2024-09-15T23:57:21.450 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T23:57:21.474 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T23:57:21.557 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906375' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on pool no app: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request sha1: 8921c8ef947807b717db041d85f0761b7914d5fc subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3393 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 294/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKTuExul2AAIKeYuusOSebmFCcJ4k4yV5vgGHAr1yENIDlAH0ga9lTWp8z/wazIA3+rFsTa6AADgoYR3nBvagkE= smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEcr+RkHirOHHIm0Xwlm7EjjzsbgRKWkVF+TG/GBw35hndqGYn7AG2AkJH4GA7HV3r/t2l8XuDHA0X07dNs7wI0= smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGM9yvaQAb4tKpe2gcLk73IqwJVeL6hzbmI8ajytgZjkf3s/6dU9JdCo0cVQlwyBYGNBuZ26Rlb7rxTe9RdS6vk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:24:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T23:57:21.558 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-15T23:57:21.558 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-15T23:57:21.559 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T23:57:21.560 INFO:teuthology.task.internal:Checking packages... 2024-09-15T23:57:21.581 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-15T23:57:21.581 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T23:57:21.581 INFO:teuthology.packaging:ref: None 2024-09-15T23:57:21.582 INFO:teuthology.packaging:tag: None 2024-09-15T23:57:21.582 INFO:teuthology.packaging:branch: squid 2024-09-15T23:57:21.582 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:57:21.582 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-15T23:57:21.780 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-15T23:57:21.782 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T23:57:21.789 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T23:57:21.789 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T23:57:21.796 INFO:teuthology.task.internal:Saving configuration 2024-09-15T23:57:21.814 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T23:57:21.822 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T23:57:21.848 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 23:52:28.155774', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKTuExul2AAIKeYuusOSebmFCcJ4k4yV5vgGHAr1yENIDlAH0ga9lTWp8z/wazIA3+rFsTa6AADgoYR3nBvagkE='} 2024-09-15T23:57:21.872 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 23:52:28.153022', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEcr+RkHirOHHIm0Xwlm7EjjzsbgRKWkVF+TG/GBw35hndqGYn7AG2AkJH4GA7HV3r/t2l8XuDHA0X07dNs7wI0='} 2024-09-15T23:57:21.896 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 23:52:28.154510', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGM9yvaQAb4tKpe2gcLk73IqwJVeL6hzbmI8ajytgZjkf3s/6dU9JdCo0cVQlwyBYGNBuZ26Rlb7rxTe9RdS6vk='} 2024-09-15T23:57:21.896 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T23:57:21.903 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-09-15T23:57:21.903 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-09-15T23:57:21.903 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-09-15T23:57:21.903 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T23:57:21.993 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1c13d5fbe0>, signals=[15]) 2024-09-15T23:57:21.993 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T23:57:22.002 INFO:teuthology.task.internal:Opening connections... 2024-09-15T23:57:22.002 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2024-09-15T23:57:22.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:57:22.081 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-09-15T23:57:22.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:57:22.160 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-09-15T23:57:22.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:57:22.235 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T23:57:22.242 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2024-09-15T23:57:22.260 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2024-09-15T23:57:22.261 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2024-09-15T23:57:22.318 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="9" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="9" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el9" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:LOGO="fedora-logo-icon" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T23:57:22.319 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T23:57:22.320 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2024-09-15T23:57:22.346 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-09-15T23:57:22.364 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-09-15T23:57:22.364 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-09-15T23:57:22.420 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2024-09-15T23:57:22.420 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2024-09-15T23:57:22.420 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2024-09-15T23:57:22.421 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2024-09-15T23:57:22.421 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2024-09-15T23:57:22.421 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2024-09-15T23:57:22.421 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T23:57:22.421 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2024-09-15T23:57:22.421 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2024-09-15T23:57:22.421 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T23:57:22.421 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2024-09-15T23:57:22.421 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T23:57:22.421 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T23:57:22.421 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T23:57:22.422 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-09-15T23:57:22.443 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-09-15T23:57:22.462 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-09-15T23:57:22.462 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-09-15T23:57:22.520 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-09-15T23:57:22.520 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2024-09-15T23:57:22.520 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-09-15T23:57:22.520 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-09-15T23:57:22.520 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2024-09-15T23:57:22.520 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2024-09-15T23:57:22.521 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T23:57:22.521 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-09-15T23:57:22.521 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2024-09-15T23:57:22.521 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T23:57:22.521 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-09-15T23:57:22.521 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T23:57:22.521 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T23:57:22.521 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T23:57:22.521 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-09-15T23:57:22.543 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T23:57:22.554 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T23:57:22.560 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T23:57:22.560 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2024-09-15T23:57:22.563 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-09-15T23:57:22.565 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-09-15T23:57:22.581 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T23:57:22.587 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T23:57:22.588 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2024-09-15T23:57:22.620 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-09-15T23:57:22.623 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-09-15T23:57:22.781 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T23:57:22.864 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T23:57:22.878 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-15T23:57:22.878 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-09-15T23:57:22.879 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-09-15T23:57:22.879 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T23:57:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:57:22.896 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-09-15T23:57:22.957 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-503.el9.x86_64 2024-09-15T23:57:22.958 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.14.0-503.el9.x86_64 2024-09-15T23:57:22.958 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum install -y kernel 2024-09-15T23:57:25.371 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-15T23:57:27.180 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-15T23:57:32.914 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.5 MB 00:01 2024-09-15T23:57:34.373 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 88 kB/s | 19 kB 00:00 2024-09-15T23:57:35.185 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-15T23:57:41.057 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-09-15T23:57:43.164 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T23:57:43.164 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T23:57:43.256 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-09-15T23:57:43.262 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-09-15T23:57:43.262 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-09-15T23:57:43.263 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-09-15T23:57:43.263 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-09-15T23:57:43.263 INFO:teuthology.orchestra.run.smithi033.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T23:57:43.263 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-09-15T23:57:43.263 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T23:57:43.263 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T23:57:43.263 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T23:57:43.263 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-15T23:57:43.263 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-09-15T23:57:43.263 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-09-15T23:57:43.263 INFO:teuthology.orchestra.run.smithi033.stdout:Install 4 Packages 2024-09-15T23:57:43.263 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-15T23:57:43.264 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 79 M 2024-09-15T23:57:43.264 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 126 M 2024-09-15T23:57:43.264 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-09-15T23:57:43.426 INFO:teuthology.orchestra.run.smithi033.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 198 kB/s | 23 kB 00:00 2024-09-15T23:57:44.501 INFO:teuthology.orchestra.run.smithi033.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-09-15T23:57:45.159 INFO:teuthology.orchestra.run.smithi033.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 16 MB/s | 29 MB 00:01 2024-09-15T23:57:45.476 INFO:teuthology.orchestra.run.smithi033.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-15T23:57:45.476 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-09-15T23:57:45.476 INFO:teuthology.orchestra.run.smithi033.stdout:Total 36 MB/s | 79 MB 00:02 2024-09-15T23:57:46.190 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-09-15T23:57:46.591 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-09-15T23:57:46.591 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-09-15T23:57:47.570 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-09-15T23:57:47.570 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-09-15T23:57:50.129 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-09-15T23:57:50.794 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T23:57:50.893 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T23:57:53.706 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T23:57:53.839 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T23:57:58.956 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T23:57:58.999 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T23:58:04.149 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T23:58:43.592 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T23:58:43.633 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T23:58:44.004 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T23:58:44.004 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T23:58:44.004 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T23:58:44.004 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T23:58:44.885 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T23:58:44.885 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-15T23:58:44.885 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-09-15T23:58:44.885 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T23:58:44.885 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T23:58:44.885 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T23:58:44.885 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T23:58:44.886 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-15T23:58:44.886 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-09-15T23:58:45.221 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T23:58:45.305 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T23:58:45.305 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T23:58:45.305 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T23:58:45.305 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T23:58:45.306 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T23:58:45.306 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-15T23:58:45.306 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T23:58:45.306 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T23:58:45.364 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T23:58:45.365 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T23:58:45.365 DEBUG:teuthology.orchestra.run.smithi033:> sudo rpm -qi grub2-tools 2024-09-15T23:58:45.403 INFO:teuthology.orchestra.run.smithi033.stdout:Name : grub2-tools 2024-09-15T23:58:45.404 INFO:teuthology.orchestra.run.smithi033.stdout:Epoch : 1 2024-09-15T23:58:45.404 INFO:teuthology.orchestra.run.smithi033.stdout:Version : 2.06 2024-09-15T23:58:45.404 INFO:teuthology.orchestra.run.smithi033.stdout:Release : 82.el9 2024-09-15T23:58:45.404 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: x86_64 2024-09-15T23:58:45.404 INFO:teuthology.orchestra.run.smithi033.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T23:58:45.404 INFO:teuthology.orchestra.run.smithi033.stdout:Group : Unspecified 2024-09-15T23:58:45.405 INFO:teuthology.orchestra.run.smithi033.stdout:Size : 8274670 2024-09-15T23:58:45.405 INFO:teuthology.orchestra.run.smithi033.stdout:License : GPLv3+ 2024-09-15T23:58:45.405 INFO:teuthology.orchestra.run.smithi033.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T23:58:45.405 INFO:teuthology.orchestra.run.smithi033.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T23:58:45.405 INFO:teuthology.orchestra.run.smithi033.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T23:58:45.405 INFO:teuthology.orchestra.run.smithi033.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T23:58:45.405 INFO:teuthology.orchestra.run.smithi033.stdout:Packager : builder@centos.org 2024-09-15T23:58:45.406 INFO:teuthology.orchestra.run.smithi033.stdout:Vendor : CentOS 2024-09-15T23:58:45.406 INFO:teuthology.orchestra.run.smithi033.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T23:58:45.406 INFO:teuthology.orchestra.run.smithi033.stdout:Summary : Support tools for GRUB. 2024-09-15T23:58:45.406 INFO:teuthology.orchestra.run.smithi033.stdout:Description : 2024-09-15T23:58:45.406 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-15T23:58:45.406 INFO:teuthology.orchestra.run.smithi033.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T23:58:45.406 INFO:teuthology.orchestra.run.smithi033.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T23:58:45.407 INFO:teuthology.orchestra.run.smithi033.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T23:58:45.407 INFO:teuthology.orchestra.run.smithi033.stdout:hardware devices. 2024-09-15T23:58:45.407 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-15T23:58:45.407 INFO:teuthology.orchestra.run.smithi033.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T23:58:45.408 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T23:58:45.409 INFO:teuthology.task.kernel:Updating grub on smithi033 to boot 5.14.0-505.el9.x86_64 2024-09-15T23:58:45.409 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T23:58:46.028 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2024-09-15T23:58:47.347 INFO:teuthology.orchestra.run.smithi033.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T23:58:47.386 INFO:teuthology.orchestra.run.smithi033.stderr:done 2024-09-15T23:58:47.388 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-09-15T23:58:47.403 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.X6R4tzZxif 2024-09-15T23:58:47.403 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.X6R4tzZxif 2024-09-15T23:58:47.469 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.X6R4tzZxif 2024-09-15T23:58:47.614 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.X6R4tzZxif is 6KB 2024-09-15T23:58:47.664 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.X6R4tzZxif 2024-09-15T23:58:47.678 DEBUG:teuthology.orchestra.run.smithi033:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T23:58:47.741 INFO:teuthology.orchestra.run.smithi033.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T23:58:47.741 INFO:teuthology.orchestra.run.smithi033.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T23:58:47.741 INFO:teuthology.orchestra.run.smithi033.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T23:58:47.741 INFO:teuthology.orchestra.run.smithi033.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T23:58:47.742 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T23:58:47.901 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2024-09-15T23:59:17.903 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-15T23:59:17.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-09-15T23:59:17.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:00:17.907 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T00:00:26.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-09-16T00:00:26.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:00:27.281 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-09-16T00:00:27.635 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-09-16T00:00:27.636 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-505.el9.x86_64"... 2024-09-16T00:00:27.636 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-09-16T00:00:27.652 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-505.el9.x86_64 2024-09-16T00:00:27.652 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T00:00:27.652 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T00:00:27.652 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T00:00:28.654 DEBUG:teuthology.parallel:result is None 2024-09-16T00:00:28.654 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T00:00:28.678 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T00:00:28.678 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T00:00:28.681 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T00:00:28.684 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T00:00:28.704 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T00:00:28.715 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T00:00:28.721 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T00:00:28.722 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T00:00:28.742 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T00:00:28.744 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T00:00:28.790 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T00:00:28.798 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T00:00:28.799 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T00:00:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T00:00:28.823 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T00:00:28.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T00:00:28.841 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T00:00:28.844 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T00:00:28.866 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T00:00:28.899 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T00:00:28.904 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T00:00:28.913 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T00:00:28.917 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T00:00:28.921 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T00:00:28.937 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T00:00:28.939 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T00:00:28.970 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T00:00:28.971 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T00:00:28.974 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T00:00:28.977 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T00:00:29.007 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T00:00:29.053 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T00:00:29.053 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T00:00:29.056 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T00:00:29.059 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T00:00:29.075 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T00:00:29.155 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T00:00:29.209 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T00:00:29.296 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T00:00:29.342 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:00:29.342 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T00:00:29.413 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T00:00:29.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T00:00:29.506 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T00:00:29.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T00:00:29.637 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-16T00:00:29.637 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T00:00:29.703 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T00:00:29.748 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T00:00:29.804 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T00:00:29.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T00:00:29.929 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T00:00:29.929 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T00:00:29.995 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2024-09-16T00:00:29.998 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-09-16T00:00:30.001 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-09-16T00:00:30.040 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T00:00:30.040 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T00:00:30.079 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T00:00:30.525 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T00:00:30.539 INFO:teuthology.task.internal:Starting timer... 2024-09-16T00:00:30.539 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T00:00:30.552 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T00:00:30.563 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-16T00:00:30.564 DEBUG:teuthology.orchestra.run.smithi033:> sudo service auditd rotate 2024-09-16T00:00:30.635 INFO:teuthology.orchestra.run.smithi033.stdout:Rotating logs: 2024-09-16T00:00:30.637 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2024-09-16T00:00:30.692 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2024-09-16T00:00:30.694 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-09-16T00:00:30.746 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-09-16T00:00:30.748 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T00:00:30.749 DEBUG:teuthology.orchestra.run.smithi033:> /usr/sbin/getenforce 2024-09-16T00:00:30.779 INFO:teuthology.orchestra.run.smithi033.stdout:Permissive 2024-09-16T00:00:30.780 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2024-09-16T00:00:30.807 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2024-09-16T00:00:30.807 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-09-16T00:00:30.842 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-09-16T00:00:30.842 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi033.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi060.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi083.front.sepia.ceph.com': 'permissive'} 2024-09-16T00:00:30.842 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T00:00:30.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T00:00:30.872 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T00:00:30.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T00:00:30.901 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T00:00:30.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T00:00:30.929 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T00:00:30.929 DEBUG:teuthology.orchestra.run.smithi033:> sudo /usr/sbin/setenforce permissive 2024-09-16T00:00:30.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2024-09-16T00:00:30.996 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-09-16T00:00:31.025 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T00:00:31.058 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T00:00:31.071 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T00:00:31.072 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi033.front.sepia.ceph.com,smithi060.front.sepia.ceph.com,smithi083.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T00:04:36.463 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi033.front.sepia.ceph.com'), Remote(name='ubuntu@smithi060.front.sepia.ceph.com'), Remote(name='ubuntu@smithi083.front.sepia.ceph.com')] 2024-09-16T00:04:36.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-09-16T00:04:36.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:04:36.569 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-09-16T00:04:36.647 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-09-16T00:04:36.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-16T00:04:36.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:04:36.744 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-09-16T00:04:36.821 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-16T00:04:36.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-16T00:04:36.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:04:36.899 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-09-16T00:04:36.976 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-16T00:04:36.977 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T00:04:36.987 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T00:04:36.987 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T00:04:36.988 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T00:04:36.990 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T00:04:36.990 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T00:04:36.993 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T00:04:36.993 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T00:04:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T00:04:37.023 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T00:04:37.038 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T00:04:37.039 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T00:04:37.043 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T00:04:37.057 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T00:04:37.064 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2024-09-16T00:04:37.067 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: ntpd: command not found 2024-09-16T00:04:37.075 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2024-09-16T00:04:37.079 INFO:teuthology.orchestra.run.smithi033.stdout:506 Cannot talk to daemon 2024-09-16T00:04:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-09-16T00:04:37.090 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T00:04:37.092 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-09-16T00:04:37.095 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T00:04:37.104 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T00:04:37.107 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T00:04:37.110 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T00:04:37.121 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T00:04:37.167 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-09-16T00:04:37.168 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-09-16T00:04:37.171 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T00:04:37.171 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-09-16T00:04:37.171 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:04:37.171 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:04:37.171 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:04:37.171 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:04:37.172 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T00:04:37.172 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-09-16T00:04:37.173 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:04:37.173 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:04:37.173 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:04:37.173 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:04:37.180 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-09-16T00:04:37.183 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T00:04:37.183 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-09-16T00:04:37.183 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:04:37.183 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:04:37.183 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:04:37.183 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:04:37.184 INFO:teuthology.run_tasks:Running task install... 2024-09-16T00:04:37.192 DEBUG:teuthology.task.install:project ceph 2024-09-16T00:04:37.193 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T00:04:37.193 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T00:04:37.193 INFO:teuthology.task.install:Using flavor: default 2024-09-16T00:04:37.198 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T00:04:37.199 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-16T00:04:37.199 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T00:04:37.200 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:04:37.202 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T00:04:37.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:04:37.203 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T00:04:37.203 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:04:37.351 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T00:04:37.351 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T00:04:37.363 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T00:04:37.363 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T00:04:37.366 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T00:04:37.366 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T00:04:37.459 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T00:04:37.460 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T00:04:37.460 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T00:04:37.471 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T00:04:37.471 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:04:37.472 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T00:04:37.486 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T00:04:37.487 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-16T00:04:37.487 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T00:04:37.488 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-16T00:04:37.488 DEBUG:teuthology.orchestra.run.smithi083:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T00:04:37.500 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-16T00:04:37.500 DEBUG:teuthology.orchestra.run.smithi033:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T00:04:37.513 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-16T00:04:37.514 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T00:04:37.553 DEBUG:teuthology.orchestra.run.smithi083:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T00:04:37.568 DEBUG:teuthology.orchestra.run.smithi033:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T00:04:37.579 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T00:04:37.628 DEBUG:teuthology.orchestra.run.smithi083:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T00:04:37.647 DEBUG:teuthology.orchestra.run.smithi033:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T00:04:37.654 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T00:04:37.694 INFO:teuthology.orchestra.run.smithi083.stdout:check_obsoletes = 1 2024-09-16T00:04:37.695 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum clean all 2024-09-16T00:04:37.714 INFO:teuthology.orchestra.run.smithi033.stdout:check_obsoletes = 1 2024-09-16T00:04:37.715 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum clean all 2024-09-16T00:04:37.718 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2024-09-16T00:04:37.719 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2024-09-16T00:04:37.920 INFO:teuthology.orchestra.run.smithi083.stdout:45 files removed 2024-09-16T00:04:37.948 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-16T00:04:37.950 INFO:teuthology.orchestra.run.smithi060.stdout:45 files removed 2024-09-16T00:04:37.959 INFO:teuthology.orchestra.run.smithi033.stdout:45 files removed 2024-09-16T00:04:37.978 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-16T00:04:37.987 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-16T00:04:38.492 INFO:teuthology.orchestra.run.smithi083.stdout:ceph packages for x86_64 310 kB/s | 83 kB 00:00 2024-09-16T00:04:38.524 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 319 kB/s | 83 kB 00:00 2024-09-16T00:04:38.553 INFO:teuthology.orchestra.run.smithi033.stdout:ceph packages for x86_64 314 kB/s | 83 kB 00:00 2024-09-16T00:04:38.776 INFO:teuthology.orchestra.run.smithi083.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-16T00:04:38.808 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-16T00:04:38.837 INFO:teuthology.orchestra.run.smithi033.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-16T00:04:39.051 INFO:teuthology.orchestra.run.smithi083.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-16T00:04:39.083 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-16T00:04:39.113 INFO:teuthology.orchestra.run.smithi033.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-16T00:04:39.718 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-16T00:04:40.350 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.2 MB 00:01 2024-09-16T00:04:40.962 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.2 MB 00:01 2024-09-16T00:04:41.703 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-16T00:04:42.617 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-16T00:04:45.979 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-16T00:04:46.563 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-16T00:04:47.169 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-16T00:04:48.313 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-16T00:04:48.653 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 19 kB 00:00 2024-09-16T00:04:49.667 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-16T00:04:49.699 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-16T00:04:50.964 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-16T00:04:53.473 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 19 kB 00:01 2024-09-16T00:04:54.260 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-16T00:04:55.607 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 3.2 kB/s | 1.7 kB 00:00 2024-09-16T00:04:55.630 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 8.0 kB/s | 1.7 kB 00:00 2024-09-16T00:04:57.875 INFO:teuthology.orchestra.run.smithi083.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T00:04:57.875 INFO:teuthology.orchestra.run.smithi083.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T00:04:57.886 INFO:teuthology.orchestra.run.smithi083.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-16T00:04:57.887 INFO:teuthology.orchestra.run.smithi083.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-16T00:04:57.888 INFO:teuthology.orchestra.run.smithi083.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-16T00:04:57.889 INFO:teuthology.orchestra.run.smithi083.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-16T00:04:57.891 INFO:teuthology.orchestra.run.smithi083.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-16T00:04:57.893 INFO:teuthology.orchestra.run.smithi083.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-16T00:04:57.894 INFO:teuthology.orchestra.run.smithi083.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-16T00:04:57.955 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-16T00:04:57.976 INFO:teuthology.orchestra.run.smithi083.stdout:====================================================================================== 2024-09-16T00:04:57.976 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-16T00:04:57.976 INFO:teuthology.orchestra.run.smithi083.stdout:====================================================================================== 2024-09-16T00:04:57.976 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-16T00:04:57.976 INFO:teuthology.orchestra.run.smithi083.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-16T00:04:57.976 INFO:teuthology.orchestra.run.smithi083.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T00:04:57.976 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T00:04:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T00:04:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T00:04:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T00:04:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T00:04:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T00:04:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T00:04:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T00:04:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T00:04:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T00:04:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T00:04:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T00:04:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-16T00:04:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-16T00:04:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-16T00:04:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-16T00:04:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-16T00:04:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-16T00:04:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T00:04:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T00:04:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T00:04:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-16T00:04:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-16T00:04:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-16T00:04:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrading: 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-16T00:04:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-16T00:04:57.980 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-16T00:04:57.980 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-16T00:04:57.980 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T00:04:57.980 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T00:04:57.980 INFO:teuthology.orchestra.run.smithi083.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-16T00:04:57.980 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-16T00:04:57.980 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-16T00:04:57.980 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T00:04:57.980 INFO:teuthology.orchestra.run.smithi083.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-16T00:04:57.980 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T00:04:57.980 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T00:04:57.980 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T00:04:57.981 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T00:04:57.982 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T00:04:57.982 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T00:04:57.982 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T00:04:57.982 INFO:teuthology.orchestra.run.smithi083.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T00:04:57.982 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T00:04:57.982 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T00:04:57.982 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T00:04:57.982 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T00:04:57.982 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T00:04:57.982 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T00:04:57.982 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T00:04:57.982 INFO:teuthology.orchestra.run.smithi083.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T00:04:57.983 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T00:04:57.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T00:04:57.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T00:04:57.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T00:04:57.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T00:04:57.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T00:04:57.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T00:04:57.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T00:04:57.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-16T00:04:57.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-16T00:04:57.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T00:04:57.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T00:04:57.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T00:04:57.985 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T00:04:57.986 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T00:04:57.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-16T00:04:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T00:04:57.989 INFO:teuthology.orchestra.run.smithi083.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-16T00:04:57.989 INFO:teuthology.orchestra.run.smithi083.stdout:Installing weak dependencies: 2024-09-16T00:04:57.989 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T00:04:57.989 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:04:57.989 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-16T00:04:57.989 INFO:teuthology.orchestra.run.smithi083.stdout:====================================================================================== 2024-09-16T00:04:57.989 INFO:teuthology.orchestra.run.smithi083.stdout:Install 148 Packages 2024-09-16T00:04:57.989 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrade 8 Packages 2024-09-16T00:04:57.989 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:04:57.990 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 226 M 2024-09-16T00:04:57.990 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-16T00:04:58.007 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T00:04:58.008 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T00:04:58.019 INFO:teuthology.orchestra.run.smithi060.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-16T00:04:58.020 INFO:teuthology.orchestra.run.smithi060.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-16T00:04:58.021 INFO:teuthology.orchestra.run.smithi060.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-16T00:04:58.022 INFO:teuthology.orchestra.run.smithi060.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-16T00:04:58.024 INFO:teuthology.orchestra.run.smithi060.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-16T00:04:58.026 INFO:teuthology.orchestra.run.smithi060.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-16T00:04:58.027 INFO:teuthology.orchestra.run.smithi060.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-16T00:04:58.095 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-16T00:04:58.123 INFO:teuthology.orchestra.run.smithi060.stdout:====================================================================================== 2024-09-16T00:04:58.123 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-16T00:04:58.124 INFO:teuthology.orchestra.run.smithi060.stdout:====================================================================================== 2024-09-16T00:04:58.124 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-16T00:04:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-16T00:04:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T00:04:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T00:04:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T00:04:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T00:04:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T00:04:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T00:04:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T00:04:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T00:04:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T00:04:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-16T00:04:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T00:04:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T00:04:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T00:04:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T00:04:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T00:04:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T00:04:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T00:04:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T00:04:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T00:04:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T00:04:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T00:04:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T00:04:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T00:04:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-16T00:04:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-16T00:04:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T00:04:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T00:04:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T00:04:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T00:04:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T00:04:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T00:04:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T00:04:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T00:04:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T00:04:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T00:04:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T00:04:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T00:04:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T00:04:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T00:04:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T00:04:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T00:04:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T00:04:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T00:04:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-16T00:04:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T00:04:58.137 INFO:teuthology.orchestra.run.smithi060.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-16T00:04:58.137 INFO:teuthology.orchestra.run.smithi060.stdout:Installing weak dependencies: 2024-09-16T00:04:58.137 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T00:04:58.137 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:04:58.137 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-16T00:04:58.137 INFO:teuthology.orchestra.run.smithi060.stdout:====================================================================================== 2024-09-16T00:04:58.137 INFO:teuthology.orchestra.run.smithi060.stdout:Install 148 Packages 2024-09-16T00:04:58.137 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 8 Packages 2024-09-16T00:04:58.137 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:04:58.138 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 226 M 2024-09-16T00:04:58.138 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-16T00:04:58.558 INFO:teuthology.orchestra.run.smithi083.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 52 kB/s | 6.5 kB 00:00 2024-09-16T00:04:58.742 INFO:teuthology.orchestra.run.smithi083.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 4.8 MB/s | 896 kB 00:00 2024-09-16T00:04:58.748 INFO:teuthology.orchestra.run.smithi060.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 51 kB/s | 6.5 kB 00:00 2024-09-16T00:04:58.842 INFO:teuthology.orchestra.run.smithi083.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 1.4 MB/s | 145 kB 00:00 2024-09-16T00:04:58.926 INFO:teuthology.orchestra.run.smithi083.stdout:(4/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 11 MB/s | 5.4 MB 00:00 2024-09-16T00:04:59.051 INFO:teuthology.orchestra.run.smithi083.stdout:(5/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 12 MB/s | 1.5 MB 00:00 2024-09-16T00:04:59.259 INFO:teuthology.orchestra.run.smithi083.stdout:(6/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 23 MB/s | 4.7 MB 00:00 2024-09-16T00:04:59.315 INFO:teuthology.orchestra.run.smithi060.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 1.5 MB/s | 896 kB 00:00 2024-09-16T00:04:59.490 INFO:teuthology.orchestra.run.smithi060.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 830 kB/s | 145 kB 00:00 2024-09-16T00:04:59.693 INFO:teuthology.orchestra.run.smithi083.stdout:(7/156): ceph-common-19.1.1-326.g8921c8ef.el9.x 17 MB/s | 21 MB 00:01 2024-09-16T00:04:59.743 INFO:teuthology.orchestra.run.smithi083.stdout:(8/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 2.6 MB/s | 2.4 MB 00:00 2024-09-16T00:04:59.819 INFO:teuthology.orchestra.run.smithi083.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 335 kB/s | 25 kB 00:00 2024-09-16T00:04:59.908 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-09-16T00:05:00.115 INFO:teuthology.orchestra.run.smithi060.stdout:(4/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 3.8 MB/s | 2.4 MB 00:00 2024-09-16T00:05:00.152 INFO:teuthology.orchestra.run.smithi083.stdout:(10/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 19 MB/s | 17 MB 00:00 2024-09-16T00:05:00.282 INFO:teuthology.orchestra.run.smithi060.stdout:(5/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 3.3 MB/s | 5.4 MB 00:01 2024-09-16T00:05:00.286 INFO:teuthology.orchestra.run.smithi083.stdout:(11/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9 18 MB/s | 11 MB 00:00 2024-09-16T00:05:00.311 INFO:teuthology.orchestra.run.smithi083.stdout:(12/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 1.4 MB/s | 228 kB 00:00 2024-09-16T00:05:00.345 INFO:teuthology.orchestra.run.smithi083.stdout:(13/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 463 kB/s | 27 kB 00:00 2024-09-16T00:05:00.362 INFO:teuthology.orchestra.run.smithi083.stdout:(14/156): libcephfs-devel-19.1.1-326.g8921c8ef. 632 kB/s | 32 kB 00:00 2024-09-16T00:05:00.445 INFO:teuthology.orchestra.run.smithi083.stdout:(15/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 7.3 MB/s | 751 kB 00:00 2024-09-16T00:05:00.466 INFO:teuthology.orchestra.run.smithi060.stdout:(6/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 4.3 MB/s | 1.5 MB 00:00 2024-09-16T00:05:00.470 INFO:teuthology.orchestra.run.smithi083.stdout:(16/156): libcephsqlite-19.1.1-326.g8921c8ef.el 1.5 MB/s | 163 kB 00:00 2024-09-16T00:05:00.496 INFO:teuthology.orchestra.run.smithi083.stdout:(17/156): librados-devel-19.1.1-326.g8921c8ef.e 2.5 MB/s | 127 kB 00:00 2024-09-16T00:05:00.546 INFO:teuthology.orchestra.run.smithi083.stdout:(18/156): libradosstriper1-19.1.1-326.g8921c8ef 6.6 MB/s | 506 kB 00:00 2024-09-16T00:05:00.638 INFO:teuthology.orchestra.run.smithi083.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 492 kB/s | 45 kB 00:00 2024-09-16T00:05:00.730 INFO:teuthology.orchestra.run.smithi083.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 1.5 MB/s | 136 kB 00:00 2024-09-16T00:05:00.830 INFO:teuthology.orchestra.run.smithi083.stdout:(21/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 16 MB/s | 5.3 MB 00:00 2024-09-16T00:05:00.833 INFO:teuthology.orchestra.run.smithi060.stdout:(7/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 8.6 MB/s | 4.7 MB 00:00 2024-09-16T00:05:00.856 INFO:teuthology.orchestra.run.smithi083.stdout:(22/156): python3-cephfs-19.1.1-326.g8921c8ef.e 1.3 MB/s | 161 kB 00:00 2024-09-16T00:05:00.906 INFO:teuthology.orchestra.run.smithi083.stdout:(23/156): python3-rados-19.1.1-326.g8921c8ef.el 4.1 MB/s | 320 kB 00:00 2024-09-16T00:05:00.957 INFO:teuthology.orchestra.run.smithi083.stdout:(24/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 2.9 MB/s | 300 kB 00:00 2024-09-16T00:05:00.982 INFO:teuthology.orchestra.run.smithi083.stdout:(25/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.3 MB/s | 99 kB 00:00 2024-09-16T00:05:01.024 INFO:teuthology.orchestra.run.smithi083.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.2 MB/s | 85 kB 00:00 2024-09-16T00:05:01.124 INFO:teuthology.orchestra.run.smithi083.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 1.7 MB/s | 170 kB 00:00 2024-09-16T00:05:01.241 INFO:teuthology.orchestra.run.smithi083.stdout:(28/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 12 MB/s | 3.1 MB 00:00 2024-09-16T00:05:01.266 INFO:teuthology.orchestra.run.smithi083.stdout:(29/156): ceph-grafana-dashboards-19.1.1-326.g8 224 kB/s | 32 kB 00:00 2024-09-16T00:05:01.317 INFO:teuthology.orchestra.run.smithi083.stdout:(30/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 1.9 MB/s | 146 kB 00:00 2024-09-16T00:05:01.541 INFO:teuthology.orchestra.run.smithi060.stdout:(8/156): ceph-common-19.1.1-326.g8921c8ef.el9.x 7.3 MB/s | 21 MB 00:02 2024-09-16T00:05:01.575 INFO:teuthology.orchestra.run.smithi083.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 12 MB/s | 3.6 MB 00:00 2024-09-16T00:05:01.600 INFO:teuthology.orchestra.run.smithi060.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 432 kB/s | 25 kB 00:00 2024-09-16T00:05:01.667 INFO:teuthology.orchestra.run.smithi083.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 2.7 MB/s | 249 kB 00:00 2024-09-16T00:05:01.734 INFO:teuthology.orchestra.run.smithi083.stdout:(33/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 740 kB/s | 49 kB 00:00 2024-09-16T00:05:01.767 INFO:teuthology.orchestra.run.smithi060.stdout:(10/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9 11 MB/s | 11 MB 00:00 2024-09-16T00:05:01.825 INFO:teuthology.orchestra.run.smithi060.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 3.8 MB/s | 228 kB 00:00 2024-09-16T00:05:01.826 INFO:teuthology.orchestra.run.smithi083.stdout:(34/156): ceph-mgr-diskprediction-local-19.1.1- 15 MB/s | 7.4 MB 00:00 2024-09-16T00:05:01.846 INFO:teuthology.orchestra.run.smithi083.stdout:(35/156): ceph-prometheus-alerts-19.1.1-326.g89 148 kB/s | 16 kB 00:00 2024-09-16T00:05:01.884 INFO:teuthology.orchestra.run.smithi060.stdout:(12/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 464 kB/s | 27 kB 00:00 2024-09-16T00:05:01.896 INFO:teuthology.orchestra.run.smithi083.stdout:(36/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 4.0 MB/s | 285 kB 00:00 2024-09-16T00:05:01.938 INFO:teuthology.orchestra.run.smithi083.stdout:(37/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 8.1 MB/s | 765 kB 00:00 2024-09-16T00:05:01.963 INFO:teuthology.orchestra.run.smithi083.stdout:(38/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 273 kB/s | 18 kB 00:00 2024-09-16T00:05:01.993 INFO:teuthology.orchestra.run.smithi060.stdout:(13/156): libcephfs-devel-19.1.1-326.g8921c8ef. 293 kB/s | 32 kB 00:00 2024-09-16T00:05:02.085 INFO:teuthology.orchestra.run.smithi060.stdout:(14/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 8.0 MB/s | 751 kB 00:00 2024-09-16T00:05:02.168 INFO:teuthology.orchestra.run.smithi060.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 1.9 MB/s | 163 kB 00:00 2024-09-16T00:05:02.243 INFO:teuthology.orchestra.run.smithi060.stdout:(16/156): librados-devel-19.1.1-326.g8921c8ef.e 1.7 MB/s | 127 kB 00:00 2024-09-16T00:05:02.335 INFO:teuthology.orchestra.run.smithi060.stdout:(17/156): libradosstriper1-19.1.1-326.g8921c8ef 5.4 MB/s | 506 kB 00:00 2024-09-16T00:05:02.354 INFO:teuthology.orchestra.run.smithi033.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T00:05:02.355 INFO:teuthology.orchestra.run.smithi033.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T00:05:02.369 INFO:teuthology.orchestra.run.smithi033.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-16T00:05:02.370 INFO:teuthology.orchestra.run.smithi033.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-16T00:05:02.371 INFO:teuthology.orchestra.run.smithi033.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-16T00:05:02.372 INFO:teuthology.orchestra.run.smithi033.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-16T00:05:02.374 INFO:teuthology.orchestra.run.smithi033.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-16T00:05:02.377 INFO:teuthology.orchestra.run.smithi033.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-16T00:05:02.378 INFO:teuthology.orchestra.run.smithi033.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-16T00:05:02.397 INFO:teuthology.orchestra.run.smithi083.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 174 kB/s | 80 kB 00:00 2024-09-16T00:05:02.445 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-09-16T00:05:02.460 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-09-16T00:05:02.460 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-09-16T00:05:02.460 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-09-16T00:05:02.460 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-09-16T00:05:02.460 INFO:teuthology.orchestra.run.smithi033.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-16T00:05:02.460 INFO:teuthology.orchestra.run.smithi033.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T00:05:02.460 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T00:05:02.460 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T00:05:02.461 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T00:05:02.461 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T00:05:02.461 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T00:05:02.461 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T00:05:02.461 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T00:05:02.461 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T00:05:02.461 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T00:05:02.461 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T00:05:02.461 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T00:05:02.461 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T00:05:02.461 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-16T00:05:02.461 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-16T00:05:02.462 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrading: 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-16T00:05:02.463 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T00:05:02.464 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T00:05:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T00:05:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T00:05:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T00:05:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T00:05:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T00:05:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T00:05:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T00:05:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T00:05:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T00:05:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T00:05:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T00:05:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T00:05:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T00:05:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T00:05:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T00:05:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-16T00:05:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T00:05:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T00:05:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T00:05:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T00:05:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T00:05:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T00:05:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T00:05:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T00:05:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T00:05:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-16T00:05:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T00:05:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-16T00:05:02.472 INFO:teuthology.orchestra.run.smithi033.stdout:Installing weak dependencies: 2024-09-16T00:05:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T00:05:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:02.473 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-09-16T00:05:02.473 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-09-16T00:05:02.473 INFO:teuthology.orchestra.run.smithi033.stdout:Install 148 Packages 2024-09-16T00:05:02.473 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrade 8 Packages 2024-09-16T00:05:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:02.473 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 226 M 2024-09-16T00:05:02.474 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-09-16T00:05:02.597 INFO:teuthology.orchestra.run.smithi083.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 957 kB/s | 191 kB 00:00 2024-09-16T00:05:02.777 INFO:teuthology.orchestra.run.smithi060.stdout:(18/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 12 MB/s | 5.3 MB 00:00 2024-09-16T00:05:02.798 INFO:teuthology.orchestra.run.smithi083.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-16T00:05:02.827 INFO:teuthology.orchestra.run.smithi060.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 905 kB/s | 45 kB 00:00 2024-09-16T00:05:02.914 INFO:teuthology.orchestra.run.smithi083.stdout:(42/156): libgfortran-11.5.0-2.el9.x86_64.rpm 842 kB/s | 801 kB 00:00 2024-09-16T00:05:02.919 INFO:teuthology.orchestra.run.smithi060.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 1.5 MB/s | 136 kB 00:00 2024-09-16T00:05:02.995 INFO:teuthology.orchestra.run.smithi060.stdout:(21/156): python3-cephfs-19.1.1-326.g8921c8ef.e 2.1 MB/s | 161 kB 00:00 2024-09-16T00:05:03.032 INFO:teuthology.orchestra.run.smithi083.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 913 kB/s | 106 kB 00:00 2024-09-16T00:05:03.072 INFO:teuthology.orchestra.run.smithi060.stdout:(22/156): python3-rados-19.1.1-326.g8921c8ef.el 4.2 MB/s | 320 kB 00:00 2024-09-16T00:05:03.145 INFO:teuthology.orchestra.run.smithi060.stdout:(23/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 3.9 MB/s | 300 kB 00:00 2024-09-16T00:05:03.149 INFO:teuthology.orchestra.run.smithi083.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-09-16T00:05:03.191 INFO:teuthology.orchestra.run.smithi083.stdout:(45/156): python3-cryptography-36.0.1-4.el9.x86 3.1 MB/s | 1.2 MB 00:00 2024-09-16T00:05:03.234 INFO:teuthology.orchestra.run.smithi033.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 42 kB/s | 6.5 kB 00:00 2024-09-16T00:05:03.484 INFO:teuthology.orchestra.run.smithi033.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 3.5 MB/s | 896 kB 00:00 2024-09-16T00:05:03.551 INFO:teuthology.orchestra.run.smithi033.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 2.1 MB/s | 145 kB 00:00 2024-09-16T00:05:03.596 INFO:teuthology.orchestra.run.smithi060.stdout:(24/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 24 MB/s | 49 MB 00:01 2024-09-16T00:05:03.613 INFO:teuthology.orchestra.run.smithi060.stdout:(25/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 213 kB/s | 99 kB 00:00 2024-09-16T00:05:03.638 INFO:teuthology.orchestra.run.smithi060.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 2.0 MB/s | 85 kB 00:00 2024-09-16T00:05:03.666 INFO:teuthology.orchestra.run.smithi083.stdout:(46/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 13 MB/s | 49 MB 00:03 2024-09-16T00:05:03.691 INFO:teuthology.orchestra.run.smithi083.stdout:(47/156): python3-requests-2.25.1-8.el9.noarch. 230 kB/s | 125 kB 00:00 2024-09-16T00:05:03.706 INFO:teuthology.orchestra.run.smithi060.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 2.5 MB/s | 170 kB 00:00 2024-09-16T00:05:03.764 INFO:teuthology.orchestra.run.smithi060.stdout:(28/156): ceph-grafana-dashboards-19.1.1-326.g8 548 kB/s | 32 kB 00:00 2024-09-16T00:05:03.783 INFO:teuthology.orchestra.run.smithi083.stdout:(48/156): python3-urllib3-1.26.5-6.el9.noarch.r 362 kB/s | 215 kB 00:00 2024-09-16T00:05:03.831 INFO:teuthology.orchestra.run.smithi060.stdout:(29/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 14 MB/s | 3.1 MB 00:00 2024-09-16T00:05:03.856 INFO:teuthology.orchestra.run.smithi060.stdout:(30/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 1.6 MB/s | 146 kB 00:00 2024-09-16T00:05:03.868 INFO:teuthology.orchestra.run.smithi033.stdout:(4/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 7.5 MB/s | 2.4 MB 00:00 2024-09-16T00:05:03.875 INFO:teuthology.orchestra.run.smithi083.stdout:(49/156): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-09-16T00:05:03.917 INFO:teuthology.orchestra.run.smithi083.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 800 kB/s | 107 kB 00:00 2024-09-16T00:05:03.959 INFO:teuthology.orchestra.run.smithi083.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 259 kB/s | 22 kB 00:00 2024-09-16T00:05:03.993 INFO:teuthology.orchestra.run.smithi083.stdout:(52/156): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-09-16T00:05:04.018 INFO:teuthology.orchestra.run.smithi083.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 3.0 MB/s | 313 kB 00:00 2024-09-16T00:05:04.043 INFO:teuthology.orchestra.run.smithi083.stdout:(54/156): flexiblas-3.0.4-8.el9.x86_64.rpm 390 kB/s | 33 kB 00:00 2024-09-16T00:05:04.051 INFO:teuthology.orchestra.run.smithi033.stdout:(5/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 5.6 MB/s | 5.4 MB 00:00 2024-09-16T00:05:04.081 INFO:teuthology.orchestra.run.smithi060.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 14 MB/s | 3.6 MB 00:00 2024-09-16T00:05:04.094 INFO:teuthology.orchestra.run.smithi083.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 242 kB/s | 18 kB 00:00 2024-09-16T00:05:04.161 INFO:teuthology.orchestra.run.smithi083.stdout:(56/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 18 MB/s | 3.0 MB 00:00 2024-09-16T00:05:04.186 INFO:teuthology.orchestra.run.smithi083.stdout:(57/156): libaio-devel-0.3.111-13.el9.x86_64.rp 78 kB/s | 11 kB 00:00 2024-09-16T00:05:04.202 INFO:teuthology.orchestra.run.smithi033.stdout:(6/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 4.6 MB/s | 1.5 MB 00:00 2024-09-16T00:05:04.207 INFO:teuthology.orchestra.run.smithi060.stdout:(32/156): ceph-mgr-diskprediction-local-19.1.1- 21 MB/s | 7.4 MB 00:00 2024-09-16T00:05:04.211 INFO:teuthology.orchestra.run.smithi083.stdout:(58/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-09-16T00:05:04.224 INFO:teuthology.orchestra.run.smithi060.stdout:(33/156): ceph-mgr-modules-core-19.1.1-326.g892 1.7 MB/s | 249 kB 00:00 2024-09-16T00:05:04.245 INFO:teuthology.orchestra.run.smithi083.stdout:(59/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 541 kB/s | 45 kB 00:00 2024-09-16T00:05:04.266 INFO:teuthology.orchestra.run.smithi060.stdout:(34/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 845 kB/s | 49 kB 00:00 2024-09-16T00:05:04.278 INFO:teuthology.orchestra.run.smithi083.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-09-16T00:05:04.291 INFO:teuthology.orchestra.run.smithi060.stdout:(35/156): ceph-prometheus-alerts-19.1.1-326.g89 246 kB/s | 16 kB 00:00 2024-09-16T00:05:04.304 INFO:teuthology.orchestra.run.smithi083.stdout:(61/156): libtool-2.4.6-46.el9.x86_64.rpm 6.2 MB/s | 581 kB 00:00 2024-09-16T00:05:04.320 INFO:teuthology.orchestra.run.smithi083.stdout:(62/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 324 kB/s | 24 kB 00:00 2024-09-16T00:05:04.325 INFO:teuthology.orchestra.run.smithi060.stdout:(36/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 4.7 MB/s | 285 kB 00:00 2024-09-16T00:05:04.375 INFO:teuthology.orchestra.run.smithi060.stdout:(37/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 9.0 MB/s | 765 kB 00:00 2024-09-16T00:05:04.379 INFO:teuthology.orchestra.run.smithi083.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-09-16T00:05:04.400 INFO:teuthology.orchestra.run.smithi060.stdout:(38/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 242 kB/s | 18 kB 00:00 2024-09-16T00:05:04.404 INFO:teuthology.orchestra.run.smithi083.stdout:(64/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 377 kB/s | 47 kB 00:00 2024-09-16T00:05:04.430 INFO:teuthology.orchestra.run.smithi083.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-09-16T00:05:04.447 INFO:teuthology.orchestra.run.smithi083.stdout:(66/156): openblas-0.3.26-2.el9.x86_64.rpm 572 kB/s | 38 kB 00:00 2024-09-16T00:05:04.460 INFO:teuthology.orchestra.run.smithi033.stdout:(7/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 12 MB/s | 4.7 MB 00:00 2024-09-16T00:05:04.581 INFO:teuthology.orchestra.run.smithi083.stdout:(67/156): openblas-openmp-0.3.26-2.el9.x86_64.r 28 MB/s | 4.9 MB 00:00 2024-09-16T00:05:04.622 INFO:teuthology.orchestra.run.smithi083.stdout:(68/156): postgresql-13.16-1.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-09-16T00:05:04.665 INFO:teuthology.orchestra.run.smithi083.stdout:(69/156): postgresql-contrib-13.16-1.el9.x86_64 11 MB/s | 910 kB 00:00 2024-09-16T00:05:04.690 INFO:teuthology.orchestra.run.smithi083.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 2.0 MB/s | 137 kB 00:00 2024-09-16T00:05:04.757 INFO:teuthology.orchestra.run.smithi083.stdout:(71/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-09-16T00:05:04.799 INFO:teuthology.orchestra.run.smithi083.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-09-16T00:05:04.824 INFO:teuthology.orchestra.run.smithi083.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-09-16T00:05:04.867 INFO:teuthology.orchestra.run.smithi060.stdout:(39/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 3.8 MB/s | 17 MB 00:04 2024-09-16T00:05:04.892 INFO:teuthology.orchestra.run.smithi060.stdout:(40/156): fuse-2.9.9-16.el9.x86_64.rpm 154 kB/s | 80 kB 00:00 2024-09-16T00:05:04.908 INFO:teuthology.orchestra.run.smithi083.stdout:(74/156): postgresql-server-13.16-1.el9.x86_64. 24 MB/s | 5.8 MB 00:00 2024-09-16T00:05:04.933 INFO:teuthology.orchestra.run.smithi083.stdout:(75/156): python3-colorama-0.4.6-3.el9.noarch.r 563 kB/s | 61 kB 00:00 2024-09-16T00:05:05.058 INFO:teuthology.orchestra.run.smithi083.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 6.0 MB 00:00 2024-09-16T00:05:05.083 INFO:teuthology.orchestra.run.smithi083.stdout:(77/156): python3-devel-3.9.19-8.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-09-16T00:05:05.094 INFO:teuthology.orchestra.run.smithi033.stdout:(8/156): ceph-common-19.1.1-326.g8921c8ef.el9.x 11 MB/s | 21 MB 00:02 2024-09-16T00:05:05.109 INFO:teuthology.orchestra.run.smithi083.stdout:(78/156): python3-jinja2-2.11.3-6.el9.noarch.rp 1.4 MB/s | 248 kB 00:00 2024-09-16T00:05:05.134 INFO:teuthology.orchestra.run.smithi083.stdout:(79/156): python3-jsonpatch-1.21-16.el9.noarch. 340 kB/s | 26 kB 00:00 2024-09-16T00:05:05.159 INFO:teuthology.orchestra.run.smithi083.stdout:(80/156): python3-jsonpointer-2.0-4.el9.noarch. 253 kB/s | 19 kB 00:00 2024-09-16T00:05:05.184 INFO:teuthology.orchestra.run.smithi060.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 868 kB/s | 253 kB 00:00 2024-09-16T00:05:05.185 INFO:teuthology.orchestra.run.smithi083.stdout:(81/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-16T00:05:05.210 INFO:teuthology.orchestra.run.smithi083.stdout:(82/156): python3-markupsafe-1.1.1-12.el9.x86_6 460 kB/s | 35 kB 00:00 2024-09-16T00:05:05.218 INFO:teuthology.orchestra.run.smithi060.stdout:(42/156): libgfortran-11.5.0-2.el9.x86_64.rpm 980 kB/s | 801 kB 00:00 2024-09-16T00:05:05.227 INFO:teuthology.orchestra.run.smithi033.stdout:(9/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9. 14 MB/s | 11 MB 00:00 2024-09-16T00:05:05.252 INFO:teuthology.orchestra.run.smithi033.stdout:(10/156): ceph-selinux-19.1.1-326.g8921c8ef.el9 159 kB/s | 25 kB 00:00 2024-09-16T00:05:05.268 INFO:teuthology.orchestra.run.smithi083.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-09-16T00:05:05.294 INFO:teuthology.orchestra.run.smithi083.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-09-16T00:05:05.311 INFO:teuthology.orchestra.run.smithi033.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 3.8 MB/s | 228 kB 00:00 2024-09-16T00:05:05.326 INFO:teuthology.orchestra.run.smithi060.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 984 kB/s | 106 kB 00:00 2024-09-16T00:05:05.369 INFO:teuthology.orchestra.run.smithi083.stdout:(85/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 5.1 MB 00:00 2024-09-16T00:05:05.394 INFO:teuthology.orchestra.run.smithi083.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 421 kB/s | 42 kB 00:00 2024-09-16T00:05:05.418 INFO:teuthology.orchestra.run.smithi060.stdout:(44/156): libquadmath-11.5.0-2.el9.x86_64.rpm 347 kB/s | 191 kB 00:00 2024-09-16T00:05:05.420 INFO:teuthology.orchestra.run.smithi083.stdout:(87/156): python3-packaging-20.9-5.el9.noarch.r 512 kB/s | 77 kB 00:00 2024-09-16T00:05:05.435 INFO:teuthology.orchestra.run.smithi060.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-09-16T00:05:05.445 INFO:teuthology.orchestra.run.smithi083.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-09-16T00:05:05.470 INFO:teuthology.orchestra.run.smithi083.stdout:(89/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-09-16T00:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout:(12/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 13 MB/s | 17 MB 00:01 2024-09-16T00:05:05.495 INFO:teuthology.orchestra.run.smithi083.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-09-16T00:05:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:(13/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 135 kB/s | 27 kB 00:00 2024-09-16T00:05:05.521 INFO:teuthology.orchestra.run.smithi083.stdout:(91/156): python3-pytz-2021.1-5.el9.noarch.rpm 674 kB/s | 51 kB 00:00 2024-09-16T00:05:05.536 INFO:teuthology.orchestra.run.smithi060.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-09-16T00:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout:(14/156): libcephfs-devel-19.1.1-326.g8921c8ef. 635 kB/s | 32 kB 00:00 2024-09-16T00:05:05.546 INFO:teuthology.orchestra.run.smithi083.stdout:(92/156): python3-requests-oauthlib-1.3.0-12.el 712 kB/s | 54 kB 00:00 2024-09-16T00:05:05.561 INFO:teuthology.orchestra.run.smithi060.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 1.7 MB/s | 215 kB 00:00 2024-09-16T00:05:05.587 INFO:teuthology.orchestra.run.smithi033.stdout:(15/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 9.8 MB/s | 751 kB 00:00 2024-09-16T00:05:05.596 INFO:teuthology.orchestra.run.smithi083.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 555 kB/s | 42 kB 00:00 2024-09-16T00:05:05.612 INFO:teuthology.orchestra.run.smithi033.stdout:(16/156): libcephsqlite-19.1.1-326.g8921c8ef.el 2.1 MB/s | 163 kB 00:00 2024-09-16T00:05:05.636 INFO:teuthology.orchestra.run.smithi060.stdout:(48/156): python3-cryptography-36.0.1-4.el9.x86 2.7 MB/s | 1.2 MB 00:00 2024-09-16T00:05:05.638 INFO:teuthology.orchestra.run.smithi083.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 480 kB/s | 44 kB 00:00 2024-09-16T00:05:05.646 INFO:teuthology.orchestra.run.smithi033.stdout:(17/156): librados-devel-19.1.1-326.g8921c8ef.e 2.1 MB/s | 127 kB 00:00 2024-09-16T00:05:05.663 INFO:teuthology.orchestra.run.smithi083.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-09-16T00:05:05.679 INFO:teuthology.orchestra.run.smithi033.stdout:(18/156): libradosstriper1-19.1.1-326.g8921c8ef 7.4 MB/s | 506 kB 00:00 2024-09-16T00:05:05.705 INFO:teuthology.orchestra.run.smithi083.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 538 kB/s | 36 kB 00:00 2024-09-16T00:05:05.731 INFO:teuthology.orchestra.run.smithi083.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 707 kB/s | 47 kB 00:00 2024-09-16T00:05:05.746 INFO:teuthology.orchestra.run.smithi033.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 678 kB/s | 45 kB 00:00 2024-09-16T00:05:05.789 INFO:teuthology.orchestra.run.smithi083.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 3.6 MB/s | 304 kB 00:00 2024-09-16T00:05:05.795 INFO:teuthology.orchestra.run.smithi060.stdout:(49/156): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 689 kB 00:00 2024-09-16T00:05:05.813 INFO:teuthology.orchestra.run.smithi033.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 2.0 MB/s | 136 kB 00:00 2024-09-16T00:05:05.815 INFO:teuthology.orchestra.run.smithi083.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 692 kB/s | 58 kB 00:00 2024-09-16T00:05:05.820 INFO:teuthology.orchestra.run.smithi060.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 583 kB/s | 107 kB 00:00 2024-09-16T00:05:05.845 INFO:teuthology.orchestra.run.smithi060.stdout:(51/156): bison-3.7.4-5.el9.x86_64.rpm 3.2 MB/s | 943 kB 00:00 2024-09-16T00:05:05.862 INFO:teuthology.orchestra.run.smithi060.stdout:(52/156): elfutils-libelf-devel-0.191-4.el9.x86 322 kB/s | 22 kB 00:00 2024-09-16T00:05:05.888 INFO:teuthology.orchestra.run.smithi060.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 777 kB/s | 33 kB 00:00 2024-09-16T00:05:05.905 INFO:teuthology.orchestra.run.smithi033.stdout:(21/156): python3-cephfs-19.1.1-326.g8921c8ef.e 1.7 MB/s | 161 kB 00:00 2024-09-16T00:05:05.930 INFO:teuthology.orchestra.run.smithi060.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-09-16T00:05:05.955 INFO:teuthology.orchestra.run.smithi060.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 273 kB/s | 18 kB 00:00 2024-09-16T00:05:05.981 INFO:teuthology.orchestra.run.smithi033.stdout:(22/156): python3-rados-19.1.1-326.g8921c8ef.el 4.2 MB/s | 320 kB 00:00 2024-09-16T00:05:05.990 INFO:teuthology.orchestra.run.smithi083.stdout:(100/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-16T00:05:06.016 INFO:teuthology.orchestra.run.smithi083.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 317 kB/s | 64 kB 00:00 2024-09-16T00:05:06.017 INFO:teuthology.orchestra.run.smithi060.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 128 kB/s | 11 kB 00:00 2024-09-16T00:05:06.041 INFO:teuthology.orchestra.run.smithi083.stdout:(102/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 381 kB/s | 96 kB 00:00 2024-09-16T00:05:06.047 INFO:teuthology.orchestra.run.smithi060.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-09-16T00:05:06.066 INFO:teuthology.orchestra.run.smithi083.stdout:(103/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 593 kB/s | 45 kB 00:00 2024-09-16T00:05:06.072 INFO:teuthology.orchestra.run.smithi060.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 828 kB/s | 45 kB 00:00 2024-09-16T00:05:06.073 INFO:teuthology.orchestra.run.smithi033.stdout:(23/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 13 MB/s | 5.3 MB 00:00 2024-09-16T00:05:06.097 INFO:teuthology.orchestra.run.smithi033.stdout:(24/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 2.5 MB/s | 300 kB 00:00 2024-09-16T00:05:06.116 INFO:teuthology.orchestra.run.smithi083.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-09-16T00:05:06.123 INFO:teuthology.orchestra.run.smithi060.stdout:(59/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-09-16T00:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout:(25/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.7 MB/s | 99 kB 00:00 2024-09-16T00:05:06.142 INFO:teuthology.orchestra.run.smithi083.stdout:(105/156): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-16T00:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.4 MB/s | 85 kB 00:00 2024-09-16T00:05:06.164 INFO:teuthology.orchestra.run.smithi060.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-16T00:05:06.176 INFO:teuthology.orchestra.run.smithi083.stdout:(106/156): gperftools-libs-2.9.1-3.el9.x86_64.r 5.1 MB/s | 308 kB 00:00 2024-09-16T00:05:06.190 INFO:teuthology.orchestra.run.smithi060.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 366 kB/s | 24 kB 00:00 2024-09-16T00:05:06.207 INFO:teuthology.orchestra.run.smithi033.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 3.4 MB/s | 170 kB 00:00 2024-09-16T00:05:06.209 INFO:teuthology.orchestra.run.smithi083.stdout:(107/156): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-09-16T00:05:06.215 INFO:teuthology.orchestra.run.smithi060.stdout:(62/156): libtool-2.4.6-46.el9.x86_64.rpm 4.0 MB/s | 581 kB 00:00 2024-09-16T00:05:06.234 INFO:teuthology.orchestra.run.smithi083.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-16T00:05:06.268 INFO:teuthology.orchestra.run.smithi083.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-16T00:05:06.291 INFO:teuthology.orchestra.run.smithi033.stdout:(28/156): ceph-grafana-dashboards-19.1.1-326.g8 383 kB/s | 32 kB 00:00 2024-09-16T00:05:06.301 INFO:teuthology.orchestra.run.smithi083.stdout:(110/156): protobuf-compiler-3.14.0-13.el9.x86_ 3.2 MB/s | 863 kB 00:00 2024-09-16T00:05:06.335 INFO:teuthology.orchestra.run.smithi083.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-09-16T00:05:06.349 INFO:teuthology.orchestra.run.smithi033.stdout:(29/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 14 MB/s | 3.1 MB 00:00 2024-09-16T00:05:06.365 INFO:teuthology.orchestra.run.smithi060.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 236 kB/s | 47 kB 00:00 2024-09-16T00:05:06.374 INFO:teuthology.orchestra.run.smithi033.stdout:(30/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 1.7 MB/s | 146 kB 00:00 2024-09-16T00:05:06.377 INFO:teuthology.orchestra.run.smithi083.stdout:(112/156): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-16T00:05:06.390 INFO:teuthology.orchestra.run.smithi060.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-16T00:05:06.416 INFO:teuthology.orchestra.run.smithi060.stdout:(65/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 292 kB 00:00 2024-09-16T00:05:06.427 INFO:teuthology.orchestra.run.smithi083.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.1 MB/s | 839 kB 00:00 2024-09-16T00:05:06.441 INFO:teuthology.orchestra.run.smithi060.stdout:(66/156): openblas-0.3.26-2.el9.x86_64.rpm 510 kB/s | 38 kB 00:00 2024-09-16T00:05:06.444 INFO:teuthology.orchestra.run.smithi083.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 2.6 MB/s | 43 kB 00:00 2024-09-16T00:05:06.470 INFO:teuthology.orchestra.run.smithi083.stdout:(115/156): python3-asyncssh-2.13.2-5.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-09-16T00:05:06.486 INFO:teuthology.orchestra.run.smithi083.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 767 kB/s | 32 kB 00:00 2024-09-16T00:05:06.512 INFO:teuthology.orchestra.run.smithi083.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 337 kB/s | 14 kB 00:00 2024-09-16T00:05:06.537 INFO:teuthology.orchestra.run.smithi083.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-16T00:05:06.541 INFO:teuthology.orchestra.run.smithi060.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-16T00:05:06.604 INFO:teuthology.orchestra.run.smithi083.stdout:(119/156): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-09-16T00:05:06.629 INFO:teuthology.orchestra.run.smithi083.stdout:(120/156): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-09-16T00:05:06.633 INFO:teuthology.orchestra.run.smithi060.stdout:(68/156): openblas-openmp-0.3.26-2.el9.x86_64.r 20 MB/s | 4.9 MB 00:00 2024-09-16T00:05:06.642 INFO:teuthology.orchestra.run.smithi033.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 12 MB/s | 3.6 MB 00:00 2024-09-16T00:05:06.655 INFO:teuthology.orchestra.run.smithi083.stdout:(121/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-09-16T00:05:06.688 INFO:teuthology.orchestra.run.smithi083.stdout:(122/156): python3-google-auth-2.34.0-1.el9.noa 2.6 MB/s | 220 kB 00:00 2024-09-16T00:05:06.692 INFO:teuthology.orchestra.run.smithi060.stdout:(69/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-09-16T00:05:06.700 INFO:teuthology.orchestra.run.smithi033.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 4.2 MB/s | 249 kB 00:00 2024-09-16T00:05:06.714 INFO:teuthology.orchestra.run.smithi083.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-09-16T00:05:06.717 INFO:teuthology.orchestra.run.smithi060.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 1.6 MB/s | 137 kB 00:00 2024-09-16T00:05:06.739 INFO:teuthology.orchestra.run.smithi083.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-09-16T00:05:06.750 INFO:teuthology.orchestra.run.smithi060.stdout:(71/156): postgresql-contrib-13.16-1.el9.x86_64 4.3 MB/s | 910 kB 00:00 2024-09-16T00:05:06.764 INFO:teuthology.orchestra.run.smithi083.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 709 kB/s | 18 kB 00:00 2024-09-16T00:05:06.789 INFO:teuthology.orchestra.run.smithi083.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-09-16T00:05:06.792 INFO:teuthology.orchestra.run.smithi033.stdout:(33/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 538 kB/s | 49 kB 00:00 2024-09-16T00:05:06.792 INFO:teuthology.orchestra.run.smithi060.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-09-16T00:05:06.814 INFO:teuthology.orchestra.run.smithi083.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-09-16T00:05:06.818 INFO:teuthology.orchestra.run.smithi060.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-09-16T00:05:06.840 INFO:teuthology.orchestra.run.smithi083.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-09-16T00:05:06.860 INFO:teuthology.orchestra.run.smithi060.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-09-16T00:05:06.865 INFO:teuthology.orchestra.run.smithi083.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-16T00:05:06.891 INFO:teuthology.orchestra.run.smithi083.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.0 MB/s | 79 kB 00:00 2024-09-16T00:05:06.916 INFO:teuthology.orchestra.run.smithi083.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-16T00:05:06.943 INFO:teuthology.orchestra.run.smithi060.stdout:(75/156): postgresql-server-13.16-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-09-16T00:05:06.949 INFO:teuthology.orchestra.run.smithi083.stdout:(132/156): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-09-16T00:05:06.968 INFO:teuthology.orchestra.run.smithi060.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-09-16T00:05:06.975 INFO:teuthology.orchestra.run.smithi083.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-09-16T00:05:07.000 INFO:teuthology.orchestra.run.smithi083.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-09-16T00:05:07.002 INFO:teuthology.orchestra.run.smithi060.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 4.1 MB/s | 248 kB 00:00 2024-09-16T00:05:07.026 INFO:teuthology.orchestra.run.smithi083.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-09-16T00:05:07.027 INFO:teuthology.orchestra.run.smithi060.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 436 kB/s | 26 kB 00:00 2024-09-16T00:05:07.051 INFO:teuthology.orchestra.run.smithi083.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-09-16T00:05:07.052 INFO:teuthology.orchestra.run.smithi060.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-09-16T00:05:07.076 INFO:teuthology.orchestra.run.smithi083.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-09-16T00:05:07.101 INFO:teuthology.orchestra.run.smithi083.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-16T00:05:07.127 INFO:teuthology.orchestra.run.smithi083.stdout:(139/156): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-09-16T00:05:07.128 INFO:teuthology.orchestra.run.smithi060.stdout:(80/156): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-09-16T00:05:07.152 INFO:teuthology.orchestra.run.smithi083.stdout:(140/156): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-09-16T00:05:07.153 INFO:teuthology.orchestra.run.smithi060.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 346 kB/s | 35 kB 00:00 2024-09-16T00:05:07.177 INFO:teuthology.orchestra.run.smithi083.stdout:(141/156): python3-typing-extensions-4.12.2-2.e 1.5 MB/s | 74 kB 00:00 2024-09-16T00:05:07.178 INFO:teuthology.orchestra.run.smithi060.stdout:(82/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-16T00:05:07.212 INFO:teuthology.orchestra.run.smithi060.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-09-16T00:05:07.217 INFO:teuthology.orchestra.run.smithi033.stdout:(34/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 24 MB/s | 49 MB 00:01 2024-09-16T00:05:07.228 INFO:teuthology.orchestra.run.smithi083.stdout:(142/156): python3-grpcio-1.46.7-10.el9.x86_64. 3.4 MB/s | 2.0 MB 00:00 2024-09-16T00:05:07.243 INFO:teuthology.orchestra.run.smithi033.stdout:(35/156): ceph-prometheus-alerts-19.1.1-326.g89 36 kB/s | 16 kB 00:00 2024-09-16T00:05:07.246 INFO:teuthology.orchestra.run.smithi060.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 3.2 MB/s | 222 kB 00:00 2024-09-16T00:05:07.261 INFO:teuthology.orchestra.run.smithi083.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-09-16T00:05:07.271 INFO:teuthology.orchestra.run.smithi060.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-09-16T00:05:07.287 INFO:teuthology.orchestra.run.smithi083.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-09-16T00:05:07.304 INFO:teuthology.orchestra.run.smithi083.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 476 kB/s | 20 kB 00:00 2024-09-16T00:05:07.305 INFO:teuthology.orchestra.run.smithi060.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 720 kB/s | 42 kB 00:00 2024-09-16T00:05:07.329 INFO:teuthology.orchestra.run.smithi083.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-16T00:05:07.330 INFO:teuthology.orchestra.run.smithi060.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-09-16T00:05:07.334 INFO:teuthology.orchestra.run.smithi033.stdout:(36/156): ceph-mgr-diskprediction-local-19.1.1- 7.7 MB/s | 7.4 MB 00:00 2024-09-16T00:05:07.354 INFO:teuthology.orchestra.run.smithi083.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-09-16T00:05:07.359 INFO:teuthology.orchestra.run.smithi033.stdout:(37/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 2.0 MB/s | 285 kB 00:00 2024-09-16T00:05:07.385 INFO:teuthology.orchestra.run.smithi033.stdout:(38/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 366 kB/s | 18 kB 00:00 2024-09-16T00:05:07.397 INFO:teuthology.orchestra.run.smithi060.stdout:(88/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 5.1 MB 00:00 2024-09-16T00:05:07.404 INFO:teuthology.orchestra.run.smithi083.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-16T00:05:07.418 INFO:teuthology.orchestra.run.smithi033.stdout:(39/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 4.3 MB/s | 765 kB 00:00 2024-09-16T00:05:07.422 INFO:teuthology.orchestra.run.smithi060.stdout:(89/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-16T00:05:07.447 INFO:teuthology.orchestra.run.smithi060.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-09-16T00:05:07.473 INFO:teuthology.orchestra.run.smithi060.stdout:(91/156): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-09-16T00:05:07.498 INFO:teuthology.orchestra.run.smithi060.stdout:(92/156): python3-requests-oauthlib-1.3.0-12.el 709 kB/s | 54 kB 00:00 2024-09-16T00:05:07.519 INFO:teuthology.orchestra.run.smithi033.stdout:(40/156): fuse-2.9.9-16.el9.x86_64.rpm 503 kB/s | 80 kB 00:00 2024-09-16T00:05:07.523 INFO:teuthology.orchestra.run.smithi060.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 829 kB/s | 42 kB 00:00 2024-09-16T00:05:07.557 INFO:teuthology.orchestra.run.smithi060.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 758 kB/s | 44 kB 00:00 2024-09-16T00:05:07.577 INFO:teuthology.orchestra.run.smithi033.stdout:(41/156): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-16T00:05:07.582 INFO:teuthology.orchestra.run.smithi060.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-16T00:05:07.615 INFO:teuthology.orchestra.run.smithi060.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 614 kB/s | 36 kB 00:00 2024-09-16T00:05:07.619 INFO:teuthology.orchestra.run.smithi033.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-09-16T00:05:07.621 INFO:teuthology.orchestra.run.smithi083.stdout:(149/156): librados2-19.1.1-326.g8921c8ef.el9.x 12 MB/s | 3.4 MB 00:00 2024-09-16T00:05:07.641 INFO:teuthology.orchestra.run.smithi060.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 808 kB/s | 47 kB 00:00 2024-09-16T00:05:07.670 INFO:teuthology.orchestra.run.smithi033.stdout:(43/156): libgfortran-11.5.0-2.el9.x86_64.rpm 2.8 MB/s | 801 kB 00:00 2024-09-16T00:05:07.680 INFO:teuthology.orchestra.run.smithi083.stdout:(150/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 9.7 MB/s | 3.2 MB 00:00 2024-09-16T00:05:07.683 INFO:teuthology.orchestra.run.smithi060.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 4.4 MB/s | 304 kB 00:00 2024-09-16T00:05:07.695 INFO:teuthology.orchestra.run.smithi033.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-09-16T00:05:07.708 INFO:teuthology.orchestra.run.smithi060.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 866 kB/s | 58 kB 00:00 2024-09-16T00:05:07.737 INFO:teuthology.orchestra.run.smithi033.stdout:(45/156): python3-cryptography-36.0.1-4.el9.x86 7.7 MB/s | 1.2 MB 00:00 2024-09-16T00:05:07.742 INFO:teuthology.orchestra.run.smithi060.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.6 MB/s | 96 kB 00:00 2024-09-16T00:05:07.762 INFO:teuthology.orchestra.run.smithi033.stdout:(46/156): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 135 kB 00:00 2024-09-16T00:05:07.768 INFO:teuthology.orchestra.run.smithi060.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-16T00:05:07.787 INFO:teuthology.orchestra.run.smithi033.stdout:(47/156): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-09-16T00:05:07.792 INFO:teuthology.orchestra.run.smithi060.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 898 kB/s | 45 kB 00:00 2024-09-16T00:05:07.812 INFO:teuthology.orchestra.run.smithi033.stdout:(48/156): python3-urllib3-1.26.5-6.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-09-16T00:05:07.851 INFO:teuthology.orchestra.run.smithi060.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-09-16T00:05:07.938 INFO:teuthology.orchestra.run.smithi033.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 855 kB/s | 107 kB 00:00 2024-09-16T00:05:07.963 INFO:teuthology.orchestra.run.smithi083.stdout:(151/156): NetworkManager-team-1.49.90-1.el9.x8 144 kB/s | 41 kB 00:00 2024-09-16T00:05:07.971 INFO:teuthology.orchestra.run.smithi033.stdout:(50/156): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-09-16T00:05:07.988 INFO:teuthology.orchestra.run.smithi033.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 434 kB/s | 22 kB 00:00 2024-09-16T00:05:08.030 INFO:teuthology.orchestra.run.smithi033.stdout:(52/156): bison-3.7.4-5.el9.x86_64.rpm 3.8 MB/s | 943 kB 00:00 2024-09-16T00:05:08.047 INFO:teuthology.orchestra.run.smithi033.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-09-16T00:05:08.060 INFO:teuthology.orchestra.run.smithi060.stdout:(104/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-09-16T00:05:08.072 INFO:teuthology.orchestra.run.smithi033.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 3.1 MB/s | 313 kB 00:00 2024-09-16T00:05:08.090 INFO:teuthology.orchestra.run.smithi033.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 432 kB/s | 18 kB 00:00 2024-09-16T00:05:08.102 INFO:teuthology.orchestra.run.smithi060.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 2.1 MB/s | 548 kB 00:00 2024-09-16T00:05:08.106 INFO:teuthology.orchestra.run.smithi033.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 330 kB/s | 11 kB 00:00 2024-09-16T00:05:08.127 INFO:teuthology.orchestra.run.smithi060.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 869 kB/s | 19 kB 00:00 2024-09-16T00:05:08.132 INFO:teuthology.orchestra.run.smithi033.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 160 kB 00:00 2024-09-16T00:05:08.149 INFO:teuthology.orchestra.run.smithi033.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-16T00:05:08.160 INFO:teuthology.orchestra.run.smithi060.stdout:(107/156): protobuf-compiler-3.14.0-13.el9.x86_ 2.3 MB/s | 863 kB 00:00 2024-09-16T00:05:08.182 INFO:teuthology.orchestra.run.smithi033.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-09-16T00:05:08.194 INFO:teuthology.orchestra.run.smithi060.stdout:(108/156): gperftools-libs-2.9.1-3.el9.x86_64.r 2.2 MB/s | 308 kB 00:00 2024-09-16T00:05:08.227 INFO:teuthology.orchestra.run.smithi060.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.5 MB/s | 49 kB 00:00 2024-09-16T00:05:08.233 INFO:teuthology.orchestra.run.smithi033.stdout:(60/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-09-16T00:05:08.249 INFO:teuthology.orchestra.run.smithi033.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 366 kB/s | 24 kB 00:00 2024-09-16T00:05:08.253 INFO:teuthology.orchestra.run.smithi060.stdout:(110/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 282 kB/s | 26 kB 00:00 2024-09-16T00:05:08.266 INFO:teuthology.orchestra.run.smithi033.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-16T00:05:08.278 INFO:teuthology.orchestra.run.smithi060.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T00:05:08.292 INFO:teuthology.orchestra.run.smithi033.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.8 MB/s | 292 kB 00:00 2024-09-16T00:05:08.312 INFO:teuthology.orchestra.run.smithi060.stdout:(112/156): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-16T00:05:08.325 INFO:teuthology.orchestra.run.smithi033.stdout:(64/156): libtool-2.4.6-46.el9.x86_64.rpm 3.2 MB/s | 581 kB 00:00 2024-09-16T00:05:08.339 INFO:teuthology.orchestra.run.smithi083.stdout:(152/156): NetworkManager-1.49.90-1.el9.x86_64. 2.5 MB/s | 2.3 MB 00:00 2024-09-16T00:05:08.342 INFO:teuthology.orchestra.run.smithi033.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 768 kB/s | 38 kB 00:00 2024-09-16T00:05:08.354 INFO:teuthology.orchestra.run.smithi060.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-09-16T00:05:08.367 INFO:teuthology.orchestra.run.smithi033.stdout:(66/156): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-09-16T00:05:08.375 INFO:teuthology.orchestra.run.smithi083.stdout:(153/156): NetworkManager-tui-1.49.90-1.el9.x86 604 kB/s | 249 kB 00:00 2024-09-16T00:05:08.387 INFO:teuthology.orchestra.run.smithi060.stdout:(114/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.6 MB/s | 839 kB 00:00 2024-09-16T00:05:08.413 INFO:teuthology.orchestra.run.smithi060.stdout:(115/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 739 kB/s | 43 kB 00:00 2024-09-16T00:05:08.430 INFO:teuthology.orchestra.run.smithi060.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 763 kB/s | 32 kB 00:00 2024-09-16T00:05:08.435 INFO:teuthology.orchestra.run.smithi033.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-09-16T00:05:08.446 INFO:teuthology.orchestra.run.smithi060.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 422 kB/s | 14 kB 00:00 2024-09-16T00:05:08.472 INFO:teuthology.orchestra.run.smithi060.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-09-16T00:05:08.481 INFO:teuthology.orchestra.run.smithi083.stdout:(154/156): openssl-3.2.2-6.el9.x86_64.rpm 9.6 MB/s | 1.4 MB 00:00 2024-09-16T00:05:08.497 INFO:teuthology.orchestra.run.smithi060.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-09-16T00:05:08.501 INFO:teuthology.orchestra.run.smithi033.stdout:(68/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2024-09-16T00:05:08.531 INFO:teuthology.orchestra.run.smithi060.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-09-16T00:05:08.556 INFO:teuthology.orchestra.run.smithi060.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-16T00:05:08.568 INFO:teuthology.orchestra.run.smithi033.stdout:(69/156): openblas-openmp-0.3.26-2.el9.x86_64.r 20 MB/s | 4.9 MB 00:00 2024-09-16T00:05:08.590 INFO:teuthology.orchestra.run.smithi060.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-09-16T00:05:08.594 INFO:teuthology.orchestra.run.smithi033.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 1.5 MB/s | 137 kB 00:00 2024-09-16T00:05:08.615 INFO:teuthology.orchestra.run.smithi060.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-09-16T00:05:08.635 INFO:teuthology.orchestra.run.smithi033.stdout:(71/156): postgresql-contrib-13.16-1.el9.x86_64 4.4 MB/s | 910 kB 00:00 2024-09-16T00:05:08.640 INFO:teuthology.orchestra.run.smithi060.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 706 kB/s | 18 kB 00:00 2024-09-16T00:05:08.665 INFO:teuthology.orchestra.run.smithi060.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 931 kB/s | 23 kB 00:00 2024-09-16T00:05:08.678 INFO:teuthology.orchestra.run.smithi033.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-09-16T00:05:08.691 INFO:teuthology.orchestra.run.smithi060.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 779 kB/s | 19 kB 00:00 2024-09-16T00:05:08.702 INFO:teuthology.orchestra.run.smithi033.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-09-16T00:05:08.716 INFO:teuthology.orchestra.run.smithi060.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 792 kB/s | 20 kB 00:00 2024-09-16T00:05:08.745 INFO:teuthology.orchestra.run.smithi033.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-09-16T00:05:08.792 INFO:teuthology.orchestra.run.smithi060.stdout:(128/156): libarrow-9.0.0-11.el9.x86_64.rpm 6.6 MB/s | 4.4 MB 00:00 2024-09-16T00:05:08.817 INFO:teuthology.orchestra.run.smithi060.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-16T00:05:08.820 INFO:teuthology.orchestra.run.smithi033.stdout:(75/156): postgresql-server-13.16-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-09-16T00:05:08.842 INFO:teuthology.orchestra.run.smithi060.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-16T00:05:08.845 INFO:teuthology.orchestra.run.smithi033.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-09-16T00:05:08.887 INFO:teuthology.orchestra.run.smithi033.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-09-16T00:05:08.892 INFO:teuthology.orchestra.run.smithi060.stdout:(131/156): python3-grpcio-1.46.7-10.el9.x86_64. 5.6 MB/s | 2.0 MB 00:00 2024-09-16T00:05:08.912 INFO:teuthology.orchestra.run.smithi033.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 384 kB/s | 26 kB 00:00 2024-09-16T00:05:08.917 INFO:teuthology.orchestra.run.smithi060.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 770 kB/s | 58 kB 00:00 2024-09-16T00:05:08.937 INFO:teuthology.orchestra.run.smithi033.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-09-16T00:05:08.943 INFO:teuthology.orchestra.run.smithi060.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 660 kB/s | 16 kB 00:00 2024-09-16T00:05:08.963 INFO:teuthology.orchestra.run.smithi033.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-16T00:05:08.968 INFO:teuthology.orchestra.run.smithi060.stdout:(134/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-09-16T00:05:08.981 INFO:teuthology.orchestra.run.smithi083.stdout:(155/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 3.9 MB/s | 2.4 MB 00:00 2024-09-16T00:05:08.988 INFO:teuthology.orchestra.run.smithi033.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 694 kB/s | 35 kB 00:00 2024-09-16T00:05:08.993 INFO:teuthology.orchestra.run.smithi060.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-09-16T00:05:09.018 INFO:teuthology.orchestra.run.smithi060.stdout:(136/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-16T00:05:09.044 INFO:teuthology.orchestra.run.smithi060.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-09-16T00:05:09.069 INFO:teuthology.orchestra.run.smithi060.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-16T00:05:09.072 INFO:teuthology.orchestra.run.smithi033.stdout:(82/156): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-16T00:05:09.094 INFO:teuthology.orchestra.run.smithi060.stdout:(139/156): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-09-16T00:05:09.097 INFO:teuthology.orchestra.run.smithi033.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-09-16T00:05:09.111 INFO:teuthology.orchestra.run.smithi060.stdout:(140/156): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-09-16T00:05:09.122 INFO:teuthology.orchestra.run.smithi033.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-09-16T00:05:09.145 INFO:teuthology.orchestra.run.smithi060.stdout:(141/156): python3-kubernetes-26.1.0-2.el9.noar 2.4 MB/s | 1.0 MB 00:00 2024-09-16T00:05:09.147 INFO:teuthology.orchestra.run.smithi033.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-16T00:05:09.165 INFO:teuthology.orchestra.run.smithi033.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-09-16T00:05:09.170 INFO:teuthology.orchestra.run.smithi060.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 987 kB/s | 74 kB 00:00 2024-09-16T00:05:09.190 INFO:teuthology.orchestra.run.smithi033.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 6.2 MB/s | 269 kB 00:00 2024-09-16T00:05:09.196 INFO:teuthology.orchestra.run.smithi060.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-16T00:05:09.198 INFO:teuthology.orchestra.run.smithi083.stdout:(156/156): NetworkManager-libnm-1.49.90-1.el9.x 1.1 MB/s | 1.8 MB 00:01 2024-09-16T00:05:09.205 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-16T00:05:09.205 INFO:teuthology.orchestra.run.smithi083.stdout:Total 20 MB/s | 226 MB 00:11 2024-09-16T00:05:09.215 INFO:teuthology.orchestra.run.smithi033.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-09-16T00:05:09.221 INFO:teuthology.orchestra.run.smithi060.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 796 kB/s | 20 kB 00:00 2024-09-16T00:05:09.240 INFO:teuthology.orchestra.run.smithi033.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-09-16T00:05:09.246 INFO:teuthology.orchestra.run.smithi060.stdout:(145/156): python3-webob-1.8.8-2.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-09-16T00:05:09.258 INFO:teuthology.orchestra.run.smithi033.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-16T00:05:09.275 INFO:teuthology.orchestra.run.smithi033.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-09-16T00:05:09.280 INFO:teuthology.orchestra.run.smithi060.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-16T00:05:09.305 INFO:teuthology.orchestra.run.smithi060.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-09-16T00:05:09.356 INFO:teuthology.orchestra.run.smithi060.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-16T00:05:09.358 INFO:teuthology.orchestra.run.smithi033.stdout:(92/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-09-16T00:05:09.375 INFO:teuthology.orchestra.run.smithi033.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-09-16T00:05:09.392 INFO:teuthology.orchestra.run.smithi033.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.3 MB/s | 44 kB 00:00 2024-09-16T00:05:09.417 INFO:teuthology.orchestra.run.smithi033.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-09-16T00:05:09.434 INFO:teuthology.orchestra.run.smithi033.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 861 kB/s | 36 kB 00:00 2024-09-16T00:05:09.451 INFO:teuthology.orchestra.run.smithi033.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-16T00:05:09.476 INFO:teuthology.orchestra.run.smithi033.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 7.1 MB/s | 304 kB 00:00 2024-09-16T00:05:09.493 INFO:teuthology.orchestra.run.smithi033.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-09-16T00:05:09.518 INFO:teuthology.orchestra.run.smithi033.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 2.2 MB/s | 96 kB 00:00 2024-09-16T00:05:09.544 INFO:teuthology.orchestra.run.smithi033.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-16T00:05:09.569 INFO:teuthology.orchestra.run.smithi033.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 893 kB/s | 45 kB 00:00 2024-09-16T00:05:09.686 INFO:teuthology.orchestra.run.smithi033.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-09-16T00:05:09.789 INFO:teuthology.orchestra.run.smithi060.stdout:(149/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 6.5 MB/s | 3.2 MB 00:00 2024-09-16T00:05:09.848 INFO:teuthology.orchestra.run.smithi060.stdout:(150/156): librados2-19.1.1-326.g8921c8ef.el9.x 6.0 MB/s | 3.4 MB 00:00 2024-09-16T00:05:09.861 INFO:teuthology.orchestra.run.smithi033.stdout:(104/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-09-16T00:05:09.911 INFO:teuthology.orchestra.run.smithi033.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 2.4 MB/s | 548 kB 00:00 2024-09-16T00:05:09.936 INFO:teuthology.orchestra.run.smithi033.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 780 kB/s | 19 kB 00:00 2024-09-16T00:05:09.978 INFO:teuthology.orchestra.run.smithi033.stdout:(107/156): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-09-16T00:05:10.012 INFO:teuthology.orchestra.run.smithi033.stdout:(108/156): gperftools-libs-2.9.1-3.el9.x86_64.r 2.0 MB/s | 308 kB 00:00 2024-09-16T00:05:10.037 INFO:teuthology.orchestra.run.smithi033.stdout:(109/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-09-16T00:05:10.062 INFO:teuthology.orchestra.run.smithi033.stdout:(110/156): liboath-2.6.7-2.el9.x86_64.rpm 1.0 MB/s | 49 kB 00:00 2024-09-16T00:05:10.088 INFO:teuthology.orchestra.run.smithi033.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T00:05:10.113 INFO:teuthology.orchestra.run.smithi033.stdout:(112/156): luarocks-3.9.2-5.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-09-16T00:05:10.131 INFO:teuthology.orchestra.run.smithi060.stdout:(151/156): NetworkManager-team-1.49.90-1.el9.x8 144 kB/s | 41 kB 00:00 2024-09-16T00:05:10.155 INFO:teuthology.orchestra.run.smithi033.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-09-16T00:05:10.180 INFO:teuthology.orchestra.run.smithi033.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-16T00:05:10.206 INFO:teuthology.orchestra.run.smithi033.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-16T00:05:10.239 INFO:teuthology.orchestra.run.smithi033.stdout:(116/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.4 MB/s | 839 kB 00:00 2024-09-16T00:05:10.264 INFO:teuthology.orchestra.run.smithi033.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 241 kB/s | 14 kB 00:00 2024-09-16T00:05:10.307 INFO:teuthology.orchestra.run.smithi033.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-09-16T00:05:10.340 INFO:teuthology.orchestra.run.smithi033.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-09-16T00:05:10.365 INFO:teuthology.orchestra.run.smithi060.stdout:(152/156): NetworkManager-1.49.90-1.el9.x86_64. 2.3 MB/s | 2.3 MB 00:01 2024-09-16T00:05:10.374 INFO:teuthology.orchestra.run.smithi033.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-09-16T00:05:10.407 INFO:teuthology.orchestra.run.smithi033.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 3.2 MB/s | 220 kB 00:00 2024-09-16T00:05:10.426 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-16T00:05:10.441 INFO:teuthology.orchestra.run.smithi033.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-09-16T00:05:10.466 INFO:teuthology.orchestra.run.smithi033.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 428 kB/s | 11 kB 00:00 2024-09-16T00:05:10.492 INFO:teuthology.orchestra.run.smithi033.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 712 kB/s | 18 kB 00:00 2024-09-16T00:05:10.498 INFO:teuthology.orchestra.run.smithi060.stdout:(153/156): openssl-3.2.2-6.el9.x86_64.rpm 10 MB/s | 1.4 MB 00:00 2024-09-16T00:05:10.517 INFO:teuthology.orchestra.run.smithi033.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 937 kB/s | 23 kB 00:00 2024-09-16T00:05:10.542 INFO:teuthology.orchestra.run.smithi033.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 780 kB/s | 19 kB 00:00 2024-09-16T00:05:10.568 INFO:teuthology.orchestra.run.smithi033.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 792 kB/s | 20 kB 00:00 2024-09-16T00:05:10.574 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-16T00:05:10.574 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-16T00:05:10.607 INFO:teuthology.orchestra.run.smithi060.stdout:(154/156): NetworkManager-tui-1.49.90-1.el9.x86 523 kB/s | 249 kB 00:00 2024-09-16T00:05:10.635 INFO:teuthology.orchestra.run.smithi033.stdout:(128/156): python3-grpcio-1.46.7-10.el9.x86_64. 7.8 MB/s | 2.0 MB 00:00 2024-09-16T00:05:10.657 INFO:teuthology.orchestra.run.smithi060.stdout:(155/156): NetworkManager-libnm-1.49.90-1.el9.x 2.1 MB/s | 1.8 MB 00:00 2024-09-16T00:05:10.660 INFO:teuthology.orchestra.run.smithi033.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-16T00:05:10.685 INFO:teuthology.orchestra.run.smithi033.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-16T00:05:10.744 INFO:teuthology.orchestra.run.smithi033.stdout:(131/156): libarrow-9.0.0-11.el9.x86_64.rpm 5.5 MB/s | 4.4 MB 00:00 2024-09-16T00:05:10.785 INFO:teuthology.orchestra.run.smithi033.stdout:(132/156): python3-kubernetes-26.1.0-2.el9.noar 4.7 MB/s | 1.0 MB 00:00 2024-09-16T00:05:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 460 kB/s | 58 kB 00:00 2024-09-16T00:05:10.836 INFO:teuthology.orchestra.run.smithi033.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-09-16T00:05:10.869 INFO:teuthology.orchestra.run.smithi033.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 853 kB/s | 28 kB 00:00 2024-09-16T00:05:10.895 INFO:teuthology.orchestra.run.smithi033.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-09-16T00:05:10.920 INFO:teuthology.orchestra.run.smithi033.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-09-16T00:05:10.945 INFO:teuthology.orchestra.run.smithi033.stdout:(138/156): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-09-16T00:05:10.970 INFO:teuthology.orchestra.run.smithi033.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-09-16T00:05:10.996 INFO:teuthology.orchestra.run.smithi033.stdout:(140/156): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-09-16T00:05:11.021 INFO:teuthology.orchestra.run.smithi033.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 475 kB/s | 36 kB 00:00 2024-09-16T00:05:11.046 INFO:teuthology.orchestra.run.smithi033.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 987 kB/s | 74 kB 00:00 2024-09-16T00:05:11.071 INFO:teuthology.orchestra.run.smithi033.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-09-16T00:05:11.097 INFO:teuthology.orchestra.run.smithi033.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-16T00:05:11.122 INFO:teuthology.orchestra.run.smithi033.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-09-16T00:05:11.148 INFO:teuthology.orchestra.run.smithi033.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-16T00:05:11.173 INFO:teuthology.orchestra.run.smithi033.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-09-16T00:05:11.231 INFO:teuthology.orchestra.run.smithi033.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-16T00:05:11.373 INFO:teuthology.orchestra.run.smithi033.stdout:(149/156): NetworkManager-1.49.90-1.el9.x86_64. 17 MB/s | 2.3 MB 00:00 2024-09-16T00:05:11.449 INFO:teuthology.orchestra.run.smithi033.stdout:(150/156): librados2-19.1.1-326.g8921c8ef.el9.x 11 MB/s | 3.4 MB 00:00 2024-09-16T00:05:11.490 INFO:teuthology.orchestra.run.smithi033.stdout:(151/156): NetworkManager-libnm-1.49.90-1.el9.x 15 MB/s | 1.8 MB 00:00 2024-09-16T00:05:11.549 INFO:teuthology.orchestra.run.smithi033.stdout:(152/156): NetworkManager-team-1.49.90-1.el9.x8 407 kB/s | 41 kB 00:00 2024-09-16T00:05:11.574 INFO:teuthology.orchestra.run.smithi033.stdout:(153/156): NetworkManager-tui-1.49.90-1.el9.x86 2.9 MB/s | 249 kB 00:00 2024-09-16T00:05:11.607 INFO:teuthology.orchestra.run.smithi060.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 2.2 MB/s | 2.4 MB 00:01 2024-09-16T00:05:11.613 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-16T00:05:11.614 INFO:teuthology.orchestra.run.smithi060.stdout:Total 17 MB/s | 226 MB 00:13 2024-09-16T00:05:11.633 INFO:teuthology.orchestra.run.smithi033.stdout:(154/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 6.9 MB/s | 3.2 MB 00:00 2024-09-16T00:05:11.675 INFO:teuthology.orchestra.run.smithi033.stdout:(155/156): openssl-3.2.2-6.el9.x86_64.rpm 11 MB/s | 1.4 MB 00:00 2024-09-16T00:05:11.716 INFO:teuthology.orchestra.run.smithi033.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 17 MB/s | 2.4 MB 00:00 2024-09-16T00:05:11.723 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-09-16T00:05:11.723 INFO:teuthology.orchestra.run.smithi033.stdout:Total 24 MB/s | 226 MB 00:09 2024-09-16T00:05:12.380 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-16T00:05:12.380 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-16T00:05:12.875 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-16T00:05:12.953 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-09-16T00:05:13.036 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-16T00:05:13.036 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-16T00:05:13.122 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-09-16T00:05:13.122 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-09-16T00:05:14.934 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-16T00:05:14.934 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-16T00:05:15.067 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-09-16T00:05:15.068 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-09-16T00:05:18.868 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-16T00:05:18.994 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-16T00:05:19.033 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-16T00:05:19.086 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-16T00:05:19.120 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-16T00:05:19.145 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-16T00:05:19.188 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-16T00:05:19.517 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-16T00:05:19.630 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:05:19.752 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:05:19.780 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:05:19.880 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:05:19.910 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T00:05:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-16T00:05:20.124 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-16T00:05:20.152 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:05:20.255 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:05:20.277 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T00:05:20.341 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T00:05:20.370 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T00:05:20.398 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T00:05:20.606 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T00:05:20.745 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T00:05:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-16T00:05:20.968 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:05:21.049 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:05:21.264 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:05:21.372 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-16T00:05:21.446 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-09-16T00:05:21.474 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-16T00:05:21.518 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-16T00:05:21.561 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-16T00:05:21.570 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-16T00:05:21.579 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-16T00:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-16T00:05:21.635 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-16T00:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-16T00:05:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-16T00:05:21.699 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-16T00:05:21.721 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-16T00:05:21.739 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-16T00:05:21.755 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-16T00:05:21.783 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-16T00:05:21.809 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-16T00:05:21.814 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-16T00:05:21.816 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-16T00:05:21.843 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-16T00:05:21.913 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-16T00:05:21.966 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-16T00:05:22.004 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-16T00:05:22.025 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-16T00:05:22.040 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-16T00:05:22.064 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-16T00:05:22.096 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-16T00:05:22.133 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-16T00:05:22.167 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-16T00:05:22.168 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-16T00:05:22.210 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-16T00:05:22.239 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-16T00:05:22.269 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-16T00:05:22.309 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-16T00:05:22.354 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-16T00:05:22.381 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:05:22.437 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-16T00:05:22.489 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:05:22.513 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-16T00:05:22.524 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:05:22.581 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-16T00:05:22.616 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:05:22.631 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-16T00:05:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:05:22.645 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T00:05:22.764 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:05:22.798 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:05:22.816 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-16T00:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-16T00:05:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:05:22.903 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:05:22.927 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T00:05:22.991 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:05:23.020 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T00:05:23.075 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T00:05:23.083 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-16T00:05:23.096 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T00:05:23.122 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T00:05:23.132 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-16T00:05:23.160 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:05:23.174 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-16T00:05:23.261 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-16T00:05:23.265 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:05:23.289 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-16T00:05:23.297 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T00:05:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T00:05:23.332 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-16T00:05:23.357 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T00:05:23.370 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-16T00:05:23.379 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T00:05:23.402 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T00:05:23.412 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-16T00:05:23.490 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T00:05:23.571 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-16T00:05:23.598 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T00:05:23.603 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-16T00:05:23.651 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-16T00:05:23.706 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-16T00:05:23.727 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:05:23.747 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T00:05:23.780 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-16T00:05:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:05:23.819 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-16T00:05:23.854 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-16T00:05:23.877 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-16T00:05:23.902 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-16T00:05:23.937 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-16T00:05:23.975 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:05:23.980 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-16T00:05:24.023 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-16T00:05:24.036 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:05:24.071 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:05:24.148 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-16T00:05:24.168 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-16T00:05:24.225 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-16T00:05:24.252 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-16T00:05:24.293 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-16T00:05:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:05:24.329 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-16T00:05:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-16T00:05:24.406 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-16T00:05:24.463 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-16T00:05:24.501 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-16T00:05:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-16T00:05:24.536 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-16T00:05:24.538 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-16T00:05:24.546 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-16T00:05:24.579 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-16T00:05:24.582 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-16T00:05:24.606 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-16T00:05:24.642 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-16T00:05:24.657 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-16T00:05:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-16T00:05:24.679 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-16T00:05:24.708 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-16T00:05:24.714 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-16T00:05:24.740 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-16T00:05:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-16T00:05:24.765 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-16T00:05:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-16T00:05:24.801 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-16T00:05:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-16T00:05:24.841 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-16T00:05:24.851 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-16T00:05:24.923 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-16T00:05:24.970 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-16T00:05:24.983 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-16T00:05:25.012 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-16T00:05:25.016 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-16T00:05:25.018 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-16T00:05:25.044 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-16T00:05:25.048 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-16T00:05:25.053 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-16T00:05:25.089 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-16T00:05:25.126 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-16T00:05:25.134 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-16T00:05:25.140 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-16T00:05:25.159 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-16T00:05:25.165 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-16T00:05:25.180 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-16T00:05:25.195 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-16T00:05:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-16T00:05:25.230 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-16T00:05:25.231 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-16T00:05:25.239 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-16T00:05:25.263 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-16T00:05:25.269 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-16T00:05:25.303 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-16T00:05:25.305 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-16T00:05:25.339 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-16T00:05:25.341 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-16T00:05:25.354 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-16T00:05:25.384 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-16T00:05:25.439 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-16T00:05:25.443 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-16T00:05:25.461 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-16T00:05:25.495 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-16T00:05:25.535 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-16T00:05:25.584 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-16T00:05:25.598 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-16T00:05:25.639 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-16T00:05:25.640 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-16T00:05:25.662 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-16T00:05:25.694 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-16T00:05:26.192 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-16T00:05:26.225 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-16T00:05:26.278 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-16T00:05:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-16T00:05:26.326 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-16T00:05:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-16T00:05:26.361 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-16T00:05:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-16T00:05:26.413 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-16T00:05:26.431 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-16T00:05:26.437 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-16T00:05:26.450 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-16T00:05:26.483 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-16T00:05:26.493 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-16T00:05:26.525 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-16T00:05:26.608 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-16T00:05:26.627 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-16T00:05:26.657 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-16T00:05:26.664 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-16T00:05:26.681 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-16T00:05:26.712 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-16T00:05:26.713 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-16T00:05:26.726 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-16T00:05:26.768 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-16T00:05:26.787 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-16T00:05:26.815 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-16T00:05:26.833 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-16T00:05:26.850 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-16T00:05:26.862 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-16T00:05:26.870 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-16T00:05:26.887 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-16T00:05:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-16T00:05:26.923 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-16T00:05:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-16T00:05:26.971 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-16T00:05:26.996 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-16T00:05:27.006 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-16T00:05:27.046 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-16T00:05:27.050 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-16T00:05:27.054 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-16T00:05:27.084 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-16T00:05:27.092 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-16T00:05:27.162 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-16T00:05:27.188 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-16T00:05:27.234 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-16T00:05:27.346 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-16T00:05:27.373 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-16T00:05:27.394 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-16T00:05:27.423 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-16T00:05:27.500 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-16T00:05:27.535 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-16T00:05:27.538 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-16T00:05:27.573 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-16T00:05:27.575 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-16T00:05:27.608 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-16T00:05:27.612 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-16T00:05:27.633 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-16T00:05:27.638 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-16T00:05:27.663 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-16T00:05:27.732 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-16T00:05:27.740 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-16T00:05:27.809 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-16T00:05:27.866 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-16T00:05:27.879 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-16T00:05:27.899 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-16T00:05:27.914 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-16T00:05:27.917 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-16T00:05:27.927 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-16T00:05:27.942 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-16T00:05:27.970 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-16T00:05:27.984 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-16T00:05:28.017 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-16T00:05:28.024 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-16T00:05:28.057 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-16T00:05:28.064 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-16T00:05:28.087 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-16T00:05:28.093 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-16T00:05:28.121 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-16T00:05:28.128 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-16T00:05:28.154 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-16T00:05:28.161 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-16T00:05:28.197 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-16T00:05:28.204 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-16T00:05:28.230 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-16T00:05:28.237 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-16T00:05:28.324 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-16T00:05:28.327 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-16T00:05:28.373 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-16T00:05:28.377 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-16T00:05:28.403 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-16T00:05:28.408 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-16T00:05:28.512 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-16T00:05:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-16T00:05:28.568 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-16T00:05:28.573 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-16T00:05:29.154 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-16T00:05:29.239 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-16T00:05:29.267 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-16T00:05:29.300 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-16T00:05:29.324 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-16T00:05:29.335 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-16T00:05:29.372 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-16T00:05:29.385 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-16T00:05:29.406 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-16T00:05:29.419 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-16T00:05:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-16T00:05:29.521 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-16T00:05:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-16T00:05:29.574 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-16T00:05:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-16T00:05:29.609 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-16T00:05:29.619 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-16T00:05:29.671 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T00:05:29.697 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-16T00:05:29.709 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-16T00:05:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-16T00:05:29.746 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-16T00:05:29.762 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T00:05:29.801 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-16T00:05:29.846 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-16T00:05:29.924 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-16T00:05:29.939 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-16T00:05:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-16T00:05:29.977 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-16T00:05:30.035 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-16T00:05:30.057 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-16T00:05:30.328 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-16T00:05:30.366 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T00:05:30.437 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T00:05:30.468 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-16T00:05:30.719 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-16T00:05:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-16T00:05:30.848 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-16T00:05:30.878 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-16T00:05:32.132 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-16T00:05:32.210 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-16T00:05:32.238 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-16T00:05:32.240 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-16T00:05:32.270 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-16T00:05:32.323 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-16T00:05:32.350 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-16T00:05:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-16T00:05:32.596 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-16T00:05:32.650 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T00:05:32.742 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T00:05:32.766 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:05:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-16T00:05:32.807 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:05:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-16T00:05:32.954 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-16T00:05:32.996 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T00:05:33.096 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T00:05:33.126 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-16T00:05:33.163 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-16T00:05:33.316 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-16T00:05:33.353 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:05:33.353 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T00:05:33.393 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T00:05:33.424 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T00:05:33.463 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-16T00:05:33.480 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T00:05:33.480 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T00:05:33.480 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T00:05:33.480 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:05:33.511 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T00:05:33.669 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-16T00:05:33.708 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T00:05:33.779 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T00:05:33.811 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-16T00:05:35.861 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:05:35.893 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:05:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:05:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:05:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:05:36.513 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T00:05:36.592 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T00:05:36.592 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T00:05:36.592 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T00:05:36.592 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:05:36.631 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T00:05:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:05:36.904 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T00:05:36.999 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T00:05:36.999 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T00:05:36.999 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T00:05:37.000 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:37.030 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T00:05:45.598 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T00:05:45.598 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /sys 2024-09-16T00:05:45.598 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /proc 2024-09-16T00:05:45.598 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /mnt 2024-09-16T00:05:45.598 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /var/tmp 2024-09-16T00:05:45.598 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /home 2024-09-16T00:05:45.598 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /root 2024-09-16T00:05:45.599 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /tmp 2024-09-16T00:05:45.599 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:05:45.767 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T00:05:45.783 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T00:05:45.783 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:45.783 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T00:05:45.783 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:05:47.245 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T00:05:47.272 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T00:05:47.272 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:47.272 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T00:05:47.272 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T00:05:47.272 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T00:05:47.273 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:05:47.496 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T00:05:47.519 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T00:05:47.519 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:47.519 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T00:05:47.519 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T00:05:47.519 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T00:05:47.519 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:05:47.971 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T00:05:47.995 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T00:05:47.995 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:47.995 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T00:05:47.995 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T00:05:47.995 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T00:05:47.995 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:05:48.110 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-16T00:05:48.167 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-16T00:05:48.222 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-16T00:05:48.411 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-16T00:05:48.464 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-16T00:05:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-16T00:05:48.631 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-16T00:05:48.772 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-16T00:05:48.823 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-16T00:05:48.854 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-16T00:05:48.882 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-16T00:05:49.004 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T00:05:49.004 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2024-09-16T00:05:49.005 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2024-09-16T00:05:49.005 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2024-09-16T00:05:49.005 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2024-09-16T00:05:49.005 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2024-09-16T00:05:49.005 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /root 2024-09-16T00:05:49.006 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2024-09-16T00:05:49.006 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:05:49.130 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T00:05:49.131 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /sys 2024-09-16T00:05:49.131 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /proc 2024-09-16T00:05:49.131 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /mnt 2024-09-16T00:05:49.131 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /var/tmp 2024-09-16T00:05:49.131 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /home 2024-09-16T00:05:49.131 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /root 2024-09-16T00:05:49.131 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /tmp 2024-09-16T00:05:49.131 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:49.192 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T00:05:49.208 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T00:05:49.208 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:49.208 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T00:05:49.209 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:05:49.277 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T00:05:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T00:05:49.317 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T00:05:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T00:05:49.333 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:49.333 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T00:05:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:49.354 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-16T00:05:49.417 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-16T00:05:49.446 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T00:05:50.492 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T00:05:50.531 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T00:05:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T00:05:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T00:05:50.773 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:50.773 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T00:05:50.773 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T00:05:50.773 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T00:05:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:05:50.861 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T00:05:50.889 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T00:05:50.889 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:50.889 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T00:05:50.889 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T00:05:50.889 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T00:05:50.889 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:51.012 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T00:05:51.036 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T00:05:51.036 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:51.036 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T00:05:51.036 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T00:05:51.036 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T00:05:51.036 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:05:51.128 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T00:05:51.152 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T00:05:51.153 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:51.153 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T00:05:51.153 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T00:05:51.153 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T00:05:51.153 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:51.495 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T00:05:51.505 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T00:05:51.528 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T00:05:51.528 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:51.529 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T00:05:51.529 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T00:05:51.529 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T00:05:51.529 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:05:51.532 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T00:05:51.612 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T00:05:51.635 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T00:05:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-16T00:05:51.636 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T00:05:51.636 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:51.637 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T00:05:51.637 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T00:05:51.637 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T00:05:51.637 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-16T00:05:51.743 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-16T00:05:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-16T00:05:51.794 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-16T00:05:51.801 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-16T00:05:51.826 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T00:05:51.849 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T00:05:51.850 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:51.850 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T00:05:51.850 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T00:05:51.850 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T00:05:51.851 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:05:51.865 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-16T00:05:51.909 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T00:05:51.925 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T00:05:51.949 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-16T00:05:51.998 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-16T00:05:52.019 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-16T00:05:52.060 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-16T00:05:52.075 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-16T00:05:52.078 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-16T00:05:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-16T00:05:52.180 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-16T00:05:52.260 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-16T00:05:52.317 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-16T00:05:52.349 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-16T00:05:52.356 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-16T00:05:52.396 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-16T00:05:52.423 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-16T00:05:52.502 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-16T00:05:52.547 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-16T00:05:52.579 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-16T00:05:52.606 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-16T00:05:52.894 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T00:05:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T00:05:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-16T00:05:52.999 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-16T00:05:53.018 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-16T00:05:53.029 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T00:05:53.053 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T00:05:53.053 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:53.053 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T00:05:53.053 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T00:05:53.053 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T00:05:53.053 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:05:53.054 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T00:05:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T00:05:53.096 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T00:05:53.104 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T00:05:53.119 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T00:05:53.119 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:53.119 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T00:05:53.119 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:05:53.137 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-16T00:05:53.163 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T00:05:53.187 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T00:05:53.187 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:53.187 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-16T00:05:53.187 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T00:05:53.188 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T00:05:53.188 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:05:53.203 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-16T00:05:53.228 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T00:05:53.489 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T00:05:53.513 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T00:05:53.513 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:53.513 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T00:05:53.513 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T00:05:53.513 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T00:05:53.513 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:05:54.151 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T00:05:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T00:05:54.330 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T00:05:54.371 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T00:05:55.210 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T00:05:55.239 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T00:05:55.347 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T00:05:55.383 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T00:05:55.413 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T00:05:55.508 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-16T00:05:55.520 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T00:05:55.533 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T00:05:55.557 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T00:05:55.557 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:55.557 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T00:05:55.557 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T00:05:55.557 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T00:05:55.558 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:05:55.609 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T00:05:55.620 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T00:05:55.700 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-16T00:05:55.722 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-16T00:05:55.731 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T00:05:55.757 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T00:05:55.758 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:55.758 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T00:05:55.758 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T00:05:55.758 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T00:05:55.758 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:55.774 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-16T00:05:55.809 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T00:05:55.819 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T00:05:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-16T00:05:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-16T00:05:56.759 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-16T00:05:56.803 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T00:05:56.827 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T00:05:56.827 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:56.827 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T00:05:56.827 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T00:05:56.827 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T00:05:56.827 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:05:56.871 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T00:05:56.893 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T00:05:56.893 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:56.893 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T00:05:56.893 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:05:56.946 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T00:05:56.949 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-16T00:05:56.969 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T00:05:56.969 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:56.969 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-16T00:05:56.969 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T00:05:56.969 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T00:05:56.969 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:05:56.976 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T00:05:57.001 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T00:05:57.002 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:57.002 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T00:05:57.002 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T00:05:57.002 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T00:05:57.002 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:57.052 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T00:05:57.077 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T00:05:57.077 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:57.077 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T00:05:57.077 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:57.128 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T00:05:57.152 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T00:05:57.153 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:57.153 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-16T00:05:57.153 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T00:05:57.153 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T00:05:57.153 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:57.288 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T00:05:57.312 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T00:05:57.312 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:57.312 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T00:05:57.312 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T00:05:57.312 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T00:05:57.312 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:05:57.485 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T00:05:57.510 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T00:05:57.511 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:05:57.511 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T00:05:57.511 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T00:05:57.511 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T00:05:57.511 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:05:58.093 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-16T00:05:58.143 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-16T00:05:58.192 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-16T00:05:58.224 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-16T00:05:58.251 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-16T00:05:58.442 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-16T00:05:58.490 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-16T00:05:58.520 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-16T00:05:58.558 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-16T00:05:58.604 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-16T00:05:59.228 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:05:59.269 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:05:59.289 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:06:00.122 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-16T00:06:00.216 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-16T00:06:00.257 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-16T00:06:00.340 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-16T00:06:00.369 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-16T00:06:00.394 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-16T00:06:00.429 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-16T00:06:00.429 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T00:06:00.521 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T00:06:00.522 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T00:06:00.606 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T00:06:00.631 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-16T00:06:00.636 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:06:00.636 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:06:00.816 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:06:00.816 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T00:06:00.900 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T00:06:00.964 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:06:02.117 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-16T00:06:02.159 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-16T00:06:02.209 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-16T00:06:02.240 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-16T00:06:02.267 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-16T00:06:02.357 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-16T00:06:02.399 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-16T00:06:02.441 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-16T00:06:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-16T00:06:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-16T00:06:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-16T00:06:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-16T00:06:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-16T00:06:02.557 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-16T00:06:02.595 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-16T00:06:02.700 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-16T00:06:02.745 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-16T00:06:02.776 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-16T00:06:02.805 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-16T00:06:02.842 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-16T00:06:03.150 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:06:03.184 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:06:03.204 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:06:03.446 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:06:03.490 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:06:03.511 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:06:04.083 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-16T00:06:04.172 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-16T00:06:04.213 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-16T00:06:04.297 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-16T00:06:04.324 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-16T00:06:04.358 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-16T00:06:04.393 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-16T00:06:04.393 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T00:06:04.415 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-16T00:06:04.485 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T00:06:04.485 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T00:06:04.503 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-16T00:06:04.562 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-16T00:06:04.569 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T00:06:04.594 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-16T00:06:04.600 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:06:04.600 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:06:04.655 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-16T00:06:04.689 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-16T00:06:04.715 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-16T00:06:04.749 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-16T00:06:04.749 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T00:06:04.788 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:06:04.788 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T00:06:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T00:06:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T00:06:04.855 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T00:06:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T00:06:04.930 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:06:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-16T00:06:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:06:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:06:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:06:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T00:06:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T00:06:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:06:06.858 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:06:06.858 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-16T00:06:06.858 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-16T00:06:06.858 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-16T00:06:06.859 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-16T00:06:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-16T00:06:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-16T00:06:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-16T00:06:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-16T00:06:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-16T00:06:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-16T00:06:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-16T00:06:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-16T00:06:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-16T00:06:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-16T00:06:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-16T00:06:06.860 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-16T00:06:06.861 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-16T00:06:06.861 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-16T00:06:06.874 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-16T00:06:06.874 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-16T00:06:06.874 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-16T00:06:06.874 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-16T00:06:06.874 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-16T00:06:06.874 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-16T00:06:06.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-16T00:06:06.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-16T00:06:06.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-16T00:06:06.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-16T00:06:06.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-16T00:06:06.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-16T00:06:06.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-16T00:06:06.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-16T00:06:06.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-16T00:06:06.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-16T00:06:06.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-16T00:06:06.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-16T00:06:06.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-16T00:06:06.877 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-16T00:06:06.878 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-16T00:06:06.879 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-16T00:06:06.880 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-16T00:06:06.881 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-16T00:06:06.882 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-16T00:06:06.883 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-16T00:06:06.884 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-16T00:06:06.884 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-16T00:06:06.884 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-16T00:06:06.884 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-16T00:06:06.884 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-16T00:06:06.884 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-16T00:06:06.884 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-16T00:06:06.884 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-16T00:06:09.476 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:06:09.476 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:06:09.476 INFO:teuthology.orchestra.run.smithi083.stdout:Upgraded: 2024-09-16T00:06:09.476 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-16T00:06:09.476 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-16T00:06:09.476 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-16T00:06:09.476 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-16T00:06:09.476 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.476 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.476 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-16T00:06:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-16T00:06:09.477 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-16T00:06:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T00:06:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: automake-1.16.2-8.el9.noarch 2024-09-16T00:06:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: bison-3.7.4-5.el9.x86_64 2024-09-16T00:06:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T00:06:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.478 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.479 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:09.479 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:09.479 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.479 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.479 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:09.479 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-16T00:06:09.479 INFO:teuthology.orchestra.run.smithi083.stdout: flex-2.6.4-9.el9.x86_64 2024-09-16T00:06:09.479 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T00:06:09.479 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T00:06:09.479 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T00:06:09.479 INFO:teuthology.orchestra.run.smithi083.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-16T00:06:09.479 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T00:06:09.480 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T00:06:09.481 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-13.16-1.el9.x86_64 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T00:06:09.482 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.483 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.483 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.483 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T00:06:09.483 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T00:06:09.483 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T00:06:09.483 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T00:06:09.483 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-16T00:06:09.483 INFO:teuthology.orchestra.run.smithi083.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-16T00:06:09.483 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T00:06:09.483 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T00:06:09.483 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T00:06:09.483 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T00:06:09.484 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T00:06:09.484 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T00:06:09.484 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T00:06:09.484 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T00:06:09.484 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T00:06:09.484 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T00:06:09.484 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T00:06:09.484 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T00:06:09.484 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T00:06:09.484 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T00:06:09.484 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T00:06:09.484 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T00:06:09.485 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T00:06:09.485 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T00:06:09.485 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T00:06:09.485 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T00:06:09.485 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T00:06:09.485 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T00:06:09.485 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T00:06:09.485 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T00:06:09.485 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T00:06:09.485 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T00:06:09.485 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T00:06:09.485 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T00:06:09.486 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T00:06:09.486 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T00:06:09.486 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T00:06:09.486 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T00:06:09.486 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-16T00:06:09.486 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T00:06:09.486 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.486 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.486 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T00:06:09.486 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T00:06:09.486 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T00:06:09.486 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T00:06:09.487 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T00:06:09.488 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T00:06:09.488 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T00:06:09.488 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.488 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.488 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:09.488 INFO:teuthology.orchestra.run.smithi083.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T00:06:09.488 INFO:teuthology.orchestra.run.smithi083.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T00:06:09.488 INFO:teuthology.orchestra.run.smithi083.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T00:06:09.488 INFO:teuthology.orchestra.run.smithi083.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-16T00:06:09.488 INFO:teuthology.orchestra.run.smithi083.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-16T00:06:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T00:06:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-16T00:06:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:06:09.489 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-16T00:06:09.950 DEBUG:teuthology.parallel:result is None 2024-09-16T00:06:10.590 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:06:10.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-16T00:06:10.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-16T00:06:10.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-16T00:06:10.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-16T00:06:10.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-16T00:06:10.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-16T00:06:10.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-16T00:06:10.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:06:10.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-16T00:06:10.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-16T00:06:10.592 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-16T00:06:10.592 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-16T00:06:10.592 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-16T00:06:10.592 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-16T00:06:10.592 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-16T00:06:10.592 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-16T00:06:10.592 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-16T00:06:10.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-16T00:06:10.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-16T00:06:10.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-16T00:06:10.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-16T00:06:10.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-16T00:06:10.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-16T00:06:10.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-16T00:06:10.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-16T00:06:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-16T00:06:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-16T00:06:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-16T00:06:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-16T00:06:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-16T00:06:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-16T00:06:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-16T00:06:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-16T00:06:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-16T00:06:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-16T00:06:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-16T00:06:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-16T00:06:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-16T00:06:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-16T00:06:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-16T00:06:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-16T00:06:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-16T00:06:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-16T00:06:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-16T00:06:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-16T00:06:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-16T00:06:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-16T00:06:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-16T00:06:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-16T00:06:10.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-16T00:06:10.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-16T00:06:10.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-16T00:06:10.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-16T00:06:10.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-16T00:06:10.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-16T00:06:10.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-16T00:06:10.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-16T00:06:10.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-16T00:06:10.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-16T00:06:10.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-16T00:06:10.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-16T00:06:10.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-16T00:06:10.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-16T00:06:10.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-16T00:06:10.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-16T00:06:10.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-16T00:06:10.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-16T00:06:10.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-16T00:06:10.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-16T00:06:10.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-16T00:06:10.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-16T00:06:10.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-16T00:06:10.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-16T00:06:10.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-16T00:06:10.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-16T00:06:10.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-16T00:06:10.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-16T00:06:10.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-16T00:06:10.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-16T00:06:10.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-16T00:06:10.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-16T00:06:10.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-16T00:06:10.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-16T00:06:10.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-16T00:06:10.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-16T00:06:10.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-16T00:06:10.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-16T00:06:10.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-16T00:06:10.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-16T00:06:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-16T00:06:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-16T00:06:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-16T00:06:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-16T00:06:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-16T00:06:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-16T00:06:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-16T00:06:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-16T00:06:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-16T00:06:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-16T00:06:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-16T00:06:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-16T00:06:10.950 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:06:10.950 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-16T00:06:10.950 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-16T00:06:10.950 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-16T00:06:10.950 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-16T00:06:10.950 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-16T00:06:10.951 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-16T00:06:10.952 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-16T00:06:10.952 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-16T00:06:10.952 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-16T00:06:10.952 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-16T00:06:10.952 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-16T00:06:10.952 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-16T00:06:10.952 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-16T00:06:10.952 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-16T00:06:10.952 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-16T00:06:10.967 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-16T00:06:10.968 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-16T00:06:10.968 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-16T00:06:10.968 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-16T00:06:10.968 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-16T00:06:10.968 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-16T00:06:10.968 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-16T00:06:10.968 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-16T00:06:10.968 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-16T00:06:10.968 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-16T00:06:10.968 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-16T00:06:10.968 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-16T00:06:10.968 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-16T00:06:10.969 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-16T00:06:10.969 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-16T00:06:10.969 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-16T00:06:10.969 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-16T00:06:10.969 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-16T00:06:10.969 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-16T00:06:10.969 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-16T00:06:10.969 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-16T00:06:10.969 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-16T00:06:10.969 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-16T00:06:10.969 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-16T00:06:10.970 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-16T00:06:10.971 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-16T00:06:10.971 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-16T00:06:10.971 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-16T00:06:10.971 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-16T00:06:10.971 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-16T00:06:10.971 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-16T00:06:10.971 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-16T00:06:10.971 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-16T00:06:10.971 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-16T00:06:10.971 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-16T00:06:10.971 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-16T00:06:10.972 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-16T00:06:10.972 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-16T00:06:10.972 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-16T00:06:10.972 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-16T00:06:10.972 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-16T00:06:10.972 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-16T00:06:10.972 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-16T00:06:10.972 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-16T00:06:10.972 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-16T00:06:10.972 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-16T00:06:10.972 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-16T00:06:10.972 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-16T00:06:10.973 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-16T00:06:10.974 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-16T00:06:10.974 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-16T00:06:10.974 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-16T00:06:10.974 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-16T00:06:10.974 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-16T00:06:10.974 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-16T00:06:10.974 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-16T00:06:10.974 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-16T00:06:10.974 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-16T00:06:10.974 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-16T00:06:10.974 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-16T00:06:10.974 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-16T00:06:10.975 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-16T00:06:10.975 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-16T00:06:10.975 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-16T00:06:10.975 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-16T00:06:10.975 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-16T00:06:10.975 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-16T00:06:10.975 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-16T00:06:10.975 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-16T00:06:10.975 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-16T00:06:10.975 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-16T00:06:10.975 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-16T00:06:10.975 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-16T00:06:10.976 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-16T00:06:10.976 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-16T00:06:10.976 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-16T00:06:10.976 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-16T00:06:10.976 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-16T00:06:10.976 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-16T00:06:10.976 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-16T00:06:10.976 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-16T00:06:10.976 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-16T00:06:10.976 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-16T00:06:10.976 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-16T00:06:10.976 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-16T00:06:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-16T00:06:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-16T00:06:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-16T00:06:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-16T00:06:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-16T00:06:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-16T00:06:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-16T00:06:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-16T00:06:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-16T00:06:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-16T00:06:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-16T00:06:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-16T00:06:10.978 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-16T00:06:10.979 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-16T00:06:10.979 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-16T00:06:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:06:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:06:13.348 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2024-09-16T00:06:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-16T00:06:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-16T00:06:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-16T00:06:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: automake-1.16.2-8.el9.noarch 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: bison-3.7.4-5.el9.x86_64 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: flex-2.6.4-9.el9.x86_64 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T00:06:13.351 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T00:06:13.352 INFO:teuthology.orchestra.run.smithi060.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-16T00:06:13.352 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T00:06:13.352 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T00:06:13.352 INFO:teuthology.orchestra.run.smithi060.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-16T00:06:13.352 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T00:06:13.352 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T00:06:13.352 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.352 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.352 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.352 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T00:06:13.352 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T00:06:13.352 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T00:06:13.353 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-13.16-1.el9.x86_64 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-16T00:06:13.354 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T00:06:13.355 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T00:06:13.355 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T00:06:13.355 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.355 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.355 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.355 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T00:06:13.355 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T00:06:13.355 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T00:06:13.355 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T00:06:13.355 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-16T00:06:13.355 INFO:teuthology.orchestra.run.smithi060.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-16T00:06:13.355 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T00:06:13.356 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T00:06:13.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T00:06:13.358 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T00:06:13.359 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T00:06:13.360 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T00:06:13.360 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T00:06:13.360 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.360 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.360 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.360 INFO:teuthology.orchestra.run.smithi060.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T00:06:13.360 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T00:06:13.360 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T00:06:13.360 INFO:teuthology.orchestra.run.smithi060.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-16T00:06:13.360 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-16T00:06:13.360 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T00:06:13.361 INFO:teuthology.orchestra.run.smithi060.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-16T00:06:13.361 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:06:13.361 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-16T00:06:13.766 DEBUG:teuthology.parallel:result is None 2024-09-16T00:06:13.887 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:06:13.887 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:06:13.887 INFO:teuthology.orchestra.run.smithi033.stdout:Upgraded: 2024-09-16T00:06:13.887 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-16T00:06:13.887 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-16T00:06:13.887 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-16T00:06:13.887 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-16T00:06:13.887 INFO:teuthology.orchestra.run.smithi033.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.887 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.887 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout: automake-1.16.2-8.el9.noarch 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout: bison-3.7.4-5.el9.x86_64 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.888 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.889 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.889 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.889 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.889 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.889 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.889 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.889 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.889 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.889 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.889 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.889 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.890 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.890 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.890 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.890 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.890 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.890 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:06:13.890 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-16T00:06:13.890 INFO:teuthology.orchestra.run.smithi033.stdout: flex-2.6.4-9.el9.x86_64 2024-09-16T00:06:13.890 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T00:06:13.890 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T00:06:13.890 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T00:06:13.890 INFO:teuthology.orchestra.run.smithi033.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-16T00:06:13.891 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T00:06:13.891 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T00:06:13.891 INFO:teuthology.orchestra.run.smithi033.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-16T00:06:13.891 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T00:06:13.891 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T00:06:13.891 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.891 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.891 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.891 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T00:06:13.891 INFO:teuthology.orchestra.run.smithi033.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T00:06:13.891 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T00:06:13.891 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T00:06:13.892 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-13.16-1.el9.x86_64 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-16T00:06:13.893 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T00:06:13.894 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T00:06:13.894 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T00:06:13.894 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.894 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.894 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.894 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T00:06:13.894 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T00:06:13.894 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T00:06:13.894 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T00:06:13.894 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-16T00:06:13.894 INFO:teuthology.orchestra.run.smithi033.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-16T00:06:13.894 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T00:06:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T00:06:13.896 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T00:06:13.896 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T00:06:13.896 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T00:06:13.896 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T00:06:13.896 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T00:06:13.896 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T00:06:13.896 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T00:06:13.896 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T00:06:13.896 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T00:06:13.896 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T00:06:13.896 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T00:06:13.896 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T00:06:13.897 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T00:06:13.898 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T00:06:13.899 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T00:06:13.899 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T00:06:13.899 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T00:06:13.899 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.899 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.899 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:06:13.899 INFO:teuthology.orchestra.run.smithi033.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T00:06:13.899 INFO:teuthology.orchestra.run.smithi033.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T00:06:13.899 INFO:teuthology.orchestra.run.smithi033.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T00:06:13.899 INFO:teuthology.orchestra.run.smithi033.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-16T00:06:13.899 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-16T00:06:13.899 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T00:06:13.900 INFO:teuthology.orchestra.run.smithi033.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-16T00:06:13.900 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:06:13.900 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-09-16T00:06:14.332 DEBUG:teuthology.parallel:result is None 2024-09-16T00:06:14.333 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:06:14.481 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T00:06:14.505 INFO:teuthology.orchestra.run.smithi033.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T00:06:14.506 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T00:06:14.506 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T00:06:14.508 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:06:14.651 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T00:06:14.677 INFO:teuthology.orchestra.run.smithi060.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T00:06:14.677 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T00:06:14.677 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T00:06:14.679 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:06:14.820 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T00:06:14.847 INFO:teuthology.orchestra.run.smithi083.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T00:06:14.847 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T00:06:14.847 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T00:06:14.849 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T00:06:14.849 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:06:14.849 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T00:06:14.880 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-16T00:06:14.880 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T00:06:14.912 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T00:06:14.913 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T00:06:14.943 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T00:06:14.943 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:06:14.943 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T00:06:14.970 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T00:06:15.037 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-16T00:06:15.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T00:06:15.067 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T00:06:15.136 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T00:06:15.136 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T00:06:15.164 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T00:06:15.232 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T00:06:15.233 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:06:15.233 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T00:06:15.260 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T00:06:15.327 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-16T00:06:15.327 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T00:06:15.357 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T00:06:15.426 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T00:06:15.427 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T00:06:15.455 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T00:06:15.523 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T00:06:15.523 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:06:15.524 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T00:06:15.551 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T00:06:15.618 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-16T00:06:15.619 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T00:06:15.648 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T00:06:15.717 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T00:06:15.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T00:06:15.746 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T00:06:15.814 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-16T00:06:15.943 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-09-16T00:06:15.943 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:06:15.944 INFO:tasks.cephadm:Cluster fsid is 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:06:15.944 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-16T00:06:15.944 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.33', 'mon.b': '172.21.15.60', 'mon.c': '172.21.15.83'} 2024-09-16T00:06:15.945 INFO:tasks.cephadm:First mon is mon.a on smithi033 2024-09-16T00:06:15.945 INFO:tasks.cephadm:First mgr is x 2024-09-16T00:06:15.945 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-16T00:06:15.946 DEBUG:teuthology.orchestra.run.smithi033:> sudo hostname $(hostname -s) 2024-09-16T00:06:15.977 DEBUG:teuthology.orchestra.run.smithi060:> sudo hostname $(hostname -s) 2024-09-16T00:06:16.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo hostname $(hostname -s) 2024-09-16T00:06:16.035 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-16T00:06:16.036 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:06:16.178 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56081/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.1-326-g8921c8ef', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.1-326.g8921c8ef'}, 'url': 'https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-15 16:06:42.385367', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-16T00:06:16.313 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref squid, sha1 8921c8ef947807b717db041d85f0761b7914d5fc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:06:16.314 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm 2024-09-16T00:06:16.314 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm 2024-09-16T00:06:16.315 DEBUG:teuthology.orchestra.run.smithi033:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T00:06:16.705 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 16 00:06 /home/ubuntu/cephtest/cephadm 2024-09-16T00:06:16.706 DEBUG:teuthology.orchestra.run.smithi060:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T00:06:17.083 INFO:teuthology.orchestra.run.smithi060.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 16 00:06 /home/ubuntu/cephtest/cephadm 2024-09-16T00:06:17.083 DEBUG:teuthology.orchestra.run.smithi083:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T00:06:17.443 INFO:teuthology.orchestra.run.smithi083.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 16 00:06 /home/ubuntu/cephtest/cephadm 2024-09-16T00:06:17.443 DEBUG:teuthology.orchestra.run.smithi033:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T00:06:17.463 DEBUG:teuthology.orchestra.run.smithi060:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T00:06:17.482 DEBUG:teuthology.orchestra.run.smithi083:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T00:06:17.567 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc on all hosts... 2024-09-16T00:06:17.567 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-16T00:06:17.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-16T00:06:17.572 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-16T00:06:19.081 INFO:teuthology.orchestra.run.smithi083.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T00:06:19.203 INFO:teuthology.orchestra.run.smithi060.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T00:06:19.291 INFO:teuthology.orchestra.run.smithi033.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T00:06:59.090 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:06:59.090 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T00:06:59.090 INFO:teuthology.orchestra.run.smithi033.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-16T00:06:59.091 INFO:teuthology.orchestra.run.smithi033.stdout: "repo_digests": [ 2024-09-16T00:06:59.091 INFO:teuthology.orchestra.run.smithi033.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-16T00:06:59.091 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-09-16T00:06:59.091 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:06:59.343 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-16T00:06:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T00:06:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-16T00:06:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: "repo_digests": [ 2024-09-16T00:06:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-16T00:06:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-16T00:06:59.343 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-16T00:06:59.560 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-16T00:06:59.560 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T00:06:59.560 INFO:teuthology.orchestra.run.smithi083.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-16T00:06:59.560 INFO:teuthology.orchestra.run.smithi083.stdout: "repo_digests": [ 2024-09-16T00:06:59.560 INFO:teuthology.orchestra.run.smithi083.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-16T00:06:59.560 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-16T00:06:59.561 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-16T00:06:59.585 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph 2024-09-16T00:06:59.618 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph 2024-09-16T00:06:59.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph 2024-09-16T00:06:59.682 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /etc/ceph 2024-09-16T00:06:59.711 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /etc/ceph 2024-09-16T00:06:59.740 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /etc/ceph 2024-09-16T00:06:59.767 INFO:tasks.cephadm:Writing seed config... 2024-09-16T00:06:59.768 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-09-16T00:06:59.769 INFO:tasks.cephadm: override: [client] debug client = 20 2024-09-16T00:06:59.769 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-16T00:06:59.769 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-09-16T00:06:59.769 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-09-16T00:06:59.769 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-09-16T00:06:59.769 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-09-16T00:06:59.770 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-09-16T00:06:59.770 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-09-16T00:06:59.770 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-09-16T00:06:59.770 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-09-16T00:06:59.770 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-09-16T00:06:59.771 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-09-16T00:06:59.771 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-09-16T00:06:59.771 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-09-16T00:06:59.771 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-09-16T00:06:59.771 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-09-16T00:06:59.771 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-09-16T00:06:59.771 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-09-16T00:06:59.771 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-09-16T00:06:59.772 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-09-16T00:06:59.772 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-09-16T00:06:59.772 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-09-16T00:06:59.772 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-16T00:06:59.772 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-16T00:06:59.772 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-16T00:06:59.772 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-16T00:06:59.772 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-16T00:06:59.772 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-16T00:06:59.773 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-09-16T00:06:59.773 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-16T00:06:59.773 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-16T00:06:59.773 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-09-16T00:06:59.773 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-16T00:06:59.773 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-09-16T00:06:59.773 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-09-16T00:06:59.773 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-16T00:06:59.774 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-16T00:06:59.774 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-09-16T00:06:59.774 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-16T00:06:59.774 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-16T00:06:59.774 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-16T00:06:59.774 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-16T00:06:59.774 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-16T00:06:59.774 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-09-16T00:06:59.774 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-16T00:06:59.775 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-09-16T00:06:59.776 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:06:59.776 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-16T00:06:59.793 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7dde3d1a-73bf-11ef-bceb-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-09-16T00:06:59.793 DEBUG:teuthology.orchestra.run.smithi033:mon.a> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.a.service 2024-09-16T00:06:59.837 DEBUG:teuthology.orchestra.run.smithi033:mgr.x> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.x.service 2024-09-16T00:06:59.884 INFO:tasks.cephadm:Bootstrapping... 2024-09-16T00:06:59.884 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc -v bootstrap --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.33 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-16T00:07:00.095 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-09-16T00:07:00.095 INFO:teuthology.orchestra.run.smithi033.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc', '-v', 'bootstrap', '--fsid', '7dde3d1a-73bf-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.33', '--skip-admin-label'] 2024-09-16T00:07:00.096 INFO:teuthology.orchestra.run.smithi033.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-16T00:07:00.096 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying podman|docker is present... 2024-09-16T00:07:00.120 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stdout 5.2.2 2024-09-16T00:07:00.120 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying lvm2 is present... 2024-09-16T00:07:00.120 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying time synchronization is in place... 2024-09-16T00:07:00.144 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-16T00:07:00.144 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-16T00:07:00.150 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-16T00:07:00.150 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-09-16T00:07:00.157 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout enabled 2024-09-16T00:07:00.164 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout active 2024-09-16T00:07:00.164 INFO:teuthology.orchestra.run.smithi033.stdout:Unit chronyd.service is enabled and running 2024-09-16T00:07:00.164 INFO:teuthology.orchestra.run.smithi033.stdout:Repeating the final host check... 2024-09-16T00:07:00.189 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stdout 5.2.2 2024-09-16T00:07:00.189 INFO:teuthology.orchestra.run.smithi033.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-16T00:07:00.189 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl is present 2024-09-16T00:07:00.189 INFO:teuthology.orchestra.run.smithi033.stdout:lvcreate is present 2024-09-16T00:07:00.196 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-16T00:07:00.196 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-16T00:07:00.202 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-16T00:07:00.203 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-09-16T00:07:00.209 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout enabled 2024-09-16T00:07:00.216 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout active 2024-09-16T00:07:00.216 INFO:teuthology.orchestra.run.smithi033.stdout:Unit chronyd.service is enabled and running 2024-09-16T00:07:00.216 INFO:teuthology.orchestra.run.smithi033.stdout:Host looks OK 2024-09-16T00:07:00.216 INFO:teuthology.orchestra.run.smithi033.stdout:Cluster fsid: 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:07:00.216 INFO:teuthology.orchestra.run.smithi033.stdout:Acquiring lock 140049212429312 on /run/cephadm/7dde3d1a-73bf-11ef-bceb-c7b262605968.lock 2024-09-16T00:07:00.217 INFO:teuthology.orchestra.run.smithi033.stdout:Lock 140049212429312 acquired on /run/cephadm/7dde3d1a-73bf-11ef-bceb-c7b262605968.lock 2024-09-16T00:07:00.217 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying IP 172.21.15.33 port 3300 ... 2024-09-16T00:07:00.217 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying IP 172.21.15.33 port 6789 ... 2024-09-16T00:07:00.217 INFO:teuthology.orchestra.run.smithi033.stdout:Base mon IP(s) is [172.21.15.33:3300, 172.21.15.33:6789], mon addrv is [v2:172.21.15.33:3300,v1:172.21.15.33:6789] 2024-09-16T00:07:00.221 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.33 metric 100 2024-09-16T00:07:00.221 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.33 metric 100 2024-09-16T00:07:00.224 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-16T00:07:00.224 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-16T00:07:00.224 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1786sec hoplimit 64 pref medium 2024-09-16T00:07:00.226 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-16T00:07:00.226 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-16T00:07:00.227 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-16T00:07:00.227 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-16T00:07:00.227 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:d294/64 scope link 2024-09-16T00:07:00.227 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-16T00:07:00.227 INFO:teuthology.orchestra.run.smithi033.stdout:Mon IP `172.21.15.33` is in CIDR network `172.21.0.0/20` 2024-09-16T00:07:00.227 INFO:teuthology.orchestra.run.smithi033.stdout:Mon IP `172.21.15.33` is in CIDR network `172.21.0.0/20` 2024-09-16T00:07:00.228 INFO:teuthology.orchestra.run.smithi033.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-16T00:07:00.228 INFO:teuthology.orchestra.run.smithi033.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-16T00:07:00.229 INFO:teuthology.orchestra.run.smithi033.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T00:07:00.699 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stdout f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee 2024-09-16T00:07:00.699 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T00:07:00.699 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Getting image source signatures 2024-09-16T00:07:00.699 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Copying blob sha256:c0574da0c3174f3dc49daa274908f97d06456bd6a3cfab87e94022c1369a83a6 2024-09-16T00:07:00.699 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-16T00:07:00.699 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Copying config sha256:f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee 2024-09-16T00:07:00.699 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-16T00:07:03.013 INFO:teuthology.orchestra.run.smithi033.stdout:ceph: stdout ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T00:07:03.013 INFO:teuthology.orchestra.run.smithi033.stdout:Ceph version: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T00:07:03.013 INFO:teuthology.orchestra.run.smithi033.stdout:Extracting ceph user uid/gid from container image... 2024-09-16T00:07:12.668 INFO:teuthology.orchestra.run.smithi033.stdout:stat: stdout 167 167 2024-09-16T00:07:12.668 INFO:teuthology.orchestra.run.smithi033.stdout:Creating initial keys... 2024-09-16T00:07:15.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph-authtool: stdout AQCydudm7EH7NRAAPjo5iQK6tf/JZr6VU22JmA== 2024-09-16T00:07:16.483 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph-authtool: stdout AQCzdudm0VUGNBAAQFoKwTEpLcA3JfMmvtum8g== 2024-09-16T00:07:17.405 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph-authtool: stdout AQC0dudmvFc4LxAA9M469sH6H49gD39HXX7KEg== 2024-09-16T00:07:17.406 INFO:teuthology.orchestra.run.smithi033.stdout:Creating initial monmap... 2024-09-16T00:07:18.280 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-16T00:07:18.280 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-16T00:07:18.280 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:07:18.281 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-16T00:07:18.281 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool for a [v2:172.21.15.33:3300,v1:172.21.15.33:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-16T00:07:18.281 INFO:teuthology.orchestra.run.smithi033.stdout:setting min_mon_release = quincy 2024-09-16T00:07:18.281 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: set fsid to 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:07:18.281 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-16T00:07:18.281 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:07:18.281 INFO:teuthology.orchestra.run.smithi033.stdout:Creating mon... 2024-09-16T00:07:19.195 INFO:teuthology.orchestra.run.smithi033.stdout:create mon.a on 2024-09-16T00:07:19.507 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-16T00:07:19.666 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-16T00:07:19.865 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968.target. 2024-09-16T00:07:19.865 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968.target. 2024-09-16T00:07:20.534 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.a 2024-09-16T00:07:20.534 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-16T00:07:20.719 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968.target.wants/ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@.service. 2024-09-16T00:07:21.215 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T00:07:21.216 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout disabled 2024-09-16T00:07:21.222 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T00:07:21.223 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-09-16T00:07:21.223 INFO:teuthology.orchestra.run.smithi033.stdout:firewalld.service is not enabled 2024-09-16T00:07:21.223 INFO:teuthology.orchestra.run.smithi033.stdout:Not possible to enable service . firewalld.service is not available 2024-09-16T00:07:21.223 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mon to start... 2024-09-16T00:07:21.223 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mon... 2024-09-16T00:07:22.235 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout cluster: 2024-09-16T00:07:22.241 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout id: 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:07:22.241 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-16T00:07:22.241 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-09-16T00:07:22.241 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout services: 2024-09-16T00:07:22.241 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.521305s) 2024-09-16T00:07:22.241 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-16T00:07:22.241 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout data: 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout pgs: 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.616+0000 7f370f494640 1 Processor -- start 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.616+0000 7f370f494640 1 -- start start 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.616+0000 7f370f494640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 0x7f3708107b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.616+0000 7f370f494640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3708108110 con 0x7f3708107740 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.617+0000 7f370d230640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 0x7f3708107b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:22.242 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.617+0000 7f370d230640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 0x7f3708107b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56454/0 (socket says 172.21.15.33:56454) 2024-09-16T00:07:22.243 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.617+0000 7f370d230640 1 -- 172.21.15.33:0/3926417521 learned_addr learned my addr 172.21.15.33:0/3926417521 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:22.243 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.617+0000 7f370d230640 1 -- 172.21.15.33:0/3926417521 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37081088a0 con 0x7f3708107740 2024-09-16T00:07:22.243 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.619+0000 7f370d230640 1 --2- 172.21.15.33:0/3926417521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 0x7f3708107b40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3704009820 tx=0x7f370402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e880e8e32c733b3e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:22.243 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.619+0000 7f36fbfff640 1 -- 172.21.15.33:0/3926417521 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3704037070 con 0x7f3708107740 2024-09-16T00:07:22.243 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.619+0000 7f36fbfff640 1 -- 172.21.15.33:0/3926417521 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3704032440 con 0x7f3708107740 2024-09-16T00:07:22.243 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.619+0000 7f36fbfff640 1 -- 172.21.15.33:0/3926417521 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3704032740 con 0x7f3708107740 2024-09-16T00:07:22.243 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.619+0000 7f370f494640 1 -- 172.21.15.33:0/3926417521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 msgr2=0x7f3708107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:22.243 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.619+0000 7f370f494640 1 --2- 172.21.15.33:0/3926417521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 0x7f3708107b40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3704009820 tx=0x7f370402a270 comp rx=0 tx=0).stop 2024-09-16T00:07:22.243 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.620+0000 7f370f494640 1 -- 172.21.15.33:0/3926417521 shutdown_connections 2024-09-16T00:07:22.243 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.620+0000 7f370f494640 1 --2- 172.21.15.33:0/3926417521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 0x7f3708107b40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:22.243 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.620+0000 7f370f494640 1 -- 172.21.15.33:0/3926417521 >> 172.21.15.33:0/3926417521 conn(0x7f3708102eb0 msgr2=0x7f3708105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:22.243 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.620+0000 7f370f494640 1 -- 172.21.15.33:0/3926417521 shutdown_connections 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.620+0000 7f370f494640 1 -- 172.21.15.33:0/3926417521 wait complete. 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.621+0000 7f370f494640 1 Processor -- start 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.621+0000 7f370f494640 1 -- start start 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.621+0000 7f370f494640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 0x7f370807db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.621+0000 7f370f494640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3708108dd0 con 0x7f3708107740 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.621+0000 7f370d230640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 0x7f370807db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.622+0000 7f370d230640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 0x7f370807db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56460/0 (socket says 172.21.15.33:56460) 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.622+0000 7f370d230640 1 -- 172.21.15.33:0/1285052862 learned_addr learned my addr 172.21.15.33:0/1285052862 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.622+0000 7f370d230640 1 -- 172.21.15.33:0/1285052862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370807e0b0 con 0x7f3708107740 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.622+0000 7f370d230640 1 --2- 172.21.15.33:0/1285052862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 0x7f370807db70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f37040090b0 tx=0x7f37040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.622+0000 7f36fa7fc640 1 -- 172.21.15.33:0/1285052862 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3704041070 con 0x7f3708107740 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.622+0000 7f36fa7fc640 1 -- 172.21.15.33:0/1285052862 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f370403c7a0 con 0x7f3708107740 2024-09-16T00:07:22.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.622+0000 7f370f494640 1 -- 172.21.15.33:0/1285052862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370807c2a0 con 0x7f3708107740 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.622+0000 7f36fa7fc640 1 -- 172.21.15.33:0/1285052862 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3704037040 con 0x7f3708107740 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.622+0000 7f370f494640 1 -- 172.21.15.33:0/1285052862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f370807c7e0 con 0x7f3708107740 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.622+0000 7f36fa7fc640 1 -- 172.21.15.33:0/1285052862 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3704044070 con 0x7f3708107740 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.623+0000 7f36fa7fc640 1 -- 172.21.15.33:0/1285052862 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f370403d3f0 con 0x7f3708107740 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.623+0000 7f370f494640 1 -- 172.21.15.33:0/1285052862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3708109420 con 0x7f3708107740 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.626+0000 7f36fa7fc640 1 -- 172.21.15.33:0/1285052862 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f3704046450 con 0x7f3708107740 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.687+0000 7f370f494640 1 -- 172.21.15.33:0/1285052862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f370807cee0 con 0x7f3708107740 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.687+0000 7f36fa7fc640 1 -- 172.21.15.33:0/1285052862 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f3704046630 con 0x7f3708107740 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.688+0000 7f370f494640 1 -- 172.21.15.33:0/1285052862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 msgr2=0x7f370807db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.688+0000 7f370f494640 1 --2- 172.21.15.33:0/1285052862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 0x7f370807db70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f37040090b0 tx=0x7f37040046c0 comp rx=0 tx=0).stop 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.688+0000 7f370f494640 1 -- 172.21.15.33:0/1285052862 shutdown_connections 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.688+0000 7f370f494640 1 --2- 172.21.15.33:0/1285052862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3708107740 0x7f370807db70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:22.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.688+0000 7f370f494640 1 -- 172.21.15.33:0/1285052862 >> 172.21.15.33:0/1285052862 conn(0x7f3708102eb0 msgr2=0x7f370818f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:22.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.688+0000 7f370f494640 1 -- 172.21.15.33:0/1285052862 shutdown_connections 2024-09-16T00:07:22.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:21.688+0000 7f370f494640 1 -- 172.21.15.33:0/1285052862 wait complete. 2024-09-16T00:07:22.246 INFO:teuthology.orchestra.run.smithi033.stdout:mon is available 2024-09-16T00:07:22.246 INFO:teuthology.orchestra.run.smithi033.stdout:Assimilating anything we can from ceph.conf... 2024-09-16T00:07:23.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:23 smithi033 ceph-mon[30871]: from='client.? 172.21.15.33:0/3550968671' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-16T00:07:23.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-09-16T00:07:23.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [global] 2024-09-16T00:07:23.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout fsid = 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:07:23.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.33:3300,v1:172.21.15.33:6789] 2024-09-16T00:07:23.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-16T00:07:23.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-16T00:07:23.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-16T00:07:23.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [osd] 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.657+0000 7ff49ee0b640 1 Processor -- start 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.657+0000 7ff49ee0b640 1 -- start start 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.657+0000 7ff49ee0b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 0x7ff498106920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:23.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.657+0000 7ff49ee0b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff498106ef0 con 0x7ff498106520 2024-09-16T00:07:23.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.658+0000 7ff49cba7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 0x7ff498106920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:23.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.658+0000 7ff49cba7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 0x7ff498106920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56466/0 (socket says 172.21.15.33:56466) 2024-09-16T00:07:23.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.658+0000 7ff49cba7640 1 -- 172.21.15.33:0/2715009934 learned_addr learned my addr 172.21.15.33:0/2715009934 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:23.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.658+0000 7ff49cba7640 1 -- 172.21.15.33:0/2715009934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff498107720 con 0x7ff498106520 2024-09-16T00:07:23.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.659+0000 7ff49cba7640 1 --2- 172.21.15.33:0/2715009934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 0x7ff498106920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff488009820 tx=0x7ff48802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ceec5e3cc4aa84d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:23.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.659+0000 7ff4977fe640 1 -- 172.21.15.33:0/2715009934 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff488037070 con 0x7ff498106520 2024-09-16T00:07:23.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.659+0000 7ff4977fe640 1 -- 172.21.15.33:0/2715009934 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff488032440 con 0x7ff498106520 2024-09-16T00:07:23.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.659+0000 7ff4977fe640 1 -- 172.21.15.33:0/2715009934 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff488032740 con 0x7ff498106520 2024-09-16T00:07:23.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.659+0000 7ff49ee0b640 1 -- 172.21.15.33:0/2715009934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 msgr2=0x7ff498106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:23.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.659+0000 7ff49ee0b640 1 --2- 172.21.15.33:0/2715009934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 0x7ff498106920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff488009820 tx=0x7ff48802a270 comp rx=0 tx=0).stop 2024-09-16T00:07:23.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.659+0000 7ff49ee0b640 1 -- 172.21.15.33:0/2715009934 shutdown_connections 2024-09-16T00:07:23.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.659+0000 7ff49ee0b640 1 --2- 172.21.15.33:0/2715009934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 0x7ff498106920 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:23.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.659+0000 7ff49ee0b640 1 -- 172.21.15.33:0/2715009934 >> 172.21.15.33:0/2715009934 conn(0x7ff498101db0 msgr2=0x7ff4981041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:23.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.659+0000 7ff49ee0b640 1 -- 172.21.15.33:0/2715009934 shutdown_connections 2024-09-16T00:07:23.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.659+0000 7ff49ee0b640 1 -- 172.21.15.33:0/2715009934 wait complete. 2024-09-16T00:07:23.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.660+0000 7ff49ee0b640 1 Processor -- start 2024-09-16T00:07:23.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.660+0000 7ff49ee0b640 1 -- start start 2024-09-16T00:07:23.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.660+0000 7ff49ee0b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 0x7ff49819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:23.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.660+0000 7ff49ee0b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff498107cc0 con 0x7ff498106520 2024-09-16T00:07:23.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.660+0000 7ff49cba7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 0x7ff49819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:23.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.660+0000 7ff49cba7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 0x7ff49819ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56480/0 (socket says 172.21.15.33:56480) 2024-09-16T00:07:23.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.660+0000 7ff49cba7640 1 -- 172.21.15.33:0/3550968671 learned_addr learned my addr 172.21.15.33:0/3550968671 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:23.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.661+0000 7ff49cba7640 1 -- 172.21.15.33:0/3550968671 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff49819b320 con 0x7ff498106520 2024-09-16T00:07:23.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.661+0000 7ff49cba7640 1 --2- 172.21.15.33:0/3550968671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 0x7ff49819ade0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff4880090b0 tx=0x7ff4880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.661+0000 7ff495ffb640 1 -- 172.21.15.33:0/3550968671 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff488041070 con 0x7ff498106520 2024-09-16T00:07:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.661+0000 7ff495ffb640 1 -- 172.21.15.33:0/3550968671 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff48803c7a0 con 0x7ff498106520 2024-09-16T00:07:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.661+0000 7ff49ee0b640 1 -- 172.21.15.33:0/3550968671 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff49819b5b0 con 0x7ff498106520 2024-09-16T00:07:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.661+0000 7ff495ffb640 1 -- 172.21.15.33:0/3550968671 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff488037040 con 0x7ff498106520 2024-09-16T00:07:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.661+0000 7ff49ee0b640 1 -- 172.21.15.33:0/3550968671 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff49819e2a0 con 0x7ff498106520 2024-09-16T00:07:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.661+0000 7ff495ffb640 1 -- 172.21.15.33:0/3550968671 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff488044070 con 0x7ff498106520 2024-09-16T00:07:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.661+0000 7ff495ffb640 1 -- 172.21.15.33:0/3550968671 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff48803d3f0 con 0x7ff498106520 2024-09-16T00:07:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.662+0000 7ff49ee0b640 1 -- 172.21.15.33:0/3550968671 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff498108310 con 0x7ff498106520 2024-09-16T00:07:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.664+0000 7ff495ffb640 1 -- 172.21.15.33:0/3550968671 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7ff488046450 con 0x7ff498106520 2024-09-16T00:07:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.724+0000 7ff49ee0b640 1 -- 172.21.15.33:0/3550968671 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7ff49802d980 con 0x7ff498106520 2024-09-16T00:07:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.727+0000 7ff495ffb640 1 -- 172.21.15.33:0/3550968671 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7ff488046630 con 0x7ff498106520 2024-09-16T00:07:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.728+0000 7ff495ffb640 1 -- 172.21.15.33:0/3550968671 <== mon.0 v2:172.21.15.33:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff488056070 con 0x7ff498106520 2024-09-16T00:07:23.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.729+0000 7ff49ee0b640 1 -- 172.21.15.33:0/3550968671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 msgr2=0x7ff49819ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:23.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.729+0000 7ff49ee0b640 1 --2- 172.21.15.33:0/3550968671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 0x7ff49819ade0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff4880090b0 tx=0x7ff4880046c0 comp rx=0 tx=0).stop 2024-09-16T00:07:23.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.729+0000 7ff49ee0b640 1 -- 172.21.15.33:0/3550968671 shutdown_connections 2024-09-16T00:07:23.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.729+0000 7ff49ee0b640 1 --2- 172.21.15.33:0/3550968671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff498106520 0x7ff49819ade0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:23.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.729+0000 7ff49ee0b640 1 -- 172.21.15.33:0/3550968671 >> 172.21.15.33:0/3550968671 conn(0x7ff498101db0 msgr2=0x7ff498191aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:23.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.729+0000 7ff49ee0b640 1 -- 172.21.15.33:0/3550968671 shutdown_connections 2024-09-16T00:07:23.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:22.729+0000 7ff49ee0b640 1 -- 172.21.15.33:0/3550968671 wait complete. 2024-09-16T00:07:23.295 INFO:teuthology.orchestra.run.smithi033.stdout:Generating new minimal ceph.conf... 2024-09-16T00:07:24.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:24 smithi033 ceph-mon[30871]: from='client.? 172.21.15.33:0/2038502292' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:07:24.362 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.664+0000 7f9598990640 1 Processor -- start 2024-09-16T00:07:24.362 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.664+0000 7f9598990640 1 -- start start 2024-09-16T00:07:24.362 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.664+0000 7f9598990640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 0x7f9594107b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:24.362 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.664+0000 7f9598990640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9594108110 con 0x7f9594107740 2024-09-16T00:07:24.362 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.664+0000 7f959259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 0x7f9594107b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.664+0000 7f959259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 0x7f9594107b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56486/0 (socket says 172.21.15.33:56486) 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.664+0000 7f959259c640 1 -- 172.21.15.33:0/1506163681 learned_addr learned my addr 172.21.15.33:0/1506163681 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.665+0000 7f959259c640 1 -- 172.21.15.33:0/1506163681 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95941088a0 con 0x7f9594107740 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.665+0000 7f959259c640 1 --2- 172.21.15.33:0/1506163681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 0x7f9594107b40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9584009820 tx=0x7f958402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a4a85297bd4670 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.665+0000 7f959159a640 1 -- 172.21.15.33:0/1506163681 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9584037070 con 0x7f9594107740 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.666+0000 7f959159a640 1 -- 172.21.15.33:0/1506163681 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9584032900 con 0x7f9594107740 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.666+0000 7f959159a640 1 -- 172.21.15.33:0/1506163681 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9584030850 con 0x7f9594107740 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.666+0000 7f9598990640 1 -- 172.21.15.33:0/1506163681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 msgr2=0x7f9594107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.666+0000 7f9598990640 1 --2- 172.21.15.33:0/1506163681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 0x7f9594107b40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9584009820 tx=0x7f958402a270 comp rx=0 tx=0).stop 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.666+0000 7f9598990640 1 -- 172.21.15.33:0/1506163681 shutdown_connections 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.666+0000 7f9598990640 1 --2- 172.21.15.33:0/1506163681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 0x7f9594107b40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.666+0000 7f9598990640 1 -- 172.21.15.33:0/1506163681 >> 172.21.15.33:0/1506163681 conn(0x7f9594102eb0 msgr2=0x7f9594105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.666+0000 7f9598990640 1 -- 172.21.15.33:0/1506163681 shutdown_connections 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.666+0000 7f9598990640 1 -- 172.21.15.33:0/1506163681 wait complete. 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.667+0000 7f9598990640 1 Processor -- start 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.667+0000 7f9598990640 1 -- start start 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.667+0000 7f9598990640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 0x7f959407db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.667+0000 7f9598990640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f959410bc50 con 0x7f9594107740 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.667+0000 7f959259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 0x7f959407db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.667+0000 7f959259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 0x7f959407db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56494/0 (socket says 172.21.15.33:56494) 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.667+0000 7f959259c640 1 -- 172.21.15.33:0/2038502292 learned_addr learned my addr 172.21.15.33:0/2038502292 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.667+0000 7f959259c640 1 -- 172.21.15.33:0/2038502292 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959407e0b0 con 0x7f9594107740 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.668+0000 7f959259c640 1 --2- 172.21.15.33:0/2038502292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 0x7f959407db70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9584009940 tx=0x7f95840326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.668+0000 7f95837fe640 1 -- 172.21.15.33:0/2038502292 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9584042070 con 0x7f9594107740 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.668+0000 7f9598990640 1 -- 172.21.15.33:0/2038502292 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959407c2a0 con 0x7f9594107740 2024-09-16T00:07:24.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.668+0000 7f95837fe640 1 -- 172.21.15.33:0/2038502292 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9584032be0 con 0x7f9594107740 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.668+0000 7f95837fe640 1 -- 172.21.15.33:0/2038502292 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9584037040 con 0x7f9594107740 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.668+0000 7f9598990640 1 -- 172.21.15.33:0/2038502292 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f959407c7e0 con 0x7f9594107740 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.668+0000 7f95837fe640 1 -- 172.21.15.33:0/2038502292 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9584045070 con 0x7f9594107740 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.668+0000 7f95837fe640 1 -- 172.21.15.33:0/2038502292 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9584047a80 con 0x7f9594107740 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.669+0000 7f9598990640 1 -- 172.21.15.33:0/2038502292 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9594107bc0 con 0x7f9594107740 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.671+0000 7f95837fe640 1 -- 172.21.15.33:0/2038502292 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f9584032d90 con 0x7f9594107740 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.731+0000 7f9598990640 1 -- 172.21.15.33:0/2038502292 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f959407cb20 con 0x7f9594107740 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.731+0000 7f95837fe640 1 -- 172.21.15.33:0/2038502292 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f958403d430 con 0x7f9594107740 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.732+0000 7f9598990640 1 -- 172.21.15.33:0/2038502292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 msgr2=0x7f959407db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.732+0000 7f9598990640 1 --2- 172.21.15.33:0/2038502292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 0x7f959407db70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9584009940 tx=0x7f95840326e0 comp rx=0 tx=0).stop 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.732+0000 7f9598990640 1 -- 172.21.15.33:0/2038502292 shutdown_connections 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.732+0000 7f9598990640 1 --2- 172.21.15.33:0/2038502292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9594107740 0x7f959407db70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.732+0000 7f9598990640 1 -- 172.21.15.33:0/2038502292 >> 172.21.15.33:0/2038502292 conn(0x7f9594102eb0 msgr2=0x7f9594103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:24.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.732+0000 7f9598990640 1 -- 172.21.15.33:0/2038502292 shutdown_connections 2024-09-16T00:07:24.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:23.732+0000 7f9598990640 1 -- 172.21.15.33:0/2038502292 wait complete. 2024-09-16T00:07:24.366 INFO:teuthology.orchestra.run.smithi033.stdout:Restarting the monitor... 2024-09-16T00:07:24.650 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:24 smithi033 systemd[1]: Stopping Ceph mon.a for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:07:24.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:24 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a[30867]: 2024-09-16T00:07:24.649+0000 7f184e68b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:07:24.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:24 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a[30867]: 2024-09-16T00:07:24.649+0000 7f184e68b640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-16T00:07:24.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:24 smithi033 podman[31118]: 2024-09-16 00:07:24.710730404 +0000 UTC m=+0.182021878 container died df41437b54bbb84e3699f9c36cd484b584890c11d1722779a76dbf9e14011f86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-16T00:07:25.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 podman[31118]: 2024-09-16 00:07:25.090786289 +0000 UTC m=+0.562077762 container cleanup df41437b54bbb84e3699f9c36cd484b584890c11d1722779a76dbf9e14011f86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, ceph=True) 2024-09-16T00:07:25.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 bash[31118]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a 2024-09-16T00:07:25.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 podman[31130]: 2024-09-16 00:07:25.291840643 +0000 UTC m=+0.575490730 container remove df41437b54bbb84e3699f9c36cd484b584890c11d1722779a76dbf9e14011f86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-16T00:07:25.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 systemd[1]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-16T00:07:25.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 systemd[1]: Stopped Ceph mon.a for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:07:25.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 systemd[1]: Starting Ceph mon.a for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:07:25.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 podman[31223]: 2024-09-16 00:07:25.531569593 +0000 UTC m=+0.017856565 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:07:25.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 podman[31223]: 2024-09-16 00:07:25.726250475 +0000 UTC m=+0.212537482 container create 831049806ec5419bdf268e2d4f962b16da7021d08b247c2c780919d4774ddace (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-8921c8e) 2024-09-16T00:07:25.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 podman[31223]: 2024-09-16 00:07:25.962115588 +0000 UTC m=+0.448402556 container init 831049806ec5419bdf268e2d4f962b16da7021d08b247c2c780919d4774ddace (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True) 2024-09-16T00:07:26.150 INFO:teuthology.orchestra.run.smithi033.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-16T00:07:26.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 podman[31223]: 2024-09-16 00:07:25.965621435 +0000 UTC m=+0.451908401 container start 831049806ec5419bdf268e2d4f962b16da7021d08b247c2c780919d4774ddace (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-16T00:07:26.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: set uid:gid to 167:167 (ceph:ceph) 2024-09-16T00:07:26.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable), process ceph-mon, pid 2 2024-09-16T00:07:26.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: pidfile_write: ignore empty --pid-file 2024-09-16T00:07:26.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: load: jerasure load: lrc 2024-09-16T00:07:26.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: RocksDB version: 7.9.2 2024-09-16T00:07:26.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Git sha 0 2024-09-16T00:07:26.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Compile date 2024-09-15 15:04:35 2024-09-16T00:07:26.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: DB SUMMARY 2024-09-16T00:07:26.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: DB Session ID: 0GP59XWETZ239M725OIG 2024-09-16T00:07:26.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: CURRENT file: CURRENT 2024-09-16T00:07:26.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: IDENTITY file: IDENTITY 2024-09-16T00:07:26.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-16T00:07:26.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-16T00:07:26.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105390 ; 2024-09-16T00:07:26.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.error_if_exists: 0 2024-09-16T00:07:26.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.create_if_missing: 0 2024-09-16T00:07:26.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.paranoid_checks: 1 2024-09-16T00:07:26.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-16T00:07:26.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-16T00:07:26.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-16T00:07:26.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.env: 0x55d95fafa940 2024-09-16T00:07:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.fs: PosixFileSystem 2024-09-16T00:07:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.info_log: 0x55d961c8b6e0 2024-09-16T00:07:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_file_opening_threads: 16 2024-09-16T00:07:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.statistics: (nil) 2024-09-16T00:07:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.use_fsync: 0 2024-09-16T00:07:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_log_file_size: 0 2024-09-16T00:07:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-16T00:07:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-16T00:07:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.keep_log_file_num: 1000 2024-09-16T00:07:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.recycle_log_file_num: 0 2024-09-16T00:07:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.allow_fallocate: 1 2024-09-16T00:07:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.allow_mmap_reads: 0 2024-09-16T00:07:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.allow_mmap_writes: 0 2024-09-16T00:07:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.use_direct_reads: 0 2024-09-16T00:07:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-16T00:07:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.create_missing_column_families: 0 2024-09-16T00:07:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.db_log_dir: 2024-09-16T00:07:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.wal_dir: 2024-09-16T00:07:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-16T00:07:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-16T00:07:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-16T00:07:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-16T00:07:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.advise_random_on_open: 1 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.db_write_buffer_size: 0 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.write_buffer_manager: 0x55d961c9adc0 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.rate_limiter: (nil) 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.wal_recovery_mode: 2 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.enable_thread_tracking: 0 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.enable_pipelined_write: 0 2024-09-16T00:07:26.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.unordered_write: 0 2024-09-16T00:07:26.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-16T00:07:26.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-16T00:07:26.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-16T00:07:26.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-16T00:07:26.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.row_cache: None 2024-09-16T00:07:26.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.wal_filter: None 2024-09-16T00:07:26.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-16T00:07:26.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.allow_ingest_behind: 0 2024-09-16T00:07:26.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.two_write_queues: 0 2024-09-16T00:07:26.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.manual_wal_flush: 0 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.wal_compression: 0 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.atomic_flush: 0 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.log_readahead_size: 0 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.best_efforts_recovery: 0 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.allow_data_in_errors: 0 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.db_host_id: __hostname__ 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-16T00:07:26.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_background_jobs: 2 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_background_compactions: -1 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_subcompactions: 1 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_total_wal_size: 0 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_open_files: -1 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bytes_per_sync: 0 2024-09-16T00:07:26.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_readahead_size: 0 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_background_flushes: -1 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Compression algorithms supported: 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: kZSTD supported: 0 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: kXpressCompression supported: 0 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: kBZip2Compression supported: 0 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: kLZ4Compression supported: 1 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: kZlibCompression supported: 1 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: kLZ4HCCompression supported: 1 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: kSnappyCompression supported: 1 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-16T00:07:26.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.merge_operator: 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_filter: None 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_filter_factory: None 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.sst_partitioner_factory: None 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d961c8b320) 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout: cache_index_and_filter_blocks: 1 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-16T00:07:26.285 INFO:journalctl@ceph.mon.a.smithi033.stdout: pin_top_level_index_and_filter: 1 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: index_type: 0 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: data_block_index_type: 0 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: index_shortening: 1 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: checksum: 4 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: no_block_cache: 0 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_cache: 0x55d961c7f1f0 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_cache_name: BinnedLRUCache 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_cache_options: 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: capacity : 536870912 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: num_shard_bits : 4 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: strict_capacity_limit : 0 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: high_pri_pool_ratio: 0.000 2024-09-16T00:07:26.286 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_cache_compressed: (nil) 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: persistent_cache: (nil) 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_size: 4096 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_size_deviation: 10 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_restart_interval: 16 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: index_block_restart_interval: 1 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: metadata_block_size: 4096 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: partition_filters: 0 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: use_delta_encoding: 1 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: filter_policy: bloomfilter 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: whole_key_filtering: 1 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: verify_compression: 0 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: read_amp_bytes_per_bit: 0 2024-09-16T00:07:26.287 INFO:journalctl@ceph.mon.a.smithi033.stdout: format_version: 5 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout: enable_index_compression: 1 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_align: 0 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout: max_auto_readahead_size: 262144 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout: prepopulate_block_cache: 0 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout: initial_auto_readahead_size: 8192 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout: num_file_reads_for_auto_readahead: 2 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.write_buffer_size: 33554432 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_write_buffer_number: 2 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compression: NoCompression 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bottommost_compression: Disabled 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.prefix_extractor: nullptr 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.num_levels: 7 2024-09-16T00:07:26.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-16T00:07:26.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compression_opts.level: 32767 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compression_opts.strategy: 0 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compression_opts.enabled: false 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.target_file_size_base: 67108864 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-16T00:07:26.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.arena_block_size: 1048576 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-16T00:07:26.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.disable_auto_compactions: 0 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-16T00:07:26.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.inplace_update_support: 0 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.bloom_locality: 0 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.max_successive_merges: 0 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.paranoid_file_checks: 0 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.force_consistency_checks: 1 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.report_bg_io_stats: 0 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.ttl: 2592000 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-16T00:07:26.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.enable_blob_files: false 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.min_blob_size: 0 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.blob_file_size: 268435456 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.blob_file_starting_level: 0 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5003d8eb-da0d-46e0-a401-32ecc7494d46 2024-09-16T00:07:26.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726445245993019, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726445245994115, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100438, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98332, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14320, "raw_average_key_size": 49, "raw_value_size": 90585, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726445245, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5003d8eb-da0d-46e0-a401-32ecc7494d46", "db_session_id": "0GP59XWETZ239M725OIG", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726445245994216, "job": 1, "event": "recovery_finished"} 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d961cace00 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: DB pointer 0x55d961da8000 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout: ** DB Stats ** 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-16T00:07:26.295 INFO:journalctl@ceph.mon.a.smithi033.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: ** Compaction Stats [default] ** 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: L0 2/0 99.94 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: Sum 2/0 99.94 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-09-16T00:07:26.296 INFO:journalctl@ceph.mon.a.smithi033.stdout: ** Compaction Stats [default] ** 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-16T00:07:26.297 INFO:journalctl@ceph.mon.a.smithi033.stdout: Cumulative compaction: 0.00 GB write, 28.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout: Interval compaction: 0.00 GB write, 28.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout: Block cache BinnedLRUCache@0x55d961c7f1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: starting mon.a rank 0 at public addrs [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] at bind addrs [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: mon.a@-1(???) e1 preinit fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: mon.a@-1(???).mds e1 new map 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: mon.a@-1(???).mds e1 print_map 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout: e1 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout: btime 2024-09-16T00:07:21:167536+0000 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-16T00:07:26.298 INFO:journalctl@ceph.mon.a.smithi033.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout: legacy client fscid: -1 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout: No filesystems configured 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:25 smithi033 ceph-mon[31243]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 ceph-mon[31243]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 ceph-mon[31243]: monmap epoch 1 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 ceph-mon[31243]: fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 ceph-mon[31243]: last_changed 2024-09-16T00:07:17.720547+0000 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 ceph-mon[31243]: created 2024-09-16T00:07:17.720547+0000 2024-09-16T00:07:26.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 ceph-mon[31243]: min_mon_release 19 (squid) 2024-09-16T00:07:26.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 ceph-mon[31243]: election_strategy: 1 2024-09-16T00:07:26.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 ceph-mon[31243]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-09-16T00:07:26.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 ceph-mon[31243]: fsmap 2024-09-16T00:07:26.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 ceph-mon[31243]: osdmap e1: 0 total, 0 up, 0 in 2024-09-16T00:07:26.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 ceph-mon[31243]: mgrmap e1: no daemons active 2024-09-16T00:07:26.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 bash[31223]: 831049806ec5419bdf268e2d4f962b16da7021d08b247c2c780919d4774ddace 2024-09-16T00:07:26.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:26 smithi033 systemd[1]: Started Ceph mon.a for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:07:27.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.734+0000 7f2a37aad640 1 Processor -- start 2024-09-16T00:07:27.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.734+0000 7f2a37aad640 1 -- start start 2024-09-16T00:07:27.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.735+0000 7f2a37aad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 0x7f2a3010a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:27.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.735+0000 7f2a37aad640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3007be30 con 0x7f2a30108110 2024-09-16T00:07:27.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.735+0000 7f2a35849640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 0x7f2a3010a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:27.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.735+0000 7f2a35849640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 0x7f2a3010a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56496/0 (socket says 172.21.15.33:56496) 2024-09-16T00:07:27.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.735+0000 7f2a35849640 1 -- 172.21.15.33:0/4082666471 learned_addr learned my addr 172.21.15.33:0/4082666471 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:27.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.735+0000 7f2a35849640 1 -- 172.21.15.33:0/4082666471 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3010aa40 con 0x7f2a30108110 2024-09-16T00:07:27.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.737+0000 7f2a35849640 1 --2- 172.21.15.33:0/4082666471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 0x7f2a3010a500 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009e80 tx=0x7f2a2002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f8bcd38289249a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.737+0000 7f2a34847640 1 -- 172.21.15.33:0/4082666471 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a20036070 con 0x7f2a30108110 2024-09-16T00:07:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.738+0000 7f2a34847640 1 -- 172.21.15.33:0/4082666471 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2a200318c0 con 0x7f2a30108110 2024-09-16T00:07:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.738+0000 7f2a34847640 1 -- 172.21.15.33:0/4082666471 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a20007ab0 con 0x7f2a30108110 2024-09-16T00:07:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.738+0000 7f2a37aad640 1 -- 172.21.15.33:0/4082666471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 msgr2=0x7f2a3010a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.738+0000 7f2a37aad640 1 --2- 172.21.15.33:0/4082666471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 0x7f2a3010a500 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009e80 tx=0x7f2a2002a7a0 comp rx=0 tx=0).stop 2024-09-16T00:07:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.738+0000 7f2a37aad640 1 -- 172.21.15.33:0/4082666471 shutdown_connections 2024-09-16T00:07:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.738+0000 7f2a37aad640 1 --2- 172.21.15.33:0/4082666471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 0x7f2a3010a500 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.738+0000 7f2a37aad640 1 -- 172.21.15.33:0/4082666471 >> 172.21.15.33:0/4082666471 conn(0x7f2a30101db0 msgr2=0x7f2a301041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.738+0000 7f2a37aad640 1 -- 172.21.15.33:0/4082666471 shutdown_connections 2024-09-16T00:07:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.738+0000 7f2a37aad640 1 -- 172.21.15.33:0/4082666471 wait complete. 2024-09-16T00:07:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.739+0000 7f2a37aad640 1 Processor -- start 2024-09-16T00:07:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.739+0000 7f2a37aad640 1 -- start start 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.739+0000 7f2a37aad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 0x7f2a3019d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.739+0000 7f2a37aad640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3010ba20 con 0x7f2a30108110 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.739+0000 7f2a35849640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 0x7f2a3019d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.739+0000 7f2a35849640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 0x7f2a3019d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56502/0 (socket says 172.21.15.33:56502) 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.739+0000 7f2a35849640 1 -- 172.21.15.33:0/1505777628 learned_addr learned my addr 172.21.15.33:0/1505777628 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.740+0000 7f2a35849640 1 -- 172.21.15.33:0/1505777628 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3019dd60 con 0x7f2a30108110 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.740+0000 7f2a35849640 1 --2- 172.21.15.33:0/1505777628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 0x7f2a3019d820 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2a20031ca0 tx=0x7f2a20031cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.740+0000 7f2a2effd640 1 -- 172.21.15.33:0/1505777628 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a20036050 con 0x7f2a30108110 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.740+0000 7f2a2effd640 1 -- 172.21.15.33:0/1505777628 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2a20042070 con 0x7f2a30108110 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.740+0000 7f2a37aad640 1 -- 172.21.15.33:0/1505777628 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3019dff0 con 0x7f2a30108110 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.740+0000 7f2a2effd640 1 -- 172.21.15.33:0/1505777628 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a2003d820 con 0x7f2a30108110 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.740+0000 7f2a37aad640 1 -- 172.21.15.33:0/1505777628 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a301a0ce0 con 0x7f2a30108110 2024-09-16T00:07:27.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.740+0000 7f2a2effd640 1 -- 172.21.15.33:0/1505777628 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2a2003db20 con 0x7f2a30108110 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.740+0000 7f2a2effd640 1 -- 172.21.15.33:0/1505777628 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2a20007e00 con 0x7f2a30108110 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.741+0000 7f2a37aad640 1 -- 172.21.15.33:0/1505777628 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a30108570 con 0x7f2a30108110 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.743+0000 7f2a2effd640 1 -- 172.21.15.33:0/1505777628 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f2a200072f0 con 0x7f2a30108110 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.802+0000 7f2a37aad640 1 -- 172.21.15.33:0/1505777628 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f2a301a12d0 con 0x7f2a30108110 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.804+0000 7f2a2effd640 1 -- 172.21.15.33:0/1505777628 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f2a20047460 con 0x7f2a30108110 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.805+0000 7f2a37aad640 1 -- 172.21.15.33:0/1505777628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 msgr2=0x7f2a3019d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.805+0000 7f2a37aad640 1 --2- 172.21.15.33:0/1505777628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 0x7f2a3019d820 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2a20031ca0 tx=0x7f2a20031cd0 comp rx=0 tx=0).stop 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.805+0000 7f2a37aad640 1 -- 172.21.15.33:0/1505777628 shutdown_connections 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.805+0000 7f2a37aad640 1 --2- 172.21.15.33:0/1505777628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a30108110 0x7f2a3019d820 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.805+0000 7f2a37aad640 1 -- 172.21.15.33:0/1505777628 >> 172.21.15.33:0/1505777628 conn(0x7f2a30101db0 msgr2=0x7f2a30102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.805+0000 7f2a37aad640 1 -- 172.21.15.33:0/1505777628 shutdown_connections 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:26.805+0000 7f2a37aad640 1 -- 172.21.15.33:0/1505777628 wait complete. 2024-09-16T00:07:27.330 INFO:teuthology.orchestra.run.smithi033.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-16T00:07:27.352 INFO:teuthology.orchestra.run.smithi033.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-16T00:07:27.353 INFO:teuthology.orchestra.run.smithi033.stdout:Creating mgr... 2024-09-16T00:07:27.353 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-16T00:07:27.353 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-16T00:07:27.621 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.x 2024-09-16T00:07:27.621 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.x.service: Unit ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.x.service not loaded. 2024-09-16T00:07:27.775 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968.target.wants/ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@.service. 2024-09-16T00:07:28.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:27 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1505777628' entity='client.admin' 2024-09-16T00:07:28.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:28 smithi033 podman[31454]: 2024-09-16 00:07:28.030108733 +0000 UTC m=+0.132493755 container create e964b53ad2d0c02ad6a4c1de177522c175e7178997af6feefa553e0241333d31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-16T00:07:28.540 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T00:07:28.540 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout disabled 2024-09-16T00:07:28.547 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T00:07:28.547 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-09-16T00:07:28.547 INFO:teuthology.orchestra.run.smithi033.stdout:firewalld.service is not enabled 2024-09-16T00:07:28.547 INFO:teuthology.orchestra.run.smithi033.stdout:Not possible to enable service . firewalld.service is not available 2024-09-16T00:07:28.557 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T00:07:28.557 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout disabled 2024-09-16T00:07:28.564 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T00:07:28.565 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-09-16T00:07:28.565 INFO:teuthology.orchestra.run.smithi033.stdout:firewalld.service is not enabled 2024-09-16T00:07:28.565 INFO:teuthology.orchestra.run.smithi033.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-16T00:07:28.565 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mgr to start... 2024-09-16T00:07:28.565 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mgr... 2024-09-16T00:07:28.638 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:28 smithi033 podman[31454]: 2024-09-16 00:07:28.440970109 +0000 UTC m=+0.543355081 container init e964b53ad2d0c02ad6a4c1de177522c175e7178997af6feefa553e0241333d31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-16T00:07:28.638 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:28 smithi033 podman[31454]: 2024-09-16 00:07:28.444073217 +0000 UTC m=+0.546458178 container start e964b53ad2d0c02ad6a4c1de177522c175e7178997af6feefa553e0241333d31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-16T00:07:28.638 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:28 smithi033 bash[31454]: e964b53ad2d0c02ad6a4c1de177522c175e7178997af6feefa553e0241333d31 2024-09-16T00:07:28.638 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:28 smithi033 systemd[1]: Started Ceph mgr.x for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:07:28.638 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:28 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:28.574+0000 7fe97ad27040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T00:07:28.638 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:28 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:28.636+0000 7fe97ad27040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T00:07:28.922 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:28 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:28.796+0000 7fe97ad27040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T00:07:28.922 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:28 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:28.866+0000 7fe97ad27040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T00:07:29.184 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:29 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4033503676' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-16T00:07:29.184 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:29 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:29.025+0000 7fe97ad27040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T00:07:29.184 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:29 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:29.183+0000 7fe97ad27040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T00:07:29.773 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:29 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:29.515+0000 7fe97ad27040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T00:07:29.829 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:29 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:29.578+0000 7fe97ad27040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T00:07:29.829 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:29 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:29.772+0000 7fe97ad27040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T00:07:29.860 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-09-16T00:07:29.861 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-09-16T00:07:29.861 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:07:29.861 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T00:07:29.861 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T00:07:29.861 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T00:07:29.861 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T00:07:29.861 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:29.861 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T00:07:29.861 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T00:07:29.874 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 0 2024-09-16T00:07:29.874 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-09-16T00:07:29.874 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T00:07:29.874 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T00:07:29.874 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-09-16T00:07:29.874 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-09-16T00:07:29.874 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T00:07:29.874 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T00:07:21:167536+0000", 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T00:07:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T00:07:29.877 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T00:07:21.168200+0000", 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.000+0000 7f71c200c640 1 Processor -- start 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.000+0000 7f71c200c640 1 -- start start 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.000+0000 7f71c200c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 0x7f71bc1062c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.000+0000 7f71c200c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc07be70 con 0x7f71bc105ec0 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.000+0000 7f71bb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 0x7f71bc1062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.001+0000 7f71bb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 0x7f71bc1062c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56522/0 (socket says 172.21.15.33:56522) 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.001+0000 7f71bb7fe640 1 -- 172.21.15.33:0/2062169465 learned_addr learned my addr 172.21.15.33:0/2062169465 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.001+0000 7f71bb7fe640 1 -- 172.21.15.33:0/2062169465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71bc106800 con 0x7f71bc105ec0 2024-09-16T00:07:29.878 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.002+0000 7f71bb7fe640 1 --2- 172.21.15.33:0/2062169465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 0x7f71bc1062c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009820 tx=0x7f71ac02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15e53c88a279b2d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.002+0000 7f71ba7fc640 1 -- 172.21.15.33:0/2062169465 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71ac037070 con 0x7f71bc105ec0 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.002+0000 7f71ba7fc640 1 -- 172.21.15.33:0/2062169465 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f71ac032900 con 0x7f71bc105ec0 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.002+0000 7f71ba7fc640 1 -- 172.21.15.33:0/2062169465 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71ac030850 con 0x7f71bc105ec0 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.003+0000 7f71c200c640 1 -- 172.21.15.33:0/2062169465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 msgr2=0x7f71bc1062c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.003+0000 7f71c200c640 1 --2- 172.21.15.33:0/2062169465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 0x7f71bc1062c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009820 tx=0x7f71ac02a270 comp rx=0 tx=0).stop 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.003+0000 7f71c200c640 1 -- 172.21.15.33:0/2062169465 shutdown_connections 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.003+0000 7f71c200c640 1 --2- 172.21.15.33:0/2062169465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 0x7f71bc1062c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.003+0000 7f71c200c640 1 -- 172.21.15.33:0/2062169465 >> 172.21.15.33:0/2062169465 conn(0x7f71bc101db0 msgr2=0x7f71bc1041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.003+0000 7f71c200c640 1 -- 172.21.15.33:0/2062169465 shutdown_connections 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.003+0000 7f71c200c640 1 -- 172.21.15.33:0/2062169465 wait complete. 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.003+0000 7f71c200c640 1 Processor -- start 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.004+0000 7f71c200c640 1 -- start start 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.004+0000 7f71c200c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 0x7f71bc19d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:29.879 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.004+0000 7f71c200c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc107900 con 0x7f71bc105ec0 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.004+0000 7f71bb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 0x7f71bc19d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.004+0000 7f71bb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 0x7f71bc19d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56524/0 (socket says 172.21.15.33:56524) 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.004+0000 7f71bb7fe640 1 -- 172.21.15.33:0/4033503676 learned_addr learned my addr 172.21.15.33:0/4033503676 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.004+0000 7f71bb7fe640 1 -- 172.21.15.33:0/4033503676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71bc19dcf0 con 0x7f71bc105ec0 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.004+0000 7f71bb7fe640 1 --2- 172.21.15.33:0/4033503676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 0x7f71bc19d7b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f71ac032ce0 tx=0x7f71ac032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.004+0000 7f71b8ff9640 1 -- 172.21.15.33:0/4033503676 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71ac042070 con 0x7f71bc105ec0 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.005+0000 7f71c200c640 1 -- 172.21.15.33:0/4033503676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71bc19df80 con 0x7f71bc105ec0 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.005+0000 7f71b8ff9640 1 -- 172.21.15.33:0/4033503676 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f71ac0308c0 con 0x7f71bc105ec0 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.005+0000 7f71b8ff9640 1 -- 172.21.15.33:0/4033503676 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71ac037040 con 0x7f71bc105ec0 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.005+0000 7f71c200c640 1 -- 172.21.15.33:0/4033503676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71bc1a0c70 con 0x7f71bc105ec0 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.005+0000 7f71b8ff9640 1 -- 172.21.15.33:0/4033503676 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f71ac045070 con 0x7f71bc105ec0 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.005+0000 7f71b8ff9640 1 -- 172.21.15.33:0/4033503676 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f71ac047bc0 con 0x7f71bc105ec0 2024-09-16T00:07:29.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.006+0000 7f71c200c640 1 -- 172.21.15.33:0/4033503676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71bc1062c0 con 0x7f71bc105ec0 2024-09-16T00:07:29.881 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.010+0000 7f71b8ff9640 1 -- 172.21.15.33:0/4033503676 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f71ac03c7e0 con 0x7f71bc105ec0 2024-09-16T00:07:29.881 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.073+0000 7f71c200c640 1 -- 172.21.15.33:0/4033503676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f71bc02d940 con 0x7f71bc105ec0 2024-09-16T00:07:29.881 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.073+0000 7f71b8ff9640 1 -- 172.21.15.33:0/4033503676 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f71ac03c9c0 con 0x7f71bc105ec0 2024-09-16T00:07:29.881 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.075+0000 7f71c200c640 1 -- 172.21.15.33:0/4033503676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 msgr2=0x7f71bc19d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:29.881 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.075+0000 7f71c200c640 1 --2- 172.21.15.33:0/4033503676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 0x7f71bc19d7b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f71ac032ce0 tx=0x7f71ac032d10 comp rx=0 tx=0).stop 2024-09-16T00:07:29.881 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.075+0000 7f71c200c640 1 -- 172.21.15.33:0/4033503676 shutdown_connections 2024-09-16T00:07:29.881 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.075+0000 7f71c200c640 1 --2- 172.21.15.33:0/4033503676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f71bc105ec0 0x7f71bc19d7b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:29.881 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.075+0000 7f71c200c640 1 -- 172.21.15.33:0/4033503676 >> 172.21.15.33:0/4033503676 conn(0x7f71bc101db0 msgr2=0x7f71bc194410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:29.881 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.075+0000 7f71c200c640 1 -- 172.21.15.33:0/4033503676 shutdown_connections 2024-09-16T00:07:29.881 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:29.075+0000 7f71c200c640 1 -- 172.21.15.33:0/4033503676 wait complete. 2024-09-16T00:07:29.881 INFO:teuthology.orchestra.run.smithi033.stdout:mgr not available, waiting (1/15)... 2024-09-16T00:07:31.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:30 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:30.696+0000 7fe97ad27040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T00:07:31.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:30 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:30.766+0000 7fe97ad27040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T00:07:31.792 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:31 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:31.522+0000 7fe97ad27040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T00:07:31.792 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:31 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:31.642+0000 7fe97ad27040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T00:07:32.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:31 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:31.791+0000 7fe97ad27040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T00:07:32.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:31 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:31.855+0000 7fe97ad27040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T00:07:32.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:32 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:32.016+0000 7fe97ad27040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T00:07:32.361 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:32 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:32.087+0000 7fe97ad27040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T00:07:32.361 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:32 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:32.154+0000 7fe97ad27040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T00:07:32.361 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:32 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:32.323+0000 7fe97ad27040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T00:07:32.621 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:32 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2086515549' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-16T00:07:32.622 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:32 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:32.398+0000 7fe97ad27040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T00:07:32.622 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:32 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:32.471+0000 7fe97ad27040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T00:07:32.622 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:32 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:32.557+0000 7fe97ad27040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T00:07:32.622 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:32 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:32.620+0000 7fe97ad27040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T00:07:32.955 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:32 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:32.683+0000 7fe97ad27040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T00:07:32.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-09-16T00:07:32.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-09-16T00:07:32.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:07:32.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T00:07:32.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T00:07:32.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T00:07:32.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T00:07:32.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:32.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T00:07:32.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T00:07:32.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 0 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T00:07:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T00:07:32.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T00:07:32.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T00:07:32.970 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T00:07:32.970 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T00:07:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T00:07:21:167536+0000", 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-09-16T00:07:32.972 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T00:07:21.168200+0000", 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.289+0000 7ff85533f640 1 Processor -- start 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.290+0000 7ff85533f640 1 -- start start 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.290+0000 7ff85533f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 0x7ff85010a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.290+0000 7ff85533f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff85007be70 con 0x7ff8501080d0 2024-09-16T00:07:32.973 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.290+0000 7ff84effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 0x7ff85010a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.290+0000 7ff84effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 0x7ff85010a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55214/0 (socket says 172.21.15.33:55214) 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.290+0000 7ff84effd640 1 -- 172.21.15.33:0/1070628609 learned_addr learned my addr 172.21.15.33:0/1070628609 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.290+0000 7ff84effd640 1 -- 172.21.15.33:0/1070628609 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85010aa00 con 0x7ff8501080d0 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.291+0000 7ff84effd640 1 --2- 172.21.15.33:0/1070628609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 0x7ff85010a4c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff840009820 tx=0x7ff84002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ea99dba48f86d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.291+0000 7ff84e7fc640 1 -- 172.21.15.33:0/1070628609 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff840037070 con 0x7ff8501080d0 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.291+0000 7ff84e7fc640 1 -- 172.21.15.33:0/1070628609 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff840032900 con 0x7ff8501080d0 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.291+0000 7ff84e7fc640 1 -- 172.21.15.33:0/1070628609 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff840030850 con 0x7ff8501080d0 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.292+0000 7ff85533f640 1 -- 172.21.15.33:0/1070628609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 msgr2=0x7ff85010a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.292+0000 7ff85533f640 1 --2- 172.21.15.33:0/1070628609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 0x7ff85010a4c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff840009820 tx=0x7ff84002a270 comp rx=0 tx=0).stop 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.292+0000 7ff85533f640 1 -- 172.21.15.33:0/1070628609 shutdown_connections 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.292+0000 7ff85533f640 1 --2- 172.21.15.33:0/1070628609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 0x7ff85010a4c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.292+0000 7ff85533f640 1 -- 172.21.15.33:0/1070628609 >> 172.21.15.33:0/1070628609 conn(0x7ff850101db0 msgr2=0x7ff8501041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:32.974 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.292+0000 7ff85533f640 1 -- 172.21.15.33:0/1070628609 shutdown_connections 2024-09-16T00:07:32.975 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.292+0000 7ff85533f640 1 -- 172.21.15.33:0/1070628609 wait complete. 2024-09-16T00:07:32.975 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.293+0000 7ff85533f640 1 Processor -- start 2024-09-16T00:07:32.975 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.293+0000 7ff85533f640 1 -- start start 2024-09-16T00:07:32.975 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.293+0000 7ff85533f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 0x7ff85019b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:32.975 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.293+0000 7ff85533f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff85010ba40 con 0x7ff8501080d0 2024-09-16T00:07:32.975 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.293+0000 7ff84effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 0x7ff85019b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:32.975 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.293+0000 7ff84effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 0x7ff85019b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55228/0 (socket says 172.21.15.33:55228) 2024-09-16T00:07:32.975 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.293+0000 7ff84effd640 1 -- 172.21.15.33:0/2086515549 learned_addr learned my addr 172.21.15.33:0/2086515549 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:32.975 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.294+0000 7ff84effd640 1 -- 172.21.15.33:0/2086515549 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85019bb50 con 0x7ff8501080d0 2024-09-16T00:07:32.975 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.294+0000 7ff84effd640 1 --2- 172.21.15.33:0/2086515549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 0x7ff85019b610 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff840032ce0 tx=0x7ff840032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:32.975 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.294+0000 7ff84cff9640 1 -- 172.21.15.33:0/2086515549 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff840042070 con 0x7ff8501080d0 2024-09-16T00:07:32.975 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.294+0000 7ff84cff9640 1 -- 172.21.15.33:0/2086515549 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff8400308c0 con 0x7ff8501080d0 2024-09-16T00:07:32.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.294+0000 7ff85533f640 1 -- 172.21.15.33:0/2086515549 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff85019bde0 con 0x7ff8501080d0 2024-09-16T00:07:32.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.294+0000 7ff84cff9640 1 -- 172.21.15.33:0/2086515549 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff840037040 con 0x7ff8501080d0 2024-09-16T00:07:32.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.294+0000 7ff85533f640 1 -- 172.21.15.33:0/2086515549 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff85019ead0 con 0x7ff8501080d0 2024-09-16T00:07:32.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.295+0000 7ff85533f640 1 -- 172.21.15.33:0/2086515549 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff850108530 con 0x7ff8501080d0 2024-09-16T00:07:32.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.295+0000 7ff84cff9640 1 -- 172.21.15.33:0/2086515549 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff840045070 con 0x7ff8501080d0 2024-09-16T00:07:32.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.295+0000 7ff84cff9640 1 -- 172.21.15.33:0/2086515549 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff840047780 con 0x7ff8501080d0 2024-09-16T00:07:32.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.297+0000 7ff84cff9640 1 -- 172.21.15.33:0/2086515549 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7ff84003d430 con 0x7ff8501080d0 2024-09-16T00:07:32.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.360+0000 7ff85533f640 1 -- 172.21.15.33:0/2086515549 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff85019eca0 con 0x7ff8501080d0 2024-09-16T00:07:32.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.360+0000 7ff84cff9640 1 -- 172.21.15.33:0/2086515549 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7ff84003d610 con 0x7ff8501080d0 2024-09-16T00:07:32.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.361+0000 7ff85533f640 1 -- 172.21.15.33:0/2086515549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 msgr2=0x7ff85019b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:32.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.361+0000 7ff85533f640 1 --2- 172.21.15.33:0/2086515549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 0x7ff85019b610 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff840032ce0 tx=0x7ff840032d10 comp rx=0 tx=0).stop 2024-09-16T00:07:32.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.361+0000 7ff85533f640 1 -- 172.21.15.33:0/2086515549 shutdown_connections 2024-09-16T00:07:32.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.361+0000 7ff85533f640 1 --2- 172.21.15.33:0/2086515549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8501080d0 0x7ff85019b610 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:32.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.361+0000 7ff85533f640 1 -- 172.21.15.33:0/2086515549 >> 172.21.15.33:0/2086515549 conn(0x7ff850101db0 msgr2=0x7ff8501028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:32.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.361+0000 7ff85533f640 1 -- 172.21.15.33:0/2086515549 shutdown_connections 2024-09-16T00:07:32.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:32.362+0000 7ff85533f640 1 -- 172.21.15.33:0/2086515549 wait complete. 2024-09-16T00:07:32.977 INFO:teuthology.orchestra.run.smithi033.stdout:mgr not available, waiting (2/15)... 2024-09-16T00:07:33.830 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:33.514+0000 7fe97ad27040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T00:07:33.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: Activating manager daemon x 2024-09-16T00:07:33.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: mgrmap e2: x(active, starting, since 0.00314555s) 2024-09-16T00:07:33.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: from='mgr.14100 172.21.15.33:0/1555969466' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-16T00:07:33.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: from='mgr.14100 172.21.15.33:0/1555969466' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-16T00:07:33.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: from='mgr.14100 172.21.15.33:0/1555969466' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-16T00:07:33.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: from='mgr.14100 172.21.15.33:0/1555969466' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:07:33.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: from='mgr.14100 172.21.15.33:0/1555969466' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-16T00:07:33.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: Manager daemon x is now available 2024-09-16T00:07:33.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: from='mgr.14100 172.21.15.33:0/1555969466' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-16T00:07:33.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: from='mgr.14100 172.21.15.33:0/1555969466' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-16T00:07:33.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: from='mgr.14100 172.21.15.33:0/1555969466' entity='mgr.x' 2024-09-16T00:07:33.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: from='mgr.14100 172.21.15.33:0/1555969466' entity='mgr.x' 2024-09-16T00:07:33.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:33 smithi033 ceph-mon[31243]: from='mgr.14100 172.21.15.33:0/1555969466' entity='mgr.x' 2024-09-16T00:07:35.648 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:35 smithi033 ceph-mon[31243]: mgrmap e3: x(active, since 1.0071s) 2024-09-16T00:07:36.208 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-09-16T00:07:36.208 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-09-16T00:07:36.208 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:07:36.208 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T00:07:36.208 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T00:07:36.208 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T00:07:36.208 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T00:07:36.208 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:36.208 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T00:07:36.208 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 0 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T00:07:36.209 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T00:07:36.210 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T00:07:36.210 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T00:07:36.210 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T00:07:36.210 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T00:07:36.210 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T00:07:36.210 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:36.210 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T00:07:36.210 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T00:07:36.210 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T00:07:36.210 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T00:07:21:167536+0000", 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:36.211 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T00:07:21.168200+0000", 2024-09-16T00:07:36.212 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T00:07:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-09-16T00:07:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T00:07:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-09-16T00:07:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.389+0000 7f00e5e3b640 1 Processor -- start 2024-09-16T00:07:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.389+0000 7f00e5e3b640 1 -- start start 2024-09-16T00:07:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.390+0000 7f00e5e3b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 0x7f00e0106920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.390+0000 7f00e5e3b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00e0106ef0 con 0x7f00e0106520 2024-09-16T00:07:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.390+0000 7f00df7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 0x7f00e0106920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.390+0000 7f00df7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 0x7f00e0106920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55310/0 (socket says 172.21.15.33:55310) 2024-09-16T00:07:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.390+0000 7f00df7fe640 1 -- 172.21.15.33:0/2269398415 learned_addr learned my addr 172.21.15.33:0/2269398415 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.390+0000 7f00df7fe640 1 -- 172.21.15.33:0/2269398415 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e0107720 con 0x7f00e0106520 2024-09-16T00:07:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.391+0000 7f00df7fe640 1 --2- 172.21.15.33:0/2269398415 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 0x7f00e0106920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009820 tx=0x7f00d002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f925d6c064c11a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.391+0000 7f00de7fc640 1 -- 172.21.15.33:0/2269398415 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00d0037070 con 0x7f00e0106520 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.391+0000 7f00de7fc640 1 -- 172.21.15.33:0/2269398415 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f00d0032900 con 0x7f00e0106520 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.391+0000 7f00de7fc640 1 -- 172.21.15.33:0/2269398415 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00d0030850 con 0x7f00e0106520 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.392+0000 7f00e5e3b640 1 -- 172.21.15.33:0/2269398415 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 msgr2=0x7f00e0106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.392+0000 7f00e5e3b640 1 --2- 172.21.15.33:0/2269398415 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 0x7f00e0106920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009820 tx=0x7f00d002a270 comp rx=0 tx=0).stop 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.392+0000 7f00e5e3b640 1 -- 172.21.15.33:0/2269398415 shutdown_connections 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.392+0000 7f00e5e3b640 1 --2- 172.21.15.33:0/2269398415 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 0x7f00e0106920 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.392+0000 7f00e5e3b640 1 -- 172.21.15.33:0/2269398415 >> 172.21.15.33:0/2269398415 conn(0x7f00e0101db0 msgr2=0x7f00e01041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.392+0000 7f00e5e3b640 1 -- 172.21.15.33:0/2269398415 shutdown_connections 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.392+0000 7f00e5e3b640 1 -- 172.21.15.33:0/2269398415 wait complete. 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.393+0000 7f00e5e3b640 1 Processor -- start 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.393+0000 7f00e5e3b640 1 -- start start 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.393+0000 7f00e5e3b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 0x7f00e019b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:36.214 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.393+0000 7f00e5e3b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00e010aad0 con 0x7f00e0106520 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.393+0000 7f00df7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 0x7f00e019b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.393+0000 7f00df7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 0x7f00e019b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55322/0 (socket says 172.21.15.33:55322) 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.393+0000 7f00df7fe640 1 -- 172.21.15.33:0/1992522005 learned_addr learned my addr 172.21.15.33:0/1992522005 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.394+0000 7f00df7fe640 1 -- 172.21.15.33:0/1992522005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e019bb80 con 0x7f00e0106520 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.394+0000 7f00df7fe640 1 --2- 172.21.15.33:0/1992522005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 0x7f00e019b640 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009940 tx=0x7f00d0032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.394+0000 7f00dcff9640 1 -- 172.21.15.33:0/1992522005 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00d0042070 con 0x7f00e0106520 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.394+0000 7f00dcff9640 1 -- 172.21.15.33:0/1992522005 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f00d00308c0 con 0x7f00e0106520 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.394+0000 7f00e5e3b640 1 -- 172.21.15.33:0/1992522005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e019be10 con 0x7f00e0106520 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.394+0000 7f00dcff9640 1 -- 172.21.15.33:0/1992522005 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00d0037040 con 0x7f00e0106520 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.394+0000 7f00e5e3b640 1 -- 172.21.15.33:0/1992522005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e019eb00 con 0x7f00e0106520 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.395+0000 7f00dcff9640 1 -- 172.21.15.33:0/1992522005 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 3) ==== 46344+0+0 (secure 0 0 0) 0x7f00d0045070 con 0x7f00e0106520 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.395+0000 7f00e5e3b640 1 -- 172.21.15.33:0/1992522005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00e010ac50 con 0x7f00e0106520 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.395+0000 7f00dcff9640 1 --2- 172.21.15.33:0/1992522005 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f00cc0390a0 0x7f00cc03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:36.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.395+0000 7f00dcff9640 1 -- 172.21.15.33:0/1992522005 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f00d006c130 con 0x7f00e0106520 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.395+0000 7f00deffd640 1 --2- 172.21.15.33:0/1992522005 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f00cc0390a0 0x7f00cc03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.396+0000 7f00deffd640 1 --2- 172.21.15.33:0/1992522005 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f00cc0390a0 0x7f00cc03b560 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f00c80098c0 tx=0x7f00c8006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.401+0000 7f00dcff9640 1 -- 172.21.15.33:0/1992522005 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f00d0045360 con 0x7f00e0106520 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.521+0000 7f00dcff9640 1 -- 172.21.15.33:0/1992522005 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7f00d003c760 con 0x7f00e0106520 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.648+0000 7f00e5e3b640 1 -- 172.21.15.33:0/1992522005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f00e00008d0 con 0x7f00e0106520 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.648+0000 7f00dcff9640 1 -- 172.21.15.33:0/1992522005 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1290 (secure 0 0 0) 0x7f00d0069090 con 0x7f00e0106520 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.650+0000 7f00e5e3b640 1 -- 172.21.15.33:0/1992522005 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f00cc0390a0 msgr2=0x7f00cc03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.650+0000 7f00e5e3b640 1 --2- 172.21.15.33:0/1992522005 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f00cc0390a0 0x7f00cc03b560 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f00c80098c0 tx=0x7f00c8006db0 comp rx=0 tx=0).stop 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.650+0000 7f00e5e3b640 1 -- 172.21.15.33:0/1992522005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 msgr2=0x7f00e019b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.650+0000 7f00e5e3b640 1 --2- 172.21.15.33:0/1992522005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 0x7f00e019b640 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009940 tx=0x7f00d0032ce0 comp rx=0 tx=0).stop 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.651+0000 7f00e5e3b640 1 -- 172.21.15.33:0/1992522005 shutdown_connections 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.651+0000 7f00e5e3b640 1 --2- 172.21.15.33:0/1992522005 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f00cc0390a0 0x7f00cc03b560 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.651+0000 7f00e5e3b640 1 --2- 172.21.15.33:0/1992522005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e0106520 0x7f00e019b640 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:36.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.651+0000 7f00e5e3b640 1 -- 172.21.15.33:0/1992522005 >> 172.21.15.33:0/1992522005 conn(0x7f00e0101db0 msgr2=0x7f00e0102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:36.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.651+0000 7f00e5e3b640 1 -- 172.21.15.33:0/1992522005 shutdown_connections 2024-09-16T00:07:36.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:35.651+0000 7f00e5e3b640 1 -- 172.21.15.33:0/1992522005 wait complete. 2024-09-16T00:07:36.217 INFO:teuthology.orchestra.run.smithi033.stdout:mgr is available 2024-09-16T00:07:36.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:36 smithi033 ceph-mon[31243]: mgrmap e4: x(active, since 2s) 2024-09-16T00:07:36.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:36 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1992522005' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-16T00:07:37.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-09-16T00:07:37.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [global] 2024-09-16T00:07:37.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout fsid = 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:07:37.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.33:3300,v1:172.21.15.33:6789] 2024-09-16T00:07:37.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [osd] 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-16T00:07:37.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-16T00:07:37.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.593+0000 7f5be243c640 1 Processor -- start 2024-09-16T00:07:37.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.593+0000 7f5be243c640 1 -- start start 2024-09-16T00:07:37.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.594+0000 7f5be243c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 0x7f5bdc107a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:37.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.594+0000 7f5be243c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc108050 con 0x7f5bdc107680 2024-09-16T00:07:37.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.594+0000 7f5bdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 0x7f5bdc107a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:37.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.594+0000 7f5bdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 0x7f5bdc107a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55326/0 (socket says 172.21.15.33:55326) 2024-09-16T00:07:37.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.594+0000 7f5bdbfff640 1 -- 172.21.15.33:0/1909168713 learned_addr learned my addr 172.21.15.33:0/1909168713 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:37.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.594+0000 7f5bdbfff640 1 -- 172.21.15.33:0/1909168713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bdc108830 con 0x7f5bdc107680 2024-09-16T00:07:37.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.595+0000 7f5bdbfff640 1 --2- 172.21.15.33:0/1909168713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 0x7f5bdc107a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009820 tx=0x7f5bcc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f6dcc68e561c74 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:37.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.595+0000 7f5bdaffd640 1 -- 172.21.15.33:0/1909168713 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bcc037070 con 0x7f5bdc107680 2024-09-16T00:07:37.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.595+0000 7f5bdaffd640 1 -- 172.21.15.33:0/1909168713 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5bcc032900 con 0x7f5bdc107680 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.595+0000 7f5bdaffd640 1 -- 172.21.15.33:0/1909168713 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bcc030850 con 0x7f5bdc107680 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.595+0000 7f5be243c640 1 -- 172.21.15.33:0/1909168713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 msgr2=0x7f5bdc107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.595+0000 7f5be243c640 1 --2- 172.21.15.33:0/1909168713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 0x7f5bdc107a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009820 tx=0x7f5bcc02a270 comp rx=0 tx=0).stop 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.596+0000 7f5be243c640 1 -- 172.21.15.33:0/1909168713 shutdown_connections 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.596+0000 7f5be243c640 1 --2- 172.21.15.33:0/1909168713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 0x7f5bdc107a80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.596+0000 7f5be243c640 1 -- 172.21.15.33:0/1909168713 >> 172.21.15.33:0/1909168713 conn(0x7f5bdc102e50 msgr2=0x7f5bdc105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.596+0000 7f5be243c640 1 -- 172.21.15.33:0/1909168713 shutdown_connections 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.596+0000 7f5be243c640 1 -- 172.21.15.33:0/1909168713 wait complete. 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.596+0000 7f5be243c640 1 Processor -- start 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.596+0000 7f5be243c640 1 -- start start 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.596+0000 7f5be243c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 0x7f5bdc19b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.597+0000 7f5be243c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc10bbe0 con 0x7f5bdc107680 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.597+0000 7f5bdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 0x7f5bdc19b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.597+0000 7f5bdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 0x7f5bdc19b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55342/0 (socket says 172.21.15.33:55342) 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.597+0000 7f5bdbfff640 1 -- 172.21.15.33:0/2147121332 learned_addr learned my addr 172.21.15.33:0/2147121332 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.597+0000 7f5bdbfff640 1 -- 172.21.15.33:0/2147121332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bdc19bbc0 con 0x7f5bdc107680 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.597+0000 7f5bdbfff640 1 --2- 172.21.15.33:0/2147121332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 0x7f5bdc19b680 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009940 tx=0x7f5bcc0326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.597+0000 7f5bd97fa640 1 -- 172.21.15.33:0/2147121332 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bcc042070 con 0x7f5bdc107680 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.597+0000 7f5be243c640 1 -- 172.21.15.33:0/2147121332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bdc19be50 con 0x7f5bdc107680 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.597+0000 7f5bd97fa640 1 -- 172.21.15.33:0/2147121332 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5bcc032be0 con 0x7f5bdc107680 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.597+0000 7f5bd97fa640 1 -- 172.21.15.33:0/2147121332 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bcc037040 con 0x7f5bdc107680 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.597+0000 7f5be243c640 1 -- 172.21.15.33:0/2147121332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bdc19eb40 con 0x7f5bdc107680 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.598+0000 7f5bd97fa640 1 -- 172.21.15.33:0/2147121332 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7f5bcc045070 con 0x7f5bdc107680 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.598+0000 7f5be243c640 1 -- 172.21.15.33:0/2147121332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bdc10bd60 con 0x7f5bdc107680 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.598+0000 7f5bd97fa640 1 --2- 172.21.15.33:0/2147121332 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f5bc4039160 0x7f5bc403b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.598+0000 7f5bd97fa640 1 -- 172.21.15.33:0/2147121332 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5bcc06c2c0 con 0x7f5bdc107680 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.599+0000 7f5bdb7fe640 1 --2- 172.21.15.33:0/2147121332 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f5bc4039160 0x7f5bc403b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:37.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.599+0000 7f5bdb7fe640 1 --2- 172.21.15.33:0/2147121332 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f5bc4039160 0x7f5bc403b620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00098c0 tx=0x7f5bd0006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.603+0000 7f5bd97fa640 1 -- 172.21.15.33:0/2147121332 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5bcc06bce0 con 0x7f5bdc107680 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.778+0000 7f5be243c640 1 -- 172.21.15.33:0/2147121332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f5bdc107b00 con 0x7f5bdc107680 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.779+0000 7f5bd97fa640 1 -- 172.21.15.33:0/2147121332 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+429 (secure 0 0 0) 0x7f5bcc03b330 con 0x7f5bdc107680 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.781+0000 7f5be243c640 1 -- 172.21.15.33:0/2147121332 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f5bc4039160 msgr2=0x7f5bc403b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.781+0000 7f5be243c640 1 --2- 172.21.15.33:0/2147121332 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f5bc4039160 0x7f5bc403b620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00098c0 tx=0x7f5bd0006db0 comp rx=0 tx=0).stop 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.781+0000 7f5be243c640 1 -- 172.21.15.33:0/2147121332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 msgr2=0x7f5bdc19b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.781+0000 7f5be243c640 1 --2- 172.21.15.33:0/2147121332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 0x7f5bdc19b680 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009940 tx=0x7f5bcc0326e0 comp rx=0 tx=0).stop 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.781+0000 7f5be243c640 1 -- 172.21.15.33:0/2147121332 shutdown_connections 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.781+0000 7f5be243c640 1 --2- 172.21.15.33:0/2147121332 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f5bc4039160 0x7f5bc403b620 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.781+0000 7f5be243c640 1 --2- 172.21.15.33:0/2147121332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bdc107680 0x7f5bdc19b680 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.781+0000 7f5be243c640 1 -- 172.21.15.33:0/2147121332 >> 172.21.15.33:0/2147121332 conn(0x7f5bdc102e50 msgr2=0x7f5bdc103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.781+0000 7f5be243c640 1 -- 172.21.15.33:0/2147121332 shutdown_connections 2024-09-16T00:07:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:36.781+0000 7f5be243c640 1 -- 172.21.15.33:0/2147121332 wait complete. 2024-09-16T00:07:37.333 INFO:teuthology.orchestra.run.smithi033.stdout:Enabling cephadm module... 2024-09-16T00:07:37.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:37 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2147121332' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-16T00:07:38.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:38 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2249827643' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-16T00:07:38.830 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:38 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: ignoring --setuser ceph since I am not root 2024-09-16T00:07:38.830 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:38 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: ignoring --setgroup ceph since I am not root 2024-09-16T00:07:38.830 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:38 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:38.674+0000 7fdf0db94040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T00:07:38.831 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:38 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:38.733+0000 7fdf0db94040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T00:07:39.113 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:38 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:38.888+0000 7fdf0db94040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T00:07:39.113 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:38 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:38.957+0000 7fdf0db94040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T00:07:39.113 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:39 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:39.112+0000 7fdf0db94040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T00:07:39.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.694+0000 7fd55f69c640 1 Processor -- start 2024-09-16T00:07:39.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.694+0000 7fd55f69c640 1 -- start start 2024-09-16T00:07:39.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.694+0000 7fd55f69c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 0x7fd558106920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:39.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.694+0000 7fd55f69c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd558106ef0 con 0x7fd558106520 2024-09-16T00:07:39.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.695+0000 7fd55d438640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 0x7fd558106920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:39.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.695+0000 7fd55d438640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 0x7fd558106920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55348/0 (socket says 172.21.15.33:55348) 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.695+0000 7fd55d438640 1 -- 172.21.15.33:0/683157840 learned_addr learned my addr 172.21.15.33:0/683157840 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.695+0000 7fd55d438640 1 -- 172.21.15.33:0/683157840 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd558107720 con 0x7fd558106520 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.696+0000 7fd55d438640 1 --2- 172.21.15.33:0/683157840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 0x7fd558106920 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd550009820 tx=0x7fd55002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b4ef431cbd2cc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.696+0000 7fd557fff640 1 -- 172.21.15.33:0/683157840 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd550037070 con 0x7fd558106520 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.696+0000 7fd557fff640 1 -- 172.21.15.33:0/683157840 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd550032900 con 0x7fd558106520 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.696+0000 7fd557fff640 1 -- 172.21.15.33:0/683157840 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd550030850 con 0x7fd558106520 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.696+0000 7fd55f69c640 1 -- 172.21.15.33:0/683157840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 msgr2=0x7fd558106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.696+0000 7fd55f69c640 1 --2- 172.21.15.33:0/683157840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 0x7fd558106920 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd550009820 tx=0x7fd55002a270 comp rx=0 tx=0).stop 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.696+0000 7fd55f69c640 1 -- 172.21.15.33:0/683157840 shutdown_connections 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.696+0000 7fd55f69c640 1 --2- 172.21.15.33:0/683157840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 0x7fd558106920 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.696+0000 7fd55f69c640 1 -- 172.21.15.33:0/683157840 >> 172.21.15.33:0/683157840 conn(0x7fd558101db0 msgr2=0x7fd5581041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.697+0000 7fd55f69c640 1 -- 172.21.15.33:0/683157840 shutdown_connections 2024-09-16T00:07:39.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.697+0000 7fd55f69c640 1 -- 172.21.15.33:0/683157840 wait complete. 2024-09-16T00:07:39.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.697+0000 7fd55f69c640 1 Processor -- start 2024-09-16T00:07:39.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.697+0000 7fd55f69c640 1 -- start start 2024-09-16T00:07:39.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.697+0000 7fd55f69c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 0x7fd55819b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:39.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.697+0000 7fd55f69c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd55810aad0 con 0x7fd558106520 2024-09-16T00:07:39.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.698+0000 7fd55d438640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 0x7fd55819b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:39.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.698+0000 7fd55d438640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 0x7fd55819b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55362/0 (socket says 172.21.15.33:55362) 2024-09-16T00:07:39.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.698+0000 7fd55d438640 1 -- 172.21.15.33:0/2249827643 learned_addr learned my addr 172.21.15.33:0/2249827643 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:39.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.698+0000 7fd55d438640 1 -- 172.21.15.33:0/2249827643 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd55819bb70 con 0x7fd558106520 2024-09-16T00:07:39.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.698+0000 7fd55d438640 1 --2- 172.21.15.33:0/2249827643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 0x7fd55819b630 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd550009940 tx=0x7fd550032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:39.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.698+0000 7fd5567fc640 1 -- 172.21.15.33:0/2249827643 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd550042070 con 0x7fd558106520 2024-09-16T00:07:39.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.698+0000 7fd5567fc640 1 -- 172.21.15.33:0/2249827643 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd5500308c0 con 0x7fd558106520 2024-09-16T00:07:39.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.698+0000 7fd55f69c640 1 -- 172.21.15.33:0/2249827643 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd55819be00 con 0x7fd558106520 2024-09-16T00:07:39.194 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.698+0000 7fd5567fc640 1 -- 172.21.15.33:0/2249827643 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd550037040 con 0x7fd558106520 2024-09-16T00:07:39.194 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.698+0000 7fd55f69c640 1 -- 172.21.15.33:0/2249827643 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd55819eaf0 con 0x7fd558106520 2024-09-16T00:07:39.194 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.699+0000 7fd5567fc640 1 -- 172.21.15.33:0/2249827643 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7fd550045070 con 0x7fd558106520 2024-09-16T00:07:39.194 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.699+0000 7fd5567fc640 1 --2- 172.21.15.33:0/2249827643 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fd5400391b0 0x7fd54003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:39.194 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.699+0000 7fd55f69c640 1 -- 172.21.15.33:0/2249827643 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd55810ac50 con 0x7fd558106520 2024-09-16T00:07:39.194 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.699+0000 7fd5567fc640 1 -- 172.21.15.33:0/2249827643 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd55006c2a0 con 0x7fd558106520 2024-09-16T00:07:39.194 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.699+0000 7fd55cc37640 1 --2- 172.21.15.33:0/2249827643 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fd5400391b0 0x7fd54003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:39.194 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.700+0000 7fd55cc37640 1 --2- 172.21.15.33:0/2249827643 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fd5400391b0 0x7fd54003b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd54c0098c0 tx=0x7fd54c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:39.194 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.704+0000 7fd5567fc640 1 -- 172.21.15.33:0/2249827643 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd55004b170 con 0x7fd558106520 2024-09-16T00:07:39.194 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:37.924+0000 7fd55f69c640 1 -- 172.21.15.33:0/2249827643 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fd55802da10 con 0x7fd558106520 2024-09-16T00:07:39.194 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:38.528+0000 7fd5567fc640 1 -- 172.21.15.33:0/2249827643 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fd55006bcc0 con 0x7fd558106520 2024-09-16T00:07:39.195 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:38.529+0000 7fd5567fc640 1 -- 172.21.15.33:0/2249827643 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7fd550045c40 con 0x7fd558106520 2024-09-16T00:07:39.195 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:38.531+0000 7fd55f69c640 1 -- 172.21.15.33:0/2249827643 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fd5400391b0 msgr2=0x7fd54003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:39.195 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:38.531+0000 7fd55f69c640 1 --2- 172.21.15.33:0/2249827643 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fd5400391b0 0x7fd54003b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd54c0098c0 tx=0x7fd54c006db0 comp rx=0 tx=0).stop 2024-09-16T00:07:39.195 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:38.531+0000 7fd55f69c640 1 -- 172.21.15.33:0/2249827643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 msgr2=0x7fd55819b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:39.195 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:38.531+0000 7fd55f69c640 1 --2- 172.21.15.33:0/2249827643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 0x7fd55819b630 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd550009940 tx=0x7fd550032ce0 comp rx=0 tx=0).stop 2024-09-16T00:07:39.195 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:38.531+0000 7fd55f69c640 1 -- 172.21.15.33:0/2249827643 shutdown_connections 2024-09-16T00:07:39.195 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:38.531+0000 7fd55f69c640 1 --2- 172.21.15.33:0/2249827643 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fd5400391b0 0x7fd54003b670 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:39.195 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:38.531+0000 7fd55f69c640 1 --2- 172.21.15.33:0/2249827643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd558106520 0x7fd55819b630 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:39.195 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:38.531+0000 7fd55f69c640 1 -- 172.21.15.33:0/2249827643 >> 172.21.15.33:0/2249827643 conn(0x7fd558101db0 msgr2=0x7fd558102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:39.195 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:38.531+0000 7fd55f69c640 1 -- 172.21.15.33:0/2249827643 shutdown_connections 2024-09-16T00:07:39.195 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:38.531+0000 7fd55f69c640 1 -- 172.21.15.33:0/2249827643 wait complete. 2024-09-16T00:07:39.427 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:39 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:39.280+0000 7fdf0db94040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T00:07:39.681 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:39 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:39.615+0000 7fdf0db94040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T00:07:39.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:39 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2249827643' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-16T00:07:39.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:39 smithi033 ceph-mon[31243]: mgrmap e5: x(active, since 5s) 2024-09-16T00:07:40.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:39 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:39.680+0000 7fdf0db94040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T00:07:40.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:39 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:39.878+0000 7fdf0db94040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T00:07:40.432 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-09-16T00:07:40.432 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-16T00:07:40.432 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T00:07:40.432 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-16T00:07:40.432 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-16T00:07:40.433 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-09-16T00:07:40.433 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.607+0000 7f2750406640 1 Processor -- start 2024-09-16T00:07:40.433 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.607+0000 7f2750406640 1 -- start start 2024-09-16T00:07:40.433 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.607+0000 7f2750406640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 0x7f274810a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:40.433 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.607+0000 7f2750406640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f274807be50 con 0x7f27481080f0 2024-09-16T00:07:40.433 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.607+0000 7f274e1a2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 0x7f274810a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:40.433 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.607+0000 7f274e1a2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 0x7f274810a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55380/0 (socket says 172.21.15.33:55380) 2024-09-16T00:07:40.433 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.607+0000 7f274e1a2640 1 -- 172.21.15.33:0/3380649010 learned_addr learned my addr 172.21.15.33:0/3380649010 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:40.433 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.608+0000 7f274e1a2640 1 -- 172.21.15.33:0/3380649010 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274810aa20 con 0x7f27481080f0 2024-09-16T00:07:40.433 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.608+0000 7f274e1a2640 1 --2- 172.21.15.33:0/3380649010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 0x7f274810a4e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2744009820 tx=0x7f274402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed316c0ada96a19e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:40.433 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.608+0000 7f274d1a0640 1 -- 172.21.15.33:0/3380649010 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2744037070 con 0x7f27481080f0 2024-09-16T00:07:40.434 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.609+0000 7f274d1a0640 1 -- 172.21.15.33:0/3380649010 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2744032900 con 0x7f27481080f0 2024-09-16T00:07:40.434 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.609+0000 7f274d1a0640 1 -- 172.21.15.33:0/3380649010 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2744030850 con 0x7f27481080f0 2024-09-16T00:07:40.434 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.609+0000 7f2750406640 1 -- 172.21.15.33:0/3380649010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 msgr2=0x7f274810a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:40.434 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.609+0000 7f2750406640 1 --2- 172.21.15.33:0/3380649010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 0x7f274810a4e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2744009820 tx=0x7f274402a270 comp rx=0 tx=0).stop 2024-09-16T00:07:40.434 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.609+0000 7f2750406640 1 -- 172.21.15.33:0/3380649010 shutdown_connections 2024-09-16T00:07:40.434 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.609+0000 7f2750406640 1 --2- 172.21.15.33:0/3380649010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 0x7f274810a4e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:40.434 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.609+0000 7f2750406640 1 -- 172.21.15.33:0/3380649010 >> 172.21.15.33:0/3380649010 conn(0x7f2748101db0 msgr2=0x7f27481041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:40.434 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.609+0000 7f2750406640 1 -- 172.21.15.33:0/3380649010 shutdown_connections 2024-09-16T00:07:40.434 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.609+0000 7f2750406640 1 -- 172.21.15.33:0/3380649010 wait complete. 2024-09-16T00:07:40.434 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.610+0000 7f2750406640 1 Processor -- start 2024-09-16T00:07:40.434 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.610+0000 7f2750406640 1 -- start start 2024-09-16T00:07:40.434 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.610+0000 7f2750406640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 0x7f274819b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.610+0000 7f2750406640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f274810ba60 con 0x7f27481080f0 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.610+0000 7f274e1a2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 0x7f274819b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.610+0000 7f274e1a2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 0x7f274819b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55388/0 (socket says 172.21.15.33:55388) 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.610+0000 7f274e1a2640 1 -- 172.21.15.33:0/825766586 learned_addr learned my addr 172.21.15.33:0/825766586 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.610+0000 7f274e1a2640 1 -- 172.21.15.33:0/825766586 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274819bb60 con 0x7f27481080f0 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.611+0000 7f274e1a2640 1 --2- 172.21.15.33:0/825766586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 0x7f274819b620 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f27440097f0 tx=0x7f2744030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.611+0000 7f273f7fe640 1 -- 172.21.15.33:0/825766586 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2744037040 con 0x7f27481080f0 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.611+0000 7f273f7fe640 1 -- 172.21.15.33:0/825766586 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2744040070 con 0x7f27481080f0 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.611+0000 7f2750406640 1 -- 172.21.15.33:0/825766586 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f274819bdf0 con 0x7f27481080f0 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.611+0000 7f273f7fe640 1 -- 172.21.15.33:0/825766586 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f274403a390 con 0x7f27481080f0 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.611+0000 7f2750406640 1 -- 172.21.15.33:0/825766586 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f274819eae0 con 0x7f27481080f0 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.612+0000 7f273f7fe640 1 -- 172.21.15.33:0/825766586 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7f274403a5e0 con 0x7f27481080f0 2024-09-16T00:07:40.435 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.612+0000 7f2750406640 1 -- 172.21.15.33:0/825766586 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2748108550 con 0x7f27481080f0 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.612+0000 7f273f7fe640 1 --2- 172.21.15.33:0/825766586 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f2734039200 0x7f273403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.612+0000 7f273f7fe640 1 -- 172.21.15.33:0/825766586 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f274406c420 con 0x7f27481080f0 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.612+0000 7f274d9a1640 1 -- 172.21.15.33:0/825766586 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f2734039200 msgr2=0x7f273403b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/961233150 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.612+0000 7f274d9a1640 1 --2- 172.21.15.33:0/825766586 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f2734039200 0x7f273403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.617+0000 7f273f7fe640 1 -- 172.21.15.33:0/825766586 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f274406c180 con 0x7f27481080f0 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.812+0000 7f274d9a1640 1 -- 172.21.15.33:0/825766586 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f2734039200 msgr2=0x7f273403b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/961233150 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.812+0000 7f274d9a1640 1 --2- 172.21.15.33:0/825766586 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f2734039200 0x7f273403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.838+0000 7f2750406640 1 -- 172.21.15.33:0/825766586 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f274819f1d0 con 0x7f27481080f0 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.838+0000 7f273f7fe640 1 -- 172.21.15.33:0/825766586 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f2744043760 con 0x7f27481080f0 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.840+0000 7f2750406640 1 -- 172.21.15.33:0/825766586 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f2734039200 msgr2=0x7f273403b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.840+0000 7f2750406640 1 --2- 172.21.15.33:0/825766586 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f2734039200 0x7f273403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.840+0000 7f2750406640 1 -- 172.21.15.33:0/825766586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 msgr2=0x7f274819b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:40.436 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.840+0000 7f2750406640 1 --2- 172.21.15.33:0/825766586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 0x7f274819b620 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f27440097f0 tx=0x7f2744030d80 comp rx=0 tx=0).stop 2024-09-16T00:07:40.437 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.840+0000 7f2750406640 1 -- 172.21.15.33:0/825766586 shutdown_connections 2024-09-16T00:07:40.437 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.840+0000 7f2750406640 1 --2- 172.21.15.33:0/825766586 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7f2734039200 0x7f273403b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:40.437 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.840+0000 7f2750406640 1 --2- 172.21.15.33:0/825766586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27481080f0 0x7f274819b620 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:40.437 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.840+0000 7f2750406640 1 -- 172.21.15.33:0/825766586 >> 172.21.15.33:0/825766586 conn(0x7f2748101db0 msgr2=0x7f2748102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:40.437 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.840+0000 7f2750406640 1 -- 172.21.15.33:0/825766586 shutdown_connections 2024-09-16T00:07:40.437 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:39.840+0000 7f2750406640 1 -- 172.21.15.33:0/825766586 wait complete. 2024-09-16T00:07:40.437 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for the mgr to restart... 2024-09-16T00:07:40.437 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mgr epoch 5... 2024-09-16T00:07:40.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:40 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/825766586' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-16T00:07:40.881 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:40 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:40.817+0000 7fdf0db94040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T00:07:41.330 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:40 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:40.891+0000 7fdf0db94040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T00:07:41.757 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:41 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:41.635+0000 7fdf0db94040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T00:07:42.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:41 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:41.756+0000 7fdf0db94040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T00:07:42.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:41 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:41.905+0000 7fdf0db94040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T00:07:42.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:41 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:41.968+0000 7fdf0db94040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T00:07:42.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:42.125+0000 7fdf0db94040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T00:07:42.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:42.196+0000 7fdf0db94040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T00:07:42.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:42.258+0000 7fdf0db94040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T00:07:42.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:42.424+0000 7fdf0db94040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T00:07:42.711 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:42.492+0000 7fdf0db94040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T00:07:42.712 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:42.561+0000 7fdf0db94040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T00:07:42.712 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:42.648+0000 7fdf0db94040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T00:07:42.712 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:42.709+0000 7fdf0db94040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T00:07:43.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:42.771+0000 7fdf0db94040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T00:07:44.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:43.597+0000 7fdf0db94040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T00:07:44.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: Active manager daemon x restarted 2024-09-16T00:07:44.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: Activating manager daemon x 2024-09-16T00:07:44.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: osdmap e2: 0 total, 0 up, 0 in 2024-09-16T00:07:44.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: mgrmap e6: x(active, starting, since 0.00312508s) 2024-09-16T00:07:44.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:07:44.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-16T00:07:44.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-16T00:07:44.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-16T00:07:44.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-16T00:07:44.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: Manager daemon x is now available 2024-09-16T00:07:44.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' 2024-09-16T00:07:44.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' 2024-09-16T00:07:44.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:07:44.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:07:44.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-16T00:07:44.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:43 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-16T00:07:45.146 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-09-16T00:07:45.147 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-16T00:07:45.147 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-16T00:07:45.147 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-09-16T00:07:45.147 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.880+0000 7fecaabfd640 1 Processor -- start 2024-09-16T00:07:45.147 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.880+0000 7fecaabfd640 1 -- start start 2024-09-16T00:07:45.147 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.880+0000 7fecaabfd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 0x7feca410a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:45.147 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.880+0000 7fecaabfd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7feca407be60 con 0x7feca4108080 2024-09-16T00:07:45.147 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.880+0000 7feca8999640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 0x7feca410a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:45.147 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.880+0000 7feca8999640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 0x7feca410a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37632/0 (socket says 172.21.15.33:37632) 2024-09-16T00:07:45.147 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.880+0000 7feca8999640 1 -- 172.21.15.33:0/3527604278 learned_addr learned my addr 172.21.15.33:0/3527604278 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:45.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.880+0000 7feca8999640 1 -- 172.21.15.33:0/3527604278 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca410a9b0 con 0x7feca4108080 2024-09-16T00:07:45.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.881+0000 7feca8999640 1 --2- 172.21.15.33:0/3527604278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 0x7feca410a470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7feca0009820 tx=0x7feca002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f32b24975e59496 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:45.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.881+0000 7fec9b7fe640 1 -- 172.21.15.33:0/3527604278 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feca0037070 con 0x7feca4108080 2024-09-16T00:07:45.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.881+0000 7fec9b7fe640 1 -- 172.21.15.33:0/3527604278 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7feca0032900 con 0x7feca4108080 2024-09-16T00:07:45.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.881+0000 7fec9b7fe640 1 -- 172.21.15.33:0/3527604278 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feca0030850 con 0x7feca4108080 2024-09-16T00:07:45.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.882+0000 7fecaabfd640 1 -- 172.21.15.33:0/3527604278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 msgr2=0x7feca410a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:45.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.882+0000 7fecaabfd640 1 --2- 172.21.15.33:0/3527604278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 0x7feca410a470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7feca0009820 tx=0x7feca002a270 comp rx=0 tx=0).stop 2024-09-16T00:07:45.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.882+0000 7fecaabfd640 1 -- 172.21.15.33:0/3527604278 shutdown_connections 2024-09-16T00:07:45.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.882+0000 7fecaabfd640 1 --2- 172.21.15.33:0/3527604278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 0x7feca410a470 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:45.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.882+0000 7fecaabfd640 1 -- 172.21.15.33:0/3527604278 >> 172.21.15.33:0/3527604278 conn(0x7feca4101ca0 msgr2=0x7feca41040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:45.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.882+0000 7fecaabfd640 1 -- 172.21.15.33:0/3527604278 shutdown_connections 2024-09-16T00:07:45.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.882+0000 7fecaabfd640 1 -- 172.21.15.33:0/3527604278 wait complete. 2024-09-16T00:07:45.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.883+0000 7fecaabfd640 1 Processor -- start 2024-09-16T00:07:45.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.883+0000 7fecaabfd640 1 -- start start 2024-09-16T00:07:45.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.883+0000 7fecaabfd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 0x7feca419b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:45.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.883+0000 7fecaabfd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7feca410b9f0 con 0x7feca4108080 2024-09-16T00:07:45.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.883+0000 7feca8999640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 0x7feca419b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:45.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.883+0000 7feca8999640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 0x7feca419b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37634/0 (socket says 172.21.15.33:37634) 2024-09-16T00:07:45.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.883+0000 7feca8999640 1 -- 172.21.15.33:0/4041352122 learned_addr learned my addr 172.21.15.33:0/4041352122 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:45.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.883+0000 7feca8999640 1 -- 172.21.15.33:0/4041352122 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca419baf0 con 0x7feca4108080 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.883+0000 7feca8999640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 0x7feca419b5b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feca0032ce0 tx=0x7feca0032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.883+0000 7fec99ffb640 1 -- 172.21.15.33:0/4041352122 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feca0042070 con 0x7feca4108080 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.884+0000 7fec99ffb640 1 -- 172.21.15.33:0/4041352122 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7feca00308c0 con 0x7feca4108080 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.884+0000 7fec99ffb640 1 -- 172.21.15.33:0/4041352122 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feca0037040 con 0x7feca4108080 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.884+0000 7fecaabfd640 1 -- 172.21.15.33:0/4041352122 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca419bd80 con 0x7feca4108080 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.884+0000 7fecaabfd640 1 -- 172.21.15.33:0/4041352122 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca419ea70 con 0x7feca4108080 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.885+0000 7fec99ffb640 1 -- 172.21.15.33:0/4041352122 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7feca0045070 con 0x7feca4108080 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.885+0000 7fec99ffb640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fec8c039200 0x7fec8c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.885+0000 7fec99ffb640 1 -- 172.21.15.33:0/4041352122 --> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fec8c03bdd0 con 0x7fec8c039200 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.885+0000 7fec99ffb640 1 -- 172.21.15.33:0/4041352122 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7feca006c1f0 con 0x7feca4108080 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.885+0000 7fec9bfff640 1 -- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fec8c039200 msgr2=0x7fec8c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/961233150 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:40.885+0000 7fec9bfff640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fec8c039200 0x7fec8c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T00:07:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:41.086+0000 7fec9bfff640 1 -- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fec8c039200 msgr2=0x7fec8c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/961233150 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:41.086+0000 7fec9bfff640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fec8c039200 0x7fec8c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:41.487+0000 7fec9bfff640 1 -- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fec8c039200 msgr2=0x7fec8c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/961233150 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:41.487+0000 7fec9bfff640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fec8c039200 0x7fec8c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:42.288+0000 7fec9bfff640 1 -- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fec8c039200 msgr2=0x7fec8c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/961233150 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:42.288+0000 7fec9bfff640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fec8c039200 0x7fec8c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:43.601+0000 7fec99ffb640 1 -- 172.21.15.33:0/4041352122 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mgrmap(e 6) ==== 46228+0+0 (secure 0 0 0) 0x7feca006ad70 con 0x7feca4108080 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:43.601+0000 7fec99ffb640 1 -- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fec8c039200 msgr2=0x7fec8c03b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:43.601+0000 7fec99ffb640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fec8c039200 0x7fec8c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.604+0000 7fec99ffb640 1 -- 172.21.15.33:0/4041352122 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 7) ==== 46355+0+0 (secure 0 0 0) 0x7feca003c780 con 0x7feca4108080 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.604+0000 7fec99ffb640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fec8c03cb30 0x7fec8c03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.604+0000 7fec99ffb640 1 -- 172.21.15.33:0/4041352122 --> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7feca002bba0 con 0x7fec8c03cb30 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.605+0000 7fec9bfff640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fec8c03cb30 0x7fec8c03ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.605+0000 7fec9bfff640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fec8c03cb30 0x7fec8c03ef20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fec94003980 tx=0x7fec940092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:45.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.606+0000 7fec99ffb640 1 -- 172.21.15.33:0/4041352122 <== mgr.14118 v2:172.21.15.33:6800/122116489 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7feca002bba0 con 0x7fec8c03cb30 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.613+0000 7fecaabfd640 1 -- 172.21.15.33:0/4041352122 --> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7feca41084e0 con 0x7fec8c03cb30 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.613+0000 7fec99ffb640 1 -- 172.21.15.33:0/4041352122 <== mgr.14118 v2:172.21.15.33:6800/122116489 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7feca41084e0 con 0x7fec8c03cb30 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.613+0000 7fecaabfd640 1 -- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fec8c03cb30 msgr2=0x7fec8c03ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.613+0000 7fecaabfd640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fec8c03cb30 0x7fec8c03ef20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fec94003980 tx=0x7fec940092b0 comp rx=0 tx=0).stop 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.613+0000 7fecaabfd640 1 -- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 msgr2=0x7feca419b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.613+0000 7fecaabfd640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 0x7feca419b5b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feca0032ce0 tx=0x7feca0032d10 comp rx=0 tx=0).stop 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.614+0000 7fecaabfd640 1 -- 172.21.15.33:0/4041352122 shutdown_connections 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.614+0000 7fecaabfd640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fec8c03cb30 0x7fec8c03ef20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.614+0000 7fecaabfd640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:6800/961233150,v1:172.21.15.33:6801/961233150] conn(0x7fec8c039200 0x7fec8c03b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.614+0000 7fecaabfd640 1 --2- 172.21.15.33:0/4041352122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4108080 0x7feca419b5b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.614+0000 7fecaabfd640 1 -- 172.21.15.33:0/4041352122 >> 172.21.15.33:0/4041352122 conn(0x7feca4101ca0 msgr2=0x7feca4102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.614+0000 7fecaabfd640 1 -- 172.21.15.33:0/4041352122 shutdown_connections 2024-09-16T00:07:45.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:44.614+0000 7fecaabfd640 1 -- 172.21.15.33:0/4041352122 wait complete. 2024-09-16T00:07:45.153 INFO:teuthology.orchestra.run.smithi033.stdout:mgr epoch 5 is available 2024-09-16T00:07:45.153 INFO:teuthology.orchestra.run.smithi033.stdout:Setting orchestrator backend to cephadm... 2024-09-16T00:07:45.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:45 smithi033 ceph-mon[31243]: Found migration_current of "None". Setting to last migration. 2024-09-16T00:07:45.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:45 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' 2024-09-16T00:07:45.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:45 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' 2024-09-16T00:07:45.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:45 smithi033 ceph-mon[31243]: mgrmap e7: x(active, since 1.00756s) 2024-09-16T00:07:46.361 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-mon[31243]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-16T00:07:46.362 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-mon[31243]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-16T00:07:46.362 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' 2024-09-16T00:07:46.362 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:07:46.362 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:07:46.362 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:46.078+0000 7fde7a6b9640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:00:07:46] ENGINE Error in HTTPServer.serve 2024-09-16T00:07:46.362 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: Traceback (most recent call last): 2024-09-16T00:07:46.362 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T00:07:46.362 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: self._connections.run(self.expiration_interval) 2024-09-16T00:07:46.363 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T00:07:46.363 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: self._run(expiration_interval) 2024-09-16T00:07:46.363 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T00:07:46.363 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T00:07:46.363 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T00:07:46.363 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T00:07:46.363 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T00:07:46.363 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: s = self.context.wrap_socket( 2024-09-16T00:07:46.363 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T00:07:46.363 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: return self.sslsocket_class._create( 2024-09-16T00:07:46.363 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T00:07:46.364 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: self.do_handshake() 2024-09-16T00:07:46.364 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T00:07:46.364 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: self._sslobj.do_handshake() 2024-09-16T00:07:46.364 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T00:07:46.364 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:07:46.369 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.552+0000 7ff201828640 1 Processor -- start 2024-09-16T00:07:46.370 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.553+0000 7ff201828640 1 -- start start 2024-09-16T00:07:46.370 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.553+0000 7ff201828640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 0x7ff1fc107b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:46.370 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.553+0000 7ff201828640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc108110 con 0x7ff1fc107740 2024-09-16T00:07:46.370 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.553+0000 7ff1faffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 0x7ff1fc107b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:46.370 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.553+0000 7ff1faffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 0x7ff1fc107b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37708/0 (socket says 172.21.15.33:37708) 2024-09-16T00:07:46.370 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.553+0000 7ff1faffd640 1 -- 172.21.15.33:0/3329371032 learned_addr learned my addr 172.21.15.33:0/3329371032 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:46.370 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.553+0000 7ff1faffd640 1 -- 172.21.15.33:0/3329371032 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1fc1088a0 con 0x7ff1fc107740 2024-09-16T00:07:46.370 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.554+0000 7ff1faffd640 1 --2- 172.21.15.33:0/3329371032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 0x7ff1fc107b40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec009820 tx=0x7ff1ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3af0567e05e3b81 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:46.370 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.554+0000 7ff1f9ffb640 1 -- 172.21.15.33:0/3329371032 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1ec037070 con 0x7ff1fc107740 2024-09-16T00:07:46.370 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.554+0000 7ff1f9ffb640 1 -- 172.21.15.33:0/3329371032 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff1ec032900 con 0x7ff1fc107740 2024-09-16T00:07:46.371 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.554+0000 7ff1f9ffb640 1 -- 172.21.15.33:0/3329371032 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1ec030850 con 0x7ff1fc107740 2024-09-16T00:07:46.371 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.555+0000 7ff201828640 1 -- 172.21.15.33:0/3329371032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 msgr2=0x7ff1fc107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:46.371 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.555+0000 7ff201828640 1 --2- 172.21.15.33:0/3329371032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 0x7ff1fc107b40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec009820 tx=0x7ff1ec02a270 comp rx=0 tx=0).stop 2024-09-16T00:07:46.371 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.555+0000 7ff201828640 1 -- 172.21.15.33:0/3329371032 shutdown_connections 2024-09-16T00:07:46.371 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.555+0000 7ff201828640 1 --2- 172.21.15.33:0/3329371032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 0x7ff1fc107b40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:46.371 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.555+0000 7ff201828640 1 -- 172.21.15.33:0/3329371032 >> 172.21.15.33:0/3329371032 conn(0x7ff1fc102eb0 msgr2=0x7ff1fc105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:46.371 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.555+0000 7ff201828640 1 -- 172.21.15.33:0/3329371032 shutdown_connections 2024-09-16T00:07:46.371 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.555+0000 7ff201828640 1 -- 172.21.15.33:0/3329371032 wait complete. 2024-09-16T00:07:46.371 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.555+0000 7ff201828640 1 Processor -- start 2024-09-16T00:07:46.371 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.555+0000 7ff201828640 1 -- start start 2024-09-16T00:07:46.371 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.556+0000 7ff201828640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 0x7ff1fc19b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.556+0000 7ff201828640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc10bc50 con 0x7ff1fc107740 2024-09-16T00:07:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.556+0000 7ff1faffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 0x7ff1fc19b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.556+0000 7ff1faffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 0x7ff1fc19b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37716/0 (socket says 172.21.15.33:37716) 2024-09-16T00:07:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.556+0000 7ff1faffd640 1 -- 172.21.15.33:0/664371715 learned_addr learned my addr 172.21.15.33:0/664371715 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.556+0000 7ff1faffd640 1 -- 172.21.15.33:0/664371715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1fc19bbc0 con 0x7ff1fc107740 2024-09-16T00:07:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.556+0000 7ff1faffd640 1 --2- 172.21.15.33:0/664371715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 0x7ff1fc19b680 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec009940 tx=0x7ff1ec0326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.556+0000 7ff200826640 1 -- 172.21.15.33:0/664371715 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1ec042070 con 0x7ff1fc107740 2024-09-16T00:07:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.556+0000 7ff200826640 1 -- 172.21.15.33:0/664371715 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff1ec032be0 con 0x7ff1fc107740 2024-09-16T00:07:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.556+0000 7ff201828640 1 -- 172.21.15.33:0/664371715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1fc19be50 con 0x7ff1fc107740 2024-09-16T00:07:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.556+0000 7ff200826640 1 -- 172.21.15.33:0/664371715 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1ec037040 con 0x7ff1fc107740 2024-09-16T00:07:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.556+0000 7ff201828640 1 -- 172.21.15.33:0/664371715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1fc19eb40 con 0x7ff1fc107740 2024-09-16T00:07:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.558+0000 7ff200826640 1 -- 172.21.15.33:0/664371715 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 7) ==== 46355+0+0 (secure 0 0 0) 0x7ff1ec045070 con 0x7ff1fc107740 2024-09-16T00:07:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.558+0000 7ff201828640 1 -- 172.21.15.33:0/664371715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1fc107bc0 con 0x7ff1fc107740 2024-09-16T00:07:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.558+0000 7ff200826640 1 --2- 172.21.15.33:0/664371715 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7ff1e40390f0 0x7ff1e403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.558+0000 7ff200826640 1 -- 172.21.15.33:0/664371715 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff1ec06c140 con 0x7ff1fc107740 2024-09-16T00:07:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.558+0000 7ff1fa7fc640 1 --2- 172.21.15.33:0/664371715 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7ff1e40390f0 0x7ff1e403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.558+0000 7ff1fa7fc640 1 --2- 172.21.15.33:0/664371715 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7ff1e40390f0 0x7ff1e403b5b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00098c0 tx=0x7ff1f0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.563+0000 7ff200826640 1 -- 172.21.15.33:0/664371715 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff1ec032e00 con 0x7ff1fc107740 2024-09-16T00:07:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.755+0000 7ff201828640 1 -- 172.21.15.33:0/664371715 --> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7ff1fc1041c0 con 0x7ff1e40390f0 2024-09-16T00:07:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.760+0000 7ff200826640 1 -- 172.21.15.33:0/664371715 <== mgr.14118 v2:172.21.15.33:6800/122116489 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff1fc1041c0 con 0x7ff1e40390f0 2024-09-16T00:07:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.763+0000 7ff201828640 1 -- 172.21.15.33:0/664371715 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7ff1e40390f0 msgr2=0x7ff1e403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.763+0000 7ff201828640 1 --2- 172.21.15.33:0/664371715 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7ff1e40390f0 0x7ff1e403b5b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00098c0 tx=0x7ff1f0006db0 comp rx=0 tx=0).stop 2024-09-16T00:07:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.763+0000 7ff201828640 1 -- 172.21.15.33:0/664371715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 msgr2=0x7ff1fc19b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:46.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.763+0000 7ff201828640 1 --2- 172.21.15.33:0/664371715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 0x7ff1fc19b680 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec009940 tx=0x7ff1ec0326e0 comp rx=0 tx=0).stop 2024-09-16T00:07:46.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.763+0000 7ff201828640 1 -- 172.21.15.33:0/664371715 shutdown_connections 2024-09-16T00:07:46.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.763+0000 7ff201828640 1 --2- 172.21.15.33:0/664371715 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7ff1e40390f0 0x7ff1e403b5b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:46.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.763+0000 7ff201828640 1 --2- 172.21.15.33:0/664371715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1fc107740 0x7ff1fc19b680 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:46.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.763+0000 7ff201828640 1 -- 172.21.15.33:0/664371715 >> 172.21.15.33:0/664371715 conn(0x7ff1fc102eb0 msgr2=0x7ff1fc103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:46.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.763+0000 7ff201828640 1 -- 172.21.15.33:0/664371715 shutdown_connections 2024-09-16T00:07:46.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:45.763+0000 7ff201828640 1 -- 172.21.15.33:0/664371715 wait complete. 2024-09-16T00:07:47.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:47 smithi033 ceph-mon[31243]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:07:47.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:47 smithi033 ceph-mon[31243]: [16/Sep/2024:00:07:45] ENGINE Bus STARTING 2024-09-16T00:07:47.332 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:47 smithi033 ceph-mon[31243]: [16/Sep/2024:00:07:46] ENGINE Serving on https://172.21.15.33:7150 2024-09-16T00:07:47.332 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:47 smithi033 ceph-mon[31243]: [16/Sep/2024:00:07:46] ENGINE Error in HTTPServer.serve 2024-09-16T00:07:47.332 INFO:journalctl@ceph.mon.a.smithi033.stdout: Traceback (most recent call last): 2024-09-16T00:07:47.332 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T00:07:47.332 INFO:journalctl@ceph.mon.a.smithi033.stdout: self._connections.run(self.expiration_interval) 2024-09-16T00:07:47.332 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T00:07:47.332 INFO:journalctl@ceph.mon.a.smithi033.stdout: self._run(expiration_interval) 2024-09-16T00:07:47.332 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T00:07:47.332 INFO:journalctl@ceph.mon.a.smithi033.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T00:07:47.332 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T00:07:47.333 INFO:journalctl@ceph.mon.a.smithi033.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T00:07:47.333 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T00:07:47.333 INFO:journalctl@ceph.mon.a.smithi033.stdout: s = self.context.wrap_socket( 2024-09-16T00:07:47.333 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T00:07:47.333 INFO:journalctl@ceph.mon.a.smithi033.stdout: return self.sslsocket_class._create( 2024-09-16T00:07:47.333 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T00:07:47.333 INFO:journalctl@ceph.mon.a.smithi033.stdout: self.do_handshake() 2024-09-16T00:07:47.333 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T00:07:47.333 INFO:journalctl@ceph.mon.a.smithi033.stdout: self._sslobj.do_handshake() 2024-09-16T00:07:47.333 INFO:journalctl@ceph.mon.a.smithi033.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T00:07:47.333 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:47 smithi033 ceph-mon[31243]: [16/Sep/2024:00:07:46] ENGINE Serving on http://172.21.15.33:8765 2024-09-16T00:07:47.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:47 smithi033 ceph-mon[31243]: [16/Sep/2024:00:07:46] ENGINE Bus STARTED 2024-09-16T00:07:47.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:47 smithi033 ceph-mon[31243]: mgrmap e8: x(active, since 2s) 2024-09-16T00:07:47.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-16T00:07:47.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.856+0000 7fb34ddc2640 1 Processor -- start 2024-09-16T00:07:47.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.856+0000 7fb34ddc2640 1 -- start start 2024-09-16T00:07:47.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.856+0000 7fb34ddc2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 0x7fb348107fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:47.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.856+0000 7fb34ddc2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb3481085b0 con 0x7fb348107be0 2024-09-16T00:07:47.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.856+0000 7fb3477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 0x7fb348107fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:47.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.856+0000 7fb3477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 0x7fb348107fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37724/0 (socket says 172.21.15.33:37724) 2024-09-16T00:07:47.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.856+0000 7fb3477fe640 1 -- 172.21.15.33:0/553105302 learned_addr learned my addr 172.21.15.33:0/553105302 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.856+0000 7fb3477fe640 1 -- 172.21.15.33:0/553105302 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3481067b0 con 0x7fb348107be0 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.857+0000 7fb3477fe640 1 --2- 172.21.15.33:0/553105302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 0x7fb348107fe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb338009820 tx=0x7fb33802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb54c63187ef6c27 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.857+0000 7fb3467fc640 1 -- 172.21.15.33:0/553105302 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb338037070 con 0x7fb348107be0 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.857+0000 7fb3467fc640 1 -- 172.21.15.33:0/553105302 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb338032900 con 0x7fb348107be0 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.857+0000 7fb3467fc640 1 -- 172.21.15.33:0/553105302 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb338030850 con 0x7fb348107be0 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.858+0000 7fb34ddc2640 1 -- 172.21.15.33:0/553105302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 msgr2=0x7fb348107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.858+0000 7fb34ddc2640 1 --2- 172.21.15.33:0/553105302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 0x7fb348107fe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb338009820 tx=0x7fb33802a270 comp rx=0 tx=0).stop 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.858+0000 7fb34ddc2640 1 -- 172.21.15.33:0/553105302 shutdown_connections 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.858+0000 7fb34ddc2640 1 --2- 172.21.15.33:0/553105302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 0x7fb348107fe0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.858+0000 7fb34ddc2640 1 -- 172.21.15.33:0/553105302 >> 172.21.15.33:0/553105302 conn(0x7fb348101490 msgr2=0x7fb3481038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.858+0000 7fb34ddc2640 1 -- 172.21.15.33:0/553105302 shutdown_connections 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.858+0000 7fb34ddc2640 1 -- 172.21.15.33:0/553105302 wait complete. 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.859+0000 7fb34ddc2640 1 Processor -- start 2024-09-16T00:07:47.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.859+0000 7fb34ddc2640 1 -- start start 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.859+0000 7fb34ddc2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 0x7fb3481918e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.859+0000 7fb34ddc2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb34810ab70 con 0x7fb348107be0 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.859+0000 7fb3477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 0x7fb3481918e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.859+0000 7fb3477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 0x7fb3481918e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37728/0 (socket says 172.21.15.33:37728) 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.859+0000 7fb3477fe640 1 -- 172.21.15.33:0/909597556 learned_addr learned my addr 172.21.15.33:0/909597556 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.859+0000 7fb3477fe640 1 -- 172.21.15.33:0/909597556 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb348191e20 con 0x7fb348107be0 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.860+0000 7fb3477fe640 1 --2- 172.21.15.33:0/909597556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 0x7fb3481918e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb338030460 tx=0x7fb338030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.860+0000 7fb344ff9640 1 -- 172.21.15.33:0/909597556 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb338042070 con 0x7fb348107be0 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.860+0000 7fb344ff9640 1 -- 172.21.15.33:0/909597556 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb338032be0 con 0x7fb348107be0 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.860+0000 7fb34ddc2640 1 -- 172.21.15.33:0/909597556 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb348190010 con 0x7fb348107be0 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.860+0000 7fb344ff9640 1 -- 172.21.15.33:0/909597556 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb338037040 con 0x7fb348107be0 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.860+0000 7fb34ddc2640 1 -- 172.21.15.33:0/909597556 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb348190550 con 0x7fb348107be0 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.861+0000 7fb344ff9640 1 -- 172.21.15.33:0/909597556 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fb338045070 con 0x7fb348107be0 2024-09-16T00:07:47.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.861+0000 7fb344ff9640 1 --2- 172.21.15.33:0/909597556 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fb3300391b0 0x7fb33003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.861+0000 7fb34ddc2640 1 -- 172.21.15.33:0/909597556 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb34810acf0 con 0x7fb348107be0 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.861+0000 7fb344ff9640 1 -- 172.21.15.33:0/909597556 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb33806c190 con 0x7fb348107be0 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.861+0000 7fb346ffd640 1 --2- 172.21.15.33:0/909597556 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fb3300391b0 0x7fb33003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.862+0000 7fb346ffd640 1 --2- 172.21.15.33:0/909597556 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fb3300391b0 0x7fb33003b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0098c0 tx=0x7fb33c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:46.866+0000 7fb344ff9640 1 -- 172.21.15.33:0/909597556 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb338045360 con 0x7fb348107be0 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:47.042+0000 7fb34ddc2640 1 -- 172.21.15.33:0/909597556 --> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fb3481956d0 con 0x7fb3300391b0 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:47.043+0000 7fb344ff9640 1 -- 172.21.15.33:0/909597556 <== mgr.14118 v2:172.21.15.33:6800/122116489 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fb3481956d0 con 0x7fb3300391b0 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:47.045+0000 7fb34ddc2640 1 -- 172.21.15.33:0/909597556 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fb3300391b0 msgr2=0x7fb33003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:47.045+0000 7fb34ddc2640 1 --2- 172.21.15.33:0/909597556 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fb3300391b0 0x7fb33003b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0098c0 tx=0x7fb33c006db0 comp rx=0 tx=0).stop 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:47.045+0000 7fb34ddc2640 1 -- 172.21.15.33:0/909597556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 msgr2=0x7fb3481918e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:47.045+0000 7fb34ddc2640 1 --2- 172.21.15.33:0/909597556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 0x7fb3481918e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb338030460 tx=0x7fb338030d80 comp rx=0 tx=0).stop 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:47.045+0000 7fb34ddc2640 1 -- 172.21.15.33:0/909597556 shutdown_connections 2024-09-16T00:07:47.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:47.045+0000 7fb34ddc2640 1 --2- 172.21.15.33:0/909597556 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fb3300391b0 0x7fb33003b670 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:47.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:47.045+0000 7fb34ddc2640 1 --2- 172.21.15.33:0/909597556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb348107be0 0x7fb3481918e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:47.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:47.045+0000 7fb34ddc2640 1 -- 172.21.15.33:0/909597556 >> 172.21.15.33:0/909597556 conn(0x7fb348101490 msgr2=0x7fb3481023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:47.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:47.045+0000 7fb34ddc2640 1 -- 172.21.15.33:0/909597556 shutdown_connections 2024-09-16T00:07:47.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:47.045+0000 7fb34ddc2640 1 -- 172.21.15.33:0/909597556 wait complete. 2024-09-16T00:07:47.621 INFO:teuthology.orchestra.run.smithi033.stdout:Generating ssh key... 2024-09-16T00:07:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-mon[31243]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:07:49.197 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: Generating public/private rsa key pair. 2024-09-16T00:07:49.198 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: Your identification has been saved in /tmp/tmp1yuj1han/key 2024-09-16T00:07:49.198 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: Your public key has been saved in /tmp/tmp1yuj1han/key.pub 2024-09-16T00:07:49.198 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: The key fingerprint is: 2024-09-16T00:07:49.198 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: SHA256:FR+v3gY512WNuHopLSpTJW4IGmnj7aJfNTnHQyu/sas ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:07:49.198 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: The key's randomart image is: 2024-09-16T00:07:49.198 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: +---[RSA 3072]----+ 2024-09-16T00:07:49.198 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: | . . | 2024-09-16T00:07:49.198 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: | o + ..| 2024-09-16T00:07:49.198 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: | . . . o o +| 2024-09-16T00:07:49.199 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: | = . +.o. + o.| 2024-09-16T00:07:49.199 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: | o = .*oSo * . .| 2024-09-16T00:07:49.199 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: | o ...*+. + * | 2024-09-16T00:07:49.199 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: | .. oo + = o | 2024-09-16T00:07:49.199 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: | ... o = + . | 2024-09-16T00:07:49.199 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: |.o.. E+=. | 2024-09-16T00:07:49.199 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:07:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: +----[SHA256]-----+ 2024-09-16T00:07:49.551 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:49 smithi033 ceph-mon[31243]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:07:49.551 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:49 smithi033 ceph-mon[31243]: Generating ssh key... 2024-09-16T00:07:49.552 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:49 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' 2024-09-16T00:07:49.552 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:49 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' 2024-09-16T00:07:49.560 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.001+0000 7f3a12893640 1 Processor -- start 2024-09-16T00:07:49.560 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.001+0000 7f3a12893640 1 -- start start 2024-09-16T00:07:49.560 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.001+0000 7f3a12893640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 0x7f3a0c105680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:49.561 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.001+0000 7f3a12893640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0c105c50 con 0x7f3a0c105280 2024-09-16T00:07:49.561 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.001+0000 7f3a0bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 0x7f3a0c105680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:49.561 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.001+0000 7f3a0bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 0x7f3a0c105680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37744/0 (socket says 172.21.15.33:37744) 2024-09-16T00:07:49.561 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.001+0000 7f3a0bfff640 1 -- 172.21.15.33:0/3831191271 learned_addr learned my addr 172.21.15.33:0/3831191271 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:49.561 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.001+0000 7f3a0bfff640 1 -- 172.21.15.33:0/3831191271 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0c105dd0 con 0x7f3a0c105280 2024-09-16T00:07:49.561 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.002+0000 7f3a0bfff640 1 --2- 172.21.15.33:0/3831191271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 0x7f3a0c105680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009820 tx=0x7f39fc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd649aabda7a1ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:49.561 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.002+0000 7f3a0affd640 1 -- 172.21.15.33:0/3831191271 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39fc037070 con 0x7f3a0c105280 2024-09-16T00:07:49.561 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.003+0000 7f3a0affd640 1 -- 172.21.15.33:0/3831191271 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f39fc032900 con 0x7f3a0c105280 2024-09-16T00:07:49.561 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.003+0000 7f3a0affd640 1 -- 172.21.15.33:0/3831191271 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39fc030850 con 0x7f3a0c105280 2024-09-16T00:07:49.561 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.003+0000 7f3a12893640 1 -- 172.21.15.33:0/3831191271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 msgr2=0x7f3a0c105680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:49.561 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.003+0000 7f3a12893640 1 --2- 172.21.15.33:0/3831191271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 0x7f3a0c105680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009820 tx=0x7f39fc02a270 comp rx=0 tx=0).stop 2024-09-16T00:07:49.561 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.003+0000 7f3a12893640 1 -- 172.21.15.33:0/3831191271 shutdown_connections 2024-09-16T00:07:49.562 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.003+0000 7f3a12893640 1 --2- 172.21.15.33:0/3831191271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 0x7f3a0c105680 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:49.562 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.003+0000 7f3a12893640 1 -- 172.21.15.33:0/3831191271 >> 172.21.15.33:0/3831191271 conn(0x7f3a0c100fa0 msgr2=0x7f3a0c103420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:49.562 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.003+0000 7f3a12893640 1 -- 172.21.15.33:0/3831191271 shutdown_connections 2024-09-16T00:07:49.562 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.003+0000 7f3a12893640 1 -- 172.21.15.33:0/3831191271 wait complete. 2024-09-16T00:07:49.562 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.004+0000 7f3a12893640 1 Processor -- start 2024-09-16T00:07:49.562 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.004+0000 7f3a12893640 1 -- start start 2024-09-16T00:07:49.562 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.004+0000 7f3a12893640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 0x7f3a0c19b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:49.563 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.004+0000 7f3a12893640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0c10ba20 con 0x7f3a0c105280 2024-09-16T00:07:49.563 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.004+0000 7f3a0bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 0x7f3a0c19b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:49.563 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.004+0000 7f3a0bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 0x7f3a0c19b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37754/0 (socket says 172.21.15.33:37754) 2024-09-16T00:07:49.563 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.004+0000 7f3a0bfff640 1 -- 172.21.15.33:0/2964974906 learned_addr learned my addr 172.21.15.33:0/2964974906 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:49.563 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.005+0000 7f3a0bfff640 1 -- 172.21.15.33:0/2964974906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0c19bbc0 con 0x7f3a0c105280 2024-09-16T00:07:49.563 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.005+0000 7f3a0bfff640 1 --2- 172.21.15.33:0/2964974906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 0x7f3a0c19b680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0097f0 tx=0x7f39fc030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:49.563 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.005+0000 7f3a097fa640 1 -- 172.21.15.33:0/2964974906 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39fc042070 con 0x7f3a0c105280 2024-09-16T00:07:49.563 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.005+0000 7f3a097fa640 1 -- 172.21.15.33:0/2964974906 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f39fc032ca0 con 0x7f3a0c105280 2024-09-16T00:07:49.563 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.005+0000 7f3a097fa640 1 -- 172.21.15.33:0/2964974906 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39fc037040 con 0x7f3a0c105280 2024-09-16T00:07:49.563 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.005+0000 7f3a12893640 1 -- 172.21.15.33:0/2964974906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a0c19be50 con 0x7f3a0c105280 2024-09-16T00:07:49.563 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.005+0000 7f3a12893640 1 -- 172.21.15.33:0/2964974906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a0c19eb40 con 0x7f3a0c105280 2024-09-16T00:07:49.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.006+0000 7f3a097fa640 1 -- 172.21.15.33:0/2964974906 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f39fc045070 con 0x7f3a0c105280 2024-09-16T00:07:49.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.006+0000 7f3a097fa640 1 --2- 172.21.15.33:0/2964974906 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f39f4039200 0x7f39f403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:49.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.006+0000 7f3a12893640 1 -- 172.21.15.33:0/2964974906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a0c105700 con 0x7f3a0c105280 2024-09-16T00:07:49.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.006+0000 7f3a097fa640 1 -- 172.21.15.33:0/2964974906 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f39fc06c1f0 con 0x7f3a0c105280 2024-09-16T00:07:49.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.006+0000 7f3a0b7fe640 1 --2- 172.21.15.33:0/2964974906 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f39f4039200 0x7f39f403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:49.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.007+0000 7f3a0b7fe640 1 --2- 172.21.15.33:0/2964974906 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f39f4039200 0x7f39f403b6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3a000098c0 tx=0x7f3a00006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:49.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.011+0000 7f3a097fa640 1 -- 172.21.15.33:0/2964974906 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f39fc045360 con 0x7f3a0c105280 2024-09-16T00:07:49.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.190+0000 7f3a12893640 1 -- 172.21.15.33:0/2964974906 --> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f3a0c1022f0 con 0x7f39f4039200 2024-09-16T00:07:49.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.939+0000 7f3a097fa640 1 -- 172.21.15.33:0/2964974906 <== mgr.14118 v2:172.21.15.33:6800/122116489 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3a0c1022f0 con 0x7f39f4039200 2024-09-16T00:07:49.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.941+0000 7f3a12893640 1 -- 172.21.15.33:0/2964974906 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f39f4039200 msgr2=0x7f39f403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:49.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.941+0000 7f3a12893640 1 --2- 172.21.15.33:0/2964974906 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f39f4039200 0x7f39f403b6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3a000098c0 tx=0x7f3a00006db0 comp rx=0 tx=0).stop 2024-09-16T00:07:49.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.941+0000 7f3a12893640 1 -- 172.21.15.33:0/2964974906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 msgr2=0x7f3a0c19b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:49.565 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.941+0000 7f3a12893640 1 --2- 172.21.15.33:0/2964974906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 0x7f3a0c19b680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0097f0 tx=0x7f39fc030ec0 comp rx=0 tx=0).stop 2024-09-16T00:07:49.565 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.942+0000 7f3a12893640 1 -- 172.21.15.33:0/2964974906 shutdown_connections 2024-09-16T00:07:49.565 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.942+0000 7f3a12893640 1 --2- 172.21.15.33:0/2964974906 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f39f4039200 0x7f39f403b6c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:49.565 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.942+0000 7f3a12893640 1 --2- 172.21.15.33:0/2964974906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a0c105280 0x7f3a0c19b680 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:49.565 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.942+0000 7f3a12893640 1 -- 172.21.15.33:0/2964974906 >> 172.21.15.33:0/2964974906 conn(0x7f3a0c100fa0 msgr2=0x7f3a0c101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:49.565 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.942+0000 7f3a12893640 1 -- 172.21.15.33:0/2964974906 shutdown_connections 2024-09-16T00:07:49.565 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:48.942+0000 7f3a12893640 1 -- 172.21.15.33:0/2964974906 wait complete. 2024-09-16T00:07:50.715 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:50 smithi033 ceph-mon[31243]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:07:50.724 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWLo+DVeRhGl4fpRXR9AdTann1eHkovQGVToV8Fb6GqW7ytfdWTo9JOIYTpAq7QVOZt+qjkQq9OEQMevdx2ZT0vexbU0ghavH0YlpUrciPQpggk4ROJPfugp+BvDfji8LQ12QaPKUclnp6O9gMLOMM+gKjixw7fh7FQv+/oIJFC2wtWnqGmbFedICLiVyeh2tDmDyCb5Q4L1rnD8wqc/AniiC2kfBD7nN29IR47LhTv71hnhCJmxJX3I46LX/fW1xqXAfLaTIsY2gwpSDasWqgsHlXR3EwUNYl/rqDQhdLtdSGp9fks3nN+rxLbMPgQYoOjKZ0ekxQ5sGbGaeTqjaP8g68FyeitSU7r9U7xKfu+MG2oTALiUp72VxS/FQSAt4kHMkmIk5JI/wKmghBlM0SXzX7aXITXydNL4S1xU88oj9E2wxuq0EM8tTIBKPZOdzUzjMfUoixwKpp2Mxkt7digFv9mwF1QAxvsDJV07ycIkF2tEzbT3oqOItlnffkAtk= ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:07:50.724 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.967+0000 7f698656b640 1 Processor -- start 2024-09-16T00:07:50.724 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.967+0000 7f698656b640 1 -- start start 2024-09-16T00:07:50.724 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.967+0000 7f698656b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 0x7f698010a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:50.724 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.967+0000 7f698656b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f698007be50 con 0x7f6980108160 2024-09-16T00:07:50.725 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.968+0000 7f697ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 0x7f698010a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:50.725 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.968+0000 7f697ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 0x7f698010a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37766/0 (socket says 172.21.15.33:37766) 2024-09-16T00:07:50.725 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.968+0000 7f697ffff640 1 -- 172.21.15.33:0/1691097346 learned_addr learned my addr 172.21.15.33:0/1691097346 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:50.725 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.968+0000 7f697ffff640 1 -- 172.21.15.33:0/1691097346 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698010aa90 con 0x7f6980108160 2024-09-16T00:07:50.725 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.969+0000 7f697ffff640 1 --2- 172.21.15.33:0/1691097346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 0x7f698010a550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6970009820 tx=0x7f697002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d799c59c063d62 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:50.725 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.969+0000 7f697effd640 1 -- 172.21.15.33:0/1691097346 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6970037070 con 0x7f6980108160 2024-09-16T00:07:50.725 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.969+0000 7f697effd640 1 -- 172.21.15.33:0/1691097346 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6970032900 con 0x7f6980108160 2024-09-16T00:07:50.725 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.969+0000 7f697effd640 1 -- 172.21.15.33:0/1691097346 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6970030850 con 0x7f6980108160 2024-09-16T00:07:50.725 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.970+0000 7f698656b640 1 -- 172.21.15.33:0/1691097346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 msgr2=0x7f698010a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:50.725 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.970+0000 7f698656b640 1 --2- 172.21.15.33:0/1691097346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 0x7f698010a550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6970009820 tx=0x7f697002a270 comp rx=0 tx=0).stop 2024-09-16T00:07:50.725 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.970+0000 7f698656b640 1 -- 172.21.15.33:0/1691097346 shutdown_connections 2024-09-16T00:07:50.725 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.970+0000 7f698656b640 1 --2- 172.21.15.33:0/1691097346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 0x7f698010a550 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.970+0000 7f698656b640 1 -- 172.21.15.33:0/1691097346 >> 172.21.15.33:0/1691097346 conn(0x7f6980101de0 msgr2=0x7f6980104200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.970+0000 7f698656b640 1 -- 172.21.15.33:0/1691097346 shutdown_connections 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.970+0000 7f698656b640 1 -- 172.21.15.33:0/1691097346 wait complete. 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.970+0000 7f698656b640 1 Processor -- start 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.970+0000 7f698656b640 1 -- start start 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.971+0000 7f698656b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 0x7f698019b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.971+0000 7f698656b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f698010ba60 con 0x7f6980108160 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.971+0000 7f697ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 0x7f698019b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.971+0000 7f697ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 0x7f698019b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37770/0 (socket says 172.21.15.33:37770) 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.971+0000 7f697ffff640 1 -- 172.21.15.33:0/3630759281 learned_addr learned my addr 172.21.15.33:0/3630759281 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.971+0000 7f697ffff640 1 -- 172.21.15.33:0/3630759281 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698019bb50 con 0x7f6980108160 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.972+0000 7f697ffff640 1 --2- 172.21.15.33:0/3630759281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 0x7f698019b610 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f69700097f0 tx=0x7f6970030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.972+0000 7f697d7fa640 1 -- 172.21.15.33:0/3630759281 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6970037040 con 0x7f6980108160 2024-09-16T00:07:50.726 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.972+0000 7f698656b640 1 -- 172.21.15.33:0/3630759281 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698019bde0 con 0x7f6980108160 2024-09-16T00:07:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.972+0000 7f697d7fa640 1 -- 172.21.15.33:0/3630759281 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6970040070 con 0x7f6980108160 2024-09-16T00:07:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.972+0000 7f697d7fa640 1 -- 172.21.15.33:0/3630759281 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f697003a390 con 0x7f6980108160 2024-09-16T00:07:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.972+0000 7f698656b640 1 -- 172.21.15.33:0/3630759281 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f698019ead0 con 0x7f6980108160 2024-09-16T00:07:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.973+0000 7f697d7fa640 1 -- 172.21.15.33:0/3630759281 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f697003a640 con 0x7f6980108160 2024-09-16T00:07:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.973+0000 7f697d7fa640 1 --2- 172.21.15.33:0/3630759281 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f6968039200 0x7f696803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.973+0000 7f697d7fa640 1 -- 172.21.15.33:0/3630759281 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f697006c8d0 con 0x7f6980108160 2024-09-16T00:07:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.973+0000 7f698656b640 1 -- 172.21.15.33:0/3630759281 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69801085c0 con 0x7f6980108160 2024-09-16T00:07:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.973+0000 7f697f7fe640 1 --2- 172.21.15.33:0/3630759281 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f6968039200 0x7f696803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.974+0000 7f697f7fe640 1 --2- 172.21.15.33:0/3630759281 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f6968039200 0x7f696803b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f69740098c0 tx=0x7f6974006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:49.978+0000 7f697d7fa640 1 -- 172.21.15.33:0/3630759281 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6970032ab0 con 0x7f6980108160 2024-09-16T00:07:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:50.157+0000 7f698656b640 1 -- 172.21.15.33:0/3630759281 --> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f6980102fc0 con 0x7f6968039200 2024-09-16T00:07:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:50.158+0000 7f697d7fa640 1 -- 172.21.15.33:0/3630759281 <== mgr.14118 v2:172.21.15.33:6800/122116489 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f6980102fc0 con 0x7f6968039200 2024-09-16T00:07:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:50.160+0000 7f698656b640 1 -- 172.21.15.33:0/3630759281 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f6968039200 msgr2=0x7f696803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:50.160+0000 7f698656b640 1 --2- 172.21.15.33:0/3630759281 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f6968039200 0x7f696803b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f69740098c0 tx=0x7f6974006db0 comp rx=0 tx=0).stop 2024-09-16T00:07:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:50.160+0000 7f698656b640 1 -- 172.21.15.33:0/3630759281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 msgr2=0x7f698019b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:50.160+0000 7f698656b640 1 --2- 172.21.15.33:0/3630759281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 0x7f698019b610 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f69700097f0 tx=0x7f6970030d80 comp rx=0 tx=0).stop 2024-09-16T00:07:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:50.160+0000 7f698656b640 1 -- 172.21.15.33:0/3630759281 shutdown_connections 2024-09-16T00:07:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:50.160+0000 7f698656b640 1 --2- 172.21.15.33:0/3630759281 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f6968039200 0x7f696803b6c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:50.160+0000 7f698656b640 1 --2- 172.21.15.33:0/3630759281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6980108160 0x7f698019b610 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:50.160+0000 7f698656b640 1 -- 172.21.15.33:0/3630759281 >> 172.21.15.33:0/3630759281 conn(0x7f6980101de0 msgr2=0x7f69801028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:50.160+0000 7f698656b640 1 -- 172.21.15.33:0/3630759281 shutdown_connections 2024-09-16T00:07:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:50.160+0000 7f698656b640 1 -- 172.21.15.33:0/3630759281 wait complete. 2024-09-16T00:07:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-16T00:07:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:Adding key to root@localhost authorized_keys... 2024-09-16T00:07:50.729 INFO:teuthology.orchestra.run.smithi033.stdout:Adding host smithi033... 2024-09-16T00:07:52.057 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:51 smithi033 ceph-mon[31243]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi033", "addr": "172.21.15.33", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:07:52.966 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:52 smithi033 ceph-mon[31243]: Deploying cephadm binary to smithi033 2024-09-16T00:07:55.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout Added host 'smithi033' with addr '172.21.15.33' 2024-09-16T00:07:55.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.145+0000 7fbccde5f640 1 Processor -- start 2024-09-16T00:07:55.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.145+0000 7fbccde5f640 1 -- start start 2024-09-16T00:07:55.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.145+0000 7fbccde5f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 0x7fbcc8107a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:55.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.145+0000 7fbccde5f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc8107fa0 con 0x7fbcc8107660 2024-09-16T00:07:55.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.145+0000 7fbcc77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 0x7fbcc8107a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:55.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.145+0000 7fbcc77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 0x7fbcc8107a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36590/0 (socket says 172.21.15.33:36590) 2024-09-16T00:07:55.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.145+0000 7fbcc77fe640 1 -- 172.21.15.33:0/3510544151 learned_addr learned my addr 172.21.15.33:0/3510544151 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:55.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.145+0000 7fbcc77fe640 1 -- 172.21.15.33:0/3510544151 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcc8108120 con 0x7fbcc8107660 2024-09-16T00:07:55.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.146+0000 7fbcc77fe640 1 --2- 172.21.15.33:0/3510544151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 0x7fbcc8107a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8009820 tx=0x7fbcb802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f3a36f850fa4e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:55.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.146+0000 7fbcc67fc640 1 -- 172.21.15.33:0/3510544151 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcb8037070 con 0x7fbcc8107660 2024-09-16T00:07:55.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.146+0000 7fbcc67fc640 1 -- 172.21.15.33:0/3510544151 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbcb8032900 con 0x7fbcc8107660 2024-09-16T00:07:55.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.146+0000 7fbcc67fc640 1 -- 172.21.15.33:0/3510544151 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcb8030850 con 0x7fbcc8107660 2024-09-16T00:07:55.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.147+0000 7fbccde5f640 1 -- 172.21.15.33:0/3510544151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 msgr2=0x7fbcc8107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:55.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.147+0000 7fbccde5f640 1 --2- 172.21.15.33:0/3510544151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 0x7fbcc8107a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8009820 tx=0x7fbcb802a270 comp rx=0 tx=0).stop 2024-09-16T00:07:55.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.147+0000 7fbccde5f640 1 -- 172.21.15.33:0/3510544151 shutdown_connections 2024-09-16T00:07:55.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.147+0000 7fbccde5f640 1 --2- 172.21.15.33:0/3510544151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 0x7fbcc8107a60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:55.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.147+0000 7fbccde5f640 1 -- 172.21.15.33:0/3510544151 >> 172.21.15.33:0/3510544151 conn(0x7fbcc8102eb0 msgr2=0x7fbcc81052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:55.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.147+0000 7fbccde5f640 1 -- 172.21.15.33:0/3510544151 shutdown_connections 2024-09-16T00:07:55.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.147+0000 7fbccde5f640 1 -- 172.21.15.33:0/3510544151 wait complete. 2024-09-16T00:07:55.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.147+0000 7fbccde5f640 1 Processor -- start 2024-09-16T00:07:55.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.148+0000 7fbccde5f640 1 -- start start 2024-09-16T00:07:55.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.148+0000 7fbccde5f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 0x7fbcc819b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:55.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.148+0000 7fbccde5f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc810bad0 con 0x7fbcc8107660 2024-09-16T00:07:55.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.148+0000 7fbcc77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 0x7fbcc819b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:55.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.148+0000 7fbcc77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 0x7fbcc819b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36598/0 (socket says 172.21.15.33:36598) 2024-09-16T00:07:55.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.148+0000 7fbcc77fe640 1 -- 172.21.15.33:0/63197032 learned_addr learned my addr 172.21.15.33:0/63197032 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:55.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.148+0000 7fbcc77fe640 1 -- 172.21.15.33:0/63197032 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcc819bb00 con 0x7fbcc8107660 2024-09-16T00:07:55.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.148+0000 7fbcc77fe640 1 --2- 172.21.15.33:0/63197032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 0x7fbcc819b5c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8032ce0 tx=0x7fbcb8032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:55.833 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.149+0000 7fbcc4ff9640 1 -- 172.21.15.33:0/63197032 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcb8042070 con 0x7fbcc8107660 2024-09-16T00:07:55.833 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.149+0000 7fbcc4ff9640 1 -- 172.21.15.33:0/63197032 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbcb80308c0 con 0x7fbcc8107660 2024-09-16T00:07:55.833 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.149+0000 7fbccde5f640 1 -- 172.21.15.33:0/63197032 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcc819bd90 con 0x7fbcc8107660 2024-09-16T00:07:55.833 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.149+0000 7fbcc4ff9640 1 -- 172.21.15.33:0/63197032 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcb8037040 con 0x7fbcc8107660 2024-09-16T00:07:55.833 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.149+0000 7fbccde5f640 1 -- 172.21.15.33:0/63197032 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcc819ea80 con 0x7fbcc8107660 2024-09-16T00:07:55.833 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.150+0000 7fbcc4ff9640 1 -- 172.21.15.33:0/63197032 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fbcb8045070 con 0x7fbcc8107660 2024-09-16T00:07:55.833 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.150+0000 7fbccde5f640 1 -- 172.21.15.33:0/63197032 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcc8107ae0 con 0x7fbcc8107660 2024-09-16T00:07:55.833 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.150+0000 7fbcc4ff9640 1 --2- 172.21.15.33:0/63197032 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fbcb00391b0 0x7fbcb003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:55.833 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.150+0000 7fbcc4ff9640 1 -- 172.21.15.33:0/63197032 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbcb806c480 con 0x7fbcc8107660 2024-09-16T00:07:55.833 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.150+0000 7fbcc6ffd640 1 --2- 172.21.15.33:0/63197032 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fbcb00391b0 0x7fbcb003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:55.834 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.151+0000 7fbcc6ffd640 1 --2- 172.21.15.33:0/63197032 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fbcb00391b0 0x7fbcb003b670 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc0098c0 tx=0x7fbcbc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:55.834 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.155+0000 7fbcc4ff9640 1 -- 172.21.15.33:0/63197032 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbcb8045520 con 0x7fbcc8107660 2024-09-16T00:07:55.834 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:51.335+0000 7fbccde5f640 1 -- 172.21.15.33:0/63197032 --> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi033", "addr": "172.21.15.33", "target": ["mon-mgr", ""]}) -- 0x7fbcc8104060 con 0x7fbcb00391b0 2024-09-16T00:07:55.834 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:55.223+0000 7fbcc4ff9640 1 -- 172.21.15.33:0/63197032 <== mgr.14118 v2:172.21.15.33:6800/122116489 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fbcc8104060 con 0x7fbcb00391b0 2024-09-16T00:07:55.834 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:55.226+0000 7fbccde5f640 1 -- 172.21.15.33:0/63197032 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fbcb00391b0 msgr2=0x7fbcb003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:55.834 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:55.226+0000 7fbccde5f640 1 --2- 172.21.15.33:0/63197032 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fbcb00391b0 0x7fbcb003b670 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc0098c0 tx=0x7fbcbc006db0 comp rx=0 tx=0).stop 2024-09-16T00:07:55.835 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:55.226+0000 7fbccde5f640 1 -- 172.21.15.33:0/63197032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 msgr2=0x7fbcc819b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:55.835 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:55.226+0000 7fbccde5f640 1 --2- 172.21.15.33:0/63197032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 0x7fbcc819b5c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8032ce0 tx=0x7fbcb8032d10 comp rx=0 tx=0).stop 2024-09-16T00:07:55.835 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:55.226+0000 7fbccde5f640 1 -- 172.21.15.33:0/63197032 shutdown_connections 2024-09-16T00:07:55.835 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:55.226+0000 7fbccde5f640 1 --2- 172.21.15.33:0/63197032 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fbcb00391b0 0x7fbcb003b670 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:55.835 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:55.226+0000 7fbccde5f640 1 --2- 172.21.15.33:0/63197032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcc8107660 0x7fbcc819b5c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:55.835 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:55.226+0000 7fbccde5f640 1 -- 172.21.15.33:0/63197032 >> 172.21.15.33:0/63197032 conn(0x7fbcc8102eb0 msgr2=0x7fbcc8103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:55.835 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:55.226+0000 7fbccde5f640 1 -- 172.21.15.33:0/63197032 shutdown_connections 2024-09-16T00:07:55.835 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:55.226+0000 7fbccde5f640 1 -- 172.21.15.33:0/63197032 wait complete. 2024-09-16T00:07:55.835 INFO:teuthology.orchestra.run.smithi033.stdout:Deploying unmanaged mon service... 2024-09-16T00:07:56.475 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:56 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' 2024-09-16T00:07:56.475 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:56 smithi033 ceph-mon[31243]: Added host smithi033 2024-09-16T00:07:56.476 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:56 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:07:57.368 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-16T00:07:57.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.252+0000 7fbecabd6640 1 Processor -- start 2024-09-16T00:07:57.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.252+0000 7fbecabd6640 1 -- start start 2024-09-16T00:07:57.373 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.252+0000 7fbecabd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 0x7fbec410a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:57.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.252+0000 7fbecabd6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbec407be70 con 0x7fbec4108140 2024-09-16T00:07:57.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.252+0000 7fbec8972640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 0x7fbec410a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:57.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.252+0000 7fbec8972640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 0x7fbec410a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36612/0 (socket says 172.21.15.33:36612) 2024-09-16T00:07:57.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.252+0000 7fbec8972640 1 -- 172.21.15.33:0/392943816 learned_addr learned my addr 172.21.15.33:0/392943816 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:57.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.252+0000 7fbec8972640 1 -- 172.21.15.33:0/392943816 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec410aa70 con 0x7fbec4108140 2024-09-16T00:07:57.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.253+0000 7fbec8972640 1 --2- 172.21.15.33:0/392943816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 0x7fbec410a530 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009820 tx=0x7fbeb402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b89893ff728ae2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:57.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.253+0000 7fbec37fe640 1 -- 172.21.15.33:0/392943816 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbeb4037070 con 0x7fbec4108140 2024-09-16T00:07:57.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.253+0000 7fbec37fe640 1 -- 172.21.15.33:0/392943816 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbeb4032900 con 0x7fbec4108140 2024-09-16T00:07:57.374 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.253+0000 7fbec37fe640 1 -- 172.21.15.33:0/392943816 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbeb4030850 con 0x7fbec4108140 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.254+0000 7fbecabd6640 1 -- 172.21.15.33:0/392943816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 msgr2=0x7fbec410a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.254+0000 7fbecabd6640 1 --2- 172.21.15.33:0/392943816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 0x7fbec410a530 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009820 tx=0x7fbeb402a270 comp rx=0 tx=0).stop 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.254+0000 7fbecabd6640 1 -- 172.21.15.33:0/392943816 shutdown_connections 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.254+0000 7fbecabd6640 1 --2- 172.21.15.33:0/392943816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 0x7fbec410a530 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.254+0000 7fbecabd6640 1 -- 172.21.15.33:0/392943816 >> 172.21.15.33:0/392943816 conn(0x7fbec4101de0 msgr2=0x7fbec4104200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.254+0000 7fbecabd6640 1 -- 172.21.15.33:0/392943816 shutdown_connections 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.254+0000 7fbecabd6640 1 -- 172.21.15.33:0/392943816 wait complete. 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.255+0000 7fbecabd6640 1 Processor -- start 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.255+0000 7fbecabd6640 1 -- start start 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.255+0000 7fbecabd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 0x7fbec419d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.255+0000 7fbecabd6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbec410ba40 con 0x7fbec4108140 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.255+0000 7fbec8972640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 0x7fbec419d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:57.375 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.255+0000 7fbec8972640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 0x7fbec419d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36618/0 (socket says 172.21.15.33:36618) 2024-09-16T00:07:57.376 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.255+0000 7fbec8972640 1 -- 172.21.15.33:0/2918743857 learned_addr learned my addr 172.21.15.33:0/2918743857 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:57.376 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.255+0000 7fbec8972640 1 -- 172.21.15.33:0/2918743857 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec419dd50 con 0x7fbec4108140 2024-09-16T00:07:57.376 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.256+0000 7fbec8972640 1 --2- 172.21.15.33:0/2918743857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 0x7fbec419d810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4032ce0 tx=0x7fbeb4032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:57.376 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.256+0000 7fbec1ffb640 1 -- 172.21.15.33:0/2918743857 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbeb4042070 con 0x7fbec4108140 2024-09-16T00:07:57.376 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.256+0000 7fbec1ffb640 1 -- 172.21.15.33:0/2918743857 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbeb40308c0 con 0x7fbec4108140 2024-09-16T00:07:57.376 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.256+0000 7fbec1ffb640 1 -- 172.21.15.33:0/2918743857 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbeb4037040 con 0x7fbec4108140 2024-09-16T00:07:57.376 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.256+0000 7fbecabd6640 1 -- 172.21.15.33:0/2918743857 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbec419dfe0 con 0x7fbec4108140 2024-09-16T00:07:57.376 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.256+0000 7fbecabd6640 1 -- 172.21.15.33:0/2918743857 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec41a0cd0 con 0x7fbec4108140 2024-09-16T00:07:57.376 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.257+0000 7fbec1ffb640 1 -- 172.21.15.33:0/2918743857 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fbeb4045070 con 0x7fbec4108140 2024-09-16T00:07:57.376 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.257+0000 7fbecabd6640 1 -- 172.21.15.33:0/2918743857 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbec41085a0 con 0x7fbec4108140 2024-09-16T00:07:57.377 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.257+0000 7fbec1ffb640 1 --2- 172.21.15.33:0/2918743857 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fbeac039200 0x7fbeac03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:57.377 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.257+0000 7fbec1ffb640 1 -- 172.21.15.33:0/2918743857 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbeb406c2c0 con 0x7fbec4108140 2024-09-16T00:07:57.377 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.257+0000 7fbec3fff640 1 --2- 172.21.15.33:0/2918743857 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fbeac039200 0x7fbeac03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:57.377 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.258+0000 7fbec3fff640 1 --2- 172.21.15.33:0/2918743857 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fbeac039200 0x7fbeac03b6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80098c0 tx=0x7fbeb8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:57.377 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.262+0000 7fbec1ffb640 1 -- 172.21.15.33:0/2918743857 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbeb404b170 con 0x7fbec4108140 2024-09-16T00:07:57.377 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.473+0000 7fbecabd6640 1 -- 172.21.15.33:0/2918743857 --> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fbec4102fc0 con 0x7fbeac039200 2024-09-16T00:07:57.377 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.478+0000 7fbec1ffb640 1 -- 172.21.15.33:0/2918743857 <== mgr.14118 v2:172.21.15.33:6800/122116489 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fbec4102fc0 con 0x7fbeac039200 2024-09-16T00:07:57.377 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.480+0000 7fbecabd6640 1 -- 172.21.15.33:0/2918743857 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fbeac039200 msgr2=0x7fbeac03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:57.378 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.480+0000 7fbecabd6640 1 --2- 172.21.15.33:0/2918743857 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fbeac039200 0x7fbeac03b6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80098c0 tx=0x7fbeb8006db0 comp rx=0 tx=0).stop 2024-09-16T00:07:57.378 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.480+0000 7fbecabd6640 1 -- 172.21.15.33:0/2918743857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 msgr2=0x7fbec419d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:57.378 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.480+0000 7fbecabd6640 1 --2- 172.21.15.33:0/2918743857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 0x7fbec419d810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4032ce0 tx=0x7fbeb4032d10 comp rx=0 tx=0).stop 2024-09-16T00:07:57.378 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.480+0000 7fbecabd6640 1 -- 172.21.15.33:0/2918743857 shutdown_connections 2024-09-16T00:07:57.378 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.480+0000 7fbecabd6640 1 --2- 172.21.15.33:0/2918743857 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fbeac039200 0x7fbeac03b6c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:57.379 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.480+0000 7fbecabd6640 1 --2- 172.21.15.33:0/2918743857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbec4108140 0x7fbec419d810 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:57.379 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.480+0000 7fbecabd6640 1 -- 172.21.15.33:0/2918743857 >> 172.21.15.33:0/2918743857 conn(0x7fbec4101de0 msgr2=0x7fbec41028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:57.379 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.480+0000 7fbecabd6640 1 -- 172.21.15.33:0/2918743857 shutdown_connections 2024-09-16T00:07:57.379 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:56.480+0000 7fbecabd6640 1 -- 172.21.15.33:0/2918743857 wait complete. 2024-09-16T00:07:57.379 INFO:teuthology.orchestra.run.smithi033.stdout:Deploying unmanaged mgr service... 2024-09-16T00:07:57.709 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:57 smithi033 ceph-mon[31243]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:07:57.709 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:57 smithi033 ceph-mon[31243]: Saving service mon spec with placement count:5 2024-09-16T00:07:57.709 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:57 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' 2024-09-16T00:07:58.655 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-16T00:07:58.655 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.800+0000 7f528b9de640 1 Processor -- start 2024-09-16T00:07:58.655 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.800+0000 7f528b9de640 1 -- start start 2024-09-16T00:07:58.655 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.800+0000 7f528b9de640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 0x7f528407c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:58.655 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.800+0000 7f528b9de640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f528407c880 con 0x7f528407de40 2024-09-16T00:07:58.655 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.801+0000 7f528977a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 0x7f528407c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:58.656 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.801+0000 7f528977a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 0x7f528407c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36632/0 (socket says 172.21.15.33:36632) 2024-09-16T00:07:58.656 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.801+0000 7f528977a640 1 -- 172.21.15.33:0/2657001806 learned_addr learned my addr 172.21.15.33:0/2657001806 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:58.656 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.801+0000 7f528977a640 1 -- 172.21.15.33:0/2657001806 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528407ca00 con 0x7f528407de40 2024-09-16T00:07:58.656 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.802+0000 7f528977a640 1 --2- 172.21.15.33:0/2657001806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 0x7f528407c340 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5280009820 tx=0x7f528002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45fe8f92ff0f07e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:58.656 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.802+0000 7f527bfff640 1 -- 172.21.15.33:0/2657001806 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5280037070 con 0x7f528407de40 2024-09-16T00:07:58.656 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.802+0000 7f527bfff640 1 -- 172.21.15.33:0/2657001806 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5280032900 con 0x7f528407de40 2024-09-16T00:07:58.656 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.802+0000 7f527bfff640 1 -- 172.21.15.33:0/2657001806 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5280030850 con 0x7f528407de40 2024-09-16T00:07:58.656 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.802+0000 7f528b9de640 1 -- 172.21.15.33:0/2657001806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 msgr2=0x7f528407c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:58.656 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.802+0000 7f528b9de640 1 --2- 172.21.15.33:0/2657001806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 0x7f528407c340 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5280009820 tx=0x7f528002a270 comp rx=0 tx=0).stop 2024-09-16T00:07:58.656 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.803+0000 7f528b9de640 1 -- 172.21.15.33:0/2657001806 shutdown_connections 2024-09-16T00:07:58.656 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.803+0000 7f528b9de640 1 --2- 172.21.15.33:0/2657001806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 0x7f528407c340 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:58.656 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.803+0000 7f528b9de640 1 -- 172.21.15.33:0/2657001806 >> 172.21.15.33:0/2657001806 conn(0x7f5284102eb0 msgr2=0x7f5284105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.803+0000 7f528b9de640 1 -- 172.21.15.33:0/2657001806 shutdown_connections 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.803+0000 7f528b9de640 1 -- 172.21.15.33:0/2657001806 wait complete. 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.803+0000 7f528b9de640 1 Processor -- start 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.803+0000 7f528b9de640 1 -- start start 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.804+0000 7f528b9de640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 0x7f52841a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.804+0000 7f528b9de640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f52841098a0 con 0x7f528407de40 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.804+0000 7f528977a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 0x7f52841a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.804+0000 7f528977a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 0x7f52841a1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36644/0 (socket says 172.21.15.33:36644) 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.804+0000 7f528977a640 1 -- 172.21.15.33:0/2923445378 learned_addr learned my addr 172.21.15.33:0/2923445378 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.804+0000 7f528977a640 1 -- 172.21.15.33:0/2923445378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52841a21b0 con 0x7f528407de40 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.804+0000 7f528977a640 1 --2- 172.21.15.33:0/2923445378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 0x7f52841a1c70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5280031000 tx=0x7f52800326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.804+0000 7f527a7fc640 1 -- 172.21.15.33:0/2923445378 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5280042070 con 0x7f528407de40 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.804+0000 7f527a7fc640 1 -- 172.21.15.33:0/2923445378 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5280032be0 con 0x7f528407de40 2024-09-16T00:07:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.804+0000 7f528b9de640 1 -- 172.21.15.33:0/2923445378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52841a2440 con 0x7f528407de40 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.804+0000 7f527a7fc640 1 -- 172.21.15.33:0/2923445378 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5280037040 con 0x7f528407de40 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.804+0000 7f528b9de640 1 -- 172.21.15.33:0/2923445378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52841a5130 con 0x7f528407de40 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.805+0000 7f527a7fc640 1 -- 172.21.15.33:0/2923445378 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f5280045070 con 0x7f528407de40 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.806+0000 7f528b9de640 1 -- 172.21.15.33:0/2923445378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f528407c3c0 con 0x7f528407de40 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.806+0000 7f527a7fc640 1 --2- 172.21.15.33:0/2923445378 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f52700391b0 0x7f527003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.806+0000 7f527a7fc640 1 -- 172.21.15.33:0/2923445378 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f528006c280 con 0x7f528407de40 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.806+0000 7f5288f79640 1 --2- 172.21.15.33:0/2923445378 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f52700391b0 0x7f527003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.806+0000 7f5288f79640 1 --2- 172.21.15.33:0/2923445378 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f52700391b0 0x7f527003b670 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f52740098c0 tx=0x7f5274006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:57.811+0000 7f527a7fc640 1 -- 172.21.15.33:0/2923445378 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f528002f2a0 con 0x7f528407de40 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:58.009+0000 7f528b9de640 1 -- 172.21.15.33:0/2923445378 --> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f528402da60 con 0x7f52700391b0 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:58.014+0000 7f527a7fc640 1 -- 172.21.15.33:0/2923445378 <== mgr.14118 v2:172.21.15.33:6800/122116489 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f528402da60 con 0x7f52700391b0 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:58.016+0000 7f528b9de640 1 -- 172.21.15.33:0/2923445378 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f52700391b0 msgr2=0x7f527003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:58.658 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:58.016+0000 7f528b9de640 1 --2- 172.21.15.33:0/2923445378 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f52700391b0 0x7f527003b670 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f52740098c0 tx=0x7f5274006db0 comp rx=0 tx=0).stop 2024-09-16T00:07:58.659 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:58.016+0000 7f528b9de640 1 -- 172.21.15.33:0/2923445378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 msgr2=0x7f52841a1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:07:58.659 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:58.016+0000 7f528b9de640 1 --2- 172.21.15.33:0/2923445378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 0x7f52841a1c70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5280031000 tx=0x7f52800326e0 comp rx=0 tx=0).stop 2024-09-16T00:07:58.659 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:58.016+0000 7f528b9de640 1 -- 172.21.15.33:0/2923445378 shutdown_connections 2024-09-16T00:07:58.659 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:58.016+0000 7f528b9de640 1 --2- 172.21.15.33:0/2923445378 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f52700391b0 0x7f527003b670 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:58.659 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:58.016+0000 7f528b9de640 1 --2- 172.21.15.33:0/2923445378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f528407de40 0x7f52841a1c70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:07:58.659 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:58.016+0000 7f528b9de640 1 -- 172.21.15.33:0/2923445378 >> 172.21.15.33:0/2923445378 conn(0x7f5284102eb0 msgr2=0x7f5284105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:07:58.659 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:58.016+0000 7f528b9de640 1 -- 172.21.15.33:0/2923445378 shutdown_connections 2024-09-16T00:07:58.659 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:58.016+0000 7f528b9de640 1 -- 172.21.15.33:0/2923445378 wait complete. 2024-09-16T00:07:59.230 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:59 smithi033 ceph-mon[31243]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:07:59.230 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:59 smithi033 ceph-mon[31243]: Saving service mgr spec with placement count:2 2024-09-16T00:07:59.230 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:07:59 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' 2024-09-16T00:08:00.021 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.042+0000 7fd5ff078640 1 Processor -- start 2024-09-16T00:08:00.021 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.042+0000 7fd5ff078640 1 -- start start 2024-09-16T00:08:00.021 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.042+0000 7fd5ff078640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 0x7fd5f8106920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:00.021 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.042+0000 7fd5ff078640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f8106ef0 con 0x7fd5f8106520 2024-09-16T00:08:00.022 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.042+0000 7fd5fce14640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 0x7fd5f8106920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:00.022 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.042+0000 7fd5fce14640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 0x7fd5f8106920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36650/0 (socket says 172.21.15.33:36650) 2024-09-16T00:08:00.022 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.042+0000 7fd5fce14640 1 -- 172.21.15.33:0/2063545671 learned_addr learned my addr 172.21.15.33:0/2063545671 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:00.022 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.043+0000 7fd5fce14640 1 -- 172.21.15.33:0/2063545671 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f8107720 con 0x7fd5f8106520 2024-09-16T00:08:00.022 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.043+0000 7fd5fce14640 1 --2- 172.21.15.33:0/2063545671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 0x7fd5f8106920 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009820 tx=0x7fd5ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7860bfac4e0ef47 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:00.022 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.044+0000 7fd5f77fe640 1 -- 172.21.15.33:0/2063545671 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5ec037070 con 0x7fd5f8106520 2024-09-16T00:08:00.022 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.044+0000 7fd5f77fe640 1 -- 172.21.15.33:0/2063545671 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd5ec032900 con 0x7fd5f8106520 2024-09-16T00:08:00.022 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.044+0000 7fd5f77fe640 1 -- 172.21.15.33:0/2063545671 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5ec030850 con 0x7fd5f8106520 2024-09-16T00:08:00.022 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.044+0000 7fd5ff078640 1 -- 172.21.15.33:0/2063545671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 msgr2=0x7fd5f8106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:00.022 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.044+0000 7fd5ff078640 1 --2- 172.21.15.33:0/2063545671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 0x7fd5f8106920 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009820 tx=0x7fd5ec02a270 comp rx=0 tx=0).stop 2024-09-16T00:08:00.022 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.044+0000 7fd5ff078640 1 -- 172.21.15.33:0/2063545671 shutdown_connections 2024-09-16T00:08:00.022 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.044+0000 7fd5ff078640 1 --2- 172.21.15.33:0/2063545671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 0x7fd5f8106920 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:00.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.044+0000 7fd5ff078640 1 -- 172.21.15.33:0/2063545671 >> 172.21.15.33:0/2063545671 conn(0x7fd5f8101db0 msgr2=0x7fd5f81041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:00.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.044+0000 7fd5ff078640 1 -- 172.21.15.33:0/2063545671 shutdown_connections 2024-09-16T00:08:00.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.044+0000 7fd5ff078640 1 -- 172.21.15.33:0/2063545671 wait complete. 2024-09-16T00:08:00.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.045+0000 7fd5ff078640 1 Processor -- start 2024-09-16T00:08:00.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.045+0000 7fd5ff078640 1 -- start start 2024-09-16T00:08:00.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.045+0000 7fd5ff078640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 0x7fd5f81993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:00.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.045+0000 7fd5ff078640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f807b150 con 0x7fd5f8106520 2024-09-16T00:08:00.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.045+0000 7fd5fce14640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 0x7fd5f81993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:00.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.045+0000 7fd5fce14640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 0x7fd5f81993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36658/0 (socket says 172.21.15.33:36658) 2024-09-16T00:08:00.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.045+0000 7fd5fce14640 1 -- 172.21.15.33:0/2863250838 learned_addr learned my addr 172.21.15.33:0/2863250838 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:00.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.045+0000 7fd5fce14640 1 -- 172.21.15.33:0/2863250838 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f8199920 con 0x7fd5f8106520 2024-09-16T00:08:00.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.046+0000 7fd5fce14640 1 --2- 172.21.15.33:0/2863250838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 0x7fd5f81993e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009940 tx=0x7fd5ec032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:00.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.046+0000 7fd5f5ffb640 1 -- 172.21.15.33:0/2863250838 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5ec042070 con 0x7fd5f8106520 2024-09-16T00:08:00.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.046+0000 7fd5f5ffb640 1 -- 172.21.15.33:0/2863250838 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd5ec0308c0 con 0x7fd5f8106520 2024-09-16T00:08:00.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.046+0000 7fd5f5ffb640 1 -- 172.21.15.33:0/2863250838 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5ec037040 con 0x7fd5f8106520 2024-09-16T00:08:00.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.046+0000 7fd5ff078640 1 -- 172.21.15.33:0/2863250838 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f8199bb0 con 0x7fd5f8106520 2024-09-16T00:08:00.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.046+0000 7fd5ff078640 1 -- 172.21.15.33:0/2863250838 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f819c8a0 con 0x7fd5f8106520 2024-09-16T00:08:00.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.047+0000 7fd5f5ffb640 1 -- 172.21.15.33:0/2863250838 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fd5ec045070 con 0x7fd5f8106520 2024-09-16T00:08:00.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.047+0000 7fd5ff078640 1 -- 172.21.15.33:0/2863250838 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f8106920 con 0x7fd5f8106520 2024-09-16T00:08:00.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.047+0000 7fd5f5ffb640 1 --2- 172.21.15.33:0/2863250838 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd5e803b460 0x7fd5e803d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:00.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.047+0000 7fd5f5ffb640 1 -- 172.21.15.33:0/2863250838 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd5ec06c2c0 con 0x7fd5f8106520 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.047+0000 7fd5f7fff640 1 --2- 172.21.15.33:0/2863250838 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd5e803b460 0x7fd5e803d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.048+0000 7fd5f7fff640 1 --2- 172.21.15.33:0/2863250838 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd5e803b460 0x7fd5e803d920 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40098c0 tx=0x7fd5e4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.052+0000 7fd5f5ffb640 1 -- 172.21.15.33:0/2863250838 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd5ec04b170 con 0x7fd5f8106520 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.228+0000 7fd5ff078640 1 -- 172.21.15.33:0/2863250838 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fd5f8192850 con 0x7fd5f8106520 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.234+0000 7fd5f5ffb640 1 -- 172.21.15.33:0/2863250838 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7fd5ec03cd60 con 0x7fd5f8106520 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.236+0000 7fd5ff078640 1 -- 172.21.15.33:0/2863250838 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd5e803b460 msgr2=0x7fd5e803d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.236+0000 7fd5ff078640 1 --2- 172.21.15.33:0/2863250838 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd5e803b460 0x7fd5e803d920 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40098c0 tx=0x7fd5e4006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.236+0000 7fd5ff078640 1 -- 172.21.15.33:0/2863250838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 msgr2=0x7fd5f81993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.236+0000 7fd5ff078640 1 --2- 172.21.15.33:0/2863250838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 0x7fd5f81993e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009940 tx=0x7fd5ec032ce0 comp rx=0 tx=0).stop 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.236+0000 7fd5ff078640 1 -- 172.21.15.33:0/2863250838 shutdown_connections 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.236+0000 7fd5ff078640 1 --2- 172.21.15.33:0/2863250838 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd5e803b460 0x7fd5e803d920 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.236+0000 7fd5ff078640 1 --2- 172.21.15.33:0/2863250838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5f8106520 0x7fd5f81993e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.236+0000 7fd5ff078640 1 -- 172.21.15.33:0/2863250838 >> 172.21.15.33:0/2863250838 conn(0x7fd5f8101db0 msgr2=0x7fd5f818f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:00.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.236+0000 7fd5ff078640 1 -- 172.21.15.33:0/2863250838 shutdown_connections 2024-09-16T00:08:00.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:07:59.236+0000 7fd5ff078640 1 -- 172.21.15.33:0/2863250838 wait complete. 2024-09-16T00:08:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:00 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2863250838' entity='client.admin' 2024-09-16T00:08:01.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:01 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' 2024-09-16T00:08:01.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:01 smithi033 ceph-mon[31243]: from='mgr.14118 172.21.15.33:0/3247662709' entity='mgr.x' 2024-09-16T00:08:01.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:01 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3546282318' entity='client.admin' 2024-09-16T00:08:01.730 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.951+0000 7f84a86ec640 1 Processor -- start 2024-09-16T00:08:01.730 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.951+0000 7f84a86ec640 1 -- start start 2024-09-16T00:08:01.730 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.951+0000 7f84a86ec640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 0x7f84a010a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:01.730 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.951+0000 7f84a86ec640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f84a007beb0 con 0x7f84a0108090 2024-09-16T00:08:01.730 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.952+0000 7f84a6488640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 0x7f84a010a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.952+0000 7f84a6488640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 0x7f84a010a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54714/0 (socket says 172.21.15.33:54714) 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.952+0000 7f84a6488640 1 -- 172.21.15.33:0/4280043565 learned_addr learned my addr 172.21.15.33:0/4280043565 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.952+0000 7f84a6488640 1 -- 172.21.15.33:0/4280043565 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a010a9c0 con 0x7f84a0108090 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.953+0000 7f84a6488640 1 --2- 172.21.15.33:0/4280043565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 0x7f84a010a480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f849c009820 tx=0x7f849c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b62b4d5ad11cb55b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.953+0000 7f84a5486640 1 -- 172.21.15.33:0/4280043565 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f849c037070 con 0x7f84a0108090 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.953+0000 7f84a5486640 1 -- 172.21.15.33:0/4280043565 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f849c032920 con 0x7f84a0108090 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.953+0000 7f84a5486640 1 -- 172.21.15.33:0/4280043565 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f849c030810 con 0x7f84a0108090 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.954+0000 7f84a86ec640 1 -- 172.21.15.33:0/4280043565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 msgr2=0x7f84a010a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.954+0000 7f84a86ec640 1 --2- 172.21.15.33:0/4280043565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 0x7f84a010a480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f849c009820 tx=0x7f849c02a270 comp rx=0 tx=0).stop 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.954+0000 7f84a86ec640 1 -- 172.21.15.33:0/4280043565 shutdown_connections 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.954+0000 7f84a86ec640 1 --2- 172.21.15.33:0/4280043565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 0x7f84a010a480 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.954+0000 7f84a86ec640 1 -- 172.21.15.33:0/4280043565 >> 172.21.15.33:0/4280043565 conn(0x7f84a0101db0 msgr2=0x7f84a01041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:01.731 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.954+0000 7f84a86ec640 1 -- 172.21.15.33:0/4280043565 shutdown_connections 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.954+0000 7f84a86ec640 1 -- 172.21.15.33:0/4280043565 wait complete. 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.954+0000 7f84a86ec640 1 Processor -- start 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.954+0000 7f84a86ec640 1 -- start start 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.955+0000 7f84a86ec640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 0x7f84a0107910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.955+0000 7f84a86ec640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f84a010baa0 con 0x7f84a0108090 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.955+0000 7f84a6488640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 0x7f84a0107910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.955+0000 7f84a6488640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 0x7f84a0107910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54716/0 (socket says 172.21.15.33:54716) 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.955+0000 7f84a6488640 1 -- 172.21.15.33:0/3546282318 learned_addr learned my addr 172.21.15.33:0/3546282318 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.955+0000 7f84a6488640 1 -- 172.21.15.33:0/3546282318 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a0107e50 con 0x7f84a0108090 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.955+0000 7f84a6488640 1 --2- 172.21.15.33:0/3546282318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 0x7f84a0107910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f849c030380 tx=0x7f849c032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.955+0000 7f84977fe640 1 -- 172.21.15.33:0/3546282318 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f849c037040 con 0x7f84a0108090 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.955+0000 7f84977fe640 1 -- 172.21.15.33:0/3546282318 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f849c040070 con 0x7f84a0108090 2024-09-16T00:08:01.732 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.955+0000 7f84a86ec640 1 -- 172.21.15.33:0/3546282318 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84a0106040 con 0x7f84a0108090 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.955+0000 7f84977fe640 1 -- 172.21.15.33:0/3546282318 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f849c03a7b0 con 0x7f84a0108090 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.955+0000 7f84a86ec640 1 -- 172.21.15.33:0/3546282318 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84a0106580 con 0x7f84a0108090 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.956+0000 7f84977fe640 1 -- 172.21.15.33:0/3546282318 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f849c03a9e0 con 0x7f84a0108090 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.956+0000 7f84a86ec640 1 -- 172.21.15.33:0/3546282318 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84a01084f0 con 0x7f84a0108090 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.957+0000 7f84977fe640 1 --2- 172.21.15.33:0/3546282318 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f848c039200 0x7f848c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.957+0000 7f84977fe640 1 -- 172.21.15.33:0/3546282318 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f849c06c8c0 con 0x7f84a0108090 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.957+0000 7f84a5c87640 1 --2- 172.21.15.33:0/3546282318 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f848c039200 0x7f848c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.957+0000 7f84a5c87640 1 --2- 172.21.15.33:0/3546282318 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f848c039200 0x7f848c03b6c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f84900098c0 tx=0x7f8490006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:00.961+0000 7f84977fe640 1 -- 172.21.15.33:0/3546282318 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f849c044c70 con 0x7f84a0108090 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:01.151+0000 7f84a86ec640 1 -- 172.21.15.33:0/3546282318 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f84a0106ed0 con 0x7f84a0108090 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:01.156+0000 7f84977fe640 1 -- 172.21.15.33:0/3546282318 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f84a0106ed0 con 0x7f84a0108090 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:01.159+0000 7f84a86ec640 1 -- 172.21.15.33:0/3546282318 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f848c039200 msgr2=0x7f848c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:01.733 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:01.159+0000 7f84a86ec640 1 --2- 172.21.15.33:0/3546282318 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f848c039200 0x7f848c03b6c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f84900098c0 tx=0x7f8490006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:01.734 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:01.159+0000 7f84a86ec640 1 -- 172.21.15.33:0/3546282318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 msgr2=0x7f84a0107910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:01.734 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:01.159+0000 7f84a86ec640 1 --2- 172.21.15.33:0/3546282318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 0x7f84a0107910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f849c030380 tx=0x7f849c032670 comp rx=0 tx=0).stop 2024-09-16T00:08:01.734 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:01.159+0000 7f84a86ec640 1 -- 172.21.15.33:0/3546282318 shutdown_connections 2024-09-16T00:08:01.734 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:01.159+0000 7f84a86ec640 1 --2- 172.21.15.33:0/3546282318 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f848c039200 0x7f848c03b6c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:01.734 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:01.159+0000 7f84a86ec640 1 --2- 172.21.15.33:0/3546282318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a0108090 0x7f84a0107910 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:01.734 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:01.159+0000 7f84a86ec640 1 -- 172.21.15.33:0/3546282318 >> 172.21.15.33:0/3546282318 conn(0x7f84a0101db0 msgr2=0x7f84a01028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:01.734 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:01.159+0000 7f84a86ec640 1 -- 172.21.15.33:0/3546282318 shutdown_connections 2024-09-16T00:08:01.734 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:01.159+0000 7f84a86ec640 1 -- 172.21.15.33:0/3546282318 wait complete. 2024-09-16T00:08:01.734 INFO:teuthology.orchestra.run.smithi033.stdout:Enabling the dashboard module... 2024-09-16T00:08:02.687 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:02 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3432238806' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-16T00:08:02.687 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:02 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: ignoring --setuser ceph since I am not root 2024-09-16T00:08:02.688 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:02 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: ignoring --setgroup ceph since I am not root 2024-09-16T00:08:02.688 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:02 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:02.586+0000 7f75bbc64040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T00:08:02.688 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:02 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:02.645+0000 7f75bbc64040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T00:08:03.030 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:02 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:02.801+0000 7f75bbc64040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T00:08:03.031 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:02 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:02.870+0000 7f75bbc64040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T00:08:03.031 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:03 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:03.028+0000 7f75bbc64040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T00:08:03.330 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:03 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:03.176+0000 7f75bbc64040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T00:08:03.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.166+0000 7fa82e374640 1 Processor -- start 2024-09-16T00:08:03.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.166+0000 7fa82e374640 1 -- start start 2024-09-16T00:08:03.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.167+0000 7fa82e374640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 0x7fa828106920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:03.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.167+0000 7fa82e374640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa828106ef0 con 0x7fa828106520 2024-09-16T00:08:03.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.167+0000 7fa827fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 0x7fa828106920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:03.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.167+0000 7fa827fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 0x7fa828106920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54726/0 (socket says 172.21.15.33:54726) 2024-09-16T00:08:03.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.167+0000 7fa827fff640 1 -- 172.21.15.33:0/3599753670 learned_addr learned my addr 172.21.15.33:0/3599753670 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:03.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.167+0000 7fa827fff640 1 -- 172.21.15.33:0/3599753670 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa828107720 con 0x7fa828106520 2024-09-16T00:08:03.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.168+0000 7fa827fff640 1 --2- 172.21.15.33:0/3599753670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 0x7fa828106920 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa818009820 tx=0x7fa81802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2e59c76757cc39 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.168+0000 7fa826ffd640 1 -- 172.21.15.33:0/3599753670 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa818037070 con 0x7fa828106520 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.168+0000 7fa826ffd640 1 -- 172.21.15.33:0/3599753670 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa818032920 con 0x7fa828106520 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.168+0000 7fa826ffd640 1 -- 172.21.15.33:0/3599753670 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa818030810 con 0x7fa828106520 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.168+0000 7fa82e374640 1 -- 172.21.15.33:0/3599753670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 msgr2=0x7fa828106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.168+0000 7fa82e374640 1 --2- 172.21.15.33:0/3599753670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 0x7fa828106920 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa818009820 tx=0x7fa81802a270 comp rx=0 tx=0).stop 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.169+0000 7fa82e374640 1 -- 172.21.15.33:0/3599753670 shutdown_connections 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.169+0000 7fa82e374640 1 --2- 172.21.15.33:0/3599753670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 0x7fa828106920 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.169+0000 7fa82e374640 1 -- 172.21.15.33:0/3599753670 >> 172.21.15.33:0/3599753670 conn(0x7fa828101db0 msgr2=0x7fa8281041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.169+0000 7fa82e374640 1 -- 172.21.15.33:0/3599753670 shutdown_connections 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.169+0000 7fa82e374640 1 -- 172.21.15.33:0/3599753670 wait complete. 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.169+0000 7fa82e374640 1 Processor -- start 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.169+0000 7fa82e374640 1 -- start start 2024-09-16T00:08:03.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.170+0000 7fa82e374640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 0x7fa82807ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.170+0000 7fa82e374640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa82810aad0 con 0x7fa828106520 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.170+0000 7fa827fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 0x7fa82807ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.170+0000 7fa827fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 0x7fa82807ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54740/0 (socket says 172.21.15.33:54740) 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.170+0000 7fa827fff640 1 -- 172.21.15.33:0/3432238806 learned_addr learned my addr 172.21.15.33:0/3432238806 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.170+0000 7fa827fff640 1 -- 172.21.15.33:0/3432238806 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82819e390 con 0x7fa828106520 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.170+0000 7fa827fff640 1 --2- 172.21.15.33:0/3432238806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 0x7fa82807ba50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa818030560 tx=0x7fa818032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.170+0000 7fa8257fa640 1 -- 172.21.15.33:0/3432238806 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa818037040 con 0x7fa828106520 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.170+0000 7fa8257fa640 1 -- 172.21.15.33:0/3432238806 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa818040070 con 0x7fa828106520 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.170+0000 7fa82e374640 1 -- 172.21.15.33:0/3432238806 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82819e560 con 0x7fa828106520 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.170+0000 7fa8257fa640 1 -- 172.21.15.33:0/3432238806 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa81803a740 con 0x7fa828106520 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.170+0000 7fa82e374640 1 -- 172.21.15.33:0/3432238806 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa82819e900 con 0x7fa828106520 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.171+0000 7fa8257fa640 1 -- 172.21.15.33:0/3432238806 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fa81803a9b0 con 0x7fa828106520 2024-09-16T00:08:03.358 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.171+0000 7fa8257fa640 1 --2- 172.21.15.33:0/3432238806 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fa8100391b0 0x7fa81003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:03.359 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.172+0000 7fa8257fa640 1 -- 172.21.15.33:0/3432238806 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa81806cad0 con 0x7fa828106520 2024-09-16T00:08:03.359 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.172+0000 7fa82e374640 1 -- 172.21.15.33:0/3432238806 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8281069a0 con 0x7fa828106520 2024-09-16T00:08:03.359 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.172+0000 7fa8277fe640 1 --2- 172.21.15.33:0/3432238806 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fa8100391b0 0x7fa81003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:03.359 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.172+0000 7fa8277fe640 1 --2- 172.21.15.33:0/3432238806 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fa8100391b0 0x7fa81003b670 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0098c0 tx=0x7fa81c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:03.359 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.177+0000 7fa8257fa640 1 -- 172.21.15.33:0/3432238806 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa81806ce20 con 0x7fa828106520 2024-09-16T00:08:03.359 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.400+0000 7fa82e374640 1 -- 172.21.15.33:0/3432238806 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fa82819efd0 con 0x7fa828106520 2024-09-16T00:08:03.359 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.424+0000 7fa8257fa640 1 -- 172.21.15.33:0/3432238806 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fa818041270 con 0x7fa828106520 2024-09-16T00:08:03.359 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.426+0000 7fa8257fa640 1 -- 172.21.15.33:0/3432238806 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7fa818045980 con 0x7fa828106520 2024-09-16T00:08:03.359 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.427+0000 7fa82e374640 1 -- 172.21.15.33:0/3432238806 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fa8100391b0 msgr2=0x7fa81003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:03.359 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.427+0000 7fa82e374640 1 --2- 172.21.15.33:0/3432238806 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fa8100391b0 0x7fa81003b670 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0098c0 tx=0x7fa81c006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:03.359 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.427+0000 7fa82e374640 1 -- 172.21.15.33:0/3432238806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 msgr2=0x7fa82807ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:03.360 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.427+0000 7fa82e374640 1 --2- 172.21.15.33:0/3432238806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 0x7fa82807ba50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa818030560 tx=0x7fa818032670 comp rx=0 tx=0).stop 2024-09-16T00:08:03.360 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.427+0000 7fa82e374640 1 -- 172.21.15.33:0/3432238806 shutdown_connections 2024-09-16T00:08:03.360 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.427+0000 7fa82e374640 1 --2- 172.21.15.33:0/3432238806 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fa8100391b0 0x7fa81003b670 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:03.360 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.427+0000 7fa82e374640 1 --2- 172.21.15.33:0/3432238806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa828106520 0x7fa82807ba50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:03.360 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.427+0000 7fa82e374640 1 -- 172.21.15.33:0/3432238806 >> 172.21.15.33:0/3432238806 conn(0x7fa828101db0 msgr2=0x7fa8281028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:03.360 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.427+0000 7fa82e374640 1 -- 172.21.15.33:0/3432238806 shutdown_connections 2024-09-16T00:08:03.360 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:02.427+0000 7fa82e374640 1 -- 172.21.15.33:0/3432238806 wait complete. 2024-09-16T00:08:03.599 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:03 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3432238806' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-16T00:08:03.599 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:03 smithi033 ceph-mon[31243]: mgrmap e9: x(active, since 18s) 2024-09-16T00:08:03.600 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:03 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:03.509+0000 7f75bbc64040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T00:08:03.600 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:03 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:03.576+0000 7f75bbc64040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T00:08:04.006 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:03 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:03.774+0000 7f75bbc64040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T00:08:04.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-09-16T00:08:04.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-16T00:08:04.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T00:08:04.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-16T00:08:04.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-16T00:08:04.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-09-16T00:08:04.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.797+0000 7fd722b91640 1 Processor -- start 2024-09-16T00:08:04.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.797+0000 7fd722b91640 1 -- start start 2024-09-16T00:08:04.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.797+0000 7fd722b91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 0x7fd71c10a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:04.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.797+0000 7fd722b91640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c07beb0 con 0x7fd71c108090 2024-09-16T00:08:04.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.797+0000 7fd72092d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 0x7fd71c10a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:04.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.797+0000 7fd72092d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 0x7fd71c10a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54782/0 (socket says 172.21.15.33:54782) 2024-09-16T00:08:04.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.797+0000 7fd72092d640 1 -- 172.21.15.33:0/2074773035 learned_addr learned my addr 172.21.15.33:0/2074773035 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:04.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.797+0000 7fd72092d640 1 -- 172.21.15.33:0/2074773035 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71c10a9c0 con 0x7fd71c108090 2024-09-16T00:08:04.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.798+0000 7fd72092d640 1 --2- 172.21.15.33:0/2074773035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 0x7fd71c10a480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd718009820 tx=0x7fd71802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54703029d13dcf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:04.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.798+0000 7fd7137fe640 1 -- 172.21.15.33:0/2074773035 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd718037070 con 0x7fd71c108090 2024-09-16T00:08:04.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.798+0000 7fd7137fe640 1 -- 172.21.15.33:0/2074773035 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd718032920 con 0x7fd71c108090 2024-09-16T00:08:04.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.799+0000 7fd7137fe640 1 -- 172.21.15.33:0/2074773035 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd718030810 con 0x7fd71c108090 2024-09-16T00:08:04.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.799+0000 7fd722b91640 1 -- 172.21.15.33:0/2074773035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 msgr2=0x7fd71c10a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:04.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.799+0000 7fd722b91640 1 --2- 172.21.15.33:0/2074773035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 0x7fd71c10a480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd718009820 tx=0x7fd71802a270 comp rx=0 tx=0).stop 2024-09-16T00:08:04.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.799+0000 7fd722b91640 1 -- 172.21.15.33:0/2074773035 shutdown_connections 2024-09-16T00:08:04.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.799+0000 7fd722b91640 1 --2- 172.21.15.33:0/2074773035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 0x7fd71c10a480 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:04.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.799+0000 7fd722b91640 1 -- 172.21.15.33:0/2074773035 >> 172.21.15.33:0/2074773035 conn(0x7fd71c101db0 msgr2=0x7fd71c1041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:04.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.799+0000 7fd722b91640 1 -- 172.21.15.33:0/2074773035 shutdown_connections 2024-09-16T00:08:04.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.799+0000 7fd722b91640 1 -- 172.21.15.33:0/2074773035 wait complete. 2024-09-16T00:08:04.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.800+0000 7fd722b91640 1 Processor -- start 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.800+0000 7fd722b91640 1 -- start start 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.800+0000 7fd722b91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 0x7fd71c19b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.800+0000 7fd722b91640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c10baa0 con 0x7fd71c108090 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.801+0000 7fd72092d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 0x7fd71c19b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.801+0000 7fd72092d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 0x7fd71c19b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54798/0 (socket says 172.21.15.33:54798) 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.801+0000 7fd72092d640 1 -- 172.21.15.33:0/1407612306 learned_addr learned my addr 172.21.15.33:0/1407612306 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.801+0000 7fd72092d640 1 -- 172.21.15.33:0/1407612306 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71c19bb70 con 0x7fd71c108090 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.801+0000 7fd72092d640 1 --2- 172.21.15.33:0/1407612306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 0x7fd71c19b630 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd7180097f0 tx=0x7fd7180327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.801+0000 7fd711ffb640 1 -- 172.21.15.33:0/1407612306 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd718037040 con 0x7fd71c108090 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.801+0000 7fd722b91640 1 -- 172.21.15.33:0/1407612306 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71c19be00 con 0x7fd71c108090 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.801+0000 7fd711ffb640 1 -- 172.21.15.33:0/1407612306 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd718030cc0 con 0x7fd71c108090 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.801+0000 7fd711ffb640 1 -- 172.21.15.33:0/1407612306 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd71803a6b0 con 0x7fd71c108090 2024-09-16T00:08:04.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.802+0000 7fd722b91640 1 -- 172.21.15.33:0/1407612306 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd71c19eaf0 con 0x7fd71c108090 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.802+0000 7fd711ffb640 1 -- 172.21.15.33:0/1407612306 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7fd71803a850 con 0x7fd71c108090 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.803+0000 7fd711ffb640 1 --2- 172.21.15.33:0/1407612306 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd704039260 0x7fd70403b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.803+0000 7fd711ffb640 1 -- 172.21.15.33:0/1407612306 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd71806c7f0 con 0x7fd71c108090 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.803+0000 7fd713fff640 1 -- 172.21.15.33:0/1407612306 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd704039260 msgr2=0x7fd70403b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/122116489 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.803+0000 7fd713fff640 1 --2- 172.21.15.33:0/1407612306 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd704039260 0x7fd70403b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.803+0000 7fd722b91640 1 -- 172.21.15.33:0/1407612306 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd71c1084f0 con 0x7fd71c108090 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:03.808+0000 7fd711ffb640 1 -- 172.21.15.33:0/1407612306 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd718042300 con 0x7fd71c108090 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.004+0000 7fd713fff640 1 -- 172.21.15.33:0/1407612306 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd704039260 msgr2=0x7fd70403b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/122116489 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.004+0000 7fd713fff640 1 --2- 172.21.15.33:0/1407612306 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd704039260 0x7fd70403b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.064+0000 7fd722b91640 1 -- 172.21.15.33:0/1407612306 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fd71c19efb0 con 0x7fd71c108090 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.064+0000 7fd711ffb640 1 -- 172.21.15.33:0/1407612306 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fd718049200 con 0x7fd71c108090 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.067+0000 7fd722b91640 1 -- 172.21.15.33:0/1407612306 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd704039260 msgr2=0x7fd70403b720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.067+0000 7fd722b91640 1 --2- 172.21.15.33:0/1407612306 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd704039260 0x7fd70403b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:04.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.067+0000 7fd722b91640 1 -- 172.21.15.33:0/1407612306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 msgr2=0x7fd71c19b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:04.635 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.067+0000 7fd722b91640 1 --2- 172.21.15.33:0/1407612306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 0x7fd71c19b630 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd7180097f0 tx=0x7fd7180327b0 comp rx=0 tx=0).stop 2024-09-16T00:08:04.635 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.067+0000 7fd722b91640 1 -- 172.21.15.33:0/1407612306 shutdown_connections 2024-09-16T00:08:04.635 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.067+0000 7fd722b91640 1 --2- 172.21.15.33:0/1407612306 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7fd704039260 0x7fd70403b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:04.635 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.067+0000 7fd722b91640 1 --2- 172.21.15.33:0/1407612306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd71c108090 0x7fd71c19b630 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:04.635 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.067+0000 7fd722b91640 1 -- 172.21.15.33:0/1407612306 >> 172.21.15.33:0/1407612306 conn(0x7fd71c101db0 msgr2=0x7fd71c1028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:04.635 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.067+0000 7fd722b91640 1 -- 172.21.15.33:0/1407612306 shutdown_connections 2024-09-16T00:08:04.635 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:04.067+0000 7fd722b91640 1 -- 172.21.15.33:0/1407612306 wait complete. 2024-09-16T00:08:04.635 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for the mgr to restart... 2024-09-16T00:08:04.635 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mgr epoch 9... 2024-09-16T00:08:04.730 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:04 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1407612306' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-16T00:08:05.039 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:04 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:04.750+0000 7f75bbc64040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T00:08:05.039 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:04 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:04.821+0000 7f75bbc64040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T00:08:05.852 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:05 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:05.579+0000 7f75bbc64040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T00:08:05.852 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:05 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:05.700+0000 7f75bbc64040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T00:08:06.143 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:05 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:05.850+0000 7f75bbc64040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T00:08:06.143 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:05 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:05.914+0000 7f75bbc64040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T00:08:06.143 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:06.070+0000 7f75bbc64040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T00:08:06.143 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:06.141+0000 7f75bbc64040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T00:08:06.439 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:06.204+0000 7f75bbc64040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T00:08:06.439 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:06.369+0000 7f75bbc64040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T00:08:06.440 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:06.437+0000 7f75bbc64040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T00:08:06.719 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:06.507+0000 7f75bbc64040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T00:08:06.719 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:06.594+0000 7f75bbc64040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T00:08:06.719 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:06.656+0000 7f75bbc64040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T00:08:06.719 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:06.717+0000 7f75bbc64040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T00:08:07.798 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:07.545+0000 7f75bbc64040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T00:08:07.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: Active manager daemon x restarted 2024-09-16T00:08:07.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: Activating manager daemon x 2024-09-16T00:08:07.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: osdmap e3: 0 total, 0 up, 0 in 2024-09-16T00:08:07.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: mgrmap e10: x(active, starting, since 0.00331331s) 2024-09-16T00:08:07.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:08:07.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-16T00:08:07.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-16T00:08:07.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-16T00:08:07.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-16T00:08:07.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: Manager daemon x is now available 2024-09-16T00:08:07.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:08:07.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-16T00:08:07.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-16T00:08:09.456 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-09-16T00:08:09.456 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-16T00:08:09.456 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-16T00:08:09.456 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-09-16T00:08:09.456 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.036+0000 7f18f82ef640 1 Processor -- start 2024-09-16T00:08:09.456 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.037+0000 7f18f82ef640 1 -- start start 2024-09-16T00:08:09.456 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.037+0000 7f18f82ef640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 0x7f18f010a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:09.456 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.037+0000 7f18f82ef640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18f007be00 con 0x7f18f01080e0 2024-09-16T00:08:09.456 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.037+0000 7f18f608b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 0x7f18f010a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:09.456 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.037+0000 7f18f608b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 0x7f18f010a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54812/0 (socket says 172.21.15.33:54812) 2024-09-16T00:08:09.457 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.037+0000 7f18f608b640 1 -- 172.21.15.33:0/1437356489 learned_addr learned my addr 172.21.15.33:0/1437356489 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:09.457 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.037+0000 7f18f608b640 1 -- 172.21.15.33:0/1437356489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f010aa10 con 0x7f18f01080e0 2024-09-16T00:08:09.457 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.038+0000 7f18f608b640 1 --2- 172.21.15.33:0/1437356489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 0x7f18f010a4d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009820 tx=0x7f18ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eecea79cc78e8b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:09.457 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.038+0000 7f18f5089640 1 -- 172.21.15.33:0/1437356489 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18ec037070 con 0x7f18f01080e0 2024-09-16T00:08:09.457 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.038+0000 7f18f5089640 1 -- 172.21.15.33:0/1437356489 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18ec032920 con 0x7f18f01080e0 2024-09-16T00:08:09.457 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.038+0000 7f18f5089640 1 -- 172.21.15.33:0/1437356489 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18ec030810 con 0x7f18f01080e0 2024-09-16T00:08:09.457 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.039+0000 7f18f82ef640 1 -- 172.21.15.33:0/1437356489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 msgr2=0x7f18f010a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:09.457 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.039+0000 7f18f82ef640 1 --2- 172.21.15.33:0/1437356489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 0x7f18f010a4d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009820 tx=0x7f18ec02a270 comp rx=0 tx=0).stop 2024-09-16T00:08:09.457 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.040+0000 7f18f82ef640 1 -- 172.21.15.33:0/1437356489 shutdown_connections 2024-09-16T00:08:09.457 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.040+0000 7f18f82ef640 1 --2- 172.21.15.33:0/1437356489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 0x7f18f010a4d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:09.457 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.040+0000 7f18f82ef640 1 -- 172.21.15.33:0/1437356489 >> 172.21.15.33:0/1437356489 conn(0x7f18f0101ca0 msgr2=0x7f18f01040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:09.457 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.040+0000 7f18f82ef640 1 -- 172.21.15.33:0/1437356489 shutdown_connections 2024-09-16T00:08:09.458 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.040+0000 7f18f82ef640 1 -- 172.21.15.33:0/1437356489 wait complete. 2024-09-16T00:08:09.458 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.041+0000 7f18f82ef640 1 Processor -- start 2024-09-16T00:08:09.458 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.041+0000 7f18f82ef640 1 -- start start 2024-09-16T00:08:09.458 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.041+0000 7f18f82ef640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 0x7f18f019b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:09.458 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.041+0000 7f18f82ef640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18f010ba30 con 0x7f18f01080e0 2024-09-16T00:08:09.458 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.041+0000 7f18f608b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 0x7f18f019b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:09.458 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.041+0000 7f18f608b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 0x7f18f019b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54828/0 (socket says 172.21.15.33:54828) 2024-09-16T00:08:09.458 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.041+0000 7f18f608b640 1 -- 172.21.15.33:0/1483330259 learned_addr learned my addr 172.21.15.33:0/1483330259 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:09.458 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.042+0000 7f18f608b640 1 -- 172.21.15.33:0/1483330259 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f019bb10 con 0x7f18f01080e0 2024-09-16T00:08:09.458 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.042+0000 7f18f608b640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 0x7f18f019b5d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0097f0 tx=0x7f18ec0327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:09.458 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.042+0000 7f18e77fe640 1 -- 172.21.15.33:0/1483330259 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18ec037040 con 0x7f18f01080e0 2024-09-16T00:08:09.458 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.042+0000 7f18e77fe640 1 -- 172.21.15.33:0/1483330259 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18ec030cc0 con 0x7f18f01080e0 2024-09-16T00:08:09.459 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.042+0000 7f18f82ef640 1 -- 172.21.15.33:0/1483330259 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f019bda0 con 0x7f18f01080e0 2024-09-16T00:08:09.459 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.042+0000 7f18e77fe640 1 -- 172.21.15.33:0/1483330259 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18ec03a6b0 con 0x7f18f01080e0 2024-09-16T00:08:09.459 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.042+0000 7f18f82ef640 1 -- 172.21.15.33:0/1483330259 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f019ea90 con 0x7f18f01080e0 2024-09-16T00:08:09.459 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.043+0000 7f18e77fe640 1 -- 172.21.15.33:0/1483330259 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7f18ec03baf0 con 0x7f18f01080e0 2024-09-16T00:08:09.459 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.044+0000 7f18e77fe640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f18dc039200 0x7f18dc03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:09.459 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.044+0000 7f18e77fe640 1 -- 172.21.15.33:0/1483330259 --> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f18dc03bdd0 con 0x7f18dc039200 2024-09-16T00:08:09.459 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.044+0000 7f18e77fe640 1 -- 172.21.15.33:0/1483330259 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f18ec06c860 con 0x7f18f01080e0 2024-09-16T00:08:09.459 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.044+0000 7f18f588a640 1 -- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f18dc039200 msgr2=0x7f18dc03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/122116489 2024-09-16T00:08:09.459 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.044+0000 7f18f588a640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f18dc039200 0x7f18dc03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T00:08:09.459 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.245+0000 7f18f588a640 1 -- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f18dc039200 msgr2=0x7f18dc03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/122116489 2024-09-16T00:08:09.459 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.245+0000 7f18f588a640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f18dc039200 0x7f18dc03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T00:08:09.459 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.646+0000 7f18f588a640 1 -- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f18dc039200 msgr2=0x7f18dc03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/122116489 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:05.646+0000 7f18f588a640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f18dc039200 0x7f18dc03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:06.447+0000 7f18f588a640 1 -- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f18dc039200 msgr2=0x7f18dc03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/122116489 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:06.447+0000 7f18f588a640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f18dc039200 0x7f18dc03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:07.549+0000 7f18e77fe640 1 -- 172.21.15.33:0/1483330259 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mgrmap(e 10) ==== 46241+0+0 (secure 0 0 0) 0x7f18ec06b310 con 0x7f18f01080e0 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:07.549+0000 7f18e77fe640 1 -- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f18dc039200 msgr2=0x7f18dc03b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:07.549+0000 7f18e77fe640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f18dc039200 0x7f18dc03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.550+0000 7f18e77fe640 1 -- 172.21.15.33:0/1483330259 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7f18ec03b3f0 con 0x7f18f01080e0 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.550+0000 7f18e77fe640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18dc03ccc0 0x7f18dc03f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.550+0000 7f18e77fe640 1 -- 172.21.15.33:0/1483330259 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f18ec03ce70 con 0x7f18dc03ccc0 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.551+0000 7f18f588a640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18dc03ccc0 0x7f18dc03f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.552+0000 7f18f588a640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18dc03ccc0 0x7f18dc03f0b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f18e0003980 tx=0x7f18e0006e10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.552+0000 7f18e77fe640 1 -- 172.21.15.33:0/1483330259 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f18ec03ce70 con 0x7f18dc03ccc0 2024-09-16T00:08:09.460 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18f82ef640 1 -- 172.21.15.33:0/1483330259 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f18f0108540 con 0x7f18dc03ccc0 2024-09-16T00:08:09.461 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18e77fe640 1 -- 172.21.15.33:0/1483330259 <== mgr.14150 v2:172.21.15.33:6800/3427860355 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f18f0108540 con 0x7f18dc03ccc0 2024-09-16T00:08:09.461 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18f82ef640 1 -- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18dc03ccc0 msgr2=0x7f18dc03f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:09.461 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18f82ef640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18dc03ccc0 0x7f18dc03f0b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f18e0003980 tx=0x7f18e0006e10 comp rx=0 tx=0).stop 2024-09-16T00:08:09.461 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18f82ef640 1 -- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 msgr2=0x7f18f019b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:09.461 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18f82ef640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 0x7f18f019b5d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0097f0 tx=0x7f18ec0327b0 comp rx=0 tx=0).stop 2024-09-16T00:08:09.461 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18f82ef640 1 -- 172.21.15.33:0/1483330259 shutdown_connections 2024-09-16T00:08:09.464 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18f82ef640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18dc03ccc0 0x7f18dc03f0b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:09.464 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18f82ef640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:6800/122116489,v1:172.21.15.33:6801/122116489] conn(0x7f18dc039200 0x7f18dc03b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:09.464 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18f82ef640 1 --2- 172.21.15.33:0/1483330259 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f01080e0 0x7f18f019b5d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:09.464 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18f82ef640 1 -- 172.21.15.33:0/1483330259 >> 172.21.15.33:0/1483330259 conn(0x7f18f0101ca0 msgr2=0x7f18f0102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:09.464 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18f82ef640 1 -- 172.21.15.33:0/1483330259 shutdown_connections 2024-09-16T00:08:09.464 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:08.560+0000 7f18f82ef640 1 -- 172.21.15.33:0/1483330259 wait complete. 2024-09-16T00:08:09.464 INFO:teuthology.orchestra.run.smithi033.stdout:mgr epoch 9 is available 2024-09-16T00:08:09.464 INFO:teuthology.orchestra.run.smithi033.stdout:Generating a dashboard self-signed certificate... 2024-09-16T00:08:09.725 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-mon[31243]: mgrmap e11: x(active, since 1.00634s) 2024-09-16T00:08:09.725 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:09.725 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-mon[31243]: [16/Sep/2024:00:08:09] ENGINE Bus STARTING 2024-09-16T00:08:09.725 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-mon[31243]: [16/Sep/2024:00:08:09] ENGINE Serving on https://172.21.15.33:7150 2024-09-16T00:08:09.725 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-mon[31243]: [16/Sep/2024:00:08:09] ENGINE Error in HTTPServer.serve 2024-09-16T00:08:09.725 INFO:journalctl@ceph.mon.a.smithi033.stdout: Traceback (most recent call last): 2024-09-16T00:08:09.725 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T00:08:09.725 INFO:journalctl@ceph.mon.a.smithi033.stdout: self._connections.run(self.expiration_interval) 2024-09-16T00:08:09.725 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T00:08:09.725 INFO:journalctl@ceph.mon.a.smithi033.stdout: self._run(expiration_interval) 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: s = self.context.wrap_socket( 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: return self.sslsocket_class._create( 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: self.do_handshake() 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: self._sslobj.do_handshake() 2024-09-16T00:08:09.726 INFO:journalctl@ceph.mon.a.smithi033.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T00:08:09.727 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:09.460+0000 7f7526f78640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:00:08:09] ENGINE Error in HTTPServer.serve 2024-09-16T00:08:09.727 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: Traceback (most recent call last): 2024-09-16T00:08:09.727 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T00:08:09.727 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: self._connections.run(self.expiration_interval) 2024-09-16T00:08:09.727 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T00:08:09.727 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: self._run(expiration_interval) 2024-09-16T00:08:09.727 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T00:08:09.727 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T00:08:09.727 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T00:08:09.727 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T00:08:09.728 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T00:08:09.728 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: s = self.context.wrap_socket( 2024-09-16T00:08:09.728 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T00:08:09.728 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: return self.sslsocket_class._create( 2024-09-16T00:08:09.728 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T00:08:09.728 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: self.do_handshake() 2024-09-16T00:08:09.728 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T00:08:09.728 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: self._sslobj.do_handshake() 2024-09-16T00:08:09.728 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T00:08:09.728 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:08:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:08:10.845 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-16T00:08:10.845 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.914+0000 7fb9737b4640 1 Processor -- start 2024-09-16T00:08:10.845 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.914+0000 7fb9737b4640 1 -- start start 2024-09-16T00:08:10.845 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.915+0000 7fb9737b4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 0x7fb96c10a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:10.845 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.915+0000 7fb9737b4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c07be30 con 0x7fb96c1080f0 2024-09-16T00:08:10.845 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.915+0000 7fb971550640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 0x7fb96c10a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:10.845 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.915+0000 7fb971550640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 0x7fb96c10a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54904/0 (socket says 172.21.15.33:54904) 2024-09-16T00:08:10.846 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.915+0000 7fb971550640 1 -- 172.21.15.33:0/1943052891 learned_addr learned my addr 172.21.15.33:0/1943052891 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:10.846 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.915+0000 7fb971550640 1 -- 172.21.15.33:0/1943052891 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96c10aa20 con 0x7fb96c1080f0 2024-09-16T00:08:10.846 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.916+0000 7fb971550640 1 --2- 172.21.15.33:0/1943052891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 0x7fb96c10a4e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb960009e80 tx=0x7fb96002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f749770e1334da4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:10.846 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.916+0000 7fb95ffff640 1 -- 172.21.15.33:0/1943052891 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb960036070 con 0x7fb96c1080f0 2024-09-16T00:08:10.846 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.916+0000 7fb95ffff640 1 -- 172.21.15.33:0/1943052891 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9600318e0 con 0x7fb96c1080f0 2024-09-16T00:08:10.846 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.916+0000 7fb95ffff640 1 -- 172.21.15.33:0/1943052891 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb960007a00 con 0x7fb96c1080f0 2024-09-16T00:08:10.846 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.917+0000 7fb9737b4640 1 -- 172.21.15.33:0/1943052891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 msgr2=0x7fb96c10a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:10.846 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.917+0000 7fb9737b4640 1 --2- 172.21.15.33:0/1943052891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 0x7fb96c10a4e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb960009e80 tx=0x7fb96002a7a0 comp rx=0 tx=0).stop 2024-09-16T00:08:10.846 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.917+0000 7fb9737b4640 1 -- 172.21.15.33:0/1943052891 shutdown_connections 2024-09-16T00:08:10.846 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.917+0000 7fb9737b4640 1 --2- 172.21.15.33:0/1943052891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 0x7fb96c10a4e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:10.846 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.917+0000 7fb9737b4640 1 -- 172.21.15.33:0/1943052891 >> 172.21.15.33:0/1943052891 conn(0x7fb96c101d50 msgr2=0x7fb96c104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.917+0000 7fb9737b4640 1 -- 172.21.15.33:0/1943052891 shutdown_connections 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.917+0000 7fb9737b4640 1 -- 172.21.15.33:0/1943052891 wait complete. 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.918+0000 7fb9737b4640 1 Processor -- start 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.918+0000 7fb9737b4640 1 -- start start 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.918+0000 7fb9737b4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 0x7fb96c19b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.918+0000 7fb9737b4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c10ba40 con 0x7fb96c1080f0 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.918+0000 7fb971550640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 0x7fb96c19b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.918+0000 7fb971550640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 0x7fb96c19b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54906/0 (socket says 172.21.15.33:54906) 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.918+0000 7fb971550640 1 -- 172.21.15.33:0/3904692836 learned_addr learned my addr 172.21.15.33:0/3904692836 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.918+0000 7fb971550640 1 -- 172.21.15.33:0/3904692836 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96c19bb70 con 0x7fb96c1080f0 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.918+0000 7fb971550640 1 --2- 172.21.15.33:0/3904692836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 0x7fb96c19b630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb96002acc0 tx=0x7fb960031cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.919+0000 7fb95e7fc640 1 -- 172.21.15.33:0/3904692836 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb960036050 con 0x7fb96c1080f0 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.919+0000 7fb95e7fc640 1 -- 172.21.15.33:0/3904692836 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb960037040 con 0x7fb96c1080f0 2024-09-16T00:08:10.847 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.919+0000 7fb9737b4640 1 -- 172.21.15.33:0/3904692836 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb96c19be00 con 0x7fb96c1080f0 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.919+0000 7fb95e7fc640 1 -- 172.21.15.33:0/3904692836 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb96003d8e0 con 0x7fb96c1080f0 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.919+0000 7fb9737b4640 1 -- 172.21.15.33:0/3904692836 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb96c19eaf0 con 0x7fb96c1080f0 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.919+0000 7fb95e7fc640 1 -- 172.21.15.33:0/3904692836 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7fb960044020 con 0x7fb96c1080f0 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.920+0000 7fb9737b4640 1 -- 172.21.15.33:0/3904692836 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb96c108550 con 0x7fb96c1080f0 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.920+0000 7fb95e7fc640 1 --2- 172.21.15.33:0/3904692836 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb958039140 0x7fb95803b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.920+0000 7fb95e7fc640 1 -- 172.21.15.33:0/3904692836 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb960042020 con 0x7fb96c1080f0 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.920+0000 7fb970d4f640 1 --2- 172.21.15.33:0/3904692836 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb958039140 0x7fb95803b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.920+0000 7fb970d4f640 1 --2- 172.21.15.33:0/3904692836 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb958039140 0x7fb95803b600 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb9680098c0 tx=0x7fb968006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:09.925+0000 7fb95e7fc640 1 -- 172.21.15.33:0/3904692836 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb96004b020 con 0x7fb96c1080f0 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.042+0000 7fb95e7fc640 1 -- 172.21.15.33:0/3904692836 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7fb96004bcc0 con 0x7fb96c1080f0 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.109+0000 7fb9737b4640 1 -- 172.21.15.33:0/3904692836 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fb96c102fc0 con 0x7fb958039140 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.206+0000 7fb95e7fc640 1 -- 172.21.15.33:0/3904692836 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fb96c102fc0 con 0x7fb958039140 2024-09-16T00:08:10.848 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.208+0000 7fb9737b4640 1 -- 172.21.15.33:0/3904692836 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb958039140 msgr2=0x7fb95803b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:10.849 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.208+0000 7fb9737b4640 1 --2- 172.21.15.33:0/3904692836 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb958039140 0x7fb95803b600 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb9680098c0 tx=0x7fb968006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:10.849 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.208+0000 7fb9737b4640 1 -- 172.21.15.33:0/3904692836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 msgr2=0x7fb96c19b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:10.849 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.208+0000 7fb9737b4640 1 --2- 172.21.15.33:0/3904692836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 0x7fb96c19b630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb96002acc0 tx=0x7fb960031cc0 comp rx=0 tx=0).stop 2024-09-16T00:08:10.849 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.209+0000 7fb9737b4640 1 -- 172.21.15.33:0/3904692836 shutdown_connections 2024-09-16T00:08:10.849 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.209+0000 7fb9737b4640 1 --2- 172.21.15.33:0/3904692836 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb958039140 0x7fb95803b600 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:10.849 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.209+0000 7fb9737b4640 1 --2- 172.21.15.33:0/3904692836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb96c1080f0 0x7fb96c19b630 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:10.849 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.209+0000 7fb9737b4640 1 -- 172.21.15.33:0/3904692836 >> 172.21.15.33:0/3904692836 conn(0x7fb96c101d50 msgr2=0x7fb96c1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:10.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.209+0000 7fb9737b4640 1 -- 172.21.15.33:0/3904692836 shutdown_connections 2024-09-16T00:08:10.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:10.209+0000 7fb9737b4640 1 -- 172.21.15.33:0/3904692836 wait complete. 2024-09-16T00:08:10.850 INFO:teuthology.orchestra.run.smithi033.stdout:Creating initial admin user... 2024-09-16T00:08:11.181 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:10 smithi033 ceph-mon[31243]: [16/Sep/2024:00:08:09] ENGINE Serving on http://172.21.15.33:8765 2024-09-16T00:08:11.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:10 smithi033 ceph-mon[31243]: [16/Sep/2024:00:08:09] ENGINE Bus STARTED 2024-09-16T00:08:11.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:10 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:11.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:10 smithi033 ceph-mon[31243]: mgrmap e12: x(active, since 2s) 2024-09-16T00:08:11.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:10 smithi033 ceph-mon[31243]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:08:11.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:10 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:11.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:10 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:12.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:11 smithi033 ceph-mon[31243]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:08:12.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:11 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:12.915 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Vke2wxPvQdBLVaKUUgMRKe48buZ870GwPR5FoORa.2MfvLpme5PTS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726445291, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-16T00:08:12.915 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.275+0000 7fa284683640 1 Processor -- start 2024-09-16T00:08:12.915 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.275+0000 7fa284683640 1 -- start start 2024-09-16T00:08:12.915 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.275+0000 7fa284683640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 0x7fa27c107b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:12.915 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.275+0000 7fa284683640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa27c108110 con 0x7fa27c107740 2024-09-16T00:08:12.915 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.275+0000 7fa28241f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 0x7fa27c107b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:12.915 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.275+0000 7fa28241f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 0x7fa27c107b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53126/0 (socket says 172.21.15.33:53126) 2024-09-16T00:08:12.915 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.275+0000 7fa28241f640 1 -- 172.21.15.33:0/1313514106 learned_addr learned my addr 172.21.15.33:0/1313514106 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:12.915 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.276+0000 7fa28241f640 1 -- 172.21.15.33:0/1313514106 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27c1088a0 con 0x7fa27c107740 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.276+0000 7fa28241f640 1 --2- 172.21.15.33:0/1313514106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 0x7fa27c107b40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa278009820 tx=0x7fa27802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c166b2d6f536cc5f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.277+0000 7fa28141d640 1 -- 172.21.15.33:0/1313514106 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa278037070 con 0x7fa27c107740 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.277+0000 7fa28141d640 1 -- 172.21.15.33:0/1313514106 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa278032920 con 0x7fa27c107740 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.277+0000 7fa28141d640 1 -- 172.21.15.33:0/1313514106 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa278030810 con 0x7fa27c107740 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.277+0000 7fa284683640 1 -- 172.21.15.33:0/1313514106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 msgr2=0x7fa27c107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.277+0000 7fa284683640 1 --2- 172.21.15.33:0/1313514106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 0x7fa27c107b40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa278009820 tx=0x7fa27802a270 comp rx=0 tx=0).stop 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.277+0000 7fa284683640 1 -- 172.21.15.33:0/1313514106 shutdown_connections 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.277+0000 7fa284683640 1 --2- 172.21.15.33:0/1313514106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 0x7fa27c107b40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.277+0000 7fa284683640 1 -- 172.21.15.33:0/1313514106 >> 172.21.15.33:0/1313514106 conn(0x7fa27c102eb0 msgr2=0x7fa27c105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.277+0000 7fa284683640 1 -- 172.21.15.33:0/1313514106 shutdown_connections 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.277+0000 7fa284683640 1 -- 172.21.15.33:0/1313514106 wait complete. 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.278+0000 7fa284683640 1 Processor -- start 2024-09-16T00:08:12.916 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.278+0000 7fa284683640 1 -- start start 2024-09-16T00:08:12.917 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.278+0000 7fa284683640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 0x7fa27c07db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:12.917 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.278+0000 7fa284683640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa27c10bc50 con 0x7fa27c107740 2024-09-16T00:08:12.917 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.278+0000 7fa28241f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 0x7fa27c07db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:12.917 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.278+0000 7fa28241f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 0x7fa27c07db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53142/0 (socket says 172.21.15.33:53142) 2024-09-16T00:08:12.917 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.278+0000 7fa28241f640 1 -- 172.21.15.33:0/3780946438 learned_addr learned my addr 172.21.15.33:0/3780946438 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:12.917 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.279+0000 7fa28241f640 1 -- 172.21.15.33:0/3780946438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27c07e0b0 con 0x7fa27c107740 2024-09-16T00:08:12.917 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.279+0000 7fa28241f640 1 --2- 172.21.15.33:0/3780946438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 0x7fa27c07db70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa278030560 tx=0x7fa278032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:12.917 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.279+0000 7fa2737fe640 1 -- 172.21.15.33:0/3780946438 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa278037040 con 0x7fa27c107740 2024-09-16T00:08:12.917 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.279+0000 7fa2737fe640 1 -- 172.21.15.33:0/3780946438 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa278040070 con 0x7fa27c107740 2024-09-16T00:08:12.917 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.279+0000 7fa284683640 1 -- 172.21.15.33:0/3780946438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa27c07c2a0 con 0x7fa27c107740 2024-09-16T00:08:12.917 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.279+0000 7fa2737fe640 1 -- 172.21.15.33:0/3780946438 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa27803a6d0 con 0x7fa27c107740 2024-09-16T00:08:12.917 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.279+0000 7fa284683640 1 -- 172.21.15.33:0/3780946438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa27c07c7e0 con 0x7fa27c107740 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.280+0000 7fa2737fe640 1 -- 172.21.15.33:0/3780946438 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7fa27803bb20 con 0x7fa27c107740 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.280+0000 7fa2737fe640 1 --2- 172.21.15.33:0/3780946438 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa268039200 0x7fa26803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.280+0000 7fa284683640 1 -- 172.21.15.33:0/3780946438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa27c107bc0 con 0x7fa27c107740 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.280+0000 7fa2737fe640 1 -- 172.21.15.33:0/3780946438 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa27806c920 con 0x7fa27c107740 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.280+0000 7fa281c1e640 1 --2- 172.21.15.33:0/3780946438 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa268039200 0x7fa26803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.281+0000 7fa281c1e640 1 --2- 172.21.15.33:0/3780946438 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa268039200 0x7fa26803b6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0098c0 tx=0x7fa26c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.285+0000 7fa2737fe640 1 -- 172.21.15.33:0/3780946438 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa2780328f0 con 0x7fa27c107740 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.489+0000 7fa284683640 1 -- 172.21.15.33:0/3780946438 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fa27c07cf50 con 0x7fa268039200 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.738+0000 7fa2737fe640 1 -- 172.21.15.33:0/3780946438 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fa27c07cf50 con 0x7fa268039200 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.740+0000 7fa284683640 1 -- 172.21.15.33:0/3780946438 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa268039200 msgr2=0x7fa26803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.741+0000 7fa284683640 1 --2- 172.21.15.33:0/3780946438 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa268039200 0x7fa26803b6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0098c0 tx=0x7fa26c006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.741+0000 7fa284683640 1 -- 172.21.15.33:0/3780946438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 msgr2=0x7fa27c07db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.741+0000 7fa284683640 1 --2- 172.21.15.33:0/3780946438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 0x7fa27c07db70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa278030560 tx=0x7fa278032670 comp rx=0 tx=0).stop 2024-09-16T00:08:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.741+0000 7fa284683640 1 -- 172.21.15.33:0/3780946438 shutdown_connections 2024-09-16T00:08:12.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.741+0000 7fa284683640 1 --2- 172.21.15.33:0/3780946438 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa268039200 0x7fa26803b6c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:12.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.741+0000 7fa284683640 1 --2- 172.21.15.33:0/3780946438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa27c107740 0x7fa27c07db70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:12.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.741+0000 7fa284683640 1 -- 172.21.15.33:0/3780946438 >> 172.21.15.33:0/3780946438 conn(0x7fa27c102eb0 msgr2=0x7fa27c103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:12.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.741+0000 7fa284683640 1 -- 172.21.15.33:0/3780946438 shutdown_connections 2024-09-16T00:08:12.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:11.741+0000 7fa284683640 1 -- 172.21.15.33:0/3780946438 wait complete. 2024-09-16T00:08:12.919 INFO:teuthology.orchestra.run.smithi033.stdout:Fetching dashboard port number... 2024-09-16T00:08:13.788 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:13 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1689236018' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-16T00:08:14.054 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 8443 2024-09-16T00:08:14.054 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.308+0000 7f2943b8a640 1 Processor -- start 2024-09-16T00:08:14.054 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.309+0000 7f2943b8a640 1 -- start start 2024-09-16T00:08:14.054 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.309+0000 7f2943b8a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 0x7f293c07c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.309+0000 7f2943b8a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f293c07c860 con 0x7f293c07de20 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.309+0000 7f2941926640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 0x7f293c07c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.309+0000 7f2941926640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 0x7f293c07c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53158/0 (socket says 172.21.15.33:53158) 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.309+0000 7f2941926640 1 -- 172.21.15.33:0/922856937 learned_addr learned my addr 172.21.15.33:0/922856937 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.309+0000 7f2941926640 1 -- 172.21.15.33:0/922856937 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293c07c9e0 con 0x7f293c07de20 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.310+0000 7f2941926640 1 --2- 172.21.15.33:0/922856937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 0x7f293c07c320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2938009820 tx=0x7f293802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85def289c890cd1b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.310+0000 7f2940924640 1 -- 172.21.15.33:0/922856937 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2938037070 con 0x7f293c07de20 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.310+0000 7f2940924640 1 -- 172.21.15.33:0/922856937 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2938032920 con 0x7f293c07de20 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.310+0000 7f2940924640 1 -- 172.21.15.33:0/922856937 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2938030810 con 0x7f293c07de20 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.311+0000 7f2943b8a640 1 -- 172.21.15.33:0/922856937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 msgr2=0x7f293c07c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.311+0000 7f2943b8a640 1 --2- 172.21.15.33:0/922856937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 0x7f293c07c320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2938009820 tx=0x7f293802a270 comp rx=0 tx=0).stop 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.311+0000 7f2943b8a640 1 -- 172.21.15.33:0/922856937 shutdown_connections 2024-09-16T00:08:14.055 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.311+0000 7f2943b8a640 1 --2- 172.21.15.33:0/922856937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 0x7f293c07c320 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:14.056 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.311+0000 7f2943b8a640 1 -- 172.21.15.33:0/922856937 >> 172.21.15.33:0/922856937 conn(0x7f293c102eb0 msgr2=0x7f293c105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:14.056 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.311+0000 7f2943b8a640 1 -- 172.21.15.33:0/922856937 shutdown_connections 2024-09-16T00:08:14.056 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.311+0000 7f2943b8a640 1 -- 172.21.15.33:0/922856937 wait complete. 2024-09-16T00:08:14.056 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.311+0000 7f2943b8a640 1 Processor -- start 2024-09-16T00:08:14.056 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.312+0000 7f2943b8a640 1 -- start start 2024-09-16T00:08:14.056 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.312+0000 7f2943b8a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 0x7f293c19d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:14.056 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.312+0000 7f2943b8a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f293c10dcc0 con 0x7f293c07de20 2024-09-16T00:08:14.056 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.312+0000 7f2941926640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 0x7f293c19d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:14.056 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.312+0000 7f2941926640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 0x7f293c19d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53174/0 (socket says 172.21.15.33:53174) 2024-09-16T00:08:14.056 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.312+0000 7f2941926640 1 -- 172.21.15.33:0/1689236018 learned_addr learned my addr 172.21.15.33:0/1689236018 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:14.056 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.312+0000 7f2941926640 1 -- 172.21.15.33:0/1689236018 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293c19de00 con 0x7f293c07de20 2024-09-16T00:08:14.056 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.312+0000 7f2941926640 1 --2- 172.21.15.33:0/1689236018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 0x7f293c19d8c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2938030420 tx=0x7f2938032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.312+0000 7f2932ffd640 1 -- 172.21.15.33:0/1689236018 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2938037040 con 0x7f293c07de20 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.312+0000 7f2932ffd640 1 -- 172.21.15.33:0/1689236018 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2938040070 con 0x7f293c07de20 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.313+0000 7f2932ffd640 1 -- 172.21.15.33:0/1689236018 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f293803a6e0 con 0x7f293c07de20 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.313+0000 7f2943b8a640 1 -- 172.21.15.33:0/1689236018 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f293c19e090 con 0x7f293c07de20 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.313+0000 7f2943b8a640 1 -- 172.21.15.33:0/1689236018 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f293c1a0d80 con 0x7f293c07de20 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.313+0000 7f2932ffd640 1 -- 172.21.15.33:0/1689236018 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7f293803bb20 con 0x7f293c07de20 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.314+0000 7f2943b8a640 1 -- 172.21.15.33:0/1689236018 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f293c07c3a0 con 0x7f293c07de20 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.314+0000 7f2932ffd640 1 --2- 172.21.15.33:0/1689236018 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2928039260 0x7f292803b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.314+0000 7f2932ffd640 1 -- 172.21.15.33:0/1689236018 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f293806c990 con 0x7f293c07de20 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.314+0000 7f2941125640 1 --2- 172.21.15.33:0/1689236018 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2928039260 0x7f292803b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.314+0000 7f2941125640 1 --2- 172.21.15.33:0/1689236018 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2928039260 0x7f292803b720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f292c0098c0 tx=0x7f292c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.318+0000 7f2932ffd640 1 -- 172.21.15.33:0/1689236018 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2938043360 con 0x7f293c07de20 2024-09-16T00:08:14.057 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.496+0000 7f2943b8a640 1 -- 172.21.15.33:0/1689236018 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f293c1a0f50 con 0x7f293c07de20 2024-09-16T00:08:14.058 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.497+0000 7f2932ffd640 1 -- 172.21.15.33:0/1689236018 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f2938040220 con 0x7f293c07de20 2024-09-16T00:08:14.058 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.499+0000 7f2943b8a640 1 -- 172.21.15.33:0/1689236018 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2928039260 msgr2=0x7f292803b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:14.058 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.499+0000 7f2943b8a640 1 --2- 172.21.15.33:0/1689236018 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2928039260 0x7f292803b720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f292c0098c0 tx=0x7f292c006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:14.058 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.499+0000 7f2943b8a640 1 -- 172.21.15.33:0/1689236018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 msgr2=0x7f293c19d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:14.058 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.499+0000 7f2943b8a640 1 --2- 172.21.15.33:0/1689236018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 0x7f293c19d8c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2938030420 tx=0x7f2938032670 comp rx=0 tx=0).stop 2024-09-16T00:08:14.058 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.499+0000 7f2943b8a640 1 -- 172.21.15.33:0/1689236018 shutdown_connections 2024-09-16T00:08:14.058 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.499+0000 7f2943b8a640 1 --2- 172.21.15.33:0/1689236018 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2928039260 0x7f292803b720 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:14.058 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.499+0000 7f2943b8a640 1 --2- 172.21.15.33:0/1689236018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f293c07de20 0x7f293c19d8c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:14.058 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.499+0000 7f2943b8a640 1 -- 172.21.15.33:0/1689236018 >> 172.21.15.33:0/1689236018 conn(0x7f293c102eb0 msgr2=0x7f293c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:14.058 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.499+0000 7f2943b8a640 1 -- 172.21.15.33:0/1689236018 shutdown_connections 2024-09-16T00:08:14.058 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:13.499+0000 7f2943b8a640 1 -- 172.21.15.33:0/1689236018 wait complete. 2024-09-16T00:08:14.062 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T00:08:14.062 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout disabled 2024-09-16T00:08:14.070 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T00:08:14.070 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-09-16T00:08:14.070 INFO:teuthology.orchestra.run.smithi033.stdout:firewalld.service is not enabled 2024-09-16T00:08:14.070 INFO:teuthology.orchestra.run.smithi033.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-16T00:08:14.072 INFO:teuthology.orchestra.run.smithi033.stdout:Ceph Dashboard is now available at: 2024-09-16T00:08:14.072 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:08:14.072 INFO:teuthology.orchestra.run.smithi033.stdout: URL: https://smithi033.front.sepia.ceph.com:8443/ 2024-09-16T00:08:14.072 INFO:teuthology.orchestra.run.smithi033.stdout: User: admin 2024-09-16T00:08:14.072 INFO:teuthology.orchestra.run.smithi033.stdout: Password: kwf1h8476b 2024-09-16T00:08:14.072 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:08:14.073 INFO:teuthology.orchestra.run.smithi033.stdout:Saving cluster configuration to /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config directory 2024-09-16T00:08:15.311 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.481+0000 7f2ee61fc640 1 Processor -- start 2024-09-16T00:08:15.311 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.481+0000 7f2ee61fc640 1 -- start start 2024-09-16T00:08:15.311 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.481+0000 7f2ee61fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 0x7f2ee0107a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:15.311 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.481+0000 7f2ee61fc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee0107fa0 con 0x7f2ee0107660 2024-09-16T00:08:15.311 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.482+0000 7f2edf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 0x7f2ee0107a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:15.311 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.482+0000 7f2edf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 0x7f2ee0107a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53188/0 (socket says 172.21.15.33:53188) 2024-09-16T00:08:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.482+0000 7f2edf7fe640 1 -- 172.21.15.33:0/3788239227 learned_addr learned my addr 172.21.15.33:0/3788239227 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.482+0000 7f2edf7fe640 1 -- 172.21.15.33:0/3788239227 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee0108120 con 0x7f2ee0107660 2024-09-16T00:08:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.483+0000 7f2edf7fe640 1 --2- 172.21.15.33:0/3788239227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 0x7f2ee0107a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009e80 tx=0x7f2ed002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b620afe560df57 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.483+0000 7f2ede7fc640 1 -- 172.21.15.33:0/3788239227 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ed0036070 con 0x7f2ee0107660 2024-09-16T00:08:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.483+0000 7f2ede7fc640 1 -- 172.21.15.33:0/3788239227 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ed00318e0 con 0x7f2ee0107660 2024-09-16T00:08:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.483+0000 7f2ede7fc640 1 -- 172.21.15.33:0/3788239227 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ed0007a00 con 0x7f2ee0107660 2024-09-16T00:08:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.483+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3788239227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 msgr2=0x7f2ee0107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.483+0000 7f2ee61fc640 1 --2- 172.21.15.33:0/3788239227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 0x7f2ee0107a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009e80 tx=0x7f2ed002a7a0 comp rx=0 tx=0).stop 2024-09-16T00:08:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.483+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3788239227 shutdown_connections 2024-09-16T00:08:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.483+0000 7f2ee61fc640 1 --2- 172.21.15.33:0/3788239227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 0x7f2ee0107a60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.483+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3788239227 >> 172.21.15.33:0/3788239227 conn(0x7f2ee0102eb0 msgr2=0x7f2ee01052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.484+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3788239227 shutdown_connections 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.484+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3788239227 wait complete. 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.484+0000 7f2ee61fc640 1 Processor -- start 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.484+0000 7f2ee61fc640 1 -- start start 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.485+0000 7f2ee61fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 0x7f2ee007db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.485+0000 7f2ee61fc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee010bad0 con 0x7f2ee0107660 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.485+0000 7f2edf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 0x7f2ee007db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.485+0000 7f2edf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 0x7f2ee007db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53198/0 (socket says 172.21.15.33:53198) 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.485+0000 7f2edf7fe640 1 -- 172.21.15.33:0/3488278043 learned_addr learned my addr 172.21.15.33:0/3488278043 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.485+0000 7f2edf7fe640 1 -- 172.21.15.33:0/3488278043 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee007e0b0 con 0x7f2ee0107660 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.485+0000 7f2edf7fe640 1 --2- 172.21.15.33:0/3488278043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 0x7f2ee007db70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0031cf0 tx=0x7f2ed0031d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.485+0000 7f2edcff9640 1 -- 172.21.15.33:0/3488278043 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ed0036050 con 0x7f2ee0107660 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.485+0000 7f2edcff9640 1 -- 172.21.15.33:0/3488278043 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ed0042070 con 0x7f2ee0107660 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.485+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3488278043 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ee007c2a0 con 0x7f2ee0107660 2024-09-16T00:08:15.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.485+0000 7f2edcff9640 1 -- 172.21.15.33:0/3488278043 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ed003d9b0 con 0x7f2ee0107660 2024-09-16T00:08:15.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.485+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3488278043 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ee007c7e0 con 0x7f2ee0107660 2024-09-16T00:08:15.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.486+0000 7f2edcff9640 1 -- 172.21.15.33:0/3488278043 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7f2ed003dc60 con 0x7f2ee0107660 2024-09-16T00:08:15.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.486+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3488278043 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ee0107ae0 con 0x7f2ee0107660 2024-09-16T00:08:15.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.487+0000 7f2edcff9640 1 --2- 172.21.15.33:0/3488278043 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2ecc039200 0x7f2ecc03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:15.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.487+0000 7f2edcff9640 1 -- 172.21.15.33:0/3488278043 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2ed006d920 con 0x7f2ee0107660 2024-09-16T00:08:15.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.487+0000 7f2edeffd640 1 --2- 172.21.15.33:0/3488278043 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2ecc039200 0x7f2ecc03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:15.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.487+0000 7f2edeffd640 1 --2- 172.21.15.33:0/3488278043 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2ecc039200 0x7f2ecc03b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80098c0 tx=0x7f2ec8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:15.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.492+0000 7f2edcff9640 1 -- 172.21.15.33:0/3488278043 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2ed0044310 con 0x7f2ee0107660 2024-09-16T00:08:15.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.550+0000 7f2edcff9640 1 -- 172.21.15.33:0/3488278043 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f2ed003bcf0 con 0x7f2ee0107660 2024-09-16T00:08:15.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.761+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3488278043 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f2ee007cf80 con 0x7f2ee0107660 2024-09-16T00:08:15.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.764+0000 7f2edcff9640 1 -- 172.21.15.33:0/3488278043 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) ==== 153+0+0 (secure 0 0 0) 0x7f2ed0045520 con 0x7f2ee0107660 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.766+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3488278043 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2ecc039200 msgr2=0x7f2ecc03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.766+0000 7f2ee61fc640 1 --2- 172.21.15.33:0/3488278043 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2ecc039200 0x7f2ecc03b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80098c0 tx=0x7f2ec8006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.766+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3488278043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 msgr2=0x7f2ee007db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.766+0000 7f2ee61fc640 1 --2- 172.21.15.33:0/3488278043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 0x7f2ee007db70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0031cf0 tx=0x7f2ed0031d20 comp rx=0 tx=0).stop 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.766+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3488278043 shutdown_connections 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.766+0000 7f2ee61fc640 1 --2- 172.21.15.33:0/3488278043 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2ecc039200 0x7f2ecc03b6c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.766+0000 7f2ee61fc640 1 --2- 172.21.15.33:0/3488278043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ee0107660 0x7f2ee007db70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.766+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3488278043 >> 172.21.15.33:0/3488278043 conn(0x7f2ee0102eb0 msgr2=0x7f2ee0103a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.766+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3488278043 shutdown_connections 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-09-16T00:08:14.766+0000 7f2ee61fc640 1 -- 172.21.15.33:0/3488278043 wait complete. 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-16T00:08:15.315 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout:Or, if you are only running a single cluster on this host: 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout: ceph telemetry on 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout:For more information see: 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-16T00:08:15.316 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:08:15.317 INFO:teuthology.orchestra.run.smithi033.stdout:Bootstrap complete. 2024-09-16T00:08:15.350 INFO:tasks.cephadm:Fetching config... 2024-09-16T00:08:15.350 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:08:15.350 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-16T00:08:15.367 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-16T00:08:15.368 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:08:15.368 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-16T00:08:15.423 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-16T00:08:15.424 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:08:15.424 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-16T00:08:15.489 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-16T00:08:15.489 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:08:15.489 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-16T00:08:15.545 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-16T00:08:15.545 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWLo+DVeRhGl4fpRXR9AdTann1eHkovQGVToV8Fb6GqW7ytfdWTo9JOIYTpAq7QVOZt+qjkQq9OEQMevdx2ZT0vexbU0ghavH0YlpUrciPQpggk4ROJPfugp+BvDfji8LQ12QaPKUclnp6O9gMLOMM+gKjixw7fh7FQv+/oIJFC2wtWnqGmbFedICLiVyeh2tDmDyCb5Q4L1rnD8wqc/AniiC2kfBD7nN29IR47LhTv71hnhCJmxJX3I46LX/fW1xqXAfLaTIsY2gwpSDasWqgsHlXR3EwUNYl/rqDQhdLtdSGp9fks3nN+rxLbMPgQYoOjKZ0ekxQ5sGbGaeTqjaP8g68FyeitSU7r9U7xKfu+MG2oTALiUp72VxS/FQSAt4kHMkmIk5JI/wKmghBlM0SXzX7aXITXydNL4S1xU88oj9E2wxuq0EM8tTIBKPZOdzUzjMfUoixwKpp2Mxkt7digFv9mwF1QAxvsDJV07ycIkF2tEzbT3oqOItlnffkAtk= ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T00:08:15.600 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:15 smithi033 ceph-mon[31243]: mgrmap e13: x(active, since 7s) 2024-09-16T00:08:15.600 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:15 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3488278043' entity='client.admin' 2024-09-16T00:08:15.623 INFO:teuthology.orchestra.run.smithi033.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWLo+DVeRhGl4fpRXR9AdTann1eHkovQGVToV8Fb6GqW7ytfdWTo9JOIYTpAq7QVOZt+qjkQq9OEQMevdx2ZT0vexbU0ghavH0YlpUrciPQpggk4ROJPfugp+BvDfji8LQ12QaPKUclnp6O9gMLOMM+gKjixw7fh7FQv+/oIJFC2wtWnqGmbFedICLiVyeh2tDmDyCb5Q4L1rnD8wqc/AniiC2kfBD7nN29IR47LhTv71hnhCJmxJX3I46LX/fW1xqXAfLaTIsY2gwpSDasWqgsHlXR3EwUNYl/rqDQhdLtdSGp9fks3nN+rxLbMPgQYoOjKZ0ekxQ5sGbGaeTqjaP8g68FyeitSU7r9U7xKfu+MG2oTALiUp72VxS/FQSAt4kHMkmIk5JI/wKmghBlM0SXzX7aXITXydNL4S1xU88oj9E2wxuq0EM8tTIBKPZOdzUzjMfUoixwKpp2Mxkt7digFv9mwF1QAxvsDJV07ycIkF2tEzbT3oqOItlnffkAtk= ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:15.635 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWLo+DVeRhGl4fpRXR9AdTann1eHkovQGVToV8Fb6GqW7ytfdWTo9JOIYTpAq7QVOZt+qjkQq9OEQMevdx2ZT0vexbU0ghavH0YlpUrciPQpggk4ROJPfugp+BvDfji8LQ12QaPKUclnp6O9gMLOMM+gKjixw7fh7FQv+/oIJFC2wtWnqGmbFedICLiVyeh2tDmDyCb5Q4L1rnD8wqc/AniiC2kfBD7nN29IR47LhTv71hnhCJmxJX3I46LX/fW1xqXAfLaTIsY2gwpSDasWqgsHlXR3EwUNYl/rqDQhdLtdSGp9fks3nN+rxLbMPgQYoOjKZ0ekxQ5sGbGaeTqjaP8g68FyeitSU7r9U7xKfu+MG2oTALiUp72VxS/FQSAt4kHMkmIk5JI/wKmghBlM0SXzX7aXITXydNL4S1xU88oj9E2wxuq0EM8tTIBKPZOdzUzjMfUoixwKpp2Mxkt7digFv9mwF1QAxvsDJV07ycIkF2tEzbT3oqOItlnffkAtk= ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T00:08:15.676 INFO:teuthology.orchestra.run.smithi060.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWLo+DVeRhGl4fpRXR9AdTann1eHkovQGVToV8Fb6GqW7ytfdWTo9JOIYTpAq7QVOZt+qjkQq9OEQMevdx2ZT0vexbU0ghavH0YlpUrciPQpggk4ROJPfugp+BvDfji8LQ12QaPKUclnp6O9gMLOMM+gKjixw7fh7FQv+/oIJFC2wtWnqGmbFedICLiVyeh2tDmDyCb5Q4L1rnD8wqc/AniiC2kfBD7nN29IR47LhTv71hnhCJmxJX3I46LX/fW1xqXAfLaTIsY2gwpSDasWqgsHlXR3EwUNYl/rqDQhdLtdSGp9fks3nN+rxLbMPgQYoOjKZ0ekxQ5sGbGaeTqjaP8g68FyeitSU7r9U7xKfu+MG2oTALiUp72VxS/FQSAt4kHMkmIk5JI/wKmghBlM0SXzX7aXITXydNL4S1xU88oj9E2wxuq0EM8tTIBKPZOdzUzjMfUoixwKpp2Mxkt7digFv9mwF1QAxvsDJV07ycIkF2tEzbT3oqOItlnffkAtk= ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:15.689 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWLo+DVeRhGl4fpRXR9AdTann1eHkovQGVToV8Fb6GqW7ytfdWTo9JOIYTpAq7QVOZt+qjkQq9OEQMevdx2ZT0vexbU0ghavH0YlpUrciPQpggk4ROJPfugp+BvDfji8LQ12QaPKUclnp6O9gMLOMM+gKjixw7fh7FQv+/oIJFC2wtWnqGmbFedICLiVyeh2tDmDyCb5Q4L1rnD8wqc/AniiC2kfBD7nN29IR47LhTv71hnhCJmxJX3I46LX/fW1xqXAfLaTIsY2gwpSDasWqgsHlXR3EwUNYl/rqDQhdLtdSGp9fks3nN+rxLbMPgQYoOjKZ0ekxQ5sGbGaeTqjaP8g68FyeitSU7r9U7xKfu+MG2oTALiUp72VxS/FQSAt4kHMkmIk5JI/wKmghBlM0SXzX7aXITXydNL4S1xU88oj9E2wxuq0EM8tTIBKPZOdzUzjMfUoixwKpp2Mxkt7digFv9mwF1QAxvsDJV07ycIkF2tEzbT3oqOItlnffkAtk= ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T00:08:15.732 INFO:teuthology.orchestra.run.smithi083.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWLo+DVeRhGl4fpRXR9AdTann1eHkovQGVToV8Fb6GqW7ytfdWTo9JOIYTpAq7QVOZt+qjkQq9OEQMevdx2ZT0vexbU0ghavH0YlpUrciPQpggk4ROJPfugp+BvDfji8LQ12QaPKUclnp6O9gMLOMM+gKjixw7fh7FQv+/oIJFC2wtWnqGmbFedICLiVyeh2tDmDyCb5Q4L1rnD8wqc/AniiC2kfBD7nN29IR47LhTv71hnhCJmxJX3I46LX/fW1xqXAfLaTIsY2gwpSDasWqgsHlXR3EwUNYl/rqDQhdLtdSGp9fks3nN+rxLbMPgQYoOjKZ0ekxQ5sGbGaeTqjaP8g68FyeitSU7r9U7xKfu+MG2oTALiUp72VxS/FQSAt4kHMkmIk5JI/wKmghBlM0SXzX7aXITXydNL4S1xU88oj9E2wxuq0EM8tTIBKPZOdzUzjMfUoixwKpp2Mxkt7digFv9mwF1QAxvsDJV07ycIkF2tEzbT3oqOItlnffkAtk= ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:15.745 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-16T00:08:15.997 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:08:16.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.507+0000 7f00dede7640 1 -- 172.21.15.33:0/1908425510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d80ff900 msgr2=0x7f00d80ffd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:16.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.507+0000 7f00dede7640 1 --2- 172.21.15.33:0/1908425510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d80ff900 0x7f00d80ffd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009880 tx=0x7f00cc02a5d0 comp rx=0 tx=0).stop 2024-09-16T00:08:16.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.507+0000 7f00dede7640 1 -- 172.21.15.33:0/1908425510 shutdown_connections 2024-09-16T00:08:16.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.507+0000 7f00dede7640 1 --2- 172.21.15.33:0/1908425510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d80ff900 0x7f00d80ffd00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:16.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.507+0000 7f00dede7640 1 -- 172.21.15.33:0/1908425510 >> 172.21.15.33:0/1908425510 conn(0x7f00d80fb450 msgr2=0x7f00d80fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:16.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.507+0000 7f00dede7640 1 -- 172.21.15.33:0/1908425510 shutdown_connections 2024-09-16T00:08:16.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.507+0000 7f00dede7640 1 -- 172.21.15.33:0/1908425510 wait complete. 2024-09-16T00:08:16.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.508+0000 7f00dede7640 1 Processor -- start 2024-09-16T00:08:16.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.508+0000 7f00dede7640 1 -- start start 2024-09-16T00:08:16.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.508+0000 7f00dede7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d80ff900 0x7f00d8102160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:16.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.508+0000 7f00dede7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00d81076e0 con 0x7f00d80ff900 2024-09-16T00:08:16.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.508+0000 7f00dcb83640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d80ff900 0x7f00d8102160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:16.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.508+0000 7f00dcb83640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d80ff900 0x7f00d8102160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53206/0 (socket says 172.21.15.33:53206) 2024-09-16T00:08:16.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.508+0000 7f00dcb83640 1 -- 172.21.15.33:0/170796572 learned_addr learned my addr 172.21.15.33:0/170796572 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:16.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.508+0000 7f00dcb83640 1 -- 172.21.15.33:0/170796572 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d81026a0 con 0x7f00d80ff900 2024-09-16T00:08:16.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.508+0000 7f00dcb83640 1 --2- 172.21.15.33:0/170796572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d80ff900 0x7f00d8102160 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f00cc004f60 tx=0x7f00cc0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:16.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.509+0000 7f00d5ffb640 1 -- 172.21.15.33:0/170796572 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00cc03f070 con 0x7f00d80ff900 2024-09-16T00:08:16.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.509+0000 7f00d5ffb640 1 -- 172.21.15.33:0/170796572 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00cc004d40 con 0x7f00d80ff900 2024-09-16T00:08:16.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.509+0000 7f00dede7640 1 -- 172.21.15.33:0/170796572 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d8100890 con 0x7f00d80ff900 2024-09-16T00:08:16.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.509+0000 7f00d5ffb640 1 -- 172.21.15.33:0/170796572 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00cc035030 con 0x7f00d80ff900 2024-09-16T00:08:16.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.509+0000 7f00dede7640 1 -- 172.21.15.33:0/170796572 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d8100dd0 con 0x7f00d80ff900 2024-09-16T00:08:16.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.510+0000 7f00d5ffb640 1 -- 172.21.15.33:0/170796572 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f00cc042070 con 0x7f00d80ff900 2024-09-16T00:08:16.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.510+0000 7f00d5ffb640 1 --2- 172.21.15.33:0/170796572 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00c80392a0 0x7f00c803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:16.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.510+0000 7f00dede7640 1 -- 172.21.15.33:0/170796572 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d8107e60 con 0x7f00d80ff900 2024-09-16T00:08:16.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.510+0000 7f00d5ffb640 1 -- 172.21.15.33:0/170796572 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f00cc06d030 con 0x7f00d80ff900 2024-09-16T00:08:16.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.511+0000 7f00d7fff640 1 --2- 172.21.15.33:0/170796572 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00c80392a0 0x7f00c803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:16.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.512+0000 7f00d7fff640 1 --2- 172.21.15.33:0/170796572 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00c80392a0 0x7f00c803b760 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f00c40098c0 tx=0x7f00c4006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:16.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.515+0000 7f00d5ffb640 1 -- 172.21.15.33:0/170796572 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f00cc039350 con 0x7f00d80ff900 2024-09-16T00:08:16.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.693+0000 7f00dede7640 1 -- 172.21.15.33:0/170796572 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f00d8190150 con 0x7f00d80ff900 2024-09-16T00:08:16.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.695+0000 7f00d5ffb640 1 -- 172.21.15.33:0/170796572 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f00cc02f300 con 0x7f00d80ff900 2024-09-16T00:08:16.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.698+0000 7f00dede7640 1 -- 172.21.15.33:0/170796572 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00c80392a0 msgr2=0x7f00c803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:16.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.698+0000 7f00dede7640 1 --2- 172.21.15.33:0/170796572 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00c80392a0 0x7f00c803b760 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f00c40098c0 tx=0x7f00c4006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:16.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.698+0000 7f00dede7640 1 -- 172.21.15.33:0/170796572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d80ff900 msgr2=0x7f00d8102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:16.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.698+0000 7f00dede7640 1 --2- 172.21.15.33:0/170796572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d80ff900 0x7f00d8102160 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f00cc004f60 tx=0x7f00cc0048f0 comp rx=0 tx=0).stop 2024-09-16T00:08:16.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.698+0000 7f00dede7640 1 -- 172.21.15.33:0/170796572 shutdown_connections 2024-09-16T00:08:16.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.698+0000 7f00dede7640 1 --2- 172.21.15.33:0/170796572 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00c80392a0 0x7f00c803b760 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:16.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.698+0000 7f00dede7640 1 --2- 172.21.15.33:0/170796572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d80ff900 0x7f00d8102160 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:16.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.698+0000 7f00dede7640 1 -- 172.21.15.33:0/170796572 >> 172.21.15.33:0/170796572 conn(0x7f00d80fb450 msgr2=0x7f00d80fbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:16.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.698+0000 7f00dede7640 1 -- 172.21.15.33:0/170796572 shutdown_connections 2024-09-16T00:08:16.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:16.698+0000 7f00dede7640 1 -- 172.21.15.33:0/170796572 wait complete. 2024-09-16T00:08:17.693 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-16T00:08:17.694 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-16T00:08:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:17 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/170796572' entity='client.admin' 2024-09-16T00:08:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:08:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:08:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:17.943 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:08:18.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.331+0000 7f26abe37640 1 -- 172.21.15.33:0/3497054912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26a4100510 msgr2=0x7f26a4100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:18.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.331+0000 7f26abe37640 1 --2- 172.21.15.33:0/3497054912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26a4100510 0x7f26a4100910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009880 tx=0x7f26a002a5d0 comp rx=0 tx=0).stop 2024-09-16T00:08:18.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.331+0000 7f26abe37640 1 -- 172.21.15.33:0/3497054912 shutdown_connections 2024-09-16T00:08:18.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.331+0000 7f26abe37640 1 --2- 172.21.15.33:0/3497054912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26a4100510 0x7f26a4100910 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:18.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.331+0000 7f26abe37640 1 -- 172.21.15.33:0/3497054912 >> 172.21.15.33:0/3497054912 conn(0x7f26a40fbc80 msgr2=0x7f26a40fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:18.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.331+0000 7f26abe37640 1 -- 172.21.15.33:0/3497054912 shutdown_connections 2024-09-16T00:08:18.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.331+0000 7f26abe37640 1 -- 172.21.15.33:0/3497054912 wait complete. 2024-09-16T00:08:18.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.331+0000 7f26abe37640 1 Processor -- start 2024-09-16T00:08:18.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.332+0000 7f26abe37640 1 -- start start 2024-09-16T00:08:18.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.332+0000 7f26abe37640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26a4100510 0x7f26a4196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:18.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.332+0000 7f26abe37640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26a4106620 con 0x7f26a4100510 2024-09-16T00:08:18.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.332+0000 7f26a9bd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26a4100510 0x7f26a4196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:18.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.332+0000 7f26a9bd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26a4100510 0x7f26a4196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53222/0 (socket says 172.21.15.33:53222) 2024-09-16T00:08:18.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.332+0000 7f26a9bd3640 1 -- 172.21.15.33:0/4255993982 learned_addr learned my addr 172.21.15.33:0/4255993982 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:18.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.332+0000 7f26a9bd3640 1 -- 172.21.15.33:0/4255993982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a41972e0 con 0x7f26a4100510 2024-09-16T00:08:18.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.332+0000 7f26a9bd3640 1 --2- 172.21.15.33:0/4255993982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26a4100510 0x7f26a4196da0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f26a0004f60 tx=0x7f26a00048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:18.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.332+0000 7f269affd640 1 -- 172.21.15.33:0/4255993982 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26a003f070 con 0x7f26a4100510 2024-09-16T00:08:18.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.333+0000 7f269affd640 1 -- 172.21.15.33:0/4255993982 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26a0004d40 con 0x7f26a4100510 2024-09-16T00:08:18.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.333+0000 7f26abe37640 1 -- 172.21.15.33:0/4255993982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a4197570 con 0x7f26a4100510 2024-09-16T00:08:18.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.333+0000 7f269affd640 1 -- 172.21.15.33:0/4255993982 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26a0035030 con 0x7f26a4100510 2024-09-16T00:08:18.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.333+0000 7f26abe37640 1 -- 172.21.15.33:0/4255993982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a419a220 con 0x7f26a4100510 2024-09-16T00:08:18.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.333+0000 7f269affd640 1 -- 172.21.15.33:0/4255993982 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f26a0042070 con 0x7f26a4100510 2024-09-16T00:08:18.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.333+0000 7f26abe37640 1 -- 172.21.15.33:0/4255993982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a4106da0 con 0x7f26a4100510 2024-09-16T00:08:18.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.334+0000 7f269affd640 1 --2- 172.21.15.33:0/4255993982 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26900392a0 0x7f269003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:18.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.334+0000 7f269affd640 1 -- 172.21.15.33:0/4255993982 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f26a006cfe0 con 0x7f26a4100510 2024-09-16T00:08:18.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.334+0000 7f26a93d2640 1 --2- 172.21.15.33:0/4255993982 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26900392a0 0x7f269003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:18.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.334+0000 7f26a93d2640 1 --2- 172.21.15.33:0/4255993982 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26900392a0 0x7f269003b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f26940098c0 tx=0x7f2694006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:18.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.338+0000 7f269affd640 1 -- 172.21.15.33:0/4255993982 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f26a0039350 con 0x7f26a4100510 2024-09-16T00:08:18.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.517+0000 7f26abe37640 1 -- 172.21.15.33:0/4255993982 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f26a402d940 con 0x7f26900392a0 2024-09-16T00:08:18.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.520+0000 7f269affd640 1 -- 172.21.15.33:0/4255993982 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f26a402d940 con 0x7f26900392a0 2024-09-16T00:08:18.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.522+0000 7f26abe37640 1 -- 172.21.15.33:0/4255993982 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26900392a0 msgr2=0x7f269003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:18.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.522+0000 7f26abe37640 1 --2- 172.21.15.33:0/4255993982 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26900392a0 0x7f269003b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f26940098c0 tx=0x7f2694006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:18.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.522+0000 7f26abe37640 1 -- 172.21.15.33:0/4255993982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26a4100510 msgr2=0x7f26a4196da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:18.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.522+0000 7f26abe37640 1 --2- 172.21.15.33:0/4255993982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26a4100510 0x7f26a4196da0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f26a0004f60 tx=0x7f26a00048f0 comp rx=0 tx=0).stop 2024-09-16T00:08:18.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.522+0000 7f26abe37640 1 -- 172.21.15.33:0/4255993982 shutdown_connections 2024-09-16T00:08:18.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.522+0000 7f26abe37640 1 --2- 172.21.15.33:0/4255993982 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26900392a0 0x7f269003b760 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:18.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.522+0000 7f26abe37640 1 --2- 172.21.15.33:0/4255993982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26a4100510 0x7f26a4196da0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:18.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.522+0000 7f26abe37640 1 -- 172.21.15.33:0/4255993982 >> 172.21.15.33:0/4255993982 conn(0x7f26a40fbc80 msgr2=0x7f26a406a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.522+0000 7f26abe37640 1 -- 172.21.15.33:0/4255993982 shutdown_connections 2024-09-16T00:08:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:18.522+0000 7f26abe37640 1 -- 172.21.15.33:0/4255993982 wait complete. 2024-09-16T00:08:19.257 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi060 2024-09-16T00:08:19.258 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-16T00:08:19.258 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/etc/ceph/ceph.conf 2024-09-16T00:08:19.277 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-16T00:08:19.277 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T00:08:19.336 INFO:tasks.cephadm:Adding host smithi060 to orchestrator... 2024-09-16T00:08:19.336 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch host add smithi060 2024-09-16T00:08:19.586 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:08:19.788 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:19 smithi033 ceph-mon[31243]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:08:19.788 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:19 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:19.788 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:19 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:08:19.788 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:19 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:19.788 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:19 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:08:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:19 smithi033 ceph-mon[31243]: Updating smithi033:/etc/ceph/ceph.conf 2024-09-16T00:08:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:19 smithi033 ceph-mon[31243]: Updating smithi033:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:19 smithi033 ceph-mon[31243]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-09-16T00:08:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:19 smithi033 ceph-mon[31243]: Updating smithi033:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T00:08:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:19 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:19 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:19 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:19.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.995+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2909793237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfec102750 msgr2=0x7fcfec102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:19.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.995+0000 7fcff2dd1640 1 --2- 172.21.15.33:0/2909793237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfec102750 0x7fcfec102b30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009880 tx=0x7fcfe002a670 comp rx=0 tx=0).stop 2024-09-16T00:08:19.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.995+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2909793237 shutdown_connections 2024-09-16T00:08:19.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.995+0000 7fcff2dd1640 1 --2- 172.21.15.33:0/2909793237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfec102750 0x7fcfec102b30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:19.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.995+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2909793237 >> 172.21.15.33:0/2909793237 conn(0x7fcfec0fc1b0 msgr2=0x7fcfec0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:19.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.995+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2909793237 shutdown_connections 2024-09-16T00:08:19.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.995+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2909793237 wait complete. 2024-09-16T00:08:20.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.996+0000 7fcff2dd1640 1 Processor -- start 2024-09-16T00:08:20.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.996+0000 7fcff2dd1640 1 -- start start 2024-09-16T00:08:20.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.996+0000 7fcff2dd1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfec102750 0x7fcfec199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:20.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.996+0000 7fcff2dd1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec10bcc0 con 0x7fcfec102750 2024-09-16T00:08:20.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.996+0000 7fcff0b6d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfec102750 0x7fcfec199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.996+0000 7fcff0b6d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfec102750 0x7fcfec199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53244/0 (socket says 172.21.15.33:53244) 2024-09-16T00:08:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.996+0000 7fcff0b6d640 1 -- 172.21.15.33:0/2846693205 learned_addr learned my addr 172.21.15.33:0/2846693205 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.996+0000 7fcff0b6d640 1 -- 172.21.15.33:0/2846693205 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfec1998c0 con 0x7fcfec102750 2024-09-16T00:08:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.996+0000 7fcff0b6d640 1 --2- 172.21.15.33:0/2846693205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfec102750 0x7fcfec199380 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009230 tx=0x7fcfe0003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.997+0000 7fcfe9ffb640 1 -- 172.21.15.33:0/2846693205 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfe003f070 con 0x7fcfec102750 2024-09-16T00:08:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.997+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2846693205 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfec199b50 con 0x7fcfec102750 2024-09-16T00:08:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.997+0000 7fcfe9ffb640 1 -- 172.21.15.33:0/2846693205 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfe003bc80 con 0x7fcfec102750 2024-09-16T00:08:20.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.997+0000 7fcfe9ffb640 1 -- 172.21.15.33:0/2846693205 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfe0035030 con 0x7fcfec102750 2024-09-16T00:08:20.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.997+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2846693205 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfec199fb0 con 0x7fcfec102750 2024-09-16T00:08:20.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.998+0000 7fcfe9ffb640 1 -- 172.21.15.33:0/2846693205 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fcfe0042070 con 0x7fcfec102750 2024-09-16T00:08:20.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.998+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2846693205 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfec06a7b0 con 0x7fcfec102750 2024-09-16T00:08:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.998+0000 7fcfe9ffb640 1 --2- 172.21.15.33:0/2846693205 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcfdc0392a0 0x7fcfdc03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.998+0000 7fcfe9ffb640 1 -- 172.21.15.33:0/2846693205 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcfe006cf90 con 0x7fcfec102750 2024-09-16T00:08:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.998+0000 7fcfebfff640 1 --2- 172.21.15.33:0/2846693205 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcfdc0392a0 0x7fcfdc03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:19.999+0000 7fcfebfff640 1 --2- 172.21.15.33:0/2846693205 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcfdc0392a0 0x7fcfdc03b760 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80098c0 tx=0x7fcfd8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:20.003+0000 7fcfe9ffb640 1 -- 172.21.15.33:0/2846693205 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcfe004c350 con 0x7fcfec102750 2024-09-16T00:08:20.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:20.185+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2846693205 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi060", "target": ["mon-mgr", ""]}) -- 0x7fcfec19a6e0 con 0x7fcfdc0392a0 2024-09-16T00:08:20.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:20 smithi033 ceph-mon[31243]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi060", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:08:21.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:21 smithi033 ceph-mon[31243]: Deploying cephadm binary to smithi060 2024-09-16T00:08:24.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:24.076+0000 7fcfe9ffb640 1 -- 172.21.15.33:0/2846693205 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fcfec19a6e0 con 0x7fcfdc0392a0 2024-09-16T00:08:24.078 INFO:teuthology.orchestra.run.smithi033.stdout:Added host 'smithi060' with addr '172.21.15.60' 2024-09-16T00:08:24.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:24.078+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2846693205 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcfdc0392a0 msgr2=0x7fcfdc03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:24.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:24.078+0000 7fcff2dd1640 1 --2- 172.21.15.33:0/2846693205 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcfdc0392a0 0x7fcfdc03b760 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80098c0 tx=0x7fcfd8006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:24.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:24.078+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2846693205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfec102750 msgr2=0x7fcfec199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:24.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:24.078+0000 7fcff2dd1640 1 --2- 172.21.15.33:0/2846693205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfec102750 0x7fcfec199380 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009230 tx=0x7fcfe0003d40 comp rx=0 tx=0).stop 2024-09-16T00:08:24.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:24.079+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2846693205 shutdown_connections 2024-09-16T00:08:24.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:24.079+0000 7fcff2dd1640 1 --2- 172.21.15.33:0/2846693205 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcfdc0392a0 0x7fcfdc03b760 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:24.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:24.079+0000 7fcff2dd1640 1 --2- 172.21.15.33:0/2846693205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfec102750 0x7fcfec199380 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:24.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:24.079+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2846693205 >> 172.21.15.33:0/2846693205 conn(0x7fcfec0fc1b0 msgr2=0x7fcfec107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:24.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:24.079+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2846693205 shutdown_connections 2024-09-16T00:08:24.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:24.079+0000 7fcff2dd1640 1 -- 172.21.15.33:0/2846693205 wait complete. 2024-09-16T00:08:24.712 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-16T00:08:24.962 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:08:25.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:25.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:25 smithi033 ceph-mon[31243]: Added host smithi060 2024-09-16T00:08:25.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:08:25.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:25.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:25.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.391+0000 7f7be4cc5640 1 -- 172.21.15.33:0/1990137531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7be0107660 msgr2=0x7f7be0107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:25.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.391+0000 7f7be4cc5640 1 --2- 172.21.15.33:0/1990137531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7be0107660 0x7f7be0107a40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009880 tx=0x7f7bd002a5d0 comp rx=0 tx=0).stop 2024-09-16T00:08:25.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.391+0000 7f7be4cc5640 1 -- 172.21.15.33:0/1990137531 shutdown_connections 2024-09-16T00:08:25.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.391+0000 7f7be4cc5640 1 --2- 172.21.15.33:0/1990137531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7be0107660 0x7f7be0107a40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:25.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.391+0000 7f7be4cc5640 1 -- 172.21.15.33:0/1990137531 >> 172.21.15.33:0/1990137531 conn(0x7f7be00fd290 msgr2=0x7f7be00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:25.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.391+0000 7f7be4cc5640 1 -- 172.21.15.33:0/1990137531 shutdown_connections 2024-09-16T00:08:25.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.391+0000 7f7be4cc5640 1 -- 172.21.15.33:0/1990137531 wait complete. 2024-09-16T00:08:25.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.392+0000 7f7be4cc5640 1 Processor -- start 2024-09-16T00:08:25.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.392+0000 7f7be4cc5640 1 -- start start 2024-09-16T00:08:25.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.392+0000 7f7be4cc5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7be0107660 0x7f7be01970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:25.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.392+0000 7f7be4cc5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7be010bc90 con 0x7f7be0107660 2024-09-16T00:08:25.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.392+0000 7f7bde59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7be0107660 0x7f7be01970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:25.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.392+0000 7f7bde59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7be0107660 0x7f7be01970f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43920/0 (socket says 172.21.15.33:43920) 2024-09-16T00:08:25.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.392+0000 7f7bde59c640 1 -- 172.21.15.33:0/3367421025 learned_addr learned my addr 172.21.15.33:0/3367421025 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:25.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.392+0000 7f7bde59c640 1 -- 172.21.15.33:0/3367421025 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be0197630 con 0x7f7be0107660 2024-09-16T00:08:25.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.392+0000 7f7bde59c640 1 --2- 172.21.15.33:0/3367421025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7be0107660 0x7f7be01970f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009230 tx=0x7f7bd0003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:25.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.393+0000 7f7bcf7fe640 1 -- 172.21.15.33:0/3367421025 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7bd003f070 con 0x7f7be0107660 2024-09-16T00:08:25.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.393+0000 7f7be4cc5640 1 -- 172.21.15.33:0/3367421025 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be01978c0 con 0x7f7be0107660 2024-09-16T00:08:25.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.393+0000 7f7bcf7fe640 1 -- 172.21.15.33:0/3367421025 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bd003bd40 con 0x7f7be0107660 2024-09-16T00:08:25.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.393+0000 7f7bcf7fe640 1 -- 172.21.15.33:0/3367421025 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7bd0035030 con 0x7f7be0107660 2024-09-16T00:08:25.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.393+0000 7f7be4cc5640 1 -- 172.21.15.33:0/3367421025 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be019b9c0 con 0x7f7be0107660 2024-09-16T00:08:25.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.393+0000 7f7bcf7fe640 1 -- 172.21.15.33:0/3367421025 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f7bd0042070 con 0x7f7be0107660 2024-09-16T00:08:25.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.394+0000 7f7bcf7fe640 1 --2- 172.21.15.33:0/3367421025 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7bc80392a0 0x7f7bc803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:25.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.394+0000 7f7bcf7fe640 1 -- 172.21.15.33:0/3367421025 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7bd006d000 con 0x7f7be0107660 2024-09-16T00:08:25.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.394+0000 7f7bddd9b640 1 --2- 172.21.15.33:0/3367421025 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7bc80392a0 0x7f7bc803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:25.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.394+0000 7f7be4cc5640 1 -- 172.21.15.33:0/3367421025 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7be0102cd0 con 0x7f7be0107660 2024-09-16T00:08:25.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.394+0000 7f7bddd9b640 1 --2- 172.21.15.33:0/3367421025 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7bc80392a0 0x7f7bc803b760 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40098c0 tx=0x7f7bd4006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:25.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.399+0000 7f7bcf7fe640 1 -- 172.21.15.33:0/3367421025 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7bd0039350 con 0x7f7be0107660 2024-09-16T00:08:25.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.579+0000 7f7be4cc5640 1 -- 172.21.15.33:0/3367421025 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7be0107ac0 con 0x7f7bc80392a0 2024-09-16T00:08:25.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.581+0000 7f7bcf7fe640 1 -- 172.21.15.33:0/3367421025 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f7be0107ac0 con 0x7f7bc80392a0 2024-09-16T00:08:25.581 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:08:25.581 INFO:teuthology.orchestra.run.smithi033.stdout:[{"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}, {"addr": "172.21.15.60", "hostname": "smithi060", "labels": [], "status": ""}] 2024-09-16T00:08:25.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.582+0000 7f7be4cc5640 1 -- 172.21.15.33:0/3367421025 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7bc80392a0 msgr2=0x7f7bc803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:25.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.582+0000 7f7be4cc5640 1 --2- 172.21.15.33:0/3367421025 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7bc80392a0 0x7f7bc803b760 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40098c0 tx=0x7f7bd4006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:25.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.582+0000 7f7be4cc5640 1 -- 172.21.15.33:0/3367421025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7be0107660 msgr2=0x7f7be01970f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:25.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.582+0000 7f7be4cc5640 1 --2- 172.21.15.33:0/3367421025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7be0107660 0x7f7be01970f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009230 tx=0x7f7bd0003d80 comp rx=0 tx=0).stop 2024-09-16T00:08:25.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.583+0000 7f7be4cc5640 1 -- 172.21.15.33:0/3367421025 shutdown_connections 2024-09-16T00:08:25.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.583+0000 7f7be4cc5640 1 --2- 172.21.15.33:0/3367421025 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7bc80392a0 0x7f7bc803b760 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:25.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.583+0000 7f7be4cc5640 1 --2- 172.21.15.33:0/3367421025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7be0107660 0x7f7be01970f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:25.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.583+0000 7f7be4cc5640 1 -- 172.21.15.33:0/3367421025 >> 172.21.15.33:0/3367421025 conn(0x7f7be00fd290 msgr2=0x7f7be00febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:25.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.583+0000 7f7be4cc5640 1 -- 172.21.15.33:0/3367421025 shutdown_connections 2024-09-16T00:08:25.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:25.583+0000 7f7be4cc5640 1 -- 172.21.15.33:0/3367421025 wait complete. 2024-09-16T00:08:26.124 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi083 2024-09-16T00:08:26.124 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T00:08:26.125 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/etc/ceph/ceph.conf 2024-09-16T00:08:26.144 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T00:08:26.145 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T00:08:26.202 INFO:tasks.cephadm:Adding host smithi083 to orchestrator... 2024-09-16T00:08:26.203 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch host add smithi083 2024-09-16T00:08:26.468 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:08:26.691 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:26 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:26.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.898+0000 7fa87ab54640 1 -- 172.21.15.33:0/2039896779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8741027b0 msgr2=0x7fa874102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:26.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.898+0000 7fa87ab54640 1 --2- 172.21.15.33:0/2039896779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8741027b0 0x7fa874102b90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa868009880 tx=0x7fa86802a670 comp rx=0 tx=0).stop 2024-09-16T00:08:26.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.898+0000 7fa87ab54640 1 -- 172.21.15.33:0/2039896779 shutdown_connections 2024-09-16T00:08:26.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.898+0000 7fa87ab54640 1 --2- 172.21.15.33:0/2039896779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8741027b0 0x7fa874102b90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:26.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.898+0000 7fa87ab54640 1 -- 172.21.15.33:0/2039896779 >> 172.21.15.33:0/2039896779 conn(0x7fa8740fc1b0 msgr2=0x7fa8740fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:26.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.898+0000 7fa87ab54640 1 -- 172.21.15.33:0/2039896779 shutdown_connections 2024-09-16T00:08:26.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.898+0000 7fa87ab54640 1 -- 172.21.15.33:0/2039896779 wait complete. 2024-09-16T00:08:26.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.899+0000 7fa87ab54640 1 Processor -- start 2024-09-16T00:08:26.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.900+0000 7fa87ab54640 1 -- start start 2024-09-16T00:08:26.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.900+0000 7fa87ab54640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8741027b0 0x7fa8741971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:26.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.900+0000 7fa87ab54640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa87410bcb0 con 0x7fa8741027b0 2024-09-16T00:08:26.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.900+0000 7fa8788f0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8741027b0 0x7fa8741971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:26.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.900+0000 7fa8788f0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8741027b0 0x7fa8741971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43942/0 (socket says 172.21.15.33:43942) 2024-09-16T00:08:26.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.900+0000 7fa8788f0640 1 -- 172.21.15.33:0/2925798264 learned_addr learned my addr 172.21.15.33:0/2925798264 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:26.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.900+0000 7fa8788f0640 1 -- 172.21.15.33:0/2925798264 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa874197700 con 0x7fa8741027b0 2024-09-16T00:08:26.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.900+0000 7fa8788f0640 1 --2- 172.21.15.33:0/2925798264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8741027b0 0x7fa8741971c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa868009230 tx=0x7fa868003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:26.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.901+0000 7fa871ffb640 1 -- 172.21.15.33:0/2925798264 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa86803f070 con 0x7fa8741027b0 2024-09-16T00:08:26.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.901+0000 7fa871ffb640 1 -- 172.21.15.33:0/2925798264 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa86803bd40 con 0x7fa8741027b0 2024-09-16T00:08:26.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.901+0000 7fa87ab54640 1 -- 172.21.15.33:0/2925798264 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa874197990 con 0x7fa8741027b0 2024-09-16T00:08:26.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.901+0000 7fa871ffb640 1 -- 172.21.15.33:0/2925798264 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa868035030 con 0x7fa8741027b0 2024-09-16T00:08:26.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.901+0000 7fa87ab54640 1 -- 172.21.15.33:0/2925798264 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa874197df0 con 0x7fa8741027b0 2024-09-16T00:08:26.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.902+0000 7fa871ffb640 1 -- 172.21.15.33:0/2925798264 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fa868042070 con 0x7fa8741027b0 2024-09-16T00:08:26.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.902+0000 7fa871ffb640 1 --2- 172.21.15.33:0/2925798264 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8640392a0 0x7fa86403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:26.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.902+0000 7fa871ffb640 1 -- 172.21.15.33:0/2925798264 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa86806df30 con 0x7fa8741027b0 2024-09-16T00:08:26.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.902+0000 7fa873fff640 1 --2- 172.21.15.33:0/2925798264 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8640392a0 0x7fa86403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:26.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.902+0000 7fa87ab54640 1 -- 172.21.15.33:0/2925798264 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa87406a750 con 0x7fa8741027b0 2024-09-16T00:08:26.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.902+0000 7fa873fff640 1 --2- 172.21.15.33:0/2925798264 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8640392a0 0x7fa86403b760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa8600098c0 tx=0x7fa860006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:26.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:26.907+0000 7fa871ffb640 1 -- 172.21.15.33:0/2925798264 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa868031de0 con 0x7fa8741027b0 2024-09-16T00:08:27.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:27.106+0000 7fa87ab54640 1 -- 172.21.15.33:0/2925798264 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi083", "target": ["mon-mgr", ""]}) -- 0x7fa8741985c0 con 0x7fa8640392a0 2024-09-16T00:08:27.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:27 smithi033 ceph-mon[31243]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:08:28.731 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:28 smithi033 ceph-mon[31243]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi083", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:08:28.731 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:28.731 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:28.731 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:28.731 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:28.731 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi060", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:08:28.731 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:28.732 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:08:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:29 smithi033 ceph-mon[31243]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:29 smithi033 ceph-mon[31243]: Updating smithi060:/etc/ceph/ceph.conf 2024-09-16T00:08:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:29 smithi033 ceph-mon[31243]: Deploying cephadm binary to smithi083 2024-09-16T00:08:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:29 smithi033 ceph-mon[31243]: Updating smithi060:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:30.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:30 smithi033 ceph-mon[31243]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-09-16T00:08:30.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:30 smithi033 ceph-mon[31243]: Updating smithi060:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T00:08:30.986 INFO:teuthology.orchestra.run.smithi033.stdout:Added host 'smithi083' with addr '172.21.15.83' 2024-09-16T00:08:30.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:30.985+0000 7fa871ffb640 1 -- 172.21.15.33:0/2925798264 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fa8741985c0 con 0x7fa8640392a0 2024-09-16T00:08:30.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:30.988+0000 7fa87ab54640 1 -- 172.21.15.33:0/2925798264 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8640392a0 msgr2=0x7fa86403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:30.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:30.988+0000 7fa87ab54640 1 --2- 172.21.15.33:0/2925798264 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8640392a0 0x7fa86403b760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa8600098c0 tx=0x7fa860006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:30.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:30.988+0000 7fa87ab54640 1 -- 172.21.15.33:0/2925798264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8741027b0 msgr2=0x7fa8741971c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:30.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:30.988+0000 7fa87ab54640 1 --2- 172.21.15.33:0/2925798264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8741027b0 0x7fa8741971c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa868009230 tx=0x7fa868003d80 comp rx=0 tx=0).stop 2024-09-16T00:08:30.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:30.988+0000 7fa87ab54640 1 -- 172.21.15.33:0/2925798264 shutdown_connections 2024-09-16T00:08:30.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:30.988+0000 7fa87ab54640 1 --2- 172.21.15.33:0/2925798264 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8640392a0 0x7fa86403b760 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:30.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:30.988+0000 7fa87ab54640 1 --2- 172.21.15.33:0/2925798264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8741027b0 0x7fa8741971c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:30.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:30.988+0000 7fa87ab54640 1 -- 172.21.15.33:0/2925798264 >> 172.21.15.33:0/2925798264 conn(0x7fa8740fc1b0 msgr2=0x7fa874105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:30.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:30.988+0000 7fa87ab54640 1 -- 172.21.15.33:0/2925798264 shutdown_connections 2024-09-16T00:08:30.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:30.988+0000 7fa87ab54640 1 -- 172.21.15.33:0/2925798264 wait complete. 2024-09-16T00:08:31.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:31 smithi033 ceph-mon[31243]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:31.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:31.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:08:31.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:31.609 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-16T00:08:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:08:32.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.305+0000 7fd559974640 1 -- 172.21.15.33:0/1228969826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd554106540 msgr2=0x7fd554106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:32.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.305+0000 7fd559974640 1 --2- 172.21.15.33:0/1228969826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd554106540 0x7fd554106920 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009810 tx=0x7fd54c02a540 comp rx=0 tx=0).stop 2024-09-16T00:08:32.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.305+0000 7fd559974640 1 -- 172.21.15.33:0/1228969826 shutdown_connections 2024-09-16T00:08:32.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.305+0000 7fd559974640 1 --2- 172.21.15.33:0/1228969826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd554106540 0x7fd554106920 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:32.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.305+0000 7fd559974640 1 -- 172.21.15.33:0/1228969826 >> 172.21.15.33:0/1228969826 conn(0x7fd554077310 msgr2=0x7fd5540fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:32.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.305+0000 7fd559974640 1 -- 172.21.15.33:0/1228969826 shutdown_connections 2024-09-16T00:08:32.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.305+0000 7fd559974640 1 -- 172.21.15.33:0/1228969826 wait complete. 2024-09-16T00:08:32.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.305+0000 7fd559974640 1 Processor -- start 2024-09-16T00:08:32.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.306+0000 7fd559974640 1 -- start start 2024-09-16T00:08:32.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.306+0000 7fd559974640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd554106540 0x7fd5540fd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:32.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.306+0000 7fd559974640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd55410ab70 con 0x7fd554106540 2024-09-16T00:08:32.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.306+0000 7fd552ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd554106540 0x7fd5540fd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:32.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.306+0000 7fd552ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd554106540 0x7fd5540fd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55928/0 (socket says 172.21.15.33:55928) 2024-09-16T00:08:32.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.306+0000 7fd552ffd640 1 -- 172.21.15.33:0/3360118948 learned_addr learned my addr 172.21.15.33:0/3360118948 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:32.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.306+0000 7fd552ffd640 1 -- 172.21.15.33:0/3360118948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5540ff270 con 0x7fd554106540 2024-09-16T00:08:32.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.306+0000 7fd552ffd640 1 --2- 172.21.15.33:0/3360118948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd554106540 0x7fd5540fd380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009930 tx=0x7fd54c004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:32.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.307+0000 7fd558972640 1 -- 172.21.15.33:0/3360118948 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd54c031400 con 0x7fd554106540 2024-09-16T00:08:32.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.307+0000 7fd558972640 1 -- 172.21.15.33:0/3360118948 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd54c0035f0 con 0x7fd554106540 2024-09-16T00:08:32.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.307+0000 7fd559974640 1 -- 172.21.15.33:0/3360118948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5540fd8c0 con 0x7fd554106540 2024-09-16T00:08:32.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.307+0000 7fd558972640 1 -- 172.21.15.33:0/3360118948 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd54c03c440 con 0x7fd554106540 2024-09-16T00:08:32.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.307+0000 7fd559974640 1 -- 172.21.15.33:0/3360118948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5540fdd80 con 0x7fd554106540 2024-09-16T00:08:32.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.308+0000 7fd558972640 1 -- 172.21.15.33:0/3360118948 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fd54c039040 con 0x7fd554106540 2024-09-16T00:08:32.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.308+0000 7fd559974640 1 -- 172.21.15.33:0/3360118948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd554101bb0 con 0x7fd554106540 2024-09-16T00:08:32.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.308+0000 7fd558972640 1 --2- 172.21.15.33:0/3360118948 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd53c0392a0 0x7fd53c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:32.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.308+0000 7fd558972640 1 -- 172.21.15.33:0/3360118948 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd54c06db20 con 0x7fd554106540 2024-09-16T00:08:32.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.308+0000 7fd5527fc640 1 --2- 172.21.15.33:0/3360118948 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd53c0392a0 0x7fd53c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:32.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.309+0000 7fd5527fc640 1 --2- 172.21.15.33:0/3360118948 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd53c0392a0 0x7fd53c03b760 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd5480098c0 tx=0x7fd548006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:32.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.313+0000 7fd558972640 1 -- 172.21.15.33:0/3360118948 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd54c040030 con 0x7fd554106540 2024-09-16T00:08:32.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.493+0000 7fd559974640 1 -- 172.21.15.33:0/3360118948 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd5540fe8b0 con 0x7fd53c0392a0 2024-09-16T00:08:32.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.494+0000 7fd558972640 1 -- 172.21.15.33:0/3360118948 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+238 (secure 0 0 0) 0x7fd5540fe8b0 con 0x7fd53c0392a0 2024-09-16T00:08:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:08:32.550 INFO:teuthology.orchestra.run.smithi033.stdout:[{"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}, {"addr": "172.21.15.60", "hostname": "smithi060", "labels": [], "status": ""}, {"addr": "172.21.15.83", "hostname": "smithi083", "labels": [], "status": ""}] 2024-09-16T00:08:32.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.496+0000 7fd559974640 1 -- 172.21.15.33:0/3360118948 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd53c0392a0 msgr2=0x7fd53c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:32.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.496+0000 7fd559974640 1 --2- 172.21.15.33:0/3360118948 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd53c0392a0 0x7fd53c03b760 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd5480098c0 tx=0x7fd548006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:32.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.496+0000 7fd559974640 1 -- 172.21.15.33:0/3360118948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd554106540 msgr2=0x7fd5540fd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:32.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.496+0000 7fd559974640 1 --2- 172.21.15.33:0/3360118948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd554106540 0x7fd5540fd380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009930 tx=0x7fd54c004570 comp rx=0 tx=0).stop 2024-09-16T00:08:32.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.496+0000 7fd559974640 1 -- 172.21.15.33:0/3360118948 shutdown_connections 2024-09-16T00:08:32.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.496+0000 7fd559974640 1 --2- 172.21.15.33:0/3360118948 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd53c0392a0 0x7fd53c03b760 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:32.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.496+0000 7fd559974640 1 --2- 172.21.15.33:0/3360118948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd554106540 0x7fd5540fd380 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:32.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.496+0000 7fd559974640 1 -- 172.21.15.33:0/3360118948 >> 172.21.15.33:0/3360118948 conn(0x7fd554077310 msgr2=0x7fd5540696b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:32.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.496+0000 7fd559974640 1 -- 172.21.15.33:0/3360118948 shutdown_connections 2024-09-16T00:08:32.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:32.496+0000 7fd559974640 1 -- 172.21.15.33:0/3360118948 wait complete. 2024-09-16T00:08:32.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:32 smithi033 ceph-mon[31243]: Added host smithi083 2024-09-16T00:08:32.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:32.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:33.134 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-16T00:08:33.134 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-16T00:08:33.386 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:08:33.720 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:33 smithi033 ceph-mon[31243]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:33.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.802+0000 7f7f7732c640 1 -- 172.21.15.33:0/1443507052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f701002e0 msgr2=0x7f7f701006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:33.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.802+0000 7f7f7732c640 1 --2- 172.21.15.33:0/1443507052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f701002e0 0x7f7f701006c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009880 tx=0x7f7f6c02a5d0 comp rx=0 tx=0).stop 2024-09-16T00:08:33.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.803+0000 7f7f7732c640 1 -- 172.21.15.33:0/1443507052 shutdown_connections 2024-09-16T00:08:33.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.803+0000 7f7f7732c640 1 --2- 172.21.15.33:0/1443507052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f701002e0 0x7f7f701006c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:33.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.803+0000 7f7f7732c640 1 -- 172.21.15.33:0/1443507052 >> 172.21.15.33:0/1443507052 conn(0x7f7f700fc1b0 msgr2=0x7f7f700fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:33.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.803+0000 7f7f7732c640 1 -- 172.21.15.33:0/1443507052 shutdown_connections 2024-09-16T00:08:33.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.803+0000 7f7f7732c640 1 -- 172.21.15.33:0/1443507052 wait complete. 2024-09-16T00:08:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.804+0000 7f7f7732c640 1 Processor -- start 2024-09-16T00:08:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.804+0000 7f7f7732c640 1 -- start start 2024-09-16T00:08:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.804+0000 7f7f7732c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f701002e0 0x7f7f701993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.804+0000 7f7f7732c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7010dea0 con 0x7f7f701002e0 2024-09-16T00:08:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.804+0000 7f7f750c8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f701002e0 0x7f7f701993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.804+0000 7f7f750c8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f701002e0 0x7f7f701993d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55946/0 (socket says 172.21.15.33:55946) 2024-09-16T00:08:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.804+0000 7f7f750c8640 1 -- 172.21.15.33:0/3304522224 learned_addr learned my addr 172.21.15.33:0/3304522224 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.804+0000 7f7f750c8640 1 -- 172.21.15.33:0/3304522224 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f70199910 con 0x7f7f701002e0 2024-09-16T00:08:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.805+0000 7f7f750c8640 1 --2- 172.21.15.33:0/3304522224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f701002e0 0x7f7f701993d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009230 tx=0x7f7f6c003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.805+0000 7f7f667fc640 1 -- 172.21.15.33:0/3304522224 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f6c03f070 con 0x7f7f701002e0 2024-09-16T00:08:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.805+0000 7f7f667fc640 1 -- 172.21.15.33:0/3304522224 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f6c03bc80 con 0x7f7f701002e0 2024-09-16T00:08:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.805+0000 7f7f7732c640 1 -- 172.21.15.33:0/3304522224 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f70199ba0 con 0x7f7f701002e0 2024-09-16T00:08:33.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.805+0000 7f7f667fc640 1 -- 172.21.15.33:0/3304522224 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f6c035030 con 0x7f7f701002e0 2024-09-16T00:08:33.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.805+0000 7f7f7732c640 1 -- 172.21.15.33:0/3304522224 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f7019a000 con 0x7f7f701002e0 2024-09-16T00:08:33.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.806+0000 7f7f667fc640 1 -- 172.21.15.33:0/3304522224 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f7f6c042070 con 0x7f7f701002e0 2024-09-16T00:08:33.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.806+0000 7f7f7732c640 1 -- 172.21.15.33:0/3304522224 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f7006a7b0 con 0x7f7f701002e0 2024-09-16T00:08:33.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.806+0000 7f7f667fc640 1 --2- 172.21.15.33:0/3304522224 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7f5c039300 0x7f7f5c03b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:33.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.806+0000 7f7f667fc640 1 -- 172.21.15.33:0/3304522224 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7f6c06d0a0 con 0x7f7f701002e0 2024-09-16T00:08:33.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.806+0000 7f7f748c7640 1 --2- 172.21.15.33:0/3304522224 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7f5c039300 0x7f7f5c03b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:33.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.807+0000 7f7f748c7640 1 --2- 172.21.15.33:0/3304522224 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7f5c039300 0x7f7f5c03b7c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7f600098c0 tx=0x7f7f60006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:33.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.811+0000 7f7f667fc640 1 -- 172.21.15.33:0/3304522224 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7f6c031dd0 con 0x7f7f701002e0 2024-09-16T00:08:33.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:33.992+0000 7f7f7732c640 1 -- 172.21.15.33:0/3304522224 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f7f7019a790 con 0x7f7f701002e0 2024-09-16T00:08:34.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:34.449+0000 7f7f667fc640 1 -- 172.21.15.33:0/3304522224 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f7f6c0314c0 con 0x7f7f701002e0 2024-09-16T00:08:34.450 INFO:teuthology.orchestra.run.smithi033.stderr:adjusted tunables profile to default 2024-09-16T00:08:34.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:34.452+0000 7f7f7732c640 1 -- 172.21.15.33:0/3304522224 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7f5c039300 msgr2=0x7f7f5c03b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:34.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:34.452+0000 7f7f7732c640 1 --2- 172.21.15.33:0/3304522224 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7f5c039300 0x7f7f5c03b7c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7f600098c0 tx=0x7f7f60006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:34.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:34.452+0000 7f7f7732c640 1 -- 172.21.15.33:0/3304522224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f701002e0 msgr2=0x7f7f701993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:34.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:34.452+0000 7f7f7732c640 1 --2- 172.21.15.33:0/3304522224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f701002e0 0x7f7f701993d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009230 tx=0x7f7f6c003d40 comp rx=0 tx=0).stop 2024-09-16T00:08:34.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:34.452+0000 7f7f7732c640 1 -- 172.21.15.33:0/3304522224 shutdown_connections 2024-09-16T00:08:34.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:34.452+0000 7f7f7732c640 1 --2- 172.21.15.33:0/3304522224 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7f5c039300 0x7f7f5c03b7c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:34.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:34.452+0000 7f7f7732c640 1 --2- 172.21.15.33:0/3304522224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f701002e0 0x7f7f701993d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:34.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:34.452+0000 7f7f7732c640 1 -- 172.21.15.33:0/3304522224 >> 172.21.15.33:0/3304522224 conn(0x7f7f700fc1b0 msgr2=0x7f7f700fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:34.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:34.452+0000 7f7f7732c640 1 -- 172.21.15.33:0/3304522224 shutdown_connections 2024-09-16T00:08:34.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:34.452+0000 7f7f7732c640 1 -- 172.21.15.33:0/3304522224 wait complete. 2024-09-16T00:08:34.757 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:34 smithi033 ceph-mon[31243]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:08:34.757 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:34 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3304522224' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-16T00:08:35.059 INFO:tasks.cephadm:Adding mon.a on smithi033 2024-09-16T00:08:35.060 INFO:tasks.cephadm:Adding mon.b on smithi060 2024-09-16T00:08:35.060 INFO:tasks.cephadm:Adding mon.c on smithi083 2024-09-16T00:08:35.060 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi033:172.21.15.33=a;smithi060:172.21.15.60=b;smithi083:172.21.15.83=c' 2024-09-16T00:08:35.362 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-16T00:08:35.407 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-16T00:08:35.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:35 smithi033 ceph-mon[31243]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:35.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:35 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3304522224' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-16T00:08:35.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:35 smithi033 ceph-mon[31243]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T00:08:35.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:35.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:35.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:35.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:35.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi083", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:08:35.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:35.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:08:35.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.893+0000 7f16ec9db640 1 -- 172.21.15.83:0/3573194433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16e8100540 msgr2=0x7f16e8100940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.893+0000 7f16ec9db640 1 --2- 172.21.15.83:0/3573194433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16e8100540 0x7f16e8100940 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f16d8009880 tx=0x7f16d802a5d0 comp rx=0 tx=0).stop 2024-09-16T00:08:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.893+0000 7f16ec9db640 1 -- 172.21.15.83:0/3573194433 shutdown_connections 2024-09-16T00:08:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.893+0000 7f16ec9db640 1 --2- 172.21.15.83:0/3573194433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16e8100540 0x7f16e8100940 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.893+0000 7f16ec9db640 1 -- 172.21.15.83:0/3573194433 >> 172.21.15.83:0/3573194433 conn(0x7f16e80fbcd0 msgr2=0x7f16e80fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.893+0000 7f16ec9db640 1 -- 172.21.15.83:0/3573194433 shutdown_connections 2024-09-16T00:08:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.893+0000 7f16ec9db640 1 -- 172.21.15.83:0/3573194433 wait complete. 2024-09-16T00:08:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.894+0000 7f16ec9db640 1 Processor -- start 2024-09-16T00:08:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.894+0000 7f16ec9db640 1 -- start start 2024-09-16T00:08:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.894+0000 7f16ec9db640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16e8100540 0x7f16e806a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.894+0000 7f16ec9db640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f16e81065f0 con 0x7f16e8100540 2024-09-16T00:08:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.894+0000 7f16e659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16e8100540 0x7f16e806a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.894+0000 7f16e659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16e8100540 0x7f16e806a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.83:46562/0 (socket says 172.21.15.83:46562) 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.894+0000 7f16e659c640 1 -- 172.21.15.83:0/3222955330 learned_addr learned my addr 172.21.15.83:0/3222955330 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.895+0000 7f16e659c640 1 -- 172.21.15.83:0/3222955330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e8199ab0 con 0x7f16e8100540 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.895+0000 7f16e659c640 1 --2- 172.21.15.83:0/3222955330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16e8100540 0x7f16e806a130 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f16d8004f60 tx=0x7f16d80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.895+0000 7f16d77fe640 1 -- 172.21.15.83:0/3222955330 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16d803f070 con 0x7f16e8100540 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.895+0000 7f16d77fe640 1 -- 172.21.15.83:0/3222955330 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16d8004d40 con 0x7f16e8100540 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.895+0000 7f16ec9db640 1 -- 172.21.15.83:0/3222955330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16e8199c80 con 0x7f16e8100540 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.895+0000 7f16d77fe640 1 -- 172.21.15.83:0/3222955330 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16d8035030 con 0x7f16e8100540 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.895+0000 7f16ec9db640 1 -- 172.21.15.83:0/3222955330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16e819a020 con 0x7f16e8100540 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.896+0000 7f16d77fe640 1 -- 172.21.15.83:0/3222955330 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f16d8042070 con 0x7f16e8100540 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.896+0000 7f16d77fe640 1 --2- 172.21.15.83:0/3222955330 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f16d00392a0 0x7f16d003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.896+0000 7f16d77fe640 1 -- 172.21.15.83:0/3222955330 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f16d806d0f0 con 0x7f16e8100540 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.896+0000 7f16ec9db640 1 -- 172.21.15.83:0/3222955330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16e8106d70 con 0x7f16e8100540 2024-09-16T00:08:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.896+0000 7f16e5d9b640 1 --2- 172.21.15.83:0/3222955330 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f16d00392a0 0x7f16d003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.897+0000 7f16e5d9b640 1 --2- 172.21.15.83:0/3222955330 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f16d00392a0 0x7f16d003b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f16dc0098c0 tx=0x7f16dc006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:35.901+0000 7f16d77fe640 1 -- 172.21.15.83:0/3222955330 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f16d8042360 con 0x7f16e8100540 2024-09-16T00:08:36.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:36.091+0000 7f16ec9db640 1 -- 172.21.15.83:0/3222955330 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi033:172.21.15.33=a;smithi060:172.21.15.60=b;smithi083:172.21.15.83=c", "target": ["mon-mgr", ""]}) -- 0x7f16e802da70 con 0x7f16d00392a0 2024-09-16T00:08:36.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:36.096+0000 7f16d77fe640 1 -- 172.21.15.83:0/3222955330 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f16e802da70 con 0x7f16d00392a0 2024-09-16T00:08:36.097 INFO:teuthology.orchestra.run.smithi083.stdout:Scheduled mon update... 2024-09-16T00:08:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:36.098+0000 7f16ec9db640 1 -- 172.21.15.83:0/3222955330 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f16d00392a0 msgr2=0x7f16d003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:36.098+0000 7f16ec9db640 1 --2- 172.21.15.83:0/3222955330 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f16d00392a0 0x7f16d003b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f16dc0098c0 tx=0x7f16dc006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:36.098+0000 7f16ec9db640 1 -- 172.21.15.83:0/3222955330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16e8100540 msgr2=0x7f16e806a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:36.098+0000 7f16ec9db640 1 --2- 172.21.15.83:0/3222955330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16e8100540 0x7f16e806a130 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f16d8004f60 tx=0x7f16d80048f0 comp rx=0 tx=0).stop 2024-09-16T00:08:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:36.098+0000 7f16ec9db640 1 -- 172.21.15.83:0/3222955330 shutdown_connections 2024-09-16T00:08:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:36.098+0000 7f16ec9db640 1 --2- 172.21.15.83:0/3222955330 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f16d00392a0 0x7f16d003b760 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:36.098+0000 7f16ec9db640 1 --2- 172.21.15.83:0/3222955330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16e8100540 0x7f16e806a130 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:36.098+0000 7f16ec9db640 1 -- 172.21.15.83:0/3222955330 >> 172.21.15.83:0/3222955330 conn(0x7f16e80fbcd0 msgr2=0x7f16e80fc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:36.099+0000 7f16ec9db640 1 -- 172.21.15.83:0/3222955330 shutdown_connections 2024-09-16T00:08:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:36.099+0000 7f16ec9db640 1 -- 172.21.15.83:0/3222955330 wait complete. 2024-09-16T00:08:36.667 DEBUG:teuthology.orchestra.run.smithi060:mon.b> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.b.service 2024-09-16T00:08:36.670 DEBUG:teuthology.orchestra.run.smithi083:mon.c> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.c.service 2024-09-16T00:08:36.672 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T00:08:36.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T00:08:36.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: Updating smithi083:/etc/ceph/ceph.conf 2024-09-16T00:08:36.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: Updating smithi083:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:36.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:36.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:36.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:36.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:36.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:36.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:36.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:36.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:36.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:08:36.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T00:08:36.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:36.942 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:37.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.333+0000 7fac6ff1b640 1 -- 172.21.15.83:0/1115974307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac68107660 msgr2=0x7fac68107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:37.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.333+0000 7fac6ff1b640 1 --2- 172.21.15.83:0/1115974307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac68107660 0x7fac68107a40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fac64009880 tx=0x7fac6402a5d0 comp rx=0 tx=0).stop 2024-09-16T00:08:37.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.333+0000 7fac6ff1b640 1 -- 172.21.15.83:0/1115974307 shutdown_connections 2024-09-16T00:08:37.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.333+0000 7fac6ff1b640 1 --2- 172.21.15.83:0/1115974307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac68107660 0x7fac68107a40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.333+0000 7fac6ff1b640 1 -- 172.21.15.83:0/1115974307 >> 172.21.15.83:0/1115974307 conn(0x7fac680fd290 msgr2=0x7fac680ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.333+0000 7fac6ff1b640 1 -- 172.21.15.83:0/1115974307 shutdown_connections 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.333+0000 7fac6ff1b640 1 -- 172.21.15.83:0/1115974307 wait complete. 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.334+0000 7fac6ff1b640 1 Processor -- start 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.334+0000 7fac6ff1b640 1 -- start start 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.334+0000 7fac6ff1b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac68107660 0x7fac680744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.334+0000 7fac6ff1b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac6810bc90 con 0x7fac68107660 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.334+0000 7fac6dcb7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac68107660 0x7fac680744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.334+0000 7fac6dcb7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac68107660 0x7fac680744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.83:46580/0 (socket says 172.21.15.83:46580) 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.334+0000 7fac6dcb7640 1 -- 172.21.15.83:0/2199210762 learned_addr learned my addr 172.21.15.83:0/2199210762 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.335+0000 7fac6dcb7640 1 -- 172.21.15.83:0/2199210762 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac680763e0 con 0x7fac68107660 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.335+0000 7fac6dcb7640 1 --2- 172.21.15.83:0/2199210762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac68107660 0x7fac680744f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fac64009230 tx=0x7fac64003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.335+0000 7fac5affd640 1 -- 172.21.15.83:0/2199210762 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac6403f070 con 0x7fac68107660 2024-09-16T00:08:37.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.335+0000 7fac6ff1b640 1 -- 172.21.15.83:0/2199210762 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac68074a30 con 0x7fac68107660 2024-09-16T00:08:37.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.335+0000 7fac5affd640 1 -- 172.21.15.83:0/2199210762 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac6403bc80 con 0x7fac68107660 2024-09-16T00:08:37.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.335+0000 7fac5affd640 1 -- 172.21.15.83:0/2199210762 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac64035030 con 0x7fac68107660 2024-09-16T00:08:37.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.335+0000 7fac6ff1b640 1 -- 172.21.15.83:0/2199210762 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac68074ef0 con 0x7fac68107660 2024-09-16T00:08:37.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.336+0000 7fac5affd640 1 -- 172.21.15.83:0/2199210762 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fac64042070 con 0x7fac68107660 2024-09-16T00:08:37.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.336+0000 7fac6ff1b640 1 -- 172.21.15.83:0/2199210762 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac68102cd0 con 0x7fac68107660 2024-09-16T00:08:37.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.336+0000 7fac5affd640 1 --2- 172.21.15.83:0/2199210762 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac540392a0 0x7fac5403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:37.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.336+0000 7fac5affd640 1 -- 172.21.15.83:0/2199210762 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fac6406d080 con 0x7fac68107660 2024-09-16T00:08:37.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.337+0000 7fac6d4b6640 1 --2- 172.21.15.83:0/2199210762 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac540392a0 0x7fac5403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:37.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.337+0000 7fac6d4b6640 1 --2- 172.21.15.83:0/2199210762 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac540392a0 0x7fac5403b760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fac5c0098c0 tx=0x7fac5c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:37.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.341+0000 7fac5affd640 1 -- 172.21.15.83:0/2199210762 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fac6404c350 con 0x7fac68107660 2024-09-16T00:08:37.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.575+0000 7fac6ff1b640 1 -- 172.21.15.83:0/2199210762 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fac68075a20 con 0x7fac68107660 2024-09-16T00:08:37.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.576+0000 7fac5affd640 1 -- 172.21.15.83:0/2199210762 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fac6404ad50 con 0x7fac68107660 2024-09-16T00:08:37.577 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:08:37.577 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":1,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","modified":"2024-09-16T00:07:17.720547Z","created":"2024-09-16T00:07:17.720547Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-16T00:08:37.577 INFO:teuthology.orchestra.run.smithi083.stderr:dumped monmap epoch 1 2024-09-16T00:08:37.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.578+0000 7fac6ff1b640 1 -- 172.21.15.83:0/2199210762 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac540392a0 msgr2=0x7fac5403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:37.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.578+0000 7fac6ff1b640 1 --2- 172.21.15.83:0/2199210762 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac540392a0 0x7fac5403b760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fac5c0098c0 tx=0x7fac5c006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:37.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.578+0000 7fac6ff1b640 1 -- 172.21.15.83:0/2199210762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac68107660 msgr2=0x7fac680744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:37.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.578+0000 7fac6ff1b640 1 --2- 172.21.15.83:0/2199210762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac68107660 0x7fac680744f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fac64009230 tx=0x7fac64003d40 comp rx=0 tx=0).stop 2024-09-16T00:08:37.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.578+0000 7fac6ff1b640 1 -- 172.21.15.83:0/2199210762 shutdown_connections 2024-09-16T00:08:37.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.578+0000 7fac6ff1b640 1 --2- 172.21.15.83:0/2199210762 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac540392a0 0x7fac5403b760 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:37.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.578+0000 7fac6ff1b640 1 --2- 172.21.15.83:0/2199210762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac68107660 0x7fac680744f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:37.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.578+0000 7fac6ff1b640 1 -- 172.21.15.83:0/2199210762 >> 172.21.15.83:0/2199210762 conn(0x7fac680fd290 msgr2=0x7fac680fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:37.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.578+0000 7fac6ff1b640 1 -- 172.21.15.83:0/2199210762 shutdown_connections 2024-09-16T00:08:37.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:37.578+0000 7fac6ff1b640 1 -- 172.21.15.83:0/2199210762 wait complete. 2024-09-16T00:08:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:37 smithi033 ceph-mon[31243]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:37 smithi033 ceph-mon[31243]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-09-16T00:08:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:37 smithi033 ceph-mon[31243]: Updating smithi083:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T00:08:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:37 smithi033 ceph-mon[31243]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi033:172.21.15.33=a;smithi060:172.21.15.60=b;smithi083:172.21.15.83=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:08:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:37 smithi033 ceph-mon[31243]: Saving service mon spec with placement smithi033:172.21.15.33=a;smithi060:172.21.15.60=b;smithi083:172.21.15.83=c;count:3 2024-09-16T00:08:37.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:37 smithi033 ceph-mon[31243]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T00:08:37.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:37 smithi033 ceph-mon[31243]: Reconfiguring daemon mon.a on smithi033 2024-09-16T00:08:38.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:38.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:38.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:08:38.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:38.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:08:38.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:38.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:08:38.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:38.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:38 smithi033 ceph-mon[31243]: from='client.? 172.21.15.83:0/2199210762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T00:08:39.273 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T00:08:39.274 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T00:08:39.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:39 smithi033 ceph-mon[31243]: Deploying daemon mon.c on smithi083 2024-09-16T00:08:39.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:39 smithi033 ceph-mon[31243]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:40.283 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:08:40.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.881+0000 7fbff57dc640 1 -- 172.21.15.83:0/3477445400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbff0107670 msgr2=0x7fbff0107a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:40.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.881+0000 7fbff57dc640 1 --2- 172.21.15.83:0/3477445400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbff0107670 0x7fbff0107a50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0009880 tx=0x7fbfe002a5d0 comp rx=0 tx=0).stop 2024-09-16T00:08:40.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.881+0000 7fbff57dc640 1 -- 172.21.15.83:0/3477445400 shutdown_connections 2024-09-16T00:08:40.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.881+0000 7fbff57dc640 1 --2- 172.21.15.83:0/3477445400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbff0107670 0x7fbff0107a50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:40.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.881+0000 7fbff57dc640 1 -- 172.21.15.83:0/3477445400 >> 172.21.15.83:0/3477445400 conn(0x7fbff00fd330 msgr2=0x7fbff00ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:40.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.881+0000 7fbff57dc640 1 -- 172.21.15.83:0/3477445400 shutdown_connections 2024-09-16T00:08:40.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.881+0000 7fbff57dc640 1 -- 172.21.15.83:0/3477445400 wait complete. 2024-09-16T00:08:40.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.882+0000 7fbff57dc640 1 Processor -- start 2024-09-16T00:08:40.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.882+0000 7fbff57dc640 1 -- start start 2024-09-16T00:08:40.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.882+0000 7fbff57dc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbff0107670 0x7fbff01972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:40.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.882+0000 7fbff57dc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbff010bd30 con 0x7fbff0107670 2024-09-16T00:08:40.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.882+0000 7fbfeeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbff0107670 0x7fbff01972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:40.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.882+0000 7fbfeeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbff0107670 0x7fbff01972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.83:46590/0 (socket says 172.21.15.83:46590) 2024-09-16T00:08:40.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.882+0000 7fbfeeffd640 1 -- 172.21.15.83:0/2945113963 learned_addr learned my addr 172.21.15.83:0/2945113963 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-16T00:08:40.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.883+0000 7fbfeeffd640 1 -- 172.21.15.83:0/2945113963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff0197800 con 0x7fbff0107670 2024-09-16T00:08:40.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.883+0000 7fbfeeffd640 1 --2- 172.21.15.83:0/2945113963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbff0107670 0x7fbff01972c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0009230 tx=0x7fbfe0003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:40.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.883+0000 7fbfe7fff640 1 -- 172.21.15.83:0/2945113963 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbfe003f070 con 0x7fbff0107670 2024-09-16T00:08:40.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.883+0000 7fbfe7fff640 1 -- 172.21.15.83:0/2945113963 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfe003bc80 con 0x7fbff0107670 2024-09-16T00:08:40.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.883+0000 7fbff57dc640 1 -- 172.21.15.83:0/2945113963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff0197af0 con 0x7fbff0107670 2024-09-16T00:08:40.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.883+0000 7fbfe7fff640 1 -- 172.21.15.83:0/2945113963 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbfe0035030 con 0x7fbff0107670 2024-09-16T00:08:40.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.883+0000 7fbff57dc640 1 -- 172.21.15.83:0/2945113963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff019bb90 con 0x7fbff0107670 2024-09-16T00:08:40.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.884+0000 7fbfe7fff640 1 -- 172.21.15.83:0/2945113963 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fbfe0042070 con 0x7fbff0107670 2024-09-16T00:08:40.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.884+0000 7fbff57dc640 1 -- 172.21.15.83:0/2945113963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff0102db0 con 0x7fbff0107670 2024-09-16T00:08:40.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.884+0000 7fbfe7fff640 1 --2- 172.21.15.83:0/2945113963 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbfdc0392a0 0x7fbfdc03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:40.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.884+0000 7fbfe7fff640 1 -- 172.21.15.83:0/2945113963 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbfe006d0c0 con 0x7fbff0107670 2024-09-16T00:08:40.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.884+0000 7fbfee7fc640 1 --2- 172.21.15.83:0/2945113963 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbfdc0392a0 0x7fbfdc03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:40.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.885+0000 7fbfee7fc640 1 --2- 172.21.15.83:0/2945113963 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbfdc0392a0 0x7fbfdc03b760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80098c0 tx=0x7fbfd8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:40.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:40.889+0000 7fbfe7fff640 1 -- 172.21.15.83:0/2945113963 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbfe0031e20 con 0x7fbff0107670 2024-09-16T00:08:41.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:41.144+0000 7fbff57dc640 1 -- 172.21.15.83:0/2945113963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fbff0198620 con 0x7fbff0107670 2024-09-16T00:08:41.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:41.145+0000 7fbfe7fff640 1 -- 172.21.15.83:0/2945113963 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fbfe00372d0 con 0x7fbff0107670 2024-09-16T00:08:41.146 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:08:41.146 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":1,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","modified":"2024-09-16T00:07:17.720547Z","created":"2024-09-16T00:07:17.720547Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-16T00:08:41.146 INFO:teuthology.orchestra.run.smithi083.stderr:dumped monmap epoch 1 2024-09-16T00:08:41.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:41.147+0000 7fbff57dc640 1 -- 172.21.15.83:0/2945113963 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbfdc0392a0 msgr2=0x7fbfdc03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:41.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:41.147+0000 7fbff57dc640 1 --2- 172.21.15.83:0/2945113963 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbfdc0392a0 0x7fbfdc03b760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80098c0 tx=0x7fbfd8006db0 comp rx=0 tx=0).stop 2024-09-16T00:08:41.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:41.147+0000 7fbff57dc640 1 -- 172.21.15.83:0/2945113963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbff0107670 msgr2=0x7fbff01972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:41.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:41.147+0000 7fbff57dc640 1 --2- 172.21.15.83:0/2945113963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbff0107670 0x7fbff01972c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0009230 tx=0x7fbfe0003d40 comp rx=0 tx=0).stop 2024-09-16T00:08:41.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:41.147+0000 7fbff57dc640 1 -- 172.21.15.83:0/2945113963 shutdown_connections 2024-09-16T00:08:41.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:41.147+0000 7fbff57dc640 1 --2- 172.21.15.83:0/2945113963 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbfdc0392a0 0x7fbfdc03b760 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:41.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:41.147+0000 7fbff57dc640 1 --2- 172.21.15.83:0/2945113963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbff0107670 0x7fbff01972c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:41.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:41.147+0000 7fbff57dc640 1 -- 172.21.15.83:0/2945113963 >> 172.21.15.83:0/2945113963 conn(0x7fbff00fd330 msgr2=0x7fbff0105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:41.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:41.148+0000 7fbff57dc640 1 -- 172.21.15.83:0/2945113963 shutdown_connections 2024-09-16T00:08:41.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:41.148+0000 7fbff57dc640 1 -- 172.21.15.83:0/2945113963 wait complete. 2024-09-16T00:08:41.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:41 smithi033 ceph-mon[31243]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:41.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:41 smithi033 ceph-mon[31243]: from='client.? 172.21.15.83:0/2945113963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T00:08:42.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:42 smithi083 systemd[1]: Started Ceph mon.c for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:08:42.740 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T00:08:42.740 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T00:08:43.005 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:08:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:08:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: mon.a calling monitor election 2024-09-16T00:08:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: mon.c calling monitor election 2024-09-16T00:08:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: monmap epoch 2 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: last_changed 2024-09-16T00:08:42.123171+0000 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: created 2024-09-16T00:07:17.720547+0000 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: min_mon_release 19 (squid) 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: election_strategy: 1 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.c 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: fsmap 2024-09-16T00:08:47.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T00:08:47.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: mgrmap e13: x(active, since 39s) 2024-09-16T00:08:47.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: overall HEALTH_OK 2024-09-16T00:08:47.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:47.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:47.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:47.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:08:47.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:47.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:47.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:08:47.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: mon.a calling monitor election 2024-09-16T00:08:47.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:47.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:47.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: mon.c calling monitor election 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: monmap epoch 2 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: last_changed 2024-09-16T00:08:42.123171+0000 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: created 2024-09-16T00:07:17.720547+0000 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: min_mon_release 19 (squid) 2024-09-16T00:08:47.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: election_strategy: 1 2024-09-16T00:08:47.671 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-09-16T00:08:47.671 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.c 2024-09-16T00:08:47.671 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: fsmap 2024-09-16T00:08:47.671 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T00:08:47.671 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: mgrmap e13: x(active, since 39s) 2024-09-16T00:08:47.671 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: overall HEALTH_OK 2024-09-16T00:08:47.671 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:47.671 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:47.671 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:47.671 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:08:47.671 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:47.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.809+0000 7fe158b66640 1 -- 172.21.15.83:0/2753245269 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe140003530 msgr2=0x7fe1400059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:47.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.809+0000 7fe158b66640 1 --2- 172.21.15.83:0/2753245269 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe140003530 0x7fe1400059c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe1541016c0 tx=0x7fe14402c490 comp rx=0 tx=0).stop 2024-09-16T00:08:47.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.809+0000 7fe158b66640 1 -- 172.21.15.83:0/2753245269 shutdown_connections 2024-09-16T00:08:47.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.809+0000 7fe158b66640 1 --2- 172.21.15.83:0/2753245269 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe140003530 0x7fe1400059c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:47.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.809+0000 7fe158b66640 1 -- 172.21.15.83:0/2753245269 >> 172.21.15.83:0/2753245269 conn(0x7fe1540fc280 msgr2=0x7fe1540fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:47.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.809+0000 7fe158b66640 1 -- 172.21.15.83:0/2753245269 shutdown_connections 2024-09-16T00:08:47.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.809+0000 7fe158b66640 1 -- 172.21.15.83:0/2753245269 wait complete. 2024-09-16T00:08:47.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.810+0000 7fe158b66640 1 Processor -- start 2024-09-16T00:08:47.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe158b66640 1 -- start start 2024-09-16T00:08:47.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe158b66640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe140003530 0x7fe154102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:47.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe158b66640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe154102750 0x7fe154102bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:47.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe158b66640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe154108690 con 0x7fe154102750 2024-09-16T00:08:47.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe158b66640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe154108510 con 0x7fe140003530 2024-09-16T00:08:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe15259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe140003530 0x7fe154102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe15259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe140003530 0x7fe154102210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35434/0 (socket says 172.21.15.83:35434) 2024-09-16T00:08:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe15259c640 1 -- 172.21.15.83:0/2690877294 learned_addr learned my addr 172.21.15.83:0/2690877294 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-16T00:08:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe14bfff640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe154102750 0x7fe154102bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe15259c640 1 -- 172.21.15.83:0/2690877294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe140003530 msgr2=0x7fe154102210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-16T00:08:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe15259c640 1 -- 172.21.15.83:0/2690877294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe140003530 msgr2=0x7fe154102210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T00:08:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe15259c640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe140003530 0x7fe154102210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T00:08:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe15259c640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe140003530 0x7fe154102210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T00:08:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe14bfff640 1 -- 172.21.15.83:0/2690877294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe140003530 msgr2=0x7fe154102210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T00:08:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe14bfff640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe140003530 0x7fe154102210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.811+0000 7fe14bfff640 1 -- 172.21.15.83:0/2690877294 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe154104100 con 0x7fe154102750 2024-09-16T00:08:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.812+0000 7fe14bfff640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe154102750 0x7fe154102bb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe13c00b400 tx=0x7fe13c00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:47.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.812+0000 7fe14b7fe640 1 -- 172.21.15.83:0/2690877294 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe13c011630 con 0x7fe154102750 2024-09-16T00:08:47.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.812+0000 7fe14b7fe640 1 -- 172.21.15.83:0/2690877294 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe13c011cb0 con 0x7fe154102750 2024-09-16T00:08:47.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.812+0000 7fe158b66640 1 -- 172.21.15.83:0/2690877294 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1541a3570 con 0x7fe154102750 2024-09-16T00:08:47.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.812+0000 7fe14b7fe640 1 -- 172.21.15.83:0/2690877294 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe13c010d80 con 0x7fe154102750 2024-09-16T00:08:47.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.812+0000 7fe158b66640 1 -- 172.21.15.83:0/2690877294 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1541a3b00 con 0x7fe154102750 2024-09-16T00:08:47.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.813+0000 7fe14b7fe640 1 -- 172.21.15.83:0/2690877294 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fe13c018020 con 0x7fe154102750 2024-09-16T00:08:47.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.813+0000 7fe158b66640 1 -- 172.21.15.83:0/2690877294 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe154101cc0 con 0x7fe154102750 2024-09-16T00:08:47.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.813+0000 7fe14b7fe640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe134039090 0x7fe13403b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:47.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.814+0000 7fe14b7fe640 1 -- 172.21.15.83:0/2690877294 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe13c04bf60 con 0x7fe154102750 2024-09-16T00:08:47.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.814+0000 7fe15259c640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe134039090 0x7fe13403b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:47.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.814+0000 7fe15259c640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe134039090 0x7fe13403b550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe144012c10 tx=0x7fe1440323a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:47.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:47.819+0000 7fe14b7fe640 1 -- 172.21.15.83:0/2690877294 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe13c01f020 con 0x7fe154102750 2024-09-16T00:08:48.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.056+0000 7fe158b66640 1 -- 172.21.15.83:0/2690877294 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe1541a3e10 con 0x7fe154102750 2024-09-16T00:08:48.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.056+0000 7fe14b7fe640 1 -- 172.21.15.83:0/2690877294 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7fe13c018310 con 0x7fe154102750 2024-09-16T00:08:48.058 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:08:48.059 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":2,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","modified":"2024-09-16T00:08:42.123171Z","created":"2024-09-16T00:07:17.720547Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-16T00:08:48.059 INFO:teuthology.orchestra.run.smithi083.stderr:dumped monmap epoch 2 2024-09-16T00:08:48.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.059+0000 7fe158b66640 1 -- 172.21.15.83:0/2690877294 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe134039090 msgr2=0x7fe13403b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:48.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.059+0000 7fe158b66640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe134039090 0x7fe13403b550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe144012c10 tx=0x7fe1440323a0 comp rx=0 tx=0).stop 2024-09-16T00:08:48.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.059+0000 7fe158b66640 1 -- 172.21.15.83:0/2690877294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe154102750 msgr2=0x7fe154102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:48.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.059+0000 7fe158b66640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe154102750 0x7fe154102bb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe13c00b400 tx=0x7fe13c00b8c0 comp rx=0 tx=0).stop 2024-09-16T00:08:48.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.059+0000 7fe158b66640 1 -- 172.21.15.83:0/2690877294 shutdown_connections 2024-09-16T00:08:48.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.059+0000 7fe158b66640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe134039090 0x7fe13403b550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:48.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.059+0000 7fe158b66640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe154102750 0x7fe154102bb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:48.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.059+0000 7fe158b66640 1 --2- 172.21.15.83:0/2690877294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe140003530 0x7fe154102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:48.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.059+0000 7fe158b66640 1 -- 172.21.15.83:0/2690877294 >> 172.21.15.83:0/2690877294 conn(0x7fe1540fc280 msgr2=0x7fe15406a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:48.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.059+0000 7fe158b66640 1 -- 172.21.15.83:0/2690877294 shutdown_connections 2024-09-16T00:08:48.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:48.059+0000 7fe158b66640 1 -- 172.21.15.83:0/2690877294 wait complete. 2024-09-16T00:08:48.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:48 smithi083 ceph-mon[35242]: Deploying daemon mon.b on smithi060 2024-09-16T00:08:48.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:48 smithi083 ceph-mon[35242]: from='client.? 172.21.15.83:0/2690877294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T00:08:48.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:48 smithi033 ceph-mon[31243]: Deploying daemon mon.b on smithi060 2024-09-16T00:08:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:48 smithi033 ceph-mon[31243]: from='client.? 172.21.15.83:0/2690877294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T00:08:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:49 smithi033 ceph-mon[31243]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:49.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:49 smithi083 ceph-mon[35242]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:49.843 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T00:08:49.844 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T00:08:50.105 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:08:50.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 -- 172.21.15.83:0/494309886 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66e4106650 msgr2=0x7f66e4106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:50.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 --2- 172.21.15.83:0/494309886 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66e4106650 0x7f66e4106a30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009c20 tx=0x7f66d402aa90 comp rx=0 tx=0).stop 2024-09-16T00:08:50.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 -- 172.21.15.83:0/494309886 shutdown_connections 2024-09-16T00:08:50.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 --2- 172.21.15.83:0/494309886 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66e4106650 0x7f66e4106a30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:50.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 -- 172.21.15.83:0/494309886 >> 172.21.15.83:0/494309886 conn(0x7f66e40fc280 msgr2=0x7f66e40fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:50.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 -- 172.21.15.83:0/494309886 shutdown_connections 2024-09-16T00:08:50.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 -- 172.21.15.83:0/494309886 wait complete. 2024-09-16T00:08:50.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 Processor -- start 2024-09-16T00:08:50.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 -- start start 2024-09-16T00:08:50.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66e4106650 0x7f66e4197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:50.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66e4197860 0x7f66e419bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:50.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f66e410ac80 con 0x7f66e4197860 2024-09-16T00:08:50.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.486+0000 7f66ec010640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f66e410ab00 con 0x7f66e4106650 2024-09-16T00:08:50.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e95ab640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66e4197860 0x7f66e419bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:50.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e9dac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66e4106650 0x7f66e4197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:50.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e95ab640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66e4197860 0x7f66e419bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.83:40942/0 (socket says 172.21.15.83:40942) 2024-09-16T00:08:50.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e95ab640 1 -- 172.21.15.83:0/2880076902 learned_addr learned my addr 172.21.15.83:0/2880076902 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-16T00:08:50.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e9dac640 1 -- 172.21.15.83:0/2880076902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66e4106650 msgr2=0x7f66e4197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-16T00:08:50.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e9dac640 1 -- 172.21.15.83:0/2880076902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66e4106650 msgr2=0x7f66e4197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T00:08:50.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e9dac640 1 --2- 172.21.15.83:0/2880076902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66e4106650 0x7f66e4197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T00:08:50.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e9dac640 1 --2- 172.21.15.83:0/2880076902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66e4106650 0x7f66e4197320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T00:08:50.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e95ab640 1 -- 172.21.15.83:0/2880076902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66e4106650 msgr2=0x7f66e4197320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T00:08:50.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e95ab640 1 --2- 172.21.15.83:0/2880076902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66e4106650 0x7f66e4197320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:50.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e95ab640 1 -- 172.21.15.83:0/2880076902 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e4197e80 con 0x7f66e4197860 2024-09-16T00:08:50.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e95ab640 1 --2- 172.21.15.83:0/2880076902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66e4197860 0x7f66e419bbf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f66d800b4a0 tx=0x7f66d800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:50.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e2ffd640 1 -- 172.21.15.83:0/2880076902 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f66d8010610 con 0x7f66e4197860 2024-09-16T00:08:50.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e2ffd640 1 -- 172.21.15.83:0/2880076902 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66d8010c90 con 0x7f66e4197860 2024-09-16T00:08:50.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66ec010640 1 -- 172.21.15.83:0/2880076902 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e419c130 con 0x7f66e4197860 2024-09-16T00:08:50.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66e2ffd640 1 -- 172.21.15.83:0/2880076902 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f66d8017e60 con 0x7f66e4197860 2024-09-16T00:08:50.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.487+0000 7f66ec010640 1 -- 172.21.15.83:0/2880076902 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e419c6f0 con 0x7f66e4197860 2024-09-16T00:08:50.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.488+0000 7f66e2ffd640 1 -- 172.21.15.83:0/2880076902 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f66d8017070 con 0x7f66e4197860 2024-09-16T00:08:50.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.488+0000 7f66ec010640 1 -- 172.21.15.83:0/2880076902 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66e4101cc0 con 0x7f66e4197860 2024-09-16T00:08:50.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.489+0000 7f66e2ffd640 1 --2- 172.21.15.83:0/2880076902 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f66cc039040 0x7f66cc03b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:50.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.489+0000 7f66e2ffd640 1 -- 172.21.15.83:0/2880076902 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f66d800d040 con 0x7f66e4197860 2024-09-16T00:08:50.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.489+0000 7f66e9dac640 1 --2- 172.21.15.83:0/2880076902 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f66cc039040 0x7f66cc03b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:50.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.489+0000 7f66e9dac640 1 --2- 172.21.15.83:0/2880076902 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f66cc039040 0x7f66cc03b500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f66d402b080 tx=0x7f66d4005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:50.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.495+0000 7f66e2ffd640 1 -- 172.21.15.83:0/2880076902 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f66d801e020 con 0x7f66e4197860 2024-09-16T00:08:50.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.721+0000 7f66ec010640 1 -- 172.21.15.83:0/2880076902 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f66e419c9b0 con 0x7f66e4197860 2024-09-16T00:08:50.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.723+0000 7f66e2ffd640 1 -- 172.21.15.83:0/2880076902 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7f66d800e2e0 con 0x7f66e4197860 2024-09-16T00:08:50.726 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:08:50.726 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":2,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","modified":"2024-09-16T00:08:42.123171Z","created":"2024-09-16T00:07:17.720547Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-16T00:08:50.727 INFO:teuthology.orchestra.run.smithi083.stderr:dumped monmap epoch 2 2024-09-16T00:08:50.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.725+0000 7f66ec010640 1 -- 172.21.15.83:0/2880076902 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f66cc039040 msgr2=0x7f66cc03b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:50.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.725+0000 7f66ec010640 1 --2- 172.21.15.83:0/2880076902 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f66cc039040 0x7f66cc03b500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f66d402b080 tx=0x7f66d4005da0 comp rx=0 tx=0).stop 2024-09-16T00:08:50.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.725+0000 7f66ec010640 1 -- 172.21.15.83:0/2880076902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66e4197860 msgr2=0x7f66e419bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:50.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.725+0000 7f66ec010640 1 --2- 172.21.15.83:0/2880076902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66e4197860 0x7f66e419bbf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f66d800b4a0 tx=0x7f66d800b960 comp rx=0 tx=0).stop 2024-09-16T00:08:50.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.725+0000 7f66ec010640 1 -- 172.21.15.83:0/2880076902 shutdown_connections 2024-09-16T00:08:50.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.725+0000 7f66ec010640 1 --2- 172.21.15.83:0/2880076902 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f66cc039040 0x7f66cc03b500 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:50.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.725+0000 7f66ec010640 1 --2- 172.21.15.83:0/2880076902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66e4197860 0x7f66e419bbf0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:50.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.725+0000 7f66ec010640 1 --2- 172.21.15.83:0/2880076902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66e4106650 0x7f66e4197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:50.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.725+0000 7f66ec010640 1 -- 172.21.15.83:0/2880076902 >> 172.21.15.83:0/2880076902 conn(0x7f66e40fc280 msgr2=0x7f66e40fc660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:50.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.725+0000 7f66ec010640 1 -- 172.21.15.83:0/2880076902 shutdown_connections 2024-09-16T00:08:50.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:50.725+0000 7f66ec010640 1 -- 172.21.15.83:0/2880076902 wait complete. 2024-09-16T00:08:51.120 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 podman[35165]: 2024-09-16 00:08:51.119956824 +0000 UTC m=+0.434564653 container init d741ec7069ef54a9f396be3774bfec01933851e43141ebaf6e336041d8dd3123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-16T00:08:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 podman[35165]: 2024-09-16 00:08:51.12314453 +0000 UTC m=+0.437752346 container start d741ec7069ef54a9f396be3774bfec01933851e43141ebaf6e336041d8dd3123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-16T00:08:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: set uid:gid to 167:167 (ceph:ceph) 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable), process ceph-mon, pid 2 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: pidfile_write: ignore empty --pid-file 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: load: jerasure load: lrc 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: RocksDB version: 7.9.2 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Git sha 0 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Compile date 2024-09-15 15:04:35 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: DB SUMMARY 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: DB Session ID: K7MG67GO1GXKQ8SVF3NO 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: CURRENT file: CURRENT 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: IDENTITY file: IDENTITY 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-16T00:08:51.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.error_if_exists: 0 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.create_if_missing: 0 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.paranoid_checks: 1 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.env: 0x5617741c3940 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.fs: PosixFileSystem 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.info_log: 0x5617767776e0 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_file_opening_threads: 16 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.statistics: (nil) 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.use_fsync: 0 2024-09-16T00:08:51.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_log_file_size: 0 2024-09-16T00:08:51.484 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-16T00:08:51.484 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-16T00:08:51.484 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.keep_log_file_num: 1000 2024-09-16T00:08:51.484 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.recycle_log_file_num: 0 2024-09-16T00:08:51.484 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.allow_fallocate: 1 2024-09-16T00:08:51.485 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.allow_mmap_reads: 0 2024-09-16T00:08:51.485 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.allow_mmap_writes: 0 2024-09-16T00:08:51.485 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.use_direct_reads: 0 2024-09-16T00:08:51.485 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-16T00:08:51.485 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.create_missing_column_families: 0 2024-09-16T00:08:51.485 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.db_log_dir: 2024-09-16T00:08:51.485 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.wal_dir: 2024-09-16T00:08:51.485 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-16T00:08:51.486 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-16T00:08:51.486 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-16T00:08:51.486 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-16T00:08:51.486 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-16T00:08:51.486 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-16T00:08:51.486 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.advise_random_on_open: 1 2024-09-16T00:08:51.486 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.db_write_buffer_size: 0 2024-09-16T00:08:51.486 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.write_buffer_manager: 0x561776786dc0 2024-09-16T00:08:51.486 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-16T00:08:51.486 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-16T00:08:51.487 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-16T00:08:51.487 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.rate_limiter: (nil) 2024-09-16T00:08:51.487 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-16T00:08:51.487 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.wal_recovery_mode: 2 2024-09-16T00:08:51.487 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.enable_thread_tracking: 0 2024-09-16T00:08:51.488 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.enable_pipelined_write: 0 2024-09-16T00:08:51.488 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.unordered_write: 0 2024-09-16T00:08:51.488 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-16T00:08:51.488 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-16T00:08:51.488 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-16T00:08:51.488 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-16T00:08:51.489 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.row_cache: None 2024-09-16T00:08:51.489 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.wal_filter: None 2024-09-16T00:08:51.489 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-16T00:08:51.489 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.allow_ingest_behind: 0 2024-09-16T00:08:51.489 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.two_write_queues: 0 2024-09-16T00:08:51.489 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.manual_wal_flush: 0 2024-09-16T00:08:51.490 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.wal_compression: 0 2024-09-16T00:08:51.490 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.atomic_flush: 0 2024-09-16T00:08:51.490 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-16T00:08:51.490 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-16T00:08:51.490 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-16T00:08:51.490 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.log_readahead_size: 0 2024-09-16T00:08:51.491 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-16T00:08:51.491 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.best_efforts_recovery: 0 2024-09-16T00:08:51.491 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-16T00:08:51.491 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-16T00:08:51.491 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.allow_data_in_errors: 0 2024-09-16T00:08:51.492 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.db_host_id: __hostname__ 2024-09-16T00:08:51.492 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-16T00:08:51.492 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_background_jobs: 2 2024-09-16T00:08:51.492 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_background_compactions: -1 2024-09-16T00:08:51.492 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_subcompactions: 1 2024-09-16T00:08:51.492 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-16T00:08:51.493 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-16T00:08:51.493 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-16T00:08:51.493 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_total_wal_size: 0 2024-09-16T00:08:51.493 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-16T00:08:51.493 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-16T00:08:51.493 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-16T00:08:51.494 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-16T00:08:51.494 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_open_files: -1 2024-09-16T00:08:51.494 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bytes_per_sync: 0 2024-09-16T00:08:51.494 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-16T00:08:51.494 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-16T00:08:51.494 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_readahead_size: 0 2024-09-16T00:08:51.495 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_background_flushes: -1 2024-09-16T00:08:51.495 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Compression algorithms supported: 2024-09-16T00:08:51.495 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: kZSTD supported: 0 2024-09-16T00:08:51.495 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: kXpressCompression supported: 0 2024-09-16T00:08:51.495 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: kBZip2Compression supported: 0 2024-09-16T00:08:51.495 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-16T00:08:51.495 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: kLZ4Compression supported: 1 2024-09-16T00:08:51.496 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: kZlibCompression supported: 1 2024-09-16T00:08:51.496 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: kLZ4HCCompression supported: 1 2024-09-16T00:08:51.496 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: kSnappyCompression supported: 1 2024-09-16T00:08:51.496 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-16T00:08:51.496 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-16T00:08:51.496 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-16T00:08:51.496 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-16T00:08:51.496 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-16T00:08:51.496 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.merge_operator: 2024-09-16T00:08:51.497 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_filter: None 2024-09-16T00:08:51.497 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_filter_factory: None 2024-09-16T00:08:51.497 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.sst_partitioner_factory: None 2024-09-16T00:08:51.497 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-16T00:08:51.497 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-16T00:08:51.497 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561776777320) 2024-09-16T00:08:51.497 INFO:journalctl@ceph.mon.b.smithi060.stdout: cache_index_and_filter_blocks: 1 2024-09-16T00:08:51.498 INFO:journalctl@ceph.mon.b.smithi060.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-16T00:08:51.498 INFO:journalctl@ceph.mon.b.smithi060.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-16T00:08:51.498 INFO:journalctl@ceph.mon.b.smithi060.stdout: pin_top_level_index_and_filter: 1 2024-09-16T00:08:51.498 INFO:journalctl@ceph.mon.b.smithi060.stdout: index_type: 0 2024-09-16T00:08:51.498 INFO:journalctl@ceph.mon.b.smithi060.stdout: data_block_index_type: 0 2024-09-16T00:08:51.498 INFO:journalctl@ceph.mon.b.smithi060.stdout: index_shortening: 1 2024-09-16T00:08:51.498 INFO:journalctl@ceph.mon.b.smithi060.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-16T00:08:51.498 INFO:journalctl@ceph.mon.b.smithi060.stdout: checksum: 4 2024-09-16T00:08:51.498 INFO:journalctl@ceph.mon.b.smithi060.stdout: no_block_cache: 0 2024-09-16T00:08:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_cache: 0x56177676b1f0 2024-09-16T00:08:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_cache_name: BinnedLRUCache 2024-09-16T00:08:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_cache_options: 2024-09-16T00:08:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout: capacity : 536870912 2024-09-16T00:08:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout: num_shard_bits : 4 2024-09-16T00:08:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout: strict_capacity_limit : 0 2024-09-16T00:08:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout: high_pri_pool_ratio: 0.000 2024-09-16T00:08:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_cache_compressed: (nil) 2024-09-16T00:08:51.500 INFO:journalctl@ceph.mon.b.smithi060.stdout: persistent_cache: (nil) 2024-09-16T00:08:51.500 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_size: 4096 2024-09-16T00:08:51.500 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_size_deviation: 10 2024-09-16T00:08:51.500 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_restart_interval: 16 2024-09-16T00:08:51.500 INFO:journalctl@ceph.mon.b.smithi060.stdout: index_block_restart_interval: 1 2024-09-16T00:08:51.500 INFO:journalctl@ceph.mon.b.smithi060.stdout: metadata_block_size: 4096 2024-09-16T00:08:51.500 INFO:journalctl@ceph.mon.b.smithi060.stdout: partition_filters: 0 2024-09-16T00:08:51.501 INFO:journalctl@ceph.mon.b.smithi060.stdout: use_delta_encoding: 1 2024-09-16T00:08:51.501 INFO:journalctl@ceph.mon.b.smithi060.stdout: filter_policy: bloomfilter 2024-09-16T00:08:51.501 INFO:journalctl@ceph.mon.b.smithi060.stdout: whole_key_filtering: 1 2024-09-16T00:08:51.501 INFO:journalctl@ceph.mon.b.smithi060.stdout: verify_compression: 0 2024-09-16T00:08:51.501 INFO:journalctl@ceph.mon.b.smithi060.stdout: read_amp_bytes_per_bit: 0 2024-09-16T00:08:51.501 INFO:journalctl@ceph.mon.b.smithi060.stdout: format_version: 5 2024-09-16T00:08:51.501 INFO:journalctl@ceph.mon.b.smithi060.stdout: enable_index_compression: 1 2024-09-16T00:08:51.501 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_align: 0 2024-09-16T00:08:51.501 INFO:journalctl@ceph.mon.b.smithi060.stdout: max_auto_readahead_size: 262144 2024-09-16T00:08:51.501 INFO:journalctl@ceph.mon.b.smithi060.stdout: prepopulate_block_cache: 0 2024-09-16T00:08:51.501 INFO:journalctl@ceph.mon.b.smithi060.stdout: initial_auto_readahead_size: 8192 2024-09-16T00:08:51.501 INFO:journalctl@ceph.mon.b.smithi060.stdout: num_file_reads_for_auto_readahead: 2 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.write_buffer_size: 33554432 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_write_buffer_number: 2 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compression: NoCompression 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bottommost_compression: Disabled 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.prefix_extractor: nullptr 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.num_levels: 7 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-16T00:08:51.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compression_opts.level: 32767 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compression_opts.strategy: 0 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-16T00:08:51.503 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compression_opts.enabled: false 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.target_file_size_base: 67108864 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-16T00:08:51.504 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-16T00:08:51.505 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-16T00:08:51.505 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-16T00:08:51.505 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-16T00:08:51.505 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-16T00:08:51.505 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-16T00:08:51.505 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.arena_block_size: 1048576 2024-09-16T00:08:51.505 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-16T00:08:51.505 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-16T00:08:51.505 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.disable_auto_compactions: 0 2024-09-16T00:08:51.505 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-16T00:08:51.505 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-16T00:08:51.506 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-16T00:08:51.506 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-16T00:08:51.506 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-16T00:08:51.506 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-16T00:08:51.506 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-16T00:08:51.506 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-16T00:08:51.507 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-16T00:08:51.507 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-16T00:08:51.507 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-16T00:08:51.507 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.inplace_update_support: 0 2024-09-16T00:08:51.507 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-16T00:08:51.507 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-16T00:08:51.508 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-16T00:08:51.508 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-16T00:08:51.508 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.bloom_locality: 0 2024-09-16T00:08:51.508 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.max_successive_merges: 0 2024-09-16T00:08:51.508 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-16T00:08:51.508 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.paranoid_file_checks: 0 2024-09-16T00:08:51.508 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.force_consistency_checks: 1 2024-09-16T00:08:51.508 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.report_bg_io_stats: 0 2024-09-16T00:08:51.508 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.ttl: 2592000 2024-09-16T00:08:51.508 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-16T00:08:51.509 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-16T00:08:51.509 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-16T00:08:51.509 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.enable_blob_files: false 2024-09-16T00:08:51.509 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.min_blob_size: 0 2024-09-16T00:08:51.509 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.blob_file_size: 268435456 2024-09-16T00:08:51.509 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-16T00:08:51.509 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-16T00:08:51.509 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-16T00:08:51.509 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-16T00:08:51.510 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.blob_file_starting_level: 0 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ee5277b2-bbae-4f58-a1b9-3e170046c311 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726445331150023, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726445331150440, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726445331, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ee5277b2-bbae-4f58-a1b9-3e170046c311", "db_session_id": "K7MG67GO1GXKQ8SVF3NO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726445331150539, "job": 1, "event": "recovery_finished"} 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561776798e00 2024-09-16T00:08:51.511 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: DB pointer 0x561776894000 2024-09-16T00:08:51.512 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-16T00:08:51.512 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-16T00:08:51.512 INFO:journalctl@ceph.mon.b.smithi060.stdout: ** DB Stats ** 2024-09-16T00:08:51.512 INFO:journalctl@ceph.mon.b.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T00:08:51.512 INFO:journalctl@ceph.mon.b.smithi060.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-16T00:08:51.513 INFO:journalctl@ceph.mon.b.smithi060.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T00:08:51.513 INFO:journalctl@ceph.mon.b.smithi060.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T00:08:51.513 INFO:journalctl@ceph.mon.b.smithi060.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-16T00:08:51.513 INFO:journalctl@ceph.mon.b.smithi060.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T00:08:51.513 INFO:journalctl@ceph.mon.b.smithi060.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T00:08:51.513 INFO:journalctl@ceph.mon.b.smithi060.stdout: 2024-09-16T00:08:51.513 INFO:journalctl@ceph.mon.b.smithi060.stdout: ** Compaction Stats [default] ** 2024-09-16T00:08:51.513 INFO:journalctl@ceph.mon.b.smithi060.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T00:08:51.513 INFO:journalctl@ceph.mon.b.smithi060.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-16T00:08:51.513 INFO:journalctl@ceph.mon.b.smithi060.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: ** Compaction Stats [default] ** 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-16T00:08:51.514 INFO:journalctl@ceph.mon.b.smithi060.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-16T00:08:51.515 INFO:journalctl@ceph.mon.b.smithi060.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-16T00:08:51.515 INFO:journalctl@ceph.mon.b.smithi060.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-16T00:08:51.515 INFO:journalctl@ceph.mon.b.smithi060.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-16T00:08:51.515 INFO:journalctl@ceph.mon.b.smithi060.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T00:08:51.515 INFO:journalctl@ceph.mon.b.smithi060.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T00:08:51.515 INFO:journalctl@ceph.mon.b.smithi060.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-16T00:08:51.515 INFO:journalctl@ceph.mon.b.smithi060.stdout: Block cache BinnedLRUCache@0x56177676b1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-16T00:08:51.515 INFO:journalctl@ceph.mon.b.smithi060.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-16T00:08:51.515 INFO:journalctl@ceph.mon.b.smithi060.stdout: 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: using public_addr v2:172.21.15.60:0/0 -> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: starting mon.b rank -1 at public addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] at bind addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(???) e0 preinit fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).mds e1 new map 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout: e1 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout: btime 2024-09-16T00:07:21:167536+0000 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout: legacy client fscid: -1 2024-09-16T00:08:51.516 INFO:journalctl@ceph.mon.b.smithi060.stdout: 2024-09-16T00:08:51.517 INFO:journalctl@ceph.mon.b.smithi060.stdout: No filesystems configured 2024-09-16T00:08:51.517 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-16T00:08:51.517 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-16T00:08:51.518 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-16T00:08:51.518 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-16T00:08:51.518 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-16T00:08:51.518 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-16T00:08:51.518 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-16T00:08:51.518 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T00:08:51.518 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T00:08:51.518 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T00:08:51.519 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:51.519 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:08:51.519 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3304522224' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-16T00:08:51.519 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:51.519 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3304522224' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-16T00:08:51.519 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T00:08:51.519 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.519 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.519 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.520 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.520 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi083", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:08:51.520 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:51.520 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:08:51.520 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: Updating smithi083:/etc/ceph/ceph.conf 2024-09-16T00:08:51.520 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: Updating smithi083:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:51.520 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.521 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.521 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.521 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.521 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.521 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.521 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.521 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.521 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:08:51.521 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T00:08:51.522 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:51.522 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:51.522 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-09-16T00:08:51.522 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: Updating smithi083:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T00:08:51.522 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi033:172.21.15.33=a;smithi060:172.21.15.60=b;smithi083:172.21.15.83=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:08:51.522 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: Saving service mon spec with placement smithi033:172.21.15.33=a;smithi060:172.21.15.60=b;smithi083:172.21.15.83=c;count:3 2024-09-16T00:08:51.522 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T00:08:51.522 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: Reconfiguring daemon mon.a on smithi033 2024-09-16T00:08:51.523 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.523 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.523 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:08:51.523 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:51.523 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:08:51.523 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.523 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:08:51.523 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:51.524 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.83:0/2199210762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T00:08:51.524 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: Deploying daemon mon.c on smithi083 2024-09-16T00:08:51.524 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:51.524 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:51.524 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.83:0/2945113963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T00:08:51.524 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:51.524 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:08:51.524 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.a calling monitor election 2024-09-16T00:08:51.524 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:51.524 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:51.524 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:51.524 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.c calling monitor election 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: monmap epoch 2 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: last_changed 2024-09-16T00:08:42.123171+0000 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: created 2024-09-16T00:07:17.720547+0000 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: min_mon_release 19 (squid) 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: election_strategy: 1 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-09-16T00:08:51.525 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.c 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: fsmap 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mgrmap e13: x(active, since 39s) 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: overall HEALTH_OK 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: Deploying daemon mon.b on smithi060 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.83:0/2690877294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:51.526 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 ceph-mon[35185]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-16T00:08:51.527 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 bash[35165]: d741ec7069ef54a9f396be3774bfec01933851e43141ebaf6e336041d8dd3123 2024-09-16T00:08:51.527 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:51 smithi060 systemd[1]: Started Ceph mon.b for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:08:52.365 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T00:08:52.366 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T00:08:52.614 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:08:56.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.228+0000 7f411fc29640 1 -- 172.21.15.83:0/3901250105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4104003530 msgr2=0x7f41040059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:56.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.228+0000 7f411fc29640 1 --2- 172.21.15.83:0/3901250105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4104003530 0x7f41040059c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4118075a20 tx=0x7f411402c150 comp rx=0 tx=0).stop 2024-09-16T00:08:56.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.228+0000 7f411fc29640 1 -- 172.21.15.83:0/3901250105 shutdown_connections 2024-09-16T00:08:56.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.228+0000 7f411fc29640 1 --2- 172.21.15.83:0/3901250105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4104003530 0x7f41040059c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:56.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.228+0000 7f411fc29640 1 -- 172.21.15.83:0/3901250105 >> 172.21.15.83:0/3901250105 conn(0x7f41180fd320 msgr2=0x7f41180ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:56.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.228+0000 7f411fc29640 1 -- 172.21.15.83:0/3901250105 shutdown_connections 2024-09-16T00:08:56.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.229+0000 7f411fc29640 1 -- 172.21.15.83:0/3901250105 wait complete. 2024-09-16T00:08:56.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.229+0000 7f411fc29640 1 Processor -- start 2024-09-16T00:08:56.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.229+0000 7f411fc29640 1 -- start start 2024-09-16T00:08:56.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411fc29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4104003530 0x7f41181a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:56.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411fc29640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41181a3b70 0x7f411819d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:56.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411fc29640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f411819dcf0 0x7f411819e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:56.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411fc29640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f411810fd70 con 0x7f411819dcf0 2024-09-16T00:08:56.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411fc29640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f411810fbf0 con 0x7f41181a3b70 2024-09-16T00:08:56.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411fc29640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f411810fef0 con 0x7f4104003530 2024-09-16T00:08:56.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411d9c5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4104003530 0x7f41181a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:56.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411d9c5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4104003530 0x7f41181a3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56746/0 (socket says 172.21.15.83:56746) 2024-09-16T00:08:56.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411d9c5640 1 -- 172.21.15.83:0/2711616772 learned_addr learned my addr 172.21.15.83:0/2711616772 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-16T00:08:56.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411e1c6640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f411819dcf0 0x7f411819e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:56.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411d9c5640 1 -- 172.21.15.83:0/2711616772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4104003530 msgr2=0x7f41181a3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-16T00:08:56.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411d9c5640 1 -- 172.21.15.83:0/2711616772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4104003530 msgr2=0x7f41181a3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T00:08:56.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411d1c4640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41181a3b70 0x7f411819d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411d9c5640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4104003530 0x7f41181a3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411d9c5640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4104003530 0x7f41181a3630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411d1c4640 1 -- 172.21.15.83:0/2711616772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41181a3b70 msgr2=0x7f411819d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411d1c4640 1 -- 172.21.15.83:0/2711616772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41181a3b70 msgr2=0x7f411819d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411d1c4640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41181a3b70 0x7f411819d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411d1c4640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41181a3b70 0x7f411819d7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411e1c6640 1 -- 172.21.15.83:0/2711616772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4104003530 msgr2=0x7f41181a3630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411e1c6640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4104003530 0x7f41181a3630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411e1c6640 1 -- 172.21.15.83:0/2711616772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41181a3b70 msgr2=0x7f411819d7b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411e1c6640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41181a3b70 0x7f411819d7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.230+0000 7f411e1c6640 1 -- 172.21.15.83:0/2711616772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41181aa490 con 0x7f411819dcf0 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.231+0000 7f411e1c6640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f411819dcf0 0x7f411819e170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f411000bd50 tx=0x7f411000be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:56.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.231+0000 7f410effd640 1 -- 172.21.15.83:0/2711616772 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110019070 con 0x7f411819dcf0 2024-09-16T00:08:56.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.231+0000 7f410effd640 1 -- 172.21.15.83:0/2711616772 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4110014920 con 0x7f411819dcf0 2024-09-16T00:08:56.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.231+0000 7f411fc29640 1 -- 172.21.15.83:0/2711616772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41181aa6c0 con 0x7f411819dcf0 2024-09-16T00:08:56.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.231+0000 7f410effd640 1 -- 172.21.15.83:0/2711616772 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110012990 con 0x7f411819dcf0 2024-09-16T00:08:56.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.231+0000 7f411fc29640 1 -- 172.21.15.83:0/2711616772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41181aac00 con 0x7f411819dcf0 2024-09-16T00:08:56.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.232+0000 7f410effd640 1 -- 172.21.15.83:0/2711616772 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f411001c070 con 0x7f411819dcf0 2024-09-16T00:08:56.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.232+0000 7f410effd640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40fc039730 0x7f40fc03bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:56.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.232+0000 7f410effd640 1 -- 172.21.15.83:0/2711616772 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4110014440 con 0x7f411819dcf0 2024-09-16T00:08:56.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.232+0000 7f411d9c5640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40fc039730 0x7f40fc03bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:56.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.233+0000 7f411fc29640 1 -- 172.21.15.83:0/2711616772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4118075cd0 con 0x7f411819dcf0 2024-09-16T00:08:56.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.233+0000 7f411d9c5640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40fc039730 0x7f40fc03bbf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4114005fd0 tx=0x7f4114005d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:56.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.238+0000 7f410effd640 1 -- 172.21.15.83:0/2711616772 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4110013c30 con 0x7f411819dcf0 2024-09-16T00:08:56.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.481+0000 7f411fc29640 1 -- 172.21.15.83:0/2711616772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f41181aaf20 con 0x7f411819dcf0 2024-09-16T00:08:56.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:08:56.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:56.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: mon.a calling monitor election 2024-09-16T00:08:56.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: mon.c calling monitor election 2024-09-16T00:08:56.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:56.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:56.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:56.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:56.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:56.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:56.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:56.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: monmap epoch 3 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: last_changed 2024-09-16T00:08:51.174408+0000 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: created 2024-09-16T00:07:17.720547+0000 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: min_mon_release 19 (squid) 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: election_strategy: 1 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.c 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: 2: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: fsmap 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T00:08:56.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: mgrmap e13: x(active, since 48s) 2024-09-16T00:08:56.485 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: overall HEALTH_OK 2024-09-16T00:08:56.485 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:56.485 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:56.485 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:56.485 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:56.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.481+0000 7f410effd640 1 -- 172.21.15.83:0/2711616772 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1275 (secure 0 0 0) 0x7f4110014640 con 0x7f411819dcf0 2024-09-16T00:08:56.485 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:08:56.485 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":3,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","modified":"2024-09-16T00:08:51.174408Z","created":"2024-09-16T00:07:17.720547Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-16T00:08:56.486 INFO:teuthology.orchestra.run.smithi083.stderr:dumped monmap epoch 3 2024-09-16T00:08:56.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.484+0000 7f411fc29640 1 -- 172.21.15.83:0/2711616772 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40fc039730 msgr2=0x7f40fc03bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:56.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.484+0000 7f411fc29640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40fc039730 0x7f40fc03bbf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4114005fd0 tx=0x7f4114005d50 comp rx=0 tx=0).stop 2024-09-16T00:08:56.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.484+0000 7f411fc29640 1 -- 172.21.15.83:0/2711616772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f411819dcf0 msgr2=0x7f411819e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:56.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.484+0000 7f411fc29640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f411819dcf0 0x7f411819e170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f411000bd50 tx=0x7f411000be40 comp rx=0 tx=0).stop 2024-09-16T00:08:56.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.484+0000 7f411fc29640 1 -- 172.21.15.83:0/2711616772 shutdown_connections 2024-09-16T00:08:56.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.484+0000 7f411fc29640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40fc039730 0x7f40fc03bbf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:56.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.484+0000 7f411fc29640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f411819dcf0 0x7f411819e170 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:56.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.484+0000 7f411fc29640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41181a3b70 0x7f411819d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:56.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.484+0000 7f411fc29640 1 --2- 172.21.15.83:0/2711616772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4104003530 0x7f41181a3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:56.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.484+0000 7f411fc29640 1 -- 172.21.15.83:0/2711616772 >> 172.21.15.83:0/2711616772 conn(0x7f41180fd320 msgr2=0x7f4118101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:56.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.484+0000 7f411fc29640 1 -- 172.21.15.83:0/2711616772 shutdown_connections 2024-09-16T00:08:56.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:56.484+0000 7f411fc29640 1 -- 172.21.15.83:0/2711616772 wait complete. 2024-09-16T00:08:56.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:08:56.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:56.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: mon.a calling monitor election 2024-09-16T00:08:56.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: mon.c calling monitor election 2024-09-16T00:08:56.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:56.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:56.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:56.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:56.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:56.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:56.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:56.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:56.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T00:08:56.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: monmap epoch 3 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: last_changed 2024-09-16T00:08:51.174408+0000 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: created 2024-09-16T00:07:17.720547+0000 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: min_mon_release 19 (squid) 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: election_strategy: 1 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.c 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: 2: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: fsmap 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: mgrmap e13: x(active, since 48s) 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: overall HEALTH_OK 2024-09-16T00:08:56.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:56.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:56.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:56.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:57.080 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-16T00:08:57.080 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-16T00:08:57.345 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:08:57.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:57 smithi083 ceph-mon[35242]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:57.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:08:57.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:57 smithi083 ceph-mon[35242]: from='client.? 172.21.15.83:0/2711616772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T00:08:57.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:57.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:57 smithi033 ceph-mon[31243]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:57.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:08:57.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:57 smithi033 ceph-mon[31243]: from='client.? 172.21.15.83:0/2711616772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T00:08:57.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:57.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.960+0000 7fae2c98f640 1 -- 172.21.15.33:0/3466702632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae28102850 msgr2=0x7fae28102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:57.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.960+0000 7fae2c98f640 1 --2- 172.21.15.33:0/3466702632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae28102850 0x7fae28102c30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fae18009880 tx=0x7fae1802a670 comp rx=0 tx=0).stop 2024-09-16T00:08:57.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.960+0000 7fae2c98f640 1 -- 172.21.15.33:0/3466702632 shutdown_connections 2024-09-16T00:08:57.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.960+0000 7fae2c98f640 1 --2- 172.21.15.33:0/3466702632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae28102850 0x7fae28102c30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:57.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.960+0000 7fae2c98f640 1 -- 172.21.15.33:0/3466702632 >> 172.21.15.33:0/3466702632 conn(0x7fae280fc250 msgr2=0x7fae280fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:57.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.960+0000 7fae2c98f640 1 -- 172.21.15.33:0/3466702632 shutdown_connections 2024-09-16T00:08:57.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.960+0000 7fae2c98f640 1 -- 172.21.15.33:0/3466702632 wait complete. 2024-09-16T00:08:57.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.961+0000 7fae2c98f640 1 Processor -- start 2024-09-16T00:08:57.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.961+0000 7fae2c98f640 1 -- start start 2024-09-16T00:08:57.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.961+0000 7fae2c98f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae28102850 0x7fae281004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:57.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.961+0000 7fae2c98f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae281009f0 0x7fae28100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:57.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.961+0000 7fae2c98f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae281a3710 0x7fae281a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:57.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.961+0000 7fae2c98f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae2810bdc0 con 0x7fae281a3710 2024-09-16T00:08:57.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.961+0000 7fae2c98f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fae2810bc40 con 0x7fae281009f0 2024-09-16T00:08:57.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.961+0000 7fae2c98f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae2810bf40 con 0x7fae28102850 2024-09-16T00:08:57.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae2659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae28102850 0x7fae281004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:57.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae2659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae28102850 0x7fae281004b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:49574/0 (socket says 172.21.15.33:49574) 2024-09-16T00:08:57.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae25d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae281009f0 0x7fae28100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:57.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae2659c640 1 -- 172.21.15.33:0/1970300049 learned_addr learned my addr 172.21.15.33:0/1970300049 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:08:57.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae26d9d640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae281a3710 0x7fae281a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:57.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae25d9b640 1 -- 172.21.15.33:0/1970300049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae281009f0 msgr2=0x7fae28100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T00:08:57.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae25d9b640 1 -- 172.21.15.33:0/1970300049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae281009f0 msgr2=0x7fae28100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T00:08:57.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae25d9b640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae281009f0 0x7fae28100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T00:08:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae25d9b640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae281009f0 0x7fae28100e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T00:08:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae26d9d640 1 -- 172.21.15.33:0/1970300049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae28102850 msgr2=0x7fae281004b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae26d9d640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae28102850 0x7fae281004b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae26d9d640 1 -- 172.21.15.33:0/1970300049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae281009f0 msgr2=0x7fae28100e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T00:08:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae26d9d640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae281009f0 0x7fae28100e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae26d9d640 1 -- 172.21.15.33:0/1970300049 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae281a61d0 con 0x7fae281a3710 2024-09-16T00:08:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae26d9d640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae281a3710 0x7fae281a5b00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fae2000bd50 tx=0x7fae2000be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae177fe640 1 -- 172.21.15.33:0/1970300049 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20019070 con 0x7fae281a3710 2024-09-16T00:08:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae177fe640 1 -- 172.21.15.33:0/1970300049 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae20014920 con 0x7fae281a3710 2024-09-16T00:08:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae2c98f640 1 -- 172.21.15.33:0/1970300049 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae281a64c0 con 0x7fae281a3710 2024-09-16T00:08:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.962+0000 7fae177fe640 1 -- 172.21.15.33:0/1970300049 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20012990 con 0x7fae281a3710 2024-09-16T00:08:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.963+0000 7fae2c98f640 1 -- 172.21.15.33:0/1970300049 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae281a6a00 con 0x7fae281a3710 2024-09-16T00:08:58.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.963+0000 7fae177fe640 1 -- 172.21.15.33:0/1970300049 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fae2001c070 con 0x7fae281a3710 2024-09-16T00:08:58.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.964+0000 7fae177fe640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae10039730 0x7fae1003bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:58.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.964+0000 7fae177fe640 1 -- 172.21.15.33:0/1970300049 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fae2004cf80 con 0x7fae281a3710 2024-09-16T00:08:58.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.964+0000 7fae2659c640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae10039730 0x7fae1003bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:58.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.964+0000 7fae2c98f640 1 -- 172.21.15.33:0/1970300049 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae28102cb0 con 0x7fae281a3710 2024-09-16T00:08:58.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.964+0000 7fae2659c640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae10039730 0x7fae1003bbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fae180096c0 tx=0x7fae1802ab70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:58.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:57.969+0000 7fae177fe640 1 -- 172.21.15.33:0/1970300049 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fae200139f0 con 0x7fae281a3710 2024-09-16T00:08:58.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.159+0000 7fae2c98f640 1 -- 172.21.15.33:0/1970300049 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fae281a6d20 con 0x7fae281a3710 2024-09-16T00:08:58.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.159+0000 7fae177fe640 1 -- 172.21.15.33:0/1970300049 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+271 (secure 0 0 0) 0x7fae20013c50 con 0x7fae281a3710 2024-09-16T00:08:58.161 INFO:teuthology.orchestra.run.smithi033.stdout:# minimal ceph.conf for 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:58.161 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2024-09-16T00:08:58.161 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:58.161 INFO:teuthology.orchestra.run.smithi033.stdout: mon_host = [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-09-16T00:08:58.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.162+0000 7fae2c98f640 1 -- 172.21.15.33:0/1970300049 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae10039730 msgr2=0x7fae1003bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:58.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.162+0000 7fae2c98f640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae10039730 0x7fae1003bbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fae180096c0 tx=0x7fae1802ab70 comp rx=0 tx=0).stop 2024-09-16T00:08:58.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.162+0000 7fae2c98f640 1 -- 172.21.15.33:0/1970300049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae281a3710 msgr2=0x7fae281a5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:58.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.162+0000 7fae2c98f640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae281a3710 0x7fae281a5b00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fae2000bd50 tx=0x7fae2000be40 comp rx=0 tx=0).stop 2024-09-16T00:08:58.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.162+0000 7fae2c98f640 1 -- 172.21.15.33:0/1970300049 shutdown_connections 2024-09-16T00:08:58.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.162+0000 7fae2c98f640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae10039730 0x7fae1003bbf0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:58.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.162+0000 7fae2c98f640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae281a3710 0x7fae281a5b00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:58.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.162+0000 7fae2c98f640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae281009f0 0x7fae28100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:58.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.162+0000 7fae2c98f640 1 --2- 172.21.15.33:0/1970300049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae28102850 0x7fae281004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:58.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.162+0000 7fae2c98f640 1 -- 172.21.15.33:0/1970300049 >> 172.21.15.33:0/1970300049 conn(0x7fae280fc250 msgr2=0x7fae28105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:58.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.162+0000 7fae2c98f640 1 -- 172.21.15.33:0/1970300049 shutdown_connections 2024-09-16T00:08:58.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:08:58.162+0000 7fae2c98f640 1 -- 172.21.15.33:0/1970300049 wait complete. 2024-09-16T00:08:58.429 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:08:58.430 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:58.430 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: mon.a calling monitor election 2024-09-16T00:08:58.430 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: mon.c calling monitor election 2024-09-16T00:08:58.430 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:08:58.430 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:58.430 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:58.430 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:58.430 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:58.430 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:58.430 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:58.430 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: monmap epoch 3 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: last_changed 2024-09-16T00:08:51.174408+0000 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: created 2024-09-16T00:07:17.720547+0000 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: min_mon_release 19 (squid) 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: election_strategy: 1 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.c 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: 2: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: fsmap 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: mgrmap e13: x(active, since 48s) 2024-09-16T00:08:58.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: overall HEALTH_OK 2024-09-16T00:08:58.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:58.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:58.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:58.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:58.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:58.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:08:58.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='client.? 172.21.15.83:0/2711616772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T00:08:58.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:08:58.825 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-16T00:08:58.825 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:08:58.825 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T00:08:58.857 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:08:58.857 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T00:08:58.923 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-16T00:08:58.923 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T00:08:58.952 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-16T00:08:58.953 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T00:08:59.018 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T00:08:59.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T00:08:59.048 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T00:08:59.048 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T00:08:59.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: mon.b calling monitor election 2024-09-16T00:08:59.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:59.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: Updating smithi033:/etc/ceph/ceph.conf 2024-09-16T00:08:59.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: Updating smithi060:/etc/ceph/ceph.conf 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: Updating smithi083:/etc/ceph/ceph.conf 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: Updating smithi083:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: Updating smithi060:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: Updating smithi033:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: mon.b calling monitor election 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: mon.c calling monitor election 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: mon.a calling monitor election 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: monmap epoch 3 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: last_changed 2024-09-16T00:08:51.174408+0000 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: created 2024-09-16T00:07:17.720547+0000 2024-09-16T00:08:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: min_mon_release 19 (squid) 2024-09-16T00:08:59.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: election_strategy: 1 2024-09-16T00:08:59.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-09-16T00:08:59.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.c 2024-09-16T00:08:59.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: 2: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-09-16T00:08:59.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: fsmap 2024-09-16T00:08:59.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T00:08:59.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: mgrmap e13: x(active, since 50s) 2024-09-16T00:08:59.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: overall HEALTH_OK 2024-09-16T00:08:59.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.083 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.083 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.083 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.083 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: Reconfiguring mon.a (monmap changed)... 2024-09-16T00:08:59.083 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:08:59.083 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T00:08:59.083 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:59.113 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: mon.b calling monitor election 2024-09-16T00:08:59.113 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:59.113 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: Updating smithi033:/etc/ceph/ceph.conf 2024-09-16T00:08:59.113 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: Updating smithi060:/etc/ceph/ceph.conf 2024-09-16T00:08:59.113 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: Updating smithi083:/etc/ceph/ceph.conf 2024-09-16T00:08:59.113 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: Updating smithi083:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: Updating smithi060:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: Updating smithi033:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: mon.b calling monitor election 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: mon.c calling monitor election 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: mon.a calling monitor election 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: monmap epoch 3 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: last_changed 2024-09-16T00:08:51.174408+0000 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: created 2024-09-16T00:07:17.720547+0000 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: min_mon_release 19 (squid) 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: election_strategy: 1 2024-09-16T00:08:59.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.c 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: 2: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: fsmap 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: mgrmap e13: x(active, since 50s) 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: overall HEALTH_OK 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: Reconfiguring mon.a (monmap changed)... 2024-09-16T00:08:59.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:08:59.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T00:08:59.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:59.117 INFO:tasks.cephadm:Adding mgr.x on smithi033 2024-09-16T00:08:59.117 INFO:tasks.cephadm:Adding mgr.y on smithi060 2024-09-16T00:08:59.117 INFO:tasks.cephadm:Adding mgr.z on smithi083 2024-09-16T00:08:59.117 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch apply mgr '3;smithi033=x;smithi060=y;smithi083=z' 2024-09-16T00:08:59.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: mon.b calling monitor election 2024-09-16T00:08:59.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:08:59.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: Updating smithi033:/etc/ceph/ceph.conf 2024-09-16T00:08:59.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: Updating smithi060:/etc/ceph/ceph.conf 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: Updating smithi083:/etc/ceph/ceph.conf 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: Updating smithi083:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: Updating smithi060:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: Updating smithi033:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: mon.b calling monitor election 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: mon.c calling monitor election 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: mon.a calling monitor election 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: monmap epoch 3 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: last_changed 2024-09-16T00:08:51.174408+0000 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: created 2024-09-16T00:07:17.720547+0000 2024-09-16T00:08:59.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: min_mon_release 19 (squid) 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: election_strategy: 1 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.c 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: 2: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: fsmap 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: mgrmap e13: x(active, since 50s) 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: overall HEALTH_OK 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.234 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.234 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:08:59.234 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: Reconfiguring mon.a (monmap changed)... 2024-09-16T00:08:59.234 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:08:59.234 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T00:08:59.234 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:08:59.397 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:08:59.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.764+0000 7f280c94b640 1 -- 172.21.15.83:0/4226922431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28081016e0 msgr2=0x7f2808101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:59.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.764+0000 7f280c94b640 1 --2- 172.21.15.83:0/4226922431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28081016e0 0x7f2808101ae0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009880 tx=0x7f27f802a5d0 comp rx=0 tx=0).stop 2024-09-16T00:08:59.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.764+0000 7f280c94b640 1 -- 172.21.15.83:0/4226922431 shutdown_connections 2024-09-16T00:08:59.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.764+0000 7f280c94b640 1 --2- 172.21.15.83:0/4226922431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28081016e0 0x7f2808101ae0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:59.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.764+0000 7f280c94b640 1 -- 172.21.15.83:0/4226922431 >> 172.21.15.83:0/4226922431 conn(0x7f28080fce90 msgr2=0x7f28080ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:59.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.764+0000 7f280c94b640 1 -- 172.21.15.83:0/4226922431 shutdown_connections 2024-09-16T00:08:59.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.764+0000 7f280c94b640 1 -- 172.21.15.83:0/4226922431 wait complete. 2024-09-16T00:08:59.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280c94b640 1 Processor -- start 2024-09-16T00:08:59.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280c94b640 1 -- start start 2024-09-16T00:08:59.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280c94b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28081016e0 0x7f2808075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:59.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280c94b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28080745e0 0x7f2808074a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:59.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280c94b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2808074fa0 0x7f28081a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:59.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280c94b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f28081077d0 con 0x7f28081016e0 2024-09-16T00:08:59.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280c94b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2808107650 con 0x7f28080745e0 2024-09-16T00:08:59.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280c94b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2808107950 con 0x7f2808074fa0 2024-09-16T00:08:59.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28081016e0 0x7f2808075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:59.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28081016e0 0x7f2808075f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.83:35266/0 (socket says 172.21.15.83:35266) 2024-09-16T00:08:59.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280659c640 1 -- 172.21.15.83:0/704142520 learned_addr learned my addr 172.21.15.83:0/704142520 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-16T00:08:59.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f2806d9d640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2808074fa0 0x7f28081a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:59.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f2805d9b640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28080745e0 0x7f2808074a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:59.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280659c640 1 -- 172.21.15.83:0/704142520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2808074fa0 msgr2=0x7f28081a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:59.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280659c640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2808074fa0 0x7f28081a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:59.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280659c640 1 -- 172.21.15.83:0/704142520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28080745e0 msgr2=0x7f2808074a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:59.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280659c640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28080745e0 0x7f2808074a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:59.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280659c640 1 -- 172.21.15.83:0/704142520 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28081a15d0 con 0x7f28081016e0 2024-09-16T00:08:59.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f2805d9b640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28080745e0 0x7f2808074a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:08:59.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.765+0000 7f280659c640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28081016e0 0x7f2808075f90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f27f8031610 tx=0x7f27f80043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:59.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.766+0000 7f27f77fe640 1 -- 172.21.15.83:0/704142520 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f803f030 con 0x7f28081016e0 2024-09-16T00:08:59.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.766+0000 7f27f77fe640 1 -- 172.21.15.83:0/704142520 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27f8003ce0 con 0x7f28081016e0 2024-09-16T00:08:59.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.766+0000 7f280c94b640 1 -- 172.21.15.83:0/704142520 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28081a1860 con 0x7f28081016e0 2024-09-16T00:08:59.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.766+0000 7f27f77fe640 1 -- 172.21.15.83:0/704142520 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f8035070 con 0x7f28081016e0 2024-09-16T00:08:59.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.766+0000 7f280c94b640 1 -- 172.21.15.83:0/704142520 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28081a1dc0 con 0x7f28081016e0 2024-09-16T00:08:59.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.767+0000 7f27f77fe640 1 -- 172.21.15.83:0/704142520 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f27f8042030 con 0x7f28081016e0 2024-09-16T00:08:59.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.767+0000 7f280c94b640 1 -- 172.21.15.83:0/704142520 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2808190590 con 0x7f28081016e0 2024-09-16T00:08:59.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.767+0000 7f27f77fe640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f27f0039730 0x7f27f003bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:08:59.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.767+0000 7f27f77fe640 1 -- 172.21.15.83:0/704142520 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f27f806cf80 con 0x7f28081016e0 2024-09-16T00:08:59.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.767+0000 7f2805d9b640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f27f0039730 0x7f27f003bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:08:59.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.768+0000 7f2805d9b640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f27f0039730 0x7f27f003bbf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f27fc006fd0 tx=0x7f27fc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:08:59.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.772+0000 7f27f77fe640 1 -- 172.21.15.83:0/704142520 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f27f803bde0 con 0x7f28081016e0 2024-09-16T00:08:59.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.950+0000 7f280c94b640 1 -- 172.21.15.83:0/704142520 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi033=x;smithi060=y;smithi083=z", "target": ["mon-mgr", ""]}) -- 0x7f28080626c0 con 0x7f27f0039730 2024-09-16T00:08:59.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.955+0000 7f27f77fe640 1 -- 172.21.15.83:0/704142520 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f28080626c0 con 0x7f27f0039730 2024-09-16T00:08:59.956 INFO:teuthology.orchestra.run.smithi083.stdout:Scheduled mgr update... 2024-09-16T00:08:59.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.957+0000 7f280c94b640 1 -- 172.21.15.83:0/704142520 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f27f0039730 msgr2=0x7f27f003bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:59.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.957+0000 7f280c94b640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f27f0039730 0x7f27f003bbf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f27fc006fd0 tx=0x7f27fc008040 comp rx=0 tx=0).stop 2024-09-16T00:08:59.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.957+0000 7f280c94b640 1 -- 172.21.15.83:0/704142520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28081016e0 msgr2=0x7f2808075f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:08:59.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.957+0000 7f280c94b640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28081016e0 0x7f2808075f90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f27f8031610 tx=0x7f27f80043c0 comp rx=0 tx=0).stop 2024-09-16T00:08:59.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.957+0000 7f280c94b640 1 -- 172.21.15.83:0/704142520 shutdown_connections 2024-09-16T00:08:59.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.957+0000 7f280c94b640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f27f0039730 0x7f27f003bbf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:59.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.957+0000 7f280c94b640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2808074fa0 0x7f28081a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:59.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.957+0000 7f280c94b640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28080745e0 0x7f2808074a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:59.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.957+0000 7f280c94b640 1 --2- 172.21.15.83:0/704142520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28081016e0 0x7f2808075f90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:08:59.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.957+0000 7f280c94b640 1 -- 172.21.15.83:0/704142520 >> 172.21.15.83:0/704142520 conn(0x7f28080fce90 msgr2=0x7f28080fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:08:59.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.958+0000 7f280c94b640 1 -- 172.21.15.83:0/704142520 shutdown_connections 2024-09-16T00:08:59.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:08:59.958+0000 7f280c94b640 1 -- 172.21.15.83:0/704142520 wait complete. 2024-09-16T00:09:00.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:59 smithi033 ceph-mon[31243]: Reconfiguring daemon mon.a on smithi033 2024-09-16T00:09:00.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:08:59 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:09:00.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:59 smithi083 ceph-mon[35242]: Reconfiguring daemon mon.a on smithi033 2024-09-16T00:09:00.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:08:59 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:09:00.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:59 smithi060 ceph-mon[35185]: Reconfiguring daemon mon.a on smithi033 2024-09-16T00:09:00.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:08:59 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:09:00.525 DEBUG:teuthology.orchestra.run.smithi060:mgr.y> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.y.service 2024-09-16T00:09:00.528 DEBUG:teuthology.orchestra.run.smithi083:mgr.z> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.z.service 2024-09-16T00:09:00.530 INFO:tasks.cephadm:Deploying OSDs... 2024-09-16T00:09:00.530 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:09:00.530 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T00:09:00.551 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T00:09:00.551 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2024-09-16T00:09:00.611 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T00:09:00.611 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T00:09:00.612 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T00:09:00.612 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:09:00.612 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T00:09:00.612 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-09-16 00:08:16.201399590 +0000 2024-09-16T00:09:00.612 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-09-16 00:04:02.160893451 +0000 2024-09-16T00:09:00.612 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-09-16 00:04:02.160893451 +0000 2024-09-16T00:09:00.612 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-09-16 00:04:02.160893451 +0000 2024-09-16T00:09:00.613 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T00:09:00.682 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-09-16T00:09:00.682 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-09-16T00:09:00.682 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000221191 s, 2.3 MB/s 2024-09-16T00:09:00.684 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T00:09:00.744 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2024-09-16T00:09:00.802 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T00:09:00.802 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T00:09:00.802 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T00:09:00.802 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:09:00.802 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T00:09:00.803 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-09-16 00:08:16.236399013 +0000 2024-09-16T00:09:00.803 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-09-16 00:04:02.498887747 +0000 2024-09-16T00:09:00.803 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-09-16 00:04:02.498887747 +0000 2024-09-16T00:09:00.803 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-09-16 00:04:02.498887747 +0000 2024-09-16T00:09:00.803 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T00:09:00.870 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-09-16T00:09:00.870 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-09-16T00:09:00.870 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000275779 s, 1.9 MB/s 2024-09-16T00:09:00.872 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T00:09:00.930 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2024-09-16T00:09:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T00:09:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T00:09:00.989 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-16T00:09:00.989 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:09:00.989 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T00:09:00.989 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-09-16 00:08:16.276398355 +0000 2024-09-16T00:09:00.989 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-09-16 00:04:02.849881824 +0000 2024-09-16T00:09:00.989 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-09-16 00:04:02.849881824 +0000 2024-09-16T00:09:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-09-16 00:04:02.849881824 +0000 2024-09-16T00:09:00.990 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T00:09:01.055 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:00 smithi033 ceph-mon[31243]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:01.055 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:01.055 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:01.055 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:00 smithi033 ceph-mon[31243]: Reconfiguring mon.b (monmap changed)... 2024-09-16T00:09:01.055 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:09:01.055 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T00:09:01.056 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:01.056 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:00 smithi033 ceph-mon[31243]: Reconfiguring daemon mon.b on smithi060 2024-09-16T00:09:01.056 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:00 smithi033 ceph-mon[31243]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi033=x;smithi060=y;smithi083=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:09:01.056 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:00 smithi033 ceph-mon[31243]: Saving service mgr spec with placement smithi033=x;smithi060=y;smithi083=z;count:3 2024-09-16T00:09:01.056 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:01.057 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-09-16T00:09:01.057 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-09-16T00:09:01.058 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000231859 s, 2.2 MB/s 2024-09-16T00:09:01.059 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T00:09:01.118 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2024-09-16T00:09:01.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:00 smithi083 ceph-mon[35242]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:01.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:01.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:01.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:00 smithi083 ceph-mon[35242]: Reconfiguring mon.b (monmap changed)... 2024-09-16T00:09:01.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:09:01.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T00:09:01.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:01.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:00 smithi083 ceph-mon[35242]: Reconfiguring daemon mon.b on smithi060 2024-09-16T00:09:01.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:00 smithi083 ceph-mon[35242]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi033=x;smithi060=y;smithi083=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:09:01.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:00 smithi083 ceph-mon[35242]: Saving service mgr spec with placement smithi033=x;smithi060=y;smithi083=z;count:3 2024-09-16T00:09:01.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:01.173 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:00 smithi060 ceph-mon[35185]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:01.173 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:01.173 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:01.173 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:00 smithi060 ceph-mon[35185]: Reconfiguring mon.b (monmap changed)... 2024-09-16T00:09:01.173 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:09:01.173 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T00:09:01.173 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:01.173 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:00 smithi060 ceph-mon[35185]: Reconfiguring daemon mon.b on smithi060 2024-09-16T00:09:01.173 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:00 smithi060 ceph-mon[35185]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi033=x;smithi060=y;smithi083=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:09:01.173 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:00 smithi060 ceph-mon[35185]: Saving service mgr spec with placement smithi033=x;smithi060=y;smithi083=z;count:3 2024-09-16T00:09:01.174 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:01.176 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T00:09:01.176 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T00:09:01.176 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T00:09:01.176 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:09:01.176 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T00:09:01.176 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-09-16 00:08:16.322397597 +0000 2024-09-16T00:09:01.177 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-09-16 00:04:03.165876491 +0000 2024-09-16T00:09:01.177 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-09-16 00:04:03.165876491 +0000 2024-09-16T00:09:01.177 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-09-16 00:04:03.165876491 +0000 2024-09-16T00:09:01.177 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T00:09:01.245 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-09-16T00:09:01.245 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-09-16T00:09:01.246 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000244045 s, 2.1 MB/s 2024-09-16T00:09:01.247 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T00:09:01.306 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-16T00:09:01.306 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T00:09:01.325 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T00:09:01.325 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-09-16T00:09:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T00:09:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T00:09:01.382 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-09-16T00:09:01.382 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:09:01.382 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T00:09:01.382 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-16 00:08:50.501577013 +0000 2024-09-16T00:09:01.382 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-16 00:04:17.916822451 +0000 2024-09-16T00:09:01.383 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-16 00:04:17.916822451 +0000 2024-09-16T00:09:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-09-16 00:04:17.916822451 +0000 2024-09-16T00:09:01.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T00:09:01.449 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-16T00:09:01.449 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-16T00:09:01.450 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.00010021 s, 5.1 MB/s 2024-09-16T00:09:01.451 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T00:09:01.509 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-09-16T00:09:01.566 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T00:09:01.566 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T00:09:01.566 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T00:09:01.566 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:09:01.567 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T00:09:01.567 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-16 00:08:50.502577003 +0000 2024-09-16T00:09:01.567 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-16 00:04:18.241819366 +0000 2024-09-16T00:09:01.567 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-16 00:04:18.241819366 +0000 2024-09-16T00:09:01.567 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-09-16 00:04:18.241819366 +0000 2024-09-16T00:09:01.567 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T00:09:01.631 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-16T00:09:01.632 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-16T00:09:01.632 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000114409 s, 4.5 MB/s 2024-09-16T00:09:01.633 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T00:09:01.691 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-09-16T00:09:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T00:09:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T00:09:01.747 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-09-16T00:09:01.747 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:09:01.747 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T00:09:01.747 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-16 00:08:50.502577003 +0000 2024-09-16T00:09:01.747 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-16 00:04:18.553816396 +0000 2024-09-16T00:09:01.747 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-16 00:04:18.553816396 +0000 2024-09-16T00:09:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-09-16 00:04:18.553816396 +0000 2024-09-16T00:09:01.748 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T00:09:01.811 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-16T00:09:01.812 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-16T00:09:01.812 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000113975 s, 4.5 MB/s 2024-09-16T00:09:01.813 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T00:09:01.870 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-09-16T00:09:01.926 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T00:09:01.926 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T00:09:01.926 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-09-16T00:09:01.926 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:09:01.927 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T00:09:01.927 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-16 00:08:50.503576994 +0000 2024-09-16T00:09:01.927 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-16 00:04:18.884813244 +0000 2024-09-16T00:09:01.927 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-16 00:04:18.884813244 +0000 2024-09-16T00:09:01.927 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-09-16 00:04:18.884813244 +0000 2024-09-16T00:09:01.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T00:09:01.990 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-16T00:09:01.991 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-16T00:09:01.991 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000110299 s, 4.6 MB/s 2024-09-16T00:09:01.992 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T00:09:02.048 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-16T00:09:02.049 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T00:09:02.065 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T00:09:02.065 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-09-16T00:09:02.121 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T00:09:02.121 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T00:09:02.122 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-16T00:09:02.122 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:09:02.122 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T00:09:02.122 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-16 00:08:41.270042365 +0000 2024-09-16T00:09:02.122 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-16 00:04:09.243462522 +0000 2024-09-16T00:09:02.122 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-16 00:04:09.243462522 +0000 2024-09-16T00:09:02.122 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-09-16 00:04:09.243462522 +0000 2024-09-16T00:09:02.122 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T00:09:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-16T00:09:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-16T00:09:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000294274 s, 1.7 MB/s 2024-09-16T00:09:02.188 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T00:09:02.245 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-09-16T00:09:02.300 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T00:09:02.300 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T00:09:02.300 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-09-16T00:09:02.300 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:09:02.300 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T00:09:02.300 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-16 00:08:41.271042341 +0000 2024-09-16T00:09:02.300 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-16 00:04:09.569454396 +0000 2024-09-16T00:09:02.300 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-16 00:04:09.569454396 +0000 2024-09-16T00:09:02.300 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-09-16 00:04:09.569454396 +0000 2024-09-16T00:09:02.301 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T00:09:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-16T00:09:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-16T00:09:02.364 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000169021 s, 3.0 MB/s 2024-09-16T00:09:02.365 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T00:09:02.420 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-09-16T00:09:02.476 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T00:09:02.476 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T00:09:02.476 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T00:09:02.476 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:09:02.476 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T00:09:02.476 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-16 00:08:41.271042341 +0000 2024-09-16T00:09:02.476 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-16 00:04:09.902446096 +0000 2024-09-16T00:09:02.476 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-16 00:04:09.902446096 +0000 2024-09-16T00:09:02.477 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-09-16 00:04:09.902446096 +0000 2024-09-16T00:09:02.477 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T00:09:02.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:02 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:02.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:02 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:02.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:02 smithi060 ceph-mon[35185]: Reconfiguring mon.c (monmap changed)... 2024-09-16T00:09:02.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:02 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:09:02.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:02 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T00:09:02.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:02 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:02.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:02 smithi060 ceph-mon[35185]: Reconfiguring daemon mon.c on smithi083 2024-09-16T00:09:02.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:02 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:02.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:02 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:02.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:02 smithi083 ceph-mon[35242]: Reconfiguring mon.c (monmap changed)... 2024-09-16T00:09:02.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:02 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:09:02.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:02 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T00:09:02.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:02 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:02.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:02 smithi083 ceph-mon[35242]: Reconfiguring daemon mon.c on smithi083 2024-09-16T00:09:02.540 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-16T00:09:02.540 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-16T00:09:02.540 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000194904 s, 2.6 MB/s 2024-09-16T00:09:02.541 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T00:09:02.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:02 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:02.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:02 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:02.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:02 smithi033 ceph-mon[31243]: Reconfiguring mon.c (monmap changed)... 2024-09-16T00:09:02.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:02 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T00:09:02.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:02 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T00:09:02.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:02 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:02.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:02 smithi033 ceph-mon[31243]: Reconfiguring daemon mon.c on smithi083 2024-09-16T00:09:02.597 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-09-16T00:09:02.653 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T00:09:02.653 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T00:09:02.653 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T00:09:02.653 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:09:02.653 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T00:09:02.653 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-16 00:08:41.271042341 +0000 2024-09-16T00:09:02.653 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-16 00:04:10.221438144 +0000 2024-09-16T00:09:02.653 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-16 00:04:10.221438144 +0000 2024-09-16T00:09:02.653 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-09-16 00:04:10.221438144 +0000 2024-09-16T00:09:02.654 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T00:09:02.717 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-16T00:09:02.717 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-16T00:09:02.717 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000224885 s, 2.3 MB/s 2024-09-16T00:09:02.718 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T00:09:02.774 INFO:tasks.cephadm:Deploying osd.0 on smithi033 with /dev/vg_nvme/lv_4... 2024-09-16T00:09:02.775 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T00:09:03.055 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:09:03.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:03 smithi083 ceph-mon[35242]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:03.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:03.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:03.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:03.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:03.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:03.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:03.349 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T00:09:03.349 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T00:09:03.349 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T00:09:03.349 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:03.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:03 smithi033 ceph-mon[31243]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:03.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:03.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:03.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:03.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:03.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:03.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:03.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T00:09:03.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T00:09:03.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T00:09:03.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:03 smithi060 ceph-mon[35185]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T00:09:03.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T00:09:03.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T00:09:03.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:04.412 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:04 smithi033 ceph-mon[31243]: Deploying daemon mgr.z on smithi083 2024-09-16T00:09:04.531 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:04 smithi083 ceph-mon[35242]: Deploying daemon mgr.z on smithi083 2024-09-16T00:09:04.532 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:04 smithi083 systemd[1]: Starting Ceph mgr.z for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:09:04.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:04 smithi060 ceph-mon[35185]: Deploying daemon mgr.z on smithi083 2024-09-16T00:09:04.783 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:04 smithi083 podman[37029]: 2024-09-16 00:09:04.433484241 +0000 UTC m=+0.015363537 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:09:04.783 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:04 smithi083 podman[37029]: 2024-09-16 00:09:04.543985953 +0000 UTC m=+0.125865282 container create 44d46e2f0d9dce3aecf338096adacf718e006a9a806c4f48bae046a12d9e1e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, ceph=True) 2024-09-16T00:09:04.783 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:04 smithi083 podman[37029]: 2024-09-16 00:09:04.75446407 +0000 UTC m=+0.336343354 container init 44d46e2f0d9dce3aecf338096adacf718e006a9a806c4f48bae046a12d9e1e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-16T00:09:04.783 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:04 smithi083 podman[37029]: 2024-09-16 00:09:04.757875279 +0000 UTC m=+0.339754579 container start 44d46e2f0d9dce3aecf338096adacf718e006a9a806c4f48bae046a12d9e1e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:09:05.098 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:04 smithi083 ceph-mgr[37048]: -- 172.21.15.83:0/3626413370 <== mon.1 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5584a860f0e0 con 0x5584a85ed400 2024-09-16T00:09:05.098 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:04 smithi083 bash[37029]: 44d46e2f0d9dce3aecf338096adacf718e006a9a806c4f48bae046a12d9e1e29 2024-09-16T00:09:05.098 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:04 smithi083 systemd[1]: Started Ceph mgr.z for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:09:05.098 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:04 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:04.883+0000 7f7c5d3b9040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T00:09:05.098 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:04 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:04.950+0000 7f7c5d3b9040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T00:09:05.098 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:05.097+0000 7f7c5d3b9040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T00:09:05.419 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:05.163+0000 7f7c5d3b9040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T00:09:05.419 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:05.311+0000 7f7c5d3b9040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T00:09:05.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-mon[35242]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:05.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:05.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:05.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:05.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T00:09:05.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T00:09:05.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T00:09:05.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:09:05.459 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_4 2024-09-16T00:09:05.484 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:05 smithi033 ceph-mon[31243]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:05.484 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:05.484 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:05.484 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:05.484 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T00:09:05.484 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T00:09:05.484 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T00:09:05.484 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:05.568 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:05 smithi060 ceph-mon[35185]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:05.569 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:05.569 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:05.569 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:05.569 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T00:09:05.569 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T00:09:05.570 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T00:09:05.570 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:05.711 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:09:05.765 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:05.450+0000 7f7c5d3b9040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T00:09:05.765 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:05.764+0000 7f7c5d3b9040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T00:09:06.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.125+0000 7fc944393640 1 -- 172.21.15.33:0/777898459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc93c102e00 msgr2=0x7fc93c103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:06.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.125+0000 7fc944393640 1 --2- 172.21.15.33:0/777898459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc93c102e00 0x7fc93c103280 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009900 tx=0x7fc92c02a410 comp rx=0 tx=0).stop 2024-09-16T00:09:06.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.125+0000 7fc944393640 1 -- 172.21.15.33:0/777898459 shutdown_connections 2024-09-16T00:09:06.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.125+0000 7fc944393640 1 --2- 172.21.15.33:0/777898459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc93c1037c0 0x7fc93c10a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:06.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.125+0000 7fc944393640 1 --2- 172.21.15.33:0/777898459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc93c102e00 0x7fc93c103280 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:06.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.125+0000 7fc944393640 1 --2- 172.21.15.33:0/777898459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc93c101c00 0x7fc93c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:06.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.125+0000 7fc944393640 1 -- 172.21.15.33:0/777898459 >> 172.21.15.33:0/777898459 conn(0x7fc93c0fd390 msgr2=0x7fc93c0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:09:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.125+0000 7fc944393640 1 -- 172.21.15.33:0/777898459 shutdown_connections 2024-09-16T00:09:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.125+0000 7fc944393640 1 -- 172.21.15.33:0/777898459 wait complete. 2024-09-16T00:09:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc944393640 1 Processor -- start 2024-09-16T00:09:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc944393640 1 -- start start 2024-09-16T00:09:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc944393640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc93c101c00 0x7fc93c075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:09:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc944393640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc93c102e00 0x7fc93c0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:09:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc944393640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc93c1037c0 0x7fc93c074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:09:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc944393640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc93c10d8b0 con 0x7fc93c102e00 2024-09-16T00:09:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc944393640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc93c10d730 con 0x7fc93c101c00 2024-09-16T00:09:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc944393640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc93c10da30 con 0x7fc93c1037c0 2024-09-16T00:09:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc94192e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc93c102e00 0x7fc93c0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:09:06.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc94192e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc93c102e00 0x7fc93c0745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54540/0 (socket says 172.21.15.33:54540) 2024-09-16T00:09:06.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc94192e640 1 -- 172.21.15.33:0/411590206 learned_addr learned my addr 172.21.15.33:0/411590206 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:09:06.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc942930640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc93c1037c0 0x7fc93c074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:09:06.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.126+0000 7fc94212f640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc93c101c00 0x7fc93c075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:09:06.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.127+0000 7fc94192e640 1 -- 172.21.15.33:0/411590206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc93c1037c0 msgr2=0x7fc93c074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:06.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.127+0000 7fc94192e640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc93c1037c0 0x7fc93c074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:06.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.127+0000 7fc94192e640 1 -- 172.21.15.33:0/411590206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc93c101c00 msgr2=0x7fc93c075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:06.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.127+0000 7fc94192e640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc93c101c00 0x7fc93c075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:06.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.127+0000 7fc94192e640 1 -- 172.21.15.33:0/411590206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc93c0753e0 con 0x7fc93c102e00 2024-09-16T00:09:06.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.127+0000 7fc942930640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc93c1037c0 0x7fc93c074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:09:06.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.127+0000 7fc94192e640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc93c102e00 0x7fc93c0745e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc92c031440 tx=0x7fc92c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:09:06.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.127+0000 7fc9337fe640 1 -- 172.21.15.33:0/411590206 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c03f030 con 0x7fc93c102e00 2024-09-16T00:09:06.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.127+0000 7fc9337fe640 1 -- 172.21.15.33:0/411590206 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc92c003ce0 con 0x7fc93c102e00 2024-09-16T00:09:06.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.127+0000 7fc944393640 1 -- 172.21.15.33:0/411590206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc93c1a2030 con 0x7fc93c102e00 2024-09-16T00:09:06.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.127+0000 7fc9337fe640 1 -- 172.21.15.33:0/411590206 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c035070 con 0x7fc93c102e00 2024-09-16T00:09:06.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.127+0000 7fc944393640 1 -- 172.21.15.33:0/411590206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc93c1a2530 con 0x7fc93c102e00 2024-09-16T00:09:06.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.128+0000 7fc9337fe640 1 -- 172.21.15.33:0/411590206 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fc92c042030 con 0x7fc93c102e00 2024-09-16T00:09:06.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.128+0000 7fc944393640 1 -- 172.21.15.33:0/411590206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc93c108fb0 con 0x7fc93c102e00 2024-09-16T00:09:06.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.128+0000 7fc9337fe640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9280394d0 0x7fc92803b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:09:06.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.128+0000 7fc9337fe640 1 -- 172.21.15.33:0/411590206 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc92c06c880 con 0x7fc93c102e00 2024-09-16T00:09:06.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.128+0000 7fc94212f640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9280394d0 0x7fc92803b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:09:06.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.129+0000 7fc94212f640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9280394d0 0x7fc92803b990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc93c102c60 tx=0x7fc938005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:09:06.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.133+0000 7fc9337fe640 1 -- 172.21.15.33:0/411590206 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc92c004750 con 0x7fc93c102e00 2024-09-16T00:09:06.169 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:05.825+0000 7f7c5d3b9040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T00:09:06.169 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:06.008+0000 7f7c5d3b9040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T00:09:06.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:06.320+0000 7fc944393640 1 -- 172.21.15.33:0/411590206 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc93c062630 con 0x7fc9280394d0 2024-09-16T00:09:06.453 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:06 smithi060 ceph-mon[35185]: Deploying daemon mgr.y on smithi060 2024-09-16T00:09:06.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:06 smithi033 ceph-mon[31243]: Deploying daemon mgr.y on smithi060 2024-09-16T00:09:06.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:06 smithi083 ceph-mon[35242]: Deploying daemon mgr.y on smithi060 2024-09-16T00:09:06.731 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:06 smithi060 systemd[1]: Starting Ceph mgr.y for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:09:07.108 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:06 smithi060 podman[36535]: 2024-09-16 00:09:06.838713066 +0000 UTC m=+0.108466475 container create 9a7609fbd20b31a3041ee24a5e1821ea2cf6b7c7752a683139cc1ba25f7f9b99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-8921c8e, GIT_CLEAN=True) 2024-09-16T00:09:07.108 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:06 smithi060 podman[36535]: 2024-09-16 00:09:06.746039423 +0000 UTC m=+0.015792792 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:09:07.108 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:07 smithi060 podman[36535]: 2024-09-16 00:09:07.032582165 +0000 UTC m=+0.302335532 container init 9a7609fbd20b31a3041ee24a5e1821ea2cf6b7c7752a683139cc1ba25f7f9b99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-16T00:09:07.108 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:07 smithi060 podman[36535]: 2024-09-16 00:09:07.035356394 +0000 UTC m=+0.305109770 container start 9a7609fbd20b31a3041ee24a5e1821ea2cf6b7c7752a683139cc1ba25f7f9b99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:09:07.108 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-mgr[36554]: -- 172.21.15.60:0/3684204397 <== mon.2 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558199aa30e0 con 0x558199a80800 2024-09-16T00:09:07.108 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:07 smithi060 bash[36535]: 9a7609fbd20b31a3041ee24a5e1821ea2cf6b7c7752a683139cc1ba25f7f9b99 2024-09-16T00:09:07.169 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:06.886+0000 7f7c5d3b9040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T00:09:07.169 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:06.953+0000 7f7c5d3b9040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T00:09:07.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:07 smithi033 ceph-mon[31243]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:07.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:07 smithi033 ceph-mon[31243]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:09:07.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:09:07.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:09:07.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:07.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:07.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:07.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:07.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:07.386 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:07 smithi060 systemd[1]: Started Ceph mgr.y for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:09:07.387 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:07.161+0000 7f28a11a8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T00:09:07.387 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:07.224+0000 7f28a11a8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T00:09:07.387 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:07.385+0000 7f28a11a8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T00:09:07.387 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-mon[35185]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:07.388 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-mon[35185]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:09:07.388 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:09:07.388 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:09:07.388 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:07.388 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:07.388 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:07.389 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:07.389 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:07.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:07 smithi083 ceph-mon[35242]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:07.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:07 smithi083 ceph-mon[35242]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:09:07.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:07 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:09:07.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:07 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:09:07.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:07 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:07.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:07 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:07.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:07 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:07.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:07 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:07.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:07 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:07.731 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:07.456+0000 7f28a11a8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T00:09:07.731 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:07.632+0000 7f28a11a8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T00:09:07.979 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:07.711+0000 7f7c5d3b9040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T00:09:07.979 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:07.827+0000 7f7c5d3b9040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T00:09:07.979 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:07.978+0000 7f7c5d3b9040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T00:09:08.053 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:07 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:07.784+0000 7f28a11a8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T00:09:08.255 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:08.039+0000 7f7c5d3b9040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T00:09:08.256 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:08.212+0000 7f7c5d3b9040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T00:09:08.402 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:08.402 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:08.402 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:08.402 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:08.402 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:08 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:08.126+0000 7f28a11a8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T00:09:08.402 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:08 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:08.190+0000 7f28a11a8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T00:09:08.402 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:08 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:08.401+0000 7f28a11a8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T00:09:08.569 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:08.569 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:08.569 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:08.570 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:08.570 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:08.283+0000 7f7c5d3b9040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T00:09:08.570 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:08.343+0000 7f7c5d3b9040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T00:09:08.570 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:08.504+0000 7f7c5d3b9040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T00:09:08.570 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:08.568+0000 7f7c5d3b9040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T00:09:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:08.832 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:08.634+0000 7f7c5d3b9040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T00:09:08.833 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:08.715+0000 7f7c5d3b9040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T00:09:08.833 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:08.774+0000 7f7c5d3b9040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T00:09:08.833 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:08.831+0000 7f7c5d3b9040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T00:09:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: Reconfiguring mgr.x (unknown last config time)... 2024-09-16T00:09:09.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T00:09:09.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T00:09:09.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:09.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: Reconfiguring daemon mgr.x on smithi033 2024-09-16T00:09:09.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3141197563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12847d0f-d226-40db-bd81-100ef8fddf1d"}]: dispatch 2024-09-16T00:09:09.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3141197563' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12847d0f-d226-40db-bd81-100ef8fddf1d"}]': finished 2024-09-16T00:09:09.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T00:09:09.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:09.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:09 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2687620037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:09:09.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:09.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:09.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:09.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:09.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:09.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:09.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: Reconfiguring mgr.x (unknown last config time)... 2024-09-16T00:09:09.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T00:09:09.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T00:09:09.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:09.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: Reconfiguring daemon mgr.x on smithi033 2024-09-16T00:09:09.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3141197563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12847d0f-d226-40db-bd81-100ef8fddf1d"}]: dispatch 2024-09-16T00:09:09.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3141197563' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12847d0f-d226-40db-bd81-100ef8fddf1d"}]': finished 2024-09-16T00:09:09.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T00:09:09.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:09.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2687620037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:09:09.730 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:09.328+0000 7f28a11a8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T00:09:09.731 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:09.398+0000 7f28a11a8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T00:09:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: Reconfiguring mgr.x (unknown last config time)... 2024-09-16T00:09:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T00:09:09.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T00:09:09.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:09.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: Reconfiguring daemon mgr.x on smithi033 2024-09-16T00:09:09.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3141197563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12847d0f-d226-40db-bd81-100ef8fddf1d"}]: dispatch 2024-09-16T00:09:09.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3141197563' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12847d0f-d226-40db-bd81-100ef8fddf1d"}]': finished 2024-09-16T00:09:09.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T00:09:09.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:09.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:09 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2687620037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:09:09.919 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:09:09 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z[37044]: 2024-09-16T00:09:09.612+0000 7f7c5d3b9040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T00:09:10.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:10.334+0000 7fc9337fe640 1 -- 172.21.15.33:0/411590206 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 14) ==== 92408+0+0 (secure 0 0 0) 0x7fc92c049060 con 0x7fc93c102e00 2024-09-16T00:09:10.433 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:10.158+0000 7f28a11a8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T00:09:10.433 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:10.279+0000 7f28a11a8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T00:09:10.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-mon[35185]: Standby manager daemon z started 2024-09-16T00:09:10.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-mon[35185]: from='mgr.? 172.21.15.83:0/2405362077' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-16T00:09:10.434 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-mon[35185]: from='mgr.? 172.21.15.83:0/2405362077' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T00:09:10.434 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-mon[35185]: from='mgr.? 172.21.15.83:0/2405362077' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-16T00:09:10.434 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-mon[35185]: from='mgr.? 172.21.15.83:0/2405362077' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T00:09:10.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:10 smithi033 ceph-mon[31243]: Standby manager daemon z started 2024-09-16T00:09:10.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:10 smithi033 ceph-mon[31243]: from='mgr.? 172.21.15.83:0/2405362077' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-16T00:09:10.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:10 smithi033 ceph-mon[31243]: from='mgr.? 172.21.15.83:0/2405362077' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T00:09:10.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:10 smithi033 ceph-mon[31243]: from='mgr.? 172.21.15.83:0/2405362077' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-16T00:09:10.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:10 smithi033 ceph-mon[31243]: from='mgr.? 172.21.15.83:0/2405362077' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T00:09:10.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:10 smithi083 ceph-mon[35242]: Standby manager daemon z started 2024-09-16T00:09:10.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:10 smithi083 ceph-mon[35242]: from='mgr.? 172.21.15.83:0/2405362077' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-16T00:09:10.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:10 smithi083 ceph-mon[35242]: from='mgr.? 172.21.15.83:0/2405362077' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T00:09:10.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:10 smithi083 ceph-mon[35242]: from='mgr.? 172.21.15.83:0/2405362077' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-16T00:09:10.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:10 smithi083 ceph-mon[35242]: from='mgr.? 172.21.15.83:0/2405362077' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T00:09:10.724 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:10.432+0000 7f28a11a8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T00:09:10.724 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:10.496+0000 7f28a11a8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T00:09:10.724 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:10.652+0000 7f28a11a8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T00:09:10.724 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:10.723+0000 7f28a11a8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T00:09:10.980 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:10.785+0000 7f28a11a8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T00:09:10.981 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:10.951+0000 7f28a11a8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T00:09:11.300 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:11 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:11.019+0000 7f28a11a8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T00:09:11.300 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:11 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:11.089+0000 7f28a11a8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T00:09:11.300 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:11 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:11.175+0000 7f28a11a8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T00:09:11.300 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:11 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:11.237+0000 7f28a11a8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T00:09:11.300 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:11 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:11.299+0000 7f28a11a8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T00:09:11.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:11 smithi083 ceph-mon[35242]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:11.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:11 smithi083 ceph-mon[35242]: mgrmap e14: x(active, since 62s), standbys: z 2024-09-16T00:09:11.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:11 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-16T00:09:11.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:11 smithi060 ceph-mon[35185]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:11.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:11 smithi060 ceph-mon[35185]: mgrmap e14: x(active, since 62s), standbys: z 2024-09-16T00:09:11.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:11 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-16T00:09:11.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:11 smithi033 ceph-mon[31243]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:11.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:11 smithi033 ceph-mon[31243]: mgrmap e14: x(active, since 62s), standbys: z 2024-09-16T00:09:11.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:11 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-16T00:09:12.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:12.343+0000 7fc9337fe640 1 -- 172.21.15.33:0/411590206 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7fc92c004fc0 con 0x7fc93c102e00 2024-09-16T00:09:12.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:12 smithi060 ceph-mon[35185]: Standby manager daemon y started 2024-09-16T00:09:12.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:12 smithi060 ceph-mon[35185]: from='mgr.? 172.21.15.60:0/869992486' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-16T00:09:12.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:12 smithi060 ceph-mon[35185]: from='mgr.? 172.21.15.60:0/869992486' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T00:09:12.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:12 smithi060 ceph-mon[35185]: from='mgr.? 172.21.15.60:0/869992486' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-16T00:09:12.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:12 smithi060 ceph-mon[35185]: from='mgr.? 172.21.15.60:0/869992486' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T00:09:12.481 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:09:12 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y[36550]: 2024-09-16T00:09:12.130+0000 7f28a11a8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T00:09:12.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:12 smithi083 ceph-mon[35242]: Standby manager daemon y started 2024-09-16T00:09:12.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:12 smithi083 ceph-mon[35242]: from='mgr.? 172.21.15.60:0/869992486' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-16T00:09:12.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:12 smithi083 ceph-mon[35242]: from='mgr.? 172.21.15.60:0/869992486' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T00:09:12.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:12 smithi083 ceph-mon[35242]: from='mgr.? 172.21.15.60:0/869992486' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-16T00:09:12.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:12 smithi083 ceph-mon[35242]: from='mgr.? 172.21.15.60:0/869992486' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T00:09:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:12 smithi033 ceph-mon[31243]: Standby manager daemon y started 2024-09-16T00:09:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:12 smithi033 ceph-mon[31243]: from='mgr.? 172.21.15.60:0/869992486' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-16T00:09:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:12 smithi033 ceph-mon[31243]: from='mgr.? 172.21.15.60:0/869992486' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T00:09:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:12 smithi033 ceph-mon[31243]: from='mgr.? 172.21.15.60:0/869992486' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-16T00:09:12.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:12 smithi033 ceph-mon[31243]: from='mgr.? 172.21.15.60:0/869992486' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T00:09:13.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:13 smithi083 ceph-mon[35242]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:13.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:13 smithi083 ceph-mon[35242]: mgrmap e15: x(active, since 64s), standbys: z, y 2024-09-16T00:09:13.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-16T00:09:13.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:13 smithi060 ceph-mon[35185]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:13.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:13 smithi060 ceph-mon[35185]: mgrmap e15: x(active, since 64s), standbys: z, y 2024-09-16T00:09:13.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-16T00:09:13.824 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:13 smithi033 ceph-mon[31243]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:13.824 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:13 smithi033 ceph-mon[31243]: mgrmap e15: x(active, since 64s), standbys: z, y 2024-09-16T00:09:13.824 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-16T00:09:15.444 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:15 smithi033 ceph-mon[31243]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:15 smithi083 ceph-mon[35242]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:15.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:15 smithi060 ceph-mon[35185]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:16.610 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:16.610 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:16.610 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:16.610 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:16 smithi033 ceph-mon[31243]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:16.610 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:16.610 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:16.610 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:16.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:16.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:16.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:16.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:16 smithi083 ceph-mon[35242]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:16.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:16.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:16.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:16.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:16.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:16.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:16.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:16 smithi060 ceph-mon[35185]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:16.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:16.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:16.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:18.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:18 smithi033 ceph-mon[31243]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:18.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-16T00:09:18.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:18.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:18 smithi033 ceph-mon[31243]: Deploying daemon osd.0 on smithi033 2024-09-16T00:09:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:18 smithi083 ceph-mon[35242]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-16T00:09:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:18 smithi083 ceph-mon[35242]: Deploying daemon osd.0 on smithi033 2024-09-16T00:09:18.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:18 smithi060 ceph-mon[35185]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:18.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-16T00:09:18.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:18.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:18 smithi060 ceph-mon[35185]: Deploying daemon osd.0 on smithi033 2024-09-16T00:09:20.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:20 smithi083 ceph-mon[35242]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:20.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:20 smithi060 ceph-mon[35185]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:21.070 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:20 smithi033 ceph-mon[31243]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:22.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:22 smithi083 ceph-mon[35242]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:22 smithi060 ceph-mon[35185]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:22.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:22 smithi033 ceph-mon[31243]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:24.886 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:24 smithi033 ceph-mon[31243]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:24.886 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:24 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:24.886 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:24 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:24.886 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:24 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:24.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:24 smithi083 ceph-mon[35242]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:24.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:24 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:24.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:24 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:24.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:24 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:24.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:24 smithi060 ceph-mon[35185]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:24.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:24 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:24.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:24 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:24.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:24 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:26.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:26 smithi033 ceph-mon[31243]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:26.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:26 smithi083 ceph-mon[35242]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:26.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:26 smithi060 ceph-mon[35185]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:27.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.860+0000 7fc9337fe640 1 -- 172.21.15.33:0/411590206 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc93c062630 con 0x7fc9280394d0 2024-09-16T00:09:27.862 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 0 on host 'smithi033' 2024-09-16T00:09:27.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:27.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:27.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:27.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:27.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:27.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.863+0000 7fc944393640 1 -- 172.21.15.33:0/411590206 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9280394d0 msgr2=0x7fc92803b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:27.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.863+0000 7fc944393640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9280394d0 0x7fc92803b990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc93c102c60 tx=0x7fc938005f90 comp rx=0 tx=0).stop 2024-09-16T00:09:27.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.863+0000 7fc944393640 1 -- 172.21.15.33:0/411590206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc93c102e00 msgr2=0x7fc93c0745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:27.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.863+0000 7fc944393640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc93c102e00 0x7fc93c0745e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc92c031440 tx=0x7fc92c032ce0 comp rx=0 tx=0).stop 2024-09-16T00:09:27.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.863+0000 7fc944393640 1 -- 172.21.15.33:0/411590206 shutdown_connections 2024-09-16T00:09:27.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.863+0000 7fc944393640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9280394d0 0x7fc92803b990 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:27.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.863+0000 7fc944393640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc93c1037c0 0x7fc93c074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:27.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.863+0000 7fc944393640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc93c102e00 0x7fc93c0745e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:27.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.863+0000 7fc944393640 1 --2- 172.21.15.33:0/411590206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc93c101c00 0x7fc93c075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:27.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.863+0000 7fc944393640 1 -- 172.21.15.33:0/411590206 >> 172.21.15.33:0/411590206 conn(0x7fc93c0fd390 msgr2=0x7fc93c106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:09:27.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.863+0000 7fc944393640 1 -- 172.21.15.33:0/411590206 shutdown_connections 2024-09-16T00:09:27.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:09:27.863+0000 7fc944393640 1 -- 172.21.15.33:0/411590206 wait complete. 2024-09-16T00:09:28.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:28.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:28.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:28.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:28.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:28.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:28.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:28.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:28.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:28.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:28.428 DEBUG:teuthology.orchestra.run.smithi033:osd.0> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.0.service 2024-09-16T00:09:28.470 INFO:tasks.cephadm:Deploying osd.1 on smithi060 with /dev/vg_nvme/lv_4... 2024-09-16T00:09:28.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T00:09:28.717 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T00:09:28.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:28 smithi033 ceph-mon[31243]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:28.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:28.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:28.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:29.147 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:28 smithi060 ceph-mon[35185]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:29.148 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:29.148 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:29.148 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:29.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:28 smithi083 ceph-mon[35242]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:29.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:30.331 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:09:30 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0[42929]: 2024-09-16T00:09:30.095+0000 7facefaa5640 -1 osd.0 0 log_to_monitors true 2024-09-16T00:09:30.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:30 smithi033 ceph-mon[31243]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:30.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:30 smithi033 ceph-mon[31243]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-16T00:09:31.120 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:30 smithi060 ceph-mon[35185]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:31.121 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:30 smithi060 ceph-mon[35185]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-16T00:09:31.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:30 smithi083 ceph-mon[35242]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:31.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:30 smithi083 ceph-mon[35242]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-16T00:09:31.170 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:09:31.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi060:vg_nvme/lv_4 2024-09-16T00:09:31.458 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T00:09:31.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.954+0000 7ff307de0640 1 -- 172.21.15.60:0/713066188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff300102660 msgr2=0x7ff300104a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:31.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.954+0000 7ff307de0640 1 --2- 172.21.15.60:0/713066188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff300102660 0x7ff300104a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009900 tx=0x7ff2fc02a410 comp rx=0 tx=0).stop 2024-09-16T00:09:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.954+0000 7ff307de0640 1 -- 172.21.15.60:0/713066188 shutdown_connections 2024-09-16T00:09:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.954+0000 7ff307de0640 1 --2- 172.21.15.60:0/713066188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3001078c0 0x7ff300109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.954+0000 7ff307de0640 1 --2- 172.21.15.60:0/713066188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff300104f90 0x7ff300107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.954+0000 7ff307de0640 1 --2- 172.21.15.60:0/713066188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff300102660 0x7ff300104a50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.954+0000 7ff307de0640 1 -- 172.21.15.60:0/713066188 >> 172.21.15.60:0/713066188 conn(0x7ff3000fc280 msgr2=0x7ff3000fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:09:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.954+0000 7ff307de0640 1 -- 172.21.15.60:0/713066188 shutdown_connections 2024-09-16T00:09:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.954+0000 7ff307de0640 1 -- 172.21.15.60:0/713066188 wait complete. 2024-09-16T00:09:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.955+0000 7ff307de0640 1 Processor -- start 2024-09-16T00:09:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.955+0000 7ff307de0640 1 -- start start 2024-09-16T00:09:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.955+0000 7ff307de0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff300102660 0x7ff300197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:09:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.955+0000 7ff307de0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff300104f90 0x7ff300198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:09:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.955+0000 7ff307de0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3001078c0 0x7ff30019f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:09:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.955+0000 7ff307de0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff30010d890 con 0x7ff300104f90 2024-09-16T00:09:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.955+0000 7ff307de0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff30010d710 con 0x7ff300102660 2024-09-16T00:09:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.955+0000 7ff307de0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff30010da10 con 0x7ff3001078c0 2024-09-16T00:09:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.955+0000 7ff305b7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff300102660 0x7ff300197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:09:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.955+0000 7ff305b7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff300102660 0x7ff300197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37222/0 (socket says 172.21.15.60:37222) 2024-09-16T00:09:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.955+0000 7ff305b7c640 1 -- 172.21.15.60:0/3747604039 learned_addr learned my addr 172.21.15.60:0/3747604039 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-09-16T00:09:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.955+0000 7ff30537b640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff300104f90 0x7ff300198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:09:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff30637d640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3001078c0 0x7ff30019f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:09:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff305b7c640 1 -- 172.21.15.60:0/3747604039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3001078c0 msgr2=0x7ff30019f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff305b7c640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3001078c0 0x7ff30019f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff305b7c640 1 -- 172.21.15.60:0/3747604039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff300104f90 msgr2=0x7ff300198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff305b7c640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff300104f90 0x7ff300198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff305b7c640 1 -- 172.21.15.60:0/3747604039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff30019fb50 con 0x7ff300102660 2024-09-16T00:09:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff30637d640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3001078c0 0x7ff30019f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:09:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff305b7c640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff300102660 0x7ff300197f20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009230 tx=0x7ff2fc031b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:09:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff2f2ffd640 1 -- 172.21.15.60:0/3747604039 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc03f030 con 0x7ff300102660 2024-09-16T00:09:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff307de0640 1 -- 172.21.15.60:0/3747604039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff30019fde0 con 0x7ff300102660 2024-09-16T00:09:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff2f2ffd640 1 -- 172.21.15.60:0/3747604039 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2fc003ad0 con 0x7ff300102660 2024-09-16T00:09:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff2f2ffd640 1 -- 172.21.15.60:0/3747604039 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc035070 con 0x7ff300102660 2024-09-16T00:09:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.956+0000 7ff307de0640 1 -- 172.21.15.60:0/3747604039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3001a0230 con 0x7ff300102660 2024-09-16T00:09:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.957+0000 7ff307de0640 1 -- 172.21.15.60:0/3747604039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3000fdd70 con 0x7ff300102660 2024-09-16T00:09:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.958+0000 7ff2f2ffd640 1 -- 172.21.15.60:0/3747604039 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7ff2fc042030 con 0x7ff300102660 2024-09-16T00:09:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.959+0000 7ff2f2ffd640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff2ec0a3260 0x7ff2ec0a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:09:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.959+0000 7ff2f2ffd640 1 -- 172.21.15.60:0/3747604039 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1487+0+0 (secure 0 0 0) 0x7ff2fc0ee250 con 0x7ff300102660 2024-09-16T00:09:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.959+0000 7ff30537b640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff2ec0a3260 0x7ff2ec0a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:09:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.960+0000 7ff30537b640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff2ec0a3260 0x7ff2ec0a5720 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff300199440 tx=0x7ff2f4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:09:31.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:31.964+0000 7ff2f2ffd640 1 -- 172.21.15.60:0/3747604039 <== mon.2 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff2fc0bcb00 con 0x7ff300102660 2024-09-16T00:09:32.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T00:09:32.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T00:09:32.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-16T00:09:32.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:32.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: Detected new or changed devices on smithi033 2024-09-16T00:09:32.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:32.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:32.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:09:32.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi033 to 17092M 2024-09-16T00:09:32.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:32.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:32.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:32.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:32.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:32.148+0000 7ff307de0640 1 -- 172.21.15.60:0/3747604039 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff300062630 con 0x7ff2ec0a3260 2024-09-16T00:09:32.149 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T00:09:32.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T00:09:32.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-16T00:09:32.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:32.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: Detected new or changed devices on smithi033 2024-09-16T00:09:32.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:32.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:32.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:09:32.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi033 to 17092M 2024-09-16T00:09:32.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:32.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:32.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:32.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:32.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T00:09:32.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T00:09:32.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-16T00:09:32.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:32.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: Detected new or changed devices on smithi033 2024-09-16T00:09:32.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:09:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi033 to 17092M 2024-09-16T00:09:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:32.580 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:09:32 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0[42929]: 2024-09-16T00:09:32.237+0000 7faceba28640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T00:09:32.580 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:09:32 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0[42929]: 2024-09-16T00:09:32.240+0000 7face3848640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T00:09:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:32 smithi060 ceph-mon[35185]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:32 smithi060 ceph-mon[35185]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-16T00:09:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:32 smithi060 ceph-mon[35185]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T00:09:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:32 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:32 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:32 smithi060 ceph-mon[35185]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:09:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:32 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:09:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:32 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:09:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:32 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:32 smithi060 ceph-mon[35185]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' 2024-09-16T00:09:33.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:32 smithi033 ceph-mon[31243]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:33.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:32 smithi033 ceph-mon[31243]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-16T00:09:33.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:32 smithi033 ceph-mon[31243]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T00:09:33.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:33.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:33.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:32 smithi033 ceph-mon[31243]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:09:33.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:09:33.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:09:33.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:33.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:32 smithi033 ceph-mon[31243]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' 2024-09-16T00:09:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:32 smithi083 ceph-mon[35242]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:32 smithi083 ceph-mon[35242]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-16T00:09:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:32 smithi083 ceph-mon[35242]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T00:09:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:32 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:32 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:32 smithi083 ceph-mon[35242]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:09:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:32 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:09:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:32 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:09:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:32 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:33.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:32 smithi083 ceph-mon[35242]: from='osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656]' entity='osd.0' 2024-09-16T00:09:34.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:33 smithi033 ceph-mon[31243]: purged_snaps scrub starts 2024-09-16T00:09:34.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:33 smithi033 ceph-mon[31243]: purged_snaps scrub ok 2024-09-16T00:09:34.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:34.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:33 smithi033 ceph-mon[31243]: osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656] boot 2024-09-16T00:09:34.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:33 smithi033 ceph-mon[31243]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T00:09:34.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:34.094 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:33 smithi060 ceph-mon[35185]: purged_snaps scrub starts 2024-09-16T00:09:34.095 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:33 smithi060 ceph-mon[35185]: purged_snaps scrub ok 2024-09-16T00:09:34.095 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:34.095 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:33 smithi060 ceph-mon[35185]: osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656] boot 2024-09-16T00:09:34.095 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:33 smithi060 ceph-mon[35185]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T00:09:34.095 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:33 smithi083 ceph-mon[35242]: purged_snaps scrub starts 2024-09-16T00:09:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:33 smithi083 ceph-mon[35242]: purged_snaps scrub ok 2024-09-16T00:09:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:33 smithi083 ceph-mon[35242]: osd.0 [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656] boot 2024-09-16T00:09:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:33 smithi083 ceph-mon[35242]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T00:09:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T00:09:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:35 smithi033 ceph-mon[31243]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:35 smithi033 ceph-mon[31243]: from='client.? 172.21.15.60:0/3952469506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d"}]: dispatch 2024-09-16T00:09:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:35 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d"}]: dispatch 2024-09-16T00:09:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:35 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d"}]': finished 2024-09-16T00:09:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:35 smithi033 ceph-mon[31243]: osdmap e9: 2 total, 1 up, 2 in 2024-09-16T00:09:35.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:35.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:35 smithi033 ceph-mon[31243]: from='client.? 172.21.15.60:0/2907491488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:09:35.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:35 smithi083 ceph-mon[35242]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:35.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:35 smithi083 ceph-mon[35242]: from='client.? 172.21.15.60:0/3952469506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d"}]: dispatch 2024-09-16T00:09:35.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:35 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d"}]: dispatch 2024-09-16T00:09:35.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:35 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d"}]': finished 2024-09-16T00:09:35.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:35 smithi083 ceph-mon[35242]: osdmap e9: 2 total, 1 up, 2 in 2024-09-16T00:09:35.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:35.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:35 smithi083 ceph-mon[35242]: from='client.? 172.21.15.60:0/2907491488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:09:35.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:35 smithi060 ceph-mon[35185]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T00:09:35.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:35 smithi060 ceph-mon[35185]: from='client.? 172.21.15.60:0/3952469506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d"}]: dispatch 2024-09-16T00:09:35.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:35 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d"}]: dispatch 2024-09-16T00:09:35.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:35 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d"}]': finished 2024-09-16T00:09:35.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:35 smithi060 ceph-mon[35185]: osdmap e9: 2 total, 1 up, 2 in 2024-09-16T00:09:35.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:35.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:35 smithi060 ceph-mon[35185]: from='client.? 172.21.15.60:0/2907491488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:09:37.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:37 smithi033 ceph-mon[31243]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:37.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:37 smithi083 ceph-mon[35242]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:37.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:37 smithi060 ceph-mon[35185]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:39.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:39 smithi060 ceph-mon[35185]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:39.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:39 smithi033 ceph-mon[31243]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:39.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:39 smithi083 ceph-mon[35242]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:41.466 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:41 smithi060 ceph-mon[35185]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:41.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:41 smithi033 ceph-mon[31243]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:41.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:41 smithi083 ceph-mon[35242]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:43.442 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:43 smithi060 ceph-mon[35185]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:43.442 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-16T00:09:43.442 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:43.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:43 smithi033 ceph-mon[31243]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:43.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-16T00:09:43.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:43.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:43 smithi083 ceph-mon[35242]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:43.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-16T00:09:43.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:44.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:44 smithi060 ceph-mon[35185]: Deploying daemon osd.1 on smithi060 2024-09-16T00:09:44.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:44 smithi033 ceph-mon[31243]: Deploying daemon osd.1 on smithi060 2024-09-16T00:09:44.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:44 smithi083 ceph-mon[35242]: Deploying daemon osd.1 on smithi060 2024-09-16T00:09:45.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:45 smithi060 ceph-mon[35185]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:45.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:45 smithi033 ceph-mon[31243]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:45.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:45 smithi083 ceph-mon[35242]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:47.556 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:47 smithi060 ceph-mon[35185]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:47.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:47 smithi033 ceph-mon[31243]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:47.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:47 smithi083 ceph-mon[35242]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:49.405 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:49 smithi060 ceph-mon[35185]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:49.405 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:49 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:49.405 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:49 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:49.405 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:49 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:49 smithi033 ceph-mon[31243]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:49 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:49 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:49 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:49.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:49 smithi083 ceph-mon[35242]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:49.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:49 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:49.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:49 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:49.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:49 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:51.419 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:51 smithi060 ceph-mon[35185]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:51.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:51 smithi033 ceph-mon[31243]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:51.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:51 smithi083 ceph-mon[35242]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:53.217 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:53 smithi060 ceph-mon[35185]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:53.217 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:53 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:53.217 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:53 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:53.217 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:53 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:53.217 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:53 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:53.217 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:53 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:53.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.254+0000 7ff2f2ffd640 1 -- 172.21.15.60:0/3747604039 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff300062630 con 0x7ff2ec0a3260 2024-09-16T00:09:53.256 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 1 on host 'smithi060' 2024-09-16T00:09:53.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.256+0000 7ff307de0640 1 -- 172.21.15.60:0/3747604039 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff2ec0a3260 msgr2=0x7ff2ec0a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:53.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.256+0000 7ff307de0640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff2ec0a3260 0x7ff2ec0a5720 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff300199440 tx=0x7ff2f4005f90 comp rx=0 tx=0).stop 2024-09-16T00:09:53.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.256+0000 7ff307de0640 1 -- 172.21.15.60:0/3747604039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff300102660 msgr2=0x7ff300197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:53.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.256+0000 7ff307de0640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff300102660 0x7ff300197f20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009230 tx=0x7ff2fc031b60 comp rx=0 tx=0).stop 2024-09-16T00:09:53.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.256+0000 7ff307de0640 1 -- 172.21.15.60:0/3747604039 shutdown_connections 2024-09-16T00:09:53.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.256+0000 7ff307de0640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff2ec0a3260 0x7ff2ec0a5720 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:53.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.256+0000 7ff307de0640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3001078c0 0x7ff30019f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:53.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.256+0000 7ff307de0640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff300104f90 0x7ff300198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:53.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.256+0000 7ff307de0640 1 --2- 172.21.15.60:0/3747604039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff300102660 0x7ff300197f20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:53.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.256+0000 7ff307de0640 1 -- 172.21.15.60:0/3747604039 >> 172.21.15.60:0/3747604039 conn(0x7ff3000fc280 msgr2=0x7ff300102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:09:53.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.256+0000 7ff307de0640 1 -- 172.21.15.60:0/3747604039 shutdown_connections 2024-09-16T00:09:53.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:09:53.256+0000 7ff307de0640 1 -- 172.21.15.60:0/3747604039 wait complete. 2024-09-16T00:09:53.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:53 smithi033 ceph-mon[31243]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:53.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:53 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:53.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:53 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:53.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:53 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:53.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:53 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:53.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:53 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:53.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:53 smithi083 ceph-mon[35242]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:53.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:53 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:53.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:53 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:53.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:53 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:53.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:53 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:53.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:53 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:53.912 DEBUG:teuthology.orchestra.run.smithi060:osd.1> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.1.service 2024-09-16T00:09:53.915 INFO:tasks.cephadm:Deploying osd.2 on smithi083 with /dev/vg_nvme/lv_4... 2024-09-16T00:09:53.915 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T00:09:54.169 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:09:54.206 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:54 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:54.206 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:54 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:54.206 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:54 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:54.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:54 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:54.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:54 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:54.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:54 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:54.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:54 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:09:54.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:54 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:54.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:54 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:55.382 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:55 smithi083 ceph-mon[35242]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:55.382 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:55 smithi083 ceph-mon[35242]: from='osd.1 [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T00:09:55.382 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:55 smithi083 ceph-mon[35242]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T00:09:55.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:55 smithi060 ceph-mon[35185]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:55.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:55 smithi060 ceph-mon[35185]: from='osd.1 [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T00:09:55.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:55 smithi060 ceph-mon[35185]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T00:09:55.481 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:09:55 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1[39634]: 2024-09-16T00:09:55.210+0000 7ff6a6df3640 -1 osd.1 0 log_to_monitors true 2024-09-16T00:09:55.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:55 smithi033 ceph-mon[31243]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:55.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:55 smithi033 ceph-mon[31243]: from='osd.1 [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T00:09:55.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:55 smithi033 ceph-mon[31243]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T00:09:56.439 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:09:56.465 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_4 2024-09-16T00:09:56.468 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:56 smithi060 ceph-mon[35185]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T00:09:56.468 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:56 smithi060 ceph-mon[35185]: from='osd.1 [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:09:56.468 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:56 smithi060 ceph-mon[35185]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T00:09:56.468 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:56.469 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:56 smithi060 ceph-mon[35185]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:09:56.469 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:56 smithi060 ceph-mon[35185]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-09-16T00:09:56.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:56 smithi033 ceph-mon[31243]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T00:09:56.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:56 smithi033 ceph-mon[31243]: from='osd.1 [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:09:56.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:56 smithi033 ceph-mon[31243]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T00:09:56.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:56.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:56 smithi033 ceph-mon[31243]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:09:56.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:56 smithi033 ceph-mon[31243]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-09-16T00:09:56.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:56 smithi083 ceph-mon[35242]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T00:09:56.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:56 smithi083 ceph-mon[35242]: from='osd.1 [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:09:56.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:56 smithi083 ceph-mon[35242]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T00:09:56.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:56.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:56 smithi083 ceph-mon[35242]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:09:56.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:56 smithi083 ceph-mon[35242]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-09-16T00:09:56.728 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:09:56.730 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:09:56 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1[39634]: 2024-09-16T00:09:56.674+0000 7ff6a2d76640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T00:09:56.730 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:09:56 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1[39634]: 2024-09-16T00:09:56.677+0000 7ff69ab96640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T00:09:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.157+0000 7fefe4dc5640 1 -- 172.21.15.83:0/2392174223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefe0101cd0 msgr2=0x7fefe0102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.157+0000 7fefe4dc5640 1 --2- 172.21.15.83:0/2392174223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefe0101cd0 0x7fefe0102150 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009900 tx=0x7fefd002a410 comp rx=0 tx=0).stop 2024-09-16T00:09:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.157+0000 7fefe4dc5640 1 -- 172.21.15.83:0/2392174223 shutdown_connections 2024-09-16T00:09:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.157+0000 7fefe4dc5640 1 --2- 172.21.15.83:0/2392174223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fefe0102690 0x7fefe0108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.157+0000 7fefe4dc5640 1 --2- 172.21.15.83:0/2392174223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefe0101cd0 0x7fefe0102150 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.157+0000 7fefe4dc5640 1 --2- 172.21.15.83:0/2392174223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe0100ad0 0x7fefe0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.157+0000 7fefe4dc5640 1 -- 172.21.15.83:0/2392174223 >> 172.21.15.83:0/2392174223 conn(0x7fefe00fc280 msgr2=0x7fefe00fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:09:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.157+0000 7fefe4dc5640 1 -- 172.21.15.83:0/2392174223 shutdown_connections 2024-09-16T00:09:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.157+0000 7fefe4dc5640 1 -- 172.21.15.83:0/2392174223 wait complete. 2024-09-16T00:09:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefe4dc5640 1 Processor -- start 2024-09-16T00:09:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefe4dc5640 1 -- start start 2024-09-16T00:09:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefe4dc5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefe0100ad0 0x7fefe0104a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:09:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefe4dc5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe0101cd0 0x7fefe01030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:09:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefe4dc5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fefe0102690 0x7fefe01035f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:09:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefe4dc5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fefe010c730 con 0x7fefe0102690 2024-09-16T00:09:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefe4dc5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fefe010c5b0 con 0x7fefe0101cd0 2024-09-16T00:09:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefe4dc5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fefe010c8b0 con 0x7fefe0100ad0 2024-09-16T00:09:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefde59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefe0100ad0 0x7fefe0104a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:09:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefde59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefe0100ad0 0x7fefe0104a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58414/0 (socket says 172.21.15.83:58414) 2024-09-16T00:09:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefde59c640 1 -- 172.21.15.83:0/391890932 learned_addr learned my addr 172.21.15.83:0/391890932 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-16T00:09:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefded9d640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fefe0102690 0x7fefe01035f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefddd9b640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe0101cd0 0x7fefe01030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefde59c640 1 -- 172.21.15.83:0/391890932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe0101cd0 msgr2=0x7fefe01030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefde59c640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe0101cd0 0x7fefe01030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefde59c640 1 -- 172.21.15.83:0/391890932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fefe0102690 msgr2=0x7fefe01035f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefde59c640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fefe0102690 0x7fefe01035f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefde59c640 1 -- 172.21.15.83:0/391890932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe0103eb0 con 0x7fefe0100ad0 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.158+0000 7fefde59c640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefe0100ad0 0x7fefe0104a60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fefd400d770 tx=0x7fefd400dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.159+0000 7fefcf7fe640 1 -- 172.21.15.83:0/391890932 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4017070 con 0x7fefe0100ad0 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.159+0000 7fefe4dc5640 1 -- 172.21.15.83:0/391890932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe01a2090 con 0x7fefe0100ad0 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.159+0000 7fefcf7fe640 1 -- 172.21.15.83:0/391890932 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefd4009c30 con 0x7fefe0100ad0 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.159+0000 7fefcf7fe640 1 -- 172.21.15.83:0/391890932 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4003dd0 con 0x7fefe0100ad0 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.159+0000 7fefe4dc5640 1 -- 172.21.15.83:0/391890932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe01a25f0 con 0x7fefe0100ad0 2024-09-16T00:09:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.160+0000 7fefe4dc5640 1 -- 172.21.15.83:0/391890932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe01a28e0 con 0x7fefe0100ad0 2024-09-16T00:09:57.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.162+0000 7fefcf7fe640 1 -- 172.21.15.83:0/391890932 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7fefd4003f70 con 0x7fefe0100ad0 2024-09-16T00:09:57.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.162+0000 7fefcf7fe640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fefc80a3230 0x7fefc80a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:09:57.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.162+0000 7fefcf7fe640 1 -- 172.21.15.83:0/391890932 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 2029+0+0 (secure 0 0 0) 0x7fefd40cde10 con 0x7fefe0100ad0 2024-09-16T00:09:57.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.162+0000 7fefddd9b640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fefc80a3230 0x7fefc80a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:09:57.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.163+0000 7fefddd9b640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fefc80a3230 0x7fefc80a56f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fefd0000c00 tx=0x7fefd00315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:09:57.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.166+0000 7fefcf7fe640 1 -- 172.21.15.83:0/391890932 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fefd4009de0 con 0x7fefe0100ad0 2024-09-16T00:09:57.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:09:57.347+0000 7fefe4dc5640 1 -- 172.21.15.83:0/391890932 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fefe006a6d0 con 0x7fefc80a3230 2024-09-16T00:09:57.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:57.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T00:09:57.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:57.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:57.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: Detected new or changed devices on smithi060 2024-09-16T00:09:57.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:57.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:57.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:09:57.349 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:57.349 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:57.349 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:57.349 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:57.349 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: from='osd.1 ' entity='osd.1' 2024-09-16T00:09:57.349 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:57.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:57.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T00:09:57.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:57.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:57.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: Detected new or changed devices on smithi060 2024-09-16T00:09:57.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:57.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:57.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:09:57.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:57.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:57.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:57.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:57.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: from='osd.1 ' entity='osd.1' 2024-09-16T00:09:57.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T00:09:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: Detected new or changed devices on smithi060 2024-09-16T00:09:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:09:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:09:57.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:09:57.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: from='osd.1 ' entity='osd.1' 2024-09-16T00:09:57.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:58.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:58 smithi033 ceph-mon[31243]: purged_snaps scrub starts 2024-09-16T00:09:58.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:58 smithi033 ceph-mon[31243]: purged_snaps scrub ok 2024-09-16T00:09:58.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:58 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi060 to 17092M 2024-09-16T00:09:58.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:58 smithi033 ceph-mon[31243]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:09:58.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:09:58.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:09:58.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:58.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:58 smithi033 ceph-mon[31243]: osd.1 [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239] boot 2024-09-16T00:09:58.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:58 smithi033 ceph-mon[31243]: osdmap e12: 2 total, 2 up, 2 in 2024-09-16T00:09:58.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:58 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:58.606 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:58 smithi083 ceph-mon[35242]: purged_snaps scrub starts 2024-09-16T00:09:58.606 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:58 smithi083 ceph-mon[35242]: purged_snaps scrub ok 2024-09-16T00:09:58.606 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:58 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi060 to 17092M 2024-09-16T00:09:58.606 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:58 smithi083 ceph-mon[35242]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:09:58.606 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:09:58.607 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:09:58.607 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:58.607 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:58 smithi083 ceph-mon[35242]: osd.1 [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239] boot 2024-09-16T00:09:58.607 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:58 smithi083 ceph-mon[35242]: osdmap e12: 2 total, 2 up, 2 in 2024-09-16T00:09:58.607 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:58 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:58.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:58 smithi060 ceph-mon[35185]: purged_snaps scrub starts 2024-09-16T00:09:58.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:58 smithi060 ceph-mon[35185]: purged_snaps scrub ok 2024-09-16T00:09:58.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:58 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi060 to 17092M 2024-09-16T00:09:58.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:58 smithi060 ceph-mon[35185]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:09:58.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:09:58.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:09:58.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:09:58.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:58 smithi060 ceph-mon[35185]: osd.1 [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239] boot 2024-09-16T00:09:58.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:58 smithi060 ceph-mon[35185]: osdmap e12: 2 total, 2 up, 2 in 2024-09-16T00:09:58.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:58 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T00:09:59.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:59 smithi033 ceph-mon[31243]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:59.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:09:59 smithi033 ceph-mon[31243]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T00:09:59.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:59 smithi083 ceph-mon[35242]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:59.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:09:59 smithi083 ceph-mon[35242]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T00:09:59.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:59 smithi060 ceph-mon[35185]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T00:09:59.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:09:59 smithi060 ceph-mon[35185]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T00:10:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:00 smithi083 ceph-mon[35242]: from='client.? 172.21.15.83:0/3173742285' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b978c9fd-d8e4-4b4a-8bc2-1671f0232a49"}]: dispatch 2024-09-16T00:10:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:00 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b978c9fd-d8e4-4b4a-8bc2-1671f0232a49"}]: dispatch 2024-09-16T00:10:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:00 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b978c9fd-d8e4-4b4a-8bc2-1671f0232a49"}]': finished 2024-09-16T00:10:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:00 smithi083 ceph-mon[35242]: osdmap e14: 3 total, 2 up, 3 in 2024-09-16T00:10:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:00.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:00 smithi083 ceph-mon[35242]: overall HEALTH_OK 2024-09-16T00:10:00.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:00 smithi083 ceph-mon[35242]: from='client.? 172.21.15.83:0/515912883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:10:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:00 smithi033 ceph-mon[31243]: from='client.? 172.21.15.83:0/3173742285' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b978c9fd-d8e4-4b4a-8bc2-1671f0232a49"}]: dispatch 2024-09-16T00:10:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:00 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b978c9fd-d8e4-4b4a-8bc2-1671f0232a49"}]: dispatch 2024-09-16T00:10:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:00 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b978c9fd-d8e4-4b4a-8bc2-1671f0232a49"}]': finished 2024-09-16T00:10:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:00 smithi033 ceph-mon[31243]: osdmap e14: 3 total, 2 up, 3 in 2024-09-16T00:10:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:00 smithi033 ceph-mon[31243]: overall HEALTH_OK 2024-09-16T00:10:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:00 smithi033 ceph-mon[31243]: from='client.? 172.21.15.83:0/515912883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:10:00.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:00 smithi060 ceph-mon[35185]: from='client.? 172.21.15.83:0/3173742285' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b978c9fd-d8e4-4b4a-8bc2-1671f0232a49"}]: dispatch 2024-09-16T00:10:00.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:00 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b978c9fd-d8e4-4b4a-8bc2-1671f0232a49"}]: dispatch 2024-09-16T00:10:00.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:00 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b978c9fd-d8e4-4b4a-8bc2-1671f0232a49"}]': finished 2024-09-16T00:10:00.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:00 smithi060 ceph-mon[35185]: osdmap e14: 3 total, 2 up, 3 in 2024-09-16T00:10:00.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:00.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:00 smithi060 ceph-mon[35185]: overall HEALTH_OK 2024-09-16T00:10:00.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:00 smithi060 ceph-mon[35185]: from='client.? 172.21.15.83:0/515912883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:10:01.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:01 smithi033 ceph-mon[31243]: pgmap v59: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:01.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:01 smithi083 ceph-mon[35242]: pgmap v59: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:01 smithi060 ceph-mon[35185]: pgmap v59: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:03 smithi033 ceph-mon[31243]: pgmap v61: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:03 smithi083 ceph-mon[35242]: pgmap v61: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:03.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:03 smithi060 ceph-mon[35185]: pgmap v61: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:05 smithi033 ceph-mon[31243]: pgmap v62: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:05.615 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:05 smithi083 ceph-mon[35242]: pgmap v62: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:05.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:05 smithi060 ceph-mon[35185]: pgmap v62: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:07.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:07 smithi033 ceph-mon[31243]: pgmap v63: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:07.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:07 smithi083 ceph-mon[35242]: pgmap v63: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:07.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:07 smithi060 ceph-mon[35185]: pgmap v63: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:08.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-16T00:10:08.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-16T00:10:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:08.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-16T00:10:08.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:09 smithi033 ceph-mon[31243]: pgmap v64: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:09.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:09 smithi033 ceph-mon[31243]: Deploying daemon osd.2 on smithi083 2024-09-16T00:10:09.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:09 smithi083 ceph-mon[35242]: pgmap v64: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:09.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:09 smithi083 ceph-mon[35242]: Deploying daemon osd.2 on smithi083 2024-09-16T00:10:09.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:09 smithi060 ceph-mon[35185]: pgmap v64: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:09 smithi060 ceph-mon[35185]: Deploying daemon osd.2 on smithi083 2024-09-16T00:10:11.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:11 smithi033 ceph-mon[31243]: pgmap v65: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:11.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:11 smithi083 ceph-mon[35242]: pgmap v65: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:11.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:11 smithi060 ceph-mon[35185]: pgmap v65: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:13.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:13 smithi083 ceph-mon[35242]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:13.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:13 smithi033 ceph-mon[31243]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:13.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:13 smithi060 ceph-mon[35185]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:14.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:10:14.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:14.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:14.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:10:14.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:14.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:14.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:10:14.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:14.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:15.570 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:15 smithi083 ceph-mon[35242]: pgmap v67: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:15 smithi033 ceph-mon[31243]: pgmap v67: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:15.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:15 smithi060 ceph-mon[35185]: pgmap v67: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:17 smithi083 ceph-mon[35242]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:17 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:17 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:17 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:17 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:10:17.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:17 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:17 smithi060 ceph-mon[35185]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:17 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:17 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:17 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:17 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:10:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:17 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:17.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:17 smithi033 ceph-mon[31243]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:17.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:17.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:17.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:17.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:10:17.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:18.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.075+0000 7fefcf7fe640 1 -- 172.21.15.83:0/391890932 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fefe006a6d0 con 0x7fefc80a3230 2024-09-16T00:10:18.076 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 2 on host 'smithi083' 2024-09-16T00:10:18.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.077+0000 7fefe4dc5640 1 -- 172.21.15.83:0/391890932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fefc80a3230 msgr2=0x7fefc80a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:10:18.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.077+0000 7fefe4dc5640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fefc80a3230 0x7fefc80a56f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fefd0000c00 tx=0x7fefd00315d0 comp rx=0 tx=0).stop 2024-09-16T00:10:18.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.077+0000 7fefe4dc5640 1 -- 172.21.15.83:0/391890932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefe0100ad0 msgr2=0x7fefe0104a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:10:18.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.077+0000 7fefe4dc5640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefe0100ad0 0x7fefe0104a60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fefd400d770 tx=0x7fefd400dc30 comp rx=0 tx=0).stop 2024-09-16T00:10:18.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.077+0000 7fefe4dc5640 1 -- 172.21.15.83:0/391890932 shutdown_connections 2024-09-16T00:10:18.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.077+0000 7fefe4dc5640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fefc80a3230 0x7fefc80a56f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:18.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.077+0000 7fefe4dc5640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fefe0102690 0x7fefe01035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:18.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.077+0000 7fefe4dc5640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe0101cd0 0x7fefe01030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:18.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.077+0000 7fefe4dc5640 1 --2- 172.21.15.83:0/391890932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefe0100ad0 0x7fefe0104a60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:18.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.077+0000 7fefe4dc5640 1 -- 172.21.15.83:0/391890932 >> 172.21.15.83:0/391890932 conn(0x7fefe00fc280 msgr2=0x7fefe0069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:10:18.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.077+0000 7fefe4dc5640 1 -- 172.21.15.83:0/391890932 shutdown_connections 2024-09-16T00:10:18.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:10:18.077+0000 7fefe4dc5640 1 -- 172.21.15.83:0/391890932 wait complete. 2024-09-16T00:10:18.381 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:10:18.381 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:18.381 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:18.382 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:10:18.382 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:18.382 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:18.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:10:18.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:18.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:18.704 DEBUG:teuthology.orchestra.run.smithi083:osd.2> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.2.service 2024-09-16T00:10:18.707 INFO:tasks.cephadm:Deploying osd.3 on smithi033 with /dev/vg_nvme/lv_3... 2024-09-16T00:10:18.707 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T00:10:18.961 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:10:19.234 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:19 smithi083 ceph-mon[35242]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:19.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:19 smithi033 ceph-mon[31243]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:19.397 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:19 smithi060 ceph-mon[35185]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:19.838 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:10:19 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2[40190]: 2024-09-16T00:10:19.749+0000 7fa2507ca640 -1 osd.2 0 log_to_monitors true 2024-09-16T00:10:20.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:20 smithi033 ceph-mon[31243]: from='osd.2 [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T00:10:20.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:20 smithi033 ceph-mon[31243]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T00:10:20.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:20 smithi083 ceph-mon[35242]: from='osd.2 [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T00:10:20.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:20 smithi083 ceph-mon[35242]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T00:10:20.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:20 smithi060 ceph-mon[35185]: from='osd.2 [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T00:10:20.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:20 smithi060 ceph-mon[35185]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T00:10:21.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:21 smithi083 ceph-mon[35242]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:21.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:21 smithi083 ceph-mon[35242]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T00:10:21.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:21 smithi083 ceph-mon[35242]: osdmap e15: 3 total, 2 up, 3 in 2024-09-16T00:10:21.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:21 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:21.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:21 smithi083 ceph-mon[35242]: from='osd.2 [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:10:21.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:21 smithi083 ceph-mon[35242]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:10:21.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:21 smithi060 ceph-mon[35185]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:21.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:21 smithi060 ceph-mon[35185]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T00:10:21.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:21 smithi060 ceph-mon[35185]: osdmap e15: 3 total, 2 up, 3 in 2024-09-16T00:10:21.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:21 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:21.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:21 smithi060 ceph-mon[35185]: from='osd.2 [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:10:21.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:21 smithi060 ceph-mon[35185]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:10:21.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:21 smithi033 ceph-mon[31243]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:21.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:21 smithi033 ceph-mon[31243]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T00:10:21.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:21 smithi033 ceph-mon[31243]: osdmap e15: 3 total, 2 up, 3 in 2024-09-16T00:10:21.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:21 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:21.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:21 smithi033 ceph-mon[31243]: from='osd.2 [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:10:21.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:21 smithi033 ceph-mon[31243]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:10:21.586 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:10:21 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2[40190]: 2024-09-16T00:10:21.523+0000 7fa24c74d640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T00:10:21.587 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:10:21 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2[40190]: 2024-09-16T00:10:21.527+0000 7fa24456d640 -1 osd.2 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T00:10:21.601 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:10:21.631 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_3 2024-09-16T00:10:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:10:22.181 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:22 smithi033 ceph-mon[31243]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-16T00:10:22.181 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:22 smithi033 ceph-mon[31243]: osdmap e16: 3 total, 2 up, 3 in 2024-09-16T00:10:22.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:22.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:22.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:22 smithi033 ceph-mon[31243]: from='osd.2 ' entity='osd.2' 2024-09-16T00:10:22.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:22.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:22.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:10:22.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:22.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:22.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:10:22.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:22.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.362+0000 7f666fcb9640 1 -- 172.21.15.33:0/3245479492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6668101cf0 msgr2=0x7f6668102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:10:22.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.362+0000 7f666fcb9640 1 --2- 172.21.15.33:0/3245479492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6668101cf0 0x7f6668102170 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6658009900 tx=0x7f665802a410 comp rx=0 tx=0).stop 2024-09-16T00:10:22.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.362+0000 7f666fcb9640 1 -- 172.21.15.33:0/3245479492 shutdown_connections 2024-09-16T00:10:22.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.362+0000 7f666fcb9640 1 --2- 172.21.15.33:0/3245479492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66681026b0 0x7f6668108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:22.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.362+0000 7f666fcb9640 1 --2- 172.21.15.33:0/3245479492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6668101cf0 0x7f6668102170 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:22.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.362+0000 7f666fcb9640 1 --2- 172.21.15.33:0/3245479492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6668100af0 0x7f6668100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:22.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.362+0000 7f666fcb9640 1 -- 172.21.15.33:0/3245479492 >> 172.21.15.33:0/3245479492 conn(0x7f66680fc280 msgr2=0x7f66680fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:10:22.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.362+0000 7f666fcb9640 1 -- 172.21.15.33:0/3245479492 shutdown_connections 2024-09-16T00:10:22.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.362+0000 7f666fcb9640 1 -- 172.21.15.33:0/3245479492 wait complete. 2024-09-16T00:10:22.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.363+0000 7f666fcb9640 1 Processor -- start 2024-09-16T00:10:22.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.363+0000 7f666fcb9640 1 -- start start 2024-09-16T00:10:22.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.363+0000 7f666fcb9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6668100af0 0x7f6668196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:10:22.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.363+0000 7f666fcb9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6668101cf0 0x7f6668197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:10:22.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.363+0000 7f666fcb9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66681026b0 0x7f666819e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:10:22.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.363+0000 7f666fcb9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f666810c750 con 0x7f6668100af0 2024-09-16T00:10:22.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.363+0000 7f666fcb9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f666810c5d0 con 0x7f6668101cf0 2024-09-16T00:10:22.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.363+0000 7f666fcb9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f666810c8d0 con 0x7f66681026b0 2024-09-16T00:10:22.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.363+0000 7f666da55640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6668100af0 0x7f6668196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:10:22.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.363+0000 7f666da55640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6668100af0 0x7f6668196e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39212/0 (socket says 172.21.15.33:39212) 2024-09-16T00:10:22.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.363+0000 7f666da55640 1 -- 172.21.15.33:0/3546694351 learned_addr learned my addr 172.21.15.33:0/3546694351 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:10:22.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.363+0000 7f666e256640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66681026b0 0x7f666819e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:10:22.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.364+0000 7f666d254640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6668101cf0 0x7f6668197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:10:22.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.364+0000 7f666da55640 1 -- 172.21.15.33:0/3546694351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66681026b0 msgr2=0x7f666819e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:10:22.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.364+0000 7f666da55640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66681026b0 0x7f666819e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:22.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.364+0000 7f666da55640 1 -- 172.21.15.33:0/3546694351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6668101cf0 msgr2=0x7f6668197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:10:22.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.364+0000 7f666da55640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6668101cf0 0x7f6668197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:22.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.364+0000 7f666da55640 1 -- 172.21.15.33:0/3546694351 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f666806a790 con 0x7f6668100af0 2024-09-16T00:10:22.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.364+0000 7f666da55640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6668100af0 0x7f6668196e00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f666400d770 tx=0x7f666400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:10:22.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.364+0000 7f665effd640 1 -- 172.21.15.33:0/3546694351 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6664017070 con 0x7f6668100af0 2024-09-16T00:10:22.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.364+0000 7f666fcb9640 1 -- 172.21.15.33:0/3546694351 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f666806aa80 con 0x7f6668100af0 2024-09-16T00:10:22.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.364+0000 7f665effd640 1 -- 172.21.15.33:0/3546694351 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6664009c30 con 0x7f6668100af0 2024-09-16T00:10:22.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.364+0000 7f665effd640 1 -- 172.21.15.33:0/3546694351 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6664003da0 con 0x7f6668100af0 2024-09-16T00:10:22.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.364+0000 7f666fcb9640 1 -- 172.21.15.33:0/3546694351 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f666806afe0 con 0x7f6668100af0 2024-09-16T00:10:22.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.365+0000 7f666fcb9640 1 -- 172.21.15.33:0/3546694351 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6668107f80 con 0x7f6668100af0 2024-09-16T00:10:22.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.367+0000 7f665effd640 1 -- 172.21.15.33:0/3546694351 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f6664003f40 con 0x7f6668100af0 2024-09-16T00:10:22.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.368+0000 7f665effd640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f66540a31e0 0x7f66540a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:10:22.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.368+0000 7f666d254640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f66540a31e0 0x7f66540a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:10:22.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.368+0000 7f665effd640 1 -- 172.21.15.33:0/3546694351 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2571+0+0 (secure 0 0 0) 0x7f66640cdeb0 con 0x7f6668100af0 2024-09-16T00:10:22.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.368+0000 7f666d254640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f66540a31e0 0x7f66540a56a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f66580314e0 tx=0x7f6658000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:10:22.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.370+0000 7f665effd640 1 -- 172.21.15.33:0/3546694351 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6664009de0 con 0x7f6668100af0 2024-09-16T00:10:22.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:22 smithi083 ceph-mon[35242]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-16T00:10:22.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:22 smithi083 ceph-mon[35242]: osdmap e16: 3 total, 2 up, 3 in 2024-09-16T00:10:22.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:22.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:22.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:22 smithi083 ceph-mon[35242]: from='osd.2 ' entity='osd.2' 2024-09-16T00:10:22.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:22.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:22.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:10:22.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:22.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:22.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:10:22.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:22.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:22 smithi060 ceph-mon[35185]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-16T00:10:22.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:22 smithi060 ceph-mon[35185]: osdmap e16: 3 total, 2 up, 3 in 2024-09-16T00:10:22.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:22.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:22.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:22 smithi060 ceph-mon[35185]: from='osd.2 ' entity='osd.2' 2024-09-16T00:10:22.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:22.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:22.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:10:22.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:22.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:22.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:10:22.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:22.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:22.557+0000 7f666fcb9640 1 -- 172.21.15.33:0/3546694351 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f666806b2b0 con 0x7f66540a31e0 2024-09-16T00:10:23.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:23 smithi033 ceph-mon[31243]: purged_snaps scrub starts 2024-09-16T00:10:23.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:23 smithi033 ceph-mon[31243]: purged_snaps scrub ok 2024-09-16T00:10:23.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:23 smithi033 ceph-mon[31243]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:23 smithi033 ceph-mon[31243]: Detected new or changed devices on smithi083 2024-09-16T00:10:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:23 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi083 to 17088M 2024-09-16T00:10:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:23 smithi033 ceph-mon[31243]: osd.2 [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043] boot 2024-09-16T00:10:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:23 smithi033 ceph-mon[31243]: osdmap e17: 3 total, 3 up, 3 in 2024-09-16T00:10:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:10:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:10:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:23 smithi083 ceph-mon[35242]: purged_snaps scrub starts 2024-09-16T00:10:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:23 smithi083 ceph-mon[35242]: purged_snaps scrub ok 2024-09-16T00:10:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:23 smithi083 ceph-mon[35242]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:23 smithi083 ceph-mon[35242]: Detected new or changed devices on smithi083 2024-09-16T00:10:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:23 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi083 to 17088M 2024-09-16T00:10:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:23 smithi083 ceph-mon[35242]: osd.2 [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043] boot 2024-09-16T00:10:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:23 smithi083 ceph-mon[35242]: osdmap e17: 3 total, 3 up, 3 in 2024-09-16T00:10:23.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:23.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:10:23.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:10:23.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:23.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:23 smithi060 ceph-mon[35185]: purged_snaps scrub starts 2024-09-16T00:10:23.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:23 smithi060 ceph-mon[35185]: purged_snaps scrub ok 2024-09-16T00:10:23.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:23 smithi060 ceph-mon[35185]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-16T00:10:23.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:23 smithi060 ceph-mon[35185]: Detected new or changed devices on smithi083 2024-09-16T00:10:23.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:23 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi083 to 17088M 2024-09-16T00:10:23.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:23.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:23 smithi060 ceph-mon[35185]: osd.2 [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043] boot 2024-09-16T00:10:23.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:23 smithi060 ceph-mon[35185]: osdmap e17: 3 total, 3 up, 3 in 2024-09-16T00:10:23.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T00:10:23.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:10:23.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:10:23.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:24.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:24 smithi033 ceph-mon[31243]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:10:24.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:24 smithi033 ceph-mon[31243]: osdmap e18: 3 total, 3 up, 3 in 2024-09-16T00:10:24.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:24 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-16T00:10:24.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:24 smithi083 ceph-mon[35242]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:10:24.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:24 smithi083 ceph-mon[35242]: osdmap e18: 3 total, 3 up, 3 in 2024-09-16T00:10:24.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:24 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-16T00:10:24.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:24 smithi060 ceph-mon[35185]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:10:24.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:24 smithi060 ceph-mon[35185]: osdmap e18: 3 total, 3 up, 3 in 2024-09-16T00:10:24.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:24 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-16T00:10:25.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:25 smithi083 ceph-mon[35242]: pgmap v76: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:25.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T00:10:25.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:25 smithi083 ceph-mon[35242]: osdmap e19: 3 total, 3 up, 3 in 2024-09-16T00:10:25.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T00:10:25.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:25 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1389186614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94dae86a-b8b7-4563-84f4-c14c9a07bda2"}]: dispatch 2024-09-16T00:10:25.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T00:10:25.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:25 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1389186614' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94dae86a-b8b7-4563-84f4-c14c9a07bda2"}]': finished 2024-09-16T00:10:25.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:25 smithi083 ceph-mon[35242]: osdmap e20: 4 total, 3 up, 4 in 2024-09-16T00:10:25.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:25.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:25 smithi060 ceph-mon[35185]: pgmap v76: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:25.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T00:10:25.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:25 smithi060 ceph-mon[35185]: osdmap e19: 3 total, 3 up, 3 in 2024-09-16T00:10:25.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T00:10:25.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:25 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1389186614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94dae86a-b8b7-4563-84f4-c14c9a07bda2"}]: dispatch 2024-09-16T00:10:25.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T00:10:25.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:25 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1389186614' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94dae86a-b8b7-4563-84f4-c14c9a07bda2"}]': finished 2024-09-16T00:10:25.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:25 smithi060 ceph-mon[35185]: osdmap e20: 4 total, 3 up, 4 in 2024-09-16T00:10:25.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:25.544 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:25 smithi033 ceph-mon[31243]: pgmap v76: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:25.545 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T00:10:25.545 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:25 smithi033 ceph-mon[31243]: osdmap e19: 3 total, 3 up, 3 in 2024-09-16T00:10:25.545 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T00:10:25.545 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:25 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1389186614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94dae86a-b8b7-4563-84f4-c14c9a07bda2"}]: dispatch 2024-09-16T00:10:25.545 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T00:10:25.545 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:25 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1389186614' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94dae86a-b8b7-4563-84f4-c14c9a07bda2"}]': finished 2024-09-16T00:10:25.545 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:25 smithi033 ceph-mon[31243]: osdmap e20: 4 total, 3 up, 4 in 2024-09-16T00:10:25.545 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:25.830 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:10:25 smithi033 sudo[47376]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T00:10:25.830 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:10:25 smithi033 sudo[47376]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T00:10:25.830 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:10:25 smithi033 sudo[47376]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T00:10:25.830 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:10:25 smithi033 sudo[47376]: pam_unix(sudo:session): session closed for user root 2024-09-16T00:10:26.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:26.150+0000 7f665effd640 1 -- 172.21.15.33:0/3546694351 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f66640982a0 con 0x7f6668100af0 2024-09-16T00:10:26.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:10:26 smithi033 sudo[47425]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T00:10:26.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:10:26 smithi033 sudo[47425]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T00:10:26.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:10:26 smithi033 sudo[47425]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T00:10:26.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:10:26 smithi033 sudo[47425]: pam_unix(sudo:session): session closed for user root 2024-09-16T00:10:26.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:26 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/465120644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:10:26.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:26 smithi033 ceph-mon[31243]: osdmap e21: 4 total, 3 up, 4 in 2024-09-16T00:10:26.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:26 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:26.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:26 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:26.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:26 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/465120644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:10:26.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:26 smithi083 ceph-mon[35242]: osdmap e21: 4 total, 3 up, 4 in 2024-09-16T00:10:26.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:26 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:26.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:26 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:26.480 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:10:26 smithi060 sudo[43361]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T00:10:26.481 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:10:26 smithi060 sudo[43361]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T00:10:26.481 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:10:26 smithi060 sudo[43361]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T00:10:26.481 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:10:26 smithi060 sudo[43361]: pam_unix(sudo:session): session closed for user root 2024-09-16T00:10:26.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:26 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/465120644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:10:26.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:26 smithi060 ceph-mon[35185]: osdmap e21: 4 total, 3 up, 4 in 2024-09-16T00:10:26.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:26 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:26.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:26 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:26.980 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:10:26 smithi060 sudo[43364]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T00:10:26.981 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:10:26 smithi060 sudo[43364]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T00:10:26.981 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:10:26 smithi060 sudo[43364]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T00:10:26.981 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:10:26 smithi060 sudo[43364]: pam_unix(sudo:session): session closed for user root 2024-09-16T00:10:27.086 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:10:26 smithi083 sudo[43860]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T00:10:27.086 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:10:26 smithi083 sudo[43860]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T00:10:27.086 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:10:26 smithi083 sudo[43860]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T00:10:27.086 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:10:26 smithi083 sudo[43860]: pam_unix(sudo:session): session closed for user root 2024-09-16T00:10:27.419 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:10:27 smithi083 sudo[43863]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T00:10:27.419 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:10:27 smithi083 sudo[43863]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T00:10:27.419 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:10:27 smithi083 sudo[43863]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T00:10:27.419 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:10:27 smithi083 sudo[43863]: pam_unix(sudo:session): session closed for user root 2024-09-16T00:10:27.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:27 smithi033 sudo[47992]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T00:10:27.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:27 smithi033 sudo[47992]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T00:10:27.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:27 smithi033 sudo[47992]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T00:10:27.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:27 smithi033 sudo[47992]: pam_unix(sudo:session): session closed for user root 2024-09-16T00:10:28.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 ceph-mon[35242]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:28.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 ceph-mon[35242]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-16T00:10:28.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:28.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:28.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 ceph-mon[35242]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T00:10:28.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:28 smithi033 sudo[48178]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T00:10:28.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:28 smithi033 sudo[48178]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T00:10:28.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:28 smithi033 sudo[48178]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T00:10:28.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:28 smithi033 sudo[48178]: pam_unix(sudo:session): session closed for user root 2024-09-16T00:10:28.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:28 smithi033 ceph-mon[31243]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:28.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:28 smithi033 ceph-mon[31243]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-16T00:10:28.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:28.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:28.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:28 smithi033 ceph-mon[31243]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T00:10:28.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:28 smithi060 ceph-mon[35185]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:28.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:28 smithi060 ceph-mon[35185]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-16T00:10:28.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:28.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:28.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:28 smithi060 ceph-mon[35185]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T00:10:28.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 sudo[43866]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T00:10:28.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 sudo[43866]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T00:10:28.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 sudo[43866]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T00:10:28.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 sudo[43866]: pam_unix(sudo:session): session closed for user root 2024-09-16T00:10:28.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 sudo[43869]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T00:10:28.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 sudo[43869]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T00:10:28.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 sudo[43869]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T00:10:28.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:28 smithi083 sudo[43869]: pam_unix(sudo:session): session closed for user root 2024-09-16T00:10:29.076 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:28 smithi060 sudo[43367]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T00:10:29.077 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:28 smithi060 sudo[43367]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T00:10:29.077 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:28 smithi060 sudo[43367]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T00:10:29.077 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:28 smithi060 sudo[43367]: pam_unix(sudo:session): session closed for user root 2024-09-16T00:10:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T00:10:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:10:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:10:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:10:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T00:10:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:10:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:10:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:10:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T00:10:29.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:10:29.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:10:29.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:10:29.455 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:29.455 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T00:10:29.455 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:10:29.455 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:10:29.455 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:10:29.455 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T00:10:29.455 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:10:29.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:10:29.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:10:29.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T00:10:29.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:10:29.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:10:29.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:10:29.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 sudo[43370]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T00:10:29.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 sudo[43370]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T00:10:29.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 sudo[43370]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T00:10:29.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 sudo[43370]: pam_unix(sudo:session): session closed for user root 2024-09-16T00:10:29.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:29.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T00:10:29.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:10:29.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:10:29.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:10:29.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T00:10:29.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:10:29.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:10:29.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:10:29.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T00:10:29.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T00:10:29.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T00:10:29.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T00:10:30.133 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:30 smithi033 ceph-mon[31243]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T00:10:30.133 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:30 smithi033 ceph-mon[31243]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T00:10:30.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:30 smithi083 ceph-mon[35242]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T00:10:30.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:30 smithi083 ceph-mon[35242]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T00:10:30.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:30 smithi060 ceph-mon[35185]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T00:10:30.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:30 smithi060 ceph-mon[35185]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T00:10:31.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:31 smithi033 ceph-mon[31243]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:31.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:31 smithi083 ceph-mon[35242]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:31.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:31 smithi060 ceph-mon[35185]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:33 smithi083 ceph-mon[35242]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-16T00:10:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:33.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:33 smithi033 ceph-mon[31243]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:33.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-16T00:10:33.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:33.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:33 smithi060 ceph-mon[35185]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:33.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-16T00:10:33.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:34.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:34 smithi033 ceph-mon[31243]: Deploying daemon osd.3 on smithi033 2024-09-16T00:10:34.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:34 smithi083 ceph-mon[35242]: Deploying daemon osd.3 on smithi033 2024-09-16T00:10:34.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:34 smithi060 ceph-mon[35185]: Deploying daemon osd.3 on smithi033 2024-09-16T00:10:35.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:35 smithi033 ceph-mon[31243]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-09-16T00:10:35.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:35 smithi083 ceph-mon[35242]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-09-16T00:10:35.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:35 smithi060 ceph-mon[35185]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-09-16T00:10:37.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:37 smithi083 ceph-mon[35242]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-16T00:10:37.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:37 smithi060 ceph-mon[35185]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-16T00:10:37.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:37 smithi033 ceph-mon[31243]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-16T00:10:39.372 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:39 smithi033 ceph-mon[31243]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-16T00:10:39.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:39 smithi083 ceph-mon[35242]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-16T00:10:39.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:39 smithi060 ceph-mon[35185]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-16T00:10:40.121 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:40 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:10:40.121 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:40 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:40.121 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:40 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:40.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:40 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:10:40.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:40 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:40.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:40 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:40.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:40 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:10:40.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:40 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:40.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:40 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:41.664 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:41 smithi033 ceph-mon[31243]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-16T00:10:41.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:41 smithi060 ceph-mon[35185]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-16T00:10:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:41 smithi083 ceph-mon[35242]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-16T00:10:42.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.977+0000 7f665effd640 1 -- 172.21.15.33:0/3546694351 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f666806b2b0 con 0x7f66540a31e0 2024-09-16T00:10:42.979 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 3 on host 'smithi033' 2024-09-16T00:10:42.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.979+0000 7f666fcb9640 1 -- 172.21.15.33:0/3546694351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f66540a31e0 msgr2=0x7f66540a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:10:42.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.979+0000 7f666fcb9640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f66540a31e0 0x7f66540a56a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f66580314e0 tx=0x7f6658000bc0 comp rx=0 tx=0).stop 2024-09-16T00:10:42.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.979+0000 7f666fcb9640 1 -- 172.21.15.33:0/3546694351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6668100af0 msgr2=0x7f6668196e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:10:42.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.979+0000 7f666fcb9640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6668100af0 0x7f6668196e00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f666400d770 tx=0x7f666400dc30 comp rx=0 tx=0).stop 2024-09-16T00:10:42.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.979+0000 7f666fcb9640 1 -- 172.21.15.33:0/3546694351 shutdown_connections 2024-09-16T00:10:42.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.979+0000 7f666fcb9640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f66540a31e0 0x7f66540a56a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:42.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.979+0000 7f666fcb9640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66681026b0 0x7f666819e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:42.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.979+0000 7f666fcb9640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6668101cf0 0x7f6668197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:42.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.979+0000 7f666fcb9640 1 --2- 172.21.15.33:0/3546694351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6668100af0 0x7f6668196e00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:42.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.979+0000 7f666fcb9640 1 -- 172.21.15.33:0/3546694351 >> 172.21.15.33:0/3546694351 conn(0x7f66680fc280 msgr2=0x7f6668105150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:10:42.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.979+0000 7f666fcb9640 1 -- 172.21.15.33:0/3546694351 shutdown_connections 2024-09-16T00:10:42.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:10:42.979+0000 7f666fcb9640 1 -- 172.21.15.33:0/3546694351 wait complete. 2024-09-16T00:10:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:43 smithi033 ceph-mon[31243]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-16T00:10:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:10:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:10:43.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:43 smithi083 ceph-mon[35242]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-16T00:10:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:10:43.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:10:43.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:43 smithi060 ceph-mon[35185]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-16T00:10:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:10:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:10:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:43.664 DEBUG:teuthology.orchestra.run.smithi033:osd.3> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.3.service 2024-09-16T00:10:43.666 INFO:tasks.cephadm:Deploying osd.4 on smithi060 with /dev/vg_nvme/lv_3... 2024-09-16T00:10:43.667 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T00:10:43.927 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T00:10:44.801 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:10:44 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3[49398]: 2024-09-16T00:10:44.472+0000 7fac0a99e640 -1 osd.3 0 log_to_monitors true 2024-09-16T00:10:45.344 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:45 smithi060 ceph-mon[35185]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-16T00:10:45.344 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:45 smithi060 ceph-mon[35185]: from='osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-16T00:10:45.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:45 smithi083 ceph-mon[35242]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-16T00:10:45.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:45 smithi083 ceph-mon[35242]: from='osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-16T00:10:45.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:45 smithi033 ceph-mon[31243]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-16T00:10:45.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:45 smithi033 ceph-mon[31243]: from='osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-16T00:10:46.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:46 smithi083 ceph-mon[35242]: from='osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T00:10:46.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:46 smithi083 ceph-mon[35242]: osdmap e22: 4 total, 3 up, 4 in 2024-09-16T00:10:46.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:46 smithi083 ceph-mon[35242]: from='osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-16T00:10:46.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:46 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:46.439 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:46 smithi060 ceph-mon[35185]: from='osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T00:10:46.439 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:46 smithi060 ceph-mon[35185]: osdmap e22: 4 total, 3 up, 4 in 2024-09-16T00:10:46.439 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:46 smithi060 ceph-mon[35185]: from='osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-16T00:10:46.439 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:46 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:46.442 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:10:46.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi060:vg_nvme/lv_3 2024-09-16T00:10:46.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:46 smithi033 ceph-mon[31243]: from='osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T00:10:46.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:46 smithi033 ceph-mon[31243]: osdmap e22: 4 total, 3 up, 4 in 2024-09-16T00:10:46.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:46 smithi033 ceph-mon[31243]: from='osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-16T00:10:46.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:46 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:46.533 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:10:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3[49398]: 2024-09-16T00:10:46.108+0000 7fac06921640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T00:10:46.533 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:10:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3[49398]: 2024-09-16T00:10:46.115+0000 7fac02749640 -1 osd.3 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T00:10:46.723 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T00:10:47.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.179+0000 7f2cd1365640 1 -- 172.21.15.60:0/3329842238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc104f90 msgr2=0x7f2ccc107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:10:47.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.179+0000 7f2cd1365640 1 --2- 172.21.15.60:0/3329842238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc104f90 0x7f2ccc107380 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009900 tx=0x7f2cbc02a410 comp rx=0 tx=0).stop 2024-09-16T00:10:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.179+0000 7f2cd1365640 1 -- 172.21.15.60:0/3329842238 shutdown_connections 2024-09-16T00:10:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.179+0000 7f2cd1365640 1 --2- 172.21.15.60:0/3329842238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ccc1078c0 0x7f2ccc109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.179+0000 7f2cd1365640 1 --2- 172.21.15.60:0/3329842238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc104f90 0x7f2ccc107380 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.179+0000 7f2cd1365640 1 --2- 172.21.15.60:0/3329842238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ccc102660 0x7f2ccc104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.179+0000 7f2cd1365640 1 -- 172.21.15.60:0/3329842238 >> 172.21.15.60:0/3329842238 conn(0x7f2ccc0fc280 msgr2=0x7f2ccc0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:10:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.179+0000 7f2cd1365640 1 -- 172.21.15.60:0/3329842238 shutdown_connections 2024-09-16T00:10:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.179+0000 7f2cd1365640 1 -- 172.21.15.60:0/3329842238 wait complete. 2024-09-16T00:10:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.180+0000 7f2cd1365640 1 Processor -- start 2024-09-16T00:10:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.180+0000 7f2cd1365640 1 -- start start 2024-09-16T00:10:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.180+0000 7f2cd1365640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ccc102660 0x7f2ccc197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:10:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.180+0000 7f2cd1365640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ccc104f90 0x7f2ccc198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:10:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.180+0000 7f2cd1365640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc1078c0 0x7f2ccc19f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:10:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2cd1365640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc10d840 con 0x7f2ccc104f90 2024-09-16T00:10:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2cd1365640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc10d6c0 con 0x7f2ccc1078c0 2024-09-16T00:10:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2cd1365640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc10d9c0 con 0x7f2ccc102660 2024-09-16T00:10:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2ccb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc1078c0 0x7f2ccc19f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:10:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2ccaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ccc102660 0x7f2ccc197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:10:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2ccb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc1078c0 0x7f2ccc19f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60658/0 (socket says 172.21.15.60:60658) 2024-09-16T00:10:47.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2ccb7fe640 1 -- 172.21.15.60:0/919647473 learned_addr learned my addr 172.21.15.60:0/919647473 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-09-16T00:10:47.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2cca7fc640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ccc104f90 0x7f2ccc198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:10:47.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2ccb7fe640 1 -- 172.21.15.60:0/919647473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ccc102660 msgr2=0x7f2ccc197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:10:47.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2ccb7fe640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ccc102660 0x7f2ccc197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:47.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2ccb7fe640 1 -- 172.21.15.60:0/919647473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ccc104f90 msgr2=0x7f2ccc198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:10:47.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2ccb7fe640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ccc104f90 0x7f2ccc198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:10:47.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2ccb7fe640 1 -- 172.21.15.60:0/919647473 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ccc19fb60 con 0x7f2ccc1078c0 2024-09-16T00:10:47.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2ccb7fe640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc1078c0 0x7f2ccc19f4f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400eb10 tx=0x7f2cc400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:10:47.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2cb3fff640 1 -- 172.21.15.60:0/919647473 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4004b20 con 0x7f2ccc1078c0 2024-09-16T00:10:47.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.181+0000 7f2cb3fff640 1 -- 172.21.15.60:0/919647473 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cc4004cc0 con 0x7f2ccc1078c0 2024-09-16T00:10:47.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.182+0000 7f2cb3fff640 1 -- 172.21.15.60:0/919647473 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4010760 con 0x7f2ccc1078c0 2024-09-16T00:10:47.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.182+0000 7f2cd1365640 1 -- 172.21.15.60:0/919647473 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ccc19fe50 con 0x7f2ccc1078c0 2024-09-16T00:10:47.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.182+0000 7f2cd1365640 1 -- 172.21.15.60:0/919647473 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ccc1a02b0 con 0x7f2ccc1078c0 2024-09-16T00:10:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.183+0000 7f2cd1365640 1 -- 172.21.15.60:0/919647473 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ccc0fdd70 con 0x7f2ccc1078c0 2024-09-16T00:10:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.184+0000 7f2cb3fff640 1 -- 172.21.15.60:0/919647473 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2cc4005020 con 0x7f2ccc1078c0 2024-09-16T00:10:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.184+0000 7f2cb3fff640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2cb40a3270 0x7f2cb40a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:10:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.185+0000 7f2cb3fff640 1 -- 172.21.15.60:0/919647473 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 3702+0+0 (secure 0 0 0) 0x7f2cc40cec90 con 0x7f2ccc1078c0 2024-09-16T00:10:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.185+0000 7f2ccaffd640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2cb40a3270 0x7f2cb40a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:10:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.185+0000 7f2ccaffd640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2cb40a3270 0x7f2cb40a5730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000bbe0 tx=0x7f2cc000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:10:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.189+0000 7f2cb3fff640 1 -- 172.21.15.60:0/919647473 <== mon.2 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2cc409cbb0 con 0x7f2ccc1078c0 2024-09-16T00:10:47.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:10:47.372+0000 7f2cd1365640 1 -- 172.21.15.60:0/919647473 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2ccc062630 con 0x7f2cb40a3270 2024-09-16T00:10:47.373 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:47.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: from='osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-16T00:10:47.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T00:10:47.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:47.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:47.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:47.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:47.375 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:10:47.375 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:10:47.375 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:47.375 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:47.375 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:10:47.375 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:47 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:47.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:47.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: from='osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-16T00:10:47.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T00:10:47.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:47.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:47.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:47.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:47.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:10:47.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:10:47.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:47.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:47.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:10:47.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:47.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:47.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: from='osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-16T00:10:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T00:10:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:10:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:10:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:10:47.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:10:48.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:48 smithi083 ceph-mon[35242]: purged_snaps scrub starts 2024-09-16T00:10:48.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:48 smithi083 ceph-mon[35242]: purged_snaps scrub ok 2024-09-16T00:10:48.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:48 smithi083 ceph-mon[35242]: Detected new or changed devices on smithi033 2024-09-16T00:10:48.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:48 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi033 to 8546M 2024-09-16T00:10:48.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:48.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:48 smithi083 ceph-mon[35242]: osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362] boot 2024-09-16T00:10:48.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:48 smithi083 ceph-mon[35242]: osdmap e24: 4 total, 4 up, 4 in 2024-09-16T00:10:48.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:48.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:48 smithi083 ceph-mon[35242]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:10:48.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:10:48.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:10:48.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:48 smithi060 ceph-mon[35185]: purged_snaps scrub starts 2024-09-16T00:10:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:48 smithi060 ceph-mon[35185]: purged_snaps scrub ok 2024-09-16T00:10:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:48 smithi060 ceph-mon[35185]: Detected new or changed devices on smithi033 2024-09-16T00:10:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:48 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi033 to 8546M 2024-09-16T00:10:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:48 smithi060 ceph-mon[35185]: osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362] boot 2024-09-16T00:10:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:48 smithi060 ceph-mon[35185]: osdmap e24: 4 total, 4 up, 4 in 2024-09-16T00:10:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:48.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:48 smithi060 ceph-mon[35185]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:10:48.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:10:48.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:10:48.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:48 smithi033 ceph-mon[31243]: purged_snaps scrub starts 2024-09-16T00:10:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:48 smithi033 ceph-mon[31243]: purged_snaps scrub ok 2024-09-16T00:10:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:48 smithi033 ceph-mon[31243]: Detected new or changed devices on smithi033 2024-09-16T00:10:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:48 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi033 to 8546M 2024-09-16T00:10:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:48 smithi033 ceph-mon[31243]: osd.3 [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362] boot 2024-09-16T00:10:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:48 smithi033 ceph-mon[31243]: osdmap e24: 4 total, 4 up, 4 in 2024-09-16T00:10:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T00:10:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:48 smithi033 ceph-mon[31243]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:10:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:10:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:10:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:49.221 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:49 smithi060 ceph-mon[35185]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:49.221 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:49 smithi060 ceph-mon[35185]: osdmap e25: 4 total, 4 up, 4 in 2024-09-16T00:10:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:49 smithi083 ceph-mon[35242]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:49 smithi083 ceph-mon[35242]: osdmap e25: 4 total, 4 up, 4 in 2024-09-16T00:10:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:49 smithi033 ceph-mon[31243]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-16T00:10:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:49 smithi033 ceph-mon[31243]: osdmap e25: 4 total, 4 up, 4 in 2024-09-16T00:10:50.382 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:50 smithi060 ceph-mon[35185]: osdmap e26: 4 total, 4 up, 4 in 2024-09-16T00:10:50.382 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:50 smithi060 ceph-mon[35185]: from='client.? 172.21.15.60:0/2553412920' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fa6bf5-b880-4cd0-9912-c64f33b78b34"}]: dispatch 2024-09-16T00:10:50.382 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:50 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fa6bf5-b880-4cd0-9912-c64f33b78b34"}]: dispatch 2024-09-16T00:10:50.382 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:50 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3fa6bf5-b880-4cd0-9912-c64f33b78b34"}]': finished 2024-09-16T00:10:50.382 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:50 smithi060 ceph-mon[35185]: osdmap e27: 5 total, 4 up, 5 in 2024-09-16T00:10:50.382 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:50 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:10:50.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:50 smithi083 ceph-mon[35242]: osdmap e26: 4 total, 4 up, 4 in 2024-09-16T00:10:50.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:50 smithi083 ceph-mon[35242]: from='client.? 172.21.15.60:0/2553412920' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fa6bf5-b880-4cd0-9912-c64f33b78b34"}]: dispatch 2024-09-16T00:10:50.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:50 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fa6bf5-b880-4cd0-9912-c64f33b78b34"}]: dispatch 2024-09-16T00:10:50.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:50 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3fa6bf5-b880-4cd0-9912-c64f33b78b34"}]': finished 2024-09-16T00:10:50.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:50 smithi083 ceph-mon[35242]: osdmap e27: 5 total, 4 up, 5 in 2024-09-16T00:10:50.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:50 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:10:50.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:50 smithi033 ceph-mon[31243]: osdmap e26: 4 total, 4 up, 4 in 2024-09-16T00:10:50.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:50 smithi033 ceph-mon[31243]: from='client.? 172.21.15.60:0/2553412920' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fa6bf5-b880-4cd0-9912-c64f33b78b34"}]: dispatch 2024-09-16T00:10:50.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:50 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fa6bf5-b880-4cd0-9912-c64f33b78b34"}]: dispatch 2024-09-16T00:10:50.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:50 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3fa6bf5-b880-4cd0-9912-c64f33b78b34"}]': finished 2024-09-16T00:10:50.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:50 smithi033 ceph-mon[31243]: osdmap e27: 5 total, 4 up, 5 in 2024-09-16T00:10:50.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:50 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:10:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:51 smithi083 ceph-mon[35242]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:10:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:51 smithi083 ceph-mon[35242]: from='client.? 172.21.15.60:0/742565318' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:10:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:51 smithi083 ceph-mon[35242]: osdmap e28: 5 total, 4 up, 5 in 2024-09-16T00:10:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:10:51.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:51 smithi060 ceph-mon[35185]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:10:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.60:0/742565318' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:10:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:51 smithi060 ceph-mon[35185]: osdmap e28: 5 total, 4 up, 5 in 2024-09-16T00:10:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:10:51.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:51 smithi033 ceph-mon[31243]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:10:51.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:51 smithi033 ceph-mon[31243]: from='client.? 172.21.15.60:0/742565318' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:10:51.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:51 smithi033 ceph-mon[31243]: osdmap e28: 5 total, 4 up, 5 in 2024-09-16T00:10:51.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:10:53.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:53 smithi083 ceph-mon[35242]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:10:53.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:53 smithi060 ceph-mon[35185]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:10:53.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:53 smithi033 ceph-mon[31243]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:10:55.300 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:55 smithi060 ceph-mon[35185]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-09-16T00:10:55.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:55 smithi083 ceph-mon[35242]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-09-16T00:10:55.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:55 smithi033 ceph-mon[31243]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-09-16T00:10:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:57 smithi083 ceph-mon[35242]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-16T00:10:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:57 smithi060 ceph-mon[35185]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-16T00:10:57.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:57 smithi033 ceph-mon[31243]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-16T00:10:59.371 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:59 smithi060 ceph-mon[35185]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T00:10:59.371 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:59 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-16T00:10:59.371 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:59 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:59.371 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:10:59 smithi060 ceph-mon[35185]: Deploying daemon osd.4 on smithi060 2024-09-16T00:10:59.397 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:59 smithi033 ceph-mon[31243]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T00:10:59.397 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:59 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-16T00:10:59.397 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:59 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:59.397 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:10:59 smithi033 ceph-mon[31243]: Deploying daemon osd.4 on smithi060 2024-09-16T00:10:59.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:59 smithi083 ceph-mon[35242]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T00:10:59.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:59 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-16T00:10:59.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:59 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:10:59.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:10:59 smithi083 ceph-mon[35242]: Deploying daemon osd.4 on smithi060 2024-09-16T00:11:01.367 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:01 smithi060 ceph-mon[35185]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-16T00:11:01.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:01 smithi083 ceph-mon[35242]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-16T00:11:01.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:01 smithi033 ceph-mon[31243]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-16T00:11:03.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:03 smithi083 ceph-mon[35242]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-16T00:11:03.457 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:03 smithi060 ceph-mon[35185]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-16T00:11:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:03 smithi033 ceph-mon[31243]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-16T00:11:05.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:05 smithi083 ceph-mon[35242]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:11:05.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:05.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:05.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:05.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:05 smithi060 ceph-mon[35185]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:11:05.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:05.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:05.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:05 smithi033 ceph-mon[31243]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:11:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:07.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:07 smithi083 ceph-mon[35242]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:07.431 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:07 smithi060 ceph-mon[35185]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:07.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:07 smithi033 ceph-mon[31243]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:08.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:08 smithi060 ceph-mon[35185]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:08.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:08.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:08.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:08.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:11:08.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:08.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.864+0000 7f2cb3fff640 1 -- 172.21.15.60:0/919647473 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2ccc062630 con 0x7f2cb40a3270 2024-09-16T00:11:08.865 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 4 on host 'smithi060' 2024-09-16T00:11:08.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.866+0000 7f2cd1365640 1 -- 172.21.15.60:0/919647473 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2cb40a3270 msgr2=0x7f2cb40a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:11:08.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.866+0000 7f2cd1365640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2cb40a3270 0x7f2cb40a5730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000bbe0 tx=0x7f2cc000b390 comp rx=0 tx=0).stop 2024-09-16T00:11:08.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.866+0000 7f2cd1365640 1 -- 172.21.15.60:0/919647473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc1078c0 msgr2=0x7f2ccc19f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:11:08.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.866+0000 7f2cd1365640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc1078c0 0x7f2ccc19f4f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400eb10 tx=0x7f2cc400c2d0 comp rx=0 tx=0).stop 2024-09-16T00:11:08.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.866+0000 7f2cd1365640 1 -- 172.21.15.60:0/919647473 shutdown_connections 2024-09-16T00:11:08.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.866+0000 7f2cd1365640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2cb40a3270 0x7f2cb40a5730 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:08.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.866+0000 7f2cd1365640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc1078c0 0x7f2ccc19f4f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:08.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.866+0000 7f2cd1365640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ccc104f90 0x7f2ccc198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:08.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.866+0000 7f2cd1365640 1 --2- 172.21.15.60:0/919647473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ccc102660 0x7f2ccc197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:08.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.866+0000 7f2cd1365640 1 -- 172.21.15.60:0/919647473 >> 172.21.15.60:0/919647473 conn(0x7f2ccc0fc280 msgr2=0x7f2ccc103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:11:08.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.866+0000 7f2cd1365640 1 -- 172.21.15.60:0/919647473 shutdown_connections 2024-09-16T00:11:08.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:11:08.866+0000 7f2cd1365640 1 -- 172.21.15.60:0/919647473 wait complete. 2024-09-16T00:11:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:08 smithi083 ceph-mon[35242]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:11:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:09.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:08 smithi033 ceph-mon[31243]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:09.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:09.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:09.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:09.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:11:09.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:09.535 DEBUG:teuthology.orchestra.run.smithi060:osd.4> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.4.service 2024-09-16T00:11:09.538 INFO:tasks.cephadm:Deploying osd.5 on smithi083 with /dev/vg_nvme/lv_3... 2024-09-16T00:11:09.538 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T00:11:09.793 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:11:09.799 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:09.799 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:09.799 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:09.914 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:09.914 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:09.914 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:10.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:10.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:10.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:11.066 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:10 smithi060 ceph-mon[35185]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:11.066 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:11:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4[46117]: 2024-09-16T00:11:10.985+0000 7f8243691640 -1 osd.4 0 log_to_monitors true 2024-09-16T00:11:11.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:10 smithi033 ceph-mon[31243]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:11.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:10 smithi083 ceph-mon[35242]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:12.154 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:11 smithi060 ceph-mon[35185]: from='osd.4 [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T00:11:12.154 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:11 smithi060 ceph-mon[35185]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T00:11:12.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:11 smithi083 ceph-mon[35242]: from='osd.4 [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T00:11:12.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:11 smithi083 ceph-mon[35242]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T00:11:12.269 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:11:12.295 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_3 2024-09-16T00:11:12.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:11 smithi033 ceph-mon[31243]: from='osd.4 [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T00:11:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:11 smithi033 ceph-mon[31243]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T00:11:12.544 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:11:12.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.970+0000 7f74722ed640 1 -- 172.21.15.83:0/1955228122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f746c101730 msgr2=0x7f746c101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:11:12.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.970+0000 7f74722ed640 1 --2- 172.21.15.83:0/1955228122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f746c101730 0x7f746c101be0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7464009ca0 tx=0x7f746402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:11:12.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.970+0000 7f74722ed640 1 -- 172.21.15.83:0/1955228122 shutdown_connections 2024-09-16T00:11:12.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.970+0000 7f74722ed640 1 --2- 172.21.15.83:0/1955228122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f746c101730 0x7f746c101be0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:12.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.970+0000 7f74722ed640 1 --2- 172.21.15.83:0/1955228122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f746c100d90 0x7f746c1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:12.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.970+0000 7f74722ed640 1 --2- 172.21.15.83:0/1955228122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f746c100450 0x7f746c100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:12.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.970+0000 7f74722ed640 1 -- 172.21.15.83:0/1955228122 >> 172.21.15.83:0/1955228122 conn(0x7f746c0fc280 msgr2=0x7f746c0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:11:12.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.970+0000 7f74722ed640 1 -- 172.21.15.83:0/1955228122 shutdown_connections 2024-09-16T00:11:12.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.970+0000 7f74722ed640 1 -- 172.21.15.83:0/1955228122 wait complete. 2024-09-16T00:11:12.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.971+0000 7f74722ed640 1 Processor -- start 2024-09-16T00:11:12.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.971+0000 7f74722ed640 1 -- start start 2024-09-16T00:11:12.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.971+0000 7f74722ed640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f746c100450 0x7f746c19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:11:12.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.971+0000 7f74722ed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f746c100d90 0x7f746c19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:11:12.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.971+0000 7f74722ed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f746c101730 0x7f746c1a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:11:12.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f74722ed640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f746c10fa70 con 0x7f746c100450 2024-09-16T00:11:12.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f74722ed640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f746c10f8f0 con 0x7f746c101730 2024-09-16T00:11:12.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f74722ed640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f746c10fbf0 con 0x7f746c100d90 2024-09-16T00:11:12.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f746b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f746c100d90 0x7f746c19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:11:12.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f746b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f746c100d90 0x7f746c19a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52052/0 (socket says 172.21.15.83:52052) 2024-09-16T00:11:12.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f746b7fe640 1 -- 172.21.15.83:0/1143634934 learned_addr learned my addr 172.21.15.83:0/1143634934 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-16T00:11:12.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f747088a640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f746c101730 0x7f746c1a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:11:12.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f746bfff640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f746c100450 0x7f746c19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:11:12.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f746b7fe640 1 -- 172.21.15.83:0/1143634934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f746c101730 msgr2=0x7f746c1a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:11:12.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f746b7fe640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f746c101730 0x7f746c1a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:12.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f746b7fe640 1 -- 172.21.15.83:0/1143634934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f746c100450 msgr2=0x7f746c19a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:11:12.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f746b7fe640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f746c100450 0x7f746c19a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:12.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f746b7fe640 1 -- 172.21.15.83:0/1143634934 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c1a1c90 con 0x7f746c100d90 2024-09-16T00:11:12.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f746bfff640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f746c100450 0x7f746c19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:11:12.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f746b7fe640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f746c100d90 0x7f746c19a5a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f746000c890 tx=0x7f746000cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:11:12.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f74697fa640 1 -- 172.21.15.83:0/1143634934 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460004220 con 0x7f746c100d90 2024-09-16T00:11:12.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f74722ed640 1 -- 172.21.15.83:0/1143634934 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f746c1a1f80 con 0x7f746c100d90 2024-09-16T00:11:12.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f74697fa640 1 -- 172.21.15.83:0/1143634934 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74600043c0 con 0x7f746c100d90 2024-09-16T00:11:12.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f74697fa640 1 -- 172.21.15.83:0/1143634934 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460004a00 con 0x7f746c100d90 2024-09-16T00:11:12.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.972+0000 7f74722ed640 1 -- 172.21.15.83:0/1143634934 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f746c1a2480 con 0x7f746c100d90 2024-09-16T00:11:12.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.973+0000 7f74722ed640 1 -- 172.21.15.83:0/1143634934 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f746c10b1a0 con 0x7f746c100d90 2024-09-16T00:11:12.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.975+0000 7f74697fa640 1 -- 172.21.15.83:0/1143634934 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f74600079b0 con 0x7f746c100d90 2024-09-16T00:11:12.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.975+0000 7f74697fa640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74540a3270 0x7f74540a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:11:12.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.975+0000 7f74697fa640 1 -- 172.21.15.83:0/1143634934 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 3821+0+0 (secure 0 0 0) 0x7f74600ce6f0 con 0x7f746c100d90 2024-09-16T00:11:12.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.975+0000 7f746bfff640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74540a3270 0x7f74540a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:11:12.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.976+0000 7f746bfff640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74540a3270 0x7f74540a5730 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f745c005d00 tx=0x7f745c005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:11:12.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:12.983+0000 7f74697fa640 1 -- 172.21.15.83:0/1143634934 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7460014030 con 0x7f746c100d90 2024-09-16T00:11:13.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:13.163+0000 7f74722ed640 1 -- 172.21.15.83:0/1143634934 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f746c062630 con 0x7f74540a3270 2024-09-16T00:11:13.164 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:13.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T00:11:13.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: from='osd.4 [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:11:13.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T00:11:13.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:13.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:11:13.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:13.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:13.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:11:13.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:11:13.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:13.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:13.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:11:13.166 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:13.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:13.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T00:11:13.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: from='osd.4 [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:11:13.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T00:11:13.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:13.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:11:13.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:13.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:13.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:11:13.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:11:13.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:13.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:13.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:11:13.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:13.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:13.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T00:11:13.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: from='osd.4 [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:11:13.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T00:11:13.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:13.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:11:13.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:13.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:13.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:11:13.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:11:13.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:13.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:13.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:11:13.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:13.730 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4[46117]: 2024-09-16T00:11:13.325+0000 7f823f614640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T00:11:13.731 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4[46117]: 2024-09-16T00:11:13.330+0000 7f8237434640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T00:11:14.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:13 smithi083 ceph-mon[35242]: Detected new or changed devices on smithi060 2024-09-16T00:11:14.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:13 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi060 to 8546M 2024-09-16T00:11:14.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:13 smithi083 ceph-mon[35242]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-09-16T00:11:14.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:13 smithi083 ceph-mon[35242]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T00:11:14.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:14.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:14.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:13 smithi083 ceph-mon[35242]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:11:14.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:11:14.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:11:14.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:14.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:13 smithi083 ceph-mon[35242]: from='osd.4 ' entity='osd.4' 2024-09-16T00:11:14.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-mon[35185]: Detected new or changed devices on smithi060 2024-09-16T00:11:14.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi060 to 8546M 2024-09-16T00:11:14.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-mon[35185]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-09-16T00:11:14.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-mon[35185]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T00:11:14.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:14.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:14.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-mon[35185]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:11:14.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:11:14.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:11:14.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:14.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:13 smithi060 ceph-mon[35185]: from='osd.4 ' entity='osd.4' 2024-09-16T00:11:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:13 smithi033 ceph-mon[31243]: Detected new or changed devices on smithi060 2024-09-16T00:11:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:13 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi060 to 8546M 2024-09-16T00:11:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:13 smithi033 ceph-mon[31243]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-09-16T00:11:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:13 smithi033 ceph-mon[31243]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T00:11:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:13 smithi033 ceph-mon[31243]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:11:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:11:14.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:11:14.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:14.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:13 smithi033 ceph-mon[31243]: from='osd.4 ' entity='osd.4' 2024-09-16T00:11:15.148 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:14 smithi083 ceph-mon[35242]: purged_snaps scrub starts 2024-09-16T00:11:15.148 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:14 smithi083 ceph-mon[35242]: purged_snaps scrub ok 2024-09-16T00:11:15.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:14 smithi083 ceph-mon[35242]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:15.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:15.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:14 smithi083 ceph-mon[35242]: osd.4 [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488] boot 2024-09-16T00:11:15.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:14 smithi083 ceph-mon[35242]: osdmap e31: 5 total, 5 up, 5 in 2024-09-16T00:11:15.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:15.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:14 smithi060 ceph-mon[35185]: purged_snaps scrub starts 2024-09-16T00:11:15.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:14 smithi060 ceph-mon[35185]: purged_snaps scrub ok 2024-09-16T00:11:15.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:14 smithi060 ceph-mon[35185]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:15.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:15.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:14 smithi060 ceph-mon[35185]: osd.4 [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488] boot 2024-09-16T00:11:15.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:14 smithi060 ceph-mon[35185]: osdmap e31: 5 total, 5 up, 5 in 2024-09-16T00:11:15.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:14 smithi033 ceph-mon[31243]: purged_snaps scrub starts 2024-09-16T00:11:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:14 smithi033 ceph-mon[31243]: purged_snaps scrub ok 2024-09-16T00:11:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:14 smithi033 ceph-mon[31243]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-16T00:11:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:14 smithi033 ceph-mon[31243]: osd.4 [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488] boot 2024-09-16T00:11:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:14 smithi033 ceph-mon[31243]: osdmap e31: 5 total, 5 up, 5 in 2024-09-16T00:11:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T00:11:16.036 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:15 smithi083 ceph-mon[35242]: from='client.? 172.21.15.83:0/238639126' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef"}]: dispatch 2024-09-16T00:11:16.036 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:15 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef"}]: dispatch 2024-09-16T00:11:16.036 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:15 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef"}]': finished 2024-09-16T00:11:16.036 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:15 smithi083 ceph-mon[35242]: osdmap e32: 6 total, 5 up, 6 in 2024-09-16T00:11:16.036 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:16.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:15 smithi060 ceph-mon[35185]: from='client.? 172.21.15.83:0/238639126' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef"}]: dispatch 2024-09-16T00:11:16.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:15 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef"}]: dispatch 2024-09-16T00:11:16.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:15 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef"}]': finished 2024-09-16T00:11:16.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:15 smithi060 ceph-mon[35185]: osdmap e32: 6 total, 5 up, 6 in 2024-09-16T00:11:16.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:16.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:15 smithi033 ceph-mon[31243]: from='client.? 172.21.15.83:0/238639126' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef"}]: dispatch 2024-09-16T00:11:16.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:15 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef"}]: dispatch 2024-09-16T00:11:16.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:15 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef"}]': finished 2024-09-16T00:11:16.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:15 smithi033 ceph-mon[31243]: osdmap e32: 6 total, 5 up, 6 in 2024-09-16T00:11:16.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:17.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:16 smithi083 ceph-mon[35242]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:17.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:16 smithi083 ceph-mon[35242]: from='client.? 172.21.15.83:0/1894701504' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:11:17.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:16 smithi060 ceph-mon[35185]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:17.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:16 smithi060 ceph-mon[35185]: from='client.? 172.21.15.83:0/1894701504' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:11:17.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:16 smithi033 ceph-mon[31243]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:17.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:16 smithi033 ceph-mon[31243]: from='client.? 172.21.15.83:0/1894701504' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:11:19.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:18 smithi083 ceph-mon[35242]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:19.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:18 smithi060 ceph-mon[35185]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:19.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:18 smithi033 ceph-mon[31243]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:21.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:20 smithi083 ceph-mon[35242]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:21.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:20 smithi060 ceph-mon[35185]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:21.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:20 smithi033 ceph-mon[31243]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:23.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:22 smithi083 ceph-mon[35242]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:23.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:22 smithi060 ceph-mon[35185]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:23.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:22 smithi033 ceph-mon[31243]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:24.118 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-16T00:11:24.118 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:24.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-16T00:11:24.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:24.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-16T00:11:24.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:25.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:24 smithi083 ceph-mon[35242]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:25.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:24 smithi083 ceph-mon[35242]: Deploying daemon osd.5 on smithi083 2024-09-16T00:11:25.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:24 smithi060 ceph-mon[35185]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:25.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:24 smithi060 ceph-mon[35185]: Deploying daemon osd.5 on smithi083 2024-09-16T00:11:25.332 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:24 smithi033 ceph-mon[31243]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:25.333 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:24 smithi033 ceph-mon[31243]: Deploying daemon osd.5 on smithi083 2024-09-16T00:11:27.134 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:26 smithi083 ceph-mon[35242]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:27.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:26 smithi060 ceph-mon[35185]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:27.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:26 smithi033 ceph-mon[31243]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:28.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:28 smithi083 ceph-mon[35242]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:29.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:28 smithi060 ceph-mon[35185]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:29.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:28 smithi033 ceph-mon[31243]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:31.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:30 smithi083 ceph-mon[35242]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:31.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:31.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:31.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:31.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:30 smithi060 ceph-mon[35185]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:31.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:30 smithi033 ceph-mon[31243]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:31.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:32 smithi083 ceph-mon[35242]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:32 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:32 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:32 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:32 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:11:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:32 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:33.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:32 smithi060 ceph-mon[35185]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:33.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:32 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:33.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:32 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:33.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:32 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:33.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:32 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:11:33.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:32 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:33.332 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:32 smithi033 ceph-mon[31243]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:33.333 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:33.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:33.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:33.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:11:33.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:33.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.973+0000 7f74697fa640 1 -- 172.21.15.83:0/1143634934 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f746c062630 con 0x7f74540a3270 2024-09-16T00:11:33.976 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 5 on host 'smithi083' 2024-09-16T00:11:33.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.975+0000 7f74722ed640 1 -- 172.21.15.83:0/1143634934 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74540a3270 msgr2=0x7f74540a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:11:33.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.975+0000 7f74722ed640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74540a3270 0x7f74540a5730 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f745c005d00 tx=0x7f745c005c70 comp rx=0 tx=0).stop 2024-09-16T00:11:33.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.975+0000 7f74722ed640 1 -- 172.21.15.83:0/1143634934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f746c100d90 msgr2=0x7f746c19a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:11:33.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.975+0000 7f74722ed640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f746c100d90 0x7f746c19a5a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f746000c890 tx=0x7f746000cd50 comp rx=0 tx=0).stop 2024-09-16T00:11:33.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.976+0000 7f74722ed640 1 -- 172.21.15.83:0/1143634934 shutdown_connections 2024-09-16T00:11:33.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.976+0000 7f74722ed640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74540a3270 0x7f74540a5730 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:33.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.976+0000 7f74722ed640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f746c101730 0x7f746c1a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:33.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.976+0000 7f74722ed640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f746c100d90 0x7f746c19a5a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:33.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.976+0000 7f74722ed640 1 --2- 172.21.15.83:0/1143634934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f746c100450 0x7f746c19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:33.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.976+0000 7f74722ed640 1 -- 172.21.15.83:0/1143634934 >> 172.21.15.83:0/1143634934 conn(0x7f746c0fc280 msgr2=0x7f746c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:11:33.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.976+0000 7f74722ed640 1 -- 172.21.15.83:0/1143634934 shutdown_connections 2024-09-16T00:11:33.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:11:33.976+0000 7f74722ed640 1 -- 172.21.15.83:0/1143634934 wait complete. 2024-09-16T00:11:34.674 DEBUG:teuthology.orchestra.run.smithi083:osd.5> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.5.service 2024-09-16T00:11:34.676 INFO:tasks.cephadm:Deploying osd.6 on smithi033 with /dev/vg_nvme/lv_2... 2024-09-16T00:11:34.677 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T00:11:34.945 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:11:35.066 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:34 smithi033 ceph-mon[31243]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:35.066 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:35.066 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:35.066 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:34 smithi083 ceph-mon[35242]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:35.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:35.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:35.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:34 smithi060 ceph-mon[35185]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:36.143 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:11:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:11:36.003+0000 7f62a8c0c640 -1 osd.5 0 log_to_monitors true 2024-09-16T00:11:37.035 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:36 smithi033 ceph-mon[31243]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:37.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:36 smithi033 ceph-mon[31243]: from='osd.5 [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T00:11:37.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:36 smithi033 ceph-mon[31243]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T00:11:37.145 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:36 smithi083 ceph-mon[35242]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:37.145 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:36 smithi083 ceph-mon[35242]: from='osd.5 [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T00:11:37.145 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:36 smithi083 ceph-mon[35242]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T00:11:37.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:36 smithi060 ceph-mon[35185]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:37.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:36 smithi060 ceph-mon[35185]: from='osd.5 [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T00:11:37.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:36 smithi060 ceph-mon[35185]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T00:11:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:11:37.880 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_2 2024-09-16T00:11:38.145 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:11:38.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T00:11:38.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: from='osd.5 [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:11:38.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: osdmap e33: 6 total, 5 up, 6 in 2024-09-16T00:11:38.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:38.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:11:38.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: Detected new or changed devices on smithi083 2024-09-16T00:11:38.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:38.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:38.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:11:38.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:11:38.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi083 to 8544M 2024-09-16T00:11:38.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:38.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:38.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:11:38.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:38.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T00:11:38.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: from='osd.5 [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:11:38.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: osdmap e33: 6 total, 5 up, 6 in 2024-09-16T00:11:38.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:38.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:11:38.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: Detected new or changed devices on smithi083 2024-09-16T00:11:38.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:38.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:38.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:11:38.269 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:11:38.269 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi083 to 8544M 2024-09-16T00:11:38.269 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:38.269 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:38.269 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:11:38.269 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:38.419 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:11:37.932+0000 7f62a4b8f640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T00:11:38.419 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:11:37.940+0000 7f62a09b7640 -1 osd.5 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T00:11:38.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T00:11:38.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: from='osd.5 [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:11:38.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: osdmap e33: 6 total, 5 up, 6 in 2024-09-16T00:11:38.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:38.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:11:38.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: Detected new or changed devices on smithi083 2024-09-16T00:11:38.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:38.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:38.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:11:38.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:11:38.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi083 to 8544M 2024-09-16T00:11:38.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:38.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:38.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:11:38.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:38.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.685+0000 7f6b1ce64640 1 -- 172.21.15.33:0/1830662467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b181078c0 msgr2=0x7f6b18109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:11:38.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.685+0000 7f6b1ce64640 1 --2- 172.21.15.33:0/1830662467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b181078c0 0x7f6b18109d10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6b10009ca0 tx=0x7f6b1002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:11:38.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.685+0000 7f6b1ce64640 1 -- 172.21.15.33:0/1830662467 shutdown_connections 2024-09-16T00:11:38.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.685+0000 7f6b1ce64640 1 --2- 172.21.15.33:0/1830662467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b181078c0 0x7f6b18109d10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:38.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.685+0000 7f6b1ce64640 1 --2- 172.21.15.33:0/1830662467 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18104f90 0x7f6b18107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:38.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.685+0000 7f6b1ce64640 1 --2- 172.21.15.33:0/1830662467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18102660 0x7f6b18104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:38.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.685+0000 7f6b1ce64640 1 -- 172.21.15.33:0/1830662467 >> 172.21.15.33:0/1830662467 conn(0x7f6b180fc280 msgr2=0x7f6b180fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:11:38.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.685+0000 7f6b1ce64640 1 -- 172.21.15.33:0/1830662467 shutdown_connections 2024-09-16T00:11:38.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.685+0000 7f6b1ce64640 1 -- 172.21.15.33:0/1830662467 wait complete. 2024-09-16T00:11:38.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.685+0000 7f6b1ce64640 1 Processor -- start 2024-09-16T00:11:38.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1ce64640 1 -- start start 2024-09-16T00:11:38.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1ce64640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102660 0x7f6b18197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:11:38.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1ce64640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18104f90 0x7f6b18198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:11:38.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1ce64640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b181078c0 0x7f6b1819f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:11:38.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1ce64640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1810d880 con 0x7f6b18102660 2024-09-16T00:11:38.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1ce64640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1810d700 con 0x7f6b18104f90 2024-09-16T00:11:38.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1ce64640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1810da00 con 0x7f6b181078c0 2024-09-16T00:11:38.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102660 0x7f6b18197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:11:38.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102660 0x7f6b18197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45152/0 (socket says 172.21.15.33:45152) 2024-09-16T00:11:38.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1659c640 1 -- 172.21.15.33:0/3499141802 learned_addr learned my addr 172.21.15.33:0/3499141802 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:11:38.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b16d9d640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b181078c0 0x7f6b1819f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:11:38.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b15d9b640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18104f90 0x7f6b18198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:11:38.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1659c640 1 -- 172.21.15.33:0/3499141802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b181078c0 msgr2=0x7f6b1819f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:11:38.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1659c640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b181078c0 0x7f6b1819f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:38.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1659c640 1 -- 172.21.15.33:0/3499141802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18104f90 msgr2=0x7f6b18198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:11:38.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1659c640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18104f90 0x7f6b18198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:11:38.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1659c640 1 -- 172.21.15.33:0/3499141802 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1819fb10 con 0x7f6b18102660 2024-09-16T00:11:38.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b16d9d640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b181078c0 0x7f6b1819f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:11:38.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b15d9b640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18104f90 0x7f6b18198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:11:38.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.686+0000 7f6b1659c640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102660 0x7f6b18197ee0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800c890 tx=0x7f6b0800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:11:38.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.687+0000 7f6b077fe640 1 -- 172.21.15.33:0/3499141802 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b08004220 con 0x7f6b18102660 2024-09-16T00:11:38.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.687+0000 7f6b077fe640 1 -- 172.21.15.33:0/3499141802 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b080043c0 con 0x7f6b18102660 2024-09-16T00:11:38.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.687+0000 7f6b1ce64640 1 -- 172.21.15.33:0/3499141802 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b1819fe00 con 0x7f6b18102660 2024-09-16T00:11:38.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.687+0000 7f6b077fe640 1 -- 172.21.15.33:0/3499141802 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b080049e0 con 0x7f6b18102660 2024-09-16T00:11:38.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.687+0000 7f6b1ce64640 1 -- 172.21.15.33:0/3499141802 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b181a02b0 con 0x7f6b18102660 2024-09-16T00:11:38.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.688+0000 7f6b1ce64640 1 -- 172.21.15.33:0/3499141802 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b180fdd70 con 0x7f6b18102660 2024-09-16T00:11:38.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.690+0000 7f6b077fe640 1 -- 172.21.15.33:0/3499141802 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f6b080079b0 con 0x7f6b18102660 2024-09-16T00:11:38.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.690+0000 7f6b077fe640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b000a32f0 0x7f6b000a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:11:38.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.690+0000 7f6b15d9b640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b000a32f0 0x7f6b000a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:11:38.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.690+0000 7f6b077fe640 1 -- 172.21.15.33:0/3499141802 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4253+0+0 (secure 0 0 0) 0x7f6b080ce9b0 con 0x7f6b18102660 2024-09-16T00:11:38.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.691+0000 7f6b15d9b640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b000a32f0 0x7f6b000a57b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6b18199400 tx=0x7f6b0c005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:11:38.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.694+0000 7f6b077fe640 1 -- 172.21.15.33:0/3499141802 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6b08014030 con 0x7f6b18102660 2024-09-16T00:11:38.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:11:38.878+0000 7f6b1ce64640 1 -- 172.21.15.33:0/3499141802 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6b18062630 con 0x7f6b000a32f0 2024-09-16T00:11:39.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:38 smithi060 ceph-mon[35185]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:39.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:38 smithi060 ceph-mon[35185]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-16T00:11:39.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:38 smithi060 ceph-mon[35185]: osdmap e34: 6 total, 5 up, 6 in 2024-09-16T00:11:39.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:38 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:39.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:38 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:39.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:38 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:11:39.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:38 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:11:39.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:38 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:39.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:38 smithi033 ceph-mon[31243]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:39.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:38 smithi033 ceph-mon[31243]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-16T00:11:39.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:38 smithi033 ceph-mon[31243]: osdmap e34: 6 total, 5 up, 6 in 2024-09-16T00:11:39.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:39.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:39.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:11:39.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:11:39.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:39.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:38 smithi083 ceph-mon[35242]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:39.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:38 smithi083 ceph-mon[35242]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-16T00:11:39.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:38 smithi083 ceph-mon[35242]: osdmap e34: 6 total, 5 up, 6 in 2024-09-16T00:11:39.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:38 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:39.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:38 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:39.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:38 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:11:39.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:38 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:11:39.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:38 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:40.234 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:39 smithi060 ceph-mon[35185]: purged_snaps scrub starts 2024-09-16T00:11:40.235 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:39 smithi060 ceph-mon[35185]: purged_snaps scrub ok 2024-09-16T00:11:40.235 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:39 smithi060 ceph-mon[35185]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:11:40.235 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:39 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:40.235 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:39 smithi060 ceph-mon[35185]: osd.5 [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095] boot 2024-09-16T00:11:40.235 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:39 smithi060 ceph-mon[35185]: osdmap e35: 6 total, 6 up, 6 in 2024-09-16T00:11:40.235 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:39 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:40.333 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:39 smithi033 ceph-mon[31243]: purged_snaps scrub starts 2024-09-16T00:11:40.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:39 smithi033 ceph-mon[31243]: purged_snaps scrub ok 2024-09-16T00:11:40.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:39 smithi033 ceph-mon[31243]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:11:40.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:39 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:40.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:39 smithi033 ceph-mon[31243]: osd.5 [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095] boot 2024-09-16T00:11:40.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:39 smithi033 ceph-mon[31243]: osdmap e35: 6 total, 6 up, 6 in 2024-09-16T00:11:40.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:39 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:40.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:39 smithi083 ceph-mon[35242]: purged_snaps scrub starts 2024-09-16T00:11:40.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:39 smithi083 ceph-mon[35242]: purged_snaps scrub ok 2024-09-16T00:11:40.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:39 smithi083 ceph-mon[35242]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:11:40.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:39 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:40.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:39 smithi083 ceph-mon[35242]: osd.5 [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095] boot 2024-09-16T00:11:40.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:39 smithi083 ceph-mon[35242]: osdmap e35: 6 total, 6 up, 6 in 2024-09-16T00:11:40.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:39 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T00:11:41.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:40 smithi060 ceph-mon[35185]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:41.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:40 smithi060 ceph-mon[35185]: osdmap e36: 6 total, 6 up, 6 in 2024-09-16T00:11:41.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:40 smithi033 ceph-mon[31243]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:41.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:40 smithi033 ceph-mon[31243]: osdmap e36: 6 total, 6 up, 6 in 2024-09-16T00:11:41.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:40 smithi083 ceph-mon[35242]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-16T00:11:41.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:40 smithi083 ceph-mon[35242]: osdmap e36: 6 total, 6 up, 6 in 2024-09-16T00:11:42.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:41 smithi060 ceph-mon[35185]: osdmap e37: 6 total, 6 up, 6 in 2024-09-16T00:11:42.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:41 smithi060 ceph-mon[35185]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T00:11:42.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:41 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2279660144' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48de98ea-b98b-4f4f-b771-38a7d816e784"}]: dispatch 2024-09-16T00:11:42.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:41 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2279660144' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48de98ea-b98b-4f4f-b771-38a7d816e784"}]': finished 2024-09-16T00:11:42.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:41 smithi060 ceph-mon[35185]: osdmap e39: 7 total, 6 up, 7 in 2024-09-16T00:11:42.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:41 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:11:42.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:41 smithi033 ceph-mon[31243]: osdmap e37: 6 total, 6 up, 6 in 2024-09-16T00:11:42.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:41 smithi033 ceph-mon[31243]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T00:11:42.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:41 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2279660144' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48de98ea-b98b-4f4f-b771-38a7d816e784"}]: dispatch 2024-09-16T00:11:42.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:41 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2279660144' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48de98ea-b98b-4f4f-b771-38a7d816e784"}]': finished 2024-09-16T00:11:42.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:41 smithi033 ceph-mon[31243]: osdmap e39: 7 total, 6 up, 7 in 2024-09-16T00:11:42.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:41 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:11:42.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:41 smithi083 ceph-mon[35242]: osdmap e37: 6 total, 6 up, 6 in 2024-09-16T00:11:42.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:41 smithi083 ceph-mon[35242]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T00:11:42.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:41 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2279660144' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48de98ea-b98b-4f4f-b771-38a7d816e784"}]: dispatch 2024-09-16T00:11:42.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:41 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2279660144' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48de98ea-b98b-4f4f-b771-38a7d816e784"}]': finished 2024-09-16T00:11:42.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:41 smithi083 ceph-mon[35242]: osdmap e39: 7 total, 6 up, 7 in 2024-09-16T00:11:42.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:41 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:11:43.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:42 smithi060 ceph-mon[35185]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:11:43.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:42 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1108997838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:11:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:42 smithi033 ceph-mon[31243]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:11:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:42 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1108997838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:11:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:42 smithi083 ceph-mon[35242]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:11:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:42 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1108997838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:11:45.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:44 smithi060 ceph-mon[35185]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-16T00:11:45.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:44 smithi033 ceph-mon[31243]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-16T00:11:45.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:44 smithi083 ceph-mon[35242]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-16T00:11:47.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:46 smithi060 ceph-mon[35185]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-16T00:11:47.231 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:46 smithi033 ceph-mon[31243]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-16T00:11:47.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:46 smithi083 ceph-mon[35242]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-16T00:11:49.058 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:48 smithi033 ceph-mon[31243]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-16T00:11:49.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:48 smithi060 ceph-mon[35185]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-16T00:11:49.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:48 smithi083 ceph-mon[35242]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-16T00:11:51.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:50 smithi033 ceph-mon[31243]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-16T00:11:51.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:50 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-16T00:11:51.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:50 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:51.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:50 smithi033 ceph-mon[31243]: Deploying daemon osd.6 on smithi033 2024-09-16T00:11:51.221 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:50 smithi083 ceph-mon[35242]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-16T00:11:51.221 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:50 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-16T00:11:51.221 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:50 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:51.221 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:50 smithi083 ceph-mon[35242]: Deploying daemon osd.6 on smithi033 2024-09-16T00:11:51.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:50 smithi060 ceph-mon[35185]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-16T00:11:51.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:50 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-16T00:11:51.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:50 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:11:51.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:50 smithi060 ceph-mon[35185]: Deploying daemon osd.6 on smithi033 2024-09-16T00:11:53.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:52 smithi033 ceph-mon[31243]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-16T00:11:53.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:52 smithi060 ceph-mon[35185]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-16T00:11:53.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:52 smithi083 ceph-mon[35242]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-16T00:11:55.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:54 smithi033 ceph-mon[31243]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:11:55.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:54 smithi060 ceph-mon[35185]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:11:55.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:54 smithi083 ceph-mon[35242]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:11:57.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:56 smithi033 ceph-mon[31243]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:11:57.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:57.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:57.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:57.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:56 smithi060 ceph-mon[35185]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:11:57.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:57.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:57.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:56 smithi083 ceph-mon[35242]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:11:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:11:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:11:59.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:11:58 smithi060 ceph-mon[35185]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:11:59.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:11:58 smithi033 ceph-mon[31243]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:11:59.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:11:58 smithi083 ceph-mon[35242]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:12:00.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.289+0000 7f6b077fe640 1 -- 172.21.15.33:0/3499141802 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6b18062630 con 0x7f6b000a32f0 2024-09-16T00:12:00.292 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 6 on host 'smithi033' 2024-09-16T00:12:00.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.292+0000 7f6b1ce64640 1 -- 172.21.15.33:0/3499141802 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b000a32f0 msgr2=0x7f6b000a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.292+0000 7f6b1ce64640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b000a32f0 0x7f6b000a57b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6b18199400 tx=0x7f6b0c005e50 comp rx=0 tx=0).stop 2024-09-16T00:12:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.292+0000 7f6b1ce64640 1 -- 172.21.15.33:0/3499141802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102660 msgr2=0x7f6b18197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.292+0000 7f6b1ce64640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102660 0x7f6b18197ee0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800c890 tx=0x7f6b0800cd50 comp rx=0 tx=0).stop 2024-09-16T00:12:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.292+0000 7f6b1ce64640 1 -- 172.21.15.33:0/3499141802 shutdown_connections 2024-09-16T00:12:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.292+0000 7f6b1ce64640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b000a32f0 0x7f6b000a57b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.292+0000 7f6b1ce64640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b181078c0 0x7f6b1819f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.292+0000 7f6b1ce64640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18104f90 0x7f6b18198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.292+0000 7f6b1ce64640 1 --2- 172.21.15.33:0/3499141802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102660 0x7f6b18197ee0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.292+0000 7f6b1ce64640 1 -- 172.21.15.33:0/3499141802 >> 172.21.15.33:0/3499141802 conn(0x7f6b180fc280 msgr2=0x7f6b18102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:12:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.292+0000 7f6b1ce64640 1 -- 172.21.15.33:0/3499141802 shutdown_connections 2024-09-16T00:12:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:00.292+0000 7f6b1ce64640 1 -- 172.21.15.33:0/3499141802 wait complete. 2024-09-16T00:12:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:00.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:00.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:00.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:00.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:00.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:01.061 DEBUG:teuthology.orchestra.run.smithi033:osd.6> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.6.service 2024-09-16T00:12:01.064 INFO:tasks.cephadm:Deploying osd.7 on smithi060 with /dev/vg_nvme/lv_2... 2024-09-16T00:12:01.064 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T00:12:01.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:01 smithi033 ceph-mon[31243]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:12:01.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:01 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:01.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:01 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:01.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:01 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:01.327 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T00:12:01.448 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:01 smithi060 ceph-mon[35185]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:12:01.448 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:01 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:01.448 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:01 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:01.448 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:01 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:01.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:01 smithi083 ceph-mon[35242]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:12:01.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:01 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:01.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:01 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:01.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:01 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:02.174 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:12:02 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:12:02.073+0000 7ffb341df640 -1 osd.6 0 log_to_monitors true 2024-09-16T00:12:02.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:02 smithi083 ceph-mon[35242]: from='osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-16T00:12:02.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:02 smithi060 ceph-mon[35185]: from='osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-16T00:12:02.498 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:02 smithi033 ceph-mon[31243]: from='osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-16T00:12:03.347 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:03 smithi060 ceph-mon[35185]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:12:03.347 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:03 smithi060 ceph-mon[35185]: from='osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T00:12:03.347 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:03 smithi060 ceph-mon[35185]: osdmap e40: 7 total, 6 up, 7 in 2024-09-16T00:12:03.347 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:03 smithi060 ceph-mon[35185]: from='osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-16T00:12:03.348 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:12:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:03 smithi033 ceph-mon[31243]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:12:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:03 smithi033 ceph-mon[31243]: from='osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T00:12:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:03 smithi033 ceph-mon[31243]: osdmap e40: 7 total, 6 up, 7 in 2024-09-16T00:12:03.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:03 smithi033 ceph-mon[31243]: from='osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-16T00:12:03.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:12:03.581 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:12:03 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:12:03.185+0000 7ffb30162640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T00:12:03.581 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:12:03 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:12:03.195+0000 7ffb2bf8a640 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T00:12:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:03 smithi083 ceph-mon[35242]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:12:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:03 smithi083 ceph-mon[35242]: from='osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T00:12:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:03 smithi083 ceph-mon[35242]: osdmap e40: 7 total, 6 up, 7 in 2024-09-16T00:12:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:03 smithi083 ceph-mon[35242]: from='osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-16T00:12:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:12:04.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: from='osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-16T00:12:04.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: osdmap e41: 7 total, 6 up, 7 in 2024-09-16T00:12:04.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:12:04.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:12:04.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:04.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:04.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:04.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:04.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:04.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:04.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:04.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:04.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:04 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:04.270 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:12:04.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi060:vg_nvme/lv_2 2024-09-16T00:12:04.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: from='osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-16T00:12:04.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: osdmap e41: 7 total, 6 up, 7 in 2024-09-16T00:12:04.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:12:04.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:12:04.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:04.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:04.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:04.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:04.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:04.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:04.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:04.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:04.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:04 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:04.544 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T00:12:04.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: from='osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-16T00:12:04.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: osdmap e41: 7 total, 6 up, 7 in 2024-09-16T00:12:04.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:12:04.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:12:04.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:04.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:04.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:04.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:04.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:04.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:04.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:04.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:04.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:04 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:05.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.051+0000 7f285f277640 1 -- 172.21.15.60:0/3572357570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2858102690 msgr2=0x7f2858108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:05.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.051+0000 7f285f277640 1 --2- 172.21.15.60:0/3572357570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2858102690 0x7f2858108f20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2850009900 tx=0x7f285002a410 comp rx=0 tx=0).stop 2024-09-16T00:12:05.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.051+0000 7f285f277640 1 -- 172.21.15.60:0/3572357570 shutdown_connections 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.051+0000 7f285f277640 1 --2- 172.21.15.60:0/3572357570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2858102690 0x7f2858108f20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.051+0000 7f285f277640 1 --2- 172.21.15.60:0/3572357570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2858101cd0 0x7f2858102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.051+0000 7f285f277640 1 --2- 172.21.15.60:0/3572357570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2858100ad0 0x7f2858100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.051+0000 7f285f277640 1 -- 172.21.15.60:0/3572357570 >> 172.21.15.60:0/3572357570 conn(0x7f28580fc280 msgr2=0x7f28580fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.051+0000 7f285f277640 1 -- 172.21.15.60:0/3572357570 shutdown_connections 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.051+0000 7f285f277640 1 -- 172.21.15.60:0/3572357570 wait complete. 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.051+0000 7f285f277640 1 Processor -- start 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.051+0000 7f285f277640 1 -- start start 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285f277640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2858100ad0 0x7f2858196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285f277640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2858101cd0 0x7f28581972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285f277640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2858102690 0x7f285819e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285f277640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f285810c750 con 0x7f2858101cd0 2024-09-16T00:12:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285f277640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f285810c5d0 con 0x7f2858100ad0 2024-09-16T00:12:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285f277640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f285810c8d0 con 0x7f2858102690 2024-09-16T00:12:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285d013640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2858100ad0 0x7f2858196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:12:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285d013640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2858100ad0 0x7f2858196d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34674/0 (socket says 172.21.15.60:34674) 2024-09-16T00:12:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285d013640 1 -- 172.21.15.60:0/1343984097 learned_addr learned my addr 172.21.15.60:0/1343984097 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-09-16T00:12:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285d814640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2858102690 0x7f285819e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:12:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285c812640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2858101cd0 0x7f28581972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:12:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285d013640 1 -- 172.21.15.60:0/1343984097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2858102690 msgr2=0x7f285819e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285d013640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2858102690 0x7f285819e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285d013640 1 -- 172.21.15.60:0/1343984097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2858101cd0 msgr2=0x7f28581972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285d013640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2858101cd0 0x7f28581972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285d013640 1 -- 172.21.15.60:0/1343984097 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285819e9b0 con 0x7f2858100ad0 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285d814640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2858102690 0x7f285819e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f285d013640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2858100ad0 0x7f2858196d90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f285400ebc0 tx=0x7f285400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.052+0000 7f284e7fc640 1 -- 172.21.15.60:0/1343984097 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854003b70 con 0x7f2858100ad0 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.053+0000 7f284e7fc640 1 -- 172.21.15.60:0/1343984097 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2854003d10 con 0x7f2858100ad0 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.053+0000 7f285f277640 1 -- 172.21.15.60:0/1343984097 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f285819eca0 con 0x7f2858100ad0 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.053+0000 7f284e7fc640 1 -- 172.21.15.60:0/1343984097 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854019980 con 0x7f2858100ad0 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.053+0000 7f285f277640 1 -- 172.21.15.60:0/1343984097 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f285819f180 con 0x7f2858100ad0 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.053+0000 7f285f277640 1 -- 172.21.15.60:0/1343984097 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2858107e80 con 0x7f2858100ad0 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.055+0000 7f284e7fc640 1 -- 172.21.15.60:0/1343984097 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2854019b20 con 0x7f2858100ad0 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.055+0000 7f284e7fc640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f28440a32c0 0x7f28440a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.056+0000 7f284e7fc640 1 -- 172.21.15.60:0/1343984097 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4965+0+0 (secure 0 0 0) 0x7f28540cf440 con 0x7f2858100ad0 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.056+0000 7f285c812640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f28440a32c0 0x7f28440a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:12:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.056+0000 7f285c812640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f28440a32c0 0x7f28440a5780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f28480058c0 tx=0x7f284800a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:12:05.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.060+0000 7f284e7fc640 1 -- 172.21.15.60:0/1343984097 <== mon.2 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f285409ce70 con 0x7f2858100ad0 2024-09-16T00:12:05.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:05 smithi060 ceph-mon[35185]: purged_snaps scrub starts 2024-09-16T00:12:05.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:05 smithi060 ceph-mon[35185]: purged_snaps scrub ok 2024-09-16T00:12:05.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:05 smithi060 ceph-mon[35185]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:12:05.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:05 smithi060 ceph-mon[35185]: Detected new or changed devices on smithi033 2024-09-16T00:12:05.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:05 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi033 to 5697M 2024-09-16T00:12:05.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:05 smithi060 ceph-mon[35185]: osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155] boot 2024-09-16T00:12:05.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:05 smithi060 ceph-mon[35185]: osdmap e42: 7 total, 7 up, 7 in 2024-09-16T00:12:05.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:12:05.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:05.247+0000 7f285f277640 1 -- 172.21.15.60:0/1343984097 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f2858062630 con 0x7f28440a32c0 2024-09-16T00:12:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:05 smithi033 ceph-mon[31243]: purged_snaps scrub starts 2024-09-16T00:12:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:05 smithi033 ceph-mon[31243]: purged_snaps scrub ok 2024-09-16T00:12:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:05 smithi033 ceph-mon[31243]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:12:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:05 smithi033 ceph-mon[31243]: Detected new or changed devices on smithi033 2024-09-16T00:12:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:05 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi033 to 5697M 2024-09-16T00:12:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:05 smithi033 ceph-mon[31243]: osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155] boot 2024-09-16T00:12:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:05 smithi033 ceph-mon[31243]: osdmap e42: 7 total, 7 up, 7 in 2024-09-16T00:12:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:12:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:05 smithi083 ceph-mon[35242]: purged_snaps scrub starts 2024-09-16T00:12:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:05 smithi083 ceph-mon[35242]: purged_snaps scrub ok 2024-09-16T00:12:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:05 smithi083 ceph-mon[35242]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-09-16T00:12:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:05 smithi083 ceph-mon[35242]: Detected new or changed devices on smithi033 2024-09-16T00:12:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:05 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi033 to 5697M 2024-09-16T00:12:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:05 smithi083 ceph-mon[35242]: osd.6 [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155] boot 2024-09-16T00:12:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:05 smithi083 ceph-mon[35242]: osdmap e42: 7 total, 7 up, 7 in 2024-09-16T00:12:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T00:12:06.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:06 smithi060 ceph-mon[35185]: osdmap e43: 7 total, 7 up, 7 in 2024-09-16T00:12:06.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:06 smithi060 ceph-mon[35185]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:12:06.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:06 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:12:06.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:06 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:12:06.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:06 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:06.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:06 smithi033 ceph-mon[31243]: osdmap e43: 7 total, 7 up, 7 in 2024-09-16T00:12:06.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:06 smithi033 ceph-mon[31243]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:12:06.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:06 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:12:06.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:06 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:12:06.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:06 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:06.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:06 smithi083 ceph-mon[35242]: osdmap e43: 7 total, 7 up, 7 in 2024-09-16T00:12:06.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:06 smithi083 ceph-mon[35242]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:12:06.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:06 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:12:06.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:06 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:12:06.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:06 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:07.269 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:07 smithi060 ceph-mon[35185]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:07.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:07 smithi033 ceph-mon[31243]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:07.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:07 smithi083 ceph-mon[35242]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:08.333 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:08 smithi060 ceph-mon[35185]: from='client.? 172.21.15.60:0/3882694854' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44266591-c35b-48fe-9a90-e1597d7e311a"}]: dispatch 2024-09-16T00:12:08.333 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:08 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44266591-c35b-48fe-9a90-e1597d7e311a"}]: dispatch 2024-09-16T00:12:08.333 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:08 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44266591-c35b-48fe-9a90-e1597d7e311a"}]': finished 2024-09-16T00:12:08.333 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:08 smithi060 ceph-mon[35185]: osdmap e44: 8 total, 7 up, 8 in 2024-09-16T00:12:08.333 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:08 smithi033 ceph-mon[31243]: from='client.? 172.21.15.60:0/3882694854' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44266591-c35b-48fe-9a90-e1597d7e311a"}]: dispatch 2024-09-16T00:12:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:08 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44266591-c35b-48fe-9a90-e1597d7e311a"}]: dispatch 2024-09-16T00:12:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:08 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44266591-c35b-48fe-9a90-e1597d7e311a"}]': finished 2024-09-16T00:12:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:08 smithi033 ceph-mon[31243]: osdmap e44: 8 total, 7 up, 8 in 2024-09-16T00:12:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:08.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:08 smithi083 ceph-mon[35242]: from='client.? 172.21.15.60:0/3882694854' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44266591-c35b-48fe-9a90-e1597d7e311a"}]: dispatch 2024-09-16T00:12:09.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:08 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44266591-c35b-48fe-9a90-e1597d7e311a"}]: dispatch 2024-09-16T00:12:09.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:08 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44266591-c35b-48fe-9a90-e1597d7e311a"}]': finished 2024-09-16T00:12:09.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:08 smithi083 ceph-mon[35242]: osdmap e44: 8 total, 7 up, 8 in 2024-09-16T00:12:09.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:09 smithi060 ceph-mon[35185]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:09 smithi060 ceph-mon[35185]: from='client.? 172.21.15.60:0/274205530' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:12:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:09 smithi033 ceph-mon[31243]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:09 smithi033 ceph-mon[31243]: from='client.? 172.21.15.60:0/274205530' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:12:09.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:09 smithi083 ceph-mon[35242]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:09.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:09 smithi083 ceph-mon[35242]: from='client.? 172.21.15.60:0/274205530' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:12:11.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:11 smithi060 ceph-mon[35185]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:11.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:11 smithi033 ceph-mon[31243]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:11.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:11 smithi083 ceph-mon[35242]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:13.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:13 smithi060 ceph-mon[35185]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:13.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:13 smithi033 ceph-mon[31243]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:13.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:13 smithi083 ceph-mon[35242]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:15.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:15 smithi060 ceph-mon[35185]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:15 smithi033 ceph-mon[31243]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:15 smithi083 ceph-mon[35242]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:17.335 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:17 smithi060 ceph-mon[35185]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:17.336 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:17 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-16T00:12:17.336 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:17 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:17.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:17 smithi033 ceph-mon[31243]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:17.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-16T00:12:17.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:17 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:17.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:17 smithi083 ceph-mon[35242]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:17.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:17 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-16T00:12:17.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:17 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:18.576 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:18 smithi060 ceph-mon[35185]: Deploying daemon osd.7 on smithi060 2024-09-16T00:12:18.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:18 smithi033 ceph-mon[31243]: Deploying daemon osd.7 on smithi060 2024-09-16T00:12:18.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:18 smithi083 ceph-mon[35242]: Deploying daemon osd.7 on smithi060 2024-09-16T00:12:19.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:19 smithi033 ceph-mon[31243]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:19.614 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:19 smithi060 ceph-mon[35185]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:19.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:19 smithi083 ceph-mon[35242]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:21.537 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:21 smithi060 ceph-mon[35185]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:21.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:21 smithi033 ceph-mon[31243]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:21.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:21 smithi083 ceph-mon[35242]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:23.531 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:23 smithi060 ceph-mon[35185]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:23.531 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:23.532 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:23.532 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:23.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:23 smithi033 ceph-mon[31243]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:23.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:23.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:23.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:23.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:23 smithi083 ceph-mon[35242]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:23.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:23.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:23.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:25.516 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:25 smithi060 ceph-mon[35185]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:25.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:25 smithi033 ceph-mon[31243]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:25.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:25 smithi083 ceph-mon[35242]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:26.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.933+0000 7f284e7fc640 1 -- 172.21.15.60:0/1343984097 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2858062630 con 0x7f28440a32c0 2024-09-16T00:12:26.935 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 7 on host 'smithi060' 2024-09-16T00:12:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.936+0000 7f285f277640 1 -- 172.21.15.60:0/1343984097 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f28440a32c0 msgr2=0x7f28440a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.936+0000 7f285f277640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f28440a32c0 0x7f28440a5780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f28480058c0 tx=0x7f284800a250 comp rx=0 tx=0).stop 2024-09-16T00:12:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.936+0000 7f285f277640 1 -- 172.21.15.60:0/1343984097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2858100ad0 msgr2=0x7f2858196d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.936+0000 7f285f277640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2858100ad0 0x7f2858196d90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f285400ebc0 tx=0x7f285400c6a0 comp rx=0 tx=0).stop 2024-09-16T00:12:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.936+0000 7f285f277640 1 -- 172.21.15.60:0/1343984097 shutdown_connections 2024-09-16T00:12:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.936+0000 7f285f277640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f28440a32c0 0x7f28440a5780 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.936+0000 7f285f277640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2858102690 0x7f285819e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.936+0000 7f285f277640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2858101cd0 0x7f28581972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.936+0000 7f285f277640 1 --2- 172.21.15.60:0/1343984097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2858100ad0 0x7f2858196d90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.936+0000 7f285f277640 1 -- 172.21.15.60:0/1343984097 >> 172.21.15.60:0/1343984097 conn(0x7f28580fc280 msgr2=0x7f2858105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:12:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.936+0000 7f285f277640 1 -- 172.21.15.60:0/1343984097 shutdown_connections 2024-09-16T00:12:26.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:12:26.936+0000 7f285f277640 1 -- 172.21.15.60:0/1343984097 wait complete. 2024-09-16T00:12:27.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:26 smithi083 ceph-mon[35242]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:27.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:26 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:27.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:26 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:27.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:26 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:27.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:26 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:27.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:26 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:27.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:26 smithi060 ceph-mon[35185]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:27.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:26 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:27.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:26 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:27.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:26 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:27.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:26 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:27.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:26 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:27.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:26 smithi033 ceph-mon[31243]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:27.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:26 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:27.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:26 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:27.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:26 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:27.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:26 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:27.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:26 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:27.742 DEBUG:teuthology.orchestra.run.smithi060:osd.7> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.7.service 2024-09-16T00:12:27.744 INFO:tasks.cephadm:Deploying osd.8 on smithi083 with /dev/vg_nvme/lv_2... 2024-09-16T00:12:27.744 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T00:12:27.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:27.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:27.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:27.998 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:12:28.119 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:28.119 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:28.119 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:28.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:28.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:28.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:29.126 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:28 smithi060 ceph-mon[35185]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:29.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:28 smithi083 ceph-mon[35242]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:29.231 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:28 smithi033 ceph-mon[31243]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:29.417 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:12:29 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:12:29.124+0000 7f3b370cd640 -1 osd.7 0 log_to_monitors true 2024-09-16T00:12:30.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:29 smithi083 ceph-mon[35242]: from='osd.7 [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T00:12:30.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:29 smithi083 ceph-mon[35242]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T00:12:30.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:29 smithi060 ceph-mon[35185]: from='osd.7 [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T00:12:30.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:29 smithi060 ceph-mon[35185]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T00:12:30.251 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:29 smithi033 ceph-mon[31243]: from='osd.7 [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T00:12:30.251 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:29 smithi033 ceph-mon[31243]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T00:12:30.596 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:12:30.625 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_2 2024-09-16T00:12:30.889 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:12:31.166 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: osdmap e45: 8 total, 7 up, 8 in 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='osd.7 [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:31.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:31.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T00:12:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: osdmap e45: 8 total, 7 up, 8 in 2024-09-16T00:12:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='osd.7 [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:12:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:12:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:31.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:31.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:31.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:31.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:31.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:31.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:31.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:31.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T00:12:31.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: osdmap e45: 8 total, 7 up, 8 in 2024-09-16T00:12:31.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='osd.7 [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:12:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:12:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:31.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:31.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.338+0000 7f98a5a0f640 1 -- 172.21.15.83:0/2084979910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98a0101c00 msgr2=0x7f98a0102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:31.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.338+0000 7f98a5a0f640 1 --2- 172.21.15.83:0/2084979910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98a0101c00 0x7f98a0102000 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9890009860 tx=0x7f989002a340 comp rx=0 tx=0).stop 2024-09-16T00:12:31.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.338+0000 7f98a5a0f640 1 -- 172.21.15.83:0/2084979910 shutdown_connections 2024-09-16T00:12:31.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.338+0000 7f98a5a0f640 1 --2- 172.21.15.83:0/2084979910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98a01037c0 0x7f98a010a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:31.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.338+0000 7f98a5a0f640 1 --2- 172.21.15.83:0/2084979910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0102e00 0x7f98a0103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:31.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.338+0000 7f98a5a0f640 1 --2- 172.21.15.83:0/2084979910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98a0101c00 0x7f98a0102000 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:31.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.338+0000 7f98a5a0f640 1 -- 172.21.15.83:0/2084979910 >> 172.21.15.83:0/2084979910 conn(0x7f98a00fd390 msgr2=0x7f98a00ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:12:31.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.338+0000 7f98a5a0f640 1 -- 172.21.15.83:0/2084979910 shutdown_connections 2024-09-16T00:12:31.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.338+0000 7f98a5a0f640 1 -- 172.21.15.83:0/2084979910 wait complete. 2024-09-16T00:12:31.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.339+0000 7f98a5a0f640 1 Processor -- start 2024-09-16T00:12:31.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.339+0000 7f98a5a0f640 1 -- start start 2024-09-16T00:12:31.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.339+0000 7f98a5a0f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0101c00 0x7f98a0197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:12:31.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.339+0000 7f98a5a0f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98a0102e00 0x7f98a0198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:12:31.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.339+0000 7f98a5a0f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98a01037c0 0x7f98a019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:12:31.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.339+0000 7f98a5a0f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f98a010d860 con 0x7f98a01037c0 2024-09-16T00:12:31.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.339+0000 7f98a5a0f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f98a010d6e0 con 0x7f98a0102e00 2024-09-16T00:12:31.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.339+0000 7f98a5a0f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f98a010d9e0 con 0x7f98a0101c00 2024-09-16T00:12:31.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f989effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0101c00 0x7f98a0197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:12:31.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f989e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98a0102e00 0x7f98a0198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:12:31.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f989f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98a01037c0 0x7f98a019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:12:31.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f989e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98a0102e00 0x7f98a0198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.83:52790/0 (socket says 172.21.15.83:52790) 2024-09-16T00:12:31.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f989e7fc640 1 -- 172.21.15.83:0/235510582 learned_addr learned my addr 172.21.15.83:0/235510582 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-16T00:12:31.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f989e7fc640 1 -- 172.21.15.83:0/235510582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0101c00 msgr2=0x7f98a0197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:31.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f989e7fc640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0101c00 0x7f98a0197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:31.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f989e7fc640 1 -- 172.21.15.83:0/235510582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98a01037c0 msgr2=0x7f98a019f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f989e7fc640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98a01037c0 0x7f98a019f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f989e7fc640 1 -- 172.21.15.83:0/235510582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a019fb10 con 0x7f98a0102e00 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f989f7fe640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98a01037c0 0x7f98a019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f989e7fc640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98a0102e00 0x7f98a0198420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f989400e9c0 tx=0x7f989400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f98a4a0d640 1 -- 172.21.15.83:0/235510582 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989400cde0 con 0x7f98a0102e00 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f98a5a0f640 1 -- 172.21.15.83:0/235510582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98a019fe00 con 0x7f98a0102e00 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f98a4a0d640 1 -- 172.21.15.83:0/235510582 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98940035f0 con 0x7f98a0102e00 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f98a4a0d640 1 -- 172.21.15.83:0/235510582 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9894010650 con 0x7f98a0102e00 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.340+0000 7f98a5a0f640 1 -- 172.21.15.83:0/235510582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98a01a02b0 con 0x7f98a0102e00 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.341+0000 7f98a5a0f640 1 -- 172.21.15.83:0/235510582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98a0108fa0 con 0x7f98a0102e00 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.343+0000 7f98a4a0d640 1 -- 172.21.15.83:0/235510582 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f98940037a0 con 0x7f98a0102e00 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.343+0000 7f98a4a0d640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f98880a3270 0x7f98880a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:12:31.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.343+0000 7f98a4a0d640 1 -- 172.21.15.83:0/235510582 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5150+0+0 (secure 0 0 0) 0x7f98940cefc0 con 0x7f98a0102e00 2024-09-16T00:12:31.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.343+0000 7f989effd640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f98880a3270 0x7f98880a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:12:31.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.344+0000 7f989effd640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f98880a3270 0x7f98880a5730 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9890009830 tx=0x7f9890030570 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:12:31.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.346+0000 7f98a4a0d640 1 -- 172.21.15.83:0/235510582 <== mon.2 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f989409c9d0 con 0x7f98a0102e00 2024-09-16T00:12:31.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:31.528+0000 7f98a5a0f640 1 -- 172.21.15.83:0/235510582 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f98a0062630 con 0x7f98880a3270 2024-09-16T00:12:31.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:12:31.332+0000 7f3b33050640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T00:12:31.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:12:31.342+0000 7f3b2ae70640 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T00:12:32.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: Detected new or changed devices on smithi060 2024-09-16T00:12:32.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi060 to 5697M 2024-09-16T00:12:32.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-09-16T00:12:32.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: osdmap e46: 8 total, 7 up, 8 in 2024-09-16T00:12:32.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: osdmap e47: 8 total, 7 up, 8 in 2024-09-16T00:12:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: from='osd.7 ' entity='osd.7' 2024-09-16T00:12:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:12:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:12:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:12:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:32.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:32.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: Detected new or changed devices on smithi060 2024-09-16T00:12:32.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi060 to 5697M 2024-09-16T00:12:32.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-09-16T00:12:32.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: osdmap e46: 8 total, 7 up, 8 in 2024-09-16T00:12:32.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:32.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:32.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: osdmap e47: 8 total, 7 up, 8 in 2024-09-16T00:12:32.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:32.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: from='osd.7 ' entity='osd.7' 2024-09-16T00:12:32.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:12:32.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:12:32.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:12:32.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:32.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:32.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: Detected new or changed devices on smithi060 2024-09-16T00:12:32.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi060 to 5697M 2024-09-16T00:12:32.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-09-16T00:12:32.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: osdmap e46: 8 total, 7 up, 8 in 2024-09-16T00:12:32.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:32.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:32.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: osdmap e47: 8 total, 7 up, 8 in 2024-09-16T00:12:32.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:32.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: from='osd.7 ' entity='osd.7' 2024-09-16T00:12:32.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:12:32.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:12:32.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:12:32.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:32.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:33.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:33 smithi083 ceph-mon[35242]: purged_snaps scrub starts 2024-09-16T00:12:33.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:33 smithi083 ceph-mon[35242]: purged_snaps scrub ok 2024-09-16T00:12:33.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:33 smithi083 ceph-mon[35242]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:33.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:33 smithi083 ceph-mon[35242]: osd.7 [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559] boot 2024-09-16T00:12:33.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:33 smithi083 ceph-mon[35242]: osdmap e48: 8 total, 8 up, 8 in 2024-09-16T00:12:33.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:33.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:33 smithi033 ceph-mon[31243]: purged_snaps scrub starts 2024-09-16T00:12:33.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:33 smithi033 ceph-mon[31243]: purged_snaps scrub ok 2024-09-16T00:12:33.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:33 smithi033 ceph-mon[31243]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:33.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:33 smithi033 ceph-mon[31243]: osd.7 [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559] boot 2024-09-16T00:12:33.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:33 smithi033 ceph-mon[31243]: osdmap e48: 8 total, 8 up, 8 in 2024-09-16T00:12:33.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:33.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:33 smithi060 ceph-mon[35185]: purged_snaps scrub starts 2024-09-16T00:12:33.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:33 smithi060 ceph-mon[35185]: purged_snaps scrub ok 2024-09-16T00:12:33.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:33 smithi060 ceph-mon[35185]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-09-16T00:12:33.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:33 smithi060 ceph-mon[35185]: osd.7 [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559] boot 2024-09-16T00:12:33.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:33 smithi060 ceph-mon[35185]: osdmap e48: 8 total, 8 up, 8 in 2024-09-16T00:12:33.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T00:12:34.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:34 smithi033 ceph-mon[31243]: osdmap e49: 8 total, 8 up, 8 in 2024-09-16T00:12:34.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:34 smithi033 ceph-mon[31243]: from='client.? 172.21.15.83:0/2879253812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fadf88b-ff77-46ed-8c31-34b974af9dc0"}]: dispatch 2024-09-16T00:12:34.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:34 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fadf88b-ff77-46ed-8c31-34b974af9dc0"}]: dispatch 2024-09-16T00:12:34.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:34 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fadf88b-ff77-46ed-8c31-34b974af9dc0"}]': finished 2024-09-16T00:12:34.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:34 smithi033 ceph-mon[31243]: osdmap e50: 9 total, 8 up, 9 in 2024-09-16T00:12:34.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:34.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:34 smithi083 ceph-mon[35242]: osdmap e49: 8 total, 8 up, 8 in 2024-09-16T00:12:34.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:34 smithi083 ceph-mon[35242]: from='client.? 172.21.15.83:0/2879253812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fadf88b-ff77-46ed-8c31-34b974af9dc0"}]: dispatch 2024-09-16T00:12:34.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:34 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fadf88b-ff77-46ed-8c31-34b974af9dc0"}]: dispatch 2024-09-16T00:12:34.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:34 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fadf88b-ff77-46ed-8c31-34b974af9dc0"}]': finished 2024-09-16T00:12:34.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:34 smithi083 ceph-mon[35242]: osdmap e50: 9 total, 8 up, 9 in 2024-09-16T00:12:34.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:34.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:34 smithi060 ceph-mon[35185]: osdmap e49: 8 total, 8 up, 8 in 2024-09-16T00:12:34.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:34 smithi060 ceph-mon[35185]: from='client.? 172.21.15.83:0/2879253812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fadf88b-ff77-46ed-8c31-34b974af9dc0"}]: dispatch 2024-09-16T00:12:34.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:34 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fadf88b-ff77-46ed-8c31-34b974af9dc0"}]: dispatch 2024-09-16T00:12:34.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:34 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fadf88b-ff77-46ed-8c31-34b974af9dc0"}]': finished 2024-09-16T00:12:34.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:34 smithi060 ceph-mon[35185]: osdmap e50: 9 total, 8 up, 9 in 2024-09-16T00:12:34.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:35.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:35 smithi033 ceph-mon[31243]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:35.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:35 smithi033 ceph-mon[31243]: from='client.? 172.21.15.83:0/1616533935' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:12:35.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:35 smithi033 ceph-mon[31243]: osdmap e51: 9 total, 8 up, 9 in 2024-09-16T00:12:35.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:35.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:35 smithi083 ceph-mon[35242]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:35.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:35 smithi083 ceph-mon[35242]: from='client.? 172.21.15.83:0/1616533935' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:12:35.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:35 smithi083 ceph-mon[35242]: osdmap e51: 9 total, 8 up, 9 in 2024-09-16T00:12:35.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:35.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:35 smithi060 ceph-mon[35185]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:35.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:35 smithi060 ceph-mon[35185]: from='client.? 172.21.15.83:0/1616533935' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:12:35.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:35 smithi060 ceph-mon[35185]: osdmap e51: 9 total, 8 up, 9 in 2024-09-16T00:12:35.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:36 smithi083 ceph-mon[35242]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:36 smithi083 ceph-mon[35242]: osdmap e52: 9 total, 8 up, 9 in 2024-09-16T00:12:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:36 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:37.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:36 smithi060 ceph-mon[35185]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:37.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:36 smithi060 ceph-mon[35185]: osdmap e52: 9 total, 8 up, 9 in 2024-09-16T00:12:37.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:36 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:37.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:36 smithi033 ceph-mon[31243]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:37.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:36 smithi033 ceph-mon[31243]: osdmap e52: 9 total, 8 up, 9 in 2024-09-16T00:12:37.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:39.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:38 smithi083 ceph-mon[35242]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:39.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:38 smithi060 ceph-mon[35185]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:39.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:38 smithi033 ceph-mon[31243]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:41.137 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:40 smithi083 ceph-mon[35242]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:41.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:40 smithi060 ceph-mon[35185]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:41.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:40 smithi033 ceph-mon[31243]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:43.090 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:42 smithi083 ceph-mon[35242]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:43.090 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:42 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-16T00:12:43.090 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:42 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:43.090 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:42 smithi083 ceph-mon[35242]: Deploying daemon osd.8 on smithi083 2024-09-16T00:12:43.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:42 smithi060 ceph-mon[35185]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:43.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:42 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-16T00:12:43.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:42 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:43.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:42 smithi060 ceph-mon[35185]: Deploying daemon osd.8 on smithi083 2024-09-16T00:12:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:42 smithi033 ceph-mon[31243]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:42 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-16T00:12:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:42 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:43.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:42 smithi033 ceph-mon[31243]: Deploying daemon osd.8 on smithi083 2024-09-16T00:12:45.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:44 smithi083 ceph-mon[35242]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:45.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:44 smithi060 ceph-mon[35185]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:45.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:44 smithi033 ceph-mon[31243]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:46.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:46 smithi083 ceph-mon[35242]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:47.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:46 smithi060 ceph-mon[35185]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:47.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:46 smithi033 ceph-mon[31243]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:49.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:48 smithi083 ceph-mon[35242]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:49.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:49.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:49.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:49.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:48 smithi060 ceph-mon[35185]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:49.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:49.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:49.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:49.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:48 smithi033 ceph-mon[31243]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:49.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:49.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:49.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:51.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:50 smithi083 ceph-mon[35242]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:51.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:50 smithi060 ceph-mon[35185]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:51.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:50 smithi033 ceph-mon[31243]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:51.950 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 8 on host 'smithi083' 2024-09-16T00:12:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.948+0000 7f98a4a0d640 1 -- 172.21.15.83:0/235510582 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f98a0062630 con 0x7f98880a3270 2024-09-16T00:12:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.951+0000 7f98a5a0f640 1 -- 172.21.15.83:0/235510582 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f98880a3270 msgr2=0x7f98880a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.951+0000 7f98a5a0f640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f98880a3270 0x7f98880a5730 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9890009830 tx=0x7f9890030570 comp rx=0 tx=0).stop 2024-09-16T00:12:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.951+0000 7f98a5a0f640 1 -- 172.21.15.83:0/235510582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98a0102e00 msgr2=0x7f98a0198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.951+0000 7f98a5a0f640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98a0102e00 0x7f98a0198420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f989400e9c0 tx=0x7f989400ee80 comp rx=0 tx=0).stop 2024-09-16T00:12:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.951+0000 7f98a5a0f640 1 -- 172.21.15.83:0/235510582 shutdown_connections 2024-09-16T00:12:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.951+0000 7f98a5a0f640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f98880a3270 0x7f98880a5730 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.951+0000 7f98a5a0f640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98a01037c0 0x7f98a019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.951+0000 7f98a5a0f640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98a0102e00 0x7f98a0198420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.951+0000 7f98a5a0f640 1 --2- 172.21.15.83:0/235510582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0101c00 0x7f98a0197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:51.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.951+0000 7f98a5a0f640 1 -- 172.21.15.83:0/235510582 >> 172.21.15.83:0/235510582 conn(0x7f98a00fd390 msgr2=0x7f98a0106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:12:51.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.951+0000 7f98a5a0f640 1 -- 172.21.15.83:0/235510582 shutdown_connections 2024-09-16T00:12:51.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:12:51.951+0000 7f98a5a0f640 1 -- 172.21.15.83:0/235510582 wait complete. 2024-09-16T00:12:52.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:52.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:52.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:52.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:52.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:52.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:52.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:52.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:52.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:12:52.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:52.627 DEBUG:teuthology.orchestra.run.smithi083:osd.8> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.8.service 2024-09-16T00:12:52.630 INFO:tasks.cephadm:Deploying osd.9 on smithi033 with /dev/vg_nvme/lv_1... 2024-09-16T00:12:52.630 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T00:12:52.897 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:12:53.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:52 smithi083 ceph-mon[35242]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:53.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:52 smithi033 ceph-mon[31243]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:53.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:52 smithi060 ceph-mon[35185]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:54.669 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:12:54 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:12:54.274+0000 7fb4af52c640 -1 osd.8 0 log_to_monitors true 2024-09-16T00:12:54.997 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:54 smithi083 ceph-mon[35242]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:54.997 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:54 smithi083 ceph-mon[35242]: from='osd.8 [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T00:12:54.997 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:54 smithi083 ceph-mon[35242]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T00:12:55.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:54 smithi033 ceph-mon[31243]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:55.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:54 smithi033 ceph-mon[31243]: from='osd.8 [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T00:12:55.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:54 smithi033 ceph-mon[31243]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T00:12:55.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:54 smithi060 ceph-mon[35185]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:55.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:54 smithi060 ceph-mon[35185]: from='osd.8 [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T00:12:55.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:54 smithi060 ceph-mon[35185]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T00:12:55.736 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:12:55.763 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_1 2024-09-16T00:12:56.028 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:12:56.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-16T00:12:56.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: osdmap e53: 9 total, 8 up, 9 in 2024-09-16T00:12:56.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:56.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='osd.8 [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:12:56.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:12:56.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: Detected new or changed devices on smithi083 2024-09-16T00:12:56.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:56.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:56.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:56.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:56.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:56.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi083 to 5696M 2024-09-16T00:12:56.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:56.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:56.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:56.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:56.419 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:12:56.010+0000 7fb4ab4af640 -1 osd.8 0 waiting for initial osdmap 2024-09-16T00:12:56.419 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:12:56.018+0000 7fb4a72d7640 -1 osd.8 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T00:12:56.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-16T00:12:56.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: osdmap e53: 9 total, 8 up, 9 in 2024-09-16T00:12:56.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:56.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='osd.8 [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:12:56.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:12:56.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: Detected new or changed devices on smithi083 2024-09-16T00:12:56.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:56.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:56.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:56.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:56.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:56.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi083 to 5696M 2024-09-16T00:12:56.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:56.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:56.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:56.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:56.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-16T00:12:56.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: osdmap e53: 9 total, 8 up, 9 in 2024-09-16T00:12:56.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:56.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='osd.8 [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:12:56.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:12:56.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: Detected new or changed devices on smithi083 2024-09-16T00:12:56.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:56.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:56.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:56.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:56.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:12:56.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi083 to 5696M 2024-09-16T00:12:56.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:56.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:56.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:12:56.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:12:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.547+0000 7f7ae3b27640 1 -- 172.21.15.33:0/1242530635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7adc102690 msgr2=0x7f7adc108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.547+0000 7f7ae3b27640 1 --2- 172.21.15.33:0/1242530635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7adc102690 0x7f7adc108f20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009ca0 tx=0x7f7ad402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:12:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.547+0000 7f7ae3b27640 1 -- 172.21.15.33:0/1242530635 shutdown_connections 2024-09-16T00:12:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.547+0000 7f7ae3b27640 1 --2- 172.21.15.33:0/1242530635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7adc102690 0x7f7adc108f20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.547+0000 7f7ae3b27640 1 --2- 172.21.15.33:0/1242530635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7adc101cd0 0x7f7adc102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.547+0000 7f7ae3b27640 1 --2- 172.21.15.33:0/1242530635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7adc100ad0 0x7f7adc100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.547+0000 7f7ae3b27640 1 -- 172.21.15.33:0/1242530635 >> 172.21.15.33:0/1242530635 conn(0x7f7adc0fc280 msgr2=0x7f7adc0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:12:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.547+0000 7f7ae3b27640 1 -- 172.21.15.33:0/1242530635 shutdown_connections 2024-09-16T00:12:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.547+0000 7f7ae3b27640 1 -- 172.21.15.33:0/1242530635 wait complete. 2024-09-16T00:12:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.548+0000 7f7ae3b27640 1 Processor -- start 2024-09-16T00:12:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.548+0000 7f7ae3b27640 1 -- start start 2024-09-16T00:12:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.548+0000 7f7ae3b27640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7adc100ad0 0x7f7adc104a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:12:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.548+0000 7f7ae3b27640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7adc101cd0 0x7f7adc1030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:12:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.548+0000 7f7ae3b27640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7adc102690 0x7f7adc1035f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:12:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.548+0000 7f7ae3b27640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc10c730 con 0x7f7adc101cd0 2024-09-16T00:12:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.548+0000 7f7ae3b27640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc10c5b0 con 0x7f7adc100ad0 2024-09-16T00:12:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.548+0000 7f7ae3b27640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc10c8b0 con 0x7f7adc102690 2024-09-16T00:12:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.548+0000 7f7ae10c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7adc101cd0 0x7f7adc1030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:12:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.548+0000 7f7ae10c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7adc101cd0 0x7f7adc1030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44122/0 (socket says 172.21.15.33:44122) 2024-09-16T00:12:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.548+0000 7f7ae10c2640 1 -- 172.21.15.33:0/87550563 learned_addr learned my addr 172.21.15.33:0/87550563 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:12:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ae20c4640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7adc102690 0x7f7adc1035f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:12:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ae18c3640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7adc100ad0 0x7f7adc104a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:12:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ae10c2640 1 -- 172.21.15.33:0/87550563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7adc102690 msgr2=0x7f7adc1035f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ae10c2640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7adc102690 0x7f7adc1035f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ae10c2640 1 -- 172.21.15.33:0/87550563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7adc100ad0 msgr2=0x7f7adc104a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:12:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ae10c2640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7adc100ad0 0x7f7adc104a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:12:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ae10c2640 1 -- 172.21.15.33:0/87550563 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7adc103eb0 con 0x7f7adc101cd0 2024-09-16T00:12:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ae18c3640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7adc100ad0 0x7f7adc104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:12:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ae10c2640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7adc101cd0 0x7f7adc1030b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7acc00c920 tx=0x7f7acc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:12:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ad2ffd640 1 -- 172.21.15.33:0/87550563 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7acc007c70 con 0x7f7adc101cd0 2024-09-16T00:12:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ad2ffd640 1 -- 172.21.15.33:0/87550563 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7acc007e10 con 0x7f7adc101cd0 2024-09-16T00:12:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ae3b27640 1 -- 172.21.15.33:0/87550563 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7adc1a1fa0 con 0x7f7adc101cd0 2024-09-16T00:12:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ad2ffd640 1 -- 172.21.15.33:0/87550563 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7acc018890 con 0x7f7adc101cd0 2024-09-16T00:12:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.549+0000 7f7ae3b27640 1 -- 172.21.15.33:0/87550563 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7adc1a2450 con 0x7f7adc101cd0 2024-09-16T00:12:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.550+0000 7f7ae3b27640 1 -- 172.21.15.33:0/87550563 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7adc1a26e0 con 0x7f7adc101cd0 2024-09-16T00:12:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.552+0000 7f7ad2ffd640 1 -- 172.21.15.33:0/87550563 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f7acc018a50 con 0x7f7adc101cd0 2024-09-16T00:12:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.553+0000 7f7ad2ffd640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ac80a32c0 0x7f7ac80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:12:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.553+0000 7f7ae18c3640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ac80a32c0 0x7f7ac80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:12:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.553+0000 7f7ad2ffd640 1 -- 172.21.15.33:0/87550563 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5862+0+0 (secure 0 0 0) 0x7f7acc0cf370 con 0x7f7adc101cd0 2024-09-16T00:12:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.554+0000 7f7ae18c3640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ac80a32c0 0x7f7ac80a5780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7adc101b30 tx=0x7f7ad8005e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:12:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.556+0000 7f7ad2ffd640 1 -- 172.21.15.33:0/87550563 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7acc014030 con 0x7f7adc101cd0 2024-09-16T00:12:56.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:12:56.741+0000 7f7ae3b27640 1 -- 172.21.15.33:0/87550563 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7adc1a28b0 con 0x7f7ac80a32c0 2024-09-16T00:12:57.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:57 smithi033 ceph-mon[31243]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:57.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:57 smithi033 ceph-mon[31243]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-16T00:12:57.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:57 smithi033 ceph-mon[31243]: osdmap e54: 9 total, 8 up, 9 in 2024-09-16T00:12:57.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:57.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:57.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:57 smithi033 ceph-mon[31243]: osd.8 [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527] boot 2024-09-16T00:12:57.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:57 smithi033 ceph-mon[31243]: osdmap e55: 9 total, 9 up, 9 in 2024-09-16T00:12:57.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:57.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:12:57.313 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:12:57.313 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:57 smithi083 ceph-mon[35242]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:57 smithi083 ceph-mon[35242]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-16T00:12:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:57 smithi083 ceph-mon[35242]: osdmap e54: 9 total, 8 up, 9 in 2024-09-16T00:12:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:57 smithi083 ceph-mon[35242]: osd.8 [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527] boot 2024-09-16T00:12:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:57 smithi083 ceph-mon[35242]: osdmap e55: 9 total, 9 up, 9 in 2024-09-16T00:12:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:12:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:12:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:57 smithi060 ceph-mon[35185]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-16T00:12:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:57 smithi060 ceph-mon[35185]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-16T00:12:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:57 smithi060 ceph-mon[35185]: osdmap e54: 9 total, 8 up, 9 in 2024-09-16T00:12:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:57 smithi060 ceph-mon[35185]: osd.8 [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527] boot 2024-09-16T00:12:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:57 smithi060 ceph-mon[35185]: osdmap e55: 9 total, 9 up, 9 in 2024-09-16T00:12:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T00:12:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:12:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:12:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:12:58.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:58 smithi033 ceph-mon[31243]: purged_snaps scrub starts 2024-09-16T00:12:58.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:58 smithi033 ceph-mon[31243]: purged_snaps scrub ok 2024-09-16T00:12:58.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:58 smithi033 ceph-mon[31243]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:12:58.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:58 smithi033 ceph-mon[31243]: osdmap e56: 9 total, 9 up, 9 in 2024-09-16T00:12:58.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:58 smithi083 ceph-mon[35242]: purged_snaps scrub starts 2024-09-16T00:12:58.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:58 smithi083 ceph-mon[35242]: purged_snaps scrub ok 2024-09-16T00:12:58.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:58 smithi083 ceph-mon[35242]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:12:58.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:58 smithi083 ceph-mon[35242]: osdmap e56: 9 total, 9 up, 9 in 2024-09-16T00:12:58.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:58 smithi060 ceph-mon[35185]: purged_snaps scrub starts 2024-09-16T00:12:58.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:58 smithi060 ceph-mon[35185]: purged_snaps scrub ok 2024-09-16T00:12:58.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:58 smithi060 ceph-mon[35185]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:12:58.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:58 smithi060 ceph-mon[35185]: osdmap e56: 9 total, 9 up, 9 in 2024-09-16T00:12:59.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:59 smithi033 ceph-mon[31243]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:12:59.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:12:59 smithi033 ceph-mon[31243]: osdmap e57: 9 total, 9 up, 9 in 2024-09-16T00:12:59.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:59 smithi083 ceph-mon[35242]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:12:59.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:12:59 smithi083 ceph-mon[35242]: osdmap e57: 9 total, 9 up, 9 in 2024-09-16T00:12:59.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:59 smithi060 ceph-mon[35185]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:12:59.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:12:59 smithi060 ceph-mon[35185]: osdmap e57: 9 total, 9 up, 9 in 2024-09-16T00:13:00.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:00 smithi033 ceph-mon[31243]: osdmap e58: 9 total, 9 up, 9 in 2024-09-16T00:13:00.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:00 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3369909816' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03160394-ba77-4a71-80fe-6c20837c3cb7"}]: dispatch 2024-09-16T00:13:00.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:00 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3369909816' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03160394-ba77-4a71-80fe-6c20837c3cb7"}]': finished 2024-09-16T00:13:00.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:00 smithi033 ceph-mon[31243]: osdmap e59: 10 total, 9 up, 10 in 2024-09-16T00:13:00.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:00.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:00 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/552187607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:13:00.373 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:00 smithi060 ceph-mon[35185]: osdmap e58: 9 total, 9 up, 9 in 2024-09-16T00:13:00.373 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:00 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3369909816' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03160394-ba77-4a71-80fe-6c20837c3cb7"}]: dispatch 2024-09-16T00:13:00.373 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:00 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3369909816' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03160394-ba77-4a71-80fe-6c20837c3cb7"}]': finished 2024-09-16T00:13:00.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:00 smithi060 ceph-mon[35185]: osdmap e59: 10 total, 9 up, 10 in 2024-09-16T00:13:00.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:00.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:00 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/552187607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:13:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:00 smithi083 ceph-mon[35242]: osdmap e58: 9 total, 9 up, 9 in 2024-09-16T00:13:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:00 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3369909816' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03160394-ba77-4a71-80fe-6c20837c3cb7"}]: dispatch 2024-09-16T00:13:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:00 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3369909816' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03160394-ba77-4a71-80fe-6c20837c3cb7"}]': finished 2024-09-16T00:13:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:00 smithi083 ceph-mon[35242]: osdmap e59: 10 total, 9 up, 10 in 2024-09-16T00:13:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:00.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:00 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/552187607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:13:01.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:01 smithi083 ceph-mon[35242]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:01 smithi060 ceph-mon[35185]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:01.833 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:01 smithi033 ceph-mon[31243]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:03 smithi083 ceph-mon[35242]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:03.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:03 smithi060 ceph-mon[35185]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:03.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:03 smithi033 ceph-mon[31243]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:05.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:05 smithi083 ceph-mon[35242]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T00:13:05.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:05 smithi060 ceph-mon[35185]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T00:13:05.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:05 smithi033 ceph-mon[31243]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T00:13:07.377 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:07 smithi033 ceph-mon[31243]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-16T00:13:07.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:07 smithi083 ceph-mon[35242]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-16T00:13:07.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:07 smithi060 ceph-mon[35185]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-16T00:13:09.380 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:09 smithi033 ceph-mon[31243]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-16T00:13:09.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-16T00:13:09.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:09.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:09 smithi033 ceph-mon[31243]: Deploying daemon osd.9 on smithi033 2024-09-16T00:13:09.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:09 smithi083 ceph-mon[35242]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-16T00:13:09.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-16T00:13:09.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:09.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:09 smithi083 ceph-mon[35242]: Deploying daemon osd.9 on smithi033 2024-09-16T00:13:09.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:09 smithi060 ceph-mon[35185]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-16T00:13:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-16T00:13:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:09 smithi060 ceph-mon[35185]: Deploying daemon osd.9 on smithi033 2024-09-16T00:13:11.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:11 smithi033 ceph-mon[31243]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-16T00:13:11.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:11 smithi083 ceph-mon[35242]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-16T00:13:11.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:11 smithi060 ceph-mon[35185]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-16T00:13:13.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:13 smithi033 ceph-mon[31243]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:13:13.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:13 smithi083 ceph-mon[35242]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:13:13.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:13 smithi060 ceph-mon[35185]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:13:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:15 smithi083 ceph-mon[35242]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:13:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:13:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:15.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:15 smithi060 ceph-mon[35185]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:13:15.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:13:15.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:15.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:15 smithi033 ceph-mon[31243]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:13:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:13:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:17.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:17 smithi033 ceph-mon[31243]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:17.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:17 smithi083 ceph-mon[35242]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:17 smithi060 ceph-mon[35185]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:18.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.457+0000 7f7ad2ffd640 1 -- 172.21.15.33:0/87550563 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7adc1a28b0 con 0x7f7ac80a32c0 2024-09-16T00:13:18.458 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 9 on host 'smithi033' 2024-09-16T00:13:18.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.460+0000 7f7ae3b27640 1 -- 172.21.15.33:0/87550563 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ac80a32c0 msgr2=0x7f7ac80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:13:18.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.460+0000 7f7ae3b27640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ac80a32c0 0x7f7ac80a5780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7adc101b30 tx=0x7f7ad8005e70 comp rx=0 tx=0).stop 2024-09-16T00:13:18.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.460+0000 7f7ae3b27640 1 -- 172.21.15.33:0/87550563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7adc101cd0 msgr2=0x7f7adc1030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:13:18.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.460+0000 7f7ae3b27640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7adc101cd0 0x7f7adc1030b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7acc00c920 tx=0x7f7acc00cde0 comp rx=0 tx=0).stop 2024-09-16T00:13:18.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.460+0000 7f7ae3b27640 1 -- 172.21.15.33:0/87550563 shutdown_connections 2024-09-16T00:13:18.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.460+0000 7f7ae3b27640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ac80a32c0 0x7f7ac80a5780 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:18.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.460+0000 7f7ae3b27640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7adc102690 0x7f7adc1035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:18.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.460+0000 7f7ae3b27640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7adc101cd0 0x7f7adc1030b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:18.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.460+0000 7f7ae3b27640 1 --2- 172.21.15.33:0/87550563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7adc100ad0 0x7f7adc104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:18.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.460+0000 7f7ae3b27640 1 -- 172.21.15.33:0/87550563 >> 172.21.15.33:0/87550563 conn(0x7f7adc0fc280 msgr2=0x7f7adc0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:13:18.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.460+0000 7f7ae3b27640 1 -- 172.21.15.33:0/87550563 shutdown_connections 2024-09-16T00:13:18.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:13:18.460+0000 7f7ae3b27640 1 -- 172.21.15.33:0/87550563 wait complete. 2024-09-16T00:13:18.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:18.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:13:18.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:18 smithi033 ceph-mon[31243]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:18.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:13:18.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:18 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:13:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:18 smithi083 ceph-mon[35242]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:13:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:18 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:18.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:13:18.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:18 smithi060 ceph-mon[35185]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:18.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:13:18.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:18.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:18 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:19.181 DEBUG:teuthology.orchestra.run.smithi033:osd.9> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.9.service 2024-09-16T00:13:19.184 INFO:tasks.cephadm:Deploying osd.10 on smithi060 with /dev/vg_nvme/lv_1... 2024-09-16T00:13:19.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T00:13:19.448 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T00:13:20.663 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:13:20 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:13:20.448+0000 7f2ce5376640 -1 osd.9 0 log_to_monitors true 2024-09-16T00:13:20.829 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:20 smithi060 ceph-mon[35185]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:20.829 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:20 smithi060 ceph-mon[35185]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-16T00:13:20.915 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:20 smithi033 ceph-mon[31243]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:20.915 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:20 smithi033 ceph-mon[31243]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-16T00:13:20.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:20 smithi083 ceph-mon[35242]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:20.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:20 smithi083 ceph-mon[35242]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-16T00:13:21.954 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:21 smithi060 ceph-mon[35185]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-16T00:13:21.954 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:21 smithi060 ceph-mon[35185]: osdmap e60: 10 total, 9 up, 10 in 2024-09-16T00:13:21.955 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:21 smithi060 ceph-mon[35185]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-16T00:13:21.955 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:21 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:22.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:21 smithi033 ceph-mon[31243]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-16T00:13:22.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:21 smithi033 ceph-mon[31243]: osdmap e60: 10 total, 9 up, 10 in 2024-09-16T00:13:22.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:21 smithi033 ceph-mon[31243]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-16T00:13:22.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:21 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:22.094 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:21 smithi083 ceph-mon[35242]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-16T00:13:22.094 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:21 smithi083 ceph-mon[35242]: osdmap e60: 10 total, 9 up, 10 in 2024-09-16T00:13:22.094 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:21 smithi083 ceph-mon[35242]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-16T00:13:22.094 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:21 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:22.284 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-16T00:13:22.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi060:vg_nvme/lv_1 2024-09-16T00:13:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T00:13:22.580 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:13:22.103+0000 7f2ce12f9640 -1 osd.9 0 waiting for initial osdmap 2024-09-16T00:13:22.580 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:13:22.113+0000 7f2cd9119640 -1 osd.9 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T00:13:22.918 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:22.918 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-16T00:13:22.918 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: osdmap e61: 10 total, 9 up, 10 in 2024-09-16T00:13:22.918 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:22.918 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:22.918 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' 2024-09-16T00:13:22.919 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: Detected new or changed devices on smithi033 2024-09-16T00:13:22.919 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:22.919 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:22.919 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:22.919 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:22.919 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:22.919 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:22.919 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi033 to 4273M 2024-09-16T00:13:22.919 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:22.919 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:22.919 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:13:22.919 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:23.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.000+0000 7f7fc05ca640 1 -- 172.21.15.60:0/2615468449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb8107960 msgr2=0x7f7fb8109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:13:23.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.000+0000 7f7fc05ca640 1 --2- 172.21.15.60:0/2615468449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb8107960 0x7f7fb8109d50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009ca0 tx=0x7f7fb002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:13:23.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.000+0000 7f7fc05ca640 1 -- 172.21.15.60:0/2615468449 shutdown_connections 2024-09-16T00:13:23.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.000+0000 7f7fc05ca640 1 --2- 172.21.15.60:0/2615468449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb8107960 0x7f7fb8109d50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:23.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.000+0000 7f7fc05ca640 1 --2- 172.21.15.60:0/2615468449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fb8105030 0x7f7fb8107420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:23.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.000+0000 7f7fc05ca640 1 --2- 172.21.15.60:0/2615468449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fb8102700 0x7f7fb8104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:23.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.000+0000 7f7fc05ca640 1 -- 172.21.15.60:0/2615468449 >> 172.21.15.60:0/2615468449 conn(0x7f7fb80fc2a0 msgr2=0x7f7fb80fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fc05ca640 1 -- 172.21.15.60:0/2615468449 shutdown_connections 2024-09-16T00:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fc05ca640 1 -- 172.21.15.60:0/2615468449 wait complete. 2024-09-16T00:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fc05ca640 1 Processor -- start 2024-09-16T00:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fc05ca640 1 -- start start 2024-09-16T00:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fc05ca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb8102700 0x7f7fb8102060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fc05ca640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fb8105030 0x7f7fb81006b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fc05ca640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fb8107960 0x7f7fb8100bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fc05ca640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb810d8a0 con 0x7f7fb8105030 2024-09-16T00:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fc05ca640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb810d720 con 0x7f7fb8102700 2024-09-16T00:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fc05ca640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb810da20 con 0x7f7fb8107960 2024-09-16T00:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fbe366640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb8102700 0x7f7fb8102060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fbe366640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb8102700 0x7f7fb8102060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45318/0 (socket says 172.21.15.60:45318) 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fbe366640 1 -- 172.21.15.60:0/3410515872 learned_addr learned my addr 172.21.15.60:0/3410515872 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.001+0000 7f7fbeb67640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fb8107960 0x7f7fb8100bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7fbdb65640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fb8105030 0x7f7fb81006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7fbe366640 1 -- 172.21.15.60:0/3410515872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fb8107960 msgr2=0x7f7fb8100bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7fbe366640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fb8107960 0x7f7fb8100bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7fbe366640 1 -- 172.21.15.60:0/3410515872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fb8105030 msgr2=0x7f7fb81006b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7fbe366640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fb8105030 0x7f7fb81006b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7fbe366640 1 -- 172.21.15.60:0/3410515872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb81014b0 con 0x7f7fb8102700 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7fbdb65640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fb8105030 0x7f7fb81006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7fbe366640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb8102700 0x7f7fb8102060 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400e9f0 tx=0x7f7fb400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7faf7fe640 1 -- 172.21.15.60:0/3410515872 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb400cd80 con 0x7f7fb8102700 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7faf7fe640 1 -- 172.21.15.60:0/3410515872 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fb4010910 con 0x7f7fb8102700 2024-09-16T00:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7fc05ca640 1 -- 172.21.15.60:0/3410515872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb81a2090 con 0x7f7fb8102700 2024-09-16T00:13:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7faf7fe640 1 -- 172.21.15.60:0/3410515872 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4018990 con 0x7f7fb8102700 2024-09-16T00:13:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.002+0000 7f7fc05ca640 1 -- 172.21.15.60:0/3410515872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb81a25f0 con 0x7f7fb8102700 2024-09-16T00:13:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.003+0000 7f7fc05ca640 1 -- 172.21.15.60:0/3410515872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fb80fdd90 con 0x7f7fb8102700 2024-09-16T00:13:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.004+0000 7f7faf7fe640 1 -- 172.21.15.60:0/3410515872 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f7fb4010ac0 con 0x7f7fb8102700 2024-09-16T00:13:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.005+0000 7f7faf7fe640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7fa40a32c0 0x7f7fa40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:13:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.005+0000 7f7faf7fe640 1 -- 172.21.15.60:0/3410515872 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5981+0+0 (secure 0 0 0) 0x7f7fb40cf700 con 0x7f7fb8102700 2024-09-16T00:13:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.005+0000 7f7fbdb65640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7fa40a32c0 0x7f7fa40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:13:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.006+0000 7f7fbdb65640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7fa40a32c0 0x7f7fa40a5780 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8101de0 tx=0x7f7fa8005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:13:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.008+0000 7f7faf7fe640 1 -- 172.21.15.60:0/3410515872 <== mon.2 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7fb409cd40 con 0x7f7fb8102700 2024-09-16T00:13:23.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:23.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-16T00:13:23.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: osdmap e61: 10 total, 9 up, 10 in 2024-09-16T00:13:23.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:23.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:23.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' 2024-09-16T00:13:23.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: Detected new or changed devices on smithi033 2024-09-16T00:13:23.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:23.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:23.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:23.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:23.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:23.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:23.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi033 to 4273M 2024-09-16T00:13:23.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:23.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:23.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:13:23.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:23.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:23.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-16T00:13:23.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: osdmap e61: 10 total, 9 up, 10 in 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241]' entity='osd.9' 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: Detected new or changed devices on smithi033 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi033 to 4273M 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:23.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:23.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:13:23.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:23.194+0000 7f7fc05ca640 1 -- 172.21.15.60:0/3410515872 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7fb8062630 con 0x7f7fa40a32c0 2024-09-16T00:13:23.863 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:23 smithi060 ceph-mon[35185]: purged_snaps scrub starts 2024-09-16T00:13:23.864 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:23 smithi060 ceph-mon[35185]: purged_snaps scrub ok 2024-09-16T00:13:23.864 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:23.864 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:23 smithi060 ceph-mon[35185]: osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241] boot 2024-09-16T00:13:23.864 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:23 smithi060 ceph-mon[35185]: osdmap e62: 10 total, 10 up, 10 in 2024-09-16T00:13:23.864 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:23.864 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:23 smithi060 ceph-mon[35185]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:13:23.864 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:13:23.865 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:13:23.865 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:24.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:23 smithi033 ceph-mon[31243]: purged_snaps scrub starts 2024-09-16T00:13:24.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:23 smithi033 ceph-mon[31243]: purged_snaps scrub ok 2024-09-16T00:13:24.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:24.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:23 smithi033 ceph-mon[31243]: osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241] boot 2024-09-16T00:13:24.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:23 smithi033 ceph-mon[31243]: osdmap e62: 10 total, 10 up, 10 in 2024-09-16T00:13:24.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:24.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:23 smithi033 ceph-mon[31243]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:13:24.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:13:24.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:13:24.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:24.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:23 smithi083 ceph-mon[35242]: purged_snaps scrub starts 2024-09-16T00:13:24.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:23 smithi083 ceph-mon[35242]: purged_snaps scrub ok 2024-09-16T00:13:24.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:24.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:23 smithi083 ceph-mon[35242]: osd.9 [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241] boot 2024-09-16T00:13:24.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:23 smithi083 ceph-mon[35242]: osdmap e62: 10 total, 10 up, 10 in 2024-09-16T00:13:24.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T00:13:24.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:23 smithi083 ceph-mon[35242]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:13:24.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:13:24.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:13:24.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:25.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:25 smithi083 ceph-mon[35242]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:25.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:25 smithi083 ceph-mon[35242]: osdmap e63: 10 total, 10 up, 10 in 2024-09-16T00:13:25.466 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:25 smithi060 ceph-mon[35185]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:25.466 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:25 smithi060 ceph-mon[35185]: osdmap e63: 10 total, 10 up, 10 in 2024-09-16T00:13:25.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:25 smithi033 ceph-mon[31243]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-09-16T00:13:25.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:25 smithi033 ceph-mon[31243]: osdmap e63: 10 total, 10 up, 10 in 2024-09-16T00:13:26.381 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:26 smithi060 ceph-mon[35185]: from='client.? 172.21.15.60:0/2645932963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae2336fa-0b07-42a2-b67f-c77b0f7db0da"}]: dispatch 2024-09-16T00:13:26.381 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:26 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae2336fa-0b07-42a2-b67f-c77b0f7db0da"}]: dispatch 2024-09-16T00:13:26.381 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:26 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae2336fa-0b07-42a2-b67f-c77b0f7db0da"}]': finished 2024-09-16T00:13:26.381 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:26 smithi060 ceph-mon[35185]: osdmap e64: 11 total, 10 up, 11 in 2024-09-16T00:13:26.381 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:26 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:26.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:26 smithi083 ceph-mon[35242]: from='client.? 172.21.15.60:0/2645932963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae2336fa-0b07-42a2-b67f-c77b0f7db0da"}]: dispatch 2024-09-16T00:13:26.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:26 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae2336fa-0b07-42a2-b67f-c77b0f7db0da"}]: dispatch 2024-09-16T00:13:26.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:26 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae2336fa-0b07-42a2-b67f-c77b0f7db0da"}]': finished 2024-09-16T00:13:26.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:26 smithi083 ceph-mon[35242]: osdmap e64: 11 total, 10 up, 11 in 2024-09-16T00:13:26.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:26 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:26.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:26 smithi033 ceph-mon[31243]: from='client.? 172.21.15.60:0/2645932963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae2336fa-0b07-42a2-b67f-c77b0f7db0da"}]: dispatch 2024-09-16T00:13:26.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:26 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae2336fa-0b07-42a2-b67f-c77b0f7db0da"}]: dispatch 2024-09-16T00:13:26.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:26 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae2336fa-0b07-42a2-b67f-c77b0f7db0da"}]': finished 2024-09-16T00:13:26.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:26 smithi033 ceph-mon[31243]: osdmap e64: 11 total, 10 up, 11 in 2024-09-16T00:13:26.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:26 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:27.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:27 smithi083 ceph-mon[35242]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:27.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:27 smithi083 ceph-mon[35242]: from='client.? 172.21.15.60:0/1426219857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:13:27.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:27 smithi060 ceph-mon[35185]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:27.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:27 smithi060 ceph-mon[35185]: from='client.? 172.21.15.60:0/1426219857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:13:27.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:27 smithi033 ceph-mon[31243]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:27.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:27 smithi033 ceph-mon[31243]: from='client.? 172.21.15.60:0/1426219857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:13:29.382 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:29 smithi033 ceph-mon[31243]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:29 smithi083 ceph-mon[35242]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:29.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:29 smithi060 ceph-mon[35185]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:31 smithi060 ceph-mon[35185]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:31.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:31 smithi083 ceph-mon[35242]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:31.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:31 smithi033 ceph-mon[31243]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:33.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:33 smithi060 ceph-mon[35185]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:33.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:33 smithi083 ceph-mon[35242]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:33.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:33 smithi033 ceph-mon[31243]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:35 smithi060 ceph-mon[35185]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-16T00:13:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:35 smithi060 ceph-mon[35185]: Deploying daemon osd.10 on smithi060 2024-09-16T00:13:35.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:35 smithi083 ceph-mon[35242]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:35.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-16T00:13:35.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:35.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:35 smithi083 ceph-mon[35242]: Deploying daemon osd.10 on smithi060 2024-09-16T00:13:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:35 smithi033 ceph-mon[31243]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-16T00:13:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:35 smithi033 ceph-mon[31243]: Deploying daemon osd.10 on smithi060 2024-09-16T00:13:37.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:37 smithi060 ceph-mon[35185]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:37.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:37 smithi083 ceph-mon[35242]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:37.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:37 smithi033 ceph-mon[31243]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:39.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:39 smithi060 ceph-mon[35185]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:39.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:39 smithi083 ceph-mon[35242]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:39.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:39 smithi033 ceph-mon[31243]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:41.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:41 smithi060 ceph-mon[35185]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:41.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:41 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:13:41.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:41 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:41.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:41 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:41.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:41 smithi083 ceph-mon[35242]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:41.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:41 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:13:41.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:41 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:41.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:41 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:41.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:41 smithi033 ceph-mon[31243]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:41.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:41 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:13:41.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:41 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:41.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:41 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:43 smithi083 ceph-mon[35242]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:13:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:43.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:43 smithi060 ceph-mon[35185]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:13:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:43.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:43 smithi033 ceph-mon[31243]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:43.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:43.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:43.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:43.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:13:43.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:43.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.863+0000 7f7faf7fe640 1 -- 172.21.15.60:0/3410515872 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f7fb8062630 con 0x7f7fa40a32c0 2024-09-16T00:13:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 10 on host 'smithi060' 2024-09-16T00:13:43.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.866+0000 7f7fc05ca640 1 -- 172.21.15.60:0/3410515872 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7fa40a32c0 msgr2=0x7f7fa40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:13:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.866+0000 7f7fc05ca640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7fa40a32c0 0x7f7fa40a5780 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8101de0 tx=0x7f7fa8005d30 comp rx=0 tx=0).stop 2024-09-16T00:13:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.866+0000 7f7fc05ca640 1 -- 172.21.15.60:0/3410515872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb8102700 msgr2=0x7f7fb8102060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:13:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.866+0000 7f7fc05ca640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb8102700 0x7f7fb8102060 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400e9f0 tx=0x7f7fb400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:13:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.866+0000 7f7fc05ca640 1 -- 172.21.15.60:0/3410515872 shutdown_connections 2024-09-16T00:13:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.866+0000 7f7fc05ca640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7fa40a32c0 0x7f7fa40a5780 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.866+0000 7f7fc05ca640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fb8107960 0x7f7fb8100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.866+0000 7f7fc05ca640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fb8105030 0x7f7fb81006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.866+0000 7f7fc05ca640 1 --2- 172.21.15.60:0/3410515872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb8102700 0x7f7fb8102060 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:43.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.866+0000 7f7fc05ca640 1 -- 172.21.15.60:0/3410515872 >> 172.21.15.60:0/3410515872 conn(0x7f7fb80fc2a0 msgr2=0x7f7fb8102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:13:43.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.866+0000 7f7fc05ca640 1 -- 172.21.15.60:0/3410515872 shutdown_connections 2024-09-16T00:13:43.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-16T00:13:43.866+0000 7f7fc05ca640 1 -- 172.21.15.60:0/3410515872 wait complete. 2024-09-16T00:13:44.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:44 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:13:44.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:44 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:44.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:44 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:44.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:44 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:13:44.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:44 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:44.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:44 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:44.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:44 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:13:44.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:44 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:44.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:44 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:44.585 DEBUG:teuthology.orchestra.run.smithi060:osd.10> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.10.service 2024-09-16T00:13:44.587 INFO:tasks.cephadm:Deploying osd.11 on smithi083 with /dev/vg_nvme/lv_1... 2024-09-16T00:13:44.588 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T00:13:44.844 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:13:45.223 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:45 smithi083 ceph-mon[35242]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:45.439 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:45 smithi060 ceph-mon[35185]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:45.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:45 smithi033 ceph-mon[31243]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:45.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:13:45 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:13:45.532+0000 7fafcd132640 -1 osd.10 0 log_to_monitors true 2024-09-16T00:13:46.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:46 smithi083 ceph-mon[35242]: from='osd.10 [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T00:13:46.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:46 smithi083 ceph-mon[35242]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T00:13:46.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:46 smithi060 ceph-mon[35185]: from='osd.10 [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T00:13:46.555 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:46 smithi060 ceph-mon[35185]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T00:13:46.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:46 smithi033 ceph-mon[31243]: from='osd.10 [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T00:13:46.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:46 smithi033 ceph-mon[31243]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T00:13:47.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:47 smithi083 ceph-mon[35242]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:47.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:47 smithi083 ceph-mon[35242]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-16T00:13:47.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:47 smithi083 ceph-mon[35242]: from='osd.10 [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:13:47.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:47 smithi083 ceph-mon[35242]: osdmap e65: 11 total, 10 up, 11 in 2024-09-16T00:13:47.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:47 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:47.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:47 smithi083 ceph-mon[35242]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:13:47.465 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-16T00:13:47.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:47 smithi060 ceph-mon[35185]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:47.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:47 smithi060 ceph-mon[35185]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-16T00:13:47.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:47 smithi060 ceph-mon[35185]: from='osd.10 [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:13:47.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:47 smithi060 ceph-mon[35185]: osdmap e65: 11 total, 10 up, 11 in 2024-09-16T00:13:47.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:47 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:47.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:47 smithi060 ceph-mon[35185]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:13:47.491 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_1 2024-09-16T00:13:47.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:47 smithi033 ceph-mon[31243]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:47.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:47 smithi033 ceph-mon[31243]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-16T00:13:47.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:47 smithi033 ceph-mon[31243]: from='osd.10 [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:13:47.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:47 smithi033 ceph-mon[31243]: osdmap e65: 11 total, 10 up, 11 in 2024-09-16T00:13:47.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:47 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:47.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:47 smithi033 ceph-mon[31243]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-09-16T00:13:47.748 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:13:47.980 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:13:47 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:13:47.577+0000 7fafc90b5640 -1 osd.10 0 waiting for initial osdmap 2024-09-16T00:13:47.980 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:13:47 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:13:47.587+0000 7fafc0ed5640 -1 osd.10 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T00:13:48.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.210+0000 7fe3d71ee640 1 -- 172.21.15.83:0/2305494139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d0102690 msgr2=0x7fe3d0108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:13:48.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.210+0000 7fe3d71ee640 1 --2- 172.21.15.83:0/2305494139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d0102690 0x7fe3d0108f20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009ca0 tx=0x7fe3c802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:13:48.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.210+0000 7fe3d71ee640 1 -- 172.21.15.83:0/2305494139 shutdown_connections 2024-09-16T00:13:48.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.210+0000 7fe3d71ee640 1 --2- 172.21.15.83:0/2305494139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d0102690 0x7fe3d0108f20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:48.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.210+0000 7fe3d71ee640 1 --2- 172.21.15.83:0/2305494139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3d0101cd0 0x7fe3d0102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:48.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.210+0000 7fe3d71ee640 1 --2- 172.21.15.83:0/2305494139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d0100ad0 0x7fe3d0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:48.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.210+0000 7fe3d71ee640 1 -- 172.21.15.83:0/2305494139 >> 172.21.15.83:0/2305494139 conn(0x7fe3d00fc280 msgr2=0x7fe3d00fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:13:48.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.210+0000 7fe3d71ee640 1 -- 172.21.15.83:0/2305494139 shutdown_connections 2024-09-16T00:13:48.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.210+0000 7fe3d71ee640 1 -- 172.21.15.83:0/2305494139 wait complete. 2024-09-16T00:13:48.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d71ee640 1 Processor -- start 2024-09-16T00:13:48.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d71ee640 1 -- start start 2024-09-16T00:13:48.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d71ee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d0100ad0 0x7fe3d006ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:13:48.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d71ee640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3d0101cd0 0x7fe3d019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:13:48.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d71ee640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d0102690 0x7fe3d019f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:13:48.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d71ee640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d010c7d0 con 0x7fe3d0101cd0 2024-09-16T00:13:48.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d71ee640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d010c650 con 0x7fe3d0100ad0 2024-09-16T00:13:48.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d71ee640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d010c950 con 0x7fe3d0102690 2024-09-16T00:13:48.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d578b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d0102690 0x7fe3d019f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:13:48.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d578b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d0102690 0x7fe3d019f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33734/0 (socket says 172.21.15.83:33734) 2024-09-16T00:13:48.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d578b640 1 -- 172.21.15.83:0/443766923 learned_addr learned my addr 172.21.15.83:0/443766923 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-16T00:13:48.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d4f8a640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d0100ad0 0x7fe3d006ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:13:48.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3cffff640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3d0101cd0 0x7fe3d019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:13:48.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d578b640 1 -- 172.21.15.83:0/443766923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d0100ad0 msgr2=0x7fe3d006ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:13:48.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d578b640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d0100ad0 0x7fe3d006ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:48.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d578b640 1 -- 172.21.15.83:0/443766923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3d0101cd0 msgr2=0x7fe3d019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:13:48.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d578b640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3d0101cd0 0x7fe3d019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:13:48.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d578b640 1 -- 172.21.15.83:0/443766923 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3d019fb50 con 0x7fe3d0102690 2024-09-16T00:13:48.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3d4f8a640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d0100ad0 0x7fe3d006ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:13:48.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.211+0000 7fe3cffff640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3d0101cd0 0x7fe3d019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:13:48.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.212+0000 7fe3d578b640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d0102690 0x7fe3d019f290 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8031ae0 tx=0x7fe3c8031ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:13:48.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.212+0000 7fe3cdffb640 1 -- 172.21.15.83:0/443766923 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8003c30 con 0x7fe3d0102690 2024-09-16T00:13:48.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.212+0000 7fe3d71ee640 1 -- 172.21.15.83:0/443766923 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3d019fde0 con 0x7fe3d0102690 2024-09-16T00:13:48.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.212+0000 7fe3cdffb640 1 -- 172.21.15.83:0/443766923 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3c8003dd0 con 0x7fe3d0102690 2024-09-16T00:13:48.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.212+0000 7fe3cdffb640 1 -- 172.21.15.83:0/443766923 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c802fcb0 con 0x7fe3d0102690 2024-09-16T00:13:48.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.212+0000 7fe3d71ee640 1 -- 172.21.15.83:0/443766923 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3d01a0310 con 0x7fe3d0102690 2024-09-16T00:13:48.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.213+0000 7fe3d71ee640 1 -- 172.21.15.83:0/443766923 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3d0107ed0 con 0x7fe3d0102690 2024-09-16T00:13:48.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.214+0000 7fe3cdffb640 1 -- 172.21.15.83:0/443766923 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe3c8004050 con 0x7fe3d0102690 2024-09-16T00:13:48.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.215+0000 7fe3cdffb640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3c00a3240 0x7fe3c00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:13:48.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.215+0000 7fe3cdffb640 1 -- 172.21.15.83:0/443766923 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6727+0+0 (secure 0 0 0) 0x7fe3c80f0650 con 0x7fe3d0102690 2024-09-16T00:13:48.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.215+0000 7fe3d4f8a640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3c00a3240 0x7fe3c00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:13:48.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.216+0000 7fe3d4f8a640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3c00a3240 0x7fe3c00a5700 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe3d006b050 tx=0x7fe3c4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:13:48.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.219+0000 7fe3cdffb640 1 -- 172.21.15.83:0/443766923 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe3c80bd900 con 0x7fe3d0102690 2024-09-16T00:13:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:13:48.400+0000 7fe3d71ee640 1 -- 172.21.15.83:0/443766923 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe3d0062630 con 0x7fe3c00a3240 2024-09-16T00:13:48.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-09-16T00:13:48.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: osdmap e66: 11 total, 10 up, 11 in 2024-09-16T00:13:48.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:48.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:48.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='osd.10 ' entity='osd.10' 2024-09-16T00:13:48.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:48.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:48.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:48.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:48.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:48.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:48.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:48.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:48.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:13:48.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-09-16T00:13:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: osdmap e66: 11 total, 10 up, 11 in 2024-09-16T00:13:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='osd.10 ' entity='osd.10' 2024-09-16T00:13:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:48.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:48.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:48.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:48.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:13:48.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-09-16T00:13:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: osdmap e66: 11 total, 10 up, 11 in 2024-09-16T00:13:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='osd.10 ' entity='osd.10' 2024-09-16T00:13:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:13:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:48.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:13:48.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:13:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:49 smithi083 ceph-mon[35242]: purged_snaps scrub starts 2024-09-16T00:13:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:49 smithi083 ceph-mon[35242]: purged_snaps scrub ok 2024-09-16T00:13:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:49 smithi083 ceph-mon[35242]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:49 smithi083 ceph-mon[35242]: Detected new or changed devices on smithi060 2024-09-16T00:13:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:49 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi060 to 4273M 2024-09-16T00:13:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:49 smithi083 ceph-mon[35242]: osd.10 [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079] boot 2024-09-16T00:13:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:49 smithi083 ceph-mon[35242]: osdmap e67: 11 total, 11 up, 11 in 2024-09-16T00:13:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:49 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:49 smithi083 ceph-mon[35242]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:13:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:49 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:13:49.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:49 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:13:49.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:49 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:49.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:49 smithi060 ceph-mon[35185]: purged_snaps scrub starts 2024-09-16T00:13:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:49 smithi060 ceph-mon[35185]: purged_snaps scrub ok 2024-09-16T00:13:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:49 smithi060 ceph-mon[35185]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:49 smithi060 ceph-mon[35185]: Detected new or changed devices on smithi060 2024-09-16T00:13:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:49 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi060 to 4273M 2024-09-16T00:13:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:49 smithi060 ceph-mon[35185]: osd.10 [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079] boot 2024-09-16T00:13:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:49 smithi060 ceph-mon[35185]: osdmap e67: 11 total, 11 up, 11 in 2024-09-16T00:13:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:49 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:49 smithi060 ceph-mon[35185]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:13:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:49 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:13:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:49 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:13:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:49 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:49 smithi033 ceph-mon[31243]: purged_snaps scrub starts 2024-09-16T00:13:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:49 smithi033 ceph-mon[31243]: purged_snaps scrub ok 2024-09-16T00:13:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:49 smithi033 ceph-mon[31243]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-16T00:13:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:49 smithi033 ceph-mon[31243]: Detected new or changed devices on smithi060 2024-09-16T00:13:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:49 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi060 to 4273M 2024-09-16T00:13:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:49 smithi033 ceph-mon[31243]: osd.10 [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079] boot 2024-09-16T00:13:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:49 smithi033 ceph-mon[31243]: osdmap e67: 11 total, 11 up, 11 in 2024-09-16T00:13:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:49 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T00:13:49.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:49 smithi033 ceph-mon[31243]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:13:49.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:49 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T00:13:49.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:49 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T00:13:49.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:49 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:13:50.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:50 smithi083 ceph-mon[35242]: osdmap e68: 11 total, 11 up, 11 in 2024-09-16T00:13:50.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:50 smithi060 ceph-mon[35185]: osdmap e68: 11 total, 11 up, 11 in 2024-09-16T00:13:50.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:50 smithi033 ceph-mon[31243]: osdmap e68: 11 total, 11 up, 11 in 2024-09-16T00:13:51.437 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:51 smithi083 ceph-mon[35242]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 373 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:13:51.437 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:51 smithi083 ceph-mon[35242]: osdmap e69: 11 total, 11 up, 11 in 2024-09-16T00:13:51.437 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:51 smithi083 ceph-mon[35242]: osdmap e70: 11 total, 11 up, 11 in 2024-09-16T00:13:51.437 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:51 smithi083 ceph-mon[35242]: from='client.? 172.21.15.83:0/3377371723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02536f55-f270-4950-8fb3-0b528f79dac8"}]: dispatch 2024-09-16T00:13:51.437 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:51 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02536f55-f270-4950-8fb3-0b528f79dac8"}]: dispatch 2024-09-16T00:13:51.437 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:51 smithi083 ceph-mon[35242]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02536f55-f270-4950-8fb3-0b528f79dac8"}]': finished 2024-09-16T00:13:51.437 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:51 smithi083 ceph-mon[35242]: osdmap e71: 12 total, 11 up, 12 in 2024-09-16T00:13:51.437 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:13:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:51 smithi060 ceph-mon[35185]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 373 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:13:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:51 smithi060 ceph-mon[35185]: osdmap e69: 11 total, 11 up, 11 in 2024-09-16T00:13:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:51 smithi060 ceph-mon[35185]: osdmap e70: 11 total, 11 up, 11 in 2024-09-16T00:13:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.83:0/3377371723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02536f55-f270-4950-8fb3-0b528f79dac8"}]: dispatch 2024-09-16T00:13:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:51 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02536f55-f270-4950-8fb3-0b528f79dac8"}]: dispatch 2024-09-16T00:13:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:51 smithi060 ceph-mon[35185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02536f55-f270-4950-8fb3-0b528f79dac8"}]': finished 2024-09-16T00:13:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:51 smithi060 ceph-mon[35185]: osdmap e71: 12 total, 11 up, 12 in 2024-09-16T00:13:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:13:51.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:51 smithi033 ceph-mon[31243]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 373 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:13:51.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:51 smithi033 ceph-mon[31243]: osdmap e69: 11 total, 11 up, 11 in 2024-09-16T00:13:51.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:51 smithi033 ceph-mon[31243]: osdmap e70: 11 total, 11 up, 11 in 2024-09-16T00:13:51.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:51 smithi033 ceph-mon[31243]: from='client.? 172.21.15.83:0/3377371723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02536f55-f270-4950-8fb3-0b528f79dac8"}]: dispatch 2024-09-16T00:13:51.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:51 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02536f55-f270-4950-8fb3-0b528f79dac8"}]: dispatch 2024-09-16T00:13:51.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:51 smithi033 ceph-mon[31243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02536f55-f270-4950-8fb3-0b528f79dac8"}]': finished 2024-09-16T00:13:51.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:51 smithi033 ceph-mon[31243]: osdmap e71: 12 total, 11 up, 12 in 2024-09-16T00:13:51.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:13:52.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:52 smithi060 ceph-mon[35185]: from='client.? 172.21.15.83:0/4009949537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:13:52.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:52 smithi033 ceph-mon[31243]: from='client.? 172.21.15.83:0/4009949537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:13:52.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:52 smithi083 ceph-mon[35242]: from='client.? 172.21.15.83:0/4009949537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T00:13:53.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:53 smithi060 ceph-mon[35185]: pgmap v233: 1 pgs: 1 peering; 577 KiB data, 373 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:13:53.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:53 smithi033 ceph-mon[31243]: pgmap v233: 1 pgs: 1 peering; 577 KiB data, 373 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:13:53.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:53 smithi083 ceph-mon[35242]: pgmap v233: 1 pgs: 1 peering; 577 KiB data, 373 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:13:55.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:55 smithi060 ceph-mon[35185]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-09-16T00:13:55.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:55 smithi033 ceph-mon[31243]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-09-16T00:13:55.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:55 smithi083 ceph-mon[35242]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-09-16T00:13:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:57 smithi083 ceph-mon[35242]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T00:13:57.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:57 smithi060 ceph-mon[35185]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T00:13:57.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:57 smithi033 ceph-mon[31243]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T00:13:59.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:13:59 smithi083 ceph-mon[35242]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-16T00:13:59.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:13:59 smithi060 ceph-mon[35185]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-16T00:13:59.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:13:59 smithi033 ceph-mon[31243]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-16T00:14:00.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-16T00:14:00.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:00.531 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-16T00:14:00.531 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-16T00:14:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:01.394 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:01 smithi083 ceph-mon[35242]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-16T00:14:01.395 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:01 smithi083 ceph-mon[35242]: Deploying daemon osd.11 on smithi083 2024-09-16T00:14:01.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:01 smithi060 ceph-mon[35185]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-16T00:14:01.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:01 smithi060 ceph-mon[35185]: Deploying daemon osd.11 on smithi083 2024-09-16T00:14:01.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:01 smithi033 ceph-mon[31243]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-16T00:14:01.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:01 smithi033 ceph-mon[31243]: Deploying daemon osd.11 on smithi083 2024-09-16T00:14:03.397 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:03 smithi083 ceph-mon[35242]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-16T00:14:03.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:03 smithi060 ceph-mon[35185]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-16T00:14:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:03 smithi033 ceph-mon[31243]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-16T00:14:05.474 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:05 smithi083 ceph-mon[35242]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:14:05.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:05 smithi060 ceph-mon[35185]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:14:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:05 smithi033 ceph-mon[31243]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T00:14:06.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:06 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:14:06.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:06 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:06.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:06 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:06.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:06 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:14:06.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:06 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:06.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:06 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:06.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:06 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:14:06.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:06 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:06.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:06 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:07.423 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:07 smithi083 ceph-mon[35242]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:07.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:07 smithi060 ceph-mon[35185]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:07.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:07 smithi033 ceph-mon[31243]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:09.474 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 11 on host 'smithi083' 2024-09-16T00:14:09.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.472+0000 7fe3cdffb640 1 -- 172.21.15.83:0/443766923 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe3d0062630 con 0x7fe3c00a3240 2024-09-16T00:14:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.475+0000 7fe3d71ee640 1 -- 172.21.15.83:0/443766923 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3c00a3240 msgr2=0x7fe3c00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.475+0000 7fe3d71ee640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3c00a3240 0x7fe3c00a5700 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe3d006b050 tx=0x7fe3c4005f90 comp rx=0 tx=0).stop 2024-09-16T00:14:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.475+0000 7fe3d71ee640 1 -- 172.21.15.83:0/443766923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d0102690 msgr2=0x7fe3d019f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.475+0000 7fe3d71ee640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d0102690 0x7fe3d019f290 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8031ae0 tx=0x7fe3c8031ee0 comp rx=0 tx=0).stop 2024-09-16T00:14:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.475+0000 7fe3d71ee640 1 -- 172.21.15.83:0/443766923 shutdown_connections 2024-09-16T00:14:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.475+0000 7fe3d71ee640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3c00a3240 0x7fe3c00a5700 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.475+0000 7fe3d71ee640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d0102690 0x7fe3d019f290 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.475+0000 7fe3d71ee640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3d0101cd0 0x7fe3d019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.475+0000 7fe3d71ee640 1 --2- 172.21.15.83:0/443766923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d0100ad0 0x7fe3d006ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.475+0000 7fe3d71ee640 1 -- 172.21.15.83:0/443766923 >> 172.21.15.83:0/443766923 conn(0x7fe3d00fc280 msgr2=0x7fe3d01050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.475+0000 7fe3d71ee640 1 -- 172.21.15.83:0/443766923 shutdown_connections 2024-09-16T00:14:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:09.475+0000 7fe3d71ee640 1 -- 172.21.15.83:0/443766923 wait complete. 2024-09-16T00:14:09.486 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:09 smithi083 ceph-mon[35242]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:09.486 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:09.486 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:09.486 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:09.486 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:14:09.486 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:09 smithi033 ceph-mon[31243]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:14:09.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:09.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:09 smithi060 ceph-mon[35185]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:09.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:14:09.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:10.108 DEBUG:teuthology.orchestra.run.smithi083:osd.11> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.11.service 2024-09-16T00:14:10.110 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-16T00:14:10.111 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-16T00:14:10.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:10 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:14:10.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:10 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:10.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:10 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:10.381 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:10.503 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:10 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:14:10.503 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:10 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:10.504 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:10 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:10.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:10 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:14:10.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:10 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:10.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:10 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:10.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.897+0000 7fae5723b640 1 -- 172.21.15.33:0/3919086144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae50107940 msgr2=0x7fae50109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:10.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.897+0000 7fae5723b640 1 --2- 172.21.15.33:0/3919086144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae50107940 0x7fae50109d30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fae48009ca0 tx=0x7fae4802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:14:10.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.897+0000 7fae5723b640 1 -- 172.21.15.33:0/3919086144 shutdown_connections 2024-09-16T00:14:10.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.897+0000 7fae5723b640 1 --2- 172.21.15.33:0/3919086144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae50107940 0x7fae50109d30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:10.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.897+0000 7fae5723b640 1 --2- 172.21.15.33:0/3919086144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae50105010 0x7fae50107400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:10.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.897+0000 7fae5723b640 1 --2- 172.21.15.33:0/3919086144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae501026e0 0x7fae50104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:10.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.897+0000 7fae5723b640 1 -- 172.21.15.33:0/3919086144 >> 172.21.15.33:0/3919086144 conn(0x7fae500fc280 msgr2=0x7fae500fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:10.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.898+0000 7fae5723b640 1 -- 172.21.15.33:0/3919086144 shutdown_connections 2024-09-16T00:14:10.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.898+0000 7fae5723b640 1 -- 172.21.15.33:0/3919086144 wait complete. 2024-09-16T00:14:10.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.898+0000 7fae5723b640 1 Processor -- start 2024-09-16T00:14:10.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.898+0000 7fae5723b640 1 -- start start 2024-09-16T00:14:10.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.898+0000 7fae5723b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae501026e0 0x7fae50197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:10.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.898+0000 7fae5723b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae50105010 0x7fae50198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:10.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.898+0000 7fae5723b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae50107940 0x7fae5019f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:10.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.898+0000 7fae5723b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae5010d880 con 0x7fae501026e0 2024-09-16T00:14:10.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.898+0000 7fae5723b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fae5010d700 con 0x7fae50107940 2024-09-16T00:14:10.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.898+0000 7fae5723b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae5010da00 con 0x7fae50105010 2024-09-16T00:14:10.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.898+0000 7fae54fd7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae501026e0 0x7fae50197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:10.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae54fd7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae501026e0 0x7fae50197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39314/0 (socket says 172.21.15.33:39314) 2024-09-16T00:14:10.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae54fd7640 1 -- 172.21.15.33:0/602142438 learned_addr learned my addr 172.21.15.33:0/602142438 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:10.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae4ffff640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae50105010 0x7fae50198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:10.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae557d8640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae50107940 0x7fae5019f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:10.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae54fd7640 1 -- 172.21.15.33:0/602142438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae50105010 msgr2=0x7fae50198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:10.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae54fd7640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae50105010 0x7fae50198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:10.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae54fd7640 1 -- 172.21.15.33:0/602142438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae50107940 msgr2=0x7fae5019f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:10.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae54fd7640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae50107940 0x7fae5019f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:10.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae54fd7640 1 -- 172.21.15.33:0/602142438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5019fb00 con 0x7fae501026e0 2024-09-16T00:14:10.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae54fd7640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae501026e0 0x7fae50197ed0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fae4000d770 tx=0x7fae4000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:10.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae4dffb640 1 -- 172.21.15.33:0/602142438 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae40017070 con 0x7fae501026e0 2024-09-16T00:14:10.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae4dffb640 1 -- 172.21.15.33:0/602142438 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae40005500 con 0x7fae501026e0 2024-09-16T00:14:10.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae5723b640 1 -- 172.21.15.33:0/602142438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae5019fdf0 con 0x7fae501026e0 2024-09-16T00:14:10.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae4dffb640 1 -- 172.21.15.33:0/602142438 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae40004e00 con 0x7fae501026e0 2024-09-16T00:14:10.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.899+0000 7fae5723b640 1 -- 172.21.15.33:0/602142438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae501a01f0 con 0x7fae501026e0 2024-09-16T00:14:10.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.901+0000 7fae5723b640 1 -- 172.21.15.33:0/602142438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae500fdd70 con 0x7fae501026e0 2024-09-16T00:14:10.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.902+0000 7fae4dffb640 1 -- 172.21.15.33:0/602142438 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fae40004fa0 con 0x7fae501026e0 2024-09-16T00:14:10.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.902+0000 7fae4dffb640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae380a3270 0x7fae380a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:10.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.903+0000 7fae4ffff640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae380a3270 0x7fae380a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:10.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.903+0000 7fae4dffb640 1 -- 172.21.15.33:0/602142438 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6809+0+0 (secure 0 0 0) 0x7fae400cf140 con 0x7fae501026e0 2024-09-16T00:14:10.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.903+0000 7fae4ffff640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae380a3270 0x7fae380a5730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fae501993f0 tx=0x7fae4400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:10.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:10.910+0000 7fae4dffb640 1 -- 172.21.15.33:0/602142438 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fae4009c440 con 0x7fae501026e0 2024-09-16T00:14:11.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.088+0000 7fae5723b640 1 -- 172.21.15.33:0/602142438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fae5002da30 con 0x7fae501026e0 2024-09-16T00:14:11.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.089+0000 7fae4dffb640 1 -- 172.21.15.33:0/602142438 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v71) ==== 74+0+133 (secure 0 0 0) 0x7fae4009bd40 con 0x7fae501026e0 2024-09-16T00:14:11.091 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.091+0000 7fae5723b640 1 -- 172.21.15.33:0/602142438 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae380a3270 msgr2=0x7fae380a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.091+0000 7fae5723b640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae380a3270 0x7fae380a5730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fae501993f0 tx=0x7fae4400a2d0 comp rx=0 tx=0).stop 2024-09-16T00:14:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.091+0000 7fae5723b640 1 -- 172.21.15.33:0/602142438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae501026e0 msgr2=0x7fae50197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.091+0000 7fae5723b640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae501026e0 0x7fae50197ed0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fae4000d770 tx=0x7fae4000dc30 comp rx=0 tx=0).stop 2024-09-16T00:14:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.091+0000 7fae5723b640 1 -- 172.21.15.33:0/602142438 shutdown_connections 2024-09-16T00:14:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.091+0000 7fae5723b640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae380a3270 0x7fae380a5730 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.091+0000 7fae5723b640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae50107940 0x7fae5019f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.091+0000 7fae5723b640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae50105010 0x7fae50198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.091+0000 7fae5723b640 1 --2- 172.21.15.33:0/602142438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae501026e0 0x7fae50197ed0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.091+0000 7fae5723b640 1 -- 172.21.15.33:0/602142438 >> 172.21.15.33:0/602142438 conn(0x7fae500fc280 msgr2=0x7fae50102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.091+0000 7fae5723b640 1 -- 172.21.15.33:0/602142438 shutdown_connections 2024-09-16T00:14:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:11.091+0000 7fae5723b640 1 -- 172.21.15.33:0/602142438 wait complete. 2024-09-16T00:14:11.725 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:11 smithi083 ceph-mon[35242]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:11.725 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:11 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/602142438' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T00:14:11.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:11 smithi060 ceph-mon[35185]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:11.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:11 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/602142438' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T00:14:11.820 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:11 smithi033 ceph-mon[31243]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:11.821 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:11 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/602142438' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T00:14:11.849 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":71,"num_osds":12,"num_up_osds":11,"osd_up_since":1726445628,"num_in_osds":12,"osd_in_since":1726445631,"num_remapped_pgs":0} 2024-09-16T00:14:12.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:14:11 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:14:11.721+0000 7f862551f640 -1 osd.11 0 log_to_monitors true 2024-09-16T00:14:12.634 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:12 smithi083 ceph-mon[35242]: from='osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T00:14:12.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:12 smithi060 ceph-mon[35185]: from='osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T00:14:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:12 smithi033 ceph-mon[31243]: from='osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T00:14:12.850 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-16T00:14:13.124 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:13.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:13.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-16T00:14:13.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: osdmap e72: 12 total, 11 up, 12 in 2024-09-16T00:14:13.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:14:13.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:14:13.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:13.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:13.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:14:13.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:14:13.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:14:13.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:14:13.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:13.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:13.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:14:13.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:13.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.591+0000 7f5cea8fd640 1 -- 172.21.15.33:0/3626523993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ce41078c0 msgr2=0x7f5ce4109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:13.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.591+0000 7f5cea8fd640 1 --2- 172.21.15.33:0/3626523993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ce41078c0 0x7f5ce4109d10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009ca0 tx=0x7f5cdc02a8c0 comp rx=0 tx=0).stop 2024-09-16T00:14:13.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.591+0000 7f5cea8fd640 1 -- 172.21.15.33:0/3626523993 shutdown_connections 2024-09-16T00:14:13.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.591+0000 7f5cea8fd640 1 --2- 172.21.15.33:0/3626523993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ce41078c0 0x7f5ce4109d10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:13.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.591+0000 7f5cea8fd640 1 --2- 172.21.15.33:0/3626523993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ce4104f90 0x7f5ce4107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:13.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.591+0000 7f5cea8fd640 1 --2- 172.21.15.33:0/3626523993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ce4102660 0x7f5ce4104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:13.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.591+0000 7f5cea8fd640 1 -- 172.21.15.33:0/3626523993 >> 172.21.15.33:0/3626523993 conn(0x7f5ce40fc280 msgr2=0x7f5ce40fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:13.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.591+0000 7f5cea8fd640 1 -- 172.21.15.33:0/3626523993 shutdown_connections 2024-09-16T00:14:13.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.591+0000 7f5cea8fd640 1 -- 172.21.15.33:0/3626523993 wait complete. 2024-09-16T00:14:13.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.591+0000 7f5cea8fd640 1 Processor -- start 2024-09-16T00:14:13.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.591+0000 7f5cea8fd640 1 -- start start 2024-09-16T00:14:13.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5cea8fd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ce4102660 0x7f5ce4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:13.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5cea8fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ce4104f90 0x7f5ce41984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:13.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5cea8fd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ce41078c0 0x7f5ce419f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:13.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5cea8fd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce410d900 con 0x7f5ce4102660 2024-09-16T00:14:13.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5cea8fd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce410d780 con 0x7f5ce4104f90 2024-09-16T00:14:13.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5cea8fd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce410da80 con 0x7f5ce41078c0 2024-09-16T00:14:13.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ce4102660 0x7f5ce4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:13.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ce4102660 0x7f5ce4197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39338/0 (socket says 172.21.15.33:39338) 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce3fff640 1 -- 172.21.15.33:0/1958435216 learned_addr learned my addr 172.21.15.33:0/1958435216 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce8e9a640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ce41078c0 0x7f5ce419f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce3fff640 1 -- 172.21.15.33:0/1958435216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ce41078c0 msgr2=0x7f5ce419f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5cdbfff640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ce4104f90 0x7f5ce41984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce3fff640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ce41078c0 0x7f5ce419f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce3fff640 1 -- 172.21.15.33:0/1958435216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ce4104f90 msgr2=0x7f5ce41984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce3fff640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ce4104f90 0x7f5ce41984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce3fff640 1 -- 172.21.15.33:0/1958435216 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce419fb90 con 0x7f5ce4102660 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce8e9a640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ce41078c0 0x7f5ce419f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5cdbfff640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ce4104f90 0x7f5ce41984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce3fff640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ce4102660 0x7f5ce4197f60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400b4a0 tx=0x7f5cd400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce1ffb640 1 -- 172.21.15.33:0/1958435216 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4014070 con 0x7f5ce4102660 2024-09-16T00:14:13.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5cea8fd640 1 -- 172.21.15.33:0/1958435216 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce419fe80 con 0x7f5ce4102660 2024-09-16T00:14:13.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce1ffb640 1 -- 172.21.15.33:0/1958435216 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cd4010af0 con 0x7f5ce4102660 2024-09-16T00:14:13.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.592+0000 7f5ce1ffb640 1 -- 172.21.15.33:0/1958435216 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4017d10 con 0x7f5ce4102660 2024-09-16T00:14:13.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.593+0000 7f5cea8fd640 1 -- 172.21.15.33:0/1958435216 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce41a03b0 con 0x7f5ce4102660 2024-09-16T00:14:13.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.593+0000 7f5cea8fd640 1 -- 172.21.15.33:0/1958435216 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce40fdd70 con 0x7f5ce4102660 2024-09-16T00:14:13.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.595+0000 7f5ce1ffb640 1 -- 172.21.15.33:0/1958435216 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f5cd4017070 con 0x7f5ce4102660 2024-09-16T00:14:13.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.595+0000 7f5ce1ffb640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5cd00a3270 0x7f5cd00a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:13.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.595+0000 7f5cdbfff640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5cd00a3270 0x7f5cd00a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:13.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.595+0000 7f5ce1ffb640 1 -- 172.21.15.33:0/1958435216 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6847+0+0 (secure 0 0 0) 0x7f5cd40ceb20 con 0x7f5ce4102660 2024-09-16T00:14:13.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.596+0000 7f5cdbfff640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5cd00a3270 0x7f5cd00a5730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4199480 tx=0x7f5ccc005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:13.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.598+0000 7f5ce1ffb640 1 -- 172.21.15.33:0/1958435216 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5cd40cee20 con 0x7f5ce4102660 2024-09-16T00:14:13.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.773+0000 7f5cea8fd640 1 -- 172.21.15.33:0/1958435216 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f5ce402d9c0 con 0x7f5ce4102660 2024-09-16T00:14:13.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.774+0000 7f5ce1ffb640 1 -- 172.21.15.33:0/1958435216 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) ==== 74+0+133 (secure 0 0 0) 0x7f5cd409be00 con 0x7f5ce4102660 2024-09-16T00:14:13.775 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:13.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.776+0000 7f5cea8fd640 1 -- 172.21.15.33:0/1958435216 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5cd00a3270 msgr2=0x7f5cd00a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:13.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.776+0000 7f5cea8fd640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5cd00a3270 0x7f5cd00a5730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4199480 tx=0x7f5ccc005e50 comp rx=0 tx=0).stop 2024-09-16T00:14:13.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.776+0000 7f5cea8fd640 1 -- 172.21.15.33:0/1958435216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ce4102660 msgr2=0x7f5ce4197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.776+0000 7f5cea8fd640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ce4102660 0x7f5ce4197f60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400b4a0 tx=0x7f5cd400b960 comp rx=0 tx=0).stop 2024-09-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.776+0000 7f5cea8fd640 1 -- 172.21.15.33:0/1958435216 shutdown_connections 2024-09-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.776+0000 7f5cea8fd640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5cd00a3270 0x7f5cd00a5730 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.776+0000 7f5cea8fd640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ce41078c0 0x7f5ce419f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.776+0000 7f5cea8fd640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ce4104f90 0x7f5ce41984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.776+0000 7f5cea8fd640 1 --2- 172.21.15.33:0/1958435216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ce4102660 0x7f5ce4197f60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.776+0000 7f5cea8fd640 1 -- 172.21.15.33:0/1958435216 >> 172.21.15.33:0/1958435216 conn(0x7f5ce40fc280 msgr2=0x7f5ce4102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.776+0000 7f5cea8fd640 1 -- 172.21.15.33:0/1958435216 shutdown_connections 2024-09-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:13.776+0000 7f5cea8fd640 1 -- 172.21.15.33:0/1958435216 wait complete. 2024-09-16T00:14:13.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:14:13.483+0000 7f86214a2640 -1 osd.11 0 waiting for initial osdmap 2024-09-16T00:14:13.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:14:13.500+0000 7f861d2ca640 -1 osd.11 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T00:14:13.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:13.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-16T00:14:13.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: osdmap e72: 12 total, 11 up, 12 in 2024-09-16T00:14:13.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:14:13.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:14:13.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:13.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:13.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:14:13.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:14:13.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:14:13.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:14:13.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:13.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:13.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:14:13.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:13.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:13.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-16T00:14:13.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: osdmap e72: 12 total, 11 up, 12 in 2024-09-16T00:14:13.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-16T00:14:13.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:14:13.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:13.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:13.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:14:13.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:14:13.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:14:13.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:14:13.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:13.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:13.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:14:13.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:14.596 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":73,"num_osds":12,"num_up_osds":11,"osd_up_since":1726445628,"num_in_osds":12,"osd_in_since":1726445631,"num_remapped_pgs":0} 2024-09-16T00:14:14.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:14 smithi033 ceph-mon[31243]: Detected new or changed devices on smithi083 2024-09-16T00:14:14.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:14 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi083 to 4272M 2024-09-16T00:14:14.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:14 smithi033 ceph-mon[31243]: from='osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-16T00:14:14.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:14 smithi033 ceph-mon[31243]: osdmap e73: 12 total, 11 up, 12 in 2024-09-16T00:14:14.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:14:14.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:14:14.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:14 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1958435216' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T00:14:14.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:14 smithi083 ceph-mon[35242]: Detected new or changed devices on smithi083 2024-09-16T00:14:14.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:14 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi083 to 4272M 2024-09-16T00:14:14.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:14 smithi083 ceph-mon[35242]: from='osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-16T00:14:14.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:14 smithi083 ceph-mon[35242]: osdmap e73: 12 total, 11 up, 12 in 2024-09-16T00:14:14.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:14:14.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:14:14.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:14 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1958435216' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T00:14:14.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:14 smithi060 ceph-mon[35185]: Detected new or changed devices on smithi083 2024-09-16T00:14:14.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:14 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi083 to 4272M 2024-09-16T00:14:14.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:14 smithi060 ceph-mon[35185]: from='osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-16T00:14:14.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:14 smithi060 ceph-mon[35185]: osdmap e73: 12 total, 11 up, 12 in 2024-09-16T00:14:14.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:14:14.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:14:14.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:14 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1958435216' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T00:14:15.597 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-16T00:14:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:15 smithi033 ceph-mon[31243]: purged_snaps scrub starts 2024-09-16T00:14:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:15 smithi033 ceph-mon[31243]: purged_snaps scrub ok 2024-09-16T00:14:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:15 smithi033 ceph-mon[31243]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:15 smithi033 ceph-mon[31243]: osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075] boot 2024-09-16T00:14:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:15 smithi033 ceph-mon[31243]: osdmap e74: 12 total, 12 up, 12 in 2024-09-16T00:14:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:14:15.867 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:15 smithi083 ceph-mon[35242]: purged_snaps scrub starts 2024-09-16T00:14:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:15 smithi083 ceph-mon[35242]: purged_snaps scrub ok 2024-09-16T00:14:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:15 smithi083 ceph-mon[35242]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:15 smithi083 ceph-mon[35242]: osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075] boot 2024-09-16T00:14:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:15 smithi083 ceph-mon[35242]: osdmap e74: 12 total, 12 up, 12 in 2024-09-16T00:14:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:14:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:15 smithi060 ceph-mon[35185]: purged_snaps scrub starts 2024-09-16T00:14:15.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:15 smithi060 ceph-mon[35185]: purged_snaps scrub ok 2024-09-16T00:14:15.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:15 smithi060 ceph-mon[35185]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-16T00:14:15.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:15 smithi060 ceph-mon[35185]: osd.11 [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075] boot 2024-09-16T00:14:15.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:15 smithi060 ceph-mon[35185]: osdmap e74: 12 total, 12 up, 12 in 2024-09-16T00:14:15.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T00:14:16.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.348+0000 7f1579b80640 1 -- 172.21.15.33:0/2757431138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1574101cf0 msgr2=0x7f1574102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:16.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.348+0000 7f1579b80640 1 --2- 172.21.15.33:0/2757431138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1574101cf0 0x7f1574102170 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f155c009900 tx=0x7f155c02a410 comp rx=0 tx=0).stop 2024-09-16T00:14:16.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.348+0000 7f1579b80640 1 -- 172.21.15.33:0/2757431138 shutdown_connections 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.348+0000 7f1579b80640 1 --2- 172.21.15.33:0/2757431138 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15741026b0 0x7f1574108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.348+0000 7f1579b80640 1 --2- 172.21.15.33:0/2757431138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1574101cf0 0x7f1574102170 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.348+0000 7f1579b80640 1 --2- 172.21.15.33:0/2757431138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1574100af0 0x7f1574100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.348+0000 7f1579b80640 1 -- 172.21.15.33:0/2757431138 >> 172.21.15.33:0/2757431138 conn(0x7f15740fc2a0 msgr2=0x7f15740fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.348+0000 7f1579b80640 1 -- 172.21.15.33:0/2757431138 shutdown_connections 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.349+0000 7f1579b80640 1 -- 172.21.15.33:0/2757431138 wait complete. 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.349+0000 7f1579b80640 1 Processor -- start 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.349+0000 7f1579b80640 1 -- start start 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.349+0000 7f1579b80640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1574100af0 0x7f15741980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.349+0000 7f1579b80640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1574101cf0 0x7f1574198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.349+0000 7f1579b80640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15741026b0 0x7f157419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.349+0000 7f1579b80640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f157410c8c0 con 0x7f1574100af0 2024-09-16T00:14:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.349+0000 7f1579b80640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f157410c740 con 0x7f15741026b0 2024-09-16T00:14:16.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.349+0000 7f1579b80640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f157410ca40 con 0x7f1574101cf0 2024-09-16T00:14:16.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f15737fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1574100af0 0x7f15741980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:16.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f15737fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1574100af0 0x7f15741980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39360/0 (socket says 172.21.15.33:39360) 2024-09-16T00:14:16.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f15737fe640 1 -- 172.21.15.33:0/1939293505 learned_addr learned my addr 172.21.15.33:0/1939293505 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:16.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f1573fff640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15741026b0 0x7f157419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:16.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f156affd640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1574101cf0 0x7f1574198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:16.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f15737fe640 1 -- 172.21.15.33:0/1939293505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1574101cf0 msgr2=0x7f1574198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:16.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f15737fe640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1574101cf0 0x7f1574198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:16.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f15737fe640 1 -- 172.21.15.33:0/1939293505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15741026b0 msgr2=0x7f157419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:16.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f15737fe640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15741026b0 0x7f157419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:16.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f15737fe640 1 -- 172.21.15.33:0/1939293505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157419fd00 con 0x7f1574100af0 2024-09-16T00:14:16.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f15737fe640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1574100af0 0x7f15741980d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f156400b450 tx=0x7f156400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:16.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f15717fa640 1 -- 172.21.15.33:0/1939293505 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1564015020 con 0x7f1574100af0 2024-09-16T00:14:16.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f15717fa640 1 -- 172.21.15.33:0/1939293505 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1564011d70 con 0x7f1574100af0 2024-09-16T00:14:16.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f1579b80640 1 -- 172.21.15.33:0/1939293505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f157419fff0 con 0x7f1574100af0 2024-09-16T00:14:16.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f15717fa640 1 -- 172.21.15.33:0/1939293505 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156400f340 con 0x7f1574100af0 2024-09-16T00:14:16.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.350+0000 7f1579b80640 1 -- 172.21.15.33:0/1939293505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15741a0520 con 0x7f1574100af0 2024-09-16T00:14:16.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.351+0000 7f1579b80640 1 -- 172.21.15.33:0/1939293505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15741a07e0 con 0x7f1574100af0 2024-09-16T00:14:16.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.353+0000 7f15717fa640 1 -- 172.21.15.33:0/1939293505 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f1564018020 con 0x7f1574100af0 2024-09-16T00:14:16.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.354+0000 7f15717fa640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15600a32f0 0x7f15600a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:16.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.354+0000 7f156affd640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15600a32f0 0x7f15600a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:16.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.354+0000 7f15717fa640 1 -- 172.21.15.33:0/1939293505 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f15640cf3e0 con 0x7f1574100af0 2024-09-16T00:14:16.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.355+0000 7f156affd640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15600a32f0 0x7f15600a57b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f155c031610 tx=0x7f155c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:16.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.357+0000 7f15717fa640 1 -- 172.21.15.33:0/1939293505 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f156401f020 con 0x7f1574100af0 2024-09-16T00:14:16.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.533+0000 7f1579b80640 1 -- 172.21.15.33:0/1939293505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f15741a09b0 con 0x7f1574100af0 2024-09-16T00:14:16.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:16 smithi033 ceph-mon[31243]: osdmap e75: 12 total, 12 up, 12 in 2024-09-16T00:14:16.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.533+0000 7f15717fa640 1 -- 172.21.15.33:0/1939293505 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) ==== 74+0+133 (secure 0 0 0) 0x7f156400d040 con 0x7f1574100af0 2024-09-16T00:14:16.534 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:16.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.535+0000 7f1579b80640 1 -- 172.21.15.33:0/1939293505 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15600a32f0 msgr2=0x7f15600a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:16.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.535+0000 7f1579b80640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15600a32f0 0x7f15600a57b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f155c031610 tx=0x7f155c005fd0 comp rx=0 tx=0).stop 2024-09-16T00:14:16.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.535+0000 7f1579b80640 1 -- 172.21.15.33:0/1939293505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1574100af0 msgr2=0x7f15741980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:16.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.535+0000 7f1579b80640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1574100af0 0x7f15741980d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f156400b450 tx=0x7f156400b910 comp rx=0 tx=0).stop 2024-09-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.536+0000 7f1579b80640 1 -- 172.21.15.33:0/1939293505 shutdown_connections 2024-09-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.536+0000 7f1579b80640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15600a32f0 0x7f15600a57b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.536+0000 7f1579b80640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15741026b0 0x7f157419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.536+0000 7f1579b80640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1574101cf0 0x7f1574198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.536+0000 7f1579b80640 1 --2- 172.21.15.33:0/1939293505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1574100af0 0x7f15741980d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.536+0000 7f1579b80640 1 -- 172.21.15.33:0/1939293505 >> 172.21.15.33:0/1939293505 conn(0x7f15740fc2a0 msgr2=0x7f157406a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.536+0000 7f1579b80640 1 -- 172.21.15.33:0/1939293505 shutdown_connections 2024-09-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:16.536+0000 7f1579b80640 1 -- 172.21.15.33:0/1939293505 wait complete. 2024-09-16T00:14:16.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:16 smithi083 ceph-mon[35242]: osdmap e75: 12 total, 12 up, 12 in 2024-09-16T00:14:16.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:16 smithi060 ceph-mon[35185]: osdmap e75: 12 total, 12 up, 12 in 2024-09-16T00:14:17.296 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":75,"num_osds":12,"num_up_osds":12,"osd_up_since":1726445654,"num_in_osds":12,"osd_in_since":1726445631,"num_remapped_pgs":0} 2024-09-16T00:14:17.296 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:14:17.550 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:17.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:17 smithi033 ceph-mon[31243]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:17.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:17 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1939293505' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T00:14:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:17 smithi083 ceph-mon[35242]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:17 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1939293505' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T00:14:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:17 smithi060 ceph-mon[35185]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:17 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1939293505' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T00:14:18.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.026+0000 7fbd5c388640 1 -- 172.21.15.33:0/3387889204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54102ef0 msgr2=0x7fbd5410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.026+0000 7fbd5c388640 1 --2- 172.21.15.33:0/3387889204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54102ef0 0x7fbd5410da40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009ca0 tx=0x7fbd4c02a950 comp rx=0 tx=0).stop 2024-09-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.026+0000 7fbd5c388640 1 -- 172.21.15.33:0/3387889204 shutdown_connections 2024-09-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.026+0000 7fbd5c388640 1 --2- 172.21.15.33:0/3387889204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54102ef0 0x7fbd5410da40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.026+0000 7fbd5c388640 1 --2- 172.21.15.33:0/3387889204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd54102570 0x7fbd541029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.026+0000 7fbd5c388640 1 --2- 172.21.15.33:0/3387889204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd54100e20 0x7fbd54101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.026+0000 7fbd5c388640 1 -- 172.21.15.33:0/3387889204 >> 172.21.15.33:0/3387889204 conn(0x7fbd540fc7d0 msgr2=0x7fbd540febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.026+0000 7fbd5c388640 1 -- 172.21.15.33:0/3387889204 shutdown_connections 2024-09-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.026+0000 7fbd5c388640 1 -- 172.21.15.33:0/3387889204 wait complete. 2024-09-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd5c388640 1 Processor -- start 2024-09-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd5c388640 1 -- start start 2024-09-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd5c388640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd54100e20 0x7fbd54197400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd5c388640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54102570 0x7fbd54197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd5c388640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd54102ef0 0x7fbd5419bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd5c388640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbd54110f80 con 0x7fbd54102570 2024-09-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd5c388640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbd54110e00 con 0x7fbd54100e20 2024-09-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd5c388640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbd54111100 con 0x7fbd54102ef0 2024-09-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd59923640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54102570 0x7fbd54197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd59923640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54102570 0x7fbd54197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39386/0 (socket says 172.21.15.33:39386) 2024-09-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd59923640 1 -- 172.21.15.33:0/2181548581 learned_addr learned my addr 172.21.15.33:0/2181548581 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd5a925640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd54102ef0 0x7fbd5419bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd5a124640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd54100e20 0x7fbd54197400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd59923640 1 -- 172.21.15.33:0/2181548581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd54102ef0 msgr2=0x7fbd5419bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd59923640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd54102ef0 0x7fbd5419bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd59923640 1 -- 172.21.15.33:0/2181548581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd54100e20 msgr2=0x7fbd54197400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd59923640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd54100e20 0x7fbd54197400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd59923640 1 -- 172.21.15.33:0/2181548581 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd5419c3b0 con 0x7fbd54102570 2024-09-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.027+0000 7fbd5a124640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd54100e20 0x7fbd54197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.028+0000 7fbd59923640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54102570 0x7fbd54197940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400c970 tx=0x7fbd4400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.028+0000 7fbd4b7fe640 1 -- 172.21.15.33:0/2181548581 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44007bb0 con 0x7fbd54102570 2024-09-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.028+0000 7fbd4b7fe640 1 -- 172.21.15.33:0/2181548581 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd44007d50 con 0x7fbd54102570 2024-09-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.028+0000 7fbd5c388640 1 -- 172.21.15.33:0/2181548581 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd5419c6a0 con 0x7fbd54102570 2024-09-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.028+0000 7fbd4b7fe640 1 -- 172.21.15.33:0/2181548581 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4400f450 con 0x7fbd54102570 2024-09-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.028+0000 7fbd5c388640 1 -- 172.21.15.33:0/2181548581 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd541a3f80 con 0x7fbd54102570 2024-09-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.029+0000 7fbd5c388640 1 -- 172.21.15.33:0/2181548581 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd54101200 con 0x7fbd54102570 2024-09-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.030+0000 7fbd4b7fe640 1 -- 172.21.15.33:0/2181548581 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fbd44005020 con 0x7fbd54102570 2024-09-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.031+0000 7fbd4b7fe640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbd400a32c0 0x7fbd400a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.031+0000 7fbd5a124640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbd400a32c0 0x7fbd400a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.031+0000 7fbd4b7fe640 1 -- 172.21.15.33:0/2181548581 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fbd440cf280 con 0x7fbd54102570 2024-09-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.031+0000 7fbd5a124640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbd400a32c0 0x7fbd400a5780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbd500058c0 tx=0x7fbd50005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.035+0000 7fbd4b7fe640 1 -- 172.21.15.33:0/2181548581 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbd44014030 con 0x7fbd54102570 2024-09-16T00:14:18.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.210+0000 7fbd5c388640 1 -- 172.21.15.33:0/2181548581 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd541a4150 con 0x7fbd54102570 2024-09-16T00:14:18.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.212+0000 7fbd4b7fe640 1 -- 172.21.15.33:0/2181548581 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+18394 (secure 0 0 0) 0x7fbd4409c440 con 0x7fbd54102570 2024-09-16T00:14:18.213 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:18.213 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":75,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:14:15.491804+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:14:18.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.214+0000 7fbd5c388640 1 -- 172.21.15.33:0/2181548581 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbd400a32c0 msgr2=0x7fbd400a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:18.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.214+0000 7fbd5c388640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbd400a32c0 0x7fbd400a5780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbd500058c0 tx=0x7fbd50005d30 comp rx=0 tx=0).stop 2024-09-16T00:14:18.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.214+0000 7fbd5c388640 1 -- 172.21.15.33:0/2181548581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54102570 msgr2=0x7fbd54197940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:18.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.214+0000 7fbd5c388640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54102570 0x7fbd54197940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400c970 tx=0x7fbd4400ce30 comp rx=0 tx=0).stop 2024-09-16T00:14:18.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.214+0000 7fbd5c388640 1 -- 172.21.15.33:0/2181548581 shutdown_connections 2024-09-16T00:14:18.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.214+0000 7fbd5c388640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbd400a32c0 0x7fbd400a5780 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:18.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.214+0000 7fbd5c388640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd54102ef0 0x7fbd5419bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:18.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.214+0000 7fbd5c388640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54102570 0x7fbd54197940 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:18.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.214+0000 7fbd5c388640 1 --2- 172.21.15.33:0/2181548581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd54100e20 0x7fbd54197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:18.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.214+0000 7fbd5c388640 1 -- 172.21.15.33:0/2181548581 >> 172.21.15.33:0/2181548581 conn(0x7fbd540fc7d0 msgr2=0x7fbd54106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:18.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.214+0000 7fbd5c388640 1 -- 172.21.15.33:0/2181548581 shutdown_connections 2024-09-16T00:14:18.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:18.214+0000 7fbd5c388640 1 -- 172.21.15.33:0/2181548581 wait complete. 2024-09-16T00:14:18.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:18 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2181548581' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:14:18.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:18 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2181548581' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:14:18.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:18 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2181548581' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:14:19.020 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T00:10:24.062476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T00:14:19.021 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-16T00:14:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:19.697 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:19 smithi033 ceph-mon[31243]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:19.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.781+0000 7f432cafb640 1 -- 172.21.15.33:0/3407664281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4328102d40 msgr2=0x7f4328103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.781+0000 7f432cafb640 1 --2- 172.21.15.33:0/3407664281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4328102d40 0x7f4328103120 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4318009ca0 tx=0x7f431802a950 comp rx=0 tx=0).stop 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.781+0000 7f432cafb640 1 -- 172.21.15.33:0/3407664281 shutdown_connections 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.781+0000 7f432cafb640 1 --2- 172.21.15.33:0/3407664281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f432810c170 0x7f432810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.781+0000 7f432cafb640 1 --2- 172.21.15.33:0/3407664281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4328103660 0x7f432810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.781+0000 7f432cafb640 1 --2- 172.21.15.33:0/3407664281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4328102d40 0x7f4328103120 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.781+0000 7f432cafb640 1 -- 172.21.15.33:0/3407664281 >> 172.21.15.33:0/3407664281 conn(0x7f43280fc7a0 msgr2=0x7f43280febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.782+0000 7f432cafb640 1 -- 172.21.15.33:0/3407664281 shutdown_connections 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.782+0000 7f432cafb640 1 -- 172.21.15.33:0/3407664281 wait complete. 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.782+0000 7f432cafb640 1 Processor -- start 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.782+0000 7f432cafb640 1 -- start start 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.782+0000 7f432cafb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4328102d40 0x7f43281984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.782+0000 7f432cafb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4328103660 0x7f4328198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.782+0000 7f432cafb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f432810c170 0x7f432819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:19.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.782+0000 7f432cafb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4328111f70 con 0x7f432810c170 2024-09-16T00:14:19.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f432cafb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4328111df0 con 0x7f4328102d40 2024-09-16T00:14:19.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f432cafb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f43281120f0 con 0x7f4328103660 2024-09-16T00:14:19.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f4326d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f432810c170 0x7f432819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:19.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f4326d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f432810c170 0x7f432819cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39408/0 (socket says 172.21.15.33:39408) 2024-09-16T00:14:19.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f4326d9d640 1 -- 172.21.15.33:0/2201143618 learned_addr learned my addr 172.21.15.33:0/2201143618 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:19.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f4325d9b640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4328103660 0x7f4328198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:19.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f432659c640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4328102d40 0x7f43281984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:19.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f4326d9d640 1 -- 172.21.15.33:0/2201143618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4328103660 msgr2=0x7f4328198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:19.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f4326d9d640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4328103660 0x7f4328198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f4326d9d640 1 -- 172.21.15.33:0/2201143618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4328102d40 msgr2=0x7f43281984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f4326d9d640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4328102d40 0x7f43281984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f4326d9d640 1 -- 172.21.15.33:0/2201143618 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432819d4a0 con 0x7f432810c170 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f4326d9d640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f432810c170 0x7f432819cdc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f432000d950 tx=0x7f432000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f43177fe640 1 -- 172.21.15.33:0/2201143618 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320017030 con 0x7f432810c170 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f43177fe640 1 -- 172.21.15.33:0/2201143618 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4320004020 con 0x7f432810c170 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f43177fe640 1 -- 172.21.15.33:0/2201143618 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320003c00 con 0x7f432810c170 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f432cafb640 1 -- 172.21.15.33:0/2201143618 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f432819d790 con 0x7f432810c170 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.783+0000 7f432cafb640 1 -- 172.21.15.33:0/2201143618 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43281a5070 con 0x7f432810c170 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.784+0000 7f432cafb640 1 -- 172.21.15.33:0/2201143618 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4328103120 con 0x7f432810c170 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.785+0000 7f43177fe640 1 -- 172.21.15.33:0/2201143618 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f432001a070 con 0x7f432810c170 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.786+0000 7f43177fe640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f43100a3240 0x7f43100a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:19.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.787+0000 7f432659c640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f43100a3240 0x7f43100a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:19.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.787+0000 7f43177fe640 1 -- 172.21.15.33:0/2201143618 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f43200cecb0 con 0x7f432810c170 2024-09-16T00:14:19.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.787+0000 7f432659c640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f43100a3240 0x7f43100a5700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4318004ad0 tx=0x7f4318004a60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:19.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.791+0000 7f43177fe640 1 -- 172.21.15.33:0/2201143618 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4320021070 con 0x7f432810c170 2024-09-16T00:14:19.881 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:19 smithi060 ceph-mon[35185]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:19 smithi083 ceph-mon[35242]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:19.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.966+0000 7f432cafb640 1 -- 172.21.15.33:0/2201143618 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4328106c50 con 0x7f432810c170 2024-09-16T00:14:19.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.967+0000 7f43177fe640 1 -- 172.21.15.33:0/2201143618 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) ==== 93+0+10 (secure 0 0 0) 0x7f432009bdd0 con 0x7f432810c170 2024-09-16T00:14:19.967 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-09-16T00:14:19.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.969+0000 7f432cafb640 1 -- 172.21.15.33:0/2201143618 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f43100a3240 msgr2=0x7f43100a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:19.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.969+0000 7f432cafb640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f43100a3240 0x7f43100a5700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4318004ad0 tx=0x7f4318004a60 comp rx=0 tx=0).stop 2024-09-16T00:14:19.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.969+0000 7f432cafb640 1 -- 172.21.15.33:0/2201143618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f432810c170 msgr2=0x7f432819cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:19.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.969+0000 7f432cafb640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f432810c170 0x7f432819cdc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f432000d950 tx=0x7f432000de10 comp rx=0 tx=0).stop 2024-09-16T00:14:19.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.969+0000 7f432cafb640 1 -- 172.21.15.33:0/2201143618 shutdown_connections 2024-09-16T00:14:19.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.969+0000 7f432cafb640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f43100a3240 0x7f43100a5700 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:19.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.969+0000 7f432cafb640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f432810c170 0x7f432819cdc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:19.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.969+0000 7f432cafb640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4328103660 0x7f4328198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:19.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.969+0000 7f432cafb640 1 --2- 172.21.15.33:0/2201143618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4328102d40 0x7f43281984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:19.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.969+0000 7f432cafb640 1 -- 172.21.15.33:0/2201143618 >> 172.21.15.33:0/2201143618 conn(0x7f43280fc7a0 msgr2=0x7f43280fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:19.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.969+0000 7f432cafb640 1 -- 172.21.15.33:0/2201143618 shutdown_connections 2024-09-16T00:14:19.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:19.969+0000 7f432cafb640 1 -- 172.21.15.33:0/2201143618 wait complete. 2024-09-16T00:14:20.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:20 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2201143618' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T00:14:20.714 INFO:tasks.cephadm:Adding mds.a on smithi033 2024-09-16T00:14:20.714 INFO:tasks.cephadm:Adding mds.d on smithi033 2024-09-16T00:14:20.714 INFO:tasks.cephadm:Adding mds.g on smithi033 2024-09-16T00:14:20.714 INFO:tasks.cephadm:Adding mds.j on smithi033 2024-09-16T00:14:20.715 INFO:tasks.cephadm:Adding mds.b on smithi060 2024-09-16T00:14:20.715 INFO:tasks.cephadm:Adding mds.e on smithi060 2024-09-16T00:14:20.715 INFO:tasks.cephadm:Adding mds.h on smithi060 2024-09-16T00:14:20.715 INFO:tasks.cephadm:Adding mds.k on smithi060 2024-09-16T00:14:20.715 INFO:tasks.cephadm:Adding mds.c on smithi083 2024-09-16T00:14:20.715 INFO:tasks.cephadm:Adding mds.f on smithi083 2024-09-16T00:14:20.715 INFO:tasks.cephadm:Adding mds.i on smithi083 2024-09-16T00:14:20.715 INFO:tasks.cephadm:Adding mds.l on smithi083 2024-09-16T00:14:20.715 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph orch apply mds all '12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi060=b;smithi060=e;smithi060=h;smithi060=k;smithi083=c;smithi083=f;smithi083=i;smithi083=l' 2024-09-16T00:14:20.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:20 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2201143618' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T00:14:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:14:20.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:20 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2201143618' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T00:14:21.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.457+0000 7fe9c38e9640 1 -- 172.21.15.83:0/4133206071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9bc1026c0 msgr2=0x7fe9bc108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:21.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.457+0000 7fe9c38e9640 1 --2- 172.21.15.83:0/4133206071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9bc1026c0 0x7fe9bc108f50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009ca0 tx=0x7fe9b402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:14:21.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.457+0000 7fe9c38e9640 1 -- 172.21.15.83:0/4133206071 shutdown_connections 2024-09-16T00:14:21.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.457+0000 7fe9c38e9640 1 --2- 172.21.15.83:0/4133206071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9bc1026c0 0x7fe9bc108f50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.457+0000 7fe9c38e9640 1 --2- 172.21.15.83:0/4133206071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9bc101d00 0x7fe9bc102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.457+0000 7fe9c38e9640 1 --2- 172.21.15.83:0/4133206071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9bc100b00 0x7fe9bc100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.457+0000 7fe9c38e9640 1 -- 172.21.15.83:0/4133206071 >> 172.21.15.83:0/4133206071 conn(0x7fe9bc0fc270 msgr2=0x7fe9bc0fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c38e9640 1 -- 172.21.15.83:0/4133206071 shutdown_connections 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c38e9640 1 -- 172.21.15.83:0/4133206071 wait complete. 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c38e9640 1 Processor -- start 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c38e9640 1 -- start start 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c38e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9bc100b00 0x7fe9bc197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c38e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9bc101d00 0x7fe9bc198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c38e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9bc1026c0 0x7fe9bc19f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c38e9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc10c800 con 0x7fe9bc101d00 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c38e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc10c680 con 0x7fe9bc1026c0 2024-09-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c38e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc10c980 con 0x7fe9bc100b00 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c1685640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9bc100b00 0x7fe9bc197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c1685640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9bc100b00 0x7fe9bc197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34702/0 (socket says 172.21.15.83:34702) 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.458+0000 7fe9c1685640 1 -- 172.21.15.83:0/3575053118 learned_addr learned my addr 172.21.15.83:0/3575053118 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9c0e84640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9bc101d00 0x7fe9bc198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9c1e86640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9bc1026c0 0x7fe9bc19f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9c1685640 1 -- 172.21.15.83:0/3575053118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9bc1026c0 msgr2=0x7fe9bc19f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9c1685640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9bc1026c0 0x7fe9bc19f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9c1685640 1 -- 172.21.15.83:0/3575053118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9bc101d00 msgr2=0x7fe9bc198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9c1685640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9bc101d00 0x7fe9bc198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9c1685640 1 -- 172.21.15.83:0/3575053118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9bc19fb50 con 0x7fe9bc100b00 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9c0e84640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9bc101d00 0x7fe9bc198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9c1685640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9bc100b00 0x7fe9bc197f20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac00b450 tx=0x7fe9ac00b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9b27fc640 1 -- 172.21.15.83:0/3575053118 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac015020 con 0x7fe9bc100b00 2024-09-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9b27fc640 1 -- 172.21.15.83:0/3575053118 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9ac011d70 con 0x7fe9bc100b00 2024-09-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9c38e9640 1 -- 172.21.15.83:0/3575053118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9bc19fe40 con 0x7fe9bc100b00 2024-09-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9b27fc640 1 -- 172.21.15.83:0/3575053118 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac00f340 con 0x7fe9bc100b00 2024-09-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.459+0000 7fe9c38e9640 1 -- 172.21.15.83:0/3575053118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9bc1a02c0 con 0x7fe9bc100b00 2024-09-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.460+0000 7fe9c38e9640 1 -- 172.21.15.83:0/3575053118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9bc1a0580 con 0x7fe9bc100b00 2024-09-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.461+0000 7fe9b27fc640 1 -- 172.21.15.83:0/3575053118 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe9ac018020 con 0x7fe9bc100b00 2024-09-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.462+0000 7fe9b27fc640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9a80a3270 0x7fe9a80a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.462+0000 7fe9b27fc640 1 -- 172.21.15.83:0/3575053118 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fe9ac0cf500 con 0x7fe9bc100b00 2024-09-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.462+0000 7fe9c0e84640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9a80a3270 0x7fe9a80a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.463+0000 7fe9c0e84640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9a80a3270 0x7fe9a80a5730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc199440 tx=0x7fe9b800a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.469+0000 7fe9b27fc640 1 -- 172.21.15.83:0/3575053118 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe9ac01f020 con 0x7fe9bc100b00 2024-09-16T00:14:21.643 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:21 smithi083 ceph-mon[35242]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:21.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.652+0000 7fe9c38e9640 1 -- 172.21.15.83:0/3575053118 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi060=b;smithi060=e;smithi060=h;smithi060=k;smithi083=c;smithi083=f;smithi083=i;smithi083=l", "target": ["mon-mgr", ""]}) -- 0x7fe9bc06aea0 con 0x7fe9a80a3270 2024-09-16T00:14:21.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.660+0000 7fe9b27fc640 1 -- 172.21.15.83:0/3575053118 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fe9bc06aea0 con 0x7fe9a80a3270 2024-09-16T00:14:21.660 INFO:teuthology.orchestra.run.smithi083.stdout:Scheduled mds.all update... 2024-09-16T00:14:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.662+0000 7fe9c38e9640 1 -- 172.21.15.83:0/3575053118 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9a80a3270 msgr2=0x7fe9a80a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.662+0000 7fe9c38e9640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9a80a3270 0x7fe9a80a5730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc199440 tx=0x7fe9b800a5c0 comp rx=0 tx=0).stop 2024-09-16T00:14:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.662+0000 7fe9c38e9640 1 -- 172.21.15.83:0/3575053118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9bc100b00 msgr2=0x7fe9bc197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.662+0000 7fe9c38e9640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9bc100b00 0x7fe9bc197f20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac00b450 tx=0x7fe9ac00b910 comp rx=0 tx=0).stop 2024-09-16T00:14:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.662+0000 7fe9c38e9640 1 -- 172.21.15.83:0/3575053118 shutdown_connections 2024-09-16T00:14:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.662+0000 7fe9c38e9640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9a80a3270 0x7fe9a80a5730 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.662+0000 7fe9c38e9640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9bc1026c0 0x7fe9bc19f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.662+0000 7fe9c38e9640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9bc101d00 0x7fe9bc198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.662+0000 7fe9c38e9640 1 --2- 172.21.15.83:0/3575053118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9bc100b00 0x7fe9bc197f20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.662+0000 7fe9c38e9640 1 -- 172.21.15.83:0/3575053118 >> 172.21.15.83:0/3575053118 conn(0x7fe9bc0fc270 msgr2=0x7fe9bc06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.662+0000 7fe9c38e9640 1 -- 172.21.15.83:0/3575053118 shutdown_connections 2024-09-16T00:14:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-16T00:14:21.662+0000 7fe9c38e9640 1 -- 172.21.15.83:0/3575053118 wait complete. 2024-09-16T00:14:21.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:21 smithi033 ceph-mon[31243]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:21.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:21 smithi060 ceph-mon[35185]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:22.333 DEBUG:teuthology.orchestra.run.smithi033:mds.a> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.a.service 2024-09-16T00:14:22.336 DEBUG:teuthology.orchestra.run.smithi033:mds.d> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.d.service 2024-09-16T00:14:22.378 DEBUG:teuthology.orchestra.run.smithi033:mds.g> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.g.service 2024-09-16T00:14:22.421 DEBUG:teuthology.orchestra.run.smithi033:mds.j> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.j.service 2024-09-16T00:14:22.465 DEBUG:teuthology.orchestra.run.smithi060:mds.b> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.b.service 2024-09-16T00:14:22.468 DEBUG:teuthology.orchestra.run.smithi060:mds.e> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.e.service 2024-09-16T00:14:22.513 DEBUG:teuthology.orchestra.run.smithi060:mds.h> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.h.service 2024-09-16T00:14:22.556 DEBUG:teuthology.orchestra.run.smithi060:mds.k> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.k.service 2024-09-16T00:14:22.599 DEBUG:teuthology.orchestra.run.smithi083:mds.c> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.c.service 2024-09-16T00:14:22.601 DEBUG:teuthology.orchestra.run.smithi083:mds.f> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.f.service 2024-09-16T00:14:22.644 DEBUG:teuthology.orchestra.run.smithi083:mds.i> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.i.service 2024-09-16T00:14:22.688 DEBUG:teuthology.orchestra.run.smithi083:mds.l> sudo journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.l.service 2024-09-16T00:14:22.730 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-09-16T00:14:22.731 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:14:22.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:22 smithi033 ceph-mon[31243]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:22.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:22 smithi033 ceph-mon[31243]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi060=b;smithi060=e;smithi060=h;smithi060=k;smithi083=c;smithi083=f;smithi083=i;smithi083=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:14:22.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:22 smithi033 ceph-mon[31243]: Saving service mds.all spec with placement smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi060=b;smithi060=e;smithi060=h;smithi060=k;smithi083=c;smithi083=f;smithi083=i;smithi083=l;count:12 2024-09-16T00:14:22.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:22.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:14:22.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:22.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:14:22.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:22.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:22.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:22.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:22.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:22 smithi033 ceph-mon[31243]: Deploying daemon mds.l on smithi083 2024-09-16T00:14:22.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:22 smithi083 ceph-mon[35242]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:22.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:22 smithi083 ceph-mon[35242]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi060=b;smithi060=e;smithi060=h;smithi060=k;smithi083=c;smithi083=f;smithi083=i;smithi083=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:14:22.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:22 smithi083 ceph-mon[35242]: Saving service mds.all spec with placement smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi060=b;smithi060=e;smithi060=h;smithi060=k;smithi083=c;smithi083=f;smithi083=i;smithi083=l;count:12 2024-09-16T00:14:22.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:22.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:14:22.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:22.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:14:22.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:22.774 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:22.774 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:22.774 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:22.774 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:22 smithi083 ceph-mon[35242]: Deploying daemon mds.l on smithi083 2024-09-16T00:14:22.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:22 smithi060 ceph-mon[35185]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:22.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:22 smithi060 ceph-mon[35185]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi060=b;smithi060=e;smithi060=h;smithi060=k;smithi083=c;smithi083=f;smithi083=i;smithi083=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:14:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:22 smithi060 ceph-mon[35185]: Saving service mds.all spec with placement smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi060=b;smithi060=e;smithi060=h;smithi060=k;smithi083=c;smithi083=f;smithi083=i;smithi083=l;count:12 2024-09-16T00:14:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:14:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:14:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:22 smithi060 ceph-mon[35185]: Deploying daemon mds.l on smithi083 2024-09-16T00:14:22.982 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:23.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.480+0000 7f13985c0640 1 -- 172.21.15.33:0/3722280852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1390106f10 msgr2=0x7f139010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:23.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.480+0000 7f13985c0640 1 --2- 172.21.15.33:0/3722280852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1390106f10 0x7f139010ea90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1388009ca0 tx=0x7f138802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:14:23.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.480+0000 7f13985c0640 1 -- 172.21.15.33:0/3722280852 shutdown_connections 2024-09-16T00:14:23.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.480+0000 7f13985c0640 1 --2- 172.21.15.33:0/3722280852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1390106f10 0x7f139010ea90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:23.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.480+0000 7f13985c0640 1 --2- 172.21.15.33:0/3722280852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1390102790 0x7f13901067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:23.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.480+0000 7f13985c0640 1 --2- 172.21.15.33:0/3722280852 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1390101de0 0x7f13901021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:23.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.480+0000 7f13985c0640 1 -- 172.21.15.33:0/3722280852 >> 172.21.15.33:0/3722280852 conn(0x7f13900fd8c0 msgr2=0x7f13900ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:23.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.480+0000 7f13985c0640 1 -- 172.21.15.33:0/3722280852 shutdown_connections 2024-09-16T00:14:23.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.480+0000 7f13985c0640 1 -- 172.21.15.33:0/3722280852 wait complete. 2024-09-16T00:14:23.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f13985c0640 1 Processor -- start 2024-09-16T00:14:23.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f13985c0640 1 -- start start 2024-09-16T00:14:23.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f13985c0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1390101de0 0x7f13900744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:23.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f13985c0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1390102790 0x7f1390074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:23.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f13985c0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1390106f10 0x7f1390074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f13985c0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1390111f70 con 0x7f1390102790 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f13985c0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1390111df0 con 0x7f1390106f10 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f13985c0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13901120f0 con 0x7f1390101de0 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f1395b5b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1390102790 0x7f1390074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f1395b5b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1390102790 0x7f1390074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45268/0 (socket says 172.21.15.33:45268) 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f1395b5b640 1 -- 172.21.15.33:0/3073092510 learned_addr learned my addr 172.21.15.33:0/3073092510 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f139635c640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1390101de0 0x7f13900744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f1395b5b640 1 -- 172.21.15.33:0/3073092510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1390101de0 msgr2=0x7f13900744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f1395b5b640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1390101de0 0x7f13900744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f1396b5d640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1390106f10 0x7f1390074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f1395b5b640 1 -- 172.21.15.33:0/3073092510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1390106f10 msgr2=0x7f1390074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f1395b5b640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1390106f10 0x7f1390074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:23.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f1395b5b640 1 -- 172.21.15.33:0/3073092510 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13901a6890 con 0x7f1390102790 2024-09-16T00:14:23.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f1396b5d640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1390106f10 0x7f1390074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:14:23.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.481+0000 7f1395b5b640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1390102790 0x7f1390074a30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f138000d950 tx=0x7f138000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:23.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.482+0000 7f13877fe640 1 -- 172.21.15.33:0/3073092510 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1380017070 con 0x7f1390102790 2024-09-16T00:14:23.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.482+0000 7f13985c0640 1 -- 172.21.15.33:0/3073092510 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13901a6b80 con 0x7f1390102790 2024-09-16T00:14:23.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.482+0000 7f13877fe640 1 -- 172.21.15.33:0/3073092510 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1380009c30 con 0x7f1390102790 2024-09-16T00:14:23.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.482+0000 7f13877fe640 1 -- 172.21.15.33:0/3073092510 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138000f6b0 con 0x7f1390102790 2024-09-16T00:14:23.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.482+0000 7f13985c0640 1 -- 172.21.15.33:0/3073092510 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13901a70c0 con 0x7f1390102790 2024-09-16T00:14:23.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.483+0000 7f13985c0640 1 -- 172.21.15.33:0/3073092510 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13901021c0 con 0x7f1390102790 2024-09-16T00:14:23.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.484+0000 7f13877fe640 1 -- 172.21.15.33:0/3073092510 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f13800109d0 con 0x7f1390102790 2024-09-16T00:14:23.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.485+0000 7f13877fe640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f137c0a3240 0x7f137c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:23.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.485+0000 7f139635c640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f137c0a3240 0x7f137c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:23.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.485+0000 7f13877fe640 1 -- 172.21.15.33:0/3073092510 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f13800cf0a0 con 0x7f1390102790 2024-09-16T00:14:23.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.485+0000 7f139635c640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f137c0a3240 0x7f137c0a5700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f138c009bd0 tx=0x7f138c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:23.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.489+0000 7f13877fe640 1 -- 172.21.15.33:0/3073092510 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1380009de0 con 0x7f1390102790 2024-09-16T00:14:23.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.701+0000 7f13985c0640 1 -- 172.21.15.33:0/3073092510 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1390109bc0 con 0x7f1390102790 2024-09-16T00:14:23.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.701+0000 7f13877fe640 1 -- 172.21.15.33:0/3073092510 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7f138009c1c0 con 0x7f1390102790 2024-09-16T00:14:23.702 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:23.702 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":1,"btime":"2024-09-16T00:07:21:167536+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-09-16T00:14:23.703 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 1 2024-09-16T00:14:23.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.704+0000 7f13985c0640 1 -- 172.21.15.33:0/3073092510 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f137c0a3240 msgr2=0x7f137c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:23.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.704+0000 7f13985c0640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f137c0a3240 0x7f137c0a5700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f138c009bd0 tx=0x7f138c005f90 comp rx=0 tx=0).stop 2024-09-16T00:14:23.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.704+0000 7f13985c0640 1 -- 172.21.15.33:0/3073092510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1390102790 msgr2=0x7f1390074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:23.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.704+0000 7f13985c0640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1390102790 0x7f1390074a30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f138000d950 tx=0x7f138000de10 comp rx=0 tx=0).stop 2024-09-16T00:14:23.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.704+0000 7f13985c0640 1 -- 172.21.15.33:0/3073092510 shutdown_connections 2024-09-16T00:14:23.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.704+0000 7f13985c0640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f137c0a3240 0x7f137c0a5700 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:23.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.704+0000 7f13985c0640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1390106f10 0x7f1390074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:23.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.704+0000 7f13985c0640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1390102790 0x7f1390074a30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:23.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.704+0000 7f13985c0640 1 --2- 172.21.15.33:0/3073092510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1390101de0 0x7f13900744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:23.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.704+0000 7f13985c0640 1 -- 172.21.15.33:0/3073092510 >> 172.21.15.33:0/3073092510 conn(0x7f13900fd8c0 msgr2=0x7f13900feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:23.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.704+0000 7f13985c0640 1 -- 172.21.15.33:0/3073092510 shutdown_connections 2024-09-16T00:14:23.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:23.704+0000 7f13985c0640 1 -- 172.21.15.33:0/3073092510 wait complete. 2024-09-16T00:14:24.169 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:14:24 smithi083 systemd[1]: Starting Ceph mds.l for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:14:24.526 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:14:24 smithi083 podman[64445]: 2024-09-16 00:14:24.172187295 +0000 UTC m=+0.016056788 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:14:24.526 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:14:24 smithi083 podman[64445]: 2024-09-16 00:14:24.271626978 +0000 UTC m=+0.115496453 container create cc64a2a63a5d6cb29cad91338e0ec552f7e3e99821a75d47dbb0973b63c1f977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-l, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-16T00:14:24.526 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:14:24 smithi083 podman[64445]: 2024-09-16 00:14:24.499870477 +0000 UTC m=+0.343739951 container init cc64a2a63a5d6cb29cad91338e0ec552f7e3e99821a75d47dbb0973b63c1f977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T00:14:24.527 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:14:24 smithi083 podman[64445]: 2024-09-16 00:14:24.502548904 +0000 UTC m=+0.346418379 container start cc64a2a63a5d6cb29cad91338e0ec552f7e3e99821a75d47dbb0973b63c1f977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, io.buildah.version=1.37.2) 2024-09-16T00:14:24.527 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:14:24 smithi083 ceph-mds[64464]: -- 172.21.15.83:0/3745621548 <== mon.1 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55591358f860 con 0x55591356d400 2024-09-16T00:14:24.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:24 smithi033 ceph-mon[31243]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:24.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:24 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3073092510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:24.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:24 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.83:6832/1083835397,v1:172.21.15.83:6833/1083835397] up:boot 2024-09-16T00:14:24.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:24 smithi033 ceph-mon[31243]: fsmap 1 up:standby 2024-09-16T00:14:24.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:24 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-16T00:14:24.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:24 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:24.918 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:14:24 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-l[64460]: starting mds.l at 2024-09-16T00:14:24.919 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:14:24 smithi083 bash[64445]: cc64a2a63a5d6cb29cad91338e0ec552f7e3e99821a75d47dbb0973b63c1f977 2024-09-16T00:14:24.919 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:14:24 smithi083 systemd[1]: Started Ceph mds.l for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:14:24.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:24 smithi083 ceph-mon[35242]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:24.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:24 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3073092510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:24.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:24 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.83:6832/1083835397,v1:172.21.15.83:6833/1083835397] up:boot 2024-09-16T00:14:24.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:24 smithi083 ceph-mon[35242]: fsmap 1 up:standby 2024-09-16T00:14:24.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:24 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-16T00:14:24.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:24 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:24.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:24 smithi060 ceph-mon[35185]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:24.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:24 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3073092510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:24.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:24 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.83:6832/1083835397,v1:172.21.15.83:6833/1083835397] up:boot 2024-09-16T00:14:24.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:24 smithi060 ceph-mon[35185]: fsmap 1 up:standby 2024-09-16T00:14:24.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:24 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-16T00:14:24.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:24 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:25.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:25.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:25.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:25.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:25.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:25.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:25 smithi083 ceph-mon[35242]: Deploying daemon mds.k on smithi060 2024-09-16T00:14:25.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:25.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:25.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:25.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:25.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:25.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:25 smithi060 ceph-mon[35185]: Deploying daemon mds.k on smithi060 2024-09-16T00:14:26.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:26.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:26.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:26.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:26.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:26.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:25 smithi033 ceph-mon[31243]: Deploying daemon mds.k on smithi060 2024-09-16T00:14:26.558 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:14:26.826 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:26.946 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:26 smithi060 ceph-mon[35185]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:26.947 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:14:26 smithi060 systemd[1]: Starting Ceph mds.k for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:14:26.947 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:26 smithi033 ceph-mon[31243]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:27.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:26 smithi083 ceph-mon[35242]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:27.216 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:14:26 smithi060 podman[64647]: 2024-09-16 00:14:26.848812386 +0000 UTC m=+0.015803855 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:14:27.216 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:14:26 smithi060 podman[64647]: 2024-09-16 00:14:26.979369369 +0000 UTC m=+0.146360872 container create 30c350dd320bd9093927f2f0e4c1623d5a4cde0abf4a41287d59f1f18437ace3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-k, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-16T00:14:27.216 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:14:27 smithi060 podman[64647]: 2024-09-16 00:14:27.215621148 +0000 UTC m=+0.382612607 container init 30c350dd320bd9093927f2f0e4c1623d5a4cde0abf4a41287d59f1f18437ace3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-k, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.291+0000 7f9a91559640 1 -- 172.21.15.33:0/2529975714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c102dd0 msgr2=0x7f9a8c1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.291+0000 7f9a91559640 1 --2- 172.21.15.33:0/2529975714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c102dd0 0x7f9a8c1031b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009cd0 tx=0x7f9a7c02a8d0 comp rx=0 tx=0).stop 2024-09-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.291+0000 7f9a91559640 1 -- 172.21.15.33:0/2529975714 shutdown_connections 2024-09-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.291+0000 7f9a91559640 1 --2- 172.21.15.33:0/2529975714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a8c108300 0x7f9a8c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.291+0000 7f9a91559640 1 --2- 172.21.15.33:0/2529975714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8c103780 0x7f9a8c107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.291+0000 7f9a91559640 1 --2- 172.21.15.33:0/2529975714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c102dd0 0x7f9a8c1031b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.291+0000 7f9a91559640 1 -- 172.21.15.33:0/2529975714 >> 172.21.15.33:0/2529975714 conn(0x7f9a8c0fc7d0 msgr2=0x7f9a8c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.291+0000 7f9a91559640 1 -- 172.21.15.33:0/2529975714 shutdown_connections 2024-09-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.291+0000 7f9a91559640 1 -- 172.21.15.33:0/2529975714 wait complete. 2024-09-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.291+0000 7f9a91559640 1 Processor -- start 2024-09-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a91559640 1 -- start start 2024-09-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a91559640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a8c102dd0 0x7f9a8c100a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a91559640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8c103780 0x7f9a8c100f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a91559640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c108300 0x7f9a8c1014b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a91559640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c112090 con 0x7f9a8c108300 2024-09-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a91559640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c111f10 con 0x7f9a8c103780 2024-09-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a91559640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c112210 con 0x7f9a8c102dd0 2024-09-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a8b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c108300 0x7f9a8c1014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a8a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8c103780 0x7f9a8c100f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a8affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a8c102dd0 0x7f9a8c100a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a8b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c108300 0x7f9a8c1014b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45284/0 (socket says 172.21.15.33:45284) 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a8b7fe640 1 -- 172.21.15.33:0/3173415320 learned_addr learned my addr 172.21.15.33:0/3173415320 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a8b7fe640 1 -- 172.21.15.33:0/3173415320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a8c102dd0 msgr2=0x7f9a8c100a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a8b7fe640 1 --2- 172.21.15.33:0/3173415320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a8c102dd0 0x7f9a8c100a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a8b7fe640 1 -- 172.21.15.33:0/3173415320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8c103780 msgr2=0x7f9a8c100f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a8b7fe640 1 --2- 172.21.15.33:0/3173415320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8c103780 0x7f9a8c100f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.292+0000 7f9a8b7fe640 1 -- 172.21.15.33:0/3173415320 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c1a6a10 con 0x7f9a8c108300 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.293+0000 7f9a8b7fe640 1 --2- 172.21.15.33:0/3173415320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c108300 0x7f9a8c1014b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400c970 tx=0x7f9a8400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.293+0000 7f9a73fff640 1 -- 172.21.15.33:0/3173415320 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84007bb0 con 0x7f9a8c108300 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.293+0000 7f9a91559640 1 -- 172.21.15.33:0/3173415320 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8c1a6d00 con 0x7f9a8c108300 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.293+0000 7f9a73fff640 1 -- 172.21.15.33:0/3173415320 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a84007d50 con 0x7f9a8c108300 2024-09-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.293+0000 7f9a73fff640 1 -- 172.21.15.33:0/3173415320 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84018800 con 0x7f9a8c108300 2024-09-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.293+0000 7f9a91559640 1 -- 172.21.15.33:0/3173415320 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8c1a71f0 con 0x7f9a8c108300 2024-09-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.294+0000 7f9a91559640 1 -- 172.21.15.33:0/3173415320 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a8c06a7b0 con 0x7f9a8c108300 2024-09-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.295+0000 7f9a73fff640 1 -- 172.21.15.33:0/3173415320 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f9a8400fa50 con 0x7f9a8c108300 2024-09-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.296+0000 7f9a73fff640 1 --2- 172.21.15.33:0/3173415320 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a740a3240 0x7f9a740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.296+0000 7f9a8affd640 1 --2- 172.21.15.33:0/3173415320 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a740a3240 0x7f9a740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.296+0000 7f9a73fff640 1 -- 172.21.15.33:0/3173415320 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f9a840cf3a0 con 0x7f9a8c108300 2024-09-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.296+0000 7f9a8affd640 1 --2- 172.21.15.33:0/3173415320 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a740a3240 0x7f9a740a5700 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c005e30 tx=0x7f9a7c031000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.300+0000 7f9a73fff640 1 -- 172.21.15.33:0/3173415320 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9a84014030 con 0x7f9a8c108300 2024-09-16T00:14:27.480 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:14:27 smithi060 podman[64647]: 2024-09-16 00:14:27.218600884 +0000 UTC m=+0.385592355 container start 30c350dd320bd9093927f2f0e4c1623d5a4cde0abf4a41287d59f1f18437ace3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-k, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T00:14:27.481 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:14:27 smithi060 ceph-mds[64666]: -- 172.21.15.60:0/3478101169 <== mon.2 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557dd8077860 con 0x557dd8055000 2024-09-16T00:14:27.481 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:14:27 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-k[64662]: starting mds.k at 2024-09-16T00:14:27.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.511+0000 7f9a91559640 1 -- 172.21.15.33:0/3173415320 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a8c10a290 con 0x7f9a8c108300 2024-09-16T00:14:27.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.511+0000 7f9a73fff640 1 -- 172.21.15.33:0/3173415320 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1435 (secure 0 0 0) 0x7f9a840865b0 con 0x7f9a8c108300 2024-09-16T00:14:27.512 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:27.512 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":2,"btime":"2024-09-16T00:14:24:530818+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-16T00:14:27.513 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 2 2024-09-16T00:14:27.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.514+0000 7f9a91559640 1 -- 172.21.15.33:0/3173415320 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a740a3240 msgr2=0x7f9a740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:27.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.514+0000 7f9a91559640 1 --2- 172.21.15.33:0/3173415320 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a740a3240 0x7f9a740a5700 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c005e30 tx=0x7f9a7c031000 comp rx=0 tx=0).stop 2024-09-16T00:14:27.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.514+0000 7f9a91559640 1 -- 172.21.15.33:0/3173415320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c108300 msgr2=0x7f9a8c1014b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:27.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.514+0000 7f9a91559640 1 --2- 172.21.15.33:0/3173415320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c108300 0x7f9a8c1014b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400c970 tx=0x7f9a8400ce30 comp rx=0 tx=0).stop 2024-09-16T00:14:27.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.514+0000 7f9a91559640 1 -- 172.21.15.33:0/3173415320 shutdown_connections 2024-09-16T00:14:27.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.514+0000 7f9a91559640 1 --2- 172.21.15.33:0/3173415320 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a740a3240 0x7f9a740a5700 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:27.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.514+0000 7f9a91559640 1 --2- 172.21.15.33:0/3173415320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c108300 0x7f9a8c1014b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:27.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.514+0000 7f9a91559640 1 --2- 172.21.15.33:0/3173415320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8c103780 0x7f9a8c100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:27.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.514+0000 7f9a91559640 1 --2- 172.21.15.33:0/3173415320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a8c102dd0 0x7f9a8c100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:27.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.514+0000 7f9a91559640 1 -- 172.21.15.33:0/3173415320 >> 172.21.15.33:0/3173415320 conn(0x7f9a8c0fc7d0 msgr2=0x7f9a8c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:27.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.514+0000 7f9a91559640 1 -- 172.21.15.33:0/3173415320 shutdown_connections 2024-09-16T00:14:27.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:27.514+0000 7f9a91559640 1 -- 172.21.15.33:0/3173415320 wait complete. 2024-09-16T00:14:27.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:27 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3173415320' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:27.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:27.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:27.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:27.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:27.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:27.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:27.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:27 smithi033 ceph-mon[31243]: Deploying daemon mds.j on smithi033 2024-09-16T00:14:27.981 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:14:27 smithi060 bash[64647]: 30c350dd320bd9093927f2f0e4c1623d5a4cde0abf4a41287d59f1f18437ace3 2024-09-16T00:14:27.981 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:14:27 smithi060 systemd[1]: Started Ceph mds.k for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:14:27.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:27 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3173415320' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:27.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:27.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:27.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:27.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:27.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:27.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:27.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:27 smithi060 ceph-mon[35185]: Deploying daemon mds.j on smithi033 2024-09-16T00:14:28.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:27 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3173415320' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:28.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:28.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:28.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:28.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:28.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:28.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:28.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:27 smithi083 ceph-mon[35242]: Deploying daemon mds.j on smithi033 2024-09-16T00:14:28.817 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:28 smithi060 ceph-mon[35185]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:28.817 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:28 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.60:6832/2130595683,v1:172.21.15.60:6833/2130595683] up:boot 2024-09-16T00:14:28.817 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:28 smithi060 ceph-mon[35185]: fsmap 2 up:standby 2024-09-16T00:14:28.817 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-16T00:14:28.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:28 smithi033 ceph-mon[31243]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:28.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:28 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.60:6832/2130595683,v1:172.21.15.60:6833/2130595683] up:boot 2024-09-16T00:14:28.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:28 smithi033 ceph-mon[31243]: fsmap 2 up:standby 2024-09-16T00:14:28.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-16T00:14:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:28 smithi083 ceph-mon[35242]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:28 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.60:6832/2130595683,v1:172.21.15.60:6833/2130595683] up:boot 2024-09-16T00:14:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:28 smithi083 ceph-mon[35242]: fsmap 2 up:standby 2024-09-16T00:14:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-16T00:14:30.299 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:14:30.324 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:14:30 smithi033 systemd[1]: Starting Ceph mds.j for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:14:30.580 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:14:30 smithi033 podman[67952]: 2024-09-16 00:14:30.266067403 +0000 UTC m=+0.023674630 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:14:30.580 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:14:30 smithi033 podman[67952]: 2024-09-16 00:14:30.395062512 +0000 UTC m=+0.152669728 container create fdf1e0d0550706c463a952074c0f3530768eb13a3d22b28e312276303f00b51f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-j, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux ) 2024-09-16T00:14:30.788 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:30.911 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:30 smithi033 ceph-mon[31243]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:30.912 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:14:30 smithi033 podman[67952]: 2024-09-16 00:14:30.650310058 +0000 UTC m=+0.407917285 container init fdf1e0d0550706c463a952074c0f3530768eb13a3d22b28e312276303f00b51f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-j, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-16T00:14:30.912 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:14:30 smithi033 podman[67952]: 2024-09-16 00:14:30.654156531 +0000 UTC m=+0.411763749 container start fdf1e0d0550706c463a952074c0f3530768eb13a3d22b28e312276303f00b51f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-j, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-16T00:14:30.912 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:14:30 smithi033 ceph-mds[68020]: -- 172.21.15.33:0/2976169463 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564de7c45c20 con 0x564de7c23400 2024-09-16T00:14:30.912 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:14:30 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-j[68003]: starting mds.j at 2024-09-16T00:14:30.912 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:14:30 smithi033 bash[67952]: fdf1e0d0550706c463a952074c0f3530768eb13a3d22b28e312276303f00b51f 2024-09-16T00:14:30.912 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:14:30 smithi033 systemd[1]: Started Ceph mds.j for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:14:30.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:30 smithi060 ceph-mon[35185]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:31.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:30 smithi083 ceph-mon[35242]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.262+0000 7f0f02bd8640 1 -- 172.21.15.33:0/3021693545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0efc102570 msgr2=0x7f0efc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.262+0000 7f0f02bd8640 1 --2- 172.21.15.33:0/3021693545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0efc102570 0x7f0efc1029b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009900 tx=0x7f0ee802a480 comp rx=0 tx=0).stop 2024-09-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.262+0000 7f0f02bd8640 1 -- 172.21.15.33:0/3021693545 shutdown_connections 2024-09-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.262+0000 7f0f02bd8640 1 --2- 172.21.15.33:0/3021693545 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0efc102ef0 0x7f0efc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.262+0000 7f0f02bd8640 1 --2- 172.21.15.33:0/3021693545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0efc102570 0x7f0efc1029b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.262+0000 7f0f02bd8640 1 --2- 172.21.15.33:0/3021693545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0efc100e20 0x7f0efc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.262+0000 7f0f02bd8640 1 -- 172.21.15.33:0/3021693545 >> 172.21.15.33:0/3021693545 conn(0x7f0efc0fc7d0 msgr2=0x7f0efc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.262+0000 7f0f02bd8640 1 -- 172.21.15.33:0/3021693545 shutdown_connections 2024-09-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.262+0000 7f0f02bd8640 1 -- 172.21.15.33:0/3021693545 wait complete. 2024-09-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.263+0000 7f0f02bd8640 1 Processor -- start 2024-09-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.263+0000 7f0f02bd8640 1 -- start start 2024-09-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.263+0000 7f0f02bd8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0efc100e20 0x7f0efc19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.263+0000 7f0f02bd8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0efc102570 0x7f0efc19ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.263+0000 7f0f02bd8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0efc102ef0 0x7f0efc19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.263+0000 7f0f02bd8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc110fa0 con 0x7f0efc102ef0 2024-09-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.263+0000 7f0f02bd8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc110e20 con 0x7f0efc100e20 2024-09-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.263+0000 7f0f02bd8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc111120 con 0x7f0efc102570 2024-09-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.263+0000 7f0f01175640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0efc102ef0 0x7f0efc19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0f01175640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0efc102ef0 0x7f0efc19eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41660/0 (socket says 172.21.15.33:41660) 2024-09-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0f01175640 1 -- 172.21.15.33:0/2813842042 learned_addr learned my addr 172.21.15.33:0/2813842042 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0efbfff640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0efc102570 0x7f0efc19ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0f00974640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0efc100e20 0x7f0efc19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0f01175640 1 -- 172.21.15.33:0/2813842042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0efc102570 msgr2=0x7f0efc19ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0f01175640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0efc102570 0x7f0efc19ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0f01175640 1 -- 172.21.15.33:0/2813842042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0efc100e20 msgr2=0x7f0efc19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0f01175640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0efc100e20 0x7f0efc19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0f01175640 1 -- 172.21.15.33:0/2813842042 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc19f6d0 con 0x7f0efc102ef0 2024-09-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0f01175640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0efc102ef0 0x7f0efc19eff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400eaa0 tx=0x7f0ef400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0ef9ffb640 1 -- 172.21.15.33:0/2813842042 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef400cdb0 con 0x7f0efc102ef0 2024-09-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0ef9ffb640 1 -- 172.21.15.33:0/2813842042 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ef4005020 con 0x7f0efc102ef0 2024-09-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0f02bd8640 1 -- 172.21.15.33:0/2813842042 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0efc19f9c0 con 0x7f0efc102ef0 2024-09-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0ef9ffb640 1 -- 172.21.15.33:0/2813842042 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef40116b0 con 0x7f0efc102ef0 2024-09-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.264+0000 7f0f02bd8640 1 -- 172.21.15.33:0/2813842042 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0efc1a72a0 con 0x7f0efc102ef0 2024-09-16T00:14:31.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.265+0000 7f0f02bd8640 1 -- 172.21.15.33:0/2813842042 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0efc101200 con 0x7f0efc102ef0 2024-09-16T00:14:31.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.266+0000 7f0ef9ffb640 1 -- 172.21.15.33:0/2813842042 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f0ef4005560 con 0x7f0efc102ef0 2024-09-16T00:14:31.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.267+0000 7f0ef9ffb640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0eec0a3240 0x7f0eec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:31.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.267+0000 7f0f00974640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0eec0a3240 0x7f0eec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:31.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.267+0000 7f0ef9ffb640 1 -- 172.21.15.33:0/2813842042 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f0ef40cf7a0 con 0x7f0efc102ef0 2024-09-16T00:14:31.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.268+0000 7f0f00974640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0eec0a3240 0x7f0eec0a5700 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0005fd0 tx=0x7f0ef0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:31.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.271+0000 7f0ef9ffb640 1 -- 172.21.15.33:0/2813842042 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0ef4016030 con 0x7f0efc102ef0 2024-09-16T00:14:31.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.496+0000 7f0f02bd8640 1 -- 172.21.15.33:0/2813842042 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0efc1a7470 con 0x7f0efc102ef0 2024-09-16T00:14:31.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.497+0000 7f0ef9ffb640 1 -- 172.21.15.33:0/2813842042 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3085 (secure 0 0 0) 0x7f0ef409c960 con 0x7f0efc102ef0 2024-09-16T00:14:31.498 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:31.498 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":4,"btime":"2024-09-16T00:14:30:735115+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-16T00:14:31.498 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 4 2024-09-16T00:14:31.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.499+0000 7f0f02bd8640 1 -- 172.21.15.33:0/2813842042 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0eec0a3240 msgr2=0x7f0eec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:31.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.499+0000 7f0f02bd8640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0eec0a3240 0x7f0eec0a5700 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0005fd0 tx=0x7f0ef0005850 comp rx=0 tx=0).stop 2024-09-16T00:14:31.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.499+0000 7f0f02bd8640 1 -- 172.21.15.33:0/2813842042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0efc102ef0 msgr2=0x7f0efc19eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:31.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.499+0000 7f0f02bd8640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0efc102ef0 0x7f0efc19eff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400eaa0 tx=0x7f0ef400ef60 comp rx=0 tx=0).stop 2024-09-16T00:14:31.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.499+0000 7f0f02bd8640 1 -- 172.21.15.33:0/2813842042 shutdown_connections 2024-09-16T00:14:31.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.499+0000 7f0f02bd8640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0eec0a3240 0x7f0eec0a5700 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:31.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.499+0000 7f0f02bd8640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0efc102ef0 0x7f0efc19eff0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:31.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.499+0000 7f0f02bd8640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0efc102570 0x7f0efc19ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:31.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.499+0000 7f0f02bd8640 1 --2- 172.21.15.33:0/2813842042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0efc100e20 0x7f0efc19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:31.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.499+0000 7f0f02bd8640 1 -- 172.21.15.33:0/2813842042 >> 172.21.15.33:0/2813842042 conn(0x7f0efc0fc7d0 msgr2=0x7f0efc0ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:31.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.500+0000 7f0f02bd8640 1 -- 172.21.15.33:0/2813842042 shutdown_connections 2024-09-16T00:14:31.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:31.500+0000 7f0f02bd8640 1 -- 172.21.15.33:0/2813842042 wait complete. 2024-09-16T00:14:32.035 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:31 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.33:6834/2093179667,v1:172.21.15.33:6835/2093179667] up:boot 2024-09-16T00:14:32.035 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:31 smithi033 ceph-mon[31243]: fsmap 3 up:standby 2024-09-16T00:14:32.035 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-16T00:14:32.035 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:32.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:32.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:32.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:32.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:32.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:32.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:31 smithi033 ceph-mon[31243]: Deploying daemon mds.d on smithi033 2024-09-16T00:14:32.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:31 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2813842042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:32.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:31 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.33:6834/2093179667,v1:172.21.15.33:6835/2093179667] up:boot 2024-09-16T00:14:32.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:31 smithi083 ceph-mon[35242]: fsmap 3 up:standby 2024-09-16T00:14:32.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-16T00:14:32.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:32.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:32.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:32.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:32.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:32.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:32.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:31 smithi083 ceph-mon[35242]: Deploying daemon mds.d on smithi033 2024-09-16T00:14:32.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:31 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2813842042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:32.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:31 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.33:6834/2093179667,v1:172.21.15.33:6835/2093179667] up:boot 2024-09-16T00:14:32.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:31 smithi060 ceph-mon[35185]: fsmap 3 up:standby 2024-09-16T00:14:32.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-16T00:14:32.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:32.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:32.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:32.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:32.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:32.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:32.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:31 smithi060 ceph-mon[35185]: Deploying daemon mds.d on smithi033 2024-09-16T00:14:32.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:31 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2813842042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:33.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:32 smithi083 ceph-mon[35242]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:33.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:32 smithi060 ceph-mon[35185]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:33.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:32 smithi033 ceph-mon[31243]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:34.056 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:14:33 smithi033 systemd[1]: Starting Ceph mds.d for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:14:34.056 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:14:34 smithi033 podman[68376]: 2024-09-16 00:14:33.958156107 +0000 UTC m=+0.016497875 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:14:34.324 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:14:34 smithi033 podman[68376]: 2024-09-16 00:14:34.079672159 +0000 UTC m=+0.138013926 container create 7ed5f1632dbd749930b874544805379e3d78845343eaa0a5bc91886f6016484d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-d, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-16T00:14:34.324 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:14:34 smithi033 podman[68376]: 2024-09-16 00:14:34.324361642 +0000 UTC m=+0.382703400 container init 7ed5f1632dbd749930b874544805379e3d78845343eaa0a5bc91886f6016484d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-16T00:14:34.580 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:14:34 smithi033 podman[68376]: 2024-09-16 00:14:34.327346093 +0000 UTC m=+0.385687854 container start 7ed5f1632dbd749930b874544805379e3d78845343eaa0a5bc91886f6016484d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-16T00:14:34.580 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:14:34 smithi033 ceph-mds[68395]: -- 172.21.15.33:0/981775884 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557d8059b860 con 0x557d80578800 2024-09-16T00:14:34.580 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:14:34 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-d[68391]: starting mds.d at 2024-09-16T00:14:34.580 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:14:34 smithi033 bash[68376]: 7ed5f1632dbd749930b874544805379e3d78845343eaa0a5bc91886f6016484d 2024-09-16T00:14:34.581 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:14:34 smithi033 systemd[1]: Started Ceph mds.d for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:14:34.921 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:14:35.182 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:35 smithi033 ceph-mon[31243]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:35.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:35.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:35.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:35.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:35 smithi033 ceph-mon[31243]: Deploying daemon mds.e on smithi060 2024-09-16T00:14:35.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:35 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.33:6836/3526002638,v1:172.21.15.33:6837/3526002638] up:boot 2024-09-16T00:14:35.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:35 smithi033 ceph-mon[31243]: fsmap 4 up:standby 2024-09-16T00:14:35.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.705+0000 7f2408068640 1 -- 172.21.15.33:0/2796691051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2400100a80 msgr2=0x7f2400100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.705+0000 7f2408068640 1 --2- 172.21.15.33:0/2796691051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2400100a80 0x7f2400100ee0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f23f0009930 tx=0x7f23f002a420 comp rx=0 tx=0).stop 2024-09-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.705+0000 7f2408068640 1 -- 172.21.15.33:0/2796691051 shutdown_connections 2024-09-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.705+0000 7f2408068640 1 --2- 172.21.15.33:0/2796691051 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2400101420 0x7f240010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.705+0000 7f2408068640 1 --2- 172.21.15.33:0/2796691051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2400100a80 0x7f2400100ee0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.705+0000 7f2408068640 1 --2- 172.21.15.33:0/2796691051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2400106a80 0x7f2400106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.705+0000 7f2408068640 1 -- 172.21.15.33:0/2796691051 >> 172.21.15.33:0/2796691051 conn(0x7f24000fc7a0 msgr2=0x7f24000febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.705+0000 7f2408068640 1 -- 172.21.15.33:0/2796691051 shutdown_connections 2024-09-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.705+0000 7f2408068640 1 -- 172.21.15.33:0/2796691051 wait complete. 2024-09-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.706+0000 7f2408068640 1 Processor -- start 2024-09-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.706+0000 7f2408068640 1 -- start start 2024-09-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.706+0000 7f2408068640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2400100a80 0x7f24001973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.706+0000 7f2408068640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2400101420 0x7f24001978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.706+0000 7f2408068640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2400106a80 0x7f240019bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.706+0000 7f2408068640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2400110e80 con 0x7f2400100a80 2024-09-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.706+0000 7f2408068640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2400110d00 con 0x7f2400101420 2024-09-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.706+0000 7f2408068640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2400111000 con 0x7f2400106a80 2024-09-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2405e04640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2400100a80 0x7f24001973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2405e04640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2400100a80 0x7f24001973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41692/0 (socket says 172.21.15.33:41692) 2024-09-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2405e04640 1 -- 172.21.15.33:0/2966928732 learned_addr learned my addr 172.21.15.33:0/2966928732 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2405603640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2400101420 0x7f24001978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2405e04640 1 -- 172.21.15.33:0/2966928732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2400106a80 msgr2=0x7f240019bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2406605640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2400106a80 0x7f240019bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2405e04640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2400106a80 0x7f240019bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2405e04640 1 -- 172.21.15.33:0/2966928732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2400101420 msgr2=0x7f24001978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2405e04640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2400101420 0x7f24001978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2405e04640 1 -- 172.21.15.33:0/2966928732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240019c350 con 0x7f2400100a80 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2405603640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2400101420 0x7f24001978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2406605640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2400106a80 0x7f240019bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2405e04640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2400100a80 0x7f24001973a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00eb00 tx=0x7f23fc00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f23f6ffd640 1 -- 172.21.15.33:0/2966928732 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00bc30 con 0x7f2400100a80 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f23f6ffd640 1 -- 172.21.15.33:0/2966928732 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23fc00bdd0 con 0x7f2400100a80 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2408068640 1 -- 172.21.15.33:0/2966928732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f240019c640 con 0x7f2400100a80 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f23f6ffd640 1 -- 172.21.15.33:0/2966928732 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc017410 con 0x7f2400100a80 2024-09-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.707+0000 7f2408068640 1 -- 172.21.15.33:0/2966928732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f240006a790 con 0x7f2400100a80 2024-09-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.708+0000 7f2408068640 1 -- 172.21.15.33:0/2966928732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24001021c0 con 0x7f2400100a80 2024-09-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.709+0000 7f23f6ffd640 1 -- 172.21.15.33:0/2966928732 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f23fc005020 con 0x7f2400100a80 2024-09-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.710+0000 7f23f6ffd640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f23ec0a32c0 0x7f23ec0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.710+0000 7f2405603640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f23ec0a32c0 0x7f23ec0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.710+0000 7f23f6ffd640 1 -- 172.21.15.33:0/2966928732 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f23fc0d0d80 con 0x7f2400100a80 2024-09-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.711+0000 7f2405603640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f23ec0a32c0 0x7f23ec0a5780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f23f0000c00 tx=0x7f23f0005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.714+0000 7f23f6ffd640 1 -- 172.21.15.33:0/2966928732 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f23fc09df40 con 0x7f2400100a80 2024-09-16T00:14:35.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:35 smithi060 ceph-mon[35185]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:35.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:35.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:35.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:35.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:35.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:35.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:35.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:35 smithi060 ceph-mon[35185]: Deploying daemon mds.e on smithi060 2024-09-16T00:14:35.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:35 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.33:6836/3526002638,v1:172.21.15.33:6837/3526002638] up:boot 2024-09-16T00:14:35.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:35 smithi060 ceph-mon[35185]: fsmap 4 up:standby 2024-09-16T00:14:35.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-16T00:14:35.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:35 smithi083 ceph-mon[35242]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:35.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:35.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:35.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:35.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:35.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:35.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:35.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:35 smithi083 ceph-mon[35242]: Deploying daemon mds.e on smithi060 2024-09-16T00:14:35.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:35 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.33:6836/3526002638,v1:172.21.15.33:6837/3526002638] up:boot 2024-09-16T00:14:35.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:35 smithi083 ceph-mon[35242]: fsmap 4 up:standby 2024-09-16T00:14:35.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-16T00:14:35.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.923+0000 7f2408068640 1 -- 172.21.15.33:0/2966928732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f240006b1b0 con 0x7f2400100a80 2024-09-16T00:14:35.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.924+0000 7f23f6ffd640 1 -- 172.21.15.33:0/2966928732 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3910 (secure 0 0 0) 0x7f23fc09d840 con 0x7f2400100a80 2024-09-16T00:14:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:35.924 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":5,"btime":"2024-09-16T00:14:34:833626+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-16T00:14:35.925 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 5 2024-09-16T00:14:35.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.926+0000 7f2408068640 1 -- 172.21.15.33:0/2966928732 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f23ec0a32c0 msgr2=0x7f23ec0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:35.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.926+0000 7f2408068640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f23ec0a32c0 0x7f23ec0a5780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f23f0000c00 tx=0x7f23f0005a20 comp rx=0 tx=0).stop 2024-09-16T00:14:35.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.926+0000 7f2408068640 1 -- 172.21.15.33:0/2966928732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2400100a80 msgr2=0x7f24001973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:35.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.926+0000 7f2408068640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2400100a80 0x7f24001973a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00eb00 tx=0x7f23fc00ef00 comp rx=0 tx=0).stop 2024-09-16T00:14:35.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.926+0000 7f2408068640 1 -- 172.21.15.33:0/2966928732 shutdown_connections 2024-09-16T00:14:35.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.926+0000 7f2408068640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f23ec0a32c0 0x7f23ec0a5780 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:35.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.926+0000 7f2408068640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2400106a80 0x7f240019bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:35.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.926+0000 7f2408068640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2400101420 0x7f24001978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:35.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.926+0000 7f2408068640 1 --2- 172.21.15.33:0/2966928732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2400100a80 0x7f24001973a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:35.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.926+0000 7f2408068640 1 -- 172.21.15.33:0/2966928732 >> 172.21.15.33:0/2966928732 conn(0x7f24000fc7a0 msgr2=0x7f24000fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:35.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.926+0000 7f2408068640 1 -- 172.21.15.33:0/2966928732 shutdown_connections 2024-09-16T00:14:35.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:35.926+0000 7f2408068640 1 -- 172.21.15.33:0/2966928732 wait complete. 2024-09-16T00:14:36.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:36 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2966928732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:36.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:36 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2966928732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:36.731 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:14:36 smithi060 systemd[1]: Starting Ceph mds.e for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:14:36.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:36 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2966928732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:37.048 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:14:36 smithi060 podman[64959]: 2024-09-16 00:14:36.696965734 +0000 UTC m=+0.015819127 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:14:37.048 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:14:36 smithi060 podman[64959]: 2024-09-16 00:14:36.811794663 +0000 UTC m=+0.130648034 container create 34c56ec6ab56af859dbe470626758f4c8464f31dde68344abad222222c464491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-e, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:14:37.048 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:14:37 smithi060 podman[64959]: 2024-09-16 00:14:37.048271334 +0000 UTC m=+0.367124715 container init 34c56ec6ab56af859dbe470626758f4c8464f31dde68344abad222222c464491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T00:14:37.480 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:14:37 smithi060 podman[64959]: 2024-09-16 00:14:37.051260353 +0000 UTC m=+0.370113727 container start 34c56ec6ab56af859dbe470626758f4c8464f31dde68344abad222222c464491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-16T00:14:37.481 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:14:37 smithi060 ceph-mds[64978]: -- 172.21.15.60:0/2056045326 <== mon.2 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559553259860 con 0x559553237000 2024-09-16T00:14:37.481 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:14:37 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-e[64974]: starting mds.e at 2024-09-16T00:14:37.481 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:14:37 smithi060 bash[64959]: 34c56ec6ab56af859dbe470626758f4c8464f31dde68344abad222222c464491 2024-09-16T00:14:37.481 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:14:37 smithi060 systemd[1]: Started Ceph mds.e for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:14:37.783 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:37 smithi083 ceph-mon[35242]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:37.783 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:37.783 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:37.783 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:37.783 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:37.783 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:37.783 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:37 smithi033 ceph-mon[31243]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:37.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:37.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:37 smithi060 ceph-mon[35185]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:38.622 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:14:38.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:38 smithi083 ceph-mon[35242]: Deploying daemon mds.f on smithi083 2024-09-16T00:14:38.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:38 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.60:6834/1002409397,v1:172.21.15.60:6835/1002409397] up:boot 2024-09-16T00:14:38.767 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:38 smithi083 ceph-mon[35242]: fsmap 5 up:standby 2024-09-16T00:14:38.767 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:38 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T00:14:38.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:38 smithi033 ceph-mon[31243]: Deploying daemon mds.f on smithi083 2024-09-16T00:14:38.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:38 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.60:6834/1002409397,v1:172.21.15.60:6835/1002409397] up:boot 2024-09-16T00:14:38.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:38 smithi033 ceph-mon[31243]: fsmap 5 up:standby 2024-09-16T00:14:38.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T00:14:38.897 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:38.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:38 smithi060 ceph-mon[35185]: Deploying daemon mds.f on smithi083 2024-09-16T00:14:38.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:38 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.60:6834/1002409397,v1:172.21.15.60:6835/1002409397] up:boot 2024-09-16T00:14:38.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:38 smithi060 ceph-mon[35185]: fsmap 5 up:standby 2024-09-16T00:14:38.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:38 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T00:14:39.169 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:14:38 smithi083 systemd[1]: Starting Ceph mds.f for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:14:39.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.361+0000 7f62915d3640 1 -- 172.21.15.33:0/3732859245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f628c101280 msgr2=0x7f628c10df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:39.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.361+0000 7f62915d3640 1 --2- 172.21.15.33:0/3732859245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f628c101280 0x7f628c10df40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6280009900 tx=0x7f628002a410 comp rx=0 tx=0).stop 2024-09-16T00:14:39.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.361+0000 7f62915d3640 1 -- 172.21.15.33:0/3732859245 shutdown_connections 2024-09-16T00:14:39.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.361+0000 7f62915d3640 1 --2- 172.21.15.33:0/3732859245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c10e480 0x7f628c110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.361+0000 7f62915d3640 1 --2- 172.21.15.33:0/3732859245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f628c101280 0x7f628c10df40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.361+0000 7f62915d3640 1 --2- 172.21.15.33:0/3732859245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c1008d0 0x7f628c100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.361+0000 7f62915d3640 1 -- 172.21.15.33:0/3732859245 >> 172.21.15.33:0/3732859245 conn(0x7f628c0fc7a0 msgr2=0x7f628c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.361+0000 7f62915d3640 1 -- 172.21.15.33:0/3732859245 shutdown_connections 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.361+0000 7f62915d3640 1 -- 172.21.15.33:0/3732859245 wait complete. 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.362+0000 7f62915d3640 1 Processor -- start 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.362+0000 7f62915d3640 1 -- start start 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.362+0000 7f62915d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f628c1008d0 0x7f628c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.362+0000 7f62915d3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c101280 0x7f628c19ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.362+0000 7f62915d3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c10e480 0x7f628c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.362+0000 7f62915d3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f628c114190 con 0x7f628c1008d0 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.362+0000 7f62915d3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f628c114010 con 0x7f628c10e480 2024-09-16T00:14:39.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.362+0000 7f62915d3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f628c114310 con 0x7f628c101280 2024-09-16T00:14:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.362+0000 7f628affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f628c1008d0 0x7f628c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.362+0000 7f628affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f628c1008d0 0x7f628c19a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41704/0 (socket says 172.21.15.33:41704) 2024-09-16T00:14:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.362+0000 7f628affd640 1 -- 172.21.15.33:0/510448758 learned_addr learned my addr 172.21.15.33:0/510448758 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.362+0000 7f628b7fe640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c10e480 0x7f628c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.363+0000 7f628a7fc640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c101280 0x7f628c19ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.363+0000 7f628affd640 1 -- 172.21.15.33:0/510448758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c101280 msgr2=0x7f628c19ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.363+0000 7f628affd640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c101280 0x7f628c19ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.363+0000 7f628affd640 1 -- 172.21.15.33:0/510448758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c10e480 msgr2=0x7f628c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.363+0000 7f628affd640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c10e480 0x7f628c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.363+0000 7f628affd640 1 -- 172.21.15.33:0/510448758 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628c19f680 con 0x7f628c1008d0 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.363+0000 7f628affd640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f628c1008d0 0x7f628c19a6d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f627c00d950 tx=0x7f627c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.363+0000 7f6273fff640 1 -- 172.21.15.33:0/510448758 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c017070 con 0x7f628c1008d0 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.363+0000 7f6273fff640 1 -- 172.21.15.33:0/510448758 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f627c009c30 con 0x7f628c1008d0 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.363+0000 7f62915d3640 1 -- 172.21.15.33:0/510448758 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f628c19f970 con 0x7f628c1008d0 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.363+0000 7f6273fff640 1 -- 172.21.15.33:0/510448758 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c00f6b0 con 0x7f628c1008d0 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.363+0000 7f62915d3640 1 -- 172.21.15.33:0/510448758 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f628c1a7250 con 0x7f628c1008d0 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.364+0000 7f62915d3640 1 -- 172.21.15.33:0/510448758 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f628c100cb0 con 0x7f628c1008d0 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.365+0000 7f6273fff640 1 -- 172.21.15.33:0/510448758 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f627c0109d0 con 0x7f628c1008d0 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.366+0000 7f6273fff640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62740a3240 0x7f62740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.366+0000 7f628a7fc640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62740a3240 0x7f62740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.366+0000 7f6273fff640 1 -- 172.21.15.33:0/510448758 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f627c0cf100 con 0x7f628c1008d0 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.366+0000 7f628a7fc640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62740a3240 0x7f62740a5700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f628c1022e0 tx=0x7f6280005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.370+0000 7f6273fff640 1 -- 172.21.15.33:0/510448758 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f627c009de0 con 0x7f628c1008d0 2024-09-16T00:14:39.431 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:14:39 smithi083 podman[64746]: 2024-09-16 00:14:39.177210091 +0000 UTC m=+0.090573766 container create 2bb32a434d0ad4ca3d847bf74ab6604e0df2b8d28e4ad7f2f972ee5fee065f68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-f, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-16T00:14:39.431 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:14:39 smithi083 podman[64746]: 2024-09-16 00:14:39.102255533 +0000 UTC m=+0.015619205 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:14:39.431 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:14:39 smithi083 podman[64746]: 2024-09-16 00:14:39.404765367 +0000 UTC m=+0.318129040 container init 2bb32a434d0ad4ca3d847bf74ab6604e0df2b8d28e4ad7f2f972ee5fee065f68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-f, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True) 2024-09-16T00:14:39.431 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:14:39 smithi083 podman[64746]: 2024-09-16 00:14:39.407687105 +0000 UTC m=+0.321050789 container start 2bb32a434d0ad4ca3d847bf74ab6604e0df2b8d28e4ad7f2f972ee5fee065f68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-f, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=squid-8921c8e) 2024-09-16T00:14:39.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:39 smithi033 ceph-mon[31243]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:39.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.584+0000 7f62915d3640 1 -- 172.21.15.33:0/510448758 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f628c109670 con 0x7f628c1008d0 2024-09-16T00:14:39.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.585+0000 7f6273fff640 1 -- 172.21.15.33:0/510448758 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5560 (secure 0 0 0) 0x7f627c09c2c0 con 0x7f628c1008d0 2024-09-16T00:14:39.586 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:39.586 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":7,"btime":"2024-09-16T00:14:39:493752+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-09-16T00:14:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 7 2024-09-16T00:14:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.588+0000 7f62915d3640 1 -- 172.21.15.33:0/510448758 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62740a3240 msgr2=0x7f62740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.588+0000 7f62915d3640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62740a3240 0x7f62740a5700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f628c1022e0 tx=0x7f6280005a20 comp rx=0 tx=0).stop 2024-09-16T00:14:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.588+0000 7f62915d3640 1 -- 172.21.15.33:0/510448758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f628c1008d0 msgr2=0x7f628c19a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.588+0000 7f62915d3640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f628c1008d0 0x7f628c19a6d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f627c00d950 tx=0x7f627c00de10 comp rx=0 tx=0).stop 2024-09-16T00:14:39.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.588+0000 7f62915d3640 1 -- 172.21.15.33:0/510448758 shutdown_connections 2024-09-16T00:14:39.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.588+0000 7f62915d3640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62740a3240 0x7f62740a5700 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:39.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.588+0000 7f62915d3640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c10e480 0x7f628c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:39.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.588+0000 7f62915d3640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c101280 0x7f628c19ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:39.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.588+0000 7f62915d3640 1 --2- 172.21.15.33:0/510448758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f628c1008d0 0x7f628c19a6d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:39.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.588+0000 7f62915d3640 1 -- 172.21.15.33:0/510448758 >> 172.21.15.33:0/510448758 conn(0x7f628c0fc7a0 msgr2=0x7f628c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:39.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.588+0000 7f62915d3640 1 -- 172.21.15.33:0/510448758 shutdown_connections 2024-09-16T00:14:39.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:39.588+0000 7f62915d3640 1 -- 172.21.15.33:0/510448758 wait complete. 2024-09-16T00:14:39.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:39 smithi060 ceph-mon[35185]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:39 smithi083 ceph-mon[35242]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:39.919 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:14:39 smithi083 ceph-mds[64765]: -- 172.21.15.83:0/3449031542 <== mon.1 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562fe0617860 con 0x562fe05f5400 2024-09-16T00:14:39.919 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:14:39 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-f[64761]: starting mds.f at 2024-09-16T00:14:39.919 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:14:39 smithi083 bash[64746]: 2bb32a434d0ad4ca3d847bf74ab6604e0df2b8d28e4ad7f2f972ee5fee065f68 2024-09-16T00:14:39.919 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:14:39 smithi083 systemd[1]: Started Ceph mds.f for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:14:40.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:40 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.83:6834/2667354509,v1:172.21.15.83:6835/2667354509] up:boot 2024-09-16T00:14:40.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:40 smithi033 ceph-mon[31243]: fsmap 6 up:standby 2024-09-16T00:14:40.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:40 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T00:14:40.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:40 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:40.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:40 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:40.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:40 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:40.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:40 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:40.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:40 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:40.585 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:40 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:40.585 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:40 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/510448758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:40.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:40 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.83:6834/2667354509,v1:172.21.15.83:6835/2667354509] up:boot 2024-09-16T00:14:40.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:40 smithi083 ceph-mon[35242]: fsmap 6 up:standby 2024-09-16T00:14:40.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:40 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T00:14:40.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:40 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:40.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:40 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:40.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:40 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:40.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:40 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:40.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:40 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:40.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:40 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:40.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:40 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/510448758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:40.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:40 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.83:6834/2667354509,v1:172.21.15.83:6835/2667354509] up:boot 2024-09-16T00:14:40.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:40 smithi060 ceph-mon[35185]: fsmap 6 up:standby 2024-09-16T00:14:40.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:40 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T00:14:40.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:40 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:40.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:40 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:40.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:40 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:40.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:40 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:40.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:40 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:40.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:40 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:40.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:40 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/510448758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:41.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:41 smithi083 ceph-mon[35242]: Deploying daemon mds.c on smithi083 2024-09-16T00:14:41.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:41 smithi083 ceph-mon[35242]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:41.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:41 smithi033 ceph-mon[31243]: Deploying daemon mds.c on smithi083 2024-09-16T00:14:41.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:41 smithi033 ceph-mon[31243]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:41 smithi060 ceph-mon[35185]: Deploying daemon mds.c on smithi083 2024-09-16T00:14:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:41 smithi060 ceph-mon[35185]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:42.307 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:14:42.418 INFO:journalctl@ceph.mds.c.smithi083.stdout:Sep 16 00:14:42 smithi083 systemd[1]: Started Ceph mds.c for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:14:42.591 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:43.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.088+0000 7fd6c32e5640 1 -- 172.21.15.33:0/634148488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6bc108290 msgr2=0x7fd6bc10ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:43.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.088+0000 7fd6c32e5640 1 --2- 172.21.15.33:0/634148488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6bc108290 0x7fd6bc10ec30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4009cd0 tx=0x7fd6b402a8e0 comp rx=0 tx=0).stop 2024-09-16T00:14:43.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.088+0000 7fd6c32e5640 1 -- 172.21.15.33:0/634148488 shutdown_connections 2024-09-16T00:14:43.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.088+0000 7fd6c32e5640 1 --2- 172.21.15.33:0/634148488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6bc108290 0x7fd6bc10ec30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:43.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.088+0000 7fd6c32e5640 1 --2- 172.21.15.33:0/634148488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6bc103740 0x7fd6bc107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:43.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.088+0000 7fd6c32e5640 1 --2- 172.21.15.33:0/634148488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6bc102d90 0x7fd6bc103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:43.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.088+0000 7fd6c32e5640 1 -- 172.21.15.33:0/634148488 >> 172.21.15.33:0/634148488 conn(0x7fd6bc0fc7d0 msgr2=0x7fd6bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:43.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.088+0000 7fd6c32e5640 1 -- 172.21.15.33:0/634148488 shutdown_connections 2024-09-16T00:14:43.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.088+0000 7fd6c32e5640 1 -- 172.21.15.33:0/634148488 wait complete. 2024-09-16T00:14:43.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.089+0000 7fd6c32e5640 1 Processor -- start 2024-09-16T00:14:43.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.089+0000 7fd6c32e5640 1 -- start start 2024-09-16T00:14:43.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.089+0000 7fd6c32e5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6bc102d90 0x7fd6bc1009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:43.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.089+0000 7fd6c32e5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6bc103740 0x7fd6bc100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:43.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.089+0000 7fd6c32e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6bc108290 0x7fd6bc101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.089+0000 7fd6c32e5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc112090 con 0x7fd6bc102d90 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.089+0000 7fd6c32e5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc111f10 con 0x7fd6bc108290 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.089+0000 7fd6c32e5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc112210 con 0x7fd6bc103740 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.089+0000 7fd6c1081640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6bc102d90 0x7fd6bc1009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c1081640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6bc102d90 0x7fd6bc1009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38768/0 (socket says 172.21.15.33:38768) 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c1081640 1 -- 172.21.15.33:0/315395402 learned_addr learned my addr 172.21.15.33:0/315395402 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c0880640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6bc103740 0x7fd6bc100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c1882640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6bc108290 0x7fd6bc101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c1081640 1 -- 172.21.15.33:0/315395402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6bc103740 msgr2=0x7fd6bc100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c1081640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6bc103740 0x7fd6bc100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c1081640 1 -- 172.21.15.33:0/315395402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6bc108290 msgr2=0x7fd6bc101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c1081640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6bc108290 0x7fd6bc101470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c1081640 1 -- 172.21.15.33:0/315395402 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6bc1a6a20 con 0x7fd6bc102d90 2024-09-16T00:14:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c0880640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6bc103740 0x7fd6bc100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c1882640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6bc108290 0x7fd6bc101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c1081640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6bc102d90 0x7fd6bc1009f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800e9f0 tx=0x7fd6b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6b27fc640 1 -- 172.21.15.33:0/315395402 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b800cd80 con 0x7fd6bc102d90 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6b27fc640 1 -- 172.21.15.33:0/315395402 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6b8010910 con 0x7fd6bc102d90 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6b27fc640 1 -- 172.21.15.33:0/315395402 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8018990 con 0x7fd6bc102d90 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c32e5640 1 -- 172.21.15.33:0/315395402 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6bc1a6d10 con 0x7fd6bc102d90 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.090+0000 7fd6c32e5640 1 -- 172.21.15.33:0/315395402 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6bc1a71f0 con 0x7fd6bc102d90 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.091+0000 7fd6c32e5640 1 -- 172.21.15.33:0/315395402 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6bc103170 con 0x7fd6bc102d90 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.093+0000 7fd6b27fc640 1 -- 172.21.15.33:0/315395402 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fd6b8010ac0 con 0x7fd6bc102d90 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.093+0000 7fd6b27fc640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6a80a3240 0x7fd6a80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.094+0000 7fd6c0880640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6a80a3240 0x7fd6a80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.094+0000 7fd6b27fc640 1 -- 172.21.15.33:0/315395402 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fd6b80cf780 con 0x7fd6bc102d90 2024-09-16T00:14:43.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.094+0000 7fd6c0880640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6a80a3240 0x7fd6a80a5700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc102480 tx=0x7fd6ac00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:43.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.098+0000 7fd6b27fc640 1 -- 172.21.15.33:0/315395402 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd6b809c2d0 con 0x7fd6bc102d90 2024-09-16T00:14:43.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:43 smithi033 ceph-mon[31243]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:43.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:43.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:43.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:43.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:43.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:43.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:43.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:43 smithi033 ceph-mon[31243]: Deploying daemon mds.h on smithi060 2024-09-16T00:14:43.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:43 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] up:boot 2024-09-16T00:14:43.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:43 smithi033 ceph-mon[31243]: fsmap 7 up:standby 2024-09-16T00:14:43.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T00:14:43.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.310+0000 7fd6c32e5640 1 -- 172.21.15.33:0/315395402 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6bc10a2d0 con 0x7fd6bc102d90 2024-09-16T00:14:43.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.310+0000 7fd6b27fc640 1 -- 172.21.15.33:0/315395402 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6385 (secure 0 0 0) 0x7fd6b80a05d0 con 0x7fd6bc102d90 2024-09-16T00:14:43.312 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:43.312 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":8,"btime":"2024-09-16T00:14:42:503075+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-09-16T00:14:43.313 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 8 2024-09-16T00:14:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.314+0000 7fd6c32e5640 1 -- 172.21.15.33:0/315395402 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6a80a3240 msgr2=0x7fd6a80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.314+0000 7fd6c32e5640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6a80a3240 0x7fd6a80a5700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc102480 tx=0x7fd6ac00b410 comp rx=0 tx=0).stop 2024-09-16T00:14:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.314+0000 7fd6c32e5640 1 -- 172.21.15.33:0/315395402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6bc102d90 msgr2=0x7fd6bc1009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.314+0000 7fd6c32e5640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6bc102d90 0x7fd6bc1009f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800e9f0 tx=0x7fd6b800eeb0 comp rx=0 tx=0).stop 2024-09-16T00:14:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.314+0000 7fd6c32e5640 1 -- 172.21.15.33:0/315395402 shutdown_connections 2024-09-16T00:14:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.314+0000 7fd6c32e5640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6a80a3240 0x7fd6a80a5700 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:43.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.314+0000 7fd6c32e5640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6bc108290 0x7fd6bc101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:43.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.314+0000 7fd6c32e5640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6bc103740 0x7fd6bc100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:43.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.314+0000 7fd6c32e5640 1 --2- 172.21.15.33:0/315395402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6bc102d90 0x7fd6bc1009f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:43.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.314+0000 7fd6c32e5640 1 -- 172.21.15.33:0/315395402 >> 172.21.15.33:0/315395402 conn(0x7fd6bc0fc7d0 msgr2=0x7fd6bc106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:43.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.314+0000 7fd6c32e5640 1 -- 172.21.15.33:0/315395402 shutdown_connections 2024-09-16T00:14:43.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:43.314+0000 7fd6c32e5640 1 -- 172.21.15.33:0/315395402 wait complete. 2024-09-16T00:14:43.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:43 smithi083 ceph-mon[35242]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:43 smithi083 ceph-mon[35242]: Deploying daemon mds.h on smithi060 2024-09-16T00:14:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:43 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] up:boot 2024-09-16T00:14:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:43 smithi083 ceph-mon[35242]: fsmap 7 up:standby 2024-09-16T00:14:43.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T00:14:43.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:43 smithi060 ceph-mon[35185]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:43 smithi060 ceph-mon[35185]: Deploying daemon mds.h on smithi060 2024-09-16T00:14:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:43 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] up:boot 2024-09-16T00:14:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:43 smithi060 ceph-mon[35185]: fsmap 7 up:standby 2024-09-16T00:14:43.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T00:14:44.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:44 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/315395402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:44.405 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:44 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/315395402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:44.405 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:14:44 smithi060 systemd[1]: Starting Ceph mds.h for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:14:44.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:44 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/315395402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:44.705 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:14:44 smithi060 podman[65263]: 2024-09-16 00:14:44.307299832 +0000 UTC m=+0.016016865 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:14:44.705 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:14:44 smithi060 podman[65263]: 2024-09-16 00:14:44.418663516 +0000 UTC m=+0.127380549 container create 4b6511ad6bcd02c165f491c012c6cb2a5a4c1df50a018a13881c0c2e27a3470a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-16T00:14:44.705 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:14:44 smithi060 podman[65263]: 2024-09-16 00:14:44.704898303 +0000 UTC m=+0.413615338 container init 4b6511ad6bcd02c165f491c012c6cb2a5a4c1df50a018a13881c0c2e27a3470a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-16T00:14:44.981 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:14:44 smithi060 podman[65263]: 2024-09-16 00:14:44.70785708 +0000 UTC m=+0.416574112 container start 4b6511ad6bcd02c165f491c012c6cb2a5a4c1df50a018a13881c0c2e27a3470a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-h, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-16T00:14:44.981 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:14:44 smithi060 ceph-mds[65282]: -- 172.21.15.60:0/2699292163 <== mon.2 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b019797860 con 0x55b019775400 2024-09-16T00:14:44.981 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:14:44 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-h[65278]: starting mds.h at 2024-09-16T00:14:44.981 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:14:44 smithi060 bash[65263]: 4b6511ad6bcd02c165f491c012c6cb2a5a4c1df50a018a13881c0c2e27a3470a 2024-09-16T00:14:44.981 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:14:44 smithi060 systemd[1]: Started Ceph mds.h for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:14:45.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:45 smithi083 ceph-mon[35242]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:45.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:45 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:45.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:45 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:45.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:45 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:45.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:45 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:45.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:45 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:45.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:45 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:45.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:45 smithi060 ceph-mon[35185]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:45.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:45 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:45.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:45 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:45.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:45 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:45.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:45 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:45.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:45 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:45.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:45 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:45.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:45 smithi033 ceph-mon[31243]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:45.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:45 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:45.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:45 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:45.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:45 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:45.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:45 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:45.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:45 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:45.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:45 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:46.022 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:14:46.296 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:46.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:46 smithi033 ceph-mon[31243]: Deploying daemon mds.i on smithi083 2024-09-16T00:14:46.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:46 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.60:6836/452012251,v1:172.21.15.60:6837/452012251] up:boot 2024-09-16T00:14:46.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:46 smithi033 ceph-mon[31243]: fsmap 8 up:standby 2024-09-16T00:14:46.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:46 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T00:14:46.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:46 smithi083 ceph-mon[35242]: Deploying daemon mds.i on smithi083 2024-09-16T00:14:46.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:46 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.60:6836/452012251,v1:172.21.15.60:6837/452012251] up:boot 2024-09-16T00:14:46.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:46 smithi083 ceph-mon[35242]: fsmap 8 up:standby 2024-09-16T00:14:46.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:46 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T00:14:46.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:46 smithi060 ceph-mon[35185]: Deploying daemon mds.i on smithi083 2024-09-16T00:14:46.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:46 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.60:6836/452012251,v1:172.21.15.60:6837/452012251] up:boot 2024-09-16T00:14:46.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:46 smithi060 ceph-mon[35185]: fsmap 8 up:standby 2024-09-16T00:14:46.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:46 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T00:14:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.787+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/1525967209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7410e4f0 msgr2=0x7fdc74110900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.787+0000 7fdc7bb2e640 1 --2- 172.21.15.33:0/1525967209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7410e4f0 0x7fdc74110900 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c009900 tx=0x7fdc6c02a410 comp rx=0 tx=0).stop 2024-09-16T00:14:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.787+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/1525967209 shutdown_connections 2024-09-16T00:14:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.787+0000 7fdc7bb2e640 1 --2- 172.21.15.33:0/1525967209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7410e4f0 0x7fdc74110900 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.787+0000 7fdc7bb2e640 1 --2- 172.21.15.33:0/1525967209 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc74101380 0x7fdc7410dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.787+0000 7fdc7bb2e640 1 --2- 172.21.15.33:0/1525967209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc741009d0 0x7fdc74100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.787+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/1525967209 >> 172.21.15.33:0/1525967209 conn(0x7fdc740fc7d0 msgr2=0x7fdc740febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.787+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/1525967209 shutdown_connections 2024-09-16T00:14:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.787+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/1525967209 wait complete. 2024-09-16T00:14:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.787+0000 7fdc7bb2e640 1 Processor -- start 2024-09-16T00:14:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.787+0000 7fdc7bb2e640 1 -- start start 2024-09-16T00:14:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7bb2e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc741009d0 0x7fdc7419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7bb2e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc74101380 0x7fdc7419ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7bb2e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7410e4f0 0x7fdc7419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7bb2e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdc74114250 con 0x7fdc7410e4f0 2024-09-16T00:14:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7bb2e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdc741140d0 con 0x7fdc74101380 2024-09-16T00:14:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7bb2e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdc741143d0 con 0x7fdc741009d0 2024-09-16T00:14:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc798ca640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc741009d0 0x7fdc7419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7a0cb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7410e4f0 0x7fdc7419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc798ca640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc741009d0 0x7fdc7419a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60704/0 (socket says 172.21.15.33:60704) 2024-09-16T00:14:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc798ca640 1 -- 172.21.15.33:0/223960992 learned_addr learned my addr 172.21.15.33:0/223960992 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc790c9640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc74101380 0x7fdc7419ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7a0cb640 1 -- 172.21.15.33:0/223960992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc741009d0 msgr2=0x7fdc7419a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7a0cb640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc741009d0 0x7fdc7419a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7a0cb640 1 -- 172.21.15.33:0/223960992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc74101380 msgr2=0x7fdc7419ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7a0cb640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc74101380 0x7fdc7419ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7a0cb640 1 -- 172.21.15.33:0/223960992 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7419f690 con 0x7fdc7410e4f0 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7a0cb640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7410e4f0 0x7fdc7419efb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c009230 tx=0x7fdc6c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc72ffd640 1 -- 172.21.15.33:0/223960992 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c03f030 con 0x7fdc7410e4f0 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/223960992 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc7419f920 con 0x7fdc7410e4f0 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc72ffd640 1 -- 172.21.15.33:0/223960992 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc6c003ad0 con 0x7fdc7410e4f0 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.788+0000 7fdc72ffd640 1 -- 172.21.15.33:0/223960992 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c035070 con 0x7fdc7410e4f0 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.789+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/223960992 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc741a7260 con 0x7fdc7410e4f0 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.789+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/223960992 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc7406a7d0 con 0x7fdc7410e4f0 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.791+0000 7fdc72ffd640 1 -- 172.21.15.33:0/223960992 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fdc6c042030 con 0x7fdc7410e4f0 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.791+0000 7fdc72ffd640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdc5c0a3240 0x7fdc5c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.791+0000 7fdc798ca640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdc5c0a3240 0x7fdc5c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.791+0000 7fdc72ffd640 1 -- 172.21.15.33:0/223960992 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fdc6c0efa70 con 0x7fdc7410e4f0 2024-09-16T00:14:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.792+0000 7fdc798ca640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdc5c0a3240 0x7fdc5c0a5700 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdc640104e0 tx=0x7fdc64009800 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:46.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:46.796+0000 7fdc72ffd640 1 -- 172.21.15.33:0/223960992 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdc6c0bc5e0 con 0x7fdc7410e4f0 2024-09-16T00:14:47.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.006+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/223960992 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc74109680 con 0x7fdc7410e4f0 2024-09-16T00:14:47.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.006+0000 7fdc72ffd640 1 -- 172.21.15.33:0/223960992 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7207 (secure 0 0 0) 0x7fdc6c0c08e0 con 0x7fdc7410e4f0 2024-09-16T00:14:47.008 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:47.008 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":9,"btime":"2024-09-16T00:14:45:108018+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-09-16T00:14:47.009 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 9 2024-09-16T00:14:47.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.008+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/223960992 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdc5c0a3240 msgr2=0x7fdc5c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:47.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.008+0000 7fdc7bb2e640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdc5c0a3240 0x7fdc5c0a5700 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdc640104e0 tx=0x7fdc64009800 comp rx=0 tx=0).stop 2024-09-16T00:14:47.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.008+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/223960992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7410e4f0 msgr2=0x7fdc7419efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:47.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.008+0000 7fdc7bb2e640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7410e4f0 0x7fdc7419efb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c009230 tx=0x7fdc6c004270 comp rx=0 tx=0).stop 2024-09-16T00:14:47.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.009+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/223960992 shutdown_connections 2024-09-16T00:14:47.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.009+0000 7fdc7bb2e640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdc5c0a3240 0x7fdc5c0a5700 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:47.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.009+0000 7fdc7bb2e640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7410e4f0 0x7fdc7419efb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:47.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.009+0000 7fdc7bb2e640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc74101380 0x7fdc7419ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:47.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.009+0000 7fdc7bb2e640 1 --2- 172.21.15.33:0/223960992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc741009d0 0x7fdc7419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:47.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.009+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/223960992 >> 172.21.15.33:0/223960992 conn(0x7fdc740fc7d0 msgr2=0x7fdc740febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:47.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.009+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/223960992 shutdown_connections 2024-09-16T00:14:47.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:47.009+0000 7fdc7bb2e640 1 -- 172.21.15.33:0/223960992 wait complete. 2024-09-16T00:14:47.121 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:14:47 smithi083 podman[65349]: 2024-09-16 00:14:47.033145372 +0000 UTC m=+0.128437653 container create 8b79c590acfae9b681cb1ee3798345307be38699bb9525336b1ca5d0508d6b84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-16T00:14:47.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:47 smithi033 ceph-mon[31243]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:47.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:47 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/223960992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:47.371 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:47 smithi083 ceph-mon[35242]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:47.371 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:47 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/223960992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:47.372 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:14:47 smithi083 podman[65349]: 2024-09-16 00:14:47.26952426 +0000 UTC m=+0.364816543 container init 8b79c590acfae9b681cb1ee3798345307be38699bb9525336b1ca5d0508d6b84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-16T00:14:47.372 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:14:47 smithi083 podman[65349]: 2024-09-16 00:14:47.272285957 +0000 UTC m=+0.367578235 container start 8b79c590acfae9b681cb1ee3798345307be38699bb9525336b1ca5d0508d6b84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux ) 2024-09-16T00:14:47.372 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:14:47 smithi083 ceph-mds[65368]: -- 172.21.15.83:0/1439252914 <== mon.1 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5559b474f860 con 0x5559b472c800 2024-09-16T00:14:47.372 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:14:47 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i[65364]: starting mds.i at 2024-09-16T00:14:47.372 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:14:47 smithi083 bash[65349]: 8b79c590acfae9b681cb1ee3798345307be38699bb9525336b1ca5d0508d6b84 2024-09-16T00:14:47.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:47 smithi060 ceph-mon[35185]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:47.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:47 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/223960992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:47.668 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:14:47 smithi083 systemd[1]: Started Ceph mds.i for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:14:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:48 smithi033 ceph-mon[31243]: Deploying daemon mds.g on smithi033 2024-09-16T00:14:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:48 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] up:boot 2024-09-16T00:14:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:48 smithi033 ceph-mon[31243]: fsmap 9 up:standby 2024-09-16T00:14:48.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:48 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T00:14:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:48.934 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:48.934 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:48.934 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:48.934 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:48.934 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:48.934 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:48 smithi060 ceph-mon[35185]: Deploying daemon mds.g on smithi033 2024-09-16T00:14:48.934 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:48 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] up:boot 2024-09-16T00:14:48.934 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:48 smithi060 ceph-mon[35185]: fsmap 9 up:standby 2024-09-16T00:14:48.934 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:48 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T00:14:48.935 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:48.935 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:48.935 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:48.935 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:48.935 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:48.936 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:48.936 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:48 smithi083 ceph-mon[35242]: Deploying daemon mds.g on smithi033 2024-09-16T00:14:48.936 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:48 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] up:boot 2024-09-16T00:14:48.936 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:48 smithi083 ceph-mon[35242]: fsmap 9 up:standby 2024-09-16T00:14:48.936 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:48 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T00:14:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:49 smithi033 ceph-mon[31243]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:49.580 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:14:49 smithi033 systemd[1]: Starting Ceph mds.g for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:14:49.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:49 smithi060 ceph-mon[35185]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:49.756 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:14:49.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:49 smithi083 ceph-mon[35242]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:49.967 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:14:49 smithi033 podman[69146]: 2024-09-16 00:14:49.603188548 +0000 UTC m=+0.016307259 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:14:49.967 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:14:49 smithi033 podman[69146]: 2024-09-16 00:14:49.722277606 +0000 UTC m=+0.135396305 container create 093a62828f9a2921f7182c120d9c7bc7104dd1faa113c82ab998ae14c86f13ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-16T00:14:49.967 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:14:49 smithi033 podman[69146]: 2024-09-16 00:14:49.966961528 +0000 UTC m=+0.380080239 container init 093a62828f9a2921f7182c120d9c7bc7104dd1faa113c82ab998ae14c86f13ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-g, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-16T00:14:50.155 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:50.278 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:14:49 smithi033 podman[69146]: 2024-09-16 00:14:49.970371756 +0000 UTC m=+0.383490463 container start 093a62828f9a2921f7182c120d9c7bc7104dd1faa113c82ab998ae14c86f13ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-16T00:14:50.278 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:14:49 smithi033 ceph-mds[69196]: -- 172.21.15.33:0/3681456699 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e1286f7860 con 0x55e1286d5000 2024-09-16T00:14:50.278 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:14:49 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-g[69179]: starting mds.g at 2024-09-16T00:14:50.278 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:14:50 smithi033 bash[69146]: 093a62828f9a2921f7182c120d9c7bc7104dd1faa113c82ab998ae14c86f13ce 2024-09-16T00:14:50.278 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:14:50 smithi033 systemd[1]: Started Ceph mds.g for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:14:50.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.660+0000 7f3e50fdd640 1 -- 172.21.15.33:0/3978035132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e4c100d00 msgr2=0x7f3e4c106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:50.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.660+0000 7f3e50fdd640 1 --2- 172.21.15.33:0/3978035132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e4c100d00 0x7f3e4c106840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3e40009930 tx=0x7f3e4002a420 comp rx=0 tx=0).stop 2024-09-16T00:14:50.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.661+0000 7f3e50fdd640 1 -- 172.21.15.33:0/3978035132 shutdown_connections 2024-09-16T00:14:50.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.661+0000 7f3e50fdd640 1 --2- 172.21.15.33:0/3978035132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e4c106f50 0x7f3e4c10eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:50.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.661+0000 7f3e50fdd640 1 --2- 172.21.15.33:0/3978035132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e4c100d00 0x7f3e4c106840 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:50.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.661+0000 7f3e50fdd640 1 --2- 172.21.15.33:0/3978035132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4c100350 0x7f3e4c100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:50.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.661+0000 7f3e50fdd640 1 -- 172.21.15.33:0/3978035132 >> 172.21.15.33:0/3978035132 conn(0x7f3e4c077210 msgr2=0x7f3e4c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:50.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.661+0000 7f3e50fdd640 1 -- 172.21.15.33:0/3978035132 shutdown_connections 2024-09-16T00:14:50.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.661+0000 7f3e50fdd640 1 -- 172.21.15.33:0/3978035132 wait complete. 2024-09-16T00:14:50.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.661+0000 7f3e50fdd640 1 Processor -- start 2024-09-16T00:14:50.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e50fdd640 1 -- start start 2024-09-16T00:14:50.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e50fdd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e4c100350 0x7f3e4c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:50.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e50fdd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e4c100d00 0x7f3e4c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:50.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e50fdd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4c106f50 0x7f3e4c19cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:50.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e50fdd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c112030 con 0x7f3e4c100d00 2024-09-16T00:14:50.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e50fdd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c111eb0 con 0x7f3e4c100350 2024-09-16T00:14:50.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e50fdd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1121b0 con 0x7f3e4c106f50 2024-09-16T00:14:50.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e49d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e4c100d00 0x7f3e4c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:50.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e49d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e4c100d00 0x7f3e4c198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34604/0 (socket says 172.21.15.33:34604) 2024-09-16T00:14:50.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e49d9b640 1 -- 172.21.15.33:0/754208226 learned_addr learned my addr 172.21.15.33:0/754208226 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:50.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e4a59c640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e4c100350 0x7f3e4c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:50.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e49d9b640 1 -- 172.21.15.33:0/754208226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4c106f50 msgr2=0x7f3e4c19cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:50.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e4ad9d640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4c106f50 0x7f3e4c19cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:50.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e49d9b640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4c106f50 0x7f3e4c19cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:50.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e49d9b640 1 -- 172.21.15.33:0/754208226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e4c100350 msgr2=0x7f3e4c198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:50.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e49d9b640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e4c100350 0x7f3e4c198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e49d9b640 1 -- 172.21.15.33:0/754208226 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4c19d4c0 con 0x7f3e4c100d00 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e4ad9d640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4c106f50 0x7f3e4c19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e49d9b640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e4c100d00 0x7f3e4c198a50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3e4002a940 tx=0x7f3e40003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.662+0000 7f3e3b7fe640 1 -- 172.21.15.33:0/754208226 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4003f070 con 0x7f3e4c100d00 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.663+0000 7f3e50fdd640 1 -- 172.21.15.33:0/754208226 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4c19d750 con 0x7f3e4c100d00 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.663+0000 7f3e3b7fe640 1 -- 172.21.15.33:0/754208226 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e40031c90 con 0x7f3e4c100d00 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.663+0000 7f3e3b7fe640 1 -- 172.21.15.33:0/754208226 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40035030 con 0x7f3e4c100d00 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.663+0000 7f3e50fdd640 1 -- 172.21.15.33:0/754208226 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e4c1a5090 con 0x7f3e4c100d00 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.664+0000 7f3e50fdd640 1 -- 172.21.15.33:0/754208226 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e4c100730 con 0x7f3e4c100d00 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.665+0000 7f3e3b7fe640 1 -- 172.21.15.33:0/754208226 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f3e40042070 con 0x7f3e4c100d00 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.665+0000 7f3e3b7fe640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3e340a3240 0x7f3e340a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.666+0000 7f3e3b7fe640 1 -- 172.21.15.33:0/754208226 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f3e400f02f0 con 0x7f3e4c100d00 2024-09-16T00:14:50.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.666+0000 7f3e4a59c640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3e340a3240 0x7f3e340a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:50.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.666+0000 7f3e4a59c640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3e340a3240 0x7f3e340a5700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00a400 tx=0x7f3e3c00d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:50.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.668+0000 7f3e3b7fe640 1 -- 172.21.15.33:0/754208226 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3e400bd4b0 con 0x7f3e4c100d00 2024-09-16T00:14:50.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.901+0000 7f3e50fdd640 1 -- 172.21.15.33:0/754208226 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e4c199870 con 0x7f3e4c100d00 2024-09-16T00:14:50.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.902+0000 7f3e3b7fe640 1 -- 172.21.15.33:0/754208226 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8860 (secure 0 0 0) 0x7f3e4c199870 con 0x7f3e4c100d00 2024-09-16T00:14:50.904 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:50.904 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":11,"btime":"2024-09-16T00:14:50:437333+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-16T00:14:50.904 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 11 2024-09-16T00:14:50.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.904+0000 7f3e50fdd640 1 -- 172.21.15.33:0/754208226 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3e340a3240 msgr2=0x7f3e340a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:50.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.904+0000 7f3e50fdd640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3e340a3240 0x7f3e340a5700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00a400 tx=0x7f3e3c00d040 comp rx=0 tx=0).stop 2024-09-16T00:14:50.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.904+0000 7f3e50fdd640 1 -- 172.21.15.33:0/754208226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e4c100d00 msgr2=0x7f3e4c198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:50.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.904+0000 7f3e50fdd640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e4c100d00 0x7f3e4c198a50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3e4002a940 tx=0x7f3e40003b90 comp rx=0 tx=0).stop 2024-09-16T00:14:50.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.904+0000 7f3e50fdd640 1 -- 172.21.15.33:0/754208226 shutdown_connections 2024-09-16T00:14:50.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.904+0000 7f3e50fdd640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3e340a3240 0x7f3e340a5700 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:50.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.904+0000 7f3e50fdd640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4c106f50 0x7f3e4c19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:50.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.904+0000 7f3e50fdd640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e4c100d00 0x7f3e4c198a50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:50.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.904+0000 7f3e50fdd640 1 --2- 172.21.15.33:0/754208226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e4c100350 0x7f3e4c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:50.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.904+0000 7f3e50fdd640 1 -- 172.21.15.33:0/754208226 >> 172.21.15.33:0/754208226 conn(0x7f3e4c077210 msgr2=0x7f3e4c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:50.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.904+0000 7f3e50fdd640 1 -- 172.21.15.33:0/754208226 shutdown_connections 2024-09-16T00:14:50.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:50.904+0000 7f3e50fdd640 1 -- 172.21.15.33:0/754208226 wait complete. 2024-09-16T00:14:51.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:51 smithi060 ceph-mon[35185]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:51.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:51.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:51.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:51.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:51.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:51.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:51.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:51 smithi060 ceph-mon[35185]: Deploying daemon mds.a on smithi033 2024-09-16T00:14:51.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:51 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.33:6838/4092505131,v1:172.21.15.33:6839/4092505131] up:boot 2024-09-16T00:14:51.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:51 smithi060 ceph-mon[35185]: fsmap 10 up:standby 2024-09-16T00:14:51.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:51 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-16T00:14:51.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/754208226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:51 smithi083 ceph-mon[35242]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:51 smithi083 ceph-mon[35242]: Deploying daemon mds.a on smithi033 2024-09-16T00:14:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:51 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.33:6838/4092505131,v1:172.21.15.33:6839/4092505131] up:boot 2024-09-16T00:14:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:51 smithi083 ceph-mon[35242]: fsmap 10 up:standby 2024-09-16T00:14:51.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:51 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-16T00:14:51.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:51 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/754208226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:51.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:51 smithi033 ceph-mon[31243]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:51.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:51.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:51.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:51.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:51.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:51.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:51.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:51 smithi033 ceph-mon[31243]: Deploying daemon mds.a on smithi033 2024-09-16T00:14:51.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:51 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.33:6838/4092505131,v1:172.21.15.33:6839/4092505131] up:boot 2024-09-16T00:14:51.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:51 smithi033 ceph-mon[31243]: fsmap 10 up:standby 2024-09-16T00:14:51.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:51 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-16T00:14:51.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:51 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/754208226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:53.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:53 smithi033 ceph-mon[31243]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:53.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:53 smithi083 ceph-mon[35242]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:53.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:53 smithi060 ceph-mon[35185]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:53.713 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:14:53.979 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:54.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.493+0000 7f9d0ff72640 1 -- 172.21.15.33:0/1841113334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d08112b90 msgr2=0x7f9d08114f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:54.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.493+0000 7f9d0ff72640 1 --2- 172.21.15.33:0/1841113334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d08112b90 0x7f9d08114f80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009ca0 tx=0x7f9d0002a950 comp rx=0 tx=0).stop 2024-09-16T00:14:54.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.494+0000 7f9d0ff72640 1 -- 172.21.15.33:0/1841113334 shutdown_connections 2024-09-16T00:14:54.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.494+0000 7f9d0ff72640 1 --2- 172.21.15.33:0/1841113334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d08112b90 0x7f9d08114f80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:54.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.494+0000 7f9d0ff72640 1 --2- 172.21.15.33:0/1841113334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d08074eb0 0x7f9d08112650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:54.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.494+0000 7f9d0ff72640 1 --2- 172.21.15.33:0/1841113334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d08074590 0x7f9d08074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:54.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.494+0000 7f9d0ff72640 1 -- 172.21.15.33:0/1841113334 >> 172.21.15.33:0/1841113334 conn(0x7f9d080fd8c0 msgr2=0x7f9d080ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:54.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.494+0000 7f9d0ff72640 1 -- 172.21.15.33:0/1841113334 shutdown_connections 2024-09-16T00:14:54.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.494+0000 7f9d0ff72640 1 -- 172.21.15.33:0/1841113334 wait complete. 2024-09-16T00:14:54.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.494+0000 7f9d0ff72640 1 Processor -- start 2024-09-16T00:14:54.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.494+0000 7f9d0ff72640 1 -- start start 2024-09-16T00:14:54.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0ff72640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d08074590 0x7f9d081a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:54.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0ff72640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d08074eb0 0x7f9d081a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:54.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0ff72640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d08112b90 0x7f9d0819e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:54.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0ff72640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9d081185b0 con 0x7f9d08074590 2024-09-16T00:14:54.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0ff72640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9d08118430 con 0x7f9d08112b90 2024-09-16T00:14:54.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0ff72640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9d08118730 con 0x7f9d08074eb0 2024-09-16T00:14:54.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0dd0e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d08074590 0x7f9d081a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0dd0e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d08074590 0x7f9d081a4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34646/0 (socket says 172.21.15.33:34646) 2024-09-16T00:14:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0dd0e640 1 -- 172.21.15.33:0/3437180714 learned_addr learned my addr 172.21.15.33:0/3437180714 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0e50f640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d08112b90 0x7f9d0819e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0d50d640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d08074eb0 0x7f9d081a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0dd0e640 1 -- 172.21.15.33:0/3437180714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d08074eb0 msgr2=0x7f9d081a4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:54.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0dd0e640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d08074eb0 0x7f9d081a4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:54.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0dd0e640 1 -- 172.21.15.33:0/3437180714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d08112b90 msgr2=0x7f9d0819e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:54.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0dd0e640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d08112b90 0x7f9d0819e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:54.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0dd0e640 1 -- 172.21.15.33:0/3437180714 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0819f010 con 0x7f9d08074590 2024-09-16T00:14:54.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0d50d640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d08074eb0 0x7f9d081a4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:14:54.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9d0dd0e640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d08074590 0x7f9d081a4720 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400c890 tx=0x7f9d0400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.495+0000 7f9cfeffd640 1 -- 172.21.15.33:0/3437180714 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04004220 con 0x7f9d08074590 2024-09-16T00:14:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.496+0000 7f9cfeffd640 1 -- 172.21.15.33:0/3437180714 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d040043c0 con 0x7f9d08074590 2024-09-16T00:14:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.496+0000 7f9cfeffd640 1 -- 172.21.15.33:0/3437180714 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d040049e0 con 0x7f9d08074590 2024-09-16T00:14:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.496+0000 7f9d0ff72640 1 -- 172.21.15.33:0/3437180714 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0819f300 con 0x7f9d08074590 2024-09-16T00:14:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.496+0000 7f9d0ff72640 1 -- 172.21.15.33:0/3437180714 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d081ab5f0 con 0x7f9d08074590 2024-09-16T00:14:54.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.497+0000 7f9d0ff72640 1 -- 172.21.15.33:0/3437180714 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d08074970 con 0x7f9d08074590 2024-09-16T00:14:54.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.498+0000 7f9cfeffd640 1 -- 172.21.15.33:0/3437180714 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f9d040079b0 con 0x7f9d08074590 2024-09-16T00:14:54.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.498+0000 7f9cfeffd640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9cf40a32c0 0x7f9cf40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:54.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.498+0000 7f9d0d50d640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9cf40a32c0 0x7f9cf40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:54.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.499+0000 7f9cfeffd640 1 -- 172.21.15.33:0/3437180714 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f9d040cf440 con 0x7f9d08074590 2024-09-16T00:14:54.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.499+0000 7f9d0d50d640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9cf40a32c0 0x7f9cf40a5780 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9d08114c30 tx=0x7f9cf8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:54.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.503+0000 7f9cfeffd640 1 -- 172.21.15.33:0/3437180714 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9d04014030 con 0x7f9d08074590 2024-09-16T00:14:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.712+0000 7f9d0ff72640 1 -- 172.21.15.33:0/3437180714 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d0819fe10 con 0x7f9d08074590 2024-09-16T00:14:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.713+0000 7f9cfeffd640 1 -- 172.21.15.33:0/3437180714 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9686 (secure 0 0 0) 0x7f9d0409c600 con 0x7f9d08074590 2024-09-16T00:14:54.714 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:54.715 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":12,"btime":"2024-09-16T00:14:54:136940+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-16T00:14:54.715 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 12 2024-09-16T00:14:54.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.716+0000 7f9d0ff72640 1 -- 172.21.15.33:0/3437180714 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9cf40a32c0 msgr2=0x7f9cf40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:54.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.716+0000 7f9d0ff72640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9cf40a32c0 0x7f9cf40a5780 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9d08114c30 tx=0x7f9cf8008040 comp rx=0 tx=0).stop 2024-09-16T00:14:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.716+0000 7f9d0ff72640 1 -- 172.21.15.33:0/3437180714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d08074590 msgr2=0x7f9d081a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.716+0000 7f9d0ff72640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d08074590 0x7f9d081a4720 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400c890 tx=0x7f9d0400cd50 comp rx=0 tx=0).stop 2024-09-16T00:14:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.716+0000 7f9d0ff72640 1 -- 172.21.15.33:0/3437180714 shutdown_connections 2024-09-16T00:14:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.716+0000 7f9d0ff72640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9cf40a32c0 0x7f9cf40a5780 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.716+0000 7f9d0ff72640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d08112b90 0x7f9d0819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.716+0000 7f9d0ff72640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d08074eb0 0x7f9d081a4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.716+0000 7f9d0ff72640 1 --2- 172.21.15.33:0/3437180714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d08074590 0x7f9d081a4720 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.716+0000 7f9d0ff72640 1 -- 172.21.15.33:0/3437180714 >> 172.21.15.33:0/3437180714 conn(0x7f9d080fd8c0 msgr2=0x7f9d080ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.716+0000 7f9d0ff72640 1 -- 172.21.15.33:0/3437180714 shutdown_connections 2024-09-16T00:14:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:54.716+0000 7f9d0ff72640 1 -- 172.21.15.33:0/3437180714 wait complete. 2024-09-16T00:14:55.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:54 smithi033 ceph-mon[31243]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:55.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:54 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:55.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:54 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:55.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:54 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:55.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:54 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:55.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:54 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:55.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:54 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:55.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:54 smithi033 ceph-mon[31243]: Deploying daemon mds.b on smithi060 2024-09-16T00:14:55.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:54 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.33:6840/1588024134,v1:172.21.15.33:6841/1588024134] up:boot 2024-09-16T00:14:55.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:54 smithi033 ceph-mon[31243]: fsmap 11 up:standby 2024-09-16T00:14:55.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:54 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-16T00:14:55.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:54 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3437180714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:55.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:54 smithi083 ceph-mon[35242]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:55.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:54 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:55.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:54 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:55.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:54 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:55.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:54 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:55.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:54 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:55.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:54 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:55.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:54 smithi083 ceph-mon[35242]: Deploying daemon mds.b on smithi060 2024-09-16T00:14:55.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:54 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.33:6840/1588024134,v1:172.21.15.33:6841/1588024134] up:boot 2024-09-16T00:14:55.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:54 smithi083 ceph-mon[35242]: fsmap 11 up:standby 2024-09-16T00:14:55.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:54 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-16T00:14:55.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:54 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3437180714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:55.201 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:54 smithi060 ceph-mon[35185]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:55.201 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:54 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:55.201 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:54 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:55.201 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:54 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:55.201 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:54 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T00:14:55.201 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:54 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T00:14:55.202 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:54 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:14:55.202 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:54 smithi060 ceph-mon[35185]: Deploying daemon mds.b on smithi060 2024-09-16T00:14:55.202 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:54 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.33:6840/1588024134,v1:172.21.15.33:6841/1588024134] up:boot 2024-09-16T00:14:55.202 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:54 smithi060 ceph-mon[35185]: fsmap 11 up:standby 2024-09-16T00:14:55.202 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:54 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-16T00:14:55.202 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:54 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3437180714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:56.152 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:14:55 smithi060 systemd[1]: Starting Ceph mds.b for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:14:56.422 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:14:56 smithi060 podman[65579]: 2024-09-16 00:14:56.054287984 +0000 UTC m=+0.016017907 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:14:56.423 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:14:56 smithi060 podman[65579]: 2024-09-16 00:14:56.169716574 +0000 UTC m=+0.131446495 container create 42c946fdaf398492d9f2b4e5ac2611c5613d7efe141d775602e43e5a4d164084 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-16T00:14:56.730 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:14:56 smithi060 podman[65579]: 2024-09-16 00:14:56.422572216 +0000 UTC m=+0.384302150 container init 42c946fdaf398492d9f2b4e5ac2611c5613d7efe141d775602e43e5a4d164084 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-16T00:14:56.730 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:14:56 smithi060 podman[65579]: 2024-09-16 00:14:56.425412944 +0000 UTC m=+0.387142864 container start 42c946fdaf398492d9f2b4e5ac2611c5613d7efe141d775602e43e5a4d164084 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b, ceph=True, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-16T00:14:56.731 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:14:56 smithi060 ceph-mds[65598]: -- 172.21.15.60:0/982320098 <== mon.2 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559f9dc19860 con 0x559f9dbf7400 2024-09-16T00:14:56.731 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:14:56 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b[65594]: starting mds.b at 2024-09-16T00:14:56.731 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:14:56 smithi060 bash[65579]: 42c946fdaf398492d9f2b4e5ac2611c5613d7efe141d775602e43e5a4d164084 2024-09-16T00:14:56.731 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:14:56 smithi060 systemd[1]: Started Ceph mds.b for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:14:57.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:56 smithi033 ceph-mon[31243]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:57.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:56 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:14:57.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:56 smithi083 ceph-mon[35242]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:57.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:56 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:14:57.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:56 smithi060 ceph-mon[35185]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:57.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:57.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:56 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:14:57.446 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:14:57.822 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:57.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:57 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] up:boot 2024-09-16T00:14:57.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:57 smithi033 ceph-mon[31243]: fsmap 12 up:standby 2024-09-16T00:14:57.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T00:14:57.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:57 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:58.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:57 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] up:boot 2024-09-16T00:14:58.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:57 smithi083 ceph-mon[35242]: fsmap 12 up:standby 2024-09-16T00:14:58.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T00:14:58.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:57 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:58.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:57 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] up:boot 2024-09-16T00:14:58.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:57 smithi060 ceph-mon[35185]: fsmap 12 up:standby 2024-09-16T00:14:58.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T00:14:58.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:57 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:58.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.309+0000 7feca9965640 1 -- 172.21.15.33:0/711148307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4102ef0 msgr2=0x7feca410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:58.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.309+0000 7feca9965640 1 --2- 172.21.15.33:0/711148307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4102ef0 0x7feca410da40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fec9c009ca0 tx=0x7fec9c02a950 comp rx=0 tx=0).stop 2024-09-16T00:14:58.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.309+0000 7feca9965640 1 -- 172.21.15.33:0/711148307 shutdown_connections 2024-09-16T00:14:58.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.309+0000 7feca9965640 1 --2- 172.21.15.33:0/711148307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4102ef0 0x7feca410da40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:58.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.309+0000 7feca9965640 1 --2- 172.21.15.33:0/711148307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feca4102570 0x7feca41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:58.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.309+0000 7feca9965640 1 --2- 172.21.15.33:0/711148307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feca4100e20 0x7feca4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:58.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.309+0000 7feca9965640 1 -- 172.21.15.33:0/711148307 >> 172.21.15.33:0/711148307 conn(0x7feca40fc7d0 msgr2=0x7feca40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:58.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.309+0000 7feca9965640 1 -- 172.21.15.33:0/711148307 shutdown_connections 2024-09-16T00:14:58.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.309+0000 7feca9965640 1 -- 172.21.15.33:0/711148307 wait complete. 2024-09-16T00:14:58.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.310+0000 7feca9965640 1 Processor -- start 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.310+0000 7feca9965640 1 -- start start 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca9965640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4100e20 0x7feca419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca9965640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feca4102570 0x7feca419ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca9965640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feca4102ef0 0x7feca419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca9965640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7feca4110f80 con 0x7feca4100e20 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca9965640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7feca4110e00 con 0x7feca4102ef0 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca9965640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feca4111100 con 0x7feca4102570 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4100e20 0x7feca419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4100e20 0x7feca419a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34656/0 (socket says 172.21.15.33:34656) 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca2ffd640 1 -- 172.21.15.33:0/3996812373 learned_addr learned my addr 172.21.15.33:0/3996812373 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca27fc640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feca4102570 0x7feca419ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca37fe640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feca4102ef0 0x7feca419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:58.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca2ffd640 1 -- 172.21.15.33:0/3996812373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feca4102570 msgr2=0x7feca419ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:58.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca2ffd640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feca4102570 0x7feca419ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:58.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca2ffd640 1 -- 172.21.15.33:0/3996812373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feca4102ef0 msgr2=0x7feca419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:58.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca2ffd640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feca4102ef0 0x7feca419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:58.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca2ffd640 1 -- 172.21.15.33:0/3996812373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca419f680 con 0x7feca4100e20 2024-09-16T00:14:58.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca27fc640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feca4102570 0x7feca419ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:14:58.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.311+0000 7feca2ffd640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4100e20 0x7feca419a6d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fec9400ea60 tx=0x7fec9400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:58.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.312+0000 7feca8963640 1 -- 172.21.15.33:0/3996812373 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9400cde0 con 0x7feca4100e20 2024-09-16T00:14:58.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.312+0000 7feca9965640 1 -- 172.21.15.33:0/3996812373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca419f970 con 0x7feca4100e20 2024-09-16T00:14:58.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.312+0000 7feca9965640 1 -- 172.21.15.33:0/3996812373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca41a7250 con 0x7feca4100e20 2024-09-16T00:14:58.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.312+0000 7feca8963640 1 -- 172.21.15.33:0/3996812373 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec94018910 con 0x7feca4100e20 2024-09-16T00:14:58.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.312+0000 7feca8963640 1 -- 172.21.15.33:0/3996812373 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec94021870 con 0x7feca4100e20 2024-09-16T00:14:58.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.313+0000 7feca9965640 1 -- 172.21.15.33:0/3996812373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec90005290 con 0x7feca4100e20 2024-09-16T00:14:58.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.315+0000 7feca8963640 1 -- 172.21.15.33:0/3996812373 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fec94018430 con 0x7feca4100e20 2024-09-16T00:14:58.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.315+0000 7feca8963640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec8c0a3240 0x7fec8c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:14:58.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.316+0000 7feca27fc640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec8c0a3240 0x7fec8c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:14:58.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.316+0000 7feca8963640 1 -- 172.21.15.33:0/3996812373 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fec940cf690 con 0x7feca4100e20 2024-09-16T00:14:58.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.316+0000 7feca27fc640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec8c0a3240 0x7fec8c0a5700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7feca410de80 tx=0x7fec9800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:14:58.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.319+0000 7feca8963640 1 -- 172.21.15.33:0/3996812373 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fec9409c840 con 0x7feca4100e20 2024-09-16T00:14:58.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.541+0000 7feca9965640 1 -- 172.21.15.33:0/3996812373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec90005580 con 0x7feca4100e20 2024-09-16T00:14:58.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.542+0000 7feca8963640 1 -- 172.21.15.33:0/3996812373 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10512 (secure 0 0 0) 0x7fec9409c140 con 0x7feca4100e20 2024-09-16T00:14:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:14:58.543 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":13,"btime":"2024-09-16T00:14:56:735443+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24494,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-09-16T00:14:58.543 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 13 2024-09-16T00:14:58.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.544+0000 7feca9965640 1 -- 172.21.15.33:0/3996812373 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec8c0a3240 msgr2=0x7fec8c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:58.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.544+0000 7feca9965640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec8c0a3240 0x7fec8c0a5700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7feca410de80 tx=0x7fec9800b410 comp rx=0 tx=0).stop 2024-09-16T00:14:58.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.544+0000 7feca9965640 1 -- 172.21.15.33:0/3996812373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4100e20 msgr2=0x7feca419a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:14:58.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.544+0000 7feca9965640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4100e20 0x7feca419a6d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fec9400ea60 tx=0x7fec9400ef20 comp rx=0 tx=0).stop 2024-09-16T00:14:58.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.544+0000 7feca9965640 1 -- 172.21.15.33:0/3996812373 shutdown_connections 2024-09-16T00:14:58.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.544+0000 7feca9965640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec8c0a3240 0x7fec8c0a5700 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:58.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.544+0000 7feca9965640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feca4102ef0 0x7feca419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:58.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.544+0000 7feca9965640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feca4102570 0x7feca419ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:58.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.544+0000 7feca9965640 1 --2- 172.21.15.33:0/3996812373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca4100e20 0x7feca419a6d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:14:58.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.544+0000 7feca9965640 1 -- 172.21.15.33:0/3996812373 >> 172.21.15.33:0/3996812373 conn(0x7feca40fc7d0 msgr2=0x7feca4069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:14:58.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.544+0000 7feca9965640 1 -- 172.21.15.33:0/3996812373 shutdown_connections 2024-09-16T00:14:58.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:14:58.544+0000 7feca9965640 1 -- 172.21.15.33:0/3996812373 wait complete. 2024-09-16T00:14:59.254 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-09-16T00:14:59.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:59 smithi060 ceph-mon[35185]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:59.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:59 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:59.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:59 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:59.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:59 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:59.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:59 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:59.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:14:59 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3996812373' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:59.548 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:14:59.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:59 smithi033 ceph-mon[31243]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:59.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:59 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:59.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:59 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:59.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:59 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:59.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:59 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:59.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:14:59 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3996812373' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:14:59.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:59 smithi083 ceph-mon[35242]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:14:59.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:59 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:59.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:59 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:59.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:59 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:59.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:59 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:14:59.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:14:59 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3996812373' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:15:00.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.049+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1278273903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce0100a60 msgr2=0x7f7ce0100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:00.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.049+0000 7f7ce7a40640 1 --2- 172.21.15.33:0/1278273903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce0100a60 0x7f7ce0100ec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009890 tx=0x7f7cd002a380 comp rx=0 tx=0).stop 2024-09-16T00:15:00.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.049+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1278273903 shutdown_connections 2024-09-16T00:15:00.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.049+0000 7f7ce7a40640 1 --2- 172.21.15.33:0/1278273903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce0101400 0x7f7ce010d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:00.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.049+0000 7f7ce7a40640 1 --2- 172.21.15.33:0/1278273903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce0100a60 0x7f7ce0100ec0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:00.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.049+0000 7f7ce7a40640 1 --2- 172.21.15.33:0/1278273903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ce0106a60 0x7f7ce0106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:00.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.049+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1278273903 >> 172.21.15.33:0/1278273903 conn(0x7f7ce00fc780 msgr2=0x7f7ce00feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:00.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.049+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1278273903 shutdown_connections 2024-09-16T00:15:00.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.049+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1278273903 wait complete. 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.050+0000 7f7ce7a40640 1 Processor -- start 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.050+0000 7f7ce7a40640 1 -- start start 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.050+0000 7f7ce7a40640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce0100a60 0x7f7ce0198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.050+0000 7f7ce7a40640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce0101400 0x7f7ce01989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.050+0000 7f7ce7a40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ce0106a60 0x7f7ce019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.050+0000 7f7ce7a40640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce0110e60 con 0x7f7ce0100a60 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.050+0000 7f7ce7a40640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce0110ce0 con 0x7f7ce0106a60 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.050+0000 7f7ce7a40640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce0110fe0 con 0x7f7ce0101400 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce4fdb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce0101400 0x7f7ce01989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce57dc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce0100a60 0x7f7ce0198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce4fdb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce0101400 0x7f7ce01989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59460/0 (socket says 172.21.15.33:59460) 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce4fdb640 1 -- 172.21.15.33:0/1352289343 learned_addr learned my addr 172.21.15.33:0/1352289343 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce5fdd640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ce0106a60 0x7f7ce019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:00.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce4fdb640 1 -- 172.21.15.33:0/1352289343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ce0106a60 msgr2=0x7f7ce019cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce4fdb640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ce0106a60 0x7f7ce019cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce4fdb640 1 -- 172.21.15.33:0/1352289343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce0100a60 msgr2=0x7f7ce0198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce4fdb640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce0100a60 0x7f7ce0198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce4fdb640 1 -- 172.21.15.33:0/1352289343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce019d430 con 0x7f7ce0101400 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce57dc640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce0100a60 0x7f7ce0198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce5fdd640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ce0106a60 0x7f7ce019cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce4fdb640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce0101400 0x7f7ce01989c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009860 tx=0x7f7cd0033d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7cd67fc640 1 -- 172.21.15.33:0/1352289343 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd003f020 con 0x7f7ce0101400 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7cd67fc640 1 -- 172.21.15.33:0/1352289343 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd0004240 con 0x7f7ce0101400 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1352289343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce019d6c0 con 0x7f7ce0101400 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7cd67fc640 1 -- 172.21.15.33:0/1352289343 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0037040 con 0x7f7ce0101400 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.051+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1352289343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce01a5000 con 0x7f7ce0101400 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.052+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1352289343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce01021a0 con 0x7f7ce0101400 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.054+0000 7f7cd67fc640 1 -- 172.21.15.33:0/1352289343 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f7cd0042020 con 0x7f7ce0101400 2024-09-16T00:15:00.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.054+0000 7f7cd67fc640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ccc0a3240 0x7f7ccc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:00.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.055+0000 7f7ce57dc640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ccc0a3240 0x7f7ccc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:00.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.055+0000 7f7cd67fc640 1 -- 172.21.15.33:0/1352289343 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f7cd00ef9a0 con 0x7f7ce0101400 2024-09-16T00:15:00.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.055+0000 7f7ce57dc640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ccc0a3240 0x7f7ccc0a5700 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc005d00 tx=0x7f7cdc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:00.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.058+0000 7f7cd67fc640 1 -- 172.21.15.33:0/1352289343 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7cd0049020 con 0x7f7ce0101400 2024-09-16T00:15:00.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.246+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1352289343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7ce0109fe0 con 0x7f7ce0101400 2024-09-16T00:15:00.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.247+0000 7f7cd67fc640 1 -- 172.21.15.33:0/1352289343 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v75) ==== 84+0+65 (secure 0 0 0) 0x7f7cd00bcac0 con 0x7f7ce0101400 2024-09-16T00:15:00.249 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:15:00.249 INFO:teuthology.orchestra.run.smithi033.stdout:[ 2024-09-16T00:15:00.249 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:15:00.249 INFO:teuthology.orchestra.run.smithi033.stdout: "poolnum": 1, 2024-09-16T00:15:00.249 INFO:teuthology.orchestra.run.smithi033.stdout: "poolname": ".mgr" 2024-09-16T00:15:00.249 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:15:00.249 INFO:teuthology.orchestra.run.smithi033.stdout:] 2024-09-16T00:15:00.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.249+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1352289343 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ccc0a3240 msgr2=0x7f7ccc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:00.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.249+0000 7f7ce7a40640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ccc0a3240 0x7f7ccc0a5700 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc005d00 tx=0x7f7cdc00b410 comp rx=0 tx=0).stop 2024-09-16T00:15:00.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.249+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1352289343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce0101400 msgr2=0x7f7ce01989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:00.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.249+0000 7f7ce7a40640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce0101400 0x7f7ce01989c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009860 tx=0x7f7cd0033d10 comp rx=0 tx=0).stop 2024-09-16T00:15:00.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.249+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1352289343 shutdown_connections 2024-09-16T00:15:00.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.249+0000 7f7ce7a40640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ccc0a3240 0x7f7ccc0a5700 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:00.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.249+0000 7f7ce7a40640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ce0106a60 0x7f7ce019cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:00.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.249+0000 7f7ce7a40640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce0101400 0x7f7ce01989c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:00.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.249+0000 7f7ce7a40640 1 --2- 172.21.15.33:0/1352289343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce0100a60 0x7f7ce0198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:00.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.249+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1352289343 >> 172.21.15.33:0/1352289343 conn(0x7f7ce00fc780 msgr2=0x7f7ce006aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:00.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.249+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1352289343 shutdown_connections 2024-09-16T00:15:00.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:00.249+0000 7f7ce7a40640 1 -- 172.21.15.33:0/1352289343 wait complete. 2024-09-16T00:15:00.391 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:00.391 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:00.391 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:15:00.391 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:15:00.391 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:00.392 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:00 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:15:00.392 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:00 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1352289343' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-16T00:15:00.549 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:00.549 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:00.549 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:15:00.549 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:15:00.550 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:00.550 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:00 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:15:00.550 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:00 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1352289343' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-16T00:15:00.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:00.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:00.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:15:00.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:15:00.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:00.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:00 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:15:00.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:00 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1352289343' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-16T00:15:01.288 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T00:15:01.288 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T00:15:01.570 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:01 smithi033 ceph-mon[31243]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:01.570 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:01 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:01.570 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:01 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:01.570 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:01 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:01.570 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:01 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:01.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:01 smithi083 ceph-mon[35242]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:01.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:01 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:01.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:01 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:01.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:01 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:01.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:01 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:01 smithi060 ceph-mon[35185]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:01 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:01 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:01 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:01.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:01 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:01.732 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:02.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.218+0000 7f301d514640 1 -- 172.21.15.33:0/1621780869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3018100a80 msgr2=0x7f3018100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:02.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.218+0000 7f301d514640 1 --2- 172.21.15.33:0/1621780869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3018100a80 0x7f3018100ee0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3000009930 tx=0x7f300002a420 comp rx=0 tx=0).stop 2024-09-16T00:15:02.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.218+0000 7f301d514640 1 -- 172.21.15.33:0/1621780869 shutdown_connections 2024-09-16T00:15:02.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.218+0000 7f301d514640 1 --2- 172.21.15.33:0/1621780869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3018101420 0x7f301810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:02.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.218+0000 7f301d514640 1 --2- 172.21.15.33:0/1621780869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3018100a80 0x7f3018100ee0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:02.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.218+0000 7f301d514640 1 --2- 172.21.15.33:0/1621780869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018106a80 0x7f3018106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:02.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.218+0000 7f301d514640 1 -- 172.21.15.33:0/1621780869 >> 172.21.15.33:0/1621780869 conn(0x7f30180fc7a0 msgr2=0x7f30180febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:02.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.218+0000 7f301d514640 1 -- 172.21.15.33:0/1621780869 shutdown_connections 2024-09-16T00:15:02.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.218+0000 7f301d514640 1 -- 172.21.15.33:0/1621780869 wait complete. 2024-09-16T00:15:02.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.219+0000 7f301d514640 1 Processor -- start 2024-09-16T00:15:02.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.219+0000 7f301d514640 1 -- start start 2024-09-16T00:15:02.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.219+0000 7f301d514640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3018100a80 0x7f3018199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:02.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.219+0000 7f301d514640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018101420 0x7f3018199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.219+0000 7f301d514640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3018106a80 0x7f30181a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.219+0000 7f301d514640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3018110e80 con 0x7f3018106a80 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.219+0000 7f301d514640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3018110d00 con 0x7f3018100a80 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.219+0000 7f301d514640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3018111000 con 0x7f3018101420 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.219+0000 7f30177fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3018106a80 0x7f30181a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f30177fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3018106a80 0x7f30181a43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50006/0 (socket says 172.21.15.33:50006) 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f30177fe640 1 -- 172.21.15.33:0/3305568610 learned_addr learned my addr 172.21.15.33:0/3305568610 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f3016ffd640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3018100a80 0x7f3018199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f300e7fc640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018101420 0x7f3018199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f30177fe640 1 -- 172.21.15.33:0/3305568610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018101420 msgr2=0x7f3018199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f30177fe640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018101420 0x7f3018199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f30177fe640 1 -- 172.21.15.33:0/3305568610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3018100a80 msgr2=0x7f3018199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f30177fe640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3018100a80 0x7f3018199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:02.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f30177fe640 1 -- 172.21.15.33:0/3305568610 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30181a48f0 con 0x7f3018106a80 2024-09-16T00:15:02.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f30177fe640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3018106a80 0x7f30181a43b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3010009fd0 tx=0x7f301000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:02.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f3014ff9640 1 -- 172.21.15.33:0/3305568610 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3010017070 con 0x7f3018106a80 2024-09-16T00:15:02.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f301d514640 1 -- 172.21.15.33:0/3305568610 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30181a4ac0 con 0x7f3018106a80 2024-09-16T00:15:02.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f3014ff9640 1 -- 172.21.15.33:0/3305568610 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3010003e50 con 0x7f3018106a80 2024-09-16T00:15:02.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f3014ff9640 1 -- 172.21.15.33:0/3305568610 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30100106c0 con 0x7f3018106a80 2024-09-16T00:15:02.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.220+0000 7f301d514640 1 -- 172.21.15.33:0/3305568610 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30181a4f80 con 0x7f3018106a80 2024-09-16T00:15:02.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.221+0000 7f301d514640 1 -- 172.21.15.33:0/3305568610 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3018100ee0 con 0x7f3018106a80 2024-09-16T00:15:02.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.222+0000 7f3014ff9640 1 -- 172.21.15.33:0/3305568610 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f3010003970 con 0x7f3018106a80 2024-09-16T00:15:02.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.223+0000 7f3014ff9640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30040a3240 0x7f30040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:02.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.223+0000 7f3016ffd640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30040a3240 0x7f30040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:02.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.223+0000 7f3014ff9640 1 -- 172.21.15.33:0/3305568610 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f30100cf2b0 con 0x7f3018106a80 2024-09-16T00:15:02.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.224+0000 7f3016ffd640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30040a3240 0x7f30040a5700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3008005fd0 tx=0x7f300800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:02.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.227+0000 7f3014ff9640 1 -- 172.21.15.33:0/3305568610 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f301009c3d0 con 0x7f3018106a80 2024-09-16T00:15:02.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:02.430+0000 7f301d514640 1 -- 172.21.15.33:0/3305568610 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f301802d7c0 con 0x7f3018106a80 2024-09-16T00:15:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.324+0000 7f3014ff9640 1 -- 172.21.15.33:0/3305568610 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v76) ==== 138+0+0 (secure 0 0 0) 0x7f301009bcd0 con 0x7f3018106a80 2024-09-16T00:15:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_metadata' created 2024-09-16T00:15:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.327+0000 7f301d514640 1 -- 172.21.15.33:0/3305568610 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30040a3240 msgr2=0x7f30040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.327+0000 7f301d514640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30040a3240 0x7f30040a5700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3008005fd0 tx=0x7f300800b390 comp rx=0 tx=0).stop 2024-09-16T00:15:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.327+0000 7f301d514640 1 -- 172.21.15.33:0/3305568610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3018106a80 msgr2=0x7f30181a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.327+0000 7f301d514640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3018106a80 0x7f30181a43b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3010009fd0 tx=0x7f301000b260 comp rx=0 tx=0).stop 2024-09-16T00:15:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.327+0000 7f301d514640 1 -- 172.21.15.33:0/3305568610 shutdown_connections 2024-09-16T00:15:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.327+0000 7f301d514640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30040a3240 0x7f30040a5700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.327+0000 7f301d514640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3018106a80 0x7f30181a43b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.327+0000 7f301d514640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018101420 0x7f3018199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:03.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.327+0000 7f301d514640 1 --2- 172.21.15.33:0/3305568610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3018100a80 0x7f3018199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:03.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.328+0000 7f301d514640 1 -- 172.21.15.33:0/3305568610 >> 172.21.15.33:0/3305568610 conn(0x7f30180fc7a0 msgr2=0x7f30180fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:03.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.328+0000 7f301d514640 1 -- 172.21.15.33:0/3305568610 shutdown_connections 2024-09-16T00:15:03.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:03.328+0000 7f301d514640 1 -- 172.21.15.33:0/3305568610 wait complete. 2024-09-16T00:15:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:03 smithi033 ceph-mon[31243]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:03 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3305568610' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-16T00:15:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:15:03.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:15:03.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:03 smithi083 ceph-mon[35242]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:03 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3305568610' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-16T00:15:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:15:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:15:03.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:03.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:03 smithi060 ceph-mon[35185]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:03 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3305568610' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-16T00:15:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:15:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:15:03.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:15:04.096 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-09-16T00:15:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:04.658 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:04 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3305568610' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-16T00:15:04.659 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:04 smithi033 ceph-mon[31243]: osdmap e76: 12 total, 12 up, 12 in 2024-09-16T00:15:04.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:04 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3305568610' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-16T00:15:04.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:04 smithi083 ceph-mon[35242]: osdmap e76: 12 total, 12 up, 12 in 2024-09-16T00:15:04.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:04 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3305568610' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-16T00:15:04.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:04 smithi060 ceph-mon[35185]: osdmap e76: 12 total, 12 up, 12 in 2024-09-16T00:15:04.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.888+0000 7f5754d26640 1 -- 172.21.15.33:0/2428088873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f575010c180 msgr2=0x7f575010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.888+0000 7f5754d26640 1 --2- 172.21.15.33:0/2428088873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f575010c180 0x7f575010e610 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5748009900 tx=0x7f574802a480 comp rx=0 tx=0).stop 2024-09-16T00:15:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.888+0000 7f5754d26640 1 -- 172.21.15.33:0/2428088873 shutdown_connections 2024-09-16T00:15:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.888+0000 7f5754d26640 1 --2- 172.21.15.33:0/2428088873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f575010c180 0x7f575010e610 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.888+0000 7f5754d26640 1 --2- 172.21.15.33:0/2428088873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5750103700 0x7f575010bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.888+0000 7f5754d26640 1 --2- 172.21.15.33:0/2428088873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5750102de0 0x7f57501031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.888+0000 7f5754d26640 1 -- 172.21.15.33:0/2428088873 >> 172.21.15.33:0/2428088873 conn(0x7f57500fc7a0 msgr2=0x7f57500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.888+0000 7f5754d26640 1 -- 172.21.15.33:0/2428088873 shutdown_connections 2024-09-16T00:15:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.888+0000 7f5754d26640 1 -- 172.21.15.33:0/2428088873 wait complete. 2024-09-16T00:15:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.889+0000 7f5754d26640 1 Processor -- start 2024-09-16T00:15:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.889+0000 7f5754d26640 1 -- start start 2024-09-16T00:15:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.889+0000 7f5754d26640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5750102de0 0x7f575019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.889+0000 7f5754d26640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5750103700 0x7f575019aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:04.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.889+0000 7f5754d26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f575010c180 0x7f575019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:04.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.889+0000 7f5754d26640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5750111f30 con 0x7f5750102de0 2024-09-16T00:15:04.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.889+0000 7f5754d26640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5750111db0 con 0x7f575010c180 2024-09-16T00:15:04.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.889+0000 7f5754d26640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f57501120b0 con 0x7f5750103700 2024-09-16T00:15:04.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.889+0000 7f574e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5750102de0 0x7f575019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:04.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.889+0000 7f574e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5750102de0 0x7f575019a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50028/0 (socket says 172.21.15.33:50028) 2024-09-16T00:15:04.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.889+0000 7f574e59c640 1 -- 172.21.15.33:0/4106209629 learned_addr learned my addr 172.21.15.33:0/4106209629 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:04.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f574dd9b640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5750103700 0x7f575019aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:04.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f574ed9d640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f575010c180 0x7f575019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:04.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f574e59c640 1 -- 172.21.15.33:0/4106209629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5750103700 msgr2=0x7f575019aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:04.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f574e59c640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5750103700 0x7f575019aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:04.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f574e59c640 1 -- 172.21.15.33:0/4106209629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f575010c180 msgr2=0x7f575019edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:04.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f574e59c640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f575010c180 0x7f575019edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:04.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f574e59c640 1 -- 172.21.15.33:0/4106209629 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575019f490 con 0x7f5750102de0 2024-09-16T00:15:04.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f574dd9b640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5750103700 0x7f575019aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:15:04.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f574ed9d640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f575010c180 0x7f575019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:15:04.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f574e59c640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5750102de0 0x7f575019a4e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f574000cce0 tx=0x7f574000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:04.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f57477fe640 1 -- 172.21.15.33:0/4106209629 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574000f650 con 0x7f5750102de0 2024-09-16T00:15:04.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f57477fe640 1 -- 172.21.15.33:0/4106209629 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f574000fcd0 con 0x7f5750102de0 2024-09-16T00:15:04.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f5754d26640 1 -- 172.21.15.33:0/4106209629 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f575019f780 con 0x7f5750102de0 2024-09-16T00:15:04.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f57477fe640 1 -- 172.21.15.33:0/4106209629 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740018c70 con 0x7f5750102de0 2024-09-16T00:15:04.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.890+0000 7f5754d26640 1 -- 172.21.15.33:0/4106209629 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57501a7060 con 0x7f5750102de0 2024-09-16T00:15:04.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.891+0000 7f5754d26640 1 -- 172.21.15.33:0/4106209629 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57501031c0 con 0x7f5750102de0 2024-09-16T00:15:04.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.893+0000 7f57477fe640 1 -- 172.21.15.33:0/4106209629 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f574000f7f0 con 0x7f5750102de0 2024-09-16T00:15:04.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.893+0000 7f57477fe640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f573c0a32c0 0x7f573c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:04.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.893+0000 7f574dd9b640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f573c0a32c0 0x7f573c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:04.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.894+0000 7f57477fe640 1 -- 172.21.15.33:0/4106209629 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7497+0+0 (secure 0 0 0) 0x7f57400cfe30 con 0x7f5750102de0 2024-09-16T00:15:04.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.894+0000 7f574dd9b640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f573c0a32c0 0x7f573c0a5780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f575010e9a0 tx=0x7f5738005d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:04.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:04.897+0000 7f57477fe640 1 -- 172.21.15.33:0/4106209629 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5740014030 con 0x7f5750102de0 2024-09-16T00:15:05.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.074+0000 7f5754d26640 1 -- 172.21.15.33:0/4106209629 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f575019b840 con 0x7f5750102de0 2024-09-16T00:15:05.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.516+0000 7f57477fe640 1 -- 172.21.15.33:0/4106209629 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v78) ==== 171+0+0 (secure 0 0 0) 0x7f574009cef0 con 0x7f5750102de0 2024-09-16T00:15:05.518 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_data' created 2024-09-16T00:15:05.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.519+0000 7f5754d26640 1 -- 172.21.15.33:0/4106209629 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f573c0a32c0 msgr2=0x7f573c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:05.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.519+0000 7f5754d26640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f573c0a32c0 0x7f573c0a5780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f575010e9a0 tx=0x7f5738005d20 comp rx=0 tx=0).stop 2024-09-16T00:15:05.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.519+0000 7f5754d26640 1 -- 172.21.15.33:0/4106209629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5750102de0 msgr2=0x7f575019a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:05.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.519+0000 7f5754d26640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5750102de0 0x7f575019a4e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f574000cce0 tx=0x7f574000a620 comp rx=0 tx=0).stop 2024-09-16T00:15:05.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.519+0000 7f5754d26640 1 -- 172.21.15.33:0/4106209629 shutdown_connections 2024-09-16T00:15:05.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.519+0000 7f5754d26640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f573c0a32c0 0x7f573c0a5780 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:05.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.519+0000 7f5754d26640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f575010c180 0x7f575019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:05.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.519+0000 7f5754d26640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5750103700 0x7f575019aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:05.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.519+0000 7f5754d26640 1 --2- 172.21.15.33:0/4106209629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5750102de0 0x7f575019a4e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:05.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.519+0000 7f5754d26640 1 -- 172.21.15.33:0/4106209629 >> 172.21.15.33:0/4106209629 conn(0x7f57500fc7a0 msgr2=0x7f575010d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:05.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.519+0000 7f5754d26640 1 -- 172.21.15.33:0/4106209629 shutdown_connections 2024-09-16T00:15:05.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:05.519+0000 7f5754d26640 1 -- 172.21.15.33:0/4106209629 wait complete. 2024-09-16T00:15:05.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:05 smithi033 ceph-mon[31243]: pgmap v274: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:05.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:05 smithi033 ceph-mon[31243]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T00:15:05.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:05 smithi033 ceph-mon[31243]: osdmap e77: 12 total, 12 up, 12 in 2024-09-16T00:15:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:05 smithi083 ceph-mon[35242]: pgmap v274: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:05.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:05 smithi083 ceph-mon[35242]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T00:15:05.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:05 smithi083 ceph-mon[35242]: osdmap e77: 12 total, 12 up, 12 in 2024-09-16T00:15:05.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:05 smithi060 ceph-mon[35185]: pgmap v274: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:05.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:05 smithi060 ceph-mon[35185]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T00:15:05.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:05 smithi060 ceph-mon[35185]: osdmap e77: 12 total, 12 up, 12 in 2024-09-16T00:15:06.271 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T00:15:06.538 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:06.797 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:06 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4106209629' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-16T00:15:06.797 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:06 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4106209629' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-16T00:15:06.797 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:06 smithi033 ceph-mon[31243]: osdmap e78: 12 total, 12 up, 12 in 2024-09-16T00:15:06.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:06 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4106209629' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-16T00:15:06.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:06 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4106209629' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-16T00:15:06.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:06 smithi083 ceph-mon[35242]: osdmap e78: 12 total, 12 up, 12 in 2024-09-16T00:15:06.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:06 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4106209629' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-16T00:15:06.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:06 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4106209629' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-16T00:15:06.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:06 smithi060 ceph-mon[35185]: osdmap e78: 12 total, 12 up, 12 in 2024-09-16T00:15:07.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.024+0000 7f5820743640 1 -- 172.21.15.33:0/4116106616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58181011a0 msgr2=0x7f5818101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:07.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.024+0000 7f5820743640 1 --2- 172.21.15.33:0/4116106616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58181011a0 0x7f5818101600 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5808009900 tx=0x7f580802a410 comp rx=0 tx=0).stop 2024-09-16T00:15:07.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.024+0000 7f5820743640 1 -- 172.21.15.33:0/4116106616 shutdown_connections 2024-09-16T00:15:07.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.024+0000 7f5820743640 1 --2- 172.21.15.33:0/4116106616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5818101b40 0x7f581810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:07.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.024+0000 7f5820743640 1 --2- 172.21.15.33:0/4116106616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58181011a0 0x7f5818101600 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:07.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.024+0000 7f5820743640 1 --2- 172.21.15.33:0/4116106616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5818106c30 0x7f5818107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:07.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.024+0000 7f5820743640 1 -- 172.21.15.33:0/4116106616 >> 172.21.15.33:0/4116106616 conn(0x7f5818077210 msgr2=0x7f58180fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:07.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f5820743640 1 -- 172.21.15.33:0/4116106616 shutdown_connections 2024-09-16T00:15:07.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f5820743640 1 -- 172.21.15.33:0/4116106616 wait complete. 2024-09-16T00:15:07.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f5820743640 1 Processor -- start 2024-09-16T00:15:07.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f5820743640 1 -- start start 2024-09-16T00:15:07.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f5820743640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58181011a0 0x7f58181993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:07.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f5820743640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5818101b40 0x7f5818199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:07.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f5820743640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5818106c30 0x7f58181a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:07.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f5820743640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5818110fc0 con 0x7f5818101b40 2024-09-16T00:15:07.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f5820743640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5818110e40 con 0x7f58181011a0 2024-09-16T00:15:07.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f5820743640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5818111140 con 0x7f5818106c30 2024-09-16T00:15:07.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f581dcde640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5818101b40 0x7f5818199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:07.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f581dcde640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5818101b40 0x7f5818199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50042/0 (socket says 172.21.15.33:50042) 2024-09-16T00:15:07.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.025+0000 7f581dcde640 1 -- 172.21.15.33:0/1609252568 learned_addr learned my addr 172.21.15.33:0/1609252568 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:07.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f581ece0640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5818106c30 0x7f58181a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:07.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f581e4df640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58181011a0 0x7f58181993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f581dcde640 1 -- 172.21.15.33:0/1609252568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5818106c30 msgr2=0x7f58181a4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f581dcde640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5818106c30 0x7f58181a4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f581dcde640 1 -- 172.21.15.33:0/1609252568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58181011a0 msgr2=0x7f58181993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f581dcde640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58181011a0 0x7f58181993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f581dcde640 1 -- 172.21.15.33:0/1609252568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58181a49b0 con 0x7f5818101b40 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f581e4df640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58181011a0 0x7f58181993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f581dcde640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5818101b40 0x7f5818199930 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5808004770 tx=0x7f5808003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f580f7fe640 1 -- 172.21.15.33:0/1609252568 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580803f070 con 0x7f5818101b40 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f580f7fe640 1 -- 172.21.15.33:0/1609252568 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5808031ca0 con 0x7f5818101b40 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f580f7fe640 1 -- 172.21.15.33:0/1609252568 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5808035030 con 0x7f5818101b40 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f5820743640 1 -- 172.21.15.33:0/1609252568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58181a4b80 con 0x7f5818101b40 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.026+0000 7f5820743640 1 -- 172.21.15.33:0/1609252568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58181a5030 con 0x7f5818101b40 2024-09-16T00:15:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.027+0000 7f5820743640 1 -- 172.21.15.33:0/1609252568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5818101600 con 0x7f5818101b40 2024-09-16T00:15:07.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.029+0000 7f580f7fe640 1 -- 172.21.15.33:0/1609252568 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f5808042070 con 0x7f5818101b40 2024-09-16T00:15:07.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.030+0000 7f580f7fe640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58040a32c0 0x7f58040a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:07.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.030+0000 7f581e4df640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58040a32c0 0x7f58040a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:07.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.030+0000 7f580f7fe640 1 -- 172.21.15.33:0/1609252568 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7879+0+0 (secure 0 0 0) 0x7f58080f0400 con 0x7f5818101b40 2024-09-16T00:15:07.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.030+0000 7f581e4df640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58040a32c0 0x7f58040a5780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5814005e00 tx=0x7f5814005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:07.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.033+0000 7f580f7fe640 1 -- 172.21.15.33:0/1609252568 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f58080bd2d0 con 0x7f5818101b40 2024-09-16T00:15:07.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.244+0000 7f5820743640 1 -- 172.21.15.33:0/1609252568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f5818108b30 con 0x7f5818101b40 2024-09-16T00:15:07.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.526+0000 7f580f7fe640 1 -- 172.21.15.33:0/1609252568 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f58080bcbd0 con 0x7f5818101b40 2024-09-16T00:15:07.528 INFO:teuthology.orchestra.run.smithi033.stderr:new fs with metadata pool 2 and data pool 3 2024-09-16T00:15:07.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.529+0000 7f5820743640 1 -- 172.21.15.33:0/1609252568 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58040a32c0 msgr2=0x7f58040a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:07.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.529+0000 7f5820743640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58040a32c0 0x7f58040a5780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5814005e00 tx=0x7f5814005c70 comp rx=0 tx=0).stop 2024-09-16T00:15:07.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.529+0000 7f5820743640 1 -- 172.21.15.33:0/1609252568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5818101b40 msgr2=0x7f5818199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:07.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.529+0000 7f5820743640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5818101b40 0x7f5818199930 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5808004770 tx=0x7f5808003c10 comp rx=0 tx=0).stop 2024-09-16T00:15:07.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.529+0000 7f5820743640 1 -- 172.21.15.33:0/1609252568 shutdown_connections 2024-09-16T00:15:07.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.529+0000 7f5820743640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58040a32c0 0x7f58040a5780 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:07.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.529+0000 7f5820743640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5818106c30 0x7f58181a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:07.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.529+0000 7f5820743640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5818101b40 0x7f5818199930 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:07.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.529+0000 7f5820743640 1 --2- 172.21.15.33:0/1609252568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58181011a0 0x7f58181993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:07.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.529+0000 7f5820743640 1 -- 172.21.15.33:0/1609252568 >> 172.21.15.33:0/1609252568 conn(0x7f5818077210 msgr2=0x7f5818103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:07.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.529+0000 7f5820743640 1 -- 172.21.15.33:0/1609252568 shutdown_connections 2024-09-16T00:15:07.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:07.529+0000 7f5820743640 1 -- 172.21.15.33:0/1609252568 wait complete. 2024-09-16T00:15:07.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:07 smithi033 ceph-mon[31243]: pgmap v277: 97 pgs: 3 creating+peering, 82 unknown, 12 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:07.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:07 smithi033 ceph-mon[31243]: osdmap e79: 12 total, 12 up, 12 in 2024-09-16T00:15:07.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:07 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1609252568' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-16T00:15:07.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:07 smithi033 ceph-mon[31243]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-16T00:15:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:07 smithi083 ceph-mon[35242]: pgmap v277: 97 pgs: 3 creating+peering, 82 unknown, 12 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:07.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:07 smithi083 ceph-mon[35242]: osdmap e79: 12 total, 12 up, 12 in 2024-09-16T00:15:07.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:07 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1609252568' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-16T00:15:07.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:07 smithi083 ceph-mon[35242]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-16T00:15:07.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:07 smithi060 ceph-mon[35185]: pgmap v277: 97 pgs: 3 creating+peering, 82 unknown, 12 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:07.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:07 smithi060 ceph-mon[35185]: osdmap e79: 12 total, 12 up, 12 in 2024-09-16T00:15:07.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:07 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1609252568' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-16T00:15:07.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:07 smithi060 ceph-mon[35185]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-16T00:15:08.243 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-09-16T00:15:08.243 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-09-16T00:15:08.499 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:08.805 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:08 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1609252568' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-16T00:15:08.805 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:08 smithi033 ceph-mon[31243]: osdmap e80: 12 total, 12 up, 12 in 2024-09-16T00:15:08.805 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:08 smithi033 ceph-mon[31243]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-16T00:15:08.805 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:08 smithi033 ceph-mon[31243]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-16T00:15:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:08 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1609252568' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-16T00:15:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:08 smithi083 ceph-mon[35242]: osdmap e80: 12 total, 12 up, 12 in 2024-09-16T00:15:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:08 smithi083 ceph-mon[35242]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-16T00:15:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:08 smithi083 ceph-mon[35242]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-16T00:15:08.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:08 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1609252568' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-16T00:15:08.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:08 smithi060 ceph-mon[35185]: osdmap e80: 12 total, 12 up, 12 in 2024-09-16T00:15:08.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:08 smithi060 ceph-mon[35185]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-16T00:15:08.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:08 smithi060 ceph-mon[35185]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-16T00:15:09.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.019+0000 7f22e31dd640 1 -- 172.21.15.33:0/2447777377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22dc102570 msgr2=0x7f22dc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:09.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.019+0000 7f22e31dd640 1 --2- 172.21.15.33:0/2447777377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22dc102570 0x7f22dc1029b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009900 tx=0x7f22cc02a480 comp rx=0 tx=0).stop 2024-09-16T00:15:09.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.019+0000 7f22e31dd640 1 -- 172.21.15.33:0/2447777377 shutdown_connections 2024-09-16T00:15:09.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.019+0000 7f22e31dd640 1 --2- 172.21.15.33:0/2447777377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22dc102ef0 0x7f22dc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:09.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.019+0000 7f22e31dd640 1 --2- 172.21.15.33:0/2447777377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22dc102570 0x7f22dc1029b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:09.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.019+0000 7f22e31dd640 1 --2- 172.21.15.33:0/2447777377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22dc100e20 0x7f22dc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:09.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.019+0000 7f22e31dd640 1 -- 172.21.15.33:0/2447777377 >> 172.21.15.33:0/2447777377 conn(0x7f22dc0fc7d0 msgr2=0x7f22dc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:09.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.019+0000 7f22e31dd640 1 -- 172.21.15.33:0/2447777377 shutdown_connections 2024-09-16T00:15:09.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.019+0000 7f22e31dd640 1 -- 172.21.15.33:0/2447777377 wait complete. 2024-09-16T00:15:09.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22e31dd640 1 Processor -- start 2024-09-16T00:15:09.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22e31dd640 1 -- start start 2024-09-16T00:15:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22e31dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22dc100e20 0x7f22dc198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22e31dd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22dc102570 0x7f22dc198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22e31dd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22dc102ef0 0x7f22dc19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22e31dd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc1120a0 con 0x7f22dc102570 2024-09-16T00:15:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22e31dd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc111f20 con 0x7f22dc100e20 2024-09-16T00:15:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22e31dd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc112220 con 0x7f22dc102ef0 2024-09-16T00:15:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22dc102570 0x7f22dc198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22dc102570 0x7f22dc198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50056/0 (socket says 172.21.15.33:50056) 2024-09-16T00:15:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22d3fff640 1 -- 172.21.15.33:0/736548988 learned_addr learned my addr 172.21.15.33:0/736548988 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22e0f79640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22dc100e20 0x7f22dc198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22e177a640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22dc102ef0 0x7f22dc19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22d3fff640 1 -- 172.21.15.33:0/736548988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22dc102ef0 msgr2=0x7f22dc19ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:09.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22d3fff640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22dc102ef0 0x7f22dc19ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:09.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.020+0000 7f22d3fff640 1 -- 172.21.15.33:0/736548988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22dc100e20 msgr2=0x7f22dc198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:09.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.021+0000 7f22d3fff640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22dc100e20 0x7f22dc198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:09.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.021+0000 7f22d3fff640 1 -- 172.21.15.33:0/736548988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc19d4f0 con 0x7f22dc102570 2024-09-16T00:15:09.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.021+0000 7f22d3fff640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22dc102570 0x7f22dc198a80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f22cc006010 tx=0x7f22cc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:09.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.021+0000 7f22d1ffb640 1 -- 172.21.15.33:0/736548988 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc03f070 con 0x7f22dc102570 2024-09-16T00:15:09.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.021+0000 7f22d1ffb640 1 -- 172.21.15.33:0/736548988 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22cc031c30 con 0x7f22dc102570 2024-09-16T00:15:09.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.021+0000 7f22e31dd640 1 -- 172.21.15.33:0/736548988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22dc19d780 con 0x7f22dc102570 2024-09-16T00:15:09.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.021+0000 7f22d1ffb640 1 -- 172.21.15.33:0/736548988 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc035030 con 0x7f22dc102570 2024-09-16T00:15:09.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.021+0000 7f22e31dd640 1 -- 172.21.15.33:0/736548988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22dc1a50c0 con 0x7f22dc102570 2024-09-16T00:15:09.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.022+0000 7f22e31dd640 1 -- 172.21.15.33:0/736548988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22dc101200 con 0x7f22dc102570 2024-09-16T00:15:09.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.023+0000 7f22d1ffb640 1 -- 172.21.15.33:0/736548988 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f22cc042070 con 0x7f22dc102570 2024-09-16T00:15:09.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.024+0000 7f22d1ffb640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f22c40a32c0 0x7f22c40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:09.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.024+0000 7f22e0f79640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f22c40a32c0 0x7f22c40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:09.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.024+0000 7f22d1ffb640 1 -- 172.21.15.33:0/736548988 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7979+0+0 (secure 0 0 0) 0x7f22cc0efef0 con 0x7f22dc102570 2024-09-16T00:15:09.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.024+0000 7f22e0f79640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f22c40a32c0 0x7f22c40a5780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f22d8005fd0 tx=0x7f22d8009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:09.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.028+0000 7f22d1ffb640 1 -- 172.21.15.33:0/736548988 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f22cc031de0 con 0x7f22dc102570 2024-09-16T00:15:09.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.211+0000 7f22e31dd640 1 -- 172.21.15.33:0/736548988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f22dc109090 con 0x7f22dc102570 2024-09-16T00:15:09.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.546+0000 7f22d1ffb640 1 -- 172.21.15.33:0/736548988 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v81) ==== 157+0+0 (secure 0 0 0) 0x7f22cc0bc710 con 0x7f22dc102570 2024-09-16T00:15:09.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.548+0000 7f22e31dd640 1 -- 172.21.15.33:0/736548988 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f22c40a32c0 msgr2=0x7f22c40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:09.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.549+0000 7f22e31dd640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f22c40a32c0 0x7f22c40a5780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f22d8005fd0 tx=0x7f22d8009680 comp rx=0 tx=0).stop 2024-09-16T00:15:09.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.549+0000 7f22e31dd640 1 -- 172.21.15.33:0/736548988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22dc102570 msgr2=0x7f22dc198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:09.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.549+0000 7f22e31dd640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22dc102570 0x7f22dc198a80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f22cc006010 tx=0x7f22cc003c10 comp rx=0 tx=0).stop 2024-09-16T00:15:09.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.549+0000 7f22e31dd640 1 -- 172.21.15.33:0/736548988 shutdown_connections 2024-09-16T00:15:09.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.549+0000 7f22e31dd640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f22c40a32c0 0x7f22c40a5780 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:09.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.549+0000 7f22e31dd640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22dc102ef0 0x7f22dc19ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:09.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.549+0000 7f22e31dd640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22dc102570 0x7f22dc198a80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:09.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.549+0000 7f22e31dd640 1 --2- 172.21.15.33:0/736548988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22dc100e20 0x7f22dc198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:09.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.549+0000 7f22e31dd640 1 -- 172.21.15.33:0/736548988 >> 172.21.15.33:0/736548988 conn(0x7f22dc0fc7d0 msgr2=0x7f22dc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:09.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.549+0000 7f22e31dd640 1 -- 172.21.15.33:0/736548988 shutdown_connections 2024-09-16T00:15:09.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:09.549+0000 7f22e31dd640 1 -- 172.21.15.33:0/736548988 wait complete. 2024-09-16T00:15:09.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:09 smithi033 ceph-mon[31243]: pgmap v280: 97 pgs: 3 creating+peering, 72 unknown, 22 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:09.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:09 smithi033 ceph-mon[31243]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-16T00:15:09.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:09 smithi033 ceph-mon[31243]: Cluster is now healthy 2024-09-16T00:15:09.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:09 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] up:active 2024-09-16T00:15:09.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:09 smithi033 ceph-mon[31243]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:09.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:09 smithi033 ceph-mon[31243]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:09.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:09 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/736548988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T00:15:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:09 smithi083 ceph-mon[35242]: pgmap v280: 97 pgs: 3 creating+peering, 72 unknown, 22 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:09 smithi083 ceph-mon[35242]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-16T00:15:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:09 smithi083 ceph-mon[35242]: Cluster is now healthy 2024-09-16T00:15:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:09 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] up:active 2024-09-16T00:15:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:09 smithi083 ceph-mon[35242]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:09 smithi083 ceph-mon[35242]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:09 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/736548988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T00:15:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:09 smithi060 ceph-mon[35185]: pgmap v280: 97 pgs: 3 creating+peering, 72 unknown, 22 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:09 smithi060 ceph-mon[35185]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-16T00:15:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:09 smithi060 ceph-mon[35185]: Cluster is now healthy 2024-09-16T00:15:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:09 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] up:active 2024-09-16T00:15:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:09 smithi060 ceph-mon[35185]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:09 smithi060 ceph-mon[35185]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:09 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/736548988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T00:15:10.265 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T00:15:10.525 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:10.802 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:10 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/736548988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-16T00:15:10.802 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:10 smithi033 ceph-mon[31243]: osdmap e81: 12 total, 12 up, 12 in 2024-09-16T00:15:10.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:10 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/736548988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-16T00:15:10.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:10 smithi083 ceph-mon[35242]: osdmap e81: 12 total, 12 up, 12 in 2024-09-16T00:15:10.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:10 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/736548988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-16T00:15:10.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:10 smithi060 ceph-mon[35185]: osdmap e81: 12 total, 12 up, 12 in 2024-09-16T00:15:11.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.014+0000 7fd6d2685640 1 -- 172.21.15.33:0/1154499817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc106f10 msgr2=0x7fd6cc10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:11.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.014+0000 7fd6d2685640 1 --2- 172.21.15.33:0/1154499817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc106f10 0x7fd6cc10ea90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009ca0 tx=0x7fd6c402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:15:11.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.014+0000 7fd6d2685640 1 -- 172.21.15.33:0/1154499817 shutdown_connections 2024-09-16T00:15:11.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.014+0000 7fd6d2685640 1 --2- 172.21.15.33:0/1154499817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc106f10 0x7fd6cc10ea90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:11.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.014+0000 7fd6d2685640 1 --2- 172.21.15.33:0/1154499817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6cc102790 0x7fd6cc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:11.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.014+0000 7fd6d2685640 1 --2- 172.21.15.33:0/1154499817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6cc101de0 0x7fd6cc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:11.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.014+0000 7fd6d2685640 1 -- 172.21.15.33:0/1154499817 >> 172.21.15.33:0/1154499817 conn(0x7fd6cc0fd8c0 msgr2=0x7fd6cc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:11.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.014+0000 7fd6d2685640 1 -- 172.21.15.33:0/1154499817 shutdown_connections 2024-09-16T00:15:11.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.014+0000 7fd6d2685640 1 -- 172.21.15.33:0/1154499817 wait complete. 2024-09-16T00:15:11.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.015+0000 7fd6d2685640 1 Processor -- start 2024-09-16T00:15:11.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.015+0000 7fd6d2685640 1 -- start start 2024-09-16T00:15:11.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.015+0000 7fd6d2685640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6cc101de0 0x7fd6cc19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:11.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.015+0000 7fd6d2685640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc102790 0x7fd6cc19ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.015+0000 7fd6d2685640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6cc106f10 0x7fd6cc19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.015+0000 7fd6d2685640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc111f70 con 0x7fd6cc102790 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6d2685640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc111df0 con 0x7fd6cc101de0 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6d2685640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc1120f0 con 0x7fd6cc106f10 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6cb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc102790 0x7fd6cc19ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6cb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc102790 0x7fd6cc19ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58856/0 (socket says 172.21.15.33:58856) 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6cb7fe640 1 -- 172.21.15.33:0/3050211235 learned_addr learned my addr 172.21.15.33:0/3050211235 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6cbfff640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6cc101de0 0x7fd6cc19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6d0c22640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6cc106f10 0x7fd6cc19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6cb7fe640 1 -- 172.21.15.33:0/3050211235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6cc106f10 msgr2=0x7fd6cc19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6cb7fe640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6cc106f10 0x7fd6cc19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6cb7fe640 1 -- 172.21.15.33:0/3050211235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6cc101de0 msgr2=0x7fd6cc19a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:11.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6cb7fe640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6cc101de0 0x7fd6cc19a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6cb7fe640 1 -- 172.21.15.33:0/3050211235 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6cc19f5a0 con 0x7fd6cc102790 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6d0c22640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6cc106f10 0x7fd6cc19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6cb7fe640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc102790 0x7fd6cc19ab30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000d950 tx=0x7fd6c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6c97fa640 1 -- 172.21.15.33:0/3050211235 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0017070 con 0x7fd6cc102790 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6d2685640 1 -- 172.21.15.33:0/3050211235 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6cc19f890 con 0x7fd6cc102790 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6c97fa640 1 -- 172.21.15.33:0/3050211235 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c0009c30 con 0x7fd6cc102790 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6c97fa640 1 -- 172.21.15.33:0/3050211235 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c000f6b0 con 0x7fd6cc102790 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.016+0000 7fd6d2685640 1 -- 172.21.15.33:0/3050211235 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6cc1a7170 con 0x7fd6cc102790 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.018+0000 7fd6d2685640 1 -- 172.21.15.33:0/3050211235 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6cc1021c0 con 0x7fd6cc102790 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.019+0000 7fd6c97fa640 1 -- 172.21.15.33:0/3050211235 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fd6c00109d0 con 0x7fd6cc102790 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.019+0000 7fd6c97fa640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6b40a3240 0x7fd6b40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.019+0000 7fd6c97fa640 1 -- 172.21.15.33:0/3050211235 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8277+0+0 (secure 0 0 0) 0x7fd6c00cf510 con 0x7fd6cc102790 2024-09-16T00:15:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.019+0000 7fd6cbfff640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6b40a3240 0x7fd6b40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.020+0000 7fd6cbfff640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6b40a3240 0x7fd6b40a5700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc005d00 tx=0x7fd6bc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.024+0000 7fd6c97fa640 1 -- 172.21.15.33:0/3050211235 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd6c0009de0 con 0x7fd6cc102790 2024-09-16T00:15:11.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:11.201+0000 7fd6d2685640 1 -- 172.21.15.33:0/3050211235 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd6cc1a7400 con 0x7fd6cc102790 2024-09-16T00:15:12.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:11 smithi060 ceph-mon[35185]: pgmap v282: 97 pgs: 20 unknown, 77 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 3 op/s 2024-09-16T00:15:12.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:11 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3050211235' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-16T00:15:12.231 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:15:11 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b[65594]: 2024-09-16T00:15:11.745+0000 7fe392696640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T00:15:12.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:11 smithi033 ceph-mon[31243]: pgmap v282: 97 pgs: 20 unknown, 77 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 3 op/s 2024-09-16T00:15:12.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:11 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3050211235' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-16T00:15:12.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:11 smithi083 ceph-mon[35242]: pgmap v282: 97 pgs: 20 unknown, 77 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 3 op/s 2024-09-16T00:15:12.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:11 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3050211235' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-16T00:15:12.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.558+0000 7fd6c97fa640 1 -- 172.21.15.33:0/3050211235 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v83) ==== 228+0+0 (secure 0 0 0) 0x7fd6c009c200 con 0x7fd6cc102790 2024-09-16T00:15:12.559 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_data_ec' created 2024-09-16T00:15:12.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.561+0000 7fd6d2685640 1 -- 172.21.15.33:0/3050211235 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6b40a3240 msgr2=0x7fd6b40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:12.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.561+0000 7fd6d2685640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6b40a3240 0x7fd6b40a5700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc005d00 tx=0x7fd6bc00b410 comp rx=0 tx=0).stop 2024-09-16T00:15:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.561+0000 7fd6d2685640 1 -- 172.21.15.33:0/3050211235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc102790 msgr2=0x7fd6cc19ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.561+0000 7fd6d2685640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc102790 0x7fd6cc19ab30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000d950 tx=0x7fd6c000de10 comp rx=0 tx=0).stop 2024-09-16T00:15:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.561+0000 7fd6d2685640 1 -- 172.21.15.33:0/3050211235 shutdown_connections 2024-09-16T00:15:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.561+0000 7fd6d2685640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6b40a3240 0x7fd6b40a5700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.561+0000 7fd6d2685640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6cc106f10 0x7fd6cc19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.561+0000 7fd6d2685640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc102790 0x7fd6cc19ab30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.561+0000 7fd6d2685640 1 --2- 172.21.15.33:0/3050211235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6cc101de0 0x7fd6cc19a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.561+0000 7fd6d2685640 1 -- 172.21.15.33:0/3050211235 >> 172.21.15.33:0/3050211235 conn(0x7fd6cc0fd8c0 msgr2=0x7fd6cc0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.561+0000 7fd6d2685640 1 -- 172.21.15.33:0/3050211235 shutdown_connections 2024-09-16T00:15:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:12.561+0000 7fd6d2685640 1 -- 172.21.15.33:0/3050211235 wait complete. 2024-09-16T00:15:13.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:12 smithi033 ceph-mon[31243]: pgmap v284: 97 pgs: 97 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-09-16T00:15:13.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:13 smithi033 ceph-mon[31243]: osdmap e82: 12 total, 12 up, 12 in 2024-09-16T00:15:13.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:13 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3050211235' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-16T00:15:13.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:13 smithi033 ceph-mon[31243]: osdmap e83: 12 total, 12 up, 12 in 2024-09-16T00:15:13.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:12 smithi083 ceph-mon[35242]: pgmap v284: 97 pgs: 97 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-09-16T00:15:13.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:12 smithi083 ceph-mon[35242]: osdmap e82: 12 total, 12 up, 12 in 2024-09-16T00:15:13.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:12 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3050211235' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-16T00:15:13.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:12 smithi083 ceph-mon[35242]: osdmap e83: 12 total, 12 up, 12 in 2024-09-16T00:15:13.433 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-09-16T00:15:13.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:12 smithi060 ceph-mon[35185]: pgmap v284: 97 pgs: 97 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-09-16T00:15:13.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:12 smithi060 ceph-mon[35185]: osdmap e82: 12 total, 12 up, 12 in 2024-09-16T00:15:13.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:12 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3050211235' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-16T00:15:13.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:12 smithi060 ceph-mon[35185]: osdmap e83: 12 total, 12 up, 12 in 2024-09-16T00:15:13.711 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:14.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.222+0000 7fbdec51a640 1 -- 172.21.15.33:0/3343143847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde41082a0 msgr2=0x7fbde410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:14.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.222+0000 7fbdec51a640 1 --2- 172.21.15.33:0/3343143847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde41082a0 0x7fbde410ec40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbddc009cd0 tx=0x7fbddc02a8e0 comp rx=0 tx=0).stop 2024-09-16T00:15:14.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.222+0000 7fbdec51a640 1 -- 172.21.15.33:0/3343143847 shutdown_connections 2024-09-16T00:15:14.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.222+0000 7fbdec51a640 1 --2- 172.21.15.33:0/3343143847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde41082a0 0x7fbde410ec40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:14.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.222+0000 7fbdec51a640 1 --2- 172.21.15.33:0/3343143847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbde41036e0 0x7fbde4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:14.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.222+0000 7fbdec51a640 1 --2- 172.21.15.33:0/3343143847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbde4102d30 0x7fbde4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:14.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.222+0000 7fbdec51a640 1 -- 172.21.15.33:0/3343143847 >> 172.21.15.33:0/3343143847 conn(0x7fbde40fc7d0 msgr2=0x7fbde40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:14.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.222+0000 7fbdec51a640 1 -- 172.21.15.33:0/3343143847 shutdown_connections 2024-09-16T00:15:14.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.222+0000 7fbdec51a640 1 -- 172.21.15.33:0/3343143847 wait complete. 2024-09-16T00:15:14.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.223+0000 7fbdec51a640 1 Processor -- start 2024-09-16T00:15:14.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.223+0000 7fbdec51a640 1 -- start start 2024-09-16T00:15:14.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.223+0000 7fbdec51a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde4102d30 0x7fbde4198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:14.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.223+0000 7fbdec51a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbde41036e0 0x7fbde4198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:14.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.223+0000 7fbdec51a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbde41082a0 0x7fbde419ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:14.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.223+0000 7fbdec51a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbde41120a0 con 0x7fbde4102d30 2024-09-16T00:15:14.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.223+0000 7fbdec51a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbde4111f20 con 0x7fbde41082a0 2024-09-16T00:15:14.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.223+0000 7fbdec51a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbde4112220 con 0x7fbde41036e0 2024-09-16T00:15:14.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.223+0000 7fbdea2b6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde4102d30 0x7fbde4198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:14.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.223+0000 7fbdea2b6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde4102d30 0x7fbde4198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58888/0 (socket says 172.21.15.33:58888) 2024-09-16T00:15:14.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.223+0000 7fbdea2b6640 1 -- 172.21.15.33:0/2984920456 learned_addr learned my addr 172.21.15.33:0/2984920456 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:14.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.223+0000 7fbde9ab5640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbde41036e0 0x7fbde4198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:14.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbdeaab7640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbde41082a0 0x7fbde419ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:14.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbdea2b6640 1 -- 172.21.15.33:0/2984920456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbde41036e0 msgr2=0x7fbde4198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:14.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbdea2b6640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbde41036e0 0x7fbde4198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:14.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbdea2b6640 1 -- 172.21.15.33:0/2984920456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbde41082a0 msgr2=0x7fbde419ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:14.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbdea2b6640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbde41082a0 0x7fbde419ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:14.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbdea2b6640 1 -- 172.21.15.33:0/2984920456 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde419d520 con 0x7fbde4102d30 2024-09-16T00:15:14.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbdeaab7640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbde41082a0 0x7fbde419ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:15:14.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbde9ab5640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbde41036e0 0x7fbde4198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:15:14.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbdea2b6640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde4102d30 0x7fbde4198570 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbde000e9f0 tx=0x7fbde000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:14.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbddb7fe640 1 -- 172.21.15.33:0/2984920456 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde000cd80 con 0x7fbde4102d30 2024-09-16T00:15:14.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbddb7fe640 1 -- 172.21.15.33:0/2984920456 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbde0005500 con 0x7fbde4102d30 2024-09-16T00:15:14.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbdec51a640 1 -- 172.21.15.33:0/2984920456 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde419d810 con 0x7fbde4102d30 2024-09-16T00:15:14.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbddb7fe640 1 -- 172.21.15.33:0/2984920456 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0010690 con 0x7fbde4102d30 2024-09-16T00:15:14.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.224+0000 7fbdec51a640 1 -- 172.21.15.33:0/2984920456 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde41a50f0 con 0x7fbde4102d30 2024-09-16T00:15:14.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.225+0000 7fbdec51a640 1 -- 172.21.15.33:0/2984920456 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde4103110 con 0x7fbde4102d30 2024-09-16T00:15:14.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.227+0000 7fbddb7fe640 1 -- 172.21.15.33:0/2984920456 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fbde00046c0 con 0x7fbde4102d30 2024-09-16T00:15:14.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.228+0000 7fbddb7fe640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbdd00a32c0 0x7fbdd00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:14.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.228+0000 7fbde9ab5640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbdd00a32c0 0x7fbdd00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:14.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.228+0000 7fbddb7fe640 1 -- 172.21.15.33:0/2984920456 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8770+0+0 (secure 0 0 0) 0x7fbde00d0250 con 0x7fbde4102d30 2024-09-16T00:15:14.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.228+0000 7fbde9ab5640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbdd00a32c0 0x7fbdd00a5780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4005e00 tx=0x7fbdd4005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:14.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.232+0000 7fbddb7fe640 1 -- 172.21.15.33:0/2984920456 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbde00541b0 con 0x7fbde4102d30 2024-09-16T00:15:14.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.408+0000 7fbdec51a640 1 -- 172.21.15.33:0/2984920456 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fbde41998d0 con 0x7fbde4102d30 2024-09-16T00:15:14.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.565+0000 7fbddb7fe640 1 -- 172.21.15.33:0/2984920456 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v85) ==== 169+0+0 (secure 0 0 0) 0x7fbde001f340 con 0x7fbde4102d30 2024-09-16T00:15:14.566 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 4 allow_ec_overwrites to true 2024-09-16T00:15:14.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.568+0000 7fbdec51a640 1 -- 172.21.15.33:0/2984920456 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbdd00a32c0 msgr2=0x7fbdd00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:14.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.568+0000 7fbdec51a640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbdd00a32c0 0x7fbdd00a5780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4005e00 tx=0x7fbdd4005c70 comp rx=0 tx=0).stop 2024-09-16T00:15:14.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.568+0000 7fbdec51a640 1 -- 172.21.15.33:0/2984920456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde4102d30 msgr2=0x7fbde4198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:14.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.568+0000 7fbdec51a640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde4102d30 0x7fbde4198570 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbde000e9f0 tx=0x7fbde000edf0 comp rx=0 tx=0).stop 2024-09-16T00:15:14.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.568+0000 7fbdec51a640 1 -- 172.21.15.33:0/2984920456 shutdown_connections 2024-09-16T00:15:14.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.568+0000 7fbdec51a640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbdd00a32c0 0x7fbdd00a5780 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:14.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.568+0000 7fbdec51a640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbde41082a0 0x7fbde419ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:14.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.568+0000 7fbdec51a640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbde41036e0 0x7fbde4198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:14.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.568+0000 7fbdec51a640 1 --2- 172.21.15.33:0/2984920456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde4102d30 0x7fbde4198570 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:14.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.568+0000 7fbdec51a640 1 -- 172.21.15.33:0/2984920456 >> 172.21.15.33:0/2984920456 conn(0x7fbde40fc7d0 msgr2=0x7fbde4105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:14.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.569+0000 7fbdec51a640 1 -- 172.21.15.33:0/2984920456 shutdown_connections 2024-09-16T00:15:14.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:14.569+0000 7fbdec51a640 1 -- 172.21.15.33:0/2984920456 wait complete. 2024-09-16T00:15:14.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:14 smithi033 ceph-mon[31243]: osdmap e84: 12 total, 12 up, 12 in 2024-09-16T00:15:14.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:14 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2984920456' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-16T00:15:14.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:14 smithi083 ceph-mon[35242]: osdmap e84: 12 total, 12 up, 12 in 2024-09-16T00:15:14.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:14 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2984920456' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-16T00:15:14.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:14 smithi060 ceph-mon[35185]: osdmap e84: 12 total, 12 up, 12 in 2024-09-16T00:15:14.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:14 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2984920456' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-16T00:15:15.244 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-09-16T00:15:15.506 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:15 smithi033 ceph-mon[31243]: pgmap v287: 129 pgs: 8 creating+peering, 24 unknown, 97 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 752 B/s wr, 7 op/s 2024-09-16T00:15:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:15 smithi033 ceph-mon[31243]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T00:15:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:15 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2984920456' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-16T00:15:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:15 smithi033 ceph-mon[31243]: osdmap e85: 12 total, 12 up, 12 in 2024-09-16T00:15:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:15 smithi083 ceph-mon[35242]: pgmap v287: 129 pgs: 8 creating+peering, 24 unknown, 97 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 752 B/s wr, 7 op/s 2024-09-16T00:15:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:15 smithi083 ceph-mon[35242]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T00:15:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:15 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2984920456' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-16T00:15:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:15 smithi083 ceph-mon[35242]: osdmap e85: 12 total, 12 up, 12 in 2024-09-16T00:15:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:15 smithi060 ceph-mon[35185]: pgmap v287: 129 pgs: 8 creating+peering, 24 unknown, 97 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 752 B/s wr, 7 op/s 2024-09-16T00:15:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:15 smithi060 ceph-mon[35185]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T00:15:15.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:15 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2984920456' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-16T00:15:15.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:15 smithi060 ceph-mon[35185]: osdmap e85: 12 total, 12 up, 12 in 2024-09-16T00:15:15.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.969+0000 7f7a59a88640 1 -- 172.21.15.33:0/2671419915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a54102790 msgr2=0x7f7a541067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:15.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.969+0000 7f7a59a88640 1 --2- 172.21.15.33:0/2671419915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a54102790 0x7f7a541067e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009900 tx=0x7f7a3c02a410 comp rx=0 tx=0).stop 2024-09-16T00:15:15.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.969+0000 7f7a59a88640 1 -- 172.21.15.33:0/2671419915 shutdown_connections 2024-09-16T00:15:15.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.969+0000 7f7a59a88640 1 --2- 172.21.15.33:0/2671419915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a54106f10 0x7f7a5410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:15.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.969+0000 7f7a59a88640 1 --2- 172.21.15.33:0/2671419915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a54102790 0x7f7a541067e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:15.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.969+0000 7f7a59a88640 1 --2- 172.21.15.33:0/2671419915 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a54101de0 0x7f7a541021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:15.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.969+0000 7f7a59a88640 1 -- 172.21.15.33:0/2671419915 >> 172.21.15.33:0/2671419915 conn(0x7f7a540fd8c0 msgr2=0x7f7a540ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:15.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.970+0000 7f7a59a88640 1 -- 172.21.15.33:0/2671419915 shutdown_connections 2024-09-16T00:15:15.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.970+0000 7f7a59a88640 1 -- 172.21.15.33:0/2671419915 wait complete. 2024-09-16T00:15:15.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.970+0000 7f7a59a88640 1 Processor -- start 2024-09-16T00:15:15.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a59a88640 1 -- start start 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a59a88640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a54101de0 0x7f7a54198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a59a88640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a54102790 0x7f7a541989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a59a88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a54106f10 0x7f7a5419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a59a88640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7a54111f90 con 0x7f7a54102790 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a59a88640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7a54111e10 con 0x7f7a54101de0 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a59a88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7a54112110 con 0x7f7a54106f10 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a52ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a54102790 0x7f7a541989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a52ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a54102790 0x7f7a541989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58912/0 (socket says 172.21.15.33:58912) 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a52ffd640 1 -- 172.21.15.33:0/2700940540 learned_addr learned my addr 172.21.15.33:0/2700940540 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a537fe640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a54101de0 0x7f7a54198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a53fff640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a54106f10 0x7f7a5419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a52ffd640 1 -- 172.21.15.33:0/2700940540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a54106f10 msgr2=0x7f7a5419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:15.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a52ffd640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a54106f10 0x7f7a5419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:15.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a52ffd640 1 -- 172.21.15.33:0/2700940540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a54101de0 msgr2=0x7f7a54198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:15.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a52ffd640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a54101de0 0x7f7a54198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:15.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.971+0000 7f7a52ffd640 1 -- 172.21.15.33:0/2700940540 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a5419d440 con 0x7f7a54102790 2024-09-16T00:15:15.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.972+0000 7f7a52ffd640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a54102790 0x7f7a541989d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c006010 tx=0x7f7a3c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:15.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.972+0000 7f7a50ff9640 1 -- 172.21.15.33:0/2700940540 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c03f030 con 0x7f7a54102790 2024-09-16T00:15:15.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.972+0000 7f7a50ff9640 1 -- 172.21.15.33:0/2700940540 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a3c003ce0 con 0x7f7a54102790 2024-09-16T00:15:15.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.972+0000 7f7a59a88640 1 -- 172.21.15.33:0/2700940540 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a5419d6d0 con 0x7f7a54102790 2024-09-16T00:15:15.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.972+0000 7f7a50ff9640 1 -- 172.21.15.33:0/2700940540 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c035070 con 0x7f7a54102790 2024-09-16T00:15:15.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.972+0000 7f7a59a88640 1 -- 172.21.15.33:0/2700940540 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a541a5010 con 0x7f7a54102790 2024-09-16T00:15:15.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.973+0000 7f7a59a88640 1 -- 172.21.15.33:0/2700940540 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a541021c0 con 0x7f7a54102790 2024-09-16T00:15:15.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.974+0000 7f7a50ff9640 1 -- 172.21.15.33:0/2700940540 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f7a3c042030 con 0x7f7a54102790 2024-09-16T00:15:15.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.975+0000 7f7a50ff9640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7a400a32c0 0x7f7a400a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:15.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.975+0000 7f7a537fe640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7a400a32c0 0x7f7a400a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:15.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.975+0000 7f7a50ff9640 1 -- 172.21.15.33:0/2700940540 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8770+0+0 (secure 0 0 0) 0x7f7a3c0efe60 con 0x7f7a54102790 2024-09-16T00:15:15.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.975+0000 7f7a537fe640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7a400a32c0 0x7f7a400a5780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009990 tx=0x7f7a44009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:15.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:15.979+0000 7f7a50ff9640 1 -- 172.21.15.33:0/2700940540 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7a3c0bc960 con 0x7f7a54102790 2024-09-16T00:15:16.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.192+0000 7f7a59a88640 1 -- 172.21.15.33:0/2700940540 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f7a541997f0 con 0x7f7a54102790 2024-09-16T00:15:16.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.574+0000 7f7a50ff9640 1 -- 172.21.15.33:0/2700940540 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f7a541997f0 con 0x7f7a54102790 2024-09-16T00:15:16.575 INFO:teuthology.orchestra.run.smithi033.stderr:added data pool 4 to fsmap 2024-09-16T00:15:16.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.576+0000 7f7a59a88640 1 -- 172.21.15.33:0/2700940540 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7a400a32c0 msgr2=0x7f7a400a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:16.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.576+0000 7f7a59a88640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7a400a32c0 0x7f7a400a5780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009990 tx=0x7f7a44009110 comp rx=0 tx=0).stop 2024-09-16T00:15:16.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.576+0000 7f7a59a88640 1 -- 172.21.15.33:0/2700940540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a54102790 msgr2=0x7f7a541989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:16.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.576+0000 7f7a59a88640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a54102790 0x7f7a541989d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c006010 tx=0x7f7a3c0042f0 comp rx=0 tx=0).stop 2024-09-16T00:15:16.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.576+0000 7f7a59a88640 1 -- 172.21.15.33:0/2700940540 shutdown_connections 2024-09-16T00:15:16.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.576+0000 7f7a59a88640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7a400a32c0 0x7f7a400a5780 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:16.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.576+0000 7f7a59a88640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a54106f10 0x7f7a5419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:16.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.576+0000 7f7a59a88640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a54102790 0x7f7a541989d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:16.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.576+0000 7f7a59a88640 1 --2- 172.21.15.33:0/2700940540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a54101de0 0x7f7a54198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:16.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.576+0000 7f7a59a88640 1 -- 172.21.15.33:0/2700940540 >> 172.21.15.33:0/2700940540 conn(0x7f7a540fd8c0 msgr2=0x7f7a540fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:16.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.576+0000 7f7a59a88640 1 -- 172.21.15.33:0/2700940540 shutdown_connections 2024-09-16T00:15:16.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:16.576+0000 7f7a59a88640 1 -- 172.21.15.33:0/2700940540 wait complete. 2024-09-16T00:15:16.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:16 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2700940540' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-16T00:15:16.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:16 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2700940540' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-16T00:15:16.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:16 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2700940540' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-16T00:15:17.317 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:15:17.582 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:17 smithi033 ceph-mon[31243]: pgmap v289: 129 pgs: 8 creating+peering, 1 unknown, 120 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 251 B/s wr, 5 op/s 2024-09-16T00:15:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:17 smithi033 ceph-mon[31243]: osdmap e86: 12 total, 12 up, 12 in 2024-09-16T00:15:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:17 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2700940540' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-16T00:15:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:17 smithi033 ceph-mon[31243]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:17 smithi083 ceph-mon[35242]: pgmap v289: 129 pgs: 8 creating+peering, 1 unknown, 120 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 251 B/s wr, 5 op/s 2024-09-16T00:15:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:17 smithi083 ceph-mon[35242]: osdmap e86: 12 total, 12 up, 12 in 2024-09-16T00:15:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:17 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2700940540' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-16T00:15:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:17 smithi083 ceph-mon[35242]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:17 smithi060 ceph-mon[35185]: pgmap v289: 129 pgs: 8 creating+peering, 1 unknown, 120 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 251 B/s wr, 5 op/s 2024-09-16T00:15:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:17 smithi060 ceph-mon[35185]: osdmap e86: 12 total, 12 up, 12 in 2024-09-16T00:15:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:17 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2700940540' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-16T00:15:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:17 smithi060 ceph-mon[35185]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:18.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.059+0000 7f33e2480640 1 -- 172.21.15.33:0/725333342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33dc074620 msgr2=0x7f33dc074a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:18.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.059+0000 7f33e2480640 1 --2- 172.21.15.33:0/725333342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33dc074620 0x7f33dc074a00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009900 tx=0x7f33cc02a480 comp rx=0 tx=0).stop 2024-09-16T00:15:18.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.059+0000 7f33e2480640 1 -- 172.21.15.33:0/725333342 shutdown_connections 2024-09-16T00:15:18.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.059+0000 7f33e2480640 1 --2- 172.21.15.33:0/725333342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33dc10e3c0 0x7f33dc110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:18.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.059+0000 7f33e2480640 1 --2- 172.21.15.33:0/725333342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33dc074f40 0x7f33dc10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:18.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.059+0000 7f33e2480640 1 --2- 172.21.15.33:0/725333342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33dc074620 0x7f33dc074a00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:18.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.059+0000 7f33e2480640 1 -- 172.21.15.33:0/725333342 >> 172.21.15.33:0/725333342 conn(0x7f33dc0fd8c0 msgr2=0x7f33dc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:18.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.059+0000 7f33e2480640 1 -- 172.21.15.33:0/725333342 shutdown_connections 2024-09-16T00:15:18.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.059+0000 7f33e2480640 1 -- 172.21.15.33:0/725333342 wait complete. 2024-09-16T00:15:18.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33e2480640 1 Processor -- start 2024-09-16T00:15:18.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33e2480640 1 -- start start 2024-09-16T00:15:18.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33e2480640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33dc074620 0x7f33dc19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:18.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33e2480640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33dc074f40 0x7f33dc19ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:18.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33e2480640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33dc10e3c0 0x7f33dc19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:18.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33e2480640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1141c0 con 0x7f33dc074620 2024-09-16T00:15:18.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33e2480640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc114040 con 0x7f33dc074f40 2024-09-16T00:15:18.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33e2480640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc114340 con 0x7f33dc10e3c0 2024-09-16T00:15:18.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33dbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33dc074620 0x7f33dc19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:18.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33dbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33dc074620 0x7f33dc19a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58932/0 (socket says 172.21.15.33:58932) 2024-09-16T00:15:18.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33dbfff640 1 -- 172.21.15.33:0/1057858517 learned_addr learned my addr 172.21.15.33:0/1057858517 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:18.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33e0a1d640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33dc10e3c0 0x7f33dc19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:18.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33db7fe640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33dc074f40 0x7f33dc19ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:18.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33dbfff640 1 -- 172.21.15.33:0/1057858517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33dc10e3c0 msgr2=0x7f33dc19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:18.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33dbfff640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33dc10e3c0 0x7f33dc19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:18.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33dbfff640 1 -- 172.21.15.33:0/1057858517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33dc074f40 msgr2=0x7f33dc19ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:18.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33dbfff640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33dc074f40 0x7f33dc19ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:18.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33dbfff640 1 -- 172.21.15.33:0/1057858517 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33dc19f670 con 0x7f33dc074620 2024-09-16T00:15:18.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33e0a1d640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33dc10e3c0 0x7f33dc19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:15:18.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.060+0000 7f33db7fe640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33dc074f40 0x7f33dc19ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:15:18.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.061+0000 7f33dbfff640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33dc074620 0x7f33dc19a6c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f33cc006010 tx=0x7f33cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:18.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.061+0000 7f33d97fa640 1 -- 172.21.15.33:0/1057858517 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc03f030 con 0x7f33dc074620 2024-09-16T00:15:18.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.061+0000 7f33d97fa640 1 -- 172.21.15.33:0/1057858517 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33cc003ce0 con 0x7f33dc074620 2024-09-16T00:15:18.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.061+0000 7f33e2480640 1 -- 172.21.15.33:0/1057858517 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33dc19f900 con 0x7f33dc074620 2024-09-16T00:15:18.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.061+0000 7f33d97fa640 1 -- 172.21.15.33:0/1057858517 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc035070 con 0x7f33dc074620 2024-09-16T00:15:18.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.061+0000 7f33e2480640 1 -- 172.21.15.33:0/1057858517 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33dc1a7240 con 0x7f33dc074620 2024-09-16T00:15:18.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.062+0000 7f33e2480640 1 -- 172.21.15.33:0/1057858517 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33dc074a00 con 0x7f33dc074620 2024-09-16T00:15:18.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.063+0000 7f33d97fa640 1 -- 172.21.15.33:0/1057858517 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f33cc042030 con 0x7f33dc074620 2024-09-16T00:15:18.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.064+0000 7f33d97fa640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f33c40a32c0 0x7f33c40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:18.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.064+0000 7f33d97fa640 1 -- 172.21.15.33:0/1057858517 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f33cc0f0190 con 0x7f33dc074620 2024-09-16T00:15:18.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.064+0000 7f33db7fe640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f33c40a32c0 0x7f33c40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:18.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.064+0000 7f33db7fe640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f33c40a32c0 0x7f33c40a5780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f33dc10f750 tx=0x7f33d0005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:18.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.068+0000 7f33d97fa640 1 -- 172.21.15.33:0/1057858517 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f33cc0bcd30 con 0x7f33dc074620 2024-09-16T00:15:18.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.280+0000 7f33e2480640 1 -- 172.21.15.33:0/1057858517 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33dc109570 con 0x7f33dc074620 2024-09-16T00:15:18.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.281+0000 7f33d97fa640 1 -- 172.21.15.33:0/1057858517 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11830 (secure 0 0 0) 0x7f33cc0dc090 con 0x7f33dc074620 2024-09-16T00:15:18.282 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:15:18.282 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":17,"btime":"2024-09-16T00:15:16:567667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:15:16.193712+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24494,"qdb_cluster":[24494]},"id":1}]} 2024-09-16T00:15:18.282 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 17 2024-09-16T00:15:18.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.283+0000 7f33e2480640 1 -- 172.21.15.33:0/1057858517 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f33c40a32c0 msgr2=0x7f33c40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:18.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.283+0000 7f33e2480640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f33c40a32c0 0x7f33c40a5780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f33dc10f750 tx=0x7f33d0005e50 comp rx=0 tx=0).stop 2024-09-16T00:15:18.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.283+0000 7f33e2480640 1 -- 172.21.15.33:0/1057858517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33dc074620 msgr2=0x7f33dc19a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:18.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.283+0000 7f33e2480640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33dc074620 0x7f33dc19a6c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f33cc006010 tx=0x7f33cc0042f0 comp rx=0 tx=0).stop 2024-09-16T00:15:18.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.283+0000 7f33e2480640 1 -- 172.21.15.33:0/1057858517 shutdown_connections 2024-09-16T00:15:18.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.283+0000 7f33e2480640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f33c40a32c0 0x7f33c40a5780 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:18.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.283+0000 7f33e2480640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33dc10e3c0 0x7f33dc19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:18.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.283+0000 7f33e2480640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33dc074f40 0x7f33dc19ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:18.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.283+0000 7f33e2480640 1 --2- 172.21.15.33:0/1057858517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33dc074620 0x7f33dc19a6c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:18.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.283+0000 7f33e2480640 1 -- 172.21.15.33:0/1057858517 >> 172.21.15.33:0/1057858517 conn(0x7f33dc0fd8c0 msgr2=0x7f33dc0fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:18.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.283+0000 7f33e2480640 1 -- 172.21.15.33:0/1057858517 shutdown_connections 2024-09-16T00:15:18.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:18.283+0000 7f33e2480640 1 -- 172.21.15.33:0/1057858517 wait complete. 2024-09-16T00:15:18.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:18 smithi033 ceph-mon[31243]: pgmap v291: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-16T00:15:18.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:18 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1057858517' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:15:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:18 smithi083 ceph-mon[35242]: pgmap v291: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-16T00:15:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:18 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1057858517' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:15:18.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:15:18.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:18 smithi060 ceph-mon[35185]: pgmap v291: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-16T00:15:18.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:18 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1057858517' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:15:19.235 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:19.671 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:19 smithi033 ceph-mon[31243]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T00:15:19.671 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:19 smithi033 ceph-mon[31243]: Cluster is now healthy 2024-09-16T00:15:19.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.753+0000 7efffba87640 1 -- 172.21.15.33:0/3126998716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efff4074f40 msgr2=0x7efff410de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:19.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.753+0000 7efffba87640 1 --2- 172.21.15.33:0/3126998716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efff4074f40 0x7efff410de80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7effe4009900 tx=0x7effe402a480 comp rx=0 tx=0).stop 2024-09-16T00:15:19.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.753+0000 7efffba87640 1 -- 172.21.15.33:0/3126998716 shutdown_connections 2024-09-16T00:15:19.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 --2- 172.21.15.33:0/3126998716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efff410e3c0 0x7efff4110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:19.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 --2- 172.21.15.33:0/3126998716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efff4074f40 0x7efff410de80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:19.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 --2- 172.21.15.33:0/3126998716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efff4074620 0x7efff4074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:19.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 -- 172.21.15.33:0/3126998716 >> 172.21.15.33:0/3126998716 conn(0x7efff40fd8c0 msgr2=0x7efff40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:19.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 -- 172.21.15.33:0/3126998716 shutdown_connections 2024-09-16T00:15:19.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 -- 172.21.15.33:0/3126998716 wait complete. 2024-09-16T00:15:19.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 Processor -- start 2024-09-16T00:15:19.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 -- start start 2024-09-16T00:15:19.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efff4074620 0x7efff419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:19.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efff4074f40 0x7efff419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:19.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efff410e3c0 0x7efff419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:19.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efff41141c0 con 0x7efff4074620 2024-09-16T00:15:19.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efff4114040 con 0x7efff4074f40 2024-09-16T00:15:19.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.754+0000 7efffba87640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efff4114340 con 0x7efff410e3c0 2024-09-16T00:15:19.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7efff9823640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efff4074620 0x7efff419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:19.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7efff9823640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efff4074620 0x7efff419a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58948/0 (socket says 172.21.15.33:58948) 2024-09-16T00:15:19.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7efff9823640 1 -- 172.21.15.33:0/1563713190 learned_addr learned my addr 172.21.15.33:0/1563713190 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7efff9022640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efff4074f40 0x7efff419abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7efff9823640 1 -- 172.21.15.33:0/1563713190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efff410e3c0 msgr2=0x7efff419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7efff9823640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efff410e3c0 0x7efff419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7efff9823640 1 -- 172.21.15.33:0/1563713190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efff4074f40 msgr2=0x7efff419abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7efff9823640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efff4074f40 0x7efff419abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7efff9823640 1 -- 172.21.15.33:0/1563713190 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff419f660 con 0x7efff4074620 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7efff9823640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efff4074620 0x7efff419a6b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efff000b4a0 tx=0x7efff000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7effeaffd640 1 -- 172.21.15.33:0/1563713190 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff0014070 con 0x7efff4074620 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7effeaffd640 1 -- 172.21.15.33:0/1563713190 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efff0010af0 con 0x7efff4074620 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7efffba87640 1 -- 172.21.15.33:0/1563713190 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff419f950 con 0x7efff4074620 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7effeaffd640 1 -- 172.21.15.33:0/1563713190 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff0017d10 con 0x7efff4074620 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.755+0000 7efffba87640 1 -- 172.21.15.33:0/1563713190 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff41a7230 con 0x7efff4074620 2024-09-16T00:15:19.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.756+0000 7efffba87640 1 -- 172.21.15.33:0/1563713190 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff4074a00 con 0x7efff4074620 2024-09-16T00:15:19.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.758+0000 7effeaffd640 1 -- 172.21.15.33:0/1563713190 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7efff0017070 con 0x7efff4074620 2024-09-16T00:15:19.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.759+0000 7effeaffd640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7effe00a3240 0x7effe00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:19.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.759+0000 7efff9022640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7effe00a3240 0x7effe00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:19.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.759+0000 7effeaffd640 1 -- 172.21.15.33:0/1563713190 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7efff00cef70 con 0x7efff4074620 2024-09-16T00:15:19.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.759+0000 7efff9022640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7effe00a3240 0x7effe00a5700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efff40ffec0 tx=0x7effe4031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:19.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.763+0000 7effeaffd640 1 -- 172.21.15.33:0/1563713190 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7efff009ba50 con 0x7efff4074620 2024-09-16T00:15:19.882 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:19 smithi060 ceph-mon[35185]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T00:15:19.882 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:19 smithi060 ceph-mon[35185]: Cluster is now healthy 2024-09-16T00:15:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:19 smithi083 ceph-mon[35242]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T00:15:19.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:19 smithi083 ceph-mon[35242]: Cluster is now healthy 2024-09-16T00:15:19.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.940+0000 7efffba87640 1 -- 172.21.15.33:0/1563713190 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efff4108fb0 con 0x7efff4074620 2024-09-16T00:15:19.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.942+0000 7effeaffd640 1 -- 172.21.15.33:0/1563713190 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24138 (secure 0 0 0) 0x7efff009b350 con 0x7efff4074620 2024-09-16T00:15:19.944 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:15:19.944 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":86,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:15:16.193527+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:15:19.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.944+0000 7efffba87640 1 -- 172.21.15.33:0/1563713190 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7effe00a3240 msgr2=0x7effe00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:19.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.944+0000 7efffba87640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7effe00a3240 0x7effe00a5700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efff40ffec0 tx=0x7effe4031490 comp rx=0 tx=0).stop 2024-09-16T00:15:19.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.944+0000 7efffba87640 1 -- 172.21.15.33:0/1563713190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efff4074620 msgr2=0x7efff419a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:19.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.944+0000 7efffba87640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efff4074620 0x7efff419a6b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efff000b4a0 tx=0x7efff000b960 comp rx=0 tx=0).stop 2024-09-16T00:15:19.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.944+0000 7efffba87640 1 -- 172.21.15.33:0/1563713190 shutdown_connections 2024-09-16T00:15:19.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.944+0000 7efffba87640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7effe00a3240 0x7effe00a5700 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:19.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.944+0000 7efffba87640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efff410e3c0 0x7efff419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:19.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.944+0000 7efffba87640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efff4074f40 0x7efff419abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:19.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.944+0000 7efffba87640 1 --2- 172.21.15.33:0/1563713190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efff4074620 0x7efff419a6b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:19.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.944+0000 7efffba87640 1 -- 172.21.15.33:0/1563713190 >> 172.21.15.33:0/1563713190 conn(0x7efff40fd8c0 msgr2=0x7efff40ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:19.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.945+0000 7efffba87640 1 -- 172.21.15.33:0/1563713190 shutdown_connections 2024-09-16T00:15:19.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:19.945+0000 7efffba87640 1 -- 172.21.15.33:0/1563713190 wait complete. 2024-09-16T00:15:20.725 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:20 smithi033 ceph-mon[31243]: pgmap v292: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-16T00:15:20.726 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:20 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1563713190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:20.762 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:15:20.893 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:20 smithi060 ceph-mon[35185]: pgmap v292: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-16T00:15:20.894 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:20 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1563713190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:20.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:20 smithi083 ceph-mon[35242]: pgmap v292: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-16T00:15:20.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:20 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1563713190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:21.025 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:21.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.503+0000 7f683c88d640 1 -- 172.21.15.33:0/1082149086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6838101400 msgr2=0x7f683810d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:21.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.503+0000 7f683c88d640 1 --2- 172.21.15.33:0/1082149086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6838101400 0x7f683810d930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6830009cd0 tx=0x7f683002a8e0 comp rx=0 tx=0).stop 2024-09-16T00:15:21.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.503+0000 7f683c88d640 1 -- 172.21.15.33:0/1082149086 shutdown_connections 2024-09-16T00:15:21.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.503+0000 7f683c88d640 1 --2- 172.21.15.33:0/1082149086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6838101400 0x7f683810d930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:21.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.503+0000 7f683c88d640 1 --2- 172.21.15.33:0/1082149086 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6838100a60 0x7f6838100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:21.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.503+0000 7f683c88d640 1 --2- 172.21.15.33:0/1082149086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6838106a60 0x7f6838106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:21.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.503+0000 7f683c88d640 1 -- 172.21.15.33:0/1082149086 >> 172.21.15.33:0/1082149086 conn(0x7f68380fc780 msgr2=0x7f68380feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:21.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.503+0000 7f683c88d640 1 -- 172.21.15.33:0/1082149086 shutdown_connections 2024-09-16T00:15:21.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.503+0000 7f683c88d640 1 -- 172.21.15.33:0/1082149086 wait complete. 2024-09-16T00:15:21.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.504+0000 7f683c88d640 1 Processor -- start 2024-09-16T00:15:21.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.504+0000 7f683c88d640 1 -- start start 2024-09-16T00:15:21.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.504+0000 7f683c88d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6838100a60 0x7f6838102630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:21.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.504+0000 7f683c88d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6838101400 0x7f6838102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:21.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683c88d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6838106a60 0x7f68381030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:21.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683c88d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6838110e60 con 0x7f6838100a60 2024-09-16T00:15:21.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683c88d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6838110ce0 con 0x7f6838101400 2024-09-16T00:15:21.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683c88d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6838110fe0 con 0x7f6838106a60 2024-09-16T00:15:21.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6838100a60 0x7f6838102630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:21.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6838100a60 0x7f6838102630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36912/0 (socket says 172.21.15.33:36912) 2024-09-16T00:15:21.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683659c640 1 -- 172.21.15.33:0/2601926004 learned_addr learned my addr 172.21.15.33:0/2601926004 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:21.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f6835d9b640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6838101400 0x7f6838102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:21.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f6836d9d640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6838106a60 0x7f68381030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:21.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683659c640 1 -- 172.21.15.33:0/2601926004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6838106a60 msgr2=0x7f68381030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683659c640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6838106a60 0x7f68381030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683659c640 1 -- 172.21.15.33:0/2601926004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6838101400 msgr2=0x7f6838102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683659c640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6838101400 0x7f6838102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683659c640 1 -- 172.21.15.33:0/2601926004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68381a6930 con 0x7f6838100a60 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f6835d9b640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6838101400 0x7f6838102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683659c640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6838100a60 0x7f6838102630 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f682800d770 tx=0x7f682800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f68277fe640 1 -- 172.21.15.33:0/2601926004 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6828017070 con 0x7f6838100a60 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683c88d640 1 -- 172.21.15.33:0/2601926004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68381a6c20 con 0x7f6838100a60 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f68277fe640 1 -- 172.21.15.33:0/2601926004 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6828005500 con 0x7f6838100a60 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f68277fe640 1 -- 172.21.15.33:0/2601926004 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6828004e00 con 0x7f6838100a60 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.505+0000 7f683c88d640 1 -- 172.21.15.33:0/2601926004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68381a7160 con 0x7f6838100a60 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.507+0000 7f683c88d640 1 -- 172.21.15.33:0/2601926004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6838100ec0 con 0x7f6838100a60 2024-09-16T00:15:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.508+0000 7f68277fe640 1 -- 172.21.15.33:0/2601926004 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f6828004fa0 con 0x7f6838100a60 2024-09-16T00:15:21.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.508+0000 7f68277fe640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68200a3240 0x7f68200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:21.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.509+0000 7f6835d9b640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68200a3240 0x7f68200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:21.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.509+0000 7f68277fe640 1 -- 172.21.15.33:0/2601926004 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f68280cf510 con 0x7f6838100a60 2024-09-16T00:15:21.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.509+0000 7f6835d9b640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68200a3240 0x7f68200a5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f68381040c0 tx=0x7f682c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:21.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.513+0000 7f68277fe640 1 -- 172.21.15.33:0/2601926004 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f682809bff0 con 0x7f6838100a60 2024-09-16T00:15:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.694+0000 7f683c88d640 1 -- 172.21.15.33:0/2601926004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68381a73d0 con 0x7f6838100a60 2024-09-16T00:15:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.695+0000 7f68277fe640 1 -- 172.21.15.33:0/2601926004 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24138 (secure 0 0 0) 0x7f682809b8f0 con 0x7f6838100a60 2024-09-16T00:15:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:15:21.696 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":86,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:15:16.193527+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:15:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.697+0000 7f683c88d640 1 -- 172.21.15.33:0/2601926004 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68200a3240 msgr2=0x7f68200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.697+0000 7f683c88d640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68200a3240 0x7f68200a5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f68381040c0 tx=0x7f682c005f90 comp rx=0 tx=0).stop 2024-09-16T00:15:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.697+0000 7f683c88d640 1 -- 172.21.15.33:0/2601926004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6838100a60 msgr2=0x7f6838102630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.697+0000 7f683c88d640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6838100a60 0x7f6838102630 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f682800d770 tx=0x7f682800dc30 comp rx=0 tx=0).stop 2024-09-16T00:15:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.697+0000 7f683c88d640 1 -- 172.21.15.33:0/2601926004 shutdown_connections 2024-09-16T00:15:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.697+0000 7f683c88d640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68200a3240 0x7f68200a5700 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.697+0000 7f683c88d640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6838106a60 0x7f68381030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.697+0000 7f683c88d640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6838101400 0x7f6838102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.697+0000 7f683c88d640 1 --2- 172.21.15.33:0/2601926004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6838100a60 0x7f6838102630 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.697+0000 7f683c88d640 1 -- 172.21.15.33:0/2601926004 >> 172.21.15.33:0/2601926004 conn(0x7f68380fc780 msgr2=0x7f683806a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.698+0000 7f683c88d640 1 -- 172.21.15.33:0/2601926004 shutdown_connections 2024-09-16T00:15:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:21.698+0000 7f683c88d640 1 -- 172.21.15.33:0/2601926004 wait complete. 2024-09-16T00:15:22.378 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:15:22.639 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:22 smithi060 ceph-mon[35185]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-16T00:15:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:22 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2601926004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:23.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:22 smithi033 ceph-mon[31243]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-16T00:15:23.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:22 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2601926004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.095+0000 7efece7f9640 1 -- 172.21.15.33:0/685428346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec8102ef0 msgr2=0x7efec810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.095+0000 7efece7f9640 1 --2- 172.21.15.33:0/685428346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec8102ef0 0x7efec810da40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efec0009ca0 tx=0x7efec002a950 comp rx=0 tx=0).stop 2024-09-16T00:15:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.095+0000 7efece7f9640 1 -- 172.21.15.33:0/685428346 shutdown_connections 2024-09-16T00:15:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.095+0000 7efece7f9640 1 --2- 172.21.15.33:0/685428346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec8102ef0 0x7efec810da40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.095+0000 7efece7f9640 1 --2- 172.21.15.33:0/685428346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efec8102570 0x7efec81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.095+0000 7efece7f9640 1 --2- 172.21.15.33:0/685428346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec8100e20 0x7efec8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.095+0000 7efece7f9640 1 -- 172.21.15.33:0/685428346 >> 172.21.15.33:0/685428346 conn(0x7efec80fc7d0 msgr2=0x7efec80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.095+0000 7efece7f9640 1 -- 172.21.15.33:0/685428346 shutdown_connections 2024-09-16T00:15:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.095+0000 7efece7f9640 1 -- 172.21.15.33:0/685428346 wait complete. 2024-09-16T00:15:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.095+0000 7efece7f9640 1 Processor -- start 2024-09-16T00:15:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.095+0000 7efece7f9640 1 -- start start 2024-09-16T00:15:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efece7f9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec8100e20 0x7efec819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efece7f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efec8102570 0x7efec819ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efece7f9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec8102ef0 0x7efec819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efece7f9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efec8110fa0 con 0x7efec8100e20 2024-09-16T00:15:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efece7f9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efec8110e20 con 0x7efec8102ef0 2024-09-16T00:15:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efece7f9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efec8111120 con 0x7efec8102570 2024-09-16T00:15:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efec7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec8100e20 0x7efec819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efec7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec8100e20 0x7efec819a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36928/0 (socket says 172.21.15.33:36928) 2024-09-16T00:15:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efec7fff640 1 -- 172.21.15.33:0/1475980076 learned_addr learned my addr 172.21.15.33:0/1475980076 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efeccd96640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec8102ef0 0x7efec819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efec77fe640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efec8102570 0x7efec819ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efec7fff640 1 -- 172.21.15.33:0/1475980076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efec8102570 msgr2=0x7efec819ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efec7fff640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efec8102570 0x7efec819ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efec7fff640 1 -- 172.21.15.33:0/1475980076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec8102ef0 msgr2=0x7efec819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efec7fff640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec8102ef0 0x7efec819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.096+0000 7efec7fff640 1 -- 172.21.15.33:0/1475980076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec819f680 con 0x7efec8100e20 2024-09-16T00:15:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.097+0000 7efec7fff640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec8100e20 0x7efec819a6d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efeb800c890 tx=0x7efeb800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.097+0000 7efec57fa640 1 -- 172.21.15.33:0/1475980076 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb8004220 con 0x7efec8100e20 2024-09-16T00:15:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.097+0000 7efec57fa640 1 -- 172.21.15.33:0/1475980076 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efeb80043c0 con 0x7efec8100e20 2024-09-16T00:15:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.097+0000 7efece7f9640 1 -- 172.21.15.33:0/1475980076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec819f970 con 0x7efec8100e20 2024-09-16T00:15:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.097+0000 7efec57fa640 1 -- 172.21.15.33:0/1475980076 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb80049e0 con 0x7efec8100e20 2024-09-16T00:15:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.097+0000 7efece7f9640 1 -- 172.21.15.33:0/1475980076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec81a7250 con 0x7efec8100e20 2024-09-16T00:15:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.098+0000 7efece7f9640 1 -- 172.21.15.33:0/1475980076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efec8101200 con 0x7efec8100e20 2024-09-16T00:15:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.099+0000 7efec57fa640 1 -- 172.21.15.33:0/1475980076 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7efeb80079b0 con 0x7efec8100e20 2024-09-16T00:15:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.100+0000 7efec57fa640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efeb00a3240 0x7efeb00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.100+0000 7efec77fe640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efeb00a3240 0x7efeb00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.100+0000 7efec57fa640 1 -- 172.21.15.33:0/1475980076 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7efeb80cfc40 con 0x7efec8100e20 2024-09-16T00:15:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.100+0000 7efec77fe640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efeb00a3240 0x7efeb00a5700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efec810ddf0 tx=0x7efebc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.103+0000 7efec57fa640 1 -- 172.21.15.33:0/1475980076 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7efeb8014030 con 0x7efec8100e20 2024-09-16T00:15:23.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:22 smithi083 ceph-mon[35242]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-16T00:15:23.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:22 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2601926004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:23.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.316+0000 7efece7f9640 1 -- 172.21.15.33:0/1475980076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efec81a74e0 con 0x7efec8100e20 2024-09-16T00:15:23.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.317+0000 7efec57fa640 1 -- 172.21.15.33:0/1475980076 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11830 (secure 0 0 0) 0x7efeb809c6c0 con 0x7efec8100e20 2024-09-16T00:15:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:15:23.318 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":17,"btime":"2024-09-16T00:15:16:567667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:15:16.193712+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24494,"qdb_cluster":[24494]},"id":1}]} 2024-09-16T00:15:23.318 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 17 2024-09-16T00:15:23.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.319+0000 7efece7f9640 1 -- 172.21.15.33:0/1475980076 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efeb00a3240 msgr2=0x7efeb00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:23.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.319+0000 7efece7f9640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efeb00a3240 0x7efeb00a5700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efec810ddf0 tx=0x7efebc008040 comp rx=0 tx=0).stop 2024-09-16T00:15:23.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.319+0000 7efece7f9640 1 -- 172.21.15.33:0/1475980076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec8100e20 msgr2=0x7efec819a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:23.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.319+0000 7efece7f9640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec8100e20 0x7efec819a6d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efeb800c890 tx=0x7efeb800cd50 comp rx=0 tx=0).stop 2024-09-16T00:15:23.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.319+0000 7efece7f9640 1 -- 172.21.15.33:0/1475980076 shutdown_connections 2024-09-16T00:15:23.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.319+0000 7efece7f9640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efeb00a3240 0x7efeb00a5700 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:23.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.319+0000 7efece7f9640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec8102ef0 0x7efec819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:23.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.319+0000 7efece7f9640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efec8102570 0x7efec819ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:23.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.319+0000 7efece7f9640 1 --2- 172.21.15.33:0/1475980076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec8100e20 0x7efec819a6d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:23.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.319+0000 7efece7f9640 1 -- 172.21.15.33:0/1475980076 >> 172.21.15.33:0/1475980076 conn(0x7efec80fc7d0 msgr2=0x7efec806a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:23.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.319+0000 7efece7f9640 1 -- 172.21.15.33:0/1475980076 shutdown_connections 2024-09-16T00:15:23.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:23.319+0000 7efece7f9640 1 -- 172.21.15.33:0/1475980076 wait complete. 2024-09-16T00:15:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:23 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1475980076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:15:23.996 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:23 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1475980076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:15:24.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:23 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1475980076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:15:24.140 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:15:24.404 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:24.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:24 smithi033 ceph-mon[31243]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:24.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.880+0000 7fb4e76e1640 1 -- 172.21.15.33:0/2224849252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4e0100ca0 msgr2=0x7fb4e0106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:24.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.880+0000 7fb4e76e1640 1 --2- 172.21.15.33:0/2224849252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4e0100ca0 0x7fb4e0106840 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009930 tx=0x7fb4d002a420 comp rx=0 tx=0).stop 2024-09-16T00:15:24.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.880+0000 7fb4e76e1640 1 -- 172.21.15.33:0/2224849252 shutdown_connections 2024-09-16T00:15:24.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.880+0000 7fb4e76e1640 1 --2- 172.21.15.33:0/2224849252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4e0106f50 0x7fb4e010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:24.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.880+0000 7fb4e76e1640 1 --2- 172.21.15.33:0/2224849252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4e0100ca0 0x7fb4e0106840 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:24.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.880+0000 7fb4e76e1640 1 --2- 172.21.15.33:0/2224849252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4e01002f0 0x7fb4e01006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:24.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.880+0000 7fb4e76e1640 1 -- 172.21.15.33:0/2224849252 >> 172.21.15.33:0/2224849252 conn(0x7fb4e0077210 msgr2=0x7fb4e00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:24.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.880+0000 7fb4e76e1640 1 -- 172.21.15.33:0/2224849252 shutdown_connections 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.880+0000 7fb4e76e1640 1 -- 172.21.15.33:0/2224849252 wait complete. 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e76e1640 1 Processor -- start 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e76e1640 1 -- start start 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e76e1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4e01002f0 0x7fb4e0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e76e1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4e0100ca0 0x7fb4e0198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e76e1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4e0106f50 0x7fb4e019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e76e1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e0111fa0 con 0x7fb4e0106f50 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e76e1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e0111e20 con 0x7fb4e0100ca0 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e76e1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e0112120 con 0x7fb4e01002f0 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e5c7e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4e0106f50 0x7fb4e019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e5c7e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4e0106f50 0x7fb4e019cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36944/0 (socket says 172.21.15.33:36944) 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e5c7e640 1 -- 172.21.15.33:0/819539966 learned_addr learned my addr 172.21.15.33:0/819539966 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:24.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e547d640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4e01002f0 0x7fb4e0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.881+0000 7fb4e4c7c640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4e0100ca0 0x7fb4e0198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.882+0000 7fb4e5c7e640 1 -- 172.21.15.33:0/819539966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4e01002f0 msgr2=0x7fb4e0198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.882+0000 7fb4e5c7e640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4e01002f0 0x7fb4e0198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.882+0000 7fb4e5c7e640 1 -- 172.21.15.33:0/819539966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4e0100ca0 msgr2=0x7fb4e0198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.882+0000 7fb4e5c7e640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4e0100ca0 0x7fb4e0198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.882+0000 7fb4e5c7e640 1 -- 172.21.15.33:0/819539966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4e019d4d0 con 0x7fb4e0106f50 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.882+0000 7fb4e547d640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4e01002f0 0x7fb4e0198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.882+0000 7fb4e5c7e640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4e0106f50 0x7fb4e019cdf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb4d800eaa0 tx=0x7fb4d800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.882+0000 7fb4d67fc640 1 -- 172.21.15.33:0/819539966 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d800cdb0 con 0x7fb4e0106f50 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.882+0000 7fb4d67fc640 1 -- 172.21.15.33:0/819539966 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4d8005020 con 0x7fb4e0106f50 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.882+0000 7fb4d67fc640 1 -- 172.21.15.33:0/819539966 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d80116b0 con 0x7fb4e0106f50 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.882+0000 7fb4e76e1640 1 -- 172.21.15.33:0/819539966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4e019d7c0 con 0x7fb4e0106f50 2024-09-16T00:15:24.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.882+0000 7fb4e76e1640 1 -- 172.21.15.33:0/819539966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4e01a50a0 con 0x7fb4e0106f50 2024-09-16T00:15:24.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.883+0000 7fb4e76e1640 1 -- 172.21.15.33:0/819539966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4e01006d0 con 0x7fb4e0106f50 2024-09-16T00:15:24.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.885+0000 7fb4d67fc640 1 -- 172.21.15.33:0/819539966 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fb4d8005560 con 0x7fb4e0106f50 2024-09-16T00:15:24.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.885+0000 7fb4d67fc640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4cc0a3240 0x7fb4cc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:24.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.885+0000 7fb4e547d640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4cc0a3240 0x7fb4cc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:24.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.885+0000 7fb4d67fc640 1 -- 172.21.15.33:0/819539966 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fb4d80cfc90 con 0x7fb4e0106f50 2024-09-16T00:15:24.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.886+0000 7fb4e547d640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4cc0a3240 0x7fb4cc0a5700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009fc0 tx=0x7fb4dc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:24.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:24.890+0000 7fb4d67fc640 1 -- 172.21.15.33:0/819539966 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb4d8016030 con 0x7fb4e0106f50 2024-09-16T00:15:24.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:24 smithi060 ceph-mon[35185]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:25.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.068+0000 7fb4e76e1640 1 -- 172.21.15.33:0/819539966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4e0109b90 con 0x7fb4e0106f50 2024-09-16T00:15:25.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.070+0000 7fb4d67fc640 1 -- 172.21.15.33:0/819539966 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24138 (secure 0 0 0) 0x7fb4d809c120 con 0x7fb4e0106f50 2024-09-16T00:15:25.070 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:15:25.071 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":86,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:15:16.193527+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:15:25.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.072+0000 7fb4e76e1640 1 -- 172.21.15.33:0/819539966 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4cc0a3240 msgr2=0x7fb4cc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:25.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.072+0000 7fb4e76e1640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4cc0a3240 0x7fb4cc0a5700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009fc0 tx=0x7fb4dc005f90 comp rx=0 tx=0).stop 2024-09-16T00:15:25.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.072+0000 7fb4e76e1640 1 -- 172.21.15.33:0/819539966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4e0106f50 msgr2=0x7fb4e019cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:25.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.072+0000 7fb4e76e1640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4e0106f50 0x7fb4e019cdf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb4d800eaa0 tx=0x7fb4d800ef60 comp rx=0 tx=0).stop 2024-09-16T00:15:25.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.072+0000 7fb4e76e1640 1 -- 172.21.15.33:0/819539966 shutdown_connections 2024-09-16T00:15:25.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.072+0000 7fb4e76e1640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4cc0a3240 0x7fb4cc0a5700 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:25.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.072+0000 7fb4e76e1640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4e0106f50 0x7fb4e019cdf0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:25.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.072+0000 7fb4e76e1640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4e0100ca0 0x7fb4e0198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:25.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.072+0000 7fb4e76e1640 1 --2- 172.21.15.33:0/819539966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4e01002f0 0x7fb4e0198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:25.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.072+0000 7fb4e76e1640 1 -- 172.21.15.33:0/819539966 >> 172.21.15.33:0/819539966 conn(0x7fb4e0077210 msgr2=0x7fb4e0105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:25.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.072+0000 7fb4e76e1640 1 -- 172.21.15.33:0/819539966 shutdown_connections 2024-09-16T00:15:25.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:25.072+0000 7fb4e76e1640 1 -- 172.21.15.33:0/819539966 wait complete. 2024-09-16T00:15:25.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:24 smithi083 ceph-mon[35242]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:25.800 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T00:15:07.246355+0000', 'modified': '2024-09-16T00:15:16.193712+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24494}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.60:6839/1222932045', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6838', 'nonce': 1222932045}, {'type': 'v1', 'addr': '172.21.15.60:6839', 'nonce': 1222932045}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24494, 'qdb_cluster': [24494]} 2024-09-16T00:15:25.800 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T00:15:25.800 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-16T00:15:25.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:25 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/819539966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:26.055 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:26.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:25 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/819539966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:26.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:25 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/819539966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.507+0000 7f0865727640 1 -- 172.21.15.33:0/1234097680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0860103660 msgr2=0x7f086010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.507+0000 7f0865727640 1 --2- 172.21.15.33:0/1234097680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0860103660 0x7f086010bb50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0848009900 tx=0x7f084802a480 comp rx=0 tx=0).stop 2024-09-16T00:15:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 -- 172.21.15.33:0/1234097680 shutdown_connections 2024-09-16T00:15:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 --2- 172.21.15.33:0/1234097680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f086010c170 0x7f086010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 --2- 172.21.15.33:0/1234097680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0860103660 0x7f086010bb50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 --2- 172.21.15.33:0/1234097680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0860102d40 0x7f0860103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 -- 172.21.15.33:0/1234097680 >> 172.21.15.33:0/1234097680 conn(0x7f08600fc7a0 msgr2=0x7f08600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 -- 172.21.15.33:0/1234097680 shutdown_connections 2024-09-16T00:15:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 -- 172.21.15.33:0/1234097680 wait complete. 2024-09-16T00:15:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 Processor -- start 2024-09-16T00:15:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 -- start start 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0860102d40 0x7f0860198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0860103660 0x7f08601989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086010c170 0x7f086019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0860111f70 con 0x7f0860103660 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0860111df0 con 0x7f086010c170 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.508+0000 7f0865727640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f08601120f0 con 0x7f0860102d40 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f0857fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0860103660 0x7f08601989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f0857fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0860103660 0x7f08601989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36960/0 (socket says 172.21.15.33:36960) 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f0857fff640 1 -- 172.21.15.33:0/2302243189 learned_addr learned my addr 172.21.15.33:0/2302243189 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f085f7fe640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086010c170 0x7f086019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f085effd640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0860102d40 0x7f0860198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f0857fff640 1 -- 172.21.15.33:0/2302243189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0860102d40 msgr2=0x7f0860198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f0857fff640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0860102d40 0x7f0860198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f0857fff640 1 -- 172.21.15.33:0/2302243189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086010c170 msgr2=0x7f086019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f0857fff640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086010c170 0x7f086019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f0857fff640 1 -- 172.21.15.33:0/2302243189 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086019d420 con 0x7f0860103660 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f0857fff640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0860103660 0x7f08601989b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0848006010 tx=0x7f08480042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f085cff9640 1 -- 172.21.15.33:0/2302243189 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084803f030 con 0x7f0860103660 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f085cff9640 1 -- 172.21.15.33:0/2302243189 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0848003ce0 con 0x7f0860103660 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f0865727640 1 -- 172.21.15.33:0/2302243189 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f086019d6b0 con 0x7f0860103660 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f085cff9640 1 -- 172.21.15.33:0/2302243189 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0848035070 con 0x7f0860103660 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.509+0000 7f0865727640 1 -- 172.21.15.33:0/2302243189 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08601a4ff0 con 0x7f0860103660 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.512+0000 7f085cff9640 1 -- 172.21.15.33:0/2302243189 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f0848042030 con 0x7f0860103660 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.512+0000 7f0867b00740 1 Processor -- start 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.512+0000 7f0867b00740 1 -- start start 2024-09-16T00:15:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.512+0000 7f0867b00740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397bde0 0x56117397c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.512+0000 7f0867b00740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397c700 0x561173986a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.512+0000 7f0867b00740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 0x5611739893b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.512+0000 7f0867b00740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5611738de090 con 0x561173986fc0 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.512+0000 7f0867b00740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561173885a70 con 0x56117397bde0 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.512+0000 7f0867b00740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5611738da050 con 0x56117397c700 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.512+0000 7f0857fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 0x5611739893b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397bde0 0x56117397c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397bde0 0x56117397c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40130/0 (socket says 172.21.15.33:40130) 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f0857fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 0x5611739893b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36976/0 (socket says 172.21.15.33:36976) 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085f7fe640 1 -- 172.21.15.33:0/2662069215 learned_addr learned my addr 172.21.15.33:0/2662069215 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085effd640 1 --2- 172.21.15.33:0/2662069215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397c700 0x561173986a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085cff9640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f084c0a32c0 0x7f084c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085f7fe640 1 -- 172.21.15.33:0/2662069215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397c700 msgr2=0x561173986a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:15:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085f7fe640 1 --2- 172.21.15.33:0/2662069215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397c700 0x561173986a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085f7fe640 1 -- 172.21.15.33:0/2662069215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 msgr2=0x5611739893b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:15:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085f7fe640 1 --2- 172.21.15.33:0/2662069215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 0x5611739893b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085f7fe640 1 -- 172.21.15.33:0/2662069215 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561173892130 con 0x56117397bde0 2024-09-16T00:15:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085cff9640 1 -- 172.21.15.33:0/2302243189 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f08480f0160 con 0x7f0860103660 2024-09-16T00:15:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085f7fe640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f084c0a32c0 0x7f084c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085f7fe640 1 --2- 172.21.15.33:0/2662069215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397bde0 0x56117397c1c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0858009fc0 tx=0x7f0858010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58ffad5e8fa0dab4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f08557fa640 1 -- 172.21.15.33:0/2662069215 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085801c900 con 0x56117397bde0 2024-09-16T00:15:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f085f7fe640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f084c0a32c0 0x7f084c0a5780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0858020000 tx=0x7f085801d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f08557fa640 1 -- 172.21.15.33:0/2662069215 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0858013600 con 0x56117397bde0 2024-09-16T00:15:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.513+0000 7f08557fa640 1 -- 172.21.15.33:0/2662069215 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08580113e0 con 0x56117397bde0 2024-09-16T00:15:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.514+0000 7f0867b00740 1 -- 172.21.15.33:0/2662069215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397bde0 msgr2=0x56117397c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.514+0000 7f0867b00740 1 --2- 172.21.15.33:0/2662069215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397bde0 0x56117397c1c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0858009fc0 tx=0x7f0858010800 comp rx=0 tx=0).stop 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.514+0000 7f0867b00740 1 -- 172.21.15.33:0/2662069215 shutdown_connections 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.514+0000 7f0867b00740 1 --2- 172.21.15.33:0/2662069215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 0x5611739893b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.514+0000 7f0867b00740 1 --2- 172.21.15.33:0/2662069215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397c700 0x561173986a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.514+0000 7f0867b00740 1 --2- 172.21.15.33:0/2662069215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397bde0 0x56117397c1c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.514+0000 7f0867b00740 1 -- 172.21.15.33:0/2662069215 >> 172.21.15.33:0/2662069215 conn(0x561173978370 msgr2=0x561173978750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.514+0000 7f0867b00740 1 -- 172.21.15.33:0/2662069215 shutdown_connections 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.514+0000 7f0867b00740 1 -- 172.21.15.33:0/2662069215 wait complete. 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.515+0000 7f0867b00740 1 -- start start 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.515+0000 7f0867b00740 1 Processor -- start 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.515+0000 7f0867b00740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397bde0 0x561173986810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.515+0000 7f0867b00740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397c700 0x561173983b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.515+0000 7f0867b00740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561173986fc0 0x5611739840a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.515+0000 7f0867b00740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561173989b90 con 0x56117397bde0 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.515+0000 7f0867b00740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561173898f20 con 0x56117397c700 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.515+0000 7f0867b00740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561173989d10 con 0x561173986fc0 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.515+0000 7f0867b00740 10 client.0 _ll_register_callbacks cb 0x561173862030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.515+0000 7f085effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397bde0 0x561173986810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.515+0000 7f085effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397bde0 0x561173986810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36990/0 (socket says 172.21.15.33:36990) 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.515+0000 7f085effd640 1 -- 172.21.15.33:0/1320900741 learned_addr learned my addr 172.21.15.33:0/1320900741 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f0857fff640 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397c700 0x561173983b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f085effd640 1 -- 172.21.15.33:0/1320900741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561173986fc0 msgr2=0x5611739840a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f085f7fe640 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561173986fc0 0x5611739840a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f085effd640 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561173986fc0 0x5611739840a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f085effd640 1 -- 172.21.15.33:0/1320900741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397c700 msgr2=0x561173983b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f085effd640 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397c700 0x561173983b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f085effd640 1 -- 172.21.15.33:0/1320900741 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56117364f2b0 con 0x56117397bde0 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f0857fff640 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397c700 0x561173983b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f085effd640 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397bde0 0x561173986810 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f085000bee0 tx=0x7f0850012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f35da0c1427dbf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f08467fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f08467fc640 1 -- 172.21.15.33:0/1320900741 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850004230 con 0x56117397bde0 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f08467fc640 1 -- 172.21.15.33:0/1320900741 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f08500043d0 con 0x56117397bde0 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f0867b00740 1 -- 172.21.15.33:0/1320900741 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5611738eb600 con 0x56117397bde0 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f08467fc640 1 -- 172.21.15.33:0/1320900741 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f08500106a0 con 0x56117397bde0 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f0867b00740 10 client.14721 fetch_fsmap learned FSMap version 17 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f0867b00740 1 -- 172.21.15.33:0/1320900741 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5611739954c0 con 0x56117397bde0 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f08467fc640 1 -- 172.21.15.33:0/1320900741 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5611738eb600 con 0x56117397bde0 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.516+0000 7f08467fc640 1 -- 172.21.15.33:0/1320900741 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f0850003c30 con 0x56117397bde0 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.517+0000 7f0867b00740 10 client.14721 fetch_fsmap finished waiting for FSMap version 17 2024-09-16T00:15:26.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.517+0000 7f0867b00740 10 client.14721 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.517+0000 7f0867b00740 20 client.14721 populate_metadata read hostname 'smithi033' 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.517+0000 7f0867b00740 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561173997b00 0x561173998f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.517+0000 7f0867b00740 4 client.14721 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.517+0000 7f0867b00740 1 -- 172.21.15.33:0/1320900741 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611738e5b10 con 0x561173997b00 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.517+0000 7f0857fff640 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561173997b00 0x561173998f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.517+0000 7f0857fff640 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561173997b00 0x561173998f70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58c6619d2d4d31b0 server_cookie=3412a1f7964530c1 in_seq=0 out_seq=0 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.517+0000 7f08467fc640 10 client.14721 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.518+0000 7f08467fc640 1 -- 172.21.15.33:0/1320900741 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5611738e5b10 con 0x561173997b00 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.518+0000 7f08467fc640 10 client.14721 handle_command_reply: tid=0 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.518+0000 7f0867b00740 1 client.14721 shutdown 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.518+0000 7f0867b00740 1 -- 172.21.15.33:0/1320900741 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397bde0 msgr2=0x561173986810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.518+0000 7f0867b00740 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397bde0 0x561173986810 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f085000bee0 tx=0x7f0850012820 comp rx=0 tx=0).stop 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.518+0000 7f0867b00740 1 -- 172.21.15.33:0/1320900741 shutdown_connections 2024-09-16T00:15:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.518+0000 7f0867b00740 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561173997b00 0x561173998f70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.518+0000 7f0867b00740 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561173986fc0 0x5611739840a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.518+0000 7f0867b00740 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397c700 0x561173983b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.518+0000 7f0867b00740 1 --2- 172.21.15.33:0/1320900741 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397bde0 0x561173986810 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.518+0000 7f0867b00740 1 -- 172.21.15.33:0/1320900741 >> 172.21.15.33:0/1320900741 conn(0x561173978370 msgr2=0x561173973020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:26.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.518+0000 7f08467fc640 0 client.14721 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:15:26.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.519+0000 7f0867b00740 1 -- 172.21.15.33:0/1320900741 shutdown_connections 2024-09-16T00:15:26.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.519+0000 7f0867b00740 1 -- 172.21.15.33:0/1320900741 wait complete. 2024-09-16T00:15:26.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.519+0000 7f0867b00740 20 client.14721 trim_cache size 0 max 16384 2024-09-16T00:15:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.537+0000 7f0867b00740 1 Processor -- start 2024-09-16T00:15:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.537+0000 7f0867b00740 1 -- start start 2024-09-16T00:15:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.537+0000 7f0867b00740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397bde0 0x561173997430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.537+0000 7f0867b00740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397c700 0x5611739988a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.537+0000 7f0867b00740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561173986fc0 0x5611739a1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.537+0000 7f0867b00740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5611739972b0 con 0x56117397c700 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.537+0000 7f0867b00740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561173997980 con 0x561173986fc0 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.537+0000 7f0867b00740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561173998050 con 0x56117397bde0 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.537+0000 7f0857fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397c700 0x5611739988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.537+0000 7f0857fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397c700 0x5611739988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36992/0 (socket says 172.21.15.33:36992) 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f0857fff640 1 -- 172.21.15.33:0/3129594939 learned_addr learned my addr 172.21.15.33:0/3129594939 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f085effd640 1 --2- 172.21.15.33:0/3129594939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397bde0 0x561173997430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f085f7fe640 1 --2- 172.21.15.33:0/3129594939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561173986fc0 0x5611739a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f0857fff640 1 -- 172.21.15.33:0/3129594939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397bde0 msgr2=0x561173997430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f0857fff640 1 --2- 172.21.15.33:0/3129594939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397bde0 0x561173997430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f0857fff640 1 -- 172.21.15.33:0/3129594939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561173986fc0 msgr2=0x5611739a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f0857fff640 1 --2- 172.21.15.33:0/3129594939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561173986fc0 0x5611739a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f0857fff640 1 -- 172.21.15.33:0/3129594939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611739820b0 con 0x56117397c700 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f085effd640 1 --2- 172.21.15.33:0/3129594939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397bde0 0x561173997430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f085f7fe640 1 --2- 172.21.15.33:0/3129594939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561173986fc0 0x5611739a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f0857fff640 1 --2- 172.21.15.33:0/3129594939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397c700 0x5611739988a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f08480b8a60 tx=0x7f08480b89d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994b3dfb507a05fc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f0854ff9640 1 -- 172.21.15.33:0/3129594939 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0848049040 con 0x56117397c700 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f0854ff9640 1 -- 172.21.15.33:0/3129594939 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08480dc030 con 0x56117397c700 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.538+0000 7f0854ff9640 1 -- 172.21.15.33:0/3129594939 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08480bc600 con 0x56117397c700 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.539+0000 7f0867b00740 1 -- 172.21.15.33:0/3129594939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397c700 msgr2=0x5611739988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.539+0000 7f0867b00740 1 --2- 172.21.15.33:0/3129594939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397c700 0x5611739988a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f08480b8a60 tx=0x7f08480b89d0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.539+0000 7f0867b00740 1 -- 172.21.15.33:0/3129594939 shutdown_connections 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.539+0000 7f0867b00740 1 --2- 172.21.15.33:0/3129594939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561173986fc0 0x5611739a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.539+0000 7f0867b00740 1 --2- 172.21.15.33:0/3129594939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56117397c700 0x5611739988a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.539+0000 7f0867b00740 1 --2- 172.21.15.33:0/3129594939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397bde0 0x561173997430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.539+0000 7f0867b00740 1 -- 172.21.15.33:0/3129594939 >> 172.21.15.33:0/3129594939 conn(0x561173978370 msgr2=0x561173982430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.539+0000 7f0867b00740 1 -- 172.21.15.33:0/3129594939 shutdown_connections 2024-09-16T00:15:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.539+0000 7f0867b00740 1 -- 172.21.15.33:0/3129594939 wait complete. 2024-09-16T00:15:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.539+0000 7f0867b00740 1 -- start start 2024-09-16T00:15:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.539+0000 7f0867b00740 1 Processor -- start 2024-09-16T00:15:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f0867b00740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397bde0 0x56117399c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f0867b00740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397c700 0x56117399c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f0867b00740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 0x561173a00200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f0867b00740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561173998df0 con 0x561173986fc0 2024-09-16T00:15:26.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f0867b00740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561173998720 con 0x56117397bde0 2024-09-16T00:15:26.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f0867b00740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5611739994c0 con 0x56117397c700 2024-09-16T00:15:26.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f0867b00740 10 client.0 _ll_register_callbacks cb 0x7f085000b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:15:26.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f085f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 0x561173a00200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f085effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397bde0 0x56117399c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f085f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 0x561173a00200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36998/0 (socket says 172.21.15.33:36998) 2024-09-16T00:15:26.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f0857fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397c700 0x56117399c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f085f7fe640 1 -- 172.21.15.33:0/3171681751 learned_addr learned my addr 172.21.15.33:0/3171681751 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:26.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f085f7fe640 1 -- 172.21.15.33:0/3171681751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397c700 msgr2=0x56117399c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f085f7fe640 1 --2- 172.21.15.33:0/3171681751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397c700 0x56117399c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f085f7fe640 1 -- 172.21.15.33:0/3171681751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397bde0 msgr2=0x56117399c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f085f7fe640 1 --2- 172.21.15.33:0/3171681751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397bde0 0x56117399c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f085f7fe640 1 -- 172.21.15.33:0/3171681751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561173a00740 con 0x561173986fc0 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f085f7fe640 1 --2- 172.21.15.33:0/3171681751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 0x561173a00200 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f08580092b0 tx=0x7f085801cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4983f318c58b2096 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.540+0000 7f08467fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f08467fc640 1 -- 172.21.15.33:0/3171681751 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0858035060 con 0x561173986fc0 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f0867b00740 1 -- 172.21.15.33:0/3171681751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08480dc030 con 0x561173986fc0 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f08467fc640 1 -- 172.21.15.33:0/3171681751 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f0858022040 con 0x561173986fc0 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f08467fc640 1 -- 172.21.15.33:0/3171681751 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f085803c050 con 0x561173986fc0 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f0867b00740 10 client.14727 fetch_fsmap learned FSMap version 17 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f0867b00740 1 -- 172.21.15.33:0/3171681751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x561173a01890 con 0x561173986fc0 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f08467fc640 1 -- 172.21.15.33:0/3171681751 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08480dc030 con 0x561173986fc0 2024-09-16T00:15:26.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f08467fc640 1 -- 172.21.15.33:0/3171681751 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f085800f040 con 0x561173986fc0 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f0867b00740 10 client.14727 fetch_fsmap finished waiting for FSMap version 17 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f0867b00740 10 client.14727 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f0867b00740 20 client.14727 populate_metadata read hostname 'smithi033' 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f0867b00740 1 --2- 172.21.15.33:0/3171681751 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561173a06240 0x561173a04f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f0867b00740 4 client.14727 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.541+0000 7f0867b00740 1 -- 172.21.15.33:0/3171681751 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "status"}) -- 0x5611738620e0 con 0x561173a06240 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f085effd640 1 --2- 172.21.15.33:0/3171681751 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561173a06240 0x561173a04f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f085effd640 1 --2- 172.21.15.33:0/3171681751 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561173a06240 0x561173a04f50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7eb4dfe72f08080 server_cookie=c89a740d860bd2a9 in_seq=0 out_seq=0 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f08467fc640 10 client.14727 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f08467fc640 1 -- 172.21.15.33:0/3171681751 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x5611738620e0 con 0x561173a06240 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f08467fc640 10 client.14727 handle_command_reply: tid=0 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f0867b00740 1 client.14727 shutdown 2024-09-16T00:15:26.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f0867b00740 1 -- 172.21.15.33:0/3171681751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 msgr2=0x561173a00200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:26.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f0867b00740 1 --2- 172.21.15.33:0/3171681751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 0x561173a00200 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f08580092b0 tx=0x7f085801cd90 comp rx=0 tx=0).stop 2024-09-16T00:15:26.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f0867b00740 1 -- 172.21.15.33:0/3171681751 shutdown_connections 2024-09-16T00:15:26.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f0867b00740 1 --2- 172.21.15.33:0/3171681751 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561173a06240 0x561173a04f50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f0867b00740 1 --2- 172.21.15.33:0/3171681751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561173986fc0 0x561173a00200 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f0867b00740 1 --2- 172.21.15.33:0/3171681751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56117397c700 0x56117399c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f0867b00740 1 --2- 172.21.15.33:0/3171681751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56117397bde0 0x56117399c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f08467fc640 0 client.14727 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:15:26.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.542+0000 7f0867b00740 1 -- 172.21.15.33:0/3171681751 >> 172.21.15.33:0/3171681751 conn(0x561173978370 msgr2=0x561173978750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:26.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0867b00740 1 -- 172.21.15.33:0/3171681751 shutdown_connections 2024-09-16T00:15:26.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0867b00740 1 -- 172.21.15.33:0/3171681751 wait complete. 2024-09-16T00:15:26.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0867b00740 20 client.14727 trim_cache size 0 max 16384 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 0, 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24494, 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 19.01374611, 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 17, 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 86, 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 86, 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 30.091514002 2024-09-16T00:15:26.549 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:15:26.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0865727640 1 -- 172.21.15.33:0/2302243189 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f084c0a32c0 msgr2=0x7f084c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:26.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0865727640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f084c0a32c0 0x7f084c0a5780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0858020000 tx=0x7f085801d000 comp rx=0 tx=0).stop 2024-09-16T00:15:26.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0865727640 1 -- 172.21.15.33:0/2302243189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0860103660 msgr2=0x7f08601989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:26.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0865727640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0860103660 0x7f08601989b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0848006010 tx=0x7f08480042f0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0865727640 1 -- 172.21.15.33:0/2302243189 shutdown_connections 2024-09-16T00:15:26.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0865727640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f084c0a32c0 0x7f084c0a5780 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0865727640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086010c170 0x7f086019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0865727640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0860103660 0x7f08601989b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0865727640 1 --2- 172.21.15.33:0/2302243189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0860102d40 0x7f0860198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:26.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0865727640 1 -- 172.21.15.33:0/2302243189 >> 172.21.15.33:0/2302243189 conn(0x7f08600fc7a0 msgr2=0x7f08600fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:26.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0865727640 1 -- 172.21.15.33:0/2302243189 shutdown_connections 2024-09-16T00:15:26.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:26.543+0000 7f0865727640 1 -- 172.21.15.33:0/2302243189 wait complete. 2024-09-16T00:15:27.275 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T00:15:27.275 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-16T00:15:27.276 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-16T00:15:27.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:27 smithi083 ceph-mon[35242]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:27.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:27 smithi060 ceph-mon[35185]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:27.531 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:27.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:27 smithi033 ceph-mon[31243]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:28.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.997+0000 7fab24c08640 1 -- 172.21.15.33:0/1095873178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab20102790 msgr2=0x7fab201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:28.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.997+0000 7fab24c08640 1 --2- 172.21.15.33:0/1095873178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab20102790 0x7fab201067e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fab14009900 tx=0x7fab1402a410 comp rx=0 tx=0).stop 2024-09-16T00:15:28.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.997+0000 7fab24c08640 1 -- 172.21.15.33:0/1095873178 shutdown_connections 2024-09-16T00:15:28.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.997+0000 7fab24c08640 1 --2- 172.21.15.33:0/1095873178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab20106f10 0x7fab2010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:28.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.997+0000 7fab24c08640 1 --2- 172.21.15.33:0/1095873178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab20102790 0x7fab201067e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:28.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.997+0000 7fab24c08640 1 --2- 172.21.15.33:0/1095873178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab20101de0 0x7fab201021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:28.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.997+0000 7fab24c08640 1 -- 172.21.15.33:0/1095873178 >> 172.21.15.33:0/1095873178 conn(0x7fab200fd8c0 msgr2=0x7fab200ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:28.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.997+0000 7fab24c08640 1 -- 172.21.15.33:0/1095873178 shutdown_connections 2024-09-16T00:15:28.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.997+0000 7fab24c08640 1 -- 172.21.15.33:0/1095873178 wait complete. 2024-09-16T00:15:28.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.998+0000 7fab24c08640 1 Processor -- start 2024-09-16T00:15:28.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.998+0000 7fab24c08640 1 -- start start 2024-09-16T00:15:28.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.998+0000 7fab24c08640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab20101de0 0x7fab201983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:28.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.998+0000 7fab24c08640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab20102790 0x7fab201988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:28.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.998+0000 7fab24c08640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab20106f10 0x7fab2019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:28.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.998+0000 7fab24c08640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab20111f70 con 0x7fab20106f10 2024-09-16T00:15:28.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.998+0000 7fab24c08640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fab20111df0 con 0x7fab20101de0 2024-09-16T00:15:28.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.998+0000 7fab24c08640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fab201120f0 con 0x7fab20102790 2024-09-16T00:15:28.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab1ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab20106f10 0x7fab2019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:28.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab1ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab20106f10 0x7fab2019cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37018/0 (socket says 172.21.15.33:37018) 2024-09-16T00:15:28.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab1ed9d640 1 -- 172.21.15.33:0/2510645669 learned_addr learned my addr 172.21.15.33:0/2510645669 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:28.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab1dd9b640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab20102790 0x7fab201988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:28.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab1e59c640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab20101de0 0x7fab201983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:28.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab1ed9d640 1 -- 172.21.15.33:0/2510645669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab20102790 msgr2=0x7fab201988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab1ed9d640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab20102790 0x7fab201988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab1ed9d640 1 -- 172.21.15.33:0/2510645669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab20101de0 msgr2=0x7fab201983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab1ed9d640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab20101de0 0x7fab201983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab1ed9d640 1 -- 172.21.15.33:0/2510645669 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2019d360 con 0x7fab20106f10 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab1ed9d640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab20106f10 0x7fab2019cc80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fab1800cce0 tx=0x7fab1800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab0f7fe640 1 -- 172.21.15.33:0/2510645669 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1800f450 con 0x7fab20106f10 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab0f7fe640 1 -- 172.21.15.33:0/2510645669 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab1800fad0 con 0x7fab20106f10 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab24c08640 1 -- 172.21.15.33:0/2510645669 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab2019d650 con 0x7fab20106f10 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab0f7fe640 1 -- 172.21.15.33:0/2510645669 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab18018c10 con 0x7fab20106f10 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:27.999+0000 7fab24c08640 1 -- 172.21.15.33:0/2510645669 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab201a4f30 con 0x7fab20106f10 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.000+0000 7fab24c08640 1 -- 172.21.15.33:0/2510645669 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab201021c0 con 0x7fab20106f10 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.001+0000 7fab0f7fe640 1 -- 172.21.15.33:0/2510645669 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fab1800f5f0 con 0x7fab20106f10 2024-09-16T00:15:28.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.002+0000 7fab0f7fe640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab080a3240 0x7fab080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:28.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.002+0000 7fab1e59c640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab080a3240 0x7fab080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:28.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.002+0000 7fab0f7fe640 1 -- 172.21.15.33:0/2510645669 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fab180d01d0 con 0x7fab20106f10 2024-09-16T00:15:28.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.003+0000 7fab1e59c640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab080a3240 0x7fab080a5700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fab10005fd0 tx=0x7fab10005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:28.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.006+0000 7fab0f7fe640 1 -- 172.21.15.33:0/2510645669 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fab18014030 con 0x7fab20106f10 2024-09-16T00:15:28.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.222+0000 7fab24c08640 1 -- 172.21.15.33:0/2510645669 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fab20109bc0 con 0x7fab20106f10 2024-09-16T00:15:28.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.222+0000 7fab0f7fe640 1 -- 172.21.15.33:0/2510645669 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fab1809ccb0 con 0x7fab20106f10 2024-09-16T00:15:28.223 INFO:teuthology.orchestra.run.smithi033.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-16T00:15:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.225+0000 7fab24c08640 1 -- 172.21.15.33:0/2510645669 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab080a3240 msgr2=0x7fab080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.225+0000 7fab24c08640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab080a3240 0x7fab080a5700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fab10005fd0 tx=0x7fab10005850 comp rx=0 tx=0).stop 2024-09-16T00:15:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.225+0000 7fab24c08640 1 -- 172.21.15.33:0/2510645669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab20106f10 msgr2=0x7fab2019cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.225+0000 7fab24c08640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab20106f10 0x7fab2019cc80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fab1800cce0 tx=0x7fab1800a620 comp rx=0 tx=0).stop 2024-09-16T00:15:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.225+0000 7fab24c08640 1 -- 172.21.15.33:0/2510645669 shutdown_connections 2024-09-16T00:15:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.225+0000 7fab24c08640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab080a3240 0x7fab080a5700 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.225+0000 7fab24c08640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab20106f10 0x7fab2019cc80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.225+0000 7fab24c08640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab20102790 0x7fab201988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.225+0000 7fab24c08640 1 --2- 172.21.15.33:0/2510645669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab20101de0 0x7fab201983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.225+0000 7fab24c08640 1 -- 172.21.15.33:0/2510645669 >> 172.21.15.33:0/2510645669 conn(0x7fab200fd8c0 msgr2=0x7fab200fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.225+0000 7fab24c08640 1 -- 172.21.15.33:0/2510645669 shutdown_connections 2024-09-16T00:15:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:28.225+0000 7fab24c08640 1 -- 172.21.15.33:0/2510645669 wait complete. 2024-09-16T00:15:28.958 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T00:15:28.959 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T00:15:28.959 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi033.front.sepia.ceph.com 2024-09-16T00:15:28.959 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-09-16T00:15:28.959 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-16T00:15:28.959 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T00:15:28.959 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T00:15:28.959 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T00:15:28.959 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-09-16T00:15:28.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:15:28.959 DEBUG:teuthology.orchestra.run.smithi033:> ip addr 2024-09-16T00:15:28.976 INFO:teuthology.orchestra.run.smithi033.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T00:15:28.976 INFO:teuthology.orchestra.run.smithi033.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T00:15:28.976 INFO:teuthology.orchestra.run.smithi033.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T00:15:28.976 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout: inet6 ::1/128 scope host 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether 0c:c4:7a:6b:fd:14 brd ff:ff:ff:ff:ff:ff 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp7s0f0 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether 0c:c4:7a:6b:fd:15 brd ff:ff:ff:ff:ff:ff 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp7s0f1 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether ac:1f:6b:a5:d2:94 brd ff:ff:ff:ff:ff:ff 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp1s0f0 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout: inet 172.21.15.33/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft 42288sec preferred_lft 42288sec 2024-09-16T00:15:28.977 INFO:teuthology.orchestra.run.smithi033.stdout: inet6 fe80::ae1f:6bff:fea5:d294/64 scope link 2024-09-16T00:15:28.978 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-09-16T00:15:28.978 INFO:teuthology.orchestra.run.smithi033.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T00:15:28.978 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether ac:1f:6b:a5:d2:95 brd ff:ff:ff:ff:ff:ff 2024-09-16T00:15:28.978 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp1s0f1 2024-09-16T00:15:28.978 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-16T00:15:28.979 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T00:15:28.979 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-09-16T00:15:28.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link add name ceph-brx type bridge 2024-09-16T00:15:28.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip addr flush dev ceph-brx 2024-09-16T00:15:28.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set ceph-brx up 2024-09-16T00:15:28.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-16T00:15:28.979 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-09-16T00:15:29.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T00:15:29.183 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:29 smithi033 ceph-mon[31243]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:29.183 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:29 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2510645669' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-16T00:15:29.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:15:29.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:15:29.194 DEBUG:teuthology.orchestra.run.smithi033:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T00:15:29.220 INFO:teuthology.orchestra.run.smithi033.stdout:1 2024-09-16T00:15:29.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:15:29.222 DEBUG:teuthology.orchestra.run.smithi033:> route 2024-09-16T00:15:29.292 INFO:teuthology.orchestra.run.smithi033.stdout:Kernel IP routing table 2024-09-16T00:15:29.292 INFO:teuthology.orchestra.run.smithi033.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T00:15:29.292 INFO:teuthology.orchestra.run.smithi033.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-16T00:15:29.292 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-16T00:15:29.293 INFO:teuthology.orchestra.run.smithi033.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T00:15:29.293 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T00:15:29.294 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-09-16T00:15:29.294 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-16T00:15:29.294 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-16T00:15:29.294 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-16T00:15:29.294 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-09-16T00:15:29.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T00:15:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:29 smithi083 ceph-mon[35242]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:29.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:29 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2510645669' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-16T00:15:29.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:29 smithi060 ceph-mon[35185]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:29.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:29 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2510645669' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-16T00:15:29.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:15:29.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:15:29.620 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list 2024-09-16T00:15:29.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:15:29.637 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list-id 2024-09-16T00:15:29.693 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T00:15:29.693 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-09-16T00:15:29.693 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-16T00:15:29.693 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-09-16T00:15:29.693 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-09-16T00:15:29.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T00:15:29.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:15:29.813 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-09-16T00:15:29.814 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T00:15:29.814 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-09-16T00:15:29.814 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-09-16T00:15:29.814 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-16T00:15:29.814 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-09-16T00:15:29.814 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-09-16T00:15:29.814 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-09-16T00:15:29.814 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-09-16T00:15:29.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T00:15:29.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:15:29.941 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T00:15:29.941 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-09-16T00:15:29.941 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set brx.0 up 2024-09-16T00:15:29.941 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set dev brx.0 master ceph-brx 2024-09-16T00:15:29.941 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-09-16T00:15:29.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T00:15:30.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:15:30.032 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-09-16T00:15:30.032 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T00:15:30.032 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-09-16T00:15:30.048 INFO:teuthology.orchestra.run.smithi033.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-09-16T00:15:30.048 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T00:15:30.048 DEBUG:teuthology.orchestra.run.smithi033:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-09-16T00:15:30.104 DEBUG:teuthology.orchestra.run.smithi033:> sudo modprobe fuse 2024-09-16T00:15:30.172 DEBUG:teuthology.orchestra.run.smithi033:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T00:15:30.229 INFO:teuthology.orchestra.run.smithi033.stdout:/proc 2024-09-16T00:15:30.230 INFO:teuthology.orchestra.run.smithi033.stdout:/sys 2024-09-16T00:15:30.230 INFO:teuthology.orchestra.run.smithi033.stdout:/dev 2024-09-16T00:15:30.230 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/security 2024-09-16T00:15:30.230 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/shm 2024-09-16T00:15:30.230 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/pts 2024-09-16T00:15:30.230 INFO:teuthology.orchestra.run.smithi033.stdout:/run 2024-09-16T00:15:30.230 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/cgroup 2024-09-16T00:15:30.230 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/pstore 2024-09-16T00:15:30.230 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/bpf 2024-09-16T00:15:30.230 INFO:teuthology.orchestra.run.smithi033.stdout:/ 2024-09-16T00:15:30.230 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/selinux 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/mqueue 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/hugepages 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/debug 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/tracing 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/fuse/connections 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/config 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-sysctl.service 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/nfs/rpc_pipefs 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/run/user/1000 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/ceph 2024-09-16T00:15:30.231 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/0ff1707cf5312bf3bd834ed867e5a141dc6e3a5bee56305e7c5b9bb2ca1b2dcb/merged 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/242317df37cba39d830e2d3fac7d9bc1178946afdd8dda1465daf405a3d33af2/merged 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/run/user/0 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/f1029826be4036c316f1aaa5a6fea5548e810e24cfdde2ad4df3b04647bf1e24/merged 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a915c506e696374a17dd1863d9cd1f93e71a4742ac0e0cf55c81a5d265c521fd/merged 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/df224b79010056ba3f2c4168747ecb733d8b0429d5b5d7763cc8679ed8ead3a0/merged 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/c872ca49468e6156c0067ad9b8cd315b6f9f448674ab7c20e78cd72a38c27581/merged 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/208d9fea0cb8276bbff38880c6192c4d64b5ce1ec461892c69b42c0a37fdfbb2/merged 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/08a98eae55ba64f67eb0aae76e50f67ed45ff2f86df680cfd004f0d9a8178285/merged 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/ca95b91caf965b1d43d8709c6fe4a2ed877885fd71340287d13abddf68217540/merged 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/f5b9d68e4c3f65263b0e8be38dcf19669d39b3c6ad0f43e023a397eaba4dca34/merged 2024-09-16T00:15:30.232 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns 2024-09-16T00:15:30.233 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-16T00:15:30.233 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-16T00:15:30.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:15:30.233 DEBUG:teuthology.orchestra.run.smithi033:> ls /sys/fs/fuse/connections 2024-09-16T00:15:30.286 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T00:15:30.287 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-09-16T00:15:30.330 DEBUG:teuthology.orchestra.run.smithi033:> sudo modprobe fuse 2024-09-16T00:15:30.356 DEBUG:teuthology.orchestra.run.smithi033:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T00:15:30.414 INFO:teuthology.orchestra.run.smithi033.stdout:/proc 2024-09-16T00:15:30.414 INFO:teuthology.orchestra.run.smithi033.stdout:/sys 2024-09-16T00:15:30.414 INFO:teuthology.orchestra.run.smithi033.stdout:/dev 2024-09-16T00:15:30.414 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/security 2024-09-16T00:15:30.415 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/shm 2024-09-16T00:15:30.415 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/pts 2024-09-16T00:15:30.415 INFO:teuthology.orchestra.run.smithi033.stdout:/run 2024-09-16T00:15:30.415 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/cgroup 2024-09-16T00:15:30.415 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/pstore 2024-09-16T00:15:30.415 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/bpf 2024-09-16T00:15:30.416 INFO:teuthology.orchestra.run.smithi033.stdout:/ 2024-09-16T00:15:30.416 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/selinux 2024-09-16T00:15:30.416 INFO:teuthology.orchestra.run.smithi033.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T00:15:30.416 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/mqueue 2024-09-16T00:15:30.416 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/hugepages 2024-09-16T00:15:30.416 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/debug 2024-09-16T00:15:30.416 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/tracing 2024-09-16T00:15:30.416 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/fuse/connections 2024-09-16T00:15:30.416 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/config 2024-09-16T00:15:30.416 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-sysctl.service 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/nfs/rpc_pipefs 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/run/user/1000 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/ceph 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/0ff1707cf5312bf3bd834ed867e5a141dc6e3a5bee56305e7c5b9bb2ca1b2dcb/merged 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/242317df37cba39d830e2d3fac7d9bc1178946afdd8dda1465daf405a3d33af2/merged 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/run/user/0 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/f1029826be4036c316f1aaa5a6fea5548e810e24cfdde2ad4df3b04647bf1e24/merged 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a915c506e696374a17dd1863d9cd1f93e71a4742ac0e0cf55c81a5d265c521fd/merged 2024-09-16T00:15:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/df224b79010056ba3f2c4168747ecb733d8b0429d5b5d7763cc8679ed8ead3a0/merged 2024-09-16T00:15:30.418 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/c872ca49468e6156c0067ad9b8cd315b6f9f448674ab7c20e78cd72a38c27581/merged 2024-09-16T00:15:30.418 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/208d9fea0cb8276bbff38880c6192c4d64b5ce1ec461892c69b42c0a37fdfbb2/merged 2024-09-16T00:15:30.418 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/08a98eae55ba64f67eb0aae76e50f67ed45ff2f86df680cfd004f0d9a8178285/merged 2024-09-16T00:15:30.418 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/ca95b91caf965b1d43d8709c6fe4a2ed877885fd71340287d13abddf68217540/merged 2024-09-16T00:15:30.418 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/f5b9d68e4c3f65263b0e8be38dcf19669d39b3c6ad0f43e023a397eaba4dca34/merged 2024-09-16T00:15:30.418 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns 2024-09-16T00:15:30.418 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-16T00:15:30.418 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-16T00:15:30.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:15:30.419 DEBUG:teuthology.orchestra.run.smithi033:> ls /sys/fs/fuse/connections 2024-09-16T00:15:30.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi033.stderr:2024-09-16T00:15:30.451+0000 7f98405c4480 -1 init, newargv = 0x562f342bbb40 newargc=15 2024-09-16T00:15:30.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi033.stderr:ceph-fuse[72777]: starting ceph client 2024-09-16T00:15:30.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi033.stderr:ceph-fuse[72777]: starting fuse 2024-09-16T00:15:30.470 INFO:teuthology.orchestra.run.smithi033.stdout:120 2024-09-16T00:15:30.470 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [120] 2024-09-16T00:15:30.471 DEBUG:teuthology.orchestra.run.smithi033:> sudo stdin-killer -- python3 -c ' 2024-09-16T00:15:30.471 DEBUG:teuthology.orchestra.run.smithi033:> import glob 2024-09-16T00:15:30.471 DEBUG:teuthology.orchestra.run.smithi033:> import re 2024-09-16T00:15:30.471 DEBUG:teuthology.orchestra.run.smithi033:> import os 2024-09-16T00:15:30.471 DEBUG:teuthology.orchestra.run.smithi033:> import subprocess 2024-09-16T00:15:30.471 DEBUG:teuthology.orchestra.run.smithi033:> 2024-09-16T00:15:30.471 DEBUG:teuthology.orchestra.run.smithi033:> def _find_admin_socket(client_name): 2024-09-16T00:15:30.471 DEBUG:teuthology.orchestra.run.smithi033:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-09-16T00:15:30.471 DEBUG:teuthology.orchestra.run.smithi033:> files = glob.glob(asok_path) 2024-09-16T00:15:30.471 DEBUG:teuthology.orchestra.run.smithi033:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-09-16T00:15:30.471 DEBUG:teuthology.orchestra.run.smithi033:> 2024-09-16T00:15:30.471 DEBUG:teuthology.orchestra.run.smithi033:> # Given a non-glob path, it better be there 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> if "*" not in asok_path: 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> assert(len(files) == 1) 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> return files[0] 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> for f in files: 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> contents = proc_f.read() 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> if mountpoint in contents: 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> return f 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T00:15:30.472 DEBUG:teuthology.orchestra.run.smithi033:> 2024-09-16T00:15:30.473 DEBUG:teuthology.orchestra.run.smithi033:> print(_find_admin_socket("client.admin")) 2024-09-16T00:15:30.473 DEBUG:teuthology.orchestra.run.smithi033:> ' 2024-09-16T00:15:30.594 INFO:teuthology.orchestra.run.smithi033.stdout:/var/run/ceph/ceph-client.admin.72777.asok 2024-09-16T00:15:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:15:30.604 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.72777.asok 2024-09-16T00:15:30.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:15:30.605 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.72777.asok status 2024-09-16T00:15:30.743 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:15:30.743 INFO:teuthology.orchestra.run.smithi033.stdout: "metadata": { 2024-09-16T00:15:30.743 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_sha1": "8921c8ef947807b717db041d85f0761b7914d5fc", 2024-09-16T00:15:30.743 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T00:15:30.744 INFO:teuthology.orchestra.run.smithi033.stdout: "entity_id": "admin", 2024-09-16T00:15:30.744 INFO:teuthology.orchestra.run.smithi033.stdout: "hostname": "smithi033", 2024-09-16T00:15:30.744 INFO:teuthology.orchestra.run.smithi033.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-09-16T00:15:30.744 INFO:teuthology.orchestra.run.smithi033.stdout: "pid": "72777", 2024-09-16T00:15:30.744 INFO:teuthology.orchestra.run.smithi033.stdout: "root": "/" 2024-09-16T00:15:30.744 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:15:30.744 INFO:teuthology.orchestra.run.smithi033.stdout: "dentry_count": 0, 2024-09-16T00:15:30.744 INFO:teuthology.orchestra.run.smithi033.stdout: "dentry_pinned_count": 0, 2024-09-16T00:15:30.744 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 14739, 2024-09-16T00:15:30.744 INFO:teuthology.orchestra.run.smithi033.stdout: "inst": { 2024-09-16T00:15:30.744 INFO:teuthology.orchestra.run.smithi033.stdout: "name": { 2024-09-16T00:15:30.744 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "client", 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: "num": 14739 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": { 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "192.168.0.1:0", 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 527339367 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": { 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "192.168.0.1:0", 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 527339367 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:15:30.745 INFO:teuthology.orchestra.run.smithi033.stdout: "inst_str": "client.14739 192.168.0.1:0/527339367", 2024-09-16T00:15:30.746 INFO:teuthology.orchestra.run.smithi033.stdout: "addr_str": "192.168.0.1:0/527339367", 2024-09-16T00:15:30.746 INFO:teuthology.orchestra.run.smithi033.stdout: "inode_count": 1, 2024-09-16T00:15:30.746 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_epoch": 17, 2024-09-16T00:15:30.746 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_epoch": 86, 2024-09-16T00:15:30.746 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_epoch_barrier": 0, 2024-09-16T00:15:30.746 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklisted": false, 2024-09-16T00:15:30.746 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs" 2024-09-16T00:15:30.746 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:15:30.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:15:30.752 DEBUG:teuthology.orchestra.run.smithi033:> stat --file-system '--printf=%T 2024-09-16T00:15:30.752 DEBUG:teuthology.orchestra.run.smithi033:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-09-16T00:15:30.768 INFO:teuthology.orchestra.run.smithi033.stdout:fuseblk 2024-09-16T00:15:30.769 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-09-16T00:15:30.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:15:30.769 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-09-16T00:15:30.836 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-09-16T00:15:30.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T00:15:30.943 INFO:teuthology.orchestra.run.smithi033.stdout:# file: . 2024-09-16T00:15:30.944 INFO:teuthology.orchestra.run.smithi033.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-09-16T00:15:30.944 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:15:30.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:15:30.950 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:15:30.950 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T00:15:30.970 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-16T00:15:31.268 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:31.390 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:31 smithi033 ceph-mon[31243]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:31.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:31 smithi083 ceph-mon[35242]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:31.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:31 smithi060 ceph-mon[35185]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:31.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.778+0000 7fe1d8d39640 1 -- 172.21.15.33:0/3998641293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1d4102790 msgr2=0x7fe1d41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.778+0000 7fe1d8d39640 1 --2- 172.21.15.33:0/3998641293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1d4102790 0x7fe1d41067e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009900 tx=0x7fe1bc02a410 comp rx=0 tx=0).stop 2024-09-16T00:15:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.778+0000 7fe1d8d39640 1 -- 172.21.15.33:0/3998641293 shutdown_connections 2024-09-16T00:15:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.778+0000 7fe1d8d39640 1 --2- 172.21.15.33:0/3998641293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d4106f10 0x7fe1d410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.778+0000 7fe1d8d39640 1 --2- 172.21.15.33:0/3998641293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1d4102790 0x7fe1d41067e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.778+0000 7fe1d8d39640 1 --2- 172.21.15.33:0/3998641293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1d4101de0 0x7fe1d41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.778+0000 7fe1d8d39640 1 -- 172.21.15.33:0/3998641293 >> 172.21.15.33:0/3998641293 conn(0x7fe1d40fd8c0 msgr2=0x7fe1d40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.778+0000 7fe1d8d39640 1 -- 172.21.15.33:0/3998641293 shutdown_connections 2024-09-16T00:15:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.778+0000 7fe1d8d39640 1 -- 172.21.15.33:0/3998641293 wait complete. 2024-09-16T00:15:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.779+0000 7fe1d8d39640 1 Processor -- start 2024-09-16T00:15:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.779+0000 7fe1d8d39640 1 -- start start 2024-09-16T00:15:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.779+0000 7fe1d8d39640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1d4101de0 0x7fe1d4198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.779+0000 7fe1d8d39640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1d4102790 0x7fe1d4198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.779+0000 7fe1d8d39640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d4106f10 0x7fe1d419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.779+0000 7fe1d8d39640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d4111f30 con 0x7fe1d4101de0 2024-09-16T00:15:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.779+0000 7fe1d8d39640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d4111db0 con 0x7fe1d4102790 2024-09-16T00:15:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.779+0000 7fe1d8d39640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d41120b0 con 0x7fe1d4106f10 2024-09-16T00:15:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1ca59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1d4101de0 0x7fe1d4198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1ca59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1d4101de0 0x7fe1d4198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38306/0 (socket says 172.21.15.33:38306) 2024-09-16T00:15:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1ca59c640 1 -- 172.21.15.33:0/441414038 learned_addr learned my addr 172.21.15.33:0/441414038 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1d259c640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1d4102790 0x7fe1d4198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1d2d9d640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d4106f10 0x7fe1d419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1ca59c640 1 -- 172.21.15.33:0/441414038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d4106f10 msgr2=0x7fe1d419cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1ca59c640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d4106f10 0x7fe1d419cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1ca59c640 1 -- 172.21.15.33:0/441414038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1d4102790 msgr2=0x7fe1d4198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1ca59c640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1d4102790 0x7fe1d4198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1ca59c640 1 -- 172.21.15.33:0/441414038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d419d400 con 0x7fe1d4101de0 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1d2d9d640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d4106f10 0x7fe1d419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1ca59c640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1d4101de0 0x7fe1d4198450 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe1c400e9f0 tx=0x7fe1c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1cbfff640 1 -- 172.21.15.33:0/441414038 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c400cd80 con 0x7fe1d4101de0 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1d8d39640 1 -- 172.21.15.33:0/441414038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d419d6f0 con 0x7fe1d4101de0 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1cbfff640 1 -- 172.21.15.33:0/441414038 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1c4010910 con 0x7fe1d4101de0 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1cbfff640 1 -- 172.21.15.33:0/441414038 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c40188f0 con 0x7fe1d4101de0 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.780+0000 7fe1d8d39640 1 -- 172.21.15.33:0/441414038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d41a4fd0 con 0x7fe1d4101de0 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.781+0000 7fe1d8d39640 1 -- 172.21.15.33:0/441414038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1d41021c0 con 0x7fe1d4101de0 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.783+0000 7fe1cbfff640 1 -- 172.21.15.33:0/441414038 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe1c4010ac0 con 0x7fe1d4101de0 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.784+0000 7fe1cbfff640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1c00a3240 0x7fe1c00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.784+0000 7fe1cbfff640 1 -- 172.21.15.33:0/441414038 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fe1c40d03a0 con 0x7fe1d4101de0 2024-09-16T00:15:31.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.784+0000 7fe1d259c640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1c00a3240 0x7fe1c00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:31.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.785+0000 7fe1d259c640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1c00a3240 0x7fe1c00a5700 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009230 tx=0x7fe1bc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:31.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.787+0000 7fe1cbfff640 1 -- 172.21.15.33:0/441414038 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe1c409ce20 con 0x7fe1d4101de0 2024-09-16T00:15:31.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.976+0000 7fe1d8d39640 1 -- 172.21.15.33:0/441414038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe1d41997b0 con 0x7fe1d4101de0 2024-09-16T00:15:31.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.976+0000 7fe1cbfff640 1 -- 172.21.15.33:0/441414038 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v86) ==== 81+0+588 (secure 0 0 0) 0x7fe1c409c720 con 0x7fe1d4101de0 2024-09-16T00:15:31.977 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/3553033370 2024-09-17T00:08:07.548547+0000 2024-09-16T00:15:31.977 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/122116489 2024-09-17T00:08:07.548547+0000 2024-09-16T00:15:31.977 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/1410359644 2024-09-17T00:08:07.548547+0000 2024-09-16T00:15:31.977 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2692588880 2024-09-17T00:08:07.548547+0000 2024-09-16T00:15:31.978 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/122116489 2024-09-17T00:08:07.548547+0000 2024-09-16T00:15:31.978 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/961233150 2024-09-17T00:07:43.600446+0000 2024-09-16T00:15:31.978 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/1160344784 2024-09-17T00:07:43.600446+0000 2024-09-16T00:15:31.978 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/961233150 2024-09-17T00:07:43.600446+0000 2024-09-16T00:15:31.978 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2039144880 2024-09-17T00:07:43.600446+0000 2024-09-16T00:15:31.978 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/3586056783 2024-09-17T00:07:43.600446+0000 2024-09-16T00:15:31.978 INFO:teuthology.orchestra.run.smithi033.stderr:listed 10 entries 2024-09-16T00:15:31.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.979+0000 7fe1d8d39640 1 -- 172.21.15.33:0/441414038 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1c00a3240 msgr2=0x7fe1c00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:31.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.979+0000 7fe1d8d39640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1c00a3240 0x7fe1c00a5700 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009230 tx=0x7fe1bc0058d0 comp rx=0 tx=0).stop 2024-09-16T00:15:31.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.979+0000 7fe1d8d39640 1 -- 172.21.15.33:0/441414038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1d4101de0 msgr2=0x7fe1d4198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:31.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.979+0000 7fe1d8d39640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1d4101de0 0x7fe1d4198450 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe1c400e9f0 tx=0x7fe1c400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:15:31.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.979+0000 7fe1d8d39640 1 -- 172.21.15.33:0/441414038 shutdown_connections 2024-09-16T00:15:31.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.979+0000 7fe1d8d39640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1c00a3240 0x7fe1c00a5700 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:31.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.979+0000 7fe1d8d39640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d4106f10 0x7fe1d419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:31.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.979+0000 7fe1d8d39640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1d4102790 0x7fe1d4198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:31.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.979+0000 7fe1d8d39640 1 --2- 172.21.15.33:0/441414038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1d4101de0 0x7fe1d4198450 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:31.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.979+0000 7fe1d8d39640 1 -- 172.21.15.33:0/441414038 >> 172.21.15.33:0/441414038 conn(0x7fe1d40fd8c0 msgr2=0x7fe1d40fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:31.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.979+0000 7fe1d8d39640 1 -- 172.21.15.33:0/441414038 shutdown_connections 2024-09-16T00:15:31.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:31.979+0000 7fe1d8d39640 1 -- 172.21.15.33:0/441414038 wait complete. 2024-09-16T00:15:32.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:32 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/441414038' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T00:15:32.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:32 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/441414038' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T00:15:32.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:32 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/441414038' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T00:15:32.820 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:15:32.820 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T00:15:32.841 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-16T00:15:33.145 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:33.406 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:33 smithi033 ceph-mon[31243]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:33.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:33 smithi083 ceph-mon[35242]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:33.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:33 smithi060 ceph-mon[35185]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:15:33.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.664+0000 7fe208e12640 1 -- 172.21.15.33:0/2279186023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe204100e20 msgr2=0x7fe204101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:33.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.664+0000 7fe208e12640 1 --2- 172.21.15.33:0/2279186023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe204100e20 0x7fe204101200 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009900 tx=0x7fe1fc02a480 comp rx=0 tx=0).stop 2024-09-16T00:15:33.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.664+0000 7fe208e12640 1 -- 172.21.15.33:0/2279186023 shutdown_connections 2024-09-16T00:15:33.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.664+0000 7fe208e12640 1 --2- 172.21.15.33:0/2279186023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204102ef0 0x7fe20410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:33.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.664+0000 7fe208e12640 1 --2- 172.21.15.33:0/2279186023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe204102570 0x7fe2041029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:33.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.664+0000 7fe208e12640 1 --2- 172.21.15.33:0/2279186023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe204100e20 0x7fe204101200 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:33.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.664+0000 7fe208e12640 1 -- 172.21.15.33:0/2279186023 >> 172.21.15.33:0/2279186023 conn(0x7fe2040fc7d0 msgr2=0x7fe2040febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:33.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.664+0000 7fe208e12640 1 -- 172.21.15.33:0/2279186023 shutdown_connections 2024-09-16T00:15:33.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.664+0000 7fe208e12640 1 -- 172.21.15.33:0/2279186023 wait complete. 2024-09-16T00:15:33.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.665+0000 7fe208e12640 1 Processor -- start 2024-09-16T00:15:33.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.665+0000 7fe208e12640 1 -- start start 2024-09-16T00:15:33.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.665+0000 7fe208e12640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204100e20 0x7fe204103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:33.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.665+0000 7fe208e12640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe204102570 0x7fe204103c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:33.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.665+0000 7fe208e12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe204102ef0 0x7fe2041041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:33.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.665+0000 7fe208e12640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe204110f80 con 0x7fe204102ef0 2024-09-16T00:15:33.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.665+0000 7fe208e12640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe204110e00 con 0x7fe204100e20 2024-09-16T00:15:33.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.665+0000 7fe208e12640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe204111100 con 0x7fe204102570 2024-09-16T00:15:33.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.665+0000 7fe202d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe204102ef0 0x7fe2041041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:33.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.665+0000 7fe202d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe204102ef0 0x7fe2041041d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38324/0 (socket says 172.21.15.33:38324) 2024-09-16T00:15:33.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.665+0000 7fe202d9d640 1 -- 172.21.15.33:0/3688731547 learned_addr learned my addr 172.21.15.33:0/3688731547 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:33.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.665+0000 7fe201d9b640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe204102570 0x7fe204103c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:33.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.666+0000 7fe20259c640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204100e20 0x7fe204103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:33.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.666+0000 7fe202d9d640 1 -- 172.21.15.33:0/3688731547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe204102570 msgr2=0x7fe204103c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:33.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.666+0000 7fe202d9d640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe204102570 0x7fe204103c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:33.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.666+0000 7fe202d9d640 1 -- 172.21.15.33:0/3688731547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204100e20 msgr2=0x7fe204103750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:33.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.666+0000 7fe202d9d640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204100e20 0x7fe204103750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:33.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.666+0000 7fe202d9d640 1 -- 172.21.15.33:0/3688731547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2041a68b0 con 0x7fe204102ef0 2024-09-16T00:15:33.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.666+0000 7fe202d9d640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe204102ef0 0x7fe2041041d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400cce0 tx=0x7fe1f400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:33.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.666+0000 7fe1fb7fe640 1 -- 172.21.15.33:0/3688731547 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f400f450 con 0x7fe204102ef0 2024-09-16T00:15:33.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.666+0000 7fe1fb7fe640 1 -- 172.21.15.33:0/3688731547 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1f400fad0 con 0x7fe204102ef0 2024-09-16T00:15:33.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.666+0000 7fe1fb7fe640 1 -- 172.21.15.33:0/3688731547 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4018b60 con 0x7fe204102ef0 2024-09-16T00:15:33.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.666+0000 7fe208e12640 1 -- 172.21.15.33:0/3688731547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2041a6ba0 con 0x7fe204102ef0 2024-09-16T00:15:33.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.666+0000 7fe208e12640 1 -- 172.21.15.33:0/3688731547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2041a7090 con 0x7fe204102ef0 2024-09-16T00:15:33.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.667+0000 7fe208e12640 1 -- 172.21.15.33:0/3688731547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe204101200 con 0x7fe204102ef0 2024-09-16T00:15:33.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.668+0000 7fe1fb7fe640 1 -- 172.21.15.33:0/3688731547 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe1f400fc80 con 0x7fe204102ef0 2024-09-16T00:15:33.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.669+0000 7fe1fb7fe640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1f00a3240 0x7fe1f00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:33.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.669+0000 7fe20259c640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1f00a3240 0x7fe1f00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:33.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.670+0000 7fe1fb7fe640 1 -- 172.21.15.33:0/3688731547 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fe1f40cff70 con 0x7fe204102ef0 2024-09-16T00:15:33.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.670+0000 7fe20259c640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1f00a3240 0x7fe1f00a5700 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc000c00 tx=0x7fe1fc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:33.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.673+0000 7fe1fb7fe640 1 -- 172.21.15.33:0/3688731547 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe1f4014030 con 0x7fe204102ef0 2024-09-16T00:15:33.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.860+0000 7fe208e12640 1 -- 172.21.15.33:0/3688731547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe204104c80 con 0x7fe204102ef0 2024-09-16T00:15:33.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.860+0000 7fe1fb7fe640 1 -- 172.21.15.33:0/3688731547 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v86) ==== 81+0+588 (secure 0 0 0) 0x7fe204104c80 con 0x7fe204102ef0 2024-09-16T00:15:33.861 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/3553033370 2024-09-17T00:08:07.548547+0000 2024-09-16T00:15:33.861 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/122116489 2024-09-17T00:08:07.548547+0000 2024-09-16T00:15:33.861 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/1410359644 2024-09-17T00:08:07.548547+0000 2024-09-16T00:15:33.862 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2692588880 2024-09-17T00:08:07.548547+0000 2024-09-16T00:15:33.862 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/122116489 2024-09-17T00:08:07.548547+0000 2024-09-16T00:15:33.862 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/961233150 2024-09-17T00:07:43.600446+0000 2024-09-16T00:15:33.862 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/1160344784 2024-09-17T00:07:43.600446+0000 2024-09-16T00:15:33.862 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/961233150 2024-09-17T00:07:43.600446+0000 2024-09-16T00:15:33.862 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2039144880 2024-09-17T00:07:43.600446+0000 2024-09-16T00:15:33.862 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/3586056783 2024-09-17T00:07:43.600446+0000 2024-09-16T00:15:33.862 INFO:teuthology.orchestra.run.smithi033.stderr:listed 10 entries 2024-09-16T00:15:33.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.862+0000 7fe208e12640 1 -- 172.21.15.33:0/3688731547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1f00a3240 msgr2=0x7fe1f00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:33.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.862+0000 7fe208e12640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1f00a3240 0x7fe1f00a5700 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc000c00 tx=0x7fe1fc0058d0 comp rx=0 tx=0).stop 2024-09-16T00:15:33.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.862+0000 7fe208e12640 1 -- 172.21.15.33:0/3688731547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe204102ef0 msgr2=0x7fe2041041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:33.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.862+0000 7fe208e12640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe204102ef0 0x7fe2041041d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400cce0 tx=0x7fe1f400a620 comp rx=0 tx=0).stop 2024-09-16T00:15:33.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.863+0000 7fe208e12640 1 -- 172.21.15.33:0/3688731547 shutdown_connections 2024-09-16T00:15:33.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.863+0000 7fe208e12640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1f00a3240 0x7fe1f00a5700 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:33.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.863+0000 7fe208e12640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe204102ef0 0x7fe2041041d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:33.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.863+0000 7fe208e12640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe204102570 0x7fe204103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:33.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.863+0000 7fe208e12640 1 --2- 172.21.15.33:0/3688731547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204100e20 0x7fe204103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:33.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.863+0000 7fe208e12640 1 -- 172.21.15.33:0/3688731547 >> 172.21.15.33:0/3688731547 conn(0x7fe2040fc7d0 msgr2=0x7fe20406a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:33.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.863+0000 7fe208e12640 1 -- 172.21.15.33:0/3688731547 shutdown_connections 2024-09-16T00:15:33.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:33.863+0000 7fe208e12640 1 -- 172.21.15.33:0/3688731547 wait complete. 2024-09-16T00:15:34.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:34 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3688731547' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T00:15:34.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:34 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3688731547' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T00:15:34.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:34 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3688731547' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T00:15:34.688 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi033.front.sepia.ceph.com... 2024-09-16T00:15:34.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:15:34.688 DEBUG:teuthology.orchestra.run.smithi033:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-09-16T00:15:34.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi033.stderr:ceph-fuse[72777]: fuse finished with error 0 and tester_r 0 2024-09-16T00:15:34.720 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T00:15:35.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:35 smithi083 ceph-mon[35242]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-16T00:15:35.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:35 smithi060 ceph-mon[35185]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-16T00:15:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:35 smithi033 ceph-mon[31243]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-16T00:15:37.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:37 smithi083 ceph-mon[35242]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-16T00:15:37.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:37 smithi060 ceph-mon[35185]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-16T00:15:37.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:37 smithi033 ceph-mon[31243]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-16T00:15:39.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:39 smithi083 ceph-mon[35242]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-16T00:15:39.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:39 smithi060 ceph-mon[35185]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-16T00:15:39.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:39 smithi033 ceph-mon[31243]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-16T00:15:40.826 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi033.front.sepia.ceph.com 2024-09-16T00:15:40.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:15:40.827 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-09-16T00:15:40.860 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:15:41.164 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:41.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:41 smithi083 ceph-mon[35242]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T00:15:41.436 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:41 smithi033 ceph-mon[31243]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T00:15:41.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:41 smithi060 ceph-mon[35185]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T00:15:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.654+0000 7f57f49d0640 1 -- 172.21.15.33:0/3448605907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57f0103660 msgr2=0x7f57f010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.654+0000 7f57f49d0640 1 --2- 172.21.15.33:0/3448605907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57f0103660 0x7f57f010bb50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f57e4009900 tx=0x7f57e402a480 comp rx=0 tx=0).stop 2024-09-16T00:15:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.654+0000 7f57f49d0640 1 -- 172.21.15.33:0/3448605907 shutdown_connections 2024-09-16T00:15:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.654+0000 7f57f49d0640 1 --2- 172.21.15.33:0/3448605907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57f010c170 0x7f57f010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.654+0000 7f57f49d0640 1 --2- 172.21.15.33:0/3448605907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57f0103660 0x7f57f010bb50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.654+0000 7f57f49d0640 1 --2- 172.21.15.33:0/3448605907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57f0102d40 0x7f57f0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.654+0000 7f57f49d0640 1 -- 172.21.15.33:0/3448605907 >> 172.21.15.33:0/3448605907 conn(0x7f57f00fc7a0 msgr2=0x7f57f00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.654+0000 7f57f49d0640 1 -- 172.21.15.33:0/3448605907 shutdown_connections 2024-09-16T00:15:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.654+0000 7f57f49d0640 1 -- 172.21.15.33:0/3448605907 wait complete. 2024-09-16T00:15:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.655+0000 7f57f49d0640 1 Processor -- start 2024-09-16T00:15:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.655+0000 7f57f49d0640 1 -- start start 2024-09-16T00:15:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.655+0000 7f57f49d0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57f0102d40 0x7f57f0198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.655+0000 7f57f49d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57f0103660 0x7f57f0198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.655+0000 7f57f49d0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57f010c170 0x7f57f019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.655+0000 7f57f49d0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0111f70 con 0x7f57f010c170 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57f49d0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0111df0 con 0x7f57f0102d40 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57f49d0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01120f0 con 0x7f57f0103660 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57eed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57f010c170 0x7f57f019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57eed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57f010c170 0x7f57f019cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56446/0 (socket says 172.21.15.33:56446) 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57eed9d640 1 -- 172.21.15.33:0/4088712845 learned_addr learned my addr 172.21.15.33:0/4088712845 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57edd9b640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57f0103660 0x7f57f0198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57ee59c640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57f0102d40 0x7f57f0198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57eed9d640 1 -- 172.21.15.33:0/4088712845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57f0103660 msgr2=0x7f57f0198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57eed9d640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57f0103660 0x7f57f0198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57eed9d640 1 -- 172.21.15.33:0/4088712845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57f0102d40 msgr2=0x7f57f0198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57eed9d640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57f0102d40 0x7f57f0198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:41.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57eed9d640 1 -- 172.21.15.33:0/4088712845 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f019d3c0 con 0x7f57f010c170 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57eed9d640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57f010c170 0x7f57f019cce0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f57e800eaa0 tx=0x7f57e800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57df7fe640 1 -- 172.21.15.33:0/4088712845 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e800cdb0 con 0x7f57f010c170 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57df7fe640 1 -- 172.21.15.33:0/4088712845 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57e8005020 con 0x7f57f010c170 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57f49d0640 1 -- 172.21.15.33:0/4088712845 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f019d6b0 con 0x7f57f010c170 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.656+0000 7f57df7fe640 1 -- 172.21.15.33:0/4088712845 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e80116b0 con 0x7f57f010c170 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.657+0000 7f57f49d0640 1 -- 172.21.15.33:0/4088712845 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f01a4f90 con 0x7f57f010c170 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.658+0000 7f57f49d0640 1 -- 172.21.15.33:0/4088712845 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57f0103120 con 0x7f57f010c170 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.659+0000 7f57df7fe640 1 -- 172.21.15.33:0/4088712845 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f57e8005560 con 0x7f57f010c170 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.659+0000 7f57df7fe640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f57d80a32c0 0x7f57d80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.660+0000 7f57ee59c640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f57d80a32c0 0x7f57d80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.660+0000 7f57df7fe640 1 -- 172.21.15.33:0/4088712845 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f57e80cfd40 con 0x7f57f010c170 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.660+0000 7f57ee59c640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f57d80a32c0 0x7f57d80a5780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f57e00058c0 tx=0x7f57e000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.664+0000 7f57df7fe640 1 -- 172.21.15.33:0/4088712845 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f57e8016030 con 0x7f57f010c170 2024-09-16T00:15:41.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.842+0000 7f57f49d0640 1 -- 172.21.15.33:0/4088712845 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57f002d7c0 con 0x7f57f010c170 2024-09-16T00:15:41.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.844+0000 7f57df7fe640 1 -- 172.21.15.33:0/4088712845 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24138 (secure 0 0 0) 0x7f57e809c7c0 con 0x7f57f010c170 2024-09-16T00:15:41.845 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:15:41.845 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":86,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:15:16.193527+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:15:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.846+0000 7f57f49d0640 1 -- 172.21.15.33:0/4088712845 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f57d80a32c0 msgr2=0x7f57d80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.846+0000 7f57f49d0640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f57d80a32c0 0x7f57d80a5780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f57e00058c0 tx=0x7f57e000a4c0 comp rx=0 tx=0).stop 2024-09-16T00:15:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.846+0000 7f57f49d0640 1 -- 172.21.15.33:0/4088712845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57f010c170 msgr2=0x7f57f019cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.846+0000 7f57f49d0640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57f010c170 0x7f57f019cce0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f57e800eaa0 tx=0x7f57e800ef60 comp rx=0 tx=0).stop 2024-09-16T00:15:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.846+0000 7f57f49d0640 1 -- 172.21.15.33:0/4088712845 shutdown_connections 2024-09-16T00:15:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.846+0000 7f57f49d0640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f57d80a32c0 0x7f57d80a5780 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.846+0000 7f57f49d0640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57f010c170 0x7f57f019cce0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.846+0000 7f57f49d0640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57f0103660 0x7f57f0198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.846+0000 7f57f49d0640 1 --2- 172.21.15.33:0/4088712845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57f0102d40 0x7f57f0198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.846+0000 7f57f49d0640 1 -- 172.21.15.33:0/4088712845 >> 172.21.15.33:0/4088712845 conn(0x7f57f00fc7a0 msgr2=0x7f57f00fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.846+0000 7f57f49d0640 1 -- 172.21.15.33:0/4088712845 shutdown_connections 2024-09-16T00:15:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:41.846+0000 7f57f49d0640 1 -- 172.21.15.33:0/4088712845 wait complete. 2024-09-16T00:15:42.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:42 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4088712845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:42.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:42 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4088712845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:42.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:42 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4088712845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:42.679 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:15:42.943 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:43.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:43 smithi033 ceph-mon[31243]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T00:15:43.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:43 smithi083 ceph-mon[35242]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T00:15:43.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.411+0000 7f63b4518640 1 -- 172.21.15.33:0/3230618598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63ac102ef0 msgr2=0x7f63ac10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:43.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.411+0000 7f63b4518640 1 --2- 172.21.15.33:0/3230618598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63ac102ef0 0x7f63ac10da40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f63a4009ca0 tx=0x7f63a402a950 comp rx=0 tx=0).stop 2024-09-16T00:15:43.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.411+0000 7f63b4518640 1 -- 172.21.15.33:0/3230618598 shutdown_connections 2024-09-16T00:15:43.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.411+0000 7f63b4518640 1 --2- 172.21.15.33:0/3230618598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63ac102ef0 0x7f63ac10da40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:43.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.411+0000 7f63b4518640 1 --2- 172.21.15.33:0/3230618598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63ac102570 0x7f63ac1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:43.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.411+0000 7f63b4518640 1 --2- 172.21.15.33:0/3230618598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63ac100e20 0x7f63ac101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:43.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.411+0000 7f63b4518640 1 -- 172.21.15.33:0/3230618598 >> 172.21.15.33:0/3230618598 conn(0x7f63ac0fc7d0 msgr2=0x7f63ac0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:43.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.411+0000 7f63b4518640 1 -- 172.21.15.33:0/3230618598 shutdown_connections 2024-09-16T00:15:43.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.411+0000 7f63b4518640 1 -- 172.21.15.33:0/3230618598 wait complete. 2024-09-16T00:15:43.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.411+0000 7f63b4518640 1 Processor -- start 2024-09-16T00:15:43.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.411+0000 7f63b4518640 1 -- start start 2024-09-16T00:15:43.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b4518640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63ac100e20 0x7f63ac198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:43.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b4518640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63ac102570 0x7f63ac1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:43.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b4518640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63ac102ef0 0x7f63ac19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:43.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b4518640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac110f80 con 0x7f63ac102ef0 2024-09-16T00:15:43.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b4518640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac110e00 con 0x7f63ac100e20 2024-09-16T00:15:43.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b4518640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac111100 con 0x7f63ac102570 2024-09-16T00:15:43.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b2ab5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63ac102ef0 0x7f63ac19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:43.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b2ab5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63ac102ef0 0x7f63ac19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56454/0 (socket says 172.21.15.33:56454) 2024-09-16T00:15:43.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b2ab5640 1 -- 172.21.15.33:0/4062442559 learned_addr learned my addr 172.21.15.33:0/4062442559 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:43.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b22b4640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63ac100e20 0x7f63ac198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:43.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b1ab3640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63ac102570 0x7f63ac1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:43.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b2ab5640 1 -- 172.21.15.33:0/4062442559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63ac102570 msgr2=0x7f63ac1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:43.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b2ab5640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63ac102570 0x7f63ac1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:43.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b2ab5640 1 -- 172.21.15.33:0/4062442559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63ac100e20 msgr2=0x7f63ac198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:43.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b2ab5640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63ac100e20 0x7f63ac198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:43.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b2ab5640 1 -- 172.21.15.33:0/4062442559 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ac19d440 con 0x7f63ac102ef0 2024-09-16T00:15:43.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.412+0000 7f63b2ab5640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63ac102ef0 0x7f63ac19cd60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f63a4004af0 tx=0x7f63a402f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:43.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.413+0000 7f63a37fe640 1 -- 172.21.15.33:0/4062442559 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4034040 con 0x7f63ac102ef0 2024-09-16T00:15:43.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.413+0000 7f63a37fe640 1 -- 172.21.15.33:0/4062442559 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a4030920 con 0x7f63ac102ef0 2024-09-16T00:15:43.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.413+0000 7f63b4518640 1 -- 172.21.15.33:0/4062442559 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ac19d6d0 con 0x7f63ac102ef0 2024-09-16T00:15:43.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.413+0000 7f63a37fe640 1 -- 172.21.15.33:0/4062442559 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4004820 con 0x7f63ac102ef0 2024-09-16T00:15:43.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.413+0000 7f63b4518640 1 -- 172.21.15.33:0/4062442559 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ac1a5010 con 0x7f63ac102ef0 2024-09-16T00:15:43.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.414+0000 7f63b4518640 1 -- 172.21.15.33:0/4062442559 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63ac101200 con 0x7f63ac102ef0 2024-09-16T00:15:43.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.415+0000 7f63a37fe640 1 -- 172.21.15.33:0/4062442559 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f63a4037020 con 0x7f63ac102ef0 2024-09-16T00:15:43.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.415+0000 7f63a37fe640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f63980a32c0 0x7f63980a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:43.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.416+0000 7f63a37fe640 1 -- 172.21.15.33:0/4062442559 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f63a40f0900 con 0x7f63ac102ef0 2024-09-16T00:15:43.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.416+0000 7f63b22b4640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f63980a32c0 0x7f63980a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:43.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.416+0000 7f63b22b4640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f63980a32c0 0x7f63980a5780 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009bd0 tx=0x7f63a8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:43.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.420+0000 7f63a37fe640 1 -- 172.21.15.33:0/4062442559 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f63a40bd3e0 con 0x7f63ac102ef0 2024-09-16T00:15:43.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:43 smithi060 ceph-mon[35185]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T00:15:43.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.596+0000 7f63b4518640 1 -- 172.21.15.33:0/4062442559 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f63ac10a0a0 con 0x7f63ac102ef0 2024-09-16T00:15:43.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.597+0000 7f63a37fe640 1 -- 172.21.15.33:0/4062442559 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24138 (secure 0 0 0) 0x7f63a40bcce0 con 0x7f63ac102ef0 2024-09-16T00:15:43.598 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:15:43.598 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":86,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:15:16.193527+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:15:43.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.600+0000 7f63b4518640 1 -- 172.21.15.33:0/4062442559 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f63980a32c0 msgr2=0x7f63980a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:43.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.600+0000 7f63b4518640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f63980a32c0 0x7f63980a5780 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009bd0 tx=0x7f63a8005f90 comp rx=0 tx=0).stop 2024-09-16T00:15:43.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.600+0000 7f63b4518640 1 -- 172.21.15.33:0/4062442559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63ac102ef0 msgr2=0x7f63ac19cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:43.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.600+0000 7f63b4518640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63ac102ef0 0x7f63ac19cd60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f63a4004af0 tx=0x7f63a402f5b0 comp rx=0 tx=0).stop 2024-09-16T00:15:43.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.600+0000 7f63b4518640 1 -- 172.21.15.33:0/4062442559 shutdown_connections 2024-09-16T00:15:43.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.600+0000 7f63b4518640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f63980a32c0 0x7f63980a5780 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:43.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.600+0000 7f63b4518640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63ac102ef0 0x7f63ac19cd60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:43.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.600+0000 7f63b4518640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63ac102570 0x7f63ac1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:43.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.600+0000 7f63b4518640 1 --2- 172.21.15.33:0/4062442559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63ac100e20 0x7f63ac198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:43.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.600+0000 7f63b4518640 1 -- 172.21.15.33:0/4062442559 >> 172.21.15.33:0/4062442559 conn(0x7f63ac0fc7d0 msgr2=0x7f63ac06a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:43.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.600+0000 7f63b4518640 1 -- 172.21.15.33:0/4062442559 shutdown_connections 2024-09-16T00:15:43.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:43.600+0000 7f63b4518640 1 -- 172.21.15.33:0/4062442559 wait complete. 2024-09-16T00:15:44.380 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:44 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4062442559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:44.412 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-09-16T00:15:44.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:44 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4062442559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:44.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:44 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4062442559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:15:44.676 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:45.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.185+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/3132934083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8b4100e20 msgr2=0x7fa8b4101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:45.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.185+0000 7fa8b8d3d640 1 --2- 172.21.15.33:0/3132934083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8b4100e20 0x7fa8b4101200 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4009900 tx=0x7fa8a402a480 comp rx=0 tx=0).stop 2024-09-16T00:15:45.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.185+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/3132934083 shutdown_connections 2024-09-16T00:15:45.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.185+0000 7fa8b8d3d640 1 --2- 172.21.15.33:0/3132934083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b4102ef0 0x7fa8b410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:45.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.185+0000 7fa8b8d3d640 1 --2- 172.21.15.33:0/3132934083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8b4102570 0x7fa8b41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:45.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.185+0000 7fa8b8d3d640 1 --2- 172.21.15.33:0/3132934083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8b4100e20 0x7fa8b4101200 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:45.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.185+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/3132934083 >> 172.21.15.33:0/3132934083 conn(0x7fa8b40fc7d0 msgr2=0x7fa8b40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:45.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.185+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/3132934083 shutdown_connections 2024-09-16T00:15:45.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.185+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/3132934083 wait complete. 2024-09-16T00:15:45.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b8d3d640 1 Processor -- start 2024-09-16T00:15:45.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b8d3d640 1 -- start start 2024-09-16T00:15:45.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b8d3d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8b4100e20 0x7fa8b4199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:45.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b8d3d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b4102570 0x7fa8b41997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:45.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b8d3d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8b4102ef0 0x7fa8b41a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:45.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b8d3d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b4110f50 con 0x7fa8b4100e20 2024-09-16T00:15:45.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b8d3d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b4110dd0 con 0x7fa8b4102570 2024-09-16T00:15:45.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b8d3d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b41110d0 con 0x7fa8b4102ef0 2024-09-16T00:15:45.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8b4100e20 0x7fa8b4199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:45.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8b4100e20 0x7fa8b4199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56470/0 (socket says 172.21.15.33:56470) 2024-09-16T00:15:45.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b259c640 1 -- 172.21.15.33:0/1481345673 learned_addr learned my addr 172.21.15.33:0/1481345673 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:45.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b2d9d640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8b4102ef0 0x7fa8b41a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:45.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8a9d9b640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b4102570 0x7fa8b41997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:45.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b259c640 1 -- 172.21.15.33:0/1481345673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8b4102ef0 msgr2=0x7fa8b41a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:45.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b259c640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8b4102ef0 0x7fa8b41a42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:45.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b259c640 1 -- 172.21.15.33:0/1481345673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b4102570 msgr2=0x7fa8b41997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:45.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b259c640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b4102570 0x7fa8b41997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:45.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.186+0000 7fa8b259c640 1 -- 172.21.15.33:0/1481345673 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b41a4820 con 0x7fa8b4100e20 2024-09-16T00:15:45.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.187+0000 7fa8b2d9d640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8b4102ef0 0x7fa8b41a42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:15:45.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.187+0000 7fa8b259c640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8b4100e20 0x7fa8b4199260 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4006010 tx=0x7fa8a4003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:45.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.187+0000 7fa8abfff640 1 -- 172.21.15.33:0/1481345673 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a403f070 con 0x7fa8b4100e20 2024-09-16T00:15:45.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.187+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/1481345673 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8b41a49f0 con 0x7fa8b4100e20 2024-09-16T00:15:45.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.187+0000 7fa8abfff640 1 -- 172.21.15.33:0/1481345673 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8a4031e30 con 0x7fa8b4100e20 2024-09-16T00:15:45.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.187+0000 7fa8abfff640 1 -- 172.21.15.33:0/1481345673 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4035030 con 0x7fa8b4100e20 2024-09-16T00:15:45.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.187+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/1481345673 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8b41a4e40 con 0x7fa8b4100e20 2024-09-16T00:15:45.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.188+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/1481345673 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8b4101200 con 0x7fa8b4100e20 2024-09-16T00:15:45.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.189+0000 7fa8abfff640 1 -- 172.21.15.33:0/1481345673 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fa8a4042070 con 0x7fa8b4100e20 2024-09-16T00:15:45.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.190+0000 7fa8abfff640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8a00a3240 0x7fa8a00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:45.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.190+0000 7fa8a9d9b640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8a00a3240 0x7fa8a00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:45.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.190+0000 7fa8abfff640 1 -- 172.21.15.33:0/1481345673 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fa8a40f0550 con 0x7fa8b4100e20 2024-09-16T00:15:45.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.190+0000 7fa8a9d9b640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8a00a3240 0x7fa8a00a5700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa89c005e10 tx=0x7fa89c005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:45.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.194+0000 7fa8abfff640 1 -- 172.21.15.33:0/1481345673 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa8a40bd030 con 0x7fa8b4100e20 2024-09-16T00:15:45.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:45.410+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/1481345673 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa8b41a5070 con 0x7fa8b4100e20 2024-09-16T00:15:45.411 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:45 smithi033 ceph-mon[31243]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T00:15:45.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:45 smithi083 ceph-mon[35242]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T00:15:45.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:45 smithi060 ceph-mon[35185]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T00:15:46.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.149+0000 7fa8abfff640 1 -- 172.21.15.33:0/1481345673 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fa8a40bc930 con 0x7fa8b4100e20 2024-09-16T00:15:46.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.152+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/1481345673 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8a00a3240 msgr2=0x7fa8a00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:46.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.152+0000 7fa8b8d3d640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8a00a3240 0x7fa8a00a5700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa89c005e10 tx=0x7fa89c005da0 comp rx=0 tx=0).stop 2024-09-16T00:15:46.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.152+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/1481345673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8b4100e20 msgr2=0x7fa8b4199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:46.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.152+0000 7fa8b8d3d640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8b4100e20 0x7fa8b4199260 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4006010 tx=0x7fa8a4003c80 comp rx=0 tx=0).stop 2024-09-16T00:15:46.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.152+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/1481345673 shutdown_connections 2024-09-16T00:15:46.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.152+0000 7fa8b8d3d640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8a00a3240 0x7fa8a00a5700 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:46.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.152+0000 7fa8b8d3d640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8b4102ef0 0x7fa8b41a42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:46.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.152+0000 7fa8b8d3d640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b4102570 0x7fa8b41997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:46.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.152+0000 7fa8b8d3d640 1 --2- 172.21.15.33:0/1481345673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8b4100e20 0x7fa8b4199260 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:46.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.152+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/1481345673 >> 172.21.15.33:0/1481345673 conn(0x7fa8b40fc7d0 msgr2=0x7fa8b41067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:46.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.153+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/1481345673 shutdown_connections 2024-09-16T00:15:46.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:46.153+0000 7fa8b8d3d640 1 -- 172.21.15.33:0/1481345673 wait complete. 2024-09-16T00:15:46.372 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:46 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1481345673' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-16T00:15:46.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:46 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1481345673' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-16T00:15:46.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:46 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1481345673' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-16T00:15:46.907 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-09-16T00:15:46.907 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-09-16T00:15:47.169 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:47.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:47 smithi033 ceph-mon[31243]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-16T00:15:47.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:47 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1481345673' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-16T00:15:47.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:47 smithi033 ceph-mon[31243]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:47.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:47 smithi083 ceph-mon[35242]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-16T00:15:47.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:47 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1481345673' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-16T00:15:47.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:47 smithi083 ceph-mon[35242]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:47.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:47 smithi060 ceph-mon[35185]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-16T00:15:47.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:47 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1481345673' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-16T00:15:47.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:47 smithi060 ceph-mon[35185]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:47.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.695+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1794195250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4102ef0 msgr2=0x7fd6c410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:47.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.695+0000 7fd6ca0a7640 1 --2- 172.21.15.33:0/1794195250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4102ef0 0x7fd6c410da40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009900 tx=0x7fd6bc02a480 comp rx=0 tx=0).stop 2024-09-16T00:15:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.695+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1794195250 shutdown_connections 2024-09-16T00:15:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.695+0000 7fd6ca0a7640 1 --2- 172.21.15.33:0/1794195250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4102ef0 0x7fd6c410da40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.695+0000 7fd6ca0a7640 1 --2- 172.21.15.33:0/1794195250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6c4102570 0x7fd6c41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.695+0000 7fd6ca0a7640 1 --2- 172.21.15.33:0/1794195250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6c4100e20 0x7fd6c4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.695+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1794195250 >> 172.21.15.33:0/1794195250 conn(0x7fd6c40fc7d0 msgr2=0x7fd6c40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.695+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1794195250 shutdown_connections 2024-09-16T00:15:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.695+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1794195250 wait complete. 2024-09-16T00:15:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.696+0000 7fd6ca0a7640 1 Processor -- start 2024-09-16T00:15:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.696+0000 7fd6ca0a7640 1 -- start start 2024-09-16T00:15:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.696+0000 7fd6ca0a7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100e20 0x7fd6c419a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.696+0000 7fd6ca0a7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6c4102570 0x7fd6c419aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.696+0000 7fd6ca0a7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6c4102ef0 0x7fd6c419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.696+0000 7fd6ca0a7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c4110f80 con 0x7fd6c4100e20 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.696+0000 7fd6ca0a7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c4110e00 con 0x7fd6c4102ef0 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.696+0000 7fd6ca0a7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c4111100 con 0x7fd6c4102570 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100e20 0x7fd6c419a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100e20 0x7fd6c419a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56484/0 (socket says 172.21.15.33:56484) 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c37fe640 1 -- 172.21.15.33:0/1337814365 learned_addr learned my addr 172.21.15.33:0/1337814365 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6c4102570 0x7fd6c419aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c3fff640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6c4102ef0 0x7fd6c419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c37fe640 1 -- 172.21.15.33:0/1337814365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6c4102570 msgr2=0x7fd6c419aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6c4102570 0x7fd6c419aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c37fe640 1 -- 172.21.15.33:0/1337814365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6c4102ef0 msgr2=0x7fd6c419ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6c4102ef0 0x7fd6c419ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c37fe640 1 -- 172.21.15.33:0/1337814365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c419f550 con 0x7fd6c4100e20 2024-09-16T00:15:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c3fff640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6c4102ef0 0x7fd6c419ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:15:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100e20 0x7fd6c419a5a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400eab0 tx=0x7fd6b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c0ff9640 1 -- 172.21.15.33:0/1337814365 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b400cd80 con 0x7fd6c4100e20 2024-09-16T00:15:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c0ff9640 1 -- 172.21.15.33:0/1337814365 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6b4011950 con 0x7fd6c4100e20 2024-09-16T00:15:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1337814365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c419f840 con 0x7fd6c4100e20 2024-09-16T00:15:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6c0ff9640 1 -- 172.21.15.33:0/1337814365 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4010a00 con 0x7fd6c4100e20 2024-09-16T00:15:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.697+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1337814365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c41a7120 con 0x7fd6c4100e20 2024-09-16T00:15:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.698+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1337814365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6c4101200 con 0x7fd6c4100e20 2024-09-16T00:15:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.700+0000 7fd6c0ff9640 1 -- 172.21.15.33:0/1337814365 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fd6b4011b00 con 0x7fd6c4100e20 2024-09-16T00:15:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.701+0000 7fd6c0ff9640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6ac0a3240 0x7fd6ac0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.701+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6ac0a3240 0x7fd6ac0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.701+0000 7fd6c0ff9640 1 -- 172.21.15.33:0/1337814365 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fd6b40d01b0 con 0x7fd6c4100e20 2024-09-16T00:15:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.701+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6ac0a3240 0x7fd6ac0a5700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4104b30 tx=0x7fd6b800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.705+0000 7fd6c0ff9640 1 -- 172.21.15.33:0/1337814365 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd6b4016030 con 0x7fd6c4100e20 2024-09-16T00:15:47.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:47.916+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1337814365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fd6c41a72f0 con 0x7fd6c4100e20 2024-09-16T00:15:48.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.155+0000 7fd6c0ff9640 1 -- 172.21.15.33:0/1337814365 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fd6b409cc90 con 0x7fd6c4100e20 2024-09-16T00:15:48.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.159+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1337814365 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6ac0a3240 msgr2=0x7fd6ac0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:48.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.159+0000 7fd6ca0a7640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6ac0a3240 0x7fd6ac0a5700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4104b30 tx=0x7fd6b800a2d0 comp rx=0 tx=0).stop 2024-09-16T00:15:48.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.159+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1337814365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100e20 msgr2=0x7fd6c419a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:48.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.159+0000 7fd6ca0a7640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100e20 0x7fd6c419a5a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400eab0 tx=0x7fd6b400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:15:48.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.159+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1337814365 shutdown_connections 2024-09-16T00:15:48.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.159+0000 7fd6ca0a7640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd6ac0a3240 0x7fd6ac0a5700 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:48.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.159+0000 7fd6ca0a7640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6c4102ef0 0x7fd6c419ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:48.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.159+0000 7fd6ca0a7640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6c4102570 0x7fd6c419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:48.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.159+0000 7fd6ca0a7640 1 --2- 172.21.15.33:0/1337814365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100e20 0x7fd6c419a5a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:48.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.159+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1337814365 >> 172.21.15.33:0/1337814365 conn(0x7fd6c40fc7d0 msgr2=0x7fd6c406a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:48.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.159+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1337814365 shutdown_connections 2024-09-16T00:15:48.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:48.159+0000 7fd6ca0a7640 1 -- 172.21.15.33:0/1337814365 wait complete. 2024-09-16T00:15:48.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:48 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1337814365' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-09-16T00:15:48.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:48 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1337814365' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-09-16T00:15:48.492 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:48 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1337814365' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-09-16T00:15:48.842 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-09-16T00:15:49.103 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:49 smithi083 ceph-mon[35242]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-16T00:15:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:49 smithi083 ceph-mon[35242]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-16T00:15:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:49 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1337814365' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-09-16T00:15:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:49 smithi083 ceph-mon[35242]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-16T00:15:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:49 smithi083 ceph-mon[35242]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:49 smithi083 ceph-mon[35242]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-09-16T00:15:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:49 smithi083 ceph-mon[35242]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-09-16T00:15:49.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:49 smithi060 ceph-mon[35185]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-16T00:15:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:49 smithi060 ceph-mon[35185]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-16T00:15:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:49 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1337814365' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-09-16T00:15:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:49 smithi060 ceph-mon[35185]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-16T00:15:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:49 smithi060 ceph-mon[35185]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:49 smithi060 ceph-mon[35185]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-09-16T00:15:49.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:49 smithi060 ceph-mon[35185]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-09-16T00:15:49.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:49 smithi033 ceph-mon[31243]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-16T00:15:49.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:49 smithi033 ceph-mon[31243]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-16T00:15:49.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:49 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1337814365' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-09-16T00:15:49.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:49 smithi033 ceph-mon[31243]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-16T00:15:49.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:49 smithi033 ceph-mon[31243]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T00:15:49.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:49 smithi033 ceph-mon[31243]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-09-16T00:15:49.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:49 smithi033 ceph-mon[31243]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-09-16T00:15:49.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.572+0000 7f8d81691640 1 -- 172.21.15.33:0/899541755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c100a80 msgr2=0x7f8d7c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:49.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.572+0000 7f8d81691640 1 --2- 172.21.15.33:0/899541755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c100a80 0x7f8d7c100ee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009930 tx=0x7f8d6402a420 comp rx=0 tx=0).stop 2024-09-16T00:15:49.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.572+0000 7f8d81691640 1 -- 172.21.15.33:0/899541755 shutdown_connections 2024-09-16T00:15:49.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.572+0000 7f8d81691640 1 --2- 172.21.15.33:0/899541755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d7c101420 0x7f8d7c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:49.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.572+0000 7f8d81691640 1 --2- 172.21.15.33:0/899541755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c100a80 0x7f8d7c100ee0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:49.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.572+0000 7f8d81691640 1 --2- 172.21.15.33:0/899541755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d7c106a80 0x7f8d7c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:49.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.572+0000 7f8d81691640 1 -- 172.21.15.33:0/899541755 >> 172.21.15.33:0/899541755 conn(0x7f8d7c0fc7a0 msgr2=0x7f8d7c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:49.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.572+0000 7f8d81691640 1 -- 172.21.15.33:0/899541755 shutdown_connections 2024-09-16T00:15:49.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.572+0000 7f8d81691640 1 -- 172.21.15.33:0/899541755 wait complete. 2024-09-16T00:15:49.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.573+0000 7f8d81691640 1 Processor -- start 2024-09-16T00:15:49.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.573+0000 7f8d81691640 1 -- start start 2024-09-16T00:15:49.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.573+0000 7f8d81691640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d7c100a80 0x7f8d7c199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:49.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.573+0000 7f8d81691640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c101420 0x7f8d7c199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:49.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.573+0000 7f8d81691640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d7c106a80 0x7f8d7c1a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:49.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.573+0000 7f8d81691640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7c110ea0 con 0x7f8d7c101420 2024-09-16T00:15:49.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.573+0000 7f8d81691640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7c110d20 con 0x7f8d7c106a80 2024-09-16T00:15:49.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.573+0000 7f8d81691640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7c111020 con 0x7f8d7c100a80 2024-09-16T00:15:49.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.573+0000 7f8d7a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c101420 0x7f8d7c199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:49.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.573+0000 7f8d7a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c101420 0x7f8d7c199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56496/0 (socket says 172.21.15.33:56496) 2024-09-16T00:15:49.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.573+0000 7f8d7a7fc640 1 -- 172.21.15.33:0/125858484 learned_addr learned my addr 172.21.15.33:0/125858484 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:49.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d7b7fe640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d7c106a80 0x7f8d7c1a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:49.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d7affd640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d7c100a80 0x7f8d7c199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:49.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d7a7fc640 1 -- 172.21.15.33:0/125858484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d7c100a80 msgr2=0x7f8d7c199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:49.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d7a7fc640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d7c100a80 0x7f8d7c199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:49.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d7a7fc640 1 -- 172.21.15.33:0/125858484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d7c106a80 msgr2=0x7f8d7c1a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:49.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d7a7fc640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d7c106a80 0x7f8d7c1a43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:49.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d7a7fc640 1 -- 172.21.15.33:0/125858484 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d7c1a48f0 con 0x7f8d7c101420 2024-09-16T00:15:49.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d7affd640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d7c100a80 0x7f8d7c199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:15:49.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d7a7fc640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c101420 0x7f8d7c199870 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009a50 tx=0x7f8d64004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:49.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d73fff640 1 -- 172.21.15.33:0/125858484 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6403f030 con 0x7f8d7c101420 2024-09-16T00:15:49.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d81691640 1 -- 172.21.15.33:0/125858484 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d7c1a4ac0 con 0x7f8d7c101420 2024-09-16T00:15:49.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d73fff640 1 -- 172.21.15.33:0/125858484 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d64003ce0 con 0x7f8d7c101420 2024-09-16T00:15:49.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d73fff640 1 -- 172.21.15.33:0/125858484 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d64035070 con 0x7f8d7c101420 2024-09-16T00:15:49.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.574+0000 7f8d81691640 1 -- 172.21.15.33:0/125858484 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d7c1a4f20 con 0x7f8d7c101420 2024-09-16T00:15:49.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.575+0000 7f8d81691640 1 -- 172.21.15.33:0/125858484 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d7c100ee0 con 0x7f8d7c101420 2024-09-16T00:15:49.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.577+0000 7f8d73fff640 1 -- 172.21.15.33:0/125858484 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f8d64042030 con 0x7f8d7c101420 2024-09-16T00:15:49.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.577+0000 7f8d73fff640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d680a3240 0x7f8d680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:49.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.578+0000 7f8d7affd640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d680a3240 0x7f8d680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:49.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.578+0000 7f8d73fff640 1 -- 172.21.15.33:0/125858484 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f8d640f0540 con 0x7f8d7c101420 2024-09-16T00:15:49.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.578+0000 7f8d7affd640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d680a3240 0x7f8d680a5700 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c0058c0 tx=0x7f8d6c005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:49.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.581+0000 7f8d73fff640 1 -- 172.21.15.33:0/125858484 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8d640bd020 con 0x7f8d7c101420 2024-09-16T00:15:49.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:49.793+0000 7f8d81691640 1 -- 172.21.15.33:0/125858484 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f8d7c02d7c0 con 0x7f8d7c101420 2024-09-16T00:15:50.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.171+0000 7f8d73fff640 1 -- 172.21.15.33:0/125858484 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f8d640bc920 con 0x7f8d7c101420 2024-09-16T00:15:50.807 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:50 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] up:active 2024-09-16T00:15:50.807 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:50 smithi060 ceph-mon[35185]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-16T00:15:50.808 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:50 smithi060 ceph-mon[35185]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-16T00:15:50.808 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:50 smithi060 ceph-mon[35185]: Cluster is now healthy 2024-09-16T00:15:50.808 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:50 smithi060 ceph-mon[35185]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-09-16T00:15:50.808 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:50 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-09-16T00:15:50.808 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:50 smithi060 ceph-mon[35185]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-16T00:15:50.808 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:50 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/125858484' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-16T00:15:50.808 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:50 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] up:active 2024-09-16T00:15:50.808 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:50 smithi083 ceph-mon[35242]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-16T00:15:50.808 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:50 smithi083 ceph-mon[35242]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-16T00:15:50.808 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:50 smithi083 ceph-mon[35242]: Cluster is now healthy 2024-09-16T00:15:50.808 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:50 smithi083 ceph-mon[35242]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-09-16T00:15:50.809 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:50 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-09-16T00:15:50.809 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:50 smithi083 ceph-mon[35242]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-16T00:15:50.809 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:50 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/125858484' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-16T00:15:50.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.174+0000 7f8d81691640 1 -- 172.21.15.33:0/125858484 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d680a3240 msgr2=0x7f8d680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:50.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.174+0000 7f8d81691640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d680a3240 0x7f8d680a5700 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c0058c0 tx=0x7f8d6c005d30 comp rx=0 tx=0).stop 2024-09-16T00:15:50.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.174+0000 7f8d81691640 1 -- 172.21.15.33:0/125858484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c101420 msgr2=0x7f8d7c199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:50.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.174+0000 7f8d81691640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c101420 0x7f8d7c199870 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009a50 tx=0x7f8d64004270 comp rx=0 tx=0).stop 2024-09-16T00:15:50.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.174+0000 7f8d81691640 1 -- 172.21.15.33:0/125858484 shutdown_connections 2024-09-16T00:15:50.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.174+0000 7f8d81691640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d680a3240 0x7f8d680a5700 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:50.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.174+0000 7f8d81691640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d7c106a80 0x7f8d7c1a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:50.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.174+0000 7f8d81691640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c101420 0x7f8d7c199870 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:50.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.174+0000 7f8d81691640 1 --2- 172.21.15.33:0/125858484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d7c100a80 0x7f8d7c199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:50.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.174+0000 7f8d81691640 1 -- 172.21.15.33:0/125858484 >> 172.21.15.33:0/125858484 conn(0x7f8d7c0fc7a0 msgr2=0x7f8d7c0fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:50.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.174+0000 7f8d81691640 1 -- 172.21.15.33:0/125858484 shutdown_connections 2024-09-16T00:15:50.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:50.174+0000 7f8d81691640 1 -- 172.21.15.33:0/125858484 wait complete. 2024-09-16T00:15:50.811 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:50 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] up:active 2024-09-16T00:15:50.811 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:50 smithi033 ceph-mon[31243]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-16T00:15:50.811 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:50 smithi033 ceph-mon[31243]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-16T00:15:50.811 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:50 smithi033 ceph-mon[31243]: Cluster is now healthy 2024-09-16T00:15:50.811 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:50 smithi033 ceph-mon[31243]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-09-16T00:15:50.811 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:50 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-09-16T00:15:50.811 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:50 smithi033 ceph-mon[31243]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-16T00:15:50.811 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:50 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/125858484' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-16T00:15:50.921 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-09-16T00:15:51.186 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:51.452 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:51 smithi033 ceph-mon[31243]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-09-16T00:15:51.453 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:51 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/125858484' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-16T00:15:51.453 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:51 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] up:active 2024-09-16T00:15:51.453 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:51 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-09-16T00:15:51.453 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:51 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:15:51.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:51 smithi060 ceph-mon[35185]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-09-16T00:15:51.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/125858484' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-16T00:15:51.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:51 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] up:active 2024-09-16T00:15:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:51 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-09-16T00:15:51.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:51 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:15:51.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:51 smithi083 ceph-mon[35242]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-09-16T00:15:51.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:51 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/125858484' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-16T00:15:51.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:51 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] up:active 2024-09-16T00:15:51.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:51 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-09-16T00:15:51.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:51 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:15:51.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.691+0000 7f8477fa0640 1 -- 172.21.15.33:0/3534834317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8470106f10 msgr2=0x7f847010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:51.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.691+0000 7f8477fa0640 1 --2- 172.21.15.33:0/3534834317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8470106f10 0x7f847010ea90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8468009ca0 tx=0x7f846802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:15:51.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.691+0000 7f8477fa0640 1 -- 172.21.15.33:0/3534834317 shutdown_connections 2024-09-16T00:15:51.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.691+0000 7f8477fa0640 1 --2- 172.21.15.33:0/3534834317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8470106f10 0x7f847010ea90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:51.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.691+0000 7f8477fa0640 1 --2- 172.21.15.33:0/3534834317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8470102790 0x7f84701067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:51.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.691+0000 7f8477fa0640 1 --2- 172.21.15.33:0/3534834317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8470101de0 0x7f84701021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:51.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.691+0000 7f8477fa0640 1 -- 172.21.15.33:0/3534834317 >> 172.21.15.33:0/3534834317 conn(0x7f84700fd8c0 msgr2=0x7f84700ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:51.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.691+0000 7f8477fa0640 1 -- 172.21.15.33:0/3534834317 shutdown_connections 2024-09-16T00:15:51.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.691+0000 7f8477fa0640 1 -- 172.21.15.33:0/3534834317 wait complete. 2024-09-16T00:15:51.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.691+0000 7f8477fa0640 1 Processor -- start 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8477fa0640 1 -- start start 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8477fa0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8470101de0 0x7f8470198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8477fa0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8470102790 0x7f8470198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8477fa0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8470106f10 0x7f847019ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8477fa0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8470111f90 con 0x7f8470101de0 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8477fa0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8470111e10 con 0x7f8470102790 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8477fa0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8470112110 con 0x7f8470106f10 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8475d3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8470101de0 0x7f8470198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8475d3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8470101de0 0x7f8470198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51316/0 (socket says 172.21.15.33:51316) 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8475d3c640 1 -- 172.21.15.33:0/104048539 learned_addr learned my addr 172.21.15.33:0/104048539 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f847653d640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8470106f10 0x7f847019ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f847553b640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8470102790 0x7f8470198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:51.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8475d3c640 1 -- 172.21.15.33:0/104048539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8470106f10 msgr2=0x7f847019ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:51.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8475d3c640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8470106f10 0x7f847019ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:51.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8475d3c640 1 -- 172.21.15.33:0/104048539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8470102790 msgr2=0x7f8470198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:51.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8475d3c640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8470102790 0x7f8470198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:51.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8475d3c640 1 -- 172.21.15.33:0/104048539 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847019d500 con 0x7f8470101de0 2024-09-16T00:15:51.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f847653d640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8470106f10 0x7f847019ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:15:51.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.692+0000 7f8475d3c640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8470101de0 0x7f8470198550 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f846c00d950 tx=0x7f846c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:51.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.693+0000 7f8466ffd640 1 -- 172.21.15.33:0/104048539 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c017070 con 0x7f8470101de0 2024-09-16T00:15:51.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.693+0000 7f8466ffd640 1 -- 172.21.15.33:0/104048539 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f846c009c30 con 0x7f8470101de0 2024-09-16T00:15:51.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.693+0000 7f8477fa0640 1 -- 172.21.15.33:0/104048539 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f847019d7f0 con 0x7f8470101de0 2024-09-16T00:15:51.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.693+0000 7f8466ffd640 1 -- 172.21.15.33:0/104048539 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c00f6b0 con 0x7f8470101de0 2024-09-16T00:15:51.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.693+0000 7f8477fa0640 1 -- 172.21.15.33:0/104048539 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84701a50d0 con 0x7f8470101de0 2024-09-16T00:15:51.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.694+0000 7f8477fa0640 1 -- 172.21.15.33:0/104048539 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84701021c0 con 0x7f8470101de0 2024-09-16T00:15:51.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.695+0000 7f8466ffd640 1 -- 172.21.15.33:0/104048539 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f846c0109d0 con 0x7f8470101de0 2024-09-16T00:15:51.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.696+0000 7f8466ffd640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f845c0a32c0 0x7f845c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:51.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.696+0000 7f847553b640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f845c0a32c0 0x7f845c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:51.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.696+0000 7f8466ffd640 1 -- 172.21.15.33:0/104048539 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f846c0cf7c0 con 0x7f8470101de0 2024-09-16T00:15:51.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.696+0000 7f847553b640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f845c0a32c0 0x7f845c0a5780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f84701087b0 tx=0x7f8460008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:51.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.699+0000 7f8466ffd640 1 -- 172.21.15.33:0/104048539 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f846c009de0 con 0x7f8470101de0 2024-09-16T00:15:51.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:51.911+0000 7f8477fa0640 1 -- 172.21.15.33:0/104048539 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f847010a150 con 0x7f8470101de0 2024-09-16T00:15:52.175 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:15:51 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b[65594]: 2024-09-16T00:15:51.748+0000 7fe392696640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-16T00:15:52.176 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:15:51 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b[65594]: 2024-09-16T00:15:51.748+0000 7fe392696640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-16T00:15:52.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.179+0000 7f8466ffd640 1 -- 172.21.15.33:0/104048539 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v25) ==== 115+0+0 (secure 0 0 0) 0x7f846c09c320 con 0x7f8470101de0 2024-09-16T00:15:52.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.182+0000 7f8477fa0640 1 -- 172.21.15.33:0/104048539 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f845c0a32c0 msgr2=0x7f845c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:52.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.182+0000 7f8477fa0640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f845c0a32c0 0x7f845c0a5780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f84701087b0 tx=0x7f8460008040 comp rx=0 tx=0).stop 2024-09-16T00:15:52.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.182+0000 7f8477fa0640 1 -- 172.21.15.33:0/104048539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8470101de0 msgr2=0x7f8470198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:52.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.182+0000 7f8477fa0640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8470101de0 0x7f8470198550 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f846c00d950 tx=0x7f846c00de10 comp rx=0 tx=0).stop 2024-09-16T00:15:52.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.182+0000 7f8477fa0640 1 -- 172.21.15.33:0/104048539 shutdown_connections 2024-09-16T00:15:52.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.182+0000 7f8477fa0640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f845c0a32c0 0x7f845c0a5780 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:52.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.182+0000 7f8477fa0640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8470106f10 0x7f847019ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:52.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.182+0000 7f8477fa0640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8470102790 0x7f8470198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:52.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.182+0000 7f8477fa0640 1 --2- 172.21.15.33:0/104048539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8470101de0 0x7f8470198550 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:52.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.182+0000 7f8477fa0640 1 -- 172.21.15.33:0/104048539 >> 172.21.15.33:0/104048539 conn(0x7f84700fd8c0 msgr2=0x7f84700fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:52.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.182+0000 7f8477fa0640 1 -- 172.21.15.33:0/104048539 shutdown_connections 2024-09-16T00:15:52.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:52.182+0000 7f8477fa0640 1 -- 172.21.15.33:0/104048539 wait complete. 2024-09-16T00:15:52.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:52 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/104048539' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-16T00:15:52.553 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:52 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/104048539' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-16T00:15:52.570 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:52 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/104048539' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-16T00:15:52.936 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-09-16T00:15:52.937 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-09-16T00:15:53.198 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:53.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:53 smithi033 ceph-mon[31243]: pgmap v308: 129 pgs: 129 active+clean; 583 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-16T00:15:53.457 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:53 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/104048539' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-16T00:15:53.457 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:53 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:15:53.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:53 smithi060 ceph-mon[35185]: pgmap v308: 129 pgs: 129 active+clean; 583 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-16T00:15:53.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:53 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/104048539' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-16T00:15:53.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:53 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:15:53.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:53 smithi083 ceph-mon[35242]: pgmap v308: 129 pgs: 129 active+clean; 583 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-16T00:15:53.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:53 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/104048539' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-16T00:15:53.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:53 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:15:53.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.677+0000 7ff9b704c640 1 -- 172.21.15.33:0/2754556993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9b0100500 msgr2=0x7ff9b010ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:53.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.677+0000 7ff9b704c640 1 --2- 172.21.15.33:0/2754556993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9b0100500 0x7ff9b010ea20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009ca0 tx=0x7ff9a802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:15:53.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.677+0000 7ff9b704c640 1 -- 172.21.15.33:0/2754556993 shutdown_connections 2024-09-16T00:15:53.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.677+0000 7ff9b704c640 1 --2- 172.21.15.33:0/2754556993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9b0100500 0x7ff9b010ea20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:53.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.677+0000 7ff9b704c640 1 --2- 172.21.15.33:0/2754556993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9b00ffb60 0x7ff9b00fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:53.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.677+0000 7ff9b704c640 1 --2- 172.21.15.33:0/2754556993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b0106b80 0x7ff9b0106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:53.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.677+0000 7ff9b704c640 1 -- 172.21.15.33:0/2754556993 >> 172.21.15.33:0/2754556993 conn(0x7ff9b00fba90 msgr2=0x7ff9b00fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:53.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.677+0000 7ff9b704c640 1 -- 172.21.15.33:0/2754556993 shutdown_connections 2024-09-16T00:15:53.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.677+0000 7ff9b704c640 1 -- 172.21.15.33:0/2754556993 wait complete. 2024-09-16T00:15:53.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9b704c640 1 Processor -- start 2024-09-16T00:15:53.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9b704c640 1 -- start start 2024-09-16T00:15:53.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9b704c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b00ffb60 0x7ff9b01983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:53.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9b704c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9b0100500 0x7ff9b01988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:53.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9b704c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9b0106b80 0x7ff9b019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:53.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9b704c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b0111fa0 con 0x7ff9b0106b80 2024-09-16T00:15:53.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9b704c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b0111e20 con 0x7ff9b0100500 2024-09-16T00:15:53.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9b704c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b0112120 con 0x7ff9b00ffb60 2024-09-16T00:15:53.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9b55e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9b0106b80 0x7ff9b019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:53.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9b55e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9b0106b80 0x7ff9b019cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51334/0 (socket says 172.21.15.33:51334) 2024-09-16T00:15:53.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9b55e9640 1 -- 172.21.15.33:0/1383610113 learned_addr learned my addr 172.21.15.33:0/1383610113 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:53.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9b4de8640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b00ffb60 0x7ff9b01983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:53.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.678+0000 7ff9affff640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9b0100500 0x7ff9b01988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:53.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.679+0000 7ff9b55e9640 1 -- 172.21.15.33:0/1383610113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b00ffb60 msgr2=0x7ff9b01983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:53.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.679+0000 7ff9b55e9640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b00ffb60 0x7ff9b01983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:53.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.679+0000 7ff9b55e9640 1 -- 172.21.15.33:0/1383610113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9b0100500 msgr2=0x7ff9b01988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:53.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.679+0000 7ff9b55e9640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9b0100500 0x7ff9b01988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:53.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.679+0000 7ff9b55e9640 1 -- 172.21.15.33:0/1383610113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b019d350 con 0x7ff9b0106b80 2024-09-16T00:15:53.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.679+0000 7ff9affff640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9b0100500 0x7ff9b01988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:15:53.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.679+0000 7ff9b55e9640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9b0106b80 0x7ff9b019cc70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8003f40 tx=0x7ff9a8003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:53.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.679+0000 7ff9adffb640 1 -- 172.21.15.33:0/1383610113 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8031790 con 0x7ff9b0106b80 2024-09-16T00:15:53.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.679+0000 7ff9adffb640 1 -- 172.21.15.33:0/1383610113 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9a8031930 con 0x7ff9b0106b80 2024-09-16T00:15:53.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.679+0000 7ff9b704c640 1 -- 172.21.15.33:0/1383610113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b019d5e0 con 0x7ff9b0106b80 2024-09-16T00:15:53.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.679+0000 7ff9adffb640 1 -- 172.21.15.33:0/1383610113 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8030bf0 con 0x7ff9b0106b80 2024-09-16T00:15:53.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.679+0000 7ff9b704c640 1 -- 172.21.15.33:0/1383610113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b01a4f20 con 0x7ff9b0106b80 2024-09-16T00:15:53.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.680+0000 7ff9b704c640 1 -- 172.21.15.33:0/1383610113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9b00fffc0 con 0x7ff9b0106b80 2024-09-16T00:15:53.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.681+0000 7ff9adffb640 1 -- 172.21.15.33:0/1383610113 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff9a8004040 con 0x7ff9b0106b80 2024-09-16T00:15:53.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.682+0000 7ff9adffb640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9980a3240 0x7ff9980a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:53.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.683+0000 7ff9b4de8640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9980a3240 0x7ff9980a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:53.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.683+0000 7ff9adffb640 1 -- 172.21.15.33:0/1383610113 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7ff9a80f10f0 con 0x7ff9b0106b80 2024-09-16T00:15:53.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.683+0000 7ff9b4de8640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9980a3240 0x7ff9980a5700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0005fd0 tx=0x7ff9a0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:53.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.687+0000 7ff9adffb640 1 -- 172.21.15.33:0/1383610113 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff9a80bdc60 con 0x7ff9b0106b80 2024-09-16T00:15:53.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.888+0000 7ff9b704c640 1 -- 172.21.15.33:0/1383610113 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff9b00fc9f0 con 0x7ff9980a3240 2024-09-16T00:15:53.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.913+0000 7ff9adffb640 1 -- 172.21.15.33:0/1383610113 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff9b00fc9f0 con 0x7ff9980a3240 2024-09-16T00:15:53.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.915+0000 7ff9b704c640 1 -- 172.21.15.33:0/1383610113 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9980a3240 msgr2=0x7ff9980a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:53.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.915+0000 7ff9b704c640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9980a3240 0x7ff9980a5700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0005fd0 tx=0x7ff9a0005850 comp rx=0 tx=0).stop 2024-09-16T00:15:53.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.915+0000 7ff9b704c640 1 -- 172.21.15.33:0/1383610113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9b0106b80 msgr2=0x7ff9b019cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:53.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.915+0000 7ff9b704c640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9b0106b80 0x7ff9b019cc70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8003f40 tx=0x7ff9a8003f70 comp rx=0 tx=0).stop 2024-09-16T00:15:53.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.916+0000 7ff9b704c640 1 -- 172.21.15.33:0/1383610113 shutdown_connections 2024-09-16T00:15:53.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.916+0000 7ff9b704c640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9980a3240 0x7ff9980a5700 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:53.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.916+0000 7ff9b704c640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9b0106b80 0x7ff9b019cc70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:53.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.916+0000 7ff9b704c640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9b0100500 0x7ff9b01988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:53.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.916+0000 7ff9b704c640 1 --2- 172.21.15.33:0/1383610113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b00ffb60 0x7ff9b01983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:53.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.916+0000 7ff9b704c640 1 -- 172.21.15.33:0/1383610113 >> 172.21.15.33:0/1383610113 conn(0x7ff9b00fba90 msgr2=0x7ff9b00fc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:53.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.916+0000 7ff9b704c640 1 -- 172.21.15.33:0/1383610113 shutdown_connections 2024-09-16T00:15:53.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:53.916+0000 7ff9b704c640 1 -- 172.21.15.33:0/1383610113 wait complete. 2024-09-16T00:15:54.330 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:15:53 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:15:53.893+0000 7f75277f9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T00:15:54.330 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:15:53 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:15:53.893+0000 7f75277f9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T00:15:54.331 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:15:53 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:15:53.893+0000 7f75277f9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T00:15:54.331 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:15:53 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:15:53.893+0000 7f75277f9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T00:15:54.331 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:15:53 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:15:53.893+0000 7f75277f9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T00:15:54.628 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-09-16T00:15:54.887 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:55.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:55 smithi033 ceph-mon[31243]: pgmap v309: 129 pgs: 129 active+clean; 584 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-09-16T00:15:55.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:55 smithi033 ceph-mon[31243]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:15:55.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.391+0000 7f355d59e640 1 -- 172.21.15.33:0/1471041610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35581034b0 msgr2=0x7f355810b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:55.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.391+0000 7f355d59e640 1 --2- 172.21.15.33:0/1471041610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35581034b0 0x7f355810b9a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3540009900 tx=0x7f354002a480 comp rx=0 tx=0).stop 2024-09-16T00:15:55.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.391+0000 7f355d59e640 1 -- 172.21.15.33:0/1471041610 shutdown_connections 2024-09-16T00:15:55.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.391+0000 7f355d59e640 1 --2- 172.21.15.33:0/1471041610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f355810bfc0 0x7f355810e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:55.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.391+0000 7f355d59e640 1 --2- 172.21.15.33:0/1471041610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35581034b0 0x7f355810b9a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:55.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.391+0000 7f355d59e640 1 --2- 172.21.15.33:0/1471041610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3558102b90 0x7f3558102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:55.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.391+0000 7f355d59e640 1 -- 172.21.15.33:0/1471041610 >> 172.21.15.33:0/1471041610 conn(0x7f35580fc630 msgr2=0x7f35580fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:55.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.391+0000 7f355d59e640 1 -- 172.21.15.33:0/1471041610 shutdown_connections 2024-09-16T00:15:55.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.391+0000 7f355d59e640 1 -- 172.21.15.33:0/1471041610 wait complete. 2024-09-16T00:15:55.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f355d59e640 1 Processor -- start 2024-09-16T00:15:55.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f355d59e640 1 -- start start 2024-09-16T00:15:55.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f355d59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3558102b90 0x7f35581982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:55.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f355d59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35581034b0 0x7f3558198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:55.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f355d59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f355810bfc0 0x7f355819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:55.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f355d59e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3558111dc0 con 0x7f3558102b90 2024-09-16T00:15:55.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f355d59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3558111c40 con 0x7f35581034b0 2024-09-16T00:15:55.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f355d59e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3558111f40 con 0x7f355810bfc0 2024-09-16T00:15:55.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f3557fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3558102b90 0x7f35581982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:55.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f3557fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3558102b90 0x7f35581982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51364/0 (socket says 172.21.15.33:51364) 2024-09-16T00:15:55.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f3557fff640 1 -- 172.21.15.33:0/2952128317 learned_addr learned my addr 172.21.15.33:0/2952128317 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:55.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f354ffff640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35581034b0 0x7f3558198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:55.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.392+0000 7f355cd9d640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f355810bfc0 0x7f355819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:55.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.393+0000 7f3557fff640 1 -- 172.21.15.33:0/2952128317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f355810bfc0 msgr2=0x7f355819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:55.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.393+0000 7f3557fff640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f355810bfc0 0x7f355819cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:55.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.393+0000 7f3557fff640 1 -- 172.21.15.33:0/2952128317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35581034b0 msgr2=0x7f3558198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:55.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.393+0000 7f3557fff640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35581034b0 0x7f3558198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:55.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.393+0000 7f3557fff640 1 -- 172.21.15.33:0/2952128317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355819d290 con 0x7f3558102b90 2024-09-16T00:15:55.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.393+0000 7f355cd9d640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f355810bfc0 0x7f355819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:15:55.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.393+0000 7f3557fff640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3558102b90 0x7f35581982e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f354800d950 tx=0x7f354800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:55.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.393+0000 7f3555ffb640 1 -- 172.21.15.33:0/2952128317 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3548017070 con 0x7f3558102b90 2024-09-16T00:15:55.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.393+0000 7f355d59e640 1 -- 172.21.15.33:0/2952128317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f355819d580 con 0x7f3558102b90 2024-09-16T00:15:55.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.393+0000 7f3555ffb640 1 -- 172.21.15.33:0/2952128317 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3548009c30 con 0x7f3558102b90 2024-09-16T00:15:55.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.393+0000 7f3555ffb640 1 -- 172.21.15.33:0/2952128317 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354800f6b0 con 0x7f3558102b90 2024-09-16T00:15:55.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.393+0000 7f355d59e640 1 -- 172.21.15.33:0/2952128317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35581a4e60 con 0x7f3558102b90 2024-09-16T00:15:55.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.394+0000 7f355d59e640 1 -- 172.21.15.33:0/2952128317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3558102f70 con 0x7f3558102b90 2024-09-16T00:15:55.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.395+0000 7f3555ffb640 1 -- 172.21.15.33:0/2952128317 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f35480109d0 con 0x7f3558102b90 2024-09-16T00:15:55.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.396+0000 7f3555ffb640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35440a3240 0x7f35440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:55.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.396+0000 7f354ffff640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35440a3240 0x7f35440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:55.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.396+0000 7f3555ffb640 1 -- 172.21.15.33:0/2952128317 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f35480cf840 con 0x7f3558102b90 2024-09-16T00:15:55.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.396+0000 7f354ffff640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35440a3240 0x7f35440a5700 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3540009230 tx=0x7f35400058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:55.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.400+0000 7f3555ffb640 1 -- 172.21.15.33:0/2952128317 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3548009de0 con 0x7f3558102b90 2024-09-16T00:15:55.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:55 smithi060 ceph-mon[35185]: pgmap v309: 129 pgs: 129 active+clean; 584 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-09-16T00:15:55.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:55 smithi060 ceph-mon[35185]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:15:55.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.602+0000 7f355d59e640 1 -- 172.21.15.33:0/2952128317 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3558062630 con 0x7f35440a3240 2024-09-16T00:15:55.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.627+0000 7f3555ffb640 1 -- 172.21.15.33:0/2952128317 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3558062630 con 0x7f35440a3240 2024-09-16T00:15:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.629+0000 7f355d59e640 1 -- 172.21.15.33:0/2952128317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35440a3240 msgr2=0x7f35440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.629+0000 7f355d59e640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35440a3240 0x7f35440a5700 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3540009230 tx=0x7f35400058d0 comp rx=0 tx=0).stop 2024-09-16T00:15:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.629+0000 7f355d59e640 1 -- 172.21.15.33:0/2952128317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3558102b90 msgr2=0x7f35581982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.629+0000 7f355d59e640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3558102b90 0x7f35581982e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f354800d950 tx=0x7f354800de10 comp rx=0 tx=0).stop 2024-09-16T00:15:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.629+0000 7f355d59e640 1 -- 172.21.15.33:0/2952128317 shutdown_connections 2024-09-16T00:15:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.629+0000 7f355d59e640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35440a3240 0x7f35440a5700 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.629+0000 7f355d59e640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f355810bfc0 0x7f355819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.629+0000 7f355d59e640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35581034b0 0x7f3558198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.629+0000 7f355d59e640 1 --2- 172.21.15.33:0/2952128317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3558102b90 0x7f35581982e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.629+0000 7f355d59e640 1 -- 172.21.15.33:0/2952128317 >> 172.21.15.33:0/2952128317 conn(0x7f35580fc630 msgr2=0x7f35580fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.630+0000 7f355d59e640 1 -- 172.21.15.33:0/2952128317 shutdown_connections 2024-09-16T00:15:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:55.630+0000 7f355d59e640 1 -- 172.21.15.33:0/2952128317 wait complete. 2024-09-16T00:15:55.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:55 smithi083 ceph-mon[35242]: pgmap v309: 129 pgs: 129 active+clean; 584 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-09-16T00:15:55.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:55 smithi083 ceph-mon[35242]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:15:56.296 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-09-16T00:15:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:57.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.004+0000 7fd234a25640 1 -- 172.21.15.33:0/4094228778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd230102570 msgr2=0x7fd2301029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:57.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.004+0000 7fd234a25640 1 --2- 172.21.15.33:0/4094228778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd230102570 0x7fd2301029b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd224009900 tx=0x7fd22402a480 comp rx=0 tx=0).stop 2024-09-16T00:15:57.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.004+0000 7fd234a25640 1 -- 172.21.15.33:0/4094228778 shutdown_connections 2024-09-16T00:15:57.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.004+0000 7fd234a25640 1 --2- 172.21.15.33:0/4094228778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd230102ef0 0x7fd23010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:57.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.004+0000 7fd234a25640 1 --2- 172.21.15.33:0/4094228778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd230102570 0x7fd2301029b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.004+0000 7fd234a25640 1 --2- 172.21.15.33:0/4094228778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd230100e20 0x7fd230101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.004+0000 7fd234a25640 1 -- 172.21.15.33:0/4094228778 >> 172.21.15.33:0/4094228778 conn(0x7fd2300fc7d0 msgr2=0x7fd2300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.004+0000 7fd234a25640 1 -- 172.21.15.33:0/4094228778 shutdown_connections 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.004+0000 7fd234a25640 1 -- 172.21.15.33:0/4094228778 wait complete. 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd234a25640 1 Processor -- start 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd234a25640 1 -- start start 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd234a25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd230100e20 0x7fd23019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd234a25640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd230102570 0x7fd23019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd234a25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd230102ef0 0x7fd23019eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd234a25640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd230110f80 con 0x7fd230102570 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd234a25640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd230110e00 con 0x7fd230100e20 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd234a25640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd230111100 con 0x7fd230102ef0 2024-09-16T00:15:57.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd22dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd230102570 0x7fd23019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd22dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd230102570 0x7fd23019ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51380/0 (socket says 172.21.15.33:51380) 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd22dd9b640 1 -- 172.21.15.33:0/1876048809 learned_addr learned my addr 172.21.15.33:0/1876048809 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd22ed9d640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd230102ef0 0x7fd23019eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd22e59c640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd230100e20 0x7fd23019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd22dd9b640 1 -- 172.21.15.33:0/1876048809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd230102ef0 msgr2=0x7fd23019eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd22dd9b640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd230102ef0 0x7fd23019eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd22dd9b640 1 -- 172.21.15.33:0/1876048809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd230100e20 msgr2=0x7fd23019a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd22dd9b640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd230100e20 0x7fd23019a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.005+0000 7fd22dd9b640 1 -- 172.21.15.33:0/1876048809 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd23019f5b0 con 0x7fd230102570 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.006+0000 7fd22dd9b640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd230102570 0x7fd23019ab40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd224006010 tx=0x7fd224003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.006+0000 7fd21f7fe640 1 -- 172.21.15.33:0/1876048809 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22403f070 con 0x7fd230102570 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.006+0000 7fd234a25640 1 -- 172.21.15.33:0/1876048809 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd23019f840 con 0x7fd230102570 2024-09-16T00:15:57.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.006+0000 7fd21f7fe640 1 -- 172.21.15.33:0/1876048809 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd224031e30 con 0x7fd230102570 2024-09-16T00:15:57.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.006+0000 7fd21f7fe640 1 -- 172.21.15.33:0/1876048809 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd224035030 con 0x7fd230102570 2024-09-16T00:15:57.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.006+0000 7fd234a25640 1 -- 172.21.15.33:0/1876048809 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2301a7180 con 0x7fd230102570 2024-09-16T00:15:57.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.007+0000 7fd234a25640 1 -- 172.21.15.33:0/1876048809 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd230101200 con 0x7fd230102570 2024-09-16T00:15:57.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.008+0000 7fd21f7fe640 1 -- 172.21.15.33:0/1876048809 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fd224042070 con 0x7fd230102570 2024-09-16T00:15:57.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.009+0000 7fd21f7fe640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2180a32d0 0x7fd2180a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:57.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.009+0000 7fd22e59c640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2180a32d0 0x7fd2180a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:57.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.009+0000 7fd21f7fe640 1 -- 172.21.15.33:0/1876048809 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fd2240f0c20 con 0x7fd230102570 2024-09-16T00:15:57.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.010+0000 7fd22e59c640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2180a32d0 0x7fd2180a5790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd220005e10 tx=0x7fd220005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:57.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.012+0000 7fd21f7fe640 1 -- 172.21.15.33:0/1876048809 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd2240bd630 con 0x7fd230102570 2024-09-16T00:15:57.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.216+0000 7fd234a25640 1 -- 172.21.15.33:0/1876048809 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fd23010cda0 con 0x7fd2180a32d0 2024-09-16T00:15:57.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.221+0000 7fd21f7fe640 1 -- 172.21.15.33:0/1876048809 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fd23010cda0 con 0x7fd2180a32d0 2024-09-16T00:15:57.222 INFO:teuthology.orchestra.run.smithi033.stdout:/volumes/qa/sv_0/ebf5e943-a091-4cf5-858d-ac2d9027b49b 2024-09-16T00:15:57.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.223+0000 7fd234a25640 1 -- 172.21.15.33:0/1876048809 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2180a32d0 msgr2=0x7fd2180a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.223+0000 7fd234a25640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2180a32d0 0x7fd2180a5790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd220005e10 tx=0x7fd220005da0 comp rx=0 tx=0).stop 2024-09-16T00:15:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.223+0000 7fd234a25640 1 -- 172.21.15.33:0/1876048809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd230102570 msgr2=0x7fd23019ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.223+0000 7fd234a25640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd230102570 0x7fd23019ab40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd224006010 tx=0x7fd224003c80 comp rx=0 tx=0).stop 2024-09-16T00:15:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.223+0000 7fd234a25640 1 -- 172.21.15.33:0/1876048809 shutdown_connections 2024-09-16T00:15:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.223+0000 7fd234a25640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2180a32d0 0x7fd2180a5790 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.223+0000 7fd234a25640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd230102ef0 0x7fd23019eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.223+0000 7fd234a25640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd230102570 0x7fd23019ab40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.223+0000 7fd234a25640 1 --2- 172.21.15.33:0/1876048809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd230100e20 0x7fd23019a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.223+0000 7fd234a25640 1 -- 172.21.15.33:0/1876048809 >> 172.21.15.33:0/1876048809 conn(0x7fd2300fc7d0 msgr2=0x7fd23006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.223+0000 7fd234a25640 1 -- 172.21.15.33:0/1876048809 shutdown_connections 2024-09-16T00:15:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:57.223+0000 7fd234a25640 1 -- 172.21.15.33:0/1876048809 wait complete. 2024-09-16T00:15:57.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:57 smithi033 ceph-mon[31243]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:15:57.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:57 smithi033 ceph-mon[31243]: pgmap v310: 129 pgs: 129 active+clean; 584 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-16T00:15:57.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:57 smithi033 ceph-mon[31243]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-16T00:15:57.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:57 smithi060 ceph-mon[35185]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:15:57.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:57 smithi060 ceph-mon[35185]: pgmap v310: 129 pgs: 129 active+clean; 584 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-16T00:15:57.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:57 smithi060 ceph-mon[35185]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-16T00:15:57.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:57 smithi083 ceph-mon[35242]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:15:57.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:57 smithi083 ceph-mon[35242]: pgmap v310: 129 pgs: 129 active+clean; 584 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-16T00:15:57.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:57 smithi083 ceph-mon[35242]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-16T00:15:57.862 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-09-16T00:15:58.116 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:15:58.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:58 smithi060 ceph-mon[35185]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:15:58.495 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:58 smithi033 ceph-mon[31243]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:15:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.581+0000 7fee8d503640 1 -- 172.21.15.33:0/1770952952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee88102570 msgr2=0x7fee881029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.581+0000 7fee8d503640 1 --2- 172.21.15.33:0/1770952952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee88102570 0x7fee881029b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fee70009900 tx=0x7fee7002a480 comp rx=0 tx=0).stop 2024-09-16T00:15:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.581+0000 7fee8d503640 1 -- 172.21.15.33:0/1770952952 shutdown_connections 2024-09-16T00:15:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.581+0000 7fee8d503640 1 --2- 172.21.15.33:0/1770952952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88102ef0 0x7fee8810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.581+0000 7fee8d503640 1 --2- 172.21.15.33:0/1770952952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee88102570 0x7fee881029b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.581+0000 7fee8d503640 1 --2- 172.21.15.33:0/1770952952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee88100e20 0x7fee88101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.581+0000 7fee8d503640 1 -- 172.21.15.33:0/1770952952 >> 172.21.15.33:0/1770952952 conn(0x7fee880fc7d0 msgr2=0x7fee880febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.581+0000 7fee8d503640 1 -- 172.21.15.33:0/1770952952 shutdown_connections 2024-09-16T00:15:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.581+0000 7fee8d503640 1 -- 172.21.15.33:0/1770952952 wait complete. 2024-09-16T00:15:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.581+0000 7fee8d503640 1 Processor -- start 2024-09-16T00:15:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee8d503640 1 -- start start 2024-09-16T00:15:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee8d503640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee88100e20 0x7fee88199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee8d503640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee88102570 0x7fee88199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee8d503640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88102ef0 0x7fee881a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee8d503640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fee88111020 con 0x7fee88102570 2024-09-16T00:15:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee8d503640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fee88110ea0 con 0x7fee88102ef0 2024-09-16T00:15:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee8d503640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee881111a0 con 0x7fee88100e20 2024-09-16T00:15:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee88102570 0x7fee88199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee88102570 0x7fee88199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51398/0 (socket says 172.21.15.33:51398) 2024-09-16T00:15:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee867fc640 1 -- 172.21.15.33:0/1805487333 learned_addr learned my addr 172.21.15.33:0/1805487333 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:15:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee86ffd640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee88100e20 0x7fee88199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee877fe640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88102ef0 0x7fee881a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee867fc640 1 -- 172.21.15.33:0/1805487333 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee88100e20 msgr2=0x7fee88199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee867fc640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee88100e20 0x7fee88199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee867fc640 1 -- 172.21.15.33:0/1805487333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88102ef0 msgr2=0x7fee881a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee867fc640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88102ef0 0x7fee881a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee867fc640 1 -- 172.21.15.33:0/1805487333 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee881a4ad0 con 0x7fee88102570 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.582+0000 7fee867fc640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee88102570 0x7fee88199a50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fee70006010 tx=0x7fee70003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.583+0000 7fee7ffff640 1 -- 172.21.15.33:0/1805487333 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7003f070 con 0x7fee88102570 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.583+0000 7fee7ffff640 1 -- 172.21.15.33:0/1805487333 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee70031e30 con 0x7fee88102570 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.583+0000 7fee8d503640 1 -- 172.21.15.33:0/1805487333 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee881a4ca0 con 0x7fee88102570 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.583+0000 7fee7ffff640 1 -- 172.21.15.33:0/1805487333 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70035030 con 0x7fee88102570 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.583+0000 7fee8d503640 1 -- 172.21.15.33:0/1805487333 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee881a50f0 con 0x7fee88102570 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.584+0000 7fee8d503640 1 -- 172.21.15.33:0/1805487333 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee88101280 con 0x7fee88102570 2024-09-16T00:15:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.585+0000 7fee7ffff640 1 -- 172.21.15.33:0/1805487333 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fee70042070 con 0x7fee88102570 2024-09-16T00:15:58.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.585+0000 7fee7ffff640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee740a32d0 0x7fee740a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:15:58.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.586+0000 7fee86ffd640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee740a32d0 0x7fee740a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:15:58.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.586+0000 7fee7ffff640 1 -- 172.21.15.33:0/1805487333 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fee700f0690 con 0x7fee88102570 2024-09-16T00:15:58.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.586+0000 7fee86ffd640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee740a32d0 0x7fee740a5790 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fee7800a400 tx=0x7fee7800d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:15:58.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.590+0000 7fee7ffff640 1 -- 172.21.15.33:0/1805487333 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fee700bcfb0 con 0x7fee88102570 2024-09-16T00:15:58.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:58 smithi083 ceph-mon[35242]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:15:58.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.792+0000 7fee8d503640 1 -- 172.21.15.33:0/1805487333 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fee88109100 con 0x7fee740a32d0 2024-09-16T00:15:58.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.816+0000 7fee7ffff640 1 -- 172.21.15.33:0/1805487333 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fee78009c30 con 0x7fee740a32d0 2024-09-16T00:15:58.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.818+0000 7fee8d503640 1 -- 172.21.15.33:0/1805487333 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee740a32d0 msgr2=0x7fee740a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:58.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.818+0000 7fee8d503640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee740a32d0 0x7fee740a5790 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fee7800a400 tx=0x7fee7800d040 comp rx=0 tx=0).stop 2024-09-16T00:15:58.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.818+0000 7fee8d503640 1 -- 172.21.15.33:0/1805487333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee88102570 msgr2=0x7fee88199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:15:58.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.818+0000 7fee8d503640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee88102570 0x7fee88199a50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fee70006010 tx=0x7fee70003c80 comp rx=0 tx=0).stop 2024-09-16T00:15:58.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.818+0000 7fee8d503640 1 -- 172.21.15.33:0/1805487333 shutdown_connections 2024-09-16T00:15:58.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.819+0000 7fee8d503640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee740a32d0 0x7fee740a5790 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:58.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.819+0000 7fee8d503640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88102ef0 0x7fee881a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.819+0000 7fee8d503640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee88102570 0x7fee88199a50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.819+0000 7fee8d503640 1 --2- 172.21.15.33:0/1805487333 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee88100e20 0x7fee88199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:15:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.819+0000 7fee8d503640 1 -- 172.21.15.33:0/1805487333 >> 172.21.15.33:0/1805487333 conn(0x7fee880fc7d0 msgr2=0x7fee880febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:15:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.819+0000 7fee8d503640 1 -- 172.21.15.33:0/1805487333 shutdown_connections 2024-09-16T00:15:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:15:58.819+0000 7fee8d503640 1 -- 172.21.15.33:0/1805487333 wait complete. 2024-09-16T00:15:59.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:15:59 smithi060 ceph-mon[35185]: pgmap v311: 129 pgs: 129 active+clean; 584 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-16T00:15:59.517 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:15:59 smithi033 ceph-mon[31243]: pgmap v311: 129 pgs: 129 active+clean; 584 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-16T00:15:59.552 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-09-16T00:15:59.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:15:59 smithi083 ceph-mon[35242]: pgmap v311: 129 pgs: 129 active+clean; 584 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-16T00:15:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:00.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.277+0000 7fdc831cc640 1 -- 172.21.15.33:0/2773971828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7c106f00 msgr2=0x7fdc7c10ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:00.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.277+0000 7fdc831cc640 1 --2- 172.21.15.33:0/2773971828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7c106f00 0x7fdc7c10ea80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009ca0 tx=0x7fdc7402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:16:00.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.278+0000 7fdc831cc640 1 -- 172.21.15.33:0/2773971828 shutdown_connections 2024-09-16T00:16:00.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.278+0000 7fdc831cc640 1 --2- 172.21.15.33:0/2773971828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7c106f00 0x7fdc7c10ea80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:00.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.278+0000 7fdc831cc640 1 --2- 172.21.15.33:0/2773971828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc7c102780 0x7fdc7c1067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:00.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.278+0000 7fdc831cc640 1 --2- 172.21.15.33:0/2773971828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc7c101dd0 0x7fdc7c1021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.278+0000 7fdc831cc640 1 -- 172.21.15.33:0/2773971828 >> 172.21.15.33:0/2773971828 conn(0x7fdc7c0fd8b0 msgr2=0x7fdc7c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.278+0000 7fdc831cc640 1 -- 172.21.15.33:0/2773971828 shutdown_connections 2024-09-16T00:16:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.278+0000 7fdc831cc640 1 -- 172.21.15.33:0/2773971828 wait complete. 2024-09-16T00:16:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.278+0000 7fdc831cc640 1 Processor -- start 2024-09-16T00:16:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.278+0000 7fdc831cc640 1 -- start start 2024-09-16T00:16:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc831cc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7c101dd0 0x7fdc7c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc831cc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc7c102780 0x7fdc7c1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc831cc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc7c106f00 0x7fdc7c19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc831cc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7c111f60 con 0x7fdc7c101dd0 2024-09-16T00:16:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc831cc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7c111de0 con 0x7fdc7c102780 2024-09-16T00:16:00.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc831cc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7c1120e0 con 0x7fdc7c106f00 2024-09-16T00:16:00.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc80f68640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7c101dd0 0x7fdc7c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:00.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc80f68640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7c101dd0 0x7fdc7c1984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40222/0 (socket says 172.21.15.33:40222) 2024-09-16T00:16:00.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc80f68640 1 -- 172.21.15.33:0/1856377925 learned_addr learned my addr 172.21.15.33:0/1856377925 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:00.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc7bfff640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc7c102780 0x7fdc7c1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:00.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc81769640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc7c106f00 0x7fdc7c19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:00.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc80f68640 1 -- 172.21.15.33:0/1856377925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc7c106f00 msgr2=0x7fdc7c19cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:00.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc80f68640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc7c106f00 0x7fdc7c19cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:00.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc80f68640 1 -- 172.21.15.33:0/1856377925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc7c102780 msgr2=0x7fdc7c1989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:00.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc80f68640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc7c102780 0x7fdc7c1989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:00.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc80f68640 1 -- 172.21.15.33:0/1856377925 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7c19d450 con 0x7fdc7c101dd0 2024-09-16T00:16:00.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc80f68640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7c101dd0 0x7fdc7c1984a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000e9f0 tx=0x7fdc7000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:00.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc79ffb640 1 -- 172.21.15.33:0/1856377925 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7000cd80 con 0x7fdc7c101dd0 2024-09-16T00:16:00.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc79ffb640 1 -- 172.21.15.33:0/1856377925 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc70010910 con 0x7fdc7c101dd0 2024-09-16T00:16:00.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc831cc640 1 -- 172.21.15.33:0/1856377925 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc7c19d740 con 0x7fdc7c101dd0 2024-09-16T00:16:00.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.279+0000 7fdc79ffb640 1 -- 172.21.15.33:0/1856377925 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc70018990 con 0x7fdc7c101dd0 2024-09-16T00:16:00.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.280+0000 7fdc831cc640 1 -- 172.21.15.33:0/1856377925 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc7c1a5020 con 0x7fdc7c101dd0 2024-09-16T00:16:00.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.281+0000 7fdc831cc640 1 -- 172.21.15.33:0/1856377925 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc7c1021b0 con 0x7fdc7c101dd0 2024-09-16T00:16:00.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.282+0000 7fdc79ffb640 1 -- 172.21.15.33:0/1856377925 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fdc70010ac0 con 0x7fdc7c101dd0 2024-09-16T00:16:00.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.283+0000 7fdc79ffb640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdc6c0a3350 0x7fdc6c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:00.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.283+0000 7fdc7bfff640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdc6c0a3350 0x7fdc6c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:00.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.283+0000 7fdc79ffb640 1 -- 172.21.15.33:0/1856377925 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fdc700d0060 con 0x7fdc7c101dd0 2024-09-16T00:16:00.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.283+0000 7fdc7bfff640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdc6c0a3350 0x7fdc6c0a5810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c108790 tx=0x7fdc68005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:00.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.287+0000 7fdc79ffb640 1 -- 172.21.15.33:0/1856377925 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdc7009c980 con 0x7fdc7c101dd0 2024-09-16T00:16:00.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:00 smithi060 ceph-mon[35185]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:16:00.495 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:00 smithi033 ceph-mon[31243]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:16:00.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.493+0000 7fdc831cc640 1 -- 172.21.15.33:0/1856377925 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fdc7c062630 con 0x7fdc6c0a3350 2024-09-16T00:16:00.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.499+0000 7fdc79ffb640 1 -- 172.21.15.33:0/1856377925 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fdc7c062630 con 0x7fdc6c0a3350 2024-09-16T00:16:00.500 INFO:teuthology.orchestra.run.smithi033.stdout:/volumes/qa/sv_1/656da54f-0733-45b2-aea2-3ad93d5723b8 2024-09-16T00:16:00.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.502+0000 7fdc831cc640 1 -- 172.21.15.33:0/1856377925 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdc6c0a3350 msgr2=0x7fdc6c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:00.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.502+0000 7fdc831cc640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdc6c0a3350 0x7fdc6c0a5810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c108790 tx=0x7fdc68005f90 comp rx=0 tx=0).stop 2024-09-16T00:16:00.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.502+0000 7fdc831cc640 1 -- 172.21.15.33:0/1856377925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7c101dd0 msgr2=0x7fdc7c1984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:00.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.502+0000 7fdc831cc640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7c101dd0 0x7fdc7c1984a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000e9f0 tx=0x7fdc7000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:16:00.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.502+0000 7fdc831cc640 1 -- 172.21.15.33:0/1856377925 shutdown_connections 2024-09-16T00:16:00.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.502+0000 7fdc831cc640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdc6c0a3350 0x7fdc6c0a5810 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:00.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.502+0000 7fdc831cc640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc7c106f00 0x7fdc7c19cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:00.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.502+0000 7fdc831cc640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc7c102780 0x7fdc7c1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:00.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.502+0000 7fdc831cc640 1 --2- 172.21.15.33:0/1856377925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc7c101dd0 0x7fdc7c1984a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:00.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.502+0000 7fdc831cc640 1 -- 172.21.15.33:0/1856377925 >> 172.21.15.33:0/1856377925 conn(0x7fdc7c0fd8b0 msgr2=0x7fdc7c0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:00.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.502+0000 7fdc831cc640 1 -- 172.21.15.33:0/1856377925 shutdown_connections 2024-09-16T00:16:00.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:00.502+0000 7fdc831cc640 1 -- 172.21.15.33:0/1856377925 wait complete. 2024-09-16T00:16:00.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:00 smithi083 ceph-mon[35242]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:16:01.239 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:16:01.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:01 smithi060 ceph-mon[35185]: pgmap v312: 129 pgs: 129 active+clean; 609 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-09-16T00:16:01.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:01 smithi060 ceph-mon[35185]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:16:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:01.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:01 smithi033 ceph-mon[31243]: pgmap v312: 129 pgs: 129 active+clean; 609 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-09-16T00:16:01.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:01 smithi033 ceph-mon[31243]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:16:01.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:01 smithi083 ceph-mon[35242]: pgmap v312: 129 pgs: 129 active+clean; 609 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-09-16T00:16:01.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:01 smithi083 ceph-mon[35242]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:16:02.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.980+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/3662776820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b14102d40 msgr2=0x7f7b14103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:02.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.980+0000 7f7b1a9b6640 1 --2- 172.21.15.33:0/3662776820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b14102d40 0x7f7b14103120 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7b04009900 tx=0x7f7b0402a480 comp rx=0 tx=0).stop 2024-09-16T00:16:02.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.980+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/3662776820 shutdown_connections 2024-09-16T00:16:02.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.980+0000 7f7b1a9b6640 1 --2- 172.21.15.33:0/3662776820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1410c170 0x7f7b1410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:02.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.980+0000 7f7b1a9b6640 1 --2- 172.21.15.33:0/3662776820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b14103660 0x7f7b1410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:02.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.980+0000 7f7b1a9b6640 1 --2- 172.21.15.33:0/3662776820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b14102d40 0x7f7b14103120 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.980+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/3662776820 >> 172.21.15.33:0/3662776820 conn(0x7f7b140fc7a0 msgr2=0x7f7b140febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.980+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/3662776820 shutdown_connections 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.980+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/3662776820 wait complete. 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.981+0000 7f7b1a9b6640 1 Processor -- start 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.981+0000 7f7b1a9b6640 1 -- start start 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.981+0000 7f7b1a9b6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b14102d40 0x7f7b141984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.981+0000 7f7b1a9b6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b14103660 0x7f7b141989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.981+0000 7f7b1a9b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1410c170 0x7f7b1419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.981+0000 7f7b1a9b6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b14111fd0 con 0x7f7b14102d40 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.981+0000 7f7b1a9b6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7b14111e50 con 0x7f7b14103660 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.981+0000 7f7b1a9b6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7b14112150 con 0x7f7b1410c170 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.981+0000 7f7b13fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b14102d40 0x7f7b141984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:02.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.981+0000 7f7b13fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b14102d40 0x7f7b141984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40234/0 (socket says 172.21.15.33:40234) 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.981+0000 7f7b13fff640 1 -- 172.21.15.33:0/2394259740 learned_addr learned my addr 172.21.15.33:0/2394259740 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b137fe640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b14103660 0x7f7b141989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b18f53640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1410c170 0x7f7b1419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b13fff640 1 -- 172.21.15.33:0/2394259740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1410c170 msgr2=0x7f7b1419cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b13fff640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1410c170 0x7f7b1419cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b13fff640 1 -- 172.21.15.33:0/2394259740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b14103660 msgr2=0x7f7b141989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b13fff640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b14103660 0x7f7b141989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b13fff640 1 -- 172.21.15.33:0/2394259740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1419d450 con 0x7f7b14102d40 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b18f53640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1410c170 0x7f7b1419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b13fff640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b14102d40 0x7f7b141984a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7b04005ab0 tx=0x7f7b040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b117fa640 1 -- 172.21.15.33:0/2394259740 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0403f030 con 0x7f7b14102d40 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/2394259740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b1419d6e0 con 0x7f7b14102d40 2024-09-16T00:16:02.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b117fa640 1 -- 172.21.15.33:0/2394259740 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b04003ce0 con 0x7f7b14102d40 2024-09-16T00:16:02.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b117fa640 1 -- 172.21.15.33:0/2394259740 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b04035070 con 0x7f7b14102d40 2024-09-16T00:16:02.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.982+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/2394259740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b141a5020 con 0x7f7b14102d40 2024-09-16T00:16:02.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.983+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/2394259740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b14103120 con 0x7f7b14102d40 2024-09-16T00:16:02.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.985+0000 7f7b117fa640 1 -- 172.21.15.33:0/2394259740 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7b04042030 con 0x7f7b14102d40 2024-09-16T00:16:02.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.985+0000 7f7b117fa640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7afc0a32d0 0x7f7afc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:02.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.985+0000 7f7b117fa640 1 -- 172.21.15.33:0/2394259740 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f7b040f05d0 con 0x7f7b14102d40 2024-09-16T00:16:02.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.985+0000 7f7b137fe640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7afc0a32d0 0x7f7afc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:02.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.986+0000 7f7b137fe640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7afc0a32d0 0x7f7afc0a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7b140feda0 tx=0x7f7b08009060 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:02.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:01.989+0000 7f7b117fa640 1 -- 172.21.15.33:0/2394259740 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7b040bcf50 con 0x7f7b14102d40 2024-09-16T00:16:02.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.202+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/2394259740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b14107210 con 0x7f7b14102d40 2024-09-16T00:16:02.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.202+0000 7f7b117fa640 1 -- 172.21.15.33:0/2394259740 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7f7b040bc850 con 0x7f7b14102d40 2024-09-16T00:16:02.204 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:02.204 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"btime":"2024-09-16T00:15:52:174386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:15:51.912925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:16:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-09-16T00:16:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.205+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/2394259740 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7afc0a32d0 msgr2=0x7f7afc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.205+0000 7f7b1a9b6640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7afc0a32d0 0x7f7afc0a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7b140feda0 tx=0x7f7b08009060 comp rx=0 tx=0).stop 2024-09-16T00:16:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.205+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/2394259740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b14102d40 msgr2=0x7f7b141984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.205+0000 7f7b1a9b6640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b14102d40 0x7f7b141984a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7b04005ab0 tx=0x7f7b040042f0 comp rx=0 tx=0).stop 2024-09-16T00:16:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.205+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/2394259740 shutdown_connections 2024-09-16T00:16:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.205+0000 7f7b1a9b6640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7afc0a32d0 0x7f7afc0a5790 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.205+0000 7f7b1a9b6640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1410c170 0x7f7b1419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.205+0000 7f7b1a9b6640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b14103660 0x7f7b141989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.205+0000 7f7b1a9b6640 1 --2- 172.21.15.33:0/2394259740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b14102d40 0x7f7b141984a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.205+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/2394259740 >> 172.21.15.33:0/2394259740 conn(0x7f7b140fc7a0 msgr2=0x7f7b140fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.205+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/2394259740 shutdown_connections 2024-09-16T00:16:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:02.205+0000 7f7b1a9b6640 1 -- 172.21.15.33:0/2394259740 wait complete. 2024-09-16T00:16:02.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:02 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2394259740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:16:02.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:02 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2394259740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:16:02.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:02 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2394259740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:16:02.901 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:16:03.184 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:03.475 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:03 smithi033 ceph-mon[31243]: pgmap v313: 129 pgs: 129 active+clean; 609 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-09-16T00:16:03.476 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:03 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:16:03.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:03 smithi060 ceph-mon[35185]: pgmap v313: 129 pgs: 129 active+clean; 609 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-09-16T00:16:03.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:03 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:16:03.606 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:03 smithi083 ceph-mon[35242]: pgmap v313: 129 pgs: 129 active+clean; 609 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-09-16T00:16:03.607 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:03 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:16:03.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.638+0000 7fafb1dc3640 1 -- 172.21.15.33:0/996069255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafac100ca0 msgr2=0x7fafac106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:03.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.638+0000 7fafb1dc3640 1 --2- 172.21.15.33:0/996069255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafac100ca0 0x7fafac106840 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009930 tx=0x7fafa002a420 comp rx=0 tx=0).stop 2024-09-16T00:16:03.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.638+0000 7fafb1dc3640 1 -- 172.21.15.33:0/996069255 shutdown_connections 2024-09-16T00:16:03.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.638+0000 7fafb1dc3640 1 --2- 172.21.15.33:0/996069255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafac106f50 0x7fafac10eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:03.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.638+0000 7fafb1dc3640 1 --2- 172.21.15.33:0/996069255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafac100ca0 0x7fafac106840 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:03.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.638+0000 7fafb1dc3640 1 --2- 172.21.15.33:0/996069255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafac1002f0 0x7fafac1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.639+0000 7fafb1dc3640 1 -- 172.21.15.33:0/996069255 >> 172.21.15.33:0/996069255 conn(0x7fafac077210 msgr2=0x7fafac0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.639+0000 7fafb1dc3640 1 -- 172.21.15.33:0/996069255 shutdown_connections 2024-09-16T00:16:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.639+0000 7fafb1dc3640 1 -- 172.21.15.33:0/996069255 wait complete. 2024-09-16T00:16:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.639+0000 7fafb1dc3640 1 Processor -- start 2024-09-16T00:16:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.639+0000 7fafb1dc3640 1 -- start start 2024-09-16T00:16:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.639+0000 7fafb1dc3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafac1002f0 0x7fafac198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.639+0000 7fafb1dc3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafac100ca0 0x7fafac198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafb1dc3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafac106f50 0x7fafac19cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafb1dc3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fafac111fa0 con 0x7fafac100ca0 2024-09-16T00:16:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafb1dc3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fafac111e20 con 0x7fafac1002f0 2024-09-16T00:16:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafb1dc3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fafac112120 con 0x7fafac106f50 2024-09-16T00:16:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafaaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafac100ca0 0x7fafac198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:03.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafaaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafac100ca0 0x7fafac198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40256/0 (socket says 172.21.15.33:40256) 2024-09-16T00:16:03.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafaaffd640 1 -- 172.21.15.33:0/1662940913 learned_addr learned my addr 172.21.15.33:0/1662940913 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:03.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafabfff640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafac106f50 0x7fafac19cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:03.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafab7fe640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafac1002f0 0x7fafac198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:03.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafaaffd640 1 -- 172.21.15.33:0/1662940913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafac106f50 msgr2=0x7fafac19cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:03.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafaaffd640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafac106f50 0x7fafac19cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:03.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafaaffd640 1 -- 172.21.15.33:0/1662940913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafac1002f0 msgr2=0x7fafac198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:03.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafaaffd640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafac1002f0 0x7fafac198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:03.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafaaffd640 1 -- 172.21.15.33:0/1662940913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafac19d4d0 con 0x7fafac100ca0 2024-09-16T00:16:03.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafabfff640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafac106f50 0x7fafac19cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:16:03.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafaaffd640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafac100ca0 0x7fafac198a60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fafa0005940 tx=0x7fafa0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:03.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafa8ff9640 1 -- 172.21.15.33:0/1662940913 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa003f030 con 0x7fafac100ca0 2024-09-16T00:16:03.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafa8ff9640 1 -- 172.21.15.33:0/1662940913 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafa0003ad0 con 0x7fafac100ca0 2024-09-16T00:16:03.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafb1dc3640 1 -- 172.21.15.33:0/1662940913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafac19d760 con 0x7fafac100ca0 2024-09-16T00:16:03.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.640+0000 7fafa8ff9640 1 -- 172.21.15.33:0/1662940913 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa0035070 con 0x7fafac100ca0 2024-09-16T00:16:03.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.641+0000 7fafb1dc3640 1 -- 172.21.15.33:0/1662940913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafac1a50a0 con 0x7fafac100ca0 2024-09-16T00:16:03.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.641+0000 7fafb1dc3640 1 -- 172.21.15.33:0/1662940913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafac1006d0 con 0x7fafac100ca0 2024-09-16T00:16:03.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.643+0000 7fafa8ff9640 1 -- 172.21.15.33:0/1662940913 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fafa0042030 con 0x7fafac100ca0 2024-09-16T00:16:03.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.643+0000 7fafa8ff9640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faf940a32d0 0x7faf940a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:03.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.644+0000 7fafab7fe640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faf940a32d0 0x7faf940a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:03.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.644+0000 7fafa8ff9640 1 -- 172.21.15.33:0/1662940913 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fafa00f0530 con 0x7fafac100ca0 2024-09-16T00:16:03.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.644+0000 7fafab7fe640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faf940a32d0 0x7faf940a5790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009bd0 tx=0x7faf9c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:03.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.646+0000 7fafa8ff9640 1 -- 172.21.15.33:0/1662940913 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fafa00bc800 con 0x7fafac100ca0 2024-09-16T00:16:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.835+0000 7fafb1dc3640 1 -- 172.21.15.33:0/1662940913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fafac199880 con 0x7fafac100ca0 2024-09-16T00:16:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.837+0000 7fafa8ff9640 1 -- 172.21.15.33:0/1662940913 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24138 (secure 0 0 0) 0x7fafac199880 con 0x7fafac100ca0 2024-09-16T00:16:03.839 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:03.839 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":86,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:15:16.193527+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:16:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.840+0000 7fafb1dc3640 1 -- 172.21.15.33:0/1662940913 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faf940a32d0 msgr2=0x7faf940a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.840+0000 7fafb1dc3640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faf940a32d0 0x7faf940a5790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009bd0 tx=0x7faf9c005f90 comp rx=0 tx=0).stop 2024-09-16T00:16:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.840+0000 7fafb1dc3640 1 -- 172.21.15.33:0/1662940913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafac100ca0 msgr2=0x7fafac198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.840+0000 7fafb1dc3640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafac100ca0 0x7fafac198a60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fafa0005940 tx=0x7fafa0004270 comp rx=0 tx=0).stop 2024-09-16T00:16:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.840+0000 7fafb1dc3640 1 -- 172.21.15.33:0/1662940913 shutdown_connections 2024-09-16T00:16:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.840+0000 7fafb1dc3640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faf940a32d0 0x7faf940a5790 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.840+0000 7fafb1dc3640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafac106f50 0x7fafac19cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.840+0000 7fafb1dc3640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafac100ca0 0x7fafac198a60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.840+0000 7fafb1dc3640 1 --2- 172.21.15.33:0/1662940913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafac1002f0 0x7fafac198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.840+0000 7fafb1dc3640 1 -- 172.21.15.33:0/1662940913 >> 172.21.15.33:0/1662940913 conn(0x7fafac077210 msgr2=0x7fafac105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.840+0000 7fafb1dc3640 1 -- 172.21.15.33:0/1662940913 shutdown_connections 2024-09-16T00:16:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:03.840+0000 7fafb1dc3640 1 -- 172.21.15.33:0/1662940913 wait complete. 2024-09-16T00:16:04.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:04 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1662940913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:16:04.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:04 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1662940913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:16:04.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:04 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1662940913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:16:05.105 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-16T00:16:05.105 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T00:16:05.388 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:05 smithi060 ceph-mon[35185]: pgmap v314: 129 pgs: 129 active+clean; 632 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2024-09-16T00:16:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:05 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:05.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:05 smithi033 ceph-mon[31243]: pgmap v314: 129 pgs: 129 active+clean; 632 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2024-09-16T00:16:05.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:05.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:05.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:05.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:05 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:05.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.856+0000 7fa78c66e640 1 -- 172.21.15.33:0/3944785163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa784100c10 msgr2=0x7fa784110840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:05.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.856+0000 7fa78c66e640 1 --2- 172.21.15.33:0/3944785163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa784100c10 0x7fa784110840 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa77c009900 tx=0x7fa77c02a480 comp rx=0 tx=0).stop 2024-09-16T00:16:05.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.856+0000 7fa78c66e640 1 -- 172.21.15.33:0/3944785163 shutdown_connections 2024-09-16T00:16:05.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.856+0000 7fa78c66e640 1 --2- 172.21.15.33:0/3944785163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa784100c10 0x7fa784110840 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:05.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.856+0000 7fa78c66e640 1 --2- 172.21.15.33:0/3944785163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa784102ea0 0x7fa7841005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:05.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.856+0000 7fa78c66e640 1 --2- 172.21.15.33:0/3944785163 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa784102ac0 0x7fa7841000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:05.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.856+0000 7fa78c66e640 1 -- 172.21.15.33:0/3944785163 >> 172.21.15.33:0/3944785163 conn(0x7fa784077210 msgr2=0x7fa7840fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:05.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.856+0000 7fa78c66e640 1 -- 172.21.15.33:0/3944785163 shutdown_connections 2024-09-16T00:16:05.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.856+0000 7fa78c66e640 1 -- 172.21.15.33:0/3944785163 wait complete. 2024-09-16T00:16:05.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa78c66e640 1 Processor -- start 2024-09-16T00:16:05.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa78c66e640 1 -- start start 2024-09-16T00:16:05.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa78c66e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa784100c10 0x7fa78419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:05.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa78c66e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa784102ac0 0x7fa78419ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:05.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa78c66e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa784102ea0 0x7fa78419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:05.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa78c66e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841141d0 con 0x7fa784102ea0 2024-09-16T00:16:05.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa78c66e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa784114050 con 0x7fa784100c10 2024-09-16T00:16:05.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa78c66e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa784114350 con 0x7fa784102ac0 2024-09-16T00:16:05.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa78ac0b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa784102ea0 0x7fa78419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:05.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa78ac0b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa784102ea0 0x7fa78419eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40276/0 (socket says 172.21.15.33:40276) 2024-09-16T00:16:05.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa789c09640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa784102ac0 0x7fa78419ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:05.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa78ac0b640 1 -- 172.21.15.33:0/24777240 learned_addr learned my addr 172.21.15.33:0/24777240 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:05.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.857+0000 7fa78a40a640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa784100c10 0x7fa78419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:05.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.858+0000 7fa78ac0b640 1 -- 172.21.15.33:0/24777240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa784102ac0 msgr2=0x7fa78419ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:05.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.858+0000 7fa78ac0b640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa784102ac0 0x7fa78419ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:05.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.858+0000 7fa78ac0b640 1 -- 172.21.15.33:0/24777240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa784100c10 msgr2=0x7fa78419a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:05.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.858+0000 7fa78ac0b640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa784100c10 0x7fa78419a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:05.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.858+0000 7fa78ac0b640 1 -- 172.21.15.33:0/24777240 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa78419f5c0 con 0x7fa784102ea0 2024-09-16T00:16:05.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.858+0000 7fa789c09640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa784102ac0 0x7fa78419ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:16:05.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.858+0000 7fa78ac0b640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa784102ea0 0x7fa78419eee0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa77c004770 tx=0x7fa77c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:05.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.858+0000 7fa7777fe640 1 -- 172.21.15.33:0/24777240 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c03f070 con 0x7fa784102ea0 2024-09-16T00:16:05.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.858+0000 7fa78c66e640 1 -- 172.21.15.33:0/24777240 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa78419f850 con 0x7fa784102ea0 2024-09-16T00:16:05.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.858+0000 7fa7777fe640 1 -- 172.21.15.33:0/24777240 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa77c031c90 con 0x7fa784102ea0 2024-09-16T00:16:05.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.858+0000 7fa7777fe640 1 -- 172.21.15.33:0/24777240 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c035030 con 0x7fa784102ea0 2024-09-16T00:16:05.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.858+0000 7fa78c66e640 1 -- 172.21.15.33:0/24777240 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7841a7190 con 0x7fa784102ea0 2024-09-16T00:16:05.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.859+0000 7fa78c66e640 1 -- 172.21.15.33:0/24777240 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7841000b0 con 0x7fa784102ea0 2024-09-16T00:16:05.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.860+0000 7fa7777fe640 1 -- 172.21.15.33:0/24777240 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa77c042070 con 0x7fa784102ea0 2024-09-16T00:16:05.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.861+0000 7fa7777fe640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa7700a32d0 0x7fa7700a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:05.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.861+0000 7fa78a40a640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa7700a32d0 0x7fa7700a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:05.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.861+0000 7fa7777fe640 1 -- 172.21.15.33:0/24777240 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fa77c0f0480 con 0x7fa784102ea0 2024-09-16T00:16:05.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.862+0000 7fa78a40a640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa7700a32d0 0x7fa7700a5790 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa77800bbe0 tx=0x7fa77800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:05.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:05.865+0000 7fa7777fe640 1 -- 172.21.15.33:0/24777240 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa77c0bc750 con 0x7fa784102ea0 2024-09-16T00:16:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:05 smithi083 ceph-mon[35242]: pgmap v314: 129 pgs: 129 active+clean; 632 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2024-09-16T00:16:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:05.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:05.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:05.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:05 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:06.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.065+0000 7fa78c66e640 1 -- 172.21.15.33:0/24777240 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa7840fde00 con 0x7fa7700a32d0 2024-09-16T00:16:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.074+0000 7fa7777fe640 1 -- 172.21.15.33:0/24777240 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340585 (secure 0 0 0) 0x7fa7840fde00 con 0x7fa7700a32d0 2024-09-16T00:16:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:06.082 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-09-16T00:16:06.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.079+0000 7fa78c66e640 1 -- 172.21.15.33:0/24777240 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa7700a32d0 msgr2=0x7fa7700a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:06.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.079+0000 7fa78c66e640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa7700a32d0 0x7fa7700a5790 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa77800bbe0 tx=0x7fa77800b390 comp rx=0 tx=0).stop 2024-09-16T00:16:06.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.079+0000 7fa78c66e640 1 -- 172.21.15.33:0/24777240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa784102ea0 msgr2=0x7fa78419eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:06.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.079+0000 7fa78c66e640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa784102ea0 0x7fa78419eee0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa77c004770 tx=0x7fa77c003b90 comp rx=0 tx=0).stop 2024-09-16T00:16:06.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.079+0000 7fa78c66e640 1 -- 172.21.15.33:0/24777240 shutdown_connections 2024-09-16T00:16:06.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.079+0000 7fa78c66e640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa7700a32d0 0x7fa7700a5790 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:06.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.079+0000 7fa78c66e640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa784102ea0 0x7fa78419eee0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:06.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.079+0000 7fa78c66e640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa784102ac0 0x7fa78419ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:06.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.079+0000 7fa78c66e640 1 --2- 172.21.15.33:0/24777240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa784100c10 0x7fa78419a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:06.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.079+0000 7fa78c66e640 1 -- 172.21.15.33:0/24777240 >> 172.21.15.33:0/24777240 conn(0x7fa784077210 msgr2=0x7fa7840fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:06.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.079+0000 7fa78c66e640 1 -- 172.21.15.33:0/24777240 shutdown_connections 2024-09-16T00:16:06.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:06.079+0000 7fa78c66e640 1 -- 172.21.15.33:0/24777240 wait complete. 2024-09-16T00:16:06.731 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-16T00:16:05.647153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":647018,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":237,"num_read_kb":224,"num_write":341,"num_write_kb":3043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":385760,"kb_used_data":11552,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124556064,"statfs":{"total":1151940427776,"available":1151545409536,"internally_reserved":0,"allocated":11829248,"data_stored":7660580,"data_compressed":319671,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":49196,"num_objects":5,"num_object_clones":0,"num_object_copies":17,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":89,"num_read_kb":94,"num_write":40,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003152"},"pg_stats":[{"pgid":"2.1f","version":"86'9","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.178868+0000","last_change":"2024-09-16T00:15:04.342338+0000","last_active":"2024-09-16T00:15:49.178868+0000","last_peered":"2024-09-16T00:15:49.178868+0000","last_clean":"2024-09-16T00:15:49.178868+0000","last_became_active":"2024-09-16T00:15:04.342029+0000","last_became_peered":"2024-09-16T00:15:04.342029+0000","last_unstale":"2024-09-16T00:15:49.178868+0000","last_undegraded":"2024-09-16T00:15:49.178868+0000","last_fullsized":"2024-09-16T00:15:49.178868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:55:21.012262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570766+0000","last_change":"2024-09-16T00:15:06.547094+0000","last_active":"2024-09-16T00:15:16.570766+0000","last_peered":"2024-09-16T00:15:16.570766+0000","last_clean":"2024-09-16T00:15:16.570766+0000","last_became_active":"2024-09-16T00:15:06.542764+0000","last_became_peered":"2024-09-16T00:15:06.542764+0000","last_unstale":"2024-09-16T00:15:16.570766+0000","last_undegraded":"2024-09-16T00:15:16.570766+0000","last_fullsized":"2024-09-16T00:15:16.570766+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:04:37.449577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571347+0000","last_change":"2024-09-16T00:15:13.578400+0000","last_active":"2024-09-16T00:15:16.571347+0000","last_peered":"2024-09-16T00:15:16.571347+0000","last_clean":"2024-09-16T00:15:16.571347+0000","last_became_active":"2024-09-16T00:15:13.577787+0000","last_became_peered":"2024-09-16T00:15:13.577787+0000","last_unstale":"2024-09-16T00:15:16.571347+0000","last_undegraded":"2024-09-16T00:15:16.571347+0000","last_fullsized":"2024-09-16T00:15:16.571347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:19:59.194152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571117+0000","last_change":"2024-09-16T00:15:04.373381+0000","last_active":"2024-09-16T00:15:16.571117+0000","last_peered":"2024-09-16T00:15:16.571117+0000","last_clean":"2024-09-16T00:15:16.571117+0000","last_became_active":"2024-09-16T00:15:04.373028+0000","last_became_peered":"2024-09-16T00:15:04.373028+0000","last_unstale":"2024-09-16T00:15:16.571117+0000","last_undegraded":"2024-09-16T00:15:16.571117+0000","last_fullsized":"2024-09-16T00:15:16.571117+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:22.902583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717843+0000","last_change":"2024-09-16T00:15:06.554048+0000","last_active":"2024-09-16T00:15:16.717843+0000","last_peered":"2024-09-16T00:15:16.717843+0000","last_clean":"2024-09-16T00:15:16.717843+0000","last_became_active":"2024-09-16T00:15:06.553044+0000","last_became_peered":"2024-09-16T00:15:06.553044+0000","last_unstale":"2024-09-16T00:15:16.717843+0000","last_undegraded":"2024-09-16T00:15:16.717843+0000","last_fullsized":"2024-09-16T00:15:16.717843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:22:53.333705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717271+0000","last_change":"2024-09-16T00:15:13.586995+0000","last_active":"2024-09-16T00:15:16.717271+0000","last_peered":"2024-09-16T00:15:16.717271+0000","last_clean":"2024-09-16T00:15:16.717271+0000","last_became_active":"2024-09-16T00:15:13.586509+0000","last_became_peered":"2024-09-16T00:15:13.586509+0000","last_unstale":"2024-09-16T00:15:16.717271+0000","last_undegraded":"2024-09-16T00:15:16.717271+0000","last_fullsized":"2024-09-16T00:15:16.717271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:39.418581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.177512+0000","last_change":"2024-09-16T00:15:04.348235+0000","last_active":"2024-09-16T00:15:49.177512+0000","last_peered":"2024-09-16T00:15:49.177512+0000","last_clean":"2024-09-16T00:15:49.177512+0000","last_became_active":"2024-09-16T00:15:04.347518+0000","last_became_peered":"2024-09-16T00:15:04.347518+0000","last_unstale":"2024-09-16T00:15:49.177512+0000","last_undegraded":"2024-09-16T00:15:49.177512+0000","last_fullsized":"2024-09-16T00:15:49.177512+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:24.396911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718974+0000","last_change":"2024-09-16T00:15:06.554067+0000","last_active":"2024-09-16T00:15:16.718974+0000","last_peered":"2024-09-16T00:15:16.718974+0000","last_clean":"2024-09-16T00:15:16.718974+0000","last_became_active":"2024-09-16T00:15:06.553715+0000","last_became_peered":"2024-09-16T00:15:06.553715+0000","last_unstale":"2024-09-16T00:15:16.718974+0000","last_undegraded":"2024-09-16T00:15:16.718974+0000","last_fullsized":"2024-09-16T00:15:16.718974+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:22.008530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718992+0000","last_change":"2024-09-16T00:15:13.586872+0000","last_active":"2024-09-16T00:15:16.718992+0000","last_peered":"2024-09-16T00:15:16.718992+0000","last_clean":"2024-09-16T00:15:16.718992+0000","last_became_active":"2024-09-16T00:15:13.586375+0000","last_became_peered":"2024-09-16T00:15:13.586375+0000","last_unstale":"2024-09-16T00:15:16.718992+0000","last_undegraded":"2024-09-16T00:15:16.718992+0000","last_fullsized":"2024-09-16T00:15:16.718992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:43.668827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.859671+0000","last_change":"2024-09-16T00:15:04.347964+0000","last_active":"2024-09-16T00:15:16.859671+0000","last_peered":"2024-09-16T00:15:16.859671+0000","last_clean":"2024-09-16T00:15:16.859671+0000","last_became_active":"2024-09-16T00:15:04.347588+0000","last_became_peered":"2024-09-16T00:15:04.347588+0000","last_unstale":"2024-09-16T00:15:16.859671+0000","last_undegraded":"2024-09-16T00:15:16.859671+0000","last_fullsized":"2024-09-16T00:15:16.859671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:14.199287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718701+0000","last_change":"2024-09-16T00:15:06.544918+0000","last_active":"2024-09-16T00:15:16.718701+0000","last_peered":"2024-09-16T00:15:16.718701+0000","last_clean":"2024-09-16T00:15:16.718701+0000","last_became_active":"2024-09-16T00:15:06.544579+0000","last_became_peered":"2024-09-16T00:15:06.544579+0000","last_unstale":"2024-09-16T00:15:16.718701+0000","last_undegraded":"2024-09-16T00:15:16.718701+0000","last_fullsized":"2024-09-16T00:15:16.718701+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:06:37.475644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717546+0000","last_change":"2024-09-16T00:15:13.581488+0000","last_active":"2024-09-16T00:15:16.717546+0000","last_peered":"2024-09-16T00:15:16.717546+0000","last_clean":"2024-09-16T00:15:16.717546+0000","last_became_active":"2024-09-16T00:15:13.581164+0000","last_became_peered":"2024-09-16T00:15:13.581164+0000","last_unstale":"2024-09-16T00:15:16.717546+0000","last_undegraded":"2024-09-16T00:15:16.717546+0000","last_fullsized":"2024-09-16T00:15:16.717546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:05.061592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571396+0000","last_change":"2024-09-16T00:15:06.551982+0000","last_active":"2024-09-16T00:15:16.571396+0000","last_peered":"2024-09-16T00:15:16.571396+0000","last_clean":"2024-09-16T00:15:16.571396+0000","last_became_active":"2024-09-16T00:15:06.551663+0000","last_became_peered":"2024-09-16T00:15:06.551663+0000","last_unstale":"2024-09-16T00:15:16.571396+0000","last_undegraded":"2024-09-16T00:15:16.571396+0000","last_fullsized":"2024-09-16T00:15:16.571396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:11.928703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.173852+0000","last_change":"2024-09-16T00:15:04.352237+0000","last_active":"2024-09-16T00:15:48.173852+0000","last_peered":"2024-09-16T00:15:48.173852+0000","last_clean":"2024-09-16T00:15:48.173852+0000","last_became_active":"2024-09-16T00:15:04.351884+0000","last_became_peered":"2024-09-16T00:15:04.351884+0000","last_unstale":"2024-09-16T00:15:48.173852+0000","last_undegraded":"2024-09-16T00:15:48.173852+0000","last_fullsized":"2024-09-16T00:15:48.173852+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:32.932020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717602+0000","last_change":"2024-09-16T00:15:13.638917+0000","last_active":"2024-09-16T00:15:16.717602+0000","last_peered":"2024-09-16T00:15:16.717602+0000","last_clean":"2024-09-16T00:15:16.717602+0000","last_became_active":"2024-09-16T00:15:13.638401+0000","last_became_peered":"2024-09-16T00:15:13.638401+0000","last_unstale":"2024-09-16T00:15:16.717602+0000","last_undegraded":"2024-09-16T00:15:16.717602+0000","last_fullsized":"2024-09-16T00:15:16.717602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:38.082407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570732+0000","last_change":"2024-09-16T00:15:06.547909+0000","last_active":"2024-09-16T00:15:16.570732+0000","last_peered":"2024-09-16T00:15:16.570732+0000","last_clean":"2024-09-16T00:15:16.570732+0000","last_became_active":"2024-09-16T00:15:06.546814+0000","last_became_peered":"2024-09-16T00:15:06.546814+0000","last_unstale":"2024-09-16T00:15:16.570732+0000","last_undegraded":"2024-09-16T00:15:16.570732+0000","last_fullsized":"2024-09-16T00:15:16.570732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:46.419406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571418+0000","last_change":"2024-09-16T00:15:04.349796+0000","last_active":"2024-09-16T00:15:16.571418+0000","last_peered":"2024-09-16T00:15:16.571418+0000","last_clean":"2024-09-16T00:15:16.571418+0000","last_became_active":"2024-09-16T00:15:04.349435+0000","last_became_peered":"2024-09-16T00:15:04.349435+0000","last_unstale":"2024-09-16T00:15:16.571418+0000","last_undegraded":"2024-09-16T00:15:16.571418+0000","last_fullsized":"2024-09-16T00:15:16.571418+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:09:43.702956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717958+0000","last_change":"2024-09-16T00:15:13.588345+0000","last_active":"2024-09-16T00:15:16.717958+0000","last_peered":"2024-09-16T00:15:16.717958+0000","last_clean":"2024-09-16T00:15:16.717958+0000","last_became_active":"2024-09-16T00:15:13.587829+0000","last_became_peered":"2024-09-16T00:15:13.587829+0000","last_unstale":"2024-09-16T00:15:16.717958+0000","last_undegraded":"2024-09-16T00:15:16.717958+0000","last_fullsized":"2024-09-16T00:15:16.717958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:04.261031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717528+0000","last_change":"2024-09-16T00:15:06.545485+0000","last_active":"2024-09-16T00:15:16.717528+0000","last_peered":"2024-09-16T00:15:16.717528+0000","last_clean":"2024-09-16T00:15:16.717528+0000","last_became_active":"2024-09-16T00:15:06.545178+0000","last_became_peered":"2024-09-16T00:15:06.545178+0000","last_unstale":"2024-09-16T00:15:16.717528+0000","last_undegraded":"2024-09-16T00:15:16.717528+0000","last_fullsized":"2024-09-16T00:15:16.717528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:26.923469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.172469+0000","last_change":"2024-09-16T00:15:04.340546+0000","last_active":"2024-09-16T00:15:48.172469+0000","last_peered":"2024-09-16T00:15:48.172469+0000","last_clean":"2024-09-16T00:15:48.172469+0000","last_became_active":"2024-09-16T00:15:04.340173+0000","last_became_peered":"2024-09-16T00:15:04.340173+0000","last_unstale":"2024-09-16T00:15:48.172469+0000","last_undegraded":"2024-09-16T00:15:48.172469+0000","last_fullsized":"2024-09-16T00:15:48.172469+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:35:01.134433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718366+0000","last_change":"2024-09-16T00:15:13.589963+0000","last_active":"2024-09-16T00:15:16.718366+0000","last_peered":"2024-09-16T00:15:16.718366+0000","last_clean":"2024-09-16T00:15:16.718366+0000","last_became_active":"2024-09-16T00:15:13.589615+0000","last_became_peered":"2024-09-16T00:15:13.589615+0000","last_unstale":"2024-09-16T00:15:16.718366+0000","last_undegraded":"2024-09-16T00:15:16.718366+0000","last_fullsized":"2024-09-16T00:15:16.718366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:37.197765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.859080+0000","last_change":"2024-09-16T00:15:06.544374+0000","last_active":"2024-09-16T00:15:16.859080+0000","last_peered":"2024-09-16T00:15:16.859080+0000","last_clean":"2024-09-16T00:15:16.859080+0000","last_became_active":"2024-09-16T00:15:06.544027+0000","last_became_peered":"2024-09-16T00:15:06.544027+0000","last_unstale":"2024-09-16T00:15:16.859080+0000","last_undegraded":"2024-09-16T00:15:16.859080+0000","last_fullsized":"2024-09-16T00:15:16.859080+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.172879+0000","last_change":"2024-09-16T00:15:04.374209+0000","last_active":"2024-09-16T00:15:48.172879+0000","last_peered":"2024-09-16T00:15:48.172879+0000","last_clean":"2024-09-16T00:15:48.172879+0000","last_became_active":"2024-09-16T00:15:04.373814+0000","last_became_peered":"2024-09-16T00:15:04.373814+0000","last_unstale":"2024-09-16T00:15:48.172879+0000","last_undegraded":"2024-09-16T00:15:48.172879+0000","last_fullsized":"2024-09-16T00:15:48.172879+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:13.516161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717939+0000","last_change":"2024-09-16T00:15:13.639210+0000","last_active":"2024-09-16T00:15:16.717939+0000","last_peered":"2024-09-16T00:15:16.717939+0000","last_clean":"2024-09-16T00:15:16.717939+0000","last_became_active":"2024-09-16T00:15:13.638719+0000","last_became_peered":"2024-09-16T00:15:13.638719+0000","last_unstale":"2024-09-16T00:15:16.717939+0000","last_undegraded":"2024-09-16T00:15:16.717939+0000","last_fullsized":"2024-09-16T00:15:16.717939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:26.130778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717624+0000","last_change":"2024-09-16T00:15:06.553108+0000","last_active":"2024-09-16T00:15:16.717624+0000","last_peered":"2024-09-16T00:15:16.717624+0000","last_clean":"2024-09-16T00:15:16.717624+0000","last_became_active":"2024-09-16T00:15:06.552817+0000","last_became_peered":"2024-09-16T00:15:06.552817+0000","last_unstale":"2024-09-16T00:15:16.717624+0000","last_undegraded":"2024-09-16T00:15:16.717624+0000","last_fullsized":"2024-09-16T00:15:16.717624+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:06.821337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.172594+0000","last_change":"2024-09-16T00:15:04.644185+0000","last_active":"2024-09-16T00:15:48.172594+0000","last_peered":"2024-09-16T00:15:48.172594+0000","last_clean":"2024-09-16T00:15:48.172594+0000","last_became_active":"2024-09-16T00:15:04.643887+0000","last_became_peered":"2024-09-16T00:15:04.643887+0000","last_unstale":"2024-09-16T00:15:48.172594+0000","last_undegraded":"2024-09-16T00:15:48.172594+0000","last_fullsized":"2024-09-16T00:15:48.172594+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:44.279272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571611+0000","last_change":"2024-09-16T00:15:13.574761+0000","last_active":"2024-09-16T00:15:16.571611+0000","last_peered":"2024-09-16T00:15:16.571611+0000","last_clean":"2024-09-16T00:15:16.571611+0000","last_became_active":"2024-09-16T00:15:13.574347+0000","last_became_peered":"2024-09-16T00:15:13.574347+0000","last_unstale":"2024-09-16T00:15:16.571611+0000","last_undegraded":"2024-09-16T00:15:16.571611+0000","last_fullsized":"2024-09-16T00:15:16.571611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:22.040121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571044+0000","last_change":"2024-09-16T00:15:06.550338+0000","last_active":"2024-09-16T00:15:16.571044+0000","last_peered":"2024-09-16T00:15:16.571044+0000","last_clean":"2024-09-16T00:15:16.571044+0000","last_became_active":"2024-09-16T00:15:06.549868+0000","last_became_peered":"2024-09-16T00:15:06.549868+0000","last_unstale":"2024-09-16T00:15:16.571044+0000","last_undegraded":"2024-09-16T00:15:16.571044+0000","last_fullsized":"2024-09-16T00:15:16.571044+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:26.643338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:16:01.543351+0000","last_change":"2024-09-16T00:15:04.342306+0000","last_active":"2024-09-16T00:16:01.543351+0000","last_peered":"2024-09-16T00:16:01.543351+0000","last_clean":"2024-09-16T00:16:01.543351+0000","last_became_active":"2024-09-16T00:15:04.341964+0000","last_became_peered":"2024-09-16T00:15:04.341964+0000","last_unstale":"2024-09-16T00:16:01.543351+0000","last_undegraded":"2024-09-16T00:16:01.543351+0000","last_fullsized":"2024-09-16T00:16:01.543351+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:14.137199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570459+0000","last_change":"2024-09-16T00:15:13.638246+0000","last_active":"2024-09-16T00:15:16.570459+0000","last_peered":"2024-09-16T00:15:16.570459+0000","last_clean":"2024-09-16T00:15:16.570459+0000","last_became_active":"2024-09-16T00:15:13.637816+0000","last_became_peered":"2024-09-16T00:15:13.637816+0000","last_unstale":"2024-09-16T00:15:16.570459+0000","last_undegraded":"2024-09-16T00:15:16.570459+0000","last_fullsized":"2024-09-16T00:15:16.570459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:27.331156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718856+0000","last_change":"2024-09-16T00:15:06.544057+0000","last_active":"2024-09-16T00:15:16.718856+0000","last_peered":"2024-09-16T00:15:16.718856+0000","last_clean":"2024-09-16T00:15:16.718856+0000","last_became_active":"2024-09-16T00:15:06.543244+0000","last_became_peered":"2024-09-16T00:15:06.543244+0000","last_unstale":"2024-09-16T00:15:16.718856+0000","last_undegraded":"2024-09-16T00:15:16.718856+0000","last_fullsized":"2024-09-16T00:15:16.718856+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:42:28.950014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.173500+0000","last_change":"2024-09-16T00:15:04.376419+0000","last_active":"2024-09-16T00:15:48.173500+0000","last_peered":"2024-09-16T00:15:48.173500+0000","last_clean":"2024-09-16T00:15:48.173500+0000","last_became_active":"2024-09-16T00:15:04.375976+0000","last_became_peered":"2024-09-16T00:15:04.375976+0000","last_unstale":"2024-09-16T00:15:48.173500+0000","last_undegraded":"2024-09-16T00:15:48.173500+0000","last_fullsized":"2024-09-16T00:15:48.173500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:33:38.652109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717793+0000","last_change":"2024-09-16T00:15:13.584306+0000","last_active":"2024-09-16T00:15:16.717793+0000","last_peered":"2024-09-16T00:15:16.717793+0000","last_clean":"2024-09-16T00:15:16.717793+0000","last_became_active":"2024-09-16T00:15:13.583840+0000","last_became_peered":"2024-09-16T00:15:13.583840+0000","last_unstale":"2024-09-16T00:15:16.717793+0000","last_undegraded":"2024-09-16T00:15:16.717793+0000","last_fullsized":"2024-09-16T00:15:16.717793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:36.660458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718524+0000","last_change":"2024-09-16T00:15:06.538902+0000","last_active":"2024-09-16T00:15:16.718524+0000","last_peered":"2024-09-16T00:15:16.718524+0000","last_clean":"2024-09-16T00:15:16.718524+0000","last_became_active":"2024-09-16T00:15:06.538537+0000","last_became_peered":"2024-09-16T00:15:06.538537+0000","last_unstale":"2024-09-16T00:15:16.718524+0000","last_undegraded":"2024-09-16T00:15:16.718524+0000","last_fullsized":"2024-09-16T00:15:16.718524+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:38.411711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718017+0000","last_change":"2024-09-16T00:15:04.374762+0000","last_active":"2024-09-16T00:15:16.718017+0000","last_peered":"2024-09-16T00:15:16.718017+0000","last_clean":"2024-09-16T00:15:16.718017+0000","last_became_active":"2024-09-16T00:15:04.374394+0000","last_became_peered":"2024-09-16T00:15:04.374394+0000","last_unstale":"2024-09-16T00:15:16.718017+0000","last_undegraded":"2024-09-16T00:15:16.718017+0000","last_fullsized":"2024-09-16T00:15:16.718017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:47.075021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.859226+0000","last_change":"2024-09-16T00:15:13.638306+0000","last_active":"2024-09-16T00:15:16.859226+0000","last_peered":"2024-09-16T00:15:16.859226+0000","last_clean":"2024-09-16T00:15:16.859226+0000","last_became_active":"2024-09-16T00:15:13.637931+0000","last_became_peered":"2024-09-16T00:15:13.637931+0000","last_unstale":"2024-09-16T00:15:16.859226+0000","last_undegraded":"2024-09-16T00:15:16.859226+0000","last_fullsized":"2024-09-16T00:15:16.859226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:41.385174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717975+0000","last_change":"2024-09-16T00:15:06.548209+0000","last_active":"2024-09-16T00:15:16.717975+0000","last_peered":"2024-09-16T00:15:16.717975+0000","last_clean":"2024-09-16T00:15:16.717975+0000","last_became_active":"2024-09-16T00:15:06.547846+0000","last_became_peered":"2024-09-16T00:15:06.547846+0000","last_unstale":"2024-09-16T00:15:16.717975+0000","last_undegraded":"2024-09-16T00:15:16.717975+0000","last_fullsized":"2024-09-16T00:15:16.717975+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:15.821847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571615+0000","last_change":"2024-09-16T00:15:04.342403+0000","last_active":"2024-09-16T00:15:16.571615+0000","last_peered":"2024-09-16T00:15:16.571615+0000","last_clean":"2024-09-16T00:15:16.571615+0000","last_became_active":"2024-09-16T00:15:04.342064+0000","last_became_peered":"2024-09-16T00:15:04.342064+0000","last_unstale":"2024-09-16T00:15:16.571615+0000","last_undegraded":"2024-09-16T00:15:16.571615+0000","last_fullsized":"2024-09-16T00:15:16.571615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:36.706776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718700+0000","last_change":"2024-09-16T00:15:13.579387+0000","last_active":"2024-09-16T00:15:16.718700+0000","last_peered":"2024-09-16T00:15:16.718700+0000","last_clean":"2024-09-16T00:15:16.718700+0000","last_became_active":"2024-09-16T00:15:13.578996+0000","last_became_peered":"2024-09-16T00:15:13.578996+0000","last_unstale":"2024-09-16T00:15:16.718700+0000","last_undegraded":"2024-09-16T00:15:16.718700+0000","last_fullsized":"2024-09-16T00:15:16.718700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:51.703060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717862+0000","last_change":"2024-09-16T00:15:06.540396+0000","last_active":"2024-09-16T00:15:16.717862+0000","last_peered":"2024-09-16T00:15:16.717862+0000","last_clean":"2024-09-16T00:15:16.717862+0000","last_became_active":"2024-09-16T00:15:06.540084+0000","last_became_peered":"2024-09-16T00:15:06.540084+0000","last_unstale":"2024-09-16T00:15:16.717862+0000","last_undegraded":"2024-09-16T00:15:16.717862+0000","last_fullsized":"2024-09-16T00:15:16.717862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:46.477143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:16:04.202657+0000","last_change":"2024-09-16T00:15:04.373896+0000","last_active":"2024-09-16T00:16:04.202657+0000","last_peered":"2024-09-16T00:16:04.202657+0000","last_clean":"2024-09-16T00:16:04.202657+0000","last_became_active":"2024-09-16T00:15:04.373493+0000","last_became_peered":"2024-09-16T00:15:04.373493+0000","last_unstale":"2024-09-16T00:16:04.202657+0000","last_undegraded":"2024-09-16T00:16:04.202657+0000","last_fullsized":"2024-09-16T00:16:04.202657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:17:10.767267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718093+0000","last_change":"2024-09-16T00:15:13.638575+0000","last_active":"2024-09-16T00:15:16.718093+0000","last_peered":"2024-09-16T00:15:16.718093+0000","last_clean":"2024-09-16T00:15:16.718093+0000","last_became_active":"2024-09-16T00:15:13.638144+0000","last_became_peered":"2024-09-16T00:15:13.638144+0000","last_unstale":"2024-09-16T00:15:16.718093+0000","last_undegraded":"2024-09-16T00:15:16.718093+0000","last_fullsized":"2024-09-16T00:15:16.718093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:39.345011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.716899+0000","last_change":"2024-09-16T00:15:06.548852+0000","last_active":"2024-09-16T00:15:16.716899+0000","last_peered":"2024-09-16T00:15:16.716899+0000","last_clean":"2024-09-16T00:15:16.716899+0000","last_became_active":"2024-09-16T00:15:06.548238+0000","last_became_peered":"2024-09-16T00:15:06.548238+0000","last_unstale":"2024-09-16T00:15:16.716899+0000","last_undegraded":"2024-09-16T00:15:16.716899+0000","last_fullsized":"2024-09-16T00:15:16.716899+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:16.115338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.177212+0000","last_change":"2024-09-16T00:15:04.350753+0000","last_active":"2024-09-16T00:15:49.177212+0000","last_peered":"2024-09-16T00:15:49.177212+0000","last_clean":"2024-09-16T00:15:49.177212+0000","last_became_active":"2024-09-16T00:15:04.350406+0000","last_became_peered":"2024-09-16T00:15:04.350406+0000","last_unstale":"2024-09-16T00:15:49.177212+0000","last_undegraded":"2024-09-16T00:15:49.177212+0000","last_fullsized":"2024-09-16T00:15:49.177212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:09:54.737287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718316+0000","last_change":"2024-09-16T00:15:13.639302+0000","last_active":"2024-09-16T00:15:16.718316+0000","last_peered":"2024-09-16T00:15:16.718316+0000","last_clean":"2024-09-16T00:15:16.718316+0000","last_became_active":"2024-09-16T00:15:13.638863+0000","last_became_peered":"2024-09-16T00:15:13.638863+0000","last_unstale":"2024-09-16T00:15:16.718316+0000","last_undegraded":"2024-09-16T00:15:16.718316+0000","last_fullsized":"2024-09-16T00:15:16.718316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:23.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570789+0000","last_change":"2024-09-16T00:15:06.545797+0000","last_active":"2024-09-16T00:15:16.570789+0000","last_peered":"2024-09-16T00:15:16.570789+0000","last_clean":"2024-09-16T00:15:16.570789+0000","last_became_active":"2024-09-16T00:15:06.545477+0000","last_became_peered":"2024-09-16T00:15:06.545477+0000","last_unstale":"2024-09-16T00:15:16.570789+0000","last_undegraded":"2024-09-16T00:15:16.570789+0000","last_fullsized":"2024-09-16T00:15:16.570789+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.887183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:50.181523+0000","last_change":"2024-09-16T00:15:04.342216+0000","last_active":"2024-09-16T00:15:50.181523+0000","last_peered":"2024-09-16T00:15:50.181523+0000","last_clean":"2024-09-16T00:15:50.181523+0000","last_became_active":"2024-09-16T00:15:04.341763+0000","last_became_peered":"2024-09-16T00:15:04.341763+0000","last_unstale":"2024-09-16T00:15:50.181523+0000","last_undegraded":"2024-09-16T00:15:50.181523+0000","last_fullsized":"2024-09-16T00:15:50.181523+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:44.938527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":341,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571538+0000","last_change":"2024-09-16T00:13:51.108571+0000","last_active":"2024-09-16T00:15:16.571538+0000","last_peered":"2024-09-16T00:15:16.571538+0000","last_clean":"2024-09-16T00:15:16.571538+0000","last_became_active":"2024-09-16T00:13:51.108088+0000","last_became_peered":"2024-09-16T00:13:51.108088+0000","last_unstale":"2024-09-16T00:15:16.571538+0000","last_undegraded":"2024-09-16T00:15:16.571538+0000","last_fullsized":"2024-09-16T00:15:16.571538+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_clean_scrub_stamp":"2024-09-16T00:10:24.533807+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:02.511064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571863+0000","last_change":"2024-09-16T00:15:13.588665+0000","last_active":"2024-09-16T00:15:16.571863+0000","last_peered":"2024-09-16T00:15:16.571863+0000","last_clean":"2024-09-16T00:15:16.571863+0000","last_became_active":"2024-09-16T00:15:13.588222+0000","last_became_peered":"2024-09-16T00:15:13.588222+0000","last_unstale":"2024-09-16T00:15:16.571863+0000","last_undegraded":"2024-09-16T00:15:16.571863+0000","last_fullsized":"2024-09-16T00:15:16.571863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:59:59.965962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571925+0000","last_change":"2024-09-16T00:15:06.553114+0000","last_active":"2024-09-16T00:15:16.571925+0000","last_peered":"2024-09-16T00:15:16.571925+0000","last_clean":"2024-09-16T00:15:16.571925+0000","last_became_active":"2024-09-16T00:15:06.552788+0000","last_became_peered":"2024-09-16T00:15:06.552788+0000","last_unstale":"2024-09-16T00:15:16.571925+0000","last_undegraded":"2024-09-16T00:15:16.571925+0000","last_fullsized":"2024-09-16T00:15:16.571925+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:05.473847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.180950+0000","last_change":"2024-09-16T00:15:04.644100+0000","last_active":"2024-09-16T00:15:49.180950+0000","last_peered":"2024-09-16T00:15:49.180950+0000","last_clean":"2024-09-16T00:15:49.180950+0000","last_became_active":"2024-09-16T00:15:04.643725+0000","last_became_peered":"2024-09-16T00:15:04.643725+0000","last_unstale":"2024-09-16T00:15:49.180950+0000","last_undegraded":"2024-09-16T00:15:49.180950+0000","last_fullsized":"2024-09-16T00:15:49.180950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:15.482080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718471+0000","last_change":"2024-09-16T00:15:13.575606+0000","last_active":"2024-09-16T00:15:16.718471+0000","last_peered":"2024-09-16T00:15:16.718471+0000","last_clean":"2024-09-16T00:15:16.718471+0000","last_became_active":"2024-09-16T00:15:13.575131+0000","last_became_peered":"2024-09-16T00:15:13.575131+0000","last_unstale":"2024-09-16T00:15:16.718471+0000","last_undegraded":"2024-09-16T00:15:16.718471+0000","last_fullsized":"2024-09-16T00:15:16.718471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:41.636058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571578+0000","last_change":"2024-09-16T00:15:06.552956+0000","last_active":"2024-09-16T00:15:16.571578+0000","last_peered":"2024-09-16T00:15:16.571578+0000","last_clean":"2024-09-16T00:15:16.571578+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:15:16.571578+0000","last_undegraded":"2024-09-16T00:15:16.571578+0000","last_fullsized":"2024-09-16T00:15:16.571578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:36:20.739676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718949+0000","last_change":"2024-09-16T00:15:04.349095+0000","last_active":"2024-09-16T00:15:16.718949+0000","last_peered":"2024-09-16T00:15:16.718949+0000","last_clean":"2024-09-16T00:15:16.718949+0000","last_became_active":"2024-09-16T00:15:04.348698+0000","last_became_peered":"2024-09-16T00:15:04.348698+0000","last_unstale":"2024-09-16T00:15:16.718949+0000","last_undegraded":"2024-09-16T00:15:16.718949+0000","last_fullsized":"2024-09-16T00:15:16.718949+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:42.282882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718208+0000","last_change":"2024-09-16T00:15:13.580494+0000","last_active":"2024-09-16T00:15:16.718208+0000","last_peered":"2024-09-16T00:15:16.718208+0000","last_clean":"2024-09-16T00:15:16.718208+0000","last_became_active":"2024-09-16T00:15:13.580091+0000","last_became_peered":"2024-09-16T00:15:13.580091+0000","last_unstale":"2024-09-16T00:15:16.718208+0000","last_undegraded":"2024-09-16T00:15:16.718208+0000","last_fullsized":"2024-09-16T00:15:16.718208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:17.314919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571885+0000","last_change":"2024-09-16T00:15:06.544859+0000","last_active":"2024-09-16T00:15:16.571885+0000","last_peered":"2024-09-16T00:15:16.571885+0000","last_clean":"2024-09-16T00:15:16.571885+0000","last_became_active":"2024-09-16T00:15:06.544289+0000","last_became_peered":"2024-09-16T00:15:06.544289+0000","last_unstale":"2024-09-16T00:15:16.571885+0000","last_undegraded":"2024-09-16T00:15:16.571885+0000","last_fullsized":"2024-09-16T00:15:16.571885+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:52.622477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718547+0000","last_change":"2024-09-16T00:15:04.351313+0000","last_active":"2024-09-16T00:15:16.718547+0000","last_peered":"2024-09-16T00:15:16.718547+0000","last_clean":"2024-09-16T00:15:16.718547+0000","last_became_active":"2024-09-16T00:15:04.350928+0000","last_became_peered":"2024-09-16T00:15:04.350928+0000","last_unstale":"2024-09-16T00:15:16.718547+0000","last_undegraded":"2024-09-16T00:15:16.718547+0000","last_fullsized":"2024-09-16T00:15:16.718547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:48.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717654+0000","last_change":"2024-09-16T00:15:13.578573+0000","last_active":"2024-09-16T00:15:16.717654+0000","last_peered":"2024-09-16T00:15:16.717654+0000","last_clean":"2024-09-16T00:15:16.717654+0000","last_became_active":"2024-09-16T00:15:13.578117+0000","last_became_peered":"2024-09-16T00:15:13.578117+0000","last_unstale":"2024-09-16T00:15:16.717654+0000","last_undegraded":"2024-09-16T00:15:16.717654+0000","last_fullsized":"2024-09-16T00:15:16.717654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:53.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718552+0000","last_change":"2024-09-16T00:15:06.553548+0000","last_active":"2024-09-16T00:15:16.718552+0000","last_peered":"2024-09-16T00:15:16.718552+0000","last_clean":"2024-09-16T00:15:16.718552+0000","last_became_active":"2024-09-16T00:15:06.552963+0000","last_became_peered":"2024-09-16T00:15:06.552963+0000","last_unstale":"2024-09-16T00:15:16.718552+0000","last_undegraded":"2024-09-16T00:15:16.718552+0000","last_fullsized":"2024-09-16T00:15:16.718552+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:43.455881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:50.181698+0000","last_change":"2024-09-16T00:15:04.351005+0000","last_active":"2024-09-16T00:15:50.181698+0000","last_peered":"2024-09-16T00:15:50.181698+0000","last_clean":"2024-09-16T00:15:50.181698+0000","last_became_active":"2024-09-16T00:15:04.350614+0000","last_became_peered":"2024-09-16T00:15:04.350614+0000","last_unstale":"2024-09-16T00:15:50.181698+0000","last_undegraded":"2024-09-16T00:15:50.181698+0000","last_fullsized":"2024-09-16T00:15:50.181698+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:39.196360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717921+0000","last_change":"2024-09-16T00:15:13.578466+0000","last_active":"2024-09-16T00:15:16.717921+0000","last_peered":"2024-09-16T00:15:16.717921+0000","last_clean":"2024-09-16T00:15:16.717921+0000","last_became_active":"2024-09-16T00:15:13.578078+0000","last_became_peered":"2024-09-16T00:15:13.578078+0000","last_unstale":"2024-09-16T00:15:16.717921+0000","last_undegraded":"2024-09-16T00:15:16.717921+0000","last_fullsized":"2024-09-16T00:15:16.717921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:41:20.439086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718413+0000","last_change":"2024-09-16T00:15:06.546325+0000","last_active":"2024-09-16T00:15:16.718413+0000","last_peered":"2024-09-16T00:15:16.718413+0000","last_clean":"2024-09-16T00:15:16.718413+0000","last_became_active":"2024-09-16T00:15:06.546028+0000","last_became_peered":"2024-09-16T00:15:06.546028+0000","last_unstale":"2024-09-16T00:15:16.718413+0000","last_undegraded":"2024-09-16T00:15:16.718413+0000","last_fullsized":"2024-09-16T00:15:16.718413+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:15.049592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.179258+0000","last_change":"2024-09-16T00:15:04.373818+0000","last_active":"2024-09-16T00:15:49.179258+0000","last_peered":"2024-09-16T00:15:49.179258+0000","last_clean":"2024-09-16T00:15:49.179258+0000","last_became_active":"2024-09-16T00:15:04.373406+0000","last_became_peered":"2024-09-16T00:15:04.373406+0000","last_unstale":"2024-09-16T00:15:49.179258+0000","last_undegraded":"2024-09-16T00:15:49.179258+0000","last_fullsized":"2024-09-16T00:15:49.179258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:08.402660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718433+0000","last_change":"2024-09-16T00:15:13.588669+0000","last_active":"2024-09-16T00:15:16.718433+0000","last_peered":"2024-09-16T00:15:16.718433+0000","last_clean":"2024-09-16T00:15:16.718433+0000","last_became_active":"2024-09-16T00:15:13.588229+0000","last_became_peered":"2024-09-16T00:15:13.588229+0000","last_unstale":"2024-09-16T00:15:16.718433+0000","last_undegraded":"2024-09-16T00:15:16.718433+0000","last_fullsized":"2024-09-16T00:15:16.718433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:30.157296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571181+0000","last_change":"2024-09-16T00:15:06.550221+0000","last_active":"2024-09-16T00:15:16.571181+0000","last_peered":"2024-09-16T00:15:16.571181+0000","last_clean":"2024-09-16T00:15:16.571181+0000","last_became_active":"2024-09-16T00:15:06.549914+0000","last_became_peered":"2024-09-16T00:15:06.549914+0000","last_unstale":"2024-09-16T00:15:16.571181+0000","last_undegraded":"2024-09-16T00:15:16.571181+0000","last_fullsized":"2024-09-16T00:15:16.571181+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:18:26.824523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":51,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:16:03.201845+0000","last_change":"2024-09-16T00:15:04.344389+0000","last_active":"2024-09-16T00:16:03.201845+0000","last_peered":"2024-09-16T00:16:03.201845+0000","last_clean":"2024-09-16T00:16:03.201845+0000","last_became_active":"2024-09-16T00:15:04.343956+0000","last_became_peered":"2024-09-16T00:15:04.343956+0000","last_unstale":"2024-09-16T00:16:03.201845+0000","last_undegraded":"2024-09-16T00:16:03.201845+0000","last_fullsized":"2024-09-16T00:16:03.201845+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:34.019861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718417+0000","last_change":"2024-09-16T00:15:13.586043+0000","last_active":"2024-09-16T00:15:16.718417+0000","last_peered":"2024-09-16T00:15:16.718417+0000","last_clean":"2024-09-16T00:15:16.718417+0000","last_became_active":"2024-09-16T00:15:13.583872+0000","last_became_peered":"2024-09-16T00:15:13.583872+0000","last_unstale":"2024-09-16T00:15:16.718417+0000","last_undegraded":"2024-09-16T00:15:16.718417+0000","last_fullsized":"2024-09-16T00:15:16.718417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:12.160443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571491+0000","last_change":"2024-09-16T00:15:06.542744+0000","last_active":"2024-09-16T00:15:16.571491+0000","last_peered":"2024-09-16T00:15:16.571491+0000","last_clean":"2024-09-16T00:15:16.571491+0000","last_became_active":"2024-09-16T00:15:06.541868+0000","last_became_peered":"2024-09-16T00:15:06.541868+0000","last_unstale":"2024-09-16T00:15:16.571491+0000","last_undegraded":"2024-09-16T00:15:16.571491+0000","last_fullsized":"2024-09-16T00:15:16.571491+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:10:06.508682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.173721+0000","last_change":"2024-09-16T00:15:04.349556+0000","last_active":"2024-09-16T00:15:48.173721+0000","last_peered":"2024-09-16T00:15:48.173721+0000","last_clean":"2024-09-16T00:15:48.173721+0000","last_became_active":"2024-09-16T00:15:04.349242+0000","last_became_peered":"2024-09-16T00:15:04.349242+0000","last_unstale":"2024-09-16T00:15:48.173721+0000","last_undegraded":"2024-09-16T00:15:48.173721+0000","last_fullsized":"2024-09-16T00:15:48.173721+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:22:18.459157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717967+0000","last_change":"2024-09-16T00:15:13.584109+0000","last_active":"2024-09-16T00:15:16.717967+0000","last_peered":"2024-09-16T00:15:16.717967+0000","last_clean":"2024-09-16T00:15:16.717967+0000","last_became_active":"2024-09-16T00:15:13.583718+0000","last_became_peered":"2024-09-16T00:15:13.583718+0000","last_unstale":"2024-09-16T00:15:16.717967+0000","last_undegraded":"2024-09-16T00:15:16.717967+0000","last_fullsized":"2024-09-16T00:15:16.717967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:29.416254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717276+0000","last_change":"2024-09-16T00:15:06.548247+0000","last_active":"2024-09-16T00:15:16.717276+0000","last_peered":"2024-09-16T00:15:16.717276+0000","last_clean":"2024-09-16T00:15:16.717276+0000","last_became_active":"2024-09-16T00:15:06.547864+0000","last_became_peered":"2024-09-16T00:15:06.547864+0000","last_unstale":"2024-09-16T00:15:16.717276+0000","last_undegraded":"2024-09-16T00:15:16.717276+0000","last_fullsized":"2024-09-16T00:15:16.717276+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:25.350197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.178625+0000","last_change":"2024-09-16T00:15:04.350965+0000","last_active":"2024-09-16T00:15:49.178625+0000","last_peered":"2024-09-16T00:15:49.178625+0000","last_clean":"2024-09-16T00:15:49.178625+0000","last_became_active":"2024-09-16T00:15:04.349124+0000","last_became_peered":"2024-09-16T00:15:04.349124+0000","last_unstale":"2024-09-16T00:15:49.178625+0000","last_undegraded":"2024-09-16T00:15:49.178625+0000","last_fullsized":"2024-09-16T00:15:49.178625+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:40.859220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718757+0000","last_change":"2024-09-16T00:15:13.581718+0000","last_active":"2024-09-16T00:15:16.718757+0000","last_peered":"2024-09-16T00:15:16.718757+0000","last_clean":"2024-09-16T00:15:16.718757+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:15:16.718757+0000","last_undegraded":"2024-09-16T00:15:16.718757+0000","last_fullsized":"2024-09-16T00:15:16.718757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:08.810606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.859429+0000","last_change":"2024-09-16T00:15:06.544507+0000","last_active":"2024-09-16T00:15:16.859429+0000","last_peered":"2024-09-16T00:15:16.859429+0000","last_clean":"2024-09-16T00:15:16.859429+0000","last_became_active":"2024-09-16T00:15:06.544178+0000","last_became_peered":"2024-09-16T00:15:06.544178+0000","last_unstale":"2024-09-16T00:15:16.859429+0000","last_undegraded":"2024-09-16T00:15:16.859429+0000","last_fullsized":"2024-09-16T00:15:16.859429+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:38.494266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"86'29","reported_seq":92,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:16:02.204274+0000","last_change":"2024-09-16T00:15:04.350871+0000","last_active":"2024-09-16T00:16:02.204274+0000","last_peered":"2024-09-16T00:16:02.204274+0000","last_clean":"2024-09-16T00:16:02.204274+0000","last_became_active":"2024-09-16T00:15:04.349900+0000","last_became_peered":"2024-09-16T00:15:04.349900+0000","last_unstale":"2024-09-16T00:16:02.204274+0000","last_undegraded":"2024-09-16T00:16:02.204274+0000","last_fullsized":"2024-09-16T00:16:02.204274+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.167301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52413,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":67,"num_write":30,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718108+0000","last_change":"2024-09-16T00:15:13.585928+0000","last_active":"2024-09-16T00:15:16.718108+0000","last_peered":"2024-09-16T00:15:16.718108+0000","last_clean":"2024-09-16T00:15:16.718108+0000","last_became_active":"2024-09-16T00:15:13.582940+0000","last_became_peered":"2024-09-16T00:15:13.582940+0000","last_unstale":"2024-09-16T00:15:16.718108+0000","last_undegraded":"2024-09-16T00:15:16.718108+0000","last_fullsized":"2024-09-16T00:15:16.718108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:57:13.251978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.719088+0000","last_change":"2024-09-16T00:15:06.546238+0000","last_active":"2024-09-16T00:15:16.719088+0000","last_peered":"2024-09-16T00:15:16.719088+0000","last_clean":"2024-09-16T00:15:16.719088+0000","last_became_active":"2024-09-16T00:15:06.545919+0000","last_became_peered":"2024-09-16T00:15:06.545919+0000","last_unstale":"2024-09-16T00:15:16.719088+0000","last_undegraded":"2024-09-16T00:15:16.719088+0000","last_fullsized":"2024-09-16T00:15:16.719088+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:09.180832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.172877+0000","last_change":"2024-09-16T00:15:04.376028+0000","last_active":"2024-09-16T00:15:48.172877+0000","last_peered":"2024-09-16T00:15:48.172877+0000","last_clean":"2024-09-16T00:15:48.172877+0000","last_became_active":"2024-09-16T00:15:04.375633+0000","last_became_peered":"2024-09-16T00:15:04.375633+0000","last_unstale":"2024-09-16T00:15:48.172877+0000","last_undegraded":"2024-09-16T00:15:48.172877+0000","last_fullsized":"2024-09-16T00:15:48.172877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:44.446342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718640+0000","last_change":"2024-09-16T00:15:13.587414+0000","last_active":"2024-09-16T00:15:16.718640+0000","last_peered":"2024-09-16T00:15:16.718640+0000","last_clean":"2024-09-16T00:15:16.718640+0000","last_became_active":"2024-09-16T00:15:13.586764+0000","last_became_peered":"2024-09-16T00:15:13.586764+0000","last_unstale":"2024-09-16T00:15:16.718640+0000","last_undegraded":"2024-09-16T00:15:16.718640+0000","last_fullsized":"2024-09-16T00:15:16.718640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:15:43.047070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717946+0000","last_change":"2024-09-16T00:15:06.550864+0000","last_active":"2024-09-16T00:15:16.717946+0000","last_peered":"2024-09-16T00:15:16.717946+0000","last_clean":"2024-09-16T00:15:16.717946+0000","last_became_active":"2024-09-16T00:15:06.547956+0000","last_became_peered":"2024-09-16T00:15:06.547956+0000","last_unstale":"2024-09-16T00:15:16.717946+0000","last_undegraded":"2024-09-16T00:15:16.717946+0000","last_fullsized":"2024-09-16T00:15:16.717946+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:26.102175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.174309+0000","last_change":"2024-09-16T00:15:04.350349+0000","last_active":"2024-09-16T00:15:48.174309+0000","last_peered":"2024-09-16T00:15:48.174309+0000","last_clean":"2024-09-16T00:15:48.174309+0000","last_became_active":"2024-09-16T00:15:04.349951+0000","last_became_peered":"2024-09-16T00:15:04.349951+0000","last_unstale":"2024-09-16T00:15:48.174309+0000","last_undegraded":"2024-09-16T00:15:48.174309+0000","last_fullsized":"2024-09-16T00:15:48.174309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:46.000831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717366+0000","last_change":"2024-09-16T00:15:13.585906+0000","last_active":"2024-09-16T00:15:16.717366+0000","last_peered":"2024-09-16T00:15:16.717366+0000","last_clean":"2024-09-16T00:15:16.717366+0000","last_became_active":"2024-09-16T00:15:13.583664+0000","last_became_peered":"2024-09-16T00:15:13.583664+0000","last_unstale":"2024-09-16T00:15:16.717366+0000","last_undegraded":"2024-09-16T00:15:16.717366+0000","last_fullsized":"2024-09-16T00:15:16.717366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:07:29.313469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.719136+0000","last_change":"2024-09-16T00:15:06.552919+0000","last_active":"2024-09-16T00:15:16.719136+0000","last_peered":"2024-09-16T00:15:16.719136+0000","last_clean":"2024-09-16T00:15:16.719136+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:15:16.719136+0000","last_undegraded":"2024-09-16T00:15:16.719136+0000","last_fullsized":"2024-09-16T00:15:16.719136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:47:38.651837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:16:01.199012+0000","last_change":"2024-09-16T00:15:04.351116+0000","last_active":"2024-09-16T00:16:01.199012+0000","last_peered":"2024-09-16T00:16:01.199012+0000","last_clean":"2024-09-16T00:16:01.199012+0000","last_became_active":"2024-09-16T00:15:04.350728+0000","last_became_peered":"2024-09-16T00:15:04.350728+0000","last_unstale":"2024-09-16T00:16:01.199012+0000","last_undegraded":"2024-09-16T00:16:01.199012+0000","last_fullsized":"2024-09-16T00:16:01.199012+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:38.314019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570602+0000","last_change":"2024-09-16T00:15:13.575961+0000","last_active":"2024-09-16T00:15:16.570602+0000","last_peered":"2024-09-16T00:15:16.570602+0000","last_clean":"2024-09-16T00:15:16.570602+0000","last_became_active":"2024-09-16T00:15:13.575550+0000","last_became_peered":"2024-09-16T00:15:13.575550+0000","last_unstale":"2024-09-16T00:15:16.570602+0000","last_undegraded":"2024-09-16T00:15:16.570602+0000","last_fullsized":"2024-09-16T00:15:16.570602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:05.515588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717415+0000","last_change":"2024-09-16T00:15:13.583583+0000","last_active":"2024-09-16T00:15:16.717415+0000","last_peered":"2024-09-16T00:15:16.717415+0000","last_clean":"2024-09-16T00:15:16.717415+0000","last_became_active":"2024-09-16T00:15:13.583145+0000","last_became_peered":"2024-09-16T00:15:13.583145+0000","last_unstale":"2024-09-16T00:15:16.717415+0000","last_undegraded":"2024-09-16T00:15:16.717415+0000","last_fullsized":"2024-09-16T00:15:16.717415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:07.320337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717281+0000","last_change":"2024-09-16T00:15:06.552017+0000","last_active":"2024-09-16T00:15:16.717281+0000","last_peered":"2024-09-16T00:15:16.717281+0000","last_clean":"2024-09-16T00:15:16.717281+0000","last_became_active":"2024-09-16T00:15:06.551688+0000","last_became_peered":"2024-09-16T00:15:06.551688+0000","last_unstale":"2024-09-16T00:15:16.717281+0000","last_undegraded":"2024-09-16T00:15:16.717281+0000","last_fullsized":"2024-09-16T00:15:16.717281+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:39.951856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718079+0000","last_change":"2024-09-16T00:15:04.350826+0000","last_active":"2024-09-16T00:15:16.718079+0000","last_peered":"2024-09-16T00:15:16.718079+0000","last_clean":"2024-09-16T00:15:16.718079+0000","last_became_active":"2024-09-16T00:15:04.350294+0000","last_became_peered":"2024-09-16T00:15:04.350294+0000","last_unstale":"2024-09-16T00:15:16.718079+0000","last_undegraded":"2024-09-16T00:15:16.718079+0000","last_fullsized":"2024-09-16T00:15:16.718079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:49.146709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718339+0000","last_change":"2024-09-16T00:15:13.580184+0000","last_active":"2024-09-16T00:15:16.718339+0000","last_peered":"2024-09-16T00:15:16.718339+0000","last_clean":"2024-09-16T00:15:16.718339+0000","last_became_active":"2024-09-16T00:15:13.579849+0000","last_became_peered":"2024-09-16T00:15:13.579849+0000","last_unstale":"2024-09-16T00:15:16.718339+0000","last_undegraded":"2024-09-16T00:15:16.718339+0000","last_fullsized":"2024-09-16T00:15:16.718339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:40.613127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570581+0000","last_change":"2024-09-16T00:15:06.545796+0000","last_active":"2024-09-16T00:15:16.570581+0000","last_peered":"2024-09-16T00:15:16.570581+0000","last_clean":"2024-09-16T00:15:16.570581+0000","last_became_active":"2024-09-16T00:15:06.544815+0000","last_became_peered":"2024-09-16T00:15:06.544815+0000","last_unstale":"2024-09-16T00:15:16.570581+0000","last_undegraded":"2024-09-16T00:15:16.570581+0000","last_fullsized":"2024-09-16T00:15:16.570581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:51:25.921949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.177230+0000","last_change":"2024-09-16T00:15:04.340101+0000","last_active":"2024-09-16T00:15:49.177230+0000","last_peered":"2024-09-16T00:15:49.177230+0000","last_clean":"2024-09-16T00:15:49.177230+0000","last_became_active":"2024-09-16T00:15:04.339788+0000","last_became_peered":"2024-09-16T00:15:04.339788+0000","last_unstale":"2024-09-16T00:15:49.177230+0000","last_undegraded":"2024-09-16T00:15:49.177230+0000","last_fullsized":"2024-09-16T00:15:49.177230+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:08.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.176612+0000","last_change":"2024-09-16T00:15:04.342298+0000","last_active":"2024-09-16T00:15:49.176612+0000","last_peered":"2024-09-16T00:15:49.176612+0000","last_clean":"2024-09-16T00:15:49.176612+0000","last_became_active":"2024-09-16T00:15:04.341930+0000","last_became_peered":"2024-09-16T00:15:04.341930+0000","last_unstale":"2024-09-16T00:15:49.176612+0000","last_undegraded":"2024-09-16T00:15:49.176612+0000","last_fullsized":"2024-09-16T00:15:49.176612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:03:43.036214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718518+0000","last_change":"2024-09-16T00:15:06.547496+0000","last_active":"2024-09-16T00:15:16.718518+0000","last_peered":"2024-09-16T00:15:16.718518+0000","last_clean":"2024-09-16T00:15:16.718518+0000","last_became_active":"2024-09-16T00:15:06.547177+0000","last_became_peered":"2024-09-16T00:15:06.547177+0000","last_unstale":"2024-09-16T00:15:16.718518+0000","last_undegraded":"2024-09-16T00:15:16.718518+0000","last_fullsized":"2024-09-16T00:15:16.718518+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:13.288827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571601+0000","last_change":"2024-09-16T00:15:13.581966+0000","last_active":"2024-09-16T00:15:16.571601+0000","last_peered":"2024-09-16T00:15:16.571601+0000","last_clean":"2024-09-16T00:15:16.571601+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:15:16.571601+0000","last_undegraded":"2024-09-16T00:15:16.571601+0000","last_fullsized":"2024-09-16T00:15:16.571601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.132773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":52,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:16:03.202440+0000","last_change":"2024-09-16T00:15:04.351765+0000","last_active":"2024-09-16T00:16:03.202440+0000","last_peered":"2024-09-16T00:16:03.202440+0000","last_clean":"2024-09-16T00:16:03.202440+0000","last_became_active":"2024-09-16T00:15:04.351398+0000","last_became_peered":"2024-09-16T00:15:04.351398+0000","last_unstale":"2024-09-16T00:16:03.202440+0000","last_undegraded":"2024-09-16T00:16:03.202440+0000","last_fullsized":"2024-09-16T00:16:03.202440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:14:28.762657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.859193+0000","last_change":"2024-09-16T00:15:06.541422+0000","last_active":"2024-09-16T00:15:16.859193+0000","last_peered":"2024-09-16T00:15:16.859193+0000","last_clean":"2024-09-16T00:15:16.859193+0000","last_became_active":"2024-09-16T00:15:06.541083+0000","last_became_peered":"2024-09-16T00:15:06.541083+0000","last_unstale":"2024-09-16T00:15:16.859193+0000","last_undegraded":"2024-09-16T00:15:16.859193+0000","last_fullsized":"2024-09-16T00:15:16.859193+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:30.138972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571945+0000","last_change":"2024-09-16T00:15:13.576877+0000","last_active":"2024-09-16T00:15:16.571945+0000","last_peered":"2024-09-16T00:15:16.571945+0000","last_clean":"2024-09-16T00:15:16.571945+0000","last_became_active":"2024-09-16T00:15:13.576480+0000","last_became_peered":"2024-09-16T00:15:13.576480+0000","last_unstale":"2024-09-16T00:15:16.571945+0000","last_undegraded":"2024-09-16T00:15:16.571945+0000","last_fullsized":"2024-09-16T00:15:16.571945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:47.322219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717463+0000","last_change":"2024-09-16T00:15:06.550329+0000","last_active":"2024-09-16T00:15:16.717463+0000","last_peered":"2024-09-16T00:15:16.717463+0000","last_clean":"2024-09-16T00:15:16.717463+0000","last_became_active":"2024-09-16T00:15:06.550008+0000","last_became_peered":"2024-09-16T00:15:06.550008+0000","last_unstale":"2024-09-16T00:15:16.717463+0000","last_undegraded":"2024-09-16T00:15:16.717463+0000","last_fullsized":"2024-09-16T00:15:16.717463+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:38.096126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571940+0000","last_change":"2024-09-16T00:15:06.550368+0000","last_active":"2024-09-16T00:15:16.571940+0000","last_peered":"2024-09-16T00:15:16.571940+0000","last_clean":"2024-09-16T00:15:16.571940+0000","last_became_active":"2024-09-16T00:15:06.549985+0000","last_became_peered":"2024-09-16T00:15:06.549985+0000","last_unstale":"2024-09-16T00:15:16.571940+0000","last_undegraded":"2024-09-16T00:15:16.571940+0000","last_fullsized":"2024-09-16T00:15:16.571940+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:16.992110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571338+0000","last_change":"2024-09-16T00:15:06.546713+0000","last_active":"2024-09-16T00:15:16.571338+0000","last_peered":"2024-09-16T00:15:16.571338+0000","last_clean":"2024-09-16T00:15:16.571338+0000","last_became_active":"2024-09-16T00:15:06.546209+0000","last_became_peered":"2024-09-16T00:15:06.546209+0000","last_unstale":"2024-09-16T00:15:16.571338+0000","last_undegraded":"2024-09-16T00:15:16.571338+0000","last_fullsized":"2024-09-16T00:15:16.571338+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:43.913748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718301+0000","last_change":"2024-09-16T00:15:06.550490+0000","last_active":"2024-09-16T00:15:16.718301+0000","last_peered":"2024-09-16T00:15:16.718301+0000","last_clean":"2024-09-16T00:15:16.718301+0000","last_became_active":"2024-09-16T00:15:06.549311+0000","last_became_peered":"2024-09-16T00:15:06.549311+0000","last_unstale":"2024-09-16T00:15:16.718301+0000","last_undegraded":"2024-09-16T00:15:16.718301+0000","last_fullsized":"2024-09-16T00:15:16.718301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:10:01.995357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718836+0000","last_change":"2024-09-16T00:15:06.548115+0000","last_active":"2024-09-16T00:15:16.718836+0000","last_peered":"2024-09-16T00:15:16.718836+0000","last_clean":"2024-09-16T00:15:16.718836+0000","last_became_active":"2024-09-16T00:15:06.547782+0000","last_became_peered":"2024-09-16T00:15:06.547782+0000","last_unstale":"2024-09-16T00:15:16.718836+0000","last_undegraded":"2024-09-16T00:15:16.718836+0000","last_fullsized":"2024-09-16T00:15:16.718836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:53.228222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718442+0000","last_change":"2024-09-16T00:15:06.554396+0000","last_active":"2024-09-16T00:15:16.718442+0000","last_peered":"2024-09-16T00:15:16.718442+0000","last_clean":"2024-09-16T00:15:16.718442+0000","last_became_active":"2024-09-16T00:15:06.554012+0000","last_became_peered":"2024-09-16T00:15:06.554012+0000","last_unstale":"2024-09-16T00:15:16.718442+0000","last_undegraded":"2024-09-16T00:15:16.718442+0000","last_fullsized":"2024-09-16T00:15:16.718442+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:31.464098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571795+0000","last_change":"2024-09-16T00:15:06.549969+0000","last_active":"2024-09-16T00:15:16.571795+0000","last_peered":"2024-09-16T00:15:16.571795+0000","last_clean":"2024-09-16T00:15:16.571795+0000","last_became_active":"2024-09-16T00:15:06.549531+0000","last_became_peered":"2024-09-16T00:15:06.549531+0000","last_unstale":"2024-09-16T00:15:16.571795+0000","last_undegraded":"2024-09-16T00:15:16.571795+0000","last_fullsized":"2024-09-16T00:15:16.571795+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:37.883612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570998+0000","last_change":"2024-09-16T00:15:06.547734+0000","last_active":"2024-09-16T00:15:16.570998+0000","last_peered":"2024-09-16T00:15:16.570998+0000","last_clean":"2024-09-16T00:15:16.570998+0000","last_became_active":"2024-09-16T00:15:06.547400+0000","last_became_peered":"2024-09-16T00:15:06.547400+0000","last_unstale":"2024-09-16T00:15:16.570998+0000","last_undegraded":"2024-09-16T00:15:16.570998+0000","last_fullsized":"2024-09-16T00:15:16.570998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:54.455313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717789+0000","last_change":"2024-09-16T00:15:06.551965+0000","last_active":"2024-09-16T00:15:16.717789+0000","last_peered":"2024-09-16T00:15:16.717789+0000","last_clean":"2024-09-16T00:15:16.717789+0000","last_became_active":"2024-09-16T00:15:06.551602+0000","last_became_peered":"2024-09-16T00:15:06.551602+0000","last_unstale":"2024-09-16T00:15:16.717789+0000","last_undegraded":"2024-09-16T00:15:16.717789+0000","last_fullsized":"2024-09-16T00:15:16.717789+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:34.951960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570606+0000","last_change":"2024-09-16T00:15:06.549340+0000","last_active":"2024-09-16T00:15:16.570606+0000","last_peered":"2024-09-16T00:15:16.570606+0000","last_clean":"2024-09-16T00:15:16.570606+0000","last_became_active":"2024-09-16T00:15:06.547424+0000","last_became_peered":"2024-09-16T00:15:06.547424+0000","last_unstale":"2024-09-16T00:15:16.570606+0000","last_undegraded":"2024-09-16T00:15:16.570606+0000","last_fullsized":"2024-09-16T00:15:16.570606+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:12.087535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718679+0000","last_change":"2024-09-16T00:15:06.555216+0000","last_active":"2024-09-16T00:15:16.718679+0000","last_peered":"2024-09-16T00:15:16.718679+0000","last_clean":"2024-09-16T00:15:16.718679+0000","last_became_active":"2024-09-16T00:15:06.554896+0000","last_became_peered":"2024-09-16T00:15:06.554896+0000","last_unstale":"2024-09-16T00:15:16.718679+0000","last_undegraded":"2024-09-16T00:15:16.718679+0000","last_fullsized":"2024-09-16T00:15:16.718679+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:20.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718416+0000","last_change":"2024-09-16T00:15:06.549752+0000","last_active":"2024-09-16T00:15:16.718416+0000","last_peered":"2024-09-16T00:15:16.718416+0000","last_clean":"2024-09-16T00:15:16.718416+0000","last_became_active":"2024-09-16T00:15:06.549350+0000","last_became_peered":"2024-09-16T00:15:06.549350+0000","last_unstale":"2024-09-16T00:15:16.718416+0000","last_undegraded":"2024-09-16T00:15:16.718416+0000","last_fullsized":"2024-09-16T00:15:16.718416+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:48.615123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718131+0000","last_change":"2024-09-16T00:15:06.549159+0000","last_active":"2024-09-16T00:15:16.718131+0000","last_peered":"2024-09-16T00:15:16.718131+0000","last_clean":"2024-09-16T00:15:16.718131+0000","last_became_active":"2024-09-16T00:15:06.548408+0000","last_became_peered":"2024-09-16T00:15:06.548408+0000","last_unstale":"2024-09-16T00:15:16.718131+0000","last_undegraded":"2024-09-16T00:15:16.718131+0000","last_fullsized":"2024-09-16T00:15:16.718131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:23.941809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571847+0000","last_change":"2024-09-16T00:15:06.544091+0000","last_active":"2024-09-16T00:15:16.571847+0000","last_peered":"2024-09-16T00:15:16.571847+0000","last_clean":"2024-09-16T00:15:16.571847+0000","last_became_active":"2024-09-16T00:15:06.543761+0000","last_became_peered":"2024-09-16T00:15:06.543761+0000","last_unstale":"2024-09-16T00:15:16.571847+0000","last_undegraded":"2024-09-16T00:15:16.571847+0000","last_fullsized":"2024-09-16T00:15:16.571847+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:36:43.253405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717983+0000","last_change":"2024-09-16T00:15:06.546408+0000","last_active":"2024-09-16T00:15:16.717983+0000","last_peered":"2024-09-16T00:15:16.717983+0000","last_clean":"2024-09-16T00:15:16.717983+0000","last_became_active":"2024-09-16T00:15:06.546092+0000","last_became_peered":"2024-09-16T00:15:06.546092+0000","last_unstale":"2024-09-16T00:15:16.717983+0000","last_undegraded":"2024-09-16T00:15:16.717983+0000","last_fullsized":"2024-09-16T00:15:16.717983+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:39:15.576097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718725+0000","last_change":"2024-09-16T00:15:06.557162+0000","last_active":"2024-09-16T00:15:16.718725+0000","last_peered":"2024-09-16T00:15:16.718725+0000","last_clean":"2024-09-16T00:15:16.718725+0000","last_became_active":"2024-09-16T00:15:06.556831+0000","last_became_peered":"2024-09-16T00:15:06.556831+0000","last_unstale":"2024-09-16T00:15:16.718725+0000","last_undegraded":"2024-09-16T00:15:16.718725+0000","last_fullsized":"2024-09-16T00:15:16.718725+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.236915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717369+0000","last_change":"2024-09-16T00:15:06.556161+0000","last_active":"2024-09-16T00:15:16.717369+0000","last_peered":"2024-09-16T00:15:16.717369+0000","last_clean":"2024-09-16T00:15:16.717369+0000","last_became_active":"2024-09-16T00:15:06.555819+0000","last_became_peered":"2024-09-16T00:15:06.555819+0000","last_unstale":"2024-09-16T00:15:16.717369+0000","last_undegraded":"2024-09-16T00:15:16.717369+0000","last_fullsized":"2024-09-16T00:15:16.717369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:29.929977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717618+0000","last_change":"2024-09-16T00:15:06.550635+0000","last_active":"2024-09-16T00:15:16.717618+0000","last_peered":"2024-09-16T00:15:16.717618+0000","last_clean":"2024-09-16T00:15:16.717618+0000","last_became_active":"2024-09-16T00:15:06.549576+0000","last_became_peered":"2024-09-16T00:15:06.549576+0000","last_unstale":"2024-09-16T00:15:16.717618+0000","last_undegraded":"2024-09-16T00:15:16.717618+0000","last_fullsized":"2024-09-16T00:15:16.717618+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:12.818199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717880+0000","last_change":"2024-09-16T00:15:06.547114+0000","last_active":"2024-09-16T00:15:16.717880+0000","last_peered":"2024-09-16T00:15:16.717880+0000","last_clean":"2024-09-16T00:15:16.717880+0000","last_became_active":"2024-09-16T00:15:06.546786+0000","last_became_peered":"2024-09-16T00:15:06.546786+0000","last_unstale":"2024-09-16T00:15:16.717880+0000","last_undegraded":"2024-09-16T00:15:16.717880+0000","last_fullsized":"2024-09-16T00:15:16.717880+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:13:26.925603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718321+0000","last_change":"2024-09-16T00:15:06.541168+0000","last_active":"2024-09-16T00:15:16.718321+0000","last_peered":"2024-09-16T00:15:16.718321+0000","last_clean":"2024-09-16T00:15:16.718321+0000","last_became_active":"2024-09-16T00:15:06.540848+0000","last_became_peered":"2024-09-16T00:15:06.540848+0000","last_unstale":"2024-09-16T00:15:16.718321+0000","last_undegraded":"2024-09-16T00:15:16.718321+0000","last_fullsized":"2024-09-16T00:15:16.718321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.958600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718097+0000","last_change":"2024-09-16T00:15:06.547429+0000","last_active":"2024-09-16T00:15:16.718097+0000","last_peered":"2024-09-16T00:15:16.718097+0000","last_clean":"2024-09-16T00:15:16.718097+0000","last_became_active":"2024-09-16T00:15:06.547081+0000","last_became_peered":"2024-09-16T00:15:06.547081+0000","last_unstale":"2024-09-16T00:15:16.718097+0000","last_undegraded":"2024-09-16T00:15:16.718097+0000","last_fullsized":"2024-09-16T00:15:16.718097+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:43.396313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571832+0000","last_change":"2024-09-16T00:15:06.551811+0000","last_active":"2024-09-16T00:15:16.571832+0000","last_peered":"2024-09-16T00:15:16.571832+0000","last_clean":"2024-09-16T00:15:16.571832+0000","last_became_active":"2024-09-16T00:15:06.551424+0000","last_became_peered":"2024-09-16T00:15:06.551424+0000","last_unstale":"2024-09-16T00:15:16.571832+0000","last_undegraded":"2024-09-16T00:15:16.571832+0000","last_fullsized":"2024-09-16T00:15:16.571832+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.943070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718670+0000","last_change":"2024-09-16T00:15:06.544697+0000","last_active":"2024-09-16T00:15:16.718670+0000","last_peered":"2024-09-16T00:15:16.718670+0000","last_clean":"2024-09-16T00:15:16.718670+0000","last_became_active":"2024-09-16T00:15:06.544182+0000","last_became_peered":"2024-09-16T00:15:06.544182+0000","last_unstale":"2024-09-16T00:15:16.718670+0000","last_undegraded":"2024-09-16T00:15:16.718670+0000","last_fullsized":"2024-09-16T00:15:16.718670+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:45.929092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.719262+0000","last_change":"2024-09-16T00:15:06.551488+0000","last_active":"2024-09-16T00:15:16.719262+0000","last_peered":"2024-09-16T00:15:16.719262+0000","last_clean":"2024-09-16T00:15:16.719262+0000","last_became_active":"2024-09-16T00:15:06.551171+0000","last_became_peered":"2024-09-16T00:15:06.551171+0000","last_unstale":"2024-09-16T00:15:16.719262+0000","last_undegraded":"2024-09-16T00:15:16.719262+0000","last_fullsized":"2024-09-16T00:15:16.719262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:23.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570559+0000","last_change":"2024-09-16T00:15:06.544043+0000","last_active":"2024-09-16T00:15:16.570559+0000","last_peered":"2024-09-16T00:15:16.570559+0000","last_clean":"2024-09-16T00:15:16.570559+0000","last_became_active":"2024-09-16T00:15:06.543014+0000","last_became_peered":"2024-09-16T00:15:06.543014+0000","last_unstale":"2024-09-16T00:15:16.570559+0000","last_undegraded":"2024-09-16T00:15:16.570559+0000","last_fullsized":"2024-09-16T00:15:16.570559+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:23.904810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571312+0000","last_change":"2024-09-16T00:15:06.546788+0000","last_active":"2024-09-16T00:15:16.571312+0000","last_peered":"2024-09-16T00:15:16.571312+0000","last_clean":"2024-09-16T00:15:16.571312+0000","last_became_active":"2024-09-16T00:15:06.546333+0000","last_became_peered":"2024-09-16T00:15:06.546333+0000","last_unstale":"2024-09-16T00:15:16.571312+0000","last_undegraded":"2024-09-16T00:15:16.571312+0000","last_fullsized":"2024-09-16T00:15:16.571312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:09.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718109+0000","last_change":"2024-09-16T00:15:06.544855+0000","last_active":"2024-09-16T00:15:16.718109+0000","last_peered":"2024-09-16T00:15:16.718109+0000","last_clean":"2024-09-16T00:15:16.718109+0000","last_became_active":"2024-09-16T00:15:06.543708+0000","last_became_peered":"2024-09-16T00:15:06.543708+0000","last_unstale":"2024-09-16T00:15:16.718109+0000","last_undegraded":"2024-09-16T00:15:16.718109+0000","last_fullsized":"2024-09-16T00:15:16.718109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:13:58.756111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.859192+0000","last_change":"2024-09-16T00:15:06.540675+0000","last_active":"2024-09-16T00:15:16.859192+0000","last_peered":"2024-09-16T00:15:16.859192+0000","last_clean":"2024-09-16T00:15:16.859192+0000","last_became_active":"2024-09-16T00:15:06.540333+0000","last_became_peered":"2024-09-16T00:15:06.540333+0000","last_unstale":"2024-09-16T00:15:16.859192+0000","last_undegraded":"2024-09-16T00:15:16.859192+0000","last_fullsized":"2024-09-16T00:15:16.859192+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:17.166987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718067+0000","last_change":"2024-09-16T00:15:06.548758+0000","last_active":"2024-09-16T00:15:16.718067+0000","last_peered":"2024-09-16T00:15:16.718067+0000","last_clean":"2024-09-16T00:15:16.718067+0000","last_became_active":"2024-09-16T00:15:06.548397+0000","last_became_peered":"2024-09-16T00:15:06.548397+0000","last_unstale":"2024-09-16T00:15:16.718067+0000","last_undegraded":"2024-09-16T00:15:16.718067+0000","last_fullsized":"2024-09-16T00:15:16.718067+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:50.320170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718682+0000","last_change":"2024-09-16T00:15:06.554898+0000","last_active":"2024-09-16T00:15:16.718682+0000","last_peered":"2024-09-16T00:15:16.718682+0000","last_clean":"2024-09-16T00:15:16.718682+0000","last_became_active":"2024-09-16T00:15:06.553005+0000","last_became_peered":"2024-09-16T00:15:06.553005+0000","last_unstale":"2024-09-16T00:15:16.718682+0000","last_undegraded":"2024-09-16T00:15:16.718682+0000","last_fullsized":"2024-09-16T00:15:16.718682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:29.630911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717525+0000","last_change":"2024-09-16T00:15:06.547265+0000","last_active":"2024-09-16T00:15:16.717525+0000","last_peered":"2024-09-16T00:15:16.717525+0000","last_clean":"2024-09-16T00:15:16.717525+0000","last_became_active":"2024-09-16T00:15:06.546938+0000","last_became_peered":"2024-09-16T00:15:06.546938+0000","last_unstale":"2024-09-16T00:15:16.717525+0000","last_undegraded":"2024-09-16T00:15:16.717525+0000","last_fullsized":"2024-09-16T00:15:16.717525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:40.247343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718626+0000","last_change":"2024-09-16T00:15:06.556735+0000","last_active":"2024-09-16T00:15:16.718626+0000","last_peered":"2024-09-16T00:15:16.718626+0000","last_clean":"2024-09-16T00:15:16.718626+0000","last_became_active":"2024-09-16T00:15:06.556324+0000","last_became_peered":"2024-09-16T00:15:06.556324+0000","last_unstale":"2024-09-16T00:15:16.718626+0000","last_undegraded":"2024-09-16T00:15:16.718626+0000","last_fullsized":"2024-09-16T00:15:16.718626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:25.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":56416,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":111,"num_read_kb":115,"num_write":158,"num_write_kb":103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":169248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249414,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579929,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":478633,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808861,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":1118707,"data_compressed":53889,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972387,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":482965,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201319,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":478789,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430253,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":1074700,"data_compressed":53889,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626483,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":474045,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855416,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":1069895,"data_compressed":53889,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986237,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":954368,"data_stored":479088,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215169,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":474239,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444103,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":527718,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607628,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":999424,"data_stored":526368,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":475433,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":52413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":53763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":54384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T00:16:06.737 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T00:16:07.002 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:07.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:06 smithi033 ceph-mon[31243]: pgmap v315: 129 pgs: 129 active+clean; 632 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.2 KiB/s wr, 10 op/s 2024-09-16T00:16:07.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:06 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:07.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:06 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:07.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:06 smithi033 ceph-mon[31243]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:16:07.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:06 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:16:07.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:06 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:16:07.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:06 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:07.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:06 smithi083 ceph-mon[35242]: pgmap v315: 129 pgs: 129 active+clean; 632 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.2 KiB/s wr, 10 op/s 2024-09-16T00:16:07.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:06 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:07.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:06 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:07.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:06 smithi083 ceph-mon[35242]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:16:07.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:06 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:16:07.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:06 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:16:07.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:06 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:07.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:06 smithi060 ceph-mon[35185]: pgmap v315: 129 pgs: 129 active+clean; 632 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.2 KiB/s wr, 10 op/s 2024-09-16T00:16:07.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:06 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:07.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:06 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:07.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:06 smithi060 ceph-mon[35185]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:16:07.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:06 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:16:07.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:06 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:16:07.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:06 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:16:07.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.472+0000 7f80c9c41640 1 -- 172.21.15.33:0/1579165775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80c4102790 msgr2=0x7f80c41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:07.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.472+0000 7f80c9c41640 1 --2- 172.21.15.33:0/1579165775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80c4102790 0x7f80c41067e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f80b8009900 tx=0x7f80b802a410 comp rx=0 tx=0).stop 2024-09-16T00:16:07.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.472+0000 7f80c9c41640 1 -- 172.21.15.33:0/1579165775 shutdown_connections 2024-09-16T00:16:07.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.472+0000 7f80c9c41640 1 --2- 172.21.15.33:0/1579165775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80c4106f10 0x7f80c410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:07.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.472+0000 7f80c9c41640 1 --2- 172.21.15.33:0/1579165775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80c4102790 0x7f80c41067e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:07.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.472+0000 7f80c9c41640 1 --2- 172.21.15.33:0/1579165775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c4101de0 0x7f80c41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:07.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.472+0000 7f80c9c41640 1 -- 172.21.15.33:0/1579165775 >> 172.21.15.33:0/1579165775 conn(0x7f80c40fd8c0 msgr2=0x7f80c40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:07.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.472+0000 7f80c9c41640 1 -- 172.21.15.33:0/1579165775 shutdown_connections 2024-09-16T00:16:07.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.472+0000 7f80c9c41640 1 -- 172.21.15.33:0/1579165775 wait complete. 2024-09-16T00:16:07.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.473+0000 7f80c9c41640 1 Processor -- start 2024-09-16T00:16:07.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.473+0000 7f80c9c41640 1 -- start start 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.473+0000 7f80c9c41640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80c4101de0 0x7f80c40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.473+0000 7f80c9c41640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c4102790 0x7f80c4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.473+0000 7f80c9c41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80c4106f10 0x7f80c4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.473+0000 7f80c9c41640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f80c4111f30 con 0x7f80c4101de0 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.473+0000 7f80c9c41640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f80c4111db0 con 0x7f80c4106f10 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.473+0000 7f80c9c41640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f80c41120b0 con 0x7f80c4102790 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.473+0000 7f80c37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80c4101de0 0x7f80c40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.473+0000 7f80c37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80c4101de0 0x7f80c40744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40292/0 (socket says 172.21.15.33:40292) 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.473+0000 7f80c37fe640 1 -- 172.21.15.33:0/1163203530 learned_addr learned my addr 172.21.15.33:0/1163203530 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c3fff640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80c4106f10 0x7f80c4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c2ffd640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c4102790 0x7f80c4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c37fe640 1 -- 172.21.15.33:0/1163203530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c4102790 msgr2=0x7f80c4074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:07.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c37fe640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c4102790 0x7f80c4074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c37fe640 1 -- 172.21.15.33:0/1163203530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80c4106f10 msgr2=0x7f80c4074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c37fe640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80c4106f10 0x7f80c4074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c37fe640 1 -- 172.21.15.33:0/1163203530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c41a68f0 con 0x7f80c4101de0 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c37fe640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80c4101de0 0x7f80c40744f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f80b400d950 tx=0x7f80b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c0ff9640 1 -- 172.21.15.33:0/1163203530 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b4017070 con 0x7f80c4101de0 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c0ff9640 1 -- 172.21.15.33:0/1163203530 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80b4009c30 con 0x7f80c4101de0 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c9c41640 1 -- 172.21.15.33:0/1163203530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c41a6be0 con 0x7f80c4101de0 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c0ff9640 1 -- 172.21.15.33:0/1163203530 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b400f6b0 con 0x7f80c4101de0 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.474+0000 7f80c9c41640 1 -- 172.21.15.33:0/1163203530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c41a7120 con 0x7f80c4101de0 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.475+0000 7f80c9c41640 1 -- 172.21.15.33:0/1163203530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80c41021c0 con 0x7f80c4101de0 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.477+0000 7f80c0ff9640 1 -- 172.21.15.33:0/1163203530 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f80b40109d0 con 0x7f80c4101de0 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.478+0000 7f80c0ff9640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f80ac0a32d0 0x7f80ac0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:07.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.478+0000 7f80c0ff9640 1 -- 172.21.15.33:0/1163203530 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f80b40cf950 con 0x7f80c4101de0 2024-09-16T00:16:07.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.478+0000 7f80c2ffd640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f80ac0a32d0 0x7f80ac0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:07.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.478+0000 7f80c2ffd640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f80ac0a32d0 0x7f80ac0a5790 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f80b80096c0 tx=0x7f80b8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:07.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.481+0000 7f80c0ff9640 1 -- 172.21.15.33:0/1163203530 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f80b4009de0 con 0x7f80c4101de0 2024-09-16T00:16:07.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.654+0000 7f80c9c41640 1 -- 172.21.15.33:0/1163203530 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f80c4062630 con 0x7f80ac0a32d0 2024-09-16T00:16:07.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.662+0000 7f80c0ff9640 1 -- 172.21.15.33:0/1163203530 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340584 (secure 0 0 0) 0x7f80c4062630 con 0x7f80ac0a32d0 2024-09-16T00:16:07.667 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:07.668 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-09-16T00:16:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.666+0000 7f80c9c41640 1 -- 172.21.15.33:0/1163203530 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f80ac0a32d0 msgr2=0x7f80ac0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.666+0000 7f80c9c41640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f80ac0a32d0 0x7f80ac0a5790 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f80b80096c0 tx=0x7f80b8000bc0 comp rx=0 tx=0).stop 2024-09-16T00:16:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.666+0000 7f80c9c41640 1 -- 172.21.15.33:0/1163203530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80c4101de0 msgr2=0x7f80c40744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:07.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.666+0000 7f80c9c41640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80c4101de0 0x7f80c40744f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f80b400d950 tx=0x7f80b400de10 comp rx=0 tx=0).stop 2024-09-16T00:16:07.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.666+0000 7f80c9c41640 1 -- 172.21.15.33:0/1163203530 shutdown_connections 2024-09-16T00:16:07.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.666+0000 7f80c9c41640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f80ac0a32d0 0x7f80ac0a5790 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:07.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.666+0000 7f80c9c41640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80c4106f10 0x7f80c4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:07.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.666+0000 7f80c9c41640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c4102790 0x7f80c4074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:07.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.666+0000 7f80c9c41640 1 --2- 172.21.15.33:0/1163203530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80c4101de0 0x7f80c40744f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:07.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.666+0000 7f80c9c41640 1 -- 172.21.15.33:0/1163203530 >> 172.21.15.33:0/1163203530 conn(0x7f80c40fd8c0 msgr2=0x7f80c40fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:07.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.666+0000 7f80c9c41640 1 -- 172.21.15.33:0/1163203530 shutdown_connections 2024-09-16T00:16:07.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:07.666+0000 7f80c9c41640 1 -- 172.21.15.33:0/1163203530 wait complete. 2024-09-16T00:16:08.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]: dispatch 2024-09-16T00:16:08.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-09-16T00:16:08.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-09-16T00:16:08.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-09-16T00:16:08.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-09-16T00:16:08.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-09-16T00:16:08.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]: dispatch 2024-09-16T00:16:08.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]: dispatch 2024-09-16T00:16:08.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-09-16T00:16:08.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-09-16T00:16:08.396 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-16T00:16:07.647848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":647018,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":237,"num_read_kb":224,"num_write":341,"num_write_kb":3043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":385760,"kb_used_data":11552,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124556064,"statfs":{"total":1151940427776,"available":1151545409536,"internally_reserved":0,"allocated":11829248,"data_stored":7660580,"data_compressed":319671,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":48803,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":62,"num_read_kb":77,"num_write":30,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003184"},"pg_stats":[{"pgid":"2.1f","version":"86'9","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.178868+0000","last_change":"2024-09-16T00:15:04.342338+0000","last_active":"2024-09-16T00:15:49.178868+0000","last_peered":"2024-09-16T00:15:49.178868+0000","last_clean":"2024-09-16T00:15:49.178868+0000","last_became_active":"2024-09-16T00:15:04.342029+0000","last_became_peered":"2024-09-16T00:15:04.342029+0000","last_unstale":"2024-09-16T00:15:49.178868+0000","last_undegraded":"2024-09-16T00:15:49.178868+0000","last_fullsized":"2024-09-16T00:15:49.178868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:55:21.012262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570766+0000","last_change":"2024-09-16T00:15:06.547094+0000","last_active":"2024-09-16T00:15:16.570766+0000","last_peered":"2024-09-16T00:15:16.570766+0000","last_clean":"2024-09-16T00:15:16.570766+0000","last_became_active":"2024-09-16T00:15:06.542764+0000","last_became_peered":"2024-09-16T00:15:06.542764+0000","last_unstale":"2024-09-16T00:15:16.570766+0000","last_undegraded":"2024-09-16T00:15:16.570766+0000","last_fullsized":"2024-09-16T00:15:16.570766+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:04:37.449577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571347+0000","last_change":"2024-09-16T00:15:13.578400+0000","last_active":"2024-09-16T00:15:16.571347+0000","last_peered":"2024-09-16T00:15:16.571347+0000","last_clean":"2024-09-16T00:15:16.571347+0000","last_became_active":"2024-09-16T00:15:13.577787+0000","last_became_peered":"2024-09-16T00:15:13.577787+0000","last_unstale":"2024-09-16T00:15:16.571347+0000","last_undegraded":"2024-09-16T00:15:16.571347+0000","last_fullsized":"2024-09-16T00:15:16.571347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:19:59.194152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571117+0000","last_change":"2024-09-16T00:15:04.373381+0000","last_active":"2024-09-16T00:15:16.571117+0000","last_peered":"2024-09-16T00:15:16.571117+0000","last_clean":"2024-09-16T00:15:16.571117+0000","last_became_active":"2024-09-16T00:15:04.373028+0000","last_became_peered":"2024-09-16T00:15:04.373028+0000","last_unstale":"2024-09-16T00:15:16.571117+0000","last_undegraded":"2024-09-16T00:15:16.571117+0000","last_fullsized":"2024-09-16T00:15:16.571117+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:22.902583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717843+0000","last_change":"2024-09-16T00:15:06.554048+0000","last_active":"2024-09-16T00:15:16.717843+0000","last_peered":"2024-09-16T00:15:16.717843+0000","last_clean":"2024-09-16T00:15:16.717843+0000","last_became_active":"2024-09-16T00:15:06.553044+0000","last_became_peered":"2024-09-16T00:15:06.553044+0000","last_unstale":"2024-09-16T00:15:16.717843+0000","last_undegraded":"2024-09-16T00:15:16.717843+0000","last_fullsized":"2024-09-16T00:15:16.717843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:22:53.333705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717271+0000","last_change":"2024-09-16T00:15:13.586995+0000","last_active":"2024-09-16T00:15:16.717271+0000","last_peered":"2024-09-16T00:15:16.717271+0000","last_clean":"2024-09-16T00:15:16.717271+0000","last_became_active":"2024-09-16T00:15:13.586509+0000","last_became_peered":"2024-09-16T00:15:13.586509+0000","last_unstale":"2024-09-16T00:15:16.717271+0000","last_undegraded":"2024-09-16T00:15:16.717271+0000","last_fullsized":"2024-09-16T00:15:16.717271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:39.418581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.177512+0000","last_change":"2024-09-16T00:15:04.348235+0000","last_active":"2024-09-16T00:15:49.177512+0000","last_peered":"2024-09-16T00:15:49.177512+0000","last_clean":"2024-09-16T00:15:49.177512+0000","last_became_active":"2024-09-16T00:15:04.347518+0000","last_became_peered":"2024-09-16T00:15:04.347518+0000","last_unstale":"2024-09-16T00:15:49.177512+0000","last_undegraded":"2024-09-16T00:15:49.177512+0000","last_fullsized":"2024-09-16T00:15:49.177512+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:24.396911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718974+0000","last_change":"2024-09-16T00:15:06.554067+0000","last_active":"2024-09-16T00:15:16.718974+0000","last_peered":"2024-09-16T00:15:16.718974+0000","last_clean":"2024-09-16T00:15:16.718974+0000","last_became_active":"2024-09-16T00:15:06.553715+0000","last_became_peered":"2024-09-16T00:15:06.553715+0000","last_unstale":"2024-09-16T00:15:16.718974+0000","last_undegraded":"2024-09-16T00:15:16.718974+0000","last_fullsized":"2024-09-16T00:15:16.718974+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:22.008530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718992+0000","last_change":"2024-09-16T00:15:13.586872+0000","last_active":"2024-09-16T00:15:16.718992+0000","last_peered":"2024-09-16T00:15:16.718992+0000","last_clean":"2024-09-16T00:15:16.718992+0000","last_became_active":"2024-09-16T00:15:13.586375+0000","last_became_peered":"2024-09-16T00:15:13.586375+0000","last_unstale":"2024-09-16T00:15:16.718992+0000","last_undegraded":"2024-09-16T00:15:16.718992+0000","last_fullsized":"2024-09-16T00:15:16.718992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:43.668827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.859671+0000","last_change":"2024-09-16T00:15:04.347964+0000","last_active":"2024-09-16T00:15:16.859671+0000","last_peered":"2024-09-16T00:15:16.859671+0000","last_clean":"2024-09-16T00:15:16.859671+0000","last_became_active":"2024-09-16T00:15:04.347588+0000","last_became_peered":"2024-09-16T00:15:04.347588+0000","last_unstale":"2024-09-16T00:15:16.859671+0000","last_undegraded":"2024-09-16T00:15:16.859671+0000","last_fullsized":"2024-09-16T00:15:16.859671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:14.199287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718701+0000","last_change":"2024-09-16T00:15:06.544918+0000","last_active":"2024-09-16T00:15:16.718701+0000","last_peered":"2024-09-16T00:15:16.718701+0000","last_clean":"2024-09-16T00:15:16.718701+0000","last_became_active":"2024-09-16T00:15:06.544579+0000","last_became_peered":"2024-09-16T00:15:06.544579+0000","last_unstale":"2024-09-16T00:15:16.718701+0000","last_undegraded":"2024-09-16T00:15:16.718701+0000","last_fullsized":"2024-09-16T00:15:16.718701+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:06:37.475644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717546+0000","last_change":"2024-09-16T00:15:13.581488+0000","last_active":"2024-09-16T00:15:16.717546+0000","last_peered":"2024-09-16T00:15:16.717546+0000","last_clean":"2024-09-16T00:15:16.717546+0000","last_became_active":"2024-09-16T00:15:13.581164+0000","last_became_peered":"2024-09-16T00:15:13.581164+0000","last_unstale":"2024-09-16T00:15:16.717546+0000","last_undegraded":"2024-09-16T00:15:16.717546+0000","last_fullsized":"2024-09-16T00:15:16.717546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:05.061592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571396+0000","last_change":"2024-09-16T00:15:06.551982+0000","last_active":"2024-09-16T00:15:16.571396+0000","last_peered":"2024-09-16T00:15:16.571396+0000","last_clean":"2024-09-16T00:15:16.571396+0000","last_became_active":"2024-09-16T00:15:06.551663+0000","last_became_peered":"2024-09-16T00:15:06.551663+0000","last_unstale":"2024-09-16T00:15:16.571396+0000","last_undegraded":"2024-09-16T00:15:16.571396+0000","last_fullsized":"2024-09-16T00:15:16.571396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:11.928703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.173852+0000","last_change":"2024-09-16T00:15:04.352237+0000","last_active":"2024-09-16T00:15:48.173852+0000","last_peered":"2024-09-16T00:15:48.173852+0000","last_clean":"2024-09-16T00:15:48.173852+0000","last_became_active":"2024-09-16T00:15:04.351884+0000","last_became_peered":"2024-09-16T00:15:04.351884+0000","last_unstale":"2024-09-16T00:15:48.173852+0000","last_undegraded":"2024-09-16T00:15:48.173852+0000","last_fullsized":"2024-09-16T00:15:48.173852+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:32.932020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717602+0000","last_change":"2024-09-16T00:15:13.638917+0000","last_active":"2024-09-16T00:15:16.717602+0000","last_peered":"2024-09-16T00:15:16.717602+0000","last_clean":"2024-09-16T00:15:16.717602+0000","last_became_active":"2024-09-16T00:15:13.638401+0000","last_became_peered":"2024-09-16T00:15:13.638401+0000","last_unstale":"2024-09-16T00:15:16.717602+0000","last_undegraded":"2024-09-16T00:15:16.717602+0000","last_fullsized":"2024-09-16T00:15:16.717602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:38.082407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570732+0000","last_change":"2024-09-16T00:15:06.547909+0000","last_active":"2024-09-16T00:15:16.570732+0000","last_peered":"2024-09-16T00:15:16.570732+0000","last_clean":"2024-09-16T00:15:16.570732+0000","last_became_active":"2024-09-16T00:15:06.546814+0000","last_became_peered":"2024-09-16T00:15:06.546814+0000","last_unstale":"2024-09-16T00:15:16.570732+0000","last_undegraded":"2024-09-16T00:15:16.570732+0000","last_fullsized":"2024-09-16T00:15:16.570732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:46.419406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571418+0000","last_change":"2024-09-16T00:15:04.349796+0000","last_active":"2024-09-16T00:15:16.571418+0000","last_peered":"2024-09-16T00:15:16.571418+0000","last_clean":"2024-09-16T00:15:16.571418+0000","last_became_active":"2024-09-16T00:15:04.349435+0000","last_became_peered":"2024-09-16T00:15:04.349435+0000","last_unstale":"2024-09-16T00:15:16.571418+0000","last_undegraded":"2024-09-16T00:15:16.571418+0000","last_fullsized":"2024-09-16T00:15:16.571418+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:09:43.702956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717958+0000","last_change":"2024-09-16T00:15:13.588345+0000","last_active":"2024-09-16T00:15:16.717958+0000","last_peered":"2024-09-16T00:15:16.717958+0000","last_clean":"2024-09-16T00:15:16.717958+0000","last_became_active":"2024-09-16T00:15:13.587829+0000","last_became_peered":"2024-09-16T00:15:13.587829+0000","last_unstale":"2024-09-16T00:15:16.717958+0000","last_undegraded":"2024-09-16T00:15:16.717958+0000","last_fullsized":"2024-09-16T00:15:16.717958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:04.261031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717528+0000","last_change":"2024-09-16T00:15:06.545485+0000","last_active":"2024-09-16T00:15:16.717528+0000","last_peered":"2024-09-16T00:15:16.717528+0000","last_clean":"2024-09-16T00:15:16.717528+0000","last_became_active":"2024-09-16T00:15:06.545178+0000","last_became_peered":"2024-09-16T00:15:06.545178+0000","last_unstale":"2024-09-16T00:15:16.717528+0000","last_undegraded":"2024-09-16T00:15:16.717528+0000","last_fullsized":"2024-09-16T00:15:16.717528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:26.923469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.172469+0000","last_change":"2024-09-16T00:15:04.340546+0000","last_active":"2024-09-16T00:15:48.172469+0000","last_peered":"2024-09-16T00:15:48.172469+0000","last_clean":"2024-09-16T00:15:48.172469+0000","last_became_active":"2024-09-16T00:15:04.340173+0000","last_became_peered":"2024-09-16T00:15:04.340173+0000","last_unstale":"2024-09-16T00:15:48.172469+0000","last_undegraded":"2024-09-16T00:15:48.172469+0000","last_fullsized":"2024-09-16T00:15:48.172469+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:35:01.134433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718366+0000","last_change":"2024-09-16T00:15:13.589963+0000","last_active":"2024-09-16T00:15:16.718366+0000","last_peered":"2024-09-16T00:15:16.718366+0000","last_clean":"2024-09-16T00:15:16.718366+0000","last_became_active":"2024-09-16T00:15:13.589615+0000","last_became_peered":"2024-09-16T00:15:13.589615+0000","last_unstale":"2024-09-16T00:15:16.718366+0000","last_undegraded":"2024-09-16T00:15:16.718366+0000","last_fullsized":"2024-09-16T00:15:16.718366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:37.197765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.859080+0000","last_change":"2024-09-16T00:15:06.544374+0000","last_active":"2024-09-16T00:15:16.859080+0000","last_peered":"2024-09-16T00:15:16.859080+0000","last_clean":"2024-09-16T00:15:16.859080+0000","last_became_active":"2024-09-16T00:15:06.544027+0000","last_became_peered":"2024-09-16T00:15:06.544027+0000","last_unstale":"2024-09-16T00:15:16.859080+0000","last_undegraded":"2024-09-16T00:15:16.859080+0000","last_fullsized":"2024-09-16T00:15:16.859080+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.172879+0000","last_change":"2024-09-16T00:15:04.374209+0000","last_active":"2024-09-16T00:15:48.172879+0000","last_peered":"2024-09-16T00:15:48.172879+0000","last_clean":"2024-09-16T00:15:48.172879+0000","last_became_active":"2024-09-16T00:15:04.373814+0000","last_became_peered":"2024-09-16T00:15:04.373814+0000","last_unstale":"2024-09-16T00:15:48.172879+0000","last_undegraded":"2024-09-16T00:15:48.172879+0000","last_fullsized":"2024-09-16T00:15:48.172879+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:13.516161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717939+0000","last_change":"2024-09-16T00:15:13.639210+0000","last_active":"2024-09-16T00:15:16.717939+0000","last_peered":"2024-09-16T00:15:16.717939+0000","last_clean":"2024-09-16T00:15:16.717939+0000","last_became_active":"2024-09-16T00:15:13.638719+0000","last_became_peered":"2024-09-16T00:15:13.638719+0000","last_unstale":"2024-09-16T00:15:16.717939+0000","last_undegraded":"2024-09-16T00:15:16.717939+0000","last_fullsized":"2024-09-16T00:15:16.717939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:26.130778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717624+0000","last_change":"2024-09-16T00:15:06.553108+0000","last_active":"2024-09-16T00:15:16.717624+0000","last_peered":"2024-09-16T00:15:16.717624+0000","last_clean":"2024-09-16T00:15:16.717624+0000","last_became_active":"2024-09-16T00:15:06.552817+0000","last_became_peered":"2024-09-16T00:15:06.552817+0000","last_unstale":"2024-09-16T00:15:16.717624+0000","last_undegraded":"2024-09-16T00:15:16.717624+0000","last_fullsized":"2024-09-16T00:15:16.717624+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:06.821337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.172594+0000","last_change":"2024-09-16T00:15:04.644185+0000","last_active":"2024-09-16T00:15:48.172594+0000","last_peered":"2024-09-16T00:15:48.172594+0000","last_clean":"2024-09-16T00:15:48.172594+0000","last_became_active":"2024-09-16T00:15:04.643887+0000","last_became_peered":"2024-09-16T00:15:04.643887+0000","last_unstale":"2024-09-16T00:15:48.172594+0000","last_undegraded":"2024-09-16T00:15:48.172594+0000","last_fullsized":"2024-09-16T00:15:48.172594+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:44.279272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571611+0000","last_change":"2024-09-16T00:15:13.574761+0000","last_active":"2024-09-16T00:15:16.571611+0000","last_peered":"2024-09-16T00:15:16.571611+0000","last_clean":"2024-09-16T00:15:16.571611+0000","last_became_active":"2024-09-16T00:15:13.574347+0000","last_became_peered":"2024-09-16T00:15:13.574347+0000","last_unstale":"2024-09-16T00:15:16.571611+0000","last_undegraded":"2024-09-16T00:15:16.571611+0000","last_fullsized":"2024-09-16T00:15:16.571611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:22.040121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571044+0000","last_change":"2024-09-16T00:15:06.550338+0000","last_active":"2024-09-16T00:15:16.571044+0000","last_peered":"2024-09-16T00:15:16.571044+0000","last_clean":"2024-09-16T00:15:16.571044+0000","last_became_active":"2024-09-16T00:15:06.549868+0000","last_became_peered":"2024-09-16T00:15:06.549868+0000","last_unstale":"2024-09-16T00:15:16.571044+0000","last_undegraded":"2024-09-16T00:15:16.571044+0000","last_fullsized":"2024-09-16T00:15:16.571044+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:26.643338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:16:01.543351+0000","last_change":"2024-09-16T00:15:04.342306+0000","last_active":"2024-09-16T00:16:01.543351+0000","last_peered":"2024-09-16T00:16:01.543351+0000","last_clean":"2024-09-16T00:16:01.543351+0000","last_became_active":"2024-09-16T00:15:04.341964+0000","last_became_peered":"2024-09-16T00:15:04.341964+0000","last_unstale":"2024-09-16T00:16:01.543351+0000","last_undegraded":"2024-09-16T00:16:01.543351+0000","last_fullsized":"2024-09-16T00:16:01.543351+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:14.137199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570459+0000","last_change":"2024-09-16T00:15:13.638246+0000","last_active":"2024-09-16T00:15:16.570459+0000","last_peered":"2024-09-16T00:15:16.570459+0000","last_clean":"2024-09-16T00:15:16.570459+0000","last_became_active":"2024-09-16T00:15:13.637816+0000","last_became_peered":"2024-09-16T00:15:13.637816+0000","last_unstale":"2024-09-16T00:15:16.570459+0000","last_undegraded":"2024-09-16T00:15:16.570459+0000","last_fullsized":"2024-09-16T00:15:16.570459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:27.331156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718856+0000","last_change":"2024-09-16T00:15:06.544057+0000","last_active":"2024-09-16T00:15:16.718856+0000","last_peered":"2024-09-16T00:15:16.718856+0000","last_clean":"2024-09-16T00:15:16.718856+0000","last_became_active":"2024-09-16T00:15:06.543244+0000","last_became_peered":"2024-09-16T00:15:06.543244+0000","last_unstale":"2024-09-16T00:15:16.718856+0000","last_undegraded":"2024-09-16T00:15:16.718856+0000","last_fullsized":"2024-09-16T00:15:16.718856+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:42:28.950014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.173500+0000","last_change":"2024-09-16T00:15:04.376419+0000","last_active":"2024-09-16T00:15:48.173500+0000","last_peered":"2024-09-16T00:15:48.173500+0000","last_clean":"2024-09-16T00:15:48.173500+0000","last_became_active":"2024-09-16T00:15:04.375976+0000","last_became_peered":"2024-09-16T00:15:04.375976+0000","last_unstale":"2024-09-16T00:15:48.173500+0000","last_undegraded":"2024-09-16T00:15:48.173500+0000","last_fullsized":"2024-09-16T00:15:48.173500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:33:38.652109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717793+0000","last_change":"2024-09-16T00:15:13.584306+0000","last_active":"2024-09-16T00:15:16.717793+0000","last_peered":"2024-09-16T00:15:16.717793+0000","last_clean":"2024-09-16T00:15:16.717793+0000","last_became_active":"2024-09-16T00:15:13.583840+0000","last_became_peered":"2024-09-16T00:15:13.583840+0000","last_unstale":"2024-09-16T00:15:16.717793+0000","last_undegraded":"2024-09-16T00:15:16.717793+0000","last_fullsized":"2024-09-16T00:15:16.717793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:36.660458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718524+0000","last_change":"2024-09-16T00:15:06.538902+0000","last_active":"2024-09-16T00:15:16.718524+0000","last_peered":"2024-09-16T00:15:16.718524+0000","last_clean":"2024-09-16T00:15:16.718524+0000","last_became_active":"2024-09-16T00:15:06.538537+0000","last_became_peered":"2024-09-16T00:15:06.538537+0000","last_unstale":"2024-09-16T00:15:16.718524+0000","last_undegraded":"2024-09-16T00:15:16.718524+0000","last_fullsized":"2024-09-16T00:15:16.718524+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:38.411711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718017+0000","last_change":"2024-09-16T00:15:04.374762+0000","last_active":"2024-09-16T00:15:16.718017+0000","last_peered":"2024-09-16T00:15:16.718017+0000","last_clean":"2024-09-16T00:15:16.718017+0000","last_became_active":"2024-09-16T00:15:04.374394+0000","last_became_peered":"2024-09-16T00:15:04.374394+0000","last_unstale":"2024-09-16T00:15:16.718017+0000","last_undegraded":"2024-09-16T00:15:16.718017+0000","last_fullsized":"2024-09-16T00:15:16.718017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:47.075021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.859226+0000","last_change":"2024-09-16T00:15:13.638306+0000","last_active":"2024-09-16T00:15:16.859226+0000","last_peered":"2024-09-16T00:15:16.859226+0000","last_clean":"2024-09-16T00:15:16.859226+0000","last_became_active":"2024-09-16T00:15:13.637931+0000","last_became_peered":"2024-09-16T00:15:13.637931+0000","last_unstale":"2024-09-16T00:15:16.859226+0000","last_undegraded":"2024-09-16T00:15:16.859226+0000","last_fullsized":"2024-09-16T00:15:16.859226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:41.385174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717975+0000","last_change":"2024-09-16T00:15:06.548209+0000","last_active":"2024-09-16T00:15:16.717975+0000","last_peered":"2024-09-16T00:15:16.717975+0000","last_clean":"2024-09-16T00:15:16.717975+0000","last_became_active":"2024-09-16T00:15:06.547846+0000","last_became_peered":"2024-09-16T00:15:06.547846+0000","last_unstale":"2024-09-16T00:15:16.717975+0000","last_undegraded":"2024-09-16T00:15:16.717975+0000","last_fullsized":"2024-09-16T00:15:16.717975+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:15.821847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571615+0000","last_change":"2024-09-16T00:15:04.342403+0000","last_active":"2024-09-16T00:15:16.571615+0000","last_peered":"2024-09-16T00:15:16.571615+0000","last_clean":"2024-09-16T00:15:16.571615+0000","last_became_active":"2024-09-16T00:15:04.342064+0000","last_became_peered":"2024-09-16T00:15:04.342064+0000","last_unstale":"2024-09-16T00:15:16.571615+0000","last_undegraded":"2024-09-16T00:15:16.571615+0000","last_fullsized":"2024-09-16T00:15:16.571615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:36.706776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718700+0000","last_change":"2024-09-16T00:15:13.579387+0000","last_active":"2024-09-16T00:15:16.718700+0000","last_peered":"2024-09-16T00:15:16.718700+0000","last_clean":"2024-09-16T00:15:16.718700+0000","last_became_active":"2024-09-16T00:15:13.578996+0000","last_became_peered":"2024-09-16T00:15:13.578996+0000","last_unstale":"2024-09-16T00:15:16.718700+0000","last_undegraded":"2024-09-16T00:15:16.718700+0000","last_fullsized":"2024-09-16T00:15:16.718700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:51.703060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717862+0000","last_change":"2024-09-16T00:15:06.540396+0000","last_active":"2024-09-16T00:15:16.717862+0000","last_peered":"2024-09-16T00:15:16.717862+0000","last_clean":"2024-09-16T00:15:16.717862+0000","last_became_active":"2024-09-16T00:15:06.540084+0000","last_became_peered":"2024-09-16T00:15:06.540084+0000","last_unstale":"2024-09-16T00:15:16.717862+0000","last_undegraded":"2024-09-16T00:15:16.717862+0000","last_fullsized":"2024-09-16T00:15:16.717862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:46.477143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:16:04.202657+0000","last_change":"2024-09-16T00:15:04.373896+0000","last_active":"2024-09-16T00:16:04.202657+0000","last_peered":"2024-09-16T00:16:04.202657+0000","last_clean":"2024-09-16T00:16:04.202657+0000","last_became_active":"2024-09-16T00:15:04.373493+0000","last_became_peered":"2024-09-16T00:15:04.373493+0000","last_unstale":"2024-09-16T00:16:04.202657+0000","last_undegraded":"2024-09-16T00:16:04.202657+0000","last_fullsized":"2024-09-16T00:16:04.202657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:17:10.767267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718093+0000","last_change":"2024-09-16T00:15:13.638575+0000","last_active":"2024-09-16T00:15:16.718093+0000","last_peered":"2024-09-16T00:15:16.718093+0000","last_clean":"2024-09-16T00:15:16.718093+0000","last_became_active":"2024-09-16T00:15:13.638144+0000","last_became_peered":"2024-09-16T00:15:13.638144+0000","last_unstale":"2024-09-16T00:15:16.718093+0000","last_undegraded":"2024-09-16T00:15:16.718093+0000","last_fullsized":"2024-09-16T00:15:16.718093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:39.345011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.716899+0000","last_change":"2024-09-16T00:15:06.548852+0000","last_active":"2024-09-16T00:15:16.716899+0000","last_peered":"2024-09-16T00:15:16.716899+0000","last_clean":"2024-09-16T00:15:16.716899+0000","last_became_active":"2024-09-16T00:15:06.548238+0000","last_became_peered":"2024-09-16T00:15:06.548238+0000","last_unstale":"2024-09-16T00:15:16.716899+0000","last_undegraded":"2024-09-16T00:15:16.716899+0000","last_fullsized":"2024-09-16T00:15:16.716899+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:16.115338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.177212+0000","last_change":"2024-09-16T00:15:04.350753+0000","last_active":"2024-09-16T00:15:49.177212+0000","last_peered":"2024-09-16T00:15:49.177212+0000","last_clean":"2024-09-16T00:15:49.177212+0000","last_became_active":"2024-09-16T00:15:04.350406+0000","last_became_peered":"2024-09-16T00:15:04.350406+0000","last_unstale":"2024-09-16T00:15:49.177212+0000","last_undegraded":"2024-09-16T00:15:49.177212+0000","last_fullsized":"2024-09-16T00:15:49.177212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:09:54.737287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718316+0000","last_change":"2024-09-16T00:15:13.639302+0000","last_active":"2024-09-16T00:15:16.718316+0000","last_peered":"2024-09-16T00:15:16.718316+0000","last_clean":"2024-09-16T00:15:16.718316+0000","last_became_active":"2024-09-16T00:15:13.638863+0000","last_became_peered":"2024-09-16T00:15:13.638863+0000","last_unstale":"2024-09-16T00:15:16.718316+0000","last_undegraded":"2024-09-16T00:15:16.718316+0000","last_fullsized":"2024-09-16T00:15:16.718316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:23.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570789+0000","last_change":"2024-09-16T00:15:06.545797+0000","last_active":"2024-09-16T00:15:16.570789+0000","last_peered":"2024-09-16T00:15:16.570789+0000","last_clean":"2024-09-16T00:15:16.570789+0000","last_became_active":"2024-09-16T00:15:06.545477+0000","last_became_peered":"2024-09-16T00:15:06.545477+0000","last_unstale":"2024-09-16T00:15:16.570789+0000","last_undegraded":"2024-09-16T00:15:16.570789+0000","last_fullsized":"2024-09-16T00:15:16.570789+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.887183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:50.181523+0000","last_change":"2024-09-16T00:15:04.342216+0000","last_active":"2024-09-16T00:15:50.181523+0000","last_peered":"2024-09-16T00:15:50.181523+0000","last_clean":"2024-09-16T00:15:50.181523+0000","last_became_active":"2024-09-16T00:15:04.341763+0000","last_became_peered":"2024-09-16T00:15:04.341763+0000","last_unstale":"2024-09-16T00:15:50.181523+0000","last_undegraded":"2024-09-16T00:15:50.181523+0000","last_fullsized":"2024-09-16T00:15:50.181523+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:44.938527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":341,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571538+0000","last_change":"2024-09-16T00:13:51.108571+0000","last_active":"2024-09-16T00:15:16.571538+0000","last_peered":"2024-09-16T00:15:16.571538+0000","last_clean":"2024-09-16T00:15:16.571538+0000","last_became_active":"2024-09-16T00:13:51.108088+0000","last_became_peered":"2024-09-16T00:13:51.108088+0000","last_unstale":"2024-09-16T00:15:16.571538+0000","last_undegraded":"2024-09-16T00:15:16.571538+0000","last_fullsized":"2024-09-16T00:15:16.571538+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_clean_scrub_stamp":"2024-09-16T00:10:24.533807+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:02.511064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571863+0000","last_change":"2024-09-16T00:15:13.588665+0000","last_active":"2024-09-16T00:15:16.571863+0000","last_peered":"2024-09-16T00:15:16.571863+0000","last_clean":"2024-09-16T00:15:16.571863+0000","last_became_active":"2024-09-16T00:15:13.588222+0000","last_became_peered":"2024-09-16T00:15:13.588222+0000","last_unstale":"2024-09-16T00:15:16.571863+0000","last_undegraded":"2024-09-16T00:15:16.571863+0000","last_fullsized":"2024-09-16T00:15:16.571863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:59:59.965962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571925+0000","last_change":"2024-09-16T00:15:06.553114+0000","last_active":"2024-09-16T00:15:16.571925+0000","last_peered":"2024-09-16T00:15:16.571925+0000","last_clean":"2024-09-16T00:15:16.571925+0000","last_became_active":"2024-09-16T00:15:06.552788+0000","last_became_peered":"2024-09-16T00:15:06.552788+0000","last_unstale":"2024-09-16T00:15:16.571925+0000","last_undegraded":"2024-09-16T00:15:16.571925+0000","last_fullsized":"2024-09-16T00:15:16.571925+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:05.473847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.180950+0000","last_change":"2024-09-16T00:15:04.644100+0000","last_active":"2024-09-16T00:15:49.180950+0000","last_peered":"2024-09-16T00:15:49.180950+0000","last_clean":"2024-09-16T00:15:49.180950+0000","last_became_active":"2024-09-16T00:15:04.643725+0000","last_became_peered":"2024-09-16T00:15:04.643725+0000","last_unstale":"2024-09-16T00:15:49.180950+0000","last_undegraded":"2024-09-16T00:15:49.180950+0000","last_fullsized":"2024-09-16T00:15:49.180950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:15.482080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718471+0000","last_change":"2024-09-16T00:15:13.575606+0000","last_active":"2024-09-16T00:15:16.718471+0000","last_peered":"2024-09-16T00:15:16.718471+0000","last_clean":"2024-09-16T00:15:16.718471+0000","last_became_active":"2024-09-16T00:15:13.575131+0000","last_became_peered":"2024-09-16T00:15:13.575131+0000","last_unstale":"2024-09-16T00:15:16.718471+0000","last_undegraded":"2024-09-16T00:15:16.718471+0000","last_fullsized":"2024-09-16T00:15:16.718471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:41.636058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571578+0000","last_change":"2024-09-16T00:15:06.552956+0000","last_active":"2024-09-16T00:15:16.571578+0000","last_peered":"2024-09-16T00:15:16.571578+0000","last_clean":"2024-09-16T00:15:16.571578+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:15:16.571578+0000","last_undegraded":"2024-09-16T00:15:16.571578+0000","last_fullsized":"2024-09-16T00:15:16.571578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:36:20.739676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718949+0000","last_change":"2024-09-16T00:15:04.349095+0000","last_active":"2024-09-16T00:15:16.718949+0000","last_peered":"2024-09-16T00:15:16.718949+0000","last_clean":"2024-09-16T00:15:16.718949+0000","last_became_active":"2024-09-16T00:15:04.348698+0000","last_became_peered":"2024-09-16T00:15:04.348698+0000","last_unstale":"2024-09-16T00:15:16.718949+0000","last_undegraded":"2024-09-16T00:15:16.718949+0000","last_fullsized":"2024-09-16T00:15:16.718949+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:42.282882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718208+0000","last_change":"2024-09-16T00:15:13.580494+0000","last_active":"2024-09-16T00:15:16.718208+0000","last_peered":"2024-09-16T00:15:16.718208+0000","last_clean":"2024-09-16T00:15:16.718208+0000","last_became_active":"2024-09-16T00:15:13.580091+0000","last_became_peered":"2024-09-16T00:15:13.580091+0000","last_unstale":"2024-09-16T00:15:16.718208+0000","last_undegraded":"2024-09-16T00:15:16.718208+0000","last_fullsized":"2024-09-16T00:15:16.718208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:17.314919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571885+0000","last_change":"2024-09-16T00:15:06.544859+0000","last_active":"2024-09-16T00:15:16.571885+0000","last_peered":"2024-09-16T00:15:16.571885+0000","last_clean":"2024-09-16T00:15:16.571885+0000","last_became_active":"2024-09-16T00:15:06.544289+0000","last_became_peered":"2024-09-16T00:15:06.544289+0000","last_unstale":"2024-09-16T00:15:16.571885+0000","last_undegraded":"2024-09-16T00:15:16.571885+0000","last_fullsized":"2024-09-16T00:15:16.571885+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:52.622477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718547+0000","last_change":"2024-09-16T00:15:04.351313+0000","last_active":"2024-09-16T00:15:16.718547+0000","last_peered":"2024-09-16T00:15:16.718547+0000","last_clean":"2024-09-16T00:15:16.718547+0000","last_became_active":"2024-09-16T00:15:04.350928+0000","last_became_peered":"2024-09-16T00:15:04.350928+0000","last_unstale":"2024-09-16T00:15:16.718547+0000","last_undegraded":"2024-09-16T00:15:16.718547+0000","last_fullsized":"2024-09-16T00:15:16.718547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:48.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717654+0000","last_change":"2024-09-16T00:15:13.578573+0000","last_active":"2024-09-16T00:15:16.717654+0000","last_peered":"2024-09-16T00:15:16.717654+0000","last_clean":"2024-09-16T00:15:16.717654+0000","last_became_active":"2024-09-16T00:15:13.578117+0000","last_became_peered":"2024-09-16T00:15:13.578117+0000","last_unstale":"2024-09-16T00:15:16.717654+0000","last_undegraded":"2024-09-16T00:15:16.717654+0000","last_fullsized":"2024-09-16T00:15:16.717654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:53.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718552+0000","last_change":"2024-09-16T00:15:06.553548+0000","last_active":"2024-09-16T00:15:16.718552+0000","last_peered":"2024-09-16T00:15:16.718552+0000","last_clean":"2024-09-16T00:15:16.718552+0000","last_became_active":"2024-09-16T00:15:06.552963+0000","last_became_peered":"2024-09-16T00:15:06.552963+0000","last_unstale":"2024-09-16T00:15:16.718552+0000","last_undegraded":"2024-09-16T00:15:16.718552+0000","last_fullsized":"2024-09-16T00:15:16.718552+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:43.455881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:50.181698+0000","last_change":"2024-09-16T00:15:04.351005+0000","last_active":"2024-09-16T00:15:50.181698+0000","last_peered":"2024-09-16T00:15:50.181698+0000","last_clean":"2024-09-16T00:15:50.181698+0000","last_became_active":"2024-09-16T00:15:04.350614+0000","last_became_peered":"2024-09-16T00:15:04.350614+0000","last_unstale":"2024-09-16T00:15:50.181698+0000","last_undegraded":"2024-09-16T00:15:50.181698+0000","last_fullsized":"2024-09-16T00:15:50.181698+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:39.196360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717921+0000","last_change":"2024-09-16T00:15:13.578466+0000","last_active":"2024-09-16T00:15:16.717921+0000","last_peered":"2024-09-16T00:15:16.717921+0000","last_clean":"2024-09-16T00:15:16.717921+0000","last_became_active":"2024-09-16T00:15:13.578078+0000","last_became_peered":"2024-09-16T00:15:13.578078+0000","last_unstale":"2024-09-16T00:15:16.717921+0000","last_undegraded":"2024-09-16T00:15:16.717921+0000","last_fullsized":"2024-09-16T00:15:16.717921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:41:20.439086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718413+0000","last_change":"2024-09-16T00:15:06.546325+0000","last_active":"2024-09-16T00:15:16.718413+0000","last_peered":"2024-09-16T00:15:16.718413+0000","last_clean":"2024-09-16T00:15:16.718413+0000","last_became_active":"2024-09-16T00:15:06.546028+0000","last_became_peered":"2024-09-16T00:15:06.546028+0000","last_unstale":"2024-09-16T00:15:16.718413+0000","last_undegraded":"2024-09-16T00:15:16.718413+0000","last_fullsized":"2024-09-16T00:15:16.718413+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:15.049592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.179258+0000","last_change":"2024-09-16T00:15:04.373818+0000","last_active":"2024-09-16T00:15:49.179258+0000","last_peered":"2024-09-16T00:15:49.179258+0000","last_clean":"2024-09-16T00:15:49.179258+0000","last_became_active":"2024-09-16T00:15:04.373406+0000","last_became_peered":"2024-09-16T00:15:04.373406+0000","last_unstale":"2024-09-16T00:15:49.179258+0000","last_undegraded":"2024-09-16T00:15:49.179258+0000","last_fullsized":"2024-09-16T00:15:49.179258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:08.402660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718433+0000","last_change":"2024-09-16T00:15:13.588669+0000","last_active":"2024-09-16T00:15:16.718433+0000","last_peered":"2024-09-16T00:15:16.718433+0000","last_clean":"2024-09-16T00:15:16.718433+0000","last_became_active":"2024-09-16T00:15:13.588229+0000","last_became_peered":"2024-09-16T00:15:13.588229+0000","last_unstale":"2024-09-16T00:15:16.718433+0000","last_undegraded":"2024-09-16T00:15:16.718433+0000","last_fullsized":"2024-09-16T00:15:16.718433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:30.157296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571181+0000","last_change":"2024-09-16T00:15:06.550221+0000","last_active":"2024-09-16T00:15:16.571181+0000","last_peered":"2024-09-16T00:15:16.571181+0000","last_clean":"2024-09-16T00:15:16.571181+0000","last_became_active":"2024-09-16T00:15:06.549914+0000","last_became_peered":"2024-09-16T00:15:06.549914+0000","last_unstale":"2024-09-16T00:15:16.571181+0000","last_undegraded":"2024-09-16T00:15:16.571181+0000","last_fullsized":"2024-09-16T00:15:16.571181+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:18:26.824523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":51,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:16:03.201845+0000","last_change":"2024-09-16T00:15:04.344389+0000","last_active":"2024-09-16T00:16:03.201845+0000","last_peered":"2024-09-16T00:16:03.201845+0000","last_clean":"2024-09-16T00:16:03.201845+0000","last_became_active":"2024-09-16T00:15:04.343956+0000","last_became_peered":"2024-09-16T00:15:04.343956+0000","last_unstale":"2024-09-16T00:16:03.201845+0000","last_undegraded":"2024-09-16T00:16:03.201845+0000","last_fullsized":"2024-09-16T00:16:03.201845+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:34.019861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718417+0000","last_change":"2024-09-16T00:15:13.586043+0000","last_active":"2024-09-16T00:15:16.718417+0000","last_peered":"2024-09-16T00:15:16.718417+0000","last_clean":"2024-09-16T00:15:16.718417+0000","last_became_active":"2024-09-16T00:15:13.583872+0000","last_became_peered":"2024-09-16T00:15:13.583872+0000","last_unstale":"2024-09-16T00:15:16.718417+0000","last_undegraded":"2024-09-16T00:15:16.718417+0000","last_fullsized":"2024-09-16T00:15:16.718417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:12.160443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571491+0000","last_change":"2024-09-16T00:15:06.542744+0000","last_active":"2024-09-16T00:15:16.571491+0000","last_peered":"2024-09-16T00:15:16.571491+0000","last_clean":"2024-09-16T00:15:16.571491+0000","last_became_active":"2024-09-16T00:15:06.541868+0000","last_became_peered":"2024-09-16T00:15:06.541868+0000","last_unstale":"2024-09-16T00:15:16.571491+0000","last_undegraded":"2024-09-16T00:15:16.571491+0000","last_fullsized":"2024-09-16T00:15:16.571491+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:10:06.508682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.173721+0000","last_change":"2024-09-16T00:15:04.349556+0000","last_active":"2024-09-16T00:15:48.173721+0000","last_peered":"2024-09-16T00:15:48.173721+0000","last_clean":"2024-09-16T00:15:48.173721+0000","last_became_active":"2024-09-16T00:15:04.349242+0000","last_became_peered":"2024-09-16T00:15:04.349242+0000","last_unstale":"2024-09-16T00:15:48.173721+0000","last_undegraded":"2024-09-16T00:15:48.173721+0000","last_fullsized":"2024-09-16T00:15:48.173721+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:22:18.459157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717967+0000","last_change":"2024-09-16T00:15:13.584109+0000","last_active":"2024-09-16T00:15:16.717967+0000","last_peered":"2024-09-16T00:15:16.717967+0000","last_clean":"2024-09-16T00:15:16.717967+0000","last_became_active":"2024-09-16T00:15:13.583718+0000","last_became_peered":"2024-09-16T00:15:13.583718+0000","last_unstale":"2024-09-16T00:15:16.717967+0000","last_undegraded":"2024-09-16T00:15:16.717967+0000","last_fullsized":"2024-09-16T00:15:16.717967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:29.416254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717276+0000","last_change":"2024-09-16T00:15:06.548247+0000","last_active":"2024-09-16T00:15:16.717276+0000","last_peered":"2024-09-16T00:15:16.717276+0000","last_clean":"2024-09-16T00:15:16.717276+0000","last_became_active":"2024-09-16T00:15:06.547864+0000","last_became_peered":"2024-09-16T00:15:06.547864+0000","last_unstale":"2024-09-16T00:15:16.717276+0000","last_undegraded":"2024-09-16T00:15:16.717276+0000","last_fullsized":"2024-09-16T00:15:16.717276+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:25.350197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.178625+0000","last_change":"2024-09-16T00:15:04.350965+0000","last_active":"2024-09-16T00:15:49.178625+0000","last_peered":"2024-09-16T00:15:49.178625+0000","last_clean":"2024-09-16T00:15:49.178625+0000","last_became_active":"2024-09-16T00:15:04.349124+0000","last_became_peered":"2024-09-16T00:15:04.349124+0000","last_unstale":"2024-09-16T00:15:49.178625+0000","last_undegraded":"2024-09-16T00:15:49.178625+0000","last_fullsized":"2024-09-16T00:15:49.178625+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:40.859220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718757+0000","last_change":"2024-09-16T00:15:13.581718+0000","last_active":"2024-09-16T00:15:16.718757+0000","last_peered":"2024-09-16T00:15:16.718757+0000","last_clean":"2024-09-16T00:15:16.718757+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:15:16.718757+0000","last_undegraded":"2024-09-16T00:15:16.718757+0000","last_fullsized":"2024-09-16T00:15:16.718757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:08.810606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.859429+0000","last_change":"2024-09-16T00:15:06.544507+0000","last_active":"2024-09-16T00:15:16.859429+0000","last_peered":"2024-09-16T00:15:16.859429+0000","last_clean":"2024-09-16T00:15:16.859429+0000","last_became_active":"2024-09-16T00:15:06.544178+0000","last_became_peered":"2024-09-16T00:15:06.544178+0000","last_unstale":"2024-09-16T00:15:16.859429+0000","last_undegraded":"2024-09-16T00:15:16.859429+0000","last_fullsized":"2024-09-16T00:15:16.859429+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:38.494266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"86'29","reported_seq":92,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:16:02.204274+0000","last_change":"2024-09-16T00:15:04.350871+0000","last_active":"2024-09-16T00:16:02.204274+0000","last_peered":"2024-09-16T00:16:02.204274+0000","last_clean":"2024-09-16T00:16:02.204274+0000","last_became_active":"2024-09-16T00:15:04.349900+0000","last_became_peered":"2024-09-16T00:15:04.349900+0000","last_unstale":"2024-09-16T00:16:02.204274+0000","last_undegraded":"2024-09-16T00:16:02.204274+0000","last_fullsized":"2024-09-16T00:16:02.204274+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.167301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52413,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":67,"num_write":30,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718108+0000","last_change":"2024-09-16T00:15:13.585928+0000","last_active":"2024-09-16T00:15:16.718108+0000","last_peered":"2024-09-16T00:15:16.718108+0000","last_clean":"2024-09-16T00:15:16.718108+0000","last_became_active":"2024-09-16T00:15:13.582940+0000","last_became_peered":"2024-09-16T00:15:13.582940+0000","last_unstale":"2024-09-16T00:15:16.718108+0000","last_undegraded":"2024-09-16T00:15:16.718108+0000","last_fullsized":"2024-09-16T00:15:16.718108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:57:13.251978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.719088+0000","last_change":"2024-09-16T00:15:06.546238+0000","last_active":"2024-09-16T00:15:16.719088+0000","last_peered":"2024-09-16T00:15:16.719088+0000","last_clean":"2024-09-16T00:15:16.719088+0000","last_became_active":"2024-09-16T00:15:06.545919+0000","last_became_peered":"2024-09-16T00:15:06.545919+0000","last_unstale":"2024-09-16T00:15:16.719088+0000","last_undegraded":"2024-09-16T00:15:16.719088+0000","last_fullsized":"2024-09-16T00:15:16.719088+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:09.180832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.172877+0000","last_change":"2024-09-16T00:15:04.376028+0000","last_active":"2024-09-16T00:15:48.172877+0000","last_peered":"2024-09-16T00:15:48.172877+0000","last_clean":"2024-09-16T00:15:48.172877+0000","last_became_active":"2024-09-16T00:15:04.375633+0000","last_became_peered":"2024-09-16T00:15:04.375633+0000","last_unstale":"2024-09-16T00:15:48.172877+0000","last_undegraded":"2024-09-16T00:15:48.172877+0000","last_fullsized":"2024-09-16T00:15:48.172877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:44.446342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718640+0000","last_change":"2024-09-16T00:15:13.587414+0000","last_active":"2024-09-16T00:15:16.718640+0000","last_peered":"2024-09-16T00:15:16.718640+0000","last_clean":"2024-09-16T00:15:16.718640+0000","last_became_active":"2024-09-16T00:15:13.586764+0000","last_became_peered":"2024-09-16T00:15:13.586764+0000","last_unstale":"2024-09-16T00:15:16.718640+0000","last_undegraded":"2024-09-16T00:15:16.718640+0000","last_fullsized":"2024-09-16T00:15:16.718640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:15:43.047070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717946+0000","last_change":"2024-09-16T00:15:06.550864+0000","last_active":"2024-09-16T00:15:16.717946+0000","last_peered":"2024-09-16T00:15:16.717946+0000","last_clean":"2024-09-16T00:15:16.717946+0000","last_became_active":"2024-09-16T00:15:06.547956+0000","last_became_peered":"2024-09-16T00:15:06.547956+0000","last_unstale":"2024-09-16T00:15:16.717946+0000","last_undegraded":"2024-09-16T00:15:16.717946+0000","last_fullsized":"2024-09-16T00:15:16.717946+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:26.102175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:48.174309+0000","last_change":"2024-09-16T00:15:04.350349+0000","last_active":"2024-09-16T00:15:48.174309+0000","last_peered":"2024-09-16T00:15:48.174309+0000","last_clean":"2024-09-16T00:15:48.174309+0000","last_became_active":"2024-09-16T00:15:04.349951+0000","last_became_peered":"2024-09-16T00:15:04.349951+0000","last_unstale":"2024-09-16T00:15:48.174309+0000","last_undegraded":"2024-09-16T00:15:48.174309+0000","last_fullsized":"2024-09-16T00:15:48.174309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:46.000831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717366+0000","last_change":"2024-09-16T00:15:13.585906+0000","last_active":"2024-09-16T00:15:16.717366+0000","last_peered":"2024-09-16T00:15:16.717366+0000","last_clean":"2024-09-16T00:15:16.717366+0000","last_became_active":"2024-09-16T00:15:13.583664+0000","last_became_peered":"2024-09-16T00:15:13.583664+0000","last_unstale":"2024-09-16T00:15:16.717366+0000","last_undegraded":"2024-09-16T00:15:16.717366+0000","last_fullsized":"2024-09-16T00:15:16.717366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:07:29.313469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.719136+0000","last_change":"2024-09-16T00:15:06.552919+0000","last_active":"2024-09-16T00:15:16.719136+0000","last_peered":"2024-09-16T00:15:16.719136+0000","last_clean":"2024-09-16T00:15:16.719136+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:15:16.719136+0000","last_undegraded":"2024-09-16T00:15:16.719136+0000","last_fullsized":"2024-09-16T00:15:16.719136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:47:38.651837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:16:01.199012+0000","last_change":"2024-09-16T00:15:04.351116+0000","last_active":"2024-09-16T00:16:01.199012+0000","last_peered":"2024-09-16T00:16:01.199012+0000","last_clean":"2024-09-16T00:16:01.199012+0000","last_became_active":"2024-09-16T00:15:04.350728+0000","last_became_peered":"2024-09-16T00:15:04.350728+0000","last_unstale":"2024-09-16T00:16:01.199012+0000","last_undegraded":"2024-09-16T00:16:01.199012+0000","last_fullsized":"2024-09-16T00:16:01.199012+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:38.314019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570602+0000","last_change":"2024-09-16T00:15:13.575961+0000","last_active":"2024-09-16T00:15:16.570602+0000","last_peered":"2024-09-16T00:15:16.570602+0000","last_clean":"2024-09-16T00:15:16.570602+0000","last_became_active":"2024-09-16T00:15:13.575550+0000","last_became_peered":"2024-09-16T00:15:13.575550+0000","last_unstale":"2024-09-16T00:15:16.570602+0000","last_undegraded":"2024-09-16T00:15:16.570602+0000","last_fullsized":"2024-09-16T00:15:16.570602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:05.515588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717415+0000","last_change":"2024-09-16T00:15:13.583583+0000","last_active":"2024-09-16T00:15:16.717415+0000","last_peered":"2024-09-16T00:15:16.717415+0000","last_clean":"2024-09-16T00:15:16.717415+0000","last_became_active":"2024-09-16T00:15:13.583145+0000","last_became_peered":"2024-09-16T00:15:13.583145+0000","last_unstale":"2024-09-16T00:15:16.717415+0000","last_undegraded":"2024-09-16T00:15:16.717415+0000","last_fullsized":"2024-09-16T00:15:16.717415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:07.320337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717281+0000","last_change":"2024-09-16T00:15:06.552017+0000","last_active":"2024-09-16T00:15:16.717281+0000","last_peered":"2024-09-16T00:15:16.717281+0000","last_clean":"2024-09-16T00:15:16.717281+0000","last_became_active":"2024-09-16T00:15:06.551688+0000","last_became_peered":"2024-09-16T00:15:06.551688+0000","last_unstale":"2024-09-16T00:15:16.717281+0000","last_undegraded":"2024-09-16T00:15:16.717281+0000","last_fullsized":"2024-09-16T00:15:16.717281+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:39.951856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718079+0000","last_change":"2024-09-16T00:15:04.350826+0000","last_active":"2024-09-16T00:15:16.718079+0000","last_peered":"2024-09-16T00:15:16.718079+0000","last_clean":"2024-09-16T00:15:16.718079+0000","last_became_active":"2024-09-16T00:15:04.350294+0000","last_became_peered":"2024-09-16T00:15:04.350294+0000","last_unstale":"2024-09-16T00:15:16.718079+0000","last_undegraded":"2024-09-16T00:15:16.718079+0000","last_fullsized":"2024-09-16T00:15:16.718079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:49.146709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718339+0000","last_change":"2024-09-16T00:15:13.580184+0000","last_active":"2024-09-16T00:15:16.718339+0000","last_peered":"2024-09-16T00:15:16.718339+0000","last_clean":"2024-09-16T00:15:16.718339+0000","last_became_active":"2024-09-16T00:15:13.579849+0000","last_became_peered":"2024-09-16T00:15:13.579849+0000","last_unstale":"2024-09-16T00:15:16.718339+0000","last_undegraded":"2024-09-16T00:15:16.718339+0000","last_fullsized":"2024-09-16T00:15:16.718339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:40.613127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570581+0000","last_change":"2024-09-16T00:15:06.545796+0000","last_active":"2024-09-16T00:15:16.570581+0000","last_peered":"2024-09-16T00:15:16.570581+0000","last_clean":"2024-09-16T00:15:16.570581+0000","last_became_active":"2024-09-16T00:15:06.544815+0000","last_became_peered":"2024-09-16T00:15:06.544815+0000","last_unstale":"2024-09-16T00:15:16.570581+0000","last_undegraded":"2024-09-16T00:15:16.570581+0000","last_fullsized":"2024-09-16T00:15:16.570581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:51:25.921949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.177230+0000","last_change":"2024-09-16T00:15:04.340101+0000","last_active":"2024-09-16T00:15:49.177230+0000","last_peered":"2024-09-16T00:15:49.177230+0000","last_clean":"2024-09-16T00:15:49.177230+0000","last_became_active":"2024-09-16T00:15:04.339788+0000","last_became_peered":"2024-09-16T00:15:04.339788+0000","last_unstale":"2024-09-16T00:15:49.177230+0000","last_undegraded":"2024-09-16T00:15:49.177230+0000","last_fullsized":"2024-09-16T00:15:49.177230+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:08.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:49.176612+0000","last_change":"2024-09-16T00:15:04.342298+0000","last_active":"2024-09-16T00:15:49.176612+0000","last_peered":"2024-09-16T00:15:49.176612+0000","last_clean":"2024-09-16T00:15:49.176612+0000","last_became_active":"2024-09-16T00:15:04.341930+0000","last_became_peered":"2024-09-16T00:15:04.341930+0000","last_unstale":"2024-09-16T00:15:49.176612+0000","last_undegraded":"2024-09-16T00:15:49.176612+0000","last_fullsized":"2024-09-16T00:15:49.176612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:03:43.036214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718518+0000","last_change":"2024-09-16T00:15:06.547496+0000","last_active":"2024-09-16T00:15:16.718518+0000","last_peered":"2024-09-16T00:15:16.718518+0000","last_clean":"2024-09-16T00:15:16.718518+0000","last_became_active":"2024-09-16T00:15:06.547177+0000","last_became_peered":"2024-09-16T00:15:06.547177+0000","last_unstale":"2024-09-16T00:15:16.718518+0000","last_undegraded":"2024-09-16T00:15:16.718518+0000","last_fullsized":"2024-09-16T00:15:16.718518+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:13.288827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571601+0000","last_change":"2024-09-16T00:15:13.581966+0000","last_active":"2024-09-16T00:15:16.571601+0000","last_peered":"2024-09-16T00:15:16.571601+0000","last_clean":"2024-09-16T00:15:16.571601+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:15:16.571601+0000","last_undegraded":"2024-09-16T00:15:16.571601+0000","last_fullsized":"2024-09-16T00:15:16.571601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.132773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":52,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:16:03.202440+0000","last_change":"2024-09-16T00:15:04.351765+0000","last_active":"2024-09-16T00:16:03.202440+0000","last_peered":"2024-09-16T00:16:03.202440+0000","last_clean":"2024-09-16T00:16:03.202440+0000","last_became_active":"2024-09-16T00:15:04.351398+0000","last_became_peered":"2024-09-16T00:15:04.351398+0000","last_unstale":"2024-09-16T00:16:03.202440+0000","last_undegraded":"2024-09-16T00:16:03.202440+0000","last_fullsized":"2024-09-16T00:16:03.202440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:14:28.762657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.859193+0000","last_change":"2024-09-16T00:15:06.541422+0000","last_active":"2024-09-16T00:15:16.859193+0000","last_peered":"2024-09-16T00:15:16.859193+0000","last_clean":"2024-09-16T00:15:16.859193+0000","last_became_active":"2024-09-16T00:15:06.541083+0000","last_became_peered":"2024-09-16T00:15:06.541083+0000","last_unstale":"2024-09-16T00:15:16.859193+0000","last_undegraded":"2024-09-16T00:15:16.859193+0000","last_fullsized":"2024-09-16T00:15:16.859193+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:30.138972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571945+0000","last_change":"2024-09-16T00:15:13.576877+0000","last_active":"2024-09-16T00:15:16.571945+0000","last_peered":"2024-09-16T00:15:16.571945+0000","last_clean":"2024-09-16T00:15:16.571945+0000","last_became_active":"2024-09-16T00:15:13.576480+0000","last_became_peered":"2024-09-16T00:15:13.576480+0000","last_unstale":"2024-09-16T00:15:16.571945+0000","last_undegraded":"2024-09-16T00:15:16.571945+0000","last_fullsized":"2024-09-16T00:15:16.571945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:47.322219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717463+0000","last_change":"2024-09-16T00:15:06.550329+0000","last_active":"2024-09-16T00:15:16.717463+0000","last_peered":"2024-09-16T00:15:16.717463+0000","last_clean":"2024-09-16T00:15:16.717463+0000","last_became_active":"2024-09-16T00:15:06.550008+0000","last_became_peered":"2024-09-16T00:15:06.550008+0000","last_unstale":"2024-09-16T00:15:16.717463+0000","last_undegraded":"2024-09-16T00:15:16.717463+0000","last_fullsized":"2024-09-16T00:15:16.717463+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:38.096126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571940+0000","last_change":"2024-09-16T00:15:06.550368+0000","last_active":"2024-09-16T00:15:16.571940+0000","last_peered":"2024-09-16T00:15:16.571940+0000","last_clean":"2024-09-16T00:15:16.571940+0000","last_became_active":"2024-09-16T00:15:06.549985+0000","last_became_peered":"2024-09-16T00:15:06.549985+0000","last_unstale":"2024-09-16T00:15:16.571940+0000","last_undegraded":"2024-09-16T00:15:16.571940+0000","last_fullsized":"2024-09-16T00:15:16.571940+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:16.992110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571338+0000","last_change":"2024-09-16T00:15:06.546713+0000","last_active":"2024-09-16T00:15:16.571338+0000","last_peered":"2024-09-16T00:15:16.571338+0000","last_clean":"2024-09-16T00:15:16.571338+0000","last_became_active":"2024-09-16T00:15:06.546209+0000","last_became_peered":"2024-09-16T00:15:06.546209+0000","last_unstale":"2024-09-16T00:15:16.571338+0000","last_undegraded":"2024-09-16T00:15:16.571338+0000","last_fullsized":"2024-09-16T00:15:16.571338+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:43.913748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718301+0000","last_change":"2024-09-16T00:15:06.550490+0000","last_active":"2024-09-16T00:15:16.718301+0000","last_peered":"2024-09-16T00:15:16.718301+0000","last_clean":"2024-09-16T00:15:16.718301+0000","last_became_active":"2024-09-16T00:15:06.549311+0000","last_became_peered":"2024-09-16T00:15:06.549311+0000","last_unstale":"2024-09-16T00:15:16.718301+0000","last_undegraded":"2024-09-16T00:15:16.718301+0000","last_fullsized":"2024-09-16T00:15:16.718301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:10:01.995357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718836+0000","last_change":"2024-09-16T00:15:06.548115+0000","last_active":"2024-09-16T00:15:16.718836+0000","last_peered":"2024-09-16T00:15:16.718836+0000","last_clean":"2024-09-16T00:15:16.718836+0000","last_became_active":"2024-09-16T00:15:06.547782+0000","last_became_peered":"2024-09-16T00:15:06.547782+0000","last_unstale":"2024-09-16T00:15:16.718836+0000","last_undegraded":"2024-09-16T00:15:16.718836+0000","last_fullsized":"2024-09-16T00:15:16.718836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:53.228222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718442+0000","last_change":"2024-09-16T00:15:06.554396+0000","last_active":"2024-09-16T00:15:16.718442+0000","last_peered":"2024-09-16T00:15:16.718442+0000","last_clean":"2024-09-16T00:15:16.718442+0000","last_became_active":"2024-09-16T00:15:06.554012+0000","last_became_peered":"2024-09-16T00:15:06.554012+0000","last_unstale":"2024-09-16T00:15:16.718442+0000","last_undegraded":"2024-09-16T00:15:16.718442+0000","last_fullsized":"2024-09-16T00:15:16.718442+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:31.464098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571795+0000","last_change":"2024-09-16T00:15:06.549969+0000","last_active":"2024-09-16T00:15:16.571795+0000","last_peered":"2024-09-16T00:15:16.571795+0000","last_clean":"2024-09-16T00:15:16.571795+0000","last_became_active":"2024-09-16T00:15:06.549531+0000","last_became_peered":"2024-09-16T00:15:06.549531+0000","last_unstale":"2024-09-16T00:15:16.571795+0000","last_undegraded":"2024-09-16T00:15:16.571795+0000","last_fullsized":"2024-09-16T00:15:16.571795+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:37.883612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570998+0000","last_change":"2024-09-16T00:15:06.547734+0000","last_active":"2024-09-16T00:15:16.570998+0000","last_peered":"2024-09-16T00:15:16.570998+0000","last_clean":"2024-09-16T00:15:16.570998+0000","last_became_active":"2024-09-16T00:15:06.547400+0000","last_became_peered":"2024-09-16T00:15:06.547400+0000","last_unstale":"2024-09-16T00:15:16.570998+0000","last_undegraded":"2024-09-16T00:15:16.570998+0000","last_fullsized":"2024-09-16T00:15:16.570998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:54.455313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717789+0000","last_change":"2024-09-16T00:15:06.551965+0000","last_active":"2024-09-16T00:15:16.717789+0000","last_peered":"2024-09-16T00:15:16.717789+0000","last_clean":"2024-09-16T00:15:16.717789+0000","last_became_active":"2024-09-16T00:15:06.551602+0000","last_became_peered":"2024-09-16T00:15:06.551602+0000","last_unstale":"2024-09-16T00:15:16.717789+0000","last_undegraded":"2024-09-16T00:15:16.717789+0000","last_fullsized":"2024-09-16T00:15:16.717789+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:34.951960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570606+0000","last_change":"2024-09-16T00:15:06.549340+0000","last_active":"2024-09-16T00:15:16.570606+0000","last_peered":"2024-09-16T00:15:16.570606+0000","last_clean":"2024-09-16T00:15:16.570606+0000","last_became_active":"2024-09-16T00:15:06.547424+0000","last_became_peered":"2024-09-16T00:15:06.547424+0000","last_unstale":"2024-09-16T00:15:16.570606+0000","last_undegraded":"2024-09-16T00:15:16.570606+0000","last_fullsized":"2024-09-16T00:15:16.570606+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:12.087535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718679+0000","last_change":"2024-09-16T00:15:06.555216+0000","last_active":"2024-09-16T00:15:16.718679+0000","last_peered":"2024-09-16T00:15:16.718679+0000","last_clean":"2024-09-16T00:15:16.718679+0000","last_became_active":"2024-09-16T00:15:06.554896+0000","last_became_peered":"2024-09-16T00:15:06.554896+0000","last_unstale":"2024-09-16T00:15:16.718679+0000","last_undegraded":"2024-09-16T00:15:16.718679+0000","last_fullsized":"2024-09-16T00:15:16.718679+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:20.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718416+0000","last_change":"2024-09-16T00:15:06.549752+0000","last_active":"2024-09-16T00:15:16.718416+0000","last_peered":"2024-09-16T00:15:16.718416+0000","last_clean":"2024-09-16T00:15:16.718416+0000","last_became_active":"2024-09-16T00:15:06.549350+0000","last_became_peered":"2024-09-16T00:15:06.549350+0000","last_unstale":"2024-09-16T00:15:16.718416+0000","last_undegraded":"2024-09-16T00:15:16.718416+0000","last_fullsized":"2024-09-16T00:15:16.718416+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:48.615123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718131+0000","last_change":"2024-09-16T00:15:06.549159+0000","last_active":"2024-09-16T00:15:16.718131+0000","last_peered":"2024-09-16T00:15:16.718131+0000","last_clean":"2024-09-16T00:15:16.718131+0000","last_became_active":"2024-09-16T00:15:06.548408+0000","last_became_peered":"2024-09-16T00:15:06.548408+0000","last_unstale":"2024-09-16T00:15:16.718131+0000","last_undegraded":"2024-09-16T00:15:16.718131+0000","last_fullsized":"2024-09-16T00:15:16.718131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:23.941809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571847+0000","last_change":"2024-09-16T00:15:06.544091+0000","last_active":"2024-09-16T00:15:16.571847+0000","last_peered":"2024-09-16T00:15:16.571847+0000","last_clean":"2024-09-16T00:15:16.571847+0000","last_became_active":"2024-09-16T00:15:06.543761+0000","last_became_peered":"2024-09-16T00:15:06.543761+0000","last_unstale":"2024-09-16T00:15:16.571847+0000","last_undegraded":"2024-09-16T00:15:16.571847+0000","last_fullsized":"2024-09-16T00:15:16.571847+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:36:43.253405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717983+0000","last_change":"2024-09-16T00:15:06.546408+0000","last_active":"2024-09-16T00:15:16.717983+0000","last_peered":"2024-09-16T00:15:16.717983+0000","last_clean":"2024-09-16T00:15:16.717983+0000","last_became_active":"2024-09-16T00:15:06.546092+0000","last_became_peered":"2024-09-16T00:15:06.546092+0000","last_unstale":"2024-09-16T00:15:16.717983+0000","last_undegraded":"2024-09-16T00:15:16.717983+0000","last_fullsized":"2024-09-16T00:15:16.717983+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:39:15.576097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718725+0000","last_change":"2024-09-16T00:15:06.557162+0000","last_active":"2024-09-16T00:15:16.718725+0000","last_peered":"2024-09-16T00:15:16.718725+0000","last_clean":"2024-09-16T00:15:16.718725+0000","last_became_active":"2024-09-16T00:15:06.556831+0000","last_became_peered":"2024-09-16T00:15:06.556831+0000","last_unstale":"2024-09-16T00:15:16.718725+0000","last_undegraded":"2024-09-16T00:15:16.718725+0000","last_fullsized":"2024-09-16T00:15:16.718725+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.236915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717369+0000","last_change":"2024-09-16T00:15:06.556161+0000","last_active":"2024-09-16T00:15:16.717369+0000","last_peered":"2024-09-16T00:15:16.717369+0000","last_clean":"2024-09-16T00:15:16.717369+0000","last_became_active":"2024-09-16T00:15:06.555819+0000","last_became_peered":"2024-09-16T00:15:06.555819+0000","last_unstale":"2024-09-16T00:15:16.717369+0000","last_undegraded":"2024-09-16T00:15:16.717369+0000","last_fullsized":"2024-09-16T00:15:16.717369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:29.929977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717618+0000","last_change":"2024-09-16T00:15:06.550635+0000","last_active":"2024-09-16T00:15:16.717618+0000","last_peered":"2024-09-16T00:15:16.717618+0000","last_clean":"2024-09-16T00:15:16.717618+0000","last_became_active":"2024-09-16T00:15:06.549576+0000","last_became_peered":"2024-09-16T00:15:06.549576+0000","last_unstale":"2024-09-16T00:15:16.717618+0000","last_undegraded":"2024-09-16T00:15:16.717618+0000","last_fullsized":"2024-09-16T00:15:16.717618+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:12.818199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717880+0000","last_change":"2024-09-16T00:15:06.547114+0000","last_active":"2024-09-16T00:15:16.717880+0000","last_peered":"2024-09-16T00:15:16.717880+0000","last_clean":"2024-09-16T00:15:16.717880+0000","last_became_active":"2024-09-16T00:15:06.546786+0000","last_became_peered":"2024-09-16T00:15:06.546786+0000","last_unstale":"2024-09-16T00:15:16.717880+0000","last_undegraded":"2024-09-16T00:15:16.717880+0000","last_fullsized":"2024-09-16T00:15:16.717880+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:13:26.925603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718321+0000","last_change":"2024-09-16T00:15:06.541168+0000","last_active":"2024-09-16T00:15:16.718321+0000","last_peered":"2024-09-16T00:15:16.718321+0000","last_clean":"2024-09-16T00:15:16.718321+0000","last_became_active":"2024-09-16T00:15:06.540848+0000","last_became_peered":"2024-09-16T00:15:06.540848+0000","last_unstale":"2024-09-16T00:15:16.718321+0000","last_undegraded":"2024-09-16T00:15:16.718321+0000","last_fullsized":"2024-09-16T00:15:16.718321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.958600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718097+0000","last_change":"2024-09-16T00:15:06.547429+0000","last_active":"2024-09-16T00:15:16.718097+0000","last_peered":"2024-09-16T00:15:16.718097+0000","last_clean":"2024-09-16T00:15:16.718097+0000","last_became_active":"2024-09-16T00:15:06.547081+0000","last_became_peered":"2024-09-16T00:15:06.547081+0000","last_unstale":"2024-09-16T00:15:16.718097+0000","last_undegraded":"2024-09-16T00:15:16.718097+0000","last_fullsized":"2024-09-16T00:15:16.718097+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:43.396313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571832+0000","last_change":"2024-09-16T00:15:06.551811+0000","last_active":"2024-09-16T00:15:16.571832+0000","last_peered":"2024-09-16T00:15:16.571832+0000","last_clean":"2024-09-16T00:15:16.571832+0000","last_became_active":"2024-09-16T00:15:06.551424+0000","last_became_peered":"2024-09-16T00:15:06.551424+0000","last_unstale":"2024-09-16T00:15:16.571832+0000","last_undegraded":"2024-09-16T00:15:16.571832+0000","last_fullsized":"2024-09-16T00:15:16.571832+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.943070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718670+0000","last_change":"2024-09-16T00:15:06.544697+0000","last_active":"2024-09-16T00:15:16.718670+0000","last_peered":"2024-09-16T00:15:16.718670+0000","last_clean":"2024-09-16T00:15:16.718670+0000","last_became_active":"2024-09-16T00:15:06.544182+0000","last_became_peered":"2024-09-16T00:15:06.544182+0000","last_unstale":"2024-09-16T00:15:16.718670+0000","last_undegraded":"2024-09-16T00:15:16.718670+0000","last_fullsized":"2024-09-16T00:15:16.718670+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:45.929092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.719262+0000","last_change":"2024-09-16T00:15:06.551488+0000","last_active":"2024-09-16T00:15:16.719262+0000","last_peered":"2024-09-16T00:15:16.719262+0000","last_clean":"2024-09-16T00:15:16.719262+0000","last_became_active":"2024-09-16T00:15:06.551171+0000","last_became_peered":"2024-09-16T00:15:06.551171+0000","last_unstale":"2024-09-16T00:15:16.719262+0000","last_undegraded":"2024-09-16T00:15:16.719262+0000","last_fullsized":"2024-09-16T00:15:16.719262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:23.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.570559+0000","last_change":"2024-09-16T00:15:06.544043+0000","last_active":"2024-09-16T00:15:16.570559+0000","last_peered":"2024-09-16T00:15:16.570559+0000","last_clean":"2024-09-16T00:15:16.570559+0000","last_became_active":"2024-09-16T00:15:06.543014+0000","last_became_peered":"2024-09-16T00:15:06.543014+0000","last_unstale":"2024-09-16T00:15:16.570559+0000","last_undegraded":"2024-09-16T00:15:16.570559+0000","last_fullsized":"2024-09-16T00:15:16.570559+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:23.904810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.571312+0000","last_change":"2024-09-16T00:15:06.546788+0000","last_active":"2024-09-16T00:15:16.571312+0000","last_peered":"2024-09-16T00:15:16.571312+0000","last_clean":"2024-09-16T00:15:16.571312+0000","last_became_active":"2024-09-16T00:15:06.546333+0000","last_became_peered":"2024-09-16T00:15:06.546333+0000","last_unstale":"2024-09-16T00:15:16.571312+0000","last_undegraded":"2024-09-16T00:15:16.571312+0000","last_fullsized":"2024-09-16T00:15:16.571312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:09.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718109+0000","last_change":"2024-09-16T00:15:06.544855+0000","last_active":"2024-09-16T00:15:16.718109+0000","last_peered":"2024-09-16T00:15:16.718109+0000","last_clean":"2024-09-16T00:15:16.718109+0000","last_became_active":"2024-09-16T00:15:06.543708+0000","last_became_peered":"2024-09-16T00:15:06.543708+0000","last_unstale":"2024-09-16T00:15:16.718109+0000","last_undegraded":"2024-09-16T00:15:16.718109+0000","last_fullsized":"2024-09-16T00:15:16.718109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:13:58.756111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.859192+0000","last_change":"2024-09-16T00:15:06.540675+0000","last_active":"2024-09-16T00:15:16.859192+0000","last_peered":"2024-09-16T00:15:16.859192+0000","last_clean":"2024-09-16T00:15:16.859192+0000","last_became_active":"2024-09-16T00:15:06.540333+0000","last_became_peered":"2024-09-16T00:15:06.540333+0000","last_unstale":"2024-09-16T00:15:16.859192+0000","last_undegraded":"2024-09-16T00:15:16.859192+0000","last_fullsized":"2024-09-16T00:15:16.859192+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:17.166987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718067+0000","last_change":"2024-09-16T00:15:06.548758+0000","last_active":"2024-09-16T00:15:16.718067+0000","last_peered":"2024-09-16T00:15:16.718067+0000","last_clean":"2024-09-16T00:15:16.718067+0000","last_became_active":"2024-09-16T00:15:06.548397+0000","last_became_peered":"2024-09-16T00:15:06.548397+0000","last_unstale":"2024-09-16T00:15:16.718067+0000","last_undegraded":"2024-09-16T00:15:16.718067+0000","last_fullsized":"2024-09-16T00:15:16.718067+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:50.320170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718682+0000","last_change":"2024-09-16T00:15:06.554898+0000","last_active":"2024-09-16T00:15:16.718682+0000","last_peered":"2024-09-16T00:15:16.718682+0000","last_clean":"2024-09-16T00:15:16.718682+0000","last_became_active":"2024-09-16T00:15:06.553005+0000","last_became_peered":"2024-09-16T00:15:06.553005+0000","last_unstale":"2024-09-16T00:15:16.718682+0000","last_undegraded":"2024-09-16T00:15:16.718682+0000","last_fullsized":"2024-09-16T00:15:16.718682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:29.630911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.717525+0000","last_change":"2024-09-16T00:15:06.547265+0000","last_active":"2024-09-16T00:15:16.717525+0000","last_peered":"2024-09-16T00:15:16.717525+0000","last_clean":"2024-09-16T00:15:16.717525+0000","last_became_active":"2024-09-16T00:15:06.546938+0000","last_became_peered":"2024-09-16T00:15:06.546938+0000","last_unstale":"2024-09-16T00:15:16.717525+0000","last_undegraded":"2024-09-16T00:15:16.717525+0000","last_fullsized":"2024-09-16T00:15:16.717525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:40.247343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T00:15:16.718626+0000","last_change":"2024-09-16T00:15:06.556735+0000","last_active":"2024-09-16T00:15:16.718626+0000","last_peered":"2024-09-16T00:15:16.718626+0000","last_clean":"2024-09-16T00:15:16.718626+0000","last_became_active":"2024-09-16T00:15:06.556324+0000","last_became_peered":"2024-09-16T00:15:06.556324+0000","last_unstale":"2024-09-16T00:15:16.718626+0000","last_undegraded":"2024-09-16T00:15:16.718626+0000","last_fullsized":"2024-09-16T00:15:16.718626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:25.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":56416,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":111,"num_read_kb":115,"num_write":158,"num_write_kb":103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":169248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249414,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579929,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":478633,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808861,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":1118707,"data_compressed":53889,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972387,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":482965,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201320,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":478789,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430253,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":1074700,"data_compressed":53889,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626483,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":474045,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855416,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":1069895,"data_compressed":53889,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986237,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":954368,"data_stored":479088,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215169,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":474239,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444103,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":527718,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607628,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":999424,"data_stored":526368,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":475433,"data_compressed":17556,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":52413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":53763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":54384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T00:16:08.401 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-16T00:16:08.402 INFO:tasks.cephadm:Setting up client nodes... 2024-09-16T00:16:08.402 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-16T00:16:08.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]: dispatch 2024-09-16T00:16:08.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-09-16T00:16:08.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-09-16T00:16:08.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-09-16T00:16:08.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-09-16T00:16:08.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-09-16T00:16:08.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]: dispatch 2024-09-16T00:16:08.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]: dispatch 2024-09-16T00:16:08.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-09-16T00:16:08.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-09-16T00:16:08.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]: dispatch 2024-09-16T00:16:08.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-09-16T00:16:08.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-09-16T00:16:08.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-09-16T00:16:08.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-09-16T00:16:08.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-09-16T00:16:08.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]: dispatch 2024-09-16T00:16:08.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]: dispatch 2024-09-16T00:16:08.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-09-16T00:16:08.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-09-16T00:16:08.661 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:09.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: pgmap v316: 129 pgs: 129 active+clean; 632 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 5.1 KiB/s wr, 7 op/s 2024-09-16T00:16:09.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:16:09.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]': finished 2024-09-16T00:16:09.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-09-16T00:16:09.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-09-16T00:16:09.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-09-16T00:16:09.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-09-16T00:16:09.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-09-16T00:16:09.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]': finished 2024-09-16T00:16:09.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]': finished 2024-09-16T00:16:09.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-09-16T00:16:09.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-09-16T00:16:09.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:09 smithi033 ceph-mon[31243]: osdmap e87: 12 total, 12 up, 12 in 2024-09-16T00:16:09.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.204+0000 7f18cfbf3640 1 -- 172.21.15.33:0/89104560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18c81026d0 msgr2=0x7f18c8108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:09.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.204+0000 7f18cfbf3640 1 --2- 172.21.15.33:0/89104560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18c81026d0 0x7f18c8108f60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f18c0009ca0 tx=0x7f18c002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:16:09.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.204+0000 7f18cfbf3640 1 -- 172.21.15.33:0/89104560 shutdown_connections 2024-09-16T00:16:09.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.204+0000 7f18cfbf3640 1 --2- 172.21.15.33:0/89104560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18c81026d0 0x7f18c8108f60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:09.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.204+0000 7f18cfbf3640 1 --2- 172.21.15.33:0/89104560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c8101d10 0x7f18c8102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:09.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.204+0000 7f18cfbf3640 1 --2- 172.21.15.33:0/89104560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18c8100b10 0x7f18c8100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:09.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.204+0000 7f18cfbf3640 1 -- 172.21.15.33:0/89104560 >> 172.21.15.33:0/89104560 conn(0x7f18c80fc2a0 msgr2=0x7f18c80fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:09.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.204+0000 7f18cfbf3640 1 -- 172.21.15.33:0/89104560 shutdown_connections 2024-09-16T00:16:09.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.204+0000 7f18cfbf3640 1 -- 172.21.15.33:0/89104560 wait complete. 2024-09-16T00:16:09.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.204+0000 7f18cfbf3640 1 Processor -- start 2024-09-16T00:16:09.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18cfbf3640 1 -- start start 2024-09-16T00:16:09.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18cfbf3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18c8100b10 0x7f18c806b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:09.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18cfbf3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c8101d10 0x7f18c819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:09.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18cfbf3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18c81026d0 0x7f18c819f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:09.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18cfbf3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18c810c890 con 0x7f18c81026d0 2024-09-16T00:16:09.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18cfbf3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f18c810c710 con 0x7f18c8101d10 2024-09-16T00:16:09.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18cfbf3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f18c810ca10 con 0x7f18c8100b10 2024-09-16T00:16:09.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18ce190640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18c81026d0 0x7f18c819f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18ce190640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18c81026d0 0x7f18c819f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40312/0 (socket says 172.21.15.33:40312) 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18ce190640 1 -- 172.21.15.33:0/4224956697 learned_addr learned my addr 172.21.15.33:0/4224956697 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18cd18e640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c8101d10 0x7f18c819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18cd98f640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18c8100b10 0x7f18c806b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18ce190640 1 -- 172.21.15.33:0/4224956697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18c8100b10 msgr2=0x7f18c806b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18ce190640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18c8100b10 0x7f18c806b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18ce190640 1 -- 172.21.15.33:0/4224956697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c8101d10 msgr2=0x7f18c819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18ce190640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c8101d10 0x7f18c819cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18ce190640 1 -- 172.21.15.33:0/4224956697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c819fce0 con 0x7f18c81026d0 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18cd18e640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c8101d10 0x7f18c819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.205+0000 7f18cd98f640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18c8100b10 0x7f18c806b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.206+0000 7f18ce190640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18c81026d0 0x7f18c819f3d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f18c00098d0 tx=0x7f18c002f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:09.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.206+0000 7f18beffd640 1 -- 172.21.15.33:0/4224956697 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c002fc30 con 0x7f18c81026d0 2024-09-16T00:16:09.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.206+0000 7f18beffd640 1 -- 172.21.15.33:0/4224956697 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18c0003e50 con 0x7f18c81026d0 2024-09-16T00:16:09.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.206+0000 7f18cfbf3640 1 -- 172.21.15.33:0/4224956697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c819ff10 con 0x7f18c81026d0 2024-09-16T00:16:09.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.206+0000 7f18beffd640 1 -- 172.21.15.33:0/4224956697 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c00305c0 con 0x7f18c81026d0 2024-09-16T00:16:09.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.206+0000 7f18cfbf3640 1 -- 172.21.15.33:0/4224956697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c81a04f0 con 0x7f18c81026d0 2024-09-16T00:16:09.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.207+0000 7f18cfbf3640 1 -- 172.21.15.33:0/4224956697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18c8107f30 con 0x7f18c81026d0 2024-09-16T00:16:09.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.208+0000 7f18beffd640 1 -- 172.21.15.33:0/4224956697 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f18c0004b40 con 0x7f18c81026d0 2024-09-16T00:16:09.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.209+0000 7f18beffd640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18b40a3350 0x7f18b40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:09.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.209+0000 7f18cd98f640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18b40a3350 0x7f18b40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:09.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.209+0000 7f18beffd640 1 -- 172.21.15.33:0/4224956697 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f18c00f11e0 con 0x7f18c81026d0 2024-09-16T00:16:09.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.209+0000 7f18cd98f640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18b40a3350 0x7f18b40a5810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f18c8101b70 tx=0x7f18c4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:09.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.212+0000 7f18beffd640 1 -- 172.21.15.33:0/4224956697 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f18c00bdb50 con 0x7f18c81026d0 2024-09-16T00:16:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: pgmap v316: 129 pgs: 129 active+clean; 632 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 5.1 KiB/s wr, 7 op/s 2024-09-16T00:16:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:16:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]': finished 2024-09-16T00:16:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-09-16T00:16:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-09-16T00:16:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-09-16T00:16:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-09-16T00:16:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-09-16T00:16:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]': finished 2024-09-16T00:16:09.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]': finished 2024-09-16T00:16:09.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-09-16T00:16:09.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-09-16T00:16:09.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:09 smithi083 ceph-mon[35242]: osdmap e87: 12 total, 12 up, 12 in 2024-09-16T00:16:09.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.452+0000 7f18cfbf3640 1 -- 172.21.15.33:0/4224956697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f18c802d7c0 con 0x7f18c81026d0 2024-09-16T00:16:09.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.455+0000 7f18beffd640 1 -- 172.21.15.33:0/4224956697 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f18c00bd450 con 0x7f18c81026d0 2024-09-16T00:16:09.456 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2024-09-16T00:16:09.456 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDJeOdmj7AdGxAA/C9SrepauOSk5IQr6hEa5g== 2024-09-16T00:16:09.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.457+0000 7f18cfbf3640 1 -- 172.21.15.33:0/4224956697 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18b40a3350 msgr2=0x7f18b40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:09.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.457+0000 7f18cfbf3640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18b40a3350 0x7f18b40a5810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f18c8101b70 tx=0x7f18c4008040 comp rx=0 tx=0).stop 2024-09-16T00:16:09.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.457+0000 7f18cfbf3640 1 -- 172.21.15.33:0/4224956697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18c81026d0 msgr2=0x7f18c819f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:09.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.457+0000 7f18cfbf3640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18c81026d0 0x7f18c819f3d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f18c00098d0 tx=0x7f18c002f840 comp rx=0 tx=0).stop 2024-09-16T00:16:09.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.457+0000 7f18cfbf3640 1 -- 172.21.15.33:0/4224956697 shutdown_connections 2024-09-16T00:16:09.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.457+0000 7f18cfbf3640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18b40a3350 0x7f18b40a5810 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:09.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.457+0000 7f18cfbf3640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18c81026d0 0x7f18c819f3d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:09.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.457+0000 7f18cfbf3640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c8101d10 0x7f18c819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:09.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.457+0000 7f18cfbf3640 1 --2- 172.21.15.33:0/4224956697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18c8100b10 0x7f18c806b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:09.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.457+0000 7f18cfbf3640 1 -- 172.21.15.33:0/4224956697 >> 172.21.15.33:0/4224956697 conn(0x7f18c80fc2a0 msgr2=0x7f18c8105100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:09.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.457+0000 7f18cfbf3640 1 -- 172.21.15.33:0/4224956697 shutdown_connections 2024-09-16T00:16:09.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:09.457+0000 7f18cfbf3640 1 -- 172.21.15.33:0/4224956697 wait complete. 2024-09-16T00:16:09.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: pgmap v316: 129 pgs: 129 active+clean; 632 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 5.1 KiB/s wr, 7 op/s 2024-09-16T00:16:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:16:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]': finished 2024-09-16T00:16:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-09-16T00:16:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-09-16T00:16:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-09-16T00:16:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-09-16T00:16:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-09-16T00:16:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]': finished 2024-09-16T00:16:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]': finished 2024-09-16T00:16:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-09-16T00:16:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-09-16T00:16:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:09 smithi060 ceph-mon[35185]: osdmap e87: 12 total, 12 up, 12 in 2024-09-16T00:16:10.141 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:10 smithi033 ceph-mon[31243]: osdmap e88: 12 total, 12 up, 12 in 2024-09-16T00:16:10.141 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:10 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4224956697' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T00:16:10.141 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:10 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4224956697' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T00:16:10.172 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:16:10.172 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-16T00:16:10.172 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T00:16:10.213 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T00:16:10.213 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-16T00:16:10.213 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-16T00:16:10.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:10 smithi083 ceph-mon[35242]: osdmap e88: 12 total, 12 up, 12 in 2024-09-16T00:16:10.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:10 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4224956697' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T00:16:10.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:10 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4224956697' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T00:16:10.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:10 smithi060 ceph-mon[35185]: osdmap e88: 12 total, 12 up, 12 in 2024-09-16T00:16:10.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:10 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4224956697' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T00:16:10.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:10 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4224956697' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T00:16:10.510 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.111+0000 7f9192d5d640 1 -- 172.21.15.33:0/1710611001 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f918c100a80 msgr2=0x7f918c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.111+0000 7f9192d5d640 1 --2- 172.21.15.33:0/1710611001 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f918c100a80 0x7f918c100ee0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f917c009cd0 tx=0x7f917c02a8e0 comp rx=0 tx=0).stop 2024-09-16T00:16:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.111+0000 7f9192d5d640 1 -- 172.21.15.33:0/1710611001 shutdown_connections 2024-09-16T00:16:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.111+0000 7f9192d5d640 1 --2- 172.21.15.33:0/1710611001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918c101420 0x7f918c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.111+0000 7f9192d5d640 1 --2- 172.21.15.33:0/1710611001 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f918c100a80 0x7f918c100ee0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.111+0000 7f9192d5d640 1 --2- 172.21.15.33:0/1710611001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f918c106a80 0x7f918c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.111+0000 7f9192d5d640 1 -- 172.21.15.33:0/1710611001 >> 172.21.15.33:0/1710611001 conn(0x7f918c0fc7a0 msgr2=0x7f918c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.111+0000 7f9192d5d640 1 -- 172.21.15.33:0/1710611001 shutdown_connections 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.111+0000 7f9192d5d640 1 -- 172.21.15.33:0/1710611001 wait complete. 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f9192d5d640 1 Processor -- start 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f9192d5d640 1 -- start start 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f9192d5d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918c100a80 0x7f918c0fce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f9192d5d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f918c101420 0x7f918c0fd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f9192d5d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f918c106a80 0x7f918c0fd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f9192d5d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f918c110e80 con 0x7f918c106a80 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f9192d5d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f918c110d00 con 0x7f918c100a80 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f9192d5d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f918c111000 con 0x7f918c101420 2024-09-16T00:16:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f91912fa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f918c106a80 0x7f918c0fd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:11.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f91912fa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f918c106a80 0x7f918c0fd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59256/0 (socket says 172.21.15.33:59256) 2024-09-16T00:16:11.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f91912fa640 1 -- 172.21.15.33:0/18886172 learned_addr learned my addr 172.21.15.33:0/18886172 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:11.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f918bfff640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f918c101420 0x7f918c0fd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:11.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f9190af9640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918c100a80 0x7f918c0fce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:11.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f91912fa640 1 -- 172.21.15.33:0/18886172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f918c101420 msgr2=0x7f918c0fd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:11.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.112+0000 7f91912fa640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f918c101420 0x7f918c0fd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:11.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.113+0000 7f91912fa640 1 -- 172.21.15.33:0/18886172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918c100a80 msgr2=0x7f918c0fce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:11.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.113+0000 7f91912fa640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918c100a80 0x7f918c0fce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:11.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.113+0000 7f91912fa640 1 -- 172.21.15.33:0/18886172 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918c1a6950 con 0x7f918c106a80 2024-09-16T00:16:11.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.113+0000 7f9190af9640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918c100a80 0x7f918c0fce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:16:11.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.113+0000 7f918bfff640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f918c101420 0x7f918c0fd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:16:11.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.113+0000 7f91912fa640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f918c106a80 0x7f918c0fd8d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f918400c970 tx=0x7f918400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.113+0000 7f9189ffb640 1 -- 172.21.15.33:0/18886172 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9184007bb0 con 0x7f918c106a80 2024-09-16T00:16:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.113+0000 7f9189ffb640 1 -- 172.21.15.33:0/18886172 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9184007d50 con 0x7f918c106a80 2024-09-16T00:16:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.113+0000 7f9192d5d640 1 -- 172.21.15.33:0/18886172 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f918c1a6c40 con 0x7f918c106a80 2024-09-16T00:16:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.113+0000 7f9189ffb640 1 -- 172.21.15.33:0/18886172 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918400f450 con 0x7f918c106a80 2024-09-16T00:16:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.113+0000 7f9192d5d640 1 -- 172.21.15.33:0/18886172 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f918c1a7130 con 0x7f918c106a80 2024-09-16T00:16:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.114+0000 7f9192d5d640 1 -- 172.21.15.33:0/18886172 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f918c100ee0 con 0x7f918c106a80 2024-09-16T00:16:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.115+0000 7f9189ffb640 1 -- 172.21.15.33:0/18886172 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f9184004f60 con 0x7f918c106a80 2024-09-16T00:16:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.116+0000 7f9189ffb640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f91780a32d0 0x7f91780a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.117+0000 7f9190af9640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f91780a32d0 0x7f91780a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.117+0000 7f9189ffb640 1 -- 172.21.15.33:0/18886172 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f91840cfbe0 con 0x7f918c106a80 2024-09-16T00:16:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.117+0000 7f9190af9640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f91780a32d0 0x7f91780a5790 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f91740058c0 tx=0x7f9174005cb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.121+0000 7f9189ffb640 1 -- 172.21.15.33:0/18886172 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9184014030 con 0x7f918c106a80 2024-09-16T00:16:11.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.344+0000 7f9192d5d640 1 -- 172.21.15.33:0/18886172 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f918c109a90 con 0x7f918c106a80 2024-09-16T00:16:11.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.349+0000 7f9189ffb640 1 -- 172.21.15.33:0/18886172 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264050 (secure 0 0 0) 0x7f918409c440 con 0x7f918c106a80 2024-09-16T00:16:11.354 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:11.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.352+0000 7f9192d5d640 1 -- 172.21.15.33:0/18886172 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f91780a32d0 msgr2=0x7f91780a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:11.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.352+0000 7f9192d5d640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f91780a32d0 0x7f91780a5790 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f91740058c0 tx=0x7f9174005cb0 comp rx=0 tx=0).stop 2024-09-16T00:16:11.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.352+0000 7f9192d5d640 1 -- 172.21.15.33:0/18886172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f918c106a80 msgr2=0x7f918c0fd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:11.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.352+0000 7f9192d5d640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f918c106a80 0x7f918c0fd8d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f918400c970 tx=0x7f918400ce30 comp rx=0 tx=0).stop 2024-09-16T00:16:11.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.352+0000 7f9192d5d640 1 -- 172.21.15.33:0/18886172 shutdown_connections 2024-09-16T00:16:11.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.352+0000 7f9192d5d640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f91780a32d0 0x7f91780a5790 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:11.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.352+0000 7f9192d5d640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f918c106a80 0x7f918c0fd8d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:11.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.352+0000 7f9192d5d640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f918c101420 0x7f918c0fd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:11.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.352+0000 7f9192d5d640 1 --2- 172.21.15.33:0/18886172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918c100a80 0x7f918c0fce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:11.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.352+0000 7f9192d5d640 1 -- 172.21.15.33:0/18886172 >> 172.21.15.33:0/18886172 conn(0x7f918c0fc7a0 msgr2=0x7f918c06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:11.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.352+0000 7f9192d5d640 1 -- 172.21.15.33:0/18886172 shutdown_connections 2024-09-16T00:16:11.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:11.352+0000 7f9192d5d640 1 -- 172.21.15.33:0/18886172 wait complete. 2024-09-16T00:16:11.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:11 smithi033 ceph-mon[31243]: pgmap v319: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.6 KiB/s wr, 8 op/s 2024-09-16T00:16:11.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:11 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/18886172' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-16T00:16:11.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:11 smithi060 ceph-mon[35185]: pgmap v319: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.6 KiB/s wr, 8 op/s 2024-09-16T00:16:11.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:11 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/18886172' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-16T00:16:11.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:11 smithi083 ceph-mon[35242]: pgmap v319: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.6 KiB/s wr, 8 op/s 2024-09-16T00:16:11.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:11 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/18886172' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-16T00:16:12.116 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":3427860355},{"type":"v1","addr":"172.21.15.33:6801","nonce":3427860355}]},"active_addr":"172.21.15.33:6801/3427860355","active_change":"2024-09-16T00:08:07.548656+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.33:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":3761745238}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":1917125516}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":1628705799}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":2583695893}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":3040183362}]}]} 2024-09-16T00:16:12.122 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-16T00:16:12.122 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-16T00:16:12.122 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:16:12.380 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:12.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.861+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/3511532023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0106a80 msgr2=0x7ff8d0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:12.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.861+0000 7ff8d6bc4640 1 --2- 172.21.15.33:0/3511532023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0106a80 0x7ff8d0106e60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4009930 tx=0x7ff8c402a420 comp rx=0 tx=0).stop 2024-09-16T00:16:12.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.861+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/3511532023 shutdown_connections 2024-09-16T00:16:12.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.861+0000 7ff8d6bc4640 1 --2- 172.21.15.33:0/3511532023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8d0101420 0x7ff8d010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:12.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.861+0000 7ff8d6bc4640 1 --2- 172.21.15.33:0/3511532023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8d0100a80 0x7ff8d0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:12.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.861+0000 7ff8d6bc4640 1 --2- 172.21.15.33:0/3511532023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0106a80 0x7ff8d0106e60 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.861+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/3511532023 >> 172.21.15.33:0/3511532023 conn(0x7ff8d00fc7a0 msgr2=0x7ff8d00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.861+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/3511532023 shutdown_connections 2024-09-16T00:16:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.861+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/3511532023 wait complete. 2024-09-16T00:16:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.861+0000 7ff8d6bc4640 1 Processor -- start 2024-09-16T00:16:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.861+0000 7ff8d6bc4640 1 -- start start 2024-09-16T00:16:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d6bc4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8d0100a80 0x7ff8d0199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d6bc4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8d0101420 0x7ff8d01997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d6bc4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0106a80 0x7ff8d01a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d6bc4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d0110e80 con 0x7ff8d0106a80 2024-09-16T00:16:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d6bc4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d0110d00 con 0x7ff8d0100a80 2024-09-16T00:16:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d6bc4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d0111000 con 0x7ff8d0101420 2024-09-16T00:16:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d5161640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0106a80 0x7ff8d01a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d5161640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0106a80 0x7ff8d01a4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59264/0 (socket says 172.21.15.33:59264) 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d5161640 1 -- 172.21.15.33:0/1223696232 learned_addr learned my addr 172.21.15.33:0/1223696232 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d4960640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8d0100a80 0x7ff8d0199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8cffff640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8d0101420 0x7ff8d01997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d5161640 1 -- 172.21.15.33:0/1223696232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8d0101420 msgr2=0x7ff8d01997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d5161640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8d0101420 0x7ff8d01997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d5161640 1 -- 172.21.15.33:0/1223696232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8d0100a80 msgr2=0x7ff8d0199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d5161640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8d0100a80 0x7ff8d0199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d5161640 1 -- 172.21.15.33:0/1223696232 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d01a4850 con 0x7ff8d0106a80 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.862+0000 7ff8d5161640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0106a80 0x7ff8d01a4310 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800cce0 tx=0x7ff8c800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.863+0000 7ff8cdffb640 1 -- 172.21.15.33:0/1223696232 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c800f450 con 0x7ff8d0106a80 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.863+0000 7ff8cdffb640 1 -- 172.21.15.33:0/1223696232 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8c800fad0 con 0x7ff8d0106a80 2024-09-16T00:16:12.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.863+0000 7ff8cdffb640 1 -- 172.21.15.33:0/1223696232 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8018c10 con 0x7ff8d0106a80 2024-09-16T00:16:12.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.863+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/1223696232 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d01a4a20 con 0x7ff8d0106a80 2024-09-16T00:16:12.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.863+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/1223696232 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8d01a4f30 con 0x7ff8d0106a80 2024-09-16T00:16:12.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.864+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/1223696232 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8d0100ee0 con 0x7ff8d0106a80 2024-09-16T00:16:12.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.865+0000 7ff8cdffb640 1 -- 172.21.15.33:0/1223696232 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff8c800f5f0 con 0x7ff8d0106a80 2024-09-16T00:16:12.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.865+0000 7ff8cdffb640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8c00a3350 0x7ff8c00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:12.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.866+0000 7ff8cdffb640 1 -- 172.21.15.33:0/1223696232 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7ff8c80d0300 con 0x7ff8d0106a80 2024-09-16T00:16:12.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.866+0000 7ff8d4960640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8c00a3350 0x7ff8c00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:12.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.866+0000 7ff8d4960640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8c00a3350 0x7ff8c00a5810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4009900 tx=0x7ff8c40058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:12.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:12.871+0000 7ff8cdffb640 1 -- 172.21.15.33:0/1223696232 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff8c8014030 con 0x7ff8d0106a80 2024-09-16T00:16:13.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.049+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/1223696232 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8d0108ff0 con 0x7ff8d0106a80 2024-09-16T00:16:13.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.052+0000 7ff8cdffb640 1 -- 172.21.15.33:0/1223696232 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24609 (secure 0 0 0) 0x7ff8c809cb60 con 0x7ff8d0106a80 2024-09-16T00:16:13.053 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:13.053 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":88,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:16:09.010428+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:16:13.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.053+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/1223696232 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8c00a3350 msgr2=0x7ff8c00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:13.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.053+0000 7ff8d6bc4640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8c00a3350 0x7ff8c00a5810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4009900 tx=0x7ff8c40058d0 comp rx=0 tx=0).stop 2024-09-16T00:16:13.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.053+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/1223696232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0106a80 msgr2=0x7ff8d01a4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:13.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.053+0000 7ff8d6bc4640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0106a80 0x7ff8d01a4310 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800cce0 tx=0x7ff8c800a620 comp rx=0 tx=0).stop 2024-09-16T00:16:13.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.053+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/1223696232 shutdown_connections 2024-09-16T00:16:13.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.053+0000 7ff8d6bc4640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8c00a3350 0x7ff8c00a5810 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:13.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.053+0000 7ff8d6bc4640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0106a80 0x7ff8d01a4310 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:13.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.053+0000 7ff8d6bc4640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8d0101420 0x7ff8d01997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:13.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.053+0000 7ff8d6bc4640 1 --2- 172.21.15.33:0/1223696232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8d0100a80 0x7ff8d0199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:13.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.053+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/1223696232 >> 172.21.15.33:0/1223696232 conn(0x7ff8d00fc7a0 msgr2=0x7ff8d00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:13.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.054+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/1223696232 shutdown_connections 2024-09-16T00:16:13.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:13.054+0000 7ff8d6bc4640 1 -- 172.21.15.33:0/1223696232 wait complete. 2024-09-16T00:16:13.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:13 smithi060 ceph-mon[35185]: pgmap v320: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 6 op/s; 72 B/s, 0 objects/s recovering 2024-09-16T00:16:13.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:13 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1223696232' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:16:13.736 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:13 smithi033 ceph-mon[31243]: pgmap v320: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 6 op/s; 72 B/s, 0 objects/s recovering 2024-09-16T00:16:13.736 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:13 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1223696232' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:16:13.768 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-16T00:16:13.768 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:16:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:13 smithi083 ceph-mon[35242]: pgmap v320: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 6 op/s; 72 B/s, 0 objects/s recovering 2024-09-16T00:16:13.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:13 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1223696232' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:16:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:14.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.514+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/3347191330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a08106a80 msgr2=0x7f5a08106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:14.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.514+0000 7f5a0f4c4640 1 --2- 172.21.15.33:0/3347191330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a08106a80 0x7f5a08106e60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009930 tx=0x7f5a0402a420 comp rx=0 tx=0).stop 2024-09-16T00:16:14.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.514+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/3347191330 shutdown_connections 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.514+0000 7f5a0f4c4640 1 --2- 172.21.15.33:0/3347191330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08101420 0x7f5a0810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.514+0000 7f5a0f4c4640 1 --2- 172.21.15.33:0/3347191330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a08100a80 0x7f5a08100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.514+0000 7f5a0f4c4640 1 --2- 172.21.15.33:0/3347191330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a08106a80 0x7f5a08106e60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.514+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/3347191330 >> 172.21.15.33:0/3347191330 conn(0x7f5a080fc7a0 msgr2=0x7f5a080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.514+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/3347191330 shutdown_connections 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.514+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/3347191330 wait complete. 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.515+0000 7f5a0f4c4640 1 Processor -- start 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.515+0000 7f5a0f4c4640 1 -- start start 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.515+0000 7f5a0f4c4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08100a80 0x7f5a080fcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.515+0000 7f5a0f4c4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a08101420 0x7f5a080fd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.515+0000 7f5a0f4c4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a08106a80 0x7f5a080fd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.515+0000 7f5a0f4c4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5a08110e80 con 0x7f5a08101420 2024-09-16T00:16:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.515+0000 7f5a0f4c4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5a08110d00 con 0x7f5a08100a80 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.515+0000 7f5a0f4c4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a08111000 con 0x7f5a08106a80 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.515+0000 7f5a0ca5f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a08101420 0x7f5a080fd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.515+0000 7f5a0ca5f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a08101420 0x7f5a080fd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59286/0 (socket says 172.21.15.33:59286) 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.515+0000 7f5a0ca5f640 1 -- 172.21.15.33:0/1807883040 learned_addr learned my addr 172.21.15.33:0/1807883040 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.515+0000 7f5a0d260640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08100a80 0x7f5a080fcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.516+0000 7f5a0ca5f640 1 -- 172.21.15.33:0/1807883040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a08106a80 msgr2=0x7f5a080fd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.516+0000 7f5a0da61640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a08106a80 0x7f5a080fd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.516+0000 7f5a0ca5f640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a08106a80 0x7f5a080fd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.516+0000 7f5a0ca5f640 1 -- 172.21.15.33:0/1807883040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08100a80 msgr2=0x7f5a080fcdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.516+0000 7f5a0ca5f640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08100a80 0x7f5a080fcdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.516+0000 7f5a0ca5f640 1 -- 172.21.15.33:0/1807883040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a081a6930 con 0x7f5a08101420 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.516+0000 7f5a0ca5f640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a08101420 0x7f5a080fd300 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f59f800e9f0 tx=0x7f59f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.516+0000 7f59fe7fc640 1 -- 172.21.15.33:0/1807883040 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f800cd80 con 0x7f5a08101420 2024-09-16T00:16:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.516+0000 7f59fe7fc640 1 -- 172.21.15.33:0/1807883040 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59f8005500 con 0x7f5a08101420 2024-09-16T00:16:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.516+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/1807883040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a081a6c20 con 0x7f5a08101420 2024-09-16T00:16:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.516+0000 7f59fe7fc640 1 -- 172.21.15.33:0/1807883040 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8010690 con 0x7f5a08101420 2024-09-16T00:16:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.516+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/1807883040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a081a7110 con 0x7f5a08101420 2024-09-16T00:16:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.517+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/1807883040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a08100ee0 con 0x7f5a08101420 2024-09-16T00:16:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.518+0000 7f59fe7fc640 1 -- 172.21.15.33:0/1807883040 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f59f80046c0 con 0x7f5a08101420 2024-09-16T00:16:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.519+0000 7f59fe7fc640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f59f40a32d0 0x7f59f40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.519+0000 7f59fe7fc640 1 -- 172.21.15.33:0/1807883040 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f59f80d0460 con 0x7f5a08101420 2024-09-16T00:16:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.519+0000 7f5a0d260640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f59f40a32d0 0x7f59f40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.520+0000 7f5a0d260640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f59f40a32d0 0x7f59f40a5790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009900 tx=0x7f5a040058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.522+0000 7f59fe7fc640 1 -- 172.21.15.33:0/1807883040 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f59f80541b0 con 0x7f5a08101420 2024-09-16T00:16:14.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.698+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/1807883040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5a080fe2f0 con 0x7f5a08101420 2024-09-16T00:16:14.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.700+0000 7f59fe7fc640 1 -- 172.21.15.33:0/1807883040 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24609 (secure 0 0 0) 0x7f5a080fe2f0 con 0x7f5a08101420 2024-09-16T00:16:14.701 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:14.701 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":88,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:16:09.010428+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:16:14.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.702+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/1807883040 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f59f40a32d0 msgr2=0x7f59f40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:14.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.702+0000 7f5a0f4c4640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f59f40a32d0 0x7f59f40a5790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009900 tx=0x7f5a040058d0 comp rx=0 tx=0).stop 2024-09-16T00:16:14.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.702+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/1807883040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a08101420 msgr2=0x7f5a080fd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:14.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.702+0000 7f5a0f4c4640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a08101420 0x7f5a080fd300 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f59f800e9f0 tx=0x7f59f800edf0 comp rx=0 tx=0).stop 2024-09-16T00:16:14.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.702+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/1807883040 shutdown_connections 2024-09-16T00:16:14.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.702+0000 7f5a0f4c4640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f59f40a32d0 0x7f59f40a5790 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:14.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.702+0000 7f5a0f4c4640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a08106a80 0x7f5a080fd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:14.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.702+0000 7f5a0f4c4640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a08101420 0x7f5a080fd300 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:14.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.702+0000 7f5a0f4c4640 1 --2- 172.21.15.33:0/1807883040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08100a80 0x7f5a080fcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:14.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.702+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/1807883040 >> 172.21.15.33:0/1807883040 conn(0x7f5a080fc7a0 msgr2=0x7f5a0806aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:14.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.702+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/1807883040 shutdown_connections 2024-09-16T00:16:14.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:14.702+0000 7f5a0f4c4640 1 -- 172.21.15.33:0/1807883040 wait complete. 2024-09-16T00:16:15.543 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-16T00:16:15.544 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-16T00:16:15.544 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-16T00:16:15.545 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-16T00:16:15.545 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-16T00:16:15.545 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-16T00:16:15.546 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-16T00:16:15.546 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-16T00:16:15.547 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-16T00:16:15.548 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-16T00:16:15.548 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-16T00:16:15.549 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-16T00:16:15.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:15 smithi060 ceph-mon[35185]: pgmap v321: 129 pgs: 1 peering, 128 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-09-16T00:16:15.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:15 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1807883040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:16:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:15 smithi033 ceph-mon[31243]: pgmap v321: 129 pgs: 1 peering, 128 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-09-16T00:16:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:15 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1807883040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:16:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:15 smithi083 ceph-mon[35242]: pgmap v321: 129 pgs: 1 peering, 128 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-09-16T00:16:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:15 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1807883040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:16:16.257 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:16.257 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:16.258 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:16.698 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:16.701 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:16.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.784+0000 7f6826438640 1 -- 172.21.15.33:0/4135535957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6820102d30 msgr2=0x7f6820103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:16.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.784+0000 7f6826438640 1 --2- 172.21.15.33:0/4135535957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6820102d30 0x7f6820103110 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6810009930 tx=0x7f681002a440 comp rx=0 tx=0).stop 2024-09-16T00:16:16.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.784+0000 7f6826438640 1 -- 172.21.15.33:0/4135535957 shutdown_connections 2024-09-16T00:16:16.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.784+0000 7f6826438640 1 --2- 172.21.15.33:0/4135535957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68201082a0 0x7f682010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:16.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.784+0000 7f6826438640 1 --2- 172.21.15.33:0/4135535957 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68201036e0 0x7f6820107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:16.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.784+0000 7f6826438640 1 --2- 172.21.15.33:0/4135535957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6820102d30 0x7f6820103110 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:16.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.784+0000 7f6826438640 1 -- 172.21.15.33:0/4135535957 >> 172.21.15.33:0/4135535957 conn(0x7f68200fc7d0 msgr2=0x7f68200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:16.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.784+0000 7f6826438640 1 -- 172.21.15.33:0/4135535957 shutdown_connections 2024-09-16T00:16:16.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.785+0000 7f6826438640 1 -- 172.21.15.33:0/4135535957 wait complete. 2024-09-16T00:16:16.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.785+0000 7f6826438640 1 Processor -- start 2024-09-16T00:16:16.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.785+0000 7f6826438640 1 -- start start 2024-09-16T00:16:16.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f6826438640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6820102d30 0x7f68201984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:16.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f6826438640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68201036e0 0x7f6820198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:16.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f6826438640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68201082a0 0x7f682019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:16.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f6826438640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f68201120e0 con 0x7f6820102d30 2024-09-16T00:16:16.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f6826438640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6820111f60 con 0x7f68201036e0 2024-09-16T00:16:16.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f6826438640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6820112260 con 0x7f68201082a0 2024-09-16T00:16:16.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f681ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6820102d30 0x7f68201984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:16.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f681ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6820102d30 0x7f68201984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59298/0 (socket says 172.21.15.33:59298) 2024-09-16T00:16:16.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f681ffff640 1 -- 172.21.15.33:0/4088916768 learned_addr learned my addr 172.21.15.33:0/4088916768 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:16.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f681f7fe640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68201036e0 0x7f6820198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:16.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f68249d5640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68201082a0 0x7f682019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:16.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f681ffff640 1 -- 172.21.15.33:0/4088916768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68201082a0 msgr2=0x7f682019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:16.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f681ffff640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68201082a0 0x7f682019cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:16.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f681ffff640 1 -- 172.21.15.33:0/4088916768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68201036e0 msgr2=0x7f6820198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:16.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f681ffff640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68201036e0 0x7f6820198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:16.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.786+0000 7f681ffff640 1 -- 172.21.15.33:0/4088916768 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682019d490 con 0x7f6820102d30 2024-09-16T00:16:16.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.787+0000 7f681ffff640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6820102d30 0x7f68201984e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f681002a960 tx=0x7f6810003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:16.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.787+0000 7f681d7fa640 1 -- 172.21.15.33:0/4088916768 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681003f070 con 0x7f6820102d30 2024-09-16T00:16:16.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.787+0000 7f6826438640 1 -- 172.21.15.33:0/4088916768 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f682019d720 con 0x7f6820102d30 2024-09-16T00:16:16.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.787+0000 7f681d7fa640 1 -- 172.21.15.33:0/4088916768 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6810031c90 con 0x7f6820102d30 2024-09-16T00:16:16.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.787+0000 7f681d7fa640 1 -- 172.21.15.33:0/4088916768 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6810035030 con 0x7f6820102d30 2024-09-16T00:16:16.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.787+0000 7f6826438640 1 -- 172.21.15.33:0/4088916768 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68201a5060 con 0x7f6820102d30 2024-09-16T00:16:16.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.788+0000 7f6826438640 1 -- 172.21.15.33:0/4088916768 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f68200fed60 con 0x7f6820102d30 2024-09-16T00:16:16.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.789+0000 7f681d7fa640 1 -- 172.21.15.33:0/4088916768 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f6810042070 con 0x7f6820102d30 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.790+0000 7f681d7fa640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68080a3350 0x7f68080a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.790+0000 7f681f7fe640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68080a3350 0x7f68080a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.790+0000 7f681d7fa640 1 -- 172.21.15.33:0/4088916768 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f68100f08c0 con 0x7f6820102d30 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.791+0000 7f681d7fa640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559] conn(0x7f68080ae520 0x7f68080b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.791+0000 7f681d7fa640 1 -- 172.21.15.33:0/4088916768 --> [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6810046e50 con 0x7f68080ae520 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.791+0000 7f681d7fa640 1 -- 172.21.15.33:0/4088916768 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f68100f0d00 con 0x7f6820102d30 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.791+0000 7f681f7fe640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68080a3350 0x7f68080a5810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f68200fdaf0 tx=0x7f681400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.791+0000 7f68249d5640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559] conn(0x7f68080ae520 0x7f68080b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.791+0000 7f68249d5640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559] conn(0x7f68080ae520 0x7f68080b09c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.792+0000 7f681d7fa640 1 -- 172.21.15.33:0/4088916768 <== osd.7 v2:172.21.15.60:6816/1473789559 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6810046e50 con 0x7f68080ae520 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.806+0000 7f6826438640 1 -- 172.21.15.33:0/4088916768 --> [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f682006a850 con 0x7f68080ae520 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.806+0000 7f681d7fa640 1 -- 172.21.15.33:0/4088916768 <== osd.7 v2:172.21.15.60:6816/1473789559 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f682006a850 con 0x7f68080ae520 2024-09-16T00:16:16.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.807+0000 7f6826438640 1 -- 172.21.15.33:0/4088916768 >> [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559] conn(0x7f68080ae520 msgr2=0x7f68080b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:16.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.807+0000 7f6826438640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559] conn(0x7f68080ae520 0x7f68080b09c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:16.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.807+0000 7f6826438640 1 -- 172.21.15.33:0/4088916768 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68080a3350 msgr2=0x7f68080a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:16.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.807+0000 7f6826438640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68080a3350 0x7f68080a5810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f68200fdaf0 tx=0x7f681400a2d0 comp rx=0 tx=0).stop 2024-09-16T00:16:16.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.807+0000 7f6826438640 1 -- 172.21.15.33:0/4088916768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6820102d30 msgr2=0x7f68201984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:16.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.807+0000 7f6826438640 1 --2- 172.21.15.33:0/4088916768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6820102d30 0x7f68201984e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f681002a960 tx=0x7f6810003c00 comp rx=0 tx=0).stop 2024-09-16T00:16:16.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.807+0000 7f68249d5640 1 -- 172.21.15.33:0/4088916768 reap_dead start 2024-09-16T00:16:16.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.807+0000 7f6826438640 1 -- 172.21.15.33:0/4088916768 shutdown_connections 2024-09-16T00:16:16.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.807+0000 7f6826438640 1 -- 172.21.15.33:0/4088916768 >> 172.21.15.33:0/4088916768 conn(0x7f68200fc7d0 msgr2=0x7f68200fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:16.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.807+0000 7f6826438640 1 -- 172.21.15.33:0/4088916768 shutdown_connections 2024-09-16T00:16:16.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:16.807+0000 7f6826438640 1 -- 172.21.15.33:0/4088916768 wait complete. 2024-09-16T00:16:17.037 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:17.037 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:17.037 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:17.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:17 smithi033 ceph-mon[31243]: pgmap v322: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.5 KiB/s wr, 7 op/s; 7.6 KiB/s, 1 objects/s recovering 2024-09-16T00:16:17.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.682+0000 7fdf604f6640 1 -- 172.21.15.33:0/2086766011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf5810e500 msgr2=0x7fdf58110910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:17.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.682+0000 7fdf604f6640 1 --2- 172.21.15.33:0/2086766011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf5810e500 0x7fdf58110910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009ca0 tx=0x7fdf5002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.682+0000 7fdf604f6640 1 -- 172.21.15.33:0/2086766011 shutdown_connections 2024-09-16T00:16:17.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.682+0000 7fdf604f6640 1 --2- 172.21.15.33:0/2086766011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf5810e500 0x7fdf58110910 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.682+0000 7fdf604f6640 1 --2- 172.21.15.33:0/2086766011 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf58101440 0x7fdf5810df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.682+0000 7fdf604f6640 1 --2- 172.21.15.33:0/2086766011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf58100b20 0x7fdf58100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.682+0000 7fdf604f6640 1 -- 172.21.15.33:0/2086766011 >> 172.21.15.33:0/2086766011 conn(0x7fdf580fc7d0 msgr2=0x7fdf580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:17.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.682+0000 7fdf604f6640 1 -- 172.21.15.33:0/2086766011 shutdown_connections 2024-09-16T00:16:17.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.682+0000 7fdf604f6640 1 -- 172.21.15.33:0/2086766011 wait complete. 2024-09-16T00:16:17.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf604f6640 1 Processor -- start 2024-09-16T00:16:17.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf604f6640 1 -- start start 2024-09-16T00:16:17.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf604f6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf58100b20 0x7fdf5819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:17.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf604f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf58101440 0x7fdf5819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:17.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf604f6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5810e500 0x7fdf5819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:17.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf604f6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdf58114250 con 0x7fdf58101440 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf604f6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdf581140d0 con 0x7fdf5810e500 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf604f6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf581143d0 con 0x7fdf58100b20 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf5da91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf58101440 0x7fdf5819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf5da91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf58101440 0x7fdf5819ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59326/0 (socket says 172.21.15.33:59326) 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf5da91640 1 -- 172.21.15.33:0/1140842175 learned_addr learned my addr 172.21.15.33:0/1140842175 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf5ea93640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5810e500 0x7fdf5819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.683+0000 7fdf5e292640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf58100b20 0x7fdf5819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.684+0000 7fdf5da91640 1 -- 172.21.15.33:0/1140842175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf58100b20 msgr2=0x7fdf5819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.684+0000 7fdf5da91640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf58100b20 0x7fdf5819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.684+0000 7fdf5da91640 1 -- 172.21.15.33:0/1140842175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5810e500 msgr2=0x7fdf5819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.684+0000 7fdf5da91640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5810e500 0x7fdf5819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.684+0000 7fdf5da91640 1 -- 172.21.15.33:0/1140842175 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5819f6e0 con 0x7fdf58101440 2024-09-16T00:16:17.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.684+0000 7fdf5da91640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf58101440 0x7fdf5819ac70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdf4800e9c0 tx=0x7fdf4800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.684+0000 7fdf4f7fe640 1 -- 172.21.15.33:0/1140842175 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4800cd80 con 0x7fdf58101440 2024-09-16T00:16:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.684+0000 7fdf4f7fe640 1 -- 172.21.15.33:0/1140842175 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf480035f0 con 0x7fdf58101440 2024-09-16T00:16:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.684+0000 7fdf4f7fe640 1 -- 172.21.15.33:0/1140842175 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48010690 con 0x7fdf58101440 2024-09-16T00:16:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.684+0000 7fdf604f6640 1 -- 172.21.15.33:0/1140842175 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf5819f9d0 con 0x7fdf58101440 2024-09-16T00:16:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.684+0000 7fdf604f6640 1 -- 172.21.15.33:0/1140842175 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf581a72b0 con 0x7fdf58101440 2024-09-16T00:16:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.685+0000 7fdf604f6640 1 -- 172.21.15.33:0/1140842175 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdf58062980 con 0x7fdf58101440 2024-09-16T00:16:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.687+0000 7fdf4f7fe640 1 -- 172.21.15.33:0/1140842175 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fdf480037a0 con 0x7fdf58101440 2024-09-16T00:16:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.687+0000 7fdf4f7fe640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdf440a3350 0x7fdf440a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.687+0000 7fdf5e292640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdf440a3350 0x7fdf440a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.687+0000 7fdf4f7fe640 1 -- 172.21.15.33:0/1140842175 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fdf480d04f0 con 0x7fdf58101440 2024-09-16T00:16:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.688+0000 7fdf4f7fe640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079] conn(0x7fdf440ae520 0x7fdf440b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.688+0000 7fdf4f7fe640 1 -- 172.21.15.33:0/1140842175 --> [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf48023e50 con 0x7fdf440ae520 2024-09-16T00:16:17.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.688+0000 7fdf4f7fe640 1 -- 172.21.15.33:0/1140842175 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fdf480d0940 con 0x7fdf58101440 2024-09-16T00:16:17.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.688+0000 7fdf5e292640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdf440a3350 0x7fdf440a5810 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdf54005fd0 tx=0x7fdf54005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:17.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.688+0000 7fdf5ea93640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079] conn(0x7fdf440ae520 0x7fdf440b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:17.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.688+0000 7fdf5ea93640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079] conn(0x7fdf440ae520 0x7fdf440b09c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:17.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.689+0000 7fdf4f7fe640 1 -- 172.21.15.33:0/1140842175 <== osd.10 v2:172.21.15.60:6824/3903707079 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdf48023e50 con 0x7fdf440ae520 2024-09-16T00:16:17.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.704+0000 7fdf604f6640 1 -- 172.21.15.33:0/1140842175 --> [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdf58062e50 con 0x7fdf440ae520 2024-09-16T00:16:17.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.705+0000 7fdf4f7fe640 1 -- 172.21.15.33:0/1140842175 <== osd.10 v2:172.21.15.60:6824/3903707079 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdf58062e50 con 0x7fdf440ae520 2024-09-16T00:16:17.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.705+0000 7fdf604f6640 1 -- 172.21.15.33:0/1140842175 >> [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079] conn(0x7fdf440ae520 msgr2=0x7fdf440b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:17.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.705+0000 7fdf604f6640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079] conn(0x7fdf440ae520 0x7fdf440b09c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.705+0000 7fdf604f6640 1 -- 172.21.15.33:0/1140842175 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdf440a3350 msgr2=0x7fdf440a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:17.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.705+0000 7fdf604f6640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdf440a3350 0x7fdf440a5810 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdf54005fd0 tx=0x7fdf54005850 comp rx=0 tx=0).stop 2024-09-16T00:16:17.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.705+0000 7fdf604f6640 1 -- 172.21.15.33:0/1140842175 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf58101440 msgr2=0x7fdf5819ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:17.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.705+0000 7fdf604f6640 1 --2- 172.21.15.33:0/1140842175 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf58101440 0x7fdf5819ac70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdf4800e9c0 tx=0x7fdf4800ee80 comp rx=0 tx=0).stop 2024-09-16T00:16:17.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.705+0000 7fdf5ea93640 1 -- 172.21.15.33:0/1140842175 reap_dead start 2024-09-16T00:16:17.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.705+0000 7fdf604f6640 1 -- 172.21.15.33:0/1140842175 shutdown_connections 2024-09-16T00:16:17.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.705+0000 7fdf604f6640 1 -- 172.21.15.33:0/1140842175 >> 172.21.15.33:0/1140842175 conn(0x7fdf580fc7d0 msgr2=0x7fdf580febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:17.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.706+0000 7fdf604f6640 1 -- 172.21.15.33:0/1140842175 shutdown_connections 2024-09-16T00:16:17.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.706+0000 7fdf604f6640 1 -- 172.21.15.33:0/1140842175 wait complete. 2024-09-16T00:16:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:17 smithi060 ceph-mon[35185]: pgmap v322: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.5 KiB/s wr, 7 op/s; 7.6 KiB/s, 1 objects/s recovering 2024-09-16T00:16:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:17 smithi083 ceph-mon[35242]: pgmap v322: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.5 KiB/s wr, 7 op/s; 7.6 KiB/s, 1 objects/s recovering 2024-09-16T00:16:17.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.954+0000 7f06a2b35640 1 -- 172.21.15.33:0/2615404116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f069c108290 msgr2=0x7f069c10ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:17.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.954+0000 7f06a2b35640 1 --2- 172.21.15.33:0/2615404116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f069c108290 0x7f069c10ec30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0694009cd0 tx=0x7f069402a8e0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.954+0000 7f06a2b35640 1 -- 172.21.15.33:0/2615404116 shutdown_connections 2024-09-16T00:16:17.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.954+0000 7f06a2b35640 1 --2- 172.21.15.33:0/2615404116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f069c108290 0x7f069c10ec30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.954+0000 7f06a2b35640 1 --2- 172.21.15.33:0/2615404116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f069c103740 0x7f069c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.954+0000 7f06a2b35640 1 --2- 172.21.15.33:0/2615404116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069c102d90 0x7f069c103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.954+0000 7f06a2b35640 1 -- 172.21.15.33:0/2615404116 >> 172.21.15.33:0/2615404116 conn(0x7f069c0fc7d0 msgr2=0x7f069c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:17.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.954+0000 7f06a2b35640 1 -- 172.21.15.33:0/2615404116 shutdown_connections 2024-09-16T00:16:17.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.954+0000 7f06a2b35640 1 -- 172.21.15.33:0/2615404116 wait complete. 2024-09-16T00:16:17.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f06a2b35640 1 Processor -- start 2024-09-16T00:16:17.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f06a2b35640 1 -- start start 2024-09-16T00:16:17.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f06a2b35640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069c102d90 0x7f069c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:17.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f06a2b35640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f069c103740 0x7f069c198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:17.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f06a2b35640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f069c108290 0x7f069c19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:17.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f06a2b35640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f069c112090 con 0x7f069c103740 2024-09-16T00:16:17.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f06a2b35640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f069c111f10 con 0x7f069c102d90 2024-09-16T00:16:17.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f06a2b35640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f069c112210 con 0x7f069c108290 2024-09-16T00:16:17.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f0693fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f069c103740 0x7f069c198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:17.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f0693fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f069c103740 0x7f069c198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59344/0 (socket says 172.21.15.33:59344) 2024-09-16T00:16:17.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f0693fff640 1 -- 172.21.15.33:0/3116339807 learned_addr learned my addr 172.21.15.33:0/3116339807 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f06a10d2640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f069c108290 0x7f069c19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f06a08d1640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069c102d90 0x7f069c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f0693fff640 1 -- 172.21.15.33:0/3116339807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f069c108290 msgr2=0x7f069c19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f0693fff640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f069c108290 0x7f069c19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f0693fff640 1 -- 172.21.15.33:0/3116339807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069c102d90 msgr2=0x7f069c198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f0693fff640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069c102d90 0x7f069c198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.955+0000 7f0693fff640 1 -- 172.21.15.33:0/3116339807 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f069c19d500 con 0x7f069c103740 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.956+0000 7f0693fff640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f069c103740 0x7f069c198a90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f068c00e9f0 tx=0x7f068c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.956+0000 7f0691ffb640 1 -- 172.21.15.33:0/3116339807 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068c00ccc0 con 0x7f069c103740 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.956+0000 7f0691ffb640 1 -- 172.21.15.33:0/3116339807 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f068c005500 con 0x7f069c103740 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.956+0000 7f06a2b35640 1 -- 172.21.15.33:0/3116339807 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f069c19d7f0 con 0x7f069c103740 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.956+0000 7f0691ffb640 1 -- 172.21.15.33:0/3116339807 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068c010690 con 0x7f069c103740 2024-09-16T00:16:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.956+0000 7f06a2b35640 1 -- 172.21.15.33:0/3116339807 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f069c1a50d0 con 0x7f069c103740 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.957+0000 7f06a2b35640 1 -- 172.21.15.33:0/3116339807 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f069c062980 con 0x7f069c103740 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.958+0000 7f0691ffb640 1 -- 172.21.15.33:0/3116339807 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f068c0046c0 con 0x7f069c103740 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.959+0000 7f0691ffb640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f06840a32d0 0x7f06840a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.959+0000 7f06a08d1640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f06840a32d0 0x7f06840a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.959+0000 7f0691ffb640 1 -- 172.21.15.33:0/3116339807 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f068c0d0610 con 0x7f069c103740 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.959+0000 7f0691ffb640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362] conn(0x7f06840ae4a0 0x7f06840b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.959+0000 7f0691ffb640 1 -- 172.21.15.33:0/3116339807 --> [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06840b1010 con 0x7f06840ae4a0 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.959+0000 7f0691ffb640 1 -- 172.21.15.33:0/3116339807 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f068c0d09c0 con 0x7f069c103740 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.959+0000 7f06a10d2640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362] conn(0x7f06840ae4a0 0x7f06840b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.959+0000 7f06a08d1640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f06840a32d0 0x7f06840a5790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0698005fd0 tx=0x7f069800b590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.960+0000 7f06a10d2640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362] conn(0x7f06840ae4a0 0x7f06840b0940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.961+0000 7f0691ffb640 1 -- 172.21.15.33:0/3116339807 <== osd.3 v2:172.21.15.33:6810/712525362 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06840b1010 con 0x7f06840ae4a0 2024-09-16T00:16:17.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.975+0000 7f06a2b35640 1 -- 172.21.15.33:0/3116339807 --> [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f069c062e00 con 0x7f06840ae4a0 2024-09-16T00:16:17.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.975+0000 7f0691ffb640 1 -- 172.21.15.33:0/3116339807 <== osd.3 v2:172.21.15.33:6810/712525362 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f069c062e00 con 0x7f06840ae4a0 2024-09-16T00:16:17.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.975+0000 7f06a2b35640 1 -- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362] conn(0x7f06840ae4a0 msgr2=0x7f06840b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:17.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.975+0000 7f06a2b35640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362] conn(0x7f06840ae4a0 0x7f06840b0940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.975+0000 7f06a2b35640 1 -- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f06840a32d0 msgr2=0x7f06840a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:17.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.975+0000 7f06a2b35640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f06840a32d0 0x7f06840a5790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0698005fd0 tx=0x7f069800b590 comp rx=0 tx=0).stop 2024-09-16T00:16:17.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.975+0000 7f06a2b35640 1 -- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f069c103740 msgr2=0x7f069c198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:17.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.975+0000 7f06a2b35640 1 --2- 172.21.15.33:0/3116339807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f069c103740 0x7f069c198a90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f068c00e9f0 tx=0x7f068c00eeb0 comp rx=0 tx=0).stop 2024-09-16T00:16:17.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.975+0000 7f06a10d2640 1 -- 172.21.15.33:0/3116339807 reap_dead start 2024-09-16T00:16:17.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.976+0000 7f06a2b35640 1 -- 172.21.15.33:0/3116339807 shutdown_connections 2024-09-16T00:16:17.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.976+0000 7f06a2b35640 1 -- 172.21.15.33:0/3116339807 >> 172.21.15.33:0/3116339807 conn(0x7f069c0fc7d0 msgr2=0x7f069c106350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:17.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.976+0000 7f06a2b35640 1 -- 172.21.15.33:0/3116339807 shutdown_connections 2024-09-16T00:16:17.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:17.976+0000 7f06a2b35640 1 -- 172.21.15.33:0/3116339807 wait complete. 2024-09-16T00:16:18.866 INFO:teuthology.orchestra.run.smithi033.stdout:206158430256 2024-09-16T00:16:18.867 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-16T00:16:19.173 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:19.173 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:19.714 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:19 smithi033 ceph-mon[31243]: pgmap v323: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s; 6.3 KiB/s, 1 objects/s recovering 2024-09-16T00:16:19.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:19 smithi060 ceph-mon[35185]: pgmap v323: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s; 6.3 KiB/s, 1 objects/s recovering 2024-09-16T00:16:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.714+0000 7fced6c26640 1 -- 172.21.15.33:0/1396387799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced01012e0 msgr2=0x7fced010dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.714+0000 7fced6c26640 1 --2- 172.21.15.33:0/1396387799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced01012e0 0x7fced010dfa0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009900 tx=0x7fcebc02a410 comp rx=0 tx=0).stop 2024-09-16T00:16:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.714+0000 7fced6c26640 1 -- 172.21.15.33:0/1396387799 shutdown_connections 2024-09-16T00:16:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.714+0000 7fced6c26640 1 --2- 172.21.15.33:0/1396387799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fced010e4e0 0x7fced01108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.714+0000 7fced6c26640 1 --2- 172.21.15.33:0/1396387799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced01012e0 0x7fced010dfa0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.714+0000 7fced6c26640 1 --2- 172.21.15.33:0/1396387799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fced0100930 0x7fced0100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.714+0000 7fced6c26640 1 -- 172.21.15.33:0/1396387799 >> 172.21.15.33:0/1396387799 conn(0x7fced00fc7a0 msgr2=0x7fced00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.714+0000 7fced6c26640 1 -- 172.21.15.33:0/1396387799 shutdown_connections 2024-09-16T00:16:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.714+0000 7fced6c26640 1 -- 172.21.15.33:0/1396387799 wait complete. 2024-09-16T00:16:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced6c26640 1 Processor -- start 2024-09-16T00:16:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced6c26640 1 -- start start 2024-09-16T00:16:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced6c26640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced0100930 0x7fced019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced6c26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fced01012e0 0x7fced019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced6c26640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fced010e4e0 0x7fced019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced6c26640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fced0114180 con 0x7fced0100930 2024-09-16T00:16:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced6c26640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fced0114000 con 0x7fced01012e0 2024-09-16T00:16:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced6c26640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fced0114300 con 0x7fced010e4e0 2024-09-16T00:16:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced49c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced0100930 0x7fced019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced49c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced0100930 0x7fced019a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59366/0 (socket says 172.21.15.33:59366) 2024-09-16T00:16:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced49c2640 1 -- 172.21.15.33:0/2069648020 learned_addr learned my addr 172.21.15.33:0/2069648020 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fcecffff640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fced01012e0 0x7fced019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced51c3640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fced010e4e0 0x7fced019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced49c2640 1 -- 172.21.15.33:0/2069648020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fced010e4e0 msgr2=0x7fced019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced49c2640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fced010e4e0 0x7fced019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced49c2640 1 -- 172.21.15.33:0/2069648020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fced01012e0 msgr2=0x7fced019aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced49c2640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fced01012e0 0x7fced019aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.715+0000 7fced49c2640 1 -- 172.21.15.33:0/2069648020 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced019f610 con 0x7fced0100930 2024-09-16T00:16:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.716+0000 7fced49c2640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced0100930 0x7fced019a660 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcec400e9f0 tx=0x7fcec400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.716+0000 7fcecdffb640 1 -- 172.21.15.33:0/2069648020 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec400cd80 con 0x7fced0100930 2024-09-16T00:16:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.716+0000 7fcecdffb640 1 -- 172.21.15.33:0/2069648020 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcec4010910 con 0x7fced0100930 2024-09-16T00:16:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.716+0000 7fced6c26640 1 -- 172.21.15.33:0/2069648020 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced019f900 con 0x7fced0100930 2024-09-16T00:16:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.716+0000 7fcecdffb640 1 -- 172.21.15.33:0/2069648020 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec4018990 con 0x7fced0100930 2024-09-16T00:16:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.716+0000 7fced6c26640 1 -- 172.21.15.33:0/2069648020 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced01a71e0 con 0x7fced0100930 2024-09-16T00:16:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.717+0000 7fced6c26640 1 -- 172.21.15.33:0/2069648020 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fced0062980 con 0x7fced0100930 2024-09-16T00:16:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.718+0000 7fcecdffb640 1 -- 172.21.15.33:0/2069648020 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fcec4010ac0 con 0x7fced0100930 2024-09-16T00:16:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.719+0000 7fcecdffb640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcec00a3350 0x7fcec00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.719+0000 7fcecffff640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcec00a3350 0x7fcec00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.719+0000 7fcecdffb640 1 -- 172.21.15.33:0/2069648020 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fcec40d0730 con 0x7fced0100930 2024-09-16T00:16:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.719+0000 7fcecdffb640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656] conn(0x7fcec00ae520 0x7fcec00b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.719+0000 7fcecdffb640 1 -- 172.21.15.33:0/2069648020 --> [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcec4019ec0 con 0x7fcec00ae520 2024-09-16T00:16:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.719+0000 7fcecdffb640 1 -- 172.21.15.33:0/2069648020 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fcec40d0b80 con 0x7fced0100930 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.719+0000 7fced51c3640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656] conn(0x7fcec00ae520 0x7fcec00b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.719+0000 7fcecffff640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcec00a3350 0x7fcec00a5810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009230 tx=0x7fcebc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.720+0000 7fced51c3640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656] conn(0x7fcec00ae520 0x7fcec00b09c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.720+0000 7fcecdffb640 1 -- 172.21.15.33:0/2069648020 <== osd.0 v2:172.21.15.33:6802/1463114656 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcec4019ec0 con 0x7fcec00ae520 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.734+0000 7fced6c26640 1 -- 172.21.15.33:0/2069648020 --> [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fced0062d80 con 0x7fcec00ae520 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.734+0000 7fcecdffb640 1 -- 172.21.15.33:0/2069648020 <== osd.0 v2:172.21.15.33:6802/1463114656 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fced0062d80 con 0x7fcec00ae520 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.734+0000 7fced6c26640 1 -- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656] conn(0x7fcec00ae520 msgr2=0x7fcec00b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.735+0000 7fced6c26640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656] conn(0x7fcec00ae520 0x7fcec00b09c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.735+0000 7fced6c26640 1 -- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcec00a3350 msgr2=0x7fcec00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.735+0000 7fced6c26640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcec00a3350 0x7fcec00a5810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009230 tx=0x7fcebc0058d0 comp rx=0 tx=0).stop 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.735+0000 7fced6c26640 1 -- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced0100930 msgr2=0x7fced019a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.735+0000 7fced6c26640 1 --2- 172.21.15.33:0/2069648020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced0100930 0x7fced019a660 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcec400e9f0 tx=0x7fcec400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:16:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.735+0000 7fced51c3640 1 -- 172.21.15.33:0/2069648020 reap_dead start 2024-09-16T00:16:19.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.735+0000 7fced6c26640 1 -- 172.21.15.33:0/2069648020 shutdown_connections 2024-09-16T00:16:19.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.735+0000 7fced6c26640 1 -- 172.21.15.33:0/2069648020 >> 172.21.15.33:0/2069648020 conn(0x7fced00fc7a0 msgr2=0x7fced00feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:19.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.735+0000 7fced6c26640 1 -- 172.21.15.33:0/2069648020 shutdown_connections 2024-09-16T00:16:19.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:19.735+0000 7fced6c26640 1 -- 172.21.15.33:0/2069648020 wait complete. 2024-09-16T00:16:19.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:19 smithi083 ceph-mon[35242]: pgmap v323: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s; 6.3 KiB/s, 1 objects/s recovering 2024-09-16T00:16:20.310 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:20.310 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:20.766 INFO:teuthology.orchestra.run.smithi033.stdout:34359738453 2024-09-16T00:16:20.766 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-16T00:16:21.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.486+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2724782224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f04106f10 msgr2=0x7f3f0410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:21.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.486+0000 7f3f08b5c640 1 --2- 172.21.15.33:0/2724782224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f04106f10 0x7f3f0410ea90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009ca0 tx=0x7f3efc02a8c0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.486+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2724782224 shutdown_connections 2024-09-16T00:16:21.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.486+0000 7f3f08b5c640 1 --2- 172.21.15.33:0/2724782224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f04106f10 0x7f3f0410ea90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.486+0000 7f3f08b5c640 1 --2- 172.21.15.33:0/2724782224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f04102790 0x7f3f041067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.486+0000 7f3f08b5c640 1 --2- 172.21.15.33:0/2724782224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f04101de0 0x7f3f041021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.486+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2724782224 >> 172.21.15.33:0/2724782224 conn(0x7f3f040fd8c0 msgr2=0x7f3f040ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:21.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.486+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2724782224 shutdown_connections 2024-09-16T00:16:21.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.486+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2724782224 wait complete. 2024-09-16T00:16:21.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.486+0000 7f3f08b5c640 1 Processor -- start 2024-09-16T00:16:21.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f08b5c640 1 -- start start 2024-09-16T00:16:21.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f08b5c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f04101de0 0x7f3f040744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:21.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f08b5c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f04102790 0x7f3f04074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:21.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f08b5c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f04106f10 0x7f3f04074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:21.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f08b5c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3f04111f70 con 0x7f3f04106f10 2024-09-16T00:16:21.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f08b5c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3f04111df0 con 0x7f3f04101de0 2024-09-16T00:16:21.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f08b5c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3f041120f0 con 0x7f3f04102790 2024-09-16T00:16:21.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f02d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f04106f10 0x7f3f04074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:21.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f02d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f04106f10 0x7f3f04074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39780/0 (socket says 172.21.15.33:39780) 2024-09-16T00:16:21.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f0259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f04101de0 0x7f3f040744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f02d9d640 1 -- 172.21.15.33:0/2024517373 learned_addr learned my addr 172.21.15.33:0/2024517373 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3ef9d9b640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f04102790 0x7f3f04074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f02d9d640 1 -- 172.21.15.33:0/2024517373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f04102790 msgr2=0x7f3f04074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f02d9d640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f04102790 0x7f3f04074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f02d9d640 1 -- 172.21.15.33:0/2024517373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f04101de0 msgr2=0x7f3f040744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f02d9d640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f04101de0 0x7f3f040744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.487+0000 7f3f02d9d640 1 -- 172.21.15.33:0/2024517373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f041a6890 con 0x7f3f04106f10 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.488+0000 7f3f02d9d640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f04106f10 0x7f3f04074f70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3efc031a20 tx=0x7f3efc031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.488+0000 7f3efbfff640 1 -- 172.21.15.33:0/2024517373 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc003c30 con 0x7f3f04106f10 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.488+0000 7f3efbfff640 1 -- 172.21.15.33:0/2024517373 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3efc003dd0 con 0x7f3f04106f10 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.488+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2024517373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f041a6b20 con 0x7f3f04106f10 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.488+0000 7f3efbfff640 1 -- 172.21.15.33:0/2024517373 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc02fcf0 con 0x7f3f04106f10 2024-09-16T00:16:21.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.488+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2024517373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f041a7060 con 0x7f3f04106f10 2024-09-16T00:16:21.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.489+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2024517373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3f040628c0 con 0x7f3f04106f10 2024-09-16T00:16:21.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.490+0000 7f3efbfff640 1 -- 172.21.15.33:0/2024517373 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f3efc004050 con 0x7f3f04106f10 2024-09-16T00:16:21.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.491+0000 7f3efbfff640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3ef00a32d0 0x7f3ef00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:21.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.491+0000 7f3f0259c640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3ef00a32d0 0x7f3ef00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:21.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.491+0000 7f3efbfff640 1 -- 172.21.15.33:0/2024517373 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f3efc0f11d0 con 0x7f3f04106f10 2024-09-16T00:16:21.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.491+0000 7f3efbfff640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527] conn(0x7f3ef00ae4a0 0x7f3ef00b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:21.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.491+0000 7f3efbfff640 1 -- 172.21.15.33:0/2024517373 --> [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ef00b1010 con 0x7f3ef00ae4a0 2024-09-16T00:16:21.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.491+0000 7f3efbfff640 1 -- 172.21.15.33:0/2024517373 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f3efc0f1620 con 0x7f3f04106f10 2024-09-16T00:16:21.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.491+0000 7f3f0259c640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3ef00a32d0 0x7f3ef00a5790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4005fd0 tx=0x7f3ef400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:21.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.491+0000 7f3ef9d9b640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527] conn(0x7f3ef00ae4a0 0x7f3ef00b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:21.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.492+0000 7f3ef9d9b640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527] conn(0x7f3ef00ae4a0 0x7f3ef00b0940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:21.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.493+0000 7f3efbfff640 1 -- 172.21.15.33:0/2024517373 <== osd.8 v2:172.21.15.83:6816/2115210527 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3ef00b1010 con 0x7f3ef00ae4a0 2024-09-16T00:16:21.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.507+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2024517373 --> [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3f04062d40 con 0x7f3ef00ae4a0 2024-09-16T00:16:21.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.508+0000 7f3efbfff640 1 -- 172.21.15.33:0/2024517373 <== osd.8 v2:172.21.15.83:6816/2115210527 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3f04062d40 con 0x7f3ef00ae4a0 2024-09-16T00:16:21.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.508+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2024517373 >> [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527] conn(0x7f3ef00ae4a0 msgr2=0x7f3ef00b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:21.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.508+0000 7f3f08b5c640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527] conn(0x7f3ef00ae4a0 0x7f3ef00b0940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.508+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2024517373 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3ef00a32d0 msgr2=0x7f3ef00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:21.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.508+0000 7f3f08b5c640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3ef00a32d0 0x7f3ef00a5790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4005fd0 tx=0x7f3ef400b390 comp rx=0 tx=0).stop 2024-09-16T00:16:21.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.508+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2024517373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f04106f10 msgr2=0x7f3f04074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:21.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.508+0000 7f3f08b5c640 1 --2- 172.21.15.33:0/2024517373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f04106f10 0x7f3f04074f70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3efc031a20 tx=0x7f3efc031a50 comp rx=0 tx=0).stop 2024-09-16T00:16:21.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.508+0000 7f3f02d9d640 1 -- 172.21.15.33:0/2024517373 reap_dead start 2024-09-16T00:16:21.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.508+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2024517373 shutdown_connections 2024-09-16T00:16:21.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.508+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2024517373 >> 172.21.15.33:0/2024517373 conn(0x7f3f040fd8c0 msgr2=0x7f3f040fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:21.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.508+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2024517373 shutdown_connections 2024-09-16T00:16:21.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.508+0000 7f3f08b5c640 1 -- 172.21.15.33:0/2024517373 wait complete. 2024-09-16T00:16:21.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:21 smithi033 ceph-mon[31243]: pgmap v324: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 288 B/s wr, 5 op/s; 5.7 KiB/s, 1 objects/s recovering 2024-09-16T00:16:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.695+0000 7f103bce1640 1 -- 172.21.15.33:0/2559678869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1034102790 msgr2=0x7f10341067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.695+0000 7f103bce1640 1 --2- 172.21.15.33:0/2559678869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1034102790 0x7f10341067e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1024009900 tx=0x7f102402a410 comp rx=0 tx=0).stop 2024-09-16T00:16:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.695+0000 7f103bce1640 1 -- 172.21.15.33:0/2559678869 shutdown_connections 2024-09-16T00:16:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.695+0000 7f103bce1640 1 --2- 172.21.15.33:0/2559678869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1034106f10 0x7f103410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.695+0000 7f103bce1640 1 --2- 172.21.15.33:0/2559678869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1034102790 0x7f10341067e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.695+0000 7f103bce1640 1 --2- 172.21.15.33:0/2559678869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1034101de0 0x7f10341021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.695+0000 7f103bce1640 1 -- 172.21.15.33:0/2559678869 >> 172.21.15.33:0/2559678869 conn(0x7f10340fd8c0 msgr2=0x7f10340ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.696+0000 7f103bce1640 1 -- 172.21.15.33:0/2559678869 shutdown_connections 2024-09-16T00:16:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.696+0000 7f103bce1640 1 -- 172.21.15.33:0/2559678869 wait complete. 2024-09-16T00:16:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.696+0000 7f103bce1640 1 Processor -- start 2024-09-16T00:16:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.696+0000 7f103bce1640 1 -- start start 2024-09-16T00:16:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.696+0000 7f103bce1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1034101de0 0x7f10340744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.696+0000 7f103bce1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1034102790 0x7f1034074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.696+0000 7f103bce1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1034106f10 0x7f1034074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.696+0000 7f103bce1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1034111f70 con 0x7f1034106f10 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.696+0000 7f103bce1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1034111df0 con 0x7f1034102790 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.696+0000 7f103bce1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f10341120f0 con 0x7f1034101de0 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103a27e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1034106f10 0x7f1034074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f1039a7d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1034101de0 0x7f10340744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103a27e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1034106f10 0x7f1034074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39800/0 (socket says 172.21.15.33:39800) 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103a27e640 1 -- 172.21.15.33:0/2036005062 learned_addr learned my addr 172.21.15.33:0/2036005062 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103927c640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1034102790 0x7f1034074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103a27e640 1 -- 172.21.15.33:0/2036005062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1034101de0 msgr2=0x7f10340744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103a27e640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1034101de0 0x7f10340744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103a27e640 1 -- 172.21.15.33:0/2036005062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1034102790 msgr2=0x7f1034074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103a27e640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1034102790 0x7f1034074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103a27e640 1 -- 172.21.15.33:0/2036005062 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10341a6890 con 0x7f1034106f10 2024-09-16T00:16:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f1039a7d640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1034101de0 0x7f10340744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:16:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103927c640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1034102790 0x7f1034074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:16:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103a27e640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1034106f10 0x7f1034074f70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f102c00cce0 tx=0x7f102c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f102affd640 1 -- 172.21.15.33:0/2036005062 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c00f450 con 0x7f1034106f10 2024-09-16T00:16:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f102affd640 1 -- 172.21.15.33:0/2036005062 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f102c00fad0 con 0x7f1034106f10 2024-09-16T00:16:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103bce1640 1 -- 172.21.15.33:0/2036005062 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10341a6b80 con 0x7f1034106f10 2024-09-16T00:16:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f102affd640 1 -- 172.21.15.33:0/2036005062 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c018c10 con 0x7f1034106f10 2024-09-16T00:16:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.697+0000 7f103bce1640 1 -- 172.21.15.33:0/2036005062 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10341a7070 con 0x7f1034106f10 2024-09-16T00:16:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.698+0000 7f103bce1640 1 -- 172.21.15.33:0/2036005062 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1034062880 con 0x7f1034106f10 2024-09-16T00:16:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.700+0000 7f102affd640 1 -- 172.21.15.33:0/2036005062 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f102c00f5f0 con 0x7f1034106f10 2024-09-16T00:16:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.700+0000 7f102affd640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10200a32d0 0x7f10200a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.700+0000 7f1039a7d640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10200a32d0 0x7f10200a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.700+0000 7f102affd640 1 -- 172.21.15.33:0/2036005062 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f102c0d06d0 con 0x7f1034106f10 2024-09-16T00:16:21.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.701+0000 7f102affd640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075] conn(0x7f10200ae4a0 0x7f10200b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:21.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.701+0000 7f102affd640 1 -- 172.21.15.33:0/2036005062 --> [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10200b1010 con 0x7f10200ae4a0 2024-09-16T00:16:21.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.701+0000 7f102affd640 1 -- 172.21.15.33:0/2036005062 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f102c0d0b20 con 0x7f1034106f10 2024-09-16T00:16:21.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.701+0000 7f1039a7d640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10200a32d0 0x7f10200a5790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1030005fd0 tx=0x7f103000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:21.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.701+0000 7f103927c640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075] conn(0x7f10200ae4a0 0x7f10200b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:21.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.701+0000 7f103927c640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075] conn(0x7f10200ae4a0 0x7f10200b0940 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:21.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.702+0000 7f102affd640 1 -- 172.21.15.33:0/2036005062 <== osd.11 v2:172.21.15.83:6824/2178604075 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10200b1010 con 0x7f10200ae4a0 2024-09-16T00:16:21.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.716+0000 7f103bce1640 1 -- 172.21.15.33:0/2036005062 --> [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1034062d00 con 0x7f10200ae4a0 2024-09-16T00:16:21.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.717+0000 7f102affd640 1 -- 172.21.15.33:0/2036005062 <== osd.11 v2:172.21.15.83:6824/2178604075 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1034062d00 con 0x7f10200ae4a0 2024-09-16T00:16:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.717+0000 7f103bce1640 1 -- 172.21.15.33:0/2036005062 >> [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075] conn(0x7f10200ae4a0 msgr2=0x7f10200b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.717+0000 7f103bce1640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075] conn(0x7f10200ae4a0 0x7f10200b0940 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.717+0000 7f103bce1640 1 -- 172.21.15.33:0/2036005062 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10200a32d0 msgr2=0x7f10200a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.717+0000 7f103bce1640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10200a32d0 0x7f10200a5790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1030005fd0 tx=0x7f103000b390 comp rx=0 tx=0).stop 2024-09-16T00:16:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.717+0000 7f103bce1640 1 -- 172.21.15.33:0/2036005062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1034106f10 msgr2=0x7f1034074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.717+0000 7f103bce1640 1 --2- 172.21.15.33:0/2036005062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1034106f10 0x7f1034074f70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f102c00cce0 tx=0x7f102c00a620 comp rx=0 tx=0).stop 2024-09-16T00:16:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.717+0000 7f103a27e640 1 -- 172.21.15.33:0/2036005062 reap_dead start 2024-09-16T00:16:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.718+0000 7f103bce1640 1 -- 172.21.15.33:0/2036005062 shutdown_connections 2024-09-16T00:16:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.718+0000 7f103bce1640 1 -- 172.21.15.33:0/2036005062 >> 172.21.15.33:0/2036005062 conn(0x7f10340fd8c0 msgr2=0x7f10340fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.718+0000 7f103bce1640 1 -- 172.21.15.33:0/2036005062 shutdown_connections 2024-09-16T00:16:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:21.718+0000 7f103bce1640 1 -- 172.21.15.33:0/2036005062 wait complete. 2024-09-16T00:16:21.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:21 smithi060 ceph-mon[35185]: pgmap v324: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 288 B/s wr, 5 op/s; 5.7 KiB/s, 1 objects/s recovering 2024-09-16T00:16:21.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:21 smithi083 ceph-mon[35242]: pgmap v324: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 288 B/s wr, 5 op/s; 5.7 KiB/s, 1 objects/s recovering 2024-09-16T00:16:22.110 INFO:teuthology.orchestra.run.smithi033.stdout:103079215172 2024-09-16T00:16:22.110 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-16T00:16:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.302+0000 7fe01b546640 1 -- 172.21.15.33:0/4244341531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014105560 msgr2=0x7fe01410d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.302+0000 7fe01b546640 1 --2- 172.21.15.33:0/4244341531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014105560 0x7fe01410d0e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe00c009ca0 tx=0x7fe00c02a8c0 comp rx=0 tx=0).stop 2024-09-16T00:16:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.302+0000 7fe01b546640 1 -- 172.21.15.33:0/4244341531 shutdown_connections 2024-09-16T00:16:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.302+0000 7fe01b546640 1 --2- 172.21.15.33:0/4244341531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014105560 0x7fe01410d0e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.302+0000 7fe01b546640 1 --2- 172.21.15.33:0/4244341531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe014100de0 0x7fe014104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.302+0000 7fe01b546640 1 --2- 172.21.15.33:0/4244341531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe014100430 0x7fe014100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.302+0000 7fe01b546640 1 -- 172.21.15.33:0/4244341531 >> 172.21.15.33:0/4244341531 conn(0x7fe014077230 msgr2=0x7fe0140fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.302+0000 7fe01b546640 1 -- 172.21.15.33:0/4244341531 shutdown_connections 2024-09-16T00:16:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.302+0000 7fe01b546640 1 -- 172.21.15.33:0/4244341531 wait complete. 2024-09-16T00:16:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe01b546640 1 Processor -- start 2024-09-16T00:16:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe01b546640 1 -- start start 2024-09-16T00:16:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe01b546640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe014100430 0x7fe0140fc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe01b546640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe014100de0 0x7fe0140fcc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe01b546640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014105560 0x7fe0140fd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe01b546640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe014110e40 con 0x7fe014105560 2024-09-16T00:16:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe01b546640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe014110cc0 con 0x7fe014100430 2024-09-16T00:16:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe01b546640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe014110fc0 con 0x7fe014100de0 2024-09-16T00:16:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe019ae3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014105560 0x7fe0140fd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe019ae3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014105560 0x7fe0140fd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39812/0 (socket says 172.21.15.33:39812) 2024-09-16T00:16:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe019ae3640 1 -- 172.21.15.33:0/1509543089 learned_addr learned my addr 172.21.15.33:0/1509543089 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe018ae1640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe014100de0 0x7fe0140fcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.303+0000 7fe0192e2640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe014100430 0x7fe0140fc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.304+0000 7fe018ae1640 1 -- 172.21.15.33:0/1509543089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe014100430 msgr2=0x7fe0140fc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.304+0000 7fe018ae1640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe014100430 0x7fe0140fc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.304+0000 7fe018ae1640 1 -- 172.21.15.33:0/1509543089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014105560 msgr2=0x7fe0140fd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.304+0000 7fe018ae1640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014105560 0x7fe0140fd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.304+0000 7fe018ae1640 1 -- 172.21.15.33:0/1509543089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0141a68a0 con 0x7fe014100de0 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.304+0000 7fe019ae3640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014105560 0x7fe0140fd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.304+0000 7fe018ae1640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe014100de0 0x7fe0140fcc40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe00400e9f0 tx=0x7fe00400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.304+0000 7fe00a7fc640 1 -- 172.21.15.33:0/1509543089 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00400ccc0 con 0x7fe014100de0 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.304+0000 7fe01b546640 1 -- 172.21.15.33:0/1509543089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0141a6b90 con 0x7fe014100de0 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.304+0000 7fe00a7fc640 1 -- 172.21.15.33:0/1509543089 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe004005020 con 0x7fe014100de0 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.304+0000 7fe00a7fc640 1 -- 172.21.15.33:0/1509543089 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0040049e0 con 0x7fe014100de0 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.304+0000 7fe01b546640 1 -- 172.21.15.33:0/1509543089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0141a70d0 con 0x7fe014100de0 2024-09-16T00:16:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.305+0000 7fe01b546640 1 -- 172.21.15.33:0/1509543089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe0140628c0 con 0x7fe014100de0 2024-09-16T00:16:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.307+0000 7fe00a7fc640 1 -- 172.21.15.33:0/1509543089 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fe004005560 con 0x7fe014100de0 2024-09-16T00:16:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.307+0000 7fe00a7fc640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0000a32d0 0x7fe0000a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.307+0000 7fe0192e2640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0000a32d0 0x7fe0000a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.307+0000 7fe00a7fc640 1 -- 172.21.15.33:0/1509543089 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fe0040cff80 con 0x7fe014100de0 2024-09-16T00:16:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.308+0000 7fe00a7fc640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488] conn(0x7fe0000ae4a0 0x7fe0000b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.308+0000 7fe00a7fc640 1 -- 172.21.15.33:0/1509543089 --> [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe0000b1010 con 0x7fe0000ae4a0 2024-09-16T00:16:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.308+0000 7fe00a7fc640 1 -- 172.21.15.33:0/1509543089 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fe0040d0330 con 0x7fe014100de0 2024-09-16T00:16:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.308+0000 7fe019ae3640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488] conn(0x7fe0000ae4a0 0x7fe0000b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.308+0000 7fe0192e2640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0000a32d0 0x7fe0000a5790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe010005d00 tx=0x7fe01000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.308+0000 7fe019ae3640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488] conn(0x7fe0000ae4a0 0x7fe0000b0940 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:22.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.309+0000 7fe00a7fc640 1 -- 172.21.15.33:0/1509543089 <== osd.4 v2:172.21.15.60:6808/3488110488 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0000b1010 con 0x7fe0000ae4a0 2024-09-16T00:16:22.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.324+0000 7fe01b546640 1 -- 172.21.15.33:0/1509543089 --> [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe014062d40 con 0x7fe0000ae4a0 2024-09-16T00:16:22.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.325+0000 7fe00a7fc640 1 -- 172.21.15.33:0/1509543089 <== osd.4 v2:172.21.15.60:6808/3488110488 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe014062d40 con 0x7fe0000ae4a0 2024-09-16T00:16:22.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.325+0000 7fe01b546640 1 -- 172.21.15.33:0/1509543089 >> [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488] conn(0x7fe0000ae4a0 msgr2=0x7fe0000b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:22.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.325+0000 7fe01b546640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488] conn(0x7fe0000ae4a0 0x7fe0000b0940 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:22.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.325+0000 7fe01b546640 1 -- 172.21.15.33:0/1509543089 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0000a32d0 msgr2=0x7fe0000a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:22.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.325+0000 7fe01b546640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0000a32d0 0x7fe0000a5790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe010005d00 tx=0x7fe01000b410 comp rx=0 tx=0).stop 2024-09-16T00:16:22.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.325+0000 7fe01b546640 1 -- 172.21.15.33:0/1509543089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe014100de0 msgr2=0x7fe0140fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:22.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.325+0000 7fe01b546640 1 --2- 172.21.15.33:0/1509543089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe014100de0 0x7fe0140fcc40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe00400e9f0 tx=0x7fe00400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:16:22.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.325+0000 7fe019ae3640 1 -- 172.21.15.33:0/1509543089 reap_dead start 2024-09-16T00:16:22.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.325+0000 7fe01b546640 1 -- 172.21.15.33:0/1509543089 shutdown_connections 2024-09-16T00:16:22.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.325+0000 7fe01b546640 1 -- 172.21.15.33:0/1509543089 >> 172.21.15.33:0/1509543089 conn(0x7fe014077230 msgr2=0x7fe01410dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:22.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.326+0000 7fe01b546640 1 -- 172.21.15.33:0/1509543089 shutdown_connections 2024-09-16T00:16:22.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:22.326+0000 7fe01b546640 1 -- 172.21.15.33:0/1509543089 wait complete. 2024-09-16T00:16:22.465 INFO:teuthology.orchestra.run.smithi033.stdout:287762808864 2024-09-16T00:16:22.465 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-16T00:16:23.554 INFO:teuthology.orchestra.run.smithi033.stdout:133143986241 2024-09-16T00:16:23.555 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-16T00:16:23.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:23 smithi060 ceph-mon[35185]: pgmap v325: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 6 op/s; 5.1 KiB/s, 1 objects/s recovering 2024-09-16T00:16:23.738 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:23 smithi033 ceph-mon[31243]: pgmap v325: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 6 op/s; 5.1 KiB/s, 1 objects/s recovering 2024-09-16T00:16:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:23 smithi083 ceph-mon[35242]: pgmap v325: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 6 op/s; 5.1 KiB/s, 1 objects/s recovering 2024-09-16T00:16:23.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.913+0000 7fea17cf2640 1 -- 172.21.15.33:0/2622181695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea10101450 msgr2=0x7fea10105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:23.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.913+0000 7fea17cf2640 1 --2- 172.21.15.33:0/2622181695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea10101450 0x7fea10105920 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fea00009900 tx=0x7fea0002a470 comp rx=0 tx=0).stop 2024-09-16T00:16:23.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.913+0000 7fea17cf2640 1 -- 172.21.15.33:0/2622181695 shutdown_connections 2024-09-16T00:16:23.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.913+0000 7fea17cf2640 1 --2- 172.21.15.33:0/2622181695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea10105f40 0x7fea101063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:23.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.913+0000 7fea17cf2640 1 --2- 172.21.15.33:0/2622181695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea10101450 0x7fea10105920 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:23.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.913+0000 7fea17cf2640 1 --2- 172.21.15.33:0/2622181695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea10100b30 0x7fea10100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:23.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.913+0000 7fea17cf2640 1 -- 172.21.15.33:0/2622181695 >> 172.21.15.33:0/2622181695 conn(0x7fea100fc7a0 msgr2=0x7fea100febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:23.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.913+0000 7fea17cf2640 1 -- 172.21.15.33:0/2622181695 shutdown_connections 2024-09-16T00:16:23.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.913+0000 7fea17cf2640 1 -- 172.21.15.33:0/2622181695 wait complete. 2024-09-16T00:16:23.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.914+0000 7fea17cf2640 1 Processor -- start 2024-09-16T00:16:23.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.914+0000 7fea17cf2640 1 -- start start 2024-09-16T00:16:23.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.914+0000 7fea17cf2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea10100b30 0x7fea101a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.914+0000 7fea17cf2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea10101450 0x7fea101a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.914+0000 7fea17cf2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea10105f40 0x7fea1019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.914+0000 7fea17cf2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fea10118540 con 0x7fea10100b30 2024-09-16T00:16:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.914+0000 7fea17cf2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fea101183c0 con 0x7fea10105f40 2024-09-16T00:16:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.914+0000 7fea17cf2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fea101186c0 con 0x7fea10101450 2024-09-16T00:16:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.914+0000 7fea15a8e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea10100b30 0x7fea101a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.914+0000 7fea15a8e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea10100b30 0x7fea101a4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39824/0 (socket says 172.21.15.33:39824) 2024-09-16T00:16:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.914+0000 7fea15a8e640 1 -- 172.21.15.33:0/4280327265 learned_addr learned my addr 172.21.15.33:0/4280327265 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.914+0000 7fea1528d640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea10101450 0x7fea101a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.915+0000 7fea15a8e640 1 -- 172.21.15.33:0/4280327265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea10101450 msgr2=0x7fea101a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.915+0000 7fea1628f640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea10105f40 0x7fea1019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.915+0000 7fea15a8e640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea10101450 0x7fea101a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.915+0000 7fea15a8e640 1 -- 172.21.15.33:0/4280327265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea10105f40 msgr2=0x7fea1019e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.915+0000 7fea15a8e640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea10105f40 0x7fea1019e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.915+0000 7fea15a8e640 1 -- 172.21.15.33:0/4280327265 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1019f070 con 0x7fea10100b30 2024-09-16T00:16:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.915+0000 7fea15a8e640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea10100b30 0x7fea101a4780 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00d950 tx=0x7fea0c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.915+0000 7fea06ffd640 1 -- 172.21.15.33:0/4280327265 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c017030 con 0x7fea10100b30 2024-09-16T00:16:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.915+0000 7fea06ffd640 1 -- 172.21.15.33:0/4280327265 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea0c004020 con 0x7fea10100b30 2024-09-16T00:16:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.915+0000 7fea17cf2640 1 -- 172.21.15.33:0/4280327265 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1019f360 con 0x7fea10100b30 2024-09-16T00:16:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.915+0000 7fea06ffd640 1 -- 172.21.15.33:0/4280327265 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c0047a0 con 0x7fea10100b30 2024-09-16T00:16:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.915+0000 7fea17cf2640 1 -- 172.21.15.33:0/4280327265 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea101ab650 con 0x7fea10100b30 2024-09-16T00:16:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.916+0000 7fea17cf2640 1 -- 172.21.15.33:0/4280327265 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fea101100e0 con 0x7fea10100b30 2024-09-16T00:16:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.917+0000 7fea06ffd640 1 -- 172.21.15.33:0/4280327265 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fea0c0041d0 con 0x7fea10100b30 2024-09-16T00:16:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.918+0000 7fea06ffd640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9fc0a32d0 0x7fe9fc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.918+0000 7fea06ffd640 1 -- 172.21.15.33:0/4280327265 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fea0c0cfed0 con 0x7fea10100b30 2024-09-16T00:16:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.918+0000 7fea1528d640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9fc0a32d0 0x7fe9fc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.918+0000 7fea06ffd640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239] conn(0x7fe9fc0ae4a0 0x7fe9fc0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.918+0000 7fea06ffd640 1 -- 172.21.15.33:0/4280327265 --> [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9fc0b1010 con 0x7fe9fc0ae4a0 2024-09-16T00:16:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.918+0000 7fea06ffd640 1 -- 172.21.15.33:0/4280327265 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fea0c0d0320 con 0x7fea10100b30 2024-09-16T00:16:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.918+0000 7fea1628f640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239] conn(0x7fe9fc0ae4a0 0x7fe9fc0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:23.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.918+0000 7fea1528d640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9fc0a32d0 0x7fe9fc0a5790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fea00009230 tx=0x7fea000058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:23.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.919+0000 7fea1628f640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239] conn(0x7fe9fc0ae4a0 0x7fe9fc0b0940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:23.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.920+0000 7fea06ffd640 1 -- 172.21.15.33:0/4280327265 <== osd.1 v2:172.21.15.60:6800/1932822239 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe9fc0b1010 con 0x7fe9fc0ae4a0 2024-09-16T00:16:23.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.933+0000 7fea17cf2640 1 -- 172.21.15.33:0/4280327265 --> [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fea1006b070 con 0x7fe9fc0ae4a0 2024-09-16T00:16:23.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.934+0000 7fea06ffd640 1 -- 172.21.15.33:0/4280327265 <== osd.1 v2:172.21.15.60:6800/1932822239 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fea1006b070 con 0x7fe9fc0ae4a0 2024-09-16T00:16:23.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.934+0000 7fea17cf2640 1 -- 172.21.15.33:0/4280327265 >> [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239] conn(0x7fe9fc0ae4a0 msgr2=0x7fe9fc0b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:23.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.934+0000 7fea17cf2640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239] conn(0x7fe9fc0ae4a0 0x7fe9fc0b0940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:23.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.934+0000 7fea17cf2640 1 -- 172.21.15.33:0/4280327265 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9fc0a32d0 msgr2=0x7fe9fc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:23.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.934+0000 7fea17cf2640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9fc0a32d0 0x7fe9fc0a5790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fea00009230 tx=0x7fea000058d0 comp rx=0 tx=0).stop 2024-09-16T00:16:23.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.934+0000 7fea17cf2640 1 -- 172.21.15.33:0/4280327265 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea10100b30 msgr2=0x7fea101a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:23.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.934+0000 7fea17cf2640 1 --2- 172.21.15.33:0/4280327265 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea10100b30 0x7fea101a4780 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00d950 tx=0x7fea0c00de10 comp rx=0 tx=0).stop 2024-09-16T00:16:23.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.934+0000 7fea1628f640 1 -- 172.21.15.33:0/4280327265 reap_dead start 2024-09-16T00:16:23.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.934+0000 7fea17cf2640 1 -- 172.21.15.33:0/4280327265 shutdown_connections 2024-09-16T00:16:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.934+0000 7fea17cf2640 1 -- 172.21.15.33:0/4280327265 >> 172.21.15.33:0/4280327265 conn(0x7fea100fc7a0 msgr2=0x7fea1010f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.935+0000 7fea17cf2640 1 -- 172.21.15.33:0/4280327265 shutdown_connections 2024-09-16T00:16:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:23.935+0000 7fea17cf2640 1 -- 172.21.15.33:0/4280327265 wait complete. 2024-09-16T00:16:24.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.377+0000 7f1613ec7640 1 -- 172.21.15.33:0/2084854883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f160c105f40 msgr2=0x7f160c1063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:24.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.377+0000 7f1613ec7640 1 --2- 172.21.15.33:0/2084854883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f160c105f40 0x7f160c1063d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1604009ca0 tx=0x7f160402a940 comp rx=0 tx=0).stop 2024-09-16T00:16:24.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.377+0000 7f1613ec7640 1 -- 172.21.15.33:0/2084854883 shutdown_connections 2024-09-16T00:16:24.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.377+0000 7f1613ec7640 1 --2- 172.21.15.33:0/2084854883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f160c105f40 0x7f160c1063d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:24.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.377+0000 7f1613ec7640 1 --2- 172.21.15.33:0/2084854883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f160c101450 0x7f160c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:24.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.377+0000 7f1613ec7640 1 --2- 172.21.15.33:0/2084854883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f160c100b30 0x7f160c100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:24.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.377+0000 7f1613ec7640 1 -- 172.21.15.33:0/2084854883 >> 172.21.15.33:0/2084854883 conn(0x7f160c0fc7a0 msgr2=0x7f160c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:24.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.377+0000 7f1613ec7640 1 -- 172.21.15.33:0/2084854883 shutdown_connections 2024-09-16T00:16:24.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.377+0000 7f1613ec7640 1 -- 172.21.15.33:0/2084854883 wait complete. 2024-09-16T00:16:24.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.378+0000 7f1613ec7640 1 Processor -- start 2024-09-16T00:16:24.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.378+0000 7f1613ec7640 1 -- start start 2024-09-16T00:16:24.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.378+0000 7f1613ec7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f160c100b30 0x7f160c1a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:24.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.378+0000 7f1613ec7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f160c101450 0x7f160c1a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:24.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.378+0000 7f1613ec7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f160c105f40 0x7f160c19e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:24.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.378+0000 7f1613ec7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f160c118560 con 0x7f160c105f40 2024-09-16T00:16:24.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.378+0000 7f1613ec7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f160c1183e0 con 0x7f160c101450 2024-09-16T00:16:24.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.378+0000 7f1613ec7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f160c1186e0 con 0x7f160c100b30 2024-09-16T00:16:24.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.378+0000 7f1612464640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f160c105f40 0x7f160c19e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:24.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.378+0000 7f1612464640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f160c105f40 0x7f160c19e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39828/0 (socket says 172.21.15.33:39828) 2024-09-16T00:16:24.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.378+0000 7f1612464640 1 -- 172.21.15.33:0/2764766575 learned_addr learned my addr 172.21.15.33:0/2764766575 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:24.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1611c63640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f160c100b30 0x7f160c1a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:24.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1611462640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f160c101450 0x7f160c1a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:24.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1612464640 1 -- 172.21.15.33:0/2764766575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f160c100b30 msgr2=0x7f160c1a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:24.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1612464640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f160c100b30 0x7f160c1a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:24.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1612464640 1 -- 172.21.15.33:0/2764766575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f160c101450 msgr2=0x7f160c1a4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:24.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1612464640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f160c101450 0x7f160c1a4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:24.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1612464640 1 -- 172.21.15.33:0/2764766575 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f160c19efe0 con 0x7f160c105f40 2024-09-16T00:16:24.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1612464640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f160c105f40 0x7f160c19e770 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1604006010 tx=0x7f160402f700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:24.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1602ffd640 1 -- 172.21.15.33:0/2764766575 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1604034040 con 0x7f160c105f40 2024-09-16T00:16:24.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1602ffd640 1 -- 172.21.15.33:0/2764766575 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1604003e50 con 0x7f160c105f40 2024-09-16T00:16:24.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1613ec7640 1 -- 172.21.15.33:0/2764766575 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f160c19f270 con 0x7f160c105f40 2024-09-16T00:16:24.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1602ffd640 1 -- 172.21.15.33:0/2764766575 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1604004800 con 0x7f160c105f40 2024-09-16T00:16:24.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.379+0000 7f1613ec7640 1 -- 172.21.15.33:0/2764766575 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f160c1ab450 con 0x7f160c105f40 2024-09-16T00:16:24.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.380+0000 7f1613ec7640 1 -- 172.21.15.33:0/2764766575 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f160c105270 con 0x7f160c105f40 2024-09-16T00:16:24.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.381+0000 7f1602ffd640 1 -- 172.21.15.33:0/2764766575 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1604037020 con 0x7f160c105f40 2024-09-16T00:16:24.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.382+0000 7f1602ffd640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15f80a32d0 0x7f15f80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:24.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.382+0000 7f1611c63640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15f80a32d0 0x7f15f80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:24.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.382+0000 7f1602ffd640 1 -- 172.21.15.33:0/2764766575 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f16040f0e60 con 0x7f160c105f40 2024-09-16T00:16:24.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.382+0000 7f1602ffd640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155] conn(0x7f15f80ae4a0 0x7f15f80b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:24.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.382+0000 7f1602ffd640 1 -- 172.21.15.33:0/2764766575 --> [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15f80b1010 con 0x7f15f80ae4a0 2024-09-16T00:16:24.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.382+0000 7f1602ffd640 1 -- 172.21.15.33:0/2764766575 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f16040f12b0 con 0x7f160c105f40 2024-09-16T00:16:24.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.383+0000 7f1611c63640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15f80a32d0 0x7f15f80a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1608005d00 tx=0x7f160800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:24.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.383+0000 7f1611462640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155] conn(0x7f15f80ae4a0 0x7f15f80b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:24.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.383+0000 7f1611462640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155] conn(0x7f15f80ae4a0 0x7f15f80b0940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:24.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.384+0000 7f1602ffd640 1 -- 172.21.15.33:0/2764766575 <== osd.6 v2:172.21.15.33:6818/2633676155 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15f80b1010 con 0x7f15f80ae4a0 2024-09-16T00:16:24.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.397+0000 7f1613ec7640 1 -- 172.21.15.33:0/2764766575 --> [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f160c06b070 con 0x7f15f80ae4a0 2024-09-16T00:16:24.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.397+0000 7f1602ffd640 1 -- 172.21.15.33:0/2764766575 <== osd.6 v2:172.21.15.33:6818/2633676155 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f160c06b070 con 0x7f15f80ae4a0 2024-09-16T00:16:24.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.398+0000 7f1613ec7640 1 -- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155] conn(0x7f15f80ae4a0 msgr2=0x7f15f80b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:24.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.398+0000 7f1613ec7640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155] conn(0x7f15f80ae4a0 0x7f15f80b0940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:24.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.398+0000 7f1613ec7640 1 -- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15f80a32d0 msgr2=0x7f15f80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:24.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.398+0000 7f1613ec7640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15f80a32d0 0x7f15f80a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1608005d00 tx=0x7f160800b410 comp rx=0 tx=0).stop 2024-09-16T00:16:24.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.398+0000 7f1613ec7640 1 -- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f160c105f40 msgr2=0x7f160c19e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:24.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.398+0000 7f1613ec7640 1 --2- 172.21.15.33:0/2764766575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f160c105f40 0x7f160c19e770 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1604006010 tx=0x7f160402f700 comp rx=0 tx=0).stop 2024-09-16T00:16:24.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.398+0000 7f1612464640 1 -- 172.21.15.33:0/2764766575 reap_dead start 2024-09-16T00:16:24.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.398+0000 7f1613ec7640 1 -- 172.21.15.33:0/2764766575 shutdown_connections 2024-09-16T00:16:24.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.398+0000 7f1613ec7640 1 -- 172.21.15.33:0/2764766575 >> 172.21.15.33:0/2764766575 conn(0x7f160c0fc7a0 msgr2=0x7f160c1038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:24.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.398+0000 7f1613ec7640 1 -- 172.21.15.33:0/2764766575 shutdown_connections 2024-09-16T00:16:24.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:24.398+0000 7f1613ec7640 1 -- 172.21.15.33:0/2764766575 wait complete. 2024-09-16T00:16:25.102 INFO:teuthology.orchestra.run.smithi033.stdout:180388626487 2024-09-16T00:16:25.102 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-16T00:16:25.673 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:25 smithi033 ceph-mon[31243]: pgmap v326: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s; 5.0 KiB/s, 0 objects/s recovering 2024-09-16T00:16:25.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:25 smithi060 ceph-mon[35185]: pgmap v326: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s; 5.0 KiB/s, 0 objects/s recovering 2024-09-16T00:16:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:25 smithi083 ceph-mon[35242]: pgmap v326: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s; 5.0 KiB/s, 0 objects/s recovering 2024-09-16T00:16:26.087 INFO:teuthology.orchestra.run.smithi033.stdout:51539607633 2024-09-16T00:16:26.088 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-16T00:16:26.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.250+0000 7f21fe2f1640 1 -- 172.21.15.33:0/729365027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f8102db0 msgr2=0x7f21f8103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:26.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.250+0000 7f21fe2f1640 1 --2- 172.21.15.33:0/729365027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f8102db0 0x7f21f8103190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f21e8009930 tx=0x7f21e802a420 comp rx=0 tx=0).stop 2024-09-16T00:16:26.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.250+0000 7f21fe2f1640 1 -- 172.21.15.33:0/729365027 shutdown_connections 2024-09-16T00:16:26.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.250+0000 7f21fe2f1640 1 --2- 172.21.15.33:0/729365027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21f81082e0 0x7f21f810ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:26.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.250+0000 7f21fe2f1640 1 --2- 172.21.15.33:0/729365027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21f8103760 0x7f21f8107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:26.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.250+0000 7f21fe2f1640 1 --2- 172.21.15.33:0/729365027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f8102db0 0x7f21f8103190 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:26.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.250+0000 7f21fe2f1640 1 -- 172.21.15.33:0/729365027 >> 172.21.15.33:0/729365027 conn(0x7f21f80fc7d0 msgr2=0x7f21f80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:26.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.250+0000 7f21fe2f1640 1 -- 172.21.15.33:0/729365027 shutdown_connections 2024-09-16T00:16:26.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.250+0000 7f21fe2f1640 1 -- 172.21.15.33:0/729365027 wait complete. 2024-09-16T00:16:26.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.250+0000 7f21fe2f1640 1 Processor -- start 2024-09-16T00:16:26.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.250+0000 7f21fe2f1640 1 -- start start 2024-09-16T00:16:26.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21fe2f1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21f8102db0 0x7f21f8100a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:26.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21fe2f1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f8103760 0x7f21f8100f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:26.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21fe2f1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21f81082e0 0x7f21f8101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:26.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21fe2f1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21f8112090 con 0x7f21f8103760 2024-09-16T00:16:26.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21fe2f1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f21f8111f10 con 0x7f21f81082e0 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21fe2f1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21f8112210 con 0x7f21f8102db0 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21f77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f8103760 0x7f21f8100f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21f77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f8103760 0x7f21f8100f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39848/0 (socket says 172.21.15.33:39848) 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21f7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21f8102db0 0x7f21f8100a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21f77fe640 1 -- 172.21.15.33:0/3630304307 learned_addr learned my addr 172.21.15.33:0/3630304307 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21fc88e640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21f81082e0 0x7f21f8101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21f77fe640 1 -- 172.21.15.33:0/3630304307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21f8102db0 msgr2=0x7f21f8100a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21f77fe640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21f8102db0 0x7f21f8100a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21f77fe640 1 -- 172.21.15.33:0/3630304307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21f81082e0 msgr2=0x7f21f8101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21f77fe640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21f81082e0 0x7f21f8101490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.251+0000 7f21f77fe640 1 -- 172.21.15.33:0/3630304307 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f81a6a80 con 0x7f21f8103760 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.252+0000 7f21f77fe640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f8103760 0x7f21f8100f50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f21ec00e9f0 tx=0x7f21ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:26.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.252+0000 7f21f57fa640 1 -- 172.21.15.33:0/3630304307 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec00ccc0 con 0x7f21f8103760 2024-09-16T00:16:26.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.252+0000 7f21f57fa640 1 -- 172.21.15.33:0/3630304307 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21ec010910 con 0x7f21f8103760 2024-09-16T00:16:26.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.252+0000 7f21f57fa640 1 -- 172.21.15.33:0/3630304307 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec018920 con 0x7f21f8103760 2024-09-16T00:16:26.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.252+0000 7f21fe2f1640 1 -- 172.21.15.33:0/3630304307 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f81a6d70 con 0x7f21f8103760 2024-09-16T00:16:26.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.252+0000 7f21fe2f1640 1 -- 172.21.15.33:0/3630304307 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f81a7270 con 0x7f21f8103760 2024-09-16T00:16:26.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.253+0000 7f21fe2f1640 1 -- 172.21.15.33:0/3630304307 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f21f8062900 con 0x7f21f8103760 2024-09-16T00:16:26.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.254+0000 7f21f57fa640 1 -- 172.21.15.33:0/3630304307 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f21ec010ac0 con 0x7f21f8103760 2024-09-16T00:16:26.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.255+0000 7f21f57fa640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21e00a3350 0x7f21e00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:26.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.255+0000 7f21f7fff640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21e00a3350 0x7f21e00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.255+0000 7f21f57fa640 1 -- 172.21.15.33:0/3630304307 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f21ec0d0670 con 0x7f21f8103760 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.255+0000 7f21f57fa640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095] conn(0x7f21e00ae520 0x7f21e00b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.255+0000 7f21f57fa640 1 -- 172.21.15.33:0/3630304307 --> [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f21e00b1090 con 0x7f21e00ae520 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.255+0000 7f21f57fa640 1 -- 172.21.15.33:0/3630304307 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f21ec0d0ac0 con 0x7f21f8103760 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.255+0000 7f21fc88e640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095] conn(0x7f21e00ae520 0x7f21e00b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.255+0000 7f21f7fff640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21e00a3350 0x7f21e00a5810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f21e8009900 tx=0x7f21e80058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.256+0000 7f21fc88e640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095] conn(0x7f21e00ae520 0x7f21e00b09c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.257+0000 7f21f57fa640 1 -- 172.21.15.33:0/3630304307 <== osd.5 v2:172.21.15.83:6808/2896581095 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f21e00b1090 con 0x7f21e00ae520 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.270+0000 7f21fe2f1640 1 -- 172.21.15.33:0/3630304307 --> [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f21f8062d30 con 0x7f21e00ae520 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.271+0000 7f21f57fa640 1 -- 172.21.15.33:0/3630304307 <== osd.5 v2:172.21.15.83:6808/2896581095 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f21f8062d30 con 0x7f21e00ae520 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.271+0000 7f21fe2f1640 1 -- 172.21.15.33:0/3630304307 >> [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095] conn(0x7f21e00ae520 msgr2=0x7f21e00b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.271+0000 7f21fe2f1640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095] conn(0x7f21e00ae520 0x7f21e00b09c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.272+0000 7f21fe2f1640 1 -- 172.21.15.33:0/3630304307 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21e00a3350 msgr2=0x7f21e00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:26.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.272+0000 7f21fe2f1640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21e00a3350 0x7f21e00a5810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f21e8009900 tx=0x7f21e80058d0 comp rx=0 tx=0).stop 2024-09-16T00:16:26.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.272+0000 7f21fe2f1640 1 -- 172.21.15.33:0/3630304307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f8103760 msgr2=0x7f21f8100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:26.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.272+0000 7f21fe2f1640 1 --2- 172.21.15.33:0/3630304307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f8103760 0x7f21f8100f50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f21ec00e9f0 tx=0x7f21ec00eeb0 comp rx=0 tx=0).stop 2024-09-16T00:16:26.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.272+0000 7f21fc88e640 1 -- 172.21.15.33:0/3630304307 reap_dead start 2024-09-16T00:16:26.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.272+0000 7f21fe2f1640 1 -- 172.21.15.33:0/3630304307 shutdown_connections 2024-09-16T00:16:26.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.272+0000 7f21fe2f1640 1 -- 172.21.15.33:0/3630304307 >> 172.21.15.33:0/3630304307 conn(0x7f21f80fc7d0 msgr2=0x7f21f8105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:26.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.272+0000 7f21fe2f1640 1 -- 172.21.15.33:0/3630304307 shutdown_connections 2024-09-16T00:16:26.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:26.272+0000 7f21fe2f1640 1 -- 172.21.15.33:0/3630304307 wait complete. 2024-09-16T00:16:27.068 INFO:teuthology.orchestra.run.smithi033.stdout:236223201323 2024-09-16T00:16:27.068 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-16T00:16:27.571 INFO:teuthology.orchestra.run.smithi033.stdout:317827579933 2024-09-16T00:16:27.571 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-16T00:16:27.796 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:27 smithi033 ceph-mon[31243]: pgmap v327: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 48 B/s, 0 objects/s recovering 2024-09-16T00:16:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.881+0000 7f130804b640 1 -- 172.21.15.33:0/1367106487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13001036e0 msgr2=0x7f1300107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:27.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.881+0000 7f130804b640 1 --2- 172.21.15.33:0/1367106487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13001036e0 0x7f1300107b90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f12f4009930 tx=0x7f12f402a420 comp rx=0 tx=0).stop 2024-09-16T00:16:27.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.881+0000 7f130804b640 1 -- 172.21.15.33:0/1367106487 shutdown_connections 2024-09-16T00:16:27.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.881+0000 7f130804b640 1 --2- 172.21.15.33:0/1367106487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13001082a0 0x7f130010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:27.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.881+0000 7f130804b640 1 --2- 172.21.15.33:0/1367106487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13001036e0 0x7f1300107b90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:27.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.881+0000 7f130804b640 1 --2- 172.21.15.33:0/1367106487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1300102d30 0x7f1300103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:27.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.881+0000 7f130804b640 1 -- 172.21.15.33:0/1367106487 >> 172.21.15.33:0/1367106487 conn(0x7f13000fc7d0 msgr2=0x7f13000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:27.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.881+0000 7f130804b640 1 -- 172.21.15.33:0/1367106487 shutdown_connections 2024-09-16T00:16:27.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.881+0000 7f130804b640 1 -- 172.21.15.33:0/1367106487 wait complete. 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f130804b640 1 Processor -- start 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f130804b640 1 -- start start 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f130804b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1300102d30 0x7f1300198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f130804b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13001036e0 0x7f1300198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f130804b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13001082a0 0x7f130019ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f130804b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1300112060 con 0x7f13001082a0 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f130804b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1300111ee0 con 0x7f13001036e0 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f130804b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13001121e0 con 0x7f1300102d30 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f13065e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13001082a0 0x7f130019ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f13065e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13001082a0 0x7f130019ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39858/0 (socket says 172.21.15.33:39858) 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f13065e8640 1 -- 172.21.15.33:0/634190038 learned_addr learned my addr 172.21.15.33:0/634190038 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f1305de7640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1300102d30 0x7f1300198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f13055e6640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13001036e0 0x7f1300198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:27.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f13065e8640 1 -- 172.21.15.33:0/634190038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1300102d30 msgr2=0x7f1300198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:27.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f13065e8640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1300102d30 0x7f1300198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:27.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f13065e8640 1 -- 172.21.15.33:0/634190038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13001036e0 msgr2=0x7f1300198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:27.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f13065e8640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13001036e0 0x7f1300198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:27.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.882+0000 7f13065e8640 1 -- 172.21.15.33:0/634190038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130019d5b0 con 0x7f13001082a0 2024-09-16T00:16:27.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.883+0000 7f13065e8640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13001082a0 0x7f130019ced0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f12f800c970 tx=0x7f12f800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:27.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.883+0000 7f12feffd640 1 -- 172.21.15.33:0/634190038 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f8007bb0 con 0x7f13001082a0 2024-09-16T00:16:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.883+0000 7f12feffd640 1 -- 172.21.15.33:0/634190038 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12f8007d50 con 0x7f13001082a0 2024-09-16T00:16:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.883+0000 7f130804b640 1 -- 172.21.15.33:0/634190038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f130019d8a0 con 0x7f13001082a0 2024-09-16T00:16:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.883+0000 7f12feffd640 1 -- 172.21.15.33:0/634190038 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f800f450 con 0x7f13001082a0 2024-09-16T00:16:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.883+0000 7f130804b640 1 -- 172.21.15.33:0/634190038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13001a5180 con 0x7f13001082a0 2024-09-16T00:16:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.884+0000 7f130804b640 1 -- 172.21.15.33:0/634190038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1300062940 con 0x7f13001082a0 2024-09-16T00:16:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.885+0000 7f12feffd640 1 -- 172.21.15.33:0/634190038 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f12f8004f00 con 0x7f13001082a0 2024-09-16T00:16:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.886+0000 7f12feffd640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12e80a3350 0x7f12e80a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.886+0000 7f12feffd640 1 -- 172.21.15.33:0/634190038 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f12f80cfe90 con 0x7f13001082a0 2024-09-16T00:16:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.886+0000 7f1305de7640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12e80a3350 0x7f12e80a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.886+0000 7f12feffd640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241] conn(0x7f12e80ae520 0x7f12e80b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.886+0000 7f12feffd640 1 -- 172.21.15.33:0/634190038 --> [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12e80b1090 con 0x7f12e80ae520 2024-09-16T00:16:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.886+0000 7f12feffd640 1 -- 172.21.15.33:0/634190038 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f12f80d0240 con 0x7f13001082a0 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.886+0000 7f13055e6640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241] conn(0x7f12e80ae520 0x7f12e80b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.886+0000 7f1305de7640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12e80a3350 0x7f12e80a5810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f12f0005c40 tx=0x7f12f000a590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.887+0000 7f13055e6640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241] conn(0x7f12e80ae520 0x7f12e80b09c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.887+0000 7f12feffd640 1 -- 172.21.15.33:0/634190038 <== osd.9 v2:172.21.15.33:6826/2906942241 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f12e80b1090 con 0x7f12e80ae520 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.901+0000 7f130804b640 1 -- 172.21.15.33:0/634190038 --> [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1300062dc0 con 0x7f12e80ae520 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.901+0000 7f12feffd640 1 -- 172.21.15.33:0/634190038 <== osd.9 v2:172.21.15.33:6826/2906942241 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1300062dc0 con 0x7f12e80ae520 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.902+0000 7f130804b640 1 -- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241] conn(0x7f12e80ae520 msgr2=0x7f12e80b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.902+0000 7f130804b640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241] conn(0x7f12e80ae520 0x7f12e80b09c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.902+0000 7f130804b640 1 -- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12e80a3350 msgr2=0x7f12e80a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.902+0000 7f130804b640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12e80a3350 0x7f12e80a5810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f12f0005c40 tx=0x7f12f000a590 comp rx=0 tx=0).stop 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.902+0000 7f130804b640 1 -- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13001082a0 msgr2=0x7f130019ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.902+0000 7f130804b640 1 --2- 172.21.15.33:0/634190038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13001082a0 0x7f130019ced0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f12f800c970 tx=0x7f12f800ce30 comp rx=0 tx=0).stop 2024-09-16T00:16:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.902+0000 7f13065e8640 1 -- 172.21.15.33:0/634190038 reap_dead start 2024-09-16T00:16:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.902+0000 7f130804b640 1 -- 172.21.15.33:0/634190038 shutdown_connections 2024-09-16T00:16:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.902+0000 7f130804b640 1 -- 172.21.15.33:0/634190038 >> 172.21.15.33:0/634190038 conn(0x7f13000fc7d0 msgr2=0x7f1300105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.902+0000 7f130804b640 1 -- 172.21.15.33:0/634190038 shutdown_connections 2024-09-16T00:16:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:27.902+0000 7f130804b640 1 -- 172.21.15.33:0/634190038 wait complete. 2024-09-16T00:16:27.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:27 smithi083 ceph-mon[35242]: pgmap v327: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 48 B/s, 0 objects/s recovering 2024-09-16T00:16:27.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:27 smithi060 ceph-mon[35185]: pgmap v327: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 48 B/s, 0 objects/s recovering 2024-09-16T00:16:28.601 INFO:teuthology.orchestra.run.smithi033.stdout:266287972392 2024-09-16T00:16:28.601 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-16T00:16:29.319 INFO:teuthology.orchestra.run.smithi033.stdout:150323855421 2024-09-16T00:16:29.319 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-16T00:16:29.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.521+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/2878675247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b38103660 msgr2=0x7f0b3810bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:29.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.521+0000 7f0b3c9ff640 1 --2- 172.21.15.33:0/2878675247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b38103660 0x7f0b3810bb50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009900 tx=0x7f0b2c02a480 comp rx=0 tx=0).stop 2024-09-16T00:16:29.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.522+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/2878675247 shutdown_connections 2024-09-16T00:16:29.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.522+0000 7f0b3c9ff640 1 --2- 172.21.15.33:0/2878675247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b3810c170 0x7f0b3810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:29.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.522+0000 7f0b3c9ff640 1 --2- 172.21.15.33:0/2878675247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b38103660 0x7f0b3810bb50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:29.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.522+0000 7f0b3c9ff640 1 --2- 172.21.15.33:0/2878675247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b38102d40 0x7f0b38103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:29.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.522+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/2878675247 >> 172.21.15.33:0/2878675247 conn(0x7f0b380fc7a0 msgr2=0x7f0b380febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:29.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.522+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/2878675247 shutdown_connections 2024-09-16T00:16:29.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.522+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/2878675247 wait complete. 2024-09-16T00:16:29.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.522+0000 7f0b3c9ff640 1 Processor -- start 2024-09-16T00:16:29.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.522+0000 7f0b3c9ff640 1 -- start start 2024-09-16T00:16:29.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3c9ff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b38102d40 0x7f0b381983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:29.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3c9ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b38103660 0x7f0b38198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:29.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3c9ff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3810c170 0x7f0b3819ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:29.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3c9ff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0b38111f70 con 0x7f0b38102d40 2024-09-16T00:16:29.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3c9ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0b38111df0 con 0x7f0b38103660 2024-09-16T00:16:29.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3c9ff640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0b381120f0 con 0x7f0b3810c170 2024-09-16T00:16:29.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b38102d40 0x7f0b381983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:29.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b38102d40 0x7f0b381983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39880/0 (socket says 172.21.15.33:39880) 2024-09-16T00:16:29.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3659c640 1 -- 172.21.15.33:0/890382601 learned_addr learned my addr 172.21.15.33:0/890382601 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:29.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b35d9b640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b38103660 0x7f0b38198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:29.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b36d9d640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3810c170 0x7f0b3819ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:29.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3659c640 1 -- 172.21.15.33:0/890382601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3810c170 msgr2=0x7f0b3819ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:29.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3659c640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3810c170 0x7f0b3819ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:29.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3659c640 1 -- 172.21.15.33:0/890382601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b38103660 msgr2=0x7f0b38198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:29.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3659c640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b38103660 0x7f0b38198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:29.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3659c640 1 -- 172.21.15.33:0/890382601 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3819d390 con 0x7f0b38102d40 2024-09-16T00:16:29.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b35d9b640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b38103660 0x7f0b38198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:16:29.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.523+0000 7f0b3659c640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b38102d40 0x7f0b381983e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800c920 tx=0x7f0b2800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:29.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.524+0000 7f0b277fe640 1 -- 172.21.15.33:0/890382601 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28007c70 con 0x7f0b38102d40 2024-09-16T00:16:29.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.524+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/890382601 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b3819d680 con 0x7f0b38102d40 2024-09-16T00:16:29.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.524+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/890382601 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b381a4f60 con 0x7f0b38102d40 2024-09-16T00:16:29.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.524+0000 7f0b277fe640 1 -- 172.21.15.33:0/890382601 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b28007e10 con 0x7f0b38102d40 2024-09-16T00:16:29.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.524+0000 7f0b277fe640 1 -- 172.21.15.33:0/890382601 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2800f450 con 0x7f0b38102d40 2024-09-16T00:16:29.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.525+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/890382601 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0b18001030 con 0x7f0b38102d40 2024-09-16T00:16:29.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.527+0000 7f0b277fe640 1 -- 172.21.15.33:0/890382601 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0b280046c0 con 0x7f0b38102d40 2024-09-16T00:16:29.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.528+0000 7f0b277fe640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b200a3350 0x7f0b200a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:29.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.528+0000 7f0b277fe640 1 -- 172.21.15.33:0/890382601 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f0b280d0330 con 0x7f0b38102d40 2024-09-16T00:16:29.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.528+0000 7f0b35d9b640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b200a3350 0x7f0b200a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:29.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.528+0000 7f0b277fe640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043] conn(0x7f0b200ae520 0x7f0b200b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:29.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.528+0000 7f0b277fe640 1 -- 172.21.15.33:0/890382601 --> [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0b2801fe50 con 0x7f0b200ae520 2024-09-16T00:16:29.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.528+0000 7f0b277fe640 1 -- 172.21.15.33:0/890382601 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f0b280d06e0 con 0x7f0b38102d40 2024-09-16T00:16:29.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.528+0000 7f0b35d9b640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b200a3350 0x7f0b200a5810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0b3810d5b0 tx=0x7f0b2c005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:29.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.528+0000 7f0b36d9d640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043] conn(0x7f0b200ae520 0x7f0b200b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:29.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.529+0000 7f0b36d9d640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043] conn(0x7f0b200ae520 0x7f0b200b09c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:29.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.530+0000 7f0b277fe640 1 -- 172.21.15.33:0/890382601 <== osd.2 v2:172.21.15.83:6800/409428043 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0b2801fe50 con 0x7f0b200ae520 2024-09-16T00:16:29.547 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:29 smithi033 ceph-mon[31243]: pgmap v328: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T00:16:29.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.545+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/890382601 --> [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0b18002d50 con 0x7f0b200ae520 2024-09-16T00:16:29.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.545+0000 7f0b277fe640 1 -- 172.21.15.33:0/890382601 <== osd.2 v2:172.21.15.83:6800/409428043 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0b18002d50 con 0x7f0b200ae520 2024-09-16T00:16:29.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.546+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/890382601 >> [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043] conn(0x7f0b200ae520 msgr2=0x7f0b200b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:29.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.546+0000 7f0b3c9ff640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043] conn(0x7f0b200ae520 0x7f0b200b09c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:29.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.546+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/890382601 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b200a3350 msgr2=0x7f0b200a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:29.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.546+0000 7f0b3c9ff640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b200a3350 0x7f0b200a5810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0b3810d5b0 tx=0x7f0b2c005a20 comp rx=0 tx=0).stop 2024-09-16T00:16:29.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.546+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/890382601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b38102d40 msgr2=0x7f0b381983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:29.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.546+0000 7f0b3c9ff640 1 --2- 172.21.15.33:0/890382601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b38102d40 0x7f0b381983e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800c920 tx=0x7f0b2800cde0 comp rx=0 tx=0).stop 2024-09-16T00:16:29.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.546+0000 7f0b36d9d640 1 -- 172.21.15.33:0/890382601 reap_dead start 2024-09-16T00:16:29.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.547+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/890382601 shutdown_connections 2024-09-16T00:16:29.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.547+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/890382601 >> 172.21.15.33:0/890382601 conn(0x7f0b380fc7a0 msgr2=0x7f0b380fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:29.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.547+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/890382601 shutdown_connections 2024-09-16T00:16:29.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:29.547+0000 7f0b3c9ff640 1 -- 172.21.15.33:0/890382601 wait complete. 2024-09-16T00:16:29.557 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:29.559 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:29.566 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:29.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:29 smithi083 ceph-mon[35242]: pgmap v328: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T00:16:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:29 smithi060 ceph-mon[35185]: pgmap v328: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T00:16:30.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.383+0000 7f5227752640 1 -- 172.21.15.33:0/1952576983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5220100e20 msgr2=0x7f5220101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:30.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.383+0000 7f5227752640 1 --2- 172.21.15.33:0/1952576983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5220100e20 0x7f5220101200 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f521c009900 tx=0x7f521c02a480 comp rx=0 tx=0).stop 2024-09-16T00:16:30.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.383+0000 7f5227752640 1 -- 172.21.15.33:0/1952576983 shutdown_connections 2024-09-16T00:16:30.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.383+0000 7f5227752640 1 --2- 172.21.15.33:0/1952576983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5220102ef0 0x7f522010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:30.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.383+0000 7f5227752640 1 --2- 172.21.15.33:0/1952576983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5220102570 0x7f52201029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:30.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.383+0000 7f5227752640 1 --2- 172.21.15.33:0/1952576983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5220100e20 0x7f5220101200 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:30.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.383+0000 7f5227752640 1 -- 172.21.15.33:0/1952576983 >> 172.21.15.33:0/1952576983 conn(0x7f52200fc7d0 msgr2=0x7f52200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:30.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.383+0000 7f5227752640 1 -- 172.21.15.33:0/1952576983 shutdown_connections 2024-09-16T00:16:30.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.383+0000 7f5227752640 1 -- 172.21.15.33:0/1952576983 wait complete. 2024-09-16T00:16:30.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.384+0000 7f5227752640 1 Processor -- start 2024-09-16T00:16:30.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.384+0000 7f5227752640 1 -- start start 2024-09-16T00:16:30.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.384+0000 7f5227752640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5220100e20 0x7f5220197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:30.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.384+0000 7f5227752640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5220102570 0x7f5220197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:30.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.384+0000 7f5227752640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5220102ef0 0x7f5220197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:30.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.384+0000 7f5227752640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5220110fa0 con 0x7f5220102ef0 2024-09-16T00:16:30.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.384+0000 7f5227752640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5220110e20 con 0x7f5220100e20 2024-09-16T00:16:30.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.384+0000 7f5227752640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5220111120 con 0x7f5220102570 2024-09-16T00:16:30.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.384+0000 7f5225cef640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5220102ef0 0x7f5220197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:30.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.384+0000 7f5225cef640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5220102ef0 0x7f5220197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51886/0 (socket says 172.21.15.33:51886) 2024-09-16T00:16:30.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.384+0000 7f5225cef640 1 -- 172.21.15.33:0/2182361115 learned_addr learned my addr 172.21.15.33:0/2182361115 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:30.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.384+0000 7f52254ee640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5220100e20 0x7f5220197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:30.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.385+0000 7f5224ced640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5220102570 0x7f5220197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:30.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.385+0000 7f5225cef640 1 -- 172.21.15.33:0/2182361115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5220102570 msgr2=0x7f5220197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:30.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.385+0000 7f5225cef640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5220102570 0x7f5220197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:30.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.385+0000 7f5225cef640 1 -- 172.21.15.33:0/2182361115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5220100e20 msgr2=0x7f5220197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:30.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.385+0000 7f5225cef640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5220100e20 0x7f5220197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.385+0000 7f5225cef640 1 -- 172.21.15.33:0/2182361115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52201a6990 con 0x7f5220102ef0 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.385+0000 7f5225cef640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5220102ef0 0x7f5220197ba0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f521800b970 tx=0x7f521800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.385+0000 7f52167fc640 1 -- 172.21.15.33:0/2182361115 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218019070 con 0x7f5220102ef0 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.385+0000 7f52167fc640 1 -- 172.21.15.33:0/2182361115 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f521800cb40 con 0x7f5220102ef0 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.385+0000 7f52167fc640 1 -- 172.21.15.33:0/2182361115 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218007650 con 0x7f5220102ef0 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.385+0000 7f5227752640 1 -- 172.21.15.33:0/2182361115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52201a6c80 con 0x7f5220102ef0 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.385+0000 7f5227752640 1 -- 172.21.15.33:0/2182361115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52201a71f0 con 0x7f5220102ef0 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.386+0000 7f5227752640 1 -- 172.21.15.33:0/2182361115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5220101200 con 0x7f5220102ef0 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.387+0000 7f52167fc640 1 -- 172.21.15.33:0/2182361115 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f521801c070 con 0x7f5220102ef0 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.388+0000 7f52167fc640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f520c0a32d0 0x7f520c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.388+0000 7f52254ee640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f520c0a32d0 0x7f520c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.388+0000 7f52167fc640 1 -- 172.21.15.33:0/2182361115 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f52180cf2d0 con 0x7f5220102ef0 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.389+0000 7f52254ee640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f520c0a32d0 0x7f520c0a5790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f521c000c00 tx=0x7f521c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:30.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.392+0000 7f52167fc640 1 -- 172.21.15.33:0/2182361115 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f521809b5f0 con 0x7f5220102ef0 2024-09-16T00:16:30.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.575+0000 7f5227752640 1 -- 172.21.15.33:0/2182361115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f52201a7480 con 0x7f5220102ef0 2024-09-16T00:16:30.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.575+0000 7f52167fc640 1 -- 172.21.15.33:0/2182361115 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f521809f8f0 con 0x7f5220102ef0 2024-09-16T00:16:30.576 INFO:teuthology.orchestra.run.smithi033.stdout:34359738455 2024-09-16T00:16:30.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.577+0000 7f5227752640 1 -- 172.21.15.33:0/2182361115 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f520c0a32d0 msgr2=0x7f520c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:30.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.577+0000 7f5227752640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f520c0a32d0 0x7f520c0a5790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f521c000c00 tx=0x7f521c0058d0 comp rx=0 tx=0).stop 2024-09-16T00:16:30.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.577+0000 7f5227752640 1 -- 172.21.15.33:0/2182361115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5220102ef0 msgr2=0x7f5220197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:30.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.577+0000 7f5227752640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5220102ef0 0x7f5220197ba0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f521800b970 tx=0x7f521800be30 comp rx=0 tx=0).stop 2024-09-16T00:16:30.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.577+0000 7f5227752640 1 -- 172.21.15.33:0/2182361115 shutdown_connections 2024-09-16T00:16:30.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.577+0000 7f5227752640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f520c0a32d0 0x7f520c0a5790 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:30.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.577+0000 7f5227752640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5220102ef0 0x7f5220197ba0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:30.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.577+0000 7f5227752640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5220102570 0x7f5220197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:30.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.577+0000 7f5227752640 1 --2- 172.21.15.33:0/2182361115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5220100e20 0x7f5220197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:30.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.577+0000 7f5227752640 1 -- 172.21.15.33:0/2182361115 >> 172.21.15.33:0/2182361115 conn(0x7f52200fc7d0 msgr2=0x7f5220106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:30.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.577+0000 7f5227752640 1 -- 172.21.15.33:0/2182361115 shutdown_connections 2024-09-16T00:16:30.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:30.577+0000 7f5227752640 1 -- 172.21.15.33:0/2182361115 wait complete. 2024-09-16T00:16:31.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.096+0000 7f5ddce79640 1 -- 172.21.15.33:0/3074471509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dd8102570 msgr2=0x7f5dd81029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:31.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.096+0000 7f5ddce79640 1 --2- 172.21.15.33:0/3074471509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dd8102570 0x7f5dd81029b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc009900 tx=0x7f5dcc02a480 comp rx=0 tx=0).stop 2024-09-16T00:16:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.096+0000 7f5ddce79640 1 -- 172.21.15.33:0/3074471509 shutdown_connections 2024-09-16T00:16:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.096+0000 7f5ddce79640 1 --2- 172.21.15.33:0/3074471509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dd8102ef0 0x7f5dd810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.096+0000 7f5ddce79640 1 --2- 172.21.15.33:0/3074471509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dd8102570 0x7f5dd81029b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.096+0000 7f5ddce79640 1 --2- 172.21.15.33:0/3074471509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dd8100e20 0x7f5dd8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.096+0000 7f5ddce79640 1 -- 172.21.15.33:0/3074471509 >> 172.21.15.33:0/3074471509 conn(0x7f5dd80fc7d0 msgr2=0x7f5dd80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.096+0000 7f5ddce79640 1 -- 172.21.15.33:0/3074471509 shutdown_connections 2024-09-16T00:16:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.096+0000 7f5ddce79640 1 -- 172.21.15.33:0/3074471509 wait complete. 2024-09-16T00:16:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5ddce79640 1 Processor -- start 2024-09-16T00:16:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5ddce79640 1 -- start start 2024-09-16T00:16:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5ddce79640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dd8100e20 0x7f5dd81985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5ddce79640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dd8102570 0x7f5dd8198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5ddce79640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dd8102ef0 0x7f5dd819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5ddce79640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd8110f80 con 0x7f5dd8100e20 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5ddce79640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd8110e00 con 0x7f5dd8102ef0 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5ddce79640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd8111100 con 0x7f5dd8102570 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5dd659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dd8100e20 0x7f5dd81985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5dd659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dd8100e20 0x7f5dd81985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51914/0 (socket says 172.21.15.33:51914) 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5dd659c640 1 -- 172.21.15.33:0/774907874 learned_addr learned my addr 172.21.15.33:0/774907874 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5dd5d9b640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dd8102570 0x7f5dd8198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.097+0000 7f5dd6d9d640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dd8102ef0 0x7f5dd819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.098+0000 7f5dd659c640 1 -- 172.21.15.33:0/774907874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dd8102570 msgr2=0x7f5dd8198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.098+0000 7f5dd659c640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dd8102570 0x7f5dd8198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.098+0000 7f5dd659c640 1 -- 172.21.15.33:0/774907874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dd8102ef0 msgr2=0x7f5dd819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.098+0000 7f5dd659c640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dd8102ef0 0x7f5dd819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:31.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.098+0000 7f5dd659c640 1 -- 172.21.15.33:0/774907874 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd819d5a0 con 0x7f5dd8100e20 2024-09-16T00:16:31.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.098+0000 7f5dd659c640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dd8100e20 0x7f5dd81985f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5dc800c890 tx=0x7f5dc800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:31.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.098+0000 7f5dc77fe640 1 -- 172.21.15.33:0/774907874 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc8004220 con 0x7f5dd8100e20 2024-09-16T00:16:31.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.098+0000 7f5ddce79640 1 -- 172.21.15.33:0/774907874 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd819d890 con 0x7f5dd8100e20 2024-09-16T00:16:31.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.098+0000 7f5dc77fe640 1 -- 172.21.15.33:0/774907874 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dc80043c0 con 0x7f5dd8100e20 2024-09-16T00:16:31.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.098+0000 7f5dc77fe640 1 -- 172.21.15.33:0/774907874 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc8004a00 con 0x7f5dd8100e20 2024-09-16T00:16:31.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.098+0000 7f5ddce79640 1 -- 172.21.15.33:0/774907874 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd81a5170 con 0x7f5dd8100e20 2024-09-16T00:16:31.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.099+0000 7f5ddce79640 1 -- 172.21.15.33:0/774907874 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd8101200 con 0x7f5dd8100e20 2024-09-16T00:16:31.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.100+0000 7f5dc77fe640 1 -- 172.21.15.33:0/774907874 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f5dc80079b0 con 0x7f5dd8100e20 2024-09-16T00:16:31.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.101+0000 7f5dc77fe640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5dc00a32d0 0x7f5dc00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:31.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.101+0000 7f5dc77fe640 1 -- 172.21.15.33:0/774907874 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f5dc80cfab0 con 0x7f5dd8100e20 2024-09-16T00:16:31.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.101+0000 7f5dd5d9b640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5dc00a32d0 0x7f5dc00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:31.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.102+0000 7f5dd5d9b640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5dc00a32d0 0x7f5dc00a5790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc009230 tx=0x7f5dcc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:31.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.105+0000 7f5dc77fe640 1 -- 172.21.15.33:0/774907874 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5dc8014030 con 0x7f5dd8100e20 2024-09-16T00:16:31.176 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:31.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.300+0000 7f5ddce79640 1 -- 172.21.15.33:0/774907874 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f5dd810a0a0 con 0x7f5dd8100e20 2024-09-16T00:16:31.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.301+0000 7f5dc77fe640 1 -- 172.21.15.33:0/774907874 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5dc809bc60 con 0x7f5dd8100e20 2024-09-16T00:16:31.301 INFO:teuthology.orchestra.run.smithi033.stdout:103079215175 2024-09-16T00:16:31.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.303+0000 7f5ddce79640 1 -- 172.21.15.33:0/774907874 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5dc00a32d0 msgr2=0x7f5dc00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.303+0000 7f5ddce79640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5dc00a32d0 0x7f5dc00a5790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc009230 tx=0x7f5dcc0058d0 comp rx=0 tx=0).stop 2024-09-16T00:16:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.303+0000 7f5ddce79640 1 -- 172.21.15.33:0/774907874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dd8100e20 msgr2=0x7f5dd81985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.303+0000 7f5ddce79640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dd8100e20 0x7f5dd81985f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5dc800c890 tx=0x7f5dc800cd50 comp rx=0 tx=0).stop 2024-09-16T00:16:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.303+0000 7f5ddce79640 1 -- 172.21.15.33:0/774907874 shutdown_connections 2024-09-16T00:16:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.303+0000 7f5ddce79640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5dc00a32d0 0x7f5dc00a5790 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.303+0000 7f5ddce79640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dd8102ef0 0x7f5dd819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.303+0000 7f5ddce79640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dd8102570 0x7f5dd8198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.303+0000 7f5ddce79640 1 --2- 172.21.15.33:0/774907874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dd8100e20 0x7f5dd81985f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.303+0000 7f5ddce79640 1 -- 172.21.15.33:0/774907874 >> 172.21.15.33:0/774907874 conn(0x7f5dd80fc7d0 msgr2=0x7f5dd806a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.303+0000 7f5ddce79640 1 -- 172.21.15.33:0/774907874 shutdown_connections 2024-09-16T00:16:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:31.303+0000 7f5ddce79640 1 -- 172.21.15.33:0/774907874 wait complete. 2024-09-16T00:16:31.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:31 smithi033 ceph-mon[31243]: pgmap v329: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:31.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:31 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2182361115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-16T00:16:31.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:31 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/774907874' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-16T00:16:31.666 INFO:teuthology.orchestra.run.smithi033.stdout:73014444109 2024-09-16T00:16:31.666 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-16T00:16:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:31 smithi083 ceph-mon[35242]: pgmap v329: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:31 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2182361115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-16T00:16:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:31 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/774907874' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-16T00:16:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:31 smithi060 ceph-mon[35185]: pgmap v329: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:31 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2182361115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-16T00:16:31.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:31 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/774907874' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-16T00:16:32.292 INFO:tasks.cephadm.ceph_manager.ceph:need seq 103079215172 got 103079215175 for osd.3 2024-09-16T00:16:32.292 DEBUG:teuthology.parallel:result is None 2024-09-16T00:16:33.462 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738453 got 34359738455 for osd.0 2024-09-16T00:16:33.462 DEBUG:teuthology.parallel:result is None 2024-09-16T00:16:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.619+0000 7fa662c3f640 1 -- 172.21.15.33:0/1852558374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c101de0 msgr2=0x7fa65c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.619+0000 7fa662c3f640 1 --2- 172.21.15.33:0/1852558374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c101de0 0x7fa65c1021c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa654009900 tx=0x7fa65402a410 comp rx=0 tx=0).stop 2024-09-16T00:16:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.619+0000 7fa662c3f640 1 -- 172.21.15.33:0/1852558374 shutdown_connections 2024-09-16T00:16:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.619+0000 7fa662c3f640 1 --2- 172.21.15.33:0/1852558374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa65c106f10 0x7fa65c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.619+0000 7fa662c3f640 1 --2- 172.21.15.33:0/1852558374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa65c102790 0x7fa65c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.619+0000 7fa662c3f640 1 --2- 172.21.15.33:0/1852558374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c101de0 0x7fa65c1021c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.619+0000 7fa662c3f640 1 -- 172.21.15.33:0/1852558374 >> 172.21.15.33:0/1852558374 conn(0x7fa65c0fd8c0 msgr2=0x7fa65c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.619+0000 7fa662c3f640 1 -- 172.21.15.33:0/1852558374 shutdown_connections 2024-09-16T00:16:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.619+0000 7fa662c3f640 1 -- 172.21.15.33:0/1852558374 wait complete. 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa662c3f640 1 Processor -- start 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa662c3f640 1 -- start start 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa662c3f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c101de0 0x7fa65c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa662c3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa65c102790 0x7fa65c1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa662c3f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa65c106f10 0x7fa65c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa662c3f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa65c111f70 con 0x7fa65c101de0 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa662c3f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa65c111df0 con 0x7fa65c102790 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa662c3f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa65c1120f0 con 0x7fa65c106f10 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa6609db640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c101de0 0x7fa65c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa6609db640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c101de0 0x7fa65c198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51942/0 (socket says 172.21.15.33:51942) 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa6609db640 1 -- 172.21.15.33:0/3140605645 learned_addr learned my addr 172.21.15.33:0/3140605645 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa65bfff640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa65c102790 0x7fa65c1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.620+0000 7fa6611dc640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa65c106f10 0x7fa65c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.621+0000 7fa6609db640 1 -- 172.21.15.33:0/3140605645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa65c106f10 msgr2=0x7fa65c19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.621+0000 7fa6609db640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa65c106f10 0x7fa65c19cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.621+0000 7fa6609db640 1 -- 172.21.15.33:0/3140605645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa65c102790 msgr2=0x7fa65c1989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.621+0000 7fa6609db640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa65c102790 0x7fa65c1989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.621+0000 7fa6609db640 1 -- 172.21.15.33:0/3140605645 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65c19d420 con 0x7fa65c101de0 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.621+0000 7fa65bfff640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa65c102790 0x7fa65c1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.621+0000 7fa6609db640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c101de0 0x7fa65c198470 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa654006010 tx=0x7fa6540042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.621+0000 7fa659ffb640 1 -- 172.21.15.33:0/3140605645 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65403f030 con 0x7fa65c101de0 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.621+0000 7fa659ffb640 1 -- 172.21.15.33:0/3140605645 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa654003ce0 con 0x7fa65c101de0 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.621+0000 7fa659ffb640 1 -- 172.21.15.33:0/3140605645 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654035070 con 0x7fa65c101de0 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.621+0000 7fa662c3f640 1 -- 172.21.15.33:0/3140605645 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa65c19d6b0 con 0x7fa65c101de0 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.621+0000 7fa662c3f640 1 -- 172.21.15.33:0/3140605645 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa65c1a4ff0 con 0x7fa65c101de0 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.622+0000 7fa662c3f640 1 -- 172.21.15.33:0/3140605645 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa65c1032c0 con 0x7fa65c101de0 2024-09-16T00:16:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.623+0000 7fa659ffb640 1 -- 172.21.15.33:0/3140605645 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa654042030 con 0x7fa65c101de0 2024-09-16T00:16:33.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.624+0000 7fa659ffb640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa6480a32d0 0x7fa6480a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:33.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.624+0000 7fa659ffb640 1 -- 172.21.15.33:0/3140605645 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fa6540f04f0 con 0x7fa65c101de0 2024-09-16T00:16:33.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.624+0000 7fa65bfff640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa6480a32d0 0x7fa6480a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:33.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.625+0000 7fa65bfff640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa6480a32d0 0x7fa6480a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0fdce0 tx=0x7fa644005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:33.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.628+0000 7fa659ffb640 1 -- 172.21.15.33:0/3140605645 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa6540bcdb0 con 0x7fa65c101de0 2024-09-16T00:16:33.688 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:33.689 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:33.689 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:33.690 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:33.695 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:33.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:33 smithi033 ceph-mon[31243]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:33.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:33 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1216342898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-16T00:16:33.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.831+0000 7fa662c3f640 1 -- 172.21.15.33:0/3140605645 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fa65c10a0f0 con 0x7fa65c101de0 2024-09-16T00:16:33.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.831+0000 7fa659ffb640 1 -- 172.21.15.33:0/3140605645 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa6540bc6b0 con 0x7fa65c101de0 2024-09-16T00:16:33.832 INFO:teuthology.orchestra.run.smithi033.stdout:236223201325 2024-09-16T00:16:33.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.833+0000 7fa662c3f640 1 -- 172.21.15.33:0/3140605645 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa6480a32d0 msgr2=0x7fa6480a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:33.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.833+0000 7fa662c3f640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa6480a32d0 0x7fa6480a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0fdce0 tx=0x7fa644005f90 comp rx=0 tx=0).stop 2024-09-16T00:16:33.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.833+0000 7fa662c3f640 1 -- 172.21.15.33:0/3140605645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c101de0 msgr2=0x7fa65c198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:33.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.833+0000 7fa662c3f640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c101de0 0x7fa65c198470 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa654006010 tx=0x7fa6540042f0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.834+0000 7fa662c3f640 1 -- 172.21.15.33:0/3140605645 shutdown_connections 2024-09-16T00:16:33.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.834+0000 7fa662c3f640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa6480a32d0 0x7fa6480a5790 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.834+0000 7fa662c3f640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa65c106f10 0x7fa65c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.834+0000 7fa662c3f640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa65c102790 0x7fa65c1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.834+0000 7fa662c3f640 1 --2- 172.21.15.33:0/3140605645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c101de0 0x7fa65c198470 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.834+0000 7fa662c3f640 1 -- 172.21.15.33:0/3140605645 >> 172.21.15.33:0/3140605645 conn(0x7fa65c0fd8c0 msgr2=0x7fa65c0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:33.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.834+0000 7fa662c3f640 1 -- 172.21.15.33:0/3140605645 shutdown_connections 2024-09-16T00:16:33.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.834+0000 7fa662c3f640 1 -- 172.21.15.33:0/3140605645 wait complete. 2024-09-16T00:16:33.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.164+0000 7f34e9748640 1 -- 172.21.15.33:0/3802149215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34e4101df0 msgr2=0x7f34e41021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:33.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.164+0000 7f34e9748640 1 --2- 172.21.15.33:0/3802149215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34e4101df0 0x7f34e41021d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f34d4009900 tx=0x7f34d402a3f0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.165+0000 7f34e9748640 1 -- 172.21.15.33:0/3802149215 shutdown_connections 2024-09-16T00:16:33.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.165+0000 7f34e9748640 1 --2- 172.21.15.33:0/3802149215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e4106f20 0x7f34e410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.165+0000 7f34e9748640 1 --2- 172.21.15.33:0/3802149215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34e41027a0 0x7f34e41067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.165+0000 7f34e9748640 1 --2- 172.21.15.33:0/3802149215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34e4101df0 0x7f34e41021d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.165+0000 7f34e9748640 1 -- 172.21.15.33:0/3802149215 >> 172.21.15.33:0/3802149215 conn(0x7f34e40fd8f0 msgr2=0x7f34e40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:33.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.165+0000 7f34e9748640 1 -- 172.21.15.33:0/3802149215 shutdown_connections 2024-09-16T00:16:33.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.165+0000 7f34e9748640 1 -- 172.21.15.33:0/3802149215 wait complete. 2024-09-16T00:16:33.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.165+0000 7f34e9748640 1 Processor -- start 2024-09-16T00:16:33.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e9748640 1 -- start start 2024-09-16T00:16:33.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e9748640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e4101df0 0x7f34e40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:33.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e9748640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34e41027a0 0x7f34e4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:33.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e9748640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34e4106f20 0x7f34e4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:33.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e9748640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f34e4111f80 con 0x7f34e41027a0 2024-09-16T00:16:33.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e9748640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f34e4111e00 con 0x7f34e4101df0 2024-09-16T00:16:33.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e9748640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f34e4112100 con 0x7f34e4106f20 2024-09-16T00:16:33.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e27fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34e41027a0 0x7f34e4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:33.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e27fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34e41027a0 0x7f34e4074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51930/0 (socket says 172.21.15.33:51930) 2024-09-16T00:16:33.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e27fc640 1 -- 172.21.15.33:0/1216342898 learned_addr learned my addr 172.21.15.33:0/1216342898 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:33.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e37fe640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34e4106f20 0x7f34e4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:33.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e2ffd640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e4101df0 0x7f34e40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:33.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e27fc640 1 -- 172.21.15.33:0/1216342898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34e4106f20 msgr2=0x7f34e4074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:33.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e27fc640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34e4106f20 0x7f34e4074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e27fc640 1 -- 172.21.15.33:0/1216342898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e4101df0 msgr2=0x7f34e40744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:33.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e27fc640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e4101df0 0x7f34e40744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.857 INFO:teuthology.orchestra.run.smithi033.stdout:206158430259 2024-09-16T00:16:33.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.166+0000 7f34e27fc640 1 -- 172.21.15.33:0/1216342898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e41a6840 con 0x7f34e41027a0 2024-09-16T00:16:33.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.167+0000 7f34e27fc640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34e41027a0 0x7f34e4074a30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f34d800c920 tx=0x7f34d800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:33.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.167+0000 7f34cbfff640 1 -- 172.21.15.33:0/1216342898 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8007c70 con 0x7f34e41027a0 2024-09-16T00:16:33.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.167+0000 7f34cbfff640 1 -- 172.21.15.33:0/1216342898 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34d8007e10 con 0x7f34e41027a0 2024-09-16T00:16:33.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.167+0000 7f34e9748640 1 -- 172.21.15.33:0/1216342898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e41a6b30 con 0x7f34e41027a0 2024-09-16T00:16:33.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.167+0000 7f34cbfff640 1 -- 172.21.15.33:0/1216342898 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8018890 con 0x7f34e41027a0 2024-09-16T00:16:33.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.167+0000 7f34e9748640 1 -- 172.21.15.33:0/1216342898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e41a70c0 con 0x7f34e41027a0 2024-09-16T00:16:33.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.168+0000 7f34e9748640 1 -- 172.21.15.33:0/1216342898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34e41021d0 con 0x7f34e41027a0 2024-09-16T00:16:33.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.169+0000 7f34cbfff640 1 -- 172.21.15.33:0/1216342898 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f34d8018a50 con 0x7f34e41027a0 2024-09-16T00:16:33.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.170+0000 7f34cbfff640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f34cc0a3350 0x7f34cc0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:33.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.170+0000 7f34e2ffd640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f34cc0a3350 0x7f34cc0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:33.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.170+0000 7f34cbfff640 1 -- 172.21.15.33:0/1216342898 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f34d80d01d0 con 0x7f34e41027a0 2024-09-16T00:16:33.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.170+0000 7f34e2ffd640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f34cc0a3350 0x7f34cc0a5810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f34d40096c0 tx=0x7f34d4031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:33.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.173+0000 7f34cbfff640 1 -- 172.21.15.33:0/1216342898 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f34d8014030 con 0x7f34e41027a0 2024-09-16T00:16:33.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.350+0000 7f34e9748640 1 -- 172.21.15.33:0/1216342898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f34e4075a20 con 0x7f34e41027a0 2024-09-16T00:16:33.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.350+0000 7f34cbfff640 1 -- 172.21.15.33:0/1216342898 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f34e4075a20 con 0x7f34e41027a0 2024-09-16T00:16:33.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.352+0000 7f34e9748640 1 -- 172.21.15.33:0/1216342898 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f34cc0a3350 msgr2=0x7f34cc0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:33.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.352+0000 7f34e9748640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f34cc0a3350 0x7f34cc0a5810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f34d40096c0 tx=0x7f34d4031490 comp rx=0 tx=0).stop 2024-09-16T00:16:33.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.352+0000 7f34e9748640 1 -- 172.21.15.33:0/1216342898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34e41027a0 msgr2=0x7f34e4074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:33.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.352+0000 7f34e9748640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34e41027a0 0x7f34e4074a30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f34d800c920 tx=0x7f34d800cde0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.353+0000 7f34e9748640 1 -- 172.21.15.33:0/1216342898 shutdown_connections 2024-09-16T00:16:33.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.353+0000 7f34e9748640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f34cc0a3350 0x7f34cc0a5810 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.353+0000 7f34e9748640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34e4106f20 0x7f34e4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.353+0000 7f34e9748640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34e41027a0 0x7f34e4074a30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.353+0000 7f34e9748640 1 --2- 172.21.15.33:0/1216342898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e4101df0 0x7f34e40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:33.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.353+0000 7f34e9748640 1 -- 172.21.15.33:0/1216342898 >> 172.21.15.33:0/1216342898 conn(0x7f34e40fd8f0 msgr2=0x7f34e40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:33.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.353+0000 7f34e9748640 1 -- 172.21.15.33:0/1216342898 shutdown_connections 2024-09-16T00:16:33.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:33.353+0000 7f34e9748640 1 -- 172.21.15.33:0/1216342898 wait complete. 2024-09-16T00:16:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:33 smithi083 ceph-mon[35242]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:33 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1216342898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-16T00:16:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:33 smithi060 ceph-mon[35185]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:33 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1216342898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-16T00:16:34.713 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:34 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3140605645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-16T00:16:34.738 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:34.745 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201323 got 236223201325 for osd.8 2024-09-16T00:16:34.745 DEBUG:teuthology.parallel:result is None 2024-09-16T00:16:34.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:34 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3140605645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-16T00:16:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:34 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3140605645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-16T00:16:35.175 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430256 got 206158430259 for osd.7 2024-09-16T00:16:35.175 DEBUG:teuthology.parallel:result is None 2024-09-16T00:16:35.653 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:35 smithi033 ceph-mon[31243]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:35.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.889+0000 7f9bd5c60640 1 -- 172.21.15.33:0/98825761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd0101df0 msgr2=0x7f9bd01021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:35.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.889+0000 7f9bd5c60640 1 --2- 172.21.15.33:0/98825761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd0101df0 0x7f9bd01021d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009900 tx=0x7f9bc002a410 comp rx=0 tx=0).stop 2024-09-16T00:16:35.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.889+0000 7f9bd5c60640 1 -- 172.21.15.33:0/98825761 shutdown_connections 2024-09-16T00:16:35.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.889+0000 7f9bd5c60640 1 --2- 172.21.15.33:0/98825761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0106f20 0x7f9bd010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.889+0000 7f9bd5c60640 1 --2- 172.21.15.33:0/98825761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bd01027a0 0x7f9bd01067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.889+0000 7f9bd5c60640 1 --2- 172.21.15.33:0/98825761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd0101df0 0x7f9bd01021d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.889+0000 7f9bd5c60640 1 -- 172.21.15.33:0/98825761 >> 172.21.15.33:0/98825761 conn(0x7f9bd00fd8b0 msgr2=0x7f9bd00ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.889+0000 7f9bd5c60640 1 -- 172.21.15.33:0/98825761 shutdown_connections 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.889+0000 7f9bd5c60640 1 -- 172.21.15.33:0/98825761 wait complete. 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.889+0000 7f9bd5c60640 1 Processor -- start 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.889+0000 7f9bd5c60640 1 -- start start 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bd5c60640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bd0101df0 0x7f9bd0198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bd5c60640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd01027a0 0x7f9bd0198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bd5c60640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0106f20 0x7f9bd019cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bd5c60640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd0111fa0 con 0x7f9bd01027a0 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bd5c60640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd0111e20 con 0x7f9bd0106f20 2024-09-16T00:16:35.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bd5c60640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd0112120 con 0x7f9bd0101df0 2024-09-16T00:16:35.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bceffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd01027a0 0x7f9bd0198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:35.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bceffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd01027a0 0x7f9bd0198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51950/0 (socket says 172.21.15.33:51950) 2024-09-16T00:16:35.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bceffd640 1 -- 172.21.15.33:0/2536449749 learned_addr learned my addr 172.21.15.33:0/2536449749 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:35.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bcffff640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0106f20 0x7f9bd019cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:35.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bcf7fe640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bd0101df0 0x7f9bd0198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:35.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bceffd640 1 -- 172.21.15.33:0/2536449749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bd0101df0 msgr2=0x7f9bd0198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:35.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bceffd640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bd0101df0 0x7f9bd0198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:35.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bceffd640 1 -- 172.21.15.33:0/2536449749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0106f20 msgr2=0x7f9bd019cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:35.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bceffd640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0106f20 0x7f9bd019cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:35.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bceffd640 1 -- 172.21.15.33:0/2536449749 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd019d3f0 con 0x7f9bd01027a0 2024-09-16T00:16:35.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bcf7fe640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bd0101df0 0x7f9bd0198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:16:35.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bceffd640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd01027a0 0x7f9bd0198980 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400c920 tx=0x7f9bc400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:35.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.890+0000 7f9bccff9640 1 -- 172.21.15.33:0/2536449749 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4007c70 con 0x7f9bd01027a0 2024-09-16T00:16:35.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.891+0000 7f9bccff9640 1 -- 172.21.15.33:0/2536449749 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bc4007e10 con 0x7f9bd01027a0 2024-09-16T00:16:35.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.891+0000 7f9bd5c60640 1 -- 172.21.15.33:0/2536449749 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd019d6e0 con 0x7f9bd01027a0 2024-09-16T00:16:35.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.891+0000 7f9bccff9640 1 -- 172.21.15.33:0/2536449749 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4018890 con 0x7f9bd01027a0 2024-09-16T00:16:35.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.891+0000 7f9bd5c60640 1 -- 172.21.15.33:0/2536449749 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd01a4fc0 con 0x7f9bd01027a0 2024-09-16T00:16:35.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.892+0000 7f9bd5c60640 1 -- 172.21.15.33:0/2536449749 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd01021d0 con 0x7f9bd01027a0 2024-09-16T00:16:35.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.893+0000 7f9bccff9640 1 -- 172.21.15.33:0/2536449749 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f9bc4018a70 con 0x7f9bd01027a0 2024-09-16T00:16:35.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.893+0000 7f9bccff9640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9bb80a3350 0x7f9bb80a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:35.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.893+0000 7f9bcf7fe640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9bb80a3350 0x7f9bb80a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:35.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.894+0000 7f9bccff9640 1 -- 172.21.15.33:0/2536449749 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f9bc40cfef0 con 0x7f9bd01027a0 2024-09-16T00:16:35.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.894+0000 7f9bcf7fe640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9bb80a3350 0x7f9bb80a5810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0005ab0 tx=0x7f9bc0005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:35.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:35.898+0000 7f9bccff9640 1 -- 172.21.15.33:0/2536449749 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9bc4014030 con 0x7f9bd01027a0 2024-09-16T00:16:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:35 smithi083 ceph-mon[35242]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:35 smithi060 ceph-mon[35185]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:36.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.075+0000 7f9bd5c60640 1 -- 172.21.15.33:0/2536449749 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f9bd0109b70 con 0x7f9bd01027a0 2024-09-16T00:16:36.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.075+0000 7f9bccff9640 1 -- 172.21.15.33:0/2536449749 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9bc409c750 con 0x7f9bd01027a0 2024-09-16T00:16:36.076 INFO:teuthology.orchestra.run.smithi033.stdout:51539607635 2024-09-16T00:16:36.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.077+0000 7f9bd5c60640 1 -- 172.21.15.33:0/2536449749 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9bb80a3350 msgr2=0x7f9bb80a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:36.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.077+0000 7f9bd5c60640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9bb80a3350 0x7f9bb80a5810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0005ab0 tx=0x7f9bc0005a20 comp rx=0 tx=0).stop 2024-09-16T00:16:36.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.077+0000 7f9bd5c60640 1 -- 172.21.15.33:0/2536449749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd01027a0 msgr2=0x7f9bd0198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:36.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.077+0000 7f9bd5c60640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd01027a0 0x7f9bd0198980 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400c920 tx=0x7f9bc400cde0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.077+0000 7f9bd5c60640 1 -- 172.21.15.33:0/2536449749 shutdown_connections 2024-09-16T00:16:36.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.077+0000 7f9bd5c60640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9bb80a3350 0x7f9bb80a5810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.077+0000 7f9bd5c60640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0106f20 0x7f9bd019cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.077+0000 7f9bd5c60640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd01027a0 0x7f9bd0198980 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.077+0000 7f9bd5c60640 1 --2- 172.21.15.33:0/2536449749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bd0101df0 0x7f9bd0198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.077+0000 7f9bd5c60640 1 -- 172.21.15.33:0/2536449749 >> 172.21.15.33:0/2536449749 conn(0x7f9bd00fd8b0 msgr2=0x7f9bd00febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:36.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.078+0000 7f9bd5c60640 1 -- 172.21.15.33:0/2536449749 shutdown_connections 2024-09-16T00:16:36.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.078+0000 7f9bd5c60640 1 -- 172.21.15.33:0/2536449749 wait complete. 2024-09-16T00:16:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.128+0000 7fe0839aa640 1 -- 172.21.15.33:0/29149000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe07c108290 msgr2=0x7fe07c10ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.128+0000 7fe0839aa640 1 --2- 172.21.15.33:0/29149000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe07c108290 0x7fe07c10ec30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe074009cd0 tx=0x7fe07402a8e0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.128+0000 7fe0839aa640 1 -- 172.21.15.33:0/29149000 shutdown_connections 2024-09-16T00:16:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.128+0000 7fe0839aa640 1 --2- 172.21.15.33:0/29149000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe07c108290 0x7fe07c10ec30 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.128+0000 7fe0839aa640 1 --2- 172.21.15.33:0/29149000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe07c103740 0x7fe07c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.128+0000 7fe0839aa640 1 --2- 172.21.15.33:0/29149000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07c102d90 0x7fe07c103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.128+0000 7fe0839aa640 1 -- 172.21.15.33:0/29149000 >> 172.21.15.33:0/29149000 conn(0x7fe07c0fc7d0 msgr2=0x7fe07c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.128+0000 7fe0839aa640 1 -- 172.21.15.33:0/29149000 shutdown_connections 2024-09-16T00:16:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.128+0000 7fe0839aa640 1 -- 172.21.15.33:0/29149000 wait complete. 2024-09-16T00:16:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe0839aa640 1 Processor -- start 2024-09-16T00:16:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe0839aa640 1 -- start start 2024-09-16T00:16:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe0839aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe07c102d90 0x7fe07c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe0839aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07c103740 0x7fe07c19ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe0839aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe07c108290 0x7fe07c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe0839aa640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c112090 con 0x7fe07c102d90 2024-09-16T00:16:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe0839aa640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c111f10 con 0x7fe07c103740 2024-09-16T00:16:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe0839aa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c112210 con 0x7fe07c108290 2024-09-16T00:16:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe081746640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe07c102d90 0x7fe07c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe081746640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe07c102d90 0x7fe07c19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51968/0 (socket says 172.21.15.33:51968) 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe081746640 1 -- 172.21.15.33:0/1677308647 learned_addr learned my addr 172.21.15.33:0/1677308647 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe080f45640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07c103740 0x7fe07c19ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.129+0000 7fe081f47640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe07c108290 0x7fe07c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.130+0000 7fe081746640 1 -- 172.21.15.33:0/1677308647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe07c108290 msgr2=0x7fe07c19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.130+0000 7fe081746640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe07c108290 0x7fe07c19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.130+0000 7fe081746640 1 -- 172.21.15.33:0/1677308647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07c103740 msgr2=0x7fe07c19ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.130+0000 7fe081746640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07c103740 0x7fe07c19ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.130+0000 7fe081746640 1 -- 172.21.15.33:0/1677308647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe07c19f6e0 con 0x7fe07c102d90 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.130+0000 7fe081746640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe07c102d90 0x7fe07c19a730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe07800e9f0 tx=0x7fe07800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.130+0000 7fe0727fc640 1 -- 172.21.15.33:0/1677308647 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07800cd80 con 0x7fe07c102d90 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.130+0000 7fe0727fc640 1 -- 172.21.15.33:0/1677308647 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe078010910 con 0x7fe07c102d90 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.130+0000 7fe0727fc640 1 -- 172.21.15.33:0/1677308647 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe078018990 con 0x7fe07c102d90 2024-09-16T00:16:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.130+0000 7fe0839aa640 1 -- 172.21.15.33:0/1677308647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe07c19f9d0 con 0x7fe07c102d90 2024-09-16T00:16:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.130+0000 7fe0839aa640 1 -- 172.21.15.33:0/1677308647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe07c1a72b0 con 0x7fe07c102d90 2024-09-16T00:16:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.131+0000 7fe0839aa640 1 -- 172.21.15.33:0/1677308647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe07c103170 con 0x7fe07c102d90 2024-09-16T00:16:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.132+0000 7fe0727fc640 1 -- 172.21.15.33:0/1677308647 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fe078010ac0 con 0x7fe07c102d90 2024-09-16T00:16:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.133+0000 7fe0727fc640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0680a32d0 0x7fe0680a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.133+0000 7fe0727fc640 1 -- 172.21.15.33:0/1677308647 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fe0780d0130 con 0x7fe07c102d90 2024-09-16T00:16:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.133+0000 7fe080f45640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0680a32d0 0x7fe0680a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.133+0000 7fe080f45640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0680a32d0 0x7fe0680a5790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe07c100af0 tx=0x7fe06c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:36.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.137+0000 7fe0727fc640 1 -- 172.21.15.33:0/1677308647 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe07809c340 con 0x7fe07c102d90 2024-09-16T00:16:36.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.313+0000 7fe0839aa640 1 -- 172.21.15.33:0/1677308647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fe07c19ba90 con 0x7fe07c102d90 2024-09-16T00:16:36.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.314+0000 7fe0727fc640 1 -- 172.21.15.33:0/1677308647 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fe0780a0640 con 0x7fe07c102d90 2024-09-16T00:16:36.314 INFO:teuthology.orchestra.run.smithi033.stdout:317827579935 2024-09-16T00:16:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.315+0000 7fe0839aa640 1 -- 172.21.15.33:0/1677308647 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0680a32d0 msgr2=0x7fe0680a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.315+0000 7fe0839aa640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0680a32d0 0x7fe0680a5790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe07c100af0 tx=0x7fe06c005f90 comp rx=0 tx=0).stop 2024-09-16T00:16:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.316+0000 7fe0839aa640 1 -- 172.21.15.33:0/1677308647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe07c102d90 msgr2=0x7fe07c19a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.316+0000 7fe0839aa640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe07c102d90 0x7fe07c19a730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe07800e9f0 tx=0x7fe07800eeb0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.316+0000 7fe0839aa640 1 -- 172.21.15.33:0/1677308647 shutdown_connections 2024-09-16T00:16:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.316+0000 7fe0839aa640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0680a32d0 0x7fe0680a5790 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.316+0000 7fe0839aa640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe07c108290 0x7fe07c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.316+0000 7fe0839aa640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07c103740 0x7fe07c19ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.316+0000 7fe0839aa640 1 --2- 172.21.15.33:0/1677308647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe07c102d90 0x7fe07c19a730 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.316+0000 7fe0839aa640 1 -- 172.21.15.33:0/1677308647 >> 172.21.15.33:0/1677308647 conn(0x7fe07c0fc7d0 msgr2=0x7fe07c101400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.316+0000 7fe0839aa640 1 -- 172.21.15.33:0/1677308647 shutdown_connections 2024-09-16T00:16:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:36.316+0000 7fe0839aa640 1 -- 172.21.15.33:0/1677308647 wait complete. 2024-09-16T00:16:36.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:36 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2536449749' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-16T00:16:36.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:36 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1677308647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-16T00:16:36.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:36 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2536449749' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-16T00:16:36.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:36 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1677308647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-16T00:16:36.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:36 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2536449749' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-16T00:16:36.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:36 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1677308647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-16T00:16:37.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.180+0000 7f2cfebba640 1 -- 172.21.15.33:0/1461932003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2cf8102ef0 msgr2=0x7f2cf810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:37.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.180+0000 7f2cfebba640 1 --2- 172.21.15.33:0/1461932003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2cf8102ef0 0x7f2cf810da40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009ca0 tx=0x7f2cf002a950 comp rx=0 tx=0).stop 2024-09-16T00:16:37.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.181+0000 7f2cfebba640 1 -- 172.21.15.33:0/1461932003 shutdown_connections 2024-09-16T00:16:37.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.181+0000 7f2cfebba640 1 --2- 172.21.15.33:0/1461932003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2cf8102ef0 0x7f2cf810da40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.181+0000 7f2cfebba640 1 --2- 172.21.15.33:0/1461932003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cf8102570 0x7f2cf81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.181+0000 7f2cfebba640 1 --2- 172.21.15.33:0/1461932003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf8100e20 0x7f2cf8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.181+0000 7f2cfebba640 1 -- 172.21.15.33:0/1461932003 >> 172.21.15.33:0/1461932003 conn(0x7f2cf80fc7d0 msgr2=0x7f2cf80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:37.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.181+0000 7f2cfebba640 1 -- 172.21.15.33:0/1461932003 shutdown_connections 2024-09-16T00:16:37.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.181+0000 7f2cfebba640 1 -- 172.21.15.33:0/1461932003 wait complete. 2024-09-16T00:16:37.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.181+0000 7f2cfebba640 1 Processor -- start 2024-09-16T00:16:37.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.181+0000 7f2cfebba640 1 -- start start 2024-09-16T00:16:37.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.181+0000 7f2cfebba640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2cf8100e20 0x7f2cf80fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:37.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.181+0000 7f2cfebba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf8102570 0x7f2cf80fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:37.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfebba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cf8102ef0 0x7f2cf80fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:37.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfebba640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf8110f80 con 0x7f2cf8100e20 2024-09-16T00:16:37.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfebba640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf8110e00 con 0x7f2cf8102ef0 2024-09-16T00:16:37.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfebba640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf8111100 con 0x7f2cf8102570 2024-09-16T00:16:37.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfc956640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2cf8100e20 0x7f2cf80fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:37.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfc956640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2cf8100e20 0x7f2cf80fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51994/0 (socket says 172.21.15.33:51994) 2024-09-16T00:16:37.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfc956640 1 -- 172.21.15.33:0/3383512470 learned_addr learned my addr 172.21.15.33:0/3383512470 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:37.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfd157640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cf8102ef0 0x7f2cf80fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:37.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cf7fff640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf8102570 0x7f2cf80fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:37.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfc956640 1 -- 172.21.15.33:0/3383512470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf8102570 msgr2=0x7f2cf80fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:37.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfc956640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf8102570 0x7f2cf80fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfc956640 1 -- 172.21.15.33:0/3383512470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cf8102ef0 msgr2=0x7f2cf80fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:37.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfc956640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cf8102ef0 0x7f2cf80fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfc956640 1 -- 172.21.15.33:0/3383512470 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf81a69a0 con 0x7f2cf8100e20 2024-09-16T00:16:37.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfc956640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2cf8100e20 0x7f2cf80fcc80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2ce800d770 tx=0x7f2ce800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:37.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cf5ffb640 1 -- 172.21.15.33:0/3383512470 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8017070 con 0x7f2cf8100e20 2024-09-16T00:16:37.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cf5ffb640 1 -- 172.21.15.33:0/3383512470 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ce8005500 con 0x7f2cf8100e20 2024-09-16T00:16:37.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cf5ffb640 1 -- 172.21.15.33:0/3383512470 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8004e00 con 0x7f2cf8100e20 2024-09-16T00:16:37.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.182+0000 7f2cfebba640 1 -- 172.21.15.33:0/3383512470 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf81a6c90 con 0x7f2cf8100e20 2024-09-16T00:16:37.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.183+0000 7f2cfebba640 1 -- 172.21.15.33:0/3383512470 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf81a7180 con 0x7f2cf8100e20 2024-09-16T00:16:37.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.184+0000 7f2cfebba640 1 -- 172.21.15.33:0/3383512470 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf8101200 con 0x7f2cf8100e20 2024-09-16T00:16:37.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.185+0000 7f2cf5ffb640 1 -- 172.21.15.33:0/3383512470 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f2ce8004fa0 con 0x7f2cf8100e20 2024-09-16T00:16:37.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.185+0000 7f2cf5ffb640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2ce00a32d0 0x7f2ce00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:37.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.185+0000 7f2cf7fff640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2ce00a32d0 0x7f2ce00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:37.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.185+0000 7f2cf5ffb640 1 -- 172.21.15.33:0/3383512470 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f2ce80cf880 con 0x7f2cf8100e20 2024-09-16T00:16:37.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.186+0000 7f2cf7fff640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2ce00a32d0 0x7f2ce00a5790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80fe710 tx=0x7f2cec005be0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:37.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.190+0000 7f2cf5ffb640 1 -- 172.21.15.33:0/3383512470 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2ce809c0e0 con 0x7f2cf8100e20 2024-09-16T00:16:37.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.366+0000 7f2cfebba640 1 -- 172.21.15.33:0/3383512470 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f2cf810a0a0 con 0x7f2cf8100e20 2024-09-16T00:16:37.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.366+0000 7f2cf5ffb640 1 -- 172.21.15.33:0/3383512470 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2ce809b9e0 con 0x7f2cf8100e20 2024-09-16T00:16:37.367 INFO:teuthology.orchestra.run.smithi033.stdout:266287972394 2024-09-16T00:16:37.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.368+0000 7f2cfebba640 1 -- 172.21.15.33:0/3383512470 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2ce00a32d0 msgr2=0x7f2ce00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:37.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.369+0000 7f2cfebba640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2ce00a32d0 0x7f2ce00a5790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80fe710 tx=0x7f2cec005be0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.369+0000 7f2cfebba640 1 -- 172.21.15.33:0/3383512470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2cf8100e20 msgr2=0x7f2cf80fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:37.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.369+0000 7f2cfebba640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2cf8100e20 0x7f2cf80fcc80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2ce800d770 tx=0x7f2ce800dc30 comp rx=0 tx=0).stop 2024-09-16T00:16:37.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.369+0000 7f2cfebba640 1 -- 172.21.15.33:0/3383512470 shutdown_connections 2024-09-16T00:16:37.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.369+0000 7f2cfebba640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2ce00a32d0 0x7f2ce00a5790 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.369+0000 7f2cfebba640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cf8102ef0 0x7f2cf80fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.369+0000 7f2cfebba640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf8102570 0x7f2cf80fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.369+0000 7f2cfebba640 1 --2- 172.21.15.33:0/3383512470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2cf8100e20 0x7f2cf80fcc80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.369+0000 7f2cfebba640 1 -- 172.21.15.33:0/3383512470 >> 172.21.15.33:0/3383512470 conn(0x7f2cf80fc7d0 msgr2=0x7f2cf806a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:37.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.369+0000 7f2cfebba640 1 -- 172.21.15.33:0/3383512470 shutdown_connections 2024-09-16T00:16:37.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.369+0000 7f2cfebba640 1 -- 172.21.15.33:0/3383512470 wait complete. 2024-09-16T00:16:37.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.392+0000 7f0b23491640 1 -- 172.21.15.33:0/4164094567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b1c108290 msgr2=0x7f0b1c10ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:37.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.392+0000 7f0b23491640 1 --2- 172.21.15.33:0/4164094567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b1c108290 0x7f0b1c10ec30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0b14009cd0 tx=0x7f0b1402a8e0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.392+0000 7f0b23491640 1 -- 172.21.15.33:0/4164094567 shutdown_connections 2024-09-16T00:16:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.392+0000 7f0b23491640 1 --2- 172.21.15.33:0/4164094567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b1c108290 0x7f0b1c10ec30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.392+0000 7f0b23491640 1 --2- 172.21.15.33:0/4164094567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b1c103740 0x7f0b1c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.392+0000 7f0b23491640 1 --2- 172.21.15.33:0/4164094567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1c102d90 0x7f0b1c103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.392+0000 7f0b23491640 1 -- 172.21.15.33:0/4164094567 >> 172.21.15.33:0/4164094567 conn(0x7f0b1c0fc7d0 msgr2=0x7f0b1c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.393+0000 7f0b23491640 1 -- 172.21.15.33:0/4164094567 shutdown_connections 2024-09-16T00:16:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.393+0000 7f0b23491640 1 -- 172.21.15.33:0/4164094567 wait complete. 2024-09-16T00:16:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.393+0000 7f0b23491640 1 Processor -- start 2024-09-16T00:16:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.393+0000 7f0b23491640 1 -- start start 2024-09-16T00:16:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.393+0000 7f0b23491640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1c102d90 0x7f0b1c1009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.393+0000 7f0b23491640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b1c103740 0x7f0b1c100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.393+0000 7f0b23491640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b1c108290 0x7f0b1c101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.393+0000 7f0b23491640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c1120a0 con 0x7f0b1c103740 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.393+0000 7f0b23491640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c111f20 con 0x7f0b1c108290 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.393+0000 7f0b23491640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c112220 con 0x7f0b1c102d90 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b20a2c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b1c103740 0x7f0b1c100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b20a2c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b1c103740 0x7f0b1c100f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52024/0 (socket says 172.21.15.33:52024) 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b20a2c640 1 -- 172.21.15.33:0/1701031313 learned_addr learned my addr 172.21.15.33:0/1701031313 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b21a2e640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b1c108290 0x7f0b1c101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b2122d640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1c102d90 0x7f0b1c1009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b20a2c640 1 -- 172.21.15.33:0/1701031313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1c102d90 msgr2=0x7f0b1c1009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b20a2c640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1c102d90 0x7f0b1c1009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b20a2c640 1 -- 172.21.15.33:0/1701031313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b1c108290 msgr2=0x7f0b1c101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b20a2c640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b1c108290 0x7f0b1c101470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b20a2c640 1 -- 172.21.15.33:0/1701031313 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1c1a6960 con 0x7f0b1c103740 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b20a2c640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b1c103740 0x7f0b1c100f30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00c970 tx=0x7f0b0c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b127fc640 1 -- 172.21.15.33:0/1701031313 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c014070 con 0x7f0b1c103740 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b23491640 1 -- 172.21.15.33:0/1701031313 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b1c1a6c50 con 0x7f0b1c103740 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b127fc640 1 -- 172.21.15.33:0/1701031313 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b0c004440 con 0x7f0b1c103740 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b127fc640 1 -- 172.21.15.33:0/1701031313 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c00f450 con 0x7f0b1c103740 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.394+0000 7f0b23491640 1 -- 172.21.15.33:0/1701031313 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b1c1a7190 con 0x7f0b1c103740 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.396+0000 7f0b23491640 1 -- 172.21.15.33:0/1701031313 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b1c103170 con 0x7f0b1c103740 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.397+0000 7f0b127fc640 1 -- 172.21.15.33:0/1701031313 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0b0c017070 con 0x7f0b1c103740 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.398+0000 7f0b127fc640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b080a32d0 0x7f0b080a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.398+0000 7f0b2122d640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b080a32d0 0x7f0b080a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.398+0000 7f0b127fc640 1 -- 172.21.15.33:0/1701031313 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f0b0c0cfb90 con 0x7f0b1c103740 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.399+0000 7f0b2122d640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b080a32d0 0x7f0b080a5790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0b18005fd0 tx=0x7f0b18005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:37.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.403+0000 7f0b127fc640 1 -- 172.21.15.33:0/1701031313 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0b0c01e020 con 0x7f0b1c103740 2024-09-16T00:16:37.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.581+0000 7f0b23491640 1 -- 172.21.15.33:0/1701031313 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f0b1c1a7480 con 0x7f0b1c103740 2024-09-16T00:16:37.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.582+0000 7f0b127fc640 1 -- 172.21.15.33:0/1701031313 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f0b0c09c3f0 con 0x7f0b1c103740 2024-09-16T00:16:37.582 INFO:teuthology.orchestra.run.smithi033.stdout:287762808868 2024-09-16T00:16:37.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.584+0000 7f0b23491640 1 -- 172.21.15.33:0/1701031313 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b080a32d0 msgr2=0x7f0b080a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:37.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.584+0000 7f0b23491640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b080a32d0 0x7f0b080a5790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0b18005fd0 tx=0x7f0b18005850 comp rx=0 tx=0).stop 2024-09-16T00:16:37.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.584+0000 7f0b23491640 1 -- 172.21.15.33:0/1701031313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b1c103740 msgr2=0x7f0b1c100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:37.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.584+0000 7f0b23491640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b1c103740 0x7f0b1c100f30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00c970 tx=0x7f0b0c00ce30 comp rx=0 tx=0).stop 2024-09-16T00:16:37.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.584+0000 7f0b23491640 1 -- 172.21.15.33:0/1701031313 shutdown_connections 2024-09-16T00:16:37.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.584+0000 7f0b23491640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b080a32d0 0x7f0b080a5790 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.584+0000 7f0b23491640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b1c108290 0x7f0b1c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.584+0000 7f0b23491640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b1c103740 0x7f0b1c100f30 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.584+0000 7f0b23491640 1 --2- 172.21.15.33:0/1701031313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1c102d90 0x7f0b1c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:37.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.584+0000 7f0b23491640 1 -- 172.21.15.33:0/1701031313 >> 172.21.15.33:0/1701031313 conn(0x7f0b1c0fc7d0 msgr2=0x7f0b1c0fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:37.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.584+0000 7f0b23491640 1 -- 172.21.15.33:0/1701031313 shutdown_connections 2024-09-16T00:16:37.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:37.584+0000 7f0b23491640 1 -- 172.21.15.33:0/1701031313 wait complete. 2024-09-16T00:16:37.642 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:37 smithi033 ceph-mon[31243]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-16T00:16:37.642 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:37 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3383512470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-16T00:16:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:37 smithi083 ceph-mon[35242]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-16T00:16:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:37 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3383512470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-16T00:16:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:37 smithi060 ceph-mon[35185]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-16T00:16:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:37 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3383512470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-16T00:16:38.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.052+0000 7f68acd9e640 1 -- 172.21.15.33:0/1639939642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68a81008d0 msgr2=0x7f68a8100d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:38.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.052+0000 7f68acd9e640 1 --2- 172.21.15.33:0/1639939642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68a81008d0 0x7f68a8100d30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f689c009930 tx=0x7f689c02a420 comp rx=0 tx=0).stop 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.052+0000 7f68acd9e640 1 -- 172.21.15.33:0/1639939642 shutdown_connections 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.052+0000 7f68acd9e640 1 --2- 172.21.15.33:0/1639939642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68a8101270 0x7f68a810d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.052+0000 7f68acd9e640 1 --2- 172.21.15.33:0/1639939642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68a81008d0 0x7f68a8100d30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.052+0000 7f68acd9e640 1 --2- 172.21.15.33:0/1639939642 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68a81068d0 0x7f68a8106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.052+0000 7f68acd9e640 1 -- 172.21.15.33:0/1639939642 >> 172.21.15.33:0/1639939642 conn(0x7f68a80fc630 msgr2=0x7f68a80fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.052+0000 7f68acd9e640 1 -- 172.21.15.33:0/1639939642 shutdown_connections 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.052+0000 7f68acd9e640 1 -- 172.21.15.33:0/1639939642 wait complete. 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.053+0000 7f68acd9e640 1 Processor -- start 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.053+0000 7f68acd9e640 1 -- start start 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.053+0000 7f68acd9e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68a81008d0 0x7f68a8198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.053+0000 7f68acd9e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68a8101270 0x7f68a8198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.053+0000 7f68acd9e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68a81068d0 0x7f68a819cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:38.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.053+0000 7f68acd9e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f68a8110cf0 con 0x7f68a8101270 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.053+0000 7f68acd9e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f68a8110b70 con 0x7f68a81008d0 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.053+0000 7f68acd9e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f68a8110e70 con 0x7f68a81068d0 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68a8101270 0x7f68a8198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68a8101270 0x7f68a8198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52040/0 (socket says 172.21.15.33:52040) 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a6ffd640 1 -- 172.21.15.33:0/2281930734 learned_addr learned my addr 172.21.15.33:0/2281930734 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a77fe640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68a81008d0 0x7f68a8198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a7fff640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68a81068d0 0x7f68a819cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a6ffd640 1 -- 172.21.15.33:0/2281930734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68a81068d0 msgr2=0x7f68a819cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a6ffd640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68a81068d0 0x7f68a819cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a6ffd640 1 -- 172.21.15.33:0/2281930734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68a81008d0 msgr2=0x7f68a8198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a6ffd640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68a81008d0 0x7f68a8198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a6ffd640 1 -- 172.21.15.33:0/2281930734 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a819d2e0 con 0x7f68a8101270 2024-09-16T00:16:38.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a7fff640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68a81068d0 0x7f68a819cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a6ffd640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68a8101270 0x7f68a8198870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f689c009a50 tx=0x7f689c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a4ff9640 1 -- 172.21.15.33:0/2281930734 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c03f030 con 0x7f68a8101270 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a4ff9640 1 -- 172.21.15.33:0/2281930734 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f689c003ce0 con 0x7f68a8101270 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68acd9e640 1 -- 172.21.15.33:0/2281930734 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a819d570 con 0x7f68a8101270 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68a4ff9640 1 -- 172.21.15.33:0/2281930734 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c035070 con 0x7f68a8101270 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.054+0000 7f68acd9e640 1 -- 172.21.15.33:0/2281930734 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a81a4eb0 con 0x7f68a8101270 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.055+0000 7f68acd9e640 1 -- 172.21.15.33:0/2281930734 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68a8100d30 con 0x7f68a8101270 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.057+0000 7f68a4ff9640 1 -- 172.21.15.33:0/2281930734 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f689c042030 con 0x7f68a8101270 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.057+0000 7f68a4ff9640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68900a32d0 0x7f68900a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.057+0000 7f68a4ff9640 1 -- 172.21.15.33:0/2281930734 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f689c0f07f0 con 0x7f68a8101270 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.057+0000 7f68a77fe640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68900a32d0 0x7f68900a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.058+0000 7f68a77fe640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68900a32d0 0x7f68900a5790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f68980058c0 tx=0x7f689800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:38.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.060+0000 7f68a4ff9640 1 -- 172.21.15.33:0/2281930734 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f689c0bd050 con 0x7f68a8101270 2024-09-16T00:16:38.128 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:38.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.242+0000 7f68acd9e640 1 -- 172.21.15.33:0/2281930734 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f68a8109e70 con 0x7f68a8101270 2024-09-16T00:16:38.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.242+0000 7f68a4ff9640 1 -- 172.21.15.33:0/2281930734 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f689c0bc950 con 0x7f68a8101270 2024-09-16T00:16:38.243 INFO:teuthology.orchestra.run.smithi033.stdout:180388626490 2024-09-16T00:16:38.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.245+0000 7f68acd9e640 1 -- 172.21.15.33:0/2281930734 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68900a32d0 msgr2=0x7f68900a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:38.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.245+0000 7f68acd9e640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68900a32d0 0x7f68900a5790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f68980058c0 tx=0x7f689800a2d0 comp rx=0 tx=0).stop 2024-09-16T00:16:38.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.245+0000 7f68acd9e640 1 -- 172.21.15.33:0/2281930734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68a8101270 msgr2=0x7f68a8198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:38.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.245+0000 7f68acd9e640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68a8101270 0x7f68a8198870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f689c009a50 tx=0x7f689c004270 comp rx=0 tx=0).stop 2024-09-16T00:16:38.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.245+0000 7f68acd9e640 1 -- 172.21.15.33:0/2281930734 shutdown_connections 2024-09-16T00:16:38.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.245+0000 7f68acd9e640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68900a32d0 0x7f68900a5790 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:38.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.245+0000 7f68acd9e640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68a81068d0 0x7f68a819cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:38.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.245+0000 7f68acd9e640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68a8101270 0x7f68a8198870 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:38.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.245+0000 7f68acd9e640 1 --2- 172.21.15.33:0/2281930734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68a81008d0 0x7f68a8198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:38.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.245+0000 7f68acd9e640 1 -- 172.21.15.33:0/2281930734 >> 172.21.15.33:0/2281930734 conn(0x7f68a80fc630 msgr2=0x7f68a806aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:38.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.245+0000 7f68acd9e640 1 -- 172.21.15.33:0/2281930734 shutdown_connections 2024-09-16T00:16:38.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:38.245+0000 7f68acd9e640 1 -- 172.21.15.33:0/2281930734 wait complete. 2024-09-16T00:16:38.357 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:38.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:38 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1701031313' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-16T00:16:38.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:38 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2281930734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-16T00:16:38.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:38 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1701031313' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-16T00:16:38.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:38 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2281930734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-16T00:16:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:38 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1701031313' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-16T00:16:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:38 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2281930734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-16T00:16:39.161 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626487 got 180388626490 for osd.6 2024-09-16T00:16:39.161 DEBUG:teuthology.parallel:result is None 2024-09-16T00:16:39.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:39 smithi033 ceph-mon[31243]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:16:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:39 smithi083 ceph-mon[35242]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:16:39.969 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972392 got 266287972394 for osd.9 2024-09-16T00:16:39.969 DEBUG:teuthology.parallel:result is None 2024-09-16T00:16:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:39 smithi060 ceph-mon[35185]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:16:40.415 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579933 got 317827579935 for osd.11 2024-09-16T00:16:40.415 DEBUG:teuthology.parallel:result is None 2024-09-16T00:16:41.247 INFO:tasks.cephadm.ceph_manager.ceph:need seq 51539607633 got 51539607635 for osd.1 2024-09-16T00:16:41.248 DEBUG:teuthology.parallel:result is None 2024-09-16T00:16:41.648 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808864 got 287762808868 for osd.10 2024-09-16T00:16:41.648 DEBUG:teuthology.parallel:result is None 2024-09-16T00:16:41.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:41 smithi033 ceph-mon[31243]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:41 smithi083 ceph-mon[35242]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:41 smithi060 ceph-mon[35185]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.056+0000 7f0260de0640 1 -- 172.21.15.33:0/1394485172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f025c106f10 msgr2=0x7f025c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.056+0000 7f0260de0640 1 --2- 172.21.15.33:0/1394485172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f025c106f10 0x7f025c10ea90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0254009ca0 tx=0x7f025402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.056+0000 7f0260de0640 1 -- 172.21.15.33:0/1394485172 shutdown_connections 2024-09-16T00:16:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.056+0000 7f0260de0640 1 --2- 172.21.15.33:0/1394485172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f025c106f10 0x7f025c10ea90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.056+0000 7f0260de0640 1 --2- 172.21.15.33:0/1394485172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f025c102790 0x7f025c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.056+0000 7f0260de0640 1 --2- 172.21.15.33:0/1394485172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f025c101de0 0x7f025c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.056+0000 7f0260de0640 1 -- 172.21.15.33:0/1394485172 >> 172.21.15.33:0/1394485172 conn(0x7f025c0fd8c0 msgr2=0x7f025c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.056+0000 7f0260de0640 1 -- 172.21.15.33:0/1394485172 shutdown_connections 2024-09-16T00:16:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.056+0000 7f0260de0640 1 -- 172.21.15.33:0/1394485172 wait complete. 2024-09-16T00:16:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.057+0000 7f0260de0640 1 Processor -- start 2024-09-16T00:16:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.057+0000 7f0260de0640 1 -- start start 2024-09-16T00:16:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.057+0000 7f0260de0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f025c101de0 0x7f025c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.057+0000 7f0260de0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f025c102790 0x7f025c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.057+0000 7f0260de0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f025c106f10 0x7f025c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.057+0000 7f0260de0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f025c111f90 con 0x7f025c102790 2024-09-16T00:16:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.057+0000 7f0260de0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f025c111e10 con 0x7f025c101de0 2024-09-16T00:16:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.057+0000 7f0260de0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f025c112110 con 0x7f025c106f10 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.057+0000 7f0259d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f025c102790 0x7f025c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.057+0000 7f0259d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f025c102790 0x7f025c198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40012/0 (socket says 172.21.15.33:40012) 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.057+0000 7f0259d9b640 1 -- 172.21.15.33:0/2376795855 learned_addr learned my addr 172.21.15.33:0/2376795855 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f025a59c640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f025c101de0 0x7f025c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f025ad9d640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f025c106f10 0x7f025c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f0259d9b640 1 -- 172.21.15.33:0/2376795855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f025c106f10 msgr2=0x7f025c19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f0259d9b640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f025c106f10 0x7f025c19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f0259d9b640 1 -- 172.21.15.33:0/2376795855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f025c101de0 msgr2=0x7f025c198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f0259d9b640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f025c101de0 0x7f025c198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f0259d9b640 1 -- 172.21.15.33:0/2376795855 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025c19d3c0 con 0x7f025c102790 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f025ad9d640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f025c106f10 0x7f025c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f0259d9b640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f025c102790 0x7f025c198950 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f025000c920 tx=0x7f025000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f024b7fe640 1 -- 172.21.15.33:0/2376795855 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0250007c70 con 0x7f025c102790 2024-09-16T00:16:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f0260de0640 1 -- 172.21.15.33:0/2376795855 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f025c19d6b0 con 0x7f025c102790 2024-09-16T00:16:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f024b7fe640 1 -- 172.21.15.33:0/2376795855 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0250007e10 con 0x7f025c102790 2024-09-16T00:16:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f024b7fe640 1 -- 172.21.15.33:0/2376795855 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0250018890 con 0x7f025c102790 2024-09-16T00:16:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.058+0000 7f0260de0640 1 -- 172.21.15.33:0/2376795855 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f025c1a4f90 con 0x7f025c102790 2024-09-16T00:16:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.059+0000 7f0260de0640 1 -- 172.21.15.33:0/2376795855 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f025c1021c0 con 0x7f025c102790 2024-09-16T00:16:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.060+0000 7f024b7fe640 1 -- 172.21.15.33:0/2376795855 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0250018a70 con 0x7f025c102790 2024-09-16T00:16:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.061+0000 7f024b7fe640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f02440a32d0 0x7f02440a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.061+0000 7f025a59c640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f02440a32d0 0x7f02440a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.062+0000 7f024b7fe640 1 -- 172.21.15.33:0/2376795855 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f02500d0060 con 0x7f025c102790 2024-09-16T00:16:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.062+0000 7f025a59c640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f02440a32d0 0x7f02440a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f024c009890 tx=0x7f024c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.066+0000 7f024b7fe640 1 -- 172.21.15.33:0/2376795855 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0250014030 con 0x7f025c102790 2024-09-16T00:16:42.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.242+0000 7f0260de0640 1 -- 172.21.15.33:0/2376795855 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f025c1a51c0 con 0x7f025c102790 2024-09-16T00:16:42.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.243+0000 7f024b7fe640 1 -- 172.21.15.33:0/2376795855 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f025000de20 con 0x7f025c102790 2024-09-16T00:16:42.243 INFO:teuthology.orchestra.run.smithi033.stdout:150323855423 2024-09-16T00:16:42.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.245+0000 7f0260de0640 1 -- 172.21.15.33:0/2376795855 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f02440a32d0 msgr2=0x7f02440a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:42.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.245+0000 7f0260de0640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f02440a32d0 0x7f02440a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f024c009890 tx=0x7f024c008040 comp rx=0 tx=0).stop 2024-09-16T00:16:42.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.245+0000 7f0260de0640 1 -- 172.21.15.33:0/2376795855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f025c102790 msgr2=0x7f025c198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:42.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.245+0000 7f0260de0640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f025c102790 0x7f025c198950 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f025000c920 tx=0x7f025000cde0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.245+0000 7f0260de0640 1 -- 172.21.15.33:0/2376795855 shutdown_connections 2024-09-16T00:16:42.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.245+0000 7f0260de0640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f02440a32d0 0x7f02440a5790 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.245+0000 7f0260de0640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f025c106f10 0x7f025c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.245+0000 7f0260de0640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f025c102790 0x7f025c198950 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.245+0000 7f0260de0640 1 --2- 172.21.15.33:0/2376795855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f025c101de0 0x7f025c198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.245+0000 7f0260de0640 1 -- 172.21.15.33:0/2376795855 >> 172.21.15.33:0/2376795855 conn(0x7f025c0fd8c0 msgr2=0x7f025c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:42.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.245+0000 7f0260de0640 1 -- 172.21.15.33:0/2376795855 shutdown_connections 2024-09-16T00:16:42.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.245+0000 7f0260de0640 1 -- 172.21.15.33:0/2376795855 wait complete. 2024-09-16T00:16:42.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.753+0000 7fa641807640 1 -- 172.21.15.33:0/1744970276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa63c102790 msgr2=0x7fa63c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:42.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.753+0000 7fa641807640 1 --2- 172.21.15.33:0/1744970276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa63c102790 0x7fa63c1067e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa630009900 tx=0x7fa63002a410 comp rx=0 tx=0).stop 2024-09-16T00:16:42.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.753+0000 7fa641807640 1 -- 172.21.15.33:0/1744970276 shutdown_connections 2024-09-16T00:16:42.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.753+0000 7fa641807640 1 --2- 172.21.15.33:0/1744970276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa63c106f10 0x7fa63c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.753+0000 7fa641807640 1 --2- 172.21.15.33:0/1744970276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa63c102790 0x7fa63c1067e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.753+0000 7fa641807640 1 --2- 172.21.15.33:0/1744970276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa63c101de0 0x7fa63c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.753+0000 7fa641807640 1 -- 172.21.15.33:0/1744970276 >> 172.21.15.33:0/1744970276 conn(0x7fa63c0fd8c0 msgr2=0x7fa63c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.754+0000 7fa641807640 1 -- 172.21.15.33:0/1744970276 shutdown_connections 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.754+0000 7fa641807640 1 -- 172.21.15.33:0/1744970276 wait complete. 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.754+0000 7fa641807640 1 Processor -- start 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.754+0000 7fa641807640 1 -- start start 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa641807640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa63c101de0 0x7fa63c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa641807640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa63c102790 0x7fa63c198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa641807640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa63c106f10 0x7fa63c19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa641807640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c111f70 con 0x7fa63c106f10 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa641807640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c111df0 con 0x7fa63c102790 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa641807640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c1120f0 con 0x7fa63c101de0 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa63b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa63c106f10 0x7fa63c19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:42.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa63b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa63c106f10 0x7fa63c19cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40034/0 (socket says 172.21.15.33:40034) 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa63a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa63c102790 0x7fa63c198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa63b7fe640 1 -- 172.21.15.33:0/2328095882 learned_addr learned my addr 172.21.15.33:0/2328095882 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa63affd640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa63c101de0 0x7fa63c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa63b7fe640 1 -- 172.21.15.33:0/2328095882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa63c101de0 msgr2=0x7fa63c1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa63b7fe640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa63c101de0 0x7fa63c1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa63b7fe640 1 -- 172.21.15.33:0/2328095882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa63c102790 msgr2=0x7fa63c198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa63b7fe640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa63c102790 0x7fa63c198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa63b7fe640 1 -- 172.21.15.33:0/2328095882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63c19d380 con 0x7fa63c106f10 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa63b7fe640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa63c106f10 0x7fa63c19cca0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa63400c970 tx=0x7fa63400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa640805640 1 -- 172.21.15.33:0/2328095882 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa634007bb0 con 0x7fa63c106f10 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa640805640 1 -- 172.21.15.33:0/2328095882 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa634007d50 con 0x7fa63c106f10 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa641807640 1 -- 172.21.15.33:0/2328095882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63c19d670 con 0x7fa63c106f10 2024-09-16T00:16:42.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.755+0000 7fa640805640 1 -- 172.21.15.33:0/2328095882 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa634018800 con 0x7fa63c106f10 2024-09-16T00:16:42.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.756+0000 7fa641807640 1 -- 172.21.15.33:0/2328095882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa63c1a4f50 con 0x7fa63c106f10 2024-09-16T00:16:42.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.756+0000 7fa641807640 1 -- 172.21.15.33:0/2328095882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa63c1021c0 con 0x7fa63c106f10 2024-09-16T00:16:42.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.758+0000 7fa640805640 1 -- 172.21.15.33:0/2328095882 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa63400fa50 con 0x7fa63c106f10 2024-09-16T00:16:42.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.758+0000 7fa640805640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa6240a32d0 0x7fa6240a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:42.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.759+0000 7fa640805640 1 -- 172.21.15.33:0/2328095882 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fa6340d0030 con 0x7fa63c106f10 2024-09-16T00:16:42.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.759+0000 7fa63affd640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa6240a32d0 0x7fa6240a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:42.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.759+0000 7fa63affd640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa6240a32d0 0x7fa6240a5790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa62c005c40 tx=0x7fa62c00a390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:42.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.762+0000 7fa640805640 1 -- 172.21.15.33:0/2328095882 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa634014030 con 0x7fa63c106f10 2024-09-16T00:16:42.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:42 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2376795855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-16T00:16:42.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:42 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2376795855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-16T00:16:42.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.937+0000 7fa641807640 1 -- 172.21.15.33:0/2328095882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa63c109bc0 con 0x7fa63c106f10 2024-09-16T00:16:42.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.938+0000 7fa640805640 1 -- 172.21.15.33:0/2328095882 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa63409c890 con 0x7fa63c106f10 2024-09-16T00:16:42.938 INFO:teuthology.orchestra.run.smithi033.stdout:73014444111 2024-09-16T00:16:42.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.940+0000 7fa641807640 1 -- 172.21.15.33:0/2328095882 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa6240a32d0 msgr2=0x7fa6240a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:42.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.940+0000 7fa641807640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa6240a32d0 0x7fa6240a5790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa62c005c40 tx=0x7fa62c00a390 comp rx=0 tx=0).stop 2024-09-16T00:16:42.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.940+0000 7fa641807640 1 -- 172.21.15.33:0/2328095882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa63c106f10 msgr2=0x7fa63c19cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:42.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.940+0000 7fa641807640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa63c106f10 0x7fa63c19cca0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa63400c970 tx=0x7fa63400ce30 comp rx=0 tx=0).stop 2024-09-16T00:16:42.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.940+0000 7fa641807640 1 -- 172.21.15.33:0/2328095882 shutdown_connections 2024-09-16T00:16:42.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.940+0000 7fa641807640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa6240a32d0 0x7fa6240a5790 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.940+0000 7fa641807640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa63c106f10 0x7fa63c19cca0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.940+0000 7fa641807640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa63c102790 0x7fa63c198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.940+0000 7fa641807640 1 --2- 172.21.15.33:0/2328095882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa63c101de0 0x7fa63c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:42.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.940+0000 7fa641807640 1 -- 172.21.15.33:0/2328095882 >> 172.21.15.33:0/2328095882 conn(0x7fa63c0fd8c0 msgr2=0x7fa63c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:42.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.940+0000 7fa641807640 1 -- 172.21.15.33:0/2328095882 shutdown_connections 2024-09-16T00:16:42.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:42.940+0000 7fa641807640 1 -- 172.21.15.33:0/2328095882 wait complete. 2024-09-16T00:16:42.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:42 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2376795855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-16T00:16:43.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.007+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/2505944872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a84102d40 msgr2=0x7f5a84103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:43.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.007+0000 7f5a8c0a6640 1 --2- 172.21.15.33:0/2505944872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a84102d40 0x7f5a84103120 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009900 tx=0x7f5a8002a480 comp rx=0 tx=0).stop 2024-09-16T00:16:43.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.007+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/2505944872 shutdown_connections 2024-09-16T00:16:43.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.007+0000 7f5a8c0a6640 1 --2- 172.21.15.33:0/2505944872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a8410c170 0x7f5a8410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:43.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.007+0000 7f5a8c0a6640 1 --2- 172.21.15.33:0/2505944872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a84103660 0x7f5a8410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:43.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.007+0000 7f5a8c0a6640 1 --2- 172.21.15.33:0/2505944872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a84102d40 0x7f5a84103120 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:43.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.007+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/2505944872 >> 172.21.15.33:0/2505944872 conn(0x7f5a840fc7a0 msgr2=0x7f5a840febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:43.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.007+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/2505944872 shutdown_connections 2024-09-16T00:16:43.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.007+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/2505944872 wait complete. 2024-09-16T00:16:43.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a8c0a6640 1 Processor -- start 2024-09-16T00:16:43.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a8c0a6640 1 -- start start 2024-09-16T00:16:43.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a8c0a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a84102d40 0x7f5a8419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a8c0a6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a84103660 0x7f5a8419ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a8c0a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8410c170 0x7f5a8419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a8c0a6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5a84111fc0 con 0x7f5a84102d40 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a8c0a6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5a84111e40 con 0x7f5a8410c170 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a8c0a6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a84112140 con 0x7f5a84103660 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a89e42640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a84102d40 0x7f5a8419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a89e42640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a84102d40 0x7f5a8419a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40050/0 (socket says 172.21.15.33:40050) 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a89e42640 1 -- 172.21.15.33:0/1786871299 learned_addr learned my addr 172.21.15.33:0/1786871299 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a89641640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a84103660 0x7f5a8419ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a8a643640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8410c170 0x7f5a8419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a89e42640 1 -- 172.21.15.33:0/1786871299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a84103660 msgr2=0x7f5a8419ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.008+0000 7f5a89e42640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a84103660 0x7f5a8419ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:43.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.009+0000 7f5a89e42640 1 -- 172.21.15.33:0/1786871299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8410c170 msgr2=0x7f5a8419ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.009+0000 7f5a89e42640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8410c170 0x7f5a8419ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.009+0000 7f5a89e42640 1 -- 172.21.15.33:0/1786871299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a8419f600 con 0x7f5a84102d40 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.009+0000 7f5a8a643640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8410c170 0x7f5a8419ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.009+0000 7f5a89e42640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a84102d40 0x7f5a8419a650 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5a80004770 tx=0x7f5a80003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.009+0000 7f5a7affd640 1 -- 172.21.15.33:0/1786871299 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8003f070 con 0x7f5a84102d40 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.009+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/1786871299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a8419f890 con 0x7f5a84102d40 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.009+0000 7f5a7affd640 1 -- 172.21.15.33:0/1786871299 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a80031ca0 con 0x7f5a84102d40 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.009+0000 7f5a7affd640 1 -- 172.21.15.33:0/1786871299 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a80035030 con 0x7f5a84102d40 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.009+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/1786871299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a841a71d0 con 0x7f5a84102d40 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.010+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/1786871299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a84103120 con 0x7f5a84102d40 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.011+0000 7f5a7affd640 1 -- 172.21.15.33:0/1786871299 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f5a80042070 con 0x7f5a84102d40 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.012+0000 7f5a7affd640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5a700a32d0 0x7f5a700a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:43.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.013+0000 7f5a89641640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5a700a32d0 0x7f5a700a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:43.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.013+0000 7f5a7affd640 1 -- 172.21.15.33:0/1786871299 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f5a800f0a90 con 0x7f5a84102d40 2024-09-16T00:16:43.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.013+0000 7f5a89641640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5a700a32d0 0x7f5a700a5790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a8410eb20 tx=0x7f5a74005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:43.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.016+0000 7f5a7affd640 1 -- 172.21.15.33:0/1786871299 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5a800bd400 con 0x7f5a84102d40 2024-09-16T00:16:43.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.192+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/1786871299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f5a841a73a0 con 0x7f5a84102d40 2024-09-16T00:16:43.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.192+0000 7f5a7affd640 1 -- 172.21.15.33:0/1786871299 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5a800bcd00 con 0x7f5a84102d40 2024-09-16T00:16:43.193 INFO:teuthology.orchestra.run.smithi033.stdout:133143986245 2024-09-16T00:16:43.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.194+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/1786871299 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5a700a32d0 msgr2=0x7f5a700a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:43.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.194+0000 7f5a8c0a6640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5a700a32d0 0x7f5a700a5790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a8410eb20 tx=0x7f5a74005eb0 comp rx=0 tx=0).stop 2024-09-16T00:16:43.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.194+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/1786871299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a84102d40 msgr2=0x7f5a8419a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:43.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.194+0000 7f5a8c0a6640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a84102d40 0x7f5a8419a650 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5a80004770 tx=0x7f5a80003c10 comp rx=0 tx=0).stop 2024-09-16T00:16:43.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.194+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/1786871299 shutdown_connections 2024-09-16T00:16:43.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.194+0000 7f5a8c0a6640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5a700a32d0 0x7f5a700a5790 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:43.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.194+0000 7f5a8c0a6640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8410c170 0x7f5a8419ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:43.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.194+0000 7f5a8c0a6640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a84103660 0x7f5a8419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:43.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.194+0000 7f5a8c0a6640 1 --2- 172.21.15.33:0/1786871299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a84102d40 0x7f5a8419a650 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:43.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.194+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/1786871299 >> 172.21.15.33:0/1786871299 conn(0x7f5a840fc7a0 msgr2=0x7f5a840fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:43.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.194+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/1786871299 shutdown_connections 2024-09-16T00:16:43.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:43.194+0000 7f5a8c0a6640 1 -- 172.21.15.33:0/1786871299 wait complete. 2024-09-16T00:16:43.702 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:43 smithi033 ceph-mon[31243]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:43.702 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:43 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2328095882' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-16T00:16:43.703 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:43 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1786871299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-16T00:16:43.733 INFO:tasks.cephadm.ceph_manager.ceph:need seq 73014444109 got 73014444111 for osd.2 2024-09-16T00:16:43.734 DEBUG:teuthology.parallel:result is None 2024-09-16T00:16:43.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:43 smithi083 ceph-mon[35242]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:43.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:43 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2328095882' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-16T00:16:43.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:43 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1786871299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-16T00:16:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:43 smithi060 ceph-mon[35185]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:43.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:43 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2328095882' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-16T00:16:43.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:43 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1786871299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-16T00:16:44.434 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855421 got 150323855423 for osd.5 2024-09-16T00:16:44.434 DEBUG:teuthology.parallel:result is None 2024-09-16T00:16:44.923 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986241 got 133143986245 for osd.4 2024-09-16T00:16:44.924 DEBUG:teuthology.parallel:result is None 2024-09-16T00:16:44.924 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-16T00:16:44.924 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T00:16:45.190 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:45.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:45 smithi033 ceph-mon[31243]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:45.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.637+0000 7fc84facd640 1 -- 172.21.15.33:0/3509829294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848102790 msgr2=0x7fc8481067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:45.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.637+0000 7fc84facd640 1 --2- 172.21.15.33:0/3509829294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848102790 0x7fc8481067e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc838009900 tx=0x7fc83802a3f0 comp rx=0 tx=0).stop 2024-09-16T00:16:45.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.637+0000 7fc84facd640 1 -- 172.21.15.33:0/3509829294 shutdown_connections 2024-09-16T00:16:45.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.637+0000 7fc84facd640 1 --2- 172.21.15.33:0/3509829294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc848106f10 0x7fc84810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:45.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.637+0000 7fc84facd640 1 --2- 172.21.15.33:0/3509829294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848102790 0x7fc8481067e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:45.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.637+0000 7fc84facd640 1 --2- 172.21.15.33:0/3509829294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc848101de0 0x7fc8481021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:45.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.637+0000 7fc84facd640 1 -- 172.21.15.33:0/3509829294 >> 172.21.15.33:0/3509829294 conn(0x7fc8480fd8c0 msgr2=0x7fc8480ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:45.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.637+0000 7fc84facd640 1 -- 172.21.15.33:0/3509829294 shutdown_connections 2024-09-16T00:16:45.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.637+0000 7fc84facd640 1 -- 172.21.15.33:0/3509829294 wait complete. 2024-09-16T00:16:45.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84facd640 1 Processor -- start 2024-09-16T00:16:45.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84facd640 1 -- start start 2024-09-16T00:16:45.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84facd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc848101de0 0x7fc848198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:45.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84facd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc848102790 0x7fc848198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:45.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84facd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848106f10 0x7fc84819ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:45.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84facd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc848111fc0 con 0x7fc848106f10 2024-09-16T00:16:45.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84facd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc848111e40 con 0x7fc848101de0 2024-09-16T00:16:45.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84facd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc848112140 con 0x7fc848102790 2024-09-16T00:16:45.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84e06a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848106f10 0x7fc84819ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:45.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84e06a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848106f10 0x7fc84819ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40062/0 (socket says 172.21.15.33:40062) 2024-09-16T00:16:45.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84e06a640 1 -- 172.21.15.33:0/2594925123 learned_addr learned my addr 172.21.15.33:0/2594925123 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:45.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84d869640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc848101de0 0x7fc848198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:45.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84d068640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc848102790 0x7fc848198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:45.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84e06a640 1 -- 172.21.15.33:0/2594925123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc848102790 msgr2=0x7fc848198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:45.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84e06a640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc848102790 0x7fc848198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:45.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84e06a640 1 -- 172.21.15.33:0/2594925123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc848101de0 msgr2=0x7fc848198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:45.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84e06a640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc848101de0 0x7fc848198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:45.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.638+0000 7fc84e06a640 1 -- 172.21.15.33:0/2594925123 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84819d3b0 con 0x7fc848106f10 2024-09-16T00:16:45.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.639+0000 7fc84e06a640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848106f10 0x7fc84819ccd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc84000cce0 tx=0x7fc84000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:45.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.639+0000 7fc83effd640 1 -- 172.21.15.33:0/2594925123 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84000f450 con 0x7fc848106f10 2024-09-16T00:16:45.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.639+0000 7fc83effd640 1 -- 172.21.15.33:0/2594925123 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc84000fad0 con 0x7fc848106f10 2024-09-16T00:16:45.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.639+0000 7fc84facd640 1 -- 172.21.15.33:0/2594925123 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84819d6a0 con 0x7fc848106f10 2024-09-16T00:16:45.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.639+0000 7fc83effd640 1 -- 172.21.15.33:0/2594925123 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840018c10 con 0x7fc848106f10 2024-09-16T00:16:45.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.639+0000 7fc84facd640 1 -- 172.21.15.33:0/2594925123 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8481a4f80 con 0x7fc848106f10 2024-09-16T00:16:45.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.640+0000 7fc84facd640 1 -- 172.21.15.33:0/2594925123 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8481021c0 con 0x7fc848106f10 2024-09-16T00:16:45.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.641+0000 7fc83effd640 1 -- 172.21.15.33:0/2594925123 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc84000f5f0 con 0x7fc848106f10 2024-09-16T00:16:45.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.642+0000 7fc83effd640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc8340a32d0 0x7fc8340a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:45.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.642+0000 7fc83effd640 1 -- 172.21.15.33:0/2594925123 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fc8400d0390 con 0x7fc848106f10 2024-09-16T00:16:45.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.642+0000 7fc84d869640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc8340a32d0 0x7fc8340a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:45.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.643+0000 7fc84d869640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc8340a32d0 0x7fc8340a5790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc8440058c0 tx=0x7fc84400a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:45.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.645+0000 7fc83effd640 1 -- 172.21.15.33:0/2594925123 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc840014030 con 0x7fc848106f10 2024-09-16T00:16:45.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.818+0000 7fc84facd640 1 -- 172.21.15.33:0/2594925123 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc848062630 con 0x7fc8340a32d0 2024-09-16T00:16:45.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.826+0000 7fc83effd640 1 -- 172.21.15.33:0/2594925123 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340640 (secure 0 0 0) 0x7fc848062630 con 0x7fc8340a32d0 2024-09-16T00:16:45.831 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:45.833 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-09-16T00:16:45.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.829+0000 7fc84facd640 1 -- 172.21.15.33:0/2594925123 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc8340a32d0 msgr2=0x7fc8340a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:45.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.829+0000 7fc84facd640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc8340a32d0 0x7fc8340a5790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc8440058c0 tx=0x7fc84400a250 comp rx=0 tx=0).stop 2024-09-16T00:16:45.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.829+0000 7fc84facd640 1 -- 172.21.15.33:0/2594925123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848106f10 msgr2=0x7fc84819ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:45.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.829+0000 7fc84facd640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848106f10 0x7fc84819ccd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc84000cce0 tx=0x7fc84000a620 comp rx=0 tx=0).stop 2024-09-16T00:16:45.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.829+0000 7fc84facd640 1 -- 172.21.15.33:0/2594925123 shutdown_connections 2024-09-16T00:16:45.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.829+0000 7fc84facd640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc8340a32d0 0x7fc8340a5790 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:45.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.829+0000 7fc84facd640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848106f10 0x7fc84819ccd0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:45.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.829+0000 7fc84facd640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc848102790 0x7fc848198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:45.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.830+0000 7fc84facd640 1 --2- 172.21.15.33:0/2594925123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc848101de0 0x7fc848198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:45.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.830+0000 7fc84facd640 1 -- 172.21.15.33:0/2594925123 >> 172.21.15.33:0/2594925123 conn(0x7fc8480fd8c0 msgr2=0x7fc8480febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:45.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.830+0000 7fc84facd640 1 -- 172.21.15.33:0/2594925123 shutdown_connections 2024-09-16T00:16:45.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:45.830+0000 7fc84facd640 1 -- 172.21.15.33:0/2594925123 wait complete. 2024-09-16T00:16:45.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:45 smithi083 ceph-mon[35242]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:45 smithi060 ceph-mon[35185]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:46.676 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-16T00:16:45.658095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657536,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":474,"num_read_kb":343,"num_write":348,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1242843,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386104,"kb_used_data":11832,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555720,"statfs":{"total":1151940427776,"available":1151545057280,"internally_reserved":0,"allocated":12115968,"data_stored":7919102,"data_compressed":400707,"data_compressed_allocated":540672,"data_compressed_original":2613248,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003521"},"pg_stats":[{"pgid":"2.1f","version":"86'9","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.034677+0000","last_change":"2024-09-16T00:16:09.034677+0000","last_active":"2024-09-16T00:16:09.034677+0000","last_peered":"2024-09-16T00:16:09.034677+0000","last_clean":"2024-09-16T00:16:09.034677+0000","last_became_active":"2024-09-16T00:16:09.024755+0000","last_became_peered":"2024-09-16T00:16:09.024755+0000","last_unstale":"2024-09-16T00:16:09.034677+0000","last_undegraded":"2024-09-16T00:16:09.034677+0000","last_fullsized":"2024-09-16T00:16:09.034677+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:09.012210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022356+0000","last_change":"2024-09-16T00:15:06.547094+0000","last_active":"2024-09-16T00:16:09.022356+0000","last_peered":"2024-09-16T00:16:09.022356+0000","last_clean":"2024-09-16T00:16:09.022356+0000","last_became_active":"2024-09-16T00:15:06.542764+0000","last_became_peered":"2024-09-16T00:15:06.542764+0000","last_unstale":"2024-09-16T00:16:09.022356+0000","last_undegraded":"2024-09-16T00:16:09.022356+0000","last_fullsized":"2024-09-16T00:16:09.022356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:04:37.449577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021501+0000","last_change":"2024-09-16T00:15:13.578400+0000","last_active":"2024-09-16T00:16:09.021501+0000","last_peered":"2024-09-16T00:16:09.021501+0000","last_clean":"2024-09-16T00:16:09.021501+0000","last_became_active":"2024-09-16T00:15:13.577787+0000","last_became_peered":"2024-09-16T00:15:13.577787+0000","last_unstale":"2024-09-16T00:16:09.021501+0000","last_undegraded":"2024-09-16T00:16:09.021501+0000","last_fullsized":"2024-09-16T00:16:09.021501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:19:59.194152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022614+0000","last_change":"2024-09-16T00:15:04.373381+0000","last_active":"2024-09-16T00:16:09.022614+0000","last_peered":"2024-09-16T00:16:09.022614+0000","last_clean":"2024-09-16T00:16:09.022614+0000","last_became_active":"2024-09-16T00:15:04.373028+0000","last_became_peered":"2024-09-16T00:15:04.373028+0000","last_unstale":"2024-09-16T00:16:09.022614+0000","last_undegraded":"2024-09-16T00:16:09.022614+0000","last_fullsized":"2024-09-16T00:16:09.022614+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:22.902583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020323+0000","last_change":"2024-09-16T00:15:06.554048+0000","last_active":"2024-09-16T00:16:09.020323+0000","last_peered":"2024-09-16T00:16:09.020323+0000","last_clean":"2024-09-16T00:16:09.020323+0000","last_became_active":"2024-09-16T00:15:06.553044+0000","last_became_peered":"2024-09-16T00:15:06.553044+0000","last_unstale":"2024-09-16T00:16:09.020323+0000","last_undegraded":"2024-09-16T00:16:09.020323+0000","last_fullsized":"2024-09-16T00:16:09.020323+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:22:53.333705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017272+0000","last_change":"2024-09-16T00:15:13.586995+0000","last_active":"2024-09-16T00:16:09.017272+0000","last_peered":"2024-09-16T00:16:09.017272+0000","last_clean":"2024-09-16T00:16:09.017272+0000","last_became_active":"2024-09-16T00:15:13.586509+0000","last_became_peered":"2024-09-16T00:15:13.586509+0000","last_unstale":"2024-09-16T00:16:09.017272+0000","last_undegraded":"2024-09-16T00:16:09.017272+0000","last_fullsized":"2024-09-16T00:16:09.017272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:39.418581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019912+0000","last_change":"2024-09-16T00:15:04.348235+0000","last_active":"2024-09-16T00:16:09.019912+0000","last_peered":"2024-09-16T00:16:09.019912+0000","last_clean":"2024-09-16T00:16:09.019912+0000","last_became_active":"2024-09-16T00:15:04.347518+0000","last_became_peered":"2024-09-16T00:15:04.347518+0000","last_unstale":"2024-09-16T00:16:09.019912+0000","last_undegraded":"2024-09-16T00:16:09.019912+0000","last_fullsized":"2024-09-16T00:16:09.019912+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:24.396911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021590+0000","last_change":"2024-09-16T00:15:06.554067+0000","last_active":"2024-09-16T00:16:09.021590+0000","last_peered":"2024-09-16T00:16:09.021590+0000","last_clean":"2024-09-16T00:16:09.021590+0000","last_became_active":"2024-09-16T00:15:06.553715+0000","last_became_peered":"2024-09-16T00:15:06.553715+0000","last_unstale":"2024-09-16T00:16:09.021590+0000","last_undegraded":"2024-09-16T00:16:09.021590+0000","last_fullsized":"2024-09-16T00:16:09.021590+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:22.008530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016383+0000","last_change":"2024-09-16T00:15:13.586872+0000","last_active":"2024-09-16T00:16:09.016383+0000","last_peered":"2024-09-16T00:16:09.016383+0000","last_clean":"2024-09-16T00:16:09.016383+0000","last_became_active":"2024-09-16T00:15:13.586375+0000","last_became_peered":"2024-09-16T00:15:13.586375+0000","last_unstale":"2024-09-16T00:16:09.016383+0000","last_undegraded":"2024-09-16T00:16:09.016383+0000","last_fullsized":"2024-09-16T00:16:09.016383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:43.668827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020333+0000","last_change":"2024-09-16T00:15:04.347964+0000","last_active":"2024-09-16T00:16:09.020333+0000","last_peered":"2024-09-16T00:16:09.020333+0000","last_clean":"2024-09-16T00:16:09.020333+0000","last_became_active":"2024-09-16T00:15:04.347588+0000","last_became_peered":"2024-09-16T00:15:04.347588+0000","last_unstale":"2024-09-16T00:16:09.020333+0000","last_undegraded":"2024-09-16T00:16:09.020333+0000","last_fullsized":"2024-09-16T00:16:09.020333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:14.199287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030005+0000","last_change":"2024-09-16T00:15:06.544918+0000","last_active":"2024-09-16T00:16:09.030005+0000","last_peered":"2024-09-16T00:16:09.030005+0000","last_clean":"2024-09-16T00:16:09.030005+0000","last_became_active":"2024-09-16T00:15:06.544579+0000","last_became_peered":"2024-09-16T00:15:06.544579+0000","last_unstale":"2024-09-16T00:16:09.030005+0000","last_undegraded":"2024-09-16T00:16:09.030005+0000","last_fullsized":"2024-09-16T00:16:09.030005+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:06:37.475644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015781+0000","last_change":"2024-09-16T00:15:13.581488+0000","last_active":"2024-09-16T00:16:09.015781+0000","last_peered":"2024-09-16T00:16:09.015781+0000","last_clean":"2024-09-16T00:16:09.015781+0000","last_became_active":"2024-09-16T00:15:13.581164+0000","last_became_peered":"2024-09-16T00:15:13.581164+0000","last_unstale":"2024-09-16T00:16:09.015781+0000","last_undegraded":"2024-09-16T00:16:09.015781+0000","last_fullsized":"2024-09-16T00:16:09.015781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:05.061592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015354+0000","last_change":"2024-09-16T00:15:06.551982+0000","last_active":"2024-09-16T00:16:09.015354+0000","last_peered":"2024-09-16T00:16:09.015354+0000","last_clean":"2024-09-16T00:16:09.015354+0000","last_became_active":"2024-09-16T00:15:06.551663+0000","last_became_peered":"2024-09-16T00:15:06.551663+0000","last_unstale":"2024-09-16T00:16:09.015354+0000","last_undegraded":"2024-09-16T00:16:09.015354+0000","last_fullsized":"2024-09-16T00:16:09.015354+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:11.928703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019021+0000","last_change":"2024-09-16T00:15:04.352237+0000","last_active":"2024-09-16T00:16:09.019021+0000","last_peered":"2024-09-16T00:16:09.019021+0000","last_clean":"2024-09-16T00:16:09.019021+0000","last_became_active":"2024-09-16T00:15:04.351884+0000","last_became_peered":"2024-09-16T00:15:04.351884+0000","last_unstale":"2024-09-16T00:16:09.019021+0000","last_undegraded":"2024-09-16T00:16:09.019021+0000","last_fullsized":"2024-09-16T00:16:09.019021+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:32.932020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016008+0000","last_change":"2024-09-16T00:15:13.638917+0000","last_active":"2024-09-16T00:16:09.016008+0000","last_peered":"2024-09-16T00:16:09.016008+0000","last_clean":"2024-09-16T00:16:09.016008+0000","last_became_active":"2024-09-16T00:15:13.638401+0000","last_became_peered":"2024-09-16T00:15:13.638401+0000","last_unstale":"2024-09-16T00:16:09.016008+0000","last_undegraded":"2024-09-16T00:16:09.016008+0000","last_fullsized":"2024-09-16T00:16:09.016008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:38.082407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022193+0000","last_change":"2024-09-16T00:15:06.547909+0000","last_active":"2024-09-16T00:16:09.022193+0000","last_peered":"2024-09-16T00:16:09.022193+0000","last_clean":"2024-09-16T00:16:09.022193+0000","last_became_active":"2024-09-16T00:15:06.546814+0000","last_became_peered":"2024-09-16T00:15:06.546814+0000","last_unstale":"2024-09-16T00:16:09.022193+0000","last_undegraded":"2024-09-16T00:16:09.022193+0000","last_fullsized":"2024-09-16T00:16:09.022193+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:46.419406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.028228+0000","last_change":"2024-09-16T00:16:09.027691+0000","last_active":"2024-09-16T00:16:09.028228+0000","last_peered":"2024-09-16T00:16:09.028228+0000","last_clean":"2024-09-16T00:16:09.028228+0000","last_became_active":"2024-09-16T00:16:09.023204+0000","last_became_peered":"2024-09-16T00:16:09.023204+0000","last_unstale":"2024-09-16T00:16:09.028228+0000","last_undegraded":"2024-09-16T00:16:09.028228+0000","last_fullsized":"2024-09-16T00:16:09.028228+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:44.647576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016601+0000","last_change":"2024-09-16T00:15:13.588345+0000","last_active":"2024-09-16T00:16:09.016601+0000","last_peered":"2024-09-16T00:16:09.016601+0000","last_clean":"2024-09-16T00:16:09.016601+0000","last_became_active":"2024-09-16T00:15:13.587829+0000","last_became_peered":"2024-09-16T00:15:13.587829+0000","last_unstale":"2024-09-16T00:16:09.016601+0000","last_undegraded":"2024-09-16T00:16:09.016601+0000","last_fullsized":"2024-09-16T00:16:09.016601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:04.261031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018874+0000","last_change":"2024-09-16T00:15:06.545485+0000","last_active":"2024-09-16T00:16:09.018874+0000","last_peered":"2024-09-16T00:16:09.018874+0000","last_clean":"2024-09-16T00:16:09.018874+0000","last_became_active":"2024-09-16T00:15:06.545178+0000","last_became_peered":"2024-09-16T00:15:06.545178+0000","last_unstale":"2024-09-16T00:16:09.018874+0000","last_undegraded":"2024-09-16T00:16:09.018874+0000","last_fullsized":"2024-09-16T00:16:09.018874+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:26.923469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022534+0000","last_change":"2024-09-16T00:15:04.340546+0000","last_active":"2024-09-16T00:16:09.022534+0000","last_peered":"2024-09-16T00:16:09.022534+0000","last_clean":"2024-09-16T00:16:09.022534+0000","last_became_active":"2024-09-16T00:15:04.340173+0000","last_became_peered":"2024-09-16T00:15:04.340173+0000","last_unstale":"2024-09-16T00:16:09.022534+0000","last_undegraded":"2024-09-16T00:16:09.022534+0000","last_fullsized":"2024-09-16T00:16:09.022534+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:35:01.134433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.013959+0000","last_change":"2024-09-16T00:15:13.589963+0000","last_active":"2024-09-16T00:16:09.013959+0000","last_peered":"2024-09-16T00:16:09.013959+0000","last_clean":"2024-09-16T00:16:09.013959+0000","last_became_active":"2024-09-16T00:15:13.589615+0000","last_became_peered":"2024-09-16T00:15:13.589615+0000","last_unstale":"2024-09-16T00:16:09.013959+0000","last_undegraded":"2024-09-16T00:16:09.013959+0000","last_fullsized":"2024-09-16T00:16:09.013959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:37.197765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019152+0000","last_change":"2024-09-16T00:15:06.544374+0000","last_active":"2024-09-16T00:16:09.019152+0000","last_peered":"2024-09-16T00:16:09.019152+0000","last_clean":"2024-09-16T00:16:09.019152+0000","last_became_active":"2024-09-16T00:15:06.544027+0000","last_became_peered":"2024-09-16T00:15:06.544027+0000","last_unstale":"2024-09-16T00:16:09.019152+0000","last_undegraded":"2024-09-16T00:16:09.019152+0000","last_fullsized":"2024-09-16T00:16:09.019152+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021199+0000","last_change":"2024-09-16T00:15:04.374209+0000","last_active":"2024-09-16T00:16:09.021199+0000","last_peered":"2024-09-16T00:16:09.021199+0000","last_clean":"2024-09-16T00:16:09.021199+0000","last_became_active":"2024-09-16T00:15:04.373814+0000","last_became_peered":"2024-09-16T00:15:04.373814+0000","last_unstale":"2024-09-16T00:16:09.021199+0000","last_undegraded":"2024-09-16T00:16:09.021199+0000","last_fullsized":"2024-09-16T00:16:09.021199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:13.516161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016200+0000","last_change":"2024-09-16T00:15:13.639210+0000","last_active":"2024-09-16T00:16:09.016200+0000","last_peered":"2024-09-16T00:16:09.016200+0000","last_clean":"2024-09-16T00:16:09.016200+0000","last_became_active":"2024-09-16T00:15:13.638719+0000","last_became_peered":"2024-09-16T00:15:13.638719+0000","last_unstale":"2024-09-16T00:16:09.016200+0000","last_undegraded":"2024-09-16T00:16:09.016200+0000","last_fullsized":"2024-09-16T00:16:09.016200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:26.130778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020290+0000","last_change":"2024-09-16T00:15:06.553108+0000","last_active":"2024-09-16T00:16:09.020290+0000","last_peered":"2024-09-16T00:16:09.020290+0000","last_clean":"2024-09-16T00:16:09.020290+0000","last_became_active":"2024-09-16T00:15:06.552817+0000","last_became_peered":"2024-09-16T00:15:06.552817+0000","last_unstale":"2024-09-16T00:16:09.020290+0000","last_undegraded":"2024-09-16T00:16:09.020290+0000","last_fullsized":"2024-09-16T00:16:09.020290+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:06.821337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016843+0000","last_change":"2024-09-16T00:15:04.644185+0000","last_active":"2024-09-16T00:16:09.016843+0000","last_peered":"2024-09-16T00:16:09.016843+0000","last_clean":"2024-09-16T00:16:09.016843+0000","last_became_active":"2024-09-16T00:15:04.643887+0000","last_became_peered":"2024-09-16T00:15:04.643887+0000","last_unstale":"2024-09-16T00:16:09.016843+0000","last_undegraded":"2024-09-16T00:16:09.016843+0000","last_fullsized":"2024-09-16T00:16:09.016843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:44.279272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021730+0000","last_change":"2024-09-16T00:15:13.574761+0000","last_active":"2024-09-16T00:16:09.021730+0000","last_peered":"2024-09-16T00:16:09.021730+0000","last_clean":"2024-09-16T00:16:09.021730+0000","last_became_active":"2024-09-16T00:15:13.574347+0000","last_became_peered":"2024-09-16T00:15:13.574347+0000","last_unstale":"2024-09-16T00:16:09.021730+0000","last_undegraded":"2024-09-16T00:16:09.021730+0000","last_fullsized":"2024-09-16T00:16:09.021730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:22.040121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022609+0000","last_change":"2024-09-16T00:15:06.550338+0000","last_active":"2024-09-16T00:16:09.022609+0000","last_peered":"2024-09-16T00:16:09.022609+0000","last_clean":"2024-09-16T00:16:09.022609+0000","last_became_active":"2024-09-16T00:15:06.549868+0000","last_became_peered":"2024-09-16T00:15:06.549868+0000","last_unstale":"2024-09-16T00:16:09.022609+0000","last_undegraded":"2024-09-16T00:16:09.022609+0000","last_fullsized":"2024-09-16T00:16:09.022609+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:26.643338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'10","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.014079+0000","last_change":"2024-09-16T00:15:04.342306+0000","last_active":"2024-09-16T00:16:09.014079+0000","last_peered":"2024-09-16T00:16:09.014079+0000","last_clean":"2024-09-16T00:16:09.014079+0000","last_became_active":"2024-09-16T00:15:04.341964+0000","last_became_peered":"2024-09-16T00:15:04.341964+0000","last_unstale":"2024-09-16T00:16:09.014079+0000","last_undegraded":"2024-09-16T00:16:09.014079+0000","last_fullsized":"2024-09-16T00:16:09.014079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:14.137199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021778+0000","last_change":"2024-09-16T00:15:13.638246+0000","last_active":"2024-09-16T00:16:09.021778+0000","last_peered":"2024-09-16T00:16:09.021778+0000","last_clean":"2024-09-16T00:16:09.021778+0000","last_became_active":"2024-09-16T00:15:13.637816+0000","last_became_peered":"2024-09-16T00:15:13.637816+0000","last_unstale":"2024-09-16T00:16:09.021778+0000","last_undegraded":"2024-09-16T00:16:09.021778+0000","last_fullsized":"2024-09-16T00:16:09.021778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:27.331156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019970+0000","last_change":"2024-09-16T00:15:06.544057+0000","last_active":"2024-09-16T00:16:09.019970+0000","last_peered":"2024-09-16T00:16:09.019970+0000","last_clean":"2024-09-16T00:16:09.019970+0000","last_became_active":"2024-09-16T00:15:06.543244+0000","last_became_peered":"2024-09-16T00:15:06.543244+0000","last_unstale":"2024-09-16T00:16:09.019970+0000","last_undegraded":"2024-09-16T00:16:09.019970+0000","last_fullsized":"2024-09-16T00:16:09.019970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:42:28.950014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020105+0000","last_change":"2024-09-16T00:15:04.376419+0000","last_active":"2024-09-16T00:16:09.020105+0000","last_peered":"2024-09-16T00:16:09.020105+0000","last_clean":"2024-09-16T00:16:09.020105+0000","last_became_active":"2024-09-16T00:15:04.375976+0000","last_became_peered":"2024-09-16T00:15:04.375976+0000","last_unstale":"2024-09-16T00:16:09.020105+0000","last_undegraded":"2024-09-16T00:16:09.020105+0000","last_fullsized":"2024-09-16T00:16:09.020105+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:33:38.652109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018162+0000","last_change":"2024-09-16T00:15:13.584306+0000","last_active":"2024-09-16T00:16:09.018162+0000","last_peered":"2024-09-16T00:16:09.018162+0000","last_clean":"2024-09-16T00:16:09.018162+0000","last_became_active":"2024-09-16T00:15:13.583840+0000","last_became_peered":"2024-09-16T00:15:13.583840+0000","last_unstale":"2024-09-16T00:16:09.018162+0000","last_undegraded":"2024-09-16T00:16:09.018162+0000","last_fullsized":"2024-09-16T00:16:09.018162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:36.660458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017267+0000","last_change":"2024-09-16T00:15:06.538902+0000","last_active":"2024-09-16T00:16:09.017267+0000","last_peered":"2024-09-16T00:16:09.017267+0000","last_clean":"2024-09-16T00:16:09.017267+0000","last_became_active":"2024-09-16T00:15:06.538537+0000","last_became_peered":"2024-09-16T00:15:06.538537+0000","last_unstale":"2024-09-16T00:16:09.017267+0000","last_undegraded":"2024-09-16T00:16:09.017267+0000","last_fullsized":"2024-09-16T00:16:09.017267+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:38.411711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.013859+0000","last_change":"2024-09-16T00:15:04.374762+0000","last_active":"2024-09-16T00:16:09.013859+0000","last_peered":"2024-09-16T00:16:09.013859+0000","last_clean":"2024-09-16T00:16:09.013859+0000","last_became_active":"2024-09-16T00:15:04.374394+0000","last_became_peered":"2024-09-16T00:15:04.374394+0000","last_unstale":"2024-09-16T00:16:09.013859+0000","last_undegraded":"2024-09-16T00:16:09.013859+0000","last_fullsized":"2024-09-16T00:16:09.013859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:47.075021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018397+0000","last_change":"2024-09-16T00:15:13.638306+0000","last_active":"2024-09-16T00:16:09.018397+0000","last_peered":"2024-09-16T00:16:09.018397+0000","last_clean":"2024-09-16T00:16:09.018397+0000","last_became_active":"2024-09-16T00:15:13.637931+0000","last_became_peered":"2024-09-16T00:15:13.637931+0000","last_unstale":"2024-09-16T00:16:09.018397+0000","last_undegraded":"2024-09-16T00:16:09.018397+0000","last_fullsized":"2024-09-16T00:16:09.018397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:41.385174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017001+0000","last_change":"2024-09-16T00:15:06.548209+0000","last_active":"2024-09-16T00:16:09.017001+0000","last_peered":"2024-09-16T00:16:09.017001+0000","last_clean":"2024-09-16T00:16:09.017001+0000","last_became_active":"2024-09-16T00:15:06.547846+0000","last_became_peered":"2024-09-16T00:15:06.547846+0000","last_unstale":"2024-09-16T00:16:09.017001+0000","last_undegraded":"2024-09-16T00:16:09.017001+0000","last_fullsized":"2024-09-16T00:16:09.017001+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:15.821847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.024470+0000","last_change":"2024-09-16T00:16:09.023918+0000","last_active":"2024-09-16T00:16:09.024470+0000","last_peered":"2024-09-16T00:16:09.024470+0000","last_clean":"2024-09-16T00:16:09.024470+0000","last_became_active":"2024-09-16T00:16:09.023230+0000","last_became_peered":"2024-09-16T00:16:09.023230+0000","last_unstale":"2024-09-16T00:16:09.024470+0000","last_undegraded":"2024-09-16T00:16:09.024470+0000","last_fullsized":"2024-09-16T00:16:09.024470+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:31.527863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019672+0000","last_change":"2024-09-16T00:15:13.579387+0000","last_active":"2024-09-16T00:16:09.019672+0000","last_peered":"2024-09-16T00:16:09.019672+0000","last_clean":"2024-09-16T00:16:09.019672+0000","last_became_active":"2024-09-16T00:15:13.578996+0000","last_became_peered":"2024-09-16T00:15:13.578996+0000","last_unstale":"2024-09-16T00:16:09.019672+0000","last_undegraded":"2024-09-16T00:16:09.019672+0000","last_fullsized":"2024-09-16T00:16:09.019672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:51.703060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.029836+0000","last_change":"2024-09-16T00:15:06.540396+0000","last_active":"2024-09-16T00:16:09.029836+0000","last_peered":"2024-09-16T00:16:09.029836+0000","last_clean":"2024-09-16T00:16:09.029836+0000","last_became_active":"2024-09-16T00:15:06.540084+0000","last_became_peered":"2024-09-16T00:15:06.540084+0000","last_unstale":"2024-09-16T00:16:09.029836+0000","last_undegraded":"2024-09-16T00:16:09.029836+0000","last_fullsized":"2024-09-16T00:16:09.029836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:46.477143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":93,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:43.254034+0000","last_change":"2024-09-16T00:15:04.373896+0000","last_active":"2024-09-16T00:16:43.254034+0000","last_peered":"2024-09-16T00:16:43.254034+0000","last_clean":"2024-09-16T00:16:43.254034+0000","last_became_active":"2024-09-16T00:15:04.373493+0000","last_became_peered":"2024-09-16T00:15:04.373493+0000","last_unstale":"2024-09-16T00:16:43.254034+0000","last_undegraded":"2024-09-16T00:16:43.254034+0000","last_fullsized":"2024-09-16T00:16:43.254034+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:17:10.767267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":55,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019826+0000","last_change":"2024-09-16T00:15:13.638575+0000","last_active":"2024-09-16T00:16:09.019826+0000","last_peered":"2024-09-16T00:16:09.019826+0000","last_clean":"2024-09-16T00:16:09.019826+0000","last_became_active":"2024-09-16T00:15:13.638144+0000","last_became_peered":"2024-09-16T00:15:13.638144+0000","last_unstale":"2024-09-16T00:16:09.019826+0000","last_undegraded":"2024-09-16T00:16:09.019826+0000","last_fullsized":"2024-09-16T00:16:09.019826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:39.345011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019993+0000","last_change":"2024-09-16T00:15:06.548852+0000","last_active":"2024-09-16T00:16:09.019993+0000","last_peered":"2024-09-16T00:16:09.019993+0000","last_clean":"2024-09-16T00:16:09.019993+0000","last_became_active":"2024-09-16T00:15:06.548238+0000","last_became_peered":"2024-09-16T00:15:06.548238+0000","last_unstale":"2024-09-16T00:16:09.019993+0000","last_undegraded":"2024-09-16T00:16:09.019993+0000","last_fullsized":"2024-09-16T00:16:09.019993+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:16.115338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.014829+0000","last_change":"2024-09-16T00:15:04.350753+0000","last_active":"2024-09-16T00:16:09.014829+0000","last_peered":"2024-09-16T00:16:09.014829+0000","last_clean":"2024-09-16T00:16:09.014829+0000","last_became_active":"2024-09-16T00:15:04.350406+0000","last_became_peered":"2024-09-16T00:15:04.350406+0000","last_unstale":"2024-09-16T00:16:09.014829+0000","last_undegraded":"2024-09-16T00:16:09.014829+0000","last_fullsized":"2024-09-16T00:16:09.014829+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:09:54.737287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030014+0000","last_change":"2024-09-16T00:15:13.639302+0000","last_active":"2024-09-16T00:16:09.030014+0000","last_peered":"2024-09-16T00:16:09.030014+0000","last_clean":"2024-09-16T00:16:09.030014+0000","last_became_active":"2024-09-16T00:15:13.638863+0000","last_became_peered":"2024-09-16T00:15:13.638863+0000","last_unstale":"2024-09-16T00:16:09.030014+0000","last_undegraded":"2024-09-16T00:16:09.030014+0000","last_fullsized":"2024-09-16T00:16:09.030014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:23.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022107+0000","last_change":"2024-09-16T00:15:06.545797+0000","last_active":"2024-09-16T00:16:09.022107+0000","last_peered":"2024-09-16T00:16:09.022107+0000","last_clean":"2024-09-16T00:16:09.022107+0000","last_became_active":"2024-09-16T00:15:06.545477+0000","last_became_peered":"2024-09-16T00:15:06.545477+0000","last_unstale":"2024-09-16T00:16:09.022107+0000","last_undegraded":"2024-09-16T00:16:09.022107+0000","last_fullsized":"2024-09-16T00:16:09.022107+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.887183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016777+0000","last_change":"2024-09-16T00:15:04.342216+0000","last_active":"2024-09-16T00:16:09.016777+0000","last_peered":"2024-09-16T00:16:09.016777+0000","last_clean":"2024-09-16T00:16:09.016777+0000","last_became_active":"2024-09-16T00:15:04.341763+0000","last_became_peered":"2024-09-16T00:15:04.341763+0000","last_unstale":"2024-09-16T00:16:09.016777+0000","last_undegraded":"2024-09-16T00:16:09.016777+0000","last_fullsized":"2024-09-16T00:16:09.016777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:44.938527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015058+0000","last_change":"2024-09-16T00:13:51.108571+0000","last_active":"2024-09-16T00:16:09.015058+0000","last_peered":"2024-09-16T00:16:09.015058+0000","last_clean":"2024-09-16T00:16:09.015058+0000","last_became_active":"2024-09-16T00:13:51.108088+0000","last_became_peered":"2024-09-16T00:13:51.108088+0000","last_unstale":"2024-09-16T00:16:09.015058+0000","last_undegraded":"2024-09-16T00:16:09.015058+0000","last_fullsized":"2024-09-16T00:16:09.015058+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_clean_scrub_stamp":"2024-09-16T00:10:24.533807+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:02.511064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015440+0000","last_change":"2024-09-16T00:15:13.588665+0000","last_active":"2024-09-16T00:16:09.015440+0000","last_peered":"2024-09-16T00:16:09.015440+0000","last_clean":"2024-09-16T00:16:09.015440+0000","last_became_active":"2024-09-16T00:15:13.588222+0000","last_became_peered":"2024-09-16T00:15:13.588222+0000","last_unstale":"2024-09-16T00:16:09.015440+0000","last_undegraded":"2024-09-16T00:16:09.015440+0000","last_fullsized":"2024-09-16T00:16:09.015440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:59:59.965962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015757+0000","last_change":"2024-09-16T00:15:06.553114+0000","last_active":"2024-09-16T00:16:09.015757+0000","last_peered":"2024-09-16T00:16:09.015757+0000","last_clean":"2024-09-16T00:16:09.015757+0000","last_became_active":"2024-09-16T00:15:06.552788+0000","last_became_peered":"2024-09-16T00:15:06.552788+0000","last_unstale":"2024-09-16T00:16:09.015757+0000","last_undegraded":"2024-09-16T00:16:09.015757+0000","last_fullsized":"2024-09-16T00:16:09.015757+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:05.473847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.031113+0000","last_change":"2024-09-16T00:16:09.030747+0000","last_active":"2024-09-16T00:16:09.031113+0000","last_peered":"2024-09-16T00:16:09.031113+0000","last_clean":"2024-09-16T00:16:09.031113+0000","last_became_active":"2024-09-16T00:16:09.021208+0000","last_became_peered":"2024-09-16T00:16:09.021208+0000","last_unstale":"2024-09-16T00:16:09.031113+0000","last_undegraded":"2024-09-16T00:16:09.031113+0000","last_fullsized":"2024-09-16T00:16:09.031113+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:40.605498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021707+0000","last_change":"2024-09-16T00:15:13.575606+0000","last_active":"2024-09-16T00:16:09.021707+0000","last_peered":"2024-09-16T00:16:09.021707+0000","last_clean":"2024-09-16T00:16:09.021707+0000","last_became_active":"2024-09-16T00:15:13.575131+0000","last_became_peered":"2024-09-16T00:15:13.575131+0000","last_unstale":"2024-09-16T00:16:09.021707+0000","last_undegraded":"2024-09-16T00:16:09.021707+0000","last_fullsized":"2024-09-16T00:16:09.021707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:41.636058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015223+0000","last_change":"2024-09-16T00:15:06.552956+0000","last_active":"2024-09-16T00:16:09.015223+0000","last_peered":"2024-09-16T00:16:09.015223+0000","last_clean":"2024-09-16T00:16:09.015223+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:16:09.015223+0000","last_undegraded":"2024-09-16T00:16:09.015223+0000","last_fullsized":"2024-09-16T00:16:09.015223+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:36:20.739676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030263+0000","last_change":"2024-09-16T00:15:04.349095+0000","last_active":"2024-09-16T00:16:09.030263+0000","last_peered":"2024-09-16T00:16:09.030263+0000","last_clean":"2024-09-16T00:16:09.030263+0000","last_became_active":"2024-09-16T00:15:04.348698+0000","last_became_peered":"2024-09-16T00:15:04.348698+0000","last_unstale":"2024-09-16T00:16:09.030263+0000","last_undegraded":"2024-09-16T00:16:09.030263+0000","last_fullsized":"2024-09-16T00:16:09.030263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:42.282882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019445+0000","last_change":"2024-09-16T00:15:13.580494+0000","last_active":"2024-09-16T00:16:09.019445+0000","last_peered":"2024-09-16T00:16:09.019445+0000","last_clean":"2024-09-16T00:16:09.019445+0000","last_became_active":"2024-09-16T00:15:13.580091+0000","last_became_peered":"2024-09-16T00:15:13.580091+0000","last_unstale":"2024-09-16T00:16:09.019445+0000","last_undegraded":"2024-09-16T00:16:09.019445+0000","last_fullsized":"2024-09-16T00:16:09.019445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:17.314919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.014736+0000","last_change":"2024-09-16T00:15:06.544859+0000","last_active":"2024-09-16T00:16:09.014736+0000","last_peered":"2024-09-16T00:16:09.014736+0000","last_clean":"2024-09-16T00:16:09.014736+0000","last_became_active":"2024-09-16T00:15:06.544289+0000","last_became_peered":"2024-09-16T00:15:06.544289+0000","last_unstale":"2024-09-16T00:16:09.014736+0000","last_undegraded":"2024-09-16T00:16:09.014736+0000","last_fullsized":"2024-09-16T00:16:09.014736+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:52.622477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020216+0000","last_change":"2024-09-16T00:15:04.351313+0000","last_active":"2024-09-16T00:16:09.020216+0000","last_peered":"2024-09-16T00:16:09.020216+0000","last_clean":"2024-09-16T00:16:09.020216+0000","last_became_active":"2024-09-16T00:15:04.350928+0000","last_became_peered":"2024-09-16T00:15:04.350928+0000","last_unstale":"2024-09-16T00:16:09.020216+0000","last_undegraded":"2024-09-16T00:16:09.020216+0000","last_fullsized":"2024-09-16T00:16:09.020216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:48.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.029450+0000","last_change":"2024-09-16T00:15:13.578573+0000","last_active":"2024-09-16T00:16:09.029450+0000","last_peered":"2024-09-16T00:16:09.029450+0000","last_clean":"2024-09-16T00:16:09.029450+0000","last_became_active":"2024-09-16T00:15:13.578117+0000","last_became_peered":"2024-09-16T00:15:13.578117+0000","last_unstale":"2024-09-16T00:16:09.029450+0000","last_undegraded":"2024-09-16T00:16:09.029450+0000","last_fullsized":"2024-09-16T00:16:09.029450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:53.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018124+0000","last_change":"2024-09-16T00:15:06.553548+0000","last_active":"2024-09-16T00:16:09.018124+0000","last_peered":"2024-09-16T00:16:09.018124+0000","last_clean":"2024-09-16T00:16:09.018124+0000","last_became_active":"2024-09-16T00:15:06.552963+0000","last_became_peered":"2024-09-16T00:15:06.552963+0000","last_unstale":"2024-09-16T00:16:09.018124+0000","last_undegraded":"2024-09-16T00:16:09.018124+0000","last_fullsized":"2024-09-16T00:16:09.018124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:43.455881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021425+0000","last_change":"2024-09-16T00:15:04.351005+0000","last_active":"2024-09-16T00:16:09.021425+0000","last_peered":"2024-09-16T00:16:09.021425+0000","last_clean":"2024-09-16T00:16:09.021425+0000","last_became_active":"2024-09-16T00:15:04.350614+0000","last_became_peered":"2024-09-16T00:15:04.350614+0000","last_unstale":"2024-09-16T00:16:09.021425+0000","last_undegraded":"2024-09-16T00:16:09.021425+0000","last_fullsized":"2024-09-16T00:16:09.021425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:39.196360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.024668+0000","last_change":"2024-09-16T00:16:09.023796+0000","last_active":"2024-09-16T00:16:09.024668+0000","last_peered":"2024-09-16T00:16:09.024668+0000","last_clean":"2024-09-16T00:16:09.024668+0000","last_became_active":"2024-09-16T00:16:09.023403+0000","last_became_peered":"2024-09-16T00:16:09.023403+0000","last_unstale":"2024-09-16T00:16:09.024668+0000","last_undegraded":"2024-09-16T00:16:09.024668+0000","last_fullsized":"2024-09-16T00:16:09.024668+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:59.488315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.029742+0000","last_change":"2024-09-16T00:15:06.546325+0000","last_active":"2024-09-16T00:16:09.029742+0000","last_peered":"2024-09-16T00:16:09.029742+0000","last_clean":"2024-09-16T00:16:09.029742+0000","last_became_active":"2024-09-16T00:15:06.546028+0000","last_became_peered":"2024-09-16T00:15:06.546028+0000","last_unstale":"2024-09-16T00:16:09.029742+0000","last_undegraded":"2024-09-16T00:16:09.029742+0000","last_fullsized":"2024-09-16T00:16:09.029742+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:15.049592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018478+0000","last_change":"2024-09-16T00:15:04.373818+0000","last_active":"2024-09-16T00:16:09.018478+0000","last_peered":"2024-09-16T00:16:09.018478+0000","last_clean":"2024-09-16T00:16:09.018478+0000","last_became_active":"2024-09-16T00:15:04.373406+0000","last_became_peered":"2024-09-16T00:15:04.373406+0000","last_unstale":"2024-09-16T00:16:09.018478+0000","last_undegraded":"2024-09-16T00:16:09.018478+0000","last_fullsized":"2024-09-16T00:16:09.018478+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:08.402660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020108+0000","last_change":"2024-09-16T00:15:13.588669+0000","last_active":"2024-09-16T00:16:09.020108+0000","last_peered":"2024-09-16T00:16:09.020108+0000","last_clean":"2024-09-16T00:16:09.020108+0000","last_became_active":"2024-09-16T00:15:13.588229+0000","last_became_peered":"2024-09-16T00:15:13.588229+0000","last_unstale":"2024-09-16T00:16:09.020108+0000","last_undegraded":"2024-09-16T00:16:09.020108+0000","last_fullsized":"2024-09-16T00:16:09.020108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:30.157296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.014528+0000","last_change":"2024-09-16T00:15:06.550221+0000","last_active":"2024-09-16T00:16:09.014528+0000","last_peered":"2024-09-16T00:16:09.014528+0000","last_clean":"2024-09-16T00:16:09.014528+0000","last_became_active":"2024-09-16T00:15:06.549914+0000","last_became_peered":"2024-09-16T00:15:06.549914+0000","last_unstale":"2024-09-16T00:16:09.014528+0000","last_undegraded":"2024-09-16T00:16:09.014528+0000","last_fullsized":"2024-09-16T00:16:09.014528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:18:26.824523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":95,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:43.254602+0000","last_change":"2024-09-16T00:15:04.344389+0000","last_active":"2024-09-16T00:16:43.254602+0000","last_peered":"2024-09-16T00:16:43.254602+0000","last_clean":"2024-09-16T00:16:43.254602+0000","last_became_active":"2024-09-16T00:15:04.343956+0000","last_became_peered":"2024-09-16T00:15:04.343956+0000","last_unstale":"2024-09-16T00:16:43.254602+0000","last_undegraded":"2024-09-16T00:16:43.254602+0000","last_fullsized":"2024-09-16T00:16:43.254602+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:34.019861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.024716+0000","last_change":"2024-09-16T00:16:09.024050+0000","last_active":"2024-09-16T00:16:09.024716+0000","last_peered":"2024-09-16T00:16:09.024716+0000","last_clean":"2024-09-16T00:16:09.024716+0000","last_became_active":"2024-09-16T00:16:09.023371+0000","last_became_peered":"2024-09-16T00:16:09.023371+0000","last_unstale":"2024-09-16T00:16:09.024716+0000","last_undegraded":"2024-09-16T00:16:09.024716+0000","last_fullsized":"2024-09-16T00:16:09.024716+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:35:19.020370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016292+0000","last_change":"2024-09-16T00:15:06.542744+0000","last_active":"2024-09-16T00:16:09.016292+0000","last_peered":"2024-09-16T00:16:09.016292+0000","last_clean":"2024-09-16T00:16:09.016292+0000","last_became_active":"2024-09-16T00:15:06.541868+0000","last_became_peered":"2024-09-16T00:15:06.541868+0000","last_unstale":"2024-09-16T00:16:09.016292+0000","last_undegraded":"2024-09-16T00:16:09.016292+0000","last_fullsized":"2024-09-16T00:16:09.016292+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:10:06.508682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030023+0000","last_change":"2024-09-16T00:15:04.349556+0000","last_active":"2024-09-16T00:16:09.030023+0000","last_peered":"2024-09-16T00:16:09.030023+0000","last_clean":"2024-09-16T00:16:09.030023+0000","last_became_active":"2024-09-16T00:15:04.349242+0000","last_became_peered":"2024-09-16T00:15:04.349242+0000","last_unstale":"2024-09-16T00:16:09.030023+0000","last_undegraded":"2024-09-16T00:16:09.030023+0000","last_fullsized":"2024-09-16T00:16:09.030023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:22:18.459157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018537+0000","last_change":"2024-09-16T00:15:13.584109+0000","last_active":"2024-09-16T00:16:09.018537+0000","last_peered":"2024-09-16T00:16:09.018537+0000","last_clean":"2024-09-16T00:16:09.018537+0000","last_became_active":"2024-09-16T00:15:13.583718+0000","last_became_peered":"2024-09-16T00:15:13.583718+0000","last_unstale":"2024-09-16T00:16:09.018537+0000","last_undegraded":"2024-09-16T00:16:09.018537+0000","last_fullsized":"2024-09-16T00:16:09.018537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:29.416254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020028+0000","last_change":"2024-09-16T00:15:06.548247+0000","last_active":"2024-09-16T00:16:09.020028+0000","last_peered":"2024-09-16T00:16:09.020028+0000","last_clean":"2024-09-16T00:16:09.020028+0000","last_became_active":"2024-09-16T00:15:06.547864+0000","last_became_peered":"2024-09-16T00:15:06.547864+0000","last_unstale":"2024-09-16T00:16:09.020028+0000","last_undegraded":"2024-09-16T00:16:09.020028+0000","last_fullsized":"2024-09-16T00:16:09.020028+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:25.350197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022815+0000","last_change":"2024-09-16T00:15:04.350965+0000","last_active":"2024-09-16T00:16:09.022815+0000","last_peered":"2024-09-16T00:16:09.022815+0000","last_clean":"2024-09-16T00:16:09.022815+0000","last_became_active":"2024-09-16T00:15:04.349124+0000","last_became_peered":"2024-09-16T00:15:04.349124+0000","last_unstale":"2024-09-16T00:16:09.022815+0000","last_undegraded":"2024-09-16T00:16:09.022815+0000","last_fullsized":"2024-09-16T00:16:09.022815+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:40.859220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.029882+0000","last_change":"2024-09-16T00:15:13.581718+0000","last_active":"2024-09-16T00:16:09.029882+0000","last_peered":"2024-09-16T00:16:09.029882+0000","last_clean":"2024-09-16T00:16:09.029882+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:16:09.029882+0000","last_undegraded":"2024-09-16T00:16:09.029882+0000","last_fullsized":"2024-09-16T00:16:09.029882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:08.810606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019805+0000","last_change":"2024-09-16T00:15:06.544507+0000","last_active":"2024-09-16T00:16:09.019805+0000","last_peered":"2024-09-16T00:16:09.019805+0000","last_clean":"2024-09-16T00:16:09.019805+0000","last_became_active":"2024-09-16T00:15:06.544178+0000","last_became_peered":"2024-09-16T00:15:06.544178+0000","last_unstale":"2024-09-16T00:16:09.019805+0000","last_undegraded":"2024-09-16T00:16:09.019805+0000","last_fullsized":"2024-09-16T00:16:09.019805+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:38.494266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"88'32","reported_seq":199,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:43.084950+0000","last_change":"2024-09-16T00:16:09.039763+0000","last_active":"2024-09-16T00:16:43.084950+0000","last_peered":"2024-09-16T00:16:43.084950+0000","last_clean":"2024-09-16T00:16:43.084950+0000","last_became_active":"2024-09-16T00:16:09.022939+0000","last_became_peered":"2024-09-16T00:16:09.022939+0000","last_unstale":"2024-09-16T00:16:43.084950+0000","last_undegraded":"2024-09-16T00:16:43.084950+0000","last_fullsized":"2024-09-16T00:16:43.084950+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:56.431159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62931,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":112,"num_read_kb":108,"num_write":33,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60943,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.013565+0000","last_change":"2024-09-16T00:15:13.585928+0000","last_active":"2024-09-16T00:16:09.013565+0000","last_peered":"2024-09-16T00:16:09.013565+0000","last_clean":"2024-09-16T00:16:09.013565+0000","last_became_active":"2024-09-16T00:15:13.582940+0000","last_became_peered":"2024-09-16T00:15:13.582940+0000","last_unstale":"2024-09-16T00:16:09.013565+0000","last_undegraded":"2024-09-16T00:16:09.013565+0000","last_fullsized":"2024-09-16T00:16:09.013565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:57:13.251978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021707+0000","last_change":"2024-09-16T00:15:06.546238+0000","last_active":"2024-09-16T00:16:09.021707+0000","last_peered":"2024-09-16T00:16:09.021707+0000","last_clean":"2024-09-16T00:16:09.021707+0000","last_became_active":"2024-09-16T00:15:06.545919+0000","last_became_peered":"2024-09-16T00:15:06.545919+0000","last_unstale":"2024-09-16T00:16:09.021707+0000","last_undegraded":"2024-09-16T00:16:09.021707+0000","last_fullsized":"2024-09-16T00:16:09.021707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:09.180832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030189+0000","last_change":"2024-09-16T00:15:04.376028+0000","last_active":"2024-09-16T00:16:09.030189+0000","last_peered":"2024-09-16T00:16:09.030189+0000","last_clean":"2024-09-16T00:16:09.030189+0000","last_became_active":"2024-09-16T00:15:04.375633+0000","last_became_peered":"2024-09-16T00:15:04.375633+0000","last_unstale":"2024-09-16T00:16:09.030189+0000","last_undegraded":"2024-09-16T00:16:09.030189+0000","last_fullsized":"2024-09-16T00:16:09.030189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:44.446342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.025524+0000","last_change":"2024-09-16T00:16:09.025524+0000","last_active":"2024-09-16T00:16:09.025524+0000","last_peered":"2024-09-16T00:16:09.025524+0000","last_clean":"2024-09-16T00:16:09.025524+0000","last_became_active":"2024-09-16T00:16:09.025035+0000","last_became_peered":"2024-09-16T00:16:09.025035+0000","last_unstale":"2024-09-16T00:16:09.025524+0000","last_undegraded":"2024-09-16T00:16:09.025524+0000","last_fullsized":"2024-09-16T00:16:09.025524+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:12.367244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020373+0000","last_change":"2024-09-16T00:15:06.550864+0000","last_active":"2024-09-16T00:16:09.020373+0000","last_peered":"2024-09-16T00:16:09.020373+0000","last_clean":"2024-09-16T00:16:09.020373+0000","last_became_active":"2024-09-16T00:15:06.547956+0000","last_became_peered":"2024-09-16T00:15:06.547956+0000","last_unstale":"2024-09-16T00:16:09.020373+0000","last_undegraded":"2024-09-16T00:16:09.020373+0000","last_fullsized":"2024-09-16T00:16:09.020373+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:26.102175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016015+0000","last_change":"2024-09-16T00:15:04.350349+0000","last_active":"2024-09-16T00:16:09.016015+0000","last_peered":"2024-09-16T00:16:09.016015+0000","last_clean":"2024-09-16T00:16:09.016015+0000","last_became_active":"2024-09-16T00:15:04.349951+0000","last_became_peered":"2024-09-16T00:15:04.349951+0000","last_unstale":"2024-09-16T00:16:09.016015+0000","last_undegraded":"2024-09-16T00:16:09.016015+0000","last_fullsized":"2024-09-16T00:16:09.016015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:46.000831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.025774+0000","last_change":"2024-09-16T00:16:09.025774+0000","last_active":"2024-09-16T00:16:09.025774+0000","last_peered":"2024-09-16T00:16:09.025774+0000","last_clean":"2024-09-16T00:16:09.025774+0000","last_became_active":"2024-09-16T00:16:09.025308+0000","last_became_peered":"2024-09-16T00:16:09.025308+0000","last_unstale":"2024-09-16T00:16:09.025774+0000","last_undegraded":"2024-09-16T00:16:09.025774+0000","last_fullsized":"2024-09-16T00:16:09.025774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:16.664850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021661+0000","last_change":"2024-09-16T00:15:06.552919+0000","last_active":"2024-09-16T00:16:09.021661+0000","last_peered":"2024-09-16T00:16:09.021661+0000","last_clean":"2024-09-16T00:16:09.021661+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:16:09.021661+0000","last_undegraded":"2024-09-16T00:16:09.021661+0000","last_fullsized":"2024-09-16T00:16:09.021661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:47:38.651837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:41.251681+0000","last_change":"2024-09-16T00:15:04.351116+0000","last_active":"2024-09-16T00:16:41.251681+0000","last_peered":"2024-09-16T00:16:41.251681+0000","last_clean":"2024-09-16T00:16:41.251681+0000","last_became_active":"2024-09-16T00:15:04.350728+0000","last_became_peered":"2024-09-16T00:15:04.350728+0000","last_unstale":"2024-09-16T00:16:41.251681+0000","last_undegraded":"2024-09-16T00:16:41.251681+0000","last_fullsized":"2024-09-16T00:16:41.251681+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:38.314019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":53,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022243+0000","last_change":"2024-09-16T00:15:13.575961+0000","last_active":"2024-09-16T00:16:09.022243+0000","last_peered":"2024-09-16T00:16:09.022243+0000","last_clean":"2024-09-16T00:16:09.022243+0000","last_became_active":"2024-09-16T00:15:13.575550+0000","last_became_peered":"2024-09-16T00:15:13.575550+0000","last_unstale":"2024-09-16T00:16:09.022243+0000","last_undegraded":"2024-09-16T00:16:09.022243+0000","last_fullsized":"2024-09-16T00:16:09.022243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:05.515588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020094+0000","last_change":"2024-09-16T00:15:13.583583+0000","last_active":"2024-09-16T00:16:09.020094+0000","last_peered":"2024-09-16T00:16:09.020094+0000","last_clean":"2024-09-16T00:16:09.020094+0000","last_became_active":"2024-09-16T00:15:13.583145+0000","last_became_peered":"2024-09-16T00:15:13.583145+0000","last_unstale":"2024-09-16T00:16:09.020094+0000","last_undegraded":"2024-09-16T00:16:09.020094+0000","last_fullsized":"2024-09-16T00:16:09.020094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:07.320337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.013547+0000","last_change":"2024-09-16T00:15:06.552017+0000","last_active":"2024-09-16T00:16:09.013547+0000","last_peered":"2024-09-16T00:16:09.013547+0000","last_clean":"2024-09-16T00:16:09.013547+0000","last_became_active":"2024-09-16T00:15:06.551688+0000","last_became_peered":"2024-09-16T00:15:06.551688+0000","last_unstale":"2024-09-16T00:16:09.013547+0000","last_undegraded":"2024-09-16T00:16:09.013547+0000","last_fullsized":"2024-09-16T00:16:09.013547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:39.951856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030263+0000","last_change":"2024-09-16T00:15:04.350826+0000","last_active":"2024-09-16T00:16:09.030263+0000","last_peered":"2024-09-16T00:16:09.030263+0000","last_clean":"2024-09-16T00:16:09.030263+0000","last_became_active":"2024-09-16T00:15:04.350294+0000","last_became_peered":"2024-09-16T00:15:04.350294+0000","last_unstale":"2024-09-16T00:16:09.030263+0000","last_undegraded":"2024-09-16T00:16:09.030263+0000","last_fullsized":"2024-09-16T00:16:09.030263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:49.146709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017920+0000","last_change":"2024-09-16T00:15:13.580184+0000","last_active":"2024-09-16T00:16:09.017920+0000","last_peered":"2024-09-16T00:16:09.017920+0000","last_clean":"2024-09-16T00:16:09.017920+0000","last_became_active":"2024-09-16T00:15:13.579849+0000","last_became_peered":"2024-09-16T00:15:13.579849+0000","last_unstale":"2024-09-16T00:16:09.017920+0000","last_undegraded":"2024-09-16T00:16:09.017920+0000","last_fullsized":"2024-09-16T00:16:09.017920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:40.613127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021921+0000","last_change":"2024-09-16T00:15:06.545796+0000","last_active":"2024-09-16T00:16:09.021921+0000","last_peered":"2024-09-16T00:16:09.021921+0000","last_clean":"2024-09-16T00:16:09.021921+0000","last_became_active":"2024-09-16T00:15:06.544815+0000","last_became_peered":"2024-09-16T00:15:06.544815+0000","last_unstale":"2024-09-16T00:16:09.021921+0000","last_undegraded":"2024-09-16T00:16:09.021921+0000","last_fullsized":"2024-09-16T00:16:09.021921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:51:25.921949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019965+0000","last_change":"2024-09-16T00:15:04.340101+0000","last_active":"2024-09-16T00:16:09.019965+0000","last_peered":"2024-09-16T00:16:09.019965+0000","last_clean":"2024-09-16T00:16:09.019965+0000","last_became_active":"2024-09-16T00:15:04.339788+0000","last_became_peered":"2024-09-16T00:15:04.339788+0000","last_unstale":"2024-09-16T00:16:09.019965+0000","last_undegraded":"2024-09-16T00:16:09.019965+0000","last_fullsized":"2024-09-16T00:16:09.019965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:08.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.028910+0000","last_change":"2024-09-16T00:16:09.028910+0000","last_active":"2024-09-16T00:16:09.028910+0000","last_peered":"2024-09-16T00:16:09.028910+0000","last_clean":"2024-09-16T00:16:09.028910+0000","last_became_active":"2024-09-16T00:16:09.023302+0000","last_became_peered":"2024-09-16T00:16:09.023302+0000","last_unstale":"2024-09-16T00:16:09.028910+0000","last_undegraded":"2024-09-16T00:16:09.028910+0000","last_fullsized":"2024-09-16T00:16:09.028910+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:35.293676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030364+0000","last_change":"2024-09-16T00:15:06.547496+0000","last_active":"2024-09-16T00:16:09.030364+0000","last_peered":"2024-09-16T00:16:09.030364+0000","last_clean":"2024-09-16T00:16:09.030364+0000","last_became_active":"2024-09-16T00:15:06.547177+0000","last_became_peered":"2024-09-16T00:15:06.547177+0000","last_unstale":"2024-09-16T00:16:09.030364+0000","last_undegraded":"2024-09-16T00:16:09.030364+0000","last_fullsized":"2024-09-16T00:16:09.030364+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:13.288827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022002+0000","last_change":"2024-09-16T00:15:13.581966+0000","last_active":"2024-09-16T00:16:09.022002+0000","last_peered":"2024-09-16T00:16:09.022002+0000","last_clean":"2024-09-16T00:16:09.022002+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:16:09.022002+0000","last_undegraded":"2024-09-16T00:16:09.022002+0000","last_fullsized":"2024-09-16T00:16:09.022002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.132773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:43.255440+0000","last_change":"2024-09-16T00:15:04.351765+0000","last_active":"2024-09-16T00:16:43.255440+0000","last_peered":"2024-09-16T00:16:43.255440+0000","last_clean":"2024-09-16T00:16:43.255440+0000","last_became_active":"2024-09-16T00:15:04.351398+0000","last_became_peered":"2024-09-16T00:15:04.351398+0000","last_unstale":"2024-09-16T00:16:43.255440+0000","last_undegraded":"2024-09-16T00:16:43.255440+0000","last_fullsized":"2024-09-16T00:16:43.255440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:14:28.762657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":57,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018785+0000","last_change":"2024-09-16T00:15:06.541422+0000","last_active":"2024-09-16T00:16:09.018785+0000","last_peered":"2024-09-16T00:16:09.018785+0000","last_clean":"2024-09-16T00:16:09.018785+0000","last_became_active":"2024-09-16T00:15:06.541083+0000","last_became_peered":"2024-09-16T00:15:06.541083+0000","last_unstale":"2024-09-16T00:16:09.018785+0000","last_undegraded":"2024-09-16T00:16:09.018785+0000","last_fullsized":"2024-09-16T00:16:09.018785+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:30.138972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022088+0000","last_change":"2024-09-16T00:15:13.576877+0000","last_active":"2024-09-16T00:16:09.022088+0000","last_peered":"2024-09-16T00:16:09.022088+0000","last_clean":"2024-09-16T00:16:09.022088+0000","last_became_active":"2024-09-16T00:15:13.576480+0000","last_became_peered":"2024-09-16T00:15:13.576480+0000","last_unstale":"2024-09-16T00:16:09.022088+0000","last_undegraded":"2024-09-16T00:16:09.022088+0000","last_fullsized":"2024-09-16T00:16:09.022088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:47.322219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018767+0000","last_change":"2024-09-16T00:15:06.550329+0000","last_active":"2024-09-16T00:16:09.018767+0000","last_peered":"2024-09-16T00:16:09.018767+0000","last_clean":"2024-09-16T00:16:09.018767+0000","last_became_active":"2024-09-16T00:15:06.550008+0000","last_became_peered":"2024-09-16T00:15:06.550008+0000","last_unstale":"2024-09-16T00:16:09.018767+0000","last_undegraded":"2024-09-16T00:16:09.018767+0000","last_fullsized":"2024-09-16T00:16:09.018767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:38.096126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021880+0000","last_change":"2024-09-16T00:15:06.550368+0000","last_active":"2024-09-16T00:16:09.021880+0000","last_peered":"2024-09-16T00:16:09.021880+0000","last_clean":"2024-09-16T00:16:09.021880+0000","last_became_active":"2024-09-16T00:15:06.549985+0000","last_became_peered":"2024-09-16T00:15:06.549985+0000","last_unstale":"2024-09-16T00:16:09.021880+0000","last_undegraded":"2024-09-16T00:16:09.021880+0000","last_fullsized":"2024-09-16T00:16:09.021880+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:16.992110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021135+0000","last_change":"2024-09-16T00:15:06.546713+0000","last_active":"2024-09-16T00:16:09.021135+0000","last_peered":"2024-09-16T00:16:09.021135+0000","last_clean":"2024-09-16T00:16:09.021135+0000","last_became_active":"2024-09-16T00:15:06.546209+0000","last_became_peered":"2024-09-16T00:15:06.546209+0000","last_unstale":"2024-09-16T00:16:09.021135+0000","last_undegraded":"2024-09-16T00:16:09.021135+0000","last_fullsized":"2024-09-16T00:16:09.021135+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:43.913748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019555+0000","last_change":"2024-09-16T00:15:06.550490+0000","last_active":"2024-09-16T00:16:09.019555+0000","last_peered":"2024-09-16T00:16:09.019555+0000","last_clean":"2024-09-16T00:16:09.019555+0000","last_became_active":"2024-09-16T00:15:06.549311+0000","last_became_peered":"2024-09-16T00:15:06.549311+0000","last_unstale":"2024-09-16T00:16:09.019555+0000","last_undegraded":"2024-09-16T00:16:09.019555+0000","last_fullsized":"2024-09-16T00:16:09.019555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:10:01.995357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019865+0000","last_change":"2024-09-16T00:15:06.548115+0000","last_active":"2024-09-16T00:16:09.019865+0000","last_peered":"2024-09-16T00:16:09.019865+0000","last_clean":"2024-09-16T00:16:09.019865+0000","last_became_active":"2024-09-16T00:15:06.547782+0000","last_became_peered":"2024-09-16T00:15:06.547782+0000","last_unstale":"2024-09-16T00:16:09.019865+0000","last_undegraded":"2024-09-16T00:16:09.019865+0000","last_fullsized":"2024-09-16T00:16:09.019865+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:53.228222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018578+0000","last_change":"2024-09-16T00:15:06.554396+0000","last_active":"2024-09-16T00:16:09.018578+0000","last_peered":"2024-09-16T00:16:09.018578+0000","last_clean":"2024-09-16T00:16:09.018578+0000","last_became_active":"2024-09-16T00:15:06.554012+0000","last_became_peered":"2024-09-16T00:15:06.554012+0000","last_unstale":"2024-09-16T00:16:09.018578+0000","last_undegraded":"2024-09-16T00:16:09.018578+0000","last_fullsized":"2024-09-16T00:16:09.018578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:31.464098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022407+0000","last_change":"2024-09-16T00:15:06.549969+0000","last_active":"2024-09-16T00:16:09.022407+0000","last_peered":"2024-09-16T00:16:09.022407+0000","last_clean":"2024-09-16T00:16:09.022407+0000","last_became_active":"2024-09-16T00:15:06.549531+0000","last_became_peered":"2024-09-16T00:15:06.549531+0000","last_unstale":"2024-09-16T00:16:09.022407+0000","last_undegraded":"2024-09-16T00:16:09.022407+0000","last_fullsized":"2024-09-16T00:16:09.022407+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:37.883612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022583+0000","last_change":"2024-09-16T00:15:06.547734+0000","last_active":"2024-09-16T00:16:09.022583+0000","last_peered":"2024-09-16T00:16:09.022583+0000","last_clean":"2024-09-16T00:16:09.022583+0000","last_became_active":"2024-09-16T00:15:06.547400+0000","last_became_peered":"2024-09-16T00:15:06.547400+0000","last_unstale":"2024-09-16T00:16:09.022583+0000","last_undegraded":"2024-09-16T00:16:09.022583+0000","last_fullsized":"2024-09-16T00:16:09.022583+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:54.455313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030297+0000","last_change":"2024-09-16T00:15:06.551965+0000","last_active":"2024-09-16T00:16:09.030297+0000","last_peered":"2024-09-16T00:16:09.030297+0000","last_clean":"2024-09-16T00:16:09.030297+0000","last_became_active":"2024-09-16T00:15:06.551602+0000","last_became_peered":"2024-09-16T00:15:06.551602+0000","last_unstale":"2024-09-16T00:16:09.030297+0000","last_undegraded":"2024-09-16T00:16:09.030297+0000","last_fullsized":"2024-09-16T00:16:09.030297+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:34.951960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022367+0000","last_change":"2024-09-16T00:15:06.549340+0000","last_active":"2024-09-16T00:16:09.022367+0000","last_peered":"2024-09-16T00:16:09.022367+0000","last_clean":"2024-09-16T00:16:09.022367+0000","last_became_active":"2024-09-16T00:15:06.547424+0000","last_became_peered":"2024-09-16T00:15:06.547424+0000","last_unstale":"2024-09-16T00:16:09.022367+0000","last_undegraded":"2024-09-16T00:16:09.022367+0000","last_fullsized":"2024-09-16T00:16:09.022367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:12.087535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021986+0000","last_change":"2024-09-16T00:15:06.555216+0000","last_active":"2024-09-16T00:16:09.021986+0000","last_peered":"2024-09-16T00:16:09.021986+0000","last_clean":"2024-09-16T00:16:09.021986+0000","last_became_active":"2024-09-16T00:15:06.554896+0000","last_became_peered":"2024-09-16T00:15:06.554896+0000","last_unstale":"2024-09-16T00:16:09.021986+0000","last_undegraded":"2024-09-16T00:16:09.021986+0000","last_fullsized":"2024-09-16T00:16:09.021986+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:20.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020299+0000","last_change":"2024-09-16T00:15:06.549752+0000","last_active":"2024-09-16T00:16:09.020299+0000","last_peered":"2024-09-16T00:16:09.020299+0000","last_clean":"2024-09-16T00:16:09.020299+0000","last_became_active":"2024-09-16T00:15:06.549350+0000","last_became_peered":"2024-09-16T00:15:06.549350+0000","last_unstale":"2024-09-16T00:16:09.020299+0000","last_undegraded":"2024-09-16T00:16:09.020299+0000","last_fullsized":"2024-09-16T00:16:09.020299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:48.615123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018663+0000","last_change":"2024-09-16T00:15:06.549159+0000","last_active":"2024-09-16T00:16:09.018663+0000","last_peered":"2024-09-16T00:16:09.018663+0000","last_clean":"2024-09-16T00:16:09.018663+0000","last_became_active":"2024-09-16T00:15:06.548408+0000","last_became_peered":"2024-09-16T00:15:06.548408+0000","last_unstale":"2024-09-16T00:16:09.018663+0000","last_undegraded":"2024-09-16T00:16:09.018663+0000","last_fullsized":"2024-09-16T00:16:09.018663+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:23.941809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022207+0000","last_change":"2024-09-16T00:15:06.544091+0000","last_active":"2024-09-16T00:16:09.022207+0000","last_peered":"2024-09-16T00:16:09.022207+0000","last_clean":"2024-09-16T00:16:09.022207+0000","last_became_active":"2024-09-16T00:15:06.543761+0000","last_became_peered":"2024-09-16T00:15:06.543761+0000","last_unstale":"2024-09-16T00:16:09.022207+0000","last_undegraded":"2024-09-16T00:16:09.022207+0000","last_fullsized":"2024-09-16T00:16:09.022207+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:36:43.253405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.013982+0000","last_change":"2024-09-16T00:15:06.546408+0000","last_active":"2024-09-16T00:16:09.013982+0000","last_peered":"2024-09-16T00:16:09.013982+0000","last_clean":"2024-09-16T00:16:09.013982+0000","last_became_active":"2024-09-16T00:15:06.546092+0000","last_became_peered":"2024-09-16T00:15:06.546092+0000","last_unstale":"2024-09-16T00:16:09.013982+0000","last_undegraded":"2024-09-16T00:16:09.013982+0000","last_fullsized":"2024-09-16T00:16:09.013982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:39:15.576097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019200+0000","last_change":"2024-09-16T00:15:06.557162+0000","last_active":"2024-09-16T00:16:09.019200+0000","last_peered":"2024-09-16T00:16:09.019200+0000","last_clean":"2024-09-16T00:16:09.019200+0000","last_became_active":"2024-09-16T00:15:06.556831+0000","last_became_peered":"2024-09-16T00:15:06.556831+0000","last_unstale":"2024-09-16T00:16:09.019200+0000","last_undegraded":"2024-09-16T00:16:09.019200+0000","last_fullsized":"2024-09-16T00:16:09.019200+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.236915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021149+0000","last_change":"2024-09-16T00:15:06.556161+0000","last_active":"2024-09-16T00:16:09.021149+0000","last_peered":"2024-09-16T00:16:09.021149+0000","last_clean":"2024-09-16T00:16:09.021149+0000","last_became_active":"2024-09-16T00:15:06.555819+0000","last_became_peered":"2024-09-16T00:15:06.555819+0000","last_unstale":"2024-09-16T00:16:09.021149+0000","last_undegraded":"2024-09-16T00:16:09.021149+0000","last_fullsized":"2024-09-16T00:16:09.021149+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:29.929977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019135+0000","last_change":"2024-09-16T00:15:06.550635+0000","last_active":"2024-09-16T00:16:09.019135+0000","last_peered":"2024-09-16T00:16:09.019135+0000","last_clean":"2024-09-16T00:16:09.019135+0000","last_became_active":"2024-09-16T00:15:06.549576+0000","last_became_peered":"2024-09-16T00:15:06.549576+0000","last_unstale":"2024-09-16T00:16:09.019135+0000","last_undegraded":"2024-09-16T00:16:09.019135+0000","last_fullsized":"2024-09-16T00:16:09.019135+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:12.818199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019645+0000","last_change":"2024-09-16T00:15:06.547114+0000","last_active":"2024-09-16T00:16:09.019645+0000","last_peered":"2024-09-16T00:16:09.019645+0000","last_clean":"2024-09-16T00:16:09.019645+0000","last_became_active":"2024-09-16T00:15:06.546786+0000","last_became_peered":"2024-09-16T00:15:06.546786+0000","last_unstale":"2024-09-16T00:16:09.019645+0000","last_undegraded":"2024-09-16T00:16:09.019645+0000","last_fullsized":"2024-09-16T00:16:09.019645+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:13:26.925603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019813+0000","last_change":"2024-09-16T00:15:06.541168+0000","last_active":"2024-09-16T00:16:09.019813+0000","last_peered":"2024-09-16T00:16:09.019813+0000","last_clean":"2024-09-16T00:16:09.019813+0000","last_became_active":"2024-09-16T00:15:06.540848+0000","last_became_peered":"2024-09-16T00:15:06.540848+0000","last_unstale":"2024-09-16T00:16:09.019813+0000","last_undegraded":"2024-09-16T00:16:09.019813+0000","last_fullsized":"2024-09-16T00:16:09.019813+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.958600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017094+0000","last_change":"2024-09-16T00:15:06.547429+0000","last_active":"2024-09-16T00:16:09.017094+0000","last_peered":"2024-09-16T00:16:09.017094+0000","last_clean":"2024-09-16T00:16:09.017094+0000","last_became_active":"2024-09-16T00:15:06.547081+0000","last_became_peered":"2024-09-16T00:15:06.547081+0000","last_unstale":"2024-09-16T00:16:09.017094+0000","last_undegraded":"2024-09-16T00:16:09.017094+0000","last_fullsized":"2024-09-16T00:16:09.017094+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:43.396313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015644+0000","last_change":"2024-09-16T00:15:06.551811+0000","last_active":"2024-09-16T00:16:09.015644+0000","last_peered":"2024-09-16T00:16:09.015644+0000","last_clean":"2024-09-16T00:16:09.015644+0000","last_became_active":"2024-09-16T00:15:06.551424+0000","last_became_peered":"2024-09-16T00:15:06.551424+0000","last_unstale":"2024-09-16T00:16:09.015644+0000","last_undegraded":"2024-09-16T00:16:09.015644+0000","last_fullsized":"2024-09-16T00:16:09.015644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.943070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017860+0000","last_change":"2024-09-16T00:15:06.544697+0000","last_active":"2024-09-16T00:16:09.017860+0000","last_peered":"2024-09-16T00:16:09.017860+0000","last_clean":"2024-09-16T00:16:09.017860+0000","last_became_active":"2024-09-16T00:15:06.544182+0000","last_became_peered":"2024-09-16T00:15:06.544182+0000","last_unstale":"2024-09-16T00:16:09.017860+0000","last_undegraded":"2024-09-16T00:16:09.017860+0000","last_fullsized":"2024-09-16T00:16:09.017860+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:45.929092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022673+0000","last_change":"2024-09-16T00:15:06.551488+0000","last_active":"2024-09-16T00:16:09.022673+0000","last_peered":"2024-09-16T00:16:09.022673+0000","last_clean":"2024-09-16T00:16:09.022673+0000","last_became_active":"2024-09-16T00:15:06.551171+0000","last_became_peered":"2024-09-16T00:15:06.551171+0000","last_unstale":"2024-09-16T00:16:09.022673+0000","last_undegraded":"2024-09-16T00:16:09.022673+0000","last_fullsized":"2024-09-16T00:16:09.022673+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:23.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021915+0000","last_change":"2024-09-16T00:15:06.544043+0000","last_active":"2024-09-16T00:16:09.021915+0000","last_peered":"2024-09-16T00:16:09.021915+0000","last_clean":"2024-09-16T00:16:09.021915+0000","last_became_active":"2024-09-16T00:15:06.543014+0000","last_became_peered":"2024-09-16T00:15:06.543014+0000","last_unstale":"2024-09-16T00:16:09.021915+0000","last_undegraded":"2024-09-16T00:16:09.021915+0000","last_fullsized":"2024-09-16T00:16:09.021915+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:23.904810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.014817+0000","last_change":"2024-09-16T00:15:06.546788+0000","last_active":"2024-09-16T00:16:09.014817+0000","last_peered":"2024-09-16T00:16:09.014817+0000","last_clean":"2024-09-16T00:16:09.014817+0000","last_became_active":"2024-09-16T00:15:06.546333+0000","last_became_peered":"2024-09-16T00:15:06.546333+0000","last_unstale":"2024-09-16T00:16:09.014817+0000","last_undegraded":"2024-09-16T00:16:09.014817+0000","last_fullsized":"2024-09-16T00:16:09.014817+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:09.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016046+0000","last_change":"2024-09-16T00:15:06.544855+0000","last_active":"2024-09-16T00:16:09.016046+0000","last_peered":"2024-09-16T00:16:09.016046+0000","last_clean":"2024-09-16T00:16:09.016046+0000","last_became_active":"2024-09-16T00:15:06.543708+0000","last_became_peered":"2024-09-16T00:15:06.543708+0000","last_unstale":"2024-09-16T00:16:09.016046+0000","last_undegraded":"2024-09-16T00:16:09.016046+0000","last_fullsized":"2024-09-16T00:16:09.016046+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:13:58.756111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018919+0000","last_change":"2024-09-16T00:15:06.540675+0000","last_active":"2024-09-16T00:16:09.018919+0000","last_peered":"2024-09-16T00:16:09.018919+0000","last_clean":"2024-09-16T00:16:09.018919+0000","last_became_active":"2024-09-16T00:15:06.540333+0000","last_became_peered":"2024-09-16T00:15:06.540333+0000","last_unstale":"2024-09-16T00:16:09.018919+0000","last_undegraded":"2024-09-16T00:16:09.018919+0000","last_fullsized":"2024-09-16T00:16:09.018919+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:17.166987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017031+0000","last_change":"2024-09-16T00:15:06.548758+0000","last_active":"2024-09-16T00:16:09.017031+0000","last_peered":"2024-09-16T00:16:09.017031+0000","last_clean":"2024-09-16T00:16:09.017031+0000","last_became_active":"2024-09-16T00:15:06.548397+0000","last_became_peered":"2024-09-16T00:15:06.548397+0000","last_unstale":"2024-09-16T00:16:09.017031+0000","last_undegraded":"2024-09-16T00:16:09.017031+0000","last_fullsized":"2024-09-16T00:16:09.017031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:50.320170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020486+0000","last_change":"2024-09-16T00:15:06.554898+0000","last_active":"2024-09-16T00:16:09.020486+0000","last_peered":"2024-09-16T00:16:09.020486+0000","last_clean":"2024-09-16T00:16:09.020486+0000","last_became_active":"2024-09-16T00:15:06.553005+0000","last_became_peered":"2024-09-16T00:15:06.553005+0000","last_unstale":"2024-09-16T00:16:09.020486+0000","last_undegraded":"2024-09-16T00:16:09.020486+0000","last_fullsized":"2024-09-16T00:16:09.020486+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:29.630911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020199+0000","last_change":"2024-09-16T00:15:06.547265+0000","last_active":"2024-09-16T00:16:09.020199+0000","last_peered":"2024-09-16T00:16:09.020199+0000","last_clean":"2024-09-16T00:16:09.020199+0000","last_became_active":"2024-09-16T00:15:06.546938+0000","last_became_peered":"2024-09-16T00:15:06.546938+0000","last_unstale":"2024-09-16T00:16:09.020199+0000","last_undegraded":"2024-09-16T00:16:09.020199+0000","last_fullsized":"2024-09-16T00:16:09.020199+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:40.247343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019051+0000","last_change":"2024-09-16T00:15:06.556735+0000","last_active":"2024-09-16T00:16:09.019051+0000","last_peered":"2024-09-16T00:16:09.019051+0000","last_clean":"2024-09-16T00:16:09.019051+0000","last_became_active":"2024-09-16T00:15:06.556324+0000","last_became_peered":"2024-09-16T00:15:06.556324+0000","last_unstale":"2024-09-16T00:16:09.019051+0000","last_undegraded":"2024-09-16T00:16:09.019051+0000","last_fullsized":"2024-09-16T00:16:09.019051+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:25.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66934,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":348,"num_read_kb":234,"num_write":165,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":62107,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":200802,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249414,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579938,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":970752,"data_stored":497547,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808870,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36440,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708712,"statfs":{"total":95995035648,"available":95957721088,"internally_reserved":0,"allocated":1118208,"data_stored":1148139,"data_compressed":59817,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972396,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":991232,"data_stored":501879,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201328,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":497121,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430262,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1036288,"data_stored":1093614,"data_compressed":59817,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626492,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":966656,"data_stored":492959,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855425,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1044480,"data_stored":1088809,"data_compressed":59817,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986246,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":561515,"data_compressed":33384,"data_compressed_allocated":45056,"data_compressed_original":122880,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215178,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":978944,"data_stored":493153,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444112,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":983040,"data_stored":493637,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607637,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1040384,"data_stored":556382,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":991232,"data_stored":494347,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":64550,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T00:16:46.698 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T00:16:46.959 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:47.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.437+0000 7f809eba0640 1 -- 172.21.15.33:0/488593119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8098102d30 msgr2=0x7f8098103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:47.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.437+0000 7f809eba0640 1 --2- 172.21.15.33:0/488593119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8098102d30 0x7f8098103110 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f808c009930 tx=0x7f808c02a420 comp rx=0 tx=0).stop 2024-09-16T00:16:47.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.437+0000 7f809eba0640 1 -- 172.21.15.33:0/488593119 shutdown_connections 2024-09-16T00:16:47.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.437+0000 7f809eba0640 1 --2- 172.21.15.33:0/488593119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80981082a0 0x7f809810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:47.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.437+0000 7f809eba0640 1 --2- 172.21.15.33:0/488593119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80981036e0 0x7f8098107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:47.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.437+0000 7f809eba0640 1 --2- 172.21.15.33:0/488593119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8098102d30 0x7f8098103110 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:47.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.437+0000 7f809eba0640 1 -- 172.21.15.33:0/488593119 >> 172.21.15.33:0/488593119 conn(0x7f80980fc7d0 msgr2=0x7f80980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:47.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.437+0000 7f809eba0640 1 -- 172.21.15.33:0/488593119 shutdown_connections 2024-09-16T00:16:47.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.438+0000 7f809eba0640 1 -- 172.21.15.33:0/488593119 wait complete. 2024-09-16T00:16:47.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.438+0000 7f809eba0640 1 Processor -- start 2024-09-16T00:16:47.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.438+0000 7f809eba0640 1 -- start start 2024-09-16T00:16:47.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809eba0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8098102d30 0x7f80981985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:47.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809eba0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80981036e0 0x7f8098198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:47.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809eba0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80981082a0 0x7f809819ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809eba0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f80981120a0 con 0x7f80981082a0 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809eba0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8098111f20 con 0x7f80981036e0 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809eba0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8098112220 con 0x7f8098102d30 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809d13d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80981082a0 0x7f809819ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809d13d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80981082a0 0x7f809819ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40072/0 (socket says 172.21.15.33:40072) 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809d13d640 1 -- 172.21.15.33:0/700604953 learned_addr learned my addr 172.21.15.33:0/700604953 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f8097fff640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80981036e0 0x7f8098198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809c93c640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8098102d30 0x7f80981985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809d13d640 1 -- 172.21.15.33:0/700604953 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8098102d30 msgr2=0x7f80981985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809d13d640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8098102d30 0x7f80981985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809d13d640 1 -- 172.21.15.33:0/700604953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80981036e0 msgr2=0x7f8098198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809d13d640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80981036e0 0x7f8098198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809d13d640 1 -- 172.21.15.33:0/700604953 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809819d550 con 0x7f80981082a0 2024-09-16T00:16:47.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.439+0000 7f809d13d640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80981082a0 0x7f809819ce70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f809000eaa0 tx=0x7f809000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:47.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.440+0000 7f8095ffb640 1 -- 172.21.15.33:0/700604953 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8090004d20 con 0x7f80981082a0 2024-09-16T00:16:47.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.440+0000 7f8095ffb640 1 -- 172.21.15.33:0/700604953 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8090005020 con 0x7f80981082a0 2024-09-16T00:16:47.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.440+0000 7f809eba0640 1 -- 172.21.15.33:0/700604953 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f809819d840 con 0x7f80981082a0 2024-09-16T00:16:47.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.440+0000 7f8095ffb640 1 -- 172.21.15.33:0/700604953 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80900116d0 con 0x7f80981082a0 2024-09-16T00:16:47.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.440+0000 7f809eba0640 1 -- 172.21.15.33:0/700604953 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80981a5120 con 0x7f80981082a0 2024-09-16T00:16:47.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.441+0000 7f809eba0640 1 -- 172.21.15.33:0/700604953 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f809806a850 con 0x7f80981082a0 2024-09-16T00:16:47.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.442+0000 7f8095ffb640 1 -- 172.21.15.33:0/700604953 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f80900051d0 con 0x7f80981082a0 2024-09-16T00:16:47.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.443+0000 7f8095ffb640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f80880a32d0 0x7f80880a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:47.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.443+0000 7f8095ffb640 1 -- 172.21.15.33:0/700604953 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f80900cffc0 con 0x7f80981082a0 2024-09-16T00:16:47.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.443+0000 7f809c93c640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f80880a32d0 0x7f80880a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:47.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.444+0000 7f809c93c640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f80880a32d0 0x7f80880a5790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f808c000c00 tx=0x7f808c005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:47.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.447+0000 7f8095ffb640 1 -- 172.21.15.33:0/700604953 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8090016030 con 0x7f80981082a0 2024-09-16T00:16:47.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.621+0000 7f809eba0640 1 -- 172.21.15.33:0/700604953 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8098062630 con 0x7f80880a32d0 2024-09-16T00:16:47.623 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:47 smithi033 ceph-mon[31243]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:47.623 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:47 smithi033 ceph-mon[31243]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:16:47.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.630+0000 7f8095ffb640 1 -- 172.21.15.33:0/700604953 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340640 (secure 0 0 0) 0x7f8098062630 con 0x7f80880a32d0 2024-09-16T00:16:47.635 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:47.636 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-09-16T00:16:47.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.633+0000 7f809eba0640 1 -- 172.21.15.33:0/700604953 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f80880a32d0 msgr2=0x7f80880a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:47.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.633+0000 7f809eba0640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f80880a32d0 0x7f80880a5790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f808c000c00 tx=0x7f808c005a20 comp rx=0 tx=0).stop 2024-09-16T00:16:47.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.633+0000 7f809eba0640 1 -- 172.21.15.33:0/700604953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80981082a0 msgr2=0x7f809819ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:47.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.633+0000 7f809eba0640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80981082a0 0x7f809819ce70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f809000eaa0 tx=0x7f809000eea0 comp rx=0 tx=0).stop 2024-09-16T00:16:47.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.633+0000 7f809eba0640 1 -- 172.21.15.33:0/700604953 shutdown_connections 2024-09-16T00:16:47.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.633+0000 7f809eba0640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f80880a32d0 0x7f80880a5790 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:47.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.633+0000 7f809eba0640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80981082a0 0x7f809819ce70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:47.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.633+0000 7f809eba0640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80981036e0 0x7f8098198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:47.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.633+0000 7f809eba0640 1 --2- 172.21.15.33:0/700604953 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8098102d30 0x7f80981985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:47.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.633+0000 7f809eba0640 1 -- 172.21.15.33:0/700604953 >> 172.21.15.33:0/700604953 conn(0x7f80980fc7d0 msgr2=0x7f8098105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:47.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.633+0000 7f809eba0640 1 -- 172.21.15.33:0/700604953 shutdown_connections 2024-09-16T00:16:47.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:47.633+0000 7f809eba0640 1 -- 172.21.15.33:0/700604953 wait complete. 2024-09-16T00:16:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:47 smithi083 ceph-mon[35242]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:47 smithi083 ceph-mon[35242]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:16:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:47 smithi060 ceph-mon[35185]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:47 smithi060 ceph-mon[35185]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:16:48.441 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-16T00:16:45.658095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657536,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":474,"num_read_kb":343,"num_write":348,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1242843,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386104,"kb_used_data":11832,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555720,"statfs":{"total":1151940427776,"available":1151545057280,"internally_reserved":0,"allocated":12115968,"data_stored":7919102,"data_compressed":400707,"data_compressed_allocated":540672,"data_compressed_original":2613248,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003521"},"pg_stats":[{"pgid":"2.1f","version":"86'9","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.034677+0000","last_change":"2024-09-16T00:16:09.034677+0000","last_active":"2024-09-16T00:16:09.034677+0000","last_peered":"2024-09-16T00:16:09.034677+0000","last_clean":"2024-09-16T00:16:09.034677+0000","last_became_active":"2024-09-16T00:16:09.024755+0000","last_became_peered":"2024-09-16T00:16:09.024755+0000","last_unstale":"2024-09-16T00:16:09.034677+0000","last_undegraded":"2024-09-16T00:16:09.034677+0000","last_fullsized":"2024-09-16T00:16:09.034677+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:09.012210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022356+0000","last_change":"2024-09-16T00:15:06.547094+0000","last_active":"2024-09-16T00:16:09.022356+0000","last_peered":"2024-09-16T00:16:09.022356+0000","last_clean":"2024-09-16T00:16:09.022356+0000","last_became_active":"2024-09-16T00:15:06.542764+0000","last_became_peered":"2024-09-16T00:15:06.542764+0000","last_unstale":"2024-09-16T00:16:09.022356+0000","last_undegraded":"2024-09-16T00:16:09.022356+0000","last_fullsized":"2024-09-16T00:16:09.022356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:04:37.449577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021501+0000","last_change":"2024-09-16T00:15:13.578400+0000","last_active":"2024-09-16T00:16:09.021501+0000","last_peered":"2024-09-16T00:16:09.021501+0000","last_clean":"2024-09-16T00:16:09.021501+0000","last_became_active":"2024-09-16T00:15:13.577787+0000","last_became_peered":"2024-09-16T00:15:13.577787+0000","last_unstale":"2024-09-16T00:16:09.021501+0000","last_undegraded":"2024-09-16T00:16:09.021501+0000","last_fullsized":"2024-09-16T00:16:09.021501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:19:59.194152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022614+0000","last_change":"2024-09-16T00:15:04.373381+0000","last_active":"2024-09-16T00:16:09.022614+0000","last_peered":"2024-09-16T00:16:09.022614+0000","last_clean":"2024-09-16T00:16:09.022614+0000","last_became_active":"2024-09-16T00:15:04.373028+0000","last_became_peered":"2024-09-16T00:15:04.373028+0000","last_unstale":"2024-09-16T00:16:09.022614+0000","last_undegraded":"2024-09-16T00:16:09.022614+0000","last_fullsized":"2024-09-16T00:16:09.022614+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:22.902583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020323+0000","last_change":"2024-09-16T00:15:06.554048+0000","last_active":"2024-09-16T00:16:09.020323+0000","last_peered":"2024-09-16T00:16:09.020323+0000","last_clean":"2024-09-16T00:16:09.020323+0000","last_became_active":"2024-09-16T00:15:06.553044+0000","last_became_peered":"2024-09-16T00:15:06.553044+0000","last_unstale":"2024-09-16T00:16:09.020323+0000","last_undegraded":"2024-09-16T00:16:09.020323+0000","last_fullsized":"2024-09-16T00:16:09.020323+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:22:53.333705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017272+0000","last_change":"2024-09-16T00:15:13.586995+0000","last_active":"2024-09-16T00:16:09.017272+0000","last_peered":"2024-09-16T00:16:09.017272+0000","last_clean":"2024-09-16T00:16:09.017272+0000","last_became_active":"2024-09-16T00:15:13.586509+0000","last_became_peered":"2024-09-16T00:15:13.586509+0000","last_unstale":"2024-09-16T00:16:09.017272+0000","last_undegraded":"2024-09-16T00:16:09.017272+0000","last_fullsized":"2024-09-16T00:16:09.017272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:39.418581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019912+0000","last_change":"2024-09-16T00:15:04.348235+0000","last_active":"2024-09-16T00:16:09.019912+0000","last_peered":"2024-09-16T00:16:09.019912+0000","last_clean":"2024-09-16T00:16:09.019912+0000","last_became_active":"2024-09-16T00:15:04.347518+0000","last_became_peered":"2024-09-16T00:15:04.347518+0000","last_unstale":"2024-09-16T00:16:09.019912+0000","last_undegraded":"2024-09-16T00:16:09.019912+0000","last_fullsized":"2024-09-16T00:16:09.019912+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:24.396911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021590+0000","last_change":"2024-09-16T00:15:06.554067+0000","last_active":"2024-09-16T00:16:09.021590+0000","last_peered":"2024-09-16T00:16:09.021590+0000","last_clean":"2024-09-16T00:16:09.021590+0000","last_became_active":"2024-09-16T00:15:06.553715+0000","last_became_peered":"2024-09-16T00:15:06.553715+0000","last_unstale":"2024-09-16T00:16:09.021590+0000","last_undegraded":"2024-09-16T00:16:09.021590+0000","last_fullsized":"2024-09-16T00:16:09.021590+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:22.008530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016383+0000","last_change":"2024-09-16T00:15:13.586872+0000","last_active":"2024-09-16T00:16:09.016383+0000","last_peered":"2024-09-16T00:16:09.016383+0000","last_clean":"2024-09-16T00:16:09.016383+0000","last_became_active":"2024-09-16T00:15:13.586375+0000","last_became_peered":"2024-09-16T00:15:13.586375+0000","last_unstale":"2024-09-16T00:16:09.016383+0000","last_undegraded":"2024-09-16T00:16:09.016383+0000","last_fullsized":"2024-09-16T00:16:09.016383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:43.668827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020333+0000","last_change":"2024-09-16T00:15:04.347964+0000","last_active":"2024-09-16T00:16:09.020333+0000","last_peered":"2024-09-16T00:16:09.020333+0000","last_clean":"2024-09-16T00:16:09.020333+0000","last_became_active":"2024-09-16T00:15:04.347588+0000","last_became_peered":"2024-09-16T00:15:04.347588+0000","last_unstale":"2024-09-16T00:16:09.020333+0000","last_undegraded":"2024-09-16T00:16:09.020333+0000","last_fullsized":"2024-09-16T00:16:09.020333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:14.199287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030005+0000","last_change":"2024-09-16T00:15:06.544918+0000","last_active":"2024-09-16T00:16:09.030005+0000","last_peered":"2024-09-16T00:16:09.030005+0000","last_clean":"2024-09-16T00:16:09.030005+0000","last_became_active":"2024-09-16T00:15:06.544579+0000","last_became_peered":"2024-09-16T00:15:06.544579+0000","last_unstale":"2024-09-16T00:16:09.030005+0000","last_undegraded":"2024-09-16T00:16:09.030005+0000","last_fullsized":"2024-09-16T00:16:09.030005+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:06:37.475644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015781+0000","last_change":"2024-09-16T00:15:13.581488+0000","last_active":"2024-09-16T00:16:09.015781+0000","last_peered":"2024-09-16T00:16:09.015781+0000","last_clean":"2024-09-16T00:16:09.015781+0000","last_became_active":"2024-09-16T00:15:13.581164+0000","last_became_peered":"2024-09-16T00:15:13.581164+0000","last_unstale":"2024-09-16T00:16:09.015781+0000","last_undegraded":"2024-09-16T00:16:09.015781+0000","last_fullsized":"2024-09-16T00:16:09.015781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:05.061592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015354+0000","last_change":"2024-09-16T00:15:06.551982+0000","last_active":"2024-09-16T00:16:09.015354+0000","last_peered":"2024-09-16T00:16:09.015354+0000","last_clean":"2024-09-16T00:16:09.015354+0000","last_became_active":"2024-09-16T00:15:06.551663+0000","last_became_peered":"2024-09-16T00:15:06.551663+0000","last_unstale":"2024-09-16T00:16:09.015354+0000","last_undegraded":"2024-09-16T00:16:09.015354+0000","last_fullsized":"2024-09-16T00:16:09.015354+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:11.928703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019021+0000","last_change":"2024-09-16T00:15:04.352237+0000","last_active":"2024-09-16T00:16:09.019021+0000","last_peered":"2024-09-16T00:16:09.019021+0000","last_clean":"2024-09-16T00:16:09.019021+0000","last_became_active":"2024-09-16T00:15:04.351884+0000","last_became_peered":"2024-09-16T00:15:04.351884+0000","last_unstale":"2024-09-16T00:16:09.019021+0000","last_undegraded":"2024-09-16T00:16:09.019021+0000","last_fullsized":"2024-09-16T00:16:09.019021+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:32.932020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016008+0000","last_change":"2024-09-16T00:15:13.638917+0000","last_active":"2024-09-16T00:16:09.016008+0000","last_peered":"2024-09-16T00:16:09.016008+0000","last_clean":"2024-09-16T00:16:09.016008+0000","last_became_active":"2024-09-16T00:15:13.638401+0000","last_became_peered":"2024-09-16T00:15:13.638401+0000","last_unstale":"2024-09-16T00:16:09.016008+0000","last_undegraded":"2024-09-16T00:16:09.016008+0000","last_fullsized":"2024-09-16T00:16:09.016008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:38.082407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022193+0000","last_change":"2024-09-16T00:15:06.547909+0000","last_active":"2024-09-16T00:16:09.022193+0000","last_peered":"2024-09-16T00:16:09.022193+0000","last_clean":"2024-09-16T00:16:09.022193+0000","last_became_active":"2024-09-16T00:15:06.546814+0000","last_became_peered":"2024-09-16T00:15:06.546814+0000","last_unstale":"2024-09-16T00:16:09.022193+0000","last_undegraded":"2024-09-16T00:16:09.022193+0000","last_fullsized":"2024-09-16T00:16:09.022193+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:46.419406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.028228+0000","last_change":"2024-09-16T00:16:09.027691+0000","last_active":"2024-09-16T00:16:09.028228+0000","last_peered":"2024-09-16T00:16:09.028228+0000","last_clean":"2024-09-16T00:16:09.028228+0000","last_became_active":"2024-09-16T00:16:09.023204+0000","last_became_peered":"2024-09-16T00:16:09.023204+0000","last_unstale":"2024-09-16T00:16:09.028228+0000","last_undegraded":"2024-09-16T00:16:09.028228+0000","last_fullsized":"2024-09-16T00:16:09.028228+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:44.647576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016601+0000","last_change":"2024-09-16T00:15:13.588345+0000","last_active":"2024-09-16T00:16:09.016601+0000","last_peered":"2024-09-16T00:16:09.016601+0000","last_clean":"2024-09-16T00:16:09.016601+0000","last_became_active":"2024-09-16T00:15:13.587829+0000","last_became_peered":"2024-09-16T00:15:13.587829+0000","last_unstale":"2024-09-16T00:16:09.016601+0000","last_undegraded":"2024-09-16T00:16:09.016601+0000","last_fullsized":"2024-09-16T00:16:09.016601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:04.261031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018874+0000","last_change":"2024-09-16T00:15:06.545485+0000","last_active":"2024-09-16T00:16:09.018874+0000","last_peered":"2024-09-16T00:16:09.018874+0000","last_clean":"2024-09-16T00:16:09.018874+0000","last_became_active":"2024-09-16T00:15:06.545178+0000","last_became_peered":"2024-09-16T00:15:06.545178+0000","last_unstale":"2024-09-16T00:16:09.018874+0000","last_undegraded":"2024-09-16T00:16:09.018874+0000","last_fullsized":"2024-09-16T00:16:09.018874+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:26.923469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022534+0000","last_change":"2024-09-16T00:15:04.340546+0000","last_active":"2024-09-16T00:16:09.022534+0000","last_peered":"2024-09-16T00:16:09.022534+0000","last_clean":"2024-09-16T00:16:09.022534+0000","last_became_active":"2024-09-16T00:15:04.340173+0000","last_became_peered":"2024-09-16T00:15:04.340173+0000","last_unstale":"2024-09-16T00:16:09.022534+0000","last_undegraded":"2024-09-16T00:16:09.022534+0000","last_fullsized":"2024-09-16T00:16:09.022534+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:35:01.134433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.013959+0000","last_change":"2024-09-16T00:15:13.589963+0000","last_active":"2024-09-16T00:16:09.013959+0000","last_peered":"2024-09-16T00:16:09.013959+0000","last_clean":"2024-09-16T00:16:09.013959+0000","last_became_active":"2024-09-16T00:15:13.589615+0000","last_became_peered":"2024-09-16T00:15:13.589615+0000","last_unstale":"2024-09-16T00:16:09.013959+0000","last_undegraded":"2024-09-16T00:16:09.013959+0000","last_fullsized":"2024-09-16T00:16:09.013959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:37.197765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019152+0000","last_change":"2024-09-16T00:15:06.544374+0000","last_active":"2024-09-16T00:16:09.019152+0000","last_peered":"2024-09-16T00:16:09.019152+0000","last_clean":"2024-09-16T00:16:09.019152+0000","last_became_active":"2024-09-16T00:15:06.544027+0000","last_became_peered":"2024-09-16T00:15:06.544027+0000","last_unstale":"2024-09-16T00:16:09.019152+0000","last_undegraded":"2024-09-16T00:16:09.019152+0000","last_fullsized":"2024-09-16T00:16:09.019152+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021199+0000","last_change":"2024-09-16T00:15:04.374209+0000","last_active":"2024-09-16T00:16:09.021199+0000","last_peered":"2024-09-16T00:16:09.021199+0000","last_clean":"2024-09-16T00:16:09.021199+0000","last_became_active":"2024-09-16T00:15:04.373814+0000","last_became_peered":"2024-09-16T00:15:04.373814+0000","last_unstale":"2024-09-16T00:16:09.021199+0000","last_undegraded":"2024-09-16T00:16:09.021199+0000","last_fullsized":"2024-09-16T00:16:09.021199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:13.516161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016200+0000","last_change":"2024-09-16T00:15:13.639210+0000","last_active":"2024-09-16T00:16:09.016200+0000","last_peered":"2024-09-16T00:16:09.016200+0000","last_clean":"2024-09-16T00:16:09.016200+0000","last_became_active":"2024-09-16T00:15:13.638719+0000","last_became_peered":"2024-09-16T00:15:13.638719+0000","last_unstale":"2024-09-16T00:16:09.016200+0000","last_undegraded":"2024-09-16T00:16:09.016200+0000","last_fullsized":"2024-09-16T00:16:09.016200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:26.130778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020290+0000","last_change":"2024-09-16T00:15:06.553108+0000","last_active":"2024-09-16T00:16:09.020290+0000","last_peered":"2024-09-16T00:16:09.020290+0000","last_clean":"2024-09-16T00:16:09.020290+0000","last_became_active":"2024-09-16T00:15:06.552817+0000","last_became_peered":"2024-09-16T00:15:06.552817+0000","last_unstale":"2024-09-16T00:16:09.020290+0000","last_undegraded":"2024-09-16T00:16:09.020290+0000","last_fullsized":"2024-09-16T00:16:09.020290+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:06.821337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016843+0000","last_change":"2024-09-16T00:15:04.644185+0000","last_active":"2024-09-16T00:16:09.016843+0000","last_peered":"2024-09-16T00:16:09.016843+0000","last_clean":"2024-09-16T00:16:09.016843+0000","last_became_active":"2024-09-16T00:15:04.643887+0000","last_became_peered":"2024-09-16T00:15:04.643887+0000","last_unstale":"2024-09-16T00:16:09.016843+0000","last_undegraded":"2024-09-16T00:16:09.016843+0000","last_fullsized":"2024-09-16T00:16:09.016843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:44.279272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021730+0000","last_change":"2024-09-16T00:15:13.574761+0000","last_active":"2024-09-16T00:16:09.021730+0000","last_peered":"2024-09-16T00:16:09.021730+0000","last_clean":"2024-09-16T00:16:09.021730+0000","last_became_active":"2024-09-16T00:15:13.574347+0000","last_became_peered":"2024-09-16T00:15:13.574347+0000","last_unstale":"2024-09-16T00:16:09.021730+0000","last_undegraded":"2024-09-16T00:16:09.021730+0000","last_fullsized":"2024-09-16T00:16:09.021730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:22.040121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022609+0000","last_change":"2024-09-16T00:15:06.550338+0000","last_active":"2024-09-16T00:16:09.022609+0000","last_peered":"2024-09-16T00:16:09.022609+0000","last_clean":"2024-09-16T00:16:09.022609+0000","last_became_active":"2024-09-16T00:15:06.549868+0000","last_became_peered":"2024-09-16T00:15:06.549868+0000","last_unstale":"2024-09-16T00:16:09.022609+0000","last_undegraded":"2024-09-16T00:16:09.022609+0000","last_fullsized":"2024-09-16T00:16:09.022609+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:26.643338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'10","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.014079+0000","last_change":"2024-09-16T00:15:04.342306+0000","last_active":"2024-09-16T00:16:09.014079+0000","last_peered":"2024-09-16T00:16:09.014079+0000","last_clean":"2024-09-16T00:16:09.014079+0000","last_became_active":"2024-09-16T00:15:04.341964+0000","last_became_peered":"2024-09-16T00:15:04.341964+0000","last_unstale":"2024-09-16T00:16:09.014079+0000","last_undegraded":"2024-09-16T00:16:09.014079+0000","last_fullsized":"2024-09-16T00:16:09.014079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:14.137199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021778+0000","last_change":"2024-09-16T00:15:13.638246+0000","last_active":"2024-09-16T00:16:09.021778+0000","last_peered":"2024-09-16T00:16:09.021778+0000","last_clean":"2024-09-16T00:16:09.021778+0000","last_became_active":"2024-09-16T00:15:13.637816+0000","last_became_peered":"2024-09-16T00:15:13.637816+0000","last_unstale":"2024-09-16T00:16:09.021778+0000","last_undegraded":"2024-09-16T00:16:09.021778+0000","last_fullsized":"2024-09-16T00:16:09.021778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:27.331156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019970+0000","last_change":"2024-09-16T00:15:06.544057+0000","last_active":"2024-09-16T00:16:09.019970+0000","last_peered":"2024-09-16T00:16:09.019970+0000","last_clean":"2024-09-16T00:16:09.019970+0000","last_became_active":"2024-09-16T00:15:06.543244+0000","last_became_peered":"2024-09-16T00:15:06.543244+0000","last_unstale":"2024-09-16T00:16:09.019970+0000","last_undegraded":"2024-09-16T00:16:09.019970+0000","last_fullsized":"2024-09-16T00:16:09.019970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:42:28.950014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020105+0000","last_change":"2024-09-16T00:15:04.376419+0000","last_active":"2024-09-16T00:16:09.020105+0000","last_peered":"2024-09-16T00:16:09.020105+0000","last_clean":"2024-09-16T00:16:09.020105+0000","last_became_active":"2024-09-16T00:15:04.375976+0000","last_became_peered":"2024-09-16T00:15:04.375976+0000","last_unstale":"2024-09-16T00:16:09.020105+0000","last_undegraded":"2024-09-16T00:16:09.020105+0000","last_fullsized":"2024-09-16T00:16:09.020105+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:33:38.652109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018162+0000","last_change":"2024-09-16T00:15:13.584306+0000","last_active":"2024-09-16T00:16:09.018162+0000","last_peered":"2024-09-16T00:16:09.018162+0000","last_clean":"2024-09-16T00:16:09.018162+0000","last_became_active":"2024-09-16T00:15:13.583840+0000","last_became_peered":"2024-09-16T00:15:13.583840+0000","last_unstale":"2024-09-16T00:16:09.018162+0000","last_undegraded":"2024-09-16T00:16:09.018162+0000","last_fullsized":"2024-09-16T00:16:09.018162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:36.660458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017267+0000","last_change":"2024-09-16T00:15:06.538902+0000","last_active":"2024-09-16T00:16:09.017267+0000","last_peered":"2024-09-16T00:16:09.017267+0000","last_clean":"2024-09-16T00:16:09.017267+0000","last_became_active":"2024-09-16T00:15:06.538537+0000","last_became_peered":"2024-09-16T00:15:06.538537+0000","last_unstale":"2024-09-16T00:16:09.017267+0000","last_undegraded":"2024-09-16T00:16:09.017267+0000","last_fullsized":"2024-09-16T00:16:09.017267+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:38.411711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.013859+0000","last_change":"2024-09-16T00:15:04.374762+0000","last_active":"2024-09-16T00:16:09.013859+0000","last_peered":"2024-09-16T00:16:09.013859+0000","last_clean":"2024-09-16T00:16:09.013859+0000","last_became_active":"2024-09-16T00:15:04.374394+0000","last_became_peered":"2024-09-16T00:15:04.374394+0000","last_unstale":"2024-09-16T00:16:09.013859+0000","last_undegraded":"2024-09-16T00:16:09.013859+0000","last_fullsized":"2024-09-16T00:16:09.013859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:47.075021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018397+0000","last_change":"2024-09-16T00:15:13.638306+0000","last_active":"2024-09-16T00:16:09.018397+0000","last_peered":"2024-09-16T00:16:09.018397+0000","last_clean":"2024-09-16T00:16:09.018397+0000","last_became_active":"2024-09-16T00:15:13.637931+0000","last_became_peered":"2024-09-16T00:15:13.637931+0000","last_unstale":"2024-09-16T00:16:09.018397+0000","last_undegraded":"2024-09-16T00:16:09.018397+0000","last_fullsized":"2024-09-16T00:16:09.018397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:41.385174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017001+0000","last_change":"2024-09-16T00:15:06.548209+0000","last_active":"2024-09-16T00:16:09.017001+0000","last_peered":"2024-09-16T00:16:09.017001+0000","last_clean":"2024-09-16T00:16:09.017001+0000","last_became_active":"2024-09-16T00:15:06.547846+0000","last_became_peered":"2024-09-16T00:15:06.547846+0000","last_unstale":"2024-09-16T00:16:09.017001+0000","last_undegraded":"2024-09-16T00:16:09.017001+0000","last_fullsized":"2024-09-16T00:16:09.017001+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:15.821847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.024470+0000","last_change":"2024-09-16T00:16:09.023918+0000","last_active":"2024-09-16T00:16:09.024470+0000","last_peered":"2024-09-16T00:16:09.024470+0000","last_clean":"2024-09-16T00:16:09.024470+0000","last_became_active":"2024-09-16T00:16:09.023230+0000","last_became_peered":"2024-09-16T00:16:09.023230+0000","last_unstale":"2024-09-16T00:16:09.024470+0000","last_undegraded":"2024-09-16T00:16:09.024470+0000","last_fullsized":"2024-09-16T00:16:09.024470+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:31.527863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019672+0000","last_change":"2024-09-16T00:15:13.579387+0000","last_active":"2024-09-16T00:16:09.019672+0000","last_peered":"2024-09-16T00:16:09.019672+0000","last_clean":"2024-09-16T00:16:09.019672+0000","last_became_active":"2024-09-16T00:15:13.578996+0000","last_became_peered":"2024-09-16T00:15:13.578996+0000","last_unstale":"2024-09-16T00:16:09.019672+0000","last_undegraded":"2024-09-16T00:16:09.019672+0000","last_fullsized":"2024-09-16T00:16:09.019672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:51.703060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.029836+0000","last_change":"2024-09-16T00:15:06.540396+0000","last_active":"2024-09-16T00:16:09.029836+0000","last_peered":"2024-09-16T00:16:09.029836+0000","last_clean":"2024-09-16T00:16:09.029836+0000","last_became_active":"2024-09-16T00:15:06.540084+0000","last_became_peered":"2024-09-16T00:15:06.540084+0000","last_unstale":"2024-09-16T00:16:09.029836+0000","last_undegraded":"2024-09-16T00:16:09.029836+0000","last_fullsized":"2024-09-16T00:16:09.029836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:46.477143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":93,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:43.254034+0000","last_change":"2024-09-16T00:15:04.373896+0000","last_active":"2024-09-16T00:16:43.254034+0000","last_peered":"2024-09-16T00:16:43.254034+0000","last_clean":"2024-09-16T00:16:43.254034+0000","last_became_active":"2024-09-16T00:15:04.373493+0000","last_became_peered":"2024-09-16T00:15:04.373493+0000","last_unstale":"2024-09-16T00:16:43.254034+0000","last_undegraded":"2024-09-16T00:16:43.254034+0000","last_fullsized":"2024-09-16T00:16:43.254034+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:17:10.767267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":55,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019826+0000","last_change":"2024-09-16T00:15:13.638575+0000","last_active":"2024-09-16T00:16:09.019826+0000","last_peered":"2024-09-16T00:16:09.019826+0000","last_clean":"2024-09-16T00:16:09.019826+0000","last_became_active":"2024-09-16T00:15:13.638144+0000","last_became_peered":"2024-09-16T00:15:13.638144+0000","last_unstale":"2024-09-16T00:16:09.019826+0000","last_undegraded":"2024-09-16T00:16:09.019826+0000","last_fullsized":"2024-09-16T00:16:09.019826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:39.345011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019993+0000","last_change":"2024-09-16T00:15:06.548852+0000","last_active":"2024-09-16T00:16:09.019993+0000","last_peered":"2024-09-16T00:16:09.019993+0000","last_clean":"2024-09-16T00:16:09.019993+0000","last_became_active":"2024-09-16T00:15:06.548238+0000","last_became_peered":"2024-09-16T00:15:06.548238+0000","last_unstale":"2024-09-16T00:16:09.019993+0000","last_undegraded":"2024-09-16T00:16:09.019993+0000","last_fullsized":"2024-09-16T00:16:09.019993+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:16.115338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.014829+0000","last_change":"2024-09-16T00:15:04.350753+0000","last_active":"2024-09-16T00:16:09.014829+0000","last_peered":"2024-09-16T00:16:09.014829+0000","last_clean":"2024-09-16T00:16:09.014829+0000","last_became_active":"2024-09-16T00:15:04.350406+0000","last_became_peered":"2024-09-16T00:15:04.350406+0000","last_unstale":"2024-09-16T00:16:09.014829+0000","last_undegraded":"2024-09-16T00:16:09.014829+0000","last_fullsized":"2024-09-16T00:16:09.014829+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:09:54.737287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030014+0000","last_change":"2024-09-16T00:15:13.639302+0000","last_active":"2024-09-16T00:16:09.030014+0000","last_peered":"2024-09-16T00:16:09.030014+0000","last_clean":"2024-09-16T00:16:09.030014+0000","last_became_active":"2024-09-16T00:15:13.638863+0000","last_became_peered":"2024-09-16T00:15:13.638863+0000","last_unstale":"2024-09-16T00:16:09.030014+0000","last_undegraded":"2024-09-16T00:16:09.030014+0000","last_fullsized":"2024-09-16T00:16:09.030014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:23.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022107+0000","last_change":"2024-09-16T00:15:06.545797+0000","last_active":"2024-09-16T00:16:09.022107+0000","last_peered":"2024-09-16T00:16:09.022107+0000","last_clean":"2024-09-16T00:16:09.022107+0000","last_became_active":"2024-09-16T00:15:06.545477+0000","last_became_peered":"2024-09-16T00:15:06.545477+0000","last_unstale":"2024-09-16T00:16:09.022107+0000","last_undegraded":"2024-09-16T00:16:09.022107+0000","last_fullsized":"2024-09-16T00:16:09.022107+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.887183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016777+0000","last_change":"2024-09-16T00:15:04.342216+0000","last_active":"2024-09-16T00:16:09.016777+0000","last_peered":"2024-09-16T00:16:09.016777+0000","last_clean":"2024-09-16T00:16:09.016777+0000","last_became_active":"2024-09-16T00:15:04.341763+0000","last_became_peered":"2024-09-16T00:15:04.341763+0000","last_unstale":"2024-09-16T00:16:09.016777+0000","last_undegraded":"2024-09-16T00:16:09.016777+0000","last_fullsized":"2024-09-16T00:16:09.016777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:44.938527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015058+0000","last_change":"2024-09-16T00:13:51.108571+0000","last_active":"2024-09-16T00:16:09.015058+0000","last_peered":"2024-09-16T00:16:09.015058+0000","last_clean":"2024-09-16T00:16:09.015058+0000","last_became_active":"2024-09-16T00:13:51.108088+0000","last_became_peered":"2024-09-16T00:13:51.108088+0000","last_unstale":"2024-09-16T00:16:09.015058+0000","last_undegraded":"2024-09-16T00:16:09.015058+0000","last_fullsized":"2024-09-16T00:16:09.015058+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_clean_scrub_stamp":"2024-09-16T00:10:24.533807+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:02.511064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015440+0000","last_change":"2024-09-16T00:15:13.588665+0000","last_active":"2024-09-16T00:16:09.015440+0000","last_peered":"2024-09-16T00:16:09.015440+0000","last_clean":"2024-09-16T00:16:09.015440+0000","last_became_active":"2024-09-16T00:15:13.588222+0000","last_became_peered":"2024-09-16T00:15:13.588222+0000","last_unstale":"2024-09-16T00:16:09.015440+0000","last_undegraded":"2024-09-16T00:16:09.015440+0000","last_fullsized":"2024-09-16T00:16:09.015440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:59:59.965962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015757+0000","last_change":"2024-09-16T00:15:06.553114+0000","last_active":"2024-09-16T00:16:09.015757+0000","last_peered":"2024-09-16T00:16:09.015757+0000","last_clean":"2024-09-16T00:16:09.015757+0000","last_became_active":"2024-09-16T00:15:06.552788+0000","last_became_peered":"2024-09-16T00:15:06.552788+0000","last_unstale":"2024-09-16T00:16:09.015757+0000","last_undegraded":"2024-09-16T00:16:09.015757+0000","last_fullsized":"2024-09-16T00:16:09.015757+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:05.473847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.031113+0000","last_change":"2024-09-16T00:16:09.030747+0000","last_active":"2024-09-16T00:16:09.031113+0000","last_peered":"2024-09-16T00:16:09.031113+0000","last_clean":"2024-09-16T00:16:09.031113+0000","last_became_active":"2024-09-16T00:16:09.021208+0000","last_became_peered":"2024-09-16T00:16:09.021208+0000","last_unstale":"2024-09-16T00:16:09.031113+0000","last_undegraded":"2024-09-16T00:16:09.031113+0000","last_fullsized":"2024-09-16T00:16:09.031113+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:40.605498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021707+0000","last_change":"2024-09-16T00:15:13.575606+0000","last_active":"2024-09-16T00:16:09.021707+0000","last_peered":"2024-09-16T00:16:09.021707+0000","last_clean":"2024-09-16T00:16:09.021707+0000","last_became_active":"2024-09-16T00:15:13.575131+0000","last_became_peered":"2024-09-16T00:15:13.575131+0000","last_unstale":"2024-09-16T00:16:09.021707+0000","last_undegraded":"2024-09-16T00:16:09.021707+0000","last_fullsized":"2024-09-16T00:16:09.021707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:41.636058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015223+0000","last_change":"2024-09-16T00:15:06.552956+0000","last_active":"2024-09-16T00:16:09.015223+0000","last_peered":"2024-09-16T00:16:09.015223+0000","last_clean":"2024-09-16T00:16:09.015223+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:16:09.015223+0000","last_undegraded":"2024-09-16T00:16:09.015223+0000","last_fullsized":"2024-09-16T00:16:09.015223+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:36:20.739676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030263+0000","last_change":"2024-09-16T00:15:04.349095+0000","last_active":"2024-09-16T00:16:09.030263+0000","last_peered":"2024-09-16T00:16:09.030263+0000","last_clean":"2024-09-16T00:16:09.030263+0000","last_became_active":"2024-09-16T00:15:04.348698+0000","last_became_peered":"2024-09-16T00:15:04.348698+0000","last_unstale":"2024-09-16T00:16:09.030263+0000","last_undegraded":"2024-09-16T00:16:09.030263+0000","last_fullsized":"2024-09-16T00:16:09.030263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:42.282882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019445+0000","last_change":"2024-09-16T00:15:13.580494+0000","last_active":"2024-09-16T00:16:09.019445+0000","last_peered":"2024-09-16T00:16:09.019445+0000","last_clean":"2024-09-16T00:16:09.019445+0000","last_became_active":"2024-09-16T00:15:13.580091+0000","last_became_peered":"2024-09-16T00:15:13.580091+0000","last_unstale":"2024-09-16T00:16:09.019445+0000","last_undegraded":"2024-09-16T00:16:09.019445+0000","last_fullsized":"2024-09-16T00:16:09.019445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:17.314919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.014736+0000","last_change":"2024-09-16T00:15:06.544859+0000","last_active":"2024-09-16T00:16:09.014736+0000","last_peered":"2024-09-16T00:16:09.014736+0000","last_clean":"2024-09-16T00:16:09.014736+0000","last_became_active":"2024-09-16T00:15:06.544289+0000","last_became_peered":"2024-09-16T00:15:06.544289+0000","last_unstale":"2024-09-16T00:16:09.014736+0000","last_undegraded":"2024-09-16T00:16:09.014736+0000","last_fullsized":"2024-09-16T00:16:09.014736+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:52.622477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020216+0000","last_change":"2024-09-16T00:15:04.351313+0000","last_active":"2024-09-16T00:16:09.020216+0000","last_peered":"2024-09-16T00:16:09.020216+0000","last_clean":"2024-09-16T00:16:09.020216+0000","last_became_active":"2024-09-16T00:15:04.350928+0000","last_became_peered":"2024-09-16T00:15:04.350928+0000","last_unstale":"2024-09-16T00:16:09.020216+0000","last_undegraded":"2024-09-16T00:16:09.020216+0000","last_fullsized":"2024-09-16T00:16:09.020216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:48.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.029450+0000","last_change":"2024-09-16T00:15:13.578573+0000","last_active":"2024-09-16T00:16:09.029450+0000","last_peered":"2024-09-16T00:16:09.029450+0000","last_clean":"2024-09-16T00:16:09.029450+0000","last_became_active":"2024-09-16T00:15:13.578117+0000","last_became_peered":"2024-09-16T00:15:13.578117+0000","last_unstale":"2024-09-16T00:16:09.029450+0000","last_undegraded":"2024-09-16T00:16:09.029450+0000","last_fullsized":"2024-09-16T00:16:09.029450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:53.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018124+0000","last_change":"2024-09-16T00:15:06.553548+0000","last_active":"2024-09-16T00:16:09.018124+0000","last_peered":"2024-09-16T00:16:09.018124+0000","last_clean":"2024-09-16T00:16:09.018124+0000","last_became_active":"2024-09-16T00:15:06.552963+0000","last_became_peered":"2024-09-16T00:15:06.552963+0000","last_unstale":"2024-09-16T00:16:09.018124+0000","last_undegraded":"2024-09-16T00:16:09.018124+0000","last_fullsized":"2024-09-16T00:16:09.018124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:43.455881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021425+0000","last_change":"2024-09-16T00:15:04.351005+0000","last_active":"2024-09-16T00:16:09.021425+0000","last_peered":"2024-09-16T00:16:09.021425+0000","last_clean":"2024-09-16T00:16:09.021425+0000","last_became_active":"2024-09-16T00:15:04.350614+0000","last_became_peered":"2024-09-16T00:15:04.350614+0000","last_unstale":"2024-09-16T00:16:09.021425+0000","last_undegraded":"2024-09-16T00:16:09.021425+0000","last_fullsized":"2024-09-16T00:16:09.021425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:39.196360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.024668+0000","last_change":"2024-09-16T00:16:09.023796+0000","last_active":"2024-09-16T00:16:09.024668+0000","last_peered":"2024-09-16T00:16:09.024668+0000","last_clean":"2024-09-16T00:16:09.024668+0000","last_became_active":"2024-09-16T00:16:09.023403+0000","last_became_peered":"2024-09-16T00:16:09.023403+0000","last_unstale":"2024-09-16T00:16:09.024668+0000","last_undegraded":"2024-09-16T00:16:09.024668+0000","last_fullsized":"2024-09-16T00:16:09.024668+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:59.488315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.029742+0000","last_change":"2024-09-16T00:15:06.546325+0000","last_active":"2024-09-16T00:16:09.029742+0000","last_peered":"2024-09-16T00:16:09.029742+0000","last_clean":"2024-09-16T00:16:09.029742+0000","last_became_active":"2024-09-16T00:15:06.546028+0000","last_became_peered":"2024-09-16T00:15:06.546028+0000","last_unstale":"2024-09-16T00:16:09.029742+0000","last_undegraded":"2024-09-16T00:16:09.029742+0000","last_fullsized":"2024-09-16T00:16:09.029742+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:15.049592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018478+0000","last_change":"2024-09-16T00:15:04.373818+0000","last_active":"2024-09-16T00:16:09.018478+0000","last_peered":"2024-09-16T00:16:09.018478+0000","last_clean":"2024-09-16T00:16:09.018478+0000","last_became_active":"2024-09-16T00:15:04.373406+0000","last_became_peered":"2024-09-16T00:15:04.373406+0000","last_unstale":"2024-09-16T00:16:09.018478+0000","last_undegraded":"2024-09-16T00:16:09.018478+0000","last_fullsized":"2024-09-16T00:16:09.018478+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:08.402660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020108+0000","last_change":"2024-09-16T00:15:13.588669+0000","last_active":"2024-09-16T00:16:09.020108+0000","last_peered":"2024-09-16T00:16:09.020108+0000","last_clean":"2024-09-16T00:16:09.020108+0000","last_became_active":"2024-09-16T00:15:13.588229+0000","last_became_peered":"2024-09-16T00:15:13.588229+0000","last_unstale":"2024-09-16T00:16:09.020108+0000","last_undegraded":"2024-09-16T00:16:09.020108+0000","last_fullsized":"2024-09-16T00:16:09.020108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:30.157296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.014528+0000","last_change":"2024-09-16T00:15:06.550221+0000","last_active":"2024-09-16T00:16:09.014528+0000","last_peered":"2024-09-16T00:16:09.014528+0000","last_clean":"2024-09-16T00:16:09.014528+0000","last_became_active":"2024-09-16T00:15:06.549914+0000","last_became_peered":"2024-09-16T00:15:06.549914+0000","last_unstale":"2024-09-16T00:16:09.014528+0000","last_undegraded":"2024-09-16T00:16:09.014528+0000","last_fullsized":"2024-09-16T00:16:09.014528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:18:26.824523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":95,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:43.254602+0000","last_change":"2024-09-16T00:15:04.344389+0000","last_active":"2024-09-16T00:16:43.254602+0000","last_peered":"2024-09-16T00:16:43.254602+0000","last_clean":"2024-09-16T00:16:43.254602+0000","last_became_active":"2024-09-16T00:15:04.343956+0000","last_became_peered":"2024-09-16T00:15:04.343956+0000","last_unstale":"2024-09-16T00:16:43.254602+0000","last_undegraded":"2024-09-16T00:16:43.254602+0000","last_fullsized":"2024-09-16T00:16:43.254602+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:34.019861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.024716+0000","last_change":"2024-09-16T00:16:09.024050+0000","last_active":"2024-09-16T00:16:09.024716+0000","last_peered":"2024-09-16T00:16:09.024716+0000","last_clean":"2024-09-16T00:16:09.024716+0000","last_became_active":"2024-09-16T00:16:09.023371+0000","last_became_peered":"2024-09-16T00:16:09.023371+0000","last_unstale":"2024-09-16T00:16:09.024716+0000","last_undegraded":"2024-09-16T00:16:09.024716+0000","last_fullsized":"2024-09-16T00:16:09.024716+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:35:19.020370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016292+0000","last_change":"2024-09-16T00:15:06.542744+0000","last_active":"2024-09-16T00:16:09.016292+0000","last_peered":"2024-09-16T00:16:09.016292+0000","last_clean":"2024-09-16T00:16:09.016292+0000","last_became_active":"2024-09-16T00:15:06.541868+0000","last_became_peered":"2024-09-16T00:15:06.541868+0000","last_unstale":"2024-09-16T00:16:09.016292+0000","last_undegraded":"2024-09-16T00:16:09.016292+0000","last_fullsized":"2024-09-16T00:16:09.016292+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:10:06.508682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030023+0000","last_change":"2024-09-16T00:15:04.349556+0000","last_active":"2024-09-16T00:16:09.030023+0000","last_peered":"2024-09-16T00:16:09.030023+0000","last_clean":"2024-09-16T00:16:09.030023+0000","last_became_active":"2024-09-16T00:15:04.349242+0000","last_became_peered":"2024-09-16T00:15:04.349242+0000","last_unstale":"2024-09-16T00:16:09.030023+0000","last_undegraded":"2024-09-16T00:16:09.030023+0000","last_fullsized":"2024-09-16T00:16:09.030023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:22:18.459157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018537+0000","last_change":"2024-09-16T00:15:13.584109+0000","last_active":"2024-09-16T00:16:09.018537+0000","last_peered":"2024-09-16T00:16:09.018537+0000","last_clean":"2024-09-16T00:16:09.018537+0000","last_became_active":"2024-09-16T00:15:13.583718+0000","last_became_peered":"2024-09-16T00:15:13.583718+0000","last_unstale":"2024-09-16T00:16:09.018537+0000","last_undegraded":"2024-09-16T00:16:09.018537+0000","last_fullsized":"2024-09-16T00:16:09.018537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:29.416254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020028+0000","last_change":"2024-09-16T00:15:06.548247+0000","last_active":"2024-09-16T00:16:09.020028+0000","last_peered":"2024-09-16T00:16:09.020028+0000","last_clean":"2024-09-16T00:16:09.020028+0000","last_became_active":"2024-09-16T00:15:06.547864+0000","last_became_peered":"2024-09-16T00:15:06.547864+0000","last_unstale":"2024-09-16T00:16:09.020028+0000","last_undegraded":"2024-09-16T00:16:09.020028+0000","last_fullsized":"2024-09-16T00:16:09.020028+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:25.350197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022815+0000","last_change":"2024-09-16T00:15:04.350965+0000","last_active":"2024-09-16T00:16:09.022815+0000","last_peered":"2024-09-16T00:16:09.022815+0000","last_clean":"2024-09-16T00:16:09.022815+0000","last_became_active":"2024-09-16T00:15:04.349124+0000","last_became_peered":"2024-09-16T00:15:04.349124+0000","last_unstale":"2024-09-16T00:16:09.022815+0000","last_undegraded":"2024-09-16T00:16:09.022815+0000","last_fullsized":"2024-09-16T00:16:09.022815+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:40.859220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.029882+0000","last_change":"2024-09-16T00:15:13.581718+0000","last_active":"2024-09-16T00:16:09.029882+0000","last_peered":"2024-09-16T00:16:09.029882+0000","last_clean":"2024-09-16T00:16:09.029882+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:16:09.029882+0000","last_undegraded":"2024-09-16T00:16:09.029882+0000","last_fullsized":"2024-09-16T00:16:09.029882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:08.810606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019805+0000","last_change":"2024-09-16T00:15:06.544507+0000","last_active":"2024-09-16T00:16:09.019805+0000","last_peered":"2024-09-16T00:16:09.019805+0000","last_clean":"2024-09-16T00:16:09.019805+0000","last_became_active":"2024-09-16T00:15:06.544178+0000","last_became_peered":"2024-09-16T00:15:06.544178+0000","last_unstale":"2024-09-16T00:16:09.019805+0000","last_undegraded":"2024-09-16T00:16:09.019805+0000","last_fullsized":"2024-09-16T00:16:09.019805+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:38.494266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"88'32","reported_seq":199,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:43.084950+0000","last_change":"2024-09-16T00:16:09.039763+0000","last_active":"2024-09-16T00:16:43.084950+0000","last_peered":"2024-09-16T00:16:43.084950+0000","last_clean":"2024-09-16T00:16:43.084950+0000","last_became_active":"2024-09-16T00:16:09.022939+0000","last_became_peered":"2024-09-16T00:16:09.022939+0000","last_unstale":"2024-09-16T00:16:43.084950+0000","last_undegraded":"2024-09-16T00:16:43.084950+0000","last_fullsized":"2024-09-16T00:16:43.084950+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:56.431159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62931,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":112,"num_read_kb":108,"num_write":33,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60943,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.013565+0000","last_change":"2024-09-16T00:15:13.585928+0000","last_active":"2024-09-16T00:16:09.013565+0000","last_peered":"2024-09-16T00:16:09.013565+0000","last_clean":"2024-09-16T00:16:09.013565+0000","last_became_active":"2024-09-16T00:15:13.582940+0000","last_became_peered":"2024-09-16T00:15:13.582940+0000","last_unstale":"2024-09-16T00:16:09.013565+0000","last_undegraded":"2024-09-16T00:16:09.013565+0000","last_fullsized":"2024-09-16T00:16:09.013565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:57:13.251978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021707+0000","last_change":"2024-09-16T00:15:06.546238+0000","last_active":"2024-09-16T00:16:09.021707+0000","last_peered":"2024-09-16T00:16:09.021707+0000","last_clean":"2024-09-16T00:16:09.021707+0000","last_became_active":"2024-09-16T00:15:06.545919+0000","last_became_peered":"2024-09-16T00:15:06.545919+0000","last_unstale":"2024-09-16T00:16:09.021707+0000","last_undegraded":"2024-09-16T00:16:09.021707+0000","last_fullsized":"2024-09-16T00:16:09.021707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:09.180832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030189+0000","last_change":"2024-09-16T00:15:04.376028+0000","last_active":"2024-09-16T00:16:09.030189+0000","last_peered":"2024-09-16T00:16:09.030189+0000","last_clean":"2024-09-16T00:16:09.030189+0000","last_became_active":"2024-09-16T00:15:04.375633+0000","last_became_peered":"2024-09-16T00:15:04.375633+0000","last_unstale":"2024-09-16T00:16:09.030189+0000","last_undegraded":"2024-09-16T00:16:09.030189+0000","last_fullsized":"2024-09-16T00:16:09.030189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:44.446342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.025524+0000","last_change":"2024-09-16T00:16:09.025524+0000","last_active":"2024-09-16T00:16:09.025524+0000","last_peered":"2024-09-16T00:16:09.025524+0000","last_clean":"2024-09-16T00:16:09.025524+0000","last_became_active":"2024-09-16T00:16:09.025035+0000","last_became_peered":"2024-09-16T00:16:09.025035+0000","last_unstale":"2024-09-16T00:16:09.025524+0000","last_undegraded":"2024-09-16T00:16:09.025524+0000","last_fullsized":"2024-09-16T00:16:09.025524+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:12.367244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020373+0000","last_change":"2024-09-16T00:15:06.550864+0000","last_active":"2024-09-16T00:16:09.020373+0000","last_peered":"2024-09-16T00:16:09.020373+0000","last_clean":"2024-09-16T00:16:09.020373+0000","last_became_active":"2024-09-16T00:15:06.547956+0000","last_became_peered":"2024-09-16T00:15:06.547956+0000","last_unstale":"2024-09-16T00:16:09.020373+0000","last_undegraded":"2024-09-16T00:16:09.020373+0000","last_fullsized":"2024-09-16T00:16:09.020373+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:26.102175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016015+0000","last_change":"2024-09-16T00:15:04.350349+0000","last_active":"2024-09-16T00:16:09.016015+0000","last_peered":"2024-09-16T00:16:09.016015+0000","last_clean":"2024-09-16T00:16:09.016015+0000","last_became_active":"2024-09-16T00:15:04.349951+0000","last_became_peered":"2024-09-16T00:15:04.349951+0000","last_unstale":"2024-09-16T00:16:09.016015+0000","last_undegraded":"2024-09-16T00:16:09.016015+0000","last_fullsized":"2024-09-16T00:16:09.016015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:46.000831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.025774+0000","last_change":"2024-09-16T00:16:09.025774+0000","last_active":"2024-09-16T00:16:09.025774+0000","last_peered":"2024-09-16T00:16:09.025774+0000","last_clean":"2024-09-16T00:16:09.025774+0000","last_became_active":"2024-09-16T00:16:09.025308+0000","last_became_peered":"2024-09-16T00:16:09.025308+0000","last_unstale":"2024-09-16T00:16:09.025774+0000","last_undegraded":"2024-09-16T00:16:09.025774+0000","last_fullsized":"2024-09-16T00:16:09.025774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:16.664850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021661+0000","last_change":"2024-09-16T00:15:06.552919+0000","last_active":"2024-09-16T00:16:09.021661+0000","last_peered":"2024-09-16T00:16:09.021661+0000","last_clean":"2024-09-16T00:16:09.021661+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:16:09.021661+0000","last_undegraded":"2024-09-16T00:16:09.021661+0000","last_fullsized":"2024-09-16T00:16:09.021661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:47:38.651837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:41.251681+0000","last_change":"2024-09-16T00:15:04.351116+0000","last_active":"2024-09-16T00:16:41.251681+0000","last_peered":"2024-09-16T00:16:41.251681+0000","last_clean":"2024-09-16T00:16:41.251681+0000","last_became_active":"2024-09-16T00:15:04.350728+0000","last_became_peered":"2024-09-16T00:15:04.350728+0000","last_unstale":"2024-09-16T00:16:41.251681+0000","last_undegraded":"2024-09-16T00:16:41.251681+0000","last_fullsized":"2024-09-16T00:16:41.251681+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:38.314019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":53,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022243+0000","last_change":"2024-09-16T00:15:13.575961+0000","last_active":"2024-09-16T00:16:09.022243+0000","last_peered":"2024-09-16T00:16:09.022243+0000","last_clean":"2024-09-16T00:16:09.022243+0000","last_became_active":"2024-09-16T00:15:13.575550+0000","last_became_peered":"2024-09-16T00:15:13.575550+0000","last_unstale":"2024-09-16T00:16:09.022243+0000","last_undegraded":"2024-09-16T00:16:09.022243+0000","last_fullsized":"2024-09-16T00:16:09.022243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:05.515588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020094+0000","last_change":"2024-09-16T00:15:13.583583+0000","last_active":"2024-09-16T00:16:09.020094+0000","last_peered":"2024-09-16T00:16:09.020094+0000","last_clean":"2024-09-16T00:16:09.020094+0000","last_became_active":"2024-09-16T00:15:13.583145+0000","last_became_peered":"2024-09-16T00:15:13.583145+0000","last_unstale":"2024-09-16T00:16:09.020094+0000","last_undegraded":"2024-09-16T00:16:09.020094+0000","last_fullsized":"2024-09-16T00:16:09.020094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:07.320337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.013547+0000","last_change":"2024-09-16T00:15:06.552017+0000","last_active":"2024-09-16T00:16:09.013547+0000","last_peered":"2024-09-16T00:16:09.013547+0000","last_clean":"2024-09-16T00:16:09.013547+0000","last_became_active":"2024-09-16T00:15:06.551688+0000","last_became_peered":"2024-09-16T00:15:06.551688+0000","last_unstale":"2024-09-16T00:16:09.013547+0000","last_undegraded":"2024-09-16T00:16:09.013547+0000","last_fullsized":"2024-09-16T00:16:09.013547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:39.951856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030263+0000","last_change":"2024-09-16T00:15:04.350826+0000","last_active":"2024-09-16T00:16:09.030263+0000","last_peered":"2024-09-16T00:16:09.030263+0000","last_clean":"2024-09-16T00:16:09.030263+0000","last_became_active":"2024-09-16T00:15:04.350294+0000","last_became_peered":"2024-09-16T00:15:04.350294+0000","last_unstale":"2024-09-16T00:16:09.030263+0000","last_undegraded":"2024-09-16T00:16:09.030263+0000","last_fullsized":"2024-09-16T00:16:09.030263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:49.146709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017920+0000","last_change":"2024-09-16T00:15:13.580184+0000","last_active":"2024-09-16T00:16:09.017920+0000","last_peered":"2024-09-16T00:16:09.017920+0000","last_clean":"2024-09-16T00:16:09.017920+0000","last_became_active":"2024-09-16T00:15:13.579849+0000","last_became_peered":"2024-09-16T00:15:13.579849+0000","last_unstale":"2024-09-16T00:16:09.017920+0000","last_undegraded":"2024-09-16T00:16:09.017920+0000","last_fullsized":"2024-09-16T00:16:09.017920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:40.613127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021921+0000","last_change":"2024-09-16T00:15:06.545796+0000","last_active":"2024-09-16T00:16:09.021921+0000","last_peered":"2024-09-16T00:16:09.021921+0000","last_clean":"2024-09-16T00:16:09.021921+0000","last_became_active":"2024-09-16T00:15:06.544815+0000","last_became_peered":"2024-09-16T00:15:06.544815+0000","last_unstale":"2024-09-16T00:16:09.021921+0000","last_undegraded":"2024-09-16T00:16:09.021921+0000","last_fullsized":"2024-09-16T00:16:09.021921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:51:25.921949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019965+0000","last_change":"2024-09-16T00:15:04.340101+0000","last_active":"2024-09-16T00:16:09.019965+0000","last_peered":"2024-09-16T00:16:09.019965+0000","last_clean":"2024-09-16T00:16:09.019965+0000","last_became_active":"2024-09-16T00:15:04.339788+0000","last_became_peered":"2024-09-16T00:15:04.339788+0000","last_unstale":"2024-09-16T00:16:09.019965+0000","last_undegraded":"2024-09-16T00:16:09.019965+0000","last_fullsized":"2024-09-16T00:16:09.019965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:08.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.028910+0000","last_change":"2024-09-16T00:16:09.028910+0000","last_active":"2024-09-16T00:16:09.028910+0000","last_peered":"2024-09-16T00:16:09.028910+0000","last_clean":"2024-09-16T00:16:09.028910+0000","last_became_active":"2024-09-16T00:16:09.023302+0000","last_became_peered":"2024-09-16T00:16:09.023302+0000","last_unstale":"2024-09-16T00:16:09.028910+0000","last_undegraded":"2024-09-16T00:16:09.028910+0000","last_fullsized":"2024-09-16T00:16:09.028910+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:35.293676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030364+0000","last_change":"2024-09-16T00:15:06.547496+0000","last_active":"2024-09-16T00:16:09.030364+0000","last_peered":"2024-09-16T00:16:09.030364+0000","last_clean":"2024-09-16T00:16:09.030364+0000","last_became_active":"2024-09-16T00:15:06.547177+0000","last_became_peered":"2024-09-16T00:15:06.547177+0000","last_unstale":"2024-09-16T00:16:09.030364+0000","last_undegraded":"2024-09-16T00:16:09.030364+0000","last_fullsized":"2024-09-16T00:16:09.030364+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:13.288827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022002+0000","last_change":"2024-09-16T00:15:13.581966+0000","last_active":"2024-09-16T00:16:09.022002+0000","last_peered":"2024-09-16T00:16:09.022002+0000","last_clean":"2024-09-16T00:16:09.022002+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:16:09.022002+0000","last_undegraded":"2024-09-16T00:16:09.022002+0000","last_fullsized":"2024-09-16T00:16:09.022002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.132773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:43.255440+0000","last_change":"2024-09-16T00:15:04.351765+0000","last_active":"2024-09-16T00:16:43.255440+0000","last_peered":"2024-09-16T00:16:43.255440+0000","last_clean":"2024-09-16T00:16:43.255440+0000","last_became_active":"2024-09-16T00:15:04.351398+0000","last_became_peered":"2024-09-16T00:15:04.351398+0000","last_unstale":"2024-09-16T00:16:43.255440+0000","last_undegraded":"2024-09-16T00:16:43.255440+0000","last_fullsized":"2024-09-16T00:16:43.255440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:14:28.762657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":57,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018785+0000","last_change":"2024-09-16T00:15:06.541422+0000","last_active":"2024-09-16T00:16:09.018785+0000","last_peered":"2024-09-16T00:16:09.018785+0000","last_clean":"2024-09-16T00:16:09.018785+0000","last_became_active":"2024-09-16T00:15:06.541083+0000","last_became_peered":"2024-09-16T00:15:06.541083+0000","last_unstale":"2024-09-16T00:16:09.018785+0000","last_undegraded":"2024-09-16T00:16:09.018785+0000","last_fullsized":"2024-09-16T00:16:09.018785+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:30.138972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022088+0000","last_change":"2024-09-16T00:15:13.576877+0000","last_active":"2024-09-16T00:16:09.022088+0000","last_peered":"2024-09-16T00:16:09.022088+0000","last_clean":"2024-09-16T00:16:09.022088+0000","last_became_active":"2024-09-16T00:15:13.576480+0000","last_became_peered":"2024-09-16T00:15:13.576480+0000","last_unstale":"2024-09-16T00:16:09.022088+0000","last_undegraded":"2024-09-16T00:16:09.022088+0000","last_fullsized":"2024-09-16T00:16:09.022088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:47.322219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018767+0000","last_change":"2024-09-16T00:15:06.550329+0000","last_active":"2024-09-16T00:16:09.018767+0000","last_peered":"2024-09-16T00:16:09.018767+0000","last_clean":"2024-09-16T00:16:09.018767+0000","last_became_active":"2024-09-16T00:15:06.550008+0000","last_became_peered":"2024-09-16T00:15:06.550008+0000","last_unstale":"2024-09-16T00:16:09.018767+0000","last_undegraded":"2024-09-16T00:16:09.018767+0000","last_fullsized":"2024-09-16T00:16:09.018767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:38.096126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021880+0000","last_change":"2024-09-16T00:15:06.550368+0000","last_active":"2024-09-16T00:16:09.021880+0000","last_peered":"2024-09-16T00:16:09.021880+0000","last_clean":"2024-09-16T00:16:09.021880+0000","last_became_active":"2024-09-16T00:15:06.549985+0000","last_became_peered":"2024-09-16T00:15:06.549985+0000","last_unstale":"2024-09-16T00:16:09.021880+0000","last_undegraded":"2024-09-16T00:16:09.021880+0000","last_fullsized":"2024-09-16T00:16:09.021880+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:16.992110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021135+0000","last_change":"2024-09-16T00:15:06.546713+0000","last_active":"2024-09-16T00:16:09.021135+0000","last_peered":"2024-09-16T00:16:09.021135+0000","last_clean":"2024-09-16T00:16:09.021135+0000","last_became_active":"2024-09-16T00:15:06.546209+0000","last_became_peered":"2024-09-16T00:15:06.546209+0000","last_unstale":"2024-09-16T00:16:09.021135+0000","last_undegraded":"2024-09-16T00:16:09.021135+0000","last_fullsized":"2024-09-16T00:16:09.021135+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:43.913748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019555+0000","last_change":"2024-09-16T00:15:06.550490+0000","last_active":"2024-09-16T00:16:09.019555+0000","last_peered":"2024-09-16T00:16:09.019555+0000","last_clean":"2024-09-16T00:16:09.019555+0000","last_became_active":"2024-09-16T00:15:06.549311+0000","last_became_peered":"2024-09-16T00:15:06.549311+0000","last_unstale":"2024-09-16T00:16:09.019555+0000","last_undegraded":"2024-09-16T00:16:09.019555+0000","last_fullsized":"2024-09-16T00:16:09.019555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:10:01.995357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019865+0000","last_change":"2024-09-16T00:15:06.548115+0000","last_active":"2024-09-16T00:16:09.019865+0000","last_peered":"2024-09-16T00:16:09.019865+0000","last_clean":"2024-09-16T00:16:09.019865+0000","last_became_active":"2024-09-16T00:15:06.547782+0000","last_became_peered":"2024-09-16T00:15:06.547782+0000","last_unstale":"2024-09-16T00:16:09.019865+0000","last_undegraded":"2024-09-16T00:16:09.019865+0000","last_fullsized":"2024-09-16T00:16:09.019865+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:53.228222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018578+0000","last_change":"2024-09-16T00:15:06.554396+0000","last_active":"2024-09-16T00:16:09.018578+0000","last_peered":"2024-09-16T00:16:09.018578+0000","last_clean":"2024-09-16T00:16:09.018578+0000","last_became_active":"2024-09-16T00:15:06.554012+0000","last_became_peered":"2024-09-16T00:15:06.554012+0000","last_unstale":"2024-09-16T00:16:09.018578+0000","last_undegraded":"2024-09-16T00:16:09.018578+0000","last_fullsized":"2024-09-16T00:16:09.018578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:31.464098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022407+0000","last_change":"2024-09-16T00:15:06.549969+0000","last_active":"2024-09-16T00:16:09.022407+0000","last_peered":"2024-09-16T00:16:09.022407+0000","last_clean":"2024-09-16T00:16:09.022407+0000","last_became_active":"2024-09-16T00:15:06.549531+0000","last_became_peered":"2024-09-16T00:15:06.549531+0000","last_unstale":"2024-09-16T00:16:09.022407+0000","last_undegraded":"2024-09-16T00:16:09.022407+0000","last_fullsized":"2024-09-16T00:16:09.022407+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:37.883612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022583+0000","last_change":"2024-09-16T00:15:06.547734+0000","last_active":"2024-09-16T00:16:09.022583+0000","last_peered":"2024-09-16T00:16:09.022583+0000","last_clean":"2024-09-16T00:16:09.022583+0000","last_became_active":"2024-09-16T00:15:06.547400+0000","last_became_peered":"2024-09-16T00:15:06.547400+0000","last_unstale":"2024-09-16T00:16:09.022583+0000","last_undegraded":"2024-09-16T00:16:09.022583+0000","last_fullsized":"2024-09-16T00:16:09.022583+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:54.455313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.030297+0000","last_change":"2024-09-16T00:15:06.551965+0000","last_active":"2024-09-16T00:16:09.030297+0000","last_peered":"2024-09-16T00:16:09.030297+0000","last_clean":"2024-09-16T00:16:09.030297+0000","last_became_active":"2024-09-16T00:15:06.551602+0000","last_became_peered":"2024-09-16T00:15:06.551602+0000","last_unstale":"2024-09-16T00:16:09.030297+0000","last_undegraded":"2024-09-16T00:16:09.030297+0000","last_fullsized":"2024-09-16T00:16:09.030297+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:34.951960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022367+0000","last_change":"2024-09-16T00:15:06.549340+0000","last_active":"2024-09-16T00:16:09.022367+0000","last_peered":"2024-09-16T00:16:09.022367+0000","last_clean":"2024-09-16T00:16:09.022367+0000","last_became_active":"2024-09-16T00:15:06.547424+0000","last_became_peered":"2024-09-16T00:15:06.547424+0000","last_unstale":"2024-09-16T00:16:09.022367+0000","last_undegraded":"2024-09-16T00:16:09.022367+0000","last_fullsized":"2024-09-16T00:16:09.022367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:12.087535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021986+0000","last_change":"2024-09-16T00:15:06.555216+0000","last_active":"2024-09-16T00:16:09.021986+0000","last_peered":"2024-09-16T00:16:09.021986+0000","last_clean":"2024-09-16T00:16:09.021986+0000","last_became_active":"2024-09-16T00:15:06.554896+0000","last_became_peered":"2024-09-16T00:15:06.554896+0000","last_unstale":"2024-09-16T00:16:09.021986+0000","last_undegraded":"2024-09-16T00:16:09.021986+0000","last_fullsized":"2024-09-16T00:16:09.021986+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:20.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020299+0000","last_change":"2024-09-16T00:15:06.549752+0000","last_active":"2024-09-16T00:16:09.020299+0000","last_peered":"2024-09-16T00:16:09.020299+0000","last_clean":"2024-09-16T00:16:09.020299+0000","last_became_active":"2024-09-16T00:15:06.549350+0000","last_became_peered":"2024-09-16T00:15:06.549350+0000","last_unstale":"2024-09-16T00:16:09.020299+0000","last_undegraded":"2024-09-16T00:16:09.020299+0000","last_fullsized":"2024-09-16T00:16:09.020299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:48.615123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018663+0000","last_change":"2024-09-16T00:15:06.549159+0000","last_active":"2024-09-16T00:16:09.018663+0000","last_peered":"2024-09-16T00:16:09.018663+0000","last_clean":"2024-09-16T00:16:09.018663+0000","last_became_active":"2024-09-16T00:15:06.548408+0000","last_became_peered":"2024-09-16T00:15:06.548408+0000","last_unstale":"2024-09-16T00:16:09.018663+0000","last_undegraded":"2024-09-16T00:16:09.018663+0000","last_fullsized":"2024-09-16T00:16:09.018663+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:23.941809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022207+0000","last_change":"2024-09-16T00:15:06.544091+0000","last_active":"2024-09-16T00:16:09.022207+0000","last_peered":"2024-09-16T00:16:09.022207+0000","last_clean":"2024-09-16T00:16:09.022207+0000","last_became_active":"2024-09-16T00:15:06.543761+0000","last_became_peered":"2024-09-16T00:15:06.543761+0000","last_unstale":"2024-09-16T00:16:09.022207+0000","last_undegraded":"2024-09-16T00:16:09.022207+0000","last_fullsized":"2024-09-16T00:16:09.022207+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:36:43.253405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.013982+0000","last_change":"2024-09-16T00:15:06.546408+0000","last_active":"2024-09-16T00:16:09.013982+0000","last_peered":"2024-09-16T00:16:09.013982+0000","last_clean":"2024-09-16T00:16:09.013982+0000","last_became_active":"2024-09-16T00:15:06.546092+0000","last_became_peered":"2024-09-16T00:15:06.546092+0000","last_unstale":"2024-09-16T00:16:09.013982+0000","last_undegraded":"2024-09-16T00:16:09.013982+0000","last_fullsized":"2024-09-16T00:16:09.013982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:39:15.576097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019200+0000","last_change":"2024-09-16T00:15:06.557162+0000","last_active":"2024-09-16T00:16:09.019200+0000","last_peered":"2024-09-16T00:16:09.019200+0000","last_clean":"2024-09-16T00:16:09.019200+0000","last_became_active":"2024-09-16T00:15:06.556831+0000","last_became_peered":"2024-09-16T00:15:06.556831+0000","last_unstale":"2024-09-16T00:16:09.019200+0000","last_undegraded":"2024-09-16T00:16:09.019200+0000","last_fullsized":"2024-09-16T00:16:09.019200+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.236915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021149+0000","last_change":"2024-09-16T00:15:06.556161+0000","last_active":"2024-09-16T00:16:09.021149+0000","last_peered":"2024-09-16T00:16:09.021149+0000","last_clean":"2024-09-16T00:16:09.021149+0000","last_became_active":"2024-09-16T00:15:06.555819+0000","last_became_peered":"2024-09-16T00:15:06.555819+0000","last_unstale":"2024-09-16T00:16:09.021149+0000","last_undegraded":"2024-09-16T00:16:09.021149+0000","last_fullsized":"2024-09-16T00:16:09.021149+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:29.929977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019135+0000","last_change":"2024-09-16T00:15:06.550635+0000","last_active":"2024-09-16T00:16:09.019135+0000","last_peered":"2024-09-16T00:16:09.019135+0000","last_clean":"2024-09-16T00:16:09.019135+0000","last_became_active":"2024-09-16T00:15:06.549576+0000","last_became_peered":"2024-09-16T00:15:06.549576+0000","last_unstale":"2024-09-16T00:16:09.019135+0000","last_undegraded":"2024-09-16T00:16:09.019135+0000","last_fullsized":"2024-09-16T00:16:09.019135+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:12.818199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019645+0000","last_change":"2024-09-16T00:15:06.547114+0000","last_active":"2024-09-16T00:16:09.019645+0000","last_peered":"2024-09-16T00:16:09.019645+0000","last_clean":"2024-09-16T00:16:09.019645+0000","last_became_active":"2024-09-16T00:15:06.546786+0000","last_became_peered":"2024-09-16T00:15:06.546786+0000","last_unstale":"2024-09-16T00:16:09.019645+0000","last_undegraded":"2024-09-16T00:16:09.019645+0000","last_fullsized":"2024-09-16T00:16:09.019645+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:13:26.925603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019813+0000","last_change":"2024-09-16T00:15:06.541168+0000","last_active":"2024-09-16T00:16:09.019813+0000","last_peered":"2024-09-16T00:16:09.019813+0000","last_clean":"2024-09-16T00:16:09.019813+0000","last_became_active":"2024-09-16T00:15:06.540848+0000","last_became_peered":"2024-09-16T00:15:06.540848+0000","last_unstale":"2024-09-16T00:16:09.019813+0000","last_undegraded":"2024-09-16T00:16:09.019813+0000","last_fullsized":"2024-09-16T00:16:09.019813+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.958600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017094+0000","last_change":"2024-09-16T00:15:06.547429+0000","last_active":"2024-09-16T00:16:09.017094+0000","last_peered":"2024-09-16T00:16:09.017094+0000","last_clean":"2024-09-16T00:16:09.017094+0000","last_became_active":"2024-09-16T00:15:06.547081+0000","last_became_peered":"2024-09-16T00:15:06.547081+0000","last_unstale":"2024-09-16T00:16:09.017094+0000","last_undegraded":"2024-09-16T00:16:09.017094+0000","last_fullsized":"2024-09-16T00:16:09.017094+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:43.396313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.015644+0000","last_change":"2024-09-16T00:15:06.551811+0000","last_active":"2024-09-16T00:16:09.015644+0000","last_peered":"2024-09-16T00:16:09.015644+0000","last_clean":"2024-09-16T00:16:09.015644+0000","last_became_active":"2024-09-16T00:15:06.551424+0000","last_became_peered":"2024-09-16T00:15:06.551424+0000","last_unstale":"2024-09-16T00:16:09.015644+0000","last_undegraded":"2024-09-16T00:16:09.015644+0000","last_fullsized":"2024-09-16T00:16:09.015644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.943070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017860+0000","last_change":"2024-09-16T00:15:06.544697+0000","last_active":"2024-09-16T00:16:09.017860+0000","last_peered":"2024-09-16T00:16:09.017860+0000","last_clean":"2024-09-16T00:16:09.017860+0000","last_became_active":"2024-09-16T00:15:06.544182+0000","last_became_peered":"2024-09-16T00:15:06.544182+0000","last_unstale":"2024-09-16T00:16:09.017860+0000","last_undegraded":"2024-09-16T00:16:09.017860+0000","last_fullsized":"2024-09-16T00:16:09.017860+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:45.929092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.022673+0000","last_change":"2024-09-16T00:15:06.551488+0000","last_active":"2024-09-16T00:16:09.022673+0000","last_peered":"2024-09-16T00:16:09.022673+0000","last_clean":"2024-09-16T00:16:09.022673+0000","last_became_active":"2024-09-16T00:15:06.551171+0000","last_became_peered":"2024-09-16T00:15:06.551171+0000","last_unstale":"2024-09-16T00:16:09.022673+0000","last_undegraded":"2024-09-16T00:16:09.022673+0000","last_fullsized":"2024-09-16T00:16:09.022673+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:23.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.021915+0000","last_change":"2024-09-16T00:15:06.544043+0000","last_active":"2024-09-16T00:16:09.021915+0000","last_peered":"2024-09-16T00:16:09.021915+0000","last_clean":"2024-09-16T00:16:09.021915+0000","last_became_active":"2024-09-16T00:15:06.543014+0000","last_became_peered":"2024-09-16T00:15:06.543014+0000","last_unstale":"2024-09-16T00:16:09.021915+0000","last_undegraded":"2024-09-16T00:16:09.021915+0000","last_fullsized":"2024-09-16T00:16:09.021915+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:23.904810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.014817+0000","last_change":"2024-09-16T00:15:06.546788+0000","last_active":"2024-09-16T00:16:09.014817+0000","last_peered":"2024-09-16T00:16:09.014817+0000","last_clean":"2024-09-16T00:16:09.014817+0000","last_became_active":"2024-09-16T00:15:06.546333+0000","last_became_peered":"2024-09-16T00:15:06.546333+0000","last_unstale":"2024-09-16T00:16:09.014817+0000","last_undegraded":"2024-09-16T00:16:09.014817+0000","last_fullsized":"2024-09-16T00:16:09.014817+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:09.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.016046+0000","last_change":"2024-09-16T00:15:06.544855+0000","last_active":"2024-09-16T00:16:09.016046+0000","last_peered":"2024-09-16T00:16:09.016046+0000","last_clean":"2024-09-16T00:16:09.016046+0000","last_became_active":"2024-09-16T00:15:06.543708+0000","last_became_peered":"2024-09-16T00:15:06.543708+0000","last_unstale":"2024-09-16T00:16:09.016046+0000","last_undegraded":"2024-09-16T00:16:09.016046+0000","last_fullsized":"2024-09-16T00:16:09.016046+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:13:58.756111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.018919+0000","last_change":"2024-09-16T00:15:06.540675+0000","last_active":"2024-09-16T00:16:09.018919+0000","last_peered":"2024-09-16T00:16:09.018919+0000","last_clean":"2024-09-16T00:16:09.018919+0000","last_became_active":"2024-09-16T00:15:06.540333+0000","last_became_peered":"2024-09-16T00:15:06.540333+0000","last_unstale":"2024-09-16T00:16:09.018919+0000","last_undegraded":"2024-09-16T00:16:09.018919+0000","last_fullsized":"2024-09-16T00:16:09.018919+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:17.166987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.017031+0000","last_change":"2024-09-16T00:15:06.548758+0000","last_active":"2024-09-16T00:16:09.017031+0000","last_peered":"2024-09-16T00:16:09.017031+0000","last_clean":"2024-09-16T00:16:09.017031+0000","last_became_active":"2024-09-16T00:15:06.548397+0000","last_became_peered":"2024-09-16T00:15:06.548397+0000","last_unstale":"2024-09-16T00:16:09.017031+0000","last_undegraded":"2024-09-16T00:16:09.017031+0000","last_fullsized":"2024-09-16T00:16:09.017031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:50.320170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020486+0000","last_change":"2024-09-16T00:15:06.554898+0000","last_active":"2024-09-16T00:16:09.020486+0000","last_peered":"2024-09-16T00:16:09.020486+0000","last_clean":"2024-09-16T00:16:09.020486+0000","last_became_active":"2024-09-16T00:15:06.553005+0000","last_became_peered":"2024-09-16T00:15:06.553005+0000","last_unstale":"2024-09-16T00:16:09.020486+0000","last_undegraded":"2024-09-16T00:16:09.020486+0000","last_fullsized":"2024-09-16T00:16:09.020486+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:29.630911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.020199+0000","last_change":"2024-09-16T00:15:06.547265+0000","last_active":"2024-09-16T00:16:09.020199+0000","last_peered":"2024-09-16T00:16:09.020199+0000","last_clean":"2024-09-16T00:16:09.020199+0000","last_became_active":"2024-09-16T00:15:06.546938+0000","last_became_peered":"2024-09-16T00:15:06.546938+0000","last_unstale":"2024-09-16T00:16:09.020199+0000","last_undegraded":"2024-09-16T00:16:09.020199+0000","last_fullsized":"2024-09-16T00:16:09.020199+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:40.247343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T00:16:09.019051+0000","last_change":"2024-09-16T00:15:06.556735+0000","last_active":"2024-09-16T00:16:09.019051+0000","last_peered":"2024-09-16T00:16:09.019051+0000","last_clean":"2024-09-16T00:16:09.019051+0000","last_became_active":"2024-09-16T00:15:06.556324+0000","last_became_peered":"2024-09-16T00:15:06.556324+0000","last_unstale":"2024-09-16T00:16:09.019051+0000","last_undegraded":"2024-09-16T00:16:09.019051+0000","last_fullsized":"2024-09-16T00:16:09.019051+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:25.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66934,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":348,"num_read_kb":234,"num_write":165,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":62107,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":200802,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249414,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579938,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":970752,"data_stored":497547,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808870,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36440,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708712,"statfs":{"total":95995035648,"available":95957721088,"internally_reserved":0,"allocated":1118208,"data_stored":1148139,"data_compressed":59817,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972396,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":991232,"data_stored":501879,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201328,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":497121,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430262,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1036288,"data_stored":1093614,"data_compressed":59817,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626492,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":966656,"data_stored":492959,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855425,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1044480,"data_stored":1088809,"data_compressed":59817,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986246,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":561515,"data_compressed":33384,"data_compressed_allocated":45056,"data_compressed_original":122880,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215178,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":978944,"data_stored":493153,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444112,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":983040,"data_stored":493637,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607637,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1040384,"data_stored":556382,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":991232,"data_stored":494347,"data_compressed":23484,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":64550,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T00:16:48.446 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-16T00:16:48.446 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T00:16:48.447 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-16T00:16:48.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-16T00:16:48.709 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.214+0000 7ff4652a6640 1 -- 172.21.15.33:0/1848205945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff460102790 msgr2=0x7ff4601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.214+0000 7ff4652a6640 1 --2- 172.21.15.33:0/1848205945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff460102790 0x7ff4601067e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff454009900 tx=0x7ff45402a410 comp rx=0 tx=0).stop 2024-09-16T00:16:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.214+0000 7ff4652a6640 1 -- 172.21.15.33:0/1848205945 shutdown_connections 2024-09-16T00:16:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.214+0000 7ff4652a6640 1 --2- 172.21.15.33:0/1848205945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff460106f10 0x7ff46010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.214+0000 7ff4652a6640 1 --2- 172.21.15.33:0/1848205945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff460102790 0x7ff4601067e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.214+0000 7ff4652a6640 1 --2- 172.21.15.33:0/1848205945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff460101de0 0x7ff4601021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.214+0000 7ff4652a6640 1 -- 172.21.15.33:0/1848205945 >> 172.21.15.33:0/1848205945 conn(0x7ff4600fd8c0 msgr2=0x7ff4600ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.214+0000 7ff4652a6640 1 -- 172.21.15.33:0/1848205945 shutdown_connections 2024-09-16T00:16:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.214+0000 7ff4652a6640 1 -- 172.21.15.33:0/1848205945 wait complete. 2024-09-16T00:16:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff4652a6640 1 Processor -- start 2024-09-16T00:16:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff4652a6640 1 -- start start 2024-09-16T00:16:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff4652a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff460101de0 0x7ff4601984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff4652a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff460102790 0x7ff460198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff4652a6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff460106f10 0x7ff46019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff4652a6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff460111f70 con 0x7ff460102790 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff4652a6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff460111df0 con 0x7ff460101de0 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff4652a6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff4601120f0 con 0x7ff460106f10 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff45e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff460102790 0x7ff460198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff45e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff460102790 0x7ff460198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40098/0 (socket says 172.21.15.33:40098) 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff45e7fc640 1 -- 172.21.15.33:0/4054654979 learned_addr learned my addr 172.21.15.33:0/4054654979 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff45effd640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff460101de0 0x7ff4601984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff45f7fe640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff460106f10 0x7ff46019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff45e7fc640 1 -- 172.21.15.33:0/4054654979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff460106f10 msgr2=0x7ff46019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff45e7fc640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff460106f10 0x7ff46019cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff45e7fc640 1 -- 172.21.15.33:0/4054654979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff460101de0 msgr2=0x7ff4601984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.215+0000 7ff45e7fc640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff460101de0 0x7ff4601984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.216+0000 7ff45e7fc640 1 -- 172.21.15.33:0/4054654979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46019d480 con 0x7ff460102790 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.216+0000 7ff45e7fc640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff460102790 0x7ff460198a10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff454006010 tx=0x7ff4540042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.216+0000 7ff447fff640 1 -- 172.21.15.33:0/4054654979 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45403f030 con 0x7ff460102790 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.216+0000 7ff447fff640 1 -- 172.21.15.33:0/4054654979 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff454003ce0 con 0x7ff460102790 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.216+0000 7ff4652a6640 1 -- 172.21.15.33:0/4054654979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff46019d710 con 0x7ff460102790 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.216+0000 7ff447fff640 1 -- 172.21.15.33:0/4054654979 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff454035070 con 0x7ff460102790 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.216+0000 7ff4652a6640 1 -- 172.21.15.33:0/4054654979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4601a5050 con 0x7ff460102790 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.217+0000 7ff4652a6640 1 -- 172.21.15.33:0/4054654979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4601032c0 con 0x7ff460102790 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.218+0000 7ff447fff640 1 -- 172.21.15.33:0/4054654979 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff454042030 con 0x7ff460102790 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.219+0000 7ff447fff640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff4480a32d0 0x7ff4480a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.219+0000 7ff45effd640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff4480a32d0 0x7ff4480a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.219+0000 7ff447fff640 1 -- 172.21.15.33:0/4054654979 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7ff4540effe0 con 0x7ff460102790 2024-09-16T00:16:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.220+0000 7ff45effd640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff4480a32d0 0x7ff4480a5790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff45000bbe0 tx=0x7ff45000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:49.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.224+0000 7ff447fff640 1 -- 172.21.15.33:0/4054654979 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff4540bc900 con 0x7ff460102790 2024-09-16T00:16:49.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.439+0000 7ff4652a6640 1 -- 172.21.15.33:0/4054654979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff46010a150 con 0x7ff460102790 2024-09-16T00:16:49.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.440+0000 7ff447fff640 1 -- 172.21.15.33:0/4054654979 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7ff4540bc200 con 0x7ff460102790 2024-09-16T00:16:49.441 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:49.441 INFO:teuthology.orchestra.run.smithi033.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T00:16:49.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.442+0000 7ff4652a6640 1 -- 172.21.15.33:0/4054654979 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff4480a32d0 msgr2=0x7ff4480a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:49.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.442+0000 7ff4652a6640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff4480a32d0 0x7ff4480a5790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff45000bbe0 tx=0x7ff45000b390 comp rx=0 tx=0).stop 2024-09-16T00:16:49.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.442+0000 7ff4652a6640 1 -- 172.21.15.33:0/4054654979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff460102790 msgr2=0x7ff460198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:49.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.442+0000 7ff4652a6640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff460102790 0x7ff460198a10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff454006010 tx=0x7ff4540042f0 comp rx=0 tx=0).stop 2024-09-16T00:16:49.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.442+0000 7ff4652a6640 1 -- 172.21.15.33:0/4054654979 shutdown_connections 2024-09-16T00:16:49.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.442+0000 7ff4652a6640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff4480a32d0 0x7ff4480a5790 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:49.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.442+0000 7ff4652a6640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff460106f10 0x7ff46019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:49.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.442+0000 7ff4652a6640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff460102790 0x7ff460198a10 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:49.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.442+0000 7ff4652a6640 1 --2- 172.21.15.33:0/4054654979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff460101de0 0x7ff4601984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:49.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.442+0000 7ff4652a6640 1 -- 172.21.15.33:0/4054654979 >> 172.21.15.33:0/4054654979 conn(0x7ff4600fd8c0 msgr2=0x7ff4600fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:49.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.442+0000 7ff4652a6640 1 -- 172.21.15.33:0/4054654979 shutdown_connections 2024-09-16T00:16:49.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:49.442+0000 7ff4652a6640 1 -- 172.21.15.33:0/4054654979 wait complete. 2024-09-16T00:16:49.783 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:49 smithi060 ceph-mon[35185]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:16:49.783 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:49 smithi060 ceph-mon[35185]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:16:49.783 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:49 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4054654979' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-16T00:16:49.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:49 smithi033 ceph-mon[31243]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:16:49.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:49 smithi033 ceph-mon[31243]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:16:49.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:49 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4054654979' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-16T00:16:49.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:49 smithi083 ceph-mon[35242]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:16:49.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:49 smithi083 ceph-mon[35242]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:16:49.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:49 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4054654979' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-16T00:16:50.240 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-16T00:16:50.241 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:16:50.506 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:50.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.973+0000 7f2ac364b640 1 -- 172.21.15.33:0/603690384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2abc102570 msgr2=0x7f2abc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:50.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.973+0000 7f2ac364b640 1 --2- 172.21.15.33:0/603690384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2abc102570 0x7f2abc1029b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009980 tx=0x7f2aac02c6b0 comp rx=0 tx=0).stop 2024-09-16T00:16:50.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.973+0000 7f2ac364b640 1 -- 172.21.15.33:0/603690384 shutdown_connections 2024-09-16T00:16:50.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.973+0000 7f2ac364b640 1 --2- 172.21.15.33:0/603690384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc102ef0 0x7f2abc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:50.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.973+0000 7f2ac364b640 1 --2- 172.21.15.33:0/603690384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2abc102570 0x7f2abc1029b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:50.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.973+0000 7f2ac364b640 1 --2- 172.21.15.33:0/603690384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2abc100e20 0x7f2abc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:50.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.973+0000 7f2ac364b640 1 -- 172.21.15.33:0/603690384 >> 172.21.15.33:0/603690384 conn(0x7f2abc0fc7d0 msgr2=0x7f2abc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:50.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.973+0000 7f2ac364b640 1 -- 172.21.15.33:0/603690384 shutdown_connections 2024-09-16T00:16:50.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.973+0000 7f2ac364b640 1 -- 172.21.15.33:0/603690384 wait complete. 2024-09-16T00:16:50.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac364b640 1 Processor -- start 2024-09-16T00:16:50.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac364b640 1 -- start start 2024-09-16T00:16:50.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac364b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2abc100e20 0x7f2abc196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:50.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac364b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc102570 0x7f2abc196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:50.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac364b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2abc102ef0 0x7f2abc19abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:50.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac364b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc10fe80 con 0x7f2abc102ef0 2024-09-16T00:16:50.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac364b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc10fd00 con 0x7f2abc100e20 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac364b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc110000 con 0x7f2abc102570 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac1be8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2abc102ef0 0x7f2abc19abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac1be8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2abc102ef0 0x7f2abc19abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50906/0 (socket says 172.21.15.33:50906) 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac1be8640 1 -- 172.21.15.33:0/3020652529 learned_addr learned my addr 172.21.15.33:0/3020652529 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac13e7640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2abc100e20 0x7f2abc196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.974+0000 7f2ac0be6640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc102570 0x7f2abc196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.975+0000 7f2ac1be8640 1 -- 172.21.15.33:0/3020652529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc102570 msgr2=0x7f2abc196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.975+0000 7f2ac1be8640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc102570 0x7f2abc196840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.975+0000 7f2ac1be8640 1 -- 172.21.15.33:0/3020652529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2abc100e20 msgr2=0x7f2abc196300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.975+0000 7f2ac1be8640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2abc100e20 0x7f2abc196300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.975+0000 7f2ac1be8640 1 -- 172.21.15.33:0/3020652529 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc19b2b0 con 0x7f2abc102ef0 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.975+0000 7f2ac1be8640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2abc102ef0 0x7f2abc19abd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400cad0 tx=0x7f2ab400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:50.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.975+0000 7f2ab27fc640 1 -- 172.21.15.33:0/3020652529 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab400ae20 con 0x7f2abc102ef0 2024-09-16T00:16:50.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.975+0000 7f2ab27fc640 1 -- 172.21.15.33:0/3020652529 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ab4005020 con 0x7f2abc102ef0 2024-09-16T00:16:50.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.975+0000 7f2ac364b640 1 -- 172.21.15.33:0/3020652529 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2abc19b5a0 con 0x7f2abc102ef0 2024-09-16T00:16:50.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.975+0000 7f2ab27fc640 1 -- 172.21.15.33:0/3020652529 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab400f6f0 con 0x7f2abc102ef0 2024-09-16T00:16:50.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.975+0000 7f2ac364b640 1 -- 172.21.15.33:0/3020652529 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2abc1a2e80 con 0x7f2abc102ef0 2024-09-16T00:16:50.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.976+0000 7f2ac364b640 1 -- 172.21.15.33:0/3020652529 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2abc101200 con 0x7f2abc102ef0 2024-09-16T00:16:50.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.977+0000 7f2ab27fc640 1 -- 172.21.15.33:0/3020652529 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f2ab40051d0 con 0x7f2abc102ef0 2024-09-16T00:16:50.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.978+0000 7f2ab27fc640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2aa80a32d0 0x7f2aa80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:50.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.978+0000 7f2ab27fc640 1 -- 172.21.15.33:0/3020652529 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f2ab40d0100 con 0x7f2abc102ef0 2024-09-16T00:16:50.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.978+0000 7f2ac13e7640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2aa80a32d0 0x7f2aa80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:50.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.979+0000 7f2ac13e7640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2aa80a32d0 0x7f2aa80a5790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8005d00 tx=0x7f2ab800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:50.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:50.981+0000 7f2ab27fc640 1 -- 172.21.15.33:0/3020652529 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2ab4014030 con 0x7f2abc102ef0 2024-09-16T00:16:51.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.194+0000 7f2ac364b640 1 -- 172.21.15.33:0/3020652529 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2abc197660 con 0x7f2abc102ef0 2024-09-16T00:16:51.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.195+0000 7f2ab27fc640 1 -- 172.21.15.33:0/3020652529 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7f2abc197660 con 0x7f2abc102ef0 2024-09-16T00:16:51.196 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:51.197 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"btime":"2024-09-16T00:15:52:174386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:15:51.912925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:16:51.197 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-09-16T00:16:51.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.197+0000 7f2ac364b640 1 -- 172.21.15.33:0/3020652529 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2aa80a32d0 msgr2=0x7f2aa80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:51.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.197+0000 7f2ac364b640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2aa80a32d0 0x7f2aa80a5790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8005d00 tx=0x7f2ab800b410 comp rx=0 tx=0).stop 2024-09-16T00:16:51.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.197+0000 7f2ac364b640 1 -- 172.21.15.33:0/3020652529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2abc102ef0 msgr2=0x7f2abc19abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:51.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.197+0000 7f2ac364b640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2abc102ef0 0x7f2abc19abd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400cad0 tx=0x7f2ab400cf90 comp rx=0 tx=0).stop 2024-09-16T00:16:51.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.197+0000 7f2ac364b640 1 -- 172.21.15.33:0/3020652529 shutdown_connections 2024-09-16T00:16:51.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.197+0000 7f2ac364b640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2aa80a32d0 0x7f2aa80a5790 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:51.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.197+0000 7f2ac364b640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2abc102ef0 0x7f2abc19abd0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:51.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.197+0000 7f2ac364b640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc102570 0x7f2abc196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:51.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.197+0000 7f2ac364b640 1 --2- 172.21.15.33:0/3020652529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2abc100e20 0x7f2abc196300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:51.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.197+0000 7f2ac364b640 1 -- 172.21.15.33:0/3020652529 >> 172.21.15.33:0/3020652529 conn(0x7f2abc0fc7d0 msgr2=0x7f2abc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:51.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.197+0000 7f2ac364b640 1 -- 172.21.15.33:0/3020652529 shutdown_connections 2024-09-16T00:16:51.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:51.197+0000 7f2ac364b640 1 -- 172.21.15.33:0/3020652529 wait complete. 2024-09-16T00:16:51.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:51 smithi033 ceph-mon[31243]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:51.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:51 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3020652529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:16:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:51 smithi083 ceph-mon[35242]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:51.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:51 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3020652529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:16:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:51 smithi060 ceph-mon[35185]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3020652529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:16:52.077 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:16:52.340 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:52.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.817+0000 7ff07196b640 1 -- 172.21.15.33:0/3019373177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c074620 msgr2=0x7ff06c074a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:52.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.817+0000 7ff07196b640 1 --2- 172.21.15.33:0/3019373177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c074620 0x7ff06c074a00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009ca0 tx=0x7ff05c02a950 comp rx=0 tx=0).stop 2024-09-16T00:16:52.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.817+0000 7ff07196b640 1 -- 172.21.15.33:0/3019373177 shutdown_connections 2024-09-16T00:16:52.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.817+0000 7ff07196b640 1 --2- 172.21.15.33:0/3019373177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff06c10e3c0 0x7ff06c110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:52.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.817+0000 7ff07196b640 1 --2- 172.21.15.33:0/3019373177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff06c074f40 0x7ff06c10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:52.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.817+0000 7ff07196b640 1 --2- 172.21.15.33:0/3019373177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c074620 0x7ff06c074a00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:52.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.817+0000 7ff07196b640 1 -- 172.21.15.33:0/3019373177 >> 172.21.15.33:0/3019373177 conn(0x7ff06c0fd8c0 msgr2=0x7ff06c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:52.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.817+0000 7ff07196b640 1 -- 172.21.15.33:0/3019373177 shutdown_connections 2024-09-16T00:16:52.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.817+0000 7ff07196b640 1 -- 172.21.15.33:0/3019373177 wait complete. 2024-09-16T00:16:52.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.817+0000 7ff07196b640 1 Processor -- start 2024-09-16T00:16:52.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.818+0000 7ff07196b640 1 -- start start 2024-09-16T00:16:52.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.818+0000 7ff07196b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c074620 0x7ff06c19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:52.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.818+0000 7ff07196b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff06c074f40 0x7ff06c19abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:52.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.818+0000 7ff07196b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff06c10e3c0 0x7ff06c19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:52.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.818+0000 7ff07196b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c114170 con 0x7ff06c074620 2024-09-16T00:16:52.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.818+0000 7ff07196b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c113ff0 con 0x7ff06c10e3c0 2024-09-16T00:16:52.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.818+0000 7ff07196b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c1142f0 con 0x7ff06c074f40 2024-09-16T00:16:52.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.818+0000 7ff06affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c074620 0x7ff06c19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:52.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.818+0000 7ff06affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c074620 0x7ff06c19a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50932/0 (socket says 172.21.15.33:50932) 2024-09-16T00:16:52.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.818+0000 7ff06affd640 1 -- 172.21.15.33:0/1349463537 learned_addr learned my addr 172.21.15.33:0/1349463537 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:52.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.818+0000 7ff06b7fe640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff06c10e3c0 0x7ff06c19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:52.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.818+0000 7ff06a7fc640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff06c074f40 0x7ff06c19abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:52.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.819+0000 7ff06affd640 1 -- 172.21.15.33:0/1349463537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff06c074f40 msgr2=0x7ff06c19abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:52.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.819+0000 7ff06affd640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff06c074f40 0x7ff06c19abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:52.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.819+0000 7ff06affd640 1 -- 172.21.15.33:0/1349463537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff06c10e3c0 msgr2=0x7ff06c19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:52.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.819+0000 7ff06affd640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff06c10e3c0 0x7ff06c19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:52.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.819+0000 7ff06affd640 1 -- 172.21.15.33:0/1349463537 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff06c19f620 con 0x7ff06c074620 2024-09-16T00:16:52.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.819+0000 7ff06affd640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c074620 0x7ff06c19a670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff05c005cc0 tx=0x7ff05c02f630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:52.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.819+0000 7ff070969640 1 -- 172.21.15.33:0/1349463537 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c034040 con 0x7ff06c074620 2024-09-16T00:16:52.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.819+0000 7ff070969640 1 -- 172.21.15.33:0/1349463537 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff05c030920 con 0x7ff06c074620 2024-09-16T00:16:52.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.819+0000 7ff07196b640 1 -- 172.21.15.33:0/1349463537 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff06c19f8b0 con 0x7ff06c074620 2024-09-16T00:16:52.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.819+0000 7ff070969640 1 -- 172.21.15.33:0/1349463537 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c004830 con 0x7ff06c074620 2024-09-16T00:16:52.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.819+0000 7ff07196b640 1 -- 172.21.15.33:0/1349463537 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff06c1a71f0 con 0x7ff06c074620 2024-09-16T00:16:52.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.820+0000 7ff07196b640 1 -- 172.21.15.33:0/1349463537 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff06c074a00 con 0x7ff06c074620 2024-09-16T00:16:52.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.821+0000 7ff070969640 1 -- 172.21.15.33:0/1349463537 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff05c037020 con 0x7ff06c074620 2024-09-16T00:16:52.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.822+0000 7ff070969640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0580a3350 0x7ff0580a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:52.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.822+0000 7ff06a7fc640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0580a3350 0x7ff0580a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:52.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.822+0000 7ff070969640 1 -- 172.21.15.33:0/1349463537 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7ff05c0f0e90 con 0x7ff06c074620 2024-09-16T00:16:52.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.823+0000 7ff06a7fc640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0580a3350 0x7ff0580a5810 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff054005fd0 tx=0x7ff054005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:52.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:52.825+0000 7ff070969640 1 -- 172.21.15.33:0/1349463537 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff05c0811b0 con 0x7ff06c074620 2024-09-16T00:16:53.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.036+0000 7ff07196b640 1 -- 172.21.15.33:0/1349463537 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff06c109540 con 0x7ff06c074620 2024-09-16T00:16:53.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.036+0000 7ff070969640 1 -- 172.21.15.33:0/1349463537 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7ff05c0bd800 con 0x7ff06c074620 2024-09-16T00:16:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:53.038 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"btime":"2024-09-16T00:15:52:174386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:15:51.912925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:16:53.038 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-09-16T00:16:53.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.038+0000 7ff07196b640 1 -- 172.21.15.33:0/1349463537 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0580a3350 msgr2=0x7ff0580a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:53.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.038+0000 7ff07196b640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0580a3350 0x7ff0580a5810 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff054005fd0 tx=0x7ff054005e20 comp rx=0 tx=0).stop 2024-09-16T00:16:53.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.038+0000 7ff07196b640 1 -- 172.21.15.33:0/1349463537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c074620 msgr2=0x7ff06c19a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:53.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.038+0000 7ff07196b640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c074620 0x7ff06c19a670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff05c005cc0 tx=0x7ff05c02f630 comp rx=0 tx=0).stop 2024-09-16T00:16:53.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.039+0000 7ff07196b640 1 -- 172.21.15.33:0/1349463537 shutdown_connections 2024-09-16T00:16:53.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.039+0000 7ff07196b640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0580a3350 0x7ff0580a5810 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:53.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.039+0000 7ff07196b640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff06c10e3c0 0x7ff06c19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:53.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.039+0000 7ff07196b640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff06c074f40 0x7ff06c19abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:53.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.039+0000 7ff07196b640 1 --2- 172.21.15.33:0/1349463537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c074620 0x7ff06c19a670 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:53.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.039+0000 7ff07196b640 1 -- 172.21.15.33:0/1349463537 >> 172.21.15.33:0/1349463537 conn(0x7ff06c0fd8c0 msgr2=0x7ff06c0fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:53.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.039+0000 7ff07196b640 1 -- 172.21.15.33:0/1349463537 shutdown_connections 2024-09-16T00:16:53.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:53.039+0000 7ff07196b640 1 -- 172.21.15.33:0/1349463537 wait complete. 2024-09-16T00:16:53.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:53 smithi033 ceph-mon[31243]: pgmap v340: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:53.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:53 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1349463537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:16:53.910 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:16:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:53 smithi083 ceph-mon[35242]: pgmap v340: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:53 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1349463537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:16:53.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:53 smithi060 ceph-mon[35185]: pgmap v340: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:53.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:53 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1349463537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:16:54.175 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:54.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.649+0000 7f29461f1640 1 -- 172.21.15.33:0/2452292112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940106f10 msgr2=0x7f294010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:54.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.649+0000 7f29461f1640 1 --2- 172.21.15.33:0/2452292112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940106f10 0x7f294010ea90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2938009ca0 tx=0x7f293802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:16:54.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.649+0000 7f29461f1640 1 -- 172.21.15.33:0/2452292112 shutdown_connections 2024-09-16T00:16:54.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.649+0000 7f29461f1640 1 --2- 172.21.15.33:0/2452292112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940106f10 0x7f294010ea90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:54.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.649+0000 7f29461f1640 1 --2- 172.21.15.33:0/2452292112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2940102790 0x7f29401067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:54.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.649+0000 7f29461f1640 1 --2- 172.21.15.33:0/2452292112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2940101de0 0x7f29401021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:54.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.649+0000 7f29461f1640 1 -- 172.21.15.33:0/2452292112 >> 172.21.15.33:0/2452292112 conn(0x7f29400fd8c0 msgr2=0x7f29400ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:54.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.649+0000 7f29461f1640 1 -- 172.21.15.33:0/2452292112 shutdown_connections 2024-09-16T00:16:54.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.649+0000 7f29461f1640 1 -- 172.21.15.33:0/2452292112 wait complete. 2024-09-16T00:16:54.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.650+0000 7f29461f1640 1 Processor -- start 2024-09-16T00:16:54.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.650+0000 7f29461f1640 1 -- start start 2024-09-16T00:16:54.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f29461f1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940101de0 0x7f29401983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:54.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f29461f1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2940102790 0x7f2940198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:54.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f29461f1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2940106f10 0x7f294019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:54.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f29461f1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2940111f70 con 0x7f2940101de0 2024-09-16T00:16:54.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f29461f1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2940111df0 con 0x7f2940106f10 2024-09-16T00:16:54.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f29461f1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29401120f0 con 0x7f2940102790 2024-09-16T00:16:54.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f293f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940101de0 0x7f29401983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:54.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f293f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940101de0 0x7f29401983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50942/0 (socket says 172.21.15.33:50942) 2024-09-16T00:16:54.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f293f7fe640 1 -- 172.21.15.33:0/3524644412 learned_addr learned my addr 172.21.15.33:0/3524644412 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:54.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f293effd640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2940102790 0x7f2940198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:54.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f293ffff640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2940106f10 0x7f294019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:54.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f293effd640 1 -- 172.21.15.33:0/3524644412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2940106f10 msgr2=0x7f294019ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:54.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f293effd640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2940106f10 0x7f294019ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:54.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f293effd640 1 -- 172.21.15.33:0/3524644412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940101de0 msgr2=0x7f29401983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:54.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f293effd640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940101de0 0x7f29401983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:54.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.651+0000 7f293effd640 1 -- 172.21.15.33:0/3524644412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294019d3a0 con 0x7f2940102790 2024-09-16T00:16:54.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.652+0000 7f293ffff640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2940106f10 0x7f294019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:16:54.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.652+0000 7f293effd640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2940102790 0x7f2940198930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2934005fd0 tx=0x7f293400dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:54.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.652+0000 7f293cff9640 1 -- 172.21.15.33:0/3524644412 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2934009c30 con 0x7f2940102790 2024-09-16T00:16:54.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.652+0000 7f29461f1640 1 -- 172.21.15.33:0/3524644412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294019d690 con 0x7f2940102790 2024-09-16T00:16:54.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.652+0000 7f293cff9640 1 -- 172.21.15.33:0/3524644412 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2934009dd0 con 0x7f2940102790 2024-09-16T00:16:54.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.652+0000 7f293cff9640 1 -- 172.21.15.33:0/3524644412 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2934010460 con 0x7f2940102790 2024-09-16T00:16:54.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.652+0000 7f29461f1640 1 -- 172.21.15.33:0/3524644412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29401a4f70 con 0x7f2940102790 2024-09-16T00:16:54.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.653+0000 7f29461f1640 1 -- 172.21.15.33:0/3524644412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29401021c0 con 0x7f2940102790 2024-09-16T00:16:54.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.654+0000 7f293cff9640 1 -- 172.21.15.33:0/3524644412 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f2934004030 con 0x7f2940102790 2024-09-16T00:16:54.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.655+0000 7f293cff9640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f29280a32d0 0x7f29280a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:54.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.655+0000 7f293f7fe640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f29280a32d0 0x7f29280a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:54.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.655+0000 7f293cff9640 1 -- 172.21.15.33:0/3524644412 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f29340cf520 con 0x7f2940102790 2024-09-16T00:16:54.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.655+0000 7f293f7fe640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f29280a32d0 0x7f29280a5790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2930009c90 tx=0x7f2930005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:54.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.660+0000 7f293cff9640 1 -- 172.21.15.33:0/3524644412 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2934017020 con 0x7f2940102790 2024-09-16T00:16:54.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.835+0000 7f29461f1640 1 -- 172.21.15.33:0/3524644412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f294010a150 con 0x7f2940102790 2024-09-16T00:16:54.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.837+0000 7f293cff9640 1 -- 172.21.15.33:0/3524644412 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24609 (secure 0 0 0) 0x7f293409bd80 con 0x7f2940102790 2024-09-16T00:16:54.840 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:54.840 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":88,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:16:09.010428+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:16:54.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.839+0000 7f29461f1640 1 -- 172.21.15.33:0/3524644412 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f29280a32d0 msgr2=0x7f29280a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:54.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.839+0000 7f29461f1640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f29280a32d0 0x7f29280a5790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2930009c90 tx=0x7f2930005f90 comp rx=0 tx=0).stop 2024-09-16T00:16:54.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.839+0000 7f29461f1640 1 -- 172.21.15.33:0/3524644412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2940102790 msgr2=0x7f2940198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:54.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.839+0000 7f29461f1640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2940102790 0x7f2940198930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2934005fd0 tx=0x7f293400dde0 comp rx=0 tx=0).stop 2024-09-16T00:16:54.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.839+0000 7f29461f1640 1 -- 172.21.15.33:0/3524644412 shutdown_connections 2024-09-16T00:16:54.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.839+0000 7f29461f1640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f29280a32d0 0x7f29280a5790 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:54.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.839+0000 7f29461f1640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2940106f10 0x7f294019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:54.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.839+0000 7f29461f1640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2940102790 0x7f2940198930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:54.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.839+0000 7f29461f1640 1 --2- 172.21.15.33:0/3524644412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940101de0 0x7f29401983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:54.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.839+0000 7f29461f1640 1 -- 172.21.15.33:0/3524644412 >> 172.21.15.33:0/3524644412 conn(0x7f29400fd8c0 msgr2=0x7f29400feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:54.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.840+0000 7f29461f1640 1 -- 172.21.15.33:0/3524644412 shutdown_connections 2024-09-16T00:16:54.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:54.840+0000 7f29461f1640 1 -- 172.21.15.33:0/3524644412 wait complete. 2024-09-16T00:16:55.653 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:16:55.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:55 smithi033 ceph-mon[31243]: pgmap v341: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:55.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:55 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3524644412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:16:55.918 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:55 smithi083 ceph-mon[35242]: pgmap v341: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:55 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3524644412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:16:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:55 smithi060 ceph-mon[35185]: pgmap v341: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:16:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:55 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3524644412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:16:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.410+0000 7f7f07d2b640 1 -- 172.21.15.33:0/3973199168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f00102790 msgr2=0x7f7f001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.410+0000 7f7f07d2b640 1 --2- 172.21.15.33:0/3973199168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f00102790 0x7f7f001067e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009900 tx=0x7f7ef002a410 comp rx=0 tx=0).stop 2024-09-16T00:16:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.410+0000 7f7f07d2b640 1 -- 172.21.15.33:0/3973199168 shutdown_connections 2024-09-16T00:16:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.410+0000 7f7f07d2b640 1 --2- 172.21.15.33:0/3973199168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f00106f10 0x7f7f0010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.410+0000 7f7f07d2b640 1 --2- 172.21.15.33:0/3973199168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f00102790 0x7f7f001067e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.410+0000 7f7f07d2b640 1 --2- 172.21.15.33:0/3973199168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f00101de0 0x7f7f001021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.410+0000 7f7f07d2b640 1 -- 172.21.15.33:0/3973199168 >> 172.21.15.33:0/3973199168 conn(0x7f7f000fd8c0 msgr2=0x7f7f000ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.410+0000 7f7f07d2b640 1 -- 172.21.15.33:0/3973199168 shutdown_connections 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.410+0000 7f7f07d2b640 1 -- 172.21.15.33:0/3973199168 wait complete. 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.411+0000 7f7f07d2b640 1 Processor -- start 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.411+0000 7f7f07d2b640 1 -- start start 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.411+0000 7f7f07d2b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f00101de0 0x7f7f00198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.411+0000 7f7f07d2b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f00102790 0x7f7f00198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.411+0000 7f7f07d2b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f00106f10 0x7f7f0019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.411+0000 7f7f07d2b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f00111f70 con 0x7f7f00101de0 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.411+0000 7f7f07d2b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7f00111df0 con 0x7f7f00106f10 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.411+0000 7f7f07d2b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001120f0 con 0x7f7f00102790 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f062c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f00106f10 0x7f7f0019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f05ac7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f00101de0 0x7f7f00198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f062c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f00106f10 0x7f7f0019cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:44582/0 (socket says 172.21.15.33:44582) 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f05ac7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f00101de0 0x7f7f00198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50962/0 (socket says 172.21.15.33:50962) 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f062c8640 1 -- 172.21.15.33:0/2512597547 learned_addr learned my addr 172.21.15.33:0/2512597547 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f052c6640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f00102790 0x7f7f00198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f05ac7640 1 -- 172.21.15.33:0/2512597547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f00102790 msgr2=0x7f7f00198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f05ac7640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f00102790 0x7f7f00198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f05ac7640 1 -- 172.21.15.33:0/2512597547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f00106f10 msgr2=0x7f7f0019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f05ac7640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f00106f10 0x7f7f0019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f05ac7640 1 -- 172.21.15.33:0/2512597547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0019d3e0 con 0x7f7f00101de0 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f05ac7640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f00101de0 0x7f7f00198430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00d6d0 tx=0x7f7efc00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7ef6ffd640 1 -- 172.21.15.33:0/2512597547 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc017070 con 0x7f7f00101de0 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7ef6ffd640 1 -- 172.21.15.33:0/2512597547 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7efc009820 con 0x7f7f00101de0 2024-09-16T00:16:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f07d2b640 1 -- 172.21.15.33:0/2512597547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f0019d6d0 con 0x7f7f00101de0 2024-09-16T00:16:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7ef6ffd640 1 -- 172.21.15.33:0/2512597547 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc004d00 con 0x7f7f00101de0 2024-09-16T00:16:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.412+0000 7f7f07d2b640 1 -- 172.21.15.33:0/2512597547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f001a4fb0 con 0x7f7f00101de0 2024-09-16T00:16:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.413+0000 7f7f07d2b640 1 -- 172.21.15.33:0/2512597547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f001021c0 con 0x7f7f00101de0 2024-09-16T00:16:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.415+0000 7f7ef6ffd640 1 -- 172.21.15.33:0/2512597547 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7efc0099d0 con 0x7f7f00101de0 2024-09-16T00:16:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.415+0000 7f7ef6ffd640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7eec0a3350 0x7f7eec0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.416+0000 7f7f052c6640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7eec0a3350 0x7f7eec0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.416+0000 7f7ef6ffd640 1 -- 172.21.15.33:0/2512597547 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f7efc0cf850 con 0x7f7f00101de0 2024-09-16T00:16:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.416+0000 7f7f052c6640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7eec0a3350 0x7f7eec0a5810 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009230 tx=0x7f7ef00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.420+0000 7f7ef6ffd640 1 -- 172.21.15.33:0/2512597547 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7efc09ba60 con 0x7f7f00101de0 2024-09-16T00:16:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.634+0000 7f7f07d2b640 1 -- 172.21.15.33:0/2512597547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f0010a120 con 0x7f7f00101de0 2024-09-16T00:16:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.635+0000 7f7ef6ffd640 1 -- 172.21.15.33:0/2512597547 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7f7efc09fd60 con 0x7f7f00101de0 2024-09-16T00:16:56.637 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:16:56.637 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"btime":"2024-09-16T00:15:52:174386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:15:51.912925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:16:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-09-16T00:16:56.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.637+0000 7f7f07d2b640 1 -- 172.21.15.33:0/2512597547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7eec0a3350 msgr2=0x7f7eec0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:56.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.637+0000 7f7f07d2b640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7eec0a3350 0x7f7eec0a5810 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009230 tx=0x7f7ef00058d0 comp rx=0 tx=0).stop 2024-09-16T00:16:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.637+0000 7f7f07d2b640 1 -- 172.21.15.33:0/2512597547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f00101de0 msgr2=0x7f7f00198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.637+0000 7f7f07d2b640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f00101de0 0x7f7f00198430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00d6d0 tx=0x7f7efc00db90 comp rx=0 tx=0).stop 2024-09-16T00:16:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.637+0000 7f7f07d2b640 1 -- 172.21.15.33:0/2512597547 shutdown_connections 2024-09-16T00:16:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.637+0000 7f7f07d2b640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7eec0a3350 0x7f7eec0a5810 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.637+0000 7f7f07d2b640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f00106f10 0x7f7f0019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.637+0000 7f7f07d2b640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f00102790 0x7f7f00198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.637+0000 7f7f07d2b640 1 --2- 172.21.15.33:0/2512597547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f00101de0 0x7f7f00198430 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.637+0000 7f7f07d2b640 1 -- 172.21.15.33:0/2512597547 >> 172.21.15.33:0/2512597547 conn(0x7f7f000fd8c0 msgr2=0x7f7f000fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.637+0000 7f7f07d2b640 1 -- 172.21.15.33:0/2512597547 shutdown_connections 2024-09-16T00:16:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:56.637+0000 7f7f07d2b640 1 -- 172.21.15.33:0/2512597547 wait complete. 2024-09-16T00:16:57.461 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-16T00:15:07.246355+0000', 'modified': '2024-09-16T00:15:51.912925+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24494, 'mds_1': 24493, 'mds_2': 24472}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.60:6835/1002409397', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6834', 'nonce': 1002409397}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 1002409397}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.60:6837/452012251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6836', 'nonce': 452012251}, {'type': 'v1', 'addr': '172.21.15.60:6837', 'nonce': 452012251}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.83:6835/2667354509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 2667354509}, {'type': 'v1', 'addr': '172.21.15.83:6835', 'nonce': 2667354509}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.83:6837/4244918448', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6836', 'nonce': 4244918448}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 4244918448}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.83:6839/3202868586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6838', 'nonce': 3202868586}, {'type': 'v1', 'addr': '172.21.15.83:6839', 'nonce': 3202868586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.60:6839/1222932045', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6838', 'nonce': 1222932045}, {'type': 'v1', 'addr': '172.21.15.60:6839', 'nonce': 1222932045}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24494, 'qdb_cluster': [24472, 24493, 24494]} 2024-09-16T00:16:57.461 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-16T00:16:57.462 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.c status 2024-09-16T00:16:57.725 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:57.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:57 smithi033 ceph-mon[31243]: pgmap v342: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:57.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:57 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2512597547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:16:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:57 smithi083 ceph-mon[35242]: pgmap v342: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:57.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:57 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2512597547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:16:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:57 smithi060 ceph-mon[35185]: pgmap v342: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:16:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:57 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2512597547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:16:58.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.197+0000 7fa34e6fc640 1 -- 172.21.15.33:0/315091775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348106f10 msgr2=0x7fa34810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:58.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.197+0000 7fa34e6fc640 1 --2- 172.21.15.33:0/315091775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348106f10 0x7fa34810ea90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa340009ca0 tx=0x7fa34002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.197+0000 7fa34e6fc640 1 -- 172.21.15.33:0/315091775 shutdown_connections 2024-09-16T00:16:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.197+0000 7fa34e6fc640 1 --2- 172.21.15.33:0/315091775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348106f10 0x7fa34810ea90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.197+0000 7fa34e6fc640 1 --2- 172.21.15.33:0/315091775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa348102790 0x7fa3481067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.197+0000 7fa34e6fc640 1 --2- 172.21.15.33:0/315091775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa348101de0 0x7fa3481021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.197+0000 7fa34e6fc640 1 -- 172.21.15.33:0/315091775 >> 172.21.15.33:0/315091775 conn(0x7fa3480fd8e0 msgr2=0x7fa3480ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.197+0000 7fa34e6fc640 1 -- 172.21.15.33:0/315091775 shutdown_connections 2024-09-16T00:16:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.197+0000 7fa34e6fc640 1 -- 172.21.15.33:0/315091775 wait complete. 2024-09-16T00:16:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.197+0000 7fa34e6fc640 1 Processor -- start 2024-09-16T00:16:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.197+0000 7fa34e6fc640 1 -- start start 2024-09-16T00:16:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa34e6fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa348101de0 0x7fa348198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa34e6fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348102790 0x7fa348198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa34e6fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa348106f10 0x7fa34819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa34e6fc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa348111f70 con 0x7fa348102790 2024-09-16T00:16:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa34e6fc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa348111df0 con 0x7fa348106f10 2024-09-16T00:16:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa34e6fc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa3481120f0 con 0x7fa348101de0 2024-09-16T00:16:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa3477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348102790 0x7fa348198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa3477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348102790 0x7fa348198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50970/0 (socket says 172.21.15.33:50970) 2024-09-16T00:16:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa3477fe640 1 -- 172.21.15.33:0/1366332145 learned_addr learned my addr 172.21.15.33:0/1366332145 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa347fff640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa348101de0 0x7fa348198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa34cc99640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa348106f10 0x7fa34819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa3477fe640 1 -- 172.21.15.33:0/1366332145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa348101de0 msgr2=0x7fa348198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa3477fe640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa348101de0 0x7fa348198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa3477fe640 1 -- 172.21.15.33:0/1366332145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa348106f10 msgr2=0x7fa34819cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa3477fe640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa348106f10 0x7fa34819cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa3477fe640 1 -- 172.21.15.33:0/1366332145 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa34819d3c0 con 0x7fa348102790 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.198+0000 7fa3477fe640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348102790 0x7fa348198950 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00d950 tx=0x7fa33c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.199+0000 7fa3457fa640 1 -- 172.21.15.33:0/1366332145 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c017030 con 0x7fa348102790 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.199+0000 7fa3457fa640 1 -- 172.21.15.33:0/1366332145 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa33c004020 con 0x7fa348102790 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.199+0000 7fa3457fa640 1 -- 172.21.15.33:0/1366332145 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0047a0 con 0x7fa348102790 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.199+0000 7fa34e6fc640 1 -- 172.21.15.33:0/1366332145 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa34819d6b0 con 0x7fa348102790 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.199+0000 7fa34e6fc640 1 -- 172.21.15.33:0/1366332145 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3481a4f90 con 0x7fa348102790 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.201+0000 7fa3457fa640 1 -- 172.21.15.33:0/1366332145 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa33c0041d0 con 0x7fa348102790 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.201+0000 7fa350ad5740 1 Processor -- start 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.201+0000 7fa350ad5740 1 -- start start 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa350ad5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a33a9d0 0x55d53a33adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa350ad5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33b2f0 0x55d53a345670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa350ad5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a345bb0 0x55d53a347fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa350ad5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d53a2a1090 con 0x55d53a345bb0 2024-09-16T00:16:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa350ad5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d53a248a70 con 0x55d53a33b2f0 2024-09-16T00:16:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa350ad5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d53a29d050 con 0x55d53a33a9d0 2024-09-16T00:16:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa3477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a345bb0 0x55d53a347fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa3477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a345bb0 0x55d53a347fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50986/0 (socket says 172.21.15.33:50986) 2024-09-16T00:16:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa3477fe640 1 -- 172.21.15.33:0/2270316028 learned_addr learned my addr 172.21.15.33:0/2270316028 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa347fff640 1 --2- 172.21.15.33:0/2270316028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33b2f0 0x55d53a345670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa34cc99640 1 --2- 172.21.15.33:0/2270316028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a33a9d0 0x55d53a33adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa3457fa640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3300a32d0 0x7fa3300a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa34cc99640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3300a32d0 0x7fa3300a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa3477fe640 1 -- 172.21.15.33:0/2270316028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a33a9d0 msgr2=0x55d53a33adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa3477fe640 1 --2- 172.21.15.33:0/2270316028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a33a9d0 0x55d53a33adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa3477fe640 1 -- 172.21.15.33:0/2270316028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33b2f0 msgr2=0x55d53a345670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa3477fe640 1 --2- 172.21.15.33:0/2270316028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33b2f0 0x55d53a345670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa3457fa640 1 -- 172.21.15.33:0/1366332145 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fa33c0cf7e0 con 0x7fa348102790 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa3477fe640 1 -- 172.21.15.33:0/2270316028 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d53a255130 con 0x55d53a345bb0 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.202+0000 7fa347fff640 1 --2- 172.21.15.33:0/2270316028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33b2f0 0x55d53a345670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.203+0000 7fa3477fe640 1 --2- 172.21.15.33:0/2270316028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a345bb0 0x55d53a347fa0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa33c09b5e0 tx=0x7fa33c050b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e43acabc12b11e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.203+0000 7fa34cc99640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3300a32d0 0x7fa3300a5790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa340004af0 tx=0x7fa34002a8c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.203+0000 7fa32e7fc640 1 -- 172.21.15.33:0/2270316028 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c09b240 con 0x55d53a345bb0 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.203+0000 7fa32e7fc640 1 -- 172.21.15.33:0/2270316028 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa33c09bfe0 con 0x55d53a345bb0 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.203+0000 7fa32e7fc640 1 -- 172.21.15.33:0/2270316028 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0cd020 con 0x55d53a345bb0 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.203+0000 7fa350ad5740 1 -- 172.21.15.33:0/2270316028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a345bb0 msgr2=0x55d53a347fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.203+0000 7fa350ad5740 1 --2- 172.21.15.33:0/2270316028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a345bb0 0x55d53a347fa0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa33c09b5e0 tx=0x7fa33c050b70 comp rx=0 tx=0).stop 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 -- 172.21.15.33:0/2270316028 shutdown_connections 2024-09-16T00:16:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 --2- 172.21.15.33:0/2270316028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a345bb0 0x55d53a347fa0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 --2- 172.21.15.33:0/2270316028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33b2f0 0x55d53a345670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 --2- 172.21.15.33:0/2270316028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a33a9d0 0x55d53a33adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 -- 172.21.15.33:0/2270316028 >> 172.21.15.33:0/2270316028 conn(0x55d53a336f60 msgr2=0x55d53a337340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 -- 172.21.15.33:0/2270316028 shutdown_connections 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 -- 172.21.15.33:0/2270316028 wait complete. 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 -- start start 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 Processor -- start 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33a9d0 0x55d53a3453c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33b2f0 0x55d53a342710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 0x55d53a342c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d53a348780 con 0x55d53a33a9d0 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d53a25bf20 con 0x55d53a33b2f0 2024-09-16T00:16:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.204+0000 7fa350ad5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d53a348900 con 0x55d53a345bb0 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa350ad5740 10 client.0 _ll_register_callbacks cb 0x55d53a225030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa347fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33a9d0 0x55d53a3453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa347fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33a9d0 0x55d53a3453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50996/0 (socket says 172.21.15.33:50996) 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa347fff640 1 -- 172.21.15.33:0/2765650022 learned_addr learned my addr 172.21.15.33:0/2765650022 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa34cc99640 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 0x55d53a342c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa3477fe640 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33b2f0 0x55d53a342710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa347fff640 1 -- 172.21.15.33:0/2765650022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 msgr2=0x55d53a342c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa347fff640 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 0x55d53a342c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa347fff640 1 -- 172.21.15.33:0/2765650022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33b2f0 msgr2=0x55d53a342710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa347fff640 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33b2f0 0x55d53a342710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa347fff640 1 -- 172.21.15.33:0/2765650022 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d53a0122b0 con 0x55d53a33a9d0 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa347fff640 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33a9d0 0x55d53a3453c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa338010820 tx=0x7fa338010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1817a59510c91c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa3277fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:16:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa3277fe640 1 -- 172.21.15.33:0/2765650022 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33800fb90 con 0x55d53a33a9d0 2024-09-16T00:16:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa3277fe640 1 -- 172.21.15.33:0/2765650022 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fa33800fd30 con 0x55d53a33a9d0 2024-09-16T00:16:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa350ad5740 1 -- 172.21.15.33:0/2765650022 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa33c09bfe0 con 0x55d53a33a9d0 2024-09-16T00:16:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa3277fe640 1 -- 172.21.15.33:0/2765650022 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa338024050 con 0x55d53a33a9d0 2024-09-16T00:16:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa350ad5740 10 client.15069 fetch_fsmap learned FSMap version 25 2024-09-16T00:16:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.205+0000 7fa350ad5740 1 -- 172.21.15.33:0/2765650022 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d53a354070 con 0x55d53a33a9d0 2024-09-16T00:16:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.206+0000 7fa3277fe640 1 -- 172.21.15.33:0/2765650022 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa33c09bfe0 con 0x55d53a33a9d0 2024-09-16T00:16:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.206+0000 7fa3277fe640 1 -- 172.21.15.33:0/2765650022 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fa33800e570 con 0x55d53a33a9d0 2024-09-16T00:16:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.206+0000 7fa350ad5740 10 client.15069 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T00:16:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.206+0000 7fa350ad5740 10 client.15069 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T00:16:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.206+0000 7fa350ad5740 20 client.15069 populate_metadata read hostname 'smithi033' 2024-09-16T00:16:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.206+0000 7fa350ad5740 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d53a356700 0x55d53a357b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.206+0000 7fa350ad5740 4 client.15069 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.206+0000 7fa350ad5740 1 -- 172.21.15.33:0/2765650022 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d53a2ae600 con 0x55d53a356700 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.206+0000 7fa3477fe640 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d53a356700 0x55d53a357b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.207+0000 7fa3277fe640 10 client.15069 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.207+0000 7fa3477fe640 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d53a356700 0x55d53a357b70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4815c33c7933b587 server_cookie=e16026d859af7bc0 in_seq=0 out_seq=0 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.207+0000 7fa3277fe640 1 -- 172.21.15.33:0/2765650022 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d53a2ae600 con 0x55d53a356700 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.207+0000 7fa3277fe640 10 client.15069 handle_command_reply: tid=0 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.207+0000 7fa350ad5740 1 client.15069 shutdown 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.208+0000 7fa350ad5740 1 -- 172.21.15.33:0/2765650022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33a9d0 msgr2=0x55d53a3453c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.208+0000 7fa350ad5740 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33a9d0 0x55d53a3453c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa338010820 tx=0x7fa338010ce0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.208+0000 7fa350ad5740 1 -- 172.21.15.33:0/2765650022 shutdown_connections 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.208+0000 7fa350ad5740 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d53a356700 0x55d53a357b70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.208+0000 7fa350ad5740 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 0x55d53a342c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.208+0000 7fa350ad5740 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33b2f0 0x55d53a342710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.208+0000 7fa350ad5740 1 --2- 172.21.15.33:0/2765650022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33a9d0 0x55d53a3453c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.208+0000 7fa3277fe640 0 client.15069 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:16:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.208+0000 7fa350ad5740 1 -- 172.21.15.33:0/2765650022 >> 172.21.15.33:0/2765650022 conn(0x55d53a336f60 msgr2=0x55d53a33cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.208+0000 7fa350ad5740 1 -- 172.21.15.33:0/2765650022 shutdown_connections 2024-09-16T00:16:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.208+0000 7fa350ad5740 1 -- 172.21.15.33:0/2765650022 wait complete. 2024-09-16T00:16:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.208+0000 7fa350ad5740 20 client.15069 trim_cache size 0 max 16384 2024-09-16T00:16:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa350ad5740 1 Processor -- start 2024-09-16T00:16:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa350ad5740 1 -- start start 2024-09-16T00:16:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa350ad5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33a9d0 0x55d53a337340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa350ad5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 0x55d53a335ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa350ad5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 0x55d53a33f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa350ad5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d53a355eb0 con 0x55d53a33b2f0 2024-09-16T00:16:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa350ad5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d53a356580 con 0x55d53a33a9d0 2024-09-16T00:16:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa350ad5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d53a356c50 con 0x55d53a345bb0 2024-09-16T00:16:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa3477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 0x55d53a335ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa3477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 0x55d53a335ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51002/0 (socket says 172.21.15.33:51002) 2024-09-16T00:16:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa3477fe640 1 -- 172.21.15.33:0/1899979435 learned_addr learned my addr 172.21.15.33:0/1899979435 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa347fff640 1 --2- 172.21.15.33:0/1899979435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33a9d0 0x55d53a337340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa34cc99640 1 --2- 172.21.15.33:0/1899979435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 0x55d53a33f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa3477fe640 1 -- 172.21.15.33:0/1899979435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 msgr2=0x55d53a33f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa3477fe640 1 --2- 172.21.15.33:0/1899979435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 0x55d53a33f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa3477fe640 1 -- 172.21.15.33:0/1899979435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33a9d0 msgr2=0x55d53a337340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.226+0000 7fa3477fe640 1 --2- 172.21.15.33:0/1899979435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33a9d0 0x55d53a337340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.227+0000 7fa3477fe640 1 -- 172.21.15.33:0/1899979435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d53a336d90 con 0x55d53a33b2f0 2024-09-16T00:16:58.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.227+0000 7fa3477fe640 1 --2- 172.21.15.33:0/1899979435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 0x55d53a335ef0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa33c097d40 tx=0x7fa33c0cfa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128360c4147e994c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:58.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.227+0000 7fa32dffb640 1 -- 172.21.15.33:0/1899979435 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c09bfe0 con 0x55d53a33b2f0 2024-09-16T00:16:58.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.227+0000 7fa32dffb640 1 -- 172.21.15.33:0/1899979435 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa33c09fc90 con 0x55d53a33b2f0 2024-09-16T00:16:58.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.227+0000 7fa32dffb640 1 -- 172.21.15.33:0/1899979435 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0980e0 con 0x55d53a33b2f0 2024-09-16T00:16:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.227+0000 7fa350ad5740 1 -- 172.21.15.33:0/1899979435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 msgr2=0x55d53a335ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.227+0000 7fa350ad5740 1 --2- 172.21.15.33:0/1899979435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 0x55d53a335ef0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa33c097d40 tx=0x7fa33c0cfa80 comp rx=0 tx=0).stop 2024-09-16T00:16:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 -- 172.21.15.33:0/1899979435 shutdown_connections 2024-09-16T00:16:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 --2- 172.21.15.33:0/1899979435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 0x55d53a33f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 --2- 172.21.15.33:0/1899979435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 0x55d53a335ef0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 --2- 172.21.15.33:0/1899979435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33a9d0 0x55d53a337340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 -- 172.21.15.33:0/1899979435 >> 172.21.15.33:0/1899979435 conn(0x55d53a336f60 msgr2=0x55d53a3357e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 -- 172.21.15.33:0/1899979435 shutdown_connections 2024-09-16T00:16:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 -- 172.21.15.33:0/1899979435 wait complete. 2024-09-16T00:16:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 -- start start 2024-09-16T00:16:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 Processor -- start 2024-09-16T00:16:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33a9d0 0x55d53a35b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 0x55d53a35b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 0x55d53a3bee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d53a3579f0 con 0x55d53a33b2f0 2024-09-16T00:16:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d53a357320 con 0x55d53a33a9d0 2024-09-16T00:16:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.228+0000 7fa350ad5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d53a3580c0 con 0x55d53a345bb0 2024-09-16T00:16:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa350ad5740 10 client.0 _ll_register_callbacks cb 0x7fa33800a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:16:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 0x55d53a35b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 0x55d53a35b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51012/0 (socket says 172.21.15.33:51012) 2024-09-16T00:16:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3477fe640 1 -- 172.21.15.33:0/1842272398 learned_addr learned my addr 172.21.15.33:0/1842272398 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa347fff640 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33a9d0 0x55d53a35b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa34cc99640 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 0x55d53a3bee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3477fe640 1 -- 172.21.15.33:0/1842272398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 msgr2=0x55d53a3bee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3477fe640 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 0x55d53a3bee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3477fe640 1 -- 172.21.15.33:0/1842272398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33a9d0 msgr2=0x55d53a35b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3477fe640 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33a9d0 0x55d53a35b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3477fe640 1 -- 172.21.15.33:0/1842272398 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d53a3bf440 con 0x55d53a33b2f0 2024-09-16T00:16:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3477fe640 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 0x55d53a35b8d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa33c005e00 tx=0x7fa33c0cd250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db3a97985a99055 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3277fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:16:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3277fe640 1 -- 172.21.15.33:0/1842272398 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c09b8e0 con 0x55d53a33b2f0 2024-09-16T00:16:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3277fe640 1 -- 172.21.15.33:0/1842272398 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fa33c0cfde0 con 0x55d53a33b2f0 2024-09-16T00:16:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa350ad5740 1 -- 172.21.15.33:0/1842272398 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa33c09fc90 con 0x55d53a33b2f0 2024-09-16T00:16:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa3277fe640 1 -- 172.21.15.33:0/1842272398 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa33c099980 con 0x55d53a33b2f0 2024-09-16T00:16:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa350ad5740 10 client.15075 fetch_fsmap learned FSMap version 25 2024-09-16T00:16:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.229+0000 7fa350ad5740 1 -- 172.21.15.33:0/1842272398 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d53a3c0720 con 0x55d53a33b2f0 2024-09-16T00:16:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.230+0000 7fa3277fe640 1 -- 172.21.15.33:0/1842272398 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa33c09fc90 con 0x55d53a33b2f0 2024-09-16T00:16:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.230+0000 7fa3277fe640 1 -- 172.21.15.33:0/1842272398 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fa33c097040 con 0x55d53a33b2f0 2024-09-16T00:16:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.230+0000 7fa350ad5740 10 client.15075 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T00:16:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.230+0000 7fa350ad5740 10 client.15075 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T00:16:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.230+0000 7fa350ad5740 20 client.15075 populate_metadata read hostname 'smithi033' 2024-09-16T00:16:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.230+0000 7fa350ad5740 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d53a3c4b90 0x55d53a3c38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.230+0000 7fa350ad5740 4 client.15075 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T00:16:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.230+0000 7fa350ad5740 1 -- 172.21.15.33:0/1842272398 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "status"}) -- 0x55d53a2a8b10 con 0x55d53a3c4b90 2024-09-16T00:16:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.230+0000 7fa347fff640 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d53a3c4b90 0x55d53a3c38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.230+0000 7fa3277fe640 10 client.15075 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:16:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.230+0000 7fa347fff640 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d53a3c4b90 0x55d53a3c38a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d1729dbc31dc33ac server_cookie=e73fc8b07c2613dc in_seq=0 out_seq=0 2024-09-16T00:16:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa3277fe640 1 -- 172.21.15.33:0/1842272398 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d53a2a8b10 con 0x55d53a3c4b90 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa3277fe640 10 client.15075 handle_command_reply: tid=0 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa350ad5740 1 client.15075 shutdown 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa350ad5740 1 -- 172.21.15.33:0/1842272398 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 msgr2=0x55d53a35b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa350ad5740 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 0x55d53a35b8d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa33c005e00 tx=0x7fa33c0cd250 comp rx=0 tx=0).stop 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa350ad5740 1 -- 172.21.15.33:0/1842272398 shutdown_connections 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa350ad5740 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d53a3c4b90 0x55d53a3c38a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa350ad5740 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d53a345bb0 0x55d53a3bee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa350ad5740 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d53a33b2f0 0x55d53a35b8d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa350ad5740 1 --2- 172.21.15.33:0/1842272398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d53a33a9d0 0x55d53a35b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa350ad5740 1 -- 172.21.15.33:0/1842272398 >> 172.21.15.33:0/1842272398 conn(0x55d53a336f60 msgr2=0x55d53a33cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa3277fe640 0 client.15075 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa350ad5740 1 -- 172.21.15.33:0/1842272398 shutdown_connections 2024-09-16T00:16:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa350ad5740 1 -- 172.21.15.33:0/1842272398 wait complete. 2024-09-16T00:16:58.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.231+0000 7fa350ad5740 20 client.15075 trim_cache size 0 max 16384 2024-09-16T00:16:58.239 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:16:58.239 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:16:58.239 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 2, 2024-09-16T00:16:58.239 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24472, 2024-09-16T00:16:58.239 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-09-16T00:16:58.239 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-09-16T00:16:58.239 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-09-16T00:16:58.239 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 69.063401912000003, 2024-09-16T00:16:58.240 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 25, 2024-09-16T00:16:58.240 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 88, 2024-09-16T00:16:58.240 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 86, 2024-09-16T00:16:58.240 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 136.25108117600001 2024-09-16T00:16:58.240 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:16:58.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.232+0000 7fa34e6fc640 1 -- 172.21.15.33:0/1366332145 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3300a32d0 msgr2=0x7fa3300a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:58.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.232+0000 7fa34e6fc640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3300a32d0 0x7fa3300a5790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa340004af0 tx=0x7fa34002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.232+0000 7fa34e6fc640 1 -- 172.21.15.33:0/1366332145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348102790 msgr2=0x7fa348198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:58.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.232+0000 7fa34e6fc640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348102790 0x7fa348198950 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00d950 tx=0x7fa33c00de10 comp rx=0 tx=0).stop 2024-09-16T00:16:58.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.232+0000 7fa34e6fc640 1 -- 172.21.15.33:0/1366332145 shutdown_connections 2024-09-16T00:16:58.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.232+0000 7fa34e6fc640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3300a32d0 0x7fa3300a5790 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.232+0000 7fa34e6fc640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa348106f10 0x7fa34819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.232+0000 7fa34e6fc640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348102790 0x7fa348198950 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.232+0000 7fa34e6fc640 1 --2- 172.21.15.33:0/1366332145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa348101de0 0x7fa348198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:58.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.232+0000 7fa34e6fc640 1 -- 172.21.15.33:0/1366332145 >> 172.21.15.33:0/1366332145 conn(0x7fa3480fd8e0 msgr2=0x7fa3480febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:58.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.232+0000 7fa34e6fc640 1 -- 172.21.15.33:0/1366332145 shutdown_connections 2024-09-16T00:16:58.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:58.232+0000 7fa34e6fc640 1 -- 172.21.15.33:0/1366332145 wait complete. 2024-09-16T00:16:58.962 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-16T00:16:59.218 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:16:59.647 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:16:59 smithi033 ceph-mon[31243]: pgmap v343: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:16:59.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.729+0000 7fb3f739a640 1 -- 172.21.15.33:0/3728161872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f010c2f0 msgr2=0x7fb3f010e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:59.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.729+0000 7fb3f739a640 1 --2- 172.21.15.33:0/3728161872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f010c2f0 0x7fb3f010e790 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8009ca0 tx=0x7fb3e802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.729+0000 7fb3f739a640 1 -- 172.21.15.33:0/3728161872 shutdown_connections 2024-09-16T00:16:59.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.729+0000 7fb3f739a640 1 --2- 172.21.15.33:0/3728161872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f010c2f0 0x7fb3f010e790 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.729+0000 7fb3f739a640 1 --2- 172.21.15.33:0/3728161872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f01037a0 0x7fb3f010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.729+0000 7fb3f739a640 1 --2- 172.21.15.33:0/3728161872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f0102df0 0x7fb3f01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.729+0000 7fb3f739a640 1 -- 172.21.15.33:0/3728161872 >> 172.21.15.33:0/3728161872 conn(0x7fb3f00fc7d0 msgr2=0x7fb3f00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:59.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.729+0000 7fb3f739a640 1 -- 172.21.15.33:0/3728161872 shutdown_connections 2024-09-16T00:16:59.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.729+0000 7fb3f739a640 1 -- 172.21.15.33:0/3728161872 wait complete. 2024-09-16T00:16:59.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f739a640 1 Processor -- start 2024-09-16T00:16:59.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f739a640 1 -- start start 2024-09-16T00:16:59.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f739a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f0102df0 0x7fb3f0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f739a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f01037a0 0x7fb3f0198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f739a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f010c2f0 0x7fb3f019cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f739a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f0112170 con 0x7fb3f0102df0 2024-09-16T00:16:59.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f739a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f0111ff0 con 0x7fb3f01037a0 2024-09-16T00:16:59.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f739a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f01122f0 con 0x7fb3f010c2f0 2024-09-16T00:16:59.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f5136640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f0102df0 0x7fb3f0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f5136640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f0102df0 0x7fb3f0198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51020/0 (socket says 172.21.15.33:51020) 2024-09-16T00:16:59.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f5136640 1 -- 172.21.15.33:0/1952757821 learned_addr learned my addr 172.21.15.33:0/1952757821 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:59.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f4935640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f01037a0 0x7fb3f0198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.730+0000 7fb3f5937640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f010c2f0 0x7fb3f019cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.731+0000 7fb3f5136640 1 -- 172.21.15.33:0/1952757821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f010c2f0 msgr2=0x7fb3f019cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:59.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.731+0000 7fb3f5136640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f010c2f0 0x7fb3f019cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.731+0000 7fb3f5136640 1 -- 172.21.15.33:0/1952757821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f01037a0 msgr2=0x7fb3f0198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:59.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.731+0000 7fb3f5136640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f01037a0 0x7fb3f0198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.731+0000 7fb3f5136640 1 -- 172.21.15.33:0/1952757821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f019d5e0 con 0x7fb3f0102df0 2024-09-16T00:16:59.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.731+0000 7fb3f5136640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f0102df0 0x7fb3f0198630 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00b4a0 tx=0x7fb3ec00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:59.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.731+0000 7fb3e67fc640 1 -- 172.21.15.33:0/1952757821 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec014070 con 0x7fb3f0102df0 2024-09-16T00:16:59.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.731+0000 7fb3f739a640 1 -- 172.21.15.33:0/1952757821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3f019d8d0 con 0x7fb3f0102df0 2024-09-16T00:16:59.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.731+0000 7fb3e67fc640 1 -- 172.21.15.33:0/1952757821 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3ec010af0 con 0x7fb3f0102df0 2024-09-16T00:16:59.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.731+0000 7fb3e67fc640 1 -- 172.21.15.33:0/1952757821 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec017d10 con 0x7fb3f0102df0 2024-09-16T00:16:59.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.731+0000 7fb3f739a640 1 -- 172.21.15.33:0/1952757821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3f01a51b0 con 0x7fb3f0102df0 2024-09-16T00:16:59.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.733+0000 7fb3e67fc640 1 -- 172.21.15.33:0/1952757821 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fb3ec017070 con 0x7fb3f0102df0 2024-09-16T00:16:59.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.734+0000 7fb3f9773740 1 Processor -- start 2024-09-16T00:16:59.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.734+0000 7fb3f9773740 1 -- start start 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.734+0000 7fb3f9773740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102359d0 0x562d10235db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.734+0000 7fb3f9773740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d102362f0 0x562d10240670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.734+0000 7fb3f9773740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d10240bb0 0x562d10242fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.734+0000 7fb3f9773740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562d1019c090 con 0x562d10240bb0 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.734+0000 7fb3f9773740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562d10143a70 con 0x562d102362f0 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.734+0000 7fb3f9773740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562d10198050 con 0x562d102359d0 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f4935640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d10240bb0 0x562d10242fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f4935640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d10240bb0 0x562d10242fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51028/0 (socket says 172.21.15.33:51028) 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f4935640 1 -- 172.21.15.33:0/1208210333 learned_addr learned my addr 172.21.15.33:0/1208210333 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3e67fc640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3dc0a32d0 0x7fb3dc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f5937640 1 --2- 172.21.15.33:0/1208210333 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102359d0 0x562d10235db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f5136640 1 --2- 172.21.15.33:0/1208210333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d102362f0 0x562d10240670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f5937640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3dc0a32d0 0x7fb3dc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f4935640 1 -- 172.21.15.33:0/1208210333 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102359d0 msgr2=0x562d10235db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3e67fc640 1 -- 172.21.15.33:0/1952757821 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fb3ec0cf120 con 0x7fb3f0102df0 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f4935640 1 --2- 172.21.15.33:0/1208210333 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102359d0 0x562d10235db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f4935640 1 -- 172.21.15.33:0/1208210333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d102362f0 msgr2=0x562d10240670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f4935640 1 --2- 172.21.15.33:0/1208210333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d102362f0 0x562d10240670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f4935640 1 -- 172.21.15.33:0/1208210333 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d10150130 con 0x562d10240bb0 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f5136640 1 --2- 172.21.15.33:0/1208210333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d102362f0 0x562d10240670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f4935640 1 --2- 172.21.15.33:0/1208210333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d10240bb0 0x562d10242fa0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000bc30 tx=0x7fb3e00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae050be6e29e46 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3db7fe640 1 -- 172.21.15.33:0/1208210333 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e001c480 con 0x562d10240bb0 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3f5937640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3dc0a32d0 0x7fb3dc0a5790 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8009a60 tx=0x7fb3e80095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3db7fe640 1 -- 172.21.15.33:0/1208210333 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3e001cb00 con 0x562d10240bb0 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.735+0000 7fb3db7fe640 1 -- 172.21.15.33:0/1208210333 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0011bc0 con 0x562d10240bb0 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.736+0000 7fb3f9773740 1 -- 172.21.15.33:0/1208210333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d10240bb0 msgr2=0x562d10242fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:59.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.736+0000 7fb3f9773740 1 --2- 172.21.15.33:0/1208210333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d10240bb0 0x562d10242fa0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000bc30 tx=0x7fb3e00107a0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.736+0000 7fb3f9773740 1 -- 172.21.15.33:0/1208210333 shutdown_connections 2024-09-16T00:16:59.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.736+0000 7fb3f9773740 1 --2- 172.21.15.33:0/1208210333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d10240bb0 0x562d10242fa0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.736+0000 7fb3f9773740 1 --2- 172.21.15.33:0/1208210333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d102362f0 0x562d10240670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.736+0000 7fb3f9773740 1 --2- 172.21.15.33:0/1208210333 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102359d0 0x562d10235db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.736+0000 7fb3f9773740 1 -- 172.21.15.33:0/1208210333 >> 172.21.15.33:0/1208210333 conn(0x562d10231f60 msgr2=0x562d10232340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:59.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.736+0000 7fb3f9773740 1 -- 172.21.15.33:0/1208210333 shutdown_connections 2024-09-16T00:16:59.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.736+0000 7fb3f9773740 1 -- 172.21.15.33:0/1208210333 wait complete. 2024-09-16T00:16:59.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f9773740 1 -- start start 2024-09-16T00:16:59.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f9773740 1 Processor -- start 2024-09-16T00:16:59.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f9773740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 0x562d102403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f9773740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102362f0 0x562d1023d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f9773740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d10240bb0 0x562d1023dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f9773740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562d10243780 con 0x562d102359d0 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f9773740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562d10156f20 con 0x562d10240bb0 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f9773740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562d10243900 con 0x562d102362f0 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f9773740 10 client.0 _ll_register_callbacks cb 0x562d10120030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f5136640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 0x562d102403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f5136640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 0x562d102403c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51030/0 (socket says 172.21.15.33:51030) 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f5136640 1 -- 172.21.15.33:0/3557513113 learned_addr learned my addr 172.21.15.33:0/3557513113 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f4935640 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102362f0 0x562d1023d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.737+0000 7fb3f5136640 1 -- 172.21.15.33:0/3557513113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102362f0 msgr2=0x562d1023d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f5937640 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d10240bb0 0x562d1023dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f5136640 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102362f0 0x562d1023d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f5136640 1 -- 172.21.15.33:0/3557513113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d10240bb0 msgr2=0x562d1023dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:59.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f5136640 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d10240bb0 0x562d1023dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f5136640 1 -- 172.21.15.33:0/3557513113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d0ff0d2b0 con 0x562d102359d0 2024-09-16T00:16:59.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f4935640 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102362f0 0x562d1023d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:16:59.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f5937640 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d10240bb0 0x562d1023dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:16:59.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f5136640 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 0x562d102403c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00be60 tx=0x7fb3ec06a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e29673da7ce5b94 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:59.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:16:59.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3d8ff9640 1 -- 172.21.15.33:0/3557513113 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec01e070 con 0x562d102359d0 2024-09-16T00:16:59.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3d8ff9640 1 -- 172.21.15.33:0/3557513113 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fb3ec0979a0 con 0x562d102359d0 2024-09-16T00:16:59.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f9773740 1 -- 172.21.15.33:0/3557513113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3e001cb00 con 0x562d102359d0 2024-09-16T00:16:59.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3d8ff9640 1 -- 172.21.15.33:0/3557513113 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb3ec09b330 con 0x562d102359d0 2024-09-16T00:16:59.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f9773740 10 client.15087 fetch_fsmap learned FSMap version 25 2024-09-16T00:16:59.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f9773740 1 -- 172.21.15.33:0/3557513113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562d1024f070 con 0x562d102359d0 2024-09-16T00:16:59.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3d8ff9640 1 -- 172.21.15.33:0/3557513113 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3e001cb00 con 0x562d102359d0 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3d8ff9640 1 -- 172.21.15.33:0/3557513113 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fb3ec0d10d0 con 0x562d102359d0 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f9773740 10 client.15087 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f9773740 10 client.15087 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f9773740 20 client.15087 populate_metadata read hostname 'smithi033' 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f9773740 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x562d10251600 0x562d10252a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f9773740 4 client.15087 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.738+0000 7fb3f9773740 1 -- 172.21.15.33:0/3557513113 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d101a9600 con 0x562d10251600 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.739+0000 7fb3f4935640 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x562d10251600 0x562d10252a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.739+0000 7fb3f4935640 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x562d10251600 0x562d10252a70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4203920fe8c7714a server_cookie=ffb78e92d90847ee in_seq=0 out_seq=0 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.739+0000 7fb3d8ff9640 10 client.15087 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.740+0000 7fb3d8ff9640 1 -- 172.21.15.33:0/3557513113 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562d101a9600 con 0x562d10251600 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.740+0000 7fb3d8ff9640 10 client.15087 handle_command_reply: tid=0 2024-09-16T00:16:59.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.740+0000 7fb3f9773740 1 client.15087 shutdown 2024-09-16T00:16:59.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.740+0000 7fb3f9773740 1 -- 172.21.15.33:0/3557513113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 msgr2=0x562d102403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:59.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.740+0000 7fb3f9773740 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 0x562d102403c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00be60 tx=0x7fb3ec06a070 comp rx=0 tx=0).stop 2024-09-16T00:16:59.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.740+0000 7fb3f9773740 1 -- 172.21.15.33:0/3557513113 shutdown_connections 2024-09-16T00:16:59.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.740+0000 7fb3f9773740 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x562d10251600 0x562d10252a70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.740+0000 7fb3f9773740 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d10240bb0 0x562d1023dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.740+0000 7fb3f9773740 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102362f0 0x562d1023d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.740+0000 7fb3f9773740 1 --2- 172.21.15.33:0/3557513113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 0x562d102403c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.740+0000 7fb3d8ff9640 0 client.15087 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:16:59.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.740+0000 7fb3f9773740 1 -- 172.21.15.33:0/3557513113 >> 172.21.15.33:0/3557513113 conn(0x562d10231f60 msgr2=0x562d10237c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:59.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.741+0000 7fb3f9773740 1 -- 172.21.15.33:0/3557513113 shutdown_connections 2024-09-16T00:16:59.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.741+0000 7fb3f9773740 1 -- 172.21.15.33:0/3557513113 wait complete. 2024-09-16T00:16:59.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.741+0000 7fb3f9773740 20 client.15087 trim_cache size 0 max 16384 2024-09-16T00:16:59.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.758+0000 7fb3f9773740 1 Processor -- start 2024-09-16T00:16:59.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.758+0000 7fb3f9773740 1 -- start start 2024-09-16T00:16:59.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f9773740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d102359d0 0x562d10252a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f9773740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102362f0 0x562d10232340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f9773740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d10240bb0 0x562d10232880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f9773740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562d10250db0 con 0x562d102362f0 2024-09-16T00:16:59.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f9773740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562d10251480 con 0x562d102359d0 2024-09-16T00:16:59.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f9773740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562d10251b50 con 0x562d10240bb0 2024-09-16T00:16:59.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f4935640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102362f0 0x562d10232340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f4935640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102362f0 0x562d10232340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51046/0 (socket says 172.21.15.33:51046) 2024-09-16T00:16:59.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f4935640 1 -- 172.21.15.33:0/4067670728 learned_addr learned my addr 172.21.15.33:0/4067670728 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:59.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f4935640 1 -- 172.21.15.33:0/4067670728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d10240bb0 msgr2=0x562d10232880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f4935640 1 --2- 172.21.15.33:0/4067670728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d10240bb0 0x562d10232880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f4935640 1 -- 172.21.15.33:0/4067670728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d102359d0 msgr2=0x562d10252a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f4935640 1 --2- 172.21.15.33:0/4067670728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d102359d0 0x562d10252a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f4935640 1 -- 172.21.15.33:0/4067670728 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d10236120 con 0x562d102362f0 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3f4935640 1 --2- 172.21.15.33:0/4067670728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102362f0 0x562d10232340 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000b9c0 tx=0x7fb3e0013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420a559de0cda25d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3daffd640 1 -- 172.21.15.33:0/4067670728 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e002c030 con 0x562d102362f0 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3daffd640 1 -- 172.21.15.33:0/4067670728 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3e000bde0 con 0x562d102362f0 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.759+0000 7fb3daffd640 1 -- 172.21.15.33:0/4067670728 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e00035f0 con 0x562d102362f0 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.760+0000 7fb3f9773740 1 -- 172.21.15.33:0/4067670728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102362f0 msgr2=0x562d10232340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.760+0000 7fb3f9773740 1 --2- 172.21.15.33:0/4067670728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102362f0 0x562d10232340 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000b9c0 tx=0x7fb3e0013040 comp rx=0 tx=0).stop 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.760+0000 7fb3f9773740 1 -- 172.21.15.33:0/4067670728 shutdown_connections 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.760+0000 7fb3f9773740 1 --2- 172.21.15.33:0/4067670728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d10240bb0 0x562d10232880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.760+0000 7fb3f9773740 1 --2- 172.21.15.33:0/4067670728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102362f0 0x562d10232340 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.760+0000 7fb3f9773740 1 --2- 172.21.15.33:0/4067670728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d102359d0 0x562d10252a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.760+0000 7fb3f9773740 1 -- 172.21.15.33:0/4067670728 >> 172.21.15.33:0/4067670728 conn(0x562d10231f60 msgr2=0x562d1023d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.760+0000 7fb3f9773740 1 -- 172.21.15.33:0/4067670728 shutdown_connections 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.760+0000 7fb3f9773740 1 -- 172.21.15.33:0/4067670728 wait complete. 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.760+0000 7fb3f9773740 1 -- start start 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.760+0000 7fb3f9773740 1 Processor -- start 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f9773740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 0x562d10253b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f9773740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102362f0 0x562d102540c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f9773740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d10240bb0 0x562d10254600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f9773740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562d102528f0 con 0x562d102359d0 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f9773740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562d10252220 con 0x562d10240bb0 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f9773740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562d10252fc0 con 0x562d102362f0 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f9773740 10 client.0 _ll_register_callbacks cb 0x7fb3ec00b3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:16:59.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f5136640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 0x562d10253b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f5136640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 0x562d10253b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51050/0 (socket says 172.21.15.33:51050) 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f5136640 1 -- 172.21.15.33:0/2032553555 learned_addr learned my addr 172.21.15.33:0/2032553555 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f5937640 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d10240bb0 0x562d10254600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f4935640 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102362f0 0x562d102540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f5136640 1 -- 172.21.15.33:0/2032553555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102362f0 msgr2=0x562d102540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f5136640 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102362f0 0x562d102540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f5136640 1 -- 172.21.15.33:0/2032553555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d10240bb0 msgr2=0x562d10254600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f5136640 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d10240bb0 0x562d10254600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f5136640 1 -- 172.21.15.33:0/2032553555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d102b9d30 con 0x562d102359d0 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f4935640 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102362f0 0x562d102540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f5136640 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 0x562d10253b80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0a3a70 tx=0x7fb3ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6768678bd12c0323 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3d8ff9640 1 -- 172.21.15.33:0/2032553555 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec01e070 con 0x562d102359d0 2024-09-16T00:16:59.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3d8ff9640 1 -- 172.21.15.33:0/2032553555 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fb3ec0d10d0 con 0x562d102359d0 2024-09-16T00:16:59.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.761+0000 7fb3f9773740 1 -- 172.21.15.33:0/2032553555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3e000bde0 con 0x562d102359d0 2024-09-16T00:16:59.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.762+0000 7fb3d8ff9640 1 -- 172.21.15.33:0/2032553555 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb3ec09b100 con 0x562d102359d0 2024-09-16T00:16:59.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.762+0000 7fb3f9773740 10 client.15093 fetch_fsmap learned FSMap version 25 2024-09-16T00:16:59.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.762+0000 7fb3f9773740 1 -- 172.21.15.33:0/2032553555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562d102bb030 con 0x562d102359d0 2024-09-16T00:16:59.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.762+0000 7fb3d8ff9640 1 -- 172.21.15.33:0/2032553555 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3e000bde0 con 0x562d102359d0 2024-09-16T00:16:59.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.762+0000 7fb3d8ff9640 1 -- 172.21.15.33:0/2032553555 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fb3ec0bb0b0 con 0x562d102359d0 2024-09-16T00:16:59.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.762+0000 7fb3f9773740 10 client.15093 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T00:16:59.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.762+0000 7fb3f9773740 10 client.15093 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-16T00:16:59.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.762+0000 7fb3f9773740 20 client.15093 populate_metadata read hostname 'smithi033' 2024-09-16T00:16:59.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.762+0000 7fb3f9773740 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x562d102bfa40 0x562d102be750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:16:59.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.762+0000 7fb3f9773740 4 client.15093 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.762+0000 7fb3f9773740 1 -- 172.21.15.33:0/2032553555 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "status"}) -- 0x562d101a3b10 con 0x562d102bfa40 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.762+0000 7fb3f4935640 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x562d102bfa40 0x562d102be750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3d8ff9640 10 client.15093 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f4935640 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x562d102bfa40 0x562d102be750 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6334c25cacb7323e server_cookie=f6a271eeeb460b35 in_seq=0 out_seq=0 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3d8ff9640 1 -- 172.21.15.33:0/2032553555 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x562d101a3b10 con 0x562d102bfa40 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3d8ff9640 10 client.15093 handle_command_reply: tid=0 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f9773740 1 client.15093 shutdown 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f9773740 1 -- 172.21.15.33:0/2032553555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 msgr2=0x562d10253b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f9773740 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 0x562d10253b80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0a3a70 tx=0x7fb3ec0046c0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f9773740 1 -- 172.21.15.33:0/2032553555 shutdown_connections 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f9773740 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x562d102bfa40 0x562d102be750 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f9773740 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562d10240bb0 0x562d10254600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f9773740 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562d102362f0 0x562d102540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f9773740 1 --2- 172.21.15.33:0/2032553555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562d102359d0 0x562d10253b80 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f9773740 1 -- 172.21.15.33:0/2032553555 >> 172.21.15.33:0/2032553555 conn(0x562d10231f60 msgr2=0x562d1023d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:59.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3d8ff9640 0 client.15093 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:16:59.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f9773740 1 -- 172.21.15.33:0/2032553555 shutdown_connections 2024-09-16T00:16:59.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f9773740 1 -- 172.21.15.33:0/2032553555 wait complete. 2024-09-16T00:16:59.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.763+0000 7fb3f9773740 20 client.15093 trim_cache size 0 max 16384 2024-09-16T00:16:59.791 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:16:59.791 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:16:59.791 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 1, 2024-09-16T00:16:59.791 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24493, 2024-09-16T00:16:59.791 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-09-16T00:16:59.792 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-09-16T00:16:59.792 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-09-16T00:16:59.792 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 71.600520466000006, 2024-09-16T00:16:59.792 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 25, 2024-09-16T00:16:59.792 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 88, 2024-09-16T00:16:59.792 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 86, 2024-09-16T00:16:59.792 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 132.46700695300001 2024-09-16T00:16:59.792 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:16:59.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.764+0000 7fb3f739a640 1 -- 172.21.15.33:0/1952757821 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3dc0a32d0 msgr2=0x7fb3dc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:59.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.764+0000 7fb3f739a640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3dc0a32d0 0x7fb3dc0a5790 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8009a60 tx=0x7fb3e80095c0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.764+0000 7fb3f739a640 1 -- 172.21.15.33:0/1952757821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f0102df0 msgr2=0x7fb3f0198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:16:59.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.764+0000 7fb3f739a640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f0102df0 0x7fb3f0198630 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00b4a0 tx=0x7fb3ec00b960 comp rx=0 tx=0).stop 2024-09-16T00:16:59.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.764+0000 7fb3f739a640 1 -- 172.21.15.33:0/1952757821 shutdown_connections 2024-09-16T00:16:59.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.764+0000 7fb3f739a640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3dc0a32d0 0x7fb3dc0a5790 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.764+0000 7fb3f739a640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f010c2f0 0x7fb3f019cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.764+0000 7fb3f739a640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f01037a0 0x7fb3f0198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.764+0000 7fb3f739a640 1 --2- 172.21.15.33:0/1952757821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3f0102df0 0x7fb3f0198630 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:16:59.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.764+0000 7fb3f739a640 1 -- 172.21.15.33:0/1952757821 >> 172.21.15.33:0/1952757821 conn(0x7fb3f00fc7d0 msgr2=0x7fb3f00fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:16:59.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.764+0000 7fb3f739a640 1 -- 172.21.15.33:0/1952757821 shutdown_connections 2024-09-16T00:16:59.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:16:59.764+0000 7fb3f739a640 1 -- 172.21.15.33:0/1952757821 wait complete. 2024-09-16T00:16:59.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:16:59 smithi083 ceph-mon[35242]: pgmap v343: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:16:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:16:59 smithi060 ceph-mon[35185]: pgmap v343: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:17:00.682 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-16T00:17:00.959 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.483+0000 7fc8362a8640 1 -- 172.21.15.33:0/501742680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8301082e0 msgr2=0x7fc83010ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.483+0000 7fc8362a8640 1 --2- 172.21.15.33:0/501742680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8301082e0 0x7fc83010ec50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc828009cd0 tx=0x7fc82802a8e0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.484+0000 7fc8362a8640 1 -- 172.21.15.33:0/501742680 shutdown_connections 2024-09-16T00:17:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.484+0000 7fc8362a8640 1 --2- 172.21.15.33:0/501742680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8301082e0 0x7fc83010ec50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.484+0000 7fc8362a8640 1 --2- 172.21.15.33:0/501742680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc830103760 0x7fc830107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.484+0000 7fc8362a8640 1 --2- 172.21.15.33:0/501742680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc830102db0 0x7fc830103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.484+0000 7fc8362a8640 1 -- 172.21.15.33:0/501742680 >> 172.21.15.33:0/501742680 conn(0x7fc8300fc7d0 msgr2=0x7fc8300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:01.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.484+0000 7fc8362a8640 1 -- 172.21.15.33:0/501742680 shutdown_connections 2024-09-16T00:17:01.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.484+0000 7fc8362a8640 1 -- 172.21.15.33:0/501742680 wait complete. 2024-09-16T00:17:01.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.484+0000 7fc8362a8640 1 Processor -- start 2024-09-16T00:17:01.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.484+0000 7fc8362a8640 1 -- start start 2024-09-16T00:17:01.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc8362a8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc830102db0 0x7fc830198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc8362a8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc830103760 0x7fc830198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc8362a8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8301082e0 0x7fc83019ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc8362a8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc8301120b0 con 0x7fc830103760 2024-09-16T00:17:01.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc8362a8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc830111f30 con 0x7fc830102db0 2024-09-16T00:17:01.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc8362a8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc830112230 con 0x7fc8301082e0 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc82ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc830102db0 0x7fc830198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc827fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc830103760 0x7fc830198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc82ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc830102db0 0x7fc830198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:43038/0 (socket says 172.21.15.33:43038) 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc827fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc830103760 0x7fc830198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55134/0 (socket says 172.21.15.33:55134) 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc82ffff640 1 -- 172.21.15.33:0/3441069491 learned_addr learned my addr 172.21.15.33:0/3441069491 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc834845640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8301082e0 0x7fc83019ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc827fff640 1 -- 172.21.15.33:0/3441069491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8301082e0 msgr2=0x7fc83019ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc827fff640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8301082e0 0x7fc83019ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc827fff640 1 -- 172.21.15.33:0/3441069491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc830102db0 msgr2=0x7fc830198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc827fff640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc830102db0 0x7fc830198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc827fff640 1 -- 172.21.15.33:0/3441069491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83019d520 con 0x7fc830103760 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.485+0000 7fc82ffff640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc830102db0 0x7fc830198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:17:01.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.486+0000 7fc827fff640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc830103760 0x7fc830198ab0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc83006a2f0 tx=0x7fc81800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:01.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.486+0000 7fc82dffb640 1 -- 172.21.15.33:0/3441069491 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc818017070 con 0x7fc830103760 2024-09-16T00:17:01.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.486+0000 7fc82dffb640 1 -- 172.21.15.33:0/3441069491 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8180099a0 con 0x7fc830103760 2024-09-16T00:17:01.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.486+0000 7fc8362a8640 1 -- 172.21.15.33:0/3441069491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83019d810 con 0x7fc830103760 2024-09-16T00:17:01.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.486+0000 7fc82dffb640 1 -- 172.21.15.33:0/3441069491 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8180047a0 con 0x7fc830103760 2024-09-16T00:17:01.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.487+0000 7fc8362a8640 1 -- 172.21.15.33:0/3441069491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8301a50f0 con 0x7fc830103760 2024-09-16T00:17:01.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.489+0000 7fc82dffb640 1 -- 172.21.15.33:0/3441069491 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc818009b50 con 0x7fc830103760 2024-09-16T00:17:01.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.489+0000 7fc838681740 1 Processor -- start 2024-09-16T00:17:01.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.489+0000 7fc838681740 1 -- start start 2024-09-16T00:17:01.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.489+0000 7fc838681740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5849d0 0x55bbcf584db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.489+0000 7fc838681740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf5852f0 0x55bbcf58f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc838681740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf58fbb0 0x55bbcf591fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc838681740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bbcf4eb090 con 0x55bbcf58fbb0 2024-09-16T00:17:01.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc838681740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55bbcf492a70 con 0x55bbcf5852f0 2024-09-16T00:17:01.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc838681740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bbcf4e7050 con 0x55bbcf5849d0 2024-09-16T00:17:01.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc827fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf58fbb0 0x55bbcf591fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc82ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf5852f0 0x55bbcf58f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc827fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf58fbb0 0x55bbcf591fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55142/0 (socket says 172.21.15.33:55142) 2024-09-16T00:17:01.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc827fff640 1 -- 172.21.15.33:0/1606803952 learned_addr learned my addr 172.21.15.33:0/1606803952 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:01.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc834845640 1 --2- 172.21.15.33:0/1606803952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5849d0 0x55bbcf584db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc82ffff640 1 -- 172.21.15.33:0/1606803952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5849d0 msgr2=0x55bbcf584db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:01.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc82ffff640 1 --2- 172.21.15.33:0/1606803952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5849d0 0x55bbcf584db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc82ffff640 1 -- 172.21.15.33:0/1606803952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf58fbb0 msgr2=0x55bbcf591fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:01.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc82ffff640 1 --2- 172.21.15.33:0/1606803952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf58fbb0 0x55bbcf591fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc82ffff640 1 -- 172.21.15.33:0/1606803952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbcf49f130 con 0x55bbcf5852f0 2024-09-16T00:17:01.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc834845640 1 --2- 172.21.15.33:0/1606803952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5849d0 0x55bbcf584db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:01.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc82dffb640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc81c0a31b0 0x7fc81c0a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc82ffff640 1 --2- 172.21.15.33:0/1606803952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf5852f0 0x55bbcf58f670 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc8200058c0 tx=0x7fc8200109e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59d3be7751ad97c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:01.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.490+0000 7fc834845640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc81c0a31b0 0x7fc81c0a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.491+0000 7fc82dffb640 1 -- 172.21.15.33:0/3441069491 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fc8180cf330 con 0x7fc830103760 2024-09-16T00:17:01.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.491+0000 7fc8267fc640 1 -- 172.21.15.33:0/1606803952 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8200117c0 con 0x55bbcf5852f0 2024-09-16T00:17:01.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.491+0000 7fc8267fc640 1 -- 172.21.15.33:0/1606803952 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc820011e40 con 0x55bbcf5852f0 2024-09-16T00:17:01.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.491+0000 7fc8267fc640 1 -- 172.21.15.33:0/1606803952 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82000d340 con 0x55bbcf5852f0 2024-09-16T00:17:01.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.492+0000 7fc834845640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc81c0a31b0 0x7fc81c0a5670 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc828005cc0 tx=0x7fc828005c30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.492+0000 7fc838681740 1 -- 172.21.15.33:0/1606803952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf5852f0 msgr2=0x55bbcf58f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.492+0000 7fc838681740 1 --2- 172.21.15.33:0/1606803952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf5852f0 0x55bbcf58f670 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc8200058c0 tx=0x7fc8200109e0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.492+0000 7fc838681740 1 -- 172.21.15.33:0/1606803952 shutdown_connections 2024-09-16T00:17:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.492+0000 7fc838681740 1 --2- 172.21.15.33:0/1606803952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf58fbb0 0x55bbcf591fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.492+0000 7fc838681740 1 --2- 172.21.15.33:0/1606803952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf5852f0 0x55bbcf58f670 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.492+0000 7fc838681740 1 --2- 172.21.15.33:0/1606803952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5849d0 0x55bbcf584db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.492+0000 7fc838681740 1 -- 172.21.15.33:0/1606803952 >> 172.21.15.33:0/1606803952 conn(0x55bbcf580f60 msgr2=0x55bbcf581340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.492+0000 7fc838681740 1 -- 172.21.15.33:0/1606803952 shutdown_connections 2024-09-16T00:17:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.492+0000 7fc838681740 1 -- 172.21.15.33:0/1606803952 wait complete. 2024-09-16T00:17:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.492+0000 7fc838681740 1 -- start start 2024-09-16T00:17:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.492+0000 7fc838681740 1 Processor -- start 2024-09-16T00:17:01.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc838681740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 0x55bbcf58f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc838681740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5852f0 0x55bbcf58c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc838681740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf58fbb0 0x55bbcf58cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc838681740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bbcf592780 con 0x55bbcf5849d0 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc838681740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55bbcf4a5f20 con 0x55bbcf58fbb0 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc838681740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bbcf592900 con 0x55bbcf5852f0 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc838681740 10 client.0 _ll_register_callbacks cb 0x55bbcf46f030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc82ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 0x55bbcf58f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc82ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 0x55bbcf58f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55144/0 (socket says 172.21.15.33:55144) 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc82ffff640 1 -- 172.21.15.33:0/594290751 learned_addr learned my addr 172.21.15.33:0/594290751 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc827fff640 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5852f0 0x55bbcf58c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc834845640 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf58fbb0 0x55bbcf58cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc82ffff640 1 -- 172.21.15.33:0/594290751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5852f0 msgr2=0x55bbcf58c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:01.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc82ffff640 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5852f0 0x55bbcf58c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc82ffff640 1 -- 172.21.15.33:0/594290751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf58fbb0 msgr2=0x55bbcf58cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc82ffff640 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf58fbb0 0x55bbcf58cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc82ffff640 1 -- 172.21.15.33:0/594290751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbcf25c2b0 con 0x55bbcf5849d0 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.493+0000 7fc827fff640 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5852f0 0x55bbcf58c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.494+0000 7fc82ffff640 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 0x55bbcf58f3a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc820010ee0 tx=0x7fc82000ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92246707df4141c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.494+0000 7fc80f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.494+0000 7fc80f7fe640 1 -- 172.21.15.33:0/594290751 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82000e780 con 0x55bbcf5849d0 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.494+0000 7fc80f7fe640 1 -- 172.21.15.33:0/594290751 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fc820004050 con 0x55bbcf5849d0 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.494+0000 7fc838681740 1 -- 172.21.15.33:0/594290751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bbcf4f8600 con 0x55bbcf5849d0 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.494+0000 7fc80f7fe640 1 -- 172.21.15.33:0/594290751 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc820004250 con 0x55bbcf5849d0 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.494+0000 7fc838681740 10 client.15105 fetch_fsmap learned FSMap version 25 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.494+0000 7fc838681740 1 -- 172.21.15.33:0/594290751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bbcf59dff0 con 0x55bbcf5849d0 2024-09-16T00:17:01.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.494+0000 7fc80f7fe640 1 -- 172.21.15.33:0/594290751 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bbcf4f8600 con 0x55bbcf5849d0 2024-09-16T00:17:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.494+0000 7fc80f7fe640 1 -- 172.21.15.33:0/594290751 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fc820011820 con 0x55bbcf5849d0 2024-09-16T00:17:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.494+0000 7fc838681740 10 client.15105 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T00:17:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.495+0000 7fc838681740 10 client.15105 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T00:17:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.495+0000 7fc838681740 20 client.15105 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.495+0000 7fc838681740 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bbcf5a0450 0x55bbcf5a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.495+0000 7fc838681740 4 client.15105 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:17:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.495+0000 7fc838681740 1 -- 172.21.15.33:0/594290751 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbcf4f2b10 con 0x55bbcf5a0450 2024-09-16T00:17:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.495+0000 7fc827fff640 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bbcf5a0450 0x55bbcf5a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.495+0000 7fc827fff640 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bbcf5a0450 0x55bbcf5a18c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bd3d224cfc5c705 server_cookie=8902b9ebfff01c0 in_seq=0 out_seq=0 2024-09-16T00:17:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.495+0000 7fc80f7fe640 10 client.15105 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.496+0000 7fc80f7fe640 1 -- 172.21.15.33:0/594290751 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bbcf4f2b10 con 0x55bbcf5a0450 2024-09-16T00:17:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.496+0000 7fc80f7fe640 10 client.15105 handle_command_reply: tid=0 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.496+0000 7fc838681740 1 client.15105 shutdown 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.496+0000 7fc838681740 1 -- 172.21.15.33:0/594290751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 msgr2=0x55bbcf58f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.496+0000 7fc838681740 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 0x55bbcf58f3a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc820010ee0 tx=0x7fc82000ef30 comp rx=0 tx=0).stop 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.496+0000 7fc838681740 1 -- 172.21.15.33:0/594290751 shutdown_connections 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.496+0000 7fc838681740 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bbcf5a0450 0x55bbcf5a18c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.496+0000 7fc838681740 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf58fbb0 0x55bbcf58cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.496+0000 7fc838681740 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5852f0 0x55bbcf58c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.496+0000 7fc838681740 1 --2- 172.21.15.33:0/594290751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 0x55bbcf58f3a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.496+0000 7fc838681740 1 -- 172.21.15.33:0/594290751 >> 172.21.15.33:0/594290751 conn(0x55bbcf580f60 msgr2=0x55bbcf586c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.496+0000 7fc80f7fe640 0 client.15105 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.497+0000 7fc838681740 1 -- 172.21.15.33:0/594290751 shutdown_connections 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.497+0000 7fc838681740 1 -- 172.21.15.33:0/594290751 wait complete. 2024-09-16T00:17:01.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.497+0000 7fc838681740 20 client.15105 trim_cache size 0 max 16384 2024-09-16T00:17:01.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.514+0000 7fc838681740 1 Processor -- start 2024-09-16T00:17:01.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.514+0000 7fc838681740 1 -- start start 2024-09-16T00:17:01.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc838681740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 0x55bbcf59fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc838681740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5852f0 0x55bbcf5a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc838681740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf58fbb0 0x55bbcf587de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc838681740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bbcf59fc00 con 0x55bbcf5849d0 2024-09-16T00:17:01.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc838681740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55bbcf5a02d0 con 0x55bbcf58fbb0 2024-09-16T00:17:01.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc838681740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bbcf5a09a0 con 0x55bbcf5852f0 2024-09-16T00:17:01.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc82ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 0x55bbcf59fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc82ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 0x55bbcf59fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55146/0 (socket says 172.21.15.33:55146) 2024-09-16T00:17:01.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc82ffff640 1 -- 172.21.15.33:0/3979826945 learned_addr learned my addr 172.21.15.33:0/3979826945 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:01.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc827fff640 1 --2- 172.21.15.33:0/3979826945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5852f0 0x55bbcf5a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc834845640 1 --2- 172.21.15.33:0/3979826945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf58fbb0 0x55bbcf587de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc82ffff640 1 -- 172.21.15.33:0/3979826945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5852f0 msgr2=0x55bbcf5a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:01.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc82ffff640 1 --2- 172.21.15.33:0/3979826945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5852f0 0x55bbcf5a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc82ffff640 1 -- 172.21.15.33:0/3979826945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf58fbb0 msgr2=0x55bbcf587de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:01.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc82ffff640 1 --2- 172.21.15.33:0/3979826945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf58fbb0 0x55bbcf587de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.515+0000 7fc82ffff640 1 -- 172.21.15.33:0/3979826945 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbcf580d90 con 0x55bbcf5849d0 2024-09-16T00:17:01.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.516+0000 7fc82ffff640 1 --2- 172.21.15.33:0/3979826945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 0x55bbcf59fd80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc82000bea0 tx=0x7fc820001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787516023c1d6807 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:01.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.516+0000 7fc825ffb640 1 -- 172.21.15.33:0/3979826945 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82000e780 con 0x55bbcf5849d0 2024-09-16T00:17:01.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.516+0000 7fc825ffb640 1 -- 172.21.15.33:0/3979826945 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc82000e920 con 0x55bbcf5849d0 2024-09-16T00:17:01.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.516+0000 7fc825ffb640 1 -- 172.21.15.33:0/3979826945 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82001b690 con 0x55bbcf5849d0 2024-09-16T00:17:01.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.517+0000 7fc838681740 1 -- 172.21.15.33:0/3979826945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 msgr2=0x55bbcf59fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:01.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.517+0000 7fc838681740 1 --2- 172.21.15.33:0/3979826945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 0x55bbcf59fd80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc82000bea0 tx=0x7fc820001e50 comp rx=0 tx=0).stop 2024-09-16T00:17:01.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.517+0000 7fc838681740 1 -- 172.21.15.33:0/3979826945 shutdown_connections 2024-09-16T00:17:01.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.517+0000 7fc838681740 1 --2- 172.21.15.33:0/3979826945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf58fbb0 0x55bbcf587de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.517+0000 7fc838681740 1 --2- 172.21.15.33:0/3979826945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf5852f0 0x55bbcf5a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.517+0000 7fc838681740 1 --2- 172.21.15.33:0/3979826945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf5849d0 0x55bbcf59fd80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.517+0000 7fc838681740 1 -- 172.21.15.33:0/3979826945 >> 172.21.15.33:0/3979826945 conn(0x55bbcf580f60 msgr2=0x55bbcf59f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:01.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.517+0000 7fc838681740 1 -- 172.21.15.33:0/3979826945 shutdown_connections 2024-09-16T00:17:01.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.517+0000 7fc838681740 1 -- 172.21.15.33:0/3979826945 wait complete. 2024-09-16T00:17:01.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.517+0000 7fc838681740 1 -- start start 2024-09-16T00:17:01.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.517+0000 7fc838681740 1 Processor -- start 2024-09-16T00:17:01.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc838681740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf5852f0 0x55bbcf5889c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc838681740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf58fbb0 0x55bbcf588f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc838681740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf589440 0x55bbcf5a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc838681740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bbcf5a1740 con 0x55bbcf589440 2024-09-16T00:17:01.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc838681740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55bbcf5a1070 con 0x55bbcf5852f0 2024-09-16T00:17:01.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc838681740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bbcf5a1e10 con 0x55bbcf58fbb0 2024-09-16T00:17:01.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc838681740 10 client.0 _ll_register_callbacks cb 0x7fc8180d2d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:01.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc834845640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf589440 0x55bbcf5a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc834845640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf589440 0x55bbcf5a8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55154/0 (socket says 172.21.15.33:55154) 2024-09-16T00:17:01.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc834845640 1 -- 172.21.15.33:0/734197477 learned_addr learned my addr 172.21.15.33:0/734197477 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:01.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc82ffff640 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf5852f0 0x55bbcf5889c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc827fff640 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf58fbb0 0x55bbcf588f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc834845640 1 -- 172.21.15.33:0/734197477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf58fbb0 msgr2=0x55bbcf588f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:01.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc834845640 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf58fbb0 0x55bbcf588f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc834845640 1 -- 172.21.15.33:0/734197477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf5852f0 msgr2=0x55bbcf5889c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc834845640 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf5852f0 0x55bbcf5889c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.518+0000 7fc834845640 1 -- 172.21.15.33:0/734197477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbcf5a90c0 con 0x55bbcf589440 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.519+0000 7fc834845640 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf589440 0x55bbcf5a8b80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc828035000 tx=0x7fc8280095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c26be3d368d3747 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.519+0000 7fc80f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.519+0000 7fc80f7fe640 1 -- 172.21.15.33:0/734197477 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc828003970 con 0x55bbcf589440 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.519+0000 7fc80f7fe640 1 -- 172.21.15.33:0/734197477 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fc828003b10 con 0x55bbcf589440 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.519+0000 7fc838681740 1 -- 172.21.15.33:0/734197477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc82000e920 con 0x55bbcf589440 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.519+0000 7fc80f7fe640 1 -- 172.21.15.33:0/734197477 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc828003d10 con 0x55bbcf589440 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.519+0000 7fc838681740 10 client.15111 fetch_fsmap learned FSMap version 25 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.519+0000 7fc838681740 1 -- 172.21.15.33:0/734197477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bbcf5aa250 con 0x55bbcf589440 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.519+0000 7fc80f7fe640 1 -- 172.21.15.33:0/734197477 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc828036040 con 0x55bbcf589440 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.520+0000 7fc80f7fe640 1 -- 172.21.15.33:0/734197477 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fc82804f070 con 0x55bbcf589440 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.520+0000 7fc838681740 10 client.15111 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T00:17:01.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.520+0000 7fc838681740 10 client.15111 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T00:17:01.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.520+0000 7fc838681740 20 client.15111 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:01.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.520+0000 7fc838681740 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bbcf613430 0x55bbcf612140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:01.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.520+0000 7fc838681740 4 client.15111 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T00:17:01.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.520+0000 7fc838681740 1 -- 172.21.15.33:0/734197477 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "status"}) -- 0x7fc82000e920 con 0x55bbcf613430 2024-09-16T00:17:01.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.520+0000 7fc82ffff640 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bbcf613430 0x55bbcf612140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:01.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.520+0000 7fc80f7fe640 10 client.15111 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:01.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.520+0000 7fc82ffff640 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bbcf613430 0x55bbcf612140 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4313b9c07bf793d server_cookie=b4a5382743813868 in_seq=0 out_seq=0 2024-09-16T00:17:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc80f7fe640 1 -- 172.21.15.33:0/734197477 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x7fc82000e920 con 0x55bbcf613430 2024-09-16T00:17:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc80f7fe640 10 client.15111 handle_command_reply: tid=0 2024-09-16T00:17:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc838681740 1 client.15111 shutdown 2024-09-16T00:17:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc838681740 1 -- 172.21.15.33:0/734197477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf589440 msgr2=0x55bbcf5a8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc838681740 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf589440 0x55bbcf5a8b80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc828035000 tx=0x7fc8280095c0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc838681740 1 -- 172.21.15.33:0/734197477 shutdown_connections 2024-09-16T00:17:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc838681740 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bbcf613430 0x55bbcf612140 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc838681740 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbcf589440 0x55bbcf5a8b80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc838681740 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bbcf58fbb0 0x55bbcf588f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc838681740 1 --2- 172.21.15.33:0/734197477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bbcf5852f0 0x55bbcf5889c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc838681740 1 -- 172.21.15.33:0/734197477 >> 172.21.15.33:0/734197477 conn(0x55bbcf580f60 msgr2=0x55bbcf587d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc80f7fe640 0 client.15111 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc838681740 1 -- 172.21.15.33:0/734197477 shutdown_connections 2024-09-16T00:17:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc838681740 1 -- 172.21.15.33:0/734197477 wait complete. 2024-09-16T00:17:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.521+0000 7fc838681740 20 client.15111 trim_cache size 0 max 16384 2024-09-16T00:17:01.527 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:17:01.528 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:17:01.528 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 0, 2024-09-16T00:17:01.528 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24494, 2024-09-16T00:17:01.528 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-09-16T00:17:01.528 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-09-16T00:17:01.528 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-09-16T00:17:01.528 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 113.993137996, 2024-09-16T00:17:01.528 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 25, 2024-09-16T00:17:01.528 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 88, 2024-09-16T00:17:01.528 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 86, 2024-09-16T00:17:01.528 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 125.07091860200001 2024-09-16T00:17:01.528 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:17:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.522+0000 7fc8362a8640 1 -- 172.21.15.33:0/3441069491 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc81c0a31b0 msgr2=0x7fc81c0a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.522+0000 7fc8362a8640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc81c0a31b0 0x7fc81c0a5670 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc828005cc0 tx=0x7fc828005c30 comp rx=0 tx=0).stop 2024-09-16T00:17:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.522+0000 7fc8362a8640 1 -- 172.21.15.33:0/3441069491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc830103760 msgr2=0x7fc830198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.522+0000 7fc8362a8640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc830103760 0x7fc830198ab0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc83006a2f0 tx=0x7fc81800dbb0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.522+0000 7fc8362a8640 1 -- 172.21.15.33:0/3441069491 shutdown_connections 2024-09-16T00:17:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.522+0000 7fc8362a8640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc81c0a31b0 0x7fc81c0a5670 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.522+0000 7fc8362a8640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8301082e0 0x7fc83019ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.522+0000 7fc8362a8640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc830103760 0x7fc830198ab0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.522+0000 7fc8362a8640 1 --2- 172.21.15.33:0/3441069491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc830102db0 0x7fc830198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.522+0000 7fc8362a8640 1 -- 172.21.15.33:0/3441069491 >> 172.21.15.33:0/3441069491 conn(0x7fc8300fc7d0 msgr2=0x7fc830105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.522+0000 7fc8362a8640 1 -- 172.21.15.33:0/3441069491 shutdown_connections 2024-09-16T00:17:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:01.522+0000 7fc8362a8640 1 -- 172.21.15.33:0/3441069491 wait complete. 2024-09-16T00:17:01.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:01 smithi033 ceph-mon[31243]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:17:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:01 smithi083 ceph-mon[35242]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:17:01.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:01 smithi060 ceph-mon[35185]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:17:02.366 INFO:tasks.cephadm:Setup complete, yielding 2024-09-16T00:17:02.366 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-16T00:17:02.378 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-09-16T00:17:02.378 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- bash -c 'ceph orch status' 2024-09-16T00:17:02.637 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:03.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.107+0000 7fc278bef640 1 -- 172.21.15.33:0/3442087149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc274102e00 msgr2=0x7fc274103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:03.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.107+0000 7fc278bef640 1 --2- 172.21.15.33:0/3442087149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc274102e00 0x7fc274103280 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc268009900 tx=0x7fc26802a410 comp rx=0 tx=0).stop 2024-09-16T00:17:03.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.108+0000 7fc278bef640 1 -- 172.21.15.33:0/3442087149 shutdown_connections 2024-09-16T00:17:03.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.108+0000 7fc278bef640 1 --2- 172.21.15.33:0/3442087149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2741037c0 0x7fc27410a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.108+0000 7fc278bef640 1 --2- 172.21.15.33:0/3442087149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc274102e00 0x7fc274103280 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.108+0000 7fc278bef640 1 --2- 172.21.15.33:0/3442087149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc274101c00 0x7fc274102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.108+0000 7fc278bef640 1 -- 172.21.15.33:0/3442087149 >> 172.21.15.33:0/3442087149 conn(0x7fc2740fd390 msgr2=0x7fc2740ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.108+0000 7fc278bef640 1 -- 172.21.15.33:0/3442087149 shutdown_connections 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.108+0000 7fc278bef640 1 -- 172.21.15.33:0/3442087149 wait complete. 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.108+0000 7fc278bef640 1 Processor -- start 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.108+0000 7fc278bef640 1 -- start start 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc278bef640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc274101c00 0x7fc274197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc278bef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc274102e00 0x7fc274198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc278bef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2741037c0 0x7fc27419f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc278bef640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc27410d860 con 0x7fc274101c00 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc278bef640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc27410d6e0 con 0x7fc2741037c0 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc278bef640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc27410d9e0 con 0x7fc274102e00 2024-09-16T00:17:03.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc27259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc274101c00 0x7fc274197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc27259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc274101c00 0x7fc274197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55168/0 (socket says 172.21.15.33:55168) 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc27259c640 1 -- 172.21.15.33:0/1235423787 learned_addr learned my addr 172.21.15.33:0/1235423787 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc272d9d640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2741037c0 0x7fc27419f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc271d9b640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc274102e00 0x7fc274198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc27259c640 1 -- 172.21.15.33:0/1235423787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc274102e00 msgr2=0x7fc274198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc27259c640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc274102e00 0x7fc274198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc27259c640 1 -- 172.21.15.33:0/1235423787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2741037c0 msgr2=0x7fc27419f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc27259c640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2741037c0 0x7fc27419f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc27259c640 1 -- 172.21.15.33:0/1235423787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27419fb10 con 0x7fc274101c00 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc27259c640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc274101c00 0x7fc274197ee0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc26400e9c0 tx=0x7fc26400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.109+0000 7fc2637fe640 1 -- 172.21.15.33:0/1235423787 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26400cde0 con 0x7fc274101c00 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.110+0000 7fc2637fe640 1 -- 172.21.15.33:0/1235423787 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc264010910 con 0x7fc274101c00 2024-09-16T00:17:03.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.110+0000 7fc278bef640 1 -- 172.21.15.33:0/1235423787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc27419fe00 con 0x7fc274101c00 2024-09-16T00:17:03.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.110+0000 7fc2637fe640 1 -- 172.21.15.33:0/1235423787 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc264018a00 con 0x7fc274101c00 2024-09-16T00:17:03.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.110+0000 7fc278bef640 1 -- 172.21.15.33:0/1235423787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2741a0250 con 0x7fc274101c00 2024-09-16T00:17:03.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.111+0000 7fc278bef640 1 -- 172.21.15.33:0/1235423787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2741090a0 con 0x7fc274101c00 2024-09-16T00:17:03.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.112+0000 7fc2637fe640 1 -- 172.21.15.33:0/1235423787 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc26401f020 con 0x7fc274101c00 2024-09-16T00:17:03.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.113+0000 7fc2637fe640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc25c0a3300 0x7fc25c0a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:03.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.113+0000 7fc271d9b640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc25c0a3300 0x7fc25c0a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:03.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.113+0000 7fc2637fe640 1 -- 172.21.15.33:0/1235423787 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fc2640d0050 con 0x7fc274101c00 2024-09-16T00:17:03.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.113+0000 7fc271d9b640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc25c0a3300 0x7fc25c0a57c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc268000c00 tx=0x7fc2680315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:03.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.117+0000 7fc2637fe640 1 -- 172.21.15.33:0/1235423787 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc26409c8b0 con 0x7fc274101c00 2024-09-16T00:17:03.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.307+0000 7fc278bef640 1 -- 172.21.15.33:0/1235423787 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fc274062630 con 0x7fc25c0a3300 2024-09-16T00:17:03.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.308+0000 7fc2637fe640 1 -- 172.21.15.33:0/1235423787 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fc274062630 con 0x7fc25c0a3300 2024-09-16T00:17:03.310 INFO:teuthology.orchestra.run.smithi033.stdout:Backend: cephadm 2024-09-16T00:17:03.310 INFO:teuthology.orchestra.run.smithi033.stdout:Available: Yes 2024-09-16T00:17:03.310 INFO:teuthology.orchestra.run.smithi033.stdout:Paused: No 2024-09-16T00:17:03.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.310+0000 7fc278bef640 1 -- 172.21.15.33:0/1235423787 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc25c0a3300 msgr2=0x7fc25c0a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.310+0000 7fc278bef640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc25c0a3300 0x7fc25c0a57c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc268000c00 tx=0x7fc2680315d0 comp rx=0 tx=0).stop 2024-09-16T00:17:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.310+0000 7fc278bef640 1 -- 172.21.15.33:0/1235423787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc274101c00 msgr2=0x7fc274197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.310+0000 7fc278bef640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc274101c00 0x7fc274197ee0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc26400e9c0 tx=0x7fc26400ee80 comp rx=0 tx=0).stop 2024-09-16T00:17:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.310+0000 7fc278bef640 1 -- 172.21.15.33:0/1235423787 shutdown_connections 2024-09-16T00:17:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.310+0000 7fc278bef640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc25c0a3300 0x7fc25c0a57c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.310+0000 7fc278bef640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2741037c0 0x7fc27419f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.310+0000 7fc278bef640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc274102e00 0x7fc274198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.310+0000 7fc278bef640 1 --2- 172.21.15.33:0/1235423787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc274101c00 0x7fc274197ee0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.310+0000 7fc278bef640 1 -- 172.21.15.33:0/1235423787 >> 172.21.15.33:0/1235423787 conn(0x7fc2740fd390 msgr2=0x7fc274106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:03.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.311+0000 7fc278bef640 1 -- 172.21.15.33:0/1235423787 shutdown_connections 2024-09-16T00:17:03.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:03.311+0000 7fc278bef640 1 -- 172.21.15.33:0/1235423787 wait complete. 2024-09-16T00:17:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:03 smithi033 ceph-mon[31243]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:17:03.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:03 smithi083 ceph-mon[35242]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:17:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:03 smithi060 ceph-mon[35185]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:17:04.157 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ps' 2024-09-16T00:17:04.420 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:04.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:04 smithi033 ceph-mon[31243]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:04.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:04 smithi083 ceph-mon[35242]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:04.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.925+0000 7f312b356640 1 -- 172.21.15.33:0/650385460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3124101bf0 msgr2=0x7f3124101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:04.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.925+0000 7f312b356640 1 --2- 172.21.15.33:0/650385460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3124101bf0 0x7f3124101ff0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3120009900 tx=0x7f312002a410 comp rx=0 tx=0).stop 2024-09-16T00:17:04.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.925+0000 7f312b356640 1 -- 172.21.15.33:0/650385460 shutdown_connections 2024-09-16T00:17:04.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.925+0000 7f312b356640 1 --2- 172.21.15.33:0/650385460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31241037b0 0x7f312410a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:04.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.925+0000 7f312b356640 1 --2- 172.21.15.33:0/650385460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3124102df0 0x7f3124103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:04.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.925+0000 7f312b356640 1 --2- 172.21.15.33:0/650385460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3124101bf0 0x7f3124101ff0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:04.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.925+0000 7f312b356640 1 -- 172.21.15.33:0/650385460 >> 172.21.15.33:0/650385460 conn(0x7f31240fd3c0 msgr2=0x7f31240ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:04.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.925+0000 7f312b356640 1 -- 172.21.15.33:0/650385460 shutdown_connections 2024-09-16T00:17:04.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.925+0000 7f312b356640 1 -- 172.21.15.33:0/650385460 wait complete. 2024-09-16T00:17:04.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.925+0000 7f312b356640 1 Processor -- start 2024-09-16T00:17:04.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f312b356640 1 -- start start 2024-09-16T00:17:04.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f312b356640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3124101bf0 0x7f3124197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:04.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f312b356640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3124102df0 0x7f31241983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:04.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f312b356640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31241037b0 0x7f312419f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:04.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f312b356640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f312410d870 con 0x7f3124101bf0 2024-09-16T00:17:04.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f312b356640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f312410d6f0 con 0x7f31241037b0 2024-09-16T00:17:04.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f312b356640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f312410d9f0 con 0x7f3124102df0 2024-09-16T00:17:04.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f31290f2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3124101bf0 0x7f3124197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:04.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f31290f2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3124101bf0 0x7f3124197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55186/0 (socket says 172.21.15.33:55186) 2024-09-16T00:17:04.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f31290f2640 1 -- 172.21.15.33:0/496918502 learned_addr learned my addr 172.21.15.33:0/496918502 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:04.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f31288f1640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3124102df0 0x7f31241983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:04.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f31298f3640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31241037b0 0x7f312419f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:04.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f31290f2640 1 -- 172.21.15.33:0/496918502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3124102df0 msgr2=0x7f31241983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:04.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f31290f2640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3124102df0 0x7f31241983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:04.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f31290f2640 1 -- 172.21.15.33:0/496918502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31241037b0 msgr2=0x7f312419f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:04.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f31290f2640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31241037b0 0x7f312419f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:04.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f31290f2640 1 -- 172.21.15.33:0/496918502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312419faa0 con 0x7f3124101bf0 2024-09-16T00:17:04.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.926+0000 7f31290f2640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3124101bf0 0x7f3124197e70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3120031610 tx=0x7f31200043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:04.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.927+0000 7f311a7fc640 1 -- 172.21.15.33:0/496918502 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312003f030 con 0x7f3124101bf0 2024-09-16T00:17:04.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.927+0000 7f311a7fc640 1 -- 172.21.15.33:0/496918502 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3120003ce0 con 0x7f3124101bf0 2024-09-16T00:17:04.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.927+0000 7f312b356640 1 -- 172.21.15.33:0/496918502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f312419fd30 con 0x7f3124101bf0 2024-09-16T00:17:04.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.927+0000 7f311a7fc640 1 -- 172.21.15.33:0/496918502 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120035070 con 0x7f3124101bf0 2024-09-16T00:17:04.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.927+0000 7f312b356640 1 -- 172.21.15.33:0/496918502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31241a01c0 con 0x7f3124101bf0 2024-09-16T00:17:04.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.928+0000 7f312b356640 1 -- 172.21.15.33:0/496918502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3124108fa0 con 0x7f3124101bf0 2024-09-16T00:17:04.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.929+0000 7f311a7fc640 1 -- 172.21.15.33:0/496918502 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f3120042030 con 0x7f3124101bf0 2024-09-16T00:17:04.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.930+0000 7f311a7fc640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f31100a3300 0x7f31100a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:04.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.930+0000 7f31288f1640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f31100a3300 0x7f31100a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:04.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.930+0000 7f311a7fc640 1 -- 172.21.15.33:0/496918502 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f31200f00b0 con 0x7f3124101bf0 2024-09-16T00:17:04.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.930+0000 7f31288f1640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f31100a3300 0x7f31100a57c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3124199390 tx=0x7f311400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:04.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:04.934+0000 7f311a7fc640 1 -- 172.21.15.33:0/496918502 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f31200bc8b0 con 0x7f3124101bf0 2024-09-16T00:17:04.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:04 smithi060 ceph-mon[35185]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:05.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.120+0000 7f312b356640 1 -- 172.21.15.33:0/496918502 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f3124062630 con 0x7f31100a3300 2024-09-16T00:17:05.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.132+0000 7f311a7fc640 1 -- 172.21.15.33:0/496918502 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4061 (secure 0 0 0) 0x7f3124062630 con 0x7f31100a3300 2024-09-16T00:17:05.134 INFO:teuthology.orchestra.run.smithi033.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-16T00:17:05.134 INFO:teuthology.orchestra.run.smithi033.stdout:mds.a smithi033 running (2m) 59s ago 2m 16.7M - 19.1.1-326-g8921c8ef f1455bd0219c 03ad602848ca 2024-09-16T00:17:05.134 INFO:teuthology.orchestra.run.smithi033.stdout:mds.b smithi060 running (2m) 60s ago 2m 28.8M - 19.1.1-326-g8921c8ef f1455bd0219c 42c946fdaf39 2024-09-16T00:17:05.134 INFO:teuthology.orchestra.run.smithi033.stdout:mds.c smithi083 running (2m) 60s ago 2m 18.7M - 19.1.1-326-g8921c8ef f1455bd0219c 56133162cff8 2024-09-16T00:17:05.134 INFO:teuthology.orchestra.run.smithi033.stdout:mds.d smithi033 running (2m) 59s ago 2m 14.5M - 19.1.1-326-g8921c8ef f1455bd0219c 7ed5f1632dbd 2024-09-16T00:17:05.134 INFO:teuthology.orchestra.run.smithi033.stdout:mds.e smithi060 running (2m) 60s ago 2m 15.3M - 19.1.1-326-g8921c8ef f1455bd0219c 34c56ec6ab56 2024-09-16T00:17:05.134 INFO:teuthology.orchestra.run.smithi033.stdout:mds.f smithi083 running (2m) 60s ago 2m 18.7M - 19.1.1-326-g8921c8ef f1455bd0219c 2bb32a434d0a 2024-09-16T00:17:05.134 INFO:teuthology.orchestra.run.smithi033.stdout:mds.g smithi033 running (2m) 59s ago 2m 13.5M - 19.1.1-326-g8921c8ef f1455bd0219c 093a62828f9a 2024-09-16T00:17:05.134 INFO:teuthology.orchestra.run.smithi033.stdout:mds.h smithi060 running (2m) 60s ago 2m 15.6M - 19.1.1-326-g8921c8ef f1455bd0219c 4b6511ad6bcd 2024-09-16T00:17:05.134 INFO:teuthology.orchestra.run.smithi033.stdout:mds.i smithi083 running (2m) 60s ago 2m 17.2M - 19.1.1-326-g8921c8ef f1455bd0219c 8b79c590acfa 2024-09-16T00:17:05.134 INFO:teuthology.orchestra.run.smithi033.stdout:mds.j smithi033 running (2m) 59s ago 2m 14.3M - 19.1.1-326-g8921c8ef f1455bd0219c fdf1e0d05507 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:mds.k smithi060 running (2m) 60s ago 2m 16.0M - 19.1.1-326-g8921c8ef f1455bd0219c 30c350dd320b 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:mds.l smithi083 running (2m) 60s ago 2m 14.5M - 19.1.1-326-g8921c8ef f1455bd0219c cc64a2a63a5d 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:mgr.x smithi033 *:9283,8765 running (9m) 59s ago 9m 507M - 19.1.1-326-g8921c8ef f1455bd0219c e964b53ad2d0 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:mgr.y smithi060 *:8443,8765 running (7m) 60s ago 7m 443M - 19.1.1-326-g8921c8ef f1455bd0219c 9a7609fbd20b 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:mgr.z smithi083 *:8443,8765 running (8m) 60s ago 8m 444M - 19.1.1-326-g8921c8ef f1455bd0219c 44d46e2f0d9d 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:mon.a smithi033 running (9m) 59s ago 9m 48.9M 2048M 19.1.1-326-g8921c8ef f1455bd0219c 831049806ec5 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:mon.b smithi060 running (8m) 60s ago 8m 41.2M 2048M 19.1.1-326-g8921c8ef f1455bd0219c d741ec7069ef 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:mon.c smithi083 running (8m) 60s ago 8m 41.0M 2048M 19.1.1-326-g8921c8ef f1455bd0219c f13645e97635 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0 smithi033 running (7m) 59s ago 7m 57.2M 4273M 19.1.1-326-g8921c8ef f1455bd0219c 662c4a8efa4c 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1 smithi060 running (7m) 60s ago 7m 56.7M 4273M 19.1.1-326-g8921c8ef f1455bd0219c e8867ba4f9d0 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2 smithi083 running (6m) 60s ago 6m 60.5M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 4eebd2012a21 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3 smithi033 running (6m) 59s ago 6m 44.2M 4273M 19.1.1-326-g8921c8ef f1455bd0219c d57387010c76 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4 smithi060 running (6m) 60s ago 6m 54.2M 4273M 19.1.1-326-g8921c8ef f1455bd0219c 9706d3b46d65 2024-09-16T00:17:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5 smithi083 running (5m) 60s ago 5m 43.9M 4272M 19.1.1-326-g8921c8ef f1455bd0219c f8ef70f296b8 2024-09-16T00:17:05.136 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6 smithi033 running (5m) 59s ago 5m 45.7M 4273M 19.1.1-326-g8921c8ef f1455bd0219c 19ddb956e0c7 2024-09-16T00:17:05.136 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7 smithi060 running (4m) 60s ago 4m 55.2M 4273M 19.1.1-326-g8921c8ef f1455bd0219c 6ab9d8a02085 2024-09-16T00:17:05.136 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8 smithi083 running (4m) 60s ago 4m 42.8M 4272M 19.1.1-326-g8921c8ef f1455bd0219c c6b122500d37 2024-09-16T00:17:05.136 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9 smithi033 running (3m) 59s ago 3m 53.8M 4273M 19.1.1-326-g8921c8ef f1455bd0219c 181e2246b1c0 2024-09-16T00:17:05.136 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10 smithi060 running (3m) 60s ago 3m 55.4M 4273M 19.1.1-326-g8921c8ef f1455bd0219c c6601bbb2bab 2024-09-16T00:17:05.136 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11 smithi083 running (2m) 60s ago 2m 40.4M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 49e47c89fb97 2024-09-16T00:17:05.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.134+0000 7f312b356640 1 -- 172.21.15.33:0/496918502 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f31100a3300 msgr2=0x7f31100a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:05.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.134+0000 7f312b356640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f31100a3300 0x7f31100a57c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3124199390 tx=0x7f311400b390 comp rx=0 tx=0).stop 2024-09-16T00:17:05.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.134+0000 7f312b356640 1 -- 172.21.15.33:0/496918502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3124101bf0 msgr2=0x7f3124197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:05.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.134+0000 7f312b356640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3124101bf0 0x7f3124197e70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3120031610 tx=0x7f31200043b0 comp rx=0 tx=0).stop 2024-09-16T00:17:05.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.134+0000 7f312b356640 1 -- 172.21.15.33:0/496918502 shutdown_connections 2024-09-16T00:17:05.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.134+0000 7f312b356640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f31100a3300 0x7f31100a57c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:05.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.134+0000 7f312b356640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31241037b0 0x7f312419f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:05.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.134+0000 7f312b356640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3124102df0 0x7f31241983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:05.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.134+0000 7f312b356640 1 --2- 172.21.15.33:0/496918502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3124101bf0 0x7f3124197e70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:05.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.134+0000 7f312b356640 1 -- 172.21.15.33:0/496918502 >> 172.21.15.33:0/496918502 conn(0x7f31240fd3c0 msgr2=0x7f3124106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:05.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.134+0000 7f312b356640 1 -- 172.21.15.33:0/496918502 shutdown_connections 2024-09-16T00:17:05.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:05.134+0000 7f312b356640 1 -- 172.21.15.33:0/496918502 wait complete. 2024-09-16T00:17:05.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:05 smithi033 ceph-mon[31243]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:17:05.900 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ls' 2024-09-16T00:17:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:05 smithi083 ceph-mon[35242]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:17:05.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:05 smithi060 ceph-mon[35185]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:17:06.165 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:06.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.628+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/1628412286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ac01037c0 msgr2=0x7f1ac010a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:06.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.628+0000 7f1ac4ef7640 1 --2- 172.21.15.33:0/1628412286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ac01037c0 0x7f1ac010a050 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009ca0 tx=0x7f1ab802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:17:06.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.628+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/1628412286 shutdown_connections 2024-09-16T00:17:06.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.628+0000 7f1ac4ef7640 1 --2- 172.21.15.33:0/1628412286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ac01037c0 0x7f1ac010a050 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:06.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.628+0000 7f1ac4ef7640 1 --2- 172.21.15.33:0/1628412286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ac0102e00 0x7f1ac0103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:06.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.628+0000 7f1ac4ef7640 1 --2- 172.21.15.33:0/1628412286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ac0101c00 0x7f1ac0102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:06.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.628+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/1628412286 >> 172.21.15.33:0/1628412286 conn(0x7f1ac00fd390 msgr2=0x7f1ac00ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:06.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.628+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/1628412286 shutdown_connections 2024-09-16T00:17:06.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.628+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/1628412286 wait complete. 2024-09-16T00:17:06.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.628+0000 7f1ac4ef7640 1 Processor -- start 2024-09-16T00:17:06.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1ac4ef7640 1 -- start start 2024-09-16T00:17:06.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1ac4ef7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ac0101c00 0x7f1ac0075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:06.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1ac4ef7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ac0102e00 0x7f1ac00745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:06.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1ac4ef7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ac01037c0 0x7f1ac0074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:06.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1ac4ef7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac010d980 con 0x7f1ac0101c00 2024-09-16T00:17:06.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1ac4ef7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac010d800 con 0x7f1ac01037c0 2024-09-16T00:17:06.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1ac4ef7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac010db00 con 0x7f1ac0102e00 2024-09-16T00:17:06.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1abe59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ac0101c00 0x7f1ac0075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:06.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1abe59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ac0101c00 0x7f1ac0075f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55208/0 (socket says 172.21.15.33:55208) 2024-09-16T00:17:06.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1abe59c640 1 -- 172.21.15.33:0/3785078721 learned_addr learned my addr 172.21.15.33:0/3785078721 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:06.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1abed9d640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ac01037c0 0x7f1ac0074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:06.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1abdd9b640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ac0102e00 0x7f1ac00745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:06.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1abe59c640 1 -- 172.21.15.33:0/3785078721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ac0102e00 msgr2=0x7f1ac00745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:06.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1abe59c640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ac0102e00 0x7f1ac00745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:06.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1abe59c640 1 -- 172.21.15.33:0/3785078721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ac01037c0 msgr2=0x7f1ac0074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:06.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1abe59c640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ac01037c0 0x7f1ac0074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.629+0000 7f1abe59c640 1 -- 172.21.15.33:0/3785078721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac00753e0 con 0x7f1ac0101c00 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.630+0000 7f1abe59c640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ac0101c00 0x7f1ac0075f90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000b450 tx=0x7f1ab000b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.630+0000 7f1aaf7fe640 1 -- 172.21.15.33:0/3785078721 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0015020 con 0x7f1ac0101c00 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.630+0000 7f1aaf7fe640 1 -- 172.21.15.33:0/3785078721 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ab0011d10 con 0x7f1ac0101c00 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.630+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/3785078721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ac01a21c0 con 0x7f1ac0101c00 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.630+0000 7f1aaf7fe640 1 -- 172.21.15.33:0/3785078721 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab000f2e0 con 0x7f1ac0101c00 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.630+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/3785078721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ac01a2750 con 0x7f1ac0101c00 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.631+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/3785078721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ac0102080 con 0x7f1ac0101c00 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.632+0000 7f1aaf7fe640 1 -- 172.21.15.33:0/3785078721 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1ab0018020 con 0x7f1ac0101c00 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.633+0000 7f1aaf7fe640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1aa80a32d0 0x7f1aa80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.633+0000 7f1abdd9b640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1aa80a32d0 0x7f1aa80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.633+0000 7f1aaf7fe640 1 -- 172.21.15.33:0/3785078721 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f1ab00cfa90 con 0x7f1ac0101c00 2024-09-16T00:17:06.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.633+0000 7f1abdd9b640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1aa80a32d0 0x7f1aa80a5790 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0075d10 tx=0x7f1ab4009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:06.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.636+0000 7f1aaf7fe640 1 -- 172.21.15.33:0/3785078721 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1ab001f020 con 0x7f1ac0101c00 2024-09-16T00:17:06.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:06 smithi033 ceph-mon[31243]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:06.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.831+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/3785078721 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f1ac0062630 con 0x7f1aa80a32d0 2024-09-16T00:17:06.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.837+0000 7f1aaf7fe640 1 -- 172.21.15.33:0/3785078721 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f1ac0062630 con 0x7f1aa80a32d0 2024-09-16T00:17:06.838 INFO:teuthology.orchestra.run.smithi033.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-16T00:17:06.839 INFO:teuthology.orchestra.run.smithi033.stdout:mds.all 12/12 62s ago 2m smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi060=b;smithi060=e;smithi060=h;smithi060=k;smithi083=c;smithi083=f;smithi083=i;smithi083=l;count:12 2024-09-16T00:17:06.839 INFO:teuthology.orchestra.run.smithi033.stdout:mgr 3/3 62s ago 8m smithi033=x;smithi060=y;smithi083=z;count:3 2024-09-16T00:17:06.839 INFO:teuthology.orchestra.run.smithi033.stdout:mon 3/3 62s ago 8m smithi033:172.21.15.33=a;smithi060:172.21.15.60=b;smithi083:172.21.15.83=c;count:3 2024-09-16T00:17:06.839 INFO:teuthology.orchestra.run.smithi033.stdout:osd 12 62s ago - 2024-09-16T00:17:06.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.839+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/3785078721 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1aa80a32d0 msgr2=0x7f1aa80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:06.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.839+0000 7f1ac4ef7640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1aa80a32d0 0x7f1aa80a5790 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0075d10 tx=0x7f1ab4009110 comp rx=0 tx=0).stop 2024-09-16T00:17:06.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.839+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/3785078721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ac0101c00 msgr2=0x7f1ac0075f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:06.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.839+0000 7f1ac4ef7640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ac0101c00 0x7f1ac0075f90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000b450 tx=0x7f1ab000b850 comp rx=0 tx=0).stop 2024-09-16T00:17:06.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.839+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/3785078721 shutdown_connections 2024-09-16T00:17:06.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.839+0000 7f1ac4ef7640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1aa80a32d0 0x7f1aa80a5790 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:06.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.839+0000 7f1ac4ef7640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ac01037c0 0x7f1ac0074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:06.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.839+0000 7f1ac4ef7640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ac0102e00 0x7f1ac00745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:06.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.840+0000 7f1ac4ef7640 1 --2- 172.21.15.33:0/3785078721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ac0101c00 0x7f1ac0075f90 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:06.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.840+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/3785078721 >> 172.21.15.33:0/3785078721 conn(0x7f1ac00fd390 msgr2=0x7f1ac0106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:06.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.840+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/3785078721 shutdown_connections 2024-09-16T00:17:06.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:06.840+0000 7f1ac4ef7640 1 -- 172.21.15.33:0/3785078721 wait complete. 2024-09-16T00:17:06.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:06 smithi083 ceph-mon[35242]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:06.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:06 smithi060 ceph-mon[35185]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:07.570 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- bash -c 'ceph orch host ls' 2024-09-16T00:17:07.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:07 smithi033 ceph-mon[31243]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:17:07.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:07 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:17:07.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:07 smithi083 ceph-mon[35242]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:17:07.892 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:07 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:17:07.952 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:07 smithi060 ceph-mon[35185]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:17:07.952 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:07 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:17:08.500 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:08.814 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:08 smithi033 ceph-mon[31243]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:08.814 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-09-16T00:17:08.814 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-09-16T00:17:08.814 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-09-16T00:17:08.814 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 6]}]: dispatch 2024-09-16T00:17:08.814 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-09-16T00:17:08.814 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-09-16T00:17:08.814 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-09-16T00:17:08.815 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2e", "id": [1, 8]}]: dispatch 2024-09-16T00:17:08.815 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]: dispatch 2024-09-16T00:17:08.815 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:08 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-09-16T00:17:08.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:08 smithi083 ceph-mon[35242]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-09-16T00:17:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-09-16T00:17:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-09-16T00:17:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 6]}]: dispatch 2024-09-16T00:17:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-09-16T00:17:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-09-16T00:17:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-09-16T00:17:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2e", "id": [1, 8]}]: dispatch 2024-09-16T00:17:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]: dispatch 2024-09-16T00:17:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:08 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-09-16T00:17:08.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:08 smithi060 ceph-mon[35185]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:08.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-09-16T00:17:08.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-09-16T00:17:08.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-09-16T00:17:08.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 6]}]: dispatch 2024-09-16T00:17:08.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-09-16T00:17:08.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-09-16T00:17:08.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-09-16T00:17:08.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2e", "id": [1, 8]}]: dispatch 2024-09-16T00:17:08.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]: dispatch 2024-09-16T00:17:08.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:08 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-09-16T00:17:09.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:08.999+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/3455157482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d98102eb0 msgr2=0x7f9d98103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:09.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:08.999+0000 7f9d9e3f8640 1 --2- 172.21.15.33:0/3455157482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d98102eb0 0x7f9d98103330 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009930 tx=0x7f9d8c02a420 comp rx=0 tx=0).stop 2024-09-16T00:17:09.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:08.999+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/3455157482 shutdown_connections 2024-09-16T00:17:09.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:08.999+0000 7f9d9e3f8640 1 --2- 172.21.15.33:0/3455157482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d98103870 0x7f9d9810a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:09.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:08.999+0000 7f9d9e3f8640 1 --2- 172.21.15.33:0/3455157482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d98102eb0 0x7f9d98103330 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:09.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:08.999+0000 7f9d9e3f8640 1 --2- 172.21.15.33:0/3455157482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d980ffed0 0x7f9d981002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:09.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:08.999+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/3455157482 >> 172.21.15.33:0/3455157482 conn(0x7f9d980fba40 msgr2=0x7f9d980fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:09.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.000+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/3455157482 shutdown_connections 2024-09-16T00:17:09.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.000+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/3455157482 wait complete. 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.000+0000 7f9d9e3f8640 1 Processor -- start 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.000+0000 7f9d9e3f8640 1 -- start start 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.000+0000 7f9d9e3f8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d980ffed0 0x7f9d98102810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.000+0000 7f9d9e3f8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d98102eb0 0x7f9d98100e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.000+0000 7f9d9e3f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d98103870 0x7f9d981013a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.000+0000 7f9d9e3f8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9810d8b0 con 0x7f9d98103870 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.000+0000 7f9d9e3f8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9810d730 con 0x7f9d98102eb0 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.000+0000 7f9d9e3f8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9810da30 con 0x7f9d980ffed0 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d9c995640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d98103870 0x7f9d981013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d9c995640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d98103870 0x7f9d981013a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55224/0 (socket says 172.21.15.33:55224) 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d97fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d980ffed0 0x7f9d98102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d98102eb0 0x7f9d98100e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d9c995640 1 -- 172.21.15.33:0/2430849410 learned_addr learned my addr 172.21.15.33:0/2430849410 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:09.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d9c995640 1 -- 172.21.15.33:0/2430849410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d980ffed0 msgr2=0x7f9d98102810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:09.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d9c995640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d980ffed0 0x7f9d98102810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:09.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d9c995640 1 -- 172.21.15.33:0/2430849410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d98102eb0 msgr2=0x7f9d98100e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:09.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d9c995640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d98102eb0 0x7f9d98100e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:09.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d9c995640 1 -- 172.21.15.33:0/2430849410 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d98101c60 con 0x7f9d98103870 2024-09-16T00:17:09.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d977fe640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d98102eb0 0x7f9d98100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:17:09.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d9c995640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d98103870 0x7f9d981013a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000eaa0 tx=0x7f9d9000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:09.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d957fa640 1 -- 172.21.15.33:0/2430849410 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9000cdb0 con 0x7f9d98103870 2024-09-16T00:17:09.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d957fa640 1 -- 172.21.15.33:0/2430849410 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d90005020 con 0x7f9d98103870 2024-09-16T00:17:09.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/2430849410 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d981a2100 con 0x7f9d98103870 2024-09-16T00:17:09.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d957fa640 1 -- 172.21.15.33:0/2430849410 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d900116b0 con 0x7f9d98103870 2024-09-16T00:17:09.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.001+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/2430849410 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d981a2660 con 0x7f9d98103870 2024-09-16T00:17:09.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.002+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/2430849410 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d981091a0 con 0x7f9d98103870 2024-09-16T00:17:09.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.004+0000 7f9d957fa640 1 -- 172.21.15.33:0/2430849410 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9d90005560 con 0x7f9d98103870 2024-09-16T00:17:09.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.004+0000 7f9d957fa640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9d800a32c0 0x7f9d800a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:09.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.004+0000 7f9d957fa640 1 -- 172.21.15.33:0/2430849410 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9382+0+0 (secure 0 0 0) 0x7f9d900d0230 con 0x7f9d98103870 2024-09-16T00:17:09.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.004+0000 7f9d97fff640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9d800a32c0 0x7f9d800a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:09.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.005+0000 7f9d97fff640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9d800a32c0 0x7f9d800a5780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9d9806a590 tx=0x7f9d88005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:09.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.008+0000 7f9d957fa640 1 -- 172.21.15.33:0/2430849410 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9d90016030 con 0x7f9d98103870 2024-09-16T00:17:09.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.197+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/2430849410 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f9d98062630 con 0x7f9d800a32c0 2024-09-16T00:17:09.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.199+0000 7f9d957fa640 1 -- 172.21.15.33:0/2430849410 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+187 (secure 0 0 0) 0x7f9d98062630 con 0x7f9d800a32c0 2024-09-16T00:17:09.200 INFO:teuthology.orchestra.run.smithi033.stdout:HOST ADDR LABELS STATUS 2024-09-16T00:17:09.200 INFO:teuthology.orchestra.run.smithi033.stdout:smithi033 172.21.15.33 2024-09-16T00:17:09.200 INFO:teuthology.orchestra.run.smithi033.stdout:smithi060 172.21.15.60 2024-09-16T00:17:09.200 INFO:teuthology.orchestra.run.smithi033.stdout:smithi083 172.21.15.83 2024-09-16T00:17:09.200 INFO:teuthology.orchestra.run.smithi033.stdout:3 hosts in cluster 2024-09-16T00:17:09.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.201+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/2430849410 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9d800a32c0 msgr2=0x7f9d800a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:09.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.201+0000 7f9d9e3f8640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9d800a32c0 0x7f9d800a5780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9d9806a590 tx=0x7f9d88005f90 comp rx=0 tx=0).stop 2024-09-16T00:17:09.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.202+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/2430849410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d98103870 msgr2=0x7f9d981013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:09.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.202+0000 7f9d9e3f8640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d98103870 0x7f9d981013a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000eaa0 tx=0x7f9d9000ef60 comp rx=0 tx=0).stop 2024-09-16T00:17:09.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.202+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/2430849410 shutdown_connections 2024-09-16T00:17:09.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.202+0000 7f9d9e3f8640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9d800a32c0 0x7f9d800a5780 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:09.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.202+0000 7f9d9e3f8640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d98103870 0x7f9d981013a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:09.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.202+0000 7f9d9e3f8640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d98102eb0 0x7f9d98100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:09.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.202+0000 7f9d9e3f8640 1 --2- 172.21.15.33:0/2430849410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d980ffed0 0x7f9d98102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:09.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.202+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/2430849410 >> 172.21.15.33:0/2430849410 conn(0x7f9d980fba40 msgr2=0x7f9d98106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:09.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.202+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/2430849410 shutdown_connections 2024-09-16T00:17:09.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:09.202+0000 7f9d9e3f8640 1 -- 172.21.15.33:0/2430849410 wait complete. 2024-09-16T00:17:09.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:17:09.872 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-09-16T00:17:09.872 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-09-16T00:17:09.872 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-09-16T00:17:09.872 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 6]}]': finished 2024-09-16T00:17:09.872 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-09-16T00:17:09.872 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-09-16T00:17:09.872 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-09-16T00:17:09.872 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2e", "id": [1, 8]}]': finished 2024-09-16T00:17:09.872 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]': finished 2024-09-16T00:17:09.872 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-09-16T00:17:09.872 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: osdmap e89: 12 total, 12 up, 12 in 2024-09-16T00:17:09.873 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-16T00:17:09.873 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.873 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.873 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.873 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.873 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.873 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:09 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.899 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- bash -c 'ceph orch device ls' 2024-09-16T00:17:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:17:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-09-16T00:17:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-09-16T00:17:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-09-16T00:17:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 6]}]': finished 2024-09-16T00:17:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-09-16T00:17:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-09-16T00:17:09.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-09-16T00:17:09.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2e", "id": [1, 8]}]': finished 2024-09-16T00:17:09.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]': finished 2024-09-16T00:17:09.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-09-16T00:17:09.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: osdmap e89: 12 total, 12 up, 12 in 2024-09-16T00:17:09.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-16T00:17:09.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:09 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:17:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-09-16T00:17:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-09-16T00:17:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-09-16T00:17:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 6]}]': finished 2024-09-16T00:17:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-09-16T00:17:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-09-16T00:17:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-09-16T00:17:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2e", "id": [1, 8]}]': finished 2024-09-16T00:17:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]': finished 2024-09-16T00:17:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-09-16T00:17:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: osdmap e89: 12 total, 12 up, 12 in 2024-09-16T00:17:09.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-16T00:17:09.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:09.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:09 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:10.175 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:10.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.767+0000 7f200b3c9640 1 -- 172.21.15.33:0/3121454480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2004101d10 msgr2=0x7f2004102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:10.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.767+0000 7f200b3c9640 1 --2- 172.21.15.33:0/3121454480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2004101d10 0x7f2004102190 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009900 tx=0x7f1ff402a410 comp rx=0 tx=0).stop 2024-09-16T00:17:10.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.767+0000 7f200b3c9640 1 -- 172.21.15.33:0/3121454480 shutdown_connections 2024-09-16T00:17:10.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.767+0000 7f200b3c9640 1 --2- 172.21.15.33:0/3121454480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20041026d0 0x7f2004108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:10.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.767+0000 7f200b3c9640 1 --2- 172.21.15.33:0/3121454480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2004101d10 0x7f2004102190 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:10.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.767+0000 7f200b3c9640 1 --2- 172.21.15.33:0/3121454480 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2004100b10 0x7f2004100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:10.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.767+0000 7f200b3c9640 1 -- 172.21.15.33:0/3121454480 >> 172.21.15.33:0/3121454480 conn(0x7f20040fc2a0 msgr2=0x7f20040fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:10.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.768+0000 7f200b3c9640 1 -- 172.21.15.33:0/3121454480 shutdown_connections 2024-09-16T00:17:10.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.768+0000 7f200b3c9640 1 -- 172.21.15.33:0/3121454480 wait complete. 2024-09-16T00:17:10.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.768+0000 7f200b3c9640 1 Processor -- start 2024-09-16T00:17:10.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.768+0000 7f200b3c9640 1 -- start start 2024-09-16T00:17:10.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.768+0000 7f200b3c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2004100b10 0x7f200419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:10.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.768+0000 7f200b3c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2004101d10 0x7f200419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:10.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.768+0000 7f200b3c9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20041026d0 0x7f20041a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:10.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.768+0000 7f200b3c9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f200410c810 con 0x7f20041026d0 2024-09-16T00:17:10.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.768+0000 7f200b3c9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f200410c690 con 0x7f2004101d10 2024-09-16T00:17:10.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.768+0000 7f200b3c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f200410c990 con 0x7f2004100b10 2024-09-16T00:17:10.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.768+0000 7f2009966640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20041026d0 0x7f20041a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:10.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f2009966640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20041026d0 0x7f20041a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33204/0 (socket says 172.21.15.33:33204) 2024-09-16T00:17:10.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f2009966640 1 -- 172.21.15.33:0/2265828500 learned_addr learned my addr 172.21.15.33:0/2265828500 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:10.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f2009165640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2004100b10 0x7f200419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f2008964640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2004101d10 0x7f200419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f2009966640 1 -- 172.21.15.33:0/2265828500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2004100b10 msgr2=0x7f200419a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f2009966640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2004100b10 0x7f200419a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f2009966640 1 -- 172.21.15.33:0/2265828500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2004101d10 msgr2=0x7f200419a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f2009966640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2004101d10 0x7f200419a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f2009966640 1 -- 172.21.15.33:0/2265828500 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20041a1df0 con 0x7f20041026d0 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f2009165640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2004100b10 0x7f200419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f2009966640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20041026d0 0x7f20041a1780 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00b970 tx=0x7f1ffc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f1ffa7fc640 1 -- 172.21.15.33:0/2265828500 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc019070 con 0x7f20041026d0 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f1ffa7fc640 1 -- 172.21.15.33:0/2265828500 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ffc00cb40 con 0x7f20041026d0 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f200b3c9640 1 -- 172.21.15.33:0/2265828500 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20041a20e0 con 0x7f20041026d0 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f1ffa7fc640 1 -- 172.21.15.33:0/2265828500 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc007650 con 0x7f20041026d0 2024-09-16T00:17:10.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.769+0000 7f200b3c9640 1 -- 172.21.15.33:0/2265828500 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20041a2620 con 0x7f20041026d0 2024-09-16T00:17:10.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.770+0000 7f200b3c9640 1 -- 172.21.15.33:0/2265828500 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20041a28e0 con 0x7f20041026d0 2024-09-16T00:17:10.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.772+0000 7f1ffa7fc640 1 -- 172.21.15.33:0/2265828500 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f1ffc01c070 con 0x7f20041026d0 2024-09-16T00:17:10.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.772+0000 7f1ffa7fc640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1ff00a3270 0x7f1ff00a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:10.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.772+0000 7f2009165640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1ff00a3270 0x7f1ff00a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:10.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.772+0000 7f1ffa7fc640 1 -- 172.21.15.33:0/2265828500 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1ffc0cf890 con 0x7f20041026d0 2024-09-16T00:17:10.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.773+0000 7f2009165640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1ff00a3270 0x7f1ff00a5730 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2004101b70 tx=0x7f2000005e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:10.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.775+0000 7f1ffa7fc640 1 -- 172.21.15.33:0/2265828500 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1ffc09c130 con 0x7f20041026d0 2024-09-16T00:17:10.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:10 smithi033 ceph-mon[31243]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:10.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:10 smithi033 ceph-mon[31243]: osdmap e90: 12 total, 12 up, 12 in 2024-09-16T00:17:10.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:10 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:17:10.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:10 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:17:10.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:10 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:10.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:10 smithi083 ceph-mon[35242]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:10.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:10 smithi083 ceph-mon[35242]: osdmap e90: 12 total, 12 up, 12 in 2024-09-16T00:17:10.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:10 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:17:10.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:10 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:17:10.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:10 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:10.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.958+0000 7f200b3c9640 1 -- 172.21.15.33:0/2265828500 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f20041a2ab0 con 0x7f1ff00a3270 2024-09-16T00:17:10.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.960+0000 7f1ffa7fc640 1 -- 172.21.15.33:0/2265828500 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f20041a2ab0 con 0x7f1ff00a3270 2024-09-16T00:17:10.961 INFO:teuthology.orchestra.run.smithi033.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-16T00:17:10.961 INFO:teuthology.orchestra.run.smithi033.stdout:smithi033 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330008Q400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-16T00:17:10.961 INFO:teuthology.orchestra.run.smithi033.stdout:smithi060 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476008A480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-09-16T00:17:10.961 INFO:teuthology.orchestra.run.smithi033.stdout:smithi083 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040148400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-16T00:17:10.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.962+0000 7f200b3c9640 1 -- 172.21.15.33:0/2265828500 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1ff00a3270 msgr2=0x7f1ff00a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:10.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.962+0000 7f200b3c9640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1ff00a3270 0x7f1ff00a5730 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2004101b70 tx=0x7f2000005e70 comp rx=0 tx=0).stop 2024-09-16T00:17:10.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.962+0000 7f200b3c9640 1 -- 172.21.15.33:0/2265828500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20041026d0 msgr2=0x7f20041a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:10.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.962+0000 7f200b3c9640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20041026d0 0x7f20041a1780 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00b970 tx=0x7f1ffc00be30 comp rx=0 tx=0).stop 2024-09-16T00:17:10.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.962+0000 7f200b3c9640 1 -- 172.21.15.33:0/2265828500 shutdown_connections 2024-09-16T00:17:10.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.962+0000 7f200b3c9640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1ff00a3270 0x7f1ff00a5730 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:10.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.962+0000 7f200b3c9640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20041026d0 0x7f20041a1780 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:10.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.962+0000 7f200b3c9640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2004101d10 0x7f200419a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:10.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.962+0000 7f200b3c9640 1 --2- 172.21.15.33:0/2265828500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2004100b10 0x7f200419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:10.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.962+0000 7f200b3c9640 1 -- 172.21.15.33:0/2265828500 >> 172.21.15.33:0/2265828500 conn(0x7f20040fc2a0 msgr2=0x7f200406a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:10.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.962+0000 7f200b3c9640 1 -- 172.21.15.33:0/2265828500 shutdown_connections 2024-09-16T00:17:10.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:10.962+0000 7f200b3c9640 1 -- 172.21.15.33:0/2265828500 wait complete. 2024-09-16T00:17:10.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:10 smithi060 ceph-mon[35185]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:10.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:10 smithi060 ceph-mon[35185]: osdmap e90: 12 total, 12 up, 12 in 2024-09-16T00:17:10.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:10 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:17:10.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:10 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:17:10.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:10 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:17:11.717 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-16T00:17:11.727 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-09-16T00:17:11.727 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- bash -c 'ceph fs dump' 2024-09-16T00:17:11.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:11 smithi083 ceph-mon[35242]: pgmap v351: 129 pgs: 5 peering, 124 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 383 B/s wr, 6 op/s 2024-09-16T00:17:11.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:11 smithi083 ceph-mon[35242]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-16T00:17:11.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:11 smithi083 ceph-mon[35242]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:11 smithi060 ceph-mon[35185]: pgmap v351: 129 pgs: 5 peering, 124 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 383 B/s wr, 6 op/s 2024-09-16T00:17:11.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:11 smithi060 ceph-mon[35185]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-16T00:17:11.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:11 smithi060 ceph-mon[35185]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:11.985 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:12.063 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:11 smithi033 ceph-mon[31243]: pgmap v351: 129 pgs: 5 peering, 124 active+clean; 642 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 383 B/s wr, 6 op/s 2024-09-16T00:17:12.063 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:11 smithi033 ceph-mon[31243]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-16T00:17:12.063 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:11 smithi033 ceph-mon[31243]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:12.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.474+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/725663563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c48107900 msgr2=0x7f3c48109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:12.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.474+0000 7f3c4e4e0640 1 --2- 172.21.15.33:0/725663563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c48107900 0x7f3c48109d50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3c40009ca0 tx=0x7f3c4002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:17:12.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.475+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/725663563 shutdown_connections 2024-09-16T00:17:12.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.475+0000 7f3c4e4e0640 1 --2- 172.21.15.33:0/725663563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c48107900 0x7f3c48109d50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:12.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.475+0000 7f3c4e4e0640 1 --2- 172.21.15.33:0/725663563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c48104fd0 0x7f3c481073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:12.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.475+0000 7f3c4e4e0640 1 --2- 172.21.15.33:0/725663563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c481026a0 0x7f3c48104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:12.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.475+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/725663563 >> 172.21.15.33:0/725663563 conn(0x7f3c480fc2a0 msgr2=0x7f3c480fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:12.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.475+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/725663563 shutdown_connections 2024-09-16T00:17:12.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.475+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/725663563 wait complete. 2024-09-16T00:17:12.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c4e4e0640 1 Processor -- start 2024-09-16T00:17:12.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c4e4e0640 1 -- start start 2024-09-16T00:17:12.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c4e4e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c481026a0 0x7f3c48197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:12.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c4e4e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c48104fd0 0x7f3c48198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:12.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c4e4e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c48107900 0x7f3c4819f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:12.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c4e4e0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4810d8c0 con 0x7f3c48104fd0 2024-09-16T00:17:12.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c4e4e0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4810d740 con 0x7f3c481026a0 2024-09-16T00:17:12.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c4e4e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4810da40 con 0x7f3c48107900 2024-09-16T00:17:12.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c48104fd0 0x7f3c48198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:12.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c48104fd0 0x7f3c48198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33232/0 (socket says 172.21.15.33:33232) 2024-09-16T00:17:12.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c477fe640 1 -- 172.21.15.33:0/2902808225 learned_addr learned my addr 172.21.15.33:0/2902808225 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:12.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c4ca7d640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c48107900 0x7f3c4819f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:12.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.476+0000 7f3c47fff640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c481026a0 0x7f3c48197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:12.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.477+0000 7f3c477fe640 1 -- 172.21.15.33:0/2902808225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c48107900 msgr2=0x7f3c4819f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:12.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.477+0000 7f3c477fe640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c48107900 0x7f3c4819f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:12.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.477+0000 7f3c477fe640 1 -- 172.21.15.33:0/2902808225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c481026a0 msgr2=0x7f3c48197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:12.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.477+0000 7f3c477fe640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c481026a0 0x7f3c48197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:12.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.477+0000 7f3c477fe640 1 -- 172.21.15.33:0/2902808225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4819fb20 con 0x7f3c48104fd0 2024-09-16T00:17:12.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.477+0000 7f3c47fff640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c481026a0 0x7f3c48197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:12.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.477+0000 7f3c477fe640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c48104fd0 0x7f3c48198430 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c00d740 tx=0x7f3c3c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:12.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.477+0000 7f3c457fa640 1 -- 172.21.15.33:0/2902808225 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c017070 con 0x7f3c48104fd0 2024-09-16T00:17:12.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.477+0000 7f3c457fa640 1 -- 172.21.15.33:0/2902808225 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c3c00de50 con 0x7f3c48104fd0 2024-09-16T00:17:12.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.477+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/2902808225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c4819fe10 con 0x7f3c48104fd0 2024-09-16T00:17:12.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.477+0000 7f3c457fa640 1 -- 172.21.15.33:0/2902808225 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c004e00 con 0x7f3c48104fd0 2024-09-16T00:17:12.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.477+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/2902808225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c481a0270 con 0x7f3c48104fd0 2024-09-16T00:17:12.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.478+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/2902808225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c480fdd90 con 0x7f3c48104fd0 2024-09-16T00:17:12.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.479+0000 7f3c457fa640 1 -- 172.21.15.33:0/2902808225 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f3c3c004fa0 con 0x7f3c48104fd0 2024-09-16T00:17:12.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.480+0000 7f3c457fa640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3c300a3270 0x7f3c300a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:12.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.480+0000 7f3c47fff640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3c300a3270 0x7f3c300a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:12.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.480+0000 7f3c457fa640 1 -- 172.21.15.33:0/2902808225 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3c3c0cf750 con 0x7f3c48104fd0 2024-09-16T00:17:12.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.481+0000 7f3c47fff640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3c300a3270 0x7f3c300a5730 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3c380058c0 tx=0x7f3c38005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:12.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.485+0000 7f3c457fa640 1 -- 172.21.15.33:0/2902808225 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3c3c09bf90 con 0x7f3c48104fd0 2024-09-16T00:17:12.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.698+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/2902808225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f3c48062df0 con 0x7f3c48104fd0 2024-09-16T00:17:12.700 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:12 smithi033 ceph-mon[31243]: pgmap v352: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 6 op/s 2024-09-16T00:17:12.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.699+0000 7f3c457fa640 1 -- 172.21.15.33:0/2902808225 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) ==== 76+0+3082 (secure 0 0 0) 0x7f3c3c09b890 con 0x7f3c48104fd0 2024-09-16T00:17:12.701 INFO:teuthology.orchestra.run.smithi033.stdout:e25 2024-09-16T00:17:12.701 INFO:teuthology.orchestra.run.smithi033.stdout:btime 2024-09-16T00:15:52:174386+0000 2024-09-16T00:17:12.701 INFO:teuthology.orchestra.run.smithi033.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-16T00:17:12.701 INFO:teuthology.orchestra.run.smithi033.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T00:17:12.701 INFO:teuthology.orchestra.run.smithi033.stdout:legacy client fscid: 1 2024-09-16T00:17:12.701 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:12.701 INFO:teuthology.orchestra.run.smithi033.stdout:Filesystem 'cephfs' (1) 2024-09-16T00:17:12.701 INFO:teuthology.orchestra.run.smithi033.stdout:fs_name cephfs 2024-09-16T00:17:12.701 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 25 2024-09-16T00:17:12.701 INFO:teuthology.orchestra.run.smithi033.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-09-16T00:17:12.701 INFO:teuthology.orchestra.run.smithi033.stdout:created 2024-09-16T00:15:07.246355+0000 2024-09-16T00:17:12.701 INFO:teuthology.orchestra.run.smithi033.stdout:modified 2024-09-16T00:15:51.912925+0000 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:tableserver 0 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:root 0 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:session_timeout 300 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:session_autoclose 300 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:max_file_size 1099511627776 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:max_xattr_size 65536 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:required_client_features {} 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:last_failure 0 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:last_failure_osd_epoch 0 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:max_mds 3 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:in 0,1,2 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:up {0=24494,1=24493,2=24472} 2024-09-16T00:17:12.702 INFO:teuthology.orchestra.run.smithi033.stdout:failed 2024-09-16T00:17:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:damaged 2024-09-16T00:17:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:stopped 2024-09-16T00:17:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:data_pools [3,4] 2024-09-16T00:17:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:metadata_pool 2 2024-09-16T00:17:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:inline_data disabled 2024-09-16T00:17:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:balancer 2024-09-16T00:17:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:bal_rank_mask -1 2024-09-16T00:17:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:standby_count_wanted 0 2024-09-16T00:17:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:qdb_cluster leader: 24494 members: 24472,24493,24494 2024-09-16T00:17:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.b{0:24494} state up:active seq 4 addr [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T00:17:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.f{0:24466} state up:standby-replay seq 1 addr [v2:172.21.15.83:6834/2667354509,v1:172.21.15.83:6835/2667354509] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T00:17:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.i{1:24493} state up:active seq 17 addr [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T00:17:12.704 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.h{1:24464} state up:standby-replay seq 1 addr [v2:172.21.15.60:6836/452012251,v1:172.21.15.60:6837/452012251] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T00:17:12.704 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.c{2:24472} state up:active seq 18 addr [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T00:17:12.704 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.e{2:24437} state up:standby-replay seq 1 addr [v2:172.21.15.60:6834/1002409397,v1:172.21.15.60:6835/1002409397] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T00:17:12.704 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:12.704 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:12.704 INFO:teuthology.orchestra.run.smithi033.stdout:Standby daemons: 2024-09-16T00:17:12.704 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:12.704 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.j{-1:14541} state up:standby seq 1 addr [v2:172.21.15.33:6834/2093179667,v1:172.21.15.33:6835/2093179667] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T00:17:12.704 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.d{-1:14553} state up:standby seq 1 addr [v2:172.21.15.33:6836/3526002638,v1:172.21.15.33:6837/3526002638] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T00:17:12.704 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.g{-1:14604} state up:standby seq 1 addr [v2:172.21.15.33:6838/4092505131,v1:172.21.15.33:6839/4092505131] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T00:17:12.704 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.a{-1:14616} state up:standby seq 1 addr [v2:172.21.15.33:6840/1588024134,v1:172.21.15.33:6841/1588024134] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T00:17:12.704 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.k{-1:24410} state up:standby seq 1 addr [v2:172.21.15.60:6832/2130595683,v1:172.21.15.60:6833/2130595683] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T00:17:12.705 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.l{-1:24433} state up:standby seq 1 addr [v2:172.21.15.83:6832/1083835397,v1:172.21.15.83:6833/1083835397] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T00:17:12.705 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-09-16T00:17:12.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.702+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/2902808225 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3c300a3270 msgr2=0x7f3c300a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:12.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.702+0000 7f3c4e4e0640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3c300a3270 0x7f3c300a5730 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3c380058c0 tx=0x7f3c38005850 comp rx=0 tx=0).stop 2024-09-16T00:17:12.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.702+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/2902808225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c48104fd0 msgr2=0x7f3c48198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:12.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.702+0000 7f3c4e4e0640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c48104fd0 0x7f3c48198430 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c00d740 tx=0x7f3c3c00dc00 comp rx=0 tx=0).stop 2024-09-16T00:17:12.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.702+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/2902808225 shutdown_connections 2024-09-16T00:17:12.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.702+0000 7f3c4e4e0640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3c300a3270 0x7f3c300a5730 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:12.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.702+0000 7f3c4e4e0640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c48107900 0x7f3c4819f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:12.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.702+0000 7f3c4e4e0640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c48104fd0 0x7f3c48198430 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:12.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.702+0000 7f3c4e4e0640 1 --2- 172.21.15.33:0/2902808225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c481026a0 0x7f3c48197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:12.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.702+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/2902808225 >> 172.21.15.33:0/2902808225 conn(0x7f3c480fc2a0 msgr2=0x7f3c48102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:12.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.702+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/2902808225 shutdown_connections 2024-09-16T00:17:12.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:12.702+0000 7f3c4e4e0640 1 -- 172.21.15.33:0/2902808225 wait complete. 2024-09-16T00:17:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:12 smithi083 ceph-mon[35242]: pgmap v352: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 6 op/s 2024-09-16T00:17:12.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:12 smithi060 ceph-mon[35185]: pgmap v352: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 6 op/s 2024-09-16T00:17:13.425 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- bash -c 'ceph osd dump' 2024-09-16T00:17:13.694 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:13.815 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:13 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2902808225' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-16T00:17:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:13 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2902808225' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-16T00:17:13.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:13 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2902808225' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-16T00:17:14.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.242+0000 7faf690f4640 1 -- 172.21.15.33:0/2745748285 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf64107900 msgr2=0x7faf64109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.242+0000 7faf690f4640 1 --2- 172.21.15.33:0/2745748285 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf64107900 0x7faf64109d50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009ca0 tx=0x7faf5c02a8c0 comp rx=0 tx=0).stop 2024-09-16T00:17:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.242+0000 7faf690f4640 1 -- 172.21.15.33:0/2745748285 shutdown_connections 2024-09-16T00:17:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.242+0000 7faf690f4640 1 --2- 172.21.15.33:0/2745748285 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf64107900 0x7faf64109d50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.242+0000 7faf690f4640 1 --2- 172.21.15.33:0/2745748285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf64104fd0 0x7faf641073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.242+0000 7faf690f4640 1 --2- 172.21.15.33:0/2745748285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf641026a0 0x7faf64104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.242+0000 7faf690f4640 1 -- 172.21.15.33:0/2745748285 >> 172.21.15.33:0/2745748285 conn(0x7faf640fc2a0 msgr2=0x7faf640fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.242+0000 7faf690f4640 1 -- 172.21.15.33:0/2745748285 shutdown_connections 2024-09-16T00:17:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.242+0000 7faf690f4640 1 -- 172.21.15.33:0/2745748285 wait complete. 2024-09-16T00:17:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.243+0000 7faf690f4640 1 Processor -- start 2024-09-16T00:17:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.243+0000 7faf690f4640 1 -- start start 2024-09-16T00:17:14.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.243+0000 7faf690f4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf641026a0 0x7faf64197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:14.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.243+0000 7faf690f4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf64104fd0 0x7faf64198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:14.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.243+0000 7faf690f4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf64107900 0x7faf6419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:14.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.243+0000 7faf690f4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410d8c0 con 0x7faf64104fd0 2024-09-16T00:17:14.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.243+0000 7faf690f4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410d740 con 0x7faf64107900 2024-09-16T00:17:14.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.243+0000 7faf690f4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410da40 con 0x7faf641026a0 2024-09-16T00:17:14.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.243+0000 7faf6259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf64104fd0 0x7faf64198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:14.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.243+0000 7faf6259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf64104fd0 0x7faf64198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33258/0 (socket says 172.21.15.33:33258) 2024-09-16T00:17:14.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.243+0000 7faf6259c640 1 -- 172.21.15.33:0/1220225688 learned_addr learned my addr 172.21.15.33:0/1220225688 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:14.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf6359e640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf64107900 0x7faf6419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:14.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf62d9d640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf641026a0 0x7faf64197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:14.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf6259c640 1 -- 172.21.15.33:0/1220225688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf641026a0 msgr2=0x7faf64197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:14.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf6259c640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf641026a0 0x7faf64197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:14.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf6259c640 1 -- 172.21.15.33:0/1220225688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf64107900 msgr2=0x7faf6419f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:14.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf6259c640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf64107900 0x7faf6419f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:14.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf6259c640 1 -- 172.21.15.33:0/1220225688 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf6419fb20 con 0x7faf64104fd0 2024-09-16T00:17:14.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf62d9d640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf641026a0 0x7faf64197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:14.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf6259c640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf64104fd0 0x7faf64198430 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7faf5400d770 tx=0x7faf5400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:14.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf5bfff640 1 -- 172.21.15.33:0/1220225688 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf54017070 con 0x7faf64104fd0 2024-09-16T00:17:14.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf690f4640 1 -- 172.21.15.33:0/1220225688 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf6419fe10 con 0x7faf64104fd0 2024-09-16T00:17:14.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf5bfff640 1 -- 172.21.15.33:0/1220225688 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf54009c30 con 0x7faf64104fd0 2024-09-16T00:17:14.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf5bfff640 1 -- 172.21.15.33:0/1220225688 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5400f620 con 0x7faf64104fd0 2024-09-16T00:17:14.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.244+0000 7faf690f4640 1 -- 172.21.15.33:0/1220225688 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf641a02c0 con 0x7faf64104fd0 2024-09-16T00:17:14.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.245+0000 7faf690f4640 1 -- 172.21.15.33:0/1220225688 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf640fdd90 con 0x7faf64104fd0 2024-09-16T00:17:14.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.246+0000 7faf5bfff640 1 -- 172.21.15.33:0/1220225688 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7faf5401a070 con 0x7faf64104fd0 2024-09-16T00:17:14.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.247+0000 7faf5bfff640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faf500a3270 0x7faf500a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:14.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.247+0000 7faf62d9d640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faf500a3270 0x7faf500a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:14.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.247+0000 7faf5bfff640 1 -- 172.21.15.33:0/1220225688 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7faf540cf990 con 0x7faf64104fd0 2024-09-16T00:17:14.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.248+0000 7faf62d9d640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faf500a3270 0x7faf500a5730 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0058c0 tx=0x7faf4c005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:14.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.251+0000 7faf5bfff640 1 -- 172.21.15.33:0/1220225688 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7faf5409c1d0 con 0x7faf64104fd0 2024-09-16T00:17:14.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.428+0000 7faf690f4640 1 -- 172.21.15.33:0/1220225688 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7faf64062df0 con 0x7faf64104fd0 2024-09-16T00:17:14.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.429+0000 7faf5bfff640 1 -- 172.21.15.33:0/1220225688 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+5829 (secure 0 0 0) 0x7faf54084e20 con 0x7faf64104fd0 2024-09-16T00:17:14.429 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 90 2024-09-16T00:17:14.429 INFO:teuthology.orchestra.run.smithi033.stdout:fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:17:14.429 INFO:teuthology.orchestra.run.smithi033.stdout:created 2024-09-16T00:07:21.167733+0000 2024-09-16T00:17:14.429 INFO:teuthology.orchestra.run.smithi033.stdout:modified 2024-09-16T00:17:09.569270+0000 2024-09-16T00:17:14.429 INFO:teuthology.orchestra.run.smithi033.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T00:17:14.430 INFO:teuthology.orchestra.run.smithi033.stdout:crush_version 27 2024-09-16T00:17:14.430 INFO:teuthology.orchestra.run.smithi033.stdout:full_ratio 0.95 2024-09-16T00:17:14.430 INFO:teuthology.orchestra.run.smithi033.stdout:backfillfull_ratio 0.9 2024-09-16T00:17:14.430 INFO:teuthology.orchestra.run.smithi033.stdout:nearfull_ratio 0.85 2024-09-16T00:17:14.430 INFO:teuthology.orchestra.run.smithi033.stdout:require_min_compat_client luminous 2024-09-16T00:17:14.430 INFO:teuthology.orchestra.run.smithi033.stdout:min_compat_client luminous 2024-09-16T00:17:14.430 INFO:teuthology.orchestra.run.smithi033.stdout:require_osd_release squid 2024-09-16T00:17:14.430 INFO:teuthology.orchestra.run.smithi033.stdout:stretch_mode_enabled false 2024-09-16T00:17:14.430 INFO:teuthology.orchestra.run.smithi033.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-16T00:17:14.431 INFO:teuthology.orchestra.run.smithi033.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-09-16T00:17:14.431 INFO:teuthology.orchestra.run.smithi033.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-09-16T00:17:14.431 INFO:teuthology.orchestra.run.smithi033.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-09-16T00:17:14.431 INFO:teuthology.orchestra.run.smithi033.stdout:max_osd 12 2024-09-16T00:17:14.431 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0 up in weight 1 up_from 8 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6802/1463114656,v1:172.21.15.33:6803/1463114656] [v2:172.21.15.33:6804/1463114656,v1:172.21.15.33:6805/1463114656] exists,up 12847d0f-d226-40db-bd81-100ef8fddf1d 2024-09-16T00:17:14.431 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1 up in weight 1 up_from 12 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6800/1932822239,v1:172.21.15.60:6801/1932822239] [v2:172.21.15.60:6802/1932822239,v1:172.21.15.60:6803/1932822239] exists,up a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d 2024-09-16T00:17:14.431 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2 up in weight 1 up_from 17 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.83:6800/409428043,v1:172.21.15.83:6801/409428043] [v2:172.21.15.83:6802/409428043,v1:172.21.15.83:6803/409428043] exists,up b978c9fd-d8e4-4b4a-8bc2-1671f0232a49 2024-09-16T00:17:14.431 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3 up in weight 1 up_from 24 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6810/712525362,v1:172.21.15.33:6811/712525362] [v2:172.21.15.33:6812/712525362,v1:172.21.15.33:6813/712525362] exists,up 94dae86a-b8b7-4563-84f4-c14c9a07bda2 2024-09-16T00:17:14.431 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4 up in weight 1 up_from 31 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6808/3488110488,v1:172.21.15.60:6809/3488110488] [v2:172.21.15.60:6810/3488110488,v1:172.21.15.60:6811/3488110488] exists,up c3fa6bf5-b880-4cd0-9912-c64f33b78b34 2024-09-16T00:17:14.431 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5 up in weight 1 up_from 35 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.83:6808/2896581095,v1:172.21.15.83:6809/2896581095] [v2:172.21.15.83:6810/2896581095,v1:172.21.15.83:6811/2896581095] exists,up cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef 2024-09-16T00:17:14.431 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6 up in weight 1 up_from 42 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6818/2633676155,v1:172.21.15.33:6819/2633676155] [v2:172.21.15.33:6820/2633676155,v1:172.21.15.33:6821/2633676155] exists,up 48de98ea-b98b-4f4f-b771-38a7d816e784 2024-09-16T00:17:14.431 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7 up in weight 1 up_from 48 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6816/1473789559,v1:172.21.15.60:6817/1473789559] [v2:172.21.15.60:6818/1473789559,v1:172.21.15.60:6819/1473789559] exists,up 44266591-c35b-48fe-9a90-e1597d7e311a 2024-09-16T00:17:14.432 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8 up in weight 1 up_from 55 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.83:6816/2115210527,v1:172.21.15.83:6817/2115210527] [v2:172.21.15.83:6818/2115210527,v1:172.21.15.83:6819/2115210527] exists,up 4fadf88b-ff77-46ed-8c31-34b974af9dc0 2024-09-16T00:17:14.432 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9 up in weight 1 up_from 62 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6826/2906942241,v1:172.21.15.33:6827/2906942241] [v2:172.21.15.33:6828/2906942241,v1:172.21.15.33:6829/2906942241] exists,up 03160394-ba77-4a71-80fe-6c20837c3cb7 2024-09-16T00:17:14.432 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10 up in weight 1 up_from 67 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6824/3903707079,v1:172.21.15.60:6825/3903707079] [v2:172.21.15.60:6826/3903707079,v1:172.21.15.60:6827/3903707079] exists,up ae2336fa-0b07-42a2-b67f-c77b0f7db0da 2024-09-16T00:17:14.432 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11 up in weight 1 up_from 74 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.83:6824/2178604075,v1:172.21.15.83:6825/2178604075] [v2:172.21.15.83:6826/2178604075,v1:172.21.15.83:6827/2178604075] exists,up 02536f55-f270-4950-8fb3-0b528f79dac8 2024-09-16T00:17:14.432 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 2.2 [7,11] 2024-09-16T00:17:14.432 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 2.a [2,11] 2024-09-16T00:17:14.432 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 2.c [2,4] 2024-09-16T00:17:14.432 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 2.14 [2,4] 2024-09-16T00:17:14.432 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 2.1a [8,1] 2024-09-16T00:17:14.432 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 2.1f [8,1] 2024-09-16T00:17:14.432 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.0 [5,9] 2024-09-16T00:17:14.432 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.2 [3,9] 2024-09-16T00:17:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.c [3,9] 2024-09-16T00:17:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.19 [3,6] 2024-09-16T00:17:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.1a [5,7] 2024-09-16T00:17:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.1b [4,8] 2024-09-16T00:17:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.21 [4,8] 2024-09-16T00:17:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.2e [1,8] 2024-09-16T00:17:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.31 [4,7] 2024-09-16T00:17:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.3a [4,6] 2024-09-16T00:17:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 4.9 [5,8] 2024-09-16T00:17:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 4.10 [9,10] 2024-09-16T00:17:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 4.13 [9,8] 2024-09-16T00:17:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 4.17 [9,8] 2024-09-16T00:17:14.434 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/3553033370 expires 2024-09-17T00:08:07.548547+0000 2024-09-16T00:17:14.434 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6801/122116489 expires 2024-09-17T00:08:07.548547+0000 2024-09-16T00:17:14.434 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/1410359644 expires 2024-09-17T00:08:07.548547+0000 2024-09-16T00:17:14.434 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/2692588880 expires 2024-09-17T00:08:07.548547+0000 2024-09-16T00:17:14.434 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6800/122116489 expires 2024-09-17T00:08:07.548547+0000 2024-09-16T00:17:14.434 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6800/961233150 expires 2024-09-17T00:07:43.600446+0000 2024-09-16T00:17:14.434 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/1160344784 expires 2024-09-17T00:07:43.600446+0000 2024-09-16T00:17:14.434 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6801/961233150 expires 2024-09-17T00:07:43.600446+0000 2024-09-16T00:17:14.434 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/2039144880 expires 2024-09-17T00:07:43.600446+0000 2024-09-16T00:17:14.435 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/3586056783 expires 2024-09-17T00:07:43.600446+0000 2024-09-16T00:17:14.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.431+0000 7faf690f4640 1 -- 172.21.15.33:0/1220225688 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faf500a3270 msgr2=0x7faf500a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:14.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.431+0000 7faf690f4640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faf500a3270 0x7faf500a5730 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0058c0 tx=0x7faf4c005d30 comp rx=0 tx=0).stop 2024-09-16T00:17:14.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.431+0000 7faf690f4640 1 -- 172.21.15.33:0/1220225688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf64104fd0 msgr2=0x7faf64198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:14.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.431+0000 7faf690f4640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf64104fd0 0x7faf64198430 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7faf5400d770 tx=0x7faf5400dc30 comp rx=0 tx=0).stop 2024-09-16T00:17:14.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.431+0000 7faf690f4640 1 -- 172.21.15.33:0/1220225688 shutdown_connections 2024-09-16T00:17:14.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.431+0000 7faf690f4640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faf500a3270 0x7faf500a5730 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:14.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.431+0000 7faf690f4640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf64107900 0x7faf6419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:14.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.431+0000 7faf690f4640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf64104fd0 0x7faf64198430 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:14.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.431+0000 7faf690f4640 1 --2- 172.21.15.33:0/1220225688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf641026a0 0x7faf64197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:14.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.431+0000 7faf690f4640 1 -- 172.21.15.33:0/1220225688 >> 172.21.15.33:0/1220225688 conn(0x7faf640fc2a0 msgr2=0x7faf64102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:14.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.431+0000 7faf690f4640 1 -- 172.21.15.33:0/1220225688 shutdown_connections 2024-09-16T00:17:14.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:14.431+0000 7faf690f4640 1 -- 172.21.15.33:0/1220225688 wait complete. 2024-09-16T00:17:14.797 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:14 smithi033 ceph-mon[31243]: pgmap v353: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 6 op/s 2024-09-16T00:17:14.797 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:14 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1220225688' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-16T00:17:14.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:14 smithi083 ceph-mon[35242]: pgmap v353: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 6 op/s 2024-09-16T00:17:14.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:14 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1220225688' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-16T00:17:14.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:14 smithi060 ceph-mon[35185]: pgmap v353: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 6 op/s 2024-09-16T00:17:14.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:14 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1220225688' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-16T00:17:15.201 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-16T00:17:15.216 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:17:15.480 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:15.860 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:15 smithi033 ceph-mon[31243]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-16T00:17:15.860 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:15 smithi033 ceph-mon[31243]: Cluster is now healthy 2024-09-16T00:17:15.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.941+0000 7f781cc29640 1 -- 172.21.15.33:0/4172607240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7818102790 msgr2=0x7f78181067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:15.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.941+0000 7f781cc29640 1 --2- 172.21.15.33:0/4172607240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7818102790 0x7f78181067e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f780c009900 tx=0x7f780c02a410 comp rx=0 tx=0).stop 2024-09-16T00:17:15.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.941+0000 7f781cc29640 1 -- 172.21.15.33:0/4172607240 shutdown_connections 2024-09-16T00:17:15.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.941+0000 7f781cc29640 1 --2- 172.21.15.33:0/4172607240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7818106f10 0x7f781810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:15.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.941+0000 7f781cc29640 1 --2- 172.21.15.33:0/4172607240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7818102790 0x7f78181067e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:15.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.941+0000 7f781cc29640 1 --2- 172.21.15.33:0/4172607240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7818101de0 0x7f78181021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:15.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.941+0000 7f781cc29640 1 -- 172.21.15.33:0/4172607240 >> 172.21.15.33:0/4172607240 conn(0x7f78180fd8c0 msgr2=0x7f78180ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:15.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.941+0000 7f781cc29640 1 -- 172.21.15.33:0/4172607240 shutdown_connections 2024-09-16T00:17:15.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.941+0000 7f781cc29640 1 -- 172.21.15.33:0/4172607240 wait complete. 2024-09-16T00:17:15.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.942+0000 7f781cc29640 1 Processor -- start 2024-09-16T00:17:15.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.942+0000 7f781cc29640 1 -- start start 2024-09-16T00:17:15.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.942+0000 7f781cc29640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7818101de0 0x7f78181984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:15.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.942+0000 7f781cc29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7818102790 0x7f78181989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:15.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.942+0000 7f781cc29640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7818106f10 0x7f781819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.942+0000 7f781cc29640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7818111fb0 con 0x7f7818101de0 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.942+0000 7f781cc29640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7818111e30 con 0x7f7818106f10 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.942+0000 7f781cc29640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7818112130 con 0x7f7818102790 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f781659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7818101de0 0x7f78181984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f781659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7818101de0 0x7f78181984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33280/0 (socket says 172.21.15.33:33280) 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f781659c640 1 -- 172.21.15.33:0/3665136918 learned_addr learned my addr 172.21.15.33:0/3665136918 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f7815d9b640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7818102790 0x7f78181989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f7816d9d640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7818106f10 0x7f781819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f781659c640 1 -- 172.21.15.33:0/3665136918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7818102790 msgr2=0x7f78181989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f781659c640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7818102790 0x7f78181989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f781659c640 1 -- 172.21.15.33:0/3665136918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7818106f10 msgr2=0x7f781819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f781659c640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7818106f10 0x7f781819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f781659c640 1 -- 172.21.15.33:0/3665136918 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f781819d460 con 0x7f7818101de0 2024-09-16T00:17:15.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f781659c640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7818101de0 0x7f78181984b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f780800d770 tx=0x7f780800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:15.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f78077fe640 1 -- 172.21.15.33:0/3665136918 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7808017070 con 0x7f7818101de0 2024-09-16T00:17:15.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f78077fe640 1 -- 172.21.15.33:0/3665136918 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7808005500 con 0x7f7818101de0 2024-09-16T00:17:15.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f78077fe640 1 -- 172.21.15.33:0/3665136918 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7808004e00 con 0x7f7818101de0 2024-09-16T00:17:15.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f781cc29640 1 -- 172.21.15.33:0/3665136918 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f781819d750 con 0x7f7818101de0 2024-09-16T00:17:15.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.943+0000 7f781cc29640 1 -- 172.21.15.33:0/3665136918 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78181a5030 con 0x7f7818101de0 2024-09-16T00:17:15.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.945+0000 7f781cc29640 1 -- 172.21.15.33:0/3665136918 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78181021c0 con 0x7f7818101de0 2024-09-16T00:17:15.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.946+0000 7f78077fe640 1 -- 172.21.15.33:0/3665136918 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7808004fa0 con 0x7f7818101de0 2024-09-16T00:17:15.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.946+0000 7f78077fe640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f78000a32c0 0x7f78000a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:15.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.947+0000 7f7815d9b640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f78000a32c0 0x7f78000a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:15.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.947+0000 7f78077fe640 1 -- 172.21.15.33:0/3665136918 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f78080cf6b0 con 0x7f7818101de0 2024-09-16T00:17:15.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.947+0000 7f7815d9b640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f78000a32c0 0x7f78000a5780 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f780c009230 tx=0x7f780c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:15.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:15.951+0000 7f78077fe640 1 -- 172.21.15.33:0/3665136918 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f780809b900 con 0x7f7818101de0 2024-09-16T00:17:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.163+0000 7f781cc29640 1 -- 172.21.15.33:0/3665136918 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f781810a1b0 con 0x7f7818101de0 2024-09-16T00:17:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.164+0000 7f78077fe640 1 -- 172.21.15.33:0/3665136918 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7f780809fc00 con 0x7f7818101de0 2024-09-16T00:17:16.165 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:16.165 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"btime":"2024-09-16T00:15:52:174386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:15:51.912925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:17:16.165 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-09-16T00:17:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.166+0000 7f781cc29640 1 -- 172.21.15.33:0/3665136918 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f78000a32c0 msgr2=0x7f78000a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.166+0000 7f781cc29640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f78000a32c0 0x7f78000a5780 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f780c009230 tx=0x7f780c0058d0 comp rx=0 tx=0).stop 2024-09-16T00:17:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.166+0000 7f781cc29640 1 -- 172.21.15.33:0/3665136918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7818101de0 msgr2=0x7f78181984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.166+0000 7f781cc29640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7818101de0 0x7f78181984b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f780800d770 tx=0x7f780800dc30 comp rx=0 tx=0).stop 2024-09-16T00:17:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.167+0000 7f781cc29640 1 -- 172.21.15.33:0/3665136918 shutdown_connections 2024-09-16T00:17:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.167+0000 7f781cc29640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f78000a32c0 0x7f78000a5780 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.167+0000 7f781cc29640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7818106f10 0x7f781819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.167+0000 7f781cc29640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7818102790 0x7f78181989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.167+0000 7f781cc29640 1 --2- 172.21.15.33:0/3665136918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7818101de0 0x7f78181984b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:16.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.167+0000 7f781cc29640 1 -- 172.21.15.33:0/3665136918 >> 172.21.15.33:0/3665136918 conn(0x7f78180fd8c0 msgr2=0x7f78180feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:16.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.167+0000 7f781cc29640 1 -- 172.21.15.33:0/3665136918 shutdown_connections 2024-09-16T00:17:16.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:16.167+0000 7f781cc29640 1 -- 172.21.15.33:0/3665136918 wait complete. 2024-09-16T00:17:16.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:15 smithi083 ceph-mon[35242]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-16T00:17:16.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:15 smithi083 ceph-mon[35242]: Cluster is now healthy 2024-09-16T00:17:16.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:15 smithi060 ceph-mon[35185]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-16T00:17:16.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:15 smithi060 ceph-mon[35185]: Cluster is now healthy 2024-09-16T00:17:16.870 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:17:16.895 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:16 smithi033 ceph-mon[31243]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-09-16T00:17:16.895 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:16 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3665136918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:17.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:16 smithi083 ceph-mon[35242]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-09-16T00:17:17.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:16 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3665136918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:17.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:16 smithi060 ceph-mon[35185]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-09-16T00:17:17.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:16 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3665136918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:17.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.600+0000 7fc1d73c1640 1 -- 172.21.15.33:0/3620164983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1d0074f40 msgr2=0x7fc1d010de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:17.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.600+0000 7fc1d73c1640 1 --2- 172.21.15.33:0/3620164983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1d0074f40 0x7fc1d010de80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009900 tx=0x7fc1c002a480 comp rx=0 tx=0).stop 2024-09-16T00:17:17.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.600+0000 7fc1d73c1640 1 -- 172.21.15.33:0/3620164983 shutdown_connections 2024-09-16T00:17:17.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.600+0000 7fc1d73c1640 1 --2- 172.21.15.33:0/3620164983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1d010e3c0 0x7fc1d0110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:17.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.600+0000 7fc1d73c1640 1 --2- 172.21.15.33:0/3620164983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1d0074f40 0x7fc1d010de80 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:17.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.600+0000 7fc1d73c1640 1 --2- 172.21.15.33:0/3620164983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1d0074620 0x7fc1d0074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:17.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.600+0000 7fc1d73c1640 1 -- 172.21.15.33:0/3620164983 >> 172.21.15.33:0/3620164983 conn(0x7fc1d00fd8c0 msgr2=0x7fc1d00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:17.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.600+0000 7fc1d73c1640 1 -- 172.21.15.33:0/3620164983 shutdown_connections 2024-09-16T00:17:17.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.601+0000 7fc1d73c1640 1 -- 172.21.15.33:0/3620164983 wait complete. 2024-09-16T00:17:17.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.601+0000 7fc1d73c1640 1 Processor -- start 2024-09-16T00:17:17.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.601+0000 7fc1d73c1640 1 -- start start 2024-09-16T00:17:17.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.601+0000 7fc1d73c1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1d0074620 0x7fc1d019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:17.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.601+0000 7fc1d73c1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1d0074f40 0x7fc1d019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:17.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.601+0000 7fc1d73c1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1d010e3c0 0x7fc1d019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.601+0000 7fc1d73c1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d0114140 con 0x7fc1d010e3c0 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.601+0000 7fc1d73c1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d0113fc0 con 0x7fc1d0074f40 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.601+0000 7fc1d73c1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d01142c0 con 0x7fc1d0074620 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.601+0000 7fc1d595e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1d010e3c0 0x7fc1d019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d595e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1d010e3c0 0x7fc1d019ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33292/0 (socket says 172.21.15.33:33292) 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d595e640 1 -- 172.21.15.33:0/765514993 learned_addr learned my addr 172.21.15.33:0/765514993 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d495c640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1d0074f40 0x7fc1d019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d515d640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1d0074620 0x7fc1d019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d595e640 1 -- 172.21.15.33:0/765514993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1d0074620 msgr2=0x7fc1d019a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d595e640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1d0074620 0x7fc1d019a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d595e640 1 -- 172.21.15.33:0/765514993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1d0074f40 msgr2=0x7fc1d019ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d595e640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1d0074f40 0x7fc1d019ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:17.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d595e640 1 -- 172.21.15.33:0/765514993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d019f5f0 con 0x7fc1d010e3c0 2024-09-16T00:17:17.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d495c640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1d0074f40 0x7fc1d019ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:17.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d595e640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1d010e3c0 0x7fc1d019ef10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800eaa0 tx=0x7fc1c800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:17.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1c67fc640 1 -- 172.21.15.33:0/765514993 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c800cdb0 con 0x7fc1d010e3c0 2024-09-16T00:17:17.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1c67fc640 1 -- 172.21.15.33:0/765514993 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c8005020 con 0x7fc1d010e3c0 2024-09-16T00:17:17.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d73c1640 1 -- 172.21.15.33:0/765514993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d019f8e0 con 0x7fc1d010e3c0 2024-09-16T00:17:17.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1c67fc640 1 -- 172.21.15.33:0/765514993 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c80116b0 con 0x7fc1d010e3c0 2024-09-16T00:17:17.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.602+0000 7fc1d73c1640 1 -- 172.21.15.33:0/765514993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d01a71c0 con 0x7fc1d010e3c0 2024-09-16T00:17:17.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.603+0000 7fc1d73c1640 1 -- 172.21.15.33:0/765514993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1d0074a00 con 0x7fc1d010e3c0 2024-09-16T00:17:17.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.605+0000 7fc1c67fc640 1 -- 172.21.15.33:0/765514993 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc1c8005560 con 0x7fc1d010e3c0 2024-09-16T00:17:17.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.605+0000 7fc1c67fc640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc1bc0a3240 0x7fc1bc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:17.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.605+0000 7fc1c67fc640 1 -- 172.21.15.33:0/765514993 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc1c80d0400 con 0x7fc1d010e3c0 2024-09-16T00:17:17.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.605+0000 7fc1d515d640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc1bc0a3240 0x7fc1bc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:17.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.606+0000 7fc1d515d640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc1bc0a3240 0x7fc1bc0a5700 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009b60 tx=0x7fc1cc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:17.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.608+0000 7fc1c67fc640 1 -- 172.21.15.33:0/765514993 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc1c8016030 con 0x7fc1d010e3c0 2024-09-16T00:17:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.823+0000 7fc1d73c1640 1 -- 172.21.15.33:0/765514993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1d019b9a0 con 0x7fc1d010e3c0 2024-09-16T00:17:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.824+0000 7fc1c67fc640 1 -- 172.21.15.33:0/765514993 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7fc1d019b9a0 con 0x7fc1d010e3c0 2024-09-16T00:17:17.824 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:17.825 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"btime":"2024-09-16T00:15:52:174386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:15:51.912925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:17:17.825 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-09-16T00:17:17.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.827+0000 7fc1d73c1640 1 -- 172.21.15.33:0/765514993 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc1bc0a3240 msgr2=0x7fc1bc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:17.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.827+0000 7fc1d73c1640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc1bc0a3240 0x7fc1bc0a5700 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009b60 tx=0x7fc1cc005f90 comp rx=0 tx=0).stop 2024-09-16T00:17:17.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.827+0000 7fc1d73c1640 1 -- 172.21.15.33:0/765514993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1d010e3c0 msgr2=0x7fc1d019ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:17.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.827+0000 7fc1d73c1640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1d010e3c0 0x7fc1d019ef10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800eaa0 tx=0x7fc1c800ef60 comp rx=0 tx=0).stop 2024-09-16T00:17:17.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.827+0000 7fc1d73c1640 1 -- 172.21.15.33:0/765514993 shutdown_connections 2024-09-16T00:17:17.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.827+0000 7fc1d73c1640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc1bc0a3240 0x7fc1bc0a5700 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:17.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.827+0000 7fc1d73c1640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1d010e3c0 0x7fc1d019ef10 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:17.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.827+0000 7fc1d73c1640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1d0074f40 0x7fc1d019ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:17.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.827+0000 7fc1d73c1640 1 --2- 172.21.15.33:0/765514993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1d0074620 0x7fc1d019a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:17.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.827+0000 7fc1d73c1640 1 -- 172.21.15.33:0/765514993 >> 172.21.15.33:0/765514993 conn(0x7fc1d00fd8c0 msgr2=0x7fc1d00fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:17.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.827+0000 7fc1d73c1640 1 -- 172.21.15.33:0/765514993 shutdown_connections 2024-09-16T00:17:17.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:17.827+0000 7fc1d73c1640 1 -- 172.21.15.33:0/765514993 wait complete. 2024-09-16T00:17:18.494 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:17:18.802 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:19.032 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:18 smithi060 ceph-mon[35185]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 225 B/s wr, 5 op/s 2024-09-16T00:17:19.033 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:18 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/765514993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:19.047 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:18 smithi033 ceph-mon[31243]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 225 B/s wr, 5 op/s 2024-09-16T00:17:19.047 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:18 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/765514993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:19.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:18 smithi083 ceph-mon[35242]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 225 B/s wr, 5 op/s 2024-09-16T00:17:19.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:18 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/765514993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.279+0000 7fc6c729a640 1 -- 172.21.15.33:0/3437130196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6c0106f20 msgr2=0x7fc6c010eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.279+0000 7fc6c729a640 1 --2- 172.21.15.33:0/3437130196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6c0106f20 0x7fc6c010eaa0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0009ca0 tx=0x7fc6b002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:17:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.279+0000 7fc6c729a640 1 -- 172.21.15.33:0/3437130196 shutdown_connections 2024-09-16T00:17:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.279+0000 7fc6c729a640 1 --2- 172.21.15.33:0/3437130196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6c0106f20 0x7fc6c010eaa0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.279+0000 7fc6c729a640 1 --2- 172.21.15.33:0/3437130196 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6c01027a0 0x7fc6c01067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.279+0000 7fc6c729a640 1 --2- 172.21.15.33:0/3437130196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0101df0 0x7fc6c01021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.279+0000 7fc6c729a640 1 -- 172.21.15.33:0/3437130196 >> 172.21.15.33:0/3437130196 conn(0x7fc6c00fd8f0 msgr2=0x7fc6c00ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.279+0000 7fc6c729a640 1 -- 172.21.15.33:0/3437130196 shutdown_connections 2024-09-16T00:17:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.279+0000 7fc6c729a640 1 -- 172.21.15.33:0/3437130196 wait complete. 2024-09-16T00:17:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c729a640 1 Processor -- start 2024-09-16T00:17:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c729a640 1 -- start start 2024-09-16T00:17:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c729a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0101df0 0x7fc6c01983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c729a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6c01027a0 0x7fc6c01988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c729a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6c0106f20 0x7fc6c019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c729a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c0111fa0 con 0x7fc6c01027a0 2024-09-16T00:17:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c729a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c0111e20 con 0x7fc6c0101df0 2024-09-16T00:17:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c729a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c0112120 con 0x7fc6c0106f20 2024-09-16T00:17:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c4835640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6c01027a0 0x7fc6c01988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c4835640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6c01027a0 0x7fc6c01988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33318/0 (socket says 172.21.15.33:33318) 2024-09-16T00:17:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c4835640 1 -- 172.21.15.33:0/327354715 learned_addr learned my addr 172.21.15.33:0/327354715 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c5036640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0101df0 0x7fc6c01983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c5837640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6c0106f20 0x7fc6c019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c4835640 1 -- 172.21.15.33:0/327354715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6c0106f20 msgr2=0x7fc6c019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c4835640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6c0106f20 0x7fc6c019cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c4835640 1 -- 172.21.15.33:0/327354715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0101df0 msgr2=0x7fc6c01983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c4835640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0101df0 0x7fc6c01983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.280+0000 7fc6c4835640 1 -- 172.21.15.33:0/327354715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c019d360 con 0x7fc6c01027a0 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.281+0000 7fc6c4835640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6c01027a0 0x7fc6c01988f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400c920 tx=0x7fc6b400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.281+0000 7fc6be7fc640 1 -- 172.21.15.33:0/327354715 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4007bb0 con 0x7fc6c01027a0 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.281+0000 7fc6c729a640 1 -- 172.21.15.33:0/327354715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c019d650 con 0x7fc6c01027a0 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.281+0000 7fc6be7fc640 1 -- 172.21.15.33:0/327354715 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6b4007d50 con 0x7fc6c01027a0 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.281+0000 7fc6be7fc640 1 -- 172.21.15.33:0/327354715 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b40187d0 con 0x7fc6c01027a0 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.281+0000 7fc6c729a640 1 -- 172.21.15.33:0/327354715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c01a4f30 con 0x7fc6c01027a0 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.282+0000 7fc6c729a640 1 -- 172.21.15.33:0/327354715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c01021d0 con 0x7fc6c01027a0 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.283+0000 7fc6be7fc640 1 -- 172.21.15.33:0/327354715 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc6b400f450 con 0x7fc6c01027a0 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.284+0000 7fc6be7fc640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6a80a3240 0x7fc6a80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.284+0000 7fc6c5036640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6a80a3240 0x7fc6a80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.284+0000 7fc6be7fc640 1 -- 172.21.15.33:0/327354715 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc6b40cfe20 con 0x7fc6c01027a0 2024-09-16T00:17:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.284+0000 7fc6c5036640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6a80a3240 0x7fc6a80a5700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8009990 tx=0x7fc6b8009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.289+0000 7fc6be7fc640 1 -- 172.21.15.33:0/327354715 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc6b4014030 con 0x7fc6c01027a0 2024-09-16T00:17:19.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.467+0000 7fc6c729a640 1 -- 172.21.15.33:0/327354715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6c0109b70 con 0x7fc6c01027a0 2024-09-16T00:17:19.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.468+0000 7fc6be7fc640 1 -- 172.21.15.33:0/327354715 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25076 (secure 0 0 0) 0x7fc6b409c6c0 con 0x7fc6c01027a0 2024-09-16T00:17:19.469 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:19.469 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":90,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:17:09.569270+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2e","mappings":[{"from":1,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:17:19.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.470+0000 7fc6c729a640 1 -- 172.21.15.33:0/327354715 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6a80a3240 msgr2=0x7fc6a80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:19.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.470+0000 7fc6c729a640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6a80a3240 0x7fc6a80a5700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8009990 tx=0x7fc6b8009110 comp rx=0 tx=0).stop 2024-09-16T00:17:19.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.470+0000 7fc6c729a640 1 -- 172.21.15.33:0/327354715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6c01027a0 msgr2=0x7fc6c01988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:19.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.470+0000 7fc6c729a640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6c01027a0 0x7fc6c01988f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400c920 tx=0x7fc6b400cd20 comp rx=0 tx=0).stop 2024-09-16T00:17:19.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.470+0000 7fc6c729a640 1 -- 172.21.15.33:0/327354715 shutdown_connections 2024-09-16T00:17:19.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.470+0000 7fc6c729a640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6a80a3240 0x7fc6a80a5700 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:19.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.470+0000 7fc6c729a640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6c0106f20 0x7fc6c019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:19.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.470+0000 7fc6c729a640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6c01027a0 0x7fc6c01988f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:19.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.470+0000 7fc6c729a640 1 --2- 172.21.15.33:0/327354715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0101df0 0x7fc6c01983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:19.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.470+0000 7fc6c729a640 1 -- 172.21.15.33:0/327354715 >> 172.21.15.33:0/327354715 conn(0x7fc6c00fd8f0 msgr2=0x7fc6c00fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:19.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.470+0000 7fc6c729a640 1 -- 172.21.15.33:0/327354715 shutdown_connections 2024-09-16T00:17:19.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:19.470+0000 7fc6c729a640 1 -- 172.21.15.33:0/327354715 wait complete. 2024-09-16T00:17:20.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:19 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/327354715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:17:20.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:19 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/327354715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:17:20.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-16T00:15:07.246355+0000', 'modified': '2024-09-16T00:15:51.912925+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24494, 'mds_1': 24493, 'mds_2': 24472}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.60:6835/1002409397', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6834', 'nonce': 1002409397}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 1002409397}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.60:6837/452012251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6836', 'nonce': 452012251}, {'type': 'v1', 'addr': '172.21.15.60:6837', 'nonce': 452012251}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.83:6835/2667354509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 2667354509}, {'type': 'v1', 'addr': '172.21.15.83:6835', 'nonce': 2667354509}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.83:6837/4244918448', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6836', 'nonce': 4244918448}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 4244918448}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.83:6839/3202868586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6838', 'nonce': 3202868586}, {'type': 'v1', 'addr': '172.21.15.83:6839', 'nonce': 3202868586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.60:6839/1222932045', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6838', 'nonce': 1222932045}, {'type': 'v1', 'addr': '172.21.15.60:6839', 'nonce': 1222932045}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24494, 'qdb_cluster': [24472, 24493, 24494]} 2024-09-16T00:17:20.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-16T00:17:20.212 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.c status 2024-09-16T00:17:20.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:19 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/327354715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:17:20.472 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:20.878 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:20 smithi033 ceph-mon[31243]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 202 B/s wr, 6 op/s 2024-09-16T00:17:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.962+0000 7f6b1cc05640 1 -- 172.21.15.33:0/3538903130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18100a80 msgr2=0x7f6b18100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:20.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.962+0000 7f6b1cc05640 1 --2- 172.21.15.33:0/3538903130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18100a80 0x7f6b18100ee0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6b00009930 tx=0x7f6b0002a420 comp rx=0 tx=0).stop 2024-09-16T00:17:20.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.962+0000 7f6b1cc05640 1 -- 172.21.15.33:0/3538903130 shutdown_connections 2024-09-16T00:17:20.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.962+0000 7f6b1cc05640 1 --2- 172.21.15.33:0/3538903130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18101420 0x7f6b1810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.962+0000 7f6b1cc05640 1 --2- 172.21.15.33:0/3538903130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18100a80 0x7f6b18100ee0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.962+0000 7f6b1cc05640 1 --2- 172.21.15.33:0/3538903130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18106a80 0x7f6b18106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.962+0000 7f6b1cc05640 1 -- 172.21.15.33:0/3538903130 >> 172.21.15.33:0/3538903130 conn(0x7f6b180fc7a0 msgr2=0x7f6b180febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:20.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.962+0000 7f6b1cc05640 1 -- 172.21.15.33:0/3538903130 shutdown_connections 2024-09-16T00:17:20.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.962+0000 7f6b1cc05640 1 -- 172.21.15.33:0/3538903130 wait complete. 2024-09-16T00:17:20.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b1cc05640 1 Processor -- start 2024-09-16T00:17:20.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b1cc05640 1 -- start start 2024-09-16T00:17:20.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b1cc05640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18100a80 0x7f6b18198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:20.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b1cc05640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18101420 0x7f6b181989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:20.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b1cc05640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18106a80 0x7f6b1819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:20.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b1cc05640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6b18110e80 con 0x7f6b18106a80 2024-09-16T00:17:20.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b1cc05640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6b18110d00 con 0x7f6b18100a80 2024-09-16T00:17:20.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b1cc05640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6b18111000 con 0x7f6b18101420 2024-09-16T00:17:20.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b16d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18106a80 0x7f6b1819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:20.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b16d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18106a80 0x7f6b1819cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34742/0 (socket says 172.21.15.33:34742) 2024-09-16T00:17:20.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b16d9d640 1 -- 172.21.15.33:0/1012375983 learned_addr learned my addr 172.21.15.33:0/1012375983 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:20.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b1659c640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18100a80 0x7f6b18198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:20.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.963+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18101420 0x7f6b181989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:20.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.964+0000 7f6b16d9d640 1 -- 172.21.15.33:0/1012375983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18101420 msgr2=0x7f6b181989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.964+0000 7f6b16d9d640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18101420 0x7f6b181989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.964+0000 7f6b16d9d640 1 -- 172.21.15.33:0/1012375983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18100a80 msgr2=0x7f6b18198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.964+0000 7f6b16d9d640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18100a80 0x7f6b18198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.964+0000 7f6b16d9d640 1 -- 172.21.15.33:0/1012375983 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1819d430 con 0x7f6b18106a80 2024-09-16T00:17:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.964+0000 7f6b16d9d640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18106a80 0x7f6b1819cd50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000eaa0 tx=0x7f6b1000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.964+0000 7f6b0f7fe640 1 -- 172.21.15.33:0/1012375983 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1000cdb0 con 0x7f6b18106a80 2024-09-16T00:17:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.964+0000 7f6b1cc05640 1 -- 172.21.15.33:0/1012375983 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b1819d720 con 0x7f6b18106a80 2024-09-16T00:17:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.964+0000 7f6b0f7fe640 1 -- 172.21.15.33:0/1012375983 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b10005020 con 0x7f6b18106a80 2024-09-16T00:17:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.964+0000 7f6b0f7fe640 1 -- 172.21.15.33:0/1012375983 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b100116d0 con 0x7f6b18106a80 2024-09-16T00:17:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.964+0000 7f6b1cc05640 1 -- 172.21.15.33:0/1012375983 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b181a5000 con 0x7f6b18106a80 2024-09-16T00:17:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.966+0000 7f6b0f7fe640 1 -- 172.21.15.33:0/1012375983 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f6b100054d0 con 0x7f6b18106a80 2024-09-16T00:17:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.966+0000 7f6b1efde740 1 Processor -- start 2024-09-16T00:17:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b1efde740 1 -- start start 2024-09-16T00:17:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b1efde740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc308b0 0x561bdfc30c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b1efde740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 0x561bdfc3fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b1efde740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc3ffc0 0x561bdfc423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b1efde740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561bdfb97090 con 0x561bdfc311d0 2024-09-16T00:17:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b1efde740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561bdfb3ea70 con 0x561bdfc308b0 2024-09-16T00:17:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b1efde740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561bdfb93050 con 0x561bdfc3ffc0 2024-09-16T00:17:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b0f7fe640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b040a3240 0x7f6b040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b1659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 0x561bdfc3fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b1659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 0x561bdfc3fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34744/0 (socket says 172.21.15.33:34744) 2024-09-16T00:17:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b1659c640 1 -- 172.21.15.33:0/825430524 learned_addr learned my addr 172.21.15.33:0/825430524 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b16d9d640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b040a3240 0x7f6b040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/825430524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc3ffc0 0x561bdfc423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b0f7fe640 1 -- 172.21.15.33:0/1012375983 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6b100cff60 con 0x7f6b18106a80 2024-09-16T00:17:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.967+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/825430524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc308b0 0x561bdfc30c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.968+0000 7f6b1659c640 1 -- 172.21.15.33:0/825430524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc3ffc0 msgr2=0x561bdfc423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.968+0000 7f6b1659c640 1 --2- 172.21.15.33:0/825430524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc3ffc0 0x561bdfc423b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.968+0000 7f6b1659c640 1 -- 172.21.15.33:0/825430524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc308b0 msgr2=0x561bdfc30c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.968+0000 7f6b1659c640 1 --2- 172.21.15.33:0/825430524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc308b0 0x561bdfc30c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.968+0000 7f6b1659c640 1 -- 172.21.15.33:0/825430524 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bdfb4b130 con 0x561bdfc311d0 2024-09-16T00:17:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.968+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/825430524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc308b0 0x561bdfc30c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.968+0000 7f6b1659c640 1 --2- 172.21.15.33:0/825430524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 0x561bdfc3fa80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800bc90 tx=0x7f6b080107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869eef73be7ba784 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.968+0000 7f6b16d9d640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b040a3240 0x7f6b040a5700 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b100a47e0 tx=0x7f6b10066b90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.968+0000 7f6b0cff9640 1 -- 172.21.15.33:0/825430524 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0801c480 con 0x561bdfc311d0 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.968+0000 7f6b0cff9640 1 -- 172.21.15.33:0/825430524 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b0801cb00 con 0x561bdfc311d0 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.968+0000 7f6b0cff9640 1 -- 172.21.15.33:0/825430524 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b08011c60 con 0x561bdfc311d0 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.969+0000 7f6b1efde740 1 -- 172.21.15.33:0/825430524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 msgr2=0x561bdfc3fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.969+0000 7f6b1efde740 1 --2- 172.21.15.33:0/825430524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 0x561bdfc3fa80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800bc90 tx=0x7f6b080107a0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.969+0000 7f6b1efde740 1 -- 172.21.15.33:0/825430524 shutdown_connections 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.969+0000 7f6b1efde740 1 --2- 172.21.15.33:0/825430524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc3ffc0 0x561bdfc423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.969+0000 7f6b1efde740 1 --2- 172.21.15.33:0/825430524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 0x561bdfc3fa80 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.969+0000 7f6b1efde740 1 --2- 172.21.15.33:0/825430524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc308b0 0x561bdfc30c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.969+0000 7f6b1efde740 1 -- 172.21.15.33:0/825430524 >> 172.21.15.33:0/825430524 conn(0x561bdfc2cf60 msgr2=0x561bdfc2d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.969+0000 7f6b1efde740 1 -- 172.21.15.33:0/825430524 shutdown_connections 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.969+0000 7f6b1efde740 1 -- 172.21.15.33:0/825430524 wait complete. 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.969+0000 7f6b1efde740 1 -- start start 2024-09-16T00:17:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1efde740 1 Processor -- start 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1efde740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc308b0 0x561bdfc3f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1efde740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 0x561bdfc3cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1efde740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc3ffc0 0x561bdfc3d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1efde740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561bdfc42b90 con 0x561bdfc311d0 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1efde740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561bdfb51f20 con 0x561bdfc3ffc0 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1efde740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561bdfc42d10 con 0x561bdfc308b0 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1efde740 10 client.0 _ll_register_callbacks cb 0x561bdfb1b030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 0x561bdfc3cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 0x561bdfc3cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34758/0 (socket says 172.21.15.33:34758) 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1659c640 1 -- 172.21.15.33:0/1128340448 learned_addr learned my addr 172.21.15.33:0/1128340448 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc308b0 0x561bdfc3f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc3ffc0 0x561bdfc3d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1659c640 1 -- 172.21.15.33:0/1128340448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc308b0 msgr2=0x561bdfc3f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1659c640 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc308b0 0x561bdfc3f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1659c640 1 -- 172.21.15.33:0/1128340448 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc3ffc0 msgr2=0x561bdfc3d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1659c640 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc3ffc0 0x561bdfc3d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1659c640 1 -- 172.21.15.33:0/1128340448 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bdf9082b0 con 0x561bdfc311d0 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc308b0 0x561bdfc3f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6b1659c640 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 0x561bdfc3cc20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6b08010ca0 tx=0x7f6b08011650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709c2591f45e4df5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.970+0000 7f6afdffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6afdffb640 1 -- 172.21.15.33:0/1128340448 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b08004020 con 0x561bdfc311d0 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6b1efde740 1 -- 172.21.15.33:0/1128340448 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b0801cb00 con 0x561bdfc311d0 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6afdffb640 1 -- 172.21.15.33:0/1128340448 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6b080041c0 con 0x561bdfc311d0 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6afdffb640 1 -- 172.21.15.33:0/1128340448 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6b08030050 con 0x561bdfc311d0 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6b1efde740 10 client.15183 fetch_fsmap learned FSMap version 25 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6b1efde740 1 -- 172.21.15.33:0/1128340448 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561bdfc4e520 con 0x561bdfc311d0 2024-09-16T00:17:20.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6afdffb640 1 -- 172.21.15.33:0/1128340448 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b0801cb00 con 0x561bdfc311d0 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6afdffb640 1 -- 172.21.15.33:0/1128340448 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f6b080054b0 con 0x561bdfc311d0 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6b1efde740 10 client.15183 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6b1efde740 10 client.15183 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6b1efde740 20 client.15183 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561bdfc508e0 0x561bdfc51d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6b1efde740 4 client.15183 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.971+0000 7f6b1efde740 1 -- 172.21.15.33:0/1128340448 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bdfba4600 con 0x561bdfc508e0 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.972+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561bdfc508e0 0x561bdfc51d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.972+0000 7f6afdffb640 10 client.15183 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.972+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561bdfc508e0 0x561bdfc51d50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9e547d3680818334 server_cookie=d8cdb3a76e6813cb in_seq=0 out_seq=0 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6afdffb640 1 -- 172.21.15.33:0/1128340448 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561bdfba4600 con 0x561bdfc508e0 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6afdffb640 10 client.15183 handle_command_reply: tid=0 2024-09-16T00:17:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6b1efde740 1 client.15183 shutdown 2024-09-16T00:17:20.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6b1efde740 1 -- 172.21.15.33:0/1128340448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 msgr2=0x561bdfc3cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:20.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 0x561bdfc3cc20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6b08010ca0 tx=0x7f6b08011650 comp rx=0 tx=0).stop 2024-09-16T00:17:20.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6b1efde740 1 -- 172.21.15.33:0/1128340448 shutdown_connections 2024-09-16T00:17:20.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561bdfc508e0 0x561bdfc51d50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc3ffc0 0x561bdfc3d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc311d0 0x561bdfc3cc20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6afdffb640 0 client.15183 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:17:20.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1128340448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc308b0 0x561bdfc3f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:20.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6b1efde740 1 -- 172.21.15.33:0/1128340448 >> 172.21.15.33:0/1128340448 conn(0x561bdfc2cf60 msgr2=0x561bdfc32cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:20.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6b1efde740 1 -- 172.21.15.33:0/1128340448 shutdown_connections 2024-09-16T00:17:20.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6b1efde740 1 -- 172.21.15.33:0/1128340448 wait complete. 2024-09-16T00:17:20.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.973+0000 7f6b1efde740 20 client.15183 trim_cache size 0 max 16384 2024-09-16T00:17:21.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.991+0000 7f6b1efde740 1 Processor -- start 2024-09-16T00:17:21.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.991+0000 7f6b1efde740 1 -- start start 2024-09-16T00:17:21.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.991+0000 7f6b1efde740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 0x561bdfc50fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:21.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.991+0000 7f6b1efde740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc311d0 0x561bdfc39e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:21.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.991+0000 7f6b1efde740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc3ffc0 0x561bdfc3a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:21.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.991+0000 7f6b1efde740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561bdfc50090 con 0x561bdfc308b0 2024-09-16T00:17:21.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.991+0000 7f6b1efde740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561bdfc50760 con 0x561bdfc3ffc0 2024-09-16T00:17:21.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.991+0000 7f6b1efde740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561bdfc50e30 con 0x561bdfc311d0 2024-09-16T00:17:21.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.991+0000 7f6b0ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 0x561bdfc50fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:21.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.991+0000 7f6b0ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 0x561bdfc50fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34774/0 (socket says 172.21.15.33:34774) 2024-09-16T00:17:21.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.991+0000 7f6b0ffff640 1 -- 172.21.15.33:0/1003929020 learned_addr learned my addr 172.21.15.33:0/1003929020 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:21.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.991+0000 7f6b1659c640 1 --2- 172.21.15.33:0/1003929020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc311d0 0x561bdfc39e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:21.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.992+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1003929020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc3ffc0 0x561bdfc3a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:21.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.992+0000 7f6b0ffff640 1 -- 172.21.15.33:0/1003929020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc311d0 msgr2=0x561bdfc39e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:21.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.992+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1003929020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc311d0 0x561bdfc39e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.992+0000 7f6b0ffff640 1 -- 172.21.15.33:0/1003929020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc3ffc0 msgr2=0x561bdfc3a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:21.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.992+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1003929020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc3ffc0 0x561bdfc3a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.992+0000 7f6b0ffff640 1 -- 172.21.15.33:0/1003929020 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bdfc2ca00 con 0x561bdfc308b0 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.992+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1003929020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 0x561bdfc50fb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6b00031550 tx=0x7f6b000058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac2f4959f89873 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.992+0000 7f6afffff640 1 -- 172.21.15.33:0/1003929020 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b00043070 con 0x561bdfc308b0 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.992+0000 7f6afffff640 1 -- 172.21.15.33:0/1003929020 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b00005020 con 0x561bdfc308b0 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.992+0000 7f6afffff640 1 -- 172.21.15.33:0/1003929020 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b00046040 con 0x561bdfc308b0 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 -- 172.21.15.33:0/1003929020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 msgr2=0x561bdfc50fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1003929020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 0x561bdfc50fb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6b00031550 tx=0x7f6b000058d0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 -- 172.21.15.33:0/1003929020 shutdown_connections 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1003929020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc3ffc0 0x561bdfc3a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1003929020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc311d0 0x561bdfc39e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1003929020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 0x561bdfc50fb0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 -- 172.21.15.33:0/1003929020 >> 172.21.15.33:0/1003929020 conn(0x561bdfc2cf60 msgr2=0x561bdfc38240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 -- 172.21.15.33:0/1003929020 shutdown_connections 2024-09-16T00:17:21.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 -- 172.21.15.33:0/1003929020 wait complete. 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 -- start start 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 Processor -- start 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 0x561bdfc53d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc311d0 0x561bdfc54240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc3ffc0 0x561bdfc54780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561bdfc51bd0 con 0x561bdfc308b0 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561bdfc51500 con 0x561bdfc311d0 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561bdfc522a0 con 0x561bdfc3ffc0 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.993+0000 7f6b1efde740 10 client.0 _ll_register_callbacks cb 0x7f6b0800bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b0ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 0x561bdfc53d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b0ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 0x561bdfc53d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34786/0 (socket says 172.21.15.33:34786) 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b0ffff640 1 -- 172.21.15.33:0/1852072867 learned_addr learned my addr 172.21.15.33:0/1852072867 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:21.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc3ffc0 0x561bdfc54780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:21.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b1659c640 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc311d0 0x561bdfc54240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:21.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b0ffff640 1 -- 172.21.15.33:0/1852072867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc3ffc0 msgr2=0x561bdfc54780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:21.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc3ffc0 0x561bdfc54780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b0ffff640 1 -- 172.21.15.33:0/1852072867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc311d0 msgr2=0x561bdfc54240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:21.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc311d0 0x561bdfc54240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b0ffff640 1 -- 172.21.15.33:0/1852072867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bdfc584f0 con 0x561bdfc308b0 2024-09-16T00:17:21.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b0ffff640 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 0x561bdfc53d00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6b0002a460 tx=0x7f6b00005350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb811a508405f15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:21.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6afdffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:21.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6afdffb640 1 -- 172.21.15.33:0/1852072867 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b00043070 con 0x561bdfc308b0 2024-09-16T00:17:21.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6afdffb640 1 -- 172.21.15.33:0/1852072867 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6b00005440 con 0x561bdfc308b0 2024-09-16T00:17:21.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b1efde740 1 -- 172.21.15.33:0/1852072867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b00005020 con 0x561bdfc308b0 2024-09-16T00:17:21.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6afdffb640 1 -- 172.21.15.33:0/1852072867 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6b00035040 con 0x561bdfc308b0 2024-09-16T00:17:21.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b1efde740 10 client.15189 fetch_fsmap learned FSMap version 25 2024-09-16T00:17:21.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.994+0000 7f6b1efde740 1 -- 172.21.15.33:0/1852072867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561bdfcbb2a0 con 0x561bdfc308b0 2024-09-16T00:17:21.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.995+0000 7f6afdffb640 1 -- 172.21.15.33:0/1852072867 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b00005020 con 0x561bdfc308b0 2024-09-16T00:17:21.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.995+0000 7f6afdffb640 1 -- 172.21.15.33:0/1852072867 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f6b00059030 con 0x561bdfc308b0 2024-09-16T00:17:21.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.995+0000 7f6b1efde740 10 client.15189 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T00:17:21.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.995+0000 7f6b1efde740 10 client.15189 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T00:17:21.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.995+0000 7f6b1efde740 20 client.15189 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:21.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.995+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561bdfcbfa70 0x561bdfcbe780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:21.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.995+0000 7f6b1efde740 4 client.15189 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T00:17:21.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.995+0000 7f6b1efde740 1 -- 172.21.15.33:0/1852072867 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "status"}) -- 0x561bdfb9eb10 con 0x561bdfcbfa70 2024-09-16T00:17:21.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.995+0000 7f6b1659c640 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561bdfcbfa70 0x561bdfcbe780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:21.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1659c640 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561bdfcbfa70 0x561bdfcbe780 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ea21010111118f95 server_cookie=3f3614d4724b8710 in_seq=0 out_seq=0 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6afdffb640 10 client.15189 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6afdffb640 1 -- 172.21.15.33:0/1852072867 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561bdfb9eb10 con 0x561bdfcbfa70 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6afdffb640 10 client.15189 handle_command_reply: tid=0 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1efde740 1 client.15189 shutdown 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1efde740 1 -- 172.21.15.33:0/1852072867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 msgr2=0x561bdfc53d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 0x561bdfc53d00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6b0002a460 tx=0x7f6b00005350 comp rx=0 tx=0).stop 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1efde740 1 -- 172.21.15.33:0/1852072867 shutdown_connections 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561bdfcbfa70 0x561bdfcbe780 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6afdffb640 0 client.15189 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561bdfc3ffc0 0x561bdfc54780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561bdfc311d0 0x561bdfc54240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1efde740 1 --2- 172.21.15.33:0/1852072867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561bdfc308b0 0x561bdfc53d00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1efde740 1 -- 172.21.15.33:0/1852072867 >> 172.21.15.33:0/1852072867 conn(0x561bdfc2cf60 msgr2=0x561bdfc2b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:21.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1efde740 1 -- 172.21.15.33:0/1852072867 shutdown_connections 2024-09-16T00:17:21.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1efde740 1 -- 172.21.15.33:0/1852072867 wait complete. 2024-09-16T00:17:21.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.996+0000 7f6b1efde740 20 client.15189 trim_cache size 0 max 16384 2024-09-16T00:17:21.010 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:17:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:17:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 2, 2024-09-16T00:17:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24472, 2024-09-16T00:17:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-09-16T00:17:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-09-16T00:17:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-09-16T00:17:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 91.827446238999997, 2024-09-16T00:17:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 25, 2024-09-16T00:17:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 90, 2024-09-16T00:17:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 86, 2024-09-16T00:17:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 159.01512509299999 2024-09-16T00:17:21.011 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:17:21.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.997+0000 7f6b1cc05640 1 -- 172.21.15.33:0/1012375983 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b040a3240 msgr2=0x7f6b040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:21.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.997+0000 7f6b1cc05640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b040a3240 0x7f6b040a5700 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b100a47e0 tx=0x7f6b10066b90 comp rx=0 tx=0).stop 2024-09-16T00:17:21.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.997+0000 7f6b1cc05640 1 -- 172.21.15.33:0/1012375983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18106a80 msgr2=0x7f6b1819cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:21.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.997+0000 7f6b1cc05640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18106a80 0x7f6b1819cd50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000eaa0 tx=0x7f6b1000ef60 comp rx=0 tx=0).stop 2024-09-16T00:17:21.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.997+0000 7f6b1cc05640 1 -- 172.21.15.33:0/1012375983 shutdown_connections 2024-09-16T00:17:21.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.997+0000 7f6b1cc05640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b040a3240 0x7f6b040a5700 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.997+0000 7f6b1cc05640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18106a80 0x7f6b1819cd50 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.997+0000 7f6b1cc05640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18101420 0x7f6b181989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.997+0000 7f6b1cc05640 1 --2- 172.21.15.33:0/1012375983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18100a80 0x7f6b18198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:21.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.997+0000 7f6b1cc05640 1 -- 172.21.15.33:0/1012375983 >> 172.21.15.33:0/1012375983 conn(0x7f6b180fc7a0 msgr2=0x7f6b1806aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:21.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.997+0000 7f6b1cc05640 1 -- 172.21.15.33:0/1012375983 shutdown_connections 2024-09-16T00:17:21.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:20.997+0000 7f6b1cc05640 1 -- 172.21.15.33:0/1012375983 wait complete. 2024-09-16T00:17:21.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:20 smithi083 ceph-mon[35242]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 202 B/s wr, 6 op/s 2024-09-16T00:17:21.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:20 smithi060 ceph-mon[35185]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 202 B/s wr, 6 op/s 2024-09-16T00:17:21.763 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-16T00:17:22.020 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:22.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.478+0000 7f3a32761640 1 -- 172.21.15.33:0/738259540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a2c101de0 msgr2=0x7f3a2c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:22.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.478+0000 7f3a32761640 1 --2- 172.21.15.33:0/738259540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a2c101de0 0x7f3a2c1021c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009900 tx=0x7f3a1c02a410 comp rx=0 tx=0).stop 2024-09-16T00:17:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.478+0000 7f3a32761640 1 -- 172.21.15.33:0/738259540 shutdown_connections 2024-09-16T00:17:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.478+0000 7f3a32761640 1 --2- 172.21.15.33:0/738259540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a2c106f10 0x7f3a2c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.478+0000 7f3a32761640 1 --2- 172.21.15.33:0/738259540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a2c102790 0x7f3a2c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.478+0000 7f3a32761640 1 --2- 172.21.15.33:0/738259540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a2c101de0 0x7f3a2c1021c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.478+0000 7f3a32761640 1 -- 172.21.15.33:0/738259540 >> 172.21.15.33:0/738259540 conn(0x7f3a2c0fd8c0 msgr2=0x7f3a2c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.479+0000 7f3a32761640 1 -- 172.21.15.33:0/738259540 shutdown_connections 2024-09-16T00:17:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.479+0000 7f3a32761640 1 -- 172.21.15.33:0/738259540 wait complete. 2024-09-16T00:17:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.479+0000 7f3a32761640 1 Processor -- start 2024-09-16T00:17:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.479+0000 7f3a32761640 1 -- start start 2024-09-16T00:17:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a32761640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a2c101de0 0x7f3a2c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a32761640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a2c102790 0x7f3a2c1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a32761640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a2c106f10 0x7f3a2c19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a32761640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c111f70 con 0x7f3a2c106f10 2024-09-16T00:17:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a32761640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c111df0 con 0x7f3a2c102790 2024-09-16T00:17:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a32761640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c1120f0 con 0x7f3a2c101de0 2024-09-16T00:17:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a30cfe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a2c106f10 0x7f3a2c19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a30cfe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a2c106f10 0x7f3a2c19cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34806/0 (socket says 172.21.15.33:34806) 2024-09-16T00:17:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a30cfe640 1 -- 172.21.15.33:0/3054527950 learned_addr learned my addr 172.21.15.33:0/3054527950 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a2bfff640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a2c101de0 0x7f3a2c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a2c102790 0x7f3a2c1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a30cfe640 1 -- 172.21.15.33:0/3054527950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a2c101de0 msgr2=0x7f3a2c198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a30cfe640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a2c101de0 0x7f3a2c198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a30cfe640 1 -- 172.21.15.33:0/3054527950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a2c102790 msgr2=0x7f3a2c1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a30cfe640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a2c102790 0x7f3a2c1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a30cfe640 1 -- 172.21.15.33:0/3054527950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2c19d410 con 0x7f3a2c106f10 2024-09-16T00:17:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a30cfe640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a2c106f10 0x7f3a2c19cd30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009fd0 tx=0x7f3a2400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a297fa640 1 -- 172.21.15.33:0/3054527950 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a24017070 con 0x7f3a2c106f10 2024-09-16T00:17:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a297fa640 1 -- 172.21.15.33:0/3054527950 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a24003e50 con 0x7f3a2c106f10 2024-09-16T00:17:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a32761640 1 -- 172.21.15.33:0/3054527950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a2c19d700 con 0x7f3a2c106f10 2024-09-16T00:17:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.480+0000 7f3a297fa640 1 -- 172.21.15.33:0/3054527950 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a240106c0 con 0x7f3a2c106f10 2024-09-16T00:17:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.481+0000 7f3a32761640 1 -- 172.21.15.33:0/3054527950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a2c1a4fe0 con 0x7f3a2c106f10 2024-09-16T00:17:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.483+0000 7f3a297fa640 1 -- 172.21.15.33:0/3054527950 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f3a24003970 con 0x7f3a2c106f10 2024-09-16T00:17:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.483+0000 7f3a34b3a740 1 Processor -- start 2024-09-16T00:17:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.483+0000 7f3a34b3a740 1 -- start start 2024-09-16T00:17:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.483+0000 7f3a34b3a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31e9d0 0x5597db31edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.483+0000 7f3a34b3a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31f2f0 0x5597db329670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.483+0000 7f3a34b3a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db32bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.483+0000 7f3a34b3a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5597db285090 con 0x5597db329bb0 2024-09-16T00:17:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.483+0000 7f3a34b3a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5597db22ca70 con 0x5597db31e9d0 2024-09-16T00:17:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.483+0000 7f3a34b3a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5597db281050 con 0x5597db31f2f0 2024-09-16T00:17:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a2b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db32bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a2b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db32bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34814/0 (socket says 172.21.15.33:34814) 2024-09-16T00:17:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/1705491782 learned_addr learned my addr 172.21.15.33:0/1705491782 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a2bfff640 1 --2- 172.21.15.33:0/1705491782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31f2f0 0x5597db329670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/1705491782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31e9d0 0x5597db31edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a297fa640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3a140a3240 0x7f3a140a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/1705491782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31f2f0 msgr2=0x5597db329670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/1705491782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31f2f0 0x5597db329670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/1705491782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31e9d0 msgr2=0x5597db31edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/1705491782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31e9d0 0x5597db31edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/1705491782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597db239130 con 0x5597db329bb0 2024-09-16T00:17:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a2bfff640 1 --2- 172.21.15.33:0/1705491782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31f2f0 0x5597db329670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a30cfe640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3a140a3240 0x7f3a140a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a297fa640 1 -- 172.21.15.33:0/3054527950 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3a240cfab0 con 0x7f3a2c106f10 2024-09-16T00:17:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/1705491782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db32bfa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3a200125a0 tx=0x7f3a2000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5050736f6254c5b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a127fc640 1 -- 172.21.15.33:0/1705491782 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20004250 con 0x5597db329bb0 2024-09-16T00:17:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a127fc640 1 -- 172.21.15.33:0/1705491782 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a200043f0 con 0x5597db329bb0 2024-09-16T00:17:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.484+0000 7f3a127fc640 1 -- 172.21.15.33:0/1705491782 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a200035f0 con 0x5597db329bb0 2024-09-16T00:17:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.485+0000 7f3a34b3a740 1 -- 172.21.15.33:0/1705491782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 msgr2=0x5597db32bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.485+0000 7f3a30cfe640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3a140a3240 0x7f3a140a5700 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3a24098100 tx=0x7f3a240c69c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.485+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/1705491782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db32bfa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3a200125a0 tx=0x7f3a2000b870 comp rx=0 tx=0).stop 2024-09-16T00:17:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.485+0000 7f3a34b3a740 1 -- 172.21.15.33:0/1705491782 shutdown_connections 2024-09-16T00:17:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.485+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/1705491782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db32bfa0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.485+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/1705491782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31f2f0 0x5597db329670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.485+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/1705491782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31e9d0 0x5597db31edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.485+0000 7f3a34b3a740 1 -- 172.21.15.33:0/1705491782 >> 172.21.15.33:0/1705491782 conn(0x5597db31af60 msgr2=0x5597db31b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.485+0000 7f3a34b3a740 1 -- 172.21.15.33:0/1705491782 shutdown_connections 2024-09-16T00:17:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.485+0000 7f3a34b3a740 1 -- 172.21.15.33:0/1705491782 wait complete. 2024-09-16T00:17:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a34b3a740 1 -- start start 2024-09-16T00:17:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a34b3a740 1 Processor -- start 2024-09-16T00:17:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a34b3a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31e9d0 0x5597db3294c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a34b3a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31f2f0 0x5597db326810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a34b3a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db326d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a34b3a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5597db32c780 con 0x5597db329bb0 2024-09-16T00:17:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a34b3a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5597db23ff20 con 0x5597db31f2f0 2024-09-16T00:17:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a34b3a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5597db32c900 con 0x5597db31e9d0 2024-09-16T00:17:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a34b3a740 10 client.0 _ll_register_callbacks cb 0x5597db209030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a2b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db326d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a2b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db326d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34820/0 (socket says 172.21.15.33:34820) 2024-09-16T00:17:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/4017305984 learned_addr learned my addr 172.21.15.33:0/4017305984 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:22.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31e9d0 0x5597db3294c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.486+0000 7f3a2bfff640 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31f2f0 0x5597db326810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/4017305984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31e9d0 msgr2=0x5597db3294c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:22.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31e9d0 0x5597db3294c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/4017305984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31f2f0 msgr2=0x5597db326810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:22.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31f2f0 0x5597db326810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/4017305984 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597daff62b0 con 0x5597db329bb0 2024-09-16T00:17:22.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db326d50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3a20005d00 tx=0x7f3a20004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85de920d9751770 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:22.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:22.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a0b7fe640 1 -- 172.21.15.33:0/4017305984 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20029040 con 0x5597db329bb0 2024-09-16T00:17:22.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a0b7fe640 1 -- 172.21.15.33:0/4017305984 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3a20005020 con 0x5597db329bb0 2024-09-16T00:17:22.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a34b3a740 1 -- 172.21.15.33:0/4017305984 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a200043f0 con 0x5597db329bb0 2024-09-16T00:17:22.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a0b7fe640 1 -- 172.21.15.33:0/4017305984 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3a20036050 con 0x5597db329bb0 2024-09-16T00:17:22.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a34b3a740 10 client.15201 fetch_fsmap learned FSMap version 25 2024-09-16T00:17:22.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a34b3a740 1 -- 172.21.15.33:0/4017305984 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5597db338100 con 0x5597db329bb0 2024-09-16T00:17:22.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.487+0000 7f3a0b7fe640 1 -- 172.21.15.33:0/4017305984 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a200043f0 con 0x5597db329bb0 2024-09-16T00:17:22.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.488+0000 7f3a0b7fe640 1 -- 172.21.15.33:0/4017305984 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f3a20010ac0 con 0x5597db329bb0 2024-09-16T00:17:22.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.488+0000 7f3a34b3a740 10 client.15201 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T00:17:22.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.488+0000 7f3a34b3a740 10 client.15201 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-16T00:17:22.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.488+0000 7f3a34b3a740 20 client.15201 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.488+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5597db33a3c0 0x5597db33b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.488+0000 7f3a34b3a740 4 client.15201 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:17:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.488+0000 7f3a34b3a740 1 -- 172.21.15.33:0/4017305984 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597db292600 con 0x5597db33a3c0 2024-09-16T00:17:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.488+0000 7f3a2bfff640 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5597db33a3c0 0x5597db33b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.488+0000 7f3a0b7fe640 10 client.15201 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:17:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.488+0000 7f3a2bfff640 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5597db33a3c0 0x5597db33b830 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5976874a837f9d66 server_cookie=d4661cfe265dc5c7 in_seq=0 out_seq=0 2024-09-16T00:17:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.489+0000 7f3a0b7fe640 1 -- 172.21.15.33:0/4017305984 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5597db292600 con 0x5597db33a3c0 2024-09-16T00:17:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.489+0000 7f3a0b7fe640 10 client.15201 handle_command_reply: tid=0 2024-09-16T00:17:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.489+0000 7f3a34b3a740 1 client.15201 shutdown 2024-09-16T00:17:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.489+0000 7f3a34b3a740 1 -- 172.21.15.33:0/4017305984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 msgr2=0x5597db326d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.489+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db326d50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3a20005d00 tx=0x7f3a20004a50 comp rx=0 tx=0).stop 2024-09-16T00:17:22.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.490+0000 7f3a34b3a740 1 -- 172.21.15.33:0/4017305984 shutdown_connections 2024-09-16T00:17:22.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.490+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db326d50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.490+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5597db33a3c0 0x5597db33b830 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.490+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31f2f0 0x5597db326810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.490+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/4017305984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31e9d0 0x5597db3294c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.490+0000 7f3a0b7fe640 0 client.15201 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:17:22.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.490+0000 7f3a34b3a740 1 -- 172.21.15.33:0/4017305984 >> 172.21.15.33:0/4017305984 conn(0x5597db31af60 msgr2=0x5597db320c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:22.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.490+0000 7f3a34b3a740 1 -- 172.21.15.33:0/4017305984 shutdown_connections 2024-09-16T00:17:22.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.490+0000 7f3a34b3a740 1 -- 172.21.15.33:0/4017305984 wait complete. 2024-09-16T00:17:22.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.490+0000 7f3a34b3a740 20 client.15201 trim_cache size 0 max 16384 2024-09-16T00:17:22.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.508+0000 7f3a34b3a740 1 Processor -- start 2024-09-16T00:17:22.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.508+0000 7f3a34b3a740 1 -- start start 2024-09-16T00:17:22.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.508+0000 7f3a34b3a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31e9d0 0x5597db33b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.508+0000 7f3a34b3a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31f2f0 0x5597db343f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.508+0000 7f3a34b3a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db3250e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.508+0000 7f3a34b3a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5597db339b70 con 0x5597db329bb0 2024-09-16T00:17:22.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.508+0000 7f3a34b3a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5597db33a240 con 0x5597db31f2f0 2024-09-16T00:17:22.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.508+0000 7f3a34b3a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5597db33a910 con 0x5597db31e9d0 2024-09-16T00:17:22.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.508+0000 7f3a2b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db3250e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.508+0000 7f3a2b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db3250e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34836/0 (socket says 172.21.15.33:34836) 2024-09-16T00:17:22.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.508+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/2950500184 learned_addr learned my addr 172.21.15.33:0/2950500184 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:22.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.508+0000 7f3a2bfff640 1 --2- 172.21.15.33:0/2950500184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31f2f0 0x5597db343f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.509+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/2950500184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31e9d0 0x5597db33b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.509+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/2950500184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31e9d0 msgr2=0x5597db33b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:22.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.509+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/2950500184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31e9d0 0x5597db33b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.509+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/2950500184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31f2f0 msgr2=0x5597db343f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:22.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.509+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/2950500184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31f2f0 0x5597db343f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.509+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/2950500184 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597db327b90 con 0x5597db329bb0 2024-09-16T00:17:22.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.509+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/2950500184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db3250e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3a20012f70 tx=0x7f3a2000bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a531c6fab9386a8c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:22.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.509+0000 7f3a11ffb640 1 -- 172.21.15.33:0/2950500184 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a200104b0 con 0x5597db329bb0 2024-09-16T00:17:22.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.509+0000 7f3a11ffb640 1 -- 172.21.15.33:0/2950500184 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a2000b7c0 con 0x5597db329bb0 2024-09-16T00:17:22.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.509+0000 7f3a11ffb640 1 -- 172.21.15.33:0/2950500184 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a200120e0 con 0x5597db329bb0 2024-09-16T00:17:22.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.510+0000 7f3a34b3a740 1 -- 172.21.15.33:0/2950500184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 msgr2=0x5597db3250e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:22.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.510+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/2950500184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db3250e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3a20012f70 tx=0x7f3a2000bf50 comp rx=0 tx=0).stop 2024-09-16T00:17:22.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.510+0000 7f3a34b3a740 1 -- 172.21.15.33:0/2950500184 shutdown_connections 2024-09-16T00:17:22.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.510+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/2950500184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db329bb0 0x5597db3250e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.510+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/2950500184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31f2f0 0x5597db343f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.510+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/2950500184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db31e9d0 0x5597db33b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.510+0000 7f3a34b3a740 1 -- 172.21.15.33:0/2950500184 >> 172.21.15.33:0/2950500184 conn(0x5597db31af60 msgr2=0x5597db3267d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:22.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.510+0000 7f3a34b3a740 1 -- 172.21.15.33:0/2950500184 shutdown_connections 2024-09-16T00:17:22.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.510+0000 7f3a34b3a740 1 -- 172.21.15.33:0/2950500184 wait complete. 2024-09-16T00:17:22.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.510+0000 7f3a34b3a740 1 -- start start 2024-09-16T00:17:22.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.510+0000 7f3a34b3a740 1 Processor -- start 2024-09-16T00:17:22.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a34b3a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31e9d0 0x5597db33d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a34b3a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db31f2f0 0x5597db33d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a34b3a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db329bb0 0x5597db39fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a34b3a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5597db33b6b0 con 0x5597db31f2f0 2024-09-16T00:17:22.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a34b3a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5597db33afe0 con 0x5597db31e9d0 2024-09-16T00:17:22.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a34b3a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5597db33bd80 con 0x5597db329bb0 2024-09-16T00:17:22.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a34b3a740 10 client.0 _ll_register_callbacks cb 0x7f3a200123f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:22.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a2bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db31f2f0 0x5597db33d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a2bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db31f2f0 0x5597db33d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34846/0 (socket says 172.21.15.33:34846) 2024-09-16T00:17:22.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a2bfff640 1 -- 172.21.15.33:0/3620882056 learned_addr learned my addr 172.21.15.33:0/3620882056 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:22.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db329bb0 0x5597db39fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31e9d0 0x5597db33d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a2bfff640 1 -- 172.21.15.33:0/3620882056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db329bb0 msgr2=0x5597db39fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:22.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a2bfff640 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db329bb0 0x5597db39fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a2bfff640 1 -- 172.21.15.33:0/3620882056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31e9d0 msgr2=0x5597db33d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:22.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a2bfff640 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31e9d0 0x5597db33d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a2bfff640 1 -- 172.21.15.33:0/3620882056 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597db3a0100 con 0x5597db31f2f0 2024-09-16T00:17:22.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31e9d0 0x5597db33d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:22.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a2bfff640 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db31f2f0 0x5597db33d980 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c000c00 tx=0x7f3a1c031aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3f8f75fccb3376 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:22.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.511+0000 7f3a0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:22.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a0b7fe640 1 -- 172.21.15.33:0/3620882056 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c035040 con 0x5597db31f2f0 2024-09-16T00:17:22.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a0b7fe640 1 -- 172.21.15.33:0/3620882056 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3a1c005020 con 0x5597db31f2f0 2024-09-16T00:17:22.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a34b3a740 1 -- 172.21.15.33:0/3620882056 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a2000b7c0 con 0x5597db31f2f0 2024-09-16T00:17:22.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a0b7fe640 1 -- 172.21.15.33:0/3620882056 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3a1c058050 con 0x5597db31f2f0 2024-09-16T00:17:22.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a34b3a740 10 client.15207 fetch_fsmap learned FSMap version 25 2024-09-16T00:17:22.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a34b3a740 1 -- 172.21.15.33:0/3620882056 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5597db3a8690 con 0x5597db31f2f0 2024-09-16T00:17:22.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a0b7fe640 1 -- 172.21.15.33:0/3620882056 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a2000b7c0 con 0x5597db31f2f0 2024-09-16T00:17:22.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a0b7fe640 1 -- 172.21.15.33:0/3620882056 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f3a1c0035f0 con 0x5597db31f2f0 2024-09-16T00:17:22.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a34b3a740 10 client.15207 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T00:17:22.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a34b3a740 10 client.15207 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-16T00:17:22.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a34b3a740 20 client.15207 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:22.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5597db3ad270 0x5597db3abf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:22.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a34b3a740 4 client.15207 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T00:17:22.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.512+0000 7f3a34b3a740 1 -- 172.21.15.33:0/3620882056 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "status"}) -- 0x5597db28cb10 con 0x5597db3ad270 2024-09-16T00:17:22.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.513+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5597db3ad270 0x5597db3abf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:22.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.513+0000 7f3a0b7fe640 10 client.15207 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:17:22.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.513+0000 7f3a2b7fe640 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5597db3ad270 0x5597db3abf80 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=167aec0b66527689 server_cookie=a3aefa185a914806 in_seq=0 out_seq=0 2024-09-16T00:17:22.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.513+0000 7f3a0b7fe640 1 -- 172.21.15.33:0/3620882056 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5597db28cb10 con 0x5597db3ad270 2024-09-16T00:17:22.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.513+0000 7f3a0b7fe640 10 client.15207 handle_command_reply: tid=0 2024-09-16T00:17:22.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.513+0000 7f3a34b3a740 1 client.15207 shutdown 2024-09-16T00:17:22.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.513+0000 7f3a34b3a740 1 -- 172.21.15.33:0/3620882056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db31f2f0 msgr2=0x5597db33d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:22.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.513+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db31f2f0 0x5597db33d980 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c000c00 tx=0x7f3a1c031aa0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a34b3a740 1 -- 172.21.15.33:0/3620882056 shutdown_connections 2024-09-16T00:17:22.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597db329bb0 0x5597db39fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597db31f2f0 0x5597db33d980 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5597db3ad270 0x5597db3abf80 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a34b3a740 1 --2- 172.21.15.33:0/3620882056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597db31e9d0 0x5597db33d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a0b7fe640 0 client.15207 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:17:22.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a34b3a740 1 -- 172.21.15.33:0/3620882056 >> 172.21.15.33:0/3620882056 conn(0x5597db31af60 msgr2=0x5597db324ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:22.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a34b3a740 1 -- 172.21.15.33:0/3620882056 shutdown_connections 2024-09-16T00:17:22.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a34b3a740 1 -- 172.21.15.33:0/3620882056 wait complete. 2024-09-16T00:17:22.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a34b3a740 20 client.15207 trim_cache size 0 max 16384 2024-09-16T00:17:22.521 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:17:22.521 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:17:22.522 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 1, 2024-09-16T00:17:22.522 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24493, 2024-09-16T00:17:22.522 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-09-16T00:17:22.522 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-09-16T00:17:22.522 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-09-16T00:17:22.522 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 94.349888821999997, 2024-09-16T00:17:22.522 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 25, 2024-09-16T00:17:22.522 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 90, 2024-09-16T00:17:22.522 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 86, 2024-09-16T00:17:22.522 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 155.21637251799999 2024-09-16T00:17:22.522 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:17:22.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a32761640 1 -- 172.21.15.33:0/3054527950 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3a140a3240 msgr2=0x7f3a140a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:22.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a32761640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3a140a3240 0x7f3a140a5700 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3a24098100 tx=0x7f3a240c69c0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a32761640 1 -- 172.21.15.33:0/3054527950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a2c106f10 msgr2=0x7f3a2c19cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:22.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.514+0000 7f3a32761640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a2c106f10 0x7f3a2c19cd30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009fd0 tx=0x7f3a2400b260 comp rx=0 tx=0).stop 2024-09-16T00:17:22.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.515+0000 7f3a32761640 1 -- 172.21.15.33:0/3054527950 shutdown_connections 2024-09-16T00:17:22.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.515+0000 7f3a32761640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3a140a3240 0x7f3a140a5700 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.515+0000 7f3a32761640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a2c106f10 0x7f3a2c19cd30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.515+0000 7f3a32761640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a2c102790 0x7f3a2c1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.515+0000 7f3a32761640 1 --2- 172.21.15.33:0/3054527950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a2c101de0 0x7f3a2c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:22.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.515+0000 7f3a32761640 1 -- 172.21.15.33:0/3054527950 >> 172.21.15.33:0/3054527950 conn(0x7f3a2c0fd8c0 msgr2=0x7f3a2c0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:22.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.515+0000 7f3a32761640 1 -- 172.21.15.33:0/3054527950 shutdown_connections 2024-09-16T00:17:22.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:22.515+0000 7f3a32761640 1 -- 172.21.15.33:0/3054527950 wait complete. 2024-09-16T00:17:23.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:22 smithi033 ceph-mon[31243]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-16T00:17:23.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:22 smithi083 ceph-mon[35242]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-16T00:17:23.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:22 smithi060 ceph-mon[35185]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-16T00:17:23.275 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-16T00:17:23.530 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:24.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.984+0000 7f61c2a7f640 1 -- 172.21.15.33:0/483941316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61bc1036e0 msgr2=0x7f61bc107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:24.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.984+0000 7f61c2a7f640 1 --2- 172.21.15.33:0/483941316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61bc1036e0 0x7f61bc107b90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009930 tx=0x7f61b002a420 comp rx=0 tx=0).stop 2024-09-16T00:17:24.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 -- 172.21.15.33:0/483941316 shutdown_connections 2024-09-16T00:17:24.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 --2- 172.21.15.33:0/483941316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61bc1082a0 0x7f61bc10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 --2- 172.21.15.33:0/483941316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61bc1036e0 0x7f61bc107b90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 --2- 172.21.15.33:0/483941316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61bc102d30 0x7f61bc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 -- 172.21.15.33:0/483941316 >> 172.21.15.33:0/483941316 conn(0x7f61bc0fc7d0 msgr2=0x7f61bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:24.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 -- 172.21.15.33:0/483941316 shutdown_connections 2024-09-16T00:17:24.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 -- 172.21.15.33:0/483941316 wait complete. 2024-09-16T00:17:24.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 Processor -- start 2024-09-16T00:17:24.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 -- start start 2024-09-16T00:17:24.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61bc102d30 0x7f61bc198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61bc1036e0 0x7f61bc198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61bc1082a0 0x7f61bc19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1120a0 con 0x7f61bc1036e0 2024-09-16T00:17:24.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc111f20 con 0x7f61bc1082a0 2024-09-16T00:17:24.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61c2a7f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc112220 con 0x7f61bc102d30 2024-09-16T00:17:24.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.985+0000 7f61bbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61bc1036e0 0x7f61bc198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61bbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61bc1036e0 0x7f61bc198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34856/0 (socket says 172.21.15.33:34856) 2024-09-16T00:17:24.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61bbfff640 1 -- 172.21.15.33:0/3148289330 learned_addr learned my addr 172.21.15.33:0/3148289330 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:24.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61c101c640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61bc1082a0 0x7f61bc19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61c081b640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61bc102d30 0x7f61bc198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61bbfff640 1 -- 172.21.15.33:0/3148289330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61bc102d30 msgr2=0x7f61bc198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:24.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61bbfff640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61bc102d30 0x7f61bc198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61bbfff640 1 -- 172.21.15.33:0/3148289330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61bc1082a0 msgr2=0x7f61bc19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:24.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61bbfff640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61bc1082a0 0x7f61bc19ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61bbfff640 1 -- 172.21.15.33:0/3148289330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61bc19d540 con 0x7f61bc1036e0 2024-09-16T00:17:24.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61c081b640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61bc102d30 0x7f61bc198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:24.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61c101c640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61bc1082a0 0x7f61bc19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:24.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61bbfff640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61bc1036e0 0x7f61bc198ad0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f61b0005940 tx=0x7f61b00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:24.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61b9ffb640 1 -- 172.21.15.33:0/3148289330 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b003f030 con 0x7f61bc1036e0 2024-09-16T00:17:24.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61b9ffb640 1 -- 172.21.15.33:0/3148289330 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61b0003ce0 con 0x7f61bc1036e0 2024-09-16T00:17:24.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61c2a7f640 1 -- 172.21.15.33:0/3148289330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61bc19d7d0 con 0x7f61bc1036e0 2024-09-16T00:17:24.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61b9ffb640 1 -- 172.21.15.33:0/3148289330 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0035070 con 0x7f61bc1036e0 2024-09-16T00:17:24.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.986+0000 7f61c2a7f640 1 -- 172.21.15.33:0/3148289330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61bc1a5110 con 0x7f61bc1036e0 2024-09-16T00:17:24.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.989+0000 7f61b9ffb640 1 -- 172.21.15.33:0/3148289330 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f61b0042030 con 0x7f61bc1036e0 2024-09-16T00:17:24.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.989+0000 7f61c4e58740 1 Processor -- start 2024-09-16T00:17:24.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c4e58740 1 -- start start 2024-09-16T00:17:24.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c4e58740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa05cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c4e58740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05d2f0 0x564ffa067670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c4e58740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa069fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c4e58740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564ff9fc3090 con 0x564ffa05c9d0 2024-09-16T00:17:24.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c4e58740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564ff9f6aa70 con 0x564ffa067bb0 2024-09-16T00:17:24.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c4e58740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564ff9fbf050 con 0x564ffa05d2f0 2024-09-16T00:17:24.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c101c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa05cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c101c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa05cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34868/0 (socket says 172.21.15.33:34868) 2024-09-16T00:17:24.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c101c640 1 -- 172.21.15.33:0/1366049247 learned_addr learned my addr 172.21.15.33:0/1366049247 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:24.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61bbfff640 1 --2- 172.21.15.33:0/1366049247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa069fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c081b640 1 --2- 172.21.15.33:0/1366049247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05d2f0 0x564ffa067670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c101c640 1 -- 172.21.15.33:0/1366049247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05d2f0 msgr2=0x564ffa067670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:24.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61b9ffb640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f61a40a3240 0x7f61a40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c101c640 1 --2- 172.21.15.33:0/1366049247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05d2f0 0x564ffa067670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c101c640 1 -- 172.21.15.33:0/1366049247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 msgr2=0x564ffa069fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:24.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c101c640 1 --2- 172.21.15.33:0/1366049247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa069fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c101c640 1 -- 172.21.15.33:0/1366049247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ff9f77130 con 0x564ffa05c9d0 2024-09-16T00:17:24.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61c081b640 1 --2- 172.21.15.33:0/1366049247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05d2f0 0x564ffa067670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:24.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.990+0000 7f61bbfff640 1 --2- 172.21.15.33:0/1366049247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa069fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:24.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.991+0000 7f61b9ffb640 1 -- 172.21.15.33:0/3148289330 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f61b00f0440 con 0x7f61bc1036e0 2024-09-16T00:17:24.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.991+0000 7f61c101c640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f61a40a3240 0x7f61a40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.991+0000 7f61c101c640 1 --2- 172.21.15.33:0/1366049247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa05cdb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f61b4007e60 tx=0x7f61b400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=635089486b8934ed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.991+0000 7f61a2ffd640 1 -- 172.21.15.33:0/1366049247 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4020070 con 0x564ffa05c9d0 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.991+0000 7f61a2ffd640 1 -- 172.21.15.33:0/1366049247 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61b401ae30 con 0x564ffa05c9d0 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.991+0000 7f61a2ffd640 1 -- 172.21.15.33:0/1366049247 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4014070 con 0x564ffa05c9d0 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.992+0000 7f61c4e58740 1 -- 172.21.15.33:0/1366049247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 msgr2=0x564ffa05cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.992+0000 7f61c101c640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f61a40a3240 0x7f61a40a5700 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f61b401d000 tx=0x7f61b4020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.992+0000 7f61c4e58740 1 --2- 172.21.15.33:0/1366049247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa05cdb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f61b4007e60 tx=0x7f61b400e720 comp rx=0 tx=0).stop 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.992+0000 7f61c4e58740 1 -- 172.21.15.33:0/1366049247 shutdown_connections 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.992+0000 7f61c4e58740 1 --2- 172.21.15.33:0/1366049247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa069fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.992+0000 7f61c4e58740 1 --2- 172.21.15.33:0/1366049247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05d2f0 0x564ffa067670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.992+0000 7f61c4e58740 1 --2- 172.21.15.33:0/1366049247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa05cdb0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.992+0000 7f61c4e58740 1 -- 172.21.15.33:0/1366049247 >> 172.21.15.33:0/1366049247 conn(0x564ffa058f60 msgr2=0x564ffa059340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.992+0000 7f61c4e58740 1 -- 172.21.15.33:0/1366049247 shutdown_connections 2024-09-16T00:17:24.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.992+0000 7f61c4e58740 1 -- 172.21.15.33:0/1366049247 wait complete. 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.992+0000 7f61c4e58740 1 -- start start 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.992+0000 7f61c4e58740 1 Processor -- start 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c4e58740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa0673e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c4e58740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa05d2f0 0x564ffa064730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c4e58740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa067bb0 0x564ffa064c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c4e58740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564ffa06a780 con 0x564ffa05c9d0 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c4e58740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564ff9f7df20 con 0x564ffa05d2f0 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c4e58740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564ffa06a900 con 0x564ffa067bb0 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c4e58740 10 client.0 _ll_register_callbacks cb 0x564ff9f47030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c081b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa0673e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c081b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa0673e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34882/0 (socket says 172.21.15.33:34882) 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c081b640 1 -- 172.21.15.33:0/1222439642 learned_addr learned my addr 172.21.15.33:0/1222439642 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:24.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c101c640 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa067bb0 0x564ffa064c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61bbfff640 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa05d2f0 0x564ffa064730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c081b640 1 -- 172.21.15.33:0/1222439642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa067bb0 msgr2=0x564ffa064c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c081b640 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa067bb0 0x564ffa064c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c081b640 1 -- 172.21.15.33:0/1222439642 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa05d2f0 msgr2=0x564ffa064730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c081b640 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa05d2f0 0x564ffa064730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c081b640 1 -- 172.21.15.33:0/1222439642 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ff9d342b0 con 0x564ffa05c9d0 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f61c081b640 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa0673e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00f670 tx=0x7f61ac00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02ee288eb7d5f1b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.993+0000 7f619bfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f619bfff640 1 -- 172.21.15.33:0/1222439642 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac016070 con 0x564ffa05c9d0 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f619bfff640 1 -- 172.21.15.33:0/1222439642 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f61ac00bb00 con 0x564ffa05c9d0 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f61c4e58740 1 -- 172.21.15.33:0/1222439642 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61b401ae30 con 0x564ffa05c9d0 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f619bfff640 1 -- 172.21.15.33:0/1222439642 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f61ac025050 con 0x564ffa05c9d0 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f61c4e58740 10 client.15219 fetch_fsmap learned FSMap version 25 2024-09-16T00:17:24.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f61c4e58740 1 -- 172.21.15.33:0/1222439642 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564ffa076090 con 0x564ffa05c9d0 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f619bfff640 1 -- 172.21.15.33:0/1222439642 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61b401ae30 con 0x564ffa05c9d0 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f619bfff640 1 -- 172.21.15.33:0/1222439642 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f61ac01a400 con 0x564ffa05c9d0 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f61c4e58740 10 client.15219 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f61c4e58740 10 client.15219 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f61c4e58740 20 client.15219 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f61c4e58740 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564ffa078720 0x564ffa079b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f61c4e58740 4 client.15219 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.994+0000 7f61c4e58740 1 -- 172.21.15.33:0/1222439642 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ff9fd0600 con 0x564ffa078720 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.995+0000 7f61bbfff640 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564ffa078720 0x564ffa079b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.995+0000 7f61bbfff640 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564ffa078720 0x564ffa079b90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cac110f76788e1a server_cookie=77e25a017b7e8c26 in_seq=0 out_seq=0 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.995+0000 7f619bfff640 10 client.15219 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f619bfff640 1 -- 172.21.15.33:0/1222439642 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564ff9fd0600 con 0x564ffa078720 2024-09-16T00:17:24.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f619bfff640 10 client.15219 handle_command_reply: tid=0 2024-09-16T00:17:24.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f61c4e58740 1 client.15219 shutdown 2024-09-16T00:17:24.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f61c4e58740 1 -- 172.21.15.33:0/1222439642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 msgr2=0x564ffa0673e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:24.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f61c4e58740 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa0673e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00f670 tx=0x7f61ac00fb30 comp rx=0 tx=0).stop 2024-09-16T00:17:24.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f61c4e58740 1 -- 172.21.15.33:0/1222439642 shutdown_connections 2024-09-16T00:17:24.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f61c4e58740 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564ffa078720 0x564ffa079b90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f61c4e58740 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa067bb0 0x564ffa064c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f61c4e58740 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa05d2f0 0x564ffa064730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f619bfff640 0 client.15219 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:24.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f61c4e58740 1 --2- 172.21.15.33:0/1222439642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa0673e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f61c4e58740 1 -- 172.21.15.33:0/1222439642 >> 172.21.15.33:0/1222439642 conn(0x564ffa058f60 msgr2=0x564ffa05ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:24.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f61c4e58740 1 -- 172.21.15.33:0/1222439642 shutdown_connections 2024-09-16T00:17:24.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f61c4e58740 1 -- 172.21.15.33:0/1222439642 wait complete. 2024-09-16T00:17:24.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:23.996+0000 7f61c4e58740 20 client.15219 trim_cache size 0 max 16384 2024-09-16T00:17:24.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.014+0000 7f61c4e58740 1 Processor -- start 2024-09-16T00:17:24.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.014+0000 7f61c4e58740 1 -- start start 2024-09-16T00:17:24.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.014+0000 7f61c4e58740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa079b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.014+0000 7f61c4e58740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05d2f0 0x564ffa053ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.014+0000 7f61c4e58740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa066590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.014+0000 7f61c4e58740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564ffa077ed0 con 0x564ffa05c9d0 2024-09-16T00:17:24.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.014+0000 7f61c4e58740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564ffa0785a0 con 0x564ffa067bb0 2024-09-16T00:17:24.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.014+0000 7f61c4e58740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564ffa078c70 con 0x564ffa05d2f0 2024-09-16T00:17:24.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61c081b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa079b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61c081b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa079b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34884/0 (socket says 172.21.15.33:34884) 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61c081b640 1 -- 172.21.15.33:0/575855942 learned_addr learned my addr 172.21.15.33:0/575855942 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61c101c640 1 --2- 172.21.15.33:0/575855942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa066590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61bbfff640 1 --2- 172.21.15.33:0/575855942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05d2f0 0x564ffa053ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61c081b640 1 -- 172.21.15.33:0/575855942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05d2f0 msgr2=0x564ffa053ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61c081b640 1 --2- 172.21.15.33:0/575855942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05d2f0 0x564ffa053ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61c081b640 1 -- 172.21.15.33:0/575855942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 msgr2=0x564ffa066590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61c081b640 1 --2- 172.21.15.33:0/575855942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa066590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61c081b640 1 -- 172.21.15.33:0/575855942 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ffa063700 con 0x564ffa05c9d0 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61c101c640 1 --2- 172.21.15.33:0/575855942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa066590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61bbfff640 1 --2- 172.21.15.33:0/575855942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05d2f0 0x564ffa053ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61c081b640 1 --2- 172.21.15.33:0/575855942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa079b90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f61ac01a820 tx=0x7f61ac0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7c769642accd0b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61a27fc640 1 -- 172.21.15.33:0/575855942 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac02f020 con 0x564ffa05c9d0 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61a27fc640 1 -- 172.21.15.33:0/575855942 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61ac01a600 con 0x564ffa05c9d0 2024-09-16T00:17:24.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.015+0000 7f61a27fc640 1 -- 172.21.15.33:0/575855942 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac016040 con 0x564ffa05c9d0 2024-09-16T00:17:24.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.016+0000 7f61c4e58740 1 -- 172.21.15.33:0/575855942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 msgr2=0x564ffa079b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:24.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.016+0000 7f61c4e58740 1 --2- 172.21.15.33:0/575855942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa079b90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f61ac01a820 tx=0x7f61ac0058a0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.016+0000 7f61c4e58740 1 -- 172.21.15.33:0/575855942 shutdown_connections 2024-09-16T00:17:24.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.016+0000 7f61c4e58740 1 --2- 172.21.15.33:0/575855942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa066590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.016+0000 7f61c4e58740 1 --2- 172.21.15.33:0/575855942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05d2f0 0x564ffa053ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.016+0000 7f61c4e58740 1 --2- 172.21.15.33:0/575855942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05c9d0 0x564ffa079b90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.016+0000 7f61c4e58740 1 -- 172.21.15.33:0/575855942 >> 172.21.15.33:0/575855942 conn(0x564ffa058f60 msgr2=0x564ffa065e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:24.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.016+0000 7f61c4e58740 1 -- 172.21.15.33:0/575855942 shutdown_connections 2024-09-16T00:17:24.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.016+0000 7f61c4e58740 1 -- 172.21.15.33:0/575855942 wait complete. 2024-09-16T00:17:24.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.016+0000 7f61c4e58740 1 -- start start 2024-09-16T00:17:24.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.016+0000 7f61c4e58740 1 Processor -- start 2024-09-16T00:17:24.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61c4e58740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05c9d0 0x564ffa0822c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61c4e58740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05d2f0 0x564ffa07d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61c4e58740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa07db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61c4e58740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564ffa079a10 con 0x564ffa05d2f0 2024-09-16T00:17:24.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61c4e58740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564ffa079340 con 0x564ffa067bb0 2024-09-16T00:17:24.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61c4e58740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564ffa07a0e0 con 0x564ffa05c9d0 2024-09-16T00:17:24.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61c4e58740 10 client.0 _ll_register_callbacks cb 0x7f61ac00d340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:24.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61bbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05d2f0 0x564ffa07d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61bbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05d2f0 0x564ffa07d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34894/0 (socket says 172.21.15.33:34894) 2024-09-16T00:17:24.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61bbfff640 1 -- 172.21.15.33:0/3897269981 learned_addr learned my addr 172.21.15.33:0/3897269981 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:24.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61c101c640 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa07db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61c081b640 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05c9d0 0x564ffa0822c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61bbfff640 1 -- 172.21.15.33:0/3897269981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05c9d0 msgr2=0x564ffa0822c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:24.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61bbfff640 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05c9d0 0x564ffa0822c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61bbfff640 1 -- 172.21.15.33:0/3897269981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 msgr2=0x564ffa07db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:24.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61bbfff640 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa07db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.017+0000 7f61bbfff640 1 -- 172.21.15.33:0/3897269981 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ffa07e0c0 con 0x564ffa05d2f0 2024-09-16T00:17:24.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f61bbfff640 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05d2f0 0x564ffa07d640 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f61b00b89e0 tx=0x7f61b002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb206a040da27479 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:24.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f619bfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:24.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f619bfff640 1 -- 172.21.15.33:0/3897269981 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0049050 con 0x564ffa05d2f0 2024-09-16T00:17:24.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f619bfff640 1 -- 172.21.15.33:0/3897269981 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f61b003b460 con 0x564ffa05d2f0 2024-09-16T00:17:24.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f61c4e58740 1 -- 172.21.15.33:0/3897269981 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61ac01a600 con 0x564ffa05d2f0 2024-09-16T00:17:24.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f619bfff640 1 -- 172.21.15.33:0/3897269981 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f61b00bc580 con 0x564ffa05d2f0 2024-09-16T00:17:24.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f61c4e58740 10 client.15225 fetch_fsmap learned FSMap version 25 2024-09-16T00:17:24.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f61c4e58740 1 -- 172.21.15.33:0/3897269981 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564ffa0e2ab0 con 0x564ffa05d2f0 2024-09-16T00:17:24.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f619bfff640 1 -- 172.21.15.33:0/3897269981 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61ac01a600 con 0x564ffa05d2f0 2024-09-16T00:17:24.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f619bfff640 1 -- 172.21.15.33:0/3897269981 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f61b00b8bf0 con 0x564ffa05d2f0 2024-09-16T00:17:24.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f61c4e58740 10 client.15225 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T00:17:24.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f61c4e58740 10 client.15225 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T00:17:24.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f61c4e58740 20 client.15225 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:24.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f61c4e58740 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564ffa078720 0x564ffa0e6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:24.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f61c4e58740 4 client.15225 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T00:17:24.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.018+0000 7f61c4e58740 1 -- 172.21.15.33:0/3897269981 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "status"}) -- 0x564ff9fcab10 con 0x564ffa078720 2024-09-16T00:17:24.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f61c081b640 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564ffa078720 0x564ffa0e6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:24.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f61c081b640 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564ffa078720 0x564ffa0e6410 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8748c6b158b85879 server_cookie=39ed625eb7f5cb2e in_seq=0 out_seq=0 2024-09-16T00:17:24.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f619bfff640 10 client.15225 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:24.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f619bfff640 1 -- 172.21.15.33:0/3897269981 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x564ff9fcab10 con 0x564ffa078720 2024-09-16T00:17:24.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f619bfff640 10 client.15225 handle_command_reply: tid=0 2024-09-16T00:17:24.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f61c4e58740 1 client.15225 shutdown 2024-09-16T00:17:24.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f61c4e58740 1 -- 172.21.15.33:0/3897269981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05d2f0 msgr2=0x564ffa07d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:24.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f61c4e58740 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05d2f0 0x564ffa07d640 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f61b00b89e0 tx=0x7f61b002a420 comp rx=0 tx=0).stop 2024-09-16T00:17:24.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f61c4e58740 1 -- 172.21.15.33:0/3897269981 shutdown_connections 2024-09-16T00:17:24.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f61c4e58740 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564ffa078720 0x564ffa0e6410 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f61c4e58740 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ffa067bb0 0x564ffa07db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f61c4e58740 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ffa05d2f0 0x564ffa07d640 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f61c4e58740 1 --2- 172.21.15.33:0/3897269981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ffa05c9d0 0x564ffa0822c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f61c4e58740 1 -- 172.21.15.33:0/3897269981 >> 172.21.15.33:0/3897269981 conn(0x564ffa058f60 msgr2=0x564ffa058000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:24.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.019+0000 7f619bfff640 0 client.15225 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:24.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c4e58740 1 -- 172.21.15.33:0/3897269981 shutdown_connections 2024-09-16T00:17:24.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c4e58740 1 -- 172.21.15.33:0/3897269981 wait complete. 2024-09-16T00:17:24.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c4e58740 20 client.15225 trim_cache size 0 max 16384 2024-09-16T00:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 0, 2024-09-16T00:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24494, 2024-09-16T00:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-09-16T00:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-09-16T00:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-09-16T00:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 136.490685899, 2024-09-16T00:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 25, 2024-09-16T00:17:24.028 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 90, 2024-09-16T00:17:24.028 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 86, 2024-09-16T00:17:24.028 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 147.568454178 2024-09-16T00:17:24.028 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:17:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c2a7f640 1 -- 172.21.15.33:0/3148289330 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f61a40a3240 msgr2=0x7f61a40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c2a7f640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f61a40a3240 0x7f61a40a5700 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f61b401d000 tx=0x7f61b4020000 comp rx=0 tx=0).stop 2024-09-16T00:17:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c2a7f640 1 -- 172.21.15.33:0/3148289330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61bc1036e0 msgr2=0x7f61bc198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c2a7f640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61bc1036e0 0x7f61bc198ad0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f61b0005940 tx=0x7f61b00042f0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c2a7f640 1 -- 172.21.15.33:0/3148289330 shutdown_connections 2024-09-16T00:17:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c2a7f640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f61a40a3240 0x7f61a40a5700 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c2a7f640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61bc1082a0 0x7f61bc19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c2a7f640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61bc1036e0 0x7f61bc198ad0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c2a7f640 1 --2- 172.21.15.33:0/3148289330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61bc102d30 0x7f61bc198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c2a7f640 1 -- 172.21.15.33:0/3148289330 >> 172.21.15.33:0/3148289330 conn(0x7f61bc0fc7d0 msgr2=0x7f61bc105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c2a7f640 1 -- 172.21.15.33:0/3148289330 shutdown_connections 2024-09-16T00:17:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:24.020+0000 7f61c2a7f640 1 -- 172.21.15.33:0/3148289330 wait complete. 2024-09-16T00:17:24.770 INFO:teuthology.run_tasks:Running task sequential... 2024-09-16T00:17:24.782 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-16T00:17:24.782 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-16T00:17:24.783 INFO:teuthology.task.print:Enabling mgr modules 2024-09-16T00:17:24.783 INFO:teuthology.run_tasks:Running task kclient... 2024-09-16T00:17:24.810 INFO:tasks.kclient:Mounting kernel clients... 2024-09-16T00:17:24.810 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-16T00:17:24.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:17:24.811 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list 2024-09-16T00:17:24.828 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-09-16T00:17:24.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:17:24.829 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-16T00:17:24.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:17:24.897 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link delete ceph-brx 2024-09-16T00:17:24.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:24 smithi033 ceph-mon[31243]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:17:24.997 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-09-16T00:17:24.997 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T00:17:24.997 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T00:17:24.997 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-16T00:17:25.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:24 smithi083 ceph-mon[35242]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:17:25.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:24 smithi060 ceph-mon[35185]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:17:25.294 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:25.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.781+0000 7f1c7f762640 1 -- 172.21.15.33:0/398271163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c7810e480 msgr2=0x7f1c78110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:25.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.781+0000 7f1c7f762640 1 --2- 172.21.15.33:0/398271163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c7810e480 0x7f1c78110870 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1c70009ca0 tx=0x7f1c7002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:17:25.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.781+0000 7f1c7f762640 1 -- 172.21.15.33:0/398271163 shutdown_connections 2024-09-16T00:17:25.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.781+0000 7f1c7f762640 1 --2- 172.21.15.33:0/398271163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c7810e480 0x7f1c78110870 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:25.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.781+0000 7f1c7f762640 1 --2- 172.21.15.33:0/398271163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c78101280 0x7f1c7810df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:25.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.781+0000 7f1c7f762640 1 --2- 172.21.15.33:0/398271163 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c781008d0 0x7f1c78100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:25.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.781+0000 7f1c7f762640 1 -- 172.21.15.33:0/398271163 >> 172.21.15.33:0/398271163 conn(0x7f1c780fc7a0 msgr2=0x7f1c780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:25.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.781+0000 7f1c7f762640 1 -- 172.21.15.33:0/398271163 shutdown_connections 2024-09-16T00:17:25.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.781+0000 7f1c7f762640 1 -- 172.21.15.33:0/398271163 wait complete. 2024-09-16T00:17:25.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.782+0000 7f1c7f762640 1 Processor -- start 2024-09-16T00:17:25.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.782+0000 7f1c7f762640 1 -- start start 2024-09-16T00:17:25.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.782+0000 7f1c7f762640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c781008d0 0x7f1c7819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:25.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.782+0000 7f1c7f762640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c78101280 0x7f1c7819ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:25.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.782+0000 7f1c7f762640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c7810e480 0x7f1c7819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:25.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.782+0000 7f1c7f762640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1c781141c0 con 0x7f1c78101280 2024-09-16T00:17:25.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.782+0000 7f1c7f762640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1c78114040 con 0x7f1c7810e480 2024-09-16T00:17:25.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.782+0000 7f1c7f762640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c78114340 con 0x7f1c781008d0 2024-09-16T00:17:25.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.782+0000 7f1c7ccfd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c78101280 0x7f1c7819ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.782+0000 7f1c7ccfd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c78101280 0x7f1c7819ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34922/0 (socket says 172.21.15.33:34922) 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.782+0000 7f1c7ccfd640 1 -- 172.21.15.33:0/3520064352 learned_addr learned my addr 172.21.15.33:0/3520064352 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c7dcff640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c7810e480 0x7f1c7819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c7d4fe640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c781008d0 0x7f1c7819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c7ccfd640 1 -- 172.21.15.33:0/3520064352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c781008d0 msgr2=0x7f1c7819a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c7ccfd640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c781008d0 0x7f1c7819a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c7ccfd640 1 -- 172.21.15.33:0/3520064352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c7810e480 msgr2=0x7f1c7819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c7ccfd640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c7810e480 0x7f1c7819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c7ccfd640 1 -- 172.21.15.33:0/3520064352 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c7819f690 con 0x7f1c78101280 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c7dcff640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c7810e480 0x7f1c7819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c7d4fe640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c781008d0 0x7f1c7819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c7ccfd640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c78101280 0x7f1c7819ac20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800b4a0 tx=0x7f1c6800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3520064352 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68014070 con 0x7f1c78101280 2024-09-16T00:17:25.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3520064352 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c68010af0 con 0x7f1c78101280 2024-09-16T00:17:25.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3520064352 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68017d10 con 0x7f1c78101280 2024-09-16T00:17:25.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c7f762640 1 -- 172.21.15.33:0/3520064352 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c7819f980 con 0x7f1c78101280 2024-09-16T00:17:25.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.783+0000 7f1c7f762640 1 -- 172.21.15.33:0/3520064352 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c781a7260 con 0x7f1c78101280 2024-09-16T00:17:25.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.784+0000 7f1c7f762640 1 -- 172.21.15.33:0/3520064352 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c78100cb0 con 0x7f1c78101280 2024-09-16T00:17:25.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.785+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3520064352 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f1c68017070 con 0x7f1c78101280 2024-09-16T00:17:25.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.786+0000 7f1c6e7fc640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c640a3240 0x7f1c640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:25.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.786+0000 7f1c7d4fe640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c640a3240 0x7f1c640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:25.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.786+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3520064352 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1c680cf210 con 0x7f1c78101280 2024-09-16T00:17:25.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.787+0000 7f1c7d4fe640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c640a3240 0x7f1c640a5700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1c740058c0 tx=0x7f1c74005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:25.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:25.792+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3520064352 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1c6809bab0 con 0x7f1c78101280 2024-09-16T00:17:26.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.006+0000 7f1c7f762640 1 -- 172.21.15.33:0/3520064352 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1c7802d7c0 con 0x7f1c78101280 2024-09-16T00:17:26.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.007+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3520064352 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+88 (secure 0 0 0) 0x7f1c6809b3b0 con 0x7f1c78101280 2024-09-16T00:17:26.007 INFO:teuthology.orchestra.run.smithi033.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-16T00:17:26.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.008+0000 7f1c7f762640 1 -- 172.21.15.33:0/3520064352 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c640a3240 msgr2=0x7f1c640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:26.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.008+0000 7f1c7f762640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c640a3240 0x7f1c640a5700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1c740058c0 tx=0x7f1c74005850 comp rx=0 tx=0).stop 2024-09-16T00:17:26.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.008+0000 7f1c7f762640 1 -- 172.21.15.33:0/3520064352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c78101280 msgr2=0x7f1c7819ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:26.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.009+0000 7f1c7f762640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c78101280 0x7f1c7819ac20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800b4a0 tx=0x7f1c6800b960 comp rx=0 tx=0).stop 2024-09-16T00:17:26.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.009+0000 7f1c7f762640 1 -- 172.21.15.33:0/3520064352 shutdown_connections 2024-09-16T00:17:26.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.009+0000 7f1c7f762640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c640a3240 0x7f1c640a5700 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:26.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.009+0000 7f1c7f762640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c7810e480 0x7f1c7819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:26.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.009+0000 7f1c7f762640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c78101280 0x7f1c7819ac20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:26.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.009+0000 7f1c7f762640 1 --2- 172.21.15.33:0/3520064352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c781008d0 0x7f1c7819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:26.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.009+0000 7f1c7f762640 1 -- 172.21.15.33:0/3520064352 >> 172.21.15.33:0/3520064352 conn(0x7f1c780fc7a0 msgr2=0x7f1c780feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:26.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.009+0000 7f1c7f762640 1 -- 172.21.15.33:0/3520064352 shutdown_connections 2024-09-16T00:17:26.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26.009+0000 7f1c7f762640 1 -- 172.21.15.33:0/3520064352 wait complete. 2024-09-16T00:17:26.710 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T00:17:26.710 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T00:17:26.710 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi033.front.sepia.ceph.com 2024-09-16T00:17:26.711 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T00:17:26.711 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T00:17:26.711 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T00:17:26.711 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T00:17:26.711 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T00:17:26.711 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T00:17:26.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:17:26.711 DEBUG:teuthology.orchestra.run.smithi033:> ip addr 2024-09-16T00:17:26.729 INFO:teuthology.orchestra.run.smithi033.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T00:17:26.729 INFO:teuthology.orchestra.run.smithi033.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T00:17:26.729 INFO:teuthology.orchestra.run.smithi033.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout: inet6 ::1/128 scope host 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether 0c:c4:7a:6b:fd:14 brd ff:ff:ff:ff:ff:ff 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp7s0f0 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether 0c:c4:7a:6b:fd:15 brd ff:ff:ff:ff:ff:ff 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp7s0f1 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether ac:1f:6b:a5:d2:94 brd ff:ff:ff:ff:ff:ff 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp1s0f0 2024-09-16T00:17:26.730 INFO:teuthology.orchestra.run.smithi033.stdout: inet 172.21.15.33/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-16T00:17:26.731 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft 42171sec preferred_lft 42171sec 2024-09-16T00:17:26.731 INFO:teuthology.orchestra.run.smithi033.stdout: inet6 fe80::ae1f:6bff:fea5:d294/64 scope link 2024-09-16T00:17:26.731 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-09-16T00:17:26.731 INFO:teuthology.orchestra.run.smithi033.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T00:17:26.731 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether ac:1f:6b:a5:d2:95 brd ff:ff:ff:ff:ff:ff 2024-09-16T00:17:26.731 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp1s0f1 2024-09-16T00:17:26.731 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-16T00:17:26.732 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T00:17:26.732 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-09-16T00:17:26.732 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link add name ceph-brx type bridge 2024-09-16T00:17:26.732 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip addr flush dev ceph-brx 2024-09-16T00:17:26.732 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set ceph-brx up 2024-09-16T00:17:26.732 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-16T00:17:26.732 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-09-16T00:17:26.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T00:17:26.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:17:26.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:17:26.876 DEBUG:teuthology.orchestra.run.smithi033:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T00:17:26.902 INFO:teuthology.orchestra.run.smithi033.stdout:1 2024-09-16T00:17:26.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:17:26.903 DEBUG:teuthology.orchestra.run.smithi033:> route 2024-09-16T00:17:26.961 INFO:teuthology.orchestra.run.smithi033.stdout:Kernel IP routing table 2024-09-16T00:17:26.961 INFO:teuthology.orchestra.run.smithi033.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T00:17:26.961 INFO:teuthology.orchestra.run.smithi033.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-16T00:17:26.961 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-16T00:17:26.961 INFO:teuthology.orchestra.run.smithi033.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T00:17:26.961 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T00:17:26.961 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-09-16T00:17:26.961 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-16T00:17:26.962 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-16T00:17:26.962 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-16T00:17:26.962 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-09-16T00:17:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T00:17:27.059 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:26 smithi033 ceph-mon[31243]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:17:27.059 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:26 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3520064352' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-16T00:17:27.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:17:27.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:17:27.091 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list 2024-09-16T00:17:27.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:17:27.107 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list-id 2024-09-16T00:17:27.163 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T00:17:27.163 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-09-16T00:17:27.163 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T00:17:27.163 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-16T00:17:27.164 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-09-16T00:17:27.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:26 smithi083 ceph-mon[35242]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:17:27.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:26 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3520064352' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-16T00:17:27.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:26 smithi060 ceph-mon[35185]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:17:27.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:26 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3520064352' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-16T00:17:27.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T00:17:27.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:17:27.283 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-16T00:17:27.283 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T00:17:27.283 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-09-16T00:17:27.283 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-16T00:17:27.283 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-16T00:17:27.283 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-16T00:17:27.283 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-16T00:17:27.284 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-16T00:17:27.284 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-09-16T00:17:27.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T00:17:27.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:17:27.401 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T00:17:27.401 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-09-16T00:17:27.401 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set brx.0 up 2024-09-16T00:17:27.401 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set dev brx.0 master ceph-brx 2024-09-16T00:17:27.401 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-09-16T00:17:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T00:17:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:17:27.481 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T00:17:27.481 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T00:17:27.497 INFO:teuthology.orchestra.run.smithi033.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T00:17:27.498 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T00:17:27.498 DEBUG:teuthology.orchestra.run.smithi033:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T00:17:27.555 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-09-16T00:17:27.555 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-09-16T00:17:27.555 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/656da54f-0733-45b2-aea2-3ad93d5723b8 2024-09-16T00:17:27.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:17:27.555 DEBUG:teuthology.orchestra.run.smithi033:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/656da54f-0733-45b2-aea2-3ad93d5723b8 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-09-16T00:17:27.640 INFO:teuthology.orchestra.run.smithi033.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-09-16T00:17:27.640 INFO:teuthology.orchestra.run.smithi033.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-09-16T00:17:27.779 INFO:teuthology.orchestra.run.smithi033.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-09-16T00:17:27.780 INFO:teuthology.orchestra.run.smithi033.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-09-16T00:17:27.780 INFO:teuthology.orchestra.run.smithi033.stdout:mount.ceph: resolved to: "172.21.15.33:3300,172.21.15.60:3300,172.21.15.83:3300" 2024-09-16T00:17:27.780 INFO:teuthology.orchestra.run.smithi033.stdout:mount.ceph: trying mount with new device syntax: 0@7dde3d1a-73bf-11ef-bceb-c7b262605968.cephfs=/volumes/qa/sv_1/656da54f-0733-45b2-aea2-3ad93d5723b8 2024-09-16T00:17:27.780 INFO:teuthology.orchestra.run.smithi033.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mon_addr=172.21.15.33:3300/172.21.15.60:3300/172.21.15.83:3300" will pass to kernel 2024-09-16T00:17:27.781 INFO:teuthology.orchestra.run.smithi033.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T00:17:27.781 INFO:teuthology.orchestra.run.smithi033.stdout: You just mounted a file system that supports labels which does not 2024-09-16T00:17:27.781 INFO:teuthology.orchestra.run.smithi033.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T00:17:27.781 INFO:teuthology.orchestra.run.smithi033.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T00:17:27.781 INFO:teuthology.orchestra.run.smithi033.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T00:17:27.782 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T00:17:27.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:17:27.783 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T00:17:27.814 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T00:17:27.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:27 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T00:17:27.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:17:27.912 INFO:teuthology.orchestra.run.smithi033.stdout:client15234 2024-09-16T00:17:27.912 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:17:27.912 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T00:17:27.929 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T00:17:28.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T00:17:28.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:17:28.027 INFO:teuthology.orchestra.run.smithi033.stdout:client15234 2024-09-16T00:17:28.028 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T00:17:28.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T00:17:28.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:17:28.085 INFO:teuthology.orchestra.run.smithi033.stdout:client15234 2024-09-16T00:17:28.085 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968.client15234/status') 2024-09-16T00:17:28.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T00:17:28.145 INFO:teuthology.orchestra.run.smithi033.stdout:instance: client.15234 (3)192.168.0.1:0/3011410190 2024-09-16T00:17:28.145 INFO:teuthology.orchestra.run.smithi033.stdout:blocklisted: false 2024-09-16T00:17:28.146 INFO:teuthology.orchestra.run.smithi033.stderr:0+1 records in 2024-09-16T00:17:28.146 INFO:teuthology.orchestra.run.smithi033.stderr:0+1 records out 2024-09-16T00:17:28.146 INFO:teuthology.orchestra.run.smithi033.stderr:70 bytes copied, 2.3787e-05 s, 2.9 MB/s 2024-09-16T00:17:28.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T00:17:28.154 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T00:17:28.164 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T00:17:28.164 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-09-16T00:17:28.164 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-09-16T00:17:28.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.268+0000 7f950f5a1640 1 -- 172.21.15.33:0/3671678907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508104f20 msgr2=0x7f9508107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:28.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.268+0000 7f950f5a1640 1 --2- 172.21.15.33:0/3671678907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508104f20 0x7f9508107310 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009900 tx=0x7f94f802a410 comp rx=0 tx=0).stop 2024-09-16T00:17:28.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.268+0000 7f950f5a1640 1 -- 172.21.15.33:0/3671678907 shutdown_connections 2024-09-16T00:17:28.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.268+0000 7f950f5a1640 1 --2- 172.21.15.33:0/3671678907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9508107850 0x7f9508109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.268+0000 7f950f5a1640 1 --2- 172.21.15.33:0/3671678907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508104f20 0x7f9508107310 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.268+0000 7f950f5a1640 1 --2- 172.21.15.33:0/3671678907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95081025f0 0x7f95081049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.268+0000 7f950f5a1640 1 -- 172.21.15.33:0/3671678907 >> 172.21.15.33:0/3671678907 conn(0x7f95080fc1f0 msgr2=0x7f95080fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:28.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.268+0000 7f950f5a1640 1 -- 172.21.15.33:0/3671678907 shutdown_connections 2024-09-16T00:17:28.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.269+0000 7f950f5a1640 1 -- 172.21.15.33:0/3671678907 wait complete. 2024-09-16T00:17:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.269+0000 7f950f5a1640 1 Processor -- start 2024-09-16T00:17:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.269+0000 7f950f5a1640 1 -- start start 2024-09-16T00:17:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.269+0000 7f950f5a1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95081025f0 0x7f9508197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.269+0000 7f950f5a1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9508104f20 0x7f9508198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950f5a1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9508107850 0x7f950819f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950f5a1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f950810d830 con 0x7f95081025f0 2024-09-16T00:17:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950f5a1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f950810d6b0 con 0x7f9508107850 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950f5a1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f950810d9b0 con 0x7f9508104f20 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950d33d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95081025f0 0x7f9508197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950d33d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95081025f0 0x7f9508197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34932/0 (socket says 172.21.15.33:34932) 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950d33d640 1 -- 172.21.15.33:0/2547694314 learned_addr learned my addr 172.21.15.33:0/2547694314 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950cb3c640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9508104f20 0x7f9508198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950db3e640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9508107850 0x7f950819f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950d33d640 1 -- 172.21.15.33:0/2547694314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9508104f20 msgr2=0x7f9508198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950d33d640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9508104f20 0x7f9508198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950d33d640 1 -- 172.21.15.33:0/2547694314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9508107850 msgr2=0x7f950819f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950d33d640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9508107850 0x7f950819f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950d33d640 1 -- 172.21.15.33:0/2547694314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950819fb70 con 0x7f95081025f0 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950d33d640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95081025f0 0x7f9508197f50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f950400e9f0 tx=0x7f950400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f94fe7fc640 1 -- 172.21.15.33:0/2547694314 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950400cd80 con 0x7f95081025f0 2024-09-16T00:17:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f94fe7fc640 1 -- 172.21.15.33:0/2547694314 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9504010910 con 0x7f95081025f0 2024-09-16T00:17:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f94fe7fc640 1 -- 172.21.15.33:0/2547694314 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504018990 con 0x7f95081025f0 2024-09-16T00:17:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950f5a1640 1 -- 172.21.15.33:0/2547694314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950819fe60 con 0x7f95081025f0 2024-09-16T00:17:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.270+0000 7f950f5a1640 1 -- 172.21.15.33:0/2547694314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95081a02d0 con 0x7f95081025f0 2024-09-16T00:17:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.271+0000 7f950f5a1640 1 -- 172.21.15.33:0/2547694314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95080fdce0 con 0x7f95081025f0 2024-09-16T00:17:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.273+0000 7f94fe7fc640 1 -- 172.21.15.33:0/2547694314 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9504010ac0 con 0x7f95081025f0 2024-09-16T00:17:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.274+0000 7f94fe7fc640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f94f40a32c0 0x7f94f40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.274+0000 7f950cb3c640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f94f40a32c0 0x7f94f40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.274+0000 7f94fe7fc640 1 -- 172.21.15.33:0/2547694314 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f95040d0430 con 0x7f95081025f0 2024-09-16T00:17:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.274+0000 7f950cb3c640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f94f40a32c0 0x7f94f40a5780 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009230 tx=0x7f94f80315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:28.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.277+0000 7f94fe7fc640 1 -- 172.21.15.33:0/2547694314 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f950409cd70 con 0x7f95081025f0 2024-09-16T00:17:28.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.492+0000 7f950f5a1640 1 -- 172.21.15.33:0/2547694314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f950802d7c0 con 0x7f95081025f0 2024-09-16T00:17:28.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.798+0000 7f94fe7fc640 1 -- 172.21.15.33:0/2547694314 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v26) ==== 118+0+0 (secure 0 0 0) 0x7f950409c670 con 0x7f95081025f0 2024-09-16T00:17:28.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.800+0000 7f950f5a1640 1 -- 172.21.15.33:0/2547694314 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f94f40a32c0 msgr2=0x7f94f40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:28.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.800+0000 7f950f5a1640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f94f40a32c0 0x7f94f40a5780 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009230 tx=0x7f94f80315b0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.800+0000 7f950f5a1640 1 -- 172.21.15.33:0/2547694314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95081025f0 msgr2=0x7f9508197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:28.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.800+0000 7f950f5a1640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95081025f0 0x7f9508197f50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f950400e9f0 tx=0x7f950400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.800+0000 7f950f5a1640 1 -- 172.21.15.33:0/2547694314 shutdown_connections 2024-09-16T00:17:28.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.800+0000 7f950f5a1640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f94f40a32c0 0x7f94f40a5780 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.800+0000 7f950f5a1640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9508107850 0x7f950819f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.800+0000 7f950f5a1640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9508104f20 0x7f9508198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.800+0000 7f950f5a1640 1 --2- 172.21.15.33:0/2547694314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f95081025f0 0x7f9508197f50 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.800+0000 7f950f5a1640 1 -- 172.21.15.33:0/2547694314 >> 172.21.15.33:0/2547694314 conn(0x7f95080fc1f0 msgr2=0x7f9508102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:28.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.800+0000 7f950f5a1640 1 -- 172.21.15.33:0/2547694314 shutdown_connections 2024-09-16T00:17:28.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.800+0000 7f950f5a1640 1 -- 172.21.15.33:0/2547694314 wait complete. 2024-09-16T00:17:28.812 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-09-16T00:17:28.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.918+0000 7f784970a640 1 -- 172.21.15.33:0/1487532814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7844100a40 msgr2=0x7f7844100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:28.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.918+0000 7f784970a640 1 --2- 172.21.15.33:0/1487532814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7844100a40 0x7f7844100e40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7834009ca0 tx=0x7f783402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.918+0000 7f784970a640 1 -- 172.21.15.33:0/1487532814 shutdown_connections 2024-09-16T00:17:28.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.918+0000 7f784970a640 1 --2- 172.21.15.33:0/1487532814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78441025f0 0x7f7844108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.918+0000 7f784970a640 1 --2- 172.21.15.33:0/1487532814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7844101c30 0x7f78441020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.918+0000 7f784970a640 1 --2- 172.21.15.33:0/1487532814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7844100a40 0x7f7844100e40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.918+0000 7f784970a640 1 -- 172.21.15.33:0/1487532814 >> 172.21.15.33:0/1487532814 conn(0x7f78440fc1c0 msgr2=0x7f78440fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:28.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.918+0000 7f784970a640 1 -- 172.21.15.33:0/1487532814 shutdown_connections 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.918+0000 7f784970a640 1 -- 172.21.15.33:0/1487532814 wait complete. 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.919+0000 7f784970a640 1 Processor -- start 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.919+0000 7f784970a640 1 -- start start 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.919+0000 7f784970a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7844100a40 0x7f7844197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.919+0000 7f784970a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7844101c30 0x7f7844198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.919+0000 7f784970a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78441025f0 0x7f784419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.919+0000 7f784970a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f784410c6a0 con 0x7f7844101c30 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.919+0000 7f784970a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f784410c520 con 0x7f7844100a40 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.919+0000 7f784970a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f784410c820 con 0x7f78441025f0 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.919+0000 7f78427fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7844101c30 0x7f7844198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f78427fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7844101c30 0x7f7844198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34956/0 (socket says 172.21.15.33:34956) 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f78427fc640 1 -- 172.21.15.33:0/734359782 learned_addr learned my addr 172.21.15.33:0/734359782 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:28.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f7842ffd640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7844100a40 0x7f7844197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:28.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f78437fe640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78441025f0 0x7f784419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:28.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f78427fc640 1 -- 172.21.15.33:0/734359782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78441025f0 msgr2=0x7f784419f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:28.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f78427fc640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78441025f0 0x7f784419f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f78427fc640 1 -- 172.21.15.33:0/734359782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7844100a40 msgr2=0x7f7844197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:28.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f78427fc640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7844100a40 0x7f7844197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:28.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f78427fc640 1 -- 172.21.15.33:0/734359782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f784419fb00 con 0x7f7844101c30 2024-09-16T00:17:28.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f78437fe640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78441025f0 0x7f784419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:28.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f7842ffd640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7844100a40 0x7f7844197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:28.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f78427fc640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7844101c30 0x7f7844198420 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f783800e9f0 tx=0x7f783800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:28.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f782bfff640 1 -- 172.21.15.33:0/734359782 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783800cd80 con 0x7f7844101c30 2024-09-16T00:17:28.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f784970a640 1 -- 172.21.15.33:0/734359782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f784419fdf0 con 0x7f7844101c30 2024-09-16T00:17:28.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f782bfff640 1 -- 172.21.15.33:0/734359782 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7838010910 con 0x7f7844101c30 2024-09-16T00:17:28.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f782bfff640 1 -- 172.21.15.33:0/734359782 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78380188f0 con 0x7f7844101c30 2024-09-16T00:17:28.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.920+0000 7f784970a640 1 -- 172.21.15.33:0/734359782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78441a02b0 con 0x7f7844101c30 2024-09-16T00:17:28.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.921+0000 7f784970a640 1 -- 172.21.15.33:0/734359782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78441a04e0 con 0x7f7844101c30 2024-09-16T00:17:28.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.923+0000 7f782bfff640 1 -- 172.21.15.33:0/734359782 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7838010ac0 con 0x7f7844101c30 2024-09-16T00:17:28.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.924+0000 7f782bfff640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f782c0a3240 0x7f782c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:28.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.924+0000 7f7842ffd640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f782c0a3240 0x7f782c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:28.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.924+0000 7f782bfff640 1 -- 172.21.15.33:0/734359782 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f78380d06d0 con 0x7f7844101c30 2024-09-16T00:17:28.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.925+0000 7f7842ffd640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f782c0a3240 0x7f782c0a5700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7834006010 tx=0x7f7834031bb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:28.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:28.927+0000 7f782bfff640 1 -- 172.21.15.33:0/734359782 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f783809cf10 con 0x7f7844101c30 2024-09-16T00:17:29.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:28 smithi033 ceph-mon[31243]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:17:29.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:28 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2547694314' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-16T00:17:29.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.134+0000 7f784970a640 1 -- 172.21.15.33:0/734359782 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f784406a120 con 0x7f782c0a3240 2024-09-16T00:17:29.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.150+0000 7f782bfff640 1 -- 172.21.15.33:0/734359782 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f784406a120 con 0x7f782c0a3240 2024-09-16T00:17:29.151 INFO:teuthology.orchestra.run.smithi033.stdout:{} 2024-09-16T00:17:29.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.153+0000 7f784970a640 1 -- 172.21.15.33:0/734359782 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f782c0a3240 msgr2=0x7f782c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.153+0000 7f784970a640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f782c0a3240 0x7f782c0a5700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7834006010 tx=0x7f7834031bb0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.153+0000 7f784970a640 1 -- 172.21.15.33:0/734359782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7844101c30 msgr2=0x7f7844198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.153+0000 7f784970a640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7844101c30 0x7f7844198420 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f783800e9f0 tx=0x7f783800eeb0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.153+0000 7f784970a640 1 -- 172.21.15.33:0/734359782 shutdown_connections 2024-09-16T00:17:29.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.153+0000 7f784970a640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f782c0a3240 0x7f782c0a5700 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.153+0000 7f784970a640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78441025f0 0x7f784419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.153+0000 7f784970a640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7844101c30 0x7f7844198420 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.153+0000 7f784970a640 1 --2- 172.21.15.33:0/734359782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7844100a40 0x7f7844197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.153+0000 7f784970a640 1 -- 172.21.15.33:0/734359782 >> 172.21.15.33:0/734359782 conn(0x7f78440fc1c0 msgr2=0x7f7844069a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:29.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.153+0000 7f784970a640 1 -- 172.21.15.33:0/734359782 shutdown_connections 2024-09-16T00:17:29.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.153+0000 7f784970a640 1 -- 172.21.15.33:0/734359782 wait complete. 2024-09-16T00:17:29.164 INFO:teuthology.run_tasks:Running task check-counter... 2024-09-16T00:17:29.177 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-09-16T00:17:29.258 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T00:17:29.259 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:28 smithi060 ceph-mon[35185]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:17:29.259 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:28 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2547694314' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-16T00:17:29.259 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:28 smithi083 ceph-mon[35242]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:17:29.259 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:28 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2547694314' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-16T00:17:29.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.363+0000 7fa138ef1640 1 -- 172.21.15.33:0/3419332869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa134107830 msgr2=0x7fa134109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.363+0000 7fa138ef1640 1 --2- 172.21.15.33:0/3419332869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa134107830 0x7fa134109ca0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009ca0 tx=0x7fa12c02a8c0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.363+0000 7fa138ef1640 1 -- 172.21.15.33:0/3419332869 shutdown_connections 2024-09-16T00:17:29.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.363+0000 7fa138ef1640 1 --2- 172.21.15.33:0/3419332869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa134107830 0x7fa134109ca0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.363+0000 7fa138ef1640 1 --2- 172.21.15.33:0/3419332869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa134104f00 0x7fa1341072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.363+0000 7fa138ef1640 1 --2- 172.21.15.33:0/3419332869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1341025d0 0x7fa1341049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.363+0000 7fa138ef1640 1 -- 172.21.15.33:0/3419332869 >> 172.21.15.33:0/3419332869 conn(0x7fa1340fc190 msgr2=0x7fa1340fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:29.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.363+0000 7fa138ef1640 1 -- 172.21.15.33:0/3419332869 shutdown_connections 2024-09-16T00:17:29.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.363+0000 7fa138ef1640 1 -- 172.21.15.33:0/3419332869 wait complete. 2024-09-16T00:17:29.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa138ef1640 1 Processor -- start 2024-09-16T00:17:29.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa138ef1640 1 -- start start 2024-09-16T00:17:29.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa138ef1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1341025d0 0x7fa134197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa138ef1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa134104f00 0x7fa1341983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa138ef1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa134107830 0x7fa13419f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa138ef1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa13410d7f0 con 0x7fa1341025d0 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa138ef1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa13410d670 con 0x7fa134107830 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa138ef1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa13410d970 con 0x7fa134104f00 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa13259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1341025d0 0x7fa134197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa13259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1341025d0 0x7fa134197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35000/0 (socket says 172.21.15.33:35000) 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa13259c640 1 -- 172.21.15.33:0/1372629487 learned_addr learned my addr 172.21.15.33:0/1372629487 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa132d9d640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa134107830 0x7fa13419f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa131d9b640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa134104f00 0x7fa1341983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa13259c640 1 -- 172.21.15.33:0/1372629487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa134104f00 msgr2=0x7fa1341983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa13259c640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa134104f00 0x7fa1341983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa13259c640 1 -- 172.21.15.33:0/1372629487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa134107830 msgr2=0x7fa13419f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa13259c640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa134107830 0x7fa13419f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.364+0000 7fa13259c640 1 -- 172.21.15.33:0/1372629487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13419fa90 con 0x7fa1341025d0 2024-09-16T00:17:29.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.365+0000 7fa13259c640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1341025d0 0x7fa134197e70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa12400d950 tx=0x7fa12400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:29.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.365+0000 7fa1237fe640 1 -- 172.21.15.33:0/1372629487 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124017070 con 0x7fa1341025d0 2024-09-16T00:17:29.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.365+0000 7fa1237fe640 1 -- 172.21.15.33:0/1372629487 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa124009c30 con 0x7fa1341025d0 2024-09-16T00:17:29.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.365+0000 7fa138ef1640 1 -- 172.21.15.33:0/1372629487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13419fd80 con 0x7fa1341025d0 2024-09-16T00:17:29.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.365+0000 7fa1237fe640 1 -- 172.21.15.33:0/1372629487 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12400f6b0 con 0x7fa1341025d0 2024-09-16T00:17:29.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.365+0000 7fa138ef1640 1 -- 172.21.15.33:0/1372629487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1341a0210 con 0x7fa1341025d0 2024-09-16T00:17:29.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.366+0000 7fa138ef1640 1 -- 172.21.15.33:0/1372629487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1340fdc80 con 0x7fa1341025d0 2024-09-16T00:17:29.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.368+0000 7fa1237fe640 1 -- 172.21.15.33:0/1372629487 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fa1240109d0 con 0x7fa1341025d0 2024-09-16T00:17:29.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.369+0000 7fa1237fe640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa11c0a3250 0x7fa11c0a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:29.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.369+0000 7fa131d9b640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa11c0a3250 0x7fa11c0a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:29.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.369+0000 7fa1237fe640 1 -- 172.21.15.33:0/1372629487 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa1240cfab0 con 0x7fa1341025d0 2024-09-16T00:17:29.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.369+0000 7fa131d9b640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa11c0a3250 0x7fa11c0a5710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa128005fd0 tx=0x7fa128005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:29.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.372+0000 7fa1237fe640 1 -- 172.21.15.33:0/1372629487 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa124009de0 con 0x7fa1341025d0 2024-09-16T00:17:29.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.549+0000 7fa138ef1640 1 -- 172.21.15.33:0/1372629487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa134062f30 con 0x7fa1341025d0 2024-09-16T00:17:29.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.551+0000 7fa1237fe640 1 -- 172.21.15.33:0/1372629487 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25076 (secure 0 0 0) 0x7fa12409c2f0 con 0x7fa1341025d0 2024-09-16T00:17:29.551 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:29.552 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":90,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:17:09.569270+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2e","mappings":[{"from":1,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:17:29.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.553+0000 7fa138ef1640 1 -- 172.21.15.33:0/1372629487 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa11c0a3250 msgr2=0x7fa11c0a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.553+0000 7fa138ef1640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa11c0a3250 0x7fa11c0a5710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa128005fd0 tx=0x7fa128005850 comp rx=0 tx=0).stop 2024-09-16T00:17:29.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.553+0000 7fa138ef1640 1 -- 172.21.15.33:0/1372629487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1341025d0 msgr2=0x7fa134197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.553+0000 7fa138ef1640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1341025d0 0x7fa134197e70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa12400d950 tx=0x7fa12400de10 comp rx=0 tx=0).stop 2024-09-16T00:17:29.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.553+0000 7fa138ef1640 1 -- 172.21.15.33:0/1372629487 shutdown_connections 2024-09-16T00:17:29.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.553+0000 7fa138ef1640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa11c0a3250 0x7fa11c0a5710 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.553+0000 7fa138ef1640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa134107830 0x7fa13419f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.553+0000 7fa138ef1640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa134104f00 0x7fa1341983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.553+0000 7fa138ef1640 1 --2- 172.21.15.33:0/1372629487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1341025d0 0x7fa134197e70 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.553+0000 7fa138ef1640 1 -- 172.21.15.33:0/1372629487 >> 172.21.15.33:0/1372629487 conn(0x7fa1340fc190 msgr2=0x7fa134102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:29.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.553+0000 7fa138ef1640 1 -- 172.21.15.33:0/1372629487 shutdown_connections 2024-09-16T00:17:29.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.553+0000 7fa138ef1640 1 -- 172.21.15.33:0/1372629487 wait complete. 2024-09-16T00:17:29.563 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:17:29 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:17:29.139+0000 7f75277f9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T00:17:29.563 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:17:29 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:17:29.139+0000 7f75277f9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T00:17:29.563 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:17:29 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:17:29.139+0000 7f75277f9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T00:17:29.563 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:17:29 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:17:29.139+0000 7f75277f9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T00:17:29.564 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:17:29 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:17:29.139+0000 7f75277f9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T00:17:29.565 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T00:10:24.062476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-16T00:15:02.871178+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-16T00:15:05.511764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-16T00:15:11.993036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-16T00:17:29.565 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T00:17:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.673+0000 7fd2ea178640 1 -- 172.21.15.33:0/3654742119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e4102600 msgr2=0x7fd2e4108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.673+0000 7fd2ea178640 1 --2- 172.21.15.33:0/3654742119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e4102600 0x7fd2e4108e90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009900 tx=0x7fd2dc02a410 comp rx=0 tx=0).stop 2024-09-16T00:17:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.673+0000 7fd2ea178640 1 -- 172.21.15.33:0/3654742119 shutdown_connections 2024-09-16T00:17:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.673+0000 7fd2ea178640 1 --2- 172.21.15.33:0/3654742119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e4102600 0x7fd2e4108e90 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.673+0000 7fd2ea178640 1 --2- 172.21.15.33:0/3654742119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2e4101c40 0x7fd2e41020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.673+0000 7fd2ea178640 1 --2- 172.21.15.33:0/3654742119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2e4100a50 0x7fd2e4100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.673+0000 7fd2ea178640 1 -- 172.21.15.33:0/3654742119 >> 172.21.15.33:0/3654742119 conn(0x7fd2e40fc190 msgr2=0x7fd2e40fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.673+0000 7fd2ea178640 1 -- 172.21.15.33:0/3654742119 shutdown_connections 2024-09-16T00:17:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.673+0000 7fd2ea178640 1 -- 172.21.15.33:0/3654742119 wait complete. 2024-09-16T00:17:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2ea178640 1 Processor -- start 2024-09-16T00:17:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2ea178640 1 -- start start 2024-09-16T00:17:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2ea178640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e4100a50 0x7fd2e41049e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2ea178640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2e4101c40 0x7fd2e4103020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2ea178640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2e4102600 0x7fd2e4103560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2ea178640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e410c720 con 0x7fd2e4100a50 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2ea178640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e410c5a0 con 0x7fd2e4102600 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2ea178640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e410c8a0 con 0x7fd2e4101c40 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2e37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e4100a50 0x7fd2e41049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2e37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e4100a50 0x7fd2e41049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35030/0 (socket says 172.21.15.33:35030) 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2e37fe640 1 -- 172.21.15.33:0/1740965523 learned_addr learned my addr 172.21.15.33:0/1740965523 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2e2ffd640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2e4101c40 0x7fd2e4103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.674+0000 7fd2e3fff640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2e4102600 0x7fd2e4103560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2e37fe640 1 -- 172.21.15.33:0/1740965523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2e4101c40 msgr2=0x7fd2e4103020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2e37fe640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2e4101c40 0x7fd2e4103020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2e37fe640 1 -- 172.21.15.33:0/1740965523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2e4102600 msgr2=0x7fd2e4103560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2e37fe640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2e4102600 0x7fd2e4103560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2e37fe640 1 -- 172.21.15.33:0/1740965523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e4103df0 con 0x7fd2e4100a50 2024-09-16T00:17:29.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2e3fff640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2e4102600 0x7fd2e4103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:29.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2e2ffd640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2e4101c40 0x7fd2e4103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:29.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2e37fe640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e4100a50 0x7fd2e41049e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400eb70 tx=0x7fd2d400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:29.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2e0ff9640 1 -- 172.21.15.33:0/1740965523 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4003b70 con 0x7fd2e4100a50 2024-09-16T00:17:29.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2e0ff9640 1 -- 172.21.15.33:0/1740965523 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2d400ee70 con 0x7fd2e4100a50 2024-09-16T00:17:29.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2ea178640 1 -- 172.21.15.33:0/1740965523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2e41a1fc0 con 0x7fd2e4100a50 2024-09-16T00:17:29.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2e0ff9640 1 -- 172.21.15.33:0/1740965523 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4010830 con 0x7fd2e4100a50 2024-09-16T00:17:29.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.675+0000 7fd2ea178640 1 -- 172.21.15.33:0/1740965523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2e41a23c0 con 0x7fd2e4100a50 2024-09-16T00:17:29.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.676+0000 7fd2ea178640 1 -- 172.21.15.33:0/1740965523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2e41a26b0 con 0x7fd2e4100a50 2024-09-16T00:17:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.677+0000 7fd2e0ff9640 1 -- 172.21.15.33:0/1740965523 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd2d4004020 con 0x7fd2e4100a50 2024-09-16T00:17:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.678+0000 7fd2e0ff9640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2d00a3250 0x7fd2d00a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.678+0000 7fd2e2ffd640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2d00a3250 0x7fd2d00a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.678+0000 7fd2e0ff9640 1 -- 172.21.15.33:0/1740965523 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd2d40d0560 con 0x7fd2e4100a50 2024-09-16T00:17:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.678+0000 7fd2e2ffd640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2d00a3250 0x7fd2d00a5710 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc005d00 tx=0x7fd2cc009550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:29.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.682+0000 7fd2e0ff9640 1 -- 172.21.15.33:0/1740965523 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd2d409ce00 con 0x7fd2e4100a50 2024-09-16T00:17:29.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.818+0000 7fd2e0ff9640 1 -- 172.21.15.33:0/1740965523 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fd2d4098d70 con 0x7fd2e4100a50 2024-09-16T00:17:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:29 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2547694314' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-16T00:17:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:29 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:17:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:29 smithi033 ceph-mon[31243]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:29 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1372629487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:17:29.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.863+0000 7fd2ea178640 1 -- 172.21.15.33:0/1740965523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd2e41a2ee0 con 0x7fd2e4100a50 2024-09-16T00:17:29.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.863+0000 7fd2e0ff9640 1 -- 172.21.15.33:0/1740965523 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fd2d4051b30 con 0x7fd2e4100a50 2024-09-16T00:17:29.864 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-09-16T00:17:29.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.865+0000 7fd2ea178640 1 -- 172.21.15.33:0/1740965523 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2d00a3250 msgr2=0x7fd2d00a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.865+0000 7fd2ea178640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2d00a3250 0x7fd2d00a5710 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc005d00 tx=0x7fd2cc009550 comp rx=0 tx=0).stop 2024-09-16T00:17:29.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.865+0000 7fd2ea178640 1 -- 172.21.15.33:0/1740965523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e4100a50 msgr2=0x7fd2e41049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.865+0000 7fd2ea178640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e4100a50 0x7fd2e41049e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400eb70 tx=0x7fd2d400c6a0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.865+0000 7fd2ea178640 1 -- 172.21.15.33:0/1740965523 shutdown_connections 2024-09-16T00:17:29.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.865+0000 7fd2ea178640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2d00a3250 0x7fd2d00a5710 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.865+0000 7fd2ea178640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2e4102600 0x7fd2e4103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.865+0000 7fd2ea178640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2e4101c40 0x7fd2e4103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.865+0000 7fd2ea178640 1 --2- 172.21.15.33:0/1740965523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e4100a50 0x7fd2e41049e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.865+0000 7fd2ea178640 1 -- 172.21.15.33:0/1740965523 >> 172.21.15.33:0/1740965523 conn(0x7fd2e40fc190 msgr2=0x7fd2e40fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:29.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.866+0000 7fd2ea178640 1 -- 172.21.15.33:0/1740965523 shutdown_connections 2024-09-16T00:17:29.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.866+0000 7fd2ea178640 1 -- 172.21.15.33:0/1740965523 wait complete. 2024-09-16T00:17:29.878 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-16T00:17:29.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.984+0000 7fb25653a640 1 -- 172.21.15.33:0/1864103540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb250102d20 msgr2=0x7fb2501031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.984+0000 7fb25653a640 1 --2- 172.21.15.33:0/1864103540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb250102d20 0x7fb2501031a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb238009900 tx=0x7fb23802a410 comp rx=0 tx=0).stop 2024-09-16T00:17:29.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.984+0000 7fb25653a640 1 -- 172.21.15.33:0/1864103540 shutdown_connections 2024-09-16T00:17:29.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.984+0000 7fb25653a640 1 --2- 172.21.15.33:0/1864103540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2501036e0 0x7fb250109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.984+0000 7fb25653a640 1 --2- 172.21.15.33:0/1864103540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb250102d20 0x7fb2501031a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.984+0000 7fb25653a640 1 --2- 172.21.15.33:0/1864103540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb250101b30 0x7fb250101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.984+0000 7fb25653a640 1 -- 172.21.15.33:0/1864103540 >> 172.21.15.33:0/1864103540 conn(0x7fb2500fd270 msgr2=0x7fb2500ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:29.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.984+0000 7fb25653a640 1 -- 172.21.15.33:0/1864103540 shutdown_connections 2024-09-16T00:17:29.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.984+0000 7fb25653a640 1 -- 172.21.15.33:0/1864103540 wait complete. 2024-09-16T00:17:29.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.985+0000 7fb25653a640 1 Processor -- start 2024-09-16T00:17:29.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.985+0000 7fb25653a640 1 -- start start 2024-09-16T00:17:29.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.985+0000 7fb25653a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb250101b30 0x7fb250075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:29.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.985+0000 7fb25653a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb250102d20 0x7fb2500745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:29.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.985+0000 7fb25653a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2501036e0 0x7fb250074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:29.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.985+0000 7fb25653a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb25010d790 con 0x7fb250102d20 2024-09-16T00:17:29.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.985+0000 7fb25653a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb25010d610 con 0x7fb250101b30 2024-09-16T00:17:29.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.985+0000 7fb25653a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb25010d910 con 0x7fb2501036e0 2024-09-16T00:17:29.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.985+0000 7fb24f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb250102d20 0x7fb2500745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:29.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.985+0000 7fb24f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb250102d20 0x7fb2500745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35050/0 (socket says 172.21.15.33:35050) 2024-09-16T00:17:29.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.985+0000 7fb24f7fe640 1 -- 172.21.15.33:0/1363530710 learned_addr learned my addr 172.21.15.33:0/1363530710 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:29.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb254ad7640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2501036e0 0x7fb250074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:29.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb24f7fe640 1 -- 172.21.15.33:0/1363530710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2501036e0 msgr2=0x7fb250074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb24ffff640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb250101b30 0x7fb250075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:29.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb24f7fe640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2501036e0 0x7fb250074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb24f7fe640 1 -- 172.21.15.33:0/1363530710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb250101b30 msgr2=0x7fb250075fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:29.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb24f7fe640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb250101b30 0x7fb250075fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:29.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb24f7fe640 1 -- 172.21.15.33:0/1363530710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2500753b0 con 0x7fb250102d20 2024-09-16T00:17:29.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb254ad7640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2501036e0 0x7fb250074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:29.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb24ffff640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb250101b30 0x7fb250075fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:29.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb24f7fe640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb250102d20 0x7fb2500745e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb238009230 tx=0x7fb238031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:29.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb24d7fa640 1 -- 172.21.15.33:0/1363530710 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23803f030 con 0x7fb250102d20 2024-09-16T00:17:29.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb24d7fa640 1 -- 172.21.15.33:0/1363530710 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb238003ad0 con 0x7fb250102d20 2024-09-16T00:17:29.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb24d7fa640 1 -- 172.21.15.33:0/1363530710 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb238035070 con 0x7fb250102d20 2024-09-16T00:17:29.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb25653a640 1 -- 172.21.15.33:0/1363530710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2501a1f00 con 0x7fb250102d20 2024-09-16T00:17:29.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.986+0000 7fb25653a640 1 -- 172.21.15.33:0/1363530710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2501a2370 con 0x7fb250102d20 2024-09-16T00:17:29.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.987+0000 7fb25653a640 1 -- 172.21.15.33:0/1363530710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb250108f90 con 0x7fb250102d20 2024-09-16T00:17:29.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.988+0000 7fb24d7fa640 1 -- 172.21.15.33:0/1363530710 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fb238042030 con 0x7fb250102d20 2024-09-16T00:17:29.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.989+0000 7fb24d7fa640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb23c0a3360 0x7fb23c0a5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:29.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.989+0000 7fb24ffff640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb23c0a3360 0x7fb23c0a5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:29.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.989+0000 7fb24d7fa640 1 -- 172.21.15.33:0/1363530710 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb2380f04b0 con 0x7fb250102d20 2024-09-16T00:17:29.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.989+0000 7fb24ffff640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb23c0a3360 0x7fb23c0a5820 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb250102b80 tx=0x7fb240005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:29.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:29.993+0000 7fb24d7fa640 1 -- 172.21.15.33:0/1363530710 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb2380bcb90 con 0x7fb250102d20 2024-09-16T00:17:30.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.168+0000 7fb25653a640 1 -- 172.21.15.33:0/1363530710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb25002da30 con 0x7fb250102d20 2024-09-16T00:17:30.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:29 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2547694314' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-16T00:17:30.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:29 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:17:30.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:29 smithi083 ceph-mon[35242]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:30.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:29 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1372629487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:17:30.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.169+0000 7fb24d7fa640 1 -- 172.21.15.33:0/1363530710 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fb2380bc490 con 0x7fb250102d20 2024-09-16T00:17:30.170 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-09-16T00:17:30.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.170+0000 7fb25653a640 1 -- 172.21.15.33:0/1363530710 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb23c0a3360 msgr2=0x7fb23c0a5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:30.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.170+0000 7fb25653a640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb23c0a3360 0x7fb23c0a5820 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb250102b80 tx=0x7fb240005f90 comp rx=0 tx=0).stop 2024-09-16T00:17:30.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.170+0000 7fb25653a640 1 -- 172.21.15.33:0/1363530710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb250102d20 msgr2=0x7fb2500745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:30.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.170+0000 7fb25653a640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb250102d20 0x7fb2500745e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb238009230 tx=0x7fb238031b60 comp rx=0 tx=0).stop 2024-09-16T00:17:30.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.171+0000 7fb25653a640 1 -- 172.21.15.33:0/1363530710 shutdown_connections 2024-09-16T00:17:30.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.171+0000 7fb25653a640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb23c0a3360 0x7fb23c0a5820 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.171+0000 7fb25653a640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2501036e0 0x7fb250074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.171+0000 7fb25653a640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb250102d20 0x7fb2500745e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.171+0000 7fb25653a640 1 --2- 172.21.15.33:0/1363530710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb250101b30 0x7fb250075fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.171+0000 7fb25653a640 1 -- 172.21.15.33:0/1363530710 >> 172.21.15.33:0/1363530710 conn(0x7fb2500fd270 msgr2=0x7fb250106160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:30.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.171+0000 7fb25653a640 1 -- 172.21.15.33:0/1363530710 shutdown_connections 2024-09-16T00:17:30.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.171+0000 7fb25653a640 1 -- 172.21.15.33:0/1363530710 wait complete. 2024-09-16T00:17:30.182 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-16T00:17:30.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:29 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2547694314' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-16T00:17:30.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:29 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:17:30.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:29 smithi060 ceph-mon[35185]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T00:17:30.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:29 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1372629487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:17:30.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.289+0000 7f02e242c640 1 -- 172.21.15.33:0/1621591674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02dc1025e0 msgr2=0x7f02dc108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:30.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.289+0000 7f02e242c640 1 --2- 172.21.15.33:0/1621591674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02dc1025e0 0x7f02dc108e70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f02d4009ca0 tx=0x7f02d402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.289+0000 7f02e242c640 1 -- 172.21.15.33:0/1621591674 shutdown_connections 2024-09-16T00:17:30.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.289+0000 7f02e242c640 1 --2- 172.21.15.33:0/1621591674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02dc1025e0 0x7f02dc108e70 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.289+0000 7f02e242c640 1 --2- 172.21.15.33:0/1621591674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02dc101c20 0x7f02dc1020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.289+0000 7f02e242c640 1 --2- 172.21.15.33:0/1621591674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02dc100a30 0x7f02dc100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.289+0000 7f02e242c640 1 -- 172.21.15.33:0/1621591674 >> 172.21.15.33:0/1621591674 conn(0x7f02dc0fc190 msgr2=0x7f02dc0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:30.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.289+0000 7f02e242c640 1 -- 172.21.15.33:0/1621591674 shutdown_connections 2024-09-16T00:17:30.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.289+0000 7f02e242c640 1 -- 172.21.15.33:0/1621591674 wait complete. 2024-09-16T00:17:30.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.289+0000 7f02e242c640 1 Processor -- start 2024-09-16T00:17:30.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.289+0000 7f02e242c640 1 -- start start 2024-09-16T00:17:30.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02e242c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02dc100a30 0x7f02dc197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:30.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02e242c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02dc101c20 0x7f02dc198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:30.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02e242c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02dc1025e0 0x7f02dc19f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:30.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02e242c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f02dc10c690 con 0x7f02dc101c20 2024-09-16T00:17:30.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02e242c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f02dc10c510 con 0x7f02dc100a30 2024-09-16T00:17:30.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02e242c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f02dc10c810 con 0x7f02dc1025e0 2024-09-16T00:17:30.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02db7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02dc101c20 0x7f02dc198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:30.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02db7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02dc101c20 0x7f02dc198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36206/0 (socket says 172.21.15.33:36206) 2024-09-16T00:17:30.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02db7fe640 1 -- 172.21.15.33:0/1600949209 learned_addr learned my addr 172.21.15.33:0/1600949209 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:30.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02dbfff640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02dc100a30 0x7f02dc197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:30.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02e09c9640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02dc1025e0 0x7f02dc19f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:30.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02db7fe640 1 -- 172.21.15.33:0/1600949209 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02dc1025e0 msgr2=0x7f02dc19f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:30.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02db7fe640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02dc1025e0 0x7f02dc19f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02db7fe640 1 -- 172.21.15.33:0/1600949209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02dc100a30 msgr2=0x7f02dc197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:30.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02db7fe640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02dc100a30 0x7f02dc197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02db7fe640 1 -- 172.21.15.33:0/1600949209 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02dc19fa30 con 0x7f02dc101c20 2024-09-16T00:17:30.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02db7fe640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02dc101c20 0x7f02dc198350 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f02d000c890 tx=0x7f02d000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:30.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02d97fa640 1 -- 172.21.15.33:0/1600949209 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d0004220 con 0x7f02dc101c20 2024-09-16T00:17:30.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02d97fa640 1 -- 172.21.15.33:0/1600949209 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02d00043c0 con 0x7f02dc101c20 2024-09-16T00:17:30.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.290+0000 7f02e242c640 1 -- 172.21.15.33:0/1600949209 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02dc19fd20 con 0x7f02dc101c20 2024-09-16T00:17:30.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.291+0000 7f02d97fa640 1 -- 172.21.15.33:0/1600949209 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d00049e0 con 0x7f02dc101c20 2024-09-16T00:17:30.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.291+0000 7f02e242c640 1 -- 172.21.15.33:0/1600949209 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02dc1a0210 con 0x7f02dc101c20 2024-09-16T00:17:30.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.292+0000 7f02e242c640 1 -- 172.21.15.33:0/1600949209 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02dc1a04a0 con 0x7f02dc101c20 2024-09-16T00:17:30.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.294+0000 7f02d97fa640 1 -- 172.21.15.33:0/1600949209 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f02d00079b0 con 0x7f02dc101c20 2024-09-16T00:17:30.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.294+0000 7f02d97fa640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f02c40a32e0 0x7f02c40a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:30.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.294+0000 7f02dbfff640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f02c40a32e0 0x7f02c40a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:30.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.294+0000 7f02d97fa640 1 -- 172.21.15.33:0/1600949209 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f02d00cff80 con 0x7f02dc101c20 2024-09-16T00:17:30.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.295+0000 7f02dbfff640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f02c40a32e0 0x7f02c40a57a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f02dc101a80 tx=0x7f02cc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:30.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.297+0000 7f02d97fa640 1 -- 172.21.15.33:0/1600949209 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f02d0014030 con 0x7f02dc101c20 2024-09-16T00:17:30.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.472+0000 7f02e242c640 1 -- 172.21.15.33:0/1600949209 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f02dc1a0670 con 0x7f02dc101c20 2024-09-16T00:17:30.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.472+0000 7f02d97fa640 1 -- 172.21.15.33:0/1600949209 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f02d009c6c0 con 0x7f02dc101c20 2024-09-16T00:17:30.473 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-09-16T00:17:30.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.474+0000 7f02e242c640 1 -- 172.21.15.33:0/1600949209 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f02c40a32e0 msgr2=0x7f02c40a57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:30.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.474+0000 7f02e242c640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f02c40a32e0 0x7f02c40a57a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f02dc101a80 tx=0x7f02cc00b390 comp rx=0 tx=0).stop 2024-09-16T00:17:30.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.474+0000 7f02e242c640 1 -- 172.21.15.33:0/1600949209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02dc101c20 msgr2=0x7f02dc198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:30.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.474+0000 7f02e242c640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02dc101c20 0x7f02dc198350 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f02d000c890 tx=0x7f02d000cd50 comp rx=0 tx=0).stop 2024-09-16T00:17:30.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.474+0000 7f02e242c640 1 -- 172.21.15.33:0/1600949209 shutdown_connections 2024-09-16T00:17:30.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.474+0000 7f02e242c640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f02c40a32e0 0x7f02c40a57a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.474+0000 7f02e242c640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02dc1025e0 0x7f02dc19f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.474+0000 7f02e242c640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02dc101c20 0x7f02dc198350 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.474+0000 7f02e242c640 1 --2- 172.21.15.33:0/1600949209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02dc100a30 0x7f02dc197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.474+0000 7f02e242c640 1 -- 172.21.15.33:0/1600949209 >> 172.21.15.33:0/1600949209 conn(0x7f02dc0fc190 msgr2=0x7f02dc069520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:30.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.474+0000 7f02e242c640 1 -- 172.21.15.33:0/1600949209 shutdown_connections 2024-09-16T00:17:30.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.474+0000 7f02e242c640 1 -- 172.21.15.33:0/1600949209 wait complete. 2024-09-16T00:17:30.485 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-16T00:17:30.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.593+0000 7f9a74103640 1 -- 172.21.15.33:0/538408298 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a6c1025a0 msgr2=0x7f9a6c104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:30.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.593+0000 7f9a74103640 1 --2- 172.21.15.33:0/538408298 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a6c1025a0 0x7f9a6c104990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009900 tx=0x7f9a6802a410 comp rx=0 tx=0).stop 2024-09-16T00:17:30.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.593+0000 7f9a74103640 1 -- 172.21.15.33:0/538408298 shutdown_connections 2024-09-16T00:17:30.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.593+0000 7f9a74103640 1 --2- 172.21.15.33:0/538408298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a6c107800 0x7f9a6c109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.593+0000 7f9a74103640 1 --2- 172.21.15.33:0/538408298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6c104ed0 0x7f9a6c1072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.593+0000 7f9a74103640 1 --2- 172.21.15.33:0/538408298 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a6c1025a0 0x7f9a6c104990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.593+0000 7f9a74103640 1 -- 172.21.15.33:0/538408298 >> 172.21.15.33:0/538408298 conn(0x7f9a6c0fc180 msgr2=0x7f9a6c0fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.593+0000 7f9a74103640 1 -- 172.21.15.33:0/538408298 shutdown_connections 2024-09-16T00:17:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.593+0000 7f9a74103640 1 -- 172.21.15.33:0/538408298 wait complete. 2024-09-16T00:17:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.593+0000 7f9a74103640 1 Processor -- start 2024-09-16T00:17:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.593+0000 7f9a74103640 1 -- start start 2024-09-16T00:17:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a74103640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a6c1025a0 0x7f9a6c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a74103640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6c104ed0 0x7f9a6c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a74103640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a6c107800 0x7f9a6c19f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a74103640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c10d810 con 0x7f9a6c1025a0 2024-09-16T00:17:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a74103640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c10d690 con 0x7f9a6c104ed0 2024-09-16T00:17:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a74103640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c10d990 con 0x7f9a6c107800 2024-09-16T00:17:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a71e9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a6c1025a0 0x7f9a6c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a71e9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a6c1025a0 0x7f9a6c197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36212/0 (socket says 172.21.15.33:36212) 2024-09-16T00:17:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a71e9f640 1 -- 172.21.15.33:0/1561573109 learned_addr learned my addr 172.21.15.33:0/1561573109 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a726a0640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a6c107800 0x7f9a6c19f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a7169e640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6c104ed0 0x7f9a6c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a71e9f640 1 -- 172.21.15.33:0/1561573109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a6c107800 msgr2=0x7f9a6c19f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a71e9f640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a6c107800 0x7f9a6c19f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a71e9f640 1 -- 172.21.15.33:0/1561573109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6c104ed0 msgr2=0x7f9a6c198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a71e9f640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6c104ed0 0x7f9a6c198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a71e9f640 1 -- 172.21.15.33:0/1561573109 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6c19fb10 con 0x7f9a6c1025a0 2024-09-16T00:17:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a71e9f640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a6c1025a0 0x7f9a6c197ef0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009230 tx=0x7f9a68004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a62ffd640 1 -- 172.21.15.33:0/1561573109 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6803f030 con 0x7f9a6c1025a0 2024-09-16T00:17:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a62ffd640 1 -- 172.21.15.33:0/1561573109 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a68003ce0 con 0x7f9a6c1025a0 2024-09-16T00:17:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a74103640 1 -- 172.21.15.33:0/1561573109 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6c19fda0 con 0x7f9a6c1025a0 2024-09-16T00:17:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.594+0000 7f9a62ffd640 1 -- 172.21.15.33:0/1561573109 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68035070 con 0x7f9a6c1025a0 2024-09-16T00:17:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.595+0000 7f9a74103640 1 -- 172.21.15.33:0/1561573109 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a6c1a0220 con 0x7f9a6c1025a0 2024-09-16T00:17:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.595+0000 7f9a74103640 1 -- 172.21.15.33:0/1561573109 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a6c0fdc70 con 0x7f9a6c1025a0 2024-09-16T00:17:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.597+0000 7f9a62ffd640 1 -- 172.21.15.33:0/1561573109 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f9a68042030 con 0x7f9a6c1025a0 2024-09-16T00:17:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.597+0000 7f9a62ffd640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a580a3360 0x7f9a580a5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.598+0000 7f9a62ffd640 1 -- 172.21.15.33:0/1561573109 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9a680f0890 con 0x7f9a6c1025a0 2024-09-16T00:17:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.598+0000 7f9a7169e640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a580a3360 0x7f9a580a5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.598+0000 7f9a7169e640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a580a3360 0x7f9a580a5820 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c005d00 tx=0x7f9a5c005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.601+0000 7f9a62ffd640 1 -- 172.21.15.33:0/1561573109 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9a680bcfd0 con 0x7f9a6c1025a0 2024-09-16T00:17:30.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.776+0000 7f9a74103640 1 -- 172.21.15.33:0/1561573109 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9a6c062ef0 con 0x7f9a6c1025a0 2024-09-16T00:17:30.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.776+0000 7f9a62ffd640 1 -- 172.21.15.33:0/1561573109 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f9a680bc8d0 con 0x7f9a6c1025a0 2024-09-16T00:17:30.777 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-09-16T00:17:30.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.778+0000 7f9a74103640 1 -- 172.21.15.33:0/1561573109 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a580a3360 msgr2=0x7f9a580a5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:30.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.778+0000 7f9a74103640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a580a3360 0x7f9a580a5820 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c005d00 tx=0x7f9a5c005c50 comp rx=0 tx=0).stop 2024-09-16T00:17:30.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.778+0000 7f9a74103640 1 -- 172.21.15.33:0/1561573109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a6c1025a0 msgr2=0x7f9a6c197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:30.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.778+0000 7f9a74103640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a6c1025a0 0x7f9a6c197ef0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009230 tx=0x7f9a68004360 comp rx=0 tx=0).stop 2024-09-16T00:17:30.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.778+0000 7f9a74103640 1 -- 172.21.15.33:0/1561573109 shutdown_connections 2024-09-16T00:17:30.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.778+0000 7f9a74103640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a580a3360 0x7f9a580a5820 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.778+0000 7f9a74103640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a6c107800 0x7f9a6c19f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.778+0000 7f9a74103640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6c104ed0 0x7f9a6c198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.778+0000 7f9a74103640 1 --2- 172.21.15.33:0/1561573109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a6c1025a0 0x7f9a6c197ef0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:30.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.778+0000 7f9a74103640 1 -- 172.21.15.33:0/1561573109 >> 172.21.15.33:0/1561573109 conn(0x7f9a6c0fc180 msgr2=0x7f9a6c102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:30.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.778+0000 7f9a74103640 1 -- 172.21.15.33:0/1561573109 shutdown_connections 2024-09-16T00:17:30.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:30.778+0000 7f9a74103640 1 -- 172.21.15.33:0/1561573109 wait complete. 2024-09-16T00:17:30.789 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-09-16T00:17:30.789 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:17:31.047 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:31.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:30 smithi033 ceph-mon[31243]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T00:17:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:30 smithi033 ceph-mon[31243]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-09-16T00:17:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:30 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1740965523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T00:17:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:30 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1363530710' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-16T00:17:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:30 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1600949209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-16T00:17:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:30 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1561573109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-16T00:17:31.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:30 smithi083 ceph-mon[35242]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T00:17:31.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:30 smithi083 ceph-mon[35242]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-09-16T00:17:31.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:30 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1740965523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T00:17:31.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:30 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1363530710' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-16T00:17:31.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:30 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1600949209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-16T00:17:31.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:30 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1561573109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-16T00:17:31.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:30 smithi060 ceph-mon[35185]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T00:17:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:30 smithi060 ceph-mon[35185]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-09-16T00:17:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:30 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1740965523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T00:17:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:30 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1363530710' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-16T00:17:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:30 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1600949209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-16T00:17:31.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:30 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1561573109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-16T00:17:31.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.597+0000 7f013cc4a640 1 -- 172.21.15.33:0/1060099149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138100a80 msgr2=0x7f0138100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:31.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.597+0000 7f013cc4a640 1 --2- 172.21.15.33:0/1060099149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138100a80 0x7f0138100ee0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0120009930 tx=0x7f012002a420 comp rx=0 tx=0).stop 2024-09-16T00:17:31.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.597+0000 7f013cc4a640 1 -- 172.21.15.33:0/1060099149 shutdown_connections 2024-09-16T00:17:31.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.597+0000 7f013cc4a640 1 --2- 172.21.15.33:0/1060099149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0138101420 0x7f013810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:31.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.597+0000 7f013cc4a640 1 --2- 172.21.15.33:0/1060099149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138100a80 0x7f0138100ee0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:31.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.597+0000 7f013cc4a640 1 --2- 172.21.15.33:0/1060099149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0138106a80 0x7f0138106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:31.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.597+0000 7f013cc4a640 1 -- 172.21.15.33:0/1060099149 >> 172.21.15.33:0/1060099149 conn(0x7f01380fc7a0 msgr2=0x7f01380febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:31.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.597+0000 7f013cc4a640 1 -- 172.21.15.33:0/1060099149 shutdown_connections 2024-09-16T00:17:31.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.597+0000 7f013cc4a640 1 -- 172.21.15.33:0/1060099149 wait complete. 2024-09-16T00:17:31.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.598+0000 7f013cc4a640 1 Processor -- start 2024-09-16T00:17:31.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.598+0000 7f013cc4a640 1 -- start start 2024-09-16T00:17:31.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.598+0000 7f013cc4a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138100a80 0x7f01381a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:31.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.598+0000 7f013cc4a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0138101420 0x7f01381a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:31.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.598+0000 7f013cc4a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0138106a80 0x7f013819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:31.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.598+0000 7f013cc4a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0138110f20 con 0x7f0138100a80 2024-09-16T00:17:31.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.598+0000 7f013cc4a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0138110da0 con 0x7f0138101420 2024-09-16T00:17:31.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f013cc4a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f01381110a0 con 0x7f0138106a80 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f013659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138100a80 0x7f01381a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f013659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138100a80 0x7f01381a0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36222/0 (socket says 172.21.15.33:36222) 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f013659c640 1 -- 172.21.15.33:0/2755694435 learned_addr learned my addr 172.21.15.33:0/2755694435 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f012dd9b640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0138101420 0x7f01381a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f0136d9d640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0138106a80 0x7f013819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f013659c640 1 -- 172.21.15.33:0/2755694435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0138106a80 msgr2=0x7f013819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f013659c640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0138106a80 0x7f013819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f013659c640 1 -- 172.21.15.33:0/2755694435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0138101420 msgr2=0x7f01381a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f013659c640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0138101420 0x7f01381a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f013659c640 1 -- 172.21.15.33:0/2755694435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013819aec0 con 0x7f0138100a80 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f0136d9d640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0138106a80 0x7f013819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f013659c640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138100a80 0x7f01381a0530 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f012800b4a0 tx=0x7f012800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f012ffff640 1 -- 172.21.15.33:0/2755694435 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128014070 con 0x7f0138100a80 2024-09-16T00:17:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f012ffff640 1 -- 172.21.15.33:0/2755694435 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0128010af0 con 0x7f0138100a80 2024-09-16T00:17:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f012ffff640 1 -- 172.21.15.33:0/2755694435 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128017d10 con 0x7f0138100a80 2024-09-16T00:17:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.599+0000 7f013cc4a640 1 -- 172.21.15.33:0/2755694435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013819b1b0 con 0x7f0138100a80 2024-09-16T00:17:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.600+0000 7f013cc4a640 1 -- 172.21.15.33:0/2755694435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01381a7520 con 0x7f0138100a80 2024-09-16T00:17:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.600+0000 7f013cc4a640 1 -- 172.21.15.33:0/2755694435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01381021c0 con 0x7f0138100a80 2024-09-16T00:17:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.602+0000 7f012ffff640 1 -- 172.21.15.33:0/2755694435 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f0128017070 con 0x7f0138100a80 2024-09-16T00:17:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.602+0000 7f012ffff640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01240a3350 0x7f01240a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.603+0000 7f012dd9b640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01240a3350 0x7f01240a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.603+0000 7f012ffff640 1 -- 172.21.15.33:0/2755694435 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f01280cf630 con 0x7f0138100a80 2024-09-16T00:17:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.603+0000 7f012dd9b640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01240a3350 0x7f01240a5810 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0120000c00 tx=0x7f0120031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.606+0000 7f012ffff640 1 -- 172.21.15.33:0/2755694435 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f012809be20 con 0x7f0138100a80 2024-09-16T00:17:31.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.819+0000 7f013cc4a640 1 -- 172.21.15.33:0/2755694435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0138100f60 con 0x7f0138100a80 2024-09-16T00:17:31.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.820+0000 7f012ffff640 1 -- 172.21.15.33:0/2755694435 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11903 (secure 0 0 0) 0x7f0138100f60 con 0x7f0138100a80 2024-09-16T00:17:31.820 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:31.821 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":26,"btime":"2024-09-16T00:17:28:789138+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:17:28.493434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:17:31.821 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 26 2024-09-16T00:17:31.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.822+0000 7f013cc4a640 1 -- 172.21.15.33:0/2755694435 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01240a3350 msgr2=0x7f01240a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:31.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.822+0000 7f013cc4a640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01240a3350 0x7f01240a5810 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0120000c00 tx=0x7f0120031490 comp rx=0 tx=0).stop 2024-09-16T00:17:31.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.822+0000 7f013cc4a640 1 -- 172.21.15.33:0/2755694435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138100a80 msgr2=0x7f01381a0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:31.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.822+0000 7f013cc4a640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138100a80 0x7f01381a0530 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f012800b4a0 tx=0x7f012800b960 comp rx=0 tx=0).stop 2024-09-16T00:17:31.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.822+0000 7f013cc4a640 1 -- 172.21.15.33:0/2755694435 shutdown_connections 2024-09-16T00:17:31.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.822+0000 7f013cc4a640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01240a3350 0x7f01240a5810 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:31.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.822+0000 7f013cc4a640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0138106a80 0x7f013819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:31.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.822+0000 7f013cc4a640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0138101420 0x7f01381a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:31.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.822+0000 7f013cc4a640 1 --2- 172.21.15.33:0/2755694435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138100a80 0x7f01381a0530 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:31.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.822+0000 7f013cc4a640 1 -- 172.21.15.33:0/2755694435 >> 172.21.15.33:0/2755694435 conn(0x7f01380fc7a0 msgr2=0x7f013810c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:31.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.823+0000 7f013cc4a640 1 -- 172.21.15.33:0/2755694435 shutdown_connections 2024-09-16T00:17:31.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:31.823+0000 7f013cc4a640 1 -- 172.21.15.33:0/2755694435 wait complete. 2024-09-16T00:17:32.548 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-09-16T00:17:32.548 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-09-16T00:17:32.548 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T00:17:32.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.656+0000 7f349b766640 1 -- 172.21.15.33:0/3784371015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3494102d30 msgr2=0x7f34941031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:32.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.656+0000 7f349b766640 1 --2- 172.21.15.33:0/3784371015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3494102d30 0x7f34941031b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3484009900 tx=0x7f348402a410 comp rx=0 tx=0).stop 2024-09-16T00:17:32.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 -- 172.21.15.33:0/3784371015 shutdown_connections 2024-09-16T00:17:32.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 --2- 172.21.15.33:0/3784371015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34941036f0 0x7f3494109f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:32.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 --2- 172.21.15.33:0/3784371015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3494102d30 0x7f34941031b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:32.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 --2- 172.21.15.33:0/3784371015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3494101b40 0x7f3494101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:32.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 -- 172.21.15.33:0/3784371015 >> 172.21.15.33:0/3784371015 conn(0x7f34940fd2a0 msgr2=0x7f34940ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:32.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 -- 172.21.15.33:0/3784371015 shutdown_connections 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 -- 172.21.15.33:0/3784371015 wait complete. 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 Processor -- start 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 -- start start 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3494101b40 0x7f3494197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3494102d30 0x7f3494198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34941036f0 0x7f349419f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f349410d7a0 con 0x7f3494102d30 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f349410d620 con 0x7f3494101b40 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.657+0000 7f349b766640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f349410d920 con 0x7f34941036f0 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f3498d01640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3494102d30 0x7f3494198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f3498d01640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3494102d30 0x7f3494198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36236/0 (socket says 172.21.15.33:36236) 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f3498d01640 1 -- 172.21.15.33:0/308802131 learned_addr learned my addr 172.21.15.33:0/308802131 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:32.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f3499502640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3494101b40 0x7f3494197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:32.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f3498d01640 1 -- 172.21.15.33:0/308802131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34941036f0 msgr2=0x7f349419f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:32.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f3498d01640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34941036f0 0x7f349419f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:32.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f3498d01640 1 -- 172.21.15.33:0/308802131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3494101b40 msgr2=0x7f3494197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:32.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f3498d01640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3494101b40 0x7f3494197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:32.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f3498d01640 1 -- 172.21.15.33:0/308802131 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349419fa70 con 0x7f3494102d30 2024-09-16T00:17:32.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f3498d01640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3494102d30 0x7f3494198390 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3484009230 tx=0x7f3484004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:32.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f348a7fc640 1 -- 172.21.15.33:0/308802131 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348403f030 con 0x7f3494102d30 2024-09-16T00:17:32.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f348a7fc640 1 -- 172.21.15.33:0/308802131 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3484003ce0 con 0x7f3494102d30 2024-09-16T00:17:32.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f349b766640 1 -- 172.21.15.33:0/308802131 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f349419fd00 con 0x7f3494102d30 2024-09-16T00:17:32.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f348a7fc640 1 -- 172.21.15.33:0/308802131 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484035070 con 0x7f3494102d30 2024-09-16T00:17:32.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.658+0000 7f349b766640 1 -- 172.21.15.33:0/308802131 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34941a01a0 con 0x7f3494102d30 2024-09-16T00:17:32.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.659+0000 7f349b766640 1 -- 172.21.15.33:0/308802131 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3494108f80 con 0x7f3494102d30 2024-09-16T00:17:32.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.660+0000 7f348a7fc640 1 -- 172.21.15.33:0/308802131 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f3484042030 con 0x7f3494102d30 2024-09-16T00:17:32.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.661+0000 7f348a7fc640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f34800a32e0 0x7f34800a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:32.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.661+0000 7f348a7fc640 1 -- 172.21.15.33:0/308802131 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f34840f0910 con 0x7f3494102d30 2024-09-16T00:17:32.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.661+0000 7f3499502640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f34800a32e0 0x7f34800a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:32.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.662+0000 7f3499502640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f34800a32e0 0x7f34800a57a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3494102b90 tx=0x7f349000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:32.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.665+0000 7f348a7fc640 1 -- 172.21.15.33:0/308802131 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f34840bd050 con 0x7f3494102d30 2024-09-16T00:17:32.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.839+0000 7f349b766640 1 -- 172.21.15.33:0/308802131 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3494062630 con 0x7f34800a32e0 2024-09-16T00:17:32.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.847+0000 7f348a7fc640 1 -- 172.21.15.33:0/308802131 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340766 (secure 0 0 0) 0x7f3494062630 con 0x7f34800a32e0 2024-09-16T00:17:32.852 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:32.858 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-09-16T00:17:32.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.851+0000 7f349b766640 1 -- 172.21.15.33:0/308802131 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f34800a32e0 msgr2=0x7f34800a57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:32.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.851+0000 7f349b766640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f34800a32e0 0x7f34800a57a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3494102b90 tx=0x7f349000a2d0 comp rx=0 tx=0).stop 2024-09-16T00:17:32.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.851+0000 7f349b766640 1 -- 172.21.15.33:0/308802131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3494102d30 msgr2=0x7f3494198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:32.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.851+0000 7f349b766640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3494102d30 0x7f3494198390 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3484009230 tx=0x7f3484004360 comp rx=0 tx=0).stop 2024-09-16T00:17:32.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.851+0000 7f349b766640 1 -- 172.21.15.33:0/308802131 shutdown_connections 2024-09-16T00:17:32.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.851+0000 7f349b766640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f34800a32e0 0x7f34800a57a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:32.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.851+0000 7f349b766640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34941036f0 0x7f349419f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:32.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.851+0000 7f349b766640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3494102d30 0x7f3494198390 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:32.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.851+0000 7f349b766640 1 --2- 172.21.15.33:0/308802131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3494101b40 0x7f3494197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:32.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.851+0000 7f349b766640 1 -- 172.21.15.33:0/308802131 >> 172.21.15.33:0/308802131 conn(0x7f34940fd2a0 msgr2=0x7f3494106150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:32.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.851+0000 7f349b766640 1 -- 172.21.15.33:0/308802131 shutdown_connections 2024-09-16T00:17:32.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:32.851+0000 7f349b766640 1 -- 172.21.15.33:0/308802131 wait complete. 2024-09-16T00:17:32.863 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-16T00:17:31.670333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658627,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":748,"num_read_kb":482,"num_write":361,"num_write_kb":3065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1242843,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386388,"kb_used_data":12124,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555436,"statfs":{"total":1151940427776,"available":1151544766464,"internally_reserved":0,"allocated":12414976,"data_stored":8161167,"data_compressed":471663,"data_compressed_allocated":638976,"data_compressed_original":2809856,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":701,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":37,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003164"},"pg_stats":[{"pgid":"2.1f","version":"86'9","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578728+0000","last_change":"2024-09-16T00:16:09.034677+0000","last_active":"2024-09-16T00:17:09.578728+0000","last_peered":"2024-09-16T00:17:09.578728+0000","last_clean":"2024-09-16T00:17:09.578728+0000","last_became_active":"2024-09-16T00:16:09.024755+0000","last_became_peered":"2024-09-16T00:16:09.024755+0000","last_unstale":"2024-09-16T00:17:09.578728+0000","last_undegraded":"2024-09-16T00:17:09.578728+0000","last_fullsized":"2024-09-16T00:17:09.578728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:09.012210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579433+0000","last_change":"2024-09-16T00:15:06.547094+0000","last_active":"2024-09-16T00:17:09.579433+0000","last_peered":"2024-09-16T00:17:09.579433+0000","last_clean":"2024-09-16T00:17:09.579433+0000","last_became_active":"2024-09-16T00:15:06.542764+0000","last_became_peered":"2024-09-16T00:15:06.542764+0000","last_unstale":"2024-09-16T00:17:09.579433+0000","last_undegraded":"2024-09-16T00:17:09.579433+0000","last_fullsized":"2024-09-16T00:17:09.579433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:04:37.449577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574641+0000","last_change":"2024-09-16T00:15:13.578400+0000","last_active":"2024-09-16T00:17:09.574641+0000","last_peered":"2024-09-16T00:17:09.574641+0000","last_clean":"2024-09-16T00:17:09.574641+0000","last_became_active":"2024-09-16T00:15:13.577787+0000","last_became_peered":"2024-09-16T00:15:13.577787+0000","last_unstale":"2024-09-16T00:17:09.574641+0000","last_undegraded":"2024-09-16T00:17:09.574641+0000","last_fullsized":"2024-09-16T00:17:09.574641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:19:59.194152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579738+0000","last_change":"2024-09-16T00:15:04.373381+0000","last_active":"2024-09-16T00:17:09.579738+0000","last_peered":"2024-09-16T00:17:09.579738+0000","last_clean":"2024-09-16T00:17:09.579738+0000","last_became_active":"2024-09-16T00:15:04.373028+0000","last_became_peered":"2024-09-16T00:15:04.373028+0000","last_unstale":"2024-09-16T00:17:09.579738+0000","last_undegraded":"2024-09-16T00:17:09.579738+0000","last_fullsized":"2024-09-16T00:17:09.579738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:22.902583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577500+0000","last_change":"2024-09-16T00:15:06.554048+0000","last_active":"2024-09-16T00:17:09.577500+0000","last_peered":"2024-09-16T00:17:09.577500+0000","last_clean":"2024-09-16T00:17:09.577500+0000","last_became_active":"2024-09-16T00:15:06.553044+0000","last_became_peered":"2024-09-16T00:15:06.553044+0000","last_unstale":"2024-09-16T00:17:09.577500+0000","last_undegraded":"2024-09-16T00:17:09.577500+0000","last_fullsized":"2024-09-16T00:17:09.577500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:22:53.333705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574960+0000","last_change":"2024-09-16T00:15:13.586995+0000","last_active":"2024-09-16T00:17:09.574960+0000","last_peered":"2024-09-16T00:17:09.574960+0000","last_clean":"2024-09-16T00:17:09.574960+0000","last_became_active":"2024-09-16T00:15:13.586509+0000","last_became_peered":"2024-09-16T00:15:13.586509+0000","last_unstale":"2024-09-16T00:17:09.574960+0000","last_undegraded":"2024-09-16T00:17:09.574960+0000","last_fullsized":"2024-09-16T00:17:09.574960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:39.418581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579612+0000","last_change":"2024-09-16T00:15:04.348235+0000","last_active":"2024-09-16T00:17:09.579612+0000","last_peered":"2024-09-16T00:17:09.579612+0000","last_clean":"2024-09-16T00:17:09.579612+0000","last_became_active":"2024-09-16T00:15:04.347518+0000","last_became_peered":"2024-09-16T00:15:04.347518+0000","last_unstale":"2024-09-16T00:17:09.579612+0000","last_undegraded":"2024-09-16T00:17:09.579612+0000","last_fullsized":"2024-09-16T00:17:09.579612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:24.396911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581041+0000","last_change":"2024-09-16T00:15:06.554067+0000","last_active":"2024-09-16T00:17:09.581041+0000","last_peered":"2024-09-16T00:17:09.581041+0000","last_clean":"2024-09-16T00:17:09.581041+0000","last_became_active":"2024-09-16T00:15:06.553715+0000","last_became_peered":"2024-09-16T00:15:06.553715+0000","last_unstale":"2024-09-16T00:17:09.581041+0000","last_undegraded":"2024-09-16T00:17:09.581041+0000","last_fullsized":"2024-09-16T00:17:09.581041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:22.008530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575352+0000","last_change":"2024-09-16T00:15:13.586872+0000","last_active":"2024-09-16T00:17:09.575352+0000","last_peered":"2024-09-16T00:17:09.575352+0000","last_clean":"2024-09-16T00:17:09.575352+0000","last_became_active":"2024-09-16T00:15:13.586375+0000","last_became_peered":"2024-09-16T00:15:13.586375+0000","last_unstale":"2024-09-16T00:17:09.575352+0000","last_undegraded":"2024-09-16T00:17:09.575352+0000","last_fullsized":"2024-09-16T00:17:09.575352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:43.668827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579613+0000","last_change":"2024-09-16T00:15:04.347964+0000","last_active":"2024-09-16T00:17:09.579613+0000","last_peered":"2024-09-16T00:17:09.579613+0000","last_clean":"2024-09-16T00:17:09.579613+0000","last_became_active":"2024-09-16T00:15:04.347588+0000","last_became_peered":"2024-09-16T00:15:04.347588+0000","last_unstale":"2024-09-16T00:17:09.579613+0000","last_undegraded":"2024-09-16T00:17:09.579613+0000","last_fullsized":"2024-09-16T00:17:09.579613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:14.199287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576911+0000","last_change":"2024-09-16T00:15:06.544918+0000","last_active":"2024-09-16T00:17:09.576911+0000","last_peered":"2024-09-16T00:17:09.576911+0000","last_clean":"2024-09-16T00:17:09.576911+0000","last_became_active":"2024-09-16T00:15:06.544579+0000","last_became_peered":"2024-09-16T00:15:06.544579+0000","last_unstale":"2024-09-16T00:17:09.576911+0000","last_undegraded":"2024-09-16T00:17:09.576911+0000","last_fullsized":"2024-09-16T00:17:09.576911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:06:37.475644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575052+0000","last_change":"2024-09-16T00:15:13.581488+0000","last_active":"2024-09-16T00:17:09.575052+0000","last_peered":"2024-09-16T00:17:09.575052+0000","last_clean":"2024-09-16T00:17:09.575052+0000","last_became_active":"2024-09-16T00:15:13.581164+0000","last_became_peered":"2024-09-16T00:15:13.581164+0000","last_unstale":"2024-09-16T00:17:09.575052+0000","last_undegraded":"2024-09-16T00:17:09.575052+0000","last_fullsized":"2024-09-16T00:17:09.575052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:05.061592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581271+0000","last_change":"2024-09-16T00:15:06.551982+0000","last_active":"2024-09-16T00:17:09.581271+0000","last_peered":"2024-09-16T00:17:09.581271+0000","last_clean":"2024-09-16T00:17:09.581271+0000","last_became_active":"2024-09-16T00:15:06.551663+0000","last_became_peered":"2024-09-16T00:15:06.551663+0000","last_unstale":"2024-09-16T00:17:09.581271+0000","last_undegraded":"2024-09-16T00:17:09.581271+0000","last_fullsized":"2024-09-16T00:17:09.581271+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:11.928703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579394+0000","last_change":"2024-09-16T00:15:04.352237+0000","last_active":"2024-09-16T00:17:09.579394+0000","last_peered":"2024-09-16T00:17:09.579394+0000","last_clean":"2024-09-16T00:17:09.579394+0000","last_became_active":"2024-09-16T00:15:04.351884+0000","last_became_peered":"2024-09-16T00:15:04.351884+0000","last_unstale":"2024-09-16T00:17:09.579394+0000","last_undegraded":"2024-09-16T00:17:09.579394+0000","last_fullsized":"2024-09-16T00:17:09.579394+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:32.932020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575580+0000","last_change":"2024-09-16T00:15:13.638917+0000","last_active":"2024-09-16T00:17:09.575580+0000","last_peered":"2024-09-16T00:17:09.575580+0000","last_clean":"2024-09-16T00:17:09.575580+0000","last_became_active":"2024-09-16T00:15:13.638401+0000","last_became_peered":"2024-09-16T00:15:13.638401+0000","last_unstale":"2024-09-16T00:17:09.575580+0000","last_undegraded":"2024-09-16T00:17:09.575580+0000","last_fullsized":"2024-09-16T00:17:09.575580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:38.082407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579388+0000","last_change":"2024-09-16T00:15:06.547909+0000","last_active":"2024-09-16T00:17:09.579388+0000","last_peered":"2024-09-16T00:17:09.579388+0000","last_clean":"2024-09-16T00:17:09.579388+0000","last_became_active":"2024-09-16T00:15:06.546814+0000","last_became_peered":"2024-09-16T00:15:06.546814+0000","last_unstale":"2024-09-16T00:17:09.579388+0000","last_undegraded":"2024-09-16T00:17:09.579388+0000","last_fullsized":"2024-09-16T00:17:09.579388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:46.419406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780898+0000","last_change":"2024-09-16T00:16:09.027691+0000","last_active":"2024-09-16T00:17:09.780898+0000","last_peered":"2024-09-16T00:17:09.780898+0000","last_clean":"2024-09-16T00:17:09.780898+0000","last_became_active":"2024-09-16T00:16:09.023204+0000","last_became_peered":"2024-09-16T00:16:09.023204+0000","last_unstale":"2024-09-16T00:17:09.780898+0000","last_undegraded":"2024-09-16T00:17:09.780898+0000","last_fullsized":"2024-09-16T00:17:09.780898+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:44.647576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575627+0000","last_change":"2024-09-16T00:15:13.588345+0000","last_active":"2024-09-16T00:17:09.575627+0000","last_peered":"2024-09-16T00:17:09.575627+0000","last_clean":"2024-09-16T00:17:09.575627+0000","last_became_active":"2024-09-16T00:15:13.587829+0000","last_became_peered":"2024-09-16T00:15:13.587829+0000","last_unstale":"2024-09-16T00:17:09.575627+0000","last_undegraded":"2024-09-16T00:17:09.575627+0000","last_fullsized":"2024-09-16T00:17:09.575627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:04.261031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781137+0000","last_change":"2024-09-16T00:15:06.545485+0000","last_active":"2024-09-16T00:17:09.781137+0000","last_peered":"2024-09-16T00:17:09.781137+0000","last_clean":"2024-09-16T00:17:09.781137+0000","last_became_active":"2024-09-16T00:15:06.545178+0000","last_became_peered":"2024-09-16T00:15:06.545178+0000","last_unstale":"2024-09-16T00:17:09.781137+0000","last_undegraded":"2024-09-16T00:17:09.781137+0000","last_fullsized":"2024-09-16T00:17:09.781137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:26.923469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579782+0000","last_change":"2024-09-16T00:15:04.340546+0000","last_active":"2024-09-16T00:17:09.579782+0000","last_peered":"2024-09-16T00:17:09.579782+0000","last_clean":"2024-09-16T00:17:09.579782+0000","last_became_active":"2024-09-16T00:15:04.340173+0000","last_became_peered":"2024-09-16T00:15:04.340173+0000","last_unstale":"2024-09-16T00:17:09.579782+0000","last_undegraded":"2024-09-16T00:17:09.579782+0000","last_fullsized":"2024-09-16T00:17:09.579782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:35:01.134433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578541+0000","last_change":"2024-09-16T00:15:13.589963+0000","last_active":"2024-09-16T00:17:09.578541+0000","last_peered":"2024-09-16T00:17:09.578541+0000","last_clean":"2024-09-16T00:17:09.578541+0000","last_became_active":"2024-09-16T00:15:13.589615+0000","last_became_peered":"2024-09-16T00:15:13.589615+0000","last_unstale":"2024-09-16T00:17:09.578541+0000","last_undegraded":"2024-09-16T00:17:09.578541+0000","last_fullsized":"2024-09-16T00:17:09.578541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:37.197765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579731+0000","last_change":"2024-09-16T00:15:06.544374+0000","last_active":"2024-09-16T00:17:09.579731+0000","last_peered":"2024-09-16T00:17:09.579731+0000","last_clean":"2024-09-16T00:17:09.579731+0000","last_became_active":"2024-09-16T00:15:06.544027+0000","last_became_peered":"2024-09-16T00:15:06.544027+0000","last_unstale":"2024-09-16T00:17:09.579731+0000","last_undegraded":"2024-09-16T00:17:09.579731+0000","last_fullsized":"2024-09-16T00:17:09.579731+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580105+0000","last_change":"2024-09-16T00:15:04.374209+0000","last_active":"2024-09-16T00:17:09.580105+0000","last_peered":"2024-09-16T00:17:09.580105+0000","last_clean":"2024-09-16T00:17:09.580105+0000","last_became_active":"2024-09-16T00:15:04.373814+0000","last_became_peered":"2024-09-16T00:15:04.373814+0000","last_unstale":"2024-09-16T00:17:09.580105+0000","last_undegraded":"2024-09-16T00:17:09.580105+0000","last_fullsized":"2024-09-16T00:17:09.580105+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:13.516161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575489+0000","last_change":"2024-09-16T00:15:13.639210+0000","last_active":"2024-09-16T00:17:09.575489+0000","last_peered":"2024-09-16T00:17:09.575489+0000","last_clean":"2024-09-16T00:17:09.575489+0000","last_became_active":"2024-09-16T00:15:13.638719+0000","last_became_peered":"2024-09-16T00:15:13.638719+0000","last_unstale":"2024-09-16T00:17:09.575489+0000","last_undegraded":"2024-09-16T00:17:09.575489+0000","last_fullsized":"2024-09-16T00:17:09.575489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:26.130778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577341+0000","last_change":"2024-09-16T00:15:06.553108+0000","last_active":"2024-09-16T00:17:09.577341+0000","last_peered":"2024-09-16T00:17:09.577341+0000","last_clean":"2024-09-16T00:17:09.577341+0000","last_became_active":"2024-09-16T00:15:06.552817+0000","last_became_peered":"2024-09-16T00:15:06.552817+0000","last_unstale":"2024-09-16T00:17:09.577341+0000","last_undegraded":"2024-09-16T00:17:09.577341+0000","last_fullsized":"2024-09-16T00:17:09.577341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:06.821337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575833+0000","last_change":"2024-09-16T00:15:04.644185+0000","last_active":"2024-09-16T00:17:09.575833+0000","last_peered":"2024-09-16T00:17:09.575833+0000","last_clean":"2024-09-16T00:17:09.575833+0000","last_became_active":"2024-09-16T00:15:04.643887+0000","last_became_peered":"2024-09-16T00:15:04.643887+0000","last_unstale":"2024-09-16T00:17:09.575833+0000","last_undegraded":"2024-09-16T00:17:09.575833+0000","last_fullsized":"2024-09-16T00:17:09.575833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:44.279272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574877+0000","last_change":"2024-09-16T00:15:13.574761+0000","last_active":"2024-09-16T00:17:09.574877+0000","last_peered":"2024-09-16T00:17:09.574877+0000","last_clean":"2024-09-16T00:17:09.574877+0000","last_became_active":"2024-09-16T00:15:13.574347+0000","last_became_peered":"2024-09-16T00:15:13.574347+0000","last_unstale":"2024-09-16T00:17:09.574877+0000","last_undegraded":"2024-09-16T00:17:09.574877+0000","last_fullsized":"2024-09-16T00:17:09.574877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:22.040121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579504+0000","last_change":"2024-09-16T00:15:06.550338+0000","last_active":"2024-09-16T00:17:09.579504+0000","last_peered":"2024-09-16T00:17:09.579504+0000","last_clean":"2024-09-16T00:17:09.579504+0000","last_became_active":"2024-09-16T00:15:06.549868+0000","last_became_peered":"2024-09-16T00:15:06.549868+0000","last_unstale":"2024-09-16T00:17:09.579504+0000","last_undegraded":"2024-09-16T00:17:09.579504+0000","last_fullsized":"2024-09-16T00:17:09.579504+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:26.643338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'12","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:13.711683+0000","last_change":"2024-09-16T00:15:04.342306+0000","last_active":"2024-09-16T00:17:13.711683+0000","last_peered":"2024-09-16T00:17:13.711683+0000","last_clean":"2024-09-16T00:17:13.711683+0000","last_became_active":"2024-09-16T00:15:04.341964+0000","last_became_peered":"2024-09-16T00:15:04.341964+0000","last_unstale":"2024-09-16T00:17:13.711683+0000","last_undegraded":"2024-09-16T00:17:13.711683+0000","last_fullsized":"2024-09-16T00:17:13.711683+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:14.137199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578972+0000","last_change":"2024-09-16T00:15:13.638246+0000","last_active":"2024-09-16T00:17:09.578972+0000","last_peered":"2024-09-16T00:17:09.578972+0000","last_clean":"2024-09-16T00:17:09.578972+0000","last_became_active":"2024-09-16T00:15:13.637816+0000","last_became_peered":"2024-09-16T00:15:13.637816+0000","last_unstale":"2024-09-16T00:17:09.578972+0000","last_undegraded":"2024-09-16T00:17:09.578972+0000","last_fullsized":"2024-09-16T00:17:09.578972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:27.331156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718239+0000","last_change":"2024-09-16T00:15:06.544057+0000","last_active":"2024-09-16T00:17:09.718239+0000","last_peered":"2024-09-16T00:17:09.718239+0000","last_clean":"2024-09-16T00:17:09.718239+0000","last_became_active":"2024-09-16T00:15:06.543244+0000","last_became_peered":"2024-09-16T00:15:06.543244+0000","last_unstale":"2024-09-16T00:17:09.718239+0000","last_undegraded":"2024-09-16T00:17:09.718239+0000","last_fullsized":"2024-09-16T00:17:09.718239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:42:28.950014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579701+0000","last_change":"2024-09-16T00:15:04.376419+0000","last_active":"2024-09-16T00:17:09.579701+0000","last_peered":"2024-09-16T00:17:09.579701+0000","last_clean":"2024-09-16T00:17:09.579701+0000","last_became_active":"2024-09-16T00:15:04.375976+0000","last_became_peered":"2024-09-16T00:15:04.375976+0000","last_unstale":"2024-09-16T00:17:09.579701+0000","last_undegraded":"2024-09-16T00:17:09.579701+0000","last_fullsized":"2024-09-16T00:17:09.579701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:33:38.652109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781038+0000","last_change":"2024-09-16T00:15:13.584306+0000","last_active":"2024-09-16T00:17:09.781038+0000","last_peered":"2024-09-16T00:17:09.781038+0000","last_clean":"2024-09-16T00:17:09.781038+0000","last_became_active":"2024-09-16T00:15:13.583840+0000","last_became_peered":"2024-09-16T00:15:13.583840+0000","last_unstale":"2024-09-16T00:17:09.781038+0000","last_undegraded":"2024-09-16T00:17:09.781038+0000","last_fullsized":"2024-09-16T00:17:09.781038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:36.660458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577806+0000","last_change":"2024-09-16T00:15:06.538902+0000","last_active":"2024-09-16T00:17:09.577806+0000","last_peered":"2024-09-16T00:17:09.577806+0000","last_clean":"2024-09-16T00:17:09.577806+0000","last_became_active":"2024-09-16T00:15:06.538537+0000","last_became_peered":"2024-09-16T00:15:06.538537+0000","last_unstale":"2024-09-16T00:17:09.577806+0000","last_undegraded":"2024-09-16T00:17:09.577806+0000","last_fullsized":"2024-09-16T00:17:09.577806+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:38.411711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578013+0000","last_change":"2024-09-16T00:15:04.374762+0000","last_active":"2024-09-16T00:17:09.578013+0000","last_peered":"2024-09-16T00:17:09.578013+0000","last_clean":"2024-09-16T00:17:09.578013+0000","last_became_active":"2024-09-16T00:15:04.374394+0000","last_became_peered":"2024-09-16T00:15:04.374394+0000","last_unstale":"2024-09-16T00:17:09.578013+0000","last_undegraded":"2024-09-16T00:17:09.578013+0000","last_fullsized":"2024-09-16T00:17:09.578013+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:47.075021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579233+0000","last_change":"2024-09-16T00:15:13.638306+0000","last_active":"2024-09-16T00:17:09.579233+0000","last_peered":"2024-09-16T00:17:09.579233+0000","last_clean":"2024-09-16T00:17:09.579233+0000","last_became_active":"2024-09-16T00:15:13.637931+0000","last_became_peered":"2024-09-16T00:15:13.637931+0000","last_unstale":"2024-09-16T00:17:09.579233+0000","last_undegraded":"2024-09-16T00:17:09.579233+0000","last_fullsized":"2024-09-16T00:17:09.579233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:41.385174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577585+0000","last_change":"2024-09-16T00:15:06.548209+0000","last_active":"2024-09-16T00:17:09.577585+0000","last_peered":"2024-09-16T00:17:09.577585+0000","last_clean":"2024-09-16T00:17:09.577585+0000","last_became_active":"2024-09-16T00:15:06.547846+0000","last_became_peered":"2024-09-16T00:15:06.547846+0000","last_unstale":"2024-09-16T00:17:09.577585+0000","last_undegraded":"2024-09-16T00:17:09.577585+0000","last_fullsized":"2024-09-16T00:17:09.577585+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:15.821847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780889+0000","last_change":"2024-09-16T00:16:09.023918+0000","last_active":"2024-09-16T00:17:09.780889+0000","last_peered":"2024-09-16T00:17:09.780889+0000","last_clean":"2024-09-16T00:17:09.780889+0000","last_became_active":"2024-09-16T00:16:09.023230+0000","last_became_peered":"2024-09-16T00:16:09.023230+0000","last_unstale":"2024-09-16T00:17:09.780889+0000","last_undegraded":"2024-09-16T00:17:09.780889+0000","last_fullsized":"2024-09-16T00:17:09.780889+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:31.527863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717949+0000","last_change":"2024-09-16T00:15:13.579387+0000","last_active":"2024-09-16T00:17:09.717949+0000","last_peered":"2024-09-16T00:17:09.717949+0000","last_clean":"2024-09-16T00:17:09.717949+0000","last_became_active":"2024-09-16T00:15:13.578996+0000","last_became_peered":"2024-09-16T00:15:13.578996+0000","last_unstale":"2024-09-16T00:17:09.717949+0000","last_undegraded":"2024-09-16T00:17:09.717949+0000","last_fullsized":"2024-09-16T00:17:09.717949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:51.703060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579796+0000","last_change":"2024-09-16T00:17:09.579796+0000","last_active":"2024-09-16T00:17:09.579796+0000","last_peered":"2024-09-16T00:17:09.579796+0000","last_clean":"2024-09-16T00:17:09.579796+0000","last_became_active":"2024-09-16T00:17:09.579386+0000","last_became_peered":"2024-09-16T00:17:09.579386+0000","last_unstale":"2024-09-16T00:17:09.579796+0000","last_undegraded":"2024-09-16T00:17:09.579796+0000","last_fullsized":"2024-09-16T00:17:09.579796+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:39:42.571193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":145,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:29.146717+0000","last_change":"2024-09-16T00:15:04.373896+0000","last_active":"2024-09-16T00:17:29.146717+0000","last_peered":"2024-09-16T00:17:29.146717+0000","last_clean":"2024-09-16T00:17:29.146717+0000","last_became_active":"2024-09-16T00:15:04.373493+0000","last_became_peered":"2024-09-16T00:15:04.373493+0000","last_unstale":"2024-09-16T00:17:29.146717+0000","last_undegraded":"2024-09-16T00:17:29.146717+0000","last_fullsized":"2024-09-16T00:17:29.146717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:17:10.767267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":100,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718128+0000","last_change":"2024-09-16T00:15:13.638575+0000","last_active":"2024-09-16T00:17:09.718128+0000","last_peered":"2024-09-16T00:17:09.718128+0000","last_clean":"2024-09-16T00:17:09.718128+0000","last_became_active":"2024-09-16T00:15:13.638144+0000","last_became_peered":"2024-09-16T00:15:13.638144+0000","last_unstale":"2024-09-16T00:17:09.718128+0000","last_undegraded":"2024-09-16T00:17:09.718128+0000","last_fullsized":"2024-09-16T00:17:09.718128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:39.345011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718312+0000","last_change":"2024-09-16T00:15:06.548852+0000","last_active":"2024-09-16T00:17:09.718312+0000","last_peered":"2024-09-16T00:17:09.718312+0000","last_clean":"2024-09-16T00:17:09.718312+0000","last_became_active":"2024-09-16T00:15:06.548238+0000","last_became_peered":"2024-09-16T00:15:06.548238+0000","last_unstale":"2024-09-16T00:17:09.718312+0000","last_undegraded":"2024-09-16T00:17:09.718312+0000","last_fullsized":"2024-09-16T00:17:09.718312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:16.115338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580690+0000","last_change":"2024-09-16T00:15:04.350753+0000","last_active":"2024-09-16T00:17:09.580690+0000","last_peered":"2024-09-16T00:17:09.580690+0000","last_clean":"2024-09-16T00:17:09.580690+0000","last_became_active":"2024-09-16T00:15:04.350406+0000","last_became_peered":"2024-09-16T00:15:04.350406+0000","last_unstale":"2024-09-16T00:17:09.580690+0000","last_undegraded":"2024-09-16T00:17:09.580690+0000","last_fullsized":"2024-09-16T00:17:09.580690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:09:54.737287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576996+0000","last_change":"2024-09-16T00:15:13.639302+0000","last_active":"2024-09-16T00:17:09.576996+0000","last_peered":"2024-09-16T00:17:09.576996+0000","last_clean":"2024-09-16T00:17:09.576996+0000","last_became_active":"2024-09-16T00:15:13.638863+0000","last_became_peered":"2024-09-16T00:15:13.638863+0000","last_unstale":"2024-09-16T00:17:09.576996+0000","last_undegraded":"2024-09-16T00:17:09.576996+0000","last_fullsized":"2024-09-16T00:17:09.576996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:23.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582176+0000","last_change":"2024-09-16T00:17:09.582176+0000","last_active":"2024-09-16T00:17:09.582176+0000","last_peered":"2024-09-16T00:17:09.582176+0000","last_clean":"2024-09-16T00:17:09.582176+0000","last_became_active":"2024-09-16T00:17:09.581740+0000","last_became_peered":"2024-09-16T00:17:09.581740+0000","last_unstale":"2024-09-16T00:17:09.582176+0000","last_undegraded":"2024-09-16T00:17:09.582176+0000","last_fullsized":"2024-09-16T00:17:09.582176+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:57.176724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575861+0000","last_change":"2024-09-16T00:15:04.342216+0000","last_active":"2024-09-16T00:17:09.575861+0000","last_peered":"2024-09-16T00:17:09.575861+0000","last_clean":"2024-09-16T00:17:09.575861+0000","last_became_active":"2024-09-16T00:15:04.341763+0000","last_became_peered":"2024-09-16T00:15:04.341763+0000","last_unstale":"2024-09-16T00:17:09.575861+0000","last_undegraded":"2024-09-16T00:17:09.575861+0000","last_fullsized":"2024-09-16T00:17:09.575861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:44.938527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":350,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580892+0000","last_change":"2024-09-16T00:13:51.108571+0000","last_active":"2024-09-16T00:17:09.580892+0000","last_peered":"2024-09-16T00:17:09.580892+0000","last_clean":"2024-09-16T00:17:09.580892+0000","last_became_active":"2024-09-16T00:13:51.108088+0000","last_became_peered":"2024-09-16T00:13:51.108088+0000","last_unstale":"2024-09-16T00:17:09.580892+0000","last_undegraded":"2024-09-16T00:17:09.580892+0000","last_fullsized":"2024-09-16T00:17:09.580892+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_clean_scrub_stamp":"2024-09-16T00:10:24.533807+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:02.511064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581143+0000","last_change":"2024-09-16T00:15:13.588665+0000","last_active":"2024-09-16T00:17:09.581143+0000","last_peered":"2024-09-16T00:17:09.581143+0000","last_clean":"2024-09-16T00:17:09.581143+0000","last_became_active":"2024-09-16T00:15:13.588222+0000","last_became_peered":"2024-09-16T00:15:13.588222+0000","last_unstale":"2024-09-16T00:17:09.581143+0000","last_undegraded":"2024-09-16T00:17:09.581143+0000","last_fullsized":"2024-09-16T00:17:09.581143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:59:59.965962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581454+0000","last_change":"2024-09-16T00:15:06.553114+0000","last_active":"2024-09-16T00:17:09.581454+0000","last_peered":"2024-09-16T00:17:09.581454+0000","last_clean":"2024-09-16T00:17:09.581454+0000","last_became_active":"2024-09-16T00:15:06.552788+0000","last_became_peered":"2024-09-16T00:15:06.552788+0000","last_unstale":"2024-09-16T00:17:09.581454+0000","last_undegraded":"2024-09-16T00:17:09.581454+0000","last_fullsized":"2024-09-16T00:17:09.581454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:05.473847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575364+0000","last_change":"2024-09-16T00:16:09.030747+0000","last_active":"2024-09-16T00:17:09.575364+0000","last_peered":"2024-09-16T00:17:09.575364+0000","last_clean":"2024-09-16T00:17:09.575364+0000","last_became_active":"2024-09-16T00:16:09.021208+0000","last_became_peered":"2024-09-16T00:16:09.021208+0000","last_unstale":"2024-09-16T00:17:09.575364+0000","last_undegraded":"2024-09-16T00:17:09.575364+0000","last_fullsized":"2024-09-16T00:17:09.575364+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:40.605498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580422+0000","last_change":"2024-09-16T00:15:13.575606+0000","last_active":"2024-09-16T00:17:09.580422+0000","last_peered":"2024-09-16T00:17:09.580422+0000","last_clean":"2024-09-16T00:17:09.580422+0000","last_became_active":"2024-09-16T00:15:13.575131+0000","last_became_peered":"2024-09-16T00:15:13.575131+0000","last_unstale":"2024-09-16T00:17:09.580422+0000","last_undegraded":"2024-09-16T00:17:09.580422+0000","last_fullsized":"2024-09-16T00:17:09.580422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:41.636058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.584161+0000","last_change":"2024-09-16T00:17:09.583522+0000","last_active":"2024-09-16T00:17:09.584161+0000","last_peered":"2024-09-16T00:17:09.584161+0000","last_clean":"2024-09-16T00:17:09.584161+0000","last_became_active":"2024-09-16T00:17:09.583082+0000","last_became_peered":"2024-09-16T00:17:09.583082+0000","last_unstale":"2024-09-16T00:17:09.584161+0000","last_undegraded":"2024-09-16T00:17:09.584161+0000","last_fullsized":"2024-09-16T00:17:09.584161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:48.582126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577245+0000","last_change":"2024-09-16T00:15:04.349095+0000","last_active":"2024-09-16T00:17:09.577245+0000","last_peered":"2024-09-16T00:17:09.577245+0000","last_clean":"2024-09-16T00:17:09.577245+0000","last_became_active":"2024-09-16T00:15:04.348698+0000","last_became_peered":"2024-09-16T00:15:04.348698+0000","last_unstale":"2024-09-16T00:17:09.577245+0000","last_undegraded":"2024-09-16T00:17:09.577245+0000","last_fullsized":"2024-09-16T00:17:09.577245+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:42.282882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577535+0000","last_change":"2024-09-16T00:15:13.580494+0000","last_active":"2024-09-16T00:17:09.577535+0000","last_peered":"2024-09-16T00:17:09.577535+0000","last_clean":"2024-09-16T00:17:09.577535+0000","last_became_active":"2024-09-16T00:15:13.580091+0000","last_became_peered":"2024-09-16T00:15:13.580091+0000","last_unstale":"2024-09-16T00:17:09.577535+0000","last_undegraded":"2024-09-16T00:17:09.577535+0000","last_fullsized":"2024-09-16T00:17:09.577535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:17.314919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580727+0000","last_change":"2024-09-16T00:15:06.544859+0000","last_active":"2024-09-16T00:17:09.580727+0000","last_peered":"2024-09-16T00:17:09.580727+0000","last_clean":"2024-09-16T00:17:09.580727+0000","last_became_active":"2024-09-16T00:15:06.544289+0000","last_became_peered":"2024-09-16T00:15:06.544289+0000","last_unstale":"2024-09-16T00:17:09.580727+0000","last_undegraded":"2024-09-16T00:17:09.580727+0000","last_fullsized":"2024-09-16T00:17:09.580727+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:52.622477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718205+0000","last_change":"2024-09-16T00:15:04.351313+0000","last_active":"2024-09-16T00:17:09.718205+0000","last_peered":"2024-09-16T00:17:09.718205+0000","last_clean":"2024-09-16T00:17:09.718205+0000","last_became_active":"2024-09-16T00:15:04.350928+0000","last_became_peered":"2024-09-16T00:15:04.350928+0000","last_unstale":"2024-09-16T00:17:09.718205+0000","last_undegraded":"2024-09-16T00:17:09.718205+0000","last_fullsized":"2024-09-16T00:17:09.718205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:48.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576571+0000","last_change":"2024-09-16T00:15:13.578573+0000","last_active":"2024-09-16T00:17:09.576571+0000","last_peered":"2024-09-16T00:17:09.576571+0000","last_clean":"2024-09-16T00:17:09.576571+0000","last_became_active":"2024-09-16T00:15:13.578117+0000","last_became_peered":"2024-09-16T00:15:13.578117+0000","last_unstale":"2024-09-16T00:17:09.576571+0000","last_undegraded":"2024-09-16T00:17:09.576571+0000","last_fullsized":"2024-09-16T00:17:09.576571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:53.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781258+0000","last_change":"2024-09-16T00:15:06.553548+0000","last_active":"2024-09-16T00:17:09.781258+0000","last_peered":"2024-09-16T00:17:09.781258+0000","last_clean":"2024-09-16T00:17:09.781258+0000","last_became_active":"2024-09-16T00:15:06.552963+0000","last_became_peered":"2024-09-16T00:15:06.552963+0000","last_unstale":"2024-09-16T00:17:09.781258+0000","last_undegraded":"2024-09-16T00:17:09.781258+0000","last_fullsized":"2024-09-16T00:17:09.781258+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:43.455881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579776+0000","last_change":"2024-09-16T00:15:04.351005+0000","last_active":"2024-09-16T00:17:09.579776+0000","last_peered":"2024-09-16T00:17:09.579776+0000","last_clean":"2024-09-16T00:17:09.579776+0000","last_became_active":"2024-09-16T00:15:04.350614+0000","last_became_peered":"2024-09-16T00:15:04.350614+0000","last_unstale":"2024-09-16T00:17:09.579776+0000","last_undegraded":"2024-09-16T00:17:09.579776+0000","last_fullsized":"2024-09-16T00:17:09.579776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:39.196360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579501+0000","last_change":"2024-09-16T00:16:09.023796+0000","last_active":"2024-09-16T00:17:09.579501+0000","last_peered":"2024-09-16T00:17:09.579501+0000","last_clean":"2024-09-16T00:17:09.579501+0000","last_became_active":"2024-09-16T00:16:09.023403+0000","last_became_peered":"2024-09-16T00:16:09.023403+0000","last_unstale":"2024-09-16T00:17:09.579501+0000","last_undegraded":"2024-09-16T00:17:09.579501+0000","last_fullsized":"2024-09-16T00:17:09.579501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:59.488315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577059+0000","last_change":"2024-09-16T00:15:06.546325+0000","last_active":"2024-09-16T00:17:09.577059+0000","last_peered":"2024-09-16T00:17:09.577059+0000","last_clean":"2024-09-16T00:17:09.577059+0000","last_became_active":"2024-09-16T00:15:06.546028+0000","last_became_peered":"2024-09-16T00:15:06.546028+0000","last_unstale":"2024-09-16T00:17:09.577059+0000","last_undegraded":"2024-09-16T00:17:09.577059+0000","last_fullsized":"2024-09-16T00:17:09.577059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:15.049592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579065+0000","last_change":"2024-09-16T00:15:04.373818+0000","last_active":"2024-09-16T00:17:09.579065+0000","last_peered":"2024-09-16T00:17:09.579065+0000","last_clean":"2024-09-16T00:17:09.579065+0000","last_became_active":"2024-09-16T00:15:04.373406+0000","last_became_peered":"2024-09-16T00:15:04.373406+0000","last_unstale":"2024-09-16T00:17:09.579065+0000","last_undegraded":"2024-09-16T00:17:09.579065+0000","last_fullsized":"2024-09-16T00:17:09.579065+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:08.402660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718065+0000","last_change":"2024-09-16T00:15:13.588669+0000","last_active":"2024-09-16T00:17:09.718065+0000","last_peered":"2024-09-16T00:17:09.718065+0000","last_clean":"2024-09-16T00:17:09.718065+0000","last_became_active":"2024-09-16T00:15:13.588229+0000","last_became_peered":"2024-09-16T00:15:13.588229+0000","last_unstale":"2024-09-16T00:17:09.718065+0000","last_undegraded":"2024-09-16T00:17:09.718065+0000","last_fullsized":"2024-09-16T00:17:09.718065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:30.157296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580525+0000","last_change":"2024-09-16T00:15:06.550221+0000","last_active":"2024-09-16T00:17:09.580525+0000","last_peered":"2024-09-16T00:17:09.580525+0000","last_clean":"2024-09-16T00:17:09.580525+0000","last_became_active":"2024-09-16T00:15:06.549914+0000","last_became_peered":"2024-09-16T00:15:06.549914+0000","last_unstale":"2024-09-16T00:17:09.580525+0000","last_undegraded":"2024-09-16T00:17:09.580525+0000","last_fullsized":"2024-09-16T00:17:09.580525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:18:26.824523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:29.315869+0000","last_change":"2024-09-16T00:15:04.344389+0000","last_active":"2024-09-16T00:17:29.315869+0000","last_peered":"2024-09-16T00:17:29.315869+0000","last_clean":"2024-09-16T00:17:29.315869+0000","last_became_active":"2024-09-16T00:15:04.343956+0000","last_became_peered":"2024-09-16T00:15:04.343956+0000","last_unstale":"2024-09-16T00:17:29.315869+0000","last_undegraded":"2024-09-16T00:17:29.315869+0000","last_fullsized":"2024-09-16T00:17:29.315869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:34.019861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579707+0000","last_change":"2024-09-16T00:16:09.024050+0000","last_active":"2024-09-16T00:17:09.579707+0000","last_peered":"2024-09-16T00:17:09.579707+0000","last_clean":"2024-09-16T00:17:09.579707+0000","last_became_active":"2024-09-16T00:16:09.023371+0000","last_became_peered":"2024-09-16T00:16:09.023371+0000","last_unstale":"2024-09-16T00:17:09.579707+0000","last_undegraded":"2024-09-16T00:17:09.579707+0000","last_fullsized":"2024-09-16T00:17:09.579707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:35:19.020370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581433+0000","last_change":"2024-09-16T00:15:06.542744+0000","last_active":"2024-09-16T00:17:09.581433+0000","last_peered":"2024-09-16T00:17:09.581433+0000","last_clean":"2024-09-16T00:17:09.581433+0000","last_became_active":"2024-09-16T00:15:06.541868+0000","last_became_peered":"2024-09-16T00:15:06.541868+0000","last_unstale":"2024-09-16T00:17:09.581433+0000","last_undegraded":"2024-09-16T00:17:09.581433+0000","last_fullsized":"2024-09-16T00:17:09.581433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:10:06.508682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577336+0000","last_change":"2024-09-16T00:15:04.349556+0000","last_active":"2024-09-16T00:17:09.577336+0000","last_peered":"2024-09-16T00:17:09.577336+0000","last_clean":"2024-09-16T00:17:09.577336+0000","last_became_active":"2024-09-16T00:15:04.349242+0000","last_became_peered":"2024-09-16T00:15:04.349242+0000","last_unstale":"2024-09-16T00:17:09.577336+0000","last_undegraded":"2024-09-16T00:17:09.577336+0000","last_fullsized":"2024-09-16T00:17:09.577336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:22:18.459157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781094+0000","last_change":"2024-09-16T00:15:13.584109+0000","last_active":"2024-09-16T00:17:09.781094+0000","last_peered":"2024-09-16T00:17:09.781094+0000","last_clean":"2024-09-16T00:17:09.781094+0000","last_became_active":"2024-09-16T00:15:13.583718+0000","last_became_peered":"2024-09-16T00:15:13.583718+0000","last_unstale":"2024-09-16T00:17:09.781094+0000","last_undegraded":"2024-09-16T00:17:09.781094+0000","last_fullsized":"2024-09-16T00:17:09.781094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:29.416254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717952+0000","last_change":"2024-09-16T00:15:06.548247+0000","last_active":"2024-09-16T00:17:09.717952+0000","last_peered":"2024-09-16T00:17:09.717952+0000","last_clean":"2024-09-16T00:17:09.717952+0000","last_became_active":"2024-09-16T00:15:06.547864+0000","last_became_peered":"2024-09-16T00:15:06.547864+0000","last_unstale":"2024-09-16T00:17:09.717952+0000","last_undegraded":"2024-09-16T00:17:09.717952+0000","last_fullsized":"2024-09-16T00:17:09.717952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:25.350197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575352+0000","last_change":"2024-09-16T00:15:04.350965+0000","last_active":"2024-09-16T00:17:09.575352+0000","last_peered":"2024-09-16T00:17:09.575352+0000","last_clean":"2024-09-16T00:17:09.575352+0000","last_became_active":"2024-09-16T00:15:04.349124+0000","last_became_peered":"2024-09-16T00:15:04.349124+0000","last_unstale":"2024-09-16T00:17:09.575352+0000","last_undegraded":"2024-09-16T00:17:09.575352+0000","last_fullsized":"2024-09-16T00:17:09.575352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:40.859220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576904+0000","last_change":"2024-09-16T00:15:13.581718+0000","last_active":"2024-09-16T00:17:09.576904+0000","last_peered":"2024-09-16T00:17:09.576904+0000","last_clean":"2024-09-16T00:17:09.576904+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:17:09.576904+0000","last_undegraded":"2024-09-16T00:17:09.576904+0000","last_fullsized":"2024-09-16T00:17:09.576904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:08.810606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579507+0000","last_change":"2024-09-16T00:15:06.544507+0000","last_active":"2024-09-16T00:17:09.579507+0000","last_peered":"2024-09-16T00:17:09.579507+0000","last_clean":"2024-09-16T00:17:09.579507+0000","last_became_active":"2024-09-16T00:15:06.544178+0000","last_became_peered":"2024-09-16T00:15:06.544178+0000","last_unstale":"2024-09-16T00:17:09.579507+0000","last_undegraded":"2024-09-16T00:17:09.579507+0000","last_fullsized":"2024-09-16T00:17:09.579507+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:38.494266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'37","reported_seq":301,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:28.155901+0000","last_change":"2024-09-16T00:16:09.039763+0000","last_active":"2024-09-16T00:17:28.155901+0000","last_peered":"2024-09-16T00:17:28.155901+0000","last_clean":"2024-09-16T00:17:28.155901+0000","last_became_active":"2024-09-16T00:16:09.022939+0000","last_became_peered":"2024-09-16T00:16:09.022939+0000","last_unstale":"2024-09-16T00:17:28.155901+0000","last_undegraded":"2024-09-16T00:17:28.155901+0000","last_fullsized":"2024-09-16T00:17:28.155901+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:56.431159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63488,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":204,"num_read_kb":155,"num_write":38,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60943,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576838+0000","last_change":"2024-09-16T00:15:13.585928+0000","last_active":"2024-09-16T00:17:09.576838+0000","last_peered":"2024-09-16T00:17:09.576838+0000","last_clean":"2024-09-16T00:17:09.576838+0000","last_became_active":"2024-09-16T00:15:13.582940+0000","last_became_peered":"2024-09-16T00:15:13.582940+0000","last_unstale":"2024-09-16T00:17:09.576838+0000","last_undegraded":"2024-09-16T00:17:09.576838+0000","last_fullsized":"2024-09-16T00:17:09.576838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:57:13.251978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580875+0000","last_change":"2024-09-16T00:15:06.546238+0000","last_active":"2024-09-16T00:17:09.580875+0000","last_peered":"2024-09-16T00:17:09.580875+0000","last_clean":"2024-09-16T00:17:09.580875+0000","last_became_active":"2024-09-16T00:15:06.545919+0000","last_became_peered":"2024-09-16T00:15:06.545919+0000","last_unstale":"2024-09-16T00:17:09.580875+0000","last_undegraded":"2024-09-16T00:17:09.580875+0000","last_fullsized":"2024-09-16T00:17:09.580875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:09.180832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577083+0000","last_change":"2024-09-16T00:15:04.376028+0000","last_active":"2024-09-16T00:17:09.577083+0000","last_peered":"2024-09-16T00:17:09.577083+0000","last_clean":"2024-09-16T00:17:09.577083+0000","last_became_active":"2024-09-16T00:15:04.375633+0000","last_became_peered":"2024-09-16T00:15:04.375633+0000","last_unstale":"2024-09-16T00:17:09.577083+0000","last_undegraded":"2024-09-16T00:17:09.577083+0000","last_fullsized":"2024-09-16T00:17:09.577083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:44.446342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575222+0000","last_change":"2024-09-16T00:16:09.025524+0000","last_active":"2024-09-16T00:17:09.575222+0000","last_peered":"2024-09-16T00:17:09.575222+0000","last_clean":"2024-09-16T00:17:09.575222+0000","last_became_active":"2024-09-16T00:16:09.025035+0000","last_became_peered":"2024-09-16T00:16:09.025035+0000","last_unstale":"2024-09-16T00:17:09.575222+0000","last_undegraded":"2024-09-16T00:17:09.575222+0000","last_fullsized":"2024-09-16T00:17:09.575222+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:12.367244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577569+0000","last_change":"2024-09-16T00:15:06.550864+0000","last_active":"2024-09-16T00:17:09.577569+0000","last_peered":"2024-09-16T00:17:09.577569+0000","last_clean":"2024-09-16T00:17:09.577569+0000","last_became_active":"2024-09-16T00:15:06.547956+0000","last_became_peered":"2024-09-16T00:15:06.547956+0000","last_unstale":"2024-09-16T00:17:09.577569+0000","last_undegraded":"2024-09-16T00:17:09.577569+0000","last_fullsized":"2024-09-16T00:17:09.577569+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:26.102175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580635+0000","last_change":"2024-09-16T00:15:04.350349+0000","last_active":"2024-09-16T00:17:09.580635+0000","last_peered":"2024-09-16T00:17:09.580635+0000","last_clean":"2024-09-16T00:17:09.580635+0000","last_became_active":"2024-09-16T00:15:04.349951+0000","last_became_peered":"2024-09-16T00:15:04.349951+0000","last_unstale":"2024-09-16T00:17:09.580635+0000","last_undegraded":"2024-09-16T00:17:09.580635+0000","last_fullsized":"2024-09-16T00:17:09.580635+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:46.000831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577297+0000","last_change":"2024-09-16T00:16:09.025774+0000","last_active":"2024-09-16T00:17:09.577297+0000","last_peered":"2024-09-16T00:17:09.577297+0000","last_clean":"2024-09-16T00:17:09.577297+0000","last_became_active":"2024-09-16T00:16:09.025308+0000","last_became_peered":"2024-09-16T00:16:09.025308+0000","last_unstale":"2024-09-16T00:17:09.577297+0000","last_undegraded":"2024-09-16T00:17:09.577297+0000","last_fullsized":"2024-09-16T00:17:09.577297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:16.664850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580734+0000","last_change":"2024-09-16T00:15:06.552919+0000","last_active":"2024-09-16T00:17:09.580734+0000","last_peered":"2024-09-16T00:17:09.580734+0000","last_clean":"2024-09-16T00:17:09.580734+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:17:09.580734+0000","last_undegraded":"2024-09-16T00:17:09.580734+0000","last_fullsized":"2024-09-16T00:17:09.580734+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:47:38.651837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":139,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:26.311189+0000","last_change":"2024-09-16T00:15:04.351116+0000","last_active":"2024-09-16T00:17:26.311189+0000","last_peered":"2024-09-16T00:17:26.311189+0000","last_clean":"2024-09-16T00:17:26.311189+0000","last_became_active":"2024-09-16T00:15:04.350728+0000","last_became_peered":"2024-09-16T00:15:04.350728+0000","last_unstale":"2024-09-16T00:17:26.311189+0000","last_undegraded":"2024-09-16T00:17:26.311189+0000","last_fullsized":"2024-09-16T00:17:26.311189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:38.314019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579276+0000","last_change":"2024-09-16T00:15:13.575961+0000","last_active":"2024-09-16T00:17:09.579276+0000","last_peered":"2024-09-16T00:17:09.579276+0000","last_clean":"2024-09-16T00:17:09.579276+0000","last_became_active":"2024-09-16T00:15:13.575550+0000","last_became_peered":"2024-09-16T00:15:13.575550+0000","last_unstale":"2024-09-16T00:17:09.579276+0000","last_undegraded":"2024-09-16T00:17:09.579276+0000","last_fullsized":"2024-09-16T00:17:09.579276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:05.515588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577699+0000","last_change":"2024-09-16T00:15:13.583583+0000","last_active":"2024-09-16T00:17:09.577699+0000","last_peered":"2024-09-16T00:17:09.577699+0000","last_clean":"2024-09-16T00:17:09.577699+0000","last_became_active":"2024-09-16T00:15:13.583145+0000","last_became_peered":"2024-09-16T00:15:13.583145+0000","last_unstale":"2024-09-16T00:17:09.577699+0000","last_undegraded":"2024-09-16T00:17:09.577699+0000","last_fullsized":"2024-09-16T00:17:09.577699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:07.320337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.719269+0000","last_change":"2024-09-16T00:17:09.719269+0000","last_active":"2024-09-16T00:17:09.719269+0000","last_peered":"2024-09-16T00:17:09.719269+0000","last_clean":"2024-09-16T00:17:09.719269+0000","last_became_active":"2024-09-16T00:17:09.718887+0000","last_became_peered":"2024-09-16T00:17:09.718887+0000","last_unstale":"2024-09-16T00:17:09.719269+0000","last_undegraded":"2024-09-16T00:17:09.719269+0000","last_fullsized":"2024-09-16T00:17:09.719269+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:14:51.675738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577333+0000","last_change":"2024-09-16T00:15:04.350826+0000","last_active":"2024-09-16T00:17:09.577333+0000","last_peered":"2024-09-16T00:17:09.577333+0000","last_clean":"2024-09-16T00:17:09.577333+0000","last_became_active":"2024-09-16T00:15:04.350294+0000","last_became_peered":"2024-09-16T00:15:04.350294+0000","last_unstale":"2024-09-16T00:17:09.577333+0000","last_undegraded":"2024-09-16T00:17:09.577333+0000","last_fullsized":"2024-09-16T00:17:09.577333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:49.146709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781194+0000","last_change":"2024-09-16T00:15:13.580184+0000","last_active":"2024-09-16T00:17:09.781194+0000","last_peered":"2024-09-16T00:17:09.781194+0000","last_clean":"2024-09-16T00:17:09.781194+0000","last_became_active":"2024-09-16T00:15:13.579849+0000","last_became_peered":"2024-09-16T00:15:13.579849+0000","last_unstale":"2024-09-16T00:17:09.781194+0000","last_undegraded":"2024-09-16T00:17:09.781194+0000","last_fullsized":"2024-09-16T00:17:09.781194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:40.613127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579207+0000","last_change":"2024-09-16T00:15:06.545796+0000","last_active":"2024-09-16T00:17:09.579207+0000","last_peered":"2024-09-16T00:17:09.579207+0000","last_clean":"2024-09-16T00:17:09.579207+0000","last_became_active":"2024-09-16T00:15:06.544815+0000","last_became_peered":"2024-09-16T00:15:06.544815+0000","last_unstale":"2024-09-16T00:17:09.579207+0000","last_undegraded":"2024-09-16T00:17:09.579207+0000","last_fullsized":"2024-09-16T00:17:09.579207+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:51:25.921949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718347+0000","last_change":"2024-09-16T00:15:04.340101+0000","last_active":"2024-09-16T00:17:09.718347+0000","last_peered":"2024-09-16T00:17:09.718347+0000","last_clean":"2024-09-16T00:17:09.718347+0000","last_became_active":"2024-09-16T00:15:04.339788+0000","last_became_peered":"2024-09-16T00:15:04.339788+0000","last_unstale":"2024-09-16T00:17:09.718347+0000","last_undegraded":"2024-09-16T00:17:09.718347+0000","last_fullsized":"2024-09-16T00:17:09.718347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:08.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577952+0000","last_change":"2024-09-16T00:16:09.028910+0000","last_active":"2024-09-16T00:17:09.577952+0000","last_peered":"2024-09-16T00:17:09.577952+0000","last_clean":"2024-09-16T00:17:09.577952+0000","last_became_active":"2024-09-16T00:16:09.023302+0000","last_became_peered":"2024-09-16T00:16:09.023302+0000","last_unstale":"2024-09-16T00:17:09.577952+0000","last_undegraded":"2024-09-16T00:17:09.577952+0000","last_fullsized":"2024-09-16T00:17:09.577952+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:35.293676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581116+0000","last_change":"2024-09-16T00:17:09.581116+0000","last_active":"2024-09-16T00:17:09.581116+0000","last_peered":"2024-09-16T00:17:09.581116+0000","last_clean":"2024-09-16T00:17:09.581116+0000","last_became_active":"2024-09-16T00:17:09.580552+0000","last_became_peered":"2024-09-16T00:17:09.580552+0000","last_unstale":"2024-09-16T00:17:09.581116+0000","last_undegraded":"2024-09-16T00:17:09.581116+0000","last_fullsized":"2024-09-16T00:17:09.581116+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:35.975542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574745+0000","last_change":"2024-09-16T00:15:13.581966+0000","last_active":"2024-09-16T00:17:09.574745+0000","last_peered":"2024-09-16T00:17:09.574745+0000","last_clean":"2024-09-16T00:17:09.574745+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:17:09.574745+0000","last_undegraded":"2024-09-16T00:17:09.574745+0000","last_fullsized":"2024-09-16T00:17:09.574745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.132773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":145,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:28.314493+0000","last_change":"2024-09-16T00:15:04.351765+0000","last_active":"2024-09-16T00:17:28.314493+0000","last_peered":"2024-09-16T00:17:28.314493+0000","last_clean":"2024-09-16T00:17:28.314493+0000","last_became_active":"2024-09-16T00:15:04.351398+0000","last_became_peered":"2024-09-16T00:15:04.351398+0000","last_unstale":"2024-09-16T00:17:28.314493+0000","last_undegraded":"2024-09-16T00:17:28.314493+0000","last_fullsized":"2024-09-16T00:17:28.314493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:14:28.762657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.583255+0000","last_change":"2024-09-16T00:17:09.583255+0000","last_active":"2024-09-16T00:17:09.583255+0000","last_peered":"2024-09-16T00:17:09.583255+0000","last_clean":"2024-09-16T00:17:09.583255+0000","last_became_active":"2024-09-16T00:17:09.582793+0000","last_became_peered":"2024-09-16T00:17:09.582793+0000","last_unstale":"2024-09-16T00:17:09.583255+0000","last_undegraded":"2024-09-16T00:17:09.583255+0000","last_fullsized":"2024-09-16T00:17:09.583255+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:28.087903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574875+0000","last_change":"2024-09-16T00:15:13.576877+0000","last_active":"2024-09-16T00:17:09.574875+0000","last_peered":"2024-09-16T00:17:09.574875+0000","last_clean":"2024-09-16T00:17:09.574875+0000","last_became_active":"2024-09-16T00:15:13.576480+0000","last_became_peered":"2024-09-16T00:15:13.576480+0000","last_unstale":"2024-09-16T00:17:09.574875+0000","last_undegraded":"2024-09-16T00:17:09.574875+0000","last_fullsized":"2024-09-16T00:17:09.574875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:47.322219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780925+0000","last_change":"2024-09-16T00:15:06.550329+0000","last_active":"2024-09-16T00:17:09.780925+0000","last_peered":"2024-09-16T00:17:09.780925+0000","last_clean":"2024-09-16T00:17:09.780925+0000","last_became_active":"2024-09-16T00:15:06.550008+0000","last_became_peered":"2024-09-16T00:15:06.550008+0000","last_unstale":"2024-09-16T00:17:09.780925+0000","last_undegraded":"2024-09-16T00:17:09.780925+0000","last_fullsized":"2024-09-16T00:17:09.780925+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:38.096126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582169+0000","last_change":"2024-09-16T00:17:09.580839+0000","last_active":"2024-09-16T00:17:09.582169+0000","last_peered":"2024-09-16T00:17:09.582169+0000","last_clean":"2024-09-16T00:17:09.582169+0000","last_became_active":"2024-09-16T00:17:09.580381+0000","last_became_peered":"2024-09-16T00:17:09.580381+0000","last_unstale":"2024-09-16T00:17:09.582169+0000","last_undegraded":"2024-09-16T00:17:09.582169+0000","last_fullsized":"2024-09-16T00:17:09.582169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:35.425316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574792+0000","last_change":"2024-09-16T00:15:06.546713+0000","last_active":"2024-09-16T00:17:09.574792+0000","last_peered":"2024-09-16T00:17:09.574792+0000","last_clean":"2024-09-16T00:17:09.574792+0000","last_became_active":"2024-09-16T00:15:06.546209+0000","last_became_peered":"2024-09-16T00:15:06.546209+0000","last_unstale":"2024-09-16T00:17:09.574792+0000","last_undegraded":"2024-09-16T00:17:09.574792+0000","last_fullsized":"2024-09-16T00:17:09.574792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:43.913748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577575+0000","last_change":"2024-09-16T00:15:06.550490+0000","last_active":"2024-09-16T00:17:09.577575+0000","last_peered":"2024-09-16T00:17:09.577575+0000","last_clean":"2024-09-16T00:17:09.577575+0000","last_became_active":"2024-09-16T00:15:06.549311+0000","last_became_peered":"2024-09-16T00:15:06.549311+0000","last_unstale":"2024-09-16T00:17:09.577575+0000","last_undegraded":"2024-09-16T00:17:09.577575+0000","last_fullsized":"2024-09-16T00:17:09.577575+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:10:01.995357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718148+0000","last_change":"2024-09-16T00:15:06.548115+0000","last_active":"2024-09-16T00:17:09.718148+0000","last_peered":"2024-09-16T00:17:09.718148+0000","last_clean":"2024-09-16T00:17:09.718148+0000","last_became_active":"2024-09-16T00:15:06.547782+0000","last_became_peered":"2024-09-16T00:15:06.547782+0000","last_unstale":"2024-09-16T00:17:09.718148+0000","last_undegraded":"2024-09-16T00:17:09.718148+0000","last_fullsized":"2024-09-16T00:17:09.718148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:53.228222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781260+0000","last_change":"2024-09-16T00:15:06.554396+0000","last_active":"2024-09-16T00:17:09.781260+0000","last_peered":"2024-09-16T00:17:09.781260+0000","last_clean":"2024-09-16T00:17:09.781260+0000","last_became_active":"2024-09-16T00:15:06.554012+0000","last_became_peered":"2024-09-16T00:15:06.554012+0000","last_unstale":"2024-09-16T00:17:09.781260+0000","last_undegraded":"2024-09-16T00:17:09.781260+0000","last_fullsized":"2024-09-16T00:17:09.781260+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:31.464098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575494+0000","last_change":"2024-09-16T00:15:06.549969+0000","last_active":"2024-09-16T00:17:09.575494+0000","last_peered":"2024-09-16T00:17:09.575494+0000","last_clean":"2024-09-16T00:17:09.575494+0000","last_became_active":"2024-09-16T00:15:06.549531+0000","last_became_peered":"2024-09-16T00:15:06.549531+0000","last_unstale":"2024-09-16T00:17:09.575494+0000","last_undegraded":"2024-09-16T00:17:09.575494+0000","last_fullsized":"2024-09-16T00:17:09.575494+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:37.883612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579397+0000","last_change":"2024-09-16T00:15:06.547734+0000","last_active":"2024-09-16T00:17:09.579397+0000","last_peered":"2024-09-16T00:17:09.579397+0000","last_clean":"2024-09-16T00:17:09.579397+0000","last_became_active":"2024-09-16T00:15:06.547400+0000","last_became_peered":"2024-09-16T00:15:06.547400+0000","last_unstale":"2024-09-16T00:17:09.579397+0000","last_undegraded":"2024-09-16T00:17:09.579397+0000","last_fullsized":"2024-09-16T00:17:09.579397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:54.455313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576721+0000","last_change":"2024-09-16T00:15:06.551965+0000","last_active":"2024-09-16T00:17:09.576721+0000","last_peered":"2024-09-16T00:17:09.576721+0000","last_clean":"2024-09-16T00:17:09.576721+0000","last_became_active":"2024-09-16T00:15:06.551602+0000","last_became_peered":"2024-09-16T00:15:06.551602+0000","last_unstale":"2024-09-16T00:17:09.576721+0000","last_undegraded":"2024-09-16T00:17:09.576721+0000","last_fullsized":"2024-09-16T00:17:09.576721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:34.951960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579571+0000","last_change":"2024-09-16T00:15:06.549340+0000","last_active":"2024-09-16T00:17:09.579571+0000","last_peered":"2024-09-16T00:17:09.579571+0000","last_clean":"2024-09-16T00:17:09.579571+0000","last_became_active":"2024-09-16T00:15:06.547424+0000","last_became_peered":"2024-09-16T00:15:06.547424+0000","last_unstale":"2024-09-16T00:17:09.579571+0000","last_undegraded":"2024-09-16T00:17:09.579571+0000","last_fullsized":"2024-09-16T00:17:09.579571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:12.087535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580115+0000","last_change":"2024-09-16T00:15:06.555216+0000","last_active":"2024-09-16T00:17:09.580115+0000","last_peered":"2024-09-16T00:17:09.580115+0000","last_clean":"2024-09-16T00:17:09.580115+0000","last_became_active":"2024-09-16T00:15:06.554896+0000","last_became_peered":"2024-09-16T00:15:06.554896+0000","last_unstale":"2024-09-16T00:17:09.580115+0000","last_undegraded":"2024-09-16T00:17:09.580115+0000","last_fullsized":"2024-09-16T00:17:09.580115+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:20.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717980+0000","last_change":"2024-09-16T00:15:06.549752+0000","last_active":"2024-09-16T00:17:09.717980+0000","last_peered":"2024-09-16T00:17:09.717980+0000","last_clean":"2024-09-16T00:17:09.717980+0000","last_became_active":"2024-09-16T00:15:06.549350+0000","last_became_peered":"2024-09-16T00:15:06.549350+0000","last_unstale":"2024-09-16T00:17:09.717980+0000","last_undegraded":"2024-09-16T00:17:09.717980+0000","last_fullsized":"2024-09-16T00:17:09.717980+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:48.615123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781099+0000","last_change":"2024-09-16T00:15:06.549159+0000","last_active":"2024-09-16T00:17:09.781099+0000","last_peered":"2024-09-16T00:17:09.781099+0000","last_clean":"2024-09-16T00:17:09.781099+0000","last_became_active":"2024-09-16T00:15:06.548408+0000","last_became_peered":"2024-09-16T00:15:06.548408+0000","last_unstale":"2024-09-16T00:17:09.781099+0000","last_undegraded":"2024-09-16T00:17:09.781099+0000","last_fullsized":"2024-09-16T00:17:09.781099+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:23.941809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575144+0000","last_change":"2024-09-16T00:15:06.544091+0000","last_active":"2024-09-16T00:17:09.575144+0000","last_peered":"2024-09-16T00:17:09.575144+0000","last_clean":"2024-09-16T00:17:09.575144+0000","last_became_active":"2024-09-16T00:15:06.543761+0000","last_became_peered":"2024-09-16T00:15:06.543761+0000","last_unstale":"2024-09-16T00:17:09.575144+0000","last_undegraded":"2024-09-16T00:17:09.575144+0000","last_fullsized":"2024-09-16T00:17:09.575144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:36:43.253405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582256+0000","last_change":"2024-09-16T00:17:09.581748+0000","last_active":"2024-09-16T00:17:09.582256+0000","last_peered":"2024-09-16T00:17:09.582256+0000","last_clean":"2024-09-16T00:17:09.582256+0000","last_became_active":"2024-09-16T00:17:09.581207+0000","last_became_peered":"2024-09-16T00:17:09.581207+0000","last_unstale":"2024-09-16T00:17:09.582256+0000","last_undegraded":"2024-09-16T00:17:09.582256+0000","last_fullsized":"2024-09-16T00:17:09.582256+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:21.521929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781344+0000","last_change":"2024-09-16T00:15:06.557162+0000","last_active":"2024-09-16T00:17:09.781344+0000","last_peered":"2024-09-16T00:17:09.781344+0000","last_clean":"2024-09-16T00:17:09.781344+0000","last_became_active":"2024-09-16T00:15:06.556831+0000","last_became_peered":"2024-09-16T00:15:06.556831+0000","last_unstale":"2024-09-16T00:17:09.781344+0000","last_undegraded":"2024-09-16T00:17:09.781344+0000","last_fullsized":"2024-09-16T00:17:09.781344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.236915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579887+0000","last_change":"2024-09-16T00:15:06.556161+0000","last_active":"2024-09-16T00:17:09.579887+0000","last_peered":"2024-09-16T00:17:09.579887+0000","last_clean":"2024-09-16T00:17:09.579887+0000","last_became_active":"2024-09-16T00:15:06.555819+0000","last_became_peered":"2024-09-16T00:15:06.555819+0000","last_unstale":"2024-09-16T00:17:09.579887+0000","last_undegraded":"2024-09-16T00:17:09.579887+0000","last_fullsized":"2024-09-16T00:17:09.579887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:29.929977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.583819+0000","last_change":"2024-09-16T00:17:09.583183+0000","last_active":"2024-09-16T00:17:09.583819+0000","last_peered":"2024-09-16T00:17:09.583819+0000","last_clean":"2024-09-16T00:17:09.583819+0000","last_became_active":"2024-09-16T00:17:09.582663+0000","last_became_peered":"2024-09-16T00:17:09.582663+0000","last_unstale":"2024-09-16T00:17:09.583819+0000","last_undegraded":"2024-09-16T00:17:09.583819+0000","last_fullsized":"2024-09-16T00:17:09.583819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:37.544124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577469+0000","last_change":"2024-09-16T00:15:06.547114+0000","last_active":"2024-09-16T00:17:09.577469+0000","last_peered":"2024-09-16T00:17:09.577469+0000","last_clean":"2024-09-16T00:17:09.577469+0000","last_became_active":"2024-09-16T00:15:06.546786+0000","last_became_peered":"2024-09-16T00:15:06.546786+0000","last_unstale":"2024-09-16T00:17:09.577469+0000","last_undegraded":"2024-09-16T00:17:09.577469+0000","last_fullsized":"2024-09-16T00:17:09.577469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:13:26.925603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717890+0000","last_change":"2024-09-16T00:15:06.541168+0000","last_active":"2024-09-16T00:17:09.717890+0000","last_peered":"2024-09-16T00:17:09.717890+0000","last_clean":"2024-09-16T00:17:09.717890+0000","last_became_active":"2024-09-16T00:15:06.540848+0000","last_became_peered":"2024-09-16T00:15:06.540848+0000","last_unstale":"2024-09-16T00:17:09.717890+0000","last_undegraded":"2024-09-16T00:17:09.717890+0000","last_fullsized":"2024-09-16T00:17:09.717890+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.958600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575787+0000","last_change":"2024-09-16T00:15:06.547429+0000","last_active":"2024-09-16T00:17:09.575787+0000","last_peered":"2024-09-16T00:17:09.575787+0000","last_clean":"2024-09-16T00:17:09.575787+0000","last_became_active":"2024-09-16T00:15:06.547081+0000","last_became_peered":"2024-09-16T00:15:06.547081+0000","last_unstale":"2024-09-16T00:17:09.575787+0000","last_undegraded":"2024-09-16T00:17:09.575787+0000","last_fullsized":"2024-09-16T00:17:09.575787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:43.396313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581283+0000","last_change":"2024-09-16T00:15:06.551811+0000","last_active":"2024-09-16T00:17:09.581283+0000","last_peered":"2024-09-16T00:17:09.581283+0000","last_clean":"2024-09-16T00:17:09.581283+0000","last_became_active":"2024-09-16T00:15:06.551424+0000","last_became_peered":"2024-09-16T00:15:06.551424+0000","last_unstale":"2024-09-16T00:17:09.581283+0000","last_undegraded":"2024-09-16T00:17:09.581283+0000","last_fullsized":"2024-09-16T00:17:09.581283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.943070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578024+0000","last_change":"2024-09-16T00:15:06.544697+0000","last_active":"2024-09-16T00:17:09.578024+0000","last_peered":"2024-09-16T00:17:09.578024+0000","last_clean":"2024-09-16T00:17:09.578024+0000","last_became_active":"2024-09-16T00:15:06.544182+0000","last_became_peered":"2024-09-16T00:15:06.544182+0000","last_unstale":"2024-09-16T00:17:09.578024+0000","last_undegraded":"2024-09-16T00:17:09.578024+0000","last_fullsized":"2024-09-16T00:17:09.578024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:45.929092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580265+0000","last_change":"2024-09-16T00:15:06.551488+0000","last_active":"2024-09-16T00:17:09.580265+0000","last_peered":"2024-09-16T00:17:09.580265+0000","last_clean":"2024-09-16T00:17:09.580265+0000","last_became_active":"2024-09-16T00:15:06.551171+0000","last_became_peered":"2024-09-16T00:15:06.551171+0000","last_unstale":"2024-09-16T00:17:09.580265+0000","last_undegraded":"2024-09-16T00:17:09.580265+0000","last_fullsized":"2024-09-16T00:17:09.580265+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:23.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579101+0000","last_change":"2024-09-16T00:15:06.544043+0000","last_active":"2024-09-16T00:17:09.579101+0000","last_peered":"2024-09-16T00:17:09.579101+0000","last_clean":"2024-09-16T00:17:09.579101+0000","last_became_active":"2024-09-16T00:15:06.543014+0000","last_became_peered":"2024-09-16T00:15:06.543014+0000","last_unstale":"2024-09-16T00:17:09.579101+0000","last_undegraded":"2024-09-16T00:17:09.579101+0000","last_fullsized":"2024-09-16T00:17:09.579101+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:23.904810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581718+0000","last_change":"2024-09-16T00:15:06.546788+0000","last_active":"2024-09-16T00:17:09.581718+0000","last_peered":"2024-09-16T00:17:09.581718+0000","last_clean":"2024-09-16T00:17:09.581718+0000","last_became_active":"2024-09-16T00:15:06.546333+0000","last_became_peered":"2024-09-16T00:15:06.546333+0000","last_unstale":"2024-09-16T00:17:09.581718+0000","last_undegraded":"2024-09-16T00:17:09.581718+0000","last_fullsized":"2024-09-16T00:17:09.581718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:09.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.719618+0000","last_change":"2024-09-16T00:17:09.719618+0000","last_active":"2024-09-16T00:17:09.719618+0000","last_peered":"2024-09-16T00:17:09.719618+0000","last_clean":"2024-09-16T00:17:09.719618+0000","last_became_active":"2024-09-16T00:17:09.719172+0000","last_became_peered":"2024-09-16T00:17:09.719172+0000","last_unstale":"2024-09-16T00:17:09.719618+0000","last_undegraded":"2024-09-16T00:17:09.719618+0000","last_fullsized":"2024-09-16T00:17:09.719618+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:51.709719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579285+0000","last_change":"2024-09-16T00:15:06.540675+0000","last_active":"2024-09-16T00:17:09.579285+0000","last_peered":"2024-09-16T00:17:09.579285+0000","last_clean":"2024-09-16T00:17:09.579285+0000","last_became_active":"2024-09-16T00:15:06.540333+0000","last_became_peered":"2024-09-16T00:15:06.540333+0000","last_unstale":"2024-09-16T00:17:09.579285+0000","last_undegraded":"2024-09-16T00:17:09.579285+0000","last_fullsized":"2024-09-16T00:17:09.579285+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:17.166987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575821+0000","last_change":"2024-09-16T00:15:06.548758+0000","last_active":"2024-09-16T00:17:09.575821+0000","last_peered":"2024-09-16T00:17:09.575821+0000","last_clean":"2024-09-16T00:17:09.575821+0000","last_became_active":"2024-09-16T00:15:06.548397+0000","last_became_peered":"2024-09-16T00:15:06.548397+0000","last_unstale":"2024-09-16T00:17:09.575821+0000","last_undegraded":"2024-09-16T00:17:09.575821+0000","last_fullsized":"2024-09-16T00:17:09.575821+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:50.320170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576644+0000","last_change":"2024-09-16T00:15:06.554898+0000","last_active":"2024-09-16T00:17:09.576644+0000","last_peered":"2024-09-16T00:17:09.576644+0000","last_clean":"2024-09-16T00:17:09.576644+0000","last_became_active":"2024-09-16T00:15:06.553005+0000","last_became_peered":"2024-09-16T00:15:06.553005+0000","last_unstale":"2024-09-16T00:17:09.576644+0000","last_undegraded":"2024-09-16T00:17:09.576644+0000","last_fullsized":"2024-09-16T00:17:09.576644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:29.630911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577231+0000","last_change":"2024-09-16T00:15:06.547265+0000","last_active":"2024-09-16T00:17:09.577231+0000","last_peered":"2024-09-16T00:17:09.577231+0000","last_clean":"2024-09-16T00:17:09.577231+0000","last_became_active":"2024-09-16T00:15:06.546938+0000","last_became_peered":"2024-09-16T00:15:06.546938+0000","last_unstale":"2024-09-16T00:17:09.577231+0000","last_undegraded":"2024-09-16T00:17:09.577231+0000","last_fullsized":"2024-09-16T00:17:09.577231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:40.247343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781308+0000","last_change":"2024-09-16T00:15:06.556735+0000","last_active":"2024-09-16T00:17:09.781308+0000","last_peered":"2024-09-16T00:17:09.781308+0000","last_clean":"2024-09-16T00:17:09.781308+0000","last_became_active":"2024-09-16T00:15:06.556324+0000","last_became_peered":"2024-09-16T00:15:06.556324+0000","last_unstale":"2024-09-16T00:17:09.781308+0000","last_undegraded":"2024-09-16T00:17:09.781308+0000","last_fullsized":"2024-09-16T00:17:09.781308+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:25.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68025,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":622,"num_read_kb":373,"num_write":178,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":62107,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":209055,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249414,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579947,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":517057,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808879,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36464,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708688,"statfs":{"total":95995035648,"available":95957696512,"internally_reserved":0,"allocated":1142784,"data_stored":1168039,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972405,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1015808,"data_stored":521389,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201337,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1007616,"data_stored":516631,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430271,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1060864,"data_stored":1113265,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626501,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":512469,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855434,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1069056,"data_stored":1108319,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986255,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1040384,"data_stored":587882,"data_compressed":39297,"data_compressed_allocated":53248,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215187,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":512663,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444121,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1007616,"data_stored":513147,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607646,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1064960,"data_stored":576449,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1015808,"data_stored":513857,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":71407,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T00:17:32.869 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T00:17:33.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.007+0000 7ff98c8d0640 1 -- 172.21.15.33:0/715037933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9881025d0 msgr2=0x7ff9881049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:33.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.007+0000 7ff98c8d0640 1 --2- 172.21.15.33:0/715037933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9881025d0 0x7ff9881049c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff978009900 tx=0x7ff97802a3f0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.007+0000 7ff98c8d0640 1 -- 172.21.15.33:0/715037933 shutdown_connections 2024-09-16T00:17:33.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.007+0000 7ff98c8d0640 1 --2- 172.21.15.33:0/715037933 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff988107830 0x7ff988109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.007+0000 7ff98c8d0640 1 --2- 172.21.15.33:0/715037933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff988104f00 0x7ff9881072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.007+0000 7ff98c8d0640 1 --2- 172.21.15.33:0/715037933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9881025d0 0x7ff9881049c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.007+0000 7ff98c8d0640 1 -- 172.21.15.33:0/715037933 >> 172.21.15.33:0/715037933 conn(0x7ff9880fc190 msgr2=0x7ff9880fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:33.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.007+0000 7ff98c8d0640 1 -- 172.21.15.33:0/715037933 shutdown_connections 2024-09-16T00:17:33.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.007+0000 7ff98c8d0640 1 -- 172.21.15.33:0/715037933 wait complete. 2024-09-16T00:17:33.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff98c8d0640 1 Processor -- start 2024-09-16T00:17:33.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff98c8d0640 1 -- start start 2024-09-16T00:17:33.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff98c8d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9881025d0 0x7ff988197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:33.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff98c8d0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff988104f00 0x7ff988198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:33.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff98c8d0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff988107830 0x7ff98819f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:33.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff98c8d0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff98810d810 con 0x7ff988104f00 2024-09-16T00:17:33.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff98c8d0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff98810d690 con 0x7ff988107830 2024-09-16T00:17:33.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff98c8d0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff98810d990 con 0x7ff9881025d0 2024-09-16T00:17:33.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff985d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff988104f00 0x7ff988198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:33.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff985d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff988104f00 0x7ff988198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36254/0 (socket says 172.21.15.33:36254) 2024-09-16T00:17:33.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff985d9b640 1 -- 172.21.15.33:0/2555682408 learned_addr learned my addr 172.21.15.33:0/2555682408 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:33.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff986d9d640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff988107830 0x7ff98819f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:33.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.008+0000 7ff98659c640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9881025d0 0x7ff988197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:33.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.009+0000 7ff985d9b640 1 -- 172.21.15.33:0/2555682408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9881025d0 msgr2=0x7ff988197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:33.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.009+0000 7ff985d9b640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9881025d0 0x7ff988197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.009+0000 7ff985d9b640 1 -- 172.21.15.33:0/2555682408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff988107830 msgr2=0x7ff98819f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:33.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.009+0000 7ff985d9b640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff988107830 0x7ff98819f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.009+0000 7ff985d9b640 1 -- 172.21.15.33:0/2555682408 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98819fa50 con 0x7ff988104f00 2024-09-16T00:17:33.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.009+0000 7ff985d9b640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff988104f00 0x7ff988198370 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00c970 tx=0x7ff97c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:33.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.009+0000 7ff9777fe640 1 -- 172.21.15.33:0/2555682408 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c014070 con 0x7ff988104f00 2024-09-16T00:17:33.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.009+0000 7ff9777fe640 1 -- 172.21.15.33:0/2555682408 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff97c004440 con 0x7ff988104f00 2024-09-16T00:17:33.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.009+0000 7ff98c8d0640 1 -- 172.21.15.33:0/2555682408 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff98819fd40 con 0x7ff988104f00 2024-09-16T00:17:33.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.009+0000 7ff9777fe640 1 -- 172.21.15.33:0/2555682408 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c00f450 con 0x7ff988104f00 2024-09-16T00:17:33.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.009+0000 7ff98c8d0640 1 -- 172.21.15.33:0/2555682408 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9881a0230 con 0x7ff988104f00 2024-09-16T00:17:33.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.010+0000 7ff98c8d0640 1 -- 172.21.15.33:0/2555682408 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9880fdc80 con 0x7ff988104f00 2024-09-16T00:17:33.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.011+0000 7ff9777fe640 1 -- 172.21.15.33:0/2555682408 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7ff97c017070 con 0x7ff988104f00 2024-09-16T00:17:33.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.012+0000 7ff9777fe640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9700a32e0 0x7ff9700a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:33.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.012+0000 7ff98659c640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9700a32e0 0x7ff9700a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:33.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.012+0000 7ff9777fe640 1 -- 172.21.15.33:0/2555682408 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff97c0cfcc0 con 0x7ff988104f00 2024-09-16T00:17:33.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.013+0000 7ff98659c640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9700a32e0 0x7ff9700a57a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff9780314e0 tx=0x7ff978000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:33.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.016+0000 7ff9777fe640 1 -- 172.21.15.33:0/2555682408 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff97c01e020 con 0x7ff988104f00 2024-09-16T00:17:33.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:32 smithi033 ceph-mon[31243]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:17:33.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:32 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2755694435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:33.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:32 smithi083 ceph-mon[35242]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:17:33.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:32 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2755694435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:33.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.190+0000 7ff98c8d0640 1 -- 172.21.15.33:0/2555682408 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff988062630 con 0x7ff9700a32e0 2024-09-16T00:17:33.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.198+0000 7ff9777fe640 1 -- 172.21.15.33:0/2555682408 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340766 (secure 0 0 0) 0x7ff988062630 con 0x7ff9700a32e0 2024-09-16T00:17:33.202 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:33.204 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-09-16T00:17:33.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.201+0000 7ff98c8d0640 1 -- 172.21.15.33:0/2555682408 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9700a32e0 msgr2=0x7ff9700a57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:33.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.201+0000 7ff98c8d0640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9700a32e0 0x7ff9700a57a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff9780314e0 tx=0x7ff978000bc0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.201+0000 7ff98c8d0640 1 -- 172.21.15.33:0/2555682408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff988104f00 msgr2=0x7ff988198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:33.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.201+0000 7ff98c8d0640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff988104f00 0x7ff988198370 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00c970 tx=0x7ff97c00ce30 comp rx=0 tx=0).stop 2024-09-16T00:17:33.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.201+0000 7ff98c8d0640 1 -- 172.21.15.33:0/2555682408 shutdown_connections 2024-09-16T00:17:33.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.201+0000 7ff98c8d0640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9700a32e0 0x7ff9700a57a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.201+0000 7ff98c8d0640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff988107830 0x7ff98819f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.201+0000 7ff98c8d0640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff988104f00 0x7ff988198370 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.201+0000 7ff98c8d0640 1 --2- 172.21.15.33:0/2555682408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9881025d0 0x7ff988197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.201+0000 7ff98c8d0640 1 -- 172.21.15.33:0/2555682408 >> 172.21.15.33:0/2555682408 conn(0x7ff9880fc190 msgr2=0x7ff988102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:33.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.202+0000 7ff98c8d0640 1 -- 172.21.15.33:0/2555682408 shutdown_connections 2024-09-16T00:17:33.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.202+0000 7ff98c8d0640 1 -- 172.21.15.33:0/2555682408 wait complete. 2024-09-16T00:17:33.212 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-16T00:17:31.670333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658627,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":748,"num_read_kb":482,"num_write":361,"num_write_kb":3065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1242843,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386388,"kb_used_data":12124,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555436,"statfs":{"total":1151940427776,"available":1151544766464,"internally_reserved":0,"allocated":12414976,"data_stored":8161167,"data_compressed":471663,"data_compressed_allocated":638976,"data_compressed_original":2809856,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":701,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":37,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003164"},"pg_stats":[{"pgid":"2.1f","version":"86'9","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578728+0000","last_change":"2024-09-16T00:16:09.034677+0000","last_active":"2024-09-16T00:17:09.578728+0000","last_peered":"2024-09-16T00:17:09.578728+0000","last_clean":"2024-09-16T00:17:09.578728+0000","last_became_active":"2024-09-16T00:16:09.024755+0000","last_became_peered":"2024-09-16T00:16:09.024755+0000","last_unstale":"2024-09-16T00:17:09.578728+0000","last_undegraded":"2024-09-16T00:17:09.578728+0000","last_fullsized":"2024-09-16T00:17:09.578728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:09.012210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579433+0000","last_change":"2024-09-16T00:15:06.547094+0000","last_active":"2024-09-16T00:17:09.579433+0000","last_peered":"2024-09-16T00:17:09.579433+0000","last_clean":"2024-09-16T00:17:09.579433+0000","last_became_active":"2024-09-16T00:15:06.542764+0000","last_became_peered":"2024-09-16T00:15:06.542764+0000","last_unstale":"2024-09-16T00:17:09.579433+0000","last_undegraded":"2024-09-16T00:17:09.579433+0000","last_fullsized":"2024-09-16T00:17:09.579433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:04:37.449577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574641+0000","last_change":"2024-09-16T00:15:13.578400+0000","last_active":"2024-09-16T00:17:09.574641+0000","last_peered":"2024-09-16T00:17:09.574641+0000","last_clean":"2024-09-16T00:17:09.574641+0000","last_became_active":"2024-09-16T00:15:13.577787+0000","last_became_peered":"2024-09-16T00:15:13.577787+0000","last_unstale":"2024-09-16T00:17:09.574641+0000","last_undegraded":"2024-09-16T00:17:09.574641+0000","last_fullsized":"2024-09-16T00:17:09.574641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:19:59.194152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579738+0000","last_change":"2024-09-16T00:15:04.373381+0000","last_active":"2024-09-16T00:17:09.579738+0000","last_peered":"2024-09-16T00:17:09.579738+0000","last_clean":"2024-09-16T00:17:09.579738+0000","last_became_active":"2024-09-16T00:15:04.373028+0000","last_became_peered":"2024-09-16T00:15:04.373028+0000","last_unstale":"2024-09-16T00:17:09.579738+0000","last_undegraded":"2024-09-16T00:17:09.579738+0000","last_fullsized":"2024-09-16T00:17:09.579738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:22.902583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577500+0000","last_change":"2024-09-16T00:15:06.554048+0000","last_active":"2024-09-16T00:17:09.577500+0000","last_peered":"2024-09-16T00:17:09.577500+0000","last_clean":"2024-09-16T00:17:09.577500+0000","last_became_active":"2024-09-16T00:15:06.553044+0000","last_became_peered":"2024-09-16T00:15:06.553044+0000","last_unstale":"2024-09-16T00:17:09.577500+0000","last_undegraded":"2024-09-16T00:17:09.577500+0000","last_fullsized":"2024-09-16T00:17:09.577500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:22:53.333705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574960+0000","last_change":"2024-09-16T00:15:13.586995+0000","last_active":"2024-09-16T00:17:09.574960+0000","last_peered":"2024-09-16T00:17:09.574960+0000","last_clean":"2024-09-16T00:17:09.574960+0000","last_became_active":"2024-09-16T00:15:13.586509+0000","last_became_peered":"2024-09-16T00:15:13.586509+0000","last_unstale":"2024-09-16T00:17:09.574960+0000","last_undegraded":"2024-09-16T00:17:09.574960+0000","last_fullsized":"2024-09-16T00:17:09.574960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:39.418581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579612+0000","last_change":"2024-09-16T00:15:04.348235+0000","last_active":"2024-09-16T00:17:09.579612+0000","last_peered":"2024-09-16T00:17:09.579612+0000","last_clean":"2024-09-16T00:17:09.579612+0000","last_became_active":"2024-09-16T00:15:04.347518+0000","last_became_peered":"2024-09-16T00:15:04.347518+0000","last_unstale":"2024-09-16T00:17:09.579612+0000","last_undegraded":"2024-09-16T00:17:09.579612+0000","last_fullsized":"2024-09-16T00:17:09.579612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:24.396911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581041+0000","last_change":"2024-09-16T00:15:06.554067+0000","last_active":"2024-09-16T00:17:09.581041+0000","last_peered":"2024-09-16T00:17:09.581041+0000","last_clean":"2024-09-16T00:17:09.581041+0000","last_became_active":"2024-09-16T00:15:06.553715+0000","last_became_peered":"2024-09-16T00:15:06.553715+0000","last_unstale":"2024-09-16T00:17:09.581041+0000","last_undegraded":"2024-09-16T00:17:09.581041+0000","last_fullsized":"2024-09-16T00:17:09.581041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:22.008530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575352+0000","last_change":"2024-09-16T00:15:13.586872+0000","last_active":"2024-09-16T00:17:09.575352+0000","last_peered":"2024-09-16T00:17:09.575352+0000","last_clean":"2024-09-16T00:17:09.575352+0000","last_became_active":"2024-09-16T00:15:13.586375+0000","last_became_peered":"2024-09-16T00:15:13.586375+0000","last_unstale":"2024-09-16T00:17:09.575352+0000","last_undegraded":"2024-09-16T00:17:09.575352+0000","last_fullsized":"2024-09-16T00:17:09.575352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:43.668827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579613+0000","last_change":"2024-09-16T00:15:04.347964+0000","last_active":"2024-09-16T00:17:09.579613+0000","last_peered":"2024-09-16T00:17:09.579613+0000","last_clean":"2024-09-16T00:17:09.579613+0000","last_became_active":"2024-09-16T00:15:04.347588+0000","last_became_peered":"2024-09-16T00:15:04.347588+0000","last_unstale":"2024-09-16T00:17:09.579613+0000","last_undegraded":"2024-09-16T00:17:09.579613+0000","last_fullsized":"2024-09-16T00:17:09.579613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:14.199287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576911+0000","last_change":"2024-09-16T00:15:06.544918+0000","last_active":"2024-09-16T00:17:09.576911+0000","last_peered":"2024-09-16T00:17:09.576911+0000","last_clean":"2024-09-16T00:17:09.576911+0000","last_became_active":"2024-09-16T00:15:06.544579+0000","last_became_peered":"2024-09-16T00:15:06.544579+0000","last_unstale":"2024-09-16T00:17:09.576911+0000","last_undegraded":"2024-09-16T00:17:09.576911+0000","last_fullsized":"2024-09-16T00:17:09.576911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:06:37.475644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575052+0000","last_change":"2024-09-16T00:15:13.581488+0000","last_active":"2024-09-16T00:17:09.575052+0000","last_peered":"2024-09-16T00:17:09.575052+0000","last_clean":"2024-09-16T00:17:09.575052+0000","last_became_active":"2024-09-16T00:15:13.581164+0000","last_became_peered":"2024-09-16T00:15:13.581164+0000","last_unstale":"2024-09-16T00:17:09.575052+0000","last_undegraded":"2024-09-16T00:17:09.575052+0000","last_fullsized":"2024-09-16T00:17:09.575052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:05.061592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581271+0000","last_change":"2024-09-16T00:15:06.551982+0000","last_active":"2024-09-16T00:17:09.581271+0000","last_peered":"2024-09-16T00:17:09.581271+0000","last_clean":"2024-09-16T00:17:09.581271+0000","last_became_active":"2024-09-16T00:15:06.551663+0000","last_became_peered":"2024-09-16T00:15:06.551663+0000","last_unstale":"2024-09-16T00:17:09.581271+0000","last_undegraded":"2024-09-16T00:17:09.581271+0000","last_fullsized":"2024-09-16T00:17:09.581271+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:11.928703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579394+0000","last_change":"2024-09-16T00:15:04.352237+0000","last_active":"2024-09-16T00:17:09.579394+0000","last_peered":"2024-09-16T00:17:09.579394+0000","last_clean":"2024-09-16T00:17:09.579394+0000","last_became_active":"2024-09-16T00:15:04.351884+0000","last_became_peered":"2024-09-16T00:15:04.351884+0000","last_unstale":"2024-09-16T00:17:09.579394+0000","last_undegraded":"2024-09-16T00:17:09.579394+0000","last_fullsized":"2024-09-16T00:17:09.579394+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:32.932020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575580+0000","last_change":"2024-09-16T00:15:13.638917+0000","last_active":"2024-09-16T00:17:09.575580+0000","last_peered":"2024-09-16T00:17:09.575580+0000","last_clean":"2024-09-16T00:17:09.575580+0000","last_became_active":"2024-09-16T00:15:13.638401+0000","last_became_peered":"2024-09-16T00:15:13.638401+0000","last_unstale":"2024-09-16T00:17:09.575580+0000","last_undegraded":"2024-09-16T00:17:09.575580+0000","last_fullsized":"2024-09-16T00:17:09.575580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:38.082407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579388+0000","last_change":"2024-09-16T00:15:06.547909+0000","last_active":"2024-09-16T00:17:09.579388+0000","last_peered":"2024-09-16T00:17:09.579388+0000","last_clean":"2024-09-16T00:17:09.579388+0000","last_became_active":"2024-09-16T00:15:06.546814+0000","last_became_peered":"2024-09-16T00:15:06.546814+0000","last_unstale":"2024-09-16T00:17:09.579388+0000","last_undegraded":"2024-09-16T00:17:09.579388+0000","last_fullsized":"2024-09-16T00:17:09.579388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:46.419406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780898+0000","last_change":"2024-09-16T00:16:09.027691+0000","last_active":"2024-09-16T00:17:09.780898+0000","last_peered":"2024-09-16T00:17:09.780898+0000","last_clean":"2024-09-16T00:17:09.780898+0000","last_became_active":"2024-09-16T00:16:09.023204+0000","last_became_peered":"2024-09-16T00:16:09.023204+0000","last_unstale":"2024-09-16T00:17:09.780898+0000","last_undegraded":"2024-09-16T00:17:09.780898+0000","last_fullsized":"2024-09-16T00:17:09.780898+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:44.647576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575627+0000","last_change":"2024-09-16T00:15:13.588345+0000","last_active":"2024-09-16T00:17:09.575627+0000","last_peered":"2024-09-16T00:17:09.575627+0000","last_clean":"2024-09-16T00:17:09.575627+0000","last_became_active":"2024-09-16T00:15:13.587829+0000","last_became_peered":"2024-09-16T00:15:13.587829+0000","last_unstale":"2024-09-16T00:17:09.575627+0000","last_undegraded":"2024-09-16T00:17:09.575627+0000","last_fullsized":"2024-09-16T00:17:09.575627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:04.261031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781137+0000","last_change":"2024-09-16T00:15:06.545485+0000","last_active":"2024-09-16T00:17:09.781137+0000","last_peered":"2024-09-16T00:17:09.781137+0000","last_clean":"2024-09-16T00:17:09.781137+0000","last_became_active":"2024-09-16T00:15:06.545178+0000","last_became_peered":"2024-09-16T00:15:06.545178+0000","last_unstale":"2024-09-16T00:17:09.781137+0000","last_undegraded":"2024-09-16T00:17:09.781137+0000","last_fullsized":"2024-09-16T00:17:09.781137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:26.923469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579782+0000","last_change":"2024-09-16T00:15:04.340546+0000","last_active":"2024-09-16T00:17:09.579782+0000","last_peered":"2024-09-16T00:17:09.579782+0000","last_clean":"2024-09-16T00:17:09.579782+0000","last_became_active":"2024-09-16T00:15:04.340173+0000","last_became_peered":"2024-09-16T00:15:04.340173+0000","last_unstale":"2024-09-16T00:17:09.579782+0000","last_undegraded":"2024-09-16T00:17:09.579782+0000","last_fullsized":"2024-09-16T00:17:09.579782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:35:01.134433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578541+0000","last_change":"2024-09-16T00:15:13.589963+0000","last_active":"2024-09-16T00:17:09.578541+0000","last_peered":"2024-09-16T00:17:09.578541+0000","last_clean":"2024-09-16T00:17:09.578541+0000","last_became_active":"2024-09-16T00:15:13.589615+0000","last_became_peered":"2024-09-16T00:15:13.589615+0000","last_unstale":"2024-09-16T00:17:09.578541+0000","last_undegraded":"2024-09-16T00:17:09.578541+0000","last_fullsized":"2024-09-16T00:17:09.578541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:37.197765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579731+0000","last_change":"2024-09-16T00:15:06.544374+0000","last_active":"2024-09-16T00:17:09.579731+0000","last_peered":"2024-09-16T00:17:09.579731+0000","last_clean":"2024-09-16T00:17:09.579731+0000","last_became_active":"2024-09-16T00:15:06.544027+0000","last_became_peered":"2024-09-16T00:15:06.544027+0000","last_unstale":"2024-09-16T00:17:09.579731+0000","last_undegraded":"2024-09-16T00:17:09.579731+0000","last_fullsized":"2024-09-16T00:17:09.579731+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580105+0000","last_change":"2024-09-16T00:15:04.374209+0000","last_active":"2024-09-16T00:17:09.580105+0000","last_peered":"2024-09-16T00:17:09.580105+0000","last_clean":"2024-09-16T00:17:09.580105+0000","last_became_active":"2024-09-16T00:15:04.373814+0000","last_became_peered":"2024-09-16T00:15:04.373814+0000","last_unstale":"2024-09-16T00:17:09.580105+0000","last_undegraded":"2024-09-16T00:17:09.580105+0000","last_fullsized":"2024-09-16T00:17:09.580105+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:13.516161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575489+0000","last_change":"2024-09-16T00:15:13.639210+0000","last_active":"2024-09-16T00:17:09.575489+0000","last_peered":"2024-09-16T00:17:09.575489+0000","last_clean":"2024-09-16T00:17:09.575489+0000","last_became_active":"2024-09-16T00:15:13.638719+0000","last_became_peered":"2024-09-16T00:15:13.638719+0000","last_unstale":"2024-09-16T00:17:09.575489+0000","last_undegraded":"2024-09-16T00:17:09.575489+0000","last_fullsized":"2024-09-16T00:17:09.575489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:26.130778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577341+0000","last_change":"2024-09-16T00:15:06.553108+0000","last_active":"2024-09-16T00:17:09.577341+0000","last_peered":"2024-09-16T00:17:09.577341+0000","last_clean":"2024-09-16T00:17:09.577341+0000","last_became_active":"2024-09-16T00:15:06.552817+0000","last_became_peered":"2024-09-16T00:15:06.552817+0000","last_unstale":"2024-09-16T00:17:09.577341+0000","last_undegraded":"2024-09-16T00:17:09.577341+0000","last_fullsized":"2024-09-16T00:17:09.577341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:06.821337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575833+0000","last_change":"2024-09-16T00:15:04.644185+0000","last_active":"2024-09-16T00:17:09.575833+0000","last_peered":"2024-09-16T00:17:09.575833+0000","last_clean":"2024-09-16T00:17:09.575833+0000","last_became_active":"2024-09-16T00:15:04.643887+0000","last_became_peered":"2024-09-16T00:15:04.643887+0000","last_unstale":"2024-09-16T00:17:09.575833+0000","last_undegraded":"2024-09-16T00:17:09.575833+0000","last_fullsized":"2024-09-16T00:17:09.575833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:44.279272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574877+0000","last_change":"2024-09-16T00:15:13.574761+0000","last_active":"2024-09-16T00:17:09.574877+0000","last_peered":"2024-09-16T00:17:09.574877+0000","last_clean":"2024-09-16T00:17:09.574877+0000","last_became_active":"2024-09-16T00:15:13.574347+0000","last_became_peered":"2024-09-16T00:15:13.574347+0000","last_unstale":"2024-09-16T00:17:09.574877+0000","last_undegraded":"2024-09-16T00:17:09.574877+0000","last_fullsized":"2024-09-16T00:17:09.574877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:22.040121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579504+0000","last_change":"2024-09-16T00:15:06.550338+0000","last_active":"2024-09-16T00:17:09.579504+0000","last_peered":"2024-09-16T00:17:09.579504+0000","last_clean":"2024-09-16T00:17:09.579504+0000","last_became_active":"2024-09-16T00:15:06.549868+0000","last_became_peered":"2024-09-16T00:15:06.549868+0000","last_unstale":"2024-09-16T00:17:09.579504+0000","last_undegraded":"2024-09-16T00:17:09.579504+0000","last_fullsized":"2024-09-16T00:17:09.579504+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:26.643338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'12","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:13.711683+0000","last_change":"2024-09-16T00:15:04.342306+0000","last_active":"2024-09-16T00:17:13.711683+0000","last_peered":"2024-09-16T00:17:13.711683+0000","last_clean":"2024-09-16T00:17:13.711683+0000","last_became_active":"2024-09-16T00:15:04.341964+0000","last_became_peered":"2024-09-16T00:15:04.341964+0000","last_unstale":"2024-09-16T00:17:13.711683+0000","last_undegraded":"2024-09-16T00:17:13.711683+0000","last_fullsized":"2024-09-16T00:17:13.711683+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:14.137199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578972+0000","last_change":"2024-09-16T00:15:13.638246+0000","last_active":"2024-09-16T00:17:09.578972+0000","last_peered":"2024-09-16T00:17:09.578972+0000","last_clean":"2024-09-16T00:17:09.578972+0000","last_became_active":"2024-09-16T00:15:13.637816+0000","last_became_peered":"2024-09-16T00:15:13.637816+0000","last_unstale":"2024-09-16T00:17:09.578972+0000","last_undegraded":"2024-09-16T00:17:09.578972+0000","last_fullsized":"2024-09-16T00:17:09.578972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:27.331156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718239+0000","last_change":"2024-09-16T00:15:06.544057+0000","last_active":"2024-09-16T00:17:09.718239+0000","last_peered":"2024-09-16T00:17:09.718239+0000","last_clean":"2024-09-16T00:17:09.718239+0000","last_became_active":"2024-09-16T00:15:06.543244+0000","last_became_peered":"2024-09-16T00:15:06.543244+0000","last_unstale":"2024-09-16T00:17:09.718239+0000","last_undegraded":"2024-09-16T00:17:09.718239+0000","last_fullsized":"2024-09-16T00:17:09.718239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:42:28.950014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579701+0000","last_change":"2024-09-16T00:15:04.376419+0000","last_active":"2024-09-16T00:17:09.579701+0000","last_peered":"2024-09-16T00:17:09.579701+0000","last_clean":"2024-09-16T00:17:09.579701+0000","last_became_active":"2024-09-16T00:15:04.375976+0000","last_became_peered":"2024-09-16T00:15:04.375976+0000","last_unstale":"2024-09-16T00:17:09.579701+0000","last_undegraded":"2024-09-16T00:17:09.579701+0000","last_fullsized":"2024-09-16T00:17:09.579701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:33:38.652109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781038+0000","last_change":"2024-09-16T00:15:13.584306+0000","last_active":"2024-09-16T00:17:09.781038+0000","last_peered":"2024-09-16T00:17:09.781038+0000","last_clean":"2024-09-16T00:17:09.781038+0000","last_became_active":"2024-09-16T00:15:13.583840+0000","last_became_peered":"2024-09-16T00:15:13.583840+0000","last_unstale":"2024-09-16T00:17:09.781038+0000","last_undegraded":"2024-09-16T00:17:09.781038+0000","last_fullsized":"2024-09-16T00:17:09.781038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:36.660458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577806+0000","last_change":"2024-09-16T00:15:06.538902+0000","last_active":"2024-09-16T00:17:09.577806+0000","last_peered":"2024-09-16T00:17:09.577806+0000","last_clean":"2024-09-16T00:17:09.577806+0000","last_became_active":"2024-09-16T00:15:06.538537+0000","last_became_peered":"2024-09-16T00:15:06.538537+0000","last_unstale":"2024-09-16T00:17:09.577806+0000","last_undegraded":"2024-09-16T00:17:09.577806+0000","last_fullsized":"2024-09-16T00:17:09.577806+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:38.411711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578013+0000","last_change":"2024-09-16T00:15:04.374762+0000","last_active":"2024-09-16T00:17:09.578013+0000","last_peered":"2024-09-16T00:17:09.578013+0000","last_clean":"2024-09-16T00:17:09.578013+0000","last_became_active":"2024-09-16T00:15:04.374394+0000","last_became_peered":"2024-09-16T00:15:04.374394+0000","last_unstale":"2024-09-16T00:17:09.578013+0000","last_undegraded":"2024-09-16T00:17:09.578013+0000","last_fullsized":"2024-09-16T00:17:09.578013+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:47.075021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579233+0000","last_change":"2024-09-16T00:15:13.638306+0000","last_active":"2024-09-16T00:17:09.579233+0000","last_peered":"2024-09-16T00:17:09.579233+0000","last_clean":"2024-09-16T00:17:09.579233+0000","last_became_active":"2024-09-16T00:15:13.637931+0000","last_became_peered":"2024-09-16T00:15:13.637931+0000","last_unstale":"2024-09-16T00:17:09.579233+0000","last_undegraded":"2024-09-16T00:17:09.579233+0000","last_fullsized":"2024-09-16T00:17:09.579233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:41.385174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577585+0000","last_change":"2024-09-16T00:15:06.548209+0000","last_active":"2024-09-16T00:17:09.577585+0000","last_peered":"2024-09-16T00:17:09.577585+0000","last_clean":"2024-09-16T00:17:09.577585+0000","last_became_active":"2024-09-16T00:15:06.547846+0000","last_became_peered":"2024-09-16T00:15:06.547846+0000","last_unstale":"2024-09-16T00:17:09.577585+0000","last_undegraded":"2024-09-16T00:17:09.577585+0000","last_fullsized":"2024-09-16T00:17:09.577585+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:15.821847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780889+0000","last_change":"2024-09-16T00:16:09.023918+0000","last_active":"2024-09-16T00:17:09.780889+0000","last_peered":"2024-09-16T00:17:09.780889+0000","last_clean":"2024-09-16T00:17:09.780889+0000","last_became_active":"2024-09-16T00:16:09.023230+0000","last_became_peered":"2024-09-16T00:16:09.023230+0000","last_unstale":"2024-09-16T00:17:09.780889+0000","last_undegraded":"2024-09-16T00:17:09.780889+0000","last_fullsized":"2024-09-16T00:17:09.780889+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:31.527863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717949+0000","last_change":"2024-09-16T00:15:13.579387+0000","last_active":"2024-09-16T00:17:09.717949+0000","last_peered":"2024-09-16T00:17:09.717949+0000","last_clean":"2024-09-16T00:17:09.717949+0000","last_became_active":"2024-09-16T00:15:13.578996+0000","last_became_peered":"2024-09-16T00:15:13.578996+0000","last_unstale":"2024-09-16T00:17:09.717949+0000","last_undegraded":"2024-09-16T00:17:09.717949+0000","last_fullsized":"2024-09-16T00:17:09.717949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:51.703060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579796+0000","last_change":"2024-09-16T00:17:09.579796+0000","last_active":"2024-09-16T00:17:09.579796+0000","last_peered":"2024-09-16T00:17:09.579796+0000","last_clean":"2024-09-16T00:17:09.579796+0000","last_became_active":"2024-09-16T00:17:09.579386+0000","last_became_peered":"2024-09-16T00:17:09.579386+0000","last_unstale":"2024-09-16T00:17:09.579796+0000","last_undegraded":"2024-09-16T00:17:09.579796+0000","last_fullsized":"2024-09-16T00:17:09.579796+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:39:42.571193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":145,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:29.146717+0000","last_change":"2024-09-16T00:15:04.373896+0000","last_active":"2024-09-16T00:17:29.146717+0000","last_peered":"2024-09-16T00:17:29.146717+0000","last_clean":"2024-09-16T00:17:29.146717+0000","last_became_active":"2024-09-16T00:15:04.373493+0000","last_became_peered":"2024-09-16T00:15:04.373493+0000","last_unstale":"2024-09-16T00:17:29.146717+0000","last_undegraded":"2024-09-16T00:17:29.146717+0000","last_fullsized":"2024-09-16T00:17:29.146717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:17:10.767267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":100,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718128+0000","last_change":"2024-09-16T00:15:13.638575+0000","last_active":"2024-09-16T00:17:09.718128+0000","last_peered":"2024-09-16T00:17:09.718128+0000","last_clean":"2024-09-16T00:17:09.718128+0000","last_became_active":"2024-09-16T00:15:13.638144+0000","last_became_peered":"2024-09-16T00:15:13.638144+0000","last_unstale":"2024-09-16T00:17:09.718128+0000","last_undegraded":"2024-09-16T00:17:09.718128+0000","last_fullsized":"2024-09-16T00:17:09.718128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:39.345011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718312+0000","last_change":"2024-09-16T00:15:06.548852+0000","last_active":"2024-09-16T00:17:09.718312+0000","last_peered":"2024-09-16T00:17:09.718312+0000","last_clean":"2024-09-16T00:17:09.718312+0000","last_became_active":"2024-09-16T00:15:06.548238+0000","last_became_peered":"2024-09-16T00:15:06.548238+0000","last_unstale":"2024-09-16T00:17:09.718312+0000","last_undegraded":"2024-09-16T00:17:09.718312+0000","last_fullsized":"2024-09-16T00:17:09.718312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:16.115338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580690+0000","last_change":"2024-09-16T00:15:04.350753+0000","last_active":"2024-09-16T00:17:09.580690+0000","last_peered":"2024-09-16T00:17:09.580690+0000","last_clean":"2024-09-16T00:17:09.580690+0000","last_became_active":"2024-09-16T00:15:04.350406+0000","last_became_peered":"2024-09-16T00:15:04.350406+0000","last_unstale":"2024-09-16T00:17:09.580690+0000","last_undegraded":"2024-09-16T00:17:09.580690+0000","last_fullsized":"2024-09-16T00:17:09.580690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:09:54.737287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576996+0000","last_change":"2024-09-16T00:15:13.639302+0000","last_active":"2024-09-16T00:17:09.576996+0000","last_peered":"2024-09-16T00:17:09.576996+0000","last_clean":"2024-09-16T00:17:09.576996+0000","last_became_active":"2024-09-16T00:15:13.638863+0000","last_became_peered":"2024-09-16T00:15:13.638863+0000","last_unstale":"2024-09-16T00:17:09.576996+0000","last_undegraded":"2024-09-16T00:17:09.576996+0000","last_fullsized":"2024-09-16T00:17:09.576996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:23.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582176+0000","last_change":"2024-09-16T00:17:09.582176+0000","last_active":"2024-09-16T00:17:09.582176+0000","last_peered":"2024-09-16T00:17:09.582176+0000","last_clean":"2024-09-16T00:17:09.582176+0000","last_became_active":"2024-09-16T00:17:09.581740+0000","last_became_peered":"2024-09-16T00:17:09.581740+0000","last_unstale":"2024-09-16T00:17:09.582176+0000","last_undegraded":"2024-09-16T00:17:09.582176+0000","last_fullsized":"2024-09-16T00:17:09.582176+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:57.176724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575861+0000","last_change":"2024-09-16T00:15:04.342216+0000","last_active":"2024-09-16T00:17:09.575861+0000","last_peered":"2024-09-16T00:17:09.575861+0000","last_clean":"2024-09-16T00:17:09.575861+0000","last_became_active":"2024-09-16T00:15:04.341763+0000","last_became_peered":"2024-09-16T00:15:04.341763+0000","last_unstale":"2024-09-16T00:17:09.575861+0000","last_undegraded":"2024-09-16T00:17:09.575861+0000","last_fullsized":"2024-09-16T00:17:09.575861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:44.938527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":350,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580892+0000","last_change":"2024-09-16T00:13:51.108571+0000","last_active":"2024-09-16T00:17:09.580892+0000","last_peered":"2024-09-16T00:17:09.580892+0000","last_clean":"2024-09-16T00:17:09.580892+0000","last_became_active":"2024-09-16T00:13:51.108088+0000","last_became_peered":"2024-09-16T00:13:51.108088+0000","last_unstale":"2024-09-16T00:17:09.580892+0000","last_undegraded":"2024-09-16T00:17:09.580892+0000","last_fullsized":"2024-09-16T00:17:09.580892+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_clean_scrub_stamp":"2024-09-16T00:10:24.533807+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:02.511064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581143+0000","last_change":"2024-09-16T00:15:13.588665+0000","last_active":"2024-09-16T00:17:09.581143+0000","last_peered":"2024-09-16T00:17:09.581143+0000","last_clean":"2024-09-16T00:17:09.581143+0000","last_became_active":"2024-09-16T00:15:13.588222+0000","last_became_peered":"2024-09-16T00:15:13.588222+0000","last_unstale":"2024-09-16T00:17:09.581143+0000","last_undegraded":"2024-09-16T00:17:09.581143+0000","last_fullsized":"2024-09-16T00:17:09.581143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:59:59.965962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581454+0000","last_change":"2024-09-16T00:15:06.553114+0000","last_active":"2024-09-16T00:17:09.581454+0000","last_peered":"2024-09-16T00:17:09.581454+0000","last_clean":"2024-09-16T00:17:09.581454+0000","last_became_active":"2024-09-16T00:15:06.552788+0000","last_became_peered":"2024-09-16T00:15:06.552788+0000","last_unstale":"2024-09-16T00:17:09.581454+0000","last_undegraded":"2024-09-16T00:17:09.581454+0000","last_fullsized":"2024-09-16T00:17:09.581454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:05.473847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575364+0000","last_change":"2024-09-16T00:16:09.030747+0000","last_active":"2024-09-16T00:17:09.575364+0000","last_peered":"2024-09-16T00:17:09.575364+0000","last_clean":"2024-09-16T00:17:09.575364+0000","last_became_active":"2024-09-16T00:16:09.021208+0000","last_became_peered":"2024-09-16T00:16:09.021208+0000","last_unstale":"2024-09-16T00:17:09.575364+0000","last_undegraded":"2024-09-16T00:17:09.575364+0000","last_fullsized":"2024-09-16T00:17:09.575364+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:40.605498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580422+0000","last_change":"2024-09-16T00:15:13.575606+0000","last_active":"2024-09-16T00:17:09.580422+0000","last_peered":"2024-09-16T00:17:09.580422+0000","last_clean":"2024-09-16T00:17:09.580422+0000","last_became_active":"2024-09-16T00:15:13.575131+0000","last_became_peered":"2024-09-16T00:15:13.575131+0000","last_unstale":"2024-09-16T00:17:09.580422+0000","last_undegraded":"2024-09-16T00:17:09.580422+0000","last_fullsized":"2024-09-16T00:17:09.580422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:41.636058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.584161+0000","last_change":"2024-09-16T00:17:09.583522+0000","last_active":"2024-09-16T00:17:09.584161+0000","last_peered":"2024-09-16T00:17:09.584161+0000","last_clean":"2024-09-16T00:17:09.584161+0000","last_became_active":"2024-09-16T00:17:09.583082+0000","last_became_peered":"2024-09-16T00:17:09.583082+0000","last_unstale":"2024-09-16T00:17:09.584161+0000","last_undegraded":"2024-09-16T00:17:09.584161+0000","last_fullsized":"2024-09-16T00:17:09.584161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:48.582126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577245+0000","last_change":"2024-09-16T00:15:04.349095+0000","last_active":"2024-09-16T00:17:09.577245+0000","last_peered":"2024-09-16T00:17:09.577245+0000","last_clean":"2024-09-16T00:17:09.577245+0000","last_became_active":"2024-09-16T00:15:04.348698+0000","last_became_peered":"2024-09-16T00:15:04.348698+0000","last_unstale":"2024-09-16T00:17:09.577245+0000","last_undegraded":"2024-09-16T00:17:09.577245+0000","last_fullsized":"2024-09-16T00:17:09.577245+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:42.282882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577535+0000","last_change":"2024-09-16T00:15:13.580494+0000","last_active":"2024-09-16T00:17:09.577535+0000","last_peered":"2024-09-16T00:17:09.577535+0000","last_clean":"2024-09-16T00:17:09.577535+0000","last_became_active":"2024-09-16T00:15:13.580091+0000","last_became_peered":"2024-09-16T00:15:13.580091+0000","last_unstale":"2024-09-16T00:17:09.577535+0000","last_undegraded":"2024-09-16T00:17:09.577535+0000","last_fullsized":"2024-09-16T00:17:09.577535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:17.314919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580727+0000","last_change":"2024-09-16T00:15:06.544859+0000","last_active":"2024-09-16T00:17:09.580727+0000","last_peered":"2024-09-16T00:17:09.580727+0000","last_clean":"2024-09-16T00:17:09.580727+0000","last_became_active":"2024-09-16T00:15:06.544289+0000","last_became_peered":"2024-09-16T00:15:06.544289+0000","last_unstale":"2024-09-16T00:17:09.580727+0000","last_undegraded":"2024-09-16T00:17:09.580727+0000","last_fullsized":"2024-09-16T00:17:09.580727+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:52.622477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718205+0000","last_change":"2024-09-16T00:15:04.351313+0000","last_active":"2024-09-16T00:17:09.718205+0000","last_peered":"2024-09-16T00:17:09.718205+0000","last_clean":"2024-09-16T00:17:09.718205+0000","last_became_active":"2024-09-16T00:15:04.350928+0000","last_became_peered":"2024-09-16T00:15:04.350928+0000","last_unstale":"2024-09-16T00:17:09.718205+0000","last_undegraded":"2024-09-16T00:17:09.718205+0000","last_fullsized":"2024-09-16T00:17:09.718205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:48.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576571+0000","last_change":"2024-09-16T00:15:13.578573+0000","last_active":"2024-09-16T00:17:09.576571+0000","last_peered":"2024-09-16T00:17:09.576571+0000","last_clean":"2024-09-16T00:17:09.576571+0000","last_became_active":"2024-09-16T00:15:13.578117+0000","last_became_peered":"2024-09-16T00:15:13.578117+0000","last_unstale":"2024-09-16T00:17:09.576571+0000","last_undegraded":"2024-09-16T00:17:09.576571+0000","last_fullsized":"2024-09-16T00:17:09.576571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:53.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781258+0000","last_change":"2024-09-16T00:15:06.553548+0000","last_active":"2024-09-16T00:17:09.781258+0000","last_peered":"2024-09-16T00:17:09.781258+0000","last_clean":"2024-09-16T00:17:09.781258+0000","last_became_active":"2024-09-16T00:15:06.552963+0000","last_became_peered":"2024-09-16T00:15:06.552963+0000","last_unstale":"2024-09-16T00:17:09.781258+0000","last_undegraded":"2024-09-16T00:17:09.781258+0000","last_fullsized":"2024-09-16T00:17:09.781258+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:43.455881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579776+0000","last_change":"2024-09-16T00:15:04.351005+0000","last_active":"2024-09-16T00:17:09.579776+0000","last_peered":"2024-09-16T00:17:09.579776+0000","last_clean":"2024-09-16T00:17:09.579776+0000","last_became_active":"2024-09-16T00:15:04.350614+0000","last_became_peered":"2024-09-16T00:15:04.350614+0000","last_unstale":"2024-09-16T00:17:09.579776+0000","last_undegraded":"2024-09-16T00:17:09.579776+0000","last_fullsized":"2024-09-16T00:17:09.579776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:39.196360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579501+0000","last_change":"2024-09-16T00:16:09.023796+0000","last_active":"2024-09-16T00:17:09.579501+0000","last_peered":"2024-09-16T00:17:09.579501+0000","last_clean":"2024-09-16T00:17:09.579501+0000","last_became_active":"2024-09-16T00:16:09.023403+0000","last_became_peered":"2024-09-16T00:16:09.023403+0000","last_unstale":"2024-09-16T00:17:09.579501+0000","last_undegraded":"2024-09-16T00:17:09.579501+0000","last_fullsized":"2024-09-16T00:17:09.579501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:59.488315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577059+0000","last_change":"2024-09-16T00:15:06.546325+0000","last_active":"2024-09-16T00:17:09.577059+0000","last_peered":"2024-09-16T00:17:09.577059+0000","last_clean":"2024-09-16T00:17:09.577059+0000","last_became_active":"2024-09-16T00:15:06.546028+0000","last_became_peered":"2024-09-16T00:15:06.546028+0000","last_unstale":"2024-09-16T00:17:09.577059+0000","last_undegraded":"2024-09-16T00:17:09.577059+0000","last_fullsized":"2024-09-16T00:17:09.577059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:15.049592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579065+0000","last_change":"2024-09-16T00:15:04.373818+0000","last_active":"2024-09-16T00:17:09.579065+0000","last_peered":"2024-09-16T00:17:09.579065+0000","last_clean":"2024-09-16T00:17:09.579065+0000","last_became_active":"2024-09-16T00:15:04.373406+0000","last_became_peered":"2024-09-16T00:15:04.373406+0000","last_unstale":"2024-09-16T00:17:09.579065+0000","last_undegraded":"2024-09-16T00:17:09.579065+0000","last_fullsized":"2024-09-16T00:17:09.579065+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:08.402660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718065+0000","last_change":"2024-09-16T00:15:13.588669+0000","last_active":"2024-09-16T00:17:09.718065+0000","last_peered":"2024-09-16T00:17:09.718065+0000","last_clean":"2024-09-16T00:17:09.718065+0000","last_became_active":"2024-09-16T00:15:13.588229+0000","last_became_peered":"2024-09-16T00:15:13.588229+0000","last_unstale":"2024-09-16T00:17:09.718065+0000","last_undegraded":"2024-09-16T00:17:09.718065+0000","last_fullsized":"2024-09-16T00:17:09.718065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:30.157296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580525+0000","last_change":"2024-09-16T00:15:06.550221+0000","last_active":"2024-09-16T00:17:09.580525+0000","last_peered":"2024-09-16T00:17:09.580525+0000","last_clean":"2024-09-16T00:17:09.580525+0000","last_became_active":"2024-09-16T00:15:06.549914+0000","last_became_peered":"2024-09-16T00:15:06.549914+0000","last_unstale":"2024-09-16T00:17:09.580525+0000","last_undegraded":"2024-09-16T00:17:09.580525+0000","last_fullsized":"2024-09-16T00:17:09.580525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:18:26.824523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:29.315869+0000","last_change":"2024-09-16T00:15:04.344389+0000","last_active":"2024-09-16T00:17:29.315869+0000","last_peered":"2024-09-16T00:17:29.315869+0000","last_clean":"2024-09-16T00:17:29.315869+0000","last_became_active":"2024-09-16T00:15:04.343956+0000","last_became_peered":"2024-09-16T00:15:04.343956+0000","last_unstale":"2024-09-16T00:17:29.315869+0000","last_undegraded":"2024-09-16T00:17:29.315869+0000","last_fullsized":"2024-09-16T00:17:29.315869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:34.019861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579707+0000","last_change":"2024-09-16T00:16:09.024050+0000","last_active":"2024-09-16T00:17:09.579707+0000","last_peered":"2024-09-16T00:17:09.579707+0000","last_clean":"2024-09-16T00:17:09.579707+0000","last_became_active":"2024-09-16T00:16:09.023371+0000","last_became_peered":"2024-09-16T00:16:09.023371+0000","last_unstale":"2024-09-16T00:17:09.579707+0000","last_undegraded":"2024-09-16T00:17:09.579707+0000","last_fullsized":"2024-09-16T00:17:09.579707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:35:19.020370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581433+0000","last_change":"2024-09-16T00:15:06.542744+0000","last_active":"2024-09-16T00:17:09.581433+0000","last_peered":"2024-09-16T00:17:09.581433+0000","last_clean":"2024-09-16T00:17:09.581433+0000","last_became_active":"2024-09-16T00:15:06.541868+0000","last_became_peered":"2024-09-16T00:15:06.541868+0000","last_unstale":"2024-09-16T00:17:09.581433+0000","last_undegraded":"2024-09-16T00:17:09.581433+0000","last_fullsized":"2024-09-16T00:17:09.581433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:10:06.508682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577336+0000","last_change":"2024-09-16T00:15:04.349556+0000","last_active":"2024-09-16T00:17:09.577336+0000","last_peered":"2024-09-16T00:17:09.577336+0000","last_clean":"2024-09-16T00:17:09.577336+0000","last_became_active":"2024-09-16T00:15:04.349242+0000","last_became_peered":"2024-09-16T00:15:04.349242+0000","last_unstale":"2024-09-16T00:17:09.577336+0000","last_undegraded":"2024-09-16T00:17:09.577336+0000","last_fullsized":"2024-09-16T00:17:09.577336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:22:18.459157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781094+0000","last_change":"2024-09-16T00:15:13.584109+0000","last_active":"2024-09-16T00:17:09.781094+0000","last_peered":"2024-09-16T00:17:09.781094+0000","last_clean":"2024-09-16T00:17:09.781094+0000","last_became_active":"2024-09-16T00:15:13.583718+0000","last_became_peered":"2024-09-16T00:15:13.583718+0000","last_unstale":"2024-09-16T00:17:09.781094+0000","last_undegraded":"2024-09-16T00:17:09.781094+0000","last_fullsized":"2024-09-16T00:17:09.781094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:29.416254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717952+0000","last_change":"2024-09-16T00:15:06.548247+0000","last_active":"2024-09-16T00:17:09.717952+0000","last_peered":"2024-09-16T00:17:09.717952+0000","last_clean":"2024-09-16T00:17:09.717952+0000","last_became_active":"2024-09-16T00:15:06.547864+0000","last_became_peered":"2024-09-16T00:15:06.547864+0000","last_unstale":"2024-09-16T00:17:09.717952+0000","last_undegraded":"2024-09-16T00:17:09.717952+0000","last_fullsized":"2024-09-16T00:17:09.717952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:25.350197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575352+0000","last_change":"2024-09-16T00:15:04.350965+0000","last_active":"2024-09-16T00:17:09.575352+0000","last_peered":"2024-09-16T00:17:09.575352+0000","last_clean":"2024-09-16T00:17:09.575352+0000","last_became_active":"2024-09-16T00:15:04.349124+0000","last_became_peered":"2024-09-16T00:15:04.349124+0000","last_unstale":"2024-09-16T00:17:09.575352+0000","last_undegraded":"2024-09-16T00:17:09.575352+0000","last_fullsized":"2024-09-16T00:17:09.575352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:40.859220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576904+0000","last_change":"2024-09-16T00:15:13.581718+0000","last_active":"2024-09-16T00:17:09.576904+0000","last_peered":"2024-09-16T00:17:09.576904+0000","last_clean":"2024-09-16T00:17:09.576904+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:17:09.576904+0000","last_undegraded":"2024-09-16T00:17:09.576904+0000","last_fullsized":"2024-09-16T00:17:09.576904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:08.810606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579507+0000","last_change":"2024-09-16T00:15:06.544507+0000","last_active":"2024-09-16T00:17:09.579507+0000","last_peered":"2024-09-16T00:17:09.579507+0000","last_clean":"2024-09-16T00:17:09.579507+0000","last_became_active":"2024-09-16T00:15:06.544178+0000","last_became_peered":"2024-09-16T00:15:06.544178+0000","last_unstale":"2024-09-16T00:17:09.579507+0000","last_undegraded":"2024-09-16T00:17:09.579507+0000","last_fullsized":"2024-09-16T00:17:09.579507+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:38.494266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'37","reported_seq":301,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:28.155901+0000","last_change":"2024-09-16T00:16:09.039763+0000","last_active":"2024-09-16T00:17:28.155901+0000","last_peered":"2024-09-16T00:17:28.155901+0000","last_clean":"2024-09-16T00:17:28.155901+0000","last_became_active":"2024-09-16T00:16:09.022939+0000","last_became_peered":"2024-09-16T00:16:09.022939+0000","last_unstale":"2024-09-16T00:17:28.155901+0000","last_undegraded":"2024-09-16T00:17:28.155901+0000","last_fullsized":"2024-09-16T00:17:28.155901+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:56.431159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63488,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":204,"num_read_kb":155,"num_write":38,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60943,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576838+0000","last_change":"2024-09-16T00:15:13.585928+0000","last_active":"2024-09-16T00:17:09.576838+0000","last_peered":"2024-09-16T00:17:09.576838+0000","last_clean":"2024-09-16T00:17:09.576838+0000","last_became_active":"2024-09-16T00:15:13.582940+0000","last_became_peered":"2024-09-16T00:15:13.582940+0000","last_unstale":"2024-09-16T00:17:09.576838+0000","last_undegraded":"2024-09-16T00:17:09.576838+0000","last_fullsized":"2024-09-16T00:17:09.576838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:57:13.251978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580875+0000","last_change":"2024-09-16T00:15:06.546238+0000","last_active":"2024-09-16T00:17:09.580875+0000","last_peered":"2024-09-16T00:17:09.580875+0000","last_clean":"2024-09-16T00:17:09.580875+0000","last_became_active":"2024-09-16T00:15:06.545919+0000","last_became_peered":"2024-09-16T00:15:06.545919+0000","last_unstale":"2024-09-16T00:17:09.580875+0000","last_undegraded":"2024-09-16T00:17:09.580875+0000","last_fullsized":"2024-09-16T00:17:09.580875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:09.180832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577083+0000","last_change":"2024-09-16T00:15:04.376028+0000","last_active":"2024-09-16T00:17:09.577083+0000","last_peered":"2024-09-16T00:17:09.577083+0000","last_clean":"2024-09-16T00:17:09.577083+0000","last_became_active":"2024-09-16T00:15:04.375633+0000","last_became_peered":"2024-09-16T00:15:04.375633+0000","last_unstale":"2024-09-16T00:17:09.577083+0000","last_undegraded":"2024-09-16T00:17:09.577083+0000","last_fullsized":"2024-09-16T00:17:09.577083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:44.446342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575222+0000","last_change":"2024-09-16T00:16:09.025524+0000","last_active":"2024-09-16T00:17:09.575222+0000","last_peered":"2024-09-16T00:17:09.575222+0000","last_clean":"2024-09-16T00:17:09.575222+0000","last_became_active":"2024-09-16T00:16:09.025035+0000","last_became_peered":"2024-09-16T00:16:09.025035+0000","last_unstale":"2024-09-16T00:17:09.575222+0000","last_undegraded":"2024-09-16T00:17:09.575222+0000","last_fullsized":"2024-09-16T00:17:09.575222+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:12.367244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577569+0000","last_change":"2024-09-16T00:15:06.550864+0000","last_active":"2024-09-16T00:17:09.577569+0000","last_peered":"2024-09-16T00:17:09.577569+0000","last_clean":"2024-09-16T00:17:09.577569+0000","last_became_active":"2024-09-16T00:15:06.547956+0000","last_became_peered":"2024-09-16T00:15:06.547956+0000","last_unstale":"2024-09-16T00:17:09.577569+0000","last_undegraded":"2024-09-16T00:17:09.577569+0000","last_fullsized":"2024-09-16T00:17:09.577569+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:26.102175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580635+0000","last_change":"2024-09-16T00:15:04.350349+0000","last_active":"2024-09-16T00:17:09.580635+0000","last_peered":"2024-09-16T00:17:09.580635+0000","last_clean":"2024-09-16T00:17:09.580635+0000","last_became_active":"2024-09-16T00:15:04.349951+0000","last_became_peered":"2024-09-16T00:15:04.349951+0000","last_unstale":"2024-09-16T00:17:09.580635+0000","last_undegraded":"2024-09-16T00:17:09.580635+0000","last_fullsized":"2024-09-16T00:17:09.580635+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:46.000831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577297+0000","last_change":"2024-09-16T00:16:09.025774+0000","last_active":"2024-09-16T00:17:09.577297+0000","last_peered":"2024-09-16T00:17:09.577297+0000","last_clean":"2024-09-16T00:17:09.577297+0000","last_became_active":"2024-09-16T00:16:09.025308+0000","last_became_peered":"2024-09-16T00:16:09.025308+0000","last_unstale":"2024-09-16T00:17:09.577297+0000","last_undegraded":"2024-09-16T00:17:09.577297+0000","last_fullsized":"2024-09-16T00:17:09.577297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:16.664850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580734+0000","last_change":"2024-09-16T00:15:06.552919+0000","last_active":"2024-09-16T00:17:09.580734+0000","last_peered":"2024-09-16T00:17:09.580734+0000","last_clean":"2024-09-16T00:17:09.580734+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:17:09.580734+0000","last_undegraded":"2024-09-16T00:17:09.580734+0000","last_fullsized":"2024-09-16T00:17:09.580734+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:47:38.651837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":139,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:26.311189+0000","last_change":"2024-09-16T00:15:04.351116+0000","last_active":"2024-09-16T00:17:26.311189+0000","last_peered":"2024-09-16T00:17:26.311189+0000","last_clean":"2024-09-16T00:17:26.311189+0000","last_became_active":"2024-09-16T00:15:04.350728+0000","last_became_peered":"2024-09-16T00:15:04.350728+0000","last_unstale":"2024-09-16T00:17:26.311189+0000","last_undegraded":"2024-09-16T00:17:26.311189+0000","last_fullsized":"2024-09-16T00:17:26.311189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:38.314019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579276+0000","last_change":"2024-09-16T00:15:13.575961+0000","last_active":"2024-09-16T00:17:09.579276+0000","last_peered":"2024-09-16T00:17:09.579276+0000","last_clean":"2024-09-16T00:17:09.579276+0000","last_became_active":"2024-09-16T00:15:13.575550+0000","last_became_peered":"2024-09-16T00:15:13.575550+0000","last_unstale":"2024-09-16T00:17:09.579276+0000","last_undegraded":"2024-09-16T00:17:09.579276+0000","last_fullsized":"2024-09-16T00:17:09.579276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:05.515588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577699+0000","last_change":"2024-09-16T00:15:13.583583+0000","last_active":"2024-09-16T00:17:09.577699+0000","last_peered":"2024-09-16T00:17:09.577699+0000","last_clean":"2024-09-16T00:17:09.577699+0000","last_became_active":"2024-09-16T00:15:13.583145+0000","last_became_peered":"2024-09-16T00:15:13.583145+0000","last_unstale":"2024-09-16T00:17:09.577699+0000","last_undegraded":"2024-09-16T00:17:09.577699+0000","last_fullsized":"2024-09-16T00:17:09.577699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:07.320337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.719269+0000","last_change":"2024-09-16T00:17:09.719269+0000","last_active":"2024-09-16T00:17:09.719269+0000","last_peered":"2024-09-16T00:17:09.719269+0000","last_clean":"2024-09-16T00:17:09.719269+0000","last_became_active":"2024-09-16T00:17:09.718887+0000","last_became_peered":"2024-09-16T00:17:09.718887+0000","last_unstale":"2024-09-16T00:17:09.719269+0000","last_undegraded":"2024-09-16T00:17:09.719269+0000","last_fullsized":"2024-09-16T00:17:09.719269+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:14:51.675738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577333+0000","last_change":"2024-09-16T00:15:04.350826+0000","last_active":"2024-09-16T00:17:09.577333+0000","last_peered":"2024-09-16T00:17:09.577333+0000","last_clean":"2024-09-16T00:17:09.577333+0000","last_became_active":"2024-09-16T00:15:04.350294+0000","last_became_peered":"2024-09-16T00:15:04.350294+0000","last_unstale":"2024-09-16T00:17:09.577333+0000","last_undegraded":"2024-09-16T00:17:09.577333+0000","last_fullsized":"2024-09-16T00:17:09.577333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:49.146709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781194+0000","last_change":"2024-09-16T00:15:13.580184+0000","last_active":"2024-09-16T00:17:09.781194+0000","last_peered":"2024-09-16T00:17:09.781194+0000","last_clean":"2024-09-16T00:17:09.781194+0000","last_became_active":"2024-09-16T00:15:13.579849+0000","last_became_peered":"2024-09-16T00:15:13.579849+0000","last_unstale":"2024-09-16T00:17:09.781194+0000","last_undegraded":"2024-09-16T00:17:09.781194+0000","last_fullsized":"2024-09-16T00:17:09.781194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:40.613127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579207+0000","last_change":"2024-09-16T00:15:06.545796+0000","last_active":"2024-09-16T00:17:09.579207+0000","last_peered":"2024-09-16T00:17:09.579207+0000","last_clean":"2024-09-16T00:17:09.579207+0000","last_became_active":"2024-09-16T00:15:06.544815+0000","last_became_peered":"2024-09-16T00:15:06.544815+0000","last_unstale":"2024-09-16T00:17:09.579207+0000","last_undegraded":"2024-09-16T00:17:09.579207+0000","last_fullsized":"2024-09-16T00:17:09.579207+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:51:25.921949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718347+0000","last_change":"2024-09-16T00:15:04.340101+0000","last_active":"2024-09-16T00:17:09.718347+0000","last_peered":"2024-09-16T00:17:09.718347+0000","last_clean":"2024-09-16T00:17:09.718347+0000","last_became_active":"2024-09-16T00:15:04.339788+0000","last_became_peered":"2024-09-16T00:15:04.339788+0000","last_unstale":"2024-09-16T00:17:09.718347+0000","last_undegraded":"2024-09-16T00:17:09.718347+0000","last_fullsized":"2024-09-16T00:17:09.718347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:08.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577952+0000","last_change":"2024-09-16T00:16:09.028910+0000","last_active":"2024-09-16T00:17:09.577952+0000","last_peered":"2024-09-16T00:17:09.577952+0000","last_clean":"2024-09-16T00:17:09.577952+0000","last_became_active":"2024-09-16T00:16:09.023302+0000","last_became_peered":"2024-09-16T00:16:09.023302+0000","last_unstale":"2024-09-16T00:17:09.577952+0000","last_undegraded":"2024-09-16T00:17:09.577952+0000","last_fullsized":"2024-09-16T00:17:09.577952+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:35.293676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581116+0000","last_change":"2024-09-16T00:17:09.581116+0000","last_active":"2024-09-16T00:17:09.581116+0000","last_peered":"2024-09-16T00:17:09.581116+0000","last_clean":"2024-09-16T00:17:09.581116+0000","last_became_active":"2024-09-16T00:17:09.580552+0000","last_became_peered":"2024-09-16T00:17:09.580552+0000","last_unstale":"2024-09-16T00:17:09.581116+0000","last_undegraded":"2024-09-16T00:17:09.581116+0000","last_fullsized":"2024-09-16T00:17:09.581116+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:35.975542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574745+0000","last_change":"2024-09-16T00:15:13.581966+0000","last_active":"2024-09-16T00:17:09.574745+0000","last_peered":"2024-09-16T00:17:09.574745+0000","last_clean":"2024-09-16T00:17:09.574745+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:17:09.574745+0000","last_undegraded":"2024-09-16T00:17:09.574745+0000","last_fullsized":"2024-09-16T00:17:09.574745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.132773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":145,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:28.314493+0000","last_change":"2024-09-16T00:15:04.351765+0000","last_active":"2024-09-16T00:17:28.314493+0000","last_peered":"2024-09-16T00:17:28.314493+0000","last_clean":"2024-09-16T00:17:28.314493+0000","last_became_active":"2024-09-16T00:15:04.351398+0000","last_became_peered":"2024-09-16T00:15:04.351398+0000","last_unstale":"2024-09-16T00:17:28.314493+0000","last_undegraded":"2024-09-16T00:17:28.314493+0000","last_fullsized":"2024-09-16T00:17:28.314493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:14:28.762657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.583255+0000","last_change":"2024-09-16T00:17:09.583255+0000","last_active":"2024-09-16T00:17:09.583255+0000","last_peered":"2024-09-16T00:17:09.583255+0000","last_clean":"2024-09-16T00:17:09.583255+0000","last_became_active":"2024-09-16T00:17:09.582793+0000","last_became_peered":"2024-09-16T00:17:09.582793+0000","last_unstale":"2024-09-16T00:17:09.583255+0000","last_undegraded":"2024-09-16T00:17:09.583255+0000","last_fullsized":"2024-09-16T00:17:09.583255+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:28.087903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574875+0000","last_change":"2024-09-16T00:15:13.576877+0000","last_active":"2024-09-16T00:17:09.574875+0000","last_peered":"2024-09-16T00:17:09.574875+0000","last_clean":"2024-09-16T00:17:09.574875+0000","last_became_active":"2024-09-16T00:15:13.576480+0000","last_became_peered":"2024-09-16T00:15:13.576480+0000","last_unstale":"2024-09-16T00:17:09.574875+0000","last_undegraded":"2024-09-16T00:17:09.574875+0000","last_fullsized":"2024-09-16T00:17:09.574875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:47.322219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780925+0000","last_change":"2024-09-16T00:15:06.550329+0000","last_active":"2024-09-16T00:17:09.780925+0000","last_peered":"2024-09-16T00:17:09.780925+0000","last_clean":"2024-09-16T00:17:09.780925+0000","last_became_active":"2024-09-16T00:15:06.550008+0000","last_became_peered":"2024-09-16T00:15:06.550008+0000","last_unstale":"2024-09-16T00:17:09.780925+0000","last_undegraded":"2024-09-16T00:17:09.780925+0000","last_fullsized":"2024-09-16T00:17:09.780925+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:38.096126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582169+0000","last_change":"2024-09-16T00:17:09.580839+0000","last_active":"2024-09-16T00:17:09.582169+0000","last_peered":"2024-09-16T00:17:09.582169+0000","last_clean":"2024-09-16T00:17:09.582169+0000","last_became_active":"2024-09-16T00:17:09.580381+0000","last_became_peered":"2024-09-16T00:17:09.580381+0000","last_unstale":"2024-09-16T00:17:09.582169+0000","last_undegraded":"2024-09-16T00:17:09.582169+0000","last_fullsized":"2024-09-16T00:17:09.582169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:35.425316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574792+0000","last_change":"2024-09-16T00:15:06.546713+0000","last_active":"2024-09-16T00:17:09.574792+0000","last_peered":"2024-09-16T00:17:09.574792+0000","last_clean":"2024-09-16T00:17:09.574792+0000","last_became_active":"2024-09-16T00:15:06.546209+0000","last_became_peered":"2024-09-16T00:15:06.546209+0000","last_unstale":"2024-09-16T00:17:09.574792+0000","last_undegraded":"2024-09-16T00:17:09.574792+0000","last_fullsized":"2024-09-16T00:17:09.574792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:43.913748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577575+0000","last_change":"2024-09-16T00:15:06.550490+0000","last_active":"2024-09-16T00:17:09.577575+0000","last_peered":"2024-09-16T00:17:09.577575+0000","last_clean":"2024-09-16T00:17:09.577575+0000","last_became_active":"2024-09-16T00:15:06.549311+0000","last_became_peered":"2024-09-16T00:15:06.549311+0000","last_unstale":"2024-09-16T00:17:09.577575+0000","last_undegraded":"2024-09-16T00:17:09.577575+0000","last_fullsized":"2024-09-16T00:17:09.577575+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:10:01.995357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718148+0000","last_change":"2024-09-16T00:15:06.548115+0000","last_active":"2024-09-16T00:17:09.718148+0000","last_peered":"2024-09-16T00:17:09.718148+0000","last_clean":"2024-09-16T00:17:09.718148+0000","last_became_active":"2024-09-16T00:15:06.547782+0000","last_became_peered":"2024-09-16T00:15:06.547782+0000","last_unstale":"2024-09-16T00:17:09.718148+0000","last_undegraded":"2024-09-16T00:17:09.718148+0000","last_fullsized":"2024-09-16T00:17:09.718148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:53.228222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781260+0000","last_change":"2024-09-16T00:15:06.554396+0000","last_active":"2024-09-16T00:17:09.781260+0000","last_peered":"2024-09-16T00:17:09.781260+0000","last_clean":"2024-09-16T00:17:09.781260+0000","last_became_active":"2024-09-16T00:15:06.554012+0000","last_became_peered":"2024-09-16T00:15:06.554012+0000","last_unstale":"2024-09-16T00:17:09.781260+0000","last_undegraded":"2024-09-16T00:17:09.781260+0000","last_fullsized":"2024-09-16T00:17:09.781260+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:31.464098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575494+0000","last_change":"2024-09-16T00:15:06.549969+0000","last_active":"2024-09-16T00:17:09.575494+0000","last_peered":"2024-09-16T00:17:09.575494+0000","last_clean":"2024-09-16T00:17:09.575494+0000","last_became_active":"2024-09-16T00:15:06.549531+0000","last_became_peered":"2024-09-16T00:15:06.549531+0000","last_unstale":"2024-09-16T00:17:09.575494+0000","last_undegraded":"2024-09-16T00:17:09.575494+0000","last_fullsized":"2024-09-16T00:17:09.575494+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:37.883612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579397+0000","last_change":"2024-09-16T00:15:06.547734+0000","last_active":"2024-09-16T00:17:09.579397+0000","last_peered":"2024-09-16T00:17:09.579397+0000","last_clean":"2024-09-16T00:17:09.579397+0000","last_became_active":"2024-09-16T00:15:06.547400+0000","last_became_peered":"2024-09-16T00:15:06.547400+0000","last_unstale":"2024-09-16T00:17:09.579397+0000","last_undegraded":"2024-09-16T00:17:09.579397+0000","last_fullsized":"2024-09-16T00:17:09.579397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:54.455313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576721+0000","last_change":"2024-09-16T00:15:06.551965+0000","last_active":"2024-09-16T00:17:09.576721+0000","last_peered":"2024-09-16T00:17:09.576721+0000","last_clean":"2024-09-16T00:17:09.576721+0000","last_became_active":"2024-09-16T00:15:06.551602+0000","last_became_peered":"2024-09-16T00:15:06.551602+0000","last_unstale":"2024-09-16T00:17:09.576721+0000","last_undegraded":"2024-09-16T00:17:09.576721+0000","last_fullsized":"2024-09-16T00:17:09.576721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:34.951960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579571+0000","last_change":"2024-09-16T00:15:06.549340+0000","last_active":"2024-09-16T00:17:09.579571+0000","last_peered":"2024-09-16T00:17:09.579571+0000","last_clean":"2024-09-16T00:17:09.579571+0000","last_became_active":"2024-09-16T00:15:06.547424+0000","last_became_peered":"2024-09-16T00:15:06.547424+0000","last_unstale":"2024-09-16T00:17:09.579571+0000","last_undegraded":"2024-09-16T00:17:09.579571+0000","last_fullsized":"2024-09-16T00:17:09.579571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:12.087535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580115+0000","last_change":"2024-09-16T00:15:06.555216+0000","last_active":"2024-09-16T00:17:09.580115+0000","last_peered":"2024-09-16T00:17:09.580115+0000","last_clean":"2024-09-16T00:17:09.580115+0000","last_became_active":"2024-09-16T00:15:06.554896+0000","last_became_peered":"2024-09-16T00:15:06.554896+0000","last_unstale":"2024-09-16T00:17:09.580115+0000","last_undegraded":"2024-09-16T00:17:09.580115+0000","last_fullsized":"2024-09-16T00:17:09.580115+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:20.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717980+0000","last_change":"2024-09-16T00:15:06.549752+0000","last_active":"2024-09-16T00:17:09.717980+0000","last_peered":"2024-09-16T00:17:09.717980+0000","last_clean":"2024-09-16T00:17:09.717980+0000","last_became_active":"2024-09-16T00:15:06.549350+0000","last_became_peered":"2024-09-16T00:15:06.549350+0000","last_unstale":"2024-09-16T00:17:09.717980+0000","last_undegraded":"2024-09-16T00:17:09.717980+0000","last_fullsized":"2024-09-16T00:17:09.717980+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:48.615123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781099+0000","last_change":"2024-09-16T00:15:06.549159+0000","last_active":"2024-09-16T00:17:09.781099+0000","last_peered":"2024-09-16T00:17:09.781099+0000","last_clean":"2024-09-16T00:17:09.781099+0000","last_became_active":"2024-09-16T00:15:06.548408+0000","last_became_peered":"2024-09-16T00:15:06.548408+0000","last_unstale":"2024-09-16T00:17:09.781099+0000","last_undegraded":"2024-09-16T00:17:09.781099+0000","last_fullsized":"2024-09-16T00:17:09.781099+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:23.941809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575144+0000","last_change":"2024-09-16T00:15:06.544091+0000","last_active":"2024-09-16T00:17:09.575144+0000","last_peered":"2024-09-16T00:17:09.575144+0000","last_clean":"2024-09-16T00:17:09.575144+0000","last_became_active":"2024-09-16T00:15:06.543761+0000","last_became_peered":"2024-09-16T00:15:06.543761+0000","last_unstale":"2024-09-16T00:17:09.575144+0000","last_undegraded":"2024-09-16T00:17:09.575144+0000","last_fullsized":"2024-09-16T00:17:09.575144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:36:43.253405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582256+0000","last_change":"2024-09-16T00:17:09.581748+0000","last_active":"2024-09-16T00:17:09.582256+0000","last_peered":"2024-09-16T00:17:09.582256+0000","last_clean":"2024-09-16T00:17:09.582256+0000","last_became_active":"2024-09-16T00:17:09.581207+0000","last_became_peered":"2024-09-16T00:17:09.581207+0000","last_unstale":"2024-09-16T00:17:09.582256+0000","last_undegraded":"2024-09-16T00:17:09.582256+0000","last_fullsized":"2024-09-16T00:17:09.582256+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:21.521929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781344+0000","last_change":"2024-09-16T00:15:06.557162+0000","last_active":"2024-09-16T00:17:09.781344+0000","last_peered":"2024-09-16T00:17:09.781344+0000","last_clean":"2024-09-16T00:17:09.781344+0000","last_became_active":"2024-09-16T00:15:06.556831+0000","last_became_peered":"2024-09-16T00:15:06.556831+0000","last_unstale":"2024-09-16T00:17:09.781344+0000","last_undegraded":"2024-09-16T00:17:09.781344+0000","last_fullsized":"2024-09-16T00:17:09.781344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.236915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579887+0000","last_change":"2024-09-16T00:15:06.556161+0000","last_active":"2024-09-16T00:17:09.579887+0000","last_peered":"2024-09-16T00:17:09.579887+0000","last_clean":"2024-09-16T00:17:09.579887+0000","last_became_active":"2024-09-16T00:15:06.555819+0000","last_became_peered":"2024-09-16T00:15:06.555819+0000","last_unstale":"2024-09-16T00:17:09.579887+0000","last_undegraded":"2024-09-16T00:17:09.579887+0000","last_fullsized":"2024-09-16T00:17:09.579887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:29.929977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.583819+0000","last_change":"2024-09-16T00:17:09.583183+0000","last_active":"2024-09-16T00:17:09.583819+0000","last_peered":"2024-09-16T00:17:09.583819+0000","last_clean":"2024-09-16T00:17:09.583819+0000","last_became_active":"2024-09-16T00:17:09.582663+0000","last_became_peered":"2024-09-16T00:17:09.582663+0000","last_unstale":"2024-09-16T00:17:09.583819+0000","last_undegraded":"2024-09-16T00:17:09.583819+0000","last_fullsized":"2024-09-16T00:17:09.583819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:37.544124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577469+0000","last_change":"2024-09-16T00:15:06.547114+0000","last_active":"2024-09-16T00:17:09.577469+0000","last_peered":"2024-09-16T00:17:09.577469+0000","last_clean":"2024-09-16T00:17:09.577469+0000","last_became_active":"2024-09-16T00:15:06.546786+0000","last_became_peered":"2024-09-16T00:15:06.546786+0000","last_unstale":"2024-09-16T00:17:09.577469+0000","last_undegraded":"2024-09-16T00:17:09.577469+0000","last_fullsized":"2024-09-16T00:17:09.577469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:13:26.925603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717890+0000","last_change":"2024-09-16T00:15:06.541168+0000","last_active":"2024-09-16T00:17:09.717890+0000","last_peered":"2024-09-16T00:17:09.717890+0000","last_clean":"2024-09-16T00:17:09.717890+0000","last_became_active":"2024-09-16T00:15:06.540848+0000","last_became_peered":"2024-09-16T00:15:06.540848+0000","last_unstale":"2024-09-16T00:17:09.717890+0000","last_undegraded":"2024-09-16T00:17:09.717890+0000","last_fullsized":"2024-09-16T00:17:09.717890+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.958600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575787+0000","last_change":"2024-09-16T00:15:06.547429+0000","last_active":"2024-09-16T00:17:09.575787+0000","last_peered":"2024-09-16T00:17:09.575787+0000","last_clean":"2024-09-16T00:17:09.575787+0000","last_became_active":"2024-09-16T00:15:06.547081+0000","last_became_peered":"2024-09-16T00:15:06.547081+0000","last_unstale":"2024-09-16T00:17:09.575787+0000","last_undegraded":"2024-09-16T00:17:09.575787+0000","last_fullsized":"2024-09-16T00:17:09.575787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:43.396313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581283+0000","last_change":"2024-09-16T00:15:06.551811+0000","last_active":"2024-09-16T00:17:09.581283+0000","last_peered":"2024-09-16T00:17:09.581283+0000","last_clean":"2024-09-16T00:17:09.581283+0000","last_became_active":"2024-09-16T00:15:06.551424+0000","last_became_peered":"2024-09-16T00:15:06.551424+0000","last_unstale":"2024-09-16T00:17:09.581283+0000","last_undegraded":"2024-09-16T00:17:09.581283+0000","last_fullsized":"2024-09-16T00:17:09.581283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.943070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578024+0000","last_change":"2024-09-16T00:15:06.544697+0000","last_active":"2024-09-16T00:17:09.578024+0000","last_peered":"2024-09-16T00:17:09.578024+0000","last_clean":"2024-09-16T00:17:09.578024+0000","last_became_active":"2024-09-16T00:15:06.544182+0000","last_became_peered":"2024-09-16T00:15:06.544182+0000","last_unstale":"2024-09-16T00:17:09.578024+0000","last_undegraded":"2024-09-16T00:17:09.578024+0000","last_fullsized":"2024-09-16T00:17:09.578024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:45.929092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580265+0000","last_change":"2024-09-16T00:15:06.551488+0000","last_active":"2024-09-16T00:17:09.580265+0000","last_peered":"2024-09-16T00:17:09.580265+0000","last_clean":"2024-09-16T00:17:09.580265+0000","last_became_active":"2024-09-16T00:15:06.551171+0000","last_became_peered":"2024-09-16T00:15:06.551171+0000","last_unstale":"2024-09-16T00:17:09.580265+0000","last_undegraded":"2024-09-16T00:17:09.580265+0000","last_fullsized":"2024-09-16T00:17:09.580265+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:23.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579101+0000","last_change":"2024-09-16T00:15:06.544043+0000","last_active":"2024-09-16T00:17:09.579101+0000","last_peered":"2024-09-16T00:17:09.579101+0000","last_clean":"2024-09-16T00:17:09.579101+0000","last_became_active":"2024-09-16T00:15:06.543014+0000","last_became_peered":"2024-09-16T00:15:06.543014+0000","last_unstale":"2024-09-16T00:17:09.579101+0000","last_undegraded":"2024-09-16T00:17:09.579101+0000","last_fullsized":"2024-09-16T00:17:09.579101+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:23.904810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581718+0000","last_change":"2024-09-16T00:15:06.546788+0000","last_active":"2024-09-16T00:17:09.581718+0000","last_peered":"2024-09-16T00:17:09.581718+0000","last_clean":"2024-09-16T00:17:09.581718+0000","last_became_active":"2024-09-16T00:15:06.546333+0000","last_became_peered":"2024-09-16T00:15:06.546333+0000","last_unstale":"2024-09-16T00:17:09.581718+0000","last_undegraded":"2024-09-16T00:17:09.581718+0000","last_fullsized":"2024-09-16T00:17:09.581718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:09.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.719618+0000","last_change":"2024-09-16T00:17:09.719618+0000","last_active":"2024-09-16T00:17:09.719618+0000","last_peered":"2024-09-16T00:17:09.719618+0000","last_clean":"2024-09-16T00:17:09.719618+0000","last_became_active":"2024-09-16T00:17:09.719172+0000","last_became_peered":"2024-09-16T00:17:09.719172+0000","last_unstale":"2024-09-16T00:17:09.719618+0000","last_undegraded":"2024-09-16T00:17:09.719618+0000","last_fullsized":"2024-09-16T00:17:09.719618+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:51.709719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579285+0000","last_change":"2024-09-16T00:15:06.540675+0000","last_active":"2024-09-16T00:17:09.579285+0000","last_peered":"2024-09-16T00:17:09.579285+0000","last_clean":"2024-09-16T00:17:09.579285+0000","last_became_active":"2024-09-16T00:15:06.540333+0000","last_became_peered":"2024-09-16T00:15:06.540333+0000","last_unstale":"2024-09-16T00:17:09.579285+0000","last_undegraded":"2024-09-16T00:17:09.579285+0000","last_fullsized":"2024-09-16T00:17:09.579285+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:17.166987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575821+0000","last_change":"2024-09-16T00:15:06.548758+0000","last_active":"2024-09-16T00:17:09.575821+0000","last_peered":"2024-09-16T00:17:09.575821+0000","last_clean":"2024-09-16T00:17:09.575821+0000","last_became_active":"2024-09-16T00:15:06.548397+0000","last_became_peered":"2024-09-16T00:15:06.548397+0000","last_unstale":"2024-09-16T00:17:09.575821+0000","last_undegraded":"2024-09-16T00:17:09.575821+0000","last_fullsized":"2024-09-16T00:17:09.575821+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:50.320170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576644+0000","last_change":"2024-09-16T00:15:06.554898+0000","last_active":"2024-09-16T00:17:09.576644+0000","last_peered":"2024-09-16T00:17:09.576644+0000","last_clean":"2024-09-16T00:17:09.576644+0000","last_became_active":"2024-09-16T00:15:06.553005+0000","last_became_peered":"2024-09-16T00:15:06.553005+0000","last_unstale":"2024-09-16T00:17:09.576644+0000","last_undegraded":"2024-09-16T00:17:09.576644+0000","last_fullsized":"2024-09-16T00:17:09.576644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:29.630911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577231+0000","last_change":"2024-09-16T00:15:06.547265+0000","last_active":"2024-09-16T00:17:09.577231+0000","last_peered":"2024-09-16T00:17:09.577231+0000","last_clean":"2024-09-16T00:17:09.577231+0000","last_became_active":"2024-09-16T00:15:06.546938+0000","last_became_peered":"2024-09-16T00:15:06.546938+0000","last_unstale":"2024-09-16T00:17:09.577231+0000","last_undegraded":"2024-09-16T00:17:09.577231+0000","last_fullsized":"2024-09-16T00:17:09.577231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:40.247343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781308+0000","last_change":"2024-09-16T00:15:06.556735+0000","last_active":"2024-09-16T00:17:09.781308+0000","last_peered":"2024-09-16T00:17:09.781308+0000","last_clean":"2024-09-16T00:17:09.781308+0000","last_became_active":"2024-09-16T00:15:06.556324+0000","last_became_peered":"2024-09-16T00:15:06.556324+0000","last_unstale":"2024-09-16T00:17:09.781308+0000","last_undegraded":"2024-09-16T00:17:09.781308+0000","last_fullsized":"2024-09-16T00:17:09.781308+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:25.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68025,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":622,"num_read_kb":373,"num_write":178,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":62107,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":209055,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249414,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579947,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":517057,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808879,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36464,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708688,"statfs":{"total":95995035648,"available":95957696512,"internally_reserved":0,"allocated":1142784,"data_stored":1168039,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972405,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1015808,"data_stored":521389,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201337,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1007616,"data_stored":516631,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430271,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1060864,"data_stored":1113265,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626501,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":512469,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855434,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1069056,"data_stored":1108319,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986255,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1040384,"data_stored":587882,"data_compressed":39297,"data_compressed_allocated":53248,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215187,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":512663,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444121,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1007616,"data_stored":513147,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607646,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1064960,"data_stored":576449,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1015808,"data_stored":513857,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":71407,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T00:17:33.218 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-09-16T00:17:33.218 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T00:17:33.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:32 smithi060 ceph-mon[35185]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:17:33.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:32 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2755694435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.356+0000 7fa8e214b640 1 -- 172.21.15.33:0/2716435303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8dc102600 msgr2=0x7fa8dc108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.356+0000 7fa8e214b640 1 --2- 172.21.15.33:0/2716435303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8dc102600 0x7fa8dc108e90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009ca0 tx=0x7fa8d402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.356+0000 7fa8e214b640 1 -- 172.21.15.33:0/2716435303 shutdown_connections 2024-09-16T00:17:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.356+0000 7fa8e214b640 1 --2- 172.21.15.33:0/2716435303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8dc102600 0x7fa8dc108e90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.356+0000 7fa8e214b640 1 --2- 172.21.15.33:0/2716435303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8dc101c40 0x7fa8dc1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.356+0000 7fa8e214b640 1 --2- 172.21.15.33:0/2716435303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8dc100a50 0x7fa8dc100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.356+0000 7fa8e214b640 1 -- 172.21.15.33:0/2716435303 >> 172.21.15.33:0/2716435303 conn(0x7fa8dc0fc190 msgr2=0x7fa8dc0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.356+0000 7fa8e214b640 1 -- 172.21.15.33:0/2716435303 shutdown_connections 2024-09-16T00:17:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.356+0000 7fa8e214b640 1 -- 172.21.15.33:0/2716435303 wait complete. 2024-09-16T00:17:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.357+0000 7fa8e214b640 1 Processor -- start 2024-09-16T00:17:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.357+0000 7fa8e214b640 1 -- start start 2024-09-16T00:17:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.357+0000 7fa8e214b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8dc100a50 0x7fa8dc197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.357+0000 7fa8e214b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8dc101c40 0x7fa8dc198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.357+0000 7fa8e214b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8dc102600 0x7fa8dc19f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.357+0000 7fa8e214b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc10d820 con 0x7fa8dc101c40 2024-09-16T00:17:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.357+0000 7fa8e214b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc10d6a0 con 0x7fa8dc100a50 2024-09-16T00:17:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.357+0000 7fa8e214b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc10d9a0 con 0x7fa8dc102600 2024-09-16T00:17:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.357+0000 7fa8daffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8dc101c40 0x7fa8dc198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.357+0000 7fa8daffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8dc101c40 0x7fa8dc198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36274/0 (socket says 172.21.15.33:36274) 2024-09-16T00:17:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.357+0000 7fa8daffd640 1 -- 172.21.15.33:0/1646697335 learned_addr learned my addr 172.21.15.33:0/1646697335 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.357+0000 7fa8dbfff640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8dc102600 0x7fa8dc19f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8db7fe640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8dc100a50 0x7fa8dc197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8daffd640 1 -- 172.21.15.33:0/1646697335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8dc102600 msgr2=0x7fa8dc19f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8daffd640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8dc102600 0x7fa8dc19f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8daffd640 1 -- 172.21.15.33:0/1646697335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8dc100a50 msgr2=0x7fa8dc197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8daffd640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8dc100a50 0x7fa8dc197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8daffd640 1 -- 172.21.15.33:0/1646697335 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8dc19f970 con 0x7fa8dc101c40 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8db7fe640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8dc100a50 0x7fa8dc197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8daffd640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8dc101c40 0x7fa8dc198290 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000c920 tx=0x7fa8d000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8d8ff9640 1 -- 172.21.15.33:0/1646697335 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0007c70 con 0x7fa8dc101c40 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8d8ff9640 1 -- 172.21.15.33:0/1646697335 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8d0007e10 con 0x7fa8dc101c40 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8e214b640 1 -- 172.21.15.33:0/1646697335 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8dc19fc60 con 0x7fa8dc101c40 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8d8ff9640 1 -- 172.21.15.33:0/1646697335 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0018890 con 0x7fa8dc101c40 2024-09-16T00:17:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.358+0000 7fa8e214b640 1 -- 172.21.15.33:0/1646697335 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8dc1a01b0 con 0x7fa8dc101c40 2024-09-16T00:17:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.359+0000 7fa8e214b640 1 -- 172.21.15.33:0/1646697335 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8dc107d70 con 0x7fa8dc101c40 2024-09-16T00:17:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.360+0000 7fa8d8ff9640 1 -- 172.21.15.33:0/1646697335 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fa8d0018a70 con 0x7fa8dc101c40 2024-09-16T00:17:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.361+0000 7fa8d8ff9640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8c40a32e0 0x7fa8c40a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.361+0000 7fa8db7fe640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8c40a32e0 0x7fa8c40a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.361+0000 7fa8d8ff9640 1 -- 172.21.15.33:0/1646697335 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa8d00d0070 con 0x7fa8dc101c40 2024-09-16T00:17:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.361+0000 7fa8db7fe640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8c40a32e0 0x7fa8c40a57a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc101aa0 tx=0x7fa8cc005e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:33.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.365+0000 7fa8d8ff9640 1 -- 172.21.15.33:0/1646697335 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa8d0014030 con 0x7fa8dc101c40 2024-09-16T00:17:33.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.537+0000 7fa8e214b640 1 -- 172.21.15.33:0/1646697335 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa8dc062630 con 0x7fa8c40a32e0 2024-09-16T00:17:33.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.545+0000 7fa8d8ff9640 1 -- 172.21.15.33:0/1646697335 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340766 (secure 0 0 0) 0x7fa8dc062630 con 0x7fa8c40a32e0 2024-09-16T00:17:33.549 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:33.551 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-09-16T00:17:33.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.548+0000 7fa8e214b640 1 -- 172.21.15.33:0/1646697335 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8c40a32e0 msgr2=0x7fa8c40a57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:33.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.548+0000 7fa8e214b640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8c40a32e0 0x7fa8c40a57a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc101aa0 tx=0x7fa8cc005e70 comp rx=0 tx=0).stop 2024-09-16T00:17:33.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.549+0000 7fa8e214b640 1 -- 172.21.15.33:0/1646697335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8dc101c40 msgr2=0x7fa8dc198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:33.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.549+0000 7fa8e214b640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8dc101c40 0x7fa8dc198290 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000c920 tx=0x7fa8d000cde0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.549+0000 7fa8e214b640 1 -- 172.21.15.33:0/1646697335 shutdown_connections 2024-09-16T00:17:33.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.549+0000 7fa8e214b640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa8c40a32e0 0x7fa8c40a57a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.549+0000 7fa8e214b640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8dc102600 0x7fa8dc19f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.549+0000 7fa8e214b640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8dc101c40 0x7fa8dc198290 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.549+0000 7fa8e214b640 1 --2- 172.21.15.33:0/1646697335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8dc100a50 0x7fa8dc197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:33.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.549+0000 7fa8e214b640 1 -- 172.21.15.33:0/1646697335 >> 172.21.15.33:0/1646697335 conn(0x7fa8dc0fc190 msgr2=0x7fa8dc104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:33.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.549+0000 7fa8e214b640 1 -- 172.21.15.33:0/1646697335 shutdown_connections 2024-09-16T00:17:33.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:33.549+0000 7fa8e214b640 1 -- 172.21.15.33:0/1646697335 wait complete. 2024-09-16T00:17:33.560 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-16T00:17:31.670333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658627,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":748,"num_read_kb":482,"num_write":361,"num_write_kb":3065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1242843,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386388,"kb_used_data":12124,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555436,"statfs":{"total":1151940427776,"available":1151544766464,"internally_reserved":0,"allocated":12414976,"data_stored":8161167,"data_compressed":471663,"data_compressed_allocated":638976,"data_compressed_original":2809856,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":701,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":37,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003164"},"pg_stats":[{"pgid":"2.1f","version":"86'9","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578728+0000","last_change":"2024-09-16T00:16:09.034677+0000","last_active":"2024-09-16T00:17:09.578728+0000","last_peered":"2024-09-16T00:17:09.578728+0000","last_clean":"2024-09-16T00:17:09.578728+0000","last_became_active":"2024-09-16T00:16:09.024755+0000","last_became_peered":"2024-09-16T00:16:09.024755+0000","last_unstale":"2024-09-16T00:17:09.578728+0000","last_undegraded":"2024-09-16T00:17:09.578728+0000","last_fullsized":"2024-09-16T00:17:09.578728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:09.012210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579433+0000","last_change":"2024-09-16T00:15:06.547094+0000","last_active":"2024-09-16T00:17:09.579433+0000","last_peered":"2024-09-16T00:17:09.579433+0000","last_clean":"2024-09-16T00:17:09.579433+0000","last_became_active":"2024-09-16T00:15:06.542764+0000","last_became_peered":"2024-09-16T00:15:06.542764+0000","last_unstale":"2024-09-16T00:17:09.579433+0000","last_undegraded":"2024-09-16T00:17:09.579433+0000","last_fullsized":"2024-09-16T00:17:09.579433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:04:37.449577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574641+0000","last_change":"2024-09-16T00:15:13.578400+0000","last_active":"2024-09-16T00:17:09.574641+0000","last_peered":"2024-09-16T00:17:09.574641+0000","last_clean":"2024-09-16T00:17:09.574641+0000","last_became_active":"2024-09-16T00:15:13.577787+0000","last_became_peered":"2024-09-16T00:15:13.577787+0000","last_unstale":"2024-09-16T00:17:09.574641+0000","last_undegraded":"2024-09-16T00:17:09.574641+0000","last_fullsized":"2024-09-16T00:17:09.574641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:19:59.194152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579738+0000","last_change":"2024-09-16T00:15:04.373381+0000","last_active":"2024-09-16T00:17:09.579738+0000","last_peered":"2024-09-16T00:17:09.579738+0000","last_clean":"2024-09-16T00:17:09.579738+0000","last_became_active":"2024-09-16T00:15:04.373028+0000","last_became_peered":"2024-09-16T00:15:04.373028+0000","last_unstale":"2024-09-16T00:17:09.579738+0000","last_undegraded":"2024-09-16T00:17:09.579738+0000","last_fullsized":"2024-09-16T00:17:09.579738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:22.902583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577500+0000","last_change":"2024-09-16T00:15:06.554048+0000","last_active":"2024-09-16T00:17:09.577500+0000","last_peered":"2024-09-16T00:17:09.577500+0000","last_clean":"2024-09-16T00:17:09.577500+0000","last_became_active":"2024-09-16T00:15:06.553044+0000","last_became_peered":"2024-09-16T00:15:06.553044+0000","last_unstale":"2024-09-16T00:17:09.577500+0000","last_undegraded":"2024-09-16T00:17:09.577500+0000","last_fullsized":"2024-09-16T00:17:09.577500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:22:53.333705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574960+0000","last_change":"2024-09-16T00:15:13.586995+0000","last_active":"2024-09-16T00:17:09.574960+0000","last_peered":"2024-09-16T00:17:09.574960+0000","last_clean":"2024-09-16T00:17:09.574960+0000","last_became_active":"2024-09-16T00:15:13.586509+0000","last_became_peered":"2024-09-16T00:15:13.586509+0000","last_unstale":"2024-09-16T00:17:09.574960+0000","last_undegraded":"2024-09-16T00:17:09.574960+0000","last_fullsized":"2024-09-16T00:17:09.574960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:39.418581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579612+0000","last_change":"2024-09-16T00:15:04.348235+0000","last_active":"2024-09-16T00:17:09.579612+0000","last_peered":"2024-09-16T00:17:09.579612+0000","last_clean":"2024-09-16T00:17:09.579612+0000","last_became_active":"2024-09-16T00:15:04.347518+0000","last_became_peered":"2024-09-16T00:15:04.347518+0000","last_unstale":"2024-09-16T00:17:09.579612+0000","last_undegraded":"2024-09-16T00:17:09.579612+0000","last_fullsized":"2024-09-16T00:17:09.579612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:24.396911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581041+0000","last_change":"2024-09-16T00:15:06.554067+0000","last_active":"2024-09-16T00:17:09.581041+0000","last_peered":"2024-09-16T00:17:09.581041+0000","last_clean":"2024-09-16T00:17:09.581041+0000","last_became_active":"2024-09-16T00:15:06.553715+0000","last_became_peered":"2024-09-16T00:15:06.553715+0000","last_unstale":"2024-09-16T00:17:09.581041+0000","last_undegraded":"2024-09-16T00:17:09.581041+0000","last_fullsized":"2024-09-16T00:17:09.581041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:22.008530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575352+0000","last_change":"2024-09-16T00:15:13.586872+0000","last_active":"2024-09-16T00:17:09.575352+0000","last_peered":"2024-09-16T00:17:09.575352+0000","last_clean":"2024-09-16T00:17:09.575352+0000","last_became_active":"2024-09-16T00:15:13.586375+0000","last_became_peered":"2024-09-16T00:15:13.586375+0000","last_unstale":"2024-09-16T00:17:09.575352+0000","last_undegraded":"2024-09-16T00:17:09.575352+0000","last_fullsized":"2024-09-16T00:17:09.575352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:43.668827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579613+0000","last_change":"2024-09-16T00:15:04.347964+0000","last_active":"2024-09-16T00:17:09.579613+0000","last_peered":"2024-09-16T00:17:09.579613+0000","last_clean":"2024-09-16T00:17:09.579613+0000","last_became_active":"2024-09-16T00:15:04.347588+0000","last_became_peered":"2024-09-16T00:15:04.347588+0000","last_unstale":"2024-09-16T00:17:09.579613+0000","last_undegraded":"2024-09-16T00:17:09.579613+0000","last_fullsized":"2024-09-16T00:17:09.579613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:14.199287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576911+0000","last_change":"2024-09-16T00:15:06.544918+0000","last_active":"2024-09-16T00:17:09.576911+0000","last_peered":"2024-09-16T00:17:09.576911+0000","last_clean":"2024-09-16T00:17:09.576911+0000","last_became_active":"2024-09-16T00:15:06.544579+0000","last_became_peered":"2024-09-16T00:15:06.544579+0000","last_unstale":"2024-09-16T00:17:09.576911+0000","last_undegraded":"2024-09-16T00:17:09.576911+0000","last_fullsized":"2024-09-16T00:17:09.576911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:06:37.475644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575052+0000","last_change":"2024-09-16T00:15:13.581488+0000","last_active":"2024-09-16T00:17:09.575052+0000","last_peered":"2024-09-16T00:17:09.575052+0000","last_clean":"2024-09-16T00:17:09.575052+0000","last_became_active":"2024-09-16T00:15:13.581164+0000","last_became_peered":"2024-09-16T00:15:13.581164+0000","last_unstale":"2024-09-16T00:17:09.575052+0000","last_undegraded":"2024-09-16T00:17:09.575052+0000","last_fullsized":"2024-09-16T00:17:09.575052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:05.061592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581271+0000","last_change":"2024-09-16T00:15:06.551982+0000","last_active":"2024-09-16T00:17:09.581271+0000","last_peered":"2024-09-16T00:17:09.581271+0000","last_clean":"2024-09-16T00:17:09.581271+0000","last_became_active":"2024-09-16T00:15:06.551663+0000","last_became_peered":"2024-09-16T00:15:06.551663+0000","last_unstale":"2024-09-16T00:17:09.581271+0000","last_undegraded":"2024-09-16T00:17:09.581271+0000","last_fullsized":"2024-09-16T00:17:09.581271+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:11.928703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579394+0000","last_change":"2024-09-16T00:15:04.352237+0000","last_active":"2024-09-16T00:17:09.579394+0000","last_peered":"2024-09-16T00:17:09.579394+0000","last_clean":"2024-09-16T00:17:09.579394+0000","last_became_active":"2024-09-16T00:15:04.351884+0000","last_became_peered":"2024-09-16T00:15:04.351884+0000","last_unstale":"2024-09-16T00:17:09.579394+0000","last_undegraded":"2024-09-16T00:17:09.579394+0000","last_fullsized":"2024-09-16T00:17:09.579394+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:32.932020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575580+0000","last_change":"2024-09-16T00:15:13.638917+0000","last_active":"2024-09-16T00:17:09.575580+0000","last_peered":"2024-09-16T00:17:09.575580+0000","last_clean":"2024-09-16T00:17:09.575580+0000","last_became_active":"2024-09-16T00:15:13.638401+0000","last_became_peered":"2024-09-16T00:15:13.638401+0000","last_unstale":"2024-09-16T00:17:09.575580+0000","last_undegraded":"2024-09-16T00:17:09.575580+0000","last_fullsized":"2024-09-16T00:17:09.575580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:38.082407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579388+0000","last_change":"2024-09-16T00:15:06.547909+0000","last_active":"2024-09-16T00:17:09.579388+0000","last_peered":"2024-09-16T00:17:09.579388+0000","last_clean":"2024-09-16T00:17:09.579388+0000","last_became_active":"2024-09-16T00:15:06.546814+0000","last_became_peered":"2024-09-16T00:15:06.546814+0000","last_unstale":"2024-09-16T00:17:09.579388+0000","last_undegraded":"2024-09-16T00:17:09.579388+0000","last_fullsized":"2024-09-16T00:17:09.579388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:46.419406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780898+0000","last_change":"2024-09-16T00:16:09.027691+0000","last_active":"2024-09-16T00:17:09.780898+0000","last_peered":"2024-09-16T00:17:09.780898+0000","last_clean":"2024-09-16T00:17:09.780898+0000","last_became_active":"2024-09-16T00:16:09.023204+0000","last_became_peered":"2024-09-16T00:16:09.023204+0000","last_unstale":"2024-09-16T00:17:09.780898+0000","last_undegraded":"2024-09-16T00:17:09.780898+0000","last_fullsized":"2024-09-16T00:17:09.780898+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:44.647576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575627+0000","last_change":"2024-09-16T00:15:13.588345+0000","last_active":"2024-09-16T00:17:09.575627+0000","last_peered":"2024-09-16T00:17:09.575627+0000","last_clean":"2024-09-16T00:17:09.575627+0000","last_became_active":"2024-09-16T00:15:13.587829+0000","last_became_peered":"2024-09-16T00:15:13.587829+0000","last_unstale":"2024-09-16T00:17:09.575627+0000","last_undegraded":"2024-09-16T00:17:09.575627+0000","last_fullsized":"2024-09-16T00:17:09.575627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:04.261031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781137+0000","last_change":"2024-09-16T00:15:06.545485+0000","last_active":"2024-09-16T00:17:09.781137+0000","last_peered":"2024-09-16T00:17:09.781137+0000","last_clean":"2024-09-16T00:17:09.781137+0000","last_became_active":"2024-09-16T00:15:06.545178+0000","last_became_peered":"2024-09-16T00:15:06.545178+0000","last_unstale":"2024-09-16T00:17:09.781137+0000","last_undegraded":"2024-09-16T00:17:09.781137+0000","last_fullsized":"2024-09-16T00:17:09.781137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:26.923469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579782+0000","last_change":"2024-09-16T00:15:04.340546+0000","last_active":"2024-09-16T00:17:09.579782+0000","last_peered":"2024-09-16T00:17:09.579782+0000","last_clean":"2024-09-16T00:17:09.579782+0000","last_became_active":"2024-09-16T00:15:04.340173+0000","last_became_peered":"2024-09-16T00:15:04.340173+0000","last_unstale":"2024-09-16T00:17:09.579782+0000","last_undegraded":"2024-09-16T00:17:09.579782+0000","last_fullsized":"2024-09-16T00:17:09.579782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:35:01.134433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578541+0000","last_change":"2024-09-16T00:15:13.589963+0000","last_active":"2024-09-16T00:17:09.578541+0000","last_peered":"2024-09-16T00:17:09.578541+0000","last_clean":"2024-09-16T00:17:09.578541+0000","last_became_active":"2024-09-16T00:15:13.589615+0000","last_became_peered":"2024-09-16T00:15:13.589615+0000","last_unstale":"2024-09-16T00:17:09.578541+0000","last_undegraded":"2024-09-16T00:17:09.578541+0000","last_fullsized":"2024-09-16T00:17:09.578541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:37.197765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579731+0000","last_change":"2024-09-16T00:15:06.544374+0000","last_active":"2024-09-16T00:17:09.579731+0000","last_peered":"2024-09-16T00:17:09.579731+0000","last_clean":"2024-09-16T00:17:09.579731+0000","last_became_active":"2024-09-16T00:15:06.544027+0000","last_became_peered":"2024-09-16T00:15:06.544027+0000","last_unstale":"2024-09-16T00:17:09.579731+0000","last_undegraded":"2024-09-16T00:17:09.579731+0000","last_fullsized":"2024-09-16T00:17:09.579731+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580105+0000","last_change":"2024-09-16T00:15:04.374209+0000","last_active":"2024-09-16T00:17:09.580105+0000","last_peered":"2024-09-16T00:17:09.580105+0000","last_clean":"2024-09-16T00:17:09.580105+0000","last_became_active":"2024-09-16T00:15:04.373814+0000","last_became_peered":"2024-09-16T00:15:04.373814+0000","last_unstale":"2024-09-16T00:17:09.580105+0000","last_undegraded":"2024-09-16T00:17:09.580105+0000","last_fullsized":"2024-09-16T00:17:09.580105+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:13.516161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575489+0000","last_change":"2024-09-16T00:15:13.639210+0000","last_active":"2024-09-16T00:17:09.575489+0000","last_peered":"2024-09-16T00:17:09.575489+0000","last_clean":"2024-09-16T00:17:09.575489+0000","last_became_active":"2024-09-16T00:15:13.638719+0000","last_became_peered":"2024-09-16T00:15:13.638719+0000","last_unstale":"2024-09-16T00:17:09.575489+0000","last_undegraded":"2024-09-16T00:17:09.575489+0000","last_fullsized":"2024-09-16T00:17:09.575489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:26.130778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577341+0000","last_change":"2024-09-16T00:15:06.553108+0000","last_active":"2024-09-16T00:17:09.577341+0000","last_peered":"2024-09-16T00:17:09.577341+0000","last_clean":"2024-09-16T00:17:09.577341+0000","last_became_active":"2024-09-16T00:15:06.552817+0000","last_became_peered":"2024-09-16T00:15:06.552817+0000","last_unstale":"2024-09-16T00:17:09.577341+0000","last_undegraded":"2024-09-16T00:17:09.577341+0000","last_fullsized":"2024-09-16T00:17:09.577341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:06.821337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575833+0000","last_change":"2024-09-16T00:15:04.644185+0000","last_active":"2024-09-16T00:17:09.575833+0000","last_peered":"2024-09-16T00:17:09.575833+0000","last_clean":"2024-09-16T00:17:09.575833+0000","last_became_active":"2024-09-16T00:15:04.643887+0000","last_became_peered":"2024-09-16T00:15:04.643887+0000","last_unstale":"2024-09-16T00:17:09.575833+0000","last_undegraded":"2024-09-16T00:17:09.575833+0000","last_fullsized":"2024-09-16T00:17:09.575833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:44.279272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574877+0000","last_change":"2024-09-16T00:15:13.574761+0000","last_active":"2024-09-16T00:17:09.574877+0000","last_peered":"2024-09-16T00:17:09.574877+0000","last_clean":"2024-09-16T00:17:09.574877+0000","last_became_active":"2024-09-16T00:15:13.574347+0000","last_became_peered":"2024-09-16T00:15:13.574347+0000","last_unstale":"2024-09-16T00:17:09.574877+0000","last_undegraded":"2024-09-16T00:17:09.574877+0000","last_fullsized":"2024-09-16T00:17:09.574877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:22.040121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579504+0000","last_change":"2024-09-16T00:15:06.550338+0000","last_active":"2024-09-16T00:17:09.579504+0000","last_peered":"2024-09-16T00:17:09.579504+0000","last_clean":"2024-09-16T00:17:09.579504+0000","last_became_active":"2024-09-16T00:15:06.549868+0000","last_became_peered":"2024-09-16T00:15:06.549868+0000","last_unstale":"2024-09-16T00:17:09.579504+0000","last_undegraded":"2024-09-16T00:17:09.579504+0000","last_fullsized":"2024-09-16T00:17:09.579504+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:26.643338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'12","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:13.711683+0000","last_change":"2024-09-16T00:15:04.342306+0000","last_active":"2024-09-16T00:17:13.711683+0000","last_peered":"2024-09-16T00:17:13.711683+0000","last_clean":"2024-09-16T00:17:13.711683+0000","last_became_active":"2024-09-16T00:15:04.341964+0000","last_became_peered":"2024-09-16T00:15:04.341964+0000","last_unstale":"2024-09-16T00:17:13.711683+0000","last_undegraded":"2024-09-16T00:17:13.711683+0000","last_fullsized":"2024-09-16T00:17:13.711683+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:14.137199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578972+0000","last_change":"2024-09-16T00:15:13.638246+0000","last_active":"2024-09-16T00:17:09.578972+0000","last_peered":"2024-09-16T00:17:09.578972+0000","last_clean":"2024-09-16T00:17:09.578972+0000","last_became_active":"2024-09-16T00:15:13.637816+0000","last_became_peered":"2024-09-16T00:15:13.637816+0000","last_unstale":"2024-09-16T00:17:09.578972+0000","last_undegraded":"2024-09-16T00:17:09.578972+0000","last_fullsized":"2024-09-16T00:17:09.578972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:27.331156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718239+0000","last_change":"2024-09-16T00:15:06.544057+0000","last_active":"2024-09-16T00:17:09.718239+0000","last_peered":"2024-09-16T00:17:09.718239+0000","last_clean":"2024-09-16T00:17:09.718239+0000","last_became_active":"2024-09-16T00:15:06.543244+0000","last_became_peered":"2024-09-16T00:15:06.543244+0000","last_unstale":"2024-09-16T00:17:09.718239+0000","last_undegraded":"2024-09-16T00:17:09.718239+0000","last_fullsized":"2024-09-16T00:17:09.718239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:42:28.950014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579701+0000","last_change":"2024-09-16T00:15:04.376419+0000","last_active":"2024-09-16T00:17:09.579701+0000","last_peered":"2024-09-16T00:17:09.579701+0000","last_clean":"2024-09-16T00:17:09.579701+0000","last_became_active":"2024-09-16T00:15:04.375976+0000","last_became_peered":"2024-09-16T00:15:04.375976+0000","last_unstale":"2024-09-16T00:17:09.579701+0000","last_undegraded":"2024-09-16T00:17:09.579701+0000","last_fullsized":"2024-09-16T00:17:09.579701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:33:38.652109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781038+0000","last_change":"2024-09-16T00:15:13.584306+0000","last_active":"2024-09-16T00:17:09.781038+0000","last_peered":"2024-09-16T00:17:09.781038+0000","last_clean":"2024-09-16T00:17:09.781038+0000","last_became_active":"2024-09-16T00:15:13.583840+0000","last_became_peered":"2024-09-16T00:15:13.583840+0000","last_unstale":"2024-09-16T00:17:09.781038+0000","last_undegraded":"2024-09-16T00:17:09.781038+0000","last_fullsized":"2024-09-16T00:17:09.781038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:36.660458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577806+0000","last_change":"2024-09-16T00:15:06.538902+0000","last_active":"2024-09-16T00:17:09.577806+0000","last_peered":"2024-09-16T00:17:09.577806+0000","last_clean":"2024-09-16T00:17:09.577806+0000","last_became_active":"2024-09-16T00:15:06.538537+0000","last_became_peered":"2024-09-16T00:15:06.538537+0000","last_unstale":"2024-09-16T00:17:09.577806+0000","last_undegraded":"2024-09-16T00:17:09.577806+0000","last_fullsized":"2024-09-16T00:17:09.577806+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:38.411711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578013+0000","last_change":"2024-09-16T00:15:04.374762+0000","last_active":"2024-09-16T00:17:09.578013+0000","last_peered":"2024-09-16T00:17:09.578013+0000","last_clean":"2024-09-16T00:17:09.578013+0000","last_became_active":"2024-09-16T00:15:04.374394+0000","last_became_peered":"2024-09-16T00:15:04.374394+0000","last_unstale":"2024-09-16T00:17:09.578013+0000","last_undegraded":"2024-09-16T00:17:09.578013+0000","last_fullsized":"2024-09-16T00:17:09.578013+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:47.075021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579233+0000","last_change":"2024-09-16T00:15:13.638306+0000","last_active":"2024-09-16T00:17:09.579233+0000","last_peered":"2024-09-16T00:17:09.579233+0000","last_clean":"2024-09-16T00:17:09.579233+0000","last_became_active":"2024-09-16T00:15:13.637931+0000","last_became_peered":"2024-09-16T00:15:13.637931+0000","last_unstale":"2024-09-16T00:17:09.579233+0000","last_undegraded":"2024-09-16T00:17:09.579233+0000","last_fullsized":"2024-09-16T00:17:09.579233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:41.385174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577585+0000","last_change":"2024-09-16T00:15:06.548209+0000","last_active":"2024-09-16T00:17:09.577585+0000","last_peered":"2024-09-16T00:17:09.577585+0000","last_clean":"2024-09-16T00:17:09.577585+0000","last_became_active":"2024-09-16T00:15:06.547846+0000","last_became_peered":"2024-09-16T00:15:06.547846+0000","last_unstale":"2024-09-16T00:17:09.577585+0000","last_undegraded":"2024-09-16T00:17:09.577585+0000","last_fullsized":"2024-09-16T00:17:09.577585+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:15.821847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780889+0000","last_change":"2024-09-16T00:16:09.023918+0000","last_active":"2024-09-16T00:17:09.780889+0000","last_peered":"2024-09-16T00:17:09.780889+0000","last_clean":"2024-09-16T00:17:09.780889+0000","last_became_active":"2024-09-16T00:16:09.023230+0000","last_became_peered":"2024-09-16T00:16:09.023230+0000","last_unstale":"2024-09-16T00:17:09.780889+0000","last_undegraded":"2024-09-16T00:17:09.780889+0000","last_fullsized":"2024-09-16T00:17:09.780889+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:31.527863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717949+0000","last_change":"2024-09-16T00:15:13.579387+0000","last_active":"2024-09-16T00:17:09.717949+0000","last_peered":"2024-09-16T00:17:09.717949+0000","last_clean":"2024-09-16T00:17:09.717949+0000","last_became_active":"2024-09-16T00:15:13.578996+0000","last_became_peered":"2024-09-16T00:15:13.578996+0000","last_unstale":"2024-09-16T00:17:09.717949+0000","last_undegraded":"2024-09-16T00:17:09.717949+0000","last_fullsized":"2024-09-16T00:17:09.717949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:51.703060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579796+0000","last_change":"2024-09-16T00:17:09.579796+0000","last_active":"2024-09-16T00:17:09.579796+0000","last_peered":"2024-09-16T00:17:09.579796+0000","last_clean":"2024-09-16T00:17:09.579796+0000","last_became_active":"2024-09-16T00:17:09.579386+0000","last_became_peered":"2024-09-16T00:17:09.579386+0000","last_unstale":"2024-09-16T00:17:09.579796+0000","last_undegraded":"2024-09-16T00:17:09.579796+0000","last_fullsized":"2024-09-16T00:17:09.579796+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:39:42.571193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":145,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:29.146717+0000","last_change":"2024-09-16T00:15:04.373896+0000","last_active":"2024-09-16T00:17:29.146717+0000","last_peered":"2024-09-16T00:17:29.146717+0000","last_clean":"2024-09-16T00:17:29.146717+0000","last_became_active":"2024-09-16T00:15:04.373493+0000","last_became_peered":"2024-09-16T00:15:04.373493+0000","last_unstale":"2024-09-16T00:17:29.146717+0000","last_undegraded":"2024-09-16T00:17:29.146717+0000","last_fullsized":"2024-09-16T00:17:29.146717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:17:10.767267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":100,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718128+0000","last_change":"2024-09-16T00:15:13.638575+0000","last_active":"2024-09-16T00:17:09.718128+0000","last_peered":"2024-09-16T00:17:09.718128+0000","last_clean":"2024-09-16T00:17:09.718128+0000","last_became_active":"2024-09-16T00:15:13.638144+0000","last_became_peered":"2024-09-16T00:15:13.638144+0000","last_unstale":"2024-09-16T00:17:09.718128+0000","last_undegraded":"2024-09-16T00:17:09.718128+0000","last_fullsized":"2024-09-16T00:17:09.718128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:39.345011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718312+0000","last_change":"2024-09-16T00:15:06.548852+0000","last_active":"2024-09-16T00:17:09.718312+0000","last_peered":"2024-09-16T00:17:09.718312+0000","last_clean":"2024-09-16T00:17:09.718312+0000","last_became_active":"2024-09-16T00:15:06.548238+0000","last_became_peered":"2024-09-16T00:15:06.548238+0000","last_unstale":"2024-09-16T00:17:09.718312+0000","last_undegraded":"2024-09-16T00:17:09.718312+0000","last_fullsized":"2024-09-16T00:17:09.718312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:16.115338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580690+0000","last_change":"2024-09-16T00:15:04.350753+0000","last_active":"2024-09-16T00:17:09.580690+0000","last_peered":"2024-09-16T00:17:09.580690+0000","last_clean":"2024-09-16T00:17:09.580690+0000","last_became_active":"2024-09-16T00:15:04.350406+0000","last_became_peered":"2024-09-16T00:15:04.350406+0000","last_unstale":"2024-09-16T00:17:09.580690+0000","last_undegraded":"2024-09-16T00:17:09.580690+0000","last_fullsized":"2024-09-16T00:17:09.580690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:09:54.737287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576996+0000","last_change":"2024-09-16T00:15:13.639302+0000","last_active":"2024-09-16T00:17:09.576996+0000","last_peered":"2024-09-16T00:17:09.576996+0000","last_clean":"2024-09-16T00:17:09.576996+0000","last_became_active":"2024-09-16T00:15:13.638863+0000","last_became_peered":"2024-09-16T00:15:13.638863+0000","last_unstale":"2024-09-16T00:17:09.576996+0000","last_undegraded":"2024-09-16T00:17:09.576996+0000","last_fullsized":"2024-09-16T00:17:09.576996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:23.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582176+0000","last_change":"2024-09-16T00:17:09.582176+0000","last_active":"2024-09-16T00:17:09.582176+0000","last_peered":"2024-09-16T00:17:09.582176+0000","last_clean":"2024-09-16T00:17:09.582176+0000","last_became_active":"2024-09-16T00:17:09.581740+0000","last_became_peered":"2024-09-16T00:17:09.581740+0000","last_unstale":"2024-09-16T00:17:09.582176+0000","last_undegraded":"2024-09-16T00:17:09.582176+0000","last_fullsized":"2024-09-16T00:17:09.582176+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:57.176724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575861+0000","last_change":"2024-09-16T00:15:04.342216+0000","last_active":"2024-09-16T00:17:09.575861+0000","last_peered":"2024-09-16T00:17:09.575861+0000","last_clean":"2024-09-16T00:17:09.575861+0000","last_became_active":"2024-09-16T00:15:04.341763+0000","last_became_peered":"2024-09-16T00:15:04.341763+0000","last_unstale":"2024-09-16T00:17:09.575861+0000","last_undegraded":"2024-09-16T00:17:09.575861+0000","last_fullsized":"2024-09-16T00:17:09.575861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:44.938527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":350,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580892+0000","last_change":"2024-09-16T00:13:51.108571+0000","last_active":"2024-09-16T00:17:09.580892+0000","last_peered":"2024-09-16T00:17:09.580892+0000","last_clean":"2024-09-16T00:17:09.580892+0000","last_became_active":"2024-09-16T00:13:51.108088+0000","last_became_peered":"2024-09-16T00:13:51.108088+0000","last_unstale":"2024-09-16T00:17:09.580892+0000","last_undegraded":"2024-09-16T00:17:09.580892+0000","last_fullsized":"2024-09-16T00:17:09.580892+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_clean_scrub_stamp":"2024-09-16T00:10:24.533807+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:02.511064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581143+0000","last_change":"2024-09-16T00:15:13.588665+0000","last_active":"2024-09-16T00:17:09.581143+0000","last_peered":"2024-09-16T00:17:09.581143+0000","last_clean":"2024-09-16T00:17:09.581143+0000","last_became_active":"2024-09-16T00:15:13.588222+0000","last_became_peered":"2024-09-16T00:15:13.588222+0000","last_unstale":"2024-09-16T00:17:09.581143+0000","last_undegraded":"2024-09-16T00:17:09.581143+0000","last_fullsized":"2024-09-16T00:17:09.581143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:59:59.965962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581454+0000","last_change":"2024-09-16T00:15:06.553114+0000","last_active":"2024-09-16T00:17:09.581454+0000","last_peered":"2024-09-16T00:17:09.581454+0000","last_clean":"2024-09-16T00:17:09.581454+0000","last_became_active":"2024-09-16T00:15:06.552788+0000","last_became_peered":"2024-09-16T00:15:06.552788+0000","last_unstale":"2024-09-16T00:17:09.581454+0000","last_undegraded":"2024-09-16T00:17:09.581454+0000","last_fullsized":"2024-09-16T00:17:09.581454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:05.473847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575364+0000","last_change":"2024-09-16T00:16:09.030747+0000","last_active":"2024-09-16T00:17:09.575364+0000","last_peered":"2024-09-16T00:17:09.575364+0000","last_clean":"2024-09-16T00:17:09.575364+0000","last_became_active":"2024-09-16T00:16:09.021208+0000","last_became_peered":"2024-09-16T00:16:09.021208+0000","last_unstale":"2024-09-16T00:17:09.575364+0000","last_undegraded":"2024-09-16T00:17:09.575364+0000","last_fullsized":"2024-09-16T00:17:09.575364+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:40.605498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580422+0000","last_change":"2024-09-16T00:15:13.575606+0000","last_active":"2024-09-16T00:17:09.580422+0000","last_peered":"2024-09-16T00:17:09.580422+0000","last_clean":"2024-09-16T00:17:09.580422+0000","last_became_active":"2024-09-16T00:15:13.575131+0000","last_became_peered":"2024-09-16T00:15:13.575131+0000","last_unstale":"2024-09-16T00:17:09.580422+0000","last_undegraded":"2024-09-16T00:17:09.580422+0000","last_fullsized":"2024-09-16T00:17:09.580422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:41.636058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.584161+0000","last_change":"2024-09-16T00:17:09.583522+0000","last_active":"2024-09-16T00:17:09.584161+0000","last_peered":"2024-09-16T00:17:09.584161+0000","last_clean":"2024-09-16T00:17:09.584161+0000","last_became_active":"2024-09-16T00:17:09.583082+0000","last_became_peered":"2024-09-16T00:17:09.583082+0000","last_unstale":"2024-09-16T00:17:09.584161+0000","last_undegraded":"2024-09-16T00:17:09.584161+0000","last_fullsized":"2024-09-16T00:17:09.584161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:48.582126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577245+0000","last_change":"2024-09-16T00:15:04.349095+0000","last_active":"2024-09-16T00:17:09.577245+0000","last_peered":"2024-09-16T00:17:09.577245+0000","last_clean":"2024-09-16T00:17:09.577245+0000","last_became_active":"2024-09-16T00:15:04.348698+0000","last_became_peered":"2024-09-16T00:15:04.348698+0000","last_unstale":"2024-09-16T00:17:09.577245+0000","last_undegraded":"2024-09-16T00:17:09.577245+0000","last_fullsized":"2024-09-16T00:17:09.577245+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:42.282882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577535+0000","last_change":"2024-09-16T00:15:13.580494+0000","last_active":"2024-09-16T00:17:09.577535+0000","last_peered":"2024-09-16T00:17:09.577535+0000","last_clean":"2024-09-16T00:17:09.577535+0000","last_became_active":"2024-09-16T00:15:13.580091+0000","last_became_peered":"2024-09-16T00:15:13.580091+0000","last_unstale":"2024-09-16T00:17:09.577535+0000","last_undegraded":"2024-09-16T00:17:09.577535+0000","last_fullsized":"2024-09-16T00:17:09.577535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:17.314919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580727+0000","last_change":"2024-09-16T00:15:06.544859+0000","last_active":"2024-09-16T00:17:09.580727+0000","last_peered":"2024-09-16T00:17:09.580727+0000","last_clean":"2024-09-16T00:17:09.580727+0000","last_became_active":"2024-09-16T00:15:06.544289+0000","last_became_peered":"2024-09-16T00:15:06.544289+0000","last_unstale":"2024-09-16T00:17:09.580727+0000","last_undegraded":"2024-09-16T00:17:09.580727+0000","last_fullsized":"2024-09-16T00:17:09.580727+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:52.622477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718205+0000","last_change":"2024-09-16T00:15:04.351313+0000","last_active":"2024-09-16T00:17:09.718205+0000","last_peered":"2024-09-16T00:17:09.718205+0000","last_clean":"2024-09-16T00:17:09.718205+0000","last_became_active":"2024-09-16T00:15:04.350928+0000","last_became_peered":"2024-09-16T00:15:04.350928+0000","last_unstale":"2024-09-16T00:17:09.718205+0000","last_undegraded":"2024-09-16T00:17:09.718205+0000","last_fullsized":"2024-09-16T00:17:09.718205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:48.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576571+0000","last_change":"2024-09-16T00:15:13.578573+0000","last_active":"2024-09-16T00:17:09.576571+0000","last_peered":"2024-09-16T00:17:09.576571+0000","last_clean":"2024-09-16T00:17:09.576571+0000","last_became_active":"2024-09-16T00:15:13.578117+0000","last_became_peered":"2024-09-16T00:15:13.578117+0000","last_unstale":"2024-09-16T00:17:09.576571+0000","last_undegraded":"2024-09-16T00:17:09.576571+0000","last_fullsized":"2024-09-16T00:17:09.576571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:53.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781258+0000","last_change":"2024-09-16T00:15:06.553548+0000","last_active":"2024-09-16T00:17:09.781258+0000","last_peered":"2024-09-16T00:17:09.781258+0000","last_clean":"2024-09-16T00:17:09.781258+0000","last_became_active":"2024-09-16T00:15:06.552963+0000","last_became_peered":"2024-09-16T00:15:06.552963+0000","last_unstale":"2024-09-16T00:17:09.781258+0000","last_undegraded":"2024-09-16T00:17:09.781258+0000","last_fullsized":"2024-09-16T00:17:09.781258+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:43.455881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579776+0000","last_change":"2024-09-16T00:15:04.351005+0000","last_active":"2024-09-16T00:17:09.579776+0000","last_peered":"2024-09-16T00:17:09.579776+0000","last_clean":"2024-09-16T00:17:09.579776+0000","last_became_active":"2024-09-16T00:15:04.350614+0000","last_became_peered":"2024-09-16T00:15:04.350614+0000","last_unstale":"2024-09-16T00:17:09.579776+0000","last_undegraded":"2024-09-16T00:17:09.579776+0000","last_fullsized":"2024-09-16T00:17:09.579776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:39.196360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579501+0000","last_change":"2024-09-16T00:16:09.023796+0000","last_active":"2024-09-16T00:17:09.579501+0000","last_peered":"2024-09-16T00:17:09.579501+0000","last_clean":"2024-09-16T00:17:09.579501+0000","last_became_active":"2024-09-16T00:16:09.023403+0000","last_became_peered":"2024-09-16T00:16:09.023403+0000","last_unstale":"2024-09-16T00:17:09.579501+0000","last_undegraded":"2024-09-16T00:17:09.579501+0000","last_fullsized":"2024-09-16T00:17:09.579501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:59.488315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577059+0000","last_change":"2024-09-16T00:15:06.546325+0000","last_active":"2024-09-16T00:17:09.577059+0000","last_peered":"2024-09-16T00:17:09.577059+0000","last_clean":"2024-09-16T00:17:09.577059+0000","last_became_active":"2024-09-16T00:15:06.546028+0000","last_became_peered":"2024-09-16T00:15:06.546028+0000","last_unstale":"2024-09-16T00:17:09.577059+0000","last_undegraded":"2024-09-16T00:17:09.577059+0000","last_fullsized":"2024-09-16T00:17:09.577059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:15.049592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579065+0000","last_change":"2024-09-16T00:15:04.373818+0000","last_active":"2024-09-16T00:17:09.579065+0000","last_peered":"2024-09-16T00:17:09.579065+0000","last_clean":"2024-09-16T00:17:09.579065+0000","last_became_active":"2024-09-16T00:15:04.373406+0000","last_became_peered":"2024-09-16T00:15:04.373406+0000","last_unstale":"2024-09-16T00:17:09.579065+0000","last_undegraded":"2024-09-16T00:17:09.579065+0000","last_fullsized":"2024-09-16T00:17:09.579065+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:08.402660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718065+0000","last_change":"2024-09-16T00:15:13.588669+0000","last_active":"2024-09-16T00:17:09.718065+0000","last_peered":"2024-09-16T00:17:09.718065+0000","last_clean":"2024-09-16T00:17:09.718065+0000","last_became_active":"2024-09-16T00:15:13.588229+0000","last_became_peered":"2024-09-16T00:15:13.588229+0000","last_unstale":"2024-09-16T00:17:09.718065+0000","last_undegraded":"2024-09-16T00:17:09.718065+0000","last_fullsized":"2024-09-16T00:17:09.718065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:30.157296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580525+0000","last_change":"2024-09-16T00:15:06.550221+0000","last_active":"2024-09-16T00:17:09.580525+0000","last_peered":"2024-09-16T00:17:09.580525+0000","last_clean":"2024-09-16T00:17:09.580525+0000","last_became_active":"2024-09-16T00:15:06.549914+0000","last_became_peered":"2024-09-16T00:15:06.549914+0000","last_unstale":"2024-09-16T00:17:09.580525+0000","last_undegraded":"2024-09-16T00:17:09.580525+0000","last_fullsized":"2024-09-16T00:17:09.580525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:18:26.824523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:29.315869+0000","last_change":"2024-09-16T00:15:04.344389+0000","last_active":"2024-09-16T00:17:29.315869+0000","last_peered":"2024-09-16T00:17:29.315869+0000","last_clean":"2024-09-16T00:17:29.315869+0000","last_became_active":"2024-09-16T00:15:04.343956+0000","last_became_peered":"2024-09-16T00:15:04.343956+0000","last_unstale":"2024-09-16T00:17:29.315869+0000","last_undegraded":"2024-09-16T00:17:29.315869+0000","last_fullsized":"2024-09-16T00:17:29.315869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:34.019861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579707+0000","last_change":"2024-09-16T00:16:09.024050+0000","last_active":"2024-09-16T00:17:09.579707+0000","last_peered":"2024-09-16T00:17:09.579707+0000","last_clean":"2024-09-16T00:17:09.579707+0000","last_became_active":"2024-09-16T00:16:09.023371+0000","last_became_peered":"2024-09-16T00:16:09.023371+0000","last_unstale":"2024-09-16T00:17:09.579707+0000","last_undegraded":"2024-09-16T00:17:09.579707+0000","last_fullsized":"2024-09-16T00:17:09.579707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:35:19.020370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581433+0000","last_change":"2024-09-16T00:15:06.542744+0000","last_active":"2024-09-16T00:17:09.581433+0000","last_peered":"2024-09-16T00:17:09.581433+0000","last_clean":"2024-09-16T00:17:09.581433+0000","last_became_active":"2024-09-16T00:15:06.541868+0000","last_became_peered":"2024-09-16T00:15:06.541868+0000","last_unstale":"2024-09-16T00:17:09.581433+0000","last_undegraded":"2024-09-16T00:17:09.581433+0000","last_fullsized":"2024-09-16T00:17:09.581433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:10:06.508682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577336+0000","last_change":"2024-09-16T00:15:04.349556+0000","last_active":"2024-09-16T00:17:09.577336+0000","last_peered":"2024-09-16T00:17:09.577336+0000","last_clean":"2024-09-16T00:17:09.577336+0000","last_became_active":"2024-09-16T00:15:04.349242+0000","last_became_peered":"2024-09-16T00:15:04.349242+0000","last_unstale":"2024-09-16T00:17:09.577336+0000","last_undegraded":"2024-09-16T00:17:09.577336+0000","last_fullsized":"2024-09-16T00:17:09.577336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:22:18.459157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781094+0000","last_change":"2024-09-16T00:15:13.584109+0000","last_active":"2024-09-16T00:17:09.781094+0000","last_peered":"2024-09-16T00:17:09.781094+0000","last_clean":"2024-09-16T00:17:09.781094+0000","last_became_active":"2024-09-16T00:15:13.583718+0000","last_became_peered":"2024-09-16T00:15:13.583718+0000","last_unstale":"2024-09-16T00:17:09.781094+0000","last_undegraded":"2024-09-16T00:17:09.781094+0000","last_fullsized":"2024-09-16T00:17:09.781094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:29.416254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717952+0000","last_change":"2024-09-16T00:15:06.548247+0000","last_active":"2024-09-16T00:17:09.717952+0000","last_peered":"2024-09-16T00:17:09.717952+0000","last_clean":"2024-09-16T00:17:09.717952+0000","last_became_active":"2024-09-16T00:15:06.547864+0000","last_became_peered":"2024-09-16T00:15:06.547864+0000","last_unstale":"2024-09-16T00:17:09.717952+0000","last_undegraded":"2024-09-16T00:17:09.717952+0000","last_fullsized":"2024-09-16T00:17:09.717952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:25.350197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575352+0000","last_change":"2024-09-16T00:15:04.350965+0000","last_active":"2024-09-16T00:17:09.575352+0000","last_peered":"2024-09-16T00:17:09.575352+0000","last_clean":"2024-09-16T00:17:09.575352+0000","last_became_active":"2024-09-16T00:15:04.349124+0000","last_became_peered":"2024-09-16T00:15:04.349124+0000","last_unstale":"2024-09-16T00:17:09.575352+0000","last_undegraded":"2024-09-16T00:17:09.575352+0000","last_fullsized":"2024-09-16T00:17:09.575352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:40.859220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576904+0000","last_change":"2024-09-16T00:15:13.581718+0000","last_active":"2024-09-16T00:17:09.576904+0000","last_peered":"2024-09-16T00:17:09.576904+0000","last_clean":"2024-09-16T00:17:09.576904+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:17:09.576904+0000","last_undegraded":"2024-09-16T00:17:09.576904+0000","last_fullsized":"2024-09-16T00:17:09.576904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:08.810606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579507+0000","last_change":"2024-09-16T00:15:06.544507+0000","last_active":"2024-09-16T00:17:09.579507+0000","last_peered":"2024-09-16T00:17:09.579507+0000","last_clean":"2024-09-16T00:17:09.579507+0000","last_became_active":"2024-09-16T00:15:06.544178+0000","last_became_peered":"2024-09-16T00:15:06.544178+0000","last_unstale":"2024-09-16T00:17:09.579507+0000","last_undegraded":"2024-09-16T00:17:09.579507+0000","last_fullsized":"2024-09-16T00:17:09.579507+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:38.494266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'37","reported_seq":301,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:28.155901+0000","last_change":"2024-09-16T00:16:09.039763+0000","last_active":"2024-09-16T00:17:28.155901+0000","last_peered":"2024-09-16T00:17:28.155901+0000","last_clean":"2024-09-16T00:17:28.155901+0000","last_became_active":"2024-09-16T00:16:09.022939+0000","last_became_peered":"2024-09-16T00:16:09.022939+0000","last_unstale":"2024-09-16T00:17:28.155901+0000","last_undegraded":"2024-09-16T00:17:28.155901+0000","last_fullsized":"2024-09-16T00:17:28.155901+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:56.431159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63488,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":204,"num_read_kb":155,"num_write":38,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60943,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576838+0000","last_change":"2024-09-16T00:15:13.585928+0000","last_active":"2024-09-16T00:17:09.576838+0000","last_peered":"2024-09-16T00:17:09.576838+0000","last_clean":"2024-09-16T00:17:09.576838+0000","last_became_active":"2024-09-16T00:15:13.582940+0000","last_became_peered":"2024-09-16T00:15:13.582940+0000","last_unstale":"2024-09-16T00:17:09.576838+0000","last_undegraded":"2024-09-16T00:17:09.576838+0000","last_fullsized":"2024-09-16T00:17:09.576838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:57:13.251978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580875+0000","last_change":"2024-09-16T00:15:06.546238+0000","last_active":"2024-09-16T00:17:09.580875+0000","last_peered":"2024-09-16T00:17:09.580875+0000","last_clean":"2024-09-16T00:17:09.580875+0000","last_became_active":"2024-09-16T00:15:06.545919+0000","last_became_peered":"2024-09-16T00:15:06.545919+0000","last_unstale":"2024-09-16T00:17:09.580875+0000","last_undegraded":"2024-09-16T00:17:09.580875+0000","last_fullsized":"2024-09-16T00:17:09.580875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:09.180832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577083+0000","last_change":"2024-09-16T00:15:04.376028+0000","last_active":"2024-09-16T00:17:09.577083+0000","last_peered":"2024-09-16T00:17:09.577083+0000","last_clean":"2024-09-16T00:17:09.577083+0000","last_became_active":"2024-09-16T00:15:04.375633+0000","last_became_peered":"2024-09-16T00:15:04.375633+0000","last_unstale":"2024-09-16T00:17:09.577083+0000","last_undegraded":"2024-09-16T00:17:09.577083+0000","last_fullsized":"2024-09-16T00:17:09.577083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:44.446342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575222+0000","last_change":"2024-09-16T00:16:09.025524+0000","last_active":"2024-09-16T00:17:09.575222+0000","last_peered":"2024-09-16T00:17:09.575222+0000","last_clean":"2024-09-16T00:17:09.575222+0000","last_became_active":"2024-09-16T00:16:09.025035+0000","last_became_peered":"2024-09-16T00:16:09.025035+0000","last_unstale":"2024-09-16T00:17:09.575222+0000","last_undegraded":"2024-09-16T00:17:09.575222+0000","last_fullsized":"2024-09-16T00:17:09.575222+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:12.367244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577569+0000","last_change":"2024-09-16T00:15:06.550864+0000","last_active":"2024-09-16T00:17:09.577569+0000","last_peered":"2024-09-16T00:17:09.577569+0000","last_clean":"2024-09-16T00:17:09.577569+0000","last_became_active":"2024-09-16T00:15:06.547956+0000","last_became_peered":"2024-09-16T00:15:06.547956+0000","last_unstale":"2024-09-16T00:17:09.577569+0000","last_undegraded":"2024-09-16T00:17:09.577569+0000","last_fullsized":"2024-09-16T00:17:09.577569+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:26.102175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580635+0000","last_change":"2024-09-16T00:15:04.350349+0000","last_active":"2024-09-16T00:17:09.580635+0000","last_peered":"2024-09-16T00:17:09.580635+0000","last_clean":"2024-09-16T00:17:09.580635+0000","last_became_active":"2024-09-16T00:15:04.349951+0000","last_became_peered":"2024-09-16T00:15:04.349951+0000","last_unstale":"2024-09-16T00:17:09.580635+0000","last_undegraded":"2024-09-16T00:17:09.580635+0000","last_fullsized":"2024-09-16T00:17:09.580635+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:46.000831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577297+0000","last_change":"2024-09-16T00:16:09.025774+0000","last_active":"2024-09-16T00:17:09.577297+0000","last_peered":"2024-09-16T00:17:09.577297+0000","last_clean":"2024-09-16T00:17:09.577297+0000","last_became_active":"2024-09-16T00:16:09.025308+0000","last_became_peered":"2024-09-16T00:16:09.025308+0000","last_unstale":"2024-09-16T00:17:09.577297+0000","last_undegraded":"2024-09-16T00:17:09.577297+0000","last_fullsized":"2024-09-16T00:17:09.577297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:16.664850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580734+0000","last_change":"2024-09-16T00:15:06.552919+0000","last_active":"2024-09-16T00:17:09.580734+0000","last_peered":"2024-09-16T00:17:09.580734+0000","last_clean":"2024-09-16T00:17:09.580734+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:17:09.580734+0000","last_undegraded":"2024-09-16T00:17:09.580734+0000","last_fullsized":"2024-09-16T00:17:09.580734+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:47:38.651837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":139,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:26.311189+0000","last_change":"2024-09-16T00:15:04.351116+0000","last_active":"2024-09-16T00:17:26.311189+0000","last_peered":"2024-09-16T00:17:26.311189+0000","last_clean":"2024-09-16T00:17:26.311189+0000","last_became_active":"2024-09-16T00:15:04.350728+0000","last_became_peered":"2024-09-16T00:15:04.350728+0000","last_unstale":"2024-09-16T00:17:26.311189+0000","last_undegraded":"2024-09-16T00:17:26.311189+0000","last_fullsized":"2024-09-16T00:17:26.311189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:38.314019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579276+0000","last_change":"2024-09-16T00:15:13.575961+0000","last_active":"2024-09-16T00:17:09.579276+0000","last_peered":"2024-09-16T00:17:09.579276+0000","last_clean":"2024-09-16T00:17:09.579276+0000","last_became_active":"2024-09-16T00:15:13.575550+0000","last_became_peered":"2024-09-16T00:15:13.575550+0000","last_unstale":"2024-09-16T00:17:09.579276+0000","last_undegraded":"2024-09-16T00:17:09.579276+0000","last_fullsized":"2024-09-16T00:17:09.579276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:05.515588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577699+0000","last_change":"2024-09-16T00:15:13.583583+0000","last_active":"2024-09-16T00:17:09.577699+0000","last_peered":"2024-09-16T00:17:09.577699+0000","last_clean":"2024-09-16T00:17:09.577699+0000","last_became_active":"2024-09-16T00:15:13.583145+0000","last_became_peered":"2024-09-16T00:15:13.583145+0000","last_unstale":"2024-09-16T00:17:09.577699+0000","last_undegraded":"2024-09-16T00:17:09.577699+0000","last_fullsized":"2024-09-16T00:17:09.577699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:07.320337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.719269+0000","last_change":"2024-09-16T00:17:09.719269+0000","last_active":"2024-09-16T00:17:09.719269+0000","last_peered":"2024-09-16T00:17:09.719269+0000","last_clean":"2024-09-16T00:17:09.719269+0000","last_became_active":"2024-09-16T00:17:09.718887+0000","last_became_peered":"2024-09-16T00:17:09.718887+0000","last_unstale":"2024-09-16T00:17:09.719269+0000","last_undegraded":"2024-09-16T00:17:09.719269+0000","last_fullsized":"2024-09-16T00:17:09.719269+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:14:51.675738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577333+0000","last_change":"2024-09-16T00:15:04.350826+0000","last_active":"2024-09-16T00:17:09.577333+0000","last_peered":"2024-09-16T00:17:09.577333+0000","last_clean":"2024-09-16T00:17:09.577333+0000","last_became_active":"2024-09-16T00:15:04.350294+0000","last_became_peered":"2024-09-16T00:15:04.350294+0000","last_unstale":"2024-09-16T00:17:09.577333+0000","last_undegraded":"2024-09-16T00:17:09.577333+0000","last_fullsized":"2024-09-16T00:17:09.577333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:49.146709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781194+0000","last_change":"2024-09-16T00:15:13.580184+0000","last_active":"2024-09-16T00:17:09.781194+0000","last_peered":"2024-09-16T00:17:09.781194+0000","last_clean":"2024-09-16T00:17:09.781194+0000","last_became_active":"2024-09-16T00:15:13.579849+0000","last_became_peered":"2024-09-16T00:15:13.579849+0000","last_unstale":"2024-09-16T00:17:09.781194+0000","last_undegraded":"2024-09-16T00:17:09.781194+0000","last_fullsized":"2024-09-16T00:17:09.781194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:40.613127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579207+0000","last_change":"2024-09-16T00:15:06.545796+0000","last_active":"2024-09-16T00:17:09.579207+0000","last_peered":"2024-09-16T00:17:09.579207+0000","last_clean":"2024-09-16T00:17:09.579207+0000","last_became_active":"2024-09-16T00:15:06.544815+0000","last_became_peered":"2024-09-16T00:15:06.544815+0000","last_unstale":"2024-09-16T00:17:09.579207+0000","last_undegraded":"2024-09-16T00:17:09.579207+0000","last_fullsized":"2024-09-16T00:17:09.579207+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:51:25.921949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718347+0000","last_change":"2024-09-16T00:15:04.340101+0000","last_active":"2024-09-16T00:17:09.718347+0000","last_peered":"2024-09-16T00:17:09.718347+0000","last_clean":"2024-09-16T00:17:09.718347+0000","last_became_active":"2024-09-16T00:15:04.339788+0000","last_became_peered":"2024-09-16T00:15:04.339788+0000","last_unstale":"2024-09-16T00:17:09.718347+0000","last_undegraded":"2024-09-16T00:17:09.718347+0000","last_fullsized":"2024-09-16T00:17:09.718347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:08.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577952+0000","last_change":"2024-09-16T00:16:09.028910+0000","last_active":"2024-09-16T00:17:09.577952+0000","last_peered":"2024-09-16T00:17:09.577952+0000","last_clean":"2024-09-16T00:17:09.577952+0000","last_became_active":"2024-09-16T00:16:09.023302+0000","last_became_peered":"2024-09-16T00:16:09.023302+0000","last_unstale":"2024-09-16T00:17:09.577952+0000","last_undegraded":"2024-09-16T00:17:09.577952+0000","last_fullsized":"2024-09-16T00:17:09.577952+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:35.293676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581116+0000","last_change":"2024-09-16T00:17:09.581116+0000","last_active":"2024-09-16T00:17:09.581116+0000","last_peered":"2024-09-16T00:17:09.581116+0000","last_clean":"2024-09-16T00:17:09.581116+0000","last_became_active":"2024-09-16T00:17:09.580552+0000","last_became_peered":"2024-09-16T00:17:09.580552+0000","last_unstale":"2024-09-16T00:17:09.581116+0000","last_undegraded":"2024-09-16T00:17:09.581116+0000","last_fullsized":"2024-09-16T00:17:09.581116+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:35.975542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574745+0000","last_change":"2024-09-16T00:15:13.581966+0000","last_active":"2024-09-16T00:17:09.574745+0000","last_peered":"2024-09-16T00:17:09.574745+0000","last_clean":"2024-09-16T00:17:09.574745+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:17:09.574745+0000","last_undegraded":"2024-09-16T00:17:09.574745+0000","last_fullsized":"2024-09-16T00:17:09.574745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.132773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":145,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:28.314493+0000","last_change":"2024-09-16T00:15:04.351765+0000","last_active":"2024-09-16T00:17:28.314493+0000","last_peered":"2024-09-16T00:17:28.314493+0000","last_clean":"2024-09-16T00:17:28.314493+0000","last_became_active":"2024-09-16T00:15:04.351398+0000","last_became_peered":"2024-09-16T00:15:04.351398+0000","last_unstale":"2024-09-16T00:17:28.314493+0000","last_undegraded":"2024-09-16T00:17:28.314493+0000","last_fullsized":"2024-09-16T00:17:28.314493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:14:28.762657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.583255+0000","last_change":"2024-09-16T00:17:09.583255+0000","last_active":"2024-09-16T00:17:09.583255+0000","last_peered":"2024-09-16T00:17:09.583255+0000","last_clean":"2024-09-16T00:17:09.583255+0000","last_became_active":"2024-09-16T00:17:09.582793+0000","last_became_peered":"2024-09-16T00:17:09.582793+0000","last_unstale":"2024-09-16T00:17:09.583255+0000","last_undegraded":"2024-09-16T00:17:09.583255+0000","last_fullsized":"2024-09-16T00:17:09.583255+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:28.087903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574875+0000","last_change":"2024-09-16T00:15:13.576877+0000","last_active":"2024-09-16T00:17:09.574875+0000","last_peered":"2024-09-16T00:17:09.574875+0000","last_clean":"2024-09-16T00:17:09.574875+0000","last_became_active":"2024-09-16T00:15:13.576480+0000","last_became_peered":"2024-09-16T00:15:13.576480+0000","last_unstale":"2024-09-16T00:17:09.574875+0000","last_undegraded":"2024-09-16T00:17:09.574875+0000","last_fullsized":"2024-09-16T00:17:09.574875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:47.322219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780925+0000","last_change":"2024-09-16T00:15:06.550329+0000","last_active":"2024-09-16T00:17:09.780925+0000","last_peered":"2024-09-16T00:17:09.780925+0000","last_clean":"2024-09-16T00:17:09.780925+0000","last_became_active":"2024-09-16T00:15:06.550008+0000","last_became_peered":"2024-09-16T00:15:06.550008+0000","last_unstale":"2024-09-16T00:17:09.780925+0000","last_undegraded":"2024-09-16T00:17:09.780925+0000","last_fullsized":"2024-09-16T00:17:09.780925+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:38.096126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582169+0000","last_change":"2024-09-16T00:17:09.580839+0000","last_active":"2024-09-16T00:17:09.582169+0000","last_peered":"2024-09-16T00:17:09.582169+0000","last_clean":"2024-09-16T00:17:09.582169+0000","last_became_active":"2024-09-16T00:17:09.580381+0000","last_became_peered":"2024-09-16T00:17:09.580381+0000","last_unstale":"2024-09-16T00:17:09.582169+0000","last_undegraded":"2024-09-16T00:17:09.582169+0000","last_fullsized":"2024-09-16T00:17:09.582169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:35.425316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574792+0000","last_change":"2024-09-16T00:15:06.546713+0000","last_active":"2024-09-16T00:17:09.574792+0000","last_peered":"2024-09-16T00:17:09.574792+0000","last_clean":"2024-09-16T00:17:09.574792+0000","last_became_active":"2024-09-16T00:15:06.546209+0000","last_became_peered":"2024-09-16T00:15:06.546209+0000","last_unstale":"2024-09-16T00:17:09.574792+0000","last_undegraded":"2024-09-16T00:17:09.574792+0000","last_fullsized":"2024-09-16T00:17:09.574792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:43.913748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577575+0000","last_change":"2024-09-16T00:15:06.550490+0000","last_active":"2024-09-16T00:17:09.577575+0000","last_peered":"2024-09-16T00:17:09.577575+0000","last_clean":"2024-09-16T00:17:09.577575+0000","last_became_active":"2024-09-16T00:15:06.549311+0000","last_became_peered":"2024-09-16T00:15:06.549311+0000","last_unstale":"2024-09-16T00:17:09.577575+0000","last_undegraded":"2024-09-16T00:17:09.577575+0000","last_fullsized":"2024-09-16T00:17:09.577575+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:10:01.995357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718148+0000","last_change":"2024-09-16T00:15:06.548115+0000","last_active":"2024-09-16T00:17:09.718148+0000","last_peered":"2024-09-16T00:17:09.718148+0000","last_clean":"2024-09-16T00:17:09.718148+0000","last_became_active":"2024-09-16T00:15:06.547782+0000","last_became_peered":"2024-09-16T00:15:06.547782+0000","last_unstale":"2024-09-16T00:17:09.718148+0000","last_undegraded":"2024-09-16T00:17:09.718148+0000","last_fullsized":"2024-09-16T00:17:09.718148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:53.228222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781260+0000","last_change":"2024-09-16T00:15:06.554396+0000","last_active":"2024-09-16T00:17:09.781260+0000","last_peered":"2024-09-16T00:17:09.781260+0000","last_clean":"2024-09-16T00:17:09.781260+0000","last_became_active":"2024-09-16T00:15:06.554012+0000","last_became_peered":"2024-09-16T00:15:06.554012+0000","last_unstale":"2024-09-16T00:17:09.781260+0000","last_undegraded":"2024-09-16T00:17:09.781260+0000","last_fullsized":"2024-09-16T00:17:09.781260+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:31.464098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575494+0000","last_change":"2024-09-16T00:15:06.549969+0000","last_active":"2024-09-16T00:17:09.575494+0000","last_peered":"2024-09-16T00:17:09.575494+0000","last_clean":"2024-09-16T00:17:09.575494+0000","last_became_active":"2024-09-16T00:15:06.549531+0000","last_became_peered":"2024-09-16T00:15:06.549531+0000","last_unstale":"2024-09-16T00:17:09.575494+0000","last_undegraded":"2024-09-16T00:17:09.575494+0000","last_fullsized":"2024-09-16T00:17:09.575494+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:37.883612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579397+0000","last_change":"2024-09-16T00:15:06.547734+0000","last_active":"2024-09-16T00:17:09.579397+0000","last_peered":"2024-09-16T00:17:09.579397+0000","last_clean":"2024-09-16T00:17:09.579397+0000","last_became_active":"2024-09-16T00:15:06.547400+0000","last_became_peered":"2024-09-16T00:15:06.547400+0000","last_unstale":"2024-09-16T00:17:09.579397+0000","last_undegraded":"2024-09-16T00:17:09.579397+0000","last_fullsized":"2024-09-16T00:17:09.579397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:54.455313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576721+0000","last_change":"2024-09-16T00:15:06.551965+0000","last_active":"2024-09-16T00:17:09.576721+0000","last_peered":"2024-09-16T00:17:09.576721+0000","last_clean":"2024-09-16T00:17:09.576721+0000","last_became_active":"2024-09-16T00:15:06.551602+0000","last_became_peered":"2024-09-16T00:15:06.551602+0000","last_unstale":"2024-09-16T00:17:09.576721+0000","last_undegraded":"2024-09-16T00:17:09.576721+0000","last_fullsized":"2024-09-16T00:17:09.576721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:34.951960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579571+0000","last_change":"2024-09-16T00:15:06.549340+0000","last_active":"2024-09-16T00:17:09.579571+0000","last_peered":"2024-09-16T00:17:09.579571+0000","last_clean":"2024-09-16T00:17:09.579571+0000","last_became_active":"2024-09-16T00:15:06.547424+0000","last_became_peered":"2024-09-16T00:15:06.547424+0000","last_unstale":"2024-09-16T00:17:09.579571+0000","last_undegraded":"2024-09-16T00:17:09.579571+0000","last_fullsized":"2024-09-16T00:17:09.579571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:12.087535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580115+0000","last_change":"2024-09-16T00:15:06.555216+0000","last_active":"2024-09-16T00:17:09.580115+0000","last_peered":"2024-09-16T00:17:09.580115+0000","last_clean":"2024-09-16T00:17:09.580115+0000","last_became_active":"2024-09-16T00:15:06.554896+0000","last_became_peered":"2024-09-16T00:15:06.554896+0000","last_unstale":"2024-09-16T00:17:09.580115+0000","last_undegraded":"2024-09-16T00:17:09.580115+0000","last_fullsized":"2024-09-16T00:17:09.580115+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:20.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717980+0000","last_change":"2024-09-16T00:15:06.549752+0000","last_active":"2024-09-16T00:17:09.717980+0000","last_peered":"2024-09-16T00:17:09.717980+0000","last_clean":"2024-09-16T00:17:09.717980+0000","last_became_active":"2024-09-16T00:15:06.549350+0000","last_became_peered":"2024-09-16T00:15:06.549350+0000","last_unstale":"2024-09-16T00:17:09.717980+0000","last_undegraded":"2024-09-16T00:17:09.717980+0000","last_fullsized":"2024-09-16T00:17:09.717980+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:48.615123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781099+0000","last_change":"2024-09-16T00:15:06.549159+0000","last_active":"2024-09-16T00:17:09.781099+0000","last_peered":"2024-09-16T00:17:09.781099+0000","last_clean":"2024-09-16T00:17:09.781099+0000","last_became_active":"2024-09-16T00:15:06.548408+0000","last_became_peered":"2024-09-16T00:15:06.548408+0000","last_unstale":"2024-09-16T00:17:09.781099+0000","last_undegraded":"2024-09-16T00:17:09.781099+0000","last_fullsized":"2024-09-16T00:17:09.781099+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:23.941809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575144+0000","last_change":"2024-09-16T00:15:06.544091+0000","last_active":"2024-09-16T00:17:09.575144+0000","last_peered":"2024-09-16T00:17:09.575144+0000","last_clean":"2024-09-16T00:17:09.575144+0000","last_became_active":"2024-09-16T00:15:06.543761+0000","last_became_peered":"2024-09-16T00:15:06.543761+0000","last_unstale":"2024-09-16T00:17:09.575144+0000","last_undegraded":"2024-09-16T00:17:09.575144+0000","last_fullsized":"2024-09-16T00:17:09.575144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:36:43.253405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582256+0000","last_change":"2024-09-16T00:17:09.581748+0000","last_active":"2024-09-16T00:17:09.582256+0000","last_peered":"2024-09-16T00:17:09.582256+0000","last_clean":"2024-09-16T00:17:09.582256+0000","last_became_active":"2024-09-16T00:17:09.581207+0000","last_became_peered":"2024-09-16T00:17:09.581207+0000","last_unstale":"2024-09-16T00:17:09.582256+0000","last_undegraded":"2024-09-16T00:17:09.582256+0000","last_fullsized":"2024-09-16T00:17:09.582256+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:21.521929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781344+0000","last_change":"2024-09-16T00:15:06.557162+0000","last_active":"2024-09-16T00:17:09.781344+0000","last_peered":"2024-09-16T00:17:09.781344+0000","last_clean":"2024-09-16T00:17:09.781344+0000","last_became_active":"2024-09-16T00:15:06.556831+0000","last_became_peered":"2024-09-16T00:15:06.556831+0000","last_unstale":"2024-09-16T00:17:09.781344+0000","last_undegraded":"2024-09-16T00:17:09.781344+0000","last_fullsized":"2024-09-16T00:17:09.781344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.236915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579887+0000","last_change":"2024-09-16T00:15:06.556161+0000","last_active":"2024-09-16T00:17:09.579887+0000","last_peered":"2024-09-16T00:17:09.579887+0000","last_clean":"2024-09-16T00:17:09.579887+0000","last_became_active":"2024-09-16T00:15:06.555819+0000","last_became_peered":"2024-09-16T00:15:06.555819+0000","last_unstale":"2024-09-16T00:17:09.579887+0000","last_undegraded":"2024-09-16T00:17:09.579887+0000","last_fullsized":"2024-09-16T00:17:09.579887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:29.929977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.583819+0000","last_change":"2024-09-16T00:17:09.583183+0000","last_active":"2024-09-16T00:17:09.583819+0000","last_peered":"2024-09-16T00:17:09.583819+0000","last_clean":"2024-09-16T00:17:09.583819+0000","last_became_active":"2024-09-16T00:17:09.582663+0000","last_became_peered":"2024-09-16T00:17:09.582663+0000","last_unstale":"2024-09-16T00:17:09.583819+0000","last_undegraded":"2024-09-16T00:17:09.583819+0000","last_fullsized":"2024-09-16T00:17:09.583819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:37.544124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577469+0000","last_change":"2024-09-16T00:15:06.547114+0000","last_active":"2024-09-16T00:17:09.577469+0000","last_peered":"2024-09-16T00:17:09.577469+0000","last_clean":"2024-09-16T00:17:09.577469+0000","last_became_active":"2024-09-16T00:15:06.546786+0000","last_became_peered":"2024-09-16T00:15:06.546786+0000","last_unstale":"2024-09-16T00:17:09.577469+0000","last_undegraded":"2024-09-16T00:17:09.577469+0000","last_fullsized":"2024-09-16T00:17:09.577469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:13:26.925603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717890+0000","last_change":"2024-09-16T00:15:06.541168+0000","last_active":"2024-09-16T00:17:09.717890+0000","last_peered":"2024-09-16T00:17:09.717890+0000","last_clean":"2024-09-16T00:17:09.717890+0000","last_became_active":"2024-09-16T00:15:06.540848+0000","last_became_peered":"2024-09-16T00:15:06.540848+0000","last_unstale":"2024-09-16T00:17:09.717890+0000","last_undegraded":"2024-09-16T00:17:09.717890+0000","last_fullsized":"2024-09-16T00:17:09.717890+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.958600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575787+0000","last_change":"2024-09-16T00:15:06.547429+0000","last_active":"2024-09-16T00:17:09.575787+0000","last_peered":"2024-09-16T00:17:09.575787+0000","last_clean":"2024-09-16T00:17:09.575787+0000","last_became_active":"2024-09-16T00:15:06.547081+0000","last_became_peered":"2024-09-16T00:15:06.547081+0000","last_unstale":"2024-09-16T00:17:09.575787+0000","last_undegraded":"2024-09-16T00:17:09.575787+0000","last_fullsized":"2024-09-16T00:17:09.575787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:43.396313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581283+0000","last_change":"2024-09-16T00:15:06.551811+0000","last_active":"2024-09-16T00:17:09.581283+0000","last_peered":"2024-09-16T00:17:09.581283+0000","last_clean":"2024-09-16T00:17:09.581283+0000","last_became_active":"2024-09-16T00:15:06.551424+0000","last_became_peered":"2024-09-16T00:15:06.551424+0000","last_unstale":"2024-09-16T00:17:09.581283+0000","last_undegraded":"2024-09-16T00:17:09.581283+0000","last_fullsized":"2024-09-16T00:17:09.581283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.943070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578024+0000","last_change":"2024-09-16T00:15:06.544697+0000","last_active":"2024-09-16T00:17:09.578024+0000","last_peered":"2024-09-16T00:17:09.578024+0000","last_clean":"2024-09-16T00:17:09.578024+0000","last_became_active":"2024-09-16T00:15:06.544182+0000","last_became_peered":"2024-09-16T00:15:06.544182+0000","last_unstale":"2024-09-16T00:17:09.578024+0000","last_undegraded":"2024-09-16T00:17:09.578024+0000","last_fullsized":"2024-09-16T00:17:09.578024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:45.929092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580265+0000","last_change":"2024-09-16T00:15:06.551488+0000","last_active":"2024-09-16T00:17:09.580265+0000","last_peered":"2024-09-16T00:17:09.580265+0000","last_clean":"2024-09-16T00:17:09.580265+0000","last_became_active":"2024-09-16T00:15:06.551171+0000","last_became_peered":"2024-09-16T00:15:06.551171+0000","last_unstale":"2024-09-16T00:17:09.580265+0000","last_undegraded":"2024-09-16T00:17:09.580265+0000","last_fullsized":"2024-09-16T00:17:09.580265+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:23.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579101+0000","last_change":"2024-09-16T00:15:06.544043+0000","last_active":"2024-09-16T00:17:09.579101+0000","last_peered":"2024-09-16T00:17:09.579101+0000","last_clean":"2024-09-16T00:17:09.579101+0000","last_became_active":"2024-09-16T00:15:06.543014+0000","last_became_peered":"2024-09-16T00:15:06.543014+0000","last_unstale":"2024-09-16T00:17:09.579101+0000","last_undegraded":"2024-09-16T00:17:09.579101+0000","last_fullsized":"2024-09-16T00:17:09.579101+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:23.904810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581718+0000","last_change":"2024-09-16T00:15:06.546788+0000","last_active":"2024-09-16T00:17:09.581718+0000","last_peered":"2024-09-16T00:17:09.581718+0000","last_clean":"2024-09-16T00:17:09.581718+0000","last_became_active":"2024-09-16T00:15:06.546333+0000","last_became_peered":"2024-09-16T00:15:06.546333+0000","last_unstale":"2024-09-16T00:17:09.581718+0000","last_undegraded":"2024-09-16T00:17:09.581718+0000","last_fullsized":"2024-09-16T00:17:09.581718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:09.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.719618+0000","last_change":"2024-09-16T00:17:09.719618+0000","last_active":"2024-09-16T00:17:09.719618+0000","last_peered":"2024-09-16T00:17:09.719618+0000","last_clean":"2024-09-16T00:17:09.719618+0000","last_became_active":"2024-09-16T00:17:09.719172+0000","last_became_peered":"2024-09-16T00:17:09.719172+0000","last_unstale":"2024-09-16T00:17:09.719618+0000","last_undegraded":"2024-09-16T00:17:09.719618+0000","last_fullsized":"2024-09-16T00:17:09.719618+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:51.709719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579285+0000","last_change":"2024-09-16T00:15:06.540675+0000","last_active":"2024-09-16T00:17:09.579285+0000","last_peered":"2024-09-16T00:17:09.579285+0000","last_clean":"2024-09-16T00:17:09.579285+0000","last_became_active":"2024-09-16T00:15:06.540333+0000","last_became_peered":"2024-09-16T00:15:06.540333+0000","last_unstale":"2024-09-16T00:17:09.579285+0000","last_undegraded":"2024-09-16T00:17:09.579285+0000","last_fullsized":"2024-09-16T00:17:09.579285+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:17.166987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575821+0000","last_change":"2024-09-16T00:15:06.548758+0000","last_active":"2024-09-16T00:17:09.575821+0000","last_peered":"2024-09-16T00:17:09.575821+0000","last_clean":"2024-09-16T00:17:09.575821+0000","last_became_active":"2024-09-16T00:15:06.548397+0000","last_became_peered":"2024-09-16T00:15:06.548397+0000","last_unstale":"2024-09-16T00:17:09.575821+0000","last_undegraded":"2024-09-16T00:17:09.575821+0000","last_fullsized":"2024-09-16T00:17:09.575821+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:50.320170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576644+0000","last_change":"2024-09-16T00:15:06.554898+0000","last_active":"2024-09-16T00:17:09.576644+0000","last_peered":"2024-09-16T00:17:09.576644+0000","last_clean":"2024-09-16T00:17:09.576644+0000","last_became_active":"2024-09-16T00:15:06.553005+0000","last_became_peered":"2024-09-16T00:15:06.553005+0000","last_unstale":"2024-09-16T00:17:09.576644+0000","last_undegraded":"2024-09-16T00:17:09.576644+0000","last_fullsized":"2024-09-16T00:17:09.576644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:29.630911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577231+0000","last_change":"2024-09-16T00:15:06.547265+0000","last_active":"2024-09-16T00:17:09.577231+0000","last_peered":"2024-09-16T00:17:09.577231+0000","last_clean":"2024-09-16T00:17:09.577231+0000","last_became_active":"2024-09-16T00:15:06.546938+0000","last_became_peered":"2024-09-16T00:15:06.546938+0000","last_unstale":"2024-09-16T00:17:09.577231+0000","last_undegraded":"2024-09-16T00:17:09.577231+0000","last_fullsized":"2024-09-16T00:17:09.577231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:40.247343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781308+0000","last_change":"2024-09-16T00:15:06.556735+0000","last_active":"2024-09-16T00:17:09.781308+0000","last_peered":"2024-09-16T00:17:09.781308+0000","last_clean":"2024-09-16T00:17:09.781308+0000","last_became_active":"2024-09-16T00:15:06.556324+0000","last_became_peered":"2024-09-16T00:15:06.556324+0000","last_unstale":"2024-09-16T00:17:09.781308+0000","last_undegraded":"2024-09-16T00:17:09.781308+0000","last_fullsized":"2024-09-16T00:17:09.781308+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:25.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68025,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":622,"num_read_kb":373,"num_write":178,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":62107,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":209055,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249414,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579947,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":517057,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808879,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36464,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708688,"statfs":{"total":95995035648,"available":95957696512,"internally_reserved":0,"allocated":1142784,"data_stored":1168039,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972405,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1015808,"data_stored":521389,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201337,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1007616,"data_stored":516631,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430271,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1060864,"data_stored":1113265,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626501,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":512469,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855434,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1069056,"data_stored":1108319,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986255,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1040384,"data_stored":587882,"data_compressed":39297,"data_compressed_allocated":53248,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215187,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":512663,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444121,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1007616,"data_stored":513147,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607646,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1064960,"data_stored":576449,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1015808,"data_stored":513857,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":71407,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T00:17:33.565 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:17:33.857 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:34.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:33 smithi033 ceph-mon[31243]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:34.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:33 smithi033 ceph-mon[31243]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:34.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:33 smithi033 ceph-mon[31243]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:34.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:33 smithi083 ceph-mon[35242]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:33 smithi083 ceph-mon[35242]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:33 smithi083 ceph-mon[35242]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:34.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:33 smithi060 ceph-mon[35185]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:34.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:33 smithi060 ceph-mon[35185]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:34.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:33 smithi060 ceph-mon[35185]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:34.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.338+0000 7fd012a68640 1 -- 172.21.15.33:0/4084660559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c101de0 msgr2=0x7fd00c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:34.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.338+0000 7fd012a68640 1 --2- 172.21.15.33:0/4084660559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c101de0 0x7fd00c1021c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009900 tx=0x7fcffc02a410 comp rx=0 tx=0).stop 2024-09-16T00:17:34.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.338+0000 7fd012a68640 1 -- 172.21.15.33:0/4084660559 shutdown_connections 2024-09-16T00:17:34.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.338+0000 7fd012a68640 1 --2- 172.21.15.33:0/4084660559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd00c106f10 0x7fd00c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:34.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.338+0000 7fd012a68640 1 --2- 172.21.15.33:0/4084660559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c102790 0x7fd00c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:34.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.338+0000 7fd012a68640 1 --2- 172.21.15.33:0/4084660559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c101de0 0x7fd00c1021c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:34.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.338+0000 7fd012a68640 1 -- 172.21.15.33:0/4084660559 >> 172.21.15.33:0/4084660559 conn(0x7fd00c0fd8c0 msgr2=0x7fd00c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:34.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.338+0000 7fd012a68640 1 -- 172.21.15.33:0/4084660559 shutdown_connections 2024-09-16T00:17:34.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.338+0000 7fd012a68640 1 -- 172.21.15.33:0/4084660559 wait complete. 2024-09-16T00:17:34.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd012a68640 1 Processor -- start 2024-09-16T00:17:34.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd012a68640 1 -- start start 2024-09-16T00:17:34.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd012a68640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd00c101de0 0x7fd00c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:34.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd012a68640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c102790 0x7fd00c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:34.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd012a68640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c106f10 0x7fd00c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:34.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd012a68640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c111f70 con 0x7fd00c102790 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd012a68640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c111df0 con 0x7fd00c101de0 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd012a68640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c1120f0 con 0x7fd00c106f10 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd00bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c102790 0x7fd00c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd00bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c102790 0x7fd00c074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36296/0 (socket says 172.21.15.33:36296) 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd00bfff640 1 -- 172.21.15.33:0/3689630839 learned_addr learned my addr 172.21.15.33:0/3689630839 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd010804640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd00c101de0 0x7fd00c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd011005640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c106f10 0x7fd00c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd00bfff640 1 -- 172.21.15.33:0/3689630839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c106f10 msgr2=0x7fd00c074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd00bfff640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c106f10 0x7fd00c074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd00bfff640 1 -- 172.21.15.33:0/3689630839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd00c101de0 msgr2=0x7fd00c0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd00bfff640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd00c101de0 0x7fd00c0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd00bfff640 1 -- 172.21.15.33:0/3689630839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd00c1a6970 con 0x7fd00c102790 2024-09-16T00:17:34.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd011005640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c106f10 0x7fd00c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:34.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.339+0000 7fd010804640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd00c101de0 0x7fd00c0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:34.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.340+0000 7fd00bfff640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c102790 0x7fd00c074a30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd00000e9f0 tx=0x7fd00000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:34.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.340+0000 7fd009ffb640 1 -- 172.21.15.33:0/3689630839 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00000cd80 con 0x7fd00c102790 2024-09-16T00:17:34.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.340+0000 7fd009ffb640 1 -- 172.21.15.33:0/3689630839 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd000005500 con 0x7fd00c102790 2024-09-16T00:17:34.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.340+0000 7fd012a68640 1 -- 172.21.15.33:0/3689630839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd00c1a6c60 con 0x7fd00c102790 2024-09-16T00:17:34.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.340+0000 7fd009ffb640 1 -- 172.21.15.33:0/3689630839 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000010690 con 0x7fd00c102790 2024-09-16T00:17:34.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.340+0000 7fd012a68640 1 -- 172.21.15.33:0/3689630839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd00c1a7150 con 0x7fd00c102790 2024-09-16T00:17:34.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.341+0000 7fd012a68640 1 -- 172.21.15.33:0/3689630839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd00c1021c0 con 0x7fd00c102790 2024-09-16T00:17:34.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.342+0000 7fd009ffb640 1 -- 172.21.15.33:0/3689630839 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fd0000046c0 con 0x7fd00c102790 2024-09-16T00:17:34.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.343+0000 7fd009ffb640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcff40a32d0 0x7fcff40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:34.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.343+0000 7fd010804640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcff40a32d0 0x7fcff40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:34.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.343+0000 7fd009ffb640 1 -- 172.21.15.33:0/3689630839 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd0000d04c0 con 0x7fd00c102790 2024-09-16T00:17:34.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.343+0000 7fd010804640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcff40a32d0 0x7fcff40a5790 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcffc005ab0 tx=0x7fcffc005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:34.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.347+0000 7fd009ffb640 1 -- 172.21.15.33:0/3689630839 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd00009cba0 con 0x7fd00c102790 2024-09-16T00:17:34.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.556+0000 7fd012a68640 1 -- 172.21.15.33:0/3689630839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd00c02d7c0 con 0x7fd00c102790 2024-09-16T00:17:34.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.557+0000 7fd009ffb640 1 -- 172.21.15.33:0/3689630839 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11903 (secure 0 0 0) 0x7fd00009c4a0 con 0x7fd00c102790 2024-09-16T00:17:34.558 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:34.558 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":26,"btime":"2024-09-16T00:17:28:789138+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:17:28.493434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:17:34.558 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 26 2024-09-16T00:17:34.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.560+0000 7fd012a68640 1 -- 172.21.15.33:0/3689630839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcff40a32d0 msgr2=0x7fcff40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:34.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.560+0000 7fd012a68640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcff40a32d0 0x7fcff40a5790 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcffc005ab0 tx=0x7fcffc005a20 comp rx=0 tx=0).stop 2024-09-16T00:17:34.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.560+0000 7fd012a68640 1 -- 172.21.15.33:0/3689630839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c102790 msgr2=0x7fd00c074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:34.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.560+0000 7fd012a68640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c102790 0x7fd00c074a30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd00000e9f0 tx=0x7fd00000edf0 comp rx=0 tx=0).stop 2024-09-16T00:17:34.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.560+0000 7fd012a68640 1 -- 172.21.15.33:0/3689630839 shutdown_connections 2024-09-16T00:17:34.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.560+0000 7fd012a68640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcff40a32d0 0x7fcff40a5790 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:34.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.560+0000 7fd012a68640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c106f10 0x7fd00c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:34.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.560+0000 7fd012a68640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c102790 0x7fd00c074a30 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:34.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.560+0000 7fd012a68640 1 --2- 172.21.15.33:0/3689630839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd00c101de0 0x7fd00c0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:34.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.560+0000 7fd012a68640 1 -- 172.21.15.33:0/3689630839 >> 172.21.15.33:0/3689630839 conn(0x7fd00c0fd8c0 msgr2=0x7fd00c0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:34.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.560+0000 7fd012a68640 1 -- 172.21.15.33:0/3689630839 shutdown_connections 2024-09-16T00:17:34.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:34.560+0000 7fd012a68640 1 -- 172.21.15.33:0/3689630839 wait complete. 2024-09-16T00:17:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:34 smithi033 ceph-mon[31243]: pgmap v363: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-16T00:17:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:34 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3689630839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:35.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:34 smithi083 ceph-mon[35242]: pgmap v363: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-16T00:17:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:34 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3689630839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:35.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:34 smithi060 ceph-mon[35185]: pgmap v363: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-16T00:17:35.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:34 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3689630839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:35.296 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:17:35.560 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:36.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.053+0000 7f1229748640 1 -- 172.21.15.33:0/2109220575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1224100e30 msgr2=0x7f1224104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:36.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.053+0000 7f1229748640 1 --2- 172.21.15.33:0/2109220575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1224100e30 0x7f1224104e80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f120c009900 tx=0x7f120c02a410 comp rx=0 tx=0).stop 2024-09-16T00:17:36.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.053+0000 7f1229748640 1 -- 172.21.15.33:0/2109220575 shutdown_connections 2024-09-16T00:17:36.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.053+0000 7f1229748640 1 --2- 172.21.15.33:0/2109220575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12241055b0 0x7f122410d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:36.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.053+0000 7f1229748640 1 --2- 172.21.15.33:0/2109220575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1224100e30 0x7f1224104e80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:36.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.053+0000 7f1229748640 1 --2- 172.21.15.33:0/2109220575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1224100480 0x7f1224100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:36.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.053+0000 7f1229748640 1 -- 172.21.15.33:0/2109220575 >> 172.21.15.33:0/2109220575 conn(0x7f1224077210 msgr2=0x7f12240fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:36.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.053+0000 7f1229748640 1 -- 172.21.15.33:0/2109220575 shutdown_connections 2024-09-16T00:17:36.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.053+0000 7f1229748640 1 -- 172.21.15.33:0/2109220575 wait complete. 2024-09-16T00:17:36.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.054+0000 7f1229748640 1 Processor -- start 2024-09-16T00:17:36.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.054+0000 7f1229748640 1 -- start start 2024-09-16T00:17:36.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.054+0000 7f1229748640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1224100480 0x7f1224198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:36.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.054+0000 7f1229748640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1224100e30 0x7f1224198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:36.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.054+0000 7f1229748640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12241055b0 0x7f122419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:36.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.054+0000 7f1229748640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1224111f90 con 0x7f12241055b0 2024-09-16T00:17:36.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.054+0000 7f1229748640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1224111e10 con 0x7f1224100e30 2024-09-16T00:17:36.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.054+0000 7f1229748640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1224112110 con 0x7f1224100480 2024-09-16T00:17:36.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.054+0000 7f12237fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12241055b0 0x7f122419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:36.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.054+0000 7f12237fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12241055b0 0x7f122419cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36314/0 (socket says 172.21.15.33:36314) 2024-09-16T00:17:36.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.054+0000 7f12237fe640 1 -- 172.21.15.33:0/1807343077 learned_addr learned my addr 172.21.15.33:0/1807343077 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:36.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.054+0000 7f121bfff640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1224100e30 0x7f1224198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:36.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f1222ffd640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1224100480 0x7f1224198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:36.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f12237fe640 1 -- 172.21.15.33:0/1807343077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1224100480 msgr2=0x7f1224198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:36.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f12237fe640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1224100480 0x7f1224198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:36.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f12237fe640 1 -- 172.21.15.33:0/1807343077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1224100e30 msgr2=0x7f1224198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:36.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f12237fe640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1224100e30 0x7f1224198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:36.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f12237fe640 1 -- 172.21.15.33:0/1807343077 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122419d400 con 0x7f12241055b0 2024-09-16T00:17:36.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f1222ffd640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1224100480 0x7f1224198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:36.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f12237fe640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12241055b0 0x7f122419cd20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f121c00eaa0 tx=0x7f121c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:36.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f1220ff9640 1 -- 172.21.15.33:0/1807343077 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c00cdb0 con 0x7f12241055b0 2024-09-16T00:17:36.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f1229748640 1 -- 172.21.15.33:0/1807343077 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f122419d6f0 con 0x7f12241055b0 2024-09-16T00:17:36.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f1220ff9640 1 -- 172.21.15.33:0/1807343077 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f121c005020 con 0x7f12241055b0 2024-09-16T00:17:36.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f1220ff9640 1 -- 172.21.15.33:0/1807343077 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c0116b0 con 0x7f12241055b0 2024-09-16T00:17:36.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.055+0000 7f1229748640 1 -- 172.21.15.33:0/1807343077 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12241a4fd0 con 0x7f12241055b0 2024-09-16T00:17:36.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.056+0000 7f1229748640 1 -- 172.21.15.33:0/1807343077 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1224100860 con 0x7f12241055b0 2024-09-16T00:17:36.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.057+0000 7f1220ff9640 1 -- 172.21.15.33:0/1807343077 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f121c005560 con 0x7f12241055b0 2024-09-16T00:17:36.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.058+0000 7f1220ff9640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12100a32d0 0x7f12100a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:36.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.058+0000 7f1222ffd640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12100a32d0 0x7f12100a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:36.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.058+0000 7f1220ff9640 1 -- 172.21.15.33:0/1807343077 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f121c0d0190 con 0x7f12241055b0 2024-09-16T00:17:36.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.059+0000 7f1222ffd640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12100a32d0 0x7f12100a5790 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f12140058c0 tx=0x7f1214005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:36.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.062+0000 7f1220ff9640 1 -- 172.21.15.33:0/1807343077 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f121c016030 con 0x7f12241055b0 2024-09-16T00:17:36.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.238+0000 7f1229748640 1 -- 172.21.15.33:0/1807343077 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f122402d7c0 con 0x7f12241055b0 2024-09-16T00:17:36.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.240+0000 7f1220ff9640 1 -- 172.21.15.33:0/1807343077 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25076 (secure 0 0 0) 0x7f121c09c870 con 0x7f12241055b0 2024-09-16T00:17:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:36.241 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":90,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:17:09.569270+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2e","mappings":[{"from":1,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:17:36.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.243+0000 7f1229748640 1 -- 172.21.15.33:0/1807343077 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12100a32d0 msgr2=0x7f12100a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:36.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.243+0000 7f1229748640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12100a32d0 0x7f12100a5790 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f12140058c0 tx=0x7f1214005850 comp rx=0 tx=0).stop 2024-09-16T00:17:36.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.243+0000 7f1229748640 1 -- 172.21.15.33:0/1807343077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12241055b0 msgr2=0x7f122419cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:36.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.243+0000 7f1229748640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12241055b0 0x7f122419cd20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f121c00eaa0 tx=0x7f121c00ef60 comp rx=0 tx=0).stop 2024-09-16T00:17:36.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.243+0000 7f1229748640 1 -- 172.21.15.33:0/1807343077 shutdown_connections 2024-09-16T00:17:36.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.243+0000 7f1229748640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12100a32d0 0x7f12100a5790 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:36.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.243+0000 7f1229748640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12241055b0 0x7f122419cd20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:36.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.243+0000 7f1229748640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1224100e30 0x7f1224198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:36.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.243+0000 7f1229748640 1 --2- 172.21.15.33:0/1807343077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1224100480 0x7f1224198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:36.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.243+0000 7f1229748640 1 -- 172.21.15.33:0/1807343077 >> 172.21.15.33:0/1807343077 conn(0x7f1224077210 msgr2=0x7f1224103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:36.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.243+0000 7f1229748640 1 -- 172.21.15.33:0/1807343077 shutdown_connections 2024-09-16T00:17:36.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:36.243+0000 7f1229748640 1 -- 172.21.15.33:0/1807343077 wait complete. 2024-09-16T00:17:36.896 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:36 smithi033 ceph-mon[31243]: pgmap v364: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-16T00:17:36.897 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:36 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1807343077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:17:36.973 DEBUG:tasks.fwd_scrub:Yielding 2024-09-16T00:17:36.973 INFO:teuthology.run_tasks:Running task quiescer... 2024-09-16T00:17:36.988 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-16T00:17:36.989 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T00:17:36.989 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-09-16T00:17:36.989 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:17:36.989 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:17:37.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:36 smithi083 ceph-mon[35242]: pgmap v364: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-16T00:17:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:36 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1807343077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:17:37.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:36 smithi060 ceph-mon[35185]: pgmap v364: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-16T00:17:37.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:36 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1807343077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:17:37.268 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:37.272 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:37.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.797+0000 7f62fa0a3640 1 -- 172.21.15.33:0/2796468998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4102ef0 msgr2=0x7f62f410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:37.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.797+0000 7f62fa0a3640 1 --2- 172.21.15.33:0/2796468998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4102ef0 0x7f62f410da40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009ca0 tx=0x7f62ec02a950 comp rx=0 tx=0).stop 2024-09-16T00:17:37.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.797+0000 7f62fa0a3640 1 -- 172.21.15.33:0/2796468998 shutdown_connections 2024-09-16T00:17:37.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.797+0000 7f62fa0a3640 1 --2- 172.21.15.33:0/2796468998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4102ef0 0x7f62f410da40 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:37.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.797+0000 7f62fa0a3640 1 --2- 172.21.15.33:0/2796468998 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62f4102570 0x7f62f41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:37.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.797+0000 7f62fa0a3640 1 --2- 172.21.15.33:0/2796468998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f4100e20 0x7f62f4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:37.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.797+0000 7f62fa0a3640 1 -- 172.21.15.33:0/2796468998 >> 172.21.15.33:0/2796468998 conn(0x7f62f40fc7d0 msgr2=0x7f62f40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:37.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.797+0000 7f62fa0a3640 1 -- 172.21.15.33:0/2796468998 shutdown_connections 2024-09-16T00:17:37.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.797+0000 7f62fa0a3640 1 -- 172.21.15.33:0/2796468998 wait complete. 2024-09-16T00:17:37.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62fa0a3640 1 Processor -- start 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62fa0a3640 1 -- start start 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62fa0a3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4100e20 0x7f62f4199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62fa0a3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62f4102570 0x7f62f41998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62fa0a3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f4102ef0 0x7f62f41a43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62fa0a3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f62f4110f80 con 0x7f62f4100e20 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62fa0a3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f62f4110e00 con 0x7f62f4102ef0 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62fa0a3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f62f4111100 con 0x7f62f4102570 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62f37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4100e20 0x7f62f4199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62f37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4100e20 0x7f62f4199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36324/0 (socket says 172.21.15.33:36324) 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62f37fe640 1 -- 172.21.15.33:0/1521663528 learned_addr learned my addr 172.21.15.33:0/1521663528 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62f3fff640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f4102ef0 0x7f62f41a43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.798+0000 7f62eaffd640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62f4102570 0x7f62f41998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:37.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.799+0000 7f62f37fe640 1 -- 172.21.15.33:0/1521663528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62f4102570 msgr2=0x7f62f41998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.799+0000 7f62f37fe640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62f4102570 0x7f62f41998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.799+0000 7f62f37fe640 1 -- 172.21.15.33:0/1521663528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f4102ef0 msgr2=0x7f62f41a43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.799+0000 7f62f37fe640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f4102ef0 0x7f62f41a43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.799+0000 7f62f37fe640 1 -- 172.21.15.33:0/1521663528 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f41a4930 con 0x7f62f4100e20 2024-09-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.799+0000 7f62f37fe640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4100e20 0x7f62f4199370 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f62e400d950 tx=0x7f62e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.799+0000 7f62f17fa640 1 -- 172.21.15.33:0/1521663528 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e4017070 con 0x7f62f4100e20 2024-09-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.799+0000 7f62f17fa640 1 -- 172.21.15.33:0/1521663528 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e4009c30 con 0x7f62f4100e20 2024-09-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.799+0000 7f62fa0a3640 1 -- 172.21.15.33:0/1521663528 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62f41a4b00 con 0x7f62f4100e20 2024-09-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.799+0000 7f62f17fa640 1 -- 172.21.15.33:0/1521663528 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e400f6b0 con 0x7f62f4100e20 2024-09-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.799+0000 7f62fa0a3640 1 -- 172.21.15.33:0/1521663528 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62f41a5010 con 0x7f62f4100e20 2024-09-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.800+0000 7f62fa0a3640 1 -- 172.21.15.33:0/1521663528 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62f4101200 con 0x7f62f4100e20 2024-09-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.801+0000 7f62f17fa640 1 -- 172.21.15.33:0/1521663528 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f62e40109d0 con 0x7f62f4100e20 2024-09-16T00:17:37.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.802+0000 7f62f17fa640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62e00a32d0 0x7f62e00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:37.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.802+0000 7f62f17fa640 1 -- 172.21.15.33:0/1521663528 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f62e40cfb20 con 0x7f62f4100e20 2024-09-16T00:17:37.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.802+0000 7f62eaffd640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62e00a32d0 0x7f62e00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:37.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.803+0000 7f62eaffd640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62e00a32d0 0x7f62e00a5790 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f62f406aac0 tx=0x7f62dc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:37.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.806+0000 7f62f17fa640 1 -- 172.21.15.33:0/1521663528 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f62e4009de0 con 0x7f62f4100e20 2024-09-16T00:17:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.978+0000 7f62fa0a3640 1 -- 172.21.15.33:0/1521663528 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f62f4106f90 con 0x7f62e00a32d0 2024-09-16T00:17:37.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.986+0000 7f62f17fa640 1 -- 172.21.15.33:0/1521663528 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340771 (secure 0 0 0) 0x7f62f4106f90 con 0x7f62e00a32d0 2024-09-16T00:17:37.989 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:37.991 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-09-16T00:17:37.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.989+0000 7f62fa0a3640 1 -- 172.21.15.33:0/1521663528 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62e00a32d0 msgr2=0x7f62e00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:37.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.989+0000 7f62fa0a3640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62e00a32d0 0x7f62e00a5790 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f62f406aac0 tx=0x7f62dc00b390 comp rx=0 tx=0).stop 2024-09-16T00:17:37.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.989+0000 7f62fa0a3640 1 -- 172.21.15.33:0/1521663528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4100e20 msgr2=0x7f62f4199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:37.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.989+0000 7f62fa0a3640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4100e20 0x7f62f4199370 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f62e400d950 tx=0x7f62e400de10 comp rx=0 tx=0).stop 2024-09-16T00:17:37.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.989+0000 7f62fa0a3640 1 -- 172.21.15.33:0/1521663528 shutdown_connections 2024-09-16T00:17:37.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.989+0000 7f62fa0a3640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62e00a32d0 0x7f62e00a5790 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:37.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.989+0000 7f62fa0a3640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f4102ef0 0x7f62f41a43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:37.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.989+0000 7f62fa0a3640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62f4102570 0x7f62f41998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:37.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.989+0000 7f62fa0a3640 1 --2- 172.21.15.33:0/1521663528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4100e20 0x7f62f4199370 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:37.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.989+0000 7f62fa0a3640 1 -- 172.21.15.33:0/1521663528 >> 172.21.15.33:0/1521663528 conn(0x7f62f40fc7d0 msgr2=0x7f62f4106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:37.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.989+0000 7f62fa0a3640 1 -- 172.21.15.33:0/1521663528 shutdown_connections 2024-09-16T00:17:37.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:37.989+0000 7f62fa0a3640 1 -- 172.21.15.33:0/1521663528 wait complete. 2024-09-16T00:17:38.764 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-16T00:17:37.672082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666913,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":780,"num_read_kb":506,"num_write":365,"num_write_kb":3075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1242843,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386388,"kb_used_data":12144,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555436,"statfs":{"total":1151940427776,"available":1151544766464,"internally_reserved":0,"allocated":12435456,"data_stored":8181353,"data_compressed":471663,"data_compressed_allocated":638976,"data_compressed_original":2809856,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8987,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":41,"num_write":10,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003263"},"pg_stats":[{"pgid":"2.1f","version":"86'9","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578728+0000","last_change":"2024-09-16T00:16:09.034677+0000","last_active":"2024-09-16T00:17:09.578728+0000","last_peered":"2024-09-16T00:17:09.578728+0000","last_clean":"2024-09-16T00:17:09.578728+0000","last_became_active":"2024-09-16T00:16:09.024755+0000","last_became_peered":"2024-09-16T00:16:09.024755+0000","last_unstale":"2024-09-16T00:17:09.578728+0000","last_undegraded":"2024-09-16T00:17:09.578728+0000","last_fullsized":"2024-09-16T00:17:09.578728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:09.012210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579433+0000","last_change":"2024-09-16T00:15:06.547094+0000","last_active":"2024-09-16T00:17:09.579433+0000","last_peered":"2024-09-16T00:17:09.579433+0000","last_clean":"2024-09-16T00:17:09.579433+0000","last_became_active":"2024-09-16T00:15:06.542764+0000","last_became_peered":"2024-09-16T00:15:06.542764+0000","last_unstale":"2024-09-16T00:17:09.579433+0000","last_undegraded":"2024-09-16T00:17:09.579433+0000","last_fullsized":"2024-09-16T00:17:09.579433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:04:37.449577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574641+0000","last_change":"2024-09-16T00:15:13.578400+0000","last_active":"2024-09-16T00:17:09.574641+0000","last_peered":"2024-09-16T00:17:09.574641+0000","last_clean":"2024-09-16T00:17:09.574641+0000","last_became_active":"2024-09-16T00:15:13.577787+0000","last_became_peered":"2024-09-16T00:15:13.577787+0000","last_unstale":"2024-09-16T00:17:09.574641+0000","last_undegraded":"2024-09-16T00:17:09.574641+0000","last_fullsized":"2024-09-16T00:17:09.574641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:19:59.194152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579738+0000","last_change":"2024-09-16T00:15:04.373381+0000","last_active":"2024-09-16T00:17:09.579738+0000","last_peered":"2024-09-16T00:17:09.579738+0000","last_clean":"2024-09-16T00:17:09.579738+0000","last_became_active":"2024-09-16T00:15:04.373028+0000","last_became_peered":"2024-09-16T00:15:04.373028+0000","last_unstale":"2024-09-16T00:17:09.579738+0000","last_undegraded":"2024-09-16T00:17:09.579738+0000","last_fullsized":"2024-09-16T00:17:09.579738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:22.902583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577500+0000","last_change":"2024-09-16T00:15:06.554048+0000","last_active":"2024-09-16T00:17:09.577500+0000","last_peered":"2024-09-16T00:17:09.577500+0000","last_clean":"2024-09-16T00:17:09.577500+0000","last_became_active":"2024-09-16T00:15:06.553044+0000","last_became_peered":"2024-09-16T00:15:06.553044+0000","last_unstale":"2024-09-16T00:17:09.577500+0000","last_undegraded":"2024-09-16T00:17:09.577500+0000","last_fullsized":"2024-09-16T00:17:09.577500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:22:53.333705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574960+0000","last_change":"2024-09-16T00:15:13.586995+0000","last_active":"2024-09-16T00:17:09.574960+0000","last_peered":"2024-09-16T00:17:09.574960+0000","last_clean":"2024-09-16T00:17:09.574960+0000","last_became_active":"2024-09-16T00:15:13.586509+0000","last_became_peered":"2024-09-16T00:15:13.586509+0000","last_unstale":"2024-09-16T00:17:09.574960+0000","last_undegraded":"2024-09-16T00:17:09.574960+0000","last_fullsized":"2024-09-16T00:17:09.574960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:39.418581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579645+0000","last_change":"2024-09-16T00:15:04.348235+0000","last_active":"2024-09-16T00:17:09.579645+0000","last_peered":"2024-09-16T00:17:09.579645+0000","last_clean":"2024-09-16T00:17:09.579645+0000","last_became_active":"2024-09-16T00:15:04.347518+0000","last_became_peered":"2024-09-16T00:15:04.347518+0000","last_unstale":"2024-09-16T00:17:09.579645+0000","last_undegraded":"2024-09-16T00:17:09.579645+0000","last_fullsized":"2024-09-16T00:17:09.579645+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:24.396911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581041+0000","last_change":"2024-09-16T00:15:06.554067+0000","last_active":"2024-09-16T00:17:09.581041+0000","last_peered":"2024-09-16T00:17:09.581041+0000","last_clean":"2024-09-16T00:17:09.581041+0000","last_became_active":"2024-09-16T00:15:06.553715+0000","last_became_peered":"2024-09-16T00:15:06.553715+0000","last_unstale":"2024-09-16T00:17:09.581041+0000","last_undegraded":"2024-09-16T00:17:09.581041+0000","last_fullsized":"2024-09-16T00:17:09.581041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:22.008530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575352+0000","last_change":"2024-09-16T00:15:13.586872+0000","last_active":"2024-09-16T00:17:09.575352+0000","last_peered":"2024-09-16T00:17:09.575352+0000","last_clean":"2024-09-16T00:17:09.575352+0000","last_became_active":"2024-09-16T00:15:13.586375+0000","last_became_peered":"2024-09-16T00:15:13.586375+0000","last_unstale":"2024-09-16T00:17:09.575352+0000","last_undegraded":"2024-09-16T00:17:09.575352+0000","last_fullsized":"2024-09-16T00:17:09.575352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:43.668827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579663+0000","last_change":"2024-09-16T00:15:04.347964+0000","last_active":"2024-09-16T00:17:09.579663+0000","last_peered":"2024-09-16T00:17:09.579663+0000","last_clean":"2024-09-16T00:17:09.579663+0000","last_became_active":"2024-09-16T00:15:04.347588+0000","last_became_peered":"2024-09-16T00:15:04.347588+0000","last_unstale":"2024-09-16T00:17:09.579663+0000","last_undegraded":"2024-09-16T00:17:09.579663+0000","last_fullsized":"2024-09-16T00:17:09.579663+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:14.199287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576911+0000","last_change":"2024-09-16T00:15:06.544918+0000","last_active":"2024-09-16T00:17:09.576911+0000","last_peered":"2024-09-16T00:17:09.576911+0000","last_clean":"2024-09-16T00:17:09.576911+0000","last_became_active":"2024-09-16T00:15:06.544579+0000","last_became_peered":"2024-09-16T00:15:06.544579+0000","last_unstale":"2024-09-16T00:17:09.576911+0000","last_undegraded":"2024-09-16T00:17:09.576911+0000","last_fullsized":"2024-09-16T00:17:09.576911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:06:37.475644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575052+0000","last_change":"2024-09-16T00:15:13.581488+0000","last_active":"2024-09-16T00:17:09.575052+0000","last_peered":"2024-09-16T00:17:09.575052+0000","last_clean":"2024-09-16T00:17:09.575052+0000","last_became_active":"2024-09-16T00:15:13.581164+0000","last_became_peered":"2024-09-16T00:15:13.581164+0000","last_unstale":"2024-09-16T00:17:09.575052+0000","last_undegraded":"2024-09-16T00:17:09.575052+0000","last_fullsized":"2024-09-16T00:17:09.575052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:05.061592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581271+0000","last_change":"2024-09-16T00:15:06.551982+0000","last_active":"2024-09-16T00:17:09.581271+0000","last_peered":"2024-09-16T00:17:09.581271+0000","last_clean":"2024-09-16T00:17:09.581271+0000","last_became_active":"2024-09-16T00:15:06.551663+0000","last_became_peered":"2024-09-16T00:15:06.551663+0000","last_unstale":"2024-09-16T00:17:09.581271+0000","last_undegraded":"2024-09-16T00:17:09.581271+0000","last_fullsized":"2024-09-16T00:17:09.581271+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:11.928703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579394+0000","last_change":"2024-09-16T00:15:04.352237+0000","last_active":"2024-09-16T00:17:09.579394+0000","last_peered":"2024-09-16T00:17:09.579394+0000","last_clean":"2024-09-16T00:17:09.579394+0000","last_became_active":"2024-09-16T00:15:04.351884+0000","last_became_peered":"2024-09-16T00:15:04.351884+0000","last_unstale":"2024-09-16T00:17:09.579394+0000","last_undegraded":"2024-09-16T00:17:09.579394+0000","last_fullsized":"2024-09-16T00:17:09.579394+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:32.932020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575580+0000","last_change":"2024-09-16T00:15:13.638917+0000","last_active":"2024-09-16T00:17:09.575580+0000","last_peered":"2024-09-16T00:17:09.575580+0000","last_clean":"2024-09-16T00:17:09.575580+0000","last_became_active":"2024-09-16T00:15:13.638401+0000","last_became_peered":"2024-09-16T00:15:13.638401+0000","last_unstale":"2024-09-16T00:17:09.575580+0000","last_undegraded":"2024-09-16T00:17:09.575580+0000","last_fullsized":"2024-09-16T00:17:09.575580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:38.082407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579388+0000","last_change":"2024-09-16T00:15:06.547909+0000","last_active":"2024-09-16T00:17:09.579388+0000","last_peered":"2024-09-16T00:17:09.579388+0000","last_clean":"2024-09-16T00:17:09.579388+0000","last_became_active":"2024-09-16T00:15:06.546814+0000","last_became_peered":"2024-09-16T00:15:06.546814+0000","last_unstale":"2024-09-16T00:17:09.579388+0000","last_undegraded":"2024-09-16T00:17:09.579388+0000","last_fullsized":"2024-09-16T00:17:09.579388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:46.419406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780898+0000","last_change":"2024-09-16T00:16:09.027691+0000","last_active":"2024-09-16T00:17:09.780898+0000","last_peered":"2024-09-16T00:17:09.780898+0000","last_clean":"2024-09-16T00:17:09.780898+0000","last_became_active":"2024-09-16T00:16:09.023204+0000","last_became_peered":"2024-09-16T00:16:09.023204+0000","last_unstale":"2024-09-16T00:17:09.780898+0000","last_undegraded":"2024-09-16T00:17:09.780898+0000","last_fullsized":"2024-09-16T00:17:09.780898+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:44.647576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575627+0000","last_change":"2024-09-16T00:15:13.588345+0000","last_active":"2024-09-16T00:17:09.575627+0000","last_peered":"2024-09-16T00:17:09.575627+0000","last_clean":"2024-09-16T00:17:09.575627+0000","last_became_active":"2024-09-16T00:15:13.587829+0000","last_became_peered":"2024-09-16T00:15:13.587829+0000","last_unstale":"2024-09-16T00:17:09.575627+0000","last_undegraded":"2024-09-16T00:17:09.575627+0000","last_fullsized":"2024-09-16T00:17:09.575627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:04.261031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781137+0000","last_change":"2024-09-16T00:15:06.545485+0000","last_active":"2024-09-16T00:17:09.781137+0000","last_peered":"2024-09-16T00:17:09.781137+0000","last_clean":"2024-09-16T00:17:09.781137+0000","last_became_active":"2024-09-16T00:15:06.545178+0000","last_became_peered":"2024-09-16T00:15:06.545178+0000","last_unstale":"2024-09-16T00:17:09.781137+0000","last_undegraded":"2024-09-16T00:17:09.781137+0000","last_fullsized":"2024-09-16T00:17:09.781137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:26.923469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579782+0000","last_change":"2024-09-16T00:15:04.340546+0000","last_active":"2024-09-16T00:17:09.579782+0000","last_peered":"2024-09-16T00:17:09.579782+0000","last_clean":"2024-09-16T00:17:09.579782+0000","last_became_active":"2024-09-16T00:15:04.340173+0000","last_became_peered":"2024-09-16T00:15:04.340173+0000","last_unstale":"2024-09-16T00:17:09.579782+0000","last_undegraded":"2024-09-16T00:17:09.579782+0000","last_fullsized":"2024-09-16T00:17:09.579782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:35:01.134433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578541+0000","last_change":"2024-09-16T00:15:13.589963+0000","last_active":"2024-09-16T00:17:09.578541+0000","last_peered":"2024-09-16T00:17:09.578541+0000","last_clean":"2024-09-16T00:17:09.578541+0000","last_became_active":"2024-09-16T00:15:13.589615+0000","last_became_peered":"2024-09-16T00:15:13.589615+0000","last_unstale":"2024-09-16T00:17:09.578541+0000","last_undegraded":"2024-09-16T00:17:09.578541+0000","last_fullsized":"2024-09-16T00:17:09.578541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:37.197765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579731+0000","last_change":"2024-09-16T00:15:06.544374+0000","last_active":"2024-09-16T00:17:09.579731+0000","last_peered":"2024-09-16T00:17:09.579731+0000","last_clean":"2024-09-16T00:17:09.579731+0000","last_became_active":"2024-09-16T00:15:06.544027+0000","last_became_peered":"2024-09-16T00:15:06.544027+0000","last_unstale":"2024-09-16T00:17:09.579731+0000","last_undegraded":"2024-09-16T00:17:09.579731+0000","last_fullsized":"2024-09-16T00:17:09.579731+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580105+0000","last_change":"2024-09-16T00:15:04.374209+0000","last_active":"2024-09-16T00:17:09.580105+0000","last_peered":"2024-09-16T00:17:09.580105+0000","last_clean":"2024-09-16T00:17:09.580105+0000","last_became_active":"2024-09-16T00:15:04.373814+0000","last_became_peered":"2024-09-16T00:15:04.373814+0000","last_unstale":"2024-09-16T00:17:09.580105+0000","last_undegraded":"2024-09-16T00:17:09.580105+0000","last_fullsized":"2024-09-16T00:17:09.580105+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:13.516161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575489+0000","last_change":"2024-09-16T00:15:13.639210+0000","last_active":"2024-09-16T00:17:09.575489+0000","last_peered":"2024-09-16T00:17:09.575489+0000","last_clean":"2024-09-16T00:17:09.575489+0000","last_became_active":"2024-09-16T00:15:13.638719+0000","last_became_peered":"2024-09-16T00:15:13.638719+0000","last_unstale":"2024-09-16T00:17:09.575489+0000","last_undegraded":"2024-09-16T00:17:09.575489+0000","last_fullsized":"2024-09-16T00:17:09.575489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:26.130778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577341+0000","last_change":"2024-09-16T00:15:06.553108+0000","last_active":"2024-09-16T00:17:09.577341+0000","last_peered":"2024-09-16T00:17:09.577341+0000","last_clean":"2024-09-16T00:17:09.577341+0000","last_became_active":"2024-09-16T00:15:06.552817+0000","last_became_peered":"2024-09-16T00:15:06.552817+0000","last_unstale":"2024-09-16T00:17:09.577341+0000","last_undegraded":"2024-09-16T00:17:09.577341+0000","last_fullsized":"2024-09-16T00:17:09.577341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:06.821337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575833+0000","last_change":"2024-09-16T00:15:04.644185+0000","last_active":"2024-09-16T00:17:09.575833+0000","last_peered":"2024-09-16T00:17:09.575833+0000","last_clean":"2024-09-16T00:17:09.575833+0000","last_became_active":"2024-09-16T00:15:04.643887+0000","last_became_peered":"2024-09-16T00:15:04.643887+0000","last_unstale":"2024-09-16T00:17:09.575833+0000","last_undegraded":"2024-09-16T00:17:09.575833+0000","last_fullsized":"2024-09-16T00:17:09.575833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:44.279272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574877+0000","last_change":"2024-09-16T00:15:13.574761+0000","last_active":"2024-09-16T00:17:09.574877+0000","last_peered":"2024-09-16T00:17:09.574877+0000","last_clean":"2024-09-16T00:17:09.574877+0000","last_became_active":"2024-09-16T00:15:13.574347+0000","last_became_peered":"2024-09-16T00:15:13.574347+0000","last_unstale":"2024-09-16T00:17:09.574877+0000","last_undegraded":"2024-09-16T00:17:09.574877+0000","last_fullsized":"2024-09-16T00:17:09.574877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:22.040121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579504+0000","last_change":"2024-09-16T00:15:06.550338+0000","last_active":"2024-09-16T00:17:09.579504+0000","last_peered":"2024-09-16T00:17:09.579504+0000","last_clean":"2024-09-16T00:17:09.579504+0000","last_became_active":"2024-09-16T00:15:06.549868+0000","last_became_peered":"2024-09-16T00:15:06.549868+0000","last_unstale":"2024-09-16T00:17:09.579504+0000","last_undegraded":"2024-09-16T00:17:09.579504+0000","last_fullsized":"2024-09-16T00:17:09.579504+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:26.643338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'13","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:28.555275+0000","last_change":"2024-09-16T00:15:04.342306+0000","last_active":"2024-09-16T00:17:28.555275+0000","last_peered":"2024-09-16T00:17:28.555275+0000","last_clean":"2024-09-16T00:17:28.555275+0000","last_became_active":"2024-09-16T00:15:04.341964+0000","last_became_peered":"2024-09-16T00:15:04.341964+0000","last_unstale":"2024-09-16T00:17:28.555275+0000","last_undegraded":"2024-09-16T00:17:28.555275+0000","last_fullsized":"2024-09-16T00:17:28.555275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:14.137199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578972+0000","last_change":"2024-09-16T00:15:13.638246+0000","last_active":"2024-09-16T00:17:09.578972+0000","last_peered":"2024-09-16T00:17:09.578972+0000","last_clean":"2024-09-16T00:17:09.578972+0000","last_became_active":"2024-09-16T00:15:13.637816+0000","last_became_peered":"2024-09-16T00:15:13.637816+0000","last_unstale":"2024-09-16T00:17:09.578972+0000","last_undegraded":"2024-09-16T00:17:09.578972+0000","last_fullsized":"2024-09-16T00:17:09.578972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:27.331156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718239+0000","last_change":"2024-09-16T00:15:06.544057+0000","last_active":"2024-09-16T00:17:09.718239+0000","last_peered":"2024-09-16T00:17:09.718239+0000","last_clean":"2024-09-16T00:17:09.718239+0000","last_became_active":"2024-09-16T00:15:06.543244+0000","last_became_peered":"2024-09-16T00:15:06.543244+0000","last_unstale":"2024-09-16T00:17:09.718239+0000","last_undegraded":"2024-09-16T00:17:09.718239+0000","last_fullsized":"2024-09-16T00:17:09.718239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:42:28.950014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579701+0000","last_change":"2024-09-16T00:15:04.376419+0000","last_active":"2024-09-16T00:17:09.579701+0000","last_peered":"2024-09-16T00:17:09.579701+0000","last_clean":"2024-09-16T00:17:09.579701+0000","last_became_active":"2024-09-16T00:15:04.375976+0000","last_became_peered":"2024-09-16T00:15:04.375976+0000","last_unstale":"2024-09-16T00:17:09.579701+0000","last_undegraded":"2024-09-16T00:17:09.579701+0000","last_fullsized":"2024-09-16T00:17:09.579701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:33:38.652109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781038+0000","last_change":"2024-09-16T00:15:13.584306+0000","last_active":"2024-09-16T00:17:09.781038+0000","last_peered":"2024-09-16T00:17:09.781038+0000","last_clean":"2024-09-16T00:17:09.781038+0000","last_became_active":"2024-09-16T00:15:13.583840+0000","last_became_peered":"2024-09-16T00:15:13.583840+0000","last_unstale":"2024-09-16T00:17:09.781038+0000","last_undegraded":"2024-09-16T00:17:09.781038+0000","last_fullsized":"2024-09-16T00:17:09.781038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:36.660458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577806+0000","last_change":"2024-09-16T00:15:06.538902+0000","last_active":"2024-09-16T00:17:09.577806+0000","last_peered":"2024-09-16T00:17:09.577806+0000","last_clean":"2024-09-16T00:17:09.577806+0000","last_became_active":"2024-09-16T00:15:06.538537+0000","last_became_peered":"2024-09-16T00:15:06.538537+0000","last_unstale":"2024-09-16T00:17:09.577806+0000","last_undegraded":"2024-09-16T00:17:09.577806+0000","last_fullsized":"2024-09-16T00:17:09.577806+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:38.411711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578013+0000","last_change":"2024-09-16T00:15:04.374762+0000","last_active":"2024-09-16T00:17:09.578013+0000","last_peered":"2024-09-16T00:17:09.578013+0000","last_clean":"2024-09-16T00:17:09.578013+0000","last_became_active":"2024-09-16T00:15:04.374394+0000","last_became_peered":"2024-09-16T00:15:04.374394+0000","last_unstale":"2024-09-16T00:17:09.578013+0000","last_undegraded":"2024-09-16T00:17:09.578013+0000","last_fullsized":"2024-09-16T00:17:09.578013+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:47.075021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579233+0000","last_change":"2024-09-16T00:15:13.638306+0000","last_active":"2024-09-16T00:17:09.579233+0000","last_peered":"2024-09-16T00:17:09.579233+0000","last_clean":"2024-09-16T00:17:09.579233+0000","last_became_active":"2024-09-16T00:15:13.637931+0000","last_became_peered":"2024-09-16T00:15:13.637931+0000","last_unstale":"2024-09-16T00:17:09.579233+0000","last_undegraded":"2024-09-16T00:17:09.579233+0000","last_fullsized":"2024-09-16T00:17:09.579233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:41.385174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577585+0000","last_change":"2024-09-16T00:15:06.548209+0000","last_active":"2024-09-16T00:17:09.577585+0000","last_peered":"2024-09-16T00:17:09.577585+0000","last_clean":"2024-09-16T00:17:09.577585+0000","last_became_active":"2024-09-16T00:15:06.547846+0000","last_became_peered":"2024-09-16T00:15:06.547846+0000","last_unstale":"2024-09-16T00:17:09.577585+0000","last_undegraded":"2024-09-16T00:17:09.577585+0000","last_fullsized":"2024-09-16T00:17:09.577585+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:15.821847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780889+0000","last_change":"2024-09-16T00:16:09.023918+0000","last_active":"2024-09-16T00:17:09.780889+0000","last_peered":"2024-09-16T00:17:09.780889+0000","last_clean":"2024-09-16T00:17:09.780889+0000","last_became_active":"2024-09-16T00:16:09.023230+0000","last_became_peered":"2024-09-16T00:16:09.023230+0000","last_unstale":"2024-09-16T00:17:09.780889+0000","last_undegraded":"2024-09-16T00:17:09.780889+0000","last_fullsized":"2024-09-16T00:17:09.780889+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:31.527863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717949+0000","last_change":"2024-09-16T00:15:13.579387+0000","last_active":"2024-09-16T00:17:09.717949+0000","last_peered":"2024-09-16T00:17:09.717949+0000","last_clean":"2024-09-16T00:17:09.717949+0000","last_became_active":"2024-09-16T00:15:13.578996+0000","last_became_peered":"2024-09-16T00:15:13.578996+0000","last_unstale":"2024-09-16T00:17:09.717949+0000","last_undegraded":"2024-09-16T00:17:09.717949+0000","last_fullsized":"2024-09-16T00:17:09.717949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:51.703060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579796+0000","last_change":"2024-09-16T00:17:09.579796+0000","last_active":"2024-09-16T00:17:09.579796+0000","last_peered":"2024-09-16T00:17:09.579796+0000","last_clean":"2024-09-16T00:17:09.579796+0000","last_became_active":"2024-09-16T00:17:09.579386+0000","last_became_peered":"2024-09-16T00:17:09.579386+0000","last_unstale":"2024-09-16T00:17:09.579796+0000","last_undegraded":"2024-09-16T00:17:09.579796+0000","last_fullsized":"2024-09-16T00:17:09.579796+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:39:42.571193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":150,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:33.321091+0000","last_change":"2024-09-16T00:15:04.373896+0000","last_active":"2024-09-16T00:17:33.321091+0000","last_peered":"2024-09-16T00:17:33.321091+0000","last_clean":"2024-09-16T00:17:33.321091+0000","last_became_active":"2024-09-16T00:15:04.373493+0000","last_became_peered":"2024-09-16T00:15:04.373493+0000","last_unstale":"2024-09-16T00:17:33.321091+0000","last_undegraded":"2024-09-16T00:17:33.321091+0000","last_fullsized":"2024-09-16T00:17:33.321091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:17:10.767267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718128+0000","last_change":"2024-09-16T00:15:13.638575+0000","last_active":"2024-09-16T00:17:09.718128+0000","last_peered":"2024-09-16T00:17:09.718128+0000","last_clean":"2024-09-16T00:17:09.718128+0000","last_became_active":"2024-09-16T00:15:13.638144+0000","last_became_peered":"2024-09-16T00:15:13.638144+0000","last_unstale":"2024-09-16T00:17:09.718128+0000","last_undegraded":"2024-09-16T00:17:09.718128+0000","last_fullsized":"2024-09-16T00:17:09.718128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:39.345011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718312+0000","last_change":"2024-09-16T00:15:06.548852+0000","last_active":"2024-09-16T00:17:09.718312+0000","last_peered":"2024-09-16T00:17:09.718312+0000","last_clean":"2024-09-16T00:17:09.718312+0000","last_became_active":"2024-09-16T00:15:06.548238+0000","last_became_peered":"2024-09-16T00:15:06.548238+0000","last_unstale":"2024-09-16T00:17:09.718312+0000","last_undegraded":"2024-09-16T00:17:09.718312+0000","last_fullsized":"2024-09-16T00:17:09.718312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:16.115338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580690+0000","last_change":"2024-09-16T00:15:04.350753+0000","last_active":"2024-09-16T00:17:09.580690+0000","last_peered":"2024-09-16T00:17:09.580690+0000","last_clean":"2024-09-16T00:17:09.580690+0000","last_became_active":"2024-09-16T00:15:04.350406+0000","last_became_peered":"2024-09-16T00:15:04.350406+0000","last_unstale":"2024-09-16T00:17:09.580690+0000","last_undegraded":"2024-09-16T00:17:09.580690+0000","last_fullsized":"2024-09-16T00:17:09.580690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:09:54.737287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576996+0000","last_change":"2024-09-16T00:15:13.639302+0000","last_active":"2024-09-16T00:17:09.576996+0000","last_peered":"2024-09-16T00:17:09.576996+0000","last_clean":"2024-09-16T00:17:09.576996+0000","last_became_active":"2024-09-16T00:15:13.638863+0000","last_became_peered":"2024-09-16T00:15:13.638863+0000","last_unstale":"2024-09-16T00:17:09.576996+0000","last_undegraded":"2024-09-16T00:17:09.576996+0000","last_fullsized":"2024-09-16T00:17:09.576996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:23.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582176+0000","last_change":"2024-09-16T00:17:09.582176+0000","last_active":"2024-09-16T00:17:09.582176+0000","last_peered":"2024-09-16T00:17:09.582176+0000","last_clean":"2024-09-16T00:17:09.582176+0000","last_became_active":"2024-09-16T00:17:09.581740+0000","last_became_peered":"2024-09-16T00:17:09.581740+0000","last_unstale":"2024-09-16T00:17:09.582176+0000","last_undegraded":"2024-09-16T00:17:09.582176+0000","last_fullsized":"2024-09-16T00:17:09.582176+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:57.176724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575861+0000","last_change":"2024-09-16T00:15:04.342216+0000","last_active":"2024-09-16T00:17:09.575861+0000","last_peered":"2024-09-16T00:17:09.575861+0000","last_clean":"2024-09-16T00:17:09.575861+0000","last_became_active":"2024-09-16T00:15:04.341763+0000","last_became_peered":"2024-09-16T00:15:04.341763+0000","last_unstale":"2024-09-16T00:17:09.575861+0000","last_undegraded":"2024-09-16T00:17:09.575861+0000","last_fullsized":"2024-09-16T00:17:09.575861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:44.938527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":350,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580892+0000","last_change":"2024-09-16T00:13:51.108571+0000","last_active":"2024-09-16T00:17:09.580892+0000","last_peered":"2024-09-16T00:17:09.580892+0000","last_clean":"2024-09-16T00:17:09.580892+0000","last_became_active":"2024-09-16T00:13:51.108088+0000","last_became_peered":"2024-09-16T00:13:51.108088+0000","last_unstale":"2024-09-16T00:17:09.580892+0000","last_undegraded":"2024-09-16T00:17:09.580892+0000","last_fullsized":"2024-09-16T00:17:09.580892+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_clean_scrub_stamp":"2024-09-16T00:10:24.533807+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:02.511064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581143+0000","last_change":"2024-09-16T00:15:13.588665+0000","last_active":"2024-09-16T00:17:09.581143+0000","last_peered":"2024-09-16T00:17:09.581143+0000","last_clean":"2024-09-16T00:17:09.581143+0000","last_became_active":"2024-09-16T00:15:13.588222+0000","last_became_peered":"2024-09-16T00:15:13.588222+0000","last_unstale":"2024-09-16T00:17:09.581143+0000","last_undegraded":"2024-09-16T00:17:09.581143+0000","last_fullsized":"2024-09-16T00:17:09.581143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:59:59.965962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581454+0000","last_change":"2024-09-16T00:15:06.553114+0000","last_active":"2024-09-16T00:17:09.581454+0000","last_peered":"2024-09-16T00:17:09.581454+0000","last_clean":"2024-09-16T00:17:09.581454+0000","last_became_active":"2024-09-16T00:15:06.552788+0000","last_became_peered":"2024-09-16T00:15:06.552788+0000","last_unstale":"2024-09-16T00:17:09.581454+0000","last_undegraded":"2024-09-16T00:17:09.581454+0000","last_fullsized":"2024-09-16T00:17:09.581454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:05.473847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575364+0000","last_change":"2024-09-16T00:16:09.030747+0000","last_active":"2024-09-16T00:17:09.575364+0000","last_peered":"2024-09-16T00:17:09.575364+0000","last_clean":"2024-09-16T00:17:09.575364+0000","last_became_active":"2024-09-16T00:16:09.021208+0000","last_became_peered":"2024-09-16T00:16:09.021208+0000","last_unstale":"2024-09-16T00:17:09.575364+0000","last_undegraded":"2024-09-16T00:17:09.575364+0000","last_fullsized":"2024-09-16T00:17:09.575364+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:40.605498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580422+0000","last_change":"2024-09-16T00:15:13.575606+0000","last_active":"2024-09-16T00:17:09.580422+0000","last_peered":"2024-09-16T00:17:09.580422+0000","last_clean":"2024-09-16T00:17:09.580422+0000","last_became_active":"2024-09-16T00:15:13.575131+0000","last_became_peered":"2024-09-16T00:15:13.575131+0000","last_unstale":"2024-09-16T00:17:09.580422+0000","last_undegraded":"2024-09-16T00:17:09.580422+0000","last_fullsized":"2024-09-16T00:17:09.580422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:41.636058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.584161+0000","last_change":"2024-09-16T00:17:09.583522+0000","last_active":"2024-09-16T00:17:09.584161+0000","last_peered":"2024-09-16T00:17:09.584161+0000","last_clean":"2024-09-16T00:17:09.584161+0000","last_became_active":"2024-09-16T00:17:09.583082+0000","last_became_peered":"2024-09-16T00:17:09.583082+0000","last_unstale":"2024-09-16T00:17:09.584161+0000","last_undegraded":"2024-09-16T00:17:09.584161+0000","last_fullsized":"2024-09-16T00:17:09.584161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:48.582126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577245+0000","last_change":"2024-09-16T00:15:04.349095+0000","last_active":"2024-09-16T00:17:09.577245+0000","last_peered":"2024-09-16T00:17:09.577245+0000","last_clean":"2024-09-16T00:17:09.577245+0000","last_became_active":"2024-09-16T00:15:04.348698+0000","last_became_peered":"2024-09-16T00:15:04.348698+0000","last_unstale":"2024-09-16T00:17:09.577245+0000","last_undegraded":"2024-09-16T00:17:09.577245+0000","last_fullsized":"2024-09-16T00:17:09.577245+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:42.282882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577535+0000","last_change":"2024-09-16T00:15:13.580494+0000","last_active":"2024-09-16T00:17:09.577535+0000","last_peered":"2024-09-16T00:17:09.577535+0000","last_clean":"2024-09-16T00:17:09.577535+0000","last_became_active":"2024-09-16T00:15:13.580091+0000","last_became_peered":"2024-09-16T00:15:13.580091+0000","last_unstale":"2024-09-16T00:17:09.577535+0000","last_undegraded":"2024-09-16T00:17:09.577535+0000","last_fullsized":"2024-09-16T00:17:09.577535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:17.314919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580727+0000","last_change":"2024-09-16T00:15:06.544859+0000","last_active":"2024-09-16T00:17:09.580727+0000","last_peered":"2024-09-16T00:17:09.580727+0000","last_clean":"2024-09-16T00:17:09.580727+0000","last_became_active":"2024-09-16T00:15:06.544289+0000","last_became_peered":"2024-09-16T00:15:06.544289+0000","last_unstale":"2024-09-16T00:17:09.580727+0000","last_undegraded":"2024-09-16T00:17:09.580727+0000","last_fullsized":"2024-09-16T00:17:09.580727+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:52.622477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718205+0000","last_change":"2024-09-16T00:15:04.351313+0000","last_active":"2024-09-16T00:17:09.718205+0000","last_peered":"2024-09-16T00:17:09.718205+0000","last_clean":"2024-09-16T00:17:09.718205+0000","last_became_active":"2024-09-16T00:15:04.350928+0000","last_became_peered":"2024-09-16T00:15:04.350928+0000","last_unstale":"2024-09-16T00:17:09.718205+0000","last_undegraded":"2024-09-16T00:17:09.718205+0000","last_fullsized":"2024-09-16T00:17:09.718205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:48.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576571+0000","last_change":"2024-09-16T00:15:13.578573+0000","last_active":"2024-09-16T00:17:09.576571+0000","last_peered":"2024-09-16T00:17:09.576571+0000","last_clean":"2024-09-16T00:17:09.576571+0000","last_became_active":"2024-09-16T00:15:13.578117+0000","last_became_peered":"2024-09-16T00:15:13.578117+0000","last_unstale":"2024-09-16T00:17:09.576571+0000","last_undegraded":"2024-09-16T00:17:09.576571+0000","last_fullsized":"2024-09-16T00:17:09.576571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:53.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781258+0000","last_change":"2024-09-16T00:15:06.553548+0000","last_active":"2024-09-16T00:17:09.781258+0000","last_peered":"2024-09-16T00:17:09.781258+0000","last_clean":"2024-09-16T00:17:09.781258+0000","last_became_active":"2024-09-16T00:15:06.552963+0000","last_became_peered":"2024-09-16T00:15:06.552963+0000","last_unstale":"2024-09-16T00:17:09.781258+0000","last_undegraded":"2024-09-16T00:17:09.781258+0000","last_fullsized":"2024-09-16T00:17:09.781258+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:43.455881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579776+0000","last_change":"2024-09-16T00:15:04.351005+0000","last_active":"2024-09-16T00:17:09.579776+0000","last_peered":"2024-09-16T00:17:09.579776+0000","last_clean":"2024-09-16T00:17:09.579776+0000","last_became_active":"2024-09-16T00:15:04.350614+0000","last_became_peered":"2024-09-16T00:15:04.350614+0000","last_unstale":"2024-09-16T00:17:09.579776+0000","last_undegraded":"2024-09-16T00:17:09.579776+0000","last_fullsized":"2024-09-16T00:17:09.579776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:39.196360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579501+0000","last_change":"2024-09-16T00:16:09.023796+0000","last_active":"2024-09-16T00:17:09.579501+0000","last_peered":"2024-09-16T00:17:09.579501+0000","last_clean":"2024-09-16T00:17:09.579501+0000","last_became_active":"2024-09-16T00:16:09.023403+0000","last_became_peered":"2024-09-16T00:16:09.023403+0000","last_unstale":"2024-09-16T00:17:09.579501+0000","last_undegraded":"2024-09-16T00:17:09.579501+0000","last_fullsized":"2024-09-16T00:17:09.579501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:59.488315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577059+0000","last_change":"2024-09-16T00:15:06.546325+0000","last_active":"2024-09-16T00:17:09.577059+0000","last_peered":"2024-09-16T00:17:09.577059+0000","last_clean":"2024-09-16T00:17:09.577059+0000","last_became_active":"2024-09-16T00:15:06.546028+0000","last_became_peered":"2024-09-16T00:15:06.546028+0000","last_unstale":"2024-09-16T00:17:09.577059+0000","last_undegraded":"2024-09-16T00:17:09.577059+0000","last_fullsized":"2024-09-16T00:17:09.577059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:15.049592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579065+0000","last_change":"2024-09-16T00:15:04.373818+0000","last_active":"2024-09-16T00:17:09.579065+0000","last_peered":"2024-09-16T00:17:09.579065+0000","last_clean":"2024-09-16T00:17:09.579065+0000","last_became_active":"2024-09-16T00:15:04.373406+0000","last_became_peered":"2024-09-16T00:15:04.373406+0000","last_unstale":"2024-09-16T00:17:09.579065+0000","last_undegraded":"2024-09-16T00:17:09.579065+0000","last_fullsized":"2024-09-16T00:17:09.579065+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:08.402660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718065+0000","last_change":"2024-09-16T00:15:13.588669+0000","last_active":"2024-09-16T00:17:09.718065+0000","last_peered":"2024-09-16T00:17:09.718065+0000","last_clean":"2024-09-16T00:17:09.718065+0000","last_became_active":"2024-09-16T00:15:13.588229+0000","last_became_peered":"2024-09-16T00:15:13.588229+0000","last_unstale":"2024-09-16T00:17:09.718065+0000","last_undegraded":"2024-09-16T00:17:09.718065+0000","last_fullsized":"2024-09-16T00:17:09.718065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:30.157296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580525+0000","last_change":"2024-09-16T00:15:06.550221+0000","last_active":"2024-09-16T00:17:09.580525+0000","last_peered":"2024-09-16T00:17:09.580525+0000","last_clean":"2024-09-16T00:17:09.580525+0000","last_became_active":"2024-09-16T00:15:06.549914+0000","last_became_peered":"2024-09-16T00:15:06.549914+0000","last_unstale":"2024-09-16T00:17:09.580525+0000","last_undegraded":"2024-09-16T00:17:09.580525+0000","last_fullsized":"2024-09-16T00:17:09.580525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:18:26.824523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":154,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:33.321594+0000","last_change":"2024-09-16T00:15:04.344389+0000","last_active":"2024-09-16T00:17:33.321594+0000","last_peered":"2024-09-16T00:17:33.321594+0000","last_clean":"2024-09-16T00:17:33.321594+0000","last_became_active":"2024-09-16T00:15:04.343956+0000","last_became_peered":"2024-09-16T00:15:04.343956+0000","last_unstale":"2024-09-16T00:17:33.321594+0000","last_undegraded":"2024-09-16T00:17:33.321594+0000","last_fullsized":"2024-09-16T00:17:33.321594+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:34.019861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579707+0000","last_change":"2024-09-16T00:16:09.024050+0000","last_active":"2024-09-16T00:17:09.579707+0000","last_peered":"2024-09-16T00:17:09.579707+0000","last_clean":"2024-09-16T00:17:09.579707+0000","last_became_active":"2024-09-16T00:16:09.023371+0000","last_became_peered":"2024-09-16T00:16:09.023371+0000","last_unstale":"2024-09-16T00:17:09.579707+0000","last_undegraded":"2024-09-16T00:17:09.579707+0000","last_fullsized":"2024-09-16T00:17:09.579707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:35:19.020370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581433+0000","last_change":"2024-09-16T00:15:06.542744+0000","last_active":"2024-09-16T00:17:09.581433+0000","last_peered":"2024-09-16T00:17:09.581433+0000","last_clean":"2024-09-16T00:17:09.581433+0000","last_became_active":"2024-09-16T00:15:06.541868+0000","last_became_peered":"2024-09-16T00:15:06.541868+0000","last_unstale":"2024-09-16T00:17:09.581433+0000","last_undegraded":"2024-09-16T00:17:09.581433+0000","last_fullsized":"2024-09-16T00:17:09.581433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:10:06.508682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577336+0000","last_change":"2024-09-16T00:15:04.349556+0000","last_active":"2024-09-16T00:17:09.577336+0000","last_peered":"2024-09-16T00:17:09.577336+0000","last_clean":"2024-09-16T00:17:09.577336+0000","last_became_active":"2024-09-16T00:15:04.349242+0000","last_became_peered":"2024-09-16T00:15:04.349242+0000","last_unstale":"2024-09-16T00:17:09.577336+0000","last_undegraded":"2024-09-16T00:17:09.577336+0000","last_fullsized":"2024-09-16T00:17:09.577336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:22:18.459157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781143+0000","last_change":"2024-09-16T00:15:13.584109+0000","last_active":"2024-09-16T00:17:09.781143+0000","last_peered":"2024-09-16T00:17:09.781143+0000","last_clean":"2024-09-16T00:17:09.781143+0000","last_became_active":"2024-09-16T00:15:13.583718+0000","last_became_peered":"2024-09-16T00:15:13.583718+0000","last_unstale":"2024-09-16T00:17:09.781143+0000","last_undegraded":"2024-09-16T00:17:09.781143+0000","last_fullsized":"2024-09-16T00:17:09.781143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:29.416254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717952+0000","last_change":"2024-09-16T00:15:06.548247+0000","last_active":"2024-09-16T00:17:09.717952+0000","last_peered":"2024-09-16T00:17:09.717952+0000","last_clean":"2024-09-16T00:17:09.717952+0000","last_became_active":"2024-09-16T00:15:06.547864+0000","last_became_peered":"2024-09-16T00:15:06.547864+0000","last_unstale":"2024-09-16T00:17:09.717952+0000","last_undegraded":"2024-09-16T00:17:09.717952+0000","last_fullsized":"2024-09-16T00:17:09.717952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:25.350197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575352+0000","last_change":"2024-09-16T00:15:04.350965+0000","last_active":"2024-09-16T00:17:09.575352+0000","last_peered":"2024-09-16T00:17:09.575352+0000","last_clean":"2024-09-16T00:17:09.575352+0000","last_became_active":"2024-09-16T00:15:04.349124+0000","last_became_peered":"2024-09-16T00:15:04.349124+0000","last_unstale":"2024-09-16T00:17:09.575352+0000","last_undegraded":"2024-09-16T00:17:09.575352+0000","last_fullsized":"2024-09-16T00:17:09.575352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:40.859220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576904+0000","last_change":"2024-09-16T00:15:13.581718+0000","last_active":"2024-09-16T00:17:09.576904+0000","last_peered":"2024-09-16T00:17:09.576904+0000","last_clean":"2024-09-16T00:17:09.576904+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:17:09.576904+0000","last_undegraded":"2024-09-16T00:17:09.576904+0000","last_fullsized":"2024-09-16T00:17:09.576904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:08.810606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579555+0000","last_change":"2024-09-16T00:15:06.544507+0000","last_active":"2024-09-16T00:17:09.579555+0000","last_peered":"2024-09-16T00:17:09.579555+0000","last_clean":"2024-09-16T00:17:09.579555+0000","last_became_active":"2024-09-16T00:15:06.544178+0000","last_became_peered":"2024-09-16T00:15:06.544178+0000","last_unstale":"2024-09-16T00:17:09.579555+0000","last_undegraded":"2024-09-16T00:17:09.579555+0000","last_fullsized":"2024-09-16T00:17:09.579555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:38.494266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'39","reported_seq":315,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:33.164505+0000","last_change":"2024-09-16T00:16:09.039763+0000","last_active":"2024-09-16T00:17:33.164505+0000","last_peered":"2024-09-16T00:17:33.164505+0000","last_clean":"2024-09-16T00:17:33.164505+0000","last_became_active":"2024-09-16T00:16:09.022939+0000","last_became_peered":"2024-09-16T00:16:09.022939+0000","last_unstale":"2024-09-16T00:17:33.164505+0000","last_undegraded":"2024-09-16T00:17:33.164505+0000","last_fullsized":"2024-09-16T00:17:33.164505+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:56.431159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71774,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":216,"num_read_kb":169,"num_write":40,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60943,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576887+0000","last_change":"2024-09-16T00:15:13.585928+0000","last_active":"2024-09-16T00:17:09.576887+0000","last_peered":"2024-09-16T00:17:09.576887+0000","last_clean":"2024-09-16T00:17:09.576887+0000","last_became_active":"2024-09-16T00:15:13.582940+0000","last_became_peered":"2024-09-16T00:15:13.582940+0000","last_unstale":"2024-09-16T00:17:09.576887+0000","last_undegraded":"2024-09-16T00:17:09.576887+0000","last_fullsized":"2024-09-16T00:17:09.576887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:57:13.251978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580875+0000","last_change":"2024-09-16T00:15:06.546238+0000","last_active":"2024-09-16T00:17:09.580875+0000","last_peered":"2024-09-16T00:17:09.580875+0000","last_clean":"2024-09-16T00:17:09.580875+0000","last_became_active":"2024-09-16T00:15:06.545919+0000","last_became_peered":"2024-09-16T00:15:06.545919+0000","last_unstale":"2024-09-16T00:17:09.580875+0000","last_undegraded":"2024-09-16T00:17:09.580875+0000","last_fullsized":"2024-09-16T00:17:09.580875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:09.180832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577083+0000","last_change":"2024-09-16T00:15:04.376028+0000","last_active":"2024-09-16T00:17:09.577083+0000","last_peered":"2024-09-16T00:17:09.577083+0000","last_clean":"2024-09-16T00:17:09.577083+0000","last_became_active":"2024-09-16T00:15:04.375633+0000","last_became_peered":"2024-09-16T00:15:04.375633+0000","last_unstale":"2024-09-16T00:17:09.577083+0000","last_undegraded":"2024-09-16T00:17:09.577083+0000","last_fullsized":"2024-09-16T00:17:09.577083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:44.446342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575275+0000","last_change":"2024-09-16T00:16:09.025524+0000","last_active":"2024-09-16T00:17:09.575275+0000","last_peered":"2024-09-16T00:17:09.575275+0000","last_clean":"2024-09-16T00:17:09.575275+0000","last_became_active":"2024-09-16T00:16:09.025035+0000","last_became_peered":"2024-09-16T00:16:09.025035+0000","last_unstale":"2024-09-16T00:17:09.575275+0000","last_undegraded":"2024-09-16T00:17:09.575275+0000","last_fullsized":"2024-09-16T00:17:09.575275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:12.367244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577617+0000","last_change":"2024-09-16T00:15:06.550864+0000","last_active":"2024-09-16T00:17:09.577617+0000","last_peered":"2024-09-16T00:17:09.577617+0000","last_clean":"2024-09-16T00:17:09.577617+0000","last_became_active":"2024-09-16T00:15:06.547956+0000","last_became_peered":"2024-09-16T00:15:06.547956+0000","last_unstale":"2024-09-16T00:17:09.577617+0000","last_undegraded":"2024-09-16T00:17:09.577617+0000","last_fullsized":"2024-09-16T00:17:09.577617+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:26.102175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580692+0000","last_change":"2024-09-16T00:15:04.350349+0000","last_active":"2024-09-16T00:17:09.580692+0000","last_peered":"2024-09-16T00:17:09.580692+0000","last_clean":"2024-09-16T00:17:09.580692+0000","last_became_active":"2024-09-16T00:15:04.349951+0000","last_became_peered":"2024-09-16T00:15:04.349951+0000","last_unstale":"2024-09-16T00:17:09.580692+0000","last_undegraded":"2024-09-16T00:17:09.580692+0000","last_fullsized":"2024-09-16T00:17:09.580692+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:46.000831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577297+0000","last_change":"2024-09-16T00:16:09.025774+0000","last_active":"2024-09-16T00:17:09.577297+0000","last_peered":"2024-09-16T00:17:09.577297+0000","last_clean":"2024-09-16T00:17:09.577297+0000","last_became_active":"2024-09-16T00:16:09.025308+0000","last_became_peered":"2024-09-16T00:16:09.025308+0000","last_unstale":"2024-09-16T00:17:09.577297+0000","last_undegraded":"2024-09-16T00:17:09.577297+0000","last_fullsized":"2024-09-16T00:17:09.577297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:16.664850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580788+0000","last_change":"2024-09-16T00:15:06.552919+0000","last_active":"2024-09-16T00:17:09.580788+0000","last_peered":"2024-09-16T00:17:09.580788+0000","last_clean":"2024-09-16T00:17:09.580788+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:17:09.580788+0000","last_undegraded":"2024-09-16T00:17:09.580788+0000","last_fullsized":"2024-09-16T00:17:09.580788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:47:38.651837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":145,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:32.319661+0000","last_change":"2024-09-16T00:15:04.351116+0000","last_active":"2024-09-16T00:17:32.319661+0000","last_peered":"2024-09-16T00:17:32.319661+0000","last_clean":"2024-09-16T00:17:32.319661+0000","last_became_active":"2024-09-16T00:15:04.350728+0000","last_became_peered":"2024-09-16T00:15:04.350728+0000","last_unstale":"2024-09-16T00:17:32.319661+0000","last_undegraded":"2024-09-16T00:17:32.319661+0000","last_fullsized":"2024-09-16T00:17:32.319661+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:38.314019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":103,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579322+0000","last_change":"2024-09-16T00:15:13.575961+0000","last_active":"2024-09-16T00:17:09.579322+0000","last_peered":"2024-09-16T00:17:09.579322+0000","last_clean":"2024-09-16T00:17:09.579322+0000","last_became_active":"2024-09-16T00:15:13.575550+0000","last_became_peered":"2024-09-16T00:15:13.575550+0000","last_unstale":"2024-09-16T00:17:09.579322+0000","last_undegraded":"2024-09-16T00:17:09.579322+0000","last_fullsized":"2024-09-16T00:17:09.579322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:05.515588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577699+0000","last_change":"2024-09-16T00:15:13.583583+0000","last_active":"2024-09-16T00:17:09.577699+0000","last_peered":"2024-09-16T00:17:09.577699+0000","last_clean":"2024-09-16T00:17:09.577699+0000","last_became_active":"2024-09-16T00:15:13.583145+0000","last_became_peered":"2024-09-16T00:15:13.583145+0000","last_unstale":"2024-09-16T00:17:09.577699+0000","last_undegraded":"2024-09-16T00:17:09.577699+0000","last_fullsized":"2024-09-16T00:17:09.577699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:07.320337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.719776+0000","last_change":"2024-09-16T00:17:09.719269+0000","last_active":"2024-09-16T00:17:09.719776+0000","last_peered":"2024-09-16T00:17:09.719776+0000","last_clean":"2024-09-16T00:17:09.719776+0000","last_became_active":"2024-09-16T00:17:09.718887+0000","last_became_peered":"2024-09-16T00:17:09.718887+0000","last_unstale":"2024-09-16T00:17:09.719776+0000","last_undegraded":"2024-09-16T00:17:09.719776+0000","last_fullsized":"2024-09-16T00:17:09.719776+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:14:51.675738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577333+0000","last_change":"2024-09-16T00:15:04.350826+0000","last_active":"2024-09-16T00:17:09.577333+0000","last_peered":"2024-09-16T00:17:09.577333+0000","last_clean":"2024-09-16T00:17:09.577333+0000","last_became_active":"2024-09-16T00:15:04.350294+0000","last_became_peered":"2024-09-16T00:15:04.350294+0000","last_unstale":"2024-09-16T00:17:09.577333+0000","last_undegraded":"2024-09-16T00:17:09.577333+0000","last_fullsized":"2024-09-16T00:17:09.577333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:49.146709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781194+0000","last_change":"2024-09-16T00:15:13.580184+0000","last_active":"2024-09-16T00:17:09.781194+0000","last_peered":"2024-09-16T00:17:09.781194+0000","last_clean":"2024-09-16T00:17:09.781194+0000","last_became_active":"2024-09-16T00:15:13.579849+0000","last_became_peered":"2024-09-16T00:15:13.579849+0000","last_unstale":"2024-09-16T00:17:09.781194+0000","last_undegraded":"2024-09-16T00:17:09.781194+0000","last_fullsized":"2024-09-16T00:17:09.781194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:40.613127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579257+0000","last_change":"2024-09-16T00:15:06.545796+0000","last_active":"2024-09-16T00:17:09.579257+0000","last_peered":"2024-09-16T00:17:09.579257+0000","last_clean":"2024-09-16T00:17:09.579257+0000","last_became_active":"2024-09-16T00:15:06.544815+0000","last_became_peered":"2024-09-16T00:15:06.544815+0000","last_unstale":"2024-09-16T00:17:09.579257+0000","last_undegraded":"2024-09-16T00:17:09.579257+0000","last_fullsized":"2024-09-16T00:17:09.579257+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:51:25.921949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718347+0000","last_change":"2024-09-16T00:15:04.340101+0000","last_active":"2024-09-16T00:17:09.718347+0000","last_peered":"2024-09-16T00:17:09.718347+0000","last_clean":"2024-09-16T00:17:09.718347+0000","last_became_active":"2024-09-16T00:15:04.339788+0000","last_became_peered":"2024-09-16T00:15:04.339788+0000","last_unstale":"2024-09-16T00:17:09.718347+0000","last_undegraded":"2024-09-16T00:17:09.718347+0000","last_fullsized":"2024-09-16T00:17:09.718347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:08.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577952+0000","last_change":"2024-09-16T00:16:09.028910+0000","last_active":"2024-09-16T00:17:09.577952+0000","last_peered":"2024-09-16T00:17:09.577952+0000","last_clean":"2024-09-16T00:17:09.577952+0000","last_became_active":"2024-09-16T00:16:09.023302+0000","last_became_peered":"2024-09-16T00:16:09.023302+0000","last_unstale":"2024-09-16T00:17:09.577952+0000","last_undegraded":"2024-09-16T00:17:09.577952+0000","last_fullsized":"2024-09-16T00:17:09.577952+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:35.293676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581116+0000","last_change":"2024-09-16T00:17:09.581116+0000","last_active":"2024-09-16T00:17:09.581116+0000","last_peered":"2024-09-16T00:17:09.581116+0000","last_clean":"2024-09-16T00:17:09.581116+0000","last_became_active":"2024-09-16T00:17:09.580552+0000","last_became_peered":"2024-09-16T00:17:09.580552+0000","last_unstale":"2024-09-16T00:17:09.581116+0000","last_undegraded":"2024-09-16T00:17:09.581116+0000","last_fullsized":"2024-09-16T00:17:09.581116+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:35.975542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574745+0000","last_change":"2024-09-16T00:15:13.581966+0000","last_active":"2024-09-16T00:17:09.574745+0000","last_peered":"2024-09-16T00:17:09.574745+0000","last_clean":"2024-09-16T00:17:09.574745+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:17:09.574745+0000","last_undegraded":"2024-09-16T00:17:09.574745+0000","last_fullsized":"2024-09-16T00:17:09.574745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.132773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":151,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:34.322964+0000","last_change":"2024-09-16T00:15:04.351765+0000","last_active":"2024-09-16T00:17:34.322964+0000","last_peered":"2024-09-16T00:17:34.322964+0000","last_clean":"2024-09-16T00:17:34.322964+0000","last_became_active":"2024-09-16T00:15:04.351398+0000","last_became_peered":"2024-09-16T00:15:04.351398+0000","last_unstale":"2024-09-16T00:17:34.322964+0000","last_undegraded":"2024-09-16T00:17:34.322964+0000","last_fullsized":"2024-09-16T00:17:34.322964+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:14:28.762657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.583630+0000","last_change":"2024-09-16T00:17:09.583255+0000","last_active":"2024-09-16T00:17:09.583630+0000","last_peered":"2024-09-16T00:17:09.583630+0000","last_clean":"2024-09-16T00:17:09.583630+0000","last_became_active":"2024-09-16T00:17:09.582793+0000","last_became_peered":"2024-09-16T00:17:09.582793+0000","last_unstale":"2024-09-16T00:17:09.583630+0000","last_undegraded":"2024-09-16T00:17:09.583630+0000","last_fullsized":"2024-09-16T00:17:09.583630+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:28.087903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574875+0000","last_change":"2024-09-16T00:15:13.576877+0000","last_active":"2024-09-16T00:17:09.574875+0000","last_peered":"2024-09-16T00:17:09.574875+0000","last_clean":"2024-09-16T00:17:09.574875+0000","last_became_active":"2024-09-16T00:15:13.576480+0000","last_became_peered":"2024-09-16T00:15:13.576480+0000","last_unstale":"2024-09-16T00:17:09.574875+0000","last_undegraded":"2024-09-16T00:17:09.574875+0000","last_fullsized":"2024-09-16T00:17:09.574875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:47.322219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780925+0000","last_change":"2024-09-16T00:15:06.550329+0000","last_active":"2024-09-16T00:17:09.780925+0000","last_peered":"2024-09-16T00:17:09.780925+0000","last_clean":"2024-09-16T00:17:09.780925+0000","last_became_active":"2024-09-16T00:15:06.550008+0000","last_became_peered":"2024-09-16T00:15:06.550008+0000","last_unstale":"2024-09-16T00:17:09.780925+0000","last_undegraded":"2024-09-16T00:17:09.780925+0000","last_fullsized":"2024-09-16T00:17:09.780925+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:38.096126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582169+0000","last_change":"2024-09-16T00:17:09.580839+0000","last_active":"2024-09-16T00:17:09.582169+0000","last_peered":"2024-09-16T00:17:09.582169+0000","last_clean":"2024-09-16T00:17:09.582169+0000","last_became_active":"2024-09-16T00:17:09.580381+0000","last_became_peered":"2024-09-16T00:17:09.580381+0000","last_unstale":"2024-09-16T00:17:09.582169+0000","last_undegraded":"2024-09-16T00:17:09.582169+0000","last_fullsized":"2024-09-16T00:17:09.582169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:35.425316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574792+0000","last_change":"2024-09-16T00:15:06.546713+0000","last_active":"2024-09-16T00:17:09.574792+0000","last_peered":"2024-09-16T00:17:09.574792+0000","last_clean":"2024-09-16T00:17:09.574792+0000","last_became_active":"2024-09-16T00:15:06.546209+0000","last_became_peered":"2024-09-16T00:15:06.546209+0000","last_unstale":"2024-09-16T00:17:09.574792+0000","last_undegraded":"2024-09-16T00:17:09.574792+0000","last_fullsized":"2024-09-16T00:17:09.574792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:43.913748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577575+0000","last_change":"2024-09-16T00:15:06.550490+0000","last_active":"2024-09-16T00:17:09.577575+0000","last_peered":"2024-09-16T00:17:09.577575+0000","last_clean":"2024-09-16T00:17:09.577575+0000","last_became_active":"2024-09-16T00:15:06.549311+0000","last_became_peered":"2024-09-16T00:15:06.549311+0000","last_unstale":"2024-09-16T00:17:09.577575+0000","last_undegraded":"2024-09-16T00:17:09.577575+0000","last_fullsized":"2024-09-16T00:17:09.577575+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:10:01.995357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718148+0000","last_change":"2024-09-16T00:15:06.548115+0000","last_active":"2024-09-16T00:17:09.718148+0000","last_peered":"2024-09-16T00:17:09.718148+0000","last_clean":"2024-09-16T00:17:09.718148+0000","last_became_active":"2024-09-16T00:15:06.547782+0000","last_became_peered":"2024-09-16T00:15:06.547782+0000","last_unstale":"2024-09-16T00:17:09.718148+0000","last_undegraded":"2024-09-16T00:17:09.718148+0000","last_fullsized":"2024-09-16T00:17:09.718148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:53.228222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781260+0000","last_change":"2024-09-16T00:15:06.554396+0000","last_active":"2024-09-16T00:17:09.781260+0000","last_peered":"2024-09-16T00:17:09.781260+0000","last_clean":"2024-09-16T00:17:09.781260+0000","last_became_active":"2024-09-16T00:15:06.554012+0000","last_became_peered":"2024-09-16T00:15:06.554012+0000","last_unstale":"2024-09-16T00:17:09.781260+0000","last_undegraded":"2024-09-16T00:17:09.781260+0000","last_fullsized":"2024-09-16T00:17:09.781260+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:31.464098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575494+0000","last_change":"2024-09-16T00:15:06.549969+0000","last_active":"2024-09-16T00:17:09.575494+0000","last_peered":"2024-09-16T00:17:09.575494+0000","last_clean":"2024-09-16T00:17:09.575494+0000","last_became_active":"2024-09-16T00:15:06.549531+0000","last_became_peered":"2024-09-16T00:15:06.549531+0000","last_unstale":"2024-09-16T00:17:09.575494+0000","last_undegraded":"2024-09-16T00:17:09.575494+0000","last_fullsized":"2024-09-16T00:17:09.575494+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:37.883612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579397+0000","last_change":"2024-09-16T00:15:06.547734+0000","last_active":"2024-09-16T00:17:09.579397+0000","last_peered":"2024-09-16T00:17:09.579397+0000","last_clean":"2024-09-16T00:17:09.579397+0000","last_became_active":"2024-09-16T00:15:06.547400+0000","last_became_peered":"2024-09-16T00:15:06.547400+0000","last_unstale":"2024-09-16T00:17:09.579397+0000","last_undegraded":"2024-09-16T00:17:09.579397+0000","last_fullsized":"2024-09-16T00:17:09.579397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:54.455313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576721+0000","last_change":"2024-09-16T00:15:06.551965+0000","last_active":"2024-09-16T00:17:09.576721+0000","last_peered":"2024-09-16T00:17:09.576721+0000","last_clean":"2024-09-16T00:17:09.576721+0000","last_became_active":"2024-09-16T00:15:06.551602+0000","last_became_peered":"2024-09-16T00:15:06.551602+0000","last_unstale":"2024-09-16T00:17:09.576721+0000","last_undegraded":"2024-09-16T00:17:09.576721+0000","last_fullsized":"2024-09-16T00:17:09.576721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:34.951960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579571+0000","last_change":"2024-09-16T00:15:06.549340+0000","last_active":"2024-09-16T00:17:09.579571+0000","last_peered":"2024-09-16T00:17:09.579571+0000","last_clean":"2024-09-16T00:17:09.579571+0000","last_became_active":"2024-09-16T00:15:06.547424+0000","last_became_peered":"2024-09-16T00:15:06.547424+0000","last_unstale":"2024-09-16T00:17:09.579571+0000","last_undegraded":"2024-09-16T00:17:09.579571+0000","last_fullsized":"2024-09-16T00:17:09.579571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:12.087535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580115+0000","last_change":"2024-09-16T00:15:06.555216+0000","last_active":"2024-09-16T00:17:09.580115+0000","last_peered":"2024-09-16T00:17:09.580115+0000","last_clean":"2024-09-16T00:17:09.580115+0000","last_became_active":"2024-09-16T00:15:06.554896+0000","last_became_peered":"2024-09-16T00:15:06.554896+0000","last_unstale":"2024-09-16T00:17:09.580115+0000","last_undegraded":"2024-09-16T00:17:09.580115+0000","last_fullsized":"2024-09-16T00:17:09.580115+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:20.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717980+0000","last_change":"2024-09-16T00:15:06.549752+0000","last_active":"2024-09-16T00:17:09.717980+0000","last_peered":"2024-09-16T00:17:09.717980+0000","last_clean":"2024-09-16T00:17:09.717980+0000","last_became_active":"2024-09-16T00:15:06.549350+0000","last_became_peered":"2024-09-16T00:15:06.549350+0000","last_unstale":"2024-09-16T00:17:09.717980+0000","last_undegraded":"2024-09-16T00:17:09.717980+0000","last_fullsized":"2024-09-16T00:17:09.717980+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:48.615123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781099+0000","last_change":"2024-09-16T00:15:06.549159+0000","last_active":"2024-09-16T00:17:09.781099+0000","last_peered":"2024-09-16T00:17:09.781099+0000","last_clean":"2024-09-16T00:17:09.781099+0000","last_became_active":"2024-09-16T00:15:06.548408+0000","last_became_peered":"2024-09-16T00:15:06.548408+0000","last_unstale":"2024-09-16T00:17:09.781099+0000","last_undegraded":"2024-09-16T00:17:09.781099+0000","last_fullsized":"2024-09-16T00:17:09.781099+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:23.941809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575144+0000","last_change":"2024-09-16T00:15:06.544091+0000","last_active":"2024-09-16T00:17:09.575144+0000","last_peered":"2024-09-16T00:17:09.575144+0000","last_clean":"2024-09-16T00:17:09.575144+0000","last_became_active":"2024-09-16T00:15:06.543761+0000","last_became_peered":"2024-09-16T00:15:06.543761+0000","last_unstale":"2024-09-16T00:17:09.575144+0000","last_undegraded":"2024-09-16T00:17:09.575144+0000","last_fullsized":"2024-09-16T00:17:09.575144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:36:43.253405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582256+0000","last_change":"2024-09-16T00:17:09.581748+0000","last_active":"2024-09-16T00:17:09.582256+0000","last_peered":"2024-09-16T00:17:09.582256+0000","last_clean":"2024-09-16T00:17:09.582256+0000","last_became_active":"2024-09-16T00:17:09.581207+0000","last_became_peered":"2024-09-16T00:17:09.581207+0000","last_unstale":"2024-09-16T00:17:09.582256+0000","last_undegraded":"2024-09-16T00:17:09.582256+0000","last_fullsized":"2024-09-16T00:17:09.582256+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:21.521929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781344+0000","last_change":"2024-09-16T00:15:06.557162+0000","last_active":"2024-09-16T00:17:09.781344+0000","last_peered":"2024-09-16T00:17:09.781344+0000","last_clean":"2024-09-16T00:17:09.781344+0000","last_became_active":"2024-09-16T00:15:06.556831+0000","last_became_peered":"2024-09-16T00:15:06.556831+0000","last_unstale":"2024-09-16T00:17:09.781344+0000","last_undegraded":"2024-09-16T00:17:09.781344+0000","last_fullsized":"2024-09-16T00:17:09.781344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.236915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579887+0000","last_change":"2024-09-16T00:15:06.556161+0000","last_active":"2024-09-16T00:17:09.579887+0000","last_peered":"2024-09-16T00:17:09.579887+0000","last_clean":"2024-09-16T00:17:09.579887+0000","last_became_active":"2024-09-16T00:15:06.555819+0000","last_became_peered":"2024-09-16T00:15:06.555819+0000","last_unstale":"2024-09-16T00:17:09.579887+0000","last_undegraded":"2024-09-16T00:17:09.579887+0000","last_fullsized":"2024-09-16T00:17:09.579887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:29.929977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.583819+0000","last_change":"2024-09-16T00:17:09.583183+0000","last_active":"2024-09-16T00:17:09.583819+0000","last_peered":"2024-09-16T00:17:09.583819+0000","last_clean":"2024-09-16T00:17:09.583819+0000","last_became_active":"2024-09-16T00:17:09.582663+0000","last_became_peered":"2024-09-16T00:17:09.582663+0000","last_unstale":"2024-09-16T00:17:09.583819+0000","last_undegraded":"2024-09-16T00:17:09.583819+0000","last_fullsized":"2024-09-16T00:17:09.583819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:37.544124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577469+0000","last_change":"2024-09-16T00:15:06.547114+0000","last_active":"2024-09-16T00:17:09.577469+0000","last_peered":"2024-09-16T00:17:09.577469+0000","last_clean":"2024-09-16T00:17:09.577469+0000","last_became_active":"2024-09-16T00:15:06.546786+0000","last_became_peered":"2024-09-16T00:15:06.546786+0000","last_unstale":"2024-09-16T00:17:09.577469+0000","last_undegraded":"2024-09-16T00:17:09.577469+0000","last_fullsized":"2024-09-16T00:17:09.577469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:13:26.925603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717955+0000","last_change":"2024-09-16T00:15:06.541168+0000","last_active":"2024-09-16T00:17:09.717955+0000","last_peered":"2024-09-16T00:17:09.717955+0000","last_clean":"2024-09-16T00:17:09.717955+0000","last_became_active":"2024-09-16T00:15:06.540848+0000","last_became_peered":"2024-09-16T00:15:06.540848+0000","last_unstale":"2024-09-16T00:17:09.717955+0000","last_undegraded":"2024-09-16T00:17:09.717955+0000","last_fullsized":"2024-09-16T00:17:09.717955+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.958600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575843+0000","last_change":"2024-09-16T00:15:06.547429+0000","last_active":"2024-09-16T00:17:09.575843+0000","last_peered":"2024-09-16T00:17:09.575843+0000","last_clean":"2024-09-16T00:17:09.575843+0000","last_became_active":"2024-09-16T00:15:06.547081+0000","last_became_peered":"2024-09-16T00:15:06.547081+0000","last_unstale":"2024-09-16T00:17:09.575843+0000","last_undegraded":"2024-09-16T00:17:09.575843+0000","last_fullsized":"2024-09-16T00:17:09.575843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:43.396313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581339+0000","last_change":"2024-09-16T00:15:06.551811+0000","last_active":"2024-09-16T00:17:09.581339+0000","last_peered":"2024-09-16T00:17:09.581339+0000","last_clean":"2024-09-16T00:17:09.581339+0000","last_became_active":"2024-09-16T00:15:06.551424+0000","last_became_peered":"2024-09-16T00:15:06.551424+0000","last_unstale":"2024-09-16T00:17:09.581339+0000","last_undegraded":"2024-09-16T00:17:09.581339+0000","last_fullsized":"2024-09-16T00:17:09.581339+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.943070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578077+0000","last_change":"2024-09-16T00:15:06.544697+0000","last_active":"2024-09-16T00:17:09.578077+0000","last_peered":"2024-09-16T00:17:09.578077+0000","last_clean":"2024-09-16T00:17:09.578077+0000","last_became_active":"2024-09-16T00:15:06.544182+0000","last_became_peered":"2024-09-16T00:15:06.544182+0000","last_unstale":"2024-09-16T00:17:09.578077+0000","last_undegraded":"2024-09-16T00:17:09.578077+0000","last_fullsized":"2024-09-16T00:17:09.578077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:45.929092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580342+0000","last_change":"2024-09-16T00:15:06.551488+0000","last_active":"2024-09-16T00:17:09.580342+0000","last_peered":"2024-09-16T00:17:09.580342+0000","last_clean":"2024-09-16T00:17:09.580342+0000","last_became_active":"2024-09-16T00:15:06.551171+0000","last_became_peered":"2024-09-16T00:15:06.551171+0000","last_unstale":"2024-09-16T00:17:09.580342+0000","last_undegraded":"2024-09-16T00:17:09.580342+0000","last_fullsized":"2024-09-16T00:17:09.580342+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:23.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579101+0000","last_change":"2024-09-16T00:15:06.544043+0000","last_active":"2024-09-16T00:17:09.579101+0000","last_peered":"2024-09-16T00:17:09.579101+0000","last_clean":"2024-09-16T00:17:09.579101+0000","last_became_active":"2024-09-16T00:15:06.543014+0000","last_became_peered":"2024-09-16T00:15:06.543014+0000","last_unstale":"2024-09-16T00:17:09.579101+0000","last_undegraded":"2024-09-16T00:17:09.579101+0000","last_fullsized":"2024-09-16T00:17:09.579101+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:23.904810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581718+0000","last_change":"2024-09-16T00:15:06.546788+0000","last_active":"2024-09-16T00:17:09.581718+0000","last_peered":"2024-09-16T00:17:09.581718+0000","last_clean":"2024-09-16T00:17:09.581718+0000","last_became_active":"2024-09-16T00:15:06.546333+0000","last_became_peered":"2024-09-16T00:15:06.546333+0000","last_unstale":"2024-09-16T00:17:09.581718+0000","last_undegraded":"2024-09-16T00:17:09.581718+0000","last_fullsized":"2024-09-16T00:17:09.581718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:09.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.719618+0000","last_change":"2024-09-16T00:17:09.719618+0000","last_active":"2024-09-16T00:17:09.719618+0000","last_peered":"2024-09-16T00:17:09.719618+0000","last_clean":"2024-09-16T00:17:09.719618+0000","last_became_active":"2024-09-16T00:17:09.719172+0000","last_became_peered":"2024-09-16T00:17:09.719172+0000","last_unstale":"2024-09-16T00:17:09.719618+0000","last_undegraded":"2024-09-16T00:17:09.719618+0000","last_fullsized":"2024-09-16T00:17:09.719618+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:51.709719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579341+0000","last_change":"2024-09-16T00:15:06.540675+0000","last_active":"2024-09-16T00:17:09.579341+0000","last_peered":"2024-09-16T00:17:09.579341+0000","last_clean":"2024-09-16T00:17:09.579341+0000","last_became_active":"2024-09-16T00:15:06.540333+0000","last_became_peered":"2024-09-16T00:15:06.540333+0000","last_unstale":"2024-09-16T00:17:09.579341+0000","last_undegraded":"2024-09-16T00:17:09.579341+0000","last_fullsized":"2024-09-16T00:17:09.579341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:17.166987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575821+0000","last_change":"2024-09-16T00:15:06.548758+0000","last_active":"2024-09-16T00:17:09.575821+0000","last_peered":"2024-09-16T00:17:09.575821+0000","last_clean":"2024-09-16T00:17:09.575821+0000","last_became_active":"2024-09-16T00:15:06.548397+0000","last_became_peered":"2024-09-16T00:15:06.548397+0000","last_unstale":"2024-09-16T00:17:09.575821+0000","last_undegraded":"2024-09-16T00:17:09.575821+0000","last_fullsized":"2024-09-16T00:17:09.575821+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:50.320170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576644+0000","last_change":"2024-09-16T00:15:06.554898+0000","last_active":"2024-09-16T00:17:09.576644+0000","last_peered":"2024-09-16T00:17:09.576644+0000","last_clean":"2024-09-16T00:17:09.576644+0000","last_became_active":"2024-09-16T00:15:06.553005+0000","last_became_peered":"2024-09-16T00:15:06.553005+0000","last_unstale":"2024-09-16T00:17:09.576644+0000","last_undegraded":"2024-09-16T00:17:09.576644+0000","last_fullsized":"2024-09-16T00:17:09.576644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:29.630911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577231+0000","last_change":"2024-09-16T00:15:06.547265+0000","last_active":"2024-09-16T00:17:09.577231+0000","last_peered":"2024-09-16T00:17:09.577231+0000","last_clean":"2024-09-16T00:17:09.577231+0000","last_became_active":"2024-09-16T00:15:06.546938+0000","last_became_peered":"2024-09-16T00:15:06.546938+0000","last_unstale":"2024-09-16T00:17:09.577231+0000","last_undegraded":"2024-09-16T00:17:09.577231+0000","last_fullsized":"2024-09-16T00:17:09.577231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:40.247343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781308+0000","last_change":"2024-09-16T00:15:06.556735+0000","last_active":"2024-09-16T00:17:09.781308+0000","last_peered":"2024-09-16T00:17:09.781308+0000","last_clean":"2024-09-16T00:17:09.781308+0000","last_became_active":"2024-09-16T00:15:06.556324+0000","last_became_peered":"2024-09-16T00:15:06.556324+0000","last_unstale":"2024-09-16T00:17:09.781308+0000","last_undegraded":"2024-09-16T00:17:09.781308+0000","last_fullsized":"2024-09-16T00:17:09.781308+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:25.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":76311,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":654,"num_read_kb":397,"num_write":182,"num_write_kb":135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":62107,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":228933,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249414,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579948,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":517057,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808880,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36468,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708684,"statfs":{"total":95995035648,"available":95957692416,"internally_reserved":0,"allocated":1150976,"data_stored":1177026,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972406,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":521389,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201339,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":516631,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430272,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1060864,"data_stored":1113658,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626502,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":512469,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855435,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1069056,"data_stored":1108319,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986256,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1044480,"data_stored":590402,"data_compressed":39297,"data_compressed_allocated":53248,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215188,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":512663,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444122,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1007616,"data_stored":513147,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607647,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":584735,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1015808,"data_stored":513857,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":72356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":73927,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":74279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T00:17:38.769 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T00:17:38.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.936+0000 7f40e9208640 1 -- 172.21.15.33:0/2298805389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40e4102ef0 msgr2=0x7f40e410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.936+0000 7f40e9208640 1 --2- 172.21.15.33:0/2298805389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40e4102ef0 0x7f40e410da40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f40dc009ca0 tx=0x7f40dc02a950 comp rx=0 tx=0).stop 2024-09-16T00:17:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.936+0000 7f40e9208640 1 -- 172.21.15.33:0/2298805389 shutdown_connections 2024-09-16T00:17:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.936+0000 7f40e9208640 1 --2- 172.21.15.33:0/2298805389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40e4102ef0 0x7f40e410da40 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.936+0000 7f40e9208640 1 --2- 172.21.15.33:0/2298805389 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40e4102570 0x7f40e41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.936+0000 7f40e9208640 1 --2- 172.21.15.33:0/2298805389 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e4100e20 0x7f40e4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.936+0000 7f40e9208640 1 -- 172.21.15.33:0/2298805389 >> 172.21.15.33:0/2298805389 conn(0x7f40e40fc7d0 msgr2=0x7f40e40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.936+0000 7f40e9208640 1 -- 172.21.15.33:0/2298805389 shutdown_connections 2024-09-16T00:17:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.936+0000 7f40e9208640 1 -- 172.21.15.33:0/2298805389 wait complete. 2024-09-16T00:17:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.937+0000 7f40e9208640 1 Processor -- start 2024-09-16T00:17:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.937+0000 7f40e9208640 1 -- start start 2024-09-16T00:17:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.937+0000 7f40e9208640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40e4100e20 0x7f40e41970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.937+0000 7f40e9208640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40e4102570 0x7f40e41975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.937+0000 7f40e9208640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e4102ef0 0x7f40e4197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e9208640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f40e4110f80 con 0x7f40e4100e20 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e9208640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f40e4110e00 con 0x7f40e4102570 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e9208640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f40e4111100 con 0x7f40e4102ef0 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40e4100e20 0x7f40e41970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40e4100e20 0x7f40e41970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36338/0 (socket says 172.21.15.33:36338) 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e2d9d640 1 -- 172.21.15.33:0/708210993 learned_addr learned my addr 172.21.15.33:0/708210993 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e359e640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e4102ef0 0x7f40e4197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e259c640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40e4102570 0x7f40e41975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e2d9d640 1 -- 172.21.15.33:0/708210993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e4102ef0 msgr2=0x7f40e4197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e2d9d640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e4102ef0 0x7f40e4197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e2d9d640 1 -- 172.21.15.33:0/708210993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40e4102570 msgr2=0x7f40e41975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e2d9d640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40e4102570 0x7f40e41975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e2d9d640 1 -- 172.21.15.33:0/708210993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e41a6920 con 0x7f40e4100e20 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e259c640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40e4102570 0x7f40e41975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e2d9d640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40e4100e20 0x7f40e41970b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f40d400c970 tx=0x7f40d400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40cbfff640 1 -- 172.21.15.33:0/708210993 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d4007bb0 con 0x7f40e4100e20 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40cbfff640 1 -- 172.21.15.33:0/708210993 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40d4007d50 con 0x7f40e4100e20 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.938+0000 7f40e9208640 1 -- 172.21.15.33:0/708210993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e41a6c10 con 0x7f40e4100e20 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.939+0000 7f40cbfff640 1 -- 172.21.15.33:0/708210993 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d400f450 con 0x7f40e4100e20 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.939+0000 7f40e9208640 1 -- 172.21.15.33:0/708210993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e41a7100 con 0x7f40e4100e20 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.941+0000 7f40cbfff640 1 -- 172.21.15.33:0/708210993 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f40d4005020 con 0x7f40e4100e20 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.941+0000 7f40eb5e1740 1 Processor -- start 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.941+0000 7f40eb5e1740 1 -- start start 2024-09-16T00:17:38.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.941+0000 7f40eb5e1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c35b60 0x55cc65c35f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.941+0000 7f40eb5e1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c36480 0x55cc65c40800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.941+0000 7f40eb5e1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c40d40 0x55cc65c43130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.941+0000 7f40eb5e1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cc65b9b280 con 0x55cc65c35b60 2024-09-16T00:17:38.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.941+0000 7f40eb5e1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55cc65b42b10 con 0x55cc65c40d40 2024-09-16T00:17:38.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.941+0000 7f40eb5e1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cc65b97240 con 0x55cc65c36480 2024-09-16T00:17:38.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40e359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c35b60 0x55cc65c35f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40e359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c35b60 0x55cc65c35f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36350/0 (socket says 172.21.15.33:36350) 2024-09-16T00:17:38.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40e359e640 1 -- 172.21.15.33:0/1124093991 learned_addr learned my addr 172.21.15.33:0/1124093991 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:38.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40e2d9d640 1 --2- 172.21.15.33:0/1124093991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c36480 0x55cc65c40800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40e259c640 1 --2- 172.21.15.33:0/1124093991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c40d40 0x55cc65c43130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40cbfff640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40cc0a32d0 0x7f40cc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40e359e640 1 -- 172.21.15.33:0/1124093991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c36480 msgr2=0x55cc65c40800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40e359e640 1 --2- 172.21.15.33:0/1124093991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c36480 0x55cc65c40800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40e359e640 1 -- 172.21.15.33:0/1124093991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c40d40 msgr2=0x55cc65c43130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40e359e640 1 --2- 172.21.15.33:0/1124093991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c40d40 0x55cc65c43130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40e359e640 1 -- 172.21.15.33:0/1124093991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc65b4f1f0 con 0x55cc65c35b60 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40cbfff640 1 -- 172.21.15.33:0/708210993 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f40d40cfd10 con 0x7f40e4100e20 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40e359e640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40cc0a32d0 0x7f40cc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.942+0000 7f40e359e640 1 --2- 172.21.15.33:0/1124093991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c35b60 0x55cc65c35f40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f40dc035000 tx=0x7f40dc009980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95ab7c9e048394c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.943+0000 7f40c97fa640 1 -- 172.21.15.33:0/1124093991 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40dc034040 con 0x55cc65c35b60 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.943+0000 7f40c97fa640 1 -- 172.21.15.33:0/1124093991 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40dc004520 con 0x55cc65c35b60 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.943+0000 7f40e359e640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40cc0a32d0 0x7f40cc0a5790 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f40dc009ca0 tx=0x7f40dc04a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.943+0000 7f40c97fa640 1 -- 172.21.15.33:0/1124093991 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40dc0469b0 con 0x55cc65c35b60 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.943+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1124093991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c35b60 msgr2=0x55cc65c35f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:38.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.943+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1124093991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c35b60 0x55cc65c35f40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f40dc035000 tx=0x7f40dc009980 comp rx=0 tx=0).stop 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1124093991 shutdown_connections 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1124093991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c40d40 0x55cc65c43130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1124093991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c36480 0x55cc65c40800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1124093991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c35b60 0x55cc65c35f40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1124093991 >> 172.21.15.33:0/1124093991 conn(0x55cc65c320f0 msgr2=0x55cc65c324d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1124093991 shutdown_connections 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1124093991 wait complete. 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 -- start start 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 Processor -- start 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c35b60 0x55cc65c3dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c36480 0x55cc65c3e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c40d40 0x55cc65c3e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cc65c43960 con 0x55cc65c36480 2024-09-16T00:17:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55cc65b55f10 con 0x55cc65c35b60 2024-09-16T00:17:38.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cc65c43ae0 con 0x55cc65c40d40 2024-09-16T00:17:38.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40eb5e1740 10 client.0 _ll_register_callbacks cb 0x55cc65b1ea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:38.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.944+0000 7f40e259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c36480 0x55cc65c3e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40e259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c36480 0x55cc65c3e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36352/0 (socket says 172.21.15.33:36352) 2024-09-16T00:17:38.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40e259c640 1 -- 172.21.15.33:0/1435767346 learned_addr learned my addr 172.21.15.33:0/1435767346 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:38.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40e359e640 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c40d40 0x55cc65c3e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40e2d9d640 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c35b60 0x55cc65c3dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40e259c640 1 -- 172.21.15.33:0/1435767346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c40d40 msgr2=0x55cc65c3e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:38.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40e259c640 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c40d40 0x55cc65c3e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40e259c640 1 -- 172.21.15.33:0/1435767346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c35b60 msgr2=0x55cc65c3dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:38.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40e259c640 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c35b60 0x55cc65c3dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40e259c640 1 -- 172.21.15.33:0/1435767346 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc6590c6a0 con 0x55cc65c36480 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40e259c640 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c36480 0x55cc65c3e120 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f40d8003b60 tx=0x7f40d80117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f8d105f6e4e423 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40c6ff5640 1 -- 172.21.15.33:0/1435767346 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d8003e50 con 0x55cc65c36480 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40c6ff5640 1 -- 172.21.15.33:0/1435767346 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f40d8005020 con 0x55cc65c36480 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1435767346 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40dc004520 con 0x55cc65c36480 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40c6ff5640 1 -- 172.21.15.33:0/1435767346 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f40d8005220 con 0x55cc65c36480 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40eb5e1740 10 client.15336 fetch_fsmap learned FSMap version 26 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.945+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1435767346 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cc65c4f590 con 0x55cc65c36480 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.946+0000 7f40c6ff5640 1 -- 172.21.15.33:0/1435767346 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40dc004520 con 0x55cc65c36480 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.946+0000 7f40c6ff5640 1 -- 172.21.15.33:0/1435767346 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f40d801b640 con 0x55cc65c36480 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.946+0000 7f40eb5e1740 10 client.15336 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.946+0000 7f40eb5e1740 10 client.15336 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:17:38.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.946+0000 7f40eb5e1740 20 client.15336 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.946+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cc65c50a50 0x55cc65c52ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.946+0000 7f40eb5e1740 4 client.15336 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:17:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.946+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1435767346 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc65ba87f0 con 0x55cc65c50a50 2024-09-16T00:17:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.946+0000 7f40e2d9d640 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cc65c50a50 0x55cc65c52ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.947+0000 7f40c6ff5640 10 client.15336 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.947+0000 7f40e2d9d640 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cc65c50a50 0x55cc65c52ea0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97d94e9598099f04 server_cookie=4229a2a66e454072 in_seq=0 out_seq=0 2024-09-16T00:17:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.947+0000 7f40c6ff5640 1 -- 172.21.15.33:0/1435767346 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cc65ba87f0 con 0x55cc65c50a50 2024-09-16T00:17:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.947+0000 7f40c6ff5640 10 client.15336 handle_command_reply: tid=0 2024-09-16T00:17:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.947+0000 7f40eb5e1740 1 client.15336 shutdown 2024-09-16T00:17:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.948+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1435767346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c36480 msgr2=0x55cc65c3e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.948+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c36480 0x55cc65c3e120 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f40d8003b60 tx=0x7f40d80117b0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.948+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1435767346 shutdown_connections 2024-09-16T00:17:38.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.948+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cc65c50a50 0x55cc65c52ea0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.948+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c40d40 0x55cc65c3e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.948+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c36480 0x55cc65c3e120 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.948+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1435767346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c35b60 0x55cc65c3dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.948+0000 7f40c6ff5640 0 client.15336 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:38.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.948+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1435767346 >> 172.21.15.33:0/1435767346 conn(0x55cc65c320f0 msgr2=0x55cc65c37dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:38.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.948+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1435767346 shutdown_connections 2024-09-16T00:17:38.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.948+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1435767346 wait complete. 2024-09-16T00:17:38.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.948+0000 7f40eb5e1740 20 client.15336 trim_cache size 0 max 16384 2024-09-16T00:17:38.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.973+0000 7f40eb5e1740 1 Processor -- start 2024-09-16T00:17:38.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40eb5e1740 1 -- start start 2024-09-16T00:17:38.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40eb5e1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c35b60 0x55cc65c3cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40eb5e1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c36480 0x55cc65c385c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40eb5e1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 0x55cc65c587b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40eb5e1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f40e41b3b50 con 0x55cc65c40d40 2024-09-16T00:17:38.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40eb5e1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f40e41b1420 con 0x55cc65c36480 2024-09-16T00:17:38.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40eb5e1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f40e41af270 con 0x55cc65c35b60 2024-09-16T00:17:38.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40e359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 0x55cc65c587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40e359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 0x55cc65c587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36366/0 (socket says 172.21.15.33:36366) 2024-09-16T00:17:38.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40e359e640 1 -- 172.21.15.33:0/1203078564 learned_addr learned my addr 172.21.15.33:0/1203078564 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:38.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40e2d9d640 1 --2- 172.21.15.33:0/1203078564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c35b60 0x55cc65c3cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40e259c640 1 --2- 172.21.15.33:0/1203078564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c36480 0x55cc65c385c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40e359e640 1 -- 172.21.15.33:0/1203078564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c35b60 msgr2=0x55cc65c3cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:38.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40e359e640 1 --2- 172.21.15.33:0/1203078564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c35b60 0x55cc65c3cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40e359e640 1 -- 172.21.15.33:0/1203078564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c36480 msgr2=0x55cc65c385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:38.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40e359e640 1 --2- 172.21.15.33:0/1203078564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c36480 0x55cc65c385c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.974+0000 7f40e359e640 1 -- 172.21.15.33:0/1203078564 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc65c331a0 con 0x55cc65c40d40 2024-09-16T00:17:38.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.975+0000 7f40e259c640 1 --2- 172.21.15.33:0/1203078564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c36480 0x55cc65c385c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:17:38.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.975+0000 7f40e359e640 1 --2- 172.21.15.33:0/1203078564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 0x55cc65c587b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0095c0 tx=0x7f40dc009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a11a1298e912ee3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:38.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.975+0000 7f40c8ff9640 1 -- 172.21.15.33:0/1203078564 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40dc02f5c0 con 0x55cc65c40d40 2024-09-16T00:17:38.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.975+0000 7f40c8ff9640 1 -- 172.21.15.33:0/1203078564 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40dc046c50 con 0x55cc65c40d40 2024-09-16T00:17:38.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.975+0000 7f40c8ff9640 1 -- 172.21.15.33:0/1203078564 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40dc0044b0 con 0x55cc65c40d40 2024-09-16T00:17:38.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.976+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1203078564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 msgr2=0x55cc65c587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:38.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.976+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1203078564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 0x55cc65c587b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0095c0 tx=0x7f40dc009a20 comp rx=0 tx=0).stop 2024-09-16T00:17:38.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.976+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1203078564 shutdown_connections 2024-09-16T00:17:38.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.976+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1203078564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 0x55cc65c587b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.976+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1203078564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c36480 0x55cc65c385c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.976+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/1203078564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c35b60 0x55cc65c3cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.976+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1203078564 >> 172.21.15.33:0/1203078564 conn(0x55cc65c320f0 msgr2=0x55cc65c3fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:38.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.976+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1203078564 shutdown_connections 2024-09-16T00:17:38.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.977+0000 7f40eb5e1740 1 -- 172.21.15.33:0/1203078564 wait complete. 2024-09-16T00:17:38.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.977+0000 7f40eb5e1740 1 -- start start 2024-09-16T00:17:38.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.977+0000 7f40eb5e1740 1 Processor -- start 2024-09-16T00:17:38.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.978+0000 7f40eb5e1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c35b60 0x55cc65c49220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.978+0000 7f40eb5e1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c36480 0x55cc65c45d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.978+0000 7f40eb5e1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 0x55cc65c46240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.978+0000 7f40eb5e1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f40e41af8e0 con 0x55cc65c40d40 2024-09-16T00:17:38.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.978+0000 7f40eb5e1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f40e41b0d50 con 0x55cc65c35b60 2024-09-16T00:17:38.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.978+0000 7f40eb5e1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f40e41affb0 con 0x55cc65c36480 2024-09-16T00:17:38.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.978+0000 7f40e359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 0x55cc65c46240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.978+0000 7f40e359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 0x55cc65c46240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36378/0 (socket says 172.21.15.33:36378) 2024-09-16T00:17:38.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.978+0000 7f40e359e640 1 -- 172.21.15.33:0/3003649715 learned_addr learned my addr 172.21.15.33:0/3003649715 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:38.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.978+0000 7f40e2d9d640 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c35b60 0x55cc65c49220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.978+0000 7f40e259c640 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c36480 0x55cc65c45d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.978+0000 7f40eb5e1740 10 client.0 _ll_register_callbacks cb 0x7f40d80040e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:38.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40e359e640 1 -- 172.21.15.33:0/3003649715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c36480 msgr2=0x55cc65c45d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:38.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40e359e640 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c36480 0x55cc65c45d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40e359e640 1 -- 172.21.15.33:0/3003649715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c35b60 msgr2=0x55cc65c49220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:38.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40e359e640 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c35b60 0x55cc65c49220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40e359e640 1 -- 172.21.15.33:0/3003649715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc65c2c680 con 0x55cc65c40d40 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40e359e640 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 0x55cc65c46240 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f40dc030040 tx=0x7f40dc02ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b4ed6f3bcec3c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40c6ff5640 1 -- 172.21.15.33:0/3003649715 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40dc060070 con 0x55cc65c40d40 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40c6ff5640 1 -- 172.21.15.33:0/3003649715 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f40dc046770 con 0x55cc65c40d40 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40eb5e1740 1 -- 172.21.15.33:0/3003649715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40dc046c50 con 0x55cc65c40d40 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40c6ff5640 1 -- 172.21.15.33:0/3003649715 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f40dc046970 con 0x55cc65c40d40 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40eb5e1740 10 client.15342 fetch_fsmap learned FSMap version 26 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.979+0000 7f40eb5e1740 1 -- 172.21.15.33:0/3003649715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cc65cbc390 con 0x55cc65c40d40 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.980+0000 7f40c6ff5640 1 -- 172.21.15.33:0/3003649715 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40dc046c50 con 0x55cc65c40d40 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.980+0000 7f40c6ff5640 1 -- 172.21.15.33:0/3003649715 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f40dc0306b0 con 0x55cc65c40d40 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.980+0000 7f40eb5e1740 10 client.15342 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:17:38.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.980+0000 7f40eb5e1740 10 client.15342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:17:38.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.980+0000 7f40eb5e1740 20 client.15342 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:38.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.980+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cc65cbc610 0x55cc65cbea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:38.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.980+0000 7f40eb5e1740 4 client.15342 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:17:38.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.980+0000 7f40eb5e1740 1 -- 172.21.15.33:0/3003649715 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cc65ba2d00 con 0x55cc65cbc610 2024-09-16T00:17:38.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.980+0000 7f40e2d9d640 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cc65cbc610 0x55cc65cbea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:38.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.981+0000 7f40e2d9d640 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cc65cbc610 0x55cc65cbea00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7413814b88cec1a2 server_cookie=8bbf40d1c3764bf2 in_seq=0 out_seq=0 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.981+0000 7f40c6ff5640 10 client.15342 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40c6ff5640 1 -- 172.21.15.33:0/3003649715 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cc65ba2d00 con 0x55cc65cbc610 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40c6ff5640 10 client.15342 handle_command_reply: tid=0 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40eb5e1740 1 client.15342 shutdown 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40eb5e1740 1 -- 172.21.15.33:0/3003649715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 msgr2=0x55cc65c46240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 0x55cc65c46240 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f40dc030040 tx=0x7f40dc02ff00 comp rx=0 tx=0).stop 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40eb5e1740 1 -- 172.21.15.33:0/3003649715 shutdown_connections 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cc65cbc610 0x55cc65cbea00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc65c40d40 0x55cc65c46240 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc65c36480 0x55cc65c45d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40eb5e1740 1 --2- 172.21.15.33:0/3003649715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc65c35b60 0x55cc65c49220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40eb5e1740 1 -- 172.21.15.33:0/3003649715 >> 172.21.15.33:0/3003649715 conn(0x55cc65c320f0 msgr2=0x55cc65c30a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:38.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40c6ff5640 0 client.15342 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:38.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40eb5e1740 1 -- 172.21.15.33:0/3003649715 shutdown_connections 2024-09-16T00:17:38.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40eb5e1740 1 -- 172.21.15.33:0/3003649715 wait complete. 2024-09-16T00:17:38.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.982+0000 7f40eb5e1740 20 client.15342 trim_cache size 0 max 16384 2024-09-16T00:17:38.986 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:17:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:17:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "96a33d0c-27bf-4b98-aed1-121ec5dad828", 2024-09-16T00:17:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:17:38.986 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:17:38.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.983+0000 7f40e9208640 1 -- 172.21.15.33:0/708210993 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40cc0a32d0 msgr2=0x7f40cc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:38.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.983+0000 7f40e9208640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40cc0a32d0 0x7f40cc0a5790 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f40dc009ca0 tx=0x7f40dc04a000 comp rx=0 tx=0).stop 2024-09-16T00:17:38.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.983+0000 7f40e9208640 1 -- 172.21.15.33:0/708210993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40e4100e20 msgr2=0x7f40e41970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:38.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.983+0000 7f40e9208640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40e4100e20 0x7f40e41970b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f40d400c970 tx=0x7f40d400ce30 comp rx=0 tx=0).stop 2024-09-16T00:17:38.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.983+0000 7f40e9208640 1 -- 172.21.15.33:0/708210993 shutdown_connections 2024-09-16T00:17:38.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.983+0000 7f40e9208640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40cc0a32d0 0x7f40cc0a5790 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.983+0000 7f40e9208640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e4102ef0 0x7f40e4197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.983+0000 7f40e9208640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40e4102570 0x7f40e41975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.983+0000 7f40e9208640 1 --2- 172.21.15.33:0/708210993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40e4100e20 0x7f40e41970b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:38.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.983+0000 7f40e9208640 1 -- 172.21.15.33:0/708210993 >> 172.21.15.33:0/708210993 conn(0x7f40e40fc7d0 msgr2=0x7f40e4105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:38.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.983+0000 7f40e9208640 1 -- 172.21.15.33:0/708210993 shutdown_connections 2024-09-16T00:17:38.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:38.983+0000 7f40e9208640 1 -- 172.21.15.33:0/708210993 wait complete. 2024-09-16T00:17:38.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:38 smithi033 ceph-mon[31243]: pgmap v365: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-09-16T00:17:38.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:38 smithi033 ceph-mon[31243]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:39.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:38 smithi083 ceph-mon[35242]: pgmap v365: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-09-16T00:17:39.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:38 smithi083 ceph-mon[35242]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:39.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:38 smithi060 ceph-mon[35185]: pgmap v365: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-09-16T00:17:39.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:38 smithi060 ceph-mon[35185]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:39.804 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:17:39.808 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:40.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:39 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:17:40.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:39 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:17:40.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:39 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:17:40.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:39 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:17:40.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:39 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:17:40.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:39 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:17:40.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:39 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:17:40.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:39 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:17:40.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:39 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:17:40.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:39 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:17:40.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:39 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:17:40.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:39 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:17:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.308+0000 7fc4b5533640 1 -- 172.21.15.33:0/2127876252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4b0101de0 msgr2=0x7fc4b01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.308+0000 7fc4b5533640 1 --2- 172.21.15.33:0/2127876252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4b0101de0 0x7fc4b01021c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009ca0 tx=0x7fc4a002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:17:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.308+0000 7fc4b5533640 1 -- 172.21.15.33:0/2127876252 shutdown_connections 2024-09-16T00:17:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.308+0000 7fc4b5533640 1 --2- 172.21.15.33:0/2127876252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4b0106f10 0x7fc4b010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.308+0000 7fc4b5533640 1 --2- 172.21.15.33:0/2127876252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0102790 0x7fc4b01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.308+0000 7fc4b5533640 1 --2- 172.21.15.33:0/2127876252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4b0101de0 0x7fc4b01021c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.308+0000 7fc4b5533640 1 -- 172.21.15.33:0/2127876252 >> 172.21.15.33:0/2127876252 conn(0x7fc4b00fd8c0 msgr2=0x7fc4b00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.308+0000 7fc4b5533640 1 -- 172.21.15.33:0/2127876252 shutdown_connections 2024-09-16T00:17:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.308+0000 7fc4b5533640 1 -- 172.21.15.33:0/2127876252 wait complete. 2024-09-16T00:17:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4b5533640 1 Processor -- start 2024-09-16T00:17:40.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4b5533640 1 -- start start 2024-09-16T00:17:40.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4b5533640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4b0101de0 0x7fc4b0198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:40.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4b5533640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0102790 0x7fc4b0198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:40.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4b5533640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4b0106f10 0x7fc4b019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:40.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4b5533640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0111f70 con 0x7fc4b0106f10 2024-09-16T00:17:40.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4b5533640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0111df0 con 0x7fc4b0102790 2024-09-16T00:17:40.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4b5533640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01120f0 con 0x7fc4b0101de0 2024-09-16T00:17:40.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4af7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4b0106f10 0x7fc4b019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:40.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4af7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4b0106f10 0x7fc4b019cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58906/0 (socket says 172.21.15.33:58906) 2024-09-16T00:17:40.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4af7fe640 1 -- 172.21.15.33:0/1014490764 learned_addr learned my addr 172.21.15.33:0/1014490764 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:40.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4aeffd640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4b0101de0 0x7fc4b0198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:40.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.309+0000 7fc4ae7fc640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0102790 0x7fc4b0198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:40.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.310+0000 7fc4af7fe640 1 -- 172.21.15.33:0/1014490764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4b0101de0 msgr2=0x7fc4b0198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:40.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.310+0000 7fc4af7fe640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4b0101de0 0x7fc4b0198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:40.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.310+0000 7fc4af7fe640 1 -- 172.21.15.33:0/1014490764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0102790 msgr2=0x7fc4b0198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:40.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.310+0000 7fc4af7fe640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0102790 0x7fc4b0198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:40.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.310+0000 7fc4af7fe640 1 -- 172.21.15.33:0/1014490764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b019d3e0 con 0x7fc4b0106f10 2024-09-16T00:17:40.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.310+0000 7fc4af7fe640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4b0106f10 0x7fc4b019cd00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc4a800d750 tx=0x7fc4a800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:40.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.310+0000 7fc4a7fff640 1 -- 172.21.15.33:0/1014490764 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8017070 con 0x7fc4b0106f10 2024-09-16T00:17:40.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.310+0000 7fc4a7fff640 1 -- 172.21.15.33:0/1014490764 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a8005500 con 0x7fc4b0106f10 2024-09-16T00:17:40.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.310+0000 7fc4b5533640 1 -- 172.21.15.33:0/1014490764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b019d6d0 con 0x7fc4b0106f10 2024-09-16T00:17:40.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.310+0000 7fc4a7fff640 1 -- 172.21.15.33:0/1014490764 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8004d90 con 0x7fc4b0106f10 2024-09-16T00:17:40.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.310+0000 7fc4b5533640 1 -- 172.21.15.33:0/1014490764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b01a4fb0 con 0x7fc4b0106f10 2024-09-16T00:17:40.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.311+0000 7fc4b5533640 1 -- 172.21.15.33:0/1014490764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4b01021c0 con 0x7fc4b0106f10 2024-09-16T00:17:40.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.312+0000 7fc4a7fff640 1 -- 172.21.15.33:0/1014490764 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fc4a8004f30 con 0x7fc4b0106f10 2024-09-16T00:17:40.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.313+0000 7fc4a7fff640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc49c0a32d0 0x7fc49c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:40.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.313+0000 7fc4aeffd640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc49c0a32d0 0x7fc49c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:40.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.313+0000 7fc4a7fff640 1 -- 172.21.15.33:0/1014490764 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc4a80cfc60 con 0x7fc4b0106f10 2024-09-16T00:17:40.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.314+0000 7fc4aeffd640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc49c0a32d0 0x7fc49c0a5790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0006010 tx=0x7fc4a0009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:40.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.316+0000 7fc4a7fff640 1 -- 172.21.15.33:0/1014490764 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc4a809c3a0 con 0x7fc4b0106f10 2024-09-16T00:17:40.358 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.498+0000 7fc4b5533640 1 -- 172.21.15.33:0/1014490764 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc4b0062630 con 0x7fc49c0a32d0 2024-09-16T00:17:40.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.506+0000 7fc4a7fff640 1 -- 172.21.15.33:0/1014490764 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340771 (secure 0 0 0) 0x7fc4b0062630 con 0x7fc49c0a32d0 2024-09-16T00:17:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:40.511 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-09-16T00:17:40.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.510+0000 7fc4b5533640 1 -- 172.21.15.33:0/1014490764 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc49c0a32d0 msgr2=0x7fc49c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:40.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.510+0000 7fc4b5533640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc49c0a32d0 0x7fc49c0a5790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0006010 tx=0x7fc4a0009360 comp rx=0 tx=0).stop 2024-09-16T00:17:40.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.510+0000 7fc4b5533640 1 -- 172.21.15.33:0/1014490764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4b0106f10 msgr2=0x7fc4b019cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:40.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.510+0000 7fc4b5533640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4b0106f10 0x7fc4b019cd00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc4a800d750 tx=0x7fc4a800dc10 comp rx=0 tx=0).stop 2024-09-16T00:17:40.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.510+0000 7fc4b5533640 1 -- 172.21.15.33:0/1014490764 shutdown_connections 2024-09-16T00:17:40.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.510+0000 7fc4b5533640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc49c0a32d0 0x7fc49c0a5790 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:40.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.510+0000 7fc4b5533640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4b0106f10 0x7fc4b019cd00 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:40.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.510+0000 7fc4b5533640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0102790 0x7fc4b0198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:40.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.510+0000 7fc4b5533640 1 --2- 172.21.15.33:0/1014490764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4b0101de0 0x7fc4b0198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:40.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.510+0000 7fc4b5533640 1 -- 172.21.15.33:0/1014490764 >> 172.21.15.33:0/1014490764 conn(0x7fc4b00fd8c0 msgr2=0x7fc4b00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:40.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.510+0000 7fc4b5533640 1 -- 172.21.15.33:0/1014490764 shutdown_connections 2024-09-16T00:17:40.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:40.510+0000 7fc4b5533640 1 -- 172.21.15.33:0/1014490764 wait complete. 2024-09-16T00:17:41.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:40 smithi033 ceph-mon[31243]: pgmap v366: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-16T00:17:41.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:40 smithi033 ceph-mon[31243]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:41.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:40 smithi083 ceph-mon[35242]: pgmap v366: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-16T00:17:41.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:40 smithi083 ceph-mon[35242]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:41.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:40 smithi060 ceph-mon[35185]: pgmap v366: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-16T00:17:41.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:40 smithi060 ceph-mon[35185]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:41.238 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-09-16T00:17:39.672521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666913,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":795,"num_read_kb":516,"num_write":365,"num_write_kb":3075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1242843,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386388,"kb_used_data":12144,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555436,"statfs":{"total":1151940427776,"available":1151544766464,"internally_reserved":0,"allocated":12435456,"data_stored":8181353,"data_compressed":471663,"data_compressed_allocated":638976,"data_compressed_original":2809856,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8987,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":79,"num_read_kb":51,"num_write":10,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003100"},"pg_stats":[{"pgid":"2.1f","version":"86'9","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578728+0000","last_change":"2024-09-16T00:16:09.034677+0000","last_active":"2024-09-16T00:17:09.578728+0000","last_peered":"2024-09-16T00:17:09.578728+0000","last_clean":"2024-09-16T00:17:09.578728+0000","last_became_active":"2024-09-16T00:16:09.024755+0000","last_became_peered":"2024-09-16T00:16:09.024755+0000","last_unstale":"2024-09-16T00:17:09.578728+0000","last_undegraded":"2024-09-16T00:17:09.578728+0000","last_fullsized":"2024-09-16T00:17:09.578728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:09.012210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579433+0000","last_change":"2024-09-16T00:15:06.547094+0000","last_active":"2024-09-16T00:17:09.579433+0000","last_peered":"2024-09-16T00:17:09.579433+0000","last_clean":"2024-09-16T00:17:09.579433+0000","last_became_active":"2024-09-16T00:15:06.542764+0000","last_became_peered":"2024-09-16T00:15:06.542764+0000","last_unstale":"2024-09-16T00:17:09.579433+0000","last_undegraded":"2024-09-16T00:17:09.579433+0000","last_fullsized":"2024-09-16T00:17:09.579433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:04:37.449577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574697+0000","last_change":"2024-09-16T00:15:13.578400+0000","last_active":"2024-09-16T00:17:09.574697+0000","last_peered":"2024-09-16T00:17:09.574697+0000","last_clean":"2024-09-16T00:17:09.574697+0000","last_became_active":"2024-09-16T00:15:13.577787+0000","last_became_peered":"2024-09-16T00:15:13.577787+0000","last_unstale":"2024-09-16T00:17:09.574697+0000","last_undegraded":"2024-09-16T00:17:09.574697+0000","last_fullsized":"2024-09-16T00:17:09.574697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:19:59.194152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579738+0000","last_change":"2024-09-16T00:15:04.373381+0000","last_active":"2024-09-16T00:17:09.579738+0000","last_peered":"2024-09-16T00:17:09.579738+0000","last_clean":"2024-09-16T00:17:09.579738+0000","last_became_active":"2024-09-16T00:15:04.373028+0000","last_became_peered":"2024-09-16T00:15:04.373028+0000","last_unstale":"2024-09-16T00:17:09.579738+0000","last_undegraded":"2024-09-16T00:17:09.579738+0000","last_fullsized":"2024-09-16T00:17:09.579738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:22.902583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577500+0000","last_change":"2024-09-16T00:15:06.554048+0000","last_active":"2024-09-16T00:17:09.577500+0000","last_peered":"2024-09-16T00:17:09.577500+0000","last_clean":"2024-09-16T00:17:09.577500+0000","last_became_active":"2024-09-16T00:15:06.553044+0000","last_became_peered":"2024-09-16T00:15:06.553044+0000","last_unstale":"2024-09-16T00:17:09.577500+0000","last_undegraded":"2024-09-16T00:17:09.577500+0000","last_fullsized":"2024-09-16T00:17:09.577500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:22:53.333705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574960+0000","last_change":"2024-09-16T00:15:13.586995+0000","last_active":"2024-09-16T00:17:09.574960+0000","last_peered":"2024-09-16T00:17:09.574960+0000","last_clean":"2024-09-16T00:17:09.574960+0000","last_became_active":"2024-09-16T00:15:13.586509+0000","last_became_peered":"2024-09-16T00:15:13.586509+0000","last_unstale":"2024-09-16T00:17:09.574960+0000","last_undegraded":"2024-09-16T00:17:09.574960+0000","last_fullsized":"2024-09-16T00:17:09.574960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:39.418581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579645+0000","last_change":"2024-09-16T00:15:04.348235+0000","last_active":"2024-09-16T00:17:09.579645+0000","last_peered":"2024-09-16T00:17:09.579645+0000","last_clean":"2024-09-16T00:17:09.579645+0000","last_became_active":"2024-09-16T00:15:04.347518+0000","last_became_peered":"2024-09-16T00:15:04.347518+0000","last_unstale":"2024-09-16T00:17:09.579645+0000","last_undegraded":"2024-09-16T00:17:09.579645+0000","last_fullsized":"2024-09-16T00:17:09.579645+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:24.396911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581095+0000","last_change":"2024-09-16T00:15:06.554067+0000","last_active":"2024-09-16T00:17:09.581095+0000","last_peered":"2024-09-16T00:17:09.581095+0000","last_clean":"2024-09-16T00:17:09.581095+0000","last_became_active":"2024-09-16T00:15:06.553715+0000","last_became_peered":"2024-09-16T00:15:06.553715+0000","last_unstale":"2024-09-16T00:17:09.581095+0000","last_undegraded":"2024-09-16T00:17:09.581095+0000","last_fullsized":"2024-09-16T00:17:09.581095+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:22.008530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575352+0000","last_change":"2024-09-16T00:15:13.586872+0000","last_active":"2024-09-16T00:17:09.575352+0000","last_peered":"2024-09-16T00:17:09.575352+0000","last_clean":"2024-09-16T00:17:09.575352+0000","last_became_active":"2024-09-16T00:15:13.586375+0000","last_became_peered":"2024-09-16T00:15:13.586375+0000","last_unstale":"2024-09-16T00:17:09.575352+0000","last_undegraded":"2024-09-16T00:17:09.575352+0000","last_fullsized":"2024-09-16T00:17:09.575352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:43.668827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579663+0000","last_change":"2024-09-16T00:15:04.347964+0000","last_active":"2024-09-16T00:17:09.579663+0000","last_peered":"2024-09-16T00:17:09.579663+0000","last_clean":"2024-09-16T00:17:09.579663+0000","last_became_active":"2024-09-16T00:15:04.347588+0000","last_became_peered":"2024-09-16T00:15:04.347588+0000","last_unstale":"2024-09-16T00:17:09.579663+0000","last_undegraded":"2024-09-16T00:17:09.579663+0000","last_fullsized":"2024-09-16T00:17:09.579663+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:14.199287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576911+0000","last_change":"2024-09-16T00:15:06.544918+0000","last_active":"2024-09-16T00:17:09.576911+0000","last_peered":"2024-09-16T00:17:09.576911+0000","last_clean":"2024-09-16T00:17:09.576911+0000","last_became_active":"2024-09-16T00:15:06.544579+0000","last_became_peered":"2024-09-16T00:15:06.544579+0000","last_unstale":"2024-09-16T00:17:09.576911+0000","last_undegraded":"2024-09-16T00:17:09.576911+0000","last_fullsized":"2024-09-16T00:17:09.576911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:06:37.475644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575052+0000","last_change":"2024-09-16T00:15:13.581488+0000","last_active":"2024-09-16T00:17:09.575052+0000","last_peered":"2024-09-16T00:17:09.575052+0000","last_clean":"2024-09-16T00:17:09.575052+0000","last_became_active":"2024-09-16T00:15:13.581164+0000","last_became_peered":"2024-09-16T00:15:13.581164+0000","last_unstale":"2024-09-16T00:17:09.575052+0000","last_undegraded":"2024-09-16T00:17:09.575052+0000","last_fullsized":"2024-09-16T00:17:09.575052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:05.061592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581271+0000","last_change":"2024-09-16T00:15:06.551982+0000","last_active":"2024-09-16T00:17:09.581271+0000","last_peered":"2024-09-16T00:17:09.581271+0000","last_clean":"2024-09-16T00:17:09.581271+0000","last_became_active":"2024-09-16T00:15:06.551663+0000","last_became_peered":"2024-09-16T00:15:06.551663+0000","last_unstale":"2024-09-16T00:17:09.581271+0000","last_undegraded":"2024-09-16T00:17:09.581271+0000","last_fullsized":"2024-09-16T00:17:09.581271+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:11.928703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579394+0000","last_change":"2024-09-16T00:15:04.352237+0000","last_active":"2024-09-16T00:17:09.579394+0000","last_peered":"2024-09-16T00:17:09.579394+0000","last_clean":"2024-09-16T00:17:09.579394+0000","last_became_active":"2024-09-16T00:15:04.351884+0000","last_became_peered":"2024-09-16T00:15:04.351884+0000","last_unstale":"2024-09-16T00:17:09.579394+0000","last_undegraded":"2024-09-16T00:17:09.579394+0000","last_fullsized":"2024-09-16T00:17:09.579394+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:32.932020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575580+0000","last_change":"2024-09-16T00:15:13.638917+0000","last_active":"2024-09-16T00:17:09.575580+0000","last_peered":"2024-09-16T00:17:09.575580+0000","last_clean":"2024-09-16T00:17:09.575580+0000","last_became_active":"2024-09-16T00:15:13.638401+0000","last_became_peered":"2024-09-16T00:15:13.638401+0000","last_unstale":"2024-09-16T00:17:09.575580+0000","last_undegraded":"2024-09-16T00:17:09.575580+0000","last_fullsized":"2024-09-16T00:17:09.575580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:38.082407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579388+0000","last_change":"2024-09-16T00:15:06.547909+0000","last_active":"2024-09-16T00:17:09.579388+0000","last_peered":"2024-09-16T00:17:09.579388+0000","last_clean":"2024-09-16T00:17:09.579388+0000","last_became_active":"2024-09-16T00:15:06.546814+0000","last_became_peered":"2024-09-16T00:15:06.546814+0000","last_unstale":"2024-09-16T00:17:09.579388+0000","last_undegraded":"2024-09-16T00:17:09.579388+0000","last_fullsized":"2024-09-16T00:17:09.579388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:46.419406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780898+0000","last_change":"2024-09-16T00:16:09.027691+0000","last_active":"2024-09-16T00:17:09.780898+0000","last_peered":"2024-09-16T00:17:09.780898+0000","last_clean":"2024-09-16T00:17:09.780898+0000","last_became_active":"2024-09-16T00:16:09.023204+0000","last_became_peered":"2024-09-16T00:16:09.023204+0000","last_unstale":"2024-09-16T00:17:09.780898+0000","last_undegraded":"2024-09-16T00:17:09.780898+0000","last_fullsized":"2024-09-16T00:17:09.780898+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:44.647576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575627+0000","last_change":"2024-09-16T00:15:13.588345+0000","last_active":"2024-09-16T00:17:09.575627+0000","last_peered":"2024-09-16T00:17:09.575627+0000","last_clean":"2024-09-16T00:17:09.575627+0000","last_became_active":"2024-09-16T00:15:13.587829+0000","last_became_peered":"2024-09-16T00:15:13.587829+0000","last_unstale":"2024-09-16T00:17:09.575627+0000","last_undegraded":"2024-09-16T00:17:09.575627+0000","last_fullsized":"2024-09-16T00:17:09.575627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:04.261031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781137+0000","last_change":"2024-09-16T00:15:06.545485+0000","last_active":"2024-09-16T00:17:09.781137+0000","last_peered":"2024-09-16T00:17:09.781137+0000","last_clean":"2024-09-16T00:17:09.781137+0000","last_became_active":"2024-09-16T00:15:06.545178+0000","last_became_peered":"2024-09-16T00:15:06.545178+0000","last_unstale":"2024-09-16T00:17:09.781137+0000","last_undegraded":"2024-09-16T00:17:09.781137+0000","last_fullsized":"2024-09-16T00:17:09.781137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:26.923469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579782+0000","last_change":"2024-09-16T00:15:04.340546+0000","last_active":"2024-09-16T00:17:09.579782+0000","last_peered":"2024-09-16T00:17:09.579782+0000","last_clean":"2024-09-16T00:17:09.579782+0000","last_became_active":"2024-09-16T00:15:04.340173+0000","last_became_peered":"2024-09-16T00:15:04.340173+0000","last_unstale":"2024-09-16T00:17:09.579782+0000","last_undegraded":"2024-09-16T00:17:09.579782+0000","last_fullsized":"2024-09-16T00:17:09.579782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:35:01.134433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578541+0000","last_change":"2024-09-16T00:15:13.589963+0000","last_active":"2024-09-16T00:17:09.578541+0000","last_peered":"2024-09-16T00:17:09.578541+0000","last_clean":"2024-09-16T00:17:09.578541+0000","last_became_active":"2024-09-16T00:15:13.589615+0000","last_became_peered":"2024-09-16T00:15:13.589615+0000","last_unstale":"2024-09-16T00:17:09.578541+0000","last_undegraded":"2024-09-16T00:17:09.578541+0000","last_fullsized":"2024-09-16T00:17:09.578541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:37.197765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579731+0000","last_change":"2024-09-16T00:15:06.544374+0000","last_active":"2024-09-16T00:17:09.579731+0000","last_peered":"2024-09-16T00:17:09.579731+0000","last_clean":"2024-09-16T00:17:09.579731+0000","last_became_active":"2024-09-16T00:15:06.544027+0000","last_became_peered":"2024-09-16T00:15:06.544027+0000","last_unstale":"2024-09-16T00:17:09.579731+0000","last_undegraded":"2024-09-16T00:17:09.579731+0000","last_fullsized":"2024-09-16T00:17:09.579731+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580105+0000","last_change":"2024-09-16T00:15:04.374209+0000","last_active":"2024-09-16T00:17:09.580105+0000","last_peered":"2024-09-16T00:17:09.580105+0000","last_clean":"2024-09-16T00:17:09.580105+0000","last_became_active":"2024-09-16T00:15:04.373814+0000","last_became_peered":"2024-09-16T00:15:04.373814+0000","last_unstale":"2024-09-16T00:17:09.580105+0000","last_undegraded":"2024-09-16T00:17:09.580105+0000","last_fullsized":"2024-09-16T00:17:09.580105+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:13.516161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575489+0000","last_change":"2024-09-16T00:15:13.639210+0000","last_active":"2024-09-16T00:17:09.575489+0000","last_peered":"2024-09-16T00:17:09.575489+0000","last_clean":"2024-09-16T00:17:09.575489+0000","last_became_active":"2024-09-16T00:15:13.638719+0000","last_became_peered":"2024-09-16T00:15:13.638719+0000","last_unstale":"2024-09-16T00:17:09.575489+0000","last_undegraded":"2024-09-16T00:17:09.575489+0000","last_fullsized":"2024-09-16T00:17:09.575489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:26.130778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577341+0000","last_change":"2024-09-16T00:15:06.553108+0000","last_active":"2024-09-16T00:17:09.577341+0000","last_peered":"2024-09-16T00:17:09.577341+0000","last_clean":"2024-09-16T00:17:09.577341+0000","last_became_active":"2024-09-16T00:15:06.552817+0000","last_became_peered":"2024-09-16T00:15:06.552817+0000","last_unstale":"2024-09-16T00:17:09.577341+0000","last_undegraded":"2024-09-16T00:17:09.577341+0000","last_fullsized":"2024-09-16T00:17:09.577341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:06.821337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575833+0000","last_change":"2024-09-16T00:15:04.644185+0000","last_active":"2024-09-16T00:17:09.575833+0000","last_peered":"2024-09-16T00:17:09.575833+0000","last_clean":"2024-09-16T00:17:09.575833+0000","last_became_active":"2024-09-16T00:15:04.643887+0000","last_became_peered":"2024-09-16T00:15:04.643887+0000","last_unstale":"2024-09-16T00:17:09.575833+0000","last_undegraded":"2024-09-16T00:17:09.575833+0000","last_fullsized":"2024-09-16T00:17:09.575833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:44.279272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574877+0000","last_change":"2024-09-16T00:15:13.574761+0000","last_active":"2024-09-16T00:17:09.574877+0000","last_peered":"2024-09-16T00:17:09.574877+0000","last_clean":"2024-09-16T00:17:09.574877+0000","last_became_active":"2024-09-16T00:15:13.574347+0000","last_became_peered":"2024-09-16T00:15:13.574347+0000","last_unstale":"2024-09-16T00:17:09.574877+0000","last_undegraded":"2024-09-16T00:17:09.574877+0000","last_fullsized":"2024-09-16T00:17:09.574877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:22.040121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579504+0000","last_change":"2024-09-16T00:15:06.550338+0000","last_active":"2024-09-16T00:17:09.579504+0000","last_peered":"2024-09-16T00:17:09.579504+0000","last_clean":"2024-09-16T00:17:09.579504+0000","last_became_active":"2024-09-16T00:15:06.549868+0000","last_became_peered":"2024-09-16T00:15:06.549868+0000","last_unstale":"2024-09-16T00:17:09.579504+0000","last_undegraded":"2024-09-16T00:17:09.579504+0000","last_fullsized":"2024-09-16T00:17:09.579504+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:26.643338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'13","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:28.555275+0000","last_change":"2024-09-16T00:15:04.342306+0000","last_active":"2024-09-16T00:17:28.555275+0000","last_peered":"2024-09-16T00:17:28.555275+0000","last_clean":"2024-09-16T00:17:28.555275+0000","last_became_active":"2024-09-16T00:15:04.341964+0000","last_became_peered":"2024-09-16T00:15:04.341964+0000","last_unstale":"2024-09-16T00:17:28.555275+0000","last_undegraded":"2024-09-16T00:17:28.555275+0000","last_fullsized":"2024-09-16T00:17:28.555275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:14.137199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578972+0000","last_change":"2024-09-16T00:15:13.638246+0000","last_active":"2024-09-16T00:17:09.578972+0000","last_peered":"2024-09-16T00:17:09.578972+0000","last_clean":"2024-09-16T00:17:09.578972+0000","last_became_active":"2024-09-16T00:15:13.637816+0000","last_became_peered":"2024-09-16T00:15:13.637816+0000","last_unstale":"2024-09-16T00:17:09.578972+0000","last_undegraded":"2024-09-16T00:17:09.578972+0000","last_fullsized":"2024-09-16T00:17:09.578972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:27.331156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718239+0000","last_change":"2024-09-16T00:15:06.544057+0000","last_active":"2024-09-16T00:17:09.718239+0000","last_peered":"2024-09-16T00:17:09.718239+0000","last_clean":"2024-09-16T00:17:09.718239+0000","last_became_active":"2024-09-16T00:15:06.543244+0000","last_became_peered":"2024-09-16T00:15:06.543244+0000","last_unstale":"2024-09-16T00:17:09.718239+0000","last_undegraded":"2024-09-16T00:17:09.718239+0000","last_fullsized":"2024-09-16T00:17:09.718239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:42:28.950014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579701+0000","last_change":"2024-09-16T00:15:04.376419+0000","last_active":"2024-09-16T00:17:09.579701+0000","last_peered":"2024-09-16T00:17:09.579701+0000","last_clean":"2024-09-16T00:17:09.579701+0000","last_became_active":"2024-09-16T00:15:04.375976+0000","last_became_peered":"2024-09-16T00:15:04.375976+0000","last_unstale":"2024-09-16T00:17:09.579701+0000","last_undegraded":"2024-09-16T00:17:09.579701+0000","last_fullsized":"2024-09-16T00:17:09.579701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:33:38.652109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781038+0000","last_change":"2024-09-16T00:15:13.584306+0000","last_active":"2024-09-16T00:17:09.781038+0000","last_peered":"2024-09-16T00:17:09.781038+0000","last_clean":"2024-09-16T00:17:09.781038+0000","last_became_active":"2024-09-16T00:15:13.583840+0000","last_became_peered":"2024-09-16T00:15:13.583840+0000","last_unstale":"2024-09-16T00:17:09.781038+0000","last_undegraded":"2024-09-16T00:17:09.781038+0000","last_fullsized":"2024-09-16T00:17:09.781038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:36.660458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577806+0000","last_change":"2024-09-16T00:15:06.538902+0000","last_active":"2024-09-16T00:17:09.577806+0000","last_peered":"2024-09-16T00:17:09.577806+0000","last_clean":"2024-09-16T00:17:09.577806+0000","last_became_active":"2024-09-16T00:15:06.538537+0000","last_became_peered":"2024-09-16T00:15:06.538537+0000","last_unstale":"2024-09-16T00:17:09.577806+0000","last_undegraded":"2024-09-16T00:17:09.577806+0000","last_fullsized":"2024-09-16T00:17:09.577806+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:38.411711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578013+0000","last_change":"2024-09-16T00:15:04.374762+0000","last_active":"2024-09-16T00:17:09.578013+0000","last_peered":"2024-09-16T00:17:09.578013+0000","last_clean":"2024-09-16T00:17:09.578013+0000","last_became_active":"2024-09-16T00:15:04.374394+0000","last_became_peered":"2024-09-16T00:15:04.374394+0000","last_unstale":"2024-09-16T00:17:09.578013+0000","last_undegraded":"2024-09-16T00:17:09.578013+0000","last_fullsized":"2024-09-16T00:17:09.578013+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:47.075021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579233+0000","last_change":"2024-09-16T00:15:13.638306+0000","last_active":"2024-09-16T00:17:09.579233+0000","last_peered":"2024-09-16T00:17:09.579233+0000","last_clean":"2024-09-16T00:17:09.579233+0000","last_became_active":"2024-09-16T00:15:13.637931+0000","last_became_peered":"2024-09-16T00:15:13.637931+0000","last_unstale":"2024-09-16T00:17:09.579233+0000","last_undegraded":"2024-09-16T00:17:09.579233+0000","last_fullsized":"2024-09-16T00:17:09.579233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:41.385174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577585+0000","last_change":"2024-09-16T00:15:06.548209+0000","last_active":"2024-09-16T00:17:09.577585+0000","last_peered":"2024-09-16T00:17:09.577585+0000","last_clean":"2024-09-16T00:17:09.577585+0000","last_became_active":"2024-09-16T00:15:06.547846+0000","last_became_peered":"2024-09-16T00:15:06.547846+0000","last_unstale":"2024-09-16T00:17:09.577585+0000","last_undegraded":"2024-09-16T00:17:09.577585+0000","last_fullsized":"2024-09-16T00:17:09.577585+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:15.821847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780889+0000","last_change":"2024-09-16T00:16:09.023918+0000","last_active":"2024-09-16T00:17:09.780889+0000","last_peered":"2024-09-16T00:17:09.780889+0000","last_clean":"2024-09-16T00:17:09.780889+0000","last_became_active":"2024-09-16T00:16:09.023230+0000","last_became_peered":"2024-09-16T00:16:09.023230+0000","last_unstale":"2024-09-16T00:17:09.780889+0000","last_undegraded":"2024-09-16T00:17:09.780889+0000","last_fullsized":"2024-09-16T00:17:09.780889+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:31.527863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717949+0000","last_change":"2024-09-16T00:15:13.579387+0000","last_active":"2024-09-16T00:17:09.717949+0000","last_peered":"2024-09-16T00:17:09.717949+0000","last_clean":"2024-09-16T00:17:09.717949+0000","last_became_active":"2024-09-16T00:15:13.578996+0000","last_became_peered":"2024-09-16T00:15:13.578996+0000","last_unstale":"2024-09-16T00:17:09.717949+0000","last_undegraded":"2024-09-16T00:17:09.717949+0000","last_fullsized":"2024-09-16T00:17:09.717949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:51.703060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579796+0000","last_change":"2024-09-16T00:17:09.579796+0000","last_active":"2024-09-16T00:17:09.579796+0000","last_peered":"2024-09-16T00:17:09.579796+0000","last_clean":"2024-09-16T00:17:09.579796+0000","last_became_active":"2024-09-16T00:17:09.579386+0000","last_became_peered":"2024-09-16T00:17:09.579386+0000","last_unstale":"2024-09-16T00:17:09.579796+0000","last_undegraded":"2024-09-16T00:17:09.579796+0000","last_fullsized":"2024-09-16T00:17:09.579796+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:39:42.571193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":150,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:33.321091+0000","last_change":"2024-09-16T00:15:04.373896+0000","last_active":"2024-09-16T00:17:33.321091+0000","last_peered":"2024-09-16T00:17:33.321091+0000","last_clean":"2024-09-16T00:17:33.321091+0000","last_became_active":"2024-09-16T00:15:04.373493+0000","last_became_peered":"2024-09-16T00:15:04.373493+0000","last_unstale":"2024-09-16T00:17:33.321091+0000","last_undegraded":"2024-09-16T00:17:33.321091+0000","last_fullsized":"2024-09-16T00:17:33.321091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:17:10.767267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718128+0000","last_change":"2024-09-16T00:15:13.638575+0000","last_active":"2024-09-16T00:17:09.718128+0000","last_peered":"2024-09-16T00:17:09.718128+0000","last_clean":"2024-09-16T00:17:09.718128+0000","last_became_active":"2024-09-16T00:15:13.638144+0000","last_became_peered":"2024-09-16T00:15:13.638144+0000","last_unstale":"2024-09-16T00:17:09.718128+0000","last_undegraded":"2024-09-16T00:17:09.718128+0000","last_fullsized":"2024-09-16T00:17:09.718128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:39.345011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718312+0000","last_change":"2024-09-16T00:15:06.548852+0000","last_active":"2024-09-16T00:17:09.718312+0000","last_peered":"2024-09-16T00:17:09.718312+0000","last_clean":"2024-09-16T00:17:09.718312+0000","last_became_active":"2024-09-16T00:15:06.548238+0000","last_became_peered":"2024-09-16T00:15:06.548238+0000","last_unstale":"2024-09-16T00:17:09.718312+0000","last_undegraded":"2024-09-16T00:17:09.718312+0000","last_fullsized":"2024-09-16T00:17:09.718312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:16.115338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580690+0000","last_change":"2024-09-16T00:15:04.350753+0000","last_active":"2024-09-16T00:17:09.580690+0000","last_peered":"2024-09-16T00:17:09.580690+0000","last_clean":"2024-09-16T00:17:09.580690+0000","last_became_active":"2024-09-16T00:15:04.350406+0000","last_became_peered":"2024-09-16T00:15:04.350406+0000","last_unstale":"2024-09-16T00:17:09.580690+0000","last_undegraded":"2024-09-16T00:17:09.580690+0000","last_fullsized":"2024-09-16T00:17:09.580690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:09:54.737287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576996+0000","last_change":"2024-09-16T00:15:13.639302+0000","last_active":"2024-09-16T00:17:09.576996+0000","last_peered":"2024-09-16T00:17:09.576996+0000","last_clean":"2024-09-16T00:17:09.576996+0000","last_became_active":"2024-09-16T00:15:13.638863+0000","last_became_peered":"2024-09-16T00:15:13.638863+0000","last_unstale":"2024-09-16T00:17:09.576996+0000","last_undegraded":"2024-09-16T00:17:09.576996+0000","last_fullsized":"2024-09-16T00:17:09.576996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:23.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582176+0000","last_change":"2024-09-16T00:17:09.582176+0000","last_active":"2024-09-16T00:17:09.582176+0000","last_peered":"2024-09-16T00:17:09.582176+0000","last_clean":"2024-09-16T00:17:09.582176+0000","last_became_active":"2024-09-16T00:17:09.581740+0000","last_became_peered":"2024-09-16T00:17:09.581740+0000","last_unstale":"2024-09-16T00:17:09.582176+0000","last_undegraded":"2024-09-16T00:17:09.582176+0000","last_fullsized":"2024-09-16T00:17:09.582176+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:57.176724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575861+0000","last_change":"2024-09-16T00:15:04.342216+0000","last_active":"2024-09-16T00:17:09.575861+0000","last_peered":"2024-09-16T00:17:09.575861+0000","last_clean":"2024-09-16T00:17:09.575861+0000","last_became_active":"2024-09-16T00:15:04.341763+0000","last_became_peered":"2024-09-16T00:15:04.341763+0000","last_unstale":"2024-09-16T00:17:09.575861+0000","last_undegraded":"2024-09-16T00:17:09.575861+0000","last_fullsized":"2024-09-16T00:17:09.575861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:44.938527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":350,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580892+0000","last_change":"2024-09-16T00:13:51.108571+0000","last_active":"2024-09-16T00:17:09.580892+0000","last_peered":"2024-09-16T00:17:09.580892+0000","last_clean":"2024-09-16T00:17:09.580892+0000","last_became_active":"2024-09-16T00:13:51.108088+0000","last_became_peered":"2024-09-16T00:13:51.108088+0000","last_unstale":"2024-09-16T00:17:09.580892+0000","last_undegraded":"2024-09-16T00:17:09.580892+0000","last_fullsized":"2024-09-16T00:17:09.580892+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_clean_scrub_stamp":"2024-09-16T00:10:24.533807+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:02.511064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581143+0000","last_change":"2024-09-16T00:15:13.588665+0000","last_active":"2024-09-16T00:17:09.581143+0000","last_peered":"2024-09-16T00:17:09.581143+0000","last_clean":"2024-09-16T00:17:09.581143+0000","last_became_active":"2024-09-16T00:15:13.588222+0000","last_became_peered":"2024-09-16T00:15:13.588222+0000","last_unstale":"2024-09-16T00:17:09.581143+0000","last_undegraded":"2024-09-16T00:17:09.581143+0000","last_fullsized":"2024-09-16T00:17:09.581143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:59:59.965962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581454+0000","last_change":"2024-09-16T00:15:06.553114+0000","last_active":"2024-09-16T00:17:09.581454+0000","last_peered":"2024-09-16T00:17:09.581454+0000","last_clean":"2024-09-16T00:17:09.581454+0000","last_became_active":"2024-09-16T00:15:06.552788+0000","last_became_peered":"2024-09-16T00:15:06.552788+0000","last_unstale":"2024-09-16T00:17:09.581454+0000","last_undegraded":"2024-09-16T00:17:09.581454+0000","last_fullsized":"2024-09-16T00:17:09.581454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:05.473847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575364+0000","last_change":"2024-09-16T00:16:09.030747+0000","last_active":"2024-09-16T00:17:09.575364+0000","last_peered":"2024-09-16T00:17:09.575364+0000","last_clean":"2024-09-16T00:17:09.575364+0000","last_became_active":"2024-09-16T00:16:09.021208+0000","last_became_peered":"2024-09-16T00:16:09.021208+0000","last_unstale":"2024-09-16T00:17:09.575364+0000","last_undegraded":"2024-09-16T00:17:09.575364+0000","last_fullsized":"2024-09-16T00:17:09.575364+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:40.605498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580422+0000","last_change":"2024-09-16T00:15:13.575606+0000","last_active":"2024-09-16T00:17:09.580422+0000","last_peered":"2024-09-16T00:17:09.580422+0000","last_clean":"2024-09-16T00:17:09.580422+0000","last_became_active":"2024-09-16T00:15:13.575131+0000","last_became_peered":"2024-09-16T00:15:13.575131+0000","last_unstale":"2024-09-16T00:17:09.580422+0000","last_undegraded":"2024-09-16T00:17:09.580422+0000","last_fullsized":"2024-09-16T00:17:09.580422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:41.636058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.584161+0000","last_change":"2024-09-16T00:17:09.583522+0000","last_active":"2024-09-16T00:17:09.584161+0000","last_peered":"2024-09-16T00:17:09.584161+0000","last_clean":"2024-09-16T00:17:09.584161+0000","last_became_active":"2024-09-16T00:17:09.583082+0000","last_became_peered":"2024-09-16T00:17:09.583082+0000","last_unstale":"2024-09-16T00:17:09.584161+0000","last_undegraded":"2024-09-16T00:17:09.584161+0000","last_fullsized":"2024-09-16T00:17:09.584161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:48.582126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577245+0000","last_change":"2024-09-16T00:15:04.349095+0000","last_active":"2024-09-16T00:17:09.577245+0000","last_peered":"2024-09-16T00:17:09.577245+0000","last_clean":"2024-09-16T00:17:09.577245+0000","last_became_active":"2024-09-16T00:15:04.348698+0000","last_became_peered":"2024-09-16T00:15:04.348698+0000","last_unstale":"2024-09-16T00:17:09.577245+0000","last_undegraded":"2024-09-16T00:17:09.577245+0000","last_fullsized":"2024-09-16T00:17:09.577245+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:42.282882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577535+0000","last_change":"2024-09-16T00:15:13.580494+0000","last_active":"2024-09-16T00:17:09.577535+0000","last_peered":"2024-09-16T00:17:09.577535+0000","last_clean":"2024-09-16T00:17:09.577535+0000","last_became_active":"2024-09-16T00:15:13.580091+0000","last_became_peered":"2024-09-16T00:15:13.580091+0000","last_unstale":"2024-09-16T00:17:09.577535+0000","last_undegraded":"2024-09-16T00:17:09.577535+0000","last_fullsized":"2024-09-16T00:17:09.577535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:17.314919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580727+0000","last_change":"2024-09-16T00:15:06.544859+0000","last_active":"2024-09-16T00:17:09.580727+0000","last_peered":"2024-09-16T00:17:09.580727+0000","last_clean":"2024-09-16T00:17:09.580727+0000","last_became_active":"2024-09-16T00:15:06.544289+0000","last_became_peered":"2024-09-16T00:15:06.544289+0000","last_unstale":"2024-09-16T00:17:09.580727+0000","last_undegraded":"2024-09-16T00:17:09.580727+0000","last_fullsized":"2024-09-16T00:17:09.580727+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:52.622477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718205+0000","last_change":"2024-09-16T00:15:04.351313+0000","last_active":"2024-09-16T00:17:09.718205+0000","last_peered":"2024-09-16T00:17:09.718205+0000","last_clean":"2024-09-16T00:17:09.718205+0000","last_became_active":"2024-09-16T00:15:04.350928+0000","last_became_peered":"2024-09-16T00:15:04.350928+0000","last_unstale":"2024-09-16T00:17:09.718205+0000","last_undegraded":"2024-09-16T00:17:09.718205+0000","last_fullsized":"2024-09-16T00:17:09.718205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:48.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576571+0000","last_change":"2024-09-16T00:15:13.578573+0000","last_active":"2024-09-16T00:17:09.576571+0000","last_peered":"2024-09-16T00:17:09.576571+0000","last_clean":"2024-09-16T00:17:09.576571+0000","last_became_active":"2024-09-16T00:15:13.578117+0000","last_became_peered":"2024-09-16T00:15:13.578117+0000","last_unstale":"2024-09-16T00:17:09.576571+0000","last_undegraded":"2024-09-16T00:17:09.576571+0000","last_fullsized":"2024-09-16T00:17:09.576571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:53.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781258+0000","last_change":"2024-09-16T00:15:06.553548+0000","last_active":"2024-09-16T00:17:09.781258+0000","last_peered":"2024-09-16T00:17:09.781258+0000","last_clean":"2024-09-16T00:17:09.781258+0000","last_became_active":"2024-09-16T00:15:06.552963+0000","last_became_peered":"2024-09-16T00:15:06.552963+0000","last_unstale":"2024-09-16T00:17:09.781258+0000","last_undegraded":"2024-09-16T00:17:09.781258+0000","last_fullsized":"2024-09-16T00:17:09.781258+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:43.455881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579776+0000","last_change":"2024-09-16T00:15:04.351005+0000","last_active":"2024-09-16T00:17:09.579776+0000","last_peered":"2024-09-16T00:17:09.579776+0000","last_clean":"2024-09-16T00:17:09.579776+0000","last_became_active":"2024-09-16T00:15:04.350614+0000","last_became_peered":"2024-09-16T00:15:04.350614+0000","last_unstale":"2024-09-16T00:17:09.579776+0000","last_undegraded":"2024-09-16T00:17:09.579776+0000","last_fullsized":"2024-09-16T00:17:09.579776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:39.196360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579501+0000","last_change":"2024-09-16T00:16:09.023796+0000","last_active":"2024-09-16T00:17:09.579501+0000","last_peered":"2024-09-16T00:17:09.579501+0000","last_clean":"2024-09-16T00:17:09.579501+0000","last_became_active":"2024-09-16T00:16:09.023403+0000","last_became_peered":"2024-09-16T00:16:09.023403+0000","last_unstale":"2024-09-16T00:17:09.579501+0000","last_undegraded":"2024-09-16T00:17:09.579501+0000","last_fullsized":"2024-09-16T00:17:09.579501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:59.488315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577059+0000","last_change":"2024-09-16T00:15:06.546325+0000","last_active":"2024-09-16T00:17:09.577059+0000","last_peered":"2024-09-16T00:17:09.577059+0000","last_clean":"2024-09-16T00:17:09.577059+0000","last_became_active":"2024-09-16T00:15:06.546028+0000","last_became_peered":"2024-09-16T00:15:06.546028+0000","last_unstale":"2024-09-16T00:17:09.577059+0000","last_undegraded":"2024-09-16T00:17:09.577059+0000","last_fullsized":"2024-09-16T00:17:09.577059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:15.049592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579065+0000","last_change":"2024-09-16T00:15:04.373818+0000","last_active":"2024-09-16T00:17:09.579065+0000","last_peered":"2024-09-16T00:17:09.579065+0000","last_clean":"2024-09-16T00:17:09.579065+0000","last_became_active":"2024-09-16T00:15:04.373406+0000","last_became_peered":"2024-09-16T00:15:04.373406+0000","last_unstale":"2024-09-16T00:17:09.579065+0000","last_undegraded":"2024-09-16T00:17:09.579065+0000","last_fullsized":"2024-09-16T00:17:09.579065+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:08.402660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718065+0000","last_change":"2024-09-16T00:15:13.588669+0000","last_active":"2024-09-16T00:17:09.718065+0000","last_peered":"2024-09-16T00:17:09.718065+0000","last_clean":"2024-09-16T00:17:09.718065+0000","last_became_active":"2024-09-16T00:15:13.588229+0000","last_became_peered":"2024-09-16T00:15:13.588229+0000","last_unstale":"2024-09-16T00:17:09.718065+0000","last_undegraded":"2024-09-16T00:17:09.718065+0000","last_fullsized":"2024-09-16T00:17:09.718065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:30.157296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580525+0000","last_change":"2024-09-16T00:15:06.550221+0000","last_active":"2024-09-16T00:17:09.580525+0000","last_peered":"2024-09-16T00:17:09.580525+0000","last_clean":"2024-09-16T00:17:09.580525+0000","last_became_active":"2024-09-16T00:15:06.549914+0000","last_became_peered":"2024-09-16T00:15:06.549914+0000","last_unstale":"2024-09-16T00:17:09.580525+0000","last_undegraded":"2024-09-16T00:17:09.580525+0000","last_fullsized":"2024-09-16T00:17:09.580525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:18:26.824523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":154,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:33.321594+0000","last_change":"2024-09-16T00:15:04.344389+0000","last_active":"2024-09-16T00:17:33.321594+0000","last_peered":"2024-09-16T00:17:33.321594+0000","last_clean":"2024-09-16T00:17:33.321594+0000","last_became_active":"2024-09-16T00:15:04.343956+0000","last_became_peered":"2024-09-16T00:15:04.343956+0000","last_unstale":"2024-09-16T00:17:33.321594+0000","last_undegraded":"2024-09-16T00:17:33.321594+0000","last_fullsized":"2024-09-16T00:17:33.321594+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:34.019861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579707+0000","last_change":"2024-09-16T00:16:09.024050+0000","last_active":"2024-09-16T00:17:09.579707+0000","last_peered":"2024-09-16T00:17:09.579707+0000","last_clean":"2024-09-16T00:17:09.579707+0000","last_became_active":"2024-09-16T00:16:09.023371+0000","last_became_peered":"2024-09-16T00:16:09.023371+0000","last_unstale":"2024-09-16T00:17:09.579707+0000","last_undegraded":"2024-09-16T00:17:09.579707+0000","last_fullsized":"2024-09-16T00:17:09.579707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:35:19.020370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581433+0000","last_change":"2024-09-16T00:15:06.542744+0000","last_active":"2024-09-16T00:17:09.581433+0000","last_peered":"2024-09-16T00:17:09.581433+0000","last_clean":"2024-09-16T00:17:09.581433+0000","last_became_active":"2024-09-16T00:15:06.541868+0000","last_became_peered":"2024-09-16T00:15:06.541868+0000","last_unstale":"2024-09-16T00:17:09.581433+0000","last_undegraded":"2024-09-16T00:17:09.581433+0000","last_fullsized":"2024-09-16T00:17:09.581433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:10:06.508682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577336+0000","last_change":"2024-09-16T00:15:04.349556+0000","last_active":"2024-09-16T00:17:09.577336+0000","last_peered":"2024-09-16T00:17:09.577336+0000","last_clean":"2024-09-16T00:17:09.577336+0000","last_became_active":"2024-09-16T00:15:04.349242+0000","last_became_peered":"2024-09-16T00:15:04.349242+0000","last_unstale":"2024-09-16T00:17:09.577336+0000","last_undegraded":"2024-09-16T00:17:09.577336+0000","last_fullsized":"2024-09-16T00:17:09.577336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:22:18.459157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781143+0000","last_change":"2024-09-16T00:15:13.584109+0000","last_active":"2024-09-16T00:17:09.781143+0000","last_peered":"2024-09-16T00:17:09.781143+0000","last_clean":"2024-09-16T00:17:09.781143+0000","last_became_active":"2024-09-16T00:15:13.583718+0000","last_became_peered":"2024-09-16T00:15:13.583718+0000","last_unstale":"2024-09-16T00:17:09.781143+0000","last_undegraded":"2024-09-16T00:17:09.781143+0000","last_fullsized":"2024-09-16T00:17:09.781143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:29.416254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717952+0000","last_change":"2024-09-16T00:15:06.548247+0000","last_active":"2024-09-16T00:17:09.717952+0000","last_peered":"2024-09-16T00:17:09.717952+0000","last_clean":"2024-09-16T00:17:09.717952+0000","last_became_active":"2024-09-16T00:15:06.547864+0000","last_became_peered":"2024-09-16T00:15:06.547864+0000","last_unstale":"2024-09-16T00:17:09.717952+0000","last_undegraded":"2024-09-16T00:17:09.717952+0000","last_fullsized":"2024-09-16T00:17:09.717952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:25.350197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575352+0000","last_change":"2024-09-16T00:15:04.350965+0000","last_active":"2024-09-16T00:17:09.575352+0000","last_peered":"2024-09-16T00:17:09.575352+0000","last_clean":"2024-09-16T00:17:09.575352+0000","last_became_active":"2024-09-16T00:15:04.349124+0000","last_became_peered":"2024-09-16T00:15:04.349124+0000","last_unstale":"2024-09-16T00:17:09.575352+0000","last_undegraded":"2024-09-16T00:17:09.575352+0000","last_fullsized":"2024-09-16T00:17:09.575352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:40.859220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576904+0000","last_change":"2024-09-16T00:15:13.581718+0000","last_active":"2024-09-16T00:17:09.576904+0000","last_peered":"2024-09-16T00:17:09.576904+0000","last_clean":"2024-09-16T00:17:09.576904+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:17:09.576904+0000","last_undegraded":"2024-09-16T00:17:09.576904+0000","last_fullsized":"2024-09-16T00:17:09.576904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:08.810606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579555+0000","last_change":"2024-09-16T00:15:06.544507+0000","last_active":"2024-09-16T00:17:09.579555+0000","last_peered":"2024-09-16T00:17:09.579555+0000","last_clean":"2024-09-16T00:17:09.579555+0000","last_became_active":"2024-09-16T00:15:06.544178+0000","last_became_peered":"2024-09-16T00:15:06.544178+0000","last_unstale":"2024-09-16T00:17:09.579555+0000","last_undegraded":"2024-09-16T00:17:09.579555+0000","last_fullsized":"2024-09-16T00:17:09.579555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:38.494266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'39","reported_seq":325,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:38.172320+0000","last_change":"2024-09-16T00:16:09.039763+0000","last_active":"2024-09-16T00:17:38.172320+0000","last_peered":"2024-09-16T00:17:38.172320+0000","last_clean":"2024-09-16T00:17:38.172320+0000","last_became_active":"2024-09-16T00:16:09.022939+0000","last_became_peered":"2024-09-16T00:16:09.022939+0000","last_unstale":"2024-09-16T00:17:38.172320+0000","last_undegraded":"2024-09-16T00:17:38.172320+0000","last_fullsized":"2024-09-16T00:17:38.172320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:56.431159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71774,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":226,"num_read_kb":174,"num_write":40,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60943,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576887+0000","last_change":"2024-09-16T00:15:13.585928+0000","last_active":"2024-09-16T00:17:09.576887+0000","last_peered":"2024-09-16T00:17:09.576887+0000","last_clean":"2024-09-16T00:17:09.576887+0000","last_became_active":"2024-09-16T00:15:13.582940+0000","last_became_peered":"2024-09-16T00:15:13.582940+0000","last_unstale":"2024-09-16T00:17:09.576887+0000","last_undegraded":"2024-09-16T00:17:09.576887+0000","last_fullsized":"2024-09-16T00:17:09.576887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:57:13.251978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580875+0000","last_change":"2024-09-16T00:15:06.546238+0000","last_active":"2024-09-16T00:17:09.580875+0000","last_peered":"2024-09-16T00:17:09.580875+0000","last_clean":"2024-09-16T00:17:09.580875+0000","last_became_active":"2024-09-16T00:15:06.545919+0000","last_became_peered":"2024-09-16T00:15:06.545919+0000","last_unstale":"2024-09-16T00:17:09.580875+0000","last_undegraded":"2024-09-16T00:17:09.580875+0000","last_fullsized":"2024-09-16T00:17:09.580875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:09.180832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577083+0000","last_change":"2024-09-16T00:15:04.376028+0000","last_active":"2024-09-16T00:17:09.577083+0000","last_peered":"2024-09-16T00:17:09.577083+0000","last_clean":"2024-09-16T00:17:09.577083+0000","last_became_active":"2024-09-16T00:15:04.375633+0000","last_became_peered":"2024-09-16T00:15:04.375633+0000","last_unstale":"2024-09-16T00:17:09.577083+0000","last_undegraded":"2024-09-16T00:17:09.577083+0000","last_fullsized":"2024-09-16T00:17:09.577083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:44.446342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575275+0000","last_change":"2024-09-16T00:16:09.025524+0000","last_active":"2024-09-16T00:17:09.575275+0000","last_peered":"2024-09-16T00:17:09.575275+0000","last_clean":"2024-09-16T00:17:09.575275+0000","last_became_active":"2024-09-16T00:16:09.025035+0000","last_became_peered":"2024-09-16T00:16:09.025035+0000","last_unstale":"2024-09-16T00:17:09.575275+0000","last_undegraded":"2024-09-16T00:17:09.575275+0000","last_fullsized":"2024-09-16T00:17:09.575275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:12.367244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577617+0000","last_change":"2024-09-16T00:15:06.550864+0000","last_active":"2024-09-16T00:17:09.577617+0000","last_peered":"2024-09-16T00:17:09.577617+0000","last_clean":"2024-09-16T00:17:09.577617+0000","last_became_active":"2024-09-16T00:15:06.547956+0000","last_became_peered":"2024-09-16T00:15:06.547956+0000","last_unstale":"2024-09-16T00:17:09.577617+0000","last_undegraded":"2024-09-16T00:17:09.577617+0000","last_fullsized":"2024-09-16T00:17:09.577617+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:26.102175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580692+0000","last_change":"2024-09-16T00:15:04.350349+0000","last_active":"2024-09-16T00:17:09.580692+0000","last_peered":"2024-09-16T00:17:09.580692+0000","last_clean":"2024-09-16T00:17:09.580692+0000","last_became_active":"2024-09-16T00:15:04.349951+0000","last_became_peered":"2024-09-16T00:15:04.349951+0000","last_unstale":"2024-09-16T00:17:09.580692+0000","last_undegraded":"2024-09-16T00:17:09.580692+0000","last_fullsized":"2024-09-16T00:17:09.580692+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:46.000831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577297+0000","last_change":"2024-09-16T00:16:09.025774+0000","last_active":"2024-09-16T00:17:09.577297+0000","last_peered":"2024-09-16T00:17:09.577297+0000","last_clean":"2024-09-16T00:17:09.577297+0000","last_became_active":"2024-09-16T00:16:09.025308+0000","last_became_peered":"2024-09-16T00:16:09.025308+0000","last_unstale":"2024-09-16T00:17:09.577297+0000","last_undegraded":"2024-09-16T00:17:09.577297+0000","last_fullsized":"2024-09-16T00:17:09.577297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:16.664850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580788+0000","last_change":"2024-09-16T00:15:06.552919+0000","last_active":"2024-09-16T00:17:09.580788+0000","last_peered":"2024-09-16T00:17:09.580788+0000","last_clean":"2024-09-16T00:17:09.580788+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:17:09.580788+0000","last_undegraded":"2024-09-16T00:17:09.580788+0000","last_fullsized":"2024-09-16T00:17:09.580788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:47:38.651837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":150,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:36.325025+0000","last_change":"2024-09-16T00:15:04.351116+0000","last_active":"2024-09-16T00:17:36.325025+0000","last_peered":"2024-09-16T00:17:36.325025+0000","last_clean":"2024-09-16T00:17:36.325025+0000","last_became_active":"2024-09-16T00:15:04.350728+0000","last_became_peered":"2024-09-16T00:15:04.350728+0000","last_unstale":"2024-09-16T00:17:36.325025+0000","last_undegraded":"2024-09-16T00:17:36.325025+0000","last_fullsized":"2024-09-16T00:17:36.325025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:38.314019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579322+0000","last_change":"2024-09-16T00:15:13.575961+0000","last_active":"2024-09-16T00:17:09.579322+0000","last_peered":"2024-09-16T00:17:09.579322+0000","last_clean":"2024-09-16T00:17:09.579322+0000","last_became_active":"2024-09-16T00:15:13.575550+0000","last_became_peered":"2024-09-16T00:15:13.575550+0000","last_unstale":"2024-09-16T00:17:09.579322+0000","last_undegraded":"2024-09-16T00:17:09.579322+0000","last_fullsized":"2024-09-16T00:17:09.579322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:05.515588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577699+0000","last_change":"2024-09-16T00:15:13.583583+0000","last_active":"2024-09-16T00:17:09.577699+0000","last_peered":"2024-09-16T00:17:09.577699+0000","last_clean":"2024-09-16T00:17:09.577699+0000","last_became_active":"2024-09-16T00:15:13.583145+0000","last_became_peered":"2024-09-16T00:15:13.583145+0000","last_unstale":"2024-09-16T00:17:09.577699+0000","last_undegraded":"2024-09-16T00:17:09.577699+0000","last_fullsized":"2024-09-16T00:17:09.577699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:07.320337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.719776+0000","last_change":"2024-09-16T00:17:09.719269+0000","last_active":"2024-09-16T00:17:09.719776+0000","last_peered":"2024-09-16T00:17:09.719776+0000","last_clean":"2024-09-16T00:17:09.719776+0000","last_became_active":"2024-09-16T00:17:09.718887+0000","last_became_peered":"2024-09-16T00:17:09.718887+0000","last_unstale":"2024-09-16T00:17:09.719776+0000","last_undegraded":"2024-09-16T00:17:09.719776+0000","last_fullsized":"2024-09-16T00:17:09.719776+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:14:51.675738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577333+0000","last_change":"2024-09-16T00:15:04.350826+0000","last_active":"2024-09-16T00:17:09.577333+0000","last_peered":"2024-09-16T00:17:09.577333+0000","last_clean":"2024-09-16T00:17:09.577333+0000","last_became_active":"2024-09-16T00:15:04.350294+0000","last_became_peered":"2024-09-16T00:15:04.350294+0000","last_unstale":"2024-09-16T00:17:09.577333+0000","last_undegraded":"2024-09-16T00:17:09.577333+0000","last_fullsized":"2024-09-16T00:17:09.577333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:49.146709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781194+0000","last_change":"2024-09-16T00:15:13.580184+0000","last_active":"2024-09-16T00:17:09.781194+0000","last_peered":"2024-09-16T00:17:09.781194+0000","last_clean":"2024-09-16T00:17:09.781194+0000","last_became_active":"2024-09-16T00:15:13.579849+0000","last_became_peered":"2024-09-16T00:15:13.579849+0000","last_unstale":"2024-09-16T00:17:09.781194+0000","last_undegraded":"2024-09-16T00:17:09.781194+0000","last_fullsized":"2024-09-16T00:17:09.781194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:40.613127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579257+0000","last_change":"2024-09-16T00:15:06.545796+0000","last_active":"2024-09-16T00:17:09.579257+0000","last_peered":"2024-09-16T00:17:09.579257+0000","last_clean":"2024-09-16T00:17:09.579257+0000","last_became_active":"2024-09-16T00:15:06.544815+0000","last_became_peered":"2024-09-16T00:15:06.544815+0000","last_unstale":"2024-09-16T00:17:09.579257+0000","last_undegraded":"2024-09-16T00:17:09.579257+0000","last_fullsized":"2024-09-16T00:17:09.579257+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:51:25.921949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718347+0000","last_change":"2024-09-16T00:15:04.340101+0000","last_active":"2024-09-16T00:17:09.718347+0000","last_peered":"2024-09-16T00:17:09.718347+0000","last_clean":"2024-09-16T00:17:09.718347+0000","last_became_active":"2024-09-16T00:15:04.339788+0000","last_became_peered":"2024-09-16T00:15:04.339788+0000","last_unstale":"2024-09-16T00:17:09.718347+0000","last_undegraded":"2024-09-16T00:17:09.718347+0000","last_fullsized":"2024-09-16T00:17:09.718347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:08.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578011+0000","last_change":"2024-09-16T00:16:09.028910+0000","last_active":"2024-09-16T00:17:09.578011+0000","last_peered":"2024-09-16T00:17:09.578011+0000","last_clean":"2024-09-16T00:17:09.578011+0000","last_became_active":"2024-09-16T00:16:09.023302+0000","last_became_peered":"2024-09-16T00:16:09.023302+0000","last_unstale":"2024-09-16T00:17:09.578011+0000","last_undegraded":"2024-09-16T00:17:09.578011+0000","last_fullsized":"2024-09-16T00:17:09.578011+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:35.293676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581116+0000","last_change":"2024-09-16T00:17:09.581116+0000","last_active":"2024-09-16T00:17:09.581116+0000","last_peered":"2024-09-16T00:17:09.581116+0000","last_clean":"2024-09-16T00:17:09.581116+0000","last_became_active":"2024-09-16T00:17:09.580552+0000","last_became_peered":"2024-09-16T00:17:09.580552+0000","last_unstale":"2024-09-16T00:17:09.581116+0000","last_undegraded":"2024-09-16T00:17:09.581116+0000","last_fullsized":"2024-09-16T00:17:09.581116+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:35.975542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574745+0000","last_change":"2024-09-16T00:15:13.581966+0000","last_active":"2024-09-16T00:17:09.574745+0000","last_peered":"2024-09-16T00:17:09.574745+0000","last_clean":"2024-09-16T00:17:09.574745+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:17:09.574745+0000","last_undegraded":"2024-09-16T00:17:09.574745+0000","last_fullsized":"2024-09-16T00:17:09.574745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.132773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":151,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:34.322964+0000","last_change":"2024-09-16T00:15:04.351765+0000","last_active":"2024-09-16T00:17:34.322964+0000","last_peered":"2024-09-16T00:17:34.322964+0000","last_clean":"2024-09-16T00:17:34.322964+0000","last_became_active":"2024-09-16T00:15:04.351398+0000","last_became_peered":"2024-09-16T00:15:04.351398+0000","last_unstale":"2024-09-16T00:17:34.322964+0000","last_undegraded":"2024-09-16T00:17:34.322964+0000","last_fullsized":"2024-09-16T00:17:34.322964+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:14:28.762657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.583630+0000","last_change":"2024-09-16T00:17:09.583255+0000","last_active":"2024-09-16T00:17:09.583630+0000","last_peered":"2024-09-16T00:17:09.583630+0000","last_clean":"2024-09-16T00:17:09.583630+0000","last_became_active":"2024-09-16T00:17:09.582793+0000","last_became_peered":"2024-09-16T00:17:09.582793+0000","last_unstale":"2024-09-16T00:17:09.583630+0000","last_undegraded":"2024-09-16T00:17:09.583630+0000","last_fullsized":"2024-09-16T00:17:09.583630+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:28.087903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574875+0000","last_change":"2024-09-16T00:15:13.576877+0000","last_active":"2024-09-16T00:17:09.574875+0000","last_peered":"2024-09-16T00:17:09.574875+0000","last_clean":"2024-09-16T00:17:09.574875+0000","last_became_active":"2024-09-16T00:15:13.576480+0000","last_became_peered":"2024-09-16T00:15:13.576480+0000","last_unstale":"2024-09-16T00:17:09.574875+0000","last_undegraded":"2024-09-16T00:17:09.574875+0000","last_fullsized":"2024-09-16T00:17:09.574875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:47.322219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780925+0000","last_change":"2024-09-16T00:15:06.550329+0000","last_active":"2024-09-16T00:17:09.780925+0000","last_peered":"2024-09-16T00:17:09.780925+0000","last_clean":"2024-09-16T00:17:09.780925+0000","last_became_active":"2024-09-16T00:15:06.550008+0000","last_became_peered":"2024-09-16T00:15:06.550008+0000","last_unstale":"2024-09-16T00:17:09.780925+0000","last_undegraded":"2024-09-16T00:17:09.780925+0000","last_fullsized":"2024-09-16T00:17:09.780925+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:38.096126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582169+0000","last_change":"2024-09-16T00:17:09.580839+0000","last_active":"2024-09-16T00:17:09.582169+0000","last_peered":"2024-09-16T00:17:09.582169+0000","last_clean":"2024-09-16T00:17:09.582169+0000","last_became_active":"2024-09-16T00:17:09.580381+0000","last_became_peered":"2024-09-16T00:17:09.580381+0000","last_unstale":"2024-09-16T00:17:09.582169+0000","last_undegraded":"2024-09-16T00:17:09.582169+0000","last_fullsized":"2024-09-16T00:17:09.582169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:35.425316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574792+0000","last_change":"2024-09-16T00:15:06.546713+0000","last_active":"2024-09-16T00:17:09.574792+0000","last_peered":"2024-09-16T00:17:09.574792+0000","last_clean":"2024-09-16T00:17:09.574792+0000","last_became_active":"2024-09-16T00:15:06.546209+0000","last_became_peered":"2024-09-16T00:15:06.546209+0000","last_unstale":"2024-09-16T00:17:09.574792+0000","last_undegraded":"2024-09-16T00:17:09.574792+0000","last_fullsized":"2024-09-16T00:17:09.574792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:43.913748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577575+0000","last_change":"2024-09-16T00:15:06.550490+0000","last_active":"2024-09-16T00:17:09.577575+0000","last_peered":"2024-09-16T00:17:09.577575+0000","last_clean":"2024-09-16T00:17:09.577575+0000","last_became_active":"2024-09-16T00:15:06.549311+0000","last_became_peered":"2024-09-16T00:15:06.549311+0000","last_unstale":"2024-09-16T00:17:09.577575+0000","last_undegraded":"2024-09-16T00:17:09.577575+0000","last_fullsized":"2024-09-16T00:17:09.577575+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:10:01.995357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718148+0000","last_change":"2024-09-16T00:15:06.548115+0000","last_active":"2024-09-16T00:17:09.718148+0000","last_peered":"2024-09-16T00:17:09.718148+0000","last_clean":"2024-09-16T00:17:09.718148+0000","last_became_active":"2024-09-16T00:15:06.547782+0000","last_became_peered":"2024-09-16T00:15:06.547782+0000","last_unstale":"2024-09-16T00:17:09.718148+0000","last_undegraded":"2024-09-16T00:17:09.718148+0000","last_fullsized":"2024-09-16T00:17:09.718148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:53.228222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781260+0000","last_change":"2024-09-16T00:15:06.554396+0000","last_active":"2024-09-16T00:17:09.781260+0000","last_peered":"2024-09-16T00:17:09.781260+0000","last_clean":"2024-09-16T00:17:09.781260+0000","last_became_active":"2024-09-16T00:15:06.554012+0000","last_became_peered":"2024-09-16T00:15:06.554012+0000","last_unstale":"2024-09-16T00:17:09.781260+0000","last_undegraded":"2024-09-16T00:17:09.781260+0000","last_fullsized":"2024-09-16T00:17:09.781260+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:31.464098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575494+0000","last_change":"2024-09-16T00:15:06.549969+0000","last_active":"2024-09-16T00:17:09.575494+0000","last_peered":"2024-09-16T00:17:09.575494+0000","last_clean":"2024-09-16T00:17:09.575494+0000","last_became_active":"2024-09-16T00:15:06.549531+0000","last_became_peered":"2024-09-16T00:15:06.549531+0000","last_unstale":"2024-09-16T00:17:09.575494+0000","last_undegraded":"2024-09-16T00:17:09.575494+0000","last_fullsized":"2024-09-16T00:17:09.575494+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:37.883612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579397+0000","last_change":"2024-09-16T00:15:06.547734+0000","last_active":"2024-09-16T00:17:09.579397+0000","last_peered":"2024-09-16T00:17:09.579397+0000","last_clean":"2024-09-16T00:17:09.579397+0000","last_became_active":"2024-09-16T00:15:06.547400+0000","last_became_peered":"2024-09-16T00:15:06.547400+0000","last_unstale":"2024-09-16T00:17:09.579397+0000","last_undegraded":"2024-09-16T00:17:09.579397+0000","last_fullsized":"2024-09-16T00:17:09.579397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:54.455313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576721+0000","last_change":"2024-09-16T00:15:06.551965+0000","last_active":"2024-09-16T00:17:09.576721+0000","last_peered":"2024-09-16T00:17:09.576721+0000","last_clean":"2024-09-16T00:17:09.576721+0000","last_became_active":"2024-09-16T00:15:06.551602+0000","last_became_peered":"2024-09-16T00:15:06.551602+0000","last_unstale":"2024-09-16T00:17:09.576721+0000","last_undegraded":"2024-09-16T00:17:09.576721+0000","last_fullsized":"2024-09-16T00:17:09.576721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:34.951960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579571+0000","last_change":"2024-09-16T00:15:06.549340+0000","last_active":"2024-09-16T00:17:09.579571+0000","last_peered":"2024-09-16T00:17:09.579571+0000","last_clean":"2024-09-16T00:17:09.579571+0000","last_became_active":"2024-09-16T00:15:06.547424+0000","last_became_peered":"2024-09-16T00:15:06.547424+0000","last_unstale":"2024-09-16T00:17:09.579571+0000","last_undegraded":"2024-09-16T00:17:09.579571+0000","last_fullsized":"2024-09-16T00:17:09.579571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:12.087535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580115+0000","last_change":"2024-09-16T00:15:06.555216+0000","last_active":"2024-09-16T00:17:09.580115+0000","last_peered":"2024-09-16T00:17:09.580115+0000","last_clean":"2024-09-16T00:17:09.580115+0000","last_became_active":"2024-09-16T00:15:06.554896+0000","last_became_peered":"2024-09-16T00:15:06.554896+0000","last_unstale":"2024-09-16T00:17:09.580115+0000","last_undegraded":"2024-09-16T00:17:09.580115+0000","last_fullsized":"2024-09-16T00:17:09.580115+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:20.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717980+0000","last_change":"2024-09-16T00:15:06.549752+0000","last_active":"2024-09-16T00:17:09.717980+0000","last_peered":"2024-09-16T00:17:09.717980+0000","last_clean":"2024-09-16T00:17:09.717980+0000","last_became_active":"2024-09-16T00:15:06.549350+0000","last_became_peered":"2024-09-16T00:15:06.549350+0000","last_unstale":"2024-09-16T00:17:09.717980+0000","last_undegraded":"2024-09-16T00:17:09.717980+0000","last_fullsized":"2024-09-16T00:17:09.717980+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:48.615123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781099+0000","last_change":"2024-09-16T00:15:06.549159+0000","last_active":"2024-09-16T00:17:09.781099+0000","last_peered":"2024-09-16T00:17:09.781099+0000","last_clean":"2024-09-16T00:17:09.781099+0000","last_became_active":"2024-09-16T00:15:06.548408+0000","last_became_peered":"2024-09-16T00:15:06.548408+0000","last_unstale":"2024-09-16T00:17:09.781099+0000","last_undegraded":"2024-09-16T00:17:09.781099+0000","last_fullsized":"2024-09-16T00:17:09.781099+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:23.941809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575144+0000","last_change":"2024-09-16T00:15:06.544091+0000","last_active":"2024-09-16T00:17:09.575144+0000","last_peered":"2024-09-16T00:17:09.575144+0000","last_clean":"2024-09-16T00:17:09.575144+0000","last_became_active":"2024-09-16T00:15:06.543761+0000","last_became_peered":"2024-09-16T00:15:06.543761+0000","last_unstale":"2024-09-16T00:17:09.575144+0000","last_undegraded":"2024-09-16T00:17:09.575144+0000","last_fullsized":"2024-09-16T00:17:09.575144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:36:43.253405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582256+0000","last_change":"2024-09-16T00:17:09.581748+0000","last_active":"2024-09-16T00:17:09.582256+0000","last_peered":"2024-09-16T00:17:09.582256+0000","last_clean":"2024-09-16T00:17:09.582256+0000","last_became_active":"2024-09-16T00:17:09.581207+0000","last_became_peered":"2024-09-16T00:17:09.581207+0000","last_unstale":"2024-09-16T00:17:09.582256+0000","last_undegraded":"2024-09-16T00:17:09.582256+0000","last_fullsized":"2024-09-16T00:17:09.582256+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:21.521929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781344+0000","last_change":"2024-09-16T00:15:06.557162+0000","last_active":"2024-09-16T00:17:09.781344+0000","last_peered":"2024-09-16T00:17:09.781344+0000","last_clean":"2024-09-16T00:17:09.781344+0000","last_became_active":"2024-09-16T00:15:06.556831+0000","last_became_peered":"2024-09-16T00:15:06.556831+0000","last_unstale":"2024-09-16T00:17:09.781344+0000","last_undegraded":"2024-09-16T00:17:09.781344+0000","last_fullsized":"2024-09-16T00:17:09.781344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.236915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579887+0000","last_change":"2024-09-16T00:15:06.556161+0000","last_active":"2024-09-16T00:17:09.579887+0000","last_peered":"2024-09-16T00:17:09.579887+0000","last_clean":"2024-09-16T00:17:09.579887+0000","last_became_active":"2024-09-16T00:15:06.555819+0000","last_became_peered":"2024-09-16T00:15:06.555819+0000","last_unstale":"2024-09-16T00:17:09.579887+0000","last_undegraded":"2024-09-16T00:17:09.579887+0000","last_fullsized":"2024-09-16T00:17:09.579887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:29.929977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.583819+0000","last_change":"2024-09-16T00:17:09.583183+0000","last_active":"2024-09-16T00:17:09.583819+0000","last_peered":"2024-09-16T00:17:09.583819+0000","last_clean":"2024-09-16T00:17:09.583819+0000","last_became_active":"2024-09-16T00:17:09.582663+0000","last_became_peered":"2024-09-16T00:17:09.582663+0000","last_unstale":"2024-09-16T00:17:09.583819+0000","last_undegraded":"2024-09-16T00:17:09.583819+0000","last_fullsized":"2024-09-16T00:17:09.583819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:37.544124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577469+0000","last_change":"2024-09-16T00:15:06.547114+0000","last_active":"2024-09-16T00:17:09.577469+0000","last_peered":"2024-09-16T00:17:09.577469+0000","last_clean":"2024-09-16T00:17:09.577469+0000","last_became_active":"2024-09-16T00:15:06.546786+0000","last_became_peered":"2024-09-16T00:15:06.546786+0000","last_unstale":"2024-09-16T00:17:09.577469+0000","last_undegraded":"2024-09-16T00:17:09.577469+0000","last_fullsized":"2024-09-16T00:17:09.577469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:13:26.925603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717955+0000","last_change":"2024-09-16T00:15:06.541168+0000","last_active":"2024-09-16T00:17:09.717955+0000","last_peered":"2024-09-16T00:17:09.717955+0000","last_clean":"2024-09-16T00:17:09.717955+0000","last_became_active":"2024-09-16T00:15:06.540848+0000","last_became_peered":"2024-09-16T00:15:06.540848+0000","last_unstale":"2024-09-16T00:17:09.717955+0000","last_undegraded":"2024-09-16T00:17:09.717955+0000","last_fullsized":"2024-09-16T00:17:09.717955+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.958600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575843+0000","last_change":"2024-09-16T00:15:06.547429+0000","last_active":"2024-09-16T00:17:09.575843+0000","last_peered":"2024-09-16T00:17:09.575843+0000","last_clean":"2024-09-16T00:17:09.575843+0000","last_became_active":"2024-09-16T00:15:06.547081+0000","last_became_peered":"2024-09-16T00:15:06.547081+0000","last_unstale":"2024-09-16T00:17:09.575843+0000","last_undegraded":"2024-09-16T00:17:09.575843+0000","last_fullsized":"2024-09-16T00:17:09.575843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:43.396313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581339+0000","last_change":"2024-09-16T00:15:06.551811+0000","last_active":"2024-09-16T00:17:09.581339+0000","last_peered":"2024-09-16T00:17:09.581339+0000","last_clean":"2024-09-16T00:17:09.581339+0000","last_became_active":"2024-09-16T00:15:06.551424+0000","last_became_peered":"2024-09-16T00:15:06.551424+0000","last_unstale":"2024-09-16T00:17:09.581339+0000","last_undegraded":"2024-09-16T00:17:09.581339+0000","last_fullsized":"2024-09-16T00:17:09.581339+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.943070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578077+0000","last_change":"2024-09-16T00:15:06.544697+0000","last_active":"2024-09-16T00:17:09.578077+0000","last_peered":"2024-09-16T00:17:09.578077+0000","last_clean":"2024-09-16T00:17:09.578077+0000","last_became_active":"2024-09-16T00:15:06.544182+0000","last_became_peered":"2024-09-16T00:15:06.544182+0000","last_unstale":"2024-09-16T00:17:09.578077+0000","last_undegraded":"2024-09-16T00:17:09.578077+0000","last_fullsized":"2024-09-16T00:17:09.578077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:45.929092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580342+0000","last_change":"2024-09-16T00:15:06.551488+0000","last_active":"2024-09-16T00:17:09.580342+0000","last_peered":"2024-09-16T00:17:09.580342+0000","last_clean":"2024-09-16T00:17:09.580342+0000","last_became_active":"2024-09-16T00:15:06.551171+0000","last_became_peered":"2024-09-16T00:15:06.551171+0000","last_unstale":"2024-09-16T00:17:09.580342+0000","last_undegraded":"2024-09-16T00:17:09.580342+0000","last_fullsized":"2024-09-16T00:17:09.580342+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:23.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579150+0000","last_change":"2024-09-16T00:15:06.544043+0000","last_active":"2024-09-16T00:17:09.579150+0000","last_peered":"2024-09-16T00:17:09.579150+0000","last_clean":"2024-09-16T00:17:09.579150+0000","last_became_active":"2024-09-16T00:15:06.543014+0000","last_became_peered":"2024-09-16T00:15:06.543014+0000","last_unstale":"2024-09-16T00:17:09.579150+0000","last_undegraded":"2024-09-16T00:17:09.579150+0000","last_fullsized":"2024-09-16T00:17:09.579150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:23.904810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581774+0000","last_change":"2024-09-16T00:15:06.546788+0000","last_active":"2024-09-16T00:17:09.581774+0000","last_peered":"2024-09-16T00:17:09.581774+0000","last_clean":"2024-09-16T00:17:09.581774+0000","last_became_active":"2024-09-16T00:15:06.546333+0000","last_became_peered":"2024-09-16T00:15:06.546333+0000","last_unstale":"2024-09-16T00:17:09.581774+0000","last_undegraded":"2024-09-16T00:17:09.581774+0000","last_fullsized":"2024-09-16T00:17:09.581774+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:09.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.720091+0000","last_change":"2024-09-16T00:17:09.719618+0000","last_active":"2024-09-16T00:17:09.720091+0000","last_peered":"2024-09-16T00:17:09.720091+0000","last_clean":"2024-09-16T00:17:09.720091+0000","last_became_active":"2024-09-16T00:17:09.719172+0000","last_became_peered":"2024-09-16T00:17:09.719172+0000","last_unstale":"2024-09-16T00:17:09.720091+0000","last_undegraded":"2024-09-16T00:17:09.720091+0000","last_fullsized":"2024-09-16T00:17:09.720091+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:51.709719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579341+0000","last_change":"2024-09-16T00:15:06.540675+0000","last_active":"2024-09-16T00:17:09.579341+0000","last_peered":"2024-09-16T00:17:09.579341+0000","last_clean":"2024-09-16T00:17:09.579341+0000","last_became_active":"2024-09-16T00:15:06.540333+0000","last_became_peered":"2024-09-16T00:15:06.540333+0000","last_unstale":"2024-09-16T00:17:09.579341+0000","last_undegraded":"2024-09-16T00:17:09.579341+0000","last_fullsized":"2024-09-16T00:17:09.579341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:17.166987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575821+0000","last_change":"2024-09-16T00:15:06.548758+0000","last_active":"2024-09-16T00:17:09.575821+0000","last_peered":"2024-09-16T00:17:09.575821+0000","last_clean":"2024-09-16T00:17:09.575821+0000","last_became_active":"2024-09-16T00:15:06.548397+0000","last_became_peered":"2024-09-16T00:15:06.548397+0000","last_unstale":"2024-09-16T00:17:09.575821+0000","last_undegraded":"2024-09-16T00:17:09.575821+0000","last_fullsized":"2024-09-16T00:17:09.575821+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:50.320170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576644+0000","last_change":"2024-09-16T00:15:06.554898+0000","last_active":"2024-09-16T00:17:09.576644+0000","last_peered":"2024-09-16T00:17:09.576644+0000","last_clean":"2024-09-16T00:17:09.576644+0000","last_became_active":"2024-09-16T00:15:06.553005+0000","last_became_peered":"2024-09-16T00:15:06.553005+0000","last_unstale":"2024-09-16T00:17:09.576644+0000","last_undegraded":"2024-09-16T00:17:09.576644+0000","last_fullsized":"2024-09-16T00:17:09.576644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:29.630911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577231+0000","last_change":"2024-09-16T00:15:06.547265+0000","last_active":"2024-09-16T00:17:09.577231+0000","last_peered":"2024-09-16T00:17:09.577231+0000","last_clean":"2024-09-16T00:17:09.577231+0000","last_became_active":"2024-09-16T00:15:06.546938+0000","last_became_peered":"2024-09-16T00:15:06.546938+0000","last_unstale":"2024-09-16T00:17:09.577231+0000","last_undegraded":"2024-09-16T00:17:09.577231+0000","last_fullsized":"2024-09-16T00:17:09.577231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:40.247343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781308+0000","last_change":"2024-09-16T00:15:06.556735+0000","last_active":"2024-09-16T00:17:09.781308+0000","last_peered":"2024-09-16T00:17:09.781308+0000","last_clean":"2024-09-16T00:17:09.781308+0000","last_became_active":"2024-09-16T00:15:06.556324+0000","last_became_peered":"2024-09-16T00:15:06.556324+0000","last_unstale":"2024-09-16T00:17:09.781308+0000","last_undegraded":"2024-09-16T00:17:09.781308+0000","last_fullsized":"2024-09-16T00:17:09.781308+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:25.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":76311,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":669,"num_read_kb":407,"num_write":182,"num_write_kb":135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":62107,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":228933,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249414,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579948,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":517057,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808881,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36468,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708684,"statfs":{"total":95995035648,"available":95957692416,"internally_reserved":0,"allocated":1150976,"data_stored":1177026,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":521389,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201339,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":516631,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430273,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1060864,"data_stored":1113658,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626502,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":512469,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855435,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1069056,"data_stored":1108319,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986256,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1044480,"data_stored":590402,"data_compressed":39297,"data_compressed_allocated":53248,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215189,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":512663,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444123,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1007616,"data_stored":513147,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607648,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":584735,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1015808,"data_stored":513857,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":72356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":73927,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":74279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T00:17:41.244 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-16T00:17:41.244 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T00:17:41.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.585+0000 7f16b5758640 1 -- 172.21.15.33:0/1227245544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010c170 msgr2=0x7f16b010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:41.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.585+0000 7f16b5758640 1 --2- 172.21.15.33:0/1227245544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010c170 0x7f16b010e610 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009ca0 tx=0x7f16a802a950 comp rx=0 tx=0).stop 2024-09-16T00:17:41.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.585+0000 7f16b5758640 1 -- 172.21.15.33:0/1227245544 shutdown_connections 2024-09-16T00:17:41.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.585+0000 7f16b5758640 1 --2- 172.21.15.33:0/1227245544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010c170 0x7f16b010e610 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.585+0000 7f16b5758640 1 --2- 172.21.15.33:0/1227245544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b0103660 0x7f16b010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.585+0000 7f16b5758640 1 --2- 172.21.15.33:0/1227245544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b0102d40 0x7f16b0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.585+0000 7f16b5758640 1 -- 172.21.15.33:0/1227245544 >> 172.21.15.33:0/1227245544 conn(0x7f16b00fc7a0 msgr2=0x7f16b00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:41.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.585+0000 7f16b5758640 1 -- 172.21.15.33:0/1227245544 shutdown_connections 2024-09-16T00:17:41.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.585+0000 7f16b5758640 1 -- 172.21.15.33:0/1227245544 wait complete. 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.586+0000 7f16b5758640 1 Processor -- start 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.586+0000 7f16b5758640 1 -- start start 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16b5758640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b0102d40 0x7f16b01009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16b5758640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b0103660 0x7f16b0100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16b5758640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010c170 0x7f16b0101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16b5758640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f16b0111f30 con 0x7f16b010c170 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16b5758640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f16b0111db0 con 0x7f16b0103660 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16b5758640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01120b0 con 0x7f16b0102d40 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16af7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010c170 0x7f16b0101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16af7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010c170 0x7f16b0101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58926/0 (socket says 172.21.15.33:58926) 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16af7fe640 1 -- 172.21.15.33:0/4098498781 learned_addr learned my addr 172.21.15.33:0/4098498781 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16aeffd640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b0102d40 0x7f16b01009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b0103660 0x7f16b0100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16af7fe640 1 -- 172.21.15.33:0/4098498781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b0102d40 msgr2=0x7f16b01009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16af7fe640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b0102d40 0x7f16b01009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16af7fe640 1 -- 172.21.15.33:0/4098498781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b0103660 msgr2=0x7f16b0100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16af7fe640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b0103660 0x7f16b0100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16af7fe640 1 -- 172.21.15.33:0/4098498781 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b01a6900 con 0x7f16b010c170 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f16af7fe640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010c170 0x7f16b0101420 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f16a8004af0 tx=0x7f16a802f3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.587+0000 7f1697fff640 1 -- 172.21.15.33:0/4098498781 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a8034040 con 0x7f16b010c170 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.588+0000 7f1697fff640 1 -- 172.21.15.33:0/4098498781 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a8030920 con 0x7f16b010c170 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.588+0000 7f16b5758640 1 -- 172.21.15.33:0/4098498781 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b01a6b90 con 0x7f16b010c170 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.588+0000 7f1697fff640 1 -- 172.21.15.33:0/4098498781 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a8004670 con 0x7f16b010c170 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.588+0000 7f16b5758640 1 -- 172.21.15.33:0/4098498781 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b01a7070 con 0x7f16b010c170 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.590+0000 7f1697fff640 1 -- 172.21.15.33:0/4098498781 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f16a8037020 con 0x7f16b010c170 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.590+0000 7f16b7b31740 1 Processor -- start 2024-09-16T00:17:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.590+0000 7f16b7b31740 1 -- start start 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.590+0000 7f16b7b31740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564b4a0 0x55fc1564b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.590+0000 7f16b7b31740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc1564bdc0 0x55fc15656140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.590+0000 7f16b7b31740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc15656680 0x55fc15658a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.590+0000 7f16b7b31740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fc155b12e0 con 0x55fc1564b4a0 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.590+0000 7f16b7b31740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55fc15558ce0 con 0x55fc1564bdc0 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.590+0000 7f16b7b31740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fc155ad2a0 con 0x55fc15656680 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f1697fff640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f16980a32d0 0x7f16980a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16ae7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564b4a0 0x55fc1564b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16ae7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564b4a0 0x55fc1564b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58928/0 (socket says 172.21.15.33:58928) 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16ae7fc640 1 -- 172.21.15.33:0/43223073 learned_addr learned my addr 172.21.15.33:0/43223073 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16af7fe640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f16980a32d0 0x7f16980a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16aeffd640 1 --2- 172.21.15.33:0/43223073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc1564bdc0 0x55fc15656140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/43223073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc15656680 0x55fc15658a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f1697fff640 1 -- 172.21.15.33:0/4098498781 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f16a80f0ab0 con 0x7f16b010c170 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16ae7fc640 1 -- 172.21.15.33:0/43223073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc15656680 msgr2=0x55fc15658a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/43223073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc15656680 0x55fc15658a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16ae7fc640 1 -- 172.21.15.33:0/43223073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc1564bdc0 msgr2=0x55fc15656140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/43223073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc1564bdc0 0x55fc15656140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16ae7fc640 1 -- 172.21.15.33:0/43223073 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc1553d440 con 0x55fc1564b4a0 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/43223073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564b4a0 0x55fc1564b880 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f16a40058c0 tx=0x7f16a4005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943fe2fb0ba0712a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16af7fe640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f16980a32d0 0x7f16980a5790 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f16a8004160 tx=0x7f16a8033000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16957fa640 1 -- 172.21.15.33:0/43223073 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a4004050 con 0x55fc1564b4a0 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16957fa640 1 -- 172.21.15.33:0/43223073 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a40041f0 con 0x55fc1564b4a0 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.591+0000 7f16957fa640 1 -- 172.21.15.33:0/43223073 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a400f720 con 0x55fc1564b4a0 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.592+0000 7f16b7b31740 1 -- 172.21.15.33:0/43223073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564b4a0 msgr2=0x55fc1564b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:41.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.592+0000 7f16b7b31740 1 --2- 172.21.15.33:0/43223073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564b4a0 0x55fc1564b880 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f16a40058c0 tx=0x7f16a4005850 comp rx=0 tx=0).stop 2024-09-16T00:17:41.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.592+0000 7f16b7b31740 1 -- 172.21.15.33:0/43223073 shutdown_connections 2024-09-16T00:17:41.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.592+0000 7f16b7b31740 1 --2- 172.21.15.33:0/43223073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc15656680 0x55fc15658a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.592+0000 7f16b7b31740 1 --2- 172.21.15.33:0/43223073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc1564bdc0 0x55fc15656140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.592+0000 7f16b7b31740 1 --2- 172.21.15.33:0/43223073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564b4a0 0x55fc1564b880 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.592+0000 7f16b7b31740 1 -- 172.21.15.33:0/43223073 >> 172.21.15.33:0/43223073 conn(0x55fc15647a30 msgr2=0x55fc15647e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:41.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.592+0000 7f16b7b31740 1 -- 172.21.15.33:0/43223073 shutdown_connections 2024-09-16T00:17:41.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.592+0000 7f16b7b31740 1 -- 172.21.15.33:0/43223073 wait complete. 2024-09-16T00:17:41.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.592+0000 7f16b7b31740 1 -- start start 2024-09-16T00:17:41.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16b7b31740 1 Processor -- start 2024-09-16T00:17:41.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16b7b31740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc1564b4a0 0x55fc15656020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16b7b31740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc156532d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16b7b31740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc15656680 0x55fc15653870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16b7b31740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fc15659250 con 0x55fc1564bdc0 2024-09-16T00:17:41.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16b7b31740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55fc1556c050 con 0x55fc15656680 2024-09-16T00:17:41.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16b7b31740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fc156593d0 con 0x55fc1564b4a0 2024-09-16T00:17:41.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16b7b31740 10 client.0 _ll_register_callbacks cb 0x55fc155ac160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:41.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16aeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc156532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16aeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc156532d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58932/0 (socket says 172.21.15.33:58932) 2024-09-16T00:17:41.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16aeffd640 1 -- 172.21.15.33:0/525745065 learned_addr learned my addr 172.21.15.33:0/525745065 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:41.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc1564b4a0 0x55fc15656020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc15656680 0x55fc15653870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16aeffd640 1 -- 172.21.15.33:0/525745065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc1564b4a0 msgr2=0x55fc15656020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:41.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16aeffd640 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc1564b4a0 0x55fc15656020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16aeffd640 1 -- 172.21.15.33:0/525745065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc15656680 msgr2=0x55fc15653870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:41.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16aeffd640 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc15656680 0x55fc15653870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16aeffd640 1 -- 172.21.15.33:0/525745065 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc153223c0 con 0x55fc1564bdc0 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc1564b4a0 0x55fc15656020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f16aeffd640 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc156532d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f16a0012740 tx=0x7f16a0012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667fefbf98abfd3d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.593+0000 7f1692ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f1692ff5640 1 -- 172.21.15.33:0/525745065 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a000f920 con 0x55fc1564bdc0 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f1692ff5640 1 -- 172.21.15.33:0/525745065 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f16a000fac0 con 0x55fc1564bdc0 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f16b7b31740 1 -- 172.21.15.33:0/525745065 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16a40041f0 con 0x55fc1564bdc0 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f1692ff5640 1 -- 172.21.15.33:0/525745065 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f16a000fcc0 con 0x55fc1564bdc0 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f16b7b31740 10 client.15360 fetch_fsmap learned FSMap version 26 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f16b7b31740 1 -- 172.21.15.33:0/525745065 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fc15664cf0 con 0x55fc1564bdc0 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f1692ff5640 1 -- 172.21.15.33:0/525745065 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a40041f0 con 0x55fc1564bdc0 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f1692ff5640 1 -- 172.21.15.33:0/525745065 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f16a001c420 con 0x55fc1564bdc0 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f16b7b31740 10 client.15360 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:17:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f16b7b31740 10 client.15360 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f16b7b31740 20 client.15360 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f16b7b31740 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fc15666250 0x55fc15668670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f16b7b31740 4 client.15360 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.594+0000 7f16b7b31740 1 -- 172.21.15.33:0/525745065 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc155be850 con 0x55fc15666250 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.595+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fc15666250 0x55fc15668670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.595+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fc15666250 0x55fc15668670 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a9c8d808daa6810 server_cookie=271212fc0af11c38 in_seq=0 out_seq=0 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.595+0000 7f1692ff5640 10 client.15360 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f1692ff5640 1 -- 172.21.15.33:0/525745065 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fc155be850 con 0x55fc15666250 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f1692ff5640 10 client.15360 handle_command_reply: tid=0 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f16b7b31740 1 client.15360 shutdown 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f16b7b31740 1 -- 172.21.15.33:0/525745065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 msgr2=0x55fc156532d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f16b7b31740 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc156532d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f16a0012740 tx=0x7f16a0012c00 comp rx=0 tx=0).stop 2024-09-16T00:17:41.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f16b7b31740 1 -- 172.21.15.33:0/525745065 shutdown_connections 2024-09-16T00:17:41.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f16b7b31740 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fc15666250 0x55fc15668670 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f16b7b31740 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc15656680 0x55fc15653870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f16b7b31740 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc156532d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f1692ff5640 0 client.15360 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:41.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f16b7b31740 1 --2- 172.21.15.33:0/525745065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc1564b4a0 0x55fc15656020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f16b7b31740 1 -- 172.21.15.33:0/525745065 >> 172.21.15.33:0/525745065 conn(0x55fc15647a30 msgr2=0x55fc1564d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:41.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f16b7b31740 1 -- 172.21.15.33:0/525745065 shutdown_connections 2024-09-16T00:17:41.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f16b7b31740 1 -- 172.21.15.33:0/525745065 wait complete. 2024-09-16T00:17:41.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.596+0000 7f16b7b31740 20 client.15360 trim_cache size 0 max 16384 2024-09-16T00:17:41.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.613+0000 7f16b7b31740 1 Processor -- start 2024-09-16T00:17:41.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16b7b31740 1 -- start start 2024-09-16T00:17:41.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16b7b31740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc1564b4a0 0x55fc156624e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16b7b31740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc1566df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16b7b31740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc15656680 0x55fc1566e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16b7b31740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01b3b00 con 0x55fc1564bdc0 2024-09-16T00:17:41.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16b7b31740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01b13d0 con 0x55fc15656680 2024-09-16T00:17:41.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16b7b31740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01af220 con 0x55fc1564b4a0 2024-09-16T00:17:41.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16aeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc1566df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16aeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc1566df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58946/0 (socket says 172.21.15.33:58946) 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16aeffd640 1 -- 172.21.15.33:0/1451098662 learned_addr learned my addr 172.21.15.33:0/1451098662 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/1451098662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc15656680 0x55fc1566e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16aeffd640 1 -- 172.21.15.33:0/1451098662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc1564b4a0 msgr2=0x55fc156624e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/1451098662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc1564b4a0 0x55fc156624e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16aeffd640 1 --2- 172.21.15.33:0/1451098662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc1564b4a0 0x55fc156624e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16aeffd640 1 -- 172.21.15.33:0/1451098662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc15656680 msgr2=0x55fc1566e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16aeffd640 1 --2- 172.21.15.33:0/1451098662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc15656680 0x55fc1566e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16aeffd640 1 -- 172.21.15.33:0/1451098662 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc15656390 con 0x55fc1564bdc0 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/1451098662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc1564b4a0 0x55fc156624e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.614+0000 7f16aeffd640 1 --2- 172.21.15.33:0/1451098662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc1566df20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f16a001ca80 tx=0x7f16a000f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31bfd2e98a8bfda server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.615+0000 7f1694ff9640 1 -- 172.21.15.33:0/1451098662 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a00122e0 con 0x55fc1564bdc0 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.615+0000 7f1694ff9640 1 -- 172.21.15.33:0/1451098662 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a0012e50 con 0x55fc1564bdc0 2024-09-16T00:17:41.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.615+0000 7f1694ff9640 1 -- 172.21.15.33:0/1451098662 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0005440 con 0x55fc1564bdc0 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.615+0000 7f16b7b31740 1 -- 172.21.15.33:0/1451098662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 msgr2=0x55fc1566df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.615+0000 7f16b7b31740 1 --2- 172.21.15.33:0/1451098662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc1566df20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f16a001ca80 tx=0x7f16a000f850 comp rx=0 tx=0).stop 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.615+0000 7f16b7b31740 1 -- 172.21.15.33:0/1451098662 shutdown_connections 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.615+0000 7f16b7b31740 1 --2- 172.21.15.33:0/1451098662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc15656680 0x55fc1566e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.615+0000 7f16b7b31740 1 --2- 172.21.15.33:0/1451098662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc1566df20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.615+0000 7f16b7b31740 1 --2- 172.21.15.33:0/1451098662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc1564b4a0 0x55fc156624e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.615+0000 7f16b7b31740 1 -- 172.21.15.33:0/1451098662 >> 172.21.15.33:0/1451098662 conn(0x55fc15647a30 msgr2=0x55fc156523c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.615+0000 7f16b7b31740 1 -- 172.21.15.33:0/1451098662 shutdown_connections 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.615+0000 7f16b7b31740 1 -- 172.21.15.33:0/1451098662 wait complete. 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16b7b31740 1 -- start start 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16b7b31740 1 Processor -- start 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16b7b31740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc1564b4a0 0x55fc1565eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16b7b31740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc15659e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16b7b31740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc15656680 0x55fc1565a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16b7b31740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01af890 con 0x55fc1564bdc0 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16b7b31740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01b0d00 con 0x55fc1564b4a0 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16b7b31740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01aff60 con 0x55fc15656680 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16b7b31740 10 client.0 _ll_register_callbacks cb 0x7f16a0009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16aeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc15659e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16aeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc15659e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58954/0 (socket says 172.21.15.33:58954) 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16aeffd640 1 -- 172.21.15.33:0/3132785888 learned_addr learned my addr 172.21.15.33:0/3132785888 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc1564b4a0 0x55fc1565eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc15656680 0x55fc1565a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16aeffd640 1 -- 172.21.15.33:0/3132785888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc15656680 msgr2=0x55fc1565a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16aeffd640 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc15656680 0x55fc1565a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16aeffd640 1 -- 172.21.15.33:0/3132785888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc1564b4a0 msgr2=0x55fc1565eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:41.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16aeffd640 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc1564b4a0 0x55fc1565eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.616+0000 7f16aeffd640 1 -- 172.21.15.33:0/3132785888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc1565a970 con 0x55fc1564bdc0 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f16aeffd640 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc15659e70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f16a001c910 tx=0x7f16a000fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606fcac51515171c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f1692ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f1692ff5640 1 -- 172.21.15.33:0/3132785888 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a001dd00 con 0x55fc1564bdc0 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f1692ff5640 1 -- 172.21.15.33:0/3132785888 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f16a002f040 con 0x55fc1564bdc0 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f16b7b31740 1 -- 172.21.15.33:0/3132785888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16a0012e50 con 0x55fc1564bdc0 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f1692ff5640 1 -- 172.21.15.33:0/3132785888 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f16a00053d0 con 0x55fc1564bdc0 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f16b7b31740 10 client.15366 fetch_fsmap learned FSMap version 26 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f16b7b31740 1 -- 172.21.15.33:0/3132785888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fc156d0c00 con 0x55fc1564bdc0 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f1692ff5640 1 -- 172.21.15.33:0/3132785888 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a0012e50 con 0x55fc1564bdc0 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f1692ff5640 1 -- 172.21.15.33:0/3132785888 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f16a0010550 con 0x55fc1564bdc0 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f16b7b31740 10 client.15366 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:17:41.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f16b7b31740 10 client.15366 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f16b7b31740 20 client.15366 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f16b7b31740 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fc156d1100 0x55fc156d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f16b7b31740 4 client.15366 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.617+0000 7f16b7b31740 1 -- 172.21.15.33:0/3132785888 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fc155b8d60 con 0x55fc156d1100 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fc156d1100 0x55fc156d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f16ae7fc640 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fc156d1100 0x55fc156d3520 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22ad36a429085e46 server_cookie=5374b349d77c0d98 in_seq=0 out_seq=0 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f1692ff5640 10 client.15366 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f1692ff5640 1 -- 172.21.15.33:0/3132785888 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fc155b8d60 con 0x55fc156d1100 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f1692ff5640 10 client.15366 handle_command_reply: tid=0 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f16b7b31740 1 client.15366 shutdown 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f16b7b31740 1 -- 172.21.15.33:0/3132785888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 msgr2=0x55fc15659e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f16b7b31740 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc15659e70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f16a001c910 tx=0x7f16a000fc20 comp rx=0 tx=0).stop 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f16b7b31740 1 -- 172.21.15.33:0/3132785888 shutdown_connections 2024-09-16T00:17:41.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f16b7b31740 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fc156d1100 0x55fc156d3520 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f16b7b31740 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc15656680 0x55fc1565a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f16b7b31740 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc1564bdc0 0x55fc15659e70 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f16b7b31740 1 --2- 172.21.15.33:0/3132785888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc1564b4a0 0x55fc1565eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f1692ff5640 0 client.15366 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:41.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.618+0000 7f16b7b31740 1 -- 172.21.15.33:0/3132785888 >> 172.21.15.33:0/3132785888 conn(0x55fc15647a30 msgr2=0x55fc15654b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:41.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b7b31740 1 -- 172.21.15.33:0/3132785888 shutdown_connections 2024-09-16T00:17:41.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b7b31740 1 -- 172.21.15.33:0/3132785888 wait complete. 2024-09-16T00:17:41.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b7b31740 20 client.15366 trim_cache size 0 max 16384 2024-09-16T00:17:41.632 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:17:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:17:41.633 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:17:41.633 INFO:teuthology.orchestra.run.smithi033.stdout: "96a33d0c-27bf-4b98-aed1-121ec5dad828": { 2024-09-16T00:17:41.633 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:17:41.633 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "96a33d0c-27bf-4b98-aed1-121ec5dad828", 2024-09-16T00:17:41.633 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:17:41.633 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:17:41.633 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:17:41.633 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:17:41.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b5758640 1 -- 172.21.15.33:0/4098498781 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f16980a32d0 msgr2=0x7f16980a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:41.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b5758640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f16980a32d0 0x7f16980a5790 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f16a8004160 tx=0x7f16a8033000 comp rx=0 tx=0).stop 2024-09-16T00:17:41.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b5758640 1 -- 172.21.15.33:0/4098498781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010c170 msgr2=0x7f16b0101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:41.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b5758640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010c170 0x7f16b0101420 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f16a8004af0 tx=0x7f16a802f3a0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b5758640 1 -- 172.21.15.33:0/4098498781 shutdown_connections 2024-09-16T00:17:41.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b5758640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f16980a32d0 0x7f16980a5790 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b5758640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010c170 0x7f16b0101420 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b5758640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b0103660 0x7f16b0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b5758640 1 --2- 172.21.15.33:0/4098498781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b0102d40 0x7f16b01009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:41.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b5758640 1 -- 172.21.15.33:0/4098498781 >> 172.21.15.33:0/4098498781 conn(0x7f16b00fc7a0 msgr2=0x7f16b00fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:41.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b5758640 1 -- 172.21.15.33:0/4098498781 shutdown_connections 2024-09-16T00:17:41.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:41.619+0000 7f16b5758640 1 -- 172.21.15.33:0/4098498781 wait complete. 2024-09-16T00:17:42.335 INFO:tasks.cephfs.filesystem:scrub status for tag:96a33d0c-27bf-4b98-aed1-121ec5dad828 - {'path': '/', 'tag': '96a33d0c-27bf-4b98-aed1-121ec5dad828', 'options': 'recursive,force'} 2024-09-16T00:17:42.400 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:42.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:42 smithi033 ceph-mon[31243]: pgmap v367: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-09-16T00:17:43.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.009+0000 7f2654a2c640 1 -- 172.21.15.33:0/1564201467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2650102790 msgr2=0x7f26501067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:43.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.009+0000 7f2654a2c640 1 --2- 172.21.15.33:0/1564201467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2650102790 0x7f26501067e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2638009900 tx=0x7f263802a410 comp rx=0 tx=0).stop 2024-09-16T00:17:43.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.009+0000 7f2654a2c640 1 -- 172.21.15.33:0/1564201467 shutdown_connections 2024-09-16T00:17:43.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.009+0000 7f2654a2c640 1 --2- 172.21.15.33:0/1564201467 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2650106f10 0x7f265010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:43.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.009+0000 7f2654a2c640 1 --2- 172.21.15.33:0/1564201467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2650102790 0x7f26501067e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:43.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.009+0000 7f2654a2c640 1 --2- 172.21.15.33:0/1564201467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2650101de0 0x7f26501021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:43.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.009+0000 7f2654a2c640 1 -- 172.21.15.33:0/1564201467 >> 172.21.15.33:0/1564201467 conn(0x7f26500fd8c0 msgr2=0x7f26500ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:43.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.009+0000 7f2654a2c640 1 -- 172.21.15.33:0/1564201467 shutdown_connections 2024-09-16T00:17:43.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.009+0000 7f2654a2c640 1 -- 172.21.15.33:0/1564201467 wait complete. 2024-09-16T00:17:43.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.010+0000 7f2654a2c640 1 Processor -- start 2024-09-16T00:17:43.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.010+0000 7f2654a2c640 1 -- start start 2024-09-16T00:17:43.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.010+0000 7f2654a2c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2650101de0 0x7f26500744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:43.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.010+0000 7f2654a2c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2650102790 0x7f2650074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:43.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.010+0000 7f2654a2c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2650106f10 0x7f2650074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:43.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.010+0000 7f2654a2c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2650111f70 con 0x7f2650101de0 2024-09-16T00:17:43.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.010+0000 7f2654a2c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2650111df0 con 0x7f2650102790 2024-09-16T00:17:43.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.010+0000 7f2654a2c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f26501120f0 con 0x7f2650106f10 2024-09-16T00:17:43.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.010+0000 7f264e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2650101de0 0x7f26500744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:43.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.010+0000 7f264e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2650101de0 0x7f26500744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58976/0 (socket says 172.21.15.33:58976) 2024-09-16T00:17:43.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.010+0000 7f264e59c640 1 -- 172.21.15.33:0/3591563384 learned_addr learned my addr 172.21.15.33:0/3591563384 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:43.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f2647fff640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2650102790 0x7f2650074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:43.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f264e59c640 1 -- 172.21.15.33:0/3591563384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2650106f10 msgr2=0x7f2650074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f264ed9d640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2650106f10 0x7f2650074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f264e59c640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2650106f10 0x7f2650074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f264e59c640 1 -- 172.21.15.33:0/3591563384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2650102790 msgr2=0x7f2650074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f264e59c640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2650102790 0x7f2650074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f264e59c640 1 -- 172.21.15.33:0/3591563384 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26501a6930 con 0x7f2650101de0 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f2647fff640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2650102790 0x7f2650074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f264e59c640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2650101de0 0x7f26500744f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f264000e9f0 tx=0x7f264000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f26477fe640 1 -- 172.21.15.33:0/3591563384 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264000cd80 con 0x7f2650101de0 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f26477fe640 1 -- 172.21.15.33:0/3591563384 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2640010910 con 0x7f2650101de0 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f26477fe640 1 -- 172.21.15.33:0/3591563384 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640018990 con 0x7f2650101de0 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f2654a2c640 1 -- 172.21.15.33:0/3591563384 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26501a6c20 con 0x7f2650101de0 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.011+0000 7f2654a2c640 1 -- 172.21.15.33:0/3591563384 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26501a7110 con 0x7f2650101de0 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.012+0000 7f2654a2c640 1 -- 172.21.15.33:0/3591563384 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26501021c0 con 0x7f2650101de0 2024-09-16T00:17:43.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.014+0000 7f26477fe640 1 -- 172.21.15.33:0/3591563384 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f2640010ac0 con 0x7f2650101de0 2024-09-16T00:17:43.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.014+0000 7f26477fe640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f263c0a32d0 0x7f263c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:43.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.014+0000 7f2647fff640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f263c0a32d0 0x7f263c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:43.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.014+0000 7f26477fe640 1 -- 172.21.15.33:0/3591563384 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f26400d0370 con 0x7f2650101de0 2024-09-16T00:17:43.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.015+0000 7f2647fff640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f263c0a32d0 0x7f263c0a5790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2650075f80 tx=0x7f2638005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:43.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.018+0000 7f26477fe640 1 -- 172.21.15.33:0/3591563384 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f264009cab0 con 0x7f2650101de0 2024-09-16T00:17:43.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:42 smithi083 ceph-mon[35242]: pgmap v367: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-09-16T00:17:43.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.191+0000 7f2654a2c640 1 -- 172.21.15.33:0/3591563384 --> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2650062630 con 0x7f263c0a32d0 2024-09-16T00:17:43.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.199+0000 7f26477fe640 1 -- 172.21.15.33:0/3591563384 <== mgr.14150 v2:172.21.15.33:6800/3427860355 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340770 (secure 0 0 0) 0x7f2650062630 con 0x7f263c0a32d0 2024-09-16T00:17:43.204 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-09-16T00:17:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.202+0000 7f2654a2c640 1 -- 172.21.15.33:0/3591563384 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f263c0a32d0 msgr2=0x7f263c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.202+0000 7f2654a2c640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f263c0a32d0 0x7f263c0a5790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2650075f80 tx=0x7f2638005a20 comp rx=0 tx=0).stop 2024-09-16T00:17:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.202+0000 7f2654a2c640 1 -- 172.21.15.33:0/3591563384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2650101de0 msgr2=0x7f26500744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.202+0000 7f2654a2c640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2650101de0 0x7f26500744f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f264000e9f0 tx=0x7f264000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:17:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.202+0000 7f2654a2c640 1 -- 172.21.15.33:0/3591563384 shutdown_connections 2024-09-16T00:17:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.202+0000 7f2654a2c640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f263c0a32d0 0x7f263c0a5790 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.202+0000 7f2654a2c640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2650106f10 0x7f2650074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.202+0000 7f2654a2c640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2650102790 0x7f2650074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.202+0000 7f2654a2c640 1 --2- 172.21.15.33:0/3591563384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2650101de0 0x7f26500744f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.202+0000 7f2654a2c640 1 -- 172.21.15.33:0/3591563384 >> 172.21.15.33:0/3591563384 conn(0x7f26500fd8c0 msgr2=0x7f26500fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.202+0000 7f2654a2c640 1 -- 172.21.15.33:0/3591563384 shutdown_connections 2024-09-16T00:17:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:43.202+0000 7f2654a2c640 1 -- 172.21.15.33:0/3591563384 wait complete. 2024-09-16T00:17:43.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:42 smithi060 ceph-mon[35185]: pgmap v367: 129 pgs: 129 active+clean; 651 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-09-16T00:17:43.905 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-16T00:17:41.673063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666913,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":810,"num_read_kb":521,"num_write":365,"num_write_kb":3075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1242843,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386388,"kb_used_data":12144,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555436,"statfs":{"total":1151940427776,"available":1151544766464,"internally_reserved":0,"allocated":12435456,"data_stored":8182491,"data_compressed":471663,"data_compressed_allocated":638976,"data_compressed_original":2809856,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8679,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":78,"num_read_kb":45,"num_write":6,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003260"},"pg_stats":[{"pgid":"2.1f","version":"86'9","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578728+0000","last_change":"2024-09-16T00:16:09.034677+0000","last_active":"2024-09-16T00:17:09.578728+0000","last_peered":"2024-09-16T00:17:09.578728+0000","last_clean":"2024-09-16T00:17:09.578728+0000","last_became_active":"2024-09-16T00:16:09.024755+0000","last_became_peered":"2024-09-16T00:16:09.024755+0000","last_unstale":"2024-09-16T00:17:09.578728+0000","last_undegraded":"2024-09-16T00:17:09.578728+0000","last_fullsized":"2024-09-16T00:17:09.578728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:09.012210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579433+0000","last_change":"2024-09-16T00:15:06.547094+0000","last_active":"2024-09-16T00:17:09.579433+0000","last_peered":"2024-09-16T00:17:09.579433+0000","last_clean":"2024-09-16T00:17:09.579433+0000","last_became_active":"2024-09-16T00:15:06.542764+0000","last_became_peered":"2024-09-16T00:15:06.542764+0000","last_unstale":"2024-09-16T00:17:09.579433+0000","last_undegraded":"2024-09-16T00:17:09.579433+0000","last_fullsized":"2024-09-16T00:17:09.579433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:04:37.449577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574697+0000","last_change":"2024-09-16T00:15:13.578400+0000","last_active":"2024-09-16T00:17:09.574697+0000","last_peered":"2024-09-16T00:17:09.574697+0000","last_clean":"2024-09-16T00:17:09.574697+0000","last_became_active":"2024-09-16T00:15:13.577787+0000","last_became_peered":"2024-09-16T00:15:13.577787+0000","last_unstale":"2024-09-16T00:17:09.574697+0000","last_undegraded":"2024-09-16T00:17:09.574697+0000","last_fullsized":"2024-09-16T00:17:09.574697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:19:59.194152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579738+0000","last_change":"2024-09-16T00:15:04.373381+0000","last_active":"2024-09-16T00:17:09.579738+0000","last_peered":"2024-09-16T00:17:09.579738+0000","last_clean":"2024-09-16T00:17:09.579738+0000","last_became_active":"2024-09-16T00:15:04.373028+0000","last_became_peered":"2024-09-16T00:15:04.373028+0000","last_unstale":"2024-09-16T00:17:09.579738+0000","last_undegraded":"2024-09-16T00:17:09.579738+0000","last_fullsized":"2024-09-16T00:17:09.579738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:22.902583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577500+0000","last_change":"2024-09-16T00:15:06.554048+0000","last_active":"2024-09-16T00:17:09.577500+0000","last_peered":"2024-09-16T00:17:09.577500+0000","last_clean":"2024-09-16T00:17:09.577500+0000","last_became_active":"2024-09-16T00:15:06.553044+0000","last_became_peered":"2024-09-16T00:15:06.553044+0000","last_unstale":"2024-09-16T00:17:09.577500+0000","last_undegraded":"2024-09-16T00:17:09.577500+0000","last_fullsized":"2024-09-16T00:17:09.577500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:22:53.333705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575027+0000","last_change":"2024-09-16T00:15:13.586995+0000","last_active":"2024-09-16T00:17:09.575027+0000","last_peered":"2024-09-16T00:17:09.575027+0000","last_clean":"2024-09-16T00:17:09.575027+0000","last_became_active":"2024-09-16T00:15:13.586509+0000","last_became_peered":"2024-09-16T00:15:13.586509+0000","last_unstale":"2024-09-16T00:17:09.575027+0000","last_undegraded":"2024-09-16T00:17:09.575027+0000","last_fullsized":"2024-09-16T00:17:09.575027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:39.418581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579645+0000","last_change":"2024-09-16T00:15:04.348235+0000","last_active":"2024-09-16T00:17:09.579645+0000","last_peered":"2024-09-16T00:17:09.579645+0000","last_clean":"2024-09-16T00:17:09.579645+0000","last_became_active":"2024-09-16T00:15:04.347518+0000","last_became_peered":"2024-09-16T00:15:04.347518+0000","last_unstale":"2024-09-16T00:17:09.579645+0000","last_undegraded":"2024-09-16T00:17:09.579645+0000","last_fullsized":"2024-09-16T00:17:09.579645+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:24.396911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581095+0000","last_change":"2024-09-16T00:15:06.554067+0000","last_active":"2024-09-16T00:17:09.581095+0000","last_peered":"2024-09-16T00:17:09.581095+0000","last_clean":"2024-09-16T00:17:09.581095+0000","last_became_active":"2024-09-16T00:15:06.553715+0000","last_became_peered":"2024-09-16T00:15:06.553715+0000","last_unstale":"2024-09-16T00:17:09.581095+0000","last_undegraded":"2024-09-16T00:17:09.581095+0000","last_fullsized":"2024-09-16T00:17:09.581095+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:22.008530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575352+0000","last_change":"2024-09-16T00:15:13.586872+0000","last_active":"2024-09-16T00:17:09.575352+0000","last_peered":"2024-09-16T00:17:09.575352+0000","last_clean":"2024-09-16T00:17:09.575352+0000","last_became_active":"2024-09-16T00:15:13.586375+0000","last_became_peered":"2024-09-16T00:15:13.586375+0000","last_unstale":"2024-09-16T00:17:09.575352+0000","last_undegraded":"2024-09-16T00:17:09.575352+0000","last_fullsized":"2024-09-16T00:17:09.575352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:43.668827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579663+0000","last_change":"2024-09-16T00:15:04.347964+0000","last_active":"2024-09-16T00:17:09.579663+0000","last_peered":"2024-09-16T00:17:09.579663+0000","last_clean":"2024-09-16T00:17:09.579663+0000","last_became_active":"2024-09-16T00:15:04.347588+0000","last_became_peered":"2024-09-16T00:15:04.347588+0000","last_unstale":"2024-09-16T00:17:09.579663+0000","last_undegraded":"2024-09-16T00:17:09.579663+0000","last_fullsized":"2024-09-16T00:17:09.579663+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:14.199287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576971+0000","last_change":"2024-09-16T00:15:06.544918+0000","last_active":"2024-09-16T00:17:09.576971+0000","last_peered":"2024-09-16T00:17:09.576971+0000","last_clean":"2024-09-16T00:17:09.576971+0000","last_became_active":"2024-09-16T00:15:06.544579+0000","last_became_peered":"2024-09-16T00:15:06.544579+0000","last_unstale":"2024-09-16T00:17:09.576971+0000","last_undegraded":"2024-09-16T00:17:09.576971+0000","last_fullsized":"2024-09-16T00:17:09.576971+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:06:37.475644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575052+0000","last_change":"2024-09-16T00:15:13.581488+0000","last_active":"2024-09-16T00:17:09.575052+0000","last_peered":"2024-09-16T00:17:09.575052+0000","last_clean":"2024-09-16T00:17:09.575052+0000","last_became_active":"2024-09-16T00:15:13.581164+0000","last_became_peered":"2024-09-16T00:15:13.581164+0000","last_unstale":"2024-09-16T00:17:09.575052+0000","last_undegraded":"2024-09-16T00:17:09.575052+0000","last_fullsized":"2024-09-16T00:17:09.575052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:05.061592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581271+0000","last_change":"2024-09-16T00:15:06.551982+0000","last_active":"2024-09-16T00:17:09.581271+0000","last_peered":"2024-09-16T00:17:09.581271+0000","last_clean":"2024-09-16T00:17:09.581271+0000","last_became_active":"2024-09-16T00:15:06.551663+0000","last_became_peered":"2024-09-16T00:15:06.551663+0000","last_unstale":"2024-09-16T00:17:09.581271+0000","last_undegraded":"2024-09-16T00:17:09.581271+0000","last_fullsized":"2024-09-16T00:17:09.581271+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:11.928703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579394+0000","last_change":"2024-09-16T00:15:04.352237+0000","last_active":"2024-09-16T00:17:09.579394+0000","last_peered":"2024-09-16T00:17:09.579394+0000","last_clean":"2024-09-16T00:17:09.579394+0000","last_became_active":"2024-09-16T00:15:04.351884+0000","last_became_peered":"2024-09-16T00:15:04.351884+0000","last_unstale":"2024-09-16T00:17:09.579394+0000","last_undegraded":"2024-09-16T00:17:09.579394+0000","last_fullsized":"2024-09-16T00:17:09.579394+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:32.932020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575580+0000","last_change":"2024-09-16T00:15:13.638917+0000","last_active":"2024-09-16T00:17:09.575580+0000","last_peered":"2024-09-16T00:17:09.575580+0000","last_clean":"2024-09-16T00:17:09.575580+0000","last_became_active":"2024-09-16T00:15:13.638401+0000","last_became_peered":"2024-09-16T00:15:13.638401+0000","last_unstale":"2024-09-16T00:17:09.575580+0000","last_undegraded":"2024-09-16T00:17:09.575580+0000","last_fullsized":"2024-09-16T00:17:09.575580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:38.082407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579388+0000","last_change":"2024-09-16T00:15:06.547909+0000","last_active":"2024-09-16T00:17:09.579388+0000","last_peered":"2024-09-16T00:17:09.579388+0000","last_clean":"2024-09-16T00:17:09.579388+0000","last_became_active":"2024-09-16T00:15:06.546814+0000","last_became_peered":"2024-09-16T00:15:06.546814+0000","last_unstale":"2024-09-16T00:17:09.579388+0000","last_undegraded":"2024-09-16T00:17:09.579388+0000","last_fullsized":"2024-09-16T00:17:09.579388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:46.419406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780898+0000","last_change":"2024-09-16T00:16:09.027691+0000","last_active":"2024-09-16T00:17:09.780898+0000","last_peered":"2024-09-16T00:17:09.780898+0000","last_clean":"2024-09-16T00:17:09.780898+0000","last_became_active":"2024-09-16T00:16:09.023204+0000","last_became_peered":"2024-09-16T00:16:09.023204+0000","last_unstale":"2024-09-16T00:17:09.780898+0000","last_undegraded":"2024-09-16T00:17:09.780898+0000","last_fullsized":"2024-09-16T00:17:09.780898+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:44.647576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575627+0000","last_change":"2024-09-16T00:15:13.588345+0000","last_active":"2024-09-16T00:17:09.575627+0000","last_peered":"2024-09-16T00:17:09.575627+0000","last_clean":"2024-09-16T00:17:09.575627+0000","last_became_active":"2024-09-16T00:15:13.587829+0000","last_became_peered":"2024-09-16T00:15:13.587829+0000","last_unstale":"2024-09-16T00:17:09.575627+0000","last_undegraded":"2024-09-16T00:17:09.575627+0000","last_fullsized":"2024-09-16T00:17:09.575627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:04.261031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781137+0000","last_change":"2024-09-16T00:15:06.545485+0000","last_active":"2024-09-16T00:17:09.781137+0000","last_peered":"2024-09-16T00:17:09.781137+0000","last_clean":"2024-09-16T00:17:09.781137+0000","last_became_active":"2024-09-16T00:15:06.545178+0000","last_became_peered":"2024-09-16T00:15:06.545178+0000","last_unstale":"2024-09-16T00:17:09.781137+0000","last_undegraded":"2024-09-16T00:17:09.781137+0000","last_fullsized":"2024-09-16T00:17:09.781137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:26.923469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579782+0000","last_change":"2024-09-16T00:15:04.340546+0000","last_active":"2024-09-16T00:17:09.579782+0000","last_peered":"2024-09-16T00:17:09.579782+0000","last_clean":"2024-09-16T00:17:09.579782+0000","last_became_active":"2024-09-16T00:15:04.340173+0000","last_became_peered":"2024-09-16T00:15:04.340173+0000","last_unstale":"2024-09-16T00:17:09.579782+0000","last_undegraded":"2024-09-16T00:17:09.579782+0000","last_fullsized":"2024-09-16T00:17:09.579782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:35:01.134433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578541+0000","last_change":"2024-09-16T00:15:13.589963+0000","last_active":"2024-09-16T00:17:09.578541+0000","last_peered":"2024-09-16T00:17:09.578541+0000","last_clean":"2024-09-16T00:17:09.578541+0000","last_became_active":"2024-09-16T00:15:13.589615+0000","last_became_peered":"2024-09-16T00:15:13.589615+0000","last_unstale":"2024-09-16T00:17:09.578541+0000","last_undegraded":"2024-09-16T00:17:09.578541+0000","last_fullsized":"2024-09-16T00:17:09.578541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:37.197765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579731+0000","last_change":"2024-09-16T00:15:06.544374+0000","last_active":"2024-09-16T00:17:09.579731+0000","last_peered":"2024-09-16T00:17:09.579731+0000","last_clean":"2024-09-16T00:17:09.579731+0000","last_became_active":"2024-09-16T00:15:06.544027+0000","last_became_peered":"2024-09-16T00:15:06.544027+0000","last_unstale":"2024-09-16T00:17:09.579731+0000","last_undegraded":"2024-09-16T00:17:09.579731+0000","last_fullsized":"2024-09-16T00:17:09.579731+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580105+0000","last_change":"2024-09-16T00:15:04.374209+0000","last_active":"2024-09-16T00:17:09.580105+0000","last_peered":"2024-09-16T00:17:09.580105+0000","last_clean":"2024-09-16T00:17:09.580105+0000","last_became_active":"2024-09-16T00:15:04.373814+0000","last_became_peered":"2024-09-16T00:15:04.373814+0000","last_unstale":"2024-09-16T00:17:09.580105+0000","last_undegraded":"2024-09-16T00:17:09.580105+0000","last_fullsized":"2024-09-16T00:17:09.580105+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:13.516161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575489+0000","last_change":"2024-09-16T00:15:13.639210+0000","last_active":"2024-09-16T00:17:09.575489+0000","last_peered":"2024-09-16T00:17:09.575489+0000","last_clean":"2024-09-16T00:17:09.575489+0000","last_became_active":"2024-09-16T00:15:13.638719+0000","last_became_peered":"2024-09-16T00:15:13.638719+0000","last_unstale":"2024-09-16T00:17:09.575489+0000","last_undegraded":"2024-09-16T00:17:09.575489+0000","last_fullsized":"2024-09-16T00:17:09.575489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:26.130778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577341+0000","last_change":"2024-09-16T00:15:06.553108+0000","last_active":"2024-09-16T00:17:09.577341+0000","last_peered":"2024-09-16T00:17:09.577341+0000","last_clean":"2024-09-16T00:17:09.577341+0000","last_became_active":"2024-09-16T00:15:06.552817+0000","last_became_peered":"2024-09-16T00:15:06.552817+0000","last_unstale":"2024-09-16T00:17:09.577341+0000","last_undegraded":"2024-09-16T00:17:09.577341+0000","last_fullsized":"2024-09-16T00:17:09.577341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:06.821337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575833+0000","last_change":"2024-09-16T00:15:04.644185+0000","last_active":"2024-09-16T00:17:09.575833+0000","last_peered":"2024-09-16T00:17:09.575833+0000","last_clean":"2024-09-16T00:17:09.575833+0000","last_became_active":"2024-09-16T00:15:04.643887+0000","last_became_peered":"2024-09-16T00:15:04.643887+0000","last_unstale":"2024-09-16T00:17:09.575833+0000","last_undegraded":"2024-09-16T00:17:09.575833+0000","last_fullsized":"2024-09-16T00:17:09.575833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:44.279272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574877+0000","last_change":"2024-09-16T00:15:13.574761+0000","last_active":"2024-09-16T00:17:09.574877+0000","last_peered":"2024-09-16T00:17:09.574877+0000","last_clean":"2024-09-16T00:17:09.574877+0000","last_became_active":"2024-09-16T00:15:13.574347+0000","last_became_peered":"2024-09-16T00:15:13.574347+0000","last_unstale":"2024-09-16T00:17:09.574877+0000","last_undegraded":"2024-09-16T00:17:09.574877+0000","last_fullsized":"2024-09-16T00:17:09.574877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:22.040121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579504+0000","last_change":"2024-09-16T00:15:06.550338+0000","last_active":"2024-09-16T00:17:09.579504+0000","last_peered":"2024-09-16T00:17:09.579504+0000","last_clean":"2024-09-16T00:17:09.579504+0000","last_became_active":"2024-09-16T00:15:06.549868+0000","last_became_peered":"2024-09-16T00:15:06.549868+0000","last_unstale":"2024-09-16T00:17:09.579504+0000","last_undegraded":"2024-09-16T00:17:09.579504+0000","last_fullsized":"2024-09-16T00:17:09.579504+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:26.643338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'13","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:28.555275+0000","last_change":"2024-09-16T00:15:04.342306+0000","last_active":"2024-09-16T00:17:28.555275+0000","last_peered":"2024-09-16T00:17:28.555275+0000","last_clean":"2024-09-16T00:17:28.555275+0000","last_became_active":"2024-09-16T00:15:04.341964+0000","last_became_peered":"2024-09-16T00:15:04.341964+0000","last_unstale":"2024-09-16T00:17:28.555275+0000","last_undegraded":"2024-09-16T00:17:28.555275+0000","last_fullsized":"2024-09-16T00:17:28.555275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:14.137199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578972+0000","last_change":"2024-09-16T00:15:13.638246+0000","last_active":"2024-09-16T00:17:09.578972+0000","last_peered":"2024-09-16T00:17:09.578972+0000","last_clean":"2024-09-16T00:17:09.578972+0000","last_became_active":"2024-09-16T00:15:13.637816+0000","last_became_peered":"2024-09-16T00:15:13.637816+0000","last_unstale":"2024-09-16T00:17:09.578972+0000","last_undegraded":"2024-09-16T00:17:09.578972+0000","last_fullsized":"2024-09-16T00:17:09.578972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:27.331156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718239+0000","last_change":"2024-09-16T00:15:06.544057+0000","last_active":"2024-09-16T00:17:09.718239+0000","last_peered":"2024-09-16T00:17:09.718239+0000","last_clean":"2024-09-16T00:17:09.718239+0000","last_became_active":"2024-09-16T00:15:06.543244+0000","last_became_peered":"2024-09-16T00:15:06.543244+0000","last_unstale":"2024-09-16T00:17:09.718239+0000","last_undegraded":"2024-09-16T00:17:09.718239+0000","last_fullsized":"2024-09-16T00:17:09.718239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:42:28.950014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579701+0000","last_change":"2024-09-16T00:15:04.376419+0000","last_active":"2024-09-16T00:17:09.579701+0000","last_peered":"2024-09-16T00:17:09.579701+0000","last_clean":"2024-09-16T00:17:09.579701+0000","last_became_active":"2024-09-16T00:15:04.375976+0000","last_became_peered":"2024-09-16T00:15:04.375976+0000","last_unstale":"2024-09-16T00:17:09.579701+0000","last_undegraded":"2024-09-16T00:17:09.579701+0000","last_fullsized":"2024-09-16T00:17:09.579701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:33:38.652109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781038+0000","last_change":"2024-09-16T00:15:13.584306+0000","last_active":"2024-09-16T00:17:09.781038+0000","last_peered":"2024-09-16T00:17:09.781038+0000","last_clean":"2024-09-16T00:17:09.781038+0000","last_became_active":"2024-09-16T00:15:13.583840+0000","last_became_peered":"2024-09-16T00:15:13.583840+0000","last_unstale":"2024-09-16T00:17:09.781038+0000","last_undegraded":"2024-09-16T00:17:09.781038+0000","last_fullsized":"2024-09-16T00:17:09.781038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:36.660458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577806+0000","last_change":"2024-09-16T00:15:06.538902+0000","last_active":"2024-09-16T00:17:09.577806+0000","last_peered":"2024-09-16T00:17:09.577806+0000","last_clean":"2024-09-16T00:17:09.577806+0000","last_became_active":"2024-09-16T00:15:06.538537+0000","last_became_peered":"2024-09-16T00:15:06.538537+0000","last_unstale":"2024-09-16T00:17:09.577806+0000","last_undegraded":"2024-09-16T00:17:09.577806+0000","last_fullsized":"2024-09-16T00:17:09.577806+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:38.411711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578013+0000","last_change":"2024-09-16T00:15:04.374762+0000","last_active":"2024-09-16T00:17:09.578013+0000","last_peered":"2024-09-16T00:17:09.578013+0000","last_clean":"2024-09-16T00:17:09.578013+0000","last_became_active":"2024-09-16T00:15:04.374394+0000","last_became_peered":"2024-09-16T00:15:04.374394+0000","last_unstale":"2024-09-16T00:17:09.578013+0000","last_undegraded":"2024-09-16T00:17:09.578013+0000","last_fullsized":"2024-09-16T00:17:09.578013+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:47.075021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579233+0000","last_change":"2024-09-16T00:15:13.638306+0000","last_active":"2024-09-16T00:17:09.579233+0000","last_peered":"2024-09-16T00:17:09.579233+0000","last_clean":"2024-09-16T00:17:09.579233+0000","last_became_active":"2024-09-16T00:15:13.637931+0000","last_became_peered":"2024-09-16T00:15:13.637931+0000","last_unstale":"2024-09-16T00:17:09.579233+0000","last_undegraded":"2024-09-16T00:17:09.579233+0000","last_fullsized":"2024-09-16T00:17:09.579233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:41.385174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577585+0000","last_change":"2024-09-16T00:15:06.548209+0000","last_active":"2024-09-16T00:17:09.577585+0000","last_peered":"2024-09-16T00:17:09.577585+0000","last_clean":"2024-09-16T00:17:09.577585+0000","last_became_active":"2024-09-16T00:15:06.547846+0000","last_became_peered":"2024-09-16T00:15:06.547846+0000","last_unstale":"2024-09-16T00:17:09.577585+0000","last_undegraded":"2024-09-16T00:17:09.577585+0000","last_fullsized":"2024-09-16T00:17:09.577585+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:15.821847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780889+0000","last_change":"2024-09-16T00:16:09.023918+0000","last_active":"2024-09-16T00:17:09.780889+0000","last_peered":"2024-09-16T00:17:09.780889+0000","last_clean":"2024-09-16T00:17:09.780889+0000","last_became_active":"2024-09-16T00:16:09.023230+0000","last_became_peered":"2024-09-16T00:16:09.023230+0000","last_unstale":"2024-09-16T00:17:09.780889+0000","last_undegraded":"2024-09-16T00:17:09.780889+0000","last_fullsized":"2024-09-16T00:17:09.780889+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:31.527863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717949+0000","last_change":"2024-09-16T00:15:13.579387+0000","last_active":"2024-09-16T00:17:09.717949+0000","last_peered":"2024-09-16T00:17:09.717949+0000","last_clean":"2024-09-16T00:17:09.717949+0000","last_became_active":"2024-09-16T00:15:13.578996+0000","last_became_peered":"2024-09-16T00:15:13.578996+0000","last_unstale":"2024-09-16T00:17:09.717949+0000","last_undegraded":"2024-09-16T00:17:09.717949+0000","last_fullsized":"2024-09-16T00:17:09.717949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:51.703060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579796+0000","last_change":"2024-09-16T00:17:09.579796+0000","last_active":"2024-09-16T00:17:09.579796+0000","last_peered":"2024-09-16T00:17:09.579796+0000","last_clean":"2024-09-16T00:17:09.579796+0000","last_became_active":"2024-09-16T00:17:09.579386+0000","last_became_peered":"2024-09-16T00:17:09.579386+0000","last_unstale":"2024-09-16T00:17:09.579796+0000","last_undegraded":"2024-09-16T00:17:09.579796+0000","last_fullsized":"2024-09-16T00:17:09.579796+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:39:42.571193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":155,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:38.327474+0000","last_change":"2024-09-16T00:15:04.373896+0000","last_active":"2024-09-16T00:17:38.327474+0000","last_peered":"2024-09-16T00:17:38.327474+0000","last_clean":"2024-09-16T00:17:38.327474+0000","last_became_active":"2024-09-16T00:15:04.373493+0000","last_became_peered":"2024-09-16T00:15:04.373493+0000","last_unstale":"2024-09-16T00:17:38.327474+0000","last_undegraded":"2024-09-16T00:17:38.327474+0000","last_fullsized":"2024-09-16T00:17:38.327474+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:17:10.767267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":110,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718128+0000","last_change":"2024-09-16T00:15:13.638575+0000","last_active":"2024-09-16T00:17:09.718128+0000","last_peered":"2024-09-16T00:17:09.718128+0000","last_clean":"2024-09-16T00:17:09.718128+0000","last_became_active":"2024-09-16T00:15:13.638144+0000","last_became_peered":"2024-09-16T00:15:13.638144+0000","last_unstale":"2024-09-16T00:17:09.718128+0000","last_undegraded":"2024-09-16T00:17:09.718128+0000","last_fullsized":"2024-09-16T00:17:09.718128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:39.345011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718312+0000","last_change":"2024-09-16T00:15:06.548852+0000","last_active":"2024-09-16T00:17:09.718312+0000","last_peered":"2024-09-16T00:17:09.718312+0000","last_clean":"2024-09-16T00:17:09.718312+0000","last_became_active":"2024-09-16T00:15:06.548238+0000","last_became_peered":"2024-09-16T00:15:06.548238+0000","last_unstale":"2024-09-16T00:17:09.718312+0000","last_undegraded":"2024-09-16T00:17:09.718312+0000","last_fullsized":"2024-09-16T00:17:09.718312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:16.115338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580690+0000","last_change":"2024-09-16T00:15:04.350753+0000","last_active":"2024-09-16T00:17:09.580690+0000","last_peered":"2024-09-16T00:17:09.580690+0000","last_clean":"2024-09-16T00:17:09.580690+0000","last_became_active":"2024-09-16T00:15:04.350406+0000","last_became_peered":"2024-09-16T00:15:04.350406+0000","last_unstale":"2024-09-16T00:17:09.580690+0000","last_undegraded":"2024-09-16T00:17:09.580690+0000","last_fullsized":"2024-09-16T00:17:09.580690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:09:54.737287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576996+0000","last_change":"2024-09-16T00:15:13.639302+0000","last_active":"2024-09-16T00:17:09.576996+0000","last_peered":"2024-09-16T00:17:09.576996+0000","last_clean":"2024-09-16T00:17:09.576996+0000","last_became_active":"2024-09-16T00:15:13.638863+0000","last_became_peered":"2024-09-16T00:15:13.638863+0000","last_unstale":"2024-09-16T00:17:09.576996+0000","last_undegraded":"2024-09-16T00:17:09.576996+0000","last_fullsized":"2024-09-16T00:17:09.576996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:23.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582176+0000","last_change":"2024-09-16T00:17:09.582176+0000","last_active":"2024-09-16T00:17:09.582176+0000","last_peered":"2024-09-16T00:17:09.582176+0000","last_clean":"2024-09-16T00:17:09.582176+0000","last_became_active":"2024-09-16T00:17:09.581740+0000","last_became_peered":"2024-09-16T00:17:09.581740+0000","last_unstale":"2024-09-16T00:17:09.582176+0000","last_undegraded":"2024-09-16T00:17:09.582176+0000","last_fullsized":"2024-09-16T00:17:09.582176+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:57.176724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575861+0000","last_change":"2024-09-16T00:15:04.342216+0000","last_active":"2024-09-16T00:17:09.575861+0000","last_peered":"2024-09-16T00:17:09.575861+0000","last_clean":"2024-09-16T00:17:09.575861+0000","last_became_active":"2024-09-16T00:15:04.341763+0000","last_became_peered":"2024-09-16T00:15:04.341763+0000","last_unstale":"2024-09-16T00:17:09.575861+0000","last_undegraded":"2024-09-16T00:17:09.575861+0000","last_fullsized":"2024-09-16T00:17:09.575861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:44.938527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":350,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580892+0000","last_change":"2024-09-16T00:13:51.108571+0000","last_active":"2024-09-16T00:17:09.580892+0000","last_peered":"2024-09-16T00:17:09.580892+0000","last_clean":"2024-09-16T00:17:09.580892+0000","last_became_active":"2024-09-16T00:13:51.108088+0000","last_became_peered":"2024-09-16T00:13:51.108088+0000","last_unstale":"2024-09-16T00:17:09.580892+0000","last_undegraded":"2024-09-16T00:17:09.580892+0000","last_fullsized":"2024-09-16T00:17:09.580892+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:10:24.533807+0000","last_clean_scrub_stamp":"2024-09-16T00:10:24.533807+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:02.511064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581143+0000","last_change":"2024-09-16T00:15:13.588665+0000","last_active":"2024-09-16T00:17:09.581143+0000","last_peered":"2024-09-16T00:17:09.581143+0000","last_clean":"2024-09-16T00:17:09.581143+0000","last_became_active":"2024-09-16T00:15:13.588222+0000","last_became_peered":"2024-09-16T00:15:13.588222+0000","last_unstale":"2024-09-16T00:17:09.581143+0000","last_undegraded":"2024-09-16T00:17:09.581143+0000","last_fullsized":"2024-09-16T00:17:09.581143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:59:59.965962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581454+0000","last_change":"2024-09-16T00:15:06.553114+0000","last_active":"2024-09-16T00:17:09.581454+0000","last_peered":"2024-09-16T00:17:09.581454+0000","last_clean":"2024-09-16T00:17:09.581454+0000","last_became_active":"2024-09-16T00:15:06.552788+0000","last_became_peered":"2024-09-16T00:15:06.552788+0000","last_unstale":"2024-09-16T00:17:09.581454+0000","last_undegraded":"2024-09-16T00:17:09.581454+0000","last_fullsized":"2024-09-16T00:17:09.581454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:05.473847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575364+0000","last_change":"2024-09-16T00:16:09.030747+0000","last_active":"2024-09-16T00:17:09.575364+0000","last_peered":"2024-09-16T00:17:09.575364+0000","last_clean":"2024-09-16T00:17:09.575364+0000","last_became_active":"2024-09-16T00:16:09.021208+0000","last_became_peered":"2024-09-16T00:16:09.021208+0000","last_unstale":"2024-09-16T00:17:09.575364+0000","last_undegraded":"2024-09-16T00:17:09.575364+0000","last_fullsized":"2024-09-16T00:17:09.575364+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:40.605498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580422+0000","last_change":"2024-09-16T00:15:13.575606+0000","last_active":"2024-09-16T00:17:09.580422+0000","last_peered":"2024-09-16T00:17:09.580422+0000","last_clean":"2024-09-16T00:17:09.580422+0000","last_became_active":"2024-09-16T00:15:13.575131+0000","last_became_peered":"2024-09-16T00:15:13.575131+0000","last_unstale":"2024-09-16T00:17:09.580422+0000","last_undegraded":"2024-09-16T00:17:09.580422+0000","last_fullsized":"2024-09-16T00:17:09.580422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:41.636058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.584161+0000","last_change":"2024-09-16T00:17:09.583522+0000","last_active":"2024-09-16T00:17:09.584161+0000","last_peered":"2024-09-16T00:17:09.584161+0000","last_clean":"2024-09-16T00:17:09.584161+0000","last_became_active":"2024-09-16T00:17:09.583082+0000","last_became_peered":"2024-09-16T00:17:09.583082+0000","last_unstale":"2024-09-16T00:17:09.584161+0000","last_undegraded":"2024-09-16T00:17:09.584161+0000","last_fullsized":"2024-09-16T00:17:09.584161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:48.582126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577245+0000","last_change":"2024-09-16T00:15:04.349095+0000","last_active":"2024-09-16T00:17:09.577245+0000","last_peered":"2024-09-16T00:17:09.577245+0000","last_clean":"2024-09-16T00:17:09.577245+0000","last_became_active":"2024-09-16T00:15:04.348698+0000","last_became_peered":"2024-09-16T00:15:04.348698+0000","last_unstale":"2024-09-16T00:17:09.577245+0000","last_undegraded":"2024-09-16T00:17:09.577245+0000","last_fullsized":"2024-09-16T00:17:09.577245+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:42.282882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577535+0000","last_change":"2024-09-16T00:15:13.580494+0000","last_active":"2024-09-16T00:17:09.577535+0000","last_peered":"2024-09-16T00:17:09.577535+0000","last_clean":"2024-09-16T00:17:09.577535+0000","last_became_active":"2024-09-16T00:15:13.580091+0000","last_became_peered":"2024-09-16T00:15:13.580091+0000","last_unstale":"2024-09-16T00:17:09.577535+0000","last_undegraded":"2024-09-16T00:17:09.577535+0000","last_fullsized":"2024-09-16T00:17:09.577535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:17.314919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580727+0000","last_change":"2024-09-16T00:15:06.544859+0000","last_active":"2024-09-16T00:17:09.580727+0000","last_peered":"2024-09-16T00:17:09.580727+0000","last_clean":"2024-09-16T00:17:09.580727+0000","last_became_active":"2024-09-16T00:15:06.544289+0000","last_became_peered":"2024-09-16T00:15:06.544289+0000","last_unstale":"2024-09-16T00:17:09.580727+0000","last_undegraded":"2024-09-16T00:17:09.580727+0000","last_fullsized":"2024-09-16T00:17:09.580727+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:52.622477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718205+0000","last_change":"2024-09-16T00:15:04.351313+0000","last_active":"2024-09-16T00:17:09.718205+0000","last_peered":"2024-09-16T00:17:09.718205+0000","last_clean":"2024-09-16T00:17:09.718205+0000","last_became_active":"2024-09-16T00:15:04.350928+0000","last_became_peered":"2024-09-16T00:15:04.350928+0000","last_unstale":"2024-09-16T00:17:09.718205+0000","last_undegraded":"2024-09-16T00:17:09.718205+0000","last_fullsized":"2024-09-16T00:17:09.718205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:48.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576571+0000","last_change":"2024-09-16T00:15:13.578573+0000","last_active":"2024-09-16T00:17:09.576571+0000","last_peered":"2024-09-16T00:17:09.576571+0000","last_clean":"2024-09-16T00:17:09.576571+0000","last_became_active":"2024-09-16T00:15:13.578117+0000","last_became_peered":"2024-09-16T00:15:13.578117+0000","last_unstale":"2024-09-16T00:17:09.576571+0000","last_undegraded":"2024-09-16T00:17:09.576571+0000","last_fullsized":"2024-09-16T00:17:09.576571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:53.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781258+0000","last_change":"2024-09-16T00:15:06.553548+0000","last_active":"2024-09-16T00:17:09.781258+0000","last_peered":"2024-09-16T00:17:09.781258+0000","last_clean":"2024-09-16T00:17:09.781258+0000","last_became_active":"2024-09-16T00:15:06.552963+0000","last_became_peered":"2024-09-16T00:15:06.552963+0000","last_unstale":"2024-09-16T00:17:09.781258+0000","last_undegraded":"2024-09-16T00:17:09.781258+0000","last_fullsized":"2024-09-16T00:17:09.781258+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:43.455881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579776+0000","last_change":"2024-09-16T00:15:04.351005+0000","last_active":"2024-09-16T00:17:09.579776+0000","last_peered":"2024-09-16T00:17:09.579776+0000","last_clean":"2024-09-16T00:17:09.579776+0000","last_became_active":"2024-09-16T00:15:04.350614+0000","last_became_peered":"2024-09-16T00:15:04.350614+0000","last_unstale":"2024-09-16T00:17:09.579776+0000","last_undegraded":"2024-09-16T00:17:09.579776+0000","last_fullsized":"2024-09-16T00:17:09.579776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:39.196360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579501+0000","last_change":"2024-09-16T00:16:09.023796+0000","last_active":"2024-09-16T00:17:09.579501+0000","last_peered":"2024-09-16T00:17:09.579501+0000","last_clean":"2024-09-16T00:17:09.579501+0000","last_became_active":"2024-09-16T00:16:09.023403+0000","last_became_peered":"2024-09-16T00:16:09.023403+0000","last_unstale":"2024-09-16T00:17:09.579501+0000","last_undegraded":"2024-09-16T00:17:09.579501+0000","last_fullsized":"2024-09-16T00:17:09.579501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:59.488315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577059+0000","last_change":"2024-09-16T00:15:06.546325+0000","last_active":"2024-09-16T00:17:09.577059+0000","last_peered":"2024-09-16T00:17:09.577059+0000","last_clean":"2024-09-16T00:17:09.577059+0000","last_became_active":"2024-09-16T00:15:06.546028+0000","last_became_peered":"2024-09-16T00:15:06.546028+0000","last_unstale":"2024-09-16T00:17:09.577059+0000","last_undegraded":"2024-09-16T00:17:09.577059+0000","last_fullsized":"2024-09-16T00:17:09.577059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:15.049592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579065+0000","last_change":"2024-09-16T00:15:04.373818+0000","last_active":"2024-09-16T00:17:09.579065+0000","last_peered":"2024-09-16T00:17:09.579065+0000","last_clean":"2024-09-16T00:17:09.579065+0000","last_became_active":"2024-09-16T00:15:04.373406+0000","last_became_peered":"2024-09-16T00:15:04.373406+0000","last_unstale":"2024-09-16T00:17:09.579065+0000","last_undegraded":"2024-09-16T00:17:09.579065+0000","last_fullsized":"2024-09-16T00:17:09.579065+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:08.402660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718115+0000","last_change":"2024-09-16T00:15:13.588669+0000","last_active":"2024-09-16T00:17:09.718115+0000","last_peered":"2024-09-16T00:17:09.718115+0000","last_clean":"2024-09-16T00:17:09.718115+0000","last_became_active":"2024-09-16T00:15:13.588229+0000","last_became_peered":"2024-09-16T00:15:13.588229+0000","last_unstale":"2024-09-16T00:17:09.718115+0000","last_undegraded":"2024-09-16T00:17:09.718115+0000","last_fullsized":"2024-09-16T00:17:09.718115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:30.157296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580525+0000","last_change":"2024-09-16T00:15:06.550221+0000","last_active":"2024-09-16T00:17:09.580525+0000","last_peered":"2024-09-16T00:17:09.580525+0000","last_clean":"2024-09-16T00:17:09.580525+0000","last_became_active":"2024-09-16T00:15:06.549914+0000","last_became_peered":"2024-09-16T00:15:06.549914+0000","last_unstale":"2024-09-16T00:17:09.580525+0000","last_undegraded":"2024-09-16T00:17:09.580525+0000","last_fullsized":"2024-09-16T00:17:09.580525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:18:26.824523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":159,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:38.327928+0000","last_change":"2024-09-16T00:15:04.344389+0000","last_active":"2024-09-16T00:17:38.327928+0000","last_peered":"2024-09-16T00:17:38.327928+0000","last_clean":"2024-09-16T00:17:38.327928+0000","last_became_active":"2024-09-16T00:15:04.343956+0000","last_became_peered":"2024-09-16T00:15:04.343956+0000","last_unstale":"2024-09-16T00:17:38.327928+0000","last_undegraded":"2024-09-16T00:17:38.327928+0000","last_fullsized":"2024-09-16T00:17:38.327928+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:36:34.019861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579707+0000","last_change":"2024-09-16T00:16:09.024050+0000","last_active":"2024-09-16T00:17:09.579707+0000","last_peered":"2024-09-16T00:17:09.579707+0000","last_clean":"2024-09-16T00:17:09.579707+0000","last_became_active":"2024-09-16T00:16:09.023371+0000","last_became_peered":"2024-09-16T00:16:09.023371+0000","last_unstale":"2024-09-16T00:17:09.579707+0000","last_undegraded":"2024-09-16T00:17:09.579707+0000","last_fullsized":"2024-09-16T00:17:09.579707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:35:19.020370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581433+0000","last_change":"2024-09-16T00:15:06.542744+0000","last_active":"2024-09-16T00:17:09.581433+0000","last_peered":"2024-09-16T00:17:09.581433+0000","last_clean":"2024-09-16T00:17:09.581433+0000","last_became_active":"2024-09-16T00:15:06.541868+0000","last_became_peered":"2024-09-16T00:15:06.541868+0000","last_unstale":"2024-09-16T00:17:09.581433+0000","last_undegraded":"2024-09-16T00:17:09.581433+0000","last_fullsized":"2024-09-16T00:17:09.581433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:10:06.508682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577336+0000","last_change":"2024-09-16T00:15:04.349556+0000","last_active":"2024-09-16T00:17:09.577336+0000","last_peered":"2024-09-16T00:17:09.577336+0000","last_clean":"2024-09-16T00:17:09.577336+0000","last_became_active":"2024-09-16T00:15:04.349242+0000","last_became_peered":"2024-09-16T00:15:04.349242+0000","last_unstale":"2024-09-16T00:17:09.577336+0000","last_undegraded":"2024-09-16T00:17:09.577336+0000","last_fullsized":"2024-09-16T00:17:09.577336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:22:18.459157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781143+0000","last_change":"2024-09-16T00:15:13.584109+0000","last_active":"2024-09-16T00:17:09.781143+0000","last_peered":"2024-09-16T00:17:09.781143+0000","last_clean":"2024-09-16T00:17:09.781143+0000","last_became_active":"2024-09-16T00:15:13.583718+0000","last_became_peered":"2024-09-16T00:15:13.583718+0000","last_unstale":"2024-09-16T00:17:09.781143+0000","last_undegraded":"2024-09-16T00:17:09.781143+0000","last_fullsized":"2024-09-16T00:17:09.781143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:29.416254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717952+0000","last_change":"2024-09-16T00:15:06.548247+0000","last_active":"2024-09-16T00:17:09.717952+0000","last_peered":"2024-09-16T00:17:09.717952+0000","last_clean":"2024-09-16T00:17:09.717952+0000","last_became_active":"2024-09-16T00:15:06.547864+0000","last_became_peered":"2024-09-16T00:15:06.547864+0000","last_unstale":"2024-09-16T00:17:09.717952+0000","last_undegraded":"2024-09-16T00:17:09.717952+0000","last_fullsized":"2024-09-16T00:17:09.717952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:25.350197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575352+0000","last_change":"2024-09-16T00:15:04.350965+0000","last_active":"2024-09-16T00:17:09.575352+0000","last_peered":"2024-09-16T00:17:09.575352+0000","last_clean":"2024-09-16T00:17:09.575352+0000","last_became_active":"2024-09-16T00:15:04.349124+0000","last_became_peered":"2024-09-16T00:15:04.349124+0000","last_unstale":"2024-09-16T00:17:09.575352+0000","last_undegraded":"2024-09-16T00:17:09.575352+0000","last_fullsized":"2024-09-16T00:17:09.575352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:40.859220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576966+0000","last_change":"2024-09-16T00:15:13.581718+0000","last_active":"2024-09-16T00:17:09.576966+0000","last_peered":"2024-09-16T00:17:09.576966+0000","last_clean":"2024-09-16T00:17:09.576966+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:17:09.576966+0000","last_undegraded":"2024-09-16T00:17:09.576966+0000","last_fullsized":"2024-09-16T00:17:09.576966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:08.810606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579555+0000","last_change":"2024-09-16T00:15:06.544507+0000","last_active":"2024-09-16T00:17:09.579555+0000","last_peered":"2024-09-16T00:17:09.579555+0000","last_clean":"2024-09-16T00:17:09.579555+0000","last_became_active":"2024-09-16T00:15:06.544178+0000","last_became_peered":"2024-09-16T00:15:06.544178+0000","last_unstale":"2024-09-16T00:17:09.579555+0000","last_undegraded":"2024-09-16T00:17:09.579555+0000","last_fullsized":"2024-09-16T00:17:09.579555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:38.494266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'39","reported_seq":325,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:38.172320+0000","last_change":"2024-09-16T00:16:09.039763+0000","last_active":"2024-09-16T00:17:38.172320+0000","last_peered":"2024-09-16T00:17:38.172320+0000","last_clean":"2024-09-16T00:17:38.172320+0000","last_became_active":"2024-09-16T00:16:09.022939+0000","last_became_peered":"2024-09-16T00:16:09.022939+0000","last_unstale":"2024-09-16T00:17:38.172320+0000","last_undegraded":"2024-09-16T00:17:38.172320+0000","last_fullsized":"2024-09-16T00:17:38.172320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:56.431159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71774,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":226,"num_read_kb":174,"num_write":40,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60943,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576887+0000","last_change":"2024-09-16T00:15:13.585928+0000","last_active":"2024-09-16T00:17:09.576887+0000","last_peered":"2024-09-16T00:17:09.576887+0000","last_clean":"2024-09-16T00:17:09.576887+0000","last_became_active":"2024-09-16T00:15:13.582940+0000","last_became_peered":"2024-09-16T00:15:13.582940+0000","last_unstale":"2024-09-16T00:17:09.576887+0000","last_undegraded":"2024-09-16T00:17:09.576887+0000","last_fullsized":"2024-09-16T00:17:09.576887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:57:13.251978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580875+0000","last_change":"2024-09-16T00:15:06.546238+0000","last_active":"2024-09-16T00:17:09.580875+0000","last_peered":"2024-09-16T00:17:09.580875+0000","last_clean":"2024-09-16T00:17:09.580875+0000","last_became_active":"2024-09-16T00:15:06.545919+0000","last_became_peered":"2024-09-16T00:15:06.545919+0000","last_unstale":"2024-09-16T00:17:09.580875+0000","last_undegraded":"2024-09-16T00:17:09.580875+0000","last_fullsized":"2024-09-16T00:17:09.580875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:09.180832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577083+0000","last_change":"2024-09-16T00:15:04.376028+0000","last_active":"2024-09-16T00:17:09.577083+0000","last_peered":"2024-09-16T00:17:09.577083+0000","last_clean":"2024-09-16T00:17:09.577083+0000","last_became_active":"2024-09-16T00:15:04.375633+0000","last_became_peered":"2024-09-16T00:15:04.375633+0000","last_unstale":"2024-09-16T00:17:09.577083+0000","last_undegraded":"2024-09-16T00:17:09.577083+0000","last_fullsized":"2024-09-16T00:17:09.577083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:44.446342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575275+0000","last_change":"2024-09-16T00:16:09.025524+0000","last_active":"2024-09-16T00:17:09.575275+0000","last_peered":"2024-09-16T00:17:09.575275+0000","last_clean":"2024-09-16T00:17:09.575275+0000","last_became_active":"2024-09-16T00:16:09.025035+0000","last_became_peered":"2024-09-16T00:16:09.025035+0000","last_unstale":"2024-09-16T00:17:09.575275+0000","last_undegraded":"2024-09-16T00:17:09.575275+0000","last_fullsized":"2024-09-16T00:17:09.575275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:12.367244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577617+0000","last_change":"2024-09-16T00:15:06.550864+0000","last_active":"2024-09-16T00:17:09.577617+0000","last_peered":"2024-09-16T00:17:09.577617+0000","last_clean":"2024-09-16T00:17:09.577617+0000","last_became_active":"2024-09-16T00:15:06.547956+0000","last_became_peered":"2024-09-16T00:15:06.547956+0000","last_unstale":"2024-09-16T00:17:09.577617+0000","last_undegraded":"2024-09-16T00:17:09.577617+0000","last_fullsized":"2024-09-16T00:17:09.577617+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:26.102175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580692+0000","last_change":"2024-09-16T00:15:04.350349+0000","last_active":"2024-09-16T00:17:09.580692+0000","last_peered":"2024-09-16T00:17:09.580692+0000","last_clean":"2024-09-16T00:17:09.580692+0000","last_became_active":"2024-09-16T00:15:04.349951+0000","last_became_peered":"2024-09-16T00:15:04.349951+0000","last_unstale":"2024-09-16T00:17:09.580692+0000","last_undegraded":"2024-09-16T00:17:09.580692+0000","last_fullsized":"2024-09-16T00:17:09.580692+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:46.000831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577297+0000","last_change":"2024-09-16T00:16:09.025774+0000","last_active":"2024-09-16T00:17:09.577297+0000","last_peered":"2024-09-16T00:17:09.577297+0000","last_clean":"2024-09-16T00:17:09.577297+0000","last_became_active":"2024-09-16T00:16:09.025308+0000","last_became_peered":"2024-09-16T00:16:09.025308+0000","last_unstale":"2024-09-16T00:17:09.577297+0000","last_undegraded":"2024-09-16T00:17:09.577297+0000","last_fullsized":"2024-09-16T00:17:09.577297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:16.664850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580788+0000","last_change":"2024-09-16T00:15:06.552919+0000","last_active":"2024-09-16T00:17:09.580788+0000","last_peered":"2024-09-16T00:17:09.580788+0000","last_clean":"2024-09-16T00:17:09.580788+0000","last_became_active":"2024-09-16T00:15:06.552616+0000","last_became_peered":"2024-09-16T00:15:06.552616+0000","last_unstale":"2024-09-16T00:17:09.580788+0000","last_undegraded":"2024-09-16T00:17:09.580788+0000","last_fullsized":"2024-09-16T00:17:09.580788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:47:38.651837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":150,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:36.325025+0000","last_change":"2024-09-16T00:15:04.351116+0000","last_active":"2024-09-16T00:17:36.325025+0000","last_peered":"2024-09-16T00:17:36.325025+0000","last_clean":"2024-09-16T00:17:36.325025+0000","last_became_active":"2024-09-16T00:15:04.350728+0000","last_became_peered":"2024-09-16T00:15:04.350728+0000","last_unstale":"2024-09-16T00:17:36.325025+0000","last_undegraded":"2024-09-16T00:17:36.325025+0000","last_fullsized":"2024-09-16T00:17:36.325025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:38.314019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579322+0000","last_change":"2024-09-16T00:15:13.575961+0000","last_active":"2024-09-16T00:17:09.579322+0000","last_peered":"2024-09-16T00:17:09.579322+0000","last_clean":"2024-09-16T00:17:09.579322+0000","last_became_active":"2024-09-16T00:15:13.575550+0000","last_became_peered":"2024-09-16T00:15:13.575550+0000","last_unstale":"2024-09-16T00:17:09.579322+0000","last_undegraded":"2024-09-16T00:17:09.579322+0000","last_fullsized":"2024-09-16T00:17:09.579322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:05.515588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577699+0000","last_change":"2024-09-16T00:15:13.583583+0000","last_active":"2024-09-16T00:17:09.577699+0000","last_peered":"2024-09-16T00:17:09.577699+0000","last_clean":"2024-09-16T00:17:09.577699+0000","last_became_active":"2024-09-16T00:15:13.583145+0000","last_became_peered":"2024-09-16T00:15:13.583145+0000","last_unstale":"2024-09-16T00:17:09.577699+0000","last_undegraded":"2024-09-16T00:17:09.577699+0000","last_fullsized":"2024-09-16T00:17:09.577699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:07.320337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.719776+0000","last_change":"2024-09-16T00:17:09.719269+0000","last_active":"2024-09-16T00:17:09.719776+0000","last_peered":"2024-09-16T00:17:09.719776+0000","last_clean":"2024-09-16T00:17:09.719776+0000","last_became_active":"2024-09-16T00:17:09.718887+0000","last_became_peered":"2024-09-16T00:17:09.718887+0000","last_unstale":"2024-09-16T00:17:09.719776+0000","last_undegraded":"2024-09-16T00:17:09.719776+0000","last_fullsized":"2024-09-16T00:17:09.719776+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:14:51.675738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577333+0000","last_change":"2024-09-16T00:15:04.350826+0000","last_active":"2024-09-16T00:17:09.577333+0000","last_peered":"2024-09-16T00:17:09.577333+0000","last_clean":"2024-09-16T00:17:09.577333+0000","last_became_active":"2024-09-16T00:15:04.350294+0000","last_became_peered":"2024-09-16T00:15:04.350294+0000","last_unstale":"2024-09-16T00:17:09.577333+0000","last_undegraded":"2024-09-16T00:17:09.577333+0000","last_fullsized":"2024-09-16T00:17:09.577333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:49.146709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781194+0000","last_change":"2024-09-16T00:15:13.580184+0000","last_active":"2024-09-16T00:17:09.781194+0000","last_peered":"2024-09-16T00:17:09.781194+0000","last_clean":"2024-09-16T00:17:09.781194+0000","last_became_active":"2024-09-16T00:15:13.579849+0000","last_became_peered":"2024-09-16T00:15:13.579849+0000","last_unstale":"2024-09-16T00:17:09.781194+0000","last_undegraded":"2024-09-16T00:17:09.781194+0000","last_fullsized":"2024-09-16T00:17:09.781194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:40.613127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579257+0000","last_change":"2024-09-16T00:15:06.545796+0000","last_active":"2024-09-16T00:17:09.579257+0000","last_peered":"2024-09-16T00:17:09.579257+0000","last_clean":"2024-09-16T00:17:09.579257+0000","last_became_active":"2024-09-16T00:15:06.544815+0000","last_became_peered":"2024-09-16T00:15:06.544815+0000","last_unstale":"2024-09-16T00:17:09.579257+0000","last_undegraded":"2024-09-16T00:17:09.579257+0000","last_fullsized":"2024-09-16T00:17:09.579257+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:51:25.921949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718347+0000","last_change":"2024-09-16T00:15:04.340101+0000","last_active":"2024-09-16T00:17:09.718347+0000","last_peered":"2024-09-16T00:17:09.718347+0000","last_clean":"2024-09-16T00:17:09.718347+0000","last_became_active":"2024-09-16T00:15:04.339788+0000","last_became_peered":"2024-09-16T00:15:04.339788+0000","last_unstale":"2024-09-16T00:17:09.718347+0000","last_undegraded":"2024-09-16T00:17:09.718347+0000","last_fullsized":"2024-09-16T00:17:09.718347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:08.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578011+0000","last_change":"2024-09-16T00:16:09.028910+0000","last_active":"2024-09-16T00:17:09.578011+0000","last_peered":"2024-09-16T00:17:09.578011+0000","last_clean":"2024-09-16T00:17:09.578011+0000","last_became_active":"2024-09-16T00:16:09.023302+0000","last_became_peered":"2024-09-16T00:16:09.023302+0000","last_unstale":"2024-09-16T00:17:09.578011+0000","last_undegraded":"2024-09-16T00:17:09.578011+0000","last_fullsized":"2024-09-16T00:17:09.578011+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:35.293676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581774+0000","last_change":"2024-09-16T00:17:09.581116+0000","last_active":"2024-09-16T00:17:09.581774+0000","last_peered":"2024-09-16T00:17:09.581774+0000","last_clean":"2024-09-16T00:17:09.581774+0000","last_became_active":"2024-09-16T00:17:09.580552+0000","last_became_peered":"2024-09-16T00:17:09.580552+0000","last_unstale":"2024-09-16T00:17:09.581774+0000","last_undegraded":"2024-09-16T00:17:09.581774+0000","last_fullsized":"2024-09-16T00:17:09.581774+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:35.975542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574745+0000","last_change":"2024-09-16T00:15:13.581966+0000","last_active":"2024-09-16T00:17:09.574745+0000","last_peered":"2024-09-16T00:17:09.574745+0000","last_clean":"2024-09-16T00:17:09.574745+0000","last_became_active":"2024-09-16T00:15:13.581282+0000","last_became_peered":"2024-09-16T00:15:13.581282+0000","last_unstale":"2024-09-16T00:17:09.574745+0000","last_undegraded":"2024-09-16T00:17:09.574745+0000","last_fullsized":"2024-09-16T00:17:09.574745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.132773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":156,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:38.328455+0000","last_change":"2024-09-16T00:15:04.351765+0000","last_active":"2024-09-16T00:17:38.328455+0000","last_peered":"2024-09-16T00:17:38.328455+0000","last_clean":"2024-09-16T00:17:38.328455+0000","last_became_active":"2024-09-16T00:15:04.351398+0000","last_became_peered":"2024-09-16T00:15:04.351398+0000","last_unstale":"2024-09-16T00:17:38.328455+0000","last_undegraded":"2024-09-16T00:17:38.328455+0000","last_fullsized":"2024-09-16T00:17:38.328455+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:03.320563+0000","last_clean_scrub_stamp":"2024-09-16T00:15:03.320563+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:14:28.762657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.583630+0000","last_change":"2024-09-16T00:17:09.583255+0000","last_active":"2024-09-16T00:17:09.583630+0000","last_peered":"2024-09-16T00:17:09.583630+0000","last_clean":"2024-09-16T00:17:09.583630+0000","last_became_active":"2024-09-16T00:17:09.582793+0000","last_became_peered":"2024-09-16T00:17:09.582793+0000","last_unstale":"2024-09-16T00:17:09.583630+0000","last_undegraded":"2024-09-16T00:17:09.583630+0000","last_fullsized":"2024-09-16T00:17:09.583630+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:28.087903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574875+0000","last_change":"2024-09-16T00:15:13.576877+0000","last_active":"2024-09-16T00:17:09.574875+0000","last_peered":"2024-09-16T00:17:09.574875+0000","last_clean":"2024-09-16T00:17:09.574875+0000","last_became_active":"2024-09-16T00:15:13.576480+0000","last_became_peered":"2024-09-16T00:15:13.576480+0000","last_unstale":"2024-09-16T00:17:09.574875+0000","last_undegraded":"2024-09-16T00:17:09.574875+0000","last_fullsized":"2024-09-16T00:17:09.574875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:12.554366+0000","last_clean_scrub_stamp":"2024-09-16T00:15:12.554366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:47.322219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.780925+0000","last_change":"2024-09-16T00:15:06.550329+0000","last_active":"2024-09-16T00:17:09.780925+0000","last_peered":"2024-09-16T00:17:09.780925+0000","last_clean":"2024-09-16T00:17:09.780925+0000","last_became_active":"2024-09-16T00:15:06.550008+0000","last_became_peered":"2024-09-16T00:15:06.550008+0000","last_unstale":"2024-09-16T00:17:09.780925+0000","last_undegraded":"2024-09-16T00:17:09.780925+0000","last_fullsized":"2024-09-16T00:17:09.780925+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:38.096126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582169+0000","last_change":"2024-09-16T00:17:09.580839+0000","last_active":"2024-09-16T00:17:09.582169+0000","last_peered":"2024-09-16T00:17:09.582169+0000","last_clean":"2024-09-16T00:17:09.582169+0000","last_became_active":"2024-09-16T00:17:09.580381+0000","last_became_peered":"2024-09-16T00:17:09.580381+0000","last_unstale":"2024-09-16T00:17:09.582169+0000","last_undegraded":"2024-09-16T00:17:09.582169+0000","last_fullsized":"2024-09-16T00:17:09.582169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:35.425316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.574792+0000","last_change":"2024-09-16T00:15:06.546713+0000","last_active":"2024-09-16T00:17:09.574792+0000","last_peered":"2024-09-16T00:17:09.574792+0000","last_clean":"2024-09-16T00:17:09.574792+0000","last_became_active":"2024-09-16T00:15:06.546209+0000","last_became_peered":"2024-09-16T00:15:06.546209+0000","last_unstale":"2024-09-16T00:17:09.574792+0000","last_undegraded":"2024-09-16T00:17:09.574792+0000","last_fullsized":"2024-09-16T00:17:09.574792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:43.913748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577575+0000","last_change":"2024-09-16T00:15:06.550490+0000","last_active":"2024-09-16T00:17:09.577575+0000","last_peered":"2024-09-16T00:17:09.577575+0000","last_clean":"2024-09-16T00:17:09.577575+0000","last_became_active":"2024-09-16T00:15:06.549311+0000","last_became_peered":"2024-09-16T00:15:06.549311+0000","last_unstale":"2024-09-16T00:17:09.577575+0000","last_undegraded":"2024-09-16T00:17:09.577575+0000","last_fullsized":"2024-09-16T00:17:09.577575+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:10:01.995357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.718148+0000","last_change":"2024-09-16T00:15:06.548115+0000","last_active":"2024-09-16T00:17:09.718148+0000","last_peered":"2024-09-16T00:17:09.718148+0000","last_clean":"2024-09-16T00:17:09.718148+0000","last_became_active":"2024-09-16T00:15:06.547782+0000","last_became_peered":"2024-09-16T00:15:06.547782+0000","last_unstale":"2024-09-16T00:17:09.718148+0000","last_undegraded":"2024-09-16T00:17:09.718148+0000","last_fullsized":"2024-09-16T00:17:09.718148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:53.228222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781260+0000","last_change":"2024-09-16T00:15:06.554396+0000","last_active":"2024-09-16T00:17:09.781260+0000","last_peered":"2024-09-16T00:17:09.781260+0000","last_clean":"2024-09-16T00:17:09.781260+0000","last_became_active":"2024-09-16T00:15:06.554012+0000","last_became_peered":"2024-09-16T00:15:06.554012+0000","last_unstale":"2024-09-16T00:17:09.781260+0000","last_undegraded":"2024-09-16T00:17:09.781260+0000","last_fullsized":"2024-09-16T00:17:09.781260+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:31.464098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575494+0000","last_change":"2024-09-16T00:15:06.549969+0000","last_active":"2024-09-16T00:17:09.575494+0000","last_peered":"2024-09-16T00:17:09.575494+0000","last_clean":"2024-09-16T00:17:09.575494+0000","last_became_active":"2024-09-16T00:15:06.549531+0000","last_became_peered":"2024-09-16T00:15:06.549531+0000","last_unstale":"2024-09-16T00:17:09.575494+0000","last_undegraded":"2024-09-16T00:17:09.575494+0000","last_fullsized":"2024-09-16T00:17:09.575494+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:37.883612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579397+0000","last_change":"2024-09-16T00:15:06.547734+0000","last_active":"2024-09-16T00:17:09.579397+0000","last_peered":"2024-09-16T00:17:09.579397+0000","last_clean":"2024-09-16T00:17:09.579397+0000","last_became_active":"2024-09-16T00:15:06.547400+0000","last_became_peered":"2024-09-16T00:15:06.547400+0000","last_unstale":"2024-09-16T00:17:09.579397+0000","last_undegraded":"2024-09-16T00:17:09.579397+0000","last_fullsized":"2024-09-16T00:17:09.579397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:54.455313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576721+0000","last_change":"2024-09-16T00:15:06.551965+0000","last_active":"2024-09-16T00:17:09.576721+0000","last_peered":"2024-09-16T00:17:09.576721+0000","last_clean":"2024-09-16T00:17:09.576721+0000","last_became_active":"2024-09-16T00:15:06.551602+0000","last_became_peered":"2024-09-16T00:15:06.551602+0000","last_unstale":"2024-09-16T00:17:09.576721+0000","last_undegraded":"2024-09-16T00:17:09.576721+0000","last_fullsized":"2024-09-16T00:17:09.576721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:34.951960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579571+0000","last_change":"2024-09-16T00:15:06.549340+0000","last_active":"2024-09-16T00:17:09.579571+0000","last_peered":"2024-09-16T00:17:09.579571+0000","last_clean":"2024-09-16T00:17:09.579571+0000","last_became_active":"2024-09-16T00:15:06.547424+0000","last_became_peered":"2024-09-16T00:15:06.547424+0000","last_unstale":"2024-09-16T00:17:09.579571+0000","last_undegraded":"2024-09-16T00:17:09.579571+0000","last_fullsized":"2024-09-16T00:17:09.579571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:12.087535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580115+0000","last_change":"2024-09-16T00:15:06.555216+0000","last_active":"2024-09-16T00:17:09.580115+0000","last_peered":"2024-09-16T00:17:09.580115+0000","last_clean":"2024-09-16T00:17:09.580115+0000","last_became_active":"2024-09-16T00:15:06.554896+0000","last_became_peered":"2024-09-16T00:15:06.554896+0000","last_unstale":"2024-09-16T00:17:09.580115+0000","last_undegraded":"2024-09-16T00:17:09.580115+0000","last_fullsized":"2024-09-16T00:17:09.580115+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:20.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717980+0000","last_change":"2024-09-16T00:15:06.549752+0000","last_active":"2024-09-16T00:17:09.717980+0000","last_peered":"2024-09-16T00:17:09.717980+0000","last_clean":"2024-09-16T00:17:09.717980+0000","last_became_active":"2024-09-16T00:15:06.549350+0000","last_became_peered":"2024-09-16T00:15:06.549350+0000","last_unstale":"2024-09-16T00:17:09.717980+0000","last_undegraded":"2024-09-16T00:17:09.717980+0000","last_fullsized":"2024-09-16T00:17:09.717980+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:48.615123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781099+0000","last_change":"2024-09-16T00:15:06.549159+0000","last_active":"2024-09-16T00:17:09.781099+0000","last_peered":"2024-09-16T00:17:09.781099+0000","last_clean":"2024-09-16T00:17:09.781099+0000","last_became_active":"2024-09-16T00:15:06.548408+0000","last_became_peered":"2024-09-16T00:15:06.548408+0000","last_unstale":"2024-09-16T00:17:09.781099+0000","last_undegraded":"2024-09-16T00:17:09.781099+0000","last_fullsized":"2024-09-16T00:17:09.781099+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:23.941809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575144+0000","last_change":"2024-09-16T00:15:06.544091+0000","last_active":"2024-09-16T00:17:09.575144+0000","last_peered":"2024-09-16T00:17:09.575144+0000","last_clean":"2024-09-16T00:17:09.575144+0000","last_became_active":"2024-09-16T00:15:06.543761+0000","last_became_peered":"2024-09-16T00:15:06.543761+0000","last_unstale":"2024-09-16T00:17:09.575144+0000","last_undegraded":"2024-09-16T00:17:09.575144+0000","last_fullsized":"2024-09-16T00:17:09.575144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:36:43.253405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.582256+0000","last_change":"2024-09-16T00:17:09.581748+0000","last_active":"2024-09-16T00:17:09.582256+0000","last_peered":"2024-09-16T00:17:09.582256+0000","last_clean":"2024-09-16T00:17:09.582256+0000","last_became_active":"2024-09-16T00:17:09.581207+0000","last_became_peered":"2024-09-16T00:17:09.581207+0000","last_unstale":"2024-09-16T00:17:09.582256+0000","last_undegraded":"2024-09-16T00:17:09.582256+0000","last_fullsized":"2024-09-16T00:17:09.582256+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:21.521929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781344+0000","last_change":"2024-09-16T00:15:06.557162+0000","last_active":"2024-09-16T00:17:09.781344+0000","last_peered":"2024-09-16T00:17:09.781344+0000","last_clean":"2024-09-16T00:17:09.781344+0000","last_became_active":"2024-09-16T00:15:06.556831+0000","last_became_peered":"2024-09-16T00:15:06.556831+0000","last_unstale":"2024-09-16T00:17:09.781344+0000","last_undegraded":"2024-09-16T00:17:09.781344+0000","last_fullsized":"2024-09-16T00:17:09.781344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:52.236915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579887+0000","last_change":"2024-09-16T00:15:06.556161+0000","last_active":"2024-09-16T00:17:09.579887+0000","last_peered":"2024-09-16T00:17:09.579887+0000","last_clean":"2024-09-16T00:17:09.579887+0000","last_became_active":"2024-09-16T00:15:06.555819+0000","last_became_peered":"2024-09-16T00:15:06.555819+0000","last_unstale":"2024-09-16T00:17:09.579887+0000","last_undegraded":"2024-09-16T00:17:09.579887+0000","last_fullsized":"2024-09-16T00:17:09.579887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:29.929977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.583819+0000","last_change":"2024-09-16T00:17:09.583183+0000","last_active":"2024-09-16T00:17:09.583819+0000","last_peered":"2024-09-16T00:17:09.583819+0000","last_clean":"2024-09-16T00:17:09.583819+0000","last_became_active":"2024-09-16T00:17:09.582663+0000","last_became_peered":"2024-09-16T00:17:09.582663+0000","last_unstale":"2024-09-16T00:17:09.583819+0000","last_undegraded":"2024-09-16T00:17:09.583819+0000","last_fullsized":"2024-09-16T00:17:09.583819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:37.544124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577469+0000","last_change":"2024-09-16T00:15:06.547114+0000","last_active":"2024-09-16T00:17:09.577469+0000","last_peered":"2024-09-16T00:17:09.577469+0000","last_clean":"2024-09-16T00:17:09.577469+0000","last_became_active":"2024-09-16T00:15:06.546786+0000","last_became_peered":"2024-09-16T00:15:06.546786+0000","last_unstale":"2024-09-16T00:17:09.577469+0000","last_undegraded":"2024-09-16T00:17:09.577469+0000","last_fullsized":"2024-09-16T00:17:09.577469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:13:26.925603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.717955+0000","last_change":"2024-09-16T00:15:06.541168+0000","last_active":"2024-09-16T00:17:09.717955+0000","last_peered":"2024-09-16T00:17:09.717955+0000","last_clean":"2024-09-16T00:17:09.717955+0000","last_became_active":"2024-09-16T00:15:06.540848+0000","last_became_peered":"2024-09-16T00:15:06.540848+0000","last_unstale":"2024-09-16T00:17:09.717955+0000","last_undegraded":"2024-09-16T00:17:09.717955+0000","last_fullsized":"2024-09-16T00:17:09.717955+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.958600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575843+0000","last_change":"2024-09-16T00:15:06.547429+0000","last_active":"2024-09-16T00:17:09.575843+0000","last_peered":"2024-09-16T00:17:09.575843+0000","last_clean":"2024-09-16T00:17:09.575843+0000","last_became_active":"2024-09-16T00:15:06.547081+0000","last_became_peered":"2024-09-16T00:15:06.547081+0000","last_unstale":"2024-09-16T00:17:09.575843+0000","last_undegraded":"2024-09-16T00:17:09.575843+0000","last_fullsized":"2024-09-16T00:17:09.575843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:43.396313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581339+0000","last_change":"2024-09-16T00:15:06.551811+0000","last_active":"2024-09-16T00:17:09.581339+0000","last_peered":"2024-09-16T00:17:09.581339+0000","last_clean":"2024-09-16T00:17:09.581339+0000","last_became_active":"2024-09-16T00:15:06.551424+0000","last_became_peered":"2024-09-16T00:15:06.551424+0000","last_unstale":"2024-09-16T00:17:09.581339+0000","last_undegraded":"2024-09-16T00:17:09.581339+0000","last_fullsized":"2024-09-16T00:17:09.581339+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.943070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.578077+0000","last_change":"2024-09-16T00:15:06.544697+0000","last_active":"2024-09-16T00:17:09.578077+0000","last_peered":"2024-09-16T00:17:09.578077+0000","last_clean":"2024-09-16T00:17:09.578077+0000","last_became_active":"2024-09-16T00:15:06.544182+0000","last_became_peered":"2024-09-16T00:15:06.544182+0000","last_unstale":"2024-09-16T00:17:09.578077+0000","last_undegraded":"2024-09-16T00:17:09.578077+0000","last_fullsized":"2024-09-16T00:17:09.578077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:45.929092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.580342+0000","last_change":"2024-09-16T00:15:06.551488+0000","last_active":"2024-09-16T00:17:09.580342+0000","last_peered":"2024-09-16T00:17:09.580342+0000","last_clean":"2024-09-16T00:17:09.580342+0000","last_became_active":"2024-09-16T00:15:06.551171+0000","last_became_peered":"2024-09-16T00:15:06.551171+0000","last_unstale":"2024-09-16T00:17:09.580342+0000","last_undegraded":"2024-09-16T00:17:09.580342+0000","last_fullsized":"2024-09-16T00:17:09.580342+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:23.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579150+0000","last_change":"2024-09-16T00:15:06.544043+0000","last_active":"2024-09-16T00:17:09.579150+0000","last_peered":"2024-09-16T00:17:09.579150+0000","last_clean":"2024-09-16T00:17:09.579150+0000","last_became_active":"2024-09-16T00:15:06.543014+0000","last_became_peered":"2024-09-16T00:15:06.543014+0000","last_unstale":"2024-09-16T00:17:09.579150+0000","last_undegraded":"2024-09-16T00:17:09.579150+0000","last_fullsized":"2024-09-16T00:17:09.579150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:23.904810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.581774+0000","last_change":"2024-09-16T00:15:06.546788+0000","last_active":"2024-09-16T00:17:09.581774+0000","last_peered":"2024-09-16T00:17:09.581774+0000","last_clean":"2024-09-16T00:17:09.581774+0000","last_became_active":"2024-09-16T00:15:06.546333+0000","last_became_peered":"2024-09-16T00:15:06.546333+0000","last_unstale":"2024-09-16T00:17:09.581774+0000","last_undegraded":"2024-09-16T00:17:09.581774+0000","last_fullsized":"2024-09-16T00:17:09.581774+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:09.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.720091+0000","last_change":"2024-09-16T00:17:09.719618+0000","last_active":"2024-09-16T00:17:09.720091+0000","last_peered":"2024-09-16T00:17:09.720091+0000","last_clean":"2024-09-16T00:17:09.720091+0000","last_became_active":"2024-09-16T00:17:09.719172+0000","last_became_peered":"2024-09-16T00:17:09.719172+0000","last_unstale":"2024-09-16T00:17:09.720091+0000","last_undegraded":"2024-09-16T00:17:09.720091+0000","last_fullsized":"2024-09-16T00:17:09.720091+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:51.709719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.579341+0000","last_change":"2024-09-16T00:15:06.540675+0000","last_active":"2024-09-16T00:17:09.579341+0000","last_peered":"2024-09-16T00:17:09.579341+0000","last_clean":"2024-09-16T00:17:09.579341+0000","last_became_active":"2024-09-16T00:15:06.540333+0000","last_became_peered":"2024-09-16T00:15:06.540333+0000","last_unstale":"2024-09-16T00:17:09.579341+0000","last_undegraded":"2024-09-16T00:17:09.579341+0000","last_fullsized":"2024-09-16T00:17:09.579341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:17.166987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.575821+0000","last_change":"2024-09-16T00:15:06.548758+0000","last_active":"2024-09-16T00:17:09.575821+0000","last_peered":"2024-09-16T00:17:09.575821+0000","last_clean":"2024-09-16T00:17:09.575821+0000","last_became_active":"2024-09-16T00:15:06.548397+0000","last_became_peered":"2024-09-16T00:15:06.548397+0000","last_unstale":"2024-09-16T00:17:09.575821+0000","last_undegraded":"2024-09-16T00:17:09.575821+0000","last_fullsized":"2024-09-16T00:17:09.575821+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:50.320170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.576644+0000","last_change":"2024-09-16T00:15:06.554898+0000","last_active":"2024-09-16T00:17:09.576644+0000","last_peered":"2024-09-16T00:17:09.576644+0000","last_clean":"2024-09-16T00:17:09.576644+0000","last_became_active":"2024-09-16T00:15:06.553005+0000","last_became_peered":"2024-09-16T00:15:06.553005+0000","last_unstale":"2024-09-16T00:17:09.576644+0000","last_undegraded":"2024-09-16T00:17:09.576644+0000","last_fullsized":"2024-09-16T00:17:09.576644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:29.630911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.577231+0000","last_change":"2024-09-16T00:15:06.547265+0000","last_active":"2024-09-16T00:17:09.577231+0000","last_peered":"2024-09-16T00:17:09.577231+0000","last_clean":"2024-09-16T00:17:09.577231+0000","last_became_active":"2024-09-16T00:15:06.546938+0000","last_became_peered":"2024-09-16T00:15:06.546938+0000","last_unstale":"2024-09-16T00:17:09.577231+0000","last_undegraded":"2024-09-16T00:17:09.577231+0000","last_fullsized":"2024-09-16T00:17:09.577231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:40.247343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T00:17:09.781308+0000","last_change":"2024-09-16T00:15:06.556735+0000","last_active":"2024-09-16T00:17:09.781308+0000","last_peered":"2024-09-16T00:17:09.781308+0000","last_clean":"2024-09-16T00:17:09.781308+0000","last_became_active":"2024-09-16T00:15:06.556324+0000","last_became_peered":"2024-09-16T00:15:06.556324+0000","last_unstale":"2024-09-16T00:17:09.781308+0000","last_undegraded":"2024-09-16T00:17:09.781308+0000","last_fullsized":"2024-09-16T00:17:09.781308+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T00:15:05.512228+0000","last_clean_scrub_stamp":"2024-09-16T00:15:05.512228+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:25.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":76311,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":684,"num_read_kb":412,"num_write":182,"num_write_kb":135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":62107,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":230071,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249414,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579949,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":517057,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808881,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36468,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708684,"statfs":{"total":95995035648,"available":95957692416,"internally_reserved":0,"allocated":1150976,"data_stored":1177026,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":521389,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201339,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":516631,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430273,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1060864,"data_stored":1113658,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626503,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":512469,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855436,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1069056,"data_stored":1108319,"data_compressed":65730,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986257,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1044480,"data_stored":591540,"data_compressed":39297,"data_compressed_allocated":53248,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215189,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":512663,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444123,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1007616,"data_stored":513147,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607648,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":584735,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1015808,"data_stored":513857,"data_compressed":29397,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34694,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36333,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":72356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":75065,"data_compressed":9900,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":74279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T00:17:43.910 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:17:44.168 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:44.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:43 smithi083 ceph-mon[35242]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:44.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:43 smithi060 ceph-mon[35185]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:44.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:43 smithi033 ceph-mon[31243]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T00:17:44.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.711+0000 7f6df579d640 1 -- 172.21.15.33:0/244613401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df0101de0 msgr2=0x7f6df01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:44.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.711+0000 7f6df579d640 1 --2- 172.21.15.33:0/244613401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df0101de0 0x7f6df01021c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009900 tx=0x7f6de002a410 comp rx=0 tx=0).stop 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.711+0000 7f6df579d640 1 -- 172.21.15.33:0/244613401 shutdown_connections 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.711+0000 7f6df579d640 1 --2- 172.21.15.33:0/244613401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6df0106f10 0x7f6df010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.711+0000 7f6df579d640 1 --2- 172.21.15.33:0/244613401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6df0102790 0x7f6df01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.711+0000 7f6df579d640 1 --2- 172.21.15.33:0/244613401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df0101de0 0x7f6df01021c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.711+0000 7f6df579d640 1 -- 172.21.15.33:0/244613401 >> 172.21.15.33:0/244613401 conn(0x7f6df00fd8c0 msgr2=0x7f6df00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.711+0000 7f6df579d640 1 -- 172.21.15.33:0/244613401 shutdown_connections 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.711+0000 7f6df579d640 1 -- 172.21.15.33:0/244613401 wait complete. 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6df579d640 1 Processor -- start 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6df579d640 1 -- start start 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6df579d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6df0101de0 0x7f6df00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6df579d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df0102790 0x7f6df0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6df579d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6df0106f10 0x7f6df0074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:44.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6df579d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6df0111f70 con 0x7f6df0102790 2024-09-16T00:17:44.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6df579d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6df0111df0 con 0x7f6df0101de0 2024-09-16T00:17:44.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6df579d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6df01120f0 con 0x7f6df0106f10 2024-09-16T00:17:44.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6de7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df0102790 0x7f6df0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:44.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6de7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df0102790 0x7f6df0074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58980/0 (socket says 172.21.15.33:58980) 2024-09-16T00:17:44.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6de7fff640 1 -- 172.21.15.33:0/2500996443 learned_addr learned my addr 172.21.15.33:0/2500996443 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:44.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6deeffd640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6df0101de0 0x7f6df00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:44.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.712+0000 7f6def7fe640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6df0106f10 0x7f6df0074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:44.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.713+0000 7f6de7fff640 1 -- 172.21.15.33:0/2500996443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6df0106f10 msgr2=0x7f6df0074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:44.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.713+0000 7f6de7fff640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6df0106f10 0x7f6df0074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:44.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.713+0000 7f6de7fff640 1 -- 172.21.15.33:0/2500996443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6df0101de0 msgr2=0x7f6df00744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.713+0000 7f6de7fff640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6df0101de0 0x7f6df00744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.713+0000 7f6de7fff640 1 -- 172.21.15.33:0/2500996443 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df01a68f0 con 0x7f6df0102790 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.713+0000 7f6def7fe640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6df0106f10 0x7f6df0074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.713+0000 7f6de7fff640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df0102790 0x7f6df0074a30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800d950 tx=0x7f6dd800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.713+0000 7f6decff9640 1 -- 172.21.15.33:0/2500996443 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8017070 con 0x7f6df0102790 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.713+0000 7f6df579d640 1 -- 172.21.15.33:0/2500996443 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df01a6be0 con 0x7f6df0102790 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.713+0000 7f6decff9640 1 -- 172.21.15.33:0/2500996443 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dd8009c30 con 0x7f6df0102790 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.713+0000 7f6decff9640 1 -- 172.21.15.33:0/2500996443 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd800f6b0 con 0x7f6df0102790 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.713+0000 7f6df579d640 1 -- 172.21.15.33:0/2500996443 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df01a7170 con 0x7f6df0102790 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.714+0000 7f6df579d640 1 -- 172.21.15.33:0/2500996443 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6df01021c0 con 0x7f6df0102790 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.715+0000 7f6decff9640 1 -- 172.21.15.33:0/2500996443 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f6dd80109d0 con 0x7f6df0102790 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.716+0000 7f6decff9640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6ddc0a32d0 0x7f6ddc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:44.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.716+0000 7f6deeffd640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6ddc0a32d0 0x7f6ddc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:44.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.716+0000 7f6decff9640 1 -- 172.21.15.33:0/2500996443 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6dd80cfba0 con 0x7f6df0102790 2024-09-16T00:17:44.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.716+0000 7f6deeffd640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6ddc0a32d0 0x7f6ddc0a5790 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009230 tx=0x7f6de00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:44.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.719+0000 7f6decff9640 1 -- 172.21.15.33:0/2500996443 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6dd80cfe20 con 0x7f6df0102790 2024-09-16T00:17:44.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.931+0000 7f6df579d640 1 -- 172.21.15.33:0/2500996443 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6df002d7c0 con 0x7f6df0102790 2024-09-16T00:17:44.933 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:44 smithi033 ceph-mon[31243]: pgmap v368: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 6 op/s 2024-09-16T00:17:44.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.932+0000 7f6decff9640 1 -- 172.21.15.33:0/2500996443 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11903 (secure 0 0 0) 0x7f6dd8009de0 con 0x7f6df0102790 2024-09-16T00:17:44.934 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:44.934 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":26,"btime":"2024-09-16T00:17:28:789138+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:17:28.493434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:17:44.934 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 26 2024-09-16T00:17:44.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.934+0000 7f6df579d640 1 -- 172.21.15.33:0/2500996443 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6ddc0a32d0 msgr2=0x7f6ddc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:44.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.934+0000 7f6df579d640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6ddc0a32d0 0x7f6ddc0a5790 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009230 tx=0x7f6de00058d0 comp rx=0 tx=0).stop 2024-09-16T00:17:44.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.934+0000 7f6df579d640 1 -- 172.21.15.33:0/2500996443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df0102790 msgr2=0x7f6df0074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:44.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.935+0000 7f6df579d640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df0102790 0x7f6df0074a30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800d950 tx=0x7f6dd800de10 comp rx=0 tx=0).stop 2024-09-16T00:17:44.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.935+0000 7f6df579d640 1 -- 172.21.15.33:0/2500996443 shutdown_connections 2024-09-16T00:17:44.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.935+0000 7f6df579d640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6ddc0a32d0 0x7f6ddc0a5790 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:44.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.935+0000 7f6df579d640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6df0106f10 0x7f6df0074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:44.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.935+0000 7f6df579d640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df0102790 0x7f6df0074a30 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:44.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.935+0000 7f6df579d640 1 --2- 172.21.15.33:0/2500996443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6df0101de0 0x7f6df00744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:44.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.935+0000 7f6df579d640 1 -- 172.21.15.33:0/2500996443 >> 172.21.15.33:0/2500996443 conn(0x7f6df00fd8c0 msgr2=0x7f6df00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:44.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.935+0000 7f6df579d640 1 -- 172.21.15.33:0/2500996443 shutdown_connections 2024-09-16T00:17:44.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:44.935+0000 7f6df579d640 1 -- 172.21.15.33:0/2500996443 wait complete. 2024-09-16T00:17:45.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:44 smithi083 ceph-mon[35242]: pgmap v368: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 6 op/s 2024-09-16T00:17:45.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:44 smithi060 ceph-mon[35185]: pgmap v368: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 6 op/s 2024-09-16T00:17:45.630 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:17:45.889 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:46.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:45 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2500996443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:46.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:45 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2500996443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:46.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:45 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2500996443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:46.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.394+0000 7f4c5d50f640 1 -- 172.21.15.33:0/710659081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c58102ef0 msgr2=0x7f4c5810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:46.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.394+0000 7f4c5d50f640 1 --2- 172.21.15.33:0/710659081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c58102ef0 0x7f4c5810da40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4c48009900 tx=0x7f4c4802a480 comp rx=0 tx=0).stop 2024-09-16T00:17:46.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.394+0000 7f4c5d50f640 1 -- 172.21.15.33:0/710659081 shutdown_connections 2024-09-16T00:17:46.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.394+0000 7f4c5d50f640 1 --2- 172.21.15.33:0/710659081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c58102ef0 0x7f4c5810da40 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:46.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.394+0000 7f4c5d50f640 1 --2- 172.21.15.33:0/710659081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c58102570 0x7f4c581029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:46.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.394+0000 7f4c5d50f640 1 --2- 172.21.15.33:0/710659081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c58100e20 0x7f4c58101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:46.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.394+0000 7f4c5d50f640 1 -- 172.21.15.33:0/710659081 >> 172.21.15.33:0/710659081 conn(0x7f4c580fc7d0 msgr2=0x7f4c580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:46.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.394+0000 7f4c5d50f640 1 -- 172.21.15.33:0/710659081 shutdown_connections 2024-09-16T00:17:46.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.394+0000 7f4c5d50f640 1 -- 172.21.15.33:0/710659081 wait complete. 2024-09-16T00:17:46.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.394+0000 7f4c5d50f640 1 Processor -- start 2024-09-16T00:17:46.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.394+0000 7f4c5d50f640 1 -- start start 2024-09-16T00:17:46.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c5d50f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c58100e20 0x7f4c58198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:46.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c5d50f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c58102570 0x7f4c58198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:46.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c5d50f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c58102ef0 0x7f4c5819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:46.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c5d50f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4c58110f80 con 0x7f4c58100e20 2024-09-16T00:17:46.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c5d50f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4c58110e00 con 0x7f4c58102ef0 2024-09-16T00:17:46.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c5d50f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4c58111100 con 0x7f4c58102570 2024-09-16T00:17:46.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c56ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c58100e20 0x7f4c58198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:46.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c56ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c58100e20 0x7f4c58198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58994/0 (socket says 172.21.15.33:58994) 2024-09-16T00:17:46.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c56ffd640 1 -- 172.21.15.33:0/2869466121 learned_addr learned my addr 172.21.15.33:0/2869466121 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:46.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c567fc640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c58102570 0x7f4c58198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:46.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c577fe640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c58102ef0 0x7f4c5819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:46.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c56ffd640 1 -- 172.21.15.33:0/2869466121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c58102570 msgr2=0x7f4c58198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:46.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c56ffd640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c58102570 0x7f4c58198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:46.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c56ffd640 1 -- 172.21.15.33:0/2869466121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c58102ef0 msgr2=0x7f4c5819cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:46.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c56ffd640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c58102ef0 0x7f4c5819cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:46.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c56ffd640 1 -- 172.21.15.33:0/2869466121 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5819d4b0 con 0x7f4c58100e20 2024-09-16T00:17:46.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.395+0000 7f4c56ffd640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c58100e20 0x7f4c58198500 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000ea80 tx=0x7f4c5000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:46.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.396+0000 7f4c4ffff640 1 -- 172.21.15.33:0/2869466121 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5000ce40 con 0x7f4c58100e20 2024-09-16T00:17:46.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.396+0000 7f4c5d50f640 1 -- 172.21.15.33:0/2869466121 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5819d7a0 con 0x7f4c58100e20 2024-09-16T00:17:46.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.396+0000 7f4c4ffff640 1 -- 172.21.15.33:0/2869466121 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c50005020 con 0x7f4c58100e20 2024-09-16T00:17:46.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.396+0000 7f4c4ffff640 1 -- 172.21.15.33:0/2869466121 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c50011760 con 0x7f4c58100e20 2024-09-16T00:17:46.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.396+0000 7f4c5d50f640 1 -- 172.21.15.33:0/2869466121 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c581a5080 con 0x7f4c58100e20 2024-09-16T00:17:46.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.397+0000 7f4c5d50f640 1 -- 172.21.15.33:0/2869466121 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c58101200 con 0x7f4c58100e20 2024-09-16T00:17:46.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.398+0000 7f4c4ffff640 1 -- 172.21.15.33:0/2869466121 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f4c500051d0 con 0x7f4c58100e20 2024-09-16T00:17:46.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.398+0000 7f4c4ffff640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4c440a32d0 0x7f4c440a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:46.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.399+0000 7f4c567fc640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4c440a32d0 0x7f4c440a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:46.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.399+0000 7f4c4ffff640 1 -- 172.21.15.33:0/2869466121 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4c500d05b0 con 0x7f4c58100e20 2024-09-16T00:17:46.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.399+0000 7f4c567fc640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4c440a32d0 0x7f4c440a5790 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4c580ff0c0 tx=0x7f4c4000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:46.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.403+0000 7f4c4ffff640 1 -- 172.21.15.33:0/2869466121 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4c50016030 con 0x7f4c58100e20 2024-09-16T00:17:46.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.618+0000 7f4c5d50f640 1 -- 172.21.15.33:0/2869466121 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c58199860 con 0x7f4c58100e20 2024-09-16T00:17:46.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.619+0000 7f4c4ffff640 1 -- 172.21.15.33:0/2869466121 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11903 (secure 0 0 0) 0x7f4c5009ccf0 con 0x7f4c58100e20 2024-09-16T00:17:46.620 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:46.620 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":26,"btime":"2024-09-16T00:17:28:789138+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:17:28.493434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:17:46.621 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 26 2024-09-16T00:17:46.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.621+0000 7f4c5d50f640 1 -- 172.21.15.33:0/2869466121 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4c440a32d0 msgr2=0x7f4c440a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:46.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.621+0000 7f4c5d50f640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4c440a32d0 0x7f4c440a5790 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4c580ff0c0 tx=0x7f4c4000b390 comp rx=0 tx=0).stop 2024-09-16T00:17:46.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.621+0000 7f4c5d50f640 1 -- 172.21.15.33:0/2869466121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c58100e20 msgr2=0x7f4c58198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:46.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.621+0000 7f4c5d50f640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c58100e20 0x7f4c58198500 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000ea80 tx=0x7f4c5000ef40 comp rx=0 tx=0).stop 2024-09-16T00:17:46.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.621+0000 7f4c5d50f640 1 -- 172.21.15.33:0/2869466121 shutdown_connections 2024-09-16T00:17:46.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.621+0000 7f4c5d50f640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4c440a32d0 0x7f4c440a5790 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:46.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.621+0000 7f4c5d50f640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c58102ef0 0x7f4c5819cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:46.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.621+0000 7f4c5d50f640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c58102570 0x7f4c58198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:46.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.621+0000 7f4c5d50f640 1 --2- 172.21.15.33:0/2869466121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c58100e20 0x7f4c58198500 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:46.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.621+0000 7f4c5d50f640 1 -- 172.21.15.33:0/2869466121 >> 172.21.15.33:0/2869466121 conn(0x7f4c580fc7d0 msgr2=0x7f4c5806a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:46.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.622+0000 7f4c5d50f640 1 -- 172.21.15.33:0/2869466121 shutdown_connections 2024-09-16T00:17:46.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:46.622+0000 7f4c5d50f640 1 -- 172.21.15.33:0/2869466121 wait complete. 2024-09-16T00:17:47.047 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:46 smithi033 ceph-mon[31243]: pgmap v369: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T00:17:47.047 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:46 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2869466121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:47.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:46 smithi083 ceph-mon[35242]: pgmap v369: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T00:17:47.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:46 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2869466121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:47.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:46 smithi060 ceph-mon[35185]: pgmap v369: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T00:17:47.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:46 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2869466121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:47.339 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:17:47.594 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:48.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.044+0000 7f6b1fa93640 1 -- 172.21.15.33:0/3396343108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18101de0 msgr2=0x7f6b181021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:48.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.044+0000 7f6b1fa93640 1 --2- 172.21.15.33:0/3396343108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18101de0 0x7f6b181021c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009900 tx=0x7f6b1402a410 comp rx=0 tx=0).stop 2024-09-16T00:17:48.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.044+0000 7f6b1fa93640 1 -- 172.21.15.33:0/3396343108 shutdown_connections 2024-09-16T00:17:48.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.044+0000 7f6b1fa93640 1 --2- 172.21.15.33:0/3396343108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18106f10 0x7f6b1810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:48.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.044+0000 7f6b1fa93640 1 --2- 172.21.15.33:0/3396343108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18102790 0x7f6b181067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:48.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.044+0000 7f6b1fa93640 1 --2- 172.21.15.33:0/3396343108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18101de0 0x7f6b181021c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:48.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.044+0000 7f6b1fa93640 1 -- 172.21.15.33:0/3396343108 >> 172.21.15.33:0/3396343108 conn(0x7f6b180fd8c0 msgr2=0x7f6b180ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:48.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.044+0000 7f6b1fa93640 1 -- 172.21.15.33:0/3396343108 shutdown_connections 2024-09-16T00:17:48.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.044+0000 7f6b1fa93640 1 -- 172.21.15.33:0/3396343108 wait complete. 2024-09-16T00:17:48.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1fa93640 1 Processor -- start 2024-09-16T00:17:48.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1fa93640 1 -- start start 2024-09-16T00:17:48.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1fa93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18101de0 0x7f6b181984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:48.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1fa93640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102790 0x7f6b181989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:48.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1fa93640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18106f10 0x7f6b1819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1fa93640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6b18111f90 con 0x7f6b18102790 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1fa93640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6b18111e10 con 0x7f6b18106f10 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1fa93640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6b18112110 con 0x7f6b18101de0 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1d02e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102790 0x7f6b181989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1d02e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102790 0x7f6b181989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59016/0 (socket says 172.21.15.33:59016) 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1d02e640 1 -- 172.21.15.33:0/4067355275 learned_addr learned my addr 172.21.15.33:0/4067355275 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1d82f640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18101de0 0x7f6b181984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1e030640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18106f10 0x7f6b1819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1d02e640 1 -- 172.21.15.33:0/4067355275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18101de0 msgr2=0x7f6b181984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.045+0000 7f6b1d02e640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18101de0 0x7f6b181984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.046+0000 7f6b1d02e640 1 -- 172.21.15.33:0/4067355275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18106f10 msgr2=0x7f6b1819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.046+0000 7f6b1d02e640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18106f10 0x7f6b1819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:48.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.046+0000 7f6b1d02e640 1 -- 172.21.15.33:0/4067355275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1819d460 con 0x7f6b18102790 2024-09-16T00:17:48.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.046+0000 7f6b1d02e640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102790 0x7f6b181989f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800b4a0 tx=0x7f6b0800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:48.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.046+0000 7f6b0effd640 1 -- 172.21.15.33:0/4067355275 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b08014070 con 0x7f6b18102790 2024-09-16T00:17:48.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.046+0000 7f6b0effd640 1 -- 172.21.15.33:0/4067355275 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b08010af0 con 0x7f6b18102790 2024-09-16T00:17:48.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.046+0000 7f6b1fa93640 1 -- 172.21.15.33:0/4067355275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b1819d750 con 0x7f6b18102790 2024-09-16T00:17:48.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.046+0000 7f6b0effd640 1 -- 172.21.15.33:0/4067355275 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b08017d10 con 0x7f6b18102790 2024-09-16T00:17:48.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.046+0000 7f6b1fa93640 1 -- 172.21.15.33:0/4067355275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b181a5030 con 0x7f6b18102790 2024-09-16T00:17:48.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.047+0000 7f6b1fa93640 1 -- 172.21.15.33:0/4067355275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b181021c0 con 0x7f6b18102790 2024-09-16T00:17:48.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.048+0000 7f6b0effd640 1 -- 172.21.15.33:0/4067355275 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f6b08017070 con 0x7f6b18102790 2024-09-16T00:17:48.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.049+0000 7f6b0effd640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b040a32d0 0x7f6b040a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:48.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.049+0000 7f6b1d82f640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b040a32d0 0x7f6b040a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:48.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.049+0000 7f6b0effd640 1 -- 172.21.15.33:0/4067355275 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6b080cf250 con 0x7f6b18102790 2024-09-16T00:17:48.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.049+0000 7f6b1d82f640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b040a32d0 0x7f6b040a5790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6b14000c00 tx=0x7f6b140058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:48.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.053+0000 7f6b0effd640 1 -- 172.21.15.33:0/4067355275 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6b0809baa0 con 0x7f6b18102790 2024-09-16T00:17:48.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.231+0000 7f6b1fa93640 1 -- 172.21.15.33:0/4067355275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b1810a150 con 0x7f6b18102790 2024-09-16T00:17:48.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.233+0000 7f6b0effd640 1 -- 172.21.15.33:0/4067355275 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25076 (secure 0 0 0) 0x7f6b0809b3a0 con 0x7f6b18102790 2024-09-16T00:17:48.235 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:48.235 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":90,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:17:09.569270+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2e","mappings":[{"from":1,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:17:48.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.236+0000 7f6b1fa93640 1 -- 172.21.15.33:0/4067355275 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b040a32d0 msgr2=0x7f6b040a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:48.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.236+0000 7f6b1fa93640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b040a32d0 0x7f6b040a5790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6b14000c00 tx=0x7f6b140058d0 comp rx=0 tx=0).stop 2024-09-16T00:17:48.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.236+0000 7f6b1fa93640 1 -- 172.21.15.33:0/4067355275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102790 msgr2=0x7f6b181989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:48.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.236+0000 7f6b1fa93640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102790 0x7f6b181989f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800b4a0 tx=0x7f6b0800b960 comp rx=0 tx=0).stop 2024-09-16T00:17:48.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.236+0000 7f6b1fa93640 1 -- 172.21.15.33:0/4067355275 shutdown_connections 2024-09-16T00:17:48.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.236+0000 7f6b1fa93640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6b040a32d0 0x7f6b040a5790 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:48.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.236+0000 7f6b1fa93640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b18106f10 0x7f6b1819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:48.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.236+0000 7f6b1fa93640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b18102790 0x7f6b181989f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:48.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.236+0000 7f6b1fa93640 1 --2- 172.21.15.33:0/4067355275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b18101de0 0x7f6b181984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:48.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.236+0000 7f6b1fa93640 1 -- 172.21.15.33:0/4067355275 >> 172.21.15.33:0/4067355275 conn(0x7f6b180fd8c0 msgr2=0x7f6b180feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:48.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.236+0000 7f6b1fa93640 1 -- 172.21.15.33:0/4067355275 shutdown_connections 2024-09-16T00:17:48.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:48.236+0000 7f6b1fa93640 1 -- 172.21.15.33:0/4067355275 wait complete. 2024-09-16T00:17:48.975 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-09-16T00:17:49.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:48 smithi083 ceph-mon[35242]: pgmap v370: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-09-16T00:17:49.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:48 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4067355275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:17:49.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:48 smithi060 ceph-mon[35185]: pgmap v370: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-09-16T00:17:49.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:48 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4067355275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:17:49.236 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:49.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:48 smithi033 ceph-mon[31243]: pgmap v370: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-09-16T00:17:49.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:48 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4067355275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:17:50.434 INFO:teuthology.orchestra.run.smithi033.stdout:{"run_dir":"/var/run/ceph"} 2024-09-16T00:17:50.435 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-09-16T00:17:50.435 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 3393 2024-09-16T00:17:50.435 DEBUG:tasks.quiescer:Yielding 2024-09-16T00:17:50.435 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T00:17:50.458 INFO:tasks.workunit:Pulling workunits from ref ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T00:17:50.459 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T00:17:50.459 INFO:tasks.workunit:timeout=3h 2024-09-16T00:17:50.459 INFO:tasks.workunit:cleanup=True 2024-09-16T00:17:50.459 DEBUG:teuthology.orchestra.run.smithi033:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T00:17:50.460 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:17:50.479 INFO:teuthology.orchestra.run.smithi033.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-16T00:17:50.479 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-16T00:17:50.479 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 78h/120d Inode: 1099511628283 Links: 2 2024-09-16T00:17:50.479 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T00:17:50.479 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-16T00:17:50.479 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-09-16 00:15:58.799003769 +0000 2024-09-16T00:17:50.479 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-09-16 00:15:58.799003769 +0000 2024-09-16T00:17:50.479 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-09-16 00:17:27.811255210 +0000 2024-09-16T00:17:50.479 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-09-16 00:15:58.799003769 +0000 2024-09-16T00:17:50.480 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-16T00:17:50.480 DEBUG:teuthology.orchestra.run.smithi033:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-16T00:17:50.555 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T00:17:50.612 INFO:tasks.workunit.client.0.smithi033.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T00:17:50.725 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:51.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:50 smithi033 ceph-mon[31243]: pgmap v371: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-16T00:17:51.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:50 smithi083 ceph-mon[35242]: pgmap v371: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-16T00:17:51.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:50 smithi060 ceph-mon[35185]: pgmap v371: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-16T00:17:51.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.243+0000 7fda71839640 1 -- 172.21.15.33:0/282797625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fda6c102570 msgr2=0x7fda6c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:51.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.243+0000 7fda71839640 1 --2- 172.21.15.33:0/282797625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fda6c102570 0x7fda6c1029b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fda60009900 tx=0x7fda6002a480 comp rx=0 tx=0).stop 2024-09-16T00:17:51.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.243+0000 7fda71839640 1 -- 172.21.15.33:0/282797625 shutdown_connections 2024-09-16T00:17:51.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.243+0000 7fda71839640 1 --2- 172.21.15.33:0/282797625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda6c102ef0 0x7fda6c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:51.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.243+0000 7fda71839640 1 --2- 172.21.15.33:0/282797625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fda6c102570 0x7fda6c1029b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:51.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.243+0000 7fda71839640 1 --2- 172.21.15.33:0/282797625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda6c100e20 0x7fda6c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:51.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.243+0000 7fda71839640 1 -- 172.21.15.33:0/282797625 >> 172.21.15.33:0/282797625 conn(0x7fda6c0fc7d0 msgr2=0x7fda6c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:51.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.243+0000 7fda71839640 1 -- 172.21.15.33:0/282797625 shutdown_connections 2024-09-16T00:17:51.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.243+0000 7fda71839640 1 -- 172.21.15.33:0/282797625 wait complete. 2024-09-16T00:17:51.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.244+0000 7fda71839640 1 Processor -- start 2024-09-16T00:17:51.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.244+0000 7fda71839640 1 -- start start 2024-09-16T00:17:51.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.244+0000 7fda71839640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda6c100e20 0x7fda6c197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:51.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.244+0000 7fda71839640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda6c102570 0x7fda6c197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:51.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.244+0000 7fda71839640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fda6c102ef0 0x7fda6c19bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:51.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.244+0000 7fda71839640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c110f80 con 0x7fda6c102ef0 2024-09-16T00:17:51.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.244+0000 7fda71839640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c110e00 con 0x7fda6c102570 2024-09-16T00:17:51.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.244+0000 7fda71839640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c111100 con 0x7fda6c100e20 2024-09-16T00:17:51.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda6b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fda6c102ef0 0x7fda6c19bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:51.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda6b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fda6c102ef0 0x7fda6c19bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33762/0 (socket says 172.21.15.33:33762) 2024-09-16T00:17:51.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda6a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda6c102570 0x7fda6c197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:51.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda6b7fe640 1 -- 172.21.15.33:0/2095172400 learned_addr learned my addr 172.21.15.33:0/2095172400 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:51.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda6affd640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda6c100e20 0x7fda6c197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:51.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda6b7fe640 1 -- 172.21.15.33:0/2095172400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda6c100e20 msgr2=0x7fda6c197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:51.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda6b7fe640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda6c100e20 0x7fda6c197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:51.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda6b7fe640 1 -- 172.21.15.33:0/2095172400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda6c102570 msgr2=0x7fda6c197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:51.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda6b7fe640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda6c102570 0x7fda6c197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:51.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda6b7fe640 1 -- 172.21.15.33:0/2095172400 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6c19c3d0 con 0x7fda6c102ef0 2024-09-16T00:17:51.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda6affd640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda6c100e20 0x7fda6c197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:51.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda6b7fe640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fda6c102ef0 0x7fda6c19bcf0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fda6400cce0 tx=0x7fda6400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:51.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda70837640 1 -- 172.21.15.33:0/2095172400 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6400f450 con 0x7fda6c102ef0 2024-09-16T00:17:51.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda70837640 1 -- 172.21.15.33:0/2095172400 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda6400fad0 con 0x7fda6c102ef0 2024-09-16T00:17:51.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda71839640 1 -- 172.21.15.33:0/2095172400 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda6c19c6c0 con 0x7fda6c102ef0 2024-09-16T00:17:51.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.245+0000 7fda70837640 1 -- 172.21.15.33:0/2095172400 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda64018c10 con 0x7fda6c102ef0 2024-09-16T00:17:51.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.246+0000 7fda71839640 1 -- 172.21.15.33:0/2095172400 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda6c1a3fa0 con 0x7fda6c102ef0 2024-09-16T00:17:51.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.247+0000 7fda71839640 1 -- 172.21.15.33:0/2095172400 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda6c101200 con 0x7fda6c102ef0 2024-09-16T00:17:51.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.249+0000 7fda70837640 1 -- 172.21.15.33:0/2095172400 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fda6400f5f0 con 0x7fda6c102ef0 2024-09-16T00:17:51.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.249+0000 7fda70837640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fda540a32d0 0x7fda540a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:51.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.249+0000 7fda6affd640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fda540a32d0 0x7fda540a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:51.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.249+0000 7fda70837640 1 -- 172.21.15.33:0/2095172400 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fda640d05a0 con 0x7fda6c102ef0 2024-09-16T00:17:51.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.250+0000 7fda6affd640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fda540a32d0 0x7fda540a5790 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fda5c0058c0 tx=0x7fda5c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:51.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.253+0000 7fda70837640 1 -- 172.21.15.33:0/2095172400 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fda64014030 con 0x7fda6c102ef0 2024-09-16T00:17:51.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.466+0000 7fda71839640 1 -- 172.21.15.33:0/2095172400 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda6c198780 con 0x7fda6c102ef0 2024-09-16T00:17:51.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.467+0000 7fda70837640 1 -- 172.21.15.33:0/2095172400 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11903 (secure 0 0 0) 0x7fda6c198780 con 0x7fda6c102ef0 2024-09-16T00:17:51.468 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:17:51.469 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":26,"btime":"2024-09-16T00:17:28:789138+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:17:28.493434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:17:51.469 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 26 2024-09-16T00:17:51.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.469+0000 7fda71839640 1 -- 172.21.15.33:0/2095172400 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fda540a32d0 msgr2=0x7fda540a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:51.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.469+0000 7fda71839640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fda540a32d0 0x7fda540a5790 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fda5c0058c0 tx=0x7fda5c005850 comp rx=0 tx=0).stop 2024-09-16T00:17:51.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.469+0000 7fda71839640 1 -- 172.21.15.33:0/2095172400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fda6c102ef0 msgr2=0x7fda6c19bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:51.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.469+0000 7fda71839640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fda6c102ef0 0x7fda6c19bcf0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fda6400cce0 tx=0x7fda6400a620 comp rx=0 tx=0).stop 2024-09-16T00:17:51.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.469+0000 7fda71839640 1 -- 172.21.15.33:0/2095172400 shutdown_connections 2024-09-16T00:17:51.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.469+0000 7fda71839640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fda540a32d0 0x7fda540a5790 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:51.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.469+0000 7fda71839640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fda6c102ef0 0x7fda6c19bcf0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:51.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.469+0000 7fda71839640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda6c102570 0x7fda6c197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:51.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.469+0000 7fda71839640 1 --2- 172.21.15.33:0/2095172400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda6c100e20 0x7fda6c197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:51.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.469+0000 7fda71839640 1 -- 172.21.15.33:0/2095172400 >> 172.21.15.33:0/2095172400 conn(0x7fda6c0fc7d0 msgr2=0x7fda6c105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:51.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.469+0000 7fda71839640 1 -- 172.21.15.33:0/2095172400 shutdown_connections 2024-09-16T00:17:51.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:51.469+0000 7fda71839640 1 -- 172.21.15.33:0/2095172400 wait complete. 2024-09-16T00:17:52.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:51 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:17:52.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:51 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2095172400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:52.162 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-16T00:15:07.246355+0000', 'modified': '2024-09-16T00:17:28.493434+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24494, 'mds_1': 24493, 'mds_2': 24472}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.60:6835/1002409397', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6834', 'nonce': 1002409397}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 1002409397}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.60:6837/452012251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6836', 'nonce': 452012251}, {'type': 'v1', 'addr': '172.21.15.60:6837', 'nonce': 452012251}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.83:6835/2667354509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 2667354509}, {'type': 'v1', 'addr': '172.21.15.83:6835', 'nonce': 2667354509}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.83:6837/4244918448', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6836', 'nonce': 4244918448}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 4244918448}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.83:6839/3202868586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6838', 'nonce': 3202868586}, {'type': 'v1', 'addr': '172.21.15.83:6839', 'nonce': 3202868586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.60:6839/1222932045', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6838', 'nonce': 1222932045}, {'type': 'v1', 'addr': '172.21.15.60:6839', 'nonce': 1222932045}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24494, 'qdb_cluster': [24494, 24493, 24472]} 2024-09-16T00:17:52.162 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-16T00:17:52.162 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.c status 2024-09-16T00:17:52.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:51 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:17:52.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:51 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2095172400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:52.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:51 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:17:52.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2095172400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:17:52.463 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:52.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.942+0000 7f1084e31640 1 -- 172.21.15.33:0/3056219061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080100a80 msgr2=0x7f1080100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:52.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.942+0000 7f1084e31640 1 --2- 172.21.15.33:0/3056219061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080100a80 0x7f1080100ee0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1074009930 tx=0x7f107402a420 comp rx=0 tx=0).stop 2024-09-16T00:17:52.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.942+0000 7f1084e31640 1 -- 172.21.15.33:0/3056219061 shutdown_connections 2024-09-16T00:17:52.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.942+0000 7f1084e31640 1 --2- 172.21.15.33:0/3056219061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1080101420 0x7f108010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.942+0000 7f1084e31640 1 --2- 172.21.15.33:0/3056219061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080100a80 0x7f1080100ee0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.942+0000 7f1084e31640 1 --2- 172.21.15.33:0/3056219061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1080106a80 0x7f1080106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.942+0000 7f1084e31640 1 -- 172.21.15.33:0/3056219061 >> 172.21.15.33:0/3056219061 conn(0x7f10800fc7a0 msgr2=0x7f10800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:52.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.942+0000 7f1084e31640 1 -- 172.21.15.33:0/3056219061 shutdown_connections 2024-09-16T00:17:52.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.942+0000 7f1084e31640 1 -- 172.21.15.33:0/3056219061 wait complete. 2024-09-16T00:17:52.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.943+0000 7f1084e31640 1 Processor -- start 2024-09-16T00:17:52.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.943+0000 7f1084e31640 1 -- start start 2024-09-16T00:17:52.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.943+0000 7f1084e31640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080100a80 0x7f1080199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.943+0000 7f1084e31640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1080101420 0x7f1080199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.943+0000 7f1084e31640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1080106a80 0x7f10801a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.943+0000 7f1084e31640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1080110f20 con 0x7f1080100a80 2024-09-16T00:17:52.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.943+0000 7f1084e31640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1080110da0 con 0x7f1080101420 2024-09-16T00:17:52.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.943+0000 7f1084e31640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f10801110a0 con 0x7f1080106a80 2024-09-16T00:17:52.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.943+0000 7f107e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080100a80 0x7f1080199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.943+0000 7f107e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080100a80 0x7f1080199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33776/0 (socket says 172.21.15.33:33776) 2024-09-16T00:17:52.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.943+0000 7f107e59c640 1 -- 172.21.15.33:0/2091323439 learned_addr learned my addr 172.21.15.33:0/2091323439 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:52.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f107dd9b640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1080101420 0x7f1080199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f107e59c640 1 -- 172.21.15.33:0/2091323439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1080106a80 msgr2=0x7f10801a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f107ed9d640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1080106a80 0x7f10801a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f107e59c640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1080106a80 0x7f10801a44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f107e59c640 1 -- 172.21.15.33:0/2091323439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1080101420 msgr2=0x7f1080199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f107e59c640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1080101420 0x7f1080199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f107e59c640 1 -- 172.21.15.33:0/2091323439 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10801a4a10 con 0x7f1080100a80 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f107dd9b640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1080101420 0x7f1080199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f107e59c640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080100a80 0x7f1080199450 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f107000b4a0 tx=0x7f107000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f106f7fe640 1 -- 172.21.15.33:0/2091323439 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070014070 con 0x7f1080100a80 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f106f7fe640 1 -- 172.21.15.33:0/2091323439 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1070010af0 con 0x7f1080100a80 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f1084e31640 1 -- 172.21.15.33:0/2091323439 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10801a4be0 con 0x7f1080100a80 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f106f7fe640 1 -- 172.21.15.33:0/2091323439 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070017d10 con 0x7f1080100a80 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.944+0000 7f1084e31640 1 -- 172.21.15.33:0/2091323439 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10801a50f0 con 0x7f1080100a80 2024-09-16T00:17:52.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.947+0000 7f108720a740 1 Processor -- start 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.947+0000 7f108720a740 1 -- start start 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.947+0000 7f106f7fe640 1 -- 172.21.15.33:0/2091323439 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f1070017070 con 0x7f1080100a80 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f108720a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9d9d0 0x5651c6c9ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f108720a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f108720a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6caafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f108720a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5651c6c04090 con 0x5651c6c9e2f0 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f108720a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5651c6baba70 con 0x5651c6c9d9d0 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f108720a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5651c6c00050 con 0x5651c6ca8bb0 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f107e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f107e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33790/0 (socket says 172.21.15.33:33790) 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f107e59c640 1 -- 172.21.15.33:0/324752192 learned_addr learned my addr 172.21.15.33:0/324752192 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f107ed9d640 1 --2- 172.21.15.33:0/324752192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9d9d0 0x5651c6c9ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f107dd9b640 1 --2- 172.21.15.33:0/324752192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6caafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f107e59c640 1 -- 172.21.15.33:0/324752192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 msgr2=0x5651c6caafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f107e59c640 1 --2- 172.21.15.33:0/324752192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6caafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f107e59c640 1 -- 172.21.15.33:0/324752192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9d9d0 msgr2=0x5651c6c9ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f107e59c640 1 --2- 172.21.15.33:0/324752192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9d9d0 0x5651c6c9ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f107e59c640 1 -- 172.21.15.33:0/324752192 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651c6bb8130 con 0x5651c6c9e2f0 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f106f7fe640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10680a32d0 0x7f10680a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.948+0000 7f107e59c640 1 --2- 172.21.15.33:0/324752192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca8670 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f10700cc7e0 tx=0x7f10700d7a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58724aa90d3f712c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.949+0000 7f107ed9d640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10680a32d0 0x7f10680a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.949+0000 7f106f7fe640 1 -- 172.21.15.33:0/2091323439 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f10700cf230 con 0x7f1080100a80 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.949+0000 7f106cff9640 1 -- 172.21.15.33:0/324752192 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070097620 con 0x5651c6c9e2f0 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.949+0000 7f106cff9640 1 -- 172.21.15.33:0/324752192 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1070085e50 con 0x5651c6c9e2f0 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.949+0000 7f106cff9640 1 -- 172.21.15.33:0/324752192 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107009ace0 con 0x5651c6c9e2f0 2024-09-16T00:17:52.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.949+0000 7f108720a740 1 -- 172.21.15.33:0/324752192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9e2f0 msgr2=0x5651c6ca8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.949+0000 7f107ed9d640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10680a32d0 0x7f10680a5790 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1078007ed0 tx=0x7f10780075c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.949+0000 7f108720a740 1 --2- 172.21.15.33:0/324752192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca8670 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f10700cc7e0 tx=0x7f10700d7a10 comp rx=0 tx=0).stop 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 -- 172.21.15.33:0/324752192 shutdown_connections 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 --2- 172.21.15.33:0/324752192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6caafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 --2- 172.21.15.33:0/324752192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca8670 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 --2- 172.21.15.33:0/324752192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9d9d0 0x5651c6c9ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 -- 172.21.15.33:0/324752192 >> 172.21.15.33:0/324752192 conn(0x5651c6c99f60 msgr2=0x5651c6c9a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 -- 172.21.15.33:0/324752192 shutdown_connections 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 -- 172.21.15.33:0/324752192 wait complete. 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 -- start start 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 Processor -- start 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9d9d0 0x5651c6ca8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6ca8bb0 0x5651c6ca5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5651c6cab780 con 0x5651c6ca8bb0 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5651c6bbef20 con 0x5651c6c9d9d0 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5651c6cab900 con 0x5651c6c9e2f0 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f108720a740 10 client.0 _ll_register_callbacks cb 0x5651c6b88030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f107ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6ca8bb0 0x5651c6ca5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f107ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6ca8bb0 0x5651c6ca5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33792/0 (socket says 172.21.15.33:33792) 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.950+0000 7f107ed9d640 1 -- 172.21.15.33:0/292211027 learned_addr learned my addr 172.21.15.33:0/292211027 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f107e59c640 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9d9d0 0x5651c6ca8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f107dd9b640 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f107ed9d640 1 -- 172.21.15.33:0/292211027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6c9e2f0 msgr2=0x5651c6ca5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f107ed9d640 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f107ed9d640 1 -- 172.21.15.33:0/292211027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9d9d0 msgr2=0x5651c6ca8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:52.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f107ed9d640 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9d9d0 0x5651c6ca8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f107ed9d640 1 -- 172.21.15.33:0/292211027 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651c69752b0 con 0x5651c6ca8bb0 2024-09-16T00:17:52.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f107e59c640 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9d9d0 0x5651c6ca8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:52.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f107ed9d640 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6ca8bb0 0x5651c6ca5ca0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1078012e00 tx=0x7f1078004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632efffdb89665c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:52.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f1065ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:52.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f1065ffb640 1 -- 172.21.15.33:0/292211027 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078023020 con 0x5651c6ca8bb0 2024-09-16T00:17:52.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f108720a740 1 -- 172.21.15.33:0/292211027 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1070085e50 con 0x5651c6ca8bb0 2024-09-16T00:17:52.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f1065ffb640 1 -- 172.21.15.33:0/292211027 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f107801f440 con 0x5651c6ca8bb0 2024-09-16T00:17:52.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f1065ffb640 1 -- 172.21.15.33:0/292211027 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f107802e050 con 0x5651c6ca8bb0 2024-09-16T00:17:52.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f108720a740 10 client.15408 fetch_fsmap learned FSMap version 26 2024-09-16T00:17:52.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.951+0000 7f108720a740 1 -- 172.21.15.33:0/292211027 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5651c6cb70d0 con 0x5651c6ca8bb0 2024-09-16T00:17:52.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.952+0000 7f1065ffb640 1 -- 172.21.15.33:0/292211027 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1070085e50 con 0x5651c6ca8bb0 2024-09-16T00:17:52.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.952+0000 7f1065ffb640 1 -- 172.21.15.33:0/292211027 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f1078011040 con 0x5651c6ca8bb0 2024-09-16T00:17:52.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.952+0000 7f108720a740 10 client.15408 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:17:52.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.952+0000 7f108720a740 10 client.15408 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T00:17:52.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.952+0000 7f108720a740 20 client.15408 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:52.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.952+0000 7f108720a740 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5651c6cb9760 0x5651c6cbabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.952+0000 7f108720a740 4 client.15408 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:17:52.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.952+0000 7f108720a740 1 -- 172.21.15.33:0/292211027 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651c6c11600 con 0x5651c6cb9760 2024-09-16T00:17:52.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.952+0000 7f107e59c640 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5651c6cb9760 0x5651c6cbabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.952+0000 7f1065ffb640 10 client.15408 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:17:52.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.952+0000 7f107e59c640 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5651c6cb9760 0x5651c6cbabd0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=461dd1c786f908b0 server_cookie=59d36ea528be0f14 in_seq=0 out_seq=0 2024-09-16T00:17:52.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f1065ffb640 1 -- 172.21.15.33:0/292211027 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5651c6c11600 con 0x5651c6cb9760 2024-09-16T00:17:52.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f1065ffb640 10 client.15408 handle_command_reply: tid=0 2024-09-16T00:17:52.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f108720a740 1 client.15408 shutdown 2024-09-16T00:17:52.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f108720a740 1 -- 172.21.15.33:0/292211027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6ca8bb0 msgr2=0x5651c6ca5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:52.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f108720a740 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6ca8bb0 0x5651c6ca5ca0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1078012e00 tx=0x7f1078004a40 comp rx=0 tx=0).stop 2024-09-16T00:17:52.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f108720a740 1 -- 172.21.15.33:0/292211027 shutdown_connections 2024-09-16T00:17:52.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f108720a740 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5651c6cb9760 0x5651c6cbabd0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f108720a740 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6ca8bb0 0x5651c6ca5ca0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f108720a740 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f1065ffb640 0 client.15408 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:17:52.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f108720a740 1 --2- 172.21.15.33:0/292211027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9d9d0 0x5651c6ca8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f108720a740 1 -- 172.21.15.33:0/292211027 >> 172.21.15.33:0/292211027 conn(0x5651c6c99f60 msgr2=0x5651c6c9fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:52.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f108720a740 1 -- 172.21.15.33:0/292211027 shutdown_connections 2024-09-16T00:17:52.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.953+0000 7f108720a740 1 -- 172.21.15.33:0/292211027 wait complete. 2024-09-16T00:17:52.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.954+0000 7f108720a740 20 client.15408 trim_cache size 0 max 16384 2024-09-16T00:17:52.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.971+0000 7f108720a740 1 Processor -- start 2024-09-16T00:17:52.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.971+0000 7f108720a740 1 -- start start 2024-09-16T00:17:52.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f108720a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 0x5651c6ca4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f108720a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f108720a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6ca4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f108720a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5651c6cb8f10 con 0x5651c6c9d9d0 2024-09-16T00:17:52.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f108720a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5651c6cb95e0 con 0x5651c6c9e2f0 2024-09-16T00:17:52.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f108720a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5651c6cb9cb0 con 0x5651c6ca8bb0 2024-09-16T00:17:52.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f107e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 0x5651c6ca4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f107e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 0x5651c6ca4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33800/0 (socket says 172.21.15.33:33800) 2024-09-16T00:17:52.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f107e59c640 1 -- 172.21.15.33:0/3868171370 learned_addr learned my addr 172.21.15.33:0/3868171370 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:52.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f107ed9d640 1 --2- 172.21.15.33:0/3868171370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6ca4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f107e59c640 1 -- 172.21.15.33:0/3868171370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 msgr2=0x5651c6ca4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:52.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f107e59c640 1 --2- 172.21.15.33:0/3868171370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6ca4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f107dd9b640 1 --2- 172.21.15.33:0/3868171370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f107e59c640 1 -- 172.21.15.33:0/3868171370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9e2f0 msgr2=0x5651c6ca4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:52.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f107e59c640 1 --2- 172.21.15.33:0/3868171370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f107e59c640 1 -- 172.21.15.33:0/3868171370 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651c6c99a00 con 0x5651c6c9d9d0 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f107ed9d640 1 --2- 172.21.15.33:0/3868171370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6ca4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f107e59c640 1 --2- 172.21.15.33:0/3868171370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 0x5651c6ca4ca0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f107000b470 tx=0x7f10700047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27cf3b320d325e96 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f1067fff640 1 -- 172.21.15.33:0/3868171370 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107001e040 con 0x5651c6c9d9d0 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f1067fff640 1 -- 172.21.15.33:0/3868171370 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10700d2d00 con 0x5651c6c9d9d0 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.972+0000 7f1067fff640 1 -- 172.21.15.33:0/3868171370 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107009aec0 con 0x5651c6c9d9d0 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.973+0000 7f108720a740 1 -- 172.21.15.33:0/3868171370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 msgr2=0x5651c6ca4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.973+0000 7f108720a740 1 --2- 172.21.15.33:0/3868171370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 0x5651c6ca4ca0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f107000b470 tx=0x7f10700047d0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.973+0000 7f108720a740 1 -- 172.21.15.33:0/3868171370 shutdown_connections 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.973+0000 7f108720a740 1 --2- 172.21.15.33:0/3868171370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6ca4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.973+0000 7f108720a740 1 --2- 172.21.15.33:0/3868171370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9e2f0 0x5651c6ca4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.973+0000 7f108720a740 1 --2- 172.21.15.33:0/3868171370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 0x5651c6ca4ca0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.973+0000 7f108720a740 1 -- 172.21.15.33:0/3868171370 >> 172.21.15.33:0/3868171370 conn(0x5651c6c99f60 msgr2=0x5651c6ca5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:52.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.973+0000 7f108720a740 1 -- 172.21.15.33:0/3868171370 shutdown_connections 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.973+0000 7f108720a740 1 -- 172.21.15.33:0/3868171370 wait complete. 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.973+0000 7f108720a740 1 -- start start 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.973+0000 7f108720a740 1 Processor -- start 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f108720a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 0x5651c6cbc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f108720a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9e2f0 0x5651c6cbc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f108720a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6d21e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f108720a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5651c6cbaa50 con 0x5651c6c9d9d0 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f108720a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5651c6cba380 con 0x5651c6c9e2f0 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f108720a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5651c6cbb120 con 0x5651c6ca8bb0 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f107e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 0x5651c6cbc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f108720a740 10 client.0 _ll_register_callbacks cb 0x7f1078007b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f107e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 0x5651c6cbc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33816/0 (socket says 172.21.15.33:33816) 2024-09-16T00:17:52.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f107e59c640 1 -- 172.21.15.33:0/1820730643 learned_addr learned my addr 172.21.15.33:0/1820730643 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f107ed9d640 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6d21e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f107dd9b640 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9e2f0 0x5651c6cbc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f107e59c640 1 -- 172.21.15.33:0/1820730643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 msgr2=0x5651c6d21e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f107e59c640 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6d21e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f107e59c640 1 -- 172.21.15.33:0/1820730643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9e2f0 msgr2=0x5651c6cbc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f107e59c640 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9e2f0 0x5651c6cbc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f107e59c640 1 -- 172.21.15.33:0/1820730643 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651c6d223a0 con 0x5651c6c9d9d0 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f107e59c640 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 0x5651c6cbc0a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1070017770 tx=0x7f107009aec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5239248a8238ca79 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f1065ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.974+0000 7f1065ffb640 1 -- 172.21.15.33:0/1820730643 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107001e070 con 0x5651c6c9d9d0 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f1065ffb640 1 -- 172.21.15.33:0/1820730643 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1070099680 con 0x5651c6c9d9d0 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f108720a740 1 -- 172.21.15.33:0/1820730643 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10700d2d00 con 0x5651c6c9d9d0 2024-09-16T00:17:52.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f1065ffb640 1 -- 172.21.15.33:0/1820730643 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f107009b150 con 0x5651c6c9d9d0 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f108720a740 10 client.15414 fetch_fsmap learned FSMap version 26 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f108720a740 1 -- 172.21.15.33:0/1820730643 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5651c6d23460 con 0x5651c6c9d9d0 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f1065ffb640 1 -- 172.21.15.33:0/1820730643 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10700d2d00 con 0x5651c6c9d9d0 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f1065ffb640 1 -- 172.21.15.33:0/1820730643 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f10700cc8f0 con 0x5651c6c9d9d0 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f108720a740 10 client.15414 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f108720a740 10 client.15414 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f108720a740 20 client.15414 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f108720a740 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5651c6d27940 0x5651c6d26650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f108720a740 4 client.15414 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f108720a740 1 -- 172.21.15.33:0/1820730643 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "status"}) -- 0x5651c6c0bb10 con 0x5651c6d27940 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.975+0000 7f107dd9b640 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5651c6d27940 0x5651c6d26650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f107dd9b640 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5651c6d27940 0x5651c6d26650 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=41fad5176d8da7ec server_cookie=1ea9ed3d51856d62 in_seq=0 out_seq=0 2024-09-16T00:17:52.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f1065ffb640 10 client.15414 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:17:52.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f1065ffb640 1 -- 172.21.15.33:0/1820730643 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5651c6c0bb10 con 0x5651c6d27940 2024-09-16T00:17:52.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f1065ffb640 10 client.15414 handle_command_reply: tid=0 2024-09-16T00:17:52.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f108720a740 1 client.15414 shutdown 2024-09-16T00:17:52.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f108720a740 1 -- 172.21.15.33:0/1820730643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 msgr2=0x5651c6cbc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:52.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f108720a740 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 0x5651c6cbc0a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1070017770 tx=0x7f107009aec0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f108720a740 1 -- 172.21.15.33:0/1820730643 shutdown_connections 2024-09-16T00:17:52.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f108720a740 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5651c6d27940 0x5651c6d26650 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f1065ffb640 0 client.15414 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:17:52.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f108720a740 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651c6ca8bb0 0x5651c6d21e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f108720a740 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651c6c9e2f0 0x5651c6cbc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f108720a740 1 --2- 172.21.15.33:0/1820730643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651c6c9d9d0 0x5651c6cbc0a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f108720a740 1 -- 172.21.15.33:0/1820730643 >> 172.21.15.33:0/1820730643 conn(0x5651c6c99f60 msgr2=0x5651c6ca3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:52.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f108720a740 1 -- 172.21.15.33:0/1820730643 shutdown_connections 2024-09-16T00:17:52.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f108720a740 1 -- 172.21.15.33:0/1820730643 wait complete. 2024-09-16T00:17:52.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.976+0000 7f108720a740 20 client.15414 trim_cache size 0 max 16384 2024-09-16T00:17:52.993 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:17:52.993 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:17:52.993 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 2, 2024-09-16T00:17:52.993 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24472, 2024-09-16T00:17:52.993 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-09-16T00:17:52.993 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-09-16T00:17:52.993 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-09-16T00:17:52.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 123.808779589, 2024-09-16T00:17:52.994 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 26, 2024-09-16T00:17:52.994 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 90, 2024-09-16T00:17:52.994 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 90, 2024-09-16T00:17:52.994 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 190.99645926299999 2024-09-16T00:17:52.994 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:17:52.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.977+0000 7f1084e31640 1 -- 172.21.15.33:0/2091323439 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10680a32d0 msgr2=0x7f10680a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:52.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.977+0000 7f1084e31640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10680a32d0 0x7f10680a5790 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1078007ed0 tx=0x7f10780075c0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.977+0000 7f1084e31640 1 -- 172.21.15.33:0/2091323439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080100a80 msgr2=0x7f1080199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:52.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.977+0000 7f1084e31640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080100a80 0x7f1080199450 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f107000b4a0 tx=0x7f107000b960 comp rx=0 tx=0).stop 2024-09-16T00:17:52.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.977+0000 7f1084e31640 1 -- 172.21.15.33:0/2091323439 shutdown_connections 2024-09-16T00:17:52.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.977+0000 7f1084e31640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10680a32d0 0x7f10680a5790 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.977+0000 7f1084e31640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1080106a80 0x7f10801a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.977+0000 7f1084e31640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1080101420 0x7f1080199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.977+0000 7f1084e31640 1 --2- 172.21.15.33:0/2091323439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080100a80 0x7f1080199450 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:52.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.977+0000 7f1084e31640 1 -- 172.21.15.33:0/2091323439 >> 172.21.15.33:0/2091323439 conn(0x7f10800fc7a0 msgr2=0x7f10800fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:52.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.977+0000 7f1084e31640 1 -- 172.21.15.33:0/2091323439 shutdown_connections 2024-09-16T00:17:52.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:52.977+0000 7f1084e31640 1 -- 172.21.15.33:0/2091323439 wait complete. 2024-09-16T00:17:53.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:52 smithi083 ceph-mon[35242]: pgmap v372: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-16T00:17:53.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:52 smithi060 ceph-mon[35185]: pgmap v372: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-16T00:17:53.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:52 smithi033 ceph-mon[31243]: pgmap v372: 129 pgs: 129 active+clean; 652 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-16T00:17:53.821 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-16T00:17:54.080 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:54.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.510+0000 7f6c21345640 1 -- 172.21.15.33:0/2354227071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c1c102d30 msgr2=0x7f6c1c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:54.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.510+0000 7f6c21345640 1 --2- 172.21.15.33:0/2354227071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c1c102d30 0x7f6c1c103110 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009cd0 tx=0x7f6c0c02a8e0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.510+0000 7f6c21345640 1 -- 172.21.15.33:0/2354227071 shutdown_connections 2024-09-16T00:17:54.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.510+0000 7f6c21345640 1 --2- 172.21.15.33:0/2354227071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c1c1082a0 0x7f6c1c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.510+0000 7f6c21345640 1 --2- 172.21.15.33:0/2354227071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c1036e0 0x7f6c1c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.510+0000 7f6c21345640 1 --2- 172.21.15.33:0/2354227071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c1c102d30 0x7f6c1c103110 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.510+0000 7f6c21345640 1 -- 172.21.15.33:0/2354227071 >> 172.21.15.33:0/2354227071 conn(0x7f6c1c0fc7d0 msgr2=0x7f6c1c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:54.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.510+0000 7f6c21345640 1 -- 172.21.15.33:0/2354227071 shutdown_connections 2024-09-16T00:17:54.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.510+0000 7f6c21345640 1 -- 172.21.15.33:0/2354227071 wait complete. 2024-09-16T00:17:54.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.510+0000 7f6c21345640 1 Processor -- start 2024-09-16T00:17:54.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c21345640 1 -- start start 2024-09-16T00:17:54.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c21345640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c1c102d30 0x7f6c1c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c21345640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c1036e0 0x7f6c1c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c21345640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c1c1082a0 0x7f6c1c19ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c21345640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c1120a0 con 0x7f6c1c1082a0 2024-09-16T00:17:54.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c21345640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c111f20 con 0x7f6c1c102d30 2024-09-16T00:17:54.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c21345640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c112220 con 0x7f6c1c1036e0 2024-09-16T00:17:54.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c1b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c1c1082a0 0x7f6c1c19ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c1b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c1c1082a0 0x7f6c1c19ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33834/0 (socket says 172.21.15.33:33834) 2024-09-16T00:17:54.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c1b7fe640 1 -- 172.21.15.33:0/4288896701 learned_addr learned my addr 172.21.15.33:0/4288896701 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:54.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c1036e0 0x7f6c1c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c1affd640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c1c102d30 0x7f6c1c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c1b7fe640 1 -- 172.21.15.33:0/4288896701 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c1036e0 msgr2=0x7f6c1c198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:54.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c1b7fe640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c1036e0 0x7f6c1c198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c1b7fe640 1 -- 172.21.15.33:0/4288896701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c1c102d30 msgr2=0x7f6c1c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:54.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c1b7fe640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c1c102d30 0x7f6c1c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.511+0000 7f6c1b7fe640 1 -- 172.21.15.33:0/4288896701 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1c19d510 con 0x7f6c1c1082a0 2024-09-16T00:17:54.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.512+0000 7f6c1b7fe640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c1c1082a0 0x7f6c1c19ce30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400c970 tx=0x7f6c1400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:54.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.512+0000 7f6c03fff640 1 -- 172.21.15.33:0/4288896701 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c14007bb0 con 0x7f6c1c1082a0 2024-09-16T00:17:54.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.512+0000 7f6c21345640 1 -- 172.21.15.33:0/4288896701 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1c19d800 con 0x7f6c1c1082a0 2024-09-16T00:17:54.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.512+0000 7f6c03fff640 1 -- 172.21.15.33:0/4288896701 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c14007d50 con 0x7f6c1c1082a0 2024-09-16T00:17:54.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.512+0000 7f6c03fff640 1 -- 172.21.15.33:0/4288896701 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c14018800 con 0x7f6c1c1082a0 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.512+0000 7f6c21345640 1 -- 172.21.15.33:0/4288896701 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c1c1a50e0 con 0x7f6c1c1082a0 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.514+0000 7f6c2371e740 1 Processor -- start 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.514+0000 7f6c03fff640 1 -- 172.21.15.33:0/4288896701 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f6c1400fa50 con 0x7f6c1c1082a0 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.514+0000 7f6c2371e740 1 -- start start 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c2371e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 0x5556282fcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c2371e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fd2f0 0x555628307670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c2371e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 0x555628309fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c2371e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555628263090 con 0x5556282fc9d0 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c2371e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55562820aa70 con 0x555628307bb0 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c2371e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55562825f050 con 0x5556282fd2f0 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c1a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 0x5556282fcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c1a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 0x5556282fcdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33838/0 (socket says 172.21.15.33:33838) 2024-09-16T00:17:54.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c1a7fc640 1 -- 172.21.15.33:0/1838711411 learned_addr learned my addr 172.21.15.33:0/1838711411 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:54.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c1affd640 1 --2- 172.21.15.33:0/1838711411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fd2f0 0x555628307670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/1838711411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 0x555628309fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c03fff640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6c040a32d0 0x7f6c040a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c1a7fc640 1 -- 172.21.15.33:0/1838711411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fd2f0 msgr2=0x555628307670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:54.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/1838711411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fd2f0 0x555628307670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c1a7fc640 1 -- 172.21.15.33:0/1838711411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 msgr2=0x555628309fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:54.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/1838711411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 0x555628309fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c1a7fc640 1 -- 172.21.15.33:0/1838711411 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555628217130 con 0x5556282fc9d0 2024-09-16T00:17:54.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c1b7fe640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6c040a32d0 0x7f6c040a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c03fff640 1 -- 172.21.15.33:0/4288896701 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6c140cfe70 con 0x7f6c1c1082a0 2024-09-16T00:17:54.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.515+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/1838711411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 0x5556282fcdb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6c100058a0 tx=0x7f6c1000b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9375de8af0f5d7fb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:54.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.516+0000 7f6c017fa640 1 -- 172.21.15.33:0/1838711411 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10004030 con 0x5556282fc9d0 2024-09-16T00:17:54.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.516+0000 7f6c017fa640 1 -- 172.21.15.33:0/1838711411 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c10005500 con 0x5556282fc9d0 2024-09-16T00:17:54.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.516+0000 7f6c017fa640 1 -- 172.21.15.33:0/1838711411 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10004c60 con 0x5556282fc9d0 2024-09-16T00:17:54.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.516+0000 7f6c2371e740 1 -- 172.21.15.33:0/1838711411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 msgr2=0x5556282fcdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:54.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.516+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1838711411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 0x5556282fcdb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6c100058a0 tx=0x7f6c1000b820 comp rx=0 tx=0).stop 2024-09-16T00:17:54.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.516+0000 7f6c1b7fe640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6c040a32d0 0x7f6c040a5790 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6c14098310 tx=0x7f6c140bc040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:54.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 -- 172.21.15.33:0/1838711411 shutdown_connections 2024-09-16T00:17:54.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1838711411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 0x555628309fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1838711411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fd2f0 0x555628307670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1838711411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 0x5556282fcdb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 -- 172.21.15.33:0/1838711411 >> 172.21.15.33:0/1838711411 conn(0x5556282f8f60 msgr2=0x5556282f9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 -- 172.21.15.33:0/1838711411 shutdown_connections 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 -- 172.21.15.33:0/1838711411 wait complete. 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 -- start start 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 Processor -- start 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fc9d0 0x555628307450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 0x5556283047a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 0x555628304ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55562830a780 con 0x5556282fd2f0 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55562821df20 con 0x555628307bb0 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55562830a900 con 0x5556282fc9d0 2024-09-16T00:17:54.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c2371e740 10 client.0 _ll_register_callbacks cb 0x5556281e7030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:54.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c1affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 0x5556283047a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c1affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 0x5556283047a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33850/0 (socket says 172.21.15.33:33850) 2024-09-16T00:17:54.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.517+0000 7f6c1affd640 1 -- 172.21.15.33:0/1810547568 learned_addr learned my addr 172.21.15.33:0/1810547568 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:54.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.518+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fc9d0 0x555628307450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.518+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 0x555628304ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.518+0000 7f6c1affd640 1 -- 172.21.15.33:0/1810547568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fc9d0 msgr2=0x555628307450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:54.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.518+0000 7f6c1affd640 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fc9d0 0x555628307450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.518+0000 7f6c1affd640 1 -- 172.21.15.33:0/1810547568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 msgr2=0x555628304ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:54.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.518+0000 7f6c1affd640 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 0x555628304ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.518+0000 7f6c1affd640 1 -- 172.21.15.33:0/1810547568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555627fd42b0 con 0x5556282fd2f0 2024-09-16T00:17:54.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.518+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fc9d0 0x555628307450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:17:54.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.518+0000 7f6c1affd640 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 0x5556283047a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009cd0 tx=0x7f6c0c031c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3debc86a0db7311a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:54.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.518+0000 7f6bfeff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:54.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.518+0000 7f6bfeff5640 1 -- 172.21.15.33:0/1810547568 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c049060 con 0x5556282fd2f0 2024-09-16T00:17:54.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.519+0000 7f6c2371e740 1 -- 172.21.15.33:0/1810547568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c10005500 con 0x5556282fd2f0 2024-09-16T00:17:54.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.519+0000 7f6bfeff5640 1 -- 172.21.15.33:0/1810547568 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6c0c003b80 con 0x5556282fd2f0 2024-09-16T00:17:54.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.519+0000 7f6bfeff5640 1 -- 172.21.15.33:0/1810547568 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6c0c031d20 con 0x5556282fd2f0 2024-09-16T00:17:54.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.519+0000 7f6c2371e740 10 client.15426 fetch_fsmap learned FSMap version 26 2024-09-16T00:17:54.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.519+0000 7f6c2371e740 1 -- 172.21.15.33:0/1810547568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5556283160f0 con 0x5556282fd2f0 2024-09-16T00:17:54.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.520+0000 7f6bfeff5640 1 -- 172.21.15.33:0/1810547568 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c10005500 con 0x5556282fd2f0 2024-09-16T00:17:54.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.520+0000 7f6bfeff5640 1 -- 172.21.15.33:0/1810547568 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f6c0c043070 con 0x5556282fd2f0 2024-09-16T00:17:54.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.520+0000 7f6c2371e740 10 client.15426 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:17:54.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.520+0000 7f6c2371e740 10 client.15426 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-16T00:17:54.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.520+0000 7f6c2371e740 20 client.15426 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:54.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.520+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5556283186f0 0x555628319b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.520+0000 7f6c2371e740 4 client.15426 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:17:54.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.520+0000 7f6c2371e740 1 -- 172.21.15.33:0/1810547568 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555628270600 con 0x5556283186f0 2024-09-16T00:17:54.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.520+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5556283186f0 0x555628319b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.521+0000 7f6bfeff5640 10 client.15426 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:17:54.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.521+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5556283186f0 0x555628319b60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67bae5579f63c470 server_cookie=950eee7f3a5e93cc in_seq=0 out_seq=0 2024-09-16T00:17:54.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.521+0000 7f6bfeff5640 1 -- 172.21.15.33:0/1810547568 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555628270600 con 0x5556283186f0 2024-09-16T00:17:54.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6bfeff5640 10 client.15426 handle_command_reply: tid=0 2024-09-16T00:17:54.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6c2371e740 1 client.15426 shutdown 2024-09-16T00:17:54.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6c2371e740 1 -- 172.21.15.33:0/1810547568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 msgr2=0x5556283047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:54.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 0x5556283047a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009cd0 tx=0x7f6c0c031c30 comp rx=0 tx=0).stop 2024-09-16T00:17:54.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6c2371e740 1 -- 172.21.15.33:0/1810547568 shutdown_connections 2024-09-16T00:17:54.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5556283186f0 0x555628319b60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 0x555628304ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 0x5556283047a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6bfeff5640 0 client.15426 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:17:54.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1810547568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fc9d0 0x555628307450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6c2371e740 1 -- 172.21.15.33:0/1810547568 >> 172.21.15.33:0/1810547568 conn(0x5556282f8f60 msgr2=0x5556282fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:54.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6c2371e740 1 -- 172.21.15.33:0/1810547568 shutdown_connections 2024-09-16T00:17:54.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6c2371e740 1 -- 172.21.15.33:0/1810547568 wait complete. 2024-09-16T00:17:54.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.522+0000 7f6c2371e740 20 client.15426 trim_cache size 0 max 16384 2024-09-16T00:17:54.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.540+0000 7f6c2371e740 1 Processor -- start 2024-09-16T00:17:54.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.540+0000 7f6c2371e740 1 -- start start 2024-09-16T00:17:54.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.540+0000 7f6c2371e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 0x555628319b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.540+0000 7f6c2371e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fd2f0 0x5556282f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.540+0000 7f6c2371e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 0x5556282ffd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.540+0000 7f6c2371e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555628317ea0 con 0x5556282fc9d0 2024-09-16T00:17:54.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.540+0000 7f6c2371e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555628318570 con 0x555628307bb0 2024-09-16T00:17:54.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.540+0000 7f6c2371e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555628318c40 con 0x5556282fd2f0 2024-09-16T00:17:54.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c1a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 0x555628319b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c1a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 0x555628319b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33864/0 (socket says 172.21.15.33:33864) 2024-09-16T00:17:54.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c1a7fc640 1 -- 172.21.15.33:0/1958549351 learned_addr learned my addr 172.21.15.33:0/1958549351 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:54.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c1affd640 1 --2- 172.21.15.33:0/1958549351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fd2f0 0x5556282f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c1a7fc640 1 -- 172.21.15.33:0/1958549351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fd2f0 msgr2=0x5556282f3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:54.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/1958549351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fd2f0 0x5556282f3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c1a7fc640 1 -- 172.21.15.33:0/1958549351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 msgr2=0x5556282ffd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:54.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/1958549351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 0x5556282ffd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c1a7fc640 1 -- 172.21.15.33:0/1958549351 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555628303750 con 0x5556282fc9d0 2024-09-16T00:17:54.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/1958549351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 0x555628319b60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6c100058a0 tx=0x7f6c1000b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24428a98885d55 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:54.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c00ff9640 1 -- 172.21.15.33:0/1958549351 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1002c390 con 0x5556282fc9d0 2024-09-16T00:17:54.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c00ff9640 1 -- 172.21.15.33:0/1958549351 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c1002ca10 con 0x5556282fc9d0 2024-09-16T00:17:54.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.541+0000 7f6c00ff9640 1 -- 172.21.15.33:0/1958549351 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10010680 con 0x5556282fc9d0 2024-09-16T00:17:54.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.542+0000 7f6c2371e740 1 -- 172.21.15.33:0/1958549351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 msgr2=0x555628319b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:54.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.542+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1958549351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 0x555628319b60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6c100058a0 tx=0x7f6c1000b820 comp rx=0 tx=0).stop 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.542+0000 7f6c2371e740 1 -- 172.21.15.33:0/1958549351 shutdown_connections 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.542+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1958549351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555628307bb0 0x5556282ffd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.542+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1958549351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556282fd2f0 0x5556282f3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.542+0000 7f6c2371e740 1 --2- 172.21.15.33:0/1958549351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fc9d0 0x555628319b60 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.542+0000 7f6c2371e740 1 -- 172.21.15.33:0/1958549351 >> 172.21.15.33:0/1958549351 conn(0x5556282f8f60 msgr2=0x5556282ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.542+0000 7f6c2371e740 1 -- 172.21.15.33:0/1958549351 shutdown_connections 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.542+0000 7f6c2371e740 1 -- 172.21.15.33:0/1958549351 wait complete. 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.542+0000 7f6c2371e740 1 -- start start 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.542+0000 7f6c2371e740 1 Processor -- start 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c2371e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5556282fc9d0 0x555628322110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c2371e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 0x555628322650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c2371e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555628307bb0 0x55562831b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c2371e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5556283199e0 con 0x5556282fd2f0 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c2371e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555628319310 con 0x5556282fc9d0 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c2371e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55562831a0b0 con 0x555628307bb0 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c2371e740 10 client.0 _ll_register_callbacks cb 0x7f6c0c0318c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c1affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 0x555628322650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c1affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 0x555628322650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33880/0 (socket says 172.21.15.33:33880) 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c1affd640 1 -- 172.21.15.33:0/2379392497 learned_addr learned my addr 172.21.15.33:0/2379392497 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555628307bb0 0x55562831b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5556282fc9d0 0x555628322110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c1affd640 1 -- 172.21.15.33:0/2379392497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555628307bb0 msgr2=0x55562831b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c1affd640 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555628307bb0 0x55562831b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c1affd640 1 -- 172.21.15.33:0/2379392497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5556282fc9d0 msgr2=0x555628322110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c1affd640 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5556282fc9d0 0x555628322110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c1affd640 1 -- 172.21.15.33:0/2379392497 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556283077c0 con 0x5556282fd2f0 2024-09-16T00:17:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5556282fc9d0 0x555628322110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6c1affd640 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 0x555628322650 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c02ae00 tx=0x7f6c0c030e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af403b6486fe3f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.543+0000 7f6bfeff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6bfeff5640 1 -- 172.21.15.33:0/2379392497 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c049040 con 0x5556282fd2f0 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6bfeff5640 1 -- 172.21.15.33:0/2379392497 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6c0c02e020 con 0x5556282fd2f0 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6c2371e740 1 -- 172.21.15.33:0/2379392497 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c1002ca10 con 0x5556282fd2f0 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6bfeff5640 1 -- 172.21.15.33:0/2379392497 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6c0c054430 con 0x5556282fd2f0 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6c2371e740 10 client.15432 fetch_fsmap learned FSMap version 26 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6c2371e740 1 -- 172.21.15.33:0/2379392497 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555628320600 con 0x5556282fd2f0 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6bfeff5640 1 -- 172.21.15.33:0/2379392497 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c1002ca10 con 0x5556282fd2f0 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6bfeff5640 1 -- 172.21.15.33:0/2379392497 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f6c0c043040 con 0x5556282fd2f0 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6c2371e740 10 client.15432 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6c2371e740 10 client.15432 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-16T00:17:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6c2371e740 20 client.15432 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:54.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6c2371e740 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x555628387380 0x555628386090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:54.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6c2371e740 4 client.15432 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T00:17:54.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.544+0000 7f6c2371e740 1 -- 172.21.15.33:0/2379392497 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "status"}) -- 0x55562826ab10 con 0x555628387380 2024-09-16T00:17:54.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.545+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x555628387380 0x555628386090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:54.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.545+0000 7f6c1a7fc640 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x555628387380 0x555628386090 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ecea31bbd329597f server_cookie=b22e488ab0829b7e in_seq=0 out_seq=0 2024-09-16T00:17:54.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.545+0000 7f6bfeff5640 10 client.15432 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:17:54.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6bfeff5640 1 -- 172.21.15.33:0/2379392497 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55562826ab10 con 0x555628387380 2024-09-16T00:17:54.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6bfeff5640 10 client.15432 handle_command_reply: tid=0 2024-09-16T00:17:54.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6c2371e740 1 client.15432 shutdown 2024-09-16T00:17:54.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6c2371e740 1 -- 172.21.15.33:0/2379392497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 msgr2=0x555628322650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:54.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6c2371e740 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 0x555628322650 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c02ae00 tx=0x7f6c0c030e00 comp rx=0 tx=0).stop 2024-09-16T00:17:54.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6c2371e740 1 -- 172.21.15.33:0/2379392497 shutdown_connections 2024-09-16T00:17:54.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6c2371e740 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555628307bb0 0x55562831b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6c2371e740 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556282fd2f0 0x555628322650 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6c2371e740 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x555628387380 0x555628386090 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6c2371e740 1 --2- 172.21.15.33:0/2379392497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5556282fc9d0 0x555628322110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6bfeff5640 0 client.15432 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:17:54.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6c2371e740 1 -- 172.21.15.33:0/2379392497 >> 172.21.15.33:0/2379392497 conn(0x5556282f8f60 msgr2=0x555628306630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:54.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6c2371e740 1 -- 172.21.15.33:0/2379392497 shutdown_connections 2024-09-16T00:17:54.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6c2371e740 1 -- 172.21.15.33:0/2379392497 wait complete. 2024-09-16T00:17:54.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.546+0000 7f6c2371e740 20 client.15432 trim_cache size 0 max 16384 2024-09-16T00:17:54.554 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:17:54.555 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:17:54.555 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 1, 2024-09-16T00:17:54.555 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24493, 2024-09-16T00:17:54.555 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-09-16T00:17:54.555 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-09-16T00:17:54.555 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-09-16T00:17:54.555 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 126.383161094, 2024-09-16T00:17:54.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 26, 2024-09-16T00:17:54.555 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 90, 2024-09-16T00:17:54.555 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 90, 2024-09-16T00:17:54.555 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 187.24964929399999 2024-09-16T00:17:54.555 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:17:54.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.547+0000 7f6c21345640 1 -- 172.21.15.33:0/4288896701 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6c040a32d0 msgr2=0x7f6c040a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:54.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.547+0000 7f6c21345640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6c040a32d0 0x7f6c040a5790 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6c14098310 tx=0x7f6c140bc040 comp rx=0 tx=0).stop 2024-09-16T00:17:54.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.547+0000 7f6c21345640 1 -- 172.21.15.33:0/4288896701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c1c1082a0 msgr2=0x7f6c1c19ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:54.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.547+0000 7f6c21345640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c1c1082a0 0x7f6c1c19ce30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400c970 tx=0x7f6c1400ce30 comp rx=0 tx=0).stop 2024-09-16T00:17:54.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.547+0000 7f6c21345640 1 -- 172.21.15.33:0/4288896701 shutdown_connections 2024-09-16T00:17:54.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.547+0000 7f6c21345640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6c040a32d0 0x7f6c040a5790 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.547+0000 7f6c21345640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c1c1082a0 0x7f6c1c19ce30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.547+0000 7f6c21345640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c1036e0 0x7f6c1c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.547+0000 7f6c21345640 1 --2- 172.21.15.33:0/4288896701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c1c102d30 0x7f6c1c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:54.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.547+0000 7f6c21345640 1 -- 172.21.15.33:0/4288896701 >> 172.21.15.33:0/4288896701 conn(0x7f6c1c0fc7d0 msgr2=0x7f6c1c105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:54.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.547+0000 7f6c21345640 1 -- 172.21.15.33:0/4288896701 shutdown_connections 2024-09-16T00:17:54.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:54.547+0000 7f6c21345640 1 -- 172.21.15.33:0/4288896701 wait complete. 2024-09-16T00:17:55.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:54 smithi083 ceph-mon[35242]: pgmap v373: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-16T00:17:55.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:54 smithi060 ceph-mon[35185]: pgmap v373: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-16T00:17:55.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:54 smithi033 ceph-mon[31243]: pgmap v373: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-16T00:17:55.417 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-16T00:17:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:17:56.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.188+0000 7fbae4b3a640 1 -- 172.21.15.33:0/2032879247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbae0100e20 msgr2=0x7fbae0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:56.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.188+0000 7fbae4b3a640 1 --2- 172.21.15.33:0/2032879247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbae0100e20 0x7fbae0101200 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009900 tx=0x7fbad002a480 comp rx=0 tx=0).stop 2024-09-16T00:17:56.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.188+0000 7fbae4b3a640 1 -- 172.21.15.33:0/2032879247 shutdown_connections 2024-09-16T00:17:56.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.188+0000 7fbae4b3a640 1 --2- 172.21.15.33:0/2032879247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbae0102ef0 0x7fbae010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.188+0000 7fbae4b3a640 1 --2- 172.21.15.33:0/2032879247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbae0102570 0x7fbae01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.188+0000 7fbae4b3a640 1 --2- 172.21.15.33:0/2032879247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbae0100e20 0x7fbae0101200 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.188+0000 7fbae4b3a640 1 -- 172.21.15.33:0/2032879247 >> 172.21.15.33:0/2032879247 conn(0x7fbae00fc7d0 msgr2=0x7fbae00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:56.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.188+0000 7fbae4b3a640 1 -- 172.21.15.33:0/2032879247 shutdown_connections 2024-09-16T00:17:56.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.188+0000 7fbae4b3a640 1 -- 172.21.15.33:0/2032879247 wait complete. 2024-09-16T00:17:56.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.189+0000 7fbae4b3a640 1 Processor -- start 2024-09-16T00:17:56.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.189+0000 7fbae4b3a640 1 -- start start 2024-09-16T00:17:56.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.189+0000 7fbae4b3a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbae0100e20 0x7fbae01973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.189+0000 7fbae4b3a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbae0102570 0x7fbae0197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.189+0000 7fbae4b3a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbae0102ef0 0x7fbae019bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.189+0000 7fbae4b3a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbae0110f80 con 0x7fbae0102ef0 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.189+0000 7fbae4b3a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbae0110e00 con 0x7fbae0100e20 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.189+0000 7fbae4b3a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbae0111100 con 0x7fbae0102570 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.189+0000 7fbaded9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbae0102ef0 0x7fbae019bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.189+0000 7fbaded9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbae0102ef0 0x7fbae019bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33886/0 (socket says 172.21.15.33:33886) 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.189+0000 7fbaded9d640 1 -- 172.21.15.33:0/4084129271 learned_addr learned my addr 172.21.15.33:0/4084129271 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbad7fff640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbae0102570 0x7fbae0197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbade59c640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbae0100e20 0x7fbae01973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbaded9d640 1 -- 172.21.15.33:0/4084129271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbae0102570 msgr2=0x7fbae0197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbaded9d640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbae0102570 0x7fbae0197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbaded9d640 1 -- 172.21.15.33:0/4084129271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbae0100e20 msgr2=0x7fbae01973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbaded9d640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbae0100e20 0x7fbae01973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbaded9d640 1 -- 172.21.15.33:0/4084129271 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae019c390 con 0x7fbae0102ef0 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbaded9d640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbae0102ef0 0x7fbae019bcb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbad800cad0 tx=0x7fbad800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbad77fe640 1 -- 172.21.15.33:0/4084129271 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad800ae20 con 0x7fbae0102ef0 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbad77fe640 1 -- 172.21.15.33:0/4084129271 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbad8005020 con 0x7fbae0102ef0 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbae4b3a640 1 -- 172.21.15.33:0/4084129271 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae019c680 con 0x7fbae0102ef0 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbad77fe640 1 -- 172.21.15.33:0/4084129271 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad800f6f0 con 0x7fbae0102ef0 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.190+0000 7fbae4b3a640 1 -- 172.21.15.33:0/4084129271 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae01a3f60 con 0x7fbae0102ef0 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.192+0000 7fbad77fe640 1 -- 172.21.15.33:0/4084129271 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fbad80051d0 con 0x7fbae0102ef0 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.193+0000 7fbae6f13740 1 Processor -- start 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.193+0000 7fbae6f13740 1 -- start start 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.193+0000 7fbae6f13740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c59d0 0x557d5d9c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.193+0000 7fbae6f13740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9c62f0 0x557d5d9d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.193+0000 7fbae6f13740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 0x557d5d9d2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.193+0000 7fbae6f13740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557d5d92c090 con 0x557d5d9d0bb0 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.193+0000 7fbae6f13740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x557d5d8d3a70 con 0x557d5d9c62f0 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.193+0000 7fbae6f13740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557d5d928050 con 0x557d5d9c59d0 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.193+0000 7fbad7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 0x557d5d9d2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.193+0000 7fbad7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 0x557d5d9d2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33890/0 (socket says 172.21.15.33:33890) 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.193+0000 7fbad7fff640 1 -- 172.21.15.33:0/375996934 learned_addr learned my addr 172.21.15.33:0/375996934 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.193+0000 7fbad7fff640 1 --2- 172.21.15.33:0/375996934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c59d0 0x557d5d9c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbade59c640 1 --2- 172.21.15.33:0/375996934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9c62f0 0x557d5d9d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbad77fe640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbacc0a3350 0x7fbacc0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbad7fff640 1 -- 172.21.15.33:0/375996934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c59d0 msgr2=0x557d5d9c5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbad7fff640 1 --2- 172.21.15.33:0/375996934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c59d0 0x557d5d9c5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbad7fff640 1 -- 172.21.15.33:0/375996934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9c62f0 msgr2=0x557d5d9d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbad7fff640 1 --2- 172.21.15.33:0/375996934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9c62f0 0x557d5d9d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbad7fff640 1 -- 172.21.15.33:0/375996934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d5d8e0130 con 0x557d5d9d0bb0 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbade59c640 1 --2- 172.21.15.33:0/375996934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9c62f0 0x557d5d9d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbaded9d640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbacc0a3350 0x7fbacc0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbad7fff640 1 --2- 172.21.15.33:0/375996934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 0x557d5d9d2fa0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbac80134e0 tx=0x7fbac800a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5572374c2588a25 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbad77fe640 1 -- 172.21.15.33:0/4084129271 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fbad80cffa0 con 0x7fbae0102ef0 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbad4ff9640 1 -- 172.21.15.33:0/375996934 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac8019030 con 0x557d5d9d0bb0 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbad4ff9640 1 -- 172.21.15.33:0/375996934 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbac8005500 con 0x557d5d9d0bb0 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.194+0000 7fbad4ff9640 1 -- 172.21.15.33:0/375996934 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac80116b0 con 0x557d5d9d0bb0 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.195+0000 7fbae6f13740 1 -- 172.21.15.33:0/375996934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 msgr2=0x557d5d9d2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.195+0000 7fbaded9d640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbacc0a3350 0x7fbacc0a5810 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbad809bf10 tx=0x7fbad8098350 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.195+0000 7fbae6f13740 1 --2- 172.21.15.33:0/375996934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 0x557d5d9d2fa0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbac80134e0 tx=0x7fbac800a750 comp rx=0 tx=0).stop 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.195+0000 7fbae6f13740 1 -- 172.21.15.33:0/375996934 shutdown_connections 2024-09-16T00:17:56.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.195+0000 7fbae6f13740 1 --2- 172.21.15.33:0/375996934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 0x557d5d9d2fa0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.195+0000 7fbae6f13740 1 --2- 172.21.15.33:0/375996934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9c62f0 0x557d5d9d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.195+0000 7fbae6f13740 1 --2- 172.21.15.33:0/375996934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c59d0 0x557d5d9c5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.195+0000 7fbae6f13740 1 -- 172.21.15.33:0/375996934 >> 172.21.15.33:0/375996934 conn(0x557d5d9c1f60 msgr2=0x557d5d9c2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.195+0000 7fbae6f13740 1 -- 172.21.15.33:0/375996934 shutdown_connections 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.195+0000 7fbae6f13740 1 -- 172.21.15.33:0/375996934 wait complete. 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.195+0000 7fbae6f13740 1 -- start start 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.195+0000 7fbae6f13740 1 Processor -- start 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbae6f13740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 0x557d5d9d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbae6f13740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c62f0 0x557d5d9cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbae6f13740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9d0bb0 0x557d5d9cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbae6f13740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557d5d9d3780 con 0x557d5d9c59d0 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbae6f13740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x557d5d8e6f20 con 0x557d5d9d0bb0 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbae6f13740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557d5d9d3900 con 0x557d5d9c62f0 2024-09-16T00:17:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbae6f13740 10 client.0 _ll_register_callbacks cb 0x557d5d8b0030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbad7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 0x557d5d9d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbad7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 0x557d5d9d04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33898/0 (socket says 172.21.15.33:33898) 2024-09-16T00:17:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbad7fff640 1 -- 172.21.15.33:0/4183736233 learned_addr learned my addr 172.21.15.33:0/4183736233 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbad7fff640 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9d0bb0 0x557d5d9cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbade59c640 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c62f0 0x557d5d9cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbad7fff640 1 -- 172.21.15.33:0/4183736233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c62f0 msgr2=0x557d5d9cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbad7fff640 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c62f0 0x557d5d9cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbad7fff640 1 -- 172.21.15.33:0/4183736233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9d0bb0 msgr2=0x557d5d9cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbad7fff640 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9d0bb0 0x557d5d9cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbad7fff640 1 -- 172.21.15.33:0/4183736233 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d5d69d2b0 con 0x557d5d9c59d0 2024-09-16T00:17:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbad7fff640 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 0x557d5d9d04e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbac8004b60 tx=0x7fbac8004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5aac769876bfc3e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.196+0000 7fbac5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbac5ffb640 1 -- 172.21.15.33:0/4183736233 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac8019040 con 0x557d5d9c59d0 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbac5ffb640 1 -- 172.21.15.33:0/4183736233 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fbac8005000 con 0x557d5d9c59d0 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbae6f13740 1 -- 172.21.15.33:0/4183736233 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbac8005500 con 0x557d5d9c59d0 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbac5ffb640 1 -- 172.21.15.33:0/4183736233 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbac8005200 con 0x557d5d9c59d0 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbae6f13740 10 client.15444 fetch_fsmap learned FSMap version 26 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbae6f13740 1 -- 172.21.15.33:0/4183736233 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557d5d9df0a0 con 0x557d5d9c59d0 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbac5ffb640 1 -- 172.21.15.33:0/4183736233 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbac8005500 con 0x557d5d9c59d0 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbac5ffb640 1 -- 172.21.15.33:0/4183736233 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fbac8011c40 con 0x557d5d9c59d0 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbae6f13740 10 client.15444 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbae6f13740 10 client.15444 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbae6f13740 20 client.15444 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbae6f13740 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557d5d9e13b0 0x557d5d9e2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbae6f13740 4 client.15444 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbae6f13740 1 -- 172.21.15.33:0/4183736233 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d5d939600 con 0x557d5d9e13b0 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.197+0000 7fbade59c640 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557d5d9e13b0 0x557d5d9e2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.198+0000 7fbade59c640 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557d5d9e13b0 0x557d5d9e2820 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a18df00dd4e0ff4e server_cookie=d0c6536956067fa3 in_seq=0 out_seq=0 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.198+0000 7fbac5ffb640 10 client.15444 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbac5ffb640 1 -- 172.21.15.33:0/4183736233 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557d5d939600 con 0x557d5d9e13b0 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbac5ffb640 10 client.15444 handle_command_reply: tid=0 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbae6f13740 1 client.15444 shutdown 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbae6f13740 1 -- 172.21.15.33:0/4183736233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 msgr2=0x557d5d9d04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbae6f13740 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 0x557d5d9d04e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbac8004b60 tx=0x7fbac8004a60 comp rx=0 tx=0).stop 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbae6f13740 1 -- 172.21.15.33:0/4183736233 shutdown_connections 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbae6f13740 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557d5d9e13b0 0x557d5d9e2820 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbae6f13740 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9d0bb0 0x557d5d9cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbac5ffb640 0 client.15444 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbae6f13740 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c62f0 0x557d5d9cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbae6f13740 1 --2- 172.21.15.33:0/4183736233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 0x557d5d9d04e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbae6f13740 1 -- 172.21.15.33:0/4183736233 >> 172.21.15.33:0/4183736233 conn(0x557d5d9c1f60 msgr2=0x557d5d9c7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbae6f13740 1 -- 172.21.15.33:0/4183736233 shutdown_connections 2024-09-16T00:17:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbae6f13740 1 -- 172.21.15.33:0/4183736233 wait complete. 2024-09-16T00:17:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.199+0000 7fbae6f13740 20 client.15444 trim_cache size 0 max 16384 2024-09-16T00:17:56.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbae6f13740 1 Processor -- start 2024-09-16T00:17:56.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbae6f13740 1 -- start start 2024-09-16T00:17:56.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbae6f13740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c59d0 0x557d5d9e1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbae6f13740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9c62f0 0x557d5d9e04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbae6f13740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 0x557d5d9c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbae6f13740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557d5d9e0b60 con 0x557d5d9d0bb0 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbae6f13740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x557d5d9e1230 con 0x557d5d9c62f0 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbae6f13740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557d5d9e1900 con 0x557d5d9c59d0 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbad7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 0x557d5d9c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbad7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 0x557d5d9c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33904/0 (socket says 172.21.15.33:33904) 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbad7fff640 1 -- 172.21.15.33:0/628307155 learned_addr learned my addr 172.21.15.33:0/628307155 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbade59c640 1 --2- 172.21.15.33:0/628307155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9c62f0 0x557d5d9e04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.217+0000 7fbad7fff640 1 -- 172.21.15.33:0/628307155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c59d0 msgr2=0x557d5d9e1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.218+0000 7fbad7fff640 1 --2- 172.21.15.33:0/628307155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c59d0 0x557d5d9e1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.218+0000 7fbad7fff640 1 -- 172.21.15.33:0/628307155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9c62f0 msgr2=0x557d5d9e04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.218+0000 7fbad7fff640 1 --2- 172.21.15.33:0/628307155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9c62f0 0x557d5d9e04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.218+0000 7fbad7fff640 1 -- 172.21.15.33:0/628307155 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d5d9d0100 con 0x557d5d9d0bb0 2024-09-16T00:17:56.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.218+0000 7fbade59c640 1 --2- 172.21.15.33:0/628307155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9c62f0 0x557d5d9e04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:17:56.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.218+0000 7fbad7fff640 1 --2- 172.21.15.33:0/628307155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 0x557d5d9c8e70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fbac8000c00 tx=0x7fbac800aea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7db317e4fbbcd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:56.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.218+0000 7fbac7fff640 1 -- 172.21.15.33:0/628307155 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac8022070 con 0x557d5d9d0bb0 2024-09-16T00:17:56.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.218+0000 7fbac7fff640 1 -- 172.21.15.33:0/628307155 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbac8005500 con 0x557d5d9d0bb0 2024-09-16T00:17:56.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.218+0000 7fbac7fff640 1 -- 172.21.15.33:0/628307155 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac8019050 con 0x557d5d9d0bb0 2024-09-16T00:17:56.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.218+0000 7fbae6f13740 1 -- 172.21.15.33:0/628307155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 msgr2=0x557d5d9c8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:56.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.218+0000 7fbae6f13740 1 --2- 172.21.15.33:0/628307155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 0x557d5d9c8e70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fbac8000c00 tx=0x7fbac800aea0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 -- 172.21.15.33:0/628307155 shutdown_connections 2024-09-16T00:17:56.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 --2- 172.21.15.33:0/628307155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9d0bb0 0x557d5d9c8e70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 --2- 172.21.15.33:0/628307155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9c62f0 0x557d5d9e04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 --2- 172.21.15.33:0/628307155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c59d0 0x557d5d9e1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 -- 172.21.15.33:0/628307155 >> 172.21.15.33:0/628307155 conn(0x557d5d9c1f60 msgr2=0x557d5d9c2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 -- 172.21.15.33:0/628307155 shutdown_connections 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 -- 172.21.15.33:0/628307155 wait complete. 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 -- start start 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 Processor -- start 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 0x557d5d9e5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c62f0 0x557d5d9e5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9d0bb0 0x557d5d9e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557d5d9e26a0 con 0x557d5d9c59d0 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x557d5d9e1fd0 con 0x557d5d9d0bb0 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557d5d9e2d70 con 0x557d5d9c62f0 2024-09-16T00:17:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbae6f13740 10 client.0 _ll_register_callbacks cb 0x7fbac800af30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbad7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 0x557d5d9e5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbad7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 0x557d5d9e5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33912/0 (socket says 172.21.15.33:33912) 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.219+0000 7fbad7fff640 1 -- 172.21.15.33:0/3249557382 learned_addr learned my addr 172.21.15.33:0/3249557382 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbad7fff640 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9d0bb0 0x557d5d9e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbade59c640 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c62f0 0x557d5d9e5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbad7fff640 1 -- 172.21.15.33:0/3249557382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c62f0 msgr2=0x557d5d9e5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbad7fff640 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c62f0 0x557d5d9e5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbad7fff640 1 -- 172.21.15.33:0/3249557382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9d0bb0 msgr2=0x557d5d9e62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbad7fff640 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9d0bb0 0x557d5d9e62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbad7fff640 1 -- 172.21.15.33:0/3249557382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d5da49ab0 con 0x557d5d9c59d0 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbad7fff640 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 0x557d5d9e5820 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fbac800a5b0 tx=0x7fbac80048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6da6a487fad79 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbac5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:17:56.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbac5ffb640 1 -- 172.21.15.33:0/3249557382 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac8022070 con 0x557d5d9c59d0 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbac5ffb640 1 -- 172.21.15.33:0/3249557382 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fbac80118b0 con 0x557d5d9c59d0 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbae6f13740 1 -- 172.21.15.33:0/3249557382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbac8005500 con 0x557d5d9c59d0 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbac5ffb640 1 -- 172.21.15.33:0/3249557382 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbac8045050 con 0x557d5d9c59d0 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbae6f13740 10 client.15450 fetch_fsmap learned FSMap version 26 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.220+0000 7fbae6f13740 1 -- 172.21.15.33:0/3249557382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557d5da4acf0 con 0x557d5d9c59d0 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.221+0000 7fbac5ffb640 1 -- 172.21.15.33:0/3249557382 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbac8005500 con 0x557d5d9c59d0 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.221+0000 7fbac5ffb640 1 -- 172.21.15.33:0/3249557382 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fbac8038450 con 0x557d5d9c59d0 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.221+0000 7fbae6f13740 10 client.15450 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.221+0000 7fbae6f13740 10 client.15450 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.221+0000 7fbae6f13740 20 client.15450 populate_metadata read hostname 'smithi033' 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.221+0000 7fbae6f13740 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557d5da4f4d0 0x557d5da4e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.221+0000 7fbae6f13740 4 client.15450 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T00:17:56.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.221+0000 7fbae6f13740 1 -- 172.21.15.33:0/3249557382 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "status"}) -- 0x557d5d933b10 con 0x557d5da4f4d0 2024-09-16T00:17:56.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.221+0000 7fbade59c640 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557d5da4f4d0 0x557d5da4e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:17:56.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.221+0000 7fbade59c640 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557d5da4f4d0 0x557d5da4e1e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b53107f3f740ef94 server_cookie=cbbb4f25bda872ce in_seq=0 out_seq=0 2024-09-16T00:17:56.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.221+0000 7fbac5ffb640 10 client.15450 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:56.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbac5ffb640 1 -- 172.21.15.33:0/3249557382 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557d5d933b10 con 0x557d5da4f4d0 2024-09-16T00:17:56.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbac5ffb640 10 client.15450 handle_command_reply: tid=0 2024-09-16T00:17:56.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae6f13740 1 client.15450 shutdown 2024-09-16T00:17:56.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae6f13740 1 -- 172.21.15.33:0/3249557382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 msgr2=0x557d5d9e5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:56.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae6f13740 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 0x557d5d9e5820 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fbac800a5b0 tx=0x7fbac80048b0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae6f13740 1 -- 172.21.15.33:0/3249557382 shutdown_connections 2024-09-16T00:17:56.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae6f13740 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557d5da4f4d0 0x557d5da4e1e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae6f13740 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557d5d9d0bb0 0x557d5d9e62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae6f13740 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557d5d9c62f0 0x557d5d9e5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae6f13740 1 --2- 172.21.15.33:0/3249557382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d5d9c59d0 0x557d5d9e5820 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae6f13740 1 -- 172.21.15.33:0/3249557382 >> 172.21.15.33:0/3249557382 conn(0x557d5d9c1f60 msgr2=0x557d5d9c8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:56.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbac5ffb640 0 client.15450 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:17:56.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae6f13740 1 -- 172.21.15.33:0/3249557382 shutdown_connections 2024-09-16T00:17:56.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae6f13740 1 -- 172.21.15.33:0/3249557382 wait complete. 2024-09-16T00:17:56.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae6f13740 20 client.15450 trim_cache size 0 max 16384 2024-09-16T00:17:56.235 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:17:56.236 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "7dde3d1a-73bf-11ef-bceb-c7b262605968", 2024-09-16T00:17:56.236 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 0, 2024-09-16T00:17:56.236 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24494, 2024-09-16T00:17:56.236 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-09-16T00:17:56.236 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-09-16T00:17:56.236 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-09-16T00:17:56.236 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 168.69418046999999, 2024-09-16T00:17:56.236 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 26, 2024-09-16T00:17:56.236 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 90, 2024-09-16T00:17:56.236 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 90, 2024-09-16T00:17:56.236 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 179.77194894900001 2024-09-16T00:17:56.236 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:17:56.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.222+0000 7fbae4b3a640 1 -- 172.21.15.33:0/4084129271 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbacc0a3350 msgr2=0x7fbacc0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:56.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.223+0000 7fbae4b3a640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbacc0a3350 0x7fbacc0a5810 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbad809bf10 tx=0x7fbad8098350 comp rx=0 tx=0).stop 2024-09-16T00:17:56.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.223+0000 7fbae4b3a640 1 -- 172.21.15.33:0/4084129271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbae0102ef0 msgr2=0x7fbae019bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:17:56.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.223+0000 7fbae4b3a640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbae0102ef0 0x7fbae019bcb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbad800cad0 tx=0x7fbad800cf90 comp rx=0 tx=0).stop 2024-09-16T00:17:56.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.223+0000 7fbae4b3a640 1 -- 172.21.15.33:0/4084129271 shutdown_connections 2024-09-16T00:17:56.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.223+0000 7fbae4b3a640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbacc0a3350 0x7fbacc0a5810 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.223+0000 7fbae4b3a640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbae0102ef0 0x7fbae019bcb0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.223+0000 7fbae4b3a640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbae0102570 0x7fbae0197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.223+0000 7fbae4b3a640 1 --2- 172.21.15.33:0/4084129271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbae0100e20 0x7fbae01973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:17:56.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.223+0000 7fbae4b3a640 1 -- 172.21.15.33:0/4084129271 >> 172.21.15.33:0/4084129271 conn(0x7fbae00fc7d0 msgr2=0x7fbae00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:17:56.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.223+0000 7fbae4b3a640 1 -- 172.21.15.33:0/4084129271 shutdown_connections 2024-09-16T00:17:56.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:17:56.223+0000 7fbae4b3a640 1 -- 172.21.15.33:0/4084129271 wait complete. 2024-09-16T00:17:57.025 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-09-16T00:17:57.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:56 smithi083 ceph-mon[35242]: pgmap v374: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-16T00:17:57.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:56 smithi060 ceph-mon[35185]: pgmap v374: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-16T00:17:57.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:56 smithi033 ceph-mon[31243]: pgmap v374: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-16T00:17:59.133 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:17:58 smithi033 ceph-mon[31243]: pgmap v375: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-09-16T00:17:59.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:17:58 smithi083 ceph-mon[35242]: pgmap v375: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-09-16T00:17:59.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:17:58 smithi060 ceph-mon[35185]: pgmap v375: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-09-16T00:18:01.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:00 smithi083 ceph-mon[35242]: pgmap v376: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-16T00:18:01.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:00 smithi060 ceph-mon[35185]: pgmap v376: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-16T00:18:01.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:00 smithi033 ceph-mon[31243]: pgmap v376: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-16T00:18:03.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:02 smithi083 ceph-mon[35242]: pgmap v377: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-16T00:18:03.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:02 smithi060 ceph-mon[35185]: pgmap v377: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-16T00:18:03.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:02 smithi033 ceph-mon[31243]: pgmap v377: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-16T00:18:05.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:04 smithi083 ceph-mon[35242]: pgmap v378: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-16T00:18:05.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:04 smithi060 ceph-mon[35185]: pgmap v378: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-16T00:18:05.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:04 smithi033 ceph-mon[31243]: pgmap v378: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-16T00:18:07.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:06 smithi083 ceph-mon[35242]: pgmap v379: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T00:18:07.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:06 smithi060 ceph-mon[35185]: pgmap v379: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T00:18:07.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:06 smithi033 ceph-mon[31243]: pgmap v379: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T00:18:09.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:08 smithi083 ceph-mon[35242]: pgmap v380: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-16T00:18:09.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:08 smithi060 ceph-mon[35185]: pgmap v380: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-16T00:18:09.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:08 smithi033 ceph-mon[31243]: pgmap v380: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-16T00:18:11.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:10 smithi083 ceph-mon[35242]: pgmap v381: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-16T00:18:11.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:10 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:18:11.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:10 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:18:11.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:10 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:18:11.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:10 smithi060 ceph-mon[35185]: pgmap v381: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-16T00:18:11.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:10 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:18:11.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:10 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:18:11.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:10 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:18:11.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:10 smithi033 ceph-mon[31243]: pgmap v381: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-16T00:18:11.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:10 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:18:11.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:10 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:18:11.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:10 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:18:12.026 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.2 2024-09-16T00:18:12.027 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:18:12.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:11 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:18:12.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:11 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:18:12.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:11 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:18:12.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:11 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:18:12.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:11 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:18:12.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:11 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:18:12.295 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:18:12.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:11 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:18:12.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:11 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:18:12.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:11 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:18:12.336 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:18:12.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.818+0000 7f8d8493b640 1 -- 172.21.15.33:0/3205811570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d80103740 msgr2=0x7f8d80107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:12.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.818+0000 7f8d8493b640 1 --2- 172.21.15.33:0/3205811570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d80103740 0x7f8d80107b80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009930 tx=0x7f8d6802a420 comp rx=0 tx=0).stop 2024-09-16T00:18:12.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.818+0000 7f8d8493b640 1 -- 172.21.15.33:0/3205811570 shutdown_connections 2024-09-16T00:18:12.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.818+0000 7f8d8493b640 1 --2- 172.21.15.33:0/3205811570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d80108290 0x7f8d8010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:12.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.818+0000 7f8d8493b640 1 --2- 172.21.15.33:0/3205811570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d80103740 0x7f8d80107b80 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:12.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.818+0000 7f8d8493b640 1 --2- 172.21.15.33:0/3205811570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d80102d90 0x7f8d80103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:12.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.818+0000 7f8d8493b640 1 -- 172.21.15.33:0/3205811570 >> 172.21.15.33:0/3205811570 conn(0x7f8d800fc7d0 msgr2=0x7f8d800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:12.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.818+0000 7f8d8493b640 1 -- 172.21.15.33:0/3205811570 shutdown_connections 2024-09-16T00:18:12.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.818+0000 7f8d8493b640 1 -- 172.21.15.33:0/3205811570 wait complete. 2024-09-16T00:18:12.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.819+0000 7f8d8493b640 1 Processor -- start 2024-09-16T00:18:12.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.819+0000 7f8d8493b640 1 -- start start 2024-09-16T00:18:12.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.819+0000 7f8d8493b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d80102d90 0x7f8d80198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:12.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.819+0000 7f8d8493b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d80103740 0x7f8d80198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:12.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.819+0000 7f8d8493b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d80108290 0x7f8d8019cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:12.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.819+0000 7f8d8493b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80112090 con 0x7f8d80103740 2024-09-16T00:18:12.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.819+0000 7f8d8493b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80111f10 con 0x7f8d80108290 2024-09-16T00:18:12.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.819+0000 7f8d8493b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80112210 con 0x7f8d80102d90 2024-09-16T00:18:12.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.819+0000 7f8d77fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d80103740 0x7f8d80198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:12.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.819+0000 7f8d77fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d80103740 0x7f8d80198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56892/0 (socket says 172.21.15.33:56892) 2024-09-16T00:18:12.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.819+0000 7f8d77fff640 1 -- 172.21.15.33:0/1482541314 learned_addr learned my addr 172.21.15.33:0/1482541314 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:12.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.819+0000 7f8d7e59c640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d80102d90 0x7f8d80198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:12.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d7ed9d640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d80108290 0x7f8d8019cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:12.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d77fff640 1 -- 172.21.15.33:0/1482541314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d80102d90 msgr2=0x7f8d80198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:12.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d77fff640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d80102d90 0x7f8d80198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:12.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d77fff640 1 -- 172.21.15.33:0/1482541314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d80108290 msgr2=0x7f8d8019cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:12.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d77fff640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d80108290 0x7f8d8019cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:12.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d77fff640 1 -- 172.21.15.33:0/1482541314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8019d5c0 con 0x7f8d80103740 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d7ed9d640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d80108290 0x7f8d8019cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d7e59c640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d80102d90 0x7f8d80198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d77fff640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d80103740 0x7f8d80198b50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8d68005940 tx=0x7f8d680042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d777fe640 1 -- 172.21.15.33:0/1482541314 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6803f030 con 0x7f8d80103740 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d8493b640 1 -- 172.21.15.33:0/1482541314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8019d850 con 0x7f8d80103740 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d777fe640 1 -- 172.21.15.33:0/1482541314 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d68003ce0 con 0x7f8d80103740 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d777fe640 1 -- 172.21.15.33:0/1482541314 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d68035070 con 0x7f8d80103740 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.820+0000 7f8d8493b640 1 -- 172.21.15.33:0/1482541314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d801a5190 con 0x7f8d80103740 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.821+0000 7f8d8493b640 1 -- 172.21.15.33:0/1482541314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d8006a7f0 con 0x7f8d80103740 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.823+0000 7f8d777fe640 1 -- 172.21.15.33:0/1482541314 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f8d68042030 con 0x7f8d80103740 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.823+0000 7f8d777fe640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d6c0a32d0 0x7f8d6c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.824+0000 7f8d7e59c640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d6c0a32d0 0x7f8d6c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.824+0000 7f8d777fe640 1 -- 172.21.15.33:0/1482541314 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8d680f07d0 con 0x7f8d80103740 2024-09-16T00:18:12.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.824+0000 7f8d7e59c640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d6c0a32d0 0x7f8d6c0a5790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8d70005fd0 tx=0x7f8d70005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:12.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:12.827+0000 7f8d777fe640 1 -- 172.21.15.33:0/1482541314 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8d680bcf10 con 0x7f8d80103740 2024-09-16T00:18:12.871 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:18:13.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.049+0000 7f8d8493b640 1 -- 172.21.15.33:0/1482541314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d8010a2a0 con 0x7f8d80103740 2024-09-16T00:18:13.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.050+0000 7f8d777fe640 1 -- 172.21.15.33:0/1482541314 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11903 (secure 0 0 0) 0x7f8d680bc810 con 0x7f8d80103740 2024-09-16T00:18:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:18:13.051 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":26,"btime":"2024-09-16T00:17:28:789138+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:17:28.493434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:18:13.051 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 26 2024-09-16T00:18:13.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.052+0000 7f8d8493b640 1 -- 172.21.15.33:0/1482541314 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d6c0a32d0 msgr2=0x7f8d6c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:13.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.052+0000 7f8d8493b640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d6c0a32d0 0x7f8d6c0a5790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8d70005fd0 tx=0x7f8d70005ea0 comp rx=0 tx=0).stop 2024-09-16T00:18:13.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.052+0000 7f8d8493b640 1 -- 172.21.15.33:0/1482541314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d80103740 msgr2=0x7f8d80198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:13.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.052+0000 7f8d8493b640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d80103740 0x7f8d80198b50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8d68005940 tx=0x7f8d680042f0 comp rx=0 tx=0).stop 2024-09-16T00:18:13.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.052+0000 7f8d8493b640 1 -- 172.21.15.33:0/1482541314 shutdown_connections 2024-09-16T00:18:13.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.052+0000 7f8d8493b640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d6c0a32d0 0x7f8d6c0a5790 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:13.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.052+0000 7f8d8493b640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d80108290 0x7f8d8019cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:13.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.052+0000 7f8d8493b640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d80103740 0x7f8d80198b50 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:13.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.052+0000 7f8d8493b640 1 --2- 172.21.15.33:0/1482541314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d80102d90 0x7f8d80198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:13.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.052+0000 7f8d8493b640 1 -- 172.21.15.33:0/1482541314 >> 172.21.15.33:0/1482541314 conn(0x7f8d800fc7d0 msgr2=0x7f8d80105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:13.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.052+0000 7f8d8493b640 1 -- 172.21.15.33:0/1482541314 shutdown_connections 2024-09-16T00:18:13.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:13.052+0000 7f8d8493b640 1 -- 172.21.15.33:0/1482541314 wait complete. 2024-09-16T00:18:13.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:13 smithi033 ceph-mon[31243]: pgmap v382: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:18:13.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:13 smithi083 ceph-mon[35242]: pgmap v382: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:18:13.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:13 smithi060 ceph-mon[35185]: pgmap v382: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:18:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:14 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1482541314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:18:14.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:14 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1482541314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:18:14.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:14 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1482541314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:18:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:15 smithi033 ceph-mon[31243]: pgmap v383: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:18:15.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:15 smithi083 ceph-mon[35242]: pgmap v383: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:18:15.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:15 smithi060 ceph-mon[35185]: pgmap v383: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T00:18:16.224 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 150.2 --await-for 45.0' 2024-09-16T00:18:16.224 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 150.2 --await-for 45.0 2024-09-16T00:18:16.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.413+0000 7f8d37014640 1 -- 172.21.15.33:0/3016091145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d301009b0 msgr2=0x7f8d30100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:16.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.413+0000 7f8d37014640 1 --2- 172.21.15.33:0/3016091145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d301009b0 0x7f8d30100d90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8d24009900 tx=0x7f8d2402a410 comp rx=0 tx=0).stop 2024-09-16T00:18:16.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.414+0000 7f8d37014640 1 -- 172.21.15.33:0/3016091145 shutdown_connections 2024-09-16T00:18:16.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.414+0000 7f8d37014640 1 --2- 172.21.15.33:0/3016091145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d3010e4d0 0x7f8d301108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.414+0000 7f8d37014640 1 --2- 172.21.15.33:0/3016091145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d30101360 0x7f8d3010df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.414+0000 7f8d37014640 1 --2- 172.21.15.33:0/3016091145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d301009b0 0x7f8d30100d90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.414+0000 7f8d37014640 1 -- 172.21.15.33:0/3016091145 >> 172.21.15.33:0/3016091145 conn(0x7f8d300fc7d0 msgr2=0x7f8d300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:16.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.414+0000 7f8d37014640 1 -- 172.21.15.33:0/3016091145 shutdown_connections 2024-09-16T00:18:16.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.414+0000 7f8d37014640 1 -- 172.21.15.33:0/3016091145 wait complete. 2024-09-16T00:18:16.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d37014640 1 Processor -- start 2024-09-16T00:18:16.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d37014640 1 -- start start 2024-09-16T00:18:16.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d37014640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d301009b0 0x7f8d301a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d37014640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d30101360 0x7f8d301a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d37014640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d3010e4d0 0x7f8d3019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d37014640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d30114230 con 0x7f8d301009b0 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d37014640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301140b0 con 0x7f8d30101360 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d37014640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301143b0 con 0x7f8d3010e4d0 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d34db0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d301009b0 0x7f8d301a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d34db0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d301009b0 0x7f8d301a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56920/0 (socket says 172.21.15.33:56920) 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d2ffff640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d30101360 0x7f8d301a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d34db0640 1 -- 172.21.15.33:0/3980746950 learned_addr learned my addr 172.21.15.33:0/3980746950 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.415+0000 7f8d355b1640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d3010e4d0 0x7f8d3019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.416+0000 7f8d34db0640 1 -- 172.21.15.33:0/3980746950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d3010e4d0 msgr2=0x7f8d3019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.416+0000 7f8d34db0640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d3010e4d0 0x7f8d3019a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.416+0000 7f8d34db0640 1 -- 172.21.15.33:0/3980746950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d30101360 msgr2=0x7f8d301a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.416+0000 7f8d34db0640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d30101360 0x7f8d301a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.416+0000 7f8d34db0640 1 -- 172.21.15.33:0/3980746950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d3019acc0 con 0x7f8d301009b0 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.416+0000 7f8d2ffff640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d30101360 0x7f8d301a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.416+0000 7f8d355b1640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d3010e4d0 0x7f8d3019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.416+0000 7f8d34db0640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d301009b0 0x7f8d301a03d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8d24006010 tx=0x7f8d240042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.416+0000 7f8d2dffb640 1 -- 172.21.15.33:0/3980746950 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2403f030 con 0x7f8d301009b0 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.416+0000 7f8d37014640 1 -- 172.21.15.33:0/3980746950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d3019af50 con 0x7f8d301009b0 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.416+0000 7f8d2dffb640 1 -- 172.21.15.33:0/3980746950 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d24003ce0 con 0x7f8d301009b0 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.416+0000 7f8d2dffb640 1 -- 172.21.15.33:0/3980746950 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24035070 con 0x7f8d301009b0 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.417+0000 7f8d37014640 1 -- 172.21.15.33:0/3980746950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d301a72a0 con 0x7f8d301009b0 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.419+0000 7f8d2dffb640 1 -- 172.21.15.33:0/3980746950 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f8d24042030 con 0x7f8d301009b0 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.420+0000 7f8d2dffb640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d200a32d0 0x7f8d200a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.420+0000 7f8d2dffb640 1 -- 172.21.15.33:0/3980746950 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8d240effe0 con 0x7f8d301009b0 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.420+0000 7f8d2ffff640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d200a32d0 0x7f8d200a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.421+0000 7f8d393ed740 1 Processor -- start 2024-09-16T00:18:16.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.421+0000 7f8d2ffff640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d200a32d0 0x7f8d200a5790 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8d3010e980 tx=0x7f8d1c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:16.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.421+0000 7f8d393ed740 1 -- start start 2024-09-16T00:18:16.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.421+0000 7f8d393ed740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 0x55e3e8a80880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.421+0000 7f8d393ed740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8a8b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.421+0000 7f8d393ed740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a8b680 0x55e3e8a8da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.421+0000 7f8d393ed740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e3e89e62e0 con 0x55e3e8a804a0 2024-09-16T00:18:16.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.421+0000 7f8d34db0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 0x55e3e8a80880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d34db0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 0x55e3e8a80880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56930/0 (socket says 172.21.15.33:56930) 2024-09-16T00:18:16.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d34db0640 1 -- 172.21.15.33:0/4243572643 learned_addr learned my addr 172.21.15.33:0/4243572643 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:16.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d393ed740 1 -- 172.21.15.33:0/4243572643 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e3e898dce0 con 0x55e3e8a80dc0 2024-09-16T00:18:16.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d393ed740 1 -- 172.21.15.33:0/4243572643 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e3e89e22a0 con 0x55e3e8a8b680 2024-09-16T00:18:16.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d2ffff640 1 --2- 172.21.15.33:0/4243572643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8a8b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d355b1640 1 --2- 172.21.15.33:0/4243572643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a8b680 0x55e3e8a8da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d34db0640 1 -- 172.21.15.33:0/4243572643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a8b680 msgr2=0x55e3e8a8da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d34db0640 1 --2- 172.21.15.33:0/4243572643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a8b680 0x55e3e8a8da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d34db0640 1 -- 172.21.15.33:0/4243572643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 msgr2=0x55e3e8a8b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d34db0640 1 --2- 172.21.15.33:0/4243572643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8a8b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d34db0640 1 -- 172.21.15.33:0/4243572643 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3e8972440 con 0x55e3e8a804a0 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d2ffff640 1 --2- 172.21.15.33:0/4243572643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8a8b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d34db0640 1 --2- 172.21.15.33:0/4243572643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 0x55e3e8a80880 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8d240ed450 tx=0x7f8d24003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4485d311c9b24471 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.422+0000 7f8d1affd640 1 -- 172.21.15.33:0/4243572643 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24049040 con 0x55e3e8a804a0 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.423+0000 7f8d1affd640 1 -- 172.21.15.33:0/4243572643 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d240bc130 con 0x55e3e8a804a0 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.423+0000 7f8d1affd640 1 -- 172.21.15.33:0/4243572643 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d240b87a0 con 0x55e3e8a804a0 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.423+0000 7f8d393ed740 1 -- 172.21.15.33:0/4243572643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 msgr2=0x55e3e8a80880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:16.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.424+0000 7f8d393ed740 1 --2- 172.21.15.33:0/4243572643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 0x55e3e8a80880 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8d240ed450 tx=0x7f8d24003f90 comp rx=0 tx=0).stop 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.424+0000 7f8d393ed740 1 -- 172.21.15.33:0/4243572643 shutdown_connections 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.424+0000 7f8d393ed740 1 --2- 172.21.15.33:0/4243572643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a8b680 0x55e3e8a8da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.424+0000 7f8d393ed740 1 --2- 172.21.15.33:0/4243572643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8a8b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.424+0000 7f8d393ed740 1 --2- 172.21.15.33:0/4243572643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 0x55e3e8a80880 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.424+0000 7f8d393ed740 1 -- 172.21.15.33:0/4243572643 >> 172.21.15.33:0/4243572643 conn(0x55e3e8a7ca30 msgr2=0x55e3e8a7ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.424+0000 7f8d393ed740 1 -- 172.21.15.33:0/4243572643 shutdown_connections 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.424+0000 7f8d393ed740 1 -- 172.21.15.33:0/4243572643 wait complete. 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.424+0000 7f8d393ed740 1 -- start start 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.424+0000 7f8d393ed740 1 Processor -- start 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.425+0000 7f8d393ed740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a804a0 0x55e3e8a8af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.425+0000 7f8d393ed740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a80dc0 0x55e3e8a88220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.425+0000 7f8d393ed740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a8b680 0x55e3e8a88760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.425+0000 7f8d393ed740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e3e8a8e010 con 0x55e3e8a8b680 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.425+0000 7f8d393ed740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e3e89a1050 con 0x55e3e8a804a0 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.425+0000 7f8d393ed740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e3e8a8e190 con 0x55e3e8a80dc0 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.425+0000 7f8d2ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a80dc0 0x55e3e8a88220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.425+0000 7f8d355b1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a8b680 0x55e3e8a88760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.425+0000 7f8d393ed740 10 client.0 _ll_register_callbacks cb 0x55e3e89e1160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.425+0000 7f8d2ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a80dc0 0x55e3e8a88220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46720/0 (socket says 172.21.15.33:46720) 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.425+0000 7f8d2ffff640 1 -- 172.21.15.33:0/1369876544 learned_addr learned my addr 172.21.15.33:0/1369876544 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.426+0000 7f8d2ffff640 1 -- 172.21.15.33:0/1369876544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a804a0 msgr2=0x55e3e8a8af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.426+0000 7f8d2ffff640 1 --2- 172.21.15.33:0/1369876544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a804a0 0x55e3e8a8af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.426+0000 7f8d2ffff640 1 -- 172.21.15.33:0/1369876544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a8b680 msgr2=0x55e3e8a88760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.426+0000 7f8d2ffff640 1 --2- 172.21.15.33:0/1369876544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a8b680 0x55e3e8a88760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.426+0000 7f8d2ffff640 1 -- 172.21.15.33:0/1369876544 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3e87573c0 con 0x55e3e8a80dc0 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.426+0000 7f8d2ffff640 1 --2- 172.21.15.33:0/1369876544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a80dc0 0x55e3e8a88220 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c011ab0 tx=0x7f8d1c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=736237d79cf92964 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:16.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.426+0000 7f8d0ffff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.426+0000 7f8d0ffff640 1 -- 172.21.15.33:0/1369876544 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c01fe00 con 0x55e3e8a80dc0 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.426+0000 7f8d0ffff640 1 -- 172.21.15.33:0/1369876544 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8d1c0273e0 con 0x55e3e8a80dc0 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.426+0000 7f8d393ed740 1 -- 172.21.15.33:0/1369876544 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d240bc130 con 0x55e3e8a80dc0 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.427+0000 7f8d0ffff640 1 -- 172.21.15.33:0/1369876544 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8d1c0275e0 con 0x55e3e8a80dc0 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.427+0000 7f8d393ed740 10 client.25063 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.427+0000 7f8d393ed740 1 -- 172.21.15.33:0/1369876544 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e3e8a99950 con 0x55e3e8a80dc0 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.427+0000 7f8d0ffff640 1 -- 172.21.15.33:0/1369876544 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d240bc130 con 0x55e3e8a80dc0 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.427+0000 7f8d0ffff640 1 -- 172.21.15.33:0/1369876544 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8d1c026070 con 0x55e3e8a80dc0 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.427+0000 7f8d393ed740 10 client.25063 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.427+0000 7f8d393ed740 10 client.25063 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.427+0000 7f8d393ed740 20 client.25063 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.427+0000 7f8d393ed740 1 --2- 172.21.15.33:0/1369876544 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3e8a9aeb0 0x55e3e8a9d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.427+0000 7f8d393ed740 4 client.25063 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.427+0000 7f8d393ed740 1 -- 172.21.15.33:0/1369876544 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3e89f3850 con 0x55e3e8a9aeb0 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.428+0000 7f8d34db0640 1 --2- 172.21.15.33:0/1369876544 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3e8a9aeb0 0x55e3e8a9d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.428+0000 7f8d0ffff640 10 client.25063 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.428+0000 7f8d34db0640 1 --2- 172.21.15.33:0/1369876544 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3e8a9aeb0 0x55e3e8a9d310 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25f837b0fe405669 server_cookie=ec48fd48a6b914f6 in_seq=0 out_seq=0 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d0ffff640 1 -- 172.21.15.33:0/1369876544 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e3e89f3850 con 0x55e3e8a9aeb0 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d0ffff640 10 client.25063 handle_command_reply: tid=0 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d393ed740 1 client.25063 shutdown 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d393ed740 1 -- 172.21.15.33:0/1369876544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a80dc0 msgr2=0x55e3e8a88220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d393ed740 1 --2- 172.21.15.33:0/1369876544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a80dc0 0x55e3e8a88220 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c011ab0 tx=0x7f8d1c004730 comp rx=0 tx=0).stop 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d393ed740 1 -- 172.21.15.33:0/1369876544 shutdown_connections 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d393ed740 1 --2- 172.21.15.33:0/1369876544 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3e8a9aeb0 0x55e3e8a9d310 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d393ed740 1 --2- 172.21.15.33:0/1369876544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a8b680 0x55e3e8a88760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d393ed740 1 --2- 172.21.15.33:0/1369876544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a80dc0 0x55e3e8a88220 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d393ed740 1 --2- 172.21.15.33:0/1369876544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a804a0 0x55e3e8a8af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d0ffff640 0 client.25063 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:16.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d393ed740 1 -- 172.21.15.33:0/1369876544 >> 172.21.15.33:0/1369876544 conn(0x55e3e8a7ca30 msgr2=0x55e3e8a7bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:16.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d393ed740 1 -- 172.21.15.33:0/1369876544 shutdown_connections 2024-09-16T00:18:16.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.429+0000 7f8d393ed740 1 -- 172.21.15.33:0/1369876544 wait complete. 2024-09-16T00:18:16.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.430+0000 7f8d393ed740 20 client.25063 trim_cache size 0 max 16384 2024-09-16T00:18:16.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.448+0000 7f8d393ed740 1 Processor -- start 2024-09-16T00:18:16.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.448+0000 7f8d393ed740 1 -- start start 2024-09-16T00:18:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d393ed740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 0x55e3e8a8a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d393ed740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8aa2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d393ed740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a8b680 0x55e3e8aa3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d393ed740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301b3bf0 con 0x55e3e8a804a0 2024-09-16T00:18:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d393ed740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301b14c0 con 0x55e3e8a80dc0 2024-09-16T00:18:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d393ed740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301af310 con 0x55e3e8a8b680 2024-09-16T00:18:16.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d2ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8aa2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d34db0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 0x55e3e8a8a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d2ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8aa2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:44040/0 (socket says 172.21.15.33:44040) 2024-09-16T00:18:16.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d2ffff640 1 -- 172.21.15.33:0/2329762218 learned_addr learned my addr 172.21.15.33:0/2329762218 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:16.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d355b1640 1 --2- 172.21.15.33:0/2329762218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a8b680 0x55e3e8aa3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d34db0640 1 -- 172.21.15.33:0/2329762218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a8b680 msgr2=0x55e3e8aa3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:16.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d34db0640 1 --2- 172.21.15.33:0/2329762218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a8b680 0x55e3e8aa3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d34db0640 1 -- 172.21.15.33:0/2329762218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 msgr2=0x55e3e8aa2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:16.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d34db0640 1 --2- 172.21.15.33:0/2329762218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8aa2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.449+0000 7f8d34db0640 1 -- 172.21.15.33:0/2329762218 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3e8a802d0 con 0x55e3e8a804a0 2024-09-16T00:18:16.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.450+0000 7f8d34db0640 1 --2- 172.21.15.33:0/2329762218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 0x55e3e8a8a190 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8d240b8590 tx=0x7f8d240bbd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0cf39ebc692763 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:16.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.450+0000 7f8d1a7fc640 1 -- 172.21.15.33:0/2329762218 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24049040 con 0x55e3e8a804a0 2024-09-16T00:18:16.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.450+0000 7f8d1a7fc640 1 -- 172.21.15.33:0/2329762218 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d240c0430 con 0x55e3e8a804a0 2024-09-16T00:18:16.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.450+0000 7f8d1a7fc640 1 -- 172.21.15.33:0/2329762218 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d240ed700 con 0x55e3e8a804a0 2024-09-16T00:18:16.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.451+0000 7f8d393ed740 1 -- 172.21.15.33:0/2329762218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 msgr2=0x55e3e8a8a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:16.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.451+0000 7f8d393ed740 1 --2- 172.21.15.33:0/2329762218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 0x55e3e8a8a190 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8d240b8590 tx=0x7f8d240bbd70 comp rx=0 tx=0).stop 2024-09-16T00:18:16.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.451+0000 7f8d393ed740 1 -- 172.21.15.33:0/2329762218 shutdown_connections 2024-09-16T00:18:16.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.451+0000 7f8d393ed740 1 --2- 172.21.15.33:0/2329762218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a8b680 0x55e3e8aa3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.451+0000 7f8d393ed740 1 --2- 172.21.15.33:0/2329762218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8aa2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.451+0000 7f8d393ed740 1 --2- 172.21.15.33:0/2329762218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a804a0 0x55e3e8a8a190 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.451+0000 7f8d393ed740 1 -- 172.21.15.33:0/2329762218 >> 172.21.15.33:0/2329762218 conn(0x55e3e8a7ca30 msgr2=0x55e3e8a89a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:16.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.451+0000 7f8d393ed740 1 -- 172.21.15.33:0/2329762218 shutdown_connections 2024-09-16T00:18:16.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.451+0000 7f8d393ed740 1 -- 172.21.15.33:0/2329762218 wait complete. 2024-09-16T00:18:16.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.452+0000 7f8d393ed740 1 -- start start 2024-09-16T00:18:16.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.452+0000 7f8d393ed740 1 Processor -- start 2024-09-16T00:18:16.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.453+0000 7f8d393ed740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a804a0 0x55e3e8afdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.453+0000 7f8d393ed740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8afe4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.453+0000 7f8d393ed740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a8b680 0x55e3e8a90860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.453+0000 7f8d393ed740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301af980 con 0x55e3e8a8b680 2024-09-16T00:18:16.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.453+0000 7f8d393ed740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301b0df0 con 0x55e3e8a80dc0 2024-09-16T00:18:16.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.453+0000 7f8d355b1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a8b680 0x55e3e8a90860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.453+0000 7f8d393ed740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301b0050 con 0x55e3e8a804a0 2024-09-16T00:18:16.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.453+0000 7f8d355b1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a8b680 0x55e3e8a90860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56954/0 (socket says 172.21.15.33:56954) 2024-09-16T00:18:16.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.453+0000 7f8d355b1640 1 -- 172.21.15.33:0/76504989 learned_addr learned my addr 172.21.15.33:0/76504989 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:16.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.453+0000 7f8d393ed740 10 client.0 _ll_register_callbacks cb 0x7f8d1c011500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:16.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.454+0000 7f8d2ffff640 1 --2- 172.21.15.33:0/76504989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8afe4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.454+0000 7f8d355b1640 1 -- 172.21.15.33:0/76504989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a804a0 msgr2=0x55e3e8afdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:16.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.454+0000 7f8d355b1640 1 --2- 172.21.15.33:0/76504989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a804a0 0x55e3e8afdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.454+0000 7f8d355b1640 1 -- 172.21.15.33:0/76504989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 msgr2=0x55e3e8afe4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:16.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.454+0000 7f8d355b1640 1 --2- 172.21.15.33:0/76504989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8afe4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.454+0000 7f8d355b1640 1 -- 172.21.15.33:0/76504989 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3e8a91060 con 0x55e3e8a8b680 2024-09-16T00:18:16.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.454+0000 7f8d355b1640 1 --2- 172.21.15.33:0/76504989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a8b680 0x55e3e8a90860 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8d28005fd0 tx=0x7f8d28010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0ca40dc79c0d87 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:16.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.454+0000 7f8d0ffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:18:16.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.454+0000 7f8d0ffff640 1 -- 172.21.15.33:0/76504989 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d28003b70 con 0x55e3e8a8b680 2024-09-16T00:18:16.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.455+0000 7f8d0ffff640 1 -- 172.21.15.33:0/76504989 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8d28003d10 con 0x55e3e8a8b680 2024-09-16T00:18:16.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.455+0000 7f8d393ed740 1 -- 172.21.15.33:0/76504989 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d240c0430 con 0x55e3e8a8b680 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.455+0000 7f8d0ffff640 1 -- 172.21.15.33:0/76504989 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8d2800e680 con 0x55e3e8a8b680 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.455+0000 7f8d393ed740 10 client.15474 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.455+0000 7f8d393ed740 1 -- 172.21.15.33:0/76504989 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e3e8b05840 con 0x55e3e8a8b680 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.455+0000 7f8d0ffff640 1 -- 172.21.15.33:0/76504989 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d240c0430 con 0x55e3e8a8b680 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.455+0000 7f8d0ffff640 1 -- 172.21.15.33:0/76504989 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8d280045c0 con 0x55e3e8a8b680 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.455+0000 7f8d393ed740 10 client.15474 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.455+0000 7f8d393ed740 10 client.15474 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.455+0000 7f8d393ed740 20 client.15474 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.455+0000 7f8d393ed740 1 --2- 172.21.15.33:0/76504989 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3e8b06040 0x55e3e8b084c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.456+0000 7f8d393ed740 4 client.15474 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.456+0000 7f8d393ed740 1 -- 172.21.15.33:0/76504989 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e3e89edd60 con 0x55e3e8b06040 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.456+0000 7f8d34db0640 1 --2- 172.21.15.33:0/76504989 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3e8b06040 0x55e3e8b084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:16.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.456+0000 7f8d34db0640 1 --2- 172.21.15.33:0/76504989 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3e8b06040 0x55e3e8b084c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f340213821af292 server_cookie=9bf3f0dae8c61bd1 in_seq=0 out_seq=0 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.456+0000 7f8d0ffff640 10 client.15474 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.457+0000 7f8d0ffff640 1 -- 172.21.15.33:0/76504989 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e3e89edd60 con 0x55e3e8b06040 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.457+0000 7f8d0ffff640 10 client.15474 handle_command_reply: tid=0 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.457+0000 7f8d393ed740 1 client.15474 shutdown 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.457+0000 7f8d393ed740 1 -- 172.21.15.33:0/76504989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a8b680 msgr2=0x55e3e8a90860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.457+0000 7f8d393ed740 1 --2- 172.21.15.33:0/76504989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a8b680 0x55e3e8a90860 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8d28005fd0 tx=0x7f8d28010800 comp rx=0 tx=0).stop 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.457+0000 7f8d393ed740 1 -- 172.21.15.33:0/76504989 shutdown_connections 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.457+0000 7f8d393ed740 1 --2- 172.21.15.33:0/76504989 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3e8b06040 0x55e3e8b084c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.457+0000 7f8d393ed740 1 --2- 172.21.15.33:0/76504989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3e8a8b680 0x55e3e8a90860 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.457+0000 7f8d393ed740 1 --2- 172.21.15.33:0/76504989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3e8a80dc0 0x55e3e8afe4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.457+0000 7f8d0ffff640 0 client.15474 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.457+0000 7f8d393ed740 1 --2- 172.21.15.33:0/76504989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3e8a804a0 0x55e3e8afdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.457+0000 7f8d393ed740 1 -- 172.21.15.33:0/76504989 >> 172.21.15.33:0/76504989 conn(0x55e3e8a7ca30 msgr2=0x55e3e8a89a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:16.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.458+0000 7f8d393ed740 1 -- 172.21.15.33:0/76504989 shutdown_connections 2024-09-16T00:18:16.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.458+0000 7f8d393ed740 1 -- 172.21.15.33:0/76504989 wait complete. 2024-09-16T00:18:16.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.458+0000 7f8d393ed740 20 client.15474 trim_cache size 0 max 16384 2024-09-16T00:18:16.461 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:18:16.461 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:18:16.461 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:18:16.461 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:18:16.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.459+0000 7f8d37014640 1 -- 172.21.15.33:0/3980746950 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d200a32d0 msgr2=0x7f8d200a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:16.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.459+0000 7f8d37014640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d200a32d0 0x7f8d200a5790 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8d3010e980 tx=0x7f8d1c00b390 comp rx=0 tx=0).stop 2024-09-16T00:18:16.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.459+0000 7f8d37014640 1 -- 172.21.15.33:0/3980746950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d301009b0 msgr2=0x7f8d301a03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:16.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.459+0000 7f8d37014640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d301009b0 0x7f8d301a03d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8d24006010 tx=0x7f8d240042f0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.459+0000 7f8d37014640 1 -- 172.21.15.33:0/3980746950 shutdown_connections 2024-09-16T00:18:16.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.459+0000 7f8d37014640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d200a32d0 0x7f8d200a5790 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.459+0000 7f8d37014640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d3010e4d0 0x7f8d3019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.459+0000 7f8d37014640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d30101360 0x7f8d301a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.459+0000 7f8d37014640 1 --2- 172.21.15.33:0/3980746950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d301009b0 0x7f8d301a03d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:16.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.459+0000 7f8d37014640 1 -- 172.21.15.33:0/3980746950 >> 172.21.15.33:0/3980746950 conn(0x7f8d300fc7d0 msgr2=0x7f8d300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:16.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.460+0000 7f8d37014640 1 -- 172.21.15.33:0/3980746950 shutdown_connections 2024-09-16T00:18:16.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:16.460+0000 7f8d37014640 1 -- 172.21.15.33:0/3980746950 wait complete. 2024-09-16T00:18:17.171 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:17 smithi033 ceph-mon[31243]: pgmap v384: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:18:17.199 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:18:17.200 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:18:17.277 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:18:17.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:17 smithi083 ceph-mon[35242]: pgmap v384: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:18:17.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:17 smithi060 ceph-mon[35185]: pgmap v384: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T00:18:17.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.784+0000 7fba82922640 1 -- 172.21.15.33:0/3547322234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba7c100e20 msgr2=0x7fba7c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:17.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.784+0000 7fba82922640 1 --2- 172.21.15.33:0/3547322234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba7c100e20 0x7fba7c101200 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fba6c009900 tx=0x7fba6c02a480 comp rx=0 tx=0).stop 2024-09-16T00:18:17.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.784+0000 7fba82922640 1 -- 172.21.15.33:0/3547322234 shutdown_connections 2024-09-16T00:18:17.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.784+0000 7fba82922640 1 --2- 172.21.15.33:0/3547322234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba7c102ef0 0x7fba7c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.784+0000 7fba82922640 1 --2- 172.21.15.33:0/3547322234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba7c102570 0x7fba7c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.784+0000 7fba82922640 1 --2- 172.21.15.33:0/3547322234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba7c100e20 0x7fba7c101200 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.784+0000 7fba82922640 1 -- 172.21.15.33:0/3547322234 >> 172.21.15.33:0/3547322234 conn(0x7fba7c0fc7d0 msgr2=0x7fba7c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.784+0000 7fba82922640 1 -- 172.21.15.33:0/3547322234 shutdown_connections 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.784+0000 7fba82922640 1 -- 172.21.15.33:0/3547322234 wait complete. 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.785+0000 7fba82922640 1 Processor -- start 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.785+0000 7fba82922640 1 -- start start 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.785+0000 7fba82922640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba7c100e20 0x7fba7c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.785+0000 7fba82922640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba7c102570 0x7fba7c198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.785+0000 7fba82922640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba7c102ef0 0x7fba7c19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba82922640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fba7c110f80 con 0x7fba7c102570 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba82922640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fba7c110e00 con 0x7fba7c100e20 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba82922640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fba7c111100 con 0x7fba7c102ef0 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba7b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba7c102570 0x7fba7c198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba7b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba7c102570 0x7fba7c198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56982/0 (socket says 172.21.15.33:56982) 2024-09-16T00:18:17.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba7b7fe640 1 -- 172.21.15.33:0/2185157212 learned_addr learned my addr 172.21.15.33:0/2185157212 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba7bfff640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba7c100e20 0x7fba7c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba80ebf640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba7c102ef0 0x7fba7c19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba7b7fe640 1 -- 172.21.15.33:0/2185157212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba7c102ef0 msgr2=0x7fba7c19ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba7b7fe640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba7c102ef0 0x7fba7c19ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba7b7fe640 1 -- 172.21.15.33:0/2185157212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba7c100e20 msgr2=0x7fba7c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba7b7fe640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba7c100e20 0x7fba7c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba7b7fe640 1 -- 172.21.15.33:0/2185157212 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7c19d4f0 con 0x7fba7c102570 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba80ebf640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba7c102ef0 0x7fba7c19ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba7bfff640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba7c100e20 0x7fba7c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba7b7fe640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba7c102570 0x7fba7c198a80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fba7000d950 tx=0x7fba7000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba797fa640 1 -- 172.21.15.33:0/2185157212 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba70017070 con 0x7fba7c102570 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.786+0000 7fba82922640 1 -- 172.21.15.33:0/2185157212 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba7c19d7e0 con 0x7fba7c102570 2024-09-16T00:18:17.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.787+0000 7fba797fa640 1 -- 172.21.15.33:0/2185157212 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba70009c30 con 0x7fba7c102570 2024-09-16T00:18:17.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.787+0000 7fba797fa640 1 -- 172.21.15.33:0/2185157212 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7000f6b0 con 0x7fba7c102570 2024-09-16T00:18:17.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.787+0000 7fba82922640 1 -- 172.21.15.33:0/2185157212 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba7c1a50c0 con 0x7fba7c102570 2024-09-16T00:18:17.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.789+0000 7fba84cfb740 1 Processor -- start 2024-09-16T00:18:17.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.789+0000 7fba84cfb740 1 -- start start 2024-09-16T00:18:17.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba84cfb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6680 0x563526ad6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba84cfb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6fa0 0x563526ae5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba84cfb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ae5d50 0x563526ae8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba84cfb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563526a3c2a0 con 0x563526ad6680 2024-09-16T00:18:17.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba84cfb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5635269e3b10 con 0x563526ad6fa0 2024-09-16T00:18:17.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba84cfb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563526a38260 con 0x563526ae5d50 2024-09-16T00:18:17.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba80ebf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6680 0x563526ad6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba80ebf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6680 0x563526ad6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56992/0 (socket says 172.21.15.33:56992) 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba80ebf640 1 -- 172.21.15.33:0/4284282343 learned_addr learned my addr 172.21.15.33:0/4284282343 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba7bfff640 1 --2- 172.21.15.33:0/4284282343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6fa0 0x563526ae5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba80ebf640 1 -- 172.21.15.33:0/4284282343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ae5d50 msgr2=0x563526ae8140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba80ebf640 1 --2- 172.21.15.33:0/4284282343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ae5d50 0x563526ae8140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba80ebf640 1 -- 172.21.15.33:0/4284282343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6fa0 msgr2=0x563526ae5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba80ebf640 1 --2- 172.21.15.33:0/4284282343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6fa0 0x563526ae5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba80ebf640 1 -- 172.21.15.33:0/4284282343 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635267adb90 con 0x563526ad6680 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba797fa640 1 -- 172.21.15.33:0/2185157212 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fba700109d0 con 0x7fba7c102570 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.790+0000 7fba80ebf640 1 --2- 172.21.15.33:0/4284282343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6680 0x563526ad6a60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fba74009e40 tx=0x7fba7400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9be8fe534bad48f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.791+0000 7fba627fc640 1 -- 172.21.15.33:0/4284282343 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74004d20 con 0x563526ad6680 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.791+0000 7fba627fc640 1 -- 172.21.15.33:0/4284282343 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba74005500 con 0x563526ad6680 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.791+0000 7fba627fc640 1 -- 172.21.15.33:0/4284282343 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba740118e0 con 0x563526ad6680 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.791+0000 7fba797fa640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fba640a32d0 0x7fba640a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.791+0000 7fba84cfb740 1 -- 172.21.15.33:0/4284282343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6680 msgr2=0x563526ad6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.791+0000 7fba84cfb740 1 --2- 172.21.15.33:0/4284282343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6680 0x563526ad6a60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fba74009e40 tx=0x7fba7400f730 comp rx=0 tx=0).stop 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.791+0000 7fba797fa640 1 -- 172.21.15.33:0/2185157212 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fba700cfc40 con 0x7fba7c102570 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.791+0000 7fba7bfff640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fba640a32d0 0x7fba640a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.792+0000 7fba84cfb740 1 -- 172.21.15.33:0/4284282343 shutdown_connections 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.792+0000 7fba84cfb740 1 --2- 172.21.15.33:0/4284282343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ae5d50 0x563526ae8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.792+0000 7fba84cfb740 1 --2- 172.21.15.33:0/4284282343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6fa0 0x563526ae5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.792+0000 7fba84cfb740 1 --2- 172.21.15.33:0/4284282343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6680 0x563526ad6a60 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.792+0000 7fba84cfb740 1 -- 172.21.15.33:0/4284282343 >> 172.21.15.33:0/4284282343 conn(0x563526ad2cf0 msgr2=0x563526ad30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.792+0000 7fba84cfb740 1 -- 172.21.15.33:0/4284282343 shutdown_connections 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.792+0000 7fba7bfff640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fba640a32d0 0x7fba640a5790 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fba6c005c40 tx=0x7fba6c031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.792+0000 7fba84cfb740 1 -- 172.21.15.33:0/4284282343 wait complete. 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.793+0000 7fba84cfb740 1 -- start start 2024-09-16T00:18:17.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.793+0000 7fba84cfb740 1 Processor -- start 2024-09-16T00:18:17.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.793+0000 7fba84cfb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6680 0x563526ae2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.793+0000 7fba84cfb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ad6fa0 0x563526ae3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.793+0000 7fba84cfb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ae5d50 0x563526ae38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.794+0000 7fba84cfb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563526ae88c0 con 0x563526ae5d50 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.794+0000 7fba84cfb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5635269c46e0 con 0x563526ad6680 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.794+0000 7fba84cfb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563526ae8a40 con 0x563526ad6fa0 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.794+0000 7fba84cfb740 10 client.0 _ll_register_callbacks cb 0x5635269e8a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.794+0000 7fba80ebf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ae5d50 0x563526ae38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.794+0000 7fba80ebf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ae5d50 0x563526ae38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57004/0 (socket says 172.21.15.33:57004) 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.794+0000 7fba80ebf640 1 -- 172.21.15.33:0/4059154035 learned_addr learned my addr 172.21.15.33:0/4059154035 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.794+0000 7fba7b7fe640 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ad6fa0 0x563526ae3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.794+0000 7fba7bfff640 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6680 0x563526ae2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.794+0000 7fba80ebf640 1 -- 172.21.15.33:0/4059154035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ad6fa0 msgr2=0x563526ae3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.794+0000 7fba80ebf640 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ad6fa0 0x563526ae3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.794+0000 7fba80ebf640 1 -- 172.21.15.33:0/4059154035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6680 msgr2=0x563526ae2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:17.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.795+0000 7fba80ebf640 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6680 0x563526ae2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.795+0000 7fba80ebf640 1 -- 172.21.15.33:0/4059154035 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563526adc3d0 con 0x563526ae5d50 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.795+0000 7fba80ebf640 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ae5d50 0x563526ae38d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fba7400a010 tx=0x7fba740058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3591e74fa58b45ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.795+0000 7fba5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.795+0000 7fba5b7fe640 1 -- 172.21.15.33:0/4059154035 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74005210 con 0x563526ae5d50 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.795+0000 7fba5b7fe640 1 -- 172.21.15.33:0/4059154035 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fba740049c0 con 0x563526ae5d50 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.795+0000 7fba84cfb740 1 -- 172.21.15.33:0/4059154035 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba74005500 con 0x563526ae5d50 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.796+0000 7fba5b7fe640 1 -- 172.21.15.33:0/4059154035 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fba74004bc0 con 0x563526ae5d50 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.796+0000 7fba84cfb740 10 client.15486 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.796+0000 7fba84cfb740 1 -- 172.21.15.33:0/4059154035 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563526af8d90 con 0x563526ae5d50 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.796+0000 7fba5b7fe640 1 -- 172.21.15.33:0/4059154035 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba74004d60 con 0x563526ae5d50 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.796+0000 7fba5b7fe640 1 -- 172.21.15.33:0/4059154035 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fba74019670 con 0x563526ae5d50 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.796+0000 7fba84cfb740 10 client.15486 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:17.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.797+0000 7fba84cfb740 10 client.15486 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:18:17.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.797+0000 7fba84cfb740 20 client.15486 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:17.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.797+0000 7fba84cfb740 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563526afa1f0 0x563526afc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.797+0000 7fba84cfb740 4 client.15486 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:18:17.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.797+0000 7fba84cfb740 1 -- 172.21.15.33:0/4059154035 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fba74005500 con 0x563526afa1f0 2024-09-16T00:18:17.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.797+0000 7fba7bfff640 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563526afa1f0 0x563526afc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.797+0000 7fba7bfff640 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563526afa1f0 0x563526afc5e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d6359bf75a5a1b6 server_cookie=ec80e895544b8072 in_seq=0 out_seq=0 2024-09-16T00:18:17.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.797+0000 7fba5b7fe640 10 client.15486 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:17.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.798+0000 7fba5b7fe640 1 -- 172.21.15.33:0/4059154035 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fba74005500 con 0x563526afa1f0 2024-09-16T00:18:17.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.798+0000 7fba5b7fe640 10 client.15486 handle_command_reply: tid=0 2024-09-16T00:18:17.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.798+0000 7fba84cfb740 1 client.15486 shutdown 2024-09-16T00:18:17.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.798+0000 7fba84cfb740 1 -- 172.21.15.33:0/4059154035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ae5d50 msgr2=0x563526ae38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:17.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.798+0000 7fba84cfb740 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ae5d50 0x563526ae38d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fba7400a010 tx=0x7fba740058e0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.798+0000 7fba84cfb740 1 -- 172.21.15.33:0/4059154035 shutdown_connections 2024-09-16T00:18:17.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.799+0000 7fba84cfb740 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563526afa1f0 0x563526afc5e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.799+0000 7fba84cfb740 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ae5d50 0x563526ae38d0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.799+0000 7fba84cfb740 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ad6fa0 0x563526ae3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.799+0000 7fba84cfb740 1 --2- 172.21.15.33:0/4059154035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6680 0x563526ae2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.799+0000 7fba5b7fe640 0 client.15486 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:17.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.799+0000 7fba84cfb740 1 -- 172.21.15.33:0/4059154035 >> 172.21.15.33:0/4059154035 conn(0x563526ad2cf0 msgr2=0x563526ad8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:17.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.799+0000 7fba84cfb740 1 -- 172.21.15.33:0/4059154035 shutdown_connections 2024-09-16T00:18:17.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.799+0000 7fba84cfb740 1 -- 172.21.15.33:0/4059154035 wait complete. 2024-09-16T00:18:17.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.799+0000 7fba84cfb740 20 client.15486 trim_cache size 0 max 16384 2024-09-16T00:18:17.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.817+0000 7fba84cfb740 1 Processor -- start 2024-09-16T00:18:17.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.817+0000 7fba84cfb740 1 -- start start 2024-09-16T00:18:17.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba84cfb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6680 0x563526b01ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba84cfb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6fa0 0x563526b021e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba84cfb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ae5d50 0x563526ae0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba84cfb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fba7c1af7d0 con 0x563526ad6680 2024-09-16T00:18:17.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba84cfb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fba7c1ad0a0 con 0x563526ad6fa0 2024-09-16T00:18:17.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba84cfb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fba7c1aaef0 con 0x563526ae5d50 2024-09-16T00:18:17.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba7b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6fa0 0x563526b021e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6680 0x563526b01ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba7b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6fa0 0x563526b021e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:44094/0 (socket says 172.21.15.33:44094) 2024-09-16T00:18:17.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba7b7fe640 1 -- 172.21.15.33:0/2504410313 learned_addr learned my addr 172.21.15.33:0/2504410313 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:17.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba80ebf640 1 --2- 172.21.15.33:0/2504410313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ae5d50 0x563526ae0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba7b7fe640 1 -- 172.21.15.33:0/2504410313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ae5d50 msgr2=0x563526ae0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:17.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba7b7fe640 1 --2- 172.21.15.33:0/2504410313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ae5d50 0x563526ae0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba7b7fe640 1 -- 172.21.15.33:0/2504410313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6680 msgr2=0x563526b01ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:17.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba7b7fe640 1 --2- 172.21.15.33:0/2504410313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6680 0x563526b01ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.818+0000 7fba7b7fe640 1 -- 172.21.15.33:0/2504410313 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563526ad6dd0 con 0x563526ad6fa0 2024-09-16T00:18:17.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.819+0000 7fba7b7fe640 1 --2- 172.21.15.33:0/2504410313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6fa0 0x563526b021e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fba700cd1a0 tx=0x7fba70005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9b25a2631b3326c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.819+0000 7fba61ffb640 1 -- 172.21.15.33:0/2504410313 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba70020040 con 0x563526ad6fa0 2024-09-16T00:18:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.819+0000 7fba61ffb640 1 -- 172.21.15.33:0/2504410313 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba700a0030 con 0x563526ad6fa0 2024-09-16T00:18:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.819+0000 7fba61ffb640 1 -- 172.21.15.33:0/2504410313 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7009b330 con 0x563526ad6fa0 2024-09-16T00:18:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.820+0000 7fba84cfb740 1 -- 172.21.15.33:0/2504410313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6fa0 msgr2=0x563526b021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.820+0000 7fba84cfb740 1 --2- 172.21.15.33:0/2504410313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6fa0 0x563526b021e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fba700cd1a0 tx=0x7fba70005880 comp rx=0 tx=0).stop 2024-09-16T00:18:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.820+0000 7fba84cfb740 1 -- 172.21.15.33:0/2504410313 shutdown_connections 2024-09-16T00:18:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.820+0000 7fba84cfb740 1 --2- 172.21.15.33:0/2504410313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ae5d50 0x563526ae0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.820+0000 7fba84cfb740 1 --2- 172.21.15.33:0/2504410313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ad6fa0 0x563526b021e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.820+0000 7fba84cfb740 1 --2- 172.21.15.33:0/2504410313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6680 0x563526b01ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.820+0000 7fba84cfb740 1 -- 172.21.15.33:0/2504410313 >> 172.21.15.33:0/2504410313 conn(0x563526ad2cf0 msgr2=0x563526ae4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.820+0000 7fba84cfb740 1 -- 172.21.15.33:0/2504410313 shutdown_connections 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.820+0000 7fba84cfb740 1 -- 172.21.15.33:0/2504410313 wait complete. 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.820+0000 7fba84cfb740 1 -- start start 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.820+0000 7fba84cfb740 1 Processor -- start 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.821+0000 7fba84cfb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ad6680 0x563526af6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.821+0000 7fba84cfb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6fa0 0x563526af6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.821+0000 7fba84cfb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ae5d50 0x563526af50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.821+0000 7fba84cfb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fba7c1ab560 con 0x563526ad6fa0 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.821+0000 7fba7b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6fa0 0x563526af6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.821+0000 7fba84cfb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fba7c1ac9d0 con 0x563526ae5d50 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.821+0000 7fba84cfb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fba7c1abc30 con 0x563526ad6680 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.821+0000 7fba7b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6fa0 0x563526af6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57024/0 (socket says 172.21.15.33:57024) 2024-09-16T00:18:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.821+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ad6680 0x563526af6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.821+0000 7fba7b7fe640 1 -- 172.21.15.33:0/324850865 learned_addr learned my addr 172.21.15.33:0/324850865 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.821+0000 7fba80ebf640 1 --2- 172.21.15.33:0/324850865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ae5d50 0x563526af50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba84cfb740 10 client.0 _ll_register_callbacks cb 0x7fba7400ff00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba7b7fe640 1 -- 172.21.15.33:0/324850865 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ad6680 msgr2=0x563526af6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba7b7fe640 1 --2- 172.21.15.33:0/324850865 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ad6680 0x563526af6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba7b7fe640 1 -- 172.21.15.33:0/324850865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ae5d50 msgr2=0x563526af50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba7b7fe640 1 --2- 172.21.15.33:0/324850865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ae5d50 0x563526af50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba7b7fe640 1 -- 172.21.15.33:0/324850865 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563526ace260 con 0x563526ad6fa0 2024-09-16T00:18:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba84cfb740 1 -- 172.21.15.33:0/324850865 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba700a0030 con 0x563526ad6fa0 2024-09-16T00:18:17.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba7b7fe640 1 --2- 172.21.15.33:0/324850865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6fa0 0x563526af6fd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fba7009bc10 tx=0x7fba70098520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42cf2927d1f2670b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:17.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba5b7fe640 10 client.15492 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:18:17.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba5b7fe640 1 -- 172.21.15.33:0/324850865 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba70065050 con 0x563526ad6fa0 2024-09-16T00:18:17.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba5b7fe640 1 -- 172.21.15.33:0/324850865 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fba7009b2e0 con 0x563526ad6fa0 2024-09-16T00:18:17.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba5b7fe640 1 -- 172.21.15.33:0/324850865 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fba70014030 con 0x563526ad6fa0 2024-09-16T00:18:17.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba84cfb740 10 client.15492 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:17.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.822+0000 7fba84cfb740 1 -- 172.21.15.33:0/324850865 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({fsmap=26}) -- 0x563526b65630 con 0x563526ad6fa0 2024-09-16T00:18:17.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.823+0000 7fba5b7fe640 1 -- 172.21.15.33:0/324850865 <== mon.0 v2:172.21.15.33:3300/0 4 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fba700979d0 con 0x563526ad6fa0 2024-09-16T00:18:17.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.823+0000 7fba84cfb740 10 client.15492 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:17.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.823+0000 7fba84cfb740 10 client.15492 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:18:17.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.823+0000 7fba84cfb740 20 client.15492 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:17.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.823+0000 7fba84cfb740 1 --2- 172.21.15.33:0/324850865 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563526b65bb0 0x563526b67fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:17.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.823+0000 7fba84cfb740 4 client.15492 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.2, "await_for": 45.0}] 2024-09-16T00:18:17.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.823+0000 7fba84cfb740 1 -- 172.21.15.33:0/324850865 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.2, "await_for": 45.0}) -- 0x563526a499d0 con 0x563526b65bb0 2024-09-16T00:18:17.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.823+0000 7fba7bfff640 1 --2- 172.21.15.33:0/324850865 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563526b65bb0 0x563526b67fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:17.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.824+0000 7fba7bfff640 1 --2- 172.21.15.33:0/324850865 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563526b65bb0 0x563526b67fa0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9bd490f4d1d7cd1 server_cookie=af0e5d7df401412d in_seq=0 out_seq=0 2024-09-16T00:18:17.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.824+0000 7fba5b7fe640 10 client.15492 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:17.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.833+0000 7fba5b7fe640 1 -- 172.21.15.33:0/324850865 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x563526a499d0 con 0x563526b65bb0 2024-09-16T00:18:17.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.833+0000 7fba5b7fe640 10 client.15492 handle_command_reply: tid=0 2024-09-16T00:18:17.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.833+0000 7fba84cfb740 1 client.15492 shutdown 2024-09-16T00:18:17.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.834+0000 7fba84cfb740 1 -- 172.21.15.33:0/324850865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6fa0 msgr2=0x563526af6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:17.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.834+0000 7fba84cfb740 1 --2- 172.21.15.33:0/324850865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6fa0 0x563526af6fd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fba7009bc10 tx=0x7fba70098520 comp rx=0 tx=0).stop 2024-09-16T00:18:17.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.834+0000 7fba84cfb740 1 -- 172.21.15.33:0/324850865 shutdown_connections 2024-09-16T00:18:17.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.834+0000 7fba84cfb740 1 --2- 172.21.15.33:0/324850865 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563526b65bb0 0x563526b67fa0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.834+0000 7fba84cfb740 1 --2- 172.21.15.33:0/324850865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563526ae5d50 0x563526af50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.834+0000 7fba84cfb740 1 --2- 172.21.15.33:0/324850865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563526ad6fa0 0x563526af6fd0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.834+0000 7fba84cfb740 1 --2- 172.21.15.33:0/324850865 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563526ad6680 0x563526af6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.834+0000 7fba5b7fe640 0 client.15492 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:17.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.834+0000 7fba84cfb740 1 -- 172.21.15.33:0/324850865 >> 172.21.15.33:0/324850865 conn(0x563526ad2cf0 msgr2=0x563526ad1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:17.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.834+0000 7fba84cfb740 1 -- 172.21.15.33:0/324850865 shutdown_connections 2024-09-16T00:18:17.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.834+0000 7fba84cfb740 1 -- 172.21.15.33:0/324850865 wait complete. 2024-09-16T00:18:17.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.834+0000 7fba84cfb740 20 client.15492 trim_cache size 0 max 16384 2024-09-16T00:18:17.836 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:18:17.836 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 26, 2024-09-16T00:18:17.836 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:18:17.836 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 2, 2024-09-16T00:18:17.836 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:18:17.836 INFO:teuthology.orchestra.run.smithi033.stdout: "18ba3a83": { 2024-09-16T00:18:17.836 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 2, 2024-09-16T00:18:17.836 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 150.2, 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:18:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:18:17.838 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:18:17.838 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:18:17.838 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:18:17.838 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:18:17.838 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:18:17.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.835+0000 7fba82922640 1 -- 172.21.15.33:0/2185157212 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fba640a32d0 msgr2=0x7fba640a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:17.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.835+0000 7fba82922640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fba640a32d0 0x7fba640a5790 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fba6c005c40 tx=0x7fba6c031400 comp rx=0 tx=0).stop 2024-09-16T00:18:17.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.835+0000 7fba82922640 1 -- 172.21.15.33:0/2185157212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba7c102570 msgr2=0x7fba7c198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:17.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.835+0000 7fba82922640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba7c102570 0x7fba7c198a80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fba7000d950 tx=0x7fba7000de10 comp rx=0 tx=0).stop 2024-09-16T00:18:17.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.835+0000 7fba82922640 1 -- 172.21.15.33:0/2185157212 shutdown_connections 2024-09-16T00:18:17.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.835+0000 7fba82922640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fba640a32d0 0x7fba640a5790 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.835+0000 7fba82922640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba7c102ef0 0x7fba7c19ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.835+0000 7fba82922640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba7c102570 0x7fba7c198a80 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.835+0000 7fba82922640 1 --2- 172.21.15.33:0/2185157212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba7c100e20 0x7fba7c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:17.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.835+0000 7fba82922640 1 -- 172.21.15.33:0/2185157212 >> 172.21.15.33:0/2185157212 conn(0x7fba7c0fc7d0 msgr2=0x7fba7c105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:17.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.835+0000 7fba82922640 1 -- 172.21.15.33:0/2185157212 shutdown_connections 2024-09-16T00:18:17.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:17.835+0000 7fba82922640 1 -- 172.21.15.33:0/2185157212 wait complete. 2024-09-16T00:18:17.847 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:18:18.901 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '18ba3a83', quiesce took 6.9 seconds. Will release after: 23.299999999999997 2024-09-16T00:18:19.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:19 smithi033 ceph-mon[31243]: pgmap v385: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:18:19.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.124+0000 7fe33829e640 1 -- 172.21.15.33:0/1793574454 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe330109120 msgr2=0x7fe330110ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:19.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.124+0000 7fe33829e640 1 --2- 172.21.15.33:0/1793574454 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe330109120 0x7fe330110ca0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe328009ca0 tx=0x7fe32802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:18:19.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.124+0000 7fe33829e640 1 -- 172.21.15.33:0/1793574454 shutdown_connections 2024-09-16T00:18:19.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.124+0000 7fe33829e640 1 --2- 172.21.15.33:0/1793574454 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe330109120 0x7fe330110ca0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:19.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.124+0000 7fe33829e640 1 --2- 172.21.15.33:0/1793574454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3301049a0 0x7fe3301089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:19.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.124+0000 7fe33829e640 1 --2- 172.21.15.33:0/1793574454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe330103ff0 0x7fe3301043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:19.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.124+0000 7fe33829e640 1 -- 172.21.15.33:0/1793574454 >> 172.21.15.33:0/1793574454 conn(0x7fe3300ffab0 msgr2=0x7fe330101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:19.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.124+0000 7fe33829e640 1 -- 172.21.15.33:0/1793574454 shutdown_connections 2024-09-16T00:18:19.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.125+0000 7fe33829e640 1 -- 172.21.15.33:0/1793574454 wait complete. 2024-09-16T00:18:19.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.125+0000 7fe33829e640 1 Processor -- start 2024-09-16T00:18:19.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe33829e640 1 -- start start 2024-09-16T00:18:19.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe33829e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe330103ff0 0x7fe33019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:19.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe33829e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3301049a0 0x7fe33019ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:19.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe33829e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe330109120 0x7fe33019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:19.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe33829e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe330114130 con 0x7fe3301049a0 2024-09-16T00:18:19.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe33829e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe330113fb0 con 0x7fe330103ff0 2024-09-16T00:18:19.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe33829e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe3301142b0 con 0x7fe330109120 2024-09-16T00:18:19.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe335839640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3301049a0 0x7fe33019ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:19.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe335839640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3301049a0 0x7fe33019ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57044/0 (socket says 172.21.15.33:57044) 2024-09-16T00:18:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe335839640 1 -- 172.21.15.33:0/1946078787 learned_addr learned my addr 172.21.15.33:0/1946078787 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe33683b640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe330109120 0x7fe33019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe33603a640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe330103ff0 0x7fe33019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe335839640 1 -- 172.21.15.33:0/1946078787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe330109120 msgr2=0x7fe33019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe335839640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe330109120 0x7fe33019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe335839640 1 -- 172.21.15.33:0/1946078787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe330103ff0 msgr2=0x7fe33019a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe335839640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe330103ff0 0x7fe33019a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe335839640 1 -- 172.21.15.33:0/1946078787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33019f5a0 con 0x7fe3301049a0 2024-09-16T00:18:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.126+0000 7fe33603a640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe330103ff0 0x7fe33019a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:18:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.127+0000 7fe335839640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3301049a0 0x7fe33019ab30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe32000e9f0 tx=0x7fe32000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.127+0000 7fe3277fe640 1 -- 172.21.15.33:0/1946078787 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32000cd80 con 0x7fe3301049a0 2024-09-16T00:18:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.127+0000 7fe3277fe640 1 -- 172.21.15.33:0/1946078787 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe320010910 con 0x7fe3301049a0 2024-09-16T00:18:19.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.127+0000 7fe3277fe640 1 -- 172.21.15.33:0/1946078787 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe320018990 con 0x7fe3301049a0 2024-09-16T00:18:19.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.127+0000 7fe33829e640 1 -- 172.21.15.33:0/1946078787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe33019f890 con 0x7fe3301049a0 2024-09-16T00:18:19.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.127+0000 7fe33829e640 1 -- 172.21.15.33:0/1946078787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3301a7170 con 0x7fe3301049a0 2024-09-16T00:18:19.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.128+0000 7fe33829e640 1 -- 172.21.15.33:0/1946078787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3301043d0 con 0x7fe3301049a0 2024-09-16T00:18:19.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.130+0000 7fe3277fe640 1 -- 172.21.15.33:0/1946078787 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fe320010ac0 con 0x7fe3301049a0 2024-09-16T00:18:19.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.131+0000 7fe3277fe640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe31c0a32d0 0x7fe31c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:19.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.132+0000 7fe33603a640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe31c0a32d0 0x7fe31c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:19.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.132+0000 7fe3277fe640 1 -- 172.21.15.33:0/1946078787 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe3200d0260 con 0x7fe3301049a0 2024-09-16T00:18:19.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.132+0000 7fe33603a640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe31c0a32d0 0x7fe31c0a5790 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe32c009c20 tx=0x7fe32c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:19.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.136+0000 7fe3277fe640 1 -- 172.21.15.33:0/1946078787 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe32009c9a0 con 0x7fe3301049a0 2024-09-16T00:18:19.282 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:19 smithi060 ceph-mon[35185]: pgmap v385: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:18:19.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.383+0000 7fe33829e640 1 -- 172.21.15.33:0/1946078787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe33010bdd0 con 0x7fe3301049a0 2024-09-16T00:18:19.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.384+0000 7fe3277fe640 1 -- 172.21.15.33:0/1946078787 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11903 (secure 0 0 0) 0x7fe32009c2a0 con 0x7fe3301049a0 2024-09-16T00:18:19.385 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:18:19.385 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":26,"btime":"2024-09-16T00:17:28:789138+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:17:28.493434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:18:19.385 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 26 2024-09-16T00:18:19.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.386+0000 7fe33829e640 1 -- 172.21.15.33:0/1946078787 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe31c0a32d0 msgr2=0x7fe31c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:19.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.386+0000 7fe33829e640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe31c0a32d0 0x7fe31c0a5790 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe32c009c20 tx=0x7fe32c005f90 comp rx=0 tx=0).stop 2024-09-16T00:18:19.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.386+0000 7fe33829e640 1 -- 172.21.15.33:0/1946078787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3301049a0 msgr2=0x7fe33019ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:19.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.386+0000 7fe33829e640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3301049a0 0x7fe33019ab30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe32000e9f0 tx=0x7fe32000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:18:19.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.387+0000 7fe33829e640 1 -- 172.21.15.33:0/1946078787 shutdown_connections 2024-09-16T00:18:19.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.387+0000 7fe33829e640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe31c0a32d0 0x7fe31c0a5790 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:19.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.387+0000 7fe33829e640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe330109120 0x7fe33019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:19.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.387+0000 7fe33829e640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3301049a0 0x7fe33019ab30 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:19.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.387+0000 7fe33829e640 1 --2- 172.21.15.33:0/1946078787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe330103ff0 0x7fe33019a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:19.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.387+0000 7fe33829e640 1 -- 172.21.15.33:0/1946078787 >> 172.21.15.33:0/1946078787 conn(0x7fe3300ffab0 msgr2=0x7fe330100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:19.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.387+0000 7fe33829e640 1 -- 172.21.15.33:0/1946078787 shutdown_connections 2024-09-16T00:18:19.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:19.387+0000 7fe33829e640 1 -- 172.21.15.33:0/1946078787 wait complete. 2024-09-16T00:18:19.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:19 smithi083 ceph-mon[35242]: pgmap v385: 129 pgs: 129 active+clean; 658 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T00:18:20.230 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:18:20.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:20 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1946078787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:18:20.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:20 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1946078787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:18:20.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:20 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1946078787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:18:20.502 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:18:21.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.030+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/4060415513 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ebc100e20 msgr2=0x7f8ebc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:21.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.030+0000 7f8ec1f7a640 1 --2- 172.21.15.33:0/4060415513 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ebc100e20 0x7f8ebc101200 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8eac009900 tx=0x7f8eac02a480 comp rx=0 tx=0).stop 2024-09-16T00:18:21.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.030+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/4060415513 shutdown_connections 2024-09-16T00:18:21.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.030+0000 7f8ec1f7a640 1 --2- 172.21.15.33:0/4060415513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ebc102ef0 0x7f8ebc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.030+0000 7f8ec1f7a640 1 --2- 172.21.15.33:0/4060415513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc102570 0x7f8ebc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.030+0000 7f8ec1f7a640 1 --2- 172.21.15.33:0/4060415513 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ebc100e20 0x7f8ebc101200 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.030+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/4060415513 >> 172.21.15.33:0/4060415513 conn(0x7f8ebc0fc7d0 msgr2=0x7f8ebc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:21.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.030+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/4060415513 shutdown_connections 2024-09-16T00:18:21.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.030+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/4060415513 wait complete. 2024-09-16T00:18:21.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.031+0000 7f8ec1f7a640 1 Processor -- start 2024-09-16T00:18:21.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.031+0000 7f8ec1f7a640 1 -- start start 2024-09-16T00:18:21.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ec1f7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc100e20 0x7f8ebc198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ec1f7a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ebc102570 0x7f8ebc198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ec1f7a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ebc102ef0 0x7f8ebc19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ec1f7a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc110f80 con 0x7f8ebc102570 2024-09-16T00:18:21.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ec1f7a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc110e00 con 0x7f8ebc102ef0 2024-09-16T00:18:21.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ec1f7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc111100 con 0x7f8ebc100e20 2024-09-16T00:18:21.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc100e20 0x7f8ebc198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ebc102570 0x7f8ebc198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ebc102ef0 0x7f8ebc19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc100e20 0x7f8ebc198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37810/0 (socket says 172.21.15.33:37810) 2024-09-16T00:18:21.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ebc102570 0x7f8ebc198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55910/0 (socket says 172.21.15.33:55910) 2024-09-16T00:18:21.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebb7fe640 1 -- 172.21.15.33:0/3558034390 learned_addr learned my addr 172.21.15.33:0/3558034390 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:21.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebb7fe640 1 -- 172.21.15.33:0/3558034390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ebc102ef0 msgr2=0x7f8ebc19ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:21.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ebc102ef0 0x7f8ebc19ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebb7fe640 1 -- 172.21.15.33:0/3558034390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ebc102570 msgr2=0x7f8ebc198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:21.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ebc102570 0x7f8ebc198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebb7fe640 1 -- 172.21.15.33:0/3558034390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ebc19d4e0 con 0x7f8ebc100e20 2024-09-16T00:18:21.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebaffd640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ebc102570 0x7f8ebc198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:21.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.032+0000 7f8ebbfff640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ebc102ef0 0x7f8ebc19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:21.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.033+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc100e20 0x7f8ebc198530 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8eac006010 tx=0x7f8eac003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:21.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.035+0000 7f8eb8ff9640 1 -- 172.21.15.33:0/3558034390 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eac03f070 con 0x7f8ebc100e20 2024-09-16T00:18:21.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.035+0000 7f8eb8ff9640 1 -- 172.21.15.33:0/3558034390 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eac031c30 con 0x7f8ebc100e20 2024-09-16T00:18:21.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.035+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/3558034390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ebc19d770 con 0x7f8ebc100e20 2024-09-16T00:18:21.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.035+0000 7f8eb8ff9640 1 -- 172.21.15.33:0/3558034390 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eac035030 con 0x7f8ebc100e20 2024-09-16T00:18:21.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.035+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/3558034390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ebc0fcbb0 con 0x7f8ebc100e20 2024-09-16T00:18:21.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.037+0000 7f8eb8ff9640 1 -- 172.21.15.33:0/3558034390 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f8eac042070 con 0x7f8ebc100e20 2024-09-16T00:18:21.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.038+0000 7f8ec4353740 1 Processor -- start 2024-09-16T00:18:21.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.038+0000 7f8ec4353740 1 -- start start 2024-09-16T00:18:21.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.038+0000 7f8eb8ff9640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ea40a3350 0x7f8ea40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ec4353740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d5b40 0x560c434d4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebbfff640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ea40a3350 0x7f8ea40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ec4353740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 0x560c434d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8eb8ff9640 1 -- 172.21.15.33:0/3558034390 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8eac0f0210 con 0x7f8ebc100e20 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ec4353740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434df370 0x560c434e1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ec4353740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560c4343a2f0 con 0x560c434d45d0 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ec4353740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560c433e1cf0 con 0x560c434df370 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ec4353740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560c434362b0 con 0x560c434d5b40 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 0x560c434d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 0x560c434d4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55924/0 (socket says 172.21.15.33:55924) 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebaffd640 1 -- 172.21.15.33:0/982556555 learned_addr learned my addr 172.21.15.33:0/982556555 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/982556555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d5b40 0x560c434d4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebbfff640 1 --2- 172.21.15.33:0/982556555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434df370 0x560c434e1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebaffd640 1 -- 172.21.15.33:0/982556555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d5b40 msgr2=0x560c434d4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:21.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebaffd640 1 --2- 172.21.15.33:0/982556555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d5b40 0x560c434d4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebaffd640 1 -- 172.21.15.33:0/982556555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434df370 msgr2=0x560c434e1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:21.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebaffd640 1 --2- 172.21.15.33:0/982556555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434df370 0x560c434e1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebaffd640 1 -- 172.21.15.33:0/982556555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c433c63a0 con 0x560c434d45d0 2024-09-16T00:18:21.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebbfff640 1 --2- 172.21.15.33:0/982556555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434df370 0x560c434e1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:18:21.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.039+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/982556555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d5b40 0x560c434d4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:21.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.040+0000 7f8ebbfff640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ea40a3350 0x7f8ea40a5810 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009fd0 tx=0x7f8eb4011f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:21.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.040+0000 7f8ebaffd640 1 --2- 172.21.15.33:0/982556555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 0x560c434d4a10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8eb00096a0 tx=0x7f8eb0009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf72741b888307ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:21.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.040+0000 7f8ea1ffb640 1 -- 172.21.15.33:0/982556555 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0008810 con 0x560c434d45d0 2024-09-16T00:18:21.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.040+0000 7f8ea1ffb640 1 -- 172.21.15.33:0/982556555 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eb001f070 con 0x560c434d45d0 2024-09-16T00:18:21.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.040+0000 7f8ea1ffb640 1 -- 172.21.15.33:0/982556555 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb001adf0 con 0x560c434d45d0 2024-09-16T00:18:21.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.041+0000 7f8ec4353740 1 -- 172.21.15.33:0/982556555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 msgr2=0x560c434d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:21.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.041+0000 7f8ec4353740 1 --2- 172.21.15.33:0/982556555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 0x560c434d4a10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8eb00096a0 tx=0x7f8eb0009b60 comp rx=0 tx=0).stop 2024-09-16T00:18:21.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.041+0000 7f8ec4353740 1 -- 172.21.15.33:0/982556555 shutdown_connections 2024-09-16T00:18:21.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.041+0000 7f8ec4353740 1 --2- 172.21.15.33:0/982556555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434df370 0x560c434e1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.041+0000 7f8ec4353740 1 --2- 172.21.15.33:0/982556555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 0x560c434d4a10 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.041+0000 7f8ec4353740 1 --2- 172.21.15.33:0/982556555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d5b40 0x560c434d4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.041+0000 7f8ec4353740 1 -- 172.21.15.33:0/982556555 >> 172.21.15.33:0/982556555 conn(0x560c4349d310 msgr2=0x560c434d0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:21.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.041+0000 7f8ec4353740 1 -- 172.21.15.33:0/982556555 shutdown_connections 2024-09-16T00:18:21.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.041+0000 7f8ec4353740 1 -- 172.21.15.33:0/982556555 wait complete. 2024-09-16T00:18:21.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.041+0000 7f8ec4353740 1 -- start start 2024-09-16T00:18:21.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ec4353740 1 Processor -- start 2024-09-16T00:18:21.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ec4353740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d45d0 0x560c434ded00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ec4353740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d5b40 0x560c434dbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ec4353740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434df370 0x560c434dc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ec4353740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560c434e2070 con 0x560c434d5b40 2024-09-16T00:18:21.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ec4353740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560c433f5060 con 0x560c434df370 2024-09-16T00:18:21.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ec4353740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560c434e21f0 con 0x560c434d45d0 2024-09-16T00:18:21.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ec4353740 10 client.0 _ll_register_callbacks cb 0x560c431488c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:21.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ebaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d5b40 0x560c434dbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ebaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d5b40 0x560c434dbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55926/0 (socket says 172.21.15.33:55926) 2024-09-16T00:18:21.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ebaffd640 1 -- 172.21.15.33:0/4187221142 learned_addr learned my addr 172.21.15.33:0/4187221142 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:21.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d45d0 0x560c434ded00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ebbfff640 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434df370 0x560c434dc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.042+0000 7f8ebaffd640 1 -- 172.21.15.33:0/4187221142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d45d0 msgr2=0x560c434ded00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8ebaffd640 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d45d0 0x560c434ded00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8ebaffd640 1 -- 172.21.15.33:0/4187221142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434df370 msgr2=0x560c434dc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8ebaffd640 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434df370 0x560c434dc530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8ebaffd640 1 -- 172.21.15.33:0/4187221142 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c431ab3c0 con 0x560c434d5b40 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d45d0 0x560c434ded00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8ebaffd640 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d5b40 0x560c434dbfb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0005d00 tx=0x7f8eb0005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728af5b7b7e9eced server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8e9affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8e9affd640 1 -- 172.21.15.33:0/4187221142 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0005240 con 0x560c434d5b40 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8e9affd640 1 -- 172.21.15.33:0/4187221142 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8eb00053e0 con 0x560c434d5b40 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8ec4353740 1 -- 172.21.15.33:0/4187221142 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8eb001f070 con 0x560c434d5b40 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8e9affd640 1 -- 172.21.15.33:0/4187221142 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8eb00055e0 con 0x560c434d5b40 2024-09-16T00:18:21.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8ec4353740 10 client.15510 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.043+0000 7f8ec4353740 1 -- 172.21.15.33:0/4187221142 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560c434edaa0 con 0x560c434d5b40 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.044+0000 7f8e9affd640 1 -- 172.21.15.33:0/4187221142 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eb001f070 con 0x560c434d5b40 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.044+0000 7f8e9affd640 1 -- 172.21.15.33:0/4187221142 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8eb0004b30 con 0x560c434d5b40 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.044+0000 7f8ec4353740 10 client.15510 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.044+0000 7f8ec4353740 10 client.15510 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.044+0000 7f8ec4353740 20 client.15510 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.044+0000 7f8ec4353740 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c434eeea0 0x560c434f1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.044+0000 7f8ec4353740 4 client.15510 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.044+0000 7f8ec4353740 1 -- 172.21.15.33:0/4187221142 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c43447860 con 0x560c434eeea0 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.044+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c434eeea0 0x560c434f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.045+0000 7f8e9affd640 10 client.15510 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.045+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c434eeea0 0x560c434f1290 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c097873b36039e18 server_cookie=57ed33d34c17966d in_seq=0 out_seq=0 2024-09-16T00:18:21.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.045+0000 7f8e9affd640 1 -- 172.21.15.33:0/4187221142 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560c43447860 con 0x560c434eeea0 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.045+0000 7f8e9affd640 10 client.15510 handle_command_reply: tid=0 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.045+0000 7f8ec4353740 1 client.15510 shutdown 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.046+0000 7f8ec4353740 1 -- 172.21.15.33:0/4187221142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d5b40 msgr2=0x560c434dbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.046+0000 7f8ec4353740 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d5b40 0x560c434dbfb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0005d00 tx=0x7f8eb0005bf0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.046+0000 7f8ec4353740 1 -- 172.21.15.33:0/4187221142 shutdown_connections 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.046+0000 7f8ec4353740 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c434eeea0 0x560c434f1290 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.046+0000 7f8ec4353740 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434df370 0x560c434dc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.046+0000 7f8ec4353740 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d5b40 0x560c434dbfb0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.046+0000 7f8ec4353740 1 --2- 172.21.15.33:0/4187221142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d45d0 0x560c434ded00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.046+0000 7f8e9affd640 0 client.15510 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.046+0000 7f8ec4353740 1 -- 172.21.15.33:0/4187221142 >> 172.21.15.33:0/4187221142 conn(0x560c4349d310 msgr2=0x560c434d6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.046+0000 7f8ec4353740 1 -- 172.21.15.33:0/4187221142 shutdown_connections 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.046+0000 7f8ec4353740 1 -- 172.21.15.33:0/4187221142 wait complete. 2024-09-16T00:18:21.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.046+0000 7f8ec4353740 20 client.15510 trim_cache size 0 max 16384 2024-09-16T00:18:21.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.059+0000 7f8ec4353740 1 Processor -- start 2024-09-16T00:18:21.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.059+0000 7f8ec4353740 1 -- start start 2024-09-16T00:18:21.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.059+0000 7f8ec4353740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434d45d0 0x560c434eb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.059+0000 7f8ec4353740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d5b40 0x560c434db870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.059+0000 7f8ec4353740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434df370 0x560c434e0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.059+0000 7f8ec4353740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc1af6b0 con 0x560c434df370 2024-09-16T00:18:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.059+0000 7f8ec4353740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc1acf80 con 0x560c434d45d0 2024-09-16T00:18:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.059+0000 7f8ec4353740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc1aadd0 con 0x560c434d5b40 2024-09-16T00:18:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.059+0000 7f8ebbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434df370 0x560c434e0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.059+0000 7f8ebbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434df370 0x560c434e0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55934/0 (socket says 172.21.15.33:55934) 2024-09-16T00:18:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.059+0000 7f8ebbfff640 1 -- 172.21.15.33:0/28088716 learned_addr learned my addr 172.21.15.33:0/28088716 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.059+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/28088716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434d45d0 0x560c434eb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.060+0000 7f8ebaffd640 1 --2- 172.21.15.33:0/28088716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d5b40 0x560c434db870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.060+0000 7f8ebbfff640 1 -- 172.21.15.33:0/28088716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d5b40 msgr2=0x560c434db870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.060+0000 7f8ebbfff640 1 --2- 172.21.15.33:0/28088716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d5b40 0x560c434db870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.060+0000 7f8ebbfff640 1 -- 172.21.15.33:0/28088716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434d45d0 msgr2=0x560c434eb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:21.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.060+0000 7f8ebbfff640 1 --2- 172.21.15.33:0/28088716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434d45d0 0x560c434eb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.060+0000 7f8ebbfff640 1 -- 172.21.15.33:0/28088716 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c434f1850 con 0x560c434df370 2024-09-16T00:18:21.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.060+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/28088716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434d45d0 0x560c434eb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:21.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.060+0000 7f8ebbfff640 1 --2- 172.21.15.33:0/28088716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434df370 0x560c434e0760 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4005f40 tx=0x7f8eb401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3737d173892656 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:21.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.060+0000 7f8ea17fa640 1 -- 172.21.15.33:0/28088716 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb401a040 con 0x560c434df370 2024-09-16T00:18:21.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.060+0000 7f8ea17fa640 1 -- 172.21.15.33:0/28088716 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eb4004530 con 0x560c434df370 2024-09-16T00:18:21.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.060+0000 7f8ea17fa640 1 -- 172.21.15.33:0/28088716 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4028630 con 0x560c434df370 2024-09-16T00:18:21.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.061+0000 7f8ec4353740 1 -- 172.21.15.33:0/28088716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434df370 msgr2=0x560c434e0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:21.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.061+0000 7f8ec4353740 1 --2- 172.21.15.33:0/28088716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434df370 0x560c434e0760 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4005f40 tx=0x7f8eb401f000 comp rx=0 tx=0).stop 2024-09-16T00:18:21.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.061+0000 7f8ec4353740 1 -- 172.21.15.33:0/28088716 shutdown_connections 2024-09-16T00:18:21.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.061+0000 7f8ec4353740 1 --2- 172.21.15.33:0/28088716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434df370 0x560c434e0760 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.061+0000 7f8ec4353740 1 --2- 172.21.15.33:0/28088716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434d5b40 0x560c434db870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.061+0000 7f8ec4353740 1 --2- 172.21.15.33:0/28088716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434d45d0 0x560c434eb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.061+0000 7f8ec4353740 1 -- 172.21.15.33:0/28088716 >> 172.21.15.33:0/28088716 conn(0x560c4349d310 msgr2=0x560c434db0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:21.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.061+0000 7f8ec4353740 1 -- 172.21.15.33:0/28088716 shutdown_connections 2024-09-16T00:18:21.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.061+0000 7f8ec4353740 1 -- 172.21.15.33:0/28088716 wait complete. 2024-09-16T00:18:21.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.061+0000 7f8ec4353740 1 -- start start 2024-09-16T00:18:21.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ec4353740 1 Processor -- start 2024-09-16T00:18:21.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ec4353740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 0x560c434e7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ec4353740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434d5b40 0x560c434f6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ec4353740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434df370 0x560c434f68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ec4353740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc1ab440 con 0x560c434d45d0 2024-09-16T00:18:21.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ec4353740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc1ac8b0 con 0x560c434d5b40 2024-09-16T00:18:21.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ec4353740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc1abb10 con 0x560c434df370 2024-09-16T00:18:21.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ec4353740 10 client.0 _ll_register_callbacks cb 0x7f8eac009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:21.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ebb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 0x560c434e7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ebb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 0x560c434e7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55948/0 (socket says 172.21.15.33:55948) 2024-09-16T00:18:21.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ebb7fe640 1 -- 172.21.15.33:0/1556389368 learned_addr learned my addr 172.21.15.33:0/1556389368 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:21.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ebaffd640 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434d5b40 0x560c434f6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ebbfff640 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434df370 0x560c434f68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ebb7fe640 1 -- 172.21.15.33:0/1556389368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434df370 msgr2=0x560c434f68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:21.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434df370 0x560c434f68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ebb7fe640 1 -- 172.21.15.33:0/1556389368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434d5b40 msgr2=0x560c434f6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:21.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434d5b40 0x560c434f6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.062+0000 7f8ebb7fe640 1 -- 172.21.15.33:0/1556389368 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c434d1960 con 0x560c434d45d0 2024-09-16T00:18:21.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.063+0000 7f8ebb7fe640 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 0x560c434e7bd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0bbfa0 tx=0x7f8eac005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de10db6051253fab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:21.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.063+0000 7f8e9affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:18:21.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.063+0000 7f8e9affd640 1 -- 172.21.15.33:0/1556389368 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eac049040 con 0x560c434d45d0 2024-09-16T00:18:21.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.063+0000 7f8e9affd640 1 -- 172.21.15.33:0/1556389368 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8eac03b460 con 0x560c434d45d0 2024-09-16T00:18:21.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.063+0000 7f8ec4353740 1 -- 172.21.15.33:0/1556389368 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8eb4004530 con 0x560c434d45d0 2024-09-16T00:18:21.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.063+0000 7f8e9affd640 1 -- 172.21.15.33:0/1556389368 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8eac0c4e60 con 0x560c434d45d0 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.063+0000 7f8ec4353740 10 client.15516 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.063+0000 7f8ec4353740 1 -- 172.21.15.33:0/1556389368 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560c43559a50 con 0x560c434d45d0 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.063+0000 7f8e9affd640 1 -- 172.21.15.33:0/1556389368 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eac0d50d0 con 0x560c434d45d0 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.063+0000 7f8e9affd640 1 -- 172.21.15.33:0/1556389368 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8eac07c210 con 0x560c434d45d0 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.064+0000 7f8ec4353740 10 client.15516 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.064+0000 7f8ec4353740 10 client.15516 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.064+0000 7f8ec4353740 20 client.15516 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.064+0000 7f8ec4353740 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c4355a050 0x560c4355c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.064+0000 7f8ec4353740 4 client.15516 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.064+0000 7f8ec4353740 1 -- 172.21.15.33:0/1556389368 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f8eb4004530 con 0x560c4355a050 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.064+0000 7f8ebaffd640 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c4355a050 0x560c4355c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.064+0000 7f8ebaffd640 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c4355a050 0x560c4355c470 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7252087f861170dd server_cookie=4b3298f320e5e2f9 in_seq=0 out_seq=0 2024-09-16T00:18:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.064+0000 7f8e9affd640 10 client.15516 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:18:21.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8e9affd640 1 -- 172.21.15.33:0/1556389368 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f8eb4004530 con 0x560c4355a050 2024-09-16T00:18:21.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8e9affd640 10 client.15516 handle_command_reply: tid=0 2024-09-16T00:18:21.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8ec4353740 1 client.15516 shutdown 2024-09-16T00:18:21.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8ec4353740 1 -- 172.21.15.33:0/1556389368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 msgr2=0x560c434e7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:21.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8ec4353740 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 0x560c434e7bd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0bbfa0 tx=0x7f8eac005be0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8ec4353740 1 -- 172.21.15.33:0/1556389368 shutdown_connections 2024-09-16T00:18:21.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8ec4353740 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c4355a050 0x560c4355c470 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8ec4353740 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c434df370 0x560c434f68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8e9affd640 0 client.15516 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:18:21.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8ec4353740 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c434d5b40 0x560c434f6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8ec4353740 1 --2- 172.21.15.33:0/1556389368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c434d45d0 0x560c434e7bd0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8ec4353740 1 -- 172.21.15.33:0/1556389368 >> 172.21.15.33:0/1556389368 conn(0x560c4349d310 msgr2=0x560c434dd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:21.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8ec4353740 1 -- 172.21.15.33:0/1556389368 shutdown_connections 2024-09-16T00:18:21.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8ec4353740 1 -- 172.21.15.33:0/1556389368 wait complete. 2024-09-16T00:18:21.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.065+0000 7f8ec4353740 20 client.15516 trim_cache size 0 max 16384 2024-09-16T00:18:21.071 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:18:21.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.066+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/3558034390 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ea40a3350 msgr2=0x7f8ea40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:21.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.066+0000 7f8ec1f7a640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ea40a3350 0x7f8ea40a5810 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009fd0 tx=0x7f8eb4011f60 comp rx=0 tx=0).stop 2024-09-16T00:18:21.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.066+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/3558034390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc100e20 msgr2=0x7f8ebc198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:21.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.066+0000 7f8ec1f7a640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc100e20 0x7f8ebc198530 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8eac006010 tx=0x7f8eac003c10 comp rx=0 tx=0).stop 2024-09-16T00:18:21.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.066+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/3558034390 shutdown_connections 2024-09-16T00:18:21.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.066+0000 7f8ec1f7a640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ea40a3350 0x7f8ea40a5810 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.066+0000 7f8ec1f7a640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ebc102ef0 0x7f8ebc19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.066+0000 7f8ec1f7a640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ebc102570 0x7f8ebc198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.066+0000 7f8ec1f7a640 1 --2- 172.21.15.33:0/3558034390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc100e20 0x7f8ebc198530 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:21.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.066+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/3558034390 >> 172.21.15.33:0/3558034390 conn(0x7f8ebc0fc7d0 msgr2=0x7f8ebc06a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:21.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.066+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/3558034390 shutdown_connections 2024-09-16T00:18:21.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:21.066+0000 7f8ec1f7a640 1 -- 172.21.15.33:0/3558034390 wait complete. 2024-09-16T00:18:21.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:21 smithi033 ceph-mon[31243]: pgmap v386: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-16T00:18:21.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:21 smithi083 ceph-mon[35242]: pgmap v386: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-16T00:18:21.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:21 smithi060 ceph-mon[35185]: pgmap v386: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-16T00:18:21.968 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:18:22.306 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:18:22.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.844+0000 7f6746d31640 1 -- 172.21.15.33:0/119164999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6740102ef0 msgr2=0x7f674010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:22.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.844+0000 7f6746d31640 1 --2- 172.21.15.33:0/119164999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6740102ef0 0x7f674010da40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6738009ca0 tx=0x7f673802a950 comp rx=0 tx=0).stop 2024-09-16T00:18:22.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.844+0000 7f6746d31640 1 -- 172.21.15.33:0/119164999 shutdown_connections 2024-09-16T00:18:22.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.844+0000 7f6746d31640 1 --2- 172.21.15.33:0/119164999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6740102ef0 0x7f674010da40 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.844+0000 7f6746d31640 1 --2- 172.21.15.33:0/119164999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6740102570 0x7f67401029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.844+0000 7f6746d31640 1 --2- 172.21.15.33:0/119164999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6740100e20 0x7f6740101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.844+0000 7f6746d31640 1 -- 172.21.15.33:0/119164999 >> 172.21.15.33:0/119164999 conn(0x7f67400fc7d0 msgr2=0x7f67400febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:22.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.845+0000 7f6746d31640 1 -- 172.21.15.33:0/119164999 shutdown_connections 2024-09-16T00:18:22.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.845+0000 7f6746d31640 1 -- 172.21.15.33:0/119164999 wait complete. 2024-09-16T00:18:22.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.845+0000 7f6746d31640 1 Processor -- start 2024-09-16T00:18:22.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.845+0000 7f6746d31640 1 -- start start 2024-09-16T00:18:22.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.845+0000 7f6746d31640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6740100e20 0x7f6740198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.845+0000 7f6746d31640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6740102570 0x7f6740198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6746d31640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6740102ef0 0x7f674019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6746d31640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6740110fd0 con 0x7f6740100e20 2024-09-16T00:18:22.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6746d31640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6740110e50 con 0x7f6740102570 2024-09-16T00:18:22.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6746d31640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6740111150 con 0x7f6740102ef0 2024-09-16T00:18:22.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6744acd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6740100e20 0x7f6740198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6744acd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6740100e20 0x7f6740198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55974/0 (socket says 172.21.15.33:55974) 2024-09-16T00:18:22.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6744acd640 1 -- 172.21.15.33:0/2621547273 learned_addr learned my addr 172.21.15.33:0/2621547273 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:22.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f673ffff640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6740102570 0x7f6740198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f67452ce640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6740102ef0 0x7f674019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6744acd640 1 -- 172.21.15.33:0/2621547273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6740102ef0 msgr2=0x7f674019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:22.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6744acd640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6740102ef0 0x7f674019cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6744acd640 1 -- 172.21.15.33:0/2621547273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6740102570 msgr2=0x7f6740198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:22.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6744acd640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6740102570 0x7f6740198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6744acd640 1 -- 172.21.15.33:0/2621547273 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674019d4d0 con 0x7f6740100e20 2024-09-16T00:18:22.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f67452ce640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6740102ef0 0x7f674019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:18:22.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.846+0000 7f6744acd640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6740100e20 0x7f6740198520 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f673400e9f0 tx=0x7f673400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:22.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.847+0000 7f673dffb640 1 -- 172.21.15.33:0/2621547273 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673400cd80 con 0x7f6740100e20 2024-09-16T00:18:22.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.847+0000 7f6746d31640 1 -- 172.21.15.33:0/2621547273 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674019d7c0 con 0x7f6740100e20 2024-09-16T00:18:22.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.847+0000 7f673dffb640 1 -- 172.21.15.33:0/2621547273 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6734010910 con 0x7f6740100e20 2024-09-16T00:18:22.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.847+0000 7f6746d31640 1 -- 172.21.15.33:0/2621547273 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67401a50a0 con 0x7f6740100e20 2024-09-16T00:18:22.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.847+0000 7f673dffb640 1 -- 172.21.15.33:0/2621547273 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340188f0 con 0x7f6740100e20 2024-09-16T00:18:22.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.849+0000 7f673dffb640 1 -- 172.21.15.33:0/2621547273 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f6734010ac0 con 0x7f6740100e20 2024-09-16T00:18:22.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.850+0000 7f673dffb640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f67300a32d0 0x7f67300a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.850+0000 7f674910a740 1 Processor -- start 2024-09-16T00:18:22.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.850+0000 7f673ffff640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f67300a32d0 0x7f67300a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.850+0000 7f673dffb640 1 -- 172.21.15.33:0/2621547273 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f67340d03e0 con 0x7f6740100e20 2024-09-16T00:18:22.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.850+0000 7f674910a740 1 -- start start 2024-09-16T00:18:22.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f674910a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 0x564e7d835c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f674910a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 0x564e7d834560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f674910a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d834b80 0x564e7d849f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f674910a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564e7d79a2f0 con 0x564e7d835840 2024-09-16T00:18:22.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f674910a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564e7d741cf0 con 0x564e7d834b80 2024-09-16T00:18:22.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f674910a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564e7d7962b0 con 0x564e7d834120 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f6744acd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 0x564e7d835c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f673ffff640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f67300a32d0 0x7f67300a5790 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f67401052a0 tx=0x7f672c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f6744acd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 0x564e7d835c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55980/0 (socket says 172.21.15.33:55980) 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f6744acd640 1 -- 172.21.15.33:0/2664957871 learned_addr learned my addr 172.21.15.33:0/2664957871 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f67452ce640 1 --2- 172.21.15.33:0/2664957871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d834b80 0x564e7d849f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f6744acd640 1 -- 172.21.15.33:0/2664957871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 msgr2=0x564e7d834560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f6744acd640 1 --2- 172.21.15.33:0/2664957871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 0x564e7d834560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f6744acd640 1 -- 172.21.15.33:0/2664957871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d834b80 msgr2=0x564e7d849f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f6744acd640 1 --2- 172.21.15.33:0/2664957871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d834b80 0x564e7d849f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.851+0000 7f6744acd640 1 -- 172.21.15.33:0/2664957871 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e7d7263a0 con 0x564e7d835840 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.852+0000 7f6744acd640 1 --2- 172.21.15.33:0/2664957871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 0x564e7d835c20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6734098d90 tx=0x7f673409cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c867b631027db0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.852+0000 7f672affd640 1 -- 172.21.15.33:0/2664957871 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340a5030 con 0x564e7d835840 2024-09-16T00:18:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.852+0000 7f672affd640 1 -- 172.21.15.33:0/2664957871 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67340cdb00 con 0x564e7d835840 2024-09-16T00:18:22.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.852+0000 7f672affd640 1 -- 172.21.15.33:0/2664957871 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673409b810 con 0x564e7d835840 2024-09-16T00:18:22.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.853+0000 7f674910a740 1 -- 172.21.15.33:0/2664957871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 msgr2=0x564e7d835c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:22.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.853+0000 7f674910a740 1 --2- 172.21.15.33:0/2664957871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 0x564e7d835c20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6734098d90 tx=0x7f673409cb80 comp rx=0 tx=0).stop 2024-09-16T00:18:22.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.853+0000 7f674910a740 1 -- 172.21.15.33:0/2664957871 shutdown_connections 2024-09-16T00:18:22.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.853+0000 7f674910a740 1 --2- 172.21.15.33:0/2664957871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d834b80 0x564e7d849f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.853+0000 7f674910a740 1 --2- 172.21.15.33:0/2664957871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 0x564e7d834560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.853+0000 7f674910a740 1 --2- 172.21.15.33:0/2664957871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 0x564e7d835c20 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.853+0000 7f674910a740 1 -- 172.21.15.33:0/2664957871 >> 172.21.15.33:0/2664957871 conn(0x564e7d7fd310 msgr2=0x564e7d830c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:22.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.853+0000 7f674910a740 1 -- 172.21.15.33:0/2664957871 shutdown_connections 2024-09-16T00:18:22.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.853+0000 7f674910a740 1 -- 172.21.15.33:0/2664957871 wait complete. 2024-09-16T00:18:22.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.853+0000 7f674910a740 1 -- start start 2024-09-16T00:18:22.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.853+0000 7f674910a740 1 Processor -- start 2024-09-16T00:18:22.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.854+0000 7f674910a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d834120 0x564e7d849f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.854+0000 7f674910a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834b80 0x564e7d847240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.854+0000 7f674910a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 0x564e7d8479d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.854+0000 7f674910a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564e7d84a930 con 0x564e7d835840 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.854+0000 7f674910a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564e7d755060 con 0x564e7d834120 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.854+0000 7f674910a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564e7d84aab0 con 0x564e7d834b80 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.854+0000 7f674910a740 10 client.0 _ll_register_callbacks cb 0x564e7d4a88c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.854+0000 7f67452ce640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 0x564e7d8479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.854+0000 7f67452ce640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 0x564e7d8479d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55984/0 (socket says 172.21.15.33:55984) 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.854+0000 7f67452ce640 1 -- 172.21.15.33:0/899935168 learned_addr learned my addr 172.21.15.33:0/899935168 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.854+0000 7f673ffff640 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834b80 0x564e7d847240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.854+0000 7f6744acd640 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d834120 0x564e7d849f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f67452ce640 1 -- 172.21.15.33:0/899935168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834b80 msgr2=0x564e7d847240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f67452ce640 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834b80 0x564e7d847240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f67452ce640 1 -- 172.21.15.33:0/899935168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d834120 msgr2=0x564e7d849f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f67452ce640 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d834120 0x564e7d849f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f67452ce640 1 -- 172.21.15.33:0/899935168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e7d50b3c0 con 0x564e7d835840 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f67452ce640 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 0x564e7d8479d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6738035000 tx=0x7f6738003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca62f06d21a4a698 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f671ffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f671ffff640 1 -- 172.21.15.33:0/899935168 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6738034040 con 0x564e7d835840 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f671ffff640 1 -- 172.21.15.33:0/899935168 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f673802f930 con 0x564e7d835840 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f674910a740 1 -- 172.21.15.33:0/899935168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67340cdb00 con 0x564e7d835840 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f671ffff640 1 -- 172.21.15.33:0/899935168 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f673804f050 con 0x564e7d835840 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f674910a740 10 client.15528 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.855+0000 7f674910a740 1 -- 172.21.15.33:0/899935168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564e7d8564d0 con 0x564e7d835840 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.856+0000 7f671ffff640 1 -- 172.21.15.33:0/899935168 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67340cdb00 con 0x564e7d835840 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.856+0000 7f671ffff640 1 -- 172.21.15.33:0/899935168 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f6738030440 con 0x564e7d835840 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.856+0000 7f674910a740 10 client.15528 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.856+0000 7f674910a740 10 client.15528 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.856+0000 7f674910a740 20 client.15528 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:22.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.856+0000 7f674910a740 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e7d857ce0 0x564e7d85a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.856+0000 7f674910a740 4 client.15528 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.856+0000 7f674910a740 1 -- 172.21.15.33:0/899935168 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e7d7a7860 con 0x564e7d857ce0 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.857+0000 7f6744acd640 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e7d857ce0 0x564e7d85a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.857+0000 7f671ffff640 10 client.15528 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.857+0000 7f6744acd640 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e7d857ce0 0x564e7d85a120 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6d63712601a6df6f server_cookie=b67b23f8c3039959 in_seq=0 out_seq=0 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.858+0000 7f671ffff640 1 -- 172.21.15.33:0/899935168 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564e7d7a7860 con 0x564e7d857ce0 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.858+0000 7f671ffff640 10 client.15528 handle_command_reply: tid=0 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.858+0000 7f674910a740 1 client.15528 shutdown 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.858+0000 7f674910a740 1 -- 172.21.15.33:0/899935168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 msgr2=0x564e7d8479d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.858+0000 7f674910a740 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 0x564e7d8479d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6738035000 tx=0x7f6738003ba0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.858+0000 7f674910a740 1 -- 172.21.15.33:0/899935168 shutdown_connections 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.858+0000 7f674910a740 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d835840 0x564e7d8479d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.858+0000 7f674910a740 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834b80 0x564e7d847240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.858+0000 7f674910a740 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e7d857ce0 0x564e7d85a120 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.858+0000 7f674910a740 1 --2- 172.21.15.33:0/899935168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d834120 0x564e7d849f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.858+0000 7f674910a740 1 -- 172.21.15.33:0/899935168 >> 172.21.15.33:0/899935168 conn(0x564e7d7fd310 msgr2=0x564e7d82fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:22.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.858+0000 7f671ffff640 0 client.15528 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:18:22.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.859+0000 7f674910a740 1 -- 172.21.15.33:0/899935168 shutdown_connections 2024-09-16T00:18:22.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.859+0000 7f674910a740 1 -- 172.21.15.33:0/899935168 wait complete. 2024-09-16T00:18:22.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.859+0000 7f674910a740 20 client.15528 trim_cache size 0 max 16384 2024-09-16T00:18:22.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.872+0000 7f674910a740 1 Processor -- start 2024-09-16T00:18:22.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.872+0000 7f674910a740 1 -- start start 2024-09-16T00:18:22.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.872+0000 7f674910a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 0x564e7d847210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.872+0000 7f674910a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 0x564e7d836c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.872+0000 7f674910a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d835840 0x564e7d85f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.872+0000 7f674910a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f67401af890 con 0x564e7d834b80 2024-09-16T00:18:22.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.872+0000 7f674910a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f67401ad160 con 0x564e7d835840 2024-09-16T00:18:22.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.872+0000 7f674910a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f67401aafb0 con 0x564e7d834120 2024-09-16T00:18:22.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.872+0000 7f673ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 0x564e7d836c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.872+0000 7f673ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 0x564e7d836c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55994/0 (socket says 172.21.15.33:55994) 2024-09-16T00:18:22.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.872+0000 7f673ffff640 1 -- 172.21.15.33:0/2962329798 learned_addr learned my addr 172.21.15.33:0/2962329798 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:22.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.873+0000 7f67452ce640 1 --2- 172.21.15.33:0/2962329798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d835840 0x564e7d85f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.873+0000 7f673ffff640 1 -- 172.21.15.33:0/2962329798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 msgr2=0x564e7d847210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:18:22.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.873+0000 7f673ffff640 1 --2- 172.21.15.33:0/2962329798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 0x564e7d847210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.873+0000 7f673ffff640 1 -- 172.21.15.33:0/2962329798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d835840 msgr2=0x564e7d85f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:22.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.873+0000 7f673ffff640 1 --2- 172.21.15.33:0/2962329798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d835840 0x564e7d85f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.873+0000 7f673ffff640 1 -- 172.21.15.33:0/2962329798 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e7d85a7a0 con 0x564e7d834b80 2024-09-16T00:18:22.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.873+0000 7f67452ce640 1 --2- 172.21.15.33:0/2962329798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d835840 0x564e7d85f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:18:22.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.873+0000 7f673ffff640 1 --2- 172.21.15.33:0/2962329798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 0x564e7d836c30 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f672c011e50 tx=0x7f672c013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25eee78f378dc68b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:22.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.873+0000 7f672a7fc640 1 -- 172.21.15.33:0/2962329798 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c003cf0 con 0x564e7d834b80 2024-09-16T00:18:22.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.873+0000 7f672a7fc640 1 -- 172.21.15.33:0/2962329798 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f672c0044d0 con 0x564e7d834b80 2024-09-16T00:18:22.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.874+0000 7f672a7fc640 1 -- 172.21.15.33:0/2962329798 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c005700 con 0x564e7d834b80 2024-09-16T00:18:22.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.874+0000 7f674910a740 1 -- 172.21.15.33:0/2962329798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 msgr2=0x564e7d836c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.874+0000 7f674910a740 1 --2- 172.21.15.33:0/2962329798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 0x564e7d836c30 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f672c011e50 tx=0x7f672c013040 comp rx=0 tx=0).stop 2024-09-16T00:18:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.875+0000 7f674910a740 1 -- 172.21.15.33:0/2962329798 shutdown_connections 2024-09-16T00:18:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.875+0000 7f674910a740 1 --2- 172.21.15.33:0/2962329798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d835840 0x564e7d85f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.875+0000 7f674910a740 1 --2- 172.21.15.33:0/2962329798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 0x564e7d836c30 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.875+0000 7f674910a740 1 --2- 172.21.15.33:0/2962329798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 0x564e7d847210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.875+0000 7f674910a740 1 -- 172.21.15.33:0/2962329798 >> 172.21.15.33:0/2962329798 conn(0x564e7d7fd310 msgr2=0x564e7d846b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.875+0000 7f674910a740 1 -- 172.21.15.33:0/2962329798 shutdown_connections 2024-09-16T00:18:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.875+0000 7f674910a740 1 -- 172.21.15.33:0/2962329798 wait complete. 2024-09-16T00:18:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.875+0000 7f674910a740 1 -- start start 2024-09-16T00:18:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.875+0000 7f674910a740 1 Processor -- start 2024-09-16T00:18:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.876+0000 7f674910a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 0x564e7d85efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.876+0000 7f674910a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 0x564e7d85f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.876+0000 7f6744acd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 0x564e7d85efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.876+0000 7f6744acd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 0x564e7d85efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37916/0 (socket says 172.21.15.33:37916) 2024-09-16T00:18:22.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.876+0000 7f673ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 0x564e7d85f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.876+0000 7f673ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 0x564e7d85f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56000/0 (socket says 172.21.15.33:56000) 2024-09-16T00:18:22.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.876+0000 7f6744acd640 1 -- 172.21.15.33:0/427181172 learned_addr learned my addr 172.21.15.33:0/427181172 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:22.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.876+0000 7f674910a740 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d835840 0x564e7d84ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.876+0000 7f674910a740 1 -- 172.21.15.33:0/427181172 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f67401ab620 con 0x564e7d834b80 2024-09-16T00:18:22.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.876+0000 7f674910a740 1 -- 172.21.15.33:0/427181172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f67401aca90 con 0x564e7d835840 2024-09-16T00:18:22.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.876+0000 7f674910a740 1 -- 172.21.15.33:0/427181172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f67401abcf0 con 0x564e7d834120 2024-09-16T00:18:22.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f67452ce640 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d835840 0x564e7d84ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f67452ce640 1 -- 172.21.15.33:0/427181172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 msgr2=0x564e7d85efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:22.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f67452ce640 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 0x564e7d85efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f67452ce640 1 -- 172.21.15.33:0/427181172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 msgr2=0x564e7d85f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:22.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f67452ce640 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 0x564e7d85f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f67452ce640 1 -- 172.21.15.33:0/427181172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e7d84f300 con 0x564e7d835840 2024-09-16T00:18:22.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f6744acd640 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 0x564e7d85efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:18:22.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f673ffff640 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 0x564e7d85f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:22.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f67452ce640 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d835840 0x564e7d84ed90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f673802ae70 tx=0x7f6738030670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33fd31771780ea06 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:22.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f674910a740 10 client.0 _ll_register_callbacks cb 0x7f6734005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:22.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f671ffff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:18:22.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f674910a740 1 -- 172.21.15.33:0/427181172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f672c0044d0 con 0x564e7d835840 2024-09-16T00:18:22.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.877+0000 7f671ffff640 1 -- 172.21.15.33:0/427181172 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6738034070 con 0x564e7d835840 2024-09-16T00:18:22.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.878+0000 7f671ffff640 1 -- 172.21.15.33:0/427181172 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f673802f930 con 0x564e7d835840 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.878+0000 7f671ffff640 1 -- 172.21.15.33:0/427181172 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6738030c90 con 0x564e7d835840 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.878+0000 7f674910a740 10 client.25181 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.878+0000 7f674910a740 1 -- 172.21.15.33:0/427181172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({fsmap=26}) -- 0x564e7d8502e0 con 0x564e7d835840 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.878+0000 7f671ffff640 1 -- 172.21.15.33:0/427181172 <== mon.2 v2:172.21.15.60:3300/0 4 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f673802fb90 con 0x564e7d835840 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.878+0000 7f674910a740 10 client.25181 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.878+0000 7f674910a740 10 client.25181 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.878+0000 7f674910a740 20 client.25181 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.878+0000 7f674910a740 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e7d8c29c0 0x564e7d8c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.878+0000 7f674910a740 4 client.25181 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.878+0000 7f674910a740 1 -- 172.21.15.33:0/427181172 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e7d7a1d70 con 0x564e7d8c29c0 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.879+0000 7f6744acd640 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e7d8c29c0 0x564e7d8c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.879+0000 7f6744acd640 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e7d8c29c0 0x564e7d8c4db0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=efed2e0f3924c593 server_cookie=ce4cb2e561ba8af1 in_seq=0 out_seq=0 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.879+0000 7f671ffff640 10 client.25181 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:18:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.879+0000 7f671ffff640 1 -- 172.21.15.33:0/427181172 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e7d7a1d70 con 0x564e7d8c29c0 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.879+0000 7f671ffff640 10 client.25181 handle_command_reply: tid=0 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.879+0000 7f674910a740 1 client.25181 shutdown 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.879+0000 7f674910a740 1 -- 172.21.15.33:0/427181172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d835840 msgr2=0x564e7d84ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.879+0000 7f674910a740 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d835840 0x564e7d84ed90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f673802ae70 tx=0x7f6738030670 comp rx=0 tx=0).stop 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.879+0000 7f674910a740 1 -- 172.21.15.33:0/427181172 shutdown_connections 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.879+0000 7f674910a740 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e7d8c29c0 0x564e7d8c4db0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.879+0000 7f674910a740 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e7d835840 0x564e7d84ed90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.880+0000 7f671ffff640 0 client.25181 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.880+0000 7f674910a740 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e7d834b80 0x564e7d85f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.880+0000 7f674910a740 1 --2- 172.21.15.33:0/427181172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e7d834120 0x564e7d85efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.880+0000 7f674910a740 1 -- 172.21.15.33:0/427181172 >> 172.21.15.33:0/427181172 conn(0x564e7d7fd310 msgr2=0x564e7d849270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.880+0000 7f674910a740 1 -- 172.21.15.33:0/427181172 shutdown_connections 2024-09-16T00:18:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.880+0000 7f674910a740 1 -- 172.21.15.33:0/427181172 wait complete. 2024-09-16T00:18:22.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.880+0000 7f674910a740 20 client.25181 trim_cache size 0 max 16384 2024-09-16T00:18:22.884 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:18:22.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.881+0000 7f6746d31640 1 -- 172.21.15.33:0/2621547273 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f67300a32d0 msgr2=0x7f67300a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:22.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.881+0000 7f6746d31640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f67300a32d0 0x7f67300a5790 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f67401052a0 tx=0x7f672c005f90 comp rx=0 tx=0).stop 2024-09-16T00:18:22.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.881+0000 7f6746d31640 1 -- 172.21.15.33:0/2621547273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6740100e20 msgr2=0x7f6740198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:22.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.881+0000 7f6746d31640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6740100e20 0x7f6740198520 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f673400e9f0 tx=0x7f673400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.881+0000 7f6746d31640 1 -- 172.21.15.33:0/2621547273 shutdown_connections 2024-09-16T00:18:22.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.881+0000 7f6746d31640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f67300a32d0 0x7f67300a5790 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.881+0000 7f6746d31640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6740102ef0 0x7f674019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.881+0000 7f6746d31640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6740102570 0x7f6740198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.881+0000 7f6746d31640 1 --2- 172.21.15.33:0/2621547273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6740100e20 0x7f6740198520 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:22.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.881+0000 7f6746d31640 1 -- 172.21.15.33:0/2621547273 >> 172.21.15.33:0/2621547273 conn(0x7f67400fc7d0 msgr2=0x7f674006a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:22.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.881+0000 7f6746d31640 1 -- 172.21.15.33:0/2621547273 shutdown_connections 2024-09-16T00:18:22.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:22.881+0000 7f6746d31640 1 -- 172.21.15.33:0/2621547273 wait complete. 2024-09-16T00:18:23.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:23 smithi033 ceph-mon[31243]: pgmap v387: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-16T00:18:23.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:23 smithi083 ceph-mon[35242]: pgmap v387: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-16T00:18:23.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:23 smithi060 ceph-mon[35185]: pgmap v387: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-16T00:18:23.629 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:18:23.917 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:18:24.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.507+0000 7f70d5324640 1 -- 172.21.15.33:0/1912853356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70d01012d0 msgr2=0x7f70d0104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:24.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.507+0000 7f70d5324640 1 --2- 172.21.15.33:0/1912853356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70d01012d0 0x7f70d0104e70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f70b8009900 tx=0x7f70b802a410 comp rx=0 tx=0).stop 2024-09-16T00:18:24.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.507+0000 7f70d5324640 1 -- 172.21.15.33:0/1912853356 shutdown_connections 2024-09-16T00:18:24.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.507+0000 7f70d5324640 1 --2- 172.21.15.33:0/1912853356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70d01055a0 0x7f70d010d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.507+0000 7f70d5324640 1 --2- 172.21.15.33:0/1912853356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70d01012d0 0x7f70d0104e70 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.507+0000 7f70d5324640 1 --2- 172.21.15.33:0/1912853356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70d0100920 0x7f70d0100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.507+0000 7f70d5324640 1 -- 172.21.15.33:0/1912853356 >> 172.21.15.33:0/1912853356 conn(0x7f70d0075f70 msgr2=0x7f70d0076380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:24.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.507+0000 7f70d5324640 1 -- 172.21.15.33:0/1912853356 shutdown_connections 2024-09-16T00:18:24.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.507+0000 7f70d5324640 1 -- 172.21.15.33:0/1912853356 wait complete. 2024-09-16T00:18:24.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.508+0000 7f70d5324640 1 Processor -- start 2024-09-16T00:18:24.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.508+0000 7f70d5324640 1 -- start start 2024-09-16T00:18:24.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.508+0000 7f70d5324640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70d0100920 0x7f70d0198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.508+0000 7f70d5324640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70d01012d0 0x7f70d0198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.508+0000 7f70d5324640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70d01055a0 0x7f70d019ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.508+0000 7f70d5324640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f70d0110e20 con 0x7f70d01012d0 2024-09-16T00:18:24.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.508+0000 7f70d5324640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f70d0110ca0 con 0x7f70d01055a0 2024-09-16T00:18:24.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.508+0000 7f70d5324640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f70d0110fa0 con 0x7f70d0100920 2024-09-16T00:18:24.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.508+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70d01012d0 0x7f70d0198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.508+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70d01012d0 0x7f70d0198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56022/0 (socket says 172.21.15.33:56022) 2024-09-16T00:18:24.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.508+0000 7f70c7fff640 1 -- 172.21.15.33:0/3301038076 learned_addr learned my addr 172.21.15.33:0/3301038076 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:24.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.509+0000 7f70ceffd640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70d0100920 0x7f70d0198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.509+0000 7f70cf7fe640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70d01055a0 0x7f70d019ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.509+0000 7f70c7fff640 1 -- 172.21.15.33:0/3301038076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70d0100920 msgr2=0x7f70d0198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:24.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.509+0000 7f70c7fff640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70d0100920 0x7f70d0198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.509+0000 7f70c7fff640 1 -- 172.21.15.33:0/3301038076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70d01055a0 msgr2=0x7f70d019ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:24.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.509+0000 7f70c7fff640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70d01055a0 0x7f70d019ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.509+0000 7f70c7fff640 1 -- 172.21.15.33:0/3301038076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d019d3d0 con 0x7f70d01012d0 2024-09-16T00:18:24.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.509+0000 7f70cf7fe640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70d01055a0 0x7f70d019ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:18:24.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.509+0000 7f70c7fff640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70d01012d0 0x7f70d0198960 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f70b8005940 tx=0x7f70b80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:24.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.509+0000 7f70ccff9640 1 -- 172.21.15.33:0/3301038076 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b803f030 con 0x7f70d01012d0 2024-09-16T00:18:24.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.509+0000 7f70ccff9640 1 -- 172.21.15.33:0/3301038076 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70b80044c0 con 0x7f70d01012d0 2024-09-16T00:18:24.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.509+0000 7f70d5324640 1 -- 172.21.15.33:0/3301038076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d019d660 con 0x7f70d01012d0 2024-09-16T00:18:24.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.510+0000 7f70ccff9640 1 -- 172.21.15.33:0/3301038076 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8035070 con 0x7f70d01012d0 2024-09-16T00:18:24.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.510+0000 7f70d5324640 1 -- 172.21.15.33:0/3301038076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d01a4fa0 con 0x7f70d01012d0 2024-09-16T00:18:24.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.513+0000 7f70ccff9640 1 -- 172.21.15.33:0/3301038076 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f70b8042030 con 0x7f70d01012d0 2024-09-16T00:18:24.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.513+0000 7f70d76fd740 1 Processor -- start 2024-09-16T00:18:24.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.513+0000 7f70d76fd740 1 -- start start 2024-09-16T00:18:24.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.514+0000 7f70d76fd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5599607075d0 0x5599607079b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.514+0000 7f70d76fd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702280 0x5599607026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.514+0000 7f70d76fd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x55996070e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.514+0000 7f70d76fd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55996065e2f0 con 0x559960702c00 2024-09-16T00:18:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.514+0000 7f70d76fd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559960605cf0 con 0x5599607075d0 2024-09-16T00:18:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.514+0000 7f70d76fd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55996065a2b0 con 0x559960702280 2024-09-16T00:18:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.514+0000 7f70ceffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702280 0x5599607026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.514+0000 7f70ceffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702280 0x5599607026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37938/0 (socket says 172.21.15.33:37938) 2024-09-16T00:18:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.514+0000 7f70ceffd640 1 -- 172.21.15.33:0/4029442921 learned_addr learned my addr 172.21.15.33:0/4029442921 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.514+0000 7f70cf7fe640 1 --2- 172.21.15.33:0/4029442921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5599607075d0 0x5599607079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.514+0000 7f70c7fff640 1 --2- 172.21.15.33:0/4029442921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x55996070e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.514+0000 7f70ccff9640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f70bc0a3350 0x7f70bc0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.515+0000 7f70ceffd640 1 -- 172.21.15.33:0/4029442921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5599607075d0 msgr2=0x5599607079b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.515+0000 7f70ceffd640 1 --2- 172.21.15.33:0/4029442921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5599607075d0 0x5599607079b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.515+0000 7f70ceffd640 1 -- 172.21.15.33:0/4029442921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 msgr2=0x55996070e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.515+0000 7f70ceffd640 1 --2- 172.21.15.33:0/4029442921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x55996070e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.515+0000 7f70ceffd640 1 -- 172.21.15.33:0/4029442921 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599605ea3a0 con 0x559960702280 2024-09-16T00:18:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.515+0000 7f70c7fff640 1 --2- 172.21.15.33:0/4029442921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x55996070e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:18:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.515+0000 7f70ccff9640 1 -- 172.21.15.33:0/3301038076 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f70b80f02f0 con 0x7f70d01012d0 2024-09-16T00:18:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.515+0000 7f70cf7fe640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f70bc0a3350 0x7f70bc0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.515+0000 7f70ceffd640 1 --2- 172.21.15.33:0/4029442921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702280 0x5599607026c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f70c0009d20 tx=0x7f70c0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47aa955d81bf55a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.515+0000 7f70c57fa640 1 -- 172.21.15.33:0/4029442921 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0004230 con 0x559960702280 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.515+0000 7f70c57fa640 1 -- 172.21.15.33:0/4029442921 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70c00043d0 con 0x559960702280 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.515+0000 7f70c57fa640 1 -- 172.21.15.33:0/4029442921 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0019920 con 0x559960702280 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.516+0000 7f70d76fd740 1 -- 172.21.15.33:0/4029442921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702280 msgr2=0x5599607026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.516+0000 7f70d76fd740 1 --2- 172.21.15.33:0/4029442921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702280 0x5599607026c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f70c0009d20 tx=0x7f70c0011810 comp rx=0 tx=0).stop 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.516+0000 7f70cf7fe640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f70bc0a3350 0x7f70bc0a5810 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f70c8005fd0 tx=0x7f70c8004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.516+0000 7f70d76fd740 1 -- 172.21.15.33:0/4029442921 shutdown_connections 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.516+0000 7f70d76fd740 1 --2- 172.21.15.33:0/4029442921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x55996070e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.516+0000 7f70d76fd740 1 --2- 172.21.15.33:0/4029442921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702280 0x5599607026c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.516+0000 7f70d76fd740 1 --2- 172.21.15.33:0/4029442921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5599607075d0 0x5599607079b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.516+0000 7f70d76fd740 1 -- 172.21.15.33:0/4029442921 >> 172.21.15.33:0/4029442921 conn(0x5599606c1310 msgr2=0x5599606f46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.516+0000 7f70d76fd740 1 -- 172.21.15.33:0/4029442921 shutdown_connections 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.516+0000 7f70d76fd740 1 -- 172.21.15.33:0/4029442921 wait complete. 2024-09-16T00:18:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.517+0000 7f70d76fd740 1 -- start start 2024-09-16T00:18:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.517+0000 7f70d76fd740 1 Processor -- start 2024-09-16T00:18:24.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.517+0000 7f70d76fd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559960702280 0x55996070e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.517+0000 7f70d76fd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702c00 0x55996070b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.517+0000 7f70d76fd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599607075d0 0x55996070b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.517+0000 7f70d76fd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55996070e8d0 con 0x5599607075d0 2024-09-16T00:18:24.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70d76fd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559960619060 con 0x559960702280 2024-09-16T00:18:24.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70d76fd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55996070ea50 con 0x559960702c00 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70d76fd740 10 client.0 _ll_register_callbacks cb 0x55996036c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702c00 0x55996070b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702c00 0x55996070b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37942/0 (socket says 172.21.15.33:37942) 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70c7fff640 1 -- 172.21.15.33:0/546794417 learned_addr learned my addr 172.21.15.33:0/546794417 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70ceffd640 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559960702280 0x55996070e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70cf7fe640 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599607075d0 0x55996070b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70c7fff640 1 -- 172.21.15.33:0/546794417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559960702280 msgr2=0x55996070e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70c7fff640 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559960702280 0x55996070e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70c7fff640 1 -- 172.21.15.33:0/546794417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599607075d0 msgr2=0x55996070b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70c7fff640 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599607075d0 0x55996070b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70c7fff640 1 -- 172.21.15.33:0/546794417 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599603cf3c0 con 0x559960702c00 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70cf7fe640 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599607075d0 0x55996070b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:18:24.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70c7fff640 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702c00 0x55996070b440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f70b80b8d00 tx=0x7f70b80bc1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac214599483e26fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.518+0000 7f70b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:18:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.519+0000 7f70b67fc640 1 -- 172.21.15.33:0/546794417 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8049050 con 0x559960702c00 2024-09-16T00:18:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.519+0000 7f70d76fd740 1 -- 172.21.15.33:0/546794417 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70c00043d0 con 0x559960702c00 2024-09-16T00:18:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.519+0000 7f70b67fc640 1 -- 172.21.15.33:0/546794417 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f70b80b89a0 con 0x559960702c00 2024-09-16T00:18:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.519+0000 7f70b67fc640 1 -- 172.21.15.33:0/546794417 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f70b803b460 con 0x559960702c00 2024-09-16T00:18:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.519+0000 7f70d76fd740 10 client.25126 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.519+0000 7f70d76fd740 1 -- 172.21.15.33:0/546794417 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55996070e930 con 0x559960702c00 2024-09-16T00:18:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.519+0000 7f70b67fc640 1 -- 172.21.15.33:0/546794417 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70c00043d0 con 0x559960702c00 2024-09-16T00:18:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.519+0000 7f70b67fc640 1 -- 172.21.15.33:0/546794417 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f70b80bc590 con 0x559960702c00 2024-09-16T00:18:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.519+0000 7f70d76fd740 10 client.25126 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.519+0000 7f70d76fd740 10 client.25126 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:18:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.519+0000 7f70d76fd740 20 client.25126 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.519+0000 7f70d76fd740 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55996071b900 0x55996071dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.520+0000 7f70d76fd740 4 client.25126 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.520+0000 7f70d76fd740 1 -- 172.21.15.33:0/546794417 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55996066b860 con 0x55996071b900 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.520+0000 7f70ceffd640 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55996071b900 0x55996071dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.520+0000 7f70ceffd640 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55996071b900 0x55996071dd40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e94e9634a7a24c1 server_cookie=e634943af310c853 in_seq=0 out_seq=0 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.520+0000 7f70b67fc640 10 client.25126 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.521+0000 7f70b67fc640 1 -- 172.21.15.33:0/546794417 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55996066b860 con 0x55996071b900 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.521+0000 7f70b67fc640 10 client.25126 handle_command_reply: tid=0 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.521+0000 7f70d76fd740 1 client.25126 shutdown 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.521+0000 7f70d76fd740 1 -- 172.21.15.33:0/546794417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702c00 msgr2=0x55996070b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.521+0000 7f70d76fd740 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702c00 0x55996070b440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f70b80b8d00 tx=0x7f70b80bc1c0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.522+0000 7f70d76fd740 1 -- 172.21.15.33:0/546794417 shutdown_connections 2024-09-16T00:18:24.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.522+0000 7f70d76fd740 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55996071b900 0x55996071dd40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.522+0000 7f70d76fd740 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5599607075d0 0x55996070b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.522+0000 7f70d76fd740 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702c00 0x55996070b440 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.522+0000 7f70d76fd740 1 --2- 172.21.15.33:0/546794417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559960702280 0x55996070e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.522+0000 7f70b67fc640 0 client.25126 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:24.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.522+0000 7f70d76fd740 1 -- 172.21.15.33:0/546794417 >> 172.21.15.33:0/546794417 conn(0x5599606c1310 msgr2=0x559960701a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:24.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.522+0000 7f70d76fd740 1 -- 172.21.15.33:0/546794417 shutdown_connections 2024-09-16T00:18:24.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.522+0000 7f70d76fd740 1 -- 172.21.15.33:0/546794417 wait complete. 2024-09-16T00:18:24.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.522+0000 7f70d76fd740 20 client.25126 trim_cache size 0 max 16384 2024-09-16T00:18:24.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.547+0000 7f70d76fd740 1 Processor -- start 2024-09-16T00:18:24.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.547+0000 7f70d76fd740 1 -- start start 2024-09-16T00:18:24.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.548+0000 7f70d76fd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702280 0x559960717b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.548+0000 7f70d76fd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x559960709d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.548+0000 7f70d76fd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5599607075d0 0x5599606f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.548+0000 7f70d76fd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f70d01af850 con 0x559960702c00 2024-09-16T00:18:24.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.548+0000 7f70d76fd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f70d01ad120 con 0x5599607075d0 2024-09-16T00:18:24.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.548+0000 7f70d76fd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f70d01aaf70 con 0x559960702280 2024-09-16T00:18:24.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.548+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x559960709d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.548+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x559960709d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56058/0 (socket says 172.21.15.33:56058) 2024-09-16T00:18:24.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.548+0000 7f70c7fff640 1 -- 172.21.15.33:0/2261849706 learned_addr learned my addr 172.21.15.33:0/2261849706 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:24.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.548+0000 7f70ceffd640 1 --2- 172.21.15.33:0/2261849706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702280 0x559960717b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.548+0000 7f70cf7fe640 1 --2- 172.21.15.33:0/2261849706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5599607075d0 0x5599606f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.549+0000 7f70c7fff640 1 -- 172.21.15.33:0/2261849706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702280 msgr2=0x559960717b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:24.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.549+0000 7f70c7fff640 1 --2- 172.21.15.33:0/2261849706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702280 0x559960717b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.549+0000 7f70c7fff640 1 -- 172.21.15.33:0/2261849706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5599607075d0 msgr2=0x5599606f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:24.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.549+0000 7f70c7fff640 1 --2- 172.21.15.33:0/2261849706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5599607075d0 0x5599606f34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.549+0000 7f70c7fff640 1 -- 172.21.15.33:0/2261849706 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599606f4c20 con 0x559960702c00 2024-09-16T00:18:24.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.549+0000 7f70c7fff640 1 --2- 172.21.15.33:0/2261849706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x559960709d10 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f70b8000c00 tx=0x7f70b80bc230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a493a4fb2167a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:24.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.549+0000 7f70c4ff9640 1 -- 172.21.15.33:0/2261849706 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8049070 con 0x559960702c00 2024-09-16T00:18:24.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.549+0000 7f70c4ff9640 1 -- 172.21.15.33:0/2261849706 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70b80bca30 con 0x559960702c00 2024-09-16T00:18:24.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.549+0000 7f70c4ff9640 1 -- 172.21.15.33:0/2261849706 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b80bc330 con 0x559960702c00 2024-09-16T00:18:24.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.550+0000 7f70d76fd740 1 -- 172.21.15.33:0/2261849706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 msgr2=0x559960709d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:24.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.550+0000 7f70d76fd740 1 --2- 172.21.15.33:0/2261849706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x559960709d10 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f70b8000c00 tx=0x7f70b80bc230 comp rx=0 tx=0).stop 2024-09-16T00:18:24.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.550+0000 7f70d76fd740 1 -- 172.21.15.33:0/2261849706 shutdown_connections 2024-09-16T00:18:24.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.550+0000 7f70d76fd740 1 --2- 172.21.15.33:0/2261849706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5599607075d0 0x5599606f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.550+0000 7f70d76fd740 1 --2- 172.21.15.33:0/2261849706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x559960709d10 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.550+0000 7f70d76fd740 1 --2- 172.21.15.33:0/2261849706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559960702280 0x559960717b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.550+0000 7f70d76fd740 1 -- 172.21.15.33:0/2261849706 >> 172.21.15.33:0/2261849706 conn(0x5599606c1310 msgr2=0x559960701a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:24.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.550+0000 7f70d76fd740 1 -- 172.21.15.33:0/2261849706 shutdown_connections 2024-09-16T00:18:24.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.550+0000 7f70d76fd740 1 -- 172.21.15.33:0/2261849706 wait complete. 2024-09-16T00:18:24.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.550+0000 7f70d76fd740 1 -- start start 2024-09-16T00:18:24.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70d76fd740 1 Processor -- start 2024-09-16T00:18:24.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70d76fd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559960702280 0x559960714010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70d76fd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x559960714550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70d76fd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599607075d0 0x559960722e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70d76fd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f70d01ab5e0 con 0x559960702c00 2024-09-16T00:18:24.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70d76fd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f70d01aca50 con 0x559960702280 2024-09-16T00:18:24.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70d76fd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f70d01abcb0 con 0x5599607075d0 2024-09-16T00:18:24.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70d76fd740 10 client.0 _ll_register_callbacks cb 0x7f70b80b7eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:24.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x559960714550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x559960714550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56074/0 (socket says 172.21.15.33:56074) 2024-09-16T00:18:24.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70c7fff640 1 -- 172.21.15.33:0/3370763200 learned_addr learned my addr 172.21.15.33:0/3370763200 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:24.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70ceffd640 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559960702280 0x559960714010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70cf7fe640 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599607075d0 0x559960722e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70c7fff640 1 -- 172.21.15.33:0/3370763200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599607075d0 msgr2=0x559960722e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:24.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70c7fff640 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599607075d0 0x559960722e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70c7fff640 1 -- 172.21.15.33:0/3370763200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559960702280 msgr2=0x559960714010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:24.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70c7fff640 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559960702280 0x559960714010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70c7fff640 1 -- 172.21.15.33:0/3370763200 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559960723660 con 0x559960702c00 2024-09-16T00:18:24.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.551+0000 7f70cf7fe640 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599607075d0 0x559960722e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:18:24.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.552+0000 7f70c7fff640 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x559960714550 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f70b80c4a70 tx=0x7f70b80bc1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441c31881d8035c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:24.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.552+0000 7f70b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:18:24.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.552+0000 7f70b67fc640 1 -- 172.21.15.33:0/3370763200 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8049050 con 0x559960702c00 2024-09-16T00:18:24.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.552+0000 7f70b67fc640 1 -- 172.21.15.33:0/3370763200 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f70b80c1a10 con 0x559960702c00 2024-09-16T00:18:24.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.552+0000 7f70d76fd740 1 -- 172.21.15.33:0/3370763200 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70b80bca30 con 0x559960702c00 2024-09-16T00:18:24.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.552+0000 7f70b67fc640 1 -- 172.21.15.33:0/3370763200 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f70b80bc330 con 0x559960702c00 2024-09-16T00:18:24.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.552+0000 7f70d76fd740 10 client.15552 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:24.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.552+0000 7f70d76fd740 1 -- 172.21.15.33:0/3370763200 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559960723660 con 0x559960702c00 2024-09-16T00:18:24.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.553+0000 7f70b67fc640 1 -- 172.21.15.33:0/3370763200 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70b80bca30 con 0x559960702c00 2024-09-16T00:18:24.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.553+0000 7f70b67fc640 1 -- 172.21.15.33:0/3370763200 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f70b80c4b10 con 0x559960702c00 2024-09-16T00:18:24.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.553+0000 7f70d76fd740 10 client.15552 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:24.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.553+0000 7f70d76fd740 10 client.15552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:18:24.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.553+0000 7f70d76fd740 20 client.15552 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:24.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.553+0000 7f70d76fd740 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55996071b900 0x559960788570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:24.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.553+0000 7f70d76fd740 4 client.15552 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:18:24.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.553+0000 7f70d76fd740 1 -- 172.21.15.33:0/3370763200 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559960665d70 con 0x55996071b900 2024-09-16T00:18:24.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.553+0000 7f70ceffd640 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55996071b900 0x559960788570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:24.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70ceffd640 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55996071b900 0x559960788570 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6214d323023f01c server_cookie=79ab3d2a4bce32d5 in_seq=0 out_seq=0 2024-09-16T00:18:24.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70b67fc640 10 client.15552 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:24.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70b67fc640 1 -- 172.21.15.33:0/3370763200 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559960665d70 con 0x55996071b900 2024-09-16T00:18:24.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70b67fc640 10 client.15552 handle_command_reply: tid=0 2024-09-16T00:18:24.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70d76fd740 1 client.15552 shutdown 2024-09-16T00:18:24.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70d76fd740 1 -- 172.21.15.33:0/3370763200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 msgr2=0x559960714550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:24.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70d76fd740 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x559960714550 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f70b80c4a70 tx=0x7f70b80bc1c0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70d76fd740 1 -- 172.21.15.33:0/3370763200 shutdown_connections 2024-09-16T00:18:24.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70d76fd740 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55996071b900 0x559960788570 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70d76fd740 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599607075d0 0x559960722e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70d76fd740 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559960702c00 0x559960714550 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70d76fd740 1 --2- 172.21.15.33:0/3370763200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559960702280 0x559960714010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70b67fc640 0 client.15552 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:24.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70d76fd740 1 -- 172.21.15.33:0/3370763200 >> 172.21.15.33:0/3370763200 conn(0x5599606c1310 msgr2=0x55996070cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:24.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70d76fd740 1 -- 172.21.15.33:0/3370763200 shutdown_connections 2024-09-16T00:18:24.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70d76fd740 1 -- 172.21.15.33:0/3370763200 wait complete. 2024-09-16T00:18:24.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.554+0000 7f70d76fd740 20 client.15552 trim_cache size 0 max 16384 2024-09-16T00:18:24.562 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:18:24.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.555+0000 7f70d5324640 1 -- 172.21.15.33:0/3301038076 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f70bc0a3350 msgr2=0x7f70bc0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:24.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.555+0000 7f70d5324640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f70bc0a3350 0x7f70bc0a5810 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f70c8005fd0 tx=0x7f70c8004a40 comp rx=0 tx=0).stop 2024-09-16T00:18:24.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.555+0000 7f70d5324640 1 -- 172.21.15.33:0/3301038076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70d01012d0 msgr2=0x7f70d0198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:24.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.555+0000 7f70d5324640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70d01012d0 0x7f70d0198960 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f70b8005940 tx=0x7f70b80040c0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.555+0000 7f70d5324640 1 -- 172.21.15.33:0/3301038076 shutdown_connections 2024-09-16T00:18:24.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.555+0000 7f70d5324640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f70bc0a3350 0x7f70bc0a5810 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.555+0000 7f70d5324640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70d01055a0 0x7f70d019ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.555+0000 7f70d5324640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70d01012d0 0x7f70d0198960 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.555+0000 7f70d5324640 1 --2- 172.21.15.33:0/3301038076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70d0100920 0x7f70d0198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:24.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.555+0000 7f70d5324640 1 -- 172.21.15.33:0/3301038076 >> 172.21.15.33:0/3301038076 conn(0x7f70d0075f70 msgr2=0x7f70d010dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:24.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.555+0000 7f70d5324640 1 -- 172.21.15.33:0/3301038076 shutdown_connections 2024-09-16T00:18:24.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:24.556+0000 7f70d5324640 1 -- 172.21.15.33:0/3301038076 wait complete. 2024-09-16T00:18:25.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:25 smithi033 ceph-mon[31243]: pgmap v388: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-16T00:18:25.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:25 smithi083 ceph-mon[35242]: pgmap v388: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-16T00:18:25.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:25 smithi060 ceph-mon[35185]: pgmap v388: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-16T00:18:26.382 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:18:26.382 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:18:26.705 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:18:27.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:27 smithi033 ceph-mon[31243]: pgmap v389: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-16T00:18:27.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.203+0000 7f15e4a25640 1 -- 172.21.15.33:0/4071707975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e0106f20 msgr2=0x7f15e010eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:27.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.203+0000 7f15e4a25640 1 --2- 172.21.15.33:0/4071707975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e0106f20 0x7f15e010eaa0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009ca0 tx=0x7f15d802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.203+0000 7f15e4a25640 1 -- 172.21.15.33:0/4071707975 shutdown_connections 2024-09-16T00:18:27.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.203+0000 7f15e4a25640 1 --2- 172.21.15.33:0/4071707975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e0106f20 0x7f15e010eaa0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.203+0000 7f15e4a25640 1 --2- 172.21.15.33:0/4071707975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e01027a0 0x7f15e01067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.203+0000 7f15e4a25640 1 --2- 172.21.15.33:0/4071707975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15e0101df0 0x7f15e01021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.203+0000 7f15e4a25640 1 -- 172.21.15.33:0/4071707975 >> 172.21.15.33:0/4071707975 conn(0x7f15e00fd8f0 msgr2=0x7f15e00ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:27.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.203+0000 7f15e4a25640 1 -- 172.21.15.33:0/4071707975 shutdown_connections 2024-09-16T00:18:27.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.203+0000 7f15e4a25640 1 -- 172.21.15.33:0/4071707975 wait complete. 2024-09-16T00:18:27.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15e4a25640 1 Processor -- start 2024-09-16T00:18:27.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15e4a25640 1 -- start start 2024-09-16T00:18:27.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15e4a25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e0101df0 0x7f15e019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15e4a25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15e01027a0 0x7f15e019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15e4a25640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e0106f20 0x7f15e010db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15e4a25640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f15e0111f80 con 0x7f15e0106f20 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15e4a25640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f15e0111e00 con 0x7f15e01027a0 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15e4a25640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f15e0112100 con 0x7f15e0101df0 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15ded9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e0106f20 0x7f15e010db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15ded9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e0106f20 0x7f15e010db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56094/0 (socket says 172.21.15.33:56094) 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15ded9d640 1 -- 172.21.15.33:0/572728372 learned_addr learned my addr 172.21.15.33:0/572728372 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15de59c640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e0101df0 0x7f15e019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15e01027a0 0x7f15e019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.205+0000 7f15de59c640 1 -- 172.21.15.33:0/572728372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15e01027a0 msgr2=0x7f15e019f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.206+0000 7f15de59c640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15e01027a0 0x7f15e019f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.206+0000 7f15de59c640 1 -- 172.21.15.33:0/572728372 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e0106f20 msgr2=0x7f15e010db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.206+0000 7f15de59c640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e0106f20 0x7f15e010db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.206+0000 7f15de59c640 1 -- 172.21.15.33:0/572728372 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e010e410 con 0x7f15e0101df0 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.206+0000 7f15de59c640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e0101df0 0x7f15e019ef70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f15d000e9f0 tx=0x7f15d000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.206+0000 7f15cf7fe640 1 -- 172.21.15.33:0/572728372 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d000cd80 con 0x7f15e0101df0 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.206+0000 7f15e4a25640 1 -- 172.21.15.33:0/572728372 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e010e700 con 0x7f15e0101df0 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.206+0000 7f15cf7fe640 1 -- 172.21.15.33:0/572728372 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15d00035f0 con 0x7f15e0101df0 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.206+0000 7f15cf7fe640 1 -- 172.21.15.33:0/572728372 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0010670 con 0x7f15e0101df0 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.206+0000 7f15e4a25640 1 -- 172.21.15.33:0/572728372 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e019ac80 con 0x7f15e0101df0 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.208+0000 7f15cf7fe640 1 -- 172.21.15.33:0/572728372 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f15d00037a0 con 0x7f15e0101df0 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.209+0000 7f15cf7fe640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15c80a32d0 0x7f15c80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.209+0000 7f15cf7fe640 1 -- 172.21.15.33:0/572728372 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f15d00d0260 con 0x7f15e0101df0 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.209+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15c80a32d0 0x7f15c80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.210+0000 7f15e6dfe740 1 Processor -- start 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.210+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15c80a32d0 0x7f15c80a5790 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f15e010ed90 tx=0x7f15d4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:27.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.210+0000 7f15e6dfe740 1 -- start start 2024-09-16T00:18:27.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.210+0000 7f15e6dfe740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba68b60 0x55ccfba68f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.210+0000 7f15e6dfe740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 0x55ccfba73800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.210+0000 7f15e6dfe740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba73d40 0x55ccfba76130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.210+0000 7f15e6dfe740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ccfb9ce280 con 0x55ccfba69480 2024-09-16T00:18:27.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.210+0000 7f15e6dfe740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ccfb975b10 con 0x55ccfba73d40 2024-09-16T00:18:27.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.210+0000 7f15e6dfe740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ccfb9ca240 con 0x55ccfba68b60 2024-09-16T00:18:27.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.210+0000 7f15ddd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 0x55ccfba73800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15ddd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 0x55ccfba73800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56096/0 (socket says 172.21.15.33:56096) 2024-09-16T00:18:27.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15ddd9b640 1 -- 172.21.15.33:0/3574857387 learned_addr learned my addr 172.21.15.33:0/3574857387 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:27.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15ded9d640 1 --2- 172.21.15.33:0/3574857387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba73d40 0x55ccfba76130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15de59c640 1 --2- 172.21.15.33:0/3574857387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba68b60 0x55ccfba68f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15ddd9b640 1 -- 172.21.15.33:0/3574857387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba68b60 msgr2=0x55ccfba68f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/3574857387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba68b60 0x55ccfba68f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15ddd9b640 1 -- 172.21.15.33:0/3574857387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba73d40 msgr2=0x55ccfba76130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/3574857387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba73d40 0x55ccfba76130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15ddd9b640 1 -- 172.21.15.33:0/3574857387 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccfb9821f0 con 0x55ccfba69480 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/3574857387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 0x55ccfba73800 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f15d4005f60 tx=0x7f15d4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b86a221b9e24b8cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15ccff9640 1 -- 172.21.15.33:0/3574857387 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4003d70 con 0x55ccfba69480 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15ccff9640 1 -- 172.21.15.33:0/3574857387 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15d40044d0 con 0x55ccfba69480 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.211+0000 7f15ccff9640 1 -- 172.21.15.33:0/3574857387 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d401e680 con 0x55ccfba69480 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.212+0000 7f15e6dfe740 1 -- 172.21.15.33:0/3574857387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 msgr2=0x55ccfba73800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.212+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/3574857387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 0x55ccfba73800 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f15d4005f60 tx=0x7f15d4014000 comp rx=0 tx=0).stop 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.212+0000 7f15e6dfe740 1 -- 172.21.15.33:0/3574857387 shutdown_connections 2024-09-16T00:18:27.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.212+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/3574857387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba73d40 0x55ccfba76130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.212+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/3574857387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 0x55ccfba73800 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.212+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/3574857387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba68b60 0x55ccfba68f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.212+0000 7f15e6dfe740 1 -- 172.21.15.33:0/3574857387 >> 172.21.15.33:0/3574857387 conn(0x55ccfba650f0 msgr2=0x55ccfba654d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.213+0000 7f15e6dfe740 1 -- 172.21.15.33:0/3574857387 shutdown_connections 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.213+0000 7f15e6dfe740 1 -- 172.21.15.33:0/3574857387 wait complete. 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.213+0000 7f15e6dfe740 1 -- start start 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.213+0000 7f15e6dfe740 1 Processor -- start 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.213+0000 7f15e6dfe740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba68b60 0x55ccfba70be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15e6dfe740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba69480 0x55ccfba71120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15e6dfe740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba73d40 0x55ccfba71660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15e6dfe740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ccfba76910 con 0x55ccfba73d40 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15e6dfe740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ccfb988f10 con 0x55ccfba69480 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15e6dfe740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ccfba76a90 con 0x55ccfba68b60 2024-09-16T00:18:27.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15e6dfe740 10 client.0 _ll_register_callbacks cb 0x55ccfb951a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:27.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15ded9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba73d40 0x55ccfba71660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15de59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba68b60 0x55ccfba70be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15de59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba68b60 0x55ccfba70be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:38006/0 (socket says 172.21.15.33:38006) 2024-09-16T00:18:27.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15de59c640 1 -- 172.21.15.33:0/941915591 learned_addr learned my addr 172.21.15.33:0/941915591 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:27.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/941915591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba69480 0x55ccfba71120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15de59c640 1 -- 172.21.15.33:0/941915591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba69480 msgr2=0x55ccfba71120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:27.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15de59c640 1 --2- 172.21.15.33:0/941915591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba69480 0x55ccfba71120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15de59c640 1 -- 172.21.15.33:0/941915591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba73d40 msgr2=0x55ccfba71660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:27.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15de59c640 1 --2- 172.21.15.33:0/941915591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba73d40 0x55ccfba71660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15de59c640 1 -- 172.21.15.33:0/941915591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccfb73f6a0 con 0x55ccfba68b60 2024-09-16T00:18:27.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15de59c640 1 --2- 172.21.15.33:0/941915591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba68b60 0x55ccfba70be0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f15d000e9c0 tx=0x7f15d009c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fbf4dddb2269565 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:27.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15c5ffb640 1 -- 172.21.15.33:0/941915591 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d00b5210 con 0x55ccfba68b60 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15e6dfe740 1 -- 172.21.15.33:0/941915591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15d40044d0 con 0x55ccfba68b60 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.214+0000 7f15c5ffb640 1 -- 172.21.15.33:0/941915591 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f15d0003b30 con 0x55ccfba68b60 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.215+0000 7f15c5ffb640 1 -- 172.21.15.33:0/941915591 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f15d00988b0 con 0x55ccfba68b60 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.215+0000 7f15e6dfe740 10 client.25141 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.215+0000 7f15e6dfe740 1 -- 172.21.15.33:0/941915591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ccfba935a0 con 0x55ccfba68b60 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.215+0000 7f15c5ffb640 1 -- 172.21.15.33:0/941915591 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15d40044d0 con 0x55ccfba68b60 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.216+0000 7f15c5ffb640 1 -- 172.21.15.33:0/941915591 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f15d00bc070 con 0x55ccfba68b60 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.216+0000 7f15e6dfe740 10 client.25141 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.216+0000 7f15e6dfe740 10 client.25141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.216+0000 7f15e6dfe740 20 client.25141 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.216+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/941915591 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccfba949b0 0x55ccfba96da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.216+0000 7f15e6dfe740 4 client.25141 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:18:27.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.216+0000 7f15e6dfe740 1 -- 172.21.15.33:0/941915591 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccfb9db7f0 con 0x55ccfba949b0 2024-09-16T00:18:27.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.216+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/941915591 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccfba949b0 0x55ccfba96da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.216+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/941915591 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccfba949b0 0x55ccfba96da0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c915ba203e9b7f7 server_cookie=a65b1c2035bd3914 in_seq=0 out_seq=0 2024-09-16T00:18:27.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.216+0000 7f15c5ffb640 10 client.25141 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:27.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15c5ffb640 1 -- 172.21.15.33:0/941915591 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ccfb9db7f0 con 0x55ccfba949b0 2024-09-16T00:18:27.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15c5ffb640 10 client.25141 handle_command_reply: tid=0 2024-09-16T00:18:27.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15e6dfe740 1 client.25141 shutdown 2024-09-16T00:18:27.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15e6dfe740 1 -- 172.21.15.33:0/941915591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba68b60 msgr2=0x55ccfba70be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:27.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/941915591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba68b60 0x55ccfba70be0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f15d000e9c0 tx=0x7f15d009c300 comp rx=0 tx=0).stop 2024-09-16T00:18:27.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15e6dfe740 1 -- 172.21.15.33:0/941915591 shutdown_connections 2024-09-16T00:18:27.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/941915591 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccfba949b0 0x55ccfba96da0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/941915591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba73d40 0x55ccfba71660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/941915591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba69480 0x55ccfba71120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/941915591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba68b60 0x55ccfba70be0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15e6dfe740 1 -- 172.21.15.33:0/941915591 >> 172.21.15.33:0/941915591 conn(0x55ccfba650f0 msgr2=0x55ccfba6add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:27.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15c5ffb640 0 client.25141 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:27.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15e6dfe740 1 -- 172.21.15.33:0/941915591 shutdown_connections 2024-09-16T00:18:27.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.217+0000 7f15e6dfe740 1 -- 172.21.15.33:0/941915591 wait complete. 2024-09-16T00:18:27.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.218+0000 7f15e6dfe740 20 client.25141 trim_cache size 0 max 16384 2024-09-16T00:18:27.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.235+0000 7f15e6dfe740 1 Processor -- start 2024-09-16T00:18:27.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.235+0000 7f15e6dfe740 1 -- start start 2024-09-16T00:18:27.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15e6dfe740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba68b60 0x55ccfba714b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15e6dfe740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 0x55ccfba9c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15e6dfe740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba73d40 0x55ccfba9cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15e6dfe740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f15e01b8090 con 0x55ccfba69480 2024-09-16T00:18:27.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15e6dfe740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f15e01b5960 con 0x55ccfba68b60 2024-09-16T00:18:27.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15e6dfe740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f15e0072fa0 con 0x55ccfba73d40 2024-09-16T00:18:27.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15ddd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 0x55ccfba9c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15ddd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 0x55ccfba9c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56112/0 (socket says 172.21.15.33:56112) 2024-09-16T00:18:27.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15ddd9b640 1 -- 172.21.15.33:0/85052442 learned_addr learned my addr 172.21.15.33:0/85052442 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:27.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15ddd9b640 1 -- 172.21.15.33:0/85052442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba73d40 msgr2=0x55ccfba9cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:27.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15ded9d640 1 --2- 172.21.15.33:0/85052442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba73d40 0x55ccfba9cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/85052442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba73d40 0x55ccfba9cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15ddd9b640 1 -- 172.21.15.33:0/85052442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba68b60 msgr2=0x55ccfba714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15de59c640 1 --2- 172.21.15.33:0/85052442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba68b60 0x55ccfba714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/85052442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba68b60 0x55ccfba714b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.236+0000 7f15ddd9b640 1 -- 172.21.15.33:0/85052442 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccfba64f20 con 0x55ccfba69480 2024-09-16T00:18:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.237+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/85052442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 0x55ccfba9c5f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f15d4005f60 tx=0x7f15d40098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=403525d941bc442f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.237+0000 7f15ded9d640 1 --2- 172.21.15.33:0/85052442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba73d40 0x55ccfba9cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:18:27.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.237+0000 7f15c7fff640 1 -- 172.21.15.33:0/85052442 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4003c70 con 0x55ccfba69480 2024-09-16T00:18:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.237+0000 7f15c7fff640 1 -- 172.21.15.33:0/85052442 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15d40044d0 con 0x55ccfba69480 2024-09-16T00:18:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.237+0000 7f15c7fff640 1 -- 172.21.15.33:0/85052442 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4004c40 con 0x55ccfba69480 2024-09-16T00:18:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.238+0000 7f15e6dfe740 1 -- 172.21.15.33:0/85052442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 msgr2=0x55ccfba9c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.238+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/85052442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 0x55ccfba9c5f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f15d4005f60 tx=0x7f15d40098c0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.238+0000 7f15e6dfe740 1 -- 172.21.15.33:0/85052442 shutdown_connections 2024-09-16T00:18:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.238+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/85052442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba73d40 0x55ccfba9cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.238+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/85052442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba69480 0x55ccfba9c5f0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.238+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/85052442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba68b60 0x55ccfba714b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.238+0000 7f15e6dfe740 1 -- 172.21.15.33:0/85052442 >> 172.21.15.33:0/85052442 conn(0x55ccfba650f0 msgr2=0x55ccfba72b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.238+0000 7f15e6dfe740 1 -- 172.21.15.33:0/85052442 shutdown_connections 2024-09-16T00:18:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.238+0000 7f15e6dfe740 1 -- 172.21.15.33:0/85052442 wait complete. 2024-09-16T00:18:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.238+0000 7f15e6dfe740 1 -- start start 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.238+0000 7f15e6dfe740 1 Processor -- start 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15e6dfe740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba68b60 0x55ccfba89220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15e6dfe740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba69480 0x55ccfba89760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15de59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba68b60 0x55ccfba89220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15de59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba68b60 0x55ccfba89220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56120/0 (socket says 172.21.15.33:56120) 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15e6dfe740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba73d40 0x55ccfbafe5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15de59c640 1 -- 172.21.15.33:0/2855452597 learned_addr learned my addr 172.21.15.33:0/2855452597 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15e6dfe740 1 -- 172.21.15.33:0/2855452597 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f15e0073610 con 0x55ccfba68b60 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15e6dfe740 1 -- 172.21.15.33:0/2855452597 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f15e01b5290 con 0x55ccfba73d40 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15e6dfe740 1 -- 172.21.15.33:0/2855452597 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f15e0073ce0 con 0x55ccfba69480 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15e6dfe740 10 client.0 _ll_register_callbacks cb 0x7f15d0098560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/2855452597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba69480 0x55ccfba89760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15de59c640 1 -- 172.21.15.33:0/2855452597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba69480 msgr2=0x55ccfba89760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:27.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15de59c640 1 --2- 172.21.15.33:0/2855452597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba69480 0x55ccfba89760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15de59c640 1 -- 172.21.15.33:0/2855452597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba73d40 msgr2=0x55ccfbafe5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:18:27.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15de59c640 1 --2- 172.21.15.33:0/2855452597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba73d40 0x55ccfbafe5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15de59c640 1 -- 172.21.15.33:0/2855452597 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccfbafeb30 con 0x55ccfba68b60 2024-09-16T00:18:27.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15de59c640 1 --2- 172.21.15.33:0/2855452597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba68b60 0x55ccfba89220 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f15d00bc410 tx=0x7f15d009e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a17d6bfc5c85ddae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:27.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.239+0000 7f15c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:18:27.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15c5ffb640 1 -- 172.21.15.33:0/2855452597 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d00ba070 con 0x55ccfba68b60 2024-09-16T00:18:27.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15c5ffb640 1 -- 172.21.15.33:0/2855452597 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f15d0085170 con 0x55ccfba68b60 2024-09-16T00:18:27.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15e6dfe740 1 -- 172.21.15.33:0/2855452597 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15d40044d0 con 0x55ccfba68b60 2024-09-16T00:18:27.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15c5ffb640 1 -- 172.21.15.33:0/2855452597 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f15d00b0050 con 0x55ccfba68b60 2024-09-16T00:18:27.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15e6dfe740 10 client.15570 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:27.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15e6dfe740 1 -- 172.21.15.33:0/2855452597 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ccfbaffdc0 con 0x55ccfba68b60 2024-09-16T00:18:27.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15c5ffb640 1 -- 172.21.15.33:0/2855452597 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15d00b5210 con 0x55ccfba68b60 2024-09-16T00:18:27.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15c5ffb640 1 -- 172.21.15.33:0/2855452597 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f15d00a0170 con 0x55ccfba68b60 2024-09-16T00:18:27.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15e6dfe740 10 client.15570 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:27.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15e6dfe740 10 client.15570 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:18:27.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15e6dfe740 20 client.15570 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:27.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/2855452597 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccfbb004e0 0x55ccfbb028d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:27.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15e6dfe740 4 client.15570 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:18:27.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.240+0000 7f15e6dfe740 1 -- 172.21.15.33:0/2855452597 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f15d40044d0 con 0x55ccfbb004e0 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.241+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/2855452597 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccfbb004e0 0x55ccfbb028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.241+0000 7f15ddd9b640 1 --2- 172.21.15.33:0/2855452597 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccfbb004e0 0x55ccfbb028d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=828d39b3815bd9d8 server_cookie=95921681fdc16c8b in_seq=0 out_seq=0 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.241+0000 7f15c5ffb640 10 client.15570 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.242+0000 7f15c5ffb640 1 -- 172.21.15.33:0/2855452597 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f15d40044d0 con 0x55ccfbb004e0 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.242+0000 7f15c5ffb640 10 client.15570 handle_command_reply: tid=0 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.242+0000 7f15e6dfe740 1 client.15570 shutdown 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.243+0000 7f15e6dfe740 1 -- 172.21.15.33:0/2855452597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba68b60 msgr2=0x55ccfba89220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.243+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/2855452597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba68b60 0x55ccfba89220 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f15d00bc410 tx=0x7f15d009e000 comp rx=0 tx=0).stop 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.243+0000 7f15e6dfe740 1 -- 172.21.15.33:0/2855452597 shutdown_connections 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.243+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/2855452597 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccfbb004e0 0x55ccfbb028d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.243+0000 7f15c5ffb640 0 client.15570 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.243+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/2855452597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccfba73d40 0x55ccfbafe5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.243+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/2855452597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccfba69480 0x55ccfba89760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.243+0000 7f15e6dfe740 1 --2- 172.21.15.33:0/2855452597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccfba68b60 0x55ccfba89220 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.243+0000 7f15e6dfe740 1 -- 172.21.15.33:0/2855452597 >> 172.21.15.33:0/2855452597 conn(0x55ccfba650f0 msgr2=0x55ccfba63a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:27.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.243+0000 7f15e6dfe740 1 -- 172.21.15.33:0/2855452597 shutdown_connections 2024-09-16T00:18:27.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.243+0000 7f15e6dfe740 1 -- 172.21.15.33:0/2855452597 wait complete. 2024-09-16T00:18:27.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.243+0000 7f15e6dfe740 20 client.15570 trim_cache size 0 max 16384 2024-09-16T00:18:27.246 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:18:27.246 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:18:27.246 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "e36cb74a-da97-4a10-ab59-84d9b69fb8b4", 2024-09-16T00:18:27.246 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:18:27.247 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:18:27.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.244+0000 7f15e4a25640 1 -- 172.21.15.33:0/572728372 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15c80a32d0 msgr2=0x7f15c80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:27.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.244+0000 7f15e4a25640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15c80a32d0 0x7f15c80a5790 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f15e010ed90 tx=0x7f15d4005f90 comp rx=0 tx=0).stop 2024-09-16T00:18:27.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.244+0000 7f15e4a25640 1 -- 172.21.15.33:0/572728372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e0101df0 msgr2=0x7f15e019ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:27.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.244+0000 7f15e4a25640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e0101df0 0x7f15e019ef70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f15d000e9f0 tx=0x7f15d000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.244+0000 7f15e4a25640 1 -- 172.21.15.33:0/572728372 shutdown_connections 2024-09-16T00:18:27.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.244+0000 7f15e4a25640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15c80a32d0 0x7f15c80a5790 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.244+0000 7f15e4a25640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e0106f20 0x7f15e010db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.244+0000 7f15e4a25640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15e01027a0 0x7f15e019f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.244+0000 7f15e4a25640 1 --2- 172.21.15.33:0/572728372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e0101df0 0x7f15e019ef70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:27.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.244+0000 7f15e4a25640 1 -- 172.21.15.33:0/572728372 >> 172.21.15.33:0/572728372 conn(0x7f15e00fd8f0 msgr2=0x7f15e00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:27.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.244+0000 7f15e4a25640 1 -- 172.21.15.33:0/572728372 shutdown_connections 2024-09-16T00:18:27.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:27.244+0000 7f15e4a25640 1 -- 172.21.15.33:0/572728372 wait complete. 2024-09-16T00:18:27.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:27 smithi083 ceph-mon[35242]: pgmap v389: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-16T00:18:27.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:27 smithi060 ceph-mon[35185]: pgmap v389: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-16T00:18:27.957 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:18:28.253 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:18:28.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:28 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:18:28.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:28 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:18:28.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:28 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:18:28.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:28 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:18:28.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:28 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:18:28.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:28 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:18:28.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:28 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:18:28.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:28 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:18:28.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:28 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:18:28.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:28 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:18:28.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:28 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:18:28.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:28 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:18:28.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.807+0000 7f0f0c645640 1 -- 172.21.15.33:0/3412037798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f04101410 msgr2=0x7f0f0410d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:28.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.807+0000 7f0f0c645640 1 --2- 172.21.15.33:0/3412037798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f04101410 0x7f0f0410d940 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009cd0 tx=0x7f0efc02ecf0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.807+0000 7f0f0c645640 1 -- 172.21.15.33:0/3412037798 shutdown_connections 2024-09-16T00:18:28.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.807+0000 7f0f0c645640 1 --2- 172.21.15.33:0/3412037798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f04101410 0x7f0f0410d940 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.807+0000 7f0f0c645640 1 --2- 172.21.15.33:0/3412037798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f04100a70 0x7f0f04100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.807+0000 7f0f0c645640 1 --2- 172.21.15.33:0/3412037798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04106a70 0x7f0f04106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.807+0000 7f0f0c645640 1 -- 172.21.15.33:0/3412037798 >> 172.21.15.33:0/3412037798 conn(0x7f0f040fc790 msgr2=0x7f0f040febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:28.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.807+0000 7f0f0c645640 1 -- 172.21.15.33:0/3412037798 shutdown_connections 2024-09-16T00:18:28.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.807+0000 7f0f0c645640 1 -- 172.21.15.33:0/3412037798 wait complete. 2024-09-16T00:18:28.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.808+0000 7f0f0c645640 1 Processor -- start 2024-09-16T00:18:28.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.808+0000 7f0f0c645640 1 -- start start 2024-09-16T00:18:28.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f0c645640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f04100a70 0x7f0f040739e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f0c645640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f04101410 0x7f0f0406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f0c645640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04106a70 0x7f0f0406cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f0c645640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0410fd60 con 0x7f0f04100a70 2024-09-16T00:18:28.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f0c645640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0410fbe0 con 0x7f0f04106a70 2024-09-16T00:18:28.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f0c645640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0410fee0 con 0x7f0f04101410 2024-09-16T00:18:28.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f09be0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f04101410 0x7f0f0406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f09be0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f04101410 0x7f0f0406ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:38038/0 (socket says 172.21.15.33:38038) 2024-09-16T00:18:28.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f09be0640 1 -- 172.21.15.33:0/404180173 learned_addr learned my addr 172.21.15.33:0/404180173 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:28.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f0abe2640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04106a70 0x7f0f0406cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f09be0640 1 -- 172.21.15.33:0/404180173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04106a70 msgr2=0x7f0f0406cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:28.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f09be0640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04106a70 0x7f0f0406cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f09be0640 1 -- 172.21.15.33:0/404180173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f04100a70 msgr2=0x7f0f040739e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:28.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f09be0640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f04100a70 0x7f0f040739e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.809+0000 7f0f09be0640 1 -- 172.21.15.33:0/404180173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0406d570 con 0x7f0f04101410 2024-09-16T00:18:28.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.810+0000 7f0f09be0640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f04101410 0x7f0f0406ca70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009990 tx=0x7f0ef4009e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:28.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.810+0000 7f0efb7fe640 1 -- 172.21.15.33:0/404180173 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef400c9e0 con 0x7f0f04101410 2024-09-16T00:18:28.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.810+0000 7f0efb7fe640 1 -- 172.21.15.33:0/404180173 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ef4005500 con 0x7f0f04101410 2024-09-16T00:18:28.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.810+0000 7f0efb7fe640 1 -- 172.21.15.33:0/404180173 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4004d50 con 0x7f0f04101410 2024-09-16T00:18:28.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.811+0000 7f0f0c645640 1 -- 172.21.15.33:0/404180173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0406d860 con 0x7f0f04101410 2024-09-16T00:18:28.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.811+0000 7f0f0c645640 1 -- 172.21.15.33:0/404180173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f04071800 con 0x7f0f04101410 2024-09-16T00:18:28.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.814+0000 7f0efb7fe640 1 -- 172.21.15.33:0/404180173 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f0ef4004ef0 con 0x7f0f04101410 2024-09-16T00:18:28.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.814+0000 7f0f0ea1e740 1 Processor -- start 2024-09-16T00:18:28.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.815+0000 7f0f0ea1e740 1 -- start start 2024-09-16T00:18:28.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.815+0000 7f0f0ea1e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa183c0 0x56064fa187a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.815+0000 7f0f0ea1e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 0x56064fa2b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.815+0000 7f0f0ea1e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa2be90 0x56064fa2e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.815+0000 7f0f0ea1e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56064f97e2e0 con 0x56064fa183c0 2024-09-16T00:18:28.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.815+0000 7f0f0ea1e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56064f925ce0 con 0x56064fa18ce0 2024-09-16T00:18:28.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.815+0000 7f0f0ea1e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56064f97a2a0 con 0x56064fa2be90 2024-09-16T00:18:28.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.815+0000 7f0f0abe2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa183c0 0x56064fa187a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.816+0000 7f0f0abe2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa183c0 0x56064fa187a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56156/0 (socket says 172.21.15.33:56156) 2024-09-16T00:18:28.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.816+0000 7f0f0abe2640 1 -- 172.21.15.33:0/3961729020 learned_addr learned my addr 172.21.15.33:0/3961729020 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:28.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.816+0000 7f0f0a3e1640 1 --2- 172.21.15.33:0/3961729020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 0x56064fa2b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.816+0000 7f0f09be0640 1 --2- 172.21.15.33:0/3961729020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa2be90 0x56064fa2e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.816+0000 7f0f0a3e1640 1 -- 172.21.15.33:0/3961729020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa2be90 msgr2=0x56064fa2e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:28.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.816+0000 7f0f0a3e1640 1 --2- 172.21.15.33:0/3961729020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa2be90 0x56064fa2e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.816+0000 7f0f0a3e1640 1 -- 172.21.15.33:0/3961729020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa183c0 msgr2=0x56064fa187a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:28.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.816+0000 7f0f0a3e1640 1 --2- 172.21.15.33:0/3961729020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa183c0 0x56064fa187a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.816+0000 7f0f0a3e1640 1 -- 172.21.15.33:0/3961729020 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56064f90a440 con 0x56064fa18ce0 2024-09-16T00:18:28.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.816+0000 7f0f0a3e1640 1 --2- 172.21.15.33:0/3961729020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 0x56064fa2b950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009ac0 tx=0x7f0f00009f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da444a8c09e6a0d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:28.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.817+0000 7f0ef8ff9640 1 -- 172.21.15.33:0/3961729020 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0000ed10 con 0x56064fa18ce0 2024-09-16T00:18:28.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.817+0000 7f0ef8ff9640 1 -- 172.21.15.33:0/3961729020 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f00005020 con 0x56064fa18ce0 2024-09-16T00:18:28.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.817+0000 7f0efb7fe640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ef00a3350 0x7f0ef00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.817+0000 7f0f0abe2640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ef00a3350 0x7f0ef00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.817+0000 7f0efb7fe640 1 -- 172.21.15.33:0/404180173 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0ef40cf5f0 con 0x7f0f04101410 2024-09-16T00:18:28.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.817+0000 7f0ef8ff9640 1 -- 172.21.15.33:0/3961729020 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f00004c40 con 0x56064fa18ce0 2024-09-16T00:18:28.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.818+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3961729020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 msgr2=0x56064fa2b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:28.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.818+0000 7f0f0abe2640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ef00a3350 0x7f0ef00a5810 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0efc02f210 tx=0x7f0efc031040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:28.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.818+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3961729020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 0x56064fa2b950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009ac0 tx=0x7f0f00009f80 comp rx=0 tx=0).stop 2024-09-16T00:18:28.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.818+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3961729020 shutdown_connections 2024-09-16T00:18:28.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.818+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3961729020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa2be90 0x56064fa2e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.818+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3961729020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 0x56064fa2b950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.818+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3961729020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa183c0 0x56064fa187a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.818+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3961729020 >> 172.21.15.33:0/3961729020 conn(0x56064fa14a30 msgr2=0x56064fa14e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:28.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.818+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3961729020 shutdown_connections 2024-09-16T00:18:28.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.818+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3961729020 wait complete. 2024-09-16T00:18:28.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.819+0000 7f0f0ea1e740 1 -- start start 2024-09-16T00:18:28.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.819+0000 7f0f0ea1e740 1 Processor -- start 2024-09-16T00:18:28.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.819+0000 7f0f0ea1e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa183c0 0x56064fa2b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.819+0000 7f0f0ea1e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa18ce0 0x56064fa289e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.819+0000 7f0f0ea1e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 0x56064fa28f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.819+0000 7f0f0ea1e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56064fa2eaa0 con 0x56064fa2be90 2024-09-16T00:18:28.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.819+0000 7f0f0ea1e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56064f939050 con 0x56064fa183c0 2024-09-16T00:18:28.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.819+0000 7f0f0ea1e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56064fa2ec20 con 0x56064fa18ce0 2024-09-16T00:18:28.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.819+0000 7f0f0ea1e740 10 client.0 _ll_register_callbacks cb 0x56064f979160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:28.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0f0abe2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 0x56064fa28f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0f0abe2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 0x56064fa28f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56164/0 (socket says 172.21.15.33:56164) 2024-09-16T00:18:28.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0f0abe2640 1 -- 172.21.15.33:0/3503941361 learned_addr learned my addr 172.21.15.33:0/3503941361 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:28.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0f0a3e1640 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa183c0 0x56064fa2b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0f09be0640 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa18ce0 0x56064fa289e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0f0a3e1640 1 -- 172.21.15.33:0/3503941361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa18ce0 msgr2=0x56064fa289e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:28.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0f0a3e1640 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa18ce0 0x56064fa289e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0f0a3e1640 1 -- 172.21.15.33:0/3503941361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 msgr2=0x56064fa28f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:28.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0f0a3e1640 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 0x56064fa28f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0f0a3e1640 1 -- 172.21.15.33:0/3503941361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56064f6ef3c0 con 0x56064fa183c0 2024-09-16T00:18:28.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0f0abe2640 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 0x56064fa28f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:28.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0f0a3e1640 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa183c0 0x56064fa2b6e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0f0000e7e0 tx=0x7f0f00005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ccba1e706b660f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:28.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.820+0000 7f0eedffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:18:28.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.821+0000 7f0eedffb640 1 -- 172.21.15.33:0/3503941361 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f00029060 con 0x56064fa183c0 2024-09-16T00:18:28.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.821+0000 7f0eedffb640 1 -- 172.21.15.33:0/3503941361 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0f00004b50 con 0x56064fa183c0 2024-09-16T00:18:28.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.821+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3503941361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f00005020 con 0x56064fa183c0 2024-09-16T00:18:28.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.821+0000 7f0eedffb640 1 -- 172.21.15.33:0/3503941361 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0f00030050 con 0x56064fa183c0 2024-09-16T00:18:28.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.821+0000 7f0f0ea1e740 10 client.25223 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.821+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3503941361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56064fa3a480 con 0x56064fa183c0 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.821+0000 7f0eedffb640 1 -- 172.21.15.33:0/3503941361 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f00005020 con 0x56064fa183c0 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.822+0000 7f0eedffb640 1 -- 172.21.15.33:0/3503941361 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f0f00018470 con 0x56064fa183c0 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.822+0000 7f0f0ea1e740 10 client.25223 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.822+0000 7f0f0ea1e740 10 client.25223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.822+0000 7f0f0ea1e740 20 client.25223 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.822+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56064fa3b880 0x56064fa3dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.822+0000 7f0f0ea1e740 4 client.25223 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.822+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3503941361 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56064f98b850 con 0x56064fa3b880 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.822+0000 7f0f09be0640 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56064fa3b880 0x56064fa3dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.823+0000 7f0f09be0640 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56064fa3b880 0x56064fa3dc70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63cafeedb922e7c2 server_cookie=ca697addfc607e49 in_seq=0 out_seq=0 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.823+0000 7f0eedffb640 10 client.25223 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:28.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.823+0000 7f0eedffb640 1 -- 172.21.15.33:0/3503941361 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56064f98b850 con 0x56064fa3b880 2024-09-16T00:18:28.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.823+0000 7f0eedffb640 10 client.25223 handle_command_reply: tid=0 2024-09-16T00:18:28.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.823+0000 7f0f0ea1e740 1 client.25223 shutdown 2024-09-16T00:18:28.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.824+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3503941361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa183c0 msgr2=0x56064fa2b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:28.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.824+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa183c0 0x56064fa2b6e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0f0000e7e0 tx=0x7f0f00005850 comp rx=0 tx=0).stop 2024-09-16T00:18:28.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.824+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3503941361 shutdown_connections 2024-09-16T00:18:28.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.824+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56064fa3b880 0x56064fa3dc70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.824+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 0x56064fa28f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.824+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa18ce0 0x56064fa289e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.824+0000 7f0eedffb640 0 client.25223 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.824+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3503941361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa183c0 0x56064fa2b6e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.824+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3503941361 >> 172.21.15.33:0/3503941361 conn(0x56064fa14a30 msgr2=0x56064fa1a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.824+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3503941361 shutdown_connections 2024-09-16T00:18:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.824+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3503941361 wait complete. 2024-09-16T00:18:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.824+0000 7f0f0ea1e740 20 client.25223 trim_cache size 0 max 16384 2024-09-16T00:18:28.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.858+0000 7f0f0ea1e740 1 Processor -- start 2024-09-16T00:18:28.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.858+0000 7f0f0ea1e740 1 -- start start 2024-09-16T00:18:28.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f0ea1e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa183c0 0x56064fa282c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f0ea1e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 0x56064fa43210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f0ea1e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 0x56064fa43750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f0ea1e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0f041b8060 con 0x56064fa2be90 2024-09-16T00:18:28.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f0ea1e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0f041b5930 con 0x56064fa18ce0 2024-09-16T00:18:28.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f0ea1e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0f041b3780 con 0x56064fa183c0 2024-09-16T00:18:28.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f09be0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 0x56064fa43210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f09be0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 0x56064fa43210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:50780/0 (socket says 172.21.15.33:50780) 2024-09-16T00:18:28.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f09be0640 1 -- 172.21.15.33:0/3692060603 learned_addr learned my addr 172.21.15.33:0/3692060603 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:28.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f0abe2640 1 --2- 172.21.15.33:0/3692060603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 0x56064fa43750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f0a3e1640 1 --2- 172.21.15.33:0/3692060603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa183c0 0x56064fa282c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f09be0640 1 -- 172.21.15.33:0/3692060603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa183c0 msgr2=0x56064fa282c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:28.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f09be0640 1 --2- 172.21.15.33:0/3692060603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa183c0 0x56064fa282c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f09be0640 1 -- 172.21.15.33:0/3692060603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 msgr2=0x56064fa43750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:28.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f09be0640 1 --2- 172.21.15.33:0/3692060603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 0x56064fa43750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f09be0640 1 -- 172.21.15.33:0/3692060603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56064fa26f30 con 0x56064fa18ce0 2024-09-16T00:18:28.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.859+0000 7f0f0abe2640 1 --2- 172.21.15.33:0/3692060603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 0x56064fa43750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:28.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.860+0000 7f0f09be0640 1 --2- 172.21.15.33:0/3692060603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 0x56064fa43210 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009960 tx=0x7f0ef40d1ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23294b594b7efa17 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:28.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.860+0000 7f0eeffff640 1 -- 172.21.15.33:0/3692060603 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4083070 con 0x56064fa18ce0 2024-09-16T00:18:28.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.860+0000 7f0eeffff640 1 -- 172.21.15.33:0/3692060603 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ef409b630 con 0x56064fa18ce0 2024-09-16T00:18:28.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.860+0000 7f0eeffff640 1 -- 172.21.15.33:0/3692060603 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef400cd30 con 0x56064fa18ce0 2024-09-16T00:18:28.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.861+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3692060603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 msgr2=0x56064fa43210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:28.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.861+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3692060603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 0x56064fa43210 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009960 tx=0x7f0ef40d1ae0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.861+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3692060603 shutdown_connections 2024-09-16T00:18:28.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.861+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3692060603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa2be90 0x56064fa43750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.861+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3692060603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa18ce0 0x56064fa43210 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.861+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/3692060603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa183c0 0x56064fa282c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.861+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3692060603 >> 172.21.15.33:0/3692060603 conn(0x56064fa14a30 msgr2=0x56064fa1a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:28.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.861+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3692060603 shutdown_connections 2024-09-16T00:18:28.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.861+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/3692060603 wait complete. 2024-09-16T00:18:28.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.862+0000 7f0f0ea1e740 1 -- start start 2024-09-16T00:18:28.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.862+0000 7f0f0ea1e740 1 Processor -- start 2024-09-16T00:18:28.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.862+0000 7f0f0ea1e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa183c0 0x56064fa34360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.862+0000 7f0f0ea1e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa18ce0 0x56064fa2f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.862+0000 7f0f0ea1e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa2be90 0x56064fa2fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.862+0000 7f0f09be0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa18ce0 0x56064fa2f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.862+0000 7f0f0a3e1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa183c0 0x56064fa34360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.862+0000 7f0f09be0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa18ce0 0x56064fa2f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56186/0 (socket says 172.21.15.33:56186) 2024-09-16T00:18:28.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.862+0000 7f0f0a3e1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa183c0 0x56064fa34360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:38078/0 (socket says 172.21.15.33:38078) 2024-09-16T00:18:28.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.862+0000 7f0f09be0640 1 -- 172.21.15.33:0/2012944745 learned_addr learned my addr 172.21.15.33:0/2012944745 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:28.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.862+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/2012944745 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0f041b3df0 con 0x56064fa18ce0 2024-09-16T00:18:28.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.862+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/2012944745 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0f041b5260 con 0x56064fa2be90 2024-09-16T00:18:28.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.863+0000 7f0f0abe2640 1 --2- 172.21.15.33:0/2012944745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa2be90 0x56064fa2fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.863+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/2012944745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0f041b44c0 con 0x56064fa183c0 2024-09-16T00:18:28.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.863+0000 7f0f0ea1e740 10 client.0 _ll_register_callbacks cb 0x7f0f00005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:28.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.863+0000 7f0f0a3e1640 1 -- 172.21.15.33:0/2012944745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa2be90 msgr2=0x56064fa2fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:28.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.863+0000 7f0f0a3e1640 1 --2- 172.21.15.33:0/2012944745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa2be90 0x56064fa2fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.863+0000 7f0f0a3e1640 1 -- 172.21.15.33:0/2012944745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa18ce0 msgr2=0x56064fa2f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:28.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.863+0000 7f0f0a3e1640 1 --2- 172.21.15.33:0/2012944745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa18ce0 0x56064fa2f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.863+0000 7f0f0a3e1640 1 -- 172.21.15.33:0/2012944745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56064fa30110 con 0x56064fa183c0 2024-09-16T00:18:28.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.863+0000 7f0f0a3e1640 1 --2- 172.21.15.33:0/2012944745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa183c0 0x56064fa34360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0f00018960 tx=0x7f0f00005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=675d9e84c92f5212 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:28.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.863+0000 7f0eedffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:18:28.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.864+0000 7f0eedffb640 1 -- 172.21.15.33:0/2012944745 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f00039060 con 0x56064fa183c0 2024-09-16T00:18:28.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.864+0000 7f0eedffb640 1 -- 172.21.15.33:0/2012944745 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0f000053e0 con 0x56064fa183c0 2024-09-16T00:18:28.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.864+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/2012944745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ef409b630 con 0x56064fa183c0 2024-09-16T00:18:28.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.864+0000 7f0eedffb640 1 -- 172.21.15.33:0/2012944745 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0f00040050 con 0x56064fa183c0 2024-09-16T00:18:28.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.864+0000 7f0f0ea1e740 10 client.25159 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:28.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.864+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/2012944745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56064faa5e80 con 0x56064fa183c0 2024-09-16T00:18:28.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.864+0000 7f0eedffb640 1 -- 172.21.15.33:0/2012944745 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ef409b630 con 0x56064fa183c0 2024-09-16T00:18:28.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.865+0000 7f0eedffb640 1 -- 172.21.15.33:0/2012944745 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f0f00004040 con 0x56064fa183c0 2024-09-16T00:18:28.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.865+0000 7f0f0ea1e740 10 client.25159 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:28.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.865+0000 7f0f0ea1e740 10 client.25159 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:18:28.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.865+0000 7f0f0ea1e740 20 client.25159 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:28.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.865+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/2012944745 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56064faa6540 0x56064faa8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:28.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.865+0000 7f0f0ea1e740 4 client.25159 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:18:28.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.865+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/2012944745 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56064f985d60 con 0x56064faa6540 2024-09-16T00:18:28.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.865+0000 7f0f09be0640 1 --2- 172.21.15.33:0/2012944745 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56064faa6540 0x56064faa8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:28.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0eedffb640 10 client.25159 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:28.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0f09be0640 1 --2- 172.21.15.33:0/2012944745 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56064faa6540 0x56064faa8960 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63c6cd439283954 server_cookie=6db2541d769921cb in_seq=0 out_seq=0 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0eedffb640 1 -- 172.21.15.33:0/2012944745 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56064f985d60 con 0x56064faa6540 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0eedffb640 10 client.25159 handle_command_reply: tid=0 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0f0ea1e740 1 client.25159 shutdown 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/2012944745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa183c0 msgr2=0x56064fa34360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/2012944745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa183c0 0x56064fa34360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0f00018960 tx=0x7f0f00005f10 comp rx=0 tx=0).stop 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/2012944745 shutdown_connections 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/2012944745 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56064faa6540 0x56064faa8960 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/2012944745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56064fa2be90 0x56064fa2fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0eedffb640 0 client.25159 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/2012944745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56064fa18ce0 0x56064fa2f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0f0ea1e740 1 --2- 172.21.15.33:0/2012944745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56064fa183c0 0x56064fa34360 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.866+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/2012944745 >> 172.21.15.33:0/2012944745 conn(0x56064fa14a30 msgr2=0x56064fa2a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:28.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.867+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/2012944745 shutdown_connections 2024-09-16T00:18:28.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.867+0000 7f0f0ea1e740 1 -- 172.21.15.33:0/2012944745 wait complete. 2024-09-16T00:18:28.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.867+0000 7f0f0ea1e740 20 client.25159 trim_cache size 0 max 16384 2024-09-16T00:18:28.871 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:18:28.871 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:18:28.871 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:18:28.871 INFO:teuthology.orchestra.run.smithi033.stdout: "e36cb74a-da97-4a10-ab59-84d9b69fb8b4": { 2024-09-16T00:18:28.871 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:18:28.871 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "e36cb74a-da97-4a10-ab59-84d9b69fb8b4", 2024-09-16T00:18:28.871 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:18:28.872 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:18:28.872 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:18:28.872 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:18:28.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.867+0000 7f0f0c645640 1 -- 172.21.15.33:0/404180173 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ef00a3350 msgr2=0x7f0ef00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:28.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.867+0000 7f0f0c645640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ef00a3350 0x7f0ef00a5810 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0efc02f210 tx=0x7f0efc031040 comp rx=0 tx=0).stop 2024-09-16T00:18:28.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.868+0000 7f0f0c645640 1 -- 172.21.15.33:0/404180173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f04101410 msgr2=0x7f0f0406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:28.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.868+0000 7f0f0c645640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f04101410 0x7f0f0406ca70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009990 tx=0x7f0ef4009e50 comp rx=0 tx=0).stop 2024-09-16T00:18:28.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.868+0000 7f0f0c645640 1 -- 172.21.15.33:0/404180173 shutdown_connections 2024-09-16T00:18:28.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.868+0000 7f0f0c645640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ef00a3350 0x7f0ef00a5810 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.868+0000 7f0f0c645640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04106a70 0x7f0f0406cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.868+0000 7f0f0c645640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f04101410 0x7f0f0406ca70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.868+0000 7f0f0c645640 1 --2- 172.21.15.33:0/404180173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f04100a70 0x7f0f040739e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:28.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.868+0000 7f0f0c645640 1 -- 172.21.15.33:0/404180173 >> 172.21.15.33:0/404180173 conn(0x7f0f040fc790 msgr2=0x7f0f040fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:28.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.868+0000 7f0f0c645640 1 -- 172.21.15.33:0/404180173 shutdown_connections 2024-09-16T00:18:28.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:28.868+0000 7f0f0c645640 1 -- 172.21.15.33:0/404180173 wait complete. 2024-09-16T00:18:29.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:29 smithi033 ceph-mon[31243]: pgmap v390: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-09-16T00:18:29.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:29 smithi083 ceph-mon[35242]: pgmap v390: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-09-16T00:18:29.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:29 smithi060 ceph-mon[35185]: pgmap v390: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-09-16T00:18:29.610 INFO:tasks.cephfs.filesystem:scrub status for tag:e36cb74a-da97-4a10-ab59-84d9b69fb8b4 - {'path': '/', 'tag': 'e36cb74a-da97-4a10-ab59-84d9b69fb8b4', 'options': 'recursive,force'} 2024-09-16T00:18:31.257 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:31 smithi033 ceph-mon[31243]: pgmap v391: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-09-16T00:18:31.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:31 smithi083 ceph-mon[35242]: pgmap v391: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-09-16T00:18:31.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:31 smithi060 ceph-mon[35185]: pgmap v391: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-09-16T00:18:32.055 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:32 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:18:32.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:32 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:18:32.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:32 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:18:33.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:33 smithi033 ceph-mon[31243]: pgmap v392: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-16T00:18:33.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:33 smithi083 ceph-mon[35242]: pgmap v392: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-16T00:18:33.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:33 smithi060 ceph-mon[35185]: pgmap v392: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-16T00:18:35.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:35 smithi033 ceph-mon[31243]: pgmap v393: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-16T00:18:35.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:35 smithi083 ceph-mon[35242]: pgmap v393: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-16T00:18:35.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:35 smithi060 ceph-mon[35185]: pgmap v393: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-16T00:18:37.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:37 smithi033 ceph-mon[31243]: pgmap v394: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-16T00:18:37.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:37 smithi083 ceph-mon[35242]: pgmap v394: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-16T00:18:37.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:37 smithi060 ceph-mon[35185]: pgmap v394: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-16T00:18:39.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:39 smithi033 ceph-mon[31243]: pgmap v395: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-16T00:18:39.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:39 smithi033 ceph-mon[31243]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-09-16T00:18:39.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:39 smithi083 ceph-mon[35242]: pgmap v395: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-16T00:18:39.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:39 smithi083 ceph-mon[35242]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-09-16T00:18:39.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:39 smithi060 ceph-mon[35185]: pgmap v395: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-16T00:18:39.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:39 smithi060 ceph-mon[35185]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-09-16T00:18:41.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:41 smithi033 ceph-mon[31243]: pgmap v396: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-16T00:18:41.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:41 smithi083 ceph-mon[35242]: pgmap v396: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-16T00:18:41.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:41 smithi060 ceph-mon[35185]: pgmap v396: 129 pgs: 129 active+clean; 663 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-16T00:18:42.223 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:18:42.520 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:18:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.093+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/2649846427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb68102ef0 msgr2=0x7ffb6810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.093+0000 7ffb6d5a1640 1 --2- 172.21.15.33:0/2649846427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb68102ef0 0x7ffb6810da40 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009ca0 tx=0x7ffb6002a950 comp rx=0 tx=0).stop 2024-09-16T00:18:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.093+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/2649846427 shutdown_connections 2024-09-16T00:18:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.093+0000 7ffb6d5a1640 1 --2- 172.21.15.33:0/2649846427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb68102ef0 0x7ffb6810da40 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.093+0000 7ffb6d5a1640 1 --2- 172.21.15.33:0/2649846427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb68102570 0x7ffb681029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.093+0000 7ffb6d5a1640 1 --2- 172.21.15.33:0/2649846427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb68100e20 0x7ffb68101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.093+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/2649846427 >> 172.21.15.33:0/2649846427 conn(0x7ffb680fc7d0 msgr2=0x7ffb680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.093+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/2649846427 shutdown_connections 2024-09-16T00:18:43.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.093+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/2649846427 wait complete. 2024-09-16T00:18:43.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.094+0000 7ffb6d5a1640 1 Processor -- start 2024-09-16T00:18:43.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.095+0000 7ffb6d5a1640 1 -- start start 2024-09-16T00:18:43.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.095+0000 7ffb6d5a1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb68100e20 0x7ffb681992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:43.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.095+0000 7ffb6d5a1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb68102570 0x7ffb681997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.095+0000 7ffb6d5a1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb68102ef0 0x7ffb681a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.095+0000 7ffb6d5a1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffb68110f80 con 0x7ffb68102570 2024-09-16T00:18:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.095+0000 7ffb6d5a1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffb68110e00 con 0x7ffb68102ef0 2024-09-16T00:18:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.095+0000 7ffb6d5a1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffb68111100 con 0x7ffb68100e20 2024-09-16T00:18:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.095+0000 7ffb66ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb68100e20 0x7ffb681992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.095+0000 7ffb66ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb68100e20 0x7ffb681992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52416/0 (socket says 172.21.15.33:52416) 2024-09-16T00:18:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.095+0000 7ffb667fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb68102570 0x7ffb681997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.095+0000 7ffb66ffd640 1 -- 172.21.15.33:0/1422102516 learned_addr learned my addr 172.21.15.33:0/1422102516 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.095+0000 7ffb677fe640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb68102ef0 0x7ffb681a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.096+0000 7ffb66ffd640 1 -- 172.21.15.33:0/1422102516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb68102ef0 msgr2=0x7ffb681a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:43.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.096+0000 7ffb66ffd640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb68102ef0 0x7ffb681a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.096+0000 7ffb66ffd640 1 -- 172.21.15.33:0/1422102516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb68102570 msgr2=0x7ffb681997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.096+0000 7ffb66ffd640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb68102570 0x7ffb681997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.096+0000 7ffb66ffd640 1 -- 172.21.15.33:0/1422102516 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb681a4870 con 0x7ffb68100e20 2024-09-16T00:18:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.096+0000 7ffb667fc640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb68102570 0x7ffb681997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.096+0000 7ffb66ffd640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb68100e20 0x7ffb681992b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800d950 tx=0x7ffb5800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.096+0000 7ffb4ffff640 1 -- 172.21.15.33:0/1422102516 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb58017030 con 0x7ffb68100e20 2024-09-16T00:18:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.096+0000 7ffb4ffff640 1 -- 172.21.15.33:0/1422102516 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb58004020 con 0x7ffb68100e20 2024-09-16T00:18:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.096+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/1422102516 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb681970a0 con 0x7ffb68100e20 2024-09-16T00:18:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.096+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/1422102516 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb68197590 con 0x7ffb68100e20 2024-09-16T00:18:43.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.097+0000 7ffb4ffff640 1 -- 172.21.15.33:0/1422102516 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb580047a0 con 0x7ffb68100e20 2024-09-16T00:18:43.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.098+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/1422102516 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb68101200 con 0x7ffb68100e20 2024-09-16T00:18:43.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.100+0000 7ffb4ffff640 1 -- 172.21.15.33:0/1422102516 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ffb580041d0 con 0x7ffb68100e20 2024-09-16T00:18:43.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.101+0000 7ffb4ffff640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffb500a3240 0x7ffb500a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:43.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.102+0000 7ffb4ffff640 1 -- 172.21.15.33:0/1422102516 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ffb580cfdf0 con 0x7ffb68100e20 2024-09-16T00:18:43.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.102+0000 7ffb667fc640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffb500a3240 0x7ffb500a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:43.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.103+0000 7ffb667fc640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffb500a3240 0x7ffb500a5700 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00ffd0 tx=0x7ffb5c00f7a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:43.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.104+0000 7ffb4ffff640 1 -- 172.21.15.33:0/1422102516 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffb5809c630 con 0x7ffb68100e20 2024-09-16T00:18:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:43 smithi033 ceph-mon[31243]: pgmap v397: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:18:43.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:43 smithi083 ceph-mon[35242]: pgmap v397: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:18:43.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:43 smithi060 ceph-mon[35185]: pgmap v397: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:18:43.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.516+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/1422102516 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb6806a560 con 0x7ffb68100e20 2024-09-16T00:18:43.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.517+0000 7ffb4ffff640 1 -- 172.21.15.33:0/1422102516 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11903 (secure 0 0 0) 0x7ffb6806a560 con 0x7ffb68100e20 2024-09-16T00:18:43.518 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:18:43.519 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":26,"btime":"2024-09-16T00:17:28:789138+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:17:28.493434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:18:43.519 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 26 2024-09-16T00:18:43.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.520+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/1422102516 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffb500a3240 msgr2=0x7ffb500a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:43.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.520+0000 7ffb6d5a1640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffb500a3240 0x7ffb500a5700 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00ffd0 tx=0x7ffb5c00f7a0 comp rx=0 tx=0).stop 2024-09-16T00:18:43.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.520+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/1422102516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb68100e20 msgr2=0x7ffb681992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:43.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.520+0000 7ffb6d5a1640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb68100e20 0x7ffb681992b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800d950 tx=0x7ffb5800de10 comp rx=0 tx=0).stop 2024-09-16T00:18:43.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.520+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/1422102516 shutdown_connections 2024-09-16T00:18:43.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.520+0000 7ffb6d5a1640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffb500a3240 0x7ffb500a5700 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:43.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.520+0000 7ffb6d5a1640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb68102ef0 0x7ffb681a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:43.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.520+0000 7ffb6d5a1640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb68102570 0x7ffb681997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:43.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.520+0000 7ffb6d5a1640 1 --2- 172.21.15.33:0/1422102516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb68100e20 0x7ffb681992b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:43.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.520+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/1422102516 >> 172.21.15.33:0/1422102516 conn(0x7ffb680fc7d0 msgr2=0x7ffb68106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:43.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.520+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/1422102516 shutdown_connections 2024-09-16T00:18:43.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:43.521+0000 7ffb6d5a1640 1 -- 172.21.15.33:0/1422102516 wait complete. 2024-09-16T00:18:44.272 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 18ba3a83 --release --await' 2024-09-16T00:18:44.272 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 18ba3a83 --release --await 2024-09-16T00:18:44.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:44 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1422102516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:18:44.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:44 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1422102516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:18:44.560 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:18:44.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:44 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1422102516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:18:45.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.077+0000 7f18da303640 1 -- 172.21.15.33:0/2036440935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18d4101df0 msgr2=0x7f18d41021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:45.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.077+0000 7f18da303640 1 --2- 172.21.15.33:0/2036440935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18d4101df0 0x7f18d41021d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009900 tx=0x7f18c402a410 comp rx=0 tx=0).stop 2024-09-16T00:18:45.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.077+0000 7f18da303640 1 -- 172.21.15.33:0/2036440935 shutdown_connections 2024-09-16T00:18:45.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.077+0000 7f18da303640 1 --2- 172.21.15.33:0/2036440935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d4106f20 0x7f18d410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.077+0000 7f18da303640 1 --2- 172.21.15.33:0/2036440935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18d41027a0 0x7f18d41067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.077+0000 7f18da303640 1 --2- 172.21.15.33:0/2036440935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18d4101df0 0x7f18d41021d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.077+0000 7f18da303640 1 -- 172.21.15.33:0/2036440935 >> 172.21.15.33:0/2036440935 conn(0x7f18d40fd8f0 msgr2=0x7f18d40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:45.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.077+0000 7f18da303640 1 -- 172.21.15.33:0/2036440935 shutdown_connections 2024-09-16T00:18:45.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.077+0000 7f18da303640 1 -- 172.21.15.33:0/2036440935 wait complete. 2024-09-16T00:18:45.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.078+0000 7f18da303640 1 Processor -- start 2024-09-16T00:18:45.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.078+0000 7f18da303640 1 -- start start 2024-09-16T00:18:45.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.078+0000 7f18da303640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18d4101df0 0x7f18d4198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.078+0000 7f18da303640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d41027a0 0x7f18d4198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.078+0000 7f18da303640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18d4106f20 0x7f18d419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18da303640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18d4111f40 con 0x7f18d4101df0 2024-09-16T00:18:45.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18da303640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f18d4111dc0 con 0x7f18d41027a0 2024-09-16T00:18:45.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18da303640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f18d41120c0 con 0x7f18d4106f20 2024-09-16T00:18:45.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18d37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d41027a0 0x7f18d4198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18d4101df0 0x7f18d4198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18d37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d41027a0 0x7f18d4198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:39166/0 (socket says 172.21.15.33:39166) 2024-09-16T00:18:45.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18d37fe640 1 -- 172.21.15.33:0/1047434903 learned_addr learned my addr 172.21.15.33:0/1047434903 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:45.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18d88a0640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18d4106f20 0x7f18d419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18d3fff640 1 -- 172.21.15.33:0/1047434903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18d4106f20 msgr2=0x7f18d419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:45.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18d3fff640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18d4106f20 0x7f18d419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18d3fff640 1 -- 172.21.15.33:0/1047434903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d41027a0 msgr2=0x7f18d4198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:45.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18d3fff640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d41027a0 0x7f18d4198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18d3fff640 1 -- 172.21.15.33:0/1047434903 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d419d3f0 con 0x7f18d4101df0 2024-09-16T00:18:45.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.079+0000 7f18d3fff640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18d4101df0 0x7f18d4198440 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f18c4005940 tx=0x7f18c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:45.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.080+0000 7f18d17fa640 1 -- 172.21.15.33:0/1047434903 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c403f030 con 0x7f18d4101df0 2024-09-16T00:18:45.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.080+0000 7f18da303640 1 -- 172.21.15.33:0/1047434903 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d419d680 con 0x7f18d4101df0 2024-09-16T00:18:45.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.080+0000 7f18d17fa640 1 -- 172.21.15.33:0/1047434903 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18c4003ce0 con 0x7f18d4101df0 2024-09-16T00:18:45.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.080+0000 7f18d17fa640 1 -- 172.21.15.33:0/1047434903 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4035070 con 0x7f18d4101df0 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.080+0000 7f18da303640 1 -- 172.21.15.33:0/1047434903 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d41a4fc0 con 0x7f18d4101df0 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.084+0000 7f18dc6dc740 1 Processor -- start 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.084+0000 7f18dc6dc740 1 -- start start 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.084+0000 7f18d17fa640 1 -- 172.21.15.33:0/1047434903 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f18c4042030 con 0x7f18d4101df0 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.084+0000 7f18dc6dc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02af5e0 0x55f0a02adf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.084+0000 7f18dc6dc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02ae440 0x55f0a02ae880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.084+0000 7f18dc6dc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 0x55f0a02c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.084+0000 7f18dc6dc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f0a0213620 con 0x55f0a02af5e0 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.084+0000 7f18dc6dc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f0a01bae50 con 0x55f0a02c18e0 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.084+0000 7f18dc6dc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f0a020f5e0 con 0x55f0a02ae440 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.084+0000 7f18d37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 0x55f0a02c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.085+0000 7f18d37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 0x55f0a02c3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:39176/0 (socket says 172.21.15.33:39176) 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.085+0000 7f18d37fe640 1 -- 172.21.15.33:0/4159005049 learned_addr learned my addr 172.21.15.33:0/4159005049 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:45.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.085+0000 7f18d88a0640 1 --2- 172.21.15.33:0/4159005049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02af5e0 0x55f0a02adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.085+0000 7f18d3fff640 1 --2- 172.21.15.33:0/4159005049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02ae440 0x55f0a02ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.085+0000 7f18d37fe640 1 -- 172.21.15.33:0/4159005049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02ae440 msgr2=0x55f0a02ae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.085+0000 7f18d37fe640 1 --2- 172.21.15.33:0/4159005049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02ae440 0x55f0a02ae880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.085+0000 7f18d37fe640 1 -- 172.21.15.33:0/4159005049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02af5e0 msgr2=0x55f0a02adf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.085+0000 7f18d37fe640 1 --2- 172.21.15.33:0/4159005049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02af5e0 0x55f0a02adf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.085+0000 7f18d37fe640 1 -- 172.21.15.33:0/4159005049 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0a01abd40 con 0x55f0a02c18e0 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.085+0000 7f18d37fe640 1 --2- 172.21.15.33:0/4159005049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 0x55f0a02c3cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f18c800bd90 tx=0x7f18c80107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e24f7b30171897f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.086+0000 7f18ba7fc640 1 -- 172.21.15.33:0/4159005049 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c8011640 con 0x55f0a02c18e0 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.086+0000 7f18ba7fc640 1 -- 172.21.15.33:0/4159005049 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18c8011cc0 con 0x55f0a02c18e0 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.086+0000 7f18d17fa640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18bc0a32c0 0x7f18bc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.086+0000 7f18d17fa640 1 -- 172.21.15.33:0/1047434903 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f18c40f0570 con 0x7f18d4101df0 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.086+0000 7f18d88a0640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18bc0a32c0 0x7f18bc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.087+0000 7f18ba7fc640 1 -- 172.21.15.33:0/4159005049 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c801bc90 con 0x55f0a02c18e0 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.087+0000 7f18dc6dc740 1 -- 172.21.15.33:0/4159005049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 msgr2=0x55f0a02c3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.087+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/4159005049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 0x55f0a02c3cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f18c800bd90 tx=0x7f18c80107a0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.087+0000 7f18d88a0640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18bc0a32c0 0x7f18bc0a5780 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f18cc005fd0 tx=0x7f18cc0096e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.087+0000 7f18dc6dc740 1 -- 172.21.15.33:0/4159005049 shutdown_connections 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.087+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/4159005049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 0x55f0a02c3cd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.087+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/4159005049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02ae440 0x55f0a02ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.087+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/4159005049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02af5e0 0x55f0a02adf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.087+0000 7f18dc6dc740 1 -- 172.21.15.33:0/4159005049 >> 172.21.15.33:0/4159005049 conn(0x55f0a02aa5f0 msgr2=0x55f0a02aa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.087+0000 7f18dc6dc740 1 -- 172.21.15.33:0/4159005049 shutdown_connections 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.087+0000 7f18dc6dc740 1 -- 172.21.15.33:0/4159005049 wait complete. 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.087+0000 7f18dc6dc740 1 -- start start 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.088+0000 7f18dc6dc740 1 Processor -- start 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.088+0000 7f18dc6dc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02ae440 0x55f0a02b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.088+0000 7f18dc6dc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02af5e0 0x55f0a02b2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.088+0000 7f18dc6dc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 0x55f0a02b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.088+0000 7f18dc6dc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f0a02c4660 con 0x55f0a02af5e0 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.088+0000 7f18dc6dc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f0a019bf10 con 0x55f0a02c18e0 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.088+0000 7f18dc6dc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f0a02c47e0 con 0x55f0a02ae440 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.088+0000 7f18d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02af5e0 0x55f0a02b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.088+0000 7f18d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02af5e0 0x55f0a02b2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33696/0 (socket says 172.21.15.33:33696) 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.088+0000 7f18d37fe640 1 -- 172.21.15.33:0/1656091507 learned_addr learned my addr 172.21.15.33:0/1656091507 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.088+0000 7f18dc6dc740 10 client.0 _ll_register_callbacks cb 0x55f0a01bfdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.088+0000 7f18d88a0640 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 0x55f0a02b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18d37fe640 1 -- 172.21.15.33:0/1656091507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02ae440 msgr2=0x55f0a02b1bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18d3fff640 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02ae440 0x55f0a02b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18d37fe640 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02ae440 0x55f0a02b1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18d37fe640 1 -- 172.21.15.33:0/1656091507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 msgr2=0x55f0a02b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:45.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18d37fe640 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 0x55f0a02b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18d37fe640 1 -- 172.21.15.33:0/1656091507 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f09ff22580 con 0x55f0a02af5e0 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18d88a0640 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 0x55f0a02b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18d37fe640 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02af5e0 0x55f0a02b2130 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f18c800bf60 tx=0x7f18c801b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332bece2c4eb5a68 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18b37fe640 1 -- 172.21.15.33:0/1656091507 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c8029060 con 0x55f0a02af5e0 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1656091507 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18c8011cc0 con 0x55f0a02af5e0 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18b37fe640 1 -- 172.21.15.33:0/1656091507 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f18c801bc40 con 0x55f0a02af5e0 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18b37fe640 1 -- 172.21.15.33:0/1656091507 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f18c8030050 con 0x55f0a02af5e0 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.089+0000 7f18dc6dc740 10 client.15603 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.090+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1656091507 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f09ffec960 con 0x55f0a02af5e0 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.090+0000 7f18b37fe640 1 -- 172.21.15.33:0/1656091507 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18c8011cc0 con 0x55f0a02af5e0 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.090+0000 7f18b37fe640 1 -- 172.21.15.33:0/1656091507 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f18c80118d0 con 0x55f0a02af5e0 2024-09-16T00:18:45.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.090+0000 7f18dc6dc740 10 client.15603 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.090+0000 7f18dc6dc740 10 client.15603 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.090+0000 7f18dc6dc740 20 client.15603 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.090+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f0a02d14f0 0x55f0a02d38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.090+0000 7f18dc6dc740 4 client.15603 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.090+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1656091507 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0a0220b90 con 0x55f0a02d14f0 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.091+0000 7f18d3fff640 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f0a02d14f0 0x55f0a02d38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.091+0000 7f18b37fe640 10 client.15603 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.091+0000 7f18d3fff640 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f0a02d14f0 0x55f0a02d38e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=131d0f91655e49d1 server_cookie=28376a217b547ebc in_seq=0 out_seq=0 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.092+0000 7f18b37fe640 1 -- 172.21.15.33:0/1656091507 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f0a0220b90 con 0x55f0a02d14f0 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.092+0000 7f18b37fe640 10 client.15603 handle_command_reply: tid=0 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.092+0000 7f18dc6dc740 1 client.15603 shutdown 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.092+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1656091507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02af5e0 msgr2=0x55f0a02b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.092+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02af5e0 0x55f0a02b2130 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f18c800bf60 tx=0x7f18c801b810 comp rx=0 tx=0).stop 2024-09-16T00:18:45.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.092+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1656091507 shutdown_connections 2024-09-16T00:18:45.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.092+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f0a02d14f0 0x55f0a02d38e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.092+0000 7f18b37fe640 0 client.15603 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:45.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.092+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02c18e0 0x55f0a02b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.092+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02af5e0 0x55f0a02b2130 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.092+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/1656091507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02ae440 0x55f0a02b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.093+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1656091507 >> 172.21.15.33:0/1656091507 conn(0x55f0a02aa5f0 msgr2=0x55f0a02b0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:45.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.093+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1656091507 shutdown_connections 2024-09-16T00:18:45.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.093+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1656091507 wait complete. 2024-09-16T00:18:45.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.093+0000 7f18dc6dc740 20 client.15603 trim_cache size 0 max 16384 2024-09-16T00:18:45.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18dc6dc740 1 Processor -- start 2024-09-16T00:18:45.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18dc6dc740 1 -- start start 2024-09-16T00:18:45.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18dc6dc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02ae440 0x55f0a02a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18dc6dc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02af5e0 0x55f0a02b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18dc6dc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02c18e0 0x55f0a02a9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18dc6dc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18d41a6fd0 con 0x55f0a02ae440 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18dc6dc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f18d4075d30 con 0x55f0a02af5e0 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18dc6dc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f18d4073b80 con 0x55f0a02c18e0 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02ae440 0x55f0a02a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02ae440 0x55f0a02a52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33702/0 (socket says 172.21.15.33:33702) 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18d3fff640 1 -- 172.21.15.33:0/4146243802 learned_addr learned my addr 172.21.15.33:0/4146243802 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18d88a0640 1 --2- 172.21.15.33:0/4146243802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02c18e0 0x55f0a02a9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.111+0000 7f18d37fe640 1 --2- 172.21.15.33:0/4146243802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02af5e0 0x55f0a02b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.112+0000 7f18d3fff640 1 -- 172.21.15.33:0/4146243802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02c18e0 msgr2=0x55f0a02a9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.112+0000 7f18d3fff640 1 --2- 172.21.15.33:0/4146243802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02c18e0 0x55f0a02a9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.112+0000 7f18d3fff640 1 -- 172.21.15.33:0/4146243802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02af5e0 msgr2=0x55f0a02b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:45.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.112+0000 7f18d3fff640 1 --2- 172.21.15.33:0/4146243802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02af5e0 0x55f0a02b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.112+0000 7f18d3fff640 1 -- 172.21.15.33:0/4146243802 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0a02b3b30 con 0x55f0a02ae440 2024-09-16T00:18:45.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.112+0000 7f18d37fe640 1 --2- 172.21.15.33:0/4146243802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02af5e0 0x55f0a02b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:45.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.112+0000 7f18d3fff640 1 --2- 172.21.15.33:0/4146243802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02ae440 0x55f0a02a52b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009900 tx=0x7f18c402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f1eca3d209caa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:45.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.112+0000 7f18b9ffb640 1 -- 172.21.15.33:0/4146243802 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4049040 con 0x55f0a02ae440 2024-09-16T00:18:45.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.112+0000 7f18b9ffb640 1 -- 172.21.15.33:0/4146243802 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18c40bce10 con 0x55f0a02ae440 2024-09-16T00:18:45.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.112+0000 7f18b9ffb640 1 -- 172.21.15.33:0/4146243802 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c40baa10 con 0x55f0a02ae440 2024-09-16T00:18:45.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.113+0000 7f18dc6dc740 1 -- 172.21.15.33:0/4146243802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02ae440 msgr2=0x55f0a02a52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:45.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.113+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/4146243802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02ae440 0x55f0a02a52b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009900 tx=0x7f18c402a410 comp rx=0 tx=0).stop 2024-09-16T00:18:45.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.113+0000 7f18dc6dc740 1 -- 172.21.15.33:0/4146243802 shutdown_connections 2024-09-16T00:18:45.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.113+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/4146243802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02c18e0 0x55f0a02a9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.113+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/4146243802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02af5e0 0x55f0a02b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.113+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/4146243802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a02ae440 0x55f0a02a52b0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.113+0000 7f18dc6dc740 1 -- 172.21.15.33:0/4146243802 >> 172.21.15.33:0/4146243802 conn(0x55f0a02aa5f0 msgr2=0x55f0a02a8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:45.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.113+0000 7f18dc6dc740 1 -- 172.21.15.33:0/4146243802 shutdown_connections 2024-09-16T00:18:45.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.113+0000 7f18dc6dc740 1 -- 172.21.15.33:0/4146243802 wait complete. 2024-09-16T00:18:45.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.114+0000 7f18dc6dc740 1 -- start start 2024-09-16T00:18:45.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.114+0000 7f18dc6dc740 1 Processor -- start 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.114+0000 7f18dc6dc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02af5e0 0x55f0a02c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.114+0000 7f18dc6dc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02c18e0 0x55f0a02c6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.114+0000 7f18dc6dc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a0334eb0 0x55f0a0335330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18dc6dc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18d40741f0 con 0x55f0a0334eb0 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18dc6dc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f18d4075660 con 0x55f0a02af5e0 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18dc6dc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f18d40748c0 con 0x55f0a02c18e0 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18d37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02c18e0 0x55f0a02c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18d3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02af5e0 0x55f0a02c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18dc6dc740 10 client.0 _ll_register_callbacks cb 0x7f18c80058c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18d37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02c18e0 0x55f0a02c6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52460/0 (socket says 172.21.15.33:52460) 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18d37fe640 1 -- 172.21.15.33:0/1904706909 learned_addr learned my addr 172.21.15.33:0/1904706909 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18d88a0640 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a0334eb0 0x55f0a0335330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18d3fff640 1 -- 172.21.15.33:0/1904706909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02c18e0 msgr2=0x55f0a02c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:45.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18d3fff640 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02c18e0 0x55f0a02c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18d3fff640 1 -- 172.21.15.33:0/1904706909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a0334eb0 msgr2=0x55f0a0335330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:18:45.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18d3fff640 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a0334eb0 0x55f0a0335330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18d3fff640 1 -- 172.21.15.33:0/1904706909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0a03358a0 con 0x55f0a02af5e0 2024-09-16T00:18:45.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.115+0000 7f18d88a0640 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a0334eb0 0x55f0a0335330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:18:45.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18d3fff640 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02af5e0 0x55f0a02c6010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55f0a02b1a10 tx=0x7f18c40bc2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79bfc59c538ff9de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:18:45.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:18:45.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18b37fe640 1 -- 172.21.15.33:0/1904706909 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4049070 con 0x55f0a02af5e0 2024-09-16T00:18:45.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18b37fe640 1 -- 172.21.15.33:0/1904706909 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f18c403b460 con 0x55f0a02af5e0 2024-09-16T00:18:45.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1904706909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18c40bce10 con 0x55f0a02af5e0 2024-09-16T00:18:45.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18b37fe640 1 -- 172.21.15.33:0/1904706909 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f18c40bc070 con 0x55f0a02af5e0 2024-09-16T00:18:45.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18dc6dc740 10 client.25250 fetch_fsmap learned FSMap version 26 2024-09-16T00:18:45.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1904706909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f0a033cc40 con 0x55f0a02af5e0 2024-09-16T00:18:45.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18b37fe640 1 -- 172.21.15.33:0/1904706909 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18c40bc710 con 0x55f0a02af5e0 2024-09-16T00:18:45.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18b37fe640 1 -- 172.21.15.33:0/1904706909 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f18c40a5020 con 0x55f0a02af5e0 2024-09-16T00:18:45.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18dc6dc740 10 client.25250 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T00:18:45.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18dc6dc740 10 client.25250 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:18:45.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18dc6dc740 20 client.25250 populate_metadata read hostname 'smithi033' 2024-09-16T00:18:45.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f0a033d310 0x55f0a033f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:18:45.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18dc6dc740 4 client.25250 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "18ba3a83", "release": true, "await": true}] 2024-09-16T00:18:45.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.116+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1904706909 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "18ba3a83", "release": true, "await": true}) -- 0x7f18c40bce10 con 0x55f0a033d310 2024-09-16T00:18:45.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.117+0000 7f18d37fe640 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f0a033d310 0x55f0a033f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.117+0000 7f18b37fe640 10 client.25250 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.117+0000 7f18d37fe640 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f0a033d310 0x55f0a033f730 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f3e09cac6f4e99 server_cookie=7f54ea5d427b6262 in_seq=0 out_seq=0 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18b37fe640 1 -- 172.21.15.33:0/1904706909 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x7f18c40bce10 con 0x55f0a033d310 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18b37fe640 10 client.25250 handle_command_reply: tid=0 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18dc6dc740 1 client.25250 shutdown 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1904706909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02af5e0 msgr2=0x55f0a02c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02af5e0 0x55f0a02c6010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55f0a02b1a10 tx=0x7f18c40bc2d0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1904706909 shutdown_connections 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f0a033d310 0x55f0a033f730 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18b37fe640 0 client.25250 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f0a0334eb0 0x55f0a0335330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f0a02c18e0 0x55f0a02c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18dc6dc740 1 --2- 172.21.15.33:0/1904706909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0a02af5e0 0x55f0a02c6010 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1904706909 >> 172.21.15.33:0/1904706909 conn(0x55f0a02aa5f0 msgr2=0x55f0a02b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:45.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1904706909 shutdown_connections 2024-09-16T00:18:45.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18dc6dc740 1 -- 172.21.15.33:0/1904706909 wait complete. 2024-09-16T00:18:45.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.119+0000 7f18dc6dc740 20 client.25250 trim_cache size 0 max 16384 2024-09-16T00:18:45.123 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:18:45.123 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 26, 2024-09-16T00:18:45.123 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:18:45.123 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 4, 2024-09-16T00:18:45.123 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:18:45.123 INFO:teuthology.orchestra.run.smithi033.stdout: "18ba3a83": { 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 4, 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 150.2, 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:18:45.124 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:18:45.125 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:18:45.125 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:18:45.125 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:18:45.125 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:18:45.125 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:18:45.125 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:18:45.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.120+0000 7f18da303640 1 -- 172.21.15.33:0/1047434903 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18bc0a32c0 msgr2=0x7f18bc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:45.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.120+0000 7f18da303640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18bc0a32c0 0x7f18bc0a5780 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f18cc005fd0 tx=0x7f18cc0096e0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.120+0000 7f18da303640 1 -- 172.21.15.33:0/1047434903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18d4101df0 msgr2=0x7f18d4198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:18:45.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.120+0000 7f18da303640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18d4101df0 0x7f18d4198440 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f18c4005940 tx=0x7f18c40042f0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.120+0000 7f18da303640 1 -- 172.21.15.33:0/1047434903 shutdown_connections 2024-09-16T00:18:45.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.120+0000 7f18da303640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18bc0a32c0 0x7f18bc0a5780 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.120+0000 7f18da303640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18d4106f20 0x7f18d419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.120+0000 7f18da303640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d41027a0 0x7f18d4198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.120+0000 7f18da303640 1 --2- 172.21.15.33:0/1047434903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18d4101df0 0x7f18d4198440 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:18:45.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.120+0000 7f18da303640 1 -- 172.21.15.33:0/1047434903 >> 172.21.15.33:0/1047434903 conn(0x7f18d40fd8f0 msgr2=0x7f18d40fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:18:45.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.120+0000 7f18da303640 1 -- 172.21.15.33:0/1047434903 shutdown_connections 2024-09-16T00:18:45.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:18:45.120+0000 7f18da303640 1 -- 172.21.15.33:0/1047434903 wait complete. 2024-09-16T00:18:45.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:45 smithi033 ceph-mon[31243]: pgmap v398: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:18:45.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:45 smithi083 ceph-mon[35242]: pgmap v398: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:18:45.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:45 smithi060 ceph-mon[35185]: pgmap v398: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:18:45.872 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '18ba3a83', total seconds elapsed: 33.8 2024-09-16T00:18:47.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:47 smithi083 ceph-mon[35242]: pgmap v399: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:18:47.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:47 smithi060 ceph-mon[35185]: pgmap v399: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:18:47.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:47 smithi033 ceph-mon[31243]: pgmap v399: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:18:49.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:49 smithi083 ceph-mon[35242]: pgmap v400: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-09-16T00:18:49.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:49 smithi060 ceph-mon[35185]: pgmap v400: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-09-16T00:18:49.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:49 smithi033 ceph-mon[31243]: pgmap v400: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-09-16T00:18:50.665 INFO:tasks.workunit.client.0.smithi033.stderr:Updating files: 94% (12214/12889) Updating files: 95% (12245/12889) Updating files: 96% (12374/12889) Updating files: 97% (12503/12889) Updating files: 98% (12632/12889) Updating files: 99% (12761/12889) Updating files: 100% (12889/12889) Updating files: 100% (12889/12889), done. 2024-09-16T00:18:51.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:51 smithi083 ceph-mon[35242]: pgmap v401: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:18:51.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:51 smithi060 ceph-mon[35185]: pgmap v401: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:18:51.503 INFO:tasks.workunit.client.0.smithi033.stderr:Note: switching to 'ad35db06e8924912f01e32b35576297c0f1352b4'. 2024-09-16T00:18:51.504 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-09-16T00:18:51.504 INFO:tasks.workunit.client.0.smithi033.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T00:18:51.504 INFO:tasks.workunit.client.0.smithi033.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T00:18:51.504 INFO:tasks.workunit.client.0.smithi033.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T00:18:51.504 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-09-16T00:18:51.504 INFO:tasks.workunit.client.0.smithi033.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T00:18:51.504 INFO:tasks.workunit.client.0.smithi033.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T00:18:51.504 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-09-16T00:18:51.504 INFO:tasks.workunit.client.0.smithi033.stderr: git switch -c 2024-09-16T00:18:51.504 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-09-16T00:18:51.504 INFO:tasks.workunit.client.0.smithi033.stderr:Or undo this operation with: 2024-09-16T00:18:51.505 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-09-16T00:18:51.505 INFO:tasks.workunit.client.0.smithi033.stderr: git switch - 2024-09-16T00:18:51.505 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-09-16T00:18:51.505 INFO:tasks.workunit.client.0.smithi033.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T00:18:51.505 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-09-16T00:18:51.505 INFO:tasks.workunit.client.0.smithi033.stderr:HEAD is now at ad35db06e89 Merge pull request #59794 from zdover23/wip-doc-2024-09-15-backport-59613-to-squid 2024-09-16T00:18:51.509 DEBUG:teuthology.orchestra.run.smithi033:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T00:18:51.541 INFO:tasks.workunit.client.0.smithi033.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T00:18:51.543 INFO:tasks.workunit.client.0.smithi033.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T00:18:51.543 INFO:tasks.workunit.client.0.smithi033.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T00:18:51.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:51 smithi033 ceph-mon[31243]: pgmap v401: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-09-16T00:18:52.249 INFO:tasks.workunit.client.0.smithi033.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T00:18:52.318 INFO:tasks.workunit.client.0.smithi033.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T00:18:52.360 INFO:tasks.workunit.client.0.smithi033.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T00:18:52.362 INFO:tasks.workunit.client.0.smithi033.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T00:18:52.362 INFO:tasks.workunit.client.0.smithi033.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T00:18:52.402 INFO:tasks.workunit.client.0.smithi033.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T00:18:52.405 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:18:52.405 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T00:18:52.463 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-09-16T00:18:52.464 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-09-16T00:18:52.464 DEBUG:teuthology.orchestra.run.smithi033:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-09-16T00:18:52.526 INFO:tasks.workunit.client.0.smithi033.stderr:+ dbench 1 2024-09-16T00:18:52.536 INFO:tasks.workunit.client.0.smithi033.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-16T00:18:52.536 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-09-16T00:18:52.536 INFO:tasks.workunit.client.0.smithi033.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-16T00:18:52.543 INFO:tasks.workunit.client.0.smithi033.stdout:0 of 1 processes prepared for launch 0 sec 2024-09-16T00:18:52.643 INFO:tasks.workunit.client.0.smithi033.stdout:1 of 1 processes prepared for launch 0 sec 2024-09-16T00:18:52.643 INFO:tasks.workunit.client.0.smithi033.stdout:releasing clients 2024-09-16T00:18:53.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:53 smithi083 ceph-mon[35242]: pgmap v402: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-16T00:18:53.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:53 smithi060 ceph-mon[35185]: pgmap v402: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-16T00:18:53.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:53 smithi033 ceph-mon[31243]: pgmap v402: 129 pgs: 129 active+clean; 664 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-16T00:18:53.643 INFO:tasks.workunit.client.0.smithi033.stdout: 1 3880 39.76 MB/sec warmup 1 sec latency 7.911 ms 2024-09-16T00:18:54.643 INFO:tasks.workunit.client.0.smithi033.stdout: 1 7415 29.44 MB/sec warmup 2 sec latency 11.780 ms 2024-09-16T00:18:55.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:55 smithi083 ceph-mon[35242]: pgmap v403: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 52 KiB/s wr, 16 op/s 2024-09-16T00:18:55.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:55 smithi060 ceph-mon[35185]: pgmap v403: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 52 KiB/s wr, 16 op/s 2024-09-16T00:18:55.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:55 smithi033 ceph-mon[31243]: pgmap v403: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 52 KiB/s wr, 16 op/s 2024-09-16T00:18:55.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 10895 26.00 MB/sec warmup 3 sec latency 11.946 ms 2024-09-16T00:18:56.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 14314 24.02 MB/sec warmup 4 sec latency 15.399 ms 2024-09-16T00:18:57.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:57 smithi083 ceph-mon[35242]: pgmap v404: 129 pgs: 129 active+clean; 2.2 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 151 KiB/s wr, 21 op/s 2024-09-16T00:18:57.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:57 smithi060 ceph-mon[35185]: pgmap v404: 129 pgs: 129 active+clean; 2.2 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 151 KiB/s wr, 21 op/s 2024-09-16T00:18:57.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:57 smithi033 ceph-mon[31243]: pgmap v404: 129 pgs: 129 active+clean; 2.2 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 151 KiB/s wr, 21 op/s 2024-09-16T00:18:57.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 15175 19.66 MB/sec warmup 5 sec latency 471.258 ms 2024-09-16T00:18:58.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:58 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:18:58.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:58 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:18:58.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:58 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:18:58.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 15175 16.38 MB/sec warmup 6 sec latency 1471.363 ms 2024-09-16T00:18:59.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:18:59 smithi083 ceph-mon[35242]: pgmap v405: 129 pgs: 129 active+clean; 2.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 189 KiB/s wr, 22 op/s 2024-09-16T00:18:59.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:18:59 smithi060 ceph-mon[35185]: pgmap v405: 129 pgs: 129 active+clean; 2.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 189 KiB/s wr, 22 op/s 2024-09-16T00:18:59.481 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:18:59 smithi033 ceph-mon[31243]: pgmap v405: 129 pgs: 129 active+clean; 2.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 189 KiB/s wr, 22 op/s 2024-09-16T00:18:59.612 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:18:59.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 15175 14.04 MB/sec warmup 7 sec latency 2471.409 ms 2024-09-16T00:18:59.890 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:00.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.381+0000 7f2052da4640 1 -- 172.21.15.33:0/1032384344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f204c100a70 msgr2=0x7f204c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:00.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.381+0000 7f2052da4640 1 --2- 172.21.15.33:0/1032384344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f204c100a70 0x7f204c100ed0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2038009930 tx=0x7f203802a420 comp rx=0 tx=0).stop 2024-09-16T00:19:00.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.381+0000 7f2052da4640 1 -- 172.21.15.33:0/1032384344 shutdown_connections 2024-09-16T00:19:00.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.381+0000 7f2052da4640 1 --2- 172.21.15.33:0/1032384344 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f204c101410 0x7f204c10d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.381+0000 7f2052da4640 1 --2- 172.21.15.33:0/1032384344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f204c100a70 0x7f204c100ed0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.381+0000 7f2052da4640 1 --2- 172.21.15.33:0/1032384344 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f204c106a70 0x7f204c106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.381+0000 7f2052da4640 1 -- 172.21.15.33:0/1032384344 >> 172.21.15.33:0/1032384344 conn(0x7f204c0fc790 msgr2=0x7f204c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:00.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.381+0000 7f2052da4640 1 -- 172.21.15.33:0/1032384344 shutdown_connections 2024-09-16T00:19:00.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.381+0000 7f2052da4640 1 -- 172.21.15.33:0/1032384344 wait complete. 2024-09-16T00:19:00.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.382+0000 7f2052da4640 1 Processor -- start 2024-09-16T00:19:00.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.382+0000 7f2052da4640 1 -- start start 2024-09-16T00:19:00.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.382+0000 7f2052da4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f204c100a70 0x7f204c0fce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.382+0000 7f2052da4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f204c101410 0x7f204c0fd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.382+0000 7f2052da4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f204c106a70 0x7f204c0fd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.382+0000 7f2052da4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f204c110e70 con 0x7f204c101410 2024-09-16T00:19:00.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.382+0000 7f2052da4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f204c110cf0 con 0x7f204c100a70 2024-09-16T00:19:00.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.382+0000 7f2052da4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f204c110ff0 con 0x7f204c106a70 2024-09-16T00:19:00.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.382+0000 7f204bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f204c101410 0x7f204c0fd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.382+0000 7f204bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f204c101410 0x7f204c0fd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50166/0 (socket says 172.21.15.33:50166) 2024-09-16T00:19:00.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.382+0000 7f204bfff640 1 -- 172.21.15.33:0/4224221298 learned_addr learned my addr 172.21.15.33:0/4224221298 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:00.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f2051341640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f204c106a70 0x7f204c0fd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f2050b40640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f204c100a70 0x7f204c0fce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f204bfff640 1 -- 172.21.15.33:0/4224221298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f204c106a70 msgr2=0x7f204c0fd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:00.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f204bfff640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f204c106a70 0x7f204c0fd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f204bfff640 1 -- 172.21.15.33:0/4224221298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f204c100a70 msgr2=0x7f204c0fce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:00.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f204bfff640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f204c100a70 0x7f204c0fce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f204bfff640 1 -- 172.21.15.33:0/4224221298 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204c1a68b0 con 0x7f204c101410 2024-09-16T00:19:00.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f204bfff640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f204c101410 0x7f204c0fd360 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2038009230 tx=0x7f2038004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f2049ffb640 1 -- 172.21.15.33:0/4224221298 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203803f030 con 0x7f204c101410 2024-09-16T00:19:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f2052da4640 1 -- 172.21.15.33:0/4224221298 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f204c1a6b40 con 0x7f204c101410 2024-09-16T00:19:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f2049ffb640 1 -- 172.21.15.33:0/4224221298 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2038004440 con 0x7f204c101410 2024-09-16T00:19:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f2049ffb640 1 -- 172.21.15.33:0/4224221298 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038035070 con 0x7f204c101410 2024-09-16T00:19:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.383+0000 7f2052da4640 1 -- 172.21.15.33:0/4224221298 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f204c1a7020 con 0x7f204c101410 2024-09-16T00:19:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.386+0000 7f2049ffb640 1 -- 172.21.15.33:0/4224221298 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2038042030 con 0x7f204c101410 2024-09-16T00:19:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.386+0000 7f2049ffb640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f203c0a3240 0x7f203c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.386+0000 7f2049ffb640 1 -- 172.21.15.33:0/4224221298 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f20380f0230 con 0x7f204c101410 2024-09-16T00:19:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.386+0000 7f2050b40640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f203c0a3240 0x7f203c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.387+0000 7f205517d740 1 Processor -- start 2024-09-16T00:19:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.387+0000 7f2050b40640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f203c0a3240 0x7f203c0a5700 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2040009ae0 tx=0x7f2040005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.387+0000 7f205517d740 1 -- start start 2024-09-16T00:19:00.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.387+0000 7f205517d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e9564424a0 0x55e956442880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.387+0000 7f205517d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e956442dc0 0x55e95644d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.387+0000 7f205517d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95644d680 0x55e95644fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.387+0000 7f205517d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e9563a82e0 con 0x55e95644d680 2024-09-16T00:19:00.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.387+0000 7f205517d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e95634fce0 con 0x55e9564424a0 2024-09-16T00:19:00.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.387+0000 7f205517d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e9563a42a0 con 0x55e956442dc0 2024-09-16T00:19:00.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.387+0000 7f2051341640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95644d680 0x55e95644fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.387+0000 7f2051341640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95644d680 0x55e95644fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50178/0 (socket says 172.21.15.33:50178) 2024-09-16T00:19:00.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.387+0000 7f2051341640 1 -- 172.21.15.33:0/806871337 learned_addr learned my addr 172.21.15.33:0/806871337 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:00.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.388+0000 7f2050b40640 1 --2- 172.21.15.33:0/806871337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e9564424a0 0x55e956442880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.388+0000 7f204bfff640 1 --2- 172.21.15.33:0/806871337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e956442dc0 0x55e95644d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.388+0000 7f2051341640 1 -- 172.21.15.33:0/806871337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e956442dc0 msgr2=0x55e95644d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:00.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.388+0000 7f2051341640 1 --2- 172.21.15.33:0/806871337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e956442dc0 0x55e95644d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.388+0000 7f2051341640 1 -- 172.21.15.33:0/806871337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e9564424a0 msgr2=0x55e956442880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:00.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.388+0000 7f2051341640 1 --2- 172.21.15.33:0/806871337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e9564424a0 0x55e956442880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.388+0000 7f2051341640 1 -- 172.21.15.33:0/806871337 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e956334440 con 0x55e95644d680 2024-09-16T00:19:00.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.388+0000 7f2050b40640 1 --2- 172.21.15.33:0/806871337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e9564424a0 0x55e956442880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:00.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.388+0000 7f2051341640 1 --2- 172.21.15.33:0/806871337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95644d680 0x55e95644fa70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2044009fc0 tx=0x7f2044010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae9a775f83d8005 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:00.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.388+0000 7f2036ffd640 1 -- 172.21.15.33:0/806871337 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044003b70 con 0x55e95644d680 2024-09-16T00:19:00.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.388+0000 7f2036ffd640 1 -- 172.21.15.33:0/806871337 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2044003d10 con 0x55e95644d680 2024-09-16T00:19:00.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.388+0000 7f2036ffd640 1 -- 172.21.15.33:0/806871337 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044011ae0 con 0x55e95644d680 2024-09-16T00:19:00.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.389+0000 7f205517d740 1 -- 172.21.15.33:0/806871337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95644d680 msgr2=0x55e95644fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:00.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.389+0000 7f205517d740 1 --2- 172.21.15.33:0/806871337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95644d680 0x55e95644fa70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2044009fc0 tx=0x7f2044010860 comp rx=0 tx=0).stop 2024-09-16T00:19:00.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.389+0000 7f205517d740 1 -- 172.21.15.33:0/806871337 shutdown_connections 2024-09-16T00:19:00.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.389+0000 7f205517d740 1 --2- 172.21.15.33:0/806871337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95644d680 0x55e95644fa70 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.389+0000 7f205517d740 1 --2- 172.21.15.33:0/806871337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e956442dc0 0x55e95644d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.389+0000 7f205517d740 1 --2- 172.21.15.33:0/806871337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e9564424a0 0x55e956442880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.389+0000 7f205517d740 1 -- 172.21.15.33:0/806871337 >> 172.21.15.33:0/806871337 conn(0x55e95643ea30 msgr2=0x55e95643ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.389+0000 7f205517d740 1 -- 172.21.15.33:0/806871337 shutdown_connections 2024-09-16T00:19:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.389+0000 7f205517d740 1 -- 172.21.15.33:0/806871337 wait complete. 2024-09-16T00:19:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.389+0000 7f205517d740 1 -- start start 2024-09-16T00:19:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f205517d740 1 Processor -- start 2024-09-16T00:19:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f205517d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9564424a0 0x55e95644cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f205517d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 0x55e95644a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f205517d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e95644d680 0x55e95644a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f205517d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e956450250 con 0x55e956442dc0 2024-09-16T00:19:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f205517d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e956363050 con 0x55e95644d680 2024-09-16T00:19:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f205517d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e9564503d0 con 0x55e9564424a0 2024-09-16T00:19:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f205517d740 10 client.0 _ll_register_callbacks cb 0x55e9563a3160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f204bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 0x55e95644a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f204bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 0x55e95644a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50182/0 (socket says 172.21.15.33:50182) 2024-09-16T00:19:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f204bfff640 1 -- 172.21.15.33:0/3883116257 learned_addr learned my addr 172.21.15.33:0/3883116257 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f2051341640 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e95644d680 0x55e95644a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f2050b40640 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9564424a0 0x55e95644cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f204bfff640 1 -- 172.21.15.33:0/3883116257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9564424a0 msgr2=0x55e95644cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f204bfff640 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9564424a0 0x55e95644cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f204bfff640 1 -- 172.21.15.33:0/3883116257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e95644d680 msgr2=0x55e95644a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f204bfff640 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e95644d680 0x55e95644a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.390+0000 7f204bfff640 1 -- 172.21.15.33:0/3883116257 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9561193c0 con 0x55e956442dc0 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f204bfff640 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 0x55e95644a1f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f2038009900 tx=0x7f20380bbd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ef6b8f6940df99 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f202bfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f202bfff640 1 -- 172.21.15.33:0/3883116257 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20380d5060 con 0x55e956442dc0 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f202bfff640 1 -- 172.21.15.33:0/3883116257 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f20380d5200 con 0x55e956442dc0 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f205517d740 1 -- 172.21.15.33:0/3883116257 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2044003d10 con 0x55e956442dc0 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f202bfff640 1 -- 172.21.15.33:0/3883116257 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2038049040 con 0x55e956442dc0 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f205517d740 10 client.15621 fetch_fsmap learned FSMap version 27 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f205517d740 1 -- 172.21.15.33:0/3883116257 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e95645bba0 con 0x55e956442dc0 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f202bfff640 1 -- 172.21.15.33:0/3883116257 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2044003d10 con 0x55e956442dc0 2024-09-16T00:19:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f202bfff640 1 -- 172.21.15.33:0/3883116257 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f20380c4b50 con 0x55e956442dc0 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f205517d740 10 client.15621 fetch_fsmap finished waiting for FSMap version 27 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f205517d740 10 client.15621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f205517d740 20 client.15621 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.391+0000 7f205517d740 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e95645d000 0x55e95645f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.392+0000 7f205517d740 4 client.15621 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.392+0000 7f205517d740 1 -- 172.21.15.33:0/3883116257 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9563b5850 con 0x55e95645d000 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.392+0000 7f2050b40640 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e95645d000 0x55e95645f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.392+0000 7f202bfff640 10 client.15621 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.392+0000 7f2050b40640 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e95645d000 0x55e95645f460 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e138f120bddd0e0d server_cookie=6749927e813cc642 in_seq=0 out_seq=0 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f202bfff640 1 -- 172.21.15.33:0/3883116257 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e9563b5850 con 0x55e95645d000 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f202bfff640 10 client.15621 handle_command_reply: tid=0 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f205517d740 1 client.15621 shutdown 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f205517d740 1 -- 172.21.15.33:0/3883116257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 msgr2=0x55e95644a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f205517d740 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 0x55e95644a1f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f2038009900 tx=0x7f20380bbd90 comp rx=0 tx=0).stop 2024-09-16T00:19:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f205517d740 1 -- 172.21.15.33:0/3883116257 shutdown_connections 2024-09-16T00:19:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f205517d740 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e95645d000 0x55e95645f460 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f205517d740 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e95644d680 0x55e95644a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f205517d740 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 0x55e95644a1f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f202bfff640 0 client.15621 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f205517d740 1 --2- 172.21.15.33:0/3883116257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9564424a0 0x55e95644cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f205517d740 1 -- 172.21.15.33:0/3883116257 >> 172.21.15.33:0/3883116257 conn(0x55e95643ea30 msgr2=0x55e956444710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f205517d740 1 -- 172.21.15.33:0/3883116257 shutdown_connections 2024-09-16T00:19:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.393+0000 7f205517d740 1 -- 172.21.15.33:0/3883116257 wait complete. 2024-09-16T00:19:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.394+0000 7f205517d740 20 client.15621 trim_cache size 0 max 16384 2024-09-16T00:19:00.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.427+0000 7f205517d740 1 Processor -- start 2024-09-16T00:19:00.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f205517d740 1 -- start start 2024-09-16T00:19:00.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f205517d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9564424a0 0x55e95644c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f205517d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 0x55e956464d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f205517d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e95644d680 0x55e9564652b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f205517d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f204c1b3b90 con 0x55e956442dc0 2024-09-16T00:19:00.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f205517d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f204c1b1460 con 0x55e95644d680 2024-09-16T00:19:00.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f205517d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f204c1af2b0 con 0x55e9564424a0 2024-09-16T00:19:00.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f204bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 0x55e956464d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f204bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 0x55e956464d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50188/0 (socket says 172.21.15.33:50188) 2024-09-16T00:19:00.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f204bfff640 1 -- 172.21.15.33:0/433832917 learned_addr learned my addr 172.21.15.33:0/433832917 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:00.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f2050b40640 1 --2- 172.21.15.33:0/433832917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9564424a0 0x55e95644c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f2051341640 1 --2- 172.21.15.33:0/433832917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e95644d680 0x55e9564652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f204bfff640 1 -- 172.21.15.33:0/433832917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9564424a0 msgr2=0x55e95644c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:00.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f204bfff640 1 --2- 172.21.15.33:0/433832917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9564424a0 0x55e95644c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f204bfff640 1 -- 172.21.15.33:0/433832917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e95644d680 msgr2=0x55e9564652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:00.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f204bfff640 1 --2- 172.21.15.33:0/433832917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e95644d680 0x55e9564652b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f204bfff640 1 -- 172.21.15.33:0/433832917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e95645fae0 con 0x55e956442dc0 2024-09-16T00:19:00.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.428+0000 7f2051341640 1 --2- 172.21.15.33:0/433832917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e95644d680 0x55e9564652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:00.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.429+0000 7f204bfff640 1 --2- 172.21.15.33:0/433832917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 0x55e956464d70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2038000c00 tx=0x7f20380ba880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896e06c68747e164 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:00.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.429+0000 7f20367fc640 1 -- 172.21.15.33:0/433832917 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20380b89e0 con 0x55e956442dc0 2024-09-16T00:19:00.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.429+0000 7f20367fc640 1 -- 172.21.15.33:0/433832917 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2038003d20 con 0x55e956442dc0 2024-09-16T00:19:00.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.429+0000 7f20367fc640 1 -- 172.21.15.33:0/433832917 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038031bd0 con 0x55e956442dc0 2024-09-16T00:19:00.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.430+0000 7f205517d740 1 -- 172.21.15.33:0/433832917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 msgr2=0x55e956464d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:00.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.430+0000 7f205517d740 1 --2- 172.21.15.33:0/433832917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 0x55e956464d70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2038000c00 tx=0x7f20380ba880 comp rx=0 tx=0).stop 2024-09-16T00:19:00.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.430+0000 7f205517d740 1 -- 172.21.15.33:0/433832917 shutdown_connections 2024-09-16T00:19:00.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.430+0000 7f205517d740 1 --2- 172.21.15.33:0/433832917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e95644d680 0x55e9564652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.430+0000 7f205517d740 1 --2- 172.21.15.33:0/433832917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e956442dc0 0x55e956464d70 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.430+0000 7f205517d740 1 --2- 172.21.15.33:0/433832917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9564424a0 0x55e95644c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.430+0000 7f205517d740 1 -- 172.21.15.33:0/433832917 >> 172.21.15.33:0/433832917 conn(0x55e95643ea30 msgr2=0x55e95644ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.430+0000 7f205517d740 1 -- 172.21.15.33:0/433832917 shutdown_connections 2024-09-16T00:19:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.430+0000 7f205517d740 1 -- 172.21.15.33:0/433832917 wait complete. 2024-09-16T00:19:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.430+0000 7f205517d740 1 -- start start 2024-09-16T00:19:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.430+0000 7f205517d740 1 Processor -- start 2024-09-16T00:19:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f205517d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9564424a0 0x55e956455cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f205517d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e956442dc0 0x55e956453040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f205517d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e95644d680 0x55e956453580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f205517d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f204c1af920 con 0x55e9564424a0 2024-09-16T00:19:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f205517d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f204c1b0d90 con 0x55e956442dc0 2024-09-16T00:19:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f205517d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f204c1afff0 con 0x55e95644d680 2024-09-16T00:19:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f205517d740 10 client.0 _ll_register_callbacks cb 0x7f20380b86e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f2050b40640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9564424a0 0x55e956455cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f2050b40640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9564424a0 0x55e956455cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50200/0 (socket says 172.21.15.33:50200) 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f2050b40640 1 -- 172.21.15.33:0/3485405552 learned_addr learned my addr 172.21.15.33:0/3485405552 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f204bfff640 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e956442dc0 0x55e956453040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f2051341640 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e95644d680 0x55e956453580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f204bfff640 1 -- 172.21.15.33:0/3485405552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e95644d680 msgr2=0x55e956453580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f204bfff640 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e95644d680 0x55e956453580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f204bfff640 1 -- 172.21.15.33:0/3485405552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9564424a0 msgr2=0x55e956455cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f204bfff640 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9564424a0 0x55e956455cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f204bfff640 1 -- 172.21.15.33:0/3485405552 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e956453db0 con 0x55e956442dc0 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.431+0000 7f2050b40640 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9564424a0 0x55e956455cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.432+0000 7f204bfff640 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e956442dc0 0x55e956453040 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55e95643e480 tx=0x7f20380bbd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee90bf5387c2cf7b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.432+0000 7f202bfff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:19:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.432+0000 7f202bfff640 1 -- 172.21.15.33:0/3485405552 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20380d5040 con 0x55e956442dc0 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.432+0000 7f205517d740 1 -- 172.21.15.33:0/3485405552 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2038003d20 con 0x55e956442dc0 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.432+0000 7f202bfff640 1 -- 172.21.15.33:0/3485405552 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2038031bd0 con 0x55e956442dc0 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.432+0000 7f202bfff640 1 -- 172.21.15.33:0/3485405552 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f20380c4fe0 con 0x55e956442dc0 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.432+0000 7f205517d740 10 client.25268 fetch_fsmap learned FSMap version 27 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.432+0000 7f205517d740 1 -- 172.21.15.33:0/3485405552 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e956453db0 con 0x55e956442dc0 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.432+0000 7f202bfff640 1 -- 172.21.15.33:0/3485405552 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2038003d20 con 0x55e956442dc0 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.432+0000 7f202bfff640 1 -- 172.21.15.33:0/3485405552 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f203803bb50 con 0x55e956442dc0 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.433+0000 7f205517d740 10 client.25268 fetch_fsmap finished waiting for FSMap version 27 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.433+0000 7f205517d740 10 client.25268 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.433+0000 7f205517d740 20 client.25268 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.433+0000 7f205517d740 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e9564c8090 0x55e9564ca480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.433+0000 7f205517d740 4 client.25268 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:19:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.433+0000 7f205517d740 1 -- 172.21.15.33:0/3485405552 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e9563afd60 con 0x55e9564c8090 2024-09-16T00:19:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.433+0000 7f2050b40640 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e9564c8090 0x55e9564ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.433+0000 7f2050b40640 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e9564c8090 0x55e9564ca480 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a3d3e3a0c8c5964 server_cookie=90ced090b0849b59 in_seq=0 out_seq=0 2024-09-16T00:19:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.433+0000 7f202bfff640 10 client.25268 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f202bfff640 1 -- 172.21.15.33:0/3485405552 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e9563afd60 con 0x55e9564c8090 2024-09-16T00:19:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f202bfff640 10 client.25268 handle_command_reply: tid=0 2024-09-16T00:19:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f205517d740 1 client.25268 shutdown 2024-09-16T00:19:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f205517d740 1 -- 172.21.15.33:0/3485405552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e956442dc0 msgr2=0x55e956453040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f205517d740 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e956442dc0 0x55e956453040 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55e95643e480 tx=0x7f20380bbd90 comp rx=0 tx=0).stop 2024-09-16T00:19:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f205517d740 1 -- 172.21.15.33:0/3485405552 shutdown_connections 2024-09-16T00:19:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f205517d740 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e9564c8090 0x55e9564ca480 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f202bfff640 0 client.25268 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f205517d740 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e95644d680 0x55e956453580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f205517d740 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e956442dc0 0x55e956453040 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f205517d740 1 --2- 172.21.15.33:0/3485405552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9564424a0 0x55e956455cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f205517d740 1 -- 172.21.15.33:0/3485405552 >> 172.21.15.33:0/3485405552 conn(0x55e95643ea30 msgr2=0x55e95643d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f205517d740 1 -- 172.21.15.33:0/3485405552 shutdown_connections 2024-09-16T00:19:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f205517d740 1 -- 172.21.15.33:0/3485405552 wait complete. 2024-09-16T00:19:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.434+0000 7f205517d740 20 client.25268 trim_cache size 0 max 16384 2024-09-16T00:19:00.441 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:19:00.441 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:19:00.442 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:19:00.442 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:19:00.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.435+0000 7f2052da4640 1 -- 172.21.15.33:0/4224221298 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f203c0a3240 msgr2=0x7f203c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.435+0000 7f2052da4640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f203c0a3240 0x7f203c0a5700 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2040009ae0 tx=0x7f2040005f90 comp rx=0 tx=0).stop 2024-09-16T00:19:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.435+0000 7f2052da4640 1 -- 172.21.15.33:0/4224221298 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f204c101410 msgr2=0x7f204c0fd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.435+0000 7f2052da4640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f204c101410 0x7f204c0fd360 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2038009230 tx=0x7f2038004040 comp rx=0 tx=0).stop 2024-09-16T00:19:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.435+0000 7f2052da4640 1 -- 172.21.15.33:0/4224221298 shutdown_connections 2024-09-16T00:19:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.435+0000 7f2052da4640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f203c0a3240 0x7f203c0a5700 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.435+0000 7f2052da4640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f204c106a70 0x7f204c0fd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.435+0000 7f2052da4640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f204c101410 0x7f204c0fd360 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.435+0000 7f2052da4640 1 --2- 172.21.15.33:0/4224221298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f204c100a70 0x7f204c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.435+0000 7f2052da4640 1 -- 172.21.15.33:0/4224221298 >> 172.21.15.33:0/4224221298 conn(0x7f204c0fc790 msgr2=0x7f204c06a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.435+0000 7f2052da4640 1 -- 172.21.15.33:0/4224221298 shutdown_connections 2024-09-16T00:19:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:00.435+0000 7f2052da4640 1 -- 172.21.15.33:0/4224221298 wait complete. 2024-09-16T00:19:00.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 15175 12.29 MB/sec warmup 8 sec latency 3471.456 ms 2024-09-16T00:19:01.188 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:19:01.188 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:19:01.252 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:01 smithi033 ceph-mon[31243]: pgmap v406: 129 pgs: 129 active+clean; 6.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 643 KiB/s wr, 86 op/s 2024-09-16T00:19:01.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:01 smithi083 ceph-mon[35242]: pgmap v406: 129 pgs: 129 active+clean; 6.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 643 KiB/s wr, 86 op/s 2024-09-16T00:19:01.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:01 smithi060 ceph-mon[35185]: pgmap v406: 129 pgs: 129 active+clean; 6.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 643 KiB/s wr, 86 op/s 2024-09-16T00:19:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:01.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 15175 10.92 MB/sec warmup 9 sec latency 4471.535 ms 2024-09-16T00:19:01.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.954+0000 7fb2470d0640 1 -- 172.21.15.33:0/1528164282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb240101420 msgr2=0x7fb24010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.954+0000 7fb2470d0640 1 --2- 172.21.15.33:0/1528164282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb240101420 0x7fb24010d950 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb238009930 tx=0x7fb23802a420 comp rx=0 tx=0).stop 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.954+0000 7fb2470d0640 1 -- 172.21.15.33:0/1528164282 shutdown_connections 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.954+0000 7fb2470d0640 1 --2- 172.21.15.33:0/1528164282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb240101420 0x7fb24010d950 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.954+0000 7fb2470d0640 1 --2- 172.21.15.33:0/1528164282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb240100a80 0x7fb240100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.954+0000 7fb2470d0640 1 --2- 172.21.15.33:0/1528164282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb240106a80 0x7fb240106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.954+0000 7fb2470d0640 1 -- 172.21.15.33:0/1528164282 >> 172.21.15.33:0/1528164282 conn(0x7fb2400fc7a0 msgr2=0x7fb2400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.954+0000 7fb2470d0640 1 -- 172.21.15.33:0/1528164282 shutdown_connections 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.954+0000 7fb2470d0640 1 -- 172.21.15.33:0/1528164282 wait complete. 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.955+0000 7fb2470d0640 1 Processor -- start 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.955+0000 7fb2470d0640 1 -- start start 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.955+0000 7fb2470d0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb240100a80 0x7fb240198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.955+0000 7fb2470d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb240101420 0x7fb240198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:01.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.955+0000 7fb2470d0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb240106a80 0x7fb24019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.955+0000 7fb2470d0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb240110ea0 con 0x7fb240100a80 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.955+0000 7fb2470d0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb240110d20 con 0x7fb240106a80 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.955+0000 7fb2470d0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb240111020 con 0x7fb240101420 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.955+0000 7fb244e6c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb240100a80 0x7fb240198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.955+0000 7fb244e6c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb240100a80 0x7fb240198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50208/0 (socket says 172.21.15.33:50208) 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.955+0000 7fb244e6c640 1 -- 172.21.15.33:0/803910056 learned_addr learned my addr 172.21.15.33:0/803910056 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.956+0000 7fb24566d640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb240106a80 0x7fb24019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.956+0000 7fb23ffff640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb240101420 0x7fb240198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.956+0000 7fb244e6c640 1 -- 172.21.15.33:0/803910056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb240101420 msgr2=0x7fb240198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.956+0000 7fb244e6c640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb240101420 0x7fb240198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.956+0000 7fb244e6c640 1 -- 172.21.15.33:0/803910056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb240106a80 msgr2=0x7fb24019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.956+0000 7fb244e6c640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb240106a80 0x7fb24019cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:01.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.956+0000 7fb244e6c640 1 -- 172.21.15.33:0/803910056 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24019d3c0 con 0x7fb240100a80 2024-09-16T00:19:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.956+0000 7fb244e6c640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb240100a80 0x7fb240198410 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fb22800b4a0 tx=0x7fb22800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.956+0000 7fb23dffb640 1 -- 172.21.15.33:0/803910056 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb228014070 con 0x7fb240100a80 2024-09-16T00:19:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.956+0000 7fb23dffb640 1 -- 172.21.15.33:0/803910056 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb228010af0 con 0x7fb240100a80 2024-09-16T00:19:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.956+0000 7fb2470d0640 1 -- 172.21.15.33:0/803910056 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb24019d6b0 con 0x7fb240100a80 2024-09-16T00:19:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.956+0000 7fb23dffb640 1 -- 172.21.15.33:0/803910056 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb228017d10 con 0x7fb240100a80 2024-09-16T00:19:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.957+0000 7fb2470d0640 1 -- 172.21.15.33:0/803910056 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2401a4f90 con 0x7fb240100a80 2024-09-16T00:19:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.958+0000 7fb2470d0640 1 -- 172.21.15.33:0/803910056 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb240100ee0 con 0x7fb240100a80 2024-09-16T00:19:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.959+0000 7fb23dffb640 1 -- 172.21.15.33:0/803910056 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb228017070 con 0x7fb240100a80 2024-09-16T00:19:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.960+0000 7fb23dffb640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb22c0a32c0 0x7fb22c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.960+0000 7fb23dffb640 1 -- 172.21.15.33:0/803910056 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb2280cf3b0 con 0x7fb240100a80 2024-09-16T00:19:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.960+0000 7fb23ffff640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb22c0a32c0 0x7fb22c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.960+0000 7fb23ffff640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb22c0a32c0 0x7fb22c0a5780 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb2400fec40 tx=0x7fb23000a590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:01.963+0000 7fb23dffb640 1 -- 172.21.15.33:0/803910056 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb22809b600 con 0x7fb240100a80 2024-09-16T00:19:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.186+0000 7fb2470d0640 1 -- 172.21.15.33:0/803910056 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2401a5290 con 0x7fb240100a80 2024-09-16T00:19:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.186+0000 7fb23dffb640 1 -- 172.21.15.33:0/803910056 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11906 (secure 0 0 0) 0x7fb22809f900 con 0x7fb240100a80 2024-09-16T00:19:02.188 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:19:02.188 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"btime":"2024-09-16T00:19:02:115556+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:19:01.460559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:19:02.188 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 28 2024-09-16T00:19:02.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.189+0000 7fb2470d0640 1 -- 172.21.15.33:0/803910056 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb22c0a32c0 msgr2=0x7fb22c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:02.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.189+0000 7fb2470d0640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb22c0a32c0 0x7fb22c0a5780 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb2400fec40 tx=0x7fb23000a590 comp rx=0 tx=0).stop 2024-09-16T00:19:02.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.189+0000 7fb2470d0640 1 -- 172.21.15.33:0/803910056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb240100a80 msgr2=0x7fb240198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:02.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.189+0000 7fb2470d0640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb240100a80 0x7fb240198410 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fb22800b4a0 tx=0x7fb22800b960 comp rx=0 tx=0).stop 2024-09-16T00:19:02.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.189+0000 7fb2470d0640 1 -- 172.21.15.33:0/803910056 shutdown_connections 2024-09-16T00:19:02.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.189+0000 7fb2470d0640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb22c0a32c0 0x7fb22c0a5780 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:02.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.189+0000 7fb2470d0640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb240106a80 0x7fb24019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:02.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.189+0000 7fb2470d0640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb240101420 0x7fb240198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:02.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.189+0000 7fb2470d0640 1 --2- 172.21.15.33:0/803910056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb240100a80 0x7fb240198410 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:02.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.189+0000 7fb2470d0640 1 -- 172.21.15.33:0/803910056 >> 172.21.15.33:0/803910056 conn(0x7fb2400fc7a0 msgr2=0x7fb24006a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:02.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.189+0000 7fb2470d0640 1 -- 172.21.15.33:0/803910056 shutdown_connections 2024-09-16T00:19:02.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:02.189+0000 7fb2470d0640 1 -- 172.21.15.33:0/803910056 wait complete. 2024-09-16T00:19:02.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 17720 11.49 MB/sec warmup 10 sec latency 4820.517 ms 2024-09-16T00:19:02.906 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:19:03.215 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:03.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:03 smithi033 ceph-mon[31243]: pgmap v407: 129 pgs: 129 active+clean; 7.7 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 775 KiB/s wr, 98 op/s 2024-09-16T00:19:03.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:03 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:19:03.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:03 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/803910056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:19:03.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:03 smithi083 ceph-mon[35242]: pgmap v407: 129 pgs: 129 active+clean; 7.7 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 775 KiB/s wr, 98 op/s 2024-09-16T00:19:03.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:03 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:19:03.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:03 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/803910056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:19:03.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:03 smithi060 ceph-mon[35185]: pgmap v407: 129 pgs: 129 active+clean; 7.7 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 775 KiB/s wr, 98 op/s 2024-09-16T00:19:03.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:03 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:19:03.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:03 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/803910056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:19:03.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 18657 10.66 MB/sec warmup 11 sec latency 485.125 ms 2024-09-16T00:19:03.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.814+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1068813270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e84102790 msgr2=0x7f2e841067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.814+0000 7f2e8bc0f640 1 --2- 172.21.15.33:0/1068813270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e84102790 0x7f2e841067e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2e74009900 tx=0x7f2e7402a3f0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.814+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1068813270 shutdown_connections 2024-09-16T00:19:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.814+0000 7f2e8bc0f640 1 --2- 172.21.15.33:0/1068813270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e84106f10 0x7f2e8410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.814+0000 7f2e8bc0f640 1 --2- 172.21.15.33:0/1068813270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e84102790 0x7f2e841067e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.814+0000 7f2e8bc0f640 1 --2- 172.21.15.33:0/1068813270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e84101de0 0x7f2e841021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.815+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1068813270 >> 172.21.15.33:0/1068813270 conn(0x7f2e840fd8c0 msgr2=0x7f2e840ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.815+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1068813270 shutdown_connections 2024-09-16T00:19:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.815+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1068813270 wait complete. 2024-09-16T00:19:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.815+0000 7f2e8bc0f640 1 Processor -- start 2024-09-16T00:19:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.815+0000 7f2e8bc0f640 1 -- start start 2024-09-16T00:19:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.815+0000 7f2e8bc0f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e84101de0 0x7f2e840744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.815+0000 7f2e8bc0f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e84102790 0x7f2e84074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.815+0000 7f2e8bc0f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e84106f10 0x7f2e84074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.815+0000 7f2e8bc0f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2e84111f80 con 0x7f2e84106f10 2024-09-16T00:19:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.815+0000 7f2e8bc0f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2e84111e00 con 0x7f2e84102790 2024-09-16T00:19:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.815+0000 7f2e8bc0f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2e84112100 con 0x7f2e84101de0 2024-09-16T00:19:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e8a1ac640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e84106f10 0x7f2e84074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e8a1ac640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e84106f10 0x7f2e84074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50238/0 (socket says 172.21.15.33:50238) 2024-09-16T00:19:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e8a1ac640 1 -- 172.21.15.33:0/1002124174 learned_addr learned my addr 172.21.15.33:0/1002124174 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e891aa640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e84102790 0x7f2e84074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e899ab640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e84101de0 0x7f2e840744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e8a1ac640 1 -- 172.21.15.33:0/1002124174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e84101de0 msgr2=0x7f2e840744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e8a1ac640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e84101de0 0x7f2e840744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e8a1ac640 1 -- 172.21.15.33:0/1002124174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e84102790 msgr2=0x7f2e84074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e8a1ac640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e84102790 0x7f2e84074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e8a1ac640 1 -- 172.21.15.33:0/1002124174 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e841a68e0 con 0x7f2e84106f10 2024-09-16T00:19:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e8a1ac640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e84106f10 0x7f2e84074f70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c00cad0 tx=0x7f2e7c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e7affd640 1 -- 172.21.15.33:0/1002124174 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c00ad60 con 0x7f2e84106f10 2024-09-16T00:19:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e7affd640 1 -- 172.21.15.33:0/1002124174 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e7c005020 con 0x7f2e84106f10 2024-09-16T00:19:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1002124174 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e841a6bd0 con 0x7f2e84106f10 2024-09-16T00:19:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e7affd640 1 -- 172.21.15.33:0/1002124174 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c00f6b0 con 0x7f2e84106f10 2024-09-16T00:19:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.816+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1002124174 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e841a7090 con 0x7f2e84106f10 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.819+0000 7f2e7affd640 1 -- 172.21.15.33:0/1002124174 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2e7c005560 con 0x7f2e84106f10 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.819+0000 7f2e8dfe8740 1 Processor -- start 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.819+0000 7f2e8dfe8740 1 -- start start 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.819+0000 7f2e8dfe8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f692a6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.819+0000 7f2e8dfe8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f692a6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.819+0000 7f2e8dfe8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692b3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.819+0000 7f2e8dfe8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557f6920c2f0 con 0x557f692b1370 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.819+0000 7f2e8dfe8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x557f691b3cf0 con 0x557f692a65d0 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.819+0000 7f2e8dfe8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557f692082b0 con 0x557f692a7b40 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.819+0000 7f2e7affd640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2e700a3240 0x7f2e700a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e891aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e891aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50240/0 (socket says 172.21.15.33:50240) 2024-09-16T00:19:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e891aa640 1 -- 172.21.15.33:0/2873107439 learned_addr learned my addr 172.21.15.33:0/2873107439 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e8a1ac640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2e700a3240 0x7f2e700a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e899ab640 1 --2- 172.21.15.33:0/2873107439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f692a6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e7affd640 1 -- 172.21.15.33:0/1002124174 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2e7c0cfcd0 con 0x7f2e84106f10 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e891aa640 1 --2- 172.21.15.33:0/2873107439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f692a6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e891aa640 1 -- 172.21.15.33:0/2873107439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 msgr2=0x557f692a6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e891aa640 1 --2- 172.21.15.33:0/2873107439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f692a6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e891aa640 1 -- 172.21.15.33:0/2873107439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 msgr2=0x557f692a6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e891aa640 1 --2- 172.21.15.33:0/2873107439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f692a6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e891aa640 1 -- 172.21.15.33:0/2873107439 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f691983a0 con 0x557f692b1370 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e899ab640 1 --2- 172.21.15.33:0/2873107439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f692a6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e8a1ac640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2e700a3240 0x7f2e700a5700 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c09bb70 tx=0x7f2e7c0668b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e891aa640 1 --2- 172.21.15.33:0/2873107439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692b3760 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2e740096c0 tx=0x7f2e74031510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa5fbb5d5ccbcfa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.820+0000 7f2e6ffff640 1 -- 172.21.15.33:0/2873107439 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74035030 con 0x557f692b1370 2024-09-16T00:19:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.821+0000 7f2e6ffff640 1 -- 172.21.15.33:0/2873107439 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e74005020 con 0x557f692b1370 2024-09-16T00:19:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.821+0000 7f2e6ffff640 1 -- 172.21.15.33:0/2873107439 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74004ac0 con 0x557f692b1370 2024-09-16T00:19:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.821+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2873107439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 msgr2=0x557f692b3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.821+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/2873107439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692b3760 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2e740096c0 tx=0x7f2e74031510 comp rx=0 tx=0).stop 2024-09-16T00:19:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.821+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2873107439 shutdown_connections 2024-09-16T00:19:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.821+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/2873107439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692b3760 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.821+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/2873107439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f692a6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.821+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/2873107439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f692a6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.821+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2873107439 >> 172.21.15.33:0/2873107439 conn(0x557f6926f310 msgr2=0x557f692a2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.821+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2873107439 shutdown_connections 2024-09-16T00:19:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.821+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2873107439 wait complete. 2024-09-16T00:19:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.822+0000 7f2e8dfe8740 1 -- start start 2024-09-16T00:19:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.822+0000 7f2e8dfe8740 1 Processor -- start 2024-09-16T00:19:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.822+0000 7f2e8dfe8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a65d0 0x557f692b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.822+0000 7f2e8dfe8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a7b40 0x557f692adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.822+0000 7f2e8dfe8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692ae570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.822+0000 7f2e8dfe8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557f692b4010 con 0x557f692b1370 2024-09-16T00:19:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.822+0000 7f2e8dfe8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x557f691c7060 con 0x557f692a7b40 2024-09-16T00:19:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.822+0000 7f2e8dfe8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557f692b4190 con 0x557f692a65d0 2024-09-16T00:19:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.822+0000 7f2e8dfe8740 10 client.0 _ll_register_callbacks cb 0x557f68f1a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.822+0000 7f2e891aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.822+0000 7f2e891aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692ae570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50256/0 (socket says 172.21.15.33:50256) 2024-09-16T00:19:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.822+0000 7f2e891aa640 1 -- 172.21.15.33:0/2585447662 learned_addr learned my addr 172.21.15.33:0/2585447662 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e891aa640 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a65d0 0x557f692b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e899ab640 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a7b40 0x557f692adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e891aa640 1 -- 172.21.15.33:0/2585447662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a65d0 msgr2=0x557f692b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e891aa640 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a65d0 0x557f692b0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e891aa640 1 -- 172.21.15.33:0/2585447662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a7b40 msgr2=0x557f692adfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e891aa640 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a7b40 0x557f692adfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e891aa640 1 -- 172.21.15.33:0/2585447662 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f68f7d3c0 con 0x557f692b1370 2024-09-16T00:19:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e899ab640 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a7b40 0x557f692adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:19:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e891aa640 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692ae570 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2e74004800 tx=0x7f2e74009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95cfd3912758c020 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e6d7fa640 1 -- 172.21.15.33:0/2585447662 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74052060 con 0x557f692b1370 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e6d7fa640 1 -- 172.21.15.33:0/2585447662 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2e74003ac0 con 0x557f692b1370 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2585447662 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e74005020 con 0x557f692b1370 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e6d7fa640 1 -- 172.21.15.33:0/2585447662 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2e74035070 con 0x557f692b1370 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e8dfe8740 10 client.15645 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2585447662 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557f692bfac0 con 0x557f692b1370 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.823+0000 7f2e6d7fa640 1 -- 172.21.15.33:0/2585447662 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e74005020 con 0x557f692b1370 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.824+0000 7f2e6d7fa640 1 -- 172.21.15.33:0/2585447662 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2e74031810 con 0x557f692b1370 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.824+0000 7f2e8dfe8740 10 client.15645 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.824+0000 7f2e8dfe8740 10 client.15645 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.824+0000 7f2e8dfe8740 20 client.15645 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.824+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x557f692c0e10 0x557f692c3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.824+0000 7f2e8dfe8740 4 client.15645 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.824+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2585447662 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f69219860 con 0x557f692c0e10 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.824+0000 7f2e899ab640 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x557f692c0e10 0x557f692c3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.824+0000 7f2e899ab640 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x557f692c0e10 0x557f692c3200 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b7b1138fb635213d server_cookie=1db97c868257b5e in_seq=0 out_seq=0 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.824+0000 7f2e6d7fa640 10 client.15645 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.825+0000 7f2e6d7fa640 1 -- 172.21.15.33:0/2585447662 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557f69219860 con 0x557f692c0e10 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.825+0000 7f2e6d7fa640 10 client.15645 handle_command_reply: tid=0 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.825+0000 7f2e8dfe8740 1 client.15645 shutdown 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.825+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2585447662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 msgr2=0x557f692ae570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.825+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692ae570 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2e74004800 tx=0x7f2e74009260 comp rx=0 tx=0).stop 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.825+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2585447662 shutdown_connections 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.825+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x557f692c0e10 0x557f692c3200 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.825+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692ae570 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.825+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a7b40 0x557f692adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.825+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/2585447662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a65d0 0x557f692b0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.825+0000 7f2e6d7fa640 0 client.15645 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:19:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.825+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2585447662 >> 172.21.15.33:0/2585447662 conn(0x557f6926f310 msgr2=0x557f692a8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.826+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2585447662 shutdown_connections 2024-09-16T00:19:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.826+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/2585447662 wait complete. 2024-09-16T00:19:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.826+0000 7f2e8dfe8740 20 client.15645 trim_cache size 0 max 16384 2024-09-16T00:19:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.839+0000 7f2e8dfe8740 1 Processor -- start 2024-09-16T00:19:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.839+0000 7f2e8dfe8740 1 -- start start 2024-09-16T00:19:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.839+0000 7f2e8dfe8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f692acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.839+0000 7f2e8dfe8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f692a9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.839+0000 7f2e8dfe8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692a9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.839+0000 7f2e8dfe8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2e841b3b60 con 0x557f692b1370 2024-09-16T00:19:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.839+0000 7f2e8dfe8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2e841b1430 con 0x557f692a65d0 2024-09-16T00:19:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.839+0000 7f2e8dfe8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2e841af280 con 0x557f692a7b40 2024-09-16T00:19:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.839+0000 7f2e891aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.839+0000 7f2e891aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692a9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50260/0 (socket says 172.21.15.33:50260) 2024-09-16T00:19:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.839+0000 7f2e891aa640 1 -- 172.21.15.33:0/486239472 learned_addr learned my addr 172.21.15.33:0/486239472 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.839+0000 7f2e891aa640 1 --2- 172.21.15.33:0/486239472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f692acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.840+0000 7f2e899ab640 1 --2- 172.21.15.33:0/486239472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f692a9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.840+0000 7f2e891aa640 1 -- 172.21.15.33:0/486239472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 msgr2=0x557f692a9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.840+0000 7f2e891aa640 1 --2- 172.21.15.33:0/486239472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f692a9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.840+0000 7f2e891aa640 1 -- 172.21.15.33:0/486239472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 msgr2=0x557f692acfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.840+0000 7f2e891aa640 1 --2- 172.21.15.33:0/486239472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f692acfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.840+0000 7f2e891aa640 1 -- 172.21.15.33:0/486239472 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f692a9ba0 con 0x557f692b1370 2024-09-16T00:19:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.840+0000 7f2e899ab640 1 --2- 172.21.15.33:0/486239472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f692a9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:19:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.840+0000 7f2e891aa640 1 --2- 172.21.15.33:0/486239472 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692a9660 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2e740096c0 tx=0x7f2e74031510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d23d15c20a52a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.840+0000 7f2e6f7fe640 1 -- 172.21.15.33:0/486239472 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74052040 con 0x557f692b1370 2024-09-16T00:19:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.840+0000 7f2e6f7fe640 1 -- 172.21.15.33:0/486239472 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e74003de0 con 0x557f692b1370 2024-09-16T00:19:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.840+0000 7f2e6f7fe640 1 -- 172.21.15.33:0/486239472 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74004ab0 con 0x557f692b1370 2024-09-16T00:19:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.841+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/486239472 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 msgr2=0x557f692a9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.841+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/486239472 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692a9660 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2e740096c0 tx=0x7f2e74031510 comp rx=0 tx=0).stop 2024-09-16T00:19:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.841+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/486239472 shutdown_connections 2024-09-16T00:19:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.841+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/486239472 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692a9660 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.841+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/486239472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f692a9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.841+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/486239472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f692acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.841+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/486239472 >> 172.21.15.33:0/486239472 conn(0x557f6926f310 msgr2=0x557f692ac8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.841+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/486239472 shutdown_connections 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.841+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/486239472 wait complete. 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.841+0000 7f2e8dfe8740 1 -- start start 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.841+0000 7f2e8dfe8740 1 Processor -- start 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e8dfe8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f69322aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e8dfe8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f69322fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e8dfe8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e8dfe8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2e841af8f0 con 0x557f692b1370 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e8dfe8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2e841b0d60 con 0x557f692a65d0 2024-09-16T00:19:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e8dfe8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2e841affc0 con 0x557f692a7b40 2024-09-16T00:19:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e8dfe8740 10 client.0 _ll_register_callbacks cb 0x7f2e8000a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e891aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e891aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692c33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50264/0 (socket says 172.21.15.33:50264) 2024-09-16T00:19:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e891aa640 1 -- 172.21.15.33:0/3778391155 learned_addr learned my addr 172.21.15.33:0/3778391155 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e899ab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f69322fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e891aa640 1 --2- 172.21.15.33:0/3778391155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f69322aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e899ab640 1 -- 172.21.15.33:0/3778391155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 msgr2=0x557f69322aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e899ab640 1 --2- 172.21.15.33:0/3778391155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f69322aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e899ab640 1 -- 172.21.15.33:0/3778391155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 msgr2=0x557f692c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e899ab640 1 --2- 172.21.15.33:0/3778391155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.842+0000 7f2e899ab640 1 -- 172.21.15.33:0/3778391155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f692c3bf0 con 0x557f692a7b40 2024-09-16T00:19:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e899ab640 1 --2- 172.21.15.33:0/3778391155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f69322fe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x557f692b41d0 tx=0x7f2e80010b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ab3e62b46ca2eb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e6d7fa640 1 -- 172.21.15.33:0/3778391155 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8000fb50 con 0x557f692a7b40 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e6d7fa640 1 -- 172.21.15.33:0/3778391155 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2e8000fcf0 con 0x557f692a7b40 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/3778391155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e74003de0 con 0x557f692a7b40 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e6d7fa640 1 -- 172.21.15.33:0/3778391155 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2e8002c050 con 0x557f692a7b40 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e8dfe8740 10 client.25213 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/3778391155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557f6932b230 con 0x557f692a7b40 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e6d7fa640 1 -- 172.21.15.33:0/3778391155 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e74003de0 con 0x557f692a7b40 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e6d7fa640 1 -- 172.21.15.33:0/3778391155 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2e80004b40 con 0x557f692a7b40 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e8dfe8740 10 client.25213 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e8dfe8740 10 client.25213 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e8dfe8740 20 client.25213 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:03.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/3778391155 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x557f6932b950 0x557f6932dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e8dfe8740 4 client.25213 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.843+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/3778391155 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557f69213d70 con 0x557f6932b950 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e891aa640 1 --2- 172.21.15.33:0/3778391155 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x557f6932b950 0x557f6932dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e891aa640 1 --2- 172.21.15.33:0/3778391155 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x557f6932b950 0x557f6932dd40 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6d67e0d6c81679f8 server_cookie=ced93b39741cefc9 in_seq=0 out_seq=0 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e6d7fa640 10 client.25213 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e6d7fa640 1 -- 172.21.15.33:0/3778391155 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557f69213d70 con 0x557f6932b950 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e6d7fa640 10 client.25213 handle_command_reply: tid=0 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e8dfe8740 1 client.25213 shutdown 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/3778391155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 msgr2=0x557f69322fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/3778391155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f69322fe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x557f692b41d0 tx=0x7f2e80010b40 comp rx=0 tx=0).stop 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/3778391155 shutdown_connections 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/3778391155 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x557f6932b950 0x557f6932dd40 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/3778391155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f692b1370 0x557f692c33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/3778391155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f692a7b40 0x557f69322fe0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e8dfe8740 1 --2- 172.21.15.33:0/3778391155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557f692a65d0 0x557f69322aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e6d7fa640 0 client.25213 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:19:03.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.844+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/3778391155 >> 172.21.15.33:0/3778391155 conn(0x557f6926f310 msgr2=0x557f692ad0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:03.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/3778391155 shutdown_connections 2024-09-16T00:19:03.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8dfe8740 1 -- 172.21.15.33:0/3778391155 wait complete. 2024-09-16T00:19:03.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8dfe8740 20 client.25213 trim_cache size 0 max 16384 2024-09-16T00:19:03.856 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:19:03.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1002124174 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2e700a3240 msgr2=0x7f2e700a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:03.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8bc0f640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2e700a3240 0x7f2e700a5700 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c09bb70 tx=0x7f2e7c0668b0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1002124174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e84106f10 msgr2=0x7f2e84074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8bc0f640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e84106f10 0x7f2e84074f70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c00cad0 tx=0x7f2e7c00ced0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1002124174 shutdown_connections 2024-09-16T00:19:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8bc0f640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2e700a3240 0x7f2e700a5700 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8bc0f640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e84106f10 0x7f2e84074f70 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8bc0f640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e84102790 0x7f2e84074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8bc0f640 1 --2- 172.21.15.33:0/1002124174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e84101de0 0x7f2e840744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1002124174 >> 172.21.15.33:0/1002124174 conn(0x7f2e840fd8c0 msgr2=0x7f2e840fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1002124174 shutdown_connections 2024-09-16T00:19:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:03.845+0000 7f2e8bc0f640 1 -- 172.21.15.33:0/1002124174 wait complete. 2024-09-16T00:19:04.543 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:19:04.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 18657 9.77 MB/sec warmup 12 sec latency 1485.171 ms 2024-09-16T00:19:04.803 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:05.225 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:05 smithi033 ceph-mon[31243]: pgmap v408: 129 pgs: 129 active+clean; 7.9 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 805 KiB/s wr, 100 op/s 2024-09-16T00:19:05.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.307+0000 7f31cbf23640 1 -- 172.21.15.33:0/3040311651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31c4101de0 msgr2=0x7f31c41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:05.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.307+0000 7f31cbf23640 1 --2- 172.21.15.33:0/3040311651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31c4101de0 0x7f31c41021c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009900 tx=0x7f31c002a410 comp rx=0 tx=0).stop 2024-09-16T00:19:05.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.307+0000 7f31cbf23640 1 -- 172.21.15.33:0/3040311651 shutdown_connections 2024-09-16T00:19:05.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.307+0000 7f31cbf23640 1 --2- 172.21.15.33:0/3040311651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c4106f10 0x7f31c410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.307+0000 7f31cbf23640 1 --2- 172.21.15.33:0/3040311651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31c4102790 0x7f31c41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.307+0000 7f31cbf23640 1 --2- 172.21.15.33:0/3040311651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31c4101de0 0x7f31c41021c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.307+0000 7f31cbf23640 1 -- 172.21.15.33:0/3040311651 >> 172.21.15.33:0/3040311651 conn(0x7f31c40fd8c0 msgr2=0x7f31c40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:05.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.307+0000 7f31cbf23640 1 -- 172.21.15.33:0/3040311651 shutdown_connections 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.307+0000 7f31cbf23640 1 -- 172.21.15.33:0/3040311651 wait complete. 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31cbf23640 1 Processor -- start 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31cbf23640 1 -- start start 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31cbf23640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31c4101de0 0x7f31c41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31cbf23640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c4102790 0x7f31c4198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31cbf23640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31c4106f10 0x7f31c419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31cbf23640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31c4111f70 con 0x7f31c4106f10 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31cbf23640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f31c4111df0 con 0x7f31c4102790 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31cbf23640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f31c41120f0 con 0x7f31c4101de0 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31ca4c0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31c4106f10 0x7f31c419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31ca4c0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31c4106f10 0x7f31c419cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50288/0 (socket says 172.21.15.33:50288) 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31ca4c0640 1 -- 172.21.15.33:0/1961003194 learned_addr learned my addr 172.21.15.33:0/1961003194 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31c9cbf640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31c4101de0 0x7f31c41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31c94be640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c4102790 0x7f31c4198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31ca4c0640 1 -- 172.21.15.33:0/1961003194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31c4101de0 msgr2=0x7f31c41984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31ca4c0640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31c4101de0 0x7f31c41984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31ca4c0640 1 -- 172.21.15.33:0/1961003194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c4102790 msgr2=0x7f31c4198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31ca4c0640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c4102790 0x7f31c4198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.308+0000 7f31ca4c0640 1 -- 172.21.15.33:0/1961003194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c419d480 con 0x7f31c4106f10 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.309+0000 7f31ca4c0640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31c4106f10 0x7f31c419cda0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00eaa0 tx=0x7f31bc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.309+0000 7f31baffd640 1 -- 172.21.15.33:0/1961003194 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc00cdb0 con 0x7f31c4106f10 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.309+0000 7f31cbf23640 1 -- 172.21.15.33:0/1961003194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c419d770 con 0x7f31c4106f10 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.309+0000 7f31baffd640 1 -- 172.21.15.33:0/1961003194 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31bc005020 con 0x7f31c4106f10 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.309+0000 7f31baffd640 1 -- 172.21.15.33:0/1961003194 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc0116b0 con 0x7f31c4106f10 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.309+0000 7f31cbf23640 1 -- 172.21.15.33:0/1961003194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c41a5050 con 0x7f31c4106f10 2024-09-16T00:19:05.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.311+0000 7f31baffd640 1 -- 172.21.15.33:0/1961003194 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f31bc005560 con 0x7f31c4106f10 2024-09-16T00:19:05.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.311+0000 7f31ce2fc740 1 Processor -- start 2024-09-16T00:19:05.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31ce2fc740 1 -- start start 2024-09-16T00:19:05.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31ce2fc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df867f4b40 0x55df867f3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31ce2fc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f35d0 0x55df867f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31ce2fc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df86802780 0x55df86804b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31ce2fc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55df867592f0 con 0x55df86802780 2024-09-16T00:19:05.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31ce2fc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55df86700cf0 con 0x55df867f35d0 2024-09-16T00:19:05.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31ce2fc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55df867552b0 con 0x55df867f4b40 2024-09-16T00:19:05.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31c94be640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df86802780 0x55df86804b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31c94be640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df86802780 0x55df86804b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50298/0 (socket says 172.21.15.33:50298) 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31c94be640 1 -- 172.21.15.33:0/3571461412 learned_addr learned my addr 172.21.15.33:0/3571461412 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31c9cbf640 1 --2- 172.21.15.33:0/3571461412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f35d0 0x55df867f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31baffd640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f31b00a3240 0x7f31b00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31c94be640 1 --2- 172.21.15.33:0/3571461412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df867f4b40 0x55df867f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31baffd640 1 -- 172.21.15.33:0/1961003194 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f31bc0cff80 con 0x7f31c4106f10 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31ca4c0640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f31b00a3240 0x7f31b00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31c94be640 1 -- 172.21.15.33:0/3571461412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df867f4b40 msgr2=0x55df867f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31c94be640 1 --2- 172.21.15.33:0/3571461412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df867f4b40 0x55df867f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31c94be640 1 -- 172.21.15.33:0/3571461412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f35d0 msgr2=0x55df867f3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31c94be640 1 --2- 172.21.15.33:0/3571461412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f35d0 0x55df867f3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.312+0000 7f31c94be640 1 -- 172.21.15.33:0/3571461412 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df866e53a0 con 0x55df86802780 2024-09-16T00:19:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.313+0000 7f31c94be640 1 --2- 172.21.15.33:0/3571461412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df86802780 0x55df86804b70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f31b4013910 tx=0x7f31b400e430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99cfae97bc9d62c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.313+0000 7f31ca4c0640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f31b00a3240 0x7f31b00a5700 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f31bc098af0 tx=0x7f31bc009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.313+0000 7f31affff640 1 -- 172.21.15.33:0/3571461412 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b4004020 con 0x55df86802780 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.313+0000 7f31affff640 1 -- 172.21.15.33:0/3571461412 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31b400bdf0 con 0x55df86802780 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.313+0000 7f31affff640 1 -- 172.21.15.33:0/3571461412 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b40039f0 con 0x55df86802780 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.314+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3571461412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df86802780 msgr2=0x55df86804b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.314+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/3571461412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df86802780 0x55df86804b70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f31b4013910 tx=0x7f31b400e430 comp rx=0 tx=0).stop 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.314+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3571461412 shutdown_connections 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.314+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/3571461412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df86802780 0x55df86804b70 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.314+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/3571461412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f35d0 0x55df867f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.314+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/3571461412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df867f4b40 0x55df867f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.314+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3571461412 >> 172.21.15.33:0/3571461412 conn(0x55df867bc310 msgr2=0x55df867efc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.314+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3571461412 shutdown_connections 2024-09-16T00:19:05.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.314+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3571461412 wait complete. 2024-09-16T00:19:05.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.314+0000 7f31ce2fc740 1 -- start start 2024-09-16T00:19:05.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.314+0000 7f31ce2fc740 1 Processor -- start 2024-09-16T00:19:05.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31ce2fc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 0x55df86802150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31ce2fc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df867f4b40 0x55df867ff400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31ce2fc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df86802780 0x55df867ff9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31ce2fc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55df86805480 con 0x55df867f35d0 2024-09-16T00:19:05.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31ce2fc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55df86714060 con 0x55df86802780 2024-09-16T00:19:05.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31ce2fc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55df86805600 con 0x55df867f4b40 2024-09-16T00:19:05.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31ce2fc740 10 client.0 _ll_register_callbacks cb 0x55df864678c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:05.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31c94be640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 0x55df86802150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31c94be640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 0x55df86802150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50304/0 (socket says 172.21.15.33:50304) 2024-09-16T00:19:05.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31c94be640 1 -- 172.21.15.33:0/3028333485 learned_addr learned my addr 172.21.15.33:0/3028333485 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:05.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31c9cbf640 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df867f4b40 0x55df867ff400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31c94be640 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df86802780 0x55df867ff9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31c94be640 1 -- 172.21.15.33:0/3028333485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df867f4b40 msgr2=0x55df867ff400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:05.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31c94be640 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df867f4b40 0x55df867ff400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31c94be640 1 -- 172.21.15.33:0/3028333485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df86802780 msgr2=0x55df867ff9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:05.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31c94be640 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df86802780 0x55df867ff9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31c94be640 1 -- 172.21.15.33:0/3028333485 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df864ca3c0 con 0x55df867f35d0 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31c94be640 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 0x55df86802150 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f31b400ee60 tx=0x7f31b400edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c95459cc5c5e5c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31ad7fa640 1 -- 172.21.15.33:0/3028333485 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b400bb10 con 0x55df867f35d0 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.315+0000 7f31ad7fa640 1 -- 172.21.15.33:0/3028333485 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f31b4005050 con 0x55df867f35d0 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3028333485 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31b400bdf0 con 0x55df867f35d0 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31ad7fa640 1 -- 172.21.15.33:0/3028333485 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f31b4031050 con 0x55df867f35d0 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31ce2fc740 10 client.15663 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3028333485 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55df86810ef0 con 0x55df867f35d0 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31ad7fa640 1 -- 172.21.15.33:0/3028333485 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31b400bdf0 con 0x55df867f35d0 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31ad7fa640 1 -- 172.21.15.33:0/3028333485 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f31b40046c0 con 0x55df867f35d0 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31ce2fc740 10 client.15663 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31ce2fc740 10 client.15663 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:19:05.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31ce2fc740 20 client.15663 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55df86812240 0x55df86814630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31ce2fc740 4 client.15663 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3028333485 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df86766860 con 0x55df86812240 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.316+0000 7f31c9cbf640 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55df86812240 0x55df86814630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.317+0000 7f31c9cbf640 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55df86812240 0x55df86814630 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a1a274a35aa7d31a server_cookie=40f24b8ac2b2fec0 in_seq=0 out_seq=0 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.317+0000 7f31ad7fa640 10 client.15663 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ad7fa640 1 -- 172.21.15.33:0/3028333485 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55df86766860 con 0x55df86812240 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ad7fa640 10 client.15663 handle_command_reply: tid=0 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ce2fc740 1 client.15663 shutdown 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3028333485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 msgr2=0x55df86802150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 0x55df86802150 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f31b400ee60 tx=0x7f31b400edb0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3028333485 shutdown_connections 2024-09-16T00:19:05.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55df86812240 0x55df86814630 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df86802780 0x55df867ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df867f4b40 0x55df867ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/3028333485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 0x55df86802150 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ad7fa640 0 client.15663 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:19:05.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3028333485 >> 172.21.15.33:0/3028333485 conn(0x55df867bc310 msgr2=0x55df867f5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:05.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3028333485 shutdown_connections 2024-09-16T00:19:05.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ce2fc740 1 -- 172.21.15.33:0/3028333485 wait complete. 2024-09-16T00:19:05.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.318+0000 7f31ce2fc740 20 client.15663 trim_cache size 0 max 16384 2024-09-16T00:19:05.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.331+0000 7f31ce2fc740 1 Processor -- start 2024-09-16T00:19:05.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.331+0000 7f31ce2fc740 1 -- start start 2024-09-16T00:19:05.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.331+0000 7f31ce2fc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 0x55df867f4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.331+0000 7f31ce2fc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f4b40 0x55df867eaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.331+0000 7f31ce2fc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df86802780 0x55df867eebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.331+0000 7f31ce2fc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31c41af750 con 0x55df867f35d0 2024-09-16T00:19:05.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.331+0000 7f31ce2fc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f31c41ad020 con 0x55df867f4b40 2024-09-16T00:19:05.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.331+0000 7f31ce2fc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f31c41aae70 con 0x55df86802780 2024-09-16T00:19:05.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31c94be640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 0x55df867f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31c94be640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 0x55df867f4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50310/0 (socket says 172.21.15.33:50310) 2024-09-16T00:19:05.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31c94be640 1 -- 172.21.15.33:0/4092625357 learned_addr learned my addr 172.21.15.33:0/4092625357 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:05.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31c9cbf640 1 --2- 172.21.15.33:0/4092625357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f4b40 0x55df867eaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31c94be640 1 --2- 172.21.15.33:0/4092625357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df86802780 0x55df867eebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31c94be640 1 -- 172.21.15.33:0/4092625357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df86802780 msgr2=0x55df867eebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:05.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31c94be640 1 --2- 172.21.15.33:0/4092625357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df86802780 0x55df867eebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31c94be640 1 -- 172.21.15.33:0/4092625357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f4b40 msgr2=0x55df867eaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:05.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31c94be640 1 --2- 172.21.15.33:0/4092625357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f4b40 0x55df867eaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31c94be640 1 -- 172.21.15.33:0/4092625357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df86801d70 con 0x55df867f35d0 2024-09-16T00:19:05.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31c94be640 1 --2- 172.21.15.33:0/4092625357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 0x55df867f4300 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f31b4006010 tx=0x7f31b400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=575a8c77160edf99 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:05.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31af7fe640 1 -- 172.21.15.33:0/4092625357 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b4004660 con 0x55df867f35d0 2024-09-16T00:19:05.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31af7fe640 1 -- 172.21.15.33:0/4092625357 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31b4028070 con 0x55df867f35d0 2024-09-16T00:19:05.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.332+0000 7f31af7fe640 1 -- 172.21.15.33:0/4092625357 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b4003b10 con 0x55df867f35d0 2024-09-16T00:19:05.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.333+0000 7f31ce2fc740 1 -- 172.21.15.33:0/4092625357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 msgr2=0x55df867f4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:05.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.333+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/4092625357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 0x55df867f4300 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f31b4006010 tx=0x7f31b400ea10 comp rx=0 tx=0).stop 2024-09-16T00:19:05.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.333+0000 7f31ce2fc740 1 -- 172.21.15.33:0/4092625357 shutdown_connections 2024-09-16T00:19:05.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.333+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/4092625357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df86802780 0x55df867eebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.333+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/4092625357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f4b40 0x55df867eaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.333+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/4092625357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f35d0 0x55df867f4300 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.333+0000 7f31ce2fc740 1 -- 172.21.15.33:0/4092625357 >> 172.21.15.33:0/4092625357 conn(0x55df867bc310 msgr2=0x55df867ee4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:05.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.333+0000 7f31ce2fc740 1 -- 172.21.15.33:0/4092625357 shutdown_connections 2024-09-16T00:19:05.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.333+0000 7f31ce2fc740 1 -- 172.21.15.33:0/4092625357 wait complete. 2024-09-16T00:19:05.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.333+0000 7f31ce2fc740 1 -- start start 2024-09-16T00:19:05.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.333+0000 7f31ce2fc740 1 Processor -- start 2024-09-16T00:19:05.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31ce2fc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f35d0 0x55df86814db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31ce2fc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f4b40 0x55df868152f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31ce2fc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df86802780 0x55df8687ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31ce2fc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31c41ab4e0 con 0x55df867f4b40 2024-09-16T00:19:05.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31ce2fc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f31c41ac950 con 0x55df867f35d0 2024-09-16T00:19:05.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31ce2fc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f31c41abbb0 con 0x55df86802780 2024-09-16T00:19:05.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31ce2fc740 10 client.0 _ll_register_callbacks cb 0x7f31b4013f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:05.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31c9cbf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f4b40 0x55df868152f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31c9cbf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f4b40 0x55df868152f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50322/0 (socket says 172.21.15.33:50322) 2024-09-16T00:19:05.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31c9cbf640 1 -- 172.21.15.33:0/1476624820 learned_addr learned my addr 172.21.15.33:0/1476624820 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:05.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31c94be640 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f35d0 0x55df86814db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31c94be640 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df86802780 0x55df8687ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31c9cbf640 1 -- 172.21.15.33:0/1476624820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df86802780 msgr2=0x55df8687ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:05.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31c9cbf640 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df86802780 0x55df8687ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31c9cbf640 1 -- 172.21.15.33:0/1476624820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f35d0 msgr2=0x55df86814db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31c9cbf640 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f35d0 0x55df86814db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31c9cbf640 1 -- 172.21.15.33:0/1476624820 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df8687b1b0 con 0x55df867f4b40 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.334+0000 7f31c94be640 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df86802780 0x55df8687ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31c9cbf640 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f4b40 0x55df868152f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009230 tx=0x7f31c0031990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e06995e8e66601a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ad7fa640 1 -- 172.21.15.33:0/1476624820 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c0035040 con 0x55df867f4b40 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ad7fa640 1 -- 172.21.15.33:0/1476624820 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f31c0004240 con 0x55df867f4b40 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ce2fc740 1 -- 172.21.15.33:0/1476624820 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31b4028070 con 0x55df867f4b40 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ad7fa640 1 -- 172.21.15.33:0/1476624820 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f31c0058050 con 0x55df867f4b40 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ce2fc740 10 client.15669 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ce2fc740 1 -- 172.21.15.33:0/1476624820 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55df8687c520 con 0x55df867f4b40 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ad7fa640 1 -- 172.21.15.33:0/1476624820 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31b4028070 con 0x55df867f4b40 2024-09-16T00:19:05.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ad7fa640 1 -- 172.21.15.33:0/1476624820 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f31c004f640 con 0x55df867f4b40 2024-09-16T00:19:05.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ce2fc740 10 client.15669 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:05.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ce2fc740 10 client.15669 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:19:05.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ce2fc740 20 client.15669 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:05.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55df8687ccd0 0x55df8687f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:05.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ce2fc740 4 client.15669 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:19:05.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.335+0000 7f31ce2fc740 1 -- 172.21.15.33:0/1476624820 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55df86760d70 con 0x55df8687ccd0 2024-09-16T00:19:05.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.336+0000 7f31c94be640 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55df8687ccd0 0x55df8687f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:05.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.336+0000 7f31c94be640 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55df8687ccd0 0x55df8687f0f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=345344ce3c48482a server_cookie=3aa6dadc563ad8b3 in_seq=0 out_seq=0 2024-09-16T00:19:05.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.336+0000 7f31ad7fa640 10 client.15669 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:19:05.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.336+0000 7f31ad7fa640 1 -- 172.21.15.33:0/1476624820 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55df86760d70 con 0x55df8687ccd0 2024-09-16T00:19:05.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.336+0000 7f31ad7fa640 10 client.15669 handle_command_reply: tid=0 2024-09-16T00:19:05.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.336+0000 7f31ce2fc740 1 client.15669 shutdown 2024-09-16T00:19:05.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.336+0000 7f31ce2fc740 1 -- 172.21.15.33:0/1476624820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f4b40 msgr2=0x55df868152f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:05.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.336+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f4b40 0x55df868152f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009230 tx=0x7f31c0031990 comp rx=0 tx=0).stop 2024-09-16T00:19:05.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31ce2fc740 1 -- 172.21.15.33:0/1476624820 shutdown_connections 2024-09-16T00:19:05.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55df86802780 0x55df8687ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df867f4b40 0x55df868152f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55df8687ccd0 0x55df8687f0f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31ce2fc740 1 --2- 172.21.15.33:0/1476624820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55df867f35d0 0x55df86814db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31ad7fa640 0 client.15669 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:19:05.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31ce2fc740 1 -- 172.21.15.33:0/1476624820 >> 172.21.15.33:0/1476624820 conn(0x55df867bc310 msgr2=0x55df867ee4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:05.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31ce2fc740 1 -- 172.21.15.33:0/1476624820 shutdown_connections 2024-09-16T00:19:05.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31ce2fc740 1 -- 172.21.15.33:0/1476624820 wait complete. 2024-09-16T00:19:05.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31ce2fc740 20 client.15669 trim_cache size 0 max 16384 2024-09-16T00:19:05.355 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:19:05.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31cbf23640 1 -- 172.21.15.33:0/1961003194 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f31b00a3240 msgr2=0x7f31b00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:05.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31cbf23640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f31b00a3240 0x7f31b00a5700 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f31bc098af0 tx=0x7f31bc009270 comp rx=0 tx=0).stop 2024-09-16T00:19:05.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31cbf23640 1 -- 172.21.15.33:0/1961003194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31c4106f10 msgr2=0x7f31c419cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:05.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31cbf23640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31c4106f10 0x7f31c419cda0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00eaa0 tx=0x7f31bc00ef60 comp rx=0 tx=0).stop 2024-09-16T00:19:05.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31cbf23640 1 -- 172.21.15.33:0/1961003194 shutdown_connections 2024-09-16T00:19:05.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31cbf23640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f31b00a3240 0x7f31b00a5700 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31cbf23640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31c4106f10 0x7f31c419cda0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31cbf23640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c4102790 0x7f31c4198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31cbf23640 1 --2- 172.21.15.33:0/1961003194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31c4101de0 0x7f31c41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:05.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31cbf23640 1 -- 172.21.15.33:0/1961003194 >> 172.21.15.33:0/1961003194 conn(0x7f31c40fd8c0 msgr2=0x7f31c40fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:05.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31cbf23640 1 -- 172.21.15.33:0/1961003194 shutdown_connections 2024-09-16T00:19:05.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:05.337+0000 7f31cbf23640 1 -- 172.21.15.33:0/1961003194 wait complete. 2024-09-16T00:19:05.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:05 smithi083 ceph-mon[35242]: pgmap v408: 129 pgs: 129 active+clean; 7.9 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 805 KiB/s wr, 100 op/s 2024-09-16T00:19:05.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:05 smithi060 ceph-mon[35185]: pgmap v408: 129 pgs: 129 active+clean; 7.9 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 805 KiB/s wr, 100 op/s 2024-09-16T00:19:05.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 18657 9.02 MB/sec warmup 13 sec latency 2485.222 ms 2024-09-16T00:19:06.146 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:19:06.403 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:06.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 18657 8.38 MB/sec warmup 14 sec latency 3485.252 ms 2024-09-16T00:19:06.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.974+0000 7fb3883e0640 1 -- 172.21.15.33:0/1294801943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb380101de0 msgr2=0x7fb3801021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:06.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.974+0000 7fb3883e0640 1 --2- 172.21.15.33:0/1294801943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb380101de0 0x7fb3801021c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009900 tx=0x7fb37c02a410 comp rx=0 tx=0).stop 2024-09-16T00:19:06.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.974+0000 7fb3883e0640 1 -- 172.21.15.33:0/1294801943 shutdown_connections 2024-09-16T00:19:06.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.974+0000 7fb3883e0640 1 --2- 172.21.15.33:0/1294801943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb380106f10 0x7fb38010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.974+0000 7fb3883e0640 1 --2- 172.21.15.33:0/1294801943 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb380102790 0x7fb3801067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.974+0000 7fb3883e0640 1 --2- 172.21.15.33:0/1294801943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb380101de0 0x7fb3801021c0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.974+0000 7fb3883e0640 1 -- 172.21.15.33:0/1294801943 >> 172.21.15.33:0/1294801943 conn(0x7fb3800fd8c0 msgr2=0x7fb3800ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:06.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.974+0000 7fb3883e0640 1 -- 172.21.15.33:0/1294801943 shutdown_connections 2024-09-16T00:19:06.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.975+0000 7fb3883e0640 1 -- 172.21.15.33:0/1294801943 wait complete. 2024-09-16T00:19:06.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb3883e0640 1 Processor -- start 2024-09-16T00:19:06.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb3883e0640 1 -- start start 2024-09-16T00:19:06.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb3883e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb380101de0 0x7fb3800744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:06.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb3883e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb380102790 0x7fb380074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:06.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb3883e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb380106f10 0x7fb380074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:06.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb3883e0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb380111fc0 con 0x7fb380102790 2024-09-16T00:19:06.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb3883e0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb380111e40 con 0x7fb380106f10 2024-09-16T00:19:06.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb3883e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb380112140 con 0x7fb380101de0 2024-09-16T00:19:06.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb38597b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb380102790 0x7fb380074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:06.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb38597b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb380102790 0x7fb380074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50346/0 (socket says 172.21.15.33:50346) 2024-09-16T00:19:06.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb38597b640 1 -- 172.21.15.33:0/3834628665 learned_addr learned my addr 172.21.15.33:0/3834628665 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:06.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb38697d640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb380106f10 0x7fb380074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.976+0000 7fb38617c640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb380101de0 0x7fb3800744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.977+0000 7fb38597b640 1 -- 172.21.15.33:0/3834628665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb380101de0 msgr2=0x7fb3800744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.977+0000 7fb38597b640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb380101de0 0x7fb3800744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.977+0000 7fb38597b640 1 -- 172.21.15.33:0/3834628665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb380106f10 msgr2=0x7fb380074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.977+0000 7fb38597b640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb380106f10 0x7fb380074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.977+0000 7fb38597b640 1 -- 172.21.15.33:0/3834628665 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3801a6960 con 0x7fb380102790 2024-09-16T00:19:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.977+0000 7fb38597b640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb380102790 0x7fb380074a30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb37000d950 tx=0x7fb37000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.977+0000 7fb3777fe640 1 -- 172.21.15.33:0/3834628665 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb370017030 con 0x7fb380102790 2024-09-16T00:19:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.977+0000 7fb3777fe640 1 -- 172.21.15.33:0/3834628665 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb370004020 con 0x7fb380102790 2024-09-16T00:19:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.977+0000 7fb3883e0640 1 -- 172.21.15.33:0/3834628665 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3801a6c50 con 0x7fb380102790 2024-09-16T00:19:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.977+0000 7fb3777fe640 1 -- 172.21.15.33:0/3834628665 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3700047a0 con 0x7fb380102790 2024-09-16T00:19:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.977+0000 7fb3883e0640 1 -- 172.21.15.33:0/3834628665 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3801a7190 con 0x7fb380102790 2024-09-16T00:19:06.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.979+0000 7fb3777fe640 1 -- 172.21.15.33:0/3834628665 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb3700041d0 con 0x7fb380102790 2024-09-16T00:19:06.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.980+0000 7fb38a7b9740 1 Processor -- start 2024-09-16T00:19:06.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.980+0000 7fb38a7b9740 1 -- start start 2024-09-16T00:19:06.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.980+0000 7fb38a7b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e9137090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:06.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.980+0000 7fb3777fe640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb36c0a3240 0x7fb36c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:06.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.980+0000 7fb38a7b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9137a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:06.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.980+0000 7fb38a7b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 0x5591e9144760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:06.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.980+0000 7fb38a7b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5591e909d2f0 con 0x5591e9138b40 2024-09-16T00:19:06.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.980+0000 7fb38a7b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5591e9044cf0 con 0x5591e9142370 2024-09-16T00:19:06.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.980+0000 7fb38a7b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5591e90992b0 con 0x5591e91375d0 2024-09-16T00:19:06.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.980+0000 7fb38697d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e9137090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:06.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.980+0000 7fb38617c640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb36c0a3240 0x7fb36c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:06.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb38697d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e9137090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50362/0 (socket says 172.21.15.33:50362) 2024-09-16T00:19:06.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb38697d640 1 -- 172.21.15.33:0/3420161800 learned_addr learned my addr 172.21.15.33:0/3420161800 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:06.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb3777fe640 1 -- 172.21.15.33:0/3834628665 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb3700cf710 con 0x7fb380102790 2024-09-16T00:19:06.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb38597b640 1 --2- 172.21.15.33:0/3420161800 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9137a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:06.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb38697d640 1 --2- 172.21.15.33:0/3420161800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 0x5591e9144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:06.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb38697d640 1 -- 172.21.15.33:0/3420161800 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 msgr2=0x5591e9137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:06.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb38697d640 1 --2- 172.21.15.33:0/3420161800 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9137a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb38697d640 1 -- 172.21.15.33:0/3420161800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 msgr2=0x5591e9144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:06.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb38697d640 1 --2- 172.21.15.33:0/3420161800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 0x5591e9144760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb38697d640 1 -- 172.21.15.33:0/3420161800 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591e90293a0 con 0x5591e9138b40 2024-09-16T00:19:06.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb38697d640 1 --2- 172.21.15.33:0/3420161800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e9137090 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb37800af40 tx=0x7fb37800a980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698775ced27604d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:06.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb38617c640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb36c0a3240 0x7fb36c0a5700 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb37c000c00 tx=0x7fb37c005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:06.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb374dea640 1 -- 172.21.15.33:0/3420161800 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378019030 con 0x5591e9138b40 2024-09-16T00:19:06.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb374dea640 1 -- 172.21.15.33:0/3420161800 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb378013e30 con 0x5591e9138b40 2024-09-16T00:19:06.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.981+0000 7fb374dea640 1 -- 172.21.15.33:0/3420161800 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378003bd0 con 0x5591e9138b40 2024-09-16T00:19:06.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.982+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3420161800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 msgr2=0x5591e9137090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:06.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.982+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/3420161800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e9137090 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb37800af40 tx=0x7fb37800a980 comp rx=0 tx=0).stop 2024-09-16T00:19:06.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.982+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3420161800 shutdown_connections 2024-09-16T00:19:06.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.982+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/3420161800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 0x5591e9144760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.982+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/3420161800 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9137a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.982+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/3420161800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e9137090 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.982+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3420161800 >> 172.21.15.33:0/3420161800 conn(0x5591e9100310 msgr2=0x5591e9133c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:06.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.982+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3420161800 shutdown_connections 2024-09-16T00:19:06.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.982+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3420161800 wait complete. 2024-09-16T00:19:06.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.982+0000 7fb38a7b9740 1 -- start start 2024-09-16T00:19:06.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38a7b9740 1 Processor -- start 2024-09-16T00:19:06.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38a7b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9141d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38a7b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e913efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38a7b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 0x5591e913f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38a7b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5591e9145070 con 0x5591e9138b40 2024-09-16T00:19:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38a7b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5591e9058060 con 0x5591e9142370 2024-09-16T00:19:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38a7b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5591e91451f0 con 0x5591e91375d0 2024-09-16T00:19:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38a7b9740 10 client.0 _ll_register_callbacks cb 0x5591e8dab8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38597b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e913efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38597b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e913efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50364/0 (socket says 172.21.15.33:50364) 2024-09-16T00:19:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38597b640 1 -- 172.21.15.33:0/3776263325 learned_addr learned my addr 172.21.15.33:0/3776263325 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38697d640 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 0x5591e913f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:06.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38617c640 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:06.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38597b640 1 -- 172.21.15.33:0/3776263325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 msgr2=0x5591e9141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:06.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38597b640 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9141d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38597b640 1 -- 172.21.15.33:0/3776263325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 msgr2=0x5591e913f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:06.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38597b640 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 0x5591e913f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38597b640 1 -- 172.21.15.33:0/3776263325 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591e8e0e3c0 con 0x5591e9138b40 2024-09-16T00:19:06.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38617c640 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:19:06.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.983+0000 7fb38697d640 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 0x5591e913f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:06.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb38597b640 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e913efb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb3700ccd30 tx=0x7fb37009b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bc755a7088440a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:06.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb369ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:06.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb369ffb640 1 -- 172.21.15.33:0/3776263325 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37009b850 con 0x5591e9138b40 2024-09-16T00:19:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb369ffb640 1 -- 172.21.15.33:0/3776263325 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb3700cf300 con 0x5591e9138b40 2024-09-16T00:19:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3776263325 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb378013e30 con 0x5591e9138b40 2024-09-16T00:19:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb369ffb640 1 -- 172.21.15.33:0/3776263325 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb3700cf500 con 0x5591e9138b40 2024-09-16T00:19:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb38a7b9740 10 client.15681 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3776263325 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5591e9150aa0 con 0x5591e9138b40 2024-09-16T00:19:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb369ffb640 1 -- 172.21.15.33:0/3776263325 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb378013e30 con 0x5591e9138b40 2024-09-16T00:19:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb369ffb640 1 -- 172.21.15.33:0/3776263325 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb370084ba0 con 0x5591e9138b40 2024-09-16T00:19:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb38a7b9740 10 client.15681 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb38a7b9740 10 client.15681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb38a7b9740 20 client.15681 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5591e9151f50 0x5591e9154390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb38a7b9740 4 client.15681 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.984+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3776263325 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591e90aa860 con 0x5591e9151f50 2024-09-16T00:19:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.985+0000 7fb38617c640 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5591e9151f50 0x5591e9154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.985+0000 7fb369ffb640 10 client.15681 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.985+0000 7fb38617c640 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5591e9151f50 0x5591e9154390 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8df7650ea2704f3 server_cookie=baa9251802e2cd98 in_seq=0 out_seq=0 2024-09-16T00:19:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb369ffb640 1 -- 172.21.15.33:0/3776263325 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5591e90aa860 con 0x5591e9151f50 2024-09-16T00:19:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb369ffb640 10 client.15681 handle_command_reply: tid=0 2024-09-16T00:19:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb38a7b9740 1 client.15681 shutdown 2024-09-16T00:19:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3776263325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 msgr2=0x5591e913efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e913efb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb3700ccd30 tx=0x7fb37009b4c0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3776263325 shutdown_connections 2024-09-16T00:19:06.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5591e9151f50 0x5591e9154390 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 0x5591e913f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e913efb0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/3776263325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:06.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb369ffb640 0 client.15681 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:06.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3776263325 >> 172.21.15.33:0/3776263325 conn(0x5591e9100310 msgr2=0x5591e9139960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:06.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3776263325 shutdown_connections 2024-09-16T00:19:06.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.986+0000 7fb38a7b9740 1 -- 172.21.15.33:0/3776263325 wait complete. 2024-09-16T00:19:06.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:06.987+0000 7fb38a7b9740 20 client.15681 trim_cache size 0 max 16384 2024-09-16T00:19:07.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38a7b9740 1 Processor -- start 2024-09-16T00:19:07.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38a7b9740 1 -- start start 2024-09-16T00:19:07.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38a7b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e914e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:07.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38a7b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9138b40 0x5591e913a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:07.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38a7b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9142370 0x5591e913a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:07.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38a7b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb3801b3bd0 con 0x5591e9142370 2024-09-16T00:19:07.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38a7b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb3801b14a0 con 0x5591e9138b40 2024-09-16T00:19:07.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38a7b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb3801af2f0 con 0x5591e91375d0 2024-09-16T00:19:07.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38697d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9142370 0x5591e913a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:07.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38697d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9142370 0x5591e913a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50368/0 (socket says 172.21.15.33:50368) 2024-09-16T00:19:07.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38697d640 1 -- 172.21.15.33:0/719557132 learned_addr learned my addr 172.21.15.33:0/719557132 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:07.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38597b640 1 --2- 172.21.15.33:0/719557132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9138b40 0x5591e913a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:07.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.001+0000 7fb38617c640 1 --2- 172.21.15.33:0/719557132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e914e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:07.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.002+0000 7fb38697d640 1 -- 172.21.15.33:0/719557132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 msgr2=0x5591e914e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:07.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.002+0000 7fb38697d640 1 --2- 172.21.15.33:0/719557132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e914e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.002+0000 7fb38697d640 1 -- 172.21.15.33:0/719557132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9138b40 msgr2=0x5591e913a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:07.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.002+0000 7fb38697d640 1 --2- 172.21.15.33:0/719557132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9138b40 0x5591e913a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.002+0000 7fb38697d640 1 -- 172.21.15.33:0/719557132 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591e9132310 con 0x5591e9142370 2024-09-16T00:19:07.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.002+0000 7fb38697d640 1 --2- 172.21.15.33:0/719557132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9142370 0x5591e913a660 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb37800af40 tx=0x7fb37800aa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4659493c064b34c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:07.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.002+0000 7fb36bfff640 1 -- 172.21.15.33:0/719557132 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378019040 con 0x5591e9142370 2024-09-16T00:19:07.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.002+0000 7fb36bfff640 1 -- 172.21.15.33:0/719557132 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb378013040 con 0x5591e9142370 2024-09-16T00:19:07.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.002+0000 7fb36bfff640 1 -- 172.21.15.33:0/719557132 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378003bd0 con 0x5591e9142370 2024-09-16T00:19:07.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.003+0000 7fb38a7b9740 1 -- 172.21.15.33:0/719557132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9142370 msgr2=0x5591e913a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:07.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.003+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/719557132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9142370 0x5591e913a660 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb37800af40 tx=0x7fb37800aa70 comp rx=0 tx=0).stop 2024-09-16T00:19:07.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.003+0000 7fb38a7b9740 1 -- 172.21.15.33:0/719557132 shutdown_connections 2024-09-16T00:19:07.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.003+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/719557132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9142370 0x5591e913a660 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.003+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/719557132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9138b40 0x5591e913a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.003+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/719557132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e914e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.003+0000 7fb38a7b9740 1 -- 172.21.15.33:0/719557132 >> 172.21.15.33:0/719557132 conn(0x5591e9100310 msgr2=0x5591e913e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:07.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.003+0000 7fb38a7b9740 1 -- 172.21.15.33:0/719557132 shutdown_connections 2024-09-16T00:19:07.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.003+0000 7fb38a7b9740 1 -- 172.21.15.33:0/719557132 wait complete. 2024-09-16T00:19:07.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.003+0000 7fb38a7b9740 1 -- start start 2024-09-16T00:19:07.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38a7b9740 1 Processor -- start 2024-09-16T00:19:07.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38a7b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9146170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:07.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38a7b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e91466b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:07.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38a7b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 0x5591e9146bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:07.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38a7b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb3801af960 con 0x5591e9138b40 2024-09-16T00:19:07.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38a7b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb3801b0dd0 con 0x5591e9142370 2024-09-16T00:19:07.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38a7b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb3801b0030 con 0x5591e91375d0 2024-09-16T00:19:07.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38a7b9740 10 client.0 _ll_register_callbacks cb 0x7fb3700046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:07.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38597b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e91466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:07.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38597b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e91466b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50380/0 (socket says 172.21.15.33:50380) 2024-09-16T00:19:07.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38597b640 1 -- 172.21.15.33:0/538879904 learned_addr learned my addr 172.21.15.33:0/538879904 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:07.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38617c640 1 --2- 172.21.15.33:0/538879904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9146170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38597b640 1 -- 172.21.15.33:0/538879904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 msgr2=0x5591e9146170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38597b640 1 --2- 172.21.15.33:0/538879904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9146170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38597b640 1 -- 172.21.15.33:0/538879904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 msgr2=0x5591e9146bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38597b640 1 --2- 172.21.15.33:0/538879904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 0x5591e9146bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.004+0000 7fb38597b640 1 -- 172.21.15.33:0/538879904 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591e9134960 con 0x5591e9138b40 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.005+0000 7fb38597b640 1 --2- 172.21.15.33:0/538879904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e91466b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb370009610 tx=0x7fb37009b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e45c83ed5a4425e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.005+0000 7fb369ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.005+0000 7fb369ffb640 1 -- 172.21.15.33:0/538879904 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3700cf510 con 0x5591e9138b40 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.005+0000 7fb369ffb640 1 -- 172.21.15.33:0/538879904 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb37009aca0 con 0x5591e9138b40 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.005+0000 7fb38a7b9740 1 -- 172.21.15.33:0/538879904 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb378013040 con 0x5591e9138b40 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.005+0000 7fb369ffb640 1 -- 172.21.15.33:0/538879904 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb370084da0 con 0x5591e9138b40 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.005+0000 7fb38a7b9740 10 client.15687 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.005+0000 7fb38a7b9740 1 -- 172.21.15.33:0/538879904 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5591e91bcd10 con 0x5591e9138b40 2024-09-16T00:19:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.005+0000 7fb369ffb640 1 -- 172.21.15.33:0/538879904 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb378013040 con 0x5591e9138b40 2024-09-16T00:19:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.006+0000 7fb369ffb640 1 -- 172.21.15.33:0/538879904 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb3700975a0 con 0x5591e9138b40 2024-09-16T00:19:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.006+0000 7fb38a7b9740 10 client.15687 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.006+0000 7fb38a7b9740 10 client.15687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.006+0000 7fb38a7b9740 20 client.15687 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.006+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/538879904 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5591e91bd3d0 0x5591e91bf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.006+0000 7fb38a7b9740 4 client.15687 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:19:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.006+0000 7fb38a7b9740 1 -- 172.21.15.33:0/538879904 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5591e90a4d70 con 0x5591e91bd3d0 2024-09-16T00:19:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.006+0000 7fb38617c640 1 --2- 172.21.15.33:0/538879904 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5591e91bd3d0 0x5591e91bf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.006+0000 7fb38617c640 1 --2- 172.21.15.33:0/538879904 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5591e91bd3d0 0x5591e91bf7f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d94b8863f6a40849 server_cookie=f25dfa6da65436e5 in_seq=0 out_seq=0 2024-09-16T00:19:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.006+0000 7fb369ffb640 10 client.15687 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb369ffb640 1 -- 172.21.15.33:0/538879904 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5591e90a4d70 con 0x5591e91bd3d0 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb369ffb640 10 client.15687 handle_command_reply: tid=0 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb38a7b9740 1 client.15687 shutdown 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb38a7b9740 1 -- 172.21.15.33:0/538879904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 msgr2=0x5591e91466b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/538879904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e91466b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb370009610 tx=0x7fb37009b620 comp rx=0 tx=0).stop 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb38a7b9740 1 -- 172.21.15.33:0/538879904 shutdown_connections 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/538879904 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5591e91bd3d0 0x5591e91bf7f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/538879904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5591e9142370 0x5591e9146bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/538879904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591e9138b40 0x5591e91466b0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb38a7b9740 1 --2- 172.21.15.33:0/538879904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591e91375d0 0x5591e9146170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb38a7b9740 1 -- 172.21.15.33:0/538879904 >> 172.21.15.33:0/538879904 conn(0x5591e9100310 msgr2=0x5591e9132500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb369ffb640 0 client.15687 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb38a7b9740 1 -- 172.21.15.33:0/538879904 shutdown_connections 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb38a7b9740 1 -- 172.21.15.33:0/538879904 wait complete. 2024-09-16T00:19:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.007+0000 7fb38a7b9740 20 client.15687 trim_cache size 0 max 16384 2024-09-16T00:19:07.015 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:19:07.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.008+0000 7fb3883e0640 1 -- 172.21.15.33:0/3834628665 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb36c0a3240 msgr2=0x7fb36c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.008+0000 7fb3883e0640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb36c0a3240 0x7fb36c0a5700 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb37c000c00 tx=0x7fb37c005b50 comp rx=0 tx=0).stop 2024-09-16T00:19:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.008+0000 7fb3883e0640 1 -- 172.21.15.33:0/3834628665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb380102790 msgr2=0x7fb380074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.008+0000 7fb3883e0640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb380102790 0x7fb380074a30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb37000d950 tx=0x7fb37000de10 comp rx=0 tx=0).stop 2024-09-16T00:19:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.008+0000 7fb3883e0640 1 -- 172.21.15.33:0/3834628665 shutdown_connections 2024-09-16T00:19:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.008+0000 7fb3883e0640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb36c0a3240 0x7fb36c0a5700 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.008+0000 7fb3883e0640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb380106f10 0x7fb380074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.008+0000 7fb3883e0640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb380102790 0x7fb380074a30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.008+0000 7fb3883e0640 1 --2- 172.21.15.33:0/3834628665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb380101de0 0x7fb3800744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.008+0000 7fb3883e0640 1 -- 172.21.15.33:0/3834628665 >> 172.21.15.33:0/3834628665 conn(0x7fb3800fd8c0 msgr2=0x7fb3800fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.008+0000 7fb3883e0640 1 -- 172.21.15.33:0/3834628665 shutdown_connections 2024-09-16T00:19:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:07.008+0000 7fb3883e0640 1 -- 172.21.15.33:0/3834628665 wait complete. 2024-09-16T00:19:07.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:07 smithi083 ceph-mon[35242]: pgmap v409: 129 pgs: 129 active+clean; 9.3 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 884 KiB/s wr, 108 op/s 2024-09-16T00:19:07.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:07 smithi060 ceph-mon[35185]: pgmap v409: 129 pgs: 129 active+clean; 9.3 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 884 KiB/s wr, 108 op/s 2024-09-16T00:19:07.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:07 smithi033 ceph-mon[31243]: pgmap v409: 129 pgs: 129 active+clean; 9.3 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 884 KiB/s wr, 108 op/s 2024-09-16T00:19:07.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 21290 8.93 MB/sec warmup 15 sec latency 3837.097 ms 2024-09-16T00:19:08.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 22198 8.52 MB/sec warmup 16 sec latency 547.742 ms 2024-09-16T00:19:08.828 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:19:08.829 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:19:09.088 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:09.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:09 smithi083 ceph-mon[35242]: pgmap v410: 129 pgs: 129 active+clean; 9.3 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 793 KiB/s wr, 103 op/s 2024-09-16T00:19:09.471 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:09 smithi033 ceph-mon[31243]: pgmap v410: 129 pgs: 129 active+clean; 9.3 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 793 KiB/s wr, 103 op/s 2024-09-16T00:19:09.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:09 smithi060 ceph-mon[35185]: pgmap v410: 129 pgs: 129 active+clean; 9.3 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 793 KiB/s wr, 103 op/s 2024-09-16T00:19:09.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.555+0000 7f6f06b5b640 1 -- 172.21.15.33:0/536314232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f00101ba0 msgr2=0x7f6f0010d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:09.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.555+0000 7f6f06b5b640 1 --2- 172.21.15.33:0/536314232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f00101ba0 0x7f6f0010d970 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009ca0 tx=0x7f6ef802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.555+0000 7f6f06b5b640 1 -- 172.21.15.33:0/536314232 shutdown_connections 2024-09-16T00:19:09.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.555+0000 7f6f06b5b640 1 --2- 172.21.15.33:0/536314232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f00101ba0 0x7f6f0010d970 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.555+0000 7f6f06b5b640 1 --2- 172.21.15.33:0/536314232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f00101200 0x7f6f00101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.555+0000 7f6f06b5b640 1 --2- 172.21.15.33:0/536314232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f00105ad0 0x7f6f00105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.555+0000 7f6f06b5b640 1 -- 172.21.15.33:0/536314232 >> 172.21.15.33:0/536314232 conn(0x7f6f00077210 msgr2=0x7f6f000fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:09.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.555+0000 7f6f06b5b640 1 -- 172.21.15.33:0/536314232 shutdown_connections 2024-09-16T00:19:09.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.555+0000 7f6f06b5b640 1 -- 172.21.15.33:0/536314232 wait complete. 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6f06b5b640 1 Processor -- start 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6f06b5b640 1 -- start start 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6f06b5b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f00101200 0x7f6f00199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6f06b5b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f00101ba0 0x7f6f00199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6f06b5b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f00105ad0 0x7f6f001a41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6f06b5b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f00110ef0 con 0x7f6f00105ad0 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6f06b5b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6f00110d70 con 0x7f6f00101ba0 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6f06b5b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f00111070 con 0x7f6f00101200 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6f050f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f00105ad0 0x7f6f001a41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6f050f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f00105ad0 0x7f6f001a41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50408/0 (socket says 172.21.15.33:50408) 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6f050f8640 1 -- 172.21.15.33:0/574868282 learned_addr learned my addr 172.21.15.33:0/574868282 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6f048f7640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f00101200 0x7f6f00199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.556+0000 7f6efffff640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f00101ba0 0x7f6f00199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.557+0000 7f6f050f8640 1 -- 172.21.15.33:0/574868282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f00101200 msgr2=0x7f6f00199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.557+0000 7f6f050f8640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f00101200 0x7f6f00199140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.557+0000 7f6f050f8640 1 -- 172.21.15.33:0/574868282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f00101ba0 msgr2=0x7f6f00199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.557+0000 7f6f050f8640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f00101ba0 0x7f6f00199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.557+0000 7f6f050f8640 1 -- 172.21.15.33:0/574868282 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f001a4700 con 0x7f6f00105ad0 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.557+0000 7f6f050f8640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f00105ad0 0x7f6f001a41c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8003f40 tx=0x7f6ef8003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.557+0000 7f6efdffb640 1 -- 172.21.15.33:0/574868282 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef80317d0 con 0x7f6f00105ad0 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.557+0000 7f6efdffb640 1 -- 172.21.15.33:0/574868282 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef8031970 con 0x7f6f00105ad0 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.557+0000 7f6f06b5b640 1 -- 172.21.15.33:0/574868282 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f001a48d0 con 0x7f6f00105ad0 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.557+0000 7f6efdffb640 1 -- 172.21.15.33:0/574868282 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8030c90 con 0x7f6f00105ad0 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.557+0000 7f6f06b5b640 1 -- 172.21.15.33:0/574868282 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f001a4d80 con 0x7f6f00105ad0 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.560+0000 7f6efdffb640 1 -- 172.21.15.33:0/574868282 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6ef8004040 con 0x7f6f00105ad0 2024-09-16T00:19:09.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.560+0000 7f6f08f34740 1 Processor -- start 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.560+0000 7f6f08f34740 1 -- start start 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.560+0000 7f6f08f34740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e3b60 0x5562a69e3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.560+0000 7f6f08f34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69e4480 0x5562a69ee800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.560+0000 7f6f08f34740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69eed40 0x5562a69f1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.560+0000 7f6f08f34740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5562a6949280 con 0x5562a69e3b60 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.560+0000 7f6f08f34740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5562a68f0b10 con 0x5562a69eed40 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.560+0000 7f6f08f34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5562a6945240 con 0x5562a69e4480 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.560+0000 7f6efffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e3b60 0x5562a69e3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.560+0000 7f6efffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e3b60 0x5562a69e3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50422/0 (socket says 172.21.15.33:50422) 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6efffff640 1 -- 172.21.15.33:0/3880950112 learned_addr learned my addr 172.21.15.33:0/3880950112 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6efdffb640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6eec0a3240 0x7f6eec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6f048f7640 1 --2- 172.21.15.33:0/3880950112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69e4480 0x5562a69ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6efffff640 1 --2- 172.21.15.33:0/3880950112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69eed40 0x5562a69f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6f050f8640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6eec0a3240 0x7f6eec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6efffff640 1 -- 172.21.15.33:0/3880950112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69e4480 msgr2=0x5562a69ee800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6efffff640 1 --2- 172.21.15.33:0/3880950112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69e4480 0x5562a69ee800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6efffff640 1 -- 172.21.15.33:0/3880950112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69eed40 msgr2=0x5562a69f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6efffff640 1 --2- 172.21.15.33:0/3880950112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69eed40 0x5562a69f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6efdffb640 1 -- 172.21.15.33:0/574868282 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6ef80f1490 con 0x7f6f00105ad0 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6efffff640 1 -- 172.21.15.33:0/3880950112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562a68fd1f0 con 0x5562a69e3b60 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6efffff640 1 --2- 172.21.15.33:0/3880950112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e3b60 0x5562a69e3f40 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0013550 tx=0x7f6ef000a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0ad18ed05f8c5c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6f050f8640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6eec0a3240 0x7f6eec0a5700 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80eeab0 tx=0x7f6ef80b2ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6ef6ffd640 1 -- 172.21.15.33:0/3880950112 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0019030 con 0x5562a69e3b60 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6ef6ffd640 1 -- 172.21.15.33:0/3880950112 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef0005500 con 0x5562a69e3b60 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.561+0000 7f6ef6ffd640 1 -- 172.21.15.33:0/3880950112 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef000f890 con 0x5562a69e3b60 2024-09-16T00:19:09.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.562+0000 7f6f08f34740 1 -- 172.21.15.33:0/3880950112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e3b60 msgr2=0x5562a69e3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.562+0000 7f6f08f34740 1 --2- 172.21.15.33:0/3880950112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e3b60 0x5562a69e3f40 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0013550 tx=0x7f6ef000a760 comp rx=0 tx=0).stop 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.562+0000 7f6f08f34740 1 -- 172.21.15.33:0/3880950112 shutdown_connections 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.562+0000 7f6f08f34740 1 --2- 172.21.15.33:0/3880950112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69eed40 0x5562a69f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.562+0000 7f6f08f34740 1 --2- 172.21.15.33:0/3880950112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69e4480 0x5562a69ee800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.562+0000 7f6f08f34740 1 --2- 172.21.15.33:0/3880950112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e3b60 0x5562a69e3f40 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.562+0000 7f6f08f34740 1 -- 172.21.15.33:0/3880950112 >> 172.21.15.33:0/3880950112 conn(0x5562a69e00f0 msgr2=0x5562a69e04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.562+0000 7f6f08f34740 1 -- 172.21.15.33:0/3880950112 shutdown_connections 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.562+0000 7f6f08f34740 1 -- 172.21.15.33:0/3880950112 wait complete. 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6f08f34740 1 -- start start 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6f08f34740 1 Processor -- start 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6f08f34740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e3b60 0x5562a69ebc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6f08f34740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 0x5562a69ec150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6f08f34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69eed40 0x5562a69ec690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6f08f34740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5562a69f1910 con 0x5562a69e4480 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6f08f34740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5562a6903f10 con 0x5562a69e3b60 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6f08f34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5562a69f1a90 con 0x5562a69eed40 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6f08f34740 10 client.0 _ll_register_callbacks cb 0x5562a68cca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6f048f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 0x5562a69ec150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6f048f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 0x5562a69ec150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50438/0 (socket says 172.21.15.33:50438) 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6f048f7640 1 -- 172.21.15.33:0/3149350186 learned_addr learned my addr 172.21.15.33:0/3149350186 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6efffff640 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69eed40 0x5562a69ec690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.563+0000 7f6efffff640 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e3b60 0x5562a69ebc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6f048f7640 1 -- 172.21.15.33:0/3149350186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69eed40 msgr2=0x5562a69ec690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6f048f7640 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69eed40 0x5562a69ec690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6f048f7640 1 -- 172.21.15.33:0/3149350186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e3b60 msgr2=0x5562a69ebc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6f048f7640 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e3b60 0x5562a69ebc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6f048f7640 1 -- 172.21.15.33:0/3149350186 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562a66ba6a0 con 0x5562a69e4480 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6f048f7640 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 0x5562a69ec150 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8012780 tx=0x7f6ee8012c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebddce65aebf5dea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6edffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6edffff640 1 -- 172.21.15.33:0/3149350186 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee801f060 con 0x5562a69e4480 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6edffff640 1 -- 172.21.15.33:0/3149350186 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6ee8010dc0 con 0x5562a69e4480 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6f08f34740 1 -- 172.21.15.33:0/3149350186 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ef0005500 con 0x5562a69e4480 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6edffff640 1 -- 172.21.15.33:0/3149350186 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6ee8026050 con 0x5562a69e4480 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6f08f34740 10 client.15699 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6f08f34740 1 -- 172.21.15.33:0/3149350186 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5562a69fd5c0 con 0x5562a69e4480 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.564+0000 7f6edffff640 1 -- 172.21.15.33:0/3149350186 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef0005500 con 0x5562a69e4480 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.565+0000 7f6edffff640 1 -- 172.21.15.33:0/3149350186 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6ee800d550 con 0x5562a69e4480 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.565+0000 7f6f08f34740 10 client.15699 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.565+0000 7f6f08f34740 10 client.15699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:09.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.565+0000 7f6f08f34740 20 client.15699 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.565+0000 7f6f08f34740 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5562a69fea80 0x5562a6a00eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.565+0000 7f6f08f34740 4 client.15699 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.565+0000 7f6f08f34740 1 -- 172.21.15.33:0/3149350186 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562a69567f0 con 0x5562a69fea80 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.565+0000 7f6efffff640 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5562a69fea80 0x5562a6a00eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.565+0000 7f6efffff640 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5562a69fea80 0x5562a6a00eb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35193d2dc39af310 server_cookie=b545d4f2a22a6cba in_seq=0 out_seq=0 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.565+0000 7f6edffff640 10 client.15699 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.566+0000 7f6edffff640 1 -- 172.21.15.33:0/3149350186 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5562a69567f0 con 0x5562a69fea80 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.566+0000 7f6edffff640 10 client.15699 handle_command_reply: tid=0 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.566+0000 7f6f08f34740 1 client.15699 shutdown 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.566+0000 7f6f08f34740 1 -- 172.21.15.33:0/3149350186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 msgr2=0x5562a69ec150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.566+0000 7f6f08f34740 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 0x5562a69ec150 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8012780 tx=0x7f6ee8012c40 comp rx=0 tx=0).stop 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.566+0000 7f6f08f34740 1 -- 172.21.15.33:0/3149350186 shutdown_connections 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.566+0000 7f6f08f34740 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5562a69fea80 0x5562a6a00eb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.566+0000 7f6f08f34740 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69eed40 0x5562a69ec690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.566+0000 7f6f08f34740 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 0x5562a69ec150 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.566+0000 7f6f08f34740 1 --2- 172.21.15.33:0/3149350186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e3b60 0x5562a69ebc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.566+0000 7f6edffff640 0 client.15699 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:09.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.566+0000 7f6f08f34740 1 -- 172.21.15.33:0/3149350186 >> 172.21.15.33:0/3149350186 conn(0x5562a69e00f0 msgr2=0x5562a69e5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:09.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.567+0000 7f6f08f34740 1 -- 172.21.15.33:0/3149350186 shutdown_connections 2024-09-16T00:19:09.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.567+0000 7f6f08f34740 1 -- 172.21.15.33:0/3149350186 wait complete. 2024-09-16T00:19:09.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.567+0000 7f6f08f34740 20 client.15699 trim_cache size 0 max 16384 2024-09-16T00:19:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f08f34740 1 Processor -- start 2024-09-16T00:19:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f08f34740 1 -- start start 2024-09-16T00:19:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f08f34740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e3b60 0x5562a69e65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f08f34740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 0x5562a69eda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f08f34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69eed40 0x5562a6a067c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f08f34740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f001b1840 con 0x5562a69e4480 2024-09-16T00:19:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f08f34740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6f001af110 con 0x5562a69e3b60 2024-09-16T00:19:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f08f34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f001acf60 con 0x5562a69eed40 2024-09-16T00:19:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f048f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 0x5562a69eda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f048f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 0x5562a69eda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50446/0 (socket says 172.21.15.33:50446) 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f048f7640 1 -- 172.21.15.33:0/899888669 learned_addr learned my addr 172.21.15.33:0/899888669 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6efffff640 1 --2- 172.21.15.33:0/899888669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e3b60 0x5562a69e65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6efffff640 1 --2- 172.21.15.33:0/899888669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69eed40 0x5562a6a067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f048f7640 1 -- 172.21.15.33:0/899888669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69eed40 msgr2=0x5562a6a067c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f048f7640 1 --2- 172.21.15.33:0/899888669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69eed40 0x5562a6a067c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f048f7640 1 -- 172.21.15.33:0/899888669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e3b60 msgr2=0x5562a69e65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f048f7640 1 --2- 172.21.15.33:0/899888669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e3b60 0x5562a69e65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.585+0000 7f6f048f7640 1 -- 172.21.15.33:0/899888669 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562a69fd3d0 con 0x5562a69e4480 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.586+0000 7f6f048f7640 1 --2- 172.21.15.33:0/899888669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 0x5562a69eda70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6ee800d9a0 tx=0x7f6ee8001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2537884aaccc2e82 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.586+0000 7f6ef67fc640 1 -- 172.21.15.33:0/899888669 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee802e030 con 0x5562a69e4480 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.586+0000 7f6ef67fc640 1 -- 172.21.15.33:0/899888669 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ee800db90 con 0x5562a69e4480 2024-09-16T00:19:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.586+0000 7f6ef67fc640 1 -- 172.21.15.33:0/899888669 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8005040 con 0x5562a69e4480 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.586+0000 7f6f08f34740 1 -- 172.21.15.33:0/899888669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 msgr2=0x5562a69eda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.586+0000 7f6f08f34740 1 --2- 172.21.15.33:0/899888669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 0x5562a69eda70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6ee800d9a0 tx=0x7f6ee8001e50 comp rx=0 tx=0).stop 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 -- 172.21.15.33:0/899888669 shutdown_connections 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 --2- 172.21.15.33:0/899888669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69eed40 0x5562a6a067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 --2- 172.21.15.33:0/899888669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69e4480 0x5562a69eda70 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 --2- 172.21.15.33:0/899888669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e3b60 0x5562a69e65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 -- 172.21.15.33:0/899888669 >> 172.21.15.33:0/899888669 conn(0x5562a69e00f0 msgr2=0x5562a69ed360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 -- 172.21.15.33:0/899888669 shutdown_connections 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 -- 172.21.15.33:0/899888669 wait complete. 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 -- start start 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 Processor -- start 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69e3b60 0x5562a69f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e4480 0x5562a69f3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69eed40 0x5562a6a68790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f001ad5d0 con 0x5562a69eed40 2024-09-16T00:19:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6f001aea40 con 0x5562a69e4480 2024-09-16T00:19:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f001adca0 con 0x5562a69e3b60 2024-09-16T00:19:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.587+0000 7f6f08f34740 10 client.0 _ll_register_callbacks cb 0x7f6ee800a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6efffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69eed40 0x5562a6a68790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6efffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69eed40 0x5562a6a68790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50454/0 (socket says 172.21.15.33:50454) 2024-09-16T00:19:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6efffff640 1 -- 172.21.15.33:0/264895092 learned_addr learned my addr 172.21.15.33:0/264895092 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6f048f7640 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e4480 0x5562a69f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6efffff640 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69e3b60 0x5562a69f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6efffff640 1 -- 172.21.15.33:0/264895092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69e3b60 msgr2=0x5562a69f2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6efffff640 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69e3b60 0x5562a69f2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6efffff640 1 -- 172.21.15.33:0/264895092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e4480 msgr2=0x5562a69f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6efffff640 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e4480 0x5562a69f3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6efffff640 1 -- 172.21.15.33:0/264895092 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562a6a68cd0 con 0x5562a69eed40 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6efffff640 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69eed40 0x5562a6a68790 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0015010 tx=0x7f6ef0020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551f0e457f490e80 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6edffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6edffff640 1 -- 172.21.15.33:0/264895092 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0032030 con 0x5562a69eed40 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6edffff640 1 -- 172.21.15.33:0/264895092 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6ef002de80 con 0x5562a69eed40 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6f08f34740 1 -- 172.21.15.33:0/264895092 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ee800db90 con 0x5562a69eed40 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6edffff640 1 -- 172.21.15.33:0/264895092 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6ef0019040 con 0x5562a69eed40 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6f08f34740 10 client.15705 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.588+0000 7f6f08f34740 1 -- 172.21.15.33:0/264895092 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5562a6a69e60 con 0x5562a69eed40 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.589+0000 7f6edffff640 1 -- 172.21.15.33:0/264895092 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ee800db90 con 0x5562a69eed40 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.589+0000 7f6edffff640 1 -- 172.21.15.33:0/264895092 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6ef000f990 con 0x5562a69eed40 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.589+0000 7f6f08f34740 10 client.15705 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.589+0000 7f6f08f34740 10 client.15705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.589+0000 7f6f08f34740 20 client.15705 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.589+0000 7f6f08f34740 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5562a6a6a330 0x5562a6a6c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.589+0000 7f6f08f34740 4 client.15705 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.589+0000 7f6f08f34740 1 -- 172.21.15.33:0/264895092 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5562a6950d00 con 0x5562a6a6a330 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.589+0000 7f6f048f7640 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5562a6a6a330 0x5562a6a6c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.590+0000 7f6edffff640 10 client.15705 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.590+0000 7f6f048f7640 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5562a6a6a330 0x5562a6a6c720 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=595dea00e3e5898e server_cookie=16745c6d88ef18d6 in_seq=0 out_seq=0 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6edffff640 1 -- 172.21.15.33:0/264895092 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5562a6950d00 con 0x5562a6a6a330 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6edffff640 10 client.15705 handle_command_reply: tid=0 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6f08f34740 1 client.15705 shutdown 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6f08f34740 1 -- 172.21.15.33:0/264895092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69eed40 msgr2=0x5562a6a68790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6f08f34740 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69eed40 0x5562a6a68790 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0015010 tx=0x7f6ef0020040 comp rx=0 tx=0).stop 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6f08f34740 1 -- 172.21.15.33:0/264895092 shutdown_connections 2024-09-16T00:19:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6f08f34740 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5562a6a6a330 0x5562a6a6c720 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6f08f34740 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562a69eed40 0x5562a6a68790 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6f08f34740 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5562a69e4480 0x5562a69f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6f08f34740 1 --2- 172.21.15.33:0/264895092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5562a69e3b60 0x5562a69f2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6f08f34740 1 -- 172.21.15.33:0/264895092 >> 172.21.15.33:0/264895092 conn(0x5562a69e00f0 msgr2=0x5562a69df270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6edffff640 0 client.15705 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6f08f34740 1 -- 172.21.15.33:0/264895092 shutdown_connections 2024-09-16T00:19:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6f08f34740 1 -- 172.21.15.33:0/264895092 wait complete. 2024-09-16T00:19:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.591+0000 7f6f08f34740 20 client.15705 trim_cache size 0 max 16384 2024-09-16T00:19:09.599 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:19:09.600 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:19:09.600 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "c0cbd37c-9c16-4d06-9ae4-7ef751697353", 2024-09-16T00:19:09.600 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:19:09.600 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:19:09.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.592+0000 7f6f06b5b640 1 -- 172.21.15.33:0/574868282 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6eec0a3240 msgr2=0x7f6eec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:09.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.592+0000 7f6f06b5b640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6eec0a3240 0x7f6eec0a5700 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80eeab0 tx=0x7f6ef80b2ab0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.592+0000 7f6f06b5b640 1 -- 172.21.15.33:0/574868282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f00105ad0 msgr2=0x7f6f001a41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:09.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.592+0000 7f6f06b5b640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f00105ad0 0x7f6f001a41c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8003f40 tx=0x7f6ef8003f70 comp rx=0 tx=0).stop 2024-09-16T00:19:09.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.592+0000 7f6f06b5b640 1 -- 172.21.15.33:0/574868282 shutdown_connections 2024-09-16T00:19:09.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.592+0000 7f6f06b5b640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6eec0a3240 0x7f6eec0a5700 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.592+0000 7f6f06b5b640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f00105ad0 0x7f6f001a41c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.592+0000 7f6f06b5b640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f00101ba0 0x7f6f00199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.592+0000 7f6f06b5b640 1 --2- 172.21.15.33:0/574868282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f00101200 0x7f6f00199140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:09.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.592+0000 7f6f06b5b640 1 -- 172.21.15.33:0/574868282 >> 172.21.15.33:0/574868282 conn(0x7f6f00077210 msgr2=0x7f6f00103910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:09.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.592+0000 7f6f06b5b640 1 -- 172.21.15.33:0/574868282 shutdown_connections 2024-09-16T00:19:09.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:09.592+0000 7f6f06b5b640 1 -- 172.21.15.33:0/574868282 wait complete. 2024-09-16T00:19:09.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 22198 8.02 MB/sec warmup 17 sec latency 1547.828 ms 2024-09-16T00:19:10.333 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:19:10.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:10 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:19:10.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:10 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:10.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:10 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:19:10.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:10 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:10.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:10 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:19:10.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:10 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:10.592 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:10.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 22198 7.57 MB/sec warmup 18 sec latency 2547.911 ms 2024-09-16T00:19:11.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.097+0000 7faaf7620640 1 -- 172.21.15.33:0/3086311866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faaf0101de0 msgr2=0x7faaf01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:11.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.097+0000 7faaf7620640 1 --2- 172.21.15.33:0/3086311866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faaf0101de0 0x7faaf01021c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7faae8009900 tx=0x7faae802a3f0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.097+0000 7faaf7620640 1 -- 172.21.15.33:0/3086311866 shutdown_connections 2024-09-16T00:19:11.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.097+0000 7faaf7620640 1 --2- 172.21.15.33:0/3086311866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf0106f10 0x7faaf010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.097+0000 7faaf7620640 1 --2- 172.21.15.33:0/3086311866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaf0102790 0x7faaf01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.097+0000 7faaf7620640 1 --2- 172.21.15.33:0/3086311866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faaf0101de0 0x7faaf01021c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.097+0000 7faaf7620640 1 -- 172.21.15.33:0/3086311866 >> 172.21.15.33:0/3086311866 conn(0x7faaf00fd8c0 msgr2=0x7faaf00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:11.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.097+0000 7faaf7620640 1 -- 172.21.15.33:0/3086311866 shutdown_connections 2024-09-16T00:19:11.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.097+0000 7faaf7620640 1 -- 172.21.15.33:0/3086311866 wait complete. 2024-09-16T00:19:11.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf7620640 1 Processor -- start 2024-09-16T00:19:11.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf7620640 1 -- start start 2024-09-16T00:19:11.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf7620640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faaf0101de0 0x7faaf0198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf7620640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaf0102790 0x7faaf01988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf7620640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf0106f10 0x7faaf019cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf7620640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faaf0111f30 con 0x7faaf0101de0 2024-09-16T00:19:11.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf7620640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faaf0111db0 con 0x7faaf0102790 2024-09-16T00:19:11.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf7620640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01120b0 con 0x7faaf0106f10 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf53bc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faaf0101de0 0x7faaf0198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf53bc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faaf0101de0 0x7faaf0198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45834/0 (socket says 172.21.15.33:45834) 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf53bc640 1 -- 172.21.15.33:0/3417990113 learned_addr learned my addr 172.21.15.33:0/3417990113 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf4bbb640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaf0102790 0x7faaf01988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf5bbd640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf0106f10 0x7faaf019cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf53bc640 1 -- 172.21.15.33:0/3417990113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf0106f10 msgr2=0x7faaf019cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf53bc640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf0106f10 0x7faaf019cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf53bc640 1 -- 172.21.15.33:0/3417990113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaf0102790 msgr2=0x7faaf01988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf53bc640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaf0102790 0x7faaf01988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.098+0000 7faaf53bc640 1 -- 172.21.15.33:0/3417990113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf019d340 con 0x7faaf0101de0 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.099+0000 7faaf53bc640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faaf0101de0 0x7faaf0198390 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7faae8006010 tx=0x7faae80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.099+0000 7faaee7fc640 1 -- 172.21.15.33:0/3417990113 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae803f030 con 0x7faaf0101de0 2024-09-16T00:19:11.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.099+0000 7faaee7fc640 1 -- 172.21.15.33:0/3417990113 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae8003ce0 con 0x7faaf0101de0 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.099+0000 7faaf7620640 1 -- 172.21.15.33:0/3417990113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf019d5d0 con 0x7faaf0101de0 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.099+0000 7faaee7fc640 1 -- 172.21.15.33:0/3417990113 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8035070 con 0x7faaf0101de0 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.099+0000 7faaf7620640 1 -- 172.21.15.33:0/3417990113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf01a4f10 con 0x7faaf0101de0 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.101+0000 7faaee7fc640 1 -- 172.21.15.33:0/3417990113 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7faae8042030 con 0x7faaf0101de0 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf99f9740 1 Processor -- start 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf99f9740 1 -- start start 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf99f9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69c0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf99f9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69c0dc0 0x556be69cb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf99f9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69cb680 0x556be69cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf99f9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556be69262e0 con 0x556be69c04a0 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf99f9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556be68cdce0 con 0x556be69cb680 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf99f9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556be69222a0 con 0x556be69c0dc0 2024-09-16T00:19:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaee7fc640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faad80a3240 0x7faad80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf5bbd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69c0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf5bbd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69c0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45848/0 (socket says 172.21.15.33:45848) 2024-09-16T00:19:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf5bbd640 1 -- 172.21.15.33:0/4058569627 learned_addr learned my addr 172.21.15.33:0/4058569627 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf5bbd640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faad80a3240 0x7faad80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf53bc640 1 --2- 172.21.15.33:0/4058569627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69c0dc0 0x556be69cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaee7fc640 1 -- 172.21.15.33:0/3417990113 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7faae80efeb0 con 0x7faaf0101de0 2024-09-16T00:19:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.102+0000 7faaf4bbb640 1 --2- 172.21.15.33:0/4058569627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69cb680 0x556be69cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.103+0000 7faaf5bbd640 1 -- 172.21.15.33:0/4058569627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69c0dc0 msgr2=0x556be69cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.103+0000 7faaf5bbd640 1 --2- 172.21.15.33:0/4058569627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69c0dc0 0x556be69cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.103+0000 7faaf5bbd640 1 -- 172.21.15.33:0/4058569627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69cb680 msgr2=0x556be69cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.103+0000 7faaf5bbd640 1 --2- 172.21.15.33:0/4058569627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69cb680 0x556be69cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.103+0000 7faaf5bbd640 1 -- 172.21.15.33:0/4058569627 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556be68b2440 con 0x556be69c04a0 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.103+0000 7faaf5bbd640 1 --2- 172.21.15.33:0/4058569627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69c0880 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7faae0012c50 tx=0x7faae0010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a77dd0f3cc4db9e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.103+0000 7faaf5bbd640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faad80a3240 0x7faad80a5700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faae0010c40 tx=0x7faae0017000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.103+0000 7faad77fe640 1 -- 172.21.15.33:0/4058569627 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae0012e00 con 0x556be69c04a0 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.103+0000 7faad77fe640 1 -- 172.21.15.33:0/4058569627 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae0012730 con 0x556be69c04a0 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.103+0000 7faad77fe640 1 -- 172.21.15.33:0/4058569627 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae0005330 con 0x556be69c04a0 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.104+0000 7faaf99f9740 1 -- 172.21.15.33:0/4058569627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 msgr2=0x556be69c0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.104+0000 7faaf99f9740 1 --2- 172.21.15.33:0/4058569627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69c0880 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7faae0012c50 tx=0x7faae0010740 comp rx=0 tx=0).stop 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.104+0000 7faaf99f9740 1 -- 172.21.15.33:0/4058569627 shutdown_connections 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.104+0000 7faaf99f9740 1 --2- 172.21.15.33:0/4058569627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69cb680 0x556be69cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.104+0000 7faaf99f9740 1 --2- 172.21.15.33:0/4058569627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69c0dc0 0x556be69cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.104+0000 7faaf99f9740 1 --2- 172.21.15.33:0/4058569627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69c0880 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.104+0000 7faaf99f9740 1 -- 172.21.15.33:0/4058569627 >> 172.21.15.33:0/4058569627 conn(0x556be69bca30 msgr2=0x556be69bce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.104+0000 7faaf99f9740 1 -- 172.21.15.33:0/4058569627 shutdown_connections 2024-09-16T00:19:11.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.104+0000 7faaf99f9740 1 -- 172.21.15.33:0/4058569627 wait complete. 2024-09-16T00:19:11.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.104+0000 7faaf99f9740 1 -- start start 2024-09-16T00:19:11.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.104+0000 7faaf99f9740 1 Processor -- start 2024-09-16T00:19:11.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf99f9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69c04a0 0x556be69cb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf99f9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c0dc0 0x556be69c82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf99f9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69cb680 0x556be69c8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf99f9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556be69ce250 con 0x556be69c0dc0 2024-09-16T00:19:11.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf99f9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556be68e1050 con 0x556be69c04a0 2024-09-16T00:19:11.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf99f9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556be69ce3d0 con 0x556be69cb680 2024-09-16T00:19:11.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf99f9740 10 client.0 _ll_register_callbacks cb 0x556be6921160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf4bbb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c0dc0 0x556be69c82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf4bbb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c0dc0 0x556be69c82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45858/0 (socket says 172.21.15.33:45858) 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf4bbb640 1 -- 172.21.15.33:0/2896208736 learned_addr learned my addr 172.21.15.33:0/2896208736 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf53bc640 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69c04a0 0x556be69cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf5bbd640 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69cb680 0x556be69c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf4bbb640 1 -- 172.21.15.33:0/2896208736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69cb680 msgr2=0x556be69c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf4bbb640 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69cb680 0x556be69c8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf4bbb640 1 -- 172.21.15.33:0/2896208736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69c04a0 msgr2=0x556be69cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf4bbb640 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69c04a0 0x556be69cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf4bbb640 1 -- 172.21.15.33:0/2896208736 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556be66973c0 con 0x556be69c0dc0 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faaf4bbb640 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c0dc0 0x556be69c82b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7faae4012770 tx=0x7faae4012c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390440702834efce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.105+0000 7faad4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:11.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faad4ff9640 1 -- 172.21.15.33:0/2896208736 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae401f060 con 0x556be69c0dc0 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faaf99f9740 1 -- 172.21.15.33:0/2896208736 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faae0012730 con 0x556be69c0dc0 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faad4ff9640 1 -- 172.21.15.33:0/2896208736 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7faae4010d60 con 0x556be69c0dc0 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faad4ff9640 1 -- 172.21.15.33:0/2896208736 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7faae4026050 con 0x556be69c0dc0 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faaf99f9740 10 client.15717 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faaf99f9740 1 -- 172.21.15.33:0/2896208736 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556be69d9ca0 con 0x556be69c0dc0 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faad4ff9640 1 -- 172.21.15.33:0/2896208736 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae0012730 con 0x556be69c0dc0 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faad4ff9640 1 -- 172.21.15.33:0/2896208736 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7faae400d4e0 con 0x556be69c0dc0 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faaf99f9740 10 client.15717 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faaf99f9740 10 client.15717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faaf99f9740 20 client.15717 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faaf99f9740 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556be69db160 0x556be69dd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faaf99f9740 4 client.15717 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:11.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.106+0000 7faaf99f9740 1 -- 172.21.15.33:0/2896208736 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556be6933850 con 0x556be69db160 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.107+0000 7faaf53bc640 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556be69db160 0x556be69dd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.107+0000 7faaf53bc640 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556be69db160 0x556be69dd550 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1547370317bcc758 server_cookie=da55bf387da6a047 in_seq=0 out_seq=0 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.107+0000 7faad4ff9640 10 client.15717 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faad4ff9640 1 -- 172.21.15.33:0/2896208736 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556be6933850 con 0x556be69db160 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faad4ff9640 10 client.15717 handle_command_reply: tid=0 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faaf99f9740 1 client.15717 shutdown 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faaf99f9740 1 -- 172.21.15.33:0/2896208736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c0dc0 msgr2=0x556be69c82b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faaf99f9740 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c0dc0 0x556be69c82b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7faae4012770 tx=0x7faae4012c30 comp rx=0 tx=0).stop 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faaf99f9740 1 -- 172.21.15.33:0/2896208736 shutdown_connections 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faaf99f9740 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556be69db160 0x556be69dd550 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faaf99f9740 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69cb680 0x556be69c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faaf99f9740 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c0dc0 0x556be69c82b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faaf99f9740 1 --2- 172.21.15.33:0/2896208736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69c04a0 0x556be69cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faad4ff9640 0 client.15717 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:11.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faaf99f9740 1 -- 172.21.15.33:0/2896208736 >> 172.21.15.33:0/2896208736 conn(0x556be69bca30 msgr2=0x556be69c2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:11.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faaf99f9740 1 -- 172.21.15.33:0/2896208736 shutdown_connections 2024-09-16T00:19:11.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faaf99f9740 1 -- 172.21.15.33:0/2896208736 wait complete. 2024-09-16T00:19:11.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.108+0000 7faaf99f9740 20 client.15717 trim_cache size 0 max 16384 2024-09-16T00:19:11.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf99f9740 1 Processor -- start 2024-09-16T00:19:11.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf99f9740 1 -- start start 2024-09-16T00:19:11.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf99f9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69ca220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf99f9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69c0dc0 0x556be69e2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf99f9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69cb680 0x556be69e3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf99f9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01b1920 con 0x556be69c04a0 2024-09-16T00:19:11.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf99f9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01af1f0 con 0x556be69c0dc0 2024-09-16T00:19:11.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf99f9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01ad040 con 0x556be69cb680 2024-09-16T00:19:11.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf53bc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf53bc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69ca220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45874/0 (socket says 172.21.15.33:45874) 2024-09-16T00:19:11.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf53bc640 1 -- 172.21.15.33:0/3812363510 learned_addr learned my addr 172.21.15.33:0/3812363510 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:11.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf4bbb640 1 --2- 172.21.15.33:0/3812363510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69c0dc0 0x556be69e2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf53bc640 1 -- 172.21.15.33:0/3812363510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69cb680 msgr2=0x556be69e3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:11.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf53bc640 1 --2- 172.21.15.33:0/3812363510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69cb680 0x556be69e3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf53bc640 1 -- 172.21.15.33:0/3812363510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69c0dc0 msgr2=0x556be69e2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:11.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf53bc640 1 --2- 172.21.15.33:0/3812363510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69c0dc0 0x556be69e2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.126+0000 7faaf53bc640 1 -- 172.21.15.33:0/3812363510 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556be69cb370 con 0x556be69c04a0 2024-09-16T00:19:11.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.127+0000 7faaf53bc640 1 --2- 172.21.15.33:0/3812363510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69ca220 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7faae80bbf80 tx=0x7faae8049bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2b20fc745b1ce3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:11.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.127+0000 7faad6ffd640 1 -- 172.21.15.33:0/3812363510 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8049040 con 0x556be69c04a0 2024-09-16T00:19:11.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.127+0000 7faad6ffd640 1 -- 172.21.15.33:0/3812363510 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae803b510 con 0x556be69c04a0 2024-09-16T00:19:11.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.127+0000 7faad6ffd640 1 -- 172.21.15.33:0/3812363510 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae80bbff0 con 0x556be69c04a0 2024-09-16T00:19:11.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.127+0000 7faaf99f9740 1 -- 172.21.15.33:0/3812363510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 msgr2=0x556be69ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:11.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.127+0000 7faaf99f9740 1 --2- 172.21.15.33:0/3812363510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69ca220 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7faae80bbf80 tx=0x7faae8049bd0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.128+0000 7faaf99f9740 1 -- 172.21.15.33:0/3812363510 shutdown_connections 2024-09-16T00:19:11.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.128+0000 7faaf99f9740 1 --2- 172.21.15.33:0/3812363510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69cb680 0x556be69e3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.128+0000 7faaf99f9740 1 --2- 172.21.15.33:0/3812363510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69c0dc0 0x556be69e2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.128+0000 7faaf99f9740 1 --2- 172.21.15.33:0/3812363510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69ca220 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.128+0000 7faaf99f9740 1 -- 172.21.15.33:0/3812363510 >> 172.21.15.33:0/3812363510 conn(0x556be69bca30 msgr2=0x556be69c9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:11.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.128+0000 7faaf99f9740 1 -- 172.21.15.33:0/3812363510 shutdown_connections 2024-09-16T00:19:11.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.128+0000 7faaf99f9740 1 -- 172.21.15.33:0/3812363510 wait complete. 2024-09-16T00:19:11.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.128+0000 7faaf99f9740 1 -- start start 2024-09-16T00:19:11.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.128+0000 7faaf99f9740 1 Processor -- start 2024-09-16T00:19:11.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf99f9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69d1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf99f9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69c0dc0 0x556be69d1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf99f9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69cb680 0x556be69d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf99f9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01ad6b0 con 0x556be69c04a0 2024-09-16T00:19:11.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf99f9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01aeb20 con 0x556be69cb680 2024-09-16T00:19:11.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf99f9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01add80 con 0x556be69c0dc0 2024-09-16T00:19:11.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf99f9740 10 client.0 _ll_register_callbacks cb 0x7faae400a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:11.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf4bbb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69c0dc0 0x556be69d1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf5bbd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69cb680 0x556be69d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf5bbd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69cb680 0x556be69d1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:48434/0 (socket says 172.21.15.33:48434) 2024-09-16T00:19:11.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf5bbd640 1 -- 172.21.15.33:0/4265931687 learned_addr learned my addr 172.21.15.33:0/4265931687 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:11.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf4bbb640 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69c0dc0 0x556be69d1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:55324/0 (socket says 172.21.15.33:55324) 2024-09-16T00:19:11.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf53bc640 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69d1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf5bbd640 1 -- 172.21.15.33:0/4265931687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69c0dc0 msgr2=0x556be69d1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:11.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf5bbd640 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69c0dc0 0x556be69d1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf5bbd640 1 -- 172.21.15.33:0/4265931687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 msgr2=0x556be69d1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:11.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf5bbd640 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69d1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf5bbd640 1 -- 172.21.15.33:0/4265931687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556be6a44e20 con 0x556be69cb680 2024-09-16T00:19:11.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf4bbb640 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69c0dc0 0x556be69d1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:19:11.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faaf5bbd640 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69cb680 0x556be69d1dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faae000ceb0 tx=0x7faae000cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a100605b3b6d1da4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:11.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.129+0000 7faad4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:19:11.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.130+0000 7faad4ff9640 1 -- 172.21.15.33:0/4265931687 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae001b040 con 0x556be69cb680 2024-09-16T00:19:11.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.130+0000 7faaf99f9740 1 -- 172.21.15.33:0/4265931687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556be692dd60 con 0x556be69cb680 2024-09-16T00:19:11.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.130+0000 7faad4ff9640 1 -- 172.21.15.33:0/4265931687 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7faae00055a0 con 0x556be69cb680 2024-09-16T00:19:11.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.130+0000 7faad4ff9640 1 -- 172.21.15.33:0/4265931687 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7faae000cc90 con 0x556be69cb680 2024-09-16T00:19:11.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.130+0000 7faaf99f9740 10 client.25343 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:11.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.130+0000 7faaf99f9740 1 -- 172.21.15.33:0/4265931687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556be6a460c0 con 0x556be69cb680 2024-09-16T00:19:11.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.130+0000 7faad4ff9640 1 -- 172.21.15.33:0/4265931687 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556be692dd60 con 0x556be69cb680 2024-09-16T00:19:11.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.130+0000 7faad4ff9640 1 -- 172.21.15.33:0/4265931687 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7faae0031480 con 0x556be69cb680 2024-09-16T00:19:11.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.130+0000 7faaf99f9740 10 client.25343 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:11.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.130+0000 7faaf99f9740 10 client.25343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:11.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.130+0000 7faaf99f9740 20 client.25343 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:11.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.130+0000 7faaf99f9740 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556be6a464b0 0x556be6a488a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:11.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.131+0000 7faaf99f9740 4 client.25343 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:19:11.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.131+0000 7faaf99f9740 1 -- 172.21.15.33:0/4265931687 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556be68aa210 con 0x556be6a464b0 2024-09-16T00:19:11.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.131+0000 7faaf53bc640 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556be6a464b0 0x556be6a488a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:11.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.131+0000 7faaf53bc640 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556be6a464b0 0x556be6a488a0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b724e67d529fa1d4 server_cookie=191c89fcdcdcb58b in_seq=0 out_seq=0 2024-09-16T00:19:11.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.131+0000 7faad4ff9640 10 client.25343 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:11.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.131+0000 7faad4ff9640 1 -- 172.21.15.33:0/4265931687 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556be68aa210 con 0x556be6a464b0 2024-09-16T00:19:11.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.131+0000 7faad4ff9640 10 client.25343 handle_command_reply: tid=0 2024-09-16T00:19:11.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.131+0000 7faaf99f9740 1 client.25343 shutdown 2024-09-16T00:19:11.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.131+0000 7faaf99f9740 1 -- 172.21.15.33:0/4265931687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69cb680 msgr2=0x556be69d1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:11.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.131+0000 7faaf99f9740 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69cb680 0x556be69d1dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faae000ceb0 tx=0x7faae000cb40 comp rx=0 tx=0).stop 2024-09-16T00:19:11.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.132+0000 7faaf99f9740 1 -- 172.21.15.33:0/4265931687 shutdown_connections 2024-09-16T00:19:11.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.132+0000 7faaf99f9740 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556be6a464b0 0x556be6a488a0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.132+0000 7faad4ff9640 0 client.25343 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:11.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.132+0000 7faaf99f9740 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556be69cb680 0x556be69d1dc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.132+0000 7faaf99f9740 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556be69c0dc0 0x556be69d1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.132+0000 7faaf99f9740 1 --2- 172.21.15.33:0/4265931687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556be69c04a0 0x556be69d1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.132+0000 7faaf99f9740 1 -- 172.21.15.33:0/4265931687 >> 172.21.15.33:0/4265931687 conn(0x556be69bca30 msgr2=0x556be69bb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:11.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.132+0000 7faaf99f9740 1 -- 172.21.15.33:0/4265931687 shutdown_connections 2024-09-16T00:19:11.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.132+0000 7faaf99f9740 1 -- 172.21.15.33:0/4265931687 wait complete. 2024-09-16T00:19:11.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.132+0000 7faaf99f9740 20 client.25343 trim_cache size 0 max 16384 2024-09-16T00:19:11.143 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:19:11.143 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:19:11.143 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:19:11.143 INFO:teuthology.orchestra.run.smithi033.stdout: "c0cbd37c-9c16-4d06-9ae4-7ef751697353": { 2024-09-16T00:19:11.143 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:19:11.143 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "c0cbd37c-9c16-4d06-9ae4-7ef751697353", 2024-09-16T00:19:11.144 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:19:11.144 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:19:11.144 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:19:11.144 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:19:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.133+0000 7faaf7620640 1 -- 172.21.15.33:0/3417990113 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faad80a3240 msgr2=0x7faad80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.133+0000 7faaf7620640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faad80a3240 0x7faad80a5700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faae0010c40 tx=0x7faae0017000 comp rx=0 tx=0).stop 2024-09-16T00:19:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.133+0000 7faaf7620640 1 -- 172.21.15.33:0/3417990113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faaf0101de0 msgr2=0x7faaf0198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.133+0000 7faaf7620640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faaf0101de0 0x7faaf0198390 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7faae8006010 tx=0x7faae80042f0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.133+0000 7faaf7620640 1 -- 172.21.15.33:0/3417990113 shutdown_connections 2024-09-16T00:19:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.133+0000 7faaf7620640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faad80a3240 0x7faad80a5700 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.133+0000 7faaf7620640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf0106f10 0x7faaf019cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.133+0000 7faaf7620640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaf0102790 0x7faaf01988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.133+0000 7faaf7620640 1 --2- 172.21.15.33:0/3417990113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faaf0101de0 0x7faaf0198390 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:11.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.133+0000 7faaf7620640 1 -- 172.21.15.33:0/3417990113 >> 172.21.15.33:0/3417990113 conn(0x7faaf00fd8c0 msgr2=0x7faaf00fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:11.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.133+0000 7faaf7620640 1 -- 172.21.15.33:0/3417990113 shutdown_connections 2024-09-16T00:19:11.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:11.133+0000 7faaf7620640 1 -- 172.21.15.33:0/3417990113 wait complete. 2024-09-16T00:19:11.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:11 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:19:11.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:11 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:11.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:11 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:19:11.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:11 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:11.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:11 smithi033 ceph-mon[31243]: pgmap v411: 129 pgs: 129 active+clean; 9.4 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 777 KiB/s wr, 104 op/s 2024-09-16T00:19:11.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:11 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:19:11.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:11 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:19:11.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:11 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:11.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:11 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:19:11.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:11 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:11.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:11 smithi083 ceph-mon[35242]: pgmap v411: 129 pgs: 129 active+clean; 9.4 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 777 KiB/s wr, 104 op/s 2024-09-16T00:19:11.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:11 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:19:11.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:11 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:19:11.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:11 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:11.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:11 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:19:11.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:11 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:11.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:11 smithi060 ceph-mon[35185]: pgmap v411: 129 pgs: 129 active+clean; 9.4 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 777 KiB/s wr, 104 op/s 2024-09-16T00:19:11.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:11 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:19:11.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 22198 7.18 MB/sec warmup 19 sec latency 3547.957 ms 2024-09-16T00:19:11.964 INFO:tasks.cephfs.filesystem:scrub status for tag:c0cbd37c-9c16-4d06-9ae4-7ef751697353 - {'path': '/', 'tag': 'c0cbd37c-9c16-4d06-9ae4-7ef751697353', 'options': 'recursive,force'} 2024-09-16T00:19:12.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 22304 6.83 MB/sec warmup 20 sec latency 3896.517 ms 2024-09-16T00:19:12.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:19:12.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:19:12.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:19:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:19:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:12.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:19:12.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:19:12.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:19:13.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 22304 6.50 MB/sec warmup 21 sec latency 1597.247 ms 2024-09-16T00:19:13.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:13 smithi060 ceph-mon[35185]: pgmap v412: 129 pgs: 129 active+clean; 11 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 459 KiB/s wr, 60 op/s 2024-09-16T00:19:13.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:13 smithi033 ceph-mon[31243]: pgmap v412: 129 pgs: 129 active+clean; 11 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 459 KiB/s wr, 60 op/s 2024-09-16T00:19:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:13 smithi083 ceph-mon[35242]: pgmap v412: 129 pgs: 129 active+clean; 11 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 459 KiB/s wr, 60 op/s 2024-09-16T00:19:14.644 INFO:tasks.workunit.client.0.smithi033.stdout: 1 22304 6.21 MB/sec warmup 22 sec latency 2597.329 ms 2024-09-16T00:19:15.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 22304 5.94 MB/sec warmup 23 sec latency 3597.405 ms 2024-09-16T00:19:15.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:15 smithi060 ceph-mon[35185]: pgmap v413: 129 pgs: 129 active+clean; 11 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 341 KiB/s wr, 50 op/s 2024-09-16T00:19:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:15 smithi033 ceph-mon[31243]: pgmap v413: 129 pgs: 129 active+clean; 11 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 341 KiB/s wr, 50 op/s 2024-09-16T00:19:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:15 smithi083 ceph-mon[35242]: pgmap v413: 129 pgs: 129 active+clean; 11 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 341 KiB/s wr, 50 op/s 2024-09-16T00:19:16.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 22304 5.69 MB/sec warmup 24 sec latency 4597.523 ms 2024-09-16T00:19:17.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 25104 6.18 MB/sec warmup 25 sec latency 4948.112 ms 2024-09-16T00:19:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:17 smithi060 ceph-mon[35185]: pgmap v414: 129 pgs: 129 active+clean; 11 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 321 KiB/s wr, 50 op/s 2024-09-16T00:19:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:17 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:19:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:17 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:19:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:17 smithi033 ceph-mon[31243]: pgmap v414: 129 pgs: 129 active+clean; 11 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 321 KiB/s wr, 50 op/s 2024-09-16T00:19:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:17 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:19:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:17 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:19:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:17 smithi083 ceph-mon[35242]: pgmap v414: 129 pgs: 129 active+clean; 11 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 321 KiB/s wr, 50 op/s 2024-09-16T00:19:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:17 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:19:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:17 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:19:18.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 25788 5.98 MB/sec warmup 26 sec latency 618.242 ms 2024-09-16T00:19:19.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 25788 5.76 MB/sec warmup 27 sec latency 1618.351 ms 2024-09-16T00:19:19.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:19 smithi060 ceph-mon[35185]: pgmap v415: 129 pgs: 129 active+clean; 11 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 190 KiB/s wr, 31 op/s 2024-09-16T00:19:19.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:19 smithi033 ceph-mon[31243]: pgmap v415: 129 pgs: 129 active+clean; 11 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 190 KiB/s wr, 31 op/s 2024-09-16T00:19:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:19 smithi083 ceph-mon[35242]: pgmap v415: 129 pgs: 129 active+clean; 11 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 190 KiB/s wr, 31 op/s 2024-09-16T00:19:20.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 25788 5.56 MB/sec warmup 28 sec latency 2618.471 ms 2024-09-16T00:19:21.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 25788 5.36 MB/sec warmup 29 sec latency 3618.520 ms 2024-09-16T00:19:21.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:21 smithi060 ceph-mon[35185]: pgmap v416: 129 pgs: 129 active+clean; 11 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 195 KiB/s wr, 31 op/s 2024-09-16T00:19:21.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:21 smithi033 ceph-mon[31243]: pgmap v416: 129 pgs: 129 active+clean; 11 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 195 KiB/s wr, 31 op/s 2024-09-16T00:19:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:21 smithi083 ceph-mon[35242]: pgmap v416: 129 pgs: 129 active+clean; 11 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 195 KiB/s wr, 31 op/s 2024-09-16T00:19:22.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 28580 5.76 MB/sec warmup 30 sec latency 3967.454 ms 2024-09-16T00:19:23.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 29290 5.63 MB/sec warmup 31 sec latency 570.617 ms 2024-09-16T00:19:23.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:23 smithi060 ceph-mon[35185]: pgmap v417: 129 pgs: 129 active+clean; 12 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 304 KiB/s wr, 47 op/s 2024-09-16T00:19:23.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:23 smithi033 ceph-mon[31243]: pgmap v417: 129 pgs: 129 active+clean; 12 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 304 KiB/s wr, 47 op/s 2024-09-16T00:19:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:23 smithi083 ceph-mon[35242]: pgmap v417: 129 pgs: 129 active+clean; 12 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 304 KiB/s wr, 47 op/s 2024-09-16T00:19:24.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 30743 5.65 MB/sec warmup 32 sec latency 1216.606 ms 2024-09-16T00:19:25.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 32838 5.87 MB/sec warmup 33 sec latency 117.102 ms 2024-09-16T00:19:25.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:25 smithi083 ceph-mon[35242]: pgmap v418: 129 pgs: 129 active+clean; 12 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 212 KiB/s wr, 31 op/s 2024-09-16T00:19:25.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:25 smithi060 ceph-mon[35185]: pgmap v418: 129 pgs: 129 active+clean; 12 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 212 KiB/s wr, 31 op/s 2024-09-16T00:19:25.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:25 smithi033 ceph-mon[31243]: pgmap v418: 129 pgs: 129 active+clean; 12 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 212 KiB/s wr, 31 op/s 2024-09-16T00:19:26.645 INFO:tasks.workunit.client.0.smithi033.stdout: 1 32838 5.70 MB/sec warmup 34 sec latency 1117.224 ms 2024-09-16T00:19:27.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 32878 5.54 MB/sec warmup 35 sec latency 1467.617 ms 2024-09-16T00:19:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:27 smithi060 ceph-mon[35185]: pgmap v419: 129 pgs: 129 active+clean; 24 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.3 MiB/s wr, 56 op/s 2024-09-16T00:19:27.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:27 smithi033 ceph-mon[31243]: pgmap v419: 129 pgs: 129 active+clean; 24 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.3 MiB/s wr, 56 op/s 2024-09-16T00:19:27.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:27 smithi083 ceph-mon[35242]: pgmap v419: 129 pgs: 129 active+clean; 24 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.3 MiB/s wr, 56 op/s 2024-09-16T00:19:28.646 INFO:tasks.workunit.client.0.smithi033.stdout: 1 32878 5.38 MB/sec warmup 36 sec latency 1618.762 ms 2024-09-16T00:19:29.646 INFO:tasks.workunit.client.0.smithi033.stdout: 1 32878 5.24 MB/sec warmup 37 sec latency 2618.797 ms 2024-09-16T00:19:29.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:29 smithi060 ceph-mon[35185]: pgmap v420: 129 pgs: 129 active+clean; 26 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 56 op/s 2024-09-16T00:19:29.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:29 smithi033 ceph-mon[31243]: pgmap v420: 129 pgs: 129 active+clean; 26 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 56 op/s 2024-09-16T00:19:29.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:29 smithi083 ceph-mon[35242]: pgmap v420: 129 pgs: 129 active+clean; 26 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 56 op/s 2024-09-16T00:19:30.646 INFO:tasks.workunit.client.0.smithi033.stdout: 1 32878 5.10 MB/sec warmup 38 sec latency 3618.855 ms 2024-09-16T00:19:31.646 INFO:tasks.workunit.client.0.smithi033.stdout: 1 32878 4.97 MB/sec warmup 39 sec latency 4618.886 ms 2024-09-16T00:19:31.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:31 smithi060 ceph-mon[35185]: pgmap v421: 129 pgs: 129 active+clean; 36 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 2.3 MiB/s wr, 83 op/s 2024-09-16T00:19:31.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:31 smithi033 ceph-mon[31243]: pgmap v421: 129 pgs: 129 active+clean; 36 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 2.3 MiB/s wr, 83 op/s 2024-09-16T00:19:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:31 smithi083 ceph-mon[35242]: pgmap v421: 129 pgs: 129 active+clean; 36 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 2.3 MiB/s wr, 83 op/s 2024-09-16T00:19:32.646 INFO:tasks.workunit.client.0.smithi033.stdout: 1 35517 5.26 MB/sec warmup 40 sec latency 4968.237 ms 2024-09-16T00:19:33.646 INFO:tasks.workunit.client.0.smithi033.stdout: 1 36381 5.19 MB/sec warmup 41 sec latency 482.274 ms 2024-09-16T00:19:33.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:33 smithi060 ceph-mon[35185]: pgmap v422: 129 pgs: 129 active+clean; 40 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 2.7 MiB/s wr, 92 op/s 2024-09-16T00:19:33.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:33 smithi033 ceph-mon[31243]: pgmap v422: 129 pgs: 129 active+clean; 40 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 2.7 MiB/s wr, 92 op/s 2024-09-16T00:19:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:33 smithi083 ceph-mon[35242]: pgmap v422: 129 pgs: 129 active+clean; 40 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 2.7 MiB/s wr, 92 op/s 2024-09-16T00:19:34.646 INFO:tasks.workunit.client.0.smithi033.stdout: 1 36381 5.07 MB/sec warmup 42 sec latency 1482.352 ms 2024-09-16T00:19:35.646 INFO:tasks.workunit.client.0.smithi033.stdout: 1 36381 4.95 MB/sec warmup 43 sec latency 2482.465 ms 2024-09-16T00:19:35.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:35 smithi033 ceph-mon[31243]: pgmap v423: 129 pgs: 129 active+clean; 41 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 3.1 MiB/s wr, 104 op/s 2024-09-16T00:19:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:35 smithi083 ceph-mon[35242]: pgmap v423: 129 pgs: 129 active+clean; 41 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 3.1 MiB/s wr, 104 op/s 2024-09-16T00:19:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:35 smithi060 ceph-mon[35185]: pgmap v423: 129 pgs: 129 active+clean; 41 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 3.1 MiB/s wr, 104 op/s 2024-09-16T00:19:36.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 36381 4.84 MB/sec warmup 44 sec latency 3482.549 ms 2024-09-16T00:19:37.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 36422 4.73 MB/sec warmup 45 sec latency 3832.068 ms 2024-09-16T00:19:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:37 smithi033 ceph-mon[31243]: pgmap v424: 129 pgs: 129 active+clean; 36 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 3.2 MiB/s wr, 118 op/s 2024-09-16T00:19:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:37 smithi083 ceph-mon[35242]: pgmap v424: 129 pgs: 129 active+clean; 36 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 3.2 MiB/s wr, 118 op/s 2024-09-16T00:19:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:37 smithi060 ceph-mon[35185]: pgmap v424: 129 pgs: 129 active+clean; 36 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 3.2 MiB/s wr, 118 op/s 2024-09-16T00:19:38.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 36422 4.63 MB/sec warmup 46 sec latency 1615.573 ms 2024-09-16T00:19:39.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 36422 4.53 MB/sec warmup 47 sec latency 2615.703 ms 2024-09-16T00:19:39.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:39 smithi033 ceph-mon[31243]: pgmap v425: 129 pgs: 129 active+clean; 36 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 2.1 MiB/s wr, 91 op/s 2024-09-16T00:19:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:39 smithi083 ceph-mon[35242]: pgmap v425: 129 pgs: 129 active+clean; 36 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 2.1 MiB/s wr, 91 op/s 2024-09-16T00:19:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:39 smithi060 ceph-mon[35185]: pgmap v425: 129 pgs: 129 active+clean; 36 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 2.1 MiB/s wr, 91 op/s 2024-09-16T00:19:40.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 36422 4.44 MB/sec warmup 48 sec latency 3615.804 ms 2024-09-16T00:19:41.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 36422 4.35 MB/sec warmup 49 sec latency 4615.869 ms 2024-09-16T00:19:41.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:41 smithi033 ceph-mon[31243]: pgmap v426: 129 pgs: 129 active+clean; 36 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.2 MiB/s wr, 95 op/s 2024-09-16T00:19:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:41 smithi083 ceph-mon[35242]: pgmap v426: 129 pgs: 129 active+clean; 36 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.2 MiB/s wr, 95 op/s 2024-09-16T00:19:41.959 INFO:tasks.quiescer.fs.[cephfs]:Total duration (39.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T00:19:41.959 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.55 2024-09-16T00:19:41.960 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:19:41.965 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:19:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:41 smithi060 ceph-mon[35185]: pgmap v426: 129 pgs: 129 active+clean; 36 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.2 MiB/s wr, 95 op/s 2024-09-16T00:19:42.282 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:42.335 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:42.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 38856 4.53 MB/sec warmup 50 sec latency 4964.650 ms 2024-09-16T00:19:43.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.117+0000 7fb43d16b640 1 -- 172.21.15.33:0/1686643371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4381008c0 msgr2=0x7fb438100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.117+0000 7fb43d16b640 1 --2- 172.21.15.33:0/1686643371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4381008c0 0x7fb438100ca0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb428009900 tx=0x7fb42802a410 comp rx=0 tx=0).stop 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.117+0000 7fb43d16b640 1 -- 172.21.15.33:0/1686643371 shutdown_connections 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.117+0000 7fb43d16b640 1 --2- 172.21.15.33:0/1686643371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb438105ec0 0x7fb438106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.117+0000 7fb43d16b640 1 --2- 172.21.15.33:0/1686643371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4381011e0 0x7fb438105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.117+0000 7fb43d16b640 1 --2- 172.21.15.33:0/1686643371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4381008c0 0x7fb438100ca0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.117+0000 7fb43d16b640 1 -- 172.21.15.33:0/1686643371 >> 172.21.15.33:0/1686643371 conn(0x7fb4380fc7d0 msgr2=0x7fb4380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.117+0000 7fb43d16b640 1 -- 172.21.15.33:0/1686643371 shutdown_connections 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.117+0000 7fb43d16b640 1 -- 172.21.15.33:0/1686643371 wait complete. 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb43d16b640 1 Processor -- start 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb43d16b640 1 -- start start 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb43d16b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4381008c0 0x7fb4381a48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb43d16b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4381011e0 0x7fb4381a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb43d16b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb438105ec0 0x7fb43819ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb43d16b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb438118610 con 0x7fb438105ec0 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb43d16b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb438118490 con 0x7fb4381011e0 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb43d16b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb438118790 con 0x7fb4381008c0 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb436d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4381008c0 0x7fb4381a48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb43759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb438105ec0 0x7fb43819ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb436d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4381008c0 0x7fb4381a48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41002/0 (socket says 172.21.15.33:41002) 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb436d9d640 1 -- 172.21.15.33:0/513119809 learned_addr learned my addr 172.21.15.33:0/513119809 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb42ffff640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4381011e0 0x7fb4381a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb436d9d640 1 -- 172.21.15.33:0/513119809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4381011e0 msgr2=0x7fb4381a4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb436d9d640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4381011e0 0x7fb4381a4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb436d9d640 1 -- 172.21.15.33:0/513119809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb438105ec0 msgr2=0x7fb43819ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb436d9d640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb438105ec0 0x7fb43819ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb436d9d640 1 -- 172.21.15.33:0/513119809 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43819f190 con 0x7fb4381008c0 2024-09-16T00:19:43.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.118+0000 7fb43759e640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb438105ec0 0x7fb43819ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:19:43.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.119+0000 7fb436d9d640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4381008c0 0x7fb4381a48a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb428006010 tx=0x7fb428003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:43.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.119+0000 7fb434d99640 1 -- 172.21.15.33:0/513119809 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42803f070 con 0x7fb4381008c0 2024-09-16T00:19:43.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.119+0000 7fb43d16b640 1 -- 172.21.15.33:0/513119809 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43819f420 con 0x7fb4381008c0 2024-09-16T00:19:43.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.119+0000 7fb434d99640 1 -- 172.21.15.33:0/513119809 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb428031c30 con 0x7fb4381008c0 2024-09-16T00:19:43.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.119+0000 7fb434d99640 1 -- 172.21.15.33:0/513119809 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb428035030 con 0x7fb4381008c0 2024-09-16T00:19:43.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.119+0000 7fb43d16b640 1 -- 172.21.15.33:0/513119809 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4381ab770 con 0x7fb4381008c0 2024-09-16T00:19:43.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.122+0000 7fb43f544740 1 Processor -- start 2024-09-16T00:19:43.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.122+0000 7fb43f544740 1 -- start start 2024-09-16T00:19:43.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.122+0000 7fb434d99640 1 -- 172.21.15.33:0/513119809 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb428042070 con 0x7fb4381008c0 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb43f544740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9e4a0 0x561df0c9e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb43f544740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9edc0 0x561df0ca9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb43f544740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0ca9680 0x561df0caba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb43f544740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561df0c042e0 con 0x561df0ca9680 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb43f544740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561df0babce0 con 0x561df0c9edc0 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb43f544740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561df0c002a0 con 0x561df0c9e4a0 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb42ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0ca9680 0x561df0caba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb43759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9e4a0 0x561df0c9e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb42ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0ca9680 0x561df0caba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35724/0 (socket says 172.21.15.33:35724) 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb42ffff640 1 -- 172.21.15.33:0/3563588362 learned_addr learned my addr 172.21.15.33:0/3563588362 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb436d9d640 1 --2- 172.21.15.33:0/3563588362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9edc0 0x561df0ca9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb43759e640 1 -- 172.21.15.33:0/3563588362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9edc0 msgr2=0x561df0ca9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb43759e640 1 --2- 172.21.15.33:0/3563588362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9edc0 0x561df0ca9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb43759e640 1 -- 172.21.15.33:0/3563588362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0ca9680 msgr2=0x561df0caba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb43759e640 1 --2- 172.21.15.33:0/3563588362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0ca9680 0x561df0caba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb43759e640 1 -- 172.21.15.33:0/3563588362 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561df0b90440 con 0x561df0c9e4a0 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.123+0000 7fb434d99640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4240a3240 0x7fb4240a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.124+0000 7fb43759e640 1 --2- 172.21.15.33:0/3563588362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9e4a0 0x561df0c9e880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb430011760 tx=0x7fb430011c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dda9e7be04c9a4d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.124+0000 7fb436d9d640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4240a3240 0x7fb4240a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.124+0000 7fb434d99640 1 -- 172.21.15.33:0/513119809 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb4280f0440 con 0x7fb4381008c0 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.124+0000 7fb42d7fa640 1 -- 172.21.15.33:0/3563588362 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430004d20 con 0x561df0c9e4a0 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.124+0000 7fb42d7fa640 1 -- 172.21.15.33:0/3563588362 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb430003970 con 0x561df0c9e4a0 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.124+0000 7fb42d7fa640 1 -- 172.21.15.33:0/3563588362 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4300044d0 con 0x561df0c9e4a0 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb43f544740 1 -- 172.21.15.33:0/3563588362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9e4a0 msgr2=0x561df0c9e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb436d9d640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4240a3240 0x7fb4240a5700 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb428005ab0 tx=0x7fb4280ed930 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb43f544740 1 --2- 172.21.15.33:0/3563588362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9e4a0 0x561df0c9e880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb430011760 tx=0x7fb430011c20 comp rx=0 tx=0).stop 2024-09-16T00:19:43.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb43f544740 1 -- 172.21.15.33:0/3563588362 shutdown_connections 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb43f544740 1 --2- 172.21.15.33:0/3563588362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0ca9680 0x561df0caba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb43f544740 1 --2- 172.21.15.33:0/3563588362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9edc0 0x561df0ca9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb43f544740 1 --2- 172.21.15.33:0/3563588362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9e4a0 0x561df0c9e880 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb43f544740 1 -- 172.21.15.33:0/3563588362 >> 172.21.15.33:0/3563588362 conn(0x561df0c9aa30 msgr2=0x561df0c9ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb43f544740 1 -- 172.21.15.33:0/3563588362 shutdown_connections 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb43f544740 1 -- 172.21.15.33:0/3563588362 wait complete. 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb43f544740 1 -- start start 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb43f544740 1 Processor -- start 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.125+0000 7fb43f544740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9e4a0 0x561df0ca8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb43f544740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9edc0 0x561df0ca6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb43f544740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0ca9680 0x561df0ca6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb43f544740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561df0cac250 con 0x561df0ca9680 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb43f544740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561df0bbf050 con 0x561df0c9e4a0 2024-09-16T00:19:43.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb43f544740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561df0cac3d0 con 0x561df0c9edc0 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb43f544740 10 client.0 _ll_register_callbacks cb 0x561df0bff160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb42ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9edc0 0x561df0ca6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb42ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9edc0 0x561df0ca6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41016/0 (socket says 172.21.15.33:41016) 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb42ffff640 1 -- 172.21.15.33:0/2898663869 learned_addr learned my addr 172.21.15.33:0/2898663869 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb43759e640 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0ca9680 0x561df0ca6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb43759e640 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9e4a0 0x561df0ca8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb42ffff640 1 -- 172.21.15.33:0/2898663869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9e4a0 msgr2=0x561df0ca8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb42ffff640 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9e4a0 0x561df0ca8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb42ffff640 1 -- 172.21.15.33:0/2898663869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0ca9680 msgr2=0x561df0ca6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb42ffff640 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0ca9680 0x561df0ca6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb42ffff640 1 -- 172.21.15.33:0/2898663869 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561df09753c0 con 0x561df0c9edc0 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb43759e640 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0ca9680 0x561df0ca6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:43.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb42ffff640 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9edc0 0x561df0ca6230 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb420012700 tx=0x7fb420012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61a1e444909cb5b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb41e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:19:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb41e7fc640 1 -- 172.21.15.33:0/2898663869 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb420004260 con 0x561df0c9edc0 2024-09-16T00:19:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb41e7fc640 1 -- 172.21.15.33:0/2898663869 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb420004400 con 0x561df0c9edc0 2024-09-16T00:19:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.126+0000 7fb43f544740 1 -- 172.21.15.33:0/2898663869 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb430003970 con 0x561df0c9edc0 2024-09-16T00:19:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.127+0000 7fb41e7fc640 1 -- 172.21.15.33:0/2898663869 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb420025050 con 0x561df0c9edc0 2024-09-16T00:19:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.127+0000 7fb43f544740 10 client.25291 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.127+0000 7fb43f544740 1 -- 172.21.15.33:0/2898663869 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561df0cb7c70 con 0x561df0c9edc0 2024-09-16T00:19:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.127+0000 7fb41e7fc640 1 -- 172.21.15.33:0/2898663869 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb430003970 con 0x561df0c9edc0 2024-09-16T00:19:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.127+0000 7fb41e7fc640 1 -- 172.21.15.33:0/2898663869 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb4200119a0 con 0x561df0c9edc0 2024-09-16T00:19:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.127+0000 7fb43f544740 10 client.25291 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.127+0000 7fb43f544740 10 client.25291 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.127+0000 7fb43f544740 20 client.25291 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.127+0000 7fb43f544740 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561df0cb9180 0x561df0cbb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.127+0000 7fb43f544740 4 client.25291 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.127+0000 7fb43f544740 1 -- 172.21.15.33:0/2898663869 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561df0c11850 con 0x561df0cb9180 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.128+0000 7fb43759e640 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561df0cb9180 0x561df0cbb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.128+0000 7fb43759e640 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561df0cb9180 0x561df0cbb5a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e8ef7cfbe27ae6 server_cookie=7d173a128b700967 in_seq=0 out_seq=0 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.128+0000 7fb41e7fc640 10 client.25291 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb41e7fc640 1 -- 172.21.15.33:0/2898663869 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561df0c11850 con 0x561df0cb9180 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb41e7fc640 10 client.25291 handle_command_reply: tid=0 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb43f544740 1 client.25291 shutdown 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb43f544740 1 -- 172.21.15.33:0/2898663869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9edc0 msgr2=0x561df0ca6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb43f544740 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9edc0 0x561df0ca6230 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb420012700 tx=0x7fb420012bc0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb43f544740 1 -- 172.21.15.33:0/2898663869 shutdown_connections 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb43f544740 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561df0cb9180 0x561df0cbb5a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb43f544740 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0ca9680 0x561df0ca6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb43f544740 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9edc0 0x561df0ca6230 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb43f544740 1 --2- 172.21.15.33:0/2898663869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9e4a0 0x561df0ca8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb41e7fc640 0 client.25291 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb43f544740 1 -- 172.21.15.33:0/2898663869 >> 172.21.15.33:0/2898663869 conn(0x561df0c9aa30 msgr2=0x561df0ca0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb43f544740 1 -- 172.21.15.33:0/2898663869 shutdown_connections 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb43f544740 1 -- 172.21.15.33:0/2898663869 wait complete. 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.129+0000 7fb43f544740 20 client.25291 trim_cache size 0 max 16384 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.152+0000 7fb43f544740 1 Processor -- start 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.152+0000 7fb43f544740 1 -- start start 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.152+0000 7fb43f544740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9e4a0 0x561df0cb54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb43f544740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 0x561df0cc0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb43f544740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0ca9680 0x561df0cc1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb43f544740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb4381b7ef0 con 0x561df0c9edc0 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb43f544740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb4381b57c0 con 0x561df0ca9680 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb43f544740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb4381b3610 con 0x561df0c9e4a0 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb42ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 0x561df0cc0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb42ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 0x561df0cc0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35742/0 (socket says 172.21.15.33:35742) 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb42ffff640 1 -- 172.21.15.33:0/1392482729 learned_addr learned my addr 172.21.15.33:0/1392482729 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb43759e640 1 --2- 172.21.15.33:0/1392482729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9e4a0 0x561df0cb54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb43759e640 1 --2- 172.21.15.33:0/1392482729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0ca9680 0x561df0cc1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb42ffff640 1 -- 172.21.15.33:0/1392482729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9e4a0 msgr2=0x561df0cb54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb42ffff640 1 --2- 172.21.15.33:0/1392482729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9e4a0 0x561df0cb54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb42ffff640 1 -- 172.21.15.33:0/1392482729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0ca9680 msgr2=0x561df0cc1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb42ffff640 1 --2- 172.21.15.33:0/1392482729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0ca9680 0x561df0cc1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb42ffff640 1 -- 172.21.15.33:0/1392482729 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561df0ca92f0 con 0x561df0c9edc0 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb43759e640 1 --2- 172.21.15.33:0/1392482729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9e4a0 0x561df0cb54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.153+0000 7fb42ffff640 1 --2- 172.21.15.33:0/1392482729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 0x561df0cc0e30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb420011fb0 tx=0x7fb420011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e971dbcabfdc51 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.154+0000 7fb42cff9640 1 -- 172.21.15.33:0/1392482729 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42002d020 con 0x561df0c9edc0 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.154+0000 7fb42cff9640 1 -- 172.21.15.33:0/1392482729 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb420003b50 con 0x561df0c9edc0 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.154+0000 7fb42cff9640 1 -- 172.21.15.33:0/1392482729 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42001c420 con 0x561df0c9edc0 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.155+0000 7fb43f544740 1 -- 172.21.15.33:0/1392482729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 msgr2=0x561df0cc0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.155+0000 7fb43f544740 1 --2- 172.21.15.33:0/1392482729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 0x561df0cc0e30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb420011fb0 tx=0x7fb420011bb0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.155+0000 7fb43f544740 1 -- 172.21.15.33:0/1392482729 shutdown_connections 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.155+0000 7fb43f544740 1 --2- 172.21.15.33:0/1392482729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0ca9680 0x561df0cc1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.155+0000 7fb43f544740 1 --2- 172.21.15.33:0/1392482729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 0x561df0cc0e30 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.155+0000 7fb43f544740 1 --2- 172.21.15.33:0/1392482729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0c9e4a0 0x561df0cb54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.155+0000 7fb43f544740 1 -- 172.21.15.33:0/1392482729 >> 172.21.15.33:0/1392482729 conn(0x561df0c9aa30 msgr2=0x561df0ca1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.155+0000 7fb43f544740 1 -- 172.21.15.33:0/1392482729 shutdown_connections 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.155+0000 7fb43f544740 1 -- 172.21.15.33:0/1392482729 wait complete. 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.156+0000 7fb43f544740 1 -- start start 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.156+0000 7fb43f544740 1 Processor -- start 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb43f544740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9e4a0 0x561df0cb1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb43f544740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 0x561df0cad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb42ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 0x561df0cad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb42ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 0x561df0cad0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35748/0 (socket says 172.21.15.33:35748) 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb43759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9e4a0 0x561df0cb1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb43759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9e4a0 0x561df0cb1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:44100/0 (socket says 172.21.15.33:44100) 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb42ffff640 1 -- 172.21.15.33:0/721301553 learned_addr learned my addr 172.21.15.33:0/721301553 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb43f544740 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0ca9680 0x561df0cad5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb43f544740 1 -- 172.21.15.33:0/721301553 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb4381b3c80 con 0x561df0c9edc0 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb43f544740 1 -- 172.21.15.33:0/721301553 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb4381b50f0 con 0x561df0c9e4a0 2024-09-16T00:19:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb43f544740 1 -- 172.21.15.33:0/721301553 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb4381b4350 con 0x561df0ca9680 2024-09-16T00:19:43.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb43759e640 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0ca9680 0x561df0cad5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb42ffff640 1 -- 172.21.15.33:0/721301553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0ca9680 msgr2=0x561df0cad5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:43.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb42ffff640 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0ca9680 0x561df0cad5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb42ffff640 1 -- 172.21.15.33:0/721301553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9e4a0 msgr2=0x561df0cb1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:43.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb42ffff640 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9e4a0 0x561df0cb1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb42ffff640 1 -- 172.21.15.33:0/721301553 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561df0caddc0 con 0x561df0c9edc0 2024-09-16T00:19:43.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.157+0000 7fb43759e640 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0ca9680 0x561df0cad5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:19:43.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.158+0000 7fb43f544740 10 client.0 _ll_register_callbacks cb 0x7fb420009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:43.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.158+0000 7fb43f544740 1 -- 172.21.15.33:0/721301553 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb420003b50 con 0x561df0c9edc0 2024-09-16T00:19:43.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.158+0000 7fb42ffff640 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 0x561df0cad0a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb420000c00 tx=0x7fb4200035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5bac1a999516d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:43.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.158+0000 7fb41e7fc640 10 client.15747 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:43.177 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:19:43.177 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:19:43.177 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:19:43.177 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:19:43.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.158+0000 7fb41e7fc640 1 -- 172.21.15.33:0/721301553 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb420040050 con 0x561df0c9edc0 2024-09-16T00:19:43.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.158+0000 7fb41e7fc640 1 -- 172.21.15.33:0/721301553 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb4200054e0 con 0x561df0c9edc0 2024-09-16T00:19:43.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.158+0000 7fb41e7fc640 1 -- 172.21.15.33:0/721301553 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb42002d030 con 0x561df0c9edc0 2024-09-16T00:19:43.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.158+0000 7fb43f544740 10 client.15747 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:43.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.158+0000 7fb43f544740 1 -- 172.21.15.33:0/721301553 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({fsmap=28}) -- 0x561df0d23ae0 con 0x561df0c9edc0 2024-09-16T00:19:43.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.159+0000 7fb41e7fc640 1 -- 172.21.15.33:0/721301553 <== mon.0 v2:172.21.15.33:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb420028040 con 0x561df0c9edc0 2024-09-16T00:19:43.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.159+0000 7fb43f544740 10 client.15747 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:43.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.159+0000 7fb43f544740 10 client.15747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:43.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.159+0000 7fb43f544740 20 client.15747 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.159+0000 7fb43f544740 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561df0d24110 0x561df0d26550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.159+0000 7fb43f544740 4 client.15747 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.159+0000 7fb43f544740 1 -- 172.21.15.33:0/721301553 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561df0c0bd60 con 0x561df0d24110 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.159+0000 7fb43759e640 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561df0d24110 0x561df0d26550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb41e7fc640 10 client.15747 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43759e640 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561df0d24110 0x561df0d26550 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dd247a639745077 server_cookie=d4fa6eb937105913 in_seq=0 out_seq=0 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb41e7fc640 1 -- 172.21.15.33:0/721301553 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561df0c0bd60 con 0x561df0d24110 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb41e7fc640 10 client.15747 handle_command_reply: tid=0 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43f544740 1 client.15747 shutdown 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43f544740 1 -- 172.21.15.33:0/721301553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 msgr2=0x561df0cad0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43f544740 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 0x561df0cad0a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb420000c00 tx=0x7fb4200035f0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43f544740 1 -- 172.21.15.33:0/721301553 shutdown_connections 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43f544740 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561df0d24110 0x561df0d26550 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43f544740 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561df0ca9680 0x561df0cad5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43f544740 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561df0c9edc0 0x561df0cad0a0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43f544740 1 --2- 172.21.15.33:0/721301553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561df0c9e4a0 0x561df0cb1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43f544740 1 -- 172.21.15.33:0/721301553 >> 172.21.15.33:0/721301553 conn(0x561df0c9aa30 msgr2=0x561df0c99350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb41e7fc640 0 client.15747 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43f544740 1 -- 172.21.15.33:0/721301553 shutdown_connections 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43f544740 1 -- 172.21.15.33:0/721301553 wait complete. 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.160+0000 7fb43f544740 20 client.15747 trim_cache size 0 max 16384 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.161+0000 7fb43d16b640 1 -- 172.21.15.33:0/513119809 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4240a3240 msgr2=0x7fb4240a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.161+0000 7fb43d16b640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4240a3240 0x7fb4240a5700 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb428005ab0 tx=0x7fb4280ed930 comp rx=0 tx=0).stop 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.161+0000 7fb43d16b640 1 -- 172.21.15.33:0/513119809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4381008c0 msgr2=0x7fb4381a48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.161+0000 7fb43d16b640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4381008c0 0x7fb4381a48a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb428006010 tx=0x7fb428003c10 comp rx=0 tx=0).stop 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.161+0000 7fb43d16b640 1 -- 172.21.15.33:0/513119809 shutdown_connections 2024-09-16T00:19:43.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.161+0000 7fb43d16b640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4240a3240 0x7fb4240a5700 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.161+0000 7fb43d16b640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb438105ec0 0x7fb43819ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.161+0000 7fb43d16b640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4381011e0 0x7fb4381a4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.161+0000 7fb43d16b640 1 --2- 172.21.15.33:0/513119809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4381008c0 0x7fb4381a48a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.161+0000 7fb43d16b640 1 -- 172.21.15.33:0/513119809 >> 172.21.15.33:0/513119809 conn(0x7fb4380fc7d0 msgr2=0x7fb438103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:43.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.161+0000 7fb43d16b640 1 -- 172.21.15.33:0/513119809 shutdown_connections 2024-09-16T00:19:43.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.161+0000 7fb43d16b640 1 -- 172.21.15.33:0/513119809 wait complete. 2024-09-16T00:19:43.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.021+0000 7ff654c29640 1 -- 172.21.15.33:0/1990458276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff65010c160 msgr2=0x7ff65010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.021+0000 7ff654c29640 1 --2- 172.21.15.33:0/1990458276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff65010c160 0x7ff65010e5e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff648009be0 tx=0x7ff64802a750 comp rx=0 tx=0).stop 2024-09-16T00:19:43.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.021+0000 7ff654c29640 1 -- 172.21.15.33:0/1990458276 shutdown_connections 2024-09-16T00:19:43.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.021+0000 7ff654c29640 1 --2- 172.21.15.33:0/1990458276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff65010c160 0x7ff65010e5e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.021+0000 7ff654c29640 1 --2- 172.21.15.33:0/1990458276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6501036a0 0x7ff65010bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.021+0000 7ff654c29640 1 --2- 172.21.15.33:0/1990458276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff650102d80 0x7ff650103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.021+0000 7ff654c29640 1 -- 172.21.15.33:0/1990458276 >> 172.21.15.33:0/1990458276 conn(0x7ff6500fc780 msgr2=0x7ff6500feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.021+0000 7ff654c29640 1 -- 172.21.15.33:0/1990458276 shutdown_connections 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.021+0000 7ff654c29640 1 -- 172.21.15.33:0/1990458276 wait complete. 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.022+0000 7ff654c29640 1 Processor -- start 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.022+0000 7ff654c29640 1 -- start start 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff654c29640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650102d80 0x7ff6501a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff654c29640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6501036a0 0x7ff6501a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff654c29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff65010c160 0x7ff65010adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff654c29640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff650111ef0 con 0x7ff6501036a0 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff654c29640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff650111d70 con 0x7ff650102d80 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff654c29640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff650112070 con 0x7ff65010c160 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff64dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6501036a0 0x7ff6501a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff64dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6501036a0 0x7ff6501a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35700/0 (socket says 172.21.15.33:35700) 2024-09-16T00:19:43.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff64dd9b640 1 -- 172.21.15.33:0/851900651 learned_addr learned my addr 172.21.15.33:0/851900651 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:43.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff64ed9d640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff65010c160 0x7ff65010adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff64e59c640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650102d80 0x7ff6501a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff64dd9b640 1 -- 172.21.15.33:0/851900651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff65010c160 msgr2=0x7ff65010adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff64dd9b640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff65010c160 0x7ff65010adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff64dd9b640 1 -- 172.21.15.33:0/851900651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650102d80 msgr2=0x7ff6501a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff64dd9b640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650102d80 0x7ff6501a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff64dd9b640 1 -- 172.21.15.33:0/851900651 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65010b4f0 con 0x7ff6501036a0 2024-09-16T00:19:43.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff64dd9b640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6501036a0 0x7ff6501a4c00 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff64400d950 tx=0x7ff64400dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:43.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.023+0000 7ff63f7fe640 1 -- 172.21.15.33:0/851900651 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff644017070 con 0x7ff6501036a0 2024-09-16T00:19:43.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.024+0000 7ff63f7fe640 1 -- 172.21.15.33:0/851900651 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff64400de80 con 0x7ff6501036a0 2024-09-16T00:19:43.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.024+0000 7ff654c29640 1 -- 172.21.15.33:0/851900651 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65010b7e0 con 0x7ff6501036a0 2024-09-16T00:19:43.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.024+0000 7ff63f7fe640 1 -- 172.21.15.33:0/851900651 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64400f620 con 0x7ff6501036a0 2024-09-16T00:19:43.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.024+0000 7ff654c29640 1 -- 172.21.15.33:0/851900651 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6501ab590 con 0x7ff6501036a0 2024-09-16T00:19:43.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.025+0000 7ff654c29640 1 -- 172.21.15.33:0/851900651 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff650103160 con 0x7ff6501036a0 2024-09-16T00:19:43.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.026+0000 7ff63f7fe640 1 -- 172.21.15.33:0/851900651 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff644009c30 con 0x7ff6501036a0 2024-09-16T00:19:43.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.027+0000 7ff63f7fe640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff6380a3240 0x7ff6380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:43.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.027+0000 7ff64e59c640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff6380a3240 0x7ff6380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:43.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.027+0000 7ff63f7fe640 1 -- 172.21.15.33:0/851900651 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff6440cf900 con 0x7ff6501036a0 2024-09-16T00:19:43.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.027+0000 7ff64e59c640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff6380a3240 0x7ff6380a5700 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff65010bf00 tx=0x7ff640009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:43.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.032+0000 7ff63f7fe640 1 -- 172.21.15.33:0/851900651 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff64409c1a0 con 0x7ff6501036a0 2024-09-16T00:19:43.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.281+0000 7ff654c29640 1 -- 172.21.15.33:0/851900651 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6501ab7f0 con 0x7ff6501036a0 2024-09-16T00:19:43.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.282+0000 7ff63f7fe640 1 -- 172.21.15.33:0/851900651 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11906 (secure 0 0 0) 0x7ff64409baa0 con 0x7ff6501036a0 2024-09-16T00:19:43.284 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:19:43.284 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"btime":"2024-09-16T00:19:02:115556+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:19:01.460559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:19:43.284 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 28 2024-09-16T00:19:43.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.284+0000 7ff654c29640 1 -- 172.21.15.33:0/851900651 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff6380a3240 msgr2=0x7ff6380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.284+0000 7ff654c29640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff6380a3240 0x7ff6380a5700 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff65010bf00 tx=0x7ff640009260 comp rx=0 tx=0).stop 2024-09-16T00:19:43.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.284+0000 7ff654c29640 1 -- 172.21.15.33:0/851900651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6501036a0 msgr2=0x7ff6501a4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:43.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.284+0000 7ff654c29640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6501036a0 0x7ff6501a4c00 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff64400d950 tx=0x7ff64400dd50 comp rx=0 tx=0).stop 2024-09-16T00:19:43.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.285+0000 7ff654c29640 1 -- 172.21.15.33:0/851900651 shutdown_connections 2024-09-16T00:19:43.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.285+0000 7ff654c29640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff6380a3240 0x7ff6380a5700 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.285+0000 7ff654c29640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff65010c160 0x7ff65010adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.285+0000 7ff654c29640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6501036a0 0x7ff6501a4c00 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.285+0000 7ff654c29640 1 --2- 172.21.15.33:0/851900651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650102d80 0x7ff6501a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:43.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.285+0000 7ff654c29640 1 -- 172.21.15.33:0/851900651 >> 172.21.15.33:0/851900651 conn(0x7ff6500fc780 msgr2=0x7ff6500fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:43.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.285+0000 7ff654c29640 1 -- 172.21.15.33:0/851900651 shutdown_connections 2024-09-16T00:19:43.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:43.285+0000 7ff654c29640 1 -- 172.21.15.33:0/851900651 wait complete. 2024-09-16T00:19:43.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:43 smithi033 ceph-mon[31243]: pgmap v427: 129 pgs: 129 active+clean; 37 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.4 MiB/s wr, 73 op/s 2024-09-16T00:19:43.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:43 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/851900651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:19:43.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39969 4.55 MB/sec warmup 51 sec latency 381.409 ms 2024-09-16T00:19:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:43 smithi083 ceph-mon[35242]: pgmap v427: 129 pgs: 129 active+clean; 37 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.4 MiB/s wr, 73 op/s 2024-09-16T00:19:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:43 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/851900651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:19:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:43 smithi060 ceph-mon[35185]: pgmap v427: 129 pgs: 129 active+clean; 37 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.4 MiB/s wr, 73 op/s 2024-09-16T00:19:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:43 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/851900651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:19:44.465 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0' 2024-09-16T00:19:44.465 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0 2024-09-16T00:19:44.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39969 4.46 MB/sec warmup 52 sec latency 1381.459 ms 2024-09-16T00:19:45.139 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:19:45.139 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:19:45.214 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:45.623 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:45 smithi033 ceph-mon[31243]: pgmap v428: 129 pgs: 129 active+clean; 38 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 1.2 MiB/s wr, 85 op/s 2024-09-16T00:19:45.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39969 4.38 MB/sec warmup 53 sec latency 2381.499 ms 2024-09-16T00:19:45.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.716+0000 7f37ffef2640 1 -- 172.21.15.33:0/251899820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37f8102570 msgr2=0x7f37f81029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:45.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.716+0000 7f37ffef2640 1 --2- 172.21.15.33:0/251899820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37f8102570 0x7f37f81029b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009900 tx=0x7f37e802a480 comp rx=0 tx=0).stop 2024-09-16T00:19:45.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.716+0000 7f37ffef2640 1 -- 172.21.15.33:0/251899820 shutdown_connections 2024-09-16T00:19:45.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.716+0000 7f37ffef2640 1 --2- 172.21.15.33:0/251899820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37f8102ef0 0x7f37f810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.716+0000 7f37ffef2640 1 --2- 172.21.15.33:0/251899820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37f8102570 0x7f37f81029b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.716+0000 7f37ffef2640 1 --2- 172.21.15.33:0/251899820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f8100e20 0x7f37f8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.716+0000 7f37ffef2640 1 -- 172.21.15.33:0/251899820 >> 172.21.15.33:0/251899820 conn(0x7f37f80fc7d0 msgr2=0x7f37f80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:45.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.716+0000 7f37ffef2640 1 -- 172.21.15.33:0/251899820 shutdown_connections 2024-09-16T00:19:45.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.716+0000 7f37ffef2640 1 -- 172.21.15.33:0/251899820 wait complete. 2024-09-16T00:19:45.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37ffef2640 1 Processor -- start 2024-09-16T00:19:45.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37ffef2640 1 -- start start 2024-09-16T00:19:45.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37ffef2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f8100e20 0x7f37f8197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37ffef2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37f8102570 0x7f37f81975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37ffef2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37f8102ef0 0x7f37f8197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37ffef2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f37f8110f80 con 0x7f37f8102ef0 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37ffef2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f37f8110e00 con 0x7f37f8100e20 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37ffef2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f37f8111100 con 0x7f37f8102570 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37fe48f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37f8102ef0 0x7f37f8197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37fe48f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37f8102ef0 0x7f37f8197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35774/0 (socket says 172.21.15.33:35774) 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37fe48f640 1 -- 172.21.15.33:0/3030351878 learned_addr learned my addr 172.21.15.33:0/3030351878 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37fd48d640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37f8102570 0x7f37f81975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37fdc8e640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f8100e20 0x7f37f8197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37fe48f640 1 -- 172.21.15.33:0/3030351878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37f8102570 msgr2=0x7f37f81975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37fe48f640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37f8102570 0x7f37f81975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37fe48f640 1 -- 172.21.15.33:0/3030351878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f8100e20 msgr2=0x7f37f8197080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37fe48f640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f8100e20 0x7f37f8197080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37fe48f640 1 -- 172.21.15.33:0/3030351878 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f81a68f0 con 0x7f37f8102ef0 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.717+0000 7f37fe48f640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37f8102ef0 0x7f37f8197b00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f37f000cce0 tx=0x7f37f000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.718+0000 7f37eeffd640 1 -- 172.21.15.33:0/3030351878 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f000f450 con 0x7f37f8102ef0 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.718+0000 7f37eeffd640 1 -- 172.21.15.33:0/3030351878 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37f000fad0 con 0x7f37f8102ef0 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.718+0000 7f37ffef2640 1 -- 172.21.15.33:0/3030351878 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f81a6be0 con 0x7f37f8102ef0 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.718+0000 7f37eeffd640 1 -- 172.21.15.33:0/3030351878 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0018c10 con 0x7f37f8102ef0 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.718+0000 7f37ffef2640 1 -- 172.21.15.33:0/3030351878 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f81a70f0 con 0x7f37f8102ef0 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.721+0000 7f38022cb740 1 Processor -- start 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.721+0000 7f37eeffd640 1 -- 172.21.15.33:0/3030351878 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f37f00213f0 con 0x7f37f8102ef0 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.721+0000 7f38022cb740 1 -- start start 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.721+0000 7f38022cb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f1850 0x55e8b48f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f38022cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 0x55e8b48fc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f38022cb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 0x55e8b48fee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f38022cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e8b4856160 con 0x55e8b48f2170 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f38022cb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e8b47fd720 con 0x55e8b48f1850 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f38022cb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e8b4852120 con 0x55e8b48fca30 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fdc8e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 0x55e8b48fc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fdc8e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 0x55e8b48fc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35778/0 (socket says 172.21.15.33:35778) 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fdc8e640 1 -- 172.21.15.33:0/2351252881 learned_addr learned my addr 172.21.15.33:0/2351252881 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37eeffd640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37e40a32c0 0x7f37e40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fd48d640 1 --2- 172.21.15.33:0/2351252881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 0x55e8b48fee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fdc8e640 1 -- 172.21.15.33:0/2351252881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 msgr2=0x55e8b48fee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fdc8e640 1 --2- 172.21.15.33:0/2351252881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 0x55e8b48fee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fd48d640 1 --2- 172.21.15.33:0/2351252881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f1850 0x55e8b48f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fdc8e640 1 -- 172.21.15.33:0/2351252881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f1850 msgr2=0x55e8b48f1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fdc8e640 1 --2- 172.21.15.33:0/2351252881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f1850 0x55e8b48f1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fdc8e640 1 -- 172.21.15.33:0/2351252881 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8b45c7ba0 con 0x55e8b48f2170 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fd48d640 1 --2- 172.21.15.33:0/2351252881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f1850 0x55e8b48f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37eeffd640 1 -- 172.21.15.33:0/3030351878 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f37f00d03f0 con 0x7f37f8102ef0 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fe48f640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37e40a32c0 0x7f37e40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.722+0000 7f37fdc8e640 1 --2- 172.21.15.33:0/2351252881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 0x55e8b48fc4f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f37f400cbf0 tx=0x7f37f400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dee37844d459ca0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f37e3fff640 1 -- 172.21.15.33:0/2351252881 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4010630 con 0x55e8b48f2170 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f37e3fff640 1 -- 172.21.15.33:0/2351252881 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37f4010cb0 con 0x55e8b48f2170 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f37e3fff640 1 -- 172.21.15.33:0/2351252881 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f400fc60 con 0x55e8b48f2170 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f37fe48f640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37e40a32c0 0x7f37e40a5780 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f37f000ab20 tx=0x7f37f000af30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f38022cb740 1 -- 172.21.15.33:0/2351252881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 msgr2=0x55e8b48fc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f38022cb740 1 --2- 172.21.15.33:0/2351252881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 0x55e8b48fc4f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f37f400cbf0 tx=0x7f37f400e720 comp rx=0 tx=0).stop 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f38022cb740 1 -- 172.21.15.33:0/2351252881 shutdown_connections 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f38022cb740 1 --2- 172.21.15.33:0/2351252881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 0x55e8b48fee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f38022cb740 1 --2- 172.21.15.33:0/2351252881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 0x55e8b48fc4f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f38022cb740 1 --2- 172.21.15.33:0/2351252881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f1850 0x55e8b48f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f38022cb740 1 -- 172.21.15.33:0/2351252881 >> 172.21.15.33:0/2351252881 conn(0x55e8b48edde0 msgr2=0x55e8b48ee1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f38022cb740 1 -- 172.21.15.33:0/2351252881 shutdown_connections 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.723+0000 7f38022cb740 1 -- 172.21.15.33:0/2351252881 wait complete. 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.724+0000 7f38022cb740 1 -- start start 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.724+0000 7f38022cb740 1 Processor -- start 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.724+0000 7f38022cb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48f1850 0x55e8b48f9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.724+0000 7f38022cb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f2170 0x55e8b48f9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.724+0000 7f38022cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48fca30 0x55e8b48fa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.724+0000 7f38022cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e8b48ff600 con 0x55e8b48fca30 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.724+0000 7f38022cb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e8b47de900 con 0x55e8b48f2170 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.724+0000 7f38022cb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e8b48ff780 con 0x55e8b48f1850 2024-09-16T00:19:45.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.724+0000 7f38022cb740 10 client.0 _ll_register_callbacks cb 0x55e8b47dc780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.724+0000 7f37fd48d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48fca30 0x55e8b48fa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.724+0000 7f37fd48d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48fca30 0x55e8b48fa2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35788/0 (socket says 172.21.15.33:35788) 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.724+0000 7f37fd48d640 1 -- 172.21.15.33:0/807293419 learned_addr learned my addr 172.21.15.33:0/807293419 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f37fdc8e640 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f2170 0x55e8b48f9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f37fd48d640 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48f1850 0x55e8b48f9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f37fd48d640 1 -- 172.21.15.33:0/807293419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48f1850 msgr2=0x55e8b48f9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f37fd48d640 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48f1850 0x55e8b48f9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f37fd48d640 1 -- 172.21.15.33:0/807293419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f2170 msgr2=0x55e8b48f9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f37fd48d640 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f2170 0x55e8b48f9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f37fd48d640 1 -- 172.21.15.33:0/807293419 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8b481ec50 con 0x55e8b48fca30 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f37fd48d640 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48fca30 0x55e8b48fa2f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009900 tx=0x7f37e802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=777731b806e749f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f37e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:45.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f37e17fa640 1 -- 172.21.15.33:0/807293419 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e8035040 con 0x55e8b48fca30 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f37e17fa640 1 -- 172.21.15.33:0/807293419 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f37e8003cf0 con 0x55e8b48fca30 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f38022cb740 1 -- 172.21.15.33:0/807293419 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37f4010cb0 con 0x55e8b48fca30 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f37e17fa640 1 -- 172.21.15.33:0/807293419 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f37e8031540 con 0x55e8b48fca30 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f38022cb740 10 client.15759 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.725+0000 7f38022cb740 1 -- 172.21.15.33:0/807293419 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e8b490b170 con 0x55e8b48fca30 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.726+0000 7f37e17fa640 1 -- 172.21.15.33:0/807293419 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37f4010cb0 con 0x55e8b48fca30 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.726+0000 7f37e17fa640 1 -- 172.21.15.33:0/807293419 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f37e803d780 con 0x55e8b48fca30 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.726+0000 7f38022cb740 10 client.15759 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.726+0000 7f38022cb740 10 client.15759 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.726+0000 7f38022cb740 20 client.15759 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.726+0000 7f38022cb740 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b490c630 0x55e8b490ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.726+0000 7f38022cb740 4 client.15759 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:45.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.726+0000 7f38022cb740 1 -- 172.21.15.33:0/807293419 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8b4863890 con 0x55e8b490c630 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.726+0000 7f37fdc8e640 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b490c630 0x55e8b490ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.727+0000 7f37e17fa640 10 client.15759 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.727+0000 7f37fdc8e640 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b490c630 0x55e8b490ea90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b74c9fef321a797 server_cookie=45cace2bf4216ae8 in_seq=0 out_seq=0 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.727+0000 7f37e17fa640 1 -- 172.21.15.33:0/807293419 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e8b4863890 con 0x55e8b490c630 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.727+0000 7f37e17fa640 10 client.15759 handle_command_reply: tid=0 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.727+0000 7f38022cb740 1 client.15759 shutdown 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.727+0000 7f38022cb740 1 -- 172.21.15.33:0/807293419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48fca30 msgr2=0x55e8b48fa2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.727+0000 7f38022cb740 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48fca30 0x55e8b48fa2f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009900 tx=0x7f37e802a480 comp rx=0 tx=0).stop 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.728+0000 7f38022cb740 1 -- 172.21.15.33:0/807293419 shutdown_connections 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.728+0000 7f38022cb740 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b490c630 0x55e8b490ea90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.728+0000 7f38022cb740 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48fca30 0x55e8b48fa2f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.728+0000 7f38022cb740 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f2170 0x55e8b48f9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.728+0000 7f37e17fa640 0 client.15759 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:45.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.728+0000 7f38022cb740 1 --2- 172.21.15.33:0/807293419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48f1850 0x55e8b48f9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.728+0000 7f38022cb740 1 -- 172.21.15.33:0/807293419 >> 172.21.15.33:0/807293419 conn(0x55e8b48edde0 msgr2=0x55e8b48f3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:45.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.728+0000 7f38022cb740 1 -- 172.21.15.33:0/807293419 shutdown_connections 2024-09-16T00:19:45.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.728+0000 7f38022cb740 1 -- 172.21.15.33:0/807293419 wait complete. 2024-09-16T00:19:45.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.728+0000 7f38022cb740 20 client.15759 trim_cache size 0 max 16384 2024-09-16T00:19:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.748+0000 7f38022cb740 1 Processor -- start 2024-09-16T00:19:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f38022cb740 1 -- start start 2024-09-16T00:19:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f38022cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f1850 0x55e8b48f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f38022cb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f2170 0x55e8b48fb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f38022cb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 0x55e8b4914360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f38022cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f37f81b3a40 con 0x55e8b48f1850 2024-09-16T00:19:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f38022cb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f37f81b1310 con 0x55e8b48f2170 2024-09-16T00:19:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f38022cb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f37f81af160 con 0x55e8b48fca30 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f37fd48d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f1850 0x55e8b48f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f37fd48d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f1850 0x55e8b48f42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35804/0 (socket says 172.21.15.33:35804) 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f37fd48d640 1 -- 172.21.15.33:0/839559707 learned_addr learned my addr 172.21.15.33:0/839559707 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f37fdc8e640 1 --2- 172.21.15.33:0/839559707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f2170 0x55e8b48fb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f37fd48d640 1 -- 172.21.15.33:0/839559707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 msgr2=0x55e8b4914360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f37fd48d640 1 --2- 172.21.15.33:0/839559707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 0x55e8b4914360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f37fd48d640 1 -- 172.21.15.33:0/839559707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f2170 msgr2=0x55e8b48fb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f37fd48d640 1 --2- 172.21.15.33:0/839559707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f2170 0x55e8b48fb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.749+0000 7f37fd48d640 1 -- 172.21.15.33:0/839559707 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8b48e93f0 con 0x55e8b48f1850 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.750+0000 7f37fd48d640 1 --2- 172.21.15.33:0/839559707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f1850 0x55e8b48f42b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009890 tx=0x7f37e8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffff23550ed8cff1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.750+0000 7f37e37fe640 1 -- 172.21.15.33:0/839559707 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e8000e60 con 0x55e8b48f1850 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.750+0000 7f37e37fe640 1 -- 172.21.15.33:0/839559707 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37e803d480 con 0x55e8b48f1850 2024-09-16T00:19:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.750+0000 7f37e37fe640 1 -- 172.21.15.33:0/839559707 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e8035040 con 0x55e8b48f1850 2024-09-16T00:19:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.750+0000 7f38022cb740 1 -- 172.21.15.33:0/839559707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f1850 msgr2=0x55e8b48f42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.750+0000 7f38022cb740 1 --2- 172.21.15.33:0/839559707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f1850 0x55e8b48f42b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009890 tx=0x7f37e8000bc0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.751+0000 7f38022cb740 1 -- 172.21.15.33:0/839559707 shutdown_connections 2024-09-16T00:19:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.751+0000 7f38022cb740 1 --2- 172.21.15.33:0/839559707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 0x55e8b4914360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.751+0000 7f38022cb740 1 --2- 172.21.15.33:0/839559707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f2170 0x55e8b48fb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.751+0000 7f38022cb740 1 --2- 172.21.15.33:0/839559707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f1850 0x55e8b48f42b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.751+0000 7f38022cb740 1 -- 172.21.15.33:0/839559707 >> 172.21.15.33:0/839559707 conn(0x55e8b48edde0 msgr2=0x55e8b48f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.751+0000 7f38022cb740 1 -- 172.21.15.33:0/839559707 shutdown_connections 2024-09-16T00:19:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.751+0000 7f38022cb740 1 -- 172.21.15.33:0/839559707 wait complete. 2024-09-16T00:19:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.751+0000 7f38022cb740 1 -- start start 2024-09-16T00:19:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.751+0000 7f38022cb740 1 Processor -- start 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f38022cb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f1850 0x55e8b4905170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f38022cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 0x55e8b49004a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f38022cb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 0x55e8b49009e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f38022cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f37f81af7d0 con 0x55e8b48f2170 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f38022cb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f37f81b0c40 con 0x55e8b48f1850 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f38022cb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f37f81afea0 con 0x55e8b48fca30 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f38022cb740 10 client.0 _ll_register_callbacks cb 0x7f37e8005bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f37fdc8e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 0x55e8b49004a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f37fdc8e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 0x55e8b49004a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35820/0 (socket says 172.21.15.33:35820) 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f37fdc8e640 1 -- 172.21.15.33:0/2333957185 learned_addr learned my addr 172.21.15.33:0/2333957185 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f37fd48d640 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f1850 0x55e8b4905170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f37fd48d640 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 0x55e8b49009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f37fdc8e640 1 -- 172.21.15.33:0/2333957185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 msgr2=0x55e8b49009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f37fdc8e640 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 0x55e8b49009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f37fdc8e640 1 -- 172.21.15.33:0/2333957185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f1850 msgr2=0x55e8b4905170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f37fdc8e640 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f1850 0x55e8b4905170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f37fdc8e640 1 -- 172.21.15.33:0/2333957185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8b48eea70 con 0x55e8b48f2170 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.752+0000 7f37fd48d640 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 0x55e8b49009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.753+0000 7f37fdc8e640 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 0x55e8b49004a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f37f400f4a0 tx=0x7f37f4005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44238e0863f672cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.753+0000 7f37e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.753+0000 7f37e17fa640 1 -- 172.21.15.33:0/2333957185 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4010e60 con 0x55e8b48f2170 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.753+0000 7f37e17fa640 1 -- 172.21.15.33:0/2333957185 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f37f4005040 con 0x55e8b48f2170 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.753+0000 7f38022cb740 1 -- 172.21.15.33:0/2333957185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37e803d480 con 0x55e8b48f2170 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.753+0000 7f37e17fa640 1 -- 172.21.15.33:0/2333957185 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f37f4037050 con 0x55e8b48f2170 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.753+0000 7f38022cb740 10 client.15765 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.753+0000 7f38022cb740 1 -- 172.21.15.33:0/2333957185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e8b497f9b0 con 0x55e8b48f2170 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.753+0000 7f37e17fa640 1 -- 172.21.15.33:0/2333957185 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37e803d480 con 0x55e8b48f2170 2024-09-16T00:19:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.754+0000 7f37e17fa640 1 -- 172.21.15.33:0/2333957185 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f37f4004660 con 0x55e8b48f2170 2024-09-16T00:19:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.754+0000 7f38022cb740 10 client.15765 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.754+0000 7f38022cb740 10 client.15765 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:19:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.754+0000 7f38022cb740 20 client.15765 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.754+0000 7f38022cb740 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b497fef0 0x55e8b4982310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.754+0000 7f38022cb740 4 client.15765 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}] 2024-09-16T00:19:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.754+0000 7f38022cb740 1 -- 172.21.15.33:0/2333957185 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}) -- 0x55e8b485dbe0 con 0x55e8b497fef0 2024-09-16T00:19:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.754+0000 7f37fd48d640 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b497fef0 0x55e8b4982310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.755+0000 7f37e17fa640 10 client.15765 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.755+0000 7f37fd48d640 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b497fef0 0x55e8b4982310 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=839ea25438ee3ad5 server_cookie=c30dedcc02b97989 in_seq=0 out_seq=0 2024-09-16T00:19:45.771 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:45.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f37e17fa640 1 -- 172.21.15.33:0/2333957185 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+587 (crc 0 0 0) 0x55e8b485dbe0 con 0x55e8b497fef0 2024-09-16T00:19:45.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f37e17fa640 10 client.15765 handle_command_reply: tid=0 2024-09-16T00:19:45.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f38022cb740 1 client.15765 shutdown 2024-09-16T00:19:45.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f38022cb740 1 -- 172.21.15.33:0/2333957185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 msgr2=0x55e8b49004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:45.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f38022cb740 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 0x55e8b49004a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f37f400f4a0 tx=0x7f37f4005ce0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f38022cb740 1 -- 172.21.15.33:0/2333957185 shutdown_connections 2024-09-16T00:19:45.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f38022cb740 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b497fef0 0x55e8b4982310 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f38022cb740 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b48fca30 0x55e8b49009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f38022cb740 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b48f2170 0x55e8b49004a0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f38022cb740 1 --2- 172.21.15.33:0/2333957185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b48f1850 0x55e8b4905170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f38022cb740 1 -- 172.21.15.33:0/2333957185 >> 172.21.15.33:0/2333957185 conn(0x55e8b48edde0 msgr2=0x55e8b48ecec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:45.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f37e17fa640 0 client.15765 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:45.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f38022cb740 1 -- 172.21.15.33:0/2333957185 shutdown_connections 2024-09-16T00:19:45.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f38022cb740 1 -- 172.21.15.33:0/2333957185 wait complete. 2024-09-16T00:19:45.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.826+0000 7f38022cb740 20 client.15765 trim_cache size 0 max 16384 2024-09-16T00:19:45.829 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:19:45.829 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 28, 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 6, 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: "5dd683": { 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 6, 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 139.6, 2024-09-16T00:19:45.830 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:19:45.831 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:19:45.831 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:19:45.831 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:19:45.831 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:19:45.831 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:19:45.831 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:19:45.831 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:19:45.831 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:19:45.831 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:19:45.831 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:19:45.831 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:19:45.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.827+0000 7f37ffef2640 1 -- 172.21.15.33:0/3030351878 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37e40a32c0 msgr2=0x7f37e40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:45.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.827+0000 7f37ffef2640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37e40a32c0 0x7f37e40a5780 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f37f000ab20 tx=0x7f37f000af30 comp rx=0 tx=0).stop 2024-09-16T00:19:45.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.827+0000 7f37ffef2640 1 -- 172.21.15.33:0/3030351878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37f8102ef0 msgr2=0x7f37f8197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:45.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.827+0000 7f37ffef2640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37f8102ef0 0x7f37f8197b00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f37f000cce0 tx=0x7f37f000a620 comp rx=0 tx=0).stop 2024-09-16T00:19:45.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.827+0000 7f37ffef2640 1 -- 172.21.15.33:0/3030351878 shutdown_connections 2024-09-16T00:19:45.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.827+0000 7f37ffef2640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37e40a32c0 0x7f37e40a5780 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.827+0000 7f37ffef2640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37f8102ef0 0x7f37f8197b00 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.827+0000 7f37ffef2640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37f8102570 0x7f37f81975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.827+0000 7f37ffef2640 1 --2- 172.21.15.33:0/3030351878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f8100e20 0x7f37f8197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:45.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.827+0000 7f37ffef2640 1 -- 172.21.15.33:0/3030351878 >> 172.21.15.33:0/3030351878 conn(0x7f37f80fc7d0 msgr2=0x7f37f80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:45.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.827+0000 7f37ffef2640 1 -- 172.21.15.33:0/3030351878 shutdown_connections 2024-09-16T00:19:45.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:45.827+0000 7f37ffef2640 1 -- 172.21.15.33:0/3030351878 wait complete. 2024-09-16T00:19:45.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:45 smithi083 ceph-mon[35242]: pgmap v428: 129 pgs: 129 active+clean; 38 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 1.2 MiB/s wr, 85 op/s 2024-09-16T00:19:45.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:45 smithi060 ceph-mon[35185]: pgmap v428: 129 pgs: 129 active+clean; 38 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 1.2 MiB/s wr, 85 op/s 2024-09-16T00:19:46.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.582+0000 7fcb125e6640 1 -- 172.21.15.33:0/4270746455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb0c100a80 msgr2=0x7fcb0c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:46.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.582+0000 7fcb125e6640 1 --2- 172.21.15.33:0/4270746455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb0c100a80 0x7fcb0c100ee0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009930 tx=0x7fcaf402a420 comp rx=0 tx=0).stop 2024-09-16T00:19:46.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.582+0000 7fcb125e6640 1 -- 172.21.15.33:0/4270746455 shutdown_connections 2024-09-16T00:19:46.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.582+0000 7fcb125e6640 1 --2- 172.21.15.33:0/4270746455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb0c101420 0x7fcb0c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:46.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.582+0000 7fcb125e6640 1 --2- 172.21.15.33:0/4270746455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb0c100a80 0x7fcb0c100ee0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:46.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.582+0000 7fcb125e6640 1 --2- 172.21.15.33:0/4270746455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c106a80 0x7fcb0c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:46.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.582+0000 7fcb125e6640 1 -- 172.21.15.33:0/4270746455 >> 172.21.15.33:0/4270746455 conn(0x7fcb0c0fc7a0 msgr2=0x7fcb0c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:46.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.582+0000 7fcb125e6640 1 -- 172.21.15.33:0/4270746455 shutdown_connections 2024-09-16T00:19:46.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.582+0000 7fcb125e6640 1 -- 172.21.15.33:0/4270746455 wait complete. 2024-09-16T00:19:46.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.583+0000 7fcb125e6640 1 Processor -- start 2024-09-16T00:19:46.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.583+0000 7fcb125e6640 1 -- start start 2024-09-16T00:19:46.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.583+0000 7fcb125e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb0c100a80 0x7fcb0c1992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:46.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.583+0000 7fcb125e6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c101420 0x7fcb0c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:46.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.583+0000 7fcb125e6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb0c106a80 0x7fcb0c1a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:46.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.583+0000 7fcb125e6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c110e80 con 0x7fcb0c100a80 2024-09-16T00:19:46.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.583+0000 7fcb125e6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c110d00 con 0x7fcb0c101420 2024-09-16T00:19:46.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.583+0000 7fcb125e6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c111000 con 0x7fcb0c106a80 2024-09-16T00:19:46.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.583+0000 7fcb0bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb0c100a80 0x7fcb0c1992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:46.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.583+0000 7fcb0bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb0c100a80 0x7fcb0c1992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35826/0 (socket says 172.21.15.33:35826) 2024-09-16T00:19:46.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.583+0000 7fcb0bfff640 1 -- 172.21.15.33:0/3841406757 learned_addr learned my addr 172.21.15.33:0/3841406757 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:46.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb10b83640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb0c106a80 0x7fcb0c1a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:46.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb0b7fe640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c101420 0x7fcb0c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:46.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb0bfff640 1 -- 172.21.15.33:0/3841406757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb0c106a80 msgr2=0x7fcb0c1a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:46.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb0bfff640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb0c106a80 0x7fcb0c1a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:46.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb0bfff640 1 -- 172.21.15.33:0/3841406757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c101420 msgr2=0x7fcb0c1997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:46.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb0bfff640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c101420 0x7fcb0c1997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:46.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb0bfff640 1 -- 172.21.15.33:0/3841406757 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c1a4870 con 0x7fcb0c100a80 2024-09-16T00:19:46.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb0b7fe640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c101420 0x7fcb0c1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:46.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb0bfff640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb0c100a80 0x7fcb0c1992b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00d950 tx=0x7fcafc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:46.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb097fa640 1 -- 172.21.15.33:0/3841406757 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc017070 con 0x7fcb0c100a80 2024-09-16T00:19:46.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb097fa640 1 -- 172.21.15.33:0/3841406757 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcafc009c30 con 0x7fcb0c100a80 2024-09-16T00:19:46.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb097fa640 1 -- 172.21.15.33:0/3841406757 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc00f6b0 con 0x7fcb0c100a80 2024-09-16T00:19:46.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb125e6640 1 -- 172.21.15.33:0/3841406757 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0c1a4a40 con 0x7fcb0c100a80 2024-09-16T00:19:46.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.584+0000 7fcb125e6640 1 -- 172.21.15.33:0/3841406757 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb0c1a4f50 con 0x7fcb0c100a80 2024-09-16T00:19:46.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.585+0000 7fcb125e6640 1 -- 172.21.15.33:0/3841406757 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb0c100ee0 con 0x7fcb0c100a80 2024-09-16T00:19:46.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.586+0000 7fcb097fa640 1 -- 172.21.15.33:0/3841406757 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fcafc0109d0 con 0x7fcb0c100a80 2024-09-16T00:19:46.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.587+0000 7fcb097fa640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcaf80a32c0 0x7fcaf80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:46.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.587+0000 7fcb0b7fe640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcaf80a32c0 0x7fcaf80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:46.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.587+0000 7fcb097fa640 1 -- 172.21.15.33:0/3841406757 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fcafc0cf9f0 con 0x7fcb0c100a80 2024-09-16T00:19:46.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.588+0000 7fcb0b7fe640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcaf80a32c0 0x7fcaf80a5780 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4000c00 tx=0x7fcaf4005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:46.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.592+0000 7fcb097fa640 1 -- 172.21.15.33:0/3841406757 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcafc009de0 con 0x7fcb0c100a80 2024-09-16T00:19:46.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 4.30 MB/sec warmup 54 sec latency 2550.182 ms 2024-09-16T00:19:46.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.804+0000 7fcb125e6640 1 -- 172.21.15.33:0/3841406757 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb0c108ff0 con 0x7fcb0c100a80 2024-09-16T00:19:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.804+0000 7fcb097fa640 1 -- 172.21.15.33:0/3841406757 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11906 (secure 0 0 0) 0x7fcafc09c230 con 0x7fcb0c100a80 2024-09-16T00:19:46.807 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:19:46.808 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"btime":"2024-09-16T00:19:02:115556+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:19:01.460559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:19:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 28 2024-09-16T00:19:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.806+0000 7fcb125e6640 1 -- 172.21.15.33:0/3841406757 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcaf80a32c0 msgr2=0x7fcaf80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.806+0000 7fcb125e6640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcaf80a32c0 0x7fcaf80a5780 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4000c00 tx=0x7fcaf4005a20 comp rx=0 tx=0).stop 2024-09-16T00:19:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.806+0000 7fcb125e6640 1 -- 172.21.15.33:0/3841406757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb0c100a80 msgr2=0x7fcb0c1992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.806+0000 7fcb125e6640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb0c100a80 0x7fcb0c1992b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00d950 tx=0x7fcafc00de10 comp rx=0 tx=0).stop 2024-09-16T00:19:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.807+0000 7fcb125e6640 1 -- 172.21.15.33:0/3841406757 shutdown_connections 2024-09-16T00:19:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.807+0000 7fcb125e6640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcaf80a32c0 0x7fcaf80a5780 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.807+0000 7fcb125e6640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb0c106a80 0x7fcb0c1a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.807+0000 7fcb125e6640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c101420 0x7fcb0c1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.807+0000 7fcb125e6640 1 --2- 172.21.15.33:0/3841406757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb0c100a80 0x7fcb0c1992b0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.807+0000 7fcb125e6640 1 -- 172.21.15.33:0/3841406757 >> 172.21.15.33:0/3841406757 conn(0x7fcb0c0fc7a0 msgr2=0x7fcb0c0fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.807+0000 7fcb125e6640 1 -- 172.21.15.33:0/3841406757 shutdown_connections 2024-09-16T00:19:46.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:46.807+0000 7fcb125e6640 1 -- 172.21.15.33:0/3841406757 wait complete. 2024-09-16T00:19:47.080 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5dd683', quiesce took 5.1 seconds. Will release after: 14.450000000000001 2024-09-16T00:19:47.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 4.22 MB/sec warmup 55 sec latency 1831.462 ms 2024-09-16T00:19:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:47 smithi033 ceph-mon[31243]: pgmap v429: 129 pgs: 129 active+clean; 35 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 729 KiB/s wr, 61 op/s 2024-09-16T00:19:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:47 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3841406757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:19:47.882 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:19:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:47 smithi083 ceph-mon[35242]: pgmap v429: 129 pgs: 129 active+clean; 35 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 729 KiB/s wr, 61 op/s 2024-09-16T00:19:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:47 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3841406757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:19:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:47 smithi060 ceph-mon[35185]: pgmap v429: 129 pgs: 129 active+clean; 35 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 729 KiB/s wr, 61 op/s 2024-09-16T00:19:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:47 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3841406757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:19:48.145 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:48.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 4.14 MB/sec warmup 56 sec latency 2831.503 ms 2024-09-16T00:19:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.635+0000 7f63b561c640 1 -- 172.21.15.33:0/3832660038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63b010c170 msgr2=0x7f63b010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.635+0000 7f63b561c640 1 --2- 172.21.15.33:0/3832660038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63b010c170 0x7f63b010e610 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009ca0 tx=0x7f63a802a950 comp rx=0 tx=0).stop 2024-09-16T00:19:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.635+0000 7f63b561c640 1 -- 172.21.15.33:0/3832660038 shutdown_connections 2024-09-16T00:19:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.635+0000 7f63b561c640 1 --2- 172.21.15.33:0/3832660038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63b010c170 0x7f63b010e610 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.635+0000 7f63b561c640 1 --2- 172.21.15.33:0/3832660038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63b0103660 0x7f63b010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.635+0000 7f63b561c640 1 --2- 172.21.15.33:0/3832660038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63b0102d40 0x7f63b0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.635+0000 7f63b561c640 1 -- 172.21.15.33:0/3832660038 >> 172.21.15.33:0/3832660038 conn(0x7f63b00fc7a0 msgr2=0x7f63b00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.635+0000 7f63b561c640 1 -- 172.21.15.33:0/3832660038 shutdown_connections 2024-09-16T00:19:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.635+0000 7f63b561c640 1 -- 172.21.15.33:0/3832660038 wait complete. 2024-09-16T00:19:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.635+0000 7f63b561c640 1 Processor -- start 2024-09-16T00:19:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63b561c640 1 -- start start 2024-09-16T00:19:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63b561c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63b0102d40 0x7f63b01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63b561c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63b0103660 0x7f63b0198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63b561c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63b010c170 0x7f63b019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63b561c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f63b0111f30 con 0x7f63b010c170 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63b561c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f63b0111db0 con 0x7f63b0103660 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63b561c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01120b0 con 0x7f63b0102d40 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63af7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63b010c170 0x7f63b019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63af7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63b010c170 0x7f63b019cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35842/0 (socket says 172.21.15.33:35842) 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63af7fe640 1 -- 172.21.15.33:0/1835233095 learned_addr learned my addr 172.21.15.33:0/1835233095 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63aeffd640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63b0102d40 0x7f63b01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63b0103660 0x7f63b0198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63af7fe640 1 -- 172.21.15.33:0/1835233095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63b0102d40 msgr2=0x7f63b01983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63af7fe640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63b0102d40 0x7f63b01983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63af7fe640 1 -- 172.21.15.33:0/1835233095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63b0103660 msgr2=0x7f63b0198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:48.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63af7fe640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63b0103660 0x7f63b0198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.636+0000 7f63af7fe640 1 -- 172.21.15.33:0/1835233095 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b019d380 con 0x7f63b010c170 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.637+0000 7f63af7fe640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63b010c170 0x7f63b019cca0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f63a802ae70 tx=0x7f63a802f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.637+0000 7f6397fff640 1 -- 172.21.15.33:0/1835233095 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a8034040 con 0x7f63b010c170 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.637+0000 7f6397fff640 1 -- 172.21.15.33:0/1835233095 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a8030920 con 0x7f63b010c170 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.637+0000 7f63b561c640 1 -- 172.21.15.33:0/1835233095 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63b019d610 con 0x7f63b010c170 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.637+0000 7f6397fff640 1 -- 172.21.15.33:0/1835233095 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a80047b0 con 0x7f63b010c170 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.637+0000 7f63b561c640 1 -- 172.21.15.33:0/1835233095 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63b01a4f50 con 0x7f63b010c170 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.639+0000 7f6397fff640 1 -- 172.21.15.33:0/1835233095 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f63a8037020 con 0x7f63b010c170 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.639+0000 7f63b79f5740 1 Processor -- start 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.639+0000 7f63b79f5740 1 -- start start 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63b79f5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45dfe090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63b79f5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45dfea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63b79f5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 0x561a45e0fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63b79f5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561a45d642f0 con 0x561a45e0d780 2024-09-16T00:19:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f6397fff640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f63980a3240 0x7f63980a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63b79f5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561a45d0bcf0 con 0x561a45dffb40 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63b79f5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561a45d602b0 con 0x561a45dfe5d0 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63af7fe640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f63980a3240 0x7f63980a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63aeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45dfea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63aeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45dfea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41124/0 (socket says 172.21.15.33:41124) 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63ae7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45dfe090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63aeffd640 1 -- 172.21.15.33:0/857799589 learned_addr learned my addr 172.21.15.33:0/857799589 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/857799589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 0x561a45e0fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f6397fff640 1 -- 172.21.15.33:0/1835233095 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f63a80f0bd0 con 0x7f63b010c170 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63aeffd640 1 -- 172.21.15.33:0/857799589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 msgr2=0x561a45dfe090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63aeffd640 1 --2- 172.21.15.33:0/857799589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45dfe090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63aeffd640 1 -- 172.21.15.33:0/857799589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 msgr2=0x561a45e0fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:48.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63aeffd640 1 --2- 172.21.15.33:0/857799589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 0x561a45e0fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63aeffd640 1 -- 172.21.15.33:0/857799589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a45cf03a0 con 0x561a45dfe5d0 2024-09-16T00:19:48.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/857799589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 0x561a45e0fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:19:48.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.640+0000 7f63af7fe640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f63980a3240 0x7f63980a5700 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f63a80bcca0 tx=0x7f63a80e7ae0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:48.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.641+0000 7f63aeffd640 1 --2- 172.21.15.33:0/857799589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45dfea10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f63a000ad00 tx=0x7f63a000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c12a8ecc9d68a5e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:48.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.641+0000 7f63957fa640 1 -- 172.21.15.33:0/857799589 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a001d070 con 0x561a45dfe5d0 2024-09-16T00:19:48.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.641+0000 7f63957fa640 1 -- 172.21.15.33:0/857799589 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a0018b20 con 0x561a45dfe5d0 2024-09-16T00:19:48.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.641+0000 7f63957fa640 1 -- 172.21.15.33:0/857799589 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0010b80 con 0x561a45dfe5d0 2024-09-16T00:19:48.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 -- 172.21.15.33:0/857799589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 msgr2=0x561a45dfea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:48.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 --2- 172.21.15.33:0/857799589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45dfea10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f63a000ad00 tx=0x7f63a000f790 comp rx=0 tx=0).stop 2024-09-16T00:19:48.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 -- 172.21.15.33:0/857799589 shutdown_connections 2024-09-16T00:19:48.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 --2- 172.21.15.33:0/857799589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 0x561a45e0fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 --2- 172.21.15.33:0/857799589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45dfea10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 --2- 172.21.15.33:0/857799589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45dfe090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 -- 172.21.15.33:0/857799589 >> 172.21.15.33:0/857799589 conn(0x561a45dc7310 msgr2=0x561a45dfac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 -- 172.21.15.33:0/857799589 shutdown_connections 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 -- 172.21.15.33:0/857799589 wait complete. 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 -- start start 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 Processor -- start 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45e0d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45e0a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 0x561a45e0aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561a45e10480 con 0x561a45e0d780 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561a45d1f060 con 0x561a45dffb40 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561a45e10600 con 0x561a45dfe5d0 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63b79f5740 10 client.0 _ll_register_callbacks cb 0x561a45a728c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:48.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63ae7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 0x561a45e0aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63ae7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 0x561a45e0aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35852/0 (socket says 172.21.15.33:35852) 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.642+0000 7f63ae7fc640 1 -- 172.21.15.33:0/2296748284 learned_addr learned my addr 172.21.15.33:0/2296748284 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45e0d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f63aeffd640 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45e0a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f63ae7fc640 1 -- 172.21.15.33:0/2296748284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 msgr2=0x561a45e0d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45e0d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f63ae7fc640 1 -- 172.21.15.33:0/2296748284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 msgr2=0x561a45e0a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45e0a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f63ae7fc640 1 -- 172.21.15.33:0/2296748284 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a45ad53c0 con 0x561a45e0d780 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 0x561a45e0aa50 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f63a4013cc0 tx=0x7f63a400bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd708afbf9cad3d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f6392ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f6392ff5640 1 -- 172.21.15.33:0/2296748284 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4004230 con 0x561a45e0d780 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f6392ff5640 1 -- 172.21.15.33:0/2296748284 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f63a40043d0 con 0x561a45e0d780 2024-09-16T00:19:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f63b79f5740 1 -- 172.21.15.33:0/2296748284 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63a0018b20 con 0x561a45e0d780 2024-09-16T00:19:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f6392ff5640 1 -- 172.21.15.33:0/2296748284 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f63a4010e70 con 0x561a45e0d780 2024-09-16T00:19:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f63b79f5740 10 client.15783 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f63b79f5740 1 -- 172.21.15.33:0/2296748284 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561a45e1bfb0 con 0x561a45e0d780 2024-09-16T00:19:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.643+0000 7f6392ff5640 1 -- 172.21.15.33:0/2296748284 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a0018b20 con 0x561a45e0d780 2024-09-16T00:19:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.644+0000 7f6392ff5640 1 -- 172.21.15.33:0/2296748284 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f63a40047a0 con 0x561a45e0d780 2024-09-16T00:19:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.644+0000 7f63b79f5740 10 client.15783 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.644+0000 7f63b79f5740 10 client.15783 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:19:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.644+0000 7f63b79f5740 20 client.15783 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.644+0000 7f63b79f5740 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561a45e1d470 0x561a45e1f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.644+0000 7f63b79f5740 4 client.15783 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.644+0000 7f63b79f5740 1 -- 172.21.15.33:0/2296748284 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a45d71860 con 0x561a45e1d470 2024-09-16T00:19:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.644+0000 7f63aeffd640 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561a45e1d470 0x561a45e1f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.644+0000 7f6392ff5640 10 client.15783 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.644+0000 7f63aeffd640 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561a45e1d470 0x561a45e1f880 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e5d2db70d541bde5 server_cookie=a393945dcd977081 in_seq=0 out_seq=0 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f6392ff5640 1 -- 172.21.15.33:0/2296748284 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561a45d71860 con 0x561a45e1d470 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f6392ff5640 10 client.15783 handle_command_reply: tid=0 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f63b79f5740 1 client.15783 shutdown 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f63b79f5740 1 -- 172.21.15.33:0/2296748284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 msgr2=0x561a45e0aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f63b79f5740 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 0x561a45e0aa50 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f63a4013cc0 tx=0x7f63a400bac0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f63b79f5740 1 -- 172.21.15.33:0/2296748284 shutdown_connections 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f63b79f5740 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561a45e1d470 0x561a45e1f880 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f63b79f5740 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45e0d780 0x561a45e0aa50 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f63b79f5740 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45e0a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f6392ff5640 0 client.15783 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:19:48.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f63b79f5740 1 --2- 172.21.15.33:0/2296748284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45e0d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f63b79f5740 1 -- 172.21.15.33:0/2296748284 >> 172.21.15.33:0/2296748284 conn(0x561a45dc7310 msgr2=0x561a45e00960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f63b79f5740 1 -- 172.21.15.33:0/2296748284 shutdown_connections 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.645+0000 7f63b79f5740 1 -- 172.21.15.33:0/2296748284 wait complete. 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.646+0000 7f63b79f5740 20 client.15783 trim_cache size 0 max 16384 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.658+0000 7f63b79f5740 1 Processor -- start 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.658+0000 7f63b79f5740 1 -- start start 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63b79f5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dfe5d0 0x561a45dfae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63b79f5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45e19720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63b79f5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45e0d780 0x561a45e25150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63b79f5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01af6e0 con 0x561a45dfe5d0 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63b79f5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01acfb0 con 0x561a45dffb40 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63b79f5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01aae00 con 0x561a45e0d780 2024-09-16T00:19:48.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63ae7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dfe5d0 0x561a45dfae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63ae7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dfe5d0 0x561a45dfae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35860/0 (socket says 172.21.15.33:35860) 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63ae7fc640 1 -- 172.21.15.33:0/3910677537 learned_addr learned my addr 172.21.15.33:0/3910677537 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63aeffd640 1 --2- 172.21.15.33:0/3910677537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45e19720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63ae7fc640 1 -- 172.21.15.33:0/3910677537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45e0d780 msgr2=0x561a45e25150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/3910677537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45e0d780 0x561a45e25150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63ae7fc640 1 -- 172.21.15.33:0/3910677537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 msgr2=0x561a45e19720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/3910677537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45e19720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63ae7fc640 1 -- 172.21.15.33:0/3910677537 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a45e01120 con 0x561a45dfe5d0 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63aeffd640 1 --2- 172.21.15.33:0/3910677537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45e19720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/3910677537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dfe5d0 0x561a45dfae10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f63a4013c90 tx=0x7f63a40058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16646a265b58bb40 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f6394ff9640 1 -- 172.21.15.33:0/3910677537 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a40130d0 con 0x561a45dfe5d0 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f6394ff9640 1 -- 172.21.15.33:0/3910677537 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a4003ad0 con 0x561a45dfe5d0 2024-09-16T00:19:48.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.659+0000 7f6394ff9640 1 -- 172.21.15.33:0/3910677537 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4005410 con 0x561a45dfe5d0 2024-09-16T00:19:48.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.660+0000 7f63b79f5740 1 -- 172.21.15.33:0/3910677537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dfe5d0 msgr2=0x561a45dfae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:48.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.660+0000 7f63b79f5740 1 --2- 172.21.15.33:0/3910677537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dfe5d0 0x561a45dfae10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f63a4013c90 tx=0x7f63a40058f0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.660+0000 7f63b79f5740 1 -- 172.21.15.33:0/3910677537 shutdown_connections 2024-09-16T00:19:48.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.660+0000 7f63b79f5740 1 --2- 172.21.15.33:0/3910677537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45e0d780 0x561a45e25150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.660+0000 7f63b79f5740 1 --2- 172.21.15.33:0/3910677537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45dffb40 0x561a45e19720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.660+0000 7f63b79f5740 1 --2- 172.21.15.33:0/3910677537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dfe5d0 0x561a45dfae10 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.660+0000 7f63b79f5740 1 -- 172.21.15.33:0/3910677537 >> 172.21.15.33:0/3910677537 conn(0x561a45dc7310 msgr2=0x561a45dfa700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:48.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.660+0000 7f63b79f5740 1 -- 172.21.15.33:0/3910677537 shutdown_connections 2024-09-16T00:19:48.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.660+0000 7f63b79f5740 1 -- 172.21.15.33:0/3910677537 wait complete. 2024-09-16T00:19:48.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63b79f5740 1 -- start start 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63b79f5740 1 Processor -- start 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63b79f5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45e11520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63b79f5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dffb40 0x561a45e11a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63b79f5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45e0d780 0x561a45e11fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63b79f5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01ab470 con 0x561a45dffb40 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63b79f5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01ac8e0 con 0x561a45e0d780 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63b79f5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01abb40 con 0x561a45dfe5d0 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63b79f5740 10 client.0 _ll_register_callbacks cb 0x7f63a400bc30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63aeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dffb40 0x561a45e11a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63aeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dffb40 0x561a45e11a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35870/0 (socket says 172.21.15.33:35870) 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63aeffd640 1 -- 172.21.15.33:0/121390207 learned_addr learned my addr 172.21.15.33:0/121390207 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.661+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45e11520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45e0d780 0x561a45e11fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f63aeffd640 1 -- 172.21.15.33:0/121390207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 msgr2=0x561a45e11520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f63aeffd640 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45e11520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f63aeffd640 1 -- 172.21.15.33:0/121390207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45e0d780 msgr2=0x561a45e11fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f63aeffd640 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45e0d780 0x561a45e11fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f63aeffd640 1 -- 172.21.15.33:0/121390207 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a45e87120 con 0x561a45dffb40 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45e0d780 0x561a45e11fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f63aeffd640 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dffb40 0x561a45e11a60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f63a000ad00 tx=0x7f63a0005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab45ee604b37d87 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f6392ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f6392ff5640 1 -- 172.21.15.33:0/121390207 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a001d040 con 0x561a45dffb40 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f6392ff5640 1 -- 172.21.15.33:0/121390207 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f63a0018540 con 0x561a45dffb40 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f63b79f5740 1 -- 172.21.15.33:0/121390207 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63a4003ad0 con 0x561a45dffb40 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f6392ff5640 1 -- 172.21.15.33:0/121390207 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f63a0039050 con 0x561a45dffb40 2024-09-16T00:19:48.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f63b79f5740 10 client.15789 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f63b79f5740 1 -- 172.21.15.33:0/121390207 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561a45e88340 con 0x561a45dffb40 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.662+0000 7f6392ff5640 1 -- 172.21.15.33:0/121390207 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a4003ad0 con 0x561a45dffb40 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.663+0000 7f6392ff5640 1 -- 172.21.15.33:0/121390207 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f63a0010660 con 0x561a45dffb40 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.663+0000 7f63b79f5740 10 client.15789 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.663+0000 7f63b79f5740 10 client.15789 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.663+0000 7f63b79f5740 20 client.15789 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.663+0000 7f63b79f5740 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561a45e88960 0x561a45e8ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.663+0000 7f63b79f5740 4 client.15789 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.663+0000 7f63b79f5740 1 -- 172.21.15.33:0/121390207 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561a45d6bd70 con 0x561a45e88960 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.663+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561a45e88960 0x561a45e8ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63ae7fc640 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561a45e88960 0x561a45e8ad80 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=53ca07889a4535b server_cookie=ad1631cbe3e887b4 in_seq=0 out_seq=0 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f6392ff5640 10 client.15789 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:19:48.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f6392ff5640 1 -- 172.21.15.33:0/121390207 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561a45d6bd70 con 0x561a45e88960 2024-09-16T00:19:48.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f6392ff5640 10 client.15789 handle_command_reply: tid=0 2024-09-16T00:19:48.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63b79f5740 1 client.15789 shutdown 2024-09-16T00:19:48.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63b79f5740 1 -- 172.21.15.33:0/121390207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dffb40 msgr2=0x561a45e11a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:48.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63b79f5740 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dffb40 0x561a45e11a60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f63a000ad00 tx=0x7f63a0005e60 comp rx=0 tx=0).stop 2024-09-16T00:19:48.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63b79f5740 1 -- 172.21.15.33:0/121390207 shutdown_connections 2024-09-16T00:19:48.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63b79f5740 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561a45e88960 0x561a45e8ad80 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63b79f5740 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561a45e0d780 0x561a45e11fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63b79f5740 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a45dffb40 0x561a45e11a60 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63b79f5740 1 --2- 172.21.15.33:0/121390207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561a45dfe5d0 0x561a45e11520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63b79f5740 1 -- 172.21.15.33:0/121390207 >> 172.21.15.33:0/121390207 conn(0x561a45dc7310 msgr2=0x561a45e00930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:48.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f6392ff5640 0 client.15789 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:19:48.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63b79f5740 1 -- 172.21.15.33:0/121390207 shutdown_connections 2024-09-16T00:19:48.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63b79f5740 1 -- 172.21.15.33:0/121390207 wait complete. 2024-09-16T00:19:48.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.664+0000 7f63b79f5740 20 client.15789 trim_cache size 0 max 16384 2024-09-16T00:19:48.682 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:19:48.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.665+0000 7f63b561c640 1 -- 172.21.15.33:0/1835233095 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f63980a3240 msgr2=0x7f63980a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:48.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.665+0000 7f63b561c640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f63980a3240 0x7f63980a5700 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f63a80bcca0 tx=0x7f63a80e7ae0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.665+0000 7f63b561c640 1 -- 172.21.15.33:0/1835233095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63b010c170 msgr2=0x7f63b019cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:48.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.665+0000 7f63b561c640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63b010c170 0x7f63b019cca0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f63a802ae70 tx=0x7f63a802f5b0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.665+0000 7f63b561c640 1 -- 172.21.15.33:0/1835233095 shutdown_connections 2024-09-16T00:19:48.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.665+0000 7f63b561c640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f63980a3240 0x7f63980a5700 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.665+0000 7f63b561c640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63b010c170 0x7f63b019cca0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.665+0000 7f63b561c640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63b0103660 0x7f63b0198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.665+0000 7f63b561c640 1 --2- 172.21.15.33:0/1835233095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63b0102d40 0x7f63b01983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:48.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.665+0000 7f63b561c640 1 -- 172.21.15.33:0/1835233095 >> 172.21.15.33:0/1835233095 conn(0x7f63b00fc7a0 msgr2=0x7f63b00fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:48.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.665+0000 7f63b561c640 1 -- 172.21.15.33:0/1835233095 shutdown_connections 2024-09-16T00:19:48.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:48.665+0000 7f63b561c640 1 -- 172.21.15.33:0/1835233095 wait complete. 2024-09-16T00:19:49.446 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:19:49.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:49 smithi033 ceph-mon[31243]: pgmap v430: 129 pgs: 129 active+clean; 35 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 568 KiB/s wr, 46 op/s 2024-09-16T00:19:49.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 4.07 MB/sec warmup 57 sec latency 3831.549 ms 2024-09-16T00:19:49.707 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:49.782 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:49 smithi060 ceph-mon[35185]: pgmap v430: 129 pgs: 129 active+clean; 35 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 568 KiB/s wr, 46 op/s 2024-09-16T00:19:49.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:49 smithi083 ceph-mon[35242]: pgmap v430: 129 pgs: 129 active+clean; 35 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 568 KiB/s wr, 46 op/s 2024-09-16T00:19:50.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.220+0000 7f8e8b997640 1 -- 172.21.15.33:0/1234399614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e84101280 msgr2=0x7f8e8410df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:50.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.220+0000 7f8e8b997640 1 --2- 172.21.15.33:0/1234399614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e84101280 0x7f8e8410df40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8e74009900 tx=0x7f8e7402a410 comp rx=0 tx=0).stop 2024-09-16T00:19:50.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.220+0000 7f8e8b997640 1 -- 172.21.15.33:0/1234399614 shutdown_connections 2024-09-16T00:19:50.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.220+0000 7f8e8b997640 1 --2- 172.21.15.33:0/1234399614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8410e480 0x7f8e84110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.220+0000 7f8e8b997640 1 --2- 172.21.15.33:0/1234399614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e84101280 0x7f8e8410df40 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.220+0000 7f8e8b997640 1 --2- 172.21.15.33:0/1234399614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e841008d0 0x7f8e84100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.220+0000 7f8e8b997640 1 -- 172.21.15.33:0/1234399614 >> 172.21.15.33:0/1234399614 conn(0x7f8e840fc7a0 msgr2=0x7f8e840febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:50.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.220+0000 7f8e8b997640 1 -- 172.21.15.33:0/1234399614 shutdown_connections 2024-09-16T00:19:50.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.220+0000 7f8e8b997640 1 -- 172.21.15.33:0/1234399614 wait complete. 2024-09-16T00:19:50.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.220+0000 7f8e8b997640 1 Processor -- start 2024-09-16T00:19:50.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.220+0000 7f8e8b997640 1 -- start start 2024-09-16T00:19:50.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e8b997640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841008d0 0x7f8e8419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e8b997640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e84101280 0x7f8e8419ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e8b997640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8410e480 0x7f8e8419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e8b997640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841141b0 con 0x7f8e841008d0 2024-09-16T00:19:50.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e8b997640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8e84114030 con 0x7f8e8410e480 2024-09-16T00:19:50.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e8b997640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8e84114330 con 0x7f8e84101280 2024-09-16T00:19:50.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e89733640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841008d0 0x7f8e8419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e89733640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841008d0 0x7f8e8419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35890/0 (socket says 172.21.15.33:35890) 2024-09-16T00:19:50.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e89733640 1 -- 172.21.15.33:0/2788055099 learned_addr learned my addr 172.21.15.33:0/2788055099 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e89f34640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8410e480 0x7f8e8419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e88f32640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e84101280 0x7f8e8419ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e89733640 1 -- 172.21.15.33:0/2788055099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e84101280 msgr2=0x7f8e8419ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e89733640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e84101280 0x7f8e8419ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e89733640 1 -- 172.21.15.33:0/2788055099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8410e480 msgr2=0x7f8e8419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e89733640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8410e480 0x7f8e8419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e89733640 1 -- 172.21.15.33:0/2788055099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8419f700 con 0x7f8e841008d0 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e89f34640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8410e480 0x7f8e8419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e89733640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841008d0 0x7f8e8419a750 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000d770 tx=0x7f8e8000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e7a7fc640 1 -- 172.21.15.33:0/2788055099 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80017070 con 0x7f8e841008d0 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e7a7fc640 1 -- 172.21.15.33:0/2788055099 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e80005500 con 0x7f8e841008d0 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e8b997640 1 -- 172.21.15.33:0/2788055099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8419f9f0 con 0x7f8e841008d0 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e7a7fc640 1 -- 172.21.15.33:0/2788055099 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80004e00 con 0x7f8e841008d0 2024-09-16T00:19:50.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.221+0000 7f8e8b997640 1 -- 172.21.15.33:0/2788055099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e841a72d0 con 0x7f8e841008d0 2024-09-16T00:19:50.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e7a7fc640 1 -- 172.21.15.33:0/2788055099 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8e80004fa0 con 0x7f8e841008d0 2024-09-16T00:19:50.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e8dd70740 1 Processor -- start 2024-09-16T00:19:50.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e8dd70740 1 -- start start 2024-09-16T00:19:50.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e8dd70740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2db40 0x55afe0b2c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e8dd70740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b2c5d0 0x55afe0b2ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e8dd70740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b37370 0x55afe0b39760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e8dd70740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55afe0a922f0 con 0x55afe0b37370 2024-09-16T00:19:50.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e8dd70740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55afe0a39cf0 con 0x55afe0b2db40 2024-09-16T00:19:50.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e8dd70740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55afe0a8e2b0 con 0x55afe0b2c5d0 2024-09-16T00:19:50.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e88f32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b37370 0x55afe0b39760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e89f34640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2db40 0x55afe0b2c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e88f32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b37370 0x55afe0b39760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35902/0 (socket says 172.21.15.33:35902) 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e89f34640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2db40 0x55afe0b2c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:44228/0 (socket says 172.21.15.33:44228) 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e88f32640 1 -- 172.21.15.33:0/3694516732 learned_addr learned my addr 172.21.15.33:0/3694516732 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e89733640 1 --2- 172.21.15.33:0/3694516732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b2c5d0 0x55afe0b2ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e88f32640 1 -- 172.21.15.33:0/3694516732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b2c5d0 msgr2=0x55afe0b2ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e88f32640 1 --2- 172.21.15.33:0/3694516732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b2c5d0 0x55afe0b2ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e88f32640 1 -- 172.21.15.33:0/3694516732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2db40 msgr2=0x55afe0b2c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.225+0000 7f8e88f32640 1 --2- 172.21.15.33:0/3694516732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2db40 0x55afe0b2c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.226+0000 7f8e88f32640 1 -- 172.21.15.33:0/3694516732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afe0a1e3a0 con 0x55afe0b37370 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.226+0000 7f8e7a7fc640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e700a32c0 0x7f8e700a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.226+0000 7f8e89f34640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e700a32c0 0x7f8e700a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.226+0000 7f8e88f32640 1 --2- 172.21.15.33:0/3694516732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b37370 0x55afe0b39760 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8e7402a930 tx=0x7f8e74005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229dfc3559baad20 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.226+0000 7f8e7a7fc640 1 -- 172.21.15.33:0/2788055099 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8e800cf810 con 0x7f8e841008d0 2024-09-16T00:19:50.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.226+0000 7f8e6f7fe640 1 -- 172.21.15.33:0/3694516732 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7403f070 con 0x55afe0b37370 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.226+0000 7f8e6f7fe640 1 -- 172.21.15.33:0/3694516732 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e74004030 con 0x55afe0b37370 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.226+0000 7f8e6f7fe640 1 -- 172.21.15.33:0/3694516732 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74035070 con 0x55afe0b37370 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.227+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3694516732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b37370 msgr2=0x55afe0b39760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.227+0000 7f8e89f34640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e700a32c0 0x7f8e700a5780 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8e8406a0f0 tx=0x7f8e7c009820 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.227+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/3694516732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b37370 0x55afe0b39760 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8e7402a930 tx=0x7f8e74005e80 comp rx=0 tx=0).stop 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.227+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3694516732 shutdown_connections 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.227+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/3694516732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b37370 0x55afe0b39760 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.227+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/3694516732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b2c5d0 0x55afe0b2ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.227+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/3694516732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2db40 0x55afe0b2c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.227+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3694516732 >> 172.21.15.33:0/3694516732 conn(0x55afe0af5310 msgr2=0x55afe0b28c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.227+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3694516732 shutdown_connections 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.227+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3694516732 wait complete. 2024-09-16T00:19:50.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.227+0000 7f8e8dd70740 1 -- start start 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.227+0000 7f8e8dd70740 1 Processor -- start 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e8dd70740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2c5d0 0x55afe0b36d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e8dd70740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 0x55afe0b34010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e8dd70740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b34600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e8dd70740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55afe0b3a010 con 0x55afe0b2db40 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e8dd70740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55afe0a4d060 con 0x55afe0b2c5d0 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e8dd70740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55afe0b3a190 con 0x55afe0b37370 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e8dd70740 10 client.0 _ll_register_callbacks cb 0x55afe07a08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e88f32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 0x55afe0b34010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e88f32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 0x55afe0b34010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35912/0 (socket says 172.21.15.33:35912) 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e88f32640 1 -- 172.21.15.33:0/2323036153 learned_addr learned my addr 172.21.15.33:0/2323036153 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e89733640 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2c5d0 0x55afe0b36d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e89f34640 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b34600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e88f32640 1 -- 172.21.15.33:0/2323036153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 msgr2=0x55afe0b34600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e88f32640 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b34600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e88f32640 1 -- 172.21.15.33:0/2323036153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2c5d0 msgr2=0x55afe0b36d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e88f32640 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2c5d0 0x55afe0b36d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e88f32640 1 -- 172.21.15.33:0/2323036153 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afe08033c0 con 0x55afe0b2db40 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e88f32640 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 0x55afe0b34010 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8e74006010 tx=0x7f8e740095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6c3a5fed28c314 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.228+0000 7f8e6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e6cff9640 1 -- 172.21.15.33:0/2323036153 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74035070 con 0x55afe0b2db40 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e6cff9640 1 -- 172.21.15.33:0/2323036153 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8e740317b0 con 0x55afe0b2db40 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2323036153 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e74004030 con 0x55afe0b2db40 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e6cff9640 1 -- 172.21.15.33:0/2323036153 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e7403f040 con 0x55afe0b2db40 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e8dd70740 10 client.15801 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2323036153 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55afe0b45ab0 con 0x55afe0b2db40 2024-09-16T00:19:50.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e6cff9640 1 -- 172.21.15.33:0/2323036153 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e74004030 con 0x55afe0b2db40 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e6cff9640 1 -- 172.21.15.33:0/2323036153 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8e74004750 con 0x55afe0b2db40 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e8dd70740 10 client.15801 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e8dd70740 10 client.15801 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e8dd70740 20 client.15801 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55afe0b46e00 0x55afe0b491f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e8dd70740 4 client.15801 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2323036153 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afe0a9f860 con 0x55afe0b46e00 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.229+0000 7f8e89733640 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55afe0b46e00 0x55afe0b491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.230+0000 7f8e6cff9640 10 client.15801 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.230+0000 7f8e89733640 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55afe0b46e00 0x55afe0b491f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=acab6a3fa0edf329 server_cookie=432425a667b81f73 in_seq=0 out_seq=0 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e6cff9640 1 -- 172.21.15.33:0/2323036153 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55afe0a9f860 con 0x55afe0b46e00 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e6cff9640 10 client.15801 handle_command_reply: tid=0 2024-09-16T00:19:50.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e8dd70740 1 client.15801 shutdown 2024-09-16T00:19:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2323036153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 msgr2=0x55afe0b34010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 0x55afe0b34010 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8e74006010 tx=0x7f8e740095f0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2323036153 shutdown_connections 2024-09-16T00:19:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b34600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 0x55afe0b34010 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55afe0b46e00 0x55afe0b491f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/2323036153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2c5d0 0x55afe0b36d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e6cff9640 0 client.15801 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:19:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2323036153 >> 172.21.15.33:0/2323036153 conn(0x55afe0af5310 msgr2=0x55afe0b2e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2323036153 shutdown_connections 2024-09-16T00:19:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2323036153 wait complete. 2024-09-16T00:19:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.231+0000 7f8e8dd70740 20 client.15801 trim_cache size 0 max 16384 2024-09-16T00:19:50.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e8dd70740 1 Processor -- start 2024-09-16T00:19:50.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e8dd70740 1 -- start start 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e8dd70740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2c5d0 0x55afe0b432b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e8dd70740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 0x55afe0b2efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e8dd70740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b2f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e8dd70740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841b3b60 con 0x55afe0b2db40 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e8dd70740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841b1430 con 0x55afe0b2c5d0 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e8dd70740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841af280 con 0x55afe0b37370 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e88f32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 0x55afe0b2efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e88f32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 0x55afe0b2efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35920/0 (socket says 172.21.15.33:35920) 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e88f32640 1 -- 172.21.15.33:0/2844650422 learned_addr learned my addr 172.21.15.33:0/2844650422 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e89733640 1 --2- 172.21.15.33:0/2844650422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2c5d0 0x55afe0b432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.244+0000 7f8e89f34640 1 --2- 172.21.15.33:0/2844650422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b2f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.245+0000 7f8e88f32640 1 -- 172.21.15.33:0/2844650422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 msgr2=0x55afe0b2f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:50.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.245+0000 7f8e88f32640 1 --2- 172.21.15.33:0/2844650422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b2f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.245+0000 7f8e88f32640 1 -- 172.21.15.33:0/2844650422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2c5d0 msgr2=0x55afe0b432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.245+0000 7f8e88f32640 1 --2- 172.21.15.33:0/2844650422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2c5d0 0x55afe0b432b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.245+0000 7f8e88f32640 1 -- 172.21.15.33:0/2844650422 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afe0b32560 con 0x55afe0b2db40 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.245+0000 7f8e89733640 1 --2- 172.21.15.33:0/2844650422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2c5d0 0x55afe0b432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.245+0000 7f8e88f32640 1 --2- 172.21.15.33:0/2844650422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 0x55afe0b2efc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8e74004bf0 tx=0x7f8e74005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6353a4c683ac7b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.245+0000 7f8e6effd640 1 -- 172.21.15.33:0/2844650422 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74035070 con 0x55afe0b2db40 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.245+0000 7f8e6effd640 1 -- 172.21.15.33:0/2844650422 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e740048c0 con 0x55afe0b2db40 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.245+0000 7f8e6effd640 1 -- 172.21.15.33:0/2844650422 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74032070 con 0x55afe0b2db40 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2844650422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 msgr2=0x55afe0b2efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/2844650422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 0x55afe0b2efc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8e74004bf0 tx=0x7f8e74005e80 comp rx=0 tx=0).stop 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2844650422 shutdown_connections 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/2844650422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b2f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/2844650422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2db40 0x55afe0b2efc0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/2844650422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2c5d0 0x55afe0b432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2844650422 >> 172.21.15.33:0/2844650422 conn(0x55afe0af5310 msgr2=0x55afe0b33100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:50.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2844650422 shutdown_connections 2024-09-16T00:19:50.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 -- 172.21.15.33:0/2844650422 wait complete. 2024-09-16T00:19:50.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 -- start start 2024-09-16T00:19:50.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 Processor -- start 2024-09-16T00:19:50.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2c5d0 0x55afe0b3fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.246+0000 7f8e8dd70740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2db40 0x55afe0b30c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e8dd70740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b31170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e8dd70740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841af8f0 con 0x55afe0b2c5d0 2024-09-16T00:19:50.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e8dd70740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841b0d60 con 0x55afe0b2db40 2024-09-16T00:19:50.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e8dd70740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841affc0 con 0x55afe0b37370 2024-09-16T00:19:50.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e8dd70740 10 client.0 _ll_register_callbacks cb 0x7f8e7c0096c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:50.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e89733640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2c5d0 0x55afe0b3fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e89733640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2c5d0 0x55afe0b3fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45902/0 (socket says 172.21.15.33:45902) 2024-09-16T00:19:50.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e89733640 1 -- 172.21.15.33:0/3345483357 learned_addr learned my addr 172.21.15.33:0/3345483357 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:50.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e88f32640 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2db40 0x55afe0b30c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e89f34640 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b31170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e89733640 1 -- 172.21.15.33:0/3345483357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 msgr2=0x55afe0b31170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:50.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e89733640 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b31170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e89733640 1 -- 172.21.15.33:0/3345483357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2db40 msgr2=0x55afe0b30c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:50.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e89733640 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2db40 0x55afe0b30c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e89733640 1 -- 172.21.15.33:0/3345483357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afe0b318b0 con 0x55afe0b2c5d0 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e89f34640 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b31170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e89733640 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2c5d0 0x55afe0b3fa60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8e800cce90 tx=0x7f8e80086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cc3e22dcd74c15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e6cff9640 1 -- 172.21.15.33:0/3345483357 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e800b4060 con 0x55afe0b2c5d0 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e6cff9640 1 -- 172.21.15.33:0/3345483357 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8e80005290 con 0x55afe0b2c5d0 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.247+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3345483357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e740048c0 con 0x55afe0b2c5d0 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.248+0000 7f8e6cff9640 1 -- 172.21.15.33:0/3345483357 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e80020040 con 0x55afe0b2c5d0 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.248+0000 7f8e8dd70740 10 client.15807 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.248+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3345483357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55afe0bb1790 con 0x55afe0b2c5d0 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.248+0000 7f8e6cff9640 1 -- 172.21.15.33:0/3345483357 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e740048c0 con 0x55afe0b2c5d0 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.248+0000 7f8e6cff9640 1 -- 172.21.15.33:0/3345483357 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8e8009cce0 con 0x55afe0b2c5d0 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.248+0000 7f8e8dd70740 10 client.15807 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:50.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.248+0000 7f8e8dd70740 10 client.15807 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:19:50.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.248+0000 7f8e8dd70740 20 client.15807 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:50.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.248+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55afe0bb1e90 0x55afe0bb42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:50.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.248+0000 7f8e8dd70740 4 client.15807 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:19:50.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.248+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3345483357 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55afe0a99d70 con 0x55afe0bb1e90 2024-09-16T00:19:50.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.248+0000 7f8e88f32640 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55afe0bb1e90 0x55afe0bb42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:50.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e88f32640 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55afe0bb1e90 0x55afe0bb42b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5890fcfecb610028 server_cookie=d0c43ff7c579cc52 in_seq=0 out_seq=0 2024-09-16T00:19:50.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e6cff9640 10 client.15807 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:19:50.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e6cff9640 1 -- 172.21.15.33:0/3345483357 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55afe0a99d70 con 0x55afe0bb1e90 2024-09-16T00:19:50.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e6cff9640 10 client.15807 handle_command_reply: tid=0 2024-09-16T00:19:50.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e8dd70740 1 client.15807 shutdown 2024-09-16T00:19:50.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3345483357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2c5d0 msgr2=0x55afe0b3fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:50.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2c5d0 0x55afe0b3fa60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8e800cce90 tx=0x7f8e80086040 comp rx=0 tx=0).stop 2024-09-16T00:19:50.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3345483357 shutdown_connections 2024-09-16T00:19:50.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55afe0b37370 0x55afe0b31170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55afe0bb1e90 0x55afe0bb42b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55afe0b2db40 0x55afe0b30c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e6cff9640 0 client.15807 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:19:50.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e8dd70740 1 --2- 172.21.15.33:0/3345483357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afe0b2c5d0 0x55afe0b3fa60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3345483357 >> 172.21.15.33:0/3345483357 conn(0x55afe0af5310 msgr2=0x55afe0b274e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:50.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3345483357 shutdown_connections 2024-09-16T00:19:50.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e8dd70740 1 -- 172.21.15.33:0/3345483357 wait complete. 2024-09-16T00:19:50.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.249+0000 7f8e8dd70740 20 client.15807 trim_cache size 0 max 16384 2024-09-16T00:19:50.259 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:19:50.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.250+0000 7f8e8b997640 1 -- 172.21.15.33:0/2788055099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e700a32c0 msgr2=0x7f8e700a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:50.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.250+0000 7f8e8b997640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e700a32c0 0x7f8e700a5780 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8e8406a0f0 tx=0x7f8e7c009820 comp rx=0 tx=0).stop 2024-09-16T00:19:50.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.250+0000 7f8e8b997640 1 -- 172.21.15.33:0/2788055099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841008d0 msgr2=0x7f8e8419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:50.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.250+0000 7f8e8b997640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841008d0 0x7f8e8419a750 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000d770 tx=0x7f8e8000dc30 comp rx=0 tx=0).stop 2024-09-16T00:19:50.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.250+0000 7f8e8b997640 1 -- 172.21.15.33:0/2788055099 shutdown_connections 2024-09-16T00:19:50.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.250+0000 7f8e8b997640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e700a32c0 0x7f8e700a5780 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.250+0000 7f8e8b997640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8410e480 0x7f8e8419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.250+0000 7f8e8b997640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e84101280 0x7f8e8419ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.250+0000 7f8e8b997640 1 --2- 172.21.15.33:0/2788055099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841008d0 0x7f8e8419a750 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:50.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.250+0000 7f8e8b997640 1 -- 172.21.15.33:0/2788055099 >> 172.21.15.33:0/2788055099 conn(0x7f8e840fc7a0 msgr2=0x7f8e840feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:50.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.250+0000 7f8e8b997640 1 -- 172.21.15.33:0/2788055099 shutdown_connections 2024-09-16T00:19:50.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:50.250+0000 7f8e8b997640 1 -- 172.21.15.33:0/2788055099 wait complete. 2024-09-16T00:19:50.647 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 4.00 MB/sec warmup 58 sec latency 4831.614 ms 2024-09-16T00:19:51.118 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:19:51.378 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:51.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.93 MB/sec warmup 59 sec latency 5831.702 ms 2024-09-16T00:19:51.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:51 smithi033 ceph-mon[31243]: pgmap v431: 129 pgs: 129 active+clean; 36 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 594 KiB/s wr, 52 op/s 2024-09-16T00:19:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:51 smithi083 ceph-mon[35242]: pgmap v431: 129 pgs: 129 active+clean; 36 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 594 KiB/s wr, 52 op/s 2024-09-16T00:19:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:51 smithi060 ceph-mon[35185]: pgmap v431: 129 pgs: 129 active+clean; 36 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 594 KiB/s wr, 52 op/s 2024-09-16T00:19:52.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.064+0000 7f9a9b04b640 1 -- 172.21.15.33:0/3817386324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a94102570 msgr2=0x7f9a941029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:52.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.064+0000 7f9a9b04b640 1 --2- 172.21.15.33:0/3817386324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a94102570 0x7f9a941029b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009900 tx=0x7f9a8002a480 comp rx=0 tx=0).stop 2024-09-16T00:19:52.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.064+0000 7f9a9b04b640 1 -- 172.21.15.33:0/3817386324 shutdown_connections 2024-09-16T00:19:52.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.064+0000 7f9a9b04b640 1 --2- 172.21.15.33:0/3817386324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a94102ef0 0x7f9a9410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.064+0000 7f9a9b04b640 1 --2- 172.21.15.33:0/3817386324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a94102570 0x7f9a941029b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.064+0000 7f9a9b04b640 1 --2- 172.21.15.33:0/3817386324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a94100e20 0x7f9a94101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.064+0000 7f9a9b04b640 1 -- 172.21.15.33:0/3817386324 >> 172.21.15.33:0/3817386324 conn(0x7f9a940fc7d0 msgr2=0x7f9a940febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:52.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.064+0000 7f9a9b04b640 1 -- 172.21.15.33:0/3817386324 shutdown_connections 2024-09-16T00:19:52.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.064+0000 7f9a9b04b640 1 -- 172.21.15.33:0/3817386324 wait complete. 2024-09-16T00:19:52.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.064+0000 7f9a9b04b640 1 Processor -- start 2024-09-16T00:19:52.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a9b04b640 1 -- start start 2024-09-16T00:19:52.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a9b04b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a94100e20 0x7f9a94198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a9b04b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a94102570 0x7f9a94198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a9b04b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a94102ef0 0x7f9a9419ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a9b04b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9a94112060 con 0x7f9a94102ef0 2024-09-16T00:19:52.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a9b04b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9a94111ee0 con 0x7f9a94100e20 2024-09-16T00:19:52.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a9b04b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941121e0 con 0x7f9a94102570 2024-09-16T00:19:52.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a995e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a94102ef0 0x7f9a9419ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a995e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a94102ef0 0x7f9a9419ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45920/0 (socket says 172.21.15.33:45920) 2024-09-16T00:19:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a995e8640 1 -- 172.21.15.33:0/2146831348 learned_addr learned my addr 172.21.15.33:0/2146831348 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a93fff640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a94102570 0x7f9a94198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a98de7640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a94100e20 0x7f9a94198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a995e8640 1 -- 172.21.15.33:0/2146831348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a94102570 msgr2=0x7f9a94198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a995e8640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a94102570 0x7f9a94198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a995e8640 1 -- 172.21.15.33:0/2146831348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a94100e20 msgr2=0x7f9a94198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a995e8640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a94100e20 0x7f9a94198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a995e8640 1 -- 172.21.15.33:0/2146831348 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9419d530 con 0x7f9a94102ef0 2024-09-16T00:19:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.065+0000 7f9a98de7640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a94100e20 0x7f9a94198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:19:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.066+0000 7f9a995e8640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a94102ef0 0x7f9a9419ce50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00cad0 tx=0x7f9a8c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.066+0000 7f9a91ffb640 1 -- 172.21.15.33:0/2146831348 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c00ae20 con 0x7f9a94102ef0 2024-09-16T00:19:52.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.066+0000 7f9a91ffb640 1 -- 172.21.15.33:0/2146831348 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a8c005020 con 0x7f9a94102ef0 2024-09-16T00:19:52.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.066+0000 7f9a91ffb640 1 -- 172.21.15.33:0/2146831348 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c00f6f0 con 0x7f9a94102ef0 2024-09-16T00:19:52.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.066+0000 7f9a9b04b640 1 -- 172.21.15.33:0/2146831348 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9419d820 con 0x7f9a94102ef0 2024-09-16T00:19:52.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.066+0000 7f9a9b04b640 1 -- 172.21.15.33:0/2146831348 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a941a5100 con 0x7f9a94102ef0 2024-09-16T00:19:52.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.068+0000 7f9a91ffb640 1 -- 172.21.15.33:0/2146831348 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9a8c0051d0 con 0x7f9a94102ef0 2024-09-16T00:19:52.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a91ffb640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a840a32c0 0x7f9a840a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a9d424740 1 Processor -- start 2024-09-16T00:19:52.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a98de7640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a840a32c0 0x7f9a840a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a91ffb640 1 -- 172.21.15.33:0/2146831348 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9a8c0cfd10 con 0x7f9a94102ef0 2024-09-16T00:19:52.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a9d424740 1 -- start start 2024-09-16T00:19:52.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a9d424740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb027ab40 0x55ceb0279090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a9d424740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb02795d0 0x55ceb0279a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a9d424740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb0284370 0x55ceb0286760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a9d424740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ceb01df2f0 con 0x55ceb027ab40 2024-09-16T00:19:52.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a9d424740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ceb0186cf0 con 0x55ceb02795d0 2024-09-16T00:19:52.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a9d424740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ceb01db2b0 con 0x55ceb0284370 2024-09-16T00:19:52.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a98de7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb027ab40 0x55ceb0279090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a98de7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb027ab40 0x55ceb0279090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45922/0 (socket says 172.21.15.33:45922) 2024-09-16T00:19:52.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a98de7640 1 -- 172.21.15.33:0/3676234325 learned_addr learned my addr 172.21.15.33:0/3676234325 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:52.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.069+0000 7f9a98de7640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a840a32c0 0x7f9a840a5780 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800baa0 tx=0x7f9a88005d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:52.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.070+0000 7f9a93fff640 1 --2- 172.21.15.33:0/3676234325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb02795d0 0x55ceb0279a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.070+0000 7f9a98de7640 1 -- 172.21.15.33:0/3676234325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb0284370 msgr2=0x55ceb0286760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:52.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.070+0000 7f9a98de7640 1 --2- 172.21.15.33:0/3676234325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb0284370 0x55ceb0286760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.070+0000 7f9a98de7640 1 -- 172.21.15.33:0/3676234325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb02795d0 msgr2=0x55ceb0279a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:52.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.070+0000 7f9a98de7640 1 --2- 172.21.15.33:0/3676234325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb02795d0 0x55ceb0279a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.070+0000 7f9a98de7640 1 -- 172.21.15.33:0/3676234325 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb016b3a0 con 0x55ceb027ab40 2024-09-16T00:19:52.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.070+0000 7f9a98de7640 1 --2- 172.21.15.33:0/3676234325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb027ab40 0x55ceb0279090 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800de90 tx=0x7f9a88010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a43cb5788a85373 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:52.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.070+0000 7f9a7effd640 1 -- 172.21.15.33:0/3676234325 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88020070 con 0x55ceb027ab40 2024-09-16T00:19:52.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.070+0000 7f9a7effd640 1 -- 172.21.15.33:0/3676234325 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a8800fe30 con 0x55ceb027ab40 2024-09-16T00:19:52.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.070+0000 7f9a7effd640 1 -- 172.21.15.33:0/3676234325 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8801bc00 con 0x55ceb027ab40 2024-09-16T00:19:52.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.071+0000 7f9a9d424740 1 -- 172.21.15.33:0/3676234325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb027ab40 msgr2=0x55ceb0279090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:52.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.071+0000 7f9a9d424740 1 --2- 172.21.15.33:0/3676234325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb027ab40 0x55ceb0279090 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800de90 tx=0x7f9a88010000 comp rx=0 tx=0).stop 2024-09-16T00:19:52.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.071+0000 7f9a9d424740 1 -- 172.21.15.33:0/3676234325 shutdown_connections 2024-09-16T00:19:52.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.071+0000 7f9a9d424740 1 --2- 172.21.15.33:0/3676234325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb0284370 0x55ceb0286760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.071+0000 7f9a9d424740 1 --2- 172.21.15.33:0/3676234325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb02795d0 0x55ceb0279a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.071+0000 7f9a9d424740 1 --2- 172.21.15.33:0/3676234325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb027ab40 0x55ceb0279090 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.071+0000 7f9a9d424740 1 -- 172.21.15.33:0/3676234325 >> 172.21.15.33:0/3676234325 conn(0x55ceb0242310 msgr2=0x55ceb0275c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:52.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.071+0000 7f9a9d424740 1 -- 172.21.15.33:0/3676234325 shutdown_connections 2024-09-16T00:19:52.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.071+0000 7f9a9d424740 1 -- 172.21.15.33:0/3676234325 wait complete. 2024-09-16T00:19:52.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.071+0000 7f9a9d424740 1 -- start start 2024-09-16T00:19:52.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.071+0000 7f9a9d424740 1 Processor -- start 2024-09-16T00:19:52.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a9d424740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb02795d0 0x55ceb0283da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a9d424740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb027ab40 0x55ceb0281050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a9d424740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb02817e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a9d424740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ceb0287070 con 0x55ceb0284370 2024-09-16T00:19:52.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a9d424740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ceb019a060 con 0x55ceb027ab40 2024-09-16T00:19:52.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a9d424740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ceb02871f0 con 0x55ceb02795d0 2024-09-16T00:19:52.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a9d424740 10 client.0 _ll_register_callbacks cb 0x55ceafeed8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:52.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a995e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb02817e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a995e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb02817e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45934/0 (socket says 172.21.15.33:45934) 2024-09-16T00:19:52.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a995e8640 1 -- 172.21.15.33:0/2827867116 learned_addr learned my addr 172.21.15.33:0/2827867116 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:52.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a93fff640 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb027ab40 0x55ceb0281050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a98de7640 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb02795d0 0x55ceb0283da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a995e8640 1 -- 172.21.15.33:0/2827867116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb02795d0 msgr2=0x55ceb0283da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:52.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a995e8640 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb02795d0 0x55ceb0283da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a995e8640 1 -- 172.21.15.33:0/2827867116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb027ab40 msgr2=0x55ceb0281050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:52.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a995e8640 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb027ab40 0x55ceb0281050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a995e8640 1 -- 172.21.15.33:0/2827867116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceaff503c0 con 0x55ceb0284370 2024-09-16T00:19:52.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a93fff640 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb027ab40 0x55ceb0281050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:52.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a98de7640 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb02795d0 0x55ceb0283da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a995e8640 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb02817e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c09be40 tx=0x7f9a8c09c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104d08302175403f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a73fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a73fff640 1 -- 172.21.15.33:0/2827867116 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c01d040 con 0x55ceb0284370 2024-09-16T00:19:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a73fff640 1 -- 172.21.15.33:0/2827867116 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9a8c0a4690 con 0x55ceb0284370 2024-09-16T00:19:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.072+0000 7f9a9d424740 1 -- 172.21.15.33:0/2827867116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a8800fe30 con 0x55ceb0284370 2024-09-16T00:19:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.073+0000 7f9a73fff640 1 -- 172.21.15.33:0/2827867116 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9a8c0a4b20 con 0x55ceb0284370 2024-09-16T00:19:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.073+0000 7f9a9d424740 10 client.15819 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.073+0000 7f9a9d424740 1 -- 172.21.15.33:0/2827867116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ceb0292bb0 con 0x55ceb0284370 2024-09-16T00:19:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.073+0000 7f9a73fff640 1 -- 172.21.15.33:0/2827867116 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a8800fe30 con 0x55ceb0284370 2024-09-16T00:19:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.073+0000 7f9a73fff640 1 -- 172.21.15.33:0/2827867116 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9a8c07ac70 con 0x55ceb0284370 2024-09-16T00:19:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.073+0000 7f9a9d424740 10 client.15819 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.073+0000 7f9a9d424740 10 client.15819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.073+0000 7f9a9d424740 20 client.15819 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.073+0000 7f9a9d424740 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ceb0293fc0 0x55ceb02963b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.073+0000 7f9a9d424740 4 client.15819 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.073+0000 7f9a9d424740 1 -- 172.21.15.33:0/2827867116 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ceb01ec860 con 0x55ceb0293fc0 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.073+0000 7f9a98de7640 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ceb0293fc0 0x55ceb02963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.074+0000 7f9a98de7640 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ceb0293fc0 0x55ceb02963b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9d021d01779f180 server_cookie=61ecb5b2ef6889c9 in_seq=0 out_seq=0 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.074+0000 7f9a73fff640 10 client.15819 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.074+0000 7f9a73fff640 1 -- 172.21.15.33:0/2827867116 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ceb01ec860 con 0x55ceb0293fc0 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.074+0000 7f9a73fff640 10 client.15819 handle_command_reply: tid=0 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.074+0000 7f9a9d424740 1 client.15819 shutdown 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.075+0000 7f9a9d424740 1 -- 172.21.15.33:0/2827867116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 msgr2=0x55ceb02817e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.075+0000 7f9a9d424740 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb02817e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c09be40 tx=0x7f9a8c09c6f0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.075+0000 7f9a9d424740 1 -- 172.21.15.33:0/2827867116 shutdown_connections 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.075+0000 7f9a9d424740 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ceb0293fc0 0x55ceb02963b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.075+0000 7f9a9d424740 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb02817e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.075+0000 7f9a9d424740 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb027ab40 0x55ceb0281050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.075+0000 7f9a73fff640 0 client.15819 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.075+0000 7f9a9d424740 1 --2- 172.21.15.33:0/2827867116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb02795d0 0x55ceb0283da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.075+0000 7f9a9d424740 1 -- 172.21.15.33:0/2827867116 >> 172.21.15.33:0/2827867116 conn(0x55ceb0242310 msgr2=0x55ceb027b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.075+0000 7f9a9d424740 1 -- 172.21.15.33:0/2827867116 shutdown_connections 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.075+0000 7f9a9d424740 1 -- 172.21.15.33:0/2827867116 wait complete. 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.075+0000 7f9a9d424740 20 client.15819 trim_cache size 0 max 16384 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a9d424740 1 Processor -- start 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a9d424740 1 -- start start 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a9d424740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb02795d0 0x55ceb027a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a9d424740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb027ab40 0x55ceb02754f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a9d424740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb0275a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a9d424740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941b1960 con 0x55ceb0284370 2024-09-16T00:19:52.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a9d424740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941af230 con 0x55ceb027ab40 2024-09-16T00:19:52.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a9d424740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941ad080 con 0x55ceb02795d0 2024-09-16T00:19:52.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a995e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb0275a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a995e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb0275a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45944/0 (socket says 172.21.15.33:45944) 2024-09-16T00:19:52.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a995e8640 1 -- 172.21.15.33:0/3072335322 learned_addr learned my addr 172.21.15.33:0/3072335322 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:52.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a98de7640 1 --2- 172.21.15.33:0/3072335322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb02795d0 0x55ceb027a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a93fff640 1 --2- 172.21.15.33:0/3072335322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb027ab40 0x55ceb02754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a995e8640 1 -- 172.21.15.33:0/3072335322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb02795d0 msgr2=0x55ceb027a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:52.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a995e8640 1 --2- 172.21.15.33:0/3072335322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb02795d0 0x55ceb027a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a995e8640 1 -- 172.21.15.33:0/3072335322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb027ab40 msgr2=0x55ceb02754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a995e8640 1 --2- 172.21.15.33:0/3072335322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb027ab40 0x55ceb02754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.088+0000 7f9a995e8640 1 -- 172.21.15.33:0/3072335322 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb027f5a0 con 0x55ceb0284370 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a995e8640 1 --2- 172.21.15.33:0/3072335322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb0275a30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c098310 tx=0x7f9a8c09a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57dc6c24b10f926b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a7e7fc640 1 -- 172.21.15.33:0/3072335322 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c07f070 con 0x55ceb0284370 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a7e7fc640 1 -- 172.21.15.33:0/3072335322 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a8c005560 con 0x55ceb0284370 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a7e7fc640 1 -- 172.21.15.33:0/3072335322 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c01d040 con 0x55ceb0284370 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a9d424740 1 -- 172.21.15.33:0/3072335322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 msgr2=0x55ceb0275a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a9d424740 1 --2- 172.21.15.33:0/3072335322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb0275a30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c098310 tx=0x7f9a8c09a3c0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a9d424740 1 -- 172.21.15.33:0/3072335322 shutdown_connections 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a9d424740 1 --2- 172.21.15.33:0/3072335322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb0275a30 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a9d424740 1 --2- 172.21.15.33:0/3072335322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb027ab40 0x55ceb02754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a9d424740 1 --2- 172.21.15.33:0/3072335322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb02795d0 0x55ceb027a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a9d424740 1 -- 172.21.15.33:0/3072335322 >> 172.21.15.33:0/3072335322 conn(0x55ceb0242310 msgr2=0x55ceb0270a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:52.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a9d424740 1 -- 172.21.15.33:0/3072335322 shutdown_connections 2024-09-16T00:19:52.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.089+0000 7f9a9d424740 1 -- 172.21.15.33:0/3072335322 wait complete. 2024-09-16T00:19:52.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a9d424740 1 -- start start 2024-09-16T00:19:52.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a9d424740 1 Processor -- start 2024-09-16T00:19:52.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a9d424740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb02795d0 0x55ceb028cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a9d424740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb027ab40 0x55ceb0288060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a9d424740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb02887f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a9d424740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941ad6f0 con 0x55ceb0284370 2024-09-16T00:19:52.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a9d424740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941aeb60 con 0x55ceb02795d0 2024-09-16T00:19:52.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a9d424740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941addc0 con 0x55ceb027ab40 2024-09-16T00:19:52.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a9d424740 10 client.0 _ll_register_callbacks cb 0x7f9a8c09b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:52.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a995e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb02887f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a995e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb02887f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45950/0 (socket says 172.21.15.33:45950) 2024-09-16T00:19:52.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a995e8640 1 -- 172.21.15.33:0/2775658368 learned_addr learned my addr 172.21.15.33:0/2775658368 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:52.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a98de7640 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb02795d0 0x55ceb028cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.090+0000 7f9a93fff640 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb027ab40 0x55ceb0288060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a995e8640 1 -- 172.21.15.33:0/2775658368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb027ab40 msgr2=0x55ceb0288060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:52.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a995e8640 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb027ab40 0x55ceb0288060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a995e8640 1 -- 172.21.15.33:0/2775658368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb02795d0 msgr2=0x55ceb028cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:52.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a995e8640 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb02795d0 0x55ceb028cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a995e8640 1 -- 172.21.15.33:0/2775658368 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceb02fdbb0 con 0x55ceb0284370 2024-09-16T00:19:52.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a93fff640 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb027ab40 0x55ceb0288060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:52.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a995e8640 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb02887f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0a45f0 tx=0x7f9a8c09b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f874a09affe98ecb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:52.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a73fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:52.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a73fff640 1 -- 172.21.15.33:0/2775658368 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c07f070 con 0x55ceb0284370 2024-09-16T00:19:52.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a73fff640 1 -- 172.21.15.33:0/2775658368 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9a8c0a4890 con 0x55ceb0284370 2024-09-16T00:19:52.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a9d424740 1 -- 172.21.15.33:0/2775658368 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a8c005560 con 0x55ceb0284370 2024-09-16T00:19:52.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a73fff640 1 -- 172.21.15.33:0/2775658368 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9a8c01d040 con 0x55ceb0284370 2024-09-16T00:19:52.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a9d424740 10 client.15825 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:52.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.091+0000 7f9a9d424740 1 -- 172.21.15.33:0/2775658368 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ceb02fdbb0 con 0x55ceb0284370 2024-09-16T00:19:52.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.092+0000 7f9a73fff640 1 -- 172.21.15.33:0/2775658368 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a8c005560 con 0x55ceb0284370 2024-09-16T00:19:52.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.092+0000 7f9a73fff640 1 -- 172.21.15.33:0/2775658368 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9a8c0a13f0 con 0x55ceb0284370 2024-09-16T00:19:52.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.092+0000 7f9a9d424740 10 client.15825 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:52.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.092+0000 7f9a9d424740 10 client.15825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:52.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.092+0000 7f9a9d424740 20 client.15825 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:52.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.092+0000 7f9a9d424740 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ceb02fef60 0x55ceb0301350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:52.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.092+0000 7f9a9d424740 4 client.15825 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:19:52.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.092+0000 7f9a9d424740 1 -- 172.21.15.33:0/2775658368 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ceb01e6d70 con 0x55ceb02fef60 2024-09-16T00:19:52.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.092+0000 7f9a98de7640 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ceb02fef60 0x55ceb0301350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:52.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a98de7640 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ceb02fef60 0x55ceb0301350 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c78f9501a8c4341f server_cookie=5a1caf5a3a0762b1 in_seq=0 out_seq=0 2024-09-16T00:19:52.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a73fff640 10 client.15825 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:52.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a73fff640 1 -- 172.21.15.33:0/2775658368 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ceb01e6d70 con 0x55ceb02fef60 2024-09-16T00:19:52.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a73fff640 10 client.15825 handle_command_reply: tid=0 2024-09-16T00:19:52.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a9d424740 1 client.15825 shutdown 2024-09-16T00:19:52.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a9d424740 1 -- 172.21.15.33:0/2775658368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 msgr2=0x55ceb02887f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:52.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a9d424740 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb02887f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0a45f0 tx=0x7f9a8c09b8e0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a9d424740 1 -- 172.21.15.33:0/2775658368 shutdown_connections 2024-09-16T00:19:52.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a9d424740 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ceb02fef60 0x55ceb0301350 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a9d424740 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb0284370 0x55ceb02887f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a9d424740 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ceb027ab40 0x55ceb0288060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a9d424740 1 --2- 172.21.15.33:0/2775658368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ceb02795d0 0x55ceb028cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a9d424740 1 -- 172.21.15.33:0/2775658368 >> 172.21.15.33:0/2775658368 conn(0x55ceb0242310 msgr2=0x55ceb02828b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:52.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a73fff640 0 client.15825 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:52.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a9d424740 1 -- 172.21.15.33:0/2775658368 shutdown_connections 2024-09-16T00:19:52.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a9d424740 1 -- 172.21.15.33:0/2775658368 wait complete. 2024-09-16T00:19:52.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.093+0000 7f9a9d424740 20 client.15825 trim_cache size 0 max 16384 2024-09-16T00:19:52.115 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:19:52.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.094+0000 7f9a9b04b640 1 -- 172.21.15.33:0/2146831348 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a840a32c0 msgr2=0x7f9a840a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:52.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.094+0000 7f9a9b04b640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a840a32c0 0x7f9a840a5780 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800baa0 tx=0x7f9a88005d90 comp rx=0 tx=0).stop 2024-09-16T00:19:52.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.094+0000 7f9a9b04b640 1 -- 172.21.15.33:0/2146831348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a94102ef0 msgr2=0x7f9a9419ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:52.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.094+0000 7f9a9b04b640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a94102ef0 0x7f9a9419ce50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00cad0 tx=0x7f9a8c00cf90 comp rx=0 tx=0).stop 2024-09-16T00:19:52.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.094+0000 7f9a9b04b640 1 -- 172.21.15.33:0/2146831348 shutdown_connections 2024-09-16T00:19:52.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.094+0000 7f9a9b04b640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a840a32c0 0x7f9a840a5780 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.094+0000 7f9a9b04b640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a94102ef0 0x7f9a9419ce50 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.094+0000 7f9a9b04b640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a94102570 0x7f9a94198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.094+0000 7f9a9b04b640 1 --2- 172.21.15.33:0/2146831348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a94100e20 0x7f9a94198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:52.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.094+0000 7f9a9b04b640 1 -- 172.21.15.33:0/2146831348 >> 172.21.15.33:0/2146831348 conn(0x7f9a940fc7d0 msgr2=0x7f9a94105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:52.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.094+0000 7f9a9b04b640 1 -- 172.21.15.33:0/2146831348 shutdown_connections 2024-09-16T00:19:52.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:52.094+0000 7f9a9b04b640 1 -- 172.21.15.33:0/2146831348 wait complete. 2024-09-16T00:19:52.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.87 MB/sec warmup 60 sec latency 6831.795 ms 2024-09-16T00:19:53.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.80 MB/sec warmup 61 sec latency 7831.845 ms 2024-09-16T00:19:53.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:53 smithi033 ceph-mon[31243]: pgmap v432: 129 pgs: 129 active+clean; 38 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 576 KiB/s wr, 49 op/s 2024-09-16T00:19:53.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:53 smithi083 ceph-mon[35242]: pgmap v432: 129 pgs: 129 active+clean; 38 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 576 KiB/s wr, 49 op/s 2024-09-16T00:19:53.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:53 smithi060 ceph-mon[35185]: pgmap v432: 129 pgs: 129 active+clean; 38 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 576 KiB/s wr, 49 op/s 2024-09-16T00:19:54.010 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:19:54.011 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:19:54.294 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:54.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.74 MB/sec warmup 62 sec latency 8831.955 ms 2024-09-16T00:19:54.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.858+0000 7f8650f67640 1 -- 172.21.15.33:0/503779117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f864c102d30 msgr2=0x7f864c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:54.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.859+0000 7f8650f67640 1 --2- 172.21.15.33:0/503779117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f864c102d30 0x7f864c103110 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8644009cd0 tx=0x7f864402a8e0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.859+0000 7f8650f67640 1 -- 172.21.15.33:0/503779117 shutdown_connections 2024-09-16T00:19:54.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.859+0000 7f8650f67640 1 --2- 172.21.15.33:0/503779117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f864c1082a0 0x7f864c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.859+0000 7f8650f67640 1 --2- 172.21.15.33:0/503779117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c1036e0 0x7f864c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.859+0000 7f8650f67640 1 --2- 172.21.15.33:0/503779117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f864c102d30 0x7f864c103110 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.859+0000 7f8650f67640 1 -- 172.21.15.33:0/503779117 >> 172.21.15.33:0/503779117 conn(0x7f864c0fc7d0 msgr2=0x7f864c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:54.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.859+0000 7f8650f67640 1 -- 172.21.15.33:0/503779117 shutdown_connections 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.859+0000 7f8650f67640 1 -- 172.21.15.33:0/503779117 wait complete. 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.859+0000 7f8650f67640 1 Processor -- start 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.859+0000 7f8650f67640 1 -- start start 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8650f67640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f864c102d30 0x7f864c100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8650f67640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f864c1036e0 0x7f864c100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8650f67640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c1082a0 0x7f864c101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8650f67640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f864c1120a0 con 0x7f864c1036e0 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8650f67640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f864c111f20 con 0x7f864c102d30 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8650f67640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f864c112220 con 0x7f864c1082a0 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8649d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f864c1036e0 0x7f864c100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8649d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f864c1036e0 0x7f864c100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45974/0 (socket says 172.21.15.33:45974) 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8649d9b640 1 -- 172.21.15.33:0/3231326598 learned_addr learned my addr 172.21.15.33:0/3231326598 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:54.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f864a59c640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f864c102d30 0x7f864c100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f864ad9d640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c1082a0 0x7f864c101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8649d9b640 1 -- 172.21.15.33:0/3231326598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c1082a0 msgr2=0x7f864c101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8649d9b640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c1082a0 0x7f864c101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8649d9b640 1 -- 172.21.15.33:0/3231326598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f864c102d30 msgr2=0x7f864c100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8649d9b640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f864c102d30 0x7f864c100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8649d9b640 1 -- 172.21.15.33:0/3231326598 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c1a6a80 con 0x7f864c1036e0 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f864a59c640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f864c102d30 0x7f864c100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.860+0000 7f8649d9b640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f864c1036e0 0x7f864c100ed0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f863400d750 tx=0x7f863400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.861+0000 7f86437fe640 1 -- 172.21.15.33:0/3231326598 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634017070 con 0x7f864c1036e0 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.861+0000 7f86437fe640 1 -- 172.21.15.33:0/3231326598 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8634005500 con 0x7f864c1036e0 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.861+0000 7f8650f67640 1 -- 172.21.15.33:0/3231326598 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f864c1a6d70 con 0x7f864c1036e0 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.861+0000 7f86437fe640 1 -- 172.21.15.33:0/3231326598 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634004e00 con 0x7f864c1036e0 2024-09-16T00:19:54.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.861+0000 7f8650f67640 1 -- 172.21.15.33:0/3231326598 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f864c1a7250 con 0x7f864c1036e0 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.863+0000 7f86437fe640 1 -- 172.21.15.33:0/3231326598 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8634004fa0 con 0x7f864c1036e0 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.863+0000 7f8653340740 1 Processor -- start 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f8653340740 1 -- start start 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f8653340740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561969513b60 0x561969513f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f8653340740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 0x56196951e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f8653340740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 0x561969521130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f8653340740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561969479280 con 0x56196951ed40 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f8653340740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561969420b10 con 0x561969513b60 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f8653340740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561969475240 con 0x561969514480 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f8649d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 0x561969521130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f8649d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 0x561969521130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45980/0 (socket says 172.21.15.33:45980) 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f8649d9b640 1 -- 172.21.15.33:0/3438524849 learned_addr learned my addr 172.21.15.33:0/3438524849 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:54.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f864ad9d640 1 --2- 172.21.15.33:0/3438524849 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561969513b60 0x561969513f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f864a59c640 1 --2- 172.21.15.33:0/3438524849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 0x56196951e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.864+0000 7f86437fe640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f86380a32c0 0x7f86380a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f8649d9b640 1 -- 172.21.15.33:0/3438524849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 msgr2=0x56196951e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:54.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f8649d9b640 1 --2- 172.21.15.33:0/3438524849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 0x56196951e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f8649d9b640 1 -- 172.21.15.33:0/3438524849 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561969513b60 msgr2=0x561969513f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:54.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f8649d9b640 1 --2- 172.21.15.33:0/3438524849 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561969513b60 0x561969513f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f8649d9b640 1 -- 172.21.15.33:0/3438524849 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56196942d1f0 con 0x56196951ed40 2024-09-16T00:19:54.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f864a59c640 1 --2- 172.21.15.33:0/3438524849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 0x56196951e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:19:54.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f864ad9d640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f86380a32c0 0x7f86380a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f86437fe640 1 -- 172.21.15.33:0/3231326598 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f86340cf7e0 con 0x7f864c1036e0 2024-09-16T00:19:54.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f8649d9b640 1 --2- 172.21.15.33:0/3438524849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 0x561969521130 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f863401aab0 tx=0x7f8634020c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9ad829d563d170 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:54.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f864ad9d640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f86380a32c0 0x7f86380a5780 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f863c005d80 tx=0x7f863c0046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f8640ff9640 1 -- 172.21.15.33:0/3438524849 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863408ee60 con 0x56196951ed40 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f8640ff9640 1 -- 172.21.15.33:0/3438524849 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f863401ab70 con 0x56196951ed40 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.865+0000 7f8640ff9640 1 -- 172.21.15.33:0/3438524849 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634086110 con 0x56196951ed40 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.866+0000 7f8653340740 1 -- 172.21.15.33:0/3438524849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 msgr2=0x561969521130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.866+0000 7f8653340740 1 --2- 172.21.15.33:0/3438524849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 0x561969521130 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f863401aab0 tx=0x7f8634020c20 comp rx=0 tx=0).stop 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.866+0000 7f8653340740 1 -- 172.21.15.33:0/3438524849 shutdown_connections 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.866+0000 7f8653340740 1 --2- 172.21.15.33:0/3438524849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 0x561969521130 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.866+0000 7f8653340740 1 --2- 172.21.15.33:0/3438524849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 0x56196951e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.866+0000 7f8653340740 1 --2- 172.21.15.33:0/3438524849 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561969513b60 0x561969513f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.866+0000 7f8653340740 1 -- 172.21.15.33:0/3438524849 >> 172.21.15.33:0/3438524849 conn(0x5619695100f0 msgr2=0x5619695104d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.866+0000 7f8653340740 1 -- 172.21.15.33:0/3438524849 shutdown_connections 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.866+0000 7f8653340740 1 -- 172.21.15.33:0/3438524849 wait complete. 2024-09-16T00:19:54.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.866+0000 7f8653340740 1 -- start start 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f8653340740 1 Processor -- start 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f8653340740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 0x56196951bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f8653340740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 0x56196951c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f8653340740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56196951ed40 0x56196951c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f8653340740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561969521910 con 0x561969513b60 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f8653340740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561969433f10 con 0x56196951ed40 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f8653340740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561969521a90 con 0x561969514480 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f8653340740 10 client.0 _ll_register_callbacks cb 0x5619693fca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f864a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 0x56196951bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f864a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 0x56196951bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45996/0 (socket says 172.21.15.33:45996) 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f864a59c640 1 -- 172.21.15.33:0/2919354564 learned_addr learned my addr 172.21.15.33:0/2919354564 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f8649d9b640 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 0x56196951c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f864ad9d640 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56196951ed40 0x56196951c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f864a59c640 1 -- 172.21.15.33:0/2919354564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 msgr2=0x56196951c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f864a59c640 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 0x56196951c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f864a59c640 1 -- 172.21.15.33:0/2919354564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56196951ed40 msgr2=0x56196951c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f864a59c640 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56196951ed40 0x56196951c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f864a59c640 1 -- 172.21.15.33:0/2919354564 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619691ea6a0 con 0x561969513b60 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f864a59c640 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 0x56196951bbb0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8644009cd0 tx=0x7f8644031c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d696c53f311b3e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.867+0000 7f8631ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8631ffb640 1 -- 172.21.15.33:0/2919354564 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644049060 con 0x561969513b60 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8631ffb640 1 -- 172.21.15.33:0/2919354564 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8644003b80 con 0x561969513b60 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8653340740 1 -- 172.21.15.33:0/2919354564 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5619694867f0 con 0x561969513b60 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8631ffb640 1 -- 172.21.15.33:0/2919354564 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8644003d80 con 0x561969513b60 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8653340740 10 client.15837 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:54.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8653340740 1 -- 172.21.15.33:0/2919354564 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56196952d4a0 con 0x561969513b60 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8631ffb640 1 -- 172.21.15.33:0/2919354564 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5619694867f0 con 0x561969513b60 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8631ffb640 1 -- 172.21.15.33:0/2919354564 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8644043070 con 0x561969513b60 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8653340740 10 client.15837 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8653340740 10 client.15837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8653340740 20 client.15837 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8653340740 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56196952e960 0x561969530d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8653340740 4 client.15837 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.868+0000 7f8653340740 1 -- 172.21.15.33:0/2919354564 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561969480d00 con 0x56196952e960 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.869+0000 7f8649d9b640 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56196952e960 0x561969530d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.869+0000 7f8649d9b640 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56196952e960 0x561969530d70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fedd9d4b139bb72 server_cookie=8d8506ee7b026926 in_seq=0 out_seq=0 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.869+0000 7f8631ffb640 10 client.15837 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8631ffb640 1 -- 172.21.15.33:0/2919354564 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561969480d00 con 0x56196952e960 2024-09-16T00:19:54.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8631ffb640 10 client.15837 handle_command_reply: tid=0 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8653340740 1 client.15837 shutdown 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8653340740 1 -- 172.21.15.33:0/2919354564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 msgr2=0x56196951bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8653340740 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 0x56196951bbb0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8644009cd0 tx=0x7f8644031c30 comp rx=0 tx=0).stop 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8653340740 1 -- 172.21.15.33:0/2919354564 shutdown_connections 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8653340740 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56196952e960 0x561969530d70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8653340740 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56196951ed40 0x56196951c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8653340740 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 0x56196951c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8631ffb640 0 client.15837 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8653340740 1 --2- 172.21.15.33:0/2919354564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 0x56196951bbb0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8653340740 1 -- 172.21.15.33:0/2919354564 >> 172.21.15.33:0/2919354564 conn(0x5619695100f0 msgr2=0x561969515e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8653340740 1 -- 172.21.15.33:0/2919354564 shutdown_connections 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8653340740 1 -- 172.21.15.33:0/2919354564 wait complete. 2024-09-16T00:19:54.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.870+0000 7f8653340740 20 client.15837 trim_cache size 0 max 16384 2024-09-16T00:19:54.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.887+0000 7f8653340740 1 Processor -- start 2024-09-16T00:19:54.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.887+0000 7f8653340740 1 -- start start 2024-09-16T00:19:54.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.887+0000 7f8653340740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969513b60 0x56196951c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.887+0000 7f8653340740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561969514480 0x561969536640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.887+0000 7f8653340740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 0x561969536b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.887+0000 7f8653340740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f864c1b3c90 con 0x56196951ed40 2024-09-16T00:19:54.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.887+0000 7f8653340740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f864c1b1560 con 0x561969514480 2024-09-16T00:19:54.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.887+0000 7f8653340740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f864c1af3b0 con 0x561969513b60 2024-09-16T00:19:54.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.887+0000 7f864ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 0x561969536b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f864ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 0x561969536b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46006/0 (socket says 172.21.15.33:46006) 2024-09-16T00:19:54.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f864ad9d640 1 -- 172.21.15.33:0/3500712837 learned_addr learned my addr 172.21.15.33:0/3500712837 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:54.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f864ad9d640 1 -- 172.21.15.33:0/3500712837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969513b60 msgr2=0x56196951c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f864a59c640 1 --2- 172.21.15.33:0/3500712837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969513b60 0x56196951c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f8649d9b640 1 --2- 172.21.15.33:0/3500712837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561969514480 0x561969536640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f864ad9d640 1 --2- 172.21.15.33:0/3500712837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969513b60 0x56196951c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f864ad9d640 1 -- 172.21.15.33:0/3500712837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561969514480 msgr2=0x561969536640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f864ad9d640 1 --2- 172.21.15.33:0/3500712837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561969514480 0x561969536640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f864ad9d640 1 -- 172.21.15.33:0/3500712837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56196950ff20 con 0x56196951ed40 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f8649d9b640 1 --2- 172.21.15.33:0/3500712837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561969514480 0x561969536640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f864ad9d640 1 --2- 172.21.15.33:0/3500712837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 0x561969536b80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f863c01d490 tx=0x7f863c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629c7bda949e4a0e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f8633fff640 1 -- 172.21.15.33:0/3500712837 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c01d0d0 con 0x56196951ed40 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f8633fff640 1 -- 172.21.15.33:0/3500712837 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f863c01d270 con 0x56196951ed40 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.888+0000 7f8633fff640 1 -- 172.21.15.33:0/3500712837 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c010790 con 0x56196951ed40 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 -- 172.21.15.33:0/3500712837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 msgr2=0x561969536b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:54.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 --2- 172.21.15.33:0/3500712837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 0x561969536b80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f863c01d490 tx=0x7f863c015000 comp rx=0 tx=0).stop 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 -- 172.21.15.33:0/3500712837 shutdown_connections 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 --2- 172.21.15.33:0/3500712837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56196951ed40 0x561969536b80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 --2- 172.21.15.33:0/3500712837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561969514480 0x561969536640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 --2- 172.21.15.33:0/3500712837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969513b60 0x56196951c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 -- 172.21.15.33:0/3500712837 >> 172.21.15.33:0/3500712837 conn(0x5619695100f0 msgr2=0x561969515dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 -- 172.21.15.33:0/3500712837 shutdown_connections 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 -- 172.21.15.33:0/3500712837 wait complete. 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 -- start start 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 Processor -- start 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 0x561969522ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 0x5619695231e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56196951ed40 0x561969523720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f864c1afa20 con 0x561969513b60 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f864c1b0e90 con 0x56196951ed40 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.889+0000 7f8653340740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f864c1b00f0 con 0x561969514480 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f8653340740 10 client.0 _ll_register_callbacks cb 0x7f8644009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f864a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 0x561969522ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f864a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 0x561969522ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46012/0 (socket says 172.21.15.33:46012) 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f864a59c640 1 -- 172.21.15.33:0/51515302 learned_addr learned my addr 172.21.15.33:0/51515302 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f864ad9d640 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56196951ed40 0x561969523720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f864a59c640 1 -- 172.21.15.33:0/51515302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 msgr2=0x5619695231e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f864a59c640 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 0x5619695231e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f864a59c640 1 -- 172.21.15.33:0/51515302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56196951ed40 msgr2=0x561969523720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f864a59c640 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56196951ed40 0x561969523720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f864a59c640 1 -- 172.21.15.33:0/51515302 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561969598860 con 0x561969513b60 2024-09-16T00:19:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f864ad9d640 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56196951ed40 0x561969523720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f864a59c640 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 0x561969522ca0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8644000f80 tx=0x7f8644009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9898948e32d8b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f8631ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f8631ffb640 1 -- 172.21.15.33:0/51515302 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644049050 con 0x561969513b60 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f8631ffb640 1 -- 172.21.15.33:0/51515302 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8644041040 con 0x561969513b60 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f8653340740 1 -- 172.21.15.33:0/51515302 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f863c01d270 con 0x561969513b60 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f8631ffb640 1 -- 172.21.15.33:0/51515302 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8644030ed0 con 0x561969513b60 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f8653340740 10 client.15843 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.890+0000 7f8653340740 1 -- 172.21.15.33:0/51515302 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561969599b70 con 0x561969513b60 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.891+0000 7f8631ffb640 1 -- 172.21.15.33:0/51515302 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f863c01d270 con 0x561969513b60 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.891+0000 7f8631ffb640 1 -- 172.21.15.33:0/51515302 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8644043040 con 0x561969513b60 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.891+0000 7f8653340740 10 client.15843 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.891+0000 7f8653340740 10 client.15843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:54.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.891+0000 7f8653340740 20 client.15843 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:54.915 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:19:54.915 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:19:54.915 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "265eb9c6-e85a-4ebb-b79d-0cb091c77e73", 2024-09-16T00:19:54.915 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:19:54.915 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:19:54.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.891+0000 7f8653340740 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56196959a2e0 0x56196959c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:54.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.891+0000 7f8653340740 4 client.15843 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:19:54.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.891+0000 7f8653340740 1 -- 172.21.15.33:0/51515302 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5619693fd110 con 0x56196959a2e0 2024-09-16T00:19:54.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.891+0000 7f8649d9b640 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56196959a2e0 0x56196959c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:54.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.891+0000 7f8631ffb640 10 client.15843 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:54.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.891+0000 7f8649d9b640 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56196959a2e0 0x56196959c720 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67418e24e4b1b7db server_cookie=b85a539d82e88e51 in_seq=0 out_seq=0 2024-09-16T00:19:54.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8631ffb640 1 -- 172.21.15.33:0/51515302 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5619693fd110 con 0x56196959a2e0 2024-09-16T00:19:54.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8631ffb640 10 client.15843 handle_command_reply: tid=0 2024-09-16T00:19:54.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8653340740 1 client.15843 shutdown 2024-09-16T00:19:54.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8653340740 1 -- 172.21.15.33:0/51515302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 msgr2=0x561969522ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:54.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8653340740 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 0x561969522ca0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8644000f80 tx=0x7f8644009540 comp rx=0 tx=0).stop 2024-09-16T00:19:54.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8653340740 1 -- 172.21.15.33:0/51515302 shutdown_connections 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8653340740 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56196959a2e0 0x56196959c720 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8653340740 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56196951ed40 0x561969523720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8653340740 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561969514480 0x5619695231e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8631ffb640 0 client.15843 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8653340740 1 --2- 172.21.15.33:0/51515302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969513b60 0x561969522ca0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8653340740 1 -- 172.21.15.33:0/51515302 >> 172.21.15.33:0/51515302 conn(0x5619695100f0 msgr2=0x56196950ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8653340740 1 -- 172.21.15.33:0/51515302 shutdown_connections 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8653340740 1 -- 172.21.15.33:0/51515302 wait complete. 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.893+0000 7f8653340740 20 client.15843 trim_cache size 0 max 16384 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.894+0000 7f8650f67640 1 -- 172.21.15.33:0/3231326598 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f86380a32c0 msgr2=0x7f86380a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.894+0000 7f8650f67640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f86380a32c0 0x7f86380a5780 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f863c005d80 tx=0x7f863c0046c0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.894+0000 7f8650f67640 1 -- 172.21.15.33:0/3231326598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f864c1036e0 msgr2=0x7f864c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.894+0000 7f8650f67640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f864c1036e0 0x7f864c100ed0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f863400d750 tx=0x7f863400dc10 comp rx=0 tx=0).stop 2024-09-16T00:19:54.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.894+0000 7f8650f67640 1 -- 172.21.15.33:0/3231326598 shutdown_connections 2024-09-16T00:19:54.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.894+0000 7f8650f67640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f86380a32c0 0x7f86380a5780 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.894+0000 7f8650f67640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c1082a0 0x7f864c101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.894+0000 7f8650f67640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f864c1036e0 0x7f864c100ed0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.894+0000 7f8650f67640 1 --2- 172.21.15.33:0/3231326598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f864c102d30 0x7f864c100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:54.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.894+0000 7f8650f67640 1 -- 172.21.15.33:0/3231326598 >> 172.21.15.33:0/3231326598 conn(0x7f864c0fc7d0 msgr2=0x7f864c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:54.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.894+0000 7f8650f67640 1 -- 172.21.15.33:0/3231326598 shutdown_connections 2024-09-16T00:19:54.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:54.894+0000 7f8650f67640 1 -- 172.21.15.33:0/3231326598 wait complete. 2024-09-16T00:19:55.642 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:19:55.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.68 MB/sec warmup 63 sec latency 9831.994 ms 2024-09-16T00:19:55.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:55 smithi033 ceph-mon[31243]: pgmap v433: 129 pgs: 129 active+clean; 38 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 478 KiB/s wr, 47 op/s 2024-09-16T00:19:55.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:55 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:19:55.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:55 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:55.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:55 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:19:55.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:55 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:19:55.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:55 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:55.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:55 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:55.903 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:19:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:55 smithi083 ceph-mon[35242]: pgmap v433: 129 pgs: 129 active+clean; 38 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 478 KiB/s wr, 47 op/s 2024-09-16T00:19:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:55 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:19:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:55 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:55.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:55 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:19:55.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:55 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:19:55.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:55 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:55.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:55 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:55 smithi060 ceph-mon[35185]: pgmap v433: 129 pgs: 129 active+clean; 38 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 478 KiB/s wr, 47 op/s 2024-09-16T00:19:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:55 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:19:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:55 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:55 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:19:55.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:55 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:19:55.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:55 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:55.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:55 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:19:56.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.432+0000 7fe94d47c640 1 -- 172.21.15.33:0/1517103178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe948103640 msgr2=0x7fe94810bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:56.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.432+0000 7fe94d47c640 1 --2- 172.21.15.33:0/1517103178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe948103640 0x7fe94810bb30 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009900 tx=0x7fe93c02a480 comp rx=0 tx=0).stop 2024-09-16T00:19:56.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.432+0000 7fe94d47c640 1 -- 172.21.15.33:0/1517103178 shutdown_connections 2024-09-16T00:19:56.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.432+0000 7fe94d47c640 1 --2- 172.21.15.33:0/1517103178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe94810c150 0x7fe94810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.432+0000 7fe94d47c640 1 --2- 172.21.15.33:0/1517103178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe948103640 0x7fe94810bb30 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.432+0000 7fe94d47c640 1 --2- 172.21.15.33:0/1517103178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe948102d20 0x7fe948103100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.432+0000 7fe94d47c640 1 -- 172.21.15.33:0/1517103178 >> 172.21.15.33:0/1517103178 conn(0x7fe9480fc780 msgr2=0x7fe9480feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:56.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.433+0000 7fe94d47c640 1 -- 172.21.15.33:0/1517103178 shutdown_connections 2024-09-16T00:19:56.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.433+0000 7fe94d47c640 1 -- 172.21.15.33:0/1517103178 wait complete. 2024-09-16T00:19:56.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.433+0000 7fe94d47c640 1 Processor -- start 2024-09-16T00:19:56.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.433+0000 7fe94d47c640 1 -- start start 2024-09-16T00:19:56.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.433+0000 7fe94d47c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe948102d20 0x7fe948198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.433+0000 7fe94d47c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe948103640 0x7fe9481989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.433+0000 7fe94d47c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe94810c150 0x7fe94819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe94d47c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe948111f50 con 0x7fe948102d20 2024-09-16T00:19:56.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe94d47c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe948111dd0 con 0x7fe94810c150 2024-09-16T00:19:56.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe94d47c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481120d0 con 0x7fe948103640 2024-09-16T00:19:56.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe946ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe948102d20 0x7fe948198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe946ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe948102d20 0x7fe948198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46032/0 (socket says 172.21.15.33:46032) 2024-09-16T00:19:56.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe946ffd640 1 -- 172.21.15.33:0/2372085112 learned_addr learned my addr 172.21.15.33:0/2372085112 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe9477fe640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe94810c150 0x7fe94819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe9467fc640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe948103640 0x7fe9481989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe946ffd640 1 -- 172.21.15.33:0/2372085112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe948103640 msgr2=0x7fe9481989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe946ffd640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe948103640 0x7fe9481989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe946ffd640 1 -- 172.21.15.33:0/2372085112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe94810c150 msgr2=0x7fe94819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe946ffd640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe94810c150 0x7fe94819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe946ffd640 1 -- 172.21.15.33:0/2372085112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe94819d430 con 0x7fe948102d20 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe9477fe640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe94810c150 0x7fe94819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe946ffd640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe948102d20 0x7fe948198480 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe93800c890 tx=0x7fe93800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe92ffff640 1 -- 172.21.15.33:0/2372085112 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938004220 con 0x7fe948102d20 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe92ffff640 1 -- 172.21.15.33:0/2372085112 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9380043c0 con 0x7fe948102d20 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe92ffff640 1 -- 172.21.15.33:0/2372085112 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938004a00 con 0x7fe948102d20 2024-09-16T00:19:56.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe94d47c640 1 -- 172.21.15.33:0/2372085112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe94819d720 con 0x7fe948102d20 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.434+0000 7fe94d47c640 1 -- 172.21.15.33:0/2372085112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9481a5000 con 0x7fe948102d20 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.437+0000 7fe92ffff640 1 -- 172.21.15.33:0/2372085112 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe9380079b0 con 0x7fe948102d20 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.437+0000 7fe94f855740 1 Processor -- start 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.437+0000 7fe94f855740 1 -- start start 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.437+0000 7fe94f855740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0a4a0 0x55ea04d0a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.437+0000 7fe94f855740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0adc0 0x55ea04d15140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.437+0000 7fe94f855740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d17a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.437+0000 7fe94f855740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ea04c702e0 con 0x55ea04d15680 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.437+0000 7fe94f855740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ea04c17ce0 con 0x55ea04d0adc0 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.437+0000 7fe94f855740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ea04c6c2a0 con 0x55ea04d0a4a0 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.437+0000 7fe9467fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d17a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe9467fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d17a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46048/0 (socket says 172.21.15.33:46048) 2024-09-16T00:19:56.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe9467fc640 1 -- 172.21.15.33:0/772088738 learned_addr learned my addr 172.21.15.33:0/772088738 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe92ffff640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9300a32c0 0x7fe9300a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe946ffd640 1 --2- 172.21.15.33:0/772088738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0adc0 0x55ea04d15140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe9477fe640 1 --2- 172.21.15.33:0/772088738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0a4a0 0x55ea04d0a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe9477fe640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9300a32c0 0x7fe9300a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe9467fc640 1 -- 172.21.15.33:0/772088738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0a4a0 msgr2=0x55ea04d0a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe9467fc640 1 --2- 172.21.15.33:0/772088738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0a4a0 0x55ea04d0a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe9467fc640 1 -- 172.21.15.33:0/772088738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0adc0 msgr2=0x55ea04d15140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe9467fc640 1 --2- 172.21.15.33:0/772088738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0adc0 0x55ea04d15140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe9467fc640 1 -- 172.21.15.33:0/772088738 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea04bfc440 con 0x55ea04d15680 2024-09-16T00:19:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe946ffd640 1 --2- 172.21.15.33:0/772088738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0adc0 0x55ea04d15140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe92ffff640 1 -- 172.21.15.33:0/2372085112 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe9380cfb00 con 0x7fe948102d20 2024-09-16T00:19:56.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe9467fc640 1 --2- 172.21.15.33:0/772088738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d17a70 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009900 tx=0x7fe93c032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48fb98dfd82b792 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe9477fe640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9300a32c0 0x7fe9300a5780 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe940005c40 tx=0x7fe940009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe92d7fa640 1 -- 172.21.15.33:0/772088738 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c035070 con 0x55ea04d15680 2024-09-16T00:19:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe92d7fa640 1 -- 172.21.15.33:0/772088738 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe93c032e20 con 0x55ea04d15680 2024-09-16T00:19:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.438+0000 7fe92d7fa640 1 -- 172.21.15.33:0/772088738 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c005610 con 0x55ea04d15680 2024-09-16T00:19:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.439+0000 7fe94f855740 1 -- 172.21.15.33:0/772088738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 msgr2=0x55ea04d17a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.439+0000 7fe94f855740 1 --2- 172.21.15.33:0/772088738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d17a70 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009900 tx=0x7fe93c032c30 comp rx=0 tx=0).stop 2024-09-16T00:19:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.439+0000 7fe94f855740 1 -- 172.21.15.33:0/772088738 shutdown_connections 2024-09-16T00:19:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.439+0000 7fe94f855740 1 --2- 172.21.15.33:0/772088738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d17a70 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.439+0000 7fe94f855740 1 --2- 172.21.15.33:0/772088738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0adc0 0x55ea04d15140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.439+0000 7fe94f855740 1 --2- 172.21.15.33:0/772088738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0a4a0 0x55ea04d0a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.439+0000 7fe94f855740 1 -- 172.21.15.33:0/772088738 >> 172.21.15.33:0/772088738 conn(0x55ea04d06a30 msgr2=0x55ea04d06e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:56.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.439+0000 7fe94f855740 1 -- 172.21.15.33:0/772088738 shutdown_connections 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.439+0000 7fe94f855740 1 -- 172.21.15.33:0/772088738 wait complete. 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.439+0000 7fe94f855740 1 -- start start 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe94f855740 1 Processor -- start 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe94f855740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0a4a0 0x55ea04d14f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe94f855740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0adc0 0x55ea04d12240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe94f855740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d12780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe94f855740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ea04d18250 con 0x55ea04d15680 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe94f855740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ea04c2b050 con 0x55ea04d0a4a0 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe94f855740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ea04d183d0 con 0x55ea04d0adc0 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe94f855740 10 client.0 _ll_register_callbacks cb 0x55ea04c6b160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe9477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d12780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe9477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d12780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46056/0 (socket says 172.21.15.33:46056) 2024-09-16T00:19:56.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe9477fe640 1 -- 172.21.15.33:0/2899981979 learned_addr learned my addr 172.21.15.33:0/2899981979 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe9467fc640 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0adc0 0x55ea04d12240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe946ffd640 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0a4a0 0x55ea04d14f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe9477fe640 1 -- 172.21.15.33:0/2899981979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0adc0 msgr2=0x55ea04d12240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe9477fe640 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0adc0 0x55ea04d12240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe9477fe640 1 -- 172.21.15.33:0/2899981979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0a4a0 msgr2=0x55ea04d14f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe9477fe640 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0a4a0 0x55ea04d14f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe9477fe640 1 -- 172.21.15.33:0/2899981979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea049e13c0 con 0x55ea04d15680 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe9467fc640 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0adc0 0x55ea04d12240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe946ffd640 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0a4a0 0x55ea04d14f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe92aff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.440+0000 7fe9477fe640 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d12780 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe940013010 tx=0x7fe940004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead14ee47be37e8d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe92aff5640 1 -- 172.21.15.33:0/2899981979 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940018070 con 0x55ea04d15680 2024-09-16T00:19:56.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe92aff5640 1 -- 172.21.15.33:0/2899981979 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe940028670 con 0x55ea04d15680 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe94f855740 1 -- 172.21.15.33:0/2899981979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe93c032e20 con 0x55ea04d15680 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe92aff5640 1 -- 172.21.15.33:0/2899981979 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe940028870 con 0x55ea04d15680 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe94f855740 10 client.15855 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe94f855740 1 -- 172.21.15.33:0/2899981979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea04d23c30 con 0x55ea04d15680 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe92aff5640 1 -- 172.21.15.33:0/2899981979 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe93c032e20 con 0x55ea04d15680 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe92aff5640 1 -- 172.21.15.33:0/2899981979 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe94001f670 con 0x55ea04d15680 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe94f855740 10 client.15855 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe94f855740 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe94f855740 20 client.15855 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe94f855740 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ea04d251c0 0x55ea04d27660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe94f855740 4 client.15855 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.441+0000 7fe94f855740 1 -- 172.21.15.33:0/2899981979 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea04c7d850 con 0x55ea04d251c0 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.442+0000 7fe946ffd640 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ea04d251c0 0x55ea04d27660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.442+0000 7fe946ffd640 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ea04d251c0 0x55ea04d27660 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7922e2d15e28898f server_cookie=98802eb02cf4387e in_seq=0 out_seq=0 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.442+0000 7fe92aff5640 10 client.15855 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe92aff5640 1 -- 172.21.15.33:0/2899981979 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ea04c7d850 con 0x55ea04d251c0 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe92aff5640 10 client.15855 handle_command_reply: tid=0 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe94f855740 1 client.15855 shutdown 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe94f855740 1 -- 172.21.15.33:0/2899981979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 msgr2=0x55ea04d12780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe94f855740 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d12780 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe940013010 tx=0x7fe940004a40 comp rx=0 tx=0).stop 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe94f855740 1 -- 172.21.15.33:0/2899981979 shutdown_connections 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe94f855740 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ea04d251c0 0x55ea04d27660 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe94f855740 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d12780 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe94f855740 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0adc0 0x55ea04d12240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe92aff5640 0 client.15855 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe94f855740 1 --2- 172.21.15.33:0/2899981979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0a4a0 0x55ea04d14f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe94f855740 1 -- 172.21.15.33:0/2899981979 >> 172.21.15.33:0/2899981979 conn(0x55ea04d06a30 msgr2=0x55ea04d0c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:56.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe94f855740 1 -- 172.21.15.33:0/2899981979 shutdown_connections 2024-09-16T00:19:56.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe94f855740 1 -- 172.21.15.33:0/2899981979 wait complete. 2024-09-16T00:19:56.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.443+0000 7fe94f855740 20 client.15855 trim_cache size 0 max 16384 2024-09-16T00:19:56.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.460+0000 7fe94f855740 1 Processor -- start 2024-09-16T00:19:56.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.460+0000 7fe94f855740 1 -- start start 2024-09-16T00:19:56.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.460+0000 7fe94f855740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0a4a0 0x55ea04d214b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.460+0000 7fe94f855740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d0adc0 0x55ea04d10b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.460+0000 7fe94f855740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d15680 0x55ea04d2cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.460+0000 7fe94f855740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481af7f0 con 0x55ea04d0adc0 2024-09-16T00:19:56.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.460+0000 7fe94f855740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481ad0c0 con 0x55ea04d15680 2024-09-16T00:19:56.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.460+0000 7fe94f855740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481aaf10 con 0x55ea04d0a4a0 2024-09-16T00:19:56.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe9467fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d0adc0 0x55ea04d10b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe946ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0a4a0 0x55ea04d214b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe9467fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d0adc0 0x55ea04d10b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46072/0 (socket says 172.21.15.33:46072) 2024-09-16T00:19:56.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe9467fc640 1 -- 172.21.15.33:0/3795751821 learned_addr learned my addr 172.21.15.33:0/3795751821 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:56.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe9477fe640 1 --2- 172.21.15.33:0/3795751821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d15680 0x55ea04d2cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe9467fc640 1 -- 172.21.15.33:0/3795751821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0a4a0 msgr2=0x55ea04d214b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:56.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe9467fc640 1 --2- 172.21.15.33:0/3795751821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0a4a0 0x55ea04d214b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe9467fc640 1 -- 172.21.15.33:0/3795751821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d15680 msgr2=0x55ea04d2cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:56.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe9467fc640 1 --2- 172.21.15.33:0/3795751821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d15680 0x55ea04d2cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe9467fc640 1 -- 172.21.15.33:0/3795751821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea04d111f0 con 0x55ea04d0adc0 2024-09-16T00:19:56.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe9467fc640 1 --2- 172.21.15.33:0/3795751821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d0adc0 0x55ea04d10b10 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe93c031550 tx=0x7fe93c02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d04cfb779af23d32 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:56.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe92cff9640 1 -- 172.21.15.33:0/3795751821 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c035070 con 0x55ea04d0adc0 2024-09-16T00:19:56.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe92cff9640 1 -- 172.21.15.33:0/3795751821 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe93c04f070 con 0x55ea04d0adc0 2024-09-16T00:19:56.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.461+0000 7fe92cff9640 1 -- 172.21.15.33:0/3795751821 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c0035f0 con 0x55ea04d0adc0 2024-09-16T00:19:56.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.462+0000 7fe94f855740 1 -- 172.21.15.33:0/3795751821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d0adc0 msgr2=0x55ea04d10b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:56.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.462+0000 7fe94f855740 1 --2- 172.21.15.33:0/3795751821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d0adc0 0x55ea04d10b10 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe93c031550 tx=0x7fe93c02a480 comp rx=0 tx=0).stop 2024-09-16T00:19:56.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.462+0000 7fe94f855740 1 -- 172.21.15.33:0/3795751821 shutdown_connections 2024-09-16T00:19:56.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.462+0000 7fe94f855740 1 --2- 172.21.15.33:0/3795751821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d15680 0x55ea04d2cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.462+0000 7fe94f855740 1 --2- 172.21.15.33:0/3795751821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d0adc0 0x55ea04d10b10 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.462+0000 7fe94f855740 1 --2- 172.21.15.33:0/3795751821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0a4a0 0x55ea04d214b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.462+0000 7fe94f855740 1 -- 172.21.15.33:0/3795751821 >> 172.21.15.33:0/3795751821 conn(0x55ea04d06a30 msgr2=0x55ea04d0d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:56.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.462+0000 7fe94f855740 1 -- 172.21.15.33:0/3795751821 shutdown_connections 2024-09-16T00:19:56.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.462+0000 7fe94f855740 1 -- 172.21.15.33:0/3795751821 wait complete. 2024-09-16T00:19:56.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.462+0000 7fe94f855740 1 -- start start 2024-09-16T00:19:56.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.462+0000 7fe94f855740 1 Processor -- start 2024-09-16T00:19:56.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe94f855740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0a4a0 0x55ea04d19390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe94f855740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0adc0 0x55ea04d198d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe94f855740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d19e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe94f855740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481ab580 con 0x55ea04d15680 2024-09-16T00:19:56.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe94f855740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481ac9f0 con 0x55ea04d0a4a0 2024-09-16T00:19:56.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe94f855740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481abc50 con 0x55ea04d0adc0 2024-09-16T00:19:56.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe94f855740 10 client.0 _ll_register_callbacks cb 0x7fe940009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:19:56.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe9477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d19e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe9477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d19e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46074/0 (socket says 172.21.15.33:46074) 2024-09-16T00:19:56.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe9477fe640 1 -- 172.21.15.33:0/3403486897 learned_addr learned my addr 172.21.15.33:0/3403486897 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:19:56.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe946ffd640 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0a4a0 0x55ea04d19390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe9467fc640 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0adc0 0x55ea04d198d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe9477fe640 1 -- 172.21.15.33:0/3403486897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0adc0 msgr2=0x55ea04d198d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:56.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe9477fe640 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0adc0 0x55ea04d198d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe9477fe640 1 -- 172.21.15.33:0/3403486897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0a4a0 msgr2=0x55ea04d19390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:19:56.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe9477fe640 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0a4a0 0x55ea04d19390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe9477fe640 1 -- 172.21.15.33:0/3403486897 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea04d15300 con 0x55ea04d15680 2024-09-16T00:19:56.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe9467fc640 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0adc0 0x55ea04d198d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:19:56.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe9477fe640 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d19e10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe940011010 tx=0x7fe94000fdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e55e4d93924662 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:19:56.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.463+0000 7fe92aff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:19:56.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe92aff5640 1 -- 172.21.15.33:0/3403486897 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940018040 con 0x55ea04d15680 2024-09-16T00:19:56.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe92aff5640 1 -- 172.21.15.33:0/3403486897 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe94001f640 con 0x55ea04d15680 2024-09-16T00:19:56.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe94f855740 1 -- 172.21.15.33:0/3403486897 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe93c04f070 con 0x55ea04d15680 2024-09-16T00:19:56.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe92aff5640 1 -- 172.21.15.33:0/3403486897 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe94000e960 con 0x55ea04d15680 2024-09-16T00:19:56.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe94f855740 10 client.15861 fetch_fsmap learned FSMap version 28 2024-09-16T00:19:56.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe94f855740 1 -- 172.21.15.33:0/3403486897 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea04d900d0 con 0x55ea04d15680 2024-09-16T00:19:56.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe92aff5640 1 -- 172.21.15.33:0/3403486897 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe93c04f070 con 0x55ea04d15680 2024-09-16T00:19:56.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe92aff5640 1 -- 172.21.15.33:0/3403486897 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe94001c420 con 0x55ea04d15680 2024-09-16T00:19:56.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe94f855740 10 client.15861 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:19:56.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe94f855740 10 client.15861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:19:56.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe94f855740 20 client.15861 populate_metadata read hostname 'smithi033' 2024-09-16T00:19:56.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe94f855740 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ea04d908d0 0x55ea04d92cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:19:56.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe94f855740 4 client.15861 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:19:56.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.464+0000 7fe94f855740 1 -- 172.21.15.33:0/3403486897 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ea04c77d60 con 0x55ea04d908d0 2024-09-16T00:19:56.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe946ffd640 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ea04d908d0 0x55ea04d92cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:19:56.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe946ffd640 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ea04d908d0 0x55ea04d92cf0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b54bded23af6e781 server_cookie=a42db7e75a55135e in_seq=0 out_seq=0 2024-09-16T00:19:56.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe92aff5640 10 client.15861 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:56.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe92aff5640 1 -- 172.21.15.33:0/3403486897 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ea04c77d60 con 0x55ea04d908d0 2024-09-16T00:19:56.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe92aff5640 10 client.15861 handle_command_reply: tid=0 2024-09-16T00:19:56.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe94f855740 1 client.15861 shutdown 2024-09-16T00:19:56.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe94f855740 1 -- 172.21.15.33:0/3403486897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 msgr2=0x55ea04d19e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:56.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe94f855740 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d19e10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe940011010 tx=0x7fe94000fdc0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe94f855740 1 -- 172.21.15.33:0/3403486897 shutdown_connections 2024-09-16T00:19:56.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe94f855740 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ea04d908d0 0x55ea04d92cf0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe94f855740 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ea04d15680 0x55ea04d19e10 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe94f855740 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ea04d0adc0 0x55ea04d198d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe94f855740 1 --2- 172.21.15.33:0/3403486897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ea04d0a4a0 0x55ea04d19390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe94f855740 1 -- 172.21.15.33:0/3403486897 >> 172.21.15.33:0/3403486897 conn(0x55ea04d06a30 msgr2=0x55ea04d13aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:56.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe92aff5640 0 client.15861 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:19:56.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.465+0000 7fe94f855740 1 -- 172.21.15.33:0/3403486897 shutdown_connections 2024-09-16T00:19:56.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94f855740 1 -- 172.21.15.33:0/3403486897 wait complete. 2024-09-16T00:19:56.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94f855740 20 client.15861 trim_cache size 0 max 16384 2024-09-16T00:19:56.476 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:19:56.476 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:19:56.476 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:19:56.476 INFO:teuthology.orchestra.run.smithi033.stdout: "265eb9c6-e85a-4ebb-b79d-0cb091c77e73": { 2024-09-16T00:19:56.476 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:19:56.476 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "265eb9c6-e85a-4ebb-b79d-0cb091c77e73", 2024-09-16T00:19:56.476 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:19:56.476 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:19:56.476 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:19:56.477 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:19:56.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94d47c640 1 -- 172.21.15.33:0/2372085112 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9300a32c0 msgr2=0x7fe9300a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:56.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94d47c640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9300a32c0 0x7fe9300a5780 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe940005c40 tx=0x7fe940009500 comp rx=0 tx=0).stop 2024-09-16T00:19:56.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94d47c640 1 -- 172.21.15.33:0/2372085112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe948102d20 msgr2=0x7fe948198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:19:56.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94d47c640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe948102d20 0x7fe948198480 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe93800c890 tx=0x7fe93800cd50 comp rx=0 tx=0).stop 2024-09-16T00:19:56.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94d47c640 1 -- 172.21.15.33:0/2372085112 shutdown_connections 2024-09-16T00:19:56.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94d47c640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9300a32c0 0x7fe9300a5780 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94d47c640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe94810c150 0x7fe94819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94d47c640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe948103640 0x7fe9481989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94d47c640 1 --2- 172.21.15.33:0/2372085112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe948102d20 0x7fe948198480 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:19:56.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94d47c640 1 -- 172.21.15.33:0/2372085112 >> 172.21.15.33:0/2372085112 conn(0x7fe9480fc780 msgr2=0x7fe9480fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:19:56.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94d47c640 1 -- 172.21.15.33:0/2372085112 shutdown_connections 2024-09-16T00:19:56.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:19:56.466+0000 7fe94d47c640 1 -- 172.21.15.33:0/2372085112 wait complete. 2024-09-16T00:19:56.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.63 MB/sec warmup 64 sec latency 10832.070 ms 2024-09-16T00:19:57.261 INFO:tasks.cephfs.filesystem:scrub status for tag:265eb9c6-e85a-4ebb-b79d-0cb091c77e73 - {'path': '/', 'tag': '265eb9c6-e85a-4ebb-b79d-0cb091c77e73', 'options': 'recursive,force'} 2024-09-16T00:19:57.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.57 MB/sec warmup 65 sec latency 11832.130 ms 2024-09-16T00:19:57.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:57 smithi033 ceph-mon[31243]: pgmap v434: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 469 KiB/s wr, 32 op/s 2024-09-16T00:19:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:57 smithi083 ceph-mon[35242]: pgmap v434: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 469 KiB/s wr, 32 op/s 2024-09-16T00:19:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:57 smithi060 ceph-mon[35185]: pgmap v434: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 469 KiB/s wr, 32 op/s 2024-09-16T00:19:58.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.52 MB/sec warmup 66 sec latency 12832.165 ms 2024-09-16T00:19:59.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.46 MB/sec warmup 67 sec latency 13832.206 ms 2024-09-16T00:19:59.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:19:59 smithi033 ceph-mon[31243]: pgmap v435: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 494 KiB/s wr, 25 op/s 2024-09-16T00:19:59.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:19:59 smithi083 ceph-mon[35242]: pgmap v435: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 494 KiB/s wr, 25 op/s 2024-09-16T00:19:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:19:59 smithi060 ceph-mon[35185]: pgmap v435: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 494 KiB/s wr, 25 op/s 2024-09-16T00:20:00.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.41 MB/sec warmup 68 sec latency 14832.250 ms 2024-09-16T00:20:00.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:00 smithi033 ceph-mon[31243]: overall HEALTH_OK 2024-09-16T00:20:00.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:00 smithi083 ceph-mon[35242]: overall HEALTH_OK 2024-09-16T00:20:00.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:00 smithi060 ceph-mon[35185]: overall HEALTH_OK 2024-09-16T00:20:01.531 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:20:01.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.36 MB/sec warmup 69 sec latency 15832.284 ms 2024-09-16T00:20:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:01 smithi060 ceph-mon[35185]: pgmap v436: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 524 KiB/s wr, 28 op/s 2024-09-16T00:20:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:01 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:20:01.806 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:01.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:01 smithi033 ceph-mon[31243]: pgmap v436: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 524 KiB/s wr, 28 op/s 2024-09-16T00:20:01.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:01 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:20:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:01 smithi083 ceph-mon[35242]: pgmap v436: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 524 KiB/s wr, 28 op/s 2024-09-16T00:20:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:01 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:20:02.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.299+0000 7fab0a77f640 1 -- 172.21.15.33:0/4197709412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab04102570 msgr2=0x7fab041029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:02.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.299+0000 7fab0a77f640 1 --2- 172.21.15.33:0/4197709412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab04102570 0x7fab041029b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7faaf8009900 tx=0x7faaf802a480 comp rx=0 tx=0).stop 2024-09-16T00:20:02.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.299+0000 7fab0a77f640 1 -- 172.21.15.33:0/4197709412 shutdown_connections 2024-09-16T00:20:02.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.299+0000 7fab0a77f640 1 --2- 172.21.15.33:0/4197709412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab04102ef0 0x7fab0410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:02.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.299+0000 7fab0a77f640 1 --2- 172.21.15.33:0/4197709412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab04102570 0x7fab041029b0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:02.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.299+0000 7fab0a77f640 1 --2- 172.21.15.33:0/4197709412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab04100e20 0x7fab04101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:02.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.299+0000 7fab0a77f640 1 -- 172.21.15.33:0/4197709412 >> 172.21.15.33:0/4197709412 conn(0x7fab040fc7d0 msgr2=0x7fab040febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:02.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.299+0000 7fab0a77f640 1 -- 172.21.15.33:0/4197709412 shutdown_connections 2024-09-16T00:20:02.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.299+0000 7fab0a77f640 1 -- 172.21.15.33:0/4197709412 wait complete. 2024-09-16T00:20:02.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab0a77f640 1 Processor -- start 2024-09-16T00:20:02.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab0a77f640 1 -- start start 2024-09-16T00:20:02.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab0a77f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab04100e20 0x7fab041993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:02.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab0a77f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab04102570 0x7fab04199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:02.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab0a77f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab04102ef0 0x7fab041a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:02.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab0a77f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab04110f80 con 0x7fab04100e20 2024-09-16T00:20:02.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab0a77f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fab04110e00 con 0x7fab04102ef0 2024-09-16T00:20:02.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab0a77f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fab04111100 con 0x7fab04102570 2024-09-16T00:20:02.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab03fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab04100e20 0x7fab041993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:02.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab03fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab04100e20 0x7fab041993d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48056/0 (socket says 172.21.15.33:48056) 2024-09-16T00:20:02.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab03fff640 1 -- 172.21.15.33:0/3057426023 learned_addr learned my addr 172.21.15.33:0/3057426023 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:02.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab08d1c640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab04102ef0 0x7fab041a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:02.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.300+0000 7fab037fe640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab04102570 0x7fab04199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:02.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.301+0000 7fab03fff640 1 -- 172.21.15.33:0/3057426023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab04102570 msgr2=0x7fab04199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:02.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.301+0000 7fab03fff640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab04102570 0x7fab04199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:02.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.301+0000 7fab03fff640 1 -- 172.21.15.33:0/3057426023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab04102ef0 msgr2=0x7fab041a4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:02.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.301+0000 7fab03fff640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab04102ef0 0x7fab041a4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:02.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.301+0000 7fab03fff640 1 -- 172.21.15.33:0/3057426023 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab041a4990 con 0x7fab04100e20 2024-09-16T00:20:02.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.301+0000 7fab037fe640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab04102570 0x7fab04199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:20:02.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.301+0000 7fab03fff640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab04100e20 0x7fab041993d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7faaf400d770 tx=0x7faaf400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:02.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.301+0000 7fab017fa640 1 -- 172.21.15.33:0/3057426023 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4017070 con 0x7fab04100e20 2024-09-16T00:20:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.301+0000 7fab017fa640 1 -- 172.21.15.33:0/3057426023 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf4005500 con 0x7fab04100e20 2024-09-16T00:20:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.301+0000 7fab0a77f640 1 -- 172.21.15.33:0/3057426023 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab041a4b60 con 0x7fab04100e20 2024-09-16T00:20:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.301+0000 7fab017fa640 1 -- 172.21.15.33:0/3057426023 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4004e00 con 0x7fab04100e20 2024-09-16T00:20:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.301+0000 7fab0a77f640 1 -- 172.21.15.33:0/3057426023 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab041a5020 con 0x7fab04100e20 2024-09-16T00:20:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.302+0000 7fab0a77f640 1 -- 172.21.15.33:0/3057426023 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab04101200 con 0x7fab04100e20 2024-09-16T00:20:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.303+0000 7fab017fa640 1 -- 172.21.15.33:0/3057426023 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7faaf4004fa0 con 0x7fab04100e20 2024-09-16T00:20:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.304+0000 7fab017fa640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faaec0a3240 0x7faaec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.304+0000 7fab017fa640 1 -- 172.21.15.33:0/3057426023 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7faaf40cf870 con 0x7fab04100e20 2024-09-16T00:20:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.304+0000 7fab037fe640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faaec0a3240 0x7faaec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.305+0000 7fab037fe640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faaec0a3240 0x7faaec0a5700 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fab0406aac0 tx=0x7faaf8005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.308+0000 7fab017fa640 1 -- 172.21.15.33:0/3057426023 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7faaf409c0b0 con 0x7fab04100e20 2024-09-16T00:20:02.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.540+0000 7fab0a77f640 1 -- 172.21.15.33:0/3057426023 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab04109090 con 0x7fab04100e20 2024-09-16T00:20:02.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.541+0000 7fab017fa640 1 -- 172.21.15.33:0/3057426023 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11906 (secure 0 0 0) 0x7faaf409b9b0 con 0x7fab04100e20 2024-09-16T00:20:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:20:02.543 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"btime":"2024-09-16T00:19:02:115556+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:19:01.460559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:20:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 28 2024-09-16T00:20:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.543+0000 7fab0a77f640 1 -- 172.21.15.33:0/3057426023 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faaec0a3240 msgr2=0x7faaec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.543+0000 7fab0a77f640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faaec0a3240 0x7faaec0a5700 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fab0406aac0 tx=0x7faaf8005a20 comp rx=0 tx=0).stop 2024-09-16T00:20:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.543+0000 7fab0a77f640 1 -- 172.21.15.33:0/3057426023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab04100e20 msgr2=0x7fab041993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.543+0000 7fab0a77f640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab04100e20 0x7fab041993d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7faaf400d770 tx=0x7faaf400dc30 comp rx=0 tx=0).stop 2024-09-16T00:20:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.543+0000 7fab0a77f640 1 -- 172.21.15.33:0/3057426023 shutdown_connections 2024-09-16T00:20:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.543+0000 7fab0a77f640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faaec0a3240 0x7faaec0a5700 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.543+0000 7fab0a77f640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab04102ef0 0x7fab041a4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.543+0000 7fab0a77f640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab04102570 0x7fab04199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.543+0000 7fab0a77f640 1 --2- 172.21.15.33:0/3057426023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab04100e20 0x7fab041993d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.543+0000 7fab0a77f640 1 -- 172.21.15.33:0/3057426023 >> 172.21.15.33:0/3057426023 conn(0x7fab040fc7d0 msgr2=0x7fab040febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.543+0000 7fab0a77f640 1 -- 172.21.15.33:0/3057426023 shutdown_connections 2024-09-16T00:20:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:02.543+0000 7fab0a77f640 1 -- 172.21.15.33:0/3057426023 wait complete. 2024-09-16T00:20:02.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.32 MB/sec warmup 70 sec latency 16832.315 ms 2024-09-16T00:20:02.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:02 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:20:02.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:02 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:20:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:02 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:20:03.343 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 5dd683 --release --await' 2024-09-16T00:20:03.343 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 5dd683 --release --await 2024-09-16T00:20:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:03 smithi033 ceph-mon[31243]: pgmap v437: 129 pgs: 129 active+clean; 36 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 606 KiB/s wr, 26 op/s 2024-09-16T00:20:03.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:03 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3057426023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:20:03.610 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:03.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 39970 3.27 MB/sec warmup 71 sec latency 17832.400 ms 2024-09-16T00:20:03.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:03 smithi083 ceph-mon[35242]: pgmap v437: 129 pgs: 129 active+clean; 36 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 606 KiB/s wr, 26 op/s 2024-09-16T00:20:03.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:03 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3057426023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:20:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:03 smithi060 ceph-mon[35185]: pgmap v437: 129 pgs: 129 active+clean; 36 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 606 KiB/s wr, 26 op/s 2024-09-16T00:20:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:03 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3057426023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:20:04.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.122+0000 7f6f63a63640 1 -- 172.21.15.33:0/2837201374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f5c106f10 msgr2=0x7f6f5c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.122+0000 7f6f63a63640 1 --2- 172.21.15.33:0/2837201374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f5c106f10 0x7f6f5c10ea90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009ca0 tx=0x7f6f5402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.123+0000 7f6f63a63640 1 -- 172.21.15.33:0/2837201374 shutdown_connections 2024-09-16T00:20:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.123+0000 7f6f63a63640 1 --2- 172.21.15.33:0/2837201374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f5c106f10 0x7f6f5c10ea90 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.123+0000 7f6f63a63640 1 --2- 172.21.15.33:0/2837201374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c102790 0x7f6f5c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.123+0000 7f6f63a63640 1 --2- 172.21.15.33:0/2837201374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f5c101de0 0x7f6f5c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.123+0000 7f6f63a63640 1 -- 172.21.15.33:0/2837201374 >> 172.21.15.33:0/2837201374 conn(0x7f6f5c0fd8c0 msgr2=0x7f6f5c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.123+0000 7f6f63a63640 1 -- 172.21.15.33:0/2837201374 shutdown_connections 2024-09-16T00:20:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.123+0000 7f6f63a63640 1 -- 172.21.15.33:0/2837201374 wait complete. 2024-09-16T00:20:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.123+0000 7f6f63a63640 1 Processor -- start 2024-09-16T00:20:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.123+0000 7f6f63a63640 1 -- start start 2024-09-16T00:20:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f63a63640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c101de0 0x7f6f5c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f63a63640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f5c102790 0x7f6f5c198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f63a63640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f5c106f10 0x7f6f5c19ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f63a63640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c111f70 con 0x7f6f5c106f10 2024-09-16T00:20:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f63a63640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c111df0 con 0x7f6f5c101de0 2024-09-16T00:20:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f63a63640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c1120f0 con 0x7f6f5c102790 2024-09-16T00:20:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f62000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f5c106f10 0x7f6f5c19ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f617ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c101de0 0x7f6f5c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f62000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f5c106f10 0x7f6f5c19ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48070/0 (socket says 172.21.15.33:48070) 2024-09-16T00:20:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f62000640 1 -- 172.21.15.33:0/3326987749 learned_addr learned my addr 172.21.15.33:0/3326987749 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f60ffe640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f5c102790 0x7f6f5c198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f62000640 1 -- 172.21.15.33:0/3326987749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f5c102790 msgr2=0x7f6f5c198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f62000640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f5c102790 0x7f6f5c198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f62000640 1 -- 172.21.15.33:0/3326987749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c101de0 msgr2=0x7f6f5c1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f62000640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c101de0 0x7f6f5c1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f62000640 1 -- 172.21.15.33:0/3326987749 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f5c19d3a0 con 0x7f6f5c106f10 2024-09-16T00:20:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f617ff640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c101de0 0x7f6f5c1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.124+0000 7f6f62000640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f5c106f10 0x7f6f5c19ccc0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6f54031a20 tx=0x7f6f54031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.125+0000 7f6f527fc640 1 -- 172.21.15.33:0/3326987749 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f54003c30 con 0x7f6f5c106f10 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.125+0000 7f6f527fc640 1 -- 172.21.15.33:0/3326987749 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f54003dd0 con 0x7f6f5c106f10 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.125+0000 7f6f63a63640 1 -- 172.21.15.33:0/3326987749 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f5c19d630 con 0x7f6f5c106f10 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.125+0000 7f6f527fc640 1 -- 172.21.15.33:0/3326987749 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5402fcf0 con 0x7f6f5c106f10 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.125+0000 7f6f63a63640 1 -- 172.21.15.33:0/3326987749 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f5c1a4f70 con 0x7f6f5c106f10 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.127+0000 7f6f527fc640 1 -- 172.21.15.33:0/3326987749 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6f54004050 con 0x7f6f5c106f10 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f65e3c740 1 Processor -- start 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f65e3c740 1 -- start start 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f65e3c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598060 0x55a0fc598440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f527fc640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6f480a3240 0x7f6f480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f65e3c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc598980 0x55a0fc5a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f65e3c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f65e3c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a0fc4fd620 con 0x55a0fc5a3240 2024-09-16T00:20:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f65e3c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a0fc4a4e50 con 0x55a0fc598060 2024-09-16T00:20:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f65e3c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a0fc4f95e0 con 0x55a0fc598980 2024-09-16T00:20:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f62000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f62000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48084/0 (socket says 172.21.15.33:48084) 2024-09-16T00:20:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f62000640 1 -- 172.21.15.33:0/2762853351 learned_addr learned my addr 172.21.15.33:0/2762853351 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f617ff640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6f480a3240 0x7f6f480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f60ffe640 1 --2- 172.21.15.33:0/2762853351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc598980 0x55a0fc5a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f60ffe640 1 --2- 172.21.15.33:0/2762853351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598060 0x55a0fc598440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f527fc640 1 -- 172.21.15.33:0/3326987749 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6f540f1140 con 0x7f6f5c106f10 2024-09-16T00:20:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f62000640 1 -- 172.21.15.33:0/2762853351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc598980 msgr2=0x55a0fc5a2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f62000640 1 --2- 172.21.15.33:0/2762853351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc598980 0x55a0fc5a2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f62000640 1 -- 172.21.15.33:0/2762853351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598060 msgr2=0x55a0fc598440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f62000640 1 --2- 172.21.15.33:0/2762853351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598060 0x55a0fc598440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.128+0000 7f6f62000640 1 -- 172.21.15.33:0/2762853351 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0fc495d40 con 0x55a0fc5a3240 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.129+0000 7f6f617ff640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6f480a3240 0x7f6f480a5700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6f58005fd0 tx=0x7f6f5800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.129+0000 7f6f62000640 1 --2- 172.21.15.33:0/2762853351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a5630 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009a30 tx=0x7f6f5402f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3534b002574ce219 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.129+0000 7f6f477fe640 1 -- 172.21.15.33:0/2762853351 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f540bcc70 con 0x55a0fc5a3240 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.129+0000 7f6f477fe640 1 -- 172.21.15.33:0/2762853351 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f54036070 con 0x55a0fc5a3240 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.129+0000 7f6f477fe640 1 -- 172.21.15.33:0/2762853351 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f540ca900 con 0x55a0fc5a3240 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.129+0000 7f6f65e3c740 1 -- 172.21.15.33:0/2762853351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 msgr2=0x55a0fc5a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.129+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/2762853351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a5630 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009a30 tx=0x7f6f5402f790 comp rx=0 tx=0).stop 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 -- 172.21.15.33:0/2762853351 shutdown_connections 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/2762853351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a5630 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/2762853351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc598980 0x55a0fc5a2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/2762853351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598060 0x55a0fc598440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 -- 172.21.15.33:0/2762853351 >> 172.21.15.33:0/2762853351 conn(0x55a0fc5945f0 msgr2=0x55a0fc5949d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 -- 172.21.15.33:0/2762853351 shutdown_connections 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 -- 172.21.15.33:0/2762853351 wait complete. 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 -- start start 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 Processor -- start 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc598060 0x55a0fc5a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598980 0x55a0fc5a0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a0fc4fd620 con 0x55a0fc5a3240 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a0fc485f10 con 0x55a0fc598980 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a0fc5a5bd0 con 0x55a0fc598060 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.130+0000 7f6f65e3c740 10 client.0 _ll_register_callbacks cb 0x55a0fc4a9db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f62000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f62000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48098/0 (socket says 172.21.15.33:48098) 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f62000640 1 -- 172.21.15.33:0/318522698 learned_addr learned my addr 172.21.15.33:0/318522698 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f617ff640 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc598060 0x55a0fc5a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f60ffe640 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598980 0x55a0fc5a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f62000640 1 -- 172.21.15.33:0/318522698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc598060 msgr2=0x55a0fc5a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f62000640 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc598060 0x55a0fc5a01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f62000640 1 -- 172.21.15.33:0/318522698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598980 msgr2=0x55a0fc5a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f62000640 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598980 0x55a0fc5a0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f62000640 1 -- 172.21.15.33:0/318522698 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0fc20c580 con 0x55a0fc5a3240 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f617ff640 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc598060 0x55a0fc5a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f60ffe640 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598980 0x55a0fc5a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f62000640 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a0c70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6f540b9770 tx=0x7f6f540bcd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc7dd9e583200be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:20:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f44ff9640 1 -- 172.21.15.33:0/318522698 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f540bda10 con 0x55a0fc5a3240 2024-09-16T00:20:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f44ff9640 1 -- 172.21.15.33:0/318522698 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6f540b9980 con 0x55a0fc5a3240 2024-09-16T00:20:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f65e3c740 1 -- 172.21.15.33:0/318522698 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f54036070 con 0x55a0fc5a3240 2024-09-16T00:20:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f44ff9640 1 -- 172.21.15.33:0/318522698 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6f540c16c0 con 0x55a0fc5a3240 2024-09-16T00:20:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f65e3c740 10 client.15879 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.131+0000 7f6f65e3c740 1 -- 172.21.15.33:0/318522698 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a0fc2d6960 con 0x55a0fc5a3240 2024-09-16T00:20:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.132+0000 7f6f44ff9640 1 -- 172.21.15.33:0/318522698 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f54036070 con 0x55a0fc5a3240 2024-09-16T00:20:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.132+0000 7f6f44ff9640 1 -- 172.21.15.33:0/318522698 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6f540c5a60 con 0x55a0fc5a3240 2024-09-16T00:20:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.132+0000 7f6f65e3c740 10 client.15879 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.132+0000 7f6f65e3c740 10 client.15879 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:20:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.132+0000 7f6f65e3c740 20 client.15879 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.132+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a0fc5b2900 0x55a0fc5b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.132+0000 7f6f65e3c740 4 client.15879 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.132+0000 7f6f65e3c740 1 -- 172.21.15.33:0/318522698 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0fc50ab90 con 0x55a0fc5b2900 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.132+0000 7f6f617ff640 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a0fc5b2900 0x55a0fc5b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f44ff9640 10 client.15879 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f617ff640 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a0fc5b2900 0x55a0fc5b4cf0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d544c4cfd6bd1c5 server_cookie=f75dab2d6faf9386 in_seq=0 out_seq=0 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f44ff9640 1 -- 172.21.15.33:0/318522698 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a0fc50ab90 con 0x55a0fc5b2900 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f44ff9640 10 client.15879 handle_command_reply: tid=0 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f65e3c740 1 client.15879 shutdown 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f65e3c740 1 -- 172.21.15.33:0/318522698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 msgr2=0x55a0fc5a0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a0c70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6f540b9770 tx=0x7f6f540bcd90 comp rx=0 tx=0).stop 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f65e3c740 1 -- 172.21.15.33:0/318522698 shutdown_connections 2024-09-16T00:20:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a0fc5b2900 0x55a0fc5b4cf0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a0c70 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598980 0x55a0fc5a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/318522698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc598060 0x55a0fc5a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f44ff9640 0 client.15879 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:04.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.133+0000 7f6f65e3c740 1 -- 172.21.15.33:0/318522698 >> 172.21.15.33:0/318522698 conn(0x55a0fc5945f0 msgr2=0x55a0fc59a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:04.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.134+0000 7f6f65e3c740 1 -- 172.21.15.33:0/318522698 shutdown_connections 2024-09-16T00:20:04.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.134+0000 7f6f65e3c740 1 -- 172.21.15.33:0/318522698 wait complete. 2024-09-16T00:20:04.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.134+0000 7f6f65e3c740 20 client.15879 trim_cache size 0 max 16384 2024-09-16T00:20:04.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f65e3c740 1 Processor -- start 2024-09-16T00:20:04.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f65e3c740 1 -- start start 2024-09-16T00:20:04.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f65e3c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598060 0x55a0fc593640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f65e3c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598980 0x55a0fc5ba5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f65e3c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc5a3240 0x55a0fc5bab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f65e3c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c1b1960 con 0x55a0fc598060 2024-09-16T00:20:04.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f65e3c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c1af230 con 0x55a0fc598980 2024-09-16T00:20:04.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f65e3c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c1ad080 con 0x55a0fc5a3240 2024-09-16T00:20:04.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f617ff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598060 0x55a0fc593640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f617ff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598060 0x55a0fc593640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48110/0 (socket says 172.21.15.33:48110) 2024-09-16T00:20:04.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f617ff640 1 -- 172.21.15.33:0/580350494 learned_addr learned my addr 172.21.15.33:0/580350494 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:04.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f60ffe640 1 --2- 172.21.15.33:0/580350494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598980 0x55a0fc5ba5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.151+0000 7f6f62000640 1 --2- 172.21.15.33:0/580350494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc5a3240 0x55a0fc5bab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.152+0000 7f6f617ff640 1 -- 172.21.15.33:0/580350494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc5a3240 msgr2=0x55a0fc5bab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.152+0000 7f6f617ff640 1 --2- 172.21.15.33:0/580350494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc5a3240 0x55a0fc5bab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.152+0000 7f6f617ff640 1 -- 172.21.15.33:0/580350494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598980 msgr2=0x55a0fc5ba5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.152+0000 7f6f617ff640 1 --2- 172.21.15.33:0/580350494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598980 0x55a0fc5ba5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.152+0000 7f6f617ff640 1 -- 172.21.15.33:0/580350494 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0fc595280 con 0x55a0fc598060 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.152+0000 7f6f62000640 1 --2- 172.21.15.33:0/580350494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc5a3240 0x55a0fc5bab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.152+0000 7f6f617ff640 1 --2- 172.21.15.33:0/580350494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598060 0x55a0fc593640 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6f58011a60 tx=0x7f6f58013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ad2e16aaa68aa5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.152+0000 7f6f46ffd640 1 -- 172.21.15.33:0/580350494 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f58004220 con 0x55a0fc598060 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.152+0000 7f6f46ffd640 1 -- 172.21.15.33:0/580350494 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f5801fe50 con 0x55a0fc598060 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.152+0000 7f6f46ffd640 1 -- 172.21.15.33:0/580350494 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f58028330 con 0x55a0fc598060 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.153+0000 7f6f65e3c740 1 -- 172.21.15.33:0/580350494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598060 msgr2=0x55a0fc593640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.153+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/580350494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598060 0x55a0fc593640 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6f58011a60 tx=0x7f6f58013000 comp rx=0 tx=0).stop 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.153+0000 7f6f65e3c740 1 -- 172.21.15.33:0/580350494 shutdown_connections 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.153+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/580350494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc5a3240 0x55a0fc5bab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.153+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/580350494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598980 0x55a0fc5ba5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.153+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/580350494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598060 0x55a0fc593640 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.153+0000 7f6f65e3c740 1 -- 172.21.15.33:0/580350494 >> 172.21.15.33:0/580350494 conn(0x55a0fc5945f0 msgr2=0x55a0fc592f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.153+0000 7f6f65e3c740 1 -- 172.21.15.33:0/580350494 shutdown_connections 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.153+0000 7f6f65e3c740 1 -- 172.21.15.33:0/580350494 wait complete. 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.153+0000 7f6f65e3c740 1 -- start start 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.153+0000 7f6f65e3c740 1 Processor -- start 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f65e3c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598060 0x55a0fc5ab390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f65e3c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598980 0x55a0fc5a76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f65e3c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f65e3c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c1ad6f0 con 0x55a0fc598980 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f65e3c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c1aeb60 con 0x55a0fc598060 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f65e3c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c1addc0 con 0x55a0fc5a3240 2024-09-16T00:20:04.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f65e3c740 10 client.0 _ll_register_callbacks cb 0x7f6f4c00cdf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f60ffe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598980 0x55a0fc5a76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f60ffe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598980 0x55a0fc5a76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48118/0 (socket says 172.21.15.33:48118) 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f60ffe640 1 -- 172.21.15.33:0/3684393550 learned_addr learned my addr 172.21.15.33:0/3684393550 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f62000640 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f617ff640 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598060 0x55a0fc5ab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f60ffe640 1 -- 172.21.15.33:0/3684393550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc5a3240 msgr2=0x55a0fc5a7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f60ffe640 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f60ffe640 1 -- 172.21.15.33:0/3684393550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598060 msgr2=0x55a0fc5ab390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f60ffe640 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598060 0x55a0fc5ab390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f60ffe640 1 -- 172.21.15.33:0/3684393550 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0fc5a8300 con 0x55a0fc598980 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f60ffe640 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598980 0x55a0fc5a76d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x55a0fc595450 tx=0x7f6f4c014f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55a5cf501b053b93 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.154+0000 7f6f44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:20:04.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f44ff9640 1 -- 172.21.15.33:0/3684393550 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c00f8f0 con 0x55a0fc598980 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f44ff9640 1 -- 172.21.15.33:0/3684393550 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6f4c00fa90 con 0x55a0fc598980 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f65e3c740 1 -- 172.21.15.33:0/3684393550 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f5801fe50 con 0x55a0fc598980 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f44ff9640 1 -- 172.21.15.33:0/3684393550 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6f4c00fc90 con 0x55a0fc598980 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f65e3c740 10 client.15885 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f65e3c740 1 -- 172.21.15.33:0/3684393550 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a0fc61e480 con 0x55a0fc598980 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f44ff9640 1 -- 172.21.15.33:0/3684393550 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f5801fe50 con 0x55a0fc598980 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f44ff9640 1 -- 172.21.15.33:0/3684393550 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6f4c01f4c0 con 0x55a0fc598980 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f65e3c740 10 client.15885 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f65e3c740 10 client.15885 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f65e3c740 20 client.15885 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a0fc61ece0 0x55a0fc621100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f65e3c740 4 client.15885 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5dd683", "release": true, "await": true}] 2024-09-16T00:20:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.155+0000 7f6f65e3c740 1 -- 172.21.15.33:0/3684393550 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5dd683", "release": true, "await": true}) -- 0x55a0fc58f950 con 0x55a0fc61ece0 2024-09-16T00:20:04.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.156+0000 7f6f617ff640 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a0fc61ece0 0x55a0fc621100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:04.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.156+0000 7f6f617ff640 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a0fc61ece0 0x55a0fc621100 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=927d660b8379c3a1 server_cookie=19ca9895a0a53978 in_seq=0 out_seq=0 2024-09-16T00:20:04.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.156+0000 7f6f44ff9640 10 client.15885 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:04.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f44ff9640 1 -- 172.21.15.33:0/3684393550 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+587 (crc 0 0 0) 0x55a0fc58f950 con 0x55a0fc61ece0 2024-09-16T00:20:04.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f44ff9640 10 client.15885 handle_command_reply: tid=0 2024-09-16T00:20:04.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f65e3c740 1 client.15885 shutdown 2024-09-16T00:20:04.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f65e3c740 1 -- 172.21.15.33:0/3684393550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598980 msgr2=0x55a0fc5a76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:04.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598980 0x55a0fc5a76d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x55a0fc595450 tx=0x7f6f4c014f60 comp rx=0 tx=0).stop 2024-09-16T00:20:04.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f65e3c740 1 -- 172.21.15.33:0/3684393550 shutdown_connections 2024-09-16T00:20:04.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a0fc61ece0 0x55a0fc621100 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0fc5a3240 0x55a0fc5a7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0fc598980 0x55a0fc5a76d0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f65e3c740 1 --2- 172.21.15.33:0/3684393550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0fc598060 0x55a0fc5ab390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f65e3c740 1 -- 172.21.15.33:0/3684393550 >> 172.21.15.33:0/3684393550 conn(0x55a0fc5945f0 msgr2=0x55a0fc59a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:04.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f44ff9640 0 client.15885 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:04.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f65e3c740 1 -- 172.21.15.33:0/3684393550 shutdown_connections 2024-09-16T00:20:04.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f65e3c740 1 -- 172.21.15.33:0/3684393550 wait complete. 2024-09-16T00:20:04.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.164+0000 7f6f65e3c740 20 client.15885 trim_cache size 0 max 16384 2024-09-16T00:20:04.167 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:20:04.167 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 28, 2024-09-16T00:20:04.167 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:20:04.167 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 8, 2024-09-16T00:20:04.167 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:20:04.167 INFO:teuthology.orchestra.run.smithi033.stdout: "5dd683": { 2024-09-16T00:20:04.167 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 8, 2024-09-16T00:20:04.167 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:20:04.167 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:20:04.167 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:20:04.167 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:20:04.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 139.6, 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:04.168 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:04.169 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:20:04.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.165+0000 7f6f63a63640 1 -- 172.21.15.33:0/3326987749 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6f480a3240 msgr2=0x7f6f480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:04.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.165+0000 7f6f63a63640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6f480a3240 0x7f6f480a5700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6f58005fd0 tx=0x7f6f5800b390 comp rx=0 tx=0).stop 2024-09-16T00:20:04.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.165+0000 7f6f63a63640 1 -- 172.21.15.33:0/3326987749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f5c106f10 msgr2=0x7f6f5c19ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:04.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.165+0000 7f6f63a63640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f5c106f10 0x7f6f5c19ccc0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6f54031a20 tx=0x7f6f54031a50 comp rx=0 tx=0).stop 2024-09-16T00:20:04.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.165+0000 7f6f63a63640 1 -- 172.21.15.33:0/3326987749 shutdown_connections 2024-09-16T00:20:04.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.165+0000 7f6f63a63640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6f480a3240 0x7f6f480a5700 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.165+0000 7f6f63a63640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f5c106f10 0x7f6f5c19ccc0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.165+0000 7f6f63a63640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f5c102790 0x7f6f5c198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.165+0000 7f6f63a63640 1 --2- 172.21.15.33:0/3326987749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c101de0 0x7f6f5c1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:04.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.165+0000 7f6f63a63640 1 -- 172.21.15.33:0/3326987749 >> 172.21.15.33:0/3326987749 conn(0x7f6f5c0fd8c0 msgr2=0x7f6f5c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:04.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.165+0000 7f6f63a63640 1 -- 172.21.15.33:0/3326987749 shutdown_connections 2024-09-16T00:20:04.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:04.165+0000 7f6f63a63640 1 -- 172.21.15.33:0/3326987749 wait complete. 2024-09-16T00:20:04.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 41619 3.32 MB/sec warmup 72 sec latency 18351.741 ms 2024-09-16T00:20:05.012 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5dd683', total seconds elapsed: 23.1 2024-09-16T00:20:05.012 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.55 2024-09-16T00:20:05.012 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:20:05.285 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:05.648 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43471 3.44 MB/sec warmup 73 sec latency 257.229 ms 2024-09-16T00:20:05.720 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:05 smithi033 ceph-mon[31243]: pgmap v438: 129 pgs: 129 active+clean; 36 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 556 KiB/s wr, 29 op/s 2024-09-16T00:20:05.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.802+0000 7f685b57d640 1 -- 172.21.15.33:0/2591212318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68541008d0 msgr2=0x7f6854100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:05.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.802+0000 7f685b57d640 1 --2- 172.21.15.33:0/2591212318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68541008d0 0x7f6854100cb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6850009900 tx=0x7f685002a410 comp rx=0 tx=0).stop 2024-09-16T00:20:05.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.802+0000 7f685b57d640 1 -- 172.21.15.33:0/2591212318 shutdown_connections 2024-09-16T00:20:05.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.802+0000 7f685b57d640 1 --2- 172.21.15.33:0/2591212318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f685410e480 0x7f6854110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:05.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.802+0000 7f685b57d640 1 --2- 172.21.15.33:0/2591212318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6854101280 0x7f685410df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:05.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.802+0000 7f685b57d640 1 --2- 172.21.15.33:0/2591212318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68541008d0 0x7f6854100cb0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:05.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.802+0000 7f685b57d640 1 -- 172.21.15.33:0/2591212318 >> 172.21.15.33:0/2591212318 conn(0x7f68540fc7a0 msgr2=0x7f68540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:05.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.802+0000 7f685b57d640 1 -- 172.21.15.33:0/2591212318 shutdown_connections 2024-09-16T00:20:05.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.802+0000 7f685b57d640 1 -- 172.21.15.33:0/2591212318 wait complete. 2024-09-16T00:20:05.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.802+0000 7f685b57d640 1 Processor -- start 2024-09-16T00:20:05.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.802+0000 7f685b57d640 1 -- start start 2024-09-16T00:20:05.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f685b57d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68541008d0 0x7f685419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:05.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f685b57d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6854101280 0x7f685419ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:05.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f685b57d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f685410e480 0x7f685419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:05.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f685b57d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f68541141b0 con 0x7f68541008d0 2024-09-16T00:20:05.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f685b57d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6854114030 con 0x7f685410e480 2024-09-16T00:20:05.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f685b57d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6854114330 con 0x7f6854101280 2024-09-16T00:20:05.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f6859319640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68541008d0 0x7f685419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:05.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f6859319640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68541008d0 0x7f685419a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48140/0 (socket says 172.21.15.33:48140) 2024-09-16T00:20:05.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f6859319640 1 -- 172.21.15.33:0/1851293424 learned_addr learned my addr 172.21.15.33:0/1851293424 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:05.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f6858b18640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6854101280 0x7f685419ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:05.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f6859b1a640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f685410e480 0x7f685419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:05.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f6859319640 1 -- 172.21.15.33:0/1851293424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6854101280 msgr2=0x7f685419ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:05.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f6859319640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6854101280 0x7f685419ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:05.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f6859319640 1 -- 172.21.15.33:0/1851293424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f685410e480 msgr2=0x7f685419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:05.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f6859319640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f685410e480 0x7f685419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:05.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f6859319640 1 -- 172.21.15.33:0/1851293424 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685419f680 con 0x7f68541008d0 2024-09-16T00:20:05.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f6859319640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68541008d0 0x7f685419a6d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6850006010 tx=0x7f68500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f684a7fc640 1 -- 172.21.15.33:0/1851293424 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685003f030 con 0x7f68541008d0 2024-09-16T00:20:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f684a7fc640 1 -- 172.21.15.33:0/1851293424 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6850003ce0 con 0x7f68541008d0 2024-09-16T00:20:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f685b57d640 1 -- 172.21.15.33:0/1851293424 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f685419f910 con 0x7f68541008d0 2024-09-16T00:20:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.803+0000 7f684a7fc640 1 -- 172.21.15.33:0/1851293424 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850035070 con 0x7f68541008d0 2024-09-16T00:20:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.804+0000 7f685b57d640 1 -- 172.21.15.33:0/1851293424 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68541a7250 con 0x7f68541008d0 2024-09-16T00:20:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.804+0000 7f685b57d640 1 -- 172.21.15.33:0/1851293424 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6854100cb0 con 0x7f68541008d0 2024-09-16T00:20:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.806+0000 7f684a7fc640 1 -- 172.21.15.33:0/1851293424 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6850042030 con 0x7f68541008d0 2024-09-16T00:20:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.806+0000 7f684a7fc640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68400a32c0 0x7f68400a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.806+0000 7f6858b18640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68400a32c0 0x7f68400a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.806+0000 7f684a7fc640 1 -- 172.21.15.33:0/1851293424 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f68500f0660 con 0x7f68541008d0 2024-09-16T00:20:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.807+0000 7f6858b18640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68400a32c0 0x7f68400a5780 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f685410e8a0 tx=0x7f684400a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:05.810+0000 7f684a7fc640 1 -- 172.21.15.33:0/1851293424 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f68500bcfc0 con 0x7f68541008d0 2024-09-16T00:20:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:05 smithi083 ceph-mon[35242]: pgmap v438: 129 pgs: 129 active+clean; 36 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 556 KiB/s wr, 29 op/s 2024-09-16T00:20:05.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:05 smithi060 ceph-mon[35185]: pgmap v438: 129 pgs: 129 active+clean; 36 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 556 KiB/s wr, 29 op/s 2024-09-16T00:20:06.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.021+0000 7f685b57d640 1 -- 172.21.15.33:0/1851293424 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6854109610 con 0x7f68541008d0 2024-09-16T00:20:06.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.021+0000 7f684a7fc640 1 -- 172.21.15.33:0/1851293424 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11906 (secure 0 0 0) 0x7f68500bc8c0 con 0x7f68541008d0 2024-09-16T00:20:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:20:06.023 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"btime":"2024-09-16T00:19:02:115556+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:19:01.460559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:20:06.023 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 28 2024-09-16T00:20:06.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.024+0000 7f685b57d640 1 -- 172.21.15.33:0/1851293424 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68400a32c0 msgr2=0x7f68400a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:06.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.024+0000 7f685b57d640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68400a32c0 0x7f68400a5780 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f685410e8a0 tx=0x7f684400a250 comp rx=0 tx=0).stop 2024-09-16T00:20:06.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.024+0000 7f685b57d640 1 -- 172.21.15.33:0/1851293424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68541008d0 msgr2=0x7f685419a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:06.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.024+0000 7f685b57d640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68541008d0 0x7f685419a6d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6850006010 tx=0x7f68500042f0 comp rx=0 tx=0).stop 2024-09-16T00:20:06.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.024+0000 7f685b57d640 1 -- 172.21.15.33:0/1851293424 shutdown_connections 2024-09-16T00:20:06.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.024+0000 7f685b57d640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68400a32c0 0x7f68400a5780 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:06.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.024+0000 7f685b57d640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f685410e480 0x7f685419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:06.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.024+0000 7f685b57d640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6854101280 0x7f685419ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:06.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.024+0000 7f685b57d640 1 --2- 172.21.15.33:0/1851293424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68541008d0 0x7f685419a6d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:06.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.024+0000 7f685b57d640 1 -- 172.21.15.33:0/1851293424 >> 172.21.15.33:0/1851293424 conn(0x7f68540fc7a0 msgr2=0x7f685410f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:06.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.024+0000 7f685b57d640 1 -- 172.21.15.33:0/1851293424 shutdown_connections 2024-09-16T00:20:06.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:06.024+0000 7f685b57d640 1 -- 172.21.15.33:0/1851293424 wait complete. 2024-09-16T00:20:06.649 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43471 3.39 MB/sec warmup 74 sec latency 1257.325 ms 2024-09-16T00:20:06.818 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:06 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1851293424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:20:06.855 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0' 2024-09-16T00:20:06.855 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0 2024-09-16T00:20:06.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:06 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1851293424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:20:06.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:06 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1851293424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:20:07.121 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:07.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.614+0000 7f87637bd640 1 -- 172.21.15.33:0/1928065629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f875c106a80 msgr2=0x7f875c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:07.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.614+0000 7f87637bd640 1 --2- 172.21.15.33:0/1928065629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f875c106a80 0x7f875c106e60 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8758009930 tx=0x7f875802a420 comp rx=0 tx=0).stop 2024-09-16T00:20:07.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.614+0000 7f87637bd640 1 -- 172.21.15.33:0/1928065629 shutdown_connections 2024-09-16T00:20:07.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.614+0000 7f87637bd640 1 --2- 172.21.15.33:0/1928065629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f875c101420 0x7f875c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.614+0000 7f87637bd640 1 --2- 172.21.15.33:0/1928065629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f875c100a80 0x7f875c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.614+0000 7f87637bd640 1 --2- 172.21.15.33:0/1928065629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f875c106a80 0x7f875c106e60 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.614+0000 7f87637bd640 1 -- 172.21.15.33:0/1928065629 >> 172.21.15.33:0/1928065629 conn(0x7f875c0fc7a0 msgr2=0x7f875c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.614+0000 7f87637bd640 1 -- 172.21.15.33:0/1928065629 shutdown_connections 2024-09-16T00:20:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.614+0000 7f87637bd640 1 -- 172.21.15.33:0/1928065629 wait complete. 2024-09-16T00:20:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f87637bd640 1 Processor -- start 2024-09-16T00:20:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f87637bd640 1 -- start start 2024-09-16T00:20:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f87637bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f875c100a80 0x7f875c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f87637bd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f875c101420 0x7f875c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f87637bd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f875c106a80 0x7f875c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f87637bd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f875c110ea0 con 0x7f875c106a80 2024-09-16T00:20:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f87637bd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f875c110d20 con 0x7f875c101420 2024-09-16T00:20:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f87637bd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f875c111020 con 0x7f875c100a80 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f8761d5a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f875c106a80 0x7f875c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f8761d5a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f875c106a80 0x7f875c19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48154/0 (socket says 172.21.15.33:48154) 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f8761d5a640 1 -- 172.21.15.33:0/984744870 learned_addr learned my addr 172.21.15.33:0/984744870 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f8760d58640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f875c101420 0x7f875c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f8761559640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f875c100a80 0x7f875c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f8761d5a640 1 -- 172.21.15.33:0/984744870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f875c100a80 msgr2=0x7f875c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f8761d5a640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f875c100a80 0x7f875c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f8761d5a640 1 -- 172.21.15.33:0/984744870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f875c101420 msgr2=0x7f875c1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f8761d5a640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f875c101420 0x7f875c1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.615+0000 7f8761d5a640 1 -- 172.21.15.33:0/984744870 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f875c19d440 con 0x7f875c106a80 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.616+0000 7f8761d5a640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f875c106a80 0x7f875c19cd60 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f875400b970 tx=0x7f875400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.616+0000 7f87527fc640 1 -- 172.21.15.33:0/984744870 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8754019070 con 0x7f875c106a80 2024-09-16T00:20:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.616+0000 7f87527fc640 1 -- 172.21.15.33:0/984744870 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f875400cb40 con 0x7f875c106a80 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.616+0000 7f87637bd640 1 -- 172.21.15.33:0/984744870 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f875c19d730 con 0x7f875c106a80 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.616+0000 7f87527fc640 1 -- 172.21.15.33:0/984744870 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8754007650 con 0x7f875c106a80 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.616+0000 7f87637bd640 1 -- 172.21.15.33:0/984744870 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f875c1a5010 con 0x7f875c106a80 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.619+0000 7f87527fc640 1 -- 172.21.15.33:0/984744870 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f875401c070 con 0x7f875c106a80 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.619+0000 7f8765b96740 1 Processor -- start 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.619+0000 7f8765b96740 1 -- start start 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.619+0000 7f8765b96740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e552850 0x55929e552c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.619+0000 7f8765b96740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 0x55929e55d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.619+0000 7f8765b96740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e55da30 0x55929e55fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.619+0000 7f8765b96740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55929e4b7160 con 0x55929e553170 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.619+0000 7f8765b96740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55929e45e720 con 0x55929e552850 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.619+0000 7f8765b96740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55929e4b3120 con 0x55929e55da30 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.619+0000 7f8761559640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 0x55929e55d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.619+0000 7f8761559640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 0x55929e55d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48162/0 (socket says 172.21.15.33:48162) 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.619+0000 7f8761559640 1 -- 172.21.15.33:0/3234394954 learned_addr learned my addr 172.21.15.33:0/3234394954 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f8760d58640 1 --2- 172.21.15.33:0/3234394954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e552850 0x55929e552c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f8760d58640 1 --2- 172.21.15.33:0/3234394954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e55da30 0x55929e55fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f87527fc640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f87480a32c0 0x7f87480a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f8761559640 1 -- 172.21.15.33:0/3234394954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e55da30 msgr2=0x55929e55fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f8761559640 1 --2- 172.21.15.33:0/3234394954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e55da30 0x55929e55fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f8761559640 1 -- 172.21.15.33:0/3234394954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e552850 msgr2=0x55929e552c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f8761559640 1 --2- 172.21.15.33:0/3234394954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e552850 0x55929e552c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f8761559640 1 -- 172.21.15.33:0/3234394954 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55929e228ba0 con 0x55929e553170 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f8761d5a640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f87480a32c0 0x7f87480a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f87527fc640 1 -- 172.21.15.33:0/984744870 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f87540cf310 con 0x7f875c106a80 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f8761559640 1 --2- 172.21.15.33:0/3234394954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 0x55929e55d4f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8758009930 tx=0x7f8758032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcf88af30f16c97 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f87477fe640 1 -- 172.21.15.33:0/3234394954 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875803f070 con 0x55929e553170 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f87477fe640 1 -- 172.21.15.33:0/3234394954 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87580317b0 con 0x55929e553170 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f8761d5a640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f87480a32c0 0x7f87480a5780 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f875409b260 tx=0x7f8754090a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.620+0000 7f87477fe640 1 -- 172.21.15.33:0/3234394954 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8758035070 con 0x55929e553170 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.621+0000 7f8765b96740 1 -- 172.21.15.33:0/3234394954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 msgr2=0x55929e55d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.621+0000 7f8765b96740 1 --2- 172.21.15.33:0/3234394954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 0x55929e55d4f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8758009930 tx=0x7f8758032ce0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.621+0000 7f8765b96740 1 -- 172.21.15.33:0/3234394954 shutdown_connections 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.621+0000 7f8765b96740 1 --2- 172.21.15.33:0/3234394954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e55da30 0x55929e55fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.621+0000 7f8765b96740 1 --2- 172.21.15.33:0/3234394954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 0x55929e55d4f0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.621+0000 7f8765b96740 1 --2- 172.21.15.33:0/3234394954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e552850 0x55929e552c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.621+0000 7f8765b96740 1 -- 172.21.15.33:0/3234394954 >> 172.21.15.33:0/3234394954 conn(0x55929e54ede0 msgr2=0x55929e54f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.621+0000 7f8765b96740 1 -- 172.21.15.33:0/3234394954 shutdown_connections 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.621+0000 7f8765b96740 1 -- 172.21.15.33:0/3234394954 wait complete. 2024-09-16T00:20:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8765b96740 1 -- start start 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8765b96740 1 Processor -- start 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8765b96740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e552850 0x55929e55a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8765b96740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e553170 0x55929e55aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8765b96740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e55da30 0x55929e55b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8765b96740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55929e560600 con 0x55929e55da30 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8765b96740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55929e43f900 con 0x55929e552850 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8765b96740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55929e560780 con 0x55929e553170 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8765b96740 10 client.0 _ll_register_callbacks cb 0x55929e43d780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8760d58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e55da30 0x55929e55b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8760d58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e55da30 0x55929e55b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48170/0 (socket says 172.21.15.33:48170) 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8760d58640 1 -- 172.21.15.33:0/645830767 learned_addr learned my addr 172.21.15.33:0/645830767 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8761559640 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e553170 0x55929e55aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.622+0000 7f8760d58640 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e552850 0x55929e55a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8760d58640 1 -- 172.21.15.33:0/645830767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e553170 msgr2=0x55929e55aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8760d58640 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e553170 0x55929e55aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8760d58640 1 -- 172.21.15.33:0/645830767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e552850 msgr2=0x55929e55a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8760d58640 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e552850 0x55929e55a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8760d58640 1 -- 172.21.15.33:0/645830767 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55929e47fc50 con 0x55929e55da30 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8761559640 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e553170 0x55929e55aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8760d58640 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e55da30 0x55929e55b3f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f874c00bfa0 tx=0x7f874c014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc3ec9f70b96d6f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8744ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8744ff9640 1 -- 172.21.15.33:0/645830767 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c004230 con 0x55929e55da30 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8765b96740 1 -- 172.21.15.33:0/645830767 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87580317b0 con 0x55929e55da30 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8744ff9640 1 -- 172.21.15.33:0/645830767 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f874c0043d0 con 0x55929e55da30 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8744ff9640 1 -- 172.21.15.33:0/645830767 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f874c028050 con 0x55929e55da30 2024-09-16T00:20:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8765b96740 10 client.15903 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8765b96740 1 -- 172.21.15.33:0/645830767 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55929e56c3b0 con 0x55929e55da30 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8744ff9640 1 -- 172.21.15.33:0/645830767 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87580317b0 con 0x55929e55da30 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.623+0000 7f8744ff9640 1 -- 172.21.15.33:0/645830767 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f874c003b80 con 0x55929e55da30 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.624+0000 7f8765b96740 10 client.15903 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.624+0000 7f8765b96740 10 client.15903 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.624+0000 7f8765b96740 20 client.15903 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.624+0000 7f8765b96740 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55929e56d870 0x55929e56fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.624+0000 7f8765b96740 4 client.15903 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.624+0000 7f8765b96740 1 -- 172.21.15.33:0/645830767 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55929e4c4890 con 0x55929e56d870 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.624+0000 7f8761559640 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55929e56d870 0x55929e56fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.624+0000 7f8744ff9640 10 client.15903 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.624+0000 7f8761559640 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55929e56d870 0x55929e56fc80 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=378372084f5a0c33 server_cookie=d252ea76b9f613a1 in_seq=0 out_seq=0 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8744ff9640 1 -- 172.21.15.33:0/645830767 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55929e4c4890 con 0x55929e56d870 2024-09-16T00:20:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8744ff9640 10 client.15903 handle_command_reply: tid=0 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8765b96740 1 client.15903 shutdown 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8765b96740 1 -- 172.21.15.33:0/645830767 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e55da30 msgr2=0x55929e55b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8765b96740 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e55da30 0x55929e55b3f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f874c00bfa0 tx=0x7f874c014040 comp rx=0 tx=0).stop 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8765b96740 1 -- 172.21.15.33:0/645830767 shutdown_connections 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8765b96740 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55929e56d870 0x55929e56fc80 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8765b96740 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e55da30 0x55929e55b3f0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8765b96740 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e553170 0x55929e55aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8765b96740 1 --2- 172.21.15.33:0/645830767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e552850 0x55929e55a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8765b96740 1 -- 172.21.15.33:0/645830767 >> 172.21.15.33:0/645830767 conn(0x55929e54ede0 msgr2=0x55929e554b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8744ff9640 0 client.15903 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8765b96740 1 -- 172.21.15.33:0/645830767 shutdown_connections 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8765b96740 1 -- 172.21.15.33:0/645830767 wait complete. 2024-09-16T00:20:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.625+0000 7f8765b96740 20 client.15903 trim_cache size 0 max 16384 2024-09-16T00:20:07.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.642+0000 7f8765b96740 1 Processor -- start 2024-09-16T00:20:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.642+0000 7f8765b96740 1 -- start start 2024-09-16T00:20:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8765b96740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e552850 0x55929e5552b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8765b96740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 0x55929e575550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8765b96740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e55da30 0x55929e575a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8765b96740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f875c1b18d0 con 0x55929e553170 2024-09-16T00:20:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8765b96740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f875c1af1a0 con 0x55929e55da30 2024-09-16T00:20:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8765b96740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f875c1acff0 con 0x55929e552850 2024-09-16T00:20:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8761559640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 0x55929e575550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8761559640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 0x55929e575550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48176/0 (socket says 172.21.15.33:48176) 2024-09-16T00:20:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8761559640 1 -- 172.21.15.33:0/2087505790 learned_addr learned my addr 172.21.15.33:0/2087505790 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8760d58640 1 --2- 172.21.15.33:0/2087505790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e552850 0x55929e5552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8760d58640 1 --2- 172.21.15.33:0/2087505790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e55da30 0x55929e575a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8761559640 1 -- 172.21.15.33:0/2087505790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e552850 msgr2=0x55929e5552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:07.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8761559640 1 --2- 172.21.15.33:0/2087505790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e552850 0x55929e5552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8761559640 1 -- 172.21.15.33:0/2087505790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e55da30 msgr2=0x55929e575a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:07.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8761559640 1 --2- 172.21.15.33:0/2087505790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e55da30 0x55929e575a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8761559640 1 -- 172.21.15.33:0/2087505790 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55929e552680 con 0x55929e553170 2024-09-16T00:20:07.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8760d58640 1 --2- 172.21.15.33:0/2087505790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e552850 0x55929e5552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:20:07.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8761559640 1 --2- 172.21.15.33:0/2087505790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 0x55929e575550 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8758009930 tx=0x7f8758000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8861eb2f87d4335 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:07.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.643+0000 7f8746ffd640 1 -- 172.21.15.33:0/2087505790 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875803f040 con 0x55929e553170 2024-09-16T00:20:07.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.644+0000 7f8746ffd640 1 -- 172.21.15.33:0/2087505790 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8758049040 con 0x55929e553170 2024-09-16T00:20:07.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.644+0000 7f8746ffd640 1 -- 172.21.15.33:0/2087505790 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8758035040 con 0x55929e553170 2024-09-16T00:20:07.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.644+0000 7f8765b96740 1 -- 172.21.15.33:0/2087505790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 msgr2=0x55929e575550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:07.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.644+0000 7f8765b96740 1 --2- 172.21.15.33:0/2087505790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 0x55929e575550 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8758009930 tx=0x7f8758000bc0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.644+0000 7f8765b96740 1 -- 172.21.15.33:0/2087505790 shutdown_connections 2024-09-16T00:20:07.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.644+0000 7f8765b96740 1 --2- 172.21.15.33:0/2087505790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e55da30 0x55929e575a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.644+0000 7f8765b96740 1 --2- 172.21.15.33:0/2087505790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e553170 0x55929e575550 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.644+0000 7f8765b96740 1 --2- 172.21.15.33:0/2087505790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e552850 0x55929e5552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.644+0000 7f8765b96740 1 -- 172.21.15.33:0/2087505790 >> 172.21.15.33:0/2087505790 conn(0x55929e54ede0 msgr2=0x55929e55c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:07.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.644+0000 7f8765b96740 1 -- 172.21.15.33:0/2087505790 shutdown_connections 2024-09-16T00:20:07.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.644+0000 7f8765b96740 1 -- 172.21.15.33:0/2087505790 wait complete. 2024-09-16T00:20:07.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8765b96740 1 -- start start 2024-09-16T00:20:07.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8765b96740 1 Processor -- start 2024-09-16T00:20:07.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8765b96740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e552850 0x55929e561a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8765b96740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e553170 0x55929e561f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8765b96740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e55da30 0x55929e5624f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8765b96740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f875c1ad660 con 0x55929e552850 2024-09-16T00:20:07.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8765b96740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f875c1aead0 con 0x55929e55da30 2024-09-16T00:20:07.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8765b96740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f875c1add30 con 0x55929e553170 2024-09-16T00:20:07.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8765b96740 10 client.0 _ll_register_callbacks cb 0x7f874c00b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:07.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8760d58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e552850 0x55929e561a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8760d58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e552850 0x55929e561a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48188/0 (socket says 172.21.15.33:48188) 2024-09-16T00:20:07.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8760d58640 1 -- 172.21.15.33:0/873500479 learned_addr learned my addr 172.21.15.33:0/873500479 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:07.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8761559640 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e553170 0x55929e561f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8760d58640 1 -- 172.21.15.33:0/873500479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e553170 msgr2=0x55929e561f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:07.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.645+0000 7f8760d58640 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e553170 0x55929e561f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8760d58640 1 -- 172.21.15.33:0/873500479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e55da30 msgr2=0x55929e5624f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:07.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8760d58640 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e55da30 0x55929e5624f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8760d58640 1 -- 172.21.15.33:0/873500479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55929e54a3f0 con 0x55929e552850 2024-09-16T00:20:07.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8761559640 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e553170 0x55929e561f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:20:07.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8760d58640 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e552850 0x55929e561a50 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f874c014010 tx=0x7f874c00f3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a7b64906c22b8bd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:07.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8744ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:20:07.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8744ff9640 1 -- 172.21.15.33:0/873500479 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c005b90 con 0x55929e552850 2024-09-16T00:20:07.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8744ff9640 1 -- 172.21.15.33:0/873500479 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f874c01dc00 con 0x55929e552850 2024-09-16T00:20:07.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8765b96740 1 -- 172.21.15.33:0/873500479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8758049040 con 0x55929e552850 2024-09-16T00:20:07.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8744ff9640 1 -- 172.21.15.33:0/873500479 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f874c03f050 con 0x55929e552850 2024-09-16T00:20:07.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8765b96740 10 client.15909 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:07.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8765b96740 1 -- 172.21.15.33:0/873500479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55929e5d8900 con 0x55929e552850 2024-09-16T00:20:07.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.646+0000 7f8744ff9640 1 -- 172.21.15.33:0/873500479 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8758049040 con 0x55929e552850 2024-09-16T00:20:07.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.647+0000 7f8744ff9640 1 -- 172.21.15.33:0/873500479 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f874c0053a0 con 0x55929e552850 2024-09-16T00:20:07.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.647+0000 7f8765b96740 10 client.15909 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:07.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.647+0000 7f8765b96740 10 client.15909 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:20:07.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.647+0000 7f8765b96740 20 client.15909 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:07.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.647+0000 7f8765b96740 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55929e5d9120 0x55929e5db580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:07.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.647+0000 7f8765b96740 4 client.15909 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}] 2024-09-16T00:20:07.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.647+0000 7f8765b96740 1 -- 172.21.15.33:0/873500479 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}) -- 0x55929e4bebe0 con 0x55929e5d9120 2024-09-16T00:20:07.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.647+0000 7f8761559640 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55929e5d9120 0x55929e5db580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:07.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.647+0000 7f8761559640 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55929e5d9120 0x55929e5db580 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66915d1308a751f2 server_cookie=fb641181dfe66994 in_seq=0 out_seq=0 2024-09-16T00:20:07.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.647+0000 7f8744ff9640 10 client.15909 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:07.653 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43577 3.35 MB/sec warmup 75 sec latency 1604.660 ms 2024-09-16T00:20:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.722+0000 7f8744ff9640 1 -- 172.21.15.33:0/873500479 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55929e4bebe0 con 0x55929e5d9120 2024-09-16T00:20:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.722+0000 7f8744ff9640 10 client.15909 handle_command_reply: tid=0 2024-09-16T00:20:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.722+0000 7f8765b96740 1 client.15909 shutdown 2024-09-16T00:20:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.723+0000 7f8765b96740 1 -- 172.21.15.33:0/873500479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e552850 msgr2=0x55929e561a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.723+0000 7f8765b96740 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e552850 0x55929e561a50 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f874c014010 tx=0x7f874c00f3f0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.723+0000 7f8765b96740 1 -- 172.21.15.33:0/873500479 shutdown_connections 2024-09-16T00:20:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.723+0000 7f8765b96740 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55929e5d9120 0x55929e5db580 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.723+0000 7f8765b96740 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55929e55da30 0x55929e5624f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.723+0000 7f8765b96740 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55929e553170 0x55929e561f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.723+0000 7f8765b96740 1 --2- 172.21.15.33:0/873500479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55929e552850 0x55929e561a50 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.723+0000 7f8744ff9640 0 client.15909 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:07.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.723+0000 7f8765b96740 1 -- 172.21.15.33:0/873500479 >> 172.21.15.33:0/873500479 conn(0x55929e54ede0 msgr2=0x55929e55a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:07.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.723+0000 7f8765b96740 1 -- 172.21.15.33:0/873500479 shutdown_connections 2024-09-16T00:20:07.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.723+0000 7f8765b96740 1 -- 172.21.15.33:0/873500479 wait complete. 2024-09-16T00:20:07.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.723+0000 7f8765b96740 20 client.15909 trim_cache size 0 max 16384 2024-09-16T00:20:07.726 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:20:07.726 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 28, 2024-09-16T00:20:07.726 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:20:07.727 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 10, 2024-09-16T00:20:07.727 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:20:07.727 INFO:teuthology.orchestra.run.smithi033.stdout: "f41e73eb": { 2024-09-16T00:20:07.727 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 10, 2024-09-16T00:20:07.727 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:20:07.727 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:20:07.727 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:20:07.727 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:20:07.727 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:20:07.727 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:20:07.727 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 139.6, 2024-09-16T00:20:07.727 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:20:07.728 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:20:07.728 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:20:07.728 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:20:07.728 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:20:07.728 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:20:07.728 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:07.728 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:07.728 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:07.728 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:07.728 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:07.728 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:20:07.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.724+0000 7f87637bd640 1 -- 172.21.15.33:0/984744870 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f87480a32c0 msgr2=0x7f87480a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:07.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.724+0000 7f87637bd640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f87480a32c0 0x7f87480a5780 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f875409b260 tx=0x7f8754090a40 comp rx=0 tx=0).stop 2024-09-16T00:20:07.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.724+0000 7f87637bd640 1 -- 172.21.15.33:0/984744870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f875c106a80 msgr2=0x7f875c19cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:07.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.724+0000 7f87637bd640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f875c106a80 0x7f875c19cd60 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f875400b970 tx=0x7f875400be30 comp rx=0 tx=0).stop 2024-09-16T00:20:07.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.724+0000 7f87637bd640 1 -- 172.21.15.33:0/984744870 shutdown_connections 2024-09-16T00:20:07.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.724+0000 7f87637bd640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f87480a32c0 0x7f87480a5780 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.724+0000 7f87637bd640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f875c106a80 0x7f875c19cd60 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.724+0000 7f87637bd640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f875c101420 0x7f875c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.724+0000 7f87637bd640 1 --2- 172.21.15.33:0/984744870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f875c100a80 0x7f875c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:07.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.724+0000 7f87637bd640 1 -- 172.21.15.33:0/984744870 >> 172.21.15.33:0/984744870 conn(0x7f875c0fc7a0 msgr2=0x7f875c104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:07.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.724+0000 7f87637bd640 1 -- 172.21.15.33:0/984744870 shutdown_connections 2024-09-16T00:20:07.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:07.724+0000 7f87637bd640 1 -- 172.21.15.33:0/984744870 wait complete. 2024-09-16T00:20:07.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:07 smithi033 ceph-mon[31243]: pgmap v439: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 837 KiB/s wr, 51 op/s 2024-09-16T00:20:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:07 smithi083 ceph-mon[35242]: pgmap v439: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 837 KiB/s wr, 51 op/s 2024-09-16T00:20:07.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:07 smithi060 ceph-mon[35185]: pgmap v439: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 837 KiB/s wr, 51 op/s 2024-09-16T00:20:08.629 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'f41e73eb', quiesce took 3.6 seconds. Will release after: 15.950000000000001 2024-09-16T00:20:08.649 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 3.31 MB/sec warmup 76 sec latency 931.673 ms 2024-09-16T00:20:09.649 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 3.26 MB/sec warmup 77 sec latency 1931.729 ms 2024-09-16T00:20:09.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:09 smithi033 ceph-mon[31243]: pgmap v440: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 675 KiB/s wr, 46 op/s 2024-09-16T00:20:09.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:09 smithi083 ceph-mon[35242]: pgmap v440: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 675 KiB/s wr, 46 op/s 2024-09-16T00:20:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:09 smithi060 ceph-mon[35185]: pgmap v440: 129 pgs: 129 active+clean; 37 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 675 KiB/s wr, 46 op/s 2024-09-16T00:20:10.649 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 3.22 MB/sec warmup 78 sec latency 2931.862 ms 2024-09-16T00:20:11.649 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 3.18 MB/sec warmup 79 sec latency 3931.992 ms 2024-09-16T00:20:11.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:11 smithi033 ceph-mon[31243]: pgmap v441: 129 pgs: 129 active+clean; 37 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 785 KiB/s wr, 55 op/s 2024-09-16T00:20:11.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:11 smithi083 ceph-mon[35242]: pgmap v441: 129 pgs: 129 active+clean; 37 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 785 KiB/s wr, 55 op/s 2024-09-16T00:20:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:11 smithi060 ceph-mon[35185]: pgmap v441: 129 pgs: 129 active+clean; 37 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 785 KiB/s wr, 55 op/s 2024-09-16T00:20:12.649 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 3.14 MB/sec warmup 80 sec latency 4932.103 ms 2024-09-16T00:20:12.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:20:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:20:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:20:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:20:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:20:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:20:12.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:20:12.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:20:12.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:20:12.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:20:12.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:20:12.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:20:13.649 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 3.10 MB/sec warmup 81 sec latency 5932.205 ms 2024-09-16T00:20:13.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:13 smithi033 ceph-mon[31243]: pgmap v442: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 781 KiB/s wr, 58 op/s 2024-09-16T00:20:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:13 smithi083 ceph-mon[35242]: pgmap v442: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 781 KiB/s wr, 58 op/s 2024-09-16T00:20:13.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:13 smithi060 ceph-mon[35185]: pgmap v442: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 781 KiB/s wr, 58 op/s 2024-09-16T00:20:14.649 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 3.06 MB/sec warmup 82 sec latency 6932.318 ms 2024-09-16T00:20:15.649 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 3.03 MB/sec warmup 83 sec latency 7932.456 ms 2024-09-16T00:20:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:15 smithi033 ceph-mon[31243]: pgmap v443: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 664 KiB/s wr, 54 op/s 2024-09-16T00:20:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:15 smithi083 ceph-mon[35242]: pgmap v443: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 664 KiB/s wr, 54 op/s 2024-09-16T00:20:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:15 smithi060 ceph-mon[35185]: pgmap v443: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 664 KiB/s wr, 54 op/s 2024-09-16T00:20:16.650 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 2.99 MB/sec warmup 84 sec latency 8932.565 ms 2024-09-16T00:20:17.650 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 2.96 MB/sec warmup 85 sec latency 9932.616 ms 2024-09-16T00:20:17.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:17 smithi033 ceph-mon[31243]: pgmap v444: 129 pgs: 129 active+clean; 37 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 643 KiB/s wr, 52 op/s 2024-09-16T00:20:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:17 smithi083 ceph-mon[35242]: pgmap v444: 129 pgs: 129 active+clean; 37 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 643 KiB/s wr, 52 op/s 2024-09-16T00:20:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:17 smithi060 ceph-mon[35185]: pgmap v444: 129 pgs: 129 active+clean; 37 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 643 KiB/s wr, 52 op/s 2024-09-16T00:20:18.650 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 2.92 MB/sec warmup 86 sec latency 10932.731 ms 2024-09-16T00:20:19.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 2.89 MB/sec warmup 87 sec latency 11932.851 ms 2024-09-16T00:20:19.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:19 smithi033 ceph-mon[31243]: pgmap v445: 129 pgs: 129 active+clean; 38 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 340 KiB/s wr, 26 op/s 2024-09-16T00:20:19.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:19 smithi083 ceph-mon[35242]: pgmap v445: 129 pgs: 129 active+clean; 38 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 340 KiB/s wr, 26 op/s 2024-09-16T00:20:19.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:19 smithi060 ceph-mon[35185]: pgmap v445: 129 pgs: 129 active+clean; 38 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 340 KiB/s wr, 26 op/s 2024-09-16T00:20:20.650 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 2.86 MB/sec warmup 88 sec latency 12932.956 ms 2024-09-16T00:20:21.650 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 2.82 MB/sec warmup 89 sec latency 13933.065 ms 2024-09-16T00:20:21.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:21 smithi033 ceph-mon[31243]: pgmap v446: 129 pgs: 129 active+clean; 37 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 431 KiB/s wr, 29 op/s 2024-09-16T00:20:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:21 smithi083 ceph-mon[35242]: pgmap v446: 129 pgs: 129 active+clean; 37 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 431 KiB/s wr, 29 op/s 2024-09-16T00:20:21.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:21 smithi060 ceph-mon[35185]: pgmap v446: 129 pgs: 129 active+clean; 37 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 431 KiB/s wr, 29 op/s 2024-09-16T00:20:22.650 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 2.79 MB/sec warmup 90 sec latency 14933.171 ms 2024-09-16T00:20:23.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 2.76 MB/sec warmup 91 sec latency 15933.294 ms 2024-09-16T00:20:23.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:23 smithi033 ceph-mon[31243]: pgmap v447: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 284 KiB/s wr, 24 op/s 2024-09-16T00:20:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:23 smithi083 ceph-mon[35242]: pgmap v447: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 284 KiB/s wr, 24 op/s 2024-09-16T00:20:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:23 smithi060 ceph-mon[35185]: pgmap v447: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 284 KiB/s wr, 24 op/s 2024-09-16T00:20:24.580 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:20:24.650 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 2.73 MB/sec warmup 92 sec latency 16933.365 ms 2024-09-16T00:20:24.851 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:25.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.355+0000 7fbe78741640 1 -- 172.21.15.33:0/50361168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe70106a80 msgr2=0x7fbe70106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:25.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.355+0000 7fbe78741640 1 --2- 172.21.15.33:0/50361168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe70106a80 0x7fbe70106e60 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c009930 tx=0x7fbe6c02a440 comp rx=0 tx=0).stop 2024-09-16T00:20:25.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.355+0000 7fbe78741640 1 -- 172.21.15.33:0/50361168 shutdown_connections 2024-09-16T00:20:25.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.355+0000 7fbe78741640 1 --2- 172.21.15.33:0/50361168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe70101420 0x7fbe7010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:25.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.355+0000 7fbe78741640 1 --2- 172.21.15.33:0/50361168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe70100a80 0x7fbe70100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:25.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.355+0000 7fbe78741640 1 --2- 172.21.15.33:0/50361168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe70106a80 0x7fbe70106e60 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:25.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.355+0000 7fbe78741640 1 -- 172.21.15.33:0/50361168 >> 172.21.15.33:0/50361168 conn(0x7fbe700fc7a0 msgr2=0x7fbe700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:25.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.355+0000 7fbe78741640 1 -- 172.21.15.33:0/50361168 shutdown_connections 2024-09-16T00:20:25.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.355+0000 7fbe78741640 1 -- 172.21.15.33:0/50361168 wait complete. 2024-09-16T00:20:25.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.356+0000 7fbe78741640 1 Processor -- start 2024-09-16T00:20:25.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.356+0000 7fbe78741640 1 -- start start 2024-09-16T00:20:25.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.356+0000 7fbe78741640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe70100a80 0x7fbe70199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:25.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.356+0000 7fbe78741640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe70101420 0x7fbe70199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:25.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.356+0000 7fbe78741640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe70106a80 0x7fbe701a4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:25.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.356+0000 7fbe78741640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe70110e40 con 0x7fbe70101420 2024-09-16T00:20:25.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.356+0000 7fbe78741640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbe70110cc0 con 0x7fbe70106a80 2024-09-16T00:20:25.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.356+0000 7fbe78741640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbe70110fc0 con 0x7fbe70100a80 2024-09-16T00:20:25.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.356+0000 7fbe75cdc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe70101420 0x7fbe70199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:25.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.356+0000 7fbe75cdc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe70101420 0x7fbe70199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46800/0 (socket says 172.21.15.33:46800) 2024-09-16T00:20:25.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.356+0000 7fbe75cdc640 1 -- 172.21.15.33:0/1706673099 learned_addr learned my addr 172.21.15.33:0/1706673099 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:25.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.356+0000 7fbe76cde640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe70106a80 0x7fbe701a4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:25.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.357+0000 7fbe764dd640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe70100a80 0x7fbe70199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:25.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.357+0000 7fbe75cdc640 1 -- 172.21.15.33:0/1706673099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe70100a80 msgr2=0x7fbe70199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:25.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.357+0000 7fbe75cdc640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe70100a80 0x7fbe70199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:25.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.357+0000 7fbe75cdc640 1 -- 172.21.15.33:0/1706673099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe70106a80 msgr2=0x7fbe701a4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:25.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.357+0000 7fbe75cdc640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe70106a80 0x7fbe701a4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:25.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.357+0000 7fbe75cdc640 1 -- 172.21.15.33:0/1706673099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe701a47c0 con 0x7fbe70101420 2024-09-16T00:20:25.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.357+0000 7fbe75cdc640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe70101420 0x7fbe70199740 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000c920 tx=0x7fbe6000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:25.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.357+0000 7fbe677fe640 1 -- 172.21.15.33:0/1706673099 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60007c70 con 0x7fbe70101420 2024-09-16T00:20:25.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.357+0000 7fbe78741640 1 -- 172.21.15.33:0/1706673099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe701a4990 con 0x7fbe70101420 2024-09-16T00:20:25.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.357+0000 7fbe677fe640 1 -- 172.21.15.33:0/1706673099 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe60007e10 con 0x7fbe70101420 2024-09-16T00:20:25.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.357+0000 7fbe677fe640 1 -- 172.21.15.33:0/1706673099 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60018890 con 0x7fbe70101420 2024-09-16T00:20:25.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.357+0000 7fbe78741640 1 -- 172.21.15.33:0/1706673099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe701a4ea0 con 0x7fbe70101420 2024-09-16T00:20:25.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.358+0000 7fbe78741640 1 -- 172.21.15.33:0/1706673099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe70100ee0 con 0x7fbe70101420 2024-09-16T00:20:25.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.359+0000 7fbe677fe640 1 -- 172.21.15.33:0/1706673099 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbe60018a50 con 0x7fbe70101420 2024-09-16T00:20:25.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.360+0000 7fbe677fe640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbe5c0a3240 0x7fbe5c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:25.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.360+0000 7fbe764dd640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbe5c0a3240 0x7fbe5c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:25.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.360+0000 7fbe677fe640 1 -- 172.21.15.33:0/1706673099 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fbe600d0170 con 0x7fbe70101420 2024-09-16T00:20:25.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.360+0000 7fbe764dd640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbe5c0a3240 0x7fbe5c0a5700 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c000c00 tx=0x7fbe6c031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:25.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.363+0000 7fbe677fe640 1 -- 172.21.15.33:0/1706673099 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbe60014030 con 0x7fbe70101420 2024-09-16T00:20:25.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.573+0000 7fbe78741640 1 -- 172.21.15.33:0/1706673099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe701a5130 con 0x7fbe70101420 2024-09-16T00:20:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.574+0000 7fbe677fe640 1 -- 172.21.15.33:0/1706673099 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11906 (secure 0 0 0) 0x7fbe6009caa0 con 0x7fbe70101420 2024-09-16T00:20:25.574 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:20:25.574 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"btime":"2024-09-16T00:19:02:115556+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:19:01.460559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:20:25.575 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 28 2024-09-16T00:20:25.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.576+0000 7fbe78741640 1 -- 172.21.15.33:0/1706673099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbe5c0a3240 msgr2=0x7fbe5c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:25.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.576+0000 7fbe78741640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbe5c0a3240 0x7fbe5c0a5700 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c000c00 tx=0x7fbe6c031490 comp rx=0 tx=0).stop 2024-09-16T00:20:25.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.576+0000 7fbe78741640 1 -- 172.21.15.33:0/1706673099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe70101420 msgr2=0x7fbe70199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:25.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.576+0000 7fbe78741640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe70101420 0x7fbe70199740 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000c920 tx=0x7fbe6000cde0 comp rx=0 tx=0).stop 2024-09-16T00:20:25.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.576+0000 7fbe78741640 1 -- 172.21.15.33:0/1706673099 shutdown_connections 2024-09-16T00:20:25.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.576+0000 7fbe78741640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbe5c0a3240 0x7fbe5c0a5700 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:25.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.576+0000 7fbe78741640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe70106a80 0x7fbe701a4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:25.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.576+0000 7fbe78741640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe70101420 0x7fbe70199740 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:25.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.576+0000 7fbe78741640 1 --2- 172.21.15.33:0/1706673099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe70100a80 0x7fbe70199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:25.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.576+0000 7fbe78741640 1 -- 172.21.15.33:0/1706673099 >> 172.21.15.33:0/1706673099 conn(0x7fbe700fc7a0 msgr2=0x7fbe700fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:25.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.576+0000 7fbe78741640 1 -- 172.21.15.33:0/1706673099 shutdown_connections 2024-09-16T00:20:25.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:25.576+0000 7fbe78741640 1 -- 172.21.15.33:0/1706673099 wait complete. 2024-09-16T00:20:25.650 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 2.70 MB/sec warmup 93 sec latency 17933.410 ms 2024-09-16T00:20:25.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:25 smithi033 ceph-mon[31243]: pgmap v448: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 248 KiB/s wr, 17 op/s 2024-09-16T00:20:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:25 smithi083 ceph-mon[35242]: pgmap v448: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 248 KiB/s wr, 17 op/s 2024-09-16T00:20:25.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:25 smithi060 ceph-mon[35185]: pgmap v448: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 248 KiB/s wr, 17 op/s 2024-09-16T00:20:26.421 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id f41e73eb --release --await' 2024-09-16T00:20:26.421 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id f41e73eb --release --await 2024-09-16T00:20:26.650 INFO:tasks.workunit.client.0.smithi033.stdout: 1 43578 2.67 MB/sec warmup 94 sec latency 18933.438 ms 2024-09-16T00:20:26.683 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:26.804 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:26 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1706673099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:20:26.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:26 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1706673099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:20:26.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:26 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1706673099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:20:27.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.172+0000 7fb91a9c5640 1 -- 172.21.15.33:0/279411318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb914102790 msgr2=0x7fb9141067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:27.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.172+0000 7fb91a9c5640 1 --2- 172.21.15.33:0/279411318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb914102790 0x7fb9141067e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb908009900 tx=0x7fb90802a410 comp rx=0 tx=0).stop 2024-09-16T00:20:27.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.172+0000 7fb91a9c5640 1 -- 172.21.15.33:0/279411318 shutdown_connections 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.172+0000 7fb91a9c5640 1 --2- 172.21.15.33:0/279411318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb914106f10 0x7fb91410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.172+0000 7fb91a9c5640 1 --2- 172.21.15.33:0/279411318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb914102790 0x7fb9141067e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.172+0000 7fb91a9c5640 1 --2- 172.21.15.33:0/279411318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb914101de0 0x7fb9141021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.172+0000 7fb91a9c5640 1 -- 172.21.15.33:0/279411318 >> 172.21.15.33:0/279411318 conn(0x7fb9140fd8c0 msgr2=0x7fb9140ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.172+0000 7fb91a9c5640 1 -- 172.21.15.33:0/279411318 shutdown_connections 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.172+0000 7fb91a9c5640 1 -- 172.21.15.33:0/279411318 wait complete. 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.173+0000 7fb91a9c5640 1 Processor -- start 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.173+0000 7fb91a9c5640 1 -- start start 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.173+0000 7fb91a9c5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb914101de0 0x7fb914198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.173+0000 7fb91a9c5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb914102790 0x7fb9141988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.173+0000 7fb91a9c5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb914106f10 0x7fb91419cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.173+0000 7fb91a9c5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb914111f30 con 0x7fb914102790 2024-09-16T00:20:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.173+0000 7fb91a9c5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb914111db0 con 0x7fb914101de0 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.173+0000 7fb91a9c5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb9141120b0 con 0x7fb914106f10 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.173+0000 7fb9137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb914102790 0x7fb9141988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.173+0000 7fb9137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb914102790 0x7fb9141988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46814/0 (socket says 172.21.15.33:46814) 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.173+0000 7fb9137fe640 1 -- 172.21.15.33:0/872470361 learned_addr learned my addr 172.21.15.33:0/872470361 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb913fff640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb914101de0 0x7fb914198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb918f62640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb914106f10 0x7fb91419cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb9137fe640 1 -- 172.21.15.33:0/872470361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb914106f10 msgr2=0x7fb91419cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb9137fe640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb914106f10 0x7fb91419cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb9137fe640 1 -- 172.21.15.33:0/872470361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb914101de0 msgr2=0x7fb914198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb9137fe640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb914101de0 0x7fb914198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb9137fe640 1 -- 172.21.15.33:0/872470361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91419d340 con 0x7fb914102790 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb918f62640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb914106f10 0x7fb91419cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:20:27.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb913fff640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb914101de0 0x7fb914198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb9137fe640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb914102790 0x7fb9141988d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb908006010 tx=0x7fb9080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb9117fa640 1 -- 172.21.15.33:0/872470361 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90803f030 con 0x7fb914102790 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb9117fa640 1 -- 172.21.15.33:0/872470361 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb908003ce0 con 0x7fb914102790 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb9117fa640 1 -- 172.21.15.33:0/872470361 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb908035070 con 0x7fb914102790 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb91a9c5640 1 -- 172.21.15.33:0/872470361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91419d5d0 con 0x7fb914102790 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.174+0000 7fb91a9c5640 1 -- 172.21.15.33:0/872470361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9141a4f10 con 0x7fb914102790 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb9117fa640 1 -- 172.21.15.33:0/872470361 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb908042030 con 0x7fb914102790 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb91cd9e740 1 Processor -- start 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb91cd9e740 1 -- start start 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb91cd9e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 0x5649b9107320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb91cd9e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9107860 0x5649b9116110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb91cd9e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9116650 0x5649b9118a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb91cd9e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5649b906c620 con 0x5649b9116650 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb91cd9e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5649b9013e50 con 0x5649b9107860 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb91cd9e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5649b90685e0 con 0x5649b9106f40 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb9137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9116650 0x5649b9118a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb9137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9116650 0x5649b9118a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46816/0 (socket says 172.21.15.33:46816) 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb9137fe640 1 -- 172.21.15.33:0/2105246327 learned_addr learned my addr 172.21.15.33:0/2105246327 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb9117fa640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb8fc0a3240 0x7fb8fc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb918f62640 1 --2- 172.21.15.33:0/2105246327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 0x5649b9107320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.177+0000 7fb913fff640 1 --2- 172.21.15.33:0/2105246327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9107860 0x5649b9116110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.178+0000 7fb918f62640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb8fc0a3240 0x7fb8fc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.178+0000 7fb9117fa640 1 -- 172.21.15.33:0/872470361 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb9080f0460 con 0x7fb914102790 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.178+0000 7fb9137fe640 1 -- 172.21.15.33:0/2105246327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 msgr2=0x5649b9107320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.178+0000 7fb9137fe640 1 --2- 172.21.15.33:0/2105246327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 0x5649b9107320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.178+0000 7fb9137fe640 1 -- 172.21.15.33:0/2105246327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9107860 msgr2=0x5649b9116110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:27.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.178+0000 7fb9137fe640 1 --2- 172.21.15.33:0/2105246327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9107860 0x5649b9116110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.178+0000 7fb9137fe640 1 -- 172.21.15.33:0/2105246327 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649b9004d40 con 0x5649b9116650 2024-09-16T00:20:27.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.178+0000 7fb9137fe640 1 --2- 172.21.15.33:0/2105246327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9116650 0x5649b9118a40 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb9080bc6f0 tx=0x7fb90802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edbc8aceb38b912f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:27.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.178+0000 7fb8fa7fc640 1 -- 172.21.15.33:0/2105246327 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb908049070 con 0x5649b9116650 2024-09-16T00:20:27.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.178+0000 7fb918f62640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb8fc0a3240 0x7fb8fc0a5700 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0124b0 tx=0x7fb90c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:27.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.178+0000 7fb8fa7fc640 1 -- 172.21.15.33:0/2105246327 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb90803b580 con 0x5649b9116650 2024-09-16T00:20:27.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.178+0000 7fb8fa7fc640 1 -- 172.21.15.33:0/2105246327 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9080bc760 con 0x5649b9116650 2024-09-16T00:20:27.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.179+0000 7fb91cd9e740 1 -- 172.21.15.33:0/2105246327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9116650 msgr2=0x5649b9118a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:27.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.179+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/2105246327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9116650 0x5649b9118a40 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb9080bc6f0 tx=0x7fb90802a410 comp rx=0 tx=0).stop 2024-09-16T00:20:27.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.179+0000 7fb91cd9e740 1 -- 172.21.15.33:0/2105246327 shutdown_connections 2024-09-16T00:20:27.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.179+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/2105246327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9116650 0x5649b9118a40 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.179+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/2105246327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9107860 0x5649b9116110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.179+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/2105246327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 0x5649b9107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.179+0000 7fb91cd9e740 1 -- 172.21.15.33:0/2105246327 >> 172.21.15.33:0/2105246327 conn(0x5649b91035f0 msgr2=0x5649b91039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.179+0000 7fb91cd9e740 1 -- 172.21.15.33:0/2105246327 shutdown_connections 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.179+0000 7fb91cd9e740 1 -- 172.21.15.33:0/2105246327 wait complete. 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.179+0000 7fb91cd9e740 1 -- start start 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb91cd9e740 1 Processor -- start 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb91cd9e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 0x5649b9113610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb91cd9e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 0x5649b9113b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb91cd9e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9116650 0x5649b9114090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb91cd9e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5649b9118fe0 con 0x5649b9107860 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb91cd9e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5649b8ff4f10 con 0x5649b9116650 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb91cd9e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5649b9119160 con 0x5649b9106f40 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb91cd9e740 10 client.0 _ll_register_callbacks cb 0x5649b9018db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:27.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb9137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 0x5649b9113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb9137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 0x5649b9113b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46818/0 (socket says 172.21.15.33:46818) 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb913fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 0x5649b9113610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb9137fe640 1 -- 172.21.15.33:0/3557719108 learned_addr learned my addr 172.21.15.33:0/3557719108 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb9137fe640 1 -- 172.21.15.33:0/3557719108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 msgr2=0x5649b9113610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb9137fe640 1 --2- 172.21.15.33:0/3557719108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 0x5649b9113610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb9137fe640 1 -- 172.21.15.33:0/3557719108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9116650 msgr2=0x5649b9114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb9137fe640 1 --2- 172.21.15.33:0/3557719108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9116650 0x5649b9114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb9137fe640 1 -- 172.21.15.33:0/3557719108 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649b8d7b580 con 0x5649b9107860 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb9137fe640 1 --2- 172.21.15.33:0/3557719108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 0x5649b9113b50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb908003f90 tx=0x7fb908005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8af72890a7eb6f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.180+0000 7fb8f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb8f37fe640 1 -- 172.21.15.33:0/3557719108 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb908049050 con 0x5649b9107860 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb8f37fe640 1 -- 172.21.15.33:0/3557719108 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb9080b8dd0 con 0x5649b9107860 2024-09-16T00:20:27.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3557719108 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5649b9079b90 con 0x5649b9107860 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb8f37fe640 1 -- 172.21.15.33:0/3557719108 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb9080bc530 con 0x5649b9107860 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb91cd9e740 10 client.15927 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3557719108 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5649b8e45960 con 0x5649b9107860 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb8f37fe640 1 -- 172.21.15.33:0/3557719108 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5649b9079b90 con 0x5649b9107860 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb8f37fe640 1 -- 172.21.15.33:0/3557719108 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb9080b84b0 con 0x5649b9107860 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb91cd9e740 10 client.15927 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb91cd9e740 10 client.15927 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb91cd9e740 20 client.15927 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/3557719108 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5649b9125e00 0x5649b91281f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb91cd9e740 4 client.15927 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.181+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3557719108 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649b90740a0 con 0x5649b9125e00 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.182+0000 7fb913fff640 1 --2- 172.21.15.33:0/3557719108 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5649b9125e00 0x5649b91281f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.182+0000 7fb8f37fe640 10 client.15927 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.182+0000 7fb913fff640 1 --2- 172.21.15.33:0/3557719108 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5649b9125e00 0x5649b91281f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9a8235d34b5dd28 server_cookie=696a2f894f068cf9 in_seq=0 out_seq=0 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb8f37fe640 1 -- 172.21.15.33:0/3557719108 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5649b90740a0 con 0x5649b9125e00 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb8f37fe640 10 client.15927 handle_command_reply: tid=0 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb91cd9e740 1 client.15927 shutdown 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3557719108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 msgr2=0x5649b9113b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/3557719108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 0x5649b9113b50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb908003f90 tx=0x7fb908005750 comp rx=0 tx=0).stop 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3557719108 shutdown_connections 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/3557719108 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5649b9125e00 0x5649b91281f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/3557719108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9116650 0x5649b9114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb8f37fe640 0 client.15927 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/3557719108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 0x5649b9113b50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/3557719108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 0x5649b9113610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3557719108 >> 172.21.15.33:0/3557719108 conn(0x5649b91035f0 msgr2=0x5649b9109350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:27.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3557719108 shutdown_connections 2024-09-16T00:20:27.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3557719108 wait complete. 2024-09-16T00:20:27.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.183+0000 7fb91cd9e740 20 client.15927 trim_cache size 0 max 16384 2024-09-16T00:20:27.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.200+0000 7fb91cd9e740 1 Processor -- start 2024-09-16T00:20:27.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.200+0000 7fb91cd9e740 1 -- start start 2024-09-16T00:20:27.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.200+0000 7fb91cd9e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9106f40 0x5649b912da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.200+0000 7fb91cd9e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9107860 0x5649b912df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.200+0000 7fb91cd9e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9116650 0x5649b9110870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.200+0000 7fb91cd9e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb9141af700 con 0x5649b9106f40 2024-09-16T00:20:27.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.200+0000 7fb91cd9e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb9141acfd0 con 0x5649b9107860 2024-09-16T00:20:27.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.200+0000 7fb91cd9e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb9141aae20 con 0x5649b9116650 2024-09-16T00:20:27.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb913fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9106f40 0x5649b912da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb913fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9106f40 0x5649b912da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46824/0 (socket says 172.21.15.33:46824) 2024-09-16T00:20:27.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb913fff640 1 -- 172.21.15.33:0/3415634942 learned_addr learned my addr 172.21.15.33:0/3415634942 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:27.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb918f62640 1 --2- 172.21.15.33:0/3415634942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9116650 0x5649b9110870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb9137fe640 1 --2- 172.21.15.33:0/3415634942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9107860 0x5649b912df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb913fff640 1 -- 172.21.15.33:0/3415634942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9116650 msgr2=0x5649b9110870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:27.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb913fff640 1 --2- 172.21.15.33:0/3415634942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9116650 0x5649b9110870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb913fff640 1 -- 172.21.15.33:0/3415634942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9107860 msgr2=0x5649b912df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:27.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb913fff640 1 --2- 172.21.15.33:0/3415634942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9107860 0x5649b912df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb913fff640 1 -- 172.21.15.33:0/3415634942 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649b9107690 con 0x5649b9106f40 2024-09-16T00:20:27.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb913fff640 1 --2- 172.21.15.33:0/3415634942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9106f40 0x5649b912da20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb904005d00 tx=0x7fb90400beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206c641ed69130df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:27.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb8f9ffb640 1 -- 172.21.15.33:0/3415634942 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904015070 con 0x5649b9106f40 2024-09-16T00:20:27.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb8f9ffb640 1 -- 172.21.15.33:0/3415634942 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb904003e00 con 0x5649b9106f40 2024-09-16T00:20:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.201+0000 7fb8f9ffb640 1 -- 172.21.15.33:0/3415634942 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90401d610 con 0x5649b9106f40 2024-09-16T00:20:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.202+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3415634942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9106f40 msgr2=0x5649b912da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.202+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/3415634942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9106f40 0x5649b912da20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb904005d00 tx=0x7fb90400beb0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.202+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3415634942 shutdown_connections 2024-09-16T00:20:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.202+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/3415634942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9116650 0x5649b9110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.202+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/3415634942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9107860 0x5649b912df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.202+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/3415634942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9106f40 0x5649b912da20 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.202+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3415634942 >> 172.21.15.33:0/3415634942 conn(0x5649b91035f0 msgr2=0x5649b9101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.202+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3415634942 shutdown_connections 2024-09-16T00:20:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.202+0000 7fb91cd9e740 1 -- 172.21.15.33:0/3415634942 wait complete. 2024-09-16T00:20:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.202+0000 7fb91cd9e740 1 -- start start 2024-09-16T00:20:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb91cd9e740 1 Processor -- start 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb91cd9e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 0x5649b911a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb91cd9e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 0x5649b911ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb91cd9e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9116650 0x5649b918f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb91cd9e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb9141ab490 con 0x5649b9107860 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb91cd9e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb9141ac900 con 0x5649b9116650 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb91cd9e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb9141abb60 con 0x5649b9106f40 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb91cd9e740 10 client.0 _ll_register_callbacks cb 0x7fb9080f07d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb9137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 0x5649b911ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb9137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 0x5649b911ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46826/0 (socket says 172.21.15.33:46826) 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb9137fe640 1 -- 172.21.15.33:0/1157203973 learned_addr learned my addr 172.21.15.33:0/1157203973 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb913fff640 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 0x5649b911a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb918f62640 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9116650 0x5649b918f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb9137fe640 1 -- 172.21.15.33:0/1157203973 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 msgr2=0x5649b911a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb9137fe640 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 0x5649b911a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb9137fe640 1 -- 172.21.15.33:0/1157203973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9116650 msgr2=0x5649b918f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb9137fe640 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9116650 0x5649b918f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb9137fe640 1 -- 172.21.15.33:0/1157203973 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649b918ff30 con 0x5649b9107860 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.203+0000 7fb918f62640 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9116650 0x5649b918f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb9137fe640 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 0x5649b911ad00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb9080b8bc0 tx=0x7fb9080bc830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ddfef7a1a4b119 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb8f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb8f37fe640 1 -- 172.21.15.33:0/1157203973 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb908049040 con 0x5649b9107860 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb8f37fe640 1 -- 172.21.15.33:0/1157203973 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb9080c4f40 con 0x5649b9107860 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb91cd9e740 1 -- 172.21.15.33:0/1157203973 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb904003e00 con 0x5649b9107860 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb8f37fe640 1 -- 172.21.15.33:0/1157203973 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb9080b8a00 con 0x5649b9107860 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb91cd9e740 10 client.15933 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb91cd9e740 1 -- 172.21.15.33:0/1157203973 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5649b918ff30 con 0x5649b9107860 2024-09-16T00:20:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb8f37fe640 1 -- 172.21.15.33:0/1157203973 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb904003e00 con 0x5649b9107860 2024-09-16T00:20:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb8f37fe640 1 -- 172.21.15.33:0/1157203973 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb908081b30 con 0x5649b9107860 2024-09-16T00:20:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb91cd9e740 10 client.15933 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb91cd9e740 10 client.15933 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:20:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb91cd9e740 20 client.15933 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5649b9191590 0x5649b91939b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb91cd9e740 4 client.15933 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f41e73eb", "release": true, "await": true}] 2024-09-16T00:20:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.204+0000 7fb91cd9e740 1 -- 172.21.15.33:0/1157203973 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f41e73eb", "release": true, "await": true}) -- 0x5649b8ff05a0 con 0x5649b9191590 2024-09-16T00:20:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.205+0000 7fb913fff640 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5649b9191590 0x5649b91939b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.205+0000 7fb913fff640 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5649b9191590 0x5649b91939b0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33c77e018d25e958 server_cookie=4f536f0f9ff460f2 in_seq=0 out_seq=0 2024-09-16T00:20:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.205+0000 7fb8f37fe640 10 client.15933 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:27.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb8f37fe640 1 -- 172.21.15.33:0/1157203973 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5649b8ff05a0 con 0x5649b9191590 2024-09-16T00:20:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb8f37fe640 10 client.15933 handle_command_reply: tid=0 2024-09-16T00:20:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb91cd9e740 1 client.15933 shutdown 2024-09-16T00:20:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb91cd9e740 1 -- 172.21.15.33:0/1157203973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 msgr2=0x5649b911ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 0x5649b911ad00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb9080b8bc0 tx=0x7fb9080bc830 comp rx=0 tx=0).stop 2024-09-16T00:20:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb91cd9e740 1 -- 172.21.15.33:0/1157203973 shutdown_connections 2024-09-16T00:20:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5649b9191590 0x5649b91939b0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b9116650 0x5649b918f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b9107860 0x5649b911ad00 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb91cd9e740 1 --2- 172.21.15.33:0/1157203973 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b9106f40 0x5649b911a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb91cd9e740 1 -- 172.21.15.33:0/1157203973 >> 172.21.15.33:0/1157203973 conn(0x5649b91035f0 msgr2=0x5649b9111db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb8f37fe640 0 client.15933 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:27.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb91cd9e740 1 -- 172.21.15.33:0/1157203973 shutdown_connections 2024-09-16T00:20:27.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb91cd9e740 1 -- 172.21.15.33:0/1157203973 wait complete. 2024-09-16T00:20:27.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.213+0000 7fb91cd9e740 20 client.15933 trim_cache size 0 max 16384 2024-09-16T00:20:27.215 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:20:27.215 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 28, 2024-09-16T00:20:27.215 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:20:27.215 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 12, 2024-09-16T00:20:27.215 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:20:27.215 INFO:teuthology.orchestra.run.smithi033.stdout: "f41e73eb": { 2024-09-16T00:20:27.216 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 12, 2024-09-16T00:20:27.216 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:20:27.216 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:20:27.216 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:20:27.216 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:20:27.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:20:27.216 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:20:27.216 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 139.6, 2024-09-16T00:20:27.216 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:20:27.216 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:20:27.217 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:20:27.217 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:20:27.217 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:20:27.217 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:20:27.217 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:27.217 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:27.217 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:27.217 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:27.217 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:27.217 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:20:27.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.214+0000 7fb91a9c5640 1 -- 172.21.15.33:0/872470361 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb8fc0a3240 msgr2=0x7fb8fc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:27.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.214+0000 7fb91a9c5640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb8fc0a3240 0x7fb8fc0a5700 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0124b0 tx=0x7fb90c005f90 comp rx=0 tx=0).stop 2024-09-16T00:20:27.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.214+0000 7fb91a9c5640 1 -- 172.21.15.33:0/872470361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb914102790 msgr2=0x7fb9141988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:27.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.214+0000 7fb91a9c5640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb914102790 0x7fb9141988d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb908006010 tx=0x7fb9080042f0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.214+0000 7fb91a9c5640 1 -- 172.21.15.33:0/872470361 shutdown_connections 2024-09-16T00:20:27.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.214+0000 7fb91a9c5640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb8fc0a3240 0x7fb8fc0a5700 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.214+0000 7fb91a9c5640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb914106f10 0x7fb91419cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.214+0000 7fb91a9c5640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb914102790 0x7fb9141988d0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.214+0000 7fb91a9c5640 1 --2- 172.21.15.33:0/872470361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb914101de0 0x7fb914198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:27.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.214+0000 7fb91a9c5640 1 -- 172.21.15.33:0/872470361 >> 172.21.15.33:0/872470361 conn(0x7fb9140fd8c0 msgr2=0x7fb9140fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:27.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.214+0000 7fb91a9c5640 1 -- 172.21.15.33:0/872470361 shutdown_connections 2024-09-16T00:20:27.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:27.214+0000 7fb91a9c5640 1 -- 172.21.15.33:0/872470361 wait complete. 2024-09-16T00:20:27.263 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:20:27.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 45371 2.75 MB/sec warmup 95 sec latency 19497.540 ms 2024-09-16T00:20:27.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:27 smithi083 ceph-mon[35242]: pgmap v449: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 250 KiB/s wr, 16 op/s 2024-09-16T00:20:27.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:27 smithi060 ceph-mon[35185]: pgmap v449: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 250 KiB/s wr, 16 op/s 2024-09-16T00:20:28.016 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:27 smithi033 ceph-mon[31243]: pgmap v449: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 250 KiB/s wr, 16 op/s 2024-09-16T00:20:28.051 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'f41e73eb', total seconds elapsed: 23.0 2024-09-16T00:20:28.071 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:28.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.610+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/2581350395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb74102570 msgr2=0x7fdb741029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:28.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.610+0000 7fdb7c6d6640 1 --2- 172.21.15.33:0/2581350395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb74102570 0x7fdb741029b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009900 tx=0x7fdb6802a480 comp rx=0 tx=0).stop 2024-09-16T00:20:28.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.610+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/2581350395 shutdown_connections 2024-09-16T00:20:28.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.610+0000 7fdb7c6d6640 1 --2- 172.21.15.33:0/2581350395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74102ef0 0x7fdb7410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.610+0000 7fdb7c6d6640 1 --2- 172.21.15.33:0/2581350395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb74102570 0x7fdb741029b0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.610+0000 7fdb7c6d6640 1 --2- 172.21.15.33:0/2581350395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb74100e20 0x7fdb74101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.610+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/2581350395 >> 172.21.15.33:0/2581350395 conn(0x7fdb740fc7d0 msgr2=0x7fdb740febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.610+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/2581350395 shutdown_connections 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.610+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/2581350395 wait complete. 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.611+0000 7fdb7c6d6640 1 Processor -- start 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.611+0000 7fdb7c6d6640 1 -- start start 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.611+0000 7fdb7c6d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb74100e20 0x7fdb741993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb7c6d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb74102570 0x7fdb741998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb7c6d6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74102ef0 0x7fdb741a4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb7c6d6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdb74110f50 con 0x7fdb74102570 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb7c6d6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdb74110dd0 con 0x7fdb74102ef0 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb7c6d6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb741110d0 con 0x7fdb74100e20 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb79c71640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb74102570 0x7fdb741998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb7a472640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb74100e20 0x7fdb741993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb79c71640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb74102570 0x7fdb741998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46834/0 (socket says 172.21.15.33:46834) 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb79c71640 1 -- 172.21.15.33:0/1766447334 learned_addr learned my addr 172.21.15.33:0/1766447334 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb7a472640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb74100e20 0x7fdb741993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60718/0 (socket says 172.21.15.33:60718) 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb7ac73640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74102ef0 0x7fdb741a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb79c71640 1 -- 172.21.15.33:0/1766447334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb74100e20 msgr2=0x7fdb741993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb79c71640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb74100e20 0x7fdb741993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb79c71640 1 -- 172.21.15.33:0/1766447334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74102ef0 msgr2=0x7fdb741a4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb79c71640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74102ef0 0x7fdb741a4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb79c71640 1 -- 172.21.15.33:0/1766447334 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb741a4960 con 0x7fdb74102570 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb79c71640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb74102570 0x7fdb741998e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdb68006010 tx=0x7fdb68003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb737fe640 1 -- 172.21.15.33:0/1766447334 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6803f070 con 0x7fdb74102570 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb737fe640 1 -- 172.21.15.33:0/1766447334 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb68031c30 con 0x7fdb74102570 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.612+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/1766447334 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb741a4b30 con 0x7fdb74102570 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.613+0000 7fdb737fe640 1 -- 172.21.15.33:0/1766447334 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68035030 con 0x7fdb74102570 2024-09-16T00:20:28.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.613+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/1766447334 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb741a4f80 con 0x7fdb74102570 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.615+0000 7fdb737fe640 1 -- 172.21.15.33:0/1766447334 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fdb68042070 con 0x7fdb74102570 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.615+0000 7fdb7eaaf740 1 Processor -- start 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.615+0000 7fdb7eaaf740 1 -- start start 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7eaaf740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c177760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7eaaf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 0x55b77c186550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7eaaf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c186a90 0x55b77c188e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7eaaf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b77c0dd2e0 con 0x55b77c177ca0 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7eaaf740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55b77c084ce0 con 0x55b77c177380 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7eaaf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b77c0d92a0 con 0x55b77c186a90 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7a472640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 0x55b77c186550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7ac73640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c177760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7a472640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 0x55b77c186550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46838/0 (socket says 172.21.15.33:46838) 2024-09-16T00:20:28.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7a472640 1 -- 172.21.15.33:0/3485411666 learned_addr learned my addr 172.21.15.33:0/3485411666 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb79c71640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c186a90 0x55b77c188e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb737fe640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdb5c0a3240 0x7fdb5c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7a472640 1 -- 172.21.15.33:0/3485411666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c186a90 msgr2=0x55b77c188e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7a472640 1 --2- 172.21.15.33:0/3485411666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c186a90 0x55b77c188e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7a472640 1 -- 172.21.15.33:0/3485411666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 msgr2=0x55b77c177760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7a472640 1 --2- 172.21.15.33:0/3485411666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c177760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7a472640 1 -- 172.21.15.33:0/3485411666 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b77c069440 con 0x55b77c177ca0 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7ac73640 1 --2- 172.21.15.33:0/3485411666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c177760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb737fe640 1 -- 172.21.15.33:0/1766447334 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fdb680f04a0 con 0x7fdb74102570 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.616+0000 7fdb7ac73640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdb5c0a3240 0x7fdb5c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.617+0000 7fdb7a472640 1 --2- 172.21.15.33:0/3485411666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 0x55b77c186550 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fdb640114f0 tx=0x7fdb640119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632aa6d07db297bc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.617+0000 7fdb70ff9640 1 -- 172.21.15.33:0/3485411666 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6401e070 con 0x55b77c177ca0 2024-09-16T00:20:28.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.617+0000 7fdb70ff9640 1 -- 172.21.15.33:0/3485411666 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb64003c30 con 0x55b77c177ca0 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.617+0000 7fdb70ff9640 1 -- 172.21.15.33:0/3485411666 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6400fa70 con 0x55b77c177ca0 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/3485411666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 msgr2=0x55b77c186550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7ac73640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdb5c0a3240 0x7fdb5c0a5700 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c009e60 tx=0x7fdb6c0046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/3485411666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 0x55b77c186550 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fdb640114f0 tx=0x7fdb640119b0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/3485411666 shutdown_connections 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/3485411666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c186a90 0x55b77c188e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/3485411666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 0x55b77c186550 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/3485411666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c177760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/3485411666 >> 172.21.15.33:0/3485411666 conn(0x55b77c173a30 msgr2=0x55b77c173e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/3485411666 shutdown_connections 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/3485411666 wait complete. 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7eaaf740 1 -- start start 2024-09-16T00:20:28.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7eaaf740 1 Processor -- start 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.618+0000 7fdb7eaaf740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c186420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7eaaf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 0x55b77c1836d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7eaaf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c186a90 0x55b77c183c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7eaaf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b77c189660 con 0x55b77c177ca0 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7eaaf740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55b77c098050 con 0x55b77c177380 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7eaaf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b77c1897e0 con 0x55b77c186a90 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7eaaf740 10 client.0 _ll_register_callbacks cb 0x55b77c0d8160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb79c71640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 0x55b77c1836d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb79c71640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 0x55b77c1836d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46846/0 (socket says 172.21.15.33:46846) 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7a472640 1 --2- 172.21.15.33:0/1724415780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c186420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7ac73640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c186a90 0x55b77c183c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb79c71640 1 -- 172.21.15.33:0/1724415780 learned_addr learned my addr 172.21.15.33:0/1724415780 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7a472640 1 -- 172.21.15.33:0/1724415780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c186a90 msgr2=0x55b77c183c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7a472640 1 --2- 172.21.15.33:0/1724415780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c186a90 0x55b77c183c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7a472640 1 -- 172.21.15.33:0/1724415780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 msgr2=0x55b77c1836d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7a472640 1 --2- 172.21.15.33:0/1724415780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 0x55b77c1836d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7a472640 1 -- 172.21.15.33:0/1724415780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b77be4e3c0 con 0x55b77c177380 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb7a472640 1 --2- 172.21.15.33:0/1724415780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c186420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdb6401c040 tx=0x7fdb64004280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4c3289e9e38b3c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb59ffb640 1 -- 172.21.15.33:0/1724415780 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb64028060 con 0x55b77c177380 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.619+0000 7fdb59ffb640 1 -- 172.21.15.33:0/1724415780 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fdb6400f770 con 0x55b77c177380 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1724415780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb64003c30 con 0x55b77c177380 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb59ffb640 1 -- 172.21.15.33:0/1724415780 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdb6402f050 con 0x55b77c177380 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb7eaaf740 10 client.25523 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1724415780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b77c1950f0 con 0x55b77c177380 2024-09-16T00:20:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb59ffb640 1 -- 172.21.15.33:0/1724415780 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb64003c30 con 0x55b77c177380 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb59ffb640 1 -- 172.21.15.33:0/1724415780 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdb64005040 con 0x55b77c177380 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb7eaaf740 10 client.25523 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb7eaaf740 10 client.25523 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb7eaaf740 20 client.25523 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/1724415780 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b77c1965b0 0x55b77c1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb7eaaf740 4 client.25523 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1724415780 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b77c0ea850 con 0x55b77c1965b0 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.620+0000 7fdb79c71640 1 --2- 172.21.15.33:0/1724415780 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b77c1965b0 0x55b77c1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.621+0000 7fdb59ffb640 10 client.25523 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.621+0000 7fdb79c71640 1 --2- 172.21.15.33:0/1724415780 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b77c1965b0 0x55b77c1989a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb2f3a3b6676a89b server_cookie=7dcbcf86a81ddb4e in_seq=0 out_seq=0 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.621+0000 7fdb59ffb640 1 -- 172.21.15.33:0/1724415780 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b77c0ea850 con 0x55b77c1965b0 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.621+0000 7fdb59ffb640 10 client.25523 handle_command_reply: tid=0 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.621+0000 7fdb7eaaf740 1 client.25523 shutdown 2024-09-16T00:20:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.622+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1724415780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 msgr2=0x55b77c186420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.622+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/1724415780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c186420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdb6401c040 tx=0x7fdb64004280 comp rx=0 tx=0).stop 2024-09-16T00:20:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.622+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1724415780 shutdown_connections 2024-09-16T00:20:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.622+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/1724415780 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b77c1965b0 0x55b77c1989a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.622+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/1724415780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c186a90 0x55b77c183c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.622+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/1724415780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c177ca0 0x55b77c1836d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.622+0000 7fdb59ffb640 0 client.25523 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.622+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/1724415780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c186420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.622+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1724415780 >> 172.21.15.33:0/1724415780 conn(0x55b77c173a30 msgr2=0x55b77c179790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.622+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1724415780 shutdown_connections 2024-09-16T00:20:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.622+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1724415780 wait complete. 2024-09-16T00:20:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.622+0000 7fdb7eaaf740 20 client.25523 trim_cache size 0 max 16384 2024-09-16T00:20:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.642+0000 7fdb7eaaf740 1 Processor -- start 2024-09-16T00:20:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.642+0000 7fdb7eaaf740 1 -- start start 2024-09-16T00:20:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.642+0000 7fdb7eaaf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c177380 0x55b77c192900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.642+0000 7fdb7eaaf740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177ca0 0x55b77c19e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.642+0000 7fdb7eaaf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 0x55b77c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.642+0000 7fdb7eaaf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdb741af780 con 0x55b77c186a90 2024-09-16T00:20:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb7eaaf740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdb741ad050 con 0x55b77c177ca0 2024-09-16T00:20:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb7eaaf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb741aaea0 con 0x55b77c177380 2024-09-16T00:20:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb7a472640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c177380 0x55b77c192900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb7ac73640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 0x55b77c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb79c71640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177ca0 0x55b77c19e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb7a472640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c177380 0x55b77c192900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60754/0 (socket says 172.21.15.33:60754) 2024-09-16T00:20:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb7ac73640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 0x55b77c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46850/0 (socket says 172.21.15.33:46850) 2024-09-16T00:20:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb7a472640 1 -- 172.21.15.33:0/1671351237 learned_addr learned my addr 172.21.15.33:0/1671351237 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb79c71640 1 -- 172.21.15.33:0/1671351237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c177380 msgr2=0x55b77c192900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb79c71640 1 --2- 172.21.15.33:0/1671351237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c177380 0x55b77c192900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb79c71640 1 -- 172.21.15.33:0/1671351237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 msgr2=0x55b77c19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb79c71640 1 --2- 172.21.15.33:0/1671351237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 0x55b77c19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb79c71640 1 -- 172.21.15.33:0/1671351237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b77c1734d0 con 0x55b77c177ca0 2024-09-16T00:20:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb7a472640 1 --2- 172.21.15.33:0/1671351237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c177380 0x55b77c192900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb79c71640 1 --2- 172.21.15.33:0/1671351237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177ca0 0x55b77c19e110 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdb680bc340 tx=0x7fdb680ed9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c80a59e71f256fb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.643+0000 7fdb5bfff640 1 -- 172.21.15.33:0/1671351237 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68049070 con 0x55b77c177ca0 2024-09-16T00:20:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.644+0000 7fdb5bfff640 1 -- 172.21.15.33:0/1671351237 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb68031de0 con 0x55b77c177ca0 2024-09-16T00:20:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.644+0000 7fdb5bfff640 1 -- 172.21.15.33:0/1671351237 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb680c09f0 con 0x55b77c177ca0 2024-09-16T00:20:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.644+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1671351237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177ca0 msgr2=0x55b77c19e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.644+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/1671351237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177ca0 0x55b77c19e110 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdb680bc340 tx=0x7fdb680ed9f0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.644+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1671351237 shutdown_connections 2024-09-16T00:20:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.644+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/1671351237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 0x55b77c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.644+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/1671351237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177ca0 0x55b77c19e110 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.644+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/1671351237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c177380 0x55b77c192900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.644+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1671351237 >> 172.21.15.33:0/1671351237 conn(0x55b77c173a30 msgr2=0x55b77c1827c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.644+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1671351237 shutdown_connections 2024-09-16T00:20:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.644+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/1671351237 wait complete. 2024-09-16T00:20:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7eaaf740 1 -- start start 2024-09-16T00:20:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7eaaf740 1 Processor -- start 2024-09-16T00:20:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7eaaf740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c17de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7eaaf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c177ca0 0x55b77c18ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7eaaf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 0x55b77c18eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7eaaf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdb741ab510 con 0x55b77c186a90 2024-09-16T00:20:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7eaaf740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdb741ac980 con 0x55b77c177380 2024-09-16T00:20:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7eaaf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb741abbe0 con 0x55b77c177ca0 2024-09-16T00:20:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7eaaf740 10 client.0 _ll_register_callbacks cb 0x7fdb6401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7ac73640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 0x55b77c18eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7ac73640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 0x55b77c18eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46866/0 (socket says 172.21.15.33:46866) 2024-09-16T00:20:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb79c71640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c177ca0 0x55b77c18ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7ac73640 1 -- 172.21.15.33:0/2612128752 learned_addr learned my addr 172.21.15.33:0/2612128752 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.645+0000 7fdb7a472640 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c17de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb7ac73640 1 -- 172.21.15.33:0/2612128752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c177ca0 msgr2=0x55b77c18ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb7ac73640 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c177ca0 0x55b77c18ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb7ac73640 1 -- 172.21.15.33:0/2612128752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 msgr2=0x55b77c17de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb7ac73640 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c17de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb7ac73640 1 -- 172.21.15.33:0/2612128752 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b77c17a1a0 con 0x55b77c186a90 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb7a472640 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c17de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb7ac73640 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 0x55b77c18eff0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0127b0 tx=0x7fdb6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ddf2dc4372782e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb59ffb640 1 -- 172.21.15.33:0/2612128752 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c02f060 con 0x55b77c186a90 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb59ffb640 1 -- 172.21.15.33:0/2612128752 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fdb6c011ac0 con 0x55b77c186a90 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/2612128752 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb68031de0 con 0x55b77c186a90 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb59ffb640 1 -- 172.21.15.33:0/2612128752 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdb6c011d00 con 0x55b77c186a90 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb7eaaf740 10 client.15951 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/2612128752 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b77c205b70 con 0x55b77c186a90 2024-09-16T00:20:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.646+0000 7fdb59ffb640 1 -- 172.21.15.33:0/2612128752 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb68031de0 con 0x55b77c186a90 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb59ffb640 1 -- 172.21.15.33:0/2612128752 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdb6c029030 con 0x55b77c186a90 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb7eaaf740 10 client.15951 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb7eaaf740 10 client.15951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb7eaaf740 20 client.15951 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b77c206300 0x55b77c208720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb7eaaf740 4 client.15951 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/2612128752 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b77c0e4d60 con 0x55b77c206300 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb7a472640 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b77c206300 0x55b77c208720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb59ffb640 10 client.15951 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb7a472640 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b77c206300 0x55b77c208720 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2a8155d07887e83 server_cookie=553150f05cadd501 in_seq=0 out_seq=0 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb59ffb640 1 -- 172.21.15.33:0/2612128752 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b77c0e4d60 con 0x55b77c206300 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb59ffb640 10 client.15951 handle_command_reply: tid=0 2024-09-16T00:20:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.647+0000 7fdb7eaaf740 1 client.15951 shutdown 2024-09-16T00:20:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.648+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/2612128752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 msgr2=0x55b77c18eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.648+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 0x55b77c18eff0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0127b0 tx=0x7fdb6c004730 comp rx=0 tx=0).stop 2024-09-16T00:20:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.648+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/2612128752 shutdown_connections 2024-09-16T00:20:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.648+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b77c206300 0x55b77c208720 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.648+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b77c186a90 0x55b77c18eff0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.648+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b77c177ca0 0x55b77c18ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.648+0000 7fdb7eaaf740 1 --2- 172.21.15.33:0/2612128752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b77c177380 0x55b77c17de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.648+0000 7fdb59ffb640 0 client.15951 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.648+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/2612128752 >> 172.21.15.33:0/2612128752 conn(0x55b77c173a30 msgr2=0x55b77c172350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.648+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/2612128752 shutdown_connections 2024-09-16T00:20:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.648+0000 7fdb7eaaf740 1 -- 172.21.15.33:0/2612128752 wait complete. 2024-09-16T00:20:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.648+0000 7fdb7eaaf740 20 client.15951 trim_cache size 0 max 16384 2024-09-16T00:20:28.661 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:20:28.661 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:20:28.661 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:20:28.661 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:20:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.649+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/1766447334 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdb5c0a3240 msgr2=0x7fdb5c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.649+0000 7fdb7c6d6640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdb5c0a3240 0x7fdb5c0a5700 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c009e60 tx=0x7fdb6c0046c0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.649+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/1766447334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb74102570 msgr2=0x7fdb741998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.649+0000 7fdb7c6d6640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb74102570 0x7fdb741998e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdb68006010 tx=0x7fdb68003c10 comp rx=0 tx=0).stop 2024-09-16T00:20:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.649+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/1766447334 shutdown_connections 2024-09-16T00:20:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.649+0000 7fdb7c6d6640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdb5c0a3240 0x7fdb5c0a5700 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.649+0000 7fdb7c6d6640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74102ef0 0x7fdb741a4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.649+0000 7fdb7c6d6640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb74102570 0x7fdb741998e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.649+0000 7fdb7c6d6640 1 --2- 172.21.15.33:0/1766447334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb74100e20 0x7fdb741993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:28.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.649+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/1766447334 >> 172.21.15.33:0/1766447334 conn(0x7fdb740fc7d0 msgr2=0x7fdb740febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:28.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.649+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/1766447334 shutdown_connections 2024-09-16T00:20:28.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:28.649+0000 7fdb7c6d6640 1 -- 172.21.15.33:0/1766447334 wait complete. 2024-09-16T00:20:28.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 48182 2.87 MB/sec warmup 96 sec latency 11.868 ms 2024-09-16T00:20:29.540 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:20:29.540 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:20:29.650 INFO:tasks.workunit.client.0.smithi033.stdout: 1 50948 3.00 MB/sec warmup 97 sec latency 16.282 ms 2024-09-16T00:20:29.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:29 smithi033 ceph-mon[31243]: pgmap v450: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 126 KiB/s wr, 13 op/s 2024-09-16T00:20:29.857 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:29.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:29 smithi083 ceph-mon[35242]: pgmap v450: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 126 KiB/s wr, 13 op/s 2024-09-16T00:20:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:29 smithi060 ceph-mon[35185]: pgmap v450: 129 pgs: 129 active+clean; 36 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 126 KiB/s wr, 13 op/s 2024-09-16T00:20:30.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.354+0000 7f5d077fe640 1 -- 172.21.15.33:0/2562204026 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc031940 con 0x7f5d08106bd0 2024-09-16T00:20:30.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.357+0000 7f5d0ef03640 1 -- 172.21.15.33:0/2562204026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d08106bd0 msgr2=0x7f5d08106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:30.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.357+0000 7f5d0ef03640 1 --2- 172.21.15.33:0/2562204026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d08106bd0 0x7f5d08106fb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc009900 tx=0x7f5cfc02a3f0 comp rx=0 tx=0).stop 2024-09-16T00:20:30.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.357+0000 7f5d0ef03640 1 -- 172.21.15.33:0/2562204026 shutdown_connections 2024-09-16T00:20:30.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.357+0000 7f5d0ef03640 1 --2- 172.21.15.33:0/2562204026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d081005f0 0x7f5d0810ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:30.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.357+0000 7f5d0ef03640 1 --2- 172.21.15.33:0/2562204026 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d080ffc50 0x7f5d081000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:30.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.357+0000 7f5d0ef03640 1 --2- 172.21.15.33:0/2562204026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d08106bd0 0x7f5d08106fb0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:30.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.357+0000 7f5d0ef03640 1 -- 172.21.15.33:0/2562204026 >> 172.21.15.33:0/2562204026 conn(0x7f5d080fbaa0 msgr2=0x7f5d080fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:30.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.358+0000 7f5d0ef03640 1 -- 172.21.15.33:0/2562204026 shutdown_connections 2024-09-16T00:20:30.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.359+0000 7f5d0ef03640 1 -- 172.21.15.33:0/2562204026 wait complete. 2024-09-16T00:20:30.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.360+0000 7f5d0ef03640 1 Processor -- start 2024-09-16T00:20:30.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.360+0000 7f5d0ef03640 1 -- start start 2024-09-16T00:20:30.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.360+0000 7f5d0ef03640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d080ffc50 0x7f5d0810e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:30.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.361+0000 7f5d0cc9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d080ffc50 0x7f5d0810e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:30.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.361+0000 7f5d0cc9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d080ffc50 0x7f5d0810e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48652/0 (socket says 172.21.15.33:48652) 2024-09-16T00:20:30.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.361+0000 7f5d0cc9f640 1 -- 172.21.15.33:0/560625221 learned_addr learned my addr 172.21.15.33:0/560625221 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:30.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.361+0000 7f5d0ef03640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d081005f0 0x7f5d0810b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:30.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.361+0000 7f5d0ef03640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d08106bd0 0x7f5d0810bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:30.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.361+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5d08103140 con 0x7f5d080ffc50 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.361+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5d08102fc0 con 0x7f5d081005f0 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.361+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d081032c0 con 0x7f5d08106bd0 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.362+0000 7f5d07fff640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d081005f0 0x7f5d0810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.362+0000 7f5d0cc9f640 1 -- 172.21.15.33:0/560625221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d08106bd0 msgr2=0x7f5d0810bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.362+0000 7f5d0cc9f640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d08106bd0 0x7f5d0810bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.362+0000 7f5d0cc9f640 1 -- 172.21.15.33:0/560625221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d081005f0 msgr2=0x7f5d0810b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.362+0000 7f5d0cc9f640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d081005f0 0x7f5d0810b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.362+0000 7f5d0cc9f640 1 -- 172.21.15.33:0/560625221 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0810c510 con 0x7f5d080ffc50 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.362+0000 7f5d07fff640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d081005f0 0x7f5d0810b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.362+0000 7f5d0cc9f640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d080ffc50 0x7f5d0810e7a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc030650 tx=0x7f5cfc030680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.363+0000 7f5d05ffb640 1 -- 172.21.15.33:0/560625221 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc03f070 con 0x7f5d080ffc50 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.363+0000 7f5d05ffb640 1 -- 172.21.15.33:0/560625221 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cfc004460 con 0x7f5d080ffc50 2024-09-16T00:20:30.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.363+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d081aae30 con 0x7f5d080ffc50 2024-09-16T00:20:30.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.363+0000 7f5d05ffb640 1 -- 172.21.15.33:0/560625221 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc035030 con 0x7f5d080ffc50 2024-09-16T00:20:30.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.363+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d081ab300 con 0x7f5d080ffc50 2024-09-16T00:20:30.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.365+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d081010d0 con 0x7f5d080ffc50 2024-09-16T00:20:30.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.367+0000 7f5d05ffb640 1 -- 172.21.15.33:0/560625221 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5cfc042070 con 0x7f5d080ffc50 2024-09-16T00:20:30.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.368+0000 7f5d05ffb640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5cf80a32c0 0x7f5cf80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:30.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.368+0000 7f5d05ffb640 1 -- 172.21.15.33:0/560625221 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5cfc0f04d0 con 0x7f5d080ffc50 2024-09-16T00:20:30.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.369+0000 7f5d07fff640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5cf80a32c0 0x7f5cf80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:30.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.371+0000 7f5d07fff640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5cf80a32c0 0x7f5cf80a5780 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5d0810d010 tx=0x7f5cf4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:30.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.377+0000 7f5d05ffb640 1 -- 172.21.15.33:0/560625221 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5cfc049070 con 0x7f5d080ffc50 2024-09-16T00:20:30.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.612+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d081ab900 con 0x7f5d080ffc50 2024-09-16T00:20:30.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.613+0000 7f5d05ffb640 1 -- 172.21.15.33:0/560625221 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11906 (secure 0 0 0) 0x7f5cfc0bcd90 con 0x7f5d080ffc50 2024-09-16T00:20:30.613 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:20:30.613 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"btime":"2024-09-16T00:19:02:115556+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:19:01.460559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:20:30.613 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 28 2024-09-16T00:20:30.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.615+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5cf80a32c0 msgr2=0x7f5cf80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:30.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.615+0000 7f5d0ef03640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5cf80a32c0 0x7f5cf80a5780 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5d0810d010 tx=0x7f5cf4005f90 comp rx=0 tx=0).stop 2024-09-16T00:20:30.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.615+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d080ffc50 msgr2=0x7f5d0810e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:30.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.615+0000 7f5d0ef03640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d080ffc50 0x7f5d0810e7a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc030650 tx=0x7f5cfc030680 comp rx=0 tx=0).stop 2024-09-16T00:20:30.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.615+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 shutdown_connections 2024-09-16T00:20:30.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.615+0000 7f5d0ef03640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5cf80a32c0 0x7f5cf80a5780 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:30.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.615+0000 7f5d0ef03640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d08106bd0 0x7f5d0810bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:30.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.615+0000 7f5d0ef03640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d081005f0 0x7f5d0810b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:30.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.615+0000 7f5d0ef03640 1 --2- 172.21.15.33:0/560625221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d080ffc50 0x7f5d0810e7a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:30.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.615+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 >> 172.21.15.33:0/560625221 conn(0x7f5d080fbaa0 msgr2=0x7f5d080fbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:30.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.615+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 shutdown_connections 2024-09-16T00:20:30.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:30.615+0000 7f5d0ef03640 1 -- 172.21.15.33:0/560625221 wait complete. 2024-09-16T00:20:30.650 INFO:tasks.workunit.client.0.smithi033.stdout: 1 54123 3.15 MB/sec warmup 98 sec latency 17.410 ms 2024-09-16T00:20:31.458 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:20:31.650 INFO:tasks.workunit.client.0.smithi033.stdout: 1 57462 3.30 MB/sec warmup 99 sec latency 14.333 ms 2024-09-16T00:20:31.724 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:31.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:31 smithi033 ceph-mon[31243]: pgmap v451: 129 pgs: 129 active+clean; 36 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 131 KiB/s wr, 14 op/s 2024-09-16T00:20:31.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:31 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/560625221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:20:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:31 smithi083 ceph-mon[35242]: pgmap v451: 129 pgs: 129 active+clean; 36 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 131 KiB/s wr, 14 op/s 2024-09-16T00:20:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:31 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/560625221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:20:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:31 smithi060 ceph-mon[35185]: pgmap v451: 129 pgs: 129 active+clean; 36 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 131 KiB/s wr, 14 op/s 2024-09-16T00:20:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:31 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/560625221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:20:32.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.270+0000 7f2935af7640 1 -- 172.21.15.33:0/1906883418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2930101200 msgr2=0x7f2930101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:32.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.270+0000 7f2935af7640 1 --2- 172.21.15.33:0/1906883418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2930101200 0x7f2930101660 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2924009890 tx=0x7f292402a360 comp rx=0 tx=0).stop 2024-09-16T00:20:32.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.270+0000 7f2935af7640 1 -- 172.21.15.33:0/1906883418 shutdown_connections 2024-09-16T00:20:32.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.270+0000 7f2935af7640 1 --2- 172.21.15.33:0/1906883418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2930101ba0 0x7f293010d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.270+0000 7f2935af7640 1 --2- 172.21.15.33:0/1906883418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2930101200 0x7f2930101660 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.270+0000 7f2935af7640 1 --2- 172.21.15.33:0/1906883418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2930105ad0 0x7f2930105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.270+0000 7f2935af7640 1 -- 172.21.15.33:0/1906883418 >> 172.21.15.33:0/1906883418 conn(0x7f2930077210 msgr2=0x7f29300fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:32.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.270+0000 7f2935af7640 1 -- 172.21.15.33:0/1906883418 shutdown_connections 2024-09-16T00:20:32.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.270+0000 7f2935af7640 1 -- 172.21.15.33:0/1906883418 wait complete. 2024-09-16T00:20:32.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f2935af7640 1 Processor -- start 2024-09-16T00:20:32.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f2935af7640 1 -- start start 2024-09-16T00:20:32.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f2935af7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2930101200 0x7f29300fc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f2935af7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2930101ba0 0x7f29300fcbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f2935af7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2930105ad0 0x7f29300fd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f2935af7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2930110eb0 con 0x7f2930101ba0 2024-09-16T00:20:32.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f2935af7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2930110d30 con 0x7f2930101200 2024-09-16T00:20:32.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f2935af7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2930111030 con 0x7f2930105ad0 2024-09-16T00:20:32.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f292effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2930101ba0 0x7f29300fcbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f292ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2930105ad0 0x7f29300fd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f292effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2930101ba0 0x7f29300fcbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48682/0 (socket says 172.21.15.33:48682) 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f292effd640 1 -- 172.21.15.33:0/1308677062 learned_addr learned my addr 172.21.15.33:0/1308677062 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f292ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2930105ad0 0x7f29300fd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41536/0 (socket says 172.21.15.33:41536) 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.271+0000 7f292f7fe640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2930101200 0x7f29300fc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.272+0000 7f292ffff640 1 -- 172.21.15.33:0/1308677062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2930101200 msgr2=0x7f29300fc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.272+0000 7f292ffff640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2930101200 0x7f29300fc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.272+0000 7f292ffff640 1 -- 172.21.15.33:0/1308677062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2930101ba0 msgr2=0x7f29300fcbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.272+0000 7f292ffff640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2930101ba0 0x7f29300fcbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.272+0000 7f292ffff640 1 -- 172.21.15.33:0/1308677062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29301a68d0 con 0x7f2930105ad0 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.272+0000 7f292f7fe640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2930101200 0x7f29300fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.272+0000 7f292ffff640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2930105ad0 0x7f29300fd110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2928005fd0 tx=0x7f292800c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.272+0000 7f292cff9640 1 -- 172.21.15.33:0/1308677062 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2928004b20 con 0x7f2930105ad0 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.272+0000 7f2935af7640 1 -- 172.21.15.33:0/1308677062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29301a6bc0 con 0x7f2930105ad0 2024-09-16T00:20:32.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.272+0000 7f292cff9640 1 -- 172.21.15.33:0/1308677062 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2928003e50 con 0x7f2930105ad0 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.272+0000 7f292cff9640 1 -- 172.21.15.33:0/1308677062 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292800b570 con 0x7f2930105ad0 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.272+0000 7f2935af7640 1 -- 172.21.15.33:0/1308677062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29301a7100 con 0x7f2930105ad0 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.274+0000 7f292cff9640 1 -- 172.21.15.33:0/1308677062 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f292801d020 con 0x7f2930105ad0 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f2937ed0740 1 Processor -- start 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f2937ed0740 1 -- start start 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f2937ed0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1bb40 0x560c84b1a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f2937ed0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1a5d0 0x560c84b1aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f292cff9640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f29180a32c0 0x7f29180a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f2937ed0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 0x560c84b27760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f2937ed0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560c84a802f0 con 0x560c84b25370 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f2937ed0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560c84a27cf0 con 0x560c84b1a5d0 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f2937ed0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560c84a7c2b0 con 0x560c84b1bb40 2024-09-16T00:20:32.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f292effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1bb40 0x560c84b1a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f292effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1bb40 0x560c84b1a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41546/0 (socket says 172.21.15.33:41546) 2024-09-16T00:20:32.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f292cff9640 1 -- 172.21.15.33:0/1308677062 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f29280cfcc0 con 0x7f2930105ad0 2024-09-16T00:20:32.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f292effd640 1 -- 172.21.15.33:0/1106754924 learned_addr learned my addr 172.21.15.33:0/1106754924 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:32.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.275+0000 7f292f7fe640 1 --2- 172.21.15.33:0/1106754924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1a5d0 0x560c84b1aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.276+0000 7f292ffff640 1 --2- 172.21.15.33:0/1106754924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 0x560c84b27760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.276+0000 7f292effd640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f29180a32c0 0x7f29180a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.276+0000 7f292f7fe640 1 -- 172.21.15.33:0/1106754924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1bb40 msgr2=0x560c84b1a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:32.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.276+0000 7f292f7fe640 1 --2- 172.21.15.33:0/1106754924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1bb40 0x560c84b1a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.276+0000 7f292f7fe640 1 -- 172.21.15.33:0/1106754924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 msgr2=0x560c84b27760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.276+0000 7f292f7fe640 1 --2- 172.21.15.33:0/1106754924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 0x560c84b27760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.276+0000 7f292f7fe640 1 -- 172.21.15.33:0/1106754924 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c84a0c3a0 con 0x560c84b1a5d0 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.276+0000 7f292ffff640 1 --2- 172.21.15.33:0/1106754924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 0x560c84b27760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.276+0000 7f292f7fe640 1 --2- 172.21.15.33:0/1106754924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1a5d0 0x560c84b1aa10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2920009bb0 tx=0x7f29200117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d46a5e7bd5323c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.276+0000 7f292effd640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f29180a32c0 0x7f29180a5780 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2924005e00 tx=0x7f292402a360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.276+0000 7f2915ffb640 1 -- 172.21.15.33:0/1106754924 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920012450 con 0x560c84b1a5d0 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.277+0000 7f2915ffb640 1 -- 172.21.15.33:0/1106754924 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2920012ad0 con 0x560c84b1a5d0 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.277+0000 7f2915ffb640 1 -- 172.21.15.33:0/1106754924 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292000faa0 con 0x560c84b1a5d0 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.277+0000 7f2937ed0740 1 -- 172.21.15.33:0/1106754924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1a5d0 msgr2=0x560c84b1aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.277+0000 7f2937ed0740 1 --2- 172.21.15.33:0/1106754924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1a5d0 0x560c84b1aa10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2920009bb0 tx=0x7f29200117b0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.277+0000 7f2937ed0740 1 -- 172.21.15.33:0/1106754924 shutdown_connections 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.277+0000 7f2937ed0740 1 --2- 172.21.15.33:0/1106754924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 0x560c84b27760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.277+0000 7f2937ed0740 1 --2- 172.21.15.33:0/1106754924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1a5d0 0x560c84b1aa10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.277+0000 7f2937ed0740 1 --2- 172.21.15.33:0/1106754924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1bb40 0x560c84b1a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.277+0000 7f2937ed0740 1 -- 172.21.15.33:0/1106754924 >> 172.21.15.33:0/1106754924 conn(0x560c84ae3310 msgr2=0x560c84b16c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.277+0000 7f2937ed0740 1 -- 172.21.15.33:0/1106754924 shutdown_connections 2024-09-16T00:20:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.278+0000 7f2937ed0740 1 -- 172.21.15.33:0/1106754924 wait complete. 2024-09-16T00:20:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.278+0000 7f2937ed0740 1 -- start start 2024-09-16T00:20:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.278+0000 7f2937ed0740 1 Processor -- start 2024-09-16T00:20:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.278+0000 7f2937ed0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1a5d0 0x560c84b24cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.278+0000 7f2937ed0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1bb40 0x560c84b21f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.278+0000 7f2937ed0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b25370 0x560c84b224d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.278+0000 7f2937ed0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560c84b28070 con 0x560c84b1a5d0 2024-09-16T00:20:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.278+0000 7f2937ed0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560c84a3b060 con 0x560c84b1bb40 2024-09-16T00:20:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.278+0000 7f2937ed0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560c84b281f0 con 0x560c84b25370 2024-09-16T00:20:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f2937ed0740 10 client.0 _ll_register_callbacks cb 0x560c8478e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f292f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1a5d0 0x560c84b24cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f292ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b25370 0x560c84b224d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f292f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1a5d0 0x560c84b24cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48706/0 (socket says 172.21.15.33:48706) 2024-09-16T00:20:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f292f7fe640 1 -- 172.21.15.33:0/370712806 learned_addr learned my addr 172.21.15.33:0/370712806 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f292effd640 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1bb40 0x560c84b21f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f292ffff640 1 -- 172.21.15.33:0/370712806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1bb40 msgr2=0x560c84b21f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f292ffff640 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1bb40 0x560c84b21f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f292ffff640 1 -- 172.21.15.33:0/370712806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1a5d0 msgr2=0x560c84b24cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f292ffff640 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1a5d0 0x560c84b24cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f292ffff640 1 -- 172.21.15.33:0/370712806 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c847f13c0 con 0x560c84b25370 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f292f7fe640 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1a5d0 0x560c84b24cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f292ffff640 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b25370 0x560c84b224d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x560c84b1b070 tx=0x7f292809c5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a351868a6d6c369a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f290effd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f290effd640 1 -- 172.21.15.33:0/370712806 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292801a040 con 0x560c84b25370 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f290effd640 1 -- 172.21.15.33:0/370712806 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f29280cd3f0 con 0x560c84b25370 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.279+0000 7f2937ed0740 1 -- 172.21.15.33:0/370712806 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2920012ad0 con 0x560c84b25370 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.280+0000 7f290effd640 1 -- 172.21.15.33:0/370712806 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f29280a4a70 con 0x560c84b25370 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.280+0000 7f2937ed0740 10 client.25453 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.280+0000 7f2937ed0740 1 -- 172.21.15.33:0/370712806 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560c84b339f0 con 0x560c84b25370 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.280+0000 7f290effd640 1 -- 172.21.15.33:0/370712806 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2920012ad0 con 0x560c84b25370 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.280+0000 7f290effd640 1 -- 172.21.15.33:0/370712806 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f292800b040 con 0x560c84b25370 2024-09-16T00:20:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.280+0000 7f2937ed0740 10 client.25453 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.280+0000 7f2937ed0740 10 client.25453 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.280+0000 7f2937ed0740 20 client.25453 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.280+0000 7f2937ed0740 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c84b34f00 0x560c84b37350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.280+0000 7f2937ed0740 4 client.25453 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.280+0000 7f2937ed0740 1 -- 172.21.15.33:0/370712806 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c84a8d860 con 0x560c84b34f00 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.280+0000 7f292f7fe640 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c84b34f00 0x560c84b37350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.281+0000 7f290effd640 10 client.25453 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.281+0000 7f292f7fe640 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c84b34f00 0x560c84b37350 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9be4312d5522eb1a server_cookie=67e800b8c17051a8 in_seq=0 out_seq=0 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.281+0000 7f290effd640 1 -- 172.21.15.33:0/370712806 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560c84a8d860 con 0x560c84b34f00 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.281+0000 7f290effd640 10 client.25453 handle_command_reply: tid=0 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.281+0000 7f2937ed0740 1 client.25453 shutdown 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.282+0000 7f2937ed0740 1 -- 172.21.15.33:0/370712806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b25370 msgr2=0x560c84b224d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.282+0000 7f2937ed0740 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b25370 0x560c84b224d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x560c84b1b070 tx=0x7f292809c5c0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.282+0000 7f2937ed0740 1 -- 172.21.15.33:0/370712806 shutdown_connections 2024-09-16T00:20:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.282+0000 7f2937ed0740 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c84b34f00 0x560c84b37350 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.282+0000 7f2937ed0740 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b25370 0x560c84b224d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.282+0000 7f2937ed0740 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1bb40 0x560c84b21f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.282+0000 7f2937ed0740 1 --2- 172.21.15.33:0/370712806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1a5d0 0x560c84b24cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.282+0000 7f290effd640 0 client.25453 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:20:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.282+0000 7f2937ed0740 1 -- 172.21.15.33:0/370712806 >> 172.21.15.33:0/370712806 conn(0x560c84ae3310 msgr2=0x560c84b1c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.282+0000 7f2937ed0740 1 -- 172.21.15.33:0/370712806 shutdown_connections 2024-09-16T00:20:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.282+0000 7f2937ed0740 1 -- 172.21.15.33:0/370712806 wait complete. 2024-09-16T00:20:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.282+0000 7f2937ed0740 20 client.25453 trim_cache size 0 max 16384 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.296+0000 7f2937ed0740 1 Processor -- start 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.296+0000 7f2937ed0740 1 -- start start 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f2937ed0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1a5d0 0x560c84b31310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f2937ed0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1bb40 0x560c84b20f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f2937ed0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 0x560c84b1d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f2937ed0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f29301b3b60 con 0x560c84b25370 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f2937ed0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f29301b1430 con 0x560c84b1bb40 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f2937ed0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29301af280 con 0x560c84b1a5d0 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f292ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 0x560c84b1d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f292effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1bb40 0x560c84b20f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f292ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 0x560c84b1d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48710/0 (socket says 172.21.15.33:48710) 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f292f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1a5d0 0x560c84b31310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f292ffff640 1 -- 172.21.15.33:0/1424644151 learned_addr learned my addr 172.21.15.33:0/1424644151 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f292ffff640 1 -- 172.21.15.33:0/1424644151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1a5d0 msgr2=0x560c84b31310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f292ffff640 1 --2- 172.21.15.33:0/1424644151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1a5d0 0x560c84b31310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f292ffff640 1 -- 172.21.15.33:0/1424644151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1bb40 msgr2=0x560c84b20f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f292ffff640 1 --2- 172.21.15.33:0/1424644151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1bb40 0x560c84b20f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f292ffff640 1 -- 172.21.15.33:0/1424644151 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c84b37980 con 0x560c84b25370 2024-09-16T00:20:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f292effd640 1 --2- 172.21.15.33:0/1424644151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1bb40 0x560c84b20f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f292ffff640 1 --2- 172.21.15.33:0/1424644151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 0x560c84b1d120 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f292809bd90 tx=0x7f292800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175c9f56a64fedc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.297+0000 7f29157fa640 1 -- 172.21.15.33:0/1424644151 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292809b440 con 0x560c84b25370 2024-09-16T00:20:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.298+0000 7f29157fa640 1 -- 172.21.15.33:0/1424644151 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f292800b040 con 0x560c84b25370 2024-09-16T00:20:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.298+0000 7f29157fa640 1 -- 172.21.15.33:0/1424644151 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292809acc0 con 0x560c84b25370 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.298+0000 7f2937ed0740 1 -- 172.21.15.33:0/1424644151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 msgr2=0x560c84b1d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.298+0000 7f2937ed0740 1 --2- 172.21.15.33:0/1424644151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 0x560c84b1d120 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f292809bd90 tx=0x7f292800cde0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.298+0000 7f2937ed0740 1 -- 172.21.15.33:0/1424644151 shutdown_connections 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.298+0000 7f2937ed0740 1 --2- 172.21.15.33:0/1424644151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b25370 0x560c84b1d120 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.298+0000 7f2937ed0740 1 --2- 172.21.15.33:0/1424644151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b1bb40 0x560c84b20f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.299+0000 7f2937ed0740 1 --2- 172.21.15.33:0/1424644151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1a5d0 0x560c84b31310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.299+0000 7f2937ed0740 1 -- 172.21.15.33:0/1424644151 >> 172.21.15.33:0/1424644151 conn(0x560c84ae3310 msgr2=0x560c84b20840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.299+0000 7f2937ed0740 1 -- 172.21.15.33:0/1424644151 shutdown_connections 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.299+0000 7f2937ed0740 1 -- 172.21.15.33:0/1424644151 wait complete. 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.299+0000 7f2937ed0740 1 -- start start 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.299+0000 7f2937ed0740 1 Processor -- start 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.299+0000 7f2937ed0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1a5d0 0x560c84b2d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.299+0000 7f2937ed0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1bb40 0x560c84b2db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f2937ed0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b25370 0x560c84b289e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f2937ed0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f29301af8f0 con 0x560c84b1bb40 2024-09-16T00:20:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f2937ed0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f29301b0d60 con 0x560c84b25370 2024-09-16T00:20:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f2937ed0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29301affc0 con 0x560c84b1a5d0 2024-09-16T00:20:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f2937ed0740 10 client.0 _ll_register_callbacks cb 0x7f2928004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1bb40 0x560c84b2db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1a5d0 0x560c84b2d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1a5d0 0x560c84b2d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41564/0 (socket says 172.21.15.33:41564) 2024-09-16T00:20:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b25370 0x560c84b289e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292f7fe640 1 -- 172.21.15.33:0/2783145006 learned_addr learned my addr 172.21.15.33:0/2783145006 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292effd640 1 -- 172.21.15.33:0/2783145006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1a5d0 msgr2=0x560c84b2d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292effd640 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1a5d0 0x560c84b2d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292effd640 1 -- 172.21.15.33:0/2783145006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b25370 msgr2=0x560c84b289e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292effd640 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b25370 0x560c84b289e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292effd640 1 -- 172.21.15.33:0/2783145006 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c84b29210 con 0x560c84b1bb40 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292ffff640 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b25370 0x560c84b289e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292f7fe640 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1a5d0 0x560c84b2d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f292effd640 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1bb40 0x560c84b2db10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2924035040 tx=0x7f2924005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86b87c61592ffb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f290effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f290effd640 1 -- 172.21.15.33:0/2783145006 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2924053020 con 0x560c84b1bb40 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.300+0000 7f290effd640 1 -- 172.21.15.33:0/2783145006 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f29240046c0 con 0x560c84b1bb40 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f2937ed0740 1 -- 172.21.15.33:0/2783145006 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f292800b040 con 0x560c84b1bb40 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f290effd640 1 -- 172.21.15.33:0/2783145006 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2924037040 con 0x560c84b1bb40 2024-09-16T00:20:32.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f2937ed0740 10 client.15972 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:32.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f2937ed0740 1 -- 172.21.15.33:0/2783145006 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560c84b9f640 con 0x560c84b1bb40 2024-09-16T00:20:32.335 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:20:32.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f290effd640 1 -- 172.21.15.33:0/2783145006 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f292800b040 con 0x560c84b1bb40 2024-09-16T00:20:32.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f290effd640 1 -- 172.21.15.33:0/2783145006 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f292404fa20 con 0x560c84b1bb40 2024-09-16T00:20:32.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f2937ed0740 10 client.15972 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:32.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f2937ed0740 10 client.15972 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:20:32.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f2937ed0740 20 client.15972 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:32.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f2937ed0740 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c84b9fe10 0x560c84ba2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:32.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f2937ed0740 4 client.15972 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:20:32.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f2937ed0740 1 -- 172.21.15.33:0/2783145006 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560c84a87d70 con 0x560c84b9fe10 2024-09-16T00:20:32.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.301+0000 7f292f7fe640 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c84b9fe10 0x560c84ba2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:32.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f292f7fe640 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c84b9fe10 0x560c84ba2230 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3697abea64d16c95 server_cookie=46b2bad2e545491c in_seq=0 out_seq=0 2024-09-16T00:20:32.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f290effd640 10 client.15972 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:20:32.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f290effd640 1 -- 172.21.15.33:0/2783145006 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560c84a87d70 con 0x560c84b9fe10 2024-09-16T00:20:32.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f290effd640 10 client.15972 handle_command_reply: tid=0 2024-09-16T00:20:32.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f2937ed0740 1 client.15972 shutdown 2024-09-16T00:20:32.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f2937ed0740 1 -- 172.21.15.33:0/2783145006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1bb40 msgr2=0x560c84b2db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:32.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f2937ed0740 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1bb40 0x560c84b2db10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2924035040 tx=0x7f2924005f90 comp rx=0 tx=0).stop 2024-09-16T00:20:32.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f2937ed0740 1 -- 172.21.15.33:0/2783145006 shutdown_connections 2024-09-16T00:20:32.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f2937ed0740 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560c84b9fe10 0x560c84ba2230 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f2937ed0740 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560c84b25370 0x560c84b289e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f2937ed0740 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560c84b1bb40 0x560c84b2db10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f290effd640 0 client.15972 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f2937ed0740 1 --2- 172.21.15.33:0/2783145006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560c84b1a5d0 0x560c84b2d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f2937ed0740 1 -- 172.21.15.33:0/2783145006 >> 172.21.15.33:0/2783145006 conn(0x560c84ae3310 msgr2=0x560c84b237d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.302+0000 7f2937ed0740 1 -- 172.21.15.33:0/2783145006 shutdown_connections 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2937ed0740 1 -- 172.21.15.33:0/2783145006 wait complete. 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2937ed0740 20 client.15972 trim_cache size 0 max 16384 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2935af7640 1 -- 172.21.15.33:0/1308677062 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f29180a32c0 msgr2=0x7f29180a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2935af7640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f29180a32c0 0x7f29180a5780 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2924005e00 tx=0x7f292402a360 comp rx=0 tx=0).stop 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2935af7640 1 -- 172.21.15.33:0/1308677062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2930105ad0 msgr2=0x7f29300fd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2935af7640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2930105ad0 0x7f29300fd110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2928005fd0 tx=0x7f292800c740 comp rx=0 tx=0).stop 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2935af7640 1 -- 172.21.15.33:0/1308677062 shutdown_connections 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2935af7640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f29180a32c0 0x7f29180a5780 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2935af7640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2930105ad0 0x7f29300fd110 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2935af7640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2930101ba0 0x7f29300fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2935af7640 1 --2- 172.21.15.33:0/1308677062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2930101200 0x7f29300fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:32.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2935af7640 1 -- 172.21.15.33:0/1308677062 >> 172.21.15.33:0/1308677062 conn(0x7f2930077210 msgr2=0x7f2930100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:32.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2935af7640 1 -- 172.21.15.33:0/1308677062 shutdown_connections 2024-09-16T00:20:32.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:32.303+0000 7f2935af7640 1 -- 172.21.15.33:0/1308677062 wait complete. 2024-09-16T00:20:32.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 60707 3.46 MB/sec warmup 100 sec latency 40.592 ms 2024-09-16T00:20:33.059 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:20:33.337 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:33.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 64003 3.60 MB/sec warmup 101 sec latency 13.759 ms 2024-09-16T00:20:33.786 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:33 smithi033 ceph-mon[31243]: pgmap v452: 129 pgs: 129 active+clean; 39 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 368 KiB/s wr, 61 op/s 2024-09-16T00:20:33.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.875+0000 7f0df8a1f640 1 -- 172.21.15.33:0/844317986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df4103660 msgr2=0x7f0df410bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:33.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.875+0000 7f0df8a1f640 1 --2- 172.21.15.33:0/844317986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df4103660 0x7f0df410bb50 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009900 tx=0x7f0de802a480 comp rx=0 tx=0).stop 2024-09-16T00:20:33.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.875+0000 7f0df8a1f640 1 -- 172.21.15.33:0/844317986 shutdown_connections 2024-09-16T00:20:33.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.875+0000 7f0df8a1f640 1 --2- 172.21.15.33:0/844317986 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df410c170 0x7f0df410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.875+0000 7f0df8a1f640 1 --2- 172.21.15.33:0/844317986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df4103660 0x7f0df410bb50 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.875+0000 7f0df8a1f640 1 --2- 172.21.15.33:0/844317986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df4102d40 0x7f0df4103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.875+0000 7f0df8a1f640 1 -- 172.21.15.33:0/844317986 >> 172.21.15.33:0/844317986 conn(0x7f0df40fc7a0 msgr2=0x7f0df40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:33.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.875+0000 7f0df8a1f640 1 -- 172.21.15.33:0/844317986 shutdown_connections 2024-09-16T00:20:33.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.875+0000 7f0df8a1f640 1 -- 172.21.15.33:0/844317986 wait complete. 2024-09-16T00:20:33.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.876+0000 7f0df8a1f640 1 Processor -- start 2024-09-16T00:20:33.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.876+0000 7f0df8a1f640 1 -- start start 2024-09-16T00:20:33.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.876+0000 7f0df8a1f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df4102d40 0x7f0df4198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.876+0000 7f0df8a1f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df4103660 0x7f0df4198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.876+0000 7f0df8a1f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df410c170 0x7f0df419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.876+0000 7f0df8a1f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0df4111f70 con 0x7f0df4102d40 2024-09-16T00:20:33.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.876+0000 7f0df8a1f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0df4111df0 con 0x7f0df410c170 2024-09-16T00:20:33.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.876+0000 7f0df8a1f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41120f0 con 0x7f0df4103660 2024-09-16T00:20:33.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.876+0000 7f0df259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df4102d40 0x7f0df4198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.876+0000 7f0df259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df4102d40 0x7f0df4198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48730/0 (socket says 172.21.15.33:48730) 2024-09-16T00:20:33.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.876+0000 7f0df259c640 1 -- 172.21.15.33:0/2819559317 learned_addr learned my addr 172.21.15.33:0/2819559317 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:33.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.876+0000 7f0df1d9b640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df4103660 0x7f0df4198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df410c170 0x7f0df419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0df259c640 1 -- 172.21.15.33:0/2819559317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df4103660 msgr2=0x7f0df4198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:33.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0df259c640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df4103660 0x7f0df4198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0df259c640 1 -- 172.21.15.33:0/2819559317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df410c170 msgr2=0x7f0df419cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:33.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0df259c640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df410c170 0x7f0df419cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0df259c640 1 -- 172.21.15.33:0/2819559317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df419d3c0 con 0x7f0df4102d40 2024-09-16T00:20:33.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0df1d9b640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df4103660 0x7f0df4198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:33.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0df259c640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df4102d40 0x7f0df4198410 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0de400e9f0 tx=0x7f0de400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:33.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0de37fe640 1 -- 172.21.15.33:0/2819559317 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de400cd80 con 0x7f0df4102d40 2024-09-16T00:20:33.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0df8a1f640 1 -- 172.21.15.33:0/2819559317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0df419d6b0 con 0x7f0df4102d40 2024-09-16T00:20:33.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0de37fe640 1 -- 172.21.15.33:0/2819559317 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0de4010910 con 0x7f0df4102d40 2024-09-16T00:20:33.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0de37fe640 1 -- 172.21.15.33:0/2819559317 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de4018990 con 0x7f0df4102d40 2024-09-16T00:20:33.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.877+0000 7f0df8a1f640 1 -- 172.21.15.33:0/2819559317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0df41a4f90 con 0x7f0df4102d40 2024-09-16T00:20:33.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.880+0000 7f0de37fe640 1 -- 172.21.15.33:0/2819559317 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0de4010ac0 con 0x7f0df4102d40 2024-09-16T00:20:33.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0dfadf8740 1 Processor -- start 2024-09-16T00:20:33.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0dfadf8740 1 -- start start 2024-09-16T00:20:33.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0dfadf8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a830ab40 0x5642a8309090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0dfadf8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a83095d0 0x5642a8309a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0de37fe640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ddc0a3240 0x7f0ddc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0dfadf8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a8316760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0dfadf8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5642a826f2f0 con 0x5642a830ab40 2024-09-16T00:20:33.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0dfadf8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5642a8216cf0 con 0x5642a83095d0 2024-09-16T00:20:33.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0dfadf8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5642a826b2b0 con 0x5642a8314370 2024-09-16T00:20:33.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0df2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a830ab40 0x5642a8309090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0df2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a830ab40 0x5642a8309090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48738/0 (socket says 172.21.15.33:48738) 2024-09-16T00:20:33.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0df1d9b640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ddc0a3240 0x7f0ddc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0df2d9d640 1 -- 172.21.15.33:0/4051436734 learned_addr learned my addr 172.21.15.33:0/4051436734 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:33.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0df259c640 1 --2- 172.21.15.33:0/4051436734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a83095d0 0x5642a8309a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.881+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/4051436734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a8316760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.882+0000 7f0de37fe640 1 -- 172.21.15.33:0/2819559317 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0de40d01b0 con 0x7f0df4102d40 2024-09-16T00:20:33.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.882+0000 7f0df2d9d640 1 -- 172.21.15.33:0/4051436734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 msgr2=0x5642a8316760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:33.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.882+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/4051436734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a8316760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.882+0000 7f0df2d9d640 1 -- 172.21.15.33:0/4051436734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a83095d0 msgr2=0x5642a8309a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:33.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.882+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/4051436734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a83095d0 0x5642a8309a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.882+0000 7f0df2d9d640 1 -- 172.21.15.33:0/4051436734 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642a81fb3a0 con 0x5642a830ab40 2024-09-16T00:20:33.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.882+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/4051436734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a830ab40 0x5642a8309090 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0dec013a80 tx=0x7f0dec00a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5030240dd08b0a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:33.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.882+0000 7f0df1d9b640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ddc0a3240 0x7f0ddc0a5700 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009230 tx=0x7f0de80058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:33.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.882+0000 7f0de0ff9640 1 -- 172.21.15.33:0/4051436734 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec004d20 con 0x5642a830ab40 2024-09-16T00:20:33.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.882+0000 7f0de0ff9640 1 -- 172.21.15.33:0/4051436734 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dec005020 con 0x5642a830ab40 2024-09-16T00:20:33.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.882+0000 7f0de0ff9640 1 -- 172.21.15.33:0/4051436734 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec00eea0 con 0x5642a830ab40 2024-09-16T00:20:33.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.883+0000 7f0dfadf8740 1 -- 172.21.15.33:0/4051436734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a830ab40 msgr2=0x5642a8309090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:33.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.883+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/4051436734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a830ab40 0x5642a8309090 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0dec013a80 tx=0x7f0dec00a630 comp rx=0 tx=0).stop 2024-09-16T00:20:33.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.883+0000 7f0dfadf8740 1 -- 172.21.15.33:0/4051436734 shutdown_connections 2024-09-16T00:20:33.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.883+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/4051436734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a8316760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.883+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/4051436734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a83095d0 0x5642a8309a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.883+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/4051436734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a830ab40 0x5642a8309090 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.883+0000 7f0dfadf8740 1 -- 172.21.15.33:0/4051436734 >> 172.21.15.33:0/4051436734 conn(0x5642a82d2310 msgr2=0x5642a8305c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:33.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.883+0000 7f0dfadf8740 1 -- 172.21.15.33:0/4051436734 shutdown_connections 2024-09-16T00:20:33.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.883+0000 7f0dfadf8740 1 -- 172.21.15.33:0/4051436734 wait complete. 2024-09-16T00:20:33.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.883+0000 7f0dfadf8740 1 -- start start 2024-09-16T00:20:33.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0dfadf8740 1 Processor -- start 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0dfadf8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a83095d0 0x5642a8313d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0dfadf8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a830ab40 0x5642a8310fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0dfadf8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a83115d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0dfadf8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5642a8317070 con 0x5642a83095d0 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0dfadf8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5642a822a060 con 0x5642a830ab40 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0dfadf8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5642a83171f0 con 0x5642a8314370 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0dfadf8740 10 client.0 _ll_register_callbacks cb 0x5642a7f7d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0df2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a83115d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0df2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a83115d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41596/0 (socket says 172.21.15.33:41596) 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0df259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a83095d0 0x5642a8313d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0df2d9d640 1 -- 172.21.15.33:0/934594731 learned_addr learned my addr 172.21.15.33:0/934594731 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0df1d9b640 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a830ab40 0x5642a8310fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0df2d9d640 1 -- 172.21.15.33:0/934594731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a830ab40 msgr2=0x5642a8310fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a830ab40 0x5642a8310fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0df2d9d640 1 -- 172.21.15.33:0/934594731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a83095d0 msgr2=0x5642a8313d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a83095d0 0x5642a8313d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0df2d9d640 1 -- 172.21.15.33:0/934594731 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642a7fe03c0 con 0x5642a8314370 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0df1d9b640 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a830ab40 0x5642a8310fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.884+0000 7f0df259c640 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a83095d0 0x5642a8313d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a83115d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0dec00a600 tx=0x7f0dec00a370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64ab987de8a3fc48 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dd9ffb640 1 -- 172.21.15.33:0/934594731 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec00edd0 con 0x5642a8314370 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dfadf8740 1 -- 172.21.15.33:0/934594731 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dec005020 con 0x5642a8314370 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dd9ffb640 1 -- 172.21.15.33:0/934594731 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0dec003970 con 0x5642a8314370 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dd9ffb640 1 -- 172.21.15.33:0/934594731 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0dec003b70 con 0x5642a8314370 2024-09-16T00:20:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dfadf8740 10 client.25471 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dfadf8740 1 -- 172.21.15.33:0/934594731 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5642a8322ad0 con 0x5642a8314370 2024-09-16T00:20:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dd9ffb640 1 -- 172.21.15.33:0/934594731 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dec005020 con 0x5642a8314370 2024-09-16T00:20:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dd9ffb640 1 -- 172.21.15.33:0/934594731 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0dec026070 con 0x5642a8314370 2024-09-16T00:20:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dfadf8740 10 client.25471 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dfadf8740 10 client.25471 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:20:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dfadf8740 20 client.25471 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.885+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5642a8323fc0 0x5642a83263b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.886+0000 7f0dfadf8740 4 client.25471 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:20:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.886+0000 7f0dfadf8740 1 -- 172.21.15.33:0/934594731 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642a827c860 con 0x5642a8323fc0 2024-09-16T00:20:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.886+0000 7f0df259c640 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5642a8323fc0 0x5642a83263b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.886+0000 7f0df259c640 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5642a8323fc0 0x5642a83263b0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f67d690b6d1a160 server_cookie=ccf1fbea91df33ff in_seq=0 out_seq=0 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.886+0000 7f0dd9ffb640 10 client.25471 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dd9ffb640 1 -- 172.21.15.33:0/934594731 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5642a827c860 con 0x5642a8323fc0 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dd9ffb640 10 client.25471 handle_command_reply: tid=0 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dfadf8740 1 client.25471 shutdown 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dfadf8740 1 -- 172.21.15.33:0/934594731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 msgr2=0x5642a83115d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a83115d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0dec00a600 tx=0x7f0dec00a370 comp rx=0 tx=0).stop 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dfadf8740 1 -- 172.21.15.33:0/934594731 shutdown_connections 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a83115d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5642a8323fc0 0x5642a83263b0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a830ab40 0x5642a8310fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/934594731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a83095d0 0x5642a8313d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dd9ffb640 0 client.25471 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:20:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dfadf8740 1 -- 172.21.15.33:0/934594731 >> 172.21.15.33:0/934594731 conn(0x5642a82d2310 msgr2=0x5642a830b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dfadf8740 1 -- 172.21.15.33:0/934594731 shutdown_connections 2024-09-16T00:20:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dfadf8740 1 -- 172.21.15.33:0/934594731 wait complete. 2024-09-16T00:20:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.887+0000 7f0dfadf8740 20 client.25471 trim_cache size 0 max 16384 2024-09-16T00:20:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0dfadf8740 1 Processor -- start 2024-09-16T00:20:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0dfadf8740 1 -- start start 2024-09-16T00:20:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0dfadf8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a83095d0 0x5642a8320310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0dfadf8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a830ab40 0x5642a830c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0dfadf8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a8314370 0x5642a830c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0dfadf8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41af790 con 0x5642a830ab40 2024-09-16T00:20:33.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0dfadf8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41ad060 con 0x5642a8314370 2024-09-16T00:20:33.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0dfadf8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41aaeb0 con 0x5642a83095d0 2024-09-16T00:20:33.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0df259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a83095d0 0x5642a8320310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0df1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a830ab40 0x5642a830c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0df259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a83095d0 0x5642a8320310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41600/0 (socket says 172.21.15.33:41600) 2024-09-16T00:20:33.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0df259c640 1 -- 172.21.15.33:0/703998322 learned_addr learned my addr 172.21.15.33:0/703998322 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:33.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/703998322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a8314370 0x5642a830c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0df259c640 1 -- 172.21.15.33:0/703998322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a8314370 msgr2=0x5642a830c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:33.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0df259c640 1 --2- 172.21.15.33:0/703998322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a8314370 0x5642a830c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0df259c640 1 -- 172.21.15.33:0/703998322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a830ab40 msgr2=0x5642a830c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:33.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0df259c640 1 --2- 172.21.15.33:0/703998322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a830ab40 0x5642a830c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.901+0000 7f0df259c640 1 -- 172.21.15.33:0/703998322 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642a8306960 con 0x5642a83095d0 2024-09-16T00:20:33.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.902+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/703998322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a8314370 0x5642a830c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:20:33.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.902+0000 7f0df259c640 1 --2- 172.21.15.33:0/703998322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a83095d0 0x5642a8320310 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0de400c6a0 tx=0x7f0de4098590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75ee9ff032d40e3c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:33.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.902+0000 7f0ddbfff640 1 -- 172.21.15.33:0/703998322 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de40a4f60 con 0x5642a83095d0 2024-09-16T00:20:33.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.902+0000 7f0ddbfff640 1 -- 172.21.15.33:0/703998322 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0de409a390 con 0x5642a83095d0 2024-09-16T00:20:33.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.902+0000 7f0ddbfff640 1 -- 172.21.15.33:0/703998322 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de4018d90 con 0x5642a83095d0 2024-09-16T00:20:33.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 -- 172.21.15.33:0/703998322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a83095d0 msgr2=0x5642a8320310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/703998322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a83095d0 0x5642a8320310 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0de400c6a0 tx=0x7f0de4098590 comp rx=0 tx=0).stop 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 -- 172.21.15.33:0/703998322 shutdown_connections 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/703998322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a8314370 0x5642a830c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/703998322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a830ab40 0x5642a830c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/703998322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a83095d0 0x5642a8320310 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 -- 172.21.15.33:0/703998322 >> 172.21.15.33:0/703998322 conn(0x5642a82d2310 msgr2=0x5642a83100d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 -- 172.21.15.33:0/703998322 shutdown_connections 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 -- 172.21.15.33:0/703998322 wait complete. 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 -- start start 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 Processor -- start 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a83095d0 0x5642a831cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a830ab40 0x5642a8318080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a8318810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41ab520 con 0x5642a83095d0 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41ac990 con 0x5642a830ab40 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41abbf0 con 0x5642a8314370 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0dfadf8740 10 client.0 _ll_register_callbacks cb 0x7f0de4097ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.903+0000 7f0df259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a83095d0 0x5642a831cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0df2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a8318810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0df1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a830ab40 0x5642a8318080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0df259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a83095d0 0x5642a831cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48774/0 (socket says 172.21.15.33:48774) 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0df259c640 1 -- 172.21.15.33:0/876950947 learned_addr learned my addr 172.21.15.33:0/876950947 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0df2d9d640 1 -- 172.21.15.33:0/876950947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a830ab40 msgr2=0x5642a8318080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/876950947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a830ab40 0x5642a8318080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0df2d9d640 1 -- 172.21.15.33:0/876950947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a83095d0 msgr2=0x5642a831cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:33.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/876950947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a83095d0 0x5642a831cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0df2d9d640 1 -- 172.21.15.33:0/876950947 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642a838d950 con 0x5642a8314370 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0df2d9d640 1 --2- 172.21.15.33:0/876950947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a8318810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0dec016040 tx=0x7f0dec013980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eef71171815be690 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0dd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0dd9ffb640 1 -- 172.21.15.33:0/876950947 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec00aad0 con 0x5642a8314370 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0dfadf8740 1 -- 172.21.15.33:0/876950947 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0de409a390 con 0x5642a8314370 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0dd9ffb640 1 -- 172.21.15.33:0/876950947 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0dec013240 con 0x5642a8314370 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0dd9ffb640 1 -- 172.21.15.33:0/876950947 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0dec03f050 con 0x5642a8314370 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0dfadf8740 10 client.25477 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.904+0000 7f0dfadf8740 1 -- 172.21.15.33:0/876950947 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5642a838ea50 con 0x5642a8314370 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.905+0000 7f0dd9ffb640 1 -- 172.21.15.33:0/876950947 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0de409a390 con 0x5642a8314370 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.905+0000 7f0dd9ffb640 1 -- 172.21.15.33:0/876950947 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0dec035030 con 0x5642a8314370 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.905+0000 7f0dfadf8740 10 client.25477 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:33.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.905+0000 7f0dfadf8740 10 client.25477 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:20:33.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.905+0000 7f0dfadf8740 20 client.25477 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:33.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.905+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/876950947 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5642a838f0b0 0x5642a83914a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:33.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.905+0000 7f0dfadf8740 4 client.25477 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:20:33.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.905+0000 7f0dfadf8740 1 -- 172.21.15.33:0/876950947 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5642a8276d70 con 0x5642a838f0b0 2024-09-16T00:20:33.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.905+0000 7f0df259c640 1 --2- 172.21.15.33:0/876950947 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5642a838f0b0 0x5642a83914a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:33.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.905+0000 7f0df259c640 1 --2- 172.21.15.33:0/876950947 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5642a838f0b0 0x5642a83914a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=842667935a1c7c96 server_cookie=d1738ba8abe0ff8a in_seq=0 out_seq=0 2024-09-16T00:20:33.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.905+0000 7f0dd9ffb640 10 client.25477 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:20:33.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dd9ffb640 1 -- 172.21.15.33:0/876950947 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5642a8276d70 con 0x5642a838f0b0 2024-09-16T00:20:33.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dd9ffb640 10 client.25477 handle_command_reply: tid=0 2024-09-16T00:20:33.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dfadf8740 1 client.25477 shutdown 2024-09-16T00:20:33.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dfadf8740 1 -- 172.21.15.33:0/876950947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 msgr2=0x5642a8318810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:33.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/876950947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a8318810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0dec016040 tx=0x7f0dec013980 comp rx=0 tx=0).stop 2024-09-16T00:20:33.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dfadf8740 1 -- 172.21.15.33:0/876950947 shutdown_connections 2024-09-16T00:20:33.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/876950947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642a8314370 0x5642a8318810 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/876950947 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5642a838f0b0 0x5642a83914a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/876950947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5642a830ab40 0x5642a8318080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dfadf8740 1 --2- 172.21.15.33:0/876950947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642a83095d0 0x5642a831cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dd9ffb640 0 client.25477 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:20:33.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dfadf8740 1 -- 172.21.15.33:0/876950947 >> 172.21.15.33:0/876950947 conn(0x5642a82d2310 msgr2=0x5642a8312840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:33.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dfadf8740 1 -- 172.21.15.33:0/876950947 shutdown_connections 2024-09-16T00:20:33.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dfadf8740 1 -- 172.21.15.33:0/876950947 wait complete. 2024-09-16T00:20:33.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.906+0000 7f0dfadf8740 20 client.25477 trim_cache size 0 max 16384 2024-09-16T00:20:33.926 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:20:33.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.907+0000 7f0df8a1f640 1 -- 172.21.15.33:0/2819559317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ddc0a3240 msgr2=0x7f0ddc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:33.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.907+0000 7f0df8a1f640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ddc0a3240 0x7f0ddc0a5700 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009230 tx=0x7f0de80058d0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.907+0000 7f0df8a1f640 1 -- 172.21.15.33:0/2819559317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df4102d40 msgr2=0x7f0df4198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:33.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.907+0000 7f0df8a1f640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df4102d40 0x7f0df4198410 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0de400e9f0 tx=0x7f0de400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.907+0000 7f0df8a1f640 1 -- 172.21.15.33:0/2819559317 shutdown_connections 2024-09-16T00:20:33.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.907+0000 7f0df8a1f640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ddc0a3240 0x7f0ddc0a5700 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.907+0000 7f0df8a1f640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df410c170 0x7f0df419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.907+0000 7f0df8a1f640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df4103660 0x7f0df4198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.907+0000 7f0df8a1f640 1 --2- 172.21.15.33:0/2819559317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0df4102d40 0x7f0df4198410 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.907+0000 7f0df8a1f640 1 -- 172.21.15.33:0/2819559317 >> 172.21.15.33:0/2819559317 conn(0x7f0df40fc7a0 msgr2=0x7f0df40fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.907+0000 7f0df8a1f640 1 -- 172.21.15.33:0/2819559317 shutdown_connections 2024-09-16T00:20:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:33.907+0000 7f0df8a1f640 1 -- 172.21.15.33:0/2819559317 wait complete. 2024-09-16T00:20:33.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:33 smithi083 ceph-mon[35242]: pgmap v452: 129 pgs: 129 active+clean; 39 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 368 KiB/s wr, 61 op/s 2024-09-16T00:20:33.992 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:33 smithi060 ceph-mon[35185]: pgmap v452: 129 pgs: 129 active+clean; 39 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 368 KiB/s wr, 61 op/s 2024-09-16T00:20:34.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 67227 3.72 MB/sec warmup 102 sec latency 15.864 ms 2024-09-16T00:20:34.764 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:20:35.049 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:35.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 69837 3.80 MB/sec warmup 103 sec latency 46.826 ms 2024-09-16T00:20:35.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:35 smithi033 ceph-mon[31243]: pgmap v453: 129 pgs: 129 active+clean; 42 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 650 KiB/s wr, 94 op/s 2024-09-16T00:20:35.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.559+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3118747569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e981037a0 msgr2=0x7f7e9810bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:35.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.559+0000 7f7e9cee1640 1 --2- 172.21.15.33:0/3118747569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e981037a0 0x7f7e9810bc00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009900 tx=0x7f7e8c02a410 comp rx=0 tx=0).stop 2024-09-16T00:20:35.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.559+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3118747569 shutdown_connections 2024-09-16T00:20:35.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.559+0000 7f7e9cee1640 1 --2- 172.21.15.33:0/3118747569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e9810c2f0 0x7f7e9810e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.559+0000 7f7e9cee1640 1 --2- 172.21.15.33:0/3118747569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e981037a0 0x7f7e9810bc00 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.559+0000 7f7e9cee1640 1 --2- 172.21.15.33:0/3118747569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e98102df0 0x7f7e981031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.559+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3118747569 >> 172.21.15.33:0/3118747569 conn(0x7f7e980fc7d0 msgr2=0x7f7e980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:35.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.559+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3118747569 shutdown_connections 2024-09-16T00:20:35.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.559+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3118747569 wait complete. 2024-09-16T00:20:35.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e9cee1640 1 Processor -- start 2024-09-16T00:20:35.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e9cee1640 1 -- start start 2024-09-16T00:20:35.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e9cee1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e98102df0 0x7f7e98198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e9cee1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e981037a0 0x7f7e98198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e9cee1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e9810c2f0 0x7f7e9819cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e9cee1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7e981120b0 con 0x7f7e9810c2f0 2024-09-16T00:20:35.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e9cee1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7e98111f30 con 0x7f7e981037a0 2024-09-16T00:20:35.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e9cee1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7e98112230 con 0x7f7e98102df0 2024-09-16T00:20:35.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e9659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e98102df0 0x7f7e98198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e9659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e98102df0 0x7f7e98198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41640/0 (socket says 172.21.15.33:41640) 2024-09-16T00:20:35.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e9659c640 1 -- 172.21.15.33:0/3022718693 learned_addr learned my addr 172.21.15.33:0/3022718693 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:35.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e95d9b640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e981037a0 0x7f7e98198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.560+0000 7f7e96d9d640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e9810c2f0 0x7f7e9819cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.561+0000 7f7e9659c640 1 -- 172.21.15.33:0/3022718693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e981037a0 msgr2=0x7f7e98198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:35.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.561+0000 7f7e9659c640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e981037a0 0x7f7e98198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.561+0000 7f7e9659c640 1 -- 172.21.15.33:0/3022718693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e9810c2f0 msgr2=0x7f7e9819cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:35.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.561+0000 7f7e9659c640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e9810c2f0 0x7f7e9819cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.561+0000 7f7e9659c640 1 -- 172.21.15.33:0/3022718693 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9819d5e0 con 0x7f7e98102df0 2024-09-16T00:20:35.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.561+0000 7f7e9659c640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e98102df0 0x7f7e98198630 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800e9f0 tx=0x7f7e8800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:35.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.561+0000 7f7e877fe640 1 -- 172.21.15.33:0/3022718693 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8800cd80 con 0x7f7e98102df0 2024-09-16T00:20:35.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.561+0000 7f7e877fe640 1 -- 172.21.15.33:0/3022718693 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e88010910 con 0x7f7e98102df0 2024-09-16T00:20:35.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.561+0000 7f7e877fe640 1 -- 172.21.15.33:0/3022718693 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e88018990 con 0x7f7e98102df0 2024-09-16T00:20:35.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.561+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3022718693 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e9819d8d0 con 0x7f7e98102df0 2024-09-16T00:20:35.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.561+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3022718693 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e981a51b0 con 0x7f7e98102df0 2024-09-16T00:20:35.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.565+0000 7f7e877fe640 1 -- 172.21.15.33:0/3022718693 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7e88010ac0 con 0x7f7e98102df0 2024-09-16T00:20:35.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.566+0000 7f7e9f2ba740 1 Processor -- start 2024-09-16T00:20:35.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.566+0000 7f7e9f2ba740 1 -- start start 2024-09-16T00:20:35.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.566+0000 7f7e9f2ba740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 0x55667955c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.566+0000 7f7e877fe640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7e800a3340 0x7f7e800a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.566+0000 7f7e9f2ba740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667955c5d0 0x55667955ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.566+0000 7f7e9f2ba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667956b780 0x55667956db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.566+0000 7f7e9f2ba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5566794c22f0 con 0x55667956b780 2024-09-16T00:20:35.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.566+0000 7f7e9f2ba740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556679469cf0 con 0x55667955db40 2024-09-16T00:20:35.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.566+0000 7f7e9f2ba740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5566794be2b0 con 0x55667955c5d0 2024-09-16T00:20:35.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.566+0000 7f7e9659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 0x55667955c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.566+0000 7f7e9659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 0x55667955c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:38824/0 (socket says 172.21.15.33:38824) 2024-09-16T00:20:35.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.566+0000 7f7e9659c640 1 -- 172.21.15.33:0/1194840474 learned_addr learned my addr 172.21.15.33:0/1194840474 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:35.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.567+0000 7f7e96d9d640 1 --2- 172.21.15.33:0/1194840474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667956b780 0x55667956db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.567+0000 7f7e95d9b640 1 --2- 172.21.15.33:0/1194840474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667955c5d0 0x55667955ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.567+0000 7f7e877fe640 1 -- 172.21.15.33:0/3022718693 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7e880d0110 con 0x7f7e98102df0 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.567+0000 7f7e9659c640 1 -- 172.21.15.33:0/1194840474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667955c5d0 msgr2=0x55667955ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.567+0000 7f7e9659c640 1 --2- 172.21.15.33:0/1194840474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667955c5d0 0x55667955ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.567+0000 7f7e9659c640 1 -- 172.21.15.33:0/1194840474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667956b780 msgr2=0x55667956db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.567+0000 7f7e96d9d640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7e800a3340 0x7f7e800a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.567+0000 7f7e9659c640 1 --2- 172.21.15.33:0/1194840474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667956b780 0x55667956db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.567+0000 7f7e9659c640 1 -- 172.21.15.33:0/1194840474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55667944e3a0 con 0x55667955db40 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.567+0000 7f7e9659c640 1 --2- 172.21.15.33:0/1194840474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 0x55667955c090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7e880986b0 tx=0x7f7e8806f190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1f4db403cfc25e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.568+0000 7f7e84ff9640 1 -- 172.21.15.33:0/1194840474 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8809c250 con 0x55667955db40 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.568+0000 7f7e84ff9640 1 -- 172.21.15.33:0/1194840474 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e880a0600 con 0x55667955db40 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.568+0000 7f7e84ff9640 1 -- 172.21.15.33:0/1194840474 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e880bc0f0 con 0x55667955db40 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.569+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1194840474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 msgr2=0x55667955c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:35.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.569+0000 7f7e96d9d640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7e800a3340 0x7f7e800a5800 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7e900099a0 tx=0x7f7e90009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:35.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.569+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/1194840474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 0x55667955c090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7e880986b0 tx=0x7f7e8806f190 comp rx=0 tx=0).stop 2024-09-16T00:20:35.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.569+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1194840474 shutdown_connections 2024-09-16T00:20:35.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.569+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/1194840474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667956b780 0x55667956db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.569+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/1194840474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667955c5d0 0x55667955ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.569+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/1194840474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 0x55667955c090 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.569+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1194840474 >> 172.21.15.33:0/1194840474 conn(0x556679525310 msgr2=0x556679558c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:35.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.569+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1194840474 shutdown_connections 2024-09-16T00:20:35.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.569+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1194840474 wait complete. 2024-09-16T00:20:35.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.569+0000 7f7e9f2ba740 1 -- start start 2024-09-16T00:20:35.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.569+0000 7f7e9f2ba740 1 Processor -- start 2024-09-16T00:20:35.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e9f2ba740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667955c5d0 0x55667956b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e9f2ba740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 0x556679568490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e9f2ba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667956b780 0x556679568c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e9f2ba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55667956e3c0 con 0x55667956b780 2024-09-16T00:20:35.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e9f2ba740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55667947d060 con 0x55667955db40 2024-09-16T00:20:35.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e9f2ba740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55667956e540 con 0x55667955c5d0 2024-09-16T00:20:35.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e9f2ba740 10 client.0 _ll_register_callbacks cb 0x5566791d08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:35.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e9659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667955c5d0 0x55667956b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e9659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667955c5d0 0x55667956b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41656/0 (socket says 172.21.15.33:41656) 2024-09-16T00:20:35.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e9659c640 1 -- 172.21.15.33:0/1133846288 learned_addr learned my addr 172.21.15.33:0/1133846288 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:35.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e95d9b640 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 0x556679568490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e96d9d640 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667956b780 0x556679568c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e95d9b640 1 -- 172.21.15.33:0/1133846288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667955c5d0 msgr2=0x55667956b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:35.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e95d9b640 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667955c5d0 0x55667956b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e95d9b640 1 -- 172.21.15.33:0/1133846288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667956b780 msgr2=0x556679568c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:35.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e95d9b640 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667956b780 0x556679568c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e95d9b640 1 -- 172.21.15.33:0/1133846288 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566792333c0 con 0x55667955db40 2024-09-16T00:20:35.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e95d9b640 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 0x556679568490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c02a930 tx=0x7f7e8c032ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65491c9330c32ba1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:35.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.570+0000 7f7e7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:20:35.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e7dffb640 1 -- 172.21.15.33:0/1133846288 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c035040 con 0x55667955db40 2024-09-16T00:20:35.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1133846288 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e880a0600 con 0x55667955db40 2024-09-16T00:20:35.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e7dffb640 1 -- 172.21.15.33:0/1133846288 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7e8c031970 con 0x55667955db40 2024-09-16T00:20:35.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e7dffb640 1 -- 172.21.15.33:0/1133846288 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7e8c04f050 con 0x55667955db40 2024-09-16T00:20:35.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e9f2ba740 10 client.25577 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:35.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1133846288 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556679579f30 con 0x55667955db40 2024-09-16T00:20:35.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e7dffb640 1 -- 172.21.15.33:0/1133846288 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e880a0600 con 0x55667955db40 2024-09-16T00:20:35.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e7dffb640 1 -- 172.21.15.33:0/1133846288 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7e8c004750 con 0x55667955db40 2024-09-16T00:20:35.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e9f2ba740 10 client.25577 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:35.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e9f2ba740 10 client.25577 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:20:35.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e9f2ba740 20 client.25577 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:35.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55667957b390 0x55667957d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e9f2ba740 4 client.25577 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:20:35.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1133846288 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566794cf860 con 0x55667957b390 2024-09-16T00:20:35.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.571+0000 7f7e9659c640 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55667957b390 0x55667957d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.572+0000 7f7e7dffb640 10 client.25577 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:35.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.572+0000 7f7e9659c640 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55667957b390 0x55667957d7b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d295ba19fd868203 server_cookie=99b63ea530bf358e in_seq=0 out_seq=0 2024-09-16T00:20:35.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.572+0000 7f7e7dffb640 1 -- 172.21.15.33:0/1133846288 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5566794cf860 con 0x55667957b390 2024-09-16T00:20:35.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.572+0000 7f7e7dffb640 10 client.25577 handle_command_reply: tid=0 2024-09-16T00:20:35.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.572+0000 7f7e9f2ba740 1 client.25577 shutdown 2024-09-16T00:20:35.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.573+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1133846288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 msgr2=0x556679568490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:35.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.573+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 0x556679568490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c02a930 tx=0x7f7e8c032ed0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.573+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1133846288 shutdown_connections 2024-09-16T00:20:35.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.573+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55667957b390 0x55667957d7b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.573+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667956b780 0x556679568c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.573+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955db40 0x556679568490 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.573+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/1133846288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667955c5d0 0x55667956b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.573+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1133846288 >> 172.21.15.33:0/1133846288 conn(0x556679525310 msgr2=0x55667955e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:35.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.573+0000 7f7e7dffb640 0 client.25577 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:35.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.573+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1133846288 shutdown_connections 2024-09-16T00:20:35.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.573+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/1133846288 wait complete. 2024-09-16T00:20:35.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.573+0000 7f7e9f2ba740 20 client.25577 trim_cache size 0 max 16384 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e9f2ba740 1 Processor -- start 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e9f2ba740 1 -- start start 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e9f2ba740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955c5d0 0x556679567470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e9f2ba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667955db40 0x556679583080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e9f2ba740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667956b780 0x5566795835c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e9f2ba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7e981af800 con 0x55667955db40 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e9f2ba740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7e981ad0d0 con 0x55667955c5d0 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e9f2ba740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7e981aaf20 con 0x55667956b780 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e9659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955c5d0 0x556679567470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e9659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955c5d0 0x556679567470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:38836/0 (socket says 172.21.15.33:38836) 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e9659c640 1 -- 172.21.15.33:0/17774981 learned_addr learned my addr 172.21.15.33:0/17774981 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e96d9d640 1 --2- 172.21.15.33:0/17774981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667956b780 0x5566795835c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.587+0000 7f7e95d9b640 1 --2- 172.21.15.33:0/17774981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667955db40 0x556679583080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.588+0000 7f7e9659c640 1 -- 172.21.15.33:0/17774981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667956b780 msgr2=0x5566795835c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:35.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.588+0000 7f7e9659c640 1 --2- 172.21.15.33:0/17774981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667956b780 0x5566795835c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.588+0000 7f7e9659c640 1 -- 172.21.15.33:0/17774981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667955db40 msgr2=0x556679583080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:35.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.588+0000 7f7e9659c640 1 --2- 172.21.15.33:0/17774981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667955db40 0x556679583080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.588+0000 7f7e9659c640 1 -- 172.21.15.33:0/17774981 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55667955f120 con 0x55667955c5d0 2024-09-16T00:20:35.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.588+0000 7f7e9659c640 1 --2- 172.21.15.33:0/17774981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955c5d0 0x556679567470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7e880986b0 tx=0x7f7e88018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=344dd5e272f373ec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:35.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.590+0000 7f7e7ffff640 1 -- 172.21.15.33:0/17774981 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e880b5070 con 0x55667955c5d0 2024-09-16T00:20:35.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.591+0000 7f7e7ffff640 1 -- 172.21.15.33:0/17774981 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e880a0600 con 0x55667955c5d0 2024-09-16T00:20:35.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.591+0000 7f7e7ffff640 1 -- 172.21.15.33:0/17774981 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e880a4a30 con 0x55667955c5d0 2024-09-16T00:20:35.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.591+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/17774981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955c5d0 msgr2=0x556679567470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:35.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.591+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/17774981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955c5d0 0x556679567470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7e880986b0 tx=0x7f7e88018420 comp rx=0 tx=0).stop 2024-09-16T00:20:35.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.591+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/17774981 shutdown_connections 2024-09-16T00:20:35.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.591+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/17774981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667956b780 0x5566795835c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.591+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/17774981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667955db40 0x556679583080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.591+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/17774981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955c5d0 0x556679567470 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.591+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/17774981 >> 172.21.15.33:0/17774981 conn(0x556679525310 msgr2=0x556679566d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:35.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.592+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/17774981 shutdown_connections 2024-09-16T00:20:35.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.592+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/17774981 wait complete. 2024-09-16T00:20:35.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.592+0000 7f7e9f2ba740 1 -- start start 2024-09-16T00:20:35.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.592+0000 7f7e9f2ba740 1 Processor -- start 2024-09-16T00:20:35.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.592+0000 7f7e9f2ba740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955c5d0 0x55667956f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.592+0000 7f7e9f2ba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667955db40 0x55667956fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.592+0000 7f7e9f2ba740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667956b780 0x556679570240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.592+0000 7f7e9f2ba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7e981ab590 con 0x55667955db40 2024-09-16T00:20:35.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.592+0000 7f7e9f2ba740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7e981aca00 con 0x55667955c5d0 2024-09-16T00:20:35.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.592+0000 7f7e9f2ba740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7e981abc60 con 0x55667956b780 2024-09-16T00:20:35.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e9f2ba740 10 client.0 _ll_register_callbacks cb 0x7f7e8c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:35.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e95d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667955db40 0x55667956fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e95d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667955db40 0x55667956fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48818/0 (socket says 172.21.15.33:48818) 2024-09-16T00:20:35.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e95d9b640 1 -- 172.21.15.33:0/907626257 learned_addr learned my addr 172.21.15.33:0/907626257 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:35.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e96d9d640 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667956b780 0x556679570240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e9659c640 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955c5d0 0x55667956f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e95d9b640 1 -- 172.21.15.33:0/907626257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667956b780 msgr2=0x556679570240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:35.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e95d9b640 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667956b780 0x556679570240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e95d9b640 1 -- 172.21.15.33:0/907626257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955c5d0 msgr2=0x55667956f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:35.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e95d9b640 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955c5d0 0x55667956f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e95d9b640 1 -- 172.21.15.33:0/907626257 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566795e52a0 con 0x55667955db40 2024-09-16T00:20:35.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e95d9b640 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667955db40 0x55667956fd00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009900 tx=0x7f7e8c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5929b0521b4a5d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:35.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:20:35.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e7dffb640 1 -- 172.21.15.33:0/907626257 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c035050 con 0x55667955db40 2024-09-16T00:20:35.890 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:20:35.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.593+0000 7f7e7dffb640 1 -- 172.21.15.33:0/907626257 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7e8c031980 con 0x55667955db40 2024-09-16T00:20:35.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/907626257 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e880a0600 con 0x55667955db40 2024-09-16T00:20:35.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e7dffb640 1 -- 172.21.15.33:0/907626257 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7e8c005d50 con 0x55667955db40 2024-09-16T00:20:35.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e9f2ba740 10 client.16008 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:35.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/907626257 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5566795e6580 con 0x55667955db40 2024-09-16T00:20:35.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e7dffb640 1 -- 172.21.15.33:0/907626257 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e880a0600 con 0x55667955db40 2024-09-16T00:20:35.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e7dffb640 1 -- 172.21.15.33:0/907626257 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7e8c05b030 con 0x55667955db40 2024-09-16T00:20:35.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e9f2ba740 10 client.16008 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:35.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e9f2ba740 10 client.16008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:20:35.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e9f2ba740 20 client.16008 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:35.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566795e6d80 0x5566795e91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e9f2ba740 4 client.16008 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/907626257 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5566794c9d70 con 0x5566795e6d80 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.594+0000 7f7e9659c640 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566795e6d80 0x5566795e91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9659c640 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566795e6d80 0x5566795e91a0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5bba88b4f81868d server_cookie=1c88ea5ecf82bb62 in_seq=0 out_seq=0 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e7dffb640 10 client.16008 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e7dffb640 1 -- 172.21.15.33:0/907626257 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5566794c9d70 con 0x5566795e6d80 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e7dffb640 10 client.16008 handle_command_reply: tid=0 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9f2ba740 1 client.16008 shutdown 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/907626257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667955db40 msgr2=0x55667956fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667955db40 0x55667956fd00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009900 tx=0x7f7e8c0048f0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/907626257 shutdown_connections 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566795e6d80 0x5566795e91a0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e7dffb640 0 client.16008 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:35.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55667956b780 0x556679570240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55667955db40 0x55667956fd00 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9f2ba740 1 --2- 172.21.15.33:0/907626257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55667955c5d0 0x55667956f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/907626257 >> 172.21.15.33:0/907626257 conn(0x556679525310 msgr2=0x556679567580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:35.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/907626257 shutdown_connections 2024-09-16T00:20:35.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9f2ba740 1 -- 172.21.15.33:0/907626257 wait complete. 2024-09-16T00:20:35.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.595+0000 7f7e9f2ba740 20 client.16008 trim_cache size 0 max 16384 2024-09-16T00:20:35.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.596+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3022718693 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7e800a3340 msgr2=0x7f7e800a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:35.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.596+0000 7f7e9cee1640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7e800a3340 0x7f7e800a5800 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7e900099a0 tx=0x7f7e90009500 comp rx=0 tx=0).stop 2024-09-16T00:20:35.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.596+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3022718693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e98102df0 msgr2=0x7f7e98198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:35.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.596+0000 7f7e9cee1640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e98102df0 0x7f7e98198630 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800e9f0 tx=0x7f7e8800eeb0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.596+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3022718693 shutdown_connections 2024-09-16T00:20:35.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.596+0000 7f7e9cee1640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7e800a3340 0x7f7e800a5800 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.596+0000 7f7e9cee1640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e9810c2f0 0x7f7e9819cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.596+0000 7f7e9cee1640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e981037a0 0x7f7e98198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.596+0000 7f7e9cee1640 1 --2- 172.21.15.33:0/3022718693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e98102df0 0x7f7e98198630 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:35.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.596+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3022718693 >> 172.21.15.33:0/3022718693 conn(0x7f7e980fc7d0 msgr2=0x7f7e980fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:35.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.596+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3022718693 shutdown_connections 2024-09-16T00:20:35.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:35.596+0000 7f7e9cee1640 1 -- 172.21.15.33:0/3022718693 wait complete. 2024-09-16T00:20:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:35 smithi083 ceph-mon[35242]: pgmap v453: 129 pgs: 129 active+clean; 42 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 650 KiB/s wr, 94 op/s 2024-09-16T00:20:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:35 smithi060 ceph-mon[35185]: pgmap v453: 129 pgs: 129 active+clean; 42 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 650 KiB/s wr, 94 op/s 2024-09-16T00:20:36.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 72382 3.93 MB/sec warmup 104 sec latency 75.279 ms 2024-09-16T00:20:37.628 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:20:37.629 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:20:37.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 75237 4.03 MB/sec warmup 105 sec latency 20.467 ms 2024-09-16T00:20:37.908 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:37 smithi083 ceph-mon[35242]: pgmap v454: 129 pgs: 129 active+clean; 43 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 963 KiB/s wr, 109 op/s 2024-09-16T00:20:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:37 smithi060 ceph-mon[35185]: pgmap v454: 129 pgs: 129 active+clean; 43 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 963 KiB/s wr, 109 op/s 2024-09-16T00:20:38.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:37 smithi033 ceph-mon[31243]: pgmap v454: 129 pgs: 129 active+clean; 43 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 963 KiB/s wr, 109 op/s 2024-09-16T00:20:38.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.418+0000 7f23b1af1640 1 -- 172.21.15.33:0/2122287188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23ac106a80 msgr2=0x7f23ac106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:38.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.418+0000 7f23b1af1640 1 --2- 172.21.15.33:0/2122287188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23ac106a80 0x7f23ac106e60 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f239c009930 tx=0x7f239c02a420 comp rx=0 tx=0).stop 2024-09-16T00:20:38.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.418+0000 7f23b1af1640 1 -- 172.21.15.33:0/2122287188 shutdown_connections 2024-09-16T00:20:38.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.418+0000 7f23b1af1640 1 --2- 172.21.15.33:0/2122287188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23ac101420 0x7f23ac10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.418+0000 7f23b1af1640 1 --2- 172.21.15.33:0/2122287188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23ac100a80 0x7f23ac100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.418+0000 7f23b1af1640 1 --2- 172.21.15.33:0/2122287188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23ac106a80 0x7f23ac106e60 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.418+0000 7f23b1af1640 1 -- 172.21.15.33:0/2122287188 >> 172.21.15.33:0/2122287188 conn(0x7f23ac0fc7a0 msgr2=0x7f23ac0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:38.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.419+0000 7f23b1af1640 1 -- 172.21.15.33:0/2122287188 shutdown_connections 2024-09-16T00:20:38.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.419+0000 7f23b1af1640 1 -- 172.21.15.33:0/2122287188 wait complete. 2024-09-16T00:20:38.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.419+0000 7f23b1af1640 1 Processor -- start 2024-09-16T00:20:38.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23b1af1640 1 -- start start 2024-09-16T00:20:38.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23b1af1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23ac100a80 0x7f23ac1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23b1af1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23ac101420 0x7f23ac199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23b1af1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23ac106a80 0x7f23ac1a4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23b1af1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac110e80 con 0x7f23ac106a80 2024-09-16T00:20:38.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23b1af1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac110d00 con 0x7f23ac100a80 2024-09-16T00:20:38.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23b1af1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac111000 con 0x7f23ac101420 2024-09-16T00:20:38.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23abfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23ac106a80 0x7f23ac1a4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23ac101420 0x7f23ac199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23ab7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23ac100a80 0x7f23ac1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23abfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23ac106a80 0x7f23ac1a4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48834/0 (socket says 172.21.15.33:48834) 2024-09-16T00:20:38.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23ab7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23ac100a80 0x7f23ac1992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:38862/0 (socket says 172.21.15.33:38862) 2024-09-16T00:20:38.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23ac101420 0x7f23ac199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41690/0 (socket says 172.21.15.33:41690) 2024-09-16T00:20:38.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23abfff640 1 -- 172.21.15.33:0/2603426838 learned_addr learned my addr 172.21.15.33:0/2603426838 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:38.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23aaffd640 1 -- 172.21.15.33:0/2603426838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23ac100a80 msgr2=0x7f23ac1992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:38.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.420+0000 7f23aaffd640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23ac100a80 0x7f23ac1992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.421+0000 7f23aaffd640 1 -- 172.21.15.33:0/2603426838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23ac106a80 msgr2=0x7f23ac1a4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:38.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.421+0000 7f23aaffd640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23ac106a80 0x7f23ac1a4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.421+0000 7f23aaffd640 1 -- 172.21.15.33:0/2603426838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ac1a48b0 con 0x7f23ac101420 2024-09-16T00:20:38.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.421+0000 7f23abfff640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23ac106a80 0x7f23ac1a4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:38.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.421+0000 7f23aaffd640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23ac101420 0x7f23ac199830 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f23a000e9f0 tx=0x7f23a000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:38.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.421+0000 7f23a8ff9640 1 -- 172.21.15.33:0/2603426838 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a000ccc0 con 0x7f23ac101420 2024-09-16T00:20:38.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.421+0000 7f23b1af1640 1 -- 172.21.15.33:0/2603426838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23ac1970e0 con 0x7f23ac101420 2024-09-16T00:20:38.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.421+0000 7f23a8ff9640 1 -- 172.21.15.33:0/2603426838 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23a0010910 con 0x7f23ac101420 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.421+0000 7f23a8ff9640 1 -- 172.21.15.33:0/2603426838 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a0018980 con 0x7f23ac101420 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.421+0000 7f23b1af1640 1 -- 172.21.15.33:0/2603426838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23ac197620 con 0x7f23ac101420 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.421+0000 7f23ab7fe640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23ac100a80 0x7f23ac1992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.424+0000 7f23a8ff9640 1 -- 172.21.15.33:0/2603426838 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f23a0010ac0 con 0x7f23ac101420 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.424+0000 7f23b3eca740 1 Processor -- start 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.424+0000 7f23b3eca740 1 -- start start 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.424+0000 7f23b3eca740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dbbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.424+0000 7f23b3eca740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.424+0000 7f23b3eca740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925dc9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.424+0000 7f23b3eca740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555925d21280 con 0x555925dbbb60 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.424+0000 7f23b3eca740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555925cc8b10 con 0x555925dc6d40 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.424+0000 7f23b3eca740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555925d1d240 con 0x555925dbc480 2024-09-16T00:20:38.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.424+0000 7f23a8ff9640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f23940a3240 0x7f23940a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.425+0000 7f23aaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925dc9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.425+0000 7f23abfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dbbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.425+0000 7f23abfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dbbf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48840/0 (socket says 172.21.15.33:48840) 2024-09-16T00:20:38.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.425+0000 7f23abfff640 1 -- 172.21.15.33:0/2981080022 learned_addr learned my addr 172.21.15.33:0/2981080022 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:38.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.425+0000 7f23a8ff9640 1 -- 172.21.15.33:0/2603426838 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f23a00d0120 con 0x7f23ac101420 2024-09-16T00:20:38.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.425+0000 7f23abfff640 1 -- 172.21.15.33:0/2981080022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 msgr2=0x555925dc6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:38.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.425+0000 7f23abfff640 1 --2- 172.21.15.33:0/2981080022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.425+0000 7f23abfff640 1 -- 172.21.15.33:0/2981080022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 msgr2=0x555925dc9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:38.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.425+0000 7f23abfff640 1 --2- 172.21.15.33:0/2981080022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925dc9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.425+0000 7f23abfff640 1 -- 172.21.15.33:0/2981080022 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555925cd51f0 con 0x555925dbbb60 2024-09-16T00:20:38.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.425+0000 7f23aaffd640 1 --2- 172.21.15.33:0/2981080022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925dc9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:20:38.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.426+0000 7f23abfff640 1 --2- 172.21.15.33:0/2981080022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dbbf40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f23a40128b0 tx=0x7f23a4012d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fdd82af4fe960a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:38.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.426+0000 7f23abfff640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f23940a3240 0x7f23940a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.426+0000 7f2391ffb640 1 -- 172.21.15.33:0/2981080022 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a401f070 con 0x555925dbbb60 2024-09-16T00:20:38.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.426+0000 7f2391ffb640 1 -- 172.21.15.33:0/2981080022 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23a4010cf0 con 0x555925dbbb60 2024-09-16T00:20:38.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.426+0000 7f2391ffb640 1 -- 172.21.15.33:0/2981080022 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a400e7f0 con 0x555925dbbb60 2024-09-16T00:20:38.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.427+0000 7f23b3eca740 1 -- 172.21.15.33:0/2981080022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 msgr2=0x555925dbbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:38.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.427+0000 7f23abfff640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f23940a3240 0x7f23940a5700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f23a401c000 tx=0x7f23a4018700 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:38.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.427+0000 7f23b3eca740 1 --2- 172.21.15.33:0/2981080022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dbbf40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f23a40128b0 tx=0x7f23a4012d70 comp rx=0 tx=0).stop 2024-09-16T00:20:38.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.427+0000 7f23b3eca740 1 -- 172.21.15.33:0/2981080022 shutdown_connections 2024-09-16T00:20:38.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.427+0000 7f23b3eca740 1 --2- 172.21.15.33:0/2981080022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925dc9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.427+0000 7f23b3eca740 1 --2- 172.21.15.33:0/2981080022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.427+0000 7f23b3eca740 1 --2- 172.21.15.33:0/2981080022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dbbf40 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.427+0000 7f23b3eca740 1 -- 172.21.15.33:0/2981080022 >> 172.21.15.33:0/2981080022 conn(0x555925db80f0 msgr2=0x555925db84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:38.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.427+0000 7f23b3eca740 1 -- 172.21.15.33:0/2981080022 shutdown_connections 2024-09-16T00:20:38.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.427+0000 7f23b3eca740 1 -- 172.21.15.33:0/2981080022 wait complete. 2024-09-16T00:20:38.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.427+0000 7f23b3eca740 1 -- start start 2024-09-16T00:20:38.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23b3eca740 1 Processor -- start 2024-09-16T00:20:38.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23b3eca740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dc3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23b3eca740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23b3eca740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925dc4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23b3eca740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555925dc9910 con 0x555925dbbb60 2024-09-16T00:20:38.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23b3eca740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555925cdbf10 con 0x555925dc6d40 2024-09-16T00:20:38.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23b3eca740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555925dc9a90 con 0x555925dbc480 2024-09-16T00:20:38.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23b3eca740 10 client.0 _ll_register_callbacks cb 0x555925ca4a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:38.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23ab7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dc3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23ab7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dc3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48854/0 (socket says 172.21.15.33:48854) 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23ab7fe640 1 -- 172.21.15.33:0/3055443150 learned_addr learned my addr 172.21.15.33:0/3055443150 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23aaffd640 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23abfff640 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925dc4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23aaffd640 1 -- 172.21.15.33:0/3055443150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 msgr2=0x555925dc4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.428+0000 7f23aaffd640 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925dc4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f23aaffd640 1 -- 172.21.15.33:0/3055443150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 msgr2=0x555925dc3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f23aaffd640 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dc3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f23aaffd640 1 -- 172.21.15.33:0/3055443150 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555925a926a0 con 0x555925dbc480 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f23ab7fe640 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dc3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f23abfff640 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925dc4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f23aaffd640 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc40f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f23a00cd7f0 tx=0x7f23a009bfb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91987e3c9eafd3e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f238affd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:20:38.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f238affd640 1 -- 172.21.15.33:0/3055443150 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a00bc2c0 con 0x555925dbc480 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f238affd640 1 -- 172.21.15.33:0/3055443150 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f23a00a0140 con 0x555925dbc480 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f23b3eca740 1 -- 172.21.15.33:0/3055443150 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23a4010cf0 con 0x555925dbc480 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f238affd640 1 -- 172.21.15.33:0/3055443150 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f23a00a4f30 con 0x555925dbc480 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f23b3eca740 10 client.25504 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f23b3eca740 1 -- 172.21.15.33:0/3055443150 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555925dd54a0 con 0x555925dbc480 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f238affd640 1 -- 172.21.15.33:0/3055443150 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23a4010cf0 con 0x555925dbc480 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.429+0000 7f238affd640 1 -- 172.21.15.33:0/3055443150 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f23a009b850 con 0x555925dbc480 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.430+0000 7f23b3eca740 10 client.25504 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.430+0000 7f23b3eca740 10 client.25504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.430+0000 7f23b3eca740 20 client.25504 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.430+0000 7f23b3eca740 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555925dd6960 0x555925dd8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.430+0000 7f23b3eca740 4 client.25504 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:20:38.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.430+0000 7f23b3eca740 1 -- 172.21.15.33:0/3055443150 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555925d2e7f0 con 0x555925dd6960 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.430+0000 7f23ab7fe640 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555925dd6960 0x555925dd8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.430+0000 7f238affd640 10 client.25504 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.430+0000 7f23ab7fe640 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555925dd6960 0x555925dd8d90 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3724777799b143b server_cookie=1d5af70208b1cc57 in_seq=0 out_seq=0 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f238affd640 1 -- 172.21.15.33:0/3055443150 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555925d2e7f0 con 0x555925dd6960 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f238affd640 10 client.25504 handle_command_reply: tid=0 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f23b3eca740 1 client.25504 shutdown 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f23b3eca740 1 -- 172.21.15.33:0/3055443150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 msgr2=0x555925dc40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f23b3eca740 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc40f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f23a00cd7f0 tx=0x7f23a009bfb0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f23b3eca740 1 -- 172.21.15.33:0/3055443150 shutdown_connections 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f23b3eca740 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555925dd6960 0x555925dd8d90 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f23b3eca740 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925dc4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f23b3eca740 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc40f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f23b3eca740 1 --2- 172.21.15.33:0/3055443150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dc3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f23b3eca740 1 -- 172.21.15.33:0/3055443150 >> 172.21.15.33:0/3055443150 conn(0x555925db80f0 msgr2=0x555925dbddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:38.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f238affd640 0 client.25504 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:38.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f23b3eca740 1 -- 172.21.15.33:0/3055443150 shutdown_connections 2024-09-16T00:20:38.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f23b3eca740 1 -- 172.21.15.33:0/3055443150 wait complete. 2024-09-16T00:20:38.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.431+0000 7f23b3eca740 20 client.25504 trim_cache size 0 max 16384 2024-09-16T00:20:38.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.451+0000 7f23b3eca740 1 Processor -- start 2024-09-16T00:20:38.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.451+0000 7f23b3eca740 1 -- start start 2024-09-16T00:20:38.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23b3eca740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dc5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23b3eca740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dde610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23b3eca740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925ddeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23b3eca740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac1af8a0 con 0x555925dbbb60 2024-09-16T00:20:38.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23b3eca740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac1ad170 con 0x555925dc6d40 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23b3eca740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac1aafc0 con 0x555925dbc480 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23ab7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dc5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dde610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dde610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41726/0 (socket says 172.21.15.33:41726) 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23aaffd640 1 -- 172.21.15.33:0/3205546045 learned_addr learned my addr 172.21.15.33:0/3205546045 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23ab7fe640 1 --2- 172.21.15.33:0/3205546045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dc5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48870/0 (socket says 172.21.15.33:48870) 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23abfff640 1 --2- 172.21.15.33:0/3205546045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925ddeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23aaffd640 1 -- 172.21.15.33:0/3205546045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 msgr2=0x555925ddeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23aaffd640 1 --2- 172.21.15.33:0/3205546045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925ddeb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23aaffd640 1 -- 172.21.15.33:0/3205546045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 msgr2=0x555925dc5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23aaffd640 1 --2- 172.21.15.33:0/3205546045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dc5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.452+0000 7f23aaffd640 1 -- 172.21.15.33:0/3205546045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555925db7f20 con 0x555925dbc480 2024-09-16T00:20:38.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.453+0000 7f23aaffd640 1 --2- 172.21.15.33:0/3205546045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dde610 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f23a00987d0 tx=0x7f23a0010e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3d47e7c1829bfad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:38.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.453+0000 7f23917fa640 1 -- 172.21.15.33:0/3205546045 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a00b5200 con 0x555925dbc480 2024-09-16T00:20:38.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.453+0000 7f23917fa640 1 -- 172.21.15.33:0/3205546045 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23a009b7c0 con 0x555925dbc480 2024-09-16T00:20:38.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.453+0000 7f23917fa640 1 -- 172.21.15.33:0/3205546045 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a00a03a0 con 0x555925dbc480 2024-09-16T00:20:38.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.453+0000 7f23b3eca740 1 -- 172.21.15.33:0/3205546045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 msgr2=0x555925dde610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:38.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.453+0000 7f23b3eca740 1 --2- 172.21.15.33:0/3205546045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dde610 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f23a00987d0 tx=0x7f23a0010e30 comp rx=0 tx=0).stop 2024-09-16T00:20:38.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.454+0000 7f23b3eca740 1 -- 172.21.15.33:0/3205546045 shutdown_connections 2024-09-16T00:20:38.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.454+0000 7f23b3eca740 1 --2- 172.21.15.33:0/3205546045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dc6d40 0x555925ddeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.454+0000 7f23b3eca740 1 --2- 172.21.15.33:0/3205546045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dde610 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.454+0000 7f23b3eca740 1 --2- 172.21.15.33:0/3205546045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dbbb60 0x555925dc5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.454+0000 7f23b3eca740 1 -- 172.21.15.33:0/3205546045 >> 172.21.15.33:0/3205546045 conn(0x555925db80f0 msgr2=0x555925dc5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.454+0000 7f23b3eca740 1 -- 172.21.15.33:0/3205546045 shutdown_connections 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.454+0000 7f23b3eca740 1 -- 172.21.15.33:0/3205546045 wait complete. 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.454+0000 7f23b3eca740 1 -- start start 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.454+0000 7f23b3eca740 1 Processor -- start 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.454+0000 7f23b3eca740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dbbb60 0x555925dc8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23b3eca740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23b3eca740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dc6d40 0x555925dc8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23b3eca740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac1ab630 con 0x555925dc6d40 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23b3eca740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac1acaa0 con 0x555925dbbb60 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23b3eca740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac1abd00 con 0x555925dbc480 2024-09-16T00:20:38.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23b3eca740 10 client.0 _ll_register_callbacks cb 0x7f23a000cf00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:38.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23abfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dc6d40 0x555925dc8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23ab7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dbbb60 0x555925dc8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23abfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dc6d40 0x555925dc8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48878/0 (socket says 172.21.15.33:48878) 2024-09-16T00:20:38.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23abfff640 1 -- 172.21.15.33:0/1260972896 learned_addr learned my addr 172.21.15.33:0/1260972896 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:38.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23aaffd640 1 -- 172.21.15.33:0/1260972896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dbbb60 msgr2=0x555925dc8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:38.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23aaffd640 1 --2- 172.21.15.33:0/1260972896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dbbb60 0x555925dc8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23aaffd640 1 -- 172.21.15.33:0/1260972896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dc6d40 msgr2=0x555925dc8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:38.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23aaffd640 1 --2- 172.21.15.33:0/1260972896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dc6d40 0x555925dc8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23aaffd640 1 -- 172.21.15.33:0/1260972896 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555925dcd3a0 con 0x555925dbc480 2024-09-16T00:20:38.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f23aaffd640 1 --2- 172.21.15.33:0/1260972896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc8540 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f23a00987d0 tx=0x7f23a009bfb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4a8cb3238c8acaa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:38.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f238affd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f238affd640 1 -- 172.21.15.33:0/1260972896 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a009c370 con 0x555925dbc480 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f23b3eca740 1 -- 172.21.15.33:0/1260972896 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23a009b7c0 con 0x555925dbc480 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.455+0000 7f238affd640 1 -- 172.21.15.33:0/1260972896 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f23a00a0140 con 0x555925dbc480 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f238affd640 1 -- 172.21.15.33:0/1260972896 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f23a00a0340 con 0x555925dbc480 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f23b3eca740 10 client.25510 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f23b3eca740 1 -- 172.21.15.33:0/1260972896 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555925e419a0 con 0x555925dbc480 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f238affd640 1 -- 172.21.15.33:0/1260972896 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23a009b7c0 con 0x555925dbc480 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f238affd640 1 -- 172.21.15.33:0/1260972896 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f23a0085190 con 0x555925dbc480 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f23b3eca740 10 client.25510 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f23b3eca740 10 client.25510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f23b3eca740 20 client.25510 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f23b3eca740 1 --2- 172.21.15.33:0/1260972896 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555925e41d10 0x555925e44130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f23b3eca740 4 client.25510 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:20:38.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f23b3eca740 1 -- 172.21.15.33:0/1260972896 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555925d28d00 con 0x555925e41d10 2024-09-16T00:20:38.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.456+0000 7f23ab7fe640 1 --2- 172.21.15.33:0/1260972896 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555925e41d10 0x555925e44130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:38.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.457+0000 7f23ab7fe640 1 --2- 172.21.15.33:0/1260972896 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555925e41d10 0x555925e44130 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb47109fa00fcd9b server_cookie=d908e80acb54234b in_seq=0 out_seq=0 2024-09-16T00:20:38.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.457+0000 7f238affd640 10 client.25510 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:38.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f238affd640 1 -- 172.21.15.33:0/1260972896 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555925d28d00 con 0x555925e41d10 2024-09-16T00:20:38.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f238affd640 10 client.25510 handle_command_reply: tid=0 2024-09-16T00:20:38.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f23b3eca740 1 client.25510 shutdown 2024-09-16T00:20:38.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f23b3eca740 1 -- 172.21.15.33:0/1260972896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 msgr2=0x555925dc8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:38.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f23b3eca740 1 --2- 172.21.15.33:0/1260972896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc8540 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f23a00987d0 tx=0x7f23a009bfb0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f23b3eca740 1 -- 172.21.15.33:0/1260972896 shutdown_connections 2024-09-16T00:20:38.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f23b3eca740 1 --2- 172.21.15.33:0/1260972896 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555925e41d10 0x555925e44130 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f23b3eca740 1 --2- 172.21.15.33:0/1260972896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555925dc6d40 0x555925dc8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f23b3eca740 1 --2- 172.21.15.33:0/1260972896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555925dbc480 0x555925dc8540 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f23b3eca740 1 --2- 172.21.15.33:0/1260972896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555925dbbb60 0x555925dc8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f238affd640 0 client.25510 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:38.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f23b3eca740 1 -- 172.21.15.33:0/1260972896 >> 172.21.15.33:0/1260972896 conn(0x555925db80f0 msgr2=0x555925db7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:38.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f23b3eca740 1 -- 172.21.15.33:0/1260972896 shutdown_connections 2024-09-16T00:20:38.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f23b3eca740 1 -- 172.21.15.33:0/1260972896 wait complete. 2024-09-16T00:20:38.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.458+0000 7f23b3eca740 20 client.25510 trim_cache size 0 max 16384 2024-09-16T00:20:38.463 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:20:38.463 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:20:38.463 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "5ce3264d-b595-48e6-9f68-73df11db7a2f", 2024-09-16T00:20:38.463 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:20:38.464 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:20:38.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.459+0000 7f23b1af1640 1 -- 172.21.15.33:0/2603426838 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f23940a3240 msgr2=0x7f23940a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:38.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.459+0000 7f23b1af1640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f23940a3240 0x7f23940a5700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f23a401c000 tx=0x7f23a4018700 comp rx=0 tx=0).stop 2024-09-16T00:20:38.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.459+0000 7f23b1af1640 1 -- 172.21.15.33:0/2603426838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23ac101420 msgr2=0x7f23ac199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:38.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.459+0000 7f23b1af1640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23ac101420 0x7f23ac199830 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f23a000e9f0 tx=0x7f23a000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.459+0000 7f23b1af1640 1 -- 172.21.15.33:0/2603426838 shutdown_connections 2024-09-16T00:20:38.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.459+0000 7f23b1af1640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f23940a3240 0x7f23940a5700 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.459+0000 7f23b1af1640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23ac106a80 0x7f23ac1a4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.459+0000 7f23b1af1640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23ac101420 0x7f23ac199830 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.459+0000 7f23b1af1640 1 --2- 172.21.15.33:0/2603426838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23ac100a80 0x7f23ac1992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:38.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.459+0000 7f23b1af1640 1 -- 172.21.15.33:0/2603426838 >> 172.21.15.33:0/2603426838 conn(0x7f23ac0fc7a0 msgr2=0x7f23ac0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:38.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.459+0000 7f23b1af1640 1 -- 172.21.15.33:0/2603426838 shutdown_connections 2024-09-16T00:20:38.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:38.459+0000 7f23b1af1640 1 -- 172.21.15.33:0/2603426838 wait complete. 2024-09-16T00:20:38.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 78303 4.16 MB/sec warmup 106 sec latency 17.170 ms 2024-09-16T00:20:38.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:38 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:20:38.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:38 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:20:38.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:38 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:20:38.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:38 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:20:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:38 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:20:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:38 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:20:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:38 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:20:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:38 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:20:39.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:38 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:20:39.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:38 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:20:39.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:38 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:20:39.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:38 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:20:39.495 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:20:39.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 80985 4.24 MB/sec warmup 107 sec latency 19.475 ms 2024-09-16T00:20:39.808 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:20:39.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:39 smithi033 ceph-mon[31243]: pgmap v455: 129 pgs: 129 active+clean; 46 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-09-16T00:20:39.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:39 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:20:39.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:39 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:20:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:39 smithi083 ceph-mon[35242]: pgmap v455: 129 pgs: 129 active+clean; 46 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-09-16T00:20:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:39 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:20:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:39 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:20:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:39 smithi060 ceph-mon[35185]: pgmap v455: 129 pgs: 129 active+clean; 46 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-09-16T00:20:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:39 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:20:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:39 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:20:40.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.295+0000 7f043315a640 1 -- 172.21.15.33:0/2950925609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f042c100a80 msgr2=0x7f042c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:40.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.295+0000 7f043315a640 1 --2- 172.21.15.33:0/2950925609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f042c100a80 0x7f042c100ee0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f041c009930 tx=0x7f041c02a420 comp rx=0 tx=0).stop 2024-09-16T00:20:40.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.295+0000 7f043315a640 1 -- 172.21.15.33:0/2950925609 shutdown_connections 2024-09-16T00:20:40.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.295+0000 7f043315a640 1 --2- 172.21.15.33:0/2950925609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f042c101420 0x7f042c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.295+0000 7f043315a640 1 --2- 172.21.15.33:0/2950925609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f042c100a80 0x7f042c100ee0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.295+0000 7f043315a640 1 --2- 172.21.15.33:0/2950925609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f042c106a80 0x7f042c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.295+0000 7f043315a640 1 -- 172.21.15.33:0/2950925609 >> 172.21.15.33:0/2950925609 conn(0x7f042c0fc7a0 msgr2=0x7f042c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:40.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.295+0000 7f043315a640 1 -- 172.21.15.33:0/2950925609 shutdown_connections 2024-09-16T00:20:40.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.295+0000 7f043315a640 1 -- 172.21.15.33:0/2950925609 wait complete. 2024-09-16T00:20:40.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f043315a640 1 Processor -- start 2024-09-16T00:20:40.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f043315a640 1 -- start start 2024-09-16T00:20:40.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f043315a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f042c100a80 0x7f042c1992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f043315a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f042c101420 0x7f042c199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f043315a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f042c106a80 0x7f042c1a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f043315a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f042c110e80 con 0x7f042c100a80 2024-09-16T00:20:40.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f043315a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f042c110d00 con 0x7f042c106a80 2024-09-16T00:20:40.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f043315a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f042c111000 con 0x7f042c101420 2024-09-16T00:20:40.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f0430ef6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f042c100a80 0x7f042c1992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f0423fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f042c101420 0x7f042c199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f0430ef6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f042c100a80 0x7f042c1992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46646/0 (socket says 172.21.15.33:46646) 2024-09-16T00:20:40.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f0430ef6640 1 -- 172.21.15.33:0/537646371 learned_addr learned my addr 172.21.15.33:0/537646371 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:40.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.296+0000 7f04316f7640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f042c106a80 0x7f042c1a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.297+0000 7f0430ef6640 1 -- 172.21.15.33:0/537646371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f042c101420 msgr2=0x7f042c199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:40.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.297+0000 7f0430ef6640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f042c101420 0x7f042c199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.297+0000 7f0430ef6640 1 -- 172.21.15.33:0/537646371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f042c106a80 msgr2=0x7f042c1a4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:40.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.297+0000 7f0430ef6640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f042c106a80 0x7f042c1a4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.297+0000 7f0430ef6640 1 -- 172.21.15.33:0/537646371 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042c1a4890 con 0x7f042c100a80 2024-09-16T00:20:40.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.297+0000 7f0423fff640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f042c101420 0x7f042c199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.297+0000 7f0430ef6640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f042c100a80 0x7f042c1992d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f042800e9f0 tx=0x7f042800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.297+0000 7f0421ffb640 1 -- 172.21.15.33:0/537646371 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042800cd80 con 0x7f042c100a80 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.297+0000 7f0421ffb640 1 -- 172.21.15.33:0/537646371 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0428010910 con 0x7f042c100a80 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.297+0000 7f043315a640 1 -- 172.21.15.33:0/537646371 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f042c1a4a60 con 0x7f042c100a80 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.297+0000 7f0421ffb640 1 -- 172.21.15.33:0/537646371 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428018990 con 0x7f042c100a80 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.297+0000 7f043315a640 1 -- 172.21.15.33:0/537646371 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f042c1a4f70 con 0x7f042c100a80 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.300+0000 7f0421ffb640 1 -- 172.21.15.33:0/537646371 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0428010ac0 con 0x7f042c100a80 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.300+0000 7f0435533740 1 Processor -- start 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.300+0000 7f0435533740 1 -- start start 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f0435533740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8d9a4a0 0x5563a8d9a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f0435533740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9adc0 0x5563a8da5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f0435533740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8da5680 0x5563a8da7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f0435533740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5563a8d002e0 con 0x5563a8da5680 2024-09-16T00:20:40.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f0435533740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5563a8ca7ce0 con 0x5563a8d9a4a0 2024-09-16T00:20:40.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f0435533740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5563a8cfc2a0 con 0x5563a8d9adc0 2024-09-16T00:20:40.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f04316f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8d9a4a0 0x5563a8d9a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f0423fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8da5680 0x5563a8da7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f0423fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8da5680 0x5563a8da7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46660/0 (socket says 172.21.15.33:46660) 2024-09-16T00:20:40.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f0421ffb640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f04140a32c0 0x7f04140a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f04316f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8d9a4a0 0x5563a8d9a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:43312/0 (socket says 172.21.15.33:43312) 2024-09-16T00:20:40.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f0423fff640 1 -- 172.21.15.33:0/2799568534 learned_addr learned my addr 172.21.15.33:0/2799568534 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:40.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f0421ffb640 1 -- 172.21.15.33:0/537646371 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f04280cffc0 con 0x7f042c100a80 2024-09-16T00:20:40.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.301+0000 7f04316f7640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f04140a32c0 0x7f04140a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.302+0000 7f0430ef6640 1 --2- 172.21.15.33:0/2799568534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9adc0 0x5563a8da5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.302+0000 7f0423fff640 1 -- 172.21.15.33:0/2799568534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9adc0 msgr2=0x5563a8da5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:40.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.302+0000 7f0423fff640 1 --2- 172.21.15.33:0/2799568534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9adc0 0x5563a8da5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.302+0000 7f0423fff640 1 -- 172.21.15.33:0/2799568534 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8d9a4a0 msgr2=0x5563a8d9a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:40.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.302+0000 7f0423fff640 1 --2- 172.21.15.33:0/2799568534 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8d9a4a0 0x5563a8d9a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.302+0000 7f0423fff640 1 -- 172.21.15.33:0/2799568534 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563a8c8c440 con 0x5563a8da5680 2024-09-16T00:20:40.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.302+0000 7f0423fff640 1 --2- 172.21.15.33:0/2799568534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8da5680 0x5563a8da7a70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f041c000c00 tx=0x7f041c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8dba940d71dfa99 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:40.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.302+0000 7f04316f7640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f04140a32c0 0x7f04140a5780 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0424005fd0 tx=0x7f0424004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:40.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.302+0000 7f0412ffd640 1 -- 172.21.15.33:0/2799568534 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c035040 con 0x5563a8da5680 2024-09-16T00:20:40.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.303+0000 7f0412ffd640 1 -- 172.21.15.33:0/2799568534 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f041c004030 con 0x5563a8da5680 2024-09-16T00:20:40.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.303+0000 7f0412ffd640 1 -- 172.21.15.33:0/2799568534 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c03ddb0 con 0x5563a8da5680 2024-09-16T00:20:40.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.303+0000 7f0435533740 1 -- 172.21.15.33:0/2799568534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8da5680 msgr2=0x5563a8da7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:40.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.303+0000 7f0435533740 1 --2- 172.21.15.33:0/2799568534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8da5680 0x5563a8da7a70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f041c000c00 tx=0x7f041c005d50 comp rx=0 tx=0).stop 2024-09-16T00:20:40.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.303+0000 7f0435533740 1 -- 172.21.15.33:0/2799568534 shutdown_connections 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.303+0000 7f0435533740 1 --2- 172.21.15.33:0/2799568534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8da5680 0x5563a8da7a70 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.303+0000 7f0435533740 1 --2- 172.21.15.33:0/2799568534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9adc0 0x5563a8da5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.303+0000 7f0435533740 1 --2- 172.21.15.33:0/2799568534 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8d9a4a0 0x5563a8d9a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.303+0000 7f0435533740 1 -- 172.21.15.33:0/2799568534 >> 172.21.15.33:0/2799568534 conn(0x5563a8d96a30 msgr2=0x5563a8d96e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.304+0000 7f0435533740 1 -- 172.21.15.33:0/2799568534 shutdown_connections 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.304+0000 7f0435533740 1 -- 172.21.15.33:0/2799568534 wait complete. 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.304+0000 7f0435533740 1 -- start start 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.304+0000 7f0435533740 1 Processor -- start 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f0435533740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9a4a0 0x5563a8da4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f0435533740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 0x5563a8da2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f0435533740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8da5680 0x5563a8da26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f0435533740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5563a8da81f0 con 0x5563a8d9adc0 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f0435533740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5563a8cbb050 con 0x5563a8da5680 2024-09-16T00:20:40.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f0435533740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5563a8da8370 con 0x5563a8d9a4a0 2024-09-16T00:20:40.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f04316f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8da5680 0x5563a8da26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f0423fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 0x5563a8da2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f04316f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8da5680 0x5563a8da26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:43318/0 (socket says 172.21.15.33:43318) 2024-09-16T00:20:40.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f04316f7640 1 -- 172.21.15.33:0/3401815143 learned_addr learned my addr 172.21.15.33:0/3401815143 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:40.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f0430ef6640 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9a4a0 0x5563a8da4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f0435533740 10 client.0 _ll_register_callbacks cb 0x5563a8cfb160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:40.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f04316f7640 1 -- 172.21.15.33:0/3401815143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9a4a0 msgr2=0x5563a8da4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:40.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f04316f7640 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9a4a0 0x5563a8da4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f04316f7640 1 -- 172.21.15.33:0/3401815143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 msgr2=0x5563a8da2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:40.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.305+0000 7f04316f7640 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 0x5563a8da2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.306+0000 7f04316f7640 1 -- 172.21.15.33:0/3401815143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563a8a713c0 con 0x5563a8da5680 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.306+0000 7f0423fff640 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 0x5563a8da2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.306+0000 7f04316f7640 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8da5680 0x5563a8da26a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f042400fc20 tx=0x7f0424004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fc50370ffb73a7f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.306+0000 7f040bfff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.306+0000 7f040bfff640 1 -- 172.21.15.33:0/3401815143 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042400fe50 con 0x5563a8da5680 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.306+0000 7f040bfff640 1 -- 172.21.15.33:0/3401815143 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0424004b20 con 0x5563a8da5680 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.306+0000 7f0435533740 1 -- 172.21.15.33:0/3401815143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f041c004030 con 0x5563a8da5680 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.307+0000 7f040bfff640 1 -- 172.21.15.33:0/3401815143 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0424004d20 con 0x5563a8da5680 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.307+0000 7f0435533740 10 client.25613 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.307+0000 7f0435533740 1 -- 172.21.15.33:0/3401815143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5563a8db3b10 con 0x5563a8da5680 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.307+0000 7f040bfff640 1 -- 172.21.15.33:0/3401815143 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f041c004030 con 0x5563a8da5680 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.307+0000 7f040bfff640 1 -- 172.21.15.33:0/3401815143 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f042401c030 con 0x5563a8da5680 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.307+0000 7f0435533740 10 client.25613 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.307+0000 7f0435533740 10 client.25613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:20:40.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.307+0000 7f0435533740 20 client.25613 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:40.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.307+0000 7f0435533740 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5563a8db4fd0 0x5563a8db7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.307+0000 7f0435533740 4 client.25613 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:20:40.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.307+0000 7f0435533740 1 -- 172.21.15.33:0/3401815143 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563a8d0d850 con 0x5563a8db4fd0 2024-09-16T00:20:40.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.307+0000 7f0430ef6640 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5563a8db4fd0 0x5563a8db7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.308+0000 7f040bfff640 10 client.25613 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:40.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.308+0000 7f0430ef6640 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5563a8db4fd0 0x5563a8db7490 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c882a916524e8085 server_cookie=fbe593654acd605c in_seq=0 out_seq=0 2024-09-16T00:20:40.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.308+0000 7f040bfff640 1 -- 172.21.15.33:0/3401815143 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5563a8d0d850 con 0x5563a8db4fd0 2024-09-16T00:20:40.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.308+0000 7f040bfff640 10 client.25613 handle_command_reply: tid=0 2024-09-16T00:20:40.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.309+0000 7f0435533740 1 client.25613 shutdown 2024-09-16T00:20:40.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.309+0000 7f0435533740 1 -- 172.21.15.33:0/3401815143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8da5680 msgr2=0x5563a8da26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:40.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.309+0000 7f0435533740 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8da5680 0x5563a8da26a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f042400fc20 tx=0x7f0424004a40 comp rx=0 tx=0).stop 2024-09-16T00:20:40.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.309+0000 7f0435533740 1 -- 172.21.15.33:0/3401815143 shutdown_connections 2024-09-16T00:20:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.309+0000 7f0435533740 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5563a8db4fd0 0x5563a8db7490 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.309+0000 7f040bfff640 0 client.25613 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.309+0000 7f0435533740 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8da5680 0x5563a8da26a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.309+0000 7f0435533740 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 0x5563a8da2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.309+0000 7f0435533740 1 --2- 172.21.15.33:0/3401815143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9a4a0 0x5563a8da4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.309+0000 7f0435533740 1 -- 172.21.15.33:0/3401815143 >> 172.21.15.33:0/3401815143 conn(0x5563a8d96a30 msgr2=0x5563a8d9c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.310+0000 7f0435533740 1 -- 172.21.15.33:0/3401815143 shutdown_connections 2024-09-16T00:20:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.310+0000 7f0435533740 1 -- 172.21.15.33:0/3401815143 wait complete. 2024-09-16T00:20:40.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.310+0000 7f0435533740 20 client.25613 trim_cache size 0 max 16384 2024-09-16T00:20:40.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0435533740 1 Processor -- start 2024-09-16T00:20:40.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0435533740 1 -- start start 2024-09-16T00:20:40.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0435533740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9a4a0 0x5563a8d96e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0435533740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8d9adc0 0x5563a8d9d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0435533740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8da5680 0x5563a8dbcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0435533740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f042c1af7c0 con 0x5563a8d9a4a0 2024-09-16T00:20:40.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0435533740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f042c1ad090 con 0x5563a8d9adc0 2024-09-16T00:20:40.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0435533740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f042c1aaee0 con 0x5563a8da5680 2024-09-16T00:20:40.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0430ef6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9a4a0 0x5563a8d96e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0430ef6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9a4a0 0x5563a8d96e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46688/0 (socket says 172.21.15.33:46688) 2024-09-16T00:20:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0423fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8d9adc0 0x5563a8d9d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0430ef6640 1 -- 172.21.15.33:0/182048375 learned_addr learned my addr 172.21.15.33:0/182048375 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f04316f7640 1 --2- 172.21.15.33:0/182048375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8da5680 0x5563a8dbcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0423fff640 1 -- 172.21.15.33:0/182048375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8da5680 msgr2=0x5563a8dbcd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0423fff640 1 --2- 172.21.15.33:0/182048375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8da5680 0x5563a8dbcd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0423fff640 1 -- 172.21.15.33:0/182048375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9a4a0 msgr2=0x5563a8d96e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0423fff640 1 --2- 172.21.15.33:0/182048375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9a4a0 0x5563a8d96e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0423fff640 1 -- 172.21.15.33:0/182048375 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563a8da5220 con 0x5563a8d9adc0 2024-09-16T00:20:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.330+0000 7f0430ef6640 1 --2- 172.21.15.33:0/182048375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9a4a0 0x5563a8d96e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:20:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.331+0000 7f0423fff640 1 --2- 172.21.15.33:0/182048375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8d9adc0 0x5563a8d9d760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f041c009900 tx=0x7f041c02ab90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33b4be4aefa01b04 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.331+0000 7f04127fc640 1 -- 172.21.15.33:0/182048375 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c035040 con 0x5563a8d9adc0 2024-09-16T00:20:40.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.331+0000 7f04127fc640 1 -- 172.21.15.33:0/182048375 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f041c031630 con 0x5563a8d9adc0 2024-09-16T00:20:40.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.331+0000 7f04127fc640 1 -- 172.21.15.33:0/182048375 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c005020 con 0x5563a8d9adc0 2024-09-16T00:20:40.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.332+0000 7f0435533740 1 -- 172.21.15.33:0/182048375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8d9adc0 msgr2=0x5563a8d9d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:40.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.332+0000 7f0435533740 1 --2- 172.21.15.33:0/182048375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8d9adc0 0x5563a8d9d760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f041c009900 tx=0x7f041c02ab90 comp rx=0 tx=0).stop 2024-09-16T00:20:40.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.332+0000 7f0435533740 1 -- 172.21.15.33:0/182048375 shutdown_connections 2024-09-16T00:20:40.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.332+0000 7f0435533740 1 --2- 172.21.15.33:0/182048375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8da5680 0x5563a8dbcd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.332+0000 7f0435533740 1 --2- 172.21.15.33:0/182048375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8d9adc0 0x5563a8d9d760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.332+0000 7f0435533740 1 --2- 172.21.15.33:0/182048375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9a4a0 0x5563a8d96e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.332+0000 7f0435533740 1 -- 172.21.15.33:0/182048375 >> 172.21.15.33:0/182048375 conn(0x5563a8d96a30 msgr2=0x5563a8d9d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:40.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.332+0000 7f0435533740 1 -- 172.21.15.33:0/182048375 shutdown_connections 2024-09-16T00:20:40.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.332+0000 7f0435533740 1 -- 172.21.15.33:0/182048375 wait complete. 2024-09-16T00:20:40.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.332+0000 7f0435533740 1 -- start start 2024-09-16T00:20:40.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.332+0000 7f0435533740 1 Processor -- start 2024-09-16T00:20:40.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0435533740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9a4a0 0x5563a8dadd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0435533740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 0x5563a8d9ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0435533740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8da5680 0x5563a8d9f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0435533740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f042c1ab550 con 0x5563a8d9adc0 2024-09-16T00:20:40.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0435533740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f042c1ac9c0 con 0x5563a8da5680 2024-09-16T00:20:40.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0435533740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f042c1abc20 con 0x5563a8d9a4a0 2024-09-16T00:20:40.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0435533740 10 client.0 _ll_register_callbacks cb 0x7f042400fab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:20:40.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0423fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 0x5563a8d9ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0423fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 0x5563a8d9ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46696/0 (socket says 172.21.15.33:46696) 2024-09-16T00:20:40.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0423fff640 1 -- 172.21.15.33:0/2676858417 learned_addr learned my addr 172.21.15.33:0/2676858417 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:20:40.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0430ef6640 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9a4a0 0x5563a8dadd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f04316f7640 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8da5680 0x5563a8d9f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0423fff640 1 -- 172.21.15.33:0/2676858417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9a4a0 msgr2=0x5563a8dadd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:40.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0423fff640 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9a4a0 0x5563a8dadd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0423fff640 1 -- 172.21.15.33:0/2676858417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8da5680 msgr2=0x5563a8d9f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:20:40.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0423fff640 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8da5680 0x5563a8d9f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0423fff640 1 -- 172.21.15.33:0/2676858417 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563a8d9fd20 con 0x5563a8d9adc0 2024-09-16T00:20:40.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f0423fff640 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 0x5563a8d9ef50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f041c02a460 tx=0x7f041c0314c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1b70a454951956 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:20:40.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f040bfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:20:40.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.333+0000 7f040bfff640 1 -- 172.21.15.33:0/2676858417 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c035050 con 0x5563a8d9adc0 2024-09-16T00:20:40.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f040bfff640 1 -- 172.21.15.33:0/2676858417 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f041c0317e0 con 0x5563a8d9adc0 2024-09-16T00:20:40.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f0435533740 1 -- 172.21.15.33:0/2676858417 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f041c031630 con 0x5563a8d9adc0 2024-09-16T00:20:40.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f040bfff640 1 -- 172.21.15.33:0/2676858417 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f041c009520 con 0x5563a8d9adc0 2024-09-16T00:20:40.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f0435533740 10 client.16044 fetch_fsmap learned FSMap version 28 2024-09-16T00:20:40.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f0435533740 1 -- 172.21.15.33:0/2676858417 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5563a8e1fdb0 con 0x5563a8d9adc0 2024-09-16T00:20:40.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f040bfff640 1 -- 172.21.15.33:0/2676858417 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f041c031630 con 0x5563a8d9adc0 2024-09-16T00:20:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f040bfff640 1 -- 172.21.15.33:0/2676858417 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f041c004770 con 0x5563a8d9adc0 2024-09-16T00:20:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f0435533740 10 client.16044 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T00:20:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f0435533740 10 client.16044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:20:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f0435533740 20 client.16044 populate_metadata read hostname 'smithi033' 2024-09-16T00:20:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f0435533740 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5563a8db4fd0 0x5563a8e220c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:20:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f0435533740 4 client.16044 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:20:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.334+0000 7f0435533740 1 -- 172.21.15.33:0/2676858417 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5563a8d07d60 con 0x5563a8db4fd0 2024-09-16T00:20:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0430ef6640 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5563a8db4fd0 0x5563a8e220c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:20:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0430ef6640 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5563a8db4fd0 0x5563a8e220c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d613f2bf96ff95ef server_cookie=c4787e9e73a30f56 in_seq=0 out_seq=0 2024-09-16T00:20:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f040bfff640 10 client.16044 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f040bfff640 1 -- 172.21.15.33:0/2676858417 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5563a8d07d60 con 0x5563a8db4fd0 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f040bfff640 10 client.16044 handle_command_reply: tid=0 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0435533740 1 client.16044 shutdown 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0435533740 1 -- 172.21.15.33:0/2676858417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 msgr2=0x5563a8d9ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0435533740 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 0x5563a8d9ef50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f041c02a460 tx=0x7f041c0314c0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0435533740 1 -- 172.21.15.33:0/2676858417 shutdown_connections 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0435533740 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5563a8db4fd0 0x5563a8e220c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0435533740 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5563a8da5680 0x5563a8d9f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0435533740 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5563a8d9adc0 0x5563a8d9ef50 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0435533740 1 --2- 172.21.15.33:0/2676858417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5563a8d9a4a0 0x5563a8dadd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0435533740 1 -- 172.21.15.33:0/2676858417 >> 172.21.15.33:0/2676858417 conn(0x5563a8d96a30 msgr2=0x5563a8da39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f040bfff640 0 client.16044 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:20:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0435533740 1 -- 172.21.15.33:0/2676858417 shutdown_connections 2024-09-16T00:20:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.335+0000 7f0435533740 1 -- 172.21.15.33:0/2676858417 wait complete. 2024-09-16T00:20:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f0435533740 20 client.16044 trim_cache size 0 max 16384 2024-09-16T00:20:40.344 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:20:40.344 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:20:40.344 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:20:40.344 INFO:teuthology.orchestra.run.smithi033.stdout: "5ce3264d-b595-48e6-9f68-73df11db7a2f": { 2024-09-16T00:20:40.344 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:20:40.344 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "5ce3264d-b595-48e6-9f68-73df11db7a2f", 2024-09-16T00:20:40.345 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:20:40.345 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:40.345 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:20:40.345 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:20:40.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f043315a640 1 -- 172.21.15.33:0/537646371 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f04140a32c0 msgr2=0x7f04140a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:40.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f043315a640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f04140a32c0 0x7f04140a5780 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0424005fd0 tx=0x7f0424004ab0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f043315a640 1 -- 172.21.15.33:0/537646371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f042c100a80 msgr2=0x7f042c1992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:20:40.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f043315a640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f042c100a80 0x7f042c1992d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f042800e9f0 tx=0x7f042800eeb0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f043315a640 1 -- 172.21.15.33:0/537646371 shutdown_connections 2024-09-16T00:20:40.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f043315a640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f04140a32c0 0x7f04140a5780 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f043315a640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f042c106a80 0x7f042c1a4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f043315a640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f042c101420 0x7f042c199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f043315a640 1 --2- 172.21.15.33:0/537646371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f042c100a80 0x7f042c1992d0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:20:40.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f043315a640 1 -- 172.21.15.33:0/537646371 >> 172.21.15.33:0/537646371 conn(0x7f042c0fc7a0 msgr2=0x7f042c0fcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:20:40.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f043315a640 1 -- 172.21.15.33:0/537646371 shutdown_connections 2024-09-16T00:20:40.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:20:40.336+0000 7f043315a640 1 -- 172.21.15.33:0/537646371 wait complete. 2024-09-16T00:20:40.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 83461 4.32 MB/sec warmup 108 sec latency 13.326 ms 2024-09-16T00:20:41.095 INFO:tasks.cephfs.filesystem:scrub status for tag:5ce3264d-b595-48e6-9f68-73df11db7a2f - {'path': '/', 'tag': '5ce3264d-b595-48e6-9f68-73df11db7a2f', 'options': 'recursive,force'} 2024-09-16T00:20:41.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 86145 4.41 MB/sec warmup 109 sec latency 64.029 ms 2024-09-16T00:20:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:41 smithi083 ceph-mon[35242]: pgmap v456: 129 pgs: 129 active+clean; 50 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 217 op/s 2024-09-16T00:20:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:41 smithi060 ceph-mon[35185]: pgmap v456: 129 pgs: 129 active+clean; 50 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 217 op/s 2024-09-16T00:20:42.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:41 smithi033 ceph-mon[31243]: pgmap v456: 129 pgs: 129 active+clean; 50 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 217 op/s 2024-09-16T00:20:42.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 89387 4.54 MB/sec warmup 110 sec latency 99.548 ms 2024-09-16T00:20:43.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 92678 4.67 MB/sec warmup 111 sec latency 11.944 ms 2024-09-16T00:20:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:43 smithi083 ceph-mon[35242]: pgmap v457: 129 pgs: 129 active+clean; 51 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.0 MiB/s wr, 233 op/s 2024-09-16T00:20:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:43 smithi060 ceph-mon[35185]: pgmap v457: 129 pgs: 129 active+clean; 51 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.0 MiB/s wr, 233 op/s 2024-09-16T00:20:44.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:43 smithi033 ceph-mon[31243]: pgmap v457: 129 pgs: 129 active+clean; 51 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.0 MiB/s wr, 233 op/s 2024-09-16T00:20:44.651 INFO:tasks.workunit.client.0.smithi033.stdout: 1 95960 4.79 MB/sec warmup 112 sec latency 13.610 ms 2024-09-16T00:20:45.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 98746 4.87 MB/sec warmup 113 sec latency 19.521 ms 2024-09-16T00:20:45.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:45 smithi083 ceph-mon[35242]: pgmap v458: 129 pgs: 129 active+clean; 54 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.0 MiB/s wr, 229 op/s 2024-09-16T00:20:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:45 smithi060 ceph-mon[35185]: pgmap v458: 129 pgs: 129 active+clean; 54 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.0 MiB/s wr, 229 op/s 2024-09-16T00:20:46.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:45 smithi033 ceph-mon[31243]: pgmap v458: 129 pgs: 129 active+clean; 54 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.0 MiB/s wr, 229 op/s 2024-09-16T00:20:46.653 INFO:tasks.workunit.client.0.smithi033.stdout: 1 101502 4.93 MB/sec warmup 114 sec latency 19.546 ms 2024-09-16T00:20:46.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:46 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:20:46.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:46 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:20:47.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:46 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:20:47.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 103899 5.01 MB/sec warmup 115 sec latency 18.244 ms 2024-09-16T00:20:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:47 smithi083 ceph-mon[35242]: pgmap v459: 129 pgs: 129 active+clean; 57 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 226 op/s 2024-09-16T00:20:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:47 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:20:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:47 smithi060 ceph-mon[35185]: pgmap v459: 129 pgs: 129 active+clean; 57 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 226 op/s 2024-09-16T00:20:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:47 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:20:48.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:47 smithi033 ceph-mon[31243]: pgmap v459: 129 pgs: 129 active+clean; 57 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 226 op/s 2024-09-16T00:20:48.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:47 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:20:48.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 107070 5.13 MB/sec warmup 116 sec latency 19.598 ms 2024-09-16T00:20:49.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 110252 5.24 MB/sec warmup 117 sec latency 15.473 ms 2024-09-16T00:20:49.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:49 smithi083 ceph-mon[35242]: pgmap v460: 129 pgs: 129 active+clean; 59 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 245 op/s 2024-09-16T00:20:49.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:49 smithi060 ceph-mon[35185]: pgmap v460: 129 pgs: 129 active+clean; 59 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 245 op/s 2024-09-16T00:20:50.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:49 smithi033 ceph-mon[31243]: pgmap v460: 129 pgs: 129 active+clean; 59 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 245 op/s 2024-09-16T00:20:50.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 113361 5.32 MB/sec warmup 118 sec latency 17.859 ms 2024-09-16T00:20:51.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 115951 5.38 MB/sec warmup 119 sec latency 13.673 ms 2024-09-16T00:20:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:51 smithi083 ceph-mon[35242]: pgmap v461: 129 pgs: 129 active+clean; 60 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-09-16T00:20:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:51 smithi060 ceph-mon[35185]: pgmap v461: 129 pgs: 129 active+clean; 60 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-09-16T00:20:52.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:51 smithi033 ceph-mon[31243]: pgmap v461: 129 pgs: 129 active+clean; 60 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-09-16T00:20:52.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:52 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:20:52.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:52 smithi083 ceph-mon[35242]: pgmap v462: 129 pgs: 129 active+clean; 65 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.8 MiB/s wr, 233 op/s 2024-09-16T00:20:52.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:52 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:20:52.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:52 smithi060 ceph-mon[35185]: pgmap v462: 129 pgs: 129 active+clean; 65 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.8 MiB/s wr, 233 op/s 2024-09-16T00:20:53.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:52 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:20:53.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:52 smithi033 ceph-mon[31243]: pgmap v462: 129 pgs: 129 active+clean; 65 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.8 MiB/s wr, 233 op/s 2024-09-16T00:20:53.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 121505 14.68 MB/sec execute 1 sec latency 13.994 ms 2024-09-16T00:20:54.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 124797 16.60 MB/sec execute 2 sec latency 13.861 ms 2024-09-16T00:20:55.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:54 smithi033 ceph-mon[31243]: pgmap v463: 129 pgs: 129 active+clean; 66 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 226 op/s 2024-09-16T00:20:55.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:54 smithi083 ceph-mon[35242]: pgmap v463: 129 pgs: 129 active+clean; 66 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 226 op/s 2024-09-16T00:20:55.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:54 smithi060 ceph-mon[35185]: pgmap v463: 129 pgs: 129 active+clean; 66 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 226 op/s 2024-09-16T00:20:55.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 128130 17.43 MB/sec execute 3 sec latency 13.186 ms 2024-09-16T00:20:56.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 131488 17.78 MB/sec execute 4 sec latency 12.264 ms 2024-09-16T00:20:57.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:56 smithi033 ceph-mon[31243]: pgmap v464: 129 pgs: 129 active+clean; 71 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.8 MiB/s wr, 244 op/s 2024-09-16T00:20:57.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:56 smithi083 ceph-mon[35242]: pgmap v464: 129 pgs: 129 active+clean; 71 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.8 MiB/s wr, 244 op/s 2024-09-16T00:20:57.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:56 smithi060 ceph-mon[35185]: pgmap v464: 129 pgs: 129 active+clean; 71 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.8 MiB/s wr, 244 op/s 2024-09-16T00:20:57.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 134625 17.17 MB/sec execute 5 sec latency 11.945 ms 2024-09-16T00:20:58.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 137216 16.26 MB/sec execute 6 sec latency 18.033 ms 2024-09-16T00:20:59.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:20:58 smithi033 ceph-mon[31243]: pgmap v465: 129 pgs: 129 active+clean; 74 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 261 op/s 2024-09-16T00:20:59.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:20:58 smithi083 ceph-mon[35242]: pgmap v465: 129 pgs: 129 active+clean; 74 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 261 op/s 2024-09-16T00:20:59.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:20:58 smithi060 ceph-mon[35185]: pgmap v465: 129 pgs: 129 active+clean; 74 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 261 op/s 2024-09-16T00:20:59.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 139793 16.39 MB/sec execute 7 sec latency 18.776 ms 2024-09-16T00:21:00.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 142700 16.15 MB/sec execute 8 sec latency 34.204 ms 2024-09-16T00:21:01.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:00 smithi033 ceph-mon[31243]: pgmap v466: 129 pgs: 129 active+clean; 75 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 239 op/s 2024-09-16T00:21:01.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:00 smithi083 ceph-mon[35242]: pgmap v466: 129 pgs: 129 active+clean; 75 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 239 op/s 2024-09-16T00:21:01.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:00 smithi060 ceph-mon[35185]: pgmap v466: 129 pgs: 129 active+clean; 75 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 239 op/s 2024-09-16T00:21:01.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 145683 16.40 MB/sec execute 9 sec latency 103.369 ms 2024-09-16T00:21:02.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 148633 16.11 MB/sec execute 10 sec latency 12.382 ms 2024-09-16T00:21:03.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:02 smithi033 ceph-mon[31243]: pgmap v467: 129 pgs: 129 active+clean; 78 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.9 MiB/s wr, 264 op/s 2024-09-16T00:21:03.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:02 smithi083 ceph-mon[35242]: pgmap v467: 129 pgs: 129 active+clean; 78 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.9 MiB/s wr, 264 op/s 2024-09-16T00:21:03.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:02 smithi060 ceph-mon[35185]: pgmap v467: 129 pgs: 129 active+clean; 78 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.9 MiB/s wr, 264 op/s 2024-09-16T00:21:03.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 151425 15.83 MB/sec execute 11 sec latency 22.593 ms 2024-09-16T00:21:04.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 153836 15.69 MB/sec execute 12 sec latency 18.097 ms 2024-09-16T00:21:05.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:04 smithi033 ceph-mon[31243]: pgmap v468: 129 pgs: 129 active+clean; 79 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 218 op/s 2024-09-16T00:21:05.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:04 smithi083 ceph-mon[35242]: pgmap v468: 129 pgs: 129 active+clean; 79 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 218 op/s 2024-09-16T00:21:05.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:04 smithi060 ceph-mon[35185]: pgmap v468: 129 pgs: 129 active+clean; 79 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 218 op/s 2024-09-16T00:21:05.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 156827 15.82 MB/sec execute 13 sec latency 13.329 ms 2024-09-16T00:21:06.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 159646 15.91 MB/sec execute 14 sec latency 102.179 ms 2024-09-16T00:21:07.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:06 smithi033 ceph-mon[31243]: pgmap v469: 129 pgs: 129 active+clean; 84 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.8 MiB/s wr, 304 op/s 2024-09-16T00:21:07.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:06 smithi083 ceph-mon[35242]: pgmap v469: 129 pgs: 129 active+clean; 84 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.8 MiB/s wr, 304 op/s 2024-09-16T00:21:07.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:06 smithi060 ceph-mon[35185]: pgmap v469: 129 pgs: 129 active+clean; 84 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.8 MiB/s wr, 304 op/s 2024-09-16T00:21:07.652 INFO:tasks.workunit.client.0.smithi033.stdout: 1 161888 15.35 MB/sec execute 15 sec latency 34.912 ms 2024-09-16T00:21:08.653 INFO:tasks.workunit.client.0.smithi033.stdout: 1 164127 15.27 MB/sec execute 16 sec latency 21.997 ms 2024-09-16T00:21:09.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:08 smithi033 ceph-mon[31243]: pgmap v470: 129 pgs: 129 active+clean; 84 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 246 op/s 2024-09-16T00:21:09.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:08 smithi083 ceph-mon[35242]: pgmap v470: 129 pgs: 129 active+clean; 84 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 246 op/s 2024-09-16T00:21:09.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:08 smithi060 ceph-mon[35185]: pgmap v470: 129 pgs: 129 active+clean; 84 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 246 op/s 2024-09-16T00:21:09.653 INFO:tasks.workunit.client.0.smithi033.stdout: 1 167303 15.45 MB/sec execute 17 sec latency 10.380 ms 2024-09-16T00:21:10.653 INFO:tasks.workunit.client.0.smithi033.stdout: 1 170540 15.64 MB/sec execute 18 sec latency 12.376 ms 2024-09-16T00:21:11.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:10 smithi033 ceph-mon[31243]: pgmap v471: 129 pgs: 129 active+clean; 84 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 223 op/s 2024-09-16T00:21:11.096 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:21:11.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:10 smithi083 ceph-mon[35242]: pgmap v471: 129 pgs: 129 active+clean; 84 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 223 op/s 2024-09-16T00:21:11.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:10 smithi060 ceph-mon[35185]: pgmap v471: 129 pgs: 129 active+clean; 84 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 223 op/s 2024-09-16T00:21:11.387 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:11.653 INFO:tasks.workunit.client.0.smithi033.stdout: 1 172663 15.20 MB/sec execute 19 sec latency 59.348 ms 2024-09-16T00:21:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.926+0000 7fa8682d2640 1 -- 172.21.15.33:0/56766153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa86010c170 msgr2=0x7fa86010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.926+0000 7fa8682d2640 1 --2- 172.21.15.33:0/56766153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa86010c170 0x7fa86010e610 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa858009ca0 tx=0x7fa85802a950 comp rx=0 tx=0).stop 2024-09-16T00:21:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.926+0000 7fa8682d2640 1 -- 172.21.15.33:0/56766153 shutdown_connections 2024-09-16T00:21:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.926+0000 7fa8682d2640 1 --2- 172.21.15.33:0/56766153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa86010c170 0x7fa86010e610 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.926+0000 7fa8682d2640 1 --2- 172.21.15.33:0/56766153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa860103660 0x7fa86010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.926+0000 7fa8682d2640 1 --2- 172.21.15.33:0/56766153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa860102d40 0x7fa860103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.926+0000 7fa8682d2640 1 -- 172.21.15.33:0/56766153 >> 172.21.15.33:0/56766153 conn(0x7fa8600fc7a0 msgr2=0x7fa8600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.926+0000 7fa8682d2640 1 -- 172.21.15.33:0/56766153 shutdown_connections 2024-09-16T00:21:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.926+0000 7fa8682d2640 1 -- 172.21.15.33:0/56766153 wait complete. 2024-09-16T00:21:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.927+0000 7fa8682d2640 1 Processor -- start 2024-09-16T00:21:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.927+0000 7fa8682d2640 1 -- start start 2024-09-16T00:21:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.927+0000 7fa8682d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa860102d40 0x7fa860198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.927+0000 7fa8682d2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa860103660 0x7fa860198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.927+0000 7fa8682d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa86010c170 0x7fa86019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.927+0000 7fa8682d2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa860111f70 con 0x7fa860102d40 2024-09-16T00:21:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.927+0000 7fa8682d2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa860111df0 con 0x7fa86010c170 2024-09-16T00:21:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.927+0000 7fa8682d2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa8601120f0 con 0x7fa860103660 2024-09-16T00:21:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa86606e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa860102d40 0x7fa860198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa86586d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa860103660 0x7fa860198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa86606e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa860102d40 0x7fa860198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52178/0 (socket says 172.21.15.33:52178) 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa86606e640 1 -- 172.21.15.33:0/3969602769 learned_addr learned my addr 172.21.15.33:0/3969602769 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa86686f640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa86010c170 0x7fa86019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa86586d640 1 -- 172.21.15.33:0/3969602769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa86010c170 msgr2=0x7fa86019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa86586d640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa86010c170 0x7fa86019cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa86586d640 1 -- 172.21.15.33:0/3969602769 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa860102d40 msgr2=0x7fa860198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa86586d640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa860102d40 0x7fa860198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa86586d640 1 -- 172.21.15.33:0/3969602769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86019d3c0 con 0x7fa860103660 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa86686f640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa86010c170 0x7fa86019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa86586d640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa860103660 0x7fa860198950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa850005fd0 tx=0x7fa85000db60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa8577fe640 1 -- 172.21.15.33:0/3969602769 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa850004230 con 0x7fa860103660 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa8577fe640 1 -- 172.21.15.33:0/3969602769 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8500043d0 con 0x7fa860103660 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa8682d2640 1 -- 172.21.15.33:0/3969602769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa86019d6b0 con 0x7fa860103660 2024-09-16T00:21:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa8577fe640 1 -- 172.21.15.33:0/3969602769 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa850010460 con 0x7fa860103660 2024-09-16T00:21:11.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.928+0000 7fa8682d2640 1 -- 172.21.15.33:0/3969602769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8601a4f90 con 0x7fa860103660 2024-09-16T00:21:11.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.931+0000 7fa8577fe640 1 -- 172.21.15.33:0/3969602769 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa850009820 con 0x7fa860103660 2024-09-16T00:21:11.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.931+0000 7fa86a6ab740 1 Processor -- start 2024-09-16T00:21:11.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.931+0000 7fa86a6ab740 1 -- start start 2024-09-16T00:21:11.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86a6ab740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a54b380 0x56146a54b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86a6ab740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 0x56146a55a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86a6ab740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a55aa90 0x56146a55ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86a6ab740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56146a4b12e0 con 0x56146a55aa90 2024-09-16T00:21:11.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86a6ab740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56146a458ce0 con 0x56146a54b380 2024-09-16T00:21:11.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86a6ab740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56146a4ad2a0 con 0x56146a54bca0 2024-09-16T00:21:11.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86606e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 0x56146a55a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86586d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a55aa90 0x56146a55ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86606e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 0x56146a55a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59988/0 (socket says 172.21.15.33:59988) 2024-09-16T00:21:11.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86606e640 1 -- 172.21.15.33:0/1681927553 learned_addr learned my addr 172.21.15.33:0/1681927553 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:11.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86686f640 1 --2- 172.21.15.33:0/1681927553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a54b380 0x56146a54b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa8577fe640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa84c0a3240 0x7fa84c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86686f640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa84c0a3240 0x7fa84c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86606e640 1 -- 172.21.15.33:0/1681927553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a54b380 msgr2=0x56146a54b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:11.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86606e640 1 --2- 172.21.15.33:0/1681927553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a54b380 0x56146a54b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86606e640 1 -- 172.21.15.33:0/1681927553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a55aa90 msgr2=0x56146a55ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:11.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86606e640 1 --2- 172.21.15.33:0/1681927553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a55aa90 0x56146a55ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa8577fe640 1 -- 172.21.15.33:0/3969602769 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa8500cf440 con 0x7fa860103660 2024-09-16T00:21:11.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.932+0000 7fa86606e640 1 -- 172.21.15.33:0/1681927553 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56146a43d440 con 0x56146a54bca0 2024-09-16T00:21:11.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.933+0000 7fa86606e640 1 --2- 172.21.15.33:0/1681927553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 0x56146a55a550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa85c009e10 tx=0x7fa85c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7529c57a3e6ff8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:11.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.933+0000 7fa86686f640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa84c0a3240 0x7fa84c0a5700 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa858005fc0 tx=0x7fa85802a950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:11.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.933+0000 7fa854dea640 1 -- 172.21.15.33:0/1681927553 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c01e070 con 0x56146a54bca0 2024-09-16T00:21:11.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.933+0000 7fa854dea640 1 -- 172.21.15.33:0/1681927553 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa85c019b10 con 0x56146a54bca0 2024-09-16T00:21:11.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.933+0000 7fa854dea640 1 -- 172.21.15.33:0/1681927553 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c00fc20 con 0x56146a54bca0 2024-09-16T00:21:11.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 -- 172.21.15.33:0/1681927553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 msgr2=0x56146a55a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:11.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/1681927553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 0x56146a55a550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa85c009e10 tx=0x7fa85c011810 comp rx=0 tx=0).stop 2024-09-16T00:21:11.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 -- 172.21.15.33:0/1681927553 shutdown_connections 2024-09-16T00:21:11.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/1681927553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a55aa90 0x56146a55ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/1681927553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 0x56146a55a550 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/1681927553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a54b380 0x56146a54b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 -- 172.21.15.33:0/1681927553 >> 172.21.15.33:0/1681927553 conn(0x56146a547a30 msgr2=0x56146a547e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:11.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 -- 172.21.15.33:0/1681927553 shutdown_connections 2024-09-16T00:21:11.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 -- 172.21.15.33:0/1681927553 wait complete. 2024-09-16T00:21:11.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 -- start start 2024-09-16T00:21:11.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 Processor -- start 2024-09-16T00:21:11.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 0x56146a55a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a54bca0 0x56146a5576e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a55aa90 0x56146a557c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.934+0000 7fa86a6ab740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56146a55d660 con 0x56146a55aa90 2024-09-16T00:21:11.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86a6ab740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56146a46c050 con 0x56146a54bca0 2024-09-16T00:21:11.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86a6ab740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56146a55d7e0 con 0x56146a54b380 2024-09-16T00:21:11.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86a6ab740 10 client.0 _ll_register_callbacks cb 0x56146a4ac160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:11.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86686f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a55aa90 0x56146a557c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86606e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 0x56146a55a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86686f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a55aa90 0x56146a557c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52196/0 (socket says 172.21.15.33:52196) 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86686f640 1 -- 172.21.15.33:0/3742240236 learned_addr learned my addr 172.21.15.33:0/3742240236 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86586d640 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a54bca0 0x56146a5576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86686f640 1 -- 172.21.15.33:0/3742240236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 msgr2=0x56146a55a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86686f640 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 0x56146a55a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86686f640 1 -- 172.21.15.33:0/3742240236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a54bca0 msgr2=0x56146a5576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86686f640 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a54bca0 0x56146a5576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86686f640 1 -- 172.21.15.33:0/3742240236 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56146a2223c0 con 0x56146a55aa90 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86606e640 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 0x56146a55a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86586d640 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a54bca0 0x56146a5576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86686f640 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a55aa90 0x56146a557c80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa858035000 tx=0x7fa8580095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90380d145f2429f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa849ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:21:11.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa849ffb640 1 -- 172.21.15.33:0/3742240236 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858034040 con 0x56146a55aa90 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa86a6ab740 1 -- 172.21.15.33:0/3742240236 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa85c019b10 con 0x56146a55aa90 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.935+0000 7fa849ffb640 1 -- 172.21.15.33:0/3742240236 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa858004450 con 0x56146a55aa90 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.936+0000 7fa849ffb640 1 -- 172.21.15.33:0/3742240236 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa858004650 con 0x56146a55aa90 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.936+0000 7fa86a6ab740 10 client.16056 fetch_fsmap learned FSMap version 29 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.936+0000 7fa86a6ab740 1 -- 172.21.15.33:0/3742240236 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56146a5690b0 con 0x56146a55aa90 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.936+0000 7fa849ffb640 1 -- 172.21.15.33:0/3742240236 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa85c019b10 con 0x56146a55aa90 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.936+0000 7fa849ffb640 1 -- 172.21.15.33:0/3742240236 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7fa858049040 con 0x56146a55aa90 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.936+0000 7fa86a6ab740 10 client.16056 fetch_fsmap finished waiting for FSMap version 29 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.936+0000 7fa86a6ab740 10 client.16056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.936+0000 7fa86a6ab740 20 client.16056 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.936+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56146a56a5c0 0x56146a56ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.936+0000 7fa86a6ab740 4 client.16056 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:21:11.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.936+0000 7fa86a6ab740 1 -- 172.21.15.33:0/3742240236 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56146a4be850 con 0x56146a56a5c0 2024-09-16T00:21:11.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.936+0000 7fa86606e640 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56146a56a5c0 0x56146a56ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.937+0000 7fa86606e640 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56146a56a5c0 0x56146a56ca40 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6096f2a32a04dd3 server_cookie=ea8dff1ff69ce1fb in_seq=0 out_seq=0 2024-09-16T00:21:11.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.937+0000 7fa849ffb640 10 client.16056 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:11.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.937+0000 7fa849ffb640 1 -- 172.21.15.33:0/3742240236 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56146a4be850 con 0x56146a56a5c0 2024-09-16T00:21:11.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.937+0000 7fa849ffb640 10 client.16056 handle_command_reply: tid=0 2024-09-16T00:21:11.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.937+0000 7fa86a6ab740 1 client.16056 shutdown 2024-09-16T00:21:11.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.937+0000 7fa86a6ab740 1 -- 172.21.15.33:0/3742240236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a55aa90 msgr2=0x56146a557c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:11.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.937+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a55aa90 0x56146a557c80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa858035000 tx=0x7fa8580095c0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.938+0000 7fa86a6ab740 1 -- 172.21.15.33:0/3742240236 shutdown_connections 2024-09-16T00:21:11.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.938+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56146a56a5c0 0x56146a56ca40 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.938+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a55aa90 0x56146a557c80 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.938+0000 7fa849ffb640 0 client.16056 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:11.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.938+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a54bca0 0x56146a5576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.938+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/3742240236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 0x56146a55a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.938+0000 7fa86a6ab740 1 -- 172.21.15.33:0/3742240236 >> 172.21.15.33:0/3742240236 conn(0x56146a547a30 msgr2=0x56146a54d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:11.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.938+0000 7fa86a6ab740 1 -- 172.21.15.33:0/3742240236 shutdown_connections 2024-09-16T00:21:11.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.938+0000 7fa86a6ab740 1 -- 172.21.15.33:0/3742240236 wait complete. 2024-09-16T00:21:11.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.938+0000 7fa86a6ab740 20 client.16056 trim_cache size 0 max 16384 2024-09-16T00:21:11.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86a6ab740 1 Processor -- start 2024-09-16T00:21:11.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86a6ab740 1 -- start start 2024-09-16T00:21:11.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86a6ab740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 0x56146a566900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86a6ab740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54bca0 0x56146a552c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86a6ab740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a55aa90 0x56146a572310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86a6ab740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8601af7f0 con 0x56146a54bca0 2024-09-16T00:21:11.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86a6ab740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa8601ad0c0 con 0x56146a55aa90 2024-09-16T00:21:11.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86a6ab740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa8601aaf10 con 0x56146a54b380 2024-09-16T00:21:11.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86606e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 0x56146a566900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86606e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 0x56146a566900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59996/0 (socket says 172.21.15.33:59996) 2024-09-16T00:21:11.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86586d640 1 --2- 172.21.15.33:0/2119955519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54bca0 0x56146a552c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86606e640 1 -- 172.21.15.33:0/2119955519 learned_addr learned my addr 172.21.15.33:0/2119955519 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:11.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86686f640 1 --2- 172.21.15.33:0/2119955519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a55aa90 0x56146a572310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86606e640 1 -- 172.21.15.33:0/2119955519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a55aa90 msgr2=0x56146a572310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86606e640 1 --2- 172.21.15.33:0/2119955519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a55aa90 0x56146a572310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86606e640 1 -- 172.21.15.33:0/2119955519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54bca0 msgr2=0x56146a552c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86606e640 1 --2- 172.21.15.33:0/2119955519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54bca0 0x56146a552c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.957+0000 7fa86606e640 1 -- 172.21.15.33:0/2119955519 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56146a54dea0 con 0x56146a54b380 2024-09-16T00:21:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.958+0000 7fa86586d640 1 --2- 172.21.15.33:0/2119955519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54bca0 0x56146a552c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.958+0000 7fa86686f640 1 --2- 172.21.15.33:0/2119955519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a55aa90 0x56146a572310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.958+0000 7fa86606e640 1 --2- 172.21.15.33:0/2119955519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 0x56146a566900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa85c009d60 tx=0x7fa85c011e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c07d6b4dd237032 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.958+0000 7fa84bfff640 1 -- 172.21.15.33:0/2119955519 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c01e040 con 0x56146a54b380 2024-09-16T00:21:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.958+0000 7fa84bfff640 1 -- 172.21.15.33:0/2119955519 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa85c01f040 con 0x56146a54b380 2024-09-16T00:21:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.958+0000 7fa84bfff640 1 -- 172.21.15.33:0/2119955519 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c004680 con 0x56146a54b380 2024-09-16T00:21:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.959+0000 7fa86a6ab740 1 -- 172.21.15.33:0/2119955519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 msgr2=0x56146a566900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.959+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/2119955519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 0x56146a566900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa85c009d60 tx=0x7fa85c011e40 comp rx=0 tx=0).stop 2024-09-16T00:21:11.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.959+0000 7fa86a6ab740 1 -- 172.21.15.33:0/2119955519 shutdown_connections 2024-09-16T00:21:11.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.959+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/2119955519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a55aa90 0x56146a572310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.959+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/2119955519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54bca0 0x56146a552c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.959+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/2119955519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54b380 0x56146a566900 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.959+0000 7fa86a6ab740 1 -- 172.21.15.33:0/2119955519 >> 172.21.15.33:0/2119955519 conn(0x56146a547a30 msgr2=0x56146a5524f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:11.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.959+0000 7fa86a6ab740 1 -- 172.21.15.33:0/2119955519 shutdown_connections 2024-09-16T00:21:11.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.959+0000 7fa86a6ab740 1 -- 172.21.15.33:0/2119955519 wait complete. 2024-09-16T00:21:11.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.959+0000 7fa86a6ab740 1 -- start start 2024-09-16T00:21:11.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.959+0000 7fa86a6ab740 1 Processor -- start 2024-09-16T00:21:11.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86a6ab740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54b380 0x56146a562da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86a6ab740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 0x56146a5632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86a6ab740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a55aa90 0x56146a55e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86a6ab740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8601ab580 con 0x56146a54b380 2024-09-16T00:21:11.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86a6ab740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa8601ac9f0 con 0x56146a55aa90 2024-09-16T00:21:11.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86a6ab740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa8601abc50 con 0x56146a54bca0 2024-09-16T00:21:11.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86a6ab740 10 client.0 _ll_register_callbacks cb 0x7fa858009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86606e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54b380 0x56146a562da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86586d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 0x56146a5632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86606e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54b380 0x56146a562da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52216/0 (socket says 172.21.15.33:52216) 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86606e640 1 -- 172.21.15.33:0/491294755 learned_addr learned my addr 172.21.15.33:0/491294755 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86686f640 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a55aa90 0x56146a55e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86606e640 1 -- 172.21.15.33:0/491294755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 msgr2=0x56146a5632e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86606e640 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 0x56146a5632e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86606e640 1 -- 172.21.15.33:0/491294755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a55aa90 msgr2=0x56146a55e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86606e640 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a55aa90 0x56146a55e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86606e640 1 -- 172.21.15.33:0/491294755 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56146a55e9b0 con 0x56146a54b380 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86586d640 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 0x56146a5632e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86686f640 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a55aa90 0x56146a55e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:11.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa86606e640 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54b380 0x56146a562da0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa85c011d10 tx=0x7fa85c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d05fa81aaeaf8d63 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.960+0000 7fa849ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa849ffb640 1 -- 172.21.15.33:0/491294755 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c03f060 con 0x56146a54b380 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa849ffb640 1 -- 172.21.15.33:0/491294755 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa85c0054b0 con 0x56146a54b380 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa86a6ab740 1 -- 172.21.15.33:0/491294755 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa85c01f040 con 0x56146a54b380 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa849ffb640 1 -- 172.21.15.33:0/491294755 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa85c01e040 con 0x56146a54b380 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa86a6ab740 10 client.16062 fetch_fsmap learned FSMap version 29 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa86a6ab740 1 -- 172.21.15.33:0/491294755 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56146a5d4ed0 con 0x56146a54b380 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa849ffb640 1 -- 172.21.15.33:0/491294755 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa85c01f040 con 0x56146a54b380 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa849ffb640 1 -- 172.21.15.33:0/491294755 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7fa85c039030 con 0x56146a54b380 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa86a6ab740 10 client.16062 fetch_fsmap finished waiting for FSMap version 29 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa86a6ab740 10 client.16062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa86a6ab740 20 client.16062 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:11.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56146a56a5c0 0x56146a5d7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.961+0000 7fa86a6ab740 4 client.16062 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.962+0000 7fa86a6ab740 1 -- 172.21.15.33:0/491294755 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56146a4b8d60 con 0x56146a56a5c0 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.962+0000 7fa86586d640 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56146a56a5c0 0x56146a5d7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.962+0000 7fa86586d640 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56146a56a5c0 0x56146a5d7400 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e71b8ae7e62156a server_cookie=d31a6be08ada796c in_seq=0 out_seq=0 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.962+0000 7fa849ffb640 10 client.16062 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.962+0000 7fa849ffb640 1 -- 172.21.15.33:0/491294755 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56146a4b8d60 con 0x56146a56a5c0 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.962+0000 7fa849ffb640 10 client.16062 handle_command_reply: tid=0 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.962+0000 7fa86a6ab740 1 client.16062 shutdown 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.962+0000 7fa86a6ab740 1 -- 172.21.15.33:0/491294755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54b380 msgr2=0x56146a562da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.963+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54b380 0x56146a562da0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa85c011d10 tx=0x7fa85c011810 comp rx=0 tx=0).stop 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.963+0000 7fa86a6ab740 1 -- 172.21.15.33:0/491294755 shutdown_connections 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.963+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56146a56a5c0 0x56146a5d7400 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.963+0000 7fa849ffb640 0 client.16062 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:11.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.963+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56146a55aa90 0x56146a55e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.963+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56146a54bca0 0x56146a5632e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.963+0000 7fa86a6ab740 1 --2- 172.21.15.33:0/491294755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56146a54b380 0x56146a562da0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.963+0000 7fa86a6ab740 1 -- 172.21.15.33:0/491294755 >> 172.21.15.33:0/491294755 conn(0x56146a547a30 msgr2=0x56146a552d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:11.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.963+0000 7fa86a6ab740 1 -- 172.21.15.33:0/491294755 shutdown_connections 2024-09-16T00:21:11.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.963+0000 7fa86a6ab740 1 -- 172.21.15.33:0/491294755 wait complete. 2024-09-16T00:21:11.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.963+0000 7fa86a6ab740 20 client.16062 trim_cache size 0 max 16384 2024-09-16T00:21:11.969 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:21:11.969 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:21:11.970 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:21:11.970 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:21:11.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.964+0000 7fa8682d2640 1 -- 172.21.15.33:0/3969602769 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa84c0a3240 msgr2=0x7fa84c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:11.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.964+0000 7fa8682d2640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa84c0a3240 0x7fa84c0a5700 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa858005fc0 tx=0x7fa85802a950 comp rx=0 tx=0).stop 2024-09-16T00:21:11.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.964+0000 7fa8682d2640 1 -- 172.21.15.33:0/3969602769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa860103660 msgr2=0x7fa860198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:11.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.964+0000 7fa8682d2640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa860103660 0x7fa860198950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa850005fd0 tx=0x7fa85000db60 comp rx=0 tx=0).stop 2024-09-16T00:21:11.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.964+0000 7fa8682d2640 1 -- 172.21.15.33:0/3969602769 shutdown_connections 2024-09-16T00:21:11.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.964+0000 7fa8682d2640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa84c0a3240 0x7fa84c0a5700 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.964+0000 7fa8682d2640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa86010c170 0x7fa86019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.964+0000 7fa8682d2640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa860103660 0x7fa860198950 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.964+0000 7fa8682d2640 1 --2- 172.21.15.33:0/3969602769 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa860102d40 0x7fa860198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:11.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.964+0000 7fa8682d2640 1 -- 172.21.15.33:0/3969602769 >> 172.21.15.33:0/3969602769 conn(0x7fa8600fc7a0 msgr2=0x7fa8600fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:11.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.964+0000 7fa8682d2640 1 -- 172.21.15.33:0/3969602769 shutdown_connections 2024-09-16T00:21:11.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:11.964+0000 7fa8682d2640 1 -- 172.21.15.33:0/3969602769 wait complete. 2024-09-16T00:21:12.653 INFO:tasks.workunit.client.0.smithi033.stdout: 1 174735 15.08 MB/sec execute 20 sec latency 38.812 ms 2024-09-16T00:21:12.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:12 smithi033 ceph-mon[31243]: pgmap v472: 129 pgs: 129 active+clean; 86 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 266 op/s 2024-09-16T00:21:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:12 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:21:12.880 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:21:12.881 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:21:13.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:12 smithi083 ceph-mon[35242]: pgmap v472: 129 pgs: 129 active+clean; 86 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 266 op/s 2024-09-16T00:21:13.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:12 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:21:13.173 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:13.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:12 smithi060 ceph-mon[35185]: pgmap v472: 129 pgs: 129 active+clean; 86 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 266 op/s 2024-09-16T00:21:13.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:12 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:21:13.653 INFO:tasks.workunit.client.0.smithi033.stdout: 1 177727 15.24 MB/sec execute 21 sec latency 32.851 ms 2024-09-16T00:21:13.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.835+0000 7fac8d975640 1 -- 172.21.15.33:0/1867060824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac88102dd0 msgr2=0x7fac881031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:13.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.835+0000 7fac8d975640 1 --2- 172.21.15.33:0/1867060824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac88102dd0 0x7fac881031b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fac78009930 tx=0x7fac7802a420 comp rx=0 tx=0).stop 2024-09-16T00:21:13.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.835+0000 7fac8d975640 1 -- 172.21.15.33:0/1867060824 shutdown_connections 2024-09-16T00:21:13.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.835+0000 7fac8d975640 1 --2- 172.21.15.33:0/1867060824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac88108300 0x7fac8810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:13.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.835+0000 7fac8d975640 1 --2- 172.21.15.33:0/1867060824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac88103780 0x7fac88107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:13.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.835+0000 7fac8d975640 1 --2- 172.21.15.33:0/1867060824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac88102dd0 0x7fac881031b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:13.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.835+0000 7fac8d975640 1 -- 172.21.15.33:0/1867060824 >> 172.21.15.33:0/1867060824 conn(0x7fac880fc7d0 msgr2=0x7fac880febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:13.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.835+0000 7fac8d975640 1 -- 172.21.15.33:0/1867060824 shutdown_connections 2024-09-16T00:21:13.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.835+0000 7fac8d975640 1 -- 172.21.15.33:0/1867060824 wait complete. 2024-09-16T00:21:13.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac8d975640 1 Processor -- start 2024-09-16T00:21:13.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac8d975640 1 -- start start 2024-09-16T00:21:13.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac8d975640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac88102dd0 0x7fac881985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:13.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac8d975640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac88103780 0x7fac88198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac8d975640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac88108300 0x7fac8819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac8d975640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac881120b0 con 0x7fac88102dd0 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac8d975640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fac88111f30 con 0x7fac88108300 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac8d975640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fac88112230 con 0x7fac88103780 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac86ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac88102dd0 0x7fac881985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac86ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac88102dd0 0x7fac881985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52234/0 (socket says 172.21.15.33:52234) 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac86ffd640 1 -- 172.21.15.33:0/1976095136 learned_addr learned my addr 172.21.15.33:0/1976095136 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac867fc640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac88103780 0x7fac88198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac877fe640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac88108300 0x7fac8819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac86ffd640 1 -- 172.21.15.33:0/1976095136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac88103780 msgr2=0x7fac88198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac86ffd640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac88103780 0x7fac88198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac86ffd640 1 -- 172.21.15.33:0/1976095136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac88108300 msgr2=0x7fac8819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:13.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.836+0000 7fac86ffd640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac88108300 0x7fac8819cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:13.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.837+0000 7fac86ffd640 1 -- 172.21.15.33:0/1976095136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8819d5a0 con 0x7fac88102dd0 2024-09-16T00:21:13.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.837+0000 7fac86ffd640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac88102dd0 0x7fac881985f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fac78009230 tx=0x7fac78004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:13.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.837+0000 7fac8c973640 1 -- 172.21.15.33:0/1976095136 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7803f030 con 0x7fac88102dd0 2024-09-16T00:21:13.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.837+0000 7fac8c973640 1 -- 172.21.15.33:0/1976095136 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac78003ad0 con 0x7fac88102dd0 2024-09-16T00:21:13.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.837+0000 7fac8d975640 1 -- 172.21.15.33:0/1976095136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac8819d830 con 0x7fac88102dd0 2024-09-16T00:21:13.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.837+0000 7fac8c973640 1 -- 172.21.15.33:0/1976095136 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac78035070 con 0x7fac88102dd0 2024-09-16T00:21:13.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.837+0000 7fac8d975640 1 -- 172.21.15.33:0/1976095136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac881a5170 con 0x7fac88102dd0 2024-09-16T00:21:13.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.838+0000 7fac8d975640 1 -- 172.21.15.33:0/1976095136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac881031b0 con 0x7fac88102dd0 2024-09-16T00:21:13.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.840+0000 7fac8c973640 1 -- 172.21.15.33:0/1976095136 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fac78042030 con 0x7fac88102dd0 2024-09-16T00:21:13.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.840+0000 7fac8c973640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac700a32c0 0x7fac700a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:13.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.840+0000 7fac867fc640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac700a32c0 0x7fac700a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:13.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.840+0000 7fac8c973640 1 -- 172.21.15.33:0/1976095136 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fac780f0370 con 0x7fac88102dd0 2024-09-16T00:21:13.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.841+0000 7fac867fc640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac700a32c0 0x7fac700a5780 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fac880feef0 tx=0x7fac7c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:13.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:13.845+0000 7fac8c973640 1 -- 172.21.15.33:0/1976095136 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fac780bcc10 con 0x7fac88102dd0 2024-09-16T00:21:14.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:21:14.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:21:14.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:21:14.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.080+0000 7fac8d975640 1 -- 172.21.15.33:0/1976095136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac8810a290 con 0x7fac88102dd0 2024-09-16T00:21:14.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.081+0000 7fac8c973640 1 -- 172.21.15.33:0/1976095136 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11904 (secure 0 0 0) 0x7fac780bc510 con 0x7fac88102dd0 2024-09-16T00:21:14.081 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:21:14.082 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":29,"btime":"2024-09-16T00:20:51:610478+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:20:51.462209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:21:14.082 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 29 2024-09-16T00:21:14.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.083+0000 7fac8d975640 1 -- 172.21.15.33:0/1976095136 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac700a32c0 msgr2=0x7fac700a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:14.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.083+0000 7fac8d975640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac700a32c0 0x7fac700a5780 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fac880feef0 tx=0x7fac7c005f90 comp rx=0 tx=0).stop 2024-09-16T00:21:14.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.083+0000 7fac8d975640 1 -- 172.21.15.33:0/1976095136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac88102dd0 msgr2=0x7fac881985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:14.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.083+0000 7fac8d975640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac88102dd0 0x7fac881985f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fac78009230 tx=0x7fac78004270 comp rx=0 tx=0).stop 2024-09-16T00:21:14.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.083+0000 7fac8d975640 1 -- 172.21.15.33:0/1976095136 shutdown_connections 2024-09-16T00:21:14.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.083+0000 7fac8d975640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac700a32c0 0x7fac700a5780 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:14.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.083+0000 7fac8d975640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac88108300 0x7fac8819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:14.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.083+0000 7fac8d975640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac88103780 0x7fac88198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:14.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.083+0000 7fac8d975640 1 --2- 172.21.15.33:0/1976095136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac88102dd0 0x7fac881985f0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:14.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.083+0000 7fac8d975640 1 -- 172.21.15.33:0/1976095136 >> 172.21.15.33:0/1976095136 conn(0x7fac880fc7d0 msgr2=0x7fac88105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:14.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.083+0000 7fac8d975640 1 -- 172.21.15.33:0/1976095136 shutdown_connections 2024-09-16T00:21:14.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:14.083+0000 7fac8d975640 1 -- 172.21.15.33:0/1976095136 wait complete. 2024-09-16T00:21:14.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:21:14.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:21:14.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:21:14.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:21:14.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:21:14.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:21:14.653 INFO:tasks.workunit.client.0.smithi033.stdout: 1 180910 15.34 MB/sec execute 22 sec latency 11.950 ms 2024-09-16T00:21:15.129 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:21:15.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:15 smithi033 ceph-mon[31243]: pgmap v473: 129 pgs: 129 active+clean; 90 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 317 op/s 2024-09-16T00:21:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:15 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1976095136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:21:15.408 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:15.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:15 smithi083 ceph-mon[35242]: pgmap v473: 129 pgs: 129 active+clean; 90 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 317 op/s 2024-09-16T00:21:15.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:15 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1976095136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:21:15.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:15 smithi060 ceph-mon[35185]: pgmap v473: 129 pgs: 129 active+clean; 90 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 317 op/s 2024-09-16T00:21:15.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:15 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1976095136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:21:15.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 183471 15.09 MB/sec execute 23 sec latency 15.899 ms 2024-09-16T00:21:16.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.032+0000 7f5be6c96640 1 -- 172.21.15.33:0/740488050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be0106a80 msgr2=0x7f5be0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:16.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.032+0000 7f5be6c96640 1 --2- 172.21.15.33:0/740488050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be0106a80 0x7f5be0106e60 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009930 tx=0x7f5bd402a420 comp rx=0 tx=0).stop 2024-09-16T00:21:16.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.032+0000 7f5be6c96640 1 -- 172.21.15.33:0/740488050 shutdown_connections 2024-09-16T00:21:16.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.032+0000 7f5be6c96640 1 --2- 172.21.15.33:0/740488050 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5be0101420 0x7f5be010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.032+0000 7f5be6c96640 1 --2- 172.21.15.33:0/740488050 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5be0100a80 0x7f5be0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.032+0000 7f5be6c96640 1 --2- 172.21.15.33:0/740488050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be0106a80 0x7f5be0106e60 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.032+0000 7f5be6c96640 1 -- 172.21.15.33:0/740488050 >> 172.21.15.33:0/740488050 conn(0x7f5be00fc7a0 msgr2=0x7f5be00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:16.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.032+0000 7f5be6c96640 1 -- 172.21.15.33:0/740488050 shutdown_connections 2024-09-16T00:21:16.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.032+0000 7f5be6c96640 1 -- 172.21.15.33:0/740488050 wait complete. 2024-09-16T00:21:16.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be6c96640 1 Processor -- start 2024-09-16T00:21:16.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be6c96640 1 -- start start 2024-09-16T00:21:16.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be6c96640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5be0100a80 0x7f5be00fcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be6c96640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5be0101420 0x7f5be00fd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be6c96640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be0106a80 0x7f5be00fd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be6c96640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5be0110ed0 con 0x7f5be0106a80 2024-09-16T00:21:16.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be6c96640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5be0110d50 con 0x7f5be0101420 2024-09-16T00:21:16.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be6c96640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5be0111050 con 0x7f5be0100a80 2024-09-16T00:21:16.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be5233640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be0106a80 0x7f5be00fd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be4a32640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5be0100a80 0x7f5be00fcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5bdffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5be0101420 0x7f5be00fd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be4a32640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5be0100a80 0x7f5be00fcdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60058/0 (socket says 172.21.15.33:60058) 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be4a32640 1 -- 172.21.15.33:0/1438147817 learned_addr learned my addr 172.21.15.33:0/1438147817 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.033+0000 7f5be5233640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be0106a80 0x7f5be00fd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52256/0 (socket says 172.21.15.33:52256) 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.034+0000 7f5be5233640 1 -- 172.21.15.33:0/1438147817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5be0100a80 msgr2=0x7f5be00fcdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.034+0000 7f5be5233640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5be0100a80 0x7f5be00fcdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.034+0000 7f5be5233640 1 -- 172.21.15.33:0/1438147817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5be0101420 msgr2=0x7f5be00fd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.034+0000 7f5be5233640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5be0101420 0x7f5be00fd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.034+0000 7f5be5233640 1 -- 172.21.15.33:0/1438147817 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be01a6930 con 0x7f5be0106a80 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.034+0000 7f5be4a32640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5be0100a80 0x7f5be00fcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.034+0000 7f5be5233640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be0106a80 0x7f5be00fd840 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800cab0 tx=0x7f5bd800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.034+0000 7f5bddffb640 1 -- 172.21.15.33:0/1438147817 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd800ae20 con 0x7f5be0106a80 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.034+0000 7f5bddffb640 1 -- 172.21.15.33:0/1438147817 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd800f930 con 0x7f5be0106a80 2024-09-16T00:21:16.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.034+0000 7f5be6c96640 1 -- 172.21.15.33:0/1438147817 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be01a6c20 con 0x7f5be0106a80 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.034+0000 7f5bddffb640 1 -- 172.21.15.33:0/1438147817 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8018b30 con 0x7f5be0106a80 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.034+0000 7f5be6c96640 1 -- 172.21.15.33:0/1438147817 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be01a7160 con 0x7f5be0106a80 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.037+0000 7f5bddffb640 1 -- 172.21.15.33:0/1438147817 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5bd800fae0 con 0x7f5be0106a80 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.037+0000 7f5be906f740 1 Processor -- start 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.037+0000 7f5be906f740 1 -- start start 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.037+0000 7f5be906f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 0x55e04d961090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.037+0000 7f5be906f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d9615d0 0x55e04d961a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.037+0000 7f5be906f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d96c370 0x55e04d96e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.037+0000 7f5be906f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e04d8c72f0 con 0x55e04d96c370 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.037+0000 7f5be906f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e04d86ecf0 con 0x55e04d962b40 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.037+0000 7f5be906f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e04d8c32b0 con 0x55e04d9615d0 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5bdffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d96c370 0x55e04d96e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5bdffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 0x55e04d961090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5be4a32640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d9615d0 0x55e04d961a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5bdffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 0x55e04d961090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51048/0 (socket says 172.21.15.33:51048) 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5bdffff640 1 -- 172.21.15.33:0/3530783130 learned_addr learned my addr 172.21.15.33:0/3530783130 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5bdffff640 1 -- 172.21.15.33:0/3530783130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d9615d0 msgr2=0x55e04d961a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5bdffff640 1 --2- 172.21.15.33:0/3530783130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d9615d0 0x55e04d961a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5bdffff640 1 -- 172.21.15.33:0/3530783130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d96c370 msgr2=0x55e04d96e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5bdffff640 1 --2- 172.21.15.33:0/3530783130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d96c370 0x55e04d96e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5bdffff640 1 -- 172.21.15.33:0/3530783130 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e04d8533a0 con 0x55e04d962b40 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5bddffb640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5bd00a32c0 0x7f5bd00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5be4a32640 1 --2- 172.21.15.33:0/3530783130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d9615d0 0x55e04d961a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5be4a32640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5bd00a32c0 0x7f5bd00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5bddffb640 1 -- 172.21.15.33:0/1438147817 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5bd80d0330 con 0x7f5be0106a80 2024-09-16T00:21:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.038+0000 7f5bdffff640 1 --2- 172.21.15.33:0/3530783130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 0x55e04d961090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55e04d962070 tx=0x7f5bcc00b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85b489d108b81e1b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.039+0000 7f5be4a32640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5bd00a32c0 0x7f5bd00a5780 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4004650 tx=0x7f5bd4031510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.039+0000 7f5bcaffd640 1 -- 172.21.15.33:0/3530783130 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc00f760 con 0x55e04d962b40 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.039+0000 7f5bcaffd640 1 -- 172.21.15.33:0/3530783130 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bcc00fde0 con 0x55e04d962b40 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.039+0000 7f5bcaffd640 1 -- 172.21.15.33:0/3530783130 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc01fd20 con 0x55e04d962b40 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.040+0000 7f5be906f740 1 -- 172.21.15.33:0/3530783130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 msgr2=0x55e04d961090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.040+0000 7f5be906f740 1 --2- 172.21.15.33:0/3530783130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 0x55e04d961090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55e04d962070 tx=0x7f5bcc00b390 comp rx=0 tx=0).stop 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.043+0000 7f5be906f740 1 -- 172.21.15.33:0/3530783130 shutdown_connections 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.043+0000 7f5be906f740 1 --2- 172.21.15.33:0/3530783130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d96c370 0x55e04d96e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.043+0000 7f5be906f740 1 --2- 172.21.15.33:0/3530783130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d9615d0 0x55e04d961a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.043+0000 7f5be906f740 1 --2- 172.21.15.33:0/3530783130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 0x55e04d961090 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.043+0000 7f5be906f740 1 -- 172.21.15.33:0/3530783130 >> 172.21.15.33:0/3530783130 conn(0x55e04d92a310 msgr2=0x55e04d95dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.048+0000 7f5be906f740 1 -- 172.21.15.33:0/3530783130 shutdown_connections 2024-09-16T00:21:16.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.048+0000 7f5be906f740 1 -- 172.21.15.33:0/3530783130 wait complete. 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.048+0000 7f5be906f740 1 -- start start 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.048+0000 7f5be906f740 1 Processor -- start 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.049+0000 7f5be906f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 0x55e04d96bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.049+0000 7f5be906f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d962b40 0x55e04d969050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.049+0000 7f5be4a32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 0x55e04d96bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.049+0000 7f5be4a32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 0x55e04d96bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52268/0 (socket says 172.21.15.33:52268) 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.049+0000 7f5be4a32640 1 -- 172.21.15.33:0/2668181784 learned_addr learned my addr 172.21.15.33:0/2668181784 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.049+0000 7f5be906f740 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d96c370 0x55e04d9697e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.049+0000 7f5be906f740 1 -- 172.21.15.33:0/2668181784 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e04d96f070 con 0x55e04d9615d0 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.049+0000 7f5be906f740 1 -- 172.21.15.33:0/2668181784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e04d882060 con 0x55e04d96c370 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.049+0000 7f5be906f740 1 -- 172.21.15.33:0/2668181784 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e04d96f1f0 con 0x55e04d962b40 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.050+0000 7f5be906f740 10 client.0 _ll_register_callbacks cb 0x55e04d5d58c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:16.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.050+0000 7f5be5233640 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d96c370 0x55e04d9697e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.051+0000 7f5bdffff640 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d962b40 0x55e04d969050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.051+0000 7f5be5233640 1 -- 172.21.15.33:0/2668181784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d962b40 msgr2=0x55e04d969050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.051+0000 7f5be5233640 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d962b40 0x55e04d969050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.051+0000 7f5be5233640 1 -- 172.21.15.33:0/2668181784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 msgr2=0x55e04d96bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.051+0000 7f5be5233640 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 0x55e04d96bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.051+0000 7f5be5233640 1 -- 172.21.15.33:0/2668181784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e04d6383c0 con 0x55e04d96c370 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.052+0000 7f5be5233640 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d96c370 0x55e04d9697e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5bd801f000 tx=0x7f5bd809be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7def2068c4b6e2c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.052+0000 7f5bbffff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.052+0000 7f5bbffff640 1 -- 172.21.15.33:0/2668181784 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd80d0170 con 0x55e04d96c370 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.052+0000 7f5bbffff640 1 -- 172.21.15.33:0/2668181784 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5bd80a4cb0 con 0x55e04d96c370 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.054+0000 7f5be906f740 1 -- 172.21.15.33:0/2668181784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bcc00fde0 con 0x55e04d96c370 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.055+0000 7f5bbffff640 1 -- 172.21.15.33:0/2668181784 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5bd809c4d0 con 0x55e04d96c370 2024-09-16T00:21:16.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.057+0000 7f5be906f740 10 client.25649 fetch_fsmap learned FSMap version 29 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.057+0000 7f5be906f740 1 -- 172.21.15.33:0/2668181784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e04d971be0 con 0x55e04d96c370 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.059+0000 7f5bbffff640 1 -- 172.21.15.33:0/2668181784 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd807d020 con 0x55e04d96c370 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.059+0000 7f5bbffff640 1 -- 172.21.15.33:0/2668181784 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7f5bd8004b40 con 0x55e04d96c370 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.060+0000 7f5be906f740 10 client.25649 fetch_fsmap finished waiting for FSMap version 29 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.060+0000 7f5be906f740 10 client.25649 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.060+0000 7f5be906f740 20 client.25649 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.060+0000 7f5be906f740 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e04d973520 0x55e04d975930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.060+0000 7f5be906f740 4 client.25649 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.060+0000 7f5be906f740 1 -- 172.21.15.33:0/2668181784 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5bcc00fde0 con 0x55e04d973520 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.061+0000 7f5be4a32640 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e04d973520 0x55e04d975930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.063+0000 7f5be4a32640 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e04d973520 0x55e04d975930 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=73dc9dab6276152a server_cookie=66b5f9a94f5d5773 in_seq=0 out_seq=0 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.064+0000 7f5bbffff640 10 client.25649 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.065+0000 7f5bbffff640 1 -- 172.21.15.33:0/2668181784 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f5bcc00fde0 con 0x55e04d973520 2024-09-16T00:21:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.065+0000 7f5bbffff640 10 client.25649 handle_command_reply: tid=0 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.065+0000 7f5be906f740 1 client.25649 shutdown 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.069+0000 7f5be906f740 1 -- 172.21.15.33:0/2668181784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d96c370 msgr2=0x55e04d9697e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.069+0000 7f5be906f740 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d96c370 0x55e04d9697e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5bd801f000 tx=0x7f5bd809be30 comp rx=0 tx=0).stop 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.073+0000 7f5be906f740 1 -- 172.21.15.33:0/2668181784 shutdown_connections 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.073+0000 7f5be906f740 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e04d973520 0x55e04d975930 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.073+0000 7f5be906f740 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d96c370 0x55e04d9697e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.073+0000 7f5be906f740 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d962b40 0x55e04d969050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.073+0000 7f5be906f740 1 --2- 172.21.15.33:0/2668181784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 0x55e04d96bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.073+0000 7f5be906f740 1 -- 172.21.15.33:0/2668181784 >> 172.21.15.33:0/2668181784 conn(0x55e04d92a310 msgr2=0x55e04d963960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.073+0000 7f5bbffff640 0 client.25649 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.076+0000 7f5be906f740 1 -- 172.21.15.33:0/2668181784 shutdown_connections 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.076+0000 7f5be906f740 1 -- 172.21.15.33:0/2668181784 wait complete. 2024-09-16T00:21:16.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.076+0000 7f5be906f740 20 client.25649 trim_cache size 0 max 16384 2024-09-16T00:21:16.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.102+0000 7f5be906f740 1 Processor -- start 2024-09-16T00:21:16.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.102+0000 7f5be906f740 1 -- start start 2024-09-16T00:21:16.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.102+0000 7f5be906f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 0x55e04d969020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.102+0000 7f5be906f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 0x55e04d97b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.102+0000 7f5be906f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d96c370 0x55e04d97b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be906f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5be01ab400 con 0x55e04d9615d0 2024-09-16T00:21:16.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be906f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5be01a8cd0 con 0x55e04d962b40 2024-09-16T00:21:16.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be906f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5be0073bb0 con 0x55e04d96c370 2024-09-16T00:21:16.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be4a32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 0x55e04d969020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be4a32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 0x55e04d969020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52272/0 (socket says 172.21.15.33:52272) 2024-09-16T00:21:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be4a32640 1 -- 172.21.15.33:0/2807668629 learned_addr learned my addr 172.21.15.33:0/2807668629 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be5233640 1 --2- 172.21.15.33:0/2807668629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d96c370 0x55e04d97b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be4a32640 1 -- 172.21.15.33:0/2807668629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d96c370 msgr2=0x55e04d97b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5bdffff640 1 --2- 172.21.15.33:0/2807668629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 0x55e04d97b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be4a32640 1 --2- 172.21.15.33:0/2807668629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d96c370 0x55e04d97b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be4a32640 1 -- 172.21.15.33:0/2807668629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 msgr2=0x55e04d97b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be4a32640 1 --2- 172.21.15.33:0/2807668629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 0x55e04d97b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be4a32640 1 -- 172.21.15.33:0/2807668629 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e04d95ded0 con 0x55e04d9615d0 2024-09-16T00:21:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5bdffff640 1 --2- 172.21.15.33:0/2807668629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 0x55e04d97b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:21:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5be4a32640 1 --2- 172.21.15.33:0/2807668629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 0x55e04d969020 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009900 tx=0x7f5bd4040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f613d2c6c69a9113 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5bca7fc640 1 -- 172.21.15.33:0/2807668629 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4035040 con 0x55e04d9615d0 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5bca7fc640 1 -- 172.21.15.33:0/2807668629 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd407a030 con 0x55e04d9615d0 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.103+0000 7f5bca7fc640 1 -- 172.21.15.33:0/2807668629 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4003850 con 0x55e04d9615d0 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.104+0000 7f5be906f740 1 -- 172.21.15.33:0/2807668629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 msgr2=0x55e04d969020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.104+0000 7f5be906f740 1 --2- 172.21.15.33:0/2807668629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 0x55e04d969020 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009900 tx=0x7f5bd4040000 comp rx=0 tx=0).stop 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.104+0000 7f5be906f740 1 -- 172.21.15.33:0/2807668629 shutdown_connections 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.104+0000 7f5be906f740 1 --2- 172.21.15.33:0/2807668629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d96c370 0x55e04d97b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.104+0000 7f5be906f740 1 --2- 172.21.15.33:0/2807668629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d962b40 0x55e04d97b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.104+0000 7f5be906f740 1 --2- 172.21.15.33:0/2807668629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d9615d0 0x55e04d969020 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.104+0000 7f5be906f740 1 -- 172.21.15.33:0/2807668629 >> 172.21.15.33:0/2807668629 conn(0x55e04d92a310 msgr2=0x55e04d968910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.104+0000 7f5be906f740 1 -- 172.21.15.33:0/2807668629 shutdown_connections 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.104+0000 7f5be906f740 1 -- 172.21.15.33:0/2807668629 wait complete. 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5be906f740 1 -- start start 2024-09-16T00:21:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5be906f740 1 Processor -- start 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5be906f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d9615d0 0x55e04d978510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5be906f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d962b40 0x55e04d978a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5be906f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d96c370 0x55e04d9edf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5be906f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5be0074220 con 0x55e04d962b40 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5be906f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5be01a8600 con 0x55e04d96c370 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5be906f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5be01a7860 con 0x55e04d9615d0 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5bdffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d962b40 0x55e04d978a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5bdffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d962b40 0x55e04d978a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52280/0 (socket says 172.21.15.33:52280) 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5bdffff640 1 -- 172.21.15.33:0/3743482673 learned_addr learned my addr 172.21.15.33:0/3743482673 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5be906f740 10 client.0 _ll_register_callbacks cb 0x7f5bd8004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.105+0000 7f5be4a32640 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d9615d0 0x55e04d978510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5be5233640 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d96c370 0x55e04d9edf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5bdffff640 1 -- 172.21.15.33:0/3743482673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d9615d0 msgr2=0x55e04d978510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5bdffff640 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d9615d0 0x55e04d978510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5bdffff640 1 -- 172.21.15.33:0/3743482673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d96c370 msgr2=0x55e04d9edf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5bdffff640 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d96c370 0x55e04d9edf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5bdffff640 1 -- 172.21.15.33:0/3743482673 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e04d9ee440 con 0x55e04d962b40 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5be4a32640 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d9615d0 0x55e04d978510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5bdffff640 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d962b40 0x55e04d978a50 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00b6b0 tx=0x7f5bcc00bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3616e8d35bc4426b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5bbffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5bbffff640 1 -- 172.21.15.33:0/3743482673 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc01fa70 con 0x55e04d962b40 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5bbffff640 1 -- 172.21.15.33:0/3743482673 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5bcc01fc10 con 0x55e04d962b40 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5be906f740 1 -- 172.21.15.33:0/3743482673 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bd407a030 con 0x55e04d962b40 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5bbffff640 1 -- 172.21.15.33:0/3743482673 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5bcc01fe10 con 0x55e04d962b40 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5be906f740 10 client.16086 fetch_fsmap learned FSMap version 29 2024-09-16T00:21:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.106+0000 7f5be906f740 1 -- 172.21.15.33:0/3743482673 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e04d9ef560 con 0x55e04d962b40 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.107+0000 7f5bbffff640 1 -- 172.21.15.33:0/3743482673 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd407a030 con 0x55e04d962b40 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.107+0000 7f5bbffff640 1 -- 172.21.15.33:0/3743482673 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7f5bcc027070 con 0x55e04d962b40 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.107+0000 7f5be906f740 10 client.16086 fetch_fsmap finished waiting for FSMap version 29 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.107+0000 7f5be906f740 10 client.16086 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.107+0000 7f5be906f740 20 client.16086 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.107+0000 7f5be906f740 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e04d9efad0 0x55e04d9f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.107+0000 7f5be906f740 4 client.16086 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.107+0000 7f5be906f740 1 -- 172.21.15.33:0/3743482673 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e04d8d4860 con 0x55e04d9efad0 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.107+0000 7f5be4a32640 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e04d9efad0 0x55e04d9f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.108+0000 7f5be4a32640 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e04d9efad0 0x55e04d9f1ef0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=65ab586611c57004 server_cookie=eb343a7737078ecb in_seq=0 out_seq=0 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.108+0000 7f5bbffff640 10 client.16086 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5bbffff640 1 -- 172.21.15.33:0/3743482673 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e04d8d4860 con 0x55e04d9efad0 2024-09-16T00:21:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5bbffff640 10 client.16086 handle_command_reply: tid=0 2024-09-16T00:21:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5be906f740 1 client.16086 shutdown 2024-09-16T00:21:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5be906f740 1 -- 172.21.15.33:0/3743482673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d962b40 msgr2=0x55e04d978a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5be906f740 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d962b40 0x55e04d978a50 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00b6b0 tx=0x7f5bcc00bfc0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5be906f740 1 -- 172.21.15.33:0/3743482673 shutdown_connections 2024-09-16T00:21:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5be906f740 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e04d9efad0 0x55e04d9f1ef0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5bbffff640 0 client.16086 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:21:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5be906f740 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e04d96c370 0x55e04d9edf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5be906f740 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e04d962b40 0x55e04d978a50 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5be906f740 1 --2- 172.21.15.33:0/3743482673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e04d9615d0 0x55e04d978510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.164 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:21:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5be906f740 1 -- 172.21.15.33:0/3743482673 >> 172.21.15.33:0/3743482673 conn(0x55e04d92a310 msgr2=0x55e04d95c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5be906f740 1 -- 172.21.15.33:0/3743482673 shutdown_connections 2024-09-16T00:21:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5be906f740 1 -- 172.21.15.33:0/3743482673 wait complete. 2024-09-16T00:21:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.109+0000 7f5be906f740 20 client.16086 trim_cache size 0 max 16384 2024-09-16T00:21:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.110+0000 7f5be6c96640 1 -- 172.21.15.33:0/1438147817 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5bd00a32c0 msgr2=0x7f5bd00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.110+0000 7f5be6c96640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5bd00a32c0 0x7f5bd00a5780 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4004650 tx=0x7f5bd4031510 comp rx=0 tx=0).stop 2024-09-16T00:21:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.110+0000 7f5be6c96640 1 -- 172.21.15.33:0/1438147817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be0106a80 msgr2=0x7f5be00fd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:16.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.110+0000 7f5be6c96640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be0106a80 0x7f5be00fd840 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800cab0 tx=0x7f5bd800cf70 comp rx=0 tx=0).stop 2024-09-16T00:21:16.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.110+0000 7f5be6c96640 1 -- 172.21.15.33:0/1438147817 shutdown_connections 2024-09-16T00:21:16.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.110+0000 7f5be6c96640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5bd00a32c0 0x7f5bd00a5780 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.110+0000 7f5be6c96640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5be0106a80 0x7f5be00fd840 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.110+0000 7f5be6c96640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5be0101420 0x7f5be00fd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.110+0000 7f5be6c96640 1 --2- 172.21.15.33:0/1438147817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5be0100a80 0x7f5be00fcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:16.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.110+0000 7f5be6c96640 1 -- 172.21.15.33:0/1438147817 >> 172.21.15.33:0/1438147817 conn(0x7f5be00fc7a0 msgr2=0x7f5be006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:16.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.110+0000 7f5be6c96640 1 -- 172.21.15.33:0/1438147817 shutdown_connections 2024-09-16T00:21:16.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:16.110+0000 7f5be6c96640 1 -- 172.21.15.33:0/1438147817 wait complete. 2024-09-16T00:21:16.653 INFO:tasks.workunit.client.0.smithi033.stdout: 1 185137 14.93 MB/sec execute 24 sec latency 91.395 ms 2024-09-16T00:21:16.948 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:21:17.230 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:17.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:17 smithi033 ceph-mon[31243]: pgmap v474: 129 pgs: 129 active+clean; 90 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 313 op/s 2024-09-16T00:21:17.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:17 smithi083 ceph-mon[35242]: pgmap v474: 129 pgs: 129 active+clean; 90 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 313 op/s 2024-09-16T00:21:17.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:17 smithi060 ceph-mon[35185]: pgmap v474: 129 pgs: 129 active+clean; 90 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 313 op/s 2024-09-16T00:21:17.653 INFO:tasks.workunit.client.0.smithi033.stdout: 1 188281 15.08 MB/sec execute 25 sec latency 43.061 ms 2024-09-16T00:21:17.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.728+0000 7f2196dbe640 1 -- 172.21.15.33:0/3232658187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2190102780 msgr2=0x7f21901067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:17.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.728+0000 7f2196dbe640 1 --2- 172.21.15.33:0/3232658187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2190102780 0x7f21901067d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f217c009900 tx=0x7f217c02a410 comp rx=0 tx=0).stop 2024-09-16T00:21:17.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.728+0000 7f2196dbe640 1 -- 172.21.15.33:0/3232658187 shutdown_connections 2024-09-16T00:21:17.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.728+0000 7f2196dbe640 1 --2- 172.21.15.33:0/3232658187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106f00 0x7f219010ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.728+0000 7f2196dbe640 1 --2- 172.21.15.33:0/3232658187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2190102780 0x7f21901067d0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.728+0000 7f2196dbe640 1 --2- 172.21.15.33:0/3232658187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2190101dd0 0x7f21901021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.728+0000 7f2196dbe640 1 -- 172.21.15.33:0/3232658187 >> 172.21.15.33:0/3232658187 conn(0x7f21900fd8b0 msgr2=0x7f21900ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.728+0000 7f2196dbe640 1 -- 172.21.15.33:0/3232658187 shutdown_connections 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.728+0000 7f2196dbe640 1 -- 172.21.15.33:0/3232658187 wait complete. 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f2196dbe640 1 Processor -- start 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f2196dbe640 1 -- start start 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f2196dbe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2190101dd0 0x7f21900744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f2196dbe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190102780 0x7f2190074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f2196dbe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2190106f00 0x7f2190074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f2196dbe640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2190111f60 con 0x7f2190101dd0 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f2196dbe640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2190111de0 con 0x7f2190102780 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f2196dbe640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21901120e0 con 0x7f2190106f00 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f2194b5a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2190101dd0 0x7f21900744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f2194b5a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2190101dd0 0x7f21900744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52290/0 (socket says 172.21.15.33:52290) 2024-09-16T00:21:17.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f2194b5a640 1 -- 172.21.15.33:0/1385792079 learned_addr learned my addr 172.21.15.33:0/1385792079 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:17.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f218ffff640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190102780 0x7f2190074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.729+0000 7f219535b640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2190106f00 0x7f2190074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.730+0000 7f2194b5a640 1 -- 172.21.15.33:0/1385792079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2190106f00 msgr2=0x7f2190074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:17.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.730+0000 7f2194b5a640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2190106f00 0x7f2190074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.730+0000 7f2194b5a640 1 -- 172.21.15.33:0/1385792079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190102780 msgr2=0x7f2190074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:17.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.730+0000 7f2194b5a640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190102780 0x7f2190074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.730+0000 7f2194b5a640 1 -- 172.21.15.33:0/1385792079 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21901a6900 con 0x7f2190101dd0 2024-09-16T00:21:17.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.730+0000 7f218ffff640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190102780 0x7f2190074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:17.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.730+0000 7f2194b5a640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2190101dd0 0x7f21900744f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f218400e9f0 tx=0x7f218400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:17.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.730+0000 7f218dffb640 1 -- 172.21.15.33:0/1385792079 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218400cd80 con 0x7f2190101dd0 2024-09-16T00:21:17.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.730+0000 7f2196dbe640 1 -- 172.21.15.33:0/1385792079 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21901a6bf0 con 0x7f2190101dd0 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.730+0000 7f218dffb640 1 -- 172.21.15.33:0/1385792079 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2184005500 con 0x7f2190101dd0 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.730+0000 7f218dffb640 1 -- 172.21.15.33:0/1385792079 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2184010690 con 0x7f2190101dd0 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.730+0000 7f2196dbe640 1 -- 172.21.15.33:0/1385792079 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21901a7100 con 0x7f2190101dd0 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.732+0000 7f218dffb640 1 -- 172.21.15.33:0/1385792079 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f21840046c0 con 0x7f2190101dd0 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.733+0000 7f2199197740 1 Processor -- start 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.733+0000 7f2199197740 1 -- start start 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.733+0000 7f218dffb640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21800a32c0 0x7f21800a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.733+0000 7f2199197740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f269fb40 0x5585f269e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.733+0000 7f219535b640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21800a32c0 0x7f21800a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.733+0000 7f2199197740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269e5d0 0x5585f269ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.733+0000 7f218dffb640 1 -- 172.21.15.33:0/1385792079 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f21840d0170 con 0x7f2190101dd0 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.733+0000 7f2199197740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f26a9370 0x5585f26ab760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.733+0000 7f2199197740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5585f26042f0 con 0x5585f269e5d0 2024-09-16T00:21:17.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.733+0000 7f2199197740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5585f25abcf0 con 0x5585f26a9370 2024-09-16T00:21:17.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.733+0000 7f2199197740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5585f26002b0 con 0x5585f269fb40 2024-09-16T00:21:17.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f2194b5a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f269fb40 0x5585f269e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f2194b5a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f269fb40 0x5585f269e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60108/0 (socket says 172.21.15.33:60108) 2024-09-16T00:21:17.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f218ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269e5d0 0x5585f269ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f2194b5a640 1 -- 172.21.15.33:0/1378247009 learned_addr learned my addr 172.21.15.33:0/1378247009 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:17.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f2194b5a640 1 -- 172.21.15.33:0/1378247009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f26a9370 msgr2=0x5585f26ab760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:21:17.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f2194b5a640 1 --2- 172.21.15.33:0/1378247009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f26a9370 0x5585f26ab760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f2194b5a640 1 -- 172.21.15.33:0/1378247009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269e5d0 msgr2=0x5585f269ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:17.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f2194b5a640 1 --2- 172.21.15.33:0/1378247009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269e5d0 0x5585f269ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f2194b5a640 1 -- 172.21.15.33:0/1378247009 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585f25903a0 con 0x5585f269fb40 2024-09-16T00:21:17.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f219535b640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21800a32c0 0x7f21800a5780 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2188005fd0 tx=0x7f2188009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f2194b5a640 1 --2- 172.21.15.33:0/1378247009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f269fb40 0x5585f269e090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f218409c240 tx=0x7f218409bc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3786bdfe00541f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f217affd640 1 -- 172.21.15.33:0/1378247009 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21840b5070 con 0x5585f269fb40 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f217affd640 1 -- 172.21.15.33:0/1378247009 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21840a0660 con 0x5585f269fb40 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.734+0000 7f217affd640 1 -- 172.21.15.33:0/1378247009 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21840a4cf0 con 0x5585f269fb40 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.735+0000 7f2199197740 1 -- 172.21.15.33:0/1378247009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f269fb40 msgr2=0x5585f269e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.735+0000 7f2199197740 1 --2- 172.21.15.33:0/1378247009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f269fb40 0x5585f269e090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f218409c240 tx=0x7f218409bc10 comp rx=0 tx=0).stop 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.735+0000 7f2199197740 1 -- 172.21.15.33:0/1378247009 shutdown_connections 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.735+0000 7f2199197740 1 --2- 172.21.15.33:0/1378247009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f26a9370 0x5585f26ab760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.735+0000 7f2199197740 1 --2- 172.21.15.33:0/1378247009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269e5d0 0x5585f269ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.735+0000 7f2199197740 1 --2- 172.21.15.33:0/1378247009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f269fb40 0x5585f269e090 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.735+0000 7f2199197740 1 -- 172.21.15.33:0/1378247009 >> 172.21.15.33:0/1378247009 conn(0x5585f2667310 msgr2=0x5585f269ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.735+0000 7f2199197740 1 -- 172.21.15.33:0/1378247009 shutdown_connections 2024-09-16T00:21:17.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.735+0000 7f2199197740 1 -- 172.21.15.33:0/1378247009 wait complete. 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f2199197740 1 -- start start 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f2199197740 1 Processor -- start 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f2199197740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 0x5585f26a8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f2199197740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269fb40 0x5585f26a6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f2199197740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f26a9370 0x5585f26a6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f2199197740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5585f26ac070 con 0x5585f269fb40 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f2199197740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5585f25bf060 con 0x5585f269e5d0 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f2199197740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5585f26ac1f0 con 0x5585f26a9370 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f2199197740 10 client.0 _ll_register_callbacks cb 0x5585f23128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f2194b5a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 0x5585f26a8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f219535b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f26a9370 0x5585f26a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f218ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269fb40 0x5585f26a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f219535b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f26a9370 0x5585f26a6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60116/0 (socket says 172.21.15.33:60116) 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f2194b5a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 0x5585f26a8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51106/0 (socket says 172.21.15.33:51106) 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f218ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269fb40 0x5585f26a6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52314/0 (socket says 172.21.15.33:52314) 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.736+0000 7f219535b640 1 -- 172.21.15.33:0/70802312 learned_addr learned my addr 172.21.15.33:0/70802312 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f219535b640 1 -- 172.21.15.33:0/70802312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 msgr2=0x5585f26a8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f219535b640 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 0x5585f26a8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f219535b640 1 -- 172.21.15.33:0/70802312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269fb40 msgr2=0x5585f26a6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f219535b640 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269fb40 0x5585f26a6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f219535b640 1 -- 172.21.15.33:0/70802312 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585f23753c0 con 0x5585f26a9370 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f2194b5a640 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 0x5585f26a8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f218ffff640 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269fb40 0x5585f26a6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f219535b640 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f26a9370 0x5585f26a6640 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2188004290 tx=0x7f2188004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5c450215fc3ca5b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f216ffff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:21:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f216ffff640 1 -- 172.21.15.33:0/70802312 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2188022070 con 0x5585f26a9370 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f216ffff640 1 -- 172.21.15.33:0/70802312 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f218800fb50 con 0x5585f26a9370 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f2199197740 1 -- 172.21.15.33:0/70802312 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21840a0660 con 0x5585f26a9370 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f216ffff640 1 -- 172.21.15.33:0/70802312 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f218802e050 con 0x5585f26a9370 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f2199197740 10 client.25582 fetch_fsmap learned FSMap version 29 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.737+0000 7f2199197740 1 -- 172.21.15.33:0/70802312 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5585f26b7bf0 con 0x5585f26a9370 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.738+0000 7f216ffff640 1 -- 172.21.15.33:0/70802312 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21840a0660 con 0x5585f26a9370 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.738+0000 7f216ffff640 1 -- 172.21.15.33:0/70802312 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7f218801a3e0 con 0x5585f26a9370 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.738+0000 7f2199197740 10 client.25582 fetch_fsmap finished waiting for FSMap version 29 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.738+0000 7f2199197740 10 client.25582 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.738+0000 7f2199197740 20 client.25582 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.738+0000 7f2199197740 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5585f26b9000 0x5585f26bb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.738+0000 7f2199197740 4 client.25582 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:21:17.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.738+0000 7f2199197740 1 -- 172.21.15.33:0/70802312 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585f2611860 con 0x5585f26b9000 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.738+0000 7f2194b5a640 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5585f26b9000 0x5585f26bb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.738+0000 7f216ffff640 10 client.25582 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.738+0000 7f2194b5a640 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5585f26b9000 0x5585f26bb440 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f141d6326da78bc server_cookie=7413aed9a02b96ae in_seq=0 out_seq=0 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f216ffff640 1 -- 172.21.15.33:0/70802312 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5585f2611860 con 0x5585f26b9000 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f216ffff640 10 client.25582 handle_command_reply: tid=0 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f2199197740 1 client.25582 shutdown 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f2199197740 1 -- 172.21.15.33:0/70802312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f26a9370 msgr2=0x5585f26a6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f2199197740 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f26a9370 0x5585f26a6640 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2188004290 tx=0x7f2188004ab0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f2199197740 1 -- 172.21.15.33:0/70802312 shutdown_connections 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f2199197740 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f26a9370 0x5585f26a6640 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f2199197740 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269fb40 0x5585f26a6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f2199197740 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5585f26b9000 0x5585f26bb440 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f2199197740 1 --2- 172.21.15.33:0/70802312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 0x5585f26a8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f216ffff640 0 client.25582 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:21:17.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f2199197740 1 -- 172.21.15.33:0/70802312 >> 172.21.15.33:0/70802312 conn(0x5585f2667310 msgr2=0x5585f26a0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:17.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.739+0000 7f2199197740 1 -- 172.21.15.33:0/70802312 shutdown_connections 2024-09-16T00:21:17.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.740+0000 7f2199197740 1 -- 172.21.15.33:0/70802312 wait complete. 2024-09-16T00:21:17.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.740+0000 7f2199197740 20 client.25582 trim_cache size 0 max 16384 2024-09-16T00:21:17.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.755+0000 7f2199197740 1 Processor -- start 2024-09-16T00:21:17.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.755+0000 7f2199197740 1 -- start start 2024-09-16T00:21:17.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.755+0000 7f2199197740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 0x5585f26a6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.755+0000 7f2199197740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f269fb40 0x5585f26b5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.755+0000 7f2199197740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f26a9370 0x5585f26c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.755+0000 7f2199197740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21901b3ad0 con 0x5585f26a9370 2024-09-16T00:21:17.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.755+0000 7f2199197740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f21901b13a0 con 0x5585f269e5d0 2024-09-16T00:21:17.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f2199197740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21901af1f0 con 0x5585f269fb40 2024-09-16T00:21:17.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f2194b5a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 0x5585f26a6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f2194b5a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 0x5585f26a6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51110/0 (socket says 172.21.15.33:51110) 2024-09-16T00:21:17.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f2194b5a640 1 -- 172.21.15.33:0/2132207864 learned_addr learned my addr 172.21.15.33:0/2132207864 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:17.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f219535b640 1 --2- 172.21.15.33:0/2132207864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f26a9370 0x5585f26c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f218ffff640 1 --2- 172.21.15.33:0/2132207864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f269fb40 0x5585f26b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f2194b5a640 1 -- 172.21.15.33:0/2132207864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f269fb40 msgr2=0x5585f26b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:17.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f2194b5a640 1 --2- 172.21.15.33:0/2132207864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f269fb40 0x5585f26b5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f2194b5a640 1 -- 172.21.15.33:0/2132207864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f26a9370 msgr2=0x5585f26c0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:17.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f2194b5a640 1 --2- 172.21.15.33:0/2132207864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f26a9370 0x5585f26c0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f2194b5a640 1 -- 172.21.15.33:0/2132207864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585f269b960 con 0x5585f269e5d0 2024-09-16T00:21:17.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f2194b5a640 1 --2- 172.21.15.33:0/2132207864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 0x5585f26a6000 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2184098710 tx=0x7f2184004b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be5fe5db1a5a946b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:17.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f217a7fc640 1 -- 172.21.15.33:0/2132207864 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21840b0030 con 0x5585f269e5d0 2024-09-16T00:21:17.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f217a7fc640 1 -- 172.21.15.33:0/2132207864 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21840b51e0 con 0x5585f269e5d0 2024-09-16T00:21:17.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.756+0000 7f217a7fc640 1 -- 172.21.15.33:0/2132207864 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21840abe80 con 0x5585f269e5d0 2024-09-16T00:21:17.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.757+0000 7f2199197740 1 -- 172.21.15.33:0/2132207864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 msgr2=0x5585f26a6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:17.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.757+0000 7f2199197740 1 --2- 172.21.15.33:0/2132207864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 0x5585f26a6000 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2184098710 tx=0x7f2184004b60 comp rx=0 tx=0).stop 2024-09-16T00:21:17.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.757+0000 7f2199197740 1 -- 172.21.15.33:0/2132207864 shutdown_connections 2024-09-16T00:21:17.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.757+0000 7f2199197740 1 --2- 172.21.15.33:0/2132207864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f26a9370 0x5585f26c0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.757+0000 7f2199197740 1 --2- 172.21.15.33:0/2132207864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f269fb40 0x5585f26b5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.757+0000 7f2199197740 1 --2- 172.21.15.33:0/2132207864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269e5d0 0x5585f26a6000 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.757+0000 7f2199197740 1 -- 172.21.15.33:0/2132207864 >> 172.21.15.33:0/2132207864 conn(0x5585f2667310 msgr2=0x5585f26a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.757+0000 7f2199197740 1 -- 172.21.15.33:0/2132207864 shutdown_connections 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.757+0000 7f2199197740 1 -- 172.21.15.33:0/2132207864 wait complete. 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.757+0000 7f2199197740 1 -- start start 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f2199197740 1 Processor -- start 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f2199197740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269e5d0 0x5585f26b1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f2199197740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269fb40 0x5585f26b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f2199197740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f26a9370 0x5585f26acd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f2199197740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21901af860 con 0x5585f269e5d0 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f2199197740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f21901b0cd0 con 0x5585f269fb40 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f2199197740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21901aff30 con 0x5585f26a9370 2024-09-16T00:21:17.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f2199197740 10 client.0 _ll_register_callbacks cb 0x7f2188003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:17.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f2194b5a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269e5d0 0x5585f26b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f2194b5a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269e5d0 0x5585f26b1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52334/0 (socket says 172.21.15.33:52334) 2024-09-16T00:21:17.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f2194b5a640 1 -- 172.21.15.33:0/3287521862 learned_addr learned my addr 172.21.15.33:0/3287521862 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:17.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f219535b640 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f26a9370 0x5585f26acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.758+0000 7f218ffff640 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269fb40 0x5585f26b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f2194b5a640 1 -- 172.21.15.33:0/3287521862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f26a9370 msgr2=0x5585f26acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:17.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f2194b5a640 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f26a9370 0x5585f26acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f2194b5a640 1 -- 172.21.15.33:0/3287521862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269fb40 msgr2=0x5585f26b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:17.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f2194b5a640 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269fb40 0x5585f26b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f2194b5a640 1 -- 172.21.15.33:0/3287521862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585f26a1120 con 0x5585f269e5d0 2024-09-16T00:21:17.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f218ffff640 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269fb40 0x5585f26b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:21:17.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f2194b5a640 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269e5d0 0x5585f26b1810 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2184009250 tx=0x7f218409c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7965841e89f59115 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:17.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f216ffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:21:17.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f216ffff640 1 -- 172.21.15.33:0/3287521862 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21840a6070 con 0x5585f269e5d0 2024-09-16T00:21:17.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f216ffff640 1 -- 172.21.15.33:0/3287521862 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2184098920 con 0x5585f269e5d0 2024-09-16T00:21:17.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f2199197740 1 -- 172.21.15.33:0/3287521862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21840b51e0 con 0x5585f269e5d0 2024-09-16T00:21:17.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f216ffff640 1 -- 172.21.15.33:0/3287521862 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f21840b0040 con 0x5585f269e5d0 2024-09-16T00:21:17.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f2199197740 10 client.16104 fetch_fsmap learned FSMap version 29 2024-09-16T00:21:17.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.759+0000 7f2199197740 1 -- 172.21.15.33:0/3287521862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5585f27237b0 con 0x5585f269e5d0 2024-09-16T00:21:17.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.760+0000 7f216ffff640 1 -- 172.21.15.33:0/3287521862 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21840b51e0 con 0x5585f269e5d0 2024-09-16T00:21:17.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.760+0000 7f216ffff640 1 -- 172.21.15.33:0/3287521862 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7f218409b700 con 0x5585f269e5d0 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.760+0000 7f2199197740 10 client.16104 fetch_fsmap finished waiting for FSMap version 29 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.760+0000 7f2199197740 10 client.16104 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.760+0000 7f2199197740 20 client.16104 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.760+0000 7f2199197740 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5585f27239d0 0x5585f2725dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.760+0000 7f2199197740 4 client.16104 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.760+0000 7f2199197740 1 -- 172.21.15.33:0/3287521862 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5585f260bd70 con 0x5585f27239d0 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.760+0000 7f218ffff640 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5585f27239d0 0x5585f2725dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.760+0000 7f216ffff640 10 client.16104 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.760+0000 7f218ffff640 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5585f27239d0 0x5585f2725dc0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4255d5db99406e97 server_cookie=a9f19a8b41ee88b0 in_seq=0 out_seq=0 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f216ffff640 1 -- 172.21.15.33:0/3287521862 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5585f260bd70 con 0x5585f27239d0 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f216ffff640 10 client.16104 handle_command_reply: tid=0 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f2199197740 1 client.16104 shutdown 2024-09-16T00:21:17.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f2199197740 1 -- 172.21.15.33:0/3287521862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269e5d0 msgr2=0x5585f26b1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:17.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f2199197740 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269e5d0 0x5585f26b1810 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2184009250 tx=0x7f218409c370 comp rx=0 tx=0).stop 2024-09-16T00:21:17.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f2199197740 1 -- 172.21.15.33:0/3287521862 shutdown_connections 2024-09-16T00:21:17.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f2199197740 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585f26a9370 0x5585f26acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f2199197740 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5585f27239d0 0x5585f2725dc0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f2199197740 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585f269fb40 0x5585f26b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f2199197740 1 --2- 172.21.15.33:0/3287521862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585f269e5d0 0x5585f26b1810 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f216ffff640 0 client.16104 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:21:17.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f2199197740 1 -- 172.21.15.33:0/3287521862 >> 172.21.15.33:0/3287521862 conn(0x5585f2667310 msgr2=0x5585f26a0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:17.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f2199197740 1 -- 172.21.15.33:0/3287521862 shutdown_connections 2024-09-16T00:21:17.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f2199197740 1 -- 172.21.15.33:0/3287521862 wait complete. 2024-09-16T00:21:17.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.761+0000 7f2199197740 20 client.16104 trim_cache size 0 max 16384 2024-09-16T00:21:17.771 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:21:17.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.762+0000 7f2196dbe640 1 -- 172.21.15.33:0/1385792079 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21800a32c0 msgr2=0x7f21800a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:17.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.762+0000 7f2196dbe640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21800a32c0 0x7f21800a5780 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2188005fd0 tx=0x7f2188009500 comp rx=0 tx=0).stop 2024-09-16T00:21:17.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.762+0000 7f2196dbe640 1 -- 172.21.15.33:0/1385792079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2190101dd0 msgr2=0x7f21900744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:17.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.762+0000 7f2196dbe640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2190101dd0 0x7f21900744f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f218400e9f0 tx=0x7f218400edf0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.762+0000 7f2196dbe640 1 -- 172.21.15.33:0/1385792079 shutdown_connections 2024-09-16T00:21:17.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.762+0000 7f2196dbe640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21800a32c0 0x7f21800a5780 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.762+0000 7f2196dbe640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2190106f00 0x7f2190074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.762+0000 7f2196dbe640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190102780 0x7f2190074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.762+0000 7f2196dbe640 1 --2- 172.21.15.33:0/1385792079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2190101dd0 0x7f21900744f0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:17.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.762+0000 7f2196dbe640 1 -- 172.21.15.33:0/1385792079 >> 172.21.15.33:0/1385792079 conn(0x7f21900fd8b0 msgr2=0x7f21900febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:17.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.762+0000 7f2196dbe640 1 -- 172.21.15.33:0/1385792079 shutdown_connections 2024-09-16T00:21:17.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:17.762+0000 7f2196dbe640 1 -- 172.21.15.33:0/1385792079 wait complete. 2024-09-16T00:21:18.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 191366 15.07 MB/sec execute 26 sec latency 13.392 ms 2024-09-16T00:21:18.685 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:21:18.959 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:19.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:19 smithi033 ceph-mon[31243]: pgmap v475: 129 pgs: 129 active+clean; 94 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 278 op/s 2024-09-16T00:21:19.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:19 smithi083 ceph-mon[35242]: pgmap v475: 129 pgs: 129 active+clean; 94 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 278 op/s 2024-09-16T00:21:19.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:19 smithi060 ceph-mon[35185]: pgmap v475: 129 pgs: 129 active+clean; 94 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 278 op/s 2024-09-16T00:21:19.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.634+0000 7efde349a640 1 -- 172.21.15.33:0/3450857987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efddc1082a0 msgr2=0x7efddc10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:19.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.634+0000 7efde349a640 1 --2- 172.21.15.33:0/3450857987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efddc1082a0 0x7efddc10ec40 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7efdd4009cd0 tx=0x7efdd402a8e0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.634+0000 7efde349a640 1 -- 172.21.15.33:0/3450857987 shutdown_connections 2024-09-16T00:21:19.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.634+0000 7efde349a640 1 --2- 172.21.15.33:0/3450857987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efddc1082a0 0x7efddc10ec40 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.634+0000 7efde349a640 1 --2- 172.21.15.33:0/3450857987 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efddc1036e0 0x7efddc107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.634+0000 7efde349a640 1 --2- 172.21.15.33:0/3450857987 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efddc102d30 0x7efddc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.634+0000 7efde349a640 1 -- 172.21.15.33:0/3450857987 >> 172.21.15.33:0/3450857987 conn(0x7efddc0fc7d0 msgr2=0x7efddc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.634+0000 7efde349a640 1 -- 172.21.15.33:0/3450857987 shutdown_connections 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.634+0000 7efde349a640 1 -- 172.21.15.33:0/3450857987 wait complete. 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde349a640 1 Processor -- start 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde349a640 1 -- start start 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde349a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efddc102d30 0x7efddc198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde349a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efddc1036e0 0x7efddc198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde349a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efddc1082a0 0x7efddc19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde349a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efddc1120a0 con 0x7efddc102d30 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde349a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efddc111f20 con 0x7efddc1036e0 2024-09-16T00:21:19.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde349a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efddc112220 con 0x7efddc1082a0 2024-09-16T00:21:19.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde1236640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efddc102d30 0x7efddc198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde1236640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efddc102d30 0x7efddc198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52344/0 (socket says 172.21.15.33:52344) 2024-09-16T00:21:19.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde1236640 1 -- 172.21.15.33:0/1839094898 learned_addr learned my addr 172.21.15.33:0/1839094898 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:19.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde1a37640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efddc1082a0 0x7efddc19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.635+0000 7efde0a35640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efddc1036e0 0x7efddc198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.636+0000 7efde1236640 1 -- 172.21.15.33:0/1839094898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efddc1082a0 msgr2=0x7efddc19ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:19.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.636+0000 7efde1236640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efddc1082a0 0x7efddc19ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.636+0000 7efde1236640 1 -- 172.21.15.33:0/1839094898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efddc1036e0 msgr2=0x7efddc198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:19.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.636+0000 7efde1236640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efddc1036e0 0x7efddc198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.636+0000 7efde1236640 1 -- 172.21.15.33:0/1839094898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efddc19d520 con 0x7efddc102d30 2024-09-16T00:21:19.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.636+0000 7efde0a35640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efddc1036e0 0x7efddc198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:21:19.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.636+0000 7efde1236640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efddc102d30 0x7efddc198570 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7efdd800e9f0 tx=0x7efdd800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.636+0000 7efdd27fc640 1 -- 172.21.15.33:0/1839094898 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd800cd80 con 0x7efddc102d30 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.636+0000 7efdd27fc640 1 -- 172.21.15.33:0/1839094898 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdd8005500 con 0x7efddc102d30 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.636+0000 7efdd27fc640 1 -- 172.21.15.33:0/1839094898 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd8010690 con 0x7efddc102d30 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.636+0000 7efde349a640 1 -- 172.21.15.33:0/1839094898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efddc19d810 con 0x7efddc102d30 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.636+0000 7efde349a640 1 -- 172.21.15.33:0/1839094898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efddc1a50f0 con 0x7efddc102d30 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.639+0000 7efdd27fc640 1 -- 172.21.15.33:0/1839094898 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7efdd80046c0 con 0x7efddc102d30 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.639+0000 7efde5873740 1 Processor -- start 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.639+0000 7efde5873740 1 -- start start 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.639+0000 7efde5873740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc57b40 0x56533fc56090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.639+0000 7efde5873740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc565d0 0x56533fc56a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.639+0000 7efde5873740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc61370 0x56533fc63760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.639+0000 7efde5873740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56533fbbc2f0 con 0x56533fc565d0 2024-09-16T00:21:19.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.639+0000 7efde5873740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56533fb63cf0 con 0x56533fc61370 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.639+0000 7efde5873740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56533fbb82b0 con 0x56533fc57b40 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.639+0000 7efde1236640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc565d0 0x56533fc56a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.639+0000 7efde1a37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc57b40 0x56533fc56090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efde0a35640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc61370 0x56533fc63760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efde1a37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc57b40 0x56533fc56090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60154/0 (socket says 172.21.15.33:60154) 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efde1a37640 1 -- 172.21.15.33:0/3205208483 learned_addr learned my addr 172.21.15.33:0/3205208483 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efdd27fc640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efdc80a32c0 0x7efdc80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efde1a37640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efdc80a32c0 0x7efdc80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efde1a37640 1 -- 172.21.15.33:0/3205208483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc61370 msgr2=0x56533fc63760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efde1a37640 1 --2- 172.21.15.33:0/3205208483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc61370 0x56533fc63760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efde1a37640 1 -- 172.21.15.33:0/3205208483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc565d0 msgr2=0x56533fc56a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efde1a37640 1 --2- 172.21.15.33:0/3205208483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc565d0 0x56533fc56a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efde1a37640 1 -- 172.21.15.33:0/3205208483 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56533fb483a0 con 0x56533fc57b40 2024-09-16T00:21:19.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efde0a35640 1 --2- 172.21.15.33:0/3205208483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc61370 0x56533fc63760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:19.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efdd27fc640 1 -- 172.21.15.33:0/1839094898 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7efdd80d0280 con 0x7efddc102d30 2024-09-16T00:21:19.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efde1a37640 1 --2- 172.21.15.33:0/3205208483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc57b40 0x56533fc56090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efdd4009430 tx=0x7efdd4031940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9736a26aab27649 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:19.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efde1a37640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efdc80a32c0 0x7efdc80a5780 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7efdd4047000 tx=0x7efdd404a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:19.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.640+0000 7efdc77fe640 1 -- 172.21.15.33:0/3205208483 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd4031430 con 0x56533fc57b40 2024-09-16T00:21:19.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.641+0000 7efdc77fe640 1 -- 172.21.15.33:0/3205208483 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdd4036070 con 0x56533fc57b40 2024-09-16T00:21:19.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.641+0000 7efdc77fe640 1 -- 172.21.15.33:0/3205208483 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd404a450 con 0x56533fc57b40 2024-09-16T00:21:19.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.641+0000 7efde5873740 1 -- 172.21.15.33:0/3205208483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc57b40 msgr2=0x56533fc56090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:19.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.641+0000 7efde5873740 1 --2- 172.21.15.33:0/3205208483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc57b40 0x56533fc56090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efdd4009430 tx=0x7efdd4031940 comp rx=0 tx=0).stop 2024-09-16T00:21:19.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.641+0000 7efde5873740 1 -- 172.21.15.33:0/3205208483 shutdown_connections 2024-09-16T00:21:19.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.641+0000 7efde5873740 1 --2- 172.21.15.33:0/3205208483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc61370 0x56533fc63760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.641+0000 7efde5873740 1 --2- 172.21.15.33:0/3205208483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc565d0 0x56533fc56a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.641+0000 7efde5873740 1 --2- 172.21.15.33:0/3205208483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc57b40 0x56533fc56090 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.642+0000 7efde5873740 1 -- 172.21.15.33:0/3205208483 >> 172.21.15.33:0/3205208483 conn(0x56533fc1f310 msgr2=0x56533fc52c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.642+0000 7efde5873740 1 -- 172.21.15.33:0/3205208483 shutdown_connections 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.642+0000 7efde5873740 1 -- 172.21.15.33:0/3205208483 wait complete. 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.642+0000 7efde5873740 1 -- start start 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.642+0000 7efde5873740 1 Processor -- start 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.643+0000 7efde5873740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc565d0 0x56533fc60d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.643+0000 7efde5873740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc57b40 0x56533fc5e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.643+0000 7efde5873740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc61370 0x56533fc5e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.643+0000 7efde5873740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56533fc63fb0 con 0x56533fc61370 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.643+0000 7efde5873740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56533fb77060 con 0x56533fc57b40 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.643+0000 7efde5873740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56533fc64130 con 0x56533fc565d0 2024-09-16T00:21:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.643+0000 7efde5873740 10 client.0 _ll_register_callbacks cb 0x56533f8ca8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.643+0000 7efde0a35640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc57b40 0x56533fc5e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.643+0000 7efde0a35640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc57b40 0x56533fc5e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51158/0 (socket says 172.21.15.33:51158) 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.643+0000 7efde0a35640 1 -- 172.21.15.33:0/3075636432 learned_addr learned my addr 172.21.15.33:0/3075636432 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.643+0000 7efde1236640 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc565d0 0x56533fc60d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.643+0000 7efde1a37640 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc61370 0x56533fc5e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efde0a35640 1 -- 172.21.15.33:0/3075636432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc565d0 msgr2=0x56533fc60d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efde0a35640 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc565d0 0x56533fc60d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efde0a35640 1 -- 172.21.15.33:0/3075636432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc61370 msgr2=0x56533fc5e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efde0a35640 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc61370 0x56533fc5e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efde0a35640 1 -- 172.21.15.33:0/3075636432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56533f92d3c0 con 0x56533fc57b40 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efde0a35640 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc57b40 0x56533fc5e050 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efdcc013710 tx=0x7efdcc013b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9b77bfa737ec7c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efdc4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:21:19.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efdc4ff9640 1 -- 172.21.15.33:0/3075636432 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdcc004020 con 0x56533fc57b40 2024-09-16T00:21:19.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efde5873740 1 -- 172.21.15.33:0/3075636432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdd4036070 con 0x56533fc57b40 2024-09-16T00:21:19.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efdc4ff9640 1 -- 172.21.15.33:0/3075636432 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7efdcc0041c0 con 0x56533fc57b40 2024-09-16T00:21:19.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efdc4ff9640 1 -- 172.21.15.33:0/3075636432 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efdcc025050 con 0x56533fc57b40 2024-09-16T00:21:19.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efde5873740 10 client.25679 fetch_fsmap learned FSMap version 29 2024-09-16T00:21:19.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.644+0000 7efde5873740 1 -- 172.21.15.33:0/3075636432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56533fc6fb30 con 0x56533fc57b40 2024-09-16T00:21:19.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.645+0000 7efdc4ff9640 1 -- 172.21.15.33:0/3075636432 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdd4036070 con 0x56533fc57b40 2024-09-16T00:21:19.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.645+0000 7efdc4ff9640 1 -- 172.21.15.33:0/3075636432 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7efdcc01a030 con 0x56533fc57b40 2024-09-16T00:21:19.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.645+0000 7efde5873740 10 client.25679 fetch_fsmap finished waiting for FSMap version 29 2024-09-16T00:21:19.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.645+0000 7efde5873740 10 client.25679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:21:19.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.645+0000 7efde5873740 20 client.25679 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:19.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.645+0000 7efde5873740 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56533fc71000 0x56533fc7c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.645+0000 7efde5873740 4 client.25679 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:21:19.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.645+0000 7efde5873740 1 -- 172.21.15.33:0/3075636432 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56533fbc9860 con 0x56533fc71000 2024-09-16T00:21:19.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.645+0000 7efde1236640 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56533fc71000 0x56533fc7c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.645+0000 7efdc4ff9640 10 client.25679 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:19.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.645+0000 7efde1236640 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56533fc71000 0x56533fc7c090 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af959ad14f7ab65b server_cookie=8274e54a072176c0 in_seq=0 out_seq=0 2024-09-16T00:21:19.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.646+0000 7efdc4ff9640 1 -- 172.21.15.33:0/3075636432 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56533fbc9860 con 0x56533fc71000 2024-09-16T00:21:19.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.646+0000 7efdc4ff9640 10 client.25679 handle_command_reply: tid=0 2024-09-16T00:21:19.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.646+0000 7efde5873740 1 client.25679 shutdown 2024-09-16T00:21:19.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.646+0000 7efde5873740 1 -- 172.21.15.33:0/3075636432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc57b40 msgr2=0x56533fc5e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:19.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.646+0000 7efde5873740 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc57b40 0x56533fc5e050 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efdcc013710 tx=0x7efdcc013b10 comp rx=0 tx=0).stop 2024-09-16T00:21:19.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.646+0000 7efde5873740 1 -- 172.21.15.33:0/3075636432 shutdown_connections 2024-09-16T00:21:19.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.646+0000 7efde5873740 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56533fc71000 0x56533fc7c090 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.646+0000 7efde5873740 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc61370 0x56533fc5e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.646+0000 7efde5873740 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc57b40 0x56533fc5e050 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.646+0000 7efde5873740 1 --2- 172.21.15.33:0/3075636432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc565d0 0x56533fc60d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.646+0000 7efde5873740 1 -- 172.21.15.33:0/3075636432 >> 172.21.15.33:0/3075636432 conn(0x56533fc1f310 msgr2=0x56533fc58960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:19.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.647+0000 7efdc4ff9640 0 client.25679 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:19.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.647+0000 7efde5873740 1 -- 172.21.15.33:0/3075636432 shutdown_connections 2024-09-16T00:21:19.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.647+0000 7efde5873740 1 -- 172.21.15.33:0/3075636432 wait complete. 2024-09-16T00:21:19.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.647+0000 7efde5873740 20 client.25679 trim_cache size 0 max 16384 2024-09-16T00:21:19.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 193977 14.95 MB/sec execute 27 sec latency 15.845 ms 2024-09-16T00:21:19.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.661+0000 7efde5873740 1 Processor -- start 2024-09-16T00:21:19.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.662+0000 7efde5873740 1 -- start start 2024-09-16T00:21:19.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.662+0000 7efde5873740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc565d0 0x56533fc6d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.662+0000 7efde5873740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc57b40 0x56533fc5d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.662+0000 7efde5873740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc61370 0x56533fc7f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.662+0000 7efde5873740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efddc1af840 con 0x56533fc57b40 2024-09-16T00:21:19.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.662+0000 7efde5873740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efddc1ad110 con 0x56533fc565d0 2024-09-16T00:21:19.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.662+0000 7efde5873740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efddc1aaf60 con 0x56533fc61370 2024-09-16T00:21:19.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.662+0000 7efde1236640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc565d0 0x56533fc6d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.662+0000 7efde0a35640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc57b40 0x56533fc5d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.662+0000 7efde1236640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc565d0 0x56533fc6d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51164/0 (socket says 172.21.15.33:51164) 2024-09-16T00:21:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.662+0000 7efde1236640 1 -- 172.21.15.33:0/3912849580 learned_addr learned my addr 172.21.15.33:0/3912849580 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.662+0000 7efde1a37640 1 --2- 172.21.15.33:0/3912849580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc61370 0x56533fc7f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.663+0000 7efde1236640 1 -- 172.21.15.33:0/3912849580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc61370 msgr2=0x56533fc7f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.663+0000 7efde1236640 1 --2- 172.21.15.33:0/3912849580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc61370 0x56533fc7f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.663+0000 7efde1236640 1 -- 172.21.15.33:0/3912849580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc57b40 msgr2=0x56533fc5d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.663+0000 7efde1236640 1 --2- 172.21.15.33:0/3912849580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc57b40 0x56533fc5d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.663+0000 7efde1236640 1 -- 172.21.15.33:0/3912849580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56533fc5ea20 con 0x56533fc565d0 2024-09-16T00:21:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.663+0000 7efde1236640 1 --2- 172.21.15.33:0/3912849580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc565d0 0x56533fc6d3d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdd800c6a0 tx=0x7efdd80bc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=742de6c8125ecd3a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:19.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.663+0000 7efdc6ffd640 1 -- 172.21.15.33:0/3912849580 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd809bf00 con 0x56533fc565d0 2024-09-16T00:21:19.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.663+0000 7efdc6ffd640 1 -- 172.21.15.33:0/3912849580 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdd809cac0 con 0x56533fc565d0 2024-09-16T00:21:19.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.663+0000 7efdc6ffd640 1 -- 172.21.15.33:0/3912849580 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd809bd20 con 0x56533fc565d0 2024-09-16T00:21:19.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.664+0000 7efde5873740 1 -- 172.21.15.33:0/3912849580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc565d0 msgr2=0x56533fc6d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:19.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.664+0000 7efde5873740 1 --2- 172.21.15.33:0/3912849580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc565d0 0x56533fc6d3d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdd800c6a0 tx=0x7efdd80bc040 comp rx=0 tx=0).stop 2024-09-16T00:21:19.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.664+0000 7efde5873740 1 -- 172.21.15.33:0/3912849580 shutdown_connections 2024-09-16T00:21:19.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.664+0000 7efde5873740 1 --2- 172.21.15.33:0/3912849580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc61370 0x56533fc7f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.664+0000 7efde5873740 1 --2- 172.21.15.33:0/3912849580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc57b40 0x56533fc5d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.664+0000 7efde5873740 1 --2- 172.21.15.33:0/3912849580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc565d0 0x56533fc6d3d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.664+0000 7efde5873740 1 -- 172.21.15.33:0/3912849580 >> 172.21.15.33:0/3912849580 conn(0x56533fc1f310 msgr2=0x56533fc5c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:19.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.664+0000 7efde5873740 1 -- 172.21.15.33:0/3912849580 shutdown_connections 2024-09-16T00:21:19.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.664+0000 7efde5873740 1 -- 172.21.15.33:0/3912849580 wait complete. 2024-09-16T00:21:19.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.665+0000 7efde5873740 1 -- start start 2024-09-16T00:21:19.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.665+0000 7efde5873740 1 Processor -- start 2024-09-16T00:21:19.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde5873740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc565d0 0x56533fc7f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde5873740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc57b40 0x56533fc7f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde5873740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc61370 0x56533fc7ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde5873740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efddc1ab5d0 con 0x56533fc61370 2024-09-16T00:21:19.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde5873740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efddc1aca40 con 0x56533fc57b40 2024-09-16T00:21:19.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde5873740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efddc1abca0 con 0x56533fc565d0 2024-09-16T00:21:19.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde1236640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc565d0 0x56533fc7f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde1236640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc565d0 0x56533fc7f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60170/0 (socket says 172.21.15.33:60170) 2024-09-16T00:21:19.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde1236640 1 -- 172.21.15.33:0/3042116665 learned_addr learned my addr 172.21.15.33:0/3042116665 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:19.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde0a35640 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc57b40 0x56533fc7f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde1a37640 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc61370 0x56533fc7ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde5873740 10 client.0 _ll_register_callbacks cb 0x7efdcc0097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:19.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde1236640 1 -- 172.21.15.33:0/3042116665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc57b40 msgr2=0x56533fc7f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:19.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde1236640 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc57b40 0x56533fc7f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde1236640 1 -- 172.21.15.33:0/3042116665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc61370 msgr2=0x56533fc7ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:19.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde1236640 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc61370 0x56533fc7ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde1236640 1 -- 172.21.15.33:0/3042116665 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56533fc7d240 con 0x56533fc565d0 2024-09-16T00:21:19.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efde1236640 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc565d0 0x56533fc7f490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efdd8098d90 tx=0x7efdd8098040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=294026a7c057695 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:19.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.666+0000 7efdc4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:21:19.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efdc4ff9640 1 -- 172.21.15.33:0/3042116665 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd809bd20 con 0x56533fc565d0 2024-09-16T00:21:19.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efdc4ff9640 1 -- 172.21.15.33:0/3042116665 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7efdd8098a30 con 0x56533fc565d0 2024-09-16T00:21:19.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efde5873740 1 -- 172.21.15.33:0/3042116665 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdd809cac0 con 0x56533fc565d0 2024-09-16T00:21:19.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efdc4ff9640 1 -- 172.21.15.33:0/3042116665 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efdd809bf00 con 0x56533fc565d0 2024-09-16T00:21:19.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efde5873740 10 client.25606 fetch_fsmap learned FSMap version 29 2024-09-16T00:21:19.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efde5873740 1 -- 172.21.15.33:0/3042116665 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56533fce4230 con 0x56533fc565d0 2024-09-16T00:21:19.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efdc4ff9640 1 -- 172.21.15.33:0/3042116665 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdd809cac0 con 0x56533fc565d0 2024-09-16T00:21:19.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efdc4ff9640 1 -- 172.21.15.33:0/3042116665 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7efdd80a4c30 con 0x56533fc565d0 2024-09-16T00:21:19.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efde5873740 10 client.25606 fetch_fsmap finished waiting for FSMap version 29 2024-09-16T00:21:19.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efde5873740 10 client.25606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:21:19.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efde5873740 20 client.25606 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:19.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efde5873740 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56533fce4880 0x56533fce6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:19.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efde5873740 4 client.25606 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:21:19.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.667+0000 7efde5873740 1 -- 172.21.15.33:0/3042116665 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56533fbc3d70 con 0x56533fce4880 2024-09-16T00:21:19.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.668+0000 7efde0a35640 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56533fce4880 0x56533fce6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:19.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.668+0000 7efdc4ff9640 10 client.25606 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:19.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.668+0000 7efde0a35640 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56533fce4880 0x56533fce6ca0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad8cc3ed32f5a8b7 server_cookie=bc1fcff0a89d3cd3 in_seq=0 out_seq=0 2024-09-16T00:21:19.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.668+0000 7efdc4ff9640 1 -- 172.21.15.33:0/3042116665 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56533fbc3d70 con 0x56533fce4880 2024-09-16T00:21:19.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.668+0000 7efdc4ff9640 10 client.25606 handle_command_reply: tid=0 2024-09-16T00:21:19.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.668+0000 7efde5873740 1 client.25606 shutdown 2024-09-16T00:21:19.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.669+0000 7efde5873740 1 -- 172.21.15.33:0/3042116665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc565d0 msgr2=0x56533fc7f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:19.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.669+0000 7efde5873740 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc565d0 0x56533fc7f490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efdd8098d90 tx=0x7efdd8098040 comp rx=0 tx=0).stop 2024-09-16T00:21:19.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.669+0000 7efde5873740 1 -- 172.21.15.33:0/3042116665 shutdown_connections 2024-09-16T00:21:19.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.669+0000 7efde5873740 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56533fce4880 0x56533fce6ca0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.669+0000 7efde5873740 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56533fc61370 0x56533fc7ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.669+0000 7efde5873740 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56533fc57b40 0x56533fc7f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.669+0000 7efde5873740 1 --2- 172.21.15.33:0/3042116665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56533fc565d0 0x56533fc7f490 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.669+0000 7efdc4ff9640 0 client.25606 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:19.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.669+0000 7efde5873740 1 -- 172.21.15.33:0/3042116665 >> 172.21.15.33:0/3042116665 conn(0x56533fc1f310 msgr2=0x56533fc5d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:19.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.669+0000 7efde5873740 1 -- 172.21.15.33:0/3042116665 shutdown_connections 2024-09-16T00:21:19.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.669+0000 7efde5873740 1 -- 172.21.15.33:0/3042116665 wait complete. 2024-09-16T00:21:19.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.669+0000 7efde5873740 20 client.25606 trim_cache size 0 max 16384 2024-09-16T00:21:19.674 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:21:19.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.670+0000 7efde349a640 1 -- 172.21.15.33:0/1839094898 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efdc80a32c0 msgr2=0x7efdc80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:19.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.670+0000 7efde349a640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efdc80a32c0 0x7efdc80a5780 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7efdd4047000 tx=0x7efdd404a000 comp rx=0 tx=0).stop 2024-09-16T00:21:19.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.670+0000 7efde349a640 1 -- 172.21.15.33:0/1839094898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efddc102d30 msgr2=0x7efddc198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:19.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.670+0000 7efde349a640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efddc102d30 0x7efddc198570 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7efdd800e9f0 tx=0x7efdd800edf0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.670+0000 7efde349a640 1 -- 172.21.15.33:0/1839094898 shutdown_connections 2024-09-16T00:21:19.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.670+0000 7efde349a640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efdc80a32c0 0x7efdc80a5780 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.670+0000 7efde349a640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efddc1082a0 0x7efddc19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.670+0000 7efde349a640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efddc1036e0 0x7efddc198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.670+0000 7efde349a640 1 --2- 172.21.15.33:0/1839094898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efddc102d30 0x7efddc198570 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:19.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.670+0000 7efde349a640 1 -- 172.21.15.33:0/1839094898 >> 172.21.15.33:0/1839094898 conn(0x7efddc0fc7d0 msgr2=0x7efddc105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:19.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.670+0000 7efde349a640 1 -- 172.21.15.33:0/1839094898 shutdown_connections 2024-09-16T00:21:19.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:19.670+0000 7efde349a640 1 -- 172.21.15.33:0/1839094898 wait complete. 2024-09-16T00:21:20.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 196251 14.91 MB/sec execute 28 sec latency 21.212 ms 2024-09-16T00:21:21.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:21 smithi033 ceph-mon[31243]: pgmap v476: 129 pgs: 129 active+clean; 96 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 325 op/s 2024-09-16T00:21:21.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:21 smithi083 ceph-mon[35242]: pgmap v476: 129 pgs: 129 active+clean; 96 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 325 op/s 2024-09-16T00:21:21.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:21 smithi060 ceph-mon[35185]: pgmap v476: 129 pgs: 129 active+clean; 96 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 325 op/s 2024-09-16T00:21:21.495 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:21:21.496 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:21:21.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 198952 14.98 MB/sec execute 29 sec latency 73.819 ms 2024-09-16T00:21:21.815 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:22.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.331+0000 7f8727931640 1 -- 172.21.15.33:0/3830644693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8720102d30 msgr2=0x7f8720103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:22.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.331+0000 7f8727931640 1 --2- 172.21.15.33:0/3830644693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8720102d30 0x7f8720103110 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f871c009930 tx=0x7f871c02a420 comp rx=0 tx=0).stop 2024-09-16T00:21:22.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.331+0000 7f8727931640 1 -- 172.21.15.33:0/3830644693 shutdown_connections 2024-09-16T00:21:22.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.331+0000 7f8727931640 1 --2- 172.21.15.33:0/3830644693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87201082a0 0x7f872010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.331+0000 7f8727931640 1 --2- 172.21.15.33:0/3830644693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87201036e0 0x7f8720107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.331+0000 7f8727931640 1 --2- 172.21.15.33:0/3830644693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8720102d30 0x7f8720103110 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.331+0000 7f8727931640 1 -- 172.21.15.33:0/3830644693 >> 172.21.15.33:0/3830644693 conn(0x7f87200fc7d0 msgr2=0x7f87200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:22.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.331+0000 7f8727931640 1 -- 172.21.15.33:0/3830644693 shutdown_connections 2024-09-16T00:21:22.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.331+0000 7f8727931640 1 -- 172.21.15.33:0/3830644693 wait complete. 2024-09-16T00:21:22.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f8727931640 1 Processor -- start 2024-09-16T00:21:22.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f8727931640 1 -- start start 2024-09-16T00:21:22.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f8727931640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8720102d30 0x7f8720100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f8727931640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87201036e0 0x7f8720100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f8727931640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87201082a0 0x7f8720101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f8727931640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f87201120a0 con 0x7f8720102d30 2024-09-16T00:21:22.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f8727931640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8720111f20 con 0x7f87201036e0 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f8727931640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8720112220 con 0x7f87201082a0 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f87256cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8720102d30 0x7f8720100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f87256cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8720102d30 0x7f8720100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50786/0 (socket says 172.21.15.33:50786) 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f87256cd640 1 -- 172.21.15.33:0/958932754 learned_addr learned my addr 172.21.15.33:0/958932754 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f8725ece640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87201082a0 0x7f8720101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.332+0000 7f8724ecc640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87201036e0 0x7f8720100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.333+0000 7f87256cd640 1 -- 172.21.15.33:0/958932754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87201082a0 msgr2=0x7f8720101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.333+0000 7f87256cd640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87201082a0 0x7f8720101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.333+0000 7f87256cd640 1 -- 172.21.15.33:0/958932754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87201036e0 msgr2=0x7f8720100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.333+0000 7f87256cd640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87201036e0 0x7f8720100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.333+0000 7f87256cd640 1 -- 172.21.15.33:0/958932754 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87201a6a80 con 0x7f8720102d30 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.333+0000 7f87256cd640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8720102d30 0x7f8720100990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f871c009230 tx=0x7f871c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:22.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.333+0000 7f87167fc640 1 -- 172.21.15.33:0/958932754 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c03f030 con 0x7f8720102d30 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.333+0000 7f87167fc640 1 -- 172.21.15.33:0/958932754 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f871c004440 con 0x7f8720102d30 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.333+0000 7f8727931640 1 -- 172.21.15.33:0/958932754 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87201a6d10 con 0x7f8720102d30 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.333+0000 7f87167fc640 1 -- 172.21.15.33:0/958932754 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c035070 con 0x7f8720102d30 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.333+0000 7f8727931640 1 -- 172.21.15.33:0/958932754 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87201a71a0 con 0x7f8720102d30 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.335+0000 7f87167fc640 1 -- 172.21.15.33:0/958932754 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f871c042030 con 0x7f8720102d30 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.336+0000 7f8729d0a740 1 Processor -- start 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.336+0000 7f8729d0a740 1 -- start start 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.336+0000 7f87167fc640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f870c0a3240 0x7f870c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.336+0000 7f8729d0a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a00f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.336+0000 7f87167fc640 1 -- 172.21.15.33:0/958932754 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f871c0f0070 con 0x7f8720102d30 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.336+0000 7f8729d0a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a01480 0x55ed01a0b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.336+0000 7f8725ece640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f870c0a3240 0x7f870c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.336+0000 7f8729d0a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a0bd40 0x55ed01a0e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.336+0000 7f8729d0a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ed01966280 con 0x55ed01a00b60 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.336+0000 7f8729d0a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ed0190db10 con 0x55ed01a01480 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.336+0000 7f8729d0a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ed01962240 con 0x55ed01a0bd40 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f87256cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a00f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f87256cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a00f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50794/0 (socket says 172.21.15.33:50794) 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f87256cd640 1 -- 172.21.15.33:0/2436494436 learned_addr learned my addr 172.21.15.33:0/2436494436 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f8724ecc640 1 --2- 172.21.15.33:0/2436494436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a01480 0x55ed01a0b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f8725ece640 1 --2- 172.21.15.33:0/2436494436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a0bd40 0x55ed01a0e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f87256cd640 1 -- 172.21.15.33:0/2436494436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a0bd40 msgr2=0x55ed01a0e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f87256cd640 1 --2- 172.21.15.33:0/2436494436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a0bd40 0x55ed01a0e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f87256cd640 1 -- 172.21.15.33:0/2436494436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a01480 msgr2=0x55ed01a0b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f87256cd640 1 --2- 172.21.15.33:0/2436494436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a01480 0x55ed01a0b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f87256cd640 1 -- 172.21.15.33:0/2436494436 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed0191a1f0 con 0x55ed01a00b60 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f8725ece640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f870c0a3240 0x7f870c0a5700 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8718005fd0 tx=0x7f8718011b40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f87256cd640 1 --2- 172.21.15.33:0/2436494436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a00f40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f871c0c4950 tx=0x7f871c02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f8ab6589968a2bc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f870b7fe640 1 -- 172.21.15.33:0/2436494436 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c049040 con 0x55ed01a00b60 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f870b7fe640 1 -- 172.21.15.33:0/2436494436 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f871c0dae50 con 0x55ed01a00b60 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.337+0000 7f870b7fe640 1 -- 172.21.15.33:0/2436494436 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c0c0160 con 0x55ed01a00b60 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.338+0000 7f8729d0a740 1 -- 172.21.15.33:0/2436494436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 msgr2=0x55ed01a00f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.338+0000 7f8729d0a740 1 --2- 172.21.15.33:0/2436494436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a00f40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f871c0c4950 tx=0x7f871c02a420 comp rx=0 tx=0).stop 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.338+0000 7f8729d0a740 1 -- 172.21.15.33:0/2436494436 shutdown_connections 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.338+0000 7f8729d0a740 1 --2- 172.21.15.33:0/2436494436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a0bd40 0x55ed01a0e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.338+0000 7f8729d0a740 1 --2- 172.21.15.33:0/2436494436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a01480 0x55ed01a0b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.338+0000 7f8729d0a740 1 --2- 172.21.15.33:0/2436494436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a00f40 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.338+0000 7f8729d0a740 1 -- 172.21.15.33:0/2436494436 >> 172.21.15.33:0/2436494436 conn(0x55ed019fd0f0 msgr2=0x55ed019fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:22.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.338+0000 7f8729d0a740 1 -- 172.21.15.33:0/2436494436 shutdown_connections 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.338+0000 7f8729d0a740 1 -- 172.21.15.33:0/2436494436 wait complete. 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f8729d0a740 1 -- start start 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f8729d0a740 1 Processor -- start 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f8729d0a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a08bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f8729d0a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 0x55ed01a09100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f8729d0a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a0bd40 0x55ed01a09640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f8729d0a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ed01a0e6d0 con 0x55ed01a00b60 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f8729d0a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ed01920f10 con 0x55ed01a0bd40 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f8729d0a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ed01a0e850 con 0x55ed01a01480 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f8729d0a740 10 client.0 _ll_register_callbacks cb 0x55ed018e9a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f87256cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a08bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f8724ecc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 0x55ed01a09100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f87256cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a08bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50800/0 (socket says 172.21.15.33:50800) 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f8724ecc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 0x55ed01a09100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:57404/0 (socket says 172.21.15.33:57404) 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f87256cd640 1 -- 172.21.15.33:0/61261742 learned_addr learned my addr 172.21.15.33:0/61261742 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.339+0000 7f8725ece640 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a0bd40 0x55ed01a09640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f87256cd640 1 -- 172.21.15.33:0/61261742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 msgr2=0x55ed01a09100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f87256cd640 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 0x55ed01a09100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f87256cd640 1 -- 172.21.15.33:0/61261742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a0bd40 msgr2=0x55ed01a09640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f87256cd640 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a0bd40 0x55ed01a09640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f87256cd640 1 -- 172.21.15.33:0/61261742 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed016d76a0 con 0x55ed01a00b60 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f8724ecc640 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 0x55ed01a09100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f8725ece640 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a0bd40 0x55ed01a09640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f87256cd640 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a08bc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f871c0f0390 tx=0x7f871c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa7b5330636ea84 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f8708ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:21:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f8708ff9640 1 -- 172.21.15.33:0/61261742 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c0da060 con 0x55ed01a00b60 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f8708ff9640 1 -- 172.21.15.33:0/61261742 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f871c003d20 con 0x55ed01a00b60 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f8729d0a740 1 -- 172.21.15.33:0/61261742 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f871c0dae50 con 0x55ed01a00b60 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f8708ff9640 1 -- 172.21.15.33:0/61261742 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f871c0d4030 con 0x55ed01a00b60 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f8729d0a740 10 client.16134 fetch_fsmap learned FSMap version 30 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.340+0000 7f8729d0a740 1 -- 172.21.15.33:0/61261742 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ed01a0e6d0 con 0x55ed01a00b60 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.341+0000 7f8708ff9640 1 -- 172.21.15.33:0/61261742 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f871c0dae50 con 0x55ed01a00b60 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.341+0000 7f8708ff9640 1 -- 172.21.15.33:0/61261742 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f871c0c01f0 con 0x55ed01a00b60 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.341+0000 7f8729d0a740 10 client.16134 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.341+0000 7f8729d0a740 10 client.16134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.341+0000 7f8729d0a740 20 client.16134 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.341+0000 7f8729d0a740 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ed01a1b370 0x55ed01a1d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.341+0000 7f8729d0a740 4 client.16134 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.341+0000 7f8729d0a740 1 -- 172.21.15.33:0/61261742 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed019737f0 con 0x55ed01a1b370 2024-09-16T00:21:22.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.341+0000 7f8724ecc640 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ed01a1b370 0x55ed01a1d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8708ff9640 10 client.16134 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8724ecc640 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ed01a1b370 0x55ed01a1d760 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2604c6cbef1c8544 server_cookie=6cf8b227c30f6df0 in_seq=0 out_seq=0 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8708ff9640 1 -- 172.21.15.33:0/61261742 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ed019737f0 con 0x55ed01a1b370 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8708ff9640 10 client.16134 handle_command_reply: tid=0 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8729d0a740 1 client.16134 shutdown 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8729d0a740 1 -- 172.21.15.33:0/61261742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 msgr2=0x55ed01a08bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8729d0a740 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a08bc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f871c0f0390 tx=0x7f871c009260 comp rx=0 tx=0).stop 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8729d0a740 1 -- 172.21.15.33:0/61261742 shutdown_connections 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8729d0a740 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ed01a1b370 0x55ed01a1d760 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8729d0a740 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a0bd40 0x55ed01a09640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8729d0a740 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 0x55ed01a09100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8708ff9640 0 client.16134 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:22.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8729d0a740 1 --2- 172.21.15.33:0/61261742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a08bc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.342+0000 7f8729d0a740 1 -- 172.21.15.33:0/61261742 >> 172.21.15.33:0/61261742 conn(0x55ed019fd0f0 msgr2=0x55ed019fc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:22.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.343+0000 7f8729d0a740 1 -- 172.21.15.33:0/61261742 shutdown_connections 2024-09-16T00:21:22.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.343+0000 7f8729d0a740 1 -- 172.21.15.33:0/61261742 wait complete. 2024-09-16T00:21:22.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.343+0000 7f8729d0a740 20 client.16134 trim_cache size 0 max 16384 2024-09-16T00:21:22.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.363+0000 7f8729d0a740 1 Processor -- start 2024-09-16T00:21:22.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.363+0000 7f8729d0a740 1 -- start start 2024-09-16T00:21:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.363+0000 7f8729d0a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a22ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.363+0000 7f8729d0a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a01480 0x55ed01a23410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.363+0000 7f8729d0a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a0bd40 0x55ed019fba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.363+0000 7f8729d0a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f87201b3c60 con 0x55ed01a00b60 2024-09-16T00:21:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.363+0000 7f8729d0a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f87201b1530 con 0x55ed01a01480 2024-09-16T00:21:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.363+0000 7f8729d0a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f87201af380 con 0x55ed01a0bd40 2024-09-16T00:21:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.363+0000 7f87256cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a22ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.363+0000 7f87256cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a22ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50814/0 (socket says 172.21.15.33:50814) 2024-09-16T00:21:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.363+0000 7f87256cd640 1 -- 172.21.15.33:0/1698505711 learned_addr learned my addr 172.21.15.33:0/1698505711 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.363+0000 7f8725ece640 1 --2- 172.21.15.33:0/1698505711 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a0bd40 0x55ed019fba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.364+0000 7f8724ecc640 1 --2- 172.21.15.33:0/1698505711 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a01480 0x55ed01a23410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.364+0000 7f87256cd640 1 -- 172.21.15.33:0/1698505711 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a0bd40 msgr2=0x55ed019fba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:22.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.364+0000 7f87256cd640 1 --2- 172.21.15.33:0/1698505711 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a0bd40 0x55ed019fba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.364+0000 7f87256cd640 1 -- 172.21.15.33:0/1698505711 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a01480 msgr2=0x55ed01a23410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:22.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.364+0000 7f87256cd640 1 --2- 172.21.15.33:0/1698505711 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a01480 0x55ed01a23410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.364+0000 7f87256cd640 1 -- 172.21.15.33:0/1698505711 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed019f7680 con 0x55ed01a00b60 2024-09-16T00:21:22.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.364+0000 7f87256cd640 1 --2- 172.21.15.33:0/1698505711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a22ed0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f871c0b8c40 tx=0x7f871c02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f287a8f506b000bb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:22.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.364+0000 7f870affd640 1 -- 172.21.15.33:0/1698505711 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c0da070 con 0x55ed01a00b60 2024-09-16T00:21:22.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.364+0000 7f870affd640 1 -- 172.21.15.33:0/1698505711 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f871c0dae50 con 0x55ed01a00b60 2024-09-16T00:21:22.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.364+0000 7f870affd640 1 -- 172.21.15.33:0/1698505711 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c049070 con 0x55ed01a00b60 2024-09-16T00:21:22.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.365+0000 7f8729d0a740 1 -- 172.21.15.33:0/1698505711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 msgr2=0x55ed01a22ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:22.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.365+0000 7f8729d0a740 1 --2- 172.21.15.33:0/1698505711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a22ed0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f871c0b8c40 tx=0x7f871c02a420 comp rx=0 tx=0).stop 2024-09-16T00:21:22.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.365+0000 7f8729d0a740 1 -- 172.21.15.33:0/1698505711 shutdown_connections 2024-09-16T00:21:22.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.365+0000 7f8729d0a740 1 --2- 172.21.15.33:0/1698505711 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a0bd40 0x55ed019fba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.365+0000 7f8729d0a740 1 --2- 172.21.15.33:0/1698505711 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a01480 0x55ed01a23410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.365+0000 7f8729d0a740 1 --2- 172.21.15.33:0/1698505711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a00b60 0x55ed01a22ed0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.365+0000 7f8729d0a740 1 -- 172.21.15.33:0/1698505711 >> 172.21.15.33:0/1698505711 conn(0x55ed019fd0f0 msgr2=0x55ed01a0ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:22.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.365+0000 7f8729d0a740 1 -- 172.21.15.33:0/1698505711 shutdown_connections 2024-09-16T00:21:22.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.365+0000 7f8729d0a740 1 -- 172.21.15.33:0/1698505711 wait complete. 2024-09-16T00:21:22.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.365+0000 7f8729d0a740 1 -- start start 2024-09-16T00:21:22.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f8729d0a740 1 Processor -- start 2024-09-16T00:21:22.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f8729d0a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a00b60 0x55ed01a141a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f8729d0a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 0x55ed01a05a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f8729d0a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a0bd40 0x55ed01a05ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f8729d0a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f87201af9f0 con 0x55ed01a0bd40 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f8729d0a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f87201b0e60 con 0x55ed01a00b60 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f8729d0a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f87201b00c0 con 0x55ed01a01480 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f8729d0a740 10 client.0 _ll_register_callbacks cb 0x7f8710009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f87256cd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a00b60 0x55ed01a141a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f8724ecc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 0x55ed01a05a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f87256cd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a00b60 0x55ed01a141a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:57086/0 (socket says 172.21.15.33:57086) 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f8725ece640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a0bd40 0x55ed01a05ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f87256cd640 1 -- 172.21.15.33:0/1596531749 learned_addr learned my addr 172.21.15.33:0/1596531749 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.366+0000 7f8724ecc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 0x55ed01a05a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:57418/0 (socket says 172.21.15.33:57418) 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f87256cd640 1 -- 172.21.15.33:0/1596531749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 msgr2=0x55ed01a05a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:22.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f87256cd640 1 --2- 172.21.15.33:0/1596531749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 0x55ed01a05a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f87256cd640 1 -- 172.21.15.33:0/1596531749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a0bd40 msgr2=0x55ed01a05ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f87256cd640 1 --2- 172.21.15.33:0/1596531749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a0bd40 0x55ed01a05ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f87256cd640 1 -- 172.21.15.33:0/1596531749 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed01a06820 con 0x55ed01a00b60 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f87256cd640 1 --2- 172.21.15.33:0/1596531749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a00b60 0x55ed01a141a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f871c0bc250 tx=0x7f871c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10dafe906f91da36 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f8708ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f8708ff9640 1 -- 172.21.15.33:0/1596531749 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c049070 con 0x55ed01a00b60 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f8708ff9640 1 -- 172.21.15.33:0/1596531749 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f871c031de0 con 0x55ed01a00b60 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f8729d0a740 1 -- 172.21.15.33:0/1596531749 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f871c0dae50 con 0x55ed01a00b60 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f8708ff9640 1 -- 172.21.15.33:0/1596531749 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f871c0da050 con 0x55ed01a00b60 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f8729d0a740 10 client.25703 fetch_fsmap learned FSMap version 30 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f8729d0a740 1 -- 172.21.15.33:0/1596531749 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ed01a06820 con 0x55ed01a00b60 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.367+0000 7f8708ff9640 1 -- 172.21.15.33:0/1596531749 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f871c0dae50 con 0x55ed01a00b60 2024-09-16T00:21:22.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.368+0000 7f8708ff9640 1 -- 172.21.15.33:0/1596531749 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f871c0ca020 con 0x55ed01a00b60 2024-09-16T00:21:22.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.368+0000 7f8729d0a740 10 client.25703 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:21:22.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.368+0000 7f8729d0a740 10 client.25703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:21:22.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.368+0000 7f8729d0a740 20 client.25703 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:22.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.368+0000 7f8729d0a740 1 --2- 172.21.15.33:0/1596531749 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ed01a1b370 0x55ed01a88270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:22.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.368+0000 7f8729d0a740 4 client.25703 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:21:22.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.368+0000 7f8729d0a740 1 -- 172.21.15.33:0/1596531749 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ed0196dd00 con 0x55ed01a1b370 2024-09-16T00:21:22.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.368+0000 7f8724ecc640 1 --2- 172.21.15.33:0/1596531749 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ed01a1b370 0x55ed01a88270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:22.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.368+0000 7f8724ecc640 1 --2- 172.21.15.33:0/1596531749 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ed01a1b370 0x55ed01a88270 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c02a1b4d9827e281 server_cookie=24e021eeda278390 in_seq=0 out_seq=0 2024-09-16T00:21:22.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.368+0000 7f8708ff9640 10 client.25703 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:22.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.369+0000 7f8708ff9640 1 -- 172.21.15.33:0/1596531749 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ed0196dd00 con 0x55ed01a1b370 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.369+0000 7f8708ff9640 10 client.25703 handle_command_reply: tid=0 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.369+0000 7f8729d0a740 1 client.25703 shutdown 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.369+0000 7f8729d0a740 1 -- 172.21.15.33:0/1596531749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a00b60 msgr2=0x55ed01a141a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.369+0000 7f8729d0a740 1 --2- 172.21.15.33:0/1596531749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a00b60 0x55ed01a141a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f871c0bc250 tx=0x7f871c009260 comp rx=0 tx=0).stop 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.369+0000 7f8729d0a740 1 -- 172.21.15.33:0/1596531749 shutdown_connections 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.369+0000 7f8729d0a740 1 --2- 172.21.15.33:0/1596531749 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ed01a1b370 0x55ed01a88270 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.369+0000 7f8729d0a740 1 --2- 172.21.15.33:0/1596531749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed01a0bd40 0x55ed01a05ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.369+0000 7f8729d0a740 1 --2- 172.21.15.33:0/1596531749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed01a01480 0x55ed01a05a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.369+0000 7f8729d0a740 1 --2- 172.21.15.33:0/1596531749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed01a00b60 0x55ed01a141a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.369+0000 7f8729d0a740 1 -- 172.21.15.33:0/1596531749 >> 172.21.15.33:0/1596531749 conn(0x55ed019fd0f0 msgr2=0x55ed01a025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.369+0000 7f8708ff9640 0 client.25703 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8729d0a740 1 -- 172.21.15.33:0/1596531749 shutdown_connections 2024-09-16T00:21:22.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8729d0a740 1 -- 172.21.15.33:0/1596531749 wait complete. 2024-09-16T00:21:22.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8729d0a740 20 client.25703 trim_cache size 0 max 16384 2024-09-16T00:21:22.375 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:21:22.375 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:21:22.375 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "1144472d-5661-40fc-8c30-f3531939ba80", 2024-09-16T00:21:22.375 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:21:22.375 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:21:22.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8727931640 1 -- 172.21.15.33:0/958932754 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f870c0a3240 msgr2=0x7f870c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:22.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8727931640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f870c0a3240 0x7f870c0a5700 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8718005fd0 tx=0x7f8718011b40 comp rx=0 tx=0).stop 2024-09-16T00:21:22.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8727931640 1 -- 172.21.15.33:0/958932754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8720102d30 msgr2=0x7f8720100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:22.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8727931640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8720102d30 0x7f8720100990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f871c009230 tx=0x7f871c004040 comp rx=0 tx=0).stop 2024-09-16T00:21:22.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8727931640 1 -- 172.21.15.33:0/958932754 shutdown_connections 2024-09-16T00:21:22.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8727931640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f870c0a3240 0x7f870c0a5700 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8727931640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87201082a0 0x7f8720101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8727931640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87201036e0 0x7f8720100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8727931640 1 --2- 172.21.15.33:0/958932754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8720102d30 0x7f8720100990 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:22.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.370+0000 7f8727931640 1 -- 172.21.15.33:0/958932754 >> 172.21.15.33:0/958932754 conn(0x7f87200fc7d0 msgr2=0x7f8720105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:22.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.371+0000 7f8727931640 1 -- 172.21.15.33:0/958932754 shutdown_connections 2024-09-16T00:21:22.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:22.371+0000 7f8727931640 1 -- 172.21.15.33:0/958932754 wait complete. 2024-09-16T00:21:22.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 202000 14.97 MB/sec execute 30 sec latency 26.759 ms 2024-09-16T00:21:23.261 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:23 smithi033 ceph-mon[31243]: pgmap v477: 129 pgs: 129 active+clean; 96 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.3 MiB/s wr, 326 op/s 2024-09-16T00:21:23.261 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:23 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:21:23.261 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:23 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:21:23.261 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:23 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:21:23.262 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:23 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:21:23.262 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:23 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:21:23.262 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:23 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:21:23.262 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:23 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:21:23.291 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:21:23.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:23 smithi083 ceph-mon[35242]: pgmap v477: 129 pgs: 129 active+clean; 96 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.3 MiB/s wr, 326 op/s 2024-09-16T00:21:23.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:23 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:21:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:23 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:21:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:23 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:21:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:23 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:21:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:23 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:21:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:23 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:21:23.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:23 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:21:23.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:23 smithi060 ceph-mon[35185]: pgmap v477: 129 pgs: 129 active+clean; 96 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.3 MiB/s wr, 326 op/s 2024-09-16T00:21:23.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:23 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:21:23.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:23 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:21:23.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:23 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:21:23.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:23 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:21:23.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:23 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:21:23.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:23 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:21:23.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:23 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:21:23.565 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:23.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 204546 14.86 MB/sec execute 31 sec latency 15.434 ms 2024-09-16T00:21:24.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.120+0000 7fb73a5a0640 1 -- 172.21.15.33:0/1934453139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb734102d90 msgr2=0x7fb734103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:24.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.120+0000 7fb73a5a0640 1 --2- 172.21.15.33:0/1934453139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb734102d90 0x7fb734103170 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb724009930 tx=0x7fb72402a420 comp rx=0 tx=0).stop 2024-09-16T00:21:24.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.120+0000 7fb73a5a0640 1 -- 172.21.15.33:0/1934453139 shutdown_connections 2024-09-16T00:21:24.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.120+0000 7fb73a5a0640 1 --2- 172.21.15.33:0/1934453139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb734108290 0x7fb73410ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.120+0000 7fb73a5a0640 1 --2- 172.21.15.33:0/1934453139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb734103740 0x7fb734107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.120+0000 7fb73a5a0640 1 --2- 172.21.15.33:0/1934453139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb734102d90 0x7fb734103170 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.120+0000 7fb73a5a0640 1 -- 172.21.15.33:0/1934453139 >> 172.21.15.33:0/1934453139 conn(0x7fb7340fc7d0 msgr2=0x7fb7340febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:24.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.120+0000 7fb73a5a0640 1 -- 172.21.15.33:0/1934453139 shutdown_connections 2024-09-16T00:21:24.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.120+0000 7fb73a5a0640 1 -- 172.21.15.33:0/1934453139 wait complete. 2024-09-16T00:21:24.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.120+0000 7fb73a5a0640 1 Processor -- start 2024-09-16T00:21:24.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.120+0000 7fb73a5a0640 1 -- start start 2024-09-16T00:21:24.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb73a5a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb734102d90 0x7fb7341985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb73a5a0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb734103740 0x7fb734198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb73a5a0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb734108290 0x7fb73419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb73a5a0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb734112090 con 0x7fb734108290 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb73a5a0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb734111f10 con 0x7fb734103740 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb73a5a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb734112210 con 0x7fb734102d90 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb738b3d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb734108290 0x7fb73419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb733fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb734102d90 0x7fb7341985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb7337fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb734103740 0x7fb734198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb738b3d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb734108290 0x7fb73419cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50856/0 (socket says 172.21.15.33:50856) 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb733fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb734102d90 0x7fb7341985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:57436/0 (socket says 172.21.15.33:57436) 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb738b3d640 1 -- 172.21.15.33:0/3837153611 learned_addr learned my addr 172.21.15.33:0/3837153611 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb738b3d640 1 -- 172.21.15.33:0/3837153611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb734102d90 msgr2=0x7fb7341985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:24.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb738b3d640 1 --2- 172.21.15.33:0/3837153611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb734102d90 0x7fb7341985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb738b3d640 1 -- 172.21.15.33:0/3837153611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb734103740 msgr2=0x7fb734198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb738b3d640 1 --2- 172.21.15.33:0/3837153611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb734103740 0x7fb734198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb738b3d640 1 -- 172.21.15.33:0/3837153611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb73419d580 con 0x7fb734108290 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.121+0000 7fb738b3d640 1 --2- 172.21.15.33:0/3837153611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb734108290 0x7fb73419cea0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009fd0 tx=0x7fb72c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.122+0000 7fb7317fa640 1 -- 172.21.15.33:0/3837153611 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c017070 con 0x7fb734108290 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.122+0000 7fb7317fa640 1 -- 172.21.15.33:0/3837153611 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb72c003e50 con 0x7fb734108290 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.122+0000 7fb73a5a0640 1 -- 172.21.15.33:0/3837153611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb73419d870 con 0x7fb734108290 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.122+0000 7fb73a5a0640 1 -- 172.21.15.33:0/3837153611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7341a5150 con 0x7fb734108290 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.122+0000 7fb7317fa640 1 -- 172.21.15.33:0/3837153611 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0106c0 con 0x7fb734108290 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.124+0000 7fb7317fa640 1 -- 172.21.15.33:0/3837153611 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb72c003970 con 0x7fb734108290 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.124+0000 7fb73c979740 1 Processor -- start 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb73c979740 1 -- start start 2024-09-16T00:21:24.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb7317fa640 1 --2- 172.21.15.33:0/3837153611 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb71c0a32c0 0x7fb71c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb73c979740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317f4a0 0x56165317f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb73c979740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165317fdc0 0x56165318a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb73c979740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165318a680 0x56165318ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb73c979740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5616530e52e0 con 0x56165318a680 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb73c979740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56165308cce0 con 0x56165317fdc0 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb73c979740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5616530e12a0 con 0x56165317f4a0 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb7337fe640 1 --2- 172.21.15.33:0/3837153611 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb71c0a32c0 0x7fb71c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb7317fa640 1 -- 172.21.15.33:0/3837153611 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb72c0cf8a0 con 0x7fb734108290 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb733fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317f4a0 0x56165317f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb733fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317f4a0 0x56165317f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:57452/0 (socket says 172.21.15.33:57452) 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb733fff640 1 -- 172.21.15.33:0/924581091 learned_addr learned my addr 172.21.15.33:0/924581091 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb738b3d640 1 --2- 172.21.15.33:0/924581091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165318a680 0x56165318ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb7337fe640 1 --2- 172.21.15.33:0/924581091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165317fdc0 0x56165318a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb733fff640 1 -- 172.21.15.33:0/924581091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165317fdc0 msgr2=0x56165318a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb733fff640 1 --2- 172.21.15.33:0/924581091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165317fdc0 0x56165318a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb733fff640 1 -- 172.21.15.33:0/924581091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165318a680 msgr2=0x56165318ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb733fff640 1 --2- 172.21.15.33:0/924581091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165318a680 0x56165318ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb733fff640 1 -- 172.21.15.33:0/924581091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561653071440 con 0x56165317f4a0 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.125+0000 7fb7337fe640 1 --2- 172.21.15.33:0/924581091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165317fdc0 0x56165318a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.126+0000 7fb7337fe640 1 --2- 172.21.15.33:0/3837153611 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb71c0a32c0 0x7fb71c0a5780 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb7340feec0 tx=0x7fb728013880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.126+0000 7fb733fff640 1 --2- 172.21.15.33:0/924581091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317f4a0 0x56165317f880 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb724009930 tx=0x7fb724031630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3580ae38d05a41a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.126+0000 7fb71a7fc640 1 -- 172.21.15.33:0/924581091 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb724040070 con 0x56165317f4a0 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.126+0000 7fb71a7fc640 1 -- 172.21.15.33:0/924581091 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb724005020 con 0x56165317f4a0 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.126+0000 7fb73c979740 1 -- 172.21.15.33:0/924581091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317f4a0 msgr2=0x56165317f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.126+0000 7fb73c979740 1 --2- 172.21.15.33:0/924581091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317f4a0 0x56165317f880 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb724009930 tx=0x7fb724031630 comp rx=0 tx=0).stop 2024-09-16T00:21:24.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.127+0000 7fb73c979740 1 -- 172.21.15.33:0/924581091 shutdown_connections 2024-09-16T00:21:24.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.127+0000 7fb73c979740 1 --2- 172.21.15.33:0/924581091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165318a680 0x56165318ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.127+0000 7fb73c979740 1 --2- 172.21.15.33:0/924581091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165317fdc0 0x56165318a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.127+0000 7fb73c979740 1 --2- 172.21.15.33:0/924581091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317f4a0 0x56165317f880 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.127+0000 7fb73c979740 1 -- 172.21.15.33:0/924581091 >> 172.21.15.33:0/924581091 conn(0x56165317ba30 msgr2=0x56165317be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:24.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.127+0000 7fb73c979740 1 -- 172.21.15.33:0/924581091 shutdown_connections 2024-09-16T00:21:24.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.127+0000 7fb73c979740 1 -- 172.21.15.33:0/924581091 wait complete. 2024-09-16T00:21:24.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.127+0000 7fb73c979740 1 -- start start 2024-09-16T00:21:24.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.127+0000 7fb73c979740 1 Processor -- start 2024-09-16T00:21:24.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.127+0000 7fb73c979740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317f4a0 0x561653189d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb73c979740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165317fdc0 0x561653187010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb73c979740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165318a680 0x561653187550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb73c979740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56165318d2f0 con 0x56165318a680 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb73c979740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5616530a0050 con 0x56165317fdc0 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb73c979740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56165318d470 con 0x56165317f4a0 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb73c979740 10 client.0 _ll_register_callbacks cb 0x5616530e0160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb738b3d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165318a680 0x561653187550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb738b3d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165318a680 0x561653187550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50870/0 (socket says 172.21.15.33:50870) 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb738b3d640 1 -- 172.21.15.33:0/751711185 learned_addr learned my addr 172.21.15.33:0/751711185 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb733fff640 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317f4a0 0x561653189d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb7337fe640 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165317fdc0 0x561653187010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb738b3d640 1 -- 172.21.15.33:0/751711185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317f4a0 msgr2=0x561653189d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:24.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb738b3d640 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317f4a0 0x561653189d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb738b3d640 1 -- 172.21.15.33:0/751711185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165317fdc0 msgr2=0x561653187010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:24.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb738b3d640 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165317fdc0 0x561653187010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb738b3d640 1 -- 172.21.15.33:0/751711185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561652e563c0 con 0x56165318a680 2024-09-16T00:21:24.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb738b3d640 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165318a680 0x561653187550 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb72c09ba80 tx=0x7fb72c09bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84130c34ecc2b7b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:24.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb7137fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:21:24.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb7137fe640 1 -- 172.21.15.33:0/751711185 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0b4060 con 0x56165318a680 2024-09-16T00:21:24.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb73c979740 1 -- 172.21.15.33:0/751711185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb724005020 con 0x56165318a680 2024-09-16T00:21:24.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.128+0000 7fb7137fe640 1 -- 172.21.15.33:0/751711185 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb72c086070 con 0x56165318a680 2024-09-16T00:21:24.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.129+0000 7fb7137fe640 1 -- 172.21.15.33:0/751711185 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb72c020040 con 0x56165318a680 2024-09-16T00:21:24.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.129+0000 7fb73c979740 10 client.16152 fetch_fsmap learned FSMap version 30 2024-09-16T00:21:24.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.129+0000 7fb73c979740 1 -- 172.21.15.33:0/751711185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561653198bb0 con 0x56165318a680 2024-09-16T00:21:24.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.129+0000 7fb7137fe640 1 -- 172.21.15.33:0/751711185 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb724005020 con 0x56165318a680 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.129+0000 7fb7137fe640 1 -- 172.21.15.33:0/751711185 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fb72c09af40 con 0x56165318a680 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.129+0000 7fb73c979740 10 client.16152 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.129+0000 7fb73c979740 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.129+0000 7fb73c979740 20 client.16152 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.129+0000 7fb73c979740 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561653199fc0 0x56165319c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.129+0000 7fb73c979740 4 client.16152 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.129+0000 7fb73c979740 1 -- 172.21.15.33:0/751711185 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616530f2850 con 0x561653199fc0 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.129+0000 7fb733fff640 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561653199fc0 0x56165319c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.130+0000 7fb733fff640 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561653199fc0 0x56165319c440 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19f884e6add8fc11 server_cookie=86db31af25cb3f75 in_seq=0 out_seq=0 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.130+0000 7fb7137fe640 10 client.16152 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.130+0000 7fb7137fe640 1 -- 172.21.15.33:0/751711185 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5616530f2850 con 0x561653199fc0 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.130+0000 7fb7137fe640 10 client.16152 handle_command_reply: tid=0 2024-09-16T00:21:24.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.130+0000 7fb73c979740 1 client.16152 shutdown 2024-09-16T00:21:24.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.131+0000 7fb73c979740 1 -- 172.21.15.33:0/751711185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165318a680 msgr2=0x561653187550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:24.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.131+0000 7fb73c979740 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165318a680 0x561653187550 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb72c09ba80 tx=0x7fb72c09bc30 comp rx=0 tx=0).stop 2024-09-16T00:21:24.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.131+0000 7fb73c979740 1 -- 172.21.15.33:0/751711185 shutdown_connections 2024-09-16T00:21:24.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.131+0000 7fb73c979740 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561653199fc0 0x56165319c440 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.131+0000 7fb73c979740 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165318a680 0x561653187550 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.131+0000 7fb73c979740 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165317fdc0 0x561653187010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.131+0000 7fb73c979740 1 --2- 172.21.15.33:0/751711185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317f4a0 0x561653189d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.131+0000 7fb7137fe640 0 client.16152 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:24.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.131+0000 7fb73c979740 1 -- 172.21.15.33:0/751711185 >> 172.21.15.33:0/751711185 conn(0x56165317ba30 msgr2=0x56165317ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:24.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.131+0000 7fb73c979740 1 -- 172.21.15.33:0/751711185 shutdown_connections 2024-09-16T00:21:24.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.131+0000 7fb73c979740 1 -- 172.21.15.33:0/751711185 wait complete. 2024-09-16T00:21:24.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.131+0000 7fb73c979740 20 client.16152 trim_cache size 0 max 16384 2024-09-16T00:21:24.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb73c979740 1 Processor -- start 2024-09-16T00:21:24.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb73c979740 1 -- start start 2024-09-16T00:21:24.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb73c979740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 0x5616531a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb73c979740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317fdc0 0x5616531a2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb73c979740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165318a680 0x56165318ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb73c979740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb7200128a0 con 0x56165317f4a0 2024-09-16T00:21:24.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb73c979740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb720010170 con 0x56165318a680 2024-09-16T00:21:24.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb73c979740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb72000dfc0 con 0x56165317fdc0 2024-09-16T00:21:24.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb733fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 0x5616531a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb7337fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317fdc0 0x5616531a2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb733fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 0x5616531a1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50880/0 (socket says 172.21.15.33:50880) 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb733fff640 1 -- 172.21.15.33:0/3877185944 learned_addr learned my addr 172.21.15.33:0/3877185944 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb738b3d640 1 --2- 172.21.15.33:0/3877185944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165318a680 0x56165318ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb7337fe640 1 -- 172.21.15.33:0/3877185944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165318a680 msgr2=0x56165318ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb7337fe640 1 --2- 172.21.15.33:0/3877185944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165318a680 0x56165318ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb7337fe640 1 -- 172.21.15.33:0/3877185944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 msgr2=0x5616531a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb7337fe640 1 --2- 172.21.15.33:0/3877185944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 0x5616531a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb7337fe640 1 -- 172.21.15.33:0/3877185944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56165317fbf0 con 0x56165317fdc0 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.149+0000 7fb733fff640 1 --2- 172.21.15.33:0/3877185944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 0x5616531a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.150+0000 7fb7337fe640 1 --2- 172.21.15.33:0/3877185944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317fdc0 0x5616531a2250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb72800aa30 tx=0x7fb72800a2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f46d6da7a6c235c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.150+0000 7fb719ffb640 1 -- 172.21.15.33:0/3877185944 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb728004df0 con 0x56165317fdc0 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.150+0000 7fb719ffb640 1 -- 172.21.15.33:0/3877185944 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7280119e0 con 0x56165317fdc0 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.150+0000 7fb719ffb640 1 -- 172.21.15.33:0/3877185944 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72801f7d0 con 0x56165317fdc0 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.151+0000 7fb73c979740 1 -- 172.21.15.33:0/3877185944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317fdc0 msgr2=0x5616531a2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:24.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.151+0000 7fb73c979740 1 --2- 172.21.15.33:0/3877185944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317fdc0 0x5616531a2250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb72800aa30 tx=0x7fb72800a2d0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.151+0000 7fb73c979740 1 -- 172.21.15.33:0/3877185944 shutdown_connections 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.151+0000 7fb73c979740 1 --2- 172.21.15.33:0/3877185944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165318a680 0x56165318ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.151+0000 7fb73c979740 1 --2- 172.21.15.33:0/3877185944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317fdc0 0x5616531a2250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.151+0000 7fb73c979740 1 --2- 172.21.15.33:0/3877185944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 0x5616531a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.151+0000 7fb73c979740 1 -- 172.21.15.33:0/3877185944 >> 172.21.15.33:0/3877185944 conn(0x56165317ba30 msgr2=0x5616531858e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.151+0000 7fb73c979740 1 -- 172.21.15.33:0/3877185944 shutdown_connections 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.151+0000 7fb73c979740 1 -- 172.21.15.33:0/3877185944 wait complete. 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.151+0000 7fb73c979740 1 -- start start 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.151+0000 7fb73c979740 1 Processor -- start 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb73c979740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 0x561653192780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb73c979740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317fdc0 0x561653192cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb73c979740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165318a680 0x56165318d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb73c979740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb72000e630 con 0x56165317f4a0 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb73c979740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb72000faa0 con 0x56165318a680 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb73c979740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb72000ed00 con 0x56165317fdc0 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb73c979740 10 client.0 _ll_register_callbacks cb 0x7fb72c09b450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb733fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 0x561653192780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb733fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 0x561653192780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50888/0 (socket says 172.21.15.33:50888) 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb7337fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317fdc0 0x561653192cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb733fff640 1 -- 172.21.15.33:0/294069545 learned_addr learned my addr 172.21.15.33:0/294069545 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb738b3d640 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165318a680 0x56165318d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb7337fe640 1 -- 172.21.15.33:0/294069545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165318a680 msgr2=0x56165318d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb7337fe640 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165318a680 0x56165318d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb7337fe640 1 -- 172.21.15.33:0/294069545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 msgr2=0x561653192780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb7337fe640 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 0x561653192780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb7337fe640 1 -- 172.21.15.33:0/294069545 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56165318a0d0 con 0x56165317fdc0 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb738b3d640 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165318a680 0x56165318d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb733fff640 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 0x561653192780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb7337fe640 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317fdc0 0x561653192cc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb728005840 tx=0x7fb728011fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5876080cf83e5c18 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.152+0000 7fb7137fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb7137fe640 1 -- 172.21.15.33:0/294069545 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb728037060 con 0x56165317fdc0 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb7137fe640 1 -- 172.21.15.33:0/294069545 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb728025040 con 0x56165317fdc0 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb73c979740 1 -- 172.21.15.33:0/294069545 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7280119e0 con 0x56165317fdc0 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb7137fe640 1 -- 172.21.15.33:0/294069545 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb72800a590 con 0x56165317fdc0 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb73c979740 10 client.25642 fetch_fsmap learned FSMap version 30 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb73c979740 1 -- 172.21.15.33:0/294069545 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561653204e20 con 0x56165317fdc0 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb7137fe640 1 -- 172.21.15.33:0/294069545 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7280119e0 con 0x56165317fdc0 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb7137fe640 1 -- 172.21.15.33:0/294069545 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fb728031070 con 0x56165317fdc0 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb73c979740 10 client.25642 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:21:24.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb73c979740 10 client.25642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:21:24.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb73c979740 20 client.25642 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:24.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb73c979740 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561653205570 0x561653207990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:24.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb73c979740 4 client.25642 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:21:24.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb73c979740 1 -- 172.21.15.33:0/294069545 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5616530ecd60 con 0x561653205570 2024-09-16T00:21:24.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.153+0000 7fb733fff640 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561653205570 0x561653207990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:24.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb733fff640 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561653205570 0x561653207990 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fd315fff74720c4 server_cookie=b1b0770c29f050ec in_seq=0 out_seq=0 2024-09-16T00:21:24.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb7137fe640 10 client.25642 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:24.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb7137fe640 1 -- 172.21.15.33:0/294069545 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5616530ecd60 con 0x561653205570 2024-09-16T00:21:24.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb7137fe640 10 client.25642 handle_command_reply: tid=0 2024-09-16T00:21:24.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb73c979740 1 client.25642 shutdown 2024-09-16T00:21:24.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb73c979740 1 -- 172.21.15.33:0/294069545 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317fdc0 msgr2=0x561653192cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:24.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb73c979740 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317fdc0 0x561653192cc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb728005840 tx=0x7fb728011fd0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb73c979740 1 -- 172.21.15.33:0/294069545 shutdown_connections 2024-09-16T00:21:24.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb73c979740 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561653205570 0x561653207990 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb73c979740 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56165318a680 0x56165318d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb73c979740 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56165317fdc0 0x561653192cc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb7137fe640 0 client.25642 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:24.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb73c979740 1 --2- 172.21.15.33:0/294069545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56165317f4a0 0x561653192780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb73c979740 1 -- 172.21.15.33:0/294069545 >> 172.21.15.33:0/294069545 conn(0x56165317ba30 msgr2=0x561653181710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:24.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb73c979740 1 -- 172.21.15.33:0/294069545 shutdown_connections 2024-09-16T00:21:24.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb73c979740 1 -- 172.21.15.33:0/294069545 wait complete. 2024-09-16T00:21:24.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.154+0000 7fb73c979740 20 client.25642 trim_cache size 0 max 16384 2024-09-16T00:21:24.162 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:21:24.162 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:21:24.162 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:21:24.162 INFO:teuthology.orchestra.run.smithi033.stdout: "1144472d-5661-40fc-8c30-f3531939ba80": { 2024-09-16T00:21:24.162 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:21:24.162 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "1144472d-5661-40fc-8c30-f3531939ba80", 2024-09-16T00:21:24.162 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:21:24.162 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:21:24.162 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:21:24.162 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:21:24.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.155+0000 7fb73a5a0640 1 -- 172.21.15.33:0/3837153611 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb71c0a32c0 msgr2=0x7fb71c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:24.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.155+0000 7fb73a5a0640 1 --2- 172.21.15.33:0/3837153611 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb71c0a32c0 0x7fb71c0a5780 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb7340feec0 tx=0x7fb728013880 comp rx=0 tx=0).stop 2024-09-16T00:21:24.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.155+0000 7fb73a5a0640 1 -- 172.21.15.33:0/3837153611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb734108290 msgr2=0x7fb73419cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:24.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.155+0000 7fb73a5a0640 1 --2- 172.21.15.33:0/3837153611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb734108290 0x7fb73419cea0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009fd0 tx=0x7fb72c00b260 comp rx=0 tx=0).stop 2024-09-16T00:21:24.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.155+0000 7fb73a5a0640 1 -- 172.21.15.33:0/3837153611 shutdown_connections 2024-09-16T00:21:24.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.155+0000 7fb73a5a0640 1 --2- 172.21.15.33:0/3837153611 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb71c0a32c0 0x7fb71c0a5780 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.155+0000 7fb73a5a0640 1 --2- 172.21.15.33:0/3837153611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb734108290 0x7fb73419cea0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.155+0000 7fb73a5a0640 1 --2- 172.21.15.33:0/3837153611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb734103740 0x7fb734198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.155+0000 7fb73a5a0640 1 --2- 172.21.15.33:0/3837153611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb734102d90 0x7fb7341985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:24.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.155+0000 7fb73a5a0640 1 -- 172.21.15.33:0/3837153611 >> 172.21.15.33:0/3837153611 conn(0x7fb7340fc7d0 msgr2=0x7fb734105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:24.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.155+0000 7fb73a5a0640 1 -- 172.21.15.33:0/3837153611 shutdown_connections 2024-09-16T00:21:24.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:24.155+0000 7fb73a5a0640 1 -- 172.21.15.33:0/3837153611 wait complete. 2024-09-16T00:21:24.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 206703 14.81 MB/sec execute 32 sec latency 36.715 ms 2024-09-16T00:21:24.933 INFO:tasks.cephfs.filesystem:scrub status for tag:1144472d-5661-40fc-8c30-f3531939ba80 - {'path': '/', 'tag': '1144472d-5661-40fc-8c30-f3531939ba80', 'options': 'recursive,force'} 2024-09-16T00:21:25.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:25 smithi033 ceph-mon[31243]: pgmap v478: 129 pgs: 129 active+clean; 98 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 327 op/s 2024-09-16T00:21:25.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:25 smithi083 ceph-mon[35242]: pgmap v478: 129 pgs: 129 active+clean; 98 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 327 op/s 2024-09-16T00:21:25.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:25 smithi060 ceph-mon[35185]: pgmap v478: 129 pgs: 129 active+clean; 98 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 327 op/s 2024-09-16T00:21:25.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 209821 14.91 MB/sec execute 33 sec latency 14.834 ms 2024-09-16T00:21:26.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 212248 14.84 MB/sec execute 34 sec latency 127.746 ms 2024-09-16T00:21:27.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:27 smithi033 ceph-mon[31243]: pgmap v479: 129 pgs: 129 active+clean; 101 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.2 MiB/s wr, 327 op/s 2024-09-16T00:21:27.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:27 smithi083 ceph-mon[35242]: pgmap v479: 129 pgs: 129 active+clean; 101 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.2 MiB/s wr, 327 op/s 2024-09-16T00:21:27.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:27 smithi060 ceph-mon[35185]: pgmap v479: 129 pgs: 129 active+clean; 101 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.2 MiB/s wr, 327 op/s 2024-09-16T00:21:27.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 214488 14.76 MB/sec execute 35 sec latency 30.868 ms 2024-09-16T00:21:28.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 217167 14.74 MB/sec execute 36 sec latency 14.081 ms 2024-09-16T00:21:29.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:29 smithi033 ceph-mon[31243]: pgmap v480: 129 pgs: 129 active+clean; 101 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 298 op/s 2024-09-16T00:21:29.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:29 smithi083 ceph-mon[35242]: pgmap v480: 129 pgs: 129 active+clean; 101 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 298 op/s 2024-09-16T00:21:29.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:29 smithi060 ceph-mon[35185]: pgmap v480: 129 pgs: 129 active+clean; 101 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 298 op/s 2024-09-16T00:21:29.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 220331 14.85 MB/sec execute 37 sec latency 21.235 ms 2024-09-16T00:21:30.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 223474 14.92 MB/sec execute 38 sec latency 12.559 ms 2024-09-16T00:21:31.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:31 smithi033 ceph-mon[31243]: pgmap v481: 129 pgs: 129 active+clean; 101 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 263 op/s 2024-09-16T00:21:31.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:31 smithi083 ceph-mon[35242]: pgmap v481: 129 pgs: 129 active+clean; 101 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 263 op/s 2024-09-16T00:21:31.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:31 smithi060 ceph-mon[35185]: pgmap v481: 129 pgs: 129 active+clean; 101 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 263 op/s 2024-09-16T00:21:31.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 225019 14.71 MB/sec execute 39 sec latency 92.378 ms 2024-09-16T00:21:32.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:32 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:21:32.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:32 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:21:32.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:32 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:21:32.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:32 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:21:32.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:32 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:21:32.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:32 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:21:32.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 227772 14.70 MB/sec execute 40 sec latency 34.581 ms 2024-09-16T00:21:33.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:33 smithi033 ceph-mon[31243]: pgmap v482: 129 pgs: 129 active+clean; 105 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 295 op/s 2024-09-16T00:21:33.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:33 smithi083 ceph-mon[35242]: pgmap v482: 129 pgs: 129 active+clean; 105 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 295 op/s 2024-09-16T00:21:33.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:33 smithi060 ceph-mon[35185]: pgmap v482: 129 pgs: 129 active+clean; 105 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 295 op/s 2024-09-16T00:21:33.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 230983 14.80 MB/sec execute 41 sec latency 42.540 ms 2024-09-16T00:21:34.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 234146 14.86 MB/sec execute 42 sec latency 15.700 ms 2024-09-16T00:21:35.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:35 smithi033 ceph-mon[31243]: pgmap v483: 129 pgs: 129 active+clean; 105 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 292 op/s 2024-09-16T00:21:35.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:35 smithi083 ceph-mon[35242]: pgmap v483: 129 pgs: 129 active+clean; 105 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 292 op/s 2024-09-16T00:21:35.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:35 smithi060 ceph-mon[35185]: pgmap v483: 129 pgs: 129 active+clean; 105 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 292 op/s 2024-09-16T00:21:35.654 INFO:tasks.workunit.client.0.smithi033.stdout: 1 236914 14.81 MB/sec execute 43 sec latency 12.298 ms 2024-09-16T00:21:36.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 238860 14.71 MB/sec execute 44 sec latency 90.891 ms 2024-09-16T00:21:37.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:37 smithi033 ceph-mon[31243]: pgmap v484: 129 pgs: 129 active+clean; 109 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.5 MiB/s wr, 331 op/s 2024-09-16T00:21:37.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:37 smithi083 ceph-mon[35242]: pgmap v484: 129 pgs: 129 active+clean; 109 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.5 MiB/s wr, 331 op/s 2024-09-16T00:21:37.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:37 smithi060 ceph-mon[35185]: pgmap v484: 129 pgs: 129 active+clean; 109 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.5 MiB/s wr, 331 op/s 2024-09-16T00:21:37.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 241915 14.76 MB/sec execute 45 sec latency 37.957 ms 2024-09-16T00:21:38.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 245043 14.85 MB/sec execute 46 sec latency 27.694 ms 2024-09-16T00:21:39.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:39 smithi033 ceph-mon[31243]: pgmap v485: 129 pgs: 129 active+clean; 110 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 249 op/s 2024-09-16T00:21:39.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:39 smithi083 ceph-mon[35242]: pgmap v485: 129 pgs: 129 active+clean; 110 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 249 op/s 2024-09-16T00:21:39.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:39 smithi060 ceph-mon[35185]: pgmap v485: 129 pgs: 129 active+clean; 110 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 249 op/s 2024-09-16T00:21:39.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 248113 14.85 MB/sec execute 47 sec latency 14.796 ms 2024-09-16T00:21:40.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 250718 14.78 MB/sec execute 48 sec latency 18.644 ms 2024-09-16T00:21:40.666 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 13.8 2024-09-16T00:21:40.667 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:21:40.940 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:41.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:41 smithi033 ceph-mon[31243]: pgmap v486: 129 pgs: 129 active+clean; 114 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 314 op/s 2024-09-16T00:21:41.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:41 smithi083 ceph-mon[35242]: pgmap v486: 129 pgs: 129 active+clean; 114 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 314 op/s 2024-09-16T00:21:41.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.436+0000 7efcf57da640 1 -- 172.21.15.33:0/320767073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcf0101470 msgr2=0x7efcf0101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:41.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.436+0000 7efcf57da640 1 --2- 172.21.15.33:0/320767073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcf0101470 0x7efcf0101850 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7efce0009900 tx=0x7efce002a410 comp rx=0 tx=0).stop 2024-09-16T00:21:41.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.436+0000 7efcf57da640 1 -- 172.21.15.33:0/320767073 shutdown_connections 2024-09-16T00:21:41.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.436+0000 7efcf57da640 1 --2- 172.21.15.33:0/320767073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcf0105de0 0x7efcf010d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.436+0000 7efcf57da640 1 --2- 172.21.15.33:0/320767073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf0101e20 0x7efcf01056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.436+0000 7efcf57da640 1 --2- 172.21.15.33:0/320767073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcf0101470 0x7efcf0101850 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.436+0000 7efcf57da640 1 -- 172.21.15.33:0/320767073 >> 172.21.15.33:0/320767073 conn(0x7efcf0077230 msgr2=0x7efcf00fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.436+0000 7efcf57da640 1 -- 172.21.15.33:0/320767073 shutdown_connections 2024-09-16T00:21:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.436+0000 7efcf57da640 1 -- 172.21.15.33:0/320767073 wait complete. 2024-09-16T00:21:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcf57da640 1 Processor -- start 2024-09-16T00:21:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcf57da640 1 -- start start 2024-09-16T00:21:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcf57da640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcf0101470 0x7efcf01970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcf57da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf0101e20 0x7efcf01975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcf57da640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcf0105de0 0x7efcf0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcf57da640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efcf0110e40 con 0x7efcf0105de0 2024-09-16T00:21:41.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcf57da640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efcf0110cc0 con 0x7efcf0101e20 2024-09-16T00:21:41.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcf57da640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efcf0110fc0 con 0x7efcf0101470 2024-09-16T00:21:41.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcef7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcf0105de0 0x7efcf0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:41.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efceeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcf0101470 0x7efcf01970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:41.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcef7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcf0105de0 0x7efcf0197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38478/0 (socket says 172.21.15.33:38478) 2024-09-16T00:21:41.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcef7fe640 1 -- 172.21.15.33:0/3791803577 learned_addr learned my addr 172.21.15.33:0/3791803577 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:41.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcee7fc640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf0101e20 0x7efcf01975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:41.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcef7fe640 1 -- 172.21.15.33:0/3791803577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcf0101470 msgr2=0x7efcf01970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:41.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcef7fe640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcf0101470 0x7efcf01970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:41.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcef7fe640 1 -- 172.21.15.33:0/3791803577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf0101e20 msgr2=0x7efcf01975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:41.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcef7fe640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf0101e20 0x7efcf01975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:41.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efcef7fe640 1 -- 172.21.15.33:0/3791803577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf01a6920 con 0x7efcf0105de0 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.437+0000 7efceeffd640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcf0101470 0x7efcf01970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.438+0000 7efcef7fe640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcf0105de0 0x7efcf0197b30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7efce800eb10 tx=0x7efce800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.438+0000 7efce7fff640 1 -- 172.21.15.33:0/3791803577 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce8004b20 con 0x7efcf0105de0 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.438+0000 7efcf57da640 1 -- 172.21.15.33:0/3791803577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf01a6c10 con 0x7efcf0105de0 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.438+0000 7efce7fff640 1 -- 172.21.15.33:0/3791803577 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efce8004cc0 con 0x7efcf0105de0 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.438+0000 7efce7fff640 1 -- 172.21.15.33:0/3791803577 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce8010760 con 0x7efcf0105de0 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.438+0000 7efcf57da640 1 -- 172.21.15.33:0/3791803577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf01a71a0 con 0x7efcf0105de0 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.439+0000 7efcf57da640 1 -- 172.21.15.33:0/3791803577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcf0101850 con 0x7efcf0105de0 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.440+0000 7efce7fff640 1 -- 172.21.15.33:0/3791803577 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7efce8005020 con 0x7efcf0105de0 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.441+0000 7efce7fff640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efcdc0a32c0 0x7efcdc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.441+0000 7efceeffd640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efcdc0a32c0 0x7efcdc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.441+0000 7efce7fff640 1 -- 172.21.15.33:0/3791803577 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7efce80d0410 con 0x7efcf0105de0 2024-09-16T00:21:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.441+0000 7efceeffd640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efcdc0a32c0 0x7efcdc0a5780 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7efce0009230 tx=0x7efce00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:41.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.445+0000 7efce7fff640 1 -- 172.21.15.33:0/3791803577 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7efce809ccb0 con 0x7efcf0105de0 2024-09-16T00:21:41.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:41 smithi060 ceph-mon[35185]: pgmap v486: 129 pgs: 129 active+clean; 114 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 314 op/s 2024-09-16T00:21:41.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 252813 14.74 MB/sec execute 49 sec latency 64.448 ms 2024-09-16T00:21:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.723+0000 7efcf57da640 1 -- 172.21.15.33:0/3791803577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcf002d7c0 con 0x7efcf0105de0 2024-09-16T00:21:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.724+0000 7efce7fff640 1 -- 172.21.15.33:0/3791803577 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7efce809c5b0 con 0x7efcf0105de0 2024-09-16T00:21:41.726 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:21:41.726 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:21:41.726 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:21:41.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.727+0000 7efcf57da640 1 -- 172.21.15.33:0/3791803577 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efcdc0a32c0 msgr2=0x7efcdc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:41.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.727+0000 7efcf57da640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efcdc0a32c0 0x7efcdc0a5780 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7efce0009230 tx=0x7efce00058d0 comp rx=0 tx=0).stop 2024-09-16T00:21:41.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.727+0000 7efcf57da640 1 -- 172.21.15.33:0/3791803577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcf0105de0 msgr2=0x7efcf0197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:41.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.727+0000 7efcf57da640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcf0105de0 0x7efcf0197b30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7efce800eb10 tx=0x7efce800c2d0 comp rx=0 tx=0).stop 2024-09-16T00:21:41.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.727+0000 7efcf57da640 1 -- 172.21.15.33:0/3791803577 shutdown_connections 2024-09-16T00:21:41.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.727+0000 7efcf57da640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efcdc0a32c0 0x7efcdc0a5780 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:41.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.727+0000 7efcf57da640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcf0105de0 0x7efcf0197b30 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:41.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.727+0000 7efcf57da640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf0101e20 0x7efcf01975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:41.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.727+0000 7efcf57da640 1 --2- 172.21.15.33:0/3791803577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcf0101470 0x7efcf01970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:41.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.727+0000 7efcf57da640 1 -- 172.21.15.33:0/3791803577 >> 172.21.15.33:0/3791803577 conn(0x7efcf0077230 msgr2=0x7efcf0104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:41.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.727+0000 7efcf57da640 1 -- 172.21.15.33:0/3791803577 shutdown_connections 2024-09-16T00:21:41.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:41.727+0000 7efcf57da640 1 -- 172.21.15.33:0/3791803577 wait complete. 2024-09-16T00:21:42.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:42 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3791803577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:21:42.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:42 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3791803577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:21:42.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:42 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3791803577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:21:42.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 255862 14.81 MB/sec execute 50 sec latency 37.873 ms 2024-09-16T00:21:42.937 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 133.8 --await-for 45.0' 2024-09-16T00:21:42.938 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 133.8 --await-for 45.0 2024-09-16T00:21:43.211 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:43.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:43 smithi033 ceph-mon[31243]: pgmap v487: 129 pgs: 129 active+clean; 114 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.2 MiB/s wr, 317 op/s 2024-09-16T00:21:43.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:43 smithi083 ceph-mon[35242]: pgmap v487: 129 pgs: 129 active+clean; 114 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.2 MiB/s wr, 317 op/s 2024-09-16T00:21:43.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:43 smithi060 ceph-mon[35185]: pgmap v487: 129 pgs: 129 active+clean; 114 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.2 MiB/s wr, 317 op/s 2024-09-16T00:21:43.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 258941 14.86 MB/sec execute 51 sec latency 63.560 ms 2024-09-16T00:21:43.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.697+0000 7fed00fa2640 1 -- 172.21.15.33:0/3711424519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecfc103720 msgr2=0x7fecfc10bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:43.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.697+0000 7fed00fa2640 1 --2- 172.21.15.33:0/3711424519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecfc103720 0x7fecfc10bc10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fece4009900 tx=0x7fece402a480 comp rx=0 tx=0).stop 2024-09-16T00:21:43.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.697+0000 7fed00fa2640 1 -- 172.21.15.33:0/3711424519 shutdown_connections 2024-09-16T00:21:43.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.697+0000 7fed00fa2640 1 --2- 172.21.15.33:0/3711424519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecfc10c1a0 0x7fecfc10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.697+0000 7fed00fa2640 1 --2- 172.21.15.33:0/3711424519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecfc103720 0x7fecfc10bc10 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.697+0000 7fed00fa2640 1 --2- 172.21.15.33:0/3711424519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecfc102e00 0x7fecfc1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.697+0000 7fed00fa2640 1 -- 172.21.15.33:0/3711424519 >> 172.21.15.33:0/3711424519 conn(0x7fecfc0fc7a0 msgr2=0x7fecfc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:43.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.697+0000 7fed00fa2640 1 -- 172.21.15.33:0/3711424519 shutdown_connections 2024-09-16T00:21:43.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.697+0000 7fed00fa2640 1 -- 172.21.15.33:0/3711424519 wait complete. 2024-09-16T00:21:43.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.698+0000 7fed00fa2640 1 Processor -- start 2024-09-16T00:21:43.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.698+0000 7fed00fa2640 1 -- start start 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fed00fa2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecfc102e00 0x7fecfc100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fed00fa2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecfc103720 0x7fecfc100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fed00fa2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecfc10c1a0 0x7fecfc1014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fed00fa2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc111f70 con 0x7fecfc103720 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fed00fa2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc111df0 con 0x7fecfc102e00 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fed00fa2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1120f0 con 0x7fecfc10c1a0 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fecfad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecfc10c1a0 0x7fecfc1014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fecf1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecfc103720 0x7fecfc100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fecfad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecfc10c1a0 0x7fecfc1014e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:49468/0 (socket says 172.21.15.33:49468) 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fecfad9d640 1 -- 172.21.15.33:0/3037137494 learned_addr learned my addr 172.21.15.33:0/3037137494 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fecf1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecfc103720 0x7fecfc100fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38486/0 (socket says 172.21.15.33:38486) 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fecfa59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecfc102e00 0x7fecfc100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fecf1d9b640 1 -- 172.21.15.33:0/3037137494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecfc10c1a0 msgr2=0x7fecfc1014e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fecf1d9b640 1 --2- 172.21.15.33:0/3037137494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecfc10c1a0 0x7fecfc1014e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fecf1d9b640 1 -- 172.21.15.33:0/3037137494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecfc102e00 msgr2=0x7fecfc100a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fecf1d9b640 1 --2- 172.21.15.33:0/3037137494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecfc102e00 0x7fecfc100a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fecf1d9b640 1 -- 172.21.15.33:0/3037137494 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecfc1a6900 con 0x7fecfc103720 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.699+0000 7fecf1d9b640 1 --2- 172.21.15.33:0/3037137494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecfc103720 0x7fecfc100fa0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fece4005940 tx=0x7fece40040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.700+0000 7fecf3fff640 1 -- 172.21.15.33:0/3037137494 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece403f030 con 0x7fecfc103720 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.700+0000 7fed00fa2640 1 -- 172.21.15.33:0/3037137494 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecfc1a6b90 con 0x7fecfc103720 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.700+0000 7fecf3fff640 1 -- 172.21.15.33:0/3037137494 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fece40044c0 con 0x7fecfc103720 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.700+0000 7fecf3fff640 1 -- 172.21.15.33:0/3037137494 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4035070 con 0x7fecfc103720 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.700+0000 7fed00fa2640 1 -- 172.21.15.33:0/3037137494 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecfc1a7070 con 0x7fecfc103720 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.702+0000 7fecf3fff640 1 -- 172.21.15.33:0/3037137494 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fece4042030 con 0x7fecfc103720 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.702+0000 7fed0337b740 1 Processor -- start 2024-09-16T00:21:43.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.702+0000 7fed0337b740 1 -- start start 2024-09-16T00:21:43.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fed0337b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f80760 0x5646f3f80b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fed0337b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f81080 0x5646f3f8b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fed0337b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f8b940 0x5646f3f8dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fed0337b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5646f3ee62a0 con 0x5646f3f8b940 2024-09-16T00:21:43.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fed0337b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5646f3e8db10 con 0x5646f3f80760 2024-09-16T00:21:43.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fed0337b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5646f3ee2260 con 0x5646f3f81080 2024-09-16T00:21:43.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecf1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f8b940 0x5646f3f8dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f80760 0x5646f3f80b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfa59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f81080 0x5646f3f8b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f80760 0x5646f3f80b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:57392/0 (socket says 172.21.15.33:57392) 2024-09-16T00:21:43.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfad9d640 1 -- 172.21.15.33:0/1958978031 learned_addr learned my addr 172.21.15.33:0/1958978031 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:43.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfa59c640 1 --2- 172.21.15.33:0/1958978031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f81080 0x5646f3f8b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:49484/0 (socket says 172.21.15.33:49484) 2024-09-16T00:21:43.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecf1d9b640 1 --2- 172.21.15.33:0/1958978031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f8b940 0x5646f3f8dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38490/0 (socket says 172.21.15.33:38490) 2024-09-16T00:21:43.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecf3fff640 1 --2- 172.21.15.33:0/3037137494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fece80a3240 0x7fece80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfa59c640 1 -- 172.21.15.33:0/1958978031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f80760 msgr2=0x5646f3f80b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:43.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfa59c640 1 --2- 172.21.15.33:0/1958978031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f80760 0x5646f3f80b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfa59c640 1 -- 172.21.15.33:0/1958978031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f8b940 msgr2=0x5646f3f8dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:43.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfa59c640 1 --2- 172.21.15.33:0/1958978031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f8b940 0x5646f3f8dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfa59c640 1 -- 172.21.15.33:0/1958978031 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646f3c57b90 con 0x5646f3f81080 2024-09-16T00:21:43.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfad9d640 1 --2- 172.21.15.33:0/1958978031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f80760 0x5646f3f80b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:43.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfad9d640 1 --2- 172.21.15.33:0/3037137494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fece80a3240 0x7fece80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecf3fff640 1 -- 172.21.15.33:0/3037137494 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fece40f0020 con 0x7fecfc103720 2024-09-16T00:21:43.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.703+0000 7fecfa59c640 1 --2- 172.21.15.33:0/1958978031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f81080 0x5646f3f8b400 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fecec009c00 tx=0x7fecec011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e71fbe830dadda14 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:43.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fecf0d99640 1 -- 172.21.15.33:0/1958978031 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec004310 con 0x5646f3f81080 2024-09-16T00:21:43.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fecfad9d640 1 --2- 172.21.15.33:0/3037137494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fece80a3240 0x7fece80a5700 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fecf4010850 tx=0x7fecf4004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:43.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fecf0d99640 1 -- 172.21.15.33:0/1958978031 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecec0044b0 con 0x5646f3f81080 2024-09-16T00:21:43.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fecf0d99640 1 -- 172.21.15.33:0/1958978031 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec003850 con 0x5646f3f81080 2024-09-16T00:21:43.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fed0337b740 1 -- 172.21.15.33:0/1958978031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f81080 msgr2=0x5646f3f8b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:43.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fed0337b740 1 --2- 172.21.15.33:0/1958978031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f81080 0x5646f3f8b400 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fecec009c00 tx=0x7fecec011440 comp rx=0 tx=0).stop 2024-09-16T00:21:43.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fed0337b740 1 -- 172.21.15.33:0/1958978031 shutdown_connections 2024-09-16T00:21:43.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fed0337b740 1 --2- 172.21.15.33:0/1958978031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f8b940 0x5646f3f8dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fed0337b740 1 --2- 172.21.15.33:0/1958978031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f81080 0x5646f3f8b400 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fed0337b740 1 --2- 172.21.15.33:0/1958978031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f80760 0x5646f3f80b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fed0337b740 1 -- 172.21.15.33:0/1958978031 >> 172.21.15.33:0/1958978031 conn(0x5646f3f7ccf0 msgr2=0x5646f3f7d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:43.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fed0337b740 1 -- 172.21.15.33:0/1958978031 shutdown_connections 2024-09-16T00:21:43.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.704+0000 7fed0337b740 1 -- 172.21.15.33:0/1958978031 wait complete. 2024-09-16T00:21:43.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fed0337b740 1 -- start start 2024-09-16T00:21:43.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fed0337b740 1 Processor -- start 2024-09-16T00:21:43.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fed0337b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f80760 0x5646f3f88840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fed0337b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f81080 0x5646f3f88d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fed0337b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f8b940 0x5646f3f892c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fed0337b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5646f3f8e510 con 0x5646f3f80760 2024-09-16T00:21:43.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fed0337b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5646f3e6e6e0 con 0x5646f3f81080 2024-09-16T00:21:43.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fed0337b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5646f3f8e690 con 0x5646f3f8b940 2024-09-16T00:21:43.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fed0337b740 10 client.0 _ll_register_callbacks cb 0x5646f3e92a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:43.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fecfa59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f80760 0x5646f3f88840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fecfad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f8b940 0x5646f3f892c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fecfa59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f80760 0x5646f3f88840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38492/0 (socket says 172.21.15.33:38492) 2024-09-16T00:21:43.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fecfa59c640 1 -- 172.21.15.33:0/1569152498 learned_addr learned my addr 172.21.15.33:0/1569152498 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:43.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.705+0000 7fecf1d9b640 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f81080 0x5646f3f88d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fecfad9d640 1 -- 172.21.15.33:0/1569152498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f81080 msgr2=0x5646f3f88d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:43.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fecfad9d640 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f81080 0x5646f3f88d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fecfad9d640 1 -- 172.21.15.33:0/1569152498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f80760 msgr2=0x5646f3f88840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:43.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fecfad9d640 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f80760 0x5646f3f88840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fecfad9d640 1 -- 172.21.15.33:0/1569152498 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646f3f97a30 con 0x5646f3f8b940 2024-09-16T00:21:43.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fecfa59c640 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f80760 0x5646f3f88840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:43.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fecfad9d640 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f8b940 0x5646f3f892c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fecf401f6f0 tx=0x7fecf4004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=456933c265e689df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:43.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fece1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:21:43.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fece1ffb640 1 -- 172.21.15.33:0/1569152498 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf401d450 con 0x5646f3f8b940 2024-09-16T00:21:43.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fece1ffb640 1 -- 172.21.15.33:0/1569152498 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fecf401d5f0 con 0x5646f3f8b940 2024-09-16T00:21:43.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fed0337b740 1 -- 172.21.15.33:0/1569152498 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecec0044b0 con 0x5646f3f8b940 2024-09-16T00:21:43.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fece1ffb640 1 -- 172.21.15.33:0/1569152498 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fecf401d7f0 con 0x5646f3f8b940 2024-09-16T00:21:43.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fed0337b740 10 client.25660 fetch_fsmap learned FSMap version 30 2024-09-16T00:21:43.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.706+0000 7fed0337b740 1 -- 172.21.15.33:0/1569152498 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5646f3f9a400 con 0x5646f3f8b940 2024-09-16T00:21:43.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.707+0000 7fece1ffb640 1 -- 172.21.15.33:0/1569152498 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecec0044b0 con 0x5646f3f8b940 2024-09-16T00:21:43.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.707+0000 7fece1ffb640 1 -- 172.21.15.33:0/1569152498 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fecf401c970 con 0x5646f3f8b940 2024-09-16T00:21:43.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.707+0000 7fed0337b740 10 client.25660 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:21:43.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.707+0000 7fed0337b740 10 client.25660 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.707+0000 7fed0337b740 20 client.25660 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.707+0000 7fed0337b740 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646f3f9b8c0 0x5646f3f9dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.707+0000 7fed0337b740 4 client.25660 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.707+0000 7fed0337b740 1 -- 172.21.15.33:0/1569152498 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646f3ef39d0 con 0x5646f3f9b8c0 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.707+0000 7fecfa59c640 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646f3f9b8c0 0x5646f3f9dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.707+0000 7fece1ffb640 10 client.25660 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.707+0000 7fecfa59c640 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646f3f9b8c0 0x5646f3f9dd30 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30db69d8e6975924 server_cookie=91470014d39325a2 in_seq=0 out_seq=0 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fece1ffb640 1 -- 172.21.15.33:0/1569152498 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5646f3ef39d0 con 0x5646f3f9b8c0 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fece1ffb640 10 client.25660 handle_command_reply: tid=0 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fed0337b740 1 client.25660 shutdown 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fed0337b740 1 -- 172.21.15.33:0/1569152498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f8b940 msgr2=0x5646f3f892c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fed0337b740 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f8b940 0x5646f3f892c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fecf401f6f0 tx=0x7fecf4004ab0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fed0337b740 1 -- 172.21.15.33:0/1569152498 shutdown_connections 2024-09-16T00:21:43.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fed0337b740 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646f3f9b8c0 0x5646f3f9dd30 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fed0337b740 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f8b940 0x5646f3f892c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fed0337b740 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f81080 0x5646f3f88d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fece1ffb640 0 client.25660 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:43.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fed0337b740 1 --2- 172.21.15.33:0/1569152498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f80760 0x5646f3f88840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fed0337b740 1 -- 172.21.15.33:0/1569152498 >> 172.21.15.33:0/1569152498 conn(0x5646f3f7ccf0 msgr2=0x5646f3f829d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:43.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fed0337b740 1 -- 172.21.15.33:0/1569152498 shutdown_connections 2024-09-16T00:21:43.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.708+0000 7fed0337b740 1 -- 172.21.15.33:0/1569152498 wait complete. 2024-09-16T00:21:43.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.709+0000 7fed0337b740 20 client.25660 trim_cache size 0 max 16384 2024-09-16T00:21:43.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.726+0000 7fed0337b740 1 Processor -- start 2024-09-16T00:21:43.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.726+0000 7fed0337b740 1 -- start start 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fed0337b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f80760 0x5646f3fa3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fed0337b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 0x5646f3fa3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fed0337b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f8b940 0x5646f3f821b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fed0337b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1b3b90 con 0x5646f3f81080 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fed0337b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1b1460 con 0x5646f3f8b940 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fed0337b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1af2b0 con 0x5646f3f80760 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fecf1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 0x5646f3fa3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fecf1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 0x5646f3fa3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38502/0 (socket says 172.21.15.33:38502) 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fecf1d9b640 1 -- 172.21.15.33:0/3248654623 learned_addr learned my addr 172.21.15.33:0/3248654623 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fecfa59c640 1 --2- 172.21.15.33:0/3248654623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f80760 0x5646f3fa3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fecfad9d640 1 --2- 172.21.15.33:0/3248654623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f8b940 0x5646f3f821b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fecfa59c640 1 -- 172.21.15.33:0/3248654623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f8b940 msgr2=0x5646f3f821b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:43.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fecfa59c640 1 --2- 172.21.15.33:0/3248654623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f8b940 0x5646f3f821b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fecfa59c640 1 -- 172.21.15.33:0/3248654623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 msgr2=0x5646f3fa3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:43.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fecfa59c640 1 --2- 172.21.15.33:0/3248654623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 0x5646f3fa3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fecfa59c640 1 -- 172.21.15.33:0/3248654623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646f3f877f0 con 0x5646f3f80760 2024-09-16T00:21:43.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fecfad9d640 1 --2- 172.21.15.33:0/3248654623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f8b940 0x5646f3f821b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:21:43.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.727+0000 7fecfa59c640 1 --2- 172.21.15.33:0/3248654623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f80760 0x5646f3fa3650 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fecec004d10 tx=0x7fecec011a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e15cf33bb5597b00 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:43.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.728+0000 7fece3fff640 1 -- 172.21.15.33:0/3248654623 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec01e040 con 0x5646f3f80760 2024-09-16T00:21:43.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.728+0000 7fece3fff640 1 -- 172.21.15.33:0/3248654623 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecec01f040 con 0x5646f3f80760 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.728+0000 7fece3fff640 1 -- 172.21.15.33:0/3248654623 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec003850 con 0x5646f3f80760 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.728+0000 7fed0337b740 1 -- 172.21.15.33:0/3248654623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f80760 msgr2=0x5646f3fa3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.728+0000 7fed0337b740 1 --2- 172.21.15.33:0/3248654623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f80760 0x5646f3fa3650 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fecec004d10 tx=0x7fecec011a70 comp rx=0 tx=0).stop 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 -- 172.21.15.33:0/3248654623 shutdown_connections 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 --2- 172.21.15.33:0/3248654623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f8b940 0x5646f3f821b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 --2- 172.21.15.33:0/3248654623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 0x5646f3fa3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 --2- 172.21.15.33:0/3248654623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f80760 0x5646f3fa3650 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 -- 172.21.15.33:0/3248654623 >> 172.21.15.33:0/3248654623 conn(0x5646f3f7ccf0 msgr2=0x5646f3f87fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 -- 172.21.15.33:0/3248654623 shutdown_connections 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 -- 172.21.15.33:0/3248654623 wait complete. 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 -- start start 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 Processor -- start 2024-09-16T00:21:43.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f80760 0x5646f3f94010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 0x5646f3f8f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f8b940 0x5646f3f8f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1af920 con 0x5646f3f81080 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1b0d90 con 0x5646f3f80760 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1afff0 con 0x5646f3f8b940 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fed0337b740 10 client.0 _ll_register_callbacks cb 0x7fecf400fd60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fecf1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 0x5646f3f8f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fecf1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 0x5646f3f8f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38516/0 (socket says 172.21.15.33:38516) 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fecf1d9b640 1 -- 172.21.15.33:0/3887962315 learned_addr learned my addr 172.21.15.33:0/3887962315 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.729+0000 7fecfad9d640 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f8b940 0x5646f3f8f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fecfa59c640 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f80760 0x5646f3f94010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fecf1d9b640 1 -- 172.21.15.33:0/3887962315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f8b940 msgr2=0x5646f3f8f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:43.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fecf1d9b640 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f8b940 0x5646f3f8f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fecf1d9b640 1 -- 172.21.15.33:0/3887962315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f80760 msgr2=0x5646f3f94010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fecf1d9b640 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f80760 0x5646f3f94010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fecf1d9b640 1 -- 172.21.15.33:0/3887962315 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646f3f900b0 con 0x5646f3f81080 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fecf1d9b640 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 0x5646f3f8f340 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fece40bbf70 tx=0x7fece4009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6be9a563c397f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fece1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fece1ffb640 1 -- 172.21.15.33:0/3887962315 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4049070 con 0x5646f3f81080 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fece1ffb640 1 -- 172.21.15.33:0/3887962315 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fece40ba570 con 0x5646f3f81080 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fed0337b740 1 -- 172.21.15.33:0/3887962315 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecec01f040 con 0x5646f3f81080 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fece1ffb640 1 -- 172.21.15.33:0/3887962315 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fece40bbbd0 con 0x5646f3f81080 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fed0337b740 10 client.16182 fetch_fsmap learned FSMap version 30 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.730+0000 7fed0337b740 1 -- 172.21.15.33:0/3887962315 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5646f40065f0 con 0x5646f3f81080 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.731+0000 7fece1ffb640 1 -- 172.21.15.33:0/3887962315 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecec01f040 con 0x5646f3f81080 2024-09-16T00:21:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.731+0000 7fece1ffb640 1 -- 172.21.15.33:0/3887962315 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fece40dc0b0 con 0x5646f3f81080 2024-09-16T00:21:43.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.731+0000 7fed0337b740 10 client.16182 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:21:43.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.731+0000 7fed0337b740 10 client.16182 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:21:43.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.731+0000 7fed0337b740 20 client.16182 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:43.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.731+0000 7fed0337b740 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646f4006d80 0x5646f40091a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:43.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.731+0000 7fed0337b740 4 client.16182 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 133.8, "await_for": 45.0}] 2024-09-16T00:21:43.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.731+0000 7fed0337b740 1 -- 172.21.15.33:0/3887962315 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 133.8, "await_for": 45.0}) -- 0x5646f3eedd20 con 0x5646f4006d80 2024-09-16T00:21:43.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.731+0000 7fecfa59c640 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646f4006d80 0x5646f40091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:43.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.731+0000 7fece1ffb640 10 client.16182 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:43.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.731+0000 7fecfa59c640 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646f4006d80 0x5646f40091a0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26e958d0a7e708fb server_cookie=888c286f3139a6e in_seq=0 out_seq=0 2024-09-16T00:21:43.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.825+0000 7fece1ffb640 1 -- 172.21.15.33:0/3887962315 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5646f3eedd20 con 0x5646f4006d80 2024-09-16T00:21:43.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.825+0000 7fece1ffb640 10 client.16182 handle_command_reply: tid=0 2024-09-16T00:21:43.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.825+0000 7fed0337b740 1 client.16182 shutdown 2024-09-16T00:21:43.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.825+0000 7fed0337b740 1 -- 172.21.15.33:0/3887962315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 msgr2=0x5646f3f8f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:43.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.825+0000 7fed0337b740 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 0x5646f3f8f340 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fece40bbf70 tx=0x7fece4009620 comp rx=0 tx=0).stop 2024-09-16T00:21:43.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.825+0000 7fed0337b740 1 -- 172.21.15.33:0/3887962315 shutdown_connections 2024-09-16T00:21:43.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.825+0000 7fed0337b740 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646f4006d80 0x5646f40091a0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.825+0000 7fed0337b740 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646f3f8b940 0x5646f3f8f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.825+0000 7fed0337b740 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646f3f81080 0x5646f3f8f340 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.825+0000 7fed0337b740 1 --2- 172.21.15.33:0/3887962315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646f3f80760 0x5646f3f94010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.825+0000 7fed0337b740 1 -- 172.21.15.33:0/3887962315 >> 172.21.15.33:0/3887962315 conn(0x5646f3f7ccf0 msgr2=0x5646f3f8a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:43.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.825+0000 7fece1ffb640 0 client.16182 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:43.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.826+0000 7fed0337b740 1 -- 172.21.15.33:0/3887962315 shutdown_connections 2024-09-16T00:21:43.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.826+0000 7fed0337b740 1 -- 172.21.15.33:0/3887962315 wait complete. 2024-09-16T00:21:43.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.826+0000 7fed0337b740 20 client.16182 trim_cache size 0 max 16384 2024-09-16T00:21:43.829 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:21:43.829 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:21:43.829 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:21:43.829 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 14, 2024-09-16T00:21:43.829 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:21:43.829 INFO:teuthology.orchestra.run.smithi033.stdout: "52b54352": { 2024-09-16T00:21:43.829 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 14, 2024-09-16T00:21:43.829 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:21:43.829 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:21:43.829 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:21:43.829 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 133.8, 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:21:43.830 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:21:43.831 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:21:43.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.826+0000 7fed00fa2640 1 -- 172.21.15.33:0/3037137494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fece80a3240 msgr2=0x7fece80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:43.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.826+0000 7fed00fa2640 1 --2- 172.21.15.33:0/3037137494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fece80a3240 0x7fece80a5700 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fecf4010850 tx=0x7fecf4004a40 comp rx=0 tx=0).stop 2024-09-16T00:21:43.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.826+0000 7fed00fa2640 1 -- 172.21.15.33:0/3037137494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecfc103720 msgr2=0x7fecfc100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:43.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.826+0000 7fed00fa2640 1 --2- 172.21.15.33:0/3037137494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecfc103720 0x7fecfc100fa0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fece4005940 tx=0x7fece40040c0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.827+0000 7fed00fa2640 1 -- 172.21.15.33:0/3037137494 shutdown_connections 2024-09-16T00:21:43.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.827+0000 7fed00fa2640 1 --2- 172.21.15.33:0/3037137494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fece80a3240 0x7fece80a5700 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.827+0000 7fed00fa2640 1 --2- 172.21.15.33:0/3037137494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecfc10c1a0 0x7fecfc1014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.827+0000 7fed00fa2640 1 --2- 172.21.15.33:0/3037137494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecfc103720 0x7fecfc100fa0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.827+0000 7fed00fa2640 1 --2- 172.21.15.33:0/3037137494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecfc102e00 0x7fecfc100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:43.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.827+0000 7fed00fa2640 1 -- 172.21.15.33:0/3037137494 >> 172.21.15.33:0/3037137494 conn(0x7fecfc0fc7a0 msgr2=0x7fecfc0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:43.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.827+0000 7fed00fa2640 1 -- 172.21.15.33:0/3037137494 shutdown_connections 2024-09-16T00:21:43.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:43.827+0000 7fed00fa2640 1 -- 172.21.15.33:0/3037137494 wait complete. 2024-09-16T00:21:44.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 14.59 MB/sec execute 52 sec latency 828.072 ms 2024-09-16T00:21:44.708 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '52b54352', quiesce took 4.0 seconds. Will release after: 9.8 2024-09-16T00:21:45.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:45 smithi033 ceph-mon[31243]: pgmap v488: 129 pgs: 129 active+clean; 117 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 313 op/s 2024-09-16T00:21:45.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:45 smithi083 ceph-mon[35242]: pgmap v488: 129 pgs: 129 active+clean; 117 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 313 op/s 2024-09-16T00:21:45.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:45 smithi060 ceph-mon[35185]: pgmap v488: 129 pgs: 129 active+clean; 117 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 313 op/s 2024-09-16T00:21:45.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 14.31 MB/sec execute 53 sec latency 1828.146 ms 2024-09-16T00:21:46.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 14.05 MB/sec execute 54 sec latency 2828.256 ms 2024-09-16T00:21:47.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:47 smithi083 ceph-mon[35242]: pgmap v489: 129 pgs: 129 active+clean; 117 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 313 op/s 2024-09-16T00:21:47.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:47 smithi060 ceph-mon[35185]: pgmap v489: 129 pgs: 129 active+clean; 117 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 313 op/s 2024-09-16T00:21:47.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:47 smithi033 ceph-mon[31243]: pgmap v489: 129 pgs: 129 active+clean; 117 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 313 op/s 2024-09-16T00:21:47.655 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 13.79 MB/sec execute 55 sec latency 3828.311 ms 2024-09-16T00:21:48.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 13.54 MB/sec execute 56 sec latency 4828.412 ms 2024-09-16T00:21:49.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:49 smithi083 ceph-mon[35242]: pgmap v490: 129 pgs: 129 active+clean; 117 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 221 op/s 2024-09-16T00:21:49.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:49 smithi060 ceph-mon[35185]: pgmap v490: 129 pgs: 129 active+clean; 117 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 221 op/s 2024-09-16T00:21:49.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:49 smithi033 ceph-mon[31243]: pgmap v490: 129 pgs: 129 active+clean; 117 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 221 op/s 2024-09-16T00:21:49.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 13.31 MB/sec execute 57 sec latency 5828.525 ms 2024-09-16T00:21:50.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 13.08 MB/sec execute 58 sec latency 6828.649 ms 2024-09-16T00:21:51.380 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:51 smithi033 ceph-mon[31243]: pgmap v491: 129 pgs: 129 active+clean; 117 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 246 op/s 2024-09-16T00:21:51.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:51 smithi083 ceph-mon[35242]: pgmap v491: 129 pgs: 129 active+clean; 117 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 246 op/s 2024-09-16T00:21:51.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:51 smithi060 ceph-mon[35185]: pgmap v491: 129 pgs: 129 active+clean; 117 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 246 op/s 2024-09-16T00:21:51.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 12.85 MB/sec execute 59 sec latency 7828.769 ms 2024-09-16T00:21:52.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 12.64 MB/sec execute 60 sec latency 8828.806 ms 2024-09-16T00:21:53.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:53 smithi083 ceph-mon[35242]: pgmap v492: 129 pgs: 129 active+clean; 116 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 190 op/s 2024-09-16T00:21:53.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:53 smithi060 ceph-mon[35185]: pgmap v492: 129 pgs: 129 active+clean; 116 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 190 op/s 2024-09-16T00:21:53.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:53 smithi033 ceph-mon[31243]: pgmap v492: 129 pgs: 129 active+clean; 116 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 190 op/s 2024-09-16T00:21:53.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 12.43 MB/sec execute 61 sec latency 9828.851 ms 2024-09-16T00:21:54.509 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:21:54.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 12.23 MB/sec execute 62 sec latency 10828.905 ms 2024-09-16T00:21:54.783 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:54.934 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:21:55.188 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:55 smithi033 ceph-mon[31243]: pgmap v493: 129 pgs: 129 active+clean; 115 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 174 op/s 2024-09-16T00:21:55.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.269+0000 7f4700136640 1 -- 172.21.15.33:0/1876190044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46f81082a0 msgr2=0x7f46f810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:55.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.269+0000 7f4700136640 1 --2- 172.21.15.33:0/1876190044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46f81082a0 0x7f46f810ec40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009cd0 tx=0x7f46f002a910 comp rx=0 tx=0).stop 2024-09-16T00:21:55.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.269+0000 7f4700136640 1 -- 172.21.15.33:0/1876190044 shutdown_connections 2024-09-16T00:21:55.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.269+0000 7f4700136640 1 --2- 172.21.15.33:0/1876190044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46f81082a0 0x7f46f810ec40 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:55.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.269+0000 7f4700136640 1 --2- 172.21.15.33:0/1876190044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46f81036e0 0x7f46f8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:55.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.269+0000 7f4700136640 1 --2- 172.21.15.33:0/1876190044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46f8102d30 0x7f46f8103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:55.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.269+0000 7f4700136640 1 -- 172.21.15.33:0/1876190044 >> 172.21.15.33:0/1876190044 conn(0x7f46f80fc7d0 msgr2=0x7f46f80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:55.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.269+0000 7f4700136640 1 -- 172.21.15.33:0/1876190044 shutdown_connections 2024-09-16T00:21:55.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.269+0000 7f4700136640 1 -- 172.21.15.33:0/1876190044 wait complete. 2024-09-16T00:21:55.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f4700136640 1 Processor -- start 2024-09-16T00:21:55.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f4700136640 1 -- start start 2024-09-16T00:21:55.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f4700136640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46f8102d30 0x7f46f8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:55.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f4700136640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46f81036e0 0x7f46f8198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:55.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f4700136640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46f81082a0 0x7f46f819d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:55.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f4700136640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f46f8112160 con 0x7f46f81082a0 2024-09-16T00:21:55.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f4700136640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f46f8111fe0 con 0x7f46f81036e0 2024-09-16T00:21:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f4700136640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f46f81122e0 con 0x7f46f8102d30 2024-09-16T00:21:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f46fe6d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46f81082a0 0x7f46f819d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f46fe6d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46f81082a0 0x7f46f819d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51098/0 (socket says 172.21.15.33:51098) 2024-09-16T00:21:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f46fe6d3640 1 -- 172.21.15.33:0/3068577187 learned_addr learned my addr 172.21.15.33:0/3068577187 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f46fded2640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46f8102d30 0x7f46f8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f46fd6d1640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46f81036e0 0x7f46f8198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f46fe6d3640 1 -- 172.21.15.33:0/3068577187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46f8102d30 msgr2=0x7f46f8198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f46fe6d3640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46f8102d30 0x7f46f8198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.270+0000 7f46fe6d3640 1 -- 172.21.15.33:0/3068577187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46f81036e0 msgr2=0x7f46f8198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.271+0000 7f46fe6d3640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46f81036e0 0x7f46f8198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.271+0000 7f46fe6d3640 1 -- 172.21.15.33:0/3068577187 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f819d6f0 con 0x7f46f81082a0 2024-09-16T00:21:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.271+0000 7f46fe6d3640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46f81082a0 0x7f46f819d010 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f46f0005cc0 tx=0x7f46f0031460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:55.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.271+0000 7f46eaffd640 1 -- 172.21.15.33:0/3068577187 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0031850 con 0x7f46f81082a0 2024-09-16T00:21:55.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.271+0000 7f46eaffd640 1 -- 172.21.15.33:0/3068577187 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46f0004520 con 0x7f46f81082a0 2024-09-16T00:21:55.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.271+0000 7f46eaffd640 1 -- 172.21.15.33:0/3068577187 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f002fd90 con 0x7f46f81082a0 2024-09-16T00:21:55.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.271+0000 7f4700136640 1 -- 172.21.15.33:0/3068577187 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f819d980 con 0x7f46f81082a0 2024-09-16T00:21:55.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.271+0000 7f4700136640 1 -- 172.21.15.33:0/3068577187 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f81a52c0 con 0x7f46f81082a0 2024-09-16T00:21:55.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.272+0000 7f4700136640 1 -- 172.21.15.33:0/3068577187 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f8103190 con 0x7f46f81082a0 2024-09-16T00:21:55.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.273+0000 7f46eaffd640 1 -- 172.21.15.33:0/3068577187 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f46f00046d0 con 0x7f46f81082a0 2024-09-16T00:21:55.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.274+0000 7f46eaffd640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f46e40a32c0 0x7f46e40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:55.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.274+0000 7f46fded2640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f46e40a32c0 0x7f46e40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:55.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.274+0000 7f46eaffd640 1 -- 172.21.15.33:0/3068577187 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f46f00f13b0 con 0x7f46f81082a0 2024-09-16T00:21:55.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.274+0000 7f46fded2640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f46e40a32c0 0x7f46e40a5780 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f46ec009890 tx=0x7f46ec008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:55.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.278+0000 7f46eaffd640 1 -- 172.21.15.33:0/3068577187 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f46f00bdc50 con 0x7f46f81082a0 2024-09-16T00:21:55.384 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:55.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:55 smithi083 ceph-mon[35242]: pgmap v493: 129 pgs: 129 active+clean; 115 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 174 op/s 2024-09-16T00:21:55.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:55 smithi060 ceph-mon[35185]: pgmap v493: 129 pgs: 129 active+clean; 115 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 174 op/s 2024-09-16T00:21:55.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.509+0000 7f4700136640 1 -- 172.21.15.33:0/3068577187 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46f810a2b0 con 0x7f46f81082a0 2024-09-16T00:21:55.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.509+0000 7f46eaffd640 1 -- 172.21.15.33:0/3068577187 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f46f00bd550 con 0x7f46f81082a0 2024-09-16T00:21:55.511 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:21:55.511 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:21:55.511 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:21:55.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.511+0000 7f4700136640 1 -- 172.21.15.33:0/3068577187 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f46e40a32c0 msgr2=0x7f46e40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:55.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.511+0000 7f4700136640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f46e40a32c0 0x7f46e40a5780 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f46ec009890 tx=0x7f46ec008040 comp rx=0 tx=0).stop 2024-09-16T00:21:55.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.511+0000 7f4700136640 1 -- 172.21.15.33:0/3068577187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46f81082a0 msgr2=0x7f46f819d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:55.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.511+0000 7f4700136640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46f81082a0 0x7f46f819d010 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f46f0005cc0 tx=0x7f46f0031460 comp rx=0 tx=0).stop 2024-09-16T00:21:55.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.511+0000 7f4700136640 1 -- 172.21.15.33:0/3068577187 shutdown_connections 2024-09-16T00:21:55.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.511+0000 7f4700136640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f46e40a32c0 0x7f46e40a5780 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:55.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.511+0000 7f4700136640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46f81082a0 0x7f46f819d010 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:55.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.511+0000 7f4700136640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46f81036e0 0x7f46f8198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:55.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.511+0000 7f4700136640 1 --2- 172.21.15.33:0/3068577187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46f8102d30 0x7f46f8198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:55.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.512+0000 7f4700136640 1 -- 172.21.15.33:0/3068577187 >> 172.21.15.33:0/3068577187 conn(0x7f46f80fc7d0 msgr2=0x7f46f8106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:55.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.512+0000 7f4700136640 1 -- 172.21.15.33:0/3068577187 shutdown_connections 2024-09-16T00:21:55.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:55.512+0000 7f4700136640 1 -- 172.21.15.33:0/3068577187 wait complete. 2024-09-16T00:21:55.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 12.04 MB/sec execute 63 sec latency 11828.945 ms 2024-09-16T00:21:56.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:56 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3068577187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:21:56.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:56 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3068577187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:21:56.537 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:56 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3068577187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:21:56.573 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 52b54352 --release --await' 2024-09-16T00:21:56.573 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 52b54352 --release --await 2024-09-16T00:21:56.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 11.85 MB/sec execute 64 sec latency 12828.996 ms 2024-09-16T00:21:57.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.073+0000 7f45703f3640 1 -- 172.21.15.33:0/471633543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568101e10 msgr2=0x7f45681021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:57.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.073+0000 7f45703f3640 1 --2- 172.21.15.33:0/471633543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568101e10 0x7f45681021f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f4564009900 tx=0x7f456402a410 comp rx=0 tx=0).stop 2024-09-16T00:21:57.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.073+0000 7f45703f3640 1 -- 172.21.15.33:0/471633543 shutdown_connections 2024-09-16T00:21:57.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.073+0000 7f45703f3640 1 --2- 172.21.15.33:0/471633543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4568106f40 0x7f456810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.073+0000 7f45703f3640 1 --2- 172.21.15.33:0/471633543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45681027c0 0x7f4568106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.073+0000 7f45703f3640 1 --2- 172.21.15.33:0/471633543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568101e10 0x7f45681021f0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.073+0000 7f45703f3640 1 -- 172.21.15.33:0/471633543 >> 172.21.15.33:0/471633543 conn(0x7f45680fd8d0 msgr2=0x7f45680ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:57.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.073+0000 7f45703f3640 1 -- 172.21.15.33:0/471633543 shutdown_connections 2024-09-16T00:21:57.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.073+0000 7f45703f3640 1 -- 172.21.15.33:0/471633543 wait complete. 2024-09-16T00:21:57.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f45703f3640 1 Processor -- start 2024-09-16T00:21:57.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f45703f3640 1 -- start start 2024-09-16T00:21:57.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f45703f3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568101e10 0x7f45681984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f45703f3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45681027c0 0x7f45681989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f45703f3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4568106f40 0x7f456819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f45703f3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4568111ff0 con 0x7f4568101e10 2024-09-16T00:21:57.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f45703f3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4568111e70 con 0x7f4568106f40 2024-09-16T00:21:57.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f45703f3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4568112170 con 0x7f45681027c0 2024-09-16T00:21:57.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f456e18f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568101e10 0x7f45681984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f456e18f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568101e10 0x7f45681984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51110/0 (socket says 172.21.15.33:51110) 2024-09-16T00:21:57.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f456e18f640 1 -- 172.21.15.33:0/1418298781 learned_addr learned my addr 172.21.15.33:0/1418298781 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:57.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f456d98e640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45681027c0 0x7f45681989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f456e990640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4568106f40 0x7f456819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f456e18f640 1 -- 172.21.15.33:0/1418298781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45681027c0 msgr2=0x7f45681989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:57.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.074+0000 7f456e18f640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45681027c0 0x7f45681989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.075+0000 7f456e18f640 1 -- 172.21.15.33:0/1418298781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4568106f40 msgr2=0x7f456819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:57.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.075+0000 7f456e18f640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4568106f40 0x7f456819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.075+0000 7f456e18f640 1 -- 172.21.15.33:0/1418298781 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f456819d460 con 0x7f4568101e10 2024-09-16T00:21:57.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.075+0000 7f456d98e640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45681027c0 0x7f45681989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:57.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.075+0000 7f456e18f640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568101e10 0x7f45681984b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f4564004770 tx=0x7f4564003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:57.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.075+0000 7f455f7fe640 1 -- 172.21.15.33:0/1418298781 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456403f070 con 0x7f4568101e10 2024-09-16T00:21:57.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.075+0000 7f45703f3640 1 -- 172.21.15.33:0/1418298781 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f456819d6f0 con 0x7f4568101e10 2024-09-16T00:21:57.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.075+0000 7f455f7fe640 1 -- 172.21.15.33:0/1418298781 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4564031ca0 con 0x7f4568101e10 2024-09-16T00:21:57.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.075+0000 7f455f7fe640 1 -- 172.21.15.33:0/1418298781 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564035030 con 0x7f4568101e10 2024-09-16T00:21:57.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.075+0000 7f45703f3640 1 -- 172.21.15.33:0/1418298781 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45681a5030 con 0x7f4568101e10 2024-09-16T00:21:57.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.077+0000 7f455f7fe640 1 -- 172.21.15.33:0/1418298781 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4564042070 con 0x7f4568101e10 2024-09-16T00:21:57.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.077+0000 7f45727cc740 1 Processor -- start 2024-09-16T00:21:57.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f45727cc740 1 -- start start 2024-09-16T00:21:57.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f45727cc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483a4a0 0x55813483a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f45727cc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483adc0 0x558134845140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f45727cc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558134845680 0x558134847a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f45727cc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5581347a02e0 con 0x55813483a4a0 2024-09-16T00:21:57.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f45727cc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558134747ce0 con 0x558134845680 2024-09-16T00:21:57.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f45727cc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55813479c2a0 con 0x55813483adc0 2024-09-16T00:21:57.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f456e990640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483a4a0 0x55813483a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f456e990640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483a4a0 0x55813483a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51114/0 (socket says 172.21.15.33:51114) 2024-09-16T00:21:57.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f456e990640 1 -- 172.21.15.33:0/2030809153 learned_addr learned my addr 172.21.15.33:0/2030809153 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:57.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f456e18f640 1 --2- 172.21.15.33:0/2030809153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483adc0 0x558134845140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f456d98e640 1 --2- 172.21.15.33:0/2030809153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558134845680 0x558134847a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f456e990640 1 -- 172.21.15.33:0/2030809153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483adc0 msgr2=0x558134845140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:57.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f456e990640 1 --2- 172.21.15.33:0/2030809153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483adc0 0x558134845140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f456e990640 1 -- 172.21.15.33:0/2030809153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558134845680 msgr2=0x558134847a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:57.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f456e990640 1 --2- 172.21.15.33:0/2030809153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558134845680 0x558134847a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f456e990640 1 -- 172.21.15.33:0/2030809153 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55813472c440 con 0x55813483a4a0 2024-09-16T00:21:57.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.078+0000 7f455f7fe640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f45540a3240 0x7f45540a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.079+0000 7f456e990640 1 --2- 172.21.15.33:0/2030809153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483a4a0 0x55813483a880 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f4560008d40 tx=0x7f456000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c1dd0ac829f2f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:57.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.079+0000 7f456e18f640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f45540a3240 0x7f45540a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.079+0000 7f455cdea640 1 -- 172.21.15.33:0/2030809153 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560014070 con 0x55813483a4a0 2024-09-16T00:21:57.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.079+0000 7f455cdea640 1 -- 172.21.15.33:0/2030809153 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4560003b70 con 0x55813483a4a0 2024-09-16T00:21:57.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.079+0000 7f455cdea640 1 -- 172.21.15.33:0/2030809153 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560010850 con 0x55813483a4a0 2024-09-16T00:21:57.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.079+0000 7f455f7fe640 1 -- 172.21.15.33:0/1418298781 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f45640f0420 con 0x7f4568101e10 2024-09-16T00:21:57.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.079+0000 7f456e18f640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f45540a3240 0x7f45540a5700 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f45640bc5f0 tx=0x7f45640f3f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:57.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.079+0000 7f45727cc740 1 -- 172.21.15.33:0/2030809153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483a4a0 msgr2=0x55813483a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:57.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.079+0000 7f45727cc740 1 --2- 172.21.15.33:0/2030809153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483a4a0 0x55813483a880 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f4560008d40 tx=0x7f456000e720 comp rx=0 tx=0).stop 2024-09-16T00:21:57.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 -- 172.21.15.33:0/2030809153 shutdown_connections 2024-09-16T00:21:57.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 --2- 172.21.15.33:0/2030809153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558134845680 0x558134847a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 --2- 172.21.15.33:0/2030809153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483adc0 0x558134845140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 --2- 172.21.15.33:0/2030809153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483a4a0 0x55813483a880 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 -- 172.21.15.33:0/2030809153 >> 172.21.15.33:0/2030809153 conn(0x558134836a30 msgr2=0x558134836e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:57.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 -- 172.21.15.33:0/2030809153 shutdown_connections 2024-09-16T00:21:57.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 -- 172.21.15.33:0/2030809153 wait complete. 2024-09-16T00:21:57.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 -- start start 2024-09-16T00:21:57.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 Processor -- start 2024-09-16T00:21:57.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 0x558134844f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55813483adc0 0x558134842220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558134845680 0x558134842760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558134848250 con 0x558134845680 2024-09-16T00:21:57.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55813475b050 con 0x55813483adc0 2024-09-16T00:21:57.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5581348483d0 con 0x55813483a4a0 2024-09-16T00:21:57.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f45727cc740 10 client.0 _ll_register_callbacks cb 0x55813479b160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:57.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.080+0000 7f456e990640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558134845680 0x558134842760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f456e990640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558134845680 0x558134842760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51124/0 (socket says 172.21.15.33:51124) 2024-09-16T00:21:57.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f456e990640 1 -- 172.21.15.33:0/2367324905 learned_addr learned my addr 172.21.15.33:0/2367324905 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:57.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f456d98e640 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55813483adc0 0x558134842220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f456e990640 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 0x558134844f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f456e990640 1 -- 172.21.15.33:0/2367324905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55813483adc0 msgr2=0x558134842220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:57.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f456e990640 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55813483adc0 0x558134842220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f456e990640 1 -- 172.21.15.33:0/2367324905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558134845680 msgr2=0x558134842760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:57.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f456e990640 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558134845680 0x558134842760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f456e990640 1 -- 172.21.15.33:0/2367324905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581345113c0 con 0x55813483a4a0 2024-09-16T00:21:57.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f456e990640 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 0x558134844f70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4560010530 tx=0x7f4560008f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54650f0a9bea15b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:57.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f4551ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:21:57.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f4551ffb640 1 -- 172.21.15.33:0/2367324905 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560014070 con 0x55813483a4a0 2024-09-16T00:21:57.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f4551ffb640 1 -- 172.21.15.33:0/2367324905 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f45600105d0 con 0x55813483a4a0 2024-09-16T00:21:57.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.081+0000 7f45727cc740 1 -- 172.21.15.33:0/2367324905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4560003b70 con 0x55813483a4a0 2024-09-16T00:21:57.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.082+0000 7f4551ffb640 1 -- 172.21.15.33:0/2367324905 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4560021040 con 0x55813483a4a0 2024-09-16T00:21:57.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.082+0000 7f45727cc740 10 client.25681 fetch_fsmap learned FSMap version 30 2024-09-16T00:21:57.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.082+0000 7f45727cc740 1 -- 172.21.15.33:0/2367324905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558134853bc0 con 0x55813483a4a0 2024-09-16T00:21:57.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.082+0000 7f4551ffb640 1 -- 172.21.15.33:0/2367324905 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4560003b70 con 0x55813483a4a0 2024-09-16T00:21:57.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.082+0000 7f4551ffb640 1 -- 172.21.15.33:0/2367324905 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f45600287b0 con 0x55813483a4a0 2024-09-16T00:21:57.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.082+0000 7f45727cc740 10 client.25681 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:21:57.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.082+0000 7f45727cc740 10 client.25681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:21:57.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.082+0000 7f45727cc740 20 client.25681 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:57.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.082+0000 7f45727cc740 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558134854fd0 0x5581348573c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.082+0000 7f45727cc740 4 client.25681 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:21:57.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.082+0000 7f45727cc740 1 -- 172.21.15.33:0/2367324905 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581347ad850 con 0x558134854fd0 2024-09-16T00:21:57.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.082+0000 7f456d98e640 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558134854fd0 0x5581348573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.083+0000 7f456d98e640 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558134854fd0 0x5581348573c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2d49524d9c77e43 server_cookie=83ef3fc910ce9dfa in_seq=0 out_seq=0 2024-09-16T00:21:57.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.083+0000 7f4551ffb640 10 client.25681 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:57.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f4551ffb640 1 -- 172.21.15.33:0/2367324905 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5581347ad850 con 0x558134854fd0 2024-09-16T00:21:57.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f4551ffb640 10 client.25681 handle_command_reply: tid=0 2024-09-16T00:21:57.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f45727cc740 1 client.25681 shutdown 2024-09-16T00:21:57.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f45727cc740 1 -- 172.21.15.33:0/2367324905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 msgr2=0x558134844f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:57.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f45727cc740 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 0x558134844f70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4560010530 tx=0x7f4560008f50 comp rx=0 tx=0).stop 2024-09-16T00:21:57.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f45727cc740 1 -- 172.21.15.33:0/2367324905 shutdown_connections 2024-09-16T00:21:57.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f45727cc740 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558134854fd0 0x5581348573c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f45727cc740 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558134845680 0x558134842760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f45727cc740 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55813483adc0 0x558134842220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f45727cc740 1 --2- 172.21.15.33:0/2367324905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 0x558134844f70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f45727cc740 1 -- 172.21.15.33:0/2367324905 >> 172.21.15.33:0/2367324905 conn(0x558134836a30 msgr2=0x55813483c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:57.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f4551ffb640 0 client.25681 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:57.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f45727cc740 1 -- 172.21.15.33:0/2367324905 shutdown_connections 2024-09-16T00:21:57.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f45727cc740 1 -- 172.21.15.33:0/2367324905 wait complete. 2024-09-16T00:21:57.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.084+0000 7f45727cc740 20 client.25681 trim_cache size 0 max 16384 2024-09-16T00:21:57.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.112+0000 7f45727cc740 1 Processor -- start 2024-09-16T00:21:57.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.112+0000 7f45727cc740 1 -- start start 2024-09-16T00:21:57.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f45727cc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55813483a4a0 0x558134840af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f45727cc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 0x55813485cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f45727cc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558134845680 0x55813485d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f45727cc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f45681b1a50 con 0x55813483adc0 2024-09-16T00:21:57.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f45727cc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f45681af320 con 0x55813483a4a0 2024-09-16T00:21:57.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f45727cc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f45681ad170 con 0x558134845680 2024-09-16T00:21:57.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f456d98e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 0x55813485cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f456d98e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 0x55813485cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51136/0 (socket says 172.21.15.33:51136) 2024-09-16T00:21:57.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f456d98e640 1 -- 172.21.15.33:0/153847409 learned_addr learned my addr 172.21.15.33:0/153847409 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:57.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f456e990640 1 --2- 172.21.15.33:0/153847409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55813483a4a0 0x558134840af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f456e990640 1 --2- 172.21.15.33:0/153847409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558134845680 0x55813485d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f456d98e640 1 -- 172.21.15.33:0/153847409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558134845680 msgr2=0x55813485d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:57.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f456d98e640 1 --2- 172.21.15.33:0/153847409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558134845680 0x55813485d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f456d98e640 1 -- 172.21.15.33:0/153847409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55813483a4a0 msgr2=0x558134840af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:57.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f456d98e640 1 --2- 172.21.15.33:0/153847409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55813483a4a0 0x558134840af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f456d98e640 1 -- 172.21.15.33:0/153847409 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581348578f0 con 0x55813483adc0 2024-09-16T00:21:57.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.113+0000 7f456e990640 1 --2- 172.21.15.33:0/153847409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55813483a4a0 0x558134840af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:57.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.114+0000 7f456d98e640 1 --2- 172.21.15.33:0/153847409 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 0x55813485cb80 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4558009fd0 tx=0x7f4558005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db33d8c2e4ebe2c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:57.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.114+0000 7f4553fff640 1 -- 172.21.15.33:0/153847409 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4558012cc0 con 0x55813483adc0 2024-09-16T00:21:57.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.114+0000 7f4553fff640 1 -- 172.21.15.33:0/153847409 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4558005500 con 0x55813483adc0 2024-09-16T00:21:57.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.114+0000 7f4553fff640 1 -- 172.21.15.33:0/153847409 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45580035f0 con 0x55813483adc0 2024-09-16T00:21:57.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.115+0000 7f45727cc740 1 -- 172.21.15.33:0/153847409 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 msgr2=0x55813485cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:57.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.115+0000 7f45727cc740 1 --2- 172.21.15.33:0/153847409 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 0x55813485cb80 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4558009fd0 tx=0x7f4558005f90 comp rx=0 tx=0).stop 2024-09-16T00:21:57.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.115+0000 7f45727cc740 1 -- 172.21.15.33:0/153847409 shutdown_connections 2024-09-16T00:21:57.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.115+0000 7f45727cc740 1 --2- 172.21.15.33:0/153847409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558134845680 0x55813485d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.115+0000 7f45727cc740 1 --2- 172.21.15.33:0/153847409 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 0x55813485cb80 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.115+0000 7f45727cc740 1 --2- 172.21.15.33:0/153847409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55813483a4a0 0x558134840af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.115+0000 7f45727cc740 1 -- 172.21.15.33:0/153847409 >> 172.21.15.33:0/153847409 conn(0x558134836a30 msgr2=0x558134835b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:57.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.115+0000 7f45727cc740 1 -- 172.21.15.33:0/153847409 shutdown_connections 2024-09-16T00:21:57.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.115+0000 7f45727cc740 1 -- 172.21.15.33:0/153847409 wait complete. 2024-09-16T00:21:57.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.115+0000 7f45727cc740 1 -- start start 2024-09-16T00:21:57.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.115+0000 7f45727cc740 1 Processor -- start 2024-09-16T00:21:57.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f45727cc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 0x55813484aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f45727cc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 0x55813484b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f45727cc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558134845680 0x5581348bac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f45727cc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f45681ad7e0 con 0x55813483adc0 2024-09-16T00:21:57.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f45727cc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f45681aec50 con 0x558134845680 2024-09-16T00:21:57.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f45727cc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f45681adeb0 con 0x55813483a4a0 2024-09-16T00:21:57.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f456e990640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 0x55813484aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f456d98e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 0x55813484b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f456e990640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 0x55813484aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36840/0 (socket says 172.21.15.33:36840) 2024-09-16T00:21:57.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f456e990640 1 -- 172.21.15.33:0/3296613779 learned_addr learned my addr 172.21.15.33:0/3296613779 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:57.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f45727cc740 10 client.0 _ll_register_callbacks cb 0x7f45640f3e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:57.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f456e990640 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558134845680 0x5581348bac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f456e990640 1 -- 172.21.15.33:0/3296613779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558134845680 msgr2=0x5581348bac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:57.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f456e990640 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558134845680 0x5581348bac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f456e990640 1 -- 172.21.15.33:0/3296613779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 msgr2=0x55813484b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:57.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f456e990640 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 0x55813484b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f456e990640 1 -- 172.21.15.33:0/3296613779 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55813483abf0 con 0x55813483a4a0 2024-09-16T00:21:57.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.116+0000 7f456d98e640 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 0x55813484b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:57.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f456e990640 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 0x55813484aca0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4560028b80 tx=0x7f4560004120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d79901fb9e67653 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:57.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f4551ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:21:57.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f4551ffb640 1 -- 172.21.15.33:0/3296613779 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560014040 con 0x55813483a4a0 2024-09-16T00:21:57.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f45727cc740 1 -- 172.21.15.33:0/3296613779 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4558005500 con 0x55813483a4a0 2024-09-16T00:21:57.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f4551ffb640 1 -- 172.21.15.33:0/3296613779 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4560004a00 con 0x55813483a4a0 2024-09-16T00:21:57.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f4551ffb640 1 -- 172.21.15.33:0/3296613779 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4560003b70 con 0x55813483a4a0 2024-09-16T00:21:57.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f45727cc740 10 client.25687 fetch_fsmap learned FSMap version 30 2024-09-16T00:21:57.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f45727cc740 1 -- 172.21.15.33:0/3296613779 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5581348b8420 con 0x55813483a4a0 2024-09-16T00:21:57.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f4551ffb640 1 -- 172.21.15.33:0/3296613779 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4558005500 con 0x55813483a4a0 2024-09-16T00:21:57.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f4551ffb640 1 -- 172.21.15.33:0/3296613779 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f4560040030 con 0x55813483a4a0 2024-09-16T00:21:57.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f45727cc740 10 client.25687 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:21:57.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f45727cc740 10 client.25687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:21:57.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f45727cc740 20 client.25687 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:57.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f45727cc740 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5581348c3f60 0x5581348c6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:57.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f45727cc740 4 client.25687 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:21:57.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.117+0000 7f45727cc740 1 -- 172.21.15.33:0/3296613779 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5581347a7d60 con 0x5581348c3f60 2024-09-16T00:21:57.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.118+0000 7f456d98e640 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5581348c3f60 0x5581348c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:57.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.118+0000 7f456d98e640 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5581348c3f60 0x5581348c6350 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f62fe6902cd865ef server_cookie=438abc6684843502 in_seq=0 out_seq=0 2024-09-16T00:21:57.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.118+0000 7f4551ffb640 10 client.25687 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:57.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.118+0000 7f4551ffb640 1 -- 172.21.15.33:0/3296613779 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5581347a7d60 con 0x5581348c3f60 2024-09-16T00:21:57.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.118+0000 7f4551ffb640 10 client.25687 handle_command_reply: tid=0 2024-09-16T00:21:57.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.118+0000 7f45727cc740 1 client.25687 shutdown 2024-09-16T00:21:57.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.118+0000 7f45727cc740 1 -- 172.21.15.33:0/3296613779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 msgr2=0x55813484aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:57.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.118+0000 7f45727cc740 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 0x55813484aca0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4560028b80 tx=0x7f4560004120 comp rx=0 tx=0).stop 2024-09-16T00:21:57.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.118+0000 7f45727cc740 1 -- 172.21.15.33:0/3296613779 shutdown_connections 2024-09-16T00:21:57.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.118+0000 7f45727cc740 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5581348c3f60 0x5581348c6350 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.118+0000 7f45727cc740 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558134845680 0x5581348bac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.119+0000 7f45727cc740 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55813483adc0 0x55813484b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.118+0000 7f4551ffb640 0 client.25687 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:57.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.119+0000 7f45727cc740 1 --2- 172.21.15.33:0/3296613779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55813483a4a0 0x55813484aca0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.119+0000 7f45727cc740 1 -- 172.21.15.33:0/3296613779 >> 172.21.15.33:0/3296613779 conn(0x558134836a30 msgr2=0x558134843a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:57.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.119+0000 7f45727cc740 1 -- 172.21.15.33:0/3296613779 shutdown_connections 2024-09-16T00:21:57.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.119+0000 7f45727cc740 1 -- 172.21.15.33:0/3296613779 wait complete. 2024-09-16T00:21:57.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.119+0000 7f45727cc740 20 client.25687 trim_cache size 0 max 16384 2024-09-16T00:21:57.131 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:21:57.131 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:21:57.131 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:21:57.131 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:21:57.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.119+0000 7f45703f3640 1 -- 172.21.15.33:0/1418298781 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f45540a3240 msgr2=0x7f45540a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:57.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.119+0000 7f45703f3640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f45540a3240 0x7f45540a5700 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f45640bc5f0 tx=0x7f45640f3f90 comp rx=0 tx=0).stop 2024-09-16T00:21:57.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.119+0000 7f45703f3640 1 -- 172.21.15.33:0/1418298781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568101e10 msgr2=0x7f45681984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:57.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.119+0000 7f45703f3640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568101e10 0x7f45681984b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f4564004770 tx=0x7f4564003c10 comp rx=0 tx=0).stop 2024-09-16T00:21:57.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.120+0000 7f45703f3640 1 -- 172.21.15.33:0/1418298781 shutdown_connections 2024-09-16T00:21:57.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.120+0000 7f45703f3640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f45540a3240 0x7f45540a5700 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.120+0000 7f45703f3640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4568106f40 0x7f456819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.120+0000 7f45703f3640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45681027c0 0x7f45681989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.120+0000 7f45703f3640 1 --2- 172.21.15.33:0/1418298781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568101e10 0x7f45681984b0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:57.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.120+0000 7f45703f3640 1 -- 172.21.15.33:0/1418298781 >> 172.21.15.33:0/1418298781 conn(0x7f45680fd8d0 msgr2=0x7f45680fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:57.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.120+0000 7f45703f3640 1 -- 172.21.15.33:0/1418298781 shutdown_connections 2024-09-16T00:21:57.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:57.120+0000 7f45703f3640 1 -- 172.21.15.33:0/1418298781 wait complete. 2024-09-16T00:21:57.139 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:57.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:57 smithi033 ceph-mon[31243]: pgmap v494: 129 pgs: 129 active+clean; 114 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 2.4 MiB/s wr, 113 op/s 2024-09-16T00:21:57.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:57 smithi083 ceph-mon[35242]: pgmap v494: 129 pgs: 129 active+clean; 114 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 2.4 MiB/s wr, 113 op/s 2024-09-16T00:21:57.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:57 smithi060 ceph-mon[35185]: pgmap v494: 129 pgs: 129 active+clean; 114 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 2.4 MiB/s wr, 113 op/s 2024-09-16T00:21:57.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259088 11.67 MB/sec execute 65 sec latency 13829.043 ms 2024-09-16T00:21:57.842 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:21:57.842 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:21:58.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.254+0000 7ff7e451a640 1 -- 172.21.15.33:0/1258141485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7dc100a80 msgr2=0x7ff7dc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:58.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.254+0000 7ff7e451a640 1 --2- 172.21.15.33:0/1258141485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7dc100a80 0x7ff7dc100ee0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009930 tx=0x7ff7cc02a420 comp rx=0 tx=0).stop 2024-09-16T00:21:58.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 -- 172.21.15.33:0/1258141485 shutdown_connections 2024-09-16T00:21:58.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 --2- 172.21.15.33:0/1258141485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7dc101420 0x7ff7dc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 --2- 172.21.15.33:0/1258141485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7dc100a80 0x7ff7dc100ee0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 --2- 172.21.15.33:0/1258141485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7dc106a80 0x7ff7dc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 -- 172.21.15.33:0/1258141485 >> 172.21.15.33:0/1258141485 conn(0x7ff7dc0fc7a0 msgr2=0x7ff7dc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:58.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 -- 172.21.15.33:0/1258141485 shutdown_connections 2024-09-16T00:21:58.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 -- 172.21.15.33:0/1258141485 wait complete. 2024-09-16T00:21:58.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 Processor -- start 2024-09-16T00:21:58.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 -- start start 2024-09-16T00:21:58.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7dc100a80 0x7ff7dc0fcde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7dc101420 0x7ff7dc0fd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7dc106a80 0x7ff7dc0fd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc110e80 con 0x7ff7dc106a80 2024-09-16T00:21:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc110d00 con 0x7ff7dc101420 2024-09-16T00:21:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.255+0000 7ff7e451a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc111000 con 0x7ff7dc100a80 2024-09-16T00:21:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e2ab7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7dc106a80 0x7ff7dc0fd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e2ab7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7dc106a80 0x7ff7dc0fd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51160/0 (socket says 172.21.15.33:51160) 2024-09-16T00:21:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e2ab7640 1 -- 172.21.15.33:0/2829295494 learned_addr learned my addr 172.21.15.33:0/2829295494 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e22b6640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7dc100a80 0x7ff7dc0fcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7dc101420 0x7ff7dc0fd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e2ab7640 1 -- 172.21.15.33:0/2829295494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7dc100a80 msgr2=0x7ff7dc0fcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e2ab7640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7dc100a80 0x7ff7dc0fcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e2ab7640 1 -- 172.21.15.33:0/2829295494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7dc101420 msgr2=0x7ff7dc0fd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e2ab7640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7dc101420 0x7ff7dc0fd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e2ab7640 1 -- 172.21.15.33:0/2829295494 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7dc1a68e0 con 0x7ff7dc106a80 2024-09-16T00:21:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e2ab7640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7dc106a80 0x7ff7dc0fd860 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400eaa0 tx=0x7ff7d400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7d37fe640 1 -- 172.21.15.33:0/2829295494 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d400cdb0 con 0x7ff7dc106a80 2024-09-16T00:21:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7d37fe640 1 -- 172.21.15.33:0/2829295494 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7d4005020 con 0x7ff7dc106a80 2024-09-16T00:21:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e451a640 1 -- 172.21.15.33:0/2829295494 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7dc1a6bd0 con 0x7ff7dc106a80 2024-09-16T00:21:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7d37fe640 1 -- 172.21.15.33:0/2829295494 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d40116b0 con 0x7ff7dc106a80 2024-09-16T00:21:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.256+0000 7ff7e451a640 1 -- 172.21.15.33:0/2829295494 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7dc1a7130 con 0x7ff7dc106a80 2024-09-16T00:21:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.259+0000 7ff7d37fe640 1 -- 172.21.15.33:0/2829295494 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff7d4005560 con 0x7ff7dc106a80 2024-09-16T00:21:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.259+0000 7ff7e68f3740 1 Processor -- start 2024-09-16T00:21:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.259+0000 7ff7d37fe640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7c80a32c0 0x7ff7c80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7c80a32c0 0x7ff7c80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e68f3740 1 -- start start 2024-09-16T00:21:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7d37fe640 1 -- 172.21.15.33:0/2829295494 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff7d40cfb80 con 0x7ff7dc106a80 2024-09-16T00:21:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e68f3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f10923060 0x564f10923440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e68f3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923980 0x564f1092dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e68f3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f1092e240 0x564f10930630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e68f3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564f10888620 con 0x564f1092e240 2024-09-16T00:21:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e68f3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564f1082fe50 con 0x564f10923060 2024-09-16T00:21:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e68f3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564f108845e0 con 0x564f10923980 2024-09-16T00:21:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e2ab7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f1092e240 0x564f10930630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e1ab5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923980 0x564f1092dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e2ab7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f1092e240 0x564f10930630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51170/0 (socket says 172.21.15.33:51170) 2024-09-16T00:21:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e2ab7640 1 -- 172.21.15.33:0/3306478947 learned_addr learned my addr 172.21.15.33:0/3306478947 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e22b6640 1 --2- 172.21.15.33:0/3306478947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f10923060 0x564f10923440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7c80a32c0 0x7ff7c80a5780 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009900 tx=0x7ff7cc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e2ab7640 1 -- 172.21.15.33:0/3306478947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923980 msgr2=0x564f1092dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e2ab7640 1 --2- 172.21.15.33:0/3306478947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923980 0x564f1092dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e2ab7640 1 -- 172.21.15.33:0/3306478947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f10923060 msgr2=0x564f10923440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e2ab7640 1 --2- 172.21.15.33:0/3306478947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f10923060 0x564f10923440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e2ab7640 1 -- 172.21.15.33:0/3306478947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f10820d40 con 0x564f1092e240 2024-09-16T00:21:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.260+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/3306478947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923980 0x564f1092dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.261+0000 7ff7e2ab7640 1 --2- 172.21.15.33:0/3306478947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f1092e240 0x564f10930630 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40981d0 tx=0x7ff7d40a01b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29a573afb4422a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.261+0000 7ff7d0ff9640 1 -- 172.21.15.33:0/3306478947 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d401f040 con 0x564f1092e240 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.261+0000 7ff7d0ff9640 1 -- 172.21.15.33:0/3306478947 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7d409a070 con 0x564f1092e240 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.261+0000 7ff7d0ff9640 1 -- 172.21.15.33:0/3306478947 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d409ac30 con 0x564f1092e240 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.261+0000 7ff7e68f3740 1 -- 172.21.15.33:0/3306478947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f1092e240 msgr2=0x564f10930630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.261+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/3306478947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f1092e240 0x564f10930630 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40981d0 tx=0x7ff7d40a01b0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.261+0000 7ff7e68f3740 1 -- 172.21.15.33:0/3306478947 shutdown_connections 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/3306478947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f1092e240 0x564f10930630 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/3306478947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923980 0x564f1092dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/3306478947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f10923060 0x564f10923440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 -- 172.21.15.33:0/3306478947 >> 172.21.15.33:0/3306478947 conn(0x564f1091f5f0 msgr2=0x564f1091f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 -- 172.21.15.33:0/3306478947 shutdown_connections 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 -- 172.21.15.33:0/3306478947 wait complete. 2024-09-16T00:21:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 -- start start 2024-09-16T00:21:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 Processor -- start 2024-09-16T00:21:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 0x564f1092b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923980 0x564f1092b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f1092e240 0x564f1092bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564f10930e10 con 0x564f10923060 2024-09-16T00:21:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564f10810f10 con 0x564f1092e240 2024-09-16T00:21:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564f10930f90 con 0x564f10923980 2024-09-16T00:21:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e68f3740 10 client.0 _ll_register_callbacks cb 0x564f10834db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.262+0000 7ff7e22b6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 0x564f1092b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e22b6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 0x564f1092b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51176/0 (socket says 172.21.15.33:51176) 2024-09-16T00:21:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e22b6640 1 -- 172.21.15.33:0/1599766580 learned_addr learned my addr 172.21.15.33:0/1599766580 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e2ab7640 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f1092e240 0x564f1092bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e22b6640 1 -- 172.21.15.33:0/1599766580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923980 msgr2=0x564f1092b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923980 0x564f1092b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e22b6640 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923980 0x564f1092b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e22b6640 1 -- 172.21.15.33:0/1599766580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f1092e240 msgr2=0x564f1092bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e22b6640 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f1092e240 0x564f1092bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e22b6640 1 -- 172.21.15.33:0/1599766580 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f10597580 con 0x564f10923060 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e2ab7640 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f1092e240 0x564f1092bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e22b6640 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 0x564f1092b110 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8010850 tx=0x7ff7d8010d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d17b878ac6d462 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7c5ffb640 1 -- 172.21.15.33:0/1599766580 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d801e060 con 0x564f10923060 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7c5ffb640 1 -- 172.21.15.33:0/1599766580 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff7d800fb90 con 0x564f10923060 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e68f3740 1 -- 172.21.15.33:0/1599766580 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7d409a070 con 0x564f10923060 2024-09-16T00:21:58.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7c5ffb640 1 -- 172.21.15.33:0/1599766580 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff7d8025050 con 0x564f10923060 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e68f3740 10 client.16218 fetch_fsmap learned FSMap version 30 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.263+0000 7ff7e68f3740 1 -- 172.21.15.33:0/1599766580 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564f10661960 con 0x564f10923060 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.264+0000 7ff7c5ffb640 1 -- 172.21.15.33:0/1599766580 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7d409a070 con 0x564f10923060 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.264+0000 7ff7c5ffb640 1 -- 172.21.15.33:0/1599766580 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff7d800e3b0 con 0x564f10923060 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.264+0000 7ff7e68f3740 10 client.16218 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.264+0000 7ff7e68f3740 10 client.16218 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.264+0000 7ff7e68f3740 20 client.16218 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.264+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564f1093ddf0 0x564f109401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.264+0000 7ff7e68f3740 4 client.16218 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.264+0000 7ff7e68f3740 1 -- 172.21.15.33:0/1599766580 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f10895b90 con 0x564f1093ddf0 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.264+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564f1093ddf0 0x564f109401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.265+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564f1093ddf0 0x564f109401e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e66345a46558cbdc server_cookie=48ec636f0b37dc7c in_seq=0 out_seq=0 2024-09-16T00:21:58.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.265+0000 7ff7c5ffb640 10 client.16218 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:58.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.265+0000 7ff7c5ffb640 1 -- 172.21.15.33:0/1599766580 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564f10895b90 con 0x564f1093ddf0 2024-09-16T00:21:58.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.265+0000 7ff7c5ffb640 10 client.16218 handle_command_reply: tid=0 2024-09-16T00:21:58.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7e68f3740 1 client.16218 shutdown 2024-09-16T00:21:58.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7e68f3740 1 -- 172.21.15.33:0/1599766580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 msgr2=0x564f1092b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:58.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 0x564f1092b110 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8010850 tx=0x7ff7d8010d10 comp rx=0 tx=0).stop 2024-09-16T00:21:58.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7e68f3740 1 -- 172.21.15.33:0/1599766580 shutdown_connections 2024-09-16T00:21:58.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564f1093ddf0 0x564f109401e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f1092e240 0x564f1092bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923980 0x564f1092b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/1599766580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 0x564f1092b110 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7c5ffb640 0 client.16218 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:58.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7e68f3740 1 -- 172.21.15.33:0/1599766580 >> 172.21.15.33:0/1599766580 conn(0x564f1091f5f0 msgr2=0x564f109252d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:58.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7e68f3740 1 -- 172.21.15.33:0/1599766580 shutdown_connections 2024-09-16T00:21:58.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7e68f3740 1 -- 172.21.15.33:0/1599766580 wait complete. 2024-09-16T00:21:58.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.266+0000 7ff7e68f3740 20 client.16218 trim_cache size 0 max 16384 2024-09-16T00:21:58.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.283+0000 7ff7e68f3740 1 Processor -- start 2024-09-16T00:21:58.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.283+0000 7ff7e68f3740 1 -- start start 2024-09-16T00:21:58.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.283+0000 7ff7e68f3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923060 0x564f1092d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.283+0000 7ff7e68f3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923980 0x564f10945a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.283+0000 7ff7e68f3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f1092e240 0x564f10945fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.283+0000 7ff7e68f3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1b3c10 con 0x564f10923980 2024-09-16T00:21:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.283+0000 7ff7e68f3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1b14e0 con 0x564f1092e240 2024-09-16T00:21:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.283+0000 7ff7e68f3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1af330 con 0x564f10923060 2024-09-16T00:21:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.283+0000 7ff7e1ab5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923980 0x564f10945a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.283+0000 7ff7e1ab5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923980 0x564f10945a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51188/0 (socket says 172.21.15.33:51188) 2024-09-16T00:21:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.283+0000 7ff7e1ab5640 1 -- 172.21.15.33:0/2154445010 learned_addr learned my addr 172.21.15.33:0/2154445010 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.284+0000 7ff7e22b6640 1 --2- 172.21.15.33:0/2154445010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923060 0x564f1092d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.284+0000 7ff7e1ab5640 1 -- 172.21.15.33:0/2154445010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923060 msgr2=0x564f1092d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.284+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/2154445010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923060 0x564f1092d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.284+0000 7ff7e1ab5640 1 -- 172.21.15.33:0/2154445010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f1092e240 msgr2=0x564f10945fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.284+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/2154445010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f1092e240 0x564f10945fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.284+0000 7ff7e1ab5640 1 -- 172.21.15.33:0/2154445010 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f10940840 con 0x564f10923980 2024-09-16T00:21:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.284+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/2154445010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923980 0x564f10945a90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009930 tx=0x7ff7cc03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254bd11893eaa423 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.284+0000 7ff7c7fff640 1 -- 172.21.15.33:0/2154445010 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc035040 con 0x564f10923980 2024-09-16T00:21:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.284+0000 7ff7c7fff640 1 -- 172.21.15.33:0/2154445010 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7cc057070 con 0x564f10923980 2024-09-16T00:21:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.284+0000 7ff7c7fff640 1 -- 172.21.15.33:0/2154445010 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc0528e0 con 0x564f10923980 2024-09-16T00:21:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2154445010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923980 msgr2=0x564f10945a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/2154445010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923980 0x564f10945a90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009930 tx=0x7ff7cc03f000 comp rx=0 tx=0).stop 2024-09-16T00:21:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2154445010 shutdown_connections 2024-09-16T00:21:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/2154445010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f1092e240 0x564f10945fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/2154445010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923980 0x564f10945a90 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/2154445010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f10923060 0x564f1092d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2154445010 >> 172.21.15.33:0/2154445010 conn(0x564f1091f5f0 msgr2=0x564f1092c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2154445010 shutdown_connections 2024-09-16T00:21:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2154445010 wait complete. 2024-09-16T00:21:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 -- start start 2024-09-16T00:21:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 Processor -- start 2024-09-16T00:21:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 0x564f109361e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f10923980 0x564f10936720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.285+0000 7ff7e68f3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f1092e240 0x564f109331b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e68f3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1af9a0 con 0x564f10923060 2024-09-16T00:21:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e68f3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1b0e10 con 0x564f10923980 2024-09-16T00:21:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e68f3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1b0070 con 0x564f1092e240 2024-09-16T00:21:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e22b6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 0x564f109361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e68f3740 10 client.0 _ll_register_callbacks cb 0x7ff7d8010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:21:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e22b6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 0x564f109361e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51204/0 (socket says 172.21.15.33:51204) 2024-09-16T00:21:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e22b6640 1 -- 172.21.15.33:0/2029362348 learned_addr learned my addr 172.21.15.33:0/2029362348 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f10923980 0x564f10936720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e2ab7640 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f1092e240 0x564f109331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e22b6640 1 -- 172.21.15.33:0/2029362348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f1092e240 msgr2=0x564f109331b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e22b6640 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f1092e240 0x564f109331b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e22b6640 1 -- 172.21.15.33:0/2029362348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f10923980 msgr2=0x564f10936720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e22b6640 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f10923980 0x564f10936720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e22b6640 1 -- 172.21.15.33:0/2029362348 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f10933770 con 0x564f10923060 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f10923980 0x564f10936720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e22b6640 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 0x564f109361e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8000c00 tx=0x7ff7d8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c629ed48bc8e0c36 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7c5ffb640 1 -- 172.21.15.33:0/2029362348 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d802d070 con 0x564f10923060 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7c5ffb640 1 -- 172.21.15.33:0/2029362348 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff7d800fda0 con 0x564f10923060 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.286+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2029362348 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7cc057070 con 0x564f10923060 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.287+0000 7ff7c5ffb640 1 -- 172.21.15.33:0/2029362348 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff7d801e020 con 0x564f10923060 2024-09-16T00:21:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.287+0000 7ff7e68f3740 10 client.16224 fetch_fsmap learned FSMap version 30 2024-09-16T00:21:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.287+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2029362348 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564f109a9400 con 0x564f10923060 2024-09-16T00:21:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.287+0000 7ff7c5ffb640 1 -- 172.21.15.33:0/2029362348 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7cc057070 con 0x564f10923060 2024-09-16T00:21:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.287+0000 7ff7c5ffb640 1 -- 172.21.15.33:0/2029362348 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff7d80299a0 con 0x564f10923060 2024-09-16T00:21:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.287+0000 7ff7e68f3740 10 client.16224 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:21:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.287+0000 7ff7e68f3740 10 client.16224 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:21:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.287+0000 7ff7e68f3740 20 client.16224 populate_metadata read hostname 'smithi033' 2024-09-16T00:21:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.287+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564f109a9bb0 0x564f109abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.287+0000 7ff7e68f3740 4 client.16224 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "52b54352", "release": true, "await": true}] 2024-09-16T00:21:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.287+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2029362348 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "52b54352", "release": true, "await": true}) -- 0x564f108900a0 con 0x564f109a9bb0 2024-09-16T00:21:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.287+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564f109a9bb0 0x564f109abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.288+0000 7ff7e1ab5640 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564f109a9bb0 0x564f109abfd0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f32a422b6a172ea3 server_cookie=d01125c6eb4d2670 in_seq=0 out_seq=0 2024-09-16T00:21:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.288+0000 7ff7c5ffb640 10 client.16224 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.299+0000 7ff7c5ffb640 1 -- 172.21.15.33:0/2029362348 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x564f108900a0 con 0x564f109a9bb0 2024-09-16T00:21:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.299+0000 7ff7c5ffb640 10 client.16224 handle_command_reply: tid=0 2024-09-16T00:21:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.299+0000 7ff7e68f3740 1 client.16224 shutdown 2024-09-16T00:21:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.299+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2029362348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 msgr2=0x564f109361e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.299+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 0x564f109361e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8000c00 tx=0x7ff7d8005780 comp rx=0 tx=0).stop 2024-09-16T00:21:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.299+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2029362348 shutdown_connections 2024-09-16T00:21:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.299+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564f109a9bb0 0x564f109abfd0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.300+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564f1092e240 0x564f109331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.300+0000 7ff7c5ffb640 0 client.16224 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:21:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.300+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564f10923980 0x564f10936720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.300+0000 7ff7e68f3740 1 --2- 172.21.15.33:0/2029362348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f10923060 0x564f109361e0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.300+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2029362348 >> 172.21.15.33:0/2029362348 conn(0x564f1091f5f0 msgr2=0x564f1091e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.300+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2029362348 shutdown_connections 2024-09-16T00:21:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.300+0000 7ff7e68f3740 1 -- 172.21.15.33:0/2029362348 wait complete. 2024-09-16T00:21:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.300+0000 7ff7e68f3740 20 client.16224 trim_cache size 0 max 16384 2024-09-16T00:21:58.303 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:21:58.303 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:21:58.303 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:21:58.303 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 16, 2024-09-16T00:21:58.303 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:21:58.303 INFO:teuthology.orchestra.run.smithi033.stdout: "52b54352": { 2024-09-16T00:21:58.303 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 16, 2024-09-16T00:21:58.303 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:21:58.303 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:21:58.304 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:21:58.304 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:21:58.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:21:58.304 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:21:58.304 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 133.8, 2024-09-16T00:21:58.304 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:21:58.304 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:21:58.304 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:21:58.304 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:21:58.304 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:21:58.304 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:21:58.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:21:58.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:21:58.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:21:58.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:21:58.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:21:58.305 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:21:58.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.300+0000 7ff7e451a640 1 -- 172.21.15.33:0/2829295494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7c80a32c0 msgr2=0x7ff7c80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:58.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.300+0000 7ff7e451a640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7c80a32c0 0x7ff7c80a5780 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009900 tx=0x7ff7cc0058d0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.300+0000 7ff7e451a640 1 -- 172.21.15.33:0/2829295494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7dc106a80 msgr2=0x7ff7dc0fd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:58.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.300+0000 7ff7e451a640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7dc106a80 0x7ff7dc0fd860 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400eaa0 tx=0x7ff7d400ef60 comp rx=0 tx=0).stop 2024-09-16T00:21:58.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.301+0000 7ff7e451a640 1 -- 172.21.15.33:0/2829295494 shutdown_connections 2024-09-16T00:21:58.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.301+0000 7ff7e451a640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7c80a32c0 0x7ff7c80a5780 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.301+0000 7ff7e451a640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7dc106a80 0x7ff7dc0fd860 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.301+0000 7ff7e451a640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7dc101420 0x7ff7dc0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.301+0000 7ff7e451a640 1 --2- 172.21.15.33:0/2829295494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7dc100a80 0x7ff7dc0fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:58.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.301+0000 7ff7e451a640 1 -- 172.21.15.33:0/2829295494 >> 172.21.15.33:0/2829295494 conn(0x7ff7dc0fc7a0 msgr2=0x7ff7dc06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:58.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.301+0000 7ff7e451a640 1 -- 172.21.15.33:0/2829295494 shutdown_connections 2024-09-16T00:21:58.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:58.301+0000 7ff7e451a640 1 -- 172.21.15.33:0/2829295494 wait complete. 2024-09-16T00:21:58.567 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:21:58.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 259504 11.51 MB/sec execute 66 sec latency 14474.335 ms 2024-09-16T00:21:59.156 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '52b54352', total seconds elapsed: 18.5 2024-09-16T00:21:59.366 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:21:59 smithi033 ceph-mon[31243]: pgmap v495: 129 pgs: 129 active+clean; 114 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-09-16T00:21:59.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:21:59 smithi083 ceph-mon[35242]: pgmap v495: 129 pgs: 129 active+clean; 114 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-09-16T00:21:59.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:21:59 smithi060 ceph-mon[35185]: pgmap v495: 129 pgs: 129 active+clean; 114 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-09-16T00:21:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.584+0000 7f21159d9640 1 -- 172.21.15.33:0/546421697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2110103760 msgr2=0x7f2110107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.584+0000 7f21159d9640 1 --2- 172.21.15.33:0/546421697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2110103760 0x7f2110107ba0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2104009930 tx=0x7f210402a420 comp rx=0 tx=0).stop 2024-09-16T00:21:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.584+0000 7f21159d9640 1 -- 172.21.15.33:0/546421697 shutdown_connections 2024-09-16T00:21:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.584+0000 7f21159d9640 1 --2- 172.21.15.33:0/546421697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21101082e0 0x7f211010ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.584+0000 7f21159d9640 1 --2- 172.21.15.33:0/546421697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2110103760 0x7f2110107ba0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.584+0000 7f21159d9640 1 --2- 172.21.15.33:0/546421697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110102db0 0x7f2110103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.584+0000 7f21159d9640 1 -- 172.21.15.33:0/546421697 >> 172.21.15.33:0/546421697 conn(0x7f21100fc7d0 msgr2=0x7f21100febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.584+0000 7f21159d9640 1 -- 172.21.15.33:0/546421697 shutdown_connections 2024-09-16T00:21:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.584+0000 7f21159d9640 1 -- 172.21.15.33:0/546421697 wait complete. 2024-09-16T00:21:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.584+0000 7f21159d9640 1 Processor -- start 2024-09-16T00:21:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f21159d9640 1 -- start start 2024-09-16T00:21:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f21159d9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2110102db0 0x7f2110100a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f21159d9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110103760 0x7f2110100f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f21159d9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21101082e0 0x7f2110101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f21159d9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2110112070 con 0x7f21101082e0 2024-09-16T00:21:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f21159d9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2110111ef0 con 0x7f2110102db0 2024-09-16T00:21:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f21159d9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21101121f0 con 0x7f2110103760 2024-09-16T00:21:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2110102db0 0x7f2110100a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21101082e0 0x7f2110101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110103760 0x7f2110100f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110103760 0x7f2110100f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36914/0 (socket says 172.21.15.33:36914) 2024-09-16T00:21:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2110102db0 0x7f2110100a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:57924/0 (socket says 172.21.15.33:57924) 2024-09-16T00:21:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210e7fc640 1 -- 172.21.15.33:0/1894960810 learned_addr learned my addr 172.21.15.33:0/1894960810 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:21:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210e7fc640 1 -- 172.21.15.33:0/1894960810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2110102db0 msgr2=0x7f2110100a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210e7fc640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2110102db0 0x7f2110100a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210e7fc640 1 -- 172.21.15.33:0/1894960810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21101082e0 msgr2=0x7f2110101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210e7fc640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21101082e0 0x7f2110101490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210e7fc640 1 -- 172.21.15.33:0/1894960810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21101a6aa0 con 0x7f2110103760 2024-09-16T00:21:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210effd640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2110102db0 0x7f2110100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:21:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.585+0000 7f210e7fc640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110103760 0x7f2110100f50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2104005940 tx=0x7f2104003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.586+0000 7f21149d7640 1 -- 172.21.15.33:0/1894960810 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210403f070 con 0x7f2110103760 2024-09-16T00:21:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.586+0000 7f21149d7640 1 -- 172.21.15.33:0/1894960810 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2104031c30 con 0x7f2110103760 2024-09-16T00:21:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.586+0000 7f21159d9640 1 -- 172.21.15.33:0/1894960810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21101a6d30 con 0x7f2110103760 2024-09-16T00:21:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.586+0000 7f21149d7640 1 -- 172.21.15.33:0/1894960810 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2104035030 con 0x7f2110103760 2024-09-16T00:21:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.586+0000 7f21159d9640 1 -- 172.21.15.33:0/1894960810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21101a71c0 con 0x7f2110103760 2024-09-16T00:21:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.587+0000 7f21159d9640 1 -- 172.21.15.33:0/1894960810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2110103190 con 0x7f2110103760 2024-09-16T00:21:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.588+0000 7f21149d7640 1 -- 172.21.15.33:0/1894960810 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2104042070 con 0x7f2110103760 2024-09-16T00:21:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.589+0000 7f21149d7640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f20f80a3240 0x7f20f80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:21:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.589+0000 7f210effd640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f20f80a3240 0x7f20f80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:21:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.589+0000 7f21149d7640 1 -- 172.21.15.33:0/1894960810 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f21040f09a0 con 0x7f2110103760 2024-09-16T00:21:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.589+0000 7f210effd640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f20f80a3240 0x7f20f80a5700 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2100005fd0 tx=0x7f2100009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:21:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.592+0000 7f21149d7640 1 -- 172.21.15.33:0/1894960810 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2104031de0 con 0x7f2110103760 2024-09-16T00:21:59.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 262739 11.62 MB/sec execute 67 sec latency 58.234 ms 2024-09-16T00:21:59.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.803+0000 7f21159d9640 1 -- 172.21.15.33:0/1894960810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f211010a2c0 con 0x7f2110103760 2024-09-16T00:21:59.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.804+0000 7f21149d7640 1 -- 172.21.15.33:0/1894960810 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f21040bcb90 con 0x7f2110103760 2024-09-16T00:21:59.806 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:21:59.806 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:21:59.806 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:21:59.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.807+0000 7f21159d9640 1 -- 172.21.15.33:0/1894960810 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f20f80a3240 msgr2=0x7f20f80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:59.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.807+0000 7f21159d9640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f20f80a3240 0x7f20f80a5700 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2100005fd0 tx=0x7f2100009500 comp rx=0 tx=0).stop 2024-09-16T00:21:59.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.807+0000 7f21159d9640 1 -- 172.21.15.33:0/1894960810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110103760 msgr2=0x7f2110100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:21:59.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.807+0000 7f21159d9640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110103760 0x7f2110100f50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2104005940 tx=0x7f2104003c10 comp rx=0 tx=0).stop 2024-09-16T00:21:59.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.807+0000 7f21159d9640 1 -- 172.21.15.33:0/1894960810 shutdown_connections 2024-09-16T00:21:59.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.807+0000 7f21159d9640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f20f80a3240 0x7f20f80a5700 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:59.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.807+0000 7f21159d9640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21101082e0 0x7f2110101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:59.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.807+0000 7f21159d9640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110103760 0x7f2110100f50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:59.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.807+0000 7f21159d9640 1 --2- 172.21.15.33:0/1894960810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2110102db0 0x7f2110100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:21:59.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.807+0000 7f21159d9640 1 -- 172.21.15.33:0/1894960810 >> 172.21.15.33:0/1894960810 conn(0x7f21100fc7d0 msgr2=0x7f2110105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:21:59.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.807+0000 7f21159d9640 1 -- 172.21.15.33:0/1894960810 shutdown_connections 2024-09-16T00:21:59.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:21:59.807+0000 7f21159d9640 1 -- 172.21.15.33:0/1894960810 wait complete. 2024-09-16T00:22:00.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:00 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1894960810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:00.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:00 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1894960810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:00.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:00 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1894960810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:00.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 265678 11.65 MB/sec execute 68 sec latency 14.898 ms 2024-09-16T00:22:00.755 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:22:01.022 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:01.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:01 smithi083 ceph-mon[35242]: pgmap v496: 129 pgs: 129 active+clean; 118 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 2.4 MiB/s wr, 98 op/s 2024-09-16T00:22:01.430 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:01 smithi033 ceph-mon[31243]: pgmap v496: 129 pgs: 129 active+clean; 118 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 2.4 MiB/s wr, 98 op/s 2024-09-16T00:22:01.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:01 smithi060 ceph-mon[35185]: pgmap v496: 129 pgs: 129 active+clean; 118 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 2.4 MiB/s wr, 98 op/s 2024-09-16T00:22:01.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.516+0000 7f92e516d640 1 -- 172.21.15.33:0/1398299797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f92e0106f10 msgr2=0x7f92e010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:01.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.516+0000 7f92e516d640 1 --2- 172.21.15.33:0/1398299797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f92e0106f10 0x7f92e010ea90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009900 tx=0x7f92d802a410 comp rx=0 tx=0).stop 2024-09-16T00:22:01.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.516+0000 7f92e516d640 1 -- 172.21.15.33:0/1398299797 shutdown_connections 2024-09-16T00:22:01.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.516+0000 7f92e516d640 1 --2- 172.21.15.33:0/1398299797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f92e0106f10 0x7f92e010ea90 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.516+0000 7f92e516d640 1 --2- 172.21.15.33:0/1398299797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92e0102790 0x7f92e01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.516+0000 7f92e516d640 1 --2- 172.21.15.33:0/1398299797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e0101de0 0x7f92e01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.516+0000 7f92e516d640 1 -- 172.21.15.33:0/1398299797 >> 172.21.15.33:0/1398299797 conn(0x7f92e00fd8c0 msgr2=0x7f92e00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:01.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.516+0000 7f92e516d640 1 -- 172.21.15.33:0/1398299797 shutdown_connections 2024-09-16T00:22:01.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.516+0000 7f92e516d640 1 -- 172.21.15.33:0/1398299797 wait complete. 2024-09-16T00:22:01.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.516+0000 7f92e516d640 1 Processor -- start 2024-09-16T00:22:01.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92e516d640 1 -- start start 2024-09-16T00:22:01.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92e516d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f92e0101de0 0x7f92e00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92e516d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e0102790 0x7f92e0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92e516d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92e0106f10 0x7f92e0074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92e516d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f92e0111f70 con 0x7f92e0101de0 2024-09-16T00:22:01.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92e516d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f92e0111df0 con 0x7f92e0106f10 2024-09-16T00:22:01.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92e516d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f92e01120f0 con 0x7f92e0102790 2024-09-16T00:22:01.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92ded9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f92e0101de0 0x7f92e00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92ded9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f92e0101de0 0x7f92e00744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42896/0 (socket says 172.21.15.33:42896) 2024-09-16T00:22:01.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92ded9d640 1 -- 172.21.15.33:0/694464688 learned_addr learned my addr 172.21.15.33:0/694464688 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:01.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92de59c640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e0102790 0x7f92e0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92df59e640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92e0106f10 0x7f92e0074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92de59c640 1 -- 172.21.15.33:0/694464688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92e0106f10 msgr2=0x7f92e0074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:01.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92de59c640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92e0106f10 0x7f92e0074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92de59c640 1 -- 172.21.15.33:0/694464688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f92e0101de0 msgr2=0x7f92e00744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:01.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92de59c640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f92e0101de0 0x7f92e00744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92de59c640 1 -- 172.21.15.33:0/694464688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e01a6870 con 0x7f92e0102790 2024-09-16T00:22:01.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.517+0000 7f92df59e640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92e0106f10 0x7f92e0074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:01.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.518+0000 7f92de59c640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e0102790 0x7f92e0074a30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f92c800e9f0 tx=0x7f92c800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:01.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.518+0000 7f92d7fff640 1 -- 172.21.15.33:0/694464688 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c800ccc0 con 0x7f92e0102790 2024-09-16T00:22:01.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.518+0000 7f92e516d640 1 -- 172.21.15.33:0/694464688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92e01a6b60 con 0x7f92e0102790 2024-09-16T00:22:01.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.518+0000 7f92d7fff640 1 -- 172.21.15.33:0/694464688 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92c8010910 con 0x7f92e0102790 2024-09-16T00:22:01.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.518+0000 7f92d7fff640 1 -- 172.21.15.33:0/694464688 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c8018960 con 0x7f92e0102790 2024-09-16T00:22:01.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.518+0000 7f92e516d640 1 -- 172.21.15.33:0/694464688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92e01a70f0 con 0x7f92e0102790 2024-09-16T00:22:01.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.520+0000 7f92d7fff640 1 -- 172.21.15.33:0/694464688 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f92c8010ac0 con 0x7f92e0102790 2024-09-16T00:22:01.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.520+0000 7f92e7546740 1 Processor -- start 2024-09-16T00:22:01.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.520+0000 7f92e7546740 1 -- start start 2024-09-16T00:22:01.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92e7546740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a195b40 0x56458a194090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92e7546740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a1945d0 0x56458a194a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92e7546740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a19f370 0x56458a1a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92e7546740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56458a0fa2f0 con 0x56458a195b40 2024-09-16T00:22:01.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92e7546740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56458a0a1cf0 con 0x56458a1945d0 2024-09-16T00:22:01.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92e7546740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56458a0f62b0 con 0x56458a19f370 2024-09-16T00:22:01.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92df59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a195b40 0x56458a194090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92df59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a195b40 0x56458a194090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42908/0 (socket says 172.21.15.33:42908) 2024-09-16T00:22:01.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92df59e640 1 -- 172.21.15.33:0/4250047273 learned_addr learned my addr 172.21.15.33:0/4250047273 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:01.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92de59c640 1 --2- 172.21.15.33:0/4250047273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a19f370 0x56458a1a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92ded9d640 1 --2- 172.21.15.33:0/4250047273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a1945d0 0x56458a194a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92d7fff640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f92cc0a3240 0x7f92cc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92df59e640 1 -- 172.21.15.33:0/4250047273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a19f370 msgr2=0x56458a1a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:01.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92df59e640 1 --2- 172.21.15.33:0/4250047273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a19f370 0x56458a1a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92df59e640 1 -- 172.21.15.33:0/4250047273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a1945d0 msgr2=0x56458a194a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:01.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92df59e640 1 --2- 172.21.15.33:0/4250047273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a1945d0 0x56458a194a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92df59e640 1 -- 172.21.15.33:0/4250047273 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56458a0863a0 con 0x56458a195b40 2024-09-16T00:22:01.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.521+0000 7f92ded9d640 1 --2- 172.21.15.33:0/4250047273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a1945d0 0x56458a194a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:01.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.522+0000 7f92d7fff640 1 -- 172.21.15.33:0/694464688 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f92c80d01b0 con 0x7f92e0102790 2024-09-16T00:22:01.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.522+0000 7f92df59e640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f92cc0a3240 0x7f92cc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.522+0000 7f92df59e640 1 --2- 172.21.15.33:0/4250047273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a195b40 0x56458a194090 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f92d8033000 tx=0x7f92d8005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4e28456134edfb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:01.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.522+0000 7f92d57fa640 1 -- 172.21.15.33:0/4250047273 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d804b070 con 0x56458a195b40 2024-09-16T00:22:01.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.522+0000 7f92d57fa640 1 -- 172.21.15.33:0/4250047273 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92d8005040 con 0x56458a195b40 2024-09-16T00:22:01.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.522+0000 7f92d57fa640 1 -- 172.21.15.33:0/4250047273 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d803f070 con 0x56458a195b40 2024-09-16T00:22:01.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.523+0000 7f92e7546740 1 -- 172.21.15.33:0/4250047273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a195b40 msgr2=0x56458a194090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:01.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.523+0000 7f92df59e640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f92cc0a3240 0x7f92cc0a5700 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f92d8032fd0 tx=0x7f92d804b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.523+0000 7f92e7546740 1 --2- 172.21.15.33:0/4250047273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a195b40 0x56458a194090 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f92d8033000 tx=0x7f92d8005e60 comp rx=0 tx=0).stop 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.523+0000 7f92e7546740 1 -- 172.21.15.33:0/4250047273 shutdown_connections 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.523+0000 7f92e7546740 1 --2- 172.21.15.33:0/4250047273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a19f370 0x56458a1a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.523+0000 7f92e7546740 1 --2- 172.21.15.33:0/4250047273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a1945d0 0x56458a194a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.523+0000 7f92e7546740 1 --2- 172.21.15.33:0/4250047273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a195b40 0x56458a194090 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.523+0000 7f92e7546740 1 -- 172.21.15.33:0/4250047273 >> 172.21.15.33:0/4250047273 conn(0x56458a15d310 msgr2=0x56458a190c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.523+0000 7f92e7546740 1 -- 172.21.15.33:0/4250047273 shutdown_connections 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.523+0000 7f92e7546740 1 -- 172.21.15.33:0/4250047273 wait complete. 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.523+0000 7f92e7546740 1 -- start start 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.523+0000 7f92e7546740 1 Processor -- start 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92e7546740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 0x56458a19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92e7546740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a19c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92e7546740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a19f370 0x56458a19c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92e7546740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56458a1a2070 con 0x56458a19f370 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92e7546740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56458a0b5060 con 0x56458a195b40 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92e7546740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56458a1a21f0 con 0x56458a1945d0 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92e7546740 10 client.0 _ll_register_callbacks cb 0x564589e088c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92ded9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 0x56458a19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92ded9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 0x56458a19ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:54734/0 (socket says 172.21.15.33:54734) 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92ded9d640 1 -- 172.21.15.33:0/2178503544 learned_addr learned my addr 172.21.15.33:0/2178503544 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92de59c640 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a19c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92df59e640 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a19f370 0x56458a19c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92de59c640 1 -- 172.21.15.33:0/2178503544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 msgr2=0x56458a19ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92de59c640 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 0x56458a19ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92de59c640 1 -- 172.21.15.33:0/2178503544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a19f370 msgr2=0x56458a19c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:01.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92de59c640 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a19f370 0x56458a19c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92de59c640 1 -- 172.21.15.33:0/2178503544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564589e6b3c0 con 0x56458a195b40 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92ded9d640 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 0x56458a19ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92df59e640 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a19f370 0x56458a19c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92de59c640 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a19c040 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x56458a195070 tx=0x7f92c8098650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a48aa7bb83ec24ab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.524+0000 7f92c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92c67fc640 1 -- 172.21.15.33:0/2178503544 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c8096e60 con 0x56458a195b40 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92e7546740 1 -- 172.21.15.33:0/2178503544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92d8005040 con 0x56458a195b40 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92c67fc640 1 -- 172.21.15.33:0/2178503544 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f92c80d20f0 con 0x56458a195b40 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92c67fc640 1 -- 172.21.15.33:0/2178503544 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f92c809bc50 con 0x56458a195b40 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92e7546740 10 client.25784 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92e7546740 1 -- 172.21.15.33:0/2178503544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56458a1adbe0 con 0x56458a195b40 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92c67fc640 1 -- 172.21.15.33:0/2178503544 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92d8005040 con 0x56458a195b40 2024-09-16T00:22:01.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92c67fc640 1 -- 172.21.15.33:0/2178503544 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f92c809b740 con 0x56458a195b40 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92e7546740 10 client.25784 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92e7546740 10 client.25784 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92e7546740 20 client.25784 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92e7546740 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56458a1af040 0x56458a1b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92e7546740 4 client.25784 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92e7546740 1 -- 172.21.15.33:0/2178503544 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56458a107860 con 0x56458a1af040 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.525+0000 7f92ded9d640 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56458a1af040 0x56458a1b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.526+0000 7f92c67fc640 10 client.25784 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.526+0000 7f92ded9d640 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56458a1af040 0x56458a1b14a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c37fe00ff2a24da6 server_cookie=274967ae6b46ef8a in_seq=0 out_seq=0 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.527+0000 7f92c67fc640 1 -- 172.21.15.33:0/2178503544 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56458a107860 con 0x56458a1af040 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.527+0000 7f92c67fc640 10 client.25784 handle_command_reply: tid=0 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.527+0000 7f92e7546740 1 client.25784 shutdown 2024-09-16T00:22:01.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.528+0000 7f92e7546740 1 -- 172.21.15.33:0/2178503544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 msgr2=0x56458a19c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:01.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.528+0000 7f92e7546740 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a19c040 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x56458a195070 tx=0x7f92c8098650 comp rx=0 tx=0).stop 2024-09-16T00:22:01.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.528+0000 7f92e7546740 1 -- 172.21.15.33:0/2178503544 shutdown_connections 2024-09-16T00:22:01.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.528+0000 7f92e7546740 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56458a1af040 0x56458a1b14a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.528+0000 7f92e7546740 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a19f370 0x56458a19c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.528+0000 7f92c67fc640 0 client.25784 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:22:01.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.528+0000 7f92e7546740 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a19c040 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.528+0000 7f92e7546740 1 --2- 172.21.15.33:0/2178503544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 0x56458a19ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.528+0000 7f92e7546740 1 -- 172.21.15.33:0/2178503544 >> 172.21.15.33:0/2178503544 conn(0x56458a15d310 msgr2=0x56458a196930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:01.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.528+0000 7f92e7546740 1 -- 172.21.15.33:0/2178503544 shutdown_connections 2024-09-16T00:22:01.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.528+0000 7f92e7546740 1 -- 172.21.15.33:0/2178503544 wait complete. 2024-09-16T00:22:01.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.528+0000 7f92e7546740 20 client.25784 trim_cache size 0 max 16384 2024-09-16T00:22:01.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.541+0000 7f92e7546740 1 Processor -- start 2024-09-16T00:22:01.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.541+0000 7f92e7546740 1 -- start start 2024-09-16T00:22:01.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.541+0000 7f92e7546740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a1945d0 0x56458a1ab360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92e7546740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a19b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92e7546740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a19f370 0x56458a197120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92e7546740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f92e01b3b00 con 0x56458a1945d0 2024-09-16T00:22:01.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92e7546740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f92e01b13d0 con 0x56458a195b40 2024-09-16T00:22:01.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92e7546740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f92e01af220 con 0x56458a19f370 2024-09-16T00:22:01.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92ded9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a1945d0 0x56458a1ab360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92ded9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a1945d0 0x56458a1ab360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42928/0 (socket says 172.21.15.33:42928) 2024-09-16T00:22:01.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92ded9d640 1 -- 172.21.15.33:0/496494621 learned_addr learned my addr 172.21.15.33:0/496494621 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:01.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92df59e640 1 --2- 172.21.15.33:0/496494621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a19f370 0x56458a197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92de59c640 1 --2- 172.21.15.33:0/496494621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a19b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92ded9d640 1 -- 172.21.15.33:0/496494621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a19f370 msgr2=0x56458a197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:01.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92ded9d640 1 --2- 172.21.15.33:0/496494621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a19f370 0x56458a197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92ded9d640 1 -- 172.21.15.33:0/496494621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 msgr2=0x56458a19b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:01.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92ded9d640 1 --2- 172.21.15.33:0/496494621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a19b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92ded9d640 1 -- 172.21.15.33:0/496494621 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56458a19f100 con 0x56458a1945d0 2024-09-16T00:22:01.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92de59c640 1 --2- 172.21.15.33:0/496494621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a19b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:22:01.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92ded9d640 1 --2- 172.21.15.33:0/496494621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a1945d0 0x56458a1ab360 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x56458a1b1cb0 tx=0x7f92d0009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d41ce6d61a87649 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:01.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92d4ff9640 1 -- 172.21.15.33:0/496494621 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d0003b70 con 0x56458a1945d0 2024-09-16T00:22:01.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92d4ff9640 1 -- 172.21.15.33:0/496494621 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92d0010de0 con 0x56458a1945d0 2024-09-16T00:22:01.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.542+0000 7f92d4ff9640 1 -- 172.21.15.33:0/496494621 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d0004820 con 0x56458a1945d0 2024-09-16T00:22:01.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.543+0000 7f92e7546740 1 -- 172.21.15.33:0/496494621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a1945d0 msgr2=0x56458a1ab360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:01.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.543+0000 7f92e7546740 1 --2- 172.21.15.33:0/496494621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a1945d0 0x56458a1ab360 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x56458a1b1cb0 tx=0x7f92d0009500 comp rx=0 tx=0).stop 2024-09-16T00:22:01.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.543+0000 7f92e7546740 1 -- 172.21.15.33:0/496494621 shutdown_connections 2024-09-16T00:22:01.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.543+0000 7f92e7546740 1 --2- 172.21.15.33:0/496494621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a19f370 0x56458a197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.543+0000 7f92e7546740 1 --2- 172.21.15.33:0/496494621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a19b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.543+0000 7f92e7546740 1 --2- 172.21.15.33:0/496494621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a1945d0 0x56458a1ab360 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.543+0000 7f92e7546740 1 -- 172.21.15.33:0/496494621 >> 172.21.15.33:0/496494621 conn(0x56458a15d310 msgr2=0x56458a19b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:01.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.543+0000 7f92e7546740 1 -- 172.21.15.33:0/496494621 shutdown_connections 2024-09-16T00:22:01.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.543+0000 7f92e7546740 1 -- 172.21.15.33:0/496494621 wait complete. 2024-09-16T00:22:01.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92e7546740 1 -- start start 2024-09-16T00:22:01.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92e7546740 1 Processor -- start 2024-09-16T00:22:01.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92e7546740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 0x56458a1a74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92e7546740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a1a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92e7546740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a19f370 0x56458a198670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92e7546740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f92e01af890 con 0x56458a19f370 2024-09-16T00:22:01.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92e7546740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f92e01b0d00 con 0x56458a195b40 2024-09-16T00:22:01.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92e7546740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f92e01aff60 con 0x56458a1945d0 2024-09-16T00:22:01.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92e7546740 10 client.0 _ll_register_callbacks cb 0x7f92c80d38e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:01.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92ded9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 0x56458a1a74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92df59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a19f370 0x56458a198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92ded9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 0x56458a1a74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:54752/0 (socket says 172.21.15.33:54752) 2024-09-16T00:22:01.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92ded9d640 1 -- 172.21.15.33:0/1197426729 learned_addr learned my addr 172.21.15.33:0/1197426729 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:01.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92de59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a1a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92df59e640 1 -- 172.21.15.33:0/1197426729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 msgr2=0x56458a1a74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:01.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.544+0000 7f92df59e640 1 --2- 172.21.15.33:0/1197426729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 0x56458a1a74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92df59e640 1 -- 172.21.15.33:0/1197426729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 msgr2=0x56458a1a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:01.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92df59e640 1 --2- 172.21.15.33:0/1197426729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a1a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92df59e640 1 -- 172.21.15.33:0/1197426729 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56458a198bb0 con 0x56458a19f370 2024-09-16T00:22:01.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92de59c640 1 --2- 172.21.15.33:0/1197426729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a1a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:01.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92df59e640 1 --2- 172.21.15.33:0/1197426729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a19f370 0x56458a198670 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009900 tx=0x7f92d8005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa5683bca2944a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92c67fc640 1 -- 172.21.15.33:0/1197426729 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d8060060 con 0x56458a19f370 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92c67fc640 1 -- 172.21.15.33:0/1197426729 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f92d8004310 con 0x56458a19f370 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92e7546740 1 -- 172.21.15.33:0/1197426729 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92d0010de0 con 0x56458a19f370 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92c67fc640 1 -- 172.21.15.33:0/1197426729 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f92d8067050 con 0x56458a19f370 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92e7546740 10 client.16248 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92e7546740 1 -- 172.21.15.33:0/1197426729 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56458a219470 con 0x56458a19f370 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92c67fc640 1 -- 172.21.15.33:0/1197426729 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92d0010de0 con 0x56458a19f370 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92c67fc640 1 -- 172.21.15.33:0/1197426729 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f92d8003830 con 0x56458a19f370 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.545+0000 7f92e7546740 10 client.16248 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.546+0000 7f92e7546740 10 client.16248 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.546+0000 7f92e7546740 20 client.16248 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:01.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.546+0000 7f92e7546740 1 --2- 172.21.15.33:0/1197426729 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56458a2196a0 0x56458a21ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.546+0000 7f92e7546740 4 client.16248 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.546+0000 7f92e7546740 1 -- 172.21.15.33:0/1197426729 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56458a101d70 con 0x56458a2196a0 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.546+0000 7f92ded9d640 1 --2- 172.21.15.33:0/1197426729 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56458a2196a0 0x56458a21ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.546+0000 7f92ded9d640 1 --2- 172.21.15.33:0/1197426729 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56458a2196a0 0x56458a21ba90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f340c40a7533ca3 server_cookie=2b06bb8561e83486 in_seq=0 out_seq=0 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.546+0000 7f92c67fc640 10 client.16248 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.549+0000 7f92c67fc640 1 -- 172.21.15.33:0/1197426729 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56458a101d70 con 0x56458a2196a0 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.549+0000 7f92c67fc640 10 client.16248 handle_command_reply: tid=0 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.549+0000 7f92e7546740 1 client.16248 shutdown 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.549+0000 7f92e7546740 1 -- 172.21.15.33:0/1197426729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a19f370 msgr2=0x56458a198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.549+0000 7f92e7546740 1 --2- 172.21.15.33:0/1197426729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a19f370 0x56458a198670 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009900 tx=0x7f92d8005730 comp rx=0 tx=0).stop 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.549+0000 7f92e7546740 1 -- 172.21.15.33:0/1197426729 shutdown_connections 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.549+0000 7f92e7546740 1 --2- 172.21.15.33:0/1197426729 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56458a2196a0 0x56458a21ba90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.549+0000 7f92e7546740 1 --2- 172.21.15.33:0/1197426729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56458a19f370 0x56458a198670 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.549+0000 7f92e7546740 1 --2- 172.21.15.33:0/1197426729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56458a195b40 0x56458a1a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.549+0000 7f92c67fc640 0 client.16248 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:22:01.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.549+0000 7f92e7546740 1 --2- 172.21.15.33:0/1197426729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56458a1945d0 0x56458a1a74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.549+0000 7f92e7546740 1 -- 172.21.15.33:0/1197426729 >> 172.21.15.33:0/1197426729 conn(0x56458a15d310 msgr2=0x56458a18f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:01.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e7546740 1 -- 172.21.15.33:0/1197426729 shutdown_connections 2024-09-16T00:22:01.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e7546740 1 -- 172.21.15.33:0/1197426729 wait complete. 2024-09-16T00:22:01.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e7546740 20 client.16248 trim_cache size 0 max 16384 2024-09-16T00:22:01.560 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:22:01.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e516d640 1 -- 172.21.15.33:0/694464688 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f92cc0a3240 msgr2=0x7f92cc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:01.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e516d640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f92cc0a3240 0x7f92cc0a5700 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f92d8032fd0 tx=0x7f92d804b000 comp rx=0 tx=0).stop 2024-09-16T00:22:01.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e516d640 1 -- 172.21.15.33:0/694464688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e0102790 msgr2=0x7f92e0074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:01.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e516d640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e0102790 0x7f92e0074a30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f92c800e9f0 tx=0x7f92c800eeb0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e516d640 1 -- 172.21.15.33:0/694464688 shutdown_connections 2024-09-16T00:22:01.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e516d640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f92cc0a3240 0x7f92cc0a5700 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e516d640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92e0106f10 0x7f92e0074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e516d640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e0102790 0x7f92e0074a30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e516d640 1 --2- 172.21.15.33:0/694464688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f92e0101de0 0x7f92e00744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:01.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e516d640 1 -- 172.21.15.33:0/694464688 >> 172.21.15.33:0/694464688 conn(0x7f92e00fd8c0 msgr2=0x7f92e00fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:01.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e516d640 1 -- 172.21.15.33:0/694464688 shutdown_connections 2024-09-16T00:22:01.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:01.550+0000 7f92e516d640 1 -- 172.21.15.33:0/694464688 wait complete. 2024-09-16T00:22:01.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 267752 11.64 MB/sec execute 69 sec latency 54.053 ms 2024-09-16T00:22:02.459 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:22:02.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 270222 11.67 MB/sec execute 70 sec latency 73.323 ms 2024-09-16T00:22:02.788 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:03.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:03 smithi033 ceph-mon[31243]: pgmap v497: 129 pgs: 129 active+clean; 117 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 2.3 MiB/s wr, 88 op/s 2024-09-16T00:22:03.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.295+0000 7f0096237640 1 -- 172.21.15.33:0/1936325726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0090100100 msgr2=0x7f0090100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:03.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.295+0000 7f0096237640 1 --2- 172.21.15.33:0/1936325726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0090100100 0x7f0090100560 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f0084009900 tx=0x7f008402a410 comp rx=0 tx=0).stop 2024-09-16T00:22:03.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.296+0000 7f0096237640 1 -- 172.21.15.33:0/1936325726 shutdown_connections 2024-09-16T00:22:03.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.296+0000 7f0096237640 1 --2- 172.21.15.33:0/1936325726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0090100aa0 0x7f009010ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.296+0000 7f0096237640 1 --2- 172.21.15.33:0/1936325726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0090100100 0x7f0090100560 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.296+0000 7f0096237640 1 --2- 172.21.15.33:0/1936325726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0090106b80 0x7f0090106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.296+0000 7f0096237640 1 -- 172.21.15.33:0/1936325726 >> 172.21.15.33:0/1936325726 conn(0x7f0090077230 msgr2=0x7f00900fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:03.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.296+0000 7f0096237640 1 -- 172.21.15.33:0/1936325726 shutdown_connections 2024-09-16T00:22:03.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.296+0000 7f0096237640 1 -- 172.21.15.33:0/1936325726 wait complete. 2024-09-16T00:22:03.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.296+0000 7f0096237640 1 Processor -- start 2024-09-16T00:22:03.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f0096237640 1 -- start start 2024-09-16T00:22:03.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f0096237640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0090100100 0x7f00901012e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f0096237640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0090100aa0 0x7f0090101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f0096237640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0090106b80 0x7f0090101d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f0096237640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0090112050 con 0x7f0090100aa0 2024-09-16T00:22:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f0096237640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0090111ed0 con 0x7f0090100100 2024-09-16T00:22:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f0096237640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00901121d0 con 0x7f0090106b80 2024-09-16T00:22:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f008effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0090100aa0 0x7f0090101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f008effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0090100aa0 0x7f0090101820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42946/0 (socket says 172.21.15.33:42946) 2024-09-16T00:22:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f008effd640 1 -- 172.21.15.33:0/670855867 learned_addr learned my addr 172.21.15.33:0/670855867 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:03.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f008f7fe640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0090100100 0x7f00901012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f008ffff640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0090106b80 0x7f0090101d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f008f7fe640 1 -- 172.21.15.33:0/670855867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0090106b80 msgr2=0x7f0090101d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:03.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f008f7fe640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0090106b80 0x7f0090101d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f008f7fe640 1 -- 172.21.15.33:0/670855867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0090100aa0 msgr2=0x7f0090101820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:03.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.297+0000 7f008f7fe640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0090100aa0 0x7f0090101820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.298+0000 7f008f7fe640 1 -- 172.21.15.33:0/670855867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00901a69d0 con 0x7f0090100100 2024-09-16T00:22:03.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.298+0000 7f008effd640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0090100aa0 0x7f0090101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:03.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.298+0000 7f008f7fe640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0090100100 0x7f00901012e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f008000e9f0 tx=0x7f008000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:03.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.298+0000 7f008cff9640 1 -- 172.21.15.33:0/670855867 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008000ccc0 con 0x7f0090100100 2024-09-16T00:22:03.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.298+0000 7f008cff9640 1 -- 172.21.15.33:0/670855867 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0080005020 con 0x7f0090100100 2024-09-16T00:22:03.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.298+0000 7f0096237640 1 -- 172.21.15.33:0/670855867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00901a6cc0 con 0x7f0090100100 2024-09-16T00:22:03.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.298+0000 7f008cff9640 1 -- 172.21.15.33:0/670855867 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00800049e0 con 0x7f0090100100 2024-09-16T00:22:03.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.298+0000 7f0096237640 1 -- 172.21.15.33:0/670855867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00901a71b0 con 0x7f0090100100 2024-09-16T00:22:03.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.300+0000 7f008cff9640 1 -- 172.21.15.33:0/670855867 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0080005560 con 0x7f0090100100 2024-09-16T00:22:03.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f0098610740 1 Processor -- start 2024-09-16T00:22:03.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f0098610740 1 -- start start 2024-09-16T00:22:03.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f0098610740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa8b40 0x558496fa7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f0098610740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa75d0 0x558496fa7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f0098610740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fb2370 0x558496fb4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f0098610740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558496f0d2f0 con 0x558496fa8b40 2024-09-16T00:22:03.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f0098610740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558496eb4cf0 con 0x558496fa75d0 2024-09-16T00:22:03.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f0098610740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558496f092b0 con 0x558496fb2370 2024-09-16T00:22:03.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f008f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa75d0 0x558496fa7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f008ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa8b40 0x558496fa7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f008f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa75d0 0x558496fa7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:49360/0 (socket says 172.21.15.33:49360) 2024-09-16T00:22:03.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f008effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fb2370 0x558496fb4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.301+0000 7f008f7fe640 1 -- 172.21.15.33:0/2282274917 learned_addr learned my addr 172.21.15.33:0/2282274917 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:03.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.302+0000 7f008cff9640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00780a32c0 0x7f00780a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.302+0000 7f008f7fe640 1 -- 172.21.15.33:0/2282274917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fb2370 msgr2=0x558496fb4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:03.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.302+0000 7f008f7fe640 1 --2- 172.21.15.33:0/2282274917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fb2370 0x558496fb4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.302+0000 7f008f7fe640 1 -- 172.21.15.33:0/2282274917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa8b40 msgr2=0x558496fa7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:03.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.302+0000 7f008f7fe640 1 --2- 172.21.15.33:0/2282274917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa8b40 0x558496fa7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.302+0000 7f008f7fe640 1 -- 172.21.15.33:0/2282274917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558496e993a0 con 0x558496fa75d0 2024-09-16T00:22:03.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.302+0000 7f008cff9640 1 -- 172.21.15.33:0/670855867 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f00800cfcc0 con 0x7f0090100100 2024-09-16T00:22:03.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.302+0000 7f008ffff640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00780a32c0 0x7f00780a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.302+0000 7f008f7fe640 1 --2- 172.21.15.33:0/2282274917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa75d0 0x558496fa7a10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f008005e5d0 tx=0x7f00800054d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad097ad15e9973be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:03.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.302+0000 7f0075ffb640 1 -- 172.21.15.33:0/2282274917 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00800d1070 con 0x558496fa75d0 2024-09-16T00:22:03.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.303+0000 7f0075ffb640 1 -- 172.21.15.33:0/2282274917 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f008005c9a0 con 0x558496fa75d0 2024-09-16T00:22:03.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.303+0000 7f0075ffb640 1 -- 172.21.15.33:0/2282274917 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008001c030 con 0x558496fa75d0 2024-09-16T00:22:03.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.304+0000 7f0098610740 1 -- 172.21.15.33:0/2282274917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa75d0 msgr2=0x558496fa7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:03.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.304+0000 7f008ffff640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00780a32c0 0x7f00780a5780 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0088009fd0 tx=0x7f0088004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:03.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.304+0000 7f0098610740 1 --2- 172.21.15.33:0/2282274917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa75d0 0x558496fa7a10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f008005e5d0 tx=0x7f00800054d0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.304+0000 7f0098610740 1 -- 172.21.15.33:0/2282274917 shutdown_connections 2024-09-16T00:22:03.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.304+0000 7f0098610740 1 --2- 172.21.15.33:0/2282274917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fb2370 0x558496fb4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.304+0000 7f0098610740 1 --2- 172.21.15.33:0/2282274917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa75d0 0x558496fa7a10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.304+0000 7f0098610740 1 --2- 172.21.15.33:0/2282274917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa8b40 0x558496fa7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.304+0000 7f0098610740 1 -- 172.21.15.33:0/2282274917 >> 172.21.15.33:0/2282274917 conn(0x558496f70310 msgr2=0x558496fa3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:03.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.304+0000 7f0098610740 1 -- 172.21.15.33:0/2282274917 shutdown_connections 2024-09-16T00:22:03.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.304+0000 7f0098610740 1 -- 172.21.15.33:0/2282274917 wait complete. 2024-09-16T00:22:03.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.304+0000 7f0098610740 1 -- start start 2024-09-16T00:22:03.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f0098610740 1 Processor -- start 2024-09-16T00:22:03.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f0098610740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 0x558496fb1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f0098610740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa8b40 0x558496faefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f0098610740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fb2370 0x558496faf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f0098610740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558496fb5010 con 0x558496fa8b40 2024-09-16T00:22:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f0098610740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558496ec8060 con 0x558496fb2370 2024-09-16T00:22:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f0098610740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558496fb5190 con 0x558496fa75d0 2024-09-16T00:22:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f0098610740 10 client.0 _ll_register_callbacks cb 0x558496c1b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f008f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 0x558496fb1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f008f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 0x558496fb1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:54792/0 (socket says 172.21.15.33:54792) 2024-09-16T00:22:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f008f7fe640 1 -- 172.21.15.33:0/3668311097 learned_addr learned my addr 172.21.15.33:0/3668311097 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f008ffff640 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fb2370 0x558496faf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f008effd640 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa8b40 0x558496faefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f008f7fe640 1 -- 172.21.15.33:0/3668311097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fb2370 msgr2=0x558496faf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f008f7fe640 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fb2370 0x558496faf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f008f7fe640 1 -- 172.21.15.33:0/3668311097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa8b40 msgr2=0x558496faefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f008f7fe640 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa8b40 0x558496faefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.305+0000 7f008f7fe640 1 -- 172.21.15.33:0/3668311097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558496c7e3c0 con 0x558496fa75d0 2024-09-16T00:22:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.306+0000 7f008f7fe640 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 0x558496fb1d20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f00800cd390 tx=0x7f008009bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d42ce04e4bc46c52 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.306+0000 7f006effd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:22:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.306+0000 7f006effd640 1 -- 172.21.15.33:0/3668311097 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008001c040 con 0x558496fa75d0 2024-09-16T00:22:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.306+0000 7f006effd640 1 -- 172.21.15.33:0/3668311097 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f00800d3430 con 0x558496fa75d0 2024-09-16T00:22:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.306+0000 7f0098610740 1 -- 172.21.15.33:0/3668311097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558496f1a860 con 0x558496fa75d0 2024-09-16T00:22:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.306+0000 7f006effd640 1 -- 172.21.15.33:0/3668311097 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f008009b7c0 con 0x558496fa75d0 2024-09-16T00:22:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.306+0000 7f0098610740 10 client.25738 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.306+0000 7f0098610740 1 -- 172.21.15.33:0/3668311097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558496fb5010 con 0x558496fa75d0 2024-09-16T00:22:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.306+0000 7f006effd640 1 -- 172.21.15.33:0/3668311097 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558496f1a860 con 0x558496fa75d0 2024-09-16T00:22:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.306+0000 7f006effd640 1 -- 172.21.15.33:0/3668311097 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f0080018430 con 0x558496fa75d0 2024-09-16T00:22:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.307+0000 7f0098610740 10 client.25738 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.307+0000 7f0098610740 10 client.25738 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:22:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.307+0000 7f0098610740 20 client.25738 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.307+0000 7f0098610740 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x558496fc1be0 0x558496fc3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.307+0000 7f0098610740 4 client.25738 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.307+0000 7f0098610740 1 -- 172.21.15.33:0/3668311097 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558496f14d70 con 0x558496fc1be0 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.307+0000 7f008effd640 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x558496fc1be0 0x558496fc3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.307+0000 7f008effd640 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x558496fc1be0 0x558496fc3fd0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=850dee8717185c0c server_cookie=ee1611d49d84ab47 in_seq=0 out_seq=0 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.307+0000 7f006effd640 10 client.25738 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.308+0000 7f006effd640 1 -- 172.21.15.33:0/3668311097 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558496f14d70 con 0x558496fc1be0 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.308+0000 7f006effd640 10 client.25738 handle_command_reply: tid=0 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.308+0000 7f0098610740 1 client.25738 shutdown 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.308+0000 7f0098610740 1 -- 172.21.15.33:0/3668311097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 msgr2=0x558496fb1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.308+0000 7f0098610740 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 0x558496fb1d20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f00800cd390 tx=0x7f008009bd20 comp rx=0 tx=0).stop 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.308+0000 7f0098610740 1 -- 172.21.15.33:0/3668311097 shutdown_connections 2024-09-16T00:22:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.308+0000 7f0098610740 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x558496fc1be0 0x558496fc3fd0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.308+0000 7f006effd640 0 client.25738 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:22:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.308+0000 7f0098610740 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fb2370 0x558496faf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.308+0000 7f0098610740 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa8b40 0x558496faefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.309+0000 7f0098610740 1 --2- 172.21.15.33:0/3668311097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 0x558496fb1d20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.309+0000 7f0098610740 1 -- 172.21.15.33:0/3668311097 >> 172.21.15.33:0/3668311097 conn(0x558496f70310 msgr2=0x558496fa99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.309+0000 7f0098610740 1 -- 172.21.15.33:0/3668311097 shutdown_connections 2024-09-16T00:22:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.309+0000 7f0098610740 1 -- 172.21.15.33:0/3668311097 wait complete. 2024-09-16T00:22:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.309+0000 7f0098610740 20 client.25738 trim_cache size 0 max 16384 2024-09-16T00:22:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.333+0000 7f0098610740 1 Processor -- start 2024-09-16T00:22:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.333+0000 7f0098610740 1 -- start start 2024-09-16T00:22:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.333+0000 7f0098610740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa75d0 0x558496fbe2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.333+0000 7f0098610740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa8b40 0x558496fa2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.333+0000 7f0098610740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fb2370 0x558496fc9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.333+0000 7f0098610740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00901b3c30 con 0x558496fa75d0 2024-09-16T00:22:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.333+0000 7f0098610740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f00901b1500 con 0x558496fa8b40 2024-09-16T00:22:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.333+0000 7f0098610740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00901af350 con 0x558496fb2370 2024-09-16T00:22:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.334+0000 7f008f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa75d0 0x558496fbe2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.334+0000 7f008effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa8b40 0x558496fa2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.334+0000 7f008f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa75d0 0x558496fbe2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42988/0 (socket says 172.21.15.33:42988) 2024-09-16T00:22:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.334+0000 7f008f7fe640 1 -- 172.21.15.33:0/2728626299 learned_addr learned my addr 172.21.15.33:0/2728626299 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:03.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.334+0000 7f008effd640 1 -- 172.21.15.33:0/2728626299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fb2370 msgr2=0x558496fc9660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:22:03.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.334+0000 7f008effd640 1 --2- 172.21.15.33:0/2728626299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fb2370 0x558496fc9660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.334+0000 7f008effd640 1 -- 172.21.15.33:0/2728626299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa75d0 msgr2=0x558496fbe2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:03.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.334+0000 7f008effd640 1 --2- 172.21.15.33:0/2728626299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa75d0 0x558496fbe2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.334+0000 7f008effd640 1 -- 172.21.15.33:0/2728626299 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558496fa4960 con 0x558496fa8b40 2024-09-16T00:22:03.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.334+0000 7f008f7fe640 1 --2- 172.21.15.33:0/2728626299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa75d0 0x558496fbe2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:03.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.334+0000 7f008effd640 1 --2- 172.21.15.33:0/2728626299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa8b40 0x558496fa2bf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0090102d70 tx=0x7f0084005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c46d7d7a4edf0d3d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:03.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.335+0000 7f00757fa640 1 -- 172.21.15.33:0/2728626299 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008404a070 con 0x558496fa8b40 2024-09-16T00:22:03.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.335+0000 7f00757fa640 1 -- 172.21.15.33:0/2728626299 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0084032a00 con 0x558496fa8b40 2024-09-16T00:22:03.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.335+0000 7f00757fa640 1 -- 172.21.15.33:0/2728626299 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084035040 con 0x558496fa8b40 2024-09-16T00:22:03.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.336+0000 7f0098610740 1 -- 172.21.15.33:0/2728626299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa8b40 msgr2=0x558496fa2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:03.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.336+0000 7f0098610740 1 --2- 172.21.15.33:0/2728626299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa8b40 0x558496fa2bf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0090102d70 tx=0x7f0084005bd0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.336+0000 7f0098610740 1 -- 172.21.15.33:0/2728626299 shutdown_connections 2024-09-16T00:22:03.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.336+0000 7f0098610740 1 --2- 172.21.15.33:0/2728626299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fb2370 0x558496fc9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.336+0000 7f0098610740 1 --2- 172.21.15.33:0/2728626299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa8b40 0x558496fa2bf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.336+0000 7f0098610740 1 --2- 172.21.15.33:0/2728626299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fa75d0 0x558496fbe2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.336+0000 7f0098610740 1 -- 172.21.15.33:0/2728626299 >> 172.21.15.33:0/2728626299 conn(0x558496f70310 msgr2=0x558496fa24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:03.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.336+0000 7f0098610740 1 -- 172.21.15.33:0/2728626299 shutdown_connections 2024-09-16T00:22:03.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.336+0000 7f0098610740 1 -- 172.21.15.33:0/2728626299 wait complete. 2024-09-16T00:22:03.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.336+0000 7f0098610740 1 -- start start 2024-09-16T00:22:03.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.336+0000 7f0098610740 1 Processor -- start 2024-09-16T00:22:03.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f0098610740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 0x558496fbaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f0098610740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa8b40 0x558496fb6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f0098610740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fb2370 0x558496fb72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f0098610740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00901af9c0 con 0x558496fb2370 2024-09-16T00:22:03.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f0098610740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f00901b0e30 con 0x558496fa8b40 2024-09-16T00:22:03.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f0098610740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00901b0090 con 0x558496fa75d0 2024-09-16T00:22:03.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f0098610740 10 client.0 _ll_register_callbacks cb 0x7f0088009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:03.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f008ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fb2370 0x558496fb72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f008ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fb2370 0x558496fb72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42992/0 (socket says 172.21.15.33:42992) 2024-09-16T00:22:03.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f008ffff640 1 -- 172.21.15.33:0/2980668707 learned_addr learned my addr 172.21.15.33:0/2980668707 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:03.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f008f7fe640 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 0x558496fbaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f008effd640 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa8b40 0x558496fb6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f008ffff640 1 -- 172.21.15.33:0/2980668707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 msgr2=0x558496fbaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:03.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f008ffff640 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 0x558496fbaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f008ffff640 1 -- 172.21.15.33:0/2980668707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa8b40 msgr2=0x558496fb6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:03.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f008ffff640 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa8b40 0x558496fb6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f008ffff640 1 -- 172.21.15.33:0/2980668707 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558496faa120 con 0x558496fb2370 2024-09-16T00:22:03.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.337+0000 7f008f7fe640 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 0x558496fbaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:03.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f008ffff640 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fb2370 0x558496fb72e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f008800fb80 tx=0x7f0088017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa4aaa588eec5b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:03.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f006effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:03.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f006effd640 1 -- 172.21.15.33:0/2980668707 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008802e060 con 0x558496fb2370 2024-09-16T00:22:03.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f0098610740 1 -- 172.21.15.33:0/2980668707 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0084032a00 con 0x558496fb2370 2024-09-16T00:22:03.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f006effd640 1 -- 172.21.15.33:0/2980668707 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f008801ec40 con 0x558496fb2370 2024-09-16T00:22:03.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f006effd640 1 -- 172.21.15.33:0/2980668707 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0088035050 con 0x558496fb2370 2024-09-16T00:22:03.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f0098610740 10 client.16266 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:03.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f0098610740 1 -- 172.21.15.33:0/2980668707 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55849702d3b0 con 0x558496fb2370 2024-09-16T00:22:03.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f006effd640 1 -- 172.21.15.33:0/2980668707 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0084032a00 con 0x558496fb2370 2024-09-16T00:22:03.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f006effd640 1 -- 172.21.15.33:0/2980668707 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f008801b4b0 con 0x558496fb2370 2024-09-16T00:22:03.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f0098610740 10 client.16266 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:03.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f0098610740 10 client.16266 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:22:03.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f0098610740 20 client.16266 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:03.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.338+0000 7f0098610740 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55849702db30 0x55849702ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:03.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.339+0000 7f0098610740 4 client.16266 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:22:03.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.339+0000 7f0098610740 1 -- 172.21.15.33:0/2980668707 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558496e91170 con 0x55849702db30 2024-09-16T00:22:03.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.339+0000 7f008f7fe640 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55849702db30 0x55849702ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:03.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.339+0000 7f006effd640 10 client.16266 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:22:03.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f008f7fe640 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55849702db30 0x55849702ff50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8c808a0c63cea8c7 server_cookie=5b6fcc3aa3d49e72 in_seq=0 out_seq=0 2024-09-16T00:22:03.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f006effd640 1 -- 172.21.15.33:0/2980668707 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558496e91170 con 0x55849702db30 2024-09-16T00:22:03.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f006effd640 10 client.16266 handle_command_reply: tid=0 2024-09-16T00:22:03.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f0098610740 1 client.16266 shutdown 2024-09-16T00:22:03.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f0098610740 1 -- 172.21.15.33:0/2980668707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fb2370 msgr2=0x558496fb72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:03.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f0098610740 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fb2370 0x558496fb72e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f008800fb80 tx=0x7f0088017000 comp rx=0 tx=0).stop 2024-09-16T00:22:03.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f0098610740 1 -- 172.21.15.33:0/2980668707 shutdown_connections 2024-09-16T00:22:03.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f0098610740 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558496fb2370 0x558496fb72e0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f0098610740 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55849702db30 0x55849702ff50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f0098610740 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558496fa8b40 0x558496fb6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f006effd640 0 client.16266 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:22:03.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f0098610740 1 --2- 172.21.15.33:0/2980668707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558496fa75d0 0x558496fbaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f0098610740 1 -- 172.21.15.33:0/2980668707 >> 172.21.15.33:0/2980668707 conn(0x558496f70310 msgr2=0x558496fa9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:03.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f0098610740 1 -- 172.21.15.33:0/2980668707 shutdown_connections 2024-09-16T00:22:03.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.340+0000 7f0098610740 1 -- 172.21.15.33:0/2980668707 wait complete. 2024-09-16T00:22:03.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.341+0000 7f0098610740 20 client.16266 trim_cache size 0 max 16384 2024-09-16T00:22:03.351 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:22:03.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.341+0000 7f0096237640 1 -- 172.21.15.33:0/670855867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00780a32c0 msgr2=0x7f00780a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:03.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.341+0000 7f0096237640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00780a32c0 0x7f00780a5780 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0088009fd0 tx=0x7f0088004a40 comp rx=0 tx=0).stop 2024-09-16T00:22:03.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.341+0000 7f0096237640 1 -- 172.21.15.33:0/670855867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0090100100 msgr2=0x7f00901012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:03.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.341+0000 7f0096237640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0090100100 0x7f00901012e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f008000e9f0 tx=0x7f008000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.341+0000 7f0096237640 1 -- 172.21.15.33:0/670855867 shutdown_connections 2024-09-16T00:22:03.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.341+0000 7f0096237640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00780a32c0 0x7f00780a5780 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.341+0000 7f0096237640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0090106b80 0x7f0090101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.341+0000 7f0096237640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0090100aa0 0x7f0090101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.341+0000 7f0096237640 1 --2- 172.21.15.33:0/670855867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0090100100 0x7f00901012e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:03.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.341+0000 7f0096237640 1 -- 172.21.15.33:0/670855867 >> 172.21.15.33:0/670855867 conn(0x7f0090077230 msgr2=0x7f0090104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:03.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.342+0000 7f0096237640 1 -- 172.21.15.33:0/670855867 shutdown_connections 2024-09-16T00:22:03.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:03.342+0000 7f0096237640 1 -- 172.21.15.33:0/670855867 wait complete. 2024-09-16T00:22:03.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:03 smithi083 ceph-mon[35242]: pgmap v497: 129 pgs: 129 active+clean; 117 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 2.3 MiB/s wr, 88 op/s 2024-09-16T00:22:03.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:03 smithi060 ceph-mon[35185]: pgmap v497: 129 pgs: 129 active+clean; 117 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 2.3 MiB/s wr, 88 op/s 2024-09-16T00:22:03.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 273219 11.76 MB/sec execute 71 sec latency 39.641 ms 2024-09-16T00:22:04.201 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:22:04.495 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:04.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 275869 11.77 MB/sec execute 72 sec latency 12.743 ms 2024-09-16T00:22:05.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.009+0000 7f309f670640 1 -- 172.21.15.33:0/3767759050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3098102570 msgr2=0x7f30981029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:05.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.009+0000 7f309f670640 1 --2- 172.21.15.33:0/3767759050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3098102570 0x7f30981029b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f3094009900 tx=0x7f309402a480 comp rx=0 tx=0).stop 2024-09-16T00:22:05.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.009+0000 7f309f670640 1 -- 172.21.15.33:0/3767759050 shutdown_connections 2024-09-16T00:22:05.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.009+0000 7f309f670640 1 --2- 172.21.15.33:0/3767759050 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3098102ef0 0x7f309810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.009+0000 7f309f670640 1 --2- 172.21.15.33:0/3767759050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3098102570 0x7f30981029b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.009+0000 7f309f670640 1 --2- 172.21.15.33:0/3767759050 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3098100e20 0x7f3098101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.009+0000 7f309f670640 1 -- 172.21.15.33:0/3767759050 >> 172.21.15.33:0/3767759050 conn(0x7f30980fc7d0 msgr2=0x7f30980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:05.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.009+0000 7f309f670640 1 -- 172.21.15.33:0/3767759050 shutdown_connections 2024-09-16T00:22:05.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.009+0000 7f309f670640 1 -- 172.21.15.33:0/3767759050 wait complete. 2024-09-16T00:22:05.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.010+0000 7f309f670640 1 Processor -- start 2024-09-16T00:22:05.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.010+0000 7f309f670640 1 -- start start 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.010+0000 7f309f670640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3098100e20 0x7f30980fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.010+0000 7f309f670640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3098102570 0x7f30980fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.010+0000 7f309f670640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3098102ef0 0x7f30980fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.010+0000 7f309f670640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3098110f80 con 0x7f3098102ef0 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.010+0000 7f309f670640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3098110e00 con 0x7f3098100e20 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.010+0000 7f309f670640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3098111100 con 0x7f3098102570 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.010+0000 7f309cc0b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3098102570 0x7f30980fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.010+0000 7f309cc0b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3098102570 0x7f30980fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:54840/0 (socket says 172.21.15.33:54840) 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.010+0000 7f309cc0b640 1 -- 172.21.15.33:0/3568444257 learned_addr learned my addr 172.21.15.33:0/3568444257 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.010+0000 7f309dc0d640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3098102ef0 0x7f30980fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f309d40c640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3098100e20 0x7f30980fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f309cc0b640 1 -- 172.21.15.33:0/3568444257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3098100e20 msgr2=0x7f30980fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:05.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f309cc0b640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3098100e20 0x7f30980fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f309cc0b640 1 -- 172.21.15.33:0/3568444257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3098102ef0 msgr2=0x7f30980fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f309cc0b640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3098102ef0 0x7f30980fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f309cc0b640 1 -- 172.21.15.33:0/3568444257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30981a6970 con 0x7f3098102570 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f309d40c640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3098100e20 0x7f30980fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f309cc0b640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3098102570 0x7f30980fd1c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3094006010 tx=0x7f3094003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f308e7fc640 1 -- 172.21.15.33:0/3568444257 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309403f070 con 0x7f3098102570 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f309f670640 1 -- 172.21.15.33:0/3568444257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30981a6c00 con 0x7f3098102570 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f308e7fc640 1 -- 172.21.15.33:0/3568444257 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3094031c30 con 0x7f3098102570 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f308e7fc640 1 -- 172.21.15.33:0/3568444257 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3094035030 con 0x7f3098102570 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.011+0000 7f309f670640 1 -- 172.21.15.33:0/3568444257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30981a7130 con 0x7f3098102570 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.014+0000 7f308e7fc640 1 -- 172.21.15.33:0/3568444257 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3094042070 con 0x7f3098102570 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.014+0000 7f30a1a49740 1 Processor -- start 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.014+0000 7f30a1a49740 1 -- start start 2024-09-16T00:22:05.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.014+0000 7f30a1a49740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7d1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.014+0000 7f30a1a49740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7d15d0 0x56415d7d1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.014+0000 7f30a1a49740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7dc370 0x56415d7de760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.014+0000 7f30a1a49740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56415d7372f0 con 0x56415d7dc370 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.014+0000 7f30a1a49740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56415d6decf0 con 0x56415d7d2b40 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.014+0000 7f30a1a49740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56415d7332b0 con 0x56415d7d15d0 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309cc0b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7dc370 0x56415d7de760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309d40c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7d15d0 0x56415d7d1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309d40c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7d15d0 0x56415d7d1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:54848/0 (socket says 172.21.15.33:54848) 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309d40c640 1 -- 172.21.15.33:0/1621952528 learned_addr learned my addr 172.21.15.33:0/1621952528 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309dc0d640 1 --2- 172.21.15.33:0/1621952528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7d1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f308e7fc640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30800a3240 0x7f30800a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309dc0d640 1 -- 172.21.15.33:0/1621952528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7d15d0 msgr2=0x56415d7d1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:05.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309dc0d640 1 --2- 172.21.15.33:0/1621952528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7d15d0 0x56415d7d1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309dc0d640 1 -- 172.21.15.33:0/1621952528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7dc370 msgr2=0x56415d7de760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309dc0d640 1 --2- 172.21.15.33:0/1621952528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7dc370 0x56415d7de760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309dc0d640 1 -- 172.21.15.33:0/1621952528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56415d6c33a0 con 0x56415d7d2b40 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f308e7fc640 1 -- 172.21.15.33:0/3568444257 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f30940f0470 con 0x7f3098102570 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309dc0d640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30800a3240 0x7f30800a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309d40c640 1 --2- 172.21.15.33:0/1621952528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7d15d0 0x56415d7d1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.015+0000 7f309dc0d640 1 --2- 172.21.15.33:0/1621952528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7d1090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3090009b80 tx=0x7f3090011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6554f2b6c2c33c70 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.016+0000 7f307f7fe640 1 -- 172.21.15.33:0/1621952528 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090019580 con 0x56415d7d2b40 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.016+0000 7f307f7fe640 1 -- 172.21.15.33:0/1621952528 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30900189d0 con 0x56415d7d2b40 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.016+0000 7f307f7fe640 1 -- 172.21.15.33:0/1621952528 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309000e7d0 con 0x56415d7d2b40 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.017+0000 7f30a1a49740 1 -- 172.21.15.33:0/1621952528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 msgr2=0x56415d7d1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.017+0000 7f30a1a49740 1 --2- 172.21.15.33:0/1621952528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7d1090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3090009b80 tx=0x7f3090011810 comp rx=0 tx=0).stop 2024-09-16T00:22:05.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.017+0000 7f309dc0d640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30800a3240 0x7f30800a5700 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3090020000 tx=0x7f3090021000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.017+0000 7f30a1a49740 1 -- 172.21.15.33:0/1621952528 shutdown_connections 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.017+0000 7f30a1a49740 1 --2- 172.21.15.33:0/1621952528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7dc370 0x56415d7de760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.017+0000 7f30a1a49740 1 --2- 172.21.15.33:0/1621952528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7d15d0 0x56415d7d1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.017+0000 7f30a1a49740 1 --2- 172.21.15.33:0/1621952528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7d1090 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.017+0000 7f30a1a49740 1 -- 172.21.15.33:0/1621952528 >> 172.21.15.33:0/1621952528 conn(0x56415d79a310 msgr2=0x56415d7cdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.017+0000 7f30a1a49740 1 -- 172.21.15.33:0/1621952528 shutdown_connections 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.017+0000 7f30a1a49740 1 -- 172.21.15.33:0/1621952528 wait complete. 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.017+0000 7f30a1a49740 1 -- start start 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.017+0000 7f30a1a49740 1 Processor -- start 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f30a1a49740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 0x56415d7dbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f30a1a49740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7d8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f30a1a49740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7dc370 0x56415d7d94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f30a1a49740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56415d7df070 con 0x56415d7d15d0 2024-09-16T00:22:05.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f30a1a49740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56415d6f2060 con 0x56415d7d2b40 2024-09-16T00:22:05.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f30a1a49740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56415d7df1f0 con 0x56415d7dc370 2024-09-16T00:22:05.060 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:22:05.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f30a1a49740 10 client.0 _ll_register_callbacks cb 0x56415d4458c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:05.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f309d40c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 0x56415d7dbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f309dc0d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7dc370 0x56415d7d94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f309d40c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 0x56415d7dbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43026/0 (socket says 172.21.15.33:43026) 2024-09-16T00:22:05.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f309cc0b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7d8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f309d40c640 1 -- 172.21.15.33:0/2919806007 learned_addr learned my addr 172.21.15.33:0/2919806007 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:05.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f309cc0b640 1 -- 172.21.15.33:0/2919806007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7dc370 msgr2=0x56415d7d94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:05.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f309cc0b640 1 --2- 172.21.15.33:0/2919806007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7dc370 0x56415d7d94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f309cc0b640 1 -- 172.21.15.33:0/2919806007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 msgr2=0x56415d7dbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f309cc0b640 1 --2- 172.21.15.33:0/2919806007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 0x56415d7dbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f309cc0b640 1 -- 172.21.15.33:0/2919806007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56415d4a83c0 con 0x56415d7d2b40 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f309dc0d640 1 --2- 172.21.15.33:0/2919806007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7dc370 0x56415d7d94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f309cc0b640 1 --2- 172.21.15.33:0/2919806007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7d8f70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f30940edb40 tx=0x7f3094009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5bc397bf9d66d29 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.018+0000 7f307cff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f307cff9640 1 -- 172.21.15.33:0/2919806007 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3094049070 con 0x56415d7d2b40 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f30a1a49740 1 -- 172.21.15.33:0/2919806007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30900189d0 con 0x56415d7d2b40 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f307cff9640 1 -- 172.21.15.33:0/2919806007 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f30940f0170 con 0x56415d7d2b40 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f307cff9640 1 -- 172.21.15.33:0/2919806007 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f30940bc150 con 0x56415d7d2b40 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f30a1a49740 10 client.25817 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f30a1a49740 1 -- 172.21.15.33:0/2919806007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56415d7eaa80 con 0x56415d7d2b40 2024-09-16T00:22:05.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f307cff9640 1 -- 172.21.15.33:0/2919806007 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30900189d0 con 0x56415d7d2b40 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f307cff9640 1 -- 172.21.15.33:0/2919806007 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f30940b8c80 con 0x56415d7d2b40 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f30a1a49740 10 client.25817 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f30a1a49740 10 client.25817 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f30a1a49740 20 client.25817 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f30a1a49740 1 --2- 172.21.15.33:0/2919806007 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56415d7ebe70 0x56415d7ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f30a1a49740 4 client.25817 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f30a1a49740 1 -- 172.21.15.33:0/2919806007 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56415d744860 con 0x56415d7ebe70 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.019+0000 7f309d40c640 1 --2- 172.21.15.33:0/2919806007 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56415d7ebe70 0x56415d7ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.020+0000 7f307cff9640 10 client.25817 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.020+0000 7f309d40c640 1 --2- 172.21.15.33:0/2919806007 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56415d7ebe70 0x56415d7ee2b0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c93bb349b82045a server_cookie=59ff288f9587e06 in_seq=0 out_seq=0 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.020+0000 7f307cff9640 1 -- 172.21.15.33:0/2919806007 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56415d744860 con 0x56415d7ebe70 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f307cff9640 10 client.25817 handle_command_reply: tid=0 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f30a1a49740 1 client.25817 shutdown 2024-09-16T00:22:05.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f30a1a49740 1 -- 172.21.15.33:0/2919806007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 msgr2=0x56415d7d8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f30a1a49740 1 --2- 172.21.15.33:0/2919806007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7d8f70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f30940edb40 tx=0x7f3094009560 comp rx=0 tx=0).stop 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f30a1a49740 1 -- 172.21.15.33:0/2919806007 shutdown_connections 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f30a1a49740 1 --2- 172.21.15.33:0/2919806007 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56415d7ebe70 0x56415d7ee2b0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f30a1a49740 1 --2- 172.21.15.33:0/2919806007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7dc370 0x56415d7d94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f30a1a49740 1 --2- 172.21.15.33:0/2919806007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7d8f70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f307cff9640 0 client.25817 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f30a1a49740 1 --2- 172.21.15.33:0/2919806007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 0x56415d7dbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f30a1a49740 1 -- 172.21.15.33:0/2919806007 >> 172.21.15.33:0/2919806007 conn(0x56415d79a310 msgr2=0x56415d7d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f30a1a49740 1 -- 172.21.15.33:0/2919806007 shutdown_connections 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f30a1a49740 1 -- 172.21.15.33:0/2919806007 wait complete. 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.021+0000 7f30a1a49740 20 client.25817 trim_cache size 0 max 16384 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f30a1a49740 1 Processor -- start 2024-09-16T00:22:05.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f30a1a49740 1 -- start start 2024-09-16T00:22:05.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f30a1a49740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d15d0 0x56415d7e8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f30a1a49740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7d2b40 0x56415d7d7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f30a1a49740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7dc370 0x56415d7d7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f30a1a49740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f30981b3ac0 con 0x56415d7dc370 2024-09-16T00:22:05.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f30a1a49740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f30981b1390 con 0x56415d7d15d0 2024-09-16T00:22:05.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f30a1a49740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f30981af1e0 con 0x56415d7d2b40 2024-09-16T00:22:05.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f309dc0d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7dc370 0x56415d7d7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f309cc0b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7d2b40 0x56415d7d7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f309dc0d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7dc370 0x56415d7d7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43038/0 (socket says 172.21.15.33:43038) 2024-09-16T00:22:05.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f309dc0d640 1 -- 172.21.15.33:0/4008726903 learned_addr learned my addr 172.21.15.33:0/4008726903 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:05.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.036+0000 7f309d40c640 1 --2- 172.21.15.33:0/4008726903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d15d0 0x56415d7e8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.037+0000 7f309dc0d640 1 -- 172.21.15.33:0/4008726903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7d2b40 msgr2=0x56415d7d7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:05.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.037+0000 7f309dc0d640 1 --2- 172.21.15.33:0/4008726903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7d2b40 0x56415d7d7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.037+0000 7f309dc0d640 1 -- 172.21.15.33:0/4008726903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d15d0 msgr2=0x56415d7e8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:05.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.037+0000 7f309dc0d640 1 --2- 172.21.15.33:0/4008726903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d15d0 0x56415d7e8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.037+0000 7f309dc0d640 1 -- 172.21.15.33:0/4008726903 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56415d7ce960 con 0x56415d7dc370 2024-09-16T00:22:05.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.037+0000 7f309d40c640 1 --2- 172.21.15.33:0/4008726903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d15d0 0x56415d7e8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:05.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.037+0000 7f309dc0d640 1 --2- 172.21.15.33:0/4008726903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7dc370 0x56415d7d7d80 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f309000e460 tx=0x7f3090026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eeec69360a3a4fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:05.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.037+0000 7f307effd640 1 -- 172.21.15.33:0/4008726903 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090032020 con 0x56415d7dc370 2024-09-16T00:22:05.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.037+0000 7f307effd640 1 -- 172.21.15.33:0/4008726903 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3090009780 con 0x56415d7dc370 2024-09-16T00:22:05.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.037+0000 7f307effd640 1 -- 172.21.15.33:0/4008726903 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090005500 con 0x56415d7dc370 2024-09-16T00:22:05.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.038+0000 7f30a1a49740 1 -- 172.21.15.33:0/4008726903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7dc370 msgr2=0x56415d7d7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:05.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.038+0000 7f30a1a49740 1 --2- 172.21.15.33:0/4008726903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7dc370 0x56415d7d7d80 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f309000e460 tx=0x7f3090026000 comp rx=0 tx=0).stop 2024-09-16T00:22:05.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.038+0000 7f30a1a49740 1 -- 172.21.15.33:0/4008726903 shutdown_connections 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.038+0000 7f30a1a49740 1 --2- 172.21.15.33:0/4008726903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7dc370 0x56415d7d7d80 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.038+0000 7f30a1a49740 1 --2- 172.21.15.33:0/4008726903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7d2b40 0x56415d7d7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.038+0000 7f30a1a49740 1 --2- 172.21.15.33:0/4008726903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d15d0 0x56415d7e8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.038+0000 7f30a1a49740 1 -- 172.21.15.33:0/4008726903 >> 172.21.15.33:0/4008726903 conn(0x56415d79a310 msgr2=0x56415d7d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.038+0000 7f30a1a49740 1 -- 172.21.15.33:0/4008726903 shutdown_connections 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.038+0000 7f30a1a49740 1 -- 172.21.15.33:0/4008726903 wait complete. 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.038+0000 7f30a1a49740 1 -- start start 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.038+0000 7f30a1a49740 1 Processor -- start 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f30a1a49740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 0x56415d7e4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f30a1a49740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7dfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f30a1a49740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7dc370 0x56415d7e0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f30a1a49740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f30981af850 con 0x56415d7d15d0 2024-09-16T00:22:05.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f30a1a49740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f30981b0cc0 con 0x56415d7d2b40 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f30a1a49740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f30981aff20 con 0x56415d7dc370 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f30a1a49740 10 client.0 _ll_register_callbacks cb 0x7f3094005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f309d40c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 0x56415d7e4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f309dc0d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7dc370 0x56415d7e0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f309d40c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 0x56415d7e4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43052/0 (socket says 172.21.15.33:43052) 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f309d40c640 1 -- 172.21.15.33:0/3911757796 learned_addr learned my addr 172.21.15.33:0/3911757796 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f309cc0b640 1 --2- 172.21.15.33:0/3911757796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7dfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f309d40c640 1 -- 172.21.15.33:0/3911757796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7dc370 msgr2=0x56415d7e0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f309d40c640 1 --2- 172.21.15.33:0/3911757796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7dc370 0x56415d7e0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f309d40c640 1 -- 172.21.15.33:0/3911757796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 msgr2=0x56415d7dfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f309d40c640 1 --2- 172.21.15.33:0/3911757796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7dfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f309d40c640 1 -- 172.21.15.33:0/3911757796 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56415d7e0bc0 con 0x56415d7d15d0 2024-09-16T00:22:05.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f309d40c640 1 --2- 172.21.15.33:0/3911757796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 0x56415d7e4b20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f30880049d0 tx=0x7f3088012d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b4b966ed1f0bb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.039+0000 7f307cff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f307cff9640 1 -- 172.21.15.33:0/3911757796 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f308802b060 con 0x56415d7d15d0 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f307cff9640 1 -- 172.21.15.33:0/3911757796 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3088010cf0 con 0x56415d7d15d0 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f30a1a49740 1 -- 172.21.15.33:0/3911757796 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3090009780 con 0x56415d7d15d0 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f307cff9640 1 -- 172.21.15.33:0/3911757796 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3088026460 con 0x56415d7d15d0 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f30a1a49740 10 client.16284 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f30a1a49740 1 -- 172.21.15.33:0/3911757796 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56415d856b20 con 0x56415d7d15d0 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f307cff9640 1 -- 172.21.15.33:0/3911757796 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3090009780 con 0x56415d7d15d0 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f307cff9640 1 -- 172.21.15.33:0/3911757796 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f308801e810 con 0x56415d7d15d0 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f30a1a49740 10 client.16284 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f30a1a49740 10 client.16284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f30a1a49740 20 client.16284 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f30a1a49740 1 --2- 172.21.15.33:0/3911757796 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56415d857240 0x56415d859660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f30a1a49740 4 client.16284 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f30a1a49740 1 -- 172.21.15.33:0/3911757796 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56415d73ed70 con 0x56415d857240 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.040+0000 7f309cc0b640 1 --2- 172.21.15.33:0/3911757796 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56415d857240 0x56415d859660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f309cc0b640 1 --2- 172.21.15.33:0/3911757796 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56415d857240 0x56415d859660 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8096deba5f97df35 server_cookie=b01380b46082537a in_seq=0 out_seq=0 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f307cff9640 10 client.16284 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f307cff9640 1 -- 172.21.15.33:0/3911757796 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56415d73ed70 con 0x56415d857240 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f307cff9640 10 client.16284 handle_command_reply: tid=0 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f30a1a49740 1 client.16284 shutdown 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f30a1a49740 1 -- 172.21.15.33:0/3911757796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 msgr2=0x56415d7e4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f30a1a49740 1 --2- 172.21.15.33:0/3911757796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 0x56415d7e4b20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f30880049d0 tx=0x7f3088012d60 comp rx=0 tx=0).stop 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f30a1a49740 1 -- 172.21.15.33:0/3911757796 shutdown_connections 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f30a1a49740 1 --2- 172.21.15.33:0/3911757796 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56415d857240 0x56415d859660 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f30a1a49740 1 --2- 172.21.15.33:0/3911757796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56415d7dc370 0x56415d7e0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f30a1a49740 1 --2- 172.21.15.33:0/3911757796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56415d7d2b40 0x56415d7dfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f30a1a49740 1 --2- 172.21.15.33:0/3911757796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56415d7d15d0 0x56415d7e4b20 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f307cff9640 0 client.16284 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f30a1a49740 1 -- 172.21.15.33:0/3911757796 >> 172.21.15.33:0/3911757796 conn(0x56415d79a310 msgr2=0x56415d7d7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f30a1a49740 1 -- 172.21.15.33:0/3911757796 shutdown_connections 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.041+0000 7f30a1a49740 1 -- 172.21.15.33:0/3911757796 wait complete. 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f30a1a49740 20 client.16284 trim_cache size 0 max 16384 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f309f670640 1 -- 172.21.15.33:0/3568444257 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30800a3240 msgr2=0x7f30800a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f309f670640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30800a3240 0x7f30800a5700 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3090020000 tx=0x7f3090021000 comp rx=0 tx=0).stop 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f309f670640 1 -- 172.21.15.33:0/3568444257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3098102570 msgr2=0x7f30980fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f309f670640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3098102570 0x7f30980fd1c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3094006010 tx=0x7f3094003c10 comp rx=0 tx=0).stop 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f309f670640 1 -- 172.21.15.33:0/3568444257 shutdown_connections 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f309f670640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30800a3240 0x7f30800a5700 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f309f670640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3098102ef0 0x7f30980fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f309f670640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3098102570 0x7f30980fd1c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f309f670640 1 --2- 172.21.15.33:0/3568444257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3098100e20 0x7f30980fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:05.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f309f670640 1 -- 172.21.15.33:0/3568444257 >> 172.21.15.33:0/3568444257 conn(0x7f30980fc7d0 msgr2=0x7f3098069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:05.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f309f670640 1 -- 172.21.15.33:0/3568444257 shutdown_connections 2024-09-16T00:22:05.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:05.042+0000 7f309f670640 1 -- 172.21.15.33:0/3568444257 wait complete. 2024-09-16T00:22:05.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:05 smithi033 ceph-mon[31243]: pgmap v498: 129 pgs: 129 active+clean; 117 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.8 MiB/s wr, 158 op/s 2024-09-16T00:22:05.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:05 smithi083 ceph-mon[35242]: pgmap v498: 129 pgs: 129 active+clean; 117 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.8 MiB/s wr, 158 op/s 2024-09-16T00:22:05.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:05 smithi060 ceph-mon[35185]: pgmap v498: 129 pgs: 129 active+clean; 117 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.8 MiB/s wr, 158 op/s 2024-09-16T00:22:05.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 278145 11.79 MB/sec execute 73 sec latency 37.972 ms 2024-09-16T00:22:06.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 280862 11.82 MB/sec execute 74 sec latency 46.998 ms 2024-09-16T00:22:07.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:07 smithi083 ceph-mon[35242]: pgmap v499: 129 pgs: 129 active+clean; 117 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 174 op/s 2024-09-16T00:22:07.458 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:22:07.459 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:22:07.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:07 smithi060 ceph-mon[35185]: pgmap v499: 129 pgs: 129 active+clean; 117 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 174 op/s 2024-09-16T00:22:07.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:07 smithi033 ceph-mon[31243]: pgmap v499: 129 pgs: 129 active+clean; 117 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 174 op/s 2024-09-16T00:22:07.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 283847 11.90 MB/sec execute 75 sec latency 117.877 ms 2024-09-16T00:22:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:08.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.289+0000 7f0cd239d640 1 -- 172.21.15.33:0/2398665014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ccc101270 msgr2=0x7f0ccc10d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:08.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.289+0000 7f0cd239d640 1 --2- 172.21.15.33:0/2398665014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ccc101270 0x7f0ccc10d7a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009930 tx=0x7f0cc402a420 comp rx=0 tx=0).stop 2024-09-16T00:22:08.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.289+0000 7f0cd239d640 1 -- 172.21.15.33:0/2398665014 shutdown_connections 2024-09-16T00:22:08.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.289+0000 7f0cd239d640 1 --2- 172.21.15.33:0/2398665014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ccc101270 0x7f0ccc10d7a0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.289+0000 7f0cd239d640 1 --2- 172.21.15.33:0/2398665014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ccc1008d0 0x7f0ccc100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.289+0000 7f0cd239d640 1 --2- 172.21.15.33:0/2398665014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ccc1068d0 0x7f0ccc106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.289+0000 7f0cd239d640 1 -- 172.21.15.33:0/2398665014 >> 172.21.15.33:0/2398665014 conn(0x7f0ccc0fc610 msgr2=0x7f0ccc0fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.289+0000 7f0cd239d640 1 -- 172.21.15.33:0/2398665014 shutdown_connections 2024-09-16T00:22:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.289+0000 7f0cd239d640 1 -- 172.21.15.33:0/2398665014 wait complete. 2024-09-16T00:22:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd239d640 1 Processor -- start 2024-09-16T00:22:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd239d640 1 -- start start 2024-09-16T00:22:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd239d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ccc1008d0 0x7f0ccc198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd239d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ccc101270 0x7f0ccc198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd239d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ccc1068d0 0x7f0ccc19cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd239d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc110d20 con 0x7f0ccc1008d0 2024-09-16T00:22:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd239d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc110ba0 con 0x7f0ccc101270 2024-09-16T00:22:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd239d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc110ea0 con 0x7f0ccc1068d0 2024-09-16T00:22:08.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd139b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ccc1008d0 0x7f0ccc198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd139b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ccc1008d0 0x7f0ccc198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43066/0 (socket says 172.21.15.33:43066) 2024-09-16T00:22:08.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd139b640 1 -- 172.21.15.33:0/874764708 learned_addr learned my addr 172.21.15.33:0/874764708 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:08.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd1b9c640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ccc1068d0 0x7f0ccc19cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ccc101270 0x7f0ccc198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.290+0000 7f0cd139b640 1 -- 172.21.15.33:0/874764708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ccc1068d0 msgr2=0x7f0ccc19cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:08.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.291+0000 7f0cd139b640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ccc1068d0 0x7f0ccc19cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.291+0000 7f0cd139b640 1 -- 172.21.15.33:0/874764708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ccc101270 msgr2=0x7f0ccc198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:08.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.291+0000 7f0cd139b640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ccc101270 0x7f0ccc198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.291+0000 7f0cd139b640 1 -- 172.21.15.33:0/874764708 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ccc19d1b0 con 0x7f0ccc1008d0 2024-09-16T00:22:08.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.291+0000 7f0cd139b640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ccc1008d0 0x7f0ccc198200 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00ebc0 tx=0x7f0cbc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:08.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.291+0000 7f0cca7fc640 1 -- 172.21.15.33:0/874764708 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc010630 con 0x7f0ccc1008d0 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.291+0000 7f0cd239d640 1 -- 172.21.15.33:0/874764708 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ccc19d4a0 con 0x7f0ccc1008d0 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.291+0000 7f0cca7fc640 1 -- 172.21.15.33:0/874764708 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cbc010cb0 con 0x7f0ccc1008d0 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.291+0000 7f0cca7fc640 1 -- 172.21.15.33:0/874764708 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc018bd0 con 0x7f0ccc1008d0 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.291+0000 7f0cd239d640 1 -- 172.21.15.33:0/874764708 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ccc0fcad0 con 0x7f0ccc1008d0 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.293+0000 7f0cca7fc640 1 -- 172.21.15.33:0/874764708 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0cbc0107d0 con 0x7f0ccc1008d0 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd59d8740 1 Processor -- start 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd59d8740 1 -- start start 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd59d8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3da150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd59d8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3da690 0x55ef6a3e4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd59d8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd59d8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ef6a33f280 con 0x55ef6a3d9d70 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd59d8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ef6a2e6b10 con 0x55ef6a3e4f50 2024-09-16T00:22:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd59d8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ef6a33b240 con 0x55ef6a3da690 2024-09-16T00:22:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd1b9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3da150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd139b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3da690 0x55ef6a3e4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd1b9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3da150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43068/0 (socket says 172.21.15.33:43068) 2024-09-16T00:22:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd1b9c640 1 -- 172.21.15.33:0/4156764691 learned_addr learned my addr 172.21.15.33:0/4156764691 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.294+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/4156764691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cd0b9a640 1 -- 172.21.15.33:0/4156764691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3da690 msgr2=0x55ef6a3e4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/4156764691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3da690 0x55ef6a3e4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cd0b9a640 1 -- 172.21.15.33:0/4156764691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 msgr2=0x55ef6a3da150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/4156764691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3da150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cd0b9a640 1 -- 172.21.15.33:0/4156764691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef6a2f31f0 con 0x55ef6a3e4f50 2024-09-16T00:22:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cd1b9c640 1 --2- 172.21.15.33:0/4156764691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3da150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cca7fc640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0cb40a3240 0x7f0cb40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cd1b9c640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0cb40a3240 0x7f0cb40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/4156764691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e7340 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0010d00 tx=0x7f0cc0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=918cb05610e95bb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cca7fc640 1 -- 172.21.15.33:0/874764708 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0cbc0d0350 con 0x7f0ccc1008d0 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cb37fe640 1 -- 172.21.15.33:0/4156764691 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0004250 con 0x55ef6a3e4f50 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cb37fe640 1 -- 172.21.15.33:0/4156764691 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cc00043f0 con 0x55ef6a3e4f50 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.295+0000 7f0cb37fe640 1 -- 172.21.15.33:0/4156764691 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc001a9d0 con 0x55ef6a3e4f50 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.296+0000 7f0cd59d8740 1 -- 172.21.15.33:0/4156764691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 msgr2=0x55ef6a3e7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.296+0000 7f0cd1b9c640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0cb40a3240 0x7f0cb40a5700 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4031550 tx=0x7f0cc40046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.296+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/4156764691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e7340 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0010d00 tx=0x7f0cc0005f90 comp rx=0 tx=0).stop 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.296+0000 7f0cd59d8740 1 -- 172.21.15.33:0/4156764691 shutdown_connections 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.296+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/4156764691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e7340 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.296+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/4156764691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3da690 0x55ef6a3e4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.296+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/4156764691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3da150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.296+0000 7f0cd59d8740 1 -- 172.21.15.33:0/4156764691 >> 172.21.15.33:0/4156764691 conn(0x55ef6a3d6300 msgr2=0x55ef6a3d66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.296+0000 7f0cd59d8740 1 -- 172.21.15.33:0/4156764691 shutdown_connections 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.296+0000 7f0cd59d8740 1 -- 172.21.15.33:0/4156764691 wait complete. 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd59d8740 1 -- start start 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd59d8740 1 Processor -- start 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd59d8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3e1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd59d8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3da690 0x55ef6a3e22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd59d8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd59d8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ef6a3e7b20 con 0x55ef6a3e4f50 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd59d8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ef6a2f9f10 con 0x55ef6a3da690 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd59d8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ef6a3e7ca0 con 0x55ef6a3d9d70 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd59d8740 10 client.0 _ll_register_callbacks cb 0x55ef6a2c2a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:08.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd1b9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd139b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3e1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd139b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3e1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:54930/0 (socket says 172.21.15.33:54930) 2024-09-16T00:22:08.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd1b9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43070/0 (socket says 172.21.15.33:43070) 2024-09-16T00:22:08.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd139b640 1 -- 172.21.15.33:0/2931694791 learned_addr learned my addr 172.21.15.33:0/2931694791 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:08.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.297+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/2931694791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3da690 0x55ef6a3e22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cd139b640 1 -- 172.21.15.33:0/2931694791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3da690 msgr2=0x55ef6a3e22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:08.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cd139b640 1 --2- 172.21.15.33:0/2931694791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3da690 0x55ef6a3e22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cd139b640 1 -- 172.21.15.33:0/2931694791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3e4f50 msgr2=0x55ef6a3e27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:08.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cd139b640 1 --2- 172.21.15.33:0/2931694791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cd139b640 1 -- 172.21.15.33:0/2931694791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef6a0b06a0 con 0x55ef6a3d9d70 2024-09-16T00:22:08.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cd139b640 1 --2- 172.21.15.33:0/2931694791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3e1d70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0989a0 tx=0x7f0cbc009a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ad5d7d9cf45609b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:08.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cb0ff9640 1 -- 172.21.15.33:0/2931694791 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc010e60 con 0x55ef6a3d9d70 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cb0ff9640 1 -- 172.21.15.33:0/2931694791 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0cbc09b990 con 0x55ef6a3d9d70 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cd59d8740 1 -- 172.21.15.33:0/2931694791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cc00043f0 con 0x55ef6a3d9d70 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cb0ff9640 1 -- 172.21.15.33:0/2931694791 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0cbc09c080 con 0x55ef6a3d9d70 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cd59d8740 10 client.25771 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cd59d8740 1 -- 172.21.15.33:0/2931694791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ef6a3f36d0 con 0x55ef6a3d9d70 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.298+0000 7f0cb0ff9640 1 -- 172.21.15.33:0/2931694791 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cc00043f0 con 0x55ef6a3d9d70 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.299+0000 7f0cb0ff9640 1 -- 172.21.15.33:0/2931694791 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f0cbc09b400 con 0x55ef6a3d9d70 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.299+0000 7f0cd59d8740 10 client.25771 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.299+0000 7f0cd59d8740 10 client.25771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.299+0000 7f0cd59d8740 20 client.25771 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.299+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/2931694791 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ef6a3f4ae0 0x55ef6a3f6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.299+0000 7f0cd59d8740 4 client.25771 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:08.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.299+0000 7f0cd59d8740 1 -- 172.21.15.33:0/2931694791 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef6a34c7f0 con 0x55ef6a3f4ae0 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.299+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/2931694791 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ef6a3f4ae0 0x55ef6a3f6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.299+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/2931694791 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ef6a3f4ae0 0x55ef6a3f6f50 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d02e4e22df61e526 server_cookie=d3f2dc900b14db85 in_seq=0 out_seq=0 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.299+0000 7f0cb0ff9640 10 client.25771 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cb0ff9640 1 -- 172.21.15.33:0/2931694791 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ef6a34c7f0 con 0x55ef6a3f4ae0 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cb0ff9640 10 client.25771 handle_command_reply: tid=0 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cd59d8740 1 client.25771 shutdown 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cd59d8740 1 -- 172.21.15.33:0/2931694791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 msgr2=0x55ef6a3e1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/2931694791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3e1d70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0989a0 tx=0x7f0cbc009a40 comp rx=0 tx=0).stop 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cd59d8740 1 -- 172.21.15.33:0/2931694791 shutdown_connections 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/2931694791 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ef6a3f4ae0 0x55ef6a3f6f50 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/2931694791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/2931694791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3da690 0x55ef6a3e22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/2931694791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3e1d70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cb0ff9640 0 client.25771 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cd59d8740 1 -- 172.21.15.33:0/2931694791 >> 172.21.15.33:0/2931694791 conn(0x55ef6a3d6300 msgr2=0x55ef6a3dc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cd59d8740 1 -- 172.21.15.33:0/2931694791 shutdown_connections 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.300+0000 7f0cd59d8740 1 -- 172.21.15.33:0/2931694791 wait complete. 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.301+0000 7f0cd59d8740 20 client.25771 trim_cache size 0 max 16384 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.320+0000 7f0cd59d8740 1 Processor -- start 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.320+0000 7f0cd59d8740 1 -- start start 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.320+0000 7f0cd59d8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3dbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.320+0000 7f0cd59d8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3da690 0x55ef6a3fc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.320+0000 7f0cd59d8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3fcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.320+0000 7f0cd59d8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1af620 con 0x55ef6a3da690 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.320+0000 7f0cd59d8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1acef0 con 0x55ef6a3e4f50 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.320+0000 7f0cd59d8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1aad40 con 0x55ef6a3d9d70 2024-09-16T00:22:08.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.320+0000 7f0cd0b9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3da690 0x55ef6a3fc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cd0b9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3da690 0x55ef6a3fc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43084/0 (socket says 172.21.15.33:43084) 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cd0b9a640 1 -- 172.21.15.33:0/1807791135 learned_addr learned my addr 172.21.15.33:0/1807791135 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cd139b640 1 --2- 172.21.15.33:0/1807791135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3dbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cd1b9c640 1 --2- 172.21.15.33:0/1807791135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3fcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cd0b9a640 1 -- 172.21.15.33:0/1807791135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 msgr2=0x55ef6a3dbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/1807791135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3dbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cd0b9a640 1 -- 172.21.15.33:0/1807791135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 msgr2=0x55ef6a3fcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/1807791135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3fcd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cd0b9a640 1 -- 172.21.15.33:0/1807791135 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef6a3d9ba0 con 0x55ef6a3da690 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cd1b9c640 1 --2- 172.21.15.33:0/1807791135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3fcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cd139b640 1 --2- 172.21.15.33:0/1807791135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3dbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/1807791135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3da690 0x55ef6a3fc810 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x55ef6a3d7680 tx=0x7f0cc0004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7540665558fd15bb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:08.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cb2ffd640 1 -- 172.21.15.33:0/1807791135 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0005350 con 0x55ef6a3da690 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cb2ffd640 1 -- 172.21.15.33:0/1807791135 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cc00054f0 con 0x55ef6a3da690 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.321+0000 7f0cb2ffd640 1 -- 172.21.15.33:0/1807791135 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0003ab0 con 0x55ef6a3da690 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.322+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1807791135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3da690 msgr2=0x55ef6a3fc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.322+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/1807791135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3da690 0x55ef6a3fc810 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x55ef6a3d7680 tx=0x7f0cc0004260 comp rx=0 tx=0).stop 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.322+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1807791135 shutdown_connections 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.322+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/1807791135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3fcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.322+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/1807791135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3da690 0x55ef6a3fc810 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.322+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/1807791135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3dbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.322+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1807791135 >> 172.21.15.33:0/1807791135 conn(0x55ef6a3d6300 msgr2=0x55ef6a3db7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.322+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1807791135 shutdown_connections 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.322+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1807791135 wait complete. 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd59d8740 1 -- start start 2024-09-16T00:22:08.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd59d8740 1 Processor -- start 2024-09-16T00:22:08.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd59d8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3ecef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd59d8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3da690 0x55ef6a3ed430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd59d8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd59d8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1ab3b0 con 0x55ef6a3d9d70 2024-09-16T00:22:08.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd59d8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1ac820 con 0x55ef6a3e4f50 2024-09-16T00:22:08.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd59d8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1aba80 con 0x55ef6a3da690 2024-09-16T00:22:08.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd59d8740 10 client.0 _ll_register_callbacks cb 0x7f0cbc00eb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:08.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd139b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3ecef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd139b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3ecef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43088/0 (socket says 172.21.15.33:43088) 2024-09-16T00:22:08.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd139b640 1 -- 172.21.15.33:0/1002760161 learned_addr learned my addr 172.21.15.33:0/1002760161 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:08.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3da690 0x55ef6a3ed430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.323+0000 7f0cd1b9c640 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cd139b640 1 -- 172.21.15.33:0/1002760161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3da690 msgr2=0x55ef6a3ed430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:08.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cd139b640 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3da690 0x55ef6a3ed430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cd139b640 1 -- 172.21.15.33:0/1002760161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 msgr2=0x55ef6a3e9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:08.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cd139b640 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cd139b640 1 -- 172.21.15.33:0/1002760161 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef6a3d6130 con 0x55ef6a3d9d70 2024-09-16T00:22:08.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3da690 0x55ef6a3ed430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:08.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cd139b640 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3ecef0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc09b7f0 tx=0x7f0cbc000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1192a7a720567b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:08.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:08.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cb0ff9640 1 -- 172.21.15.33:0/1002760161 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc010e60 con 0x55ef6a3d9d70 2024-09-16T00:22:08.547 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:22:08.548 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:22:08.548 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "1af9c623-e845-42dc-8f50-4d5aaf1bdf4b", 2024-09-16T00:22:08.548 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:22:08.548 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:22:08.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cb0ff9640 1 -- 172.21.15.33:0/1002760161 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0cbc0984a0 con 0x55ef6a3d9d70 2024-09-16T00:22:08.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1002760161 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cc00054f0 con 0x55ef6a3d9d70 2024-09-16T00:22:08.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cb0ff9640 1 -- 172.21.15.33:0/1002760161 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0cbc09bea0 con 0x55ef6a3d9d70 2024-09-16T00:22:08.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cd59d8740 10 client.16302 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:08.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1002760161 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ef6a45ffd0 con 0x55ef6a3d9d70 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.324+0000 7f0cb0ff9640 1 -- 172.21.15.33:0/1002760161 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cc00054f0 con 0x55ef6a3d9d70 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.325+0000 7f0cb0ff9640 1 -- 172.21.15.33:0/1002760161 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f0cbc0b5080 con 0x55ef6a3d9d70 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.325+0000 7f0cd59d8740 10 client.16302 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.325+0000 7f0cd59d8740 10 client.16302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.325+0000 7f0cd59d8740 20 client.16302 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.325+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ef6a460540 0x55ef6a462960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.325+0000 7f0cd59d8740 4 client.16302 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.325+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1002760161 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ef6a346d00 con 0x55ef6a460540 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.325+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ef6a460540 0x55ef6a462960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.325+0000 7f0cb0ff9640 10 client.16302 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.325+0000 7f0cd0b9a640 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ef6a460540 0x55ef6a462960 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11334290b3e18290 server_cookie=83ba5f267292874b in_seq=0 out_seq=0 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.326+0000 7f0cb0ff9640 1 -- 172.21.15.33:0/1002760161 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ef6a346d00 con 0x55ef6a460540 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.326+0000 7f0cb0ff9640 10 client.16302 handle_command_reply: tid=0 2024-09-16T00:22:08.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.326+0000 7f0cd59d8740 1 client.16302 shutdown 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.326+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1002760161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 msgr2=0x55ef6a3ecef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.326+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3ecef0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc09b7f0 tx=0x7f0cbc000f40 comp rx=0 tx=0).stop 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.326+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1002760161 shutdown_connections 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.326+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ef6a460540 0x55ef6a462960 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.326+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ef6a3e4f50 0x55ef6a3e9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.326+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef6a3da690 0x55ef6a3ed430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.326+0000 7f0cd59d8740 1 --2- 172.21.15.33:0/1002760161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef6a3d9d70 0x55ef6a3ecef0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.326+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1002760161 >> 172.21.15.33:0/1002760161 conn(0x55ef6a3d6300 msgr2=0x55ef6a3dbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.326+0000 7f0cb0ff9640 0 client.16302 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1002760161 shutdown_connections 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd59d8740 1 -- 172.21.15.33:0/1002760161 wait complete. 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd59d8740 20 client.16302 trim_cache size 0 max 16384 2024-09-16T00:22:08.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd239d640 1 -- 172.21.15.33:0/874764708 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0cb40a3240 msgr2=0x7f0cb40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:08.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd239d640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0cb40a3240 0x7f0cb40a5700 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4031550 tx=0x7f0cc40046c0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd239d640 1 -- 172.21.15.33:0/874764708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ccc1008d0 msgr2=0x7f0ccc198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:08.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd239d640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ccc1008d0 0x7f0ccc198200 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00ebc0 tx=0x7f0cbc00c6a0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd239d640 1 -- 172.21.15.33:0/874764708 shutdown_connections 2024-09-16T00:22:08.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd239d640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0cb40a3240 0x7f0cb40a5700 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd239d640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ccc1068d0 0x7f0ccc19cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd239d640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ccc101270 0x7f0ccc198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd239d640 1 --2- 172.21.15.33:0/874764708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ccc1008d0 0x7f0ccc198200 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:08.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd239d640 1 -- 172.21.15.33:0/874764708 >> 172.21.15.33:0/874764708 conn(0x7f0ccc0fc610 msgr2=0x7f0ccc06a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:08.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd239d640 1 -- 172.21.15.33:0/874764708 shutdown_connections 2024-09-16T00:22:08.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:08.327+0000 7f0cd239d640 1 -- 172.21.15.33:0/874764708 wait complete. 2024-09-16T00:22:08.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 286264 11.86 MB/sec execute 76 sec latency 33.827 ms 2024-09-16T00:22:09.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:09 smithi033 ceph-mon[31243]: pgmap v500: 129 pgs: 129 active+clean; 116 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 160 op/s 2024-09-16T00:22:09.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:09 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:22:09.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:09 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:09.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:09 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:22:09.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:09 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:22:09.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:09 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:09.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:09 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:09.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:09 smithi083 ceph-mon[35242]: pgmap v500: 129 pgs: 129 active+clean; 116 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 160 op/s 2024-09-16T00:22:09.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:09 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:22:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:09 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:09 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:22:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:09 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:22:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:09 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:09.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:09 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:09.460 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:22:09.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:09 smithi060 ceph-mon[35185]: pgmap v500: 129 pgs: 129 active+clean; 116 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 160 op/s 2024-09-16T00:22:09.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:09 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:22:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:09 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:09 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:22:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:09 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:22:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:09 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:09.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:09 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:09.656 INFO:tasks.workunit.client.0.smithi033.stdout: 1 288516 11.88 MB/sec execute 77 sec latency 14.676 ms 2024-09-16T00:22:09.795 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:10.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.274+0000 7f1de0dc3640 1 -- 172.21.15.33:0/3431770815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ddc1082a0 msgr2=0x7f1ddc10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:10.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.274+0000 7f1de0dc3640 1 --2- 172.21.15.33:0/3431770815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ddc1082a0 0x7f1ddc10ec40 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc009930 tx=0x7f1dcc02a420 comp rx=0 tx=0).stop 2024-09-16T00:22:10.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.274+0000 7f1de0dc3640 1 -- 172.21.15.33:0/3431770815 shutdown_connections 2024-09-16T00:22:10.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.274+0000 7f1de0dc3640 1 --2- 172.21.15.33:0/3431770815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ddc1082a0 0x7f1ddc10ec40 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.274+0000 7f1de0dc3640 1 --2- 172.21.15.33:0/3431770815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ddc1036e0 0x7f1ddc107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.274+0000 7f1de0dc3640 1 --2- 172.21.15.33:0/3431770815 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ddc102d30 0x7f1ddc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.274+0000 7f1de0dc3640 1 -- 172.21.15.33:0/3431770815 >> 172.21.15.33:0/3431770815 conn(0x7f1ddc0fc7d0 msgr2=0x7f1ddc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:10.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.274+0000 7f1de0dc3640 1 -- 172.21.15.33:0/3431770815 shutdown_connections 2024-09-16T00:22:10.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.274+0000 7f1de0dc3640 1 -- 172.21.15.33:0/3431770815 wait complete. 2024-09-16T00:22:10.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1de0dc3640 1 Processor -- start 2024-09-16T00:22:10.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1de0dc3640 1 -- start start 2024-09-16T00:22:10.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1de0dc3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ddc102d30 0x7f1ddc198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1de0dc3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ddc1036e0 0x7f1ddc198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1de0dc3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ddc1082a0 0x7f1ddc19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1de0dc3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc1120a0 con 0x7f1ddc102d30 2024-09-16T00:22:10.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1de0dc3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc111f20 con 0x7f1ddc1036e0 2024-09-16T00:22:10.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1de0dc3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc112220 con 0x7f1ddc1082a0 2024-09-16T00:22:10.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1dda59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ddc102d30 0x7f1ddc198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1dda59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ddc102d30 0x7f1ddc198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56844/0 (socket says 172.21.15.33:56844) 2024-09-16T00:22:10.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1dda59c640 1 -- 172.21.15.33:0/216403652 learned_addr learned my addr 172.21.15.33:0/216403652 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:10.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ddc1036e0 0x7f1ddc198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1ddad9d640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ddc1082a0 0x7f1ddc19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1dd9d9b640 1 -- 172.21.15.33:0/216403652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ddc1082a0 msgr2=0x7f1ddc19ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:10.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ddc1082a0 0x7f1ddc19ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1dd9d9b640 1 -- 172.21.15.33:0/216403652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ddc102d30 msgr2=0x7f1ddc198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:10.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ddc102d30 0x7f1ddc198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1dd9d9b640 1 -- 172.21.15.33:0/216403652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ddc19d520 con 0x7f1ddc1036e0 2024-09-16T00:22:10.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.275+0000 7f1dda59c640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ddc102d30 0x7f1ddc198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:10.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.276+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ddc1036e0 0x7f1ddc198ab0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0697d0 tx=0x7f1dc400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:10.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.276+0000 7f1dd37fe640 1 -- 172.21.15.33:0/216403652 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc400cc40 con 0x7f1ddc1036e0 2024-09-16T00:22:10.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.276+0000 7f1dd37fe640 1 -- 172.21.15.33:0/216403652 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dc400cde0 con 0x7f1ddc1036e0 2024-09-16T00:22:10.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.276+0000 7f1de0dc3640 1 -- 172.21.15.33:0/216403652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ddc19d810 con 0x7f1ddc1036e0 2024-09-16T00:22:10.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.276+0000 7f1dd37fe640 1 -- 172.21.15.33:0/216403652 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc40188e0 con 0x7f1ddc1036e0 2024-09-16T00:22:10.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.276+0000 7f1de0dc3640 1 -- 172.21.15.33:0/216403652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ddc1a50f0 con 0x7f1ddc1036e0 2024-09-16T00:22:10.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.279+0000 7f1dd37fe640 1 -- 172.21.15.33:0/216403652 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1dc4018a80 con 0x7f1ddc1036e0 2024-09-16T00:22:10.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.279+0000 7f1de319c740 1 Processor -- start 2024-09-16T00:22:10.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1de319c740 1 -- start start 2024-09-16T00:22:10.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1dd37fe640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1dc80a32c0 0x7f1dc80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1de319c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc74a0 0x559aa7dc7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1de319c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc7dc0 0x559aa7dd2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1de319c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dd2680 0x559aa7dd4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1de319c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559aa7d2d2e0 con 0x559aa7dc7dc0 2024-09-16T00:22:10.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1de319c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559aa7cd4ce0 con 0x559aa7dd2680 2024-09-16T00:22:10.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1de319c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559aa7d292a0 con 0x559aa7dc74a0 2024-09-16T00:22:10.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1dda59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc74a0 0x559aa7dc7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1ddad9d640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1dc80a32c0 0x7f1dc80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1dda59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc74a0 0x559aa7dc7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:38672/0 (socket says 172.21.15.33:38672) 2024-09-16T00:22:10.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1dda59c640 1 -- 172.21.15.33:0/3449911583 learned_addr learned my addr 172.21.15.33:0/3449911583 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:10.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1dd37fe640 1 -- 172.21.15.33:0/216403652 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1dc40cff30 con 0x7f1ddc1036e0 2024-09-16T00:22:10.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/3449911583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc7dc0 0x559aa7dd2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1ddad9d640 1 --2- 172.21.15.33:0/3449911583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dd2680 0x559aa7dd4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1dda59c640 1 -- 172.21.15.33:0/3449911583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dd2680 msgr2=0x559aa7dd4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:10.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1dda59c640 1 --2- 172.21.15.33:0/3449911583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dd2680 0x559aa7dd4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1dda59c640 1 -- 172.21.15.33:0/3449911583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc7dc0 msgr2=0x559aa7dd2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:10.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1dda59c640 1 --2- 172.21.15.33:0/3449911583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc7dc0 0x559aa7dd2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1dda59c640 1 -- 172.21.15.33:0/3449911583 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559aa7cb9440 con 0x559aa7dc74a0 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.280+0000 7f1ddad9d640 1 --2- 172.21.15.33:0/3449911583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dd2680 0x559aa7dd4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.281+0000 7f1dda59c640 1 --2- 172.21.15.33:0/3449911583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc74a0 0x559aa7dc7880 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4011930 tx=0x7f1dd4011df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=953df5e3d8bf2489 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.281+0000 7f1ddad9d640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1dc80a32c0 0x7f1dc80a5780 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc006010 tx=0x7f1dcc031480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.281+0000 7f1dd0ff9640 1 -- 172.21.15.33:0/3449911583 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4011330 con 0x559aa7dc74a0 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.281+0000 7f1dd0ff9640 1 -- 172.21.15.33:0/3449911583 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dd4010430 con 0x559aa7dc74a0 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.281+0000 7f1dd0ff9640 1 -- 172.21.15.33:0/3449911583 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4019640 con 0x559aa7dc74a0 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.281+0000 7f1de319c740 1 -- 172.21.15.33:0/3449911583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc74a0 msgr2=0x559aa7dc7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.281+0000 7f1de319c740 1 --2- 172.21.15.33:0/3449911583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc74a0 0x559aa7dc7880 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4011930 tx=0x7f1dd4011df0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 -- 172.21.15.33:0/3449911583 shutdown_connections 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 --2- 172.21.15.33:0/3449911583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dd2680 0x559aa7dd4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 --2- 172.21.15.33:0/3449911583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc7dc0 0x559aa7dd2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 --2- 172.21.15.33:0/3449911583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc74a0 0x559aa7dc7880 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 -- 172.21.15.33:0/3449911583 >> 172.21.15.33:0/3449911583 conn(0x559aa7dc3a30 msgr2=0x559aa7dc3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:10.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 -- 172.21.15.33:0/3449911583 shutdown_connections 2024-09-16T00:22:10.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 -- 172.21.15.33:0/3449911583 wait complete. 2024-09-16T00:22:10.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 -- start start 2024-09-16T00:22:10.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 Processor -- start 2024-09-16T00:22:10.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc74a0 0x559aa7dd1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc7dc0 0x559aa7dcf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dd2680 0x559aa7dcf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559aa7dd5250 con 0x559aa7dc74a0 2024-09-16T00:22:10.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559aa7ce8050 con 0x559aa7dd2680 2024-09-16T00:22:10.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559aa7dd53d0 con 0x559aa7dc7dc0 2024-09-16T00:22:10.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1dda59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc74a0 0x559aa7dd1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.282+0000 7f1de319c740 10 client.0 _ll_register_callbacks cb 0x559aa7d28160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:10.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dda59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc74a0 0x559aa7dd1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56860/0 (socket says 172.21.15.33:56860) 2024-09-16T00:22:10.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dda59c640 1 -- 172.21.15.33:0/4140036794 learned_addr learned my addr 172.21.15.33:0/4140036794 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:10.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1ddad9d640 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dd2680 0x559aa7dcf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc7dc0 0x559aa7dcf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dda59c640 1 -- 172.21.15.33:0/4140036794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc7dc0 msgr2=0x559aa7dcf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:10.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dda59c640 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc7dc0 0x559aa7dcf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dda59c640 1 -- 172.21.15.33:0/4140036794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dd2680 msgr2=0x559aa7dcf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:10.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dda59c640 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dd2680 0x559aa7dcf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dda59c640 1 -- 172.21.15.33:0/4140036794 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559aa7a9e3c0 con 0x559aa7dc74a0 2024-09-16T00:22:10.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc7dc0 0x559aa7dcf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:10.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dda59c640 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc74a0 0x559aa7dd1e70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4010e10 tx=0x7f1dd4004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e9cb9e50ca5f20 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:10.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:10.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dc1ffb640 1 -- 172.21.15.33:0/4140036794 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4028020 con 0x559aa7dc74a0 2024-09-16T00:22:10.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1de319c740 1 -- 172.21.15.33:0/4140036794 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dd4010430 con 0x559aa7dc74a0 2024-09-16T00:22:10.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dc1ffb640 1 -- 172.21.15.33:0/4140036794 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1dd400fc80 con 0x559aa7dc74a0 2024-09-16T00:22:10.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1dc1ffb640 1 -- 172.21.15.33:0/4140036794 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1dd4033050 con 0x559aa7dc74a0 2024-09-16T00:22:10.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1de319c740 10 client.16314 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:10.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.283+0000 7f1de319c740 1 -- 172.21.15.33:0/4140036794 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559aa7de0b20 con 0x559aa7dc74a0 2024-09-16T00:22:10.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.284+0000 7f1dc1ffb640 1 -- 172.21.15.33:0/4140036794 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dd4010430 con 0x559aa7dc74a0 2024-09-16T00:22:10.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.284+0000 7f1dc1ffb640 1 -- 172.21.15.33:0/4140036794 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f1dd4018070 con 0x559aa7dc74a0 2024-09-16T00:22:10.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.284+0000 7f1de319c740 10 client.16314 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:10.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.284+0000 7f1de319c740 10 client.16314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:10.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.284+0000 7f1de319c740 20 client.16314 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:10.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.284+0000 7f1de319c740 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559aa7de1f30 0x559aa7de4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.284+0000 7f1de319c740 4 client.16314 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:10.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.284+0000 7f1de319c740 1 -- 172.21.15.33:0/4140036794 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559aa7d3a850 con 0x559aa7de1f30 2024-09-16T00:22:10.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.284+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559aa7de1f30 0x559aa7de4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559aa7de1f30 0x559aa7de4320 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7979a1495ab30098 server_cookie=83e386110d61c6be in_seq=0 out_seq=0 2024-09-16T00:22:10.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1dc1ffb640 10 client.16314 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:10.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1dc1ffb640 1 -- 172.21.15.33:0/4140036794 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559aa7d3a850 con 0x559aa7de1f30 2024-09-16T00:22:10.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1dc1ffb640 10 client.16314 handle_command_reply: tid=0 2024-09-16T00:22:10.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1de319c740 1 client.16314 shutdown 2024-09-16T00:22:10.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1de319c740 1 -- 172.21.15.33:0/4140036794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc74a0 msgr2=0x559aa7dd1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:10.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1de319c740 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc74a0 0x559aa7dd1e70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4010e10 tx=0x7f1dd4004cf0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1de319c740 1 -- 172.21.15.33:0/4140036794 shutdown_connections 2024-09-16T00:22:10.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1de319c740 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559aa7de1f30 0x559aa7de4320 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1de319c740 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dd2680 0x559aa7dcf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1de319c740 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dc7dc0 0x559aa7dcf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1dc1ffb640 0 client.16314 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:10.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1de319c740 1 --2- 172.21.15.33:0/4140036794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc74a0 0x559aa7dd1e70 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.285+0000 7f1de319c740 1 -- 172.21.15.33:0/4140036794 >> 172.21.15.33:0/4140036794 conn(0x559aa7dc3a30 msgr2=0x559aa7dc9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:10.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.286+0000 7f1de319c740 1 -- 172.21.15.33:0/4140036794 shutdown_connections 2024-09-16T00:22:10.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.286+0000 7f1de319c740 1 -- 172.21.15.33:0/4140036794 wait complete. 2024-09-16T00:22:10.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.286+0000 7f1de319c740 20 client.16314 trim_cache size 0 max 16384 2024-09-16T00:22:10.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.303+0000 7f1de319c740 1 Processor -- start 2024-09-16T00:22:10.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.303+0000 7f1de319c740 1 -- start start 2024-09-16T00:22:10.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.303+0000 7f1de319c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc74a0 0x559aa7dca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.303+0000 7f1de319c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc7dc0 0x559aa7dc9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.303+0000 7f1de319c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dd2680 0x559aa7de9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.303+0000 7f1de319c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc1b1a70 con 0x559aa7dc74a0 2024-09-16T00:22:10.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.303+0000 7f1de319c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc1af340 con 0x559aa7dc7dc0 2024-09-16T00:22:10.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.303+0000 7f1de319c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc1ad190 con 0x559aa7dd2680 2024-09-16T00:22:10.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dd9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc7dc0 0x559aa7dc9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dda59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc74a0 0x559aa7dca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dd9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc7dc0 0x559aa7dc9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:56478/0 (socket says 172.21.15.33:56478) 2024-09-16T00:22:10.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dd9d9b640 1 -- 172.21.15.33:0/1255891018 learned_addr learned my addr 172.21.15.33:0/1255891018 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:10.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1ddad9d640 1 --2- 172.21.15.33:0/1255891018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dd2680 0x559aa7de9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dd9d9b640 1 -- 172.21.15.33:0/1255891018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dd2680 msgr2=0x559aa7de9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:10.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/1255891018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dd2680 0x559aa7de9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dd9d9b640 1 -- 172.21.15.33:0/1255891018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc74a0 msgr2=0x559aa7dca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:10.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/1255891018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc74a0 0x559aa7dca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dd9d9b640 1 -- 172.21.15.33:0/1255891018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559aa7dc3860 con 0x559aa7dc7dc0 2024-09-16T00:22:10.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/1255891018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc7dc0 0x559aa7dc9600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40cd600 tx=0x7f1dc40cd480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecc1d7f6218f1419 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:10.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dc3fff640 1 -- 172.21.15.33:0/1255891018 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc409a220 con 0x559aa7dc7dc0 2024-09-16T00:22:10.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dc3fff640 1 -- 172.21.15.33:0/1255891018 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dc409a480 con 0x559aa7dc7dc0 2024-09-16T00:22:10.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.304+0000 7f1dc3fff640 1 -- 172.21.15.33:0/1255891018 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc406c1e0 con 0x559aa7dc7dc0 2024-09-16T00:22:10.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.305+0000 7f1de319c740 1 -- 172.21.15.33:0/1255891018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc7dc0 msgr2=0x559aa7dc9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:10.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.305+0000 7f1de319c740 1 --2- 172.21.15.33:0/1255891018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc7dc0 0x559aa7dc9600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40cd600 tx=0x7f1dc40cd480 comp rx=0 tx=0).stop 2024-09-16T00:22:10.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.305+0000 7f1de319c740 1 -- 172.21.15.33:0/1255891018 shutdown_connections 2024-09-16T00:22:10.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.305+0000 7f1de319c740 1 --2- 172.21.15.33:0/1255891018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dd2680 0x559aa7de9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.305+0000 7f1de319c740 1 --2- 172.21.15.33:0/1255891018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc7dc0 0x559aa7dc9600 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.305+0000 7f1de319c740 1 --2- 172.21.15.33:0/1255891018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc74a0 0x559aa7dca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.305+0000 7f1de319c740 1 -- 172.21.15.33:0/1255891018 >> 172.21.15.33:0/1255891018 conn(0x559aa7dc3a30 msgr2=0x559aa7dc8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:10.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.305+0000 7f1de319c740 1 -- 172.21.15.33:0/1255891018 shutdown_connections 2024-09-16T00:22:10.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.305+0000 7f1de319c740 1 -- 172.21.15.33:0/1255891018 wait complete. 2024-09-16T00:22:10.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.305+0000 7f1de319c740 1 -- start start 2024-09-16T00:22:10.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1de319c740 1 Processor -- start 2024-09-16T00:22:10.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1de319c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc74a0 0x559aa7dc9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1de319c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc7dc0 0x559aa7dc9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1de319c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dd2680 0x559aa7e53320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1de319c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc1ad800 con 0x559aa7dc7dc0 2024-09-16T00:22:10.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1de319c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc1aec70 con 0x559aa7dc74a0 2024-09-16T00:22:10.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1de319c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc1aded0 con 0x559aa7dd2680 2024-09-16T00:22:10.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1de319c740 10 client.0 _ll_register_callbacks cb 0x7f1dd40114d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:10.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1dd9d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc7dc0 0x559aa7dc9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1dda59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc74a0 0x559aa7dc9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1dd9d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc7dc0 0x559aa7dc9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56880/0 (socket says 172.21.15.33:56880) 2024-09-16T00:22:10.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1dd9d9b640 1 -- 172.21.15.33:0/66405164 learned_addr learned my addr 172.21.15.33:0/66405164 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:10.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1ddad9d640 1 --2- 172.21.15.33:0/66405164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dd2680 0x559aa7e53320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1dda59c640 1 -- 172.21.15.33:0/66405164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dd2680 msgr2=0x559aa7e53320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:10.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1dda59c640 1 --2- 172.21.15.33:0/66405164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dd2680 0x559aa7e53320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1dda59c640 1 -- 172.21.15.33:0/66405164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc7dc0 msgr2=0x559aa7dc9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:10.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1dda59c640 1 --2- 172.21.15.33:0/66405164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc7dc0 0x559aa7dc9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.306+0000 7f1dda59c640 1 -- 172.21.15.33:0/66405164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559aa7e53860 con 0x559aa7dc74a0 2024-09-16T00:22:10.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.307+0000 7f1dda59c640 1 --2- 172.21.15.33:0/66405164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc74a0 0x559aa7dc9720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1dd401aaf0 tx=0x7f1dd40049d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a45db472766f7663 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:10.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.307+0000 7f1dc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:22:10.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.307+0000 7f1dc1ffb640 1 -- 172.21.15.33:0/66405164 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4028040 con 0x559aa7dc74a0 2024-09-16T00:22:10.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.307+0000 7f1de319c740 1 -- 172.21.15.33:0/66405164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dc409a480 con 0x559aa7dc74a0 2024-09-16T00:22:10.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.307+0000 7f1dc1ffb640 1 -- 172.21.15.33:0/66405164 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1dd401a650 con 0x559aa7dc74a0 2024-09-16T00:22:10.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.307+0000 7f1dc1ffb640 1 -- 172.21.15.33:0/66405164 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1dd401f040 con 0x559aa7dc74a0 2024-09-16T00:22:10.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.307+0000 7f1de319c740 10 client.25859 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.307+0000 7f1de319c740 1 -- 172.21.15.33:0/66405164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559aa7e549c0 con 0x559aa7dc74a0 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.308+0000 7f1dc1ffb640 1 -- 172.21.15.33:0/66405164 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dc409a480 con 0x559aa7dc74a0 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.308+0000 7f1dc1ffb640 1 -- 172.21.15.33:0/66405164 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f1dd4018040 con 0x559aa7dc74a0 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.308+0000 7f1de319c740 10 client.25859 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.308+0000 7f1de319c740 10 client.25859 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.308+0000 7f1de319c740 20 client.25859 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.308+0000 7f1de319c740 1 --2- 172.21.15.33:0/66405164 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559aa7de1f30 0x559aa7e57030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.308+0000 7f1de319c740 4 client.25859 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.308+0000 7f1de319c740 1 -- 172.21.15.33:0/66405164 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559aa7d34d60 con 0x559aa7de1f30 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.308+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/66405164 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559aa7de1f30 0x559aa7e57030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.308+0000 7f1dc1ffb640 10 client.25859 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.308+0000 7f1dd9d9b640 1 --2- 172.21.15.33:0/66405164 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559aa7de1f30 0x559aa7e57030 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf18fc77266d4451 server_cookie=9641d34f1e2ae781 in_seq=0 out_seq=0 2024-09-16T00:22:10.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1dc1ffb640 1 -- 172.21.15.33:0/66405164 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559aa7d34d60 con 0x559aa7de1f30 2024-09-16T00:22:10.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1dc1ffb640 10 client.25859 handle_command_reply: tid=0 2024-09-16T00:22:10.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de319c740 1 client.25859 shutdown 2024-09-16T00:22:10.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de319c740 1 -- 172.21.15.33:0/66405164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc74a0 msgr2=0x559aa7dc9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:10.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de319c740 1 --2- 172.21.15.33:0/66405164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc74a0 0x559aa7dc9720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1dd401aaf0 tx=0x7f1dd40049d0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de319c740 1 -- 172.21.15.33:0/66405164 shutdown_connections 2024-09-16T00:22:10.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de319c740 1 --2- 172.21.15.33:0/66405164 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559aa7de1f30 0x559aa7e57030 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de319c740 1 --2- 172.21.15.33:0/66405164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559aa7dd2680 0x559aa7e53320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de319c740 1 --2- 172.21.15.33:0/66405164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559aa7dc7dc0 0x559aa7dc9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de319c740 1 --2- 172.21.15.33:0/66405164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559aa7dc74a0 0x559aa7dc9720 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1dc1ffb640 0 client.25859 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:10.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de319c740 1 -- 172.21.15.33:0/66405164 >> 172.21.15.33:0/66405164 conn(0x559aa7dc3a30 msgr2=0x559aa7dd0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:10.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de319c740 1 -- 172.21.15.33:0/66405164 shutdown_connections 2024-09-16T00:22:10.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de319c740 1 -- 172.21.15.33:0/66405164 wait complete. 2024-09-16T00:22:10.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de319c740 20 client.25859 trim_cache size 0 max 16384 2024-09-16T00:22:10.324 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:22:10.324 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:22:10.324 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:22:10.324 INFO:teuthology.orchestra.run.smithi033.stdout: "1af9c623-e845-42dc-8f50-4d5aaf1bdf4b": { 2024-09-16T00:22:10.324 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:22:10.324 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "1af9c623-e845-42dc-8f50-4d5aaf1bdf4b", 2024-09-16T00:22:10.324 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:22:10.324 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:10.324 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:10.325 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:22:10.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de0dc3640 1 -- 172.21.15.33:0/216403652 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1dc80a32c0 msgr2=0x7f1dc80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:10.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de0dc3640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1dc80a32c0 0x7f1dc80a5780 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc006010 tx=0x7f1dcc031480 comp rx=0 tx=0).stop 2024-09-16T00:22:10.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de0dc3640 1 -- 172.21.15.33:0/216403652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ddc1036e0 msgr2=0x7f1ddc198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:10.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.309+0000 7f1de0dc3640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ddc1036e0 0x7f1ddc198ab0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0697d0 tx=0x7f1dc400ee50 comp rx=0 tx=0).stop 2024-09-16T00:22:10.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.310+0000 7f1de0dc3640 1 -- 172.21.15.33:0/216403652 shutdown_connections 2024-09-16T00:22:10.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.310+0000 7f1de0dc3640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1dc80a32c0 0x7f1dc80a5780 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.310+0000 7f1de0dc3640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ddc1082a0 0x7f1ddc19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.310+0000 7f1de0dc3640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ddc1036e0 0x7f1ddc198ab0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.310+0000 7f1de0dc3640 1 --2- 172.21.15.33:0/216403652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ddc102d30 0x7f1ddc198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:10.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.310+0000 7f1de0dc3640 1 -- 172.21.15.33:0/216403652 >> 172.21.15.33:0/216403652 conn(0x7f1ddc0fc7d0 msgr2=0x7f1ddc105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:10.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.310+0000 7f1de0dc3640 1 -- 172.21.15.33:0/216403652 shutdown_connections 2024-09-16T00:22:10.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:10.310+0000 7f1de0dc3640 1 -- 172.21.15.33:0/216403652 wait complete. 2024-09-16T00:22:10.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 291542 11.95 MB/sec execute 78 sec latency 23.255 ms 2024-09-16T00:22:11.216 INFO:tasks.cephfs.filesystem:scrub status for tag:1af9c623-e845-42dc-8f50-4d5aaf1bdf4b - {'path': '/', 'tag': '1af9c623-e845-42dc-8f50-4d5aaf1bdf4b', 'options': 'recursive,force'} 2024-09-16T00:22:11.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:11 smithi033 ceph-mon[31243]: pgmap v501: 129 pgs: 129 active+clean; 118 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 221 op/s 2024-09-16T00:22:11.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:11 smithi083 ceph-mon[35242]: pgmap v501: 129 pgs: 129 active+clean; 118 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 221 op/s 2024-09-16T00:22:11.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:11 smithi060 ceph-mon[35185]: pgmap v501: 129 pgs: 129 active+clean; 118 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 221 op/s 2024-09-16T00:22:11.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 294722 12.03 MB/sec execute 79 sec latency 23.101 ms 2024-09-16T00:22:12.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 297346 12.04 MB/sec execute 80 sec latency 63.782 ms 2024-09-16T00:22:13.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:13 smithi083 ceph-mon[35242]: pgmap v502: 129 pgs: 129 active+clean; 122 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 290 op/s 2024-09-16T00:22:13.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:13 smithi060 ceph-mon[35185]: pgmap v502: 129 pgs: 129 active+clean; 122 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 290 op/s 2024-09-16T00:22:13.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:13 smithi033 ceph-mon[31243]: pgmap v502: 129 pgs: 129 active+clean; 122 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 290 op/s 2024-09-16T00:22:13.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 299046 12.00 MB/sec execute 81 sec latency 63.991 ms 2024-09-16T00:22:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:22:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:22:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:22:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:22:14.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:22:14.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:22:14.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:22:14.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:22:14.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:22:14.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:22:14.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:22:14.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:22:14.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 302099 12.06 MB/sec execute 82 sec latency 7.935 ms 2024-09-16T00:22:15.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 305225 12.13 MB/sec execute 83 sec latency 15.725 ms 2024-09-16T00:22:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:15 smithi083 ceph-mon[35242]: pgmap v503: 129 pgs: 129 active+clean; 125 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 334 op/s 2024-09-16T00:22:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:15 smithi060 ceph-mon[35185]: pgmap v503: 129 pgs: 129 active+clean; 125 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 334 op/s 2024-09-16T00:22:16.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:15 smithi033 ceph-mon[31243]: pgmap v503: 129 pgs: 129 active+clean; 125 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 334 op/s 2024-09-16T00:22:16.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 307980 12.15 MB/sec execute 84 sec latency 12.290 ms 2024-09-16T00:22:16.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:16 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:22:16.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:16 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:22:17.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:16 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:22:17.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 310023 12.15 MB/sec execute 85 sec latency 34.321 ms 2024-09-16T00:22:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:17 smithi083 ceph-mon[35242]: pgmap v504: 129 pgs: 129 active+clean; 125 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.9 MiB/s wr, 283 op/s 2024-09-16T00:22:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:17 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:22:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:17 smithi060 ceph-mon[35185]: pgmap v504: 129 pgs: 129 active+clean; 125 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.9 MiB/s wr, 283 op/s 2024-09-16T00:22:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:17 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:22:18.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:17 smithi033 ceph-mon[31243]: pgmap v504: 129 pgs: 129 active+clean; 125 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.9 MiB/s wr, 283 op/s 2024-09-16T00:22:18.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:17 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:22:18.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 312832 12.17 MB/sec execute 86 sec latency 28.931 ms 2024-09-16T00:22:18.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:18 smithi083 ceph-mon[35242]: pgmap v505: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 259 op/s 2024-09-16T00:22:18.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:18 smithi060 ceph-mon[35185]: pgmap v505: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 259 op/s 2024-09-16T00:22:19.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:18 smithi033 ceph-mon[31243]: pgmap v505: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 259 op/s 2024-09-16T00:22:19.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 315945 12.25 MB/sec execute 87 sec latency 16.438 ms 2024-09-16T00:22:20.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 318701 12.25 MB/sec execute 88 sec latency 19.525 ms 2024-09-16T00:22:21.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:20 smithi033 ceph-mon[31243]: pgmap v506: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 326 op/s 2024-09-16T00:22:21.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:20 smithi083 ceph-mon[35242]: pgmap v506: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 326 op/s 2024-09-16T00:22:21.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:20 smithi060 ceph-mon[35185]: pgmap v506: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 326 op/s 2024-09-16T00:22:21.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 321089 12.25 MB/sec execute 89 sec latency 15.712 ms 2024-09-16T00:22:22.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 323787 12.28 MB/sec execute 90 sec latency 13.487 ms 2024-09-16T00:22:23.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:22 smithi033 ceph-mon[31243]: pgmap v507: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 277 op/s 2024-09-16T00:22:23.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:22 smithi083 ceph-mon[35242]: pgmap v507: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 277 op/s 2024-09-16T00:22:23.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:22 smithi060 ceph-mon[35185]: pgmap v507: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 277 op/s 2024-09-16T00:22:23.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 327004 12.34 MB/sec execute 91 sec latency 16.666 ms 2024-09-16T00:22:24.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 330150 12.41 MB/sec execute 92 sec latency 16.448 ms 2024-09-16T00:22:24.786 INFO:tasks.quiescer.fs.[cephfs]:Total duration (50.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T00:22:24.786 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.35 2024-09-16T00:22:24.786 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:22:25.060 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:25.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:24 smithi033 ceph-mon[31243]: pgmap v508: 129 pgs: 129 active+clean; 133 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 235 op/s 2024-09-16T00:22:25.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:24 smithi083 ceph-mon[35242]: pgmap v508: 129 pgs: 129 active+clean; 133 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 235 op/s 2024-09-16T00:22:25.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:24 smithi060 ceph-mon[35185]: pgmap v508: 129 pgs: 129 active+clean; 133 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 235 op/s 2024-09-16T00:22:25.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.544+0000 7f0fedace640 1 -- 172.21.15.33:0/3533241455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe8106f20 msgr2=0x7f0fe810eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:25.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.544+0000 7f0fedace640 1 --2- 172.21.15.33:0/3533241455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe8106f20 0x7f0fe810eaa0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0009ca0 tx=0x7f0fe002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:22:25.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.544+0000 7f0fedace640 1 -- 172.21.15.33:0/3533241455 shutdown_connections 2024-09-16T00:22:25.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.544+0000 7f0fedace640 1 --2- 172.21.15.33:0/3533241455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe8106f20 0x7f0fe810eaa0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:25.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.544+0000 7f0fedace640 1 --2- 172.21.15.33:0/3533241455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fe81027a0 0x7f0fe81067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:25.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.544+0000 7f0fedace640 1 --2- 172.21.15.33:0/3533241455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fe8101df0 0x7f0fe81021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:25.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.544+0000 7f0fedace640 1 -- 172.21.15.33:0/3533241455 >> 172.21.15.33:0/3533241455 conn(0x7f0fe80fd8f0 msgr2=0x7f0fe80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:25.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.544+0000 7f0fedace640 1 -- 172.21.15.33:0/3533241455 shutdown_connections 2024-09-16T00:22:25.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.544+0000 7f0fedace640 1 -- 172.21.15.33:0/3533241455 wait complete. 2024-09-16T00:22:25.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fedace640 1 Processor -- start 2024-09-16T00:22:25.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fedace640 1 -- start start 2024-09-16T00:22:25.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fedace640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fe8101df0 0x7f0fe81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:25.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fedace640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe81027a0 0x7f0fe8198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:25.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fedace640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fe8106f20 0x7f0fe819cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:25.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fedace640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe8111f80 con 0x7f0fe81027a0 2024-09-16T00:22:25.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fedace640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe8111e00 con 0x7f0fe8106f20 2024-09-16T00:22:25.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fedace640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe8112100 con 0x7f0fe8101df0 2024-09-16T00:22:25.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fe77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fe8101df0 0x7f0fe81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fdffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe81027a0 0x7f0fe8198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fe77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fe8101df0 0x7f0fe81983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40932/0 (socket says 172.21.15.33:40932) 2024-09-16T00:22:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fe77fe640 1 -- 172.21.15.33:0/2047154110 learned_addr learned my addr 172.21.15.33:0/2047154110 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fe7fff640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fe8106f20 0x7f0fe819cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fe77fe640 1 -- 172.21.15.33:0/2047154110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fe8106f20 msgr2=0x7f0fe819cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fe77fe640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fe8106f20 0x7f0fe819cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fe77fe640 1 -- 172.21.15.33:0/2047154110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe81027a0 msgr2=0x7f0fe8198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fe77fe640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe81027a0 0x7f0fe8198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.545+0000 7f0fe77fe640 1 -- 172.21.15.33:0/2047154110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe819d370 con 0x7f0fe8101df0 2024-09-16T00:22:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.546+0000 7f0fe77fe640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fe8101df0 0x7f0fe81983c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800ea60 tx=0x7f0fd800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.546+0000 7f0fe57fa640 1 -- 172.21.15.33:0/2047154110 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd800cde0 con 0x7f0fe8101df0 2024-09-16T00:22:25.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.546+0000 7f0fe57fa640 1 -- 172.21.15.33:0/2047154110 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fd80035f0 con 0x7f0fe8101df0 2024-09-16T00:22:25.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.546+0000 7f0fedace640 1 -- 172.21.15.33:0/2047154110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe819d660 con 0x7f0fe8101df0 2024-09-16T00:22:25.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.546+0000 7f0fe57fa640 1 -- 172.21.15.33:0/2047154110 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8010670 con 0x7f0fe8101df0 2024-09-16T00:22:25.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.546+0000 7f0fedace640 1 -- 172.21.15.33:0/2047154110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe81a4f40 con 0x7f0fe8101df0 2024-09-16T00:22:25.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.547+0000 7f0fedace640 1 -- 172.21.15.33:0/2047154110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe81021d0 con 0x7f0fe8101df0 2024-09-16T00:22:25.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.548+0000 7f0fe57fa640 1 -- 172.21.15.33:0/2047154110 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0fd80037a0 con 0x7f0fe8101df0 2024-09-16T00:22:25.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.549+0000 7f0fe57fa640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0fd40a3240 0x7f0fd40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:25.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.549+0000 7f0fdffff640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0fd40a3240 0x7f0fd40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:25.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.549+0000 7f0fe57fa640 1 -- 172.21.15.33:0/2047154110 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0fd80d00a0 con 0x7f0fe8101df0 2024-09-16T00:22:25.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.549+0000 7f0fdffff640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0fd40a3240 0x7f0fd40a5700 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00095f0 tx=0x7f0fd0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:25.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.555+0000 7f0fe57fa640 1 -- 172.21.15.33:0/2047154110 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0fd809c8e0 con 0x7f0fe8101df0 2024-09-16T00:22:25.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 333058 12.42 MB/sec execute 93 sec latency 17.101 ms 2024-09-16T00:22:25.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.785+0000 7f0fedace640 1 -- 172.21.15.33:0/2047154110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0fe810a1c0 con 0x7f0fe8101df0 2024-09-16T00:22:25.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.786+0000 7f0fe57fa640 1 -- 172.21.15.33:0/2047154110 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f0fd80bae20 con 0x7f0fe8101df0 2024-09-16T00:22:25.787 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:22:25.787 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:22:25.787 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:22:25.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.789+0000 7f0fedace640 1 -- 172.21.15.33:0/2047154110 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0fd40a3240 msgr2=0x7f0fd40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:25.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.789+0000 7f0fedace640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0fd40a3240 0x7f0fd40a5700 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00095f0 tx=0x7f0fd0005f90 comp rx=0 tx=0).stop 2024-09-16T00:22:25.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.789+0000 7f0fedace640 1 -- 172.21.15.33:0/2047154110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fe8101df0 msgr2=0x7f0fe81983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:25.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.789+0000 7f0fedace640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fe8101df0 0x7f0fe81983c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800ea60 tx=0x7f0fd800ef20 comp rx=0 tx=0).stop 2024-09-16T00:22:25.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.789+0000 7f0fedace640 1 -- 172.21.15.33:0/2047154110 shutdown_connections 2024-09-16T00:22:25.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.789+0000 7f0fedace640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0fd40a3240 0x7f0fd40a5700 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:25.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.789+0000 7f0fedace640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fe8106f20 0x7f0fe819cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:25.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.789+0000 7f0fedace640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe81027a0 0x7f0fe8198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:25.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.789+0000 7f0fedace640 1 --2- 172.21.15.33:0/2047154110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fe8101df0 0x7f0fe81983c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:25.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.789+0000 7f0fedace640 1 -- 172.21.15.33:0/2047154110 >> 172.21.15.33:0/2047154110 conn(0x7f0fe80fd8f0 msgr2=0x7f0fe80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:25.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.789+0000 7f0fedace640 1 -- 172.21.15.33:0/2047154110 shutdown_connections 2024-09-16T00:22:25.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:25.789+0000 7f0fedace640 1 -- 172.21.15.33:0/2047154110 wait complete. 2024-09-16T00:22:26.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:25 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2047154110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:26.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:25 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2047154110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:26.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:25 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2047154110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:26.532 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0' 2024-09-16T00:22:26.533 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0 2024-09-16T00:22:26.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 335823 12.43 MB/sec execute 94 sec latency 12.576 ms 2024-09-16T00:22:26.828 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:27.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:26 smithi033 ceph-mon[31243]: pgmap v509: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 242 op/s 2024-09-16T00:22:27.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:26 smithi083 ceph-mon[35242]: pgmap v509: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 242 op/s 2024-09-16T00:22:27.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:26 smithi060 ceph-mon[35185]: pgmap v509: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 242 op/s 2024-09-16T00:22:27.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.376+0000 7f4784623640 1 -- 172.21.15.33:0/2770932818 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f477c10c170 msgr2=0x7f477c10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:27.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.376+0000 7f4784623640 1 --2- 172.21.15.33:0/2770932818 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f477c10c170 0x7f477c10e610 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4774009ca0 tx=0x7f477402a950 comp rx=0 tx=0).stop 2024-09-16T00:22:27.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.376+0000 7f4784623640 1 -- 172.21.15.33:0/2770932818 shutdown_connections 2024-09-16T00:22:27.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.376+0000 7f4784623640 1 --2- 172.21.15.33:0/2770932818 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f477c10c170 0x7f477c10e610 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.376+0000 7f4784623640 1 --2- 172.21.15.33:0/2770932818 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f477c103660 0x7f477c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.376+0000 7f4784623640 1 --2- 172.21.15.33:0/2770932818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f477c102d40 0x7f477c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.376+0000 7f4784623640 1 -- 172.21.15.33:0/2770932818 >> 172.21.15.33:0/2770932818 conn(0x7f477c0fc7a0 msgr2=0x7f477c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:27.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.376+0000 7f4784623640 1 -- 172.21.15.33:0/2770932818 shutdown_connections 2024-09-16T00:22:27.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.376+0000 7f4784623640 1 -- 172.21.15.33:0/2770932818 wait complete. 2024-09-16T00:22:27.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f4784623640 1 Processor -- start 2024-09-16T00:22:27.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f4784623640 1 -- start start 2024-09-16T00:22:27.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f4784623640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f477c102d40 0x7f477c1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f4784623640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f477c103660 0x7f477c100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f4784623640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f477c10c170 0x7f477c101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f4784623640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f477c111f70 con 0x7f477c102d40 2024-09-16T00:22:27.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f4784623640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f477c111df0 con 0x7f477c103660 2024-09-16T00:22:27.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f4784623640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f477c1120f0 con 0x7f477c10c170 2024-09-16T00:22:27.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f47823bf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f477c102d40 0x7f477c1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f47823bf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f477c102d40 0x7f477c1009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37622/0 (socket says 172.21.15.33:37622) 2024-09-16T00:22:27.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f47823bf640 1 -- 172.21.15.33:0/3036590892 learned_addr learned my addr 172.21.15.33:0/3036590892 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f4781bbe640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f477c103660 0x7f477c100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.377+0000 7f4782bc0640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f477c10c170 0x7f477c101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.378+0000 7f47823bf640 1 -- 172.21.15.33:0/3036590892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f477c10c170 msgr2=0x7f477c101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.378+0000 7f47823bf640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f477c10c170 0x7f477c101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.378+0000 7f47823bf640 1 -- 172.21.15.33:0/3036590892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f477c103660 msgr2=0x7f477c100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.378+0000 7f47823bf640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f477c103660 0x7f477c100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.378+0000 7f47823bf640 1 -- 172.21.15.33:0/3036590892 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f477c1a6910 con 0x7f477c102d40 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.378+0000 7f4781bbe640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f477c103660 0x7f477c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.378+0000 7f47823bf640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f477c102d40 0x7f477c1009a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f477800e9f0 tx=0x7f477800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.378+0000 7f47737fe640 1 -- 172.21.15.33:0/3036590892 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477800cd80 con 0x7f477c102d40 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.378+0000 7f4784623640 1 -- 172.21.15.33:0/3036590892 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f477c1a6c00 con 0x7f477c102d40 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.378+0000 7f47737fe640 1 -- 172.21.15.33:0/3036590892 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4778010910 con 0x7f477c102d40 2024-09-16T00:22:27.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.378+0000 7f47737fe640 1 -- 172.21.15.33:0/3036590892 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47780188f0 con 0x7f477c102d40 2024-09-16T00:22:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.378+0000 7f4784623640 1 -- 172.21.15.33:0/3036590892 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f477c1a7190 con 0x7f477c102d40 2024-09-16T00:22:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.381+0000 7f47737fe640 1 -- 172.21.15.33:0/3036590892 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4778010ac0 con 0x7f477c102d40 2024-09-16T00:22:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.381+0000 7f47869fc740 1 Processor -- start 2024-09-16T00:22:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.381+0000 7f47869fc740 1 -- start start 2024-09-16T00:22:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f47869fc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de1850 0x562574de1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f47869fc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de2170 0x562574dec4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f47869fc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574deca30 0x562574deee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f47869fc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562574d46160 con 0x562574de2170 2024-09-16T00:22:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f47869fc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562574ced720 con 0x562574de1850 2024-09-16T00:22:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f47869fc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562574d42120 con 0x562574deca30 2024-09-16T00:22:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f4782bc0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de1850 0x562574de1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f47823bf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de2170 0x562574dec4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f4782bc0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de1850 0x562574de1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:48950/0 (socket says 172.21.15.33:48950) 2024-09-16T00:22:27.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f4782bc0640 1 -- 172.21.15.33:0/320712746 learned_addr learned my addr 172.21.15.33:0/320712746 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:27.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f4781bbe640 1 --2- 172.21.15.33:0/320712746 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574deca30 0x562574deee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f4782bc0640 1 -- 172.21.15.33:0/320712746 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574deca30 msgr2=0x562574deee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:27.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f4782bc0640 1 --2- 172.21.15.33:0/320712746 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574deca30 0x562574deee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f4782bc0640 1 -- 172.21.15.33:0/320712746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de2170 msgr2=0x562574dec4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:27.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f4782bc0640 1 --2- 172.21.15.33:0/320712746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de2170 0x562574dec4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f4782bc0640 1 -- 172.21.15.33:0/320712746 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562574ab7ba0 con 0x562574de1850 2024-09-16T00:22:27.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f4781bbe640 1 --2- 172.21.15.33:0/320712746 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574deca30 0x562574deee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:27.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f47737fe640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f47680a3240 0x7f47680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.382+0000 7f4782bc0640 1 --2- 172.21.15.33:0/320712746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de1850 0x562574de1c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4774035000 tx=0x7f47740093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88436bc0e3a3401b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:27.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.383+0000 7f47823bf640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f47680a3240 0x7f47680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.383+0000 7f47737fe640 1 -- 172.21.15.33:0/3036590892 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f47780d0160 con 0x7f477c102d40 2024-09-16T00:22:27.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.383+0000 7f4770ff9640 1 -- 172.21.15.33:0/320712746 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774034040 con 0x562574de1850 2024-09-16T00:22:27.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.383+0000 7f4770ff9640 1 -- 172.21.15.33:0/320712746 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f477402f940 con 0x562574de1850 2024-09-16T00:22:27.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.383+0000 7f4770ff9640 1 -- 172.21.15.33:0/320712746 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774004280 con 0x562574de1850 2024-09-16T00:22:27.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.383+0000 7f47869fc740 1 -- 172.21.15.33:0/320712746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de1850 msgr2=0x562574de1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:27.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.383+0000 7f47823bf640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f47680a3240 0x7f47680a5700 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f47780a49e0 tx=0x7f47780d3ba0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:27.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.383+0000 7f47869fc740 1 --2- 172.21.15.33:0/320712746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de1850 0x562574de1c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4774035000 tx=0x7f47740093e0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 -- 172.21.15.33:0/320712746 shutdown_connections 2024-09-16T00:22:27.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 --2- 172.21.15.33:0/320712746 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574deca30 0x562574deee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 --2- 172.21.15.33:0/320712746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de2170 0x562574dec4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 --2- 172.21.15.33:0/320712746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de1850 0x562574de1c30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 -- 172.21.15.33:0/320712746 >> 172.21.15.33:0/320712746 conn(0x562574dddde0 msgr2=0x562574dde1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:27.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 -- 172.21.15.33:0/320712746 shutdown_connections 2024-09-16T00:22:27.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 -- 172.21.15.33:0/320712746 wait complete. 2024-09-16T00:22:27.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 -- start start 2024-09-16T00:22:27.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 Processor -- start 2024-09-16T00:22:27.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574de1850 0x562574de98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de2170 0x562574de9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574deca30 0x562574dea370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562574def600 con 0x562574de2170 2024-09-16T00:22:27.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562574cce900 con 0x562574deca30 2024-09-16T00:22:27.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562574def780 con 0x562574de1850 2024-09-16T00:22:27.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.384+0000 7f47869fc740 10 client.0 _ll_register_callbacks cb 0x562574ccc780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:27.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4781bbe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de2170 0x562574de9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4781bbe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de2170 0x562574de9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37638/0 (socket says 172.21.15.33:37638) 2024-09-16T00:22:27.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4781bbe640 1 -- 172.21.15.33:0/2614197640 learned_addr learned my addr 172.21.15.33:0/2614197640 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:27.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4782bc0640 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574de1850 0x562574de98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4782bc0640 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574deca30 0x562574dea370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4781bbe640 1 -- 172.21.15.33:0/2614197640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574de1850 msgr2=0x562574de98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:27.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4781bbe640 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574de1850 0x562574de98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4781bbe640 1 -- 172.21.15.33:0/2614197640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574deca30 msgr2=0x562574dea370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:27.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4781bbe640 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574deca30 0x562574dea370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4781bbe640 1 -- 172.21.15.33:0/2614197640 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562574d0ec50 con 0x562574de2170 2024-09-16T00:22:27.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4782bc0640 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574de1850 0x562574de98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:27.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4781bbe640 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de2170 0x562574de9e30 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f476c0129d0 tx=0x7f476c012e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e63d3b5de31b56f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:27.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4765ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:27.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4765ffb640 1 -- 172.21.15.33:0/2614197640 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c00fa40 con 0x562574de2170 2024-09-16T00:22:27.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4765ffb640 1 -- 172.21.15.33:0/2614197640 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f476c00fbe0 con 0x562574de2170 2024-09-16T00:22:27.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f47869fc740 1 -- 172.21.15.33:0/2614197640 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f477402f940 con 0x562574de2170 2024-09-16T00:22:27.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.385+0000 7f4765ffb640 1 -- 172.21.15.33:0/2614197640 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f476c00fde0 con 0x562574de2170 2024-09-16T00:22:27.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.386+0000 7f47869fc740 10 client.16338 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:27.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.386+0000 7f47869fc740 1 -- 172.21.15.33:0/2614197640 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562574dfb310 con 0x562574de2170 2024-09-16T00:22:27.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.386+0000 7f4765ffb640 1 -- 172.21.15.33:0/2614197640 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f477402f940 con 0x562574de2170 2024-09-16T00:22:27.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.386+0000 7f4765ffb640 1 -- 172.21.15.33:0/2614197640 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f476c01c4b0 con 0x562574de2170 2024-09-16T00:22:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.386+0000 7f47869fc740 10 client.16338 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.386+0000 7f47869fc740 10 client.16338 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:22:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.386+0000 7f47869fc740 20 client.16338 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.386+0000 7f47869fc740 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562574dfc870 0x562574dfec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.386+0000 7f47869fc740 4 client.16338 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.386+0000 7f47869fc740 1 -- 172.21.15.33:0/2614197640 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562574d53890 con 0x562574dfc870 2024-09-16T00:22:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.386+0000 7f4782bc0640 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562574dfc870 0x562574dfec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f4782bc0640 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562574dfc870 0x562574dfec90 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ae9808b4446e815 server_cookie=94a0a7012be9f8e in_seq=0 out_seq=0 2024-09-16T00:22:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f4765ffb640 10 client.16338 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f4765ffb640 1 -- 172.21.15.33:0/2614197640 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562574d53890 con 0x562574dfc870 2024-09-16T00:22:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f4765ffb640 10 client.16338 handle_command_reply: tid=0 2024-09-16T00:22:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f47869fc740 1 client.16338 shutdown 2024-09-16T00:22:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f47869fc740 1 -- 172.21.15.33:0/2614197640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de2170 msgr2=0x562574de9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f47869fc740 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de2170 0x562574de9e30 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f476c0129d0 tx=0x7f476c012e90 comp rx=0 tx=0).stop 2024-09-16T00:22:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f47869fc740 1 -- 172.21.15.33:0/2614197640 shutdown_connections 2024-09-16T00:22:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f47869fc740 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562574dfc870 0x562574dfec90 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f47869fc740 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574deca30 0x562574dea370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f47869fc740 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de2170 0x562574de9e30 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f4765ffb640 0 client.16338 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f47869fc740 1 --2- 172.21.15.33:0/2614197640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574de1850 0x562574de98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f47869fc740 1 -- 172.21.15.33:0/2614197640 >> 172.21.15.33:0/2614197640 conn(0x562574dddde0 msgr2=0x562574de3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.387+0000 7f47869fc740 1 -- 172.21.15.33:0/2614197640 shutdown_connections 2024-09-16T00:22:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.388+0000 7f47869fc740 1 -- 172.21.15.33:0/2614197640 wait complete. 2024-09-16T00:22:27.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.388+0000 7f47869fc740 20 client.16338 trim_cache size 0 max 16384 2024-09-16T00:22:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.405+0000 7f47869fc740 1 Processor -- start 2024-09-16T00:22:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.405+0000 7f47869fc740 1 -- start start 2024-09-16T00:22:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.405+0000 7f47869fc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574de1850 0x562574dea1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.405+0000 7f47869fc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de2170 0x562574ddcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.405+0000 7f47869fc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574deca30 0x562574ddd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.405+0000 7f47869fc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f477c1b3c00 con 0x562574deca30 2024-09-16T00:22:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.405+0000 7f47869fc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f477c1b14d0 con 0x562574de2170 2024-09-16T00:22:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.405+0000 7f47869fc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f477c1af320 con 0x562574de1850 2024-09-16T00:22:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4782bc0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574deca30 0x562574ddd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4782bc0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574deca30 0x562574ddd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37652/0 (socket says 172.21.15.33:37652) 2024-09-16T00:22:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4782bc0640 1 -- 172.21.15.33:0/1545362854 learned_addr learned my addr 172.21.15.33:0/1545362854 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4782bc0640 1 --2- 172.21.15.33:0/1545362854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574de1850 0x562574dea1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4781bbe640 1 --2- 172.21.15.33:0/1545362854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de2170 0x562574ddcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4782bc0640 1 -- 172.21.15.33:0/1545362854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574de1850 msgr2=0x562574dea1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4782bc0640 1 --2- 172.21.15.33:0/1545362854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574de1850 0x562574dea1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4782bc0640 1 -- 172.21.15.33:0/1545362854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de2170 msgr2=0x562574ddcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4782bc0640 1 --2- 172.21.15.33:0/1545362854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de2170 0x562574ddcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4782bc0640 1 -- 172.21.15.33:0/1545362854 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562574de1fa0 con 0x562574deca30 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4781bbe640 1 --2- 172.21.15.33:0/1545362854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de2170 0x562574ddcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4782bc0640 1 --2- 172.21.15.33:0/1545362854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574deca30 0x562574ddd2f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f47740039e0 tx=0x7f4774046040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151968ca19fad1e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4767fff640 1 -- 172.21.15.33:0/1545362854 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774049040 con 0x562574deca30 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4767fff640 1 -- 172.21.15.33:0/1545362854 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f477402fad0 con 0x562574deca30 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.406+0000 7f4767fff640 1 -- 172.21.15.33:0/1545362854 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774052b90 con 0x562574deca30 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.407+0000 7f47869fc740 1 -- 172.21.15.33:0/1545362854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574deca30 msgr2=0x562574ddd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.407+0000 7f47869fc740 1 --2- 172.21.15.33:0/1545362854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574deca30 0x562574ddd2f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f47740039e0 tx=0x7f4774046040 comp rx=0 tx=0).stop 2024-09-16T00:22:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.407+0000 7f47869fc740 1 -- 172.21.15.33:0/1545362854 shutdown_connections 2024-09-16T00:22:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.407+0000 7f47869fc740 1 --2- 172.21.15.33:0/1545362854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574deca30 0x562574ddd2f0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.407+0000 7f47869fc740 1 --2- 172.21.15.33:0/1545362854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de2170 0x562574ddcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.407+0000 7f47869fc740 1 --2- 172.21.15.33:0/1545362854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574de1850 0x562574dea1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.407+0000 7f47869fc740 1 -- 172.21.15.33:0/1545362854 >> 172.21.15.33:0/1545362854 conn(0x562574dddde0 msgr2=0x562574ddc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.407+0000 7f47869fc740 1 -- 172.21.15.33:0/1545362854 shutdown_connections 2024-09-16T00:22:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.407+0000 7f47869fc740 1 -- 172.21.15.33:0/1545362854 wait complete. 2024-09-16T00:22:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.407+0000 7f47869fc740 1 -- start start 2024-09-16T00:22:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f47869fc740 1 Processor -- start 2024-09-16T00:22:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f47869fc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de1850 0x562574df4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f47869fc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de2170 0x562574df52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f47869fc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574deca30 0x562574df1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f47869fc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f477c1af990 con 0x562574de1850 2024-09-16T00:22:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f47869fc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f477c1b0e00 con 0x562574de2170 2024-09-16T00:22:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f47869fc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f477c1b0060 con 0x562574deca30 2024-09-16T00:22:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f47869fc740 10 client.0 _ll_register_callbacks cb 0x7f476c009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f4782bc0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de1850 0x562574df4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f4782bc0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de1850 0x562574df4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37654/0 (socket says 172.21.15.33:37654) 2024-09-16T00:22:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f4782bc0640 1 -- 172.21.15.33:0/4017568803 learned_addr learned my addr 172.21.15.33:0/4017568803 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f4781bbe640 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de2170 0x562574df52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f4782bc0640 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574deca30 0x562574df1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f4781bbe640 1 -- 172.21.15.33:0/4017568803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574deca30 msgr2=0x562574df1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f4781bbe640 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574deca30 0x562574df1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f4781bbe640 1 -- 172.21.15.33:0/4017568803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de1850 msgr2=0x562574df4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f4781bbe640 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de1850 0x562574df4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f4781bbe640 1 -- 172.21.15.33:0/4017568803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562574df1b70 con 0x562574de2170 2024-09-16T00:22:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.408+0000 7f4782bc0640 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de1850 0x562574df4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f4781bbe640 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de2170 0x562574df52f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f476c01c9a0 tx=0x7f476c01cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b335467e107b9f6f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f4765ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:22:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f4765ffb640 1 -- 172.21.15.33:0/4017568803 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c01cd80 con 0x562574de2170 2024-09-16T00:22:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f4765ffb640 1 -- 172.21.15.33:0/4017568803 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f476c01c630 con 0x562574de2170 2024-09-16T00:22:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f47869fc740 1 -- 172.21.15.33:0/4017568803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f477402fad0 con 0x562574de2170 2024-09-16T00:22:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f4765ffb640 1 -- 172.21.15.33:0/4017568803 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f476c005070 con 0x562574de2170 2024-09-16T00:22:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f47869fc740 10 client.25883 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f47869fc740 1 -- 172.21.15.33:0/4017568803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562574e68190 con 0x562574de2170 2024-09-16T00:22:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f4765ffb640 1 -- 172.21.15.33:0/4017568803 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f477402fad0 con 0x562574de2170 2024-09-16T00:22:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f4765ffb640 1 -- 172.21.15.33:0/4017568803 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f476c003c50 con 0x562574de2170 2024-09-16T00:22:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f47869fc740 10 client.25883 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f47869fc740 10 client.25883 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:22:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f47869fc740 20 client.25883 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f47869fc740 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562574e68920 0x562574e6ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f47869fc740 4 client.25883 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}] 2024-09-16T00:22:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.409+0000 7f47869fc740 1 -- 172.21.15.33:0/4017568803 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}) -- 0x562574d4dbe0 con 0x562574e68920 2024-09-16T00:22:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.410+0000 7f4782bc0640 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562574e68920 0x562574e6ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.410+0000 7f4782bc0640 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562574e68920 0x562574e6ad40 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e77492eb0ee4787b server_cookie=c36cb9692052b2f in_seq=0 out_seq=0 2024-09-16T00:22:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.410+0000 7f4765ffb640 10 client.25883 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.477+0000 7f4765ffb640 1 -- 172.21.15.33:0/4017568803 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x562574d4dbe0 con 0x562574e68920 2024-09-16T00:22:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f4765ffb640 10 client.25883 handle_command_reply: tid=0 2024-09-16T00:22:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f47869fc740 1 client.25883 shutdown 2024-09-16T00:22:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f47869fc740 1 -- 172.21.15.33:0/4017568803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de2170 msgr2=0x562574df52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f47869fc740 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de2170 0x562574df52f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f476c01c9a0 tx=0x7f476c01cb90 comp rx=0 tx=0).stop 2024-09-16T00:22:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f47869fc740 1 -- 172.21.15.33:0/4017568803 shutdown_connections 2024-09-16T00:22:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f47869fc740 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562574e68920 0x562574e6ad40 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f47869fc740 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562574deca30 0x562574df1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f47869fc740 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562574de2170 0x562574df52f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f47869fc740 1 --2- 172.21.15.33:0/4017568803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562574de1850 0x562574df4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f4765ffb640 0 client.25883 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f47869fc740 1 -- 172.21.15.33:0/4017568803 >> 172.21.15.33:0/4017568803 conn(0x562574dddde0 msgr2=0x562574de3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f47869fc740 1 -- 172.21.15.33:0/4017568803 shutdown_connections 2024-09-16T00:22:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f47869fc740 1 -- 172.21.15.33:0/4017568803 wait complete. 2024-09-16T00:22:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.478+0000 7f47869fc740 20 client.25883 trim_cache size 0 max 16384 2024-09-16T00:22:27.480 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:22:27.480 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 18, 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "9279b40d": { 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 18, 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 145.3, 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:22:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:22:27.482 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:22:27.482 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:22:27.482 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:22:27.482 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:22:27.482 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:27.482 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:27.482 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:27.482 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:27.482 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:27.482 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:22:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.479+0000 7f4784623640 1 -- 172.21.15.33:0/3036590892 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f47680a3240 msgr2=0x7f47680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.479+0000 7f4784623640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f47680a3240 0x7f47680a5700 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f47780a49e0 tx=0x7f47780d3ba0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.479+0000 7f4784623640 1 -- 172.21.15.33:0/3036590892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f477c102d40 msgr2=0x7f477c1009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.479+0000 7f4784623640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f477c102d40 0x7f477c1009a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f477800e9f0 tx=0x7f477800eeb0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.479+0000 7f4784623640 1 -- 172.21.15.33:0/3036590892 shutdown_connections 2024-09-16T00:22:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.479+0000 7f4784623640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f47680a3240 0x7f47680a5700 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.479+0000 7f4784623640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f477c10c170 0x7f477c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.479+0000 7f4784623640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f477c103660 0x7f477c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.479+0000 7f4784623640 1 --2- 172.21.15.33:0/3036590892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f477c102d40 0x7f477c1009a0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.479+0000 7f4784623640 1 -- 172.21.15.33:0/3036590892 >> 172.21.15.33:0/3036590892 conn(0x7f477c0fc7a0 msgr2=0x7f477c0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.479+0000 7f4784623640 1 -- 172.21.15.33:0/3036590892 shutdown_connections 2024-09-16T00:22:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:27.479+0000 7f4784623640 1 -- 172.21.15.33:0/3036590892 wait complete. 2024-09-16T00:22:27.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 12.43 MB/sec execute 95 sec latency 228.190 ms 2024-09-16T00:22:28.262 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '9279b40d', quiesce took 3.5 seconds. Will release after: 21.85 2024-09-16T00:22:28.657 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 12.30 MB/sec execute 96 sec latency 1228.283 ms 2024-09-16T00:22:29.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:28 smithi033 ceph-mon[31243]: pgmap v510: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.8 MiB/s wr, 211 op/s 2024-09-16T00:22:29.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:28 smithi083 ceph-mon[35242]: pgmap v510: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.8 MiB/s wr, 211 op/s 2024-09-16T00:22:29.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:28 smithi060 ceph-mon[35185]: pgmap v510: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.8 MiB/s wr, 211 op/s 2024-09-16T00:22:29.658 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 12.17 MB/sec execute 97 sec latency 2228.388 ms 2024-09-16T00:22:30.658 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 12.05 MB/sec execute 98 sec latency 3228.496 ms 2024-09-16T00:22:31.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:30 smithi033 ceph-mon[31243]: pgmap v511: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 214 op/s 2024-09-16T00:22:31.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:30 smithi083 ceph-mon[35242]: pgmap v511: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 214 op/s 2024-09-16T00:22:31.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:30 smithi060 ceph-mon[35185]: pgmap v511: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 214 op/s 2024-09-16T00:22:31.658 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 11.92 MB/sec execute 99 sec latency 4228.626 ms 2024-09-16T00:22:32.658 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 11.81 MB/sec execute 100 sec latency 5228.680 ms 2024-09-16T00:22:33.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:32 smithi033 ceph-mon[31243]: pgmap v512: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-09-16T00:22:33.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:32 smithi083 ceph-mon[35242]: pgmap v512: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-09-16T00:22:33.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:32 smithi060 ceph-mon[35185]: pgmap v512: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-09-16T00:22:33.658 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 11.69 MB/sec execute 101 sec latency 6228.787 ms 2024-09-16T00:22:34.658 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 11.57 MB/sec execute 102 sec latency 7228.929 ms 2024-09-16T00:22:35.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:34 smithi083 ceph-mon[35242]: pgmap v513: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 169 op/s 2024-09-16T00:22:35.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:34 smithi060 ceph-mon[35185]: pgmap v513: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 169 op/s 2024-09-16T00:22:35.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:34 smithi033 ceph-mon[31243]: pgmap v513: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 169 op/s 2024-09-16T00:22:35.658 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 11.46 MB/sec execute 103 sec latency 8229.034 ms 2024-09-16T00:22:36.658 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 11.35 MB/sec execute 104 sec latency 9229.122 ms 2024-09-16T00:22:37.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:36 smithi083 ceph-mon[35242]: pgmap v514: 129 pgs: 129 active+clean; 149 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 141 op/s 2024-09-16T00:22:37.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:36 smithi060 ceph-mon[35185]: pgmap v514: 129 pgs: 129 active+clean; 149 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 141 op/s 2024-09-16T00:22:37.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:36 smithi033 ceph-mon[31243]: pgmap v514: 129 pgs: 129 active+clean; 149 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 141 op/s 2024-09-16T00:22:37.658 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 11.24 MB/sec execute 105 sec latency 10229.172 ms 2024-09-16T00:22:38.659 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 11.14 MB/sec execute 106 sec latency 11229.280 ms 2024-09-16T00:22:39.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:38 smithi083 ceph-mon[35242]: pgmap v515: 129 pgs: 129 active+clean; 149 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 1.1 MiB/s wr, 71 op/s 2024-09-16T00:22:39.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:38 smithi060 ceph-mon[35185]: pgmap v515: 129 pgs: 129 active+clean; 149 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 1.1 MiB/s wr, 71 op/s 2024-09-16T00:22:39.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:38 smithi033 ceph-mon[31243]: pgmap v515: 129 pgs: 129 active+clean; 149 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 1.1 MiB/s wr, 71 op/s 2024-09-16T00:22:39.659 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 11.03 MB/sec execute 107 sec latency 12229.390 ms 2024-09-16T00:22:40.659 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 10.93 MB/sec execute 108 sec latency 13229.510 ms 2024-09-16T00:22:41.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:40 smithi083 ceph-mon[35242]: pgmap v516: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-16T00:22:41.217 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:22:41.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:40 smithi060 ceph-mon[35185]: pgmap v516: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-16T00:22:41.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:40 smithi033 ceph-mon[31243]: pgmap v516: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-16T00:22:41.488 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:41.659 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 10.83 MB/sec execute 109 sec latency 14229.648 ms 2024-09-16T00:22:42.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.142+0000 7f6aa4f78640 1 -- 172.21.15.33:0/3441206369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6aa0101420 msgr2=0x7f6aa010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:42.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.142+0000 7f6aa4f78640 1 --2- 172.21.15.33:0/3441206369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6aa0101420 0x7f6aa010d950 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009cd0 tx=0x7f6a9802a8e0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.142+0000 7f6aa4f78640 1 -- 172.21.15.33:0/3441206369 shutdown_connections 2024-09-16T00:22:42.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.142+0000 7f6aa4f78640 1 --2- 172.21.15.33:0/3441206369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6aa0101420 0x7f6aa010d950 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.142+0000 7f6aa4f78640 1 --2- 172.21.15.33:0/3441206369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aa0100a80 0x7f6aa0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.142+0000 7f6aa4f78640 1 --2- 172.21.15.33:0/3441206369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6aa0106a80 0x7f6aa0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.142+0000 7f6aa4f78640 1 -- 172.21.15.33:0/3441206369 >> 172.21.15.33:0/3441206369 conn(0x7f6aa00fc7a0 msgr2=0x7f6aa00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:42.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.142+0000 7f6aa4f78640 1 -- 172.21.15.33:0/3441206369 shutdown_connections 2024-09-16T00:22:42.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.142+0000 7f6aa4f78640 1 -- 172.21.15.33:0/3441206369 wait complete. 2024-09-16T00:22:42.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.143+0000 7f6aa4f78640 1 Processor -- start 2024-09-16T00:22:42.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.143+0000 7f6aa4f78640 1 -- start start 2024-09-16T00:22:42.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.143+0000 7f6aa4f78640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6aa0100a80 0x7f6aa01029c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.143+0000 7f6aa4f78640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6aa0101420 0x7f6aa0102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.143+0000 7f6aa4f78640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aa0106a80 0x7f6aa0103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.143+0000 7f6aa4f78640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa0110e80 con 0x7f6aa0100a80 2024-09-16T00:22:42.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.143+0000 7f6aa4f78640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa0110d00 con 0x7f6aa0106a80 2024-09-16T00:22:42.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.143+0000 7f6aa4f78640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa0111000 con 0x7f6aa0101420 2024-09-16T00:22:42.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a9e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6aa0100a80 0x7f6aa01029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a9e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6aa0100a80 0x7f6aa01029c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36074/0 (socket says 172.21.15.33:36074) 2024-09-16T00:22:42.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a9e59c640 1 -- 172.21.15.33:0/2454579881 learned_addr learned my addr 172.21.15.33:0/2454579881 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:42.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a9ed9d640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aa0106a80 0x7f6aa0103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a9dd9b640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6aa0101420 0x7f6aa0102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a9e59c640 1 -- 172.21.15.33:0/2454579881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6aa0101420 msgr2=0x7f6aa0102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6aa0101420 0x7f6aa0102f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a9e59c640 1 -- 172.21.15.33:0/2454579881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aa0106a80 msgr2=0x7f6aa0103440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aa0106a80 0x7f6aa0103440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a9e59c640 1 -- 172.21.15.33:0/2454579881 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa01a6840 con 0x7f6aa0100a80 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a9dd9b640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6aa0101420 0x7f6aa0102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6aa0100a80 0x7f6aa01029c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000c920 tx=0x7f6a9000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a8f7fe640 1 -- 172.21.15.33:0/2454579881 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a90007c70 con 0x7f6aa0100a80 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a8f7fe640 1 -- 172.21.15.33:0/2454579881 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a90007e10 con 0x7f6aa0100a80 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6aa4f78640 1 -- 172.21.15.33:0/2454579881 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa01a6b30 con 0x7f6aa0100a80 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6a8f7fe640 1 -- 172.21.15.33:0/2454579881 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a90018890 con 0x7f6aa0100a80 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.144+0000 7f6aa4f78640 1 -- 172.21.15.33:0/2454579881 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa01a7070 con 0x7f6aa0100a80 2024-09-16T00:22:42.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.147+0000 7f6a8f7fe640 1 -- 172.21.15.33:0/2454579881 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6a90018a70 con 0x7f6aa0100a80 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.147+0000 7f6aa7351740 1 Processor -- start 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.147+0000 7f6aa7351740 1 -- start start 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.147+0000 7f6aa7351740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c4d4a0 0x55f425c4d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.147+0000 7f6aa7351740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c58140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.147+0000 7f6aa7351740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 0x55f425c5aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.147+0000 7f6aa7351740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f425bb32e0 con 0x55f425c58680 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.147+0000 7f6aa7351740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f425b5ace0 con 0x55f425c4d4a0 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.147+0000 7f6aa7351740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f425baf2a0 con 0x55f425c4ddc0 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.147+0000 7f6a9dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 0x55f425c5aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.147+0000 7f6a9dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 0x55f425c5aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36078/0 (socket says 172.21.15.33:36078) 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.147+0000 7f6a9dd9b640 1 -- 172.21.15.33:0/339091750 learned_addr learned my addr 172.21.15.33:0/339091750 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/339091750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c58140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a9ed9d640 1 --2- 172.21.15.33:0/339091750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c4d4a0 0x55f425c4d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a8f7fe640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6a880a3240 0x7f6a880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a9dd9b640 1 -- 172.21.15.33:0/339091750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 msgr2=0x55f425c58140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:42.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a9dd9b640 1 --2- 172.21.15.33:0/339091750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c58140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a9dd9b640 1 -- 172.21.15.33:0/339091750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c4d4a0 msgr2=0x55f425c4d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:42.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a9dd9b640 1 --2- 172.21.15.33:0/339091750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c4d4a0 0x55f425c4d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a9dd9b640 1 -- 172.21.15.33:0/339091750 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f425b3f440 con 0x55f425c58680 2024-09-16T00:22:42.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a9ed9d640 1 --2- 172.21.15.33:0/339091750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c4d4a0 0x55f425c4d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:42.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/339091750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c58140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:42.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a9ed9d640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6a880a3240 0x7f6a880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a8f7fe640 1 -- 172.21.15.33:0/2454579881 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6a900cfe60 con 0x7f6aa0100a80 2024-09-16T00:22:42.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a9dd9b640 1 --2- 172.21.15.33:0/339091750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 0x55f425c5aa70 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009c20 tx=0x7f6a9400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182deb927cd26a4d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:42.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a8cff9640 1 -- 172.21.15.33:0/339091750 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94016070 con 0x55f425c58680 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a8cff9640 1 -- 172.21.15.33:0/339091750 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a94011950 con 0x55f425c58680 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.148+0000 7f6a8cff9640 1 -- 172.21.15.33:0/339091750 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a940107f0 con 0x55f425c58680 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.149+0000 7f6aa7351740 1 -- 172.21.15.33:0/339091750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 msgr2=0x55f425c5aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.149+0000 7f6a9ed9d640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6a880a3240 0x7f6a880a5700 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009a60 tx=0x7f6a9803e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.149+0000 7f6aa7351740 1 --2- 172.21.15.33:0/339091750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 0x55f425c5aa70 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009c20 tx=0x7f6a9400e6c0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.149+0000 7f6aa7351740 1 -- 172.21.15.33:0/339091750 shutdown_connections 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.149+0000 7f6aa7351740 1 --2- 172.21.15.33:0/339091750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 0x55f425c5aa70 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.149+0000 7f6aa7351740 1 --2- 172.21.15.33:0/339091750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c58140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.149+0000 7f6aa7351740 1 --2- 172.21.15.33:0/339091750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c4d4a0 0x55f425c4d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.149+0000 7f6aa7351740 1 -- 172.21.15.33:0/339091750 >> 172.21.15.33:0/339091750 conn(0x55f425c49a30 msgr2=0x55f425c49e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.149+0000 7f6aa7351740 1 -- 172.21.15.33:0/339091750 shutdown_connections 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.149+0000 7f6aa7351740 1 -- 172.21.15.33:0/339091750 wait complete. 2024-09-16T00:22:42.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.149+0000 7f6aa7351740 1 -- start start 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6aa7351740 1 Processor -- start 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6aa7351740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4d4a0 0x55f425c57f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6aa7351740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c4ddc0 0x55f425c55210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6aa7351740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 0x55f425c55750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6aa7351740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f425c5b250 con 0x55f425c58680 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6aa7351740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f425b6e050 con 0x55f425c4ddc0 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6aa7351740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f425c5b3d0 con 0x55f425c4d4a0 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6aa7351740 10 client.0 _ll_register_callbacks cb 0x55f425bae160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6a9ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 0x55f425c55750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6a9ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 0x55f425c55750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36094/0 (socket says 172.21.15.33:36094) 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6a9ed9d640 1 -- 172.21.15.33:0/3326635907 learned_addr learned my addr 172.21.15.33:0/3326635907 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4d4a0 0x55f425c57f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6a9dd9b640 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c4ddc0 0x55f425c55210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6a9ed9d640 1 -- 172.21.15.33:0/3326635907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4d4a0 msgr2=0x55f425c57f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:42.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6a9ed9d640 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4d4a0 0x55f425c57f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6a9ed9d640 1 -- 172.21.15.33:0/3326635907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c4ddc0 msgr2=0x55f425c55210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:42.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6a9ed9d640 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c4ddc0 0x55f425c55210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6a9ed9d640 1 -- 172.21.15.33:0/3326635907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4259243c0 con 0x55f425c58680 2024-09-16T00:22:42.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.150+0000 7f6a9dd9b640 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c4ddc0 0x55f425c55210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:42.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6a9ed9d640 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 0x55f425c55750 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6a98035000 tx=0x7f6a98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a3e0bc835f81290 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:42.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6a85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:42.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6a85ffb640 1 -- 172.21.15.33:0/3326635907 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a980305a0 con 0x55f425c58680 2024-09-16T00:22:42.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6aa7351740 1 -- 172.21.15.33:0/3326635907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a94011950 con 0x55f425c58680 2024-09-16T00:22:42.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6a85ffb640 1 -- 172.21.15.33:0/3326635907 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6a98030740 con 0x55f425c58680 2024-09-16T00:22:42.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6a85ffb640 1 -- 172.21.15.33:0/3326635907 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6a98059050 con 0x55f425c58680 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6aa7351740 10 client.16356 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6aa7351740 1 -- 172.21.15.33:0/3326635907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f425c66c10 con 0x55f425c58680 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6a85ffb640 1 -- 172.21.15.33:0/3326635907 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a94011950 con 0x55f425c58680 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6a85ffb640 1 -- 172.21.15.33:0/3326635907 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f6a98030070 con 0x55f425c58680 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6aa7351740 10 client.16356 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6aa7351740 10 client.16356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6aa7351740 20 client.16356 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6aa7351740 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f425c68020 0x55f425c6a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6aa7351740 4 client.16356 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.151+0000 7f6aa7351740 1 -- 172.21.15.33:0/3326635907 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f425bc0850 con 0x55f425c68020 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.152+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f425c68020 0x55f425c6a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.152+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f425c68020 0x55f425c6a470 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=883e5143459b1da9 server_cookie=1d67a6850a2c9142 in_seq=0 out_seq=0 2024-09-16T00:22:42.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.152+0000 7f6a85ffb640 10 client.16356 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:42.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6a85ffb640 1 -- 172.21.15.33:0/3326635907 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f425bc0850 con 0x55f425c68020 2024-09-16T00:22:42.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6a85ffb640 10 client.16356 handle_command_reply: tid=0 2024-09-16T00:22:42.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6aa7351740 1 client.16356 shutdown 2024-09-16T00:22:42.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6aa7351740 1 -- 172.21.15.33:0/3326635907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 msgr2=0x55f425c55750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:42.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6aa7351740 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 0x55f425c55750 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6a98035000 tx=0x7f6a98004a40 comp rx=0 tx=0).stop 2024-09-16T00:22:42.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6aa7351740 1 -- 172.21.15.33:0/3326635907 shutdown_connections 2024-09-16T00:22:42.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6aa7351740 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f425c68020 0x55f425c6a470 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6aa7351740 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c58680 0x55f425c55750 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6aa7351740 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c4ddc0 0x55f425c55210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6a85ffb640 0 client.16356 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:42.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6aa7351740 1 --2- 172.21.15.33:0/3326635907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4d4a0 0x55f425c57f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6aa7351740 1 -- 172.21.15.33:0/3326635907 >> 172.21.15.33:0/3326635907 conn(0x55f425c49a30 msgr2=0x55f425c4f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:42.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6aa7351740 1 -- 172.21.15.33:0/3326635907 shutdown_connections 2024-09-16T00:22:42.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6aa7351740 1 -- 172.21.15.33:0/3326635907 wait complete. 2024-09-16T00:22:42.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.153+0000 7f6aa7351740 20 client.16356 trim_cache size 0 max 16384 2024-09-16T00:22:42.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.170+0000 7f6aa7351740 1 Processor -- start 2024-09-16T00:22:42.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.170+0000 7f6aa7351740 1 -- start start 2024-09-16T00:22:42.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.170+0000 7f6aa7351740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 0x55f425c644b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.170+0000 7f6aa7351740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c50760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.170+0000 7f6aa7351740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c58680 0x55f425c6fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.170+0000 7f6aa7351740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa01b3a40 con 0x55f425c4d4a0 2024-09-16T00:22:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.170+0000 7f6aa7351740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa01b1310 con 0x55f425c58680 2024-09-16T00:22:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.170+0000 7f6aa7351740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa01af160 con 0x55f425c4ddc0 2024-09-16T00:22:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a9e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 0x55f425c644b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a9e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 0x55f425c644b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36106/0 (socket says 172.21.15.33:36106) 2024-09-16T00:22:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a9e59c640 1 -- 172.21.15.33:0/2292407935 learned_addr learned my addr 172.21.15.33:0/2292407935 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a9dd9b640 1 --2- 172.21.15.33:0/2292407935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c50760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a9e59c640 1 -- 172.21.15.33:0/2292407935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 msgr2=0x55f425c50760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/2292407935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c50760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a9ed9d640 1 --2- 172.21.15.33:0/2292407935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c58680 0x55f425c6fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a9e59c640 1 -- 172.21.15.33:0/2292407935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c58680 msgr2=0x55f425c6fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/2292407935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c58680 0x55f425c6fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a9e59c640 1 -- 172.21.15.33:0/2292407935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f425c582d0 con 0x55f425c4d4a0 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/2292407935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 0x55f425c644b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6a9009bf90 tx=0x7f6a900d5040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d6175a29d6032f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a87fff640 1 -- 172.21.15.33:0/2292407935 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a900a4c70 con 0x55f425c4d4a0 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a87fff640 1 -- 172.21.15.33:0/2292407935 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a900bc0b0 con 0x55f425c4d4a0 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.171+0000 7f6a87fff640 1 -- 172.21.15.33:0/2292407935 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a900a47e0 con 0x55f425c4d4a0 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 -- 172.21.15.33:0/2292407935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 msgr2=0x55f425c644b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 --2- 172.21.15.33:0/2292407935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 0x55f425c644b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6a9009bf90 tx=0x7f6a900d5040 comp rx=0 tx=0).stop 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 -- 172.21.15.33:0/2292407935 shutdown_connections 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 --2- 172.21.15.33:0/2292407935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c58680 0x55f425c6fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 --2- 172.21.15.33:0/2292407935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c50760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 --2- 172.21.15.33:0/2292407935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 0x55f425c644b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 -- 172.21.15.33:0/2292407935 >> 172.21.15.33:0/2292407935 conn(0x55f425c49a30 msgr2=0x55f425c50050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 -- 172.21.15.33:0/2292407935 shutdown_connections 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 -- 172.21.15.33:0/2292407935 wait complete. 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 -- start start 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 Processor -- start 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 0x55f425c60db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.172+0000 7f6aa7351740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c5c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6aa7351740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c58680 0x55f425c5c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6aa7351740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa01af7d0 con 0x55f425c4d4a0 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6aa7351740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa01b0c40 con 0x55f425c58680 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6aa7351740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa01afea0 con 0x55f425c4ddc0 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6aa7351740 10 client.0 _ll_register_callbacks cb 0x7f6a98005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a9e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 0x55f425c60db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a9e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 0x55f425c60db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36114/0 (socket says 172.21.15.33:36114) 2024-09-16T00:22:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a9e59c640 1 -- 172.21.15.33:0/3757347516 learned_addr learned my addr 172.21.15.33:0/3757347516 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a9dd9b640 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c5c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a9ed9d640 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c58680 0x55f425c5c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a9e59c640 1 -- 172.21.15.33:0/3757347516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 msgr2=0x55f425c5c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c5c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a9e59c640 1 -- 172.21.15.33:0/3757347516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c58680 msgr2=0x55f425c5c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c58680 0x55f425c5c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a9e59c640 1 -- 172.21.15.33:0/3757347516 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f425c5ce70 con 0x55f425c4d4a0 2024-09-16T00:22:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a9ed9d640 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c58680 0x55f425c5c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a9e59c640 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 0x55f425c60db0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6a9009bd00 tx=0x7f6a9009c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a7e0e4729da42c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a85ffb640 1 -- 172.21.15.33:0/3757347516 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a90018d60 con 0x55f425c4d4a0 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a85ffb640 1 -- 172.21.15.33:0/3757347516 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6a90098670 con 0x55f425c4d4a0 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6aa7351740 1 -- 172.21.15.33:0/3757347516 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a900bc0b0 con 0x55f425c4d4a0 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.173+0000 7f6a85ffb640 1 -- 172.21.15.33:0/3757347516 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6a900a47e0 con 0x55f425c4d4a0 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.174+0000 7f6aa7351740 10 client.16362 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.174+0000 7f6aa7351740 1 -- 172.21.15.33:0/3757347516 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f425cd2d50 con 0x55f425c4d4a0 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.174+0000 7f6a85ffb640 1 -- 172.21.15.33:0/3757347516 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a900bc0b0 con 0x55f425c4d4a0 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.174+0000 7f6a85ffb640 1 -- 172.21.15.33:0/3757347516 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f6a90096b70 con 0x55f425c4d4a0 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.174+0000 7f6aa7351740 10 client.16362 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.174+0000 7f6aa7351740 10 client.16362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.174+0000 7f6aa7351740 20 client.16362 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.174+0000 7f6aa7351740 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f425cd33b0 0x55f425cd57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.174+0000 7f6aa7351740 4 client.16362 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:22:42.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.174+0000 7f6aa7351740 1 -- 172.21.15.33:0/3757347516 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f425bbad60 con 0x55f425cd33b0 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.174+0000 7f6a9dd9b640 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f425cd33b0 0x55f425cd57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6a9dd9b640 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f425cd33b0 0x55f425cd57d0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d388fda63f6c449 server_cookie=30d0304e24e695ea in_seq=0 out_seq=0 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6a85ffb640 10 client.16362 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6a85ffb640 1 -- 172.21.15.33:0/3757347516 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f425bbad60 con 0x55f425cd33b0 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6a85ffb640 10 client.16362 handle_command_reply: tid=0 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6aa7351740 1 client.16362 shutdown 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6aa7351740 1 -- 172.21.15.33:0/3757347516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 msgr2=0x55f425c60db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6aa7351740 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 0x55f425c60db0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6a9009bd00 tx=0x7f6a9009c180 comp rx=0 tx=0).stop 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6aa7351740 1 -- 172.21.15.33:0/3757347516 shutdown_connections 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6aa7351740 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f425cd33b0 0x55f425cd57d0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6aa7351740 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f425c58680 0x55f425c5c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6aa7351740 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f425c4ddc0 0x55f425c5c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6aa7351740 1 --2- 172.21.15.33:0/3757347516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f425c4d4a0 0x55f425c60db0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6a85ffb640 0 client.16362 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:42.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6aa7351740 1 -- 172.21.15.33:0/3757347516 >> 172.21.15.33:0/3757347516 conn(0x55f425c49a30 msgr2=0x55f425c4eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:42.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6aa7351740 1 -- 172.21.15.33:0/3757347516 shutdown_connections 2024-09-16T00:22:42.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6aa7351740 1 -- 172.21.15.33:0/3757347516 wait complete. 2024-09-16T00:22:42.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.175+0000 7f6aa7351740 20 client.16362 trim_cache size 0 max 16384 2024-09-16T00:22:42.188 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:22:42.188 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:22:42.188 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:22:42.188 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:22:42.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.176+0000 7f6aa4f78640 1 -- 172.21.15.33:0/2454579881 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6a880a3240 msgr2=0x7f6a880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:42.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.176+0000 7f6aa4f78640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6a880a3240 0x7f6a880a5700 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009a60 tx=0x7f6a9803e040 comp rx=0 tx=0).stop 2024-09-16T00:22:42.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.176+0000 7f6aa4f78640 1 -- 172.21.15.33:0/2454579881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6aa0100a80 msgr2=0x7f6aa01029c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:42.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.176+0000 7f6aa4f78640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6aa0100a80 0x7f6aa01029c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000c920 tx=0x7f6a9000cde0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.176+0000 7f6aa4f78640 1 -- 172.21.15.33:0/2454579881 shutdown_connections 2024-09-16T00:22:42.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.176+0000 7f6aa4f78640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6a880a3240 0x7f6a880a5700 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.176+0000 7f6aa4f78640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aa0106a80 0x7f6aa0103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.176+0000 7f6aa4f78640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6aa0101420 0x7f6aa0102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.176+0000 7f6aa4f78640 1 --2- 172.21.15.33:0/2454579881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6aa0100a80 0x7f6aa01029c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:42.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.176+0000 7f6aa4f78640 1 -- 172.21.15.33:0/2454579881 >> 172.21.15.33:0/2454579881 conn(0x7f6aa00fc7a0 msgr2=0x7f6aa0104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:42.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.176+0000 7f6aa4f78640 1 -- 172.21.15.33:0/2454579881 shutdown_connections 2024-09-16T00:22:42.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:42.176+0000 7f6aa4f78640 1 -- 172.21.15.33:0/2454579881 wait complete. 2024-09-16T00:22:42.659 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 10.73 MB/sec execute 110 sec latency 15229.755 ms 2024-09-16T00:22:43.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:42 smithi033 ceph-mon[31243]: pgmap v517: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-16T00:22:43.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:42 smithi083 ceph-mon[35242]: pgmap v517: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-16T00:22:43.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:42 smithi060 ceph-mon[35185]: pgmap v517: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-16T00:22:43.301 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:22:43.301 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:22:43.597 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:43.659 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 10.64 MB/sec execute 111 sec latency 16229.865 ms 2024-09-16T00:22:44.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.261+0000 7fab844e8640 1 -- 172.21.15.33:0/2007117758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab7c1027c0 msgr2=0x7fab7c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:44.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.261+0000 7fab844e8640 1 --2- 172.21.15.33:0/2007117758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab7c1027c0 0x7fab7c106810 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fab78009900 tx=0x7fab7802a410 comp rx=0 tx=0).stop 2024-09-16T00:22:44.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.261+0000 7fab844e8640 1 -- 172.21.15.33:0/2007117758 shutdown_connections 2024-09-16T00:22:44.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.261+0000 7fab844e8640 1 --2- 172.21.15.33:0/2007117758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab7c106f40 0x7fab7c10eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:44.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.261+0000 7fab844e8640 1 --2- 172.21.15.33:0/2007117758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab7c1027c0 0x7fab7c106810 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:44.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.261+0000 7fab844e8640 1 --2- 172.21.15.33:0/2007117758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab7c101e10 0x7fab7c1021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:44.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.262+0000 7fab844e8640 1 -- 172.21.15.33:0/2007117758 >> 172.21.15.33:0/2007117758 conn(0x7fab7c0fd8d0 msgr2=0x7fab7c0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:44.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.262+0000 7fab844e8640 1 -- 172.21.15.33:0/2007117758 shutdown_connections 2024-09-16T00:22:44.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.262+0000 7fab844e8640 1 -- 172.21.15.33:0/2007117758 wait complete. 2024-09-16T00:22:44.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.262+0000 7fab844e8640 1 Processor -- start 2024-09-16T00:22:44.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.262+0000 7fab844e8640 1 -- start start 2024-09-16T00:22:44.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.262+0000 7fab844e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab7c101e10 0x7fab7c19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:44.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.262+0000 7fab844e8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab7c1027c0 0x7fab7c19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:44.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.262+0000 7fab844e8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab7c106f40 0x7fab7c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:44.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.262+0000 7fab844e8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c111fa0 con 0x7fab7c101e10 2024-09-16T00:22:44.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.262+0000 7fab844e8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c111e20 con 0x7fab7c106f40 2024-09-16T00:22:44.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.262+0000 7fab844e8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c112120 con 0x7fab7c1027c0 2024-09-16T00:22:44.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.262+0000 7fab82284640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab7c101e10 0x7fab7c19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:44.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab82284640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab7c101e10 0x7fab7c19a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36124/0 (socket says 172.21.15.33:36124) 2024-09-16T00:22:44.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab82284640 1 -- 172.21.15.33:0/4139758936 learned_addr learned my addr 172.21.15.33:0/4139758936 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:44.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab82a85640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab7c106f40 0x7fab7c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:44.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab81a83640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab7c1027c0 0x7fab7c19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:44.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab82284640 1 -- 172.21.15.33:0/4139758936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab7c1027c0 msgr2=0x7fab7c19aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:44.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab82284640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab7c1027c0 0x7fab7c19aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:44.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab82284640 1 -- 172.21.15.33:0/4139758936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab7c106f40 msgr2=0x7fab7c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:44.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab82284640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab7c106f40 0x7fab7c19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:44.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab82284640 1 -- 172.21.15.33:0/4139758936 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7c19f610 con 0x7fab7c101e10 2024-09-16T00:22:44.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab82284640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab7c101e10 0x7fab7c19a660 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00d950 tx=0x7fab6c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:44.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab737fe640 1 -- 172.21.15.33:0/4139758936 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6c017070 con 0x7fab7c101e10 2024-09-16T00:22:44.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab737fe640 1 -- 172.21.15.33:0/4139758936 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab6c009c30 con 0x7fab7c101e10 2024-09-16T00:22:44.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab737fe640 1 -- 172.21.15.33:0/4139758936 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6c00f6b0 con 0x7fab7c101e10 2024-09-16T00:22:44.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab844e8640 1 -- 172.21.15.33:0/4139758936 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab7c19f900 con 0x7fab7c101e10 2024-09-16T00:22:44.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.263+0000 7fab844e8640 1 -- 172.21.15.33:0/4139758936 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab7c1a71e0 con 0x7fab7c101e10 2024-09-16T00:22:44.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.264+0000 7fab844e8640 1 -- 172.21.15.33:0/4139758936 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab7c1021f0 con 0x7fab7c101e10 2024-09-16T00:22:44.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.266+0000 7fab737fe640 1 -- 172.21.15.33:0/4139758936 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fab6c0109d0 con 0x7fab7c101e10 2024-09-16T00:22:44.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.267+0000 7fab737fe640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab680a3240 0x7fab680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:44.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.267+0000 7fab81a83640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab680a3240 0x7fab680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:44.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.267+0000 7fab737fe640 1 -- 172.21.15.33:0/4139758936 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fab6c0cfa80 con 0x7fab7c101e10 2024-09-16T00:22:44.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.268+0000 7fab81a83640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab680a3240 0x7fab680a5700 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0fdff0 tx=0x7fab78005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:44.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.270+0000 7fab737fe640 1 -- 172.21.15.33:0/4139758936 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fab6c009de0 con 0x7fab7c101e10 2024-09-16T00:22:44.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.483+0000 7fab844e8640 1 -- 172.21.15.33:0/4139758936 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab7c1a7480 con 0x7fab7c101e10 2024-09-16T00:22:44.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.483+0000 7fab737fe640 1 -- 172.21.15.33:0/4139758936 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7fab6c09c320 con 0x7fab7c101e10 2024-09-16T00:22:44.485 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:22:44.485 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:22:44.485 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:22:44.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.486+0000 7fab844e8640 1 -- 172.21.15.33:0/4139758936 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab680a3240 msgr2=0x7fab680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:44.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.486+0000 7fab844e8640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab680a3240 0x7fab680a5700 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0fdff0 tx=0x7fab78005a20 comp rx=0 tx=0).stop 2024-09-16T00:22:44.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.486+0000 7fab844e8640 1 -- 172.21.15.33:0/4139758936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab7c101e10 msgr2=0x7fab7c19a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:44.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.486+0000 7fab844e8640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab7c101e10 0x7fab7c19a660 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00d950 tx=0x7fab6c00de10 comp rx=0 tx=0).stop 2024-09-16T00:22:44.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.486+0000 7fab844e8640 1 -- 172.21.15.33:0/4139758936 shutdown_connections 2024-09-16T00:22:44.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.486+0000 7fab844e8640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab680a3240 0x7fab680a5700 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:44.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.486+0000 7fab844e8640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab7c106f40 0x7fab7c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:44.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.486+0000 7fab844e8640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab7c1027c0 0x7fab7c19aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:44.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.486+0000 7fab844e8640 1 --2- 172.21.15.33:0/4139758936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab7c101e10 0x7fab7c19a660 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:44.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.486+0000 7fab844e8640 1 -- 172.21.15.33:0/4139758936 >> 172.21.15.33:0/4139758936 conn(0x7fab7c0fd8d0 msgr2=0x7fab7c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:44.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.486+0000 7fab844e8640 1 -- 172.21.15.33:0/4139758936 shutdown_connections 2024-09-16T00:22:44.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:44.486+0000 7fab844e8640 1 -- 172.21.15.33:0/4139758936 wait complete. 2024-09-16T00:22:44.659 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 10.54 MB/sec execute 112 sec latency 17229.922 ms 2024-09-16T00:22:45.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:44 smithi083 ceph-mon[35242]: pgmap v518: 129 pgs: 129 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.0 MiB/s wr, 64 op/s 2024-09-16T00:22:45.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:44 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4139758936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:45.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:44 smithi060 ceph-mon[35185]: pgmap v518: 129 pgs: 129 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.0 MiB/s wr, 64 op/s 2024-09-16T00:22:45.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:44 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4139758936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:45.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:44 smithi033 ceph-mon[31243]: pgmap v518: 129 pgs: 129 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.0 MiB/s wr, 64 op/s 2024-09-16T00:22:45.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:44 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4139758936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:45.310 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:22:45.573 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:45.659 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 10.45 MB/sec execute 113 sec latency 18229.952 ms 2024-09-16T00:22:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.197+0000 7fb0ba402640 1 -- 172.21.15.33:0/3853930130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0b4103660 msgr2=0x7fb0b410bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.197+0000 7fb0ba402640 1 --2- 172.21.15.33:0/3853930130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0b4103660 0x7fb0b410bb50 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb09c009900 tx=0x7fb09c02a480 comp rx=0 tx=0).stop 2024-09-16T00:22:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.197+0000 7fb0ba402640 1 -- 172.21.15.33:0/3853930130 shutdown_connections 2024-09-16T00:22:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.197+0000 7fb0ba402640 1 --2- 172.21.15.33:0/3853930130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0b410c170 0x7fb0b410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.197+0000 7fb0ba402640 1 --2- 172.21.15.33:0/3853930130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0b4103660 0x7fb0b410bb50 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.197+0000 7fb0ba402640 1 --2- 172.21.15.33:0/3853930130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0b4102d40 0x7fb0b4103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.197+0000 7fb0ba402640 1 -- 172.21.15.33:0/3853930130 >> 172.21.15.33:0/3853930130 conn(0x7fb0b40fc7a0 msgr2=0x7fb0b40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.197+0000 7fb0ba402640 1 -- 172.21.15.33:0/3853930130 shutdown_connections 2024-09-16T00:22:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.197+0000 7fb0ba402640 1 -- 172.21.15.33:0/3853930130 wait complete. 2024-09-16T00:22:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.197+0000 7fb0ba402640 1 Processor -- start 2024-09-16T00:22:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.197+0000 7fb0ba402640 1 -- start start 2024-09-16T00:22:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0ba402640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0b4102d40 0x7fb0b41009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0ba402640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0b4103660 0x7fb0b4100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0ba402640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0b410c170 0x7fb0b4101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0ba402640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b4112030 con 0x7fb0b4102d40 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0ba402640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b4111eb0 con 0x7fb0b410c170 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0ba402640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41121b0 con 0x7fb0b4103660 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0b3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0b4102d40 0x7fb0b41009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0b3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0b4102d40 0x7fb0b41009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36144/0 (socket says 172.21.15.33:36144) 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0b3fff640 1 -- 172.21.15.33:0/1231313445 learned_addr learned my addr 172.21.15.33:0/1231313445 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0b899f640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0b410c170 0x7fb0b4101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0abfff640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0b4103660 0x7fb0b4100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0b3fff640 1 -- 172.21.15.33:0/1231313445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0b4103660 msgr2=0x7fb0b4100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0b4103660 0x7fb0b4100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0b3fff640 1 -- 172.21.15.33:0/1231313445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0b410c170 msgr2=0x7fb0b4101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0b410c170 0x7fb0b4101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0b3fff640 1 -- 172.21.15.33:0/1231313445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b41a6a40 con 0x7fb0b4102d40 2024-09-16T00:22:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0abfff640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0b4103660 0x7fb0b4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0b4102d40 0x7fb0b41009a0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400b4a0 tx=0x7fb0a400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.198+0000 7fb0b1ffb640 1 -- 172.21.15.33:0/1231313445 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4014070 con 0x7fb0b4102d40 2024-09-16T00:22:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.199+0000 7fb0b1ffb640 1 -- 172.21.15.33:0/1231313445 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0a4010af0 con 0x7fb0b4102d40 2024-09-16T00:22:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.199+0000 7fb0ba402640 1 -- 172.21.15.33:0/1231313445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b41a6d30 con 0x7fb0b4102d40 2024-09-16T00:22:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.199+0000 7fb0b1ffb640 1 -- 172.21.15.33:0/1231313445 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4017d10 con 0x7fb0b4102d40 2024-09-16T00:22:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.199+0000 7fb0ba402640 1 -- 172.21.15.33:0/1231313445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b41a7210 con 0x7fb0b4102d40 2024-09-16T00:22:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.201+0000 7fb0b1ffb640 1 -- 172.21.15.33:0/1231313445 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb0a4017070 con 0x7fb0b4102d40 2024-09-16T00:22:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.201+0000 7fb0bc7db740 1 Processor -- start 2024-09-16T00:22:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.201+0000 7fb0bc7db740 1 -- start start 2024-09-16T00:22:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.201+0000 7fb0bc7db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 0x56411c889090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.201+0000 7fb0bc7db740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 0x56411c889a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.201+0000 7fb0bc7db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c896760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.201+0000 7fb0bc7db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56411c7ef2f0 con 0x56411c88ab40 2024-09-16T00:22:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.201+0000 7fb0bc7db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56411c796cf0 con 0x56411c894370 2024-09-16T00:22:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.201+0000 7fb0bc7db740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56411c7eb2b0 con 0x56411c8895d0 2024-09-16T00:22:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.201+0000 7fb0b899f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 0x56411c889090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0b899f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 0x56411c889090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36150/0 (socket says 172.21.15.33:36150) 2024-09-16T00:22:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0b899f640 1 -- 172.21.15.33:0/1360336605 learned_addr learned my addr 172.21.15.33:0/1360336605 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0abfff640 1 --2- 172.21.15.33:0/1360336605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c896760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/1360336605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 0x56411c889a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0b899f640 1 -- 172.21.15.33:0/1360336605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 msgr2=0x56411c889a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0b899f640 1 --2- 172.21.15.33:0/1360336605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 0x56411c889a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0b1ffb640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb0a00a3240 0x7fb0a00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0b899f640 1 -- 172.21.15.33:0/1360336605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 msgr2=0x56411c896760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0b899f640 1 --2- 172.21.15.33:0/1360336605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c896760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0b899f640 1 -- 172.21.15.33:0/1360336605 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56411c77b3a0 con 0x56411c88ab40 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0b1ffb640 1 -- 172.21.15.33:0/1231313445 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb0a40cf160 con 0x7fb0b4102d40 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0b899f640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb0a00a3240 0x7fb0a00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0b899f640 1 --2- 172.21.15.33:0/1360336605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 0x56411c889090 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00a430 tx=0x7fb0ac009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7193c7167399ccc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.202+0000 7fb0aa7fc640 1 -- 172.21.15.33:0/1360336605 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac010c10 con 0x56411c88ab40 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.203+0000 7fb0aa7fc640 1 -- 172.21.15.33:0/1360336605 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0ac014dd0 con 0x56411c88ab40 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.203+0000 7fb0aa7fc640 1 -- 172.21.15.33:0/1360336605 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac01f5a0 con 0x56411c88ab40 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.203+0000 7fb0bc7db740 1 -- 172.21.15.33:0/1360336605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 msgr2=0x56411c889090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.203+0000 7fb0b899f640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb0a00a3240 0x7fb0a00a5700 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac01e000 tx=0x7fb0ac01f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.203+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/1360336605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 0x56411c889090 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00a430 tx=0x7fb0ac009650 comp rx=0 tx=0).stop 2024-09-16T00:22:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.203+0000 7fb0bc7db740 1 -- 172.21.15.33:0/1360336605 shutdown_connections 2024-09-16T00:22:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.203+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/1360336605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c896760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.203+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/1360336605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 0x56411c889a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.203+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/1360336605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 0x56411c889090 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.203+0000 7fb0bc7db740 1 -- 172.21.15.33:0/1360336605 >> 172.21.15.33:0/1360336605 conn(0x56411c852310 msgr2=0x56411c885c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.203+0000 7fb0bc7db740 1 -- 172.21.15.33:0/1360336605 shutdown_connections 2024-09-16T00:22:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.203+0000 7fb0bc7db740 1 -- 172.21.15.33:0/1360336605 wait complete. 2024-09-16T00:22:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0bc7db740 1 -- start start 2024-09-16T00:22:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0bc7db740 1 Processor -- start 2024-09-16T00:22:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0bc7db740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 0x56411c893d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0bc7db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c88ab40 0x56411c891000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0bc7db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c894370 0x56411c8915d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0bc7db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56411c897070 con 0x56411c894370 2024-09-16T00:22:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0bc7db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56411c7aa060 con 0x56411c88ab40 2024-09-16T00:22:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0bc7db740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56411c8971f0 con 0x56411c8895d0 2024-09-16T00:22:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0bc7db740 10 client.0 _ll_register_callbacks cb 0x56411c4fd8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0b899f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c894370 0x56411c8915d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0b899f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c894370 0x56411c8915d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36154/0 (socket says 172.21.15.33:36154) 2024-09-16T00:22:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0b899f640 1 -- 172.21.15.33:0/2059688775 learned_addr learned my addr 172.21.15.33:0/2059688775 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.204+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 0x56411c893d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0abfff640 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c88ab40 0x56411c891000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0b899f640 1 -- 172.21.15.33:0/2059688775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 msgr2=0x56411c893d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0b899f640 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 0x56411c893d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0b899f640 1 -- 172.21.15.33:0/2059688775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c88ab40 msgr2=0x56411c891000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0b899f640 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c88ab40 0x56411c891000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0b899f640 1 -- 172.21.15.33:0/2059688775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56411c5603c0 con 0x56411c894370 2024-09-16T00:22:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0b899f640 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c894370 0x56411c8915d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac004b90 tx=0x7fb0ac010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f16f0d1d9ddc64e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0937fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0937fe640 1 -- 172.21.15.33:0/2059688775 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac02f020 con 0x56411c894370 2024-09-16T00:22:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0937fe640 1 -- 172.21.15.33:0/2059688775 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb0ac021840 con 0x56411c894370 2024-09-16T00:22:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0bc7db740 1 -- 172.21.15.33:0/2059688775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0ac014dd0 con 0x56411c894370 2024-09-16T00:22:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0937fe640 1 -- 172.21.15.33:0/2059688775 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb0ac03a050 con 0x56411c894370 2024-09-16T00:22:46.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0bc7db740 10 client.16380 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:46.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0bc7db740 1 -- 172.21.15.33:0/2059688775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56411c8a2af0 con 0x56411c894370 2024-09-16T00:22:46.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.205+0000 7fb0937fe640 1 -- 172.21.15.33:0/2059688775 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0ac014dd0 con 0x56411c894370 2024-09-16T00:22:46.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.206+0000 7fb0937fe640 1 -- 172.21.15.33:0/2059688775 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fb0ac02a070 con 0x56411c894370 2024-09-16T00:22:46.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.206+0000 7fb0bc7db740 10 client.16380 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:46.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.206+0000 7fb0bc7db740 10 client.16380 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:22:46.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.206+0000 7fb0bc7db740 20 client.16380 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:46.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.206+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56411c8a3f50 0x56411c8a6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.206+0000 7fb0bc7db740 4 client.16380 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:46.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.206+0000 7fb0bc7db740 1 -- 172.21.15.33:0/2059688775 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56411c7fc860 con 0x56411c8a3f50 2024-09-16T00:22:46.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.206+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56411c8a3f50 0x56411c8a6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.206+0000 7fb0937fe640 10 client.16380 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:22:46.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.206+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56411c8a3f50 0x56411c8a6340 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5bdea3c8fc2c97ef server_cookie=1f2329aefe7efb2d in_seq=0 out_seq=0 2024-09-16T00:22:46.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0937fe640 1 -- 172.21.15.33:0/2059688775 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56411c7fc860 con 0x56411c8a3f50 2024-09-16T00:22:46.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0937fe640 10 client.16380 handle_command_reply: tid=0 2024-09-16T00:22:46.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0bc7db740 1 client.16380 shutdown 2024-09-16T00:22:46.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0bc7db740 1 -- 172.21.15.33:0/2059688775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c894370 msgr2=0x56411c8915d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:46.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c894370 0x56411c8915d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac004b90 tx=0x7fb0ac010c20 comp rx=0 tx=0).stop 2024-09-16T00:22:46.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0bc7db740 1 -- 172.21.15.33:0/2059688775 shutdown_connections 2024-09-16T00:22:46.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56411c8a3f50 0x56411c8a6340 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c894370 0x56411c8915d0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c88ab40 0x56411c891000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0937fe640 0 client.16380 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:22:46.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/2059688775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 0x56411c893d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0bc7db740 1 -- 172.21.15.33:0/2059688775 >> 172.21.15.33:0/2059688775 conn(0x56411c852310 msgr2=0x56411c88b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:46.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.207+0000 7fb0bc7db740 1 -- 172.21.15.33:0/2059688775 shutdown_connections 2024-09-16T00:22:46.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.208+0000 7fb0bc7db740 1 -- 172.21.15.33:0/2059688775 wait complete. 2024-09-16T00:22:46.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.208+0000 7fb0bc7db740 20 client.16380 trim_cache size 0 max 16384 2024-09-16T00:22:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0bc7db740 1 Processor -- start 2024-09-16T00:22:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0bc7db740 1 -- start start 2024-09-16T00:22:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0bc7db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c8895d0 0x56411c8a0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0bc7db740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c88ab40 0x56411c88c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0bc7db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c88c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0bc7db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41b3c70 con 0x56411c8895d0 2024-09-16T00:22:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0bc7db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41b1540 con 0x56411c894370 2024-09-16T00:22:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0bc7db740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41af390 con 0x56411c88ab40 2024-09-16T00:22:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0b3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c8895d0 0x56411c8a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0b3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c8895d0 0x56411c8a0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36166/0 (socket says 172.21.15.33:36166) 2024-09-16T00:22:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0b3fff640 1 -- 172.21.15.33:0/4175046781 learned_addr learned my addr 172.21.15.33:0/4175046781 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0b899f640 1 --2- 172.21.15.33:0/4175046781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c88c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0abfff640 1 --2- 172.21.15.33:0/4175046781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c88ab40 0x56411c88c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0b3fff640 1 -- 172.21.15.33:0/4175046781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c88ab40 msgr2=0x56411c88c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/4175046781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c88ab40 0x56411c88c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0b3fff640 1 -- 172.21.15.33:0/4175046781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 msgr2=0x56411c88c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/4175046781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c88c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0b3fff640 1 -- 172.21.15.33:0/4175046781 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56411c88cba0 con 0x56411c8895d0 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0abfff640 1 --2- 172.21.15.33:0/4175046781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c88ab40 0x56411c88c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0b899f640 1 --2- 172.21.15.33:0/4175046781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c88c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.221+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/4175046781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c8895d0 0x56411c8a0310 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4097c70 tx=0x7fb0a40bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c4a73e186d2a4d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.222+0000 7fb0a9ffb640 1 -- 172.21.15.33:0/4175046781 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a401e040 con 0x56411c8895d0 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.222+0000 7fb0a9ffb640 1 -- 172.21.15.33:0/4175046781 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0a409f650 con 0x56411c8895d0 2024-09-16T00:22:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.222+0000 7fb0a9ffb640 1 -- 172.21.15.33:0/4175046781 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4097910 con 0x56411c8895d0 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.222+0000 7fb0bc7db740 1 -- 172.21.15.33:0/4175046781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c8895d0 msgr2=0x56411c8a0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.222+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/4175046781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c8895d0 0x56411c8a0310 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4097c70 tx=0x7fb0a40bb040 comp rx=0 tx=0).stop 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.222+0000 7fb0bc7db740 1 -- 172.21.15.33:0/4175046781 shutdown_connections 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.222+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/4175046781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c88c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.222+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/4175046781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c88ab40 0x56411c88c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.222+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/4175046781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c8895d0 0x56411c8a0310 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.222+0000 7fb0bc7db740 1 -- 172.21.15.33:0/4175046781 >> 172.21.15.33:0/4175046781 conn(0x56411c852310 msgr2=0x56411c8900f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.222+0000 7fb0bc7db740 1 -- 172.21.15.33:0/4175046781 shutdown_connections 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.222+0000 7fb0bc7db740 1 -- 172.21.15.33:0/4175046781 wait complete. 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0bc7db740 1 -- start start 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0bc7db740 1 Processor -- start 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0bc7db740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 0x56411c89caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0bc7db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 0x56411c88dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0bc7db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c88e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0bc7db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41afa00 con 0x56411c88ab40 2024-09-16T00:22:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0bc7db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41b0e70 con 0x56411c894370 2024-09-16T00:22:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0bc7db740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41b00d0 con 0x56411c8895d0 2024-09-16T00:22:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0bc7db740 10 client.0 _ll_register_callbacks cb 0x7fb0b406b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0abfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 0x56411c88dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0abfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 0x56411c88dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36170/0 (socket says 172.21.15.33:36170) 2024-09-16T00:22:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0abfff640 1 -- 172.21.15.33:0/3919550200 learned_addr learned my addr 172.21.15.33:0/3919550200 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 0x56411c89caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.223+0000 7fb0b899f640 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c88e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0abfff640 1 -- 172.21.15.33:0/3919550200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 msgr2=0x56411c89caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0abfff640 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 0x56411c89caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0abfff640 1 -- 172.21.15.33:0/3919550200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 msgr2=0x56411c88e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:46.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0abfff640 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c88e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0abfff640 1 -- 172.21.15.33:0/3919550200 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56411c88e840 con 0x56411c88ab40 2024-09-16T00:22:46.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0abfff640 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 0x56411c88dc10 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fb09c009900 tx=0x7fb09c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9d24e4794efe2c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:46.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0937fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:46.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0937fe640 1 -- 172.21.15.33:0/3919550200 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09c035040 con 0x56411c88ab40 2024-09-16T00:22:46.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0937fe640 1 -- 172.21.15.33:0/3919550200 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb09c0037f0 con 0x56411c88ab40 2024-09-16T00:22:46.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0bc7db740 1 -- 172.21.15.33:0/3919550200 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0a409f650 con 0x56411c88ab40 2024-09-16T00:22:46.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0937fe640 1 -- 172.21.15.33:0/3919550200 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb09c0039f0 con 0x56411c88ab40 2024-09-16T00:22:46.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0bc7db740 10 client.16386 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:46.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.224+0000 7fb0bc7db740 1 -- 172.21.15.33:0/3919550200 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56411c90e7a0 con 0x56411c88ab40 2024-09-16T00:22:46.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.225+0000 7fb0937fe640 1 -- 172.21.15.33:0/3919550200 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0a409f650 con 0x56411c88ab40 2024-09-16T00:22:46.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.225+0000 7fb0937fe640 1 -- 172.21.15.33:0/3919550200 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fb09c044070 con 0x56411c88ab40 2024-09-16T00:22:46.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.225+0000 7fb0bc7db740 10 client.16386 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:46.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.225+0000 7fb0bc7db740 10 client.16386 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:22:46.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.225+0000 7fb0bc7db740 20 client.16386 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:46.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.225+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56411c90ef20 0x56411c911340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:46.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.225+0000 7fb0bc7db740 4 client.16386 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:22:46.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.225+0000 7fb0bc7db740 1 -- 172.21.15.33:0/3919550200 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56411c7f6d70 con 0x56411c90ef20 2024-09-16T00:22:46.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.225+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56411c90ef20 0x56411c911340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:46.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.225+0000 7fb0b3fff640 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56411c90ef20 0x56411c911340 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4b5bedd55348541d server_cookie=b8cbcd5334fddf62 in_seq=0 out_seq=0 2024-09-16T00:22:46.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.225+0000 7fb0937fe640 10 client.16386 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:22:46.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0937fe640 1 -- 172.21.15.33:0/3919550200 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56411c7f6d70 con 0x56411c90ef20 2024-09-16T00:22:46.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0937fe640 10 client.16386 handle_command_reply: tid=0 2024-09-16T00:22:46.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0bc7db740 1 client.16386 shutdown 2024-09-16T00:22:46.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0bc7db740 1 -- 172.21.15.33:0/3919550200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 msgr2=0x56411c88dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:46.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 0x56411c88dc10 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fb09c009900 tx=0x7fb09c004270 comp rx=0 tx=0).stop 2024-09-16T00:22:46.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0bc7db740 1 -- 172.21.15.33:0/3919550200 shutdown_connections 2024-09-16T00:22:46.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56411c90ef20 0x56411c911340 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56411c894370 0x56411c88e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0937fe640 0 client.16386 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:22:46.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56411c88ab40 0x56411c88dc10 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0bc7db740 1 --2- 172.21.15.33:0/3919550200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56411c8895d0 0x56411c89caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0bc7db740 1 -- 172.21.15.33:0/3919550200 >> 172.21.15.33:0/3919550200 conn(0x56411c852310 msgr2=0x56411c892860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:46.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0bc7db740 1 -- 172.21.15.33:0/3919550200 shutdown_connections 2024-09-16T00:22:46.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0bc7db740 1 -- 172.21.15.33:0/3919550200 wait complete. 2024-09-16T00:22:46.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0bc7db740 20 client.16386 trim_cache size 0 max 16384 2024-09-16T00:22:46.239 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:22:46.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0ba402640 1 -- 172.21.15.33:0/1231313445 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb0a00a3240 msgr2=0x7fb0a00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:46.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0ba402640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb0a00a3240 0x7fb0a00a5700 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac01e000 tx=0x7fb0ac01f000 comp rx=0 tx=0).stop 2024-09-16T00:22:46.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0ba402640 1 -- 172.21.15.33:0/1231313445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0b4102d40 msgr2=0x7fb0b41009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:46.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.226+0000 7fb0ba402640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0b4102d40 0x7fb0b41009a0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400b4a0 tx=0x7fb0a400b960 comp rx=0 tx=0).stop 2024-09-16T00:22:46.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.227+0000 7fb0ba402640 1 -- 172.21.15.33:0/1231313445 shutdown_connections 2024-09-16T00:22:46.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.227+0000 7fb0ba402640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb0a00a3240 0x7fb0a00a5700 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.227+0000 7fb0ba402640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0b410c170 0x7fb0b4101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.227+0000 7fb0ba402640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0b4103660 0x7fb0b4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.227+0000 7fb0ba402640 1 --2- 172.21.15.33:0/1231313445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0b4102d40 0x7fb0b41009a0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:46.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.227+0000 7fb0ba402640 1 -- 172.21.15.33:0/1231313445 >> 172.21.15.33:0/1231313445 conn(0x7fb0b40fc7a0 msgr2=0x7fb0b40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:46.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.227+0000 7fb0ba402640 1 -- 172.21.15.33:0/1231313445 shutdown_connections 2024-09-16T00:22:46.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:46.227+0000 7fb0ba402640 1 -- 172.21.15.33:0/1231313445 wait complete. 2024-09-16T00:22:46.659 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 10.36 MB/sec execute 114 sec latency 19230.031 ms 2024-09-16T00:22:47.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:46 smithi033 ceph-mon[31243]: pgmap v519: 129 pgs: 129 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.3 MiB/s wr, 78 op/s 2024-09-16T00:22:47.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:46 smithi083 ceph-mon[35242]: pgmap v519: 129 pgs: 129 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.3 MiB/s wr, 78 op/s 2024-09-16T00:22:47.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:46 smithi060 ceph-mon[35185]: pgmap v519: 129 pgs: 129 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.3 MiB/s wr, 78 op/s 2024-09-16T00:22:47.278 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:22:47.541 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:47.659 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 10.27 MB/sec execute 115 sec latency 20230.123 ms 2024-09-16T00:22:48.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.030+0000 7f89fa914640 1 -- 172.21.15.33:0/1756426462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f4102570 msgr2=0x7f89f41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:48.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.030+0000 7f89fa914640 1 --2- 172.21.15.33:0/1756426462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f4102570 0x7f89f41029b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009900 tx=0x7f89dc02a480 comp rx=0 tx=0).stop 2024-09-16T00:22:48.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.030+0000 7f89fa914640 1 -- 172.21.15.33:0/1756426462 shutdown_connections 2024-09-16T00:22:48.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.030+0000 7f89fa914640 1 --2- 172.21.15.33:0/1756426462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f4102ef0 0x7f89f410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.030+0000 7f89fa914640 1 --2- 172.21.15.33:0/1756426462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f4102570 0x7f89f41029b0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.030+0000 7f89fa914640 1 --2- 172.21.15.33:0/1756426462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89f4100e20 0x7f89f4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.030+0000 7f89fa914640 1 -- 172.21.15.33:0/1756426462 >> 172.21.15.33:0/1756426462 conn(0x7f89f40fc7d0 msgr2=0x7f89f40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:48.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.030+0000 7f89fa914640 1 -- 172.21.15.33:0/1756426462 shutdown_connections 2024-09-16T00:22:48.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.030+0000 7f89fa914640 1 -- 172.21.15.33:0/1756426462 wait complete. 2024-09-16T00:22:48.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89fa914640 1 Processor -- start 2024-09-16T00:22:48.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89fa914640 1 -- start start 2024-09-16T00:22:48.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89fa914640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f4100e20 0x7f89f41973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89fa914640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89f4102570 0x7f89f4197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89fa914640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f4102ef0 0x7f89f419bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89fa914640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f89f4110f80 con 0x7f89f4102ef0 2024-09-16T00:22:48.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89fa914640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f89f4110e00 con 0x7f89f4102570 2024-09-16T00:22:48.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89fa914640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89f4111100 con 0x7f89f4100e20 2024-09-16T00:22:48.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89f8eb1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f4102ef0 0x7f89f419bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89f8eb1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f4102ef0 0x7f89f419bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36194/0 (socket says 172.21.15.33:36194) 2024-09-16T00:22:48.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89f8eb1640 1 -- 172.21.15.33:0/949520702 learned_addr learned my addr 172.21.15.33:0/949520702 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:48.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89f3fff640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f4100e20 0x7f89f41973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89ebfff640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89f4102570 0x7f89f4197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89f8eb1640 1 -- 172.21.15.33:0/949520702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f4100e20 msgr2=0x7f89f41973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:48.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89f8eb1640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f4100e20 0x7f89f41973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89f8eb1640 1 -- 172.21.15.33:0/949520702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89f4102570 msgr2=0x7f89f4197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:48.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89f8eb1640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89f4102570 0x7f89f4197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89f8eb1640 1 -- 172.21.15.33:0/949520702 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f419c390 con 0x7f89f4102ef0 2024-09-16T00:22:48.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.031+0000 7f89f8eb1640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f4102ef0 0x7f89f419bcb0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00eaa0 tx=0x7f89ec00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:48.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.032+0000 7f89f1ffb640 1 -- 172.21.15.33:0/949520702 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec00cdb0 con 0x7f89f4102ef0 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.032+0000 7f89fa914640 1 -- 172.21.15.33:0/949520702 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89f419c680 con 0x7f89f4102ef0 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.032+0000 7f89f1ffb640 1 -- 172.21.15.33:0/949520702 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89ec005020 con 0x7f89f4102ef0 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.032+0000 7f89f1ffb640 1 -- 172.21.15.33:0/949520702 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec0116d0 con 0x7f89f4102ef0 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.032+0000 7f89fa914640 1 -- 172.21.15.33:0/949520702 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89f41a3f60 con 0x7f89f4102ef0 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.034+0000 7f89f1ffb640 1 -- 172.21.15.33:0/949520702 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f89ec0054d0 con 0x7f89f4102ef0 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.034+0000 7f89fcced740 1 Processor -- start 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.034+0000 7f89fcced740 1 -- start start 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.034+0000 7f89fcced740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b5586d50 0x55c4b55852a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.034+0000 7f89fcced740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b55857e0 0x55c4b5585c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.034+0000 7f89fcced740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5590580 0x55c4b5592970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.034+0000 7f89fcced740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c4b54e92f0 con 0x55c4b55857e0 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.034+0000 7f89fcced740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c4b5490cf0 con 0x55c4b5590580 2024-09-16T00:22:48.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.034+0000 7f89fcced740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c4b54e52b0 con 0x55c4b5586d50 2024-09-16T00:22:48.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89f3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b55857e0 0x55c4b5585c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b5586d50 0x55c4b55852a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b5586d50 0x55c4b55852a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42264/0 (socket says 172.21.15.33:42264) 2024-09-16T00:22:48.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89f3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b55857e0 0x55c4b5585c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36210/0 (socket says 172.21.15.33:36210) 2024-09-16T00:22:48.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89ebfff640 1 -- 172.21.15.33:0/1308624955 learned_addr learned my addr 172.21.15.33:0/1308624955 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:48.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89ebfff640 1 --2- 172.21.15.33:0/1308624955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5590580 0x55c4b5592970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89f1ffb640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89e00a3240 0x7f89e00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89f8eb1640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89e00a3240 0x7f89e00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89f3fff640 1 -- 172.21.15.33:0/1308624955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b5586d50 msgr2=0x55c4b55852a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:48.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89f3fff640 1 --2- 172.21.15.33:0/1308624955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b5586d50 0x55c4b55852a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89f3fff640 1 -- 172.21.15.33:0/1308624955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5590580 msgr2=0x55c4b5592970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:48.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89f3fff640 1 --2- 172.21.15.33:0/1308624955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5590580 0x55c4b5592970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89f3fff640 1 -- 172.21.15.33:0/1308624955 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4b54753a0 con 0x55c4b55857e0 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89f1ffb640 1 -- 172.21.15.33:0/949520702 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f89ec0cffd0 con 0x7f89f4102ef0 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89f3fff640 1 --2- 172.21.15.33:0/1308624955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b55857e0 0x55c4b5585c20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f89e400bbf0 tx=0x7f89e40107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49fe6f4096b6b97 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89f8eb1640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89e00a3240 0x7f89e00a5700 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009270 tx=0x7f89ec091650 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89ea7fc640 1 -- 172.21.15.33:0/1308624955 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e4004030 con 0x55c4b55857e0 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.035+0000 7f89ea7fc640 1 -- 172.21.15.33:0/1308624955 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e40041d0 con 0x55c4b55857e0 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.036+0000 7f89ea7fc640 1 -- 172.21.15.33:0/1308624955 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e40116a0 con 0x55c4b55857e0 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.036+0000 7f89fcced740 1 -- 172.21.15.33:0/1308624955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b55857e0 msgr2=0x55c4b5585c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.036+0000 7f89fcced740 1 --2- 172.21.15.33:0/1308624955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b55857e0 0x55c4b5585c20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f89e400bbf0 tx=0x7f89e40107a0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.036+0000 7f89fcced740 1 -- 172.21.15.33:0/1308624955 shutdown_connections 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.036+0000 7f89fcced740 1 --2- 172.21.15.33:0/1308624955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5590580 0x55c4b5592970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.036+0000 7f89fcced740 1 --2- 172.21.15.33:0/1308624955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b55857e0 0x55c4b5585c20 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.036+0000 7f89fcced740 1 --2- 172.21.15.33:0/1308624955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b5586d50 0x55c4b55852a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.036+0000 7f89fcced740 1 -- 172.21.15.33:0/1308624955 >> 172.21.15.33:0/1308624955 conn(0x55c4b554c310 msgr2=0x55c4b5581e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.036+0000 7f89fcced740 1 -- 172.21.15.33:0/1308624955 shutdown_connections 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.036+0000 7f89fcced740 1 -- 172.21.15.33:0/1308624955 wait complete. 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89fcced740 1 -- start start 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89fcced740 1 Processor -- start 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89fcced740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b55857e0 0x55c4b558ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89fcced740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5586d50 0x55c4b558d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89fcced740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b558d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89fcced740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c4b5593280 con 0x55c4b5590580 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89fcced740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c4b54a4060 con 0x55c4b5586d50 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89fcced740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c4b5593400 con 0x55c4b55857e0 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89fcced740 10 client.0 _ll_register_callbacks cb 0x55c4b51f78c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:48.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b558d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b558d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36220/0 (socket says 172.21.15.33:36220) 2024-09-16T00:22:48.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89ebfff640 1 -- 172.21.15.33:0/4294173887 learned_addr learned my addr 172.21.15.33:0/4294173887 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:48.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89ebfff640 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b55857e0 0x55c4b558ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.037+0000 7f89f3fff640 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5586d50 0x55c4b558d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89ebfff640 1 -- 172.21.15.33:0/4294173887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b55857e0 msgr2=0x55c4b558ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:48.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89ebfff640 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b55857e0 0x55c4b558ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89ebfff640 1 -- 172.21.15.33:0/4294173887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5586d50 msgr2=0x55c4b558d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:48.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89ebfff640 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5586d50 0x55c4b558d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89ebfff640 1 -- 172.21.15.33:0/4294173887 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4b525a3c0 con 0x55c4b5590580 2024-09-16T00:22:48.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89ebfff640 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b558d9f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009590 tx=0x7f89dc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc028efbe7ebbb02 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:48.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:48.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89d37fe640 1 -- 172.21.15.33:0/4294173887 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc035040 con 0x55c4b5590580 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89fcced740 1 -- 172.21.15.33:0/4294173887 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89e40041d0 con 0x55c4b5590580 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89d37fe640 1 -- 172.21.15.33:0/4294173887 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f89dc0035f0 con 0x55c4b5590580 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89d37fe640 1 -- 172.21.15.33:0/4294173887 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f89dc0037f0 con 0x55c4b5590580 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89fcced740 10 client.16398 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.038+0000 7f89fcced740 1 -- 172.21.15.33:0/4294173887 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c4b559ed60 con 0x55c4b5590580 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.039+0000 7f89d37fe640 1 -- 172.21.15.33:0/4294173887 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e40041d0 con 0x55c4b5590580 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.039+0000 7f89d37fe640 1 -- 172.21.15.33:0/4294173887 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f89dc03a040 con 0x55c4b5590580 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.039+0000 7f89fcced740 10 client.16398 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.039+0000 7f89fcced740 10 client.16398 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.039+0000 7f89fcced740 20 client.16398 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.039+0000 7f89fcced740 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c4b55a0220 0x55c4b55a2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.039+0000 7f89fcced740 4 client.16398 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.039+0000 7f89fcced740 1 -- 172.21.15.33:0/4294173887 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4b54f6860 con 0x55c4b55a0220 2024-09-16T00:22:48.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.039+0000 7f89f3fff640 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c4b55a0220 0x55c4b55a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.040+0000 7f89d37fe640 10 client.16398 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.040+0000 7f89f3fff640 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c4b55a0220 0x55c4b55a2610 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6febdfa88da39b41 server_cookie=96293a5900c8ce4c in_seq=0 out_seq=0 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.040+0000 7f89d37fe640 1 -- 172.21.15.33:0/4294173887 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c4b54f6860 con 0x55c4b55a0220 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.040+0000 7f89d37fe640 10 client.16398 handle_command_reply: tid=0 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.040+0000 7f89fcced740 1 client.16398 shutdown 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.040+0000 7f89fcced740 1 -- 172.21.15.33:0/4294173887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 msgr2=0x55c4b558d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.040+0000 7f89fcced740 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b558d9f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009590 tx=0x7f89dc005d50 comp rx=0 tx=0).stop 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.041+0000 7f89fcced740 1 -- 172.21.15.33:0/4294173887 shutdown_connections 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.041+0000 7f89fcced740 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b558d9f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.041+0000 7f89fcced740 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c4b55a0220 0x55c4b55a2610 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.041+0000 7f89fcced740 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5586d50 0x55c4b558d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.041+0000 7f89fcced740 1 --2- 172.21.15.33:0/4294173887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b55857e0 0x55c4b558ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.041+0000 7f89fcced740 1 -- 172.21.15.33:0/4294173887 >> 172.21.15.33:0/4294173887 conn(0x55c4b554c310 msgr2=0x55c4b5587b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:48.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.041+0000 7f89d37fe640 0 client.16398 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:22:48.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.041+0000 7f89fcced740 1 -- 172.21.15.33:0/4294173887 shutdown_connections 2024-09-16T00:22:48.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.041+0000 7f89fcced740 1 -- 172.21.15.33:0/4294173887 wait complete. 2024-09-16T00:22:48.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.041+0000 7f89fcced740 20 client.16398 trim_cache size 0 max 16384 2024-09-16T00:22:48.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89fcced740 1 Processor -- start 2024-09-16T00:22:48.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89fcced740 1 -- start start 2024-09-16T00:22:48.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89fcced740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b55857e0 0x55c4b5582020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89fcced740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b5586d50 0x55c4b55a7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89fcced740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b55a83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89fcced740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41af860 con 0x55c4b5590580 2024-09-16T00:22:48.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89fcced740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41ad130 con 0x55c4b55857e0 2024-09-16T00:22:48.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89fcced740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41aaf80 con 0x55c4b5586d50 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b55a83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b55a83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36224/0 (socket says 172.21.15.33:36224) 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89ebfff640 1 -- 172.21.15.33:0/2440560753 learned_addr learned my addr 172.21.15.33:0/2440560753 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89ebfff640 1 --2- 172.21.15.33:0/2440560753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b55857e0 0x55c4b5582020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89f3fff640 1 --2- 172.21.15.33:0/2440560753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b5586d50 0x55c4b55a7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89ebfff640 1 -- 172.21.15.33:0/2440560753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b5586d50 msgr2=0x55c4b55a7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89ebfff640 1 --2- 172.21.15.33:0/2440560753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b5586d50 0x55c4b55a7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89ebfff640 1 -- 172.21.15.33:0/2440560753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b55857e0 msgr2=0x55c4b5582020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89ebfff640 1 --2- 172.21.15.33:0/2440560753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b55857e0 0x55c4b5582020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89ebfff640 1 -- 172.21.15.33:0/2440560753 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4b55a2c70 con 0x55c4b5590580 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.054+0000 7f89ebfff640 1 --2- 172.21.15.33:0/2440560753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b55a83e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f89dc004650 tx=0x7f89dc02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e3520c66b3de48 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.055+0000 7f89e9ffb640 1 -- 172.21.15.33:0/2440560753 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc035030 con 0x55c4b5590580 2024-09-16T00:22:48.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.055+0000 7f89e9ffb640 1 -- 172.21.15.33:0/2440560753 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89dc041040 con 0x55c4b5590580 2024-09-16T00:22:48.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.055+0000 7f89e9ffb640 1 -- 172.21.15.33:0/2440560753 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc0054c0 con 0x55c4b5590580 2024-09-16T00:22:48.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.055+0000 7f89fcced740 1 -- 172.21.15.33:0/2440560753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 msgr2=0x55c4b55a83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:48.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.055+0000 7f89fcced740 1 --2- 172.21.15.33:0/2440560753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b55a83e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f89dc004650 tx=0x7f89dc02a480 comp rx=0 tx=0).stop 2024-09-16T00:22:48.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.055+0000 7f89fcced740 1 -- 172.21.15.33:0/2440560753 shutdown_connections 2024-09-16T00:22:48.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.055+0000 7f89fcced740 1 --2- 172.21.15.33:0/2440560753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b55a83e0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.055+0000 7f89fcced740 1 --2- 172.21.15.33:0/2440560753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b5586d50 0x55c4b55a7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.055+0000 7f89fcced740 1 --2- 172.21.15.33:0/2440560753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b55857e0 0x55c4b5582020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.055+0000 7f89fcced740 1 -- 172.21.15.33:0/2440560753 >> 172.21.15.33:0/2440560753 conn(0x55c4b554c310 msgr2=0x55c4b5581910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.055+0000 7f89fcced740 1 -- 172.21.15.33:0/2440560753 shutdown_connections 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.055+0000 7f89fcced740 1 -- 172.21.15.33:0/2440560753 wait complete. 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.056+0000 7f89fcced740 1 -- start start 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.056+0000 7f89fcced740 1 Processor -- start 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.056+0000 7f89fcced740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b55857e0 0x55c4b5598690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.056+0000 7f89fcced740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5586d50 0x55c4b5598bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.056+0000 7f89fcced740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b5595660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.056+0000 7f89fcced740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41ab5f0 con 0x55c4b5590580 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.056+0000 7f89fcced740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41aca60 con 0x55c4b5586d50 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.056+0000 7f89fcced740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41abcc0 con 0x55c4b55857e0 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.056+0000 7f89fcced740 10 client.0 _ll_register_callbacks cb 0x7f89dc032880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.056+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b5595660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.056+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b5595660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36228/0 (socket says 172.21.15.33:36228) 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89ebfff640 1 -- 172.21.15.33:0/1804534687 learned_addr learned my addr 172.21.15.33:0/1804534687 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89ebfff640 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b55857e0 0x55c4b5598690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89f3fff640 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5586d50 0x55c4b5598bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89ebfff640 1 -- 172.21.15.33:0/1804534687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b55857e0 msgr2=0x55c4b5598690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89ebfff640 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b55857e0 0x55c4b5598690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89ebfff640 1 -- 172.21.15.33:0/1804534687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5586d50 msgr2=0x55c4b5598bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89ebfff640 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5586d50 0x55c4b5598bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89ebfff640 1 -- 172.21.15.33:0/1804534687 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4b5595cd0 con 0x55c4b5590580 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89ebfff640 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b5595660 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009680 tx=0x7f89dc0050c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7ee666e295aca1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89d37fe640 1 -- 172.21.15.33:0/1804534687 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc035040 con 0x55c4b5590580 2024-09-16T00:22:48.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89d37fe640 1 -- 172.21.15.33:0/1804534687 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f89dc0035f0 con 0x55c4b5590580 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89fcced740 1 -- 172.21.15.33:0/1804534687 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89dc041040 con 0x55c4b5590580 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89d37fe640 1 -- 172.21.15.33:0/1804534687 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f89dc032e00 con 0x55c4b5590580 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89fcced740 10 client.16404 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.057+0000 7f89fcced740 1 -- 172.21.15.33:0/1804534687 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c4b560b7e0 con 0x55c4b5590580 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.058+0000 7f89d37fe640 1 -- 172.21.15.33:0/1804534687 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89dc041040 con 0x55c4b5590580 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.058+0000 7f89d37fe640 1 -- 172.21.15.33:0/1804534687 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f89dc058030 con 0x55c4b5590580 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.058+0000 7f89fcced740 10 client.16404 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.058+0000 7f89fcced740 10 client.16404 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.058+0000 7f89fcced740 20 client.16404 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.058+0000 7f89fcced740 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c4b560be70 0x55c4b560e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.058+0000 7f89fcced740 4 client.16404 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.058+0000 7f89fcced740 1 -- 172.21.15.33:0/1804534687 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c4b54f0d70 con 0x55c4b560be70 2024-09-16T00:22:48.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.058+0000 7f89f3fff640 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c4b560be70 0x55c4b560e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89f3fff640 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c4b560be70 0x55c4b560e290 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d741f2d9f8592e4e server_cookie=55be83e34690e8d0 in_seq=0 out_seq=0 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89d37fe640 10 client.16404 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89d37fe640 1 -- 172.21.15.33:0/1804534687 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c4b54f0d70 con 0x55c4b560be70 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89d37fe640 10 client.16404 handle_command_reply: tid=0 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89fcced740 1 client.16404 shutdown 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89fcced740 1 -- 172.21.15.33:0/1804534687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 msgr2=0x55c4b5595660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89fcced740 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b5595660 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009680 tx=0x7f89dc0050c0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89fcced740 1 -- 172.21.15.33:0/1804534687 shutdown_connections 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89fcced740 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4b5590580 0x55c4b5595660 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89fcced740 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c4b560be70 0x55c4b560e290 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89fcced740 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4b5586d50 0x55c4b5598bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89fcced740 1 --2- 172.21.15.33:0/1804534687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4b55857e0 0x55c4b5598690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89fcced740 1 -- 172.21.15.33:0/1804534687 >> 172.21.15.33:0/1804534687 conn(0x55c4b554c310 msgr2=0x55c4b558f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:48.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89d37fe640 0 client.16404 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:22:48.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89fcced740 1 -- 172.21.15.33:0/1804534687 shutdown_connections 2024-09-16T00:22:48.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89fcced740 1 -- 172.21.15.33:0/1804534687 wait complete. 2024-09-16T00:22:48.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.059+0000 7f89fcced740 20 client.16404 trim_cache size 0 max 16384 2024-09-16T00:22:48.066 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:22:48.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.060+0000 7f89fa914640 1 -- 172.21.15.33:0/949520702 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89e00a3240 msgr2=0x7f89e00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:48.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.060+0000 7f89fa914640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89e00a3240 0x7f89e00a5700 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009270 tx=0x7f89ec091650 comp rx=0 tx=0).stop 2024-09-16T00:22:48.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.060+0000 7f89fa914640 1 -- 172.21.15.33:0/949520702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f4102ef0 msgr2=0x7f89f419bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:48.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.060+0000 7f89fa914640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f4102ef0 0x7f89f419bcb0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00eaa0 tx=0x7f89ec00ef60 comp rx=0 tx=0).stop 2024-09-16T00:22:48.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.060+0000 7f89fa914640 1 -- 172.21.15.33:0/949520702 shutdown_connections 2024-09-16T00:22:48.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.060+0000 7f89fa914640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89e00a3240 0x7f89e00a5700 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.060+0000 7f89fa914640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89f4102ef0 0x7f89f419bcb0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.060+0000 7f89fa914640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89f4102570 0x7f89f4197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.060+0000 7f89fa914640 1 --2- 172.21.15.33:0/949520702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f4100e20 0x7f89f41973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:48.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.060+0000 7f89fa914640 1 -- 172.21.15.33:0/949520702 >> 172.21.15.33:0/949520702 conn(0x7f89f40fc7d0 msgr2=0x7f89f41068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:48.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.060+0000 7f89fa914640 1 -- 172.21.15.33:0/949520702 shutdown_connections 2024-09-16T00:22:48.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:48.060+0000 7f89fa914640 1 -- 172.21.15.33:0/949520702 wait complete. 2024-09-16T00:22:48.659 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 10.18 MB/sec execute 116 sec latency 21230.215 ms 2024-09-16T00:22:49.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:48 smithi033 ceph-mon[31243]: pgmap v520: 129 pgs: 129 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 1.9 MiB/s wr, 71 op/s 2024-09-16T00:22:49.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:48 smithi083 ceph-mon[35242]: pgmap v520: 129 pgs: 129 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 1.9 MiB/s wr, 71 op/s 2024-09-16T00:22:49.212 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:22:49.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:48 smithi060 ceph-mon[35185]: pgmap v520: 129 pgs: 129 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 1.9 MiB/s wr, 71 op/s 2024-09-16T00:22:49.476 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:49.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 10.09 MB/sec execute 117 sec latency 22230.320 ms 2024-09-16T00:22:49.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.950+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/3124820747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a58102790 msgr2=0x7f9a581067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:49.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.950+0000 7f9a5dbbf640 1 --2- 172.21.15.33:0/3124820747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a58102790 0x7f9a581067e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009900 tx=0x7f9a4002a410 comp rx=0 tx=0).stop 2024-09-16T00:22:49.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.950+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/3124820747 shutdown_connections 2024-09-16T00:22:49.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.950+0000 7f9a5dbbf640 1 --2- 172.21.15.33:0/3124820747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a58106f10 0x7f9a5810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.950+0000 7f9a5dbbf640 1 --2- 172.21.15.33:0/3124820747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a58102790 0x7f9a581067e0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.950+0000 7f9a5dbbf640 1 --2- 172.21.15.33:0/3124820747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a58101de0 0x7f9a581021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.950+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/3124820747 >> 172.21.15.33:0/3124820747 conn(0x7f9a580fd8c0 msgr2=0x7f9a580ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:49.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.950+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/3124820747 shutdown_connections 2024-09-16T00:22:49.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.950+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/3124820747 wait complete. 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a5dbbf640 1 Processor -- start 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a5dbbf640 1 -- start start 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a5dbbf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a58101de0 0x7f9a581983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a5dbbf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a58102790 0x7f9a581988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a5dbbf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a58106f10 0x7f9a5819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a5dbbf640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9a58111f70 con 0x7f9a58106f10 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a5dbbf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9a58111df0 con 0x7f9a58101de0 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a5dbbf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581120f0 con 0x7f9a58102790 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a57fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a58106f10 0x7f9a5819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a57fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a58106f10 0x7f9a5819cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36254/0 (socket says 172.21.15.33:36254) 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a57fff640 1 -- 172.21.15.33:0/2157318221 learned_addr learned my addr 172.21.15.33:0/2157318221 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a577fe640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a58101de0 0x7f9a581983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.951+0000 7f9a4effd640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a58102790 0x7f9a581988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.952+0000 7f9a57fff640 1 -- 172.21.15.33:0/2157318221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a58102790 msgr2=0x7f9a581988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.952+0000 7f9a57fff640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a58102790 0x7f9a581988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.952+0000 7f9a57fff640 1 -- 172.21.15.33:0/2157318221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a58101de0 msgr2=0x7f9a581983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.952+0000 7f9a57fff640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a58101de0 0x7f9a581983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.952+0000 7f9a57fff640 1 -- 172.21.15.33:0/2157318221 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5819d360 con 0x7f9a58106f10 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.952+0000 7f9a4effd640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a58102790 0x7f9a581988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.952+0000 7f9a57fff640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a58106f10 0x7f9a5819cc80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000cce0 tx=0x7f9a5000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.952+0000 7f9a557fa640 1 -- 172.21.15.33:0/2157318221 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5000f450 con 0x7f9a58106f10 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.952+0000 7f9a557fa640 1 -- 172.21.15.33:0/2157318221 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a5000fad0 con 0x7f9a58106f10 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.952+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/2157318221 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a5819d650 con 0x7f9a58106f10 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.952+0000 7f9a557fa640 1 -- 172.21.15.33:0/2157318221 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a50018c10 con 0x7f9a58106f10 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.952+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/2157318221 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a581a4f30 con 0x7f9a58106f10 2024-09-16T00:22:49.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a557fa640 1 -- 172.21.15.33:0/2157318221 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9a5000f5f0 con 0x7f9a58106f10 2024-09-16T00:22:49.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a5ff98740 1 Processor -- start 2024-09-16T00:22:49.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a5ff98740 1 -- start start 2024-09-16T00:22:49.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a5ff98740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb651b40 0x55c2fb650090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a5ff98740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb650a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a5ff98740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb65b370 0x55c2fb65d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a5ff98740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c2fb5b62f0 con 0x55c2fb65b370 2024-09-16T00:22:49.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a5ff98740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c2fb55dcf0 con 0x55c2fb651b40 2024-09-16T00:22:49.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a5ff98740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c2fb5b22b0 con 0x55c2fb6505d0 2024-09-16T00:22:49.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a4effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb65b370 0x55c2fb65d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a4effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb65b370 0x55c2fb65d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36266/0 (socket says 172.21.15.33:36266) 2024-09-16T00:22:49.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a4effd640 1 -- 172.21.15.33:0/2333241820 learned_addr learned my addr 172.21.15.33:0/2333241820 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:49.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a4effd640 1 --2- 172.21.15.33:0/2333241820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb651b40 0x55c2fb650090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.955+0000 7f9a577fe640 1 --2- 172.21.15.33:0/2333241820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb650a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.956+0000 7f9a557fa640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a440a3240 0x7f9a440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.956+0000 7f9a4effd640 1 -- 172.21.15.33:0/2333241820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 msgr2=0x55c2fb650a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:49.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.956+0000 7f9a4effd640 1 --2- 172.21.15.33:0/2333241820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb650a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.956+0000 7f9a4effd640 1 -- 172.21.15.33:0/2333241820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb651b40 msgr2=0x55c2fb650090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:49.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.956+0000 7f9a4effd640 1 --2- 172.21.15.33:0/2333241820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb651b40 0x55c2fb650090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.956+0000 7f9a4effd640 1 -- 172.21.15.33:0/2333241820 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2fb5423a0 con 0x55c2fb65b370 2024-09-16T00:22:49.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.956+0000 7f9a57fff640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a440a3240 0x7f9a440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.956+0000 7f9a557fa640 1 -- 172.21.15.33:0/2157318221 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9a500d03c0 con 0x7f9a58106f10 2024-09-16T00:22:49.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.956+0000 7f9a4effd640 1 --2- 172.21.15.33:0/2333241820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb65b370 0x55c2fb65d760 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009900 tx=0x7f9a400056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f58fa1697b47ea5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:49.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.956+0000 7f9a4dffb640 1 -- 172.21.15.33:0/2333241820 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40035040 con 0x55c2fb65b370 2024-09-16T00:22:49.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.956+0000 7f9a4dffb640 1 -- 172.21.15.33:0/2333241820 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a400035f0 con 0x55c2fb65b370 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.956+0000 7f9a4dffb640 1 -- 172.21.15.33:0/2333241820 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4003d680 con 0x55c2fb65b370 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.957+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2333241820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb65b370 msgr2=0x55c2fb65d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.957+0000 7f9a57fff640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a440a3240 0x7f9a440a5700 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9a500a4ca0 tx=0x7f9a5006e330 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.957+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/2333241820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb65b370 0x55c2fb65d760 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009900 tx=0x7f9a400056d0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.957+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2333241820 shutdown_connections 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.957+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/2333241820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb65b370 0x55c2fb65d760 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.957+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/2333241820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb650a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.957+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/2333241820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb651b40 0x55c2fb650090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.957+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2333241820 >> 172.21.15.33:0/2333241820 conn(0x55c2fb619310 msgr2=0x55c2fb64cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.957+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2333241820 shutdown_connections 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.957+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2333241820 wait complete. 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.957+0000 7f9a5ff98740 1 -- start start 2024-09-16T00:22:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a5ff98740 1 Processor -- start 2024-09-16T00:22:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a5ff98740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb65ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a5ff98740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb658020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a5ff98740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 0x55c2fb658610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a5ff98740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c2fb65e010 con 0x55c2fb651b40 2024-09-16T00:22:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a5ff98740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c2fb571060 con 0x55c2fb65b370 2024-09-16T00:22:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a5ff98740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c2fb65e190 con 0x55c2fb6505d0 2024-09-16T00:22:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a5ff98740 10 client.0 _ll_register_callbacks cb 0x55c2fb2c48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a577fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb658020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a577fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb658020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36276/0 (socket says 172.21.15.33:36276) 2024-09-16T00:22:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a577fe640 1 -- 172.21.15.33:0/1918153354 learned_addr learned my addr 172.21.15.33:0/1918153354 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a4effd640 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb65ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a4effd640 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 0x55c2fb658610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a577fe640 1 -- 172.21.15.33:0/1918153354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 msgr2=0x55c2fb65ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a577fe640 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb65ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a577fe640 1 -- 172.21.15.33:0/1918153354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 msgr2=0x55c2fb658610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a577fe640 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 0x55c2fb658610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a577fe640 1 -- 172.21.15.33:0/1918153354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2fb3273c0 con 0x55c2fb651b40 2024-09-16T00:22:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.958+0000 7f9a4effd640 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb65ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a577fe640 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb658020 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f9a48012740 tx=0x7f9a48012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4f9cecca54190f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a36ffd640 1 -- 172.21.15.33:0/1918153354 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4800f920 con 0x55c2fb651b40 2024-09-16T00:22:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a36ffd640 1 -- 172.21.15.33:0/1918153354 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9a4800fac0 con 0x55c2fb651b40 2024-09-16T00:22:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1918153354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a400035f0 con 0x55c2fb651b40 2024-09-16T00:22:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a36ffd640 1 -- 172.21.15.33:0/1918153354 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9a4800fcc0 con 0x55c2fb651b40 2024-09-16T00:22:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a5ff98740 10 client.16416 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1918153354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c2fb669b10 con 0x55c2fb651b40 2024-09-16T00:22:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a36ffd640 1 -- 172.21.15.33:0/1918153354 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a400035f0 con 0x55c2fb651b40 2024-09-16T00:22:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a36ffd640 1 -- 172.21.15.33:0/1918153354 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f9a4801c420 con 0x55c2fb651b40 2024-09-16T00:22:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a5ff98740 10 client.16416 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a5ff98740 10 client.16416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a5ff98740 20 client.16416 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c2fb66b070 0x55c2fb66d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a5ff98740 4 client.16416 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.959+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1918153354 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2fb5c3860 con 0x55c2fb66b070 2024-09-16T00:22:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.960+0000 7f9a4effd640 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c2fb66b070 0x55c2fb66d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.960+0000 7f9a36ffd640 10 client.16416 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.960+0000 7f9a4effd640 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c2fb66b070 0x55c2fb66d490 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3ef9c5040437898 server_cookie=f161906eb320152c in_seq=0 out_seq=0 2024-09-16T00:22:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a36ffd640 1 -- 172.21.15.33:0/1918153354 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c2fb5c3860 con 0x55c2fb66b070 2024-09-16T00:22:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a36ffd640 10 client.16416 handle_command_reply: tid=0 2024-09-16T00:22:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a5ff98740 1 client.16416 shutdown 2024-09-16T00:22:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1918153354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 msgr2=0x55c2fb658020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb658020 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f9a48012740 tx=0x7f9a48012c00 comp rx=0 tx=0).stop 2024-09-16T00:22:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1918153354 shutdown_connections 2024-09-16T00:22:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c2fb66b070 0x55c2fb66d490 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 0x55c2fb658610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb658020 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/1918153354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb65ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a36ffd640 0 client.16416 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1918153354 >> 172.21.15.33:0/1918153354 conn(0x55c2fb619310 msgr2=0x55c2fb652960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1918153354 shutdown_connections 2024-09-16T00:22:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1918153354 wait complete. 2024-09-16T00:22:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.961+0000 7f9a5ff98740 20 client.16416 trim_cache size 0 max 16384 2024-09-16T00:22:49.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.974+0000 7f9a5ff98740 1 Processor -- start 2024-09-16T00:22:49.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.974+0000 7f9a5ff98740 1 -- start start 2024-09-16T00:22:49.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a5ff98740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb64cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a5ff98740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb6672b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a5ff98740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 0x55c2fb657020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a5ff98740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581b18d0 con 0x55c2fb651b40 2024-09-16T00:22:49.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a5ff98740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581af1a0 con 0x55c2fb65b370 2024-09-16T00:22:49.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a5ff98740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581acff0 con 0x55c2fb6505d0 2024-09-16T00:22:49.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a577fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb6672b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a577fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb6672b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36292/0 (socket says 172.21.15.33:36292) 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a577fe640 1 -- 172.21.15.33:0/1625634572 learned_addr learned my addr 172.21.15.33:0/1625634572 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a4effd640 1 --2- 172.21.15.33:0/1625634572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 0x55c2fb657020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a577fe640 1 -- 172.21.15.33:0/1625634572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 msgr2=0x55c2fb64cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a577fe640 1 --2- 172.21.15.33:0/1625634572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb64cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a577fe640 1 -- 172.21.15.33:0/1625634572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 msgr2=0x55c2fb657020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a577fe640 1 --2- 172.21.15.33:0/1625634572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 0x55c2fb657020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a577fe640 1 -- 172.21.15.33:0/1625634572 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2fb64d960 con 0x55c2fb651b40 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a577fe640 1 --2- 172.21.15.33:0/1625634572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb6672b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f9a4801c910 tx=0x7f9a480096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586a10ce5543e87b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a4d7fa640 1 -- 172.21.15.33:0/1625634572 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a48009de0 con 0x55c2fb651b40 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a4d7fa640 1 -- 172.21.15.33:0/1625634572 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a48012e50 con 0x55c2fb651b40 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.975+0000 7f9a4d7fa640 1 -- 172.21.15.33:0/1625634572 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a48005430 con 0x55c2fb651b40 2024-09-16T00:22:49.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.976+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1625634572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 msgr2=0x55c2fb6672b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.976+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/1625634572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb6672b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f9a4801c910 tx=0x7f9a480096c0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.976+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1625634572 shutdown_connections 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.976+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/1625634572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 0x55c2fb657020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.976+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/1625634572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb6672b0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.976+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/1625634572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb64cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.976+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1625634572 >> 172.21.15.33:0/1625634572 conn(0x55c2fb619310 msgr2=0x55c2fb656910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.976+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1625634572 shutdown_connections 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.976+0000 7f9a5ff98740 1 -- 172.21.15.33:0/1625634572 wait complete. 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.976+0000 7f9a5ff98740 1 -- start start 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a5ff98740 1 Processor -- start 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a5ff98740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb663910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a5ff98740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb654b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a5ff98740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 0x55c2fb655050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a5ff98740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581ad660 con 0x55c2fb651b40 2024-09-16T00:22:49.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a5ff98740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581aead0 con 0x55c2fb65b370 2024-09-16T00:22:49.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a5ff98740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581add30 con 0x55c2fb6505d0 2024-09-16T00:22:49.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a5ff98740 10 client.0 _ll_register_callbacks cb 0x7f9a48009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:49.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a577fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb654b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a577fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb654b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36302/0 (socket says 172.21.15.33:36302) 2024-09-16T00:22:49.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a577fe640 1 -- 172.21.15.33:0/2653831081 learned_addr learned my addr 172.21.15.33:0/2653831081 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:49.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a4effd640 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb663910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a577fe640 1 -- 172.21.15.33:0/2653831081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 msgr2=0x55c2fb663910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:49.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a577fe640 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb663910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a577fe640 1 -- 172.21.15.33:0/2653831081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 msgr2=0x55c2fb655050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:49.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a577fe640 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 0x55c2fb655050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a577fe640 1 -- 172.21.15.33:0/2653831081 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2fb653120 con 0x55c2fb651b40 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.977+0000 7f9a4effd640 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb663910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a577fe640 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb654b10 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9a4801c890 tx=0x7f9a4801c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6d9fa82663ce68 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a36ffd640 1 -- 172.21.15.33:0/2653831081 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4801cc90 con 0x55c2fb651b40 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a36ffd640 1 -- 172.21.15.33:0/2653831081 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9a480051b0 con 0x55c2fb651b40 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2653831081 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a48012e50 con 0x55c2fb651b40 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a36ffd640 1 -- 172.21.15.33:0/2653831081 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9a4803b050 con 0x55c2fb651b40 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a5ff98740 10 client.16422 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2653831081 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c2fb6d5970 con 0x55c2fb651b40 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a36ffd640 1 -- 172.21.15.33:0/2653831081 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a48012e50 con 0x55c2fb651b40 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a36ffd640 1 -- 172.21.15.33:0/2653831081 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f9a48029030 con 0x55c2fb651b40 2024-09-16T00:22:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a5ff98740 10 client.16422 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a5ff98740 10 client.16422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a5ff98740 20 client.16422 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c2fb6d5f20 0x55c2fb6d8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a5ff98740 4 client.16422 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.978+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2653831081 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c2fb5bdd70 con 0x55c2fb6d5f20 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.979+0000 7f9a4effd640 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c2fb6d5f20 0x55c2fb6d8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.979+0000 7f9a4effd640 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c2fb6d5f20 0x55c2fb6d8310 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47f47e72e962b6a6 server_cookie=89bdd0e3f2c5a87 in_seq=0 out_seq=0 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.979+0000 7f9a36ffd640 10 client.16422 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.979+0000 7f9a36ffd640 1 -- 172.21.15.33:0/2653831081 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c2fb5bdd70 con 0x55c2fb6d5f20 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.979+0000 7f9a36ffd640 10 client.16422 handle_command_reply: tid=0 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.979+0000 7f9a5ff98740 1 client.16422 shutdown 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.980+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2653831081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 msgr2=0x55c2fb654b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.980+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb654b10 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9a4801c890 tx=0x7f9a4801c650 comp rx=0 tx=0).stop 2024-09-16T00:22:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.980+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2653831081 shutdown_connections 2024-09-16T00:22:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.980+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c2fb6d5f20 0x55c2fb6d8310 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.980+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2fb65b370 0x55c2fb655050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.980+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2fb651b40 0x55c2fb654b10 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.980+0000 7f9a5ff98740 1 --2- 172.21.15.33:0/2653831081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2fb6505d0 0x55c2fb663910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.980+0000 7f9a36ffd640 0 client.16422 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.980+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2653831081 >> 172.21.15.33:0/2653831081 conn(0x55c2fb619310 msgr2=0x55c2fb659880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.980+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2653831081 shutdown_connections 2024-09-16T00:22:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.980+0000 7f9a5ff98740 1 -- 172.21.15.33:0/2653831081 wait complete. 2024-09-16T00:22:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.980+0000 7f9a5ff98740 20 client.16422 trim_cache size 0 max 16384 2024-09-16T00:22:49.993 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:22:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.981+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/2157318221 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a440a3240 msgr2=0x7f9a440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.981+0000 7f9a5dbbf640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a440a3240 0x7f9a440a5700 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9a500a4ca0 tx=0x7f9a5006e330 comp rx=0 tx=0).stop 2024-09-16T00:22:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.981+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/2157318221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a58106f10 msgr2=0x7f9a5819cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.981+0000 7f9a5dbbf640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a58106f10 0x7f9a5819cc80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000cce0 tx=0x7f9a5000a620 comp rx=0 tx=0).stop 2024-09-16T00:22:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.981+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/2157318221 shutdown_connections 2024-09-16T00:22:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.981+0000 7f9a5dbbf640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a440a3240 0x7f9a440a5700 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.981+0000 7f9a5dbbf640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a58106f10 0x7f9a5819cc80 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.981+0000 7f9a5dbbf640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9a58102790 0x7f9a581988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.981+0000 7f9a5dbbf640 1 --2- 172.21.15.33:0/2157318221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a58101de0 0x7f9a581983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.981+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/2157318221 >> 172.21.15.33:0/2157318221 conn(0x7f9a580fd8c0 msgr2=0x7f9a580fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.981+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/2157318221 shutdown_connections 2024-09-16T00:22:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:49.981+0000 7f9a5dbbf640 1 -- 172.21.15.33:0/2157318221 wait complete. 2024-09-16T00:22:50.113 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:22:50.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 10.00 MB/sec execute 118 sec latency 23230.404 ms 2024-09-16T00:22:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:51.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:50 smithi033 ceph-mon[31243]: pgmap v521: 129 pgs: 129 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 2.4 MiB/s wr, 85 op/s 2024-09-16T00:22:51.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:50 smithi083 ceph-mon[35242]: pgmap v521: 129 pgs: 129 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 2.4 MiB/s wr, 85 op/s 2024-09-16T00:22:51.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:50 smithi060 ceph-mon[35185]: pgmap v521: 129 pgs: 129 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 2.4 MiB/s wr, 85 op/s 2024-09-16T00:22:51.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.404+0000 7fe682dbe640 1 -- 172.21.15.33:0/3928883465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe67c10c1b0 msgr2=0x7fe67c10e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:51.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.404+0000 7fe682dbe640 1 --2- 172.21.15.33:0/3928883465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe67c10c1b0 0x7fe67c10e620 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe674009ca0 tx=0x7fe67402a950 comp rx=0 tx=0).stop 2024-09-16T00:22:51.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.404+0000 7fe682dbe640 1 -- 172.21.15.33:0/3928883465 shutdown_connections 2024-09-16T00:22:51.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.404+0000 7fe682dbe640 1 --2- 172.21.15.33:0/3928883465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe67c10c1b0 0x7fe67c10e620 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:51.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.404+0000 7fe682dbe640 1 --2- 172.21.15.33:0/3928883465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe67c103730 0x7fe67c10bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:51.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.404+0000 7fe682dbe640 1 --2- 172.21.15.33:0/3928883465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe67c102e10 0x7fe67c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:51.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.404+0000 7fe682dbe640 1 -- 172.21.15.33:0/3928883465 >> 172.21.15.33:0/3928883465 conn(0x7fe67c0fc7d0 msgr2=0x7fe67c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:51.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.405+0000 7fe682dbe640 1 -- 172.21.15.33:0/3928883465 shutdown_connections 2024-09-16T00:22:51.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.405+0000 7fe682dbe640 1 -- 172.21.15.33:0/3928883465 wait complete. 2024-09-16T00:22:51.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.405+0000 7fe682dbe640 1 Processor -- start 2024-09-16T00:22:51.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.405+0000 7fe682dbe640 1 -- start start 2024-09-16T00:22:51.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.405+0000 7fe682dbe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe67c102e10 0x7fe67c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:51.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.405+0000 7fe682dbe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe67c103730 0x7fe67c19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:51.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.405+0000 7fe682dbe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe67c10c1b0 0x7fe67c19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:51.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.405+0000 7fe682dbe640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c111f80 con 0x7fe67c10c1b0 2024-09-16T00:22:51.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.405+0000 7fe682dbe640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c111e00 con 0x7fe67c103730 2024-09-16T00:22:51.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.405+0000 7fe682dbe640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c112100 con 0x7fe67c102e10 2024-09-16T00:22:51.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe68135b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe67c10c1b0 0x7fe67c19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:51.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe68135b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe67c10c1b0 0x7fe67c19eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43138/0 (socket says 172.21.15.33:43138) 2024-09-16T00:22:51.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe68135b640 1 -- 172.21.15.33:0/844435612 learned_addr learned my addr 172.21.15.33:0/844435612 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:51.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe67bfff640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe67c103730 0x7fe67c19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:51.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe680b5a640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe67c102e10 0x7fe67c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:51.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe68135b640 1 -- 172.21.15.33:0/844435612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe67c102e10 msgr2=0x7fe67c19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:51.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe68135b640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe67c102e10 0x7fe67c19a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:51.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe68135b640 1 -- 172.21.15.33:0/844435612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe67c103730 msgr2=0x7fe67c19ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:51.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe68135b640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe67c103730 0x7fe67c19ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:51.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe68135b640 1 -- 172.21.15.33:0/844435612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe67c19f5c0 con 0x7fe67c10c1b0 2024-09-16T00:22:51.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe67bfff640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe67c103730 0x7fe67c19ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:51.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe68135b640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe67c10c1b0 0x7fe67c19eee0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe674004af0 tx=0x7fe67402f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:51.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe679ffb640 1 -- 172.21.15.33:0/844435612 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674034040 con 0x7fe67c10c1b0 2024-09-16T00:22:51.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe682dbe640 1 -- 172.21.15.33:0/844435612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe67c19f850 con 0x7fe67c10c1b0 2024-09-16T00:22:51.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe679ffb640 1 -- 172.21.15.33:0/844435612 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe674030920 con 0x7fe67c10c1b0 2024-09-16T00:22:51.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe679ffb640 1 -- 172.21.15.33:0/844435612 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6740047b0 con 0x7fe67c10c1b0 2024-09-16T00:22:51.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.406+0000 7fe682dbe640 1 -- 172.21.15.33:0/844435612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe67c1a7190 con 0x7fe67c10c1b0 2024-09-16T00:22:51.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.408+0000 7fe682dbe640 1 -- 172.21.15.33:0/844435612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe67c1031f0 con 0x7fe67c10c1b0 2024-09-16T00:22:51.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.409+0000 7fe679ffb640 1 -- 172.21.15.33:0/844435612 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe674037020 con 0x7fe67c10c1b0 2024-09-16T00:22:51.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.409+0000 7fe679ffb640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6640a3240 0x7fe6640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:51.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.410+0000 7fe679ffb640 1 -- 172.21.15.33:0/844435612 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe6740f0eb0 con 0x7fe67c10c1b0 2024-09-16T00:22:51.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.410+0000 7fe680b5a640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6640a3240 0x7fe6640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:51.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.410+0000 7fe680b5a640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6640a3240 0x7fe6640a5700 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe66c009aa0 tx=0x7fe66c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:51.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.413+0000 7fe679ffb640 1 -- 172.21.15.33:0/844435612 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe6740811b0 con 0x7fe67c10c1b0 2024-09-16T00:22:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.623+0000 7fe682dbe640 1 -- 172.21.15.33:0/844435612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe67c19b970 con 0x7fe67c10c1b0 2024-09-16T00:22:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.624+0000 7fe679ffb640 1 -- 172.21.15.33:0/844435612 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7fe67c19b970 con 0x7fe67c10c1b0 2024-09-16T00:22:51.625 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:22:51.625 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:22:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:22:51.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.626+0000 7fe682dbe640 1 -- 172.21.15.33:0/844435612 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6640a3240 msgr2=0x7fe6640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:51.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.626+0000 7fe682dbe640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6640a3240 0x7fe6640a5700 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe66c009aa0 tx=0x7fe66c005f90 comp rx=0 tx=0).stop 2024-09-16T00:22:51.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.626+0000 7fe682dbe640 1 -- 172.21.15.33:0/844435612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe67c10c1b0 msgr2=0x7fe67c19eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:51.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.626+0000 7fe682dbe640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe67c10c1b0 0x7fe67c19eee0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe674004af0 tx=0x7fe67402f5b0 comp rx=0 tx=0).stop 2024-09-16T00:22:51.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.626+0000 7fe682dbe640 1 -- 172.21.15.33:0/844435612 shutdown_connections 2024-09-16T00:22:51.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.626+0000 7fe682dbe640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6640a3240 0x7fe6640a5700 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:51.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.626+0000 7fe682dbe640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe67c10c1b0 0x7fe67c19eee0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:51.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.626+0000 7fe682dbe640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe67c103730 0x7fe67c19ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:51.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.626+0000 7fe682dbe640 1 --2- 172.21.15.33:0/844435612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe67c102e10 0x7fe67c19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:51.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.626+0000 7fe682dbe640 1 -- 172.21.15.33:0/844435612 >> 172.21.15.33:0/844435612 conn(0x7fe67c0fc7d0 msgr2=0x7fe67c10d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:51.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.626+0000 7fe682dbe640 1 -- 172.21.15.33:0/844435612 shutdown_connections 2024-09-16T00:22:51.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:51.626+0000 7fe682dbe640 1 -- 172.21.15.33:0/844435612 wait complete. 2024-09-16T00:22:51.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 9.92 MB/sec execute 119 sec latency 24230.472 ms 2024-09-16T00:22:51.798 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:22:51.798 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:22:52.052 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:51 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/844435612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:52.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:51 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/844435612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:52.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/844435612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:52.444 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 9279b40d --release --await' 2024-09-16T00:22:52.444 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 9279b40d --release --await 2024-09-16T00:22:52.493 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:52.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 9.84 MB/sec execute 120 sec latency 25230.513 ms 2024-09-16T00:22:53.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.002+0000 7f056f1c2640 1 -- 172.21.15.33:0/3391526271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05681027a0 msgr2=0x7f05681067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:53.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.002+0000 7f056f1c2640 1 --2- 172.21.15.33:0/3391526271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05681027a0 0x7f05681067f0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0554009900 tx=0x7f055402a410 comp rx=0 tx=0).stop 2024-09-16T00:22:53.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.003+0000 7f056f1c2640 1 -- 172.21.15.33:0/3391526271 shutdown_connections 2024-09-16T00:22:53.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.003+0000 7f056f1c2640 1 --2- 172.21.15.33:0/3391526271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0568106f20 0x7f056810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.003+0000 7f056f1c2640 1 --2- 172.21.15.33:0/3391526271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05681027a0 0x7f05681067f0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.003+0000 7f056f1c2640 1 --2- 172.21.15.33:0/3391526271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0568101df0 0x7f05681021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.003+0000 7f056f1c2640 1 -- 172.21.15.33:0/3391526271 >> 172.21.15.33:0/3391526271 conn(0x7f05680fd8f0 msgr2=0x7f05680ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:53.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.003+0000 7f056f1c2640 1 -- 172.21.15.33:0/3391526271 shutdown_connections 2024-09-16T00:22:53.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.003+0000 7f056f1c2640 1 -- 172.21.15.33:0/3391526271 wait complete. 2024-09-16T00:22:53.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.003+0000 7f056f1c2640 1 Processor -- start 2024-09-16T00:22:53.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.003+0000 7f056f1c2640 1 -- start start 2024-09-16T00:22:53.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056f1c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0568101df0 0x7f05680744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056f1c2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05681027a0 0x7f0568074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056f1c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0568106f20 0x7f0568074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056f1c2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0568111fd0 con 0x7f0568106f20 2024-09-16T00:22:53.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056f1c2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0568111e50 con 0x7f05681027a0 2024-09-16T00:22:53.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056f1c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0568112150 con 0x7f0568101df0 2024-09-16T00:22:53.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056cf5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0568101df0 0x7f05680744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056d75f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0568106f20 0x7f0568074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056cf5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0568101df0 0x7f05680744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52306/0 (socket says 172.21.15.33:52306) 2024-09-16T00:22:53.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f0567fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05681027a0 0x7f0568074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056d75f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0568106f20 0x7f0568074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43152/0 (socket says 172.21.15.33:43152) 2024-09-16T00:22:53.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056cf5e640 1 -- 172.21.15.33:0/3034747142 learned_addr learned my addr 172.21.15.33:0/3034747142 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:53.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056cf5e640 1 -- 172.21.15.33:0/3034747142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05681027a0 msgr2=0x7f0568074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:53.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056cf5e640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05681027a0 0x7f0568074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056cf5e640 1 -- 172.21.15.33:0/3034747142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0568106f20 msgr2=0x7f0568074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:53.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056cf5e640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0568106f20 0x7f0568074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056cf5e640 1 -- 172.21.15.33:0/3034747142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05681a68f0 con 0x7f0568101df0 2024-09-16T00:22:53.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056d75f640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0568106f20 0x7f0568074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:53.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.004+0000 7f056cf5e640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0568101df0 0x7f05680744f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f055c00d950 tx=0x7f055c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:53.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.005+0000 7f0565ffb640 1 -- 172.21.15.33:0/3034747142 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c017030 con 0x7f0568101df0 2024-09-16T00:22:53.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.005+0000 7f056f1c2640 1 -- 172.21.15.33:0/3034747142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05681a6be0 con 0x7f0568101df0 2024-09-16T00:22:53.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.005+0000 7f0565ffb640 1 -- 172.21.15.33:0/3034747142 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f055c004020 con 0x7f0568101df0 2024-09-16T00:22:53.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.005+0000 7f0565ffb640 1 -- 172.21.15.33:0/3034747142 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0047a0 con 0x7f0568101df0 2024-09-16T00:22:53.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.005+0000 7f056f1c2640 1 -- 172.21.15.33:0/3034747142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05681a7140 con 0x7f0568101df0 2024-09-16T00:22:53.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.007+0000 7f0565ffb640 1 -- 172.21.15.33:0/3034747142 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f055c0041d0 con 0x7f0568101df0 2024-09-16T00:22:53.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.008+0000 7f057159b740 1 Processor -- start 2024-09-16T00:22:53.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.008+0000 7f0565ffb640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f05580a32c0 0x7f05580a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.008+0000 7f057159b740 1 -- start start 2024-09-16T00:22:53.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.008+0000 7f056d75f640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f05580a32c0 0x7f05580a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.008+0000 7f0565ffb640 1 -- 172.21.15.33:0/3034747142 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f055c0cf6b0 con 0x7f0568101df0 2024-09-16T00:22:53.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.008+0000 7f057159b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647377b60 0x55d647377f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.008+0000 7f057159b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d647382800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.008+0000 7f057159b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647382d40 0x55d647385130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.008+0000 7f057159b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d6472dd280 con 0x55d647378480 2024-09-16T00:22:53.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.008+0000 7f057159b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d647284b10 con 0x55d647377b60 2024-09-16T00:22:53.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.008+0000 7f057159b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d6472d9240 con 0x55d647382d40 2024-09-16T00:22:53.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f0567fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d647382800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f0567fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d647382800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43156/0 (socket says 172.21.15.33:43156) 2024-09-16T00:22:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f0567fff640 1 -- 172.21.15.33:0/576510099 learned_addr learned my addr 172.21.15.33:0/576510099 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f056cf5e640 1 --2- 172.21.15.33:0/576510099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647377b60 0x55d647377f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f056d75f640 1 --2- 172.21.15.33:0/576510099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647382d40 0x55d647385130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f056d75f640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f05580a32c0 0x7f05580a5780 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0560009fd0 tx=0x7f0560010a60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f0567fff640 1 -- 172.21.15.33:0/576510099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647382d40 msgr2=0x55d647385130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f0567fff640 1 --2- 172.21.15.33:0/576510099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647382d40 0x55d647385130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f0567fff640 1 -- 172.21.15.33:0/576510099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647377b60 msgr2=0x55d647377f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f0567fff640 1 --2- 172.21.15.33:0/576510099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647377b60 0x55d647377f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f0567fff640 1 -- 172.21.15.33:0/576510099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6472911f0 con 0x55d647378480 2024-09-16T00:22:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f0567fff640 1 --2- 172.21.15.33:0/576510099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d647382800 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0554009900 tx=0x7f05540040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ed32344343d45e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.009+0000 7f0552ffd640 1 -- 172.21.15.33:0/576510099 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0554035040 con 0x55d647378480 2024-09-16T00:22:53.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.010+0000 7f0552ffd640 1 -- 172.21.15.33:0/576510099 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0554005520 con 0x55d647378480 2024-09-16T00:22:53.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.010+0000 7f0552ffd640 1 -- 172.21.15.33:0/576510099 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0554003850 con 0x55d647378480 2024-09-16T00:22:53.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.011+0000 7f057159b740 1 -- 172.21.15.33:0/576510099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 msgr2=0x55d647382800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:53.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.011+0000 7f057159b740 1 --2- 172.21.15.33:0/576510099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d647382800 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0554009900 tx=0x7f05540040f0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.011+0000 7f057159b740 1 -- 172.21.15.33:0/576510099 shutdown_connections 2024-09-16T00:22:53.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.011+0000 7f057159b740 1 --2- 172.21.15.33:0/576510099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647382d40 0x55d647385130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.011+0000 7f057159b740 1 --2- 172.21.15.33:0/576510099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d647382800 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.011+0000 7f057159b740 1 --2- 172.21.15.33:0/576510099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647377b60 0x55d647377f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.011+0000 7f057159b740 1 -- 172.21.15.33:0/576510099 >> 172.21.15.33:0/576510099 conn(0x55d6473740f0 msgr2=0x55d6473744d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:53.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.011+0000 7f057159b740 1 -- 172.21.15.33:0/576510099 shutdown_connections 2024-09-16T00:22:53.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.011+0000 7f057159b740 1 -- 172.21.15.33:0/576510099 wait complete. 2024-09-16T00:22:53.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.011+0000 7f057159b740 1 -- start start 2024-09-16T00:22:53.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.011+0000 7f057159b740 1 Processor -- start 2024-09-16T00:22:53.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.012+0000 7f057159b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647377b60 0x55d64737fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.012+0000 7f057159b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d647380150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.012+0000 7f057159b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647382d40 0x55d647380690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.012+0000 7f0567fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d647380150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.012+0000 7f0567fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d647380150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43162/0 (socket says 172.21.15.33:43162) 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.012+0000 7f056cf5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647377b60 0x55d64737fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.012+0000 7f0567fff640 1 -- 172.21.15.33:0/1127667271 learned_addr learned my addr 172.21.15.33:0/1127667271 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.012+0000 7f057159b740 1 -- 172.21.15.33:0/1127667271 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d647385910 con 0x55d647378480 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.012+0000 7f057159b740 1 -- 172.21.15.33:0/1127667271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d647297f10 con 0x55d647382d40 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.012+0000 7f057159b740 1 -- 172.21.15.33:0/1127667271 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d647385a90 con 0x55d647377b60 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f056d75f640 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647382d40 0x55d647380690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f0567fff640 1 -- 172.21.15.33:0/1127667271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647377b60 msgr2=0x55d64737fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f0567fff640 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647377b60 0x55d64737fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f0567fff640 1 -- 172.21.15.33:0/1127667271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647382d40 msgr2=0x55d647380690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:53.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f0567fff640 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647382d40 0x55d647380690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f0567fff640 1 -- 172.21.15.33:0/1127667271 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d64704e6a0 con 0x55d647378480 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f056d75f640 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647382d40 0x55d647380690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f056cf5e640 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647377b60 0x55d64737fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f057159b740 10 client.0 _ll_register_callbacks cb 0x55d647260a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f057159b740 1 -- 172.21.15.33:0/1127667271 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0554005520 con 0x55d647378480 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f0567fff640 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d647380150 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0554004950 tx=0x7f055402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf4796056c78ff7c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f0547fff640 10 client.16440 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f0547fff640 1 -- 172.21.15.33:0/1127667271 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0554035070 con 0x55d647378480 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.013+0000 7f0547fff640 1 -- 172.21.15.33:0/1127667271 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0554004f90 con 0x55d647378480 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.014+0000 7f0547fff640 1 -- 172.21.15.33:0/1127667271 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0554005230 con 0x55d647378480 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.014+0000 7f057159b740 10 client.16440 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.014+0000 7f057159b740 1 -- 172.21.15.33:0/1127667271 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55d6473912d0 con 0x55d647378480 2024-09-16T00:22:53.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.014+0000 7f0547fff640 1 -- 172.21.15.33:0/1127667271 <== mon.0 v2:172.21.15.33:3300/0 4 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f05540035f0 con 0x55d647378480 2024-09-16T00:22:53.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.014+0000 7f057159b740 10 client.16440 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:53.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.014+0000 7f057159b740 10 client.16440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:53.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.014+0000 7f057159b740 20 client.16440 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:53.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.014+0000 7f057159b740 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d647392620 0x55d647394a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.014+0000 7f057159b740 4 client.16440 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:53.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.014+0000 7f057159b740 1 -- 172.21.15.33:0/1127667271 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6472ea7f0 con 0x55d647392620 2024-09-16T00:22:53.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.014+0000 7f056cf5e640 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d647392620 0x55d647394a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.015+0000 7f056cf5e640 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d647392620 0x55d647394a10 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5531d91d2283f75e server_cookie=8082df871e390163 in_seq=0 out_seq=0 2024-09-16T00:22:53.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.015+0000 7f0547fff640 10 client.16440 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:53.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.016+0000 7f0547fff640 1 -- 172.21.15.33:0/1127667271 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d6472ea7f0 con 0x55d647392620 2024-09-16T00:22:53.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.016+0000 7f0547fff640 10 client.16440 handle_command_reply: tid=0 2024-09-16T00:22:53.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.016+0000 7f057159b740 1 client.16440 shutdown 2024-09-16T00:22:53.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.016+0000 7f057159b740 1 -- 172.21.15.33:0/1127667271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 msgr2=0x55d647380150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:53.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.016+0000 7f057159b740 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d647380150 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0554004950 tx=0x7f055402a410 comp rx=0 tx=0).stop 2024-09-16T00:22:53.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.016+0000 7f057159b740 1 -- 172.21.15.33:0/1127667271 shutdown_connections 2024-09-16T00:22:53.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.016+0000 7f057159b740 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d647392620 0x55d647394a10 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.016+0000 7f0547fff640 0 client.16440 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:53.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.016+0000 7f057159b740 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647382d40 0x55d647380690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.016+0000 7f057159b740 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d647380150 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.016+0000 7f057159b740 1 --2- 172.21.15.33:0/1127667271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647377b60 0x55d64737fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.016+0000 7f057159b740 1 -- 172.21.15.33:0/1127667271 >> 172.21.15.33:0/1127667271 conn(0x55d6473740f0 msgr2=0x55d647373270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:53.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.017+0000 7f057159b740 1 -- 172.21.15.33:0/1127667271 shutdown_connections 2024-09-16T00:22:53.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.017+0000 7f057159b740 1 -- 172.21.15.33:0/1127667271 wait complete. 2024-09-16T00:22:53.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.017+0000 7f057159b740 20 client.16440 trim_cache size 0 max 16384 2024-09-16T00:22:53.032 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:53.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f057159b740 1 Processor -- start 2024-09-16T00:22:53.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f057159b740 1 -- start start 2024-09-16T00:22:53.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f057159b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647377b60 0x55d6473804e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f057159b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647378480 0x55d647399fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f057159b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647382d40 0x55d64739a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f057159b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f05681b3b40 con 0x55d647377b60 2024-09-16T00:22:53.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f057159b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f05681b1410 con 0x55d647382d40 2024-09-16T00:22:53.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f057159b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f05681af260 con 0x55d647378480 2024-09-16T00:22:53.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f056cf5e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647377b60 0x55d6473804e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f056cf5e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647377b60 0x55d6473804e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43170/0 (socket says 172.21.15.33:43170) 2024-09-16T00:22:53.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f056cf5e640 1 -- 172.21.15.33:0/261001321 learned_addr learned my addr 172.21.15.33:0/261001321 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:53.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f056d75f640 1 --2- 172.21.15.33:0/261001321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647382d40 0x55d64739a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.037+0000 7f0567fff640 1 --2- 172.21.15.33:0/261001321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647378480 0x55d647399fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.040 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:52 smithi033 ceph-mon[31243]: pgmap v522: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.3 MiB/s wr, 87 op/s 2024-09-16T00:22:53.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.038+0000 7f056cf5e640 1 -- 172.21.15.33:0/261001321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647378480 msgr2=0x55d647399fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:53.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.038+0000 7f056cf5e640 1 --2- 172.21.15.33:0/261001321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647378480 0x55d647399fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.038+0000 7f056cf5e640 1 -- 172.21.15.33:0/261001321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647382d40 msgr2=0x55d64739a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:53.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.038+0000 7f056cf5e640 1 --2- 172.21.15.33:0/261001321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647382d40 0x55d64739a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.038+0000 7f056cf5e640 1 -- 172.21.15.33:0/261001321 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6473824e0 con 0x55d647377b60 2024-09-16T00:22:53.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.038+0000 7f0567fff640 1 --2- 172.21.15.33:0/261001321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647378480 0x55d647399fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:53.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.038+0000 7f056cf5e640 1 --2- 172.21.15.33:0/261001321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647377b60 0x55d6473804e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f055c0981c0 tx=0x7f055c01e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a873774bc7ee4c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:53.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.038+0000 7f05527fc640 1 -- 172.21.15.33:0/261001321 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0b9070 con 0x55d647377b60 2024-09-16T00:22:53.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.038+0000 7f05527fc640 1 -- 172.21.15.33:0/261001321 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f055c09fba0 con 0x55d647377b60 2024-09-16T00:22:53.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.038+0000 7f05527fc640 1 -- 172.21.15.33:0/261001321 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0a3fd0 con 0x55d647377b60 2024-09-16T00:22:53.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.039+0000 7f057159b740 1 -- 172.21.15.33:0/261001321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647377b60 msgr2=0x55d6473804e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:53.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.039+0000 7f057159b740 1 --2- 172.21.15.33:0/261001321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647377b60 0x55d6473804e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f055c0981c0 tx=0x7f055c01e040 comp rx=0 tx=0).stop 2024-09-16T00:22:53.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.039+0000 7f057159b740 1 -- 172.21.15.33:0/261001321 shutdown_connections 2024-09-16T00:22:53.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.039+0000 7f057159b740 1 --2- 172.21.15.33:0/261001321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647382d40 0x55d64739a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.039+0000 7f057159b740 1 --2- 172.21.15.33:0/261001321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647378480 0x55d647399fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.039+0000 7f057159b740 1 --2- 172.21.15.33:0/261001321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647377b60 0x55d6473804e0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.039+0000 7f057159b740 1 -- 172.21.15.33:0/261001321 >> 172.21.15.33:0/261001321 conn(0x55d6473740f0 msgr2=0x55d647381b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:53.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.039+0000 7f057159b740 1 -- 172.21.15.33:0/261001321 shutdown_connections 2024-09-16T00:22:53.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.039+0000 7f057159b740 1 -- 172.21.15.33:0/261001321 wait complete. 2024-09-16T00:22:53.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.040+0000 7f057159b740 1 -- start start 2024-09-16T00:22:53.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.040+0000 7f057159b740 1 Processor -- start 2024-09-16T00:22:53.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.040+0000 7f057159b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647377b60 0x55d647387360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.040+0000 7f057159b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d6473878a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.040+0000 7f057159b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647382d40 0x55d6473f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f057159b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f05681af8d0 con 0x55d647378480 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f057159b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f05681b0d40 con 0x55d647377b60 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f057159b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f05681affa0 con 0x55d647382d40 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f057159b740 10 client.0 _ll_register_callbacks cb 0x7f055c004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f056cf5e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647377b60 0x55d647387360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f056d75f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647382d40 0x55d6473f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f056cf5e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647377b60 0x55d647387360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:49910/0 (socket says 172.21.15.33:49910) 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f056cf5e640 1 -- 172.21.15.33:0/1436794949 learned_addr learned my addr 172.21.15.33:0/1436794949 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f056cf5e640 1 -- 172.21.15.33:0/1436794949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647382d40 msgr2=0x55d6473f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f056cf5e640 1 --2- 172.21.15.33:0/1436794949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647382d40 0x55d6473f5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f056cf5e640 1 -- 172.21.15.33:0/1436794949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 msgr2=0x55d6473878a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f056cf5e640 1 --2- 172.21.15.33:0/1436794949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d6473878a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f056cf5e640 1 -- 172.21.15.33:0/1436794949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6473f6600 con 0x55d647377b60 2024-09-16T00:22:53.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f056d75f640 1 --2- 172.21.15.33:0/1436794949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647382d40 0x55d6473f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:53.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f056cf5e640 1 --2- 172.21.15.33:0/1436794949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647377b60 0x55d647387360 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f055c005fd0 tx=0x7f055c09b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe6c8ed8187deb32 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:53.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f0547fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:22:53.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.041+0000 7f0547fff640 1 -- 172.21.15.33:0/1436794949 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c07f060 con 0x55d647377b60 2024-09-16T00:22:53.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.042+0000 7f0547fff640 1 -- 172.21.15.33:0/1436794949 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f055c0ccde0 con 0x55d647377b60 2024-09-16T00:22:53.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.042+0000 7f057159b740 1 -- 172.21.15.33:0/1436794949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f055c09fba0 con 0x55d647377b60 2024-09-16T00:22:53.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.042+0000 7f0547fff640 1 -- 172.21.15.33:0/1436794949 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f055c0b9030 con 0x55d647377b60 2024-09-16T00:22:53.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.042+0000 7f057159b740 10 client.25973 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:53.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.042+0000 7f057159b740 1 -- 172.21.15.33:0/1436794949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d6473fd9f0 con 0x55d647377b60 2024-09-16T00:22:53.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.042+0000 7f0547fff640 1 -- 172.21.15.33:0/1436794949 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f055c09fba0 con 0x55d647377b60 2024-09-16T00:22:53.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.042+0000 7f0547fff640 1 -- 172.21.15.33:0/1436794949 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f055c09ac40 con 0x55d647377b60 2024-09-16T00:22:53.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.042+0000 7f057159b740 10 client.25973 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:53.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.042+0000 7f057159b740 10 client.25973 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:53.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.043+0000 7f057159b740 20 client.25973 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:53.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.043+0000 7f057159b740 1 --2- 172.21.15.33:0/1436794949 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d647392620 0x55d647400080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:53.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.043+0000 7f057159b740 4 client.25973 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:22:53.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.043+0000 7f057159b740 1 -- 172.21.15.33:0/1436794949 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d6472e4d00 con 0x55d647392620 2024-09-16T00:22:53.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.043+0000 7f0567fff640 1 --2- 172.21.15.33:0/1436794949 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d647392620 0x55d647400080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.043+0000 7f0567fff640 1 --2- 172.21.15.33:0/1436794949 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d647392620 0x55d647400080 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddcc35831c6d3683 server_cookie=a83b76e90f0c541b in_seq=0 out_seq=0 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.043+0000 7f0547fff640 10 client.25973 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.044+0000 7f0547fff640 1 -- 172.21.15.33:0/1436794949 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d6472e4d00 con 0x55d647392620 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.044+0000 7f0547fff640 10 client.25973 handle_command_reply: tid=0 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.044+0000 7f057159b740 1 client.25973 shutdown 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.044+0000 7f057159b740 1 -- 172.21.15.33:0/1436794949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647377b60 msgr2=0x55d647387360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.044+0000 7f057159b740 1 --2- 172.21.15.33:0/1436794949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647377b60 0x55d647387360 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f055c005fd0 tx=0x7f055c09b460 comp rx=0 tx=0).stop 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.044+0000 7f057159b740 1 -- 172.21.15.33:0/1436794949 shutdown_connections 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.044+0000 7f057159b740 1 --2- 172.21.15.33:0/1436794949 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d647392620 0x55d647400080 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.044+0000 7f057159b740 1 --2- 172.21.15.33:0/1436794949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d647382d40 0x55d6473f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.044+0000 7f057159b740 1 --2- 172.21.15.33:0/1436794949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d647378480 0x55d6473878a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.044+0000 7f057159b740 1 --2- 172.21.15.33:0/1436794949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d647377b60 0x55d647387360 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.045+0000 7f057159b740 1 -- 172.21.15.33:0/1436794949 >> 172.21.15.33:0/1436794949 conn(0x55d6473740f0 msgr2=0x55d6473795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:53.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.045+0000 7f0547fff640 0 client.25973 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:53.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.045+0000 7f057159b740 1 -- 172.21.15.33:0/1436794949 shutdown_connections 2024-09-16T00:22:53.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.045+0000 7f057159b740 1 -- 172.21.15.33:0/1436794949 wait complete. 2024-09-16T00:22:53.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.045+0000 7f057159b740 20 client.25973 trim_cache size 0 max 16384 2024-09-16T00:22:53.050 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:22:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:22:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "62cb8df5-8435-4d2d-9546-aeef8bf67fb3", 2024-09-16T00:22:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:22:53.050 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:22:53.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.046+0000 7f056f1c2640 1 -- 172.21.15.33:0/3034747142 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f05580a32c0 msgr2=0x7f05580a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:53.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.046+0000 7f056f1c2640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f05580a32c0 0x7f05580a5780 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0560009fd0 tx=0x7f0560010a60 comp rx=0 tx=0).stop 2024-09-16T00:22:53.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.046+0000 7f056f1c2640 1 -- 172.21.15.33:0/3034747142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0568101df0 msgr2=0x7f05680744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:53.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.046+0000 7f056f1c2640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0568101df0 0x7f05680744f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f055c00d950 tx=0x7f055c00de10 comp rx=0 tx=0).stop 2024-09-16T00:22:53.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.046+0000 7f056f1c2640 1 -- 172.21.15.33:0/3034747142 shutdown_connections 2024-09-16T00:22:53.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.046+0000 7f056f1c2640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f05580a32c0 0x7f05580a5780 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.046+0000 7f056f1c2640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0568106f20 0x7f0568074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.046+0000 7f056f1c2640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05681027a0 0x7f0568074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.046+0000 7f056f1c2640 1 --2- 172.21.15.33:0/3034747142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0568101df0 0x7f05680744f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:53.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.046+0000 7f056f1c2640 1 -- 172.21.15.33:0/3034747142 >> 172.21.15.33:0/3034747142 conn(0x7f05680fd8f0 msgr2=0x7f05680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:53.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.046+0000 7f056f1c2640 1 -- 172.21.15.33:0/3034747142 shutdown_connections 2024-09-16T00:22:53.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:53.046+0000 7f056f1c2640 1 -- 172.21.15.33:0/3034747142 wait complete. 2024-09-16T00:22:53.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:52 smithi083 ceph-mon[35242]: pgmap v522: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.3 MiB/s wr, 87 op/s 2024-09-16T00:22:53.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:52 smithi060 ceph-mon[35185]: pgmap v522: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.3 MiB/s wr, 87 op/s 2024-09-16T00:22:53.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 337644 9.76 MB/sec execute 121 sec latency 26230.615 ms 2024-09-16T00:22:54.018 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:22:54.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:53 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:22:54.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:53 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:54.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:53 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:22:54.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:53 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:54.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:53 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:22:54.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:53 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:54.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:53 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:22:54.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:53 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:54.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:53 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:22:54.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:53 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:54.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:53 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:22:54.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:53 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:54.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.224+0000 7f6f7836f640 1 -- 172.21.15.33:0/341174853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f70100e20 msgr2=0x7f6f70101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:54.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.224+0000 7f6f7836f640 1 --2- 172.21.15.33:0/341174853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f70100e20 0x7f6f70101200 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c009900 tx=0x7f6f6c02a480 comp rx=0 tx=0).stop 2024-09-16T00:22:54.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.224+0000 7f6f7836f640 1 -- 172.21.15.33:0/341174853 shutdown_connections 2024-09-16T00:22:54.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.224+0000 7f6f7836f640 1 --2- 172.21.15.33:0/341174853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f70102ef0 0x7f6f7010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.224+0000 7f6f7836f640 1 --2- 172.21.15.33:0/341174853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f70102570 0x7f6f701029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.224+0000 7f6f7836f640 1 --2- 172.21.15.33:0/341174853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f70100e20 0x7f6f70101200 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.224+0000 7f6f7836f640 1 -- 172.21.15.33:0/341174853 >> 172.21.15.33:0/341174853 conn(0x7f6f700fc7d0 msgr2=0x7f6f700febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.224+0000 7f6f7836f640 1 -- 172.21.15.33:0/341174853 shutdown_connections 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.224+0000 7f6f7836f640 1 -- 172.21.15.33:0/341174853 wait complete. 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7836f640 1 Processor -- start 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7836f640 1 -- start start 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7836f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f70100e20 0x7f6f7010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7836f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f70102570 0x7f6f7010ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7836f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f70102ef0 0x7f6f7010ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7836f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701120a0 con 0x7f6f70102ef0 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7836f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6f70111f20 con 0x7f6f70100e20 2024-09-16T00:22:54.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7836f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f70112220 con 0x7f6f70102570 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7690c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f70102ef0 0x7f6f7010ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7690c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f70102ef0 0x7f6f7010ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43186/0 (socket says 172.21.15.33:43186) 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7690c640 1 -- 172.21.15.33:0/4231193004 learned_addr learned my addr 172.21.15.33:0/4231193004 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7610b640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f70100e20 0x7f6f7010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.225+0000 7f6f7590a640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f70102570 0x7f6f7010ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.226+0000 7f6f7690c640 1 -- 172.21.15.33:0/4231193004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f70102570 msgr2=0x7f6f7010ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.226+0000 7f6f7690c640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f70102570 0x7f6f7010ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.226+0000 7f6f7690c640 1 -- 172.21.15.33:0/4231193004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f70100e20 msgr2=0x7f6f7010e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.226+0000 7f6f7690c640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f70100e20 0x7f6f7010e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.226+0000 7f6f7690c640 1 -- 172.21.15.33:0/4231193004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f701a69a0 con 0x7f6f70102ef0 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.226+0000 7f6f7690c640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f70102ef0 0x7f6f7010ef50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800eb10 tx=0x7f6f6800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.226+0000 7f6f677fe640 1 -- 172.21.15.33:0/4231193004 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f68004b20 con 0x7f6f70102ef0 2024-09-16T00:22:54.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.226+0000 7f6f677fe640 1 -- 172.21.15.33:0/4231193004 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f68004cc0 con 0x7f6f70102ef0 2024-09-16T00:22:54.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.226+0000 7f6f7836f640 1 -- 172.21.15.33:0/4231193004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f701a6c90 con 0x7f6f70102ef0 2024-09-16T00:22:54.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.226+0000 7f6f677fe640 1 -- 172.21.15.33:0/4231193004 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f68010760 con 0x7f6f70102ef0 2024-09-16T00:22:54.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.226+0000 7f6f7836f640 1 -- 172.21.15.33:0/4231193004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f701a71d0 con 0x7f6f70102ef0 2024-09-16T00:22:54.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.229+0000 7f6f677fe640 1 -- 172.21.15.33:0/4231193004 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6f68005020 con 0x7f6f70102ef0 2024-09-16T00:22:54.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.229+0000 7f6f7a748740 1 Processor -- start 2024-09-16T00:22:54.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.229+0000 7f6f7a748740 1 -- start start 2024-09-16T00:22:54.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7a748740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 0x5633e1020440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7a748740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e1020980 0x5633e102ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7a748740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e102b240 0x5633e102d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7a748740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5633e0f85620 con 0x5633e1020060 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7a748740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5633e0f2ce50 con 0x5633e102b240 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7a748740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5633e0f815e0 con 0x5633e1020980 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7590a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 0x5633e1020440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7590a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 0x5633e1020440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43192/0 (socket says 172.21.15.33:43192) 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7590a640 1 -- 172.21.15.33:0/4241471828 learned_addr learned my addr 172.21.15.33:0/4241471828 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7610b640 1 --2- 172.21.15.33:0/4241471828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e1020980 0x5633e102ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7590a640 1 --2- 172.21.15.33:0/4241471828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e102b240 0x5633e102d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f677fe640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6f5c0a3240 0x7f6f5c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7610b640 1 -- 172.21.15.33:0/4241471828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e102b240 msgr2=0x5633e102d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7610b640 1 --2- 172.21.15.33:0/4241471828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e102b240 0x5633e102d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7610b640 1 -- 172.21.15.33:0/4241471828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 msgr2=0x5633e1020440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:54.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7610b640 1 --2- 172.21.15.33:0/4241471828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 0x5633e1020440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7610b640 1 -- 172.21.15.33:0/4241471828 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633e0f1dd40 con 0x5633e1020980 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7690c640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6f5c0a3240 0x7f6f5c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f7590a640 1 --2- 172.21.15.33:0/4241471828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 0x5633e1020440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.230+0000 7f6f677fe640 1 -- 172.21.15.33:0/4231193004 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6f680d0420 con 0x7f6f70102ef0 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.231+0000 7f6f7610b640 1 --2- 172.21.15.33:0/4241471828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e1020980 0x5633e102ad00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c005940 tx=0x7f6f6c005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91305059b11766d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.231+0000 7f6f7690c640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6f5c0a3240 0x7f6f5c0a5700 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6f68019fb0 tx=0x7f6f68066fa0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.231+0000 7f6f64dea640 1 -- 172.21.15.33:0/4241471828 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c035030 con 0x5633e1020980 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.231+0000 7f6f64dea640 1 -- 172.21.15.33:0/4241471828 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f6c040070 con 0x5633e1020980 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.231+0000 7f6f64dea640 1 -- 172.21.15.33:0/4241471828 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c004d10 con 0x5633e1020980 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.232+0000 7f6f7a748740 1 -- 172.21.15.33:0/4241471828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e1020980 msgr2=0x5633e102ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.232+0000 7f6f7a748740 1 --2- 172.21.15.33:0/4241471828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e1020980 0x5633e102ad00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c005940 tx=0x7f6f6c005e80 comp rx=0 tx=0).stop 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.232+0000 7f6f7a748740 1 -- 172.21.15.33:0/4241471828 shutdown_connections 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.232+0000 7f6f7a748740 1 --2- 172.21.15.33:0/4241471828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e102b240 0x5633e102d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.232+0000 7f6f7a748740 1 --2- 172.21.15.33:0/4241471828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e1020980 0x5633e102ad00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.232+0000 7f6f7a748740 1 --2- 172.21.15.33:0/4241471828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 0x5633e1020440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.232+0000 7f6f7a748740 1 -- 172.21.15.33:0/4241471828 >> 172.21.15.33:0/4241471828 conn(0x5633e101c5f0 msgr2=0x5633e101c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.232+0000 7f6f7a748740 1 -- 172.21.15.33:0/4241471828 shutdown_connections 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.232+0000 7f6f7a748740 1 -- 172.21.15.33:0/4241471828 wait complete. 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.232+0000 7f6f7a748740 1 -- start start 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.232+0000 7f6f7a748740 1 Processor -- start 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7a748740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020060 0x5633e10280a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7a748740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e1020980 0x5633e10285e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7a748740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e102b240 0x5633e1028b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7a748740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5633e102de10 con 0x5633e102b240 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7a748740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5633e0f0df10 con 0x5633e1020060 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7a748740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5633e102df90 con 0x5633e1020980 2024-09-16T00:22:54.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7a748740 10 client.0 _ll_register_callbacks cb 0x5633e0f31db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7590a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e102b240 0x5633e1028b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7590a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e102b240 0x5633e1028b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43208/0 (socket says 172.21.15.33:43208) 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7590a640 1 -- 172.21.15.33:0/2431496906 learned_addr learned my addr 172.21.15.33:0/2431496906 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7590a640 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020060 0x5633e10280a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7610b640 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e1020980 0x5633e10285e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7590a640 1 -- 172.21.15.33:0/2431496906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e1020980 msgr2=0x5633e10285e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7590a640 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e1020980 0x5633e10285e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7590a640 1 -- 172.21.15.33:0/2431496906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020060 msgr2=0x5633e10280a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7590a640 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020060 0x5633e10280a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7590a640 1 -- 172.21.15.33:0/2431496906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633e0c94580 con 0x5633e102b240 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f7590a640 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e102b240 0x5633e1028b20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f6f60012d90 tx=0x7f6f6000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91954818f90e031a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:54.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.233+0000 7f6f59ffb640 1 -- 172.21.15.33:0/2431496906 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60012070 con 0x5633e102b240 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.234+0000 7f6f7a748740 1 -- 172.21.15.33:0/2431496906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f6c040070 con 0x5633e102b240 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.234+0000 7f6f59ffb640 1 -- 172.21.15.33:0/2431496906 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6f60012210 con 0x5633e102b240 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.234+0000 7f6f59ffb640 1 -- 172.21.15.33:0/2431496906 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6f60004020 con 0x5633e102b240 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.234+0000 7f6f7a748740 10 client.16455 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.234+0000 7f6f7a748740 1 -- 172.21.15.33:0/2431496906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5633e0d5e960 con 0x5633e102b240 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.234+0000 7f6f59ffb640 1 -- 172.21.15.33:0/2431496906 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f6c040070 con 0x5633e102b240 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.234+0000 7f6f59ffb640 1 -- 172.21.15.33:0/2431496906 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f6f60024030 con 0x5633e102b240 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.234+0000 7f6f7a748740 10 client.16455 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.234+0000 7f6f7a748740 10 client.16455 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.234+0000 7f6f7a748740 20 client.16455 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.235+0000 7f6f7a748740 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5633e103acd0 0x5633e103d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.235+0000 7f6f7a748740 4 client.16455 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:54.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.235+0000 7f6f7a748740 1 -- 172.21.15.33:0/2431496906 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633e0f92b90 con 0x5633e103acd0 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.235+0000 7f6f7610b640 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5633e103acd0 0x5633e103d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.235+0000 7f6f7610b640 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5633e103acd0 0x5633e103d130 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8222e37578adeac server_cookie=f0d80c41acbcd89d in_seq=0 out_seq=0 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.235+0000 7f6f59ffb640 10 client.16455 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f59ffb640 1 -- 172.21.15.33:0/2431496906 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5633e0f92b90 con 0x5633e103acd0 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f59ffb640 10 client.16455 handle_command_reply: tid=0 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f7a748740 1 client.16455 shutdown 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f7a748740 1 -- 172.21.15.33:0/2431496906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e102b240 msgr2=0x5633e1028b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f7a748740 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e102b240 0x5633e1028b20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f6f60012d90 tx=0x7f6f6000b390 comp rx=0 tx=0).stop 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f7a748740 1 -- 172.21.15.33:0/2431496906 shutdown_connections 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f7a748740 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5633e103acd0 0x5633e103d130 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f7a748740 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e102b240 0x5633e1028b20 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f59ffb640 0 client.16455 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f7a748740 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e1020980 0x5633e10285e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f7a748740 1 --2- 172.21.15.33:0/2431496906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020060 0x5633e10280a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f7a748740 1 -- 172.21.15.33:0/2431496906 >> 172.21.15.33:0/2431496906 conn(0x5633e101c5f0 msgr2=0x5633e101b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:54.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f7a748740 1 -- 172.21.15.33:0/2431496906 shutdown_connections 2024-09-16T00:22:54.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f7a748740 1 -- 172.21.15.33:0/2431496906 wait complete. 2024-09-16T00:22:54.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.236+0000 7f6f7a748740 20 client.16455 trim_cache size 0 max 16384 2024-09-16T00:22:54.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7a748740 1 Processor -- start 2024-09-16T00:22:54.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7a748740 1 -- start start 2024-09-16T00:22:54.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7a748740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020060 0x5633e1022ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7a748740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020980 0x5633e101b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7a748740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e102b240 0x5633e1042a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7a748740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701b3c20 con 0x5633e1020980 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7a748740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701b14f0 con 0x5633e1020060 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7a748740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701af340 con 0x5633e102b240 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7610b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020980 0x5633e101b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7610b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020980 0x5633e101b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43218/0 (socket says 172.21.15.33:43218) 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7610b640 1 -- 172.21.15.33:0/1161197003 learned_addr learned my addr 172.21.15.33:0/1161197003 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7590a640 1 --2- 172.21.15.33:0/1161197003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020060 0x5633e1022ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7590a640 1 --2- 172.21.15.33:0/1161197003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e102b240 0x5633e1042a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7610b640 1 -- 172.21.15.33:0/1161197003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e102b240 msgr2=0x5633e1042a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.254+0000 7f6f7610b640 1 --2- 172.21.15.33:0/1161197003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e102b240 0x5633e1042a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.255+0000 7f6f7610b640 1 -- 172.21.15.33:0/1161197003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020060 msgr2=0x5633e1022ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.255+0000 7f6f7610b640 1 --2- 172.21.15.33:0/1161197003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020060 0x5633e1022ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.255+0000 7f6f7610b640 1 -- 172.21.15.33:0/1161197003 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633e101c090 con 0x5633e1020980 2024-09-16T00:22:54.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.255+0000 7f6f7610b640 1 --2- 172.21.15.33:0/1161197003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020980 0x5633e101b640 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c031470 tx=0x7f6f6c02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d68aa38ef8848 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:54.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.255+0000 7f6f5bfff640 1 -- 172.21.15.33:0/1161197003 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c035040 con 0x5633e1020980 2024-09-16T00:22:54.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.255+0000 7f6f5bfff640 1 -- 172.21.15.33:0/1161197003 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f6c040040 con 0x5633e1020980 2024-09-16T00:22:54.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.255+0000 7f6f5bfff640 1 -- 172.21.15.33:0/1161197003 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c0317a0 con 0x5633e1020980 2024-09-16T00:22:54.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.256+0000 7f6f7a748740 1 -- 172.21.15.33:0/1161197003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020980 msgr2=0x5633e101b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:54.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.256+0000 7f6f7a748740 1 --2- 172.21.15.33:0/1161197003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020980 0x5633e101b640 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c031470 tx=0x7f6f6c02a480 comp rx=0 tx=0).stop 2024-09-16T00:22:54.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.256+0000 7f6f7a748740 1 -- 172.21.15.33:0/1161197003 shutdown_connections 2024-09-16T00:22:54.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.256+0000 7f6f7a748740 1 --2- 172.21.15.33:0/1161197003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e102b240 0x5633e1042a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.256+0000 7f6f7a748740 1 --2- 172.21.15.33:0/1161197003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020980 0x5633e101b640 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.256+0000 7f6f7a748740 1 --2- 172.21.15.33:0/1161197003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020060 0x5633e1022ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.256+0000 7f6f7a748740 1 -- 172.21.15.33:0/1161197003 >> 172.21.15.33:0/1161197003 conn(0x5633e101c5f0 msgr2=0x5633e101af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:54.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.256+0000 7f6f7a748740 1 -- 172.21.15.33:0/1161197003 shutdown_connections 2024-09-16T00:22:54.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.256+0000 7f6f7a748740 1 -- 172.21.15.33:0/1161197003 wait complete. 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7a748740 1 -- start start 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7a748740 1 Processor -- start 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7a748740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 0x5633e10333c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7a748740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020980 0x5633e1033900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7a748740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e102b240 0x5633e10301b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7a748740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701af9b0 con 0x5633e1020060 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7a748740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701b0e20 con 0x5633e1020980 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7a748740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701b0080 con 0x5633e102b240 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7a748740 10 client.0 _ll_register_callbacks cb 0x7f6f60012830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7590a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 0x5633e10333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7590a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 0x5633e10333c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43222/0 (socket says 172.21.15.33:43222) 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7590a640 1 -- 172.21.15.33:0/3829551908 learned_addr learned my addr 172.21.15.33:0/3829551908 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:54.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.257+0000 7f6f7610b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020980 0x5633e1033900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f7590a640 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e102b240 0x5633e10301b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f7590a640 1 -- 172.21.15.33:0/3829551908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e102b240 msgr2=0x5633e10301b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f7590a640 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e102b240 0x5633e10301b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f7590a640 1 -- 172.21.15.33:0/3829551908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020980 msgr2=0x5633e1033900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f7590a640 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020980 0x5633e1033900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f7590a640 1 -- 172.21.15.33:0/3829551908 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633e10308a0 con 0x5633e1020060 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f7610b640 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020980 0x5633e1033900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f7590a640 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 0x5633e10333c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6f60003d60 tx=0x7f6f6000b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e3e5db12834040 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f59ffb640 1 -- 172.21.15.33:0/3829551908 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60035020 con 0x5633e1020060 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f59ffb640 1 -- 172.21.15.33:0/3829551908 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6f60003a00 con 0x5633e1020060 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f7a748740 1 -- 172.21.15.33:0/3829551908 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f6c040040 con 0x5633e1020060 2024-09-16T00:22:54.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f59ffb640 1 -- 172.21.15.33:0/3829551908 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6f60040050 con 0x5633e1020060 2024-09-16T00:22:54.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f7a748740 10 client.16461 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:54.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f7a748740 1 -- 172.21.15.33:0/3829551908 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5633e10a6410 con 0x5633e1020060 2024-09-16T00:22:54.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.258+0000 7f6f59ffb640 1 -- 172.21.15.33:0/3829551908 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f6000ee70 con 0x5633e1020060 2024-09-16T00:22:54.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.259+0000 7f6f59ffb640 1 -- 172.21.15.33:0/3829551908 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f6f60024070 con 0x5633e1020060 2024-09-16T00:22:54.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.259+0000 7f6f7a748740 10 client.16461 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:54.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.259+0000 7f6f7a748740 10 client.16461 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:22:54.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.259+0000 7f6f7a748740 20 client.16461 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:54.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.259+0000 7f6f7a748740 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5633e103acd0 0x5633e10a8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:54.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.259+0000 7f6f7a748740 4 client.16461 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "9279b40d", "release": true, "await": true}] 2024-09-16T00:22:54.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.259+0000 7f6f7a748740 1 -- 172.21.15.33:0/3829551908 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "9279b40d", "release": true, "await": true}) -- 0x7f6f6c040040 con 0x5633e103acd0 2024-09-16T00:22:54.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.260+0000 7f6f7610b640 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5633e103acd0 0x5633e10a8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:54.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.260+0000 7f6f7610b640 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5633e103acd0 0x5633e10a8b10 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb33492fdb498516 server_cookie=2c1ebf807ce2cd33 in_seq=0 out_seq=0 2024-09-16T00:22:54.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.260+0000 7f6f59ffb640 10 client.16461 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:54.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.272+0000 7f6f59ffb640 1 -- 172.21.15.33:0/3829551908 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f6f6c040040 con 0x5633e103acd0 2024-09-16T00:22:54.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.272+0000 7f6f59ffb640 10 client.16461 handle_command_reply: tid=0 2024-09-16T00:22:54.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f7a748740 1 client.16461 shutdown 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f7a748740 1 -- 172.21.15.33:0/3829551908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 msgr2=0x5633e10333c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f7a748740 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 0x5633e10333c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6f60003d60 tx=0x7f6f6000b570 comp rx=0 tx=0).stop 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f7a748740 1 -- 172.21.15.33:0/3829551908 shutdown_connections 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f7a748740 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5633e103acd0 0x5633e10a8b10 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f7a748740 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5633e102b240 0x5633e10301b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f7a748740 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5633e1020980 0x5633e1033900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f7a748740 1 --2- 172.21.15.33:0/3829551908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633e1020060 0x5633e10333c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f7a748740 1 -- 172.21.15.33:0/3829551908 >> 172.21.15.33:0/3829551908 conn(0x5633e101c5f0 msgr2=0x5633e1028060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f59ffb640 0 client.16461 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f7a748740 1 -- 172.21.15.33:0/3829551908 shutdown_connections 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f7a748740 1 -- 172.21.15.33:0/3829551908 wait complete. 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.273+0000 7f6f7a748740 20 client.16461 trim_cache size 0 max 16384 2024-09-16T00:22:54.275 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:22:54.276 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:22:54.276 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:22:54.276 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 20, 2024-09-16T00:22:54.276 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:22:54.276 INFO:teuthology.orchestra.run.smithi033.stdout: "9279b40d": { 2024-09-16T00:22:54.276 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 20, 2024-09-16T00:22:54.276 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:22:54.276 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:22:54.276 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:22:54.276 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:22:54.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:22:54.276 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 145.3, 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:54.277 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:22:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.275+0000 7f6f7836f640 1 -- 172.21.15.33:0/4231193004 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6f5c0a3240 msgr2=0x7f6f5c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.275+0000 7f6f7836f640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6f5c0a3240 0x7f6f5c0a5700 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6f68019fb0 tx=0x7f6f68066fa0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.275+0000 7f6f7836f640 1 -- 172.21.15.33:0/4231193004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f70102ef0 msgr2=0x7f6f7010ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.275+0000 7f6f7836f640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f70102ef0 0x7f6f7010ef50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800eb10 tx=0x7f6f6800c2d0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.275+0000 7f6f7836f640 1 -- 172.21.15.33:0/4231193004 shutdown_connections 2024-09-16T00:22:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.275+0000 7f6f7836f640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6f5c0a3240 0x7f6f5c0a5700 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.275+0000 7f6f7836f640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f70102ef0 0x7f6f7010ef50 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.275+0000 7f6f7836f640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f70102570 0x7f6f7010ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.275+0000 7f6f7836f640 1 --2- 172.21.15.33:0/4231193004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f70100e20 0x7f6f7010e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.275+0000 7f6f7836f640 1 -- 172.21.15.33:0/4231193004 >> 172.21.15.33:0/4231193004 conn(0x7f6f700fc7d0 msgr2=0x7f6f700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.275+0000 7f6f7836f640 1 -- 172.21.15.33:0/4231193004 shutdown_connections 2024-09-16T00:22:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:54.275+0000 7f6f7836f640 1 -- 172.21.15.33:0/4231193004 wait complete. 2024-09-16T00:22:54.551 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:54.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 338569 9.72 MB/sec execute 122 sec latency 26844.805 ms 2024-09-16T00:22:55.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:54 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:22:55.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:54 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:55.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:54 smithi033 ceph-mon[31243]: pgmap v523: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.1 MiB/s wr, 85 op/s 2024-09-16T00:22:55.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:54 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:22:55.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:54 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:55.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:54 smithi083 ceph-mon[35242]: pgmap v523: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.1 MiB/s wr, 85 op/s 2024-09-16T00:22:55.188 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '9279b40d', total seconds elapsed: 30.4 2024-09-16T00:22:55.189 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.35 2024-09-16T00:22:55.189 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:22:55.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:54 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:22:55.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:54 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:22:55.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:54 smithi060 ceph-mon[35185]: pgmap v523: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.1 MiB/s wr, 85 op/s 2024-09-16T00:22:55.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 341242 9.75 MB/sec execute 123 sec latency 17.426 ms 2024-09-16T00:22:55.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.735+0000 7f2906dec640 1 -- 172.21.15.33:0/1460687198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2900102d90 msgr2=0x7f2900103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:55.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.735+0000 7f2906dec640 1 --2- 172.21.15.33:0/1460687198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2900102d90 0x7f2900103170 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f28f0009930 tx=0x7f28f002a420 comp rx=0 tx=0).stop 2024-09-16T00:22:55.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.735+0000 7f2906dec640 1 -- 172.21.15.33:0/1460687198 shutdown_connections 2024-09-16T00:22:55.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.735+0000 7f2906dec640 1 --2- 172.21.15.33:0/1460687198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2900108290 0x7f290010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.735+0000 7f2906dec640 1 --2- 172.21.15.33:0/1460687198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2900103740 0x7f2900107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.735+0000 7f2906dec640 1 --2- 172.21.15.33:0/1460687198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2900102d90 0x7f2900103170 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.735+0000 7f2906dec640 1 -- 172.21.15.33:0/1460687198 >> 172.21.15.33:0/1460687198 conn(0x7f29000fc7d0 msgr2=0x7f29000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:55.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.735+0000 7f2906dec640 1 -- 172.21.15.33:0/1460687198 shutdown_connections 2024-09-16T00:22:55.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.735+0000 7f2906dec640 1 -- 172.21.15.33:0/1460687198 wait complete. 2024-09-16T00:22:55.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2906dec640 1 Processor -- start 2024-09-16T00:22:55.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2906dec640 1 -- start start 2024-09-16T00:22:55.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2906dec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2900102d90 0x7f29001009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2906dec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2900103740 0x7f2900100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2906dec640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2900108290 0x7f2900101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2906dec640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2900112090 con 0x7f2900108290 2024-09-16T00:22:55.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2906dec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2900111f10 con 0x7f2900102d90 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2906dec640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2900112210 con 0x7f2900103740 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f28fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2900103740 0x7f2900100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2905389640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2900108290 0x7f2900101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f28fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2900103740 0x7f2900100f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52406/0 (socket says 172.21.15.33:52406) 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f28fffff640 1 -- 172.21.15.33:0/724385666 learned_addr learned my addr 172.21.15.33:0/724385666 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2904b88640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2900102d90 0x7f29001009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2905389640 1 -- 172.21.15.33:0/724385666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2900103740 msgr2=0x7f2900100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2905389640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2900103740 0x7f2900100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2905389640 1 -- 172.21.15.33:0/724385666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2900102d90 msgr2=0x7f29001009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2905389640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2900102d90 0x7f29001009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.736+0000 7f2905389640 1 -- 172.21.15.33:0/724385666 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29001a6a90 con 0x7f2900108290 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.737+0000 7f2905389640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2900108290 0x7f2900101470 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f28f800da10 tx=0x7f28f800ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:55.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.737+0000 7f28fdffb640 1 -- 172.21.15.33:0/724385666 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f800bd70 con 0x7f2900108290 2024-09-16T00:22:55.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.737+0000 7f2906dec640 1 -- 172.21.15.33:0/724385666 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29001a6d80 con 0x7f2900108290 2024-09-16T00:22:55.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.737+0000 7f28fdffb640 1 -- 172.21.15.33:0/724385666 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f8003e50 con 0x7f2900108290 2024-09-16T00:22:55.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.737+0000 7f28fdffb640 1 -- 172.21.15.33:0/724385666 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f80106a0 con 0x7f2900108290 2024-09-16T00:22:55.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.737+0000 7f2906dec640 1 -- 172.21.15.33:0/724385666 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29001a7270 con 0x7f2900108290 2024-09-16T00:22:55.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.739+0000 7f28fdffb640 1 -- 172.21.15.33:0/724385666 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f28f8003970 con 0x7f2900108290 2024-09-16T00:22:55.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f29091c5740 1 Processor -- start 2024-09-16T00:22:55.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f29091c5740 1 -- start start 2024-09-16T00:22:55.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f28fdffb640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f28e80a3240 0x7f28e80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f29091c5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace28c4a0 0x560ace28c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f29091c5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28cdc0 0x560ace297140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f29091c5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace297680 0x560ace299a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f29091c5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560ace1f22e0 con 0x560ace28c4a0 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f29091c5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560ace199ce0 con 0x560ace28cdc0 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f29091c5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560ace1ee2a0 con 0x560ace297680 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f28fdffb640 1 -- 172.21.15.33:0/724385666 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f28f80d00a0 con 0x7f2900108290 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f2904b88640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace28c4a0 0x560ace28c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f2904b88640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace28c4a0 0x560ace28c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43248/0 (socket says 172.21.15.33:43248) 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f2904b88640 1 -- 172.21.15.33:0/1511958425 learned_addr learned my addr 172.21.15.33:0/1511958425 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f2905389640 1 --2- 172.21.15.33:0/1511958425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace297680 0x560ace299a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.740+0000 7f28fffff640 1 --2- 172.21.15.33:0/1511958425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28cdc0 0x560ace297140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.741+0000 7f28fffff640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f28e80a3240 0x7f28e80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.741+0000 7f28fffff640 1 -- 172.21.15.33:0/1511958425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace297680 msgr2=0x560ace299a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:55.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.741+0000 7f28fffff640 1 --2- 172.21.15.33:0/1511958425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace297680 0x560ace299a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.741+0000 7f28fffff640 1 -- 172.21.15.33:0/1511958425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace28c4a0 msgr2=0x560ace28c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.741+0000 7f28fffff640 1 --2- 172.21.15.33:0/1511958425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace28c4a0 0x560ace28c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.741+0000 7f28fffff640 1 -- 172.21.15.33:0/1511958425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ace17e440 con 0x560ace28cdc0 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.741+0000 7f2904b88640 1 --2- 172.21.15.33:0/1511958425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace28c4a0 0x560ace28c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.741+0000 7f28fffff640 1 --2- 172.21.15.33:0/1511958425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28cdc0 0x560ace297140 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f28f40097f0 tx=0x7f28f4011ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b1ef6de2f47954a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.741+0000 7f28fffff640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f28e80a3240 0x7f28e80a5700 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f28f4011fd0 tx=0x7f28f4013040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.742+0000 7f28e6ffd640 1 -- 172.21.15.33:0/1511958425 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4022070 con 0x560ace28cdc0 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.742+0000 7f28e6ffd640 1 -- 172.21.15.33:0/1511958425 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f40035f0 con 0x560ace28cdc0 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.742+0000 7f28e6ffd640 1 -- 172.21.15.33:0/1511958425 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4022070 con 0x560ace28cdc0 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.742+0000 7f29091c5740 1 -- 172.21.15.33:0/1511958425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28cdc0 msgr2=0x560ace297140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.742+0000 7f29091c5740 1 --2- 172.21.15.33:0/1511958425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28cdc0 0x560ace297140 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f28f40097f0 tx=0x7f28f4011ad0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.743+0000 7f29091c5740 1 -- 172.21.15.33:0/1511958425 shutdown_connections 2024-09-16T00:22:55.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.743+0000 7f29091c5740 1 --2- 172.21.15.33:0/1511958425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace297680 0x560ace299a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.743+0000 7f29091c5740 1 --2- 172.21.15.33:0/1511958425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28cdc0 0x560ace297140 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.743+0000 7f29091c5740 1 --2- 172.21.15.33:0/1511958425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace28c4a0 0x560ace28c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.743+0000 7f29091c5740 1 -- 172.21.15.33:0/1511958425 >> 172.21.15.33:0/1511958425 conn(0x560ace288a30 msgr2=0x560ace288e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.743+0000 7f29091c5740 1 -- 172.21.15.33:0/1511958425 shutdown_connections 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.743+0000 7f29091c5740 1 -- 172.21.15.33:0/1511958425 wait complete. 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.743+0000 7f29091c5740 1 -- start start 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.743+0000 7f29091c5740 1 Processor -- start 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f29091c5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28c4a0 0x560ace296f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f29091c5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28cdc0 0x560ace294240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f29091c5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 0x560ace294780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f29091c5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560ace29a250 con 0x560ace297680 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f29091c5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560ace1ad050 con 0x560ace28cdc0 2024-09-16T00:22:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f29091c5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560ace29a3d0 con 0x560ace28c4a0 2024-09-16T00:22:55.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f28fffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28cdc0 0x560ace294240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f2904b88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28c4a0 0x560ace296f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f2904b88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28c4a0 0x560ace296f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52420/0 (socket says 172.21.15.33:52420) 2024-09-16T00:22:55.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f2904b88640 1 -- 172.21.15.33:0/2768418831 learned_addr learned my addr 172.21.15.33:0/2768418831 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:55.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f29091c5740 10 client.0 _ll_register_callbacks cb 0x560ace1ed160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:55.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f2905389640 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 0x560ace294780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f2904b88640 1 -- 172.21.15.33:0/2768418831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28cdc0 msgr2=0x560ace294240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:55.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f2904b88640 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28cdc0 0x560ace294240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f2904b88640 1 -- 172.21.15.33:0/2768418831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 msgr2=0x560ace294780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:55.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f2904b88640 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 0x560ace294780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.744+0000 7f2904b88640 1 -- 172.21.15.33:0/2768418831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560acdf633c0 con 0x560ace28c4a0 2024-09-16T00:22:55.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.745+0000 7f28fffff640 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28cdc0 0x560ace294240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:55.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.745+0000 7f2904b88640 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28c4a0 0x560ace296f90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f28f0005940 tx=0x7f28f00317e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c39a237b6ab98bae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:55.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.745+0000 7f28dffff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:22:55.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.745+0000 7f28dffff640 1 -- 172.21.15.33:0/2768418831 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0035040 con 0x560ace28c4a0 2024-09-16T00:22:55.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.745+0000 7f29091c5740 1 -- 172.21.15.33:0/2768418831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28f40035f0 con 0x560ace28c4a0 2024-09-16T00:22:55.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.745+0000 7f28dffff640 1 -- 172.21.15.33:0/2768418831 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f28f0004240 con 0x560ace28c4a0 2024-09-16T00:22:55.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.745+0000 7f28dffff640 1 -- 172.21.15.33:0/2768418831 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f28f0004440 con 0x560ace28c4a0 2024-09-16T00:22:55.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.745+0000 7f29091c5740 10 client.25915 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:55.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.745+0000 7f29091c5740 1 -- 172.21.15.33:0/2768418831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560ace2a5c10 con 0x560ace28c4a0 2024-09-16T00:22:55.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.746+0000 7f28dffff640 1 -- 172.21.15.33:0/2768418831 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f40035f0 con 0x560ace28c4a0 2024-09-16T00:22:55.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.746+0000 7f28dffff640 1 -- 172.21.15.33:0/2768418831 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f28f00035f0 con 0x560ace28c4a0 2024-09-16T00:22:55.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.746+0000 7f29091c5740 10 client.25915 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:55.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.746+0000 7f29091c5740 10 client.25915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:55.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.746+0000 7f29091c5740 20 client.25915 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:55.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.746+0000 7f29091c5740 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560ace2a7020 0x560ace2a9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.746+0000 7f29091c5740 4 client.25915 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:55.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.746+0000 7f29091c5740 1 -- 172.21.15.33:0/2768418831 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ace1ff850 con 0x560ace2a7020 2024-09-16T00:22:55.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.746+0000 7f28fffff640 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560ace2a7020 0x560ace2a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.746+0000 7f28dffff640 10 client.25915 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:55.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f28fffff640 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560ace2a7020 0x560ace2a9430 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=914b1d9eefe14d8a server_cookie=50c645ab4ef83f87 in_seq=0 out_seq=0 2024-09-16T00:22:55.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f28dffff640 1 -- 172.21.15.33:0/2768418831 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560ace1ff850 con 0x560ace2a7020 2024-09-16T00:22:55.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f28dffff640 10 client.25915 handle_command_reply: tid=0 2024-09-16T00:22:55.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f29091c5740 1 client.25915 shutdown 2024-09-16T00:22:55.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f29091c5740 1 -- 172.21.15.33:0/2768418831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28c4a0 msgr2=0x560ace296f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:55.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f29091c5740 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28c4a0 0x560ace296f90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f28f0005940 tx=0x7f28f00317e0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f29091c5740 1 -- 172.21.15.33:0/2768418831 shutdown_connections 2024-09-16T00:22:55.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f29091c5740 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560ace2a7020 0x560ace2a9430 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f29091c5740 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 0x560ace294780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f29091c5740 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28cdc0 0x560ace294240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f29091c5740 1 --2- 172.21.15.33:0/2768418831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28c4a0 0x560ace296f90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f29091c5740 1 -- 172.21.15.33:0/2768418831 >> 172.21.15.33:0/2768418831 conn(0x560ace288a30 msgr2=0x560ace28e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:55.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.747+0000 7f28dffff640 0 client.25915 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:55.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.748+0000 7f29091c5740 1 -- 172.21.15.33:0/2768418831 shutdown_connections 2024-09-16T00:22:55.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.748+0000 7f29091c5740 1 -- 172.21.15.33:0/2768418831 wait complete. 2024-09-16T00:22:55.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.748+0000 7f29091c5740 20 client.25915 trim_cache size 0 max 16384 2024-09-16T00:22:55.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.773+0000 7f29091c5740 1 Processor -- start 2024-09-16T00:22:55.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.773+0000 7f29091c5740 1 -- start start 2024-09-16T00:22:55.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.773+0000 7f29091c5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28c4a0 0x560ace2a34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.773+0000 7f29091c5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28cdc0 0x560ace28f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.773+0000 7f29091c5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 0x560ace2aed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.773+0000 7f29091c5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f29001af770 con 0x560ace297680 2024-09-16T00:22:55.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.773+0000 7f29091c5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f29001ad040 con 0x560ace28c4a0 2024-09-16T00:22:55.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.773+0000 7f29091c5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29001aae90 con 0x560ace28cdc0 2024-09-16T00:22:55.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f28fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28cdc0 0x560ace28f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f2904b88640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28c4a0 0x560ace2a34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f28fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28cdc0 0x560ace28f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52428/0 (socket says 172.21.15.33:52428) 2024-09-16T00:22:55.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f28fffff640 1 -- 172.21.15.33:0/1288894819 learned_addr learned my addr 172.21.15.33:0/1288894819 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:55.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f28fffff640 1 -- 172.21.15.33:0/1288894819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28c4a0 msgr2=0x560ace2a34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:55.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f28fffff640 1 --2- 172.21.15.33:0/1288894819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28c4a0 0x560ace2a34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f28fffff640 1 -- 172.21.15.33:0/1288894819 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 msgr2=0x560ace2aed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:55.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f28fffff640 1 --2- 172.21.15.33:0/1288894819 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 0x560ace2aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f28fffff640 1 -- 172.21.15.33:0/1288894819 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ace288860 con 0x560ace28cdc0 2024-09-16T00:22:55.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f28fffff640 1 --2- 172.21.15.33:0/1288894819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28cdc0 0x560ace28f7f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f28f4013010 tx=0x7f28f40096e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=918abe47917bf800 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:55.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f28e67fc640 1 -- 172.21.15.33:0/1288894819 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4022040 con 0x560ace28cdc0 2024-09-16T00:22:55.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f28e67fc640 1 -- 172.21.15.33:0/1288894819 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f400fb50 con 0x560ace28cdc0 2024-09-16T00:22:55.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.774+0000 7f28e67fc640 1 -- 172.21.15.33:0/1288894819 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4004bc0 con 0x560ace28cdc0 2024-09-16T00:22:55.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.775+0000 7f29091c5740 1 -- 172.21.15.33:0/1288894819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28cdc0 msgr2=0x560ace28f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:55.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.775+0000 7f29091c5740 1 --2- 172.21.15.33:0/1288894819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28cdc0 0x560ace28f7f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f28f4013010 tx=0x7f28f40096e0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.775+0000 7f29091c5740 1 -- 172.21.15.33:0/1288894819 shutdown_connections 2024-09-16T00:22:55.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.775+0000 7f29091c5740 1 --2- 172.21.15.33:0/1288894819 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 0x560ace2aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.775+0000 7f29091c5740 1 --2- 172.21.15.33:0/1288894819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace28cdc0 0x560ace28f7f0 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f28f4013010 tx=0x7f28f40096e0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.775+0000 7f29091c5740 1 --2- 172.21.15.33:0/1288894819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28c4a0 0x560ace2a34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.775+0000 7f29091c5740 1 -- 172.21.15.33:0/1288894819 >> 172.21.15.33:0/1288894819 conn(0x560ace288a30 msgr2=0x560ace28f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:55.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.776+0000 7f29091c5740 1 -- 172.21.15.33:0/1288894819 shutdown_connections 2024-09-16T00:22:55.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.776+0000 7f29091c5740 1 -- 172.21.15.33:0/1288894819 wait complete. 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.776+0000 7f29091c5740 1 -- start start 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.776+0000 7f29091c5740 1 Processor -- start 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f29091c5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28c4a0 0x560ace29fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f29091c5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 0x560ace29aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f29091c5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace29b3f0 0x560ace29b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f29091c5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f29001ab500 con 0x560ace297680 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f29091c5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f29001ac970 con 0x560ace28c4a0 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f29091c5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29001abbd0 con 0x560ace29b3f0 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f29091c5740 10 client.0 _ll_register_callbacks cb 0x7f28f0009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f2905389640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace29b3f0 0x560ace29b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f28fffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 0x560ace29aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f2905389640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace29b3f0 0x560ace29b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52440/0 (socket says 172.21.15.33:52440) 2024-09-16T00:22:55.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f2905389640 1 -- 172.21.15.33:0/3643778924 learned_addr learned my addr 172.21.15.33:0/3643778924 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:55.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f2904b88640 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28c4a0 0x560ace29fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f2905389640 1 -- 172.21.15.33:0/3643778924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28c4a0 msgr2=0x560ace29fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:55.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f2905389640 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28c4a0 0x560ace29fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f2905389640 1 -- 172.21.15.33:0/3643778924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 msgr2=0x560ace29aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:55.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f2905389640 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 0x560ace29aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f2905389640 1 -- 172.21.15.33:0/3643778924 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ace310cd0 con 0x560ace29b3f0 2024-09-16T00:22:55.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.777+0000 7f2904b88640 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28c4a0 0x560ace29fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:55.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f2905389640 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace29b3f0 0x560ace29b870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f28f80986a0 tx=0x7f28f8079730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7bdeb62cc5d83e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:55.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f28dffff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:22:55.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f28dffff640 1 -- 172.21.15.33:0/3643778924 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f8079260 con 0x560ace29b3f0 2024-09-16T00:22:55.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f29091c5740 1 -- 172.21.15.33:0/3643778924 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28f400fb50 con 0x560ace29b3f0 2024-09-16T00:22:55.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f28dffff640 1 -- 172.21.15.33:0/3643778924 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f28f8004000 con 0x560ace29b3f0 2024-09-16T00:22:55.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f28dffff640 1 -- 172.21.15.33:0/3643778924 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f28f809c240 con 0x560ace29b3f0 2024-09-16T00:22:55.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f29091c5740 10 client.25921 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:55.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f29091c5740 1 -- 172.21.15.33:0/3643778924 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560ace311f90 con 0x560ace29b3f0 2024-09-16T00:22:55.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f28dffff640 1 -- 172.21.15.33:0/3643778924 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f80a05f0 con 0x560ace29b3f0 2024-09-16T00:22:55.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f28dffff640 1 -- 172.21.15.33:0/3643778924 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f28f8024060 con 0x560ace29b3f0 2024-09-16T00:22:55.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f29091c5740 10 client.25921 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:55.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f29091c5740 10 client.25921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:22:55.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.778+0000 7f29091c5740 20 client.25921 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.779+0000 7f29091c5740 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560ace3126b0 0x560ace314aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.779+0000 7f29091c5740 4 client.25921 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:22:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.779+0000 7f29091c5740 1 -- 172.21.15.33:0/3643778924 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f28f400fb50 con 0x560ace3126b0 2024-09-16T00:22:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.779+0000 7f2904b88640 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560ace3126b0 0x560ace314aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.779+0000 7f28dffff640 10 client.25921 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.779+0000 7f2904b88640 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560ace3126b0 0x560ace314aa0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47849e35894cada server_cookie=4af0bedc5b4354b5 in_seq=0 out_seq=0 2024-09-16T00:22:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f28dffff640 1 -- 172.21.15.33:0/3643778924 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f28f400fb50 con 0x560ace3126b0 2024-09-16T00:22:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f28dffff640 10 client.25921 handle_command_reply: tid=0 2024-09-16T00:22:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f29091c5740 1 client.25921 shutdown 2024-09-16T00:22:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f29091c5740 1 -- 172.21.15.33:0/3643778924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace29b3f0 msgr2=0x560ace29b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f29091c5740 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace29b3f0 0x560ace29b870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f28f80986a0 tx=0x7f28f8079730 comp rx=0 tx=0).stop 2024-09-16T00:22:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f29091c5740 1 -- 172.21.15.33:0/3643778924 shutdown_connections 2024-09-16T00:22:55.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f29091c5740 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560ace3126b0 0x560ace314aa0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f29091c5740 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ace29b3f0 0x560ace29b870 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f29091c5740 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ace297680 0x560ace29aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f29091c5740 1 --2- 172.21.15.33:0/3643778924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ace28c4a0 0x560ace29fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f28dffff640 0 client.25921 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:55.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f29091c5740 1 -- 172.21.15.33:0/3643778924 >> 172.21.15.33:0/3643778924 conn(0x560ace288a30 msgr2=0x560ace287b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:55.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f29091c5740 1 -- 172.21.15.33:0/3643778924 shutdown_connections 2024-09-16T00:22:55.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f29091c5740 1 -- 172.21.15.33:0/3643778924 wait complete. 2024-09-16T00:22:55.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.780+0000 7f29091c5740 20 client.25921 trim_cache size 0 max 16384 2024-09-16T00:22:55.786 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:22:55.786 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:22:55.786 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:22:55.786 INFO:teuthology.orchestra.run.smithi033.stdout: "62cb8df5-8435-4d2d-9546-aeef8bf67fb3": { 2024-09-16T00:22:55.786 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:22:55.786 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "62cb8df5-8435-4d2d-9546-aeef8bf67fb3", 2024-09-16T00:22:55.786 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:22:55.786 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:55.786 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:55.786 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:22:55.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.781+0000 7f2906dec640 1 -- 172.21.15.33:0/724385666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f28e80a3240 msgr2=0x7f28e80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:55.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.781+0000 7f2906dec640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f28e80a3240 0x7f28e80a5700 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f28f4011fd0 tx=0x7f28f4013040 comp rx=0 tx=0).stop 2024-09-16T00:22:55.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.781+0000 7f2906dec640 1 -- 172.21.15.33:0/724385666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2900108290 msgr2=0x7f2900101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:55.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.781+0000 7f2906dec640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2900108290 0x7f2900101470 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f28f800da10 tx=0x7f28f800ded0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.781+0000 7f2906dec640 1 -- 172.21.15.33:0/724385666 shutdown_connections 2024-09-16T00:22:55.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.781+0000 7f2906dec640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f28e80a3240 0x7f28e80a5700 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.781+0000 7f2906dec640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2900108290 0x7f2900101470 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.781+0000 7f2906dec640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2900103740 0x7f2900100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.781+0000 7f2906dec640 1 --2- 172.21.15.33:0/724385666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2900102d90 0x7f29001009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:55.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.781+0000 7f2906dec640 1 -- 172.21.15.33:0/724385666 >> 172.21.15.33:0/724385666 conn(0x7f29000fc7d0 msgr2=0x7f2900105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:55.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.782+0000 7f2906dec640 1 -- 172.21.15.33:0/724385666 shutdown_connections 2024-09-16T00:22:55.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:55.782+0000 7f2906dec640 1 -- 172.21.15.33:0/724385666 wait complete. 2024-09-16T00:22:56.660 INFO:tasks.cephfs.filesystem:scrub status for tag:62cb8df5-8435-4d2d-9546-aeef8bf67fb3 - {'path': '/', 'tag': '62cb8df5-8435-4d2d-9546-aeef8bf67fb3', 'options': 'recursive,force'} 2024-09-16T00:22:56.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 344450 9.83 MB/sec execute 124 sec latency 8.544 ms 2024-09-16T00:22:56.732 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:56.890 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:56 smithi033 ceph-mon[31243]: pgmap v524: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 2.3 MiB/s wr, 100 op/s 2024-09-16T00:22:57.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:56 smithi083 ceph-mon[35242]: pgmap v524: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 2.3 MiB/s wr, 100 op/s 2024-09-16T00:22:57.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:56 smithi060 ceph-mon[35185]: pgmap v524: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 2.3 MiB/s wr, 100 op/s 2024-09-16T00:22:57.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.263+0000 7f7bb5daf640 1 -- 172.21.15.33:0/972304938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bb0102ef0 msgr2=0x7f7bb010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:57.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.263+0000 7f7bb5daf640 1 --2- 172.21.15.33:0/972304938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bb0102ef0 0x7f7bb010da40 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8009ca0 tx=0x7f7ba802a950 comp rx=0 tx=0).stop 2024-09-16T00:22:57.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.263+0000 7f7bb5daf640 1 -- 172.21.15.33:0/972304938 shutdown_connections 2024-09-16T00:22:57.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.263+0000 7f7bb5daf640 1 --2- 172.21.15.33:0/972304938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bb0102ef0 0x7f7bb010da40 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:57.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.263+0000 7f7bb5daf640 1 --2- 172.21.15.33:0/972304938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bb0102570 0x7f7bb01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:57.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.263+0000 7f7bb5daf640 1 --2- 172.21.15.33:0/972304938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bb0100e20 0x7f7bb0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:57.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.263+0000 7f7bb5daf640 1 -- 172.21.15.33:0/972304938 >> 172.21.15.33:0/972304938 conn(0x7f7bb00fc7d0 msgr2=0x7f7bb00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:57.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.263+0000 7f7bb5daf640 1 -- 172.21.15.33:0/972304938 shutdown_connections 2024-09-16T00:22:57.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.263+0000 7f7bb5daf640 1 -- 172.21.15.33:0/972304938 wait complete. 2024-09-16T00:22:57.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7bb5daf640 1 Processor -- start 2024-09-16T00:22:57.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7bb5daf640 1 -- start start 2024-09-16T00:22:57.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7bb5daf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bb0100e20 0x7f7bb0199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:57.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7bb5daf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bb0102570 0x7f7bb0199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:57.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7bb5daf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bb0102ef0 0x7f7bb01a43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:57.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7bb5daf640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb0110fd0 con 0x7f7bb0102ef0 2024-09-16T00:22:57.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7baffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bb0102ef0 0x7f7bb01a43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:57.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7bb5daf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb0110e50 con 0x7f7bb0102570 2024-09-16T00:22:57.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7bb5daf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb0111150 con 0x7f7bb0100e20 2024-09-16T00:22:57.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7baf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bb0100e20 0x7f7bb0199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:57.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7baffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bb0102ef0 0x7f7bb01a43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43300/0 (socket says 172.21.15.33:43300) 2024-09-16T00:22:57.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7baffff640 1 -- 172.21.15.33:0/3040149384 learned_addr learned my addr 172.21.15.33:0/3040149384 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:57.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.264+0000 7f7baf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bb0100e20 0x7f7bb0199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52456/0 (socket says 172.21.15.33:52456) 2024-09-16T00:22:57.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7baeffd640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bb0102570 0x7f7bb0199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:57.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7baf7fe640 1 -- 172.21.15.33:0/3040149384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bb0102570 msgr2=0x7f7bb0199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:57.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7baf7fe640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bb0102570 0x7f7bb0199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:57.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7baf7fe640 1 -- 172.21.15.33:0/3040149384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bb0102ef0 msgr2=0x7f7bb01a43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:57.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7baf7fe640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bb0102ef0 0x7f7bb01a43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:57.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7baf7fe640 1 -- 172.21.15.33:0/3040149384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb0197130 con 0x7f7bb0100e20 2024-09-16T00:22:57.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7baeffd640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bb0102570 0x7f7bb0199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:57.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7baf7fe640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bb0100e20 0x7f7bb0199340 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000e9f0 tx=0x7f7ba000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:57.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7bacff9640 1 -- 172.21.15.33:0/3040149384 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba000ccc0 con 0x7f7bb0100e20 2024-09-16T00:22:57.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7bacff9640 1 -- 172.21.15.33:0/3040149384 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ba0010910 con 0x7f7bb0100e20 2024-09-16T00:22:57.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7bb5daf640 1 -- 172.21.15.33:0/3040149384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb0197420 con 0x7f7bb0100e20 2024-09-16T00:22:57.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7bacff9640 1 -- 172.21.15.33:0/3040149384 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba00189c0 con 0x7f7bb0100e20 2024-09-16T00:22:57.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.265+0000 7f7bb5daf640 1 -- 172.21.15.33:0/3040149384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb0197960 con 0x7f7bb0100e20 2024-09-16T00:22:57.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.267+0000 7f7bb5daf640 1 -- 172.21.15.33:0/3040149384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bb01021b0 con 0x7f7bb0100e20 2024-09-16T00:22:57.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.268+0000 7f7bacff9640 1 -- 172.21.15.33:0/3040149384 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7ba0010ac0 con 0x7f7bb0100e20 2024-09-16T00:22:57.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.269+0000 7f7bacff9640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b980a3240 0x7f7b980a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:57.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.269+0000 7f7baeffd640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b980a3240 0x7f7b980a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:57.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.269+0000 7f7bacff9640 1 -- 172.21.15.33:0/3040149384 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7ba00d0280 con 0x7f7bb0100e20 2024-09-16T00:22:57.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.270+0000 7f7baeffd640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b980a3240 0x7f7b980a5700 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7bb006aac0 tx=0x7f7ba4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:57.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.275+0000 7f7bacff9640 1 -- 172.21.15.33:0/3040149384 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7ba009cac0 con 0x7f7bb0100e20 2024-09-16T00:22:57.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.524+0000 7f7bb5daf640 1 -- 172.21.15.33:0/3040149384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bb006a560 con 0x7f7bb0100e20 2024-09-16T00:22:57.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.525+0000 7f7bacff9640 1 -- 172.21.15.33:0/3040149384 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f7ba009c3c0 con 0x7f7bb0100e20 2024-09-16T00:22:57.526 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:22:57.526 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:22:57.527 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:22:57.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.527+0000 7f7bb5daf640 1 -- 172.21.15.33:0/3040149384 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b980a3240 msgr2=0x7f7b980a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:57.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.527+0000 7f7bb5daf640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b980a3240 0x7f7b980a5700 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7bb006aac0 tx=0x7f7ba4005f90 comp rx=0 tx=0).stop 2024-09-16T00:22:57.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.527+0000 7f7bb5daf640 1 -- 172.21.15.33:0/3040149384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bb0100e20 msgr2=0x7f7bb0199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:57.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.527+0000 7f7bb5daf640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bb0100e20 0x7f7bb0199340 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000e9f0 tx=0x7f7ba000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:22:57.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.527+0000 7f7bb5daf640 1 -- 172.21.15.33:0/3040149384 shutdown_connections 2024-09-16T00:22:57.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.527+0000 7f7bb5daf640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b980a3240 0x7f7b980a5700 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:57.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.527+0000 7f7bb5daf640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bb0102ef0 0x7f7bb01a43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:57.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.527+0000 7f7bb5daf640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bb0102570 0x7f7bb0199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:57.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.527+0000 7f7bb5daf640 1 --2- 172.21.15.33:0/3040149384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bb0100e20 0x7f7bb0199340 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:57.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.527+0000 7f7bb5daf640 1 -- 172.21.15.33:0/3040149384 >> 172.21.15.33:0/3040149384 conn(0x7f7bb00fc7d0 msgr2=0x7f7bb0106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:57.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.527+0000 7f7bb5daf640 1 -- 172.21.15.33:0/3040149384 shutdown_connections 2024-09-16T00:22:57.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:57.527+0000 7f7bb5daf640 1 -- 172.21.15.33:0/3040149384 wait complete. 2024-09-16T00:22:57.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 347060 9.85 MB/sec execute 125 sec latency 20.100 ms 2024-09-16T00:22:58.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:57 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3040149384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:58.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:57 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3040149384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:58.241 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:57 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3040149384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:22:58.269 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0' 2024-09-16T00:22:58.269 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0 2024-09-16T00:22:58.593 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:22:58.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 348683 9.84 MB/sec execute 126 sec latency 57.740 ms 2024-09-16T00:22:59.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:22:58 smithi033 ceph-mon[31243]: pgmap v525: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-16T00:22:59.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.127+0000 7fd04bb8c640 1 -- 172.21.15.33:0/1721985570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd044106f10 msgr2=0x7fd04410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:59.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.127+0000 7fd04bb8c640 1 --2- 172.21.15.33:0/1721985570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd044106f10 0x7fd04410ea90 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009ca0 tx=0x7fd03c02a8c0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.127+0000 7fd04bb8c640 1 -- 172.21.15.33:0/1721985570 shutdown_connections 2024-09-16T00:22:59.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.127+0000 7fd04bb8c640 1 --2- 172.21.15.33:0/1721985570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd044106f10 0x7fd04410ea90 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.127+0000 7fd04bb8c640 1 --2- 172.21.15.33:0/1721985570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd044102790 0x7fd0441067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.127+0000 7fd04bb8c640 1 --2- 172.21.15.33:0/1721985570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd044101de0 0x7fd0441021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.127+0000 7fd04bb8c640 1 -- 172.21.15.33:0/1721985570 >> 172.21.15.33:0/1721985570 conn(0x7fd0440fd8c0 msgr2=0x7fd0440ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.128+0000 7fd04bb8c640 1 -- 172.21.15.33:0/1721985570 shutdown_connections 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.128+0000 7fd04bb8c640 1 -- 172.21.15.33:0/1721985570 wait complete. 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.128+0000 7fd04bb8c640 1 Processor -- start 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.128+0000 7fd04bb8c640 1 -- start start 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.128+0000 7fd04bb8c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd044101de0 0x7fd0440744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.128+0000 7fd04bb8c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd044102790 0x7fd044074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.128+0000 7fd04bb8c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd044106f10 0x7fd044074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.128+0000 7fd04bb8c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd044111f70 con 0x7fd044106f10 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.128+0000 7fd04bb8c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd044111df0 con 0x7fd044101de0 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.128+0000 7fd04bb8c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441120f0 con 0x7fd044102790 2024-09-16T00:22:59.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd04a129640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd044106f10 0x7fd044074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd049127640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd044102790 0x7fd044074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd04a129640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd044106f10 0x7fd044074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43308/0 (socket says 172.21.15.33:43308) 2024-09-16T00:22:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd049127640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd044102790 0x7fd044074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52462/0 (socket says 172.21.15.33:52462) 2024-09-16T00:22:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd04a129640 1 -- 172.21.15.33:0/4138168928 learned_addr learned my addr 172.21.15.33:0/4138168928 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd049928640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd044101de0 0x7fd0440744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd049127640 1 -- 172.21.15.33:0/4138168928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd044101de0 msgr2=0x7fd0440744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd049127640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd044101de0 0x7fd0440744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd049127640 1 -- 172.21.15.33:0/4138168928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd044106f10 msgr2=0x7fd044074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd049127640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd044106f10 0x7fd044074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd049127640 1 -- 172.21.15.33:0/4138168928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0441a6890 con 0x7fd044102790 2024-09-16T00:22:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd04a129640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd044106f10 0x7fd044074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:22:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd049127640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd044102790 0x7fd044074a30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd03400e9f0 tx=0x7fd03400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:59.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd03affd640 1 -- 172.21.15.33:0/4138168928 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03400ccc0 con 0x7fd044102790 2024-09-16T00:22:59.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd04bb8c640 1 -- 172.21.15.33:0/4138168928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0441a6b80 con 0x7fd044102790 2024-09-16T00:22:59.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd03affd640 1 -- 172.21.15.33:0/4138168928 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd034010910 con 0x7fd044102790 2024-09-16T00:22:59.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd03affd640 1 -- 172.21.15.33:0/4138168928 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd034018960 con 0x7fd044102790 2024-09-16T00:22:59.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.129+0000 7fd04bb8c640 1 -- 172.21.15.33:0/4138168928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0441a7110 con 0x7fd044102790 2024-09-16T00:22:59.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.132+0000 7fd03affd640 1 -- 172.21.15.33:0/4138168928 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd034010ac0 con 0x7fd044102790 2024-09-16T00:22:59.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.132+0000 7fd04df65740 1 Processor -- start 2024-09-16T00:22:59.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.132+0000 7fd04df65740 1 -- start start 2024-09-16T00:22:59.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.132+0000 7fd04df65740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ab850 0x5638fa3abc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.132+0000 7fd04df65740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ac170 0x5638fa3b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.132+0000 7fd04df65740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3b6a30 0x5638fa3b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.132+0000 7fd04df65740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5638fa310160 con 0x5638fa3ac170 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.132+0000 7fd04df65740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5638fa2b7720 con 0x5638fa3b6a30 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.132+0000 7fd04df65740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5638fa30c120 con 0x5638fa3ab850 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.132+0000 7fd049928640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ac170 0x5638fa3b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.132+0000 7fd04a129640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ab850 0x5638fa3abc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd04a129640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ab850 0x5638fa3abc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52472/0 (socket says 172.21.15.33:52472) 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd04a129640 1 -- 172.21.15.33:0/1481416440 learned_addr learned my addr 172.21.15.33:0/1481416440 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd049928640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ac170 0x5638fa3b64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43316/0 (socket says 172.21.15.33:43316) 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd049127640 1 --2- 172.21.15.33:0/1481416440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3b6a30 0x5638fa3b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd04a129640 1 -- 172.21.15.33:0/1481416440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3b6a30 msgr2=0x5638fa3b8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd04a129640 1 --2- 172.21.15.33:0/1481416440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3b6a30 0x5638fa3b8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd04a129640 1 -- 172.21.15.33:0/1481416440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ac170 msgr2=0x5638fa3b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:59.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd04a129640 1 --2- 172.21.15.33:0/1481416440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ac170 0x5638fa3b64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd03affd640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd0300a3240 0x7fd0300a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd04a129640 1 -- 172.21.15.33:0/1481416440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638fa081ba0 con 0x5638fa3ab850 2024-09-16T00:22:59.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd049127640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd0300a3240 0x7fd0300a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd04a129640 1 --2- 172.21.15.33:0/1481416440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ab850 0x5638fa3abc30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd03c031f80 tx=0x7fd03c005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77cefe198dde4302 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:59.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd03affd640 1 -- 172.21.15.33:0/4138168928 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd0340d0100 con 0x7fd044102790 2024-09-16T00:22:59.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd02ffff640 1 -- 172.21.15.33:0/1481416440 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c003b70 con 0x5638fa3ab850 2024-09-16T00:22:59.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd02ffff640 1 -- 172.21.15.33:0/1481416440 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd03c036040 con 0x5638fa3ab850 2024-09-16T00:22:59.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.133+0000 7fd02ffff640 1 -- 172.21.15.33:0/1481416440 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c0046a0 con 0x5638fa3ab850 2024-09-16T00:22:59.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.134+0000 7fd04df65740 1 -- 172.21.15.33:0/1481416440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ab850 msgr2=0x5638fa3abc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:59.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.134+0000 7fd04df65740 1 --2- 172.21.15.33:0/1481416440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ab850 0x5638fa3abc30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd03c031f80 tx=0x7fd03c005f70 comp rx=0 tx=0).stop 2024-09-16T00:22:59.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.134+0000 7fd049127640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd0300a3240 0x7fd0300a5700 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd03409bff0 tx=0x7fd0340d3b40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.134+0000 7fd04df65740 1 -- 172.21.15.33:0/1481416440 shutdown_connections 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.134+0000 7fd04df65740 1 --2- 172.21.15.33:0/1481416440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3b6a30 0x5638fa3b8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.134+0000 7fd04df65740 1 --2- 172.21.15.33:0/1481416440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ac170 0x5638fa3b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.134+0000 7fd04df65740 1 --2- 172.21.15.33:0/1481416440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ab850 0x5638fa3abc30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.134+0000 7fd04df65740 1 -- 172.21.15.33:0/1481416440 >> 172.21.15.33:0/1481416440 conn(0x5638fa3a7de0 msgr2=0x5638fa3a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.134+0000 7fd04df65740 1 -- 172.21.15.33:0/1481416440 shutdown_connections 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.135+0000 7fd04df65740 1 -- 172.21.15.33:0/1481416440 wait complete. 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.135+0000 7fd04df65740 1 -- start start 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.135+0000 7fd04df65740 1 Processor -- start 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd04df65740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ab850 0x5638fa3b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd04df65740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 0x5638fa3b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd04df65740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3b6a30 0x5638fa3b4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd04df65740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5638fa3b9480 con 0x5638fa3b6a30 2024-09-16T00:22:59.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd04df65740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5638fa298900 con 0x5638fa3ac170 2024-09-16T00:22:59.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd04df65740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5638fa3b9600 con 0x5638fa3ab850 2024-09-16T00:22:59.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd04df65740 10 client.0 _ll_register_callbacks cb 0x5638fa296780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:59.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd049928640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 0x5638fa3b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd049928640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 0x5638fa3b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:50088/0 (socket says 172.21.15.33:50088) 2024-09-16T00:22:59.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd049928640 1 -- 172.21.15.33:0/251554681 learned_addr learned my addr 172.21.15.33:0/251554681 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:59.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd04a129640 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ab850 0x5638fa3b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd04a129640 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3b6a30 0x5638fa3b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd049928640 1 -- 172.21.15.33:0/251554681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ab850 msgr2=0x5638fa3b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:59.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd049928640 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ab850 0x5638fa3b38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd049928640 1 -- 172.21.15.33:0/251554681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3b6a30 msgr2=0x5638fa3b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:59.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd049928640 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3b6a30 0x5638fa3b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd049928640 1 -- 172.21.15.33:0/251554681 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638fa2d8c50 con 0x5638fa3ac170 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd04a129640 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3b6a30 0x5638fa3b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd049928640 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 0x5638fa3b3e20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd040012760 tx=0x7fd040012c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7639faadd80b41d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.136+0000 7fd02d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.137+0000 7fd02d7fa640 1 -- 172.21.15.33:0/251554681 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04000f920 con 0x5638fa3ac170 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.137+0000 7fd04df65740 1 -- 172.21.15.33:0/251554681 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd03c036040 con 0x5638fa3ac170 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.137+0000 7fd02d7fa640 1 -- 172.21.15.33:0/251554681 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd04000fac0 con 0x5638fa3ac170 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.137+0000 7fd02d7fa640 1 -- 172.21.15.33:0/251554681 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd04000fcc0 con 0x5638fa3ac170 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.137+0000 7fd04df65740 10 client.26027 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.137+0000 7fd04df65740 1 -- 172.21.15.33:0/251554681 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5638fa3c51b0 con 0x5638fa3ac170 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.137+0000 7fd02d7fa640 1 -- 172.21.15.33:0/251554681 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd03c036040 con 0x5638fa3ac170 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.138+0000 7fd02d7fa640 1 -- 172.21.15.33:0/251554681 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fd04001c420 con 0x5638fa3ac170 2024-09-16T00:22:59.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.138+0000 7fd04df65740 10 client.26027 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.138+0000 7fd04df65740 10 client.26027 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.138+0000 7fd04df65740 20 client.26027 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.138+0000 7fd04df65740 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638fa3c6710 0x5638fa3c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.138+0000 7fd04df65740 4 client.26027 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.138+0000 7fd04df65740 1 -- 172.21.15.33:0/251554681 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638fa31d890 con 0x5638fa3c6710 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.138+0000 7fd04a129640 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638fa3c6710 0x5638fa3c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.138+0000 7fd04a129640 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638fa3c6710 0x5638fa3c8b30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1bb6003224f038c server_cookie=5ffce122264f2488 in_seq=0 out_seq=0 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.138+0000 7fd02d7fa640 10 client.26027 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd02d7fa640 1 -- 172.21.15.33:0/251554681 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5638fa31d890 con 0x5638fa3c6710 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd02d7fa640 10 client.26027 handle_command_reply: tid=0 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd04df65740 1 client.26027 shutdown 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd04df65740 1 -- 172.21.15.33:0/251554681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 msgr2=0x5638fa3b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:59.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd04df65740 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 0x5638fa3b3e20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd040012760 tx=0x7fd040012c20 comp rx=0 tx=0).stop 2024-09-16T00:22:59.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd04df65740 1 -- 172.21.15.33:0/251554681 shutdown_connections 2024-09-16T00:22:59.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd04df65740 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638fa3c6710 0x5638fa3c8b30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd04df65740 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3b6a30 0x5638fa3b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd04df65740 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 0x5638fa3b3e20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd04df65740 1 --2- 172.21.15.33:0/251554681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ab850 0x5638fa3b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd02d7fa640 0 client.26027 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:59.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd04df65740 1 -- 172.21.15.33:0/251554681 >> 172.21.15.33:0/251554681 conn(0x5638fa3a7de0 msgr2=0x5638fa3adb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:59.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd04df65740 1 -- 172.21.15.33:0/251554681 shutdown_connections 2024-09-16T00:22:59.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd04df65740 1 -- 172.21.15.33:0/251554681 wait complete. 2024-09-16T00:22:59.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.139+0000 7fd04df65740 20 client.26027 trim_cache size 0 max 16384 2024-09-16T00:22:59.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd04df65740 1 Processor -- start 2024-09-16T00:22:59.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd04df65740 1 -- start start 2024-09-16T00:22:59.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd04df65740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 0x5638fa3aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd04df65740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 0x5638fa3b5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd04df65740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3b6a30 0x5638fa3ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd04df65740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441b3ae0 con 0x5638fa3ab850 2024-09-16T00:22:59.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd04df65740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441b13b0 con 0x5638fa3ac170 2024-09-16T00:22:59.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd04df65740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441af200 con 0x5638fa3b6a30 2024-09-16T00:22:59.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd04a129640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 0x5638fa3aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd049928640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 0x5638fa3b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd04a129640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 0x5638fa3aec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43332/0 (socket says 172.21.15.33:43332) 2024-09-16T00:22:59.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd04a129640 1 -- 172.21.15.33:0/810799358 learned_addr learned my addr 172.21.15.33:0/810799358 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:59.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd049928640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 0x5638fa3b5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:50098/0 (socket says 172.21.15.33:50098) 2024-09-16T00:22:59.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.157+0000 7fd04a129640 1 --2- 172.21.15.33:0/810799358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3b6a30 0x5638fa3ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.158+0000 7fd049928640 1 -- 172.21.15.33:0/810799358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3b6a30 msgr2=0x5638fa3ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:59.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.158+0000 7fd049928640 1 --2- 172.21.15.33:0/810799358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3b6a30 0x5638fa3ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.158+0000 7fd049928640 1 -- 172.21.15.33:0/810799358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 msgr2=0x5638fa3aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:59.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.158+0000 7fd049928640 1 --2- 172.21.15.33:0/810799358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 0x5638fa3aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.158+0000 7fd049928640 1 -- 172.21.15.33:0/810799358 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638fa3b2890 con 0x5638fa3ac170 2024-09-16T00:22:59.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.158+0000 7fd04a129640 1 --2- 172.21.15.33:0/810799358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3b6a30 0x5638fa3ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:22:59.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.158+0000 7fd049928640 1 --2- 172.21.15.33:0/810799358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 0x5638fa3b5850 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd04001c890 tx=0x7fd04000f810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44cc2a83dcc6e1ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:59.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.158+0000 7fd02f7fe640 1 -- 172.21.15.33:0/810799358 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd040009da0 con 0x5638fa3ac170 2024-09-16T00:22:59.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.158+0000 7fd02f7fe640 1 -- 172.21.15.33:0/810799358 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0400055c0 con 0x5638fa3ac170 2024-09-16T00:22:59.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.158+0000 7fd02f7fe640 1 -- 172.21.15.33:0/810799358 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04001c660 con 0x5638fa3ac170 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.159+0000 7fd04df65740 1 -- 172.21.15.33:0/810799358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 msgr2=0x5638fa3b5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.159+0000 7fd04df65740 1 --2- 172.21.15.33:0/810799358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 0x5638fa3b5850 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd04001c890 tx=0x7fd04000f810 comp rx=0 tx=0).stop 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.159+0000 7fd04df65740 1 -- 172.21.15.33:0/810799358 shutdown_connections 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.159+0000 7fd04df65740 1 --2- 172.21.15.33:0/810799358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3b6a30 0x5638fa3ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.159+0000 7fd04df65740 1 --2- 172.21.15.33:0/810799358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3ac170 0x5638fa3b5850 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.159+0000 7fd04df65740 1 --2- 172.21.15.33:0/810799358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 0x5638fa3aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.159+0000 7fd04df65740 1 -- 172.21.15.33:0/810799358 >> 172.21.15.33:0/810799358 conn(0x5638fa3a7de0 msgr2=0x5638fa3b5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.159+0000 7fd04df65740 1 -- 172.21.15.33:0/810799358 shutdown_connections 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.159+0000 7fd04df65740 1 -- 172.21.15.33:0/810799358 wait complete. 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.159+0000 7fd04df65740 1 -- start start 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.159+0000 7fd04df65740 1 Processor -- start 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04df65740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 0x5638fa3bea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04df65740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ac170 0x5638fa3bef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04df65740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3b6a30 0x5638fa3bb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04df65740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441af870 con 0x5638fa3ab850 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04df65740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441b0ce0 con 0x5638fa3b6a30 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04df65740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441aff40 con 0x5638fa3ac170 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04df65740 10 client.0 _ll_register_callbacks cb 0x7fd040009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04a129640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 0x5638fa3bea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04a129640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 0x5638fa3bea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43346/0 (socket says 172.21.15.33:43346) 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04a129640 1 -- 172.21.15.33:0/722989445 learned_addr learned my addr 172.21.15.33:0/722989445 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd049928640 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ac170 0x5638fa3bef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04a129640 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3b6a30 0x5638fa3bb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04a129640 1 -- 172.21.15.33:0/722989445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ac170 msgr2=0x5638fa3bef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04a129640 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ac170 0x5638fa3bef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04a129640 1 -- 172.21.15.33:0/722989445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3b6a30 msgr2=0x5638fa3bb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04a129640 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3b6a30 0x5638fa3bb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04a129640 1 -- 172.21.15.33:0/722989445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638fa3bc050 con 0x5638fa3ab850 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd04a129640 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 0x5638fa3bea30 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd03c02f470 tx=0x7fd03c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8169641e56b7f07 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd02d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.160+0000 7fd02d7fa640 1 -- 172.21.15.33:0/722989445 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c003970 con 0x5638fa3ab850 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd02d7fa640 1 -- 172.21.15.33:0/722989445 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd03c003b10 con 0x5638fa3ab850 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd04df65740 1 -- 172.21.15.33:0/722989445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0400055c0 con 0x5638fa3ab850 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd02d7fa640 1 -- 172.21.15.33:0/722989445 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd03c003d10 con 0x5638fa3ab850 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd04df65740 10 client.16500 fetch_fsmap learned FSMap version 30 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd04df65740 1 -- 172.21.15.33:0/722989445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5638fa431ea0 con 0x5638fa3ab850 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd02d7fa640 1 -- 172.21.15.33:0/722989445 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0400055c0 con 0x5638fa3ab850 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd02d7fa640 1 -- 172.21.15.33:0/722989445 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fd03c043070 con 0x5638fa3ab850 2024-09-16T00:22:59.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd04df65740 10 client.16500 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:22:59.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd04df65740 10 client.16500 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:22:59.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd04df65740 20 client.16500 populate_metadata read hostname 'smithi033' 2024-09-16T00:22:59.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd04df65740 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638fa432300 0x5638fa4346f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:22:59.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd04df65740 4 client.16500 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}] 2024-09-16T00:22:59.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd04df65740 1 -- 172.21.15.33:0/722989445 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}) -- 0x5638fa317be0 con 0x5638fa432300 2024-09-16T00:22:59.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.161+0000 7fd049928640 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638fa432300 0x5638fa4346f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:22:59.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.162+0000 7fd049928640 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638fa432300 0x5638fa4346f0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83e83fbdf9c2f9c5 server_cookie=ee3e43edbac5f1aa in_seq=0 out_seq=0 2024-09-16T00:22:59.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.162+0000 7fd02d7fa640 10 client.16500 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:59.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:22:58 smithi083 ceph-mon[35242]: pgmap v525: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-16T00:22:59.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:22:58 smithi060 ceph-mon[35185]: pgmap v525: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-16T00:22:59.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.247+0000 7fd02d7fa640 1 -- 172.21.15.33:0/722989445 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5638fa317be0 con 0x5638fa432300 2024-09-16T00:22:59.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.247+0000 7fd02d7fa640 10 client.16500 handle_command_reply: tid=0 2024-09-16T00:22:59.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.247+0000 7fd04df65740 1 client.16500 shutdown 2024-09-16T00:22:59.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04df65740 1 -- 172.21.15.33:0/722989445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 msgr2=0x5638fa3bea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:59.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04df65740 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 0x5638fa3bea30 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd03c02f470 tx=0x7fd03c02a8c0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04df65740 1 -- 172.21.15.33:0/722989445 shutdown_connections 2024-09-16T00:22:59.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04df65740 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638fa432300 0x5638fa4346f0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04df65740 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638fa3b6a30 0x5638fa3bb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd02d7fa640 0 client.16500 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:22:59.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04df65740 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638fa3ac170 0x5638fa3bef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04df65740 1 --2- 172.21.15.33:0/722989445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638fa3ab850 0x5638fa3bea30 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04df65740 1 -- 172.21.15.33:0/722989445 >> 172.21.15.33:0/722989445 conn(0x5638fa3a7de0 msgr2=0x5638fa3adac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:59.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04df65740 1 -- 172.21.15.33:0/722989445 shutdown_connections 2024-09-16T00:22:59.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04df65740 1 -- 172.21.15.33:0/722989445 wait complete. 2024-09-16T00:22:59.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04df65740 20 client.16500 trim_cache size 0 max 16384 2024-09-16T00:22:59.251 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:22:59.251 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:22:59.251 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:22:59.251 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 22, 2024-09-16T00:22:59.251 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: "fbb869a9": { 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 22, 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 145.3, 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:22:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:22:59.253 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:22:59.253 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:22:59.253 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:59.253 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:59.253 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:59.253 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:59.253 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:22:59.253 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:22:59.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04bb8c640 1 -- 172.21.15.33:0/4138168928 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd0300a3240 msgr2=0x7fd0300a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:59.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04bb8c640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd0300a3240 0x7fd0300a5700 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd03409bff0 tx=0x7fd0340d3b40 comp rx=0 tx=0).stop 2024-09-16T00:22:59.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04bb8c640 1 -- 172.21.15.33:0/4138168928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd044102790 msgr2=0x7fd044074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:22:59.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04bb8c640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd044102790 0x7fd044074a30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd03400e9f0 tx=0x7fd03400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04bb8c640 1 -- 172.21.15.33:0/4138168928 shutdown_connections 2024-09-16T00:22:59.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.248+0000 7fd04bb8c640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd0300a3240 0x7fd0300a5700 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.249+0000 7fd04bb8c640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd044106f10 0x7fd044074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.249+0000 7fd04bb8c640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd044102790 0x7fd044074a30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.249+0000 7fd04bb8c640 1 --2- 172.21.15.33:0/4138168928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd044101de0 0x7fd0440744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:22:59.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.249+0000 7fd04bb8c640 1 -- 172.21.15.33:0/4138168928 >> 172.21.15.33:0/4138168928 conn(0x7fd0440fd8c0 msgr2=0x7fd0440feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:22:59.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.249+0000 7fd04bb8c640 1 -- 172.21.15.33:0/4138168928 shutdown_connections 2024-09-16T00:22:59.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:22:59.249+0000 7fd04bb8c640 1 -- 172.21.15.33:0/4138168928 wait complete. 2024-09-16T00:22:59.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 9.81 MB/sec execute 127 sec latency 491.249 ms 2024-09-16T00:23:00.089 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'fbb869a9', quiesce took 4.9 seconds. Will release after: 20.450000000000003 2024-09-16T00:23:00.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 9.73 MB/sec execute 128 sec latency 1491.366 ms 2024-09-16T00:23:01.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:00 smithi083 ceph-mon[35242]: pgmap v526: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-09-16T00:23:01.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:00 smithi060 ceph-mon[35185]: pgmap v526: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-09-16T00:23:01.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:00 smithi033 ceph-mon[31243]: pgmap v526: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-09-16T00:23:01.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 9.66 MB/sec execute 129 sec latency 2491.490 ms 2024-09-16T00:23:02.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:01 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:23:02.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:01 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:23:02.231 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:01 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:23:02.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 9.58 MB/sec execute 130 sec latency 3491.576 ms 2024-09-16T00:23:03.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:02 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:23:03.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:02 smithi083 ceph-mon[35242]: pgmap v527: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-09-16T00:23:03.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:02 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:23:03.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:02 smithi060 ceph-mon[35185]: pgmap v527: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-09-16T00:23:03.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:02 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:23:03.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:02 smithi033 ceph-mon[31243]: pgmap v527: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-09-16T00:23:03.660 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 9.51 MB/sec execute 131 sec latency 4491.677 ms 2024-09-16T00:23:04.661 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 9.44 MB/sec execute 132 sec latency 5491.728 ms 2024-09-16T00:23:05.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:04 smithi083 ceph-mon[35242]: pgmap v528: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 174 op/s 2024-09-16T00:23:05.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:04 smithi060 ceph-mon[35185]: pgmap v528: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 174 op/s 2024-09-16T00:23:05.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:04 smithi033 ceph-mon[31243]: pgmap v528: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 174 op/s 2024-09-16T00:23:05.661 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 9.37 MB/sec execute 133 sec latency 6491.818 ms 2024-09-16T00:23:06.661 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 9.30 MB/sec execute 134 sec latency 7491.986 ms 2024-09-16T00:23:07.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:06 smithi083 ceph-mon[35242]: pgmap v529: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 162 op/s 2024-09-16T00:23:07.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:06 smithi060 ceph-mon[35185]: pgmap v529: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 162 op/s 2024-09-16T00:23:07.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:06 smithi033 ceph-mon[31243]: pgmap v529: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 162 op/s 2024-09-16T00:23:07.661 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 9.23 MB/sec execute 135 sec latency 8492.099 ms 2024-09-16T00:23:08.661 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 9.16 MB/sec execute 136 sec latency 9492.209 ms 2024-09-16T00:23:09.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:08 smithi083 ceph-mon[35242]: pgmap v530: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 133 op/s 2024-09-16T00:23:09.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:08 smithi060 ceph-mon[35185]: pgmap v530: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 133 op/s 2024-09-16T00:23:09.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:08 smithi033 ceph-mon[31243]: pgmap v530: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 133 op/s 2024-09-16T00:23:09.661 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 9.09 MB/sec execute 137 sec latency 10492.283 ms 2024-09-16T00:23:10.661 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 9.03 MB/sec execute 138 sec latency 11492.389 ms 2024-09-16T00:23:11.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:10 smithi083 ceph-mon[35242]: pgmap v531: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 127 op/s 2024-09-16T00:23:11.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:10 smithi060 ceph-mon[35185]: pgmap v531: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 127 op/s 2024-09-16T00:23:11.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:10 smithi033 ceph-mon[31243]: pgmap v531: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 127 op/s 2024-09-16T00:23:11.661 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 8.96 MB/sec execute 139 sec latency 12492.509 ms 2024-09-16T00:23:12.661 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 8.90 MB/sec execute 140 sec latency 13492.562 ms 2024-09-16T00:23:13.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:12 smithi083 ceph-mon[35242]: pgmap v532: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 889 KiB/s wr, 56 op/s 2024-09-16T00:23:13.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:12 smithi060 ceph-mon[35185]: pgmap v532: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 889 KiB/s wr, 56 op/s 2024-09-16T00:23:13.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:12 smithi033 ceph-mon[31243]: pgmap v532: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 889 KiB/s wr, 56 op/s 2024-09-16T00:23:13.661 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 8.84 MB/sec execute 141 sec latency 14492.664 ms 2024-09-16T00:23:14.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:13 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:23:14.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:13 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:23:14.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:13 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:23:14.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 8.77 MB/sec execute 142 sec latency 15492.769 ms 2024-09-16T00:23:15.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:14 smithi060 ceph-mon[35185]: pgmap v533: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 434 KiB/s wr, 34 op/s 2024-09-16T00:23:15.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:23:15.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:23:15.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:14 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:23:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:14 smithi033 ceph-mon[31243]: pgmap v533: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 434 KiB/s wr, 34 op/s 2024-09-16T00:23:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:23:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:23:15.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:14 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:23:15.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:14 smithi083 ceph-mon[35242]: pgmap v533: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 434 KiB/s wr, 34 op/s 2024-09-16T00:23:15.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:23:15.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:23:15.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:14 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:23:15.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 8.71 MB/sec execute 143 sec latency 16492.897 ms 2024-09-16T00:23:16.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 8.65 MB/sec execute 144 sec latency 17492.969 ms 2024-09-16T00:23:17.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:16 smithi060 ceph-mon[35185]: pgmap v534: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-09-16T00:23:17.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:16 smithi033 ceph-mon[31243]: pgmap v534: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-09-16T00:23:17.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:16 smithi083 ceph-mon[35242]: pgmap v534: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-09-16T00:23:17.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 8.59 MB/sec execute 145 sec latency 18493.081 ms 2024-09-16T00:23:18.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 8.53 MB/sec execute 146 sec latency 19493.198 ms 2024-09-16T00:23:19.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:18 smithi060 ceph-mon[35185]: pgmap v535: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-09-16T00:23:19.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:18 smithi033 ceph-mon[31243]: pgmap v535: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-09-16T00:23:19.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:18 smithi083 ceph-mon[35242]: pgmap v535: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-09-16T00:23:19.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 8.47 MB/sec execute 147 sec latency 20493.307 ms 2024-09-16T00:23:20.539 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:23:20.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 8.42 MB/sec execute 148 sec latency 21493.360 ms 2024-09-16T00:23:20.813 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:23:21.223 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:20 smithi033 ceph-mon[31243]: pgmap v536: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-16T00:23:21.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:20 smithi060 ceph-mon[35185]: pgmap v536: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-16T00:23:21.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.307+0000 7fc855492640 1 -- 172.21.15.33:0/2244439384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850100a80 msgr2=0x7fc850100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:21.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.307+0000 7fc855492640 1 --2- 172.21.15.33:0/2244439384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850100a80 0x7fc850100ee0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc844009930 tx=0x7fc84402a420 comp rx=0 tx=0).stop 2024-09-16T00:23:21.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.307+0000 7fc855492640 1 -- 172.21.15.33:0/2244439384 shutdown_connections 2024-09-16T00:23:21.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.307+0000 7fc855492640 1 --2- 172.21.15.33:0/2244439384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc850101420 0x7fc85010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:21.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.307+0000 7fc855492640 1 --2- 172.21.15.33:0/2244439384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850100a80 0x7fc850100ee0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:21.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.307+0000 7fc855492640 1 --2- 172.21.15.33:0/2244439384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc850106a80 0x7fc850106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:21.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.307+0000 7fc855492640 1 -- 172.21.15.33:0/2244439384 >> 172.21.15.33:0/2244439384 conn(0x7fc8500fc7a0 msgr2=0x7fc8500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:21.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.307+0000 7fc855492640 1 -- 172.21.15.33:0/2244439384 shutdown_connections 2024-09-16T00:23:21.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.307+0000 7fc855492640 1 -- 172.21.15.33:0/2244439384 wait complete. 2024-09-16T00:23:21.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.308+0000 7fc855492640 1 Processor -- start 2024-09-16T00:23:21.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.308+0000 7fc855492640 1 -- start start 2024-09-16T00:23:21.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.308+0000 7fc855492640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc850100a80 0x7fc8501984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:21.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.308+0000 7fc855492640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc850101420 0x7fc850198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:21.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.308+0000 7fc855492640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850106a80 0x7fc85019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:21.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.308+0000 7fc855492640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc850110ea0 con 0x7fc850106a80 2024-09-16T00:23:21.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.308+0000 7fc855492640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc850110d20 con 0x7fc850100a80 2024-09-16T00:23:21.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.308+0000 7fc855492640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc850111020 con 0x7fc850101420 2024-09-16T00:23:21.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc84f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850106a80 0x7fc85019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:21.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc84f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850106a80 0x7fc85019cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36068/0 (socket says 172.21.15.33:36068) 2024-09-16T00:23:21.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc84f7fe640 1 -- 172.21.15.33:0/678749565 learned_addr learned my addr 172.21.15.33:0/678749565 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:21.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc84effd640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc850100a80 0x7fc8501984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:21.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc84e7fc640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc850101420 0x7fc850198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:21.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc84f7fe640 1 -- 172.21.15.33:0/678749565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc850101420 msgr2=0x7fc850198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:21.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc84f7fe640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc850101420 0x7fc850198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:21.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc84f7fe640 1 -- 172.21.15.33:0/678749565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc850100a80 msgr2=0x7fc8501984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:21.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc84f7fe640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc850100a80 0x7fc8501984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:21.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc84f7fe640 1 -- 172.21.15.33:0/678749565 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc85019d480 con 0x7fc850106a80 2024-09-16T00:23:21.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc84e7fc640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc850101420 0x7fc850198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:23:21.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc84f7fe640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850106a80 0x7fc85019cda0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc84800c970 tx=0x7fc84800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:21.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc837fff640 1 -- 172.21.15.33:0/678749565 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc848007bb0 con 0x7fc850106a80 2024-09-16T00:23:21.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc837fff640 1 -- 172.21.15.33:0/678749565 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc848007d50 con 0x7fc850106a80 2024-09-16T00:23:21.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc855492640 1 -- 172.21.15.33:0/678749565 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc85019d770 con 0x7fc850106a80 2024-09-16T00:23:21.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc837fff640 1 -- 172.21.15.33:0/678749565 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84800f450 con 0x7fc850106a80 2024-09-16T00:23:21.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.309+0000 7fc855492640 1 -- 172.21.15.33:0/678749565 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8501a5050 con 0x7fc850106a80 2024-09-16T00:23:21.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.310+0000 7fc855492640 1 -- 172.21.15.33:0/678749565 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc850100ee0 con 0x7fc850106a80 2024-09-16T00:23:21.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.311+0000 7fc837fff640 1 -- 172.21.15.33:0/678749565 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc848004f60 con 0x7fc850106a80 2024-09-16T00:23:21.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.312+0000 7fc837fff640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc8380a3240 0x7fc8380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:21.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.312+0000 7fc84effd640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc8380a3240 0x7fc8380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:21.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.312+0000 7fc837fff640 1 -- 172.21.15.33:0/678749565 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc8480cfb60 con 0x7fc850106a80 2024-09-16T00:23:21.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.313+0000 7fc84effd640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc8380a3240 0x7fc8380a5700 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc84000a400 tx=0x7fc84000d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:21.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.316+0000 7fc837fff640 1 -- 172.21.15.33:0/678749565 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc848014030 con 0x7fc850106a80 2024-09-16T00:23:21.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:20 smithi083 ceph-mon[35242]: pgmap v536: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-16T00:23:21.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.526+0000 7fc855492640 1 -- 172.21.15.33:0/678749565 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc85002d7c0 con 0x7fc850106a80 2024-09-16T00:23:21.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.527+0000 7fc837fff640 1 -- 172.21.15.33:0/678749565 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7fc84809c3a0 con 0x7fc850106a80 2024-09-16T00:23:21.528 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:23:21.528 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:23:21.528 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:23:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.529+0000 7fc855492640 1 -- 172.21.15.33:0/678749565 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc8380a3240 msgr2=0x7fc8380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.529+0000 7fc855492640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc8380a3240 0x7fc8380a5700 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc84000a400 tx=0x7fc84000d040 comp rx=0 tx=0).stop 2024-09-16T00:23:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.529+0000 7fc855492640 1 -- 172.21.15.33:0/678749565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850106a80 msgr2=0x7fc85019cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.529+0000 7fc855492640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850106a80 0x7fc85019cda0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc84800c970 tx=0x7fc84800ce30 comp rx=0 tx=0).stop 2024-09-16T00:23:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.529+0000 7fc855492640 1 -- 172.21.15.33:0/678749565 shutdown_connections 2024-09-16T00:23:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.529+0000 7fc855492640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc8380a3240 0x7fc8380a5700 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.529+0000 7fc855492640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850106a80 0x7fc85019cda0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.529+0000 7fc855492640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc850101420 0x7fc850198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.529+0000 7fc855492640 1 --2- 172.21.15.33:0/678749565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc850100a80 0x7fc8501984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.529+0000 7fc855492640 1 -- 172.21.15.33:0/678749565 >> 172.21.15.33:0/678749565 conn(0x7fc8500fc7a0 msgr2=0x7fc85006aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:21.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.530+0000 7fc855492640 1 -- 172.21.15.33:0/678749565 shutdown_connections 2024-09-16T00:23:21.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:21.530+0000 7fc855492640 1 -- 172.21.15.33:0/678749565 wait complete. 2024-09-16T00:23:21.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 8.36 MB/sec execute 149 sec latency 22493.454 ms 2024-09-16T00:23:22.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:21 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/678749565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:23:22.320 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:21 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/678749565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:23:22.351 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id fbb869a9 --release --await' 2024-09-16T00:23:22.351 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id fbb869a9 --release --await 2024-09-16T00:23:22.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:21 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/678749565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:23:22.611 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:23:22.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 350206 8.31 MB/sec execute 150 sec latency 23493.538 ms 2024-09-16T00:23:22.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:22 smithi033 ceph-mon[31243]: pgmap v537: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 8 op/s 2024-09-16T00:23:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.079+0000 7ff92eab3640 1 -- 172.21.15.33:0/3318678211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff92810c180 msgr2=0x7ff92810e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.079+0000 7ff92eab3640 1 --2- 172.21.15.33:0/3318678211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff92810c180 0x7ff92810e600 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff920009ca0 tx=0x7ff92002a950 comp rx=0 tx=0).stop 2024-09-16T00:23:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.079+0000 7ff92eab3640 1 -- 172.21.15.33:0/3318678211 shutdown_connections 2024-09-16T00:23:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.079+0000 7ff92eab3640 1 --2- 172.21.15.33:0/3318678211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff92810c180 0x7ff92810e600 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.079+0000 7ff92eab3640 1 --2- 172.21.15.33:0/3318678211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9281036c0 0x7ff92810bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.079+0000 7ff92eab3640 1 --2- 172.21.15.33:0/3318678211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff928102da0 0x7ff928103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.079+0000 7ff92eab3640 1 -- 172.21.15.33:0/3318678211 >> 172.21.15.33:0/3318678211 conn(0x7ff9280fc7a0 msgr2=0x7ff9280febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.079+0000 7ff92eab3640 1 -- 172.21.15.33:0/3318678211 shutdown_connections 2024-09-16T00:23:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.079+0000 7ff92eab3640 1 -- 172.21.15.33:0/3318678211 wait complete. 2024-09-16T00:23:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.080+0000 7ff92eab3640 1 Processor -- start 2024-09-16T00:23:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.080+0000 7ff92eab3640 1 -- start start 2024-09-16T00:23:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.080+0000 7ff92eab3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff928102da0 0x7ff928100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.080+0000 7ff92eab3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9281036c0 0x7ff928100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.080+0000 7ff92eab3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff92810c180 0x7ff928101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.080+0000 7ff92eab3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff928111f60 con 0x7ff9281036c0 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.080+0000 7ff92eab3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff928111de0 con 0x7ff928102da0 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.080+0000 7ff92eab3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281120e0 con 0x7ff92810c180 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff927fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9281036c0 0x7ff928100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff927fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9281036c0 0x7ff928100f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36084/0 (socket says 172.21.15.33:36084) 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff927fff640 1 -- 172.21.15.33:0/1880527153 learned_addr learned my addr 172.21.15.33:0/1880527153 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff92d050640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff92810c180 0x7ff928101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff92c84f640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff928102da0 0x7ff928100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff927fff640 1 -- 172.21.15.33:0/1880527153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff92810c180 msgr2=0x7ff928101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff927fff640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff92810c180 0x7ff928101480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff927fff640 1 -- 172.21.15.33:0/1880527153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff928102da0 msgr2=0x7ff928100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff927fff640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff928102da0 0x7ff928100a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff927fff640 1 -- 172.21.15.33:0/1880527153 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9281a68a0 con 0x7ff9281036c0 2024-09-16T00:23:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff927fff640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9281036c0 0x7ff928100f40 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff91400d950 tx=0x7ff91400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff925ffb640 1 -- 172.21.15.33:0/1880527153 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff914017030 con 0x7ff9281036c0 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff925ffb640 1 -- 172.21.15.33:0/1880527153 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff914004020 con 0x7ff9281036c0 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff92eab3640 1 -- 172.21.15.33:0/1880527153 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9281a6b90 con 0x7ff9281036c0 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.081+0000 7ff925ffb640 1 -- 172.21.15.33:0/1880527153 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9140047a0 con 0x7ff9281036c0 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.082+0000 7ff92eab3640 1 -- 172.21.15.33:0/1880527153 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9281a70d0 con 0x7ff9281036c0 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.084+0000 7ff925ffb640 1 -- 172.21.15.33:0/1880527153 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff9140041d0 con 0x7ff9281036c0 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff930e8c740 1 Processor -- start 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff930e8c740 1 -- start start 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff930e8c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd3d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff930e8c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd3d980 0x55a08dd47d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff930e8c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd48240 0x55a08dd4a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff930e8c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a08dca2620 con 0x55a08dd3d060 2024-09-16T00:23:23.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff930e8c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a08dc49e50 con 0x55a08dd3d980 2024-09-16T00:23:23.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff930e8c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a08dc9e5e0 con 0x55a08dd48240 2024-09-16T00:23:23.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff92d050640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd3d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff92d050640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd3d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36096/0 (socket says 172.21.15.33:36096) 2024-09-16T00:23:23.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff92d050640 1 -- 172.21.15.33:0/1242326673 learned_addr learned my addr 172.21.15.33:0/1242326673 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:23.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff927fff640 1 --2- 172.21.15.33:0/1242326673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd48240 0x55a08dd4a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff925ffb640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9180a3240 0x7ff9180a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff92c84f640 1 --2- 172.21.15.33:0/1242326673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd3d980 0x55a08dd47d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff92d050640 1 -- 172.21.15.33:0/1242326673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd48240 msgr2=0x55a08dd4a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:23.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff92d050640 1 --2- 172.21.15.33:0/1242326673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd48240 0x55a08dd4a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff92d050640 1 -- 172.21.15.33:0/1242326673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd3d980 msgr2=0x55a08dd47d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:23.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff92d050640 1 --2- 172.21.15.33:0/1242326673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd3d980 0x55a08dd47d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.085+0000 7ff92d050640 1 -- 172.21.15.33:0/1242326673 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a08dc3ad40 con 0x55a08dd3d060 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.086+0000 7ff925ffb640 1 -- 172.21.15.33:0/1880527153 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff9140cf630 con 0x7ff9281036c0 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.086+0000 7ff92d050640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9180a3240 0x7ff9180a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.086+0000 7ff92d050640 1 --2- 172.21.15.33:0/1242326673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd3d440 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff920009ca0 tx=0x7ff9200098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f937e447a40e586 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.086+0000 7ff912ffd640 1 -- 172.21.15.33:0/1242326673 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920034040 con 0x55a08dd3d060 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.086+0000 7ff912ffd640 1 -- 172.21.15.33:0/1242326673 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff920003c30 con 0x55a08dd3d060 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.086+0000 7ff912ffd640 1 -- 172.21.15.33:0/1242326673 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920046750 con 0x55a08dd3d060 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.086+0000 7ff92d050640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9180a3240 0x7ff9180a5700 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff92004d000 tx=0x7ff92004a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.087+0000 7ff930e8c740 1 -- 172.21.15.33:0/1242326673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 msgr2=0x55a08dd3d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.087+0000 7ff930e8c740 1 --2- 172.21.15.33:0/1242326673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd3d440 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff920009ca0 tx=0x7ff9200098f0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.087+0000 7ff930e8c740 1 -- 172.21.15.33:0/1242326673 shutdown_connections 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.087+0000 7ff930e8c740 1 --2- 172.21.15.33:0/1242326673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd48240 0x55a08dd4a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.087+0000 7ff930e8c740 1 --2- 172.21.15.33:0/1242326673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd3d980 0x55a08dd47d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.087+0000 7ff930e8c740 1 --2- 172.21.15.33:0/1242326673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd3d440 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.087+0000 7ff930e8c740 1 -- 172.21.15.33:0/1242326673 >> 172.21.15.33:0/1242326673 conn(0x55a08dd395f0 msgr2=0x55a08dd399d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.087+0000 7ff930e8c740 1 -- 172.21.15.33:0/1242326673 shutdown_connections 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.087+0000 7ff930e8c740 1 -- 172.21.15.33:0/1242326673 wait complete. 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.087+0000 7ff930e8c740 1 -- start start 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.087+0000 7ff930e8c740 1 Processor -- start 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff930e8c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd450d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff930e8c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 0x55a08dd45610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff930e8c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd48240 0x55a08dd45b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff930e8c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a08dd4ae10 con 0x55a08dd3d060 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff930e8c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a08dc2af10 con 0x55a08dd48240 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff930e8c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a08dd4af90 con 0x55a08dd3d980 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff930e8c740 10 client.0 _ll_register_callbacks cb 0x55a08dc4edb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:23.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff92c84f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff92c84f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd450d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36112/0 (socket says 172.21.15.33:36112) 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff92c84f640 1 -- 172.21.15.33:0/2306093015 learned_addr learned my addr 172.21.15.33:0/2306093015 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff927fff640 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 0x55a08dd45610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff92d050640 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd48240 0x55a08dd45b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff92c84f640 1 -- 172.21.15.33:0/2306093015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 msgr2=0x55a08dd45610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff92c84f640 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 0x55a08dd45610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff92c84f640 1 -- 172.21.15.33:0/2306093015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd48240 msgr2=0x55a08dd45b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff92c84f640 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd48240 0x55a08dd45b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff92c84f640 1 -- 172.21.15.33:0/2306093015 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a08d9b1580 con 0x55a08dd3d060 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff92c84f640 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd450d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff91c00b920 tx=0x7ff91c00bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b60e10ddd5d86e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.088+0000 7ff907fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff907fff640 1 -- 172.21.15.33:0/2306093015 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c01b440 con 0x55a08dd3d060 2024-09-16T00:23:23.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff907fff640 1 -- 172.21.15.33:0/2306093015 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff91c01b5e0 con 0x55a08dd3d060 2024-09-16T00:23:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff930e8c740 1 -- 172.21.15.33:0/2306093015 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff920003c30 con 0x55a08dd3d060 2024-09-16T00:23:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff907fff640 1 -- 172.21.15.33:0/2306093015 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff91c01b7e0 con 0x55a08dd3d060 2024-09-16T00:23:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff930e8c740 10 client.16518 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff930e8c740 1 -- 172.21.15.33:0/2306093015 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a08da7b960 con 0x55a08dd3d060 2024-09-16T00:23:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff907fff640 1 -- 172.21.15.33:0/2306093015 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff920003c30 con 0x55a08dd3d060 2024-09-16T00:23:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff907fff640 1 -- 172.21.15.33:0/2306093015 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff91c019020 con 0x55a08dd3d060 2024-09-16T00:23:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff930e8c740 10 client.16518 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff930e8c740 10 client.16518 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:23:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff930e8c740 20 client.16518 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff930e8c740 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08dd57da0 0x55a08dd5a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff930e8c740 4 client.16518 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff930e8c740 1 -- 172.21.15.33:0/2306093015 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a08dcafb90 con 0x55a08dd57da0 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.089+0000 7ff927fff640 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08dd57da0 0x55a08dd5a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.090+0000 7ff927fff640 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08dd57da0 0x55a08dd5a1b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1435e6ceecb0fb38 server_cookie=2f286eadd253bbf8 in_seq=0 out_seq=0 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.090+0000 7ff907fff640 10 client.16518 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.090+0000 7ff907fff640 1 -- 172.21.15.33:0/2306093015 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a08dcafb90 con 0x55a08dd57da0 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff907fff640 10 client.16518 handle_command_reply: tid=0 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff930e8c740 1 client.16518 shutdown 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff930e8c740 1 -- 172.21.15.33:0/2306093015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 msgr2=0x55a08dd450d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff930e8c740 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd450d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff91c00b920 tx=0x7ff91c00bd20 comp rx=0 tx=0).stop 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff930e8c740 1 -- 172.21.15.33:0/2306093015 shutdown_connections 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff930e8c740 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08dd57da0 0x55a08dd5a1b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff930e8c740 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd48240 0x55a08dd45b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff930e8c740 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 0x55a08dd45610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff930e8c740 1 --2- 172.21.15.33:0/2306093015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd450d0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff907fff640 0 client.16518 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff930e8c740 1 -- 172.21.15.33:0/2306093015 >> 172.21.15.33:0/2306093015 conn(0x55a08dd395f0 msgr2=0x55a08dd38750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff930e8c740 1 -- 172.21.15.33:0/2306093015 shutdown_connections 2024-09-16T00:23:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff930e8c740 1 -- 172.21.15.33:0/2306093015 wait complete. 2024-09-16T00:23:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.091+0000 7ff930e8c740 20 client.16518 trim_cache size 0 max 16384 2024-09-16T00:23:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.108+0000 7ff930e8c740 1 Processor -- start 2024-09-16T00:23:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.108+0000 7ff930e8c740 1 -- start start 2024-09-16T00:23:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.108+0000 7ff930e8c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd3fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.108+0000 7ff930e8c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 0x55a08dd38640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.108+0000 7ff930e8c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd48240 0x55a08dd5fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.108+0000 7ff930e8c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281b3bd0 con 0x55a08dd3d060 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.108+0000 7ff930e8c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281b14a0 con 0x55a08dd48240 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.108+0000 7ff930e8c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281af2f0 con 0x55a08dd3d980 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.108+0000 7ff92c84f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd3fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff92c84f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd3fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36126/0 (socket says 172.21.15.33:36126) 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff92c84f640 1 -- 172.21.15.33:0/2886163534 learned_addr learned my addr 172.21.15.33:0/2886163534 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff927fff640 1 --2- 172.21.15.33:0/2886163534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 0x55a08dd38640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff92d050640 1 --2- 172.21.15.33:0/2886163534 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd48240 0x55a08dd5fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff92c84f640 1 -- 172.21.15.33:0/2886163534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 msgr2=0x55a08dd38640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff92c84f640 1 --2- 172.21.15.33:0/2886163534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 0x55a08dd38640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff92c84f640 1 -- 172.21.15.33:0/2886163534 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd48240 msgr2=0x55a08dd5fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff92c84f640 1 --2- 172.21.15.33:0/2886163534 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd48240 0x55a08dd5fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff92c84f640 1 -- 172.21.15.33:0/2886163534 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a08dd39090 con 0x55a08dd3d060 2024-09-16T00:23:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff92d050640 1 --2- 172.21.15.33:0/2886163534 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd48240 0x55a08dd5fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:23:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff92c84f640 1 --2- 172.21.15.33:0/2886163534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd3fac0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff91c005c30 tx=0x7ff91c005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eec74c9901e98c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff9127fc640 1 -- 172.21.15.33:0/2886163534 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c01b640 con 0x55a08dd3d060 2024-09-16T00:23:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff9127fc640 1 -- 172.21.15.33:0/2886163534 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff91c011440 con 0x55a08dd3d060 2024-09-16T00:23:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.109+0000 7ff9127fc640 1 -- 172.21.15.33:0/2886163534 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c005430 con 0x55a08dd3d060 2024-09-16T00:23:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.110+0000 7ff930e8c740 1 -- 172.21.15.33:0/2886163534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 msgr2=0x55a08dd3fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.110+0000 7ff930e8c740 1 --2- 172.21.15.33:0/2886163534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd3fac0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff91c005c30 tx=0x7ff91c005c60 comp rx=0 tx=0).stop 2024-09-16T00:23:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.110+0000 7ff930e8c740 1 -- 172.21.15.33:0/2886163534 shutdown_connections 2024-09-16T00:23:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.110+0000 7ff930e8c740 1 --2- 172.21.15.33:0/2886163534 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd48240 0x55a08dd5fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.110+0000 7ff930e8c740 1 --2- 172.21.15.33:0/2886163534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 0x55a08dd38640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.110+0000 7ff930e8c740 1 --2- 172.21.15.33:0/2886163534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd3d060 0x55a08dd3fac0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.110+0000 7ff930e8c740 1 -- 172.21.15.33:0/2886163534 >> 172.21.15.33:0/2886163534 conn(0x55a08dd395f0 msgr2=0x55a08dd37f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.110+0000 7ff930e8c740 1 -- 172.21.15.33:0/2886163534 shutdown_connections 2024-09-16T00:23:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.110+0000 7ff930e8c740 1 -- 172.21.15.33:0/2886163534 wait complete. 2024-09-16T00:23:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.110+0000 7ff930e8c740 1 -- start start 2024-09-16T00:23:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.110+0000 7ff930e8c740 1 Processor -- start 2024-09-16T00:23:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff930e8c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd3d060 0x55a08dd4c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff930e8c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 0x55a08dd4c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff930e8c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd48240 0x55a08ddc1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff930e8c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281af960 con 0x55a08dd48240 2024-09-16T00:23:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff930e8c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281b0dd0 con 0x55a08dd3d060 2024-09-16T00:23:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff930e8c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281b0030 con 0x55a08dd3d980 2024-09-16T00:23:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff930e8c740 10 client.0 _ll_register_callbacks cb 0x7ff91c004500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff92d050640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd48240 0x55a08ddc1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff92d050640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd48240 0x55a08ddc1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36138/0 (socket says 172.21.15.33:36138) 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff92d050640 1 -- 172.21.15.33:0/1083223168 learned_addr learned my addr 172.21.15.33:0/1083223168 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff927fff640 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 0x55a08dd4c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff92c84f640 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd3d060 0x55a08dd4c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff92d050640 1 -- 172.21.15.33:0/1083223168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 msgr2=0x55a08dd4c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff92d050640 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 0x55a08dd4c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff92d050640 1 -- 172.21.15.33:0/1083223168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd3d060 msgr2=0x55a08dd4c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff92d050640 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd3d060 0x55a08dd4c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff92d050640 1 -- 172.21.15.33:0/1083223168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a08ddc1f90 con 0x55a08dd48240 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff92d050640 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd48240 0x55a08ddc1a50 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff92002a990 tx=0x7ff920046ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d6369aa5e02f7a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.111+0000 7ff907fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff907fff640 1 -- 172.21.15.33:0/1083223168 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92002fdc0 con 0x55a08dd48240 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff907fff640 1 -- 172.21.15.33:0/1083223168 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff920003a30 con 0x55a08dd48240 2024-09-16T00:23:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff930e8c740 1 -- 172.21.15.33:0/1083223168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff91c011440 con 0x55a08dd48240 2024-09-16T00:23:23.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff907fff640 1 -- 172.21.15.33:0/1083223168 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff920003c30 con 0x55a08dd48240 2024-09-16T00:23:23.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff930e8c740 10 client.16524 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:23.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff930e8c740 1 -- 172.21.15.33:0/1083223168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a08ddc3120 con 0x55a08dd48240 2024-09-16T00:23:23.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff907fff640 1 -- 172.21.15.33:0/1083223168 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff91c011440 con 0x55a08dd48240 2024-09-16T00:23:23.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff907fff640 1 -- 172.21.15.33:0/1083223168 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff920051440 con 0x55a08dd48240 2024-09-16T00:23:23.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff930e8c740 10 client.16524 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:23.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff930e8c740 10 client.16524 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:23:23.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff930e8c740 20 client.16524 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:23.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff930e8c740 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08ddc3450 0x55a08ddc5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:23.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff930e8c740 4 client.16524 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "fbb869a9", "release": true, "await": true}] 2024-09-16T00:23:23.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.112+0000 7ff930e8c740 1 -- 172.21.15.33:0/1083223168 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "fbb869a9", "release": true, "await": true}) -- 0x55a08dcaa0a0 con 0x55a08ddc3450 2024-09-16T00:23:23.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.113+0000 7ff92c84f640 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08ddc3450 0x55a08ddc5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:23.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.113+0000 7ff92c84f640 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08ddc3450 0x55a08ddc5840 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a942a19505f4aee1 server_cookie=65e64d6a4e331b0 in_seq=0 out_seq=0 2024-09-16T00:23:23.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.113+0000 7ff907fff640 10 client.16524 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:23.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.123+0000 7ff907fff640 1 -- 172.21.15.33:0/1083223168 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55a08dcaa0a0 con 0x55a08ddc3450 2024-09-16T00:23:23.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.123+0000 7ff907fff640 10 client.16524 handle_command_reply: tid=0 2024-09-16T00:23:23.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.123+0000 7ff930e8c740 1 client.16524 shutdown 2024-09-16T00:23:23.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.124+0000 7ff930e8c740 1 -- 172.21.15.33:0/1083223168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd48240 msgr2=0x55a08ddc1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:23.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.124+0000 7ff930e8c740 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd48240 0x55a08ddc1a50 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff92002a990 tx=0x7ff920046ca0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.124+0000 7ff930e8c740 1 -- 172.21.15.33:0/1083223168 shutdown_connections 2024-09-16T00:23:23.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.124+0000 7ff930e8c740 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08ddc3450 0x55a08ddc5840 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.124+0000 7ff930e8c740 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08dd48240 0x55a08ddc1a50 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.124+0000 7ff930e8c740 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08dd3d980 0x55a08dd4c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.124+0000 7ff930e8c740 1 --2- 172.21.15.33:0/1083223168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08dd3d060 0x55a08dd4c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.124+0000 7ff930e8c740 1 -- 172.21.15.33:0/1083223168 >> 172.21.15.33:0/1083223168 conn(0x55a08dd395f0 msgr2=0x55a08dd38750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:23.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.124+0000 7ff907fff640 0 client.16524 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:23.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.124+0000 7ff930e8c740 1 -- 172.21.15.33:0/1083223168 shutdown_connections 2024-09-16T00:23:23.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.124+0000 7ff930e8c740 1 -- 172.21.15.33:0/1083223168 wait complete. 2024-09-16T00:23:23.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.124+0000 7ff930e8c740 20 client.16524 trim_cache size 0 max 16384 2024-09-16T00:23:23.126 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:23:23.126 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:23:23.126 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:23:23.126 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 24, 2024-09-16T00:23:23.126 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:23:23.126 INFO:teuthology.orchestra.run.smithi033.stdout: "fbb869a9": { 2024-09-16T00:23:23.126 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 24, 2024-09-16T00:23:23.126 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:23:23.126 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:23:23.126 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:23:23.126 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 145.3, 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:23:23.127 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:23:23.128 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:23:23.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.125+0000 7ff92eab3640 1 -- 172.21.15.33:0/1880527153 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9180a3240 msgr2=0x7ff9180a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:23.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.125+0000 7ff92eab3640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9180a3240 0x7ff9180a5700 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff92004d000 tx=0x7ff92004a000 comp rx=0 tx=0).stop 2024-09-16T00:23:23.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.125+0000 7ff92eab3640 1 -- 172.21.15.33:0/1880527153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9281036c0 msgr2=0x7ff928100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:23.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.125+0000 7ff92eab3640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9281036c0 0x7ff928100f40 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff91400d950 tx=0x7ff91400de10 comp rx=0 tx=0).stop 2024-09-16T00:23:23.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.125+0000 7ff92eab3640 1 -- 172.21.15.33:0/1880527153 shutdown_connections 2024-09-16T00:23:23.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.125+0000 7ff92eab3640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff9180a3240 0x7ff9180a5700 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.125+0000 7ff92eab3640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff92810c180 0x7ff928101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.125+0000 7ff92eab3640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9281036c0 0x7ff928100f40 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.125+0000 7ff92eab3640 1 --2- 172.21.15.33:0/1880527153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff928102da0 0x7ff928100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:23.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.125+0000 7ff92eab3640 1 -- 172.21.15.33:0/1880527153 >> 172.21.15.33:0/1880527153 conn(0x7ff9280fc7a0 msgr2=0x7ff9280fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:23.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.125+0000 7ff92eab3640 1 -- 172.21.15.33:0/1880527153 shutdown_connections 2024-09-16T00:23:23.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:23.125+0000 7ff92eab3640 1 -- 172.21.15.33:0/1880527153 wait complete. 2024-09-16T00:23:23.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:22 smithi060 ceph-mon[35185]: pgmap v537: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 8 op/s 2024-09-16T00:23:23.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:22 smithi083 ceph-mon[35242]: pgmap v537: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 8 op/s 2024-09-16T00:23:23.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 351574 8.32 MB/sec execute 151 sec latency 23956.217 ms 2024-09-16T00:23:23.887 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'fbb869a9', total seconds elapsed: 28.7 2024-09-16T00:23:24.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 354810 8.39 MB/sec execute 152 sec latency 11.863 ms 2024-09-16T00:23:25.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:24 smithi060 ceph-mon[35185]: pgmap v538: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 9.7 KiB/s wr, 8 op/s 2024-09-16T00:23:25.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:24 smithi033 ceph-mon[31243]: pgmap v538: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 9.7 KiB/s wr, 8 op/s 2024-09-16T00:23:25.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:24 smithi083 ceph-mon[35242]: pgmap v538: 129 pgs: 129 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 9.7 KiB/s wr, 8 op/s 2024-09-16T00:23:25.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 357673 8.42 MB/sec execute 153 sec latency 14.184 ms 2024-09-16T00:23:26.661 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:23:26.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 360057 8.44 MB/sec execute 154 sec latency 11.891 ms 2024-09-16T00:23:26.931 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:23:27.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:27 smithi033 ceph-mon[31243]: pgmap v539: 129 pgs: 129 active+clean; 138 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 46 KiB/s wr, 12 op/s 2024-09-16T00:23:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.454+0000 7ffb79079640 1 -- 172.21.15.33:0/1976831886 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb74102790 msgr2=0x7ffb741067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.454+0000 7ffb79079640 1 --2- 172.21.15.33:0/1976831886 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb74102790 0x7ffb741067e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ffb68009900 tx=0x7ffb6802a410 comp rx=0 tx=0).stop 2024-09-16T00:23:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.454+0000 7ffb79079640 1 -- 172.21.15.33:0/1976831886 shutdown_connections 2024-09-16T00:23:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.454+0000 7ffb79079640 1 --2- 172.21.15.33:0/1976831886 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb74106f10 0x7ffb7410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.454+0000 7ffb79079640 1 --2- 172.21.15.33:0/1976831886 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb74102790 0x7ffb741067e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.454+0000 7ffb79079640 1 --2- 172.21.15.33:0/1976831886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb74101de0 0x7ffb741021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.454+0000 7ffb79079640 1 -- 172.21.15.33:0/1976831886 >> 172.21.15.33:0/1976831886 conn(0x7ffb740fd8c0 msgr2=0x7ffb740ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.454+0000 7ffb79079640 1 -- 172.21.15.33:0/1976831886 shutdown_connections 2024-09-16T00:23:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.454+0000 7ffb79079640 1 -- 172.21.15.33:0/1976831886 wait complete. 2024-09-16T00:23:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.454+0000 7ffb79079640 1 Processor -- start 2024-09-16T00:23:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.455+0000 7ffb79079640 1 -- start start 2024-09-16T00:23:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.455+0000 7ffb79079640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb74101de0 0x7ffb74198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.455+0000 7ffb79079640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb74102790 0x7ffb74198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.455+0000 7ffb79079640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb74106f10 0x7ffb7419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.455+0000 7ffb79079640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffb74111fc0 con 0x7ffb74106f10 2024-09-16T00:23:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.455+0000 7ffb79079640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffb74111e40 con 0x7ffb74101de0 2024-09-16T00:23:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.455+0000 7ffb79079640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffb74112140 con 0x7ffb74102790 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.455+0000 7ffb7359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb74106f10 0x7ffb7419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.455+0000 7ffb72d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb74101de0 0x7ffb74198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.455+0000 7ffb7359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb74106f10 0x7ffb7419cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36158/0 (socket says 172.21.15.33:36158) 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.455+0000 7ffb7359e640 1 -- 172.21.15.33:0/3988323690 learned_addr learned my addr 172.21.15.33:0/3988323690 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.455+0000 7ffb7259c640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb74102790 0x7ffb74198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.456+0000 7ffb72d9d640 1 -- 172.21.15.33:0/3988323690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb74102790 msgr2=0x7ffb74198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.456+0000 7ffb72d9d640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb74102790 0x7ffb74198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.456+0000 7ffb72d9d640 1 -- 172.21.15.33:0/3988323690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb74106f10 msgr2=0x7ffb7419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.456+0000 7ffb72d9d640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb74106f10 0x7ffb7419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.456+0000 7ffb72d9d640 1 -- 172.21.15.33:0/3988323690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7419d3f0 con 0x7ffb74101de0 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.456+0000 7ffb7359e640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb74106f10 0x7ffb7419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.456+0000 7ffb72d9d640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb74101de0 0x7ffb74198440 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffb6400e9f0 tx=0x7ffb6400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.456+0000 7ffb5bfff640 1 -- 172.21.15.33:0/3988323690 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6400ccc0 con 0x7ffb74101de0 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.456+0000 7ffb79079640 1 -- 172.21.15.33:0/3988323690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb7419d6e0 con 0x7ffb74101de0 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.456+0000 7ffb5bfff640 1 -- 172.21.15.33:0/3988323690 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb64005020 con 0x7ffb74101de0 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.456+0000 7ffb5bfff640 1 -- 172.21.15.33:0/3988323690 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb640049e0 con 0x7ffb74101de0 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.456+0000 7ffb79079640 1 -- 172.21.15.33:0/3988323690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb741a4fc0 con 0x7ffb74101de0 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.458+0000 7ffb5bfff640 1 -- 172.21.15.33:0/3988323690 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ffb64005560 con 0x7ffb74101de0 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.458+0000 7ffb7b452740 1 Processor -- start 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7b452740 1 -- start start 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7b452740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e2b3c0 0x558889e2b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7b452740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 0x558889e3a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7b452740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e3aa90 0x558889e3ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7b452740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558889d912e0 con 0x558889e2bce0 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7b452740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558889d38ce0 con 0x558889e2b3c0 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7b452740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558889d8d2a0 con 0x558889e3aa90 2024-09-16T00:23:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb72d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 0x558889e3a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e2b3c0 0x558889e2b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb72d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 0x558889e3a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36172/0 (socket says 172.21.15.33:36172) 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e3aa90 0x558889e3ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e2b3c0 0x558889e2b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:59582/0 (socket says 172.21.15.33:59582) 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb72d9d640 1 -- 172.21.15.33:0/191526726 learned_addr learned my addr 172.21.15.33:0/191526726 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb5bfff640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffb5c0a3240 0x7ffb5c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7259c640 1 -- 172.21.15.33:0/191526726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e2b3c0 msgr2=0x558889e2b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7259c640 1 --2- 172.21.15.33:0/191526726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e2b3c0 0x558889e2b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7259c640 1 -- 172.21.15.33:0/191526726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 msgr2=0x558889e3a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7259c640 1 --2- 172.21.15.33:0/191526726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 0x558889e3a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7359e640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffb5c0a3240 0x7ffb5c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb7259c640 1 -- 172.21.15.33:0/191526726 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558889d1d440 con 0x558889e3aa90 2024-09-16T00:23:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.459+0000 7ffb5bfff640 1 -- 172.21.15.33:0/3988323690 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ffb640cfcd0 con 0x7ffb74101de0 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.460+0000 7ffb7259c640 1 --2- 172.21.15.33:0/191526726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e3aa90 0x558889e3ce80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffb6802aa00 tx=0x7ffb68005de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d48bfc371293b2c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.460+0000 7ffb7359e640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffb5c0a3240 0x7ffb5c0a5700 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c005c40 tx=0x7ffb6c004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.460+0000 7ffb597fa640 1 -- 172.21.15.33:0/191526726 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68035040 con 0x558889e3aa90 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.460+0000 7ffb597fa640 1 -- 172.21.15.33:0/191526726 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb68004400 con 0x558889e3aa90 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.460+0000 7ffb597fa640 1 -- 172.21.15.33:0/191526726 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb680035f0 con 0x558889e3aa90 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.461+0000 7ffb7b452740 1 -- 172.21.15.33:0/191526726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e3aa90 msgr2=0x558889e3ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.461+0000 7ffb7b452740 1 --2- 172.21.15.33:0/191526726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e3aa90 0x558889e3ce80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffb6802aa00 tx=0x7ffb68005de0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.461+0000 7ffb7b452740 1 -- 172.21.15.33:0/191526726 shutdown_connections 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.461+0000 7ffb7b452740 1 --2- 172.21.15.33:0/191526726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e3aa90 0x558889e3ce80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.461+0000 7ffb7b452740 1 --2- 172.21.15.33:0/191526726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 0x558889e3a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.461+0000 7ffb7b452740 1 --2- 172.21.15.33:0/191526726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e2b3c0 0x558889e2b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.461+0000 7ffb7b452740 1 -- 172.21.15.33:0/191526726 >> 172.21.15.33:0/191526726 conn(0x558889e27a30 msgr2=0x558889e27e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.461+0000 7ffb7b452740 1 -- 172.21.15.33:0/191526726 shutdown_connections 2024-09-16T00:23:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.461+0000 7ffb7b452740 1 -- 172.21.15.33:0/191526726 wait complete. 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.461+0000 7ffb7b452740 1 -- start start 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.461+0000 7ffb7b452740 1 Processor -- start 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7b452740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2b3c0 0x558889e3a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7b452740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e2bce0 0x558889e37700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7b452740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e3aa90 0x558889e37cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7b452740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558889e3d5a0 con 0x558889e3aa90 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7b452740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558889d4c050 con 0x558889e2bce0 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7b452740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558889e3d720 con 0x558889e2b3c0 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7b452740 10 client.0 _ll_register_callbacks cb 0x558889d8c160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e3aa90 0x558889e37cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb72d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2b3c0 0x558889e3a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e2bce0 0x558889e37700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e3aa90 0x558889e37cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36184/0 (socket says 172.21.15.33:36184) 2024-09-16T00:23:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7359e640 1 -- 172.21.15.33:0/4192181484 learned_addr learned my addr 172.21.15.33:0/4192181484 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7359e640 1 -- 172.21.15.33:0/4192181484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2b3c0 msgr2=0x558889e3a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7359e640 1 --2- 172.21.15.33:0/4192181484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2b3c0 0x558889e3a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7359e640 1 -- 172.21.15.33:0/4192181484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e2bce0 msgr2=0x558889e37700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7359e640 1 --2- 172.21.15.33:0/4192181484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e2bce0 0x558889e37700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7359e640 1 -- 172.21.15.33:0/4192181484 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558889b023c0 con 0x558889e3aa90 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7259c640 1 --2- 172.21.15.33:0/4192181484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e2bce0 0x558889e37700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7359e640 1 --2- 172.21.15.33:0/4192181484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e3aa90 0x558889e37cf0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c010690 tx=0x7ffb6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7f7e159062d3f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb56ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb56ff5640 1 -- 172.21.15.33:0/4192181484 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c024030 con 0x558889e3aa90 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb56ff5640 1 -- 172.21.15.33:0/4192181484 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ffb6c01f630 con 0x558889e3aa90 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.462+0000 7ffb7b452740 1 -- 172.21.15.33:0/4192181484 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb68004400 con 0x558889e3aa90 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.463+0000 7ffb56ff5640 1 -- 172.21.15.33:0/4192181484 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ffb6c01f830 con 0x558889e3aa90 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.463+0000 7ffb7b452740 10 client.16536 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.463+0000 7ffb7b452740 1 -- 172.21.15.33:0/4192181484 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558889e49060 con 0x558889e3aa90 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.463+0000 7ffb56ff5640 1 -- 172.21.15.33:0/4192181484 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb68004400 con 0x558889e3aa90 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.463+0000 7ffb56ff5640 1 -- 172.21.15.33:0/4192181484 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ffb6c01f040 con 0x558889e3aa90 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.463+0000 7ffb7b452740 10 client.16536 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.463+0000 7ffb7b452740 10 client.16536 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.463+0000 7ffb7b452740 20 client.16536 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.463+0000 7ffb7b452740 1 --2- 172.21.15.33:0/4192181484 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558889e4a5c0 0x558889e4c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.463+0000 7ffb7b452740 4 client.16536 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.463+0000 7ffb7b452740 1 -- 172.21.15.33:0/4192181484 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558889d9e850 con 0x558889e4a5c0 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.463+0000 7ffb72d9d640 1 --2- 172.21.15.33:0/4192181484 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558889e4a5c0 0x558889e4c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.464+0000 7ffb72d9d640 1 --2- 172.21.15.33:0/4192181484 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558889e4a5c0 0x558889e4c9d0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e908836d37ba85ad server_cookie=8c44dd69c937586b in_seq=0 out_seq=0 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.464+0000 7ffb56ff5640 10 client.16536 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.464+0000 7ffb56ff5640 1 -- 172.21.15.33:0/4192181484 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558889d9e850 con 0x558889e4a5c0 2024-09-16T00:23:27.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.464+0000 7ffb56ff5640 10 client.16536 handle_command_reply: tid=0 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.464+0000 7ffb7b452740 1 client.16536 shutdown 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.464+0000 7ffb7b452740 1 -- 172.21.15.33:0/4192181484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e3aa90 msgr2=0x558889e37cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.464+0000 7ffb7b452740 1 --2- 172.21.15.33:0/4192181484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e3aa90 0x558889e37cf0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c010690 tx=0x7ffb6c004a40 comp rx=0 tx=0).stop 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.465+0000 7ffb7b452740 1 -- 172.21.15.33:0/4192181484 shutdown_connections 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.465+0000 7ffb7b452740 1 --2- 172.21.15.33:0/4192181484 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558889e4a5c0 0x558889e4c9d0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.465+0000 7ffb7b452740 1 --2- 172.21.15.33:0/4192181484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e3aa90 0x558889e37cf0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.465+0000 7ffb7b452740 1 --2- 172.21.15.33:0/4192181484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e2bce0 0x558889e37700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.465+0000 7ffb7b452740 1 --2- 172.21.15.33:0/4192181484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2b3c0 0x558889e3a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.465+0000 7ffb7b452740 1 -- 172.21.15.33:0/4192181484 >> 172.21.15.33:0/4192181484 conn(0x558889e27a30 msgr2=0x558889e26b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.465+0000 7ffb56ff5640 0 client.16536 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.465+0000 7ffb7b452740 1 -- 172.21.15.33:0/4192181484 shutdown_connections 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.465+0000 7ffb7b452740 1 -- 172.21.15.33:0/4192181484 wait complete. 2024-09-16T00:23:27.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.465+0000 7ffb7b452740 20 client.16536 trim_cache size 0 max 16384 2024-09-16T00:23:27.487 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:27 smithi060 ceph-mon[35185]: pgmap v539: 129 pgs: 129 active+clean; 138 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 46 KiB/s wr, 12 op/s 2024-09-16T00:23:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb7b452740 1 Processor -- start 2024-09-16T00:23:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb7b452740 1 -- start start 2024-09-16T00:23:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb7b452740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2b3c0 0x558889e46840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb7b452740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2bce0 0x558889e522e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb7b452740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e3aa90 0x558889e52820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb7b452740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffb741af780 con 0x558889e2b3c0 2024-09-16T00:23:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb7b452740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffb741ad050 con 0x558889e3aa90 2024-09-16T00:23:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb7b452740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffb741aaea0 con 0x558889e2bce0 2024-09-16T00:23:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb72d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2b3c0 0x558889e46840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb72d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2b3c0 0x558889e46840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36196/0 (socket says 172.21.15.33:36196) 2024-09-16T00:23:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb7359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e3aa90 0x558889e52820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb72d9d640 1 -- 172.21.15.33:0/1922966911 learned_addr learned my addr 172.21.15.33:0/1922966911 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.483+0000 7ffb7259c640 1 --2- 172.21.15.33:0/1922966911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2bce0 0x558889e522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.484+0000 7ffb7359e640 1 -- 172.21.15.33:0/1922966911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2bce0 msgr2=0x558889e522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.484+0000 7ffb7359e640 1 --2- 172.21.15.33:0/1922966911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2bce0 0x558889e522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.484+0000 7ffb7359e640 1 -- 172.21.15.33:0/1922966911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2b3c0 msgr2=0x558889e46840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.484+0000 7ffb7359e640 1 --2- 172.21.15.33:0/1922966911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2b3c0 0x558889e46840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.484+0000 7ffb7359e640 1 -- 172.21.15.33:0/1922966911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558889e2deb0 con 0x558889e3aa90 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.484+0000 7ffb72d9d640 1 --2- 172.21.15.33:0/1922966911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2b3c0 0x558889e46840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.484+0000 7ffb7359e640 1 --2- 172.21.15.33:0/1922966911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e3aa90 0x558889e52820 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c029d50 tx=0x7ffb6c01f7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=542adf14448d565c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.484+0000 7ffb58ff9640 1 -- 172.21.15.33:0/1922966911 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c010e60 con 0x558889e3aa90 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.484+0000 7ffb58ff9640 1 -- 172.21.15.33:0/1922966911 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb6c0298f0 con 0x558889e3aa90 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.484+0000 7ffb58ff9640 1 -- 172.21.15.33:0/1922966911 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c024070 con 0x558889e3aa90 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 -- 172.21.15.33:0/1922966911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e3aa90 msgr2=0x558889e52820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 --2- 172.21.15.33:0/1922966911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e3aa90 0x558889e52820 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c029d50 tx=0x7ffb6c01f7c0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 -- 172.21.15.33:0/1922966911 shutdown_connections 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 --2- 172.21.15.33:0/1922966911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e3aa90 0x558889e52820 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 --2- 172.21.15.33:0/1922966911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2bce0 0x558889e522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 --2- 172.21.15.33:0/1922966911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2b3c0 0x558889e46840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 -- 172.21.15.33:0/1922966911 >> 172.21.15.33:0/1922966911 conn(0x558889e27a30 msgr2=0x558889e367f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 -- 172.21.15.33:0/1922966911 shutdown_connections 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 -- 172.21.15.33:0/1922966911 wait complete. 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 -- start start 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 Processor -- start 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2b3c0 0x558889e431b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 0x558889e3e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e3aa90 0x558889e3ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffb741ab510 con 0x558889e2bce0 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffb741ac980 con 0x558889e3aa90 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.485+0000 7ffb7b452740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffb741abbe0 con 0x558889e2b3c0 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb7b452740 10 client.0 _ll_register_callbacks cb 0x7ffb6c00ef20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb7259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 0x558889e3e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb72d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2b3c0 0x558889e431b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb72d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2b3c0 0x558889e431b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:51954/0 (socket says 172.21.15.33:51954) 2024-09-16T00:23:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb7359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e3aa90 0x558889e3ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb72d9d640 1 -- 172.21.15.33:0/3866521894 learned_addr learned my addr 172.21.15.33:0/3866521894 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb7259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 0x558889e3e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36212/0 (socket says 172.21.15.33:36212) 2024-09-16T00:23:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb72d9d640 1 -- 172.21.15.33:0/3866521894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e3aa90 msgr2=0x558889e3ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb72d9d640 1 --2- 172.21.15.33:0/3866521894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e3aa90 0x558889e3ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb72d9d640 1 -- 172.21.15.33:0/3866521894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 msgr2=0x558889e3e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb72d9d640 1 --2- 172.21.15.33:0/3866521894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 0x558889e3e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb72d9d640 1 -- 172.21.15.33:0/3866521894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558889e3f290 con 0x558889e2b3c0 2024-09-16T00:23:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb7259c640 1 --2- 172.21.15.33:0/3866521894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 0x558889e3e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb72d9d640 1 --2- 172.21.15.33:0/3866521894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2b3c0 0x558889e431b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffb640a45b0 tx=0x7ffb64098240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3acf6d9225086379 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb56ff5640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb56ff5640 1 -- 172.21.15.33:0/3866521894 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6401c040 con 0x558889e2b3c0 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb7b452740 1 -- 172.21.15.33:0/3866521894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb6c0298f0 con 0x558889e2b3c0 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb56ff5640 1 -- 172.21.15.33:0/3866521894 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ffb640984e0 con 0x558889e2b3c0 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb56ff5640 1 -- 172.21.15.33:0/3866521894 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ffb64068050 con 0x558889e2b3c0 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb7b452740 10 client.25978 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.486+0000 7ffb7b452740 1 -- 172.21.15.33:0/3866521894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558889eb53c0 con 0x558889e2b3c0 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.487+0000 7ffb56ff5640 1 -- 172.21.15.33:0/3866521894 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb6c0298f0 con 0x558889e2b3c0 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.487+0000 7ffb56ff5640 1 -- 172.21.15.33:0/3866521894 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ffb64004c60 con 0x558889e2b3c0 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.487+0000 7ffb7b452740 10 client.25978 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.487+0000 7ffb7b452740 10 client.25978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.487+0000 7ffb7b452740 20 client.25978 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.487+0000 7ffb7b452740 1 --2- 172.21.15.33:0/3866521894 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558889eb5b10 0x558889eb7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.487+0000 7ffb7b452740 4 client.25978 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.487+0000 7ffb7b452740 1 -- 172.21.15.33:0/3866521894 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558889d98d60 con 0x558889eb5b10 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.487+0000 7ffb7259c640 1 --2- 172.21.15.33:0/3866521894 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558889eb5b10 0x558889eb7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7259c640 1 --2- 172.21.15.33:0/3866521894 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558889eb5b10 0x558889eb7f30 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=121cb87beeb51c13 server_cookie=be0cdb5103f4a88c in_seq=0 out_seq=0 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb56ff5640 10 client.25978 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb56ff5640 1 -- 172.21.15.33:0/3866521894 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558889d98d60 con 0x558889eb5b10 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb56ff5640 10 client.25978 handle_command_reply: tid=0 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7b452740 1 client.25978 shutdown 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7b452740 1 -- 172.21.15.33:0/3866521894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2b3c0 msgr2=0x558889e431b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7b452740 1 --2- 172.21.15.33:0/3866521894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2b3c0 0x558889e431b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffb640a45b0 tx=0x7ffb64098240 comp rx=0 tx=0).stop 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7b452740 1 -- 172.21.15.33:0/3866521894 shutdown_connections 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7b452740 1 --2- 172.21.15.33:0/3866521894 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558889eb5b10 0x558889eb7f30 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7b452740 1 --2- 172.21.15.33:0/3866521894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558889e3aa90 0x558889e3ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7b452740 1 --2- 172.21.15.33:0/3866521894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558889e2bce0 0x558889e3e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb56ff5640 0 client.25978 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:27.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7b452740 1 --2- 172.21.15.33:0/3866521894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558889e2b3c0 0x558889e431b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7b452740 1 -- 172.21.15.33:0/3866521894 >> 172.21.15.33:0/3866521894 conn(0x558889e27a30 msgr2=0x558889e324f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:27.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7b452740 1 -- 172.21.15.33:0/3866521894 shutdown_connections 2024-09-16T00:23:27.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7b452740 1 -- 172.21.15.33:0/3866521894 wait complete. 2024-09-16T00:23:27.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.488+0000 7ffb7b452740 20 client.25978 trim_cache size 0 max 16384 2024-09-16T00:23:27.498 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:23:27.498 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:23:27.498 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:23:27.498 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:23:27.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.489+0000 7ffb79079640 1 -- 172.21.15.33:0/3988323690 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffb5c0a3240 msgr2=0x7ffb5c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:27.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.489+0000 7ffb79079640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffb5c0a3240 0x7ffb5c0a5700 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c005c40 tx=0x7ffb6c004ab0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.489+0000 7ffb79079640 1 -- 172.21.15.33:0/3988323690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb74101de0 msgr2=0x7ffb74198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:27.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.489+0000 7ffb79079640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb74101de0 0x7ffb74198440 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffb6400e9f0 tx=0x7ffb6400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.489+0000 7ffb79079640 1 -- 172.21.15.33:0/3988323690 shutdown_connections 2024-09-16T00:23:27.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.489+0000 7ffb79079640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffb5c0a3240 0x7ffb5c0a5700 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.489+0000 7ffb79079640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb74106f10 0x7ffb7419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.489+0000 7ffb79079640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb74102790 0x7ffb74198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.489+0000 7ffb79079640 1 --2- 172.21.15.33:0/3988323690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb74101de0 0x7ffb74198440 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:27.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.489+0000 7ffb79079640 1 -- 172.21.15.33:0/3988323690 >> 172.21.15.33:0/3988323690 conn(0x7ffb740fd8c0 msgr2=0x7ffb740fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:27.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.489+0000 7ffb79079640 1 -- 172.21.15.33:0/3988323690 shutdown_connections 2024-09-16T00:23:27.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:27.489+0000 7ffb79079640 1 -- 172.21.15.33:0/3988323690 wait complete. 2024-09-16T00:23:27.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 362765 8.48 MB/sec execute 155 sec latency 13.846 ms 2024-09-16T00:23:27.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:27 smithi083 ceph-mon[35242]: pgmap v539: 129 pgs: 129 active+clean; 138 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 46 KiB/s wr, 12 op/s 2024-09-16T00:23:28.457 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:23:28.457 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:23:28.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 365942 8.55 MB/sec execute 156 sec latency 16.394 ms 2024-09-16T00:23:28.735 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:23:29.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.266+0000 7ff1053a0640 1 -- 172.21.15.33:0/80393194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff100102380 msgr2=0x7ff10010e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:29.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.266+0000 7ff1053a0640 1 --2- 172.21.15.33:0/80393194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff100102380 0x7ff10010e9c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009cd0 tx=0x7ff0f802a8e0 comp rx=0 tx=0).stop 2024-09-16T00:23:29.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.266+0000 7ff1053a0640 1 -- 172.21.15.33:0/80393194 shutdown_connections 2024-09-16T00:23:29.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.266+0000 7ff1053a0640 1 --2- 172.21.15.33:0/80393194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff100102380 0x7ff10010e9c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:29.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.266+0000 7ff1053a0640 1 --2- 172.21.15.33:0/80393194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001019e0 0x7ff100101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:29.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.266+0000 7ff1053a0640 1 --2- 172.21.15.33:0/80393194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff100107af0 0x7ff100107ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:29.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.267+0000 7ff1053a0640 1 -- 172.21.15.33:0/80393194 >> 172.21.15.33:0/80393194 conn(0x7ff1000fd700 msgr2=0x7ff1000ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:29.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.267+0000 7ff1053a0640 1 -- 172.21.15.33:0/80393194 shutdown_connections 2024-09-16T00:23:29.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.267+0000 7ff1053a0640 1 -- 172.21.15.33:0/80393194 wait complete. 2024-09-16T00:23:29.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.267+0000 7ff1053a0640 1 Processor -- start 2024-09-16T00:23:29.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.267+0000 7ff1053a0640 1 -- start start 2024-09-16T00:23:29.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff1053a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001019e0 0x7ff1001982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:29.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff1053a0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff100102380 0x7ff100198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:29.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff1053a0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff100107af0 0x7ff10019cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:29.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff1053a0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff100111ef0 con 0x7ff100107af0 2024-09-16T00:23:29.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff1053a0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff100111d70 con 0x7ff100102380 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff1053a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff100112070 con 0x7ff1001019e0 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff0fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001019e0 0x7ff1001982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff104b9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff100107af0 0x7ff10019cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff0fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001019e0 0x7ff1001982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:51968/0 (socket says 172.21.15.33:51968) 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff0fffff640 1 -- 172.21.15.33:0/1071707067 learned_addr learned my addr 172.21.15.33:0/1071707067 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff0ff7fe640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff100102380 0x7ff100198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff0fffff640 1 -- 172.21.15.33:0/1071707067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff100102380 msgr2=0x7ff100198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff0fffff640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff100102380 0x7ff100198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff0fffff640 1 -- 172.21.15.33:0/1071707067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff100107af0 msgr2=0x7ff10019cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff0fffff640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff100107af0 0x7ff10019cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff0fffff640 1 -- 172.21.15.33:0/1071707067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10019d2a0 con 0x7ff1001019e0 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.268+0000 7ff0fffff640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001019e0 0x7ff1001982f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000e9f0 tx=0x7ff0f000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.269+0000 7ff0fd7fa640 1 -- 172.21.15.33:0/1071707067 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f000cd80 con 0x7ff1001019e0 2024-09-16T00:23:29.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.269+0000 7ff0fd7fa640 1 -- 172.21.15.33:0/1071707067 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0f0010910 con 0x7ff1001019e0 2024-09-16T00:23:29.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.269+0000 7ff0fd7fa640 1 -- 172.21.15.33:0/1071707067 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0018990 con 0x7ff1001019e0 2024-09-16T00:23:29.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.269+0000 7ff1053a0640 1 -- 172.21.15.33:0/1071707067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff10019d590 con 0x7ff1001019e0 2024-09-16T00:23:29.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.269+0000 7ff1053a0640 1 -- 172.21.15.33:0/1071707067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1001a4e70 con 0x7ff1001019e0 2024-09-16T00:23:29.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.270+0000 7ff1053a0640 1 -- 172.21.15.33:0/1071707067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff100101e40 con 0x7ff1001019e0 2024-09-16T00:23:29.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.271+0000 7ff0fd7fa640 1 -- 172.21.15.33:0/1071707067 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff0f0010ac0 con 0x7ff1001019e0 2024-09-16T00:23:29.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.272+0000 7ff0fd7fa640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0e80a32c0 0x7ff0e80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:29.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.272+0000 7ff0ff7fe640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0e80a32c0 0x7ff0e80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:29.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.272+0000 7ff0fd7fa640 1 -- 172.21.15.33:0/1071707067 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff0f00d0320 con 0x7ff1001019e0 2024-09-16T00:23:29.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.272+0000 7ff0ff7fe640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0e80a32c0 0x7ff0e80a5780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff100107040 tx=0x7ff0f4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:29.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.277+0000 7ff0fd7fa640 1 -- 172.21.15.33:0/1071707067 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff0f009c510 con 0x7ff1001019e0 2024-09-16T00:23:29.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:29 smithi060 ceph-mon[35185]: pgmap v540: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 380 KiB/s wr, 68 op/s 2024-09-16T00:23:29.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.505+0000 7ff1053a0640 1 -- 172.21.15.33:0/1071707067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff100199650 con 0x7ff1001019e0 2024-09-16T00:23:29.505 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:29 smithi033 ceph-mon[31243]: pgmap v540: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 380 KiB/s wr, 68 op/s 2024-09-16T00:23:29.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.506+0000 7ff0fd7fa640 1 -- 172.21.15.33:0/1071707067 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7ff0f00a0810 con 0x7ff1001019e0 2024-09-16T00:23:29.506 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:23:29.506 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:23:29.507 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:23:29.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.508+0000 7ff1053a0640 1 -- 172.21.15.33:0/1071707067 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0e80a32c0 msgr2=0x7ff0e80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:29.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.508+0000 7ff1053a0640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0e80a32c0 0x7ff0e80a5780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff100107040 tx=0x7ff0f4005f90 comp rx=0 tx=0).stop 2024-09-16T00:23:29.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.508+0000 7ff1053a0640 1 -- 172.21.15.33:0/1071707067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001019e0 msgr2=0x7ff1001982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:29.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.508+0000 7ff1053a0640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001019e0 0x7ff1001982f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000e9f0 tx=0x7ff0f000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:23:29.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.508+0000 7ff1053a0640 1 -- 172.21.15.33:0/1071707067 shutdown_connections 2024-09-16T00:23:29.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.508+0000 7ff1053a0640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0e80a32c0 0x7ff0e80a5780 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:29.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.508+0000 7ff1053a0640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff100107af0 0x7ff10019cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:29.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.508+0000 7ff1053a0640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff100102380 0x7ff100198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:29.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.508+0000 7ff1053a0640 1 --2- 172.21.15.33:0/1071707067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001019e0 0x7ff1001982f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:29.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.508+0000 7ff1053a0640 1 -- 172.21.15.33:0/1071707067 >> 172.21.15.33:0/1071707067 conn(0x7ff1000fd700 msgr2=0x7ff1000fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:29.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.508+0000 7ff1053a0640 1 -- 172.21.15.33:0/1071707067 shutdown_connections 2024-09-16T00:23:29.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:29.508+0000 7ff1053a0640 1 -- 172.21.15.33:0/1071707067 wait complete. 2024-09-16T00:23:29.662 INFO:tasks.workunit.client.0.smithi033.stdout: 1 368716 8.59 MB/sec execute 157 sec latency 50.564 ms 2024-09-16T00:23:29.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:29 smithi083 ceph-mon[35242]: pgmap v540: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 380 KiB/s wr, 68 op/s 2024-09-16T00:23:30.355 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:23:30.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:30 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1071707067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:23:30.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:30 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1071707067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:23:30.634 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:23:30.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 371024 8.60 MB/sec execute 158 sec latency 23.157 ms 2024-09-16T00:23:30.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:30 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1071707067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:23:31.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.146+0000 7f01c52da640 1 -- 172.21.15.33:0/3492468268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01c010c170 msgr2=0x7f01c010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:31.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.146+0000 7f01c52da640 1 --2- 172.21.15.33:0/3492468268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01c010c170 0x7f01c010e610 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009ca0 tx=0x7f01b802a950 comp rx=0 tx=0).stop 2024-09-16T00:23:31.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.146+0000 7f01c52da640 1 -- 172.21.15.33:0/3492468268 shutdown_connections 2024-09-16T00:23:31.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.146+0000 7f01c52da640 1 --2- 172.21.15.33:0/3492468268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01c010c170 0x7f01c010e610 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.146+0000 7f01c52da640 1 --2- 172.21.15.33:0/3492468268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01c0103660 0x7f01c010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.146+0000 7f01c52da640 1 --2- 172.21.15.33:0/3492468268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c0102d40 0x7f01c0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.146+0000 7f01c52da640 1 -- 172.21.15.33:0/3492468268 >> 172.21.15.33:0/3492468268 conn(0x7f01c00fc7a0 msgr2=0x7f01c00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:31.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.146+0000 7f01c52da640 1 -- 172.21.15.33:0/3492468268 shutdown_connections 2024-09-16T00:23:31.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.146+0000 7f01c52da640 1 -- 172.21.15.33:0/3492468268 wait complete. 2024-09-16T00:23:31.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01c52da640 1 Processor -- start 2024-09-16T00:23:31.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01c52da640 1 -- start start 2024-09-16T00:23:31.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01c52da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c0102d40 0x7f01c01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01c52da640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01c0103660 0x7f01c0198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01c52da640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01c010c170 0x7f01c019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01c52da640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01c0111f70 con 0x7f01c0103660 2024-09-16T00:23:31.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01c52da640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f01c0111df0 con 0x7f01c0102d40 2024-09-16T00:23:31.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01c52da640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f01c01120f0 con 0x7f01c010c170 2024-09-16T00:23:31.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01b7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01c0103660 0x7f01c0198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01b7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01c0103660 0x7f01c0198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59526/0 (socket says 172.21.15.33:59526) 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01b7fff640 1 -- 172.21.15.33:0/3503090622 learned_addr learned my addr 172.21.15.33:0/3503090622 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01beffd640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c0102d40 0x7f01c01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01bf7fe640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01c010c170 0x7f01c019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01b7fff640 1 -- 172.21.15.33:0/3503090622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01c010c170 msgr2=0x7f01c019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01b7fff640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01c010c170 0x7f01c019cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01b7fff640 1 -- 172.21.15.33:0/3503090622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c0102d40 msgr2=0x7f01c01983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01b7fff640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c0102d40 0x7f01c01983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01b7fff640 1 -- 172.21.15.33:0/3503090622 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c019d380 con 0x7f01c0103660 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.147+0000 7f01beffd640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c0102d40 0x7f01c01983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.148+0000 7f01b7fff640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01c0103660 0x7f01c0198910 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f01a800e9f0 tx=0x7f01a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.148+0000 7f01bcff9640 1 -- 172.21.15.33:0/3503090622 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a800ccc0 con 0x7f01c0103660 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.148+0000 7f01c52da640 1 -- 172.21.15.33:0/3503090622 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c019d670 con 0x7f01c0103660 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.148+0000 7f01bcff9640 1 -- 172.21.15.33:0/3503090622 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01a8010910 con 0x7f01c0103660 2024-09-16T00:23:31.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.148+0000 7f01c52da640 1 -- 172.21.15.33:0/3503090622 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c01a4f50 con 0x7f01c0103660 2024-09-16T00:23:31.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.148+0000 7f01bcff9640 1 -- 172.21.15.33:0/3503090622 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a8018920 con 0x7f01c0103660 2024-09-16T00:23:31.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.150+0000 7f01bcff9640 1 -- 172.21.15.33:0/3503090622 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f01a8010430 con 0x7f01c0103660 2024-09-16T00:23:31.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01c76b3740 1 Processor -- start 2024-09-16T00:23:31.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01c76b3740 1 -- start start 2024-09-16T00:23:31.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01c76b3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4cb40 0x565173b4b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01c76b3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4b5d0 0x565173b4ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01c76b3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b56370 0x565173b58760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01c76b3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x565173ab12f0 con 0x565173b4b5d0 2024-09-16T00:23:31.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01c76b3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x565173a58cf0 con 0x565173b4cb40 2024-09-16T00:23:31.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01c76b3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x565173aad2b0 con 0x565173b56370 2024-09-16T00:23:31.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01bcff9640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01ac0a3240 0x7f01ac0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01b7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b56370 0x565173b58760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01beffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4b5d0 0x565173b4ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01b7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b56370 0x565173b58760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36758/0 (socket says 172.21.15.33:36758) 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01beffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4b5d0 0x565173b4ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59542/0 (socket says 172.21.15.33:59542) 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01b7fff640 1 -- 172.21.15.33:0/2672188084 learned_addr learned my addr 172.21.15.33:0/2672188084 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01bf7fe640 1 --2- 172.21.15.33:0/2672188084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4cb40 0x565173b4b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.151+0000 7f01bf7fe640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01ac0a3240 0x7f01ac0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.152+0000 7f01bcff9640 1 -- 172.21.15.33:0/3503090622 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f01a80cff40 con 0x7f01c0103660 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.152+0000 7f01b7fff640 1 -- 172.21.15.33:0/2672188084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4cb40 msgr2=0x565173b4b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.152+0000 7f01b7fff640 1 --2- 172.21.15.33:0/2672188084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4cb40 0x565173b4b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.152+0000 7f01b7fff640 1 -- 172.21.15.33:0/2672188084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4b5d0 msgr2=0x565173b4ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.152+0000 7f01b7fff640 1 --2- 172.21.15.33:0/2672188084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4b5d0 0x565173b4ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.152+0000 7f01b7fff640 1 -- 172.21.15.33:0/2672188084 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565173a3d3a0 con 0x565173b56370 2024-09-16T00:23:31.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.152+0000 7f01b7fff640 1 --2- 172.21.15.33:0/2672188084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b56370 0x565173b58760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f01a809c120 tx=0x7f01a8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd646e7fc3969286 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:31.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.152+0000 7f01bf7fe640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01ac0a3240 0x7f01ac0a5700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009430 tx=0x7f01b8005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:31.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.152+0000 7f01b57fa640 1 -- 172.21.15.33:0/2672188084 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a80a0490 con 0x565173b56370 2024-09-16T00:23:31.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.152+0000 7f01b57fa640 1 -- 172.21.15.33:0/2672188084 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01a809c190 con 0x565173b56370 2024-09-16T00:23:31.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.152+0000 7f01b57fa640 1 -- 172.21.15.33:0/2672188084 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a8098800 con 0x565173b56370 2024-09-16T00:23:31.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.153+0000 7f01c76b3740 1 -- 172.21.15.33:0/2672188084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b56370 msgr2=0x565173b58760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:31.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.153+0000 7f01c76b3740 1 --2- 172.21.15.33:0/2672188084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b56370 0x565173b58760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f01a809c120 tx=0x7f01a8009f90 comp rx=0 tx=0).stop 2024-09-16T00:23:31.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.153+0000 7f01c76b3740 1 -- 172.21.15.33:0/2672188084 shutdown_connections 2024-09-16T00:23:31.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.153+0000 7f01c76b3740 1 --2- 172.21.15.33:0/2672188084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b56370 0x565173b58760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.153+0000 7f01c76b3740 1 --2- 172.21.15.33:0/2672188084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4b5d0 0x565173b4ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.153+0000 7f01c76b3740 1 --2- 172.21.15.33:0/2672188084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4cb40 0x565173b4b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.153+0000 7f01c76b3740 1 -- 172.21.15.33:0/2672188084 >> 172.21.15.33:0/2672188084 conn(0x565173b14310 msgr2=0x565173b47c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:31.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.153+0000 7f01c76b3740 1 -- 172.21.15.33:0/2672188084 shutdown_connections 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.153+0000 7f01c76b3740 1 -- 172.21.15.33:0/2672188084 wait complete. 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.153+0000 7f01c76b3740 1 -- start start 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01c76b3740 1 Processor -- start 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01c76b3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4b5d0 0x565173b55d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01c76b3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b4cb40 0x565173b53020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01c76b3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b56370 0x565173b53610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01c76b3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x565173b58fb0 con 0x565173b4b5d0 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01c76b3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x565173a6c060 con 0x565173b56370 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01c76b3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x565173b59130 con 0x565173b4cb40 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01c76b3740 10 client.0 _ll_register_callbacks cb 0x5651737bf8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01beffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4b5d0 0x565173b55d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01b7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b4cb40 0x565173b53020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01b7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b4cb40 0x565173b53020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36760/0 (socket says 172.21.15.33:36760) 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01b7fff640 1 -- 172.21.15.33:0/330985708 learned_addr learned my addr 172.21.15.33:0/330985708 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01bf7fe640 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b56370 0x565173b53610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01b7fff640 1 -- 172.21.15.33:0/330985708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b56370 msgr2=0x565173b53610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01b7fff640 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b56370 0x565173b53610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01b7fff640 1 -- 172.21.15.33:0/330985708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4b5d0 msgr2=0x565173b55d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01b7fff640 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4b5d0 0x565173b55d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01b7fff640 1 -- 172.21.15.33:0/330985708 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651738223c0 con 0x565173b4cb40 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01beffd640 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4b5d0 0x565173b55d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01b7fff640 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b4cb40 0x565173b53020 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f01a800c6a0 tx=0x7f01a800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bea72470a3f92c6b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.154+0000 7f01a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01a67fc640 1 -- 172.21.15.33:0/330985708 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a80b4020 con 0x565173b4cb40 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01a67fc640 1 -- 172.21.15.33:0/330985708 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f01a8097ee0 con 0x565173b4cb40 2024-09-16T00:23:31.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01c76b3740 1 -- 172.21.15.33:0/330985708 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01a809c190 con 0x565173b4cb40 2024-09-16T00:23:31.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01a67fc640 1 -- 172.21.15.33:0/330985708 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f01a80bc080 con 0x565173b4cb40 2024-09-16T00:23:31.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01c76b3740 10 client.25996 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:31.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01c76b3740 1 -- 172.21.15.33:0/330985708 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x565173b58fb0 con 0x565173b4cb40 2024-09-16T00:23:31.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01a67fc640 1 -- 172.21.15.33:0/330985708 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01a809c190 con 0x565173b4cb40 2024-09-16T00:23:31.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01a67fc640 1 -- 172.21.15.33:0/330985708 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f01a80a1620 con 0x565173b4cb40 2024-09-16T00:23:31.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01c76b3740 10 client.25996 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:31.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01c76b3740 10 client.25996 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:23:31.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01c76b3740 20 client.25996 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:31.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01c76b3740 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x565173b65bd0 0x565173b67fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01c76b3740 4 client.25996 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:23:31.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01c76b3740 1 -- 172.21.15.33:0/330985708 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565173abe860 con 0x565173b65bd0 2024-09-16T00:23:31.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.155+0000 7f01beffd640 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x565173b65bd0 0x565173b67fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.156+0000 7f01beffd640 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x565173b65bd0 0x565173b67fc0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a37bda99a2ac71c1 server_cookie=b7e660138bc489f6 in_seq=0 out_seq=0 2024-09-16T00:23:31.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.156+0000 7f01a67fc640 10 client.25996 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:23:31.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01a67fc640 1 -- 172.21.15.33:0/330985708 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x565173abe860 con 0x565173b65bd0 2024-09-16T00:23:31.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01a67fc640 10 client.25996 handle_command_reply: tid=0 2024-09-16T00:23:31.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01c76b3740 1 client.25996 shutdown 2024-09-16T00:23:31.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01c76b3740 1 -- 172.21.15.33:0/330985708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b4cb40 msgr2=0x565173b53020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:31.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01c76b3740 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b4cb40 0x565173b53020 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f01a800c6a0 tx=0x7f01a800cf60 comp rx=0 tx=0).stop 2024-09-16T00:23:31.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01c76b3740 1 -- 172.21.15.33:0/330985708 shutdown_connections 2024-09-16T00:23:31.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01c76b3740 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x565173b65bd0 0x565173b67fc0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01c76b3740 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b56370 0x565173b53610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01c76b3740 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b4cb40 0x565173b53020 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01a67fc640 0 client.25996 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:23:31.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01c76b3740 1 --2- 172.21.15.33:0/330985708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4b5d0 0x565173b55d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01c76b3740 1 -- 172.21.15.33:0/330985708 >> 172.21.15.33:0/330985708 conn(0x565173b14310 msgr2=0x565173b4d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:31.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01c76b3740 1 -- 172.21.15.33:0/330985708 shutdown_connections 2024-09-16T00:23:31.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.159+0000 7f01c76b3740 1 -- 172.21.15.33:0/330985708 wait complete. 2024-09-16T00:23:31.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.160+0000 7f01c76b3740 20 client.25996 trim_cache size 0 max 16384 2024-09-16T00:23:31.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.173+0000 7f01c76b3740 1 Processor -- start 2024-09-16T00:23:31.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.173+0000 7f01c76b3740 1 -- start start 2024-09-16T00:23:31.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.173+0000 7f01c76b3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b4b5d0 0x565173b47d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.173+0000 7f01c76b3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4cb40 0x565173b62250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.173+0000 7f01c76b3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b56370 0x565173b52000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.173+0000 7f01c76b3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01c01b1a20 con 0x565173b56370 2024-09-16T00:23:31.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01c76b3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f01c01af2f0 con 0x565173b4cb40 2024-09-16T00:23:31.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01c76b3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f01c01ad140 con 0x565173b4b5d0 2024-09-16T00:23:31.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01bf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b56370 0x565173b52000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01bf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b56370 0x565173b52000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59556/0 (socket says 172.21.15.33:59556) 2024-09-16T00:23:31.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4cb40 0x565173b62250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01bf7fe640 1 -- 172.21.15.33:0/3677398254 learned_addr learned my addr 172.21.15.33:0/3677398254 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:31.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01beffd640 1 --2- 172.21.15.33:0/3677398254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b4b5d0 0x565173b47d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01b7fff640 1 -- 172.21.15.33:0/3677398254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b4b5d0 msgr2=0x565173b47d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:31.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01b7fff640 1 --2- 172.21.15.33:0/3677398254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b4b5d0 0x565173b47d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01b7fff640 1 -- 172.21.15.33:0/3677398254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b56370 msgr2=0x565173b52000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:31.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01b7fff640 1 --2- 172.21.15.33:0/3677398254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b56370 0x565173b52000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01b7fff640 1 -- 172.21.15.33:0/3677398254 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565173b51570 con 0x565173b4cb40 2024-09-16T00:23:31.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01bf7fe640 1 --2- 172.21.15.33:0/3677398254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b56370 0x565173b52000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:31.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.174+0000 7f01b7fff640 1 --2- 172.21.15.33:0/3677398254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4cb40 0x565173b62250 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f01a809a440 tx=0x7f01a8009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24ff7ba46e1f944d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:31.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.175+0000 7f01b4ff9640 1 -- 172.21.15.33:0/3677398254 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a8010dc0 con 0x565173b4cb40 2024-09-16T00:23:31.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.175+0000 7f01b4ff9640 1 -- 172.21.15.33:0/3677398254 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01a80a0490 con 0x565173b4cb40 2024-09-16T00:23:31.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.175+0000 7f01b4ff9640 1 -- 172.21.15.33:0/3677398254 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a8005040 con 0x565173b4cb40 2024-09-16T00:23:31.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.175+0000 7f01c76b3740 1 -- 172.21.15.33:0/3677398254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4cb40 msgr2=0x565173b62250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:31.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.175+0000 7f01c76b3740 1 --2- 172.21.15.33:0/3677398254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4cb40 0x565173b62250 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f01a809a440 tx=0x7f01a8009f90 comp rx=0 tx=0).stop 2024-09-16T00:23:31.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.175+0000 7f01c76b3740 1 -- 172.21.15.33:0/3677398254 shutdown_connections 2024-09-16T00:23:31.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 --2- 172.21.15.33:0/3677398254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b56370 0x565173b52000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 --2- 172.21.15.33:0/3677398254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4cb40 0x565173b62250 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 --2- 172.21.15.33:0/3677398254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b4b5d0 0x565173b47d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 -- 172.21.15.33:0/3677398254 >> 172.21.15.33:0/3677398254 conn(0x565173b14310 msgr2=0x565173b518f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 -- 172.21.15.33:0/3677398254 shutdown_connections 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 -- 172.21.15.33:0/3677398254 wait complete. 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 -- start start 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 Processor -- start 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4b5d0 0x565173b5eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4cb40 0x565173b59ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b56370 0x565173b5a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01c01ad7b0 con 0x565173b4cb40 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f01c01aec20 con 0x565173b4b5d0 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f01c01ade80 con 0x565173b56370 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01c76b3740 10 client.0 _ll_register_callbacks cb 0x7f01b000bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:31.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01b7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4cb40 0x565173b59ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.176+0000 7f01beffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4b5d0 0x565173b5eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01b7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4cb40 0x565173b59ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59558/0 (socket says 172.21.15.33:59558) 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01b7fff640 1 -- 172.21.15.33:0/3142708069 learned_addr learned my addr 172.21.15.33:0/3142708069 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01bf7fe640 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b56370 0x565173b5a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01beffd640 1 -- 172.21.15.33:0/3142708069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b56370 msgr2=0x565173b5a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01beffd640 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b56370 0x565173b5a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01beffd640 1 -- 172.21.15.33:0/3142708069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4cb40 msgr2=0x565173b59ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01beffd640 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4cb40 0x565173b59ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01beffd640 1 -- 172.21.15.33:0/3142708069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565173b5ac60 con 0x565173b4b5d0 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01b7fff640 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4cb40 0x565173b59ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01beffd640 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4b5d0 0x565173b5eba0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x565173a6c120 tx=0x7f01b0012cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=872b91d65e3b000d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:23:31.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01a67fc640 1 -- 172.21.15.33:0/3142708069 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b0010e30 con 0x565173b4b5d0 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01a67fc640 1 -- 172.21.15.33:0/3142708069 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f01b0004020 con 0x565173b4b5d0 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01c76b3740 1 -- 172.21.15.33:0/3142708069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01a80a0490 con 0x565173b4b5d0 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01a67fc640 1 -- 172.21.15.33:0/3142708069 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f01b0004220 con 0x565173b4b5d0 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.177+0000 7f01c76b3740 10 client.26093 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.178+0000 7f01c76b3740 1 -- 172.21.15.33:0/3142708069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x565173bd0740 con 0x565173b4b5d0 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.178+0000 7f01a67fc640 1 -- 172.21.15.33:0/3142708069 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01a80a0490 con 0x565173b4b5d0 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.178+0000 7f01a67fc640 1 -- 172.21.15.33:0/3142708069 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f01b0004aa0 con 0x565173b4b5d0 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.178+0000 7f01c76b3740 10 client.26093 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.178+0000 7f01c76b3740 10 client.26093 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.178+0000 7f01c76b3740 20 client.26093 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.178+0000 7f01c76b3740 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x565173bd0eb0 0x565173bd32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.178+0000 7f01c76b3740 4 client.26093 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:23:31.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.178+0000 7f01c76b3740 1 -- 172.21.15.33:0/3142708069 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565173ab8d70 con 0x565173bd0eb0 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.178+0000 7f01b7fff640 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x565173bd0eb0 0x565173bd32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01b7fff640 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x565173bd0eb0 0x565173bd32a0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bede66897ac4d138 server_cookie=5626e95050a501bd in_seq=0 out_seq=0 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01a67fc640 10 client.26093 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01a67fc640 1 -- 172.21.15.33:0/3142708069 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565173ab8d70 con 0x565173bd0eb0 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01a67fc640 10 client.26093 handle_command_reply: tid=0 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01c76b3740 1 client.26093 shutdown 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01c76b3740 1 -- 172.21.15.33:0/3142708069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4b5d0 msgr2=0x565173b5eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01c76b3740 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4b5d0 0x565173b5eba0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x565173a6c120 tx=0x7f01b0012cb0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01c76b3740 1 -- 172.21.15.33:0/3142708069 shutdown_connections 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01c76b3740 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x565173bd0eb0 0x565173bd32a0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01c76b3740 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565173b56370 0x565173b5a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01c76b3740 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565173b4cb40 0x565173b59ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01a67fc640 0 client.26093 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:23:31.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01c76b3740 1 --2- 172.21.15.33:0/3142708069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565173b4b5d0 0x565173b5eba0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01c76b3740 1 -- 172.21.15.33:0/3142708069 >> 172.21.15.33:0/3142708069 conn(0x565173b14310 msgr2=0x565173b464e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:31.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01c76b3740 1 -- 172.21.15.33:0/3142708069 shutdown_connections 2024-09-16T00:23:31.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01c76b3740 1 -- 172.21.15.33:0/3142708069 wait complete. 2024-09-16T00:23:31.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.179+0000 7f01c76b3740 20 client.26093 trim_cache size 0 max 16384 2024-09-16T00:23:31.185 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:23:31.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.180+0000 7f01c52da640 1 -- 172.21.15.33:0/3503090622 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01ac0a3240 msgr2=0x7f01ac0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:31.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.180+0000 7f01c52da640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01ac0a3240 0x7f01ac0a5700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009430 tx=0x7f01b8005c50 comp rx=0 tx=0).stop 2024-09-16T00:23:31.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.180+0000 7f01c52da640 1 -- 172.21.15.33:0/3503090622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01c0103660 msgr2=0x7f01c0198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:31.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.180+0000 7f01c52da640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01c0103660 0x7f01c0198910 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f01a800e9f0 tx=0x7f01a800eeb0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.180+0000 7f01c52da640 1 -- 172.21.15.33:0/3503090622 shutdown_connections 2024-09-16T00:23:31.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.180+0000 7f01c52da640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01ac0a3240 0x7f01ac0a5700 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.180+0000 7f01c52da640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01c010c170 0x7f01c019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.180+0000 7f01c52da640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01c0103660 0x7f01c0198910 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.180+0000 7f01c52da640 1 --2- 172.21.15.33:0/3503090622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c0102d40 0x7f01c01983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:31.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.180+0000 7f01c52da640 1 -- 172.21.15.33:0/3503090622 >> 172.21.15.33:0/3503090622 conn(0x7f01c00fc7a0 msgr2=0x7f01c00fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:31.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.180+0000 7f01c52da640 1 -- 172.21.15.33:0/3503090622 shutdown_connections 2024-09-16T00:23:31.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:31.180+0000 7f01c52da640 1 -- 172.21.15.33:0/3503090622 wait complete. 2024-09-16T00:23:31.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:31 smithi033 ceph-mon[31243]: pgmap v541: 129 pgs: 129 active+clean; 142 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 530 KiB/s wr, 79 op/s 2024-09-16T00:23:31.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:31 smithi060 ceph-mon[35185]: pgmap v541: 129 pgs: 129 active+clean; 142 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 530 KiB/s wr, 79 op/s 2024-09-16T00:23:31.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 373411 8.63 MB/sec execute 159 sec latency 20.286 ms 2024-09-16T00:23:31.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:31 smithi083 ceph-mon[35242]: pgmap v541: 129 pgs: 129 active+clean; 142 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 530 KiB/s wr, 79 op/s 2024-09-16T00:23:31.903 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:23:32.176 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:23:32.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 376417 8.69 MB/sec execute 160 sec latency 35.990 ms 2024-09-16T00:23:32.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.665+0000 7f2ffe2da640 1 -- 172.21.15.33:0/186122116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ff8102770 msgr2=0x7f2ff81067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:32.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.665+0000 7f2ffe2da640 1 --2- 172.21.15.33:0/186122116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ff8102770 0x7f2ff81067c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009900 tx=0x7f2fe802a410 comp rx=0 tx=0).stop 2024-09-16T00:23:32.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.665+0000 7f2ffe2da640 1 -- 172.21.15.33:0/186122116 shutdown_connections 2024-09-16T00:23:32.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.665+0000 7f2ffe2da640 1 --2- 172.21.15.33:0/186122116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ff8106ef0 0x7f2ff810ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.665+0000 7f2ffe2da640 1 --2- 172.21.15.33:0/186122116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ff8102770 0x7f2ff81067c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.665+0000 7f2ffe2da640 1 --2- 172.21.15.33:0/186122116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ff8101dc0 0x7f2ff81021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.665+0000 7f2ffe2da640 1 -- 172.21.15.33:0/186122116 >> 172.21.15.33:0/186122116 conn(0x7f2ff80fd880 msgr2=0x7f2ff80ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:32.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.665+0000 7f2ffe2da640 1 -- 172.21.15.33:0/186122116 shutdown_connections 2024-09-16T00:23:32.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.665+0000 7f2ffe2da640 1 -- 172.21.15.33:0/186122116 wait complete. 2024-09-16T00:23:32.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.665+0000 7f2ffe2da640 1 Processor -- start 2024-09-16T00:23:32.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.665+0000 7f2ffe2da640 1 -- start start 2024-09-16T00:23:32.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ffe2da640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ff8101dc0 0x7f2ff80744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ffe2da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ff8102770 0x7f2ff8074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ffe2da640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ff8106ef0 0x7f2ff8074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ffe2da640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff8111f50 con 0x7f2ff8101dc0 2024-09-16T00:23:32.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ffe2da640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff8111dd0 con 0x7f2ff8102770 2024-09-16T00:23:32.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ffe2da640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff81120d0 con 0x7f2ff8106ef0 2024-09-16T00:23:32.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ff8101dc0 0x7f2ff80744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ff8101dc0 0x7f2ff80744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59580/0 (socket says 172.21.15.33:59580) 2024-09-16T00:23:32.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ff7fff640 1 -- 172.21.15.33:0/1823915502 learned_addr learned my addr 172.21.15.33:0/1823915502 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:32.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ff77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ff8102770 0x7f2ff8074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ffc877640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ff8106ef0 0x7f2ff8074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ff7fff640 1 -- 172.21.15.33:0/1823915502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ff8106ef0 msgr2=0x7f2ff8074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:32.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ff8106ef0 0x7f2ff8074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ff7fff640 1 -- 172.21.15.33:0/1823915502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ff8102770 msgr2=0x7f2ff8074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:32.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ff8102770 0x7f2ff8074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ff7fff640 1 -- 172.21.15.33:0/1823915502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff81a6910 con 0x7f2ff8101dc0 2024-09-16T00:23:32.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ff8101dc0 0x7f2ff80744f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000d950 tx=0x7f2fe000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:32.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.666+0000 7f2ff57fa640 1 -- 172.21.15.33:0/1823915502 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0017030 con 0x7f2ff8101dc0 2024-09-16T00:23:32.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.667+0000 7f2ff57fa640 1 -- 172.21.15.33:0/1823915502 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fe0004020 con 0x7f2ff8101dc0 2024-09-16T00:23:32.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.667+0000 7f2ffe2da640 1 -- 172.21.15.33:0/1823915502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff81a6c00 con 0x7f2ff8101dc0 2024-09-16T00:23:32.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.667+0000 7f2ff57fa640 1 -- 172.21.15.33:0/1823915502 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0005650 con 0x7f2ff8101dc0 2024-09-16T00:23:32.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.667+0000 7f2ffe2da640 1 -- 172.21.15.33:0/1823915502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff81a7110 con 0x7f2ff8101dc0 2024-09-16T00:23:32.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.669+0000 7f2ff57fa640 1 -- 172.21.15.33:0/1823915502 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2fe00041d0 con 0x7f2ff8101dc0 2024-09-16T00:23:32.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.669+0000 7f30006b3740 1 Processor -- start 2024-09-16T00:23:32.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.669+0000 7f2ff57fa640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2fe40a3240 0x7f2fe40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.669+0000 7f30006b3740 1 -- start start 2024-09-16T00:23:32.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ffc877640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2fe40a3240 0x7f2fe40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ff57fa640 1 -- 172.21.15.33:0/1823915502 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2fe00cf700 con 0x7f2ff8101dc0 2024-09-16T00:23:32.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f30006b3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512fb40 0x55c2b512e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f30006b3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b512e5d0 0x55c2b512ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f30006b3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b5139370 0x55c2b513b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f30006b3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c2b50942f0 con 0x55c2b5139370 2024-09-16T00:23:32.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f30006b3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c2b503bcf0 con 0x55c2b512fb40 2024-09-16T00:23:32.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f30006b3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c2b50902b0 con 0x55c2b512e5d0 2024-09-16T00:23:32.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512fb40 0x55c2b512e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ff77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b512e5d0 0x55c2b512ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512fb40 0x55c2b512e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:58602/0 (socket says 172.21.15.33:58602) 2024-09-16T00:23:32.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ff7fff640 1 -- 172.21.15.33:0/3620313178 learned_addr learned my addr 172.21.15.33:0/3620313178 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:32.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ffc877640 1 --2- 172.21.15.33:0/3620313178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b5139370 0x55c2b513b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ff7fff640 1 -- 172.21.15.33:0/3620313178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b512e5d0 msgr2=0x55c2b512ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:32.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/3620313178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b512e5d0 0x55c2b512ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ff7fff640 1 -- 172.21.15.33:0/3620313178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b5139370 msgr2=0x55c2b513b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:32.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/3620313178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b5139370 0x55c2b513b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ff7fff640 1 -- 172.21.15.33:0/3620313178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2b50203a0 con 0x55c2b512fb40 2024-09-16T00:23:32.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ffc877640 1 --2- 172.21.15.33:0/3620313178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b5139370 0x55c2b513b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:23:32.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ffc877640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2fe40a3240 0x7f2fe40a5700 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00099a0 tx=0x7f2ff0011810 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:32.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.670+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/3620313178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512fb40 0x55c2b512e090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2fe009b830 tx=0x7f2fe009bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=280842f64ca7e978 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:32.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.671+0000 7f2fee7fc640 1 -- 172.21.15.33:0/3620313178 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe009fc50 con 0x55c2b512fb40 2024-09-16T00:23:32.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.671+0000 7f2fee7fc640 1 -- 172.21.15.33:0/3620313178 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fe00a4510 con 0x55c2b512fb40 2024-09-16T00:23:32.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.671+0000 7f2fee7fc640 1 -- 172.21.15.33:0/3620313178 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe009cbd0 con 0x55c2b512fb40 2024-09-16T00:23:32.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.671+0000 7f30006b3740 1 -- 172.21.15.33:0/3620313178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512fb40 msgr2=0x55c2b512e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:32.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.671+0000 7f30006b3740 1 --2- 172.21.15.33:0/3620313178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512fb40 0x55c2b512e090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2fe009b830 tx=0x7f2fe009bfa0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.671+0000 7f30006b3740 1 -- 172.21.15.33:0/3620313178 shutdown_connections 2024-09-16T00:23:32.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.671+0000 7f30006b3740 1 --2- 172.21.15.33:0/3620313178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b5139370 0x55c2b513b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.671+0000 7f30006b3740 1 --2- 172.21.15.33:0/3620313178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b512e5d0 0x55c2b512ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.671+0000 7f30006b3740 1 --2- 172.21.15.33:0/3620313178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512fb40 0x55c2b512e090 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.671+0000 7f30006b3740 1 -- 172.21.15.33:0/3620313178 >> 172.21.15.33:0/3620313178 conn(0x55c2b50f7310 msgr2=0x55c2b512ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:32.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.671+0000 7f30006b3740 1 -- 172.21.15.33:0/3620313178 shutdown_connections 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.671+0000 7f30006b3740 1 -- 172.21.15.33:0/3620313178 wait complete. 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.672+0000 7f30006b3740 1 -- start start 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.672+0000 7f30006b3740 1 Processor -- start 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.672+0000 7f30006b3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512e5d0 0x55c2b5138d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.672+0000 7f30006b3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512fb40 0x55c2b5136030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.672+0000 7f30006b3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b5139370 0x55c2b5136640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f30006b3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c2b513c070 con 0x55c2b512fb40 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f30006b3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c2b504f060 con 0x55c2b512e5d0 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f30006b3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c2b513c1f0 con 0x55c2b5139370 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f30006b3740 10 client.0 _ll_register_callbacks cb 0x55c2b4da28c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2ff77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512fb40 0x55c2b5136030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512e5d0 0x55c2b5138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2ff77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512fb40 0x55c2b5136030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59608/0 (socket says 172.21.15.33:59608) 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2ff77fe640 1 -- 172.21.15.33:0/739585587 learned_addr learned my addr 172.21.15.33:0/739585587 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2ffc877640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b5139370 0x55c2b5136640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2ff7fff640 1 -- 172.21.15.33:0/739585587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b5139370 msgr2=0x55c2b5136640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/739585587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b5139370 0x55c2b5136640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2ff7fff640 1 -- 172.21.15.33:0/739585587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512fb40 msgr2=0x55c2b5136030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/739585587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512fb40 0x55c2b5136030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2ff7fff640 1 -- 172.21.15.33:0/739585587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2b4e053c0 con 0x55c2b512e5d0 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2ff77fe640 1 --2- 172.21.15.33:0/739585587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512fb40 0x55c2b5136030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/739585587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512e5d0 0x55c2b5138d80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2fe009b830 tx=0x7f2fe009fc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=882efa7667c160fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2fd77fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2fd77fe640 1 -- 172.21.15.33:0/739585587 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe00ba270 con 0x55c2b512e5d0 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f2fd77fe640 1 -- 172.21.15.33:0/739585587 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2fe00cf440 con 0x55c2b512e5d0 2024-09-16T00:23:32.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.673+0000 7f30006b3740 1 -- 172.21.15.33:0/739585587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fe00a4510 con 0x55c2b512e5d0 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.674+0000 7f2fd77fe640 1 -- 172.21.15.33:0/739585587 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2fe0097f10 con 0x55c2b512e5d0 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.674+0000 7f30006b3740 10 client.26105 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.674+0000 7f30006b3740 1 -- 172.21.15.33:0/739585587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c2b5147b00 con 0x55c2b512e5d0 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.674+0000 7f2fd77fe640 1 -- 172.21.15.33:0/739585587 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fe00a4510 con 0x55c2b512e5d0 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.674+0000 7f2fd77fe640 1 -- 172.21.15.33:0/739585587 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f2fe0062c70 con 0x55c2b512e5d0 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.674+0000 7f30006b3740 10 client.26105 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.674+0000 7f30006b3740 10 client.26105 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.674+0000 7f30006b3740 20 client.26105 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.674+0000 7f30006b3740 1 --2- 172.21.15.33:0/739585587 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c2b5148e50 0x55c2b514b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.674+0000 7f30006b3740 4 client.26105 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.674+0000 7f30006b3740 1 -- 172.21.15.33:0/739585587 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2b50a1860 con 0x55c2b5148e50 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.674+0000 7f2ff77fe640 1 --2- 172.21.15.33:0/739585587 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c2b5148e50 0x55c2b514b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.675+0000 7f2fd77fe640 10 client.26105 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:23:32.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.675+0000 7f2ff77fe640 1 --2- 172.21.15.33:0/739585587 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c2b5148e50 0x55c2b514b240 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67e04510ca2d1b10 server_cookie=8f46c9faa3d4f71 in_seq=0 out_seq=0 2024-09-16T00:23:32.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.675+0000 7f2fd77fe640 1 -- 172.21.15.33:0/739585587 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c2b50a1860 con 0x55c2b5148e50 2024-09-16T00:23:32.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.675+0000 7f2fd77fe640 10 client.26105 handle_command_reply: tid=0 2024-09-16T00:23:32.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.675+0000 7f30006b3740 1 client.26105 shutdown 2024-09-16T00:23:32.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.676+0000 7f30006b3740 1 -- 172.21.15.33:0/739585587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512e5d0 msgr2=0x55c2b5138d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:32.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.676+0000 7f30006b3740 1 --2- 172.21.15.33:0/739585587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512e5d0 0x55c2b5138d80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2fe009b830 tx=0x7f2fe009fc50 comp rx=0 tx=0).stop 2024-09-16T00:23:32.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.676+0000 7f30006b3740 1 -- 172.21.15.33:0/739585587 shutdown_connections 2024-09-16T00:23:32.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.676+0000 7f30006b3740 1 --2- 172.21.15.33:0/739585587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b5139370 0x55c2b5136640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.676+0000 7f30006b3740 1 --2- 172.21.15.33:0/739585587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512fb40 0x55c2b5136030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.676+0000 7f30006b3740 1 --2- 172.21.15.33:0/739585587 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c2b5148e50 0x55c2b514b240 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.676+0000 7f30006b3740 1 --2- 172.21.15.33:0/739585587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512e5d0 0x55c2b5138d80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.676+0000 7f2fd77fe640 0 client.26105 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:23:32.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.676+0000 7f30006b3740 1 -- 172.21.15.33:0/739585587 >> 172.21.15.33:0/739585587 conn(0x55c2b50f7310 msgr2=0x55c2b51309c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:32.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.676+0000 7f30006b3740 1 -- 172.21.15.33:0/739585587 shutdown_connections 2024-09-16T00:23:32.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.676+0000 7f30006b3740 1 -- 172.21.15.33:0/739585587 wait complete. 2024-09-16T00:23:32.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.676+0000 7f30006b3740 20 client.26105 trim_cache size 0 max 16384 2024-09-16T00:23:32.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.690+0000 7f30006b3740 1 Processor -- start 2024-09-16T00:23:32.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.690+0000 7f30006b3740 1 -- start start 2024-09-16T00:23:32.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.690+0000 7f30006b3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 0x55c2b51454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.690+0000 7f30006b3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b512fb40 0x55c2b5131120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.690+0000 7f30006b3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b5139370 0x55c2b5131660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.690+0000 7f30006b3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff81b3b40 con 0x55c2b512e5d0 2024-09-16T00:23:32.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f30006b3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff81b1410 con 0x55c2b5139370 2024-09-16T00:23:32.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f30006b3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff81af260 con 0x55c2b512fb40 2024-09-16T00:23:32.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 0x55c2b51454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ffc877640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b5139370 0x55c2b5131660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 0x55c2b51454e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59616/0 (socket says 172.21.15.33:59616) 2024-09-16T00:23:32.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ff7fff640 1 -- 172.21.15.33:0/1477844917 learned_addr learned my addr 172.21.15.33:0/1477844917 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:32.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ff77fe640 1 --2- 172.21.15.33:0/1477844917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b512fb40 0x55c2b5131120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ff7fff640 1 -- 172.21.15.33:0/1477844917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b512fb40 msgr2=0x55c2b5131120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:32.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/1477844917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b512fb40 0x55c2b5131120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ff7fff640 1 -- 172.21.15.33:0/1477844917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b5139370 msgr2=0x55c2b5131660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:32.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/1477844917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b5139370 0x55c2b5131660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ff7fff640 1 -- 172.21.15.33:0/1477844917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2b512a510 con 0x55c2b512e5d0 2024-09-16T00:23:32.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ffc877640 1 --2- 172.21.15.33:0/1477844917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b5139370 0x55c2b5131660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:32.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ff77fe640 1 --2- 172.21.15.33:0/1477844917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b512fb40 0x55c2b5131120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:23:32.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/1477844917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 0x55c2b51454e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0005c60 tx=0x7f2fe009bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baad250e1005acb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:32.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.691+0000 7f2fedffb640 1 -- 172.21.15.33:0/1477844917 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0086070 con 0x55c2b512e5d0 2024-09-16T00:23:32.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.692+0000 7f2fedffb640 1 -- 172.21.15.33:0/1477844917 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fe00b9070 con 0x55c2b512e5d0 2024-09-16T00:23:32.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.692+0000 7f2fedffb640 1 -- 172.21.15.33:0/1477844917 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe00ba270 con 0x55c2b512e5d0 2024-09-16T00:23:32.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.692+0000 7f30006b3740 1 -- 172.21.15.33:0/1477844917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 msgr2=0x55c2b51454e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:32.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.692+0000 7f30006b3740 1 --2- 172.21.15.33:0/1477844917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 0x55c2b51454e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0005c60 tx=0x7f2fe009bfa0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.692+0000 7f30006b3740 1 -- 172.21.15.33:0/1477844917 shutdown_connections 2024-09-16T00:23:32.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.692+0000 7f30006b3740 1 --2- 172.21.15.33:0/1477844917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b5139370 0x55c2b5131660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.692+0000 7f30006b3740 1 --2- 172.21.15.33:0/1477844917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b512fb40 0x55c2b5131120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.692+0000 7f30006b3740 1 --2- 172.21.15.33:0/1477844917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 0x55c2b51454e0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.692+0000 7f30006b3740 1 -- 172.21.15.33:0/1477844917 >> 172.21.15.33:0/1477844917 conn(0x55c2b50f7310 msgr2=0x55c2b5134ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:32.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.692+0000 7f30006b3740 1 -- 172.21.15.33:0/1477844917 shutdown_connections 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.692+0000 7f30006b3740 1 -- 172.21.15.33:0/1477844917 wait complete. 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f30006b3740 1 -- start start 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f30006b3740 1 Processor -- start 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f30006b3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 0x55c2b514e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f30006b3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512fb40 0x55c2b514b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f30006b3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b5139370 0x55c2b514bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f30006b3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff81af8d0 con 0x55c2b512e5d0 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f30006b3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff81b0d40 con 0x55c2b512fb40 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f30006b3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff81affa0 con 0x55c2b5139370 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f30006b3740 10 client.0 _ll_register_callbacks cb 0x7f2fe8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 0x55c2b514e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 0x55c2b514e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59618/0 (socket says 172.21.15.33:59618) 2024-09-16T00:23:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f2ff7fff640 1 -- 172.21.15.33:0/1636807652 learned_addr learned my addr 172.21.15.33:0/1636807652 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f2ff77fe640 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512fb40 0x55c2b514b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.693+0000 7f2ffc877640 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b5139370 0x55c2b514bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f2ff7fff640 1 -- 172.21.15.33:0/1636807652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b5139370 msgr2=0x55c2b514bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b5139370 0x55c2b514bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f2ff7fff640 1 -- 172.21.15.33:0/1636807652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512fb40 msgr2=0x55c2b514b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512fb40 0x55c2b514b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f2ff7fff640 1 -- 172.21.15.33:0/1636807652 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2b514c440 con 0x55c2b512e5d0 2024-09-16T00:23:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f2ff7fff640 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 0x55c2b514e670 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0005fd0 tx=0x7f2fe009b470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4822295a550a529a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f2fd77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:23:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f2fd77fe640 1 -- 172.21.15.33:0/1636807652 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0060db0 con 0x55c2b512e5d0 2024-09-16T00:23:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f2fd77fe640 1 -- 172.21.15.33:0/1636807652 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2fe0084880 con 0x55c2b512e5d0 2024-09-16T00:23:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f30006b3740 1 -- 172.21.15.33:0/1636807652 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fe00b9070 con 0x55c2b512e5d0 2024-09-16T00:23:32.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f2fd77fe640 1 -- 172.21.15.33:0/1636807652 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2fe00a9050 con 0x55c2b512e5d0 2024-09-16T00:23:32.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f30006b3740 10 client.16581 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:32.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.694+0000 7f30006b3740 1 -- 172.21.15.33:0/1636807652 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c2b514c440 con 0x55c2b512e5d0 2024-09-16T00:23:32.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.695+0000 7f2fd77fe640 1 -- 172.21.15.33:0/1636807652 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fe00b9070 con 0x55c2b512e5d0 2024-09-16T00:23:32.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.695+0000 7f2fd77fe640 1 -- 172.21.15.33:0/1636807652 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f2fe009acf0 con 0x55c2b512e5d0 2024-09-16T00:23:32.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.695+0000 7f30006b3740 10 client.16581 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:32.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.695+0000 7f30006b3740 10 client.16581 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:23:32.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.695+0000 7f30006b3740 20 client.16581 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:32.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.695+0000 7f30006b3740 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c2b51b36e0 0x55c2b51b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:32.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.695+0000 7f30006b3740 4 client.16581 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:23:32.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.695+0000 7f30006b3740 1 -- 172.21.15.33:0/1636807652 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c2b509bd70 con 0x55c2b51b36e0 2024-09-16T00:23:32.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.695+0000 7f2ff77fe640 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c2b51b36e0 0x55c2b51b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:32.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.695+0000 7f2fd77fe640 10 client.16581 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:23:32.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.695+0000 7f2ff77fe640 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c2b51b36e0 0x55c2b51b5ad0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b38c4008ab69fdf server_cookie=bd4fbe9d879a8167 in_seq=0 out_seq=0 2024-09-16T00:23:32.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f2fd77fe640 1 -- 172.21.15.33:0/1636807652 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c2b509bd70 con 0x55c2b51b36e0 2024-09-16T00:23:32.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f2fd77fe640 10 client.16581 handle_command_reply: tid=0 2024-09-16T00:23:32.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f30006b3740 1 client.16581 shutdown 2024-09-16T00:23:32.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f30006b3740 1 -- 172.21.15.33:0/1636807652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 msgr2=0x55c2b514e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:32.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f30006b3740 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 0x55c2b514e670 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0005fd0 tx=0x7f2fe009b470 comp rx=0 tx=0).stop 2024-09-16T00:23:32.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f30006b3740 1 -- 172.21.15.33:0/1636807652 shutdown_connections 2024-09-16T00:23:32.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f30006b3740 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2b5139370 0x55c2b514bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f30006b3740 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c2b51b36e0 0x55c2b51b5ad0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f30006b3740 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c2b512fb40 0x55c2b514b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f2fd77fe640 0 client.16581 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:23:32.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f30006b3740 1 --2- 172.21.15.33:0/1636807652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2b512e5d0 0x55c2b514e670 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f30006b3740 1 -- 172.21.15.33:0/1636807652 >> 172.21.15.33:0/1636807652 conn(0x55c2b50f7310 msgr2=0x55c2b51347d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:32.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f30006b3740 1 -- 172.21.15.33:0/1636807652 shutdown_connections 2024-09-16T00:23:32.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f30006b3740 1 -- 172.21.15.33:0/1636807652 wait complete. 2024-09-16T00:23:32.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.696+0000 7f30006b3740 20 client.16581 trim_cache size 0 max 16384 2024-09-16T00:23:32.704 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:23:32.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.697+0000 7f2ffe2da640 1 -- 172.21.15.33:0/1823915502 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2fe40a3240 msgr2=0x7f2fe40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:32.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.697+0000 7f2ffe2da640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2fe40a3240 0x7f2fe40a5700 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00099a0 tx=0x7f2ff0011810 comp rx=0 tx=0).stop 2024-09-16T00:23:32.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.697+0000 7f2ffe2da640 1 -- 172.21.15.33:0/1823915502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ff8101dc0 msgr2=0x7f2ff80744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:32.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.697+0000 7f2ffe2da640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ff8101dc0 0x7f2ff80744f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000d950 tx=0x7f2fe000dd50 comp rx=0 tx=0).stop 2024-09-16T00:23:32.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.697+0000 7f2ffe2da640 1 -- 172.21.15.33:0/1823915502 shutdown_connections 2024-09-16T00:23:32.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.697+0000 7f2ffe2da640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2fe40a3240 0x7f2fe40a5700 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.697+0000 7f2ffe2da640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ff8106ef0 0x7f2ff8074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.697+0000 7f2ffe2da640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ff8102770 0x7f2ff8074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.697+0000 7f2ffe2da640 1 --2- 172.21.15.33:0/1823915502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ff8101dc0 0x7f2ff80744f0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:32.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.697+0000 7f2ffe2da640 1 -- 172.21.15.33:0/1823915502 >> 172.21.15.33:0/1823915502 conn(0x7f2ff80fd880 msgr2=0x7f2ff80fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:32.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.697+0000 7f2ffe2da640 1 -- 172.21.15.33:0/1823915502 shutdown_connections 2024-09-16T00:23:32.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:32.697+0000 7f2ffe2da640 1 -- 172.21.15.33:0/1823915502 wait complete. 2024-09-16T00:23:33.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:33 smithi033 ceph-mon[31243]: pgmap v542: 129 pgs: 129 active+clean; 144 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-09-16T00:23:33.412 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:23:33.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:33 smithi060 ceph-mon[35185]: pgmap v542: 129 pgs: 129 active+clean; 144 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-09-16T00:23:33.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 379718 8.75 MB/sec execute 161 sec latency 23.974 ms 2024-09-16T00:23:33.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:33 smithi083 ceph-mon[35242]: pgmap v542: 129 pgs: 129 active+clean; 144 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-09-16T00:23:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:23:34.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.224+0000 7f05543e0640 1 -- 172.21.15.33:0/134944183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f054c109450 msgr2=0x7f054c110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:34.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.224+0000 7f05543e0640 1 --2- 172.21.15.33:0/134944183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f054c109450 0x7f054c110250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0548009900 tx=0x7f054802a3f0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.224+0000 7f05543e0640 1 -- 172.21.15.33:0/134944183 shutdown_connections 2024-09-16T00:23:34.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.224+0000 7f05543e0640 1 --2- 172.21.15.33:0/134944183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f054c11b140 0x7f054c11d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.224+0000 7f05543e0640 1 --2- 172.21.15.33:0/134944183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f054c109830 0x7f054c109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.224+0000 7f05543e0640 1 --2- 172.21.15.33:0/134944183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f054c109450 0x7f054c110250 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.224+0000 7f05543e0640 1 -- 172.21.15.33:0/134944183 >> 172.21.15.33:0/134944183 conn(0x7f054c06e140 msgr2=0x7f054c06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:34.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.224+0000 7f05543e0640 1 -- 172.21.15.33:0/134944183 shutdown_connections 2024-09-16T00:23:34.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.224+0000 7f05543e0640 1 -- 172.21.15.33:0/134944183 wait complete. 2024-09-16T00:23:34.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f05543e0640 1 Processor -- start 2024-09-16T00:23:34.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f05543e0640 1 -- start start 2024-09-16T00:23:34.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f05543e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f054c109450 0x7f054c1a72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f05543e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f054c109830 0x7f054c1a77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f05543e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f054c11b140 0x7f054c1abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f05543e0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f054c120df0 con 0x7f054c109450 2024-09-16T00:23:34.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f05543e0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f054c120c70 con 0x7f054c11b140 2024-09-16T00:23:34.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f05543e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f054c120f70 con 0x7f054c109830 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f055217c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f054c109450 0x7f054c1a72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f055217c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f054c109450 0x7f054c1a72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59642/0 (socket says 172.21.15.33:59642) 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f055217c640 1 -- 172.21.15.33:0/2305565559 learned_addr learned my addr 172.21.15.33:0/2305565559 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f055197b640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f054c109830 0x7f054c1a77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f055297d640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f054c11b140 0x7f054c1abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f055217c640 1 -- 172.21.15.33:0/2305565559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f054c109830 msgr2=0x7f054c1a77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f055217c640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f054c109830 0x7f054c1a77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.225+0000 7f055217c640 1 -- 172.21.15.33:0/2305565559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f054c11b140 msgr2=0x7f054c1abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.226+0000 7f055217c640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f054c11b140 0x7f054c1abb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.226+0000 7f055217c640 1 -- 172.21.15.33:0/2305565559 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c1ac260 con 0x7f054c109450 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.226+0000 7f055297d640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f054c11b140 0x7f054c1abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.226+0000 7f055217c640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f054c109450 0x7f054c1a72b0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0548004770 tx=0x7f0548003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:34.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.226+0000 7f05437fe640 1 -- 172.21.15.33:0/2305565559 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054803f070 con 0x7f054c109450 2024-09-16T00:23:34.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.226+0000 7f05543e0640 1 -- 172.21.15.33:0/2305565559 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f054c1ac4f0 con 0x7f054c109450 2024-09-16T00:23:34.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.226+0000 7f05437fe640 1 -- 172.21.15.33:0/2305565559 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0548031ca0 con 0x7f054c109450 2024-09-16T00:23:34.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.226+0000 7f05437fe640 1 -- 172.21.15.33:0/2305565559 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548035030 con 0x7f054c109450 2024-09-16T00:23:34.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.226+0000 7f05543e0640 1 -- 172.21.15.33:0/2305565559 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f054c1b3e30 con 0x7f054c109450 2024-09-16T00:23:34.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.228+0000 7f05437fe640 1 -- 172.21.15.33:0/2305565559 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0548042070 con 0x7f054c109450 2024-09-16T00:23:34.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f05437fe640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f05380a3240 0x7f05380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f05567b9740 1 Processor -- start 2024-09-16T00:23:34.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f05437fe640 1 -- 172.21.15.33:0/2305565559 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f05480f03b0 con 0x7f054c109450 2024-09-16T00:23:34.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f055197b640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f05380a3240 0x7f05380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f05567b9740 1 -- start start 2024-09-16T00:23:34.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f05567b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d61b40 0x55b6f0d60090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f05567b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 0x55b6f0d60a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f05567b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d6d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f05567b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0cc62f0 con 0x55b6f0d6b370 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f05567b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0c6dcf0 con 0x55b6f0d605d0 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f05567b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0cc22b0 con 0x55b6f0d61b40 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f055217c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d61b40 0x55b6f0d60090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.229+0000 7f055297d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d6d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f055297d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d6d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59650/0 (socket says 172.21.15.33:59650) 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f055217c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d61b40 0x55b6f0d60090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36904/0 (socket says 172.21.15.33:36904) 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f055197b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 0x55b6f0d60a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f055297d640 1 -- 172.21.15.33:0/3785354444 learned_addr learned my addr 172.21.15.33:0/3785354444 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f055197b640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f05380a3240 0x7f05380a5700 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f054c11d8d0 tx=0x7f053c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f055217c640 1 -- 172.21.15.33:0/3785354444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 msgr2=0x55b6f0d60a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:34.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f055217c640 1 --2- 172.21.15.33:0/3785354444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 0x55b6f0d60a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f055217c640 1 -- 172.21.15.33:0/3785354444 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 msgr2=0x55b6f0d6d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:34.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f055217c640 1 --2- 172.21.15.33:0/3785354444 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d6d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f055217c640 1 -- 172.21.15.33:0/3785354444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6f0c523a0 con 0x55b6f0d61b40 2024-09-16T00:23:34.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f055297d640 1 --2- 172.21.15.33:0/3785354444 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d6d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:34.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f055217c640 1 --2- 172.21.15.33:0/3785354444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d61b40 0x55b6f0d60090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0548000c00 tx=0x7f054802a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e193a2ebc10f5418 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:34.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f0540dea640 1 -- 172.21.15.33:0/3785354444 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548049070 con 0x55b6f0d61b40 2024-09-16T00:23:34.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f0540dea640 1 -- 172.21.15.33:0/3785354444 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05480f0170 con 0x55b6f0d61b40 2024-09-16T00:23:34.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.230+0000 7f0540dea640 1 -- 172.21.15.33:0/3785354444 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054803b4f0 con 0x55b6f0d61b40 2024-09-16T00:23:34.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.231+0000 7f05567b9740 1 -- 172.21.15.33:0/3785354444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d61b40 msgr2=0x55b6f0d60090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:34.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.231+0000 7f05567b9740 1 --2- 172.21.15.33:0/3785354444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d61b40 0x55b6f0d60090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0548000c00 tx=0x7f054802a3f0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.231+0000 7f05567b9740 1 -- 172.21.15.33:0/3785354444 shutdown_connections 2024-09-16T00:23:34.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.231+0000 7f05567b9740 1 --2- 172.21.15.33:0/3785354444 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d6d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.231+0000 7f05567b9740 1 --2- 172.21.15.33:0/3785354444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 0x55b6f0d60a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.231+0000 7f05567b9740 1 --2- 172.21.15.33:0/3785354444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d61b40 0x55b6f0d60090 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.231+0000 7f05567b9740 1 -- 172.21.15.33:0/3785354444 >> 172.21.15.33:0/3785354444 conn(0x55b6f0d29310 msgr2=0x55b6f0d5cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:34.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.231+0000 7f05567b9740 1 -- 172.21.15.33:0/3785354444 shutdown_connections 2024-09-16T00:23:34.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.231+0000 7f05567b9740 1 -- 172.21.15.33:0/3785354444 wait complete. 2024-09-16T00:23:34.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.231+0000 7f05567b9740 1 -- start start 2024-09-16T00:23:34.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f05567b9740 1 Processor -- start 2024-09-16T00:23:34.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f05567b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 0x55b6f0d6add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f05567b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d61b40 0x55b6f0d68080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f05567b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d6b370 0x55b6f0d68810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f05567b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0d6de30 con 0x55b6f0d61b40 2024-09-16T00:23:34.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f05567b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0c81060 con 0x55b6f0d605d0 2024-09-16T00:23:34.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f05567b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0d6dfb0 con 0x55b6f0d6b370 2024-09-16T00:23:34.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f05567b9740 10 client.0 _ll_register_callbacks cb 0x55b6f09d48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:34.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f055197b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d61b40 0x55b6f0d68080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f055197b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d61b40 0x55b6f0d68080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59658/0 (socket says 172.21.15.33:59658) 2024-09-16T00:23:34.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f055197b640 1 -- 172.21.15.33:0/2558373958 learned_addr learned my addr 172.21.15.33:0/2558373958 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:34.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f055217c640 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 0x55b6f0d6add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f055297d640 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d6b370 0x55b6f0d68810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f055197b640 1 -- 172.21.15.33:0/2558373958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d6b370 msgr2=0x55b6f0d68810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:34.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f055197b640 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d6b370 0x55b6f0d68810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f055197b640 1 -- 172.21.15.33:0/2558373958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 msgr2=0x55b6f0d6add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:34.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f055197b640 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 0x55b6f0d6add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.232+0000 7f055197b640 1 -- 172.21.15.33:0/2558373958 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6f0a373c0 con 0x55b6f0d61b40 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.233+0000 7f055197b640 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d61b40 0x55b6f0d68080 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f053c0038c0 tx=0x7f053c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2c356de2218d15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.233+0000 7f0535ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.233+0000 7f0535ffb640 1 -- 172.21.15.33:0/2558373958 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c005020 con 0x55b6f0d61b40 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.233+0000 7f0535ffb640 1 -- 172.21.15.33:0/2558373958 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f053c0051c0 con 0x55b6f0d61b40 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.233+0000 7f05567b9740 1 -- 172.21.15.33:0/2558373958 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05480f0170 con 0x55b6f0d61b40 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.233+0000 7f0535ffb640 1 -- 172.21.15.33:0/2558373958 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f053c0053c0 con 0x55b6f0d61b40 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.233+0000 7f05567b9740 10 client.16593 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.233+0000 7f05567b9740 1 -- 172.21.15.33:0/2558373958 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b6f0d798a0 con 0x55b6f0d61b40 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.233+0000 7f0535ffb640 1 -- 172.21.15.33:0/2558373958 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05480f0170 con 0x55b6f0d61b40 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.234+0000 7f0535ffb640 1 -- 172.21.15.33:0/2558373958 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f053c00e8b0 con 0x55b6f0d61b40 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.234+0000 7f05567b9740 10 client.16593 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:34.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.234+0000 7f05567b9740 10 client.16593 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:23:34.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.234+0000 7f05567b9740 20 client.16593 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:34.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.234+0000 7f05567b9740 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b6f0d7ad60 0x55b6f0d7d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.234+0000 7f05567b9740 4 client.16593 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:23:34.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.234+0000 7f05567b9740 1 -- 172.21.15.33:0/2558373958 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6f0cd3860 con 0x55b6f0d7ad60 2024-09-16T00:23:34.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.234+0000 7f055217c640 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b6f0d7ad60 0x55b6f0d7d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.234+0000 7f055217c640 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b6f0d7ad60 0x55b6f0d7d1b0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1322e667e3fe2758 server_cookie=75a0628f92cdace1 in_seq=0 out_seq=0 2024-09-16T00:23:34.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.234+0000 7f0535ffb640 10 client.16593 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:34.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f0535ffb640 1 -- 172.21.15.33:0/2558373958 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b6f0cd3860 con 0x55b6f0d7ad60 2024-09-16T00:23:34.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f0535ffb640 10 client.16593 handle_command_reply: tid=0 2024-09-16T00:23:34.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f05567b9740 1 client.16593 shutdown 2024-09-16T00:23:34.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f05567b9740 1 -- 172.21.15.33:0/2558373958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d61b40 msgr2=0x55b6f0d68080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:34.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f05567b9740 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d61b40 0x55b6f0d68080 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f053c0038c0 tx=0x7f053c004730 comp rx=0 tx=0).stop 2024-09-16T00:23:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f05567b9740 1 -- 172.21.15.33:0/2558373958 shutdown_connections 2024-09-16T00:23:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f05567b9740 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b6f0d7ad60 0x55b6f0d7d1b0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f05567b9740 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d6b370 0x55b6f0d68810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f05567b9740 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d61b40 0x55b6f0d68080 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f05567b9740 1 --2- 172.21.15.33:0/2558373958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 0x55b6f0d6add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f05567b9740 1 -- 172.21.15.33:0/2558373958 >> 172.21.15.33:0/2558373958 conn(0x55b6f0d29310 msgr2=0x55b6f0d62960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f0535ffb640 0 client.16593 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f05567b9740 1 -- 172.21.15.33:0/2558373958 shutdown_connections 2024-09-16T00:23:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f05567b9740 1 -- 172.21.15.33:0/2558373958 wait complete. 2024-09-16T00:23:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.235+0000 7f05567b9740 20 client.16593 trim_cache size 0 max 16384 2024-09-16T00:23:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f05567b9740 1 Processor -- start 2024-09-16T00:23:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f05567b9740 1 -- start start 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f05567b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 0x55b6f0d67940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f05567b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d66970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f05567b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d63a20 0x55b6f0d66eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f05567b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1c07d0 con 0x55b6f0d6b370 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f05567b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1be0a0 con 0x55b6f0d605d0 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f05567b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1bbef0 con 0x55b6f0d63a20 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f055197b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d66970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f055197b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d66970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59674/0 (socket says 172.21.15.33:59674) 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f055197b640 1 -- 172.21.15.33:0/3260554195 learned_addr learned my addr 172.21.15.33:0/3260554195 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f055217c640 1 --2- 172.21.15.33:0/3260554195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 0x55b6f0d67940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.250+0000 7f055297d640 1 --2- 172.21.15.33:0/3260554195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d63a20 0x55b6f0d66eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.251+0000 7f055197b640 1 -- 172.21.15.33:0/3260554195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d63a20 msgr2=0x55b6f0d66eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:34.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.251+0000 7f055197b640 1 --2- 172.21.15.33:0/3260554195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d63a20 0x55b6f0d66eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.251+0000 7f055197b640 1 -- 172.21.15.33:0/3260554195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 msgr2=0x55b6f0d67940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:34.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.251+0000 7f055197b640 1 --2- 172.21.15.33:0/3260554195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 0x55b6f0d67940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.251+0000 7f055197b640 1 -- 172.21.15.33:0/3260554195 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6f0d63120 con 0x55b6f0d6b370 2024-09-16T00:23:34.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.251+0000 7f055197b640 1 --2- 172.21.15.33:0/3260554195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d66970 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f053c00ecd0 tx=0x7f053c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6294bd929c027136 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:34.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.251+0000 7f0537fff640 1 -- 172.21.15.33:0/3260554195 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c038020 con 0x55b6f0d6b370 2024-09-16T00:23:34.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.251+0000 7f0537fff640 1 -- 172.21.15.33:0/3260554195 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f053c027470 con 0x55b6f0d6b370 2024-09-16T00:23:34.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.251+0000 7f0537fff640 1 -- 172.21.15.33:0/3260554195 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c0279e0 con 0x55b6f0d6b370 2024-09-16T00:23:34.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.252+0000 7f05567b9740 1 -- 172.21.15.33:0/3260554195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 msgr2=0x55b6f0d66970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:34.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.252+0000 7f05567b9740 1 --2- 172.21.15.33:0/3260554195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d66970 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f053c00ecd0 tx=0x7f053c009240 comp rx=0 tx=0).stop 2024-09-16T00:23:34.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.252+0000 7f05567b9740 1 -- 172.21.15.33:0/3260554195 shutdown_connections 2024-09-16T00:23:34.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.252+0000 7f05567b9740 1 --2- 172.21.15.33:0/3260554195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d63a20 0x55b6f0d66eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.252+0000 7f05567b9740 1 --2- 172.21.15.33:0/3260554195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d66970 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.252+0000 7f05567b9740 1 --2- 172.21.15.33:0/3260554195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d605d0 0x55b6f0d67940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.252+0000 7f05567b9740 1 -- 172.21.15.33:0/3260554195 >> 172.21.15.33:0/3260554195 conn(0x55b6f0d29310 msgr2=0x55b6f0d63310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.252+0000 7f05567b9740 1 -- 172.21.15.33:0/3260554195 shutdown_connections 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.252+0000 7f05567b9740 1 -- 172.21.15.33:0/3260554195 wait complete. 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f05567b9740 1 -- start start 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f05567b9740 1 Processor -- start 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f05567b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d605d0 0x55b6f0d73300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f05567b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d63a20 0x55b6f0d73840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f05567b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d64900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f05567b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1bc560 con 0x55b6f0d6b370 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f05567b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1bd9d0 con 0x55b6f0d63a20 2024-09-16T00:23:34.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f05567b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1bcc30 con 0x55b6f0d605d0 2024-09-16T00:23:34.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f05567b9740 10 client.0 _ll_register_callbacks cb 0x7f053c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:34.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f055217c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d605d0 0x55b6f0d73300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f055197b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d63a20 0x55b6f0d73840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f055217c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d605d0 0x55b6f0d73300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36932/0 (socket says 172.21.15.33:36932) 2024-09-16T00:23:34.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f055197b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d63a20 0x55b6f0d73840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:58674/0 (socket says 172.21.15.33:58674) 2024-09-16T00:23:34.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f055217c640 1 -- 172.21.15.33:0/372456636 learned_addr learned my addr 172.21.15.33:0/372456636 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:34.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.253+0000 7f055297d640 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d64900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.254+0000 7f055217c640 1 -- 172.21.15.33:0/372456636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d63a20 msgr2=0x55b6f0d73840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:34.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.254+0000 7f055217c640 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d63a20 0x55b6f0d73840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.254+0000 7f055217c640 1 -- 172.21.15.33:0/372456636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 msgr2=0x55b6f0d64900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:34.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.254+0000 7f055217c640 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d64900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.254+0000 7f055217c640 1 -- 172.21.15.33:0/372456636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6f0d65040 con 0x55b6f0d605d0 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.254+0000 7f055197b640 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d63a20 0x55b6f0d73840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.254+0000 7f055217c640 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d605d0 0x55b6f0d73300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f05480b9030 tx=0x7f0548005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a58970c92e8c471 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.254+0000 7f0535ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.254+0000 7f0535ffb640 1 -- 172.21.15.33:0/372456636 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548049040 con 0x55b6f0d605d0 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.254+0000 7f05567b9740 1 -- 172.21.15.33:0/372456636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f053c027470 con 0x55b6f0d605d0 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.254+0000 7f0535ffb640 1 -- 172.21.15.33:0/372456636 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f05480ba960 con 0x55b6f0d605d0 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.255+0000 7f0535ffb640 1 -- 172.21.15.33:0/372456636 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f05480c5220 con 0x55b6f0d605d0 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.255+0000 7f05567b9740 10 client.26038 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.255+0000 7f05567b9740 1 -- 172.21.15.33:0/372456636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b6f0de54b0 con 0x55b6f0d605d0 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.255+0000 7f0535ffb640 1 -- 172.21.15.33:0/372456636 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f053c027470 con 0x55b6f0d605d0 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.255+0000 7f0535ffb640 1 -- 172.21.15.33:0/372456636 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f05480ab070 con 0x55b6f0d605d0 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.255+0000 7f05567b9740 10 client.26038 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.255+0000 7f05567b9740 10 client.26038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:23:34.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.255+0000 7f05567b9740 20 client.26038 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.255+0000 7f05567b9740 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b6f0de58b0 0x55b6f0de7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.255+0000 7f05567b9740 4 client.26038 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.255+0000 7f05567b9740 1 -- 172.21.15.33:0/372456636 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b6f0ccdd70 con 0x55b6f0de58b0 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.255+0000 7f055197b640 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b6f0de58b0 0x55b6f0de7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f0535ffb640 10 client.26038 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f055197b640 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b6f0de58b0 0x55b6f0de7ca0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b173abfe7c75b647 server_cookie=c66cb9cd7e8539ed in_seq=0 out_seq=0 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f0535ffb640 1 -- 172.21.15.33:0/372456636 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b6f0ccdd70 con 0x55b6f0de58b0 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f0535ffb640 10 client.26038 handle_command_reply: tid=0 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f05567b9740 1 client.26038 shutdown 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f05567b9740 1 -- 172.21.15.33:0/372456636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d605d0 msgr2=0x55b6f0d73300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f05567b9740 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d605d0 0x55b6f0d73300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f05480b9030 tx=0x7f0548005e50 comp rx=0 tx=0).stop 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f05567b9740 1 -- 172.21.15.33:0/372456636 shutdown_connections 2024-09-16T00:23:34.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f05567b9740 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b6f0de58b0 0x55b6f0de7ca0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f05567b9740 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6f0d6b370 0x55b6f0d64900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f05567b9740 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b6f0d63a20 0x55b6f0d73840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f0535ffb640 0 client.26038 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:34.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f05567b9740 1 --2- 172.21.15.33:0/372456636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6f0d605d0 0x55b6f0d73300 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f05567b9740 1 -- 172.21.15.33:0/372456636 >> 172.21.15.33:0/372456636 conn(0x55b6f0d29310 msgr2=0x55b6f0d668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:34.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f05567b9740 1 -- 172.21.15.33:0/372456636 shutdown_connections 2024-09-16T00:23:34.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f05567b9740 1 -- 172.21.15.33:0/372456636 wait complete. 2024-09-16T00:23:34.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.256+0000 7f05567b9740 20 client.26038 trim_cache size 0 max 16384 2024-09-16T00:23:34.273 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:23:34.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.257+0000 7f05543e0640 1 -- 172.21.15.33:0/2305565559 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f05380a3240 msgr2=0x7f05380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:34.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.257+0000 7f05543e0640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f05380a3240 0x7f05380a5700 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f054c11d8d0 tx=0x7f053c005f90 comp rx=0 tx=0).stop 2024-09-16T00:23:34.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.257+0000 7f05543e0640 1 -- 172.21.15.33:0/2305565559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f054c109450 msgr2=0x7f054c1a72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:34.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.257+0000 7f05543e0640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f054c109450 0x7f054c1a72b0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0548004770 tx=0x7f0548003c10 comp rx=0 tx=0).stop 2024-09-16T00:23:34.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.257+0000 7f05543e0640 1 -- 172.21.15.33:0/2305565559 shutdown_connections 2024-09-16T00:23:34.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.257+0000 7f05543e0640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f05380a3240 0x7f05380a5700 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.257+0000 7f05543e0640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f054c11b140 0x7f054c1abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.257+0000 7f05543e0640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f054c109830 0x7f054c1a77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.257+0000 7f05543e0640 1 --2- 172.21.15.33:0/2305565559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f054c109450 0x7f054c1a72b0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:34.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.257+0000 7f05543e0640 1 -- 172.21.15.33:0/2305565559 >> 172.21.15.33:0/2305565559 conn(0x7f054c06e140 msgr2=0x7f054c11bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:34.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.257+0000 7f05543e0640 1 -- 172.21.15.33:0/2305565559 shutdown_connections 2024-09-16T00:23:34.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:34.257+0000 7f05543e0640 1 -- 172.21.15.33:0/2305565559 wait complete. 2024-09-16T00:23:34.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 381988 8.75 MB/sec execute 162 sec latency 81.513 ms 2024-09-16T00:23:35.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:35 smithi060 ceph-mon[35185]: pgmap v543: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-09-16T00:23:35.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:35 smithi033 ceph-mon[31243]: pgmap v543: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-09-16T00:23:35.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 384268 8.78 MB/sec execute 163 sec latency 12.519 ms 2024-09-16T00:23:35.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:35 smithi083 ceph-mon[35242]: pgmap v543: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-09-16T00:23:36.235 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:23:36.236 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:23:36.497 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:23:36.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 387409 8.83 MB/sec execute 164 sec latency 13.730 ms 2024-09-16T00:23:37.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.979+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/1437843026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6b81082a0 msgr2=0x7fe6b810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:37.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.979+0000 7fe6bd2b5640 1 --2- 172.21.15.33:0/1437843026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6b81082a0 0x7fe6b810ec40 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009cd0 tx=0x7fe6b002a8e0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.979+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/1437843026 shutdown_connections 2024-09-16T00:23:37.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.979+0000 7fe6bd2b5640 1 --2- 172.21.15.33:0/1437843026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6b81082a0 0x7fe6b810ec40 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.979+0000 7fe6bd2b5640 1 --2- 172.21.15.33:0/1437843026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6b81036e0 0x7fe6b8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.979+0000 7fe6bd2b5640 1 --2- 172.21.15.33:0/1437843026 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b8102d30 0x7fe6b8103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.979+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/1437843026 >> 172.21.15.33:0/1437843026 conn(0x7fe6b80fc7d0 msgr2=0x7fe6b80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.979+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/1437843026 shutdown_connections 2024-09-16T00:23:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.979+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/1437843026 wait complete. 2024-09-16T00:23:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.980+0000 7fe6bd2b5640 1 Processor -- start 2024-09-16T00:23:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.980+0000 7fe6bd2b5640 1 -- start start 2024-09-16T00:23:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.980+0000 7fe6bd2b5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b8102d30 0x7fe6b8100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.980+0000 7fe6bd2b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6b81036e0 0x7fe6b8100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.980+0000 7fe6bd2b5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6b81082a0 0x7fe6b8101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.980+0000 7fe6bd2b5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b81120a0 con 0x7fe6b81082a0 2024-09-16T00:23:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.980+0000 7fe6bd2b5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b8111f20 con 0x7fe6b8102d30 2024-09-16T00:23:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.980+0000 7fe6bd2b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b8112220 con 0x7fe6b81036e0 2024-09-16T00:23:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6b81082a0 0x7fe6b8101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6b6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b8102d30 0x7fe6b8100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6b67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6b81036e0 0x7fe6b8100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6b6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b8102d30 0x7fe6b8100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:58706/0 (socket says 172.21.15.33:58706) 2024-09-16T00:23:37.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6b6ffd640 1 -- 172.21.15.33:0/892066626 learned_addr learned my addr 172.21.15.33:0/892066626 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:37.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6b67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6b81036e0 0x7fe6b8100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36944/0 (socket says 172.21.15.33:36944) 2024-09-16T00:23:37.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6b6ffd640 1 -- 172.21.15.33:0/892066626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6b81036e0 msgr2=0x7fe6b8100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:37.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6b6ffd640 1 --2- 172.21.15.33:0/892066626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6b81036e0 0x7fe6b8100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6b6ffd640 1 -- 172.21.15.33:0/892066626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6b81082a0 msgr2=0x7fe6b8101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:37.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6b6ffd640 1 --2- 172.21.15.33:0/892066626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6b81082a0 0x7fe6b8101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6b6ffd640 1 -- 172.21.15.33:0/892066626 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b81a69d0 con 0x7fe6b8102d30 2024-09-16T00:23:37.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6b6ffd640 1 --2- 172.21.15.33:0/892066626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b8102d30 0x7fe6b8100990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800ea60 tx=0x7fe6a800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:37.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe69ffff640 1 -- 172.21.15.33:0/892066626 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a800cde0 con 0x7fe6b8102d30 2024-09-16T00:23:37.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/892066626 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b81a6cc0 con 0x7fe6b8102d30 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe69ffff640 1 -- 172.21.15.33:0/892066626 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6a80035f0 con 0x7fe6b8102d30 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe69ffff640 1 -- 172.21.15.33:0/892066626 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8010670 con 0x7fe6b8102d30 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.981+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/892066626 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b81a71b0 con 0x7fe6b8102d30 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.984+0000 7fe69ffff640 1 -- 172.21.15.33:0/892066626 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe6a80037a0 con 0x7fe6b8102d30 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.984+0000 7fe6bf68e740 1 Processor -- start 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.984+0000 7fe6bf68e740 1 -- start start 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.984+0000 7fe6bf68e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f546f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.984+0000 7fe6bf68e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f551800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6bf68e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f554130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6bf68e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a88f4ac280 con 0x55a88f547480 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6bf68e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a88f453b10 con 0x55a88f551d40 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6bf68e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a88f4a8240 con 0x55a88f546b60 2024-09-16T00:23:37.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6b6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f551800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6b77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f546f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6b77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f546f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36960/0 (socket says 172.21.15.33:36960) 2024-09-16T00:23:37.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6b77fe640 1 -- 172.21.15.33:0/1272357830 learned_addr learned my addr 172.21.15.33:0/1272357830 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:37.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6b67fc640 1 --2- 172.21.15.33:0/1272357830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f554130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6b77fe640 1 -- 172.21.15.33:0/1272357830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 msgr2=0x55a88f554130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:37.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/1272357830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f554130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6b77fe640 1 -- 172.21.15.33:0/1272357830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 msgr2=0x55a88f551800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:37.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/1272357830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f551800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6b77fe640 1 -- 172.21.15.33:0/1272357830 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a88f4601f0 con 0x55a88f546b60 2024-09-16T00:23:37.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe69ffff640 1 --2- 172.21.15.33:0/892066626 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6a00a3240 0x7fe6a00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/1272357830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f546f40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009cd0 tx=0x7fe6b0009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db6e7a9b76b07505 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:37.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe6b6ffd640 1 --2- 172.21.15.33:0/892066626 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6a00a3240 0x7fe6a00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.985+0000 7fe69ffff640 1 -- 172.21.15.33:0/892066626 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe6a80d0040 con 0x7fe6b8102d30 2024-09-16T00:23:37.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.986+0000 7fe69d7fa640 1 -- 172.21.15.33:0/1272357830 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0048070 con 0x55a88f546b60 2024-09-16T00:23:37.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.986+0000 7fe69d7fa640 1 -- 172.21.15.33:0/1272357830 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b0036070 con 0x55a88f546b60 2024-09-16T00:23:37.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.986+0000 7fe69d7fa640 1 -- 172.21.15.33:0/1272357830 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b00043e0 con 0x55a88f546b60 2024-09-16T00:23:37.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.986+0000 7fe6bf68e740 1 -- 172.21.15.33:0/1272357830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 msgr2=0x55a88f546f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:37.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.986+0000 7fe6b6ffd640 1 --2- 172.21.15.33:0/892066626 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6a00a3240 0x7fe6a00a5700 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8098bb0 tx=0x7fe6a80d3b10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:37.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.986+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/1272357830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f546f40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009cd0 tx=0x7fe6b0009a20 comp rx=0 tx=0).stop 2024-09-16T00:23:37.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 -- 172.21.15.33:0/1272357830 shutdown_connections 2024-09-16T00:23:37.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/1272357830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f554130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/1272357830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f551800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/1272357830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f546f40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 -- 172.21.15.33:0/1272357830 >> 172.21.15.33:0/1272357830 conn(0x55a88f5430f0 msgr2=0x55a88f5434d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 -- 172.21.15.33:0/1272357830 shutdown_connections 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 -- 172.21.15.33:0/1272357830 wait complete. 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 -- start start 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 Processor -- start 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f546b60 0x55a88f54ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f547480 0x55a88f54f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f54f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a88f554910 con 0x55a88f546b60 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a88f466f10 con 0x55a88f551d40 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a88f554a90 con 0x55a88f547480 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.987+0000 7fe6bf68e740 10 client.0 _ll_register_callbacks cb 0x55a88f42fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f546b60 0x55a88f54ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f546b60 0x55a88f54ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59716/0 (socket says 172.21.15.33:59716) 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6b77fe640 1 -- 172.21.15.33:0/3234215190 learned_addr learned my addr 172.21.15.33:0/3234215190 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6b67fc640 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f547480 0x55a88f54f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f54f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6b77fe640 1 -- 172.21.15.33:0/3234215190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f547480 msgr2=0x55a88f54f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f547480 0x55a88f54f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6b77fe640 1 -- 172.21.15.33:0/3234215190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 msgr2=0x55a88f54f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f54f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6b77fe640 1 -- 172.21.15.33:0/3234215190 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a88f21d6a0 con 0x55a88f546b60 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f546b60 0x55a88f54ebf0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009620 tx=0x7fe6b0003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b76aee80bd1a1a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe69aff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe69aff5640 1 -- 172.21.15.33:0/3234215190 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0054020 con 0x55a88f546b60 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe69aff5640 1 -- 172.21.15.33:0/3234215190 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe6b0004140 con 0x55a88f546b60 2024-09-16T00:23:37.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6bf68e740 1 -- 172.21.15.33:0/3234215190 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6b0036070 con 0x55a88f546b60 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe69aff5640 1 -- 172.21.15.33:0/3234215190 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe6b0048040 con 0x55a88f546b60 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6bf68e740 10 client.16611 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.988+0000 7fe6bf68e740 1 -- 172.21.15.33:0/3234215190 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a88f554910 con 0x55a88f546b60 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.989+0000 7fe69aff5640 1 -- 172.21.15.33:0/3234215190 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b0036070 con 0x55a88f546b60 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.989+0000 7fe69aff5640 1 -- 172.21.15.33:0/3234215190 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fe6b0042030 con 0x55a88f546b60 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.989+0000 7fe6bf68e740 10 client.16611 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.989+0000 7fe6bf68e740 10 client.16611 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.989+0000 7fe6bf68e740 20 client.16611 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.989+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a88f5616c0 0x55a88f563ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.989+0000 7fe6bf68e740 4 client.16611 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.989+0000 7fe6bf68e740 1 -- 172.21.15.33:0/3234215190 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a88f4b97f0 con 0x55a88f5616c0 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.989+0000 7fe6b67fc640 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a88f5616c0 0x55a88f563ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.989+0000 7fe6b67fc640 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a88f5616c0 0x55a88f563ab0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fd742309b1a008b server_cookie=1a6ab830bf1aadea in_seq=0 out_seq=0 2024-09-16T00:23:37.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.989+0000 7fe69aff5640 10 client.16611 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:37.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe69aff5640 1 -- 172.21.15.33:0/3234215190 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a88f4b97f0 con 0x55a88f5616c0 2024-09-16T00:23:37.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe69aff5640 10 client.16611 handle_command_reply: tid=0 2024-09-16T00:23:37.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe6bf68e740 1 client.16611 shutdown 2024-09-16T00:23:37.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe6bf68e740 1 -- 172.21.15.33:0/3234215190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f546b60 msgr2=0x55a88f54ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:37.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f546b60 0x55a88f54ebf0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009620 tx=0x7fe6b0003f60 comp rx=0 tx=0).stop 2024-09-16T00:23:37.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe6bf68e740 1 -- 172.21.15.33:0/3234215190 shutdown_connections 2024-09-16T00:23:37.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a88f5616c0 0x55a88f563ab0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f54f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f547480 0x55a88f54f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/3234215190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f546b60 0x55a88f54ebf0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe69aff5640 0 client.16611 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:37.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe6bf68e740 1 -- 172.21.15.33:0/3234215190 >> 172.21.15.33:0/3234215190 conn(0x55a88f5430f0 msgr2=0x55a88f548e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:37.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe6bf68e740 1 -- 172.21.15.33:0/3234215190 shutdown_connections 2024-09-16T00:23:37.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.990+0000 7fe6bf68e740 1 -- 172.21.15.33:0/3234215190 wait complete. 2024-09-16T00:23:37.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:36.991+0000 7fe6bf68e740 20 client.16611 trim_cache size 0 max 16384 2024-09-16T00:23:37.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.009+0000 7fe6bf68e740 1 Processor -- start 2024-09-16T00:23:37.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.009+0000 7fe6bf68e740 1 -- start start 2024-09-16T00:23:37.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6bf68e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f550a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6bf68e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f569100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6bf68e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f569640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6bf68e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b81b3b60 con 0x55a88f547480 2024-09-16T00:23:37.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6bf68e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b81b1430 con 0x55a88f551d40 2024-09-16T00:23:37.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6bf68e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b81af280 con 0x55a88f546b60 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f550a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f569100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f550a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36982/0 (socket says 172.21.15.33:36982) 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f569100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59728/0 (socket says 172.21.15.33:59728) 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b77fe640 1 -- 172.21.15.33:0/2382732109 learned_addr learned my addr 172.21.15.33:0/2382732109 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/2382732109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f569640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b77fe640 1 -- 172.21.15.33:0/2382732109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 msgr2=0x55a88f569640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/2382732109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f569640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b77fe640 1 -- 172.21.15.33:0/2382732109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 msgr2=0x55a88f569100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/2382732109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f569100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b77fe640 1 -- 172.21.15.33:0/2382732109 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a88f544280 con 0x55a88f546b60 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b67fc640 1 --2- 172.21.15.33:0/2382732109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f569100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:37.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.010+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/2382732109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f550a50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0003c50 tx=0x7fe6b002a8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fa569134a96c2d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:37.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.011+0000 7fe69cff9640 1 -- 172.21.15.33:0/2382732109 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0054040 con 0x55a88f546b60 2024-09-16T00:23:37.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.011+0000 7fe69cff9640 1 -- 172.21.15.33:0/2382732109 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b0036030 con 0x55a88f546b60 2024-09-16T00:23:37.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.011+0000 7fe69cff9640 1 -- 172.21.15.33:0/2382732109 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0030800 con 0x55a88f546b60 2024-09-16T00:23:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.011+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2382732109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 msgr2=0x55a88f550a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.011+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/2382732109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f550a50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0003c50 tx=0x7fe6b002a8e0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.011+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2382732109 shutdown_connections 2024-09-16T00:23:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.011+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/2382732109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f569640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.011+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/2382732109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f569100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.011+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/2382732109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f550a50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.011+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2382732109 >> 172.21.15.33:0/2382732109 conn(0x55a88f5430f0 msgr2=0x55a88f550340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.011+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2382732109 shutdown_connections 2024-09-16T00:23:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.011+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2382732109 wait complete. 2024-09-16T00:23:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6bf68e740 1 -- start start 2024-09-16T00:23:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6bf68e740 1 Processor -- start 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6bf68e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f555a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6bf68e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f555fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6bf68e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f5564e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6bf68e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b81af8f0 con 0x55a88f547480 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6bf68e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b81b0d60 con 0x55a88f551d40 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6bf68e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b81affc0 con 0x55a88f546b60 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6bf68e740 10 client.0 _ll_register_callbacks cb 0x7fe6b0046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6b77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f555a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6b67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f555fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6b77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f555a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36994/0 (socket says 172.21.15.33:36994) 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6b77fe640 1 -- 172.21.15.33:0/2892491952 learned_addr learned my addr 172.21.15.33:0/2892491952 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.012+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f5564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe6b67fc640 1 -- 172.21.15.33:0/2892491952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 msgr2=0x55a88f555a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe6b67fc640 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f555a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe6b67fc640 1 -- 172.21.15.33:0/2892491952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 msgr2=0x55a88f5564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe6b67fc640 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f5564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe6b67fc640 1 -- 172.21.15.33:0/2892491952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a88f5cb0d0 con 0x55a88f547480 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f5564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe6b67fc640 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f555fa0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x55a88f544470 tx=0x7fe6ac01d6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b411db4d3064d2cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe69aff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe69aff5640 1 -- 172.21.15.33:0/2892491952 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac004230 con 0x55a88f547480 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe69aff5640 1 -- 172.21.15.33:0/2892491952 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe6ac0043d0 con 0x55a88f547480 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2892491952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6b0036030 con 0x55a88f547480 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe69aff5640 1 -- 172.21.15.33:0/2892491952 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe6ac02c050 con 0x55a88f547480 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe6bf68e740 10 client.16617 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2892491952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a88f5cc420 con 0x55a88f547480 2024-09-16T00:23:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.013+0000 7fe69aff5640 1 -- 172.21.15.33:0/2892491952 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b0036030 con 0x55a88f547480 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.014+0000 7fe69aff5640 1 -- 172.21.15.33:0/2892491952 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fe6ac003a70 con 0x55a88f547480 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.014+0000 7fe6bf68e740 10 client.16617 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.014+0000 7fe6bf68e740 10 client.16617 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.014+0000 7fe6bf68e740 20 client.16617 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.014+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a88f5ccba0 0x55a88f5cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.014+0000 7fe6bf68e740 4 client.16617 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.014+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2892491952 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a88f4b3d00 con 0x55a88f5ccba0 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.014+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a88f5ccba0 0x55a88f5cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.014+0000 7fe6b77fe640 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a88f5ccba0 0x55a88f5cefc0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a28b60124e9267c server_cookie=a78fd6550a442ca3 in_seq=0 out_seq=0 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.014+0000 7fe69aff5640 10 client.16617 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe69aff5640 1 -- 172.21.15.33:0/2892491952 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a88f4b3d00 con 0x55a88f5ccba0 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe69aff5640 10 client.16617 handle_command_reply: tid=0 2024-09-16T00:23:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe6bf68e740 1 client.16617 shutdown 2024-09-16T00:23:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2892491952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 msgr2=0x55a88f555fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f555fa0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x55a88f544470 tx=0x7fe6ac01d6e0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2892491952 shutdown_connections 2024-09-16T00:23:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a88f5ccba0 0x55a88f5cefc0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a88f551d40 0x55a88f5564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe69aff5640 0 client.16617 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a88f547480 0x55a88f555fa0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe6bf68e740 1 --2- 172.21.15.33:0/2892491952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a88f546b60 0x55a88f555a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2892491952 >> 172.21.15.33:0/2892491952 conn(0x55a88f5430f0 msgr2=0x55a88f548dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2892491952 shutdown_connections 2024-09-16T00:23:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.015+0000 7fe6bf68e740 1 -- 172.21.15.33:0/2892491952 wait complete. 2024-09-16T00:23:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.016+0000 7fe6bf68e740 20 client.16617 trim_cache size 0 max 16384 2024-09-16T00:23:37.032 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:23:37.032 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:23:37.032 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "f7028255-7cc1-4fc7-9308-1ae759a9f6cf", 2024-09-16T00:23:37.032 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:23:37.032 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:23:37.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.016+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/892066626 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6a00a3240 msgr2=0x7fe6a00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:37.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.016+0000 7fe6bd2b5640 1 --2- 172.21.15.33:0/892066626 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6a00a3240 0x7fe6a00a5700 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8098bb0 tx=0x7fe6a80d3b10 comp rx=0 tx=0).stop 2024-09-16T00:23:37.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.016+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/892066626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b8102d30 msgr2=0x7fe6b8100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:37.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.016+0000 7fe6bd2b5640 1 --2- 172.21.15.33:0/892066626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b8102d30 0x7fe6b8100990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800ea60 tx=0x7fe6a800ef20 comp rx=0 tx=0).stop 2024-09-16T00:23:37.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.016+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/892066626 shutdown_connections 2024-09-16T00:23:37.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.016+0000 7fe6bd2b5640 1 --2- 172.21.15.33:0/892066626 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6a00a3240 0x7fe6a00a5700 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.016+0000 7fe6bd2b5640 1 --2- 172.21.15.33:0/892066626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6b81082a0 0x7fe6b8101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.016+0000 7fe6bd2b5640 1 --2- 172.21.15.33:0/892066626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6b81036e0 0x7fe6b8100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.016+0000 7fe6bd2b5640 1 --2- 172.21.15.33:0/892066626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b8102d30 0x7fe6b8100990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:37.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.016+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/892066626 >> 172.21.15.33:0/892066626 conn(0x7fe6b80fc7d0 msgr2=0x7fe6b8105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:37.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.017+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/892066626 shutdown_connections 2024-09-16T00:23:37.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:37.017+0000 7fe6bd2b5640 1 -- 172.21.15.33:0/892066626 wait complete. 2024-09-16T00:23:37.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:37 smithi060 ceph-mon[35185]: pgmap v544: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.3 MiB/s wr, 220 op/s 2024-09-16T00:23:37.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:37 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:23:37.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:37 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:23:37.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:37 smithi033 ceph-mon[31243]: pgmap v544: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.3 MiB/s wr, 220 op/s 2024-09-16T00:23:37.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:37 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:23:37.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:37 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:23:37.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 390610 8.89 MB/sec execute 165 sec latency 14.611 ms 2024-09-16T00:23:37.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:37 smithi083 ceph-mon[35242]: pgmap v544: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.3 MiB/s wr, 220 op/s 2024-09-16T00:23:37.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:37 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:23:37.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:37 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:23:37.786 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:23:38.070 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:23:38.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:38 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:23:38.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:38 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:23:38.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:38 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:23:38.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:38 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:23:38.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:38 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:23:38.495 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:38 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:23:38.495 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:38 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:23:38.495 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:38 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:23:38.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.581+0000 7ff0bb63b640 1 -- 172.21.15.33:0/3619046577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0b4101430 msgr2=0x7ff0b410d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:38.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.581+0000 7ff0bb63b640 1 --2- 172.21.15.33:0/3619046577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0b4101430 0x7ff0b410d960 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac009cd0 tx=0x7ff0ac02a8e0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.581+0000 7ff0bb63b640 1 -- 172.21.15.33:0/3619046577 shutdown_connections 2024-09-16T00:23:38.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.581+0000 7ff0bb63b640 1 --2- 172.21.15.33:0/3619046577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0b4101430 0x7ff0b410d960 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.581+0000 7ff0bb63b640 1 --2- 172.21.15.33:0/3619046577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0b4100a90 0x7ff0b4100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.581+0000 7ff0bb63b640 1 --2- 172.21.15.33:0/3619046577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b4106a90 0x7ff0b4106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.581+0000 7ff0bb63b640 1 -- 172.21.15.33:0/3619046577 >> 172.21.15.33:0/3619046577 conn(0x7ff0b40fc7d0 msgr2=0x7ff0b40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:38.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.582+0000 7ff0bb63b640 1 -- 172.21.15.33:0/3619046577 shutdown_connections 2024-09-16T00:23:38.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.582+0000 7ff0bb63b640 1 -- 172.21.15.33:0/3619046577 wait complete. 2024-09-16T00:23:38.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.582+0000 7ff0bb63b640 1 Processor -- start 2024-09-16T00:23:38.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.582+0000 7ff0bb63b640 1 -- start start 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0bb63b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b4100a90 0x7ff0b4199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0bb63b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0b4101430 0x7ff0b41997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0bb63b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0b4106a90 0x7ff0b41a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0bb63b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b4110e90 con 0x7ff0b4106a90 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0bb63b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b4110d10 con 0x7ff0b4100a90 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0bb63b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b4111010 con 0x7ff0b4101430 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0b9bd8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0b4106a90 0x7ff0b41a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0b9bd8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0b4106a90 0x7ff0b41a4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59746/0 (socket says 172.21.15.33:59746) 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0b9bd8640 1 -- 172.21.15.33:0/637510395 learned_addr learned my addr 172.21.15.33:0/637510395 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0b93d7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b4100a90 0x7ff0b4199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0b4101430 0x7ff0b41997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0b8bd6640 1 -- 172.21.15.33:0/637510395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b4100a90 msgr2=0x7ff0b4199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:38.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b4100a90 0x7ff0b4199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0b8bd6640 1 -- 172.21.15.33:0/637510395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0b4106a90 msgr2=0x7ff0b41a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0b4106a90 0x7ff0b41a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0b8bd6640 1 -- 172.21.15.33:0/637510395 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b41a4840 con 0x7ff0b4101430 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.583+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0b4101430 0x7ff0b41997c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff0a400e9f0 tx=0x7ff0a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.584+0000 7ff0aa7fc640 1 -- 172.21.15.33:0/637510395 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a400ccc0 con 0x7ff0b4101430 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.584+0000 7ff0aa7fc640 1 -- 172.21.15.33:0/637510395 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0a4005020 con 0x7ff0b4101430 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.584+0000 7ff0aa7fc640 1 -- 172.21.15.33:0/637510395 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a40049e0 con 0x7ff0b4101430 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.584+0000 7ff0bb63b640 1 -- 172.21.15.33:0/637510395 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b41a4a10 con 0x7ff0b4101430 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.584+0000 7ff0bb63b640 1 -- 172.21.15.33:0/637510395 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b41a4ed0 con 0x7ff0b4101430 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.586+0000 7ff0aa7fc640 1 -- 172.21.15.33:0/637510395 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff0a4005560 con 0x7ff0b4101430 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0bda14740 1 Processor -- start 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0bda14740 1 -- start start 2024-09-16T00:23:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0bda14740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45ef4a0 0x55ebd45ef880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0bda14740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd45fa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0bda14740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45fa680 0x55ebd45fca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0bda14740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ebd45552e0 con 0x55ebd45efdc0 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0bda14740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ebd44fcce0 con 0x55ebd45fa680 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0bda14740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ebd45512a0 con 0x55ebd45ef4a0 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0b93d7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd45fa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0b93d7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd45fa140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59758/0 (socket says 172.21.15.33:59758) 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0b93d7640 1 -- 172.21.15.33:0/1311300599 learned_addr learned my addr 172.21.15.33:0/1311300599 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0b9bd8640 1 --2- 172.21.15.33:0/1311300599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45ef4a0 0x55ebd45ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0aa7fc640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0a00a3240 0x7ff0a00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.587+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/1311300599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45fa680 0x55ebd45fca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.588+0000 7ff0b9bd8640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0a00a3240 0x7ff0a00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.588+0000 7ff0b93d7640 1 -- 172.21.15.33:0/1311300599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45ef4a0 msgr2=0x55ebd45ef880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.588+0000 7ff0b93d7640 1 --2- 172.21.15.33:0/1311300599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45ef4a0 0x55ebd45ef880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.588+0000 7ff0b93d7640 1 -- 172.21.15.33:0/1311300599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45fa680 msgr2=0x55ebd45fca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.588+0000 7ff0b93d7640 1 --2- 172.21.15.33:0/1311300599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45fa680 0x55ebd45fca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.588+0000 7ff0b93d7640 1 -- 172.21.15.33:0/1311300599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebd44e1440 con 0x55ebd45efdc0 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.588+0000 7ff0aa7fc640 1 -- 172.21.15.33:0/637510395 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff0a40cfce0 con 0x7ff0b4101430 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.588+0000 7ff0b93d7640 1 --2- 172.21.15.33:0/1311300599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd45fa140 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009c70 tx=0x7ff0b0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f296c23344fce61 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.588+0000 7ff0b9bd8640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0a00a3240 0x7ff0a00a5700 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac004ab0 tx=0x7ff0ac02a8e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.588+0000 7ff09f7fe640 1 -- 172.21.15.33:0/1311300599 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0004230 con 0x55ebd45efdc0 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.588+0000 7ff09f7fe640 1 -- 172.21.15.33:0/1311300599 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0b00043d0 con 0x55ebd45efdc0 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.588+0000 7ff09f7fe640 1 -- 172.21.15.33:0/1311300599 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00197a0 con 0x55ebd45efdc0 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.589+0000 7ff0bda14740 1 -- 172.21.15.33:0/1311300599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 msgr2=0x55ebd45fa140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.589+0000 7ff0bda14740 1 --2- 172.21.15.33:0/1311300599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd45fa140 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009c70 tx=0x7ff0b0011810 comp rx=0 tx=0).stop 2024-09-16T00:23:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.589+0000 7ff0bda14740 1 -- 172.21.15.33:0/1311300599 shutdown_connections 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.589+0000 7ff0bda14740 1 --2- 172.21.15.33:0/1311300599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45fa680 0x55ebd45fca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.589+0000 7ff0bda14740 1 --2- 172.21.15.33:0/1311300599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd45fa140 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.589+0000 7ff0bda14740 1 --2- 172.21.15.33:0/1311300599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45ef4a0 0x55ebd45ef880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.589+0000 7ff0bda14740 1 -- 172.21.15.33:0/1311300599 >> 172.21.15.33:0/1311300599 conn(0x55ebd45eba30 msgr2=0x55ebd45ebe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.589+0000 7ff0bda14740 1 -- 172.21.15.33:0/1311300599 shutdown_connections 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.589+0000 7ff0bda14740 1 -- 172.21.15.33:0/1311300599 wait complete. 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.589+0000 7ff0bda14740 1 -- start start 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.589+0000 7ff0bda14740 1 Processor -- start 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0bda14740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd45f9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0bda14740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd45f7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0bda14740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45fa680 0x55ebd45f7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0bda14740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ebd45fd250 con 0x55ebd45efdc0 2024-09-16T00:23:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0bda14740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ebd4510050 con 0x55ebd45ef4a0 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0bda14740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ebd45fd3d0 con 0x55ebd45fa680 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0bda14740 10 client.0 _ll_register_callbacks cb 0x55ebd4550160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0b8bd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd45f7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0b8bd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd45f7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59762/0 (socket says 172.21.15.33:59762) 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0b93d7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd45f9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0b8bd6640 1 -- 172.21.15.33:0/3901509752 learned_addr learned my addr 172.21.15.33:0/3901509752 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0b9bd8640 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45fa680 0x55ebd45f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0b93d7640 1 -- 172.21.15.33:0/3901509752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45fa680 msgr2=0x55ebd45f7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0b93d7640 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45fa680 0x55ebd45f7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.590+0000 7ff0b93d7640 1 -- 172.21.15.33:0/3901509752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 msgr2=0x55ebd45f7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.591+0000 7ff0b93d7640 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd45f7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.591+0000 7ff0b93d7640 1 -- 172.21.15.33:0/3901509752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebd42c63c0 con 0x55ebd45ef4a0 2024-09-16T00:23:38.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.591+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd45f7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.591+0000 7ff0b93d7640 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd45f9f70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0003ef0 tx=0x7ff0b0003db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e1ddefb7dbd87be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.591+0000 7ff09cff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.591+0000 7ff09cff9640 1 -- 172.21.15.33:0/3901509752 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0027020 con 0x55ebd45ef4a0 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.591+0000 7ff09cff9640 1 -- 172.21.15.33:0/3901509752 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff0b0003800 con 0x55ebd45ef4a0 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.591+0000 7ff0bda14740 1 -- 172.21.15.33:0/3901509752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0b00043d0 con 0x55ebd45ef4a0 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.591+0000 7ff09cff9640 1 -- 172.21.15.33:0/3901509752 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff0b0032050 con 0x55ebd45ef4a0 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.591+0000 7ff0bda14740 10 client.26153 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.591+0000 7ff0bda14740 1 -- 172.21.15.33:0/3901509752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ebd4608bc0 con 0x55ebd45ef4a0 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.591+0000 7ff09cff9640 1 -- 172.21.15.33:0/3901509752 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0b00043d0 con 0x55ebd45ef4a0 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.592+0000 7ff09cff9640 1 -- 172.21.15.33:0/3901509752 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff0b0018070 con 0x55ebd45ef4a0 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.592+0000 7ff0bda14740 10 client.26153 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.592+0000 7ff0bda14740 10 client.26153 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.592+0000 7ff0bda14740 20 client.26153 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.592+0000 7ff0bda14740 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ebd460a080 0x55ebd460c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.592+0000 7ff0bda14740 4 client.26153 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.592+0000 7ff0bda14740 1 -- 172.21.15.33:0/3901509752 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebd4562850 con 0x55ebd460a080 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.592+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ebd460a080 0x55ebd460c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.592+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ebd460a080 0x55ebd460c470 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=359c3be9f1d2c9ee server_cookie=c52bd0186a29c9ac in_seq=0 out_seq=0 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.592+0000 7ff09cff9640 10 client.26153 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff09cff9640 1 -- 172.21.15.33:0/3901509752 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ebd4562850 con 0x55ebd460a080 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff09cff9640 10 client.26153 handle_command_reply: tid=0 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff0bda14740 1 client.26153 shutdown 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff0bda14740 1 -- 172.21.15.33:0/3901509752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 msgr2=0x55ebd45f9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff0bda14740 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd45f9f70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0003ef0 tx=0x7ff0b0003db0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff0bda14740 1 -- 172.21.15.33:0/3901509752 shutdown_connections 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff0bda14740 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ebd460a080 0x55ebd460c470 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff0bda14740 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45fa680 0x55ebd45f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff0bda14740 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd45f7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff0bda14740 1 --2- 172.21.15.33:0/3901509752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd45f9f70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff09cff9640 0 client.26153 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:38.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff0bda14740 1 -- 172.21.15.33:0/3901509752 >> 172.21.15.33:0/3901509752 conn(0x55ebd45eba30 msgr2=0x55ebd45f1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:38.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff0bda14740 1 -- 172.21.15.33:0/3901509752 shutdown_connections 2024-09-16T00:23:38.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff0bda14740 1 -- 172.21.15.33:0/3901509752 wait complete. 2024-09-16T00:23:38.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.593+0000 7ff0bda14740 20 client.26153 trim_cache size 0 max 16384 2024-09-16T00:23:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0bda14740 1 Processor -- start 2024-09-16T00:23:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0bda14740 1 -- start start 2024-09-16T00:23:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0bda14740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd45f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0bda14740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd4611c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0bda14740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45fa680 0x55ebd4612160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0bda14740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b41b1930 con 0x55ebd45efdc0 2024-09-16T00:23:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0bda14740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b41af200 con 0x55ebd45ef4a0 2024-09-16T00:23:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0bda14740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b41ad050 con 0x55ebd45fa680 2024-09-16T00:23:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0b8bd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd4611c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0b8bd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd4611c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59772/0 (socket says 172.21.15.33:59772) 2024-09-16T00:23:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0b8bd6640 1 -- 172.21.15.33:0/402011282 learned_addr learned my addr 172.21.15.33:0/402011282 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0b93d7640 1 --2- 172.21.15.33:0/402011282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd45f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.612+0000 7ff0b9bd8640 1 --2- 172.21.15.33:0/402011282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45fa680 0x55ebd4612160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.613+0000 7ff0b8bd6640 1 -- 172.21.15.33:0/402011282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45fa680 msgr2=0x55ebd4612160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.613+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/402011282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45fa680 0x55ebd4612160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.613+0000 7ff0b8bd6640 1 -- 172.21.15.33:0/402011282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 msgr2=0x55ebd45f5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.613+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/402011282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd45f5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.613+0000 7ff0b8bd6640 1 -- 172.21.15.33:0/402011282 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebd45fa2e0 con 0x55ebd45efdc0 2024-09-16T00:23:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.613+0000 7ff0b93d7640 1 --2- 172.21.15.33:0/402011282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd45f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:23:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.613+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/402011282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd4611c20 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff0a40982e0 tx=0x7ff0a4097a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d453f9a159073778 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.613+0000 7ff09effd640 1 -- 172.21.15.33:0/402011282 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a401c040 con 0x55ebd45efdc0 2024-09-16T00:23:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.613+0000 7ff09effd640 1 -- 172.21.15.33:0/402011282 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0a40051d0 con 0x55ebd45efdc0 2024-09-16T00:23:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.613+0000 7ff09effd640 1 -- 172.21.15.33:0/402011282 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a409a210 con 0x55ebd45efdc0 2024-09-16T00:23:38.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.614+0000 7ff0bda14740 1 -- 172.21.15.33:0/402011282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 msgr2=0x55ebd4611c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:38.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.614+0000 7ff0bda14740 1 --2- 172.21.15.33:0/402011282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd4611c20 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff0a40982e0 tx=0x7ff0a4097a00 comp rx=0 tx=0).stop 2024-09-16T00:23:38.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.614+0000 7ff0bda14740 1 -- 172.21.15.33:0/402011282 shutdown_connections 2024-09-16T00:23:38.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.614+0000 7ff0bda14740 1 --2- 172.21.15.33:0/402011282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45fa680 0x55ebd4612160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.614+0000 7ff0bda14740 1 --2- 172.21.15.33:0/402011282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45efdc0 0x55ebd4611c20 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.614+0000 7ff0bda14740 1 --2- 172.21.15.33:0/402011282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd45f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.614+0000 7ff0bda14740 1 -- 172.21.15.33:0/402011282 >> 172.21.15.33:0/402011282 conn(0x55ebd45eba30 msgr2=0x55ebd45eab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:38.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.614+0000 7ff0bda14740 1 -- 172.21.15.33:0/402011282 shutdown_connections 2024-09-16T00:23:38.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.614+0000 7ff0bda14740 1 -- 172.21.15.33:0/402011282 wait complete. 2024-09-16T00:23:38.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.614+0000 7ff0bda14740 1 -- start start 2024-09-16T00:23:38.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0bda14740 1 Processor -- start 2024-09-16T00:23:38.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0bda14740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd46029b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0bda14740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45efdc0 0x55ebd4602ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0bda14740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45fa680 0x55ebd45fff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0bda14740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b41ad6c0 con 0x55ebd45fa680 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0bda14740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b41aeb30 con 0x55ebd45ef4a0 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0bda14740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b41add90 con 0x55ebd45efdc0 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0bda14740 10 client.0 _ll_register_callbacks cb 0x7ff0a40967c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0b93d7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd46029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0b9bd8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45fa680 0x55ebd45fff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0b93d7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd46029b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:58792/0 (socket says 172.21.15.33:58792) 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0b93d7640 1 -- 172.21.15.33:0/3614588330 learned_addr learned my addr 172.21.15.33:0/3614588330 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.615+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45efdc0 0x55ebd4602ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff0b93d7640 1 -- 172.21.15.33:0/3614588330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45efdc0 msgr2=0x55ebd4602ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:38.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff0b93d7640 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45efdc0 0x55ebd4602ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff0b93d7640 1 -- 172.21.15.33:0/3614588330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45fa680 msgr2=0x55ebd45fff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff0b93d7640 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45fa680 0x55ebd45fff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff0b93d7640 1 -- 172.21.15.33:0/3614588330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebd4600780 con 0x55ebd45ef4a0 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45efdc0 0x55ebd4602ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff0b93d7640 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd46029b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009c70 tx=0x7ff0b0003a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=836ae9953271166a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff09cff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff09cff9640 1 -- 172.21.15.33:0/3614588330 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0027070 con 0x55ebd45ef4a0 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff09cff9640 1 -- 172.21.15.33:0/3614588330 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff0b0004720 con 0x55ebd45ef4a0 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff0bda14740 1 -- 172.21.15.33:0/3614588330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0a40051d0 con 0x55ebd45ef4a0 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff09cff9640 1 -- 172.21.15.33:0/3614588330 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff0b001f040 con 0x55ebd45ef4a0 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff0bda14740 10 client.26159 fetch_fsmap learned FSMap version 30 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff0bda14740 1 -- 172.21.15.33:0/3614588330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ebd467d080 con 0x55ebd45ef4a0 2024-09-16T00:23:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.616+0000 7ff09cff9640 1 -- 172.21.15.33:0/3614588330 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0a40051d0 con 0x55ebd45ef4a0 2024-09-16T00:23:38.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.617+0000 7ff09cff9640 1 -- 172.21.15.33:0/3614588330 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff0b0018040 con 0x55ebd45ef4a0 2024-09-16T00:23:38.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.617+0000 7ff0bda14740 10 client.26159 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:23:38.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.617+0000 7ff0bda14740 10 client.26159 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:23:38.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.617+0000 7ff0bda14740 20 client.26159 populate_metadata read hostname 'smithi033' 2024-09-16T00:23:38.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.617+0000 7ff0bda14740 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ebd467d790 0x55ebd467fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:23:38.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.617+0000 7ff0bda14740 4 client.26159 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:23:38.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.617+0000 7ff0bda14740 1 -- 172.21.15.33:0/3614588330 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ebd455cd60 con 0x55ebd467d790 2024-09-16T00:23:38.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.617+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ebd467d790 0x55ebd467fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:23:38.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.617+0000 7ff09cff9640 10 client.26159 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:38.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.617+0000 7ff0b8bd6640 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ebd467d790 0x55ebd467fb80 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad5fa6230facae25 server_cookie=cbadaf8a410bbd4d in_seq=0 out_seq=0 2024-09-16T00:23:38.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff09cff9640 1 -- 172.21.15.33:0/3614588330 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ebd455cd60 con 0x55ebd467d790 2024-09-16T00:23:38.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff09cff9640 10 client.26159 handle_command_reply: tid=0 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff0bda14740 1 client.26159 shutdown 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff0bda14740 1 -- 172.21.15.33:0/3614588330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 msgr2=0x55ebd46029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff0bda14740 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd46029b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009c70 tx=0x7ff0b0003a30 comp rx=0 tx=0).stop 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff0bda14740 1 -- 172.21.15.33:0/3614588330 shutdown_connections 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff0bda14740 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ebd467d790 0x55ebd467fb80 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff0bda14740 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebd45fa680 0x55ebd45fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff0bda14740 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebd45efdc0 0x55ebd4602ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff09cff9640 0 client.26159 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff0bda14740 1 --2- 172.21.15.33:0/3614588330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ebd45ef4a0 0x55ebd46029b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff0bda14740 1 -- 172.21.15.33:0/3614588330 >> 172.21.15.33:0/3614588330 conn(0x55ebd45eba30 msgr2=0x55ebd45f8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff0bda14740 1 -- 172.21.15.33:0/3614588330 shutdown_connections 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff0bda14740 1 -- 172.21.15.33:0/3614588330 wait complete. 2024-09-16T00:23:38.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.618+0000 7ff0bda14740 20 client.26159 trim_cache size 0 max 16384 2024-09-16T00:23:38.624 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:23:38.624 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:23:38.624 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:23:38.624 INFO:teuthology.orchestra.run.smithi033.stdout: "f7028255-7cc1-4fc7-9308-1ae759a9f6cf": { 2024-09-16T00:23:38.624 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:23:38.624 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "f7028255-7cc1-4fc7-9308-1ae759a9f6cf", 2024-09-16T00:23:38.624 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:23:38.624 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:23:38.624 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:23:38.624 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:23:38.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.619+0000 7ff0bb63b640 1 -- 172.21.15.33:0/637510395 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0a00a3240 msgr2=0x7ff0a00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:38.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.619+0000 7ff0bb63b640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0a00a3240 0x7ff0a00a5700 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac004ab0 tx=0x7ff0ac02a8e0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.619+0000 7ff0bb63b640 1 -- 172.21.15.33:0/637510395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0b4101430 msgr2=0x7ff0b41997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:23:38.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.619+0000 7ff0bb63b640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0b4101430 0x7ff0b41997c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff0a400e9f0 tx=0x7ff0a400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.619+0000 7ff0bb63b640 1 -- 172.21.15.33:0/637510395 shutdown_connections 2024-09-16T00:23:38.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.619+0000 7ff0bb63b640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0a00a3240 0x7ff0a00a5700 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.619+0000 7ff0bb63b640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0b4106a90 0x7ff0b41a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.619+0000 7ff0bb63b640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0b4101430 0x7ff0b41997c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.619+0000 7ff0bb63b640 1 --2- 172.21.15.33:0/637510395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b4100a90 0x7ff0b4199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:23:38.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.619+0000 7ff0bb63b640 1 -- 172.21.15.33:0/637510395 >> 172.21.15.33:0/637510395 conn(0x7ff0b40fc7d0 msgr2=0x7ff0b40fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:23:38.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.619+0000 7ff0bb63b640 1 -- 172.21.15.33:0/637510395 shutdown_connections 2024-09-16T00:23:38.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:23:38.619+0000 7ff0bb63b640 1 -- 172.21.15.33:0/637510395 wait complete. 2024-09-16T00:23:38.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 393848 8.95 MB/sec execute 166 sec latency 17.987 ms 2024-09-16T00:23:38.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:38 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:23:38.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:38 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:23:38.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:38 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:23:38.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:38 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:23:39.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:39 smithi033 ceph-mon[31243]: pgmap v545: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 217 op/s 2024-09-16T00:23:39.303 INFO:tasks.cephfs.filesystem:scrub status for tag:f7028255-7cc1-4fc7-9308-1ae759a9f6cf - {'path': '/', 'tag': 'f7028255-7cc1-4fc7-9308-1ae759a9f6cf', 'options': 'recursive,force'} 2024-09-16T00:23:39.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:39 smithi060 ceph-mon[35185]: pgmap v545: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 217 op/s 2024-09-16T00:23:39.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 396646 8.97 MB/sec execute 167 sec latency 18.621 ms 2024-09-16T00:23:39.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:39 smithi083 ceph-mon[35242]: pgmap v545: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 217 op/s 2024-09-16T00:23:40.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 399004 8.99 MB/sec execute 168 sec latency 11.748 ms 2024-09-16T00:23:41.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:41 smithi060 ceph-mon[35185]: pgmap v546: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-09-16T00:23:41.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:41 smithi033 ceph-mon[31243]: pgmap v546: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-09-16T00:23:41.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 401666 9.02 MB/sec execute 169 sec latency 18.140 ms 2024-09-16T00:23:41.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:41 smithi083 ceph-mon[35242]: pgmap v546: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-09-16T00:23:42.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 404862 9.08 MB/sec execute 170 sec latency 12.346 ms 2024-09-16T00:23:43.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:43 smithi060 ceph-mon[35185]: pgmap v547: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-09-16T00:23:43.493 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:43 smithi033 ceph-mon[31243]: pgmap v547: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-09-16T00:23:43.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 408190 9.14 MB/sec execute 171 sec latency 17.954 ms 2024-09-16T00:23:43.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:43 smithi083 ceph-mon[35242]: pgmap v547: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-09-16T00:23:44.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 410868 9.16 MB/sec execute 172 sec latency 67.430 ms 2024-09-16T00:23:45.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:45 smithi060 ceph-mon[35185]: pgmap v548: 129 pgs: 129 active+clean; 157 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 225 op/s 2024-09-16T00:23:45.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:45 smithi033 ceph-mon[31243]: pgmap v548: 129 pgs: 129 active+clean; 157 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 225 op/s 2024-09-16T00:23:45.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 413131 9.17 MB/sec execute 173 sec latency 47.254 ms 2024-09-16T00:23:45.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:45 smithi083 ceph-mon[35242]: pgmap v548: 129 pgs: 129 active+clean; 157 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 225 op/s 2024-09-16T00:23:46.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 415775 9.20 MB/sec execute 174 sec latency 15.945 ms 2024-09-16T00:23:47.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:47 smithi060 ceph-mon[35185]: pgmap v549: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 222 op/s 2024-09-16T00:23:47.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:47 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:23:47.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:47 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:23:47.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:47 smithi033 ceph-mon[31243]: pgmap v549: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 222 op/s 2024-09-16T00:23:47.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:47 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:23:47.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:47 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:23:47.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 418966 9.26 MB/sec execute 175 sec latency 11.091 ms 2024-09-16T00:23:47.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:47 smithi083 ceph-mon[35242]: pgmap v549: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 222 op/s 2024-09-16T00:23:47.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:47 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:23:47.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:47 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:23:48.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 422109 9.30 MB/sec execute 176 sec latency 14.985 ms 2024-09-16T00:23:49.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:49 smithi060 ceph-mon[35185]: pgmap v550: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 241 op/s 2024-09-16T00:23:49.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:49 smithi033 ceph-mon[31243]: pgmap v550: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 241 op/s 2024-09-16T00:23:49.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 424611 9.31 MB/sec execute 177 sec latency 16.496 ms 2024-09-16T00:23:49.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:49 smithi083 ceph-mon[35242]: pgmap v550: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 241 op/s 2024-09-16T00:23:50.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 427108 9.35 MB/sec execute 178 sec latency 12.386 ms 2024-09-16T00:23:51.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:51 smithi033 ceph-mon[31243]: pgmap v551: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 248 op/s 2024-09-16T00:23:51.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 429889 9.37 MB/sec execute 179 sec latency 20.468 ms 2024-09-16T00:23:51.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:51 smithi083 ceph-mon[35242]: pgmap v551: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 248 op/s 2024-09-16T00:23:51.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:51 smithi060 ceph-mon[35185]: pgmap v551: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 248 op/s 2024-09-16T00:23:52.663 INFO:tasks.workunit.client.0.smithi033.stdout: 1 433028 9.42 MB/sec execute 180 sec latency 12.782 ms 2024-09-16T00:23:53.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:53 smithi033 ceph-mon[31243]: pgmap v552: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 230 op/s 2024-09-16T00:23:53.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 435786 9.44 MB/sec execute 181 sec latency 14.090 ms 2024-09-16T00:23:53.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:53 smithi083 ceph-mon[35242]: pgmap v552: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 230 op/s 2024-09-16T00:23:53.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:53 smithi060 ceph-mon[35185]: pgmap v552: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 230 op/s 2024-09-16T00:23:54.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 438207 9.46 MB/sec execute 182 sec latency 11.773 ms 2024-09-16T00:23:55.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:55 smithi033 ceph-mon[31243]: pgmap v553: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 220 op/s 2024-09-16T00:23:55.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 440534 9.48 MB/sec execute 183 sec latency 25.516 ms 2024-09-16T00:23:55.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:55 smithi083 ceph-mon[35242]: pgmap v553: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 220 op/s 2024-09-16T00:23:55.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:55 smithi060 ceph-mon[35185]: pgmap v553: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 220 op/s 2024-09-16T00:23:56.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 443649 9.53 MB/sec execute 184 sec latency 30.327 ms 2024-09-16T00:23:57.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:57 smithi033 ceph-mon[31243]: pgmap v554: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.0 MiB/s wr, 254 op/s 2024-09-16T00:23:57.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 446416 9.55 MB/sec execute 185 sec latency 16.398 ms 2024-09-16T00:23:57.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:57 smithi083 ceph-mon[35242]: pgmap v554: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.0 MiB/s wr, 254 op/s 2024-09-16T00:23:57.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:57 smithi060 ceph-mon[35185]: pgmap v554: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.0 MiB/s wr, 254 op/s 2024-09-16T00:23:58.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 449245 9.57 MB/sec execute 186 sec latency 41.315 ms 2024-09-16T00:23:59.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:23:59 smithi033 ceph-mon[31243]: pgmap v555: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 215 op/s 2024-09-16T00:23:59.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 451603 9.60 MB/sec execute 187 sec latency 12.071 ms 2024-09-16T00:23:59.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:23:59 smithi083 ceph-mon[35242]: pgmap v555: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 215 op/s 2024-09-16T00:23:59.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:23:59 smithi060 ceph-mon[35185]: pgmap v555: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 215 op/s 2024-09-16T00:24:00.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 454463 9.64 MB/sec execute 188 sec latency 85.686 ms 2024-09-16T00:24:01.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:01 smithi033 ceph-mon[31243]: pgmap v556: 129 pgs: 129 active+clean; 178 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-09-16T00:24:01.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 457560 9.68 MB/sec execute 189 sec latency 20.490 ms 2024-09-16T00:24:01.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:01 smithi083 ceph-mon[35242]: pgmap v556: 129 pgs: 129 active+clean; 178 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-09-16T00:24:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:01 smithi060 ceph-mon[35185]: pgmap v556: 129 pgs: 129 active+clean; 178 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-09-16T00:24:02.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 459248 9.75 MB/sec execute 190 sec latency 42.381 ms 2024-09-16T00:24:03.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:03 smithi033 ceph-mon[31243]: pgmap v557: 129 pgs: 129 active+clean; 186 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 254 op/s 2024-09-16T00:24:03.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 462477 9.80 MB/sec execute 191 sec latency 16.720 ms 2024-09-16T00:24:03.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:03 smithi083 ceph-mon[35242]: pgmap v557: 129 pgs: 129 active+clean; 186 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 254 op/s 2024-09-16T00:24:03.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:03 smithi060 ceph-mon[35185]: pgmap v557: 129 pgs: 129 active+clean; 186 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 254 op/s 2024-09-16T00:24:04.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 465768 9.84 MB/sec execute 192 sec latency 12.993 ms 2024-09-16T00:24:05.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:05 smithi033 ceph-mon[31243]: pgmap v558: 129 pgs: 129 active+clean; 181 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 234 op/s 2024-09-16T00:24:05.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 468319 9.85 MB/sec execute 193 sec latency 103.943 ms 2024-09-16T00:24:05.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:05 smithi083 ceph-mon[35242]: pgmap v558: 129 pgs: 129 active+clean; 181 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 234 op/s 2024-09-16T00:24:05.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:05 smithi060 ceph-mon[35185]: pgmap v558: 129 pgs: 129 active+clean; 181 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 234 op/s 2024-09-16T00:24:06.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 470249 9.85 MB/sec execute 194 sec latency 30.095 ms 2024-09-16T00:24:07.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:07 smithi033 ceph-mon[31243]: pgmap v559: 129 pgs: 129 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 284 op/s 2024-09-16T00:24:07.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 473345 9.89 MB/sec execute 195 sec latency 13.171 ms 2024-09-16T00:24:07.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:07 smithi083 ceph-mon[35242]: pgmap v559: 129 pgs: 129 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 284 op/s 2024-09-16T00:24:07.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:07 smithi060 ceph-mon[35185]: pgmap v559: 129 pgs: 129 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 284 op/s 2024-09-16T00:24:08.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 476478 9.94 MB/sec execute 196 sec latency 21.801 ms 2024-09-16T00:24:09.305 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:24:09.572 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:24:09.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:09 smithi033 ceph-mon[31243]: pgmap v560: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 242 op/s 2024-09-16T00:24:09.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 479884 9.98 MB/sec execute 197 sec latency 13.091 ms 2024-09-16T00:24:09.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:09 smithi083 ceph-mon[35242]: pgmap v560: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 242 op/s 2024-09-16T00:24:09.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:09 smithi060 ceph-mon[35185]: pgmap v560: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 242 op/s 2024-09-16T00:24:10.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.167+0000 7f776f64c640 1 -- 172.21.15.33:0/3020092957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77681139e0 msgr2=0x7f7768115e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:10.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.167+0000 7f776f64c640 1 --2- 172.21.15.33:0/3020092957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77681139e0 0x7f7768115e70 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7760009ca0 tx=0x7f776002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.167+0000 7f776f64c640 1 -- 172.21.15.33:0/3020092957 shutdown_connections 2024-09-16T00:24:10.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.167+0000 7f776f64c640 1 --2- 172.21.15.33:0/3020092957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77681139e0 0x7f7768115e70 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.167+0000 7f776f64c640 1 --2- 172.21.15.33:0/3020092957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f776810ae80 0x7f7768113370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.167+0000 7f776f64c640 1 --2- 172.21.15.33:0/3020092957 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f776810a560 0x7f776810a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.167+0000 7f776f64c640 1 -- 172.21.15.33:0/3020092957 >> 172.21.15.33:0/3020092957 conn(0x7f776806e2d0 msgr2=0x7f776806e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:10.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.167+0000 7f776f64c640 1 -- 172.21.15.33:0/3020092957 shutdown_connections 2024-09-16T00:24:10.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.167+0000 7f776f64c640 1 -- 172.21.15.33:0/3020092957 wait complete. 2024-09-16T00:24:10.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.168+0000 7f776f64c640 1 Processor -- start 2024-09-16T00:24:10.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.168+0000 7f776f64c640 1 -- start start 2024-09-16T00:24:10.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.168+0000 7f776f64c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f776810a560 0x7f77681a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.168+0000 7f776f64c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f776810ae80 0x7f77681a6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.168+0000 7f776f64c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77681139e0 0x7f77681a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.168+0000 7f776f64c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f776811a830 con 0x7f776810a560 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.168+0000 7f776f64c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f776811a6b0 con 0x7f77681139e0 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.168+0000 7f776f64c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f776811a9b0 con 0x7f776810ae80 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776d3e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f776810a560 0x7f77681a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776cbe7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f776810ae80 0x7f77681a6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776d3e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f776810a560 0x7f77681a6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39186/0 (socket says 172.21.15.33:39186) 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776cbe7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f776810ae80 0x7f77681a6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:50878/0 (socket says 172.21.15.33:50878) 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776d3e8640 1 -- 172.21.15.33:0/2693934750 learned_addr learned my addr 172.21.15.33:0/2693934750 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776dbe9640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77681139e0 0x7f77681a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776cbe7640 1 -- 172.21.15.33:0/2693934750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77681139e0 msgr2=0x7f77681a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776cbe7640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77681139e0 0x7f77681a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776cbe7640 1 -- 172.21.15.33:0/2693934750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f776810a560 msgr2=0x7f77681a6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:10.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776cbe7640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f776810a560 0x7f77681a6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776cbe7640 1 -- 172.21.15.33:0/2693934750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77681a1230 con 0x7f776810ae80 2024-09-16T00:24:10.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776dbe9640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77681139e0 0x7f77681a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:24:10.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776cbe7640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f776810ae80 0x7f77681a6e80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f775800e9c0 tx=0x7f775800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:10.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f775e7fc640 1 -- 172.21.15.33:0/2693934750 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775800ccc0 con 0x7f776810ae80 2024-09-16T00:24:10.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776f64c640 1 -- 172.21.15.33:0/2693934750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77681a1520 con 0x7f776810ae80 2024-09-16T00:24:10.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f775e7fc640 1 -- 172.21.15.33:0/2693934750 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77580035f0 con 0x7f776810ae80 2024-09-16T00:24:10.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f775e7fc640 1 -- 172.21.15.33:0/2693934750 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775800ce60 con 0x7f776810ae80 2024-09-16T00:24:10.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.169+0000 7f776f64c640 1 -- 172.21.15.33:0/2693934750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77681ad810 con 0x7f776810ae80 2024-09-16T00:24:10.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.172+0000 7f775e7fc640 1 -- 172.21.15.33:0/2693934750 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7758010670 con 0x7f776810ae80 2024-09-16T00:24:10.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.172+0000 7f7771a25740 1 Processor -- start 2024-09-16T00:24:10.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.172+0000 7f7771a25740 1 -- start start 2024-09-16T00:24:10.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.172+0000 7f7771a25740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf2f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.172+0000 7f7771a25740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf3a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.172+0000 7f7771a25740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf3a680 0x55ae5cf3ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.172+0000 7f7771a25740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ae5ce952e0 con 0x55ae5cf2f4a0 2024-09-16T00:24:10.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.172+0000 7f7771a25740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ae5ce3cce0 con 0x55ae5cf3a680 2024-09-16T00:24:10.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.172+0000 7f7771a25740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ae5ce912a0 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f776dbe9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf2f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f776d3e8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf3a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f776dbe9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf2f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39194/0 (socket says 172.21.15.33:39194) 2024-09-16T00:24:10.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f776dbe9640 1 -- 172.21.15.33:0/2578346816 learned_addr learned my addr 172.21.15.33:0/2578346816 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:10.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f776cbe7640 1 --2- 172.21.15.33:0/2578346816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf3a680 0x55ae5cf3ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f775e7fc640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f77540a32c0 0x7f77540a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f776d3e8640 1 -- 172.21.15.33:0/2578346816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf3a680 msgr2=0x55ae5cf3ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:10.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f776d3e8640 1 --2- 172.21.15.33:0/2578346816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf3a680 0x55ae5cf3ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f776d3e8640 1 -- 172.21.15.33:0/2578346816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2f4a0 msgr2=0x55ae5cf2f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:10.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f776d3e8640 1 --2- 172.21.15.33:0/2578346816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf2f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f776d3e8640 1 -- 172.21.15.33:0/2578346816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae5ce21440 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f776dbe9640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f77540a32c0 0x7f77540a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f775e7fc640 1 -- 172.21.15.33:0/2693934750 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f77580d0240 con 0x7f776810ae80 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f776d3e8640 1 --2- 172.21.15.33:0/2578346816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf3a140 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7764009cf0 tx=0x7f7764011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2225ff6bd0b07de6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.173+0000 7f77537fe640 1 -- 172.21.15.33:0/2578346816 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764005220 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.174+0000 7f77537fe640 1 -- 172.21.15.33:0/2578346816 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7764004c80 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.174+0000 7f77537fe640 1 -- 172.21.15.33:0/2578346816 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764005220 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.174+0000 7f7771a25740 1 -- 172.21.15.33:0/2578346816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf2fdc0 msgr2=0x55ae5cf3a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.174+0000 7f776dbe9640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f77540a32c0 0x7f77540a5780 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7760009430 tx=0x7f77600095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.174+0000 7f7771a25740 1 --2- 172.21.15.33:0/2578346816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf3a140 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7764009cf0 tx=0x7f7764011810 comp rx=0 tx=0).stop 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.174+0000 7f7771a25740 1 -- 172.21.15.33:0/2578346816 shutdown_connections 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.174+0000 7f7771a25740 1 --2- 172.21.15.33:0/2578346816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf3a680 0x55ae5cf3ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.174+0000 7f7771a25740 1 --2- 172.21.15.33:0/2578346816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf3a140 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.174+0000 7f7771a25740 1 --2- 172.21.15.33:0/2578346816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf2f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.174+0000 7f7771a25740 1 -- 172.21.15.33:0/2578346816 >> 172.21.15.33:0/2578346816 conn(0x55ae5cf2ba30 msgr2=0x55ae5cf2be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:10.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.174+0000 7f7771a25740 1 -- 172.21.15.33:0/2578346816 shutdown_connections 2024-09-16T00:24:10.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.174+0000 7f7771a25740 1 -- 172.21.15.33:0/2578346816 wait complete. 2024-09-16T00:24:10.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.175+0000 7f7771a25740 1 -- start start 2024-09-16T00:24:10.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.175+0000 7f7771a25740 1 Processor -- start 2024-09-16T00:24:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.175+0000 7f7771a25740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf39ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.175+0000 7f7771a25740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf371a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.175+0000 7f7771a25740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf3a680 0x55ae5cf376e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.175+0000 7f7771a25740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ae5cf3d2a0 con 0x55ae5cf2f4a0 2024-09-16T00:24:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.175+0000 7f7771a25740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ae5ce50050 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.175+0000 7f7771a25740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ae5cf3d420 con 0x55ae5cf3a680 2024-09-16T00:24:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.175+0000 7f7771a25740 10 client.0 _ll_register_callbacks cb 0x55ae5ce90160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.175+0000 7f776cbe7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f776dbe9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf3a680 0x55ae5cf376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f776dbe9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf3a680 0x55ae5cf376e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:50884/0 (socket says 172.21.15.33:50884) 2024-09-16T00:24:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f776dbe9640 1 -- 172.21.15.33:0/2792025580 learned_addr learned my addr 172.21.15.33:0/2792025580 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:10.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f776cbe7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf371a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51766/0 (socket says 172.21.15.33:51766) 2024-09-16T00:24:10.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f776d3e8640 1 --2- 172.21.15.33:0/2792025580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf39ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f776dbe9640 1 -- 172.21.15.33:0/2792025580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2fdc0 msgr2=0x55ae5cf371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:10.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f776dbe9640 1 --2- 172.21.15.33:0/2792025580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf371a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f776dbe9640 1 -- 172.21.15.33:0/2792025580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2f4a0 msgr2=0x55ae5cf39ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:10.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f776dbe9640 1 --2- 172.21.15.33:0/2792025580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf39ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f776dbe9640 1 -- 172.21.15.33:0/2792025580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae5cc063c0 con 0x55ae5cf3a680 2024-09-16T00:24:10.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f776dbe9640 1 --2- 172.21.15.33:0/2792025580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf3a680 0x55ae5cf376e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7760006010 tx=0x7f776002a8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14b321b1a605e8b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:10.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f7750ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:24:10.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f7750ff9640 1 -- 172.21.15.33:0/2792025580 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776002fa20 con 0x55ae5cf3a680 2024-09-16T00:24:10.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f7750ff9640 1 -- 172.21.15.33:0/2792025580 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f776002fbc0 con 0x55ae5cf3a680 2024-09-16T00:24:10.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.176+0000 7f7771a25740 1 -- 172.21.15.33:0/2792025580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7764004c80 con 0x55ae5cf3a680 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.177+0000 7f7750ff9640 1 -- 172.21.15.33:0/2792025580 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7760031990 con 0x55ae5cf3a680 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.177+0000 7f7771a25740 10 client.26086 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.177+0000 7f7771a25740 1 -- 172.21.15.33:0/2792025580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ae5cf48b90 con 0x55ae5cf3a680 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.177+0000 7f7750ff9640 1 -- 172.21.15.33:0/2792025580 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7764004c80 con 0x55ae5cf3a680 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.177+0000 7f7750ff9640 1 -- 172.21.15.33:0/2792025580 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f7760052590 con 0x55ae5cf3a680 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.177+0000 7f7771a25740 10 client.26086 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.177+0000 7f7771a25740 10 client.26086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.177+0000 7f7771a25740 20 client.26086 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.177+0000 7f7771a25740 1 --2- 172.21.15.33:0/2792025580 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ae5cf49f50 0x55ae5cf4c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.177+0000 7f7771a25740 4 client.26086 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.177+0000 7f7771a25740 1 -- 172.21.15.33:0/2792025580 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae5cea2850 con 0x55ae5cf49f50 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.177+0000 7f776d3e8640 1 --2- 172.21.15.33:0/2792025580 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ae5cf49f50 0x55ae5cf4c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.178+0000 7f776d3e8640 1 --2- 172.21.15.33:0/2792025580 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ae5cf49f50 0x55ae5cf4c3a0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b3715c8ae129515 server_cookie=914d6d3a9f48e2ff in_seq=0 out_seq=0 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.178+0000 7f7750ff9640 10 client.26086 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.178+0000 7f7750ff9640 1 -- 172.21.15.33:0/2792025580 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ae5cea2850 con 0x55ae5cf49f50 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.178+0000 7f7750ff9640 10 client.26086 handle_command_reply: tid=0 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7771a25740 1 client.26086 shutdown 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7771a25740 1 -- 172.21.15.33:0/2792025580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf3a680 msgr2=0x55ae5cf376e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7771a25740 1 --2- 172.21.15.33:0/2792025580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf3a680 0x55ae5cf376e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7760006010 tx=0x7f776002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7771a25740 1 -- 172.21.15.33:0/2792025580 shutdown_connections 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7771a25740 1 --2- 172.21.15.33:0/2792025580 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ae5cf49f50 0x55ae5cf4c3a0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7771a25740 1 --2- 172.21.15.33:0/2792025580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf3a680 0x55ae5cf376e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7771a25740 1 --2- 172.21.15.33:0/2792025580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf371a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7750ff9640 0 client.26086 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7771a25740 1 --2- 172.21.15.33:0/2792025580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf39ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7771a25740 1 -- 172.21.15.33:0/2792025580 >> 172.21.15.33:0/2792025580 conn(0x55ae5cf2ba30 msgr2=0x55ae5cf31760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:10.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7771a25740 1 -- 172.21.15.33:0/2792025580 shutdown_connections 2024-09-16T00:24:10.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7771a25740 1 -- 172.21.15.33:0/2792025580 wait complete. 2024-09-16T00:24:10.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.179+0000 7f7771a25740 20 client.26086 trim_cache size 0 max 16384 2024-09-16T00:24:10.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.196+0000 7f7771a25740 1 Processor -- start 2024-09-16T00:24:10.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.196+0000 7f7771a25740 1 -- start start 2024-09-16T00:24:10.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.196+0000 7f7771a25740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf42a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.196+0000 7f7771a25740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf42f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.196+0000 7f7771a25740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf3a680 0x55ae5cf51c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.196+0000 7f7771a25740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f77681ba240 con 0x55ae5cf3a680 2024-09-16T00:24:10.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.196+0000 7f7771a25740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f77681b7b10 con 0x55ae5cf2f4a0 2024-09-16T00:24:10.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.196+0000 7f7771a25740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f77681b5960 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f776dbe9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf3a680 0x55ae5cf51c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f776dbe9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf3a680 0x55ae5cf51c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39212/0 (socket says 172.21.15.33:39212) 2024-09-16T00:24:10.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f776dbe9640 1 -- 172.21.15.33:0/105658154 learned_addr learned my addr 172.21.15.33:0/105658154 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:10.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f776cbe7640 1 --2- 172.21.15.33:0/105658154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf42f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f776d3e8640 1 --2- 172.21.15.33:0/105658154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf42a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f776dbe9640 1 -- 172.21.15.33:0/105658154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf2fdc0 msgr2=0x55ae5cf42f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:10.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f776dbe9640 1 --2- 172.21.15.33:0/105658154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf42f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f776dbe9640 1 -- 172.21.15.33:0/105658154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2f4a0 msgr2=0x55ae5cf42a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:10.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f776dbe9640 1 --2- 172.21.15.33:0/105658154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf42a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f776dbe9640 1 -- 172.21.15.33:0/105658154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae5cf2b860 con 0x55ae5cf3a680 2024-09-16T00:24:10.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f776d3e8640 1 --2- 172.21.15.33:0/105658154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf42a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:24:10.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f776dbe9640 1 --2- 172.21.15.33:0/105658154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf3a680 0x55ae5cf51c70 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7760052a00 tx=0x7f7760009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5832325d3022ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:10.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f7752ffd640 1 -- 172.21.15.33:0/105658154 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7760064070 con 0x55ae5cf3a680 2024-09-16T00:24:10.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f7752ffd640 1 -- 172.21.15.33:0/105658154 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f776004d070 con 0x55ae5cf3a680 2024-09-16T00:24:10.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.197+0000 7f7752ffd640 1 -- 172.21.15.33:0/105658154 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7760030800 con 0x55ae5cf3a680 2024-09-16T00:24:10.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.198+0000 7f7771a25740 1 -- 172.21.15.33:0/105658154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf3a680 msgr2=0x55ae5cf51c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:10.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.198+0000 7f7771a25740 1 --2- 172.21.15.33:0/105658154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf3a680 0x55ae5cf51c70 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7760052a00 tx=0x7f7760009b20 comp rx=0 tx=0).stop 2024-09-16T00:24:10.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.198+0000 7f7771a25740 1 -- 172.21.15.33:0/105658154 shutdown_connections 2024-09-16T00:24:10.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.198+0000 7f7771a25740 1 --2- 172.21.15.33:0/105658154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf3a680 0x55ae5cf51c70 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.198+0000 7f7771a25740 1 --2- 172.21.15.33:0/105658154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf42f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.198+0000 7f7771a25740 1 --2- 172.21.15.33:0/105658154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf42a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.198+0000 7f7771a25740 1 -- 172.21.15.33:0/105658154 >> 172.21.15.33:0/105658154 conn(0x55ae5cf2ba30 msgr2=0x55ae5cf35a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:10.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.198+0000 7f7771a25740 1 -- 172.21.15.33:0/105658154 shutdown_connections 2024-09-16T00:24:10.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.198+0000 7f7771a25740 1 -- 172.21.15.33:0/105658154 wait complete. 2024-09-16T00:24:10.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.198+0000 7f7771a25740 1 -- start start 2024-09-16T00:24:10.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f7771a25740 1 Processor -- start 2024-09-16T00:24:10.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f7771a25740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf34700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f7771a25740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf34c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f7771a25740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf3a680 0x55ae5cf33410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f7771a25740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f77681b5fd0 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f7771a25740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f77681b7440 con 0x55ae5cf2f4a0 2024-09-16T00:24:10.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f7771a25740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f77681b66a0 con 0x55ae5cf3a680 2024-09-16T00:24:10.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f7771a25740 10 client.0 _ll_register_callbacks cb 0x7f7764005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:10.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f776cbe7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf34c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f776cbe7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf34c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39226/0 (socket says 172.21.15.33:39226) 2024-09-16T00:24:10.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f776cbe7640 1 -- 172.21.15.33:0/3592382052 learned_addr learned my addr 172.21.15.33:0/3592382052 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:10.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f776d3e8640 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf34700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.199+0000 7f776dbe9640 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf3a680 0x55ae5cf33410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f776cbe7640 1 -- 172.21.15.33:0/3592382052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf3a680 msgr2=0x55ae5cf33410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:10.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f776cbe7640 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf3a680 0x55ae5cf33410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f776cbe7640 1 -- 172.21.15.33:0/3592382052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2f4a0 msgr2=0x55ae5cf34700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f776cbe7640 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf34700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f776cbe7640 1 -- 172.21.15.33:0/3592382052 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae5cf2f2d0 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f776d3e8640 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf34700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f776cbe7640 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf34c40 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f775809c380 tx=0x7f775809ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a148b9c0d618b7f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f7750ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f7750ff9640 1 -- 172.21.15.33:0/3592382052 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77580a4ff0 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f7750ff9640 1 -- 172.21.15.33:0/3592382052 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7758003b10 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f7771a25740 1 -- 172.21.15.33:0/3592382052 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f776004d070 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f7750ff9640 1 -- 172.21.15.33:0/3592382052 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f77580b5260 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f7771a25740 10 client.16653 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f7771a25740 1 -- 172.21.15.33:0/3592382052 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ae5cfb9540 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f7750ff9640 1 -- 172.21.15.33:0/3592382052 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f776004d070 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.200+0000 7f7750ff9640 1 -- 172.21.15.33:0/3592382052 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f775809b2b0 con 0x55ae5cf2fdc0 2024-09-16T00:24:10.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.201+0000 7f7771a25740 10 client.16653 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:10.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.201+0000 7f7771a25740 10 client.16653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:24:10.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.201+0000 7f7771a25740 20 client.16653 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:10.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.201+0000 7f7771a25740 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ae5cfb9c70 0x55ae5cfbc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:10.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.201+0000 7f7771a25740 4 client.16653 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:24:10.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.201+0000 7f7771a25740 1 -- 172.21.15.33:0/3592382052 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ae5cf37a70 con 0x55ae5cfb9c70 2024-09-16T00:24:10.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.201+0000 7f776d3e8640 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ae5cfb9c70 0x55ae5cfbc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:10.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.201+0000 7f776d3e8640 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ae5cfb9c70 0x55ae5cfbc090 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc0189f6620e76ea server_cookie=7523d462320b8c60 in_seq=0 out_seq=0 2024-09-16T00:24:10.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.201+0000 7f7750ff9640 10 client.16653 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:10.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7750ff9640 1 -- 172.21.15.33:0/3592382052 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ae5cf37a70 con 0x55ae5cfb9c70 2024-09-16T00:24:10.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7750ff9640 10 client.16653 handle_command_reply: tid=0 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7771a25740 1 client.16653 shutdown 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7771a25740 1 -- 172.21.15.33:0/3592382052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2fdc0 msgr2=0x55ae5cf34c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7771a25740 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf34c40 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f775809c380 tx=0x7f775809ca80 comp rx=0 tx=0).stop 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7771a25740 1 -- 172.21.15.33:0/3592382052 shutdown_connections 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7771a25740 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ae5cfb9c70 0x55ae5cfbc090 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7771a25740 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae5cf3a680 0x55ae5cf33410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7771a25740 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ae5cf2fdc0 0x55ae5cf34c40 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7771a25740 1 --2- 172.21.15.33:0/3592382052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae5cf2f4a0 0x55ae5cf34700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7771a25740 1 -- 172.21.15.33:0/3592382052 >> 172.21.15.33:0/3592382052 conn(0x55ae5cf2ba30 msgr2=0x55ae5cf38a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7750ff9640 0 client.16653 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7771a25740 1 -- 172.21.15.33:0/3592382052 shutdown_connections 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7771a25740 1 -- 172.21.15.33:0/3592382052 wait complete. 2024-09-16T00:24:10.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.202+0000 7f7771a25740 20 client.16653 trim_cache size 0 max 16384 2024-09-16T00:24:10.210 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:24:10.210 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:24:10.210 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:24:10.210 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:24:10.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.203+0000 7f776f64c640 1 -- 172.21.15.33:0/2693934750 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f77540a32c0 msgr2=0x7f77540a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:10.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.203+0000 7f776f64c640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f77540a32c0 0x7f77540a5780 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7760009430 tx=0x7f77600095c0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.203+0000 7f776f64c640 1 -- 172.21.15.33:0/2693934750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f776810ae80 msgr2=0x7f77681a6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:10.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.203+0000 7f776f64c640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f776810ae80 0x7f77681a6e80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f775800e9c0 tx=0x7f775800ee80 comp rx=0 tx=0).stop 2024-09-16T00:24:10.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.203+0000 7f776f64c640 1 -- 172.21.15.33:0/2693934750 shutdown_connections 2024-09-16T00:24:10.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.203+0000 7f776f64c640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f77540a32c0 0x7f77540a5780 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.203+0000 7f776f64c640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77681139e0 0x7f77681a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.203+0000 7f776f64c640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f776810ae80 0x7f77681a6e80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.203+0000 7f776f64c640 1 --2- 172.21.15.33:0/2693934750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f776810a560 0x7f77681a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:10.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.203+0000 7f776f64c640 1 -- 172.21.15.33:0/2693934750 >> 172.21.15.33:0/2693934750 conn(0x7f776806e2d0 msgr2=0x7f7768071ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:10.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.203+0000 7f776f64c640 1 -- 172.21.15.33:0/2693934750 shutdown_connections 2024-09-16T00:24:10.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:10.203+0000 7f776f64c640 1 -- 172.21.15.33:0/2693934750 wait complete. 2024-09-16T00:24:10.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 482046 9.97 MB/sec execute 198 sec latency 96.163 ms 2024-09-16T00:24:11.118 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:24:11.118 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:24:11.390 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:24:11.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:11 smithi033 ceph-mon[31243]: pgmap v561: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 261 op/s 2024-09-16T00:24:11.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 484197 9.98 MB/sec execute 199 sec latency 21.491 ms 2024-09-16T00:24:11.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:11 smithi083 ceph-mon[35242]: pgmap v561: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 261 op/s 2024-09-16T00:24:11.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:11 smithi060 ceph-mon[35185]: pgmap v561: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 261 op/s 2024-09-16T00:24:11.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.911+0000 7f812bd3f640 1 -- 172.21.15.33:0/168077204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8124102ef0 msgr2=0x7f812410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:11.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.911+0000 7f812bd3f640 1 --2- 172.21.15.33:0/168077204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8124102ef0 0x7f812410da40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f811c009ca0 tx=0x7f811c02a950 comp rx=0 tx=0).stop 2024-09-16T00:24:11.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.911+0000 7f812bd3f640 1 -- 172.21.15.33:0/168077204 shutdown_connections 2024-09-16T00:24:11.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.911+0000 7f812bd3f640 1 --2- 172.21.15.33:0/168077204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8124102ef0 0x7f812410da40 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:11.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.911+0000 7f812bd3f640 1 --2- 172.21.15.33:0/168077204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8124102570 0x7f81241029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:11.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.911+0000 7f812bd3f640 1 --2- 172.21.15.33:0/168077204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8124100e20 0x7f8124101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:11.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.911+0000 7f812bd3f640 1 -- 172.21.15.33:0/168077204 >> 172.21.15.33:0/168077204 conn(0x7f81240fc7d0 msgr2=0x7f81240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:11.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.911+0000 7f812bd3f640 1 -- 172.21.15.33:0/168077204 shutdown_connections 2024-09-16T00:24:11.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.911+0000 7f812bd3f640 1 -- 172.21.15.33:0/168077204 wait complete. 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.912+0000 7f812bd3f640 1 Processor -- start 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.912+0000 7f812bd3f640 1 -- start start 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.912+0000 7f812bd3f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8124100e20 0x7f81241992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.912+0000 7f812bd3f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8124102570 0x7f81241997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.912+0000 7f812bd3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8124102ef0 0x7f81241a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.912+0000 7f812bd3f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8124110f80 con 0x7f8124100e20 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.912+0000 7f812bd3f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8124110e00 con 0x7f8124102ef0 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.912+0000 7f812bd3f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8124111100 con 0x7f8124102570 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.912+0000 7f8129adb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8124100e20 0x7f81241992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.912+0000 7f8129adb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8124100e20 0x7f81241992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55900/0 (socket says 172.21.15.33:55900) 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.912+0000 7f8129adb640 1 -- 172.21.15.33:0/2274595657 learned_addr learned my addr 172.21.15.33:0/2274595657 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.912+0000 7f81292da640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8124102570 0x7f81241997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:11.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.913+0000 7f812a2dc640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8124102ef0 0x7f81241a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.913+0000 7f8129adb640 1 -- 172.21.15.33:0/2274595657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8124102570 msgr2=0x7f81241997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.913+0000 7f8129adb640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8124102570 0x7f81241997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.913+0000 7f8129adb640 1 -- 172.21.15.33:0/2274595657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8124102ef0 msgr2=0x7f81241a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.913+0000 7f8129adb640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8124102ef0 0x7f81241a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.913+0000 7f8129adb640 1 -- 172.21.15.33:0/2274595657 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81241a4870 con 0x7f8124100e20 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.913+0000 7f8129adb640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8124100e20 0x7f81241992b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f812000ea60 tx=0x7f812000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.913+0000 7f811affd640 1 -- 172.21.15.33:0/2274595657 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812000cde0 con 0x7f8124100e20 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.913+0000 7f811affd640 1 -- 172.21.15.33:0/2274595657 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81200035f0 con 0x7f8124100e20 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.913+0000 7f812bd3f640 1 -- 172.21.15.33:0/2274595657 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81241a4a40 con 0x7f8124100e20 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.913+0000 7f811affd640 1 -- 172.21.15.33:0/2274595657 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120010670 con 0x7f8124100e20 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.913+0000 7f812bd3f640 1 -- 172.21.15.33:0/2274595657 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81241a4f50 con 0x7f8124100e20 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.914+0000 7f812bd3f640 1 -- 172.21.15.33:0/2274595657 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8124101200 con 0x7f8124100e20 2024-09-16T00:24:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.916+0000 7f811affd640 1 -- 172.21.15.33:0/2274595657 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f81200037a0 con 0x7f8124100e20 2024-09-16T00:24:11.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.917+0000 7f811affd640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81100a3240 0x7f81100a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:11.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.917+0000 7f81292da640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81100a3240 0x7f81100a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:11.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.917+0000 7f811affd640 1 -- 172.21.15.33:0/2274595657 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f81200d0190 con 0x7f8124100e20 2024-09-16T00:24:11.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.917+0000 7f81292da640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81100a3240 0x7f81100a5700 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f812406aac0 tx=0x7f811400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:11.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:11.921+0000 7f811affd640 1 -- 172.21.15.33:0/2274595657 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f812009ca60 con 0x7f8124100e20 2024-09-16T00:24:12.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.151+0000 7f812bd3f640 1 -- 172.21.15.33:0/2274595657 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f812406a560 con 0x7f8124100e20 2024-09-16T00:24:12.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.152+0000 7f811affd640 1 -- 172.21.15.33:0/2274595657 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f812406a560 con 0x7f8124100e20 2024-09-16T00:24:12.153 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:24:12.153 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:24:12.154 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:24:12.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.154+0000 7f812bd3f640 1 -- 172.21.15.33:0/2274595657 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81100a3240 msgr2=0x7f81100a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:12.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.154+0000 7f812bd3f640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81100a3240 0x7f81100a5700 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f812406aac0 tx=0x7f811400a2d0 comp rx=0 tx=0).stop 2024-09-16T00:24:12.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.154+0000 7f812bd3f640 1 -- 172.21.15.33:0/2274595657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8124100e20 msgr2=0x7f81241992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:12.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.154+0000 7f812bd3f640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8124100e20 0x7f81241992b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f812000ea60 tx=0x7f812000ef20 comp rx=0 tx=0).stop 2024-09-16T00:24:12.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.154+0000 7f812bd3f640 1 -- 172.21.15.33:0/2274595657 shutdown_connections 2024-09-16T00:24:12.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.154+0000 7f812bd3f640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81100a3240 0x7f81100a5700 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:12.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.154+0000 7f812bd3f640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8124102ef0 0x7f81241a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:12.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.154+0000 7f812bd3f640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8124102570 0x7f81241997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:12.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.154+0000 7f812bd3f640 1 --2- 172.21.15.33:0/2274595657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8124100e20 0x7f81241992b0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:12.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.154+0000 7f812bd3f640 1 -- 172.21.15.33:0/2274595657 >> 172.21.15.33:0/2274595657 conn(0x7f81240fc7d0 msgr2=0x7f8124106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:12.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.154+0000 7f812bd3f640 1 -- 172.21.15.33:0/2274595657 shutdown_connections 2024-09-16T00:24:12.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:12.154+0000 7f812bd3f640 1 -- 172.21.15.33:0/2274595657 wait complete. 2024-09-16T00:24:12.521 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:12 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2274595657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:24:12.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 487495 10.03 MB/sec execute 200 sec latency 12.592 ms 2024-09-16T00:24:12.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:12 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2274595657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:24:12.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:12 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2274595657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:24:12.994 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:24:13.266 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:24:13.536 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:13 smithi033 ceph-mon[31243]: pgmap v562: 129 pgs: 129 active+clean; 178 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 294 op/s 2024-09-16T00:24:13.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 490764 10.07 MB/sec execute 201 sec latency 15.606 ms 2024-09-16T00:24:13.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:13 smithi083 ceph-mon[35242]: pgmap v562: 129 pgs: 129 active+clean; 178 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 294 op/s 2024-09-16T00:24:13.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:13 smithi060 ceph-mon[35185]: pgmap v562: 129 pgs: 129 active+clean; 178 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 294 op/s 2024-09-16T00:24:14.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.016+0000 7f37311c2640 1 -- 172.21.15.33:0/4292329081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f372c102d30 msgr2=0x7f372c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:14.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.016+0000 7f37311c2640 1 --2- 172.21.15.33:0/4292329081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f372c102d30 0x7f372c103110 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f371c009860 tx=0x7f371c02a340 comp rx=0 tx=0).stop 2024-09-16T00:24:14.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.016+0000 7f37311c2640 1 -- 172.21.15.33:0/4292329081 shutdown_connections 2024-09-16T00:24:14.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.016+0000 7f37311c2640 1 --2- 172.21.15.33:0/4292329081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c1082a0 0x7f372c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.016+0000 7f37311c2640 1 --2- 172.21.15.33:0/4292329081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f372c1036e0 0x7f372c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.016+0000 7f37311c2640 1 --2- 172.21.15.33:0/4292329081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f372c102d30 0x7f372c103110 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.016+0000 7f37311c2640 1 -- 172.21.15.33:0/4292329081 >> 172.21.15.33:0/4292329081 conn(0x7f372c0fc7d0 msgr2=0x7f372c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:14.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.016+0000 7f37311c2640 1 -- 172.21.15.33:0/4292329081 shutdown_connections 2024-09-16T00:24:14.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.016+0000 7f37311c2640 1 -- 172.21.15.33:0/4292329081 wait complete. 2024-09-16T00:24:14.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.017+0000 7f37311c2640 1 Processor -- start 2024-09-16T00:24:14.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.017+0000 7f37311c2640 1 -- start start 2024-09-16T00:24:14.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f37311c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f372c102d30 0x7f372c100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f37311c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f372c1036e0 0x7f372c100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f37311c2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c1082a0 0x7f372c101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f37311c2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1120a0 con 0x7f372c1036e0 2024-09-16T00:24:14.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f37311c2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f372c111f20 con 0x7f372c1082a0 2024-09-16T00:24:14.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f37311c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f372c112220 con 0x7f372c102d30 2024-09-16T00:24:14.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f372ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f372c102d30 0x7f372c100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f372ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f372c102d30 0x7f372c100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37772/0 (socket says 172.21.15.33:37772) 2024-09-16T00:24:14.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f372ad9d640 1 -- 172.21.15.33:0/4041641639 learned_addr learned my addr 172.21.15.33:0/4041641639 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:14.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f372b59e640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c1082a0 0x7f372c101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f372ad9d640 1 -- 172.21.15.33:0/4041641639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c1082a0 msgr2=0x7f372c101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:14.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f372ad9d640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c1082a0 0x7f372c101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f372ad9d640 1 -- 172.21.15.33:0/4041641639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f372c1036e0 msgr2=0x7f372c100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f372a59c640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f372c1036e0 0x7f372c100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f372ad9d640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f372c1036e0 0x7f372c100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f372ad9d640 1 -- 172.21.15.33:0/4041641639 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f372c1a6ab0 con 0x7f372c102d30 2024-09-16T00:24:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.018+0000 7f372a59c640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f372c1036e0 0x7f372c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:24:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.019+0000 7f372ad9d640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f372c102d30 0x7f372c100990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f371c035040 tx=0x7f371c033b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.019+0000 7f3713fff640 1 -- 172.21.15.33:0/4041641639 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c037040 con 0x7f372c102d30 2024-09-16T00:24:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.019+0000 7f3713fff640 1 -- 172.21.15.33:0/4041641639 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f371c040070 con 0x7f372c102d30 2024-09-16T00:24:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.019+0000 7f37311c2640 1 -- 172.21.15.33:0/4041641639 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f372c1a6d40 con 0x7f372c102d30 2024-09-16T00:24:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.019+0000 7f3713fff640 1 -- 172.21.15.33:0/4041641639 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c003a10 con 0x7f372c102d30 2024-09-16T00:24:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.019+0000 7f37311c2640 1 -- 172.21.15.33:0/4041641639 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f372c1a7210 con 0x7f372c102d30 2024-09-16T00:24:14.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.022+0000 7f3713fff640 1 -- 172.21.15.33:0/4041641639 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f371c003bb0 con 0x7f372c102d30 2024-09-16T00:24:14.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.022+0000 7f373359b740 1 Processor -- start 2024-09-16T00:24:14.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.022+0000 7f373359b740 1 -- start start 2024-09-16T00:24:14.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f373359b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 0x560bda4de090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f373359b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4de5d0 0x560bda4dea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f373359b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4f1b80 0x560bda4f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f373359b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560bda4442f0 con 0x560bda4dfb40 2024-09-16T00:24:14.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f373359b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560bda3ebcf0 con 0x560bda4de5d0 2024-09-16T00:24:14.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f373359b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560bda4402b0 con 0x560bda4f1b80 2024-09-16T00:24:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f372b59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 0x560bda4de090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f372ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4de5d0 0x560bda4dea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f372b59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 0x560bda4de090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55926/0 (socket says 172.21.15.33:55926) 2024-09-16T00:24:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f372b59e640 1 -- 172.21.15.33:0/1267498235 learned_addr learned my addr 172.21.15.33:0/1267498235 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f372a59c640 1 --2- 172.21.15.33:0/1267498235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4f1b80 0x560bda4f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f372b59e640 1 -- 172.21.15.33:0/1267498235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4f1b80 msgr2=0x560bda4f3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f372b59e640 1 --2- 172.21.15.33:0/1267498235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4f1b80 0x560bda4f3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f372b59e640 1 -- 172.21.15.33:0/1267498235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4de5d0 msgr2=0x560bda4dea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f372b59e640 1 --2- 172.21.15.33:0/1267498235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4de5d0 0x560bda4dea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f372b59e640 1 -- 172.21.15.33:0/1267498235 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bda3d03a0 con 0x560bda4dfb40 2024-09-16T00:24:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f372ad9d640 1 --2- 172.21.15.33:0/1267498235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4de5d0 0x560bda4dea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:14.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f3713fff640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37140a3340 0x7f37140a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.023+0000 7f372b59e640 1 --2- 172.21.15.33:0/1267498235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 0x560bda4de090 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3724009f20 tx=0x7f3724010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8231ba14149a56df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:14.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.024+0000 7f372ad9d640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37140a3340 0x7f37140a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.024+0000 7f3713fff640 1 -- 172.21.15.33:0/4041641639 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f371c0eff50 con 0x7f372c102d30 2024-09-16T00:24:14.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.024+0000 7f37117fa640 1 -- 172.21.15.33:0/1267498235 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3724011700 con 0x560bda4dfb40 2024-09-16T00:24:14.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.024+0000 7f37117fa640 1 -- 172.21.15.33:0/1267498235 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3724011d80 con 0x560bda4dfb40 2024-09-16T00:24:14.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.024+0000 7f37117fa640 1 -- 172.21.15.33:0/1267498235 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372401a380 con 0x560bda4dfb40 2024-09-16T00:24:14.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.024+0000 7f373359b740 1 -- 172.21.15.33:0/1267498235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 msgr2=0x560bda4de090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:14.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.024+0000 7f372ad9d640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37140a3340 0x7f37140a5800 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f371c02a930 tx=0x7f371c0f3910 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:14.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.024+0000 7f373359b740 1 --2- 172.21.15.33:0/1267498235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 0x560bda4de090 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3724009f20 tx=0x7f3724010860 comp rx=0 tx=0).stop 2024-09-16T00:24:14.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.025+0000 7f373359b740 1 -- 172.21.15.33:0/1267498235 shutdown_connections 2024-09-16T00:24:14.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.025+0000 7f373359b740 1 --2- 172.21.15.33:0/1267498235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4f1b80 0x560bda4f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.025+0000 7f373359b740 1 --2- 172.21.15.33:0/1267498235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4de5d0 0x560bda4dea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.025+0000 7f373359b740 1 --2- 172.21.15.33:0/1267498235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 0x560bda4de090 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.025+0000 7f373359b740 1 -- 172.21.15.33:0/1267498235 >> 172.21.15.33:0/1267498235 conn(0x560bda4a7310 msgr2=0x560bda4dac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:14.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.025+0000 7f373359b740 1 -- 172.21.15.33:0/1267498235 shutdown_connections 2024-09-16T00:24:14.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.025+0000 7f373359b740 1 -- 172.21.15.33:0/1267498235 wait complete. 2024-09-16T00:24:14.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.025+0000 7f373359b740 1 -- start start 2024-09-16T00:24:14.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.025+0000 7f373359b740 1 Processor -- start 2024-09-16T00:24:14.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f373359b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4f1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f373359b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 0x560bda4ee810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f373359b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4f1b80 0x560bda4eee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f373359b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560bda4f4880 con 0x560bda4dfb40 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f373359b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560bda3ff060 con 0x560bda4f1b80 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f373359b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560bda4f4a00 con 0x560bda4de5d0 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f373359b740 10 client.0 _ll_register_callbacks cb 0x560bda1528c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 0x560bda4ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372b59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4f1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372b59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4f1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37792/0 (socket says 172.21.15.33:37792) 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 0x560bda4ee810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55940/0 (socket says 172.21.15.33:55940) 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372b59e640 1 -- 172.21.15.33:0/202382935 learned_addr learned my addr 172.21.15.33:0/202382935 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372b59e640 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4f1b80 0x560bda4eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372a59c640 1 -- 172.21.15.33:0/202382935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 msgr2=0x560bda4f1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372a59c640 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4f1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372a59c640 1 -- 172.21.15.33:0/202382935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4f1b80 msgr2=0x560bda4eee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372a59c640 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4f1b80 0x560bda4eee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372a59c640 1 -- 172.21.15.33:0/202382935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bda1b53c0 con 0x560bda4dfb40 2024-09-16T00:24:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372b59e640 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4f1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f372a59c640 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 0x560bda4ee810 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x560bda4df070 tx=0x7f37200127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad361c84c24ef29 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f370eff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:24:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.026+0000 7f370eff5640 1 -- 172.21.15.33:0/202382935 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3720004220 con 0x560bda4dfb40 2024-09-16T00:24:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f373359b740 1 -- 172.21.15.33:0/202382935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3724011d80 con 0x560bda4dfb40 2024-09-16T00:24:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f370eff5640 1 -- 172.21.15.33:0/202382935 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f37200043c0 con 0x560bda4dfb40 2024-09-16T00:24:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f370eff5640 1 -- 172.21.15.33:0/202382935 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f37200035f0 con 0x560bda4dfb40 2024-09-16T00:24:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f373359b740 10 client.16668 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f373359b740 1 -- 172.21.15.33:0/202382935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560bda500380 con 0x560bda4dfb40 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f370eff5640 1 -- 172.21.15.33:0/202382935 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3724011d80 con 0x560bda4dfb40 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f370eff5640 1 -- 172.21.15.33:0/202382935 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f3720019030 con 0x560bda4dfb40 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f373359b740 10 client.16668 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f373359b740 10 client.16668 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f373359b740 20 client.16668 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f373359b740 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560bda501cf0 0x560bda504110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f373359b740 4 client.16668 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f373359b740 1 -- 172.21.15.33:0/202382935 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bda451860 con 0x560bda501cf0 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.027+0000 7f372b59e640 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560bda501cf0 0x560bda504110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.028+0000 7f372b59e640 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560bda501cf0 0x560bda504110 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=95ed4823c283abbd server_cookie=e6d183dd6f0e356d in_seq=0 out_seq=0 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.028+0000 7f370eff5640 10 client.16668 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:24:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f370eff5640 1 -- 172.21.15.33:0/202382935 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560bda451860 con 0x560bda501cf0 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f370eff5640 10 client.16668 handle_command_reply: tid=0 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f373359b740 1 client.16668 shutdown 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f373359b740 1 -- 172.21.15.33:0/202382935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 msgr2=0x560bda4ee810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f373359b740 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 0x560bda4ee810 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x560bda4df070 tx=0x7f37200127c0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f373359b740 1 -- 172.21.15.33:0/202382935 shutdown_connections 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f373359b740 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560bda501cf0 0x560bda504110 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f373359b740 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4f1b80 0x560bda4eee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f373359b740 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4dfb40 0x560bda4ee810 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f373359b740 1 --2- 172.21.15.33:0/202382935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4f1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f373359b740 1 -- 172.21.15.33:0/202382935 >> 172.21.15.33:0/202382935 conn(0x560bda4a7310 msgr2=0x560bda4e0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f370eff5640 0 client.16668 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f373359b740 1 -- 172.21.15.33:0/202382935 shutdown_connections 2024-09-16T00:24:14.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f373359b740 1 -- 172.21.15.33:0/202382935 wait complete. 2024-09-16T00:24:14.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.029+0000 7f373359b740 20 client.16668 trim_cache size 0 max 16384 2024-09-16T00:24:14.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f373359b740 1 Processor -- start 2024-09-16T00:24:14.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f373359b740 1 -- start start 2024-09-16T00:24:14.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f373359b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4fdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f373359b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 0x560bda4ed0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f373359b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4f1b80 0x560bda4ed620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f373359b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1af8e0 con 0x560bda4f1b80 2024-09-16T00:24:14.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f373359b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1ad1b0 con 0x560bda4dfb40 2024-09-16T00:24:14.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f373359b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1ab000 con 0x560bda4de5d0 2024-09-16T00:24:14.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f372a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 0x560bda4ed0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f372a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 0x560bda4ed0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:49664/0 (socket says 172.21.15.33:49664) 2024-09-16T00:24:14.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f372a59c640 1 -- 172.21.15.33:0/539873634 learned_addr learned my addr 172.21.15.33:0/539873634 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:14.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f372b59e640 1 --2- 172.21.15.33:0/539873634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4fdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f372b59e640 1 --2- 172.21.15.33:0/539873634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4f1b80 0x560bda4ed620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f372a59c640 1 -- 172.21.15.33:0/539873634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 msgr2=0x560bda4fdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:14.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f372a59c640 1 --2- 172.21.15.33:0/539873634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4fdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f372a59c640 1 -- 172.21.15.33:0/539873634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4f1b80 msgr2=0x560bda4ed620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:14.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f372a59c640 1 --2- 172.21.15.33:0/539873634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4f1b80 0x560bda4ed620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.043+0000 7f372a59c640 1 -- 172.21.15.33:0/539873634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bda4f1880 con 0x560bda4dfb40 2024-09-16T00:24:14.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.044+0000 7f372b59e640 1 --2- 172.21.15.33:0/539873634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4fdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:24:14.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.044+0000 7f372a59c640 1 --2- 172.21.15.33:0/539873634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 0x560bda4ed0e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3720009970 tx=0x7f37200107f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2073f804397061be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:14.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.044+0000 7f3710ff9640 1 -- 172.21.15.33:0/539873634 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372002e070 con 0x560bda4dfb40 2024-09-16T00:24:14.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.044+0000 7f3710ff9640 1 -- 172.21.15.33:0/539873634 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3720005520 con 0x560bda4dfb40 2024-09-16T00:24:14.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.044+0000 7f3710ff9640 1 -- 172.21.15.33:0/539873634 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37200048e0 con 0x560bda4dfb40 2024-09-16T00:24:14.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.045+0000 7f373359b740 1 -- 172.21.15.33:0/539873634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 msgr2=0x560bda4ed0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:14.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.045+0000 7f373359b740 1 --2- 172.21.15.33:0/539873634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 0x560bda4ed0e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3720009970 tx=0x7f37200107f0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.045+0000 7f373359b740 1 -- 172.21.15.33:0/539873634 shutdown_connections 2024-09-16T00:24:14.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.045+0000 7f373359b740 1 --2- 172.21.15.33:0/539873634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4f1b80 0x560bda4ed620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.045+0000 7f373359b740 1 --2- 172.21.15.33:0/539873634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 0x560bda4ed0e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.045+0000 7f373359b740 1 --2- 172.21.15.33:0/539873634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4fdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.045+0000 7f373359b740 1 -- 172.21.15.33:0/539873634 >> 172.21.15.33:0/539873634 conn(0x560bda4a7310 msgr2=0x560bda4e9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.045+0000 7f373359b740 1 -- 172.21.15.33:0/539873634 shutdown_connections 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.045+0000 7f373359b740 1 -- 172.21.15.33:0/539873634 wait complete. 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f373359b740 1 -- start start 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f373359b740 1 Processor -- start 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f373359b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4f9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f373359b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 0x560bda4fa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f373359b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4f1b80 0x560bda4f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f373359b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1ab670 con 0x560bda4f1b80 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f373359b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1acae0 con 0x560bda4dfb40 2024-09-16T00:24:14.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f373359b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1abd40 con 0x560bda4de5d0 2024-09-16T00:24:14.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f373359b740 10 client.0 _ll_register_callbacks cb 0x7f3720009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:14.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f372b59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4f9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f372a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 0x560bda4fa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f372b59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4f9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37808/0 (socket says 172.21.15.33:37808) 2024-09-16T00:24:14.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f372a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 0x560bda4fa480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:49670/0 (socket says 172.21.15.33:49670) 2024-09-16T00:24:14.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f372b59e640 1 -- 172.21.15.33:0/1255631413 learned_addr learned my addr 172.21.15.33:0/1255631413 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:14.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f372b59e640 1 --2- 172.21.15.33:0/1255631413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4f1b80 0x560bda4f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f372a59c640 1 -- 172.21.15.33:0/1255631413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 msgr2=0x560bda4f9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:14.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f372a59c640 1 --2- 172.21.15.33:0/1255631413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4f9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f372a59c640 1 -- 172.21.15.33:0/1255631413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4f1b80 msgr2=0x560bda4f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:14.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f372a59c640 1 --2- 172.21.15.33:0/1255631413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4f1b80 0x560bda4f4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.046+0000 7f372a59c640 1 -- 172.21.15.33:0/1255631413 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bda4f5510 con 0x560bda4dfb40 2024-09-16T00:24:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.047+0000 7f372a59c640 1 --2- 172.21.15.33:0/1255631413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 0x560bda4fa480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x560bda4cf380 tx=0x7f3720005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7493e94d635d3cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.047+0000 7f370eff5640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:24:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.047+0000 7f370eff5640 1 -- 172.21.15.33:0/1255631413 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372002e040 con 0x560bda4dfb40 2024-09-16T00:24:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.047+0000 7f373359b740 1 -- 172.21.15.33:0/1255631413 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3720005520 con 0x560bda4dfb40 2024-09-16T00:24:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.047+0000 7f370eff5640 1 -- 172.21.15.33:0/1255631413 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f372000f5c0 con 0x560bda4dfb40 2024-09-16T00:24:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.047+0000 7f370eff5640 1 -- 172.21.15.33:0/1255631413 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3720040050 con 0x560bda4dfb40 2024-09-16T00:24:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.047+0000 7f373359b740 10 client.26192 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.047+0000 7f373359b740 1 -- 172.21.15.33:0/1255631413 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560bda56b560 con 0x560bda4dfb40 2024-09-16T00:24:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.047+0000 7f370eff5640 1 -- 172.21.15.33:0/1255631413 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3720005520 con 0x560bda4dfb40 2024-09-16T00:24:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f370eff5640 1 -- 172.21.15.33:0/1255631413 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f3720019040 con 0x560bda4dfb40 2024-09-16T00:24:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f373359b740 10 client.26192 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f373359b740 10 client.26192 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:24:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f373359b740 20 client.26192 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f373359b740 1 --2- 172.21.15.33:0/1255631413 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560bda56bbb0 0x560bda56dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f373359b740 4 client.26192 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:24:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f373359b740 1 -- 172.21.15.33:0/1255631413 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560bda44bd70 con 0x560bda56bbb0 2024-09-16T00:24:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f372b59e640 1 --2- 172.21.15.33:0/1255631413 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560bda56bbb0 0x560bda56dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f372b59e640 1 --2- 172.21.15.33:0/1255631413 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560bda56bbb0 0x560bda56dfa0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=94b414d6a2ba2d94 server_cookie=41e2a98f884abc29 in_seq=0 out_seq=0 2024-09-16T00:24:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f370eff5640 10 client.26192 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:24:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f370eff5640 1 -- 172.21.15.33:0/1255631413 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560bda44bd70 con 0x560bda56bbb0 2024-09-16T00:24:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f370eff5640 10 client.26192 handle_command_reply: tid=0 2024-09-16T00:24:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.048+0000 7f373359b740 1 client.26192 shutdown 2024-09-16T00:24:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f373359b740 1 -- 172.21.15.33:0/1255631413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 msgr2=0x560bda4fa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:14.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f373359b740 1 --2- 172.21.15.33:0/1255631413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 0x560bda4fa480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x560bda4cf380 tx=0x7f3720005e80 comp rx=0 tx=0).stop 2024-09-16T00:24:14.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f373359b740 1 -- 172.21.15.33:0/1255631413 shutdown_connections 2024-09-16T00:24:14.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f373359b740 1 --2- 172.21.15.33:0/1255631413 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x560bda56bbb0 0x560bda56dfa0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f373359b740 1 --2- 172.21.15.33:0/1255631413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bda4f1b80 0x560bda4f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f373359b740 1 --2- 172.21.15.33:0/1255631413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bda4dfb40 0x560bda4fa480 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f373359b740 1 --2- 172.21.15.33:0/1255631413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bda4de5d0 0x560bda4f9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f373359b740 1 -- 172.21.15.33:0/1255631413 >> 172.21.15.33:0/1255631413 conn(0x560bda4a7310 msgr2=0x560bda4ed020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:14.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f370eff5640 0 client.26192 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:24:14.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f373359b740 1 -- 172.21.15.33:0/1255631413 shutdown_connections 2024-09-16T00:24:14.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f373359b740 1 -- 172.21.15.33:0/1255631413 wait complete. 2024-09-16T00:24:14.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f373359b740 20 client.26192 trim_cache size 0 max 16384 2024-09-16T00:24:14.059 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:24:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f37311c2640 1 -- 172.21.15.33:0/4041641639 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37140a3340 msgr2=0x7f37140a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f37311c2640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37140a3340 0x7f37140a5800 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f371c02a930 tx=0x7f371c0f3910 comp rx=0 tx=0).stop 2024-09-16T00:24:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.049+0000 7f37311c2640 1 -- 172.21.15.33:0/4041641639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f372c102d30 msgr2=0x7f372c100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.050+0000 7f37311c2640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f372c102d30 0x7f372c100990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f371c035040 tx=0x7f371c033b60 comp rx=0 tx=0).stop 2024-09-16T00:24:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.050+0000 7f37311c2640 1 -- 172.21.15.33:0/4041641639 shutdown_connections 2024-09-16T00:24:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.050+0000 7f37311c2640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37140a3340 0x7f37140a5800 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.050+0000 7f37311c2640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c1082a0 0x7f372c101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.050+0000 7f37311c2640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f372c1036e0 0x7f372c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.050+0000 7f37311c2640 1 --2- 172.21.15.33:0/4041641639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f372c102d30 0x7f372c100990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.050+0000 7f37311c2640 1 -- 172.21.15.33:0/4041641639 >> 172.21.15.33:0/4041641639 conn(0x7f372c0fc7d0 msgr2=0x7f372c105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.050+0000 7f37311c2640 1 -- 172.21.15.33:0/4041641639 shutdown_connections 2024-09-16T00:24:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:14.050+0000 7f37311c2640 1 -- 172.21.15.33:0/4041641639 wait complete. 2024-09-16T00:24:14.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 493942 10.11 MB/sec execute 202 sec latency 20.455 ms 2024-09-16T00:24:14.873 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:24:15.160 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:24:15.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: pgmap v563: 129 pgs: 129 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 258 op/s 2024-09-16T00:24:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:24:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:24:15.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:24:15.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:24:15.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 496198 10.10 MB/sec execute 203 sec latency 78.928 ms 2024-09-16T00:24:15.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: pgmap v563: 129 pgs: 129 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 258 op/s 2024-09-16T00:24:15.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:24:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:24:15.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:24:15.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:24:15.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.694+0000 7fec2059b640 1 -- 172.21.15.33:0/710449386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec18103640 msgr2=0x7fec1810bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:15.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.694+0000 7fec2059b640 1 --2- 172.21.15.33:0/710449386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec18103640 0x7fec1810bb30 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009900 tx=0x7fec0c02a480 comp rx=0 tx=0).stop 2024-09-16T00:24:15.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.694+0000 7fec2059b640 1 -- 172.21.15.33:0/710449386 shutdown_connections 2024-09-16T00:24:15.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.694+0000 7fec2059b640 1 --2- 172.21.15.33:0/710449386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec1810c150 0x7fec1810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.694+0000 7fec2059b640 1 --2- 172.21.15.33:0/710449386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec18103640 0x7fec1810bb30 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.694+0000 7fec2059b640 1 --2- 172.21.15.33:0/710449386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec18102d20 0x7fec18103100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.694+0000 7fec2059b640 1 -- 172.21.15.33:0/710449386 >> 172.21.15.33:0/710449386 conn(0x7fec180fc780 msgr2=0x7fec180feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:15.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.694+0000 7fec2059b640 1 -- 172.21.15.33:0/710449386 shutdown_connections 2024-09-16T00:24:15.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.694+0000 7fec2059b640 1 -- 172.21.15.33:0/710449386 wait complete. 2024-09-16T00:24:15.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.694+0000 7fec2059b640 1 Processor -- start 2024-09-16T00:24:15.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.694+0000 7fec2059b640 1 -- start start 2024-09-16T00:24:15.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec2059b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec18102d20 0x7fec18198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec2059b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec18103640 0x7fec181989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec2059b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec1810c150 0x7fec1819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec2059b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fec18111f50 con 0x7fec18102d20 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec2059b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fec18111dd0 con 0x7fec18103640 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec2059b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fec181120d0 con 0x7fec1810c150 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1e337640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec18102d20 0x7fec18198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1db36640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec18103640 0x7fec181989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1e337640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec18102d20 0x7fec18198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55970/0 (socket says 172.21.15.33:55970) 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1eb38640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec1810c150 0x7fec1819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1e337640 1 -- 172.21.15.33:0/3507525891 learned_addr learned my addr 172.21.15.33:0/3507525891 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1db36640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec18103640 0x7fec181989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:49696/0 (socket says 172.21.15.33:49696) 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1e337640 1 -- 172.21.15.33:0/3507525891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec1810c150 msgr2=0x7fec1819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1e337640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec1810c150 0x7fec1819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1e337640 1 -- 172.21.15.33:0/3507525891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec18103640 msgr2=0x7fec181989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1e337640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec18103640 0x7fec181989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1e337640 1 -- 172.21.15.33:0/3507525891 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec1819d430 con 0x7fec18102d20 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1db36640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec18103640 0x7fec181989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec1e337640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec18102d20 0x7fec18198480 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fec1000e9f0 tx=0x7fec1000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.695+0000 7fec177fe640 1 -- 172.21.15.33:0/3507525891 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1000cd80 con 0x7fec18102d20 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.696+0000 7fec177fe640 1 -- 172.21.15.33:0/3507525891 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec10010910 con 0x7fec18102d20 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.696+0000 7fec2059b640 1 -- 172.21.15.33:0/3507525891 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec1819d720 con 0x7fec18102d20 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.696+0000 7fec177fe640 1 -- 172.21.15.33:0/3507525891 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec10018990 con 0x7fec18102d20 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.696+0000 7fec2059b640 1 -- 172.21.15.33:0/3507525891 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec181a5000 con 0x7fec18102d20 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.699+0000 7fec177fe640 1 -- 172.21.15.33:0/3507525891 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fec10010ac0 con 0x7fec18102d20 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.699+0000 7fec22974740 1 Processor -- start 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.699+0000 7fec22974740 1 -- start start 2024-09-16T00:24:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.699+0000 7fec22974740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 0x55672af86090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.699+0000 7fec22974740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af865d0 0x55672af86a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.699+0000 7fec22974740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af91370 0x55672af93760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.699+0000 7fec22974740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55672aeec2f0 con 0x55672af87b40 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.699+0000 7fec22974740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55672ae93cf0 con 0x55672af91370 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.699+0000 7fec22974740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55672aee82b0 con 0x55672af865d0 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1e337640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af865d0 0x55672af86a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1eb38640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 0x55672af86090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1e337640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af865d0 0x55672af86a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37816/0 (socket says 172.21.15.33:37816) 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1db36640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af91370 0x55672af93760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1eb38640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 0x55672af86090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55982/0 (socket says 172.21.15.33:55982) 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1e337640 1 -- 172.21.15.33:0/880723261 learned_addr learned my addr 172.21.15.33:0/880723261 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec177fe640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec000a32c0 0x7fec000a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1eb38640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec000a32c0 0x7fec000a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1db36640 1 -- 172.21.15.33:0/880723261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af865d0 msgr2=0x55672af86a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:15.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec177fe640 1 -- 172.21.15.33:0/3507525891 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fec100cfff0 con 0x7fec18102d20 2024-09-16T00:24:15.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1db36640 1 --2- 172.21.15.33:0/880723261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af865d0 0x55672af86a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1db36640 1 -- 172.21.15.33:0/880723261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 msgr2=0x55672af86090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:15.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1db36640 1 --2- 172.21.15.33:0/880723261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 0x55672af86090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1db36640 1 -- 172.21.15.33:0/880723261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55672ae783a0 con 0x55672af91370 2024-09-16T00:24:15.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1eb38640 1 --2- 172.21.15.33:0/880723261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 0x55672af86090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:15.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1db36640 1 --2- 172.21.15.33:0/880723261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af91370 0x55672af93760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fec0c005940 tx=0x7fec0c005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d455fab5e02904e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:15.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec1eb38640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec000a32c0 0x7fec000a5780 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fec08009fd0 tx=0x7fec08004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:15.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.700+0000 7fec14ff9640 1 -- 172.21.15.33:0/880723261 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c035040 con 0x55672af91370 2024-09-16T00:24:15.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.701+0000 7fec14ff9640 1 -- 172.21.15.33:0/880723261 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec0c031830 con 0x55672af91370 2024-09-16T00:24:15.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.701+0000 7fec14ff9640 1 -- 172.21.15.33:0/880723261 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c004750 con 0x55672af91370 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.701+0000 7fec22974740 1 -- 172.21.15.33:0/880723261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af91370 msgr2=0x55672af93760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.701+0000 7fec22974740 1 --2- 172.21.15.33:0/880723261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af91370 0x55672af93760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fec0c005940 tx=0x7fec0c005d50 comp rx=0 tx=0).stop 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.701+0000 7fec22974740 1 -- 172.21.15.33:0/880723261 shutdown_connections 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.701+0000 7fec22974740 1 --2- 172.21.15.33:0/880723261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af91370 0x55672af93760 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.701+0000 7fec22974740 1 --2- 172.21.15.33:0/880723261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af865d0 0x55672af86a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.701+0000 7fec22974740 1 --2- 172.21.15.33:0/880723261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 0x55672af86090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.701+0000 7fec22974740 1 -- 172.21.15.33:0/880723261 >> 172.21.15.33:0/880723261 conn(0x55672af4f310 msgr2=0x55672af82c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.701+0000 7fec22974740 1 -- 172.21.15.33:0/880723261 shutdown_connections 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.701+0000 7fec22974740 1 -- 172.21.15.33:0/880723261 wait complete. 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec22974740 1 -- start start 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec22974740 1 Processor -- start 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec22974740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af865d0 0x55672af90cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec22974740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 0x55672af8df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec22974740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af91370 0x55672af8e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec22974740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55672af94010 con 0x55672af87b40 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec22974740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55672aea7060 con 0x55672af865d0 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec22974740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55672af94190 con 0x55672af91370 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec22974740 10 client.0 _ll_register_callbacks cb 0x55672abfa8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec1db36640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 0x55672af8df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec1db36640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 0x55672af8df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55992/0 (socket says 172.21.15.33:55992) 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec1db36640 1 -- 172.21.15.33:0/1748246761 learned_addr learned my addr 172.21.15.33:0/1748246761 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec1eb38640 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af91370 0x55672af8e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.702+0000 7fec1e337640 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af865d0 0x55672af90cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7fec1db36640 1 -- 172.21.15.33:0/1748246761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af91370 msgr2=0x55672af8e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7fec1db36640 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af91370 0x55672af8e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7fec1db36640 1 -- 172.21.15.33:0/1748246761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af865d0 msgr2=0x55672af90cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:15.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7fec1db36640 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af865d0 0x55672af90cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7fec1db36640 1 -- 172.21.15.33:0/1748246761 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55672ac5d3c0 con 0x55672af87b40 2024-09-16T00:24:15.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7fec1e337640 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af865d0 0x55672af90cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:24:15.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7fec1db36640 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 0x55672af8df70 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fec0c02a9a0 tx=0x7fec0c031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d8f4de26049d822 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:15.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7febfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:24:15.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7febfdffb640 1 -- 172.21.15.33:0/1748246761 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c054060 con 0x55672af87b40 2024-09-16T00:24:15.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7fec22974740 1 -- 172.21.15.33:0/1748246761 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec0c031830 con 0x55672af87b40 2024-09-16T00:24:15.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7febfdffb640 1 -- 172.21.15.33:0/1748246761 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fec0c031630 con 0x55672af87b40 2024-09-16T00:24:15.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7febfdffb640 1 -- 172.21.15.33:0/1748246761 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fec0c05b050 con 0x55672af87b40 2024-09-16T00:24:15.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7fec22974740 10 client.16686 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:15.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7fec22974740 1 -- 172.21.15.33:0/1748246761 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55672af9fa20 con 0x55672af87b40 2024-09-16T00:24:15.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.703+0000 7febfdffb640 1 -- 172.21.15.33:0/1748246761 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec0c031830 con 0x55672af87b40 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.704+0000 7febfdffb640 1 -- 172.21.15.33:0/1748246761 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fec0c046450 con 0x55672af87b40 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.704+0000 7fec22974740 10 client.16686 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.704+0000 7fec22974740 10 client.16686 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.704+0000 7fec22974740 20 client.16686 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.704+0000 7fec22974740 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55672afa0d70 0x55672afa3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.704+0000 7fec22974740 4 client.16686 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.704+0000 7fec22974740 1 -- 172.21.15.33:0/1748246761 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55672aef9860 con 0x55672afa0d70 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.704+0000 7fec1e337640 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55672afa0d70 0x55672afa3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.704+0000 7fec1e337640 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55672afa0d70 0x55672afa3160 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2e227e97fcc1128f server_cookie=712b0f82bf644132 in_seq=0 out_seq=0 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.704+0000 7febfdffb640 10 client.16686 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7febfdffb640 1 -- 172.21.15.33:0/1748246761 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55672aef9860 con 0x55672afa0d70 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7febfdffb640 10 client.16686 handle_command_reply: tid=0 2024-09-16T00:24:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7fec22974740 1 client.16686 shutdown 2024-09-16T00:24:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7fec22974740 1 -- 172.21.15.33:0/1748246761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 msgr2=0x55672af8df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7fec22974740 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 0x55672af8df70 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fec0c02a9a0 tx=0x7fec0c031a90 comp rx=0 tx=0).stop 2024-09-16T00:24:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7fec22974740 1 -- 172.21.15.33:0/1748246761 shutdown_connections 2024-09-16T00:24:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7fec22974740 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af91370 0x55672af8e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7fec22974740 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af87b40 0x55672af8df70 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7fec22974740 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55672afa0d70 0x55672afa3160 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7fec22974740 1 --2- 172.21.15.33:0/1748246761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af865d0 0x55672af90cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7fec22974740 1 -- 172.21.15.33:0/1748246761 >> 172.21.15.33:0/1748246761 conn(0x55672af4f310 msgr2=0x55672af88960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7febfdffb640 0 client.16686 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:24:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7fec22974740 1 -- 172.21.15.33:0/1748246761 shutdown_connections 2024-09-16T00:24:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7fec22974740 1 -- 172.21.15.33:0/1748246761 wait complete. 2024-09-16T00:24:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.705+0000 7fec22974740 20 client.16686 trim_cache size 0 max 16384 2024-09-16T00:24:15.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.718+0000 7fec22974740 1 Processor -- start 2024-09-16T00:24:15.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.718+0000 7fec22974740 1 -- start start 2024-09-16T00:24:15.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec22974740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af865d0 0x55672af82e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec22974740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af87b40 0x55672af89120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec22974740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af91370 0x55672af89660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec22974740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fec181b1a40 con 0x55672af91370 2024-09-16T00:24:15.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec22974740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fec181af310 con 0x55672af87b40 2024-09-16T00:24:15.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec22974740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fec181ad160 con 0x55672af865d0 2024-09-16T00:24:15.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec1eb38640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af91370 0x55672af89660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec1db36640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af87b40 0x55672af89120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec1e337640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af865d0 0x55672af82e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec1eb38640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af91370 0x55672af89660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56008/0 (socket says 172.21.15.33:56008) 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec1eb38640 1 -- 172.21.15.33:0/3314326926 learned_addr learned my addr 172.21.15.33:0/3314326926 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec1db36640 1 -- 172.21.15.33:0/3314326926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af865d0 msgr2=0x55672af82e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec1db36640 1 --2- 172.21.15.33:0/3314326926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af865d0 0x55672af82e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec1db36640 1 -- 172.21.15.33:0/3314326926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af91370 msgr2=0x55672af89660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec1db36640 1 --2- 172.21.15.33:0/3314326926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af91370 0x55672af89660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec1db36640 1 -- 172.21.15.33:0/3314326926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55672af90ff0 con 0x55672af87b40 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec1eb38640 1 --2- 172.21.15.33:0/3314326926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af91370 0x55672af89660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7fec1db36640 1 --2- 172.21.15.33:0/3314326926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af87b40 0x55672af89120 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fec0c031550 tx=0x7fec0c031b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e2e80bec553b74b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.719+0000 7febfffff640 1 -- 172.21.15.33:0/3314326926 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c054070 con 0x55672af87b40 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.720+0000 7febfffff640 1 -- 172.21.15.33:0/3314326926 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec0c046510 con 0x55672af87b40 2024-09-16T00:24:15.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.720+0000 7febfffff640 1 -- 172.21.15.33:0/3314326926 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c035070 con 0x55672af87b40 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.720+0000 7fec22974740 1 -- 172.21.15.33:0/3314326926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af87b40 msgr2=0x55672af89120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.720+0000 7fec22974740 1 --2- 172.21.15.33:0/3314326926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af87b40 0x55672af89120 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fec0c031550 tx=0x7fec0c031b00 comp rx=0 tx=0).stop 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.720+0000 7fec22974740 1 -- 172.21.15.33:0/3314326926 shutdown_connections 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.720+0000 7fec22974740 1 --2- 172.21.15.33:0/3314326926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af91370 0x55672af89660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.720+0000 7fec22974740 1 --2- 172.21.15.33:0/3314326926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af87b40 0x55672af89120 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.720+0000 7fec22974740 1 --2- 172.21.15.33:0/3314326926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af865d0 0x55672af82e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.720+0000 7fec22974740 1 -- 172.21.15.33:0/3314326926 >> 172.21.15.33:0/3314326926 conn(0x55672af4f310 msgr2=0x55672af82700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.720+0000 7fec22974740 1 -- 172.21.15.33:0/3314326926 shutdown_connections 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.720+0000 7fec22974740 1 -- 172.21.15.33:0/3314326926 wait complete. 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.720+0000 7fec22974740 1 -- start start 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec22974740 1 Processor -- start 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec22974740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af865d0 0x55672afa63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec22974740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af87b40 0x55672afa3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec22974740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af91370 0x55672afa3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec22974740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fec181ad7d0 con 0x55672af865d0 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec22974740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fec181aec40 con 0x55672af91370 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec22974740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fec181adea0 con 0x55672af87b40 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec22974740 10 client.0 _ll_register_callbacks cb 0x7fec0c040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec1e337640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af865d0 0x55672afa63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec1e337640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af865d0 0x55672afa63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56012/0 (socket says 172.21.15.33:56012) 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec1e337640 1 -- 172.21.15.33:0/1971636165 learned_addr learned my addr 172.21.15.33:0/1971636165 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec1db36640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af87b40 0x55672afa3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec1eb38640 1 --2- 172.21.15.33:0/1971636165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af91370 0x55672afa3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec1e337640 1 -- 172.21.15.33:0/1971636165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af87b40 msgr2=0x55672afa3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec1e337640 1 --2- 172.21.15.33:0/1971636165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af87b40 0x55672afa3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec1e337640 1 -- 172.21.15.33:0/1971636165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af91370 msgr2=0x55672afa3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec1e337640 1 --2- 172.21.15.33:0/1971636165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af91370 0x55672afa3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.721+0000 7fec1e337640 1 -- 172.21.15.33:0/1971636165 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55672afa4360 con 0x55672af865d0 2024-09-16T00:24:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7fec1e337640 1 --2- 172.21.15.33:0/1971636165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af865d0 0x55672afa63e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fec100cd6c0 tx=0x7fec1009c080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d70c9e399f1c73 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7febfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:24:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7febfdffb640 1 -- 172.21.15.33:0/1971636165 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec10026040 con 0x55672af865d0 2024-09-16T00:24:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7febfdffb640 1 -- 172.21.15.33:0/1971636165 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fec100bc0b0 con 0x55672af865d0 2024-09-16T00:24:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7fec22974740 1 -- 172.21.15.33:0/1971636165 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec0c046510 con 0x55672af865d0 2024-09-16T00:24:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7febfdffb640 1 -- 172.21.15.33:0/1971636165 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fec1009c240 con 0x55672af865d0 2024-09-16T00:24:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7fec22974740 10 client.16692 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7fec22974740 1 -- 172.21.15.33:0/1971636165 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55672b00fa00 con 0x55672af865d0 2024-09-16T00:24:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7febfdffb640 1 -- 172.21.15.33:0/1971636165 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec0c046510 con 0x55672af865d0 2024-09-16T00:24:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7febfdffb640 1 -- 172.21.15.33:0/1971636165 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fec100a4970 con 0x55672af865d0 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7fec22974740 10 client.16692 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7fec22974740 10 client.16692 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7fec22974740 20 client.16692 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7fec22974740 1 --2- 172.21.15.33:0/1971636165 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55672b0100d0 0x55672b0124f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7fec22974740 4 client.16692 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.722+0000 7fec22974740 1 -- 172.21.15.33:0/1971636165 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55672aef3d70 con 0x55672b0100d0 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7fec1db36640 1 --2- 172.21.15.33:0/1971636165 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55672b0100d0 0x55672b0124f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7fec1db36640 1 --2- 172.21.15.33:0/1971636165 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55672b0100d0 0x55672b0124f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2aa1010d3614a3c8 server_cookie=d9df0a291caf9c78 in_seq=0 out_seq=0 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7febfdffb640 10 client.16692 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7febfdffb640 1 -- 172.21.15.33:0/1971636165 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55672aef3d70 con 0x55672b0100d0 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7febfdffb640 10 client.16692 handle_command_reply: tid=0 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7fec22974740 1 client.16692 shutdown 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7fec22974740 1 -- 172.21.15.33:0/1971636165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af865d0 msgr2=0x55672afa63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:15.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7fec22974740 1 --2- 172.21.15.33:0/1971636165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af865d0 0x55672afa63e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fec100cd6c0 tx=0x7fec1009c080 comp rx=0 tx=0).stop 2024-09-16T00:24:15.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7fec22974740 1 -- 172.21.15.33:0/1971636165 shutdown_connections 2024-09-16T00:24:15.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7fec22974740 1 --2- 172.21.15.33:0/1971636165 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55672b0100d0 0x55672b0124f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7fec22974740 1 --2- 172.21.15.33:0/1971636165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55672af91370 0x55672afa3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7fec22974740 1 --2- 172.21.15.33:0/1971636165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55672af87b40 0x55672afa3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7fec22974740 1 --2- 172.21.15.33:0/1971636165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55672af865d0 0x55672afa63e0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7fec22974740 1 -- 172.21.15.33:0/1971636165 >> 172.21.15.33:0/1971636165 conn(0x55672af4f310 msgr2=0x55672af8f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:15.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.723+0000 7febfdffb640 0 client.16692 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:24:15.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec22974740 1 -- 172.21.15.33:0/1971636165 shutdown_connections 2024-09-16T00:24:15.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec22974740 1 -- 172.21.15.33:0/1971636165 wait complete. 2024-09-16T00:24:15.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec22974740 20 client.16692 trim_cache size 0 max 16384 2024-09-16T00:24:15.737 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:24:15.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec2059b640 1 -- 172.21.15.33:0/3507525891 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec000a32c0 msgr2=0x7fec000a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:15.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec2059b640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec000a32c0 0x7fec000a5780 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fec08009fd0 tx=0x7fec08004ab0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec2059b640 1 -- 172.21.15.33:0/3507525891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec18102d20 msgr2=0x7fec18198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:15.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec2059b640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec18102d20 0x7fec18198480 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fec1000e9f0 tx=0x7fec1000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec2059b640 1 -- 172.21.15.33:0/3507525891 shutdown_connections 2024-09-16T00:24:15.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec2059b640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec000a32c0 0x7fec000a5780 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec2059b640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec1810c150 0x7fec1819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec2059b640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec18103640 0x7fec181989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec2059b640 1 --2- 172.21.15.33:0/3507525891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec18102d20 0x7fec18198480 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:15.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec2059b640 1 -- 172.21.15.33:0/3507525891 >> 172.21.15.33:0/3507525891 conn(0x7fec180fc780 msgr2=0x7fec180fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:15.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec2059b640 1 -- 172.21.15.33:0/3507525891 shutdown_connections 2024-09-16T00:24:15.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:15.724+0000 7fec2059b640 1 -- 172.21.15.33:0/3507525891 wait complete. 2024-09-16T00:24:15.738 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: pgmap v563: 129 pgs: 129 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 258 op/s 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.739 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:24:15.740 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:24:15.740 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:24:15.740 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:24:16.449 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:24:16.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:16 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi060 to 177.0M 2024-09-16T00:24:16.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:16 smithi033 ceph-mon[31243]: Unable to set osd_memory_target on smithi060 to 185699225: error parsing value: Value '185699225' is below minimum 939524096 2024-09-16T00:24:16.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:16 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi083 to 176.0M 2024-09-16T00:24:16.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:16 smithi033 ceph-mon[31243]: Unable to set osd_memory_target on smithi083 to 184589619: error parsing value: Value '184589619' is below minimum 939524096 2024-09-16T00:24:16.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:16 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi033 to 177.0M 2024-09-16T00:24:16.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:16 smithi033 ceph-mon[31243]: Unable to set osd_memory_target on smithi033 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-16T00:24:16.664 INFO:tasks.workunit.client.0.smithi033.stdout: 1 498312 10.11 MB/sec execute 204 sec latency 22.699 ms 2024-09-16T00:24:16.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:16 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi060 to 177.0M 2024-09-16T00:24:16.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:16 smithi083 ceph-mon[35242]: Unable to set osd_memory_target on smithi060 to 185699225: error parsing value: Value '185699225' is below minimum 939524096 2024-09-16T00:24:16.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:16 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi083 to 176.0M 2024-09-16T00:24:16.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:16 smithi083 ceph-mon[35242]: Unable to set osd_memory_target on smithi083 to 184589619: error parsing value: Value '184589619' is below minimum 939524096 2024-09-16T00:24:16.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:16 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi033 to 177.0M 2024-09-16T00:24:16.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:16 smithi083 ceph-mon[35242]: Unable to set osd_memory_target on smithi033 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-16T00:24:16.724 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:24:16.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:16 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi060 to 177.0M 2024-09-16T00:24:16.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:16 smithi060 ceph-mon[35185]: Unable to set osd_memory_target on smithi060 to 185699225: error parsing value: Value '185699225' is below minimum 939524096 2024-09-16T00:24:16.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:16 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi083 to 176.0M 2024-09-16T00:24:16.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:16 smithi060 ceph-mon[35185]: Unable to set osd_memory_target on smithi083 to 184589619: error parsing value: Value '184589619' is below minimum 939524096 2024-09-16T00:24:16.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:16 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi033 to 177.0M 2024-09-16T00:24:16.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:16 smithi060 ceph-mon[35185]: Unable to set osd_memory_target on smithi033 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-16T00:24:17.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.232+0000 7fc299f01640 1 -- 172.21.15.33:0/3448733501 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc294102570 msgr2=0x7fc2941029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:17.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.232+0000 7fc299f01640 1 --2- 172.21.15.33:0/3448733501 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc294102570 0x7fc2941029b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc288009900 tx=0x7fc28802a480 comp rx=0 tx=0).stop 2024-09-16T00:24:17.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.232+0000 7fc299f01640 1 -- 172.21.15.33:0/3448733501 shutdown_connections 2024-09-16T00:24:17.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.233+0000 7fc299f01640 1 --2- 172.21.15.33:0/3448733501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc294102ef0 0x7fc29410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.233+0000 7fc299f01640 1 --2- 172.21.15.33:0/3448733501 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc294102570 0x7fc2941029b0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.233+0000 7fc299f01640 1 --2- 172.21.15.33:0/3448733501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc294100e20 0x7fc294101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.233+0000 7fc299f01640 1 -- 172.21.15.33:0/3448733501 >> 172.21.15.33:0/3448733501 conn(0x7fc2940fc7d0 msgr2=0x7fc2940febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:17.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.233+0000 7fc299f01640 1 -- 172.21.15.33:0/3448733501 shutdown_connections 2024-09-16T00:24:17.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.233+0000 7fc299f01640 1 -- 172.21.15.33:0/3448733501 wait complete. 2024-09-16T00:24:17.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.233+0000 7fc299f01640 1 Processor -- start 2024-09-16T00:24:17.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.234+0000 7fc299f01640 1 -- start start 2024-09-16T00:24:17.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.234+0000 7fc299f01640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc294100e20 0x7fc2940fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.234+0000 7fc299f01640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc294102570 0x7fc2940fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.234+0000 7fc299f01640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc294102ef0 0x7fc2940fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.234+0000 7fc299f01640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc294110f80 con 0x7fc294102ef0 2024-09-16T00:24:17.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.234+0000 7fc299f01640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc294110e00 con 0x7fc294102570 2024-09-16T00:24:17.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.234+0000 7fc299f01640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc294111100 con 0x7fc294100e20 2024-09-16T00:24:17.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.234+0000 7fc2937fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc294100e20 0x7fc2940fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.234+0000 7fc293fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc294102ef0 0x7fc2940fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.234+0000 7fc2937fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc294100e20 0x7fc2940fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37862/0 (socket says 172.21.15.33:37862) 2024-09-16T00:24:17.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.234+0000 7fc2937fe640 1 -- 172.21.15.33:0/259085493 learned_addr learned my addr 172.21.15.33:0/259085493 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:17.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.235+0000 7fc292ffd640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc294102570 0x7fc2940fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.235+0000 7fc2937fe640 1 -- 172.21.15.33:0/259085493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc294102570 msgr2=0x7fc2940fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:17.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.235+0000 7fc2937fe640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc294102570 0x7fc2940fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.235+0000 7fc2937fe640 1 -- 172.21.15.33:0/259085493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc294102ef0 msgr2=0x7fc2940fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:17.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.235+0000 7fc2937fe640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc294102ef0 0x7fc2940fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.235+0000 7fc2937fe640 1 -- 172.21.15.33:0/259085493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2941a69d0 con 0x7fc294100e20 2024-09-16T00:24:17.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.235+0000 7fc2937fe640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc294100e20 0x7fc2940fcc80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc28400e9f0 tx=0x7fc28400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:17.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.235+0000 7fc290ff9640 1 -- 172.21.15.33:0/259085493 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28400cd80 con 0x7fc294100e20 2024-09-16T00:24:17.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.235+0000 7fc299f01640 1 -- 172.21.15.33:0/259085493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2941a6cc0 con 0x7fc294100e20 2024-09-16T00:24:17.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.235+0000 7fc290ff9640 1 -- 172.21.15.33:0/259085493 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc284005500 con 0x7fc294100e20 2024-09-16T00:24:17.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.235+0000 7fc290ff9640 1 -- 172.21.15.33:0/259085493 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284010690 con 0x7fc294100e20 2024-09-16T00:24:17.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.235+0000 7fc299f01640 1 -- 172.21.15.33:0/259085493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2941a7180 con 0x7fc294100e20 2024-09-16T00:24:17.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.238+0000 7fc290ff9640 1 -- 172.21.15.33:0/259085493 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc2840046c0 con 0x7fc294100e20 2024-09-16T00:24:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.238+0000 7fc29c2da740 1 Processor -- start 2024-09-16T00:24:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.238+0000 7fc29c2da740 1 -- start start 2024-09-16T00:24:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc29c2da740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 0x55ad019ca090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc29c2da740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019ca5d0 0x55ad019caa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc29c2da740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019d7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc29c2da740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ad019302f0 con 0x55ad019d5370 2024-09-16T00:24:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc29c2da740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ad018d7cf0 con 0x55ad019ca5d0 2024-09-16T00:24:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc29c2da740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ad0192c2b0 con 0x55ad019cbb40 2024-09-16T00:24:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc293fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 0x55ad019ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc2937fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019ca5d0 0x55ad019caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc292ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019d7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc293fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 0x55ad019ca090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37872/0 (socket says 172.21.15.33:37872) 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc293fff640 1 -- 172.21.15.33:0/3921497673 learned_addr learned my addr 172.21.15.33:0/3921497673 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc292ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019d7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56032/0 (socket says 172.21.15.33:56032) 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc293fff640 1 -- 172.21.15.33:0/3921497673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019ca5d0 msgr2=0x55ad019caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc293fff640 1 --2- 172.21.15.33:0/3921497673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019ca5d0 0x55ad019caa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc293fff640 1 -- 172.21.15.33:0/3921497673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 msgr2=0x55ad019d7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc293fff640 1 --2- 172.21.15.33:0/3921497673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019d7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc293fff640 1 -- 172.21.15.33:0/3921497673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad018bc3a0 con 0x55ad019cbb40 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.239+0000 7fc292ffd640 1 --2- 172.21.15.33:0/3921497673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019d7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.240+0000 7fc293fff640 1 --2- 172.21.15.33:0/3921497673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 0x55ad019ca090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009f90 tx=0x7fc28c010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23740a1aa0b722d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.240+0000 7fc290ff9640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc27c0a3240 0x7fc27c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.240+0000 7fc279ffb640 1 -- 172.21.15.33:0/3921497673 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28c011700 con 0x55ad019cbb40 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.240+0000 7fc2937fe640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc27c0a3240 0x7fc27c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.240+0000 7fc279ffb640 1 -- 172.21.15.33:0/3921497673 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc28c011d80 con 0x55ad019cbb40 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.240+0000 7fc279ffb640 1 -- 172.21.15.33:0/3921497673 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28c018380 con 0x55ad019cbb40 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.240+0000 7fc290ff9640 1 -- 172.21.15.33:0/259085493 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc2840d01b0 con 0x7fc294100e20 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.241+0000 7fc29c2da740 1 -- 172.21.15.33:0/3921497673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 msgr2=0x55ad019ca090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.241+0000 7fc2937fe640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc27c0a3240 0x7fc27c0a5700 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc2840cdba0 tx=0x7fc2840baf90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.241+0000 7fc29c2da740 1 --2- 172.21.15.33:0/3921497673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 0x55ad019ca090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009f90 tx=0x7fc28c010860 comp rx=0 tx=0).stop 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.241+0000 7fc29c2da740 1 -- 172.21.15.33:0/3921497673 shutdown_connections 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.241+0000 7fc29c2da740 1 --2- 172.21.15.33:0/3921497673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019d7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.241+0000 7fc29c2da740 1 --2- 172.21.15.33:0/3921497673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019ca5d0 0x55ad019caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.241+0000 7fc29c2da740 1 --2- 172.21.15.33:0/3921497673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 0x55ad019ca090 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.241+0000 7fc29c2da740 1 -- 172.21.15.33:0/3921497673 >> 172.21.15.33:0/3921497673 conn(0x55ad01993310 msgr2=0x55ad019c6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.241+0000 7fc29c2da740 1 -- 172.21.15.33:0/3921497673 shutdown_connections 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.241+0000 7fc29c2da740 1 -- 172.21.15.33:0/3921497673 wait complete. 2024-09-16T00:24:17.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc29c2da740 1 -- start start 2024-09-16T00:24:17.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc29c2da740 1 Processor -- start 2024-09-16T00:24:17.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc29c2da740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019ca5d0 0x55ad019d4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc29c2da740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 0x55ad019d2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc29c2da740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc29c2da740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ad019d8070 con 0x55ad019d5370 2024-09-16T00:24:17.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc29c2da740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ad018eb060 con 0x55ad019cbb40 2024-09-16T00:24:17.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc29c2da740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ad019d81f0 con 0x55ad019ca5d0 2024-09-16T00:24:17.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc29c2da740 10 client.0 _ll_register_callbacks cb 0x55ad0163e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:17.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc292ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 0x55ad019d2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc293fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019ca5d0 0x55ad019d4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc293fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc292ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 0x55ad019d2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:49750/0 (socket says 172.21.15.33:49750) 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.242+0000 7fc292ffd640 1 -- 172.21.15.33:0/3803089567 learned_addr learned my addr 172.21.15.33:0/3803089567 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc292ffd640 1 -- 172.21.15.33:0/3803089567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019ca5d0 msgr2=0x55ad019d4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc292ffd640 1 --2- 172.21.15.33:0/3803089567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019ca5d0 0x55ad019d4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc292ffd640 1 -- 172.21.15.33:0/3803089567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 msgr2=0x55ad019d25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc292ffd640 1 --2- 172.21.15.33:0/3803089567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019d25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc292ffd640 1 -- 172.21.15.33:0/3803089567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad016a13c0 con 0x55ad019cbb40 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc293fff640 1 --2- 172.21.15.33:0/3803089567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019ca5d0 0x55ad019d4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc292ffd640 1 --2- 172.21.15.33:0/3803089567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 0x55ad019d2000 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc288000c00 tx=0x7fc2880043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74633d1e6cf805f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc272ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc272ffd640 1 -- 172.21.15.33:0/3803089567 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc288035040 con 0x55ad019cbb40 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc29c2da740 1 -- 172.21.15.33:0/3803089567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc28c011d80 con 0x55ad019cbb40 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc272ffd640 1 -- 172.21.15.33:0/3803089567 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc288003a00 con 0x55ad019cbb40 2024-09-16T00:24:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc272ffd640 1 -- 172.21.15.33:0/3803089567 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc28804e050 con 0x55ad019cbb40 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc29c2da740 10 client.26222 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.243+0000 7fc29c2da740 1 -- 172.21.15.33:0/3803089567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ad019e3b40 con 0x55ad019cbb40 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.244+0000 7fc272ffd640 1 -- 172.21.15.33:0/3803089567 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc28c011d80 con 0x55ad019cbb40 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.244+0000 7fc272ffd640 1 -- 172.21.15.33:0/3803089567 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fc28803c440 con 0x55ad019cbb40 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.244+0000 7fc29c2da740 10 client.26222 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.244+0000 7fc29c2da740 10 client.26222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.244+0000 7fc29c2da740 20 client.26222 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.244+0000 7fc29c2da740 1 --2- 172.21.15.33:0/3803089567 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad019e4fa0 0x55ad019e73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.244+0000 7fc29c2da740 4 client.26222 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.244+0000 7fc29c2da740 1 -- 172.21.15.33:0/3803089567 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad0193d860 con 0x55ad019e4fa0 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.244+0000 7fc293fff640 1 --2- 172.21.15.33:0/3803089567 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad019e4fa0 0x55ad019e73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.245+0000 7fc272ffd640 10 client.26222 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:17.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.245+0000 7fc293fff640 1 --2- 172.21.15.33:0/3803089567 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad019e4fa0 0x55ad019e73c0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ab98bb1ac222a0 server_cookie=6ac8dc4ea6cd8ff8 in_seq=0 out_seq=0 2024-09-16T00:24:17.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.245+0000 7fc272ffd640 1 -- 172.21.15.33:0/3803089567 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ad0193d860 con 0x55ad019e4fa0 2024-09-16T00:24:17.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.245+0000 7fc272ffd640 10 client.26222 handle_command_reply: tid=0 2024-09-16T00:24:17.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.245+0000 7fc29c2da740 1 client.26222 shutdown 2024-09-16T00:24:17.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.246+0000 7fc29c2da740 1 -- 172.21.15.33:0/3803089567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 msgr2=0x55ad019d2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:17.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.246+0000 7fc29c2da740 1 --2- 172.21.15.33:0/3803089567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 0x55ad019d2000 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc288000c00 tx=0x7fc2880043b0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.246+0000 7fc29c2da740 1 -- 172.21.15.33:0/3803089567 shutdown_connections 2024-09-16T00:24:17.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.246+0000 7fc29c2da740 1 --2- 172.21.15.33:0/3803089567 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad019e4fa0 0x55ad019e73c0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.246+0000 7fc29c2da740 1 --2- 172.21.15.33:0/3803089567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019d25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.246+0000 7fc29c2da740 1 --2- 172.21.15.33:0/3803089567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 0x55ad019d2000 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.246+0000 7fc272ffd640 0 client.26222 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:17.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.246+0000 7fc29c2da740 1 --2- 172.21.15.33:0/3803089567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019ca5d0 0x55ad019d4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.246+0000 7fc29c2da740 1 -- 172.21.15.33:0/3803089567 >> 172.21.15.33:0/3803089567 conn(0x55ad01993310 msgr2=0x55ad019cc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:17.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.246+0000 7fc29c2da740 1 -- 172.21.15.33:0/3803089567 shutdown_connections 2024-09-16T00:24:17.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.246+0000 7fc29c2da740 1 -- 172.21.15.33:0/3803089567 wait complete. 2024-09-16T00:24:17.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.246+0000 7fc29c2da740 20 client.26222 trim_cache size 0 max 16384 2024-09-16T00:24:17.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc29c2da740 1 Processor -- start 2024-09-16T00:24:17.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc29c2da740 1 -- start start 2024-09-16T00:24:17.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc29c2da740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019ca5d0 0x55ad019e1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc29c2da740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 0x55ad019ecc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc29c2da740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019d5370 0x55ad019ed1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc29c2da740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc2941b3bb0 con 0x55ad019ca5d0 2024-09-16T00:24:17.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc29c2da740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc2941b1480 con 0x55ad019d5370 2024-09-16T00:24:17.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc29c2da740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2941af2d0 con 0x55ad019cbb40 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc293fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019ca5d0 0x55ad019e1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc293fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019ca5d0 0x55ad019e1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56048/0 (socket says 172.21.15.33:56048) 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc293fff640 1 -- 172.21.15.33:0/2787317523 learned_addr learned my addr 172.21.15.33:0/2787317523 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc292ffd640 1 --2- 172.21.15.33:0/2787317523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 0x55ad019ecc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc293fff640 1 --2- 172.21.15.33:0/2787317523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019d5370 0x55ad019ed1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc293fff640 1 -- 172.21.15.33:0/2787317523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 msgr2=0x55ad019ecc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc293fff640 1 --2- 172.21.15.33:0/2787317523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 0x55ad019ecc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc293fff640 1 -- 172.21.15.33:0/2787317523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019d5370 msgr2=0x55ad019ed1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc293fff640 1 --2- 172.21.15.33:0/2787317523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019d5370 0x55ad019ed1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc293fff640 1 -- 172.21.15.33:0/2787317523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad019cd120 con 0x55ad019ca5d0 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.260+0000 7fc292ffd640 1 --2- 172.21.15.33:0/2787317523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 0x55ad019ecc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.261+0000 7fc293fff640 1 --2- 172.21.15.33:0/2787317523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019ca5d0 0x55ad019e1310 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc28c019960 tx=0x7fc28c0096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f8f862751570d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:17.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.261+0000 7fc2797fa640 1 -- 172.21.15.33:0/2787317523 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28c01f040 con 0x55ad019ca5d0 2024-09-16T00:24:17.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.261+0000 7fc2797fa640 1 -- 172.21.15.33:0/2787317523 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc28c003e50 con 0x55ad019ca5d0 2024-09-16T00:24:17.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.261+0000 7fc2797fa640 1 -- 172.21.15.33:0/2787317523 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28c0056f0 con 0x55ad019ca5d0 2024-09-16T00:24:17.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.261+0000 7fc29c2da740 1 -- 172.21.15.33:0/2787317523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019ca5d0 msgr2=0x55ad019e1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:17.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.261+0000 7fc29c2da740 1 --2- 172.21.15.33:0/2787317523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019ca5d0 0x55ad019e1310 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc28c019960 tx=0x7fc28c0096a0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.262+0000 7fc29c2da740 1 -- 172.21.15.33:0/2787317523 shutdown_connections 2024-09-16T00:24:17.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.262+0000 7fc29c2da740 1 --2- 172.21.15.33:0/2787317523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019d5370 0x55ad019ed1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.262+0000 7fc29c2da740 1 --2- 172.21.15.33:0/2787317523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019cbb40 0x55ad019ecc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.262+0000 7fc29c2da740 1 --2- 172.21.15.33:0/2787317523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019ca5d0 0x55ad019e1310 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.262+0000 7fc29c2da740 1 -- 172.21.15.33:0/2787317523 >> 172.21.15.33:0/2787317523 conn(0x55ad01993310 msgr2=0x55ad019d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:17.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.262+0000 7fc29c2da740 1 -- 172.21.15.33:0/2787317523 shutdown_connections 2024-09-16T00:24:17.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.262+0000 7fc29c2da740 1 -- 172.21.15.33:0/2787317523 wait complete. 2024-09-16T00:24:17.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.262+0000 7fc29c2da740 1 -- start start 2024-09-16T00:24:17.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc29c2da740 1 Processor -- start 2024-09-16T00:24:17.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc29c2da740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019ca5d0 0x55ad019dd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc29c2da740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 0x55ad019dd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc29c2da740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019da450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc29c2da740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc2941af940 con 0x55ad019d5370 2024-09-16T00:24:17.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc29c2da740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc2941b0db0 con 0x55ad019cbb40 2024-09-16T00:24:17.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc29c2da740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2941b0010 con 0x55ad019ca5d0 2024-09-16T00:24:17.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc29c2da740 10 client.0 _ll_register_callbacks cb 0x7fc288009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:17.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc293fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019da450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc292ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 0x55ad019dd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc293fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019da450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56064/0 (socket says 172.21.15.33:56064) 2024-09-16T00:24:17.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc293fff640 1 -- 172.21.15.33:0/1534631316 learned_addr learned my addr 172.21.15.33:0/1534631316 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc292ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 0x55ad019dd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:49762/0 (socket says 172.21.15.33:49762) 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc293fff640 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019ca5d0 0x55ad019dd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc292ffd640 1 -- 172.21.15.33:0/1534631316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019ca5d0 msgr2=0x55ad019dd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc292ffd640 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019ca5d0 0x55ad019dd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc292ffd640 1 -- 172.21.15.33:0/1534631316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 msgr2=0x55ad019da450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc292ffd640 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019da450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc292ffd640 1 -- 172.21.15.33:0/1534631316 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad019da990 con 0x55ad019cbb40 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.263+0000 7fc293fff640 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019da450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.264+0000 7fc292ffd640 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 0x55ad019dd9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc28803c890 tx=0x7fc28803c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1994db1491232d51 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.264+0000 7fc272ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.264+0000 7fc272ffd640 1 -- 172.21.15.33:0/1534631316 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc288035040 con 0x55ad019cbb40 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.264+0000 7fc29c2da740 1 -- 172.21.15.33:0/1534631316 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc28c003e50 con 0x55ad019cbb40 2024-09-16T00:24:17.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.264+0000 7fc272ffd640 1 -- 172.21.15.33:0/1534631316 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc288005230 con 0x55ad019cbb40 2024-09-16T00:24:17.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.264+0000 7fc272ffd640 1 -- 172.21.15.33:0/1534631316 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc288005d50 con 0x55ad019cbb40 2024-09-16T00:24:17.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.264+0000 7fc29c2da740 10 client.26228 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:17.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.264+0000 7fc29c2da740 1 -- 172.21.15.33:0/1534631316 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ad01a50380 con 0x55ad019cbb40 2024-09-16T00:24:17.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.264+0000 7fc272ffd640 1 -- 172.21.15.33:0/1534631316 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc28c003e50 con 0x55ad019cbb40 2024-09-16T00:24:17.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.265+0000 7fc272ffd640 1 -- 172.21.15.33:0/1534631316 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fc288003800 con 0x55ad019cbb40 2024-09-16T00:24:17.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.265+0000 7fc29c2da740 10 client.26228 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:17.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.265+0000 7fc29c2da740 10 client.26228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:24:17.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.265+0000 7fc29c2da740 20 client.26228 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:17.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.265+0000 7fc29c2da740 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad01a50aa0 0x55ad01a52ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:17.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.265+0000 7fc29c2da740 4 client.26228 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:24:17.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.265+0000 7fc29c2da740 1 -- 172.21.15.33:0/1534631316 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad01937d70 con 0x55ad01a50aa0 2024-09-16T00:24:17.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.265+0000 7fc293fff640 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad01a50aa0 0x55ad01a52ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:17.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.265+0000 7fc293fff640 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad01a50aa0 0x55ad01a52ec0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed1ceb2256b264e server_cookie=66e00d41638dc98e in_seq=0 out_seq=0 2024-09-16T00:24:17.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.265+0000 7fc272ffd640 10 client.26228 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:17.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc272ffd640 1 -- 172.21.15.33:0/1534631316 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad01937d70 con 0x55ad01a50aa0 2024-09-16T00:24:17.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc272ffd640 10 client.26228 handle_command_reply: tid=0 2024-09-16T00:24:17.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc29c2da740 1 client.26228 shutdown 2024-09-16T00:24:17.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc29c2da740 1 -- 172.21.15.33:0/1534631316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 msgr2=0x55ad019dd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:17.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc29c2da740 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 0x55ad019dd9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc28803c890 tx=0x7fc28803c670 comp rx=0 tx=0).stop 2024-09-16T00:24:17.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc29c2da740 1 -- 172.21.15.33:0/1534631316 shutdown_connections 2024-09-16T00:24:17.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc29c2da740 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad01a50aa0 0x55ad01a52ec0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc29c2da740 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad019d5370 0x55ad019da450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc29c2da740 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad019cbb40 0x55ad019dd9c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc272ffd640 0 client.26228 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:17.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc29c2da740 1 --2- 172.21.15.33:0/1534631316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad019ca5d0 0x55ad019dd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc29c2da740 1 -- 172.21.15.33:0/1534631316 >> 172.21.15.33:0/1534631316 conn(0x55ad01993310 msgr2=0x55ad019c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:17.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc29c2da740 1 -- 172.21.15.33:0/1534631316 shutdown_connections 2024-09-16T00:24:17.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc29c2da740 1 -- 172.21.15.33:0/1534631316 wait complete. 2024-09-16T00:24:17.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.266+0000 7fc29c2da740 20 client.26228 trim_cache size 0 max 16384 2024-09-16T00:24:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:24:17.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.267+0000 7fc299f01640 1 -- 172.21.15.33:0/259085493 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc27c0a3240 msgr2=0x7fc27c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:17.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.267+0000 7fc299f01640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc27c0a3240 0x7fc27c0a5700 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc2840cdba0 tx=0x7fc2840baf90 comp rx=0 tx=0).stop 2024-09-16T00:24:17.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.267+0000 7fc299f01640 1 -- 172.21.15.33:0/259085493 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc294100e20 msgr2=0x7fc2940fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:17.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.267+0000 7fc299f01640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc294100e20 0x7fc2940fcc80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc28400e9f0 tx=0x7fc28400edf0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.267+0000 7fc299f01640 1 -- 172.21.15.33:0/259085493 shutdown_connections 2024-09-16T00:24:17.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.267+0000 7fc299f01640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc27c0a3240 0x7fc27c0a5700 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.267+0000 7fc299f01640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc294102ef0 0x7fc2940fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.267+0000 7fc299f01640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc294102570 0x7fc2940fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.267+0000 7fc299f01640 1 --2- 172.21.15.33:0/259085493 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc294100e20 0x7fc2940fcc80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:17.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.267+0000 7fc299f01640 1 -- 172.21.15.33:0/259085493 >> 172.21.15.33:0/259085493 conn(0x7fc2940fc7d0 msgr2=0x7fc29406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:17.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.267+0000 7fc299f01640 1 -- 172.21.15.33:0/259085493 shutdown_connections 2024-09-16T00:24:17.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:17.267+0000 7fc299f01640 1 -- 172.21.15.33:0/259085493 wait complete. 2024-09-16T00:24:17.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:17 smithi033 ceph-mon[31243]: pgmap v564: 129 pgs: 129 active+clean; 181 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 323 op/s 2024-09-16T00:24:17.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 501541 10.15 MB/sec execute 205 sec latency 7.075 ms 2024-09-16T00:24:17.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:17 smithi083 ceph-mon[35242]: pgmap v564: 129 pgs: 129 active+clean; 181 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 323 op/s 2024-09-16T00:24:17.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:17 smithi060 ceph-mon[35185]: pgmap v564: 129 pgs: 129 active+clean; 181 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 323 op/s 2024-09-16T00:24:18.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 504799 10.19 MB/sec execute 206 sec latency 13.103 ms 2024-09-16T00:24:19.179 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:24:19.180 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:24:19.451 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:24:19.573 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:19 smithi033 ceph-mon[31243]: pgmap v565: 129 pgs: 129 active+clean; 181 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 3.4 MiB/s wr, 243 op/s 2024-09-16T00:24:19.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 507838 10.22 MB/sec execute 207 sec latency 36.460 ms 2024-09-16T00:24:19.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:19 smithi083 ceph-mon[35242]: pgmap v565: 129 pgs: 129 active+clean; 181 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 3.4 MiB/s wr, 243 op/s 2024-09-16T00:24:19.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:19 smithi060 ceph-mon[35185]: pgmap v565: 129 pgs: 129 active+clean; 181 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 3.4 MiB/s wr, 243 op/s 2024-09-16T00:24:20.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.021+0000 7f76dca34640 1 -- 172.21.15.33:0/3046975498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76d8106a70 msgr2=0x7f76d8106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.021+0000 7f76dca34640 1 --2- 172.21.15.33:0/3046975498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76d8106a70 0x7f76d8106e50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009c10 tx=0x7f76c802a6e0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.021+0000 7f76dca34640 1 -- 172.21.15.33:0/3046975498 shutdown_connections 2024-09-16T00:24:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.021+0000 7f76dca34640 1 --2- 172.21.15.33:0/3046975498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76d8101410 0x7f76d810d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.021+0000 7f76dca34640 1 --2- 172.21.15.33:0/3046975498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d8100a70 0x7f76d8100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.021+0000 7f76dca34640 1 --2- 172.21.15.33:0/3046975498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76d8106a70 0x7f76d8106e50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.021+0000 7f76dca34640 1 -- 172.21.15.33:0/3046975498 >> 172.21.15.33:0/3046975498 conn(0x7f76d80fc7d0 msgr2=0x7f76d80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.021+0000 7f76dca34640 1 -- 172.21.15.33:0/3046975498 shutdown_connections 2024-09-16T00:24:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.021+0000 7f76dca34640 1 -- 172.21.15.33:0/3046975498 wait complete. 2024-09-16T00:24:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.022+0000 7f76dca34640 1 Processor -- start 2024-09-16T00:24:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.022+0000 7f76dca34640 1 -- start start 2024-09-16T00:24:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.022+0000 7f76dca34640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d8100a70 0x7f76d81973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76dca34640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76d8101410 0x7f76d8197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76dca34640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76d8106a70 0x7f76d819bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76dca34640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8110e20 con 0x7f76d8106a70 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76dca34640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8110ca0 con 0x7f76d8100a70 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76dca34640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8110fa0 con 0x7f76d8101410 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76d659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d8100a70 0x7f76d81973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76cdd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76d8101410 0x7f76d8197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76d659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d8100a70 0x7f76d81973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:49772/0 (socket says 172.21.15.33:49772) 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76d6d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76d8106a70 0x7f76d819bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76cdd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76d8101410 0x7f76d8197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37926/0 (socket says 172.21.15.33:37926) 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76cdd9b640 1 -- 172.21.15.33:0/1759643713 learned_addr learned my addr 172.21.15.33:0/1759643713 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76cdd9b640 1 -- 172.21.15.33:0/1759643713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d8100a70 msgr2=0x7f76d81973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d8100a70 0x7f76d81973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76cdd9b640 1 -- 172.21.15.33:0/1759643713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76d8106a70 msgr2=0x7f76d819bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76d8106a70 0x7f76d819bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76cdd9b640 1 -- 172.21.15.33:0/1759643713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d819c390 con 0x7f76d8101410 2024-09-16T00:24:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76d6d9d640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76d8106a70 0x7f76d819bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.023+0000 7f76d659c640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d8100a70 0x7f76d81973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.024+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76d8101410 0x7f76d8197920 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f76c000e9f0 tx=0x7f76c000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.024+0000 7f76cffff640 1 -- 172.21.15.33:0/1759643713 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c000cd80 con 0x7f76d8101410 2024-09-16T00:24:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.024+0000 7f76cffff640 1 -- 172.21.15.33:0/1759643713 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c0005500 con 0x7f76d8101410 2024-09-16T00:24:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.024+0000 7f76dca34640 1 -- 172.21.15.33:0/1759643713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d819c680 con 0x7f76d8101410 2024-09-16T00:24:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.024+0000 7f76cffff640 1 -- 172.21.15.33:0/1759643713 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c0010690 con 0x7f76d8101410 2024-09-16T00:24:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.024+0000 7f76dca34640 1 -- 172.21.15.33:0/1759643713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d806a790 con 0x7f76d8101410 2024-09-16T00:24:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.027+0000 7f76cffff640 1 -- 172.21.15.33:0/1759643713 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f76c0010830 con 0x7f76d8101410 2024-09-16T00:24:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.027+0000 7f76dee0d740 1 Processor -- start 2024-09-16T00:24:20.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76dee0d740 1 -- start start 2024-09-16T00:24:20.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76dee0d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d7b60 0x55d4768d7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76dee0d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d8480 0x55d4768e2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76dee0d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 0x55d4768e5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76dee0d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d47683d280 con 0x55d4768e2d40 2024-09-16T00:24:20.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76dee0d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d4767e4b10 con 0x55d4768d8480 2024-09-16T00:24:20.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76dee0d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d476839240 con 0x55d4768d7b60 2024-09-16T00:24:20.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76cffff640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f76c40a32c0 0x7f76c40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76cdd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 0x55d4768e5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76d6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d7b60 0x55d4768d7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76d6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d7b60 0x55d4768d7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37940/0 (socket says 172.21.15.33:37940) 2024-09-16T00:24:20.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76d6d9d640 1 -- 172.21.15.33:0/1220539131 learned_addr learned my addr 172.21.15.33:0/1220539131 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:20.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.028+0000 7f76d6d9d640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f76c40a32c0 0x7f76c40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76d659c640 1 --2- 172.21.15.33:0/1220539131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d8480 0x55d4768e2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76cffff640 1 -- 172.21.15.33:0/1759643713 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f76c00d0130 con 0x7f76d8101410 2024-09-16T00:24:20.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76d6d9d640 1 -- 172.21.15.33:0/1220539131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d8480 msgr2=0x55d4768e2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:20.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76d6d9d640 1 --2- 172.21.15.33:0/1220539131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d8480 0x55d4768e2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76d6d9d640 1 -- 172.21.15.33:0/1220539131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 msgr2=0x55d4768e5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:20.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76d6d9d640 1 --2- 172.21.15.33:0/1220539131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 0x55d4768e5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76d6d9d640 1 -- 172.21.15.33:0/1220539131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4767f11f0 con 0x55d4768d7b60 2024-09-16T00:24:20.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/1220539131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 0x55d4768e5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:20.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76d6d9d640 1 --2- 172.21.15.33:0/1220539131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d7b60 0x55d4768d7f40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f76d0013e50 tx=0x7f76d00117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c066f6564de88d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:20.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76d6d9d640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f76c40a32c0 0x7f76c40a5780 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f76d0011bf0 tx=0x7f76d0018000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:20.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76ccd99640 1 -- 172.21.15.33:0/1220539131 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0013960 con 0x55d4768d7b60 2024-09-16T00:24:20.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76ccd99640 1 -- 172.21.15.33:0/1220539131 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d000ebe0 con 0x55d4768d7b60 2024-09-16T00:24:20.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.029+0000 7f76ccd99640 1 -- 172.21.15.33:0/1220539131 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d00035f0 con 0x55d4768d7b60 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.030+0000 7f76dee0d740 1 -- 172.21.15.33:0/1220539131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d7b60 msgr2=0x55d4768d7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.030+0000 7f76dee0d740 1 --2- 172.21.15.33:0/1220539131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d7b60 0x55d4768d7f40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f76d0013e50 tx=0x7f76d00117b0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.030+0000 7f76dee0d740 1 -- 172.21.15.33:0/1220539131 shutdown_connections 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.030+0000 7f76dee0d740 1 --2- 172.21.15.33:0/1220539131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 0x55d4768e5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.030+0000 7f76dee0d740 1 --2- 172.21.15.33:0/1220539131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d8480 0x55d4768e2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.030+0000 7f76dee0d740 1 --2- 172.21.15.33:0/1220539131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d7b60 0x55d4768d7f40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.030+0000 7f76dee0d740 1 -- 172.21.15.33:0/1220539131 >> 172.21.15.33:0/1220539131 conn(0x55d4768d40f0 msgr2=0x55d4768d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.031+0000 7f76dee0d740 1 -- 172.21.15.33:0/1220539131 shutdown_connections 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.031+0000 7f76dee0d740 1 -- 172.21.15.33:0/1220539131 wait complete. 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.031+0000 7f76dee0d740 1 -- start start 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.031+0000 7f76dee0d740 1 Processor -- start 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.031+0000 7f76dee0d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d7b60 0x55d4768dfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.031+0000 7f76dee0d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 0x55d4768e01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.031+0000 7f76dee0d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 0x55d4768e06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.031+0000 7f76dee0d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d4768e5910 con 0x55d4768e2d40 2024-09-16T00:24:20.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.031+0000 7f76dee0d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d4767f7f10 con 0x55d4768d7b60 2024-09-16T00:24:20.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.031+0000 7f76dee0d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d4768e5a90 con 0x55d4768d8480 2024-09-16T00:24:20.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.031+0000 7f76dee0d740 10 client.0 _ll_register_callbacks cb 0x55d4767c0a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:20.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76cdd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 0x55d4768e01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76d6d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 0x55d4768e06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76d6d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 0x55d4768e06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56120/0 (socket says 172.21.15.33:56120) 2024-09-16T00:24:20.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76cdd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 0x55d4768e01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37950/0 (socket says 172.21.15.33:37950) 2024-09-16T00:24:20.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76d6d9d640 1 -- 172.21.15.33:0/3907755008 learned_addr learned my addr 172.21.15.33:0/3907755008 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:20.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76d659c640 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d7b60 0x55d4768dfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76cdd9b640 1 -- 172.21.15.33:0/3907755008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d7b60 msgr2=0x55d4768dfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:20.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d7b60 0x55d4768dfc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76cdd9b640 1 -- 172.21.15.33:0/3907755008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 msgr2=0x55d4768e06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:20.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 0x55d4768e06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76cdd9b640 1 -- 172.21.15.33:0/3907755008 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4765ae6a0 con 0x55d4768d8480 2024-09-16T00:24:20.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76d6d9d640 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 0x55d4768e06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:20.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 0x55d4768e01a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f76c00cd800 tx=0x7f76c0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e85d739f7d31cb17 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:20.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.032+0000 7f76bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:24:20.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76bdffb640 1 -- 172.21.15.33:0/3907755008 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c009a1b0 con 0x55d4768d8480 2024-09-16T00:24:20.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76dee0d740 1 -- 172.21.15.33:0/3907755008 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76d000ebe0 con 0x55d4768d8480 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76bdffb640 1 -- 172.21.15.33:0/3907755008 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f76c009c740 con 0x55d4768d8480 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76bdffb640 1 -- 172.21.15.33:0/3907755008 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f76c0099920 con 0x55d4768d8480 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76dee0d740 10 client.26164 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76dee0d740 1 -- 172.21.15.33:0/3907755008 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d4768f1600 con 0x55d4768d8480 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76bdffb640 1 -- 172.21.15.33:0/3907755008 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d000ebe0 con 0x55d4768d8480 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76bdffb640 1 -- 172.21.15.33:0/3907755008 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f76c00921b0 con 0x55d4768d8480 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76dee0d740 10 client.26164 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76dee0d740 10 client.26164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76dee0d740 20 client.26164 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d4768f2ba0 0x55d4768f4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76dee0d740 4 client.26164 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.033+0000 7f76dee0d740 1 -- 172.21.15.33:0/3907755008 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d47684a7f0 con 0x55d4768f2ba0 2024-09-16T00:24:20.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.034+0000 7f76d659c640 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d4768f2ba0 0x55d4768f4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.034+0000 7f76bdffb640 10 client.26164 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:20.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.034+0000 7f76d659c640 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d4768f2ba0 0x55d4768f4f90 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48a844c29bd4b8d9 server_cookie=f08f1a1a43bce40d in_seq=0 out_seq=0 2024-09-16T00:24:20.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76bdffb640 1 -- 172.21.15.33:0/3907755008 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d47684a7f0 con 0x55d4768f2ba0 2024-09-16T00:24:20.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76bdffb640 10 client.26164 handle_command_reply: tid=0 2024-09-16T00:24:20.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76dee0d740 1 client.26164 shutdown 2024-09-16T00:24:20.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76dee0d740 1 -- 172.21.15.33:0/3907755008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 msgr2=0x55d4768e01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:20.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 0x55d4768e01a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f76c00cd800 tx=0x7f76c0017000 comp rx=0 tx=0).stop 2024-09-16T00:24:20.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76dee0d740 1 -- 172.21.15.33:0/3907755008 shutdown_connections 2024-09-16T00:24:20.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d4768f2ba0 0x55d4768f4f90 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768e2d40 0x55d4768e06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 0x55d4768e01a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3907755008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d7b60 0x55d4768dfc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76dee0d740 1 -- 172.21.15.33:0/3907755008 >> 172.21.15.33:0/3907755008 conn(0x55d4768d40f0 msgr2=0x55d4768d9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:20.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76bdffb640 0 client.26164 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:20.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76dee0d740 1 -- 172.21.15.33:0/3907755008 shutdown_connections 2024-09-16T00:24:20.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76dee0d740 1 -- 172.21.15.33:0/3907755008 wait complete. 2024-09-16T00:24:20.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.035+0000 7f76dee0d740 20 client.26164 trim_cache size 0 max 16384 2024-09-16T00:24:20.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76dee0d740 1 Processor -- start 2024-09-16T00:24:20.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76dee0d740 1 -- start start 2024-09-16T00:24:20.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76dee0d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d7b60 0x55d4768d9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76dee0d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 0x55d4768fa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76dee0d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768e2d40 0x55d4768fac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76dee0d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81b3ba0 con 0x55d4768d7b60 2024-09-16T00:24:20.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76dee0d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81b1470 con 0x55d4768e2d40 2024-09-16T00:24:20.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76dee0d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81af2c0 con 0x55d4768d8480 2024-09-16T00:24:20.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76d659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d7b60 0x55d4768d9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76cdd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 0x55d4768fa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76cdd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 0x55d4768fa6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37954/0 (socket says 172.21.15.33:37954) 2024-09-16T00:24:20.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76d659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d7b60 0x55d4768d9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56122/0 (socket says 172.21.15.33:56122) 2024-09-16T00:24:20.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76cdd9b640 1 -- 172.21.15.33:0/3366043399 learned_addr learned my addr 172.21.15.33:0/3366043399 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:20.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76d6d9d640 1 --2- 172.21.15.33:0/3366043399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768e2d40 0x55d4768fac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76cdd9b640 1 -- 172.21.15.33:0/3366043399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768e2d40 msgr2=0x55d4768fac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:20.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/3366043399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768e2d40 0x55d4768fac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76cdd9b640 1 -- 172.21.15.33:0/3366043399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d7b60 msgr2=0x55d4768d9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:20.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/3366043399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d7b60 0x55d4768d9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76cdd9b640 1 -- 172.21.15.33:0/3366043399 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4768de1f0 con 0x55d4768d8480 2024-09-16T00:24:20.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/3366043399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 0x55d4768fa6d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f76c0098ca0 tx=0x7f76c009c1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54211dbbc3cb293d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:20.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76bffff640 1 -- 172.21.15.33:0/3366043399 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c0099b00 con 0x55d4768d8480 2024-09-16T00:24:20.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76bffff640 1 -- 172.21.15.33:0/3366043399 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c009a670 con 0x55d4768d8480 2024-09-16T00:24:20.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.055+0000 7f76bffff640 1 -- 172.21.15.33:0/3366043399 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c009c2d0 con 0x55d4768d8480 2024-09-16T00:24:20.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.056+0000 7f76dee0d740 1 -- 172.21.15.33:0/3366043399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 msgr2=0x55d4768fa6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:20.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.056+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3366043399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 0x55d4768fa6d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f76c0098ca0 tx=0x7f76c009c1d0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.056+0000 7f76dee0d740 1 -- 172.21.15.33:0/3366043399 shutdown_connections 2024-09-16T00:24:20.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.056+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3366043399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768e2d40 0x55d4768fac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.056+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3366043399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768d8480 0x55d4768fa6d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.056+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3366043399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d7b60 0x55d4768d9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.056+0000 7f76dee0d740 1 -- 172.21.15.33:0/3366043399 >> 172.21.15.33:0/3366043399 conn(0x55d4768d40f0 msgr2=0x55d4768d9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:20.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.056+0000 7f76dee0d740 1 -- 172.21.15.33:0/3366043399 shutdown_connections 2024-09-16T00:24:20.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.056+0000 7f76dee0d740 1 -- 172.21.15.33:0/3366043399 wait complete. 2024-09-16T00:24:20.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.056+0000 7f76dee0d740 1 -- start start 2024-09-16T00:24:20.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76dee0d740 1 Processor -- start 2024-09-16T00:24:20.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76dee0d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d7b60 0x55d4768dceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76dee0d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d8480 0x55d4768dd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76dee0d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768e2d40 0x55d4768dd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76dee0d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81af930 con 0x55d4768d8480 2024-09-16T00:24:20.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76dee0d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81b0da0 con 0x55d4768d7b60 2024-09-16T00:24:20.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76dee0d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81b0000 con 0x55d4768e2d40 2024-09-16T00:24:20.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76dee0d740 10 client.0 _ll_register_callbacks cb 0x7f76c000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:20.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76cdd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d8480 0x55d4768dd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76cdd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d8480 0x55d4768dd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56134/0 (socket says 172.21.15.33:56134) 2024-09-16T00:24:20.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76d6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768e2d40 0x55d4768dd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76cdd9b640 1 -- 172.21.15.33:0/3248389308 learned_addr learned my addr 172.21.15.33:0/3248389308 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:20.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76d659c640 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d7b60 0x55d4768dceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.057+0000 7f76cdd9b640 1 -- 172.21.15.33:0/3248389308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768e2d40 msgr2=0x55d4768dd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:20.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768e2d40 0x55d4768dd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76cdd9b640 1 -- 172.21.15.33:0/3248389308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d7b60 msgr2=0x55d4768dceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:20.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d7b60 0x55d4768dceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76cdd9b640 1 -- 172.21.15.33:0/3248389308 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d47695c8a0 con 0x55d4768d8480 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76d659c640 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d7b60 0x55d4768dceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76d6d9d640 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768e2d40 0x55d4768dd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76cdd9b640 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d8480 0x55d4768dd3f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f76c009bfd0 tx=0x7f76c0099b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf9fa4a17599815 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76bdffb640 1 -- 172.21.15.33:0/3248389308 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c009c2d0 con 0x55d4768d8480 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76bdffb640 1 -- 172.21.15.33:0/3248389308 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f76c00bc0e0 con 0x55d4768d8480 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76dee0d740 1 -- 172.21.15.33:0/3248389308 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76c009a670 con 0x55d4768d8480 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76bdffb640 1 -- 172.21.15.33:0/3248389308 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f76c00a0680 con 0x55d4768d8480 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76dee0d740 10 client.16728 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76dee0d740 1 -- 172.21.15.33:0/3248389308 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d47695dbc0 con 0x55d4768d8480 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.058+0000 7f76bdffb640 1 -- 172.21.15.33:0/3248389308 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c009a670 con 0x55d4768d8480 2024-09-16T00:24:20.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.059+0000 7f76bdffb640 1 -- 172.21.15.33:0/3248389308 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f76c0010c40 con 0x55d4768d8480 2024-09-16T00:24:20.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.059+0000 7f76dee0d740 10 client.16728 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:20.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.059+0000 7f76dee0d740 10 client.16728 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:24:20.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.059+0000 7f76dee0d740 20 client.16728 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:20.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.059+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d47695e040 0x55d476960430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:20.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.059+0000 7f76dee0d740 4 client.16728 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:24:20.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.059+0000 7f76dee0d740 1 -- 172.21.15.33:0/3248389308 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d476844d00 con 0x55d47695e040 2024-09-16T00:24:20.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.059+0000 7f76d659c640 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d47695e040 0x55d476960430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:20.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.060+0000 7f76bdffb640 10 client.16728 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:20.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.060+0000 7f76d659c640 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d47695e040 0x55d476960430 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adfdbe8423d95dd1 server_cookie=717f2a2561551c01 in_seq=0 out_seq=0 2024-09-16T00:24:20.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76bdffb640 1 -- 172.21.15.33:0/3248389308 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d476844d00 con 0x55d47695e040 2024-09-16T00:24:20.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76bdffb640 10 client.16728 handle_command_reply: tid=0 2024-09-16T00:24:20.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76dee0d740 1 client.16728 shutdown 2024-09-16T00:24:20.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76dee0d740 1 -- 172.21.15.33:0/3248389308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d8480 msgr2=0x55d4768dd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:20.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d8480 0x55d4768dd3f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f76c009bfd0 tx=0x7f76c0099b80 comp rx=0 tx=0).stop 2024-09-16T00:24:20.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76dee0d740 1 -- 172.21.15.33:0/3248389308 shutdown_connections 2024-09-16T00:24:20.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d47695e040 0x55d476960430 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d4768e2d40 0x55d4768dd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d4768d8480 0x55d4768dd3f0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76dee0d740 1 --2- 172.21.15.33:0/3248389308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d4768d7b60 0x55d4768dceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76dee0d740 1 -- 172.21.15.33:0/3248389308 >> 172.21.15.33:0/3248389308 conn(0x55d4768d40f0 msgr2=0x55d4768e0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:20.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76bdffb640 0 client.16728 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:20.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76dee0d740 1 -- 172.21.15.33:0/3248389308 shutdown_connections 2024-09-16T00:24:20.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76dee0d740 1 -- 172.21.15.33:0/3248389308 wait complete. 2024-09-16T00:24:20.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.061+0000 7f76dee0d740 20 client.16728 trim_cache size 0 max 16384 2024-09-16T00:24:20.068 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:24:20.068 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:24:20.068 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "a691b521-414e-43a7-83bb-2bbde6f0c950", 2024-09-16T00:24:20.068 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:24:20.068 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:24:20.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.062+0000 7f76dca34640 1 -- 172.21.15.33:0/1759643713 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f76c40a32c0 msgr2=0x7f76c40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:20.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.062+0000 7f76dca34640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f76c40a32c0 0x7f76c40a5780 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f76d0011bf0 tx=0x7f76d0018000 comp rx=0 tx=0).stop 2024-09-16T00:24:20.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.062+0000 7f76dca34640 1 -- 172.21.15.33:0/1759643713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76d8101410 msgr2=0x7f76d8197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:20.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.062+0000 7f76dca34640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76d8101410 0x7f76d8197920 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f76c000e9f0 tx=0x7f76c000edf0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.062+0000 7f76dca34640 1 -- 172.21.15.33:0/1759643713 shutdown_connections 2024-09-16T00:24:20.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.062+0000 7f76dca34640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f76c40a32c0 0x7f76c40a5780 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.062+0000 7f76dca34640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76d8106a70 0x7f76d819bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.062+0000 7f76dca34640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76d8101410 0x7f76d8197920 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.062+0000 7f76dca34640 1 --2- 172.21.15.33:0/1759643713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d8100a70 0x7f76d81973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:20.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.062+0000 7f76dca34640 1 -- 172.21.15.33:0/1759643713 >> 172.21.15.33:0/1759643713 conn(0x7f76d80fc7d0 msgr2=0x7f76d80fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:20.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.062+0000 7f76dca34640 1 -- 172.21.15.33:0/1759643713 shutdown_connections 2024-09-16T00:24:20.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:20.062+0000 7f76dca34640 1 -- 172.21.15.33:0/1759643713 wait complete. 2024-09-16T00:24:20.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 510117 10.22 MB/sec execute 208 sec latency 92.451 ms 2024-09-16T00:24:20.832 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:24:21.170 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:24:21.532 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:21 smithi060 ceph-mon[35185]: pgmap v566: 129 pgs: 129 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 3.7 MiB/s wr, 262 op/s 2024-09-16T00:24:21.532 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:21 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:24:21.533 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:21 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:24:21.533 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:21 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:24:21.533 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:21 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:24:21.533 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:21 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:24:21.533 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:21 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:24:21.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:21 smithi033 ceph-mon[31243]: pgmap v566: 129 pgs: 129 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 3.7 MiB/s wr, 262 op/s 2024-09-16T00:24:21.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:21 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:24:21.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:21 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:24:21.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:21 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:24:21.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:21 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:24:21.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:21 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:24:21.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:21 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:24:21.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 512325 10.23 MB/sec execute 209 sec latency 42.123 ms 2024-09-16T00:24:21.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:21 smithi083 ceph-mon[35242]: pgmap v566: 129 pgs: 129 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 3.7 MiB/s wr, 262 op/s 2024-09-16T00:24:21.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:21 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:24:21.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:21 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:24:21.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:21 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:24:21.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:21 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:24:21.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:21 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:24:21.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:21 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:24:21.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.648+0000 7f79494af640 1 -- 172.21.15.33:0/763450925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7944103780 msgr2=0x7f7944107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:21.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.648+0000 7f79494af640 1 --2- 172.21.15.33:0/763450925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7944103780 0x7f7944107bc0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f792c009930 tx=0x7f792c02a420 comp rx=0 tx=0).stop 2024-09-16T00:24:21.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.648+0000 7f79494af640 1 -- 172.21.15.33:0/763450925 shutdown_connections 2024-09-16T00:24:21.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.648+0000 7f79494af640 1 --2- 172.21.15.33:0/763450925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7944108300 0x7f794410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.648+0000 7f79494af640 1 --2- 172.21.15.33:0/763450925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7944103780 0x7f7944107bc0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.648+0000 7f79494af640 1 --2- 172.21.15.33:0/763450925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7944102dd0 0x7f79441031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.648+0000 7f79494af640 1 -- 172.21.15.33:0/763450925 >> 172.21.15.33:0/763450925 conn(0x7f79440fc7d0 msgr2=0x7f79440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.649+0000 7f79494af640 1 -- 172.21.15.33:0/763450925 shutdown_connections 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.649+0000 7f79494af640 1 -- 172.21.15.33:0/763450925 wait complete. 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.649+0000 7f79494af640 1 Processor -- start 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.649+0000 7f79494af640 1 -- start start 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.649+0000 7f79494af640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7944102dd0 0x7f79441985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.649+0000 7f79494af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7944103780 0x7f7944198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.649+0000 7f79494af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7944108300 0x7f794419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f79494af640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7944112050 con 0x7f7944102dd0 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f79494af640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7944111ed0 con 0x7f7944108300 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f79494af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f79441121d0 con 0x7f7944103780 2024-09-16T00:24:21.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f7942ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7944102dd0 0x7f79441985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f7942ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7944102dd0 0x7f79441985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45532/0 (socket says 172.21.15.33:45532) 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f7942ffd640 1 -- 172.21.15.33:0/878569558 learned_addr learned my addr 172.21.15.33:0/878569558 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f793bfff640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7944103780 0x7f7944198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f79437fe640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7944108300 0x7f794419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f7942ffd640 1 -- 172.21.15.33:0/878569558 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7944103780 msgr2=0x7f7944198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f7942ffd640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7944103780 0x7f7944198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f7942ffd640 1 -- 172.21.15.33:0/878569558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7944108300 msgr2=0x7f794419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f7942ffd640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7944108300 0x7f794419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f7942ffd640 1 -- 172.21.15.33:0/878569558 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794419d580 con 0x7f7944102dd0 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f7942ffd640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7944102dd0 0x7f79441985d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f793400e9f0 tx=0x7f793400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f7940ff9640 1 -- 172.21.15.33:0/878569558 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793400cd80 con 0x7f7944102dd0 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f79494af640 1 -- 172.21.15.33:0/878569558 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794419d870 con 0x7f7944102dd0 2024-09-16T00:24:21.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f7940ff9640 1 -- 172.21.15.33:0/878569558 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7934010910 con 0x7f7944102dd0 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f7940ff9640 1 -- 172.21.15.33:0/878569558 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7934018990 con 0x7f7944102dd0 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.650+0000 7f79494af640 1 -- 172.21.15.33:0/878569558 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79441a5150 con 0x7f7944102dd0 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.653+0000 7f794b888740 1 Processor -- start 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.653+0000 7f7940ff9640 1 -- 172.21.15.33:0/878569558 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7934010ac0 con 0x7f7944102dd0 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.653+0000 7f794b888740 1 -- start start 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.653+0000 7f794b888740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d4510880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.653+0000 7f794b888740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d451b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.653+0000 7f794b888740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d451da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.653+0000 7f794b888740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d5d44762e0 con 0x55d5d451b680 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.653+0000 7f794b888740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d5d441dce0 con 0x55d5d45104a0 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.653+0000 7f794b888740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d5d44722a0 con 0x55d5d4510dc0 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.653+0000 7f79437fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d4510880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f79437fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d4510880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51696/0 (socket says 172.21.15.33:51696) 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f79437fe640 1 -- 172.21.15.33:0/3517788828 learned_addr learned my addr 172.21.15.33:0/3517788828 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f7942ffd640 1 --2- 172.21.15.33:0/3517788828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d451b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f793bfff640 1 --2- 172.21.15.33:0/3517788828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d451da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f79437fe640 1 -- 172.21.15.33:0/3517788828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 msgr2=0x55d5d451b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f79437fe640 1 --2- 172.21.15.33:0/3517788828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d451b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f79437fe640 1 -- 172.21.15.33:0/3517788828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 msgr2=0x55d5d451da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f79437fe640 1 --2- 172.21.15.33:0/3517788828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d451da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f79437fe640 1 -- 172.21.15.33:0/3517788828 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5d4402440 con 0x55d5d45104a0 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f7940ff9640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f79300a3240 0x7f79300a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f7942ffd640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f79300a3240 0x7f79300a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f79437fe640 1 --2- 172.21.15.33:0/3517788828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d4510880 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f793c009f50 tx=0x7f793c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ff65256c95b525 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.654+0000 7f7940ff9640 1 -- 172.21.15.33:0/878569558 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f79340d00c0 con 0x7f7944102dd0 2024-09-16T00:24:21.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.655+0000 7f79397fa640 1 -- 172.21.15.33:0/3517788828 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c004d20 con 0x55d5d45104a0 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.655+0000 7f79397fa640 1 -- 172.21.15.33:0/3517788828 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f793c004050 con 0x55d5d45104a0 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.655+0000 7f79397fa640 1 -- 172.21.15.33:0/3517788828 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c005280 con 0x55d5d45104a0 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.655+0000 7f794b888740 1 -- 172.21.15.33:0/3517788828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 msgr2=0x55d5d4510880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.655+0000 7f7942ffd640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f79300a3240 0x7f79300a5700 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f79340cdab0 tx=0x7f79340d3ba0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.655+0000 7f794b888740 1 --2- 172.21.15.33:0/3517788828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d4510880 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f793c009f50 tx=0x7f793c0118d0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.655+0000 7f794b888740 1 -- 172.21.15.33:0/3517788828 shutdown_connections 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.655+0000 7f794b888740 1 --2- 172.21.15.33:0/3517788828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d451da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.655+0000 7f794b888740 1 --2- 172.21.15.33:0/3517788828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d451b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.655+0000 7f794b888740 1 --2- 172.21.15.33:0/3517788828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d4510880 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.655+0000 7f794b888740 1 -- 172.21.15.33:0/3517788828 >> 172.21.15.33:0/3517788828 conn(0x55d5d450ca30 msgr2=0x55d5d450ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f794b888740 1 -- 172.21.15.33:0/3517788828 shutdown_connections 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f794b888740 1 -- 172.21.15.33:0/3517788828 wait complete. 2024-09-16T00:24:21.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f794b888740 1 -- start start 2024-09-16T00:24:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f794b888740 1 Processor -- start 2024-09-16T00:24:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f794b888740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d451afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f794b888740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d4518290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f794b888740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d45187f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f794b888740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d5d451e250 con 0x55d5d451b680 2024-09-16T00:24:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f794b888740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d5d4431050 con 0x55d5d45104a0 2024-09-16T00:24:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f794b888740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d5d451e3d0 con 0x55d5d4510dc0 2024-09-16T00:24:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f794b888740 10 client.0 _ll_register_callbacks cb 0x55d5d4471160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f79437fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d45187f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f79437fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d45187f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45542/0 (socket says 172.21.15.33:45542) 2024-09-16T00:24:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f79437fe640 1 -- 172.21.15.33:0/2612529422 learned_addr learned my addr 172.21.15.33:0/2612529422 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f79437fe640 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d451afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.656+0000 7f793bfff640 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d4518290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f79437fe640 1 -- 172.21.15.33:0/2612529422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 msgr2=0x55d5d4518290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:21.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f79437fe640 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d4518290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f79437fe640 1 -- 172.21.15.33:0/2612529422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 msgr2=0x55d5d451afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:21.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f79437fe640 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d451afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f79437fe640 1 -- 172.21.15.33:0/2612529422 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5d41e73c0 con 0x55d5d451b680 2024-09-16T00:24:21.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f793bfff640 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d4518290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:24:21.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f79437fe640 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d45187f0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f793c009d60 tx=0x7f793c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08e6ff20c7887f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:21.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f792a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:24:21.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f792a7fc640 1 -- 172.21.15.33:0/2612529422 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c02b020 con 0x55d5d451b680 2024-09-16T00:24:21.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f792a7fc640 1 -- 172.21.15.33:0/2612529422 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f793c027640 con 0x55d5d451b680 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f794b888740 1 -- 172.21.15.33:0/2612529422 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f793c004050 con 0x55d5d451b680 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f792a7fc640 1 -- 172.21.15.33:0/2612529422 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f793c036050 con 0x55d5d451b680 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f794b888740 10 client.16740 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f794b888740 1 -- 172.21.15.33:0/2612529422 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d5d4529c20 con 0x55d5d451b680 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.657+0000 7f792a7fc640 1 -- 172.21.15.33:0/2612529422 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f793c004050 con 0x55d5d451b680 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.658+0000 7f792a7fc640 1 -- 172.21.15.33:0/2612529422 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f793c00f8a0 con 0x55d5d451b680 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.658+0000 7f794b888740 10 client.16740 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.658+0000 7f794b888740 10 client.16740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.658+0000 7f794b888740 20 client.16740 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.658+0000 7f794b888740 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d5d452b080 0x55d5d452d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.658+0000 7f794b888740 4 client.16740 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.658+0000 7f794b888740 1 -- 172.21.15.33:0/2612529422 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5d4483850 con 0x55d5d452b080 2024-09-16T00:24:21.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.658+0000 7f793bfff640 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d5d452b080 0x55d5d452d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.658+0000 7f793bfff640 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d5d452b080 0x55d5d452d470 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49082c0e4ae9378e server_cookie=f7d5a41376bb387d in_seq=0 out_seq=0 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.658+0000 7f792a7fc640 10 client.16740 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.659+0000 7f792a7fc640 1 -- 172.21.15.33:0/2612529422 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d5d4483850 con 0x55d5d452b080 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.659+0000 7f792a7fc640 10 client.16740 handle_command_reply: tid=0 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.659+0000 7f794b888740 1 client.16740 shutdown 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.659+0000 7f794b888740 1 -- 172.21.15.33:0/2612529422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 msgr2=0x55d5d45187f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.659+0000 7f794b888740 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d45187f0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f793c009d60 tx=0x7f793c005bd0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.660+0000 7f794b888740 1 -- 172.21.15.33:0/2612529422 shutdown_connections 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.660+0000 7f794b888740 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d5d452b080 0x55d5d452d470 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.660+0000 7f794b888740 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d45187f0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.660+0000 7f794b888740 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d4518290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.660+0000 7f792a7fc640 0 client.16740 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.660+0000 7f794b888740 1 --2- 172.21.15.33:0/2612529422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d451afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.660+0000 7f794b888740 1 -- 172.21.15.33:0/2612529422 >> 172.21.15.33:0/2612529422 conn(0x55d5d450ca30 msgr2=0x55d5d4512710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.660+0000 7f794b888740 1 -- 172.21.15.33:0/2612529422 shutdown_connections 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.660+0000 7f794b888740 1 -- 172.21.15.33:0/2612529422 wait complete. 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.660+0000 7f794b888740 20 client.16740 trim_cache size 0 max 16384 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.678+0000 7f794b888740 1 Processor -- start 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.678+0000 7f794b888740 1 -- start start 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f794b888740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d45104a0 0x55d5d45274b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f794b888740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d4510dc0 0x55d5d450ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f794b888740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d451b680 0x55d5d4517270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f794b888740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f79441b1b60 con 0x55d5d45104a0 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f794b888740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f79441af430 con 0x55d5d4510dc0 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f794b888740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f79441ad280 con 0x55d5d451b680 2024-09-16T00:24:21.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f79437fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d45104a0 0x55d5d45274b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f79437fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d45104a0 0x55d5d45274b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45554/0 (socket says 172.21.15.33:45554) 2024-09-16T00:24:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f79437fe640 1 -- 172.21.15.33:0/2979629198 learned_addr learned my addr 172.21.15.33:0/2979629198 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f793bfff640 1 --2- 172.21.15.33:0/2979629198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d4510dc0 0x55d5d450ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f79437fe640 1 --2- 172.21.15.33:0/2979629198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d451b680 0x55d5d4517270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f79437fe640 1 -- 172.21.15.33:0/2979629198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d451b680 msgr2=0x55d5d4517270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f79437fe640 1 --2- 172.21.15.33:0/2979629198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d451b680 0x55d5d4517270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f79437fe640 1 -- 172.21.15.33:0/2979629198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d4510dc0 msgr2=0x55d5d450ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f79437fe640 1 --2- 172.21.15.33:0/2979629198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d4510dc0 0x55d5d450ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.679+0000 7f79437fe640 1 -- 172.21.15.33:0/2979629198 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5d45102d0 con 0x55d5d45104a0 2024-09-16T00:24:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.680+0000 7f79437fe640 1 --2- 172.21.15.33:0/2979629198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d45104a0 0x55d5d45274b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f793c002230 tx=0x7f793c009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8934742d809eb2b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.680+0000 7f7938ff9640 1 -- 172.21.15.33:0/2979629198 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c02b070 con 0x55d5d45104a0 2024-09-16T00:24:21.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.680+0000 7f7938ff9640 1 -- 172.21.15.33:0/2979629198 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f793c009b30 con 0x55d5d45104a0 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.680+0000 7f7938ff9640 1 -- 172.21.15.33:0/2979629198 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c0195e0 con 0x55d5d45104a0 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.681+0000 7f794b888740 1 -- 172.21.15.33:0/2979629198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d45104a0 msgr2=0x55d5d45274b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.681+0000 7f794b888740 1 --2- 172.21.15.33:0/2979629198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d45104a0 0x55d5d45274b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f793c002230 tx=0x7f793c009230 comp rx=0 tx=0).stop 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.681+0000 7f794b888740 1 -- 172.21.15.33:0/2979629198 shutdown_connections 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.681+0000 7f794b888740 1 --2- 172.21.15.33:0/2979629198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d451b680 0x55d5d4517270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.681+0000 7f794b888740 1 --2- 172.21.15.33:0/2979629198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d4510dc0 0x55d5d450ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.681+0000 7f794b888740 1 --2- 172.21.15.33:0/2979629198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d45104a0 0x55d5d45274b0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.681+0000 7f794b888740 1 -- 172.21.15.33:0/2979629198 >> 172.21.15.33:0/2979629198 conn(0x55d5d450ca30 msgr2=0x55d5d4516b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.681+0000 7f794b888740 1 -- 172.21.15.33:0/2979629198 shutdown_connections 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.681+0000 7f794b888740 1 -- 172.21.15.33:0/2979629198 wait complete. 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.681+0000 7f794b888740 1 -- start start 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.681+0000 7f794b888740 1 Processor -- start 2024-09-16T00:24:21.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f794b888740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d4523cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f794b888740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d451f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f794b888740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d451f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f794b888740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f79441ad8f0 con 0x55d5d451b680 2024-09-16T00:24:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f794b888740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f79441aed60 con 0x55d5d45104a0 2024-09-16T00:24:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f794b888740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f79441adfc0 con 0x55d5d4510dc0 2024-09-16T00:24:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f794b888740 10 client.0 _ll_register_callbacks cb 0x7f7934005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f793bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d451f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f79437fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d451f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f793bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d451f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46226/0 (socket says 172.21.15.33:46226) 2024-09-16T00:24:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f79437fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d451f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45556/0 (socket says 172.21.15.33:45556) 2024-09-16T00:24:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f793bfff640 1 -- 172.21.15.33:0/1809123170 learned_addr learned my addr 172.21.15.33:0/1809123170 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:21.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f79437fe640 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d4523cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f793bfff640 1 -- 172.21.15.33:0/1809123170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 msgr2=0x55d5d4523cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f793bfff640 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d4523cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f793bfff640 1 -- 172.21.15.33:0/1809123170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 msgr2=0x55d5d451f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f793bfff640 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d451f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f793bfff640 1 -- 172.21.15.33:0/1809123170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5d451fd50 con 0x55d5d4510dc0 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f79437fe640 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d451f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f793bfff640 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d451f030 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f792c0096c0 tx=0x7f792c005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c65aed4dddc3969 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.682+0000 7f792a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f792a7fc640 1 -- 172.21.15.33:0/1809123170 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c035040 con 0x55d5d4510dc0 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f794b888740 1 -- 172.21.15.33:0/1809123170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f793c009b30 con 0x55d5d4510dc0 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f792a7fc640 1 -- 172.21.15.33:0/1809123170 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f792c005040 con 0x55d5d4510dc0 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f792a7fc640 1 -- 172.21.15.33:0/1809123170 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f792c054050 con 0x55d5d4510dc0 2024-09-16T00:24:21.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f794b888740 10 client.26188 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f794b888740 1 -- 172.21.15.33:0/1809123170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d5d4595b20 con 0x55d5d4510dc0 2024-09-16T00:24:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f792a7fc640 1 -- 172.21.15.33:0/1809123170 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f793c009b30 con 0x55d5d4510dc0 2024-09-16T00:24:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f792a7fc640 1 -- 172.21.15.33:0/1809123170 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f792c0035f0 con 0x55d5d4510dc0 2024-09-16T00:24:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f794b888740 10 client.26188 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f794b888740 10 client.26188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:24:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f794b888740 20 client.26188 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f794b888740 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d5d45962d0 0x55d5d45986f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f794b888740 4 client.26188 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:24:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.683+0000 7f794b888740 1 -- 172.21.15.33:0/1809123170 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d5d447dd60 con 0x55d5d45962d0 2024-09-16T00:24:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f79437fe640 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d5d45962d0 0x55d5d45986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f79437fe640 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d5d45962d0 0x55d5d45986f0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f404e1e00902b730 server_cookie=477bc25e7e3ddabd in_seq=0 out_seq=0 2024-09-16T00:24:21.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f792a7fc640 10 client.26188 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f792a7fc640 1 -- 172.21.15.33:0/1809123170 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d5d447dd60 con 0x55d5d45962d0 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f792a7fc640 10 client.26188 handle_command_reply: tid=0 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f794b888740 1 client.26188 shutdown 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f794b888740 1 -- 172.21.15.33:0/1809123170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 msgr2=0x55d5d451f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f794b888740 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d451f030 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f792c0096c0 tx=0x7f792c005850 comp rx=0 tx=0).stop 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f794b888740 1 -- 172.21.15.33:0/1809123170 shutdown_connections 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f794b888740 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d5d45962d0 0x55d5d45986f0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f794b888740 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5d451b680 0x55d5d451f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f794b888740 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5d4510dc0 0x55d5d451f030 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f792a7fc640 0 client.26188 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f794b888740 1 --2- 172.21.15.33:0/1809123170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5d45104a0 0x55d5d4523cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f794b888740 1 -- 172.21.15.33:0/1809123170 >> 172.21.15.33:0/1809123170 conn(0x55d5d450ca30 msgr2=0x55d5d4519af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:21.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f794b888740 1 -- 172.21.15.33:0/1809123170 shutdown_connections 2024-09-16T00:24:21.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f794b888740 1 -- 172.21.15.33:0/1809123170 wait complete. 2024-09-16T00:24:21.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.684+0000 7f794b888740 20 client.26188 trim_cache size 0 max 16384 2024-09-16T00:24:21.702 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:24:21.702 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:24:21.702 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:24:21.702 INFO:teuthology.orchestra.run.smithi033.stdout: "a691b521-414e-43a7-83bb-2bbde6f0c950": { 2024-09-16T00:24:21.702 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:24:21.702 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "a691b521-414e-43a7-83bb-2bbde6f0c950", 2024-09-16T00:24:21.702 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:24:21.703 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:24:21.703 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:24:21.703 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:24:21.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.685+0000 7f79494af640 1 -- 172.21.15.33:0/878569558 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f79300a3240 msgr2=0x7f79300a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:21.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.685+0000 7f79494af640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f79300a3240 0x7f79300a5700 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f79340cdab0 tx=0x7f79340d3ba0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.685+0000 7f79494af640 1 -- 172.21.15.33:0/878569558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7944102dd0 msgr2=0x7f79441985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:21.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.685+0000 7f79494af640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7944102dd0 0x7f79441985d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f793400e9f0 tx=0x7f793400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.685+0000 7f79494af640 1 -- 172.21.15.33:0/878569558 shutdown_connections 2024-09-16T00:24:21.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.685+0000 7f79494af640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f79300a3240 0x7f79300a5700 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.685+0000 7f79494af640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7944108300 0x7f794419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.685+0000 7f79494af640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7944103780 0x7f7944198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.685+0000 7f79494af640 1 --2- 172.21.15.33:0/878569558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7944102dd0 0x7f79441985d0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:21.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.685+0000 7f79494af640 1 -- 172.21.15.33:0/878569558 >> 172.21.15.33:0/878569558 conn(0x7f79440fc7d0 msgr2=0x7f7944105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:21.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.685+0000 7f79494af640 1 -- 172.21.15.33:0/878569558 shutdown_connections 2024-09-16T00:24:21.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:21.685+0000 7f79494af640 1 -- 172.21.15.33:0/878569558 wait complete. 2024-09-16T00:24:22.409 INFO:tasks.cephfs.filesystem:scrub status for tag:a691b521-414e-43a7-83bb-2bbde6f0c950 - {'path': '/', 'tag': 'a691b521-414e-43a7-83bb-2bbde6f0c950', 'options': 'recursive,force'} 2024-09-16T00:24:22.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 515671 10.27 MB/sec execute 210 sec latency 7.560 ms 2024-09-16T00:24:23.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:23 smithi033 ceph-mon[31243]: pgmap v567: 129 pgs: 129 active+clean; 185 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 324 op/s 2024-09-16T00:24:23.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 519040 10.32 MB/sec execute 211 sec latency 11.749 ms 2024-09-16T00:24:23.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:23 smithi083 ceph-mon[35242]: pgmap v567: 129 pgs: 129 active+clean; 185 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 324 op/s 2024-09-16T00:24:23.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:23 smithi060 ceph-mon[35185]: pgmap v567: 129 pgs: 129 active+clean; 185 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 324 op/s 2024-09-16T00:24:24.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 522159 10.35 MB/sec execute 212 sec latency 23.690 ms 2024-09-16T00:24:25.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:25 smithi033 ceph-mon[31243]: pgmap v568: 129 pgs: 129 active+clean; 186 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 4.1 MiB/s wr, 298 op/s 2024-09-16T00:24:25.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 524088 10.34 MB/sec execute 213 sec latency 80.752 ms 2024-09-16T00:24:25.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:25 smithi083 ceph-mon[35242]: pgmap v568: 129 pgs: 129 active+clean; 186 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 4.1 MiB/s wr, 298 op/s 2024-09-16T00:24:25.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:25 smithi060 ceph-mon[35185]: pgmap v568: 129 pgs: 129 active+clean; 186 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 4.1 MiB/s wr, 298 op/s 2024-09-16T00:24:26.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 526648 10.35 MB/sec execute 214 sec latency 24.439 ms 2024-09-16T00:24:27.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:27 smithi033 ceph-mon[31243]: pgmap v569: 129 pgs: 129 active+clean; 188 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 4.2 MiB/s wr, 307 op/s 2024-09-16T00:24:27.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:27 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:24:27.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:27 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:24:27.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 529909 10.39 MB/sec execute 215 sec latency 6.396 ms 2024-09-16T00:24:27.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:27 smithi083 ceph-mon[35242]: pgmap v569: 129 pgs: 129 active+clean; 188 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 4.2 MiB/s wr, 307 op/s 2024-09-16T00:24:27.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:27 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:24:27.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:27 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:24:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:27 smithi060 ceph-mon[35185]: pgmap v569: 129 pgs: 129 active+clean; 188 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 4.2 MiB/s wr, 307 op/s 2024-09-16T00:24:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:27 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:24:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:27 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:24:28.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 533188 10.43 MB/sec execute 216 sec latency 20.305 ms 2024-09-16T00:24:29.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:29 smithi033 ceph-mon[31243]: pgmap v570: 129 pgs: 129 active+clean; 188 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 3.2 MiB/s wr, 221 op/s 2024-09-16T00:24:29.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 535876 10.44 MB/sec execute 217 sec latency 58.973 ms 2024-09-16T00:24:29.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:29 smithi083 ceph-mon[35242]: pgmap v570: 129 pgs: 129 active+clean; 188 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 3.2 MiB/s wr, 221 op/s 2024-09-16T00:24:29.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:29 smithi060 ceph-mon[35185]: pgmap v570: 129 pgs: 129 active+clean; 188 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 3.2 MiB/s wr, 221 op/s 2024-09-16T00:24:30.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 538125 10.45 MB/sec execute 218 sec latency 87.980 ms 2024-09-16T00:24:31.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:31 smithi033 ceph-mon[31243]: pgmap v571: 129 pgs: 129 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.8 MiB/s wr, 299 op/s 2024-09-16T00:24:31.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 540734 10.47 MB/sec execute 219 sec latency 102.253 ms 2024-09-16T00:24:31.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:31 smithi083 ceph-mon[35242]: pgmap v571: 129 pgs: 129 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.8 MiB/s wr, 299 op/s 2024-09-16T00:24:31.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:31 smithi060 ceph-mon[35185]: pgmap v571: 129 pgs: 129 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.8 MiB/s wr, 299 op/s 2024-09-16T00:24:32.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 544053 10.50 MB/sec execute 220 sec latency 6.475 ms 2024-09-16T00:24:33.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:33 smithi033 ceph-mon[31243]: pgmap v572: 129 pgs: 129 active+clean; 192 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 298 op/s 2024-09-16T00:24:33.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 547390 10.54 MB/sec execute 221 sec latency 12.109 ms 2024-09-16T00:24:33.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:33 smithi083 ceph-mon[35242]: pgmap v572: 129 pgs: 129 active+clean; 192 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 298 op/s 2024-09-16T00:24:33.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:33 smithi060 ceph-mon[35185]: pgmap v572: 129 pgs: 129 active+clean; 192 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 298 op/s 2024-09-16T00:24:34.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 550371 10.56 MB/sec execute 222 sec latency 17.212 ms 2024-09-16T00:24:35.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:35 smithi033 ceph-mon[31243]: pgmap v573: 129 pgs: 129 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 3.9 MiB/s wr, 300 op/s 2024-09-16T00:24:35.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 553105 10.57 MB/sec execute 223 sec latency 15.586 ms 2024-09-16T00:24:35.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:35 smithi083 ceph-mon[35242]: pgmap v573: 129 pgs: 129 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 3.9 MiB/s wr, 300 op/s 2024-09-16T00:24:35.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:35 smithi060 ceph-mon[35185]: pgmap v573: 129 pgs: 129 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 3.9 MiB/s wr, 300 op/s 2024-09-16T00:24:36.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 554780 10.57 MB/sec execute 224 sec latency 145.029 ms 2024-09-16T00:24:37.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:37 smithi033 ceph-mon[31243]: pgmap v574: 129 pgs: 129 active+clean; 191 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 3.8 MiB/s wr, 269 op/s 2024-09-16T00:24:37.665 INFO:tasks.workunit.client.0.smithi033.stdout: 1 557976 10.61 MB/sec execute 225 sec latency 12.749 ms 2024-09-16T00:24:37.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:37 smithi083 ceph-mon[35242]: pgmap v574: 129 pgs: 129 active+clean; 191 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 3.8 MiB/s wr, 269 op/s 2024-09-16T00:24:37.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:37 smithi060 ceph-mon[35185]: pgmap v574: 129 pgs: 129 active+clean; 191 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 3.8 MiB/s wr, 269 op/s 2024-09-16T00:24:38.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 561042 10.63 MB/sec execute 226 sec latency 15.071 ms 2024-09-16T00:24:39.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:39 smithi033 ceph-mon[31243]: pgmap v575: 129 pgs: 129 active+clean; 191 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 3.4 MiB/s wr, 240 op/s 2024-09-16T00:24:39.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 563696 10.63 MB/sec execute 227 sec latency 12.018 ms 2024-09-16T00:24:39.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:39 smithi083 ceph-mon[35242]: pgmap v575: 129 pgs: 129 active+clean; 191 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 3.4 MiB/s wr, 240 op/s 2024-09-16T00:24:39.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:39 smithi060 ceph-mon[35185]: pgmap v575: 129 pgs: 129 active+clean; 191 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 3.4 MiB/s wr, 240 op/s 2024-09-16T00:24:40.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 566290 10.66 MB/sec execute 228 sec latency 11.774 ms 2024-09-16T00:24:41.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:41 smithi033 ceph-mon[31243]: pgmap v576: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 326 op/s 2024-09-16T00:24:41.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 568724 10.68 MB/sec execute 229 sec latency 51.954 ms 2024-09-16T00:24:41.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:41 smithi083 ceph-mon[35242]: pgmap v576: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 326 op/s 2024-09-16T00:24:41.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:41 smithi060 ceph-mon[35185]: pgmap v576: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 326 op/s 2024-09-16T00:24:42.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 571898 10.70 MB/sec execute 230 sec latency 11.954 ms 2024-09-16T00:24:43.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:43 smithi033 ceph-mon[31243]: pgmap v577: 129 pgs: 129 active+clean; 216 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 293 op/s 2024-09-16T00:24:43.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 574494 10.70 MB/sec execute 231 sec latency 14.630 ms 2024-09-16T00:24:43.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:43 smithi083 ceph-mon[35242]: pgmap v577: 129 pgs: 129 active+clean; 216 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 293 op/s 2024-09-16T00:24:43.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:43 smithi060 ceph-mon[35185]: pgmap v577: 129 pgs: 129 active+clean; 216 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 293 op/s 2024-09-16T00:24:44.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 577043 10.72 MB/sec execute 232 sec latency 12.914 ms 2024-09-16T00:24:45.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:45 smithi033 ceph-mon[31243]: pgmap v578: 129 pgs: 129 active+clean; 216 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.5 MiB/s wr, 297 op/s 2024-09-16T00:24:45.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 579719 10.74 MB/sec execute 233 sec latency 25.360 ms 2024-09-16T00:24:45.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:45 smithi083 ceph-mon[35242]: pgmap v578: 129 pgs: 129 active+clean; 216 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.5 MiB/s wr, 297 op/s 2024-09-16T00:24:45.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:45 smithi060 ceph-mon[35185]: pgmap v578: 129 pgs: 129 active+clean; 216 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.5 MiB/s wr, 297 op/s 2024-09-16T00:24:46.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 582530 10.77 MB/sec execute 234 sec latency 112.259 ms 2024-09-16T00:24:47.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:47 smithi033 ceph-mon[31243]: pgmap v579: 129 pgs: 129 active+clean; 221 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.8 MiB/s wr, 305 op/s 2024-09-16T00:24:47.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 584510 10.75 MB/sec execute 235 sec latency 20.038 ms 2024-09-16T00:24:47.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:47 smithi083 ceph-mon[35242]: pgmap v579: 129 pgs: 129 active+clean; 221 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.8 MiB/s wr, 305 op/s 2024-09-16T00:24:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:47 smithi060 ceph-mon[35185]: pgmap v579: 129 pgs: 129 active+clean; 221 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.8 MiB/s wr, 305 op/s 2024-09-16T00:24:48.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 586984 10.77 MB/sec execute 236 sec latency 16.314 ms 2024-09-16T00:24:49.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:49 smithi033 ceph-mon[31243]: pgmap v580: 129 pgs: 129 active+clean; 221 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.0 MiB/s wr, 280 op/s 2024-09-16T00:24:49.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 590059 10.80 MB/sec execute 237 sec latency 18.099 ms 2024-09-16T00:24:49.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:49 smithi083 ceph-mon[35242]: pgmap v580: 129 pgs: 129 active+clean; 221 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.0 MiB/s wr, 280 op/s 2024-09-16T00:24:49.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:49 smithi060 ceph-mon[35185]: pgmap v580: 129 pgs: 129 active+clean; 221 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.0 MiB/s wr, 280 op/s 2024-09-16T00:24:50.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 593235 10.83 MB/sec execute 238 sec latency 26.727 ms 2024-09-16T00:24:51.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:51 smithi033 ceph-mon[31243]: pgmap v581: 129 pgs: 129 active+clean; 221 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.5 MiB/s wr, 308 op/s 2024-09-16T00:24:51.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 594783 10.81 MB/sec execute 239 sec latency 73.133 ms 2024-09-16T00:24:51.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:51 smithi083 ceph-mon[35242]: pgmap v581: 129 pgs: 129 active+clean; 221 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.5 MiB/s wr, 308 op/s 2024-09-16T00:24:51.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:51 smithi060 ceph-mon[35185]: pgmap v581: 129 pgs: 129 active+clean; 221 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.5 MiB/s wr, 308 op/s 2024-09-16T00:24:52.411 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:24:52.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 597331 10.82 MB/sec execute 240 sec latency 20.211 ms 2024-09-16T00:24:52.685 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:24:53.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.202+0000 7f17ab5a3640 1 -- 172.21.15.33:0/2154817797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a4100e20 msgr2=0x7f17a4101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:53.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.202+0000 7f17ab5a3640 1 --2- 172.21.15.33:0/2154817797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a4100e20 0x7f17a4101200 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009900 tx=0x7f17a002a480 comp rx=0 tx=0).stop 2024-09-16T00:24:53.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.202+0000 7f17ab5a3640 1 -- 172.21.15.33:0/2154817797 shutdown_connections 2024-09-16T00:24:53.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.202+0000 7f17ab5a3640 1 --2- 172.21.15.33:0/2154817797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17a4102ef0 0x7f17a410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.202+0000 7f17ab5a3640 1 --2- 172.21.15.33:0/2154817797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17a4102570 0x7f17a41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.202+0000 7f17ab5a3640 1 --2- 172.21.15.33:0/2154817797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a4100e20 0x7f17a4101200 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.202+0000 7f17ab5a3640 1 -- 172.21.15.33:0/2154817797 >> 172.21.15.33:0/2154817797 conn(0x7f17a40fc7d0 msgr2=0x7f17a40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:53.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.202+0000 7f17ab5a3640 1 -- 172.21.15.33:0/2154817797 shutdown_connections 2024-09-16T00:24:53.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.202+0000 7f17ab5a3640 1 -- 172.21.15.33:0/2154817797 wait complete. 2024-09-16T00:24:53.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17ab5a3640 1 Processor -- start 2024-09-16T00:24:53.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17ab5a3640 1 -- start start 2024-09-16T00:24:53.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17ab5a3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17a4100e20 0x7f17a419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17ab5a3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17a4102570 0x7f17a419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17ab5a3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a4102ef0 0x7f17a419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17ab5a3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f17a410fe80 con 0x7f17a4102ef0 2024-09-16T00:24:53.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17ab5a3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f17a410fd00 con 0x7f17a4102570 2024-09-16T00:24:53.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17ab5a3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f17a4110000 con 0x7f17a4100e20 2024-09-16T00:24:53.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17a9b40640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a4102ef0 0x7f17a419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17a933f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17a4100e20 0x7f17a419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17a9b40640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a4102ef0 0x7f17a419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34392/0 (socket says 172.21.15.33:34392) 2024-09-16T00:24:53.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17a8b3e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17a4102570 0x7f17a419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17a9b40640 1 -- 172.21.15.33:0/3139874839 learned_addr learned my addr 172.21.15.33:0/3139874839 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:53.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17a933f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17a4100e20 0x7f17a419a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41164/0 (socket says 172.21.15.33:41164) 2024-09-16T00:24:53.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17a9b40640 1 -- 172.21.15.33:0/3139874839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17a4100e20 msgr2=0x7f17a419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:53.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17a9b40640 1 --2- 172.21.15.33:0/3139874839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17a4100e20 0x7f17a419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.203+0000 7f17a9b40640 1 -- 172.21.15.33:0/3139874839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17a4102570 msgr2=0x7f17a419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.204+0000 7f17a9b40640 1 --2- 172.21.15.33:0/3139874839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17a4102570 0x7f17a419ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.204+0000 7f17a9b40640 1 -- 172.21.15.33:0/3139874839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a419f6a0 con 0x7f17a4102ef0 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.204+0000 7f17a9b40640 1 --2- 172.21.15.33:0/3139874839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a4102ef0 0x7f17a419efc0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f179c00da90 tx=0x7f179c00df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.204+0000 7f179a7fc640 1 -- 172.21.15.33:0/3139874839 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c01b020 con 0x7f17a4102ef0 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.204+0000 7f179a7fc640 1 -- 172.21.15.33:0/3139874839 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f179c00bc30 con 0x7f17a4102ef0 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.204+0000 7f17ab5a3640 1 -- 172.21.15.33:0/3139874839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a419f990 con 0x7f17a4102ef0 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.204+0000 7f179a7fc640 1 -- 172.21.15.33:0/3139874839 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c0173f0 con 0x7f17a4102ef0 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.204+0000 7f17ab5a3640 1 -- 172.21.15.33:0/3139874839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a41a7270 con 0x7f17a4102ef0 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f179a7fc640 1 -- 172.21.15.33:0/3139874839 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f179c004a40 con 0x7f17a4102ef0 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17ad97c740 1 Processor -- start 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17ad97c740 1 -- start start 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17ad97c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c2e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17ad97c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2edc0 0x55e180c39140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17ad97c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c39680 0x55e180c3ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17ad97c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e180b942e0 con 0x55e180c2edc0 2024-09-16T00:24:53.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17ad97c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e180b3bce0 con 0x55e180c39680 2024-09-16T00:24:53.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17ad97c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e180b902a0 con 0x55e180c2e4a0 2024-09-16T00:24:53.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17a8b3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c2e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17a933f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2edc0 0x55e180c39140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17a8b3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c2e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41180/0 (socket says 172.21.15.33:41180) 2024-09-16T00:24:53.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17a8b3e640 1 -- 172.21.15.33:0/1574281130 learned_addr learned my addr 172.21.15.33:0/1574281130 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:53.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.207+0000 7f17a8b3e640 1 --2- 172.21.15.33:0/1574281130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c39680 0x55e180c3ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f179a7fc640 1 --2- 172.21.15.33:0/3139874839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f17900a32c0 0x7f17900a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f17a8b3e640 1 -- 172.21.15.33:0/1574281130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c39680 msgr2=0x55e180c3ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:53.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f17a8b3e640 1 --2- 172.21.15.33:0/1574281130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c39680 0x55e180c3ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f17a8b3e640 1 -- 172.21.15.33:0/1574281130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2edc0 msgr2=0x55e180c39140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f17a8b3e640 1 --2- 172.21.15.33:0/1574281130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2edc0 0x55e180c39140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f17a8b3e640 1 -- 172.21.15.33:0/1574281130 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e180b20440 con 0x55e180c2e4a0 2024-09-16T00:24:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f17a933f640 1 --2- 172.21.15.33:0/1574281130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2edc0 0x55e180c39140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f17a8b3e640 1 --2- 172.21.15.33:0/1574281130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c2e880 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1794009fd0 tx=0x7f1794005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=372fd6c76ebce813 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f178f7fe640 1 -- 172.21.15.33:0/1574281130 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1794004040 con 0x55e180c2e4a0 2024-09-16T00:24:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f17a9b40640 1 --2- 172.21.15.33:0/3139874839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f17900a32c0 0x7f17900a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f178f7fe640 1 -- 172.21.15.33:0/1574281130 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1794005500 con 0x55e180c2e4a0 2024-09-16T00:24:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f178f7fe640 1 -- 172.21.15.33:0/1574281130 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1794004d30 con 0x55e180c2e4a0 2024-09-16T00:24:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.208+0000 7f179a7fc640 1 -- 172.21.15.33:0/3139874839 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f179c0cfb70 con 0x7f17a4102ef0 2024-09-16T00:24:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.209+0000 7f17ad97c740 1 -- 172.21.15.33:0/1574281130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 msgr2=0x55e180c2e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.209+0000 7f17a9b40640 1 --2- 172.21.15.33:0/3139874839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f17900a32c0 0x7f17900a5780 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f179c09bd30 tx=0x7f179c0866e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.209+0000 7f17ad97c740 1 --2- 172.21.15.33:0/1574281130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c2e880 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1794009fd0 tx=0x7f1794005850 comp rx=0 tx=0).stop 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.209+0000 7f17ad97c740 1 -- 172.21.15.33:0/1574281130 shutdown_connections 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.209+0000 7f17ad97c740 1 --2- 172.21.15.33:0/1574281130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c39680 0x55e180c3ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.209+0000 7f17ad97c740 1 --2- 172.21.15.33:0/1574281130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2edc0 0x55e180c39140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.209+0000 7f17ad97c740 1 --2- 172.21.15.33:0/1574281130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c2e880 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.209+0000 7f17ad97c740 1 -- 172.21.15.33:0/1574281130 >> 172.21.15.33:0/1574281130 conn(0x55e180c2aa30 msgr2=0x55e180c2ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.209+0000 7f17ad97c740 1 -- 172.21.15.33:0/1574281130 shutdown_connections 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.209+0000 7f17ad97c740 1 -- 172.21.15.33:0/1574281130 wait complete. 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17ad97c740 1 -- start start 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17ad97c740 1 Processor -- start 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17ad97c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c38f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17ad97c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c36240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17ad97c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c39680 0x55e180c36780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17ad97c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e180c3c250 con 0x55e180c39680 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17ad97c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e180b4f050 con 0x55e180c2edc0 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17ad97c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e180c3c3d0 con 0x55e180c2e4a0 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17ad97c740 10 client.0 _ll_register_callbacks cb 0x55e180b8f160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17a8b3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c38f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17a933f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c36240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17a8b3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c38f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41182/0 (socket says 172.21.15.33:41182) 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17a933f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c36240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:47242/0 (socket says 172.21.15.33:47242) 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17a8b3e640 1 -- 172.21.15.33:0/4029477874 learned_addr learned my addr 172.21.15.33:0/4029477874 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.210+0000 7f17a8b3e640 1 --2- 172.21.15.33:0/4029477874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c39680 0x55e180c36780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f17a933f640 1 -- 172.21.15.33:0/4029477874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 msgr2=0x55e180c38f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f17a933f640 1 --2- 172.21.15.33:0/4029477874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c38f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f17a933f640 1 -- 172.21.15.33:0/4029477874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c39680 msgr2=0x55e180c36780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:53.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f17a933f640 1 --2- 172.21.15.33:0/4029477874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c39680 0x55e180c36780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f17a933f640 1 -- 172.21.15.33:0/4029477874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1809053c0 con 0x55e180c2edc0 2024-09-16T00:24:53.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f17a933f640 1 --2- 172.21.15.33:0/4029477874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c36240 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17a0005940 tx=0x7f17a00042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74611d23d5fd9807 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:53.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f178cff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:24:53.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f178cff9640 1 -- 172.21.15.33:0/4029477874 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a0035040 con 0x55e180c2edc0 2024-09-16T00:24:53.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f17ad97c740 1 -- 172.21.15.33:0/4029477874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1794005500 con 0x55e180c2edc0 2024-09-16T00:24:53.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f178cff9640 1 -- 172.21.15.33:0/4029477874 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f17a0003a00 con 0x55e180c2edc0 2024-09-16T00:24:53.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f178cff9640 1 -- 172.21.15.33:0/4029477874 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f17a004e050 con 0x55e180c2edc0 2024-09-16T00:24:53.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f17ad97c740 10 client.26276 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:53.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.211+0000 7f17ad97c740 1 -- 172.21.15.33:0/4029477874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e180c47c20 con 0x55e180c2edc0 2024-09-16T00:24:53.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.212+0000 7f178cff9640 1 -- 172.21.15.33:0/4029477874 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1794005500 con 0x55e180c2edc0 2024-09-16T00:24:53.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.212+0000 7f178cff9640 1 -- 172.21.15.33:0/4029477874 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f17a003c440 con 0x55e180c2edc0 2024-09-16T00:24:53.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.212+0000 7f17ad97c740 10 client.26276 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:53.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.212+0000 7f17ad97c740 10 client.26276 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:24:53.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.212+0000 7f17ad97c740 20 client.26276 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:53.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.212+0000 7f17ad97c740 1 --2- 172.21.15.33:0/4029477874 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e180c49080 0x55e180c4b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.212+0000 7f17ad97c740 4 client.26276 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:24:53.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.212+0000 7f17ad97c740 1 -- 172.21.15.33:0/4029477874 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e180ba1850 con 0x55e180c49080 2024-09-16T00:24:53.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.212+0000 7f17a8b3e640 1 --2- 172.21.15.33:0/4029477874 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e180c49080 0x55e180c4b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.212+0000 7f17a8b3e640 1 --2- 172.21.15.33:0/4029477874 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e180c49080 0x55e180c4b470 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56d3a14564b48e40 server_cookie=977f7f0808de3741 in_seq=0 out_seq=0 2024-09-16T00:24:53.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.212+0000 7f178cff9640 10 client.26276 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:53.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f178cff9640 1 -- 172.21.15.33:0/4029477874 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e180ba1850 con 0x55e180c49080 2024-09-16T00:24:53.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f178cff9640 10 client.26276 handle_command_reply: tid=0 2024-09-16T00:24:53.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f17ad97c740 1 client.26276 shutdown 2024-09-16T00:24:53.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f17ad97c740 1 -- 172.21.15.33:0/4029477874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 msgr2=0x55e180c36240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:53.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f17ad97c740 1 --2- 172.21.15.33:0/4029477874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c36240 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17a0005940 tx=0x7f17a00042e0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f17ad97c740 1 -- 172.21.15.33:0/4029477874 shutdown_connections 2024-09-16T00:24:53.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f17ad97c740 1 --2- 172.21.15.33:0/4029477874 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e180c49080 0x55e180c4b470 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f17ad97c740 1 --2- 172.21.15.33:0/4029477874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c39680 0x55e180c36780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f17ad97c740 1 --2- 172.21.15.33:0/4029477874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c36240 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f178cff9640 0 client.26276 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:53.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f17ad97c740 1 --2- 172.21.15.33:0/4029477874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c38f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f17ad97c740 1 -- 172.21.15.33:0/4029477874 >> 172.21.15.33:0/4029477874 conn(0x55e180c2aa30 msgr2=0x55e180c30760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:53.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f17ad97c740 1 -- 172.21.15.33:0/4029477874 shutdown_connections 2024-09-16T00:24:53.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f17ad97c740 1 -- 172.21.15.33:0/4029477874 wait complete. 2024-09-16T00:24:53.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.213+0000 7f17ad97c740 20 client.26276 trim_cache size 0 max 16384 2024-09-16T00:24:53.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.232+0000 7f17ad97c740 1 Processor -- start 2024-09-16T00:24:53.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.232+0000 7f17ad97c740 1 -- start start 2024-09-16T00:24:53.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.232+0000 7f17ad97c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c454b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.232+0000 7f17ad97c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c50d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.232+0000 7f17ad97c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c39680 0x55e180c51260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.232+0000 7f17ad97c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41b3b00 con 0x55e180c39680 2024-09-16T00:24:53.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.232+0000 7f17ad97c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41b13d0 con 0x55e180c2edc0 2024-09-16T00:24:53.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.232+0000 7f17ad97c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41af220 con 0x55e180c2e4a0 2024-09-16T00:24:53.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f17a933f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c50d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f17a8b3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f17a933f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c50d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:47254/0 (socket says 172.21.15.33:47254) 2024-09-16T00:24:53.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f17a933f640 1 -- 172.21.15.33:0/2564661634 learned_addr learned my addr 172.21.15.33:0/2564661634 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:53.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f17a8b3e640 1 --2- 172.21.15.33:0/2564661634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c39680 0x55e180c51260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f17a933f640 1 -- 172.21.15.33:0/2564661634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 msgr2=0x55e180c454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:53.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f17a933f640 1 --2- 172.21.15.33:0/2564661634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f17a933f640 1 -- 172.21.15.33:0/2564661634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c39680 msgr2=0x55e180c51260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:53.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f17a933f640 1 --2- 172.21.15.33:0/2564661634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c39680 0x55e180c51260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f17a933f640 1 -- 172.21.15.33:0/2564661634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e180c2ebf0 con 0x55e180c2edc0 2024-09-16T00:24:53.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f17a933f640 1 --2- 172.21.15.33:0/2564661634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c50d20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f17a002a9a0 tx=0x7f17a0009230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d1408ea285bf95c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:53.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f178effd640 1 -- 172.21.15.33:0/2564661634 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a004b070 con 0x55e180c2edc0 2024-09-16T00:24:53.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f178effd640 1 -- 172.21.15.33:0/2564661634 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17a003cbe0 con 0x55e180c2edc0 2024-09-16T00:24:53.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.233+0000 7f178effd640 1 -- 172.21.15.33:0/2564661634 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a003ce60 con 0x55e180c2edc0 2024-09-16T00:24:53.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.234+0000 7f17ad97c740 1 -- 172.21.15.33:0/2564661634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 msgr2=0x55e180c50d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:53.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.234+0000 7f17ad97c740 1 --2- 172.21.15.33:0/2564661634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c50d20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f17a002a9a0 tx=0x7f17a0009230 comp rx=0 tx=0).stop 2024-09-16T00:24:53.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.234+0000 7f17ad97c740 1 -- 172.21.15.33:0/2564661634 shutdown_connections 2024-09-16T00:24:53.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.234+0000 7f17ad97c740 1 --2- 172.21.15.33:0/2564661634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c39680 0x55e180c51260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.234+0000 7f17ad97c740 1 --2- 172.21.15.33:0/2564661634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c50d20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.234+0000 7f17ad97c740 1 --2- 172.21.15.33:0/2564661634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c2e4a0 0x55e180c454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.234+0000 7f17ad97c740 1 -- 172.21.15.33:0/2564661634 >> 172.21.15.33:0/2564661634 conn(0x55e180c2aa30 msgr2=0x55e180c31900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:53.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.234+0000 7f17ad97c740 1 -- 172.21.15.33:0/2564661634 shutdown_connections 2024-09-16T00:24:53.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.234+0000 7f17ad97c740 1 -- 172.21.15.33:0/2564661634 wait complete. 2024-09-16T00:24:53.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.234+0000 7f17ad97c740 1 -- start start 2024-09-16T00:24:53.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17ad97c740 1 Processor -- start 2024-09-16T00:24:53.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17ad97c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2e4a0 0x55e180c41c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17ad97c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c3cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17ad97c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c39680 0x55e180c3d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17ad97c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41af890 con 0x55e180c2e4a0 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17ad97c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41b0d00 con 0x55e180c2edc0 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17ad97c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41aff60 con 0x55e180c39680 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17ad97c740 10 client.0 _ll_register_callbacks cb 0x7f17a0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17a8b3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2e4a0 0x55e180c41c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17a8b3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2e4a0 0x55e180c41c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34426/0 (socket says 172.21.15.33:34426) 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17a8b3e640 1 -- 172.21.15.33:0/1962347330 learned_addr learned my addr 172.21.15.33:0/1962347330 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17a8b3e640 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c39680 0x55e180c3d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.235+0000 7f17a933f640 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c3cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f17a8b3e640 1 -- 172.21.15.33:0/1962347330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c39680 msgr2=0x55e180c3d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f17a8b3e640 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c39680 0x55e180c3d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f17a8b3e640 1 -- 172.21.15.33:0/1962347330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 msgr2=0x55e180c3cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f17a8b3e640 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c3cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f17a8b3e640 1 -- 172.21.15.33:0/1962347330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e180c3dbd0 con 0x55e180c2e4a0 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f17a933f640 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c3cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f17a8b3e640 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2e4a0 0x55e180c41c20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1794004920 tx=0x7f1794005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c39ce49bd47eb2f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f178cff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f178cff9640 1 -- 172.21.15.33:0/1962347330 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1794004130 con 0x55e180c2e4a0 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f178cff9640 1 -- 172.21.15.33:0/1962347330 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f17940042d0 con 0x55e180c2e4a0 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f17ad97c740 1 -- 172.21.15.33:0/1962347330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17a003cbe0 con 0x55e180c2e4a0 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f178cff9640 1 -- 172.21.15.33:0/1962347330 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f17940044d0 con 0x55e180c2e4a0 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f17ad97c740 10 client.16764 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f17ad97c740 1 -- 172.21.15.33:0/1962347330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e180cb3ba0 con 0x55e180c2e4a0 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.236+0000 7f178cff9640 1 -- 172.21.15.33:0/1962347330 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17a003cbe0 con 0x55e180c2e4a0 2024-09-16T00:24:53.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f178cff9640 1 -- 172.21.15.33:0/1962347330 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f179400fc10 con 0x55e180c2e4a0 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f17ad97c740 10 client.16764 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f17ad97c740 10 client.16764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f17ad97c740 20 client.16764 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f17ad97c740 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e180cb40d0 0x55e180cb64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f17ad97c740 4 client.16764 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f17ad97c740 1 -- 172.21.15.33:0/1962347330 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e180b9bd60 con 0x55e180cb40d0 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f17a933f640 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e180cb40d0 0x55e180cb64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f17a933f640 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e180cb40d0 0x55e180cb64c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f68e37320b5e9b3 server_cookie=281feee3707a24b8 in_seq=0 out_seq=0 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f178cff9640 10 client.16764 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f178cff9640 1 -- 172.21.15.33:0/1962347330 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e180b9bd60 con 0x55e180cb40d0 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f178cff9640 10 client.16764 handle_command_reply: tid=0 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.237+0000 7f17ad97c740 1 client.16764 shutdown 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.238+0000 7f17ad97c740 1 -- 172.21.15.33:0/1962347330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2e4a0 msgr2=0x55e180c41c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.238+0000 7f17ad97c740 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2e4a0 0x55e180c41c20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1794004920 tx=0x7f1794005850 comp rx=0 tx=0).stop 2024-09-16T00:24:53.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.238+0000 7f17ad97c740 1 -- 172.21.15.33:0/1962347330 shutdown_connections 2024-09-16T00:24:53.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.238+0000 7f17ad97c740 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e180cb40d0 0x55e180cb64c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.238+0000 7f17ad97c740 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e180c39680 0x55e180c3d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.238+0000 7f17ad97c740 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e180c2edc0 0x55e180c3cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.238+0000 7f17ad97c740 1 --2- 172.21.15.33:0/1962347330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e180c2e4a0 0x55e180c41c20 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.238+0000 7f17ad97c740 1 -- 172.21.15.33:0/1962347330 >> 172.21.15.33:0/1962347330 conn(0x55e180c2aa30 msgr2=0x55e180c29350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:53.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.238+0000 7f178cff9640 0 client.16764 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:24:53.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.238+0000 7f17ad97c740 1 -- 172.21.15.33:0/1962347330 shutdown_connections 2024-09-16T00:24:53.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.238+0000 7f17ad97c740 1 -- 172.21.15.33:0/1962347330 wait complete. 2024-09-16T00:24:53.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.238+0000 7f17ad97c740 20 client.16764 trim_cache size 0 max 16384 2024-09-16T00:24:53.245 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:24:53.245 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:24:53.245 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:24:53.245 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:24:53.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.239+0000 7f17ab5a3640 1 -- 172.21.15.33:0/3139874839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f17900a32c0 msgr2=0x7f17900a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:53.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.239+0000 7f17ab5a3640 1 --2- 172.21.15.33:0/3139874839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f17900a32c0 0x7f17900a5780 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f179c09bd30 tx=0x7f179c0866e0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.239+0000 7f17ab5a3640 1 -- 172.21.15.33:0/3139874839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a4102ef0 msgr2=0x7f17a419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:53.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.239+0000 7f17ab5a3640 1 --2- 172.21.15.33:0/3139874839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a4102ef0 0x7f17a419efc0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f179c00da90 tx=0x7f179c00df50 comp rx=0 tx=0).stop 2024-09-16T00:24:53.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.239+0000 7f17ab5a3640 1 -- 172.21.15.33:0/3139874839 shutdown_connections 2024-09-16T00:24:53.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.239+0000 7f17ab5a3640 1 --2- 172.21.15.33:0/3139874839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f17900a32c0 0x7f17900a5780 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.239+0000 7f17ab5a3640 1 --2- 172.21.15.33:0/3139874839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a4102ef0 0x7f17a419efc0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.239+0000 7f17ab5a3640 1 --2- 172.21.15.33:0/3139874839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17a4102570 0x7f17a419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.239+0000 7f17ab5a3640 1 --2- 172.21.15.33:0/3139874839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17a4100e20 0x7f17a419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:53.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.239+0000 7f17ab5a3640 1 -- 172.21.15.33:0/3139874839 >> 172.21.15.33:0/3139874839 conn(0x7f17a40fc7d0 msgr2=0x7f17a40fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:53.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.239+0000 7f17ab5a3640 1 -- 172.21.15.33:0/3139874839 shutdown_connections 2024-09-16T00:24:53.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:53.239+0000 7f17ab5a3640 1 -- 172.21.15.33:0/3139874839 wait complete. 2024-09-16T00:24:53.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:53 smithi033 ceph-mon[31243]: pgmap v582: 129 pgs: 129 active+clean; 223 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.0 MiB/s wr, 269 op/s 2024-09-16T00:24:53.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 600396 10.85 MB/sec execute 241 sec latency 30.358 ms 2024-09-16T00:24:53.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:53 smithi083 ceph-mon[35242]: pgmap v582: 129 pgs: 129 active+clean; 223 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.0 MiB/s wr, 269 op/s 2024-09-16T00:24:53.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:53 smithi060 ceph-mon[35185]: pgmap v582: 129 pgs: 129 active+clean; 223 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.0 MiB/s wr, 269 op/s 2024-09-16T00:24:54.171 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:24:54.172 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:24:54.452 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:24:54.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 602825 10.85 MB/sec execute 242 sec latency 23.550 ms 2024-09-16T00:24:55.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:55 smithi033 ceph-mon[31243]: pgmap v583: 129 pgs: 129 active+clean; 227 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 296 op/s 2024-09-16T00:24:55.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 604782 10.85 MB/sec execute 243 sec latency 65.582 ms 2024-09-16T00:24:55.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:55 smithi083 ceph-mon[35242]: pgmap v583: 129 pgs: 129 active+clean; 227 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 296 op/s 2024-09-16T00:24:55.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:55 smithi060 ceph-mon[35185]: pgmap v583: 129 pgs: 129 active+clean; 227 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 296 op/s 2024-09-16T00:24:55.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.917+0000 7feb8dfbf640 1 -- 172.21.15.33:0/1258127038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88101300 msgr2=0x7feb8810dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:55.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.917+0000 7feb8dfbf640 1 --2- 172.21.15.33:0/1258127038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88101300 0x7feb8810dfc0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7feb70009900 tx=0x7feb7002a410 comp rx=0 tx=0).stop 2024-09-16T00:24:55.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.917+0000 7feb8dfbf640 1 -- 172.21.15.33:0/1258127038 shutdown_connections 2024-09-16T00:24:55.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.917+0000 7feb8dfbf640 1 --2- 172.21.15.33:0/1258127038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb8810e500 0x7feb881108f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:55.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.917+0000 7feb8dfbf640 1 --2- 172.21.15.33:0/1258127038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88101300 0x7feb8810dfc0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:55.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.917+0000 7feb8dfbf640 1 --2- 172.21.15.33:0/1258127038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb88100950 0x7feb88100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:55.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.917+0000 7feb8dfbf640 1 -- 172.21.15.33:0/1258127038 >> 172.21.15.33:0/1258127038 conn(0x7feb880fc7a0 msgr2=0x7feb880febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:55.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.917+0000 7feb8dfbf640 1 -- 172.21.15.33:0/1258127038 shutdown_connections 2024-09-16T00:24:55.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.917+0000 7feb8dfbf640 1 -- 172.21.15.33:0/1258127038 wait complete. 2024-09-16T00:24:55.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb8dfbf640 1 Processor -- start 2024-09-16T00:24:55.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb8dfbf640 1 -- start start 2024-09-16T00:24:55.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb8dfbf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb88100950 0x7feb8819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:55.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb8dfbf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88101300 0x7feb8819abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:55.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb8dfbf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb8810e500 0x7feb8819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:55.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb8dfbf640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7feb881141a0 con 0x7feb88101300 2024-09-16T00:24:55.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb8dfbf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7feb88114020 con 0x7feb8810e500 2024-09-16T00:24:55.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb8dfbf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feb88114320 con 0x7feb88100950 2024-09-16T00:24:55.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb7effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88101300 0x7feb8819abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:55.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb877fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb88100950 0x7feb8819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:55.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb877fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb88100950 0x7feb8819a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41218/0 (socket says 172.21.15.33:41218) 2024-09-16T00:24:55.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb7effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88101300 0x7feb8819abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34438/0 (socket says 172.21.15.33:34438) 2024-09-16T00:24:55.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb877fe640 1 -- 172.21.15.33:0/3384848652 learned_addr learned my addr 172.21.15.33:0/3384848652 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:55.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.919+0000 7feb87fff640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb8810e500 0x7feb8819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.920+0000 7feb877fe640 1 -- 172.21.15.33:0/3384848652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb8810e500 msgr2=0x7feb8819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.920+0000 7feb877fe640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb8810e500 0x7feb8819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.920+0000 7feb877fe640 1 -- 172.21.15.33:0/3384848652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88101300 msgr2=0x7feb8819abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.920+0000 7feb877fe640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88101300 0x7feb8819abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.920+0000 7feb877fe640 1 -- 172.21.15.33:0/3384848652 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb8819f650 con 0x7feb88100950 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.920+0000 7feb877fe640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb88100950 0x7feb8819a6a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feb7800d950 tx=0x7feb7800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.920+0000 7feb857fa640 1 -- 172.21.15.33:0/3384848652 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb78017070 con 0x7feb88100950 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.920+0000 7feb8dfbf640 1 -- 172.21.15.33:0/3384848652 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb8819f940 con 0x7feb88100950 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.920+0000 7feb857fa640 1 -- 172.21.15.33:0/3384848652 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb78009c30 con 0x7feb88100950 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.920+0000 7feb857fa640 1 -- 172.21.15.33:0/3384848652 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7800f6b0 con 0x7feb88100950 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.920+0000 7feb8dfbf640 1 -- 172.21.15.33:0/3384848652 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb881a7220 con 0x7feb88100950 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.921+0000 7feb8dfbf640 1 -- 172.21.15.33:0/3384848652 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb88100d30 con 0x7feb88100950 2024-09-16T00:24:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.922+0000 7feb857fa640 1 -- 172.21.15.33:0/3384848652 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7feb780109d0 con 0x7feb88100950 2024-09-16T00:24:55.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.923+0000 7feb857fa640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7feb740a3240 0x7feb740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:55.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.923+0000 7feb7effd640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7feb740a3240 0x7feb740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:55.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.923+0000 7feb857fa640 1 -- 172.21.15.33:0/3384848652 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7feb780cf910 con 0x7feb88100950 2024-09-16T00:24:55.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.924+0000 7feb7effd640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7feb740a3240 0x7feb740a5700 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7feb70009230 tx=0x7feb700058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:55.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:55.926+0000 7feb857fa640 1 -- 172.21.15.33:0/3384848652 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7feb78009de0 con 0x7feb88100950 2024-09-16T00:24:56.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.206+0000 7feb8dfbf640 1 -- 172.21.15.33:0/3384848652 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb88109690 con 0x7feb88100950 2024-09-16T00:24:56.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.207+0000 7feb857fa640 1 -- 172.21.15.33:0/3384848652 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7feb7809bb60 con 0x7feb88100950 2024-09-16T00:24:56.209 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:24:56.209 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:24:56.210 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:24:56.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.213+0000 7feb8dfbf640 1 -- 172.21.15.33:0/3384848652 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7feb740a3240 msgr2=0x7feb740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:56.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.213+0000 7feb8dfbf640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7feb740a3240 0x7feb740a5700 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7feb70009230 tx=0x7feb700058d0 comp rx=0 tx=0).stop 2024-09-16T00:24:56.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.213+0000 7feb8dfbf640 1 -- 172.21.15.33:0/3384848652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb88100950 msgr2=0x7feb8819a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:56.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.213+0000 7feb8dfbf640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb88100950 0x7feb8819a6a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feb7800d950 tx=0x7feb7800de10 comp rx=0 tx=0).stop 2024-09-16T00:24:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.215+0000 7feb8dfbf640 1 -- 172.21.15.33:0/3384848652 shutdown_connections 2024-09-16T00:24:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.215+0000 7feb8dfbf640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7feb740a3240 0x7feb740a5700 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.215+0000 7feb8dfbf640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb8810e500 0x7feb8819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.215+0000 7feb8dfbf640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88101300 0x7feb8819abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.215+0000 7feb8dfbf640 1 --2- 172.21.15.33:0/3384848652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb88100950 0x7feb8819a6a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.215+0000 7feb8dfbf640 1 -- 172.21.15.33:0/3384848652 >> 172.21.15.33:0/3384848652 conn(0x7feb880fc7a0 msgr2=0x7feb8810f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:56.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.215+0000 7feb8dfbf640 1 -- 172.21.15.33:0/3384848652 shutdown_connections 2024-09-16T00:24:56.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:56.216+0000 7feb8dfbf640 1 -- 172.21.15.33:0/3384848652 wait complete. 2024-09-16T00:24:56.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:56 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3384848652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:24:56.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 607618 10.88 MB/sec execute 244 sec latency 88.984 ms 2024-09-16T00:24:56.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:56 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3384848652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:24:56.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:56 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3384848652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:24:57.100 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:24:57.413 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:24:57.569 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:57 smithi083 ceph-mon[35242]: pgmap v584: 129 pgs: 129 active+clean; 228 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 327 op/s 2024-09-16T00:24:57.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:57 smithi033 ceph-mon[31243]: pgmap v584: 129 pgs: 129 active+clean; 228 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 327 op/s 2024-09-16T00:24:57.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 609948 10.87 MB/sec execute 245 sec latency 18.516 ms 2024-09-16T00:24:57.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:57 smithi060 ceph-mon[35185]: pgmap v584: 129 pgs: 129 active+clean; 228 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 327 op/s 2024-09-16T00:24:58.045 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.7 2024-09-16T00:24:58.045 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:24:58.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.158+0000 7fcebaf04640 1 -- 172.21.15.33:0/2854871662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceb4103740 msgr2=0x7fceb4107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:58.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.158+0000 7fcebaf04640 1 --2- 172.21.15.33:0/2854871662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceb4103740 0x7fceb4107b80 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009930 tx=0x7fcea002a420 comp rx=0 tx=0).stop 2024-09-16T00:24:58.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.158+0000 7fcebaf04640 1 -- 172.21.15.33:0/2854871662 shutdown_connections 2024-09-16T00:24:58.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.158+0000 7fcebaf04640 1 --2- 172.21.15.33:0/2854871662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fceb4108290 0x7fceb410ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.158+0000 7fcebaf04640 1 --2- 172.21.15.33:0/2854871662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceb4103740 0x7fceb4107b80 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.158+0000 7fcebaf04640 1 --2- 172.21.15.33:0/2854871662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fceb4102d90 0x7fceb4103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.158+0000 7fcebaf04640 1 -- 172.21.15.33:0/2854871662 >> 172.21.15.33:0/2854871662 conn(0x7fceb40fc7d0 msgr2=0x7fceb40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.158+0000 7fcebaf04640 1 -- 172.21.15.33:0/2854871662 shutdown_connections 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.158+0000 7fcebaf04640 1 -- 172.21.15.33:0/2854871662 wait complete. 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fcebaf04640 1 Processor -- start 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fcebaf04640 1 -- start start 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fcebaf04640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fceb4102d90 0x7fceb4198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fcebaf04640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fceb4103740 0x7fceb4198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fcebaf04640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceb4108290 0x7fceb419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fcebaf04640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fceb4112080 con 0x7fceb4108290 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fcebaf04640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fceb4111f00 con 0x7fceb4103740 2024-09-16T00:24:58.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fcebaf04640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fceb4112200 con 0x7fceb4102d90 2024-09-16T00:24:58.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fceb8ca0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fceb4102d90 0x7fceb4198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fceb94a1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceb4108290 0x7fceb419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fceb8ca0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fceb4102d90 0x7fceb4198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41236/0 (socket says 172.21.15.33:41236) 2024-09-16T00:24:58.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fceb8ca0640 1 -- 172.21.15.33:0/4131148944 learned_addr learned my addr 172.21.15.33:0/4131148944 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:58.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fceb3fff640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fceb4103740 0x7fceb4198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fceb8ca0640 1 -- 172.21.15.33:0/4131148944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fceb4103740 msgr2=0x7fceb4198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:58.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fceb8ca0640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fceb4103740 0x7fceb4198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fceb8ca0640 1 -- 172.21.15.33:0/4131148944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceb4108290 msgr2=0x7fceb419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:58.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fceb8ca0640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceb4108290 0x7fceb419ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fceb8ca0640 1 -- 172.21.15.33:0/4131148944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb419d5b0 con 0x7fceb4102d90 2024-09-16T00:24:58.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.159+0000 7fceb94a1640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceb4108290 0x7fceb419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:58.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.160+0000 7fceb8ca0640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fceb4102d90 0x7fceb4198600 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcea800e9f0 tx=0x7fcea800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.160+0000 7fceb1ffb640 1 -- 172.21.15.33:0/4131148944 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea800cd80 con 0x7fceb4102d90 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.160+0000 7fcebaf04640 1 -- 172.21.15.33:0/4131148944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb419d8a0 con 0x7fceb4102d90 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.160+0000 7fceb1ffb640 1 -- 172.21.15.33:0/4131148944 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcea8010910 con 0x7fceb4102d90 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.160+0000 7fceb1ffb640 1 -- 172.21.15.33:0/4131148944 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea80188f0 con 0x7fceb4102d90 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.160+0000 7fcebaf04640 1 -- 172.21.15.33:0/4131148944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb41a5180 con 0x7fceb4102d90 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.163+0000 7fceb1ffb640 1 -- 172.21.15.33:0/4131148944 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fcea8010ac0 con 0x7fceb4102d90 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.163+0000 7fcebd2dd740 1 Processor -- start 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fcebd2dd740 1 -- start start 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fcebd2dd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd7db40 0x555e0dd7c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fcebd2dd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd7ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fcebd2dd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd87370 0x555e0dd89760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fcebd2dd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555e0dce22f0 con 0x555e0dd7c5d0 2024-09-16T00:24:58.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fcebd2dd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555e0dc89cf0 con 0x555e0dd87370 2024-09-16T00:24:58.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fcebd2dd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555e0dcde2b0 con 0x555e0dd7db40 2024-09-16T00:24:58.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb8ca0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd7ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb94a1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd7db40 0x555e0dd7c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb8ca0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd7ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34462/0 (socket says 172.21.15.33:34462) 2024-09-16T00:24:58.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb8ca0640 1 -- 172.21.15.33:0/2578196927 learned_addr learned my addr 172.21.15.33:0/2578196927 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:58.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb94a1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd7db40 0x555e0dd7c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41250/0 (socket says 172.21.15.33:41250) 2024-09-16T00:24:58.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb3fff640 1 --2- 172.21.15.33:0/2578196927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd87370 0x555e0dd89760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb1ffb640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcea40a3240 0x7fcea40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb94a1640 1 -- 172.21.15.33:0/2578196927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd87370 msgr2=0x555e0dd89760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:58.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb94a1640 1 --2- 172.21.15.33:0/2578196927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd87370 0x555e0dd89760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb94a1640 1 -- 172.21.15.33:0/2578196927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 msgr2=0x555e0dd7ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:58.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb94a1640 1 --2- 172.21.15.33:0/2578196927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd7ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb94a1640 1 -- 172.21.15.33:0/2578196927 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e0dc6e3a0 con 0x555e0dd7db40 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.164+0000 7fceb3fff640 1 --2- 172.21.15.33:0/2578196927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd87370 0x555e0dd89760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.165+0000 7fceb1ffb640 1 -- 172.21.15.33:0/4131148944 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fcea80d01d0 con 0x7fceb4102d90 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.165+0000 7fceb94a1640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcea40a3240 0x7fcea40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.165+0000 7fceb94a1640 1 --2- 172.21.15.33:0/2578196927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd7db40 0x555e0dd7c090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fceac005fd0 tx=0x7fceac011bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43c02a7eeb4c0e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.165+0000 7fce9effd640 1 -- 172.21.15.33:0/2578196927 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac020070 con 0x555e0dd7db40 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.165+0000 7fce9effd640 1 -- 172.21.15.33:0/2578196927 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fceac004050 con 0x555e0dd7db40 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.165+0000 7fce9effd640 1 -- 172.21.15.33:0/2578196927 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac01b8d0 con 0x555e0dd7db40 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.166+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2578196927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd7db40 msgr2=0x555e0dd7c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.166+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2578196927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd7db40 0x555e0dd7c090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fceac005fd0 tx=0x7fceac011bf0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.166+0000 7fceb94a1640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcea40a3240 0x7fcea40a5700 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fceac01d000 tx=0x7fceac01bf00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.166+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2578196927 shutdown_connections 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.166+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2578196927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd87370 0x555e0dd89760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.166+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2578196927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd7ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.166+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2578196927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd7db40 0x555e0dd7c090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.166+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2578196927 >> 172.21.15.33:0/2578196927 conn(0x555e0dd45310 msgr2=0x555e0dd78c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:58.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.166+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2578196927 shutdown_connections 2024-09-16T00:24:58.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.166+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2578196927 wait complete. 2024-09-16T00:24:58.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.166+0000 7fcebd2dd740 1 -- start start 2024-09-16T00:24:58.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.166+0000 7fcebd2dd740 1 Processor -- start 2024-09-16T00:24:58.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fcebd2dd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd86cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fcebd2dd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7db40 0x555e0dd83f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fcebd2dd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd844d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fcebd2dd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555e0dd8e480 con 0x555e0dd7c5d0 2024-09-16T00:24:58.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fcebd2dd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555e0dc9d060 con 0x555e0dd7db40 2024-09-16T00:24:58.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fcebd2dd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555e0dd8e600 con 0x555e0dd87370 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fcebd2dd740 10 client.0 _ll_register_callbacks cb 0x555e0d9f08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fceb3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7db40 0x555e0dd83f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fceb3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7db40 0x555e0dd83f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:47316/0 (socket says 172.21.15.33:47316) 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fceb3fff640 1 -- 172.21.15.33:0/278324735 learned_addr learned my addr 172.21.15.33:0/278324735 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fceb94a1640 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd844d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fceb8ca0640 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd86cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fceb3fff640 1 -- 172.21.15.33:0/278324735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 msgr2=0x555e0dd844d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fceb3fff640 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd844d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.167+0000 7fceb3fff640 1 -- 172.21.15.33:0/278324735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 msgr2=0x555e0dd86cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.168+0000 7fceb3fff640 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd86cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.168+0000 7fceb3fff640 1 -- 172.21.15.33:0/278324735 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e0da533c0 con 0x555e0dd7db40 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.168+0000 7fceb94a1640 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd844d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:58.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.168+0000 7fceb3fff640 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7db40 0x555e0dd83f70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009930 tx=0x7fcea0005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31e48fb3e19b5c82 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.168+0000 7fce93fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.168+0000 7fce93fff640 1 -- 172.21.15.33:0/278324735 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea0035040 con 0x555e0dd7db40 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.168+0000 7fcebd2dd740 1 -- 172.21.15.33:0/278324735 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceac004050 con 0x555e0dd7db40 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.168+0000 7fce93fff640 1 -- 172.21.15.33:0/278324735 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fcea00048c0 con 0x555e0dd7db40 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.168+0000 7fce93fff640 1 -- 172.21.15.33:0/278324735 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcea0005d50 con 0x555e0dd7db40 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.168+0000 7fcebd2dd740 10 client.26300 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.168+0000 7fcebd2dd740 1 -- 172.21.15.33:0/278324735 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555e0dd99ee0 con 0x555e0dd7db40 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.169+0000 7fce93fff640 1 -- 172.21.15.33:0/278324735 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fceac004050 con 0x555e0dd7db40 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.169+0000 7fce93fff640 1 -- 172.21.15.33:0/278324735 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fcea0003b70 con 0x555e0dd7db40 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.169+0000 7fcebd2dd740 10 client.26300 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.169+0000 7fcebd2dd740 10 client.26300 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.169+0000 7fcebd2dd740 20 client.26300 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:58.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.169+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555e0dd9b3a0 0x555e0dd9d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.169+0000 7fcebd2dd740 4 client.26300 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:24:58.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.169+0000 7fcebd2dd740 1 -- 172.21.15.33:0/278324735 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e0dcef860 con 0x555e0dd9b3a0 2024-09-16T00:24:58.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.169+0000 7fceb8ca0640 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555e0dd9b3a0 0x555e0dd9d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.170+0000 7fce93fff640 10 client.26300 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:24:58.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.170+0000 7fceb8ca0640 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555e0dd9b3a0 0x555e0dd9d7e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=29beaa1a222927a2 server_cookie=1bda8888e109f0d3 in_seq=0 out_seq=0 2024-09-16T00:24:58.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.171+0000 7fce93fff640 1 -- 172.21.15.33:0/278324735 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555e0dcef860 con 0x555e0dd9b3a0 2024-09-16T00:24:58.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.171+0000 7fce93fff640 10 client.26300 handle_command_reply: tid=0 2024-09-16T00:24:58.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.171+0000 7fcebd2dd740 1 client.26300 shutdown 2024-09-16T00:24:58.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.171+0000 7fcebd2dd740 1 -- 172.21.15.33:0/278324735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7db40 msgr2=0x555e0dd83f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:58.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.171+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7db40 0x555e0dd83f70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009930 tx=0x7fcea0005270 comp rx=0 tx=0).stop 2024-09-16T00:24:58.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.171+0000 7fcebd2dd740 1 -- 172.21.15.33:0/278324735 shutdown_connections 2024-09-16T00:24:58.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.171+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555e0dd9b3a0 0x555e0dd9d7e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.171+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd844d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.171+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7db40 0x555e0dd83f70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.171+0000 7fce93fff640 0 client.26300 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:24:58.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.171+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/278324735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd86cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.171+0000 7fcebd2dd740 1 -- 172.21.15.33:0/278324735 >> 172.21.15.33:0/278324735 conn(0x555e0dd45310 msgr2=0x555e0dd7e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:58.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.172+0000 7fcebd2dd740 1 -- 172.21.15.33:0/278324735 shutdown_connections 2024-09-16T00:24:58.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.172+0000 7fcebd2dd740 1 -- 172.21.15.33:0/278324735 wait complete. 2024-09-16T00:24:58.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.172+0000 7fcebd2dd740 20 client.26300 trim_cache size 0 max 16384 2024-09-16T00:24:58.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fcebd2dd740 1 Processor -- start 2024-09-16T00:24:58.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fcebd2dd740 1 -- start start 2024-09-16T00:24:58.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fcebd2dd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7c5d0 0x555e0dd97720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fcebd2dd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7db40 0x555e0dd83830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fcebd2dd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd83d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fcebd2dd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41af800 con 0x555e0dd7db40 2024-09-16T00:24:58.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fcebd2dd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41ad0d0 con 0x555e0dd7c5d0 2024-09-16T00:24:58.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fcebd2dd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41aaf20 con 0x555e0dd87370 2024-09-16T00:24:58.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fceb8ca0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7c5d0 0x555e0dd97720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fceb8ca0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7c5d0 0x555e0dd97720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:47330/0 (socket says 172.21.15.33:47330) 2024-09-16T00:24:58.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fceb8ca0640 1 -- 172.21.15.33:0/2714578987 learned_addr learned my addr 172.21.15.33:0/2714578987 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:58.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fceb94a1640 1 --2- 172.21.15.33:0/2714578987 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd83d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fceb3fff640 1 --2- 172.21.15.33:0/2714578987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7db40 0x555e0dd83830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.188+0000 7fceb8ca0640 1 -- 172.21.15.33:0/2714578987 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 msgr2=0x555e0dd83d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:58.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.189+0000 7fceb8ca0640 1 --2- 172.21.15.33:0/2714578987 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd83d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.189+0000 7fceb8ca0640 1 -- 172.21.15.33:0/2714578987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7db40 msgr2=0x555e0dd83830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:58.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.189+0000 7fceb8ca0640 1 --2- 172.21.15.33:0/2714578987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7db40 0x555e0dd83830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.189+0000 7fceb8ca0640 1 -- 172.21.15.33:0/2714578987 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e0dd89ad0 con 0x555e0dd7c5d0 2024-09-16T00:24:58.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.189+0000 7fceb8ca0640 1 --2- 172.21.15.33:0/2714578987 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7c5d0 0x555e0dd97720 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcea80987d0 tx=0x7fcea80bc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7395d49495192f3b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:58.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.189+0000 7fce9e7fc640 1 -- 172.21.15.33:0/2714578987 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea80d3e50 con 0x555e0dd7c5d0 2024-09-16T00:24:58.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.189+0000 7fce9e7fc640 1 -- 172.21.15.33:0/2714578987 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcea809ca70 con 0x555e0dd7c5d0 2024-09-16T00:24:58.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.189+0000 7fce9e7fc640 1 -- 172.21.15.33:0/2714578987 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea80989e0 con 0x555e0dd7c5d0 2024-09-16T00:24:58.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.190+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2714578987 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7c5d0 msgr2=0x555e0dd97720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:58.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.190+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2714578987 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7c5d0 0x555e0dd97720 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcea80987d0 tx=0x7fcea80bc040 comp rx=0 tx=0).stop 2024-09-16T00:24:58.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.190+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2714578987 shutdown_connections 2024-09-16T00:24:58.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.190+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2714578987 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd83d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.190+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2714578987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7db40 0x555e0dd83830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.190+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2714578987 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7c5d0 0x555e0dd97720 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.190+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2714578987 >> 172.21.15.33:0/2714578987 conn(0x555e0dd45310 msgr2=0x555e0dd7f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:58.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.190+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2714578987 shutdown_connections 2024-09-16T00:24:58.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.190+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2714578987 wait complete. 2024-09-16T00:24:58.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.190+0000 7fcebd2dd740 1 -- start start 2024-09-16T00:24:58.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.191+0000 7fcebd2dd740 1 Processor -- start 2024-09-16T00:24:58.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.191+0000 7fcebd2dd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd93c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.191+0000 7fcebd2dd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7db40 0x555e0dd94180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.191+0000 7fcebd2dd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd8f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.191+0000 7fceb8ca0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd93c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.191+0000 7fcebd2dd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41ab590 con 0x555e0dd7c5d0 2024-09-16T00:24:58.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.191+0000 7fcebd2dd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41aca00 con 0x555e0dd7db40 2024-09-16T00:24:58.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.191+0000 7fcebd2dd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41abc60 con 0x555e0dd87370 2024-09-16T00:24:58.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.191+0000 7fceb8ca0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd93c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34494/0 (socket says 172.21.15.33:34494) 2024-09-16T00:24:58.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.191+0000 7fceb8ca0640 1 -- 172.21.15.33:0/2484375215 learned_addr learned my addr 172.21.15.33:0/2484375215 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.191+0000 7fceb94a1640 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd8f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.191+0000 7fceb3fff640 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7db40 0x555e0dd94180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fcebd2dd740 10 client.0 _ll_register_callbacks cb 0x7fcea8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fceb8ca0640 1 -- 172.21.15.33:0/2484375215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 msgr2=0x555e0dd8f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fceb8ca0640 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd8f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fceb8ca0640 1 -- 172.21.15.33:0/2484375215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7db40 msgr2=0x555e0dd94180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fceb8ca0640 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7db40 0x555e0dd94180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fceb8ca0640 1 -- 172.21.15.33:0/2484375215 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e0dd8f9f0 con 0x555e0dd7c5d0 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fceb94a1640 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd8f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fceb8ca0640 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd93c40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcea80bc010 tx=0x7fcea8005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c06b5b38925301 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fce93fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fce93fff640 1 -- 172.21.15.33:0/2484375215 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea809c370 con 0x555e0dd7c5d0 2024-09-16T00:24:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fce93fff640 1 -- 172.21.15.33:0/2484375215 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fcea80d5e00 con 0x555e0dd7c5d0 2024-09-16T00:24:58.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2484375215 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcea809ca70 con 0x555e0dd7c5d0 2024-09-16T00:24:58.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fce93fff640 1 -- 172.21.15.33:0/2484375215 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcea80a0720 con 0x555e0dd7c5d0 2024-09-16T00:24:58.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fcebd2dd740 10 client.16788 fetch_fsmap learned FSMap version 30 2024-09-16T00:24:58.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.192+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2484375215 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555e0de0a4a0 con 0x555e0dd7c5d0 2024-09-16T00:24:58.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.193+0000 7fce93fff640 1 -- 172.21.15.33:0/2484375215 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcea809ca70 con 0x555e0dd7c5d0 2024-09-16T00:24:58.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.193+0000 7fce93fff640 1 -- 172.21.15.33:0/2484375215 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fcea809b7c0 con 0x555e0dd7c5d0 2024-09-16T00:24:58.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.193+0000 7fcebd2dd740 10 client.16788 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:24:58.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.193+0000 7fcebd2dd740 10 client.16788 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:24:58.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.193+0000 7fcebd2dd740 20 client.16788 populate_metadata read hostname 'smithi033' 2024-09-16T00:24:58.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.193+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555e0dd9b3a0 0x555e0de0cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:58.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.193+0000 7fcebd2dd740 4 client.16788 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:24:58.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.193+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2484375215 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555e0dce9d70 con 0x555e0dd9b3a0 2024-09-16T00:24:58.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.193+0000 7fceb3fff640 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555e0dd9b3a0 0x555e0de0cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:58.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.193+0000 7fce93fff640 10 client.16788 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:24:58.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.193+0000 7fceb3fff640 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555e0dd9b3a0 0x555e0de0cb20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2b4fdf47e459128a server_cookie=53bc0a706db3f282 in_seq=0 out_seq=0 2024-09-16T00:24:58.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fce93fff640 1 -- 172.21.15.33:0/2484375215 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555e0dce9d70 con 0x555e0dd9b3a0 2024-09-16T00:24:58.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fce93fff640 10 client.16788 handle_command_reply: tid=0 2024-09-16T00:24:58.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fcebd2dd740 1 client.16788 shutdown 2024-09-16T00:24:58.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2484375215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 msgr2=0x555e0dd93c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:58.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd93c40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcea80bc010 tx=0x7fcea8005c70 comp rx=0 tx=0).stop 2024-09-16T00:24:58.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2484375215 shutdown_connections 2024-09-16T00:24:58.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555e0dd9b3a0 0x555e0de0cb20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555e0dd87370 0x555e0dd8f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555e0dd7db40 0x555e0dd94180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fcebd2dd740 1 --2- 172.21.15.33:0/2484375215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e0dd7c5d0 0x555e0dd93c40 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2484375215 >> 172.21.15.33:0/2484375215 conn(0x555e0dd45310 msgr2=0x555e0dd857d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:58.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fce93fff640 0 client.16788 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:24:58.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2484375215 shutdown_connections 2024-09-16T00:24:58.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.194+0000 7fcebd2dd740 1 -- 172.21.15.33:0/2484375215 wait complete. 2024-09-16T00:24:58.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebd2dd740 20 client.16788 trim_cache size 0 max 16384 2024-09-16T00:24:58.200 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:24:58.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebaf04640 1 -- 172.21.15.33:0/4131148944 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcea40a3240 msgr2=0x7fcea40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:58.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebaf04640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcea40a3240 0x7fcea40a5700 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fceac01d000 tx=0x7fceac01bf00 comp rx=0 tx=0).stop 2024-09-16T00:24:58.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebaf04640 1 -- 172.21.15.33:0/4131148944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fceb4102d90 msgr2=0x7fceb4198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:58.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebaf04640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fceb4102d90 0x7fceb4198600 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcea800e9f0 tx=0x7fcea800eeb0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebaf04640 1 -- 172.21.15.33:0/4131148944 shutdown_connections 2024-09-16T00:24:58.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebaf04640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcea40a3240 0x7fcea40a5700 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebaf04640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceb4108290 0x7fceb419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebaf04640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fceb4103740 0x7fceb4198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebaf04640 1 --2- 172.21.15.33:0/4131148944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fceb4102d90 0x7fceb4198600 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:58.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebaf04640 1 -- 172.21.15.33:0/4131148944 >> 172.21.15.33:0/4131148944 conn(0x7fceb40fc7d0 msgr2=0x7fceb4106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:58.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebaf04640 1 -- 172.21.15.33:0/4131148944 shutdown_connections 2024-09-16T00:24:58.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:58.195+0000 7fcebaf04640 1 -- 172.21.15.33:0/4131148944 wait complete. 2024-09-16T00:24:58.667 INFO:tasks.workunit.client.0.smithi033.stdout: 1 612074 10.88 MB/sec execute 246 sec latency 56.599 ms 2024-09-16T00:24:58.983 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:24:59.065 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:24:59.519 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:24:59 smithi033 ceph-mon[31243]: pgmap v585: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 236 op/s 2024-09-16T00:24:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.607+0000 7f4e27333640 1 -- 172.21.15.33:0/2531362413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e20100b20 msgr2=0x7f4e20100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.607+0000 7f4e27333640 1 --2- 172.21.15.33:0/2531362413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e20100b20 0x7f4e20100f00 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009900 tx=0x7f4e1c02a410 comp rx=0 tx=0).stop 2024-09-16T00:24:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.607+0000 7f4e27333640 1 -- 172.21.15.33:0/2531362413 shutdown_connections 2024-09-16T00:24:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.607+0000 7f4e27333640 1 --2- 172.21.15.33:0/2531362413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e2010e500 0x7f4e20110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.607+0000 7f4e27333640 1 --2- 172.21.15.33:0/2531362413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e20101440 0x7f4e2010df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.607+0000 7f4e27333640 1 --2- 172.21.15.33:0/2531362413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e20100b20 0x7f4e20100f00 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.607+0000 7f4e27333640 1 -- 172.21.15.33:0/2531362413 >> 172.21.15.33:0/2531362413 conn(0x7f4e200fc7d0 msgr2=0x7f4e200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.607+0000 7f4e27333640 1 -- 172.21.15.33:0/2531362413 shutdown_connections 2024-09-16T00:24:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.607+0000 7f4e27333640 1 -- 172.21.15.33:0/2531362413 wait complete. 2024-09-16T00:24:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.607+0000 7f4e27333640 1 Processor -- start 2024-09-16T00:24:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.607+0000 7f4e27333640 1 -- start start 2024-09-16T00:24:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e27333640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e20100b20 0x7f4e2019a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e27333640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e20101440 0x7f4e2019ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e27333640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e2010e500 0x7f4e2019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e27333640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4e20114280 con 0x7f4e2010e500 2024-09-16T00:24:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e27333640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4e20114100 con 0x7f4e20101440 2024-09-16T00:24:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e27333640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4e20114400 con 0x7f4e20100b20 2024-09-16T00:24:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e250cf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e20100b20 0x7f4e2019a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e258d0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e2010e500 0x7f4e2019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e250cf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e20100b20 0x7f4e2019a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:41296/0 (socket says 172.21.15.33:41296) 2024-09-16T00:24:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e248ce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e20101440 0x7f4e2019ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e250cf640 1 -- 172.21.15.33:0/2867621207 learned_addr learned my addr 172.21.15.33:0/2867621207 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:24:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e250cf640 1 -- 172.21.15.33:0/2867621207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e20101440 msgr2=0x7f4e2019ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e250cf640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e20101440 0x7f4e2019ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e250cf640 1 -- 172.21.15.33:0/2867621207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e2010e500 msgr2=0x7f4e2019e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e250cf640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e2010e500 0x7f4e2019e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e250cf640 1 -- 172.21.15.33:0/2867621207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2019e730 con 0x7f4e20100b20 2024-09-16T00:24:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e248ce640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e20101440 0x7f4e2019ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e258d0640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e2010e500 0x7f4e2019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:24:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.608+0000 7f4e250cf640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e20100b20 0x7f4e2019a7e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c006010 tx=0x7f4e1c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.609+0000 7f4e167fc640 1 -- 172.21.15.33:0/2867621207 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c03f030 con 0x7f4e20100b20 2024-09-16T00:24:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.609+0000 7f4e167fc640 1 -- 172.21.15.33:0/2867621207 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e1c004440 con 0x7f4e20100b20 2024-09-16T00:24:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.609+0000 7f4e27333640 1 -- 172.21.15.33:0/2867621207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e2019e9c0 con 0x7f4e20100b20 2024-09-16T00:24:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.609+0000 7f4e167fc640 1 -- 172.21.15.33:0/2867621207 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c035070 con 0x7f4e20100b20 2024-09-16T00:24:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.609+0000 7f4e27333640 1 -- 172.21.15.33:0/2867621207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e201a7390 con 0x7f4e20100b20 2024-09-16T00:24:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.610+0000 7f4e27333640 1 -- 172.21.15.33:0/2867621207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e2006a850 con 0x7f4e20100b20 2024-09-16T00:24:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.611+0000 7f4e167fc640 1 -- 172.21.15.33:0/2867621207 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4e1c042030 con 0x7f4e20100b20 2024-09-16T00:24:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.612+0000 7f4e167fc640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4e0c0a3240 0x7f4e0c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:24:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.612+0000 7f4e248ce640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4e0c0a3240 0x7f4e0c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:24:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.612+0000 7f4e167fc640 1 -- 172.21.15.33:0/2867621207 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4e1c0f03f0 con 0x7f4e20100b20 2024-09-16T00:24:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.612+0000 7f4e248ce640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4e0c0a3240 0x7f4e0c0a5700 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4e2010ec60 tx=0x7f4e10005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:24:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.617+0000 7f4e167fc640 1 -- 172.21.15.33:0/2867621207 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4e1c0bc700 con 0x7f4e20100b20 2024-09-16T00:24:59.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 614817 10.90 MB/sec execute 247 sec latency 70.865 ms 2024-09-16T00:24:59.668 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:24:59.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:24:59 smithi083 ceph-mon[35242]: pgmap v585: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 236 op/s 2024-09-16T00:24:59.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:24:59 smithi060 ceph-mon[35185]: pgmap v585: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 236 op/s 2024-09-16T00:24:59.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.849+0000 7f4e27333640 1 -- 172.21.15.33:0/2867621207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e2019bb70 con 0x7f4e20100b20 2024-09-16T00:24:59.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.849+0000 7f4e167fc640 1 -- 172.21.15.33:0/2867621207 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f4e2019bb70 con 0x7f4e20100b20 2024-09-16T00:24:59.851 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:24:59.851 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:24:59.851 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:24:59.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.852+0000 7f4e27333640 1 -- 172.21.15.33:0/2867621207 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4e0c0a3240 msgr2=0x7f4e0c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:59.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.852+0000 7f4e27333640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4e0c0a3240 0x7f4e0c0a5700 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4e2010ec60 tx=0x7f4e10005f90 comp rx=0 tx=0).stop 2024-09-16T00:24:59.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.852+0000 7f4e27333640 1 -- 172.21.15.33:0/2867621207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e20100b20 msgr2=0x7f4e2019a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:24:59.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.852+0000 7f4e27333640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e20100b20 0x7f4e2019a7e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c006010 tx=0x7f4e1c004040 comp rx=0 tx=0).stop 2024-09-16T00:24:59.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.852+0000 7f4e27333640 1 -- 172.21.15.33:0/2867621207 shutdown_connections 2024-09-16T00:24:59.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.852+0000 7f4e27333640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4e0c0a3240 0x7f4e0c0a5700 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:59.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.852+0000 7f4e27333640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e2010e500 0x7f4e2019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:59.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.852+0000 7f4e27333640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e20101440 0x7f4e2019ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:59.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.852+0000 7f4e27333640 1 --2- 172.21.15.33:0/2867621207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e20100b20 0x7f4e2019a7e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:24:59.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.852+0000 7f4e27333640 1 -- 172.21.15.33:0/2867621207 >> 172.21.15.33:0/2867621207 conn(0x7f4e200fc7d0 msgr2=0x7f4e200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:24:59.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.852+0000 7f4e27333640 1 -- 172.21.15.33:0/2867621207 shutdown_connections 2024-09-16T00:24:59.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:24:59.852+0000 7f4e27333640 1 -- 172.21.15.33:0/2867621207 wait complete. 2024-09-16T00:25:00.391 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:00 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2867621207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:25:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.390+0000 7f78386a9640 1 -- 172.21.15.33:0/2025610474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7830101de0 msgr2=0x7f78301021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.390+0000 7f78386a9640 1 --2- 172.21.15.33:0/2025610474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7830101de0 0x7f78301021c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f782c009900 tx=0x7f782c02a410 comp rx=0 tx=0).stop 2024-09-16T00:25:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.390+0000 7f78386a9640 1 -- 172.21.15.33:0/2025610474 shutdown_connections 2024-09-16T00:25:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.390+0000 7f78386a9640 1 --2- 172.21.15.33:0/2025610474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7830106f10 0x7f783010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.390+0000 7f78386a9640 1 --2- 172.21.15.33:0/2025610474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7830102790 0x7f78301067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.390+0000 7f78386a9640 1 --2- 172.21.15.33:0/2025610474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7830101de0 0x7f78301021c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.390+0000 7f78386a9640 1 -- 172.21.15.33:0/2025610474 >> 172.21.15.33:0/2025610474 conn(0x7f78300fd8c0 msgr2=0x7f78300ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.390+0000 7f78386a9640 1 -- 172.21.15.33:0/2025610474 shutdown_connections 2024-09-16T00:25:00.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.390+0000 7f78386a9640 1 -- 172.21.15.33:0/2025610474 wait complete. 2024-09-16T00:25:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.391+0000 7f78386a9640 1 Processor -- start 2024-09-16T00:25:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.391+0000 7f78386a9640 1 -- start start 2024-09-16T00:25:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.391+0000 7f78386a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7830101de0 0x7f78301984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.391+0000 7f78386a9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7830102790 0x7f7830198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.391+0000 7f78386a9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7830106f10 0x7f783019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f78386a9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7830111f70 con 0x7f7830102790 2024-09-16T00:25:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f78386a9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7830111df0 con 0x7f7830101de0 2024-09-16T00:25:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f78386a9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f78301120f0 con 0x7f7830106f10 2024-09-16T00:25:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f7836445640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7830101de0 0x7f78301984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f7835c44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7830102790 0x7f7830198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f7836445640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7830101de0 0x7f78301984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55778/0 (socket says 172.21.15.33:55778) 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f7836445640 1 -- 172.21.15.33:0/3396434398 learned_addr learned my addr 172.21.15.33:0/3396434398 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f7836c46640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7830106f10 0x7f783019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f7835c44640 1 -- 172.21.15.33:0/3396434398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7830106f10 msgr2=0x7f783019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f7835c44640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7830106f10 0x7f783019cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f7835c44640 1 -- 172.21.15.33:0/3396434398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7830101de0 msgr2=0x7f78301984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f7835c44640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7830101de0 0x7f78301984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f7835c44640 1 -- 172.21.15.33:0/3396434398 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783019d480 con 0x7f7830102790 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f7836445640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7830101de0 0x7f78301984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f7835c44640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7830102790 0x7f7830198a10 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f782000d6d0 tx=0x7f782000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f78277fe640 1 -- 172.21.15.33:0/3396434398 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7820017070 con 0x7f7830102790 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f78277fe640 1 -- 172.21.15.33:0/3396434398 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7820009820 con 0x7f7830102790 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f78386a9640 1 -- 172.21.15.33:0/3396434398 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783019d770 con 0x7f7830102790 2024-09-16T00:25:00.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.392+0000 7f78277fe640 1 -- 172.21.15.33:0/3396434398 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7820004d00 con 0x7f7830102790 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.393+0000 7f78386a9640 1 -- 172.21.15.33:0/3396434398 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78301a5050 con 0x7f7830102790 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.395+0000 7f78277fe640 1 -- 172.21.15.33:0/3396434398 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7820010670 con 0x7f7830102790 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.395+0000 7f783aa82740 1 Processor -- start 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.395+0000 7f783aa82740 1 -- start start 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f783aa82740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847e3b40 0x5564847e2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f783aa82740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847e2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f783aa82740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847f1780 0x5564847f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f783aa82740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5564847482f0 con 0x5564847e3b40 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f783aa82740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5564846efcf0 con 0x5564847f1780 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f783aa82740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5564847442b0 con 0x5564847e25d0 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f7836c46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847e3b40 0x5564847e2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f7835c44640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847f1780 0x5564847f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f7836445640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847e2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f7836445640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847e2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52736/0 (socket says 172.21.15.33:52736) 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f7836445640 1 -- 172.21.15.33:0/3698807214 learned_addr learned my addr 172.21.15.33:0/3698807214 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f78277fe640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f781c0a32c0 0x7f781c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f7835c44640 1 -- 172.21.15.33:0/3698807214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 msgr2=0x5564847e2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f7835c44640 1 --2- 172.21.15.33:0/3698807214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847e2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f7835c44640 1 -- 172.21.15.33:0/3698807214 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847e3b40 msgr2=0x5564847e2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f7835c44640 1 --2- 172.21.15.33:0/3698807214 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847e3b40 0x5564847e2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f7835c44640 1 -- 172.21.15.33:0/3698807214 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564846d43a0 con 0x5564847f1780 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f7836c46640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f781c0a32c0 0x7f781c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.396+0000 7f78277fe640 1 -- 172.21.15.33:0/3396434398 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f78200cf810 con 0x7f7830102790 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.397+0000 7f7835c44640 1 --2- 172.21.15.33:0/3698807214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847f1780 0x5564847f3b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f78200cce60 tx=0x7f7820097e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fea8196fa19f81ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.397+0000 7f7824ff9640 1 -- 172.21.15.33:0/3698807214 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7820020030 con 0x5564847f1780 2024-09-16T00:25:00.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.397+0000 7f7824ff9640 1 -- 172.21.15.33:0/3698807214 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f782009ba60 con 0x5564847f1780 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.397+0000 7f7824ff9640 1 -- 172.21.15.33:0/3698807214 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782009b3c0 con 0x5564847f1780 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.397+0000 7f7836c46640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f781c0a32c0 0x7f781c0a5780 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7828005c40 tx=0x7f7828004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 -- 172.21.15.33:0/3698807214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847f1780 msgr2=0x5564847f3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 --2- 172.21.15.33:0/3698807214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847f1780 0x5564847f3b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f78200cce60 tx=0x7f7820097e30 comp rx=0 tx=0).stop 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 -- 172.21.15.33:0/3698807214 shutdown_connections 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 --2- 172.21.15.33:0/3698807214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847f1780 0x5564847f3b70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 --2- 172.21.15.33:0/3698807214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847e2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 --2- 172.21.15.33:0/3698807214 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847e3b40 0x5564847e2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 -- 172.21.15.33:0/3698807214 >> 172.21.15.33:0/3698807214 conn(0x5564847ab310 msgr2=0x5564847dec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 -- 172.21.15.33:0/3698807214 shutdown_connections 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 -- 172.21.15.33:0/3698807214 wait complete. 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 -- start start 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 Processor -- start 2024-09-16T00:25:00.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847f1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847e3b40 0x5564847ee3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 0x5564847ee980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5564847f4420 con 0x5564847f1780 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556484703060 con 0x5564847e3b40 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.398+0000 7f783aa82740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5564847f45a0 con 0x5564847e25d0 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f783aa82740 10 client.0 _ll_register_callbacks cb 0x5564844568c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7836c46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 0x5564847ee980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7836445640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7836c46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 0x5564847ee980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52982/0 (socket says 172.21.15.33:52982) 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7836445640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847f1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52750/0 (socket says 172.21.15.33:52750) 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7836c46640 1 -- 172.21.15.33:0/2872696146 learned_addr learned my addr 172.21.15.33:0/2872696146 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7835c44640 1 --2- 172.21.15.33:0/2872696146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847e3b40 0x5564847ee3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7836c46640 1 -- 172.21.15.33:0/2872696146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 msgr2=0x5564847f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7836c46640 1 --2- 172.21.15.33:0/2872696146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7836c46640 1 -- 172.21.15.33:0/2872696146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847e3b40 msgr2=0x5564847ee3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7836c46640 1 --2- 172.21.15.33:0/2872696146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847e3b40 0x5564847ee3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7836c46640 1 -- 172.21.15.33:0/2872696146 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564844b93c0 con 0x5564847f1780 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7836c46640 1 --2- 172.21.15.33:0/2872696146 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 0x5564847ee980 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f782800fac0 tx=0x7f7828004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d932b375bada73 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7819ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7819ffb640 1 -- 172.21.15.33:0/2872696146 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782802b060 con 0x5564847f1780 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f7819ffb640 1 -- 172.21.15.33:0/2872696146 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f782801d5f0 con 0x5564847f1780 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.399+0000 7f783aa82740 1 -- 172.21.15.33:0/2872696146 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f782009ba60 con 0x5564847f1780 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.400+0000 7f7819ffb640 1 -- 172.21.15.33:0/2872696146 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7828032050 con 0x5564847f1780 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.400+0000 7f783aa82740 10 client.16806 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.400+0000 7f783aa82740 1 -- 172.21.15.33:0/2872696146 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5564847fff20 con 0x5564847f1780 2024-09-16T00:25:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.400+0000 7f7819ffb640 1 -- 172.21.15.33:0/2872696146 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f782009ba60 con 0x5564847f1780 2024-09-16T00:25:00.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.400+0000 7f7819ffb640 1 -- 172.21.15.33:0/2872696146 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f7828026970 con 0x5564847f1780 2024-09-16T00:25:00.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.400+0000 7f783aa82740 10 client.16806 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:00.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.400+0000 7f783aa82740 10 client.16806 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:25:00.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.400+0000 7f783aa82740 20 client.16806 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:00.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.400+0000 7f783aa82740 1 --2- 172.21.15.33:0/2872696146 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5564848013e0 0x5564848037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.400+0000 7f783aa82740 4 client.16806 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:25:00.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.400+0000 7f783aa82740 1 -- 172.21.15.33:0/2872696146 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556484755860 con 0x5564848013e0 2024-09-16T00:25:00.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.400+0000 7f7836445640 1 --2- 172.21.15.33:0/2872696146 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5564848013e0 0x5564848037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f7836445640 1 --2- 172.21.15.33:0/2872696146 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5564848013e0 0x5564848037d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d832b8a073389ff server_cookie=e4c44a1d12ae002d in_seq=0 out_seq=0 2024-09-16T00:25:00.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f7819ffb640 10 client.16806 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:25:00.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f7819ffb640 1 -- 172.21.15.33:0/2872696146 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556484755860 con 0x5564848013e0 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f7819ffb640 10 client.16806 handle_command_reply: tid=0 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f783aa82740 1 client.16806 shutdown 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f783aa82740 1 -- 172.21.15.33:0/2872696146 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 msgr2=0x5564847ee980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f783aa82740 1 --2- 172.21.15.33:0/2872696146 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 0x5564847ee980 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f782800fac0 tx=0x7f7828004ab0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f783aa82740 1 -- 172.21.15.33:0/2872696146 shutdown_connections 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f783aa82740 1 --2- 172.21.15.33:0/2872696146 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 0x5564847ee980 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f783aa82740 1 --2- 172.21.15.33:0/2872696146 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5564848013e0 0x5564848037d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f783aa82740 1 --2- 172.21.15.33:0/2872696146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847e3b40 0x5564847ee3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f783aa82740 1 --2- 172.21.15.33:0/2872696146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f783aa82740 1 -- 172.21.15.33:0/2872696146 >> 172.21.15.33:0/2872696146 conn(0x5564847ab310 msgr2=0x5564847e4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.401+0000 7f7819ffb640 0 client.16806 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.402+0000 7f783aa82740 1 -- 172.21.15.33:0/2872696146 shutdown_connections 2024-09-16T00:25:00.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.402+0000 7f783aa82740 1 -- 172.21.15.33:0/2872696146 wait complete. 2024-09-16T00:25:00.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.402+0000 7f783aa82740 20 client.16806 trim_cache size 0 max 16384 2024-09-16T00:25:00.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.415+0000 7f783aa82740 1 Processor -- start 2024-09-16T00:25:00.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.415+0000 7f783aa82740 1 -- start start 2024-09-16T00:25:00.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.415+0000 7f783aa82740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847dee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.415+0000 7f783aa82740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847e3b40 0x5564847fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.415+0000 7f783aa82740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847f1780 0x556484809080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.415+0000 7f783aa82740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f78301af840 con 0x5564847e3b40 2024-09-16T00:25:00.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.415+0000 7f783aa82740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f78301ad110 con 0x5564847f1780 2024-09-16T00:25:00.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.415+0000 7f783aa82740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f78301aaf60 con 0x5564847e25d0 2024-09-16T00:25:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f7835c44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847e3b40 0x5564847fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f7835c44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847e3b40 0x5564847fd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52984/0 (socket says 172.21.15.33:52984) 2024-09-16T00:25:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f7835c44640 1 -- 172.21.15.33:0/3602453127 learned_addr learned my addr 172.21.15.33:0/3602453127 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f7836445640 1 --2- 172.21.15.33:0/3602453127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847dee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f7836c46640 1 --2- 172.21.15.33:0/3602453127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847f1780 0x556484809080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f7835c44640 1 -- 172.21.15.33:0/3602453127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 msgr2=0x5564847dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f7835c44640 1 --2- 172.21.15.33:0/3602453127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847dee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f7835c44640 1 -- 172.21.15.33:0/3602453127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847f1780 msgr2=0x556484809080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f7835c44640 1 --2- 172.21.15.33:0/3602453127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847f1780 0x556484809080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f7835c44640 1 -- 172.21.15.33:0/3602453127 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564847e5120 con 0x5564847e3b40 2024-09-16T00:25:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f7836c46640 1 --2- 172.21.15.33:0/3602453127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847f1780 0x556484809080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:25:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f7835c44640 1 --2- 172.21.15.33:0/3602453127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847e3b40 0x5564847fd6c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7820000c00 tx=0x7f78200cfb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43855b814260c225 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:00.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f781bfff640 1 -- 172.21.15.33:0/3602453127 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78200b4070 con 0x5564847e3b40 2024-09-16T00:25:00.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f781bfff640 1 -- 172.21.15.33:0/3602453127 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f782009fd60 con 0x5564847e3b40 2024-09-16T00:25:00.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.416+0000 7f781bfff640 1 -- 172.21.15.33:0/3602453127 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7820020040 con 0x5564847e3b40 2024-09-16T00:25:00.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.417+0000 7f783aa82740 1 -- 172.21.15.33:0/3602453127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847e3b40 msgr2=0x5564847fd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:00.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.418+0000 7f783aa82740 1 --2- 172.21.15.33:0/3602453127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847e3b40 0x5564847fd6c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7820000c00 tx=0x7f78200cfb50 comp rx=0 tx=0).stop 2024-09-16T00:25:00.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.418+0000 7f783aa82740 1 -- 172.21.15.33:0/3602453127 shutdown_connections 2024-09-16T00:25:00.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.418+0000 7f783aa82740 1 --2- 172.21.15.33:0/3602453127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847f1780 0x556484809080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.418+0000 7f783aa82740 1 --2- 172.21.15.33:0/3602453127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847e3b40 0x5564847fd6c0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.418+0000 7f783aa82740 1 --2- 172.21.15.33:0/3602453127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847e25d0 0x5564847dee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.418+0000 7f783aa82740 1 -- 172.21.15.33:0/3602453127 >> 172.21.15.33:0/3602453127 conn(0x5564847ab310 msgr2=0x5564847de700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:00.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.418+0000 7f783aa82740 1 -- 172.21.15.33:0/3602453127 shutdown_connections 2024-09-16T00:25:00.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.418+0000 7f783aa82740 1 -- 172.21.15.33:0/3602453127 wait complete. 2024-09-16T00:25:00.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.418+0000 7f783aa82740 1 -- start start 2024-09-16T00:25:00.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.418+0000 7f783aa82740 1 Processor -- start 2024-09-16T00:25:00.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f783aa82740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847e25d0 0x5564847f9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f783aa82740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 0x5564847f9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f783aa82740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847fa2b0 0x5564847f6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f783aa82740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f78301ab5d0 con 0x5564847f1780 2024-09-16T00:25:00.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f7835c44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 0x5564847f9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f783aa82740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f78301aca40 con 0x5564847e25d0 2024-09-16T00:25:00.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f7835c44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 0x5564847f9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53000/0 (socket says 172.21.15.33:53000) 2024-09-16T00:25:00.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f7835c44640 1 -- 172.21.15.33:0/350655598 learned_addr learned my addr 172.21.15.33:0/350655598 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:00.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f7836c46640 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847fa2b0 0x5564847f6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f783aa82740 1 -- 172.21.15.33:0/350655598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f78301abca0 con 0x5564847fa2b0 2024-09-16T00:25:00.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f7836445640 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847e25d0 0x5564847f9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f783aa82740 10 client.0 _ll_register_callbacks cb 0x7f7828009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f7836c46640 1 -- 172.21.15.33:0/350655598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847e25d0 msgr2=0x5564847f9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f7836c46640 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847e25d0 0x5564847f9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f7836c46640 1 -- 172.21.15.33:0/350655598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 msgr2=0x5564847f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.419+0000 7f7836c46640 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 0x5564847f9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.420+0000 7f7836c46640 1 -- 172.21.15.33:0/350655598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564847f7030 con 0x5564847fa2b0 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.420+0000 7f7835c44640 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 0x5564847f9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.420+0000 7f7836445640 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847e25d0 0x5564847f9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.420+0000 7f7836c46640 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847fa2b0 0x5564847f6800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7828026d10 tx=0x7f782800f9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebdd933690653554 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.420+0000 7f7819ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.421+0000 7f7819ffb640 1 -- 172.21.15.33:0/350655598 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782801b410 con 0x5564847fa2b0 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.421+0000 7f7819ffb640 1 -- 172.21.15.33:0/350655598 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7828004b20 con 0x5564847fa2b0 2024-09-16T00:25:00.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.421+0000 7f783aa82740 1 -- 172.21.15.33:0/350655598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f782009fd60 con 0x5564847fa2b0 2024-09-16T00:25:00.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.421+0000 7f7819ffb640 1 -- 172.21.15.33:0/350655598 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7828004d40 con 0x5564847fa2b0 2024-09-16T00:25:00.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.421+0000 7f783aa82740 10 client.26254 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:00.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.421+0000 7f783aa82740 1 -- 172.21.15.33:0/350655598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55648486ca70 con 0x5564847fa2b0 2024-09-16T00:25:00.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.421+0000 7f7819ffb640 1 -- 172.21.15.33:0/350655598 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f782009fd60 con 0x5564847fa2b0 2024-09-16T00:25:00.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.421+0000 7f7819ffb640 1 -- 172.21.15.33:0/350655598 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f782801b980 con 0x5564847fa2b0 2024-09-16T00:25:00.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.422+0000 7f783aa82740 10 client.26254 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:00.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.422+0000 7f783aa82740 10 client.26254 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:25:00.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.422+0000 7f783aa82740 20 client.26254 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:00.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.422+0000 7f783aa82740 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55648486d1f0 0x55648486f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:00.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.422+0000 7f783aa82740 4 client.26254 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:25:00.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.422+0000 7f783aa82740 1 -- 172.21.15.33:0/350655598 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55648474fd70 con 0x55648486d1f0 2024-09-16T00:25:00.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.422+0000 7f7836445640 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55648486d1f0 0x55648486f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:00.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.422+0000 7f7819ffb640 10 client.26254 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:25:00.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.422+0000 7f7836445640 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55648486d1f0 0x55648486f610 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67d01585fd2b7d77 server_cookie=67b7026421218f89 in_seq=0 out_seq=0 2024-09-16T00:25:00.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f7819ffb640 1 -- 172.21.15.33:0/350655598 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55648474fd70 con 0x55648486d1f0 2024-09-16T00:25:00.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f7819ffb640 10 client.26254 handle_command_reply: tid=0 2024-09-16T00:25:00.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f783aa82740 1 client.26254 shutdown 2024-09-16T00:25:00.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f783aa82740 1 -- 172.21.15.33:0/350655598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847fa2b0 msgr2=0x5564847f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:00.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f783aa82740 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847fa2b0 0x5564847f6800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7828026d10 tx=0x7f782800f9f0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f783aa82740 1 -- 172.21.15.33:0/350655598 shutdown_connections 2024-09-16T00:25:00.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f783aa82740 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564847fa2b0 0x5564847f6800 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f783aa82740 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564847f1780 0x5564847f9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f783aa82740 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55648486d1f0 0x55648486f610 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f7819ffb640 0 client.26254 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:25:00.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f783aa82740 1 --2- 172.21.15.33:0/350655598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564847e25d0 0x5564847f9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f783aa82740 1 -- 172.21.15.33:0/350655598 >> 172.21.15.33:0/350655598 conn(0x5564847ab310 msgr2=0x5564847dd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:00.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f783aa82740 1 -- 172.21.15.33:0/350655598 shutdown_connections 2024-09-16T00:25:00.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f783aa82740 1 -- 172.21.15.33:0/350655598 wait complete. 2024-09-16T00:25:00.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.423+0000 7f783aa82740 20 client.26254 trim_cache size 0 max 16384 2024-09-16T00:25:00.428 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:25:00.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.424+0000 7f78386a9640 1 -- 172.21.15.33:0/3396434398 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f781c0a32c0 msgr2=0x7f781c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:00.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.424+0000 7f78386a9640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f781c0a32c0 0x7f781c0a5780 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7828005c40 tx=0x7f7828004a40 comp rx=0 tx=0).stop 2024-09-16T00:25:00.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.424+0000 7f78386a9640 1 -- 172.21.15.33:0/3396434398 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7830102790 msgr2=0x7f7830198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:00.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.424+0000 7f78386a9640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7830102790 0x7f7830198a10 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f782000d6d0 tx=0x7f782000db90 comp rx=0 tx=0).stop 2024-09-16T00:25:00.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.424+0000 7f78386a9640 1 -- 172.21.15.33:0/3396434398 shutdown_connections 2024-09-16T00:25:00.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.424+0000 7f78386a9640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f781c0a32c0 0x7f781c0a5780 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.424+0000 7f78386a9640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7830106f10 0x7f783019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.424+0000 7f78386a9640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7830102790 0x7f7830198a10 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.424+0000 7f78386a9640 1 --2- 172.21.15.33:0/3396434398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7830101de0 0x7f78301984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:00.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.424+0000 7f78386a9640 1 -- 172.21.15.33:0/3396434398 >> 172.21.15.33:0/3396434398 conn(0x7f78300fd8c0 msgr2=0x7f78300fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:00.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.424+0000 7f78386a9640 1 -- 172.21.15.33:0/3396434398 shutdown_connections 2024-09-16T00:25:00.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:00.424+0000 7f78386a9640 1 -- 172.21.15.33:0/3396434398 wait complete. 2024-09-16T00:25:00.666 INFO:tasks.workunit.client.0.smithi033.stdout: 1 617624 10.91 MB/sec execute 248 sec latency 41.778 ms 2024-09-16T00:25:00.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:00 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2867621207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:25:00.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:00 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2867621207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:25:01.013 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 143.7 --await-for 45.0' 2024-09-16T00:25:01.013 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 143.7 --await-for 45.0 2024-09-16T00:25:01.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:01 smithi033 ceph-mon[31243]: pgmap v586: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.9 MiB/s wr, 253 op/s 2024-09-16T00:25:01.667 INFO:tasks.workunit.client.0.smithi033.stdout: 1 619267 10.90 MB/sec execute 249 sec latency 91.446 ms 2024-09-16T00:25:01.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:01 smithi083 ceph-mon[35242]: pgmap v586: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.9 MiB/s wr, 253 op/s 2024-09-16T00:25:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:01 smithi060 ceph-mon[35185]: pgmap v586: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.9 MiB/s wr, 253 op/s 2024-09-16T00:25:01.739 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:25:01.813 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:02.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.337+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/378183684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb58103570 msgr2=0x7fbb5810ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:02.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.337+0000 7fbb5d1a0640 1 --2- 172.21.15.33:0/378183684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb58103570 0x7fbb5810ba60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009900 tx=0x7fbb4002a480 comp rx=0 tx=0).stop 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.337+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/378183684 shutdown_connections 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.337+0000 7fbb5d1a0640 1 --2- 172.21.15.33:0/378183684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb5810bff0 0x7fbb5810e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.337+0000 7fbb5d1a0640 1 --2- 172.21.15.33:0/378183684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb58103570 0x7fbb5810ba60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.337+0000 7fbb5d1a0640 1 --2- 172.21.15.33:0/378183684 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb58102c50 0x7fbb58103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.337+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/378183684 >> 172.21.15.33:0/378183684 conn(0x7fbb580fc610 msgr2=0x7fbb580fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.337+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/378183684 shutdown_connections 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.337+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/378183684 wait complete. 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.338+0000 7fbb5d1a0640 1 Processor -- start 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.338+0000 7fbb5d1a0640 1 -- start start 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb5d1a0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb58102c50 0x7fbb58198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb5d1a0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb58103570 0x7fbb58198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb5d1a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb5810bff0 0x7fbb5819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb5d1a0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbb58111e10 con 0x7fbb58103570 2024-09-16T00:25:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb5d1a0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbb58111c90 con 0x7fbb58102c50 2024-09-16T00:25:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb5d1a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbb58111f90 con 0x7fbb5810bff0 2024-09-16T00:25:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb4ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb58103570 0x7fbb58198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb57fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb58102c50 0x7fbb58198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb57fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb58102c50 0x7fbb58198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55834/0 (socket says 172.21.15.33:55834) 2024-09-16T00:25:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb57fff640 1 -- 172.21.15.33:0/1903774299 learned_addr learned my addr 172.21.15.33:0/1903774299 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb5c99f640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb5810bff0 0x7fbb5819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb57fff640 1 -- 172.21.15.33:0/1903774299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb5810bff0 msgr2=0x7fbb5819cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb57fff640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb5810bff0 0x7fbb5819cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb57fff640 1 -- 172.21.15.33:0/1903774299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb58103570 msgr2=0x7fbb58198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb57fff640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb58103570 0x7fbb58198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb57fff640 1 -- 172.21.15.33:0/1903774299 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5819d2c0 con 0x7fbb58102c50 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb4ffff640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb58103570 0x7fbb58198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb5c99f640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb5810bff0 0x7fbb5819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.339+0000 7fbb57fff640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb58102c50 0x7fbb58198310 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009c20 tx=0x7fbb4800b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.340+0000 7fbb55ffb640 1 -- 172.21.15.33:0/1903774299 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4801e070 con 0x7fbb58102c50 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.340+0000 7fbb55ffb640 1 -- 172.21.15.33:0/1903774299 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb480198f0 con 0x7fbb58102c50 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.340+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/1903774299 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5819d5b0 con 0x7fbb58102c50 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.340+0000 7fbb55ffb640 1 -- 172.21.15.33:0/1903774299 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480187f0 con 0x7fbb58102c50 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.340+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/1903774299 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb581007e0 con 0x7fbb58102c50 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.342+0000 7fbb55ffb640 1 -- 172.21.15.33:0/1903774299 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbb48021070 con 0x7fbb58102c50 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.342+0000 7fbb607db740 1 Processor -- start 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.342+0000 7fbb607db740 1 -- start start 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb607db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 0x55f985623ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb607db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985624220 0x55f98562e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb607db740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f98562eae0 0x55f985630ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb55ffb640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbb440a32c0 0x7fbb440a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb607db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f9855892a0 con 0x55f985623900 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb607db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f985530b10 con 0x55f985624220 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb607db740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f985585260 con 0x55f98562eae0 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb5c99f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 0x55f985623ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb5c99f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 0x55f985623ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53032/0 (socket says 172.21.15.33:53032) 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb5c99f640 1 -- 172.21.15.33:0/3452098733 learned_addr learned my addr 172.21.15.33:0/3452098733 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb55ffb640 1 -- 172.21.15.33:0/1903774299 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fbb480d3660 con 0x7fbb58102c50 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb57fff640 1 --2- 172.21.15.33:0/3452098733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985624220 0x55f98562e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb4ffff640 1 --2- 172.21.15.33:0/3452098733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f98562eae0 0x55f985630ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb5c99f640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbb440a32c0 0x7fbb440a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb57fff640 1 -- 172.21.15.33:0/3452098733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f98562eae0 msgr2=0x55f985630ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:02.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb57fff640 1 --2- 172.21.15.33:0/3452098733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f98562eae0 0x55f985630ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb57fff640 1 -- 172.21.15.33:0/3452098733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 msgr2=0x55f985623ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb57fff640 1 --2- 172.21.15.33:0/3452098733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 0x55f985623ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.343+0000 7fbb57fff640 1 -- 172.21.15.33:0/3452098733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9852fab90 con 0x55f985624220 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.344+0000 7fbb57fff640 1 --2- 172.21.15.33:0/3452098733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985624220 0x55f98562e5a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800bb90 tx=0x7fbb4809f390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36b824260bbaa069 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.344+0000 7fbb5c99f640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbb440a32c0 0x7fbb440a5780 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbb500099e0 tx=0x7fbb50009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.344+0000 7fbb4e7fc640 1 -- 172.21.15.33:0/3452098733 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480a3b50 con 0x55f985624220 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.344+0000 7fbb4e7fc640 1 -- 172.21.15.33:0/3452098733 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb480a8410 con 0x55f985624220 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.344+0000 7fbb4e7fc640 1 -- 172.21.15.33:0/3452098733 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480d3280 con 0x55f985624220 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.345+0000 7fbb607db740 1 -- 172.21.15.33:0/3452098733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985624220 msgr2=0x55f98562e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.345+0000 7fbb607db740 1 --2- 172.21.15.33:0/3452098733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985624220 0x55f98562e5a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800bb90 tx=0x7fbb4809f390 comp rx=0 tx=0).stop 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.345+0000 7fbb607db740 1 -- 172.21.15.33:0/3452098733 shutdown_connections 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.345+0000 7fbb607db740 1 --2- 172.21.15.33:0/3452098733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f98562eae0 0x55f985630ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.345+0000 7fbb607db740 1 --2- 172.21.15.33:0/3452098733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985624220 0x55f98562e5a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.345+0000 7fbb607db740 1 --2- 172.21.15.33:0/3452098733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 0x55f985623ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.345+0000 7fbb607db740 1 -- 172.21.15.33:0/3452098733 >> 172.21.15.33:0/3452098733 conn(0x55f98561fe90 msgr2=0x55f985620270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.345+0000 7fbb607db740 1 -- 172.21.15.33:0/3452098733 shutdown_connections 2024-09-16T00:25:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.345+0000 7fbb607db740 1 -- 172.21.15.33:0/3452098733 wait complete. 2024-09-16T00:25:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.345+0000 7fbb607db740 1 -- start start 2024-09-16T00:25:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.345+0000 7fbb607db740 1 Processor -- start 2024-09-16T00:25:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb607db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985623900 0x55f98562bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb607db740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985624220 0x55f98562c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb607db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f98562eae0 0x55f98562c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb607db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f9856316b0 con 0x55f98562eae0 2024-09-16T00:25:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb607db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f9855116e0 con 0x55f985623900 2024-09-16T00:25:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb607db740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f985631830 con 0x55f985624220 2024-09-16T00:25:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb607db740 10 client.0 _ll_register_callbacks cb 0x55f985535a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb57fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985623900 0x55f98562bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb57fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985623900 0x55f98562bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55854/0 (socket says 172.21.15.33:55854) 2024-09-16T00:25:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb57fff640 1 -- 172.21.15.33:0/4002692736 learned_addr learned my addr 172.21.15.33:0/4002692736 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb5c99f640 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f98562eae0 0x55f98562c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb4ffff640 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985624220 0x55f98562c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb57fff640 1 -- 172.21.15.33:0/4002692736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985624220 msgr2=0x55f98562c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb57fff640 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985624220 0x55f98562c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb57fff640 1 -- 172.21.15.33:0/4002692736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f98562eae0 msgr2=0x55f98562c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb57fff640 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f98562eae0 0x55f98562c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb57fff640 1 -- 172.21.15.33:0/4002692736 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f98563acc0 con 0x55f985623900 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb57fff640 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985623900 0x55f98562bae0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbb48000c00 tx=0x7fbb4800bec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=778aaa8603599f99 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.346+0000 7fbb377fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb377fe640 1 -- 172.21.15.33:0/4002692736 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480a3b50 con 0x55f985623900 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb607db740 1 -- 172.21.15.33:0/4002692736 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb480a8410 con 0x55f985623900 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb377fe640 1 -- 172.21.15.33:0/4002692736 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fbb480d3280 con 0x55f985623900 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb377fe640 1 -- 172.21.15.33:0/4002692736 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbb48088ba0 con 0x55f985623900 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb607db740 10 client.26339 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb607db740 1 -- 172.21.15.33:0/4002692736 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f98563d650 con 0x55f985623900 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb377fe640 1 -- 172.21.15.33:0/4002692736 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb480a8410 con 0x55f985623900 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb377fe640 1 -- 172.21.15.33:0/4002692736 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fbb48018990 con 0x55f985623900 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb607db740 10 client.26339 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb607db740 10 client.26339 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb607db740 20 client.26339 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb607db740 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f98563e9a0 0x55f985640d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb607db740 4 client.26339 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.347+0000 7fbb607db740 1 -- 172.21.15.33:0/4002692736 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9855969d0 con 0x55f98563e9a0 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.348+0000 7fbb4ffff640 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f98563e9a0 0x55f985640d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.348+0000 7fbb4ffff640 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f98563e9a0 0x55f985640d90 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8cf21a9ca1d386f server_cookie=b3c0cb1f9db0ea1 in_seq=0 out_seq=0 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.348+0000 7fbb377fe640 10 client.26339 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb377fe640 1 -- 172.21.15.33:0/4002692736 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f9855969d0 con 0x55f98563e9a0 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb377fe640 10 client.26339 handle_command_reply: tid=0 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb607db740 1 client.26339 shutdown 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb607db740 1 -- 172.21.15.33:0/4002692736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985623900 msgr2=0x55f98562bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb607db740 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985623900 0x55f98562bae0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbb48000c00 tx=0x7fbb4800bec0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb607db740 1 -- 172.21.15.33:0/4002692736 shutdown_connections 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb607db740 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f98563e9a0 0x55f985640d90 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb607db740 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f98562eae0 0x55f98562c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb377fe640 0 client.26339 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:02.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb607db740 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985624220 0x55f98562c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb607db740 1 --2- 172.21.15.33:0/4002692736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985623900 0x55f98562bae0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb607db740 1 -- 172.21.15.33:0/4002692736 >> 172.21.15.33:0/4002692736 conn(0x55f98561fe90 msgr2=0x55f985625bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:02.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb607db740 1 -- 172.21.15.33:0/4002692736 shutdown_connections 2024-09-16T00:25:02.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb607db740 1 -- 172.21.15.33:0/4002692736 wait complete. 2024-09-16T00:25:02.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.349+0000 7fbb607db740 20 client.26339 trim_cache size 0 max 16384 2024-09-16T00:25:02.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.368+0000 7fbb607db740 1 Processor -- start 2024-09-16T00:25:02.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.368+0000 7fbb607db740 1 -- start start 2024-09-16T00:25:02.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.368+0000 7fbb607db740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985623900 0x55f98563a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb607db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985624220 0x55f985643db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb607db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f98562eae0 0x55f9856442f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb607db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbb581af710 con 0x55f98562eae0 2024-09-16T00:25:02.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb607db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbb581acfe0 con 0x55f985624220 2024-09-16T00:25:02.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb607db740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbb581aae30 con 0x55f985623900 2024-09-16T00:25:02.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb4ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985624220 0x55f985643db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb5c99f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f98562eae0 0x55f9856442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb57fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985623900 0x55f98563a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb57fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985623900 0x55f98563a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52824/0 (socket says 172.21.15.33:52824) 2024-09-16T00:25:02.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb57fff640 1 -- 172.21.15.33:0/2981043377 learned_addr learned my addr 172.21.15.33:0/2981043377 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:02.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb4ffff640 1 -- 172.21.15.33:0/2981043377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985623900 msgr2=0x55f98563a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:02.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb4ffff640 1 --2- 172.21.15.33:0/2981043377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985623900 0x55f98563a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb4ffff640 1 -- 172.21.15.33:0/2981043377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f98562eae0 msgr2=0x55f9856442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:02.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb4ffff640 1 --2- 172.21.15.33:0/2981043377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f98562eae0 0x55f9856442f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb4ffff640 1 -- 172.21.15.33:0/2981043377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f98561e480 con 0x55f985624220 2024-09-16T00:25:02.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.369+0000 7fbb4ffff640 1 --2- 172.21.15.33:0/2981043377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985624220 0x55f985643db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbb4002aa70 tx=0x7fbb40009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f4951fef0298f3b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:02.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.370+0000 7fbb4dffb640 1 -- 172.21.15.33:0/2981043377 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb40035040 con 0x55f985624220 2024-09-16T00:25:02.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.370+0000 7fbb4dffb640 1 -- 172.21.15.33:0/2981043377 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb4004b070 con 0x55f985624220 2024-09-16T00:25:02.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.370+0000 7fbb4dffb640 1 -- 172.21.15.33:0/2981043377 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb400318a0 con 0x55f985624220 2024-09-16T00:25:02.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.370+0000 7fbb607db740 1 -- 172.21.15.33:0/2981043377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985624220 msgr2=0x55f985643db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:02.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.370+0000 7fbb607db740 1 --2- 172.21.15.33:0/2981043377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985624220 0x55f985643db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbb4002aa70 tx=0x7fbb40009260 comp rx=0 tx=0).stop 2024-09-16T00:25:02.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.371+0000 7fbb607db740 1 -- 172.21.15.33:0/2981043377 shutdown_connections 2024-09-16T00:25:02.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.371+0000 7fbb607db740 1 --2- 172.21.15.33:0/2981043377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f98562eae0 0x55f9856442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.371+0000 7fbb607db740 1 --2- 172.21.15.33:0/2981043377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f985624220 0x55f985643db0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.371+0000 7fbb607db740 1 --2- 172.21.15.33:0/2981043377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985623900 0x55f98563a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.371+0000 7fbb607db740 1 -- 172.21.15.33:0/2981043377 >> 172.21.15.33:0/2981043377 conn(0x55f98561fe90 msgr2=0x55f98562c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:02.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.371+0000 7fbb607db740 1 -- 172.21.15.33:0/2981043377 shutdown_connections 2024-09-16T00:25:02.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.371+0000 7fbb607db740 1 -- 172.21.15.33:0/2981043377 wait complete. 2024-09-16T00:25:02.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.371+0000 7fbb607db740 1 -- start start 2024-09-16T00:25:02.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.371+0000 7fbb607db740 1 Processor -- start 2024-09-16T00:25:02.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb607db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 0x55f985641d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb607db740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985624220 0x55f985642290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb607db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f98562eae0 0x55f9856427d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb57fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 0x55f985641d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb57fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 0x55f985641d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53070/0 (socket says 172.21.15.33:53070) 2024-09-16T00:25:02.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb57fff640 1 -- 172.21.15.33:0/3960484399 learned_addr learned my addr 172.21.15.33:0/3960484399 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:02.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb4ffff640 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985624220 0x55f985642290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb607db740 1 -- 172.21.15.33:0/3960484399 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbb581ab4a0 con 0x55f985623900 2024-09-16T00:25:02.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb607db740 1 -- 172.21.15.33:0/3960484399 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbb581ac910 con 0x55f98562eae0 2024-09-16T00:25:02.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb607db740 1 -- 172.21.15.33:0/3960484399 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbb581abb70 con 0x55f985624220 2024-09-16T00:25:02.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb5c99f640 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f98562eae0 0x55f9856427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb57fff640 1 -- 172.21.15.33:0/3960484399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985624220 msgr2=0x55f985642290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb57fff640 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985624220 0x55f985642290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb57fff640 1 -- 172.21.15.33:0/3960484399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f98562eae0 msgr2=0x55f9856427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb57fff640 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f98562eae0 0x55f9856427d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb57fff640 1 -- 172.21.15.33:0/3960484399 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9856a84e0 con 0x55f985623900 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb607db740 10 client.0 _ll_register_callbacks cb 0x7fbb4809f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.372+0000 7fbb607db740 1 -- 172.21.15.33:0/3960484399 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb4004b070 con 0x55f985623900 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.373+0000 7fbb57fff640 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 0x55f985641d50 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009bf0 tx=0x7fbb48019da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a97cac5dca552d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.373+0000 7fbb377fe640 10 client.16830 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.373+0000 7fbb377fe640 1 -- 172.21.15.33:0/3960484399 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480cf070 con 0x55f985623900 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.373+0000 7fbb377fe640 1 -- 172.21.15.33:0/3960484399 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fbb4809b890 con 0x55f985623900 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.373+0000 7fbb377fe640 1 -- 172.21.15.33:0/3960484399 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbb4809f1b0 con 0x55f985623900 2024-09-16T00:25:02.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.373+0000 7fbb607db740 10 client.16830 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:02.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.373+0000 7fbb607db740 1 -- 172.21.15.33:0/3960484399 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55f9856a9520 con 0x55f985623900 2024-09-16T00:25:02.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.373+0000 7fbb377fe640 1 -- 172.21.15.33:0/3960484399 <== mon.0 v2:172.21.15.33:3300/0 4 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fbb48018990 con 0x55f985623900 2024-09-16T00:25:02.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.373+0000 7fbb607db740 10 client.16830 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:02.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.373+0000 7fbb607db740 10 client.16830 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:25:02.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.374+0000 7fbb607db740 20 client.16830 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:02.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.374+0000 7fbb607db740 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f9856a9a90 0x55f9856abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:02.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.374+0000 7fbb607db740 4 client.16830 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.7, "await_for": 45.0}] 2024-09-16T00:25:02.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.374+0000 7fbb607db740 1 -- 172.21.15.33:0/3960484399 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.7, "await_for": 45.0}) -- 0x55f985590d20 con 0x55f9856a9a90 2024-09-16T00:25:02.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.374+0000 7fbb4ffff640 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f9856a9a90 0x55f9856abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:02.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.374+0000 7fbb377fe640 10 client.16830 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:02.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.374+0000 7fbb4ffff640 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f9856a9a90 0x55f9856abf50 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=728ce939b815d683 server_cookie=7fb8cff87dda1296 in_seq=0 out_seq=0 2024-09-16T00:25:02.397 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:02.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb377fe640 1 -- 172.21.15.33:0/3960484399 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f985590d20 con 0x55f9856a9a90 2024-09-16T00:25:02.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb377fe640 10 client.16830 handle_command_reply: tid=0 2024-09-16T00:25:02.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb607db740 1 client.16830 shutdown 2024-09-16T00:25:02.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb607db740 1 -- 172.21.15.33:0/3960484399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 msgr2=0x55f985641d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:02.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb607db740 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 0x55f985641d50 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009bf0 tx=0x7fbb48019da0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb607db740 1 -- 172.21.15.33:0/3960484399 shutdown_connections 2024-09-16T00:25:02.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb607db740 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f9856a9a90 0x55f9856abf50 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb607db740 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f98562eae0 0x55f9856427d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb607db740 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f985624220 0x55f985642290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb377fe640 0 client.16830 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:02.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb607db740 1 --2- 172.21.15.33:0/3960484399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f985623900 0x55f985641d50 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb607db740 1 -- 172.21.15.33:0/3960484399 >> 172.21.15.33:0/3960484399 conn(0x55f98561fe90 msgr2=0x55f98562db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:02.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb607db740 1 -- 172.21.15.33:0/3960484399 shutdown_connections 2024-09-16T00:25:02.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb607db740 1 -- 172.21.15.33:0/3960484399 wait complete. 2024-09-16T00:25:02.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.484+0000 7fbb607db740 20 client.16830 trim_cache size 0 max 16384 2024-09-16T00:25:02.487 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:25:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:25:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:25:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 26, 2024-09-16T00:25:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:25:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "ed19ea81": { 2024-09-16T00:25:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 26, 2024-09-16T00:25:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:25:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:25:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:25:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:25:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:25:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:25:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 143.7, 2024-09-16T00:25:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:25:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:25:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:25:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:25:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:25:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:25:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:02.489 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:25:02.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.485+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/1903774299 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbb440a32c0 msgr2=0x7fbb440a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:02.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.485+0000 7fbb5d1a0640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbb440a32c0 0x7fbb440a5780 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbb500099e0 tx=0x7fbb50009500 comp rx=0 tx=0).stop 2024-09-16T00:25:02.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.485+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/1903774299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb58102c50 msgr2=0x7fbb58198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:02.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.485+0000 7fbb5d1a0640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb58102c50 0x7fbb58198310 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009c20 tx=0x7fbb4800b690 comp rx=0 tx=0).stop 2024-09-16T00:25:02.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.485+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/1903774299 shutdown_connections 2024-09-16T00:25:02.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.485+0000 7fbb5d1a0640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbb440a32c0 0x7fbb440a5780 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.485+0000 7fbb5d1a0640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb5810bff0 0x7fbb5819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.485+0000 7fbb5d1a0640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb58103570 0x7fbb58198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.485+0000 7fbb5d1a0640 1 --2- 172.21.15.33:0/1903774299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb58102c50 0x7fbb58198310 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:02.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.485+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/1903774299 >> 172.21.15.33:0/1903774299 conn(0x7fbb580fc610 msgr2=0x7fbb580fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:02.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.485+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/1903774299 shutdown_connections 2024-09-16T00:25:02.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:02.485+0000 7fbb5d1a0640 1 -- 172.21.15.33:0/1903774299 wait complete. 2024-09-16T00:25:02.667 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.92 MB/sec execute 250 sec latency 269.467 ms 2024-09-16T00:25:03.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:03 smithi033 ceph-mon[31243]: pgmap v587: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 301 op/s 2024-09-16T00:25:03.622 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ed19ea81', quiesce took 5.6 seconds. Will release after: 18.1 2024-09-16T00:25:03.667 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.88 MB/sec execute 251 sec latency 1269.569 ms 2024-09-16T00:25:03.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:03 smithi083 ceph-mon[35242]: pgmap v587: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 301 op/s 2024-09-16T00:25:03.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:03 smithi060 ceph-mon[35185]: pgmap v587: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 301 op/s 2024-09-16T00:25:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.823+0000 7f56c4f19640 1 -- 172.21.15.33:0/2049191032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56c0101790 msgr2=0x7f56c0101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.823+0000 7f56c4f19640 1 --2- 172.21.15.33:0/2049191032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56c0101790 0x7f56c0101bd0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009900 tx=0x7f56b402a480 comp rx=0 tx=0).stop 2024-09-16T00:25:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.823+0000 7f56c4f19640 1 -- 172.21.15.33:0/2049191032 shutdown_connections 2024-09-16T00:25:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.823+0000 7f56c4f19640 1 --2- 172.21.15.33:0/2049191032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c0102110 0x7f56c010d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.823+0000 7f56c4f19640 1 --2- 172.21.15.33:0/2049191032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56c0101790 0x7f56c0101bd0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.823+0000 7f56c4f19640 1 --2- 172.21.15.33:0/2049191032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c0105ba0 0x7f56c0105f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.823+0000 7f56c4f19640 1 -- 172.21.15.33:0/2049191032 >> 172.21.15.33:0/2049191032 conn(0x7f56c0077210 msgr2=0x7f56c0077620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.823+0000 7f56c4f19640 1 -- 172.21.15.33:0/2049191032 shutdown_connections 2024-09-16T00:25:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.823+0000 7f56c4f19640 1 -- 172.21.15.33:0/2049191032 wait complete. 2024-09-16T00:25:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.824+0000 7f56c4f19640 1 Processor -- start 2024-09-16T00:25:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.824+0000 7f56c4f19640 1 -- start start 2024-09-16T00:25:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56c4f19640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56c0101790 0x7f56c010e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56c4f19640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c0102110 0x7f56c010e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56c4f19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c0105ba0 0x7f56c010edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56c4f19640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f56c0111f80 con 0x7f56c0101790 2024-09-16T00:25:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56c4f19640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f56c0111e00 con 0x7f56c0105ba0 2024-09-16T00:25:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56c4f19640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f56c0112100 con 0x7f56c0102110 2024-09-16T00:25:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56be59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56c0101790 0x7f56c010e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56be59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56c0101790 0x7f56c010e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53082/0 (socket says 172.21.15.33:53082) 2024-09-16T00:25:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56be59c640 1 -- 172.21.15.33:0/896530247 learned_addr learned my addr 172.21.15.33:0/896530247 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56bdd9b640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c0102110 0x7f56c010e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56bed9d640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c0105ba0 0x7f56c010edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56be59c640 1 -- 172.21.15.33:0/896530247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c0102110 msgr2=0x7f56c010e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56be59c640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c0102110 0x7f56c010e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56be59c640 1 -- 172.21.15.33:0/896530247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c0105ba0 msgr2=0x7f56c010edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56be59c640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c0105ba0 0x7f56c010edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56be59c640 1 -- 172.21.15.33:0/896530247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c01a6880 con 0x7f56c0101790 2024-09-16T00:25:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.825+0000 7f56be59c640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56c0101790 0x7f56c010e350 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f56b000d950 tx=0x7f56b000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.826+0000 7f56af7fe640 1 -- 172.21.15.33:0/896530247 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0017070 con 0x7f56c0101790 2024-09-16T00:25:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.826+0000 7f56af7fe640 1 -- 172.21.15.33:0/896530247 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56b0009c30 con 0x7f56c0101790 2024-09-16T00:25:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.826+0000 7f56c4f19640 1 -- 172.21.15.33:0/896530247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c01a6b70 con 0x7f56c0101790 2024-09-16T00:25:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.826+0000 7f56af7fe640 1 -- 172.21.15.33:0/896530247 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b000f6b0 con 0x7f56c0101790 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.826+0000 7f56c4f19640 1 -- 172.21.15.33:0/896530247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c01a7080 con 0x7f56c0101790 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.828+0000 7f56af7fe640 1 -- 172.21.15.33:0/896530247 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f56b00109d0 con 0x7f56c0101790 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.828+0000 7f56c72f2740 1 Processor -- start 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56c72f2740 1 -- start start 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56c72f2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e40b40 0x563014e3f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56c72f2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e3f5d0 0x563014e3fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56c72f2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e4c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56c72f2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563014da52f0 con 0x563014e4a370 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56c72f2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x563014d4ccf0 con 0x563014e3f5d0 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56c72f2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563014da12b0 con 0x563014e40b40 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56af7fe640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f56a80a3240 0x7f56a80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56be59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e3f5d0 0x563014e3fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56bdd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e4c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56bdd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e4c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53086/0 (socket says 172.21.15.33:53086) 2024-09-16T00:25:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56be59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e3f5d0 0x563014e3fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55902/0 (socket says 172.21.15.33:55902) 2024-09-16T00:25:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56bed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e40b40 0x563014e3f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56bdd9b640 1 -- 172.21.15.33:0/1840888595 learned_addr learned my addr 172.21.15.33:0/1840888595 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.829+0000 7f56bed9d640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f56a80a3240 0x7f56a80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.830+0000 7f56af7fe640 1 -- 172.21.15.33:0/896530247 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f56b00cf880 con 0x7f56c0101790 2024-09-16T00:25:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.830+0000 7f56bed9d640 1 -- 172.21.15.33:0/1840888595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e3f5d0 msgr2=0x563014e3fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.830+0000 7f56bed9d640 1 --2- 172.21.15.33:0/1840888595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e3f5d0 0x563014e3fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.830+0000 7f56bed9d640 1 -- 172.21.15.33:0/1840888595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 msgr2=0x563014e4c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.830+0000 7f56bed9d640 1 --2- 172.21.15.33:0/1840888595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e4c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.830+0000 7f56bed9d640 1 -- 172.21.15.33:0/1840888595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563014d313a0 con 0x563014e40b40 2024-09-16T00:25:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.830+0000 7f56bdd9b640 1 --2- 172.21.15.33:0/1840888595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e4c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.830+0000 7f56bed9d640 1 --2- 172.21.15.33:0/1840888595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e40b40 0x563014e3f090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f56b8005fd0 tx=0x7f56b8012de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2343b9ec59b829b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.830+0000 7f56acff9640 1 -- 172.21.15.33:0/1840888595 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b801db50 con 0x563014e40b40 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.830+0000 7f56acff9640 1 -- 172.21.15.33:0/1840888595 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56b8022070 con 0x563014e40b40 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.830+0000 7f56acff9640 1 -- 172.21.15.33:0/1840888595 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8026da0 con 0x563014e40b40 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.831+0000 7f56c72f2740 1 -- 172.21.15.33:0/1840888595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e40b40 msgr2=0x563014e3f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.831+0000 7f56bed9d640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f56a80a3240 0x7f56a80a5700 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f56b8021000 tx=0x7f56b801f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.831+0000 7f56c72f2740 1 --2- 172.21.15.33:0/1840888595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e40b40 0x563014e3f090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f56b8005fd0 tx=0x7f56b8012de0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.831+0000 7f56c72f2740 1 -- 172.21.15.33:0/1840888595 shutdown_connections 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.831+0000 7f56c72f2740 1 --2- 172.21.15.33:0/1840888595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e4c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.831+0000 7f56c72f2740 1 --2- 172.21.15.33:0/1840888595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e3f5d0 0x563014e3fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.831+0000 7f56c72f2740 1 --2- 172.21.15.33:0/1840888595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e40b40 0x563014e3f090 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.831+0000 7f56c72f2740 1 -- 172.21.15.33:0/1840888595 >> 172.21.15.33:0/1840888595 conn(0x563014e08310 msgr2=0x563014e3bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.831+0000 7f56c72f2740 1 -- 172.21.15.33:0/1840888595 shutdown_connections 2024-09-16T00:25:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.831+0000 7f56c72f2740 1 -- 172.21.15.33:0/1840888595 wait complete. 2024-09-16T00:25:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.831+0000 7f56c72f2740 1 -- start start 2024-09-16T00:25:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56c72f2740 1 Processor -- start 2024-09-16T00:25:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56c72f2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e49d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56c72f2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e40b40 0x563014e47000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56c72f2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e4a370 0x563014e475d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56c72f2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563014e4d070 con 0x563014e40b40 2024-09-16T00:25:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56c72f2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x563014d60060 con 0x563014e4a370 2024-09-16T00:25:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56c72f2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563014e4d1f0 con 0x563014e3f5d0 2024-09-16T00:25:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56c72f2740 10 client.0 _ll_register_callbacks cb 0x563014ab38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56bdd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e40b40 0x563014e47000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56be59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e49d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56bdd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e40b40 0x563014e47000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53100/0 (socket says 172.21.15.33:53100) 2024-09-16T00:25:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56bdd9b640 1 -- 172.21.15.33:0/417753679 learned_addr learned my addr 172.21.15.33:0/417753679 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56bed9d640 1 --2- 172.21.15.33:0/417753679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e4a370 0x563014e475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56be59c640 1 -- 172.21.15.33:0/417753679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e4a370 msgr2=0x563014e475d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56be59c640 1 --2- 172.21.15.33:0/417753679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e4a370 0x563014e475d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56be59c640 1 -- 172.21.15.33:0/417753679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e40b40 msgr2=0x563014e47000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56be59c640 1 --2- 172.21.15.33:0/417753679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e40b40 0x563014e47000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.832+0000 7f56be59c640 1 -- 172.21.15.33:0/417753679 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563014b163c0 con 0x563014e3f5d0 2024-09-16T00:25:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56be59c640 1 --2- 172.21.15.33:0/417753679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e49d50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f56b00984a0 tx=0x7f56b0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17714181bb42cf47 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:25:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56a5ffb640 1 -- 172.21.15.33:0/417753679 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0020070 con 0x563014e3f5d0 2024-09-16T00:25:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56a5ffb640 1 -- 172.21.15.33:0/417753679 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f56b00860d0 con 0x563014e3f5d0 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56c72f2740 1 -- 172.21.15.33:0/417753679 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56b8022070 con 0x563014e3f5d0 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56a5ffb640 1 -- 172.21.15.33:0/417753679 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f56b00862d0 con 0x563014e3f5d0 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56c72f2740 10 client.26284 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56c72f2740 1 -- 172.21.15.33:0/417753679 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563014e58b70 con 0x563014e3f5d0 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56a5ffb640 1 -- 172.21.15.33:0/417753679 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56b8022070 con 0x563014e3f5d0 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56a5ffb640 1 -- 172.21.15.33:0/417753679 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f56b009a990 con 0x563014e3f5d0 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56c72f2740 10 client.26284 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56c72f2740 10 client.26284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.833+0000 7f56c72f2740 20 client.26284 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.834+0000 7f56c72f2740 1 --2- 172.21.15.33:0/417753679 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563014e5a030 0x563014e5c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.834+0000 7f56c72f2740 4 client.26284 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.834+0000 7f56c72f2740 1 -- 172.21.15.33:0/417753679 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563014db2860 con 0x563014e5a030 2024-09-16T00:25:03.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.834+0000 7f56bdd9b640 1 --2- 172.21.15.33:0/417753679 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563014e5a030 0x563014e5c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.835+0000 7f56a5ffb640 10 client.26284 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.835+0000 7f56bdd9b640 1 --2- 172.21.15.33:0/417753679 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563014e5a030 0x563014e5c420 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=409275d99ca5a634 server_cookie=379cd4cdd43ba9e4 in_seq=0 out_seq=0 2024-09-16T00:25:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.835+0000 7f56a5ffb640 1 -- 172.21.15.33:0/417753679 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563014db2860 con 0x563014e5a030 2024-09-16T00:25:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.835+0000 7f56a5ffb640 10 client.26284 handle_command_reply: tid=0 2024-09-16T00:25:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56c72f2740 1 client.26284 shutdown 2024-09-16T00:25:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56c72f2740 1 -- 172.21.15.33:0/417753679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 msgr2=0x563014e49d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56c72f2740 1 --2- 172.21.15.33:0/417753679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e49d50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f56b00984a0 tx=0x7f56b0005f90 comp rx=0 tx=0).stop 2024-09-16T00:25:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56c72f2740 1 -- 172.21.15.33:0/417753679 shutdown_connections 2024-09-16T00:25:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56c72f2740 1 --2- 172.21.15.33:0/417753679 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563014e5a030 0x563014e5c420 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56c72f2740 1 --2- 172.21.15.33:0/417753679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e4a370 0x563014e475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56c72f2740 1 --2- 172.21.15.33:0/417753679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e40b40 0x563014e47000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56c72f2740 1 --2- 172.21.15.33:0/417753679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e49d50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56c72f2740 1 -- 172.21.15.33:0/417753679 >> 172.21.15.33:0/417753679 conn(0x563014e08310 msgr2=0x563014e41960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56a5ffb640 0 client.26284 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56c72f2740 1 -- 172.21.15.33:0/417753679 shutdown_connections 2024-09-16T00:25:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56c72f2740 1 -- 172.21.15.33:0/417753679 wait complete. 2024-09-16T00:25:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.836+0000 7f56c72f2740 20 client.26284 trim_cache size 0 max 16384 2024-09-16T00:25:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.849+0000 7f56c72f2740 1 Processor -- start 2024-09-16T00:25:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.849+0000 7f56c72f2740 1 -- start start 2024-09-16T00:25:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56c72f2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e3be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56c72f2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e40b40 0x563014e56340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56c72f2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e42120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56c72f2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f56c01b3b80 con 0x563014e4a370 2024-09-16T00:25:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56c72f2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f56c01b1450 con 0x563014e40b40 2024-09-16T00:25:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56c72f2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f56c01af2a0 con 0x563014e3f5d0 2024-09-16T00:25:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56bed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e42120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56bed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e42120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53108/0 (socket says 172.21.15.33:53108) 2024-09-16T00:25:03.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56bed9d640 1 -- 172.21.15.33:0/1423571787 learned_addr learned my addr 172.21.15.33:0/1423571787 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56be59c640 1 --2- 172.21.15.33:0/1423571787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e3be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56bdd9b640 1 --2- 172.21.15.33:0/1423571787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e40b40 0x563014e56340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56bed9d640 1 -- 172.21.15.33:0/1423571787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 msgr2=0x563014e3be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56bed9d640 1 --2- 172.21.15.33:0/1423571787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e3be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56bed9d640 1 -- 172.21.15.33:0/1423571787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e40b40 msgr2=0x563014e56340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56bed9d640 1 --2- 172.21.15.33:0/1423571787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e40b40 0x563014e56340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56bed9d640 1 -- 172.21.15.33:0/1423571787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563014e3c960 con 0x563014e4a370 2024-09-16T00:25:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56be59c640 1 --2- 172.21.15.33:0/1423571787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e3be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:25:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56bed9d640 1 --2- 172.21.15.33:0/1423571787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e42120 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009b90 tx=0x7f56b8009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc7ac85b502dc12 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.850+0000 7f56a7fff640 1 -- 172.21.15.33:0/1423571787 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b802a040 con 0x563014e4a370 2024-09-16T00:25:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.851+0000 7f56a7fff640 1 -- 172.21.15.33:0/1423571787 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56b8034070 con 0x563014e4a370 2024-09-16T00:25:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.851+0000 7f56a7fff640 1 -- 172.21.15.33:0/1423571787 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8005040 con 0x563014e4a370 2024-09-16T00:25:03.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.851+0000 7f56c72f2740 1 -- 172.21.15.33:0/1423571787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 msgr2=0x563014e42120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:03.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.851+0000 7f56c72f2740 1 --2- 172.21.15.33:0/1423571787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e42120 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009b90 tx=0x7f56b8009a00 comp rx=0 tx=0).stop 2024-09-16T00:25:03.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.851+0000 7f56c72f2740 1 -- 172.21.15.33:0/1423571787 shutdown_connections 2024-09-16T00:25:03.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.851+0000 7f56c72f2740 1 --2- 172.21.15.33:0/1423571787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e42120 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.851+0000 7f56c72f2740 1 --2- 172.21.15.33:0/1423571787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e40b40 0x563014e56340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.851+0000 7f56c72f2740 1 --2- 172.21.15.33:0/1423571787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e3be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.851+0000 7f56c72f2740 1 -- 172.21.15.33:0/1423571787 >> 172.21.15.33:0/1423571787 conn(0x563014e08310 msgr2=0x563014e3b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:03.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.851+0000 7f56c72f2740 1 -- 172.21.15.33:0/1423571787 shutdown_connections 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.851+0000 7f56c72f2740 1 -- 172.21.15.33:0/1423571787 wait complete. 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.851+0000 7f56c72f2740 1 -- start start 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56c72f2740 1 Processor -- start 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56c72f2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e52690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56c72f2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e40b40 0x563014e52bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56c72f2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e43830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56c72f2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f56c01af910 con 0x563014e4a370 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56c72f2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f56c01b0d80 con 0x563014e40b40 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56c72f2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f56c01affe0 con 0x563014e3f5d0 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56c72f2740 10 client.0 _ll_register_callbacks cb 0x7f56b009b8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56bed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e43830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56be59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e52690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56bed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e43830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53122/0 (socket says 172.21.15.33:53122) 2024-09-16T00:25:03.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56bdd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e40b40 0x563014e52bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56bed9d640 1 -- 172.21.15.33:0/2633644821 learned_addr learned my addr 172.21.15.33:0/2633644821 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56bed9d640 1 -- 172.21.15.33:0/2633644821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 msgr2=0x563014e52690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56bed9d640 1 --2- 172.21.15.33:0/2633644821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e52690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56bed9d640 1 -- 172.21.15.33:0/2633644821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e40b40 msgr2=0x563014e52bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56bed9d640 1 --2- 172.21.15.33:0/2633644821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e40b40 0x563014e52bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56bed9d640 1 -- 172.21.15.33:0/2633644821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563014e43ec0 con 0x563014e4a370 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.852+0000 7f56bdd9b640 1 --2- 172.21.15.33:0/2633644821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e40b40 0x563014e52bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56bed9d640 1 --2- 172.21.15.33:0/2633644821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e43830 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f56b8012e20 tx=0x7f56b8009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ee6d3b82d2550e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56a5ffb640 1 -- 172.21.15.33:0/2633644821 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b802a070 con 0x563014e4a370 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56a5ffb640 1 -- 172.21.15.33:0/2633644821 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f56b801d950 con 0x563014e4a370 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56c72f2740 1 -- 172.21.15.33:0/2633644821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56b8034070 con 0x563014e4a370 2024-09-16T00:25:03.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56a5ffb640 1 -- 172.21.15.33:0/2633644821 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f56b8004a00 con 0x563014e4a370 2024-09-16T00:25:03.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56c72f2740 10 client.16848 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:03.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56c72f2740 1 -- 172.21.15.33:0/2633644821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563014ec45a0 con 0x563014e4a370 2024-09-16T00:25:03.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56a5ffb640 1 -- 172.21.15.33:0/2633644821 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56b8034070 con 0x563014e4a370 2024-09-16T00:25:03.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56a5ffb640 1 -- 172.21.15.33:0/2633644821 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f56b800f040 con 0x563014e4a370 2024-09-16T00:25:03.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56c72f2740 10 client.16848 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:03.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56c72f2740 10 client.16848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:03.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56c72f2740 20 client.16848 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:03.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56c72f2740 1 --2- 172.21.15.33:0/2633644821 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563014ec4b20 0x563014ec6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:03.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.853+0000 7f56c72f2740 4 client.16848 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:25:03.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.854+0000 7f56c72f2740 1 -- 172.21.15.33:0/2633644821 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563014dacd70 con 0x563014ec4b20 2024-09-16T00:25:03.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.854+0000 7f56be59c640 1 --2- 172.21.15.33:0/2633644821 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563014ec4b20 0x563014ec6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:03.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.854+0000 7f56a5ffb640 10 client.16848 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:03.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.854+0000 7f56be59c640 1 --2- 172.21.15.33:0/2633644821 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563014ec4b20 0x563014ec6f10 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=967556a489439eed server_cookie=ef03f4e91d7bcda8 in_seq=0 out_seq=0 2024-09-16T00:25:03.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.854+0000 7f56a5ffb640 1 -- 172.21.15.33:0/2633644821 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563014dacd70 con 0x563014ec4b20 2024-09-16T00:25:03.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.854+0000 7f56a5ffb640 10 client.16848 handle_command_reply: tid=0 2024-09-16T00:25:03.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.854+0000 7f56c72f2740 1 client.16848 shutdown 2024-09-16T00:25:03.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.854+0000 7f56c72f2740 1 -- 172.21.15.33:0/2633644821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 msgr2=0x563014e43830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:03.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.854+0000 7f56c72f2740 1 --2- 172.21.15.33:0/2633644821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e43830 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f56b8012e20 tx=0x7f56b8009760 comp rx=0 tx=0).stop 2024-09-16T00:25:03.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.854+0000 7f56c72f2740 1 -- 172.21.15.33:0/2633644821 shutdown_connections 2024-09-16T00:25:03.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.854+0000 7f56c72f2740 1 --2- 172.21.15.33:0/2633644821 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563014ec4b20 0x563014ec6f10 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56a5ffb640 0 client.16848 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:03.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.854+0000 7f56c72f2740 1 --2- 172.21.15.33:0/2633644821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563014e4a370 0x563014e43830 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c72f2740 1 --2- 172.21.15.33:0/2633644821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563014e40b40 0x563014e52bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c72f2740 1 --2- 172.21.15.33:0/2633644821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563014e3f5d0 0x563014e52690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c72f2740 1 -- 172.21.15.33:0/2633644821 >> 172.21.15.33:0/2633644821 conn(0x563014e08310 msgr2=0x563014e3a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:03.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c72f2740 1 -- 172.21.15.33:0/2633644821 shutdown_connections 2024-09-16T00:25:03.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c72f2740 1 -- 172.21.15.33:0/2633644821 wait complete. 2024-09-16T00:25:03.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c72f2740 20 client.16848 trim_cache size 0 max 16384 2024-09-16T00:25:03.862 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:25:03.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c4f19640 1 -- 172.21.15.33:0/896530247 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f56a80a3240 msgr2=0x7f56a80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:03.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c4f19640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f56a80a3240 0x7f56a80a5700 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f56b8021000 tx=0x7f56b801f000 comp rx=0 tx=0).stop 2024-09-16T00:25:03.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c4f19640 1 -- 172.21.15.33:0/896530247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56c0101790 msgr2=0x7f56c010e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:03.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c4f19640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56c0101790 0x7f56c010e350 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f56b000d950 tx=0x7f56b000de10 comp rx=0 tx=0).stop 2024-09-16T00:25:03.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c4f19640 1 -- 172.21.15.33:0/896530247 shutdown_connections 2024-09-16T00:25:03.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c4f19640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f56a80a3240 0x7f56a80a5700 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c4f19640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c0105ba0 0x7f56c010edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c4f19640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c0102110 0x7f56c010e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c4f19640 1 --2- 172.21.15.33:0/896530247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56c0101790 0x7f56c010e350 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:03.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.855+0000 7f56c4f19640 1 -- 172.21.15.33:0/896530247 >> 172.21.15.33:0/896530247 conn(0x7f56c0077210 msgr2=0x7f56c00ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:03.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.856+0000 7f56c4f19640 1 -- 172.21.15.33:0/896530247 shutdown_connections 2024-09-16T00:25:03.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:03.856+0000 7f56c4f19640 1 -- 172.21.15.33:0/896530247 wait complete. 2024-09-16T00:25:04.667 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.83 MB/sec execute 252 sec latency 2269.671 ms 2024-09-16T00:25:05.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:05 smithi033 ceph-mon[31243]: pgmap v588: 129 pgs: 129 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 273 op/s 2024-09-16T00:25:05.667 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.79 MB/sec execute 253 sec latency 3269.748 ms 2024-09-16T00:25:05.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:05 smithi083 ceph-mon[35242]: pgmap v588: 129 pgs: 129 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 273 op/s 2024-09-16T00:25:05.705 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:25:05.705 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:25:05.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:05 smithi060 ceph-mon[35185]: pgmap v588: 129 pgs: 129 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 273 op/s 2024-09-16T00:25:05.966 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:06.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.514+0000 7f910dbdf640 1 -- 172.21.15.33:0/3807472693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9108106a80 msgr2=0x7f9108106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:06.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.514+0000 7f910dbdf640 1 --2- 172.21.15.33:0/3807472693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9108106a80 0x7f9108106e60 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009cd0 tx=0x7f90f802a8e0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.514+0000 7f910dbdf640 1 -- 172.21.15.33:0/3807472693 shutdown_connections 2024-09-16T00:25:06.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.514+0000 7f910dbdf640 1 --2- 172.21.15.33:0/3807472693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9108101420 0x7f910810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.514+0000 7f910dbdf640 1 --2- 172.21.15.33:0/3807472693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9108100a80 0x7f9108100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.514+0000 7f910dbdf640 1 --2- 172.21.15.33:0/3807472693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9108106a80 0x7f9108106e60 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.514+0000 7f910dbdf640 1 -- 172.21.15.33:0/3807472693 >> 172.21.15.33:0/3807472693 conn(0x7f91080fc7a0 msgr2=0x7f91080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:06.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.514+0000 7f910dbdf640 1 -- 172.21.15.33:0/3807472693 shutdown_connections 2024-09-16T00:25:06.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.514+0000 7f910dbdf640 1 -- 172.21.15.33:0/3807472693 wait complete. 2024-09-16T00:25:06.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.515+0000 7f910dbdf640 1 Processor -- start 2024-09-16T00:25:06.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.515+0000 7f910dbdf640 1 -- start start 2024-09-16T00:25:06.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.515+0000 7f910dbdf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9108100a80 0x7f9108198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.515+0000 7f910dbdf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9108101420 0x7f9108198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.515+0000 7f910dbdf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9108106a80 0x7f910819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.515+0000 7f910dbdf640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9108110e80 con 0x7f9108100a80 2024-09-16T00:25:06.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.515+0000 7f910dbdf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9108110d00 con 0x7f9108101420 2024-09-16T00:25:06.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.515+0000 7f910dbdf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9108111000 con 0x7f9108106a80 2024-09-16T00:25:06.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.515+0000 7f91077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9108100a80 0x7f9108198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.515+0000 7f91077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9108100a80 0x7f9108198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53128/0 (socket says 172.21.15.33:53128) 2024-09-16T00:25:06.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.515+0000 7f91077fe640 1 -- 172.21.15.33:0/4076381928 learned_addr learned my addr 172.21.15.33:0/4076381928 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:06.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f9107fff640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9108106a80 0x7f910819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f9106ffd640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9108101420 0x7f9108198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f9107fff640 1 -- 172.21.15.33:0/4076381928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9108101420 msgr2=0x7f9108198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:06.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f9107fff640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9108101420 0x7f9108198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f9107fff640 1 -- 172.21.15.33:0/4076381928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9108100a80 msgr2=0x7f9108198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:06.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f9107fff640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9108100a80 0x7f9108198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f9107fff640 1 -- 172.21.15.33:0/4076381928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910819d4b0 con 0x7f9108106a80 2024-09-16T00:25:06.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f91077fe640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9108100a80 0x7f9108198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:06.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f9107fff640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9108106a80 0x7f910819cdd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f910000e970 tx=0x7f910000ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:06.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f9104ff9640 1 -- 172.21.15.33:0/4076381928 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910000ccc0 con 0x7f9108106a80 2024-09-16T00:25:06.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f910dbdf640 1 -- 172.21.15.33:0/4076381928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f910819d7a0 con 0x7f9108106a80 2024-09-16T00:25:06.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f9104ff9640 1 -- 172.21.15.33:0/4076381928 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9100005020 con 0x7f9108106a80 2024-09-16T00:25:06.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f9104ff9640 1 -- 172.21.15.33:0/4076381928 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100004a00 con 0x7f9108106a80 2024-09-16T00:25:06.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.516+0000 7f910dbdf640 1 -- 172.21.15.33:0/4076381928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91081a5080 con 0x7f9108106a80 2024-09-16T00:25:06.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.519+0000 7f9104ff9640 1 -- 172.21.15.33:0/4076381928 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f91000051d0 con 0x7f9108106a80 2024-09-16T00:25:06.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.519+0000 7f910ffb8740 1 Processor -- start 2024-09-16T00:25:06.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.519+0000 7f910ffb8740 1 -- start start 2024-09-16T00:25:06.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.519+0000 7f910ffb8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 0x55638aaf6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.519+0000 7f910ffb8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638aaf7380 0x55638ab05c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.519+0000 7f910ffb8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 0x55638ab08540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.519+0000 7f910ffb8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55638aa5c280 con 0x55638aaf6a60 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.519+0000 7f910ffb8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55638aa03b10 con 0x55638ab06150 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.519+0000 7f910ffb8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55638aa58240 con 0x55638aaf7380 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.519+0000 7f9106ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 0x55638aaf6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.519+0000 7f9106ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 0x55638aaf6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53130/0 (socket says 172.21.15.33:53130) 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f9106ffd640 1 -- 172.21.15.33:0/798984643 learned_addr learned my addr 172.21.15.33:0/798984643 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f91077fe640 1 --2- 172.21.15.33:0/798984643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638aaf7380 0x55638ab05c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f9106ffd640 1 --2- 172.21.15.33:0/798984643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 0x55638ab08540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f9104ff9640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f90f40a3240 0x7f90f40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f91077fe640 1 -- 172.21.15.33:0/798984643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 msgr2=0x55638ab08540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f91077fe640 1 --2- 172.21.15.33:0/798984643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 0x55638ab08540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f91077fe640 1 -- 172.21.15.33:0/798984643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 msgr2=0x55638aaf6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f91077fe640 1 --2- 172.21.15.33:0/798984643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 0x55638aaf6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f91077fe640 1 -- 172.21.15.33:0/798984643 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55638aa101f0 con 0x55638aaf7380 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f9106ffd640 1 --2- 172.21.15.33:0/798984643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 0x55638aaf6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f9104ff9640 1 -- 172.21.15.33:0/4076381928 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f91000cfd20 con 0x7f9108106a80 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f9107fff640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f90f40a3240 0x7f90f40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f91077fe640 1 --2- 172.21.15.33:0/798984643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638aaf7380 0x55638ab05c10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f90f8000f80 tx=0x7f90f8005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a400fc1c694de982 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f90fdffb640 1 -- 172.21.15.33:0/798984643 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8048070 con 0x55638aaf7380 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f90fdffb640 1 -- 172.21.15.33:0/798984643 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90f8036070 con 0x55638aaf7380 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.520+0000 7f90fdffb640 1 -- 172.21.15.33:0/798984643 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f80046c0 con 0x55638aaf7380 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.521+0000 7f910ffb8740 1 -- 172.21.15.33:0/798984643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638aaf7380 msgr2=0x55638ab05c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.521+0000 7f910ffb8740 1 --2- 172.21.15.33:0/798984643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638aaf7380 0x55638ab05c10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f90f8000f80 tx=0x7f90f8005ef0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.521+0000 7f9107fff640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f90f40a3240 0x7f90f40a5700 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f91000cd340 tx=0x7f9100005750 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.521+0000 7f910ffb8740 1 -- 172.21.15.33:0/798984643 shutdown_connections 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.521+0000 7f910ffb8740 1 --2- 172.21.15.33:0/798984643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 0x55638ab08540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.521+0000 7f910ffb8740 1 --2- 172.21.15.33:0/798984643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638aaf7380 0x55638ab05c10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.521+0000 7f910ffb8740 1 --2- 172.21.15.33:0/798984643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 0x55638aaf6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.521+0000 7f910ffb8740 1 -- 172.21.15.33:0/798984643 >> 172.21.15.33:0/798984643 conn(0x55638aaf30f0 msgr2=0x55638aaf34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.521+0000 7f910ffb8740 1 -- 172.21.15.33:0/798984643 shutdown_connections 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.521+0000 7f910ffb8740 1 -- 172.21.15.33:0/798984643 wait complete. 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.521+0000 7f910ffb8740 1 -- start start 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f910ffb8740 1 Processor -- start 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f910ffb8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 0x55638ab030f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f910ffb8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638aaf7380 0x55638ab03630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f910ffb8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 0x55638ab03b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f910ffb8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55638ab08d20 con 0x55638aaf6a60 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f910ffb8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55638aa16f10 con 0x55638ab06150 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f910ffb8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55638ab08ea0 con 0x55638aaf7380 2024-09-16T00:25:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f910ffb8740 10 client.0 _ll_register_callbacks cb 0x55638a9dfa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:06.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f91077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638aaf7380 0x55638ab03630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f9106ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 0x55638ab030f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f91077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638aaf7380 0x55638ab03630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52930/0 (socket says 172.21.15.33:52930) 2024-09-16T00:25:06.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f91077fe640 1 -- 172.21.15.33:0/3152080071 learned_addr learned my addr 172.21.15.33:0/3152080071 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:06.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f9106ffd640 1 -- 172.21.15.33:0/3152080071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638aaf7380 msgr2=0x55638ab03630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:06.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f9106ffd640 1 --2- 172.21.15.33:0/3152080071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638aaf7380 0x55638ab03630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f9106ffd640 1 -- 172.21.15.33:0/3152080071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 msgr2=0x55638ab03b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:06.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f9106ffd640 1 --2- 172.21.15.33:0/3152080071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 0x55638ab03b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.522+0000 7f9106ffd640 1 -- 172.21.15.33:0/3152080071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55638a7cd6a0 con 0x55638aaf6a60 2024-09-16T00:25:06.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.523+0000 7f9106ffd640 1 --2- 172.21.15.33:0/3152080071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 0x55638ab030f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009710 tx=0x7f90f000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42cbcd3198eaefec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:06.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.523+0000 7f90e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:06.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.523+0000 7f90e6ffd640 1 -- 172.21.15.33:0/3152080071 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f0013c70 con 0x55638aaf6a60 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.523+0000 7f90e6ffd640 1 -- 172.21.15.33:0/3152080071 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f90f0005020 con 0x55638aaf6a60 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.523+0000 7f910ffb8740 1 -- 172.21.15.33:0/3152080071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90f8036070 con 0x55638aaf6a60 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.523+0000 7f90e6ffd640 1 -- 172.21.15.33:0/3152080071 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f90f0013e10 con 0x55638aaf6a60 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.523+0000 7f910ffb8740 10 client.16860 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.523+0000 7f910ffb8740 1 -- 172.21.15.33:0/3152080071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55638ab14a80 con 0x55638aaf6a60 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.523+0000 7f90e6ffd640 1 -- 172.21.15.33:0/3152080071 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90f8036070 con 0x55638aaf6a60 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.523+0000 7f90e6ffd640 1 -- 172.21.15.33:0/3152080071 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f90f001d070 con 0x55638aaf6a60 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.524+0000 7f910ffb8740 10 client.16860 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.524+0000 7f910ffb8740 10 client.16860 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.524+0000 7f910ffb8740 20 client.16860 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.524+0000 7f910ffb8740 1 --2- 172.21.15.33:0/3152080071 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55638ab15f40 0x55638ab18350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.524+0000 7f910ffb8740 4 client.16860 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:25:06.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.524+0000 7f910ffb8740 1 -- 172.21.15.33:0/3152080071 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55638aa697f0 con 0x55638ab15f40 2024-09-16T00:25:06.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.524+0000 7f91077fe640 1 --2- 172.21.15.33:0/3152080071 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55638ab15f40 0x55638ab18350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.524+0000 7f91077fe640 1 --2- 172.21.15.33:0/3152080071 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55638ab15f40 0x55638ab18350 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9cd67b151729996 server_cookie=fe024e419aba3bc4 in_seq=0 out_seq=0 2024-09-16T00:25:06.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.524+0000 7f90e6ffd640 10 client.16860 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:06.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.525+0000 7f90e6ffd640 1 -- 172.21.15.33:0/3152080071 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55638aa697f0 con 0x55638ab15f40 2024-09-16T00:25:06.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.525+0000 7f90e6ffd640 10 client.16860 handle_command_reply: tid=0 2024-09-16T00:25:06.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.525+0000 7f910ffb8740 1 client.16860 shutdown 2024-09-16T00:25:06.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.525+0000 7f910ffb8740 1 -- 172.21.15.33:0/3152080071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 msgr2=0x55638ab030f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:06.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.525+0000 7f910ffb8740 1 --2- 172.21.15.33:0/3152080071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 0x55638ab030f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009710 tx=0x7f90f000f790 comp rx=0 tx=0).stop 2024-09-16T00:25:06.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.525+0000 7f910ffb8740 1 -- 172.21.15.33:0/3152080071 shutdown_connections 2024-09-16T00:25:06.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.525+0000 7f910ffb8740 1 --2- 172.21.15.33:0/3152080071 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55638ab15f40 0x55638ab18350 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.525+0000 7f90e6ffd640 0 client.16860 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:06.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.525+0000 7f910ffb8740 1 --2- 172.21.15.33:0/3152080071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 0x55638ab03b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.525+0000 7f910ffb8740 1 --2- 172.21.15.33:0/3152080071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638aaf7380 0x55638ab03630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.525+0000 7f910ffb8740 1 --2- 172.21.15.33:0/3152080071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf6a60 0x55638ab030f0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.525+0000 7f910ffb8740 1 -- 172.21.15.33:0/3152080071 >> 172.21.15.33:0/3152080071 conn(0x55638aaf30f0 msgr2=0x55638aaf8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:06.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.526+0000 7f910ffb8740 1 -- 172.21.15.33:0/3152080071 shutdown_connections 2024-09-16T00:25:06.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.526+0000 7f910ffb8740 1 -- 172.21.15.33:0/3152080071 wait complete. 2024-09-16T00:25:06.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.526+0000 7f910ffb8740 20 client.16860 trim_cache size 0 max 16384 2024-09-16T00:25:06.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f910ffb8740 1 Processor -- start 2024-09-16T00:25:06.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f910ffb8740 1 -- start start 2024-09-16T00:25:06.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f910ffb8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638aaf7380 0x55638ab1dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f910ffb8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638ab06150 0x55638ab1e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f910ffb8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638ab15f40 0x55638ab00350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f910ffb8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f91081af7c0 con 0x55638ab15f40 2024-09-16T00:25:06.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f910ffb8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f91081ad090 con 0x55638aaf7380 2024-09-16T00:25:06.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f910ffb8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f91081aaee0 con 0x55638ab06150 2024-09-16T00:25:06.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f9106ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638ab15f40 0x55638ab00350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f9106ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638ab15f40 0x55638ab00350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53148/0 (socket says 172.21.15.33:53148) 2024-09-16T00:25:06.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f9106ffd640 1 -- 172.21.15.33:0/2932780294 learned_addr learned my addr 172.21.15.33:0/2932780294 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:06.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f9106ffd640 1 --2- 172.21.15.33:0/2932780294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638aaf7380 0x55638ab1dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.543+0000 7f91077fe640 1 --2- 172.21.15.33:0/2932780294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638ab06150 0x55638ab1e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.544+0000 7f9106ffd640 1 -- 172.21.15.33:0/2932780294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638ab06150 msgr2=0x55638ab1e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:06.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.544+0000 7f9106ffd640 1 --2- 172.21.15.33:0/2932780294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638ab06150 0x55638ab1e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.544+0000 7f9106ffd640 1 -- 172.21.15.33:0/2932780294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638aaf7380 msgr2=0x55638ab1dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.544+0000 7f9106ffd640 1 --2- 172.21.15.33:0/2932780294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638aaf7380 0x55638ab1dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.544+0000 7f9106ffd640 1 -- 172.21.15.33:0/2932780294 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55638aaf71b0 con 0x55638ab15f40 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.544+0000 7f9106ffd640 1 --2- 172.21.15.33:0/2932780294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638ab15f40 0x55638ab00350 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f90f00137f0 tx=0x7f90f0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26339c3f741568a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.544+0000 7f90fd7fa640 1 -- 172.21.15.33:0/2932780294 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f0010500 con 0x55638ab15f40 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.544+0000 7f90fd7fa640 1 -- 172.21.15.33:0/2932780294 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90f00106a0 con 0x55638ab15f40 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.544+0000 7f90fd7fa640 1 -- 172.21.15.33:0/2932780294 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f0005280 con 0x55638ab15f40 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.545+0000 7f910ffb8740 1 -- 172.21.15.33:0/2932780294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638ab15f40 msgr2=0x55638ab00350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.545+0000 7f910ffb8740 1 --2- 172.21.15.33:0/2932780294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638ab15f40 0x55638ab00350 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f90f00137f0 tx=0x7f90f0005f90 comp rx=0 tx=0).stop 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.545+0000 7f910ffb8740 1 -- 172.21.15.33:0/2932780294 shutdown_connections 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.545+0000 7f910ffb8740 1 --2- 172.21.15.33:0/2932780294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638ab15f40 0x55638ab00350 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.545+0000 7f910ffb8740 1 --2- 172.21.15.33:0/2932780294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638ab06150 0x55638ab1e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.545+0000 7f910ffb8740 1 --2- 172.21.15.33:0/2932780294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638aaf7380 0x55638ab1dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.545+0000 7f910ffb8740 1 -- 172.21.15.33:0/2932780294 >> 172.21.15.33:0/2932780294 conn(0x55638aaf30f0 msgr2=0x55638ab05060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:06.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.545+0000 7f910ffb8740 1 -- 172.21.15.33:0/2932780294 shutdown_connections 2024-09-16T00:25:06.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.545+0000 7f910ffb8740 1 -- 172.21.15.33:0/2932780294 wait complete. 2024-09-16T00:25:06.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.545+0000 7f910ffb8740 1 -- start start 2024-09-16T00:25:06.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.545+0000 7f910ffb8740 1 Processor -- start 2024-09-16T00:25:06.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.545+0000 7f910ffb8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf7380 0x55638ab0e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f910ffb8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 0x55638ab0e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f910ffb8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638ab15f40 0x55638aafaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f910ffb8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f91081ab550 con 0x55638aaf7380 2024-09-16T00:25:06.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f910ffb8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f91081ac9c0 con 0x55638ab06150 2024-09-16T00:25:06.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f910ffb8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f91081abc20 con 0x55638ab15f40 2024-09-16T00:25:06.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f910ffb8740 10 client.0 _ll_register_callbacks cb 0x7f90f0009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:06.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f9106ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf7380 0x55638ab0e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f9106ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf7380 0x55638ab0e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53160/0 (socket says 172.21.15.33:53160) 2024-09-16T00:25:06.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f9106ffd640 1 -- 172.21.15.33:0/1997743260 learned_addr learned my addr 172.21.15.33:0/1997743260 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:06.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f91077fe640 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 0x55638ab0e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f9106ffd640 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638ab15f40 0x55638aafaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f9106ffd640 1 -- 172.21.15.33:0/1997743260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638ab15f40 msgr2=0x55638aafaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:06.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f9106ffd640 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638ab15f40 0x55638aafaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f9106ffd640 1 -- 172.21.15.33:0/1997743260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 msgr2=0x55638ab0e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:06.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f9106ffd640 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 0x55638ab0e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f9106ffd640 1 -- 172.21.15.33:0/1997743260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55638aafd7c0 con 0x55638aaf7380 2024-09-16T00:25:06.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f9106ffd640 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf7380 0x55638ab0e130 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f90f000fd10 tx=0x7f90f0009980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d63ee5ed9dda6b67 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:06.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f90e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:06.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f90e6ffd640 1 -- 172.21.15.33:0/1997743260 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f0010500 con 0x55638aaf7380 2024-09-16T00:25:06.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f910ffb8740 1 -- 172.21.15.33:0/1997743260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90f00106a0 con 0x55638aaf7380 2024-09-16T00:25:06.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.546+0000 7f90e6ffd640 1 -- 172.21.15.33:0/1997743260 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f90f0004040 con 0x55638aaf7380 2024-09-16T00:25:06.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.547+0000 7f90e6ffd640 1 -- 172.21.15.33:0/1997743260 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f90f0005d30 con 0x55638aaf7380 2024-09-16T00:25:06.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.547+0000 7f910ffb8740 10 client.16866 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:06.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.547+0000 7f910ffb8740 1 -- 172.21.15.33:0/1997743260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55638ab85880 con 0x55638aaf7380 2024-09-16T00:25:06.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.547+0000 7f90e6ffd640 1 -- 172.21.15.33:0/1997743260 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90f00106a0 con 0x55638aaf7380 2024-09-16T00:25:06.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.547+0000 7f90e6ffd640 1 -- 172.21.15.33:0/1997743260 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f90f001d040 con 0x55638aaf7380 2024-09-16T00:25:06.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.547+0000 7f910ffb8740 10 client.16866 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:06.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.547+0000 7f910ffb8740 10 client.16866 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:06.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.547+0000 7f910ffb8740 20 client.16866 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:06.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.547+0000 7f910ffb8740 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55638ab85ea0 0x55638ab88290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:06.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.547+0000 7f910ffb8740 4 client.16866 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:25:06.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.547+0000 7f910ffb8740 1 -- 172.21.15.33:0/1997743260 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55638aa63d00 con 0x55638ab85ea0 2024-09-16T00:25:06.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.547+0000 7f91077fe640 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55638ab85ea0 0x55638ab88290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:06.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.548+0000 7f91077fe640 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55638ab85ea0 0x55638ab88290 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83b976ac7b58e606 server_cookie=e1971d9909d792c3 in_seq=0 out_seq=0 2024-09-16T00:25:06.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.548+0000 7f90e6ffd640 10 client.16866 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:06.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f90e6ffd640 1 -- 172.21.15.33:0/1997743260 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f90f8036080 con 0x55638ab85ea0 2024-09-16T00:25:06.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f90e6ffd640 10 client.16866 handle_command_reply: tid=0 2024-09-16T00:25:06.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f910ffb8740 1 client.16866 shutdown 2024-09-16T00:25:06.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f910ffb8740 1 -- 172.21.15.33:0/1997743260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf7380 msgr2=0x55638ab0e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:06.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f910ffb8740 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf7380 0x55638ab0e130 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f90f000fd10 tx=0x7f90f0009980 comp rx=0 tx=0).stop 2024-09-16T00:25:06.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f910ffb8740 1 -- 172.21.15.33:0/1997743260 shutdown_connections 2024-09-16T00:25:06.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f910ffb8740 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55638ab85ea0 0x55638ab88290 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f910ffb8740 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55638ab15f40 0x55638aafaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f910ffb8740 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55638ab06150 0x55638ab0e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f910ffb8740 1 --2- 172.21.15.33:0/1997743260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55638aaf7380 0x55638ab0e130 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f90e6ffd640 0 client.16866 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:06.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f910ffb8740 1 -- 172.21.15.33:0/1997743260 >> 172.21.15.33:0/1997743260 conn(0x55638aaf30f0 msgr2=0x55638aaf8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:06.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f910ffb8740 1 -- 172.21.15.33:0/1997743260 shutdown_connections 2024-09-16T00:25:06.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f910ffb8740 1 -- 172.21.15.33:0/1997743260 wait complete. 2024-09-16T00:25:06.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.549+0000 7f910ffb8740 20 client.16866 trim_cache size 0 max 16384 2024-09-16T00:25:06.560 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:25:06.560 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:25:06.560 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "7fa834f9-a598-47e1-a5b1-ce47798be560", 2024-09-16T00:25:06.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:25:06.560 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:25:06.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.550+0000 7f910dbdf640 1 -- 172.21.15.33:0/4076381928 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f90f40a3240 msgr2=0x7f90f40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:06.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.550+0000 7f910dbdf640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f90f40a3240 0x7f90f40a5700 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f91000cd340 tx=0x7f9100005750 comp rx=0 tx=0).stop 2024-09-16T00:25:06.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.550+0000 7f910dbdf640 1 -- 172.21.15.33:0/4076381928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9108106a80 msgr2=0x7f910819cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:06.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.550+0000 7f910dbdf640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9108106a80 0x7f910819cdd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f910000e970 tx=0x7f910000ee30 comp rx=0 tx=0).stop 2024-09-16T00:25:06.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.550+0000 7f910dbdf640 1 -- 172.21.15.33:0/4076381928 shutdown_connections 2024-09-16T00:25:06.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.550+0000 7f910dbdf640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f90f40a3240 0x7f90f40a5700 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.550+0000 7f910dbdf640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9108106a80 0x7f910819cdd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.550+0000 7f910dbdf640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9108101420 0x7f9108198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.550+0000 7f910dbdf640 1 --2- 172.21.15.33:0/4076381928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9108100a80 0x7f9108198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:06.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.550+0000 7f910dbdf640 1 -- 172.21.15.33:0/4076381928 >> 172.21.15.33:0/4076381928 conn(0x7f91080fc7a0 msgr2=0x7f910806aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:06.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.550+0000 7f910dbdf640 1 -- 172.21.15.33:0/4076381928 shutdown_connections 2024-09-16T00:25:06.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:06.550+0000 7f910dbdf640 1 -- 172.21.15.33:0/4076381928 wait complete. 2024-09-16T00:25:06.667 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.75 MB/sec execute 254 sec latency 4269.844 ms 2024-09-16T00:25:07.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:07 smithi033 ceph-mon[31243]: pgmap v589: 129 pgs: 129 active+clean; 233 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 252 op/s 2024-09-16T00:25:07.386 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:07 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:25:07.386 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:07 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:07.411 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:25:07.667 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.70 MB/sec execute 255 sec latency 5269.901 ms 2024-09-16T00:25:07.668 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:07.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:07 smithi083 ceph-mon[35242]: pgmap v589: 129 pgs: 129 active+clean; 233 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 252 op/s 2024-09-16T00:25:07.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:07 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:25:07.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:07 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:07.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:07 smithi060 ceph-mon[35185]: pgmap v589: 129 pgs: 129 active+clean; 233 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 252 op/s 2024-09-16T00:25:07.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:07 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:25:07.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:07 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:08.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.222+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2172985154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff358100e20 msgr2=0x7ff358101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:08.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.222+0000 7ff35eb7d640 1 --2- 172.21.15.33:0/2172985154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff358100e20 0x7ff358101200 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff354009900 tx=0x7ff35402a480 comp rx=0 tx=0).stop 2024-09-16T00:25:08.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.222+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2172985154 shutdown_connections 2024-09-16T00:25:08.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.222+0000 7ff35eb7d640 1 --2- 172.21.15.33:0/2172985154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff358102ef0 0x7ff35810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.222+0000 7ff35eb7d640 1 --2- 172.21.15.33:0/2172985154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff358102570 0x7ff3581029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.222+0000 7ff35eb7d640 1 --2- 172.21.15.33:0/2172985154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff358100e20 0x7ff358101200 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.222+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2172985154 >> 172.21.15.33:0/2172985154 conn(0x7ff3580fc7d0 msgr2=0x7ff3580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:08.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.222+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2172985154 shutdown_connections 2024-09-16T00:25:08.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.222+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2172985154 wait complete. 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.222+0000 7ff35eb7d640 1 Processor -- start 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.222+0000 7ff35eb7d640 1 -- start start 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35eb7d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff358100e20 0x7ff3581984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35eb7d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff358102570 0x7ff358198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35eb7d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff358102ef0 0x7ff35819cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35eb7d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff358112060 con 0x7ff358102ef0 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35eb7d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff358111ee0 con 0x7ff358100e20 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35eb7d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff3581121e0 con 0x7ff358102570 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35d11a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff358102ef0 0x7ff35819cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35d11a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff358102ef0 0x7ff35819cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53164/0 (socket says 172.21.15.33:53164) 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35d11a640 1 -- 172.21.15.33:0/2461827862 learned_addr learned my addr 172.21.15.33:0/2461827862 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35c919640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff358100e20 0x7ff3581984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff34ffff640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff358102570 0x7ff358198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35d11a640 1 -- 172.21.15.33:0/2461827862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff358102570 msgr2=0x7ff358198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:08.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35d11a640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff358102570 0x7ff358198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35d11a640 1 -- 172.21.15.33:0/2461827862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff358100e20 msgr2=0x7ff3581984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:08.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35d11a640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff358100e20 0x7ff3581984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35d11a640 1 -- 172.21.15.33:0/2461827862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35819d470 con 0x7ff358102ef0 2024-09-16T00:25:08.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35d11a640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff358102ef0 0x7ff35819cd90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff35000cad0 tx=0x7ff35000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:08.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff34dffb640 1 -- 172.21.15.33:0/2461827862 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35000ae20 con 0x7ff358102ef0 2024-09-16T00:25:08.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff34dffb640 1 -- 172.21.15.33:0/2461827862 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff350005020 con 0x7ff358102ef0 2024-09-16T00:25:08.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.223+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2461827862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff35819d760 con 0x7ff358102ef0 2024-09-16T00:25:08.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.224+0000 7ff34dffb640 1 -- 172.21.15.33:0/2461827862 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35000f6f0 con 0x7ff358102ef0 2024-09-16T00:25:08.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.224+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2461827862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3581a5040 con 0x7ff358102ef0 2024-09-16T00:25:08.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.226+0000 7ff34dffb640 1 -- 172.21.15.33:0/2461827862 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff3500051d0 con 0x7ff358102ef0 2024-09-16T00:25:08.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff360f56740 1 Processor -- start 2024-09-16T00:25:08.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff360f56740 1 -- start start 2024-09-16T00:25:08.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff360f56740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294d4a0 0x564bf294d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff34dffb640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff3400a3240 0x7ff3400a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff360f56740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294ddc0 0x564bf2958140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff360f56740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf2958680 0x564bf295aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff360f56740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564bf28b32e0 con 0x564bf294ddc0 2024-09-16T00:25:08.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff360f56740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564bf285ace0 con 0x564bf294d4a0 2024-09-16T00:25:08.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff360f56740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564bf28af2a0 con 0x564bf2958680 2024-09-16T00:25:08.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff35c919640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff3400a3240 0x7ff3400a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff34ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294ddc0 0x564bf2958140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff34ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294ddc0 0x564bf2958140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53168/0 (socket says 172.21.15.33:53168) 2024-09-16T00:25:08.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff34ffff640 1 -- 172.21.15.33:0/29463799 learned_addr learned my addr 172.21.15.33:0/29463799 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:08.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff34dffb640 1 -- 172.21.15.33:0/2461827862 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff3500cfe90 con 0x7ff358102ef0 2024-09-16T00:25:08.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff35d11a640 1 --2- 172.21.15.33:0/29463799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf2958680 0x564bf295aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.227+0000 7ff34ffff640 1 --2- 172.21.15.33:0/29463799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294d4a0 0x564bf294d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.228+0000 7ff34ffff640 1 -- 172.21.15.33:0/29463799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf2958680 msgr2=0x564bf295aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:08.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.228+0000 7ff34ffff640 1 --2- 172.21.15.33:0/29463799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf2958680 0x564bf295aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.228+0000 7ff34ffff640 1 -- 172.21.15.33:0/29463799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294d4a0 msgr2=0x564bf294d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:08.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.228+0000 7ff34ffff640 1 --2- 172.21.15.33:0/29463799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294d4a0 0x564bf294d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.228+0000 7ff34ffff640 1 -- 172.21.15.33:0/29463799 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bf283f440 con 0x564bf294ddc0 2024-09-16T00:25:08.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.228+0000 7ff35c919640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff3400a3240 0x7ff3400a5700 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff354000c00 tx=0x7ff3540058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:08.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.228+0000 7ff34ffff640 1 --2- 172.21.15.33:0/29463799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294ddc0 0x564bf2958140 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff348013930 tx=0x7ff34800b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c5531fa695ea2b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:08.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.228+0000 7ff33effd640 1 -- 172.21.15.33:0/29463799 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34800e950 con 0x564bf294ddc0 2024-09-16T00:25:08.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.228+0000 7ff33effd640 1 -- 172.21.15.33:0/29463799 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff348010430 con 0x564bf294ddc0 2024-09-16T00:25:08.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.228+0000 7ff33effd640 1 -- 172.21.15.33:0/29463799 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34800f600 con 0x564bf294ddc0 2024-09-16T00:25:08.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.229+0000 7ff360f56740 1 -- 172.21.15.33:0/29463799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294ddc0 msgr2=0x564bf2958140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:08.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.229+0000 7ff360f56740 1 --2- 172.21.15.33:0/29463799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294ddc0 0x564bf2958140 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff348013930 tx=0x7ff34800b460 comp rx=0 tx=0).stop 2024-09-16T00:25:08.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.229+0000 7ff360f56740 1 -- 172.21.15.33:0/29463799 shutdown_connections 2024-09-16T00:25:08.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.229+0000 7ff360f56740 1 --2- 172.21.15.33:0/29463799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf2958680 0x564bf295aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.229+0000 7ff360f56740 1 --2- 172.21.15.33:0/29463799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294ddc0 0x564bf2958140 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.229+0000 7ff360f56740 1 --2- 172.21.15.33:0/29463799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294d4a0 0x564bf294d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.229+0000 7ff360f56740 1 -- 172.21.15.33:0/29463799 >> 172.21.15.33:0/29463799 conn(0x564bf2949a30 msgr2=0x564bf2949e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:08.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.229+0000 7ff360f56740 1 -- 172.21.15.33:0/29463799 shutdown_connections 2024-09-16T00:25:08.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.229+0000 7ff360f56740 1 -- 172.21.15.33:0/29463799 wait complete. 2024-09-16T00:25:08.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.229+0000 7ff360f56740 1 -- start start 2024-09-16T00:25:08.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.229+0000 7ff360f56740 1 Processor -- start 2024-09-16T00:25:08.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff360f56740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294d4a0 0x564bf2957fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff360f56740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294ddc0 0x564bf2955250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff360f56740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2958680 0x564bf2955790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff360f56740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564bf295b250 con 0x564bf2958680 2024-09-16T00:25:08.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff360f56740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564bf286e050 con 0x564bf294d4a0 2024-09-16T00:25:08.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff360f56740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564bf295b3d0 con 0x564bf294ddc0 2024-09-16T00:25:08.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff360f56740 10 client.0 _ll_register_callbacks cb 0x564bf28ae160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:08.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff35d11a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2958680 0x564bf2955790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff35d11a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2958680 0x564bf2955790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53170/0 (socket says 172.21.15.33:53170) 2024-09-16T00:25:08.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff35d11a640 1 -- 172.21.15.33:0/1151130268 learned_addr learned my addr 172.21.15.33:0/1151130268 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:08.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff35c919640 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294d4a0 0x564bf2957fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff34ffff640 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294ddc0 0x564bf2955250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff35d11a640 1 -- 172.21.15.33:0/1151130268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294ddc0 msgr2=0x564bf2955250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:08.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff35d11a640 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294ddc0 0x564bf2955250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff35d11a640 1 -- 172.21.15.33:0/1151130268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294d4a0 msgr2=0x564bf2957fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:08.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff35d11a640 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294d4a0 0x564bf2957fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff35d11a640 1 -- 172.21.15.33:0/1151130268 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bf26243c0 con 0x564bf2958680 2024-09-16T00:25:08.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff35c919640 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294d4a0 0x564bf2957fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:25:08.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff35d11a640 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2958680 0x564bf2955790 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff35009bcd0 tx=0x7ff3500aecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6345f6bd7fb55d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:08.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff337fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:08.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff337fff640 1 -- 172.21.15.33:0/1151130268 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35001d040 con 0x564bf2958680 2024-09-16T00:25:08.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff360f56740 1 -- 172.21.15.33:0/1151130268 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff348010430 con 0x564bf2958680 2024-09-16T00:25:08.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.230+0000 7ff337fff640 1 -- 172.21.15.33:0/1151130268 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff35009a2d0 con 0x564bf2958680 2024-09-16T00:25:08.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.231+0000 7ff337fff640 1 -- 172.21.15.33:0/1151130268 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff3500ae8e0 con 0x564bf2958680 2024-09-16T00:25:08.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.231+0000 7ff360f56740 10 client.16878 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:08.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.231+0000 7ff360f56740 1 -- 172.21.15.33:0/1151130268 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564bf2966c30 con 0x564bf2958680 2024-09-16T00:25:08.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.231+0000 7ff337fff640 1 -- 172.21.15.33:0/1151130268 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff348010430 con 0x564bf2958680 2024-09-16T00:25:08.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.231+0000 7ff337fff640 1 -- 172.21.15.33:0/1151130268 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff350098640 con 0x564bf2958680 2024-09-16T00:25:08.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.231+0000 7ff360f56740 10 client.16878 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:08.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.231+0000 7ff360f56740 10 client.16878 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:08.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.231+0000 7ff360f56740 20 client.16878 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:08.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.231+0000 7ff360f56740 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564bf2968040 0x564bf296a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.231+0000 7ff360f56740 4 client.16878 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:25:08.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.231+0000 7ff360f56740 1 -- 172.21.15.33:0/1151130268 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bf28c0850 con 0x564bf2968040 2024-09-16T00:25:08.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.231+0000 7ff35c919640 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564bf2968040 0x564bf296a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.232+0000 7ff35c919640 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564bf2968040 0x564bf296a470 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffaa3681011b9cac server_cookie=87c1263c14681823 in_seq=0 out_seq=0 2024-09-16T00:25:08.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.232+0000 7ff337fff640 10 client.16878 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:08.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.232+0000 7ff337fff640 1 -- 172.21.15.33:0/1151130268 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564bf28c0850 con 0x564bf2968040 2024-09-16T00:25:08.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.232+0000 7ff337fff640 10 client.16878 handle_command_reply: tid=0 2024-09-16T00:25:08.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.232+0000 7ff360f56740 1 client.16878 shutdown 2024-09-16T00:25:08.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.233+0000 7ff360f56740 1 -- 172.21.15.33:0/1151130268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2958680 msgr2=0x564bf2955790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:08.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.233+0000 7ff360f56740 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2958680 0x564bf2955790 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff35009bcd0 tx=0x7ff3500aecf0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.233+0000 7ff360f56740 1 -- 172.21.15.33:0/1151130268 shutdown_connections 2024-09-16T00:25:08.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.233+0000 7ff360f56740 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564bf2968040 0x564bf296a470 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.233+0000 7ff360f56740 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2958680 0x564bf2955790 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.233+0000 7ff360f56740 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294ddc0 0x564bf2955250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.233+0000 7ff360f56740 1 --2- 172.21.15.33:0/1151130268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294d4a0 0x564bf2957fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.233+0000 7ff337fff640 0 client.16878 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:08.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.233+0000 7ff360f56740 1 -- 172.21.15.33:0/1151130268 >> 172.21.15.33:0/1151130268 conn(0x564bf2949a30 msgr2=0x564bf294f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:08.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.233+0000 7ff360f56740 1 -- 172.21.15.33:0/1151130268 shutdown_connections 2024-09-16T00:25:08.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.233+0000 7ff360f56740 1 -- 172.21.15.33:0/1151130268 wait complete. 2024-09-16T00:25:08.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.233+0000 7ff360f56740 20 client.16878 trim_cache size 0 max 16384 2024-09-16T00:25:08.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.250+0000 7ff360f56740 1 Processor -- start 2024-09-16T00:25:08.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.250+0000 7ff360f56740 1 -- start start 2024-09-16T00:25:08.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.250+0000 7ff360f56740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2949a30 0x564bf29571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.250+0000 7ff360f56740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294d4a0 0x564bf296fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.250+0000 7ff360f56740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294ddc0 0x564bf29702d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.250+0000 7ff360f56740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff3581b1990 con 0x564bf2949a30 2024-09-16T00:25:08.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.250+0000 7ff360f56740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff3581af260 con 0x564bf294ddc0 2024-09-16T00:25:08.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.250+0000 7ff360f56740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff3581ad0b0 con 0x564bf294d4a0 2024-09-16T00:25:08.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.250+0000 7ff35c919640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2949a30 0x564bf29571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.250+0000 7ff35c919640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2949a30 0x564bf29571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53182/0 (socket says 172.21.15.33:53182) 2024-09-16T00:25:08.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.250+0000 7ff35c919640 1 -- 172.21.15.33:0/3127203505 learned_addr learned my addr 172.21.15.33:0/3127203505 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:08.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.251+0000 7ff35d11a640 1 --2- 172.21.15.33:0/3127203505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294ddc0 0x564bf29702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.251+0000 7ff35c919640 1 -- 172.21.15.33:0/3127203505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294d4a0 msgr2=0x564bf296fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:08.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.251+0000 7ff35c919640 1 --2- 172.21.15.33:0/3127203505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294d4a0 0x564bf296fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.251+0000 7ff35c919640 1 -- 172.21.15.33:0/3127203505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294ddc0 msgr2=0x564bf29702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.251+0000 7ff35c919640 1 --2- 172.21.15.33:0/3127203505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294ddc0 0x564bf29702d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.251+0000 7ff35c919640 1 -- 172.21.15.33:0/3127203505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bf29494d0 con 0x564bf2949a30 2024-09-16T00:25:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.251+0000 7ff35d11a640 1 --2- 172.21.15.33:0/3127203505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294ddc0 0x564bf29702d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:25:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.251+0000 7ff35c919640 1 --2- 172.21.15.33:0/3127203505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2949a30 0x564bf29571c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff354009900 tx=0x7ff35403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ad2b0ce9187186 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.251+0000 7ff33e7fc640 1 -- 172.21.15.33:0/3127203505 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff354035050 con 0x564bf2949a30 2024-09-16T00:25:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.251+0000 7ff33e7fc640 1 -- 172.21.15.33:0/3127203505 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff354056070 con 0x564bf2949a30 2024-09-16T00:25:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.251+0000 7ff33e7fc640 1 -- 172.21.15.33:0/3127203505 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff354003e60 con 0x564bf2949a30 2024-09-16T00:25:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.252+0000 7ff360f56740 1 -- 172.21.15.33:0/3127203505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2949a30 msgr2=0x564bf29571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.252+0000 7ff360f56740 1 --2- 172.21.15.33:0/3127203505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2949a30 0x564bf29571c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff354009900 tx=0x7ff35403e000 comp rx=0 tx=0).stop 2024-09-16T00:25:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.252+0000 7ff360f56740 1 -- 172.21.15.33:0/3127203505 shutdown_connections 2024-09-16T00:25:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.252+0000 7ff360f56740 1 --2- 172.21.15.33:0/3127203505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf294ddc0 0x564bf29702d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.252+0000 7ff360f56740 1 --2- 172.21.15.33:0/3127203505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294d4a0 0x564bf296fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.252+0000 7ff360f56740 1 --2- 172.21.15.33:0/3127203505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf2949a30 0x564bf29571c0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.252+0000 7ff360f56740 1 -- 172.21.15.33:0/3127203505 >> 172.21.15.33:0/3127203505 conn(0x564bf29558f0 msgr2=0x564bf2956ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.252+0000 7ff360f56740 1 -- 172.21.15.33:0/3127203505 shutdown_connections 2024-09-16T00:25:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.252+0000 7ff360f56740 1 -- 172.21.15.33:0/3127203505 wait complete. 2024-09-16T00:25:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.252+0000 7ff360f56740 1 -- start start 2024-09-16T00:25:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.252+0000 7ff360f56740 1 Processor -- start 2024-09-16T00:25:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff360f56740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf2949a30 0x564bf2960db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff360f56740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294d4a0 0x564bf295e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff360f56740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294ddc0 0x564bf295e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff360f56740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff3581ad720 con 0x564bf294d4a0 2024-09-16T00:25:08.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff360f56740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff3581aeb90 con 0x564bf2949a30 2024-09-16T00:25:08.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff360f56740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff3581addf0 con 0x564bf294ddc0 2024-09-16T00:25:08.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff360f56740 10 client.0 _ll_register_callbacks cb 0x7ff35009bde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:08.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff34ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294d4a0 0x564bf295e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff34ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294d4a0 0x564bf295e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53186/0 (socket says 172.21.15.33:53186) 2024-09-16T00:25:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff34ffff640 1 -- 172.21.15.33:0/3124872305 learned_addr learned my addr 172.21.15.33:0/3124872305 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff35d11a640 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294ddc0 0x564bf295e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff35c919640 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf2949a30 0x564bf2960db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff34ffff640 1 -- 172.21.15.33:0/3124872305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294ddc0 msgr2=0x564bf295e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff34ffff640 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294ddc0 0x564bf295e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff34ffff640 1 -- 172.21.15.33:0/3124872305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf2949a30 msgr2=0x564bf2960db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff34ffff640 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf2949a30 0x564bf2960db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff34ffff640 1 -- 172.21.15.33:0/3124872305 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bf295ee40 con 0x564bf294d4a0 2024-09-16T00:25:08.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff35c919640 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf2949a30 0x564bf2960db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:25:08.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff34ffff640 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294d4a0 0x564bf295e100 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ff348013930 tx=0x7ff348010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eeeebb7d3b10539 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:08.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.253+0000 7ff337fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:08.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff337fff640 1 -- 172.21.15.33:0/3124872305 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34800bb80 con 0x564bf294d4a0 2024-09-16T00:25:08.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff360f56740 1 -- 172.21.15.33:0/3124872305 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff354056070 con 0x564bf294d4a0 2024-09-16T00:25:08.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff337fff640 1 -- 172.21.15.33:0/3124872305 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff348005110 con 0x564bf294d4a0 2024-09-16T00:25:08.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff337fff640 1 -- 172.21.15.33:0/3124872305 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff348031050 con 0x564bf294d4a0 2024-09-16T00:25:08.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff360f56740 10 client.16884 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:08.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff360f56740 1 -- 172.21.15.33:0/3124872305 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564bf29d2d20 con 0x564bf294d4a0 2024-09-16T00:25:08.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff337fff640 1 -- 172.21.15.33:0/3124872305 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff354056070 con 0x564bf294d4a0 2024-09-16T00:25:08.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff337fff640 1 -- 172.21.15.33:0/3124872305 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff348026020 con 0x564bf294d4a0 2024-09-16T00:25:08.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff360f56740 10 client.16884 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:08.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff360f56740 10 client.16884 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:08.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff360f56740 20 client.16884 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:08.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff360f56740 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564bf2958680 0x564bf29d5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:08.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff360f56740 4 client.16884 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:25:08.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff360f56740 1 -- 172.21.15.33:0/3124872305 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564bf28bad60 con 0x564bf2958680 2024-09-16T00:25:08.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.254+0000 7ff35c919640 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564bf2958680 0x564bf29d5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff337fff640 10 client.16884 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff35c919640 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564bf2958680 0x564bf29d5240 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed8b86c0104cf94c server_cookie=66643ff765b793ab in_seq=0 out_seq=0 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff337fff640 1 -- 172.21.15.33:0/3124872305 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564bf28bad60 con 0x564bf2958680 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff337fff640 10 client.16884 handle_command_reply: tid=0 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff360f56740 1 client.16884 shutdown 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff360f56740 1 -- 172.21.15.33:0/3124872305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294d4a0 msgr2=0x564bf295e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff360f56740 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294d4a0 0x564bf295e100 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ff348013930 tx=0x7ff348010660 comp rx=0 tx=0).stop 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff360f56740 1 -- 172.21.15.33:0/3124872305 shutdown_connections 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff360f56740 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564bf2958680 0x564bf29d5240 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff337fff640 0 client.16884 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff360f56740 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564bf294ddc0 0x564bf295e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff360f56740 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564bf294d4a0 0x564bf295e100 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff360f56740 1 --2- 172.21.15.33:0/3124872305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564bf2949a30 0x564bf2960db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff360f56740 1 -- 172.21.15.33:0/3124872305 >> 172.21.15.33:0/3124872305 conn(0x564bf29558f0 msgr2=0x564bf29572d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:08.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff360f56740 1 -- 172.21.15.33:0/3124872305 shutdown_connections 2024-09-16T00:25:08.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.255+0000 7ff360f56740 1 -- 172.21.15.33:0/3124872305 wait complete. 2024-09-16T00:25:08.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff360f56740 20 client.16884 trim_cache size 0 max 16384 2024-09-16T00:25:08.282 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:25:08.282 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:25:08.282 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:25:08.282 INFO:teuthology.orchestra.run.smithi033.stdout: "7fa834f9-a598-47e1-a5b1-ce47798be560": { 2024-09-16T00:25:08.282 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:25:08.283 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "7fa834f9-a598-47e1-a5b1-ce47798be560", 2024-09-16T00:25:08.283 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:25:08.283 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:08.283 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:08.283 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:25:08.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2461827862 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff3400a3240 msgr2=0x7ff3400a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:08.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff35eb7d640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff3400a3240 0x7ff3400a5700 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff354000c00 tx=0x7ff3540058d0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2461827862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff358102ef0 msgr2=0x7ff35819cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:08.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff35eb7d640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff358102ef0 0x7ff35819cd90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff35000cad0 tx=0x7ff35000cf90 comp rx=0 tx=0).stop 2024-09-16T00:25:08.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2461827862 shutdown_connections 2024-09-16T00:25:08.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff35eb7d640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff3400a3240 0x7ff3400a5700 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff35eb7d640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff358102ef0 0x7ff35819cd90 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff35eb7d640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff358102570 0x7ff358198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff35eb7d640 1 --2- 172.21.15.33:0/2461827862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff358100e20 0x7ff3581984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:08.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2461827862 >> 172.21.15.33:0/2461827862 conn(0x7ff3580fc7d0 msgr2=0x7ff3581068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:08.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2461827862 shutdown_connections 2024-09-16T00:25:08.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:08.256+0000 7ff35eb7d640 1 -- 172.21.15.33:0/2461827862 wait complete. 2024-09-16T00:25:08.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:08 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:25:08.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:08 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:08 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:25:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:08 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:08.667 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.66 MB/sec execute 256 sec latency 6269.942 ms 2024-09-16T00:25:08.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:08 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:25:08.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:08 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:08.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:08 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:25:08.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:08 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:08.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:08 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:25:08.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:08 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:08.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:08 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:25:08.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:08 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:09.015 INFO:tasks.cephfs.filesystem:scrub status for tag:7fa834f9-a598-47e1-a5b1-ce47798be560 - {'path': '/', 'tag': '7fa834f9-a598-47e1-a5b1-ce47798be560', 'options': 'recursive,force'} 2024-09-16T00:25:09.667 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.62 MB/sec execute 257 sec latency 7270.006 ms 2024-09-16T00:25:09.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:09 smithi083 ceph-mon[35242]: pgmap v590: 129 pgs: 129 active+clean; 233 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 196 op/s 2024-09-16T00:25:09.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:09 smithi060 ceph-mon[35185]: pgmap v590: 129 pgs: 129 active+clean; 233 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 196 op/s 2024-09-16T00:25:09.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:09 smithi033 ceph-mon[31243]: pgmap v590: 129 pgs: 129 active+clean; 233 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 196 op/s 2024-09-16T00:25:10.667 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.58 MB/sec execute 258 sec latency 8270.098 ms 2024-09-16T00:25:11.667 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.54 MB/sec execute 259 sec latency 9270.203 ms 2024-09-16T00:25:11.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:11 smithi083 ceph-mon[35242]: pgmap v591: 129 pgs: 129 active+clean; 232 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 207 op/s 2024-09-16T00:25:11.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:11 smithi060 ceph-mon[35185]: pgmap v591: 129 pgs: 129 active+clean; 232 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 207 op/s 2024-09-16T00:25:11.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:11 smithi033 ceph-mon[31243]: pgmap v591: 129 pgs: 129 active+clean; 232 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 207 op/s 2024-09-16T00:25:12.668 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.50 MB/sec execute 260 sec latency 10270.317 ms 2024-09-16T00:25:13.668 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.46 MB/sec execute 261 sec latency 11270.439 ms 2024-09-16T00:25:13.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:13 smithi083 ceph-mon[35242]: pgmap v592: 129 pgs: 129 active+clean; 231 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 204 op/s 2024-09-16T00:25:13.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:13 smithi060 ceph-mon[35185]: pgmap v592: 129 pgs: 129 active+clean; 231 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 204 op/s 2024-09-16T00:25:13.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:13 smithi033 ceph-mon[31243]: pgmap v592: 129 pgs: 129 active+clean; 231 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 204 op/s 2024-09-16T00:25:14.668 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.42 MB/sec execute 262 sec latency 12270.561 ms 2024-09-16T00:25:15.668 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.38 MB/sec execute 263 sec latency 13270.612 ms 2024-09-16T00:25:15.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:15 smithi083 ceph-mon[35242]: pgmap v593: 129 pgs: 129 active+clean; 231 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.8 MiB/s wr, 142 op/s 2024-09-16T00:25:15.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:15 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:25:15.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:15 smithi060 ceph-mon[35185]: pgmap v593: 129 pgs: 129 active+clean; 231 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.8 MiB/s wr, 142 op/s 2024-09-16T00:25:15.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:15 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:25:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:15 smithi033 ceph-mon[31243]: pgmap v593: 129 pgs: 129 active+clean; 231 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.8 MiB/s wr, 142 op/s 2024-09-16T00:25:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:15 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:25:16.668 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.34 MB/sec execute 264 sec latency 14270.654 ms 2024-09-16T00:25:16.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:25:16.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:25:16.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:25:16.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:25:16.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:25:16.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:25:16.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:25:16.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:25:16.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:25:17.668 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.30 MB/sec execute 265 sec latency 15270.712 ms 2024-09-16T00:25:17.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:17 smithi083 ceph-mon[35242]: pgmap v594: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 2.8 MiB/s wr, 135 op/s 2024-09-16T00:25:17.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:17 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:25:17.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:17 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:25:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:17 smithi060 ceph-mon[35185]: pgmap v594: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 2.8 MiB/s wr, 135 op/s 2024-09-16T00:25:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:17 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:25:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:17 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:25:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:17 smithi033 ceph-mon[31243]: pgmap v594: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 2.8 MiB/s wr, 135 op/s 2024-09-16T00:25:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:17 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:25:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:17 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:25:18.668 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.26 MB/sec execute 266 sec latency 16270.774 ms 2024-09-16T00:25:19.668 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.22 MB/sec execute 267 sec latency 17270.863 ms 2024-09-16T00:25:19.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:19 smithi083 ceph-mon[35242]: pgmap v595: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 2.2 MiB/s wr, 82 op/s 2024-09-16T00:25:19.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:19 smithi060 ceph-mon[35185]: pgmap v595: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 2.2 MiB/s wr, 82 op/s 2024-09-16T00:25:19.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:19 smithi033 ceph-mon[31243]: pgmap v595: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 2.2 MiB/s wr, 82 op/s 2024-09-16T00:25:20.668 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.19 MB/sec execute 268 sec latency 18270.950 ms 2024-09-16T00:25:21.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:21 smithi083 ceph-mon[35242]: pgmap v596: 129 pgs: 129 active+clean; 228 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 2.6 MiB/s wr, 94 op/s 2024-09-16T00:25:21.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.15 MB/sec execute 269 sec latency 19271.067 ms 2024-09-16T00:25:21.723 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:25:21.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:21 smithi060 ceph-mon[35185]: pgmap v596: 129 pgs: 129 active+clean; 228 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 2.6 MiB/s wr, 94 op/s 2024-09-16T00:25:21.762 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:21 smithi033 ceph-mon[31243]: pgmap v596: 129 pgs: 129 active+clean; 228 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 2.6 MiB/s wr, 94 op/s 2024-09-16T00:25:22.033 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:22.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.532+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3250480072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e81082a0 msgr2=0x7f30e810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:22.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.532+0000 7f30ed2ed640 1 --2- 172.21.15.33:0/3250480072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e81082a0 0x7f30e810ec40 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f30e0009cd0 tx=0x7f30e002a8e0 comp rx=0 tx=0).stop 2024-09-16T00:25:22.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.532+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3250480072 shutdown_connections 2024-09-16T00:25:22.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.532+0000 7f30ed2ed640 1 --2- 172.21.15.33:0/3250480072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e81082a0 0x7f30e810ec40 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:22.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.532+0000 7f30ed2ed640 1 --2- 172.21.15.33:0/3250480072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e81036e0 0x7f30e8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:22.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.532+0000 7f30ed2ed640 1 --2- 172.21.15.33:0/3250480072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30e8102d30 0x7f30e8103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:22.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.532+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3250480072 >> 172.21.15.33:0/3250480072 conn(0x7f30e80fc7d0 msgr2=0x7f30e80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:22.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.532+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3250480072 shutdown_connections 2024-09-16T00:25:22.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.532+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3250480072 wait complete. 2024-09-16T00:25:22.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.532+0000 7f30ed2ed640 1 Processor -- start 2024-09-16T00:25:22.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30ed2ed640 1 -- start start 2024-09-16T00:25:22.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30ed2ed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e8102d30 0x7f30e81984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:22.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30ed2ed640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e81036e0 0x7f30e8198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:22.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30ed2ed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30e81082a0 0x7f30e819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:22.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30ed2ed640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f30e81120a0 con 0x7f30e81036e0 2024-09-16T00:25:22.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30ed2ed640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f30e8111f20 con 0x7f30e8102d30 2024-09-16T00:25:22.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30ed2ed640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f30e8112220 con 0x7f30e81082a0 2024-09-16T00:25:22.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30de7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e81036e0 0x7f30e8198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:22.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30de7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e81036e0 0x7f30e8198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52952/0 (socket says 172.21.15.33:52952) 2024-09-16T00:25:22.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30de7fc640 1 -- 172.21.15.33:0/3919756466 learned_addr learned my addr 172.21.15.33:0/3919756466 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:22.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30e77fe640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30e81082a0 0x7f30e819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:22.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30e6ffd640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e8102d30 0x7f30e81984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:22.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30de7fc640 1 -- 172.21.15.33:0/3919756466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30e81082a0 msgr2=0x7f30e819cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:22.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30de7fc640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30e81082a0 0x7f30e819cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30de7fc640 1 -- 172.21.15.33:0/3919756466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e8102d30 msgr2=0x7f30e81984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30de7fc640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e8102d30 0x7f30e81984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.533+0000 7f30de7fc640 1 -- 172.21.15.33:0/3919756466 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e819d4a0 con 0x7f30e81036e0 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.534+0000 7f30de7fc640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e81036e0 0x7f30e8198a30 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f30d000c8c0 tx=0x7f30d000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.534+0000 7f30e4ff9640 1 -- 172.21.15.33:0/3919756466 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0007bb0 con 0x7f30e81036e0 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.534+0000 7f30e4ff9640 1 -- 172.21.15.33:0/3919756466 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d0007d50 con 0x7f30e81036e0 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.534+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3919756466 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e819d790 con 0x7f30e81036e0 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.534+0000 7f30e4ff9640 1 -- 172.21.15.33:0/3919756466 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0018830 con 0x7f30e81036e0 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.534+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3919756466 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e81a5070 con 0x7f30e81036e0 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.535+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3919756466 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e8103110 con 0x7f30e81036e0 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.536+0000 7f30e4ff9640 1 -- 172.21.15.33:0/3919756466 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f30d00189f0 con 0x7f30e81036e0 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.537+0000 7f30e4ff9640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30d40a3240 0x7f30d40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:22.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.537+0000 7f30e6ffd640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30d40a3240 0x7f30d40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:22.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.537+0000 7f30e4ff9640 1 -- 172.21.15.33:0/3919756466 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f30d00d0000 con 0x7f30e81036e0 2024-09-16T00:25:22.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.538+0000 7f30e6ffd640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30d40a3240 0x7f30d40a5700 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f30d8005fd0 tx=0x7f30d800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:22.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.540+0000 7f30e4ff9640 1 -- 172.21.15.33:0/3919756466 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f30d0014030 con 0x7f30e81036e0 2024-09-16T00:25:22.668 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.11 MB/sec execute 270 sec latency 20271.120 ms 2024-09-16T00:25:22.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.755+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3919756466 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30e8199850 con 0x7f30e81036e0 2024-09-16T00:25:22.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.756+0000 7f30e4ff9640 1 -- 172.21.15.33:0/3919756466 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f30e8199850 con 0x7f30e81036e0 2024-09-16T00:25:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:25:22.756 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:25:22.756 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:25:22.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.758+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3919756466 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30d40a3240 msgr2=0x7f30d40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:22.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.758+0000 7f30ed2ed640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30d40a3240 0x7f30d40a5700 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f30d8005fd0 tx=0x7f30d800b390 comp rx=0 tx=0).stop 2024-09-16T00:25:22.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.758+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3919756466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e81036e0 msgr2=0x7f30e8198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:22.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.758+0000 7f30ed2ed640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e81036e0 0x7f30e8198a30 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f30d000c8c0 tx=0x7f30d000cd80 comp rx=0 tx=0).stop 2024-09-16T00:25:22.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.758+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3919756466 shutdown_connections 2024-09-16T00:25:22.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.758+0000 7f30ed2ed640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f30d40a3240 0x7f30d40a5700 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:22.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.758+0000 7f30ed2ed640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30e81082a0 0x7f30e819cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:22.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.758+0000 7f30ed2ed640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30e81036e0 0x7f30e8198a30 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:22.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.758+0000 7f30ed2ed640 1 --2- 172.21.15.33:0/3919756466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e8102d30 0x7f30e81984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:22.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.758+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3919756466 >> 172.21.15.33:0/3919756466 conn(0x7f30e80fc7d0 msgr2=0x7f30e8106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:22.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.758+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3919756466 shutdown_connections 2024-09-16T00:25:22.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:22.758+0000 7f30ed2ed640 1 -- 172.21.15.33:0/3919756466 wait complete. 2024-09-16T00:25:23.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:23 smithi033 ceph-mon[31243]: pgmap v597: 129 pgs: 129 active+clean; 228 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 2.4 MiB/s wr, 95 op/s 2024-09-16T00:25:23.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:23 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3919756466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:25:23.509 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id ed19ea81 --release --await' 2024-09-16T00:25:23.510 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id ed19ea81 --release --await 2024-09-16T00:25:23.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:23 smithi083 ceph-mon[35242]: pgmap v597: 129 pgs: 129 active+clean; 228 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 2.4 MiB/s wr, 95 op/s 2024-09-16T00:25:23.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:23 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3919756466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:25:23.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621518 10.07 MB/sec execute 271 sec latency 21271.171 ms 2024-09-16T00:25:23.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:23 smithi060 ceph-mon[35185]: pgmap v597: 129 pgs: 129 active+clean; 228 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 2.4 MiB/s wr, 95 op/s 2024-09-16T00:25:23.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:23 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3919756466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:25:23.808 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.274+0000 7f79cffe0640 1 -- 172.21.15.33:0/2991662237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c81027a0 msgr2=0x7f79c81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.274+0000 7f79cffe0640 1 --2- 172.21.15.33:0/2991662237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c81027a0 0x7f79c81067f0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009900 tx=0x7f79bc02a410 comp rx=0 tx=0).stop 2024-09-16T00:25:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.274+0000 7f79cffe0640 1 -- 172.21.15.33:0/2991662237 shutdown_connections 2024-09-16T00:25:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.274+0000 7f79cffe0640 1 --2- 172.21.15.33:0/2991662237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79c8106f20 0x7f79c810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.274+0000 7f79cffe0640 1 --2- 172.21.15.33:0/2991662237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c81027a0 0x7f79c81067f0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.274+0000 7f79cffe0640 1 --2- 172.21.15.33:0/2991662237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79c8101df0 0x7f79c81021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.274+0000 7f79cffe0640 1 -- 172.21.15.33:0/2991662237 >> 172.21.15.33:0/2991662237 conn(0x7f79c80fd8f0 msgr2=0x7f79c80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.274+0000 7f79cffe0640 1 -- 172.21.15.33:0/2991662237 shutdown_connections 2024-09-16T00:25:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.274+0000 7f79cffe0640 1 -- 172.21.15.33:0/2991662237 wait complete. 2024-09-16T00:25:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cffe0640 1 Processor -- start 2024-09-16T00:25:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cffe0640 1 -- start start 2024-09-16T00:25:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cffe0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c8101df0 0x7f79c8198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cffe0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79c81027a0 0x7f79c81989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cffe0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79c8106f20 0x7f79c819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cffe0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8111f80 con 0x7f79c8101df0 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cffe0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8111e00 con 0x7f79c8106f20 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cffe0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8112100 con 0x7f79c81027a0 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cdd7c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c8101df0 0x7f79c8198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cdd7c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c8101df0 0x7f79c8198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52976/0 (socket says 172.21.15.33:52976) 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cdd7c640 1 -- 172.21.15.33:0/4195038281 learned_addr learned my addr 172.21.15.33:0/4195038281 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cd57b640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79c81027a0 0x7f79c81989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79ce57d640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79c8106f20 0x7f79c819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cdd7c640 1 -- 172.21.15.33:0/4195038281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79c81027a0 msgr2=0x7f79c81989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79c81027a0 0x7f79c81989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cdd7c640 1 -- 172.21.15.33:0/4195038281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79c8106f20 msgr2=0x7f79c819cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79c8106f20 0x7f79c819cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.275+0000 7f79cdd7c640 1 -- 172.21.15.33:0/4195038281 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c819d410 con 0x7f79c8101df0 2024-09-16T00:25:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.276+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c8101df0 0x7f79c8198460 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f79c000d770 tx=0x7f79c000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:24.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.276+0000 7f79c6ffd640 1 -- 172.21.15.33:0/4195038281 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c0017070 con 0x7f79c8101df0 2024-09-16T00:25:24.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.276+0000 7f79cffe0640 1 -- 172.21.15.33:0/4195038281 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c819d700 con 0x7f79c8101df0 2024-09-16T00:25:24.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.276+0000 7f79c6ffd640 1 -- 172.21.15.33:0/4195038281 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c0005500 con 0x7f79c8101df0 2024-09-16T00:25:24.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.276+0000 7f79c6ffd640 1 -- 172.21.15.33:0/4195038281 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c0004e00 con 0x7f79c8101df0 2024-09-16T00:25:24.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.276+0000 7f79cffe0640 1 -- 172.21.15.33:0/4195038281 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c81a4fe0 con 0x7f79c8101df0 2024-09-16T00:25:24.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.278+0000 7f79c6ffd640 1 -- 172.21.15.33:0/4195038281 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f79c0009c30 con 0x7f79c8101df0 2024-09-16T00:25:24.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.279+0000 7f79c6ffd640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f79b00a3110 0x7f79b00a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.279+0000 7f79cd57b640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f79b00a3110 0x7f79b00a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.279+0000 7f79c6ffd640 1 -- 172.21.15.33:0/4195038281 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f79c00cf3e0 con 0x7f79c8101df0 2024-09-16T00:25:24.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.279+0000 7f79d23b9740 1 Processor -- start 2024-09-16T00:25:24.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.279+0000 7f79d23b9740 1 -- start start 2024-09-16T00:25:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79d23b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0bf4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79d23b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0bf4860 0x5602e0c03110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79d23b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0c03650 0x5602e0c05a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79d23b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5602e0b59620 con 0x5602e0bf4860 2024-09-16T00:25:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79cd57b640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f79b00a3110 0x7f79b00a55d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009230 tx=0x7f79bc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79d23b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5602e0b00e50 con 0x5602e0bf3f40 2024-09-16T00:25:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79d23b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5602e0b555e0 con 0x5602e0c03650 2024-09-16T00:25:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79cd57b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0bf4860 0x5602e0c03110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79cd57b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0bf4860 0x5602e0c03110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52988/0 (socket says 172.21.15.33:52988) 2024-09-16T00:25:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79cd57b640 1 -- 172.21.15.33:0/4066404407 learned_addr learned my addr 172.21.15.33:0/4066404407 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/4066404407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0bf4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79ce57d640 1 --2- 172.21.15.33:0/4066404407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0c03650 0x5602e0c05a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79cd57b640 1 -- 172.21.15.33:0/4066404407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0c03650 msgr2=0x5602e0c05a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79cd57b640 1 --2- 172.21.15.33:0/4066404407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0c03650 0x5602e0c05a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79cd57b640 1 -- 172.21.15.33:0/4066404407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 msgr2=0x5602e0bf4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79cd57b640 1 --2- 172.21.15.33:0/4066404407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0bf4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79cd57b640 1 -- 172.21.15.33:0/4066404407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602e0af1d40 con 0x5602e0bf4860 2024-09-16T00:25:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/4066404407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0bf4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:25:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.280+0000 7f79cd57b640 1 --2- 172.21.15.33:0/4066404407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0bf4860 0x5602e0c03110 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009900 tx=0x7f79bc03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd00caed4a50e12 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.281+0000 7f79affff640 1 -- 172.21.15.33:0/4066404407 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc035040 con 0x5602e0bf4860 2024-09-16T00:25:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.281+0000 7f79affff640 1 -- 172.21.15.33:0/4066404407 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79bc04e070 con 0x5602e0bf4860 2024-09-16T00:25:24.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.281+0000 7f79affff640 1 -- 172.21.15.33:0/4066404407 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc0496e0 con 0x5602e0bf4860 2024-09-16T00:25:24.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.282+0000 7f79d23b9740 1 -- 172.21.15.33:0/4066404407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0bf4860 msgr2=0x5602e0c03110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:24.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.282+0000 7f79d23b9740 1 --2- 172.21.15.33:0/4066404407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0bf4860 0x5602e0c03110 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009900 tx=0x7f79bc03d000 comp rx=0 tx=0).stop 2024-09-16T00:25:24.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.282+0000 7f79d23b9740 1 -- 172.21.15.33:0/4066404407 shutdown_connections 2024-09-16T00:25:24.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.282+0000 7f79d23b9740 1 --2- 172.21.15.33:0/4066404407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0c03650 0x5602e0c05a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.282+0000 7f79d23b9740 1 --2- 172.21.15.33:0/4066404407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0bf4860 0x5602e0c03110 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.282+0000 7f79d23b9740 1 --2- 172.21.15.33:0/4066404407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0bf4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.282+0000 7f79d23b9740 1 -- 172.21.15.33:0/4066404407 >> 172.21.15.33:0/4066404407 conn(0x5602e0bf05f0 msgr2=0x5602e0bf09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:24.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.282+0000 7f79d23b9740 1 -- 172.21.15.33:0/4066404407 shutdown_connections 2024-09-16T00:25:24.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.282+0000 7f79d23b9740 1 -- 172.21.15.33:0/4066404407 wait complete. 2024-09-16T00:25:24.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.282+0000 7f79d23b9740 1 -- start start 2024-09-16T00:25:24.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.282+0000 7f79d23b9740 1 Processor -- start 2024-09-16T00:25:24.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79d23b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0c005a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79d23b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf4860 0x5602e0c00ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79d23b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c01020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79d23b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5602e0c06220 con 0x5602e0c03650 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79d23b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5602e0ae1f10 con 0x5602e0bf3f40 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79d23b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5602e0c063a0 con 0x5602e0bf4860 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79ce57d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c01020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79d23b9740 10 client.0 _ll_register_callbacks cb 0x5602e0b05db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79ce57d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c01020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52990/0 (socket says 172.21.15.33:52990) 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79ce57d640 1 -- 172.21.15.33:0/3263656502 learned_addr learned my addr 172.21.15.33:0/3263656502 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79cd57b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf4860 0x5602e0c00ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0c005a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79ce57d640 1 -- 172.21.15.33:0/3263656502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf4860 msgr2=0x5602e0c00ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79ce57d640 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf4860 0x5602e0c00ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79ce57d640 1 -- 172.21.15.33:0/3263656502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 msgr2=0x5602e0c005a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79ce57d640 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0c005a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79ce57d640 1 -- 172.21.15.33:0/3263656502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602e0868580 con 0x5602e0c03650 2024-09-16T00:25:24.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79cd57b640 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf4860 0x5602e0c00ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.283+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0c005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.284+0000 7f79ce57d640 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c01020 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f79b80126a0 tx=0x7f79b8012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a841d13a529c99 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.284+0000 7f79ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.284+0000 7f79ad7fa640 1 -- 172.21.15.33:0/3263656502 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b8010d60 con 0x5602e0c03650 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.284+0000 7f79ad7fa640 1 -- 172.21.15.33:0/3263656502 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f79b8003970 con 0x5602e0c03650 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.284+0000 7f79d23b9740 1 -- 172.21.15.33:0/3263656502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79bc04e070 con 0x5602e0c03650 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.284+0000 7f79ad7fa640 1 -- 172.21.15.33:0/3263656502 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f79b8025050 con 0x5602e0c03650 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.284+0000 7f79d23b9740 10 client.16902 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.284+0000 7f79d23b9740 1 -- 172.21.15.33:0/3263656502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5602e0932960 con 0x5602e0c03650 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.284+0000 7f79ad7fa640 1 -- 172.21.15.33:0/3263656502 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79bc04e070 con 0x5602e0c03650 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.285+0000 7f79ad7fa640 1 -- 172.21.15.33:0/3263656502 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f79b8019030 con 0x5602e0c03650 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.285+0000 7f79d23b9740 10 client.16902 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.285+0000 7f79d23b9740 10 client.16902 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:25:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.285+0000 7f79d23b9740 20 client.16902 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.285+0000 7f79d23b9740 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5602e0c13310 0x5602e0c15770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.285+0000 7f79d23b9740 4 client.16902 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:25:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.285+0000 7f79d23b9740 1 -- 172.21.15.33:0/3263656502 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602e0b66b90 con 0x5602e0c13310 2024-09-16T00:25:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.285+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5602e0c13310 0x5602e0c15770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.285+0000 7f79ad7fa640 10 client.16902 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.285+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5602e0c13310 0x5602e0c15770 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9698707189d6efe server_cookie=e3f25e54ea583de0 in_seq=0 out_seq=0 2024-09-16T00:25:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.286+0000 7f79ad7fa640 1 -- 172.21.15.33:0/3263656502 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5602e0b66b90 con 0x5602e0c13310 2024-09-16T00:25:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.286+0000 7f79ad7fa640 10 client.16902 handle_command_reply: tid=0 2024-09-16T00:25:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.286+0000 7f79d23b9740 1 client.16902 shutdown 2024-09-16T00:25:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.286+0000 7f79d23b9740 1 -- 172.21.15.33:0/3263656502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 msgr2=0x5602e0c01020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.286+0000 7f79d23b9740 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c01020 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f79b80126a0 tx=0x7f79b8012b60 comp rx=0 tx=0).stop 2024-09-16T00:25:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.286+0000 7f79d23b9740 1 -- 172.21.15.33:0/3263656502 shutdown_connections 2024-09-16T00:25:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.286+0000 7f79d23b9740 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5602e0c13310 0x5602e0c15770 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.286+0000 7f79d23b9740 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c01020 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.286+0000 7f79d23b9740 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf4860 0x5602e0c00ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.286+0000 7f79ad7fa640 0 client.16902 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.286+0000 7f79d23b9740 1 --2- 172.21.15.33:0/3263656502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0c005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.286+0000 7f79d23b9740 1 -- 172.21.15.33:0/3263656502 >> 172.21.15.33:0/3263656502 conn(0x5602e0bf05f0 msgr2=0x5602e0bf6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.287+0000 7f79d23b9740 1 -- 172.21.15.33:0/3263656502 shutdown_connections 2024-09-16T00:25:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.287+0000 7f79d23b9740 1 -- 172.21.15.33:0/3263656502 wait complete. 2024-09-16T00:25:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.287+0000 7f79d23b9740 20 client.16902 trim_cache size 0 max 16384 2024-09-16T00:25:24.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.304+0000 7f79d23b9740 1 Processor -- start 2024-09-16T00:25:24.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.304+0000 7f79d23b9740 1 -- start start 2024-09-16T00:25:24.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79d23b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0c02470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79d23b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf4860 0x5602e0c1b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79d23b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c1b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79d23b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f79c81af400 con 0x5602e0c03650 2024-09-16T00:25:24.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79d23b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f79c81accd0 con 0x5602e0bf3f40 2024-09-16T00:25:24.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79d23b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f79c81aab20 con 0x5602e0bf4860 2024-09-16T00:25:24.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79cdd7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0c02470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79ce57d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c1b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79cdd7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0c02470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:33362/0 (socket says 172.21.15.33:33362) 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79cdd7c640 1 -- 172.21.15.33:0/2992615688 learned_addr learned my addr 172.21.15.33:0/2992615688 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79cd57b640 1 --2- 172.21.15.33:0/2992615688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf4860 0x5602e0c1b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79ce57d640 1 -- 172.21.15.33:0/2992615688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf4860 msgr2=0x5602e0c1b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79ce57d640 1 --2- 172.21.15.33:0/2992615688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf4860 0x5602e0c1b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79ce57d640 1 -- 172.21.15.33:0/2992615688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 msgr2=0x5602e0c02470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79ce57d640 1 --2- 172.21.15.33:0/2992615688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0c02470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79ce57d640 1 -- 172.21.15.33:0/2992615688 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602e0bf0090 con 0x5602e0c03650 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79cd57b640 1 --2- 172.21.15.33:0/2992615688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf4860 0x5602e0c1b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/2992615688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0c02470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.305+0000 7f79ce57d640 1 --2- 172.21.15.33:0/2992615688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c1b5d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f79b8000f80 tx=0x7f79b8005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0160e1a0fd7e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.306+0000 7f79af7fe640 1 -- 172.21.15.33:0/2992615688 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b8010b60 con 0x5602e0c03650 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.306+0000 7f79af7fe640 1 -- 172.21.15.33:0/2992615688 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79b802e070 con 0x5602e0c03650 2024-09-16T00:25:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.306+0000 7f79af7fe640 1 -- 172.21.15.33:0/2992615688 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b800ed80 con 0x5602e0c03650 2024-09-16T00:25:24.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.306+0000 7f79d23b9740 1 -- 172.21.15.33:0/2992615688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 msgr2=0x5602e0c1b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:24.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.306+0000 7f79d23b9740 1 --2- 172.21.15.33:0/2992615688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c1b5d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f79b8000f80 tx=0x7f79b8005fb0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.307+0000 7f79d23b9740 1 -- 172.21.15.33:0/2992615688 shutdown_connections 2024-09-16T00:25:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.307+0000 7f79d23b9740 1 --2- 172.21.15.33:0/2992615688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c1b5d0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.307+0000 7f79d23b9740 1 --2- 172.21.15.33:0/2992615688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf4860 0x5602e0c1b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.307+0000 7f79d23b9740 1 --2- 172.21.15.33:0/2992615688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf3f40 0x5602e0c02470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.307+0000 7f79d23b9740 1 -- 172.21.15.33:0/2992615688 >> 172.21.15.33:0/2992615688 conn(0x5602e0bf05f0 msgr2=0x5602e0bef750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.307+0000 7f79d23b9740 1 -- 172.21.15.33:0/2992615688 shutdown_connections 2024-09-16T00:25:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.307+0000 7f79d23b9740 1 -- 172.21.15.33:0/2992615688 wait complete. 2024-09-16T00:25:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.307+0000 7f79d23b9740 1 -- start start 2024-09-16T00:25:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.307+0000 7f79d23b9740 1 Processor -- start 2024-09-16T00:25:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79d23b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf3f40 0x5602e0c0bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79d23b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf4860 0x5602e0c07e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79d23b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c08350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79ce57d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c08350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79cdd7c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf3f40 0x5602e0c0bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79ce57d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c08350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53004/0 (socket says 172.21.15.33:53004) 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79ce57d640 1 -- 172.21.15.33:0/3968364622 learned_addr learned my addr 172.21.15.33:0/3968364622 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79d23b9740 1 -- 172.21.15.33:0/3968364622 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f79c81ab190 con 0x5602e0c03650 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79d23b9740 1 -- 172.21.15.33:0/3968364622 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f79c81ac600 con 0x5602e0bf4860 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79d23b9740 1 -- 172.21.15.33:0/3968364622 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f79c81ab860 con 0x5602e0bf3f40 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79d23b9740 10 client.0 _ll_register_callbacks cb 0x7f79b8009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79cd57b640 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf4860 0x5602e0c07e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79ce57d640 1 -- 172.21.15.33:0/3968364622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf3f40 msgr2=0x5602e0c0bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79ce57d640 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf3f40 0x5602e0c0bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79ce57d640 1 -- 172.21.15.33:0/3968364622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf4860 msgr2=0x5602e0c07e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79ce57d640 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf4860 0x5602e0c07e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79ce57d640 1 -- 172.21.15.33:0/3968364622 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602e0c11eb0 con 0x5602e0c03650 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.308+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf3f40 0x5602e0c0bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79ce57d640 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c08350 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f79b802b040 tx=0x7f79b8010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76ca3bf0d32368c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79ad7fa640 1 -- 172.21.15.33:0/3968364622 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b800ea00 con 0x5602e0c03650 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79ad7fa640 1 -- 172.21.15.33:0/3968364622 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f79b800eba0 con 0x5602e0c03650 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79d23b9740 1 -- 172.21.15.33:0/3968364622 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79b802e070 con 0x5602e0c03650 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79ad7fa640 1 -- 172.21.15.33:0/3968364622 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f79b8043050 con 0x5602e0c03650 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79d23b9740 10 client.16908 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79d23b9740 1 -- 172.21.15.33:0/3968364622 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5602e0c7ec10 con 0x5602e0c03650 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79ad7fa640 1 -- 172.21.15.33:0/3968364622 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79b802e070 con 0x5602e0c03650 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79ad7fa640 1 -- 172.21.15.33:0/3968364622 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f79b8019040 con 0x5602e0c03650 2024-09-16T00:25:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79d23b9740 10 client.16908 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:24.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79d23b9740 10 client.16908 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:25:24.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79d23b9740 20 client.16908 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:24.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79d23b9740 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5602e0c7f1c0 0x5602e0c815d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:24.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79d23b9740 4 client.16908 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ed19ea81", "release": true, "await": true}] 2024-09-16T00:25:24.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.309+0000 7f79d23b9740 1 -- 172.21.15.33:0/3968364622 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ed19ea81", "release": true, "await": true}) -- 0x5602e0b610a0 con 0x5602e0c7f1c0 2024-09-16T00:25:24.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.310+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5602e0c7f1c0 0x5602e0c815d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:24.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.310+0000 7f79ad7fa640 10 client.16908 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:24.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.310+0000 7f79cdd7c640 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5602e0c7f1c0 0x5602e0c815d0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73b3e4ed80277414 server_cookie=ae69515c4f169a55 in_seq=0 out_seq=0 2024-09-16T00:25:24.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.320+0000 7f79ad7fa640 1 -- 172.21.15.33:0/3968364622 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5602e0b610a0 con 0x5602e0c7f1c0 2024-09-16T00:25:24.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.320+0000 7f79ad7fa640 10 client.16908 handle_command_reply: tid=0 2024-09-16T00:25:24.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.320+0000 7f79d23b9740 1 client.16908 shutdown 2024-09-16T00:25:24.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.321+0000 7f79d23b9740 1 -- 172.21.15.33:0/3968364622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 msgr2=0x5602e0c08350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:24.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.321+0000 7f79d23b9740 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c08350 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f79b802b040 tx=0x7f79b8010740 comp rx=0 tx=0).stop 2024-09-16T00:25:24.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.321+0000 7f79d23b9740 1 -- 172.21.15.33:0/3968364622 shutdown_connections 2024-09-16T00:25:24.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.321+0000 7f79d23b9740 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5602e0c7f1c0 0x5602e0c815d0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.321+0000 7f79d23b9740 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602e0c03650 0x5602e0c08350 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.321+0000 7f79d23b9740 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5602e0bf4860 0x5602e0c07e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.321+0000 7f79d23b9740 1 --2- 172.21.15.33:0/3968364622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602e0bf3f40 0x5602e0c0bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.321+0000 7f79d23b9740 1 -- 172.21.15.33:0/3968364622 >> 172.21.15.33:0/3968364622 conn(0x5602e0bf05f0 msgr2=0x5602e0beef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:24.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.321+0000 7f79ad7fa640 0 client.16908 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:24.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.321+0000 7f79d23b9740 1 -- 172.21.15.33:0/3968364622 shutdown_connections 2024-09-16T00:25:24.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.321+0000 7f79d23b9740 1 -- 172.21.15.33:0/3968364622 wait complete. 2024-09-16T00:25:24.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.321+0000 7f79d23b9740 20 client.16908 trim_cache size 0 max 16384 2024-09-16T00:25:24.323 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:25:24.323 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:25:24.323 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:25:24.323 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 28, 2024-09-16T00:25:24.323 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:25:24.323 INFO:teuthology.orchestra.run.smithi033.stdout: "ed19ea81": { 2024-09-16T00:25:24.323 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 28, 2024-09-16T00:25:24.323 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:25:24.323 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 143.7, 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:24.324 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:24.325 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:24.325 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:24.325 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:24.325 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:25:24.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.322+0000 7f79cffe0640 1 -- 172.21.15.33:0/4195038281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f79b00a3110 msgr2=0x7f79b00a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:24.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.322+0000 7f79cffe0640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f79b00a3110 0x7f79b00a55d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009230 tx=0x7f79bc0058d0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.322+0000 7f79cffe0640 1 -- 172.21.15.33:0/4195038281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c8101df0 msgr2=0x7f79c8198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:24.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.322+0000 7f79cffe0640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c8101df0 0x7f79c8198460 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f79c000d770 tx=0x7f79c000dc30 comp rx=0 tx=0).stop 2024-09-16T00:25:24.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.322+0000 7f79cffe0640 1 -- 172.21.15.33:0/4195038281 shutdown_connections 2024-09-16T00:25:24.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.322+0000 7f79cffe0640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f79b00a3110 0x7f79b00a55d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.322+0000 7f79cffe0640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79c8106f20 0x7f79c819cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.322+0000 7f79cffe0640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79c81027a0 0x7f79c81989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.322+0000 7f79cffe0640 1 --2- 172.21.15.33:0/4195038281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79c8101df0 0x7f79c8198460 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:24.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.322+0000 7f79cffe0640 1 -- 172.21.15.33:0/4195038281 >> 172.21.15.33:0/4195038281 conn(0x7f79c80fd8f0 msgr2=0x7f79c80fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:24.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.322+0000 7f79cffe0640 1 -- 172.21.15.33:0/4195038281 shutdown_connections 2024-09-16T00:25:24.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:24.322+0000 7f79cffe0640 1 -- 172.21.15.33:0/4195038281 wait complete. 2024-09-16T00:25:24.668 INFO:tasks.workunit.client.0.smithi033.stdout: 1 621970 10.04 MB/sec execute 272 sec latency 21925.800 ms 2024-09-16T00:25:25.081 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ed19ea81', total seconds elapsed: 27.0 2024-09-16T00:25:25.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:25 smithi083 ceph-mon[35242]: pgmap v598: 129 pgs: 129 active+clean; 227 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 2.2 MiB/s wr, 88 op/s 2024-09-16T00:25:25.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 625230 10.07 MB/sec execute 273 sec latency 11.434 ms 2024-09-16T00:25:25.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:25 smithi060 ceph-mon[35185]: pgmap v598: 129 pgs: 129 active+clean; 227 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 2.2 MiB/s wr, 88 op/s 2024-09-16T00:25:25.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:25 smithi033 ceph-mon[31243]: pgmap v598: 129 pgs: 129 active+clean; 227 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 2.2 MiB/s wr, 88 op/s 2024-09-16T00:25:26.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 628097 10.08 MB/sec execute 274 sec latency 16.850 ms 2024-09-16T00:25:27.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:27 smithi083 ceph-mon[35242]: pgmap v599: 129 pgs: 129 active+clean; 227 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 2.4 MiB/s wr, 109 op/s 2024-09-16T00:25:27.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 629771 10.08 MB/sec execute 275 sec latency 49.644 ms 2024-09-16T00:25:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:27 smithi060 ceph-mon[35185]: pgmap v599: 129 pgs: 129 active+clean; 227 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 2.4 MiB/s wr, 109 op/s 2024-09-16T00:25:27.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:27 smithi033 ceph-mon[31243]: pgmap v599: 129 pgs: 129 active+clean; 227 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 2.4 MiB/s wr, 109 op/s 2024-09-16T00:25:28.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 632833 10.10 MB/sec execute 276 sec latency 10.493 ms 2024-09-16T00:25:29.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:29 smithi083 ceph-mon[35242]: pgmap v600: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-09-16T00:25:29.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 635952 10.14 MB/sec execute 277 sec latency 16.328 ms 2024-09-16T00:25:29.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:29 smithi060 ceph-mon[35185]: pgmap v600: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-09-16T00:25:29.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:29 smithi033 ceph-mon[31243]: pgmap v600: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-09-16T00:25:30.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 638681 10.15 MB/sec execute 278 sec latency 15.981 ms 2024-09-16T00:25:31.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:31 smithi083 ceph-mon[35242]: pgmap v601: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 2.3 MiB/s wr, 110 op/s 2024-09-16T00:25:31.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 640710 10.15 MB/sec execute 279 sec latency 62.408 ms 2024-09-16T00:25:31.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:31 smithi060 ceph-mon[35185]: pgmap v601: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 2.3 MiB/s wr, 110 op/s 2024-09-16T00:25:31.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:31 smithi033 ceph-mon[31243]: pgmap v601: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 2.3 MiB/s wr, 110 op/s 2024-09-16T00:25:32.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 643217 10.16 MB/sec execute 280 sec latency 65.824 ms 2024-09-16T00:25:33.630 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:33 smithi033 ceph-mon[31243]: pgmap v602: 129 pgs: 129 active+clean; 231 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 136 op/s 2024-09-16T00:25:33.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:33 smithi083 ceph-mon[35242]: pgmap v602: 129 pgs: 129 active+clean; 231 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 136 op/s 2024-09-16T00:25:33.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 646114 10.18 MB/sec execute 281 sec latency 20.319 ms 2024-09-16T00:25:33.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:33 smithi060 ceph-mon[35185]: pgmap v602: 129 pgs: 129 active+clean; 231 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 136 op/s 2024-09-16T00:25:34.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 648689 10.18 MB/sec execute 282 sec latency 17.117 ms 2024-09-16T00:25:35.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:35 smithi083 ceph-mon[35242]: pgmap v603: 129 pgs: 129 active+clean; 237 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 217 op/s 2024-09-16T00:25:35.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 650819 10.20 MB/sec execute 283 sec latency 43.106 ms 2024-09-16T00:25:35.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:35 smithi060 ceph-mon[35185]: pgmap v603: 129 pgs: 129 active+clean; 237 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 217 op/s 2024-09-16T00:25:35.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:35 smithi033 ceph-mon[31243]: pgmap v603: 129 pgs: 129 active+clean; 237 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 217 op/s 2024-09-16T00:25:36.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 653880 10.22 MB/sec execute 284 sec latency 11.006 ms 2024-09-16T00:25:37.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:37 smithi083 ceph-mon[35242]: pgmap v604: 129 pgs: 129 active+clean; 236 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 234 op/s 2024-09-16T00:25:37.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 656990 10.25 MB/sec execute 285 sec latency 25.097 ms 2024-09-16T00:25:37.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:37 smithi060 ceph-mon[35185]: pgmap v604: 129 pgs: 129 active+clean; 236 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 234 op/s 2024-09-16T00:25:37.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:37 smithi033 ceph-mon[31243]: pgmap v604: 129 pgs: 129 active+clean; 236 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 234 op/s 2024-09-16T00:25:38.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 659462 10.25 MB/sec execute 286 sec latency 17.391 ms 2024-09-16T00:25:39.016 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:25:39.312 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:39.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:39 smithi083 ceph-mon[35242]: pgmap v605: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 199 op/s 2024-09-16T00:25:39.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 661588 10.26 MB/sec execute 287 sec latency 41.471 ms 2024-09-16T00:25:39.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:39 smithi033 ceph-mon[31243]: pgmap v605: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 199 op/s 2024-09-16T00:25:39.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:39 smithi060 ceph-mon[35185]: pgmap v605: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 199 op/s 2024-09-16T00:25:39.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.824+0000 7f5431b63640 1 -- 172.21.15.33:0/374045437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f542c102570 msgr2=0x7f542c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:39.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.824+0000 7f5431b63640 1 --2- 172.21.15.33:0/374045437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f542c102570 0x7f542c1029b0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5420009900 tx=0x7f542002a480 comp rx=0 tx=0).stop 2024-09-16T00:25:39.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.824+0000 7f5431b63640 1 -- 172.21.15.33:0/374045437 shutdown_connections 2024-09-16T00:25:39.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.824+0000 7f5431b63640 1 --2- 172.21.15.33:0/374045437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f542c102ef0 0x7f542c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.824+0000 7f5431b63640 1 --2- 172.21.15.33:0/374045437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f542c102570 0x7f542c1029b0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.824+0000 7f5431b63640 1 --2- 172.21.15.33:0/374045437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f542c100e20 0x7f542c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.824+0000 7f5431b63640 1 -- 172.21.15.33:0/374045437 >> 172.21.15.33:0/374045437 conn(0x7f542c0fc7d0 msgr2=0x7f542c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:39.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.824+0000 7f5431b63640 1 -- 172.21.15.33:0/374045437 shutdown_connections 2024-09-16T00:25:39.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.824+0000 7f5431b63640 1 -- 172.21.15.33:0/374045437 wait complete. 2024-09-16T00:25:39.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.824+0000 7f5431b63640 1 Processor -- start 2024-09-16T00:25:39.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.824+0000 7f5431b63640 1 -- start start 2024-09-16T00:25:39.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f5431b63640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f542c100e20 0x7f542c110640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f5431b63640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f542c102570 0x7f542c110b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f5431b63640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f542c102ef0 0x7f542c1110c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f5431b63640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f542c06a980 con 0x7f542c102ef0 2024-09-16T00:25:39.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f5431b63640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f542c06a800 con 0x7f542c102570 2024-09-16T00:25:39.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f5431b63640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f542c06ab00 con 0x7f542c100e20 2024-09-16T00:25:39.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f542bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f542c102ef0 0x7f542c1110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f542bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f542c102ef0 0x7f542c1110c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42656/0 (socket says 172.21.15.33:42656) 2024-09-16T00:25:39.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f542affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f542c102570 0x7f542c110b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f542bfff640 1 -- 172.21.15.33:0/3020081580 learned_addr learned my addr 172.21.15.33:0/3020081580 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:39.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f542b7fe640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f542c100e20 0x7f542c110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f542bfff640 1 -- 172.21.15.33:0/3020081580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f542c100e20 msgr2=0x7f542c110640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:39.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f542bfff640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f542c100e20 0x7f542c110640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f542bfff640 1 -- 172.21.15.33:0/3020081580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f542c102570 msgr2=0x7f542c110b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:39.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f542bfff640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f542c102570 0x7f542c110b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f542bfff640 1 -- 172.21.15.33:0/3020081580 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c1a69a0 con 0x7f542c102ef0 2024-09-16T00:25:39.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f542bfff640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f542c102ef0 0x7f542c1110c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f542400eb10 tx=0x7f542400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:39.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f5428ff9640 1 -- 172.21.15.33:0/3020081580 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424004b20 con 0x7f542c102ef0 2024-09-16T00:25:39.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f5428ff9640 1 -- 172.21.15.33:0/3020081580 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5424004cc0 con 0x7f542c102ef0 2024-09-16T00:25:39.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.825+0000 7f5431b63640 1 -- 172.21.15.33:0/3020081580 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542c1a6c90 con 0x7f542c102ef0 2024-09-16T00:25:39.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.826+0000 7f5428ff9640 1 -- 172.21.15.33:0/3020081580 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424010760 con 0x7f542c102ef0 2024-09-16T00:25:39.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.826+0000 7f5431b63640 1 -- 172.21.15.33:0/3020081580 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f542c1a71a0 con 0x7f542c102ef0 2024-09-16T00:25:39.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.828+0000 7f5428ff9640 1 -- 172.21.15.33:0/3020081580 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5424010900 con 0x7f542c102ef0 2024-09-16T00:25:39.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.828+0000 7f5433f3c740 1 Processor -- start 2024-09-16T00:25:39.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.828+0000 7f5433f3c740 1 -- start start 2024-09-16T00:25:39.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f5433f3c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ed9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f5433f3c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f5433f3c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ee4680 0x55cf56ee6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f5433f3c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cf56e3f2e0 con 0x55cf56ee4680 2024-09-16T00:25:39.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f5433f3c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55cf56de6ce0 con 0x55cf56ed9dc0 2024-09-16T00:25:39.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f5433f3c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cf56e3b2a0 con 0x55cf56ed94a0 2024-09-16T00:25:39.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f542b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f542b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42458/0 (socket says 172.21.15.33:42458) 2024-09-16T00:25:39.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f542b7fe640 1 -- 172.21.15.33:0/2692570074 learned_addr learned my addr 172.21.15.33:0/2692570074 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:39.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f542affd640 1 --2- 172.21.15.33:0/2692570074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ee4680 0x55cf56ee6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f542affd640 1 --2- 172.21.15.33:0/2692570074 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ed9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f5428ff9640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f54140a3240 0x7f54140a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f542b7fe640 1 -- 172.21.15.33:0/2692570074 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 msgr2=0x55cf56ed9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:39.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f542b7fe640 1 --2- 172.21.15.33:0/2692570074 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ed9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f542b7fe640 1 -- 172.21.15.33:0/2692570074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ee4680 msgr2=0x55cf56ee6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:39.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f542b7fe640 1 --2- 172.21.15.33:0/2692570074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ee4680 0x55cf56ee6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.829+0000 7f542b7fe640 1 -- 172.21.15.33:0/2692570074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf56dcb440 con 0x55cf56ed9dc0 2024-09-16T00:25:39.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.830+0000 7f5428ff9640 1 -- 172.21.15.33:0/3020081580 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f54240d03d0 con 0x7f542c102ef0 2024-09-16T00:25:39.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.830+0000 7f542bfff640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f54140a3240 0x7f54140a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.830+0000 7f542b7fe640 1 --2- 172.21.15.33:0/2692570074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee4140 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f541c009cf0 tx=0x7f541c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db6aecab24079da2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.830+0000 7f5411ffb640 1 -- 172.21.15.33:0/2692570074 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c004230 con 0x55cf56ed9dc0 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.830+0000 7f5411ffb640 1 -- 172.21.15.33:0/2692570074 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541c0043d0 con 0x55cf56ed9dc0 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.830+0000 7f5411ffb640 1 -- 172.21.15.33:0/2692570074 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c0198c0 con 0x55cf56ed9dc0 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 -- 172.21.15.33:0/2692570074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 msgr2=0x55cf56ee4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2692570074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee4140 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f541c009cf0 tx=0x7f541c011810 comp rx=0 tx=0).stop 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f542bfff640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f54140a3240 0x7f54140a5700 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5424005ba0 tx=0x7f54240058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 -- 172.21.15.33:0/2692570074 shutdown_connections 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2692570074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ee4680 0x55cf56ee6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2692570074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee4140 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2692570074 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ed9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 -- 172.21.15.33:0/2692570074 >> 172.21.15.33:0/2692570074 conn(0x55cf56ed5a30 msgr2=0x55cf56ed5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 -- 172.21.15.33:0/2692570074 shutdown_connections 2024-09-16T00:25:39.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 -- 172.21.15.33:0/2692570074 wait complete. 2024-09-16T00:25:39.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 -- start start 2024-09-16T00:25:39.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 Processor -- start 2024-09-16T00:25:39.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ee3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ee4680 0x55cf56ee16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cf56ee7250 con 0x55cf56ed9dc0 2024-09-16T00:25:39.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55cf56dfa050 con 0x55cf56ee4680 2024-09-16T00:25:39.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.831+0000 7f5433f3c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cf56ee73d0 con 0x55cf56ed94a0 2024-09-16T00:25:39.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f5433f3c740 10 client.0 _ll_register_callbacks cb 0x55cf56e3a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:39.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f542affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ee3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f542affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ee3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40918/0 (socket says 172.21.15.33:40918) 2024-09-16T00:25:39.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f542affd640 1 -- 172.21.15.33:0/3739673636 learned_addr learned my addr 172.21.15.33:0/3739673636 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:39.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f542b7fe640 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f542affd640 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ee4680 0x55cf56ee16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f542affd640 1 -- 172.21.15.33:0/3739673636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ee4680 msgr2=0x55cf56ee16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:39.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f542affd640 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ee4680 0x55cf56ee16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f542affd640 1 -- 172.21.15.33:0/3739673636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 msgr2=0x55cf56ee1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:39.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f542affd640 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f542affd640 1 -- 172.21.15.33:0/3739673636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf56bb03c0 con 0x55cf56ed94a0 2024-09-16T00:25:39.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f542b7fe640 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:39.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f542affd640 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ee3ed0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5420009900 tx=0x7f54200058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=652f0225cdb17f5f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:39.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f540affd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:25:39.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f540affd640 1 -- 172.21.15.33:0/3739673636 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420035040 con 0x55cf56ed94a0 2024-09-16T00:25:39.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f540affd640 1 -- 172.21.15.33:0/3739673636 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5420003890 con 0x55cf56ed94a0 2024-09-16T00:25:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.832+0000 7f5433f3c740 1 -- 172.21.15.33:0/3739673636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f541c0043d0 con 0x55cf56ed94a0 2024-09-16T00:25:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.833+0000 7f540affd640 1 -- 172.21.15.33:0/3739673636 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5420054050 con 0x55cf56ed94a0 2024-09-16T00:25:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.833+0000 7f5433f3c740 10 client.26356 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.833+0000 7f5433f3c740 1 -- 172.21.15.33:0/3739673636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cf56ef2b20 con 0x55cf56ed94a0 2024-09-16T00:25:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.833+0000 7f540affd640 1 -- 172.21.15.33:0/3739673636 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541c0043d0 con 0x55cf56ed94a0 2024-09-16T00:25:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.833+0000 7f540affd640 1 -- 172.21.15.33:0/3739673636 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f5420031bb0 con 0x55cf56ed94a0 2024-09-16T00:25:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.833+0000 7f5433f3c740 10 client.26356 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.833+0000 7f5433f3c740 10 client.26356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.833+0000 7f5433f3c740 20 client.26356 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.833+0000 7f5433f3c740 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cf56ef3f30 0x55cf56ef6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.833+0000 7f5433f3c740 4 client.26356 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:25:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.833+0000 7f5433f3c740 1 -- 172.21.15.33:0/3739673636 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf56e4c850 con 0x55cf56ef3f30 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.833+0000 7f542b7fe640 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cf56ef3f30 0x55cf56ef6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.834+0000 7f542b7fe640 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cf56ef3f30 0x55cf56ef6320 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3455ab6ef77e142 server_cookie=f9b793e59fc344c5 in_seq=0 out_seq=0 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.834+0000 7f540affd640 10 client.26356 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f540affd640 1 -- 172.21.15.33:0/3739673636 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cf56e4c850 con 0x55cf56ef3f30 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f540affd640 10 client.26356 handle_command_reply: tid=0 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f5433f3c740 1 client.26356 shutdown 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f5433f3c740 1 -- 172.21.15.33:0/3739673636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 msgr2=0x55cf56ee3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f5433f3c740 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ee3ed0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5420009900 tx=0x7f54200058d0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f5433f3c740 1 -- 172.21.15.33:0/3739673636 shutdown_connections 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f5433f3c740 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cf56ef3f30 0x55cf56ef6320 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f5433f3c740 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ee4680 0x55cf56ee16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f5433f3c740 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f5433f3c740 1 --2- 172.21.15.33:0/3739673636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ee3ed0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f5433f3c740 1 -- 172.21.15.33:0/3739673636 >> 172.21.15.33:0/3739673636 conn(0x55cf56ed5a30 msgr2=0x55cf56edb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:39.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f540affd640 0 client.26356 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:39.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.835+0000 7f5433f3c740 1 -- 172.21.15.33:0/3739673636 shutdown_connections 2024-09-16T00:25:39.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.836+0000 7f5433f3c740 1 -- 172.21.15.33:0/3739673636 wait complete. 2024-09-16T00:25:39.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.837+0000 7f5433f3c740 20 client.26356 trim_cache size 0 max 16384 2024-09-16T00:25:39.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.855+0000 7f5433f3c740 1 Processor -- start 2024-09-16T00:25:39.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f5433f3c740 1 -- start start 2024-09-16T00:25:39.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f5433f3c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed94a0 0x55cf56ef04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f5433f3c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 0x55cf56edc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f5433f3c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ee4680 0x55cf56edb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f5433f3c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1b3c10 con 0x55cf56ed9dc0 2024-09-16T00:25:39.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f5433f3c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1b14e0 con 0x55cf56ed94a0 2024-09-16T00:25:39.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f5433f3c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1af330 con 0x55cf56ee4680 2024-09-16T00:25:39.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f542b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 0x55cf56edc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f542b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 0x55cf56edc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42674/0 (socket says 172.21.15.33:42674) 2024-09-16T00:25:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f542b7fe640 1 -- 172.21.15.33:0/2887966413 learned_addr learned my addr 172.21.15.33:0/2887966413 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f542affd640 1 --2- 172.21.15.33:0/2887966413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ee4680 0x55cf56edb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f542affd640 1 --2- 172.21.15.33:0/2887966413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed94a0 0x55cf56ef04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f542b7fe640 1 -- 172.21.15.33:0/2887966413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ee4680 msgr2=0x55cf56edb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f542b7fe640 1 --2- 172.21.15.33:0/2887966413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ee4680 0x55cf56edb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f542b7fe640 1 -- 172.21.15.33:0/2887966413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed94a0 msgr2=0x55cf56ef04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f542b7fe640 1 --2- 172.21.15.33:0/2887966413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed94a0 0x55cf56ef04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f542b7fe640 1 -- 172.21.15.33:0/2887966413 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf56ed5860 con 0x55cf56ed9dc0 2024-09-16T00:25:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.856+0000 7f542b7fe640 1 --2- 172.21.15.33:0/2887966413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 0x55cf56edc870 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f541c019490 tx=0x7f541c0194c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ebe3da19c8be06 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.857+0000 7f54117fa640 1 -- 172.21.15.33:0/2887966413 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c031070 con 0x55cf56ed9dc0 2024-09-16T00:25:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.857+0000 7f54117fa640 1 -- 172.21.15.33:0/2887966413 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541c0035f0 con 0x55cf56ed9dc0 2024-09-16T00:25:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.857+0000 7f54117fa640 1 -- 172.21.15.33:0/2887966413 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c004820 con 0x55cf56ed9dc0 2024-09-16T00:25:39.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.857+0000 7f5433f3c740 1 -- 172.21.15.33:0/2887966413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 msgr2=0x55cf56edc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:39.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.857+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2887966413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 0x55cf56edc870 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f541c019490 tx=0x7f541c0194c0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.857+0000 7f5433f3c740 1 -- 172.21.15.33:0/2887966413 shutdown_connections 2024-09-16T00:25:39.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.857+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2887966413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ee4680 0x55cf56edb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.857+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2887966413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ed9dc0 0x55cf56edc870 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.857+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2887966413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed94a0 0x55cf56ef04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.857+0000 7f5433f3c740 1 -- 172.21.15.33:0/2887966413 >> 172.21.15.33:0/2887966413 conn(0x55cf56ed5a30 msgr2=0x55cf56edaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:39.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.858+0000 7f5433f3c740 1 -- 172.21.15.33:0/2887966413 shutdown_connections 2024-09-16T00:25:39.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.858+0000 7f5433f3c740 1 -- 172.21.15.33:0/2887966413 wait complete. 2024-09-16T00:25:39.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.858+0000 7f5433f3c740 1 -- start start 2024-09-16T00:25:39.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.858+0000 7f5433f3c740 1 Processor -- start 2024-09-16T00:25:39.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.858+0000 7f5433f3c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ee8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.858+0000 7f5433f3c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.858+0000 7f5433f3c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ee4680 0x55cf56ee8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.858+0000 7f5433f3c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1af9a0 con 0x55cf56ee4680 2024-09-16T00:25:39.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.858+0000 7f5433f3c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1b0e10 con 0x55cf56ed9dc0 2024-09-16T00:25:39.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.858+0000 7f5433f3c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1b0070 con 0x55cf56ed94a0 2024-09-16T00:25:39.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.858+0000 7f5433f3c740 10 client.0 _ll_register_callbacks cb 0x7f5420009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:39.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ee4680 0x55cf56ee8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ee4680 0x55cf56ee8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42682/0 (socket says 172.21.15.33:42682) 2024-09-16T00:25:39.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42478/0 (socket says 172.21.15.33:42478) 2024-09-16T00:25:39.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542affd640 1 -- 172.21.15.33:0/2858309010 learned_addr learned my addr 172.21.15.33:0/2858309010 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:39.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542affd640 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ee8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542b7fe640 1 -- 172.21.15.33:0/2858309010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 msgr2=0x55cf56ee8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542b7fe640 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ee8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542b7fe640 1 -- 172.21.15.33:0/2858309010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ee4680 msgr2=0x55cf56ee8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542b7fe640 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ee4680 0x55cf56ee8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542b7fe640 1 -- 172.21.15.33:0/2858309010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf56f5db80 con 0x55cf56ed9dc0 2024-09-16T00:25:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542affd640 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ee4680 0x55cf56ee8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f542b7fe640 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee88a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x55cf56ef2aa0 tx=0x7f541c003ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d92595800061263a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f540affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:25:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f540affd640 1 -- 172.21.15.33:0/2858309010 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c039070 con 0x55cf56ed9dc0 2024-09-16T00:25:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.859+0000 7f540affd640 1 -- 172.21.15.33:0/2858309010 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f541c0039a0 con 0x55cf56ed9dc0 2024-09-16T00:25:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.860+0000 7f5433f3c740 1 -- 172.21.15.33:0/2858309010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f541c0035f0 con 0x55cf56ed9dc0 2024-09-16T00:25:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.860+0000 7f540affd640 1 -- 172.21.15.33:0/2858309010 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f541c031040 con 0x55cf56ed9dc0 2024-09-16T00:25:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.860+0000 7f5433f3c740 10 client.26429 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.860+0000 7f5433f3c740 1 -- 172.21.15.33:0/2858309010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cf56f5ee10 con 0x55cf56ed9dc0 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.860+0000 7f540affd640 1 -- 172.21.15.33:0/2858309010 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541c003bc0 con 0x55cf56ed9dc0 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.860+0000 7f540affd640 1 -- 172.21.15.33:0/2858309010 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f541c03c020 con 0x55cf56ed9dc0 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f5433f3c740 10 client.26429 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f5433f3c740 10 client.26429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f5433f3c740 20 client.26429 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cf56f5f3a0 0x55cf56f617c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f5433f3c740 4 client.26429 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f5433f3c740 1 -- 172.21.15.33:0/2858309010 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f541c0035f0 con 0x55cf56f5f3a0 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f542affd640 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cf56f5f3a0 0x55cf56f617c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f540affd640 10 client.26429 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f542affd640 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cf56f5f3a0 0x55cf56f617c0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4edce3595431dbdd server_cookie=ce072136b411e113 in_seq=0 out_seq=0 2024-09-16T00:25:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f540affd640 1 -- 172.21.15.33:0/2858309010 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f541c0035f0 con 0x55cf56f5f3a0 2024-09-16T00:25:39.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f540affd640 10 client.26429 handle_command_reply: tid=0 2024-09-16T00:25:39.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f5433f3c740 1 client.26429 shutdown 2024-09-16T00:25:39.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.861+0000 7f5433f3c740 1 -- 172.21.15.33:0/2858309010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 msgr2=0x55cf56ee88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:39.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee88a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x55cf56ef2aa0 tx=0x7f541c003ef0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5433f3c740 1 -- 172.21.15.33:0/2858309010 shutdown_connections 2024-09-16T00:25:39.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55cf56f5f3a0 0x55cf56f617c0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf56ee4680 0x55cf56ee8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cf56ed9dc0 0x55cf56ee88a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f540affd640 0 client.26429 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:39.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5433f3c740 1 --2- 172.21.15.33:0/2858309010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cf56ed94a0 0x55cf56ee8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5433f3c740 1 -- 172.21.15.33:0/2858309010 >> 172.21.15.33:0/2858309010 conn(0x55cf56ed5a30 msgr2=0x55cf56ee29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:39.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5433f3c740 1 -- 172.21.15.33:0/2858309010 shutdown_connections 2024-09-16T00:25:39.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5433f3c740 1 -- 172.21.15.33:0/2858309010 wait complete. 2024-09-16T00:25:39.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5433f3c740 20 client.26429 trim_cache size 0 max 16384 2024-09-16T00:25:39.869 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:25:39.869 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:25:39.869 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:25:39.869 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:25:39.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5431b63640 1 -- 172.21.15.33:0/3020081580 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f54140a3240 msgr2=0x7f54140a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:39.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5431b63640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f54140a3240 0x7f54140a5700 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5424005ba0 tx=0x7f54240058d0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5431b63640 1 -- 172.21.15.33:0/3020081580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f542c102ef0 msgr2=0x7f542c1110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:39.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5431b63640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f542c102ef0 0x7f542c1110c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f542400eb10 tx=0x7f542400c2d0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5431b63640 1 -- 172.21.15.33:0/3020081580 shutdown_connections 2024-09-16T00:25:39.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5431b63640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f54140a3240 0x7f54140a5700 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5431b63640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f542c102ef0 0x7f542c1110c0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5431b63640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f542c102570 0x7f542c110b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5431b63640 1 --2- 172.21.15.33:0/3020081580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f542c100e20 0x7f542c110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:39.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.862+0000 7f5431b63640 1 -- 172.21.15.33:0/3020081580 >> 172.21.15.33:0/3020081580 conn(0x7f542c0fc7d0 msgr2=0x7f542c1068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:39.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.863+0000 7f5431b63640 1 -- 172.21.15.33:0/3020081580 shutdown_connections 2024-09-16T00:25:39.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:39.863+0000 7f5431b63640 1 -- 172.21.15.33:0/3020081580 wait complete. 2024-09-16T00:25:40.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 664617 10.28 MB/sec execute 288 sec latency 32.685 ms 2024-09-16T00:25:40.729 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:25:40.729 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:25:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:41.430 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:41 smithi033 ceph-mon[31243]: pgmap v606: 129 pgs: 129 active+clean; 237 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 250 op/s 2024-09-16T00:25:41.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.521+0000 7f1cd8199640 1 -- 172.21.15.33:0/1773911570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cd0101de0 msgr2=0x7f1cd01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:41.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.521+0000 7f1cd8199640 1 --2- 172.21.15.33:0/1773911570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cd0101de0 0x7f1cd01021c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009900 tx=0x7f1ccc02a410 comp rx=0 tx=0).stop 2024-09-16T00:25:41.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.522+0000 7f1cd8199640 1 -- 172.21.15.33:0/1773911570 shutdown_connections 2024-09-16T00:25:41.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.522+0000 7f1cd8199640 1 --2- 172.21.15.33:0/1773911570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd0106f10 0x7f1cd010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:41.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.522+0000 7f1cd8199640 1 --2- 172.21.15.33:0/1773911570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cd0102790 0x7f1cd01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:41.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.522+0000 7f1cd8199640 1 --2- 172.21.15.33:0/1773911570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cd0101de0 0x7f1cd01021c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:41.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.522+0000 7f1cd8199640 1 -- 172.21.15.33:0/1773911570 >> 172.21.15.33:0/1773911570 conn(0x7f1cd00fd8e0 msgr2=0x7f1cd00ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:41.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.522+0000 7f1cd8199640 1 -- 172.21.15.33:0/1773911570 shutdown_connections 2024-09-16T00:25:41.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.522+0000 7f1cd8199640 1 -- 172.21.15.33:0/1773911570 wait complete. 2024-09-16T00:25:41.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.522+0000 7f1cd8199640 1 Processor -- start 2024-09-16T00:25:41.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.522+0000 7f1cd8199640 1 -- start start 2024-09-16T00:25:41.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd8199640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cd0101de0 0x7f1cd00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:41.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd8199640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd0102790 0x7f1cd0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd8199640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cd0106f10 0x7f1cd0074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd8199640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd0111f70 con 0x7f1cd0106f10 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd8199640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd0111df0 con 0x7f1cd0102790 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd8199640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd01120f0 con 0x7f1cd0101de0 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd6736640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cd0106f10 0x7f1cd0074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd5f35640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cd0101de0 0x7f1cd00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd6736640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cd0106f10 0x7f1cd0074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38892/0 (socket says 172.21.15.33:38892) 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd6736640 1 -- 172.21.15.33:0/2375353115 learned_addr learned my addr 172.21.15.33:0/2375353115 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd5734640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd0102790 0x7f1cd0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd6736640 1 -- 172.21.15.33:0/2375353115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cd0101de0 msgr2=0x7f1cd00744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd6736640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cd0101de0 0x7f1cd00744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd6736640 1 -- 172.21.15.33:0/2375353115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd0102790 msgr2=0x7f1cd0074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:41.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd6736640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd0102790 0x7f1cd0074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:41.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd6736640 1 -- 172.21.15.33:0/2375353115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd01a6970 con 0x7f1cd0106f10 2024-09-16T00:25:41.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd6736640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cd0106f10 0x7f1cd0074f70 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009fd0 tx=0x7f1cc800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:41.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cc6ffd640 1 -- 172.21.15.33:0/2375353115 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8017070 con 0x7f1cd0106f10 2024-09-16T00:25:41.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cd8199640 1 -- 172.21.15.33:0/2375353115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd01a6c60 con 0x7f1cd0106f10 2024-09-16T00:25:41.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.523+0000 7f1cc6ffd640 1 -- 172.21.15.33:0/2375353115 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cc8003e50 con 0x7f1cd0106f10 2024-09-16T00:25:41.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.524+0000 7f1cc6ffd640 1 -- 172.21.15.33:0/2375353115 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc80106c0 con 0x7f1cd0106f10 2024-09-16T00:25:41.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.524+0000 7f1cd8199640 1 -- 172.21.15.33:0/2375353115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cd01a71a0 con 0x7f1cd0106f10 2024-09-16T00:25:41.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.525+0000 7f1cd8199640 1 -- 172.21.15.33:0/2375353115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cd01021c0 con 0x7f1cd0106f10 2024-09-16T00:25:41.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.526+0000 7f1cc6ffd640 1 -- 172.21.15.33:0/2375353115 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1cc8003970 con 0x7f1cd0106f10 2024-09-16T00:25:41.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.527+0000 7f1cc6ffd640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1cbc0a32c0 0x7f1cbc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:41.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.527+0000 7f1cd5f35640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1cbc0a32c0 0x7f1cbc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:41.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.527+0000 7f1cc6ffd640 1 -- 172.21.15.33:0/2375353115 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1cc80cfa40 con 0x7f1cd0106f10 2024-09-16T00:25:41.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.528+0000 7f1cd5f35640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1cbc0a32c0 0x7f1cbc0a5780 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009230 tx=0x7f1ccc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:41.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.531+0000 7f1cc6ffd640 1 -- 172.21.15.33:0/2375353115 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1cc809bcb0 con 0x7f1cd0106f10 2024-09-16T00:25:41.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:41 smithi083 ceph-mon[35242]: pgmap v606: 129 pgs: 129 active+clean; 237 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 250 op/s 2024-09-16T00:25:41.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 667729 10.31 MB/sec execute 289 sec latency 39.865 ms 2024-09-16T00:25:41.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:41 smithi060 ceph-mon[35185]: pgmap v606: 129 pgs: 129 active+clean; 237 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 250 op/s 2024-09-16T00:25:41.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.759+0000 7f1cd8199640 1 -- 172.21.15.33:0/2375353115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cd0075a20 con 0x7f1cd0106f10 2024-09-16T00:25:41.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.760+0000 7f1cc6ffd640 1 -- 172.21.15.33:0/2375353115 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f1cd0075a20 con 0x7f1cd0106f10 2024-09-16T00:25:41.761 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:25:41.761 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:25:41.761 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:25:41.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.763+0000 7f1cd8199640 1 -- 172.21.15.33:0/2375353115 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1cbc0a32c0 msgr2=0x7f1cbc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:41.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.763+0000 7f1cd8199640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1cbc0a32c0 0x7f1cbc0a5780 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009230 tx=0x7f1ccc0058d0 comp rx=0 tx=0).stop 2024-09-16T00:25:41.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.763+0000 7f1cd8199640 1 -- 172.21.15.33:0/2375353115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cd0106f10 msgr2=0x7f1cd0074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:41.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.763+0000 7f1cd8199640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cd0106f10 0x7f1cd0074f70 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009fd0 tx=0x7f1cc800b260 comp rx=0 tx=0).stop 2024-09-16T00:25:41.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.763+0000 7f1cd8199640 1 -- 172.21.15.33:0/2375353115 shutdown_connections 2024-09-16T00:25:41.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.763+0000 7f1cd8199640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1cbc0a32c0 0x7f1cbc0a5780 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:41.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.763+0000 7f1cd8199640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cd0106f10 0x7f1cd0074f70 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:41.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.763+0000 7f1cd8199640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd0102790 0x7f1cd0074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:41.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.763+0000 7f1cd8199640 1 --2- 172.21.15.33:0/2375353115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cd0101de0 0x7f1cd00744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:41.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.763+0000 7f1cd8199640 1 -- 172.21.15.33:0/2375353115 >> 172.21.15.33:0/2375353115 conn(0x7f1cd00fd8e0 msgr2=0x7f1cd00feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:41.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.763+0000 7f1cd8199640 1 -- 172.21.15.33:0/2375353115 shutdown_connections 2024-09-16T00:25:41.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:41.763+0000 7f1cd8199640 1 -- 172.21.15.33:0/2375353115 wait complete. 2024-09-16T00:25:42.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:42 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2375353115' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:25:42.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 670553 10.32 MB/sec execute 290 sec latency 29.741 ms 2024-09-16T00:25:42.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:42 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2375353115' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:25:42.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:42 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2375353115' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:25:42.958 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:25:43.246 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:43.658 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:43 smithi033 ceph-mon[31243]: pgmap v607: 129 pgs: 129 active+clean; 242 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 288 op/s 2024-09-16T00:25:43.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:43 smithi083 ceph-mon[35242]: pgmap v607: 129 pgs: 129 active+clean; 242 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 288 op/s 2024-09-16T00:25:43.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 672687 10.32 MB/sec execute 291 sec latency 36.023 ms 2024-09-16T00:25:43.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:43 smithi060 ceph-mon[35185]: pgmap v607: 129 pgs: 129 active+clean; 242 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 288 op/s 2024-09-16T00:25:43.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.753+0000 7f3294389640 1 -- 172.21.15.33:0/2504995291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f328c1009b0 msgr2=0x7f328c100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:43.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.753+0000 7f3294389640 1 --2- 172.21.15.33:0/2504995291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f328c1009b0 0x7f328c100d90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3288009860 tx=0x7f328802a340 comp rx=0 tx=0).stop 2024-09-16T00:25:43.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.753+0000 7f3294389640 1 -- 172.21.15.33:0/2504995291 shutdown_connections 2024-09-16T00:25:43.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.753+0000 7f3294389640 1 --2- 172.21.15.33:0/2504995291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f328c10e4d0 0x7f328c1108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.753+0000 7f3294389640 1 --2- 172.21.15.33:0/2504995291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c101360 0x7f328c10df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.753+0000 7f3294389640 1 --2- 172.21.15.33:0/2504995291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f328c1009b0 0x7f328c100d90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.754+0000 7f3294389640 1 -- 172.21.15.33:0/2504995291 >> 172.21.15.33:0/2504995291 conn(0x7f328c0fc7d0 msgr2=0x7f328c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:43.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.754+0000 7f3294389640 1 -- 172.21.15.33:0/2504995291 shutdown_connections 2024-09-16T00:25:43.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.754+0000 7f3294389640 1 -- 172.21.15.33:0/2504995291 wait complete. 2024-09-16T00:25:43.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.754+0000 7f3294389640 1 Processor -- start 2024-09-16T00:25:43.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.754+0000 7f3294389640 1 -- start start 2024-09-16T00:25:43.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.754+0000 7f3294389640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f328c1009b0 0x7f328c1a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.754+0000 7f3294389640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c101360 0x7f328c1a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.754+0000 7f3294389640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f328c10e4d0 0x7f328c19a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3294389640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f328c114210 con 0x7f328c1009b0 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3294389640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f328c114090 con 0x7f328c10e4d0 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3294389640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f328c114390 con 0x7f328c101360 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3292125640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f328c1009b0 0x7f328c1a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3292125640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f328c1009b0 0x7f328c1a0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38908/0 (socket says 172.21.15.33:38908) 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3292125640 1 -- 172.21.15.33:0/3048275924 learned_addr learned my addr 172.21.15.33:0/3048275924 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3291924640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c101360 0x7f328c1a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3292926640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f328c10e4d0 0x7f328c19a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3292125640 1 -- 172.21.15.33:0/3048275924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c101360 msgr2=0x7f328c1a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3292125640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c101360 0x7f328c1a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3292125640 1 -- 172.21.15.33:0/3048275924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f328c10e4d0 msgr2=0x7f328c19a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3292125640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f328c10e4d0 0x7f328c19a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3292125640 1 -- 172.21.15.33:0/3048275924 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328c19ad30 con 0x7f328c1009b0 2024-09-16T00:25:43.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3292926640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f328c10e4d0 0x7f328c19a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3292125640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f328c1009b0 0x7f328c1a0440 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f3288035040 tx=0x7f3288033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f32837fe640 1 -- 172.21.15.33:0/3048275924 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328803f020 con 0x7f328c1009b0 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3294389640 1 -- 172.21.15.33:0/3048275924 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f328c19afc0 con 0x7f328c1009b0 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f32837fe640 1 -- 172.21.15.33:0/3048275924 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3288032770 con 0x7f328c1009b0 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f32837fe640 1 -- 172.21.15.33:0/3048275924 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3288037040 con 0x7f328c1009b0 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.755+0000 7f3294389640 1 -- 172.21.15.33:0/3048275924 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f328c1a7310 con 0x7f328c1009b0 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.758+0000 7f32837fe640 1 -- 172.21.15.33:0/3048275924 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3288042020 con 0x7f328c1009b0 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.758+0000 7f3296762740 1 Processor -- start 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.758+0000 7f3296762740 1 -- start start 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3296762740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea707b40 0x555bea706090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3296762740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 0x555bea706a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3296762740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea711370 0x555bea713760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3296762740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555bea66c2f0 con 0x555bea7065d0 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3296762740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555bea613cf0 con 0x555bea711370 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3296762740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555bea6682b0 con 0x555bea707b40 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3292125640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 0x555bea706a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f32837fe640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f32780a3240 0x7f32780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3292125640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 0x555bea706a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38910/0 (socket says 172.21.15.33:38910) 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3292125640 1 -- 172.21.15.33:0/3426280441 learned_addr learned my addr 172.21.15.33:0/3426280441 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3291924640 1 --2- 172.21.15.33:0/3426280441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea711370 0x555bea713760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3292926640 1 --2- 172.21.15.33:0/3426280441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea707b40 0x555bea706090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f32837fe640 1 -- 172.21.15.33:0/3048275924 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f32880f0060 con 0x7f328c1009b0 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3291924640 1 -- 172.21.15.33:0/3426280441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea707b40 msgr2=0x555bea706090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3291924640 1 --2- 172.21.15.33:0/3426280441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea707b40 0x555bea706090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3291924640 1 -- 172.21.15.33:0/3426280441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 msgr2=0x555bea706a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3291924640 1 --2- 172.21.15.33:0/3426280441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 0x555bea706a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3291924640 1 -- 172.21.15.33:0/3426280441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bea5f83a0 con 0x555bea711370 2024-09-16T00:25:43.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.759+0000 7f3292926640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f32780a3240 0x7f32780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.760+0000 7f3291924640 1 --2- 172.21.15.33:0/3426280441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea711370 0x555bea713760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f328c19b740 tx=0x7f327c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51ee0bef7eec5c5e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:43.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.760+0000 7f3280dea640 1 -- 172.21.15.33:0/3426280441 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c009d70 con 0x555bea711370 2024-09-16T00:25:43.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.760+0000 7f3280dea640 1 -- 172.21.15.33:0/3426280441 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f327c005020 con 0x555bea711370 2024-09-16T00:25:43.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.760+0000 7f3280dea640 1 -- 172.21.15.33:0/3426280441 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c004b90 con 0x555bea711370 2024-09-16T00:25:43.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 -- 172.21.15.33:0/3426280441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea711370 msgr2=0x555bea713760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:43.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3292926640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f32780a3240 0x7f32780a5700 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3284005fd0 tx=0x7f3284004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:43.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 --2- 172.21.15.33:0/3426280441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea711370 0x555bea713760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f328c19b740 tx=0x7f327c011810 comp rx=0 tx=0).stop 2024-09-16T00:25:43.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 -- 172.21.15.33:0/3426280441 shutdown_connections 2024-09-16T00:25:43.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 --2- 172.21.15.33:0/3426280441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea711370 0x555bea713760 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 --2- 172.21.15.33:0/3426280441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 0x555bea706a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 --2- 172.21.15.33:0/3426280441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea707b40 0x555bea706090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 -- 172.21.15.33:0/3426280441 >> 172.21.15.33:0/3426280441 conn(0x555bea6cf310 msgr2=0x555bea702c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 -- 172.21.15.33:0/3426280441 shutdown_connections 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 -- 172.21.15.33:0/3426280441 wait complete. 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 -- start start 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 Processor -- start 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea7065d0 0x555bea710d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 0x555bea70dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea711370 0x555bea70e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555bea714070 con 0x555bea707b40 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555bea627060 con 0x555bea7065d0 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.761+0000 7f3296762740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555bea7141f0 con 0x555bea711370 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3296762740 10 client.0 _ll_register_callbacks cb 0x555bea37a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:43.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3291924640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 0x555bea70dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3292125640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea7065d0 0x555bea710d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3291924640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 0x555bea70dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38916/0 (socket says 172.21.15.33:38916) 2024-09-16T00:25:43.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3291924640 1 -- 172.21.15.33:0/565020738 learned_addr learned my addr 172.21.15.33:0/565020738 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:43.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3292926640 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea711370 0x555bea70e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3292125640 1 -- 172.21.15.33:0/565020738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea711370 msgr2=0x555bea70e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:43.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3292125640 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea711370 0x555bea70e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3292125640 1 -- 172.21.15.33:0/565020738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 msgr2=0x555bea70dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:43.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3292125640 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 0x555bea70dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3292125640 1 -- 172.21.15.33:0/565020738 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bea3dd3c0 con 0x555bea7065d0 2024-09-16T00:25:43.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3292926640 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea711370 0x555bea70e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:43.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3291924640 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 0x555bea70dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:43.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3292125640 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea7065d0 0x555bea710d30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f32880bc130 tx=0x7f3288005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0c17e5bfb9b11a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3275ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3275ffb640 1 -- 172.21.15.33:0/565020738 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32880bc1a0 con 0x555bea7065d0 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3296762740 1 -- 172.21.15.33:0/565020738 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f327c005020 con 0x555bea7065d0 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.762+0000 7f3275ffb640 1 -- 172.21.15.33:0/565020738 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f32880042d0 con 0x555bea7065d0 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.763+0000 7f3275ffb640 1 -- 172.21.15.33:0/565020738 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f32880c0550 con 0x555bea7065d0 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.763+0000 7f3296762740 10 client.26444 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.763+0000 7f3296762740 1 -- 172.21.15.33:0/565020738 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555bea71fad0 con 0x555bea7065d0 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.763+0000 7f3275ffb640 1 -- 172.21.15.33:0/565020738 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f327c005020 con 0x555bea7065d0 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.763+0000 7f3275ffb640 1 -- 172.21.15.33:0/565020738 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f328803a030 con 0x555bea7065d0 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.763+0000 7f3296762740 10 client.26444 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.763+0000 7f3296762740 10 client.26444 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.763+0000 7f3296762740 20 client.26444 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:43.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.763+0000 7f3296762740 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555bea720f80 0x555bea7233d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.763+0000 7f3296762740 4 client.26444 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:25:43.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.763+0000 7f3296762740 1 -- 172.21.15.33:0/565020738 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bea679860 con 0x555bea720f80 2024-09-16T00:25:43.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.763+0000 7f3291924640 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555bea720f80 0x555bea7233d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.764+0000 7f3291924640 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555bea720f80 0x555bea7233d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bbbe7175391eba3c server_cookie=b88878c8ad448f94 in_seq=0 out_seq=0 2024-09-16T00:25:43.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.764+0000 7f3275ffb640 10 client.26444 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:25:43.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.765+0000 7f3275ffb640 1 -- 172.21.15.33:0/565020738 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555bea679860 con 0x555bea720f80 2024-09-16T00:25:43.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.765+0000 7f3275ffb640 10 client.26444 handle_command_reply: tid=0 2024-09-16T00:25:43.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.765+0000 7f3296762740 1 client.26444 shutdown 2024-09-16T00:25:43.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.766+0000 7f3296762740 1 -- 172.21.15.33:0/565020738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea7065d0 msgr2=0x555bea710d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:43.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.766+0000 7f3296762740 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea7065d0 0x555bea710d30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f32880bc130 tx=0x7f3288005dc0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.766+0000 7f3296762740 1 -- 172.21.15.33:0/565020738 shutdown_connections 2024-09-16T00:25:43.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.766+0000 7f3296762740 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555bea720f80 0x555bea7233d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.766+0000 7f3296762740 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea711370 0x555bea70e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.766+0000 7f3296762740 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 0x555bea70dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.766+0000 7f3275ffb640 0 client.26444 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:25:43.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.766+0000 7f3296762740 1 --2- 172.21.15.33:0/565020738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea7065d0 0x555bea710d30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.766+0000 7f3296762740 1 -- 172.21.15.33:0/565020738 >> 172.21.15.33:0/565020738 conn(0x555bea6cf310 msgr2=0x555bea708960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:43.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.766+0000 7f3296762740 1 -- 172.21.15.33:0/565020738 shutdown_connections 2024-09-16T00:25:43.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.766+0000 7f3296762740 1 -- 172.21.15.33:0/565020738 wait complete. 2024-09-16T00:25:43.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.766+0000 7f3296762740 20 client.26444 trim_cache size 0 max 16384 2024-09-16T00:25:43.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3296762740 1 Processor -- start 2024-09-16T00:25:43.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3296762740 1 -- start start 2024-09-16T00:25:43.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3296762740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 0x555bea707410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3296762740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea707b40 0x555bea701bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3296762740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea711370 0x555bea709120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3296762740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f328c1b3c10 con 0x555bea7065d0 2024-09-16T00:25:43.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3296762740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f328c1b14e0 con 0x555bea711370 2024-09-16T00:25:43.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3296762740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f328c1af330 con 0x555bea707b40 2024-09-16T00:25:43.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3292125640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 0x555bea707410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3291924640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea707b40 0x555bea701bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3292125640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 0x555bea707410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38918/0 (socket says 172.21.15.33:38918) 2024-09-16T00:25:43.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3292125640 1 -- 172.21.15.33:0/1586713342 learned_addr learned my addr 172.21.15.33:0/1586713342 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:43.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.780+0000 7f3292926640 1 --2- 172.21.15.33:0/1586713342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea711370 0x555bea709120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.781+0000 7f3292125640 1 -- 172.21.15.33:0/1586713342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea707b40 msgr2=0x555bea701bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:43.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.781+0000 7f3292125640 1 --2- 172.21.15.33:0/1586713342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea707b40 0x555bea701bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.781+0000 7f3292125640 1 -- 172.21.15.33:0/1586713342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea711370 msgr2=0x555bea709120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:43.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.781+0000 7f3292125640 1 --2- 172.21.15.33:0/1586713342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea711370 0x555bea709120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.781+0000 7f3292125640 1 -- 172.21.15.33:0/1586713342 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bea703960 con 0x555bea7065d0 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.781+0000 7f3292125640 1 --2- 172.21.15.33:0/1586713342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 0x555bea707410 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f32880bbea0 tx=0x7f328802a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67979cbf070abeb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.781+0000 7f3277fff640 1 -- 172.21.15.33:0/1586713342 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32880bc1a0 con 0x555bea7065d0 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.781+0000 7f3277fff640 1 -- 172.21.15.33:0/1586713342 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32880c4e10 con 0x555bea7065d0 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.781+0000 7f3277fff640 1 -- 172.21.15.33:0/1586713342 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32880bbb00 con 0x555bea7065d0 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.782+0000 7f3296762740 1 -- 172.21.15.33:0/1586713342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 msgr2=0x555bea707410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.782+0000 7f3296762740 1 --2- 172.21.15.33:0/1586713342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 0x555bea707410 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f32880bbea0 tx=0x7f328802a340 comp rx=0 tx=0).stop 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.782+0000 7f3296762740 1 -- 172.21.15.33:0/1586713342 shutdown_connections 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.782+0000 7f3296762740 1 --2- 172.21.15.33:0/1586713342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea711370 0x555bea709120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.782+0000 7f3296762740 1 --2- 172.21.15.33:0/1586713342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea707b40 0x555bea701bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.782+0000 7f3296762740 1 --2- 172.21.15.33:0/1586713342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea7065d0 0x555bea707410 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.782+0000 7f3296762740 1 -- 172.21.15.33:0/1586713342 >> 172.21.15.33:0/1586713342 conn(0x555bea6cf310 msgr2=0x555bea7014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.782+0000 7f3296762740 1 -- 172.21.15.33:0/1586713342 shutdown_connections 2024-09-16T00:25:43.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.782+0000 7f3296762740 1 -- 172.21.15.33:0/1586713342 wait complete. 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.782+0000 7f3296762740 1 -- start start 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3296762740 1 Processor -- start 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3296762740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea7065d0 0x555bea719ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3296762740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 0x555bea70ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3296762740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea711370 0x555bea70b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3296762740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f328c1af9a0 con 0x555bea707b40 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3296762740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f328c1b0e10 con 0x555bea7065d0 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3296762740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f328c1b0070 con 0x555bea711370 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3296762740 10 client.0 _ll_register_callbacks cb 0x7f32880bc020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3291924640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 0x555bea70ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3292125640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea7065d0 0x555bea719ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3291924640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 0x555bea70ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38932/0 (socket says 172.21.15.33:38932) 2024-09-16T00:25:43.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3291924640 1 -- 172.21.15.33:0/1077730359 learned_addr learned my addr 172.21.15.33:0/1077730359 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:43.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3292926640 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea711370 0x555bea70b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3291924640 1 -- 172.21.15.33:0/1077730359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea711370 msgr2=0x555bea70b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:43.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3291924640 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea711370 0x555bea70b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3291924640 1 -- 172.21.15.33:0/1077730359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea7065d0 msgr2=0x555bea719ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:43.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3291924640 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea7065d0 0x555bea719ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3291924640 1 -- 172.21.15.33:0/1077730359 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bea70b8f0 con 0x555bea707b40 2024-09-16T00:25:43.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.783+0000 7f3292125640 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea7065d0 0x555bea719ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:43.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.784+0000 7f3291924640 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 0x555bea70ac70 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f327c009bc0 tx=0x7f327c0046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208a26679996b518 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:43.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.784+0000 7f3275ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:43.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.784+0000 7f3275ffb640 1 -- 172.21.15.33:0/1077730359 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c004880 con 0x555bea707b40 2024-09-16T00:25:43.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.784+0000 7f3275ffb640 1 -- 172.21.15.33:0/1077730359 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f327c004a20 con 0x555bea707b40 2024-09-16T00:25:43.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.784+0000 7f3296762740 1 -- 172.21.15.33:0/1077730359 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32880c4e10 con 0x555bea707b40 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.784+0000 7f3275ffb640 1 -- 172.21.15.33:0/1077730359 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f327c03a050 con 0x555bea707b40 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.784+0000 7f3296762740 10 client.16950 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.784+0000 7f3296762740 1 -- 172.21.15.33:0/1077730359 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555bea78bba0 con 0x555bea707b40 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.784+0000 7f3275ffb640 1 -- 172.21.15.33:0/1077730359 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32880c4e10 con 0x555bea707b40 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.784+0000 7f3275ffb640 1 -- 172.21.15.33:0/1077730359 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f327c0249b0 con 0x555bea707b40 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.785+0000 7f3296762740 10 client.16950 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.785+0000 7f3296762740 10 client.16950 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.785+0000 7f3296762740 20 client.16950 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.785+0000 7f3296762740 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555bea78c180 0x555bea78e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.785+0000 7f3296762740 4 client.16950 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.785+0000 7f3296762740 1 -- 172.21.15.33:0/1077730359 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555bea673d70 con 0x555bea78c180 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.785+0000 7f3292125640 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555bea78c180 0x555bea78e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:43.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.785+0000 7f3292125640 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555bea78c180 0x555bea78e5a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b726f49c607193cc server_cookie=22d248ababf96b17 in_seq=0 out_seq=0 2024-09-16T00:25:43.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.785+0000 7f3275ffb640 10 client.16950 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:25:43.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3275ffb640 1 -- 172.21.15.33:0/1077730359 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555bea673d70 con 0x555bea78c180 2024-09-16T00:25:43.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3275ffb640 10 client.16950 handle_command_reply: tid=0 2024-09-16T00:25:43.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3296762740 1 client.16950 shutdown 2024-09-16T00:25:43.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3296762740 1 -- 172.21.15.33:0/1077730359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 msgr2=0x555bea70ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:43.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3296762740 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 0x555bea70ac70 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f327c009bc0 tx=0x7f327c0046f0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3296762740 1 -- 172.21.15.33:0/1077730359 shutdown_connections 2024-09-16T00:25:43.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3296762740 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x555bea78c180 0x555bea78e5a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3296762740 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555bea711370 0x555bea70b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3296762740 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bea707b40 0x555bea70ac70 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3296762740 1 --2- 172.21.15.33:0/1077730359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555bea7065d0 0x555bea719ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3296762740 1 -- 172.21.15.33:0/1077730359 >> 172.21.15.33:0/1077730359 conn(0x555bea6cf310 msgr2=0x555bea71c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:43.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3275ffb640 0 client.16950 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:25:43.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3296762740 1 -- 172.21.15.33:0/1077730359 shutdown_connections 2024-09-16T00:25:43.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3296762740 1 -- 172.21.15.33:0/1077730359 wait complete. 2024-09-16T00:25:43.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.787+0000 7f3296762740 20 client.16950 trim_cache size 0 max 16384 2024-09-16T00:25:43.798 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:25:43.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.788+0000 7f3294389640 1 -- 172.21.15.33:0/3048275924 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f32780a3240 msgr2=0x7f32780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:43.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.788+0000 7f3294389640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f32780a3240 0x7f32780a5700 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3284005fd0 tx=0x7f3284004ab0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.788+0000 7f3294389640 1 -- 172.21.15.33:0/3048275924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f328c1009b0 msgr2=0x7f328c1a0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:43.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.788+0000 7f3294389640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f328c1009b0 0x7f328c1a0440 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f3288035040 tx=0x7f3288033d90 comp rx=0 tx=0).stop 2024-09-16T00:25:43.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.788+0000 7f3294389640 1 -- 172.21.15.33:0/3048275924 shutdown_connections 2024-09-16T00:25:43.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.788+0000 7f3294389640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f32780a3240 0x7f32780a5700 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.788+0000 7f3294389640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f328c10e4d0 0x7f328c19a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.788+0000 7f3294389640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c101360 0x7f328c1a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.788+0000 7f3294389640 1 --2- 172.21.15.33:0/3048275924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f328c1009b0 0x7f328c1a0440 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:43.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.788+0000 7f3294389640 1 -- 172.21.15.33:0/3048275924 >> 172.21.15.33:0/3048275924 conn(0x7f328c0fc7d0 msgr2=0x7f328c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:43.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.788+0000 7f3294389640 1 -- 172.21.15.33:0/3048275924 shutdown_connections 2024-09-16T00:25:43.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:43.788+0000 7f3294389640 1 -- 172.21.15.33:0/3048275924 wait complete. 2024-09-16T00:25:44.594 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:25:44.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 675634 10.34 MB/sec execute 292 sec latency 15.503 ms 2024-09-16T00:25:44.868 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:45.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.387+0000 7f51da8ae640 1 -- 172.21.15.33:0/3502477694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51d4102790 msgr2=0x7f51d41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:45.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.387+0000 7f51da8ae640 1 --2- 172.21.15.33:0/3502477694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51d4102790 0x7f51d41067e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009900 tx=0x7f51c802a410 comp rx=0 tx=0).stop 2024-09-16T00:25:45.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.387+0000 7f51da8ae640 1 -- 172.21.15.33:0/3502477694 shutdown_connections 2024-09-16T00:25:45.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.387+0000 7f51da8ae640 1 --2- 172.21.15.33:0/3502477694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d4106f10 0x7f51d410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.387+0000 7f51da8ae640 1 --2- 172.21.15.33:0/3502477694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51d4102790 0x7f51d41067e0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.387+0000 7f51da8ae640 1 --2- 172.21.15.33:0/3502477694 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51d4101de0 0x7f51d41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.387+0000 7f51da8ae640 1 -- 172.21.15.33:0/3502477694 >> 172.21.15.33:0/3502477694 conn(0x7f51d40fd8c0 msgr2=0x7f51d40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:45.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.387+0000 7f51da8ae640 1 -- 172.21.15.33:0/3502477694 shutdown_connections 2024-09-16T00:25:45.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.387+0000 7f51da8ae640 1 -- 172.21.15.33:0/3502477694 wait complete. 2024-09-16T00:25:45.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.388+0000 7f51da8ae640 1 Processor -- start 2024-09-16T00:25:45.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.388+0000 7f51da8ae640 1 -- start start 2024-09-16T00:25:45.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.388+0000 7f51da8ae640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51d4101de0 0x7f51d419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.388+0000 7f51da8ae640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51d4102790 0x7f51d419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.388+0000 7f51da8ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d4106f10 0x7f51d419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.388+0000 7f51da8ae640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51d4111f80 con 0x7f51d4102790 2024-09-16T00:25:45.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.388+0000 7f51da8ae640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f51d4111e00 con 0x7f51d4101de0 2024-09-16T00:25:45.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.388+0000 7f51da8ae640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f51d4112100 con 0x7f51d4106f10 2024-09-16T00:25:45.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.388+0000 7f51d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51d4102790 0x7f51d419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.388+0000 7f51d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51d4102790 0x7f51d419abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38964/0 (socket says 172.21.15.33:38964) 2024-09-16T00:25:45.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.388+0000 7f51d37fe640 1 -- 172.21.15.33:0/2553295845 learned_addr learned my addr 172.21.15.33:0/2553295845 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:45.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.388+0000 7f51d8e4b640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d4106f10 0x7f51d419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51d3fff640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51d4101de0 0x7f51d419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51d8e4b640 1 -- 172.21.15.33:0/2553295845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51d4101de0 msgr2=0x7f51d419a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:45.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51d8e4b640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51d4101de0 0x7f51d419a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51d8e4b640 1 -- 172.21.15.33:0/2553295845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51d4102790 msgr2=0x7f51d419abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51d8e4b640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51d4102790 0x7f51d419abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51d8e4b640 1 -- 172.21.15.33:0/2553295845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d419f640 con 0x7f51d4106f10 2024-09-16T00:25:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51d37fe640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51d4102790 0x7f51d419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51d8e4b640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d4106f10 0x7f51d419ef60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009fd0 tx=0x7f51cc00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51d17fa640 1 -- 172.21.15.33:0/2553295845 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51cc00ce20 con 0x7f51d4106f10 2024-09-16T00:25:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51da8ae640 1 -- 172.21.15.33:0/2553295845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d419f930 con 0x7f51d4106f10 2024-09-16T00:25:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51d17fa640 1 -- 172.21.15.33:0/2553295845 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51cc005020 con 0x7f51d4106f10 2024-09-16T00:25:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51d17fa640 1 -- 172.21.15.33:0/2553295845 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51cc010720 con 0x7f51d4106f10 2024-09-16T00:25:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.389+0000 7f51da8ae640 1 -- 172.21.15.33:0/2553295845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d41a7210 con 0x7f51d4106f10 2024-09-16T00:25:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51d17fa640 1 -- 172.21.15.33:0/2553295845 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f51cc005540 con 0x7f51d4106f10 2024-09-16T00:25:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51dcc87740 1 Processor -- start 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51dcc87740 1 -- start start 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51dcc87740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830a8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51dcc87740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a85d0 0x55e2830a8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51dcc87740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830b3370 0x55e2830b5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51dcc87740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e28300e2f0 con 0x55e2830a9b40 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51dcc87740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e282fb5cf0 con 0x55e2830b3370 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51dcc87740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e28300a2b0 con 0x55e2830a85d0 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830a8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830a8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38978/0 (socket says 172.21.15.33:38978) 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51d37fe640 1 -- 172.21.15.33:0/429018723 learned_addr learned my addr 172.21.15.33:0/429018723 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51d3fff640 1 --2- 172.21.15.33:0/429018723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a85d0 0x55e2830a8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.392+0000 7f51d37fe640 1 --2- 172.21.15.33:0/429018723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830b3370 0x55e2830b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.393+0000 7f51d3fff640 1 -- 172.21.15.33:0/429018723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830b3370 msgr2=0x55e2830b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:45.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.393+0000 7f51d3fff640 1 --2- 172.21.15.33:0/429018723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830b3370 0x55e2830b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.393+0000 7f51d3fff640 1 -- 172.21.15.33:0/429018723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 msgr2=0x55e2830a8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:45.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.393+0000 7f51d3fff640 1 --2- 172.21.15.33:0/429018723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830a8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.393+0000 7f51d3fff640 1 -- 172.21.15.33:0/429018723 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e282f9a3a0 con 0x55e2830a85d0 2024-09-16T00:25:45.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.393+0000 7f51d17fa640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f51bc0a3240 0x7f51bc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.393+0000 7f51d37fe640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f51bc0a3240 0x7f51bc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.393+0000 7f51d3fff640 1 --2- 172.21.15.33:0/429018723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a85d0 0x55e2830a8a10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009c20 tx=0x7f51c4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4224fd080c2d86c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:45.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.393+0000 7f51d17fa640 1 -- 172.21.15.33:0/2553295845 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f51cc0d0410 con 0x7f51d4106f10 2024-09-16T00:25:45.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.393+0000 7f51d37fe640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f51bc0a3240 0x7f51bc0a5700 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f51c8004650 tx=0x7f51c80314b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:45.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.397+0000 7f51ba7fc640 1 -- 172.21.15.33:0/429018723 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c4004050 con 0x55e2830a85d0 2024-09-16T00:25:45.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.397+0000 7f51ba7fc640 1 -- 172.21.15.33:0/429018723 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c4005500 con 0x55e2830a85d0 2024-09-16T00:25:45.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.397+0000 7f51ba7fc640 1 -- 172.21.15.33:0/429018723 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c4004dd0 con 0x55e2830a85d0 2024-09-16T00:25:45.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.398+0000 7f51dcc87740 1 -- 172.21.15.33:0/429018723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a85d0 msgr2=0x55e2830a8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:45.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.398+0000 7f51dcc87740 1 --2- 172.21.15.33:0/429018723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a85d0 0x55e2830a8a10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009c20 tx=0x7f51c4011810 comp rx=0 tx=0).stop 2024-09-16T00:25:45.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.398+0000 7f51dcc87740 1 -- 172.21.15.33:0/429018723 shutdown_connections 2024-09-16T00:25:45.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.398+0000 7f51dcc87740 1 --2- 172.21.15.33:0/429018723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830b3370 0x55e2830b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.398+0000 7f51dcc87740 1 --2- 172.21.15.33:0/429018723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a85d0 0x55e2830a8a10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.398+0000 7f51dcc87740 1 --2- 172.21.15.33:0/429018723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830a8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.398+0000 7f51dcc87740 1 -- 172.21.15.33:0/429018723 >> 172.21.15.33:0/429018723 conn(0x55e283071310 msgr2=0x55e2830a4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:45.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.398+0000 7f51dcc87740 1 -- 172.21.15.33:0/429018723 shutdown_connections 2024-09-16T00:25:45.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.398+0000 7f51dcc87740 1 -- 172.21.15.33:0/429018723 wait complete. 2024-09-16T00:25:45.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.398+0000 7f51dcc87740 1 -- start start 2024-09-16T00:25:45.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.398+0000 7f51dcc87740 1 Processor -- start 2024-09-16T00:25:45.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51dcc87740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830a85d0 0x55e2830b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51dcc87740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830aff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51dcc87740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830b3370 0x55e2830b0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51dcc87740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e2830b6070 con 0x55e2830a9b40 2024-09-16T00:25:45.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51dcc87740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e282fc9060 con 0x55e2830a85d0 2024-09-16T00:25:45.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51dcc87740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e2830b61f0 con 0x55e2830b3370 2024-09-16T00:25:45.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51dcc87740 10 client.0 _ll_register_callbacks cb 0x55e282d1c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:45.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830aff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38994/0 (socket says 172.21.15.33:38994) 2024-09-16T00:25:45.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51d8e4b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830b3370 0x55e2830b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51d37fe640 1 -- 172.21.15.33:0/1807412871 learned_addr learned my addr 172.21.15.33:0/1807412871 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:45.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51d3fff640 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830a85d0 0x55e2830b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51d37fe640 1 -- 172.21.15.33:0/1807412871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830b3370 msgr2=0x55e2830b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:45.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51d37fe640 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830b3370 0x55e2830b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51d37fe640 1 -- 172.21.15.33:0/1807412871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830a85d0 msgr2=0x55e2830b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:45.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51d37fe640 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830a85d0 0x55e2830b2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51d37fe640 1 -- 172.21.15.33:0/1807412871 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e282d7f3c0 con 0x55e2830a9b40 2024-09-16T00:25:45.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.399+0000 7f51d8e4b640 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830b3370 0x55e2830b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:45.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.400+0000 7f51d37fe640 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830aff40 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f51c8004770 tx=0x7f51c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e1a6ce6399ad70 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:45.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.400+0000 7f51b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:45.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.400+0000 7f51b37fe640 1 -- 172.21.15.33:0/1807412871 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8050070 con 0x55e2830a9b40 2024-09-16T00:25:45.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.400+0000 7f51b37fe640 1 -- 172.21.15.33:0/1807412871 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f51c8003a00 con 0x55e2830a9b40 2024-09-16T00:25:45.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.400+0000 7f51dcc87740 1 -- 172.21.15.33:0/1807412871 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51c4005500 con 0x55e2830a9b40 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.400+0000 7f51b37fe640 1 -- 172.21.15.33:0/1807412871 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f51c805c050 con 0x55e2830a9b40 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.400+0000 7f51dcc87740 10 client.16962 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.400+0000 7f51dcc87740 1 -- 172.21.15.33:0/1807412871 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e2830c1a60 con 0x55e2830a9b40 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.400+0000 7f51b37fe640 1 -- 172.21.15.33:0/1807412871 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c4005500 con 0x55e2830a9b40 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.400+0000 7f51b37fe640 1 -- 172.21.15.33:0/1807412871 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f51c8049050 con 0x55e2830a9b40 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.400+0000 7f51dcc87740 10 client.16962 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.401+0000 7f51dcc87740 10 client.16962 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.401+0000 7f51dcc87740 20 client.16962 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.401+0000 7f51dcc87740 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55e2830c2ec0 0x55e2830c5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.401+0000 7f51dcc87740 4 client.16962 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.401+0000 7f51dcc87740 1 -- 172.21.15.33:0/1807412871 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e28301b860 con 0x55e2830c2ec0 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.401+0000 7f51d3fff640 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55e2830c2ec0 0x55e2830c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.401+0000 7f51d3fff640 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55e2830c2ec0 0x55e2830c5300 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=117d5f4035ff74e server_cookie=55617c6090d4b392 in_seq=0 out_seq=0 2024-09-16T00:25:45.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.401+0000 7f51b37fe640 10 client.16962 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:25:45.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51b37fe640 1 -- 172.21.15.33:0/1807412871 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e28301b860 con 0x55e2830c2ec0 2024-09-16T00:25:45.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51b37fe640 10 client.16962 handle_command_reply: tid=0 2024-09-16T00:25:45.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51dcc87740 1 client.16962 shutdown 2024-09-16T00:25:45.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51dcc87740 1 -- 172.21.15.33:0/1807412871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 msgr2=0x55e2830aff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:45.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51dcc87740 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830aff40 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f51c8004770 tx=0x7f51c80046c0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51dcc87740 1 -- 172.21.15.33:0/1807412871 shutdown_connections 2024-09-16T00:25:45.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51dcc87740 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830b3370 0x55e2830b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51dcc87740 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830aff40 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51dcc87740 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55e2830c2ec0 0x55e2830c5300 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51dcc87740 1 --2- 172.21.15.33:0/1807412871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830a85d0 0x55e2830b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51b37fe640 0 client.16962 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:25:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51dcc87740 1 -- 172.21.15.33:0/1807412871 >> 172.21.15.33:0/1807412871 conn(0x55e283071310 msgr2=0x55e2830aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51dcc87740 1 -- 172.21.15.33:0/1807412871 shutdown_connections 2024-09-16T00:25:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51dcc87740 1 -- 172.21.15.33:0/1807412871 wait complete. 2024-09-16T00:25:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.402+0000 7f51dcc87740 20 client.16962 trim_cache size 0 max 16384 2024-09-16T00:25:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51dcc87740 1 Processor -- start 2024-09-16T00:25:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51dcc87740 1 -- start start 2024-09-16T00:25:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51dcc87740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830a85d0 0x55e2830a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51dcc87740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a9b40 0x55e2830bf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51dcc87740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830b3370 0x55e2830cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51dcc87740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51d41b3c00 con 0x55e2830b3370 2024-09-16T00:25:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51dcc87740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f51d41b14d0 con 0x55e2830a85d0 2024-09-16T00:25:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51dcc87740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f51d41af320 con 0x55e2830a9b40 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51d8e4b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830b3370 0x55e2830cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51d37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a9b40 0x55e2830bf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51d8e4b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830b3370 0x55e2830cabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39000/0 (socket says 172.21.15.33:39000) 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51d8e4b640 1 -- 172.21.15.33:0/1133882550 learned_addr learned my addr 172.21.15.33:0/1133882550 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.418+0000 7f51d3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830a85d0 0x55e2830a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.419+0000 7f51d3fff640 1 -- 172.21.15.33:0/1133882550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a9b40 msgr2=0x55e2830bf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.419+0000 7f51d3fff640 1 --2- 172.21.15.33:0/1133882550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a9b40 0x55e2830bf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.419+0000 7f51d3fff640 1 -- 172.21.15.33:0/1133882550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830b3370 msgr2=0x55e2830cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.419+0000 7f51d3fff640 1 --2- 172.21.15.33:0/1133882550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830b3370 0x55e2830cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.419+0000 7f51d3fff640 1 -- 172.21.15.33:0/1133882550 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2830a5dd0 con 0x55e2830a85d0 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.419+0000 7f51d37fe640 1 --2- 172.21.15.33:0/1133882550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a9b40 0x55e2830bf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.419+0000 7f51d8e4b640 1 --2- 172.21.15.33:0/1133882550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830b3370 0x55e2830cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.419+0000 7f51d3fff640 1 --2- 172.21.15.33:0/1133882550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830a85d0 0x55e2830a4e10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f51c4000c00 tx=0x7f51c4009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73a0598e7a2f8d8a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.419+0000 7f51b9ffb640 1 -- 172.21.15.33:0/1133882550 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c4004240 con 0x55e2830a85d0 2024-09-16T00:25:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.419+0000 7f51b9ffb640 1 -- 172.21.15.33:0/1133882550 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c40043e0 con 0x55e2830a85d0 2024-09-16T00:25:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.419+0000 7f51b9ffb640 1 -- 172.21.15.33:0/1133882550 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c402e630 con 0x55e2830a85d0 2024-09-16T00:25:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.420+0000 7f51dcc87740 1 -- 172.21.15.33:0/1133882550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830a85d0 msgr2=0x55e2830a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.420+0000 7f51dcc87740 1 --2- 172.21.15.33:0/1133882550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830a85d0 0x55e2830a4e10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f51c4000c00 tx=0x7f51c4009680 comp rx=0 tx=0).stop 2024-09-16T00:25:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.420+0000 7f51dcc87740 1 -- 172.21.15.33:0/1133882550 shutdown_connections 2024-09-16T00:25:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.420+0000 7f51dcc87740 1 --2- 172.21.15.33:0/1133882550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830b3370 0x55e2830cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.420+0000 7f51dcc87740 1 --2- 172.21.15.33:0/1133882550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a9b40 0x55e2830bf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.420+0000 7f51dcc87740 1 --2- 172.21.15.33:0/1133882550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830a85d0 0x55e2830a4e10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.420+0000 7f51dcc87740 1 -- 172.21.15.33:0/1133882550 >> 172.21.15.33:0/1133882550 conn(0x55e283071310 msgr2=0x55e2830a4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.420+0000 7f51dcc87740 1 -- 172.21.15.33:0/1133882550 shutdown_connections 2024-09-16T00:25:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.420+0000 7f51dcc87740 1 -- 172.21.15.33:0/1133882550 wait complete. 2024-09-16T00:25:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.420+0000 7f51dcc87740 1 -- start start 2024-09-16T00:25:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51dcc87740 1 Processor -- start 2024-09-16T00:25:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51dcc87740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a85d0 0x55e2830bb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51dcc87740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830ac910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51dcc87740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830b3370 0x55e2830ace50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51dcc87740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51d41af990 con 0x55e2830a9b40 2024-09-16T00:25:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51dcc87740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f51d41b0e00 con 0x55e2830b3370 2024-09-16T00:25:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51dcc87740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f51d41b0060 con 0x55e2830a85d0 2024-09-16T00:25:45.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51dcc87740 10 client.0 _ll_register_callbacks cb 0x7f51c8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:45.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51d3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a85d0 0x55e2830bb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830ac910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39010/0 (socket says 172.21.15.33:39010) 2024-09-16T00:25:45.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51d37fe640 1 -- 172.21.15.33:0/27627877 learned_addr learned my addr 172.21.15.33:0/27627877 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:45.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51d8e4b640 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830b3370 0x55e2830ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51d37fe640 1 -- 172.21.15.33:0/27627877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a85d0 msgr2=0x55e2830bb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:45.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51d37fe640 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a85d0 0x55e2830bb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51d37fe640 1 -- 172.21.15.33:0/27627877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830b3370 msgr2=0x55e2830ace50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:45.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51d37fe640 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830b3370 0x55e2830ace50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51d37fe640 1 -- 172.21.15.33:0/27627877 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2830ab120 con 0x55e2830a9b40 2024-09-16T00:25:45.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.421+0000 7f51d3fff640 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a85d0 0x55e2830bb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:45.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51d37fe640 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830ac910 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f51c804cf00 tx=0x7f51c8031ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333abd445dd57b26 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:45.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:45.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51b37fe640 1 -- 172.21.15.33:0/27627877 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8065060 con 0x55e2830a9b40 2024-09-16T00:25:45.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51b37fe640 1 -- 172.21.15.33:0/27627877 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f51c804c680 con 0x55e2830a9b40 2024-09-16T00:25:45.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51dcc87740 1 -- 172.21.15.33:0/27627877 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51c40043e0 con 0x55e2830a9b40 2024-09-16T00:25:45.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51b37fe640 1 -- 172.21.15.33:0/27627877 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f51c806c050 con 0x55e2830a9b40 2024-09-16T00:25:45.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51dcc87740 10 client.16968 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:45.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51dcc87740 1 -- 172.21.15.33:0/27627877 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e28312d710 con 0x55e2830a9b40 2024-09-16T00:25:45.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51b37fe640 1 -- 172.21.15.33:0/27627877 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c40043e0 con 0x55e2830a9b40 2024-09-16T00:25:45.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51b37fe640 1 -- 172.21.15.33:0/27627877 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f51c8049040 con 0x55e2830a9b40 2024-09-16T00:25:45.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51dcc87740 10 client.16968 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:45.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51dcc87740 10 client.16968 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:25:45.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51dcc87740 20 client.16968 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51dcc87740 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55e28312dec0 0x55e2831302b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.422+0000 7f51dcc87740 4 client.16968 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:25:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.423+0000 7f51dcc87740 1 -- 172.21.15.33:0/27627877 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e283015d70 con 0x55e28312dec0 2024-09-16T00:25:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.423+0000 7f51d3fff640 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55e28312dec0 0x55e2831302b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.423+0000 7f51b37fe640 10 client.16968 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:25:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.423+0000 7f51d3fff640 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55e28312dec0 0x55e2831302b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ff7298eb7b648ebc server_cookie=ba21bf6f12b9ef33 in_seq=0 out_seq=0 2024-09-16T00:25:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.423+0000 7f51b37fe640 1 -- 172.21.15.33:0/27627877 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e283015d70 con 0x55e28312dec0 2024-09-16T00:25:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.423+0000 7f51b37fe640 10 client.16968 handle_command_reply: tid=0 2024-09-16T00:25:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.423+0000 7f51dcc87740 1 client.16968 shutdown 2024-09-16T00:25:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.423+0000 7f51dcc87740 1 -- 172.21.15.33:0/27627877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 msgr2=0x55e2830ac910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.423+0000 7f51dcc87740 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830ac910 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f51c804cf00 tx=0x7f51c8031ca0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.423+0000 7f51dcc87740 1 -- 172.21.15.33:0/27627877 shutdown_connections 2024-09-16T00:25:45.473 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:25:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.423+0000 7f51dcc87740 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55e28312dec0 0x55e2831302b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.423+0000 7f51dcc87740 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2830b3370 0x55e2830ace50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51dcc87740 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2830a9b40 0x55e2830ac910 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51b37fe640 0 client.16968 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:25:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51dcc87740 1 --2- 172.21.15.33:0/27627877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2830a85d0 0x55e2830bb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51dcc87740 1 -- 172.21.15.33:0/27627877 >> 172.21.15.33:0/27627877 conn(0x55e283071310 msgr2=0x55e2830aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51dcc87740 1 -- 172.21.15.33:0/27627877 shutdown_connections 2024-09-16T00:25:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51dcc87740 1 -- 172.21.15.33:0/27627877 wait complete. 2024-09-16T00:25:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51dcc87740 20 client.16968 trim_cache size 0 max 16384 2024-09-16T00:25:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51da8ae640 1 -- 172.21.15.33:0/2553295845 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f51bc0a3240 msgr2=0x7f51bc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51da8ae640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f51bc0a3240 0x7f51bc0a5700 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f51c8004650 tx=0x7f51c80314b0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51da8ae640 1 -- 172.21.15.33:0/2553295845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d4106f10 msgr2=0x7f51d419ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51da8ae640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d4106f10 0x7f51d419ef60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009fd0 tx=0x7f51cc00ef10 comp rx=0 tx=0).stop 2024-09-16T00:25:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51da8ae640 1 -- 172.21.15.33:0/2553295845 shutdown_connections 2024-09-16T00:25:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51da8ae640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f51bc0a3240 0x7f51bc0a5700 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51da8ae640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d4106f10 0x7f51d419ef60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51da8ae640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51d4102790 0x7f51d419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51da8ae640 1 --2- 172.21.15.33:0/2553295845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51d4101de0 0x7f51d419a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51da8ae640 1 -- 172.21.15.33:0/2553295845 >> 172.21.15.33:0/2553295845 conn(0x7f51d40fd8c0 msgr2=0x7f51d40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51da8ae640 1 -- 172.21.15.33:0/2553295845 shutdown_connections 2024-09-16T00:25:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:45.424+0000 7f51da8ae640 1 -- 172.21.15.33:0/2553295845 wait complete. 2024-09-16T00:25:45.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:45 smithi033 ceph-mon[31243]: pgmap v608: 129 pgs: 129 active+clean; 244 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 282 op/s 2024-09-16T00:25:45.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:45 smithi083 ceph-mon[35242]: pgmap v608: 129 pgs: 129 active+clean; 244 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 282 op/s 2024-09-16T00:25:45.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 678778 10.37 MB/sec execute 293 sec latency 29.633 ms 2024-09-16T00:25:45.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:45 smithi060 ceph-mon[35185]: pgmap v608: 129 pgs: 129 active+clean; 244 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 282 op/s 2024-09-16T00:25:46.246 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:25:46.519 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:46.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 682079 10.40 MB/sec execute 294 sec latency 20.054 ms 2024-09-16T00:25:47.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.026+0000 7f8d53f8d640 1 -- 172.21.15.33:0/1031760036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d4c10e480 msgr2=0x7f8d4c110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:47.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.026+0000 7f8d53f8d640 1 --2- 172.21.15.33:0/1031760036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d4c10e480 0x7f8d4c110870 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f8d44009ca0 tx=0x7f8d4402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.026+0000 7f8d53f8d640 1 -- 172.21.15.33:0/1031760036 shutdown_connections 2024-09-16T00:25:47.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.026+0000 7f8d53f8d640 1 --2- 172.21.15.33:0/1031760036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d4c10e480 0x7f8d4c110870 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.026+0000 7f8d53f8d640 1 --2- 172.21.15.33:0/1031760036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d4c101280 0x7f8d4c10df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.026+0000 7f8d53f8d640 1 --2- 172.21.15.33:0/1031760036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d4c1008d0 0x7f8d4c100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.026+0000 7f8d53f8d640 1 -- 172.21.15.33:0/1031760036 >> 172.21.15.33:0/1031760036 conn(0x7f8d4c0fc7a0 msgr2=0x7f8d4c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.026+0000 7f8d53f8d640 1 -- 172.21.15.33:0/1031760036 shutdown_connections 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.026+0000 7f8d53f8d640 1 -- 172.21.15.33:0/1031760036 wait complete. 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.026+0000 7f8d53f8d640 1 Processor -- start 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d53f8d640 1 -- start start 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d53f8d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d4c1008d0 0x7f8d4c19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d53f8d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d4c101280 0x7f8d4c19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d53f8d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d4c10e480 0x7f8d4c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d53f8d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4c114160 con 0x7f8d4c1008d0 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d53f8d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4c113fe0 con 0x7f8d4c10e480 2024-09-16T00:25:47.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d53f8d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4c1142e0 con 0x7f8d4c101280 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d51d29640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d4c1008d0 0x7f8d4c19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d51d29640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d4c1008d0 0x7f8d4c19a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39022/0 (socket says 172.21.15.33:39022) 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d51d29640 1 -- 172.21.15.33:0/4265902808 learned_addr learned my addr 172.21.15.33:0/4265902808 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d51528640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d4c101280 0x7f8d4c19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d5252a640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d4c10e480 0x7f8d4c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d51d29640 1 -- 172.21.15.33:0/4265902808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d4c101280 msgr2=0x7f8d4c19ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d51d29640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d4c101280 0x7f8d4c19ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d51d29640 1 -- 172.21.15.33:0/4265902808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d4c10e480 msgr2=0x7f8d4c19ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d51d29640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d4c10e480 0x7f8d4c19ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d51d29640 1 -- 172.21.15.33:0/4265902808 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d4c19f5f0 con 0x7f8d4c1008d0 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.027+0000 7f8d51d29640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d4c1008d0 0x7f8d4c19a640 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800d950 tx=0x7f8d4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.028+0000 7f8d42ffd640 1 -- 172.21.15.33:0/4265902808 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d48017030 con 0x7f8d4c1008d0 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.028+0000 7f8d53f8d640 1 -- 172.21.15.33:0/4265902808 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d4c19f8e0 con 0x7f8d4c1008d0 2024-09-16T00:25:47.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.028+0000 7f8d53f8d640 1 -- 172.21.15.33:0/4265902808 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d4c1a71c0 con 0x7f8d4c1008d0 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.028+0000 7f8d42ffd640 1 -- 172.21.15.33:0/4265902808 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d48004020 con 0x7f8d4c1008d0 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.028+0000 7f8d42ffd640 1 -- 172.21.15.33:0/4265902808 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d480047a0 con 0x7f8d4c1008d0 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.031+0000 7f8d42ffd640 1 -- 172.21.15.33:0/4265902808 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8d480041d0 con 0x7f8d4c1008d0 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.031+0000 7f8d56366740 1 Processor -- start 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.031+0000 7f8d56366740 1 -- start start 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.031+0000 7f8d56366740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e2b40 0x559e5a8e1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.031+0000 7f8d56366740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 0x559e5a8e1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.031+0000 7f8d56366740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8ec370 0x559e5a8ee760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d56366740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559e5a8472f0 con 0x559e5a8e15d0 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d56366740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559e5a7eecf0 con 0x559e5a8e2b40 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d56366740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559e5a8432b0 con 0x559e5a8ec370 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d51d29640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 0x559e5a8e1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d5252a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e2b40 0x559e5a8e1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d5252a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e2b40 0x559e5a8e1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:38014/0 (socket says 172.21.15.33:38014) 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d5252a640 1 -- 172.21.15.33:0/3704052712 learned_addr learned my addr 172.21.15.33:0/3704052712 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d51d29640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 0x559e5a8e1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39026/0 (socket says 172.21.15.33:39026) 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d51528640 1 --2- 172.21.15.33:0/3704052712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8ec370 0x559e5a8ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d5252a640 1 -- 172.21.15.33:0/3704052712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8ec370 msgr2=0x559e5a8ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d5252a640 1 --2- 172.21.15.33:0/3704052712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8ec370 0x559e5a8ee760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d5252a640 1 -- 172.21.15.33:0/3704052712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 msgr2=0x559e5a8e1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d5252a640 1 --2- 172.21.15.33:0/3704052712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 0x559e5a8e1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d5252a640 1 -- 172.21.15.33:0/3704052712 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e5a7d33a0 con 0x559e5a8e2b40 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d51d29640 1 --2- 172.21.15.33:0/3704052712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 0x559e5a8e1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d42ffd640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d380a32c0 0x7f8d380a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d5252a640 1 --2- 172.21.15.33:0/3704052712 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e2b40 0x559e5a8e1090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8d44005cc0 tx=0x7f8d44009930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f83c4a9b1b9eb291 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.032+0000 7f8d51d29640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d380a32c0 0x7f8d380a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.033+0000 7f8d42ffd640 1 -- 172.21.15.33:0/4265902808 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8d480cf610 con 0x7f8d4c1008d0 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.033+0000 7f8d37fff640 1 -- 172.21.15.33:0/3704052712 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d44003970 con 0x559e5a8e2b40 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.033+0000 7f8d37fff640 1 -- 172.21.15.33:0/3704052712 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d44036050 con 0x559e5a8e2b40 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.033+0000 7f8d37fff640 1 -- 172.21.15.33:0/3704052712 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d440043a0 con 0x559e5a8e2b40 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.033+0000 7f8d51d29640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d380a32c0 0x7f8d380a5780 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8d480cd000 tx=0x7f8d480d5550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.033+0000 7f8d56366740 1 -- 172.21.15.33:0/3704052712 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e2b40 msgr2=0x559e5a8e1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.033+0000 7f8d56366740 1 --2- 172.21.15.33:0/3704052712 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e2b40 0x559e5a8e1090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8d44005cc0 tx=0x7f8d44009930 comp rx=0 tx=0).stop 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.033+0000 7f8d56366740 1 -- 172.21.15.33:0/3704052712 shutdown_connections 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.033+0000 7f8d56366740 1 --2- 172.21.15.33:0/3704052712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8ec370 0x559e5a8ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.033+0000 7f8d56366740 1 --2- 172.21.15.33:0/3704052712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 0x559e5a8e1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.033+0000 7f8d56366740 1 --2- 172.21.15.33:0/3704052712 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e2b40 0x559e5a8e1090 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.033+0000 7f8d56366740 1 -- 172.21.15.33:0/3704052712 >> 172.21.15.33:0/3704052712 conn(0x559e5a8aa310 msgr2=0x559e5a8ddc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.034+0000 7f8d56366740 1 -- 172.21.15.33:0/3704052712 shutdown_connections 2024-09-16T00:25:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.034+0000 7f8d56366740 1 -- 172.21.15.33:0/3704052712 wait complete. 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.034+0000 7f8d56366740 1 -- start start 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.034+0000 7f8d56366740 1 Processor -- start 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d56366740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 0x559e5a8ebc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d56366740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e2b40 0x559e5a8e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d56366740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8ec370 0x559e5a8e9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d56366740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559e5a8eee90 con 0x559e5a8e15d0 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d56366740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559e5a802060 con 0x559e5a8ec370 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d56366740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559e5a8ef010 con 0x559e5a8e2b40 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d5252a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 0x559e5a8ebc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d5252a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 0x559e5a8ebc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39028/0 (socket says 172.21.15.33:39028) 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d5252a640 1 -- 172.21.15.33:0/1312822587 learned_addr learned my addr 172.21.15.33:0/1312822587 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d5252a640 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8ec370 0x559e5a8e9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d51528640 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e2b40 0x559e5a8e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d56366740 10 client.0 _ll_register_callbacks cb 0x559e5a5558c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d51528640 1 -- 172.21.15.33:0/1312822587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8ec370 msgr2=0x559e5a8e9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d51528640 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8ec370 0x559e5a8e9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d51528640 1 -- 172.21.15.33:0/1312822587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 msgr2=0x559e5a8ebc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d51528640 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 0x559e5a8ebc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d51528640 1 -- 172.21.15.33:0/1312822587 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e5a5b83c0 con 0x559e5a8e2b40 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d5252a640 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 0x559e5a8ebc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d51528640 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e2b40 0x559e5a8e8f40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c10ed80 tx=0x7f8d3c010bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d8855c1cd4b7006 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.035+0000 7f8d357fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d357fa640 1 -- 172.21.15.33:0/1312822587 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c00fc20 con 0x559e5a8e2b40 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d357fa640 1 -- 172.21.15.33:0/1312822587 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8d3c010e00 con 0x559e5a8e2b40 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d56366740 1 -- 172.21.15.33:0/1312822587 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d44036050 con 0x559e5a8e2b40 2024-09-16T00:25:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d357fa640 1 -- 172.21.15.33:0/1312822587 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8d3c024050 con 0x559e5a8e2b40 2024-09-16T00:25:47.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d56366740 10 client.26416 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:47.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d56366740 1 -- 172.21.15.33:0/1312822587 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559e5a8fa950 con 0x559e5a8e2b40 2024-09-16T00:25:47.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d357fa640 1 -- 172.21.15.33:0/1312822587 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d44036050 con 0x559e5a8e2b40 2024-09-16T00:25:47.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d357fa640 1 -- 172.21.15.33:0/1312822587 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f8d3c00e420 con 0x559e5a8e2b40 2024-09-16T00:25:47.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d56366740 10 client.26416 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:47.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d56366740 10 client.26416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:47.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d56366740 20 client.26416 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:47.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d56366740 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559e5a8fbed0 0x559e5a8fe2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d56366740 4 client.26416 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.036+0000 7f8d56366740 1 -- 172.21.15.33:0/1312822587 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e5a854860 con 0x559e5a8fbed0 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.037+0000 7f8d5252a640 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559e5a8fbed0 0x559e5a8fe2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.038+0000 7f8d5252a640 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559e5a8fbed0 0x559e5a8fe2e0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1998221e3facdc9 server_cookie=db94a2bf084239ef in_seq=0 out_seq=0 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.038+0000 7f8d357fa640 10 client.26416 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.038+0000 7f8d357fa640 1 -- 172.21.15.33:0/1312822587 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559e5a854860 con 0x559e5a8fbed0 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.038+0000 7f8d357fa640 10 client.26416 handle_command_reply: tid=0 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.038+0000 7f8d56366740 1 client.26416 shutdown 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.039+0000 7f8d56366740 1 -- 172.21.15.33:0/1312822587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e2b40 msgr2=0x559e5a8e8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.039+0000 7f8d56366740 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e2b40 0x559e5a8e8f40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c10ed80 tx=0x7f8d3c010bb0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.039+0000 7f8d56366740 1 -- 172.21.15.33:0/1312822587 shutdown_connections 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.039+0000 7f8d56366740 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559e5a8fbed0 0x559e5a8fe2e0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.039+0000 7f8d56366740 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8ec370 0x559e5a8e9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.039+0000 7f8d56366740 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e2b40 0x559e5a8e8f40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.039+0000 7f8d357fa640 0 client.26416 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:47.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.039+0000 7f8d56366740 1 --2- 172.21.15.33:0/1312822587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8e15d0 0x559e5a8ebc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.039+0000 7f8d56366740 1 -- 172.21.15.33:0/1312822587 >> 172.21.15.33:0/1312822587 conn(0x559e5a8aa310 msgr2=0x559e5a8e3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:47.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.039+0000 7f8d56366740 1 -- 172.21.15.33:0/1312822587 shutdown_connections 2024-09-16T00:25:47.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.039+0000 7f8d56366740 1 -- 172.21.15.33:0/1312822587 wait complete. 2024-09-16T00:25:47.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.039+0000 7f8d56366740 20 client.26416 trim_cache size 0 max 16384 2024-09-16T00:25:47.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d56366740 1 Processor -- start 2024-09-16T00:25:47.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d56366740 1 -- start start 2024-09-16T00:25:47.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d56366740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8aa310 0x559e5a8f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d56366740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e15d0 0x559e5a8e4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d56366740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e2b40 0x559e5a903bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d56366740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d30012860 con 0x559e5a8aa310 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d56366740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8d30010130 con 0x559e5a8e2b40 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d56366740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3000df80 con 0x559e5a8e15d0 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d5252a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8aa310 0x559e5a8f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d51528640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e15d0 0x559e5a8e4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d5252a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8aa310 0x559e5a8f81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39032/0 (socket says 172.21.15.33:39032) 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d5252a640 1 -- 172.21.15.33:0/1203911133 learned_addr learned my addr 172.21.15.33:0/1203911133 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d5252a640 1 --2- 172.21.15.33:0/1203911133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e2b40 0x559e5a903bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d51528640 1 -- 172.21.15.33:0/1203911133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e2b40 msgr2=0x559e5a903bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d51528640 1 --2- 172.21.15.33:0/1203911133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e2b40 0x559e5a903bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d51528640 1 -- 172.21.15.33:0/1203911133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8aa310 msgr2=0x559e5a8f81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:47.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d51528640 1 --2- 172.21.15.33:0/1203911133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8aa310 0x559e5a8f81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.057+0000 7f8d51528640 1 -- 172.21.15.33:0/1203911133 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e5a8fe960 con 0x559e5a8e15d0 2024-09-16T00:25:47.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.058+0000 7f8d51528640 1 --2- 172.21.15.33:0/1203911133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e15d0 0x559e5a8e4310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00ea30 tx=0x7f8d3c00f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a9108912550b487 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:47.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.058+0000 7f8d377fe640 1 -- 172.21.15.33:0/1203911133 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c02d070 con 0x559e5a8e15d0 2024-09-16T00:25:47.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.058+0000 7f8d377fe640 1 -- 172.21.15.33:0/1203911133 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d3c00a340 con 0x559e5a8e15d0 2024-09-16T00:25:47.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.058+0000 7f8d377fe640 1 -- 172.21.15.33:0/1203911133 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c00fc20 con 0x559e5a8e15d0 2024-09-16T00:25:47.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.059+0000 7f8d56366740 1 -- 172.21.15.33:0/1203911133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e15d0 msgr2=0x559e5a8e4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:47.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.059+0000 7f8d56366740 1 --2- 172.21.15.33:0/1203911133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e15d0 0x559e5a8e4310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00ea30 tx=0x7f8d3c00f7f0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.059+0000 7f8d56366740 1 -- 172.21.15.33:0/1203911133 shutdown_connections 2024-09-16T00:25:47.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.059+0000 7f8d56366740 1 --2- 172.21.15.33:0/1203911133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e2b40 0x559e5a903bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.059+0000 7f8d56366740 1 --2- 172.21.15.33:0/1203911133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e15d0 0x559e5a8e4310 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.059+0000 7f8d56366740 1 --2- 172.21.15.33:0/1203911133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8aa310 0x559e5a8f81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.059+0000 7f8d56366740 1 -- 172.21.15.33:0/1203911133 >> 172.21.15.33:0/1203911133 conn(0x559e5a8e95e0 msgr2=0x559e5a8e8030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:47.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.059+0000 7f8d56366740 1 -- 172.21.15.33:0/1203911133 shutdown_connections 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.059+0000 7f8d56366740 1 -- 172.21.15.33:0/1203911133 wait complete. 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.059+0000 7f8d56366740 1 -- start start 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.059+0000 7f8d56366740 1 Processor -- start 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d56366740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8aa310 0x559e5a8f0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d56366740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e15d0 0x559e5a8f0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d56366740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e2b40 0x559e5a8f0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d56366740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3000e5f0 con 0x559e5a8aa310 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d56366740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3000fa60 con 0x559e5a8e15d0 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d56366740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3000ecc0 con 0x559e5a8e2b40 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d56366740 10 client.0 _ll_register_callbacks cb 0x7f8d3c00a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d5252a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8aa310 0x559e5a8f0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d5252a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8aa310 0x559e5a8f0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39038/0 (socket says 172.21.15.33:39038) 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d51528640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e15d0 0x559e5a8f0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d5252a640 1 -- 172.21.15.33:0/71815846 learned_addr learned my addr 172.21.15.33:0/71815846 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:47.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d5252a640 1 --2- 172.21.15.33:0/71815846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e2b40 0x559e5a8f0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d51528640 1 -- 172.21.15.33:0/71815846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e2b40 msgr2=0x559e5a8f0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:47.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d51528640 1 --2- 172.21.15.33:0/71815846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e2b40 0x559e5a8f0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d51528640 1 -- 172.21.15.33:0/71815846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8aa310 msgr2=0x559e5a8f0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:47.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d51528640 1 --2- 172.21.15.33:0/71815846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8aa310 0x559e5a8f0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.060+0000 7f8d51528640 1 -- 172.21.15.33:0/71815846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e5a965e60 con 0x559e5a8e15d0 2024-09-16T00:25:47.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d51528640 1 --2- 172.21.15.33:0/71815846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e15d0 0x559e5a8f0880 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x559e5a8ec130 tx=0x7f8d3c00ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f21c86ac9551eb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:47.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d357fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:25:47.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d357fa640 1 -- 172.21.15.33:0/71815846 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c02d040 con 0x559e5a8e15d0 2024-09-16T00:25:47.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d357fa640 1 -- 172.21.15.33:0/71815846 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8d3c00fc20 con 0x559e5a8e15d0 2024-09-16T00:25:47.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d56366740 1 -- 172.21.15.33:0/71815846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d3c00a340 con 0x559e5a8e15d0 2024-09-16T00:25:47.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d357fa640 1 -- 172.21.15.33:0/71815846 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8d3c03e050 con 0x559e5a8e15d0 2024-09-16T00:25:47.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d56366740 10 client.26483 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:47.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d56366740 1 -- 172.21.15.33:0/71815846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559e5a967160 con 0x559e5a8e15d0 2024-09-16T00:25:47.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d357fa640 1 -- 172.21.15.33:0/71815846 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d3c00a340 con 0x559e5a8e15d0 2024-09-16T00:25:47.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d357fa640 1 -- 172.21.15.33:0/71815846 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f8d3c004660 con 0x559e5a8e15d0 2024-09-16T00:25:47.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d56366740 10 client.26483 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:47.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d56366740 10 client.26483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:47.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.061+0000 7f8d56366740 20 client.26483 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:47.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.062+0000 7f8d56366740 1 --2- 172.21.15.33:0/71815846 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559e5a8ec370 0x559e5a969820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:47.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.062+0000 7f8d56366740 4 client.26483 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:25:47.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.062+0000 7f8d56366740 1 -- 172.21.15.33:0/71815846 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559e5a84ed70 con 0x559e5a8ec370 2024-09-16T00:25:47.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.062+0000 7f8d5252a640 1 --2- 172.21.15.33:0/71815846 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559e5a8ec370 0x559e5a969820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:47.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.062+0000 7f8d5252a640 1 --2- 172.21.15.33:0/71815846 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559e5a8ec370 0x559e5a969820 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1637bf78f3189c6b server_cookie=7e603939451ac0ff in_seq=0 out_seq=0 2024-09-16T00:25:47.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.062+0000 7f8d357fa640 10 client.26483 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:47.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.062+0000 7f8d357fa640 1 -- 172.21.15.33:0/71815846 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559e5a84ed70 con 0x559e5a8ec370 2024-09-16T00:25:47.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.062+0000 7f8d357fa640 10 client.26483 handle_command_reply: tid=0 2024-09-16T00:25:47.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.062+0000 7f8d56366740 1 client.26483 shutdown 2024-09-16T00:25:47.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.062+0000 7f8d56366740 1 -- 172.21.15.33:0/71815846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e15d0 msgr2=0x559e5a8f0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:47.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.062+0000 7f8d56366740 1 --2- 172.21.15.33:0/71815846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e15d0 0x559e5a8f0880 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x559e5a8ec130 tx=0x7f8d3c00ec90 comp rx=0 tx=0).stop 2024-09-16T00:25:47.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d56366740 1 -- 172.21.15.33:0/71815846 shutdown_connections 2024-09-16T00:25:47.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d56366740 1 --2- 172.21.15.33:0/71815846 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559e5a8ec370 0x559e5a969820 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d56366740 1 --2- 172.21.15.33:0/71815846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559e5a8e2b40 0x559e5a8f0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d357fa640 0 client.26483 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:47.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d56366740 1 --2- 172.21.15.33:0/71815846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559e5a8e15d0 0x559e5a8f0880 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d56366740 1 --2- 172.21.15.33:0/71815846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e5a8aa310 0x559e5a8f0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d56366740 1 -- 172.21.15.33:0/71815846 >> 172.21.15.33:0/71815846 conn(0x559e5a8e95e0 msgr2=0x559e5a8ea7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:47.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d56366740 1 -- 172.21.15.33:0/71815846 shutdown_connections 2024-09-16T00:25:47.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d56366740 1 -- 172.21.15.33:0/71815846 wait complete. 2024-09-16T00:25:47.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d56366740 20 client.26483 trim_cache size 0 max 16384 2024-09-16T00:25:47.068 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:25:47.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d53f8d640 1 -- 172.21.15.33:0/4265902808 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d380a32c0 msgr2=0x7f8d380a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:47.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d53f8d640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d380a32c0 0x7f8d380a5780 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8d480cd000 tx=0x7f8d480d5550 comp rx=0 tx=0).stop 2024-09-16T00:25:47.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d53f8d640 1 -- 172.21.15.33:0/4265902808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d4c1008d0 msgr2=0x7f8d4c19a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:47.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d53f8d640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d4c1008d0 0x7f8d4c19a640 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800d950 tx=0x7f8d4800de10 comp rx=0 tx=0).stop 2024-09-16T00:25:47.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d53f8d640 1 -- 172.21.15.33:0/4265902808 shutdown_connections 2024-09-16T00:25:47.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d53f8d640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8d380a32c0 0x7f8d380a5780 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d53f8d640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d4c10e480 0x7f8d4c19ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d53f8d640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d4c101280 0x7f8d4c19ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d53f8d640 1 --2- 172.21.15.33:0/4265902808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d4c1008d0 0x7f8d4c19a640 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:47.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.063+0000 7f8d53f8d640 1 -- 172.21.15.33:0/4265902808 >> 172.21.15.33:0/4265902808 conn(0x7f8d4c0fc7a0 msgr2=0x7f8d4c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:47.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.064+0000 7f8d53f8d640 1 -- 172.21.15.33:0/4265902808 shutdown_connections 2024-09-16T00:25:47.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:47.064+0000 7f8d53f8d640 1 -- 172.21.15.33:0/4265902808 wait complete. 2024-09-16T00:25:47.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:47 smithi083 ceph-mon[35242]: pgmap v609: 129 pgs: 129 active+clean; 246 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 212 op/s 2024-09-16T00:25:47.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 684464 10.39 MB/sec execute 295 sec latency 42.890 ms 2024-09-16T00:25:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:47 smithi060 ceph-mon[35185]: pgmap v609: 129 pgs: 129 active+clean; 246 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 212 op/s 2024-09-16T00:25:47.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:47 smithi033 ceph-mon[31243]: pgmap v609: 129 pgs: 129 active+clean; 246 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 212 op/s 2024-09-16T00:25:48.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 686868 10.41 MB/sec execute 296 sec latency 19.584 ms 2024-09-16T00:25:48.809 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:25:48.809 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:25:49.079 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:49.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:49 smithi033 ceph-mon[31243]: pgmap v610: 129 pgs: 129 active+clean; 246 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 186 op/s 2024-09-16T00:25:49.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:49 smithi083 ceph-mon[35242]: pgmap v610: 129 pgs: 129 active+clean; 246 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 186 op/s 2024-09-16T00:25:49.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 689615 10.42 MB/sec execute 297 sec latency 39.814 ms 2024-09-16T00:25:49.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:49 smithi060 ceph-mon[35185]: pgmap v610: 129 pgs: 129 active+clean; 246 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 186 op/s 2024-09-16T00:25:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.949+0000 7fae8042b640 1 -- 172.21.15.33:0/3246397743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae78102ef0 msgr2=0x7fae7810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.949+0000 7fae8042b640 1 --2- 172.21.15.33:0/3246397743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae78102ef0 0x7fae7810da40 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fae70009ca0 tx=0x7fae7002a950 comp rx=0 tx=0).stop 2024-09-16T00:25:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.950+0000 7fae8042b640 1 -- 172.21.15.33:0/3246397743 shutdown_connections 2024-09-16T00:25:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.950+0000 7fae8042b640 1 --2- 172.21.15.33:0/3246397743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae78102ef0 0x7fae7810da40 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.950+0000 7fae8042b640 1 --2- 172.21.15.33:0/3246397743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae78102570 0x7fae781029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.950+0000 7fae8042b640 1 --2- 172.21.15.33:0/3246397743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae78100e20 0x7fae78101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.950+0000 7fae8042b640 1 -- 172.21.15.33:0/3246397743 >> 172.21.15.33:0/3246397743 conn(0x7fae780fc7d0 msgr2=0x7fae780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.950+0000 7fae8042b640 1 -- 172.21.15.33:0/3246397743 shutdown_connections 2024-09-16T00:25:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.950+0000 7fae8042b640 1 -- 172.21.15.33:0/3246397743 wait complete. 2024-09-16T00:25:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.950+0000 7fae8042b640 1 Processor -- start 2024-09-16T00:25:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae8042b640 1 -- start start 2024-09-16T00:25:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae8042b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae78100e20 0x7fae781973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae8042b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae78102570 0x7fae78197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae8042b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae78102ef0 0x7fae7819bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae8042b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae78110f80 con 0x7fae78100e20 2024-09-16T00:25:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae8042b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fae78110e00 con 0x7fae78102ef0 2024-09-16T00:25:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae8042b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae78111100 con 0x7fae78102570 2024-09-16T00:25:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae7e1c7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae78100e20 0x7fae781973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae7e1c7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae78100e20 0x7fae781973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39058/0 (socket says 172.21.15.33:39058) 2024-09-16T00:25:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae7e1c7640 1 -- 172.21.15.33:0/1085509624 learned_addr learned my addr 172.21.15.33:0/1085509624 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae78102570 0x7fae78197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae7e9c8640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae78102ef0 0x7fae7819bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae7e1c7640 1 -- 172.21.15.33:0/1085509624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae78102570 msgr2=0x7fae78197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae7e1c7640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae78102570 0x7fae78197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae7e1c7640 1 -- 172.21.15.33:0/1085509624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae78102ef0 msgr2=0x7fae7819bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae7e1c7640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae78102ef0 0x7fae7819bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae7e1c7640 1 -- 172.21.15.33:0/1085509624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae7819c390 con 0x7fae78100e20 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae7e9c8640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae78102ef0 0x7fae7819bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae7e1c7640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae78100e20 0x7fae781973e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fae7400e9f0 tx=0x7fae7400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.951+0000 7fae6f7fe640 1 -- 172.21.15.33:0/1085509624 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7400cd80 con 0x7fae78100e20 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.952+0000 7fae6f7fe640 1 -- 172.21.15.33:0/1085509624 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae74010910 con 0x7fae78100e20 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.952+0000 7fae8042b640 1 -- 172.21.15.33:0/1085509624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae7819c680 con 0x7fae78100e20 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.952+0000 7fae6f7fe640 1 -- 172.21.15.33:0/1085509624 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74018990 con 0x7fae78100e20 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.952+0000 7fae8042b640 1 -- 172.21.15.33:0/1085509624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae78069f70 con 0x7fae78100e20 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.954+0000 7fae6f7fe640 1 -- 172.21.15.33:0/1085509624 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fae74018b30 con 0x7fae78100e20 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.954+0000 7fae82804740 1 Processor -- start 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.954+0000 7fae82804740 1 -- start start 2024-09-16T00:25:49.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.954+0000 7fae82804740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32ded9a60 0x55d32ded9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.954+0000 7fae82804740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32deda380 0x55d32dee8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae82804740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32deeb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae82804740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d32de3f280 con 0x55d32dee9150 2024-09-16T00:25:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae82804740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d32dde6b10 con 0x55d32ded9a60 2024-09-16T00:25:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae82804740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d32de3b240 con 0x55d32deda380 2024-09-16T00:25:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7e1c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32deda380 0x55d32dee8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7e9c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32ded9a60 0x55d32ded9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7d9c6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32deeb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7e9c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32ded9a60 0x55d32ded9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:38072/0 (socket says 172.21.15.33:38072) 2024-09-16T00:25:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7e9c8640 1 -- 172.21.15.33:0/3413121907 learned_addr learned my addr 172.21.15.33:0/3413121907 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7d9c6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32deeb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39068/0 (socket says 172.21.15.33:39068) 2024-09-16T00:25:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae6f7fe640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae640a3240 0x7fae640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7d9c6640 1 -- 172.21.15.33:0/3413121907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32deda380 msgr2=0x55d32dee8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/3413121907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32deda380 0x55d32dee8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7e9c8640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae640a3240 0x7fae640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7d9c6640 1 -- 172.21.15.33:0/3413121907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32ded9a60 msgr2=0x55d32ded9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/3413121907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32ded9a60 0x55d32ded9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7d9c6640 1 -- 172.21.15.33:0/3413121907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d32ddf31f0 con 0x55d32dee9150 2024-09-16T00:25:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae6f7fe640 1 -- 172.21.15.33:0/1085509624 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fae740d0240 con 0x7fae78100e20 2024-09-16T00:25:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.955+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/3413121907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32deeb540 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fae68009bf0 tx=0x7fae68011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c876f198d246fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.956+0000 7fae6cdea640 1 -- 172.21.15.33:0/3413121907 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae68004050 con 0x55d32dee9150 2024-09-16T00:25:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.956+0000 7fae6cdea640 1 -- 172.21.15.33:0/3413121907 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae68005500 con 0x55d32dee9150 2024-09-16T00:25:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.956+0000 7fae6cdea640 1 -- 172.21.15.33:0/3413121907 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae68004dd0 con 0x55d32dee9150 2024-09-16T00:25:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.956+0000 7fae82804740 1 -- 172.21.15.33:0/3413121907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 msgr2=0x55d32deeb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.956+0000 7fae82804740 1 --2- 172.21.15.33:0/3413121907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32deeb540 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fae68009bf0 tx=0x7fae68011810 comp rx=0 tx=0).stop 2024-09-16T00:25:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.956+0000 7fae7e9c8640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae640a3240 0x7fae640a5700 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fae70009430 tx=0x7fae7002a950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 -- 172.21.15.33:0/3413121907 shutdown_connections 2024-09-16T00:25:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 --2- 172.21.15.33:0/3413121907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32deeb540 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 --2- 172.21.15.33:0/3413121907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32deda380 0x55d32dee8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 --2- 172.21.15.33:0/3413121907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32ded9a60 0x55d32ded9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 -- 172.21.15.33:0/3413121907 >> 172.21.15.33:0/3413121907 conn(0x55d32ded60f0 msgr2=0x55d32ded64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 -- 172.21.15.33:0/3413121907 shutdown_connections 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 -- 172.21.15.33:0/3413121907 wait complete. 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 -- start start 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 Processor -- start 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32ded9a60 0x55d32dee60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32deda380 0x55d32dee6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32dee9150 0x55d32dee6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d32deebd80 con 0x55d32deda380 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d32ddf9f10 con 0x55d32ded9a60 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.957+0000 7fae82804740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d32deebf00 con 0x55d32dee9150 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae82804740 10 client.0 _ll_register_callbacks cb 0x55d32ddc2a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:49.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae7d9c6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32deda380 0x55d32dee6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae7d9c6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32deda380 0x55d32dee6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39072/0 (socket says 172.21.15.33:39072) 2024-09-16T00:25:49.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae7d9c6640 1 -- 172.21.15.33:0/1971395772 learned_addr learned my addr 172.21.15.33:0/1971395772 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:49.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae7e9c8640 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32dee9150 0x55d32dee6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae7e1c7640 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32ded9a60 0x55d32dee60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae7d9c6640 1 -- 172.21.15.33:0/1971395772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32dee9150 msgr2=0x55d32dee6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:49.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32dee9150 0x55d32dee6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae7d9c6640 1 -- 172.21.15.33:0/1971395772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32ded9a60 msgr2=0x55d32dee60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:49.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32ded9a60 0x55d32dee60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae7d9c6640 1 -- 172.21.15.33:0/1971395772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d32dbb06a0 con 0x55d32deda380 2024-09-16T00:25:49.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae7e1c7640 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32ded9a60 0x55d32dee60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:25:49.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32deda380 0x55d32dee6610 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fae68004650 tx=0x7fae68009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c125ea4e3bf1f53 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:49.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae61ffb640 1 -- 172.21.15.33:0/1971395772 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae68029070 con 0x55d32deda380 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae82804740 1 -- 172.21.15.33:0/1971395772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae68005500 con 0x55d32deda380 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.958+0000 7fae61ffb640 1 -- 172.21.15.33:0/1971395772 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fae68004f20 con 0x55d32deda380 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.959+0000 7fae61ffb640 1 -- 172.21.15.33:0/1971395772 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fae68035050 con 0x55d32deda380 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.959+0000 7fae82804740 10 client.16998 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.959+0000 7fae82804740 1 -- 172.21.15.33:0/1971395772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d32def7a70 con 0x55d32deda380 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.959+0000 7fae61ffb640 1 -- 172.21.15.33:0/1971395772 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae68005500 con 0x55d32deda380 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.959+0000 7fae61ffb640 1 -- 172.21.15.33:0/1971395772 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fae68003c10 con 0x55d32deda380 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.959+0000 7fae82804740 10 client.16998 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.959+0000 7fae82804740 10 client.16998 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.959+0000 7fae82804740 20 client.16998 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.959+0000 7fae82804740 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d32def8e20 0x55d32defb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.959+0000 7fae82804740 4 client.16998 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:25:49.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.959+0000 7fae82804740 1 -- 172.21.15.33:0/1971395772 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d32de4c7f0 con 0x55d32def8e20 2024-09-16T00:25:49.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.959+0000 7fae7e1c7640 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d32def8e20 0x55d32defb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.960+0000 7fae7e1c7640 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d32def8e20 0x55d32defb210 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a6b51238d8c021b server_cookie=cf6656fc521bbfb8 in_seq=0 out_seq=0 2024-09-16T00:25:49.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.960+0000 7fae61ffb640 10 client.16998 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:49.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.960+0000 7fae61ffb640 1 -- 172.21.15.33:0/1971395772 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d32de4c7f0 con 0x55d32def8e20 2024-09-16T00:25:49.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.960+0000 7fae61ffb640 10 client.16998 handle_command_reply: tid=0 2024-09-16T00:25:49.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.960+0000 7fae82804740 1 client.16998 shutdown 2024-09-16T00:25:49.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.960+0000 7fae82804740 1 -- 172.21.15.33:0/1971395772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32deda380 msgr2=0x55d32dee6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:49.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.960+0000 7fae82804740 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32deda380 0x55d32dee6610 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fae68004650 tx=0x7fae68009fd0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.960+0000 7fae82804740 1 -- 172.21.15.33:0/1971395772 shutdown_connections 2024-09-16T00:25:49.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.960+0000 7fae82804740 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d32def8e20 0x55d32defb210 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.961+0000 7fae61ffb640 0 client.16998 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:49.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.961+0000 7fae82804740 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32dee9150 0x55d32dee6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.961+0000 7fae82804740 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32deda380 0x55d32dee6610 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.961+0000 7fae82804740 1 --2- 172.21.15.33:0/1971395772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32ded9a60 0x55d32dee60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.961+0000 7fae82804740 1 -- 172.21.15.33:0/1971395772 >> 172.21.15.33:0/1971395772 conn(0x55d32ded60f0 msgr2=0x55d32dedbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:49.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.961+0000 7fae82804740 1 -- 172.21.15.33:0/1971395772 shutdown_connections 2024-09-16T00:25:49.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.961+0000 7fae82804740 1 -- 172.21.15.33:0/1971395772 wait complete. 2024-09-16T00:25:49.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.961+0000 7fae82804740 20 client.16998 trim_cache size 0 max 16384 2024-09-16T00:25:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae82804740 1 Processor -- start 2024-09-16T00:25:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae82804740 1 -- start start 2024-09-16T00:25:49.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae82804740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 0x55d32dee7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae82804740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32deda380 0x55d32df009d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae82804740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32df00f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae82804740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae781af740 con 0x55d32dee9150 2024-09-16T00:25:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae82804740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fae781ad010 con 0x55d32deda380 2024-09-16T00:25:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae82804740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae781aae60 con 0x55d32ded9a60 2024-09-16T00:25:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7e9c8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32df00f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7d9c6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32deda380 0x55d32df009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7e1c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 0x55d32dee7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7e9c8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32df00f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39084/0 (socket says 172.21.15.33:39084) 2024-09-16T00:25:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7e1c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 0x55d32dee7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34400/0 (socket says 172.21.15.33:34400) 2024-09-16T00:25:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7e1c7640 1 -- 172.21.15.33:0/1213542928 learned_addr learned my addr 172.21.15.33:0/1213542928 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:49.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7d9c6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32deda380 0x55d32df009d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:38094/0 (socket says 172.21.15.33:38094) 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7d9c6640 1 -- 172.21.15.33:0/1213542928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 msgr2=0x55d32dee7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/1213542928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 0x55d32dee7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7d9c6640 1 -- 172.21.15.33:0/1213542928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 msgr2=0x55d32df00f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/1213542928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32df00f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7d9c6640 1 -- 172.21.15.33:0/1213542928 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d32ded1830 con 0x55d32deda380 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7e9c8640 1 --2- 172.21.15.33:0/1213542928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32df00f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.980+0000 7fae7e1c7640 1 --2- 172.21.15.33:0/1213542928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 0x55d32dee7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.981+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/1213542928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32deda380 0x55d32df009d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae68000c00 tx=0x7fae68005b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f86400a4f9651b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.981+0000 7fae63fff640 1 -- 172.21.15.33:0/1213542928 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae68029070 con 0x55d32deda380 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.981+0000 7fae63fff640 1 -- 172.21.15.33:0/1213542928 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae68004680 con 0x55d32deda380 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.981+0000 7fae63fff640 1 -- 172.21.15.33:0/1213542928 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae68003a10 con 0x55d32deda380 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.982+0000 7fae82804740 1 -- 172.21.15.33:0/1213542928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32deda380 msgr2=0x55d32df009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.982+0000 7fae82804740 1 --2- 172.21.15.33:0/1213542928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32deda380 0x55d32df009d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae68000c00 tx=0x7fae68005b90 comp rx=0 tx=0).stop 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.982+0000 7fae82804740 1 -- 172.21.15.33:0/1213542928 shutdown_connections 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.982+0000 7fae82804740 1 --2- 172.21.15.33:0/1213542928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32df00f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.982+0000 7fae82804740 1 --2- 172.21.15.33:0/1213542928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32deda380 0x55d32df009d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.982+0000 7fae82804740 1 --2- 172.21.15.33:0/1213542928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 0x55d32dee7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.982+0000 7fae82804740 1 -- 172.21.15.33:0/1213542928 >> 172.21.15.33:0/1213542928 conn(0x55d32ded60f0 msgr2=0x55d32dee7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.982+0000 7fae82804740 1 -- 172.21.15.33:0/1213542928 shutdown_connections 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.982+0000 7fae82804740 1 -- 172.21.15.33:0/1213542928 wait complete. 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.982+0000 7fae82804740 1 -- start start 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.982+0000 7fae82804740 1 Processor -- start 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.982+0000 7fae82804740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 0x55d32def1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae82804740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32deda380 0x55d32deeca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae82804740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32deecf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae82804740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae781ab4d0 con 0x55d32dee9150 2024-09-16T00:25:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae82804740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fae781ac940 con 0x55d32deda380 2024-09-16T00:25:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae82804740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae781abba0 con 0x55d32ded9a60 2024-09-16T00:25:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae7e9c8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32deecf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae7e1c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 0x55d32def1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae82804740 10 client.0 _ll_register_callbacks cb 0x7fae68005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae7e9c8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32deecf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39098/0 (socket says 172.21.15.33:39098) 2024-09-16T00:25:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae7e9c8640 1 -- 172.21.15.33:0/1808456211 learned_addr learned my addr 172.21.15.33:0/1808456211 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32deda380 0x55d32deeca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae7e1c7640 1 -- 172.21.15.33:0/1808456211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32deda380 msgr2=0x55d32deeca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae7e1c7640 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32deda380 0x55d32deeca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae7e1c7640 1 -- 172.21.15.33:0/1808456211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 msgr2=0x55d32deecf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae7e1c7640 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32deecf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae7e1c7640 1 -- 172.21.15.33:0/1808456211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d32deed5d0 con 0x55d32ded9a60 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae7e9c8640 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32deecf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae7e1c7640 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 0x55d32def1720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fae7400c6a0 tx=0x7fae74015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb0082ed512934c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae61ffb640 1 -- 172.21.15.33:0/1808456211 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae740a0730 con 0x55d32ded9a60 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae61ffb640 1 -- 172.21.15.33:0/1808456211 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fae74010ac0 con 0x55d32ded9a60 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.983+0000 7fae82804740 1 -- 172.21.15.33:0/1808456211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae68004680 con 0x55d32ded9a60 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.984+0000 7fae61ffb640 1 -- 172.21.15.33:0/1808456211 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fae740a4ff0 con 0x55d32ded9a60 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.984+0000 7fae82804740 10 client.26440 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.984+0000 7fae82804740 1 -- 172.21.15.33:0/1808456211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d32df63890 con 0x55d32ded9a60 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.984+0000 7fae61ffb640 1 -- 172.21.15.33:0/1808456211 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae68004680 con 0x55d32ded9a60 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.984+0000 7fae61ffb640 1 -- 172.21.15.33:0/1808456211 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fae740980d0 con 0x55d32ded9a60 2024-09-16T00:25:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.984+0000 7fae82804740 10 client.26440 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.984+0000 7fae82804740 10 client.26440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.984+0000 7fae82804740 20 client.26440 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.984+0000 7fae82804740 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d32df63f60 0x55d32df66350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.984+0000 7fae82804740 4 client.26440 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:25:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.984+0000 7fae82804740 1 -- 172.21.15.33:0/1808456211 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d32de46d00 con 0x55d32df63f60 2024-09-16T00:25:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.984+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d32df63f60 0x55d32df66350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.985+0000 7fae7d9c6640 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d32df63f60 0x55d32df66350 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1eafdfe75e6a908c server_cookie=aefd9fe3de24a498 in_seq=0 out_seq=0 2024-09-16T00:25:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.985+0000 7fae61ffb640 10 client.26440 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae61ffb640 1 -- 172.21.15.33:0/1808456211 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d32de46d00 con 0x55d32df63f60 2024-09-16T00:25:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae61ffb640 10 client.26440 handle_command_reply: tid=0 2024-09-16T00:25:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae82804740 1 client.26440 shutdown 2024-09-16T00:25:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae82804740 1 -- 172.21.15.33:0/1808456211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 msgr2=0x55d32def1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae82804740 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 0x55d32def1720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fae7400c6a0 tx=0x7fae74015000 comp rx=0 tx=0).stop 2024-09-16T00:25:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae82804740 1 -- 172.21.15.33:0/1808456211 shutdown_connections 2024-09-16T00:25:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae82804740 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d32df63f60 0x55d32df66350 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae82804740 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d32dee9150 0x55d32deecf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae82804740 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d32deda380 0x55d32deeca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae82804740 1 --2- 172.21.15.33:0/1808456211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d32ded9a60 0x55d32def1720 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae61ffb640 0 client.26440 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae82804740 1 -- 172.21.15.33:0/1808456211 >> 172.21.15.33:0/1808456211 conn(0x55d32ded60f0 msgr2=0x55d32dedbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae82804740 1 -- 172.21.15.33:0/1808456211 shutdown_connections 2024-09-16T00:25:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae82804740 1 -- 172.21.15.33:0/1808456211 wait complete. 2024-09-16T00:25:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.986+0000 7fae82804740 20 client.26440 trim_cache size 0 max 16384 2024-09-16T00:25:49.998 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:25:49.998 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:25:49.998 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "ec5de78d-1dda-4e19-bded-cedf5af849af", 2024-09-16T00:25:49.998 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:25:49.998 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:25:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.987+0000 7fae8042b640 1 -- 172.21.15.33:0/1085509624 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae640a3240 msgr2=0x7fae640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:49.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.987+0000 7fae8042b640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae640a3240 0x7fae640a5700 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fae70009430 tx=0x7fae7002a950 comp rx=0 tx=0).stop 2024-09-16T00:25:49.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.987+0000 7fae8042b640 1 -- 172.21.15.33:0/1085509624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae78100e20 msgr2=0x7fae781973e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:49.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.987+0000 7fae8042b640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae78100e20 0x7fae781973e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fae7400e9f0 tx=0x7fae7400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.987+0000 7fae8042b640 1 -- 172.21.15.33:0/1085509624 shutdown_connections 2024-09-16T00:25:49.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.987+0000 7fae8042b640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae640a3240 0x7fae640a5700 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.987+0000 7fae8042b640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae78102ef0 0x7fae7819bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.987+0000 7fae8042b640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae78102570 0x7fae78197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.987+0000 7fae8042b640 1 --2- 172.21.15.33:0/1085509624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae78100e20 0x7fae781973e0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:49.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.987+0000 7fae8042b640 1 -- 172.21.15.33:0/1085509624 >> 172.21.15.33:0/1085509624 conn(0x7fae780fc7d0 msgr2=0x7fae780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:49.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.987+0000 7fae8042b640 1 -- 172.21.15.33:0/1085509624 shutdown_connections 2024-09-16T00:25:49.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:49.987+0000 7fae8042b640 1 -- 172.21.15.33:0/1085509624 wait complete. 2024-09-16T00:25:50.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:50 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:25:50.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:50 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:50.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 692608 10.44 MB/sec execute 298 sec latency 56.378 ms 2024-09-16T00:25:50.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:50 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:25:50.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:50 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:50.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:50 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:25:50.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:50 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:50.775 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:25:51.047 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:25:51.468 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:51 smithi033 ceph-mon[31243]: pgmap v611: 129 pgs: 129 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-09-16T00:25:51.468 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:51 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:25:51.468 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:51 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:51.469 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:51 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:25:51.469 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:51 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:51.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.555+0000 7f362723e640 1 -- 172.21.15.33:0/3227686150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3620102d90 msgr2=0x7f3620103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:51.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.555+0000 7f362723e640 1 --2- 172.21.15.33:0/3227686150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3620102d90 0x7f3620103170 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3608009930 tx=0x7f360802a420 comp rx=0 tx=0).stop 2024-09-16T00:25:51.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.556+0000 7f362723e640 1 -- 172.21.15.33:0/3227686150 shutdown_connections 2024-09-16T00:25:51.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.556+0000 7f362723e640 1 --2- 172.21.15.33:0/3227686150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3620108290 0x7f362010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.556+0000 7f362723e640 1 --2- 172.21.15.33:0/3227686150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3620103740 0x7f3620107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.556+0000 7f362723e640 1 --2- 172.21.15.33:0/3227686150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3620102d90 0x7f3620103170 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.556+0000 7f362723e640 1 -- 172.21.15.33:0/3227686150 >> 172.21.15.33:0/3227686150 conn(0x7f36200fc7d0 msgr2=0x7f36200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.556+0000 7f362723e640 1 -- 172.21.15.33:0/3227686150 shutdown_connections 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.556+0000 7f362723e640 1 -- 172.21.15.33:0/3227686150 wait complete. 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.556+0000 7f362723e640 1 Processor -- start 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.556+0000 7f362723e640 1 -- start start 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f362723e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3620102d90 0x7f36201009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f362723e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3620103740 0x7f3620100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f362723e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3620108290 0x7f3620101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f362723e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3620112090 con 0x7f3620103740 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f362723e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3620111f10 con 0x7f3620108290 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f362723e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3620112210 con 0x7f3620102d90 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f361ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3620103740 0x7f3620100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f3624fda640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3620102d90 0x7f36201009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f3624fda640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3620102d90 0x7f36201009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:57074/0 (socket says 172.21.15.33:57074) 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f361ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3620103740 0x7f3620100f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50502/0 (socket says 172.21.15.33:50502) 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f36257db640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3620108290 0x7f3620101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f3624fda640 1 -- 172.21.15.33:0/3239652654 learned_addr learned my addr 172.21.15.33:0/3239652654 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f361ffff640 1 -- 172.21.15.33:0/3239652654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3620102d90 msgr2=0x7f36201009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f361ffff640 1 --2- 172.21.15.33:0/3239652654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3620102d90 0x7f36201009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f361ffff640 1 -- 172.21.15.33:0/3239652654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3620108290 msgr2=0x7f3620101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f361ffff640 1 --2- 172.21.15.33:0/3239652654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3620108290 0x7f3620101470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f361ffff640 1 -- 172.21.15.33:0/3239652654 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36201a6a90 con 0x7f3620103740 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f361ffff640 1 --2- 172.21.15.33:0/3239652654 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3620103740 0x7f3620100f30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f361000c940 tx=0x7f361000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.557+0000 7f361dffb640 1 -- 172.21.15.33:0/3239652654 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610007bb0 con 0x7f3620103740 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.558+0000 7f361dffb640 1 -- 172.21.15.33:0/3239652654 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3610007d50 con 0x7f3620103740 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.558+0000 7f361dffb640 1 -- 172.21.15.33:0/3239652654 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361000f450 con 0x7f3620103740 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.558+0000 7f362723e640 1 -- 172.21.15.33:0/3239652654 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36201a6d80 con 0x7f3620103740 2024-09-16T00:25:51.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.558+0000 7f362723e640 1 -- 172.21.15.33:0/3239652654 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36201a7240 con 0x7f3620103740 2024-09-16T00:25:51.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.560+0000 7f361dffb640 1 -- 172.21.15.33:0/3239652654 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3610004f20 con 0x7f3620103740 2024-09-16T00:25:51.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f3629617740 1 Processor -- start 2024-09-16T00:25:51.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f3629617740 1 -- start start 2024-09-16T00:25:51.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f3629617740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897d4a0 0x55e25897d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f3629617740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897ddc0 0x55e258988140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f3629617740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e258988680 0x55e25898aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f3629617740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e2588e32e0 con 0x55e258988680 2024-09-16T00:25:51.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f361dffb640 1 --2- 172.21.15.33:0/3239652654 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f360c0a32c0 0x7f360c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f3629617740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e25888ace0 con 0x55e25897d4a0 2024-09-16T00:25:51.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f3629617740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e2588df2a0 con 0x55e25897ddc0 2024-09-16T00:25:51.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f36257db640 1 --2- 172.21.15.33:0/3239652654 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f360c0a32c0 0x7f360c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f361ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e258988680 0x55e25898aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f3624fda640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897ddc0 0x55e258988140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f36257db640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897d4a0 0x55e25897d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f3624fda640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897ddc0 0x55e258988140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:57090/0 (socket says 172.21.15.33:57090) 2024-09-16T00:25:51.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f361ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e258988680 0x55e25898aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50510/0 (socket says 172.21.15.33:50510) 2024-09-16T00:25:51.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f3624fda640 1 -- 172.21.15.33:0/3685239531 learned_addr learned my addr 172.21.15.33:0/3685239531 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:51.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.561+0000 7f361dffb640 1 -- 172.21.15.33:0/3239652654 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f36100cfcd0 con 0x7f3620103740 2024-09-16T00:25:51.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.562+0000 7f3624fda640 1 -- 172.21.15.33:0/3685239531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897d4a0 msgr2=0x55e25897d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:51.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.562+0000 7f3624fda640 1 --2- 172.21.15.33:0/3685239531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897d4a0 0x55e25897d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.562+0000 7f3624fda640 1 -- 172.21.15.33:0/3685239531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e258988680 msgr2=0x55e25898aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:51.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.562+0000 7f3624fda640 1 --2- 172.21.15.33:0/3685239531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e258988680 0x55e25898aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.562+0000 7f3624fda640 1 -- 172.21.15.33:0/3685239531 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e25886f440 con 0x55e25897ddc0 2024-09-16T00:25:51.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.562+0000 7f3624fda640 1 --2- 172.21.15.33:0/3685239531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897ddc0 0x55e258988140 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3608006010 tx=0x7f3608009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=514c825ba6d3daf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:51.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.562+0000 7f36257db640 1 --2- 172.21.15.33:0/3239652654 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f360c0a32c0 0x7f360c0a5780 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3618005fd0 tx=0x7f3618009e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:51.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.562+0000 7f3616ffd640 1 -- 172.21.15.33:0/3685239531 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608040070 con 0x55e25897ddc0 2024-09-16T00:25:51.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.563+0000 7f3616ffd640 1 -- 172.21.15.33:0/3685239531 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3608005020 con 0x55e25897ddc0 2024-09-16T00:25:51.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.563+0000 7f3616ffd640 1 -- 172.21.15.33:0/3685239531 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608035040 con 0x55e25897ddc0 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.563+0000 7f3629617740 1 -- 172.21.15.33:0/3685239531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897ddc0 msgr2=0x55e258988140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.563+0000 7f3629617740 1 --2- 172.21.15.33:0/3685239531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897ddc0 0x55e258988140 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3608006010 tx=0x7f3608009680 comp rx=0 tx=0).stop 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.563+0000 7f3629617740 1 -- 172.21.15.33:0/3685239531 shutdown_connections 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.563+0000 7f3629617740 1 --2- 172.21.15.33:0/3685239531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e258988680 0x55e25898aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.563+0000 7f3629617740 1 --2- 172.21.15.33:0/3685239531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897ddc0 0x55e258988140 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.563+0000 7f3629617740 1 --2- 172.21.15.33:0/3685239531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897d4a0 0x55e25897d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.563+0000 7f3629617740 1 -- 172.21.15.33:0/3685239531 >> 172.21.15.33:0/3685239531 conn(0x55e258979a30 msgr2=0x55e258979e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.564+0000 7f3629617740 1 -- 172.21.15.33:0/3685239531 shutdown_connections 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.564+0000 7f3629617740 1 -- 172.21.15.33:0/3685239531 wait complete. 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.564+0000 7f3629617740 1 -- start start 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.564+0000 7f3629617740 1 Processor -- start 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.564+0000 7f3629617740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897d4a0 0x55e258987ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.564+0000 7f3629617740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897ddc0 0x55e2589851a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.564+0000 7f3629617740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e258988680 0x55e2589856e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.564+0000 7f3629617740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e25898b250 con 0x55e258988680 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.564+0000 7f3629617740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e25889e050 con 0x55e25897ddc0 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.564+0000 7f3629617740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e25898b3d0 con 0x55e25897d4a0 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f3629617740 10 client.0 _ll_register_callbacks cb 0x55e2588de160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f3624fda640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897d4a0 0x55e258987ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f36257db640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e258988680 0x55e2589856e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f3624fda640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897d4a0 0x55e258987ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:57092/0 (socket says 172.21.15.33:57092) 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f3624fda640 1 -- 172.21.15.33:0/2356740868 learned_addr learned my addr 172.21.15.33:0/2356740868 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f361ffff640 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897ddc0 0x55e2589851a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f36257db640 1 -- 172.21.15.33:0/2356740868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897d4a0 msgr2=0x55e258987ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f36257db640 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897d4a0 0x55e258987ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f36257db640 1 -- 172.21.15.33:0/2356740868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897ddc0 msgr2=0x55e2589851a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:51.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f36257db640 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897ddc0 0x55e2589851a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f36257db640 1 -- 172.21.15.33:0/2356740868 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2586543c0 con 0x55e258988680 2024-09-16T00:25:51.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f3624fda640 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897d4a0 0x55e258987ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:51.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f36257db640 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e258988680 0x55e2589856e0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3618009fd0 tx=0x7f361801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a10829f1ba3301 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:51.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f35fffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:25:51.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f35fffff640 1 -- 172.21.15.33:0/2356740868 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3618024070 con 0x55e258988680 2024-09-16T00:25:51.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f35fffff640 1 -- 172.21.15.33:0/2356740868 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3618004790 con 0x55e258988680 2024-09-16T00:25:51.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.565+0000 7f3629617740 1 -- 172.21.15.33:0/2356740868 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3608005020 con 0x55e258988680 2024-09-16T00:25:51.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.566+0000 7f35fffff640 1 -- 172.21.15.33:0/2356740868 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3618033050 con 0x55e258988680 2024-09-16T00:25:51.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.566+0000 7f3629617740 10 client.17016 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:51.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.566+0000 7f3629617740 1 -- 172.21.15.33:0/2356740868 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e258996b40 con 0x55e258988680 2024-09-16T00:25:51.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.566+0000 7f35fffff640 1 -- 172.21.15.33:0/2356740868 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3608005020 con 0x55e258988680 2024-09-16T00:25:51.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.566+0000 7f35fffff640 1 -- 172.21.15.33:0/2356740868 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f3618028400 con 0x55e258988680 2024-09-16T00:25:51.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.566+0000 7f3629617740 10 client.17016 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:51.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.566+0000 7f3629617740 10 client.17016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:51.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.566+0000 7f3629617740 20 client.17016 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:51.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.566+0000 7f3629617740 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e258998000 0x55e25899a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.566+0000 7f3629617740 4 client.17016 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:25:51.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.566+0000 7f3629617740 1 -- 172.21.15.33:0/2356740868 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2588f0850 con 0x55e258998000 2024-09-16T00:25:51.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.566+0000 7f3624fda640 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e258998000 0x55e25899a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.567+0000 7f3624fda640 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e258998000 0x55e25899a4a0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1853031582933f51 server_cookie=2bcc0bb612fab83 in_seq=0 out_seq=0 2024-09-16T00:25:51.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.567+0000 7f35fffff640 10 client.17016 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:51.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f35fffff640 1 -- 172.21.15.33:0/2356740868 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e2588f0850 con 0x55e258998000 2024-09-16T00:25:51.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f35fffff640 10 client.17016 handle_command_reply: tid=0 2024-09-16T00:25:51.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f3629617740 1 client.17016 shutdown 2024-09-16T00:25:51.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f3629617740 1 -- 172.21.15.33:0/2356740868 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e258988680 msgr2=0x55e2589856e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:51.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f3629617740 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e258988680 0x55e2589856e0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3618009fd0 tx=0x7f361801d000 comp rx=0 tx=0).stop 2024-09-16T00:25:51.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f3629617740 1 -- 172.21.15.33:0/2356740868 shutdown_connections 2024-09-16T00:25:51.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f3629617740 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e258998000 0x55e25899a4a0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f3629617740 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e258988680 0x55e2589856e0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f3629617740 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897ddc0 0x55e2589851a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f3629617740 1 --2- 172.21.15.33:0/2356740868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e25897d4a0 0x55e258987ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f3629617740 1 -- 172.21.15.33:0/2356740868 >> 172.21.15.33:0/2356740868 conn(0x55e258979a30 msgr2=0x55e25897f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:51.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f35fffff640 0 client.17016 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:51.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f3629617740 1 -- 172.21.15.33:0/2356740868 shutdown_connections 2024-09-16T00:25:51.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f3629617740 1 -- 172.21.15.33:0/2356740868 wait complete. 2024-09-16T00:25:51.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.568+0000 7f3629617740 20 client.17016 trim_cache size 0 max 16384 2024-09-16T00:25:51.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3629617740 1 Processor -- start 2024-09-16T00:25:51.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3629617740 1 -- start start 2024-09-16T00:25:51.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3629617740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897d4a0 0x55e25899a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3629617740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897ddc0 0x55e258987110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3629617740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e258988680 0x55e25899fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3629617740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36201b3b90 con 0x55e25897d4a0 2024-09-16T00:25:51.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3629617740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f36201b1460 con 0x55e25897ddc0 2024-09-16T00:25:51.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3629617740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f36201af2b0 con 0x55e258988680 2024-09-16T00:25:51.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3624fda640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897d4a0 0x55e25899a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3624fda640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897d4a0 0x55e25899a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50534/0 (socket says 172.21.15.33:50534) 2024-09-16T00:25:51.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3624fda640 1 -- 172.21.15.33:0/1349749968 learned_addr learned my addr 172.21.15.33:0/1349749968 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:51.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f361ffff640 1 --2- 172.21.15.33:0/1349749968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897ddc0 0x55e258987110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f36257db640 1 --2- 172.21.15.33:0/1349749968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e258988680 0x55e25899fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3624fda640 1 -- 172.21.15.33:0/1349749968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e258988680 msgr2=0x55e25899fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:51.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3624fda640 1 --2- 172.21.15.33:0/1349749968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e258988680 0x55e25899fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3624fda640 1 -- 172.21.15.33:0/1349749968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897ddc0 msgr2=0x55e258987110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:51.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3624fda640 1 --2- 172.21.15.33:0/1349749968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897ddc0 0x55e258987110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.587+0000 7f3624fda640 1 -- 172.21.15.33:0/1349749968 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e258984170 con 0x55e25897d4a0 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.588+0000 7f3624fda640 1 --2- 172.21.15.33:0/1349749968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897d4a0 0x55e25899a010 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f3608009900 tx=0x7f3608009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1529b4d075d14ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.588+0000 7f36167fc640 1 -- 172.21.15.33:0/1349749968 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608040050 con 0x55e25897d4a0 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.588+0000 7f36167fc640 1 -- 172.21.15.33:0/1349749968 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3608005500 con 0x55e25897d4a0 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.588+0000 7f36167fc640 1 -- 172.21.15.33:0/1349749968 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608035040 con 0x55e25897d4a0 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.588+0000 7f3629617740 1 -- 172.21.15.33:0/1349749968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897d4a0 msgr2=0x55e25899a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.588+0000 7f3629617740 1 --2- 172.21.15.33:0/1349749968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897d4a0 0x55e25899a010 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f3608009900 tx=0x7f3608009610 comp rx=0 tx=0).stop 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 -- 172.21.15.33:0/1349749968 shutdown_connections 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 --2- 172.21.15.33:0/1349749968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e258988680 0x55e25899fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 --2- 172.21.15.33:0/1349749968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897ddc0 0x55e258987110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 --2- 172.21.15.33:0/1349749968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897d4a0 0x55e25899a010 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 -- 172.21.15.33:0/1349749968 >> 172.21.15.33:0/1349749968 conn(0x55e258979a30 msgr2=0x55e258986a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 -- 172.21.15.33:0/1349749968 shutdown_connections 2024-09-16T00:25:51.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 -- 172.21.15.33:0/1349749968 wait complete. 2024-09-16T00:25:51.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 -- start start 2024-09-16T00:25:51.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 Processor -- start 2024-09-16T00:25:51.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897d4a0 0x55e258990d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897ddc0 0x55e25898e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e258988680 0x55e25898e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36201af920 con 0x55e25897ddc0 2024-09-16T00:25:51.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f36201b0d90 con 0x55e25897d4a0 2024-09-16T00:25:51.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.589+0000 7f3629617740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f36201afff0 con 0x55e258988680 2024-09-16T00:25:51.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f3629617740 10 client.0 _ll_register_callbacks cb 0x7f3618009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:25:51.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f361ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897ddc0 0x55e25898e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f3624fda640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897d4a0 0x55e258990d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f36257db640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e258988680 0x55e25898e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f361ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897ddc0 0x55e25898e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50538/0 (socket says 172.21.15.33:50538) 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f3624fda640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897d4a0 0x55e258990d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:36766/0 (socket says 172.21.15.33:36766) 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f361ffff640 1 -- 172.21.15.33:0/190999523 learned_addr learned my addr 172.21.15.33:0/190999523 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f36257db640 1 -- 172.21.15.33:0/190999523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897d4a0 msgr2=0x55e258990d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f36257db640 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897d4a0 0x55e258990d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f36257db640 1 -- 172.21.15.33:0/190999523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897ddc0 msgr2=0x55e25898e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f36257db640 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897ddc0 0x55e25898e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f36257db640 1 -- 172.21.15.33:0/190999523 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e258988260 con 0x55e258988680 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f361ffff640 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897ddc0 0x55e25898e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f3624fda640 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897d4a0 0x55e258990d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f36257db640 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e258988680 0x55e25898e650 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f36180287f0 tx=0x7f36180286b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=135ba02bfb73263a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:25:51.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f35fffff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f35fffff640 1 -- 172.21.15.33:0/190999523 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3618024040 con 0x55e258988680 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f3629617740 1 -- 172.21.15.33:0/190999523 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3608005500 con 0x55e258988680 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.590+0000 7f35fffff640 1 -- 172.21.15.33:0/190999523 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3618028c70 con 0x55e258988680 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.591+0000 7f35fffff640 1 -- 172.21.15.33:0/190999523 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3618048050 con 0x55e258988680 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.591+0000 7f3629617740 10 client.26458 fetch_fsmap learned FSMap version 30 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.591+0000 7f3629617740 1 -- 172.21.15.33:0/190999523 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e258a02c00 con 0x55e258988680 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.591+0000 7f35fffff640 1 -- 172.21.15.33:0/190999523 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3608005500 con 0x55e258988680 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.591+0000 7f35fffff640 1 -- 172.21.15.33:0/190999523 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f361801f450 con 0x55e258988680 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.591+0000 7f3629617740 10 client.26458 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.591+0000 7f3629617740 10 client.26458 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.591+0000 7f3629617740 20 client.26458 populate_metadata read hostname 'smithi033' 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.591+0000 7f3629617740 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e258a03260 0x55e258a05650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.591+0000 7f3629617740 4 client.26458 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:25:51.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.591+0000 7f3629617740 1 -- 172.21.15.33:0/190999523 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e2588ead60 con 0x55e258a03260 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.591+0000 7f3624fda640 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e258a03260 0x55e258a05650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f35fffff640 10 client.26458 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3624fda640 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e258a03260 0x55e258a05650 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e30bd3512e914371 server_cookie=fb9757022b49426e in_seq=0 out_seq=0 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f35fffff640 1 -- 172.21.15.33:0/190999523 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e2588ead60 con 0x55e258a03260 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f35fffff640 10 client.26458 handle_command_reply: tid=0 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3629617740 1 client.26458 shutdown 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3629617740 1 -- 172.21.15.33:0/190999523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e258988680 msgr2=0x55e25898e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3629617740 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e258988680 0x55e25898e650 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f36180287f0 tx=0x7f36180286b0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3629617740 1 -- 172.21.15.33:0/190999523 shutdown_connections 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3629617740 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e258a03260 0x55e258a05650 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3629617740 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e258988680 0x55e25898e650 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3629617740 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e25897ddc0 0x55e25898e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f35fffff640 0 client.26458 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:25:51.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3629617740 1 --2- 172.21.15.33:0/190999523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e25897d4a0 0x55e258990d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3629617740 1 -- 172.21.15.33:0/190999523 >> 172.21.15.33:0/190999523 conn(0x55e258979a30 msgr2=0x55e258987220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:51.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3629617740 1 -- 172.21.15.33:0/190999523 shutdown_connections 2024-09-16T00:25:51.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3629617740 1 -- 172.21.15.33:0/190999523 wait complete. 2024-09-16T00:25:51.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.592+0000 7f3629617740 20 client.26458 trim_cache size 0 max 16384 2024-09-16T00:25:51.600 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:25:51.600 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:25:51.600 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:25:51.600 INFO:teuthology.orchestra.run.smithi033.stdout: "ec5de78d-1dda-4e19-bded-cedf5af849af": { 2024-09-16T00:25:51.601 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:25:51.601 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "ec5de78d-1dda-4e19-bded-cedf5af849af", 2024-09-16T00:25:51.601 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:25:51.601 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:51.601 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:25:51.601 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:25:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.593+0000 7f362723e640 1 -- 172.21.15.33:0/3239652654 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f360c0a32c0 msgr2=0x7f360c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.593+0000 7f362723e640 1 --2- 172.21.15.33:0/3239652654 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f360c0a32c0 0x7f360c0a5780 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3618005fd0 tx=0x7f3618009e80 comp rx=0 tx=0).stop 2024-09-16T00:25:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.593+0000 7f362723e640 1 -- 172.21.15.33:0/3239652654 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3620103740 msgr2=0x7f3620100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:25:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.593+0000 7f362723e640 1 --2- 172.21.15.33:0/3239652654 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3620103740 0x7f3620100f30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f361000c940 tx=0x7f361000ce00 comp rx=0 tx=0).stop 2024-09-16T00:25:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.593+0000 7f362723e640 1 -- 172.21.15.33:0/3239652654 shutdown_connections 2024-09-16T00:25:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.593+0000 7f362723e640 1 --2- 172.21.15.33:0/3239652654 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f360c0a32c0 0x7f360c0a5780 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.593+0000 7f362723e640 1 --2- 172.21.15.33:0/3239652654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3620108290 0x7f3620101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.593+0000 7f362723e640 1 --2- 172.21.15.33:0/3239652654 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3620103740 0x7f3620100f30 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.593+0000 7f362723e640 1 --2- 172.21.15.33:0/3239652654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3620102d90 0x7f36201009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:25:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.593+0000 7f362723e640 1 -- 172.21.15.33:0/3239652654 >> 172.21.15.33:0/3239652654 conn(0x7f36200fc7d0 msgr2=0x7f3620106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:25:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.593+0000 7f362723e640 1 -- 172.21.15.33:0/3239652654 shutdown_connections 2024-09-16T00:25:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:25:51.593+0000 7f362723e640 1 -- 172.21.15.33:0/3239652654 wait complete. 2024-09-16T00:25:51.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:51 smithi083 ceph-mon[35242]: pgmap v611: 129 pgs: 129 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-09-16T00:25:51.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:51 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:25:51.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:51 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:51.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:51 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:25:51.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:51 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:51.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 694932 10.44 MB/sec execute 299 sec latency 28.597 ms 2024-09-16T00:25:51.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:51 smithi060 ceph-mon[35185]: pgmap v611: 129 pgs: 129 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-09-16T00:25:51.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:51 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:25:51.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:51 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:51.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:51 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:25:51.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:51 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:25:52.308 INFO:tasks.cephfs.filesystem:scrub status for tag:ec5de78d-1dda-4e19-bded-cedf5af849af - {'path': '/', 'tag': 'ec5de78d-1dda-4e19-bded-cedf5af849af', 'options': 'recursive,force'} 2024-09-16T00:25:52.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 697195 10.45 MB/sec execute 300 sec latency 16.256 ms 2024-09-16T00:25:53.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:53 smithi083 ceph-mon[35242]: pgmap v612: 129 pgs: 129 active+clean; 253 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-09-16T00:25:53.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 700150 10.47 MB/sec execute 301 sec latency 14.229 ms 2024-09-16T00:25:53.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:53 smithi060 ceph-mon[35185]: pgmap v612: 129 pgs: 129 active+clean; 253 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-09-16T00:25:53.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:53 smithi033 ceph-mon[31243]: pgmap v612: 129 pgs: 129 active+clean; 253 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-09-16T00:25:54.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 703248 10.50 MB/sec execute 302 sec latency 20.832 ms 2024-09-16T00:25:55.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:55 smithi083 ceph-mon[35242]: pgmap v613: 129 pgs: 129 active+clean; 257 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 227 op/s 2024-09-16T00:25:55.669 INFO:tasks.workunit.client.0.smithi033.stdout: 1 705706 10.49 MB/sec execute 303 sec latency 17.289 ms 2024-09-16T00:25:55.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:55 smithi060 ceph-mon[35185]: pgmap v613: 129 pgs: 129 active+clean; 257 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 227 op/s 2024-09-16T00:25:55.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:55 smithi033 ceph-mon[31243]: pgmap v613: 129 pgs: 129 active+clean; 257 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 227 op/s 2024-09-16T00:25:56.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 708083 10.51 MB/sec execute 304 sec latency 11.750 ms 2024-09-16T00:25:57.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:57 smithi083 ceph-mon[35242]: pgmap v614: 129 pgs: 129 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 202 op/s 2024-09-16T00:25:57.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:57 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:25:57.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:57 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:25:57.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 710788 10.52 MB/sec execute 305 sec latency 28.107 ms 2024-09-16T00:25:57.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:57 smithi060 ceph-mon[35185]: pgmap v614: 129 pgs: 129 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 202 op/s 2024-09-16T00:25:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:57 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:25:57.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:57 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:25:57.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:57 smithi033 ceph-mon[31243]: pgmap v614: 129 pgs: 129 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 202 op/s 2024-09-16T00:25:57.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:57 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:25:57.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:57 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:25:58.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 713763 10.55 MB/sec execute 306 sec latency 32.208 ms 2024-09-16T00:25:59.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:25:59 smithi083 ceph-mon[35242]: pgmap v615: 129 pgs: 129 active+clean; 259 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 177 op/s 2024-09-16T00:25:59.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 716183 10.54 MB/sec execute 307 sec latency 16.175 ms 2024-09-16T00:25:59.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:25:59 smithi060 ceph-mon[35185]: pgmap v615: 129 pgs: 129 active+clean; 259 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 177 op/s 2024-09-16T00:25:59.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:25:59 smithi033 ceph-mon[31243]: pgmap v615: 129 pgs: 129 active+clean; 259 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 177 op/s 2024-09-16T00:26:00.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 718386 10.55 MB/sec execute 308 sec latency 20.667 ms 2024-09-16T00:26:01.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:01 smithi083 ceph-mon[35242]: pgmap v616: 129 pgs: 129 active+clean; 265 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-09-16T00:26:01.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 721414 10.57 MB/sec execute 309 sec latency 6.583 ms 2024-09-16T00:26:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:01 smithi060 ceph-mon[35185]: pgmap v616: 129 pgs: 129 active+clean; 265 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-09-16T00:26:01.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:01 smithi033 ceph-mon[31243]: pgmap v616: 129 pgs: 129 active+clean; 265 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-09-16T00:26:02.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 724612 10.60 MB/sec execute 310 sec latency 16.253 ms 2024-09-16T00:26:03.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:03 smithi083 ceph-mon[35242]: pgmap v617: 129 pgs: 129 active+clean; 265 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 200 op/s 2024-09-16T00:26:03.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 726383 10.58 MB/sec execute 311 sec latency 102.548 ms 2024-09-16T00:26:03.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:03 smithi060 ceph-mon[35185]: pgmap v617: 129 pgs: 129 active+clean; 265 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 200 op/s 2024-09-16T00:26:03.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:03 smithi033 ceph-mon[31243]: pgmap v617: 129 pgs: 129 active+clean; 265 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 200 op/s 2024-09-16T00:26:04.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 728927 10.60 MB/sec execute 312 sec latency 14.029 ms 2024-09-16T00:26:05.671 INFO:tasks.workunit.client.0.smithi033.stdout: 1 732063 10.62 MB/sec execute 313 sec latency 14.315 ms 2024-09-16T00:26:05.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:05 smithi060 ceph-mon[35185]: pgmap v618: 129 pgs: 129 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 259 op/s 2024-09-16T00:26:05.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:05 smithi033 ceph-mon[31243]: pgmap v618: 129 pgs: 129 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 259 op/s 2024-09-16T00:26:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:05 smithi083 ceph-mon[35242]: pgmap v618: 129 pgs: 129 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 259 op/s 2024-09-16T00:26:06.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 735317 10.65 MB/sec execute 314 sec latency 13.226 ms 2024-09-16T00:26:07.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 738247 10.66 MB/sec execute 315 sec latency 13.478 ms 2024-09-16T00:26:07.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:07 smithi060 ceph-mon[35185]: pgmap v619: 129 pgs: 129 active+clean; 272 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 248 op/s 2024-09-16T00:26:07.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:07 smithi033 ceph-mon[31243]: pgmap v619: 129 pgs: 129 active+clean; 272 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 248 op/s 2024-09-16T00:26:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:07 smithi083 ceph-mon[35242]: pgmap v619: 129 pgs: 129 active+clean; 272 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 248 op/s 2024-09-16T00:26:08.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 739589 10.64 MB/sec execute 316 sec latency 84.062 ms 2024-09-16T00:26:09.096 INFO:tasks.quiescer.fs.[cephfs]:Total duration (44.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T00:26:09.096 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.35 2024-09-16T00:26:09.097 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:26:09.374 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:09.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 742839 10.67 MB/sec execute 317 sec latency 12.635 ms 2024-09-16T00:26:09.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:09 smithi060 ceph-mon[35185]: pgmap v620: 129 pgs: 129 active+clean; 272 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 244 op/s 2024-09-16T00:26:09.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:09 smithi033 ceph-mon[31243]: pgmap v620: 129 pgs: 129 active+clean; 272 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 244 op/s 2024-09-16T00:26:09.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.867+0000 7f145e0ed640 1 -- 172.21.15.33:0/2455187540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14581036e0 msgr2=0x7f1458107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:09.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.867+0000 7f145e0ed640 1 --2- 172.21.15.33:0/2455187540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14581036e0 0x7f1458107b90 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f1440009930 tx=0x7f144002a420 comp rx=0 tx=0).stop 2024-09-16T00:26:09.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.867+0000 7f145e0ed640 1 -- 172.21.15.33:0/2455187540 shutdown_connections 2024-09-16T00:26:09.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.867+0000 7f145e0ed640 1 --2- 172.21.15.33:0/2455187540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14581082a0 0x7f145810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:09.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.867+0000 7f145e0ed640 1 --2- 172.21.15.33:0/2455187540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14581036e0 0x7f1458107b90 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:09.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.867+0000 7f145e0ed640 1 --2- 172.21.15.33:0/2455187540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1458102d30 0x7f1458103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:09.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.867+0000 7f145e0ed640 1 -- 172.21.15.33:0/2455187540 >> 172.21.15.33:0/2455187540 conn(0x7f14580fc7d0 msgr2=0x7f14580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:09.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.867+0000 7f145e0ed640 1 -- 172.21.15.33:0/2455187540 shutdown_connections 2024-09-16T00:26:09.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.867+0000 7f145e0ed640 1 -- 172.21.15.33:0/2455187540 wait complete. 2024-09-16T00:26:09.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f145e0ed640 1 Processor -- start 2024-09-16T00:26:09.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f145e0ed640 1 -- start start 2024-09-16T00:26:09.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f145e0ed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1458102d30 0x7f14581985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:09.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f145e0ed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14581036e0 0x7f1458198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:09.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f145e0ed640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14581082a0 0x7f145819ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:09.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f145e0ed640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f14581120a0 con 0x7f14581082a0 2024-09-16T00:26:09.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f145e0ed640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1458111f20 con 0x7f1458102d30 2024-09-16T00:26:09.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f145e0ed640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1458112220 con 0x7f14581036e0 2024-09-16T00:26:09.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f1457fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14581082a0 0x7f145819ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:09.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f144ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14581036e0 0x7f1458198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:09.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f1457fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14581082a0 0x7f145819ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55784/0 (socket says 172.21.15.33:55784) 2024-09-16T00:26:09.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f14577fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1458102d30 0x7f14581985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:09.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.868+0000 7f1457fff640 1 -- 172.21.15.33:0/49544934 learned_addr learned my addr 172.21.15.33:0/49544934 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:09.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.869+0000 7f1457fff640 1 -- 172.21.15.33:0/49544934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14581036e0 msgr2=0x7f1458198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:09.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.869+0000 7f1457fff640 1 --2- 172.21.15.33:0/49544934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14581036e0 0x7f1458198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:09.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.869+0000 7f1457fff640 1 -- 172.21.15.33:0/49544934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1458102d30 msgr2=0x7f14581985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:09.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.869+0000 7f1457fff640 1 --2- 172.21.15.33:0/49544934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1458102d30 0x7f14581985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:09.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.869+0000 7f1457fff640 1 -- 172.21.15.33:0/49544934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145819d590 con 0x7f14581082a0 2024-09-16T00:26:09.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.869+0000 7f1457fff640 1 --2- 172.21.15.33:0/49544934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14581082a0 0x7f145819ceb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f145000eb10 tx=0x7f145000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:09.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.869+0000 7f14557fa640 1 -- 172.21.15.33:0/49544934 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1450004b20 con 0x7f14581082a0 2024-09-16T00:26:09.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.869+0000 7f14557fa640 1 -- 172.21.15.33:0/49544934 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1450004cc0 con 0x7f14581082a0 2024-09-16T00:26:09.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.869+0000 7f145e0ed640 1 -- 172.21.15.33:0/49544934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f145819d880 con 0x7f14581082a0 2024-09-16T00:26:09.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.869+0000 7f14557fa640 1 -- 172.21.15.33:0/49544934 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1450010760 con 0x7f14581082a0 2024-09-16T00:26:09.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.869+0000 7f145e0ed640 1 -- 172.21.15.33:0/49544934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14581008c0 con 0x7f14581082a0 2024-09-16T00:26:09.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.870+0000 7f145e0ed640 1 -- 172.21.15.33:0/49544934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f145806a850 con 0x7f14581082a0 2024-09-16T00:26:09.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.871+0000 7f14557fa640 1 -- 172.21.15.33:0/49544934 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1450005020 con 0x7f14581082a0 2024-09-16T00:26:09.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.872+0000 7f14557fa640 1 --2- 172.21.15.33:0/49544934 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f14440a32c0 0x7f14440a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:09.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.872+0000 7f14577fe640 1 --2- 172.21.15.33:0/49544934 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f14440a32c0 0x7f14440a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:09.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.872+0000 7f14557fa640 1 -- 172.21.15.33:0/49544934 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f14500d0220 con 0x7f14581082a0 2024-09-16T00:26:09.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.873+0000 7f14577fe640 1 --2- 172.21.15.33:0/49544934 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f14440a32c0 0x7f14440a5780 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1448010d80 tx=0x7f1448012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:09.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:09.877+0000 7f14557fa640 1 -- 172.21.15.33:0/49544934 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f145009cac0 con 0x7f14581082a0 2024-09-16T00:26:09.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:09 smithi083 ceph-mon[35242]: pgmap v620: 129 pgs: 129 active+clean; 272 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 244 op/s 2024-09-16T00:26:10.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.103+0000 7f145e0ed640 1 -- 172.21.15.33:0/49544934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f145810a2b0 con 0x7f14581082a0 2024-09-16T00:26:10.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.104+0000 7f14557fa640 1 -- 172.21.15.33:0/49544934 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f145009c3c0 con 0x7f14581082a0 2024-09-16T00:26:10.106 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:26:10.106 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:26:10.106 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:26:10.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.106+0000 7f145e0ed640 1 -- 172.21.15.33:0/49544934 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f14440a32c0 msgr2=0x7f14440a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:10.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.106+0000 7f145e0ed640 1 --2- 172.21.15.33:0/49544934 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f14440a32c0 0x7f14440a5780 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1448010d80 tx=0x7f1448012040 comp rx=0 tx=0).stop 2024-09-16T00:26:10.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.106+0000 7f145e0ed640 1 -- 172.21.15.33:0/49544934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14581082a0 msgr2=0x7f145819ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:10.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.106+0000 7f145e0ed640 1 --2- 172.21.15.33:0/49544934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14581082a0 0x7f145819ceb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f145000eb10 tx=0x7f145000c2d0 comp rx=0 tx=0).stop 2024-09-16T00:26:10.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.106+0000 7f145e0ed640 1 -- 172.21.15.33:0/49544934 shutdown_connections 2024-09-16T00:26:10.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.106+0000 7f145e0ed640 1 --2- 172.21.15.33:0/49544934 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f14440a32c0 0x7f14440a5780 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:10.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.106+0000 7f145e0ed640 1 --2- 172.21.15.33:0/49544934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14581082a0 0x7f145819ceb0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:10.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.106+0000 7f145e0ed640 1 --2- 172.21.15.33:0/49544934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14581036e0 0x7f1458198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:10.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.106+0000 7f145e0ed640 1 --2- 172.21.15.33:0/49544934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1458102d30 0x7f14581985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:10.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.106+0000 7f145e0ed640 1 -- 172.21.15.33:0/49544934 >> 172.21.15.33:0/49544934 conn(0x7f14580fc7d0 msgr2=0x7f1458105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:10.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.107+0000 7f145e0ed640 1 -- 172.21.15.33:0/49544934 shutdown_connections 2024-09-16T00:26:10.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:10.107+0000 7f145e0ed640 1 -- 172.21.15.33:0/49544934 wait complete. 2024-09-16T00:26:10.671 INFO:tasks.workunit.client.0.smithi033.stdout: 1 745968 10.69 MB/sec execute 318 sec latency 33.167 ms 2024-09-16T00:26:10.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:10 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/49544934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:10.816 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:10 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/49544934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:10.843 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0' 2024-09-16T00:26:10.843 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0 2024-09-16T00:26:10.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:10 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/49544934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:11.118 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:11.504 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:11 smithi033 ceph-mon[31243]: pgmap v621: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 290 op/s 2024-09-16T00:26:11.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.603+0000 7fb550eb0640 1 -- 172.21.15.33:0/3470371831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb54c103660 msgr2=0x7fb54c10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:11.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.603+0000 7fb550eb0640 1 --2- 172.21.15.33:0/3470371831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb54c103660 0x7fb54c10bb50 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fb540009900 tx=0x7fb54002a480 comp rx=0 tx=0).stop 2024-09-16T00:26:11.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.603+0000 7fb550eb0640 1 -- 172.21.15.33:0/3470371831 shutdown_connections 2024-09-16T00:26:11.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.603+0000 7fb550eb0640 1 --2- 172.21.15.33:0/3470371831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c10c170 0x7fb54c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.603+0000 7fb550eb0640 1 --2- 172.21.15.33:0/3470371831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb54c103660 0x7fb54c10bb50 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.603+0000 7fb550eb0640 1 --2- 172.21.15.33:0/3470371831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb54c102d40 0x7fb54c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.603+0000 7fb550eb0640 1 -- 172.21.15.33:0/3470371831 >> 172.21.15.33:0/3470371831 conn(0x7fb54c0fc7a0 msgr2=0x7fb54c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:11.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.603+0000 7fb550eb0640 1 -- 172.21.15.33:0/3470371831 shutdown_connections 2024-09-16T00:26:11.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.603+0000 7fb550eb0640 1 -- 172.21.15.33:0/3470371831 wait complete. 2024-09-16T00:26:11.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.604+0000 7fb550eb0640 1 Processor -- start 2024-09-16T00:26:11.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.604+0000 7fb550eb0640 1 -- start start 2024-09-16T00:26:11.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.604+0000 7fb550eb0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb54c102d40 0x7fb54c1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.604+0000 7fb550eb0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb54c103660 0x7fb54c100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.604+0000 7fb550eb0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c10c170 0x7fb54c101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.604+0000 7fb550eb0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c111f70 con 0x7fb54c103660 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.604+0000 7fb550eb0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c111df0 con 0x7fb54c10c170 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.604+0000 7fb550eb0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c1120f0 con 0x7fb54c102d40 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb549d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb54c103660 0x7fb54c100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb549d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb54c103660 0x7fb54c100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58312/0 (socket says 172.21.15.33:58312) 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb549d9b640 1 -- 172.21.15.33:0/403005075 learned_addr learned my addr 172.21.15.33:0/403005075 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c10c170 0x7fb54c101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb54a59c640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb54c102d40 0x7fb54c1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb549d9b640 1 -- 172.21.15.33:0/403005075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb54c102d40 msgr2=0x7fb54c1009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb549d9b640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb54c102d40 0x7fb54c1009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb549d9b640 1 -- 172.21.15.33:0/403005075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c10c170 msgr2=0x7fb54c101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:11.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb549d9b640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c10c170 0x7fb54c101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb549d9b640 1 -- 172.21.15.33:0/403005075 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54c1a6910 con 0x7fb54c103660 2024-09-16T00:26:11.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c10c170 0x7fb54c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:11.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb549d9b640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb54c103660 0x7fb54c100ee0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fb540006010 tx=0x7fb540031b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:11.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb53b7fe640 1 -- 172.21.15.33:0/403005075 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54003f030 con 0x7fb54c103660 2024-09-16T00:26:11.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb53b7fe640 1 -- 172.21.15.33:0/403005075 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5400043d0 con 0x7fb54c103660 2024-09-16T00:26:11.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb550eb0640 1 -- 172.21.15.33:0/403005075 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54c1a6ba0 con 0x7fb54c103660 2024-09-16T00:26:11.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.605+0000 7fb53b7fe640 1 -- 172.21.15.33:0/403005075 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb540035070 con 0x7fb54c103660 2024-09-16T00:26:11.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.606+0000 7fb550eb0640 1 -- 172.21.15.33:0/403005075 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54c1a7080 con 0x7fb54c103660 2024-09-16T00:26:11.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.608+0000 7fb53b7fe640 1 -- 172.21.15.33:0/403005075 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb540042030 con 0x7fb54c103660 2024-09-16T00:26:11.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.609+0000 7fb553289740 1 Processor -- start 2024-09-16T00:26:11.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.609+0000 7fb553289740 1 -- start start 2024-09-16T00:26:11.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.609+0000 7fb553289740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6379850 0x558ec6379c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.609+0000 7fb553289740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 0x558ec63844f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.609+0000 7fb553289740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6384a30 0x558ec6386e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.609+0000 7fb553289740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558ec62de160 con 0x558ec6384a30 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.609+0000 7fb553289740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558ec6285720 con 0x558ec6379850 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.609+0000 7fb553289740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558ec62da120 con 0x558ec637a170 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.609+0000 7fb54ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6379850 0x558ec6379c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.609+0000 7fb54ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6379850 0x558ec6379c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:53266/0 (socket says 172.21.15.33:53266) 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.609+0000 7fb54ad9d640 1 -- 172.21.15.33:0/765928293 learned_addr learned my addr 172.21.15.33:0/765928293 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.609+0000 7fb54a59c640 1 --2- 172.21.15.33:0/765928293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 0x558ec63844f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb549d9b640 1 --2- 172.21.15.33:0/765928293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6384a30 0x558ec6386e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb53b7fe640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb5340a3240 0x7fb5340a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb54ad9d640 1 -- 172.21.15.33:0/765928293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 msgr2=0x558ec63844f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:11.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/765928293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 0x558ec63844f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb54ad9d640 1 -- 172.21.15.33:0/765928293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6384a30 msgr2=0x558ec6386e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/765928293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6384a30 0x558ec6386e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb54ad9d640 1 -- 172.21.15.33:0/765928293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ec604fba0 con 0x558ec6379850 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb54a59c640 1 --2- 172.21.15.33:0/765928293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 0x558ec63844f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb53b7fe640 1 -- 172.21.15.33:0/403005075 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb5400efff0 con 0x7fb54c103660 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb5340a3240 0x7fb5340a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/765928293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6379850 0x558ec6379c30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb5440118d0 tx=0x7fb544011d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9992365db6183a3e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb538ff9640 1 -- 172.21.15.33:0/765928293 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb544017050 con 0x558ec6379850 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb538ff9640 1 -- 172.21.15.33:0/765928293 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb544026440 con 0x558ec6379850 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb5340a3240 0x7fb5340a5700 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb544023000 tx=0x7fb544024000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.610+0000 7fb538ff9640 1 -- 172.21.15.33:0/765928293 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54400e610 con 0x558ec6379850 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.611+0000 7fb553289740 1 -- 172.21.15.33:0/765928293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6379850 msgr2=0x558ec6379c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.611+0000 7fb553289740 1 --2- 172.21.15.33:0/765928293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6379850 0x558ec6379c30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb5440118d0 tx=0x7fb544011d90 comp rx=0 tx=0).stop 2024-09-16T00:26:11.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.611+0000 7fb553289740 1 -- 172.21.15.33:0/765928293 shutdown_connections 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.611+0000 7fb553289740 1 --2- 172.21.15.33:0/765928293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6384a30 0x558ec6386e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.611+0000 7fb553289740 1 --2- 172.21.15.33:0/765928293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 0x558ec63844f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.611+0000 7fb553289740 1 --2- 172.21.15.33:0/765928293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6379850 0x558ec6379c30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.611+0000 7fb553289740 1 -- 172.21.15.33:0/765928293 >> 172.21.15.33:0/765928293 conn(0x558ec6375de0 msgr2=0x558ec63761c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.611+0000 7fb553289740 1 -- 172.21.15.33:0/765928293 shutdown_connections 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.611+0000 7fb553289740 1 -- 172.21.15.33:0/765928293 wait complete. 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb553289740 1 -- start start 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb553289740 1 Processor -- start 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb553289740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 0x558ec6381900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb553289740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 0x558ec6381e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb553289740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6384a30 0x558ec6382380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb553289740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558ec6387600 con 0x558ec6379850 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb553289740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558ec6266900 con 0x558ec6384a30 2024-09-16T00:26:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb553289740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558ec6387780 con 0x558ec637a170 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb553289740 10 client.0 _ll_register_callbacks cb 0x558ec6264780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb54a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 0x558ec6381900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb549d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 0x558ec6381e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb54a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 0x558ec6381900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58332/0 (socket says 172.21.15.33:58332) 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb54a59c640 1 -- 172.21.15.33:0/2984478574 learned_addr learned my addr 172.21.15.33:0/2984478574 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb549d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 0x558ec6381e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:39424/0 (socket says 172.21.15.33:39424) 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.612+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6384a30 0x558ec6382380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb549d9b640 1 -- 172.21.15.33:0/2984478574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6384a30 msgr2=0x558ec6382380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb549d9b640 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6384a30 0x558ec6382380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb549d9b640 1 -- 172.21.15.33:0/2984478574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 msgr2=0x558ec6381900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb549d9b640 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 0x558ec6381900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb549d9b640 1 -- 172.21.15.33:0/2984478574 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ec62a6c50 con 0x558ec637a170 2024-09-16T00:26:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6384a30 0x558ec6382380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:26:11.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb549d9b640 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 0x558ec6381e40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb5400ed6c0 tx=0x7fb5400bbf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f64d85a2ddd3b84 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:11.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb531ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:26:11.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb531ffb640 1 -- 172.21.15.33:0/2984478574 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb540049070 con 0x558ec637a170 2024-09-16T00:26:11.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb531ffb640 1 -- 172.21.15.33:0/2984478574 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb54008d100 con 0x558ec637a170 2024-09-16T00:26:11.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb553289740 1 -- 172.21.15.33:0/2984478574 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb544026440 con 0x558ec637a170 2024-09-16T00:26:11.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb531ffb640 1 -- 172.21.15.33:0/2984478574 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb5400c4ec0 con 0x558ec637a170 2024-09-16T00:26:11.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb553289740 10 client.26476 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:11.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.613+0000 7fb553289740 1 -- 172.21.15.33:0/2984478574 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558ec6387600 con 0x558ec637a170 2024-09-16T00:26:11.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.614+0000 7fb531ffb640 1 -- 172.21.15.33:0/2984478574 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb544026440 con 0x558ec637a170 2024-09-16T00:26:11.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.614+0000 7fb531ffb640 1 -- 172.21.15.33:0/2984478574 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fb5400dc0b0 con 0x558ec637a170 2024-09-16T00:26:11.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.614+0000 7fb553289740 10 client.26476 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:11.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.614+0000 7fb553289740 10 client.26476 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:26:11.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.614+0000 7fb553289740 20 client.26476 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:11.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.614+0000 7fb553289740 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558ec6394660 0x558ec6396aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.614+0000 7fb553289740 4 client.26476 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:26:11.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.614+0000 7fb553289740 1 -- 172.21.15.33:0/2984478574 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ec62eb890 con 0x558ec6394660 2024-09-16T00:26:11.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.614+0000 7fb54a59c640 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558ec6394660 0x558ec6396aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.615+0000 7fb54a59c640 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558ec6394660 0x558ec6396aa0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c876a17b26effcab server_cookie=2694ad2abb9778a8 in_seq=0 out_seq=0 2024-09-16T00:26:11.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.615+0000 7fb531ffb640 10 client.26476 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:11.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.615+0000 7fb531ffb640 1 -- 172.21.15.33:0/2984478574 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558ec62eb890 con 0x558ec6394660 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.615+0000 7fb531ffb640 10 client.26476 handle_command_reply: tid=0 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.615+0000 7fb553289740 1 client.26476 shutdown 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.615+0000 7fb553289740 1 -- 172.21.15.33:0/2984478574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 msgr2=0x558ec6381e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.615+0000 7fb553289740 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 0x558ec6381e40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb5400ed6c0 tx=0x7fb5400bbf00 comp rx=0 tx=0).stop 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.615+0000 7fb553289740 1 -- 172.21.15.33:0/2984478574 shutdown_connections 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.616+0000 7fb553289740 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558ec6394660 0x558ec6396aa0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.616+0000 7fb553289740 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec6384a30 0x558ec6382380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.616+0000 7fb553289740 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec637a170 0x558ec6381e40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.616+0000 7fb553289740 1 --2- 172.21.15.33:0/2984478574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 0x558ec6381900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.616+0000 7fb531ffb640 0 client.26476 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.616+0000 7fb553289740 1 -- 172.21.15.33:0/2984478574 >> 172.21.15.33:0/2984478574 conn(0x558ec6375de0 msgr2=0x558ec637bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.616+0000 7fb553289740 1 -- 172.21.15.33:0/2984478574 shutdown_connections 2024-09-16T00:26:11.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.616+0000 7fb553289740 1 -- 172.21.15.33:0/2984478574 wait complete. 2024-09-16T00:26:11.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.616+0000 7fb553289740 20 client.26476 trim_cache size 0 max 16384 2024-09-16T00:26:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.633+0000 7fb553289740 1 Processor -- start 2024-09-16T00:26:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.633+0000 7fb553289740 1 -- start start 2024-09-16T00:26:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb553289740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec6379850 0x558ec63801d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb553289740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec637a170 0x558ec639c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb553289740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6384a30 0x558ec639c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb553289740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c1b3ab0 con 0x558ec6384a30 2024-09-16T00:26:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb553289740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c1b1380 con 0x558ec637a170 2024-09-16T00:26:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb553289740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c1af1d0 con 0x558ec6379850 2024-09-16T00:26:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb549d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec637a170 0x558ec639c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb54a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec6379850 0x558ec63801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb549d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec637a170 0x558ec639c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:53288/0 (socket says 172.21.15.33:53288) 2024-09-16T00:26:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb549d9b640 1 -- 172.21.15.33:0/1976234233 learned_addr learned my addr 172.21.15.33:0/1976234233 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:11.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/1976234233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6384a30 0x558ec639c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb549d9b640 1 -- 172.21.15.33:0/1976234233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec6379850 msgr2=0x558ec63801d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:11.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb549d9b640 1 --2- 172.21.15.33:0/1976234233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec6379850 0x558ec63801d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb549d9b640 1 -- 172.21.15.33:0/1976234233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6384a30 msgr2=0x558ec639c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:11.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb549d9b640 1 --2- 172.21.15.33:0/1976234233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6384a30 0x558ec639c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb549d9b640 1 -- 172.21.15.33:0/1976234233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ec637d2b0 con 0x558ec637a170 2024-09-16T00:26:11.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.634+0000 7fb54a59c640 1 --2- 172.21.15.33:0/1976234233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec6379850 0x558ec63801d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:11.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.635+0000 7fb549d9b640 1 --2- 172.21.15.33:0/1976234233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec637a170 0x558ec639c370 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb5400ed9e0 tx=0x7fb54002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2260cb847820485d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:11.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.635+0000 7fb533fff640 1 -- 172.21.15.33:0/1976234233 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb540049070 con 0x558ec637a170 2024-09-16T00:26:11.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.635+0000 7fb533fff640 1 -- 172.21.15.33:0/1976234233 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54008d100 con 0x558ec637a170 2024-09-16T00:26:11.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.635+0000 7fb533fff640 1 -- 172.21.15.33:0/1976234233 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5400dc040 con 0x558ec637a170 2024-09-16T00:26:11.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.635+0000 7fb553289740 1 -- 172.21.15.33:0/1976234233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec637a170 msgr2=0x558ec639c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:11.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.635+0000 7fb553289740 1 --2- 172.21.15.33:0/1976234233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec637a170 0x558ec639c370 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb5400ed9e0 tx=0x7fb54002a480 comp rx=0 tx=0).stop 2024-09-16T00:26:11.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 -- 172.21.15.33:0/1976234233 shutdown_connections 2024-09-16T00:26:11.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 --2- 172.21.15.33:0/1976234233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6384a30 0x558ec639c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 --2- 172.21.15.33:0/1976234233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec637a170 0x558ec639c370 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 --2- 172.21.15.33:0/1976234233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec6379850 0x558ec63801d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 -- 172.21.15.33:0/1976234233 >> 172.21.15.33:0/1976234233 conn(0x558ec6375de0 msgr2=0x558ec637bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 -- 172.21.15.33:0/1976234233 shutdown_connections 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 -- 172.21.15.33:0/1976234233 wait complete. 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 -- start start 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 Processor -- start 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 0x558ec638ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec637a170 0x558ec63891f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec6384a30 0x558ec6389730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c1af840 con 0x558ec6379850 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c1b0cb0 con 0x558ec637a170 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb553289740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c1aff10 con 0x558ec6384a30 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb54a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 0x558ec638ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb549d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec637a170 0x558ec63891f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb54a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 0x558ec638ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58354/0 (socket says 172.21.15.33:58354) 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb54ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec6384a30 0x558ec6389730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.636+0000 7fb54a59c640 1 -- 172.21.15.33:0/721609962 learned_addr learned my addr 172.21.15.33:0/721609962 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb553289740 10 client.0 _ll_register_callbacks cb 0x7fb540005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb54ad9d640 1 -- 172.21.15.33:0/721609962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec637a170 msgr2=0x558ec63891f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/721609962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec637a170 0x558ec63891f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb54ad9d640 1 -- 172.21.15.33:0/721609962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 msgr2=0x558ec638ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/721609962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 0x558ec638ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb54ad9d640 1 -- 172.21.15.33:0/721609962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ec63808b0 con 0x558ec6384a30 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb54a59c640 1 --2- 172.21.15.33:0/721609962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 0x558ec638ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb54ad9d640 1 --2- 172.21.15.33:0/721609962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec6384a30 0x558ec6389730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb54400f790 tx=0x7fb544004dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=511ea2d3a02ea571 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb531ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:26:11.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb531ffb640 1 -- 172.21.15.33:0/721609962 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54400e5f0 con 0x558ec6384a30 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb553289740 1 -- 172.21.15.33:0/721609962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb54008d100 con 0x558ec6384a30 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb531ffb640 1 -- 172.21.15.33:0/721609962 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb54402b050 con 0x558ec6384a30 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb531ffb640 1 -- 172.21.15.33:0/721609962 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb544017020 con 0x558ec6384a30 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb553289740 10 client.26482 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.637+0000 7fb553289740 1 -- 172.21.15.33:0/721609962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558ec6379fa0 con 0x558ec6384a30 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.638+0000 7fb531ffb640 1 -- 172.21.15.33:0/721609962 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54008d100 con 0x558ec6384a30 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.638+0000 7fb531ffb640 1 -- 172.21.15.33:0/721609962 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fb54400ebb0 con 0x558ec6384a30 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.638+0000 7fb553289740 10 client.26482 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.638+0000 7fb553289740 10 client.26482 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.638+0000 7fb553289740 20 client.26482 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.638+0000 7fb553289740 1 --2- 172.21.15.33:0/721609962 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558ec6400300 0x558ec64026f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.638+0000 7fb553289740 4 client.26482 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}] 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.638+0000 7fb553289740 1 -- 172.21.15.33:0/721609962 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}) -- 0x558ec62e5be0 con 0x558ec6400300 2024-09-16T00:26:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.638+0000 7fb54a59c640 1 --2- 172.21.15.33:0/721609962 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558ec6400300 0x558ec64026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:11.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.639+0000 7fb531ffb640 10 client.26482 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:11.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.639+0000 7fb54a59c640 1 --2- 172.21.15.33:0/721609962 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558ec6400300 0x558ec64026f0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95e8806d4eef8d77 server_cookie=f9f6268936e3d62 in_seq=0 out_seq=0 2024-09-16T00:26:11.670 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748599 10.70 MB/sec execute 319 sec latency 43.921 ms 2024-09-16T00:26:11.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:11 smithi060 ceph-mon[35185]: pgmap v621: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 290 op/s 2024-09-16T00:26:11.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb531ffb640 1 -- 172.21.15.33:0/721609962 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x558ec62e5be0 con 0x558ec6400300 2024-09-16T00:26:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb531ffb640 10 client.26482 handle_command_reply: tid=0 2024-09-16T00:26:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb553289740 1 client.26482 shutdown 2024-09-16T00:26:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb553289740 1 -- 172.21.15.33:0/721609962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec6384a30 msgr2=0x558ec6389730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb553289740 1 --2- 172.21.15.33:0/721609962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec6384a30 0x558ec6389730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb54400f790 tx=0x7fb544004dd0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb553289740 1 -- 172.21.15.33:0/721609962 shutdown_connections 2024-09-16T00:26:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb553289740 1 --2- 172.21.15.33:0/721609962 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558ec6400300 0x558ec64026f0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb553289740 1 --2- 172.21.15.33:0/721609962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558ec6384a30 0x558ec6389730 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb553289740 1 --2- 172.21.15.33:0/721609962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558ec637a170 0x558ec63891f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb553289740 1 --2- 172.21.15.33:0/721609962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558ec6379850 0x558ec638ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb531ffb640 0 client.26482 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb553289740 1 -- 172.21.15.33:0/721609962 >> 172.21.15.33:0/721609962 conn(0x558ec6375de0 msgr2=0x558ec63818c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb553289740 1 -- 172.21.15.33:0/721609962 shutdown_connections 2024-09-16T00:26:11.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.733+0000 7fb553289740 1 -- 172.21.15.33:0/721609962 wait complete. 2024-09-16T00:26:11.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb553289740 20 client.26482 trim_cache size 0 max 16384 2024-09-16T00:26:11.736 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:26:11.736 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:26:11.736 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:26:11.736 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 30, 2024-09-16T00:26:11.736 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:26:11.736 INFO:teuthology.orchestra.run.smithi033.stdout: "de406742": { 2024-09-16T00:26:11.736 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 30, 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 142.3, 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:26:11.737 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:11.738 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:11.738 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:11.738 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:11.738 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:11.738 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:26:11.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb550eb0640 1 -- 172.21.15.33:0/403005075 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb5340a3240 msgr2=0x7fb5340a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:11.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb550eb0640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb5340a3240 0x7fb5340a5700 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb544023000 tx=0x7fb544024000 comp rx=0 tx=0).stop 2024-09-16T00:26:11.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb550eb0640 1 -- 172.21.15.33:0/403005075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb54c103660 msgr2=0x7fb54c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:11.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb550eb0640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb54c103660 0x7fb54c100ee0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fb540006010 tx=0x7fb540031b20 comp rx=0 tx=0).stop 2024-09-16T00:26:11.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb550eb0640 1 -- 172.21.15.33:0/403005075 shutdown_connections 2024-09-16T00:26:11.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb550eb0640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb5340a3240 0x7fb5340a5700 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb550eb0640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c10c170 0x7fb54c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb550eb0640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb54c103660 0x7fb54c100ee0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb550eb0640 1 --2- 172.21.15.33:0/403005075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb54c102d40 0x7fb54c1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:11.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb550eb0640 1 -- 172.21.15.33:0/403005075 >> 172.21.15.33:0/403005075 conn(0x7fb54c0fc7a0 msgr2=0x7fb54c0fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:11.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb550eb0640 1 -- 172.21.15.33:0/403005075 shutdown_connections 2024-09-16T00:26:11.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:11.734+0000 7fb550eb0640 1 -- 172.21.15.33:0/403005075 wait complete. 2024-09-16T00:26:11.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:11 smithi083 ceph-mon[35242]: pgmap v621: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 290 op/s 2024-09-16T00:26:12.671 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.67 MB/sec execute 320 sec latency 937.145 ms 2024-09-16T00:26:12.673 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'de406742', quiesce took 3.6 seconds. Will release after: 18.75 2024-09-16T00:26:13.671 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.63 MB/sec execute 321 sec latency 1937.251 ms 2024-09-16T00:26:13.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:13 smithi060 ceph-mon[35185]: pgmap v622: 129 pgs: 129 active+clean; 275 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 255 op/s 2024-09-16T00:26:13.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:13 smithi033 ceph-mon[31243]: pgmap v622: 129 pgs: 129 active+clean; 275 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 255 op/s 2024-09-16T00:26:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:13 smithi083 ceph-mon[35242]: pgmap v622: 129 pgs: 129 active+clean; 275 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 255 op/s 2024-09-16T00:26:14.671 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.60 MB/sec execute 322 sec latency 2937.356 ms 2024-09-16T00:26:15.671 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.57 MB/sec execute 323 sec latency 3937.446 ms 2024-09-16T00:26:15.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:15 smithi060 ceph-mon[35185]: pgmap v623: 129 pgs: 129 active+clean; 279 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 298 op/s 2024-09-16T00:26:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:15 smithi033 ceph-mon[31243]: pgmap v623: 129 pgs: 129 active+clean; 279 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 298 op/s 2024-09-16T00:26:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:15 smithi083 ceph-mon[35242]: pgmap v623: 129 pgs: 129 active+clean; 279 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 298 op/s 2024-09-16T00:26:16.671 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.53 MB/sec execute 324 sec latency 4937.579 ms 2024-09-16T00:26:16.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:26:16.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:26:16.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:26:16.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:26:16.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:26:16.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:26:16.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:26:16.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:26:16.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:26:16.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:26:16.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:26:16.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:26:17.671 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.50 MB/sec execute 325 sec latency 5937.678 ms 2024-09-16T00:26:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:17 smithi060 ceph-mon[35185]: pgmap v624: 129 pgs: 129 active+clean; 285 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 256 op/s 2024-09-16T00:26:17.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:17 smithi033 ceph-mon[31243]: pgmap v624: 129 pgs: 129 active+clean; 285 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 256 op/s 2024-09-16T00:26:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:17 smithi083 ceph-mon[35242]: pgmap v624: 129 pgs: 129 active+clean; 285 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 256 op/s 2024-09-16T00:26:18.671 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.47 MB/sec execute 326 sec latency 6937.777 ms 2024-09-16T00:26:19.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.44 MB/sec execute 327 sec latency 7937.870 ms 2024-09-16T00:26:19.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:19 smithi060 ceph-mon[35185]: pgmap v625: 129 pgs: 129 active+clean; 286 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 195 op/s 2024-09-16T00:26:19.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:19 smithi033 ceph-mon[31243]: pgmap v625: 129 pgs: 129 active+clean; 286 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 195 op/s 2024-09-16T00:26:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:19 smithi083 ceph-mon[35242]: pgmap v625: 129 pgs: 129 active+clean; 286 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 195 op/s 2024-09-16T00:26:20.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.41 MB/sec execute 328 sec latency 8937.991 ms 2024-09-16T00:26:21.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.37 MB/sec execute 329 sec latency 9938.117 ms 2024-09-16T00:26:21.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:21 smithi060 ceph-mon[35185]: pgmap v626: 129 pgs: 129 active+clean; 285 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 201 op/s 2024-09-16T00:26:21.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:21 smithi033 ceph-mon[31243]: pgmap v626: 129 pgs: 129 active+clean; 285 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 201 op/s 2024-09-16T00:26:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:21 smithi083 ceph-mon[35242]: pgmap v626: 129 pgs: 129 active+clean; 285 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 201 op/s 2024-09-16T00:26:22.310 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:26:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:22.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.34 MB/sec execute 330 sec latency 10938.162 ms 2024-09-16T00:26:23.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.066+0000 7f13b9868640 1 -- 172.21.15.33:0/3991261258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13b4101420 msgr2=0x7f13b410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:23.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.066+0000 7f13b9868640 1 --2- 172.21.15.33:0/3991261258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13b4101420 0x7f13b410d950 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f13ac009cd0 tx=0x7f13ac02a8e0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.066+0000 7f13b9868640 1 -- 172.21.15.33:0/3991261258 shutdown_connections 2024-09-16T00:26:23.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.066+0000 7f13b9868640 1 --2- 172.21.15.33:0/3991261258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13b4101420 0x7f13b410d950 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.066+0000 7f13b9868640 1 --2- 172.21.15.33:0/3991261258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b4100a80 0x7f13b4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.066+0000 7f13b9868640 1 --2- 172.21.15.33:0/3991261258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13b4106a80 0x7f13b4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.066+0000 7f13b9868640 1 -- 172.21.15.33:0/3991261258 >> 172.21.15.33:0/3991261258 conn(0x7f13b40fc7a0 msgr2=0x7f13b40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:23.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.066+0000 7f13b9868640 1 -- 172.21.15.33:0/3991261258 shutdown_connections 2024-09-16T00:26:23.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.066+0000 7f13b9868640 1 -- 172.21.15.33:0/3991261258 wait complete. 2024-09-16T00:26:23.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.067+0000 7f13b9868640 1 Processor -- start 2024-09-16T00:26:23.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.067+0000 7f13b9868640 1 -- start start 2024-09-16T00:26:23.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.067+0000 7f13b9868640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13b4100a80 0x7f13b40fcd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.067+0000 7f13b9868640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b4101420 0x7f13b40fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.067+0000 7f13b9868640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13b4106a80 0x7f13b40fd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.067+0000 7f13b9868640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f13b4110e80 con 0x7f13b4100a80 2024-09-16T00:26:23.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.067+0000 7f13b9868640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f13b4110d00 con 0x7f13b4106a80 2024-09-16T00:26:23.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.067+0000 7f13b9868640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13b4111000 con 0x7f13b4101420 2024-09-16T00:26:23.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.067+0000 7f13b2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13b4100a80 0x7f13b40fcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.067+0000 7f13b2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13b4100a80 0x7f13b40fcd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43722/0 (socket says 172.21.15.33:43722) 2024-09-16T00:26:23.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.067+0000 7f13b2ffd640 1 -- 172.21.15.33:0/3229783408 learned_addr learned my addr 172.21.15.33:0/3229783408 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:23.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.067+0000 7f13b27fc640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b4101420 0x7f13b40fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b37fe640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13b4106a80 0x7f13b40fd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b2ffd640 1 -- 172.21.15.33:0/3229783408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b4101420 msgr2=0x7f13b40fd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:23.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b4101420 0x7f13b40fd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b2ffd640 1 -- 172.21.15.33:0/3229783408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13b4106a80 msgr2=0x7f13b40fd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:23.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13b4106a80 0x7f13b40fd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b2ffd640 1 -- 172.21.15.33:0/3229783408 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b41a6870 con 0x7f13b4100a80 2024-09-16T00:26:23.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b37fe640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13b4106a80 0x7f13b40fd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:26:23.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13b4100a80 0x7f13b40fcd30 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f13a400c890 tx=0x7f13a400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:23.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b8866640 1 -- 172.21.15.33:0/3229783408 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4004220 con 0x7f13b4100a80 2024-09-16T00:26:23.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b8866640 1 -- 172.21.15.33:0/3229783408 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a40043c0 con 0x7f13b4100a80 2024-09-16T00:26:23.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b9868640 1 -- 172.21.15.33:0/3229783408 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b41a6b60 con 0x7f13b4100a80 2024-09-16T00:26:23.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b8866640 1 -- 172.21.15.33:0/3229783408 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4004a00 con 0x7f13b4100a80 2024-09-16T00:26:23.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.068+0000 7f13b9868640 1 -- 172.21.15.33:0/3229783408 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b41a70a0 con 0x7f13b4100a80 2024-09-16T00:26:23.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.070+0000 7f13b8866640 1 -- 172.21.15.33:0/3229783408 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f13a40079b0 con 0x7f13b4100a80 2024-09-16T00:26:23.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13bbc41740 1 Processor -- start 2024-09-16T00:26:23.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13bbc41740 1 -- start start 2024-09-16T00:26:23.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13bbc41740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28a4a0 0x55beed28a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13bbc41740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28adc0 0x55beed295140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13bbc41740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed295680 0x55beed297a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13bbc41740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55beed1f02e0 con 0x55beed28adc0 2024-09-16T00:26:23.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13bbc41740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55beed197ce0 con 0x55beed28a4a0 2024-09-16T00:26:23.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13bbc41740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55beed1ec2a0 con 0x55beed295680 2024-09-16T00:26:23.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28adc0 0x55beed295140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28adc0 0x55beed295140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43728/0 (socket says 172.21.15.33:43728) 2024-09-16T00:26:23.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b2ffd640 1 -- 172.21.15.33:0/1056393585 learned_addr learned my addr 172.21.15.33:0/1056393585 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b8866640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f139c0a3240 0x7f139c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b27fc640 1 --2- 172.21.15.33:0/1056393585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed295680 0x55beed297a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b37fe640 1 --2- 172.21.15.33:0/1056393585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28a4a0 0x55beed28a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b37fe640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f139c0a3240 0x7f139c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b8866640 1 -- 172.21.15.33:0/3229783408 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f13a40cfa40 con 0x7f13b4100a80 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b2ffd640 1 -- 172.21.15.33:0/1056393585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed295680 msgr2=0x55beed297a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/1056393585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed295680 0x55beed297a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b2ffd640 1 -- 172.21.15.33:0/1056393585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28a4a0 msgr2=0x55beed28a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/1056393585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28a4a0 0x55beed28a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.071+0000 7f13b2ffd640 1 -- 172.21.15.33:0/1056393585 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55beed17c440 con 0x55beed28adc0 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.072+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/1056393585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28adc0 0x55beed295140 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f13a406a210 tx=0x7f13a4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a30f211be0ed78 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.072+0000 7f1399ffb640 1 -- 172.21.15.33:0/1056393585 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a401d070 con 0x55beed28adc0 2024-09-16T00:26:23.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.072+0000 7f1399ffb640 1 -- 172.21.15.33:0/1056393585 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a409c280 con 0x55beed28adc0 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.072+0000 7f1399ffb640 1 -- 172.21.15.33:0/1056393585 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a406ef00 con 0x55beed28adc0 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 -- 172.21.15.33:0/1056393585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28adc0 msgr2=0x55beed295140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13b37fe640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f139c0a3240 0x7f139c0a5700 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f13ac006010 tx=0x7f13ac0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 --2- 172.21.15.33:0/1056393585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28adc0 0x55beed295140 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f13a406a210 tx=0x7f13a4000bc0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 -- 172.21.15.33:0/1056393585 shutdown_connections 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 --2- 172.21.15.33:0/1056393585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed295680 0x55beed297a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 --2- 172.21.15.33:0/1056393585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28adc0 0x55beed295140 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 --2- 172.21.15.33:0/1056393585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28a4a0 0x55beed28a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 -- 172.21.15.33:0/1056393585 >> 172.21.15.33:0/1056393585 conn(0x55beed286a30 msgr2=0x55beed286e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 -- 172.21.15.33:0/1056393585 shutdown_connections 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 -- 172.21.15.33:0/1056393585 wait complete. 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 -- start start 2024-09-16T00:26:23.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 Processor -- start 2024-09-16T00:26:23.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28a4a0 0x55beed294f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28adc0 0x55beed292200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.073+0000 7f13bbc41740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 0x55beed292740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13bbc41740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55beed298250 con 0x55beed295680 2024-09-16T00:26:23.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13bbc41740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55beed1ab050 con 0x55beed28adc0 2024-09-16T00:26:23.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13bbc41740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55beed2983d0 con 0x55beed28a4a0 2024-09-16T00:26:23.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13bbc41740 10 client.0 _ll_register_callbacks cb 0x55beed1eb160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:23.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13b37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 0x55beed292740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13b37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 0x55beed292740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43736/0 (socket says 172.21.15.33:43736) 2024-09-16T00:26:23.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13b37fe640 1 -- 172.21.15.33:0/1342304603 learned_addr learned my addr 172.21.15.33:0/1342304603 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:23.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28a4a0 0x55beed294f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13b27fc640 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28adc0 0x55beed292200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13b37fe640 1 -- 172.21.15.33:0/1342304603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28a4a0 msgr2=0x55beed294f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:23.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13b37fe640 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28a4a0 0x55beed294f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13b37fe640 1 -- 172.21.15.33:0/1342304603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28adc0 msgr2=0x55beed292200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:23.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13b37fe640 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28adc0 0x55beed292200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13b37fe640 1 -- 172.21.15.33:0/1342304603 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55beecf613c0 con 0x55beed295680 2024-09-16T00:26:23.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13b37fe640 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 0x55beed292740 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f13ac003bf0 tx=0x7f13ac02a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b72e5c05bfba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:23.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f1392ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:26:23.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f1392ffd640 1 -- 172.21.15.33:0/1342304603 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac0047c0 con 0x55beed295680 2024-09-16T00:26:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f1392ffd640 1 -- 172.21.15.33:0/1342304603 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f13ac0305a0 con 0x55beed295680 2024-09-16T00:26:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13bbc41740 1 -- 172.21.15.33:0/1342304603 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a409c280 con 0x55beed295680 2024-09-16T00:26:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f1392ffd640 1 -- 172.21.15.33:0/1342304603 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f13ac0307a0 con 0x55beed295680 2024-09-16T00:26:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.074+0000 7f13bbc41740 10 client.17058 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.075+0000 7f13bbc41740 1 -- 172.21.15.33:0/1342304603 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55beed2a3bd0 con 0x55beed295680 2024-09-16T00:26:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.075+0000 7f1392ffd640 1 -- 172.21.15.33:0/1342304603 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a409c280 con 0x55beed295680 2024-09-16T00:26:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.075+0000 7f1392ffd640 1 -- 172.21.15.33:0/1342304603 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f13ac047040 con 0x55beed295680 2024-09-16T00:26:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.075+0000 7f13bbc41740 10 client.17058 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.075+0000 7f13bbc41740 10 client.17058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:26:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.075+0000 7f13bbc41740 20 client.17058 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.075+0000 7f13bbc41740 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55beed2a50e0 0x55beed2a7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.075+0000 7f13bbc41740 4 client.17058 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:26:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.075+0000 7f13bbc41740 1 -- 172.21.15.33:0/1342304603 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55beed1fd850 con 0x55beed2a50e0 2024-09-16T00:26:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.075+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55beed2a50e0 0x55beed2a7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55beed2a50e0 0x55beed2a7520 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e588838734c7d79a server_cookie=d915ebc077fd164f in_seq=0 out_seq=0 2024-09-16T00:26:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f1392ffd640 10 client.17058 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f1392ffd640 1 -- 172.21.15.33:0/1342304603 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55beed1fd850 con 0x55beed2a50e0 2024-09-16T00:26:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f1392ffd640 10 client.17058 handle_command_reply: tid=0 2024-09-16T00:26:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f13bbc41740 1 client.17058 shutdown 2024-09-16T00:26:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f13bbc41740 1 -- 172.21.15.33:0/1342304603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 msgr2=0x55beed292740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f13bbc41740 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 0x55beed292740 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f13ac003bf0 tx=0x7f13ac02a8e0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f13bbc41740 1 -- 172.21.15.33:0/1342304603 shutdown_connections 2024-09-16T00:26:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f13bbc41740 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55beed2a50e0 0x55beed2a7520 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f13bbc41740 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 0x55beed292740 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f13bbc41740 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28adc0 0x55beed292200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f13bbc41740 1 --2- 172.21.15.33:0/1342304603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28a4a0 0x55beed294f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f13bbc41740 1 -- 172.21.15.33:0/1342304603 >> 172.21.15.33:0/1342304603 conn(0x55beed286a30 msgr2=0x55beed28c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.076+0000 7f1392ffd640 0 client.17058 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.077+0000 7f13bbc41740 1 -- 172.21.15.33:0/1342304603 shutdown_connections 2024-09-16T00:26:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.077+0000 7f13bbc41740 1 -- 172.21.15.33:0/1342304603 wait complete. 2024-09-16T00:26:23.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.077+0000 7f13bbc41740 20 client.17058 trim_cache size 0 max 16384 2024-09-16T00:26:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.094+0000 7f13bbc41740 1 Processor -- start 2024-09-16T00:26:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.094+0000 7f13bbc41740 1 -- start start 2024-09-16T00:26:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13bbc41740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28a4a0 0x55beed2a14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13bbc41740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28adc0 0x55beed286e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13bbc41740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed295680 0x55beed28d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13bbc41740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f13b41b3c30 con 0x55beed28a4a0 2024-09-16T00:26:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13bbc41740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f13b41b1500 con 0x55beed295680 2024-09-16T00:26:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13bbc41740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13b41af350 con 0x55beed28adc0 2024-09-16T00:26:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13b2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28a4a0 0x55beed2a14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13b2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28a4a0 0x55beed2a14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43750/0 (socket says 172.21.15.33:43750) 2024-09-16T00:26:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13b2ffd640 1 -- 172.21.15.33:0/3221352673 learned_addr learned my addr 172.21.15.33:0/3221352673 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13b37fe640 1 --2- 172.21.15.33:0/3221352673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed295680 0x55beed28d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13b27fc640 1 --2- 172.21.15.33:0/3221352673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28adc0 0x55beed286e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13b2ffd640 1 -- 172.21.15.33:0/3221352673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28adc0 msgr2=0x55beed286e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/3221352673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28adc0 0x55beed286e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13b2ffd640 1 -- 172.21.15.33:0/3221352673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed295680 msgr2=0x55beed28d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/3221352673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed295680 0x55beed28d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13b2ffd640 1 -- 172.21.15.33:0/3221352673 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55beed2911b0 con 0x55beed28a4a0 2024-09-16T00:26:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13b37fe640 1 --2- 172.21.15.33:0/3221352673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed295680 0x55beed28d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:26:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/3221352673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28a4a0 0x55beed2a14e0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f13a4005fd0 tx=0x7f13a40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b839923eb67d4d5f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13997fa640 1 -- 172.21.15.33:0/3221352673 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a401d040 con 0x55beed28a4a0 2024-09-16T00:26:23.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13997fa640 1 -- 172.21.15.33:0/3221352673 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a409c280 con 0x55beed28a4a0 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.095+0000 7f13997fa640 1 -- 172.21.15.33:0/3221352673 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a40981f0 con 0x55beed28a4a0 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.096+0000 7f13bbc41740 1 -- 172.21.15.33:0/3221352673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28a4a0 msgr2=0x55beed2a14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.096+0000 7f13bbc41740 1 --2- 172.21.15.33:0/3221352673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28a4a0 0x55beed2a14e0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f13a4005fd0 tx=0x7f13a40057c0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.096+0000 7f13bbc41740 1 -- 172.21.15.33:0/3221352673 shutdown_connections 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.096+0000 7f13bbc41740 1 --2- 172.21.15.33:0/3221352673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed295680 0x55beed28d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.096+0000 7f13bbc41740 1 --2- 172.21.15.33:0/3221352673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28adc0 0x55beed286e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.096+0000 7f13bbc41740 1 --2- 172.21.15.33:0/3221352673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed28a4a0 0x55beed2a14e0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.096+0000 7f13bbc41740 1 -- 172.21.15.33:0/3221352673 >> 172.21.15.33:0/3221352673 conn(0x55beed286a30 msgr2=0x55beed28d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.096+0000 7f13bbc41740 1 -- 172.21.15.33:0/3221352673 shutdown_connections 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.096+0000 7f13bbc41740 1 -- 172.21.15.33:0/3221352673 wait complete. 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13bbc41740 1 -- start start 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13bbc41740 1 Processor -- start 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13bbc41740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28a4a0 0x55beed29df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13bbc41740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28adc0 0x55beed28f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13bbc41740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 0x55beed28f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13bbc41740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f13b41af9c0 con 0x55beed295680 2024-09-16T00:26:23.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13bbc41740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f13b41b0e30 con 0x55beed28a4a0 2024-09-16T00:26:23.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13bbc41740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13b41b0090 con 0x55beed28adc0 2024-09-16T00:26:23.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13bbc41740 10 client.0 _ll_register_callbacks cb 0x7f13ac009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:23.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13b37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 0x55beed28f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13b37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 0x55beed28f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43766/0 (socket says 172.21.15.33:43766) 2024-09-16T00:26:23.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13b37fe640 1 -- 172.21.15.33:0/780771863 learned_addr learned my addr 172.21.15.33:0/780771863 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:23.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13b27fc640 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28adc0 0x55beed28f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.097+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28a4a0 0x55beed29df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f13b37fe640 1 -- 172.21.15.33:0/780771863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28adc0 msgr2=0x55beed28f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:23.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f13b37fe640 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28adc0 0x55beed28f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f13b37fe640 1 -- 172.21.15.33:0/780771863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28a4a0 msgr2=0x55beed29df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:23.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f13b37fe640 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28a4a0 0x55beed29df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f13b37fe640 1 -- 172.21.15.33:0/780771863 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55beed2952c0 con 0x55beed295680 2024-09-16T00:26:23.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f13b37fe640 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 0x55beed28f800 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f13ac045040 tx=0x7f13ac04a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55bf121e8f8236de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:23.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f1392ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:26:23.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f1392ffd640 1 -- 172.21.15.33:0/780771863 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac0318c0 con 0x55beed295680 2024-09-16T00:26:23.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f1392ffd640 1 -- 172.21.15.33:0/780771863 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f13ac004620 con 0x55beed295680 2024-09-16T00:26:23.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f13bbc41740 1 -- 172.21.15.33:0/780771863 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a409c280 con 0x55beed295680 2024-09-16T00:26:23.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f1392ffd640 1 -- 172.21.15.33:0/780771863 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f13ac004820 con 0x55beed295680 2024-09-16T00:26:23.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f13bbc41740 10 client.17064 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:23.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f13bbc41740 1 -- 172.21.15.33:0/780771863 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55beed30fd70 con 0x55beed295680 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f1392ffd640 1 -- 172.21.15.33:0/780771863 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a409c280 con 0x55beed295680 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.098+0000 7f1392ffd640 1 -- 172.21.15.33:0/780771863 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f13ac05f070 con 0x55beed295680 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.099+0000 7f13bbc41740 10 client.17064 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.099+0000 7f13bbc41740 10 client.17064 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.099+0000 7f13bbc41740 20 client.17064 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.099+0000 7f13bbc41740 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55beed310540 0x55beed312960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.099+0000 7f13bbc41740 4 client.17064 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.099+0000 7f13bbc41740 1 -- 172.21.15.33:0/780771863 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55beed1f7d60 con 0x55beed310540 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.099+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55beed310540 0x55beed312960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.099+0000 7f1392ffd640 10 client.17064 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.099+0000 7f13b2ffd640 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55beed310540 0x55beed312960 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=102a87926fab7a15 server_cookie=bc33ffb602e4dad9 in_seq=0 out_seq=0 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.099+0000 7f1392ffd640 1 -- 172.21.15.33:0/780771863 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55beed1f7d60 con 0x55beed310540 2024-09-16T00:26:23.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.099+0000 7f1392ffd640 10 client.17064 handle_command_reply: tid=0 2024-09-16T00:26:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.099+0000 7f13bbc41740 1 client.17064 shutdown 2024-09-16T00:26:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13bbc41740 1 -- 172.21.15.33:0/780771863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 msgr2=0x55beed28f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13bbc41740 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 0x55beed28f800 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f13ac045040 tx=0x7f13ac04a680 comp rx=0 tx=0).stop 2024-09-16T00:26:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13bbc41740 1 -- 172.21.15.33:0/780771863 shutdown_connections 2024-09-16T00:26:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13bbc41740 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55beed310540 0x55beed312960 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13bbc41740 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55beed295680 0x55beed28f800 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13bbc41740 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55beed28adc0 0x55beed28f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13bbc41740 1 --2- 172.21.15.33:0/780771863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55beed28a4a0 0x55beed29df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13bbc41740 1 -- 172.21.15.33:0/780771863 >> 172.21.15.33:0/780771863 conn(0x55beed286a30 msgr2=0x55beed294280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f1392ffd640 0 client.17064 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13bbc41740 1 -- 172.21.15.33:0/780771863 shutdown_connections 2024-09-16T00:26:23.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13bbc41740 1 -- 172.21.15.33:0/780771863 wait complete. 2024-09-16T00:26:23.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13bbc41740 20 client.17064 trim_cache size 0 max 16384 2024-09-16T00:26:23.116 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:26:23.116 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:26:23.116 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:26:23.116 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:26:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13b9868640 1 -- 172.21.15.33:0/3229783408 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f139c0a3240 msgr2=0x7f139c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13b9868640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f139c0a3240 0x7f139c0a5700 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f13ac006010 tx=0x7f13ac0095c0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13b9868640 1 -- 172.21.15.33:0/3229783408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13b4100a80 msgr2=0x7f13b40fcd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13b9868640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13b4100a80 0x7f13b40fcd30 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f13a400c890 tx=0x7f13a400cc90 comp rx=0 tx=0).stop 2024-09-16T00:26:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13b9868640 1 -- 172.21.15.33:0/3229783408 shutdown_connections 2024-09-16T00:26:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13b9868640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f139c0a3240 0x7f139c0a5700 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13b9868640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13b4106a80 0x7f13b40fd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13b9868640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b4101420 0x7f13b40fd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13b9868640 1 --2- 172.21.15.33:0/3229783408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13b4100a80 0x7f13b40fcd30 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.100+0000 7f13b9868640 1 -- 172.21.15.33:0/3229783408 >> 172.21.15.33:0/3229783408 conn(0x7f13b40fc7a0 msgr2=0x7f13b406a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.101+0000 7f13b9868640 1 -- 172.21.15.33:0/3229783408 shutdown_connections 2024-09-16T00:26:23.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:23.101+0000 7f13b9868640 1 -- 172.21.15.33:0/3229783408 wait complete. 2024-09-16T00:26:23.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.31 MB/sec execute 331 sec latency 11938.280 ms 2024-09-16T00:26:23.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:23 smithi060 ceph-mon[35185]: pgmap v627: 129 pgs: 129 active+clean; 283 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 172 op/s 2024-09-16T00:26:23.731 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:23 smithi033 ceph-mon[31243]: pgmap v627: 129 pgs: 129 active+clean; 283 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 172 op/s 2024-09-16T00:26:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:23 smithi083 ceph-mon[35242]: pgmap v627: 129 pgs: 129 active+clean; 283 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 172 op/s 2024-09-16T00:26:24.156 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:26:24.157 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:26:24.421 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:24.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.28 MB/sec execute 332 sec latency 12938.327 ms 2024-09-16T00:26:24.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.897+0000 7f373ef55640 1 -- 172.21.15.33:0/3525450616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3738106a60 msgr2=0x7f3738106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:24.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.897+0000 7f373ef55640 1 --2- 172.21.15.33:0/3525450616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3738106a60 0x7f3738106e40 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3728009930 tx=0x7f372802a420 comp rx=0 tx=0).stop 2024-09-16T00:26:24.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.897+0000 7f373ef55640 1 -- 172.21.15.33:0/3525450616 shutdown_connections 2024-09-16T00:26:24.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.897+0000 7f373ef55640 1 --2- 172.21.15.33:0/3525450616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3738101400 0x7f373810d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.897+0000 7f373ef55640 1 --2- 172.21.15.33:0/3525450616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3738100a60 0x7f3738100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.897+0000 7f373ef55640 1 --2- 172.21.15.33:0/3525450616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3738106a60 0x7f3738106e40 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.897+0000 7f373ef55640 1 -- 172.21.15.33:0/3525450616 >> 172.21.15.33:0/3525450616 conn(0x7f37380fc780 msgr2=0x7f37380feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.897+0000 7f373ef55640 1 -- 172.21.15.33:0/3525450616 shutdown_connections 2024-09-16T00:26:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.897+0000 7f373ef55640 1 -- 172.21.15.33:0/3525450616 wait complete. 2024-09-16T00:26:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.897+0000 7f373ef55640 1 Processor -- start 2024-09-16T00:26:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.897+0000 7f373ef55640 1 -- start start 2024-09-16T00:26:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f373ef55640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3738100a60 0x7f3738102630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f373ef55640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3738101400 0x7f3738102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f373ef55640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3738106a60 0x7f37381030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f373ef55640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3738110e60 con 0x7f3738101400 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f373ef55640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3738110ce0 con 0x7f3738106a60 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f373ef55640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3738110fe0 con 0x7f3738100a60 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f3737fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3738101400 0x7f3738102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f3737fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3738101400 0x7f3738102b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43776/0 (socket says 172.21.15.33:43776) 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f3737fff640 1 -- 172.21.15.33:0/545065186 learned_addr learned my addr 172.21.15.33:0/545065186 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f373d4f2640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3738106a60 0x7f37381030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f373ccf1640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3738100a60 0x7f3738102630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f3737fff640 1 -- 172.21.15.33:0/545065186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3738100a60 msgr2=0x7f3738102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f3737fff640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3738100a60 0x7f3738102630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f3737fff640 1 -- 172.21.15.33:0/545065186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3738106a60 msgr2=0x7f37381030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f3737fff640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3738106a60 0x7f37381030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f3737fff640 1 -- 172.21.15.33:0/545065186 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37381a68f0 con 0x7f3738101400 2024-09-16T00:26:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f373ccf1640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3738100a60 0x7f3738102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:26:24.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f3737fff640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3738101400 0x7f3738102b70 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f372c00c890 tx=0x7f372c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:24.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.898+0000 7f3735ffb640 1 -- 172.21.15.33:0/545065186 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c004220 con 0x7f3738101400 2024-09-16T00:26:24.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.899+0000 7f373ef55640 1 -- 172.21.15.33:0/545065186 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37381a6be0 con 0x7f3738101400 2024-09-16T00:26:24.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.899+0000 7f3735ffb640 1 -- 172.21.15.33:0/545065186 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f372c0043c0 con 0x7f3738101400 2024-09-16T00:26:24.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.899+0000 7f3735ffb640 1 -- 172.21.15.33:0/545065186 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c004a00 con 0x7f3738101400 2024-09-16T00:26:24.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.899+0000 7f373ef55640 1 -- 172.21.15.33:0/545065186 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37381a7170 con 0x7f3738101400 2024-09-16T00:26:24.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.900+0000 7f373ef55640 1 -- 172.21.15.33:0/545065186 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3738100ec0 con 0x7f3738101400 2024-09-16T00:26:24.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.901+0000 7f3735ffb640 1 -- 172.21.15.33:0/545065186 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f372c004c60 con 0x7f3738101400 2024-09-16T00:26:24.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.902+0000 7f3735ffb640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37200a3240 0x7f37200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:24.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.902+0000 7f3735ffb640 1 -- 172.21.15.33:0/545065186 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f372c0cfac0 con 0x7f3738101400 2024-09-16T00:26:24.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.902+0000 7f373ccf1640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37200a3240 0x7f37200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:24.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.903+0000 7f373ccf1640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37200a3240 0x7f37200a5700 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3728000c00 tx=0x7f3728005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:24.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:24.906+0000 7f3735ffb640 1 -- 172.21.15.33:0/545065186 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f372c014030 con 0x7f3738101400 2024-09-16T00:26:25.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.119+0000 7f373ef55640 1 -- 172.21.15.33:0/545065186 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37381a73a0 con 0x7f3738101400 2024-09-16T00:26:25.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.120+0000 7f3735ffb640 1 -- 172.21.15.33:0/545065186 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f372c09c360 con 0x7f3738101400 2024-09-16T00:26:25.120 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:26:25.120 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:26:25.120 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:26:25.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.122+0000 7f373ef55640 1 -- 172.21.15.33:0/545065186 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37200a3240 msgr2=0x7f37200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:25.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.122+0000 7f373ef55640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37200a3240 0x7f37200a5700 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3728000c00 tx=0x7f3728005a20 comp rx=0 tx=0).stop 2024-09-16T00:26:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.122+0000 7f373ef55640 1 -- 172.21.15.33:0/545065186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3738101400 msgr2=0x7f3738102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.122+0000 7f373ef55640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3738101400 0x7f3738102b70 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f372c00c890 tx=0x7f372c00cd50 comp rx=0 tx=0).stop 2024-09-16T00:26:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.122+0000 7f373ef55640 1 -- 172.21.15.33:0/545065186 shutdown_connections 2024-09-16T00:26:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.122+0000 7f373ef55640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f37200a3240 0x7f37200a5700 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.122+0000 7f373ef55640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3738106a60 0x7f37381030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.122+0000 7f373ef55640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3738101400 0x7f3738102b70 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.122+0000 7f373ef55640 1 --2- 172.21.15.33:0/545065186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3738100a60 0x7f3738102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.122+0000 7f373ef55640 1 -- 172.21.15.33:0/545065186 >> 172.21.15.33:0/545065186 conn(0x7f37380fc780 msgr2=0x7f373806a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.122+0000 7f373ef55640 1 -- 172.21.15.33:0/545065186 shutdown_connections 2024-09-16T00:26:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:25.122+0000 7f373ef55640 1 -- 172.21.15.33:0/545065186 wait complete. 2024-09-16T00:26:25.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.25 MB/sec execute 333 sec latency 13938.395 ms 2024-09-16T00:26:25.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:25 smithi060 ceph-mon[35185]: pgmap v628: 129 pgs: 129 active+clean; 282 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 2.6 MiB/s wr, 138 op/s 2024-09-16T00:26:25.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:25 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/545065186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:25.795 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:25 smithi033 ceph-mon[31243]: pgmap v628: 129 pgs: 129 active+clean; 282 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 2.6 MiB/s wr, 138 op/s 2024-09-16T00:26:25.795 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:25 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/545065186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:25 smithi083 ceph-mon[35242]: pgmap v628: 129 pgs: 129 active+clean; 282 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 2.6 MiB/s wr, 138 op/s 2024-09-16T00:26:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:25 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/545065186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:26.258 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:26:26.518 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:26.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.22 MB/sec execute 334 sec latency 14938.490 ms 2024-09-16T00:26:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.021+0000 7f361b03c640 1 -- 172.21.15.33:0/1484467826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3614100a50 msgr2=0x7f361410ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.021+0000 7f361b03c640 1 --2- 172.21.15.33:0/1484467826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3614100a50 0x7f361410ea40 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f360c009ca0 tx=0x7f360c02a8c0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.021+0000 7f361b03c640 1 -- 172.21.15.33:0/1484467826 shutdown_connections 2024-09-16T00:26:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.021+0000 7f361b03c640 1 --2- 172.21.15.33:0/1484467826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3614100a50 0x7f361410ea40 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.021+0000 7f361b03c640 1 --2- 172.21.15.33:0/1484467826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36141000b0 0x7f3614100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.021+0000 7f361b03c640 1 --2- 172.21.15.33:0/1484467826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3614106b80 0x7f3614106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.021+0000 7f361b03c640 1 -- 172.21.15.33:0/1484467826 >> 172.21.15.33:0/1484467826 conn(0x7f36140fbee0 msgr2=0x7f36140fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.021+0000 7f361b03c640 1 -- 172.21.15.33:0/1484467826 shutdown_connections 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.021+0000 7f361b03c640 1 -- 172.21.15.33:0/1484467826 wait complete. 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f361b03c640 1 Processor -- start 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f361b03c640 1 -- start start 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f361b03c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36141000b0 0x7f3614101290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f361b03c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3614100a50 0x7f36141017d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f361b03c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3614106b80 0x7f3614101d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f361b03c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3614112070 con 0x7f3614106b80 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f361b03c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3614111ef0 con 0x7f3614100a50 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f361b03c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f36141121f0 con 0x7f36141000b0 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f36195d9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3614106b80 0x7f3614101d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f36195d9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3614106b80 0x7f3614101d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43784/0 (socket says 172.21.15.33:43784) 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f36195d9640 1 -- 172.21.15.33:0/3572668200 learned_addr learned my addr 172.21.15.33:0/3572668200 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f3618dd8640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36141000b0 0x7f3614101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f3613fff640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3614100a50 0x7f36141017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f36195d9640 1 -- 172.21.15.33:0/3572668200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36141000b0 msgr2=0x7f3614101290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f36195d9640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36141000b0 0x7f3614101290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f36195d9640 1 -- 172.21.15.33:0/3572668200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3614100a50 msgr2=0x7f36141017d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f36195d9640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3614100a50 0x7f36141017d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.022+0000 7f36195d9640 1 -- 172.21.15.33:0/3572668200 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36141a68f0 con 0x7f3614106b80 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.023+0000 7f36195d9640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3614106b80 0x7f3614101d10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f360c006010 tx=0x7f360c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.023+0000 7f3611ffb640 1 -- 172.21.15.33:0/3572668200 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c031650 con 0x7f3614106b80 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.023+0000 7f3611ffb640 1 -- 172.21.15.33:0/3572668200 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f360c0317f0 con 0x7f3614106b80 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.023+0000 7f361b03c640 1 -- 172.21.15.33:0/3572668200 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36141a6b80 con 0x7f3614106b80 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.023+0000 7f3611ffb640 1 -- 172.21.15.33:0/3572668200 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c0309e0 con 0x7f3614106b80 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.023+0000 7f361b03c640 1 -- 172.21.15.33:0/3572668200 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36141a70c0 con 0x7f3614106b80 2024-09-16T00:26:27.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.025+0000 7f3611ffb640 1 -- 172.21.15.33:0/3572668200 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f360c030c20 con 0x7f3614106b80 2024-09-16T00:26:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f361d415740 1 Processor -- start 2024-09-16T00:26:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f361d415740 1 -- start start 2024-09-16T00:26:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f361d415740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dcbb40 0x55dc92dca090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f361d415740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dca5d0 0x55dc92dcaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f361d415740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dd5370 0x55dc92dd7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f361d415740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dc92d302f0 con 0x55dc92dcbb40 2024-09-16T00:26:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f361d415740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55dc92cd7cf0 con 0x55dc92dd5370 2024-09-16T00:26:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f361d415740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dc92d2c2b0 con 0x55dc92dca5d0 2024-09-16T00:26:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f3613fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dcbb40 0x55dc92dca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f3613fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dcbb40 0x55dc92dca090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43790/0 (socket says 172.21.15.33:43790) 2024-09-16T00:26:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f3613fff640 1 -- 172.21.15.33:0/2301677671 learned_addr learned my addr 172.21.15.33:0/2301677671 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f3611ffb640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35fc0a3240 0x7f35fc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f3618dd8640 1 --2- 172.21.15.33:0/2301677671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dca5d0 0x55dc92dcaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f36195d9640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35fc0a3240 0x7f35fc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f3613fff640 1 -- 172.21.15.33:0/2301677671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dca5d0 msgr2=0x55dc92dcaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f3613fff640 1 --2- 172.21.15.33:0/2301677671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dca5d0 0x55dc92dcaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f3613fff640 1 -- 172.21.15.33:0/2301677671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dd5370 msgr2=0x55dc92dd7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f3613fff640 1 --2- 172.21.15.33:0/2301677671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dd5370 0x55dc92dd7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f3613fff640 1 -- 172.21.15.33:0/2301677671 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc92cbc3a0 con 0x55dc92dcbb40 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.026+0000 7f3611ffb640 1 -- 172.21.15.33:0/3572668200 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f360c0f1220 con 0x7f3614106b80 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.027+0000 7f3613fff640 1 --2- 172.21.15.33:0/2301677671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dcbb40 0x55dc92dca090 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f360800b6c0 tx=0x7f36080127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3043fd3dbe3e6115 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.027+0000 7f35faffd640 1 -- 172.21.15.33:0/2301677671 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360801b470 con 0x55dc92dcbb40 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.027+0000 7f35faffd640 1 -- 172.21.15.33:0/2301677671 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3608016070 con 0x55dc92dcbb40 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.027+0000 7f36195d9640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35fc0a3240 0x7f35fc0a5700 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f360c045000 tx=0x7f360c090300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.027+0000 7f35faffd640 1 -- 172.21.15.33:0/2301677671 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608010830 con 0x55dc92dcbb40 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.027+0000 7f361d415740 1 -- 172.21.15.33:0/2301677671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dcbb40 msgr2=0x55dc92dca090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.027+0000 7f361d415740 1 --2- 172.21.15.33:0/2301677671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dcbb40 0x55dc92dca090 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f360800b6c0 tx=0x7f36080127c0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 -- 172.21.15.33:0/2301677671 shutdown_connections 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 --2- 172.21.15.33:0/2301677671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dd5370 0x55dc92dd7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 --2- 172.21.15.33:0/2301677671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dca5d0 0x55dc92dcaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 --2- 172.21.15.33:0/2301677671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dcbb40 0x55dc92dca090 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 -- 172.21.15.33:0/2301677671 >> 172.21.15.33:0/2301677671 conn(0x55dc92d93310 msgr2=0x55dc92dc6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 -- 172.21.15.33:0/2301677671 shutdown_connections 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 -- 172.21.15.33:0/2301677671 wait complete. 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 -- start start 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 Processor -- start 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dca5d0 0x55dc92dd4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dcbb40 0x55dc92dd2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 0x55dc92dd25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dc92dd8070 con 0x55dc92dd5370 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55dc92ceb060 con 0x55dc92dcbb40 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dc92dd81f0 con 0x55dc92dca5d0 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.028+0000 7f361d415740 10 client.0 _ll_register_callbacks cb 0x55dc92a3e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f3613fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 0x55dc92dd25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f3613fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 0x55dc92dd25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43800/0 (socket says 172.21.15.33:43800) 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f3613fff640 1 -- 172.21.15.33:0/3974968305 learned_addr learned my addr 172.21.15.33:0/3974968305 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f3618dd8640 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dcbb40 0x55dc92dd2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f3613fff640 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dca5d0 0x55dc92dd4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f3618dd8640 1 -- 172.21.15.33:0/3974968305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dca5d0 msgr2=0x55dc92dd4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f3618dd8640 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dca5d0 0x55dc92dd4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f3618dd8640 1 -- 172.21.15.33:0/3974968305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 msgr2=0x55dc92dd25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:27.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f3618dd8640 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 0x55dc92dd25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f3618dd8640 1 -- 172.21.15.33:0/3974968305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc92aa13c0 con 0x55dc92dcbb40 2024-09-16T00:26:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f3613fff640 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 0x55dc92dd25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f3618dd8640 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dcbb40 0x55dc92dd2000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f360400b9c0 tx=0x7f360400be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e09fc0202d81a1cb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f35f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:26:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f35f3fff640 1 -- 172.21.15.33:0/3974968305 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3604003e20 con 0x55dc92dcbb40 2024-09-16T00:26:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f35f3fff640 1 -- 172.21.15.33:0/3974968305 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3604005020 con 0x55dc92dcbb40 2024-09-16T00:26:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.029+0000 7f361d415740 1 -- 172.21.15.33:0/3974968305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3608016070 con 0x55dc92dcbb40 2024-09-16T00:26:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.030+0000 7f35f3fff640 1 -- 172.21.15.33:0/3974968305 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3604024050 con 0x55dc92dcbb40 2024-09-16T00:26:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.030+0000 7f361d415740 10 client.26561 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.030+0000 7f361d415740 1 -- 172.21.15.33:0/3974968305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dc92de3b80 con 0x55dc92dcbb40 2024-09-16T00:26:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.030+0000 7f35f3fff640 1 -- 172.21.15.33:0/3974968305 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3608016070 con 0x55dc92dcbb40 2024-09-16T00:26:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.030+0000 7f35f3fff640 1 -- 172.21.15.33:0/3974968305 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f360401b470 con 0x55dc92dcbb40 2024-09-16T00:26:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.030+0000 7f361d415740 10 client.26561 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.030+0000 7f361d415740 10 client.26561 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:26:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.030+0000 7f361d415740 20 client.26561 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.030+0000 7f361d415740 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55dc92de5090 0x55dc92de74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.030+0000 7f361d415740 4 client.26561 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:26:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.030+0000 7f361d415740 1 -- 172.21.15.33:0/3974968305 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc92d3d860 con 0x55dc92de5090 2024-09-16T00:26:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.030+0000 7f3613fff640 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55dc92de5090 0x55dc92de74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.031+0000 7f35f3fff640 10 client.26561 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:26:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.031+0000 7f3613fff640 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55dc92de5090 0x55dc92de74b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=131d0546e01730c8 server_cookie=4e4df41e06754197 in_seq=0 out_seq=0 2024-09-16T00:26:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f35f3fff640 1 -- 172.21.15.33:0/3974968305 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dc92d3d860 con 0x55dc92de5090 2024-09-16T00:26:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f35f3fff640 10 client.26561 handle_command_reply: tid=0 2024-09-16T00:26:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f361d415740 1 client.26561 shutdown 2024-09-16T00:26:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f361d415740 1 -- 172.21.15.33:0/3974968305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dcbb40 msgr2=0x55dc92dd2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f361d415740 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dcbb40 0x55dc92dd2000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f360400b9c0 tx=0x7f360400be80 comp rx=0 tx=0).stop 2024-09-16T00:26:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f361d415740 1 -- 172.21.15.33:0/3974968305 shutdown_connections 2024-09-16T00:26:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f361d415740 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55dc92de5090 0x55dc92de74b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f361d415740 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 0x55dc92dd25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f35f3fff640 0 client.26561 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:26:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f361d415740 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dcbb40 0x55dc92dd2000 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f361d415740 1 --2- 172.21.15.33:0/3974968305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dca5d0 0x55dc92dd4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f361d415740 1 -- 172.21.15.33:0/3974968305 >> 172.21.15.33:0/3974968305 conn(0x55dc92d93310 msgr2=0x55dc92dcc930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f361d415740 1 -- 172.21.15.33:0/3974968305 shutdown_connections 2024-09-16T00:26:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f361d415740 1 -- 172.21.15.33:0/3974968305 wait complete. 2024-09-16T00:26:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.032+0000 7f361d415740 20 client.26561 trim_cache size 0 max 16384 2024-09-16T00:26:27.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.045+0000 7f361d415740 1 Processor -- start 2024-09-16T00:26:27.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.045+0000 7f361d415740 1 -- start start 2024-09-16T00:26:27.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f361d415740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dca5d0 0x55dc92dc6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f361d415740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dcbb40 0x55dc92de1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f361d415740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 0x55dc92dd18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f361d415740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36141b3b60 con 0x55dc92dd5370 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f361d415740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f36141b1430 con 0x55dc92dca5d0 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f361d415740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f36141af280 con 0x55dc92dcbb40 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f3613fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 0x55dc92dd18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f3613fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 0x55dc92dd18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43816/0 (socket says 172.21.15.33:43816) 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f3613fff640 1 -- 172.21.15.33:0/1604827939 learned_addr learned my addr 172.21.15.33:0/1604827939 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f3613fff640 1 --2- 172.21.15.33:0/1604827939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dca5d0 0x55dc92dc6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f3618dd8640 1 --2- 172.21.15.33:0/1604827939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dcbb40 0x55dc92de1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f3613fff640 1 -- 172.21.15.33:0/1604827939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dcbb40 msgr2=0x55dc92de1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f3613fff640 1 --2- 172.21.15.33:0/1604827939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dcbb40 0x55dc92de1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f3613fff640 1 -- 172.21.15.33:0/1604827939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dca5d0 msgr2=0x55dc92dc6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f3613fff640 1 --2- 172.21.15.33:0/1604827939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dca5d0 0x55dc92dc6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f3613fff640 1 -- 172.21.15.33:0/1604827939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc92dd0fb0 con 0x55dc92dd5370 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.046+0000 7f3613fff640 1 --2- 172.21.15.33:0/1604827939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 0x55dc92dd18c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f3608005920 tx=0x7f360800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d419584d478698f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.047+0000 7f35fa7fc640 1 -- 172.21.15.33:0/1604827939 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608005150 con 0x55dc92dd5370 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.047+0000 7f35fa7fc640 1 -- 172.21.15.33:0/1604827939 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3608031070 con 0x55dc92dd5370 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.047+0000 7f35fa7fc640 1 -- 172.21.15.33:0/1604827939 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36080048c0 con 0x55dc92dd5370 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.047+0000 7f361d415740 1 -- 172.21.15.33:0/1604827939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 msgr2=0x55dc92dd18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.047+0000 7f361d415740 1 --2- 172.21.15.33:0/1604827939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 0x55dc92dd18c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f3608005920 tx=0x7f360800b650 comp rx=0 tx=0).stop 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.047+0000 7f361d415740 1 -- 172.21.15.33:0/1604827939 shutdown_connections 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.047+0000 7f361d415740 1 --2- 172.21.15.33:0/1604827939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dd5370 0x55dc92dd18c0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.047+0000 7f361d415740 1 --2- 172.21.15.33:0/1604827939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dcbb40 0x55dc92de1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.047+0000 7f361d415740 1 --2- 172.21.15.33:0/1604827939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dca5d0 0x55dc92dc6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.047+0000 7f361d415740 1 -- 172.21.15.33:0/1604827939 >> 172.21.15.33:0/1604827939 conn(0x55dc92d93310 msgr2=0x55dc92dc6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.047+0000 7f361d415740 1 -- 172.21.15.33:0/1604827939 shutdown_connections 2024-09-16T00:26:27.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f361d415740 1 -- 172.21.15.33:0/1604827939 wait complete. 2024-09-16T00:26:27.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f361d415740 1 -- start start 2024-09-16T00:26:27.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f361d415740 1 Processor -- start 2024-09-16T00:26:27.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f361d415740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dca5d0 0x55dc92ddd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f361d415740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dcbb40 0x55dc92dddde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f361d415740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dd5370 0x55dc92dd8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f361d415740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36141af8f0 con 0x55dc92dca5d0 2024-09-16T00:26:27.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f361d415740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f36141b0d60 con 0x55dc92dd5370 2024-09-16T00:26:27.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f361d415740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f36141affc0 con 0x55dc92dcbb40 2024-09-16T00:26:27.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f361d415740 10 client.0 _ll_register_callbacks cb 0x7f3604004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:27.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f3613fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dca5d0 0x55dc92ddd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f3613fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dca5d0 0x55dc92ddd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43824/0 (socket says 172.21.15.33:43824) 2024-09-16T00:26:27.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.048+0000 7f3613fff640 1 -- 172.21.15.33:0/247472224 learned_addr learned my addr 172.21.15.33:0/247472224 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f3613fff640 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dd5370 0x55dc92dd8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f3618dd8640 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dcbb40 0x55dc92dddde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f3613fff640 1 -- 172.21.15.33:0/247472224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dcbb40 msgr2=0x55dc92dddde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f3613fff640 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dcbb40 0x55dc92dddde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f3613fff640 1 -- 172.21.15.33:0/247472224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dd5370 msgr2=0x55dc92dd8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f3613fff640 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dd5370 0x55dc92dd8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f3613fff640 1 -- 172.21.15.33:0/247472224 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc92dd9650 con 0x55dc92dca5d0 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f3618dd8640 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dcbb40 0x55dc92dddde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f3613fff640 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dca5d0 0x55dc92ddd8a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f360800bd10 tx=0x7f3608012cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f63287ab061baf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f35f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f35f3fff640 1 -- 172.21.15.33:0/247472224 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360801b850 con 0x55dc92dca5d0 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f35f3fff640 1 -- 172.21.15.33:0/247472224 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3608005150 con 0x55dc92dca5d0 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f361d415740 1 -- 172.21.15.33:0/247472224 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3608031070 con 0x55dc92dca5d0 2024-09-16T00:26:27.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f35f3fff640 1 -- 172.21.15.33:0/247472224 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3608044050 con 0x55dc92dca5d0 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f361d415740 10 client.17088 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.049+0000 7f361d415740 1 -- 172.21.15.33:0/247472224 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dc92e4f9c0 con 0x55dc92dca5d0 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.050+0000 7f35f3fff640 1 -- 172.21.15.33:0/247472224 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3608031070 con 0x55dc92dca5d0 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.050+0000 7f35f3fff640 1 -- 172.21.15.33:0/247472224 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f360802d3d0 con 0x55dc92dca5d0 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.050+0000 7f361d415740 10 client.17088 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.050+0000 7f361d415740 10 client.17088 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.050+0000 7f361d415740 20 client.17088 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.050+0000 7f361d415740 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55dc92e4fef0 0x55dc92e522e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.050+0000 7f361d415740 4 client.17088 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.050+0000 7f361d415740 1 -- 172.21.15.33:0/247472224 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dc92d37d70 con 0x55dc92e4fef0 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.050+0000 7f3618dd8640 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55dc92e4fef0 0x55dc92e522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.050+0000 7f3618dd8640 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55dc92e4fef0 0x55dc92e522e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4f4abb23a95e9ab1 server_cookie=577ec8dcab70873b in_seq=0 out_seq=0 2024-09-16T00:26:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.050+0000 7f35f3fff640 10 client.17088 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:26:27.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f35f3fff640 1 -- 172.21.15.33:0/247472224 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dc92d37d70 con 0x55dc92e4fef0 2024-09-16T00:26:27.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f35f3fff640 10 client.17088 handle_command_reply: tid=0 2024-09-16T00:26:27.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f361d415740 1 client.17088 shutdown 2024-09-16T00:26:27.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f361d415740 1 -- 172.21.15.33:0/247472224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dca5d0 msgr2=0x55dc92ddd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:27.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f361d415740 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dca5d0 0x55dc92ddd8a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f360800bd10 tx=0x7f3608012cc0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f361d415740 1 -- 172.21.15.33:0/247472224 shutdown_connections 2024-09-16T00:26:27.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f361d415740 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55dc92e4fef0 0x55dc92e522e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f361d415740 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dc92dd5370 0x55dc92dd8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f361d415740 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc92dcbb40 0x55dc92dddde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f361d415740 1 --2- 172.21.15.33:0/247472224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc92dca5d0 0x55dc92ddd8a0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f35f3fff640 0 client.17088 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:26:27.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f361d415740 1 -- 172.21.15.33:0/247472224 >> 172.21.15.33:0/247472224 conn(0x55dc92d93310 msgr2=0x55dc92dd4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:27.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f361d415740 1 -- 172.21.15.33:0/247472224 shutdown_connections 2024-09-16T00:26:27.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f361d415740 1 -- 172.21.15.33:0/247472224 wait complete. 2024-09-16T00:26:27.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.051+0000 7f361d415740 20 client.17088 trim_cache size 0 max 16384 2024-09-16T00:26:27.067 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:26:27.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.052+0000 7f361b03c640 1 -- 172.21.15.33:0/3572668200 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35fc0a3240 msgr2=0x7f35fc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:27.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.052+0000 7f361b03c640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35fc0a3240 0x7f35fc0a5700 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f360c045000 tx=0x7f360c090300 comp rx=0 tx=0).stop 2024-09-16T00:26:27.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.052+0000 7f361b03c640 1 -- 172.21.15.33:0/3572668200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3614106b80 msgr2=0x7f3614101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:27.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.052+0000 7f361b03c640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3614106b80 0x7f3614101d10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f360c006010 tx=0x7f360c003f10 comp rx=0 tx=0).stop 2024-09-16T00:26:27.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.052+0000 7f361b03c640 1 -- 172.21.15.33:0/3572668200 shutdown_connections 2024-09-16T00:26:27.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.052+0000 7f361b03c640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35fc0a3240 0x7f35fc0a5700 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.052+0000 7f361b03c640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3614106b80 0x7f3614101d10 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.052+0000 7f361b03c640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3614100a50 0x7f36141017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.052+0000 7f361b03c640 1 --2- 172.21.15.33:0/3572668200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36141000b0 0x7f3614101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:27.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.052+0000 7f361b03c640 1 -- 172.21.15.33:0/3572668200 >> 172.21.15.33:0/3572668200 conn(0x7f36140fbee0 msgr2=0x7f36140fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:27.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.052+0000 7f361b03c640 1 -- 172.21.15.33:0/3572668200 shutdown_connections 2024-09-16T00:26:27.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:27.052+0000 7f361b03c640 1 -- 172.21.15.33:0/3572668200 wait complete. 2024-09-16T00:26:27.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.19 MB/sec execute 335 sec latency 15938.583 ms 2024-09-16T00:26:27.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:27 smithi083 ceph-mon[35242]: pgmap v629: 129 pgs: 129 active+clean; 282 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 2.4 MiB/s wr, 96 op/s 2024-09-16T00:26:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:27 smithi060 ceph-mon[35185]: pgmap v629: 129 pgs: 129 active+clean; 282 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 2.4 MiB/s wr, 96 op/s 2024-09-16T00:26:27.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:27 smithi033 ceph-mon[31243]: pgmap v629: 129 pgs: 129 active+clean; 282 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 2.4 MiB/s wr, 96 op/s 2024-09-16T00:26:27.888 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:26:28.145 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.632+0000 7f518c191640 1 -- 172.21.15.33:0/4278424745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5184106f10 msgr2=0x7f518410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.632+0000 7f518c191640 1 --2- 172.21.15.33:0/4278424745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5184106f10 0x7f518410ea90 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f517c009ca0 tx=0x7f517c02a8c0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.632+0000 7f518c191640 1 -- 172.21.15.33:0/4278424745 shutdown_connections 2024-09-16T00:26:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.632+0000 7f518c191640 1 --2- 172.21.15.33:0/4278424745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5184106f10 0x7f518410ea90 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.632+0000 7f518c191640 1 --2- 172.21.15.33:0/4278424745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5184102790 0x7f51841067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.632+0000 7f518c191640 1 --2- 172.21.15.33:0/4278424745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5184101de0 0x7f51841021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.632+0000 7f518c191640 1 -- 172.21.15.33:0/4278424745 >> 172.21.15.33:0/4278424745 conn(0x7f51840fd8c0 msgr2=0x7f51840ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.632+0000 7f518c191640 1 -- 172.21.15.33:0/4278424745 shutdown_connections 2024-09-16T00:26:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.632+0000 7f518c191640 1 -- 172.21.15.33:0/4278424745 wait complete. 2024-09-16T00:26:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f518c191640 1 Processor -- start 2024-09-16T00:26:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f518c191640 1 -- start start 2024-09-16T00:26:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f518c191640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5184101de0 0x7f51840744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f518c191640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5184102790 0x7f5184074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f518c191640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5184106f10 0x7f5184074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f518c191640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5184111f90 con 0x7f5184106f10 2024-09-16T00:26:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f518c191640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5184111e10 con 0x7f5184102790 2024-09-16T00:26:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f518c191640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5184112110 con 0x7f5184101de0 2024-09-16T00:26:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f518a72e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5184106f10 0x7f5184074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f518a72e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5184106f10 0x7f5184074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43834/0 (socket says 172.21.15.33:43834) 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f518a72e640 1 -- 172.21.15.33:0/1022863024 learned_addr learned my addr 172.21.15.33:0/1022863024 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f5189f2d640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5184101de0 0x7f51840744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.633+0000 7f518972c640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5184102790 0x7f5184074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.634+0000 7f518a72e640 1 -- 172.21.15.33:0/1022863024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5184101de0 msgr2=0x7f51840744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.634+0000 7f518a72e640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5184101de0 0x7f51840744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.634+0000 7f518a72e640 1 -- 172.21.15.33:0/1022863024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5184102790 msgr2=0x7f5184074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.634+0000 7f518a72e640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5184102790 0x7f5184074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.634+0000 7f518a72e640 1 -- 172.21.15.33:0/1022863024 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51841a6930 con 0x7f5184106f10 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.634+0000 7f518a72e640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5184106f10 0x7f5184074f70 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f517c031a60 tx=0x7f517c031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.634+0000 7f517affd640 1 -- 172.21.15.33:0/1022863024 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f517c003b70 con 0x7f5184106f10 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.634+0000 7f517affd640 1 -- 172.21.15.33:0/1022863024 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f517c003d10 con 0x7f5184106f10 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.634+0000 7f518c191640 1 -- 172.21.15.33:0/1022863024 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51841a6bc0 con 0x7f5184106f10 2024-09-16T00:26:28.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.634+0000 7f517affd640 1 -- 172.21.15.33:0/1022863024 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f517c004770 con 0x7f5184106f10 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.634+0000 7f518c191640 1 -- 172.21.15.33:0/1022863024 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51841a7150 con 0x7f5184106f10 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.636+0000 7f517affd640 1 -- 172.21.15.33:0/1022863024 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f517c02f790 con 0x7f5184106f10 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518e56a740 1 Processor -- start 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518e56a740 1 -- start start 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518e56a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5ccb40 0x561c3e5cb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518e56a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 0x561c3e5cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518e56a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5d8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518e56a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561c3e5312f0 con 0x561c3e5d6370 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518e56a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561c3e4d8cf0 con 0x561c3e5ccb40 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518e56a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561c3e52d2b0 con 0x561c3e5cb5d0 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f517affd640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f51700a32c0 0x7f51700a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518972c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5d8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518972c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5d8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43842/0 (socket says 172.21.15.33:43842) 2024-09-16T00:26:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518972c640 1 -- 172.21.15.33:0/1542182522 learned_addr learned my addr 172.21.15.33:0/1542182522 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518972c640 1 --2- 172.21.15.33:0/1542182522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5ccb40 0x561c3e5cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f5189f2d640 1 --2- 172.21.15.33:0/1542182522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 0x561c3e5cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.637+0000 7f518a72e640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f51700a32c0 0x7f51700a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.638+0000 7f517affd640 1 -- 172.21.15.33:0/1022863024 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f517c0f0fe0 con 0x7f5184106f10 2024-09-16T00:26:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.638+0000 7f518972c640 1 -- 172.21.15.33:0/1542182522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 msgr2=0x561c3e5cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.638+0000 7f518972c640 1 --2- 172.21.15.33:0/1542182522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 0x561c3e5cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.638+0000 7f518972c640 1 -- 172.21.15.33:0/1542182522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 msgr2=0x561c3e5d8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.638+0000 7f518972c640 1 --2- 172.21.15.33:0/1542182522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5d8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.638+0000 7f518972c640 1 -- 172.21.15.33:0/1542182522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c3e4bd3a0 con 0x561c3e5ccb40 2024-09-16T00:26:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.638+0000 7f518972c640 1 --2- 172.21.15.33:0/1542182522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5ccb40 0x561c3e5cb090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5174006aa0 tx=0x7f5174009110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b2f3740e4ee961a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:28.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.638+0000 7f518a72e640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f51700a32c0 0x7f51700a5780 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f517c04c000 tx=0x7f517c08cef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.638+0000 7f516ffff640 1 -- 172.21.15.33:0/1542182522 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51740136e0 con 0x561c3e5ccb40 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.638+0000 7f516ffff640 1 -- 172.21.15.33:0/1542182522 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f517401d070 con 0x561c3e5ccb40 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.638+0000 7f516ffff640 1 -- 172.21.15.33:0/1542182522 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174018ab0 con 0x561c3e5ccb40 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.639+0000 7f518e56a740 1 -- 172.21.15.33:0/1542182522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5ccb40 msgr2=0x561c3e5cb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.639+0000 7f518e56a740 1 --2- 172.21.15.33:0/1542182522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5ccb40 0x561c3e5cb090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5174006aa0 tx=0x7f5174009110 comp rx=0 tx=0).stop 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.639+0000 7f518e56a740 1 -- 172.21.15.33:0/1542182522 shutdown_connections 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.639+0000 7f518e56a740 1 --2- 172.21.15.33:0/1542182522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5d8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.639+0000 7f518e56a740 1 --2- 172.21.15.33:0/1542182522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 0x561c3e5cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.639+0000 7f518e56a740 1 --2- 172.21.15.33:0/1542182522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5ccb40 0x561c3e5cb090 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.639+0000 7f518e56a740 1 -- 172.21.15.33:0/1542182522 >> 172.21.15.33:0/1542182522 conn(0x561c3e594310 msgr2=0x561c3e5c7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.639+0000 7f518e56a740 1 -- 172.21.15.33:0/1542182522 shutdown_connections 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.639+0000 7f518e56a740 1 -- 172.21.15.33:0/1542182522 wait complete. 2024-09-16T00:26:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.639+0000 7f518e56a740 1 -- start start 2024-09-16T00:26:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.639+0000 7f518e56a740 1 Processor -- start 2024-09-16T00:26:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518e56a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 0x561c3e5d5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518e56a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5ccb40 0x561c3e5d3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518e56a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5d3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518e56a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561c3e5d9010 con 0x561c3e5d6370 2024-09-16T00:26:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518e56a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561c3e4ec060 con 0x561c3e5ccb40 2024-09-16T00:26:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518e56a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561c3e5d9190 con 0x561c3e5cb5d0 2024-09-16T00:26:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518e56a740 10 client.0 _ll_register_callbacks cb 0x561c3e23f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518972c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5d3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518972c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5d3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43844/0 (socket says 172.21.15.33:43844) 2024-09-16T00:26:28.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518972c640 1 -- 172.21.15.33:0/3953269173 learned_addr learned my addr 172.21.15.33:0/3953269173 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:28.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f5189f2d640 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5ccb40 0x561c3e5d3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518972c640 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 0x561c3e5d5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518972c640 1 -- 172.21.15.33:0/3953269173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 msgr2=0x561c3e5d5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518972c640 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 0x561c3e5d5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518972c640 1 -- 172.21.15.33:0/3953269173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5ccb40 msgr2=0x561c3e5d3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518972c640 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5ccb40 0x561c3e5d3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518972c640 1 -- 172.21.15.33:0/3953269173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c3e2a23c0 con 0x561c3e5d6370 2024-09-16T00:26:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f518972c640 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5d3600 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5174018750 tx=0x7f5174008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538d84ed91ef5174 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.640+0000 7f516d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:26:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f516d7fa640 1 -- 172.21.15.33:0/3953269173 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174019c80 con 0x561c3e5d6370 2024-09-16T00:26:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f518e56a740 1 -- 172.21.15.33:0/3953269173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f517401d070 con 0x561c3e5d6370 2024-09-16T00:26:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f516d7fa640 1 -- 172.21.15.33:0/3953269173 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f51740136e0 con 0x561c3e5d6370 2024-09-16T00:26:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f516d7fa640 1 -- 172.21.15.33:0/3953269173 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f51740138e0 con 0x561c3e5d6370 2024-09-16T00:26:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f518e56a740 10 client.17100 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f518e56a740 1 -- 172.21.15.33:0/3953269173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561c3e5e4ae0 con 0x561c3e5d6370 2024-09-16T00:26:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f516d7fa640 1 -- 172.21.15.33:0/3953269173 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f517401d070 con 0x561c3e5d6370 2024-09-16T00:26:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f516d7fa640 1 -- 172.21.15.33:0/3953269173 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f5174024970 con 0x561c3e5d6370 2024-09-16T00:26:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f518e56a740 10 client.17100 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f518e56a740 10 client.17100 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:26:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f518e56a740 20 client.17100 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f518e56a740 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561c3e5e5e30 0x561c3e5e8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f518e56a740 4 client.17100 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:26:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f518e56a740 1 -- 172.21.15.33:0/3953269173 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c3e53e860 con 0x561c3e5e5e30 2024-09-16T00:26:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.641+0000 7f5189f2d640 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561c3e5e5e30 0x561c3e5e8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.642+0000 7f5189f2d640 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561c3e5e5e30 0x561c3e5e8220 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3be52d2396593925 server_cookie=fb15f3a35008eb73 in_seq=0 out_seq=0 2024-09-16T00:26:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.642+0000 7f516d7fa640 10 client.17100 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:26:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.642+0000 7f516d7fa640 1 -- 172.21.15.33:0/3953269173 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561c3e53e860 con 0x561c3e5e5e30 2024-09-16T00:26:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.642+0000 7f516d7fa640 10 client.17100 handle_command_reply: tid=0 2024-09-16T00:26:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.642+0000 7f518e56a740 1 client.17100 shutdown 2024-09-16T00:26:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.643+0000 7f518e56a740 1 -- 172.21.15.33:0/3953269173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 msgr2=0x561c3e5d3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:28.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.643+0000 7f518e56a740 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5d3600 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5174018750 tx=0x7f5174008040 comp rx=0 tx=0).stop 2024-09-16T00:26:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.643+0000 7f518e56a740 1 -- 172.21.15.33:0/3953269173 shutdown_connections 2024-09-16T00:26:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.643+0000 7f518e56a740 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5d3600 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.643+0000 7f518e56a740 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561c3e5e5e30 0x561c3e5e8220 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.643+0000 7f518e56a740 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5ccb40 0x561c3e5d3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.643+0000 7f518e56a740 1 --2- 172.21.15.33:0/3953269173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 0x561c3e5d5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.643+0000 7f516d7fa640 0 client.17100 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:26:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.643+0000 7f518e56a740 1 -- 172.21.15.33:0/3953269173 >> 172.21.15.33:0/3953269173 conn(0x561c3e594310 msgr2=0x561c3e5cd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.643+0000 7f518e56a740 1 -- 172.21.15.33:0/3953269173 shutdown_connections 2024-09-16T00:26:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.643+0000 7f518e56a740 1 -- 172.21.15.33:0/3953269173 wait complete. 2024-09-16T00:26:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.643+0000 7f518e56a740 20 client.17100 trim_cache size 0 max 16384 2024-09-16T00:26:28.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f518e56a740 1 Processor -- start 2024-09-16T00:26:28.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f518e56a740 1 -- start start 2024-09-16T00:26:28.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f518e56a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5cb5d0 0x561c3e5e22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f518e56a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5ccb40 0x561c3e5d1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f518e56a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5ed850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f518e56a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51841b3bd0 con 0x561c3e5d6370 2024-09-16T00:26:28.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f518e56a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f51841b14a0 con 0x561c3e5cb5d0 2024-09-16T00:26:28.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f518e56a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f51841af2f0 con 0x561c3e5ccb40 2024-09-16T00:26:28.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f5189f2d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5ccb40 0x561c3e5d1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f518972c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5ed850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f5189f2d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5ccb40 0x561c3e5d1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:33424/0 (socket says 172.21.15.33:33424) 2024-09-16T00:26:28.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f5189f2d640 1 -- 172.21.15.33:0/260046709 learned_addr learned my addr 172.21.15.33:0/260046709 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:28.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.656+0000 7f518972c640 1 --2- 172.21.15.33:0/260046709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5cb5d0 0x561c3e5e22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.657+0000 7f518972c640 1 -- 172.21.15.33:0/260046709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5ccb40 msgr2=0x561c3e5d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:28.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.657+0000 7f518972c640 1 --2- 172.21.15.33:0/260046709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5ccb40 0x561c3e5d1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.657+0000 7f518972c640 1 -- 172.21.15.33:0/260046709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5cb5d0 msgr2=0x561c3e5e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:28.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.657+0000 7f518972c640 1 --2- 172.21.15.33:0/260046709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5cb5d0 0x561c3e5e22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.657+0000 7f518972c640 1 -- 172.21.15.33:0/260046709 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c3e5ce120 con 0x561c3e5d6370 2024-09-16T00:26:28.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.657+0000 7f5189f2d640 1 --2- 172.21.15.33:0/260046709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5ccb40 0x561c3e5d1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:28.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.657+0000 7f518972c640 1 --2- 172.21.15.33:0/260046709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5ed850 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5174008010 tx=0x7f5174000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bac3c9f42d099a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:28.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.657+0000 7f516f7fe640 1 -- 172.21.15.33:0/260046709 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174019c80 con 0x561c3e5d6370 2024-09-16T00:26:28.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.657+0000 7f516f7fe640 1 -- 172.21.15.33:0/260046709 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f517401d040 con 0x561c3e5d6370 2024-09-16T00:26:28.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.657+0000 7f516f7fe640 1 -- 172.21.15.33:0/260046709 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51740099b0 con 0x561c3e5d6370 2024-09-16T00:26:28.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.658+0000 7f518e56a740 1 -- 172.21.15.33:0/260046709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 msgr2=0x561c3e5ed850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:28.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.658+0000 7f518e56a740 1 --2- 172.21.15.33:0/260046709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5ed850 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5174008010 tx=0x7f5174000bc0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.658+0000 7f518e56a740 1 -- 172.21.15.33:0/260046709 shutdown_connections 2024-09-16T00:26:28.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.658+0000 7f518e56a740 1 --2- 172.21.15.33:0/260046709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5d6370 0x561c3e5ed850 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.658+0000 7f518e56a740 1 --2- 172.21.15.33:0/260046709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5ccb40 0x561c3e5d1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.658+0000 7f518e56a740 1 --2- 172.21.15.33:0/260046709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5cb5d0 0x561c3e5e22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.658+0000 7f518e56a740 1 -- 172.21.15.33:0/260046709 >> 172.21.15.33:0/260046709 conn(0x561c3e594310 msgr2=0x561c3e5d18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:28.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.658+0000 7f518e56a740 1 -- 172.21.15.33:0/260046709 shutdown_connections 2024-09-16T00:26:28.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.658+0000 7f518e56a740 1 -- 172.21.15.33:0/260046709 wait complete. 2024-09-16T00:26:28.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.658+0000 7f518e56a740 1 -- start start 2024-09-16T00:26:28.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.658+0000 7f518e56a740 1 Processor -- start 2024-09-16T00:26:28.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f518e56a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 0x561c3e5decc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f518e56a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5ccb40 0x561c3e5d9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f518e56a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5d6370 0x561c3e5da5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f518e56a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51841af960 con 0x561c3e5ccb40 2024-09-16T00:26:28.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f518e56a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f51841b0dd0 con 0x561c3e5d6370 2024-09-16T00:26:28.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f518e56a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f51841b0030 con 0x561c3e5cb5d0 2024-09-16T00:26:28.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f518e56a740 10 client.0 _ll_register_callbacks cb 0x7f5174009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:28.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f5189f2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5ccb40 0x561c3e5d9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f5189f2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5ccb40 0x561c3e5d9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43866/0 (socket says 172.21.15.33:43866) 2024-09-16T00:26:28.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f5189f2d640 1 -- 172.21.15.33:0/1779173889 learned_addr learned my addr 172.21.15.33:0/1779173889 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:28.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f518972c640 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5d6370 0x561c3e5da5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f518972c640 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 0x561c3e5decc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f5189f2d640 1 -- 172.21.15.33:0/1779173889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 msgr2=0x561c3e5decc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:28.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f5189f2d640 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 0x561c3e5decc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f5189f2d640 1 -- 172.21.15.33:0/1779173889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5d6370 msgr2=0x561c3e5da5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:28.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f5189f2d640 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5d6370 0x561c3e5da5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f5189f2d640 1 -- 172.21.15.33:0/1779173889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c3e5dae10 con 0x561c3e5ccb40 2024-09-16T00:26:28.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f5189f2d640 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5ccb40 0x561c3e5d9ff0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f51800058c0 tx=0x7f518001f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0071404014d9d80 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:28.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f516d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:26:28.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f516d7fa640 1 -- 172.21.15.33:0/1779173889 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518002a030 con 0x561c3e5ccb40 2024-09-16T00:26:28.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f516d7fa640 1 -- 172.21.15.33:0/1779173889 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5180004020 con 0x561c3e5ccb40 2024-09-16T00:26:28.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.659+0000 7f518e56a740 1 -- 172.21.15.33:0/1779173889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f517401d040 con 0x561c3e5ccb40 2024-09-16T00:26:28.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.660+0000 7f516d7fa640 1 -- 172.21.15.33:0/1779173889 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5180033050 con 0x561c3e5ccb40 2024-09-16T00:26:28.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.660+0000 7f518e56a740 10 client.17106 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:28.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.660+0000 7f518e56a740 1 -- 172.21.15.33:0/1779173889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561c3e650930 con 0x561c3e5ccb40 2024-09-16T00:26:28.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.660+0000 7f516d7fa640 1 -- 172.21.15.33:0/1779173889 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f517401d040 con 0x561c3e5ccb40 2024-09-16T00:26:28.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.660+0000 7f516d7fa640 1 -- 172.21.15.33:0/1779173889 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f5180010980 con 0x561c3e5ccb40 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.660+0000 7f518e56a740 10 client.17106 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.660+0000 7f518e56a740 10 client.17106 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.660+0000 7f518e56a740 20 client.17106 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.660+0000 7f518e56a740 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561c3e651110 0x561c3e653530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.660+0000 7f518e56a740 4 client.17106 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.660+0000 7f518e56a740 1 -- 172.21.15.33:0/1779173889 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561c3e538d70 con 0x561c3e651110 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.660+0000 7f518972c640 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561c3e651110 0x561c3e653530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518972c640 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561c3e651110 0x561c3e653530 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=baefd50b435b44c4 server_cookie=bca4312e8ca6a3d in_seq=0 out_seq=0 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f516d7fa640 10 client.17106 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f516d7fa640 1 -- 172.21.15.33:0/1779173889 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561c3e538d70 con 0x561c3e651110 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f516d7fa640 10 client.17106 handle_command_reply: tid=0 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518e56a740 1 client.17106 shutdown 2024-09-16T00:26:28.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518e56a740 1 -- 172.21.15.33:0/1779173889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5ccb40 msgr2=0x561c3e5d9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518e56a740 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5ccb40 0x561c3e5d9ff0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f51800058c0 tx=0x7f518001f850 comp rx=0 tx=0).stop 2024-09-16T00:26:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518e56a740 1 -- 172.21.15.33:0/1779173889 shutdown_connections 2024-09-16T00:26:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518e56a740 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561c3e651110 0x561c3e653530 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518e56a740 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c3e5d6370 0x561c3e5da5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518e56a740 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c3e5ccb40 0x561c3e5d9ff0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f516d7fa640 0 client.17106 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:26:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518e56a740 1 --2- 172.21.15.33:0/1779173889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c3e5cb5d0 0x561c3e5decc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518e56a740 1 -- 172.21.15.33:0/1779173889 >> 172.21.15.33:0/1779173889 conn(0x561c3e594310 msgr2=0x561c3e5d2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518e56a740 1 -- 172.21.15.33:0/1779173889 shutdown_connections 2024-09-16T00:26:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518e56a740 1 -- 172.21.15.33:0/1779173889 wait complete. 2024-09-16T00:26:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.661+0000 7f518e56a740 20 client.17106 trim_cache size 0 max 16384 2024-09-16T00:26:28.681 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:26:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.662+0000 7f518c191640 1 -- 172.21.15.33:0/1022863024 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f51700a32c0 msgr2=0x7f51700a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.662+0000 7f518c191640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f51700a32c0 0x7f51700a5780 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f517c04c000 tx=0x7f517c08cef0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.662+0000 7f518c191640 1 -- 172.21.15.33:0/1022863024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5184106f10 msgr2=0x7f5184074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.662+0000 7f518c191640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5184106f10 0x7f5184074f70 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f517c031a60 tx=0x7f517c031a90 comp rx=0 tx=0).stop 2024-09-16T00:26:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.662+0000 7f518c191640 1 -- 172.21.15.33:0/1022863024 shutdown_connections 2024-09-16T00:26:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.662+0000 7f518c191640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f51700a32c0 0x7f51700a5780 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.662+0000 7f518c191640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5184106f10 0x7f5184074f70 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.662+0000 7f518c191640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5184102790 0x7f5184074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.662+0000 7f518c191640 1 --2- 172.21.15.33:0/1022863024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5184101de0 0x7f51840744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.662+0000 7f518c191640 1 -- 172.21.15.33:0/1022863024 >> 172.21.15.33:0/1022863024 conn(0x7f51840fd8c0 msgr2=0x7f51840fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.662+0000 7f518c191640 1 -- 172.21.15.33:0/1022863024 shutdown_connections 2024-09-16T00:26:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:28.662+0000 7f518c191640 1 -- 172.21.15.33:0/1022863024 wait complete. 2024-09-16T00:26:28.683 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.16 MB/sec execute 336 sec latency 16938.637 ms 2024-09-16T00:26:29.381 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:26:29.638 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:29.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.13 MB/sec execute 337 sec latency 17938.665 ms 2024-09-16T00:26:29.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:29 smithi060 ceph-mon[35185]: pgmap v630: 129 pgs: 129 active+clean; 282 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.9 MiB/s wr, 74 op/s 2024-09-16T00:26:29.760 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:29 smithi033 ceph-mon[31243]: pgmap v630: 129 pgs: 129 active+clean; 282 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.9 MiB/s wr, 74 op/s 2024-09-16T00:26:29.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:29 smithi083 ceph-mon[35242]: pgmap v630: 129 pgs: 129 active+clean; 282 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.9 MiB/s wr, 74 op/s 2024-09-16T00:26:30.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.114+0000 7fef7ca20640 1 -- 172.21.15.33:0/2821246174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef78100e20 msgr2=0x7fef78101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.114+0000 7fef7ca20640 1 --2- 172.21.15.33:0/2821246174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef78100e20 0x7fef78101200 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fef68009900 tx=0x7fef6802a480 comp rx=0 tx=0).stop 2024-09-16T00:26:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.114+0000 7fef7ca20640 1 -- 172.21.15.33:0/2821246174 shutdown_connections 2024-09-16T00:26:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.114+0000 7fef7ca20640 1 --2- 172.21.15.33:0/2821246174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef78102ef0 0x7fef7810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.114+0000 7fef7ca20640 1 --2- 172.21.15.33:0/2821246174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef78102570 0x7fef781029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.114+0000 7fef7ca20640 1 --2- 172.21.15.33:0/2821246174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef78100e20 0x7fef78101200 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.114+0000 7fef7ca20640 1 -- 172.21.15.33:0/2821246174 >> 172.21.15.33:0/2821246174 conn(0x7fef780fc7d0 msgr2=0x7fef780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.114+0000 7fef7ca20640 1 -- 172.21.15.33:0/2821246174 shutdown_connections 2024-09-16T00:26:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.114+0000 7fef7ca20640 1 -- 172.21.15.33:0/2821246174 wait complete. 2024-09-16T00:26:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.115+0000 7fef7ca20640 1 Processor -- start 2024-09-16T00:26:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.115+0000 7fef7ca20640 1 -- start start 2024-09-16T00:26:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef7ca20640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef78100e20 0x7fef78198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef7ca20640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef78102570 0x7fef781989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef7ca20640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef78102ef0 0x7fef7819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef7ca20640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fef78110fd0 con 0x7fef78102570 2024-09-16T00:26:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef7ca20640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fef78110e50 con 0x7fef78102ef0 2024-09-16T00:26:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef7ca20640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fef78111150 con 0x7fef78100e20 2024-09-16T00:26:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef6dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef78102570 0x7fef781989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef6dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef78102570 0x7fef781989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43878/0 (socket says 172.21.15.33:43878) 2024-09-16T00:26:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef6dd9b640 1 -- 172.21.15.33:0/2115332932 learned_addr learned my addr 172.21.15.33:0/2115332932 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef7659c640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef78100e20 0x7fef78198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef76d9d640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef78102ef0 0x7fef7819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef6dd9b640 1 -- 172.21.15.33:0/2115332932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef78100e20 msgr2=0x7fef78198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef78100e20 0x7fef78198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef6dd9b640 1 -- 172.21.15.33:0/2115332932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef78102ef0 msgr2=0x7fef7819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef78102ef0 0x7fef7819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef6dd9b640 1 -- 172.21.15.33:0/2115332932 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7819d440 con 0x7fef78102570 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef76d9d640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef78102ef0 0x7fef7819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef7659c640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef78100e20 0x7fef78198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef78102570 0x7fef781989d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fef6000c890 tx=0x7fef6000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.116+0000 7fef6ffff640 1 -- 172.21.15.33:0/2115332932 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60004220 con 0x7fef78102570 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.117+0000 7fef6ffff640 1 -- 172.21.15.33:0/2115332932 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef600043c0 con 0x7fef78102570 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.117+0000 7fef7ca20640 1 -- 172.21.15.33:0/2115332932 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef7819d730 con 0x7fef78102570 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.117+0000 7fef6ffff640 1 -- 172.21.15.33:0/2115332932 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef600049e0 con 0x7fef78102570 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.117+0000 7fef7ca20640 1 -- 172.21.15.33:0/2115332932 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef781a5010 con 0x7fef78102570 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.119+0000 7fef6ffff640 1 -- 172.21.15.33:0/2115332932 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fef600079b0 con 0x7fef78102570 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.119+0000 7fef7edf9740 1 Processor -- start 2024-09-16T00:26:30.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef7edf9740 1 -- start start 2024-09-16T00:26:30.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef7edf9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264d9b40 0x5654264d8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef7edf9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 0x5654264d8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef7edf9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 0x5654264e5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef7edf9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56542643e2f0 con 0x5654264e3370 2024-09-16T00:26:30.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef7edf9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5654263e5cf0 con 0x5654264d9b40 2024-09-16T00:26:30.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef7edf9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56542643a2b0 con 0x5654264d85d0 2024-09-16T00:26:30.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef6dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 0x5654264e5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef6ffff640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fef640a3240 0x7fef640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef7659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 0x5654264d8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef6dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 0x5654264e5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43886/0 (socket says 172.21.15.33:43886) 2024-09-16T00:26:30.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef6dd9b640 1 -- 172.21.15.33:0/1695252305 learned_addr learned my addr 172.21.15.33:0/1695252305 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:30.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef76d9d640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fef640a3240 0x7fef640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef76d9d640 1 --2- 172.21.15.33:0/1695252305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264d9b40 0x5654264d8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.120+0000 7fef6ffff640 1 -- 172.21.15.33:0/2115332932 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fef600cfb90 con 0x7fef78102570 2024-09-16T00:26:30.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.121+0000 7fef6dd9b640 1 -- 172.21.15.33:0/1695252305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 msgr2=0x5654264d8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:30.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.121+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/1695252305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 0x5654264d8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.121+0000 7fef6dd9b640 1 -- 172.21.15.33:0/1695252305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264d9b40 msgr2=0x5654264d8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:30.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.121+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/1695252305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264d9b40 0x5654264d8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.121+0000 7fef6dd9b640 1 -- 172.21.15.33:0/1695252305 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654263ca3a0 con 0x5654264e3370 2024-09-16T00:26:30.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.121+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/1695252305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 0x5654264e5760 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fef600b0ac0 tx=0x7fef60005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eed7659f435abfc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:30.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.121+0000 7fef76d9d640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fef640a3240 0x7fef640a5700 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fef700138e0 tx=0x7fef700074c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.121+0000 7fef6cd99640 1 -- 172.21.15.33:0/1695252305 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6001d070 con 0x5654264e3370 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.121+0000 7fef6cd99640 1 -- 172.21.15.33:0/1695252305 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef6009c3d0 con 0x5654264e3370 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.121+0000 7fef6cd99640 1 -- 172.21.15.33:0/1695252305 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60007ca0 con 0x5654264e3370 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.122+0000 7fef7edf9740 1 -- 172.21.15.33:0/1695252305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 msgr2=0x5654264e5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.122+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1695252305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 0x5654264e5760 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fef600b0ac0 tx=0x7fef60005300 comp rx=0 tx=0).stop 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.122+0000 7fef7edf9740 1 -- 172.21.15.33:0/1695252305 shutdown_connections 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.122+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1695252305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 0x5654264e5760 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.122+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1695252305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 0x5654264d8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.122+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1695252305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264d9b40 0x5654264d8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.122+0000 7fef7edf9740 1 -- 172.21.15.33:0/1695252305 >> 172.21.15.33:0/1695252305 conn(0x5654264a1310 msgr2=0x5654264d4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.122+0000 7fef7edf9740 1 -- 172.21.15.33:0/1695252305 shutdown_connections 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.122+0000 7fef7edf9740 1 -- 172.21.15.33:0/1695252305 wait complete. 2024-09-16T00:26:30.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.122+0000 7fef7edf9740 1 -- start start 2024-09-16T00:26:30.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.122+0000 7fef7edf9740 1 Processor -- start 2024-09-16T00:26:30.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7edf9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d85d0 0x5654264e2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7edf9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d9b40 0x5654264e0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7edf9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264e3370 0x5654264e05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7edf9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5654264e5fb0 con 0x5654264d85d0 2024-09-16T00:26:30.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7edf9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5654263f9060 con 0x5654264e3370 2024-09-16T00:26:30.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7edf9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5654264e6130 con 0x5654264d9b40 2024-09-16T00:26:30.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7edf9740 10 client.0 _ll_register_callbacks cb 0x56542614c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:30.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d85d0 0x5654264e2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d85d0 0x5654264e2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43890/0 (socket says 172.21.15.33:43890) 2024-09-16T00:26:30.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7659c640 1 -- 172.21.15.33:0/1702354924 learned_addr learned my addr 172.21.15.33:0/1702354924 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:30.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d9b40 0x5654264e0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef76d9d640 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264e3370 0x5654264e05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7659c640 1 -- 172.21.15.33:0/1702354924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d9b40 msgr2=0x5654264e0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7659c640 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d9b40 0x5654264e0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7659c640 1 -- 172.21.15.33:0/1702354924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264e3370 msgr2=0x5654264e05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7659c640 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264e3370 0x5654264e05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7659c640 1 -- 172.21.15.33:0/1702354924 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654261af3c0 con 0x5654264d85d0 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef7659c640 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d85d0 0x5654264e2d50 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fef68005c40 tx=0x7fef68003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb1234df0478ad1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.123+0000 7fef5dffb640 1 -- 172.21.15.33:0/1702354924 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68035040 con 0x5654264d85d0 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef7edf9740 1 -- 172.21.15.33:0/1702354924 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef6009c3d0 con 0x5654264d85d0 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef5dffb640 1 -- 172.21.15.33:0/1702354924 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fef68003c10 con 0x5654264d85d0 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef5dffb640 1 -- 172.21.15.33:0/1702354924 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fef6804d050 con 0x5654264d85d0 2024-09-16T00:26:30.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef7edf9740 10 client.17118 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:30.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef7edf9740 1 -- 172.21.15.33:0/1702354924 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5654264f1a80 con 0x5654264d85d0 2024-09-16T00:26:30.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef5dffb640 1 -- 172.21.15.33:0/1702354924 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef6009c3d0 con 0x5654264d85d0 2024-09-16T00:26:30.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef5dffb640 1 -- 172.21.15.33:0/1702354924 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fef6803c600 con 0x5654264d85d0 2024-09-16T00:26:30.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef7edf9740 10 client.17118 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:30.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef7edf9740 10 client.17118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:26:30.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef7edf9740 20 client.17118 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:30.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5654264f2e90 0x5654264f5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef7edf9740 4 client.17118 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:26:30.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef7edf9740 1 -- 172.21.15.33:0/1702354924 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56542644b860 con 0x5654264f2e90 2024-09-16T00:26:30.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.124+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5654264f2e90 0x5654264f5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.125+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5654264f2e90 0x5654264f5280 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1182130c9f593ab9 server_cookie=d40fccec6271abe0 in_seq=0 out_seq=0 2024-09-16T00:26:30.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.125+0000 7fef5dffb640 10 client.17118 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:30.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef5dffb640 1 -- 172.21.15.33:0/1702354924 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56542644b860 con 0x5654264f2e90 2024-09-16T00:26:30.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef5dffb640 10 client.17118 handle_command_reply: tid=0 2024-09-16T00:26:30.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef7edf9740 1 client.17118 shutdown 2024-09-16T00:26:30.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef7edf9740 1 -- 172.21.15.33:0/1702354924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d85d0 msgr2=0x5654264e2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:30.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d85d0 0x5654264e2d50 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fef68005c40 tx=0x7fef68003820 comp rx=0 tx=0).stop 2024-09-16T00:26:30.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef7edf9740 1 -- 172.21.15.33:0/1702354924 shutdown_connections 2024-09-16T00:26:30.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5654264f2e90 0x5654264f5280 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264e3370 0x5654264e05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d9b40 0x5654264e0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef5dffb640 0 client.17118 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:30.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1702354924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d85d0 0x5654264e2d50 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef7edf9740 1 -- 172.21.15.33:0/1702354924 >> 172.21.15.33:0/1702354924 conn(0x5654264a1310 msgr2=0x5654264da960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:30.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef7edf9740 1 -- 172.21.15.33:0/1702354924 shutdown_connections 2024-09-16T00:26:30.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef7edf9740 1 -- 172.21.15.33:0/1702354924 wait complete. 2024-09-16T00:26:30.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.126+0000 7fef7edf9740 20 client.17118 trim_cache size 0 max 16384 2024-09-16T00:26:30.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.139+0000 7fef7edf9740 1 Processor -- start 2024-09-16T00:26:30.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.139+0000 7fef7edf9740 1 -- start start 2024-09-16T00:26:30.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.139+0000 7fef7edf9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 0x5654264db120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.139+0000 7fef7edf9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d9b40 0x5654264db660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.139+0000 7fef7edf9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264e3370 0x5654264fab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.139+0000 7fef7edf9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fef781b1980 con 0x5654264d9b40 2024-09-16T00:26:30.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.139+0000 7fef7edf9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fef781af250 con 0x5654264e3370 2024-09-16T00:26:30.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.139+0000 7fef7edf9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fef781ad0a0 con 0x5654264d85d0 2024-09-16T00:26:30.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.139+0000 7fef6dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d9b40 0x5654264db660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.139+0000 7fef6dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d9b40 0x5654264db660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43896/0 (socket says 172.21.15.33:43896) 2024-09-16T00:26:30.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.139+0000 7fef6dd9b640 1 -- 172.21.15.33:0/3328780149 learned_addr learned my addr 172.21.15.33:0/3328780149 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:30.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.140+0000 7fef7659c640 1 --2- 172.21.15.33:0/3328780149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 0x5654264db120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.140+0000 7fef76d9d640 1 --2- 172.21.15.33:0/3328780149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264e3370 0x5654264fab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.140+0000 7fef6dd9b640 1 -- 172.21.15.33:0/3328780149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 msgr2=0x5654264db120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:30.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.140+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/3328780149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 0x5654264db120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.140+0000 7fef6dd9b640 1 -- 172.21.15.33:0/3328780149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264e3370 msgr2=0x5654264fab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:30.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.140+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/3328780149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264e3370 0x5654264fab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.140+0000 7fef6dd9b640 1 -- 172.21.15.33:0/3328780149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654264d9780 con 0x5654264d9b40 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.140+0000 7fef76d9d640 1 --2- 172.21.15.33:0/3328780149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264e3370 0x5654264fab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.140+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/3328780149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d9b40 0x5654264db660 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fef6009bc60 tx=0x7fef60000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13fb5857e4dce796 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.140+0000 7fef5ffff640 1 -- 172.21.15.33:0/3328780149 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6001d040 con 0x5654264d9b40 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.140+0000 7fef5ffff640 1 -- 172.21.15.33:0/3328780149 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef6009c3d0 con 0x5654264d9b40 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.140+0000 7fef5ffff640 1 -- 172.21.15.33:0/3328780149 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6009b120 con 0x5654264d9b40 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.141+0000 7fef7edf9740 1 -- 172.21.15.33:0/3328780149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d9b40 msgr2=0x5654264db660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.141+0000 7fef7edf9740 1 --2- 172.21.15.33:0/3328780149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d9b40 0x5654264db660 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fef6009bc60 tx=0x7fef60000bc0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.141+0000 7fef7edf9740 1 -- 172.21.15.33:0/3328780149 shutdown_connections 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.141+0000 7fef7edf9740 1 --2- 172.21.15.33:0/3328780149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264e3370 0x5654264fab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.141+0000 7fef7edf9740 1 --2- 172.21.15.33:0/3328780149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264d9b40 0x5654264db660 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.141+0000 7fef7edf9740 1 --2- 172.21.15.33:0/3328780149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 0x5654264db120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.141+0000 7fef7edf9740 1 -- 172.21.15.33:0/3328780149 >> 172.21.15.33:0/3328780149 conn(0x5654264a1310 msgr2=0x5654264d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.141+0000 7fef7edf9740 1 -- 172.21.15.33:0/3328780149 shutdown_connections 2024-09-16T00:26:30.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.141+0000 7fef7edf9740 1 -- 172.21.15.33:0/3328780149 wait complete. 2024-09-16T00:26:30.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.141+0000 7fef7edf9740 1 -- start start 2024-09-16T00:26:30.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.141+0000 7fef7edf9740 1 Processor -- start 2024-09-16T00:26:30.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef7edf9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 0x5654264e7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef7edf9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264d9b40 0x5654264e7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef7edf9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 0x5654264e7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef7edf9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fef781ad710 con 0x5654264e3370 2024-09-16T00:26:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef7edf9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fef781aeb80 con 0x5654264d9b40 2024-09-16T00:26:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef7edf9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fef781adde0 con 0x5654264d85d0 2024-09-16T00:26:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef7edf9740 10 client.0 _ll_register_callbacks cb 0x7fef68004500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef76d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 0x5654264e7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef76d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 0x5654264e7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43906/0 (socket says 172.21.15.33:43906) 2024-09-16T00:26:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef76d9d640 1 -- 172.21.15.33:0/1004901617 learned_addr learned my addr 172.21.15.33:0/1004901617 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef7659c640 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 0x5654264e7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef6dd9b640 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264d9b40 0x5654264e7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef76d9d640 1 -- 172.21.15.33:0/1004901617 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 msgr2=0x5654264e7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef76d9d640 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 0x5654264e7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef76d9d640 1 -- 172.21.15.33:0/1004901617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264d9b40 msgr2=0x5654264e7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef76d9d640 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264d9b40 0x5654264e7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef76d9d640 1 -- 172.21.15.33:0/1004901617 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56542655cb00 con 0x5654264e3370 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef76d9d640 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 0x5654264e7b80 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fef7000ee50 tx=0x7fef70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=350a26f086dd0cb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.142+0000 7fef5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef5dffb640 1 -- 172.21.15.33:0/1004901617 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7002d060 con 0x5654264e3370 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef5dffb640 1 -- 172.21.15.33:0/1004901617 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fef70011850 con 0x5654264e3370 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef7edf9740 1 -- 172.21.15.33:0/1004901617 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef6009c3d0 con 0x5654264e3370 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef5dffb640 1 -- 172.21.15.33:0/1004901617 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fef70034050 con 0x5654264e3370 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef7edf9740 10 client.17124 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef7edf9740 1 -- 172.21.15.33:0/1004901617 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x565426562230 con 0x5654264e3370 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef5dffb640 1 -- 172.21.15.33:0/1004901617 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef6009c3d0 con 0x5654264e3370 2024-09-16T00:26:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef5dffb640 1 -- 172.21.15.33:0/1004901617 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fef70027030 con 0x5654264e3370 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef7edf9740 10 client.17124 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef7edf9740 10 client.17124 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef7edf9740 20 client.17124 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5654265629b0 0x565426564dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef7edf9740 4 client.17124 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.143+0000 7fef7edf9740 1 -- 172.21.15.33:0/1004901617 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565426445d70 con 0x5654265629b0 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef7659c640 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5654265629b0 0x565426564dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef7659c640 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5654265629b0 0x565426564dd0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2af227f254ab05e server_cookie=cbaf815daff4712 in_seq=0 out_seq=0 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef5dffb640 10 client.17124 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef5dffb640 1 -- 172.21.15.33:0/1004901617 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565426445d70 con 0x5654265629b0 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef5dffb640 10 client.17124 handle_command_reply: tid=0 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef7edf9740 1 client.17124 shutdown 2024-09-16T00:26:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef7edf9740 1 -- 172.21.15.33:0/1004901617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 msgr2=0x5654264e7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:30.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 0x5654264e7b80 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fef7000ee50 tx=0x7fef70004a40 comp rx=0 tx=0).stop 2024-09-16T00:26:30.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef7edf9740 1 -- 172.21.15.33:0/1004901617 shutdown_connections 2024-09-16T00:26:30.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5654265629b0 0x565426564dd0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654264e3370 0x5654264e7b80 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5654264d9b40 0x5654264e7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef7edf9740 1 --2- 172.21.15.33:0/1004901617 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654264d85d0 0x5654264e7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef7edf9740 1 -- 172.21.15.33:0/1004901617 >> 172.21.15.33:0/1004901617 conn(0x5654264a1310 msgr2=0x5654264e1860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:30.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.144+0000 7fef5dffb640 0 client.17124 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:30.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7edf9740 1 -- 172.21.15.33:0/1004901617 shutdown_connections 2024-09-16T00:26:30.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7edf9740 1 -- 172.21.15.33:0/1004901617 wait complete. 2024-09-16T00:26:30.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7edf9740 20 client.17124 trim_cache size 0 max 16384 2024-09-16T00:26:30.163 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:26:30.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7ca20640 1 -- 172.21.15.33:0/2115332932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fef640a3240 msgr2=0x7fef640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:30.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7ca20640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fef640a3240 0x7fef640a5700 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fef700138e0 tx=0x7fef700074c0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7ca20640 1 -- 172.21.15.33:0/2115332932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef78102570 msgr2=0x7fef781989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:30.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7ca20640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef78102570 0x7fef781989d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fef6000c890 tx=0x7fef6000cd50 comp rx=0 tx=0).stop 2024-09-16T00:26:30.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7ca20640 1 -- 172.21.15.33:0/2115332932 shutdown_connections 2024-09-16T00:26:30.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7ca20640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fef640a3240 0x7fef640a5700 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7ca20640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef78102ef0 0x7fef7819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7ca20640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef78102570 0x7fef781989d0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7ca20640 1 --2- 172.21.15.33:0/2115332932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef78100e20 0x7fef78198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:30.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7ca20640 1 -- 172.21.15.33:0/2115332932 >> 172.21.15.33:0/2115332932 conn(0x7fef780fc7d0 msgr2=0x7fef7806a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:30.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7ca20640 1 -- 172.21.15.33:0/2115332932 shutdown_connections 2024-09-16T00:26:30.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:30.145+0000 7fef7ca20640 1 -- 172.21.15.33:0/2115332932 wait complete. 2024-09-16T00:26:30.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.10 MB/sec execute 338 sec latency 18938.765 ms 2024-09-16T00:26:31.425 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:26:31.674 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.07 MB/sec execute 339 sec latency 19938.812 ms 2024-09-16T00:26:31.696 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:31.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:31 smithi060 ceph-mon[35185]: pgmap v631: 129 pgs: 129 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 2.0 MiB/s wr, 85 op/s 2024-09-16T00:26:31.817 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:31 smithi033 ceph-mon[31243]: pgmap v631: 129 pgs: 129 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 2.0 MiB/s wr, 85 op/s 2024-09-16T00:26:31.842 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:26:31.842 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:26:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:31 smithi083 ceph-mon[35242]: pgmap v631: 129 pgs: 129 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 2.0 MiB/s wr, 85 op/s 2024-09-16T00:26:32.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.179+0000 7f424f598640 1 -- 172.21.15.33:0/1459009835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4248106f10 msgr2=0x7f424810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:32.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.179+0000 7f424f598640 1 --2- 172.21.15.33:0/1459009835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4248106f10 0x7f424810ea90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4240009ca0 tx=0x7f424002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:26:32.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.179+0000 7f424f598640 1 -- 172.21.15.33:0/1459009835 shutdown_connections 2024-09-16T00:26:32.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.179+0000 7f424f598640 1 --2- 172.21.15.33:0/1459009835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4248106f10 0x7f424810ea90 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:32.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.179+0000 7f424f598640 1 --2- 172.21.15.33:0/1459009835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4248102790 0x7f42481067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:32.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.179+0000 7f424f598640 1 --2- 172.21.15.33:0/1459009835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4248101de0 0x7f42481021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:32.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.179+0000 7f424f598640 1 -- 172.21.15.33:0/1459009835 >> 172.21.15.33:0/1459009835 conn(0x7f42480fd8c0 msgr2=0x7f42480ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:32.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.179+0000 7f424f598640 1 -- 172.21.15.33:0/1459009835 shutdown_connections 2024-09-16T00:26:32.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.179+0000 7f424f598640 1 -- 172.21.15.33:0/1459009835 wait complete. 2024-09-16T00:26:32.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.180+0000 7f424f598640 1 Processor -- start 2024-09-16T00:26:32.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.180+0000 7f424f598640 1 -- start start 2024-09-16T00:26:32.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.180+0000 7f424f598640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4248101de0 0x7f4248198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:32.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.180+0000 7f424f598640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4248102790 0x7f4248198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:32.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.180+0000 7f424f598640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4248106f10 0x7f424819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:32.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.180+0000 7f424f598640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4248111f70 con 0x7f4248102790 2024-09-16T00:26:32.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.180+0000 7f424f598640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4248111df0 con 0x7f4248106f10 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.180+0000 7f424f598640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f42481120f0 con 0x7f4248101de0 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.180+0000 7f424cb33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4248102790 0x7f4248198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.180+0000 7f424cb33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4248102790 0x7f4248198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34278/0 (socket says 172.21.15.33:34278) 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.180+0000 7f424cb33640 1 -- 172.21.15.33:0/2329161562 learned_addr learned my addr 172.21.15.33:0/2329161562 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.180+0000 7f424d334640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4248101de0 0x7f4248198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.181+0000 7f424db35640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4248106f10 0x7f424819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.181+0000 7f424cb33640 1 -- 172.21.15.33:0/2329161562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4248101de0 msgr2=0x7f4248198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.181+0000 7f424cb33640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4248101de0 0x7f4248198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.181+0000 7f424cb33640 1 -- 172.21.15.33:0/2329161562 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4248106f10 msgr2=0x7f424819cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.181+0000 7f424cb33640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4248106f10 0x7f424819cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.181+0000 7f424cb33640 1 -- 172.21.15.33:0/2329161562 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424819d2f0 con 0x7f4248102790 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.181+0000 7f424cb33640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4248102790 0x7f4248198880 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f423800c920 tx=0x7f423800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.181+0000 7f423e7fc640 1 -- 172.21.15.33:0/2329161562 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238007c70 con 0x7f4248102790 2024-09-16T00:26:32.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.181+0000 7f424f598640 1 -- 172.21.15.33:0/2329161562 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f424819d5e0 con 0x7f4248102790 2024-09-16T00:26:32.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.181+0000 7f423e7fc640 1 -- 172.21.15.33:0/2329161562 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4238007e10 con 0x7f4248102790 2024-09-16T00:26:32.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.181+0000 7f423e7fc640 1 -- 172.21.15.33:0/2329161562 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238018890 con 0x7f4248102790 2024-09-16T00:26:32.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.181+0000 7f424f598640 1 -- 172.21.15.33:0/2329161562 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42481a4ec0 con 0x7f4248102790 2024-09-16T00:26:32.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.182+0000 7f424f598640 1 -- 172.21.15.33:0/2329161562 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42481021c0 con 0x7f4248102790 2024-09-16T00:26:32.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.183+0000 7f423e7fc640 1 -- 172.21.15.33:0/2329161562 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4238018b10 con 0x7f4248102790 2024-09-16T00:26:32.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.184+0000 7f423e7fc640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42340a3240 0x7f42340a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:32.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.184+0000 7f424d334640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42340a3240 0x7f42340a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:32.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.184+0000 7f423e7fc640 1 -- 172.21.15.33:0/2329161562 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f42380cfef0 con 0x7f4248102790 2024-09-16T00:26:32.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.184+0000 7f424d334640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42340a3240 0x7f42340a5700 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4244005fd0 tx=0x7f424400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:32.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.188+0000 7f423e7fc640 1 -- 172.21.15.33:0/2329161562 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4238014030 con 0x7f4248102790 2024-09-16T00:26:32.339 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:32.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.485+0000 7f424f598640 1 -- 172.21.15.33:0/2329161562 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4248109bc0 con 0x7f4248102790 2024-09-16T00:26:32.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.486+0000 7f423e7fc640 1 -- 172.21.15.33:0/2329161562 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f423809c730 con 0x7f4248102790 2024-09-16T00:26:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:26:32.488 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:26:32.489 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:26:32.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.488+0000 7f424f598640 1 -- 172.21.15.33:0/2329161562 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42340a3240 msgr2=0x7f42340a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:32.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.488+0000 7f424f598640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42340a3240 0x7f42340a5700 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4244005fd0 tx=0x7f424400b390 comp rx=0 tx=0).stop 2024-09-16T00:26:32.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.488+0000 7f424f598640 1 -- 172.21.15.33:0/2329161562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4248102790 msgr2=0x7f4248198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:32.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.488+0000 7f424f598640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4248102790 0x7f4248198880 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f423800c920 tx=0x7f423800cde0 comp rx=0 tx=0).stop 2024-09-16T00:26:32.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.488+0000 7f424f598640 1 -- 172.21.15.33:0/2329161562 shutdown_connections 2024-09-16T00:26:32.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.488+0000 7f424f598640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42340a3240 0x7f42340a5700 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:32.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.488+0000 7f424f598640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4248106f10 0x7f424819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:32.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.488+0000 7f424f598640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4248102790 0x7f4248198880 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:32.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.488+0000 7f424f598640 1 --2- 172.21.15.33:0/2329161562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4248101de0 0x7f4248198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:32.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.488+0000 7f424f598640 1 -- 172.21.15.33:0/2329161562 >> 172.21.15.33:0/2329161562 conn(0x7f42480fd8c0 msgr2=0x7f42480febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:32.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.488+0000 7f424f598640 1 -- 172.21.15.33:0/2329161562 shutdown_connections 2024-09-16T00:26:32.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:32.488+0000 7f424f598640 1 -- 172.21.15.33:0/2329161562 wait complete. 2024-09-16T00:26:32.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.04 MB/sec execute 340 sec latency 20938.859 ms 2024-09-16T00:26:33.352 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id de406742 --release --await' 2024-09-16T00:26:33.352 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id de406742 --release --await 2024-09-16T00:26:33.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:33 smithi033 ceph-mon[31243]: pgmap v632: 129 pgs: 129 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 2.2 MiB/s wr, 90 op/s 2024-09-16T00:26:33.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:33 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2329161562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:33.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.626+0000 7f126b8ed640 1 -- 172.21.15.33:0/2050574408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1264103740 msgr2=0x7f1264107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:33.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.626+0000 7f126b8ed640 1 --2- 172.21.15.33:0/2050574408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1264103740 0x7f1264107b80 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1260009930 tx=0x7f126002a420 comp rx=0 tx=0).stop 2024-09-16T00:26:33.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.626+0000 7f126b8ed640 1 -- 172.21.15.33:0/2050574408 shutdown_connections 2024-09-16T00:26:33.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.626+0000 7f126b8ed640 1 --2- 172.21.15.33:0/2050574408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1264108290 0x7f126410ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.626+0000 7f126b8ed640 1 --2- 172.21.15.33:0/2050574408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1264103740 0x7f1264107b80 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.626+0000 7f126b8ed640 1 --2- 172.21.15.33:0/2050574408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1264102d90 0x7f1264103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.626+0000 7f126b8ed640 1 -- 172.21.15.33:0/2050574408 >> 172.21.15.33:0/2050574408 conn(0x7f12640fc7d0 msgr2=0x7f12640febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:33.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.626+0000 7f126b8ed640 1 -- 172.21.15.33:0/2050574408 shutdown_connections 2024-09-16T00:26:33.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.626+0000 7f126b8ed640 1 -- 172.21.15.33:0/2050574408 wait complete. 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.626+0000 7f126b8ed640 1 Processor -- start 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.626+0000 7f126b8ed640 1 -- start start 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f126b8ed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1264102d90 0x7f1264198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f126b8ed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1264103740 0x7f1264198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f126b8ed640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1264108290 0x7f126419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f126b8ed640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1264112090 con 0x7f1264108290 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f126b8ed640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1264111f10 con 0x7f1264103740 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f126b8ed640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1264112210 con 0x7f1264102d90 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f1269e8a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1264108290 0x7f126419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f1269e8a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1264108290 0x7f126419cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34294/0 (socket says 172.21.15.33:34294) 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f1269e8a640 1 -- 172.21.15.33:0/3619962289 learned_addr learned my addr 172.21.15.33:0/3619962289 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f1268e88640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1264103740 0x7f1264198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f1269689640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1264102d90 0x7f1264198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f1269e8a640 1 -- 172.21.15.33:0/3619962289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1264102d90 msgr2=0x7f1264198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:33.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f1269e8a640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1264102d90 0x7f1264198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f1269e8a640 1 -- 172.21.15.33:0/3619962289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1264103740 msgr2=0x7f1264198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:33.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f1269e8a640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1264103740 0x7f1264198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f1269e8a640 1 -- 172.21.15.33:0/3619962289 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126419d5c0 con 0x7f1264108290 2024-09-16T00:26:33.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f1268e88640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1264103740 0x7f1264198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:33.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f1269e8a640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1264108290 0x7f126419cee0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f125c00cad0 tx=0x7f125c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:33.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.627+0000 7f12567fc640 1 -- 172.21.15.33:0/3619962289 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c00ae20 con 0x7f1264108290 2024-09-16T00:26:33.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.628+0000 7f126b8ed640 1 -- 172.21.15.33:0/3619962289 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f126419d8b0 con 0x7f1264108290 2024-09-16T00:26:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.628+0000 7f12567fc640 1 -- 172.21.15.33:0/3619962289 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f125c005020 con 0x7f1264108290 2024-09-16T00:26:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.628+0000 7f12567fc640 1 -- 172.21.15.33:0/3619962289 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c00f6f0 con 0x7f1264108290 2024-09-16T00:26:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.628+0000 7f126b8ed640 1 -- 172.21.15.33:0/3619962289 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12641a5190 con 0x7f1264108290 2024-09-16T00:26:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.630+0000 7f12567fc640 1 -- 172.21.15.33:0/3619962289 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f125c0051d0 con 0x7f1264108290 2024-09-16T00:26:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.630+0000 7f126dcc6740 1 Processor -- start 2024-09-16T00:26:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f126dcc6740 1 -- start start 2024-09-16T00:26:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f126dcc6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 0x561c76279f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f126dcc6740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c7627a480 0x561c76284800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f126dcc6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c76287130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f126dcc6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561c761df280 con 0x561c76279b60 2024-09-16T00:26:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f126dcc6740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561c76186b10 con 0x561c7627a480 2024-09-16T00:26:33.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f126dcc6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561c761db240 con 0x561c76284d40 2024-09-16T00:26:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f1268e88640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 0x561c76279f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f1268e88640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 0x561c76279f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34308/0 (socket says 172.21.15.33:34308) 2024-09-16T00:26:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f1268e88640 1 -- 172.21.15.33:0/823544971 learned_addr learned my addr 172.21.15.33:0/823544971 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f1269689640 1 --2- 172.21.15.33:0/823544971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c7627a480 0x561c76284800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f1268e88640 1 --2- 172.21.15.33:0/823544971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c76287130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f12567fc640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12500a3240 0x7f12500a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f1269689640 1 -- 172.21.15.33:0/823544971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 msgr2=0x561c76287130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f1269689640 1 --2- 172.21.15.33:0/823544971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c76287130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f1269e8a640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12500a3240 0x7f12500a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f1269689640 1 -- 172.21.15.33:0/823544971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 msgr2=0x561c76279f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f1269689640 1 --2- 172.21.15.33:0/823544971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 0x561c76279f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f1269689640 1 -- 172.21.15.33:0/823544971 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c761931f0 con 0x561c7627a480 2024-09-16T00:26:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f1268e88640 1 --2- 172.21.15.33:0/823544971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 0x561c76279f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:33.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.631+0000 7f12567fc640 1 -- 172.21.15.33:0/3619962289 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f125c0cfd10 con 0x7f1264108290 2024-09-16T00:26:33.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.632+0000 7f1269689640 1 --2- 172.21.15.33:0/823544971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c7627a480 0x561c76284800 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1258004630 tx=0x7f125800ba60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f78ec00f9bcf60 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:33.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.632+0000 7f1269e8a640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12500a3240 0x7f12500a5700 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f125c0983c0 tx=0x7f125c091440 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:33.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.632+0000 7f124f7fe640 1 -- 172.21.15.33:0/823544971 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258003a90 con 0x561c7627a480 2024-09-16T00:26:33.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.632+0000 7f124f7fe640 1 -- 172.21.15.33:0/823544971 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1258015070 con 0x561c7627a480 2024-09-16T00:26:33.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.632+0000 7f124f7fe640 1 -- 172.21.15.33:0/823544971 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258004c60 con 0x561c7627a480 2024-09-16T00:26:33.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.633+0000 7f126dcc6740 1 -- 172.21.15.33:0/823544971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c7627a480 msgr2=0x561c76284800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:33.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.633+0000 7f126dcc6740 1 --2- 172.21.15.33:0/823544971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c7627a480 0x561c76284800 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1258004630 tx=0x7f125800ba60 comp rx=0 tx=0).stop 2024-09-16T00:26:33.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.633+0000 7f126dcc6740 1 -- 172.21.15.33:0/823544971 shutdown_connections 2024-09-16T00:26:33.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.633+0000 7f126dcc6740 1 --2- 172.21.15.33:0/823544971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c76287130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.633+0000 7f126dcc6740 1 --2- 172.21.15.33:0/823544971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c7627a480 0x561c76284800 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.633+0000 7f126dcc6740 1 --2- 172.21.15.33:0/823544971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 0x561c76279f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.633+0000 7f126dcc6740 1 -- 172.21.15.33:0/823544971 >> 172.21.15.33:0/823544971 conn(0x561c762760f0 msgr2=0x561c762764d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:33.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.633+0000 7f126dcc6740 1 -- 172.21.15.33:0/823544971 shutdown_connections 2024-09-16T00:26:33.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.633+0000 7f126dcc6740 1 -- 172.21.15.33:0/823544971 wait complete. 2024-09-16T00:26:33.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.633+0000 7f126dcc6740 1 -- start start 2024-09-16T00:26:33.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.633+0000 7f126dcc6740 1 Processor -- start 2024-09-16T00:26:33.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f126dcc6740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c76279b60 0x561c76281ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f126dcc6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 0x561c762820e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f126dcc6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c76282620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f126dcc6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561c76287910 con 0x561c7627a480 2024-09-16T00:26:33.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f126dcc6740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561c76199f10 con 0x561c76279b60 2024-09-16T00:26:33.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f126dcc6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561c76287a90 con 0x561c76284d40 2024-09-16T00:26:33.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f126dcc6740 10 client.0 _ll_register_callbacks cb 0x561c76162a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:33.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f1269689640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 0x561c762820e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f1269689640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 0x561c762820e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34322/0 (socket says 172.21.15.33:34322) 2024-09-16T00:26:33.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f1269689640 1 -- 172.21.15.33:0/3432754921 learned_addr learned my addr 172.21.15.33:0/3432754921 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:33.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f1268e88640 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c76279b60 0x561c76281ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f1268e88640 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c76282620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f1269689640 1 -- 172.21.15.33:0/3432754921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 msgr2=0x561c76282620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:33.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f1269689640 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c76282620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f1269689640 1 -- 172.21.15.33:0/3432754921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c76279b60 msgr2=0x561c76281ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:33.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f1269689640 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c76279b60 0x561c76281ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f1269689640 1 -- 172.21.15.33:0/3432754921 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c75f506a0 con 0x561c7627a480 2024-09-16T00:26:33.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f1269689640 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 0x561c762820e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f1258005cb0 tx=0x7f1258003ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a9234187eef8ca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:33.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.634+0000 7f124cff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:26:33.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f124cff9640 1 -- 172.21.15.33:0/3432754921 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258028060 con 0x561c7627a480 2024-09-16T00:26:33.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f126dcc6740 1 -- 172.21.15.33:0/3432754921 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1258015070 con 0x561c7627a480 2024-09-16T00:26:33.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f124cff9640 1 -- 172.21.15.33:0/3432754921 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1258011640 con 0x561c7627a480 2024-09-16T00:26:33.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f124cff9640 1 -- 172.21.15.33:0/3432754921 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f125802f050 con 0x561c7627a480 2024-09-16T00:26:33.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f126dcc6740 10 client.17142 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:33.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f126dcc6740 1 -- 172.21.15.33:0/3432754921 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561c762934a0 con 0x561c7627a480 2024-09-16T00:26:33.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f124cff9640 1 -- 172.21.15.33:0/3432754921 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1258015070 con 0x561c7627a480 2024-09-16T00:26:33.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f124cff9640 1 -- 172.21.15.33:0/3432754921 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f1258022070 con 0x561c7627a480 2024-09-16T00:26:33.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f126dcc6740 10 client.17142 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:33.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f126dcc6740 10 client.17142 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:26:33.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f126dcc6740 20 client.17142 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:33.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f126dcc6740 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561c76294960 0x561c76296d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f126dcc6740 4 client.17142 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:26:33.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.635+0000 7f126dcc6740 1 -- 172.21.15.33:0/3432754921 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c761ec7f0 con 0x561c76294960 2024-09-16T00:26:33.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.636+0000 7f1268e88640 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561c76294960 0x561c76296d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.636+0000 7f124cff9640 10 client.17142 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:33.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.636+0000 7f1268e88640 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561c76294960 0x561c76296d50 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0331af6f595a6dc server_cookie=badf679d569c58e7 in_seq=0 out_seq=0 2024-09-16T00:26:33.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f124cff9640 1 -- 172.21.15.33:0/3432754921 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561c761ec7f0 con 0x561c76294960 2024-09-16T00:26:33.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f124cff9640 10 client.17142 handle_command_reply: tid=0 2024-09-16T00:26:33.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f126dcc6740 1 client.17142 shutdown 2024-09-16T00:26:33.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f126dcc6740 1 -- 172.21.15.33:0/3432754921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 msgr2=0x561c762820e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:33.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f126dcc6740 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 0x561c762820e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f1258005cb0 tx=0x7f1258003ac0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f126dcc6740 1 -- 172.21.15.33:0/3432754921 shutdown_connections 2024-09-16T00:26:33.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f126dcc6740 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561c76294960 0x561c76296d50 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f126dcc6740 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c76282620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f126dcc6740 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 0x561c762820e0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f124cff9640 0 client.17142 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:33.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f126dcc6740 1 --2- 172.21.15.33:0/3432754921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c76279b60 0x561c76281ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f126dcc6740 1 -- 172.21.15.33:0/3432754921 >> 172.21.15.33:0/3432754921 conn(0x561c762760f0 msgr2=0x561c76275270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:33.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f126dcc6740 1 -- 172.21.15.33:0/3432754921 shutdown_connections 2024-09-16T00:26:33.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f126dcc6740 1 -- 172.21.15.33:0/3432754921 wait complete. 2024-09-16T00:26:33.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.637+0000 7f126dcc6740 20 client.17142 trim_cache size 0 max 16384 2024-09-16T00:26:33.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f126dcc6740 1 Processor -- start 2024-09-16T00:26:33.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f126dcc6740 1 -- start start 2024-09-16T00:26:33.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f126dcc6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 0x561c7627ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f126dcc6740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c7627a480 0x561c7627f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f126dcc6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c762832f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f126dcc6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f12641b1ab0 con 0x561c76279b60 2024-09-16T00:26:33.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f126dcc6740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f12641af380 con 0x561c7627a480 2024-09-16T00:26:33.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f126dcc6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f12641ad1d0 con 0x561c76284d40 2024-09-16T00:26:33.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f1268e88640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 0x561c7627ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f1268e88640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 0x561c7627ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34326/0 (socket says 172.21.15.33:34326) 2024-09-16T00:26:33.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f1268e88640 1 -- 172.21.15.33:0/1213669504 learned_addr learned my addr 172.21.15.33:0/1213669504 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:33.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f1269689640 1 --2- 172.21.15.33:0/1213669504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c7627a480 0x561c7627f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.655+0000 7f1268e88640 1 --2- 172.21.15.33:0/1213669504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c762832f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.656+0000 7f1269689640 1 -- 172.21.15.33:0/1213669504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 msgr2=0x561c762832f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:33.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.656+0000 7f1269689640 1 --2- 172.21.15.33:0/1213669504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c762832f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.656+0000 7f1269689640 1 -- 172.21.15.33:0/1213669504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 msgr2=0x561c7627ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:33.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.656+0000 7f1269689640 1 --2- 172.21.15.33:0/1213669504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 0x561c7627ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.656+0000 7f1269689640 1 -- 172.21.15.33:0/1213669504 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c76275f20 con 0x561c7627a480 2024-09-16T00:26:33.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.656+0000 7f1268e88640 1 --2- 172.21.15.33:0/1213669504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c762832f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:33.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.656+0000 7f1269689640 1 --2- 172.21.15.33:0/1213669504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c7627a480 0x561c7627f340 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1258019840 tx=0x7f125800b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa42025f7e4e49f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.656+0000 7f124effd640 1 -- 172.21.15.33:0/1213669504 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258038070 con 0x561c7627a480 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.656+0000 7f124effd640 1 -- 172.21.15.33:0/1213669504 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1258015050 con 0x561c7627a480 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.656+0000 7f124effd640 1 -- 172.21.15.33:0/1213669504 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258005030 con 0x561c7627a480 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.657+0000 7f126dcc6740 1 -- 172.21.15.33:0/1213669504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c7627a480 msgr2=0x561c7627f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.657+0000 7f126dcc6740 1 --2- 172.21.15.33:0/1213669504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c7627a480 0x561c7627f340 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1258019840 tx=0x7f125800b390 comp rx=0 tx=0).stop 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.657+0000 7f126dcc6740 1 -- 172.21.15.33:0/1213669504 shutdown_connections 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.657+0000 7f126dcc6740 1 --2- 172.21.15.33:0/1213669504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76284d40 0x561c762832f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.657+0000 7f126dcc6740 1 --2- 172.21.15.33:0/1213669504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c7627a480 0x561c7627f340 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.657+0000 7f126dcc6740 1 --2- 172.21.15.33:0/1213669504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c76279b60 0x561c7627ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.657+0000 7f126dcc6740 1 -- 172.21.15.33:0/1213669504 >> 172.21.15.33:0/1213669504 conn(0x561c762760f0 msgr2=0x561c76280c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.657+0000 7f126dcc6740 1 -- 172.21.15.33:0/1213669504 shutdown_connections 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.657+0000 7f126dcc6740 1 -- 172.21.15.33:0/1213669504 wait complete. 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.657+0000 7f126dcc6740 1 -- start start 2024-09-16T00:26:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f126dcc6740 1 Processor -- start 2024-09-16T00:26:33.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f126dcc6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76279b60 0x561c7629a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f126dcc6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 0x561c76297610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f126dcc6740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c76284d40 0x561c76297b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f126dcc6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f12641ad840 con 0x561c7627a480 2024-09-16T00:26:33.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f126dcc6740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f12641aecb0 con 0x561c76284d40 2024-09-16T00:26:33.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f126dcc6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f12641adf10 con 0x561c76279b60 2024-09-16T00:26:33.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f126dcc6740 10 client.0 _ll_register_callbacks cb 0x7f12600316d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:33.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f1269689640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 0x561c76297610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f1268e88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76279b60 0x561c7629a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f1268e88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76279b60 0x561c7629a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37464/0 (socket says 172.21.15.33:37464) 2024-09-16T00:26:33.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f1268e88640 1 -- 172.21.15.33:0/2259481250 learned_addr learned my addr 172.21.15.33:0/2259481250 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:33.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f1269689640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 0x561c76297610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34336/0 (socket says 172.21.15.33:34336) 2024-09-16T00:26:33.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f1268e88640 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c76284d40 0x561c76297b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f1268e88640 1 -- 172.21.15.33:0/2259481250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c76284d40 msgr2=0x561c76297b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:33.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f1268e88640 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c76284d40 0x561c76297b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f1268e88640 1 -- 172.21.15.33:0/2259481250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 msgr2=0x561c76297610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:33.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f1268e88640 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 0x561c76297610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f1268e88640 1 -- 172.21.15.33:0/2259481250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c76298110 con 0x561c76279b60 2024-09-16T00:26:33.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.658+0000 7f1269689640 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 0x561c76297610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:33.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.659+0000 7f1268e88640 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76279b60 0x561c7629a270 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1260009900 tx=0x7f1260040040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e24bc6a730929b88 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:33.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.659+0000 7f124cff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:26:33.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.659+0000 7f124cff9640 1 -- 172.21.15.33:0/2259481250 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1260051040 con 0x561c76279b60 2024-09-16T00:26:33.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.659+0000 7f124cff9640 1 -- 172.21.15.33:0/2259481250 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f126004fa50 con 0x561c76279b60 2024-09-16T00:26:33.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.659+0000 7f126dcc6740 1 -- 172.21.15.33:0/2259481250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1258015050 con 0x561c76279b60 2024-09-16T00:26:33.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.659+0000 7f124cff9640 1 -- 172.21.15.33:0/2259481250 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1260035040 con 0x561c76279b60 2024-09-16T00:26:33.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.659+0000 7f126dcc6740 10 client.26569 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:33.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.659+0000 7f126dcc6740 1 -- 172.21.15.33:0/2259481250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561c762ff070 con 0x561c76279b60 2024-09-16T00:26:33.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.660+0000 7f124cff9640 1 -- 172.21.15.33:0/2259481250 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1258015050 con 0x561c76279b60 2024-09-16T00:26:33.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.660+0000 7f124cff9640 1 -- 172.21.15.33:0/2259481250 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f1260057040 con 0x561c76279b60 2024-09-16T00:26:33.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.660+0000 7f126dcc6740 10 client.26569 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:33.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.660+0000 7f126dcc6740 10 client.26569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:26:33.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.660+0000 7f126dcc6740 20 client.26569 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:33.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.660+0000 7f126dcc6740 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561c762ff7e0 0x561c76301c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:33.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.660+0000 7f126dcc6740 4 client.26569 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:26:33.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.660+0000 7f126dcc6740 1 -- 172.21.15.33:0/2259481250 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561c761e6d00 con 0x561c762ff7e0 2024-09-16T00:26:33.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.660+0000 7f1269689640 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561c762ff7e0 0x561c76301c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:33.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.661+0000 7f124cff9640 10 client.26569 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:33.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.661+0000 7f1269689640 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561c762ff7e0 0x561c76301c00 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f3727581f15c5e2 server_cookie=b2a58f595c3900cf in_seq=0 out_seq=0 2024-09-16T00:26:33.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f124cff9640 1 -- 172.21.15.33:0/2259481250 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561c761e6d00 con 0x561c762ff7e0 2024-09-16T00:26:33.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f124cff9640 10 client.26569 handle_command_reply: tid=0 2024-09-16T00:26:33.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f126dcc6740 1 client.26569 shutdown 2024-09-16T00:26:33.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f126dcc6740 1 -- 172.21.15.33:0/2259481250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76279b60 msgr2=0x561c7629a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:33.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f126dcc6740 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76279b60 0x561c7629a270 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1260009900 tx=0x7f1260040040 comp rx=0 tx=0).stop 2024-09-16T00:26:33.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f126dcc6740 1 -- 172.21.15.33:0/2259481250 shutdown_connections 2024-09-16T00:26:33.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f126dcc6740 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561c762ff7e0 0x561c76301c00 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f126dcc6740 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561c76284d40 0x561c76297b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f126dcc6740 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561c7627a480 0x561c76297610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f126dcc6740 1 --2- 172.21.15.33:0/2259481250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c76279b60 0x561c7629a270 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f126dcc6740 1 -- 172.21.15.33:0/2259481250 >> 172.21.15.33:0/2259481250 conn(0x561c762760f0 msgr2=0x561c76274a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:33.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f124cff9640 0 client.26569 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:33.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f126dcc6740 1 -- 172.21.15.33:0/2259481250 shutdown_connections 2024-09-16T00:26:33.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.662+0000 7f126dcc6740 1 -- 172.21.15.33:0/2259481250 wait complete. 2024-09-16T00:26:33.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.663+0000 7f126dcc6740 20 client.26569 trim_cache size 0 max 16384 2024-09-16T00:26:33.676 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:26:33.677 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:26:33.677 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "6f95a3b6-ce5b-4e58-8c2d-0a5d9c7a385f", 2024-09-16T00:26:33.677 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:26:33.677 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:26:33.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.663+0000 7f126b8ed640 1 -- 172.21.15.33:0/3619962289 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12500a3240 msgr2=0x7f12500a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:33.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.663+0000 7f126b8ed640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12500a3240 0x7f12500a5700 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f125c0983c0 tx=0x7f125c091440 comp rx=0 tx=0).stop 2024-09-16T00:26:33.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.663+0000 7f126b8ed640 1 -- 172.21.15.33:0/3619962289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1264108290 msgr2=0x7f126419cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:33.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.663+0000 7f126b8ed640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1264108290 0x7f126419cee0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f125c00cad0 tx=0x7f125c00cf90 comp rx=0 tx=0).stop 2024-09-16T00:26:33.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.663+0000 7f126b8ed640 1 -- 172.21.15.33:0/3619962289 shutdown_connections 2024-09-16T00:26:33.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.663+0000 7f126b8ed640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f12500a3240 0x7f12500a5700 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.663+0000 7f126b8ed640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1264108290 0x7f126419cee0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.663+0000 7f126b8ed640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1264103740 0x7f1264198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.663+0000 7f126b8ed640 1 --2- 172.21.15.33:0/3619962289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1264102d90 0x7f1264198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.663+0000 7f126b8ed640 1 -- 172.21.15.33:0/3619962289 >> 172.21.15.33:0/3619962289 conn(0x7f12640fc7d0 msgr2=0x7f1264105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.664+0000 7f126b8ed640 1 -- 172.21.15.33:0/3619962289 shutdown_connections 2024-09-16T00:26:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:33.664+0000 7f126b8ed640 1 -- 172.21.15.33:0/3619962289 wait complete. 2024-09-16T00:26:33.680 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 10.01 MB/sec execute 341 sec latency 21938.884 ms 2024-09-16T00:26:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:33 smithi083 ceph-mon[35242]: pgmap v632: 129 pgs: 129 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 2.2 MiB/s wr, 90 op/s 2024-09-16T00:26:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:33 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2329161562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:33 smithi060 ceph-mon[35185]: pgmap v632: 129 pgs: 129 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 2.2 MiB/s wr, 90 op/s 2024-09-16T00:26:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:33 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2329161562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:34.462 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:34.490 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:26:34.673 INFO:tasks.workunit.client.0.smithi033.stdout: 1 748600 9.98 MB/sec execute 342 sec latency 22938.934 ms 2024-09-16T00:26:34.771 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:34 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:26:34.771 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:34 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:26:34.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:34 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:26:34.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:34 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:26:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:34 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:26:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:34 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:26:35.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.021+0000 7f58dfc96640 1 -- 172.21.15.33:0/3100664242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58d8102570 msgr2=0x7f58d81029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:35.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.021+0000 7f58dfc96640 1 --2- 172.21.15.33:0/3100664242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58d8102570 0x7f58d81029b0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009900 tx=0x7f58cc02a480 comp rx=0 tx=0).stop 2024-09-16T00:26:35.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.021+0000 7f58dfc96640 1 -- 172.21.15.33:0/3100664242 shutdown_connections 2024-09-16T00:26:35.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.021+0000 7f58dfc96640 1 --2- 172.21.15.33:0/3100664242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58d8102ef0 0x7f58d810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.021+0000 7f58dfc96640 1 --2- 172.21.15.33:0/3100664242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58d8102570 0x7f58d81029b0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.021+0000 7f58dfc96640 1 --2- 172.21.15.33:0/3100664242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58d8100e20 0x7f58d8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.021+0000 7f58dfc96640 1 -- 172.21.15.33:0/3100664242 >> 172.21.15.33:0/3100664242 conn(0x7f58d80fc7d0 msgr2=0x7f58d80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:35.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.021+0000 7f58dfc96640 1 -- 172.21.15.33:0/3100664242 shutdown_connections 2024-09-16T00:26:35.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.021+0000 7f58dfc96640 1 -- 172.21.15.33:0/3100664242 wait complete. 2024-09-16T00:26:35.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.021+0000 7f58dfc96640 1 Processor -- start 2024-09-16T00:26:35.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dfc96640 1 -- start start 2024-09-16T00:26:35.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dfc96640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58d8100e20 0x7f58d80fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dfc96640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58d8102570 0x7f58d80fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dfc96640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58d8102ef0 0x7f58d80fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dfc96640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58d8110f50 con 0x7f58d8100e20 2024-09-16T00:26:35.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dfc96640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f58d8110dd0 con 0x7f58d8102ef0 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dfc96640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81110d0 con 0x7f58d8102570 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dda32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58d8100e20 0x7f58d80fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dda32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58d8100e20 0x7f58d80fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34352/0 (socket says 172.21.15.33:34352) 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dda32640 1 -- 172.21.15.33:0/1222350471 learned_addr learned my addr 172.21.15.33:0/1222350471 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58de233640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58d8102ef0 0x7f58d80fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dd231640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58d8102570 0x7f58d80fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dda32640 1 -- 172.21.15.33:0/1222350471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58d8102570 msgr2=0x7f58d80fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dda32640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58d8102570 0x7f58d80fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dda32640 1 -- 172.21.15.33:0/1222350471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58d8102ef0 msgr2=0x7f58d80fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dda32640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58d8102ef0 0x7f58d80fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.022+0000 7f58dda32640 1 -- 172.21.15.33:0/1222350471 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d81a6a10 con 0x7f58d8100e20 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.023+0000 7f58dda32640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58d8100e20 0x7f58d80fcc80 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f58c800cce0 tx=0x7f58c800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.023+0000 7f58d6ffd640 1 -- 172.21.15.33:0/1222350471 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c8004b20 con 0x7f58d8100e20 2024-09-16T00:26:35.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.023+0000 7f58d6ffd640 1 -- 172.21.15.33:0/1222350471 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58c8004cc0 con 0x7f58d8100e20 2024-09-16T00:26:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.023+0000 7f58dfc96640 1 -- 172.21.15.33:0/1222350471 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d81a6d00 con 0x7f58d8100e20 2024-09-16T00:26:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.023+0000 7f58d6ffd640 1 -- 172.21.15.33:0/1222350471 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c800f7d0 con 0x7f58d8100e20 2024-09-16T00:26:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.023+0000 7f58dfc96640 1 -- 172.21.15.33:0/1222350471 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d81a7200 con 0x7f58d8100e20 2024-09-16T00:26:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58e206f740 1 Processor -- start 2024-09-16T00:26:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58e206f740 1 -- start start 2024-09-16T00:26:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58e206f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604060 0x561239604440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58e206f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604980 0x56123960ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58e206f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 0x561239611630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58e206f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561239569620 con 0x56123960f240 2024-09-16T00:26:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58e206f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561239510e50 con 0x561239604060 2024-09-16T00:26:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58e206f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5612395655e0 con 0x561239604980 2024-09-16T00:26:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58dd231640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 0x561239611630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58dd231640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 0x561239611630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34362/0 (socket says 172.21.15.33:34362) 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58dd231640 1 -- 172.21.15.33:0/1131267948 learned_addr learned my addr 172.21.15.33:0/1131267948 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58d6ffd640 1 -- 172.21.15.33:0/1222350471 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f58c800f970 con 0x7f58d8100e20 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.026+0000 7f58de233640 1 --2- 172.21.15.33:0/1131267948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604060 0x561239604440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58dda32640 1 --2- 172.21.15.33:0/1131267948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604980 0x56123960ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58dd231640 1 -- 172.21.15.33:0/1131267948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604980 msgr2=0x56123960ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58dd231640 1 --2- 172.21.15.33:0/1131267948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604980 0x56123960ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58dd231640 1 -- 172.21.15.33:0/1131267948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604060 msgr2=0x561239604440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58dd231640 1 --2- 172.21.15.33:0/1131267948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604060 0x561239604440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58dd231640 1 -- 172.21.15.33:0/1131267948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561239501d40 con 0x56123960f240 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58dda32640 1 --2- 172.21.15.33:0/1131267948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604980 0x56123960ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58dd231640 1 --2- 172.21.15.33:0/1131267948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 0x561239611630 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009900 tx=0x7f58cc004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5458d98d5903a885 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:35.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58bffff640 1 -- 172.21.15.33:0/1131267948 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc035030 con 0x56123960f240 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58d6ffd640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58c00a32c0 0x7f58c00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58bffff640 1 -- 172.21.15.33:0/1131267948 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58cc040070 con 0x56123960f240 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58de233640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58c00a32c0 0x7f58c00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.027+0000 7f58bffff640 1 -- 172.21.15.33:0/1131267948 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc031ab0 con 0x56123960f240 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.028+0000 7f58d6ffd640 1 -- 172.21.15.33:0/1222350471 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f58c80cfd40 con 0x7f58d8100e20 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.028+0000 7f58de233640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58c00a32c0 0x7f58c00a5780 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f58d000b650 tx=0x7f58d000b2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.028+0000 7f58e206f740 1 -- 172.21.15.33:0/1131267948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 msgr2=0x561239611630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.029+0000 7f58e206f740 1 --2- 172.21.15.33:0/1131267948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 0x561239611630 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009900 tx=0x7f58cc004170 comp rx=0 tx=0).stop 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.029+0000 7f58e206f740 1 -- 172.21.15.33:0/1131267948 shutdown_connections 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.029+0000 7f58e206f740 1 --2- 172.21.15.33:0/1131267948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 0x561239611630 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.029+0000 7f58e206f740 1 --2- 172.21.15.33:0/1131267948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604980 0x56123960ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.029+0000 7f58e206f740 1 --2- 172.21.15.33:0/1131267948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604060 0x561239604440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.029+0000 7f58e206f740 1 -- 172.21.15.33:0/1131267948 >> 172.21.15.33:0/1131267948 conn(0x5612396005f0 msgr2=0x5612396009d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.029+0000 7f58e206f740 1 -- 172.21.15.33:0/1131267948 shutdown_connections 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.029+0000 7f58e206f740 1 -- 172.21.15.33:0/1131267948 wait complete. 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.029+0000 7f58e206f740 1 -- start start 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.029+0000 7f58e206f740 1 Processor -- start 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58e206f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 0x56123960c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58e206f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604980 0x56123960c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58e206f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56123960f240 0x56123960cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58e206f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561239611e10 con 0x561239604060 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58e206f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5612394f1f10 con 0x56123960f240 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58e206f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561239611f90 con 0x561239604980 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58dda32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 0x56123960c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58dda32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 0x56123960c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34376/0 (socket says 172.21.15.33:34376) 2024-09-16T00:26:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58dda32640 1 -- 172.21.15.33:0/1726194961 learned_addr learned my addr 172.21.15.33:0/1726194961 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58e206f740 10 client.0 _ll_register_callbacks cb 0x561239515db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58de233640 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56123960f240 0x56123960cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58dd231640 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604980 0x56123960c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58dda32640 1 -- 172.21.15.33:0/1726194961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604980 msgr2=0x56123960c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58dda32640 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604980 0x56123960c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58dda32640 1 -- 172.21.15.33:0/1726194961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56123960f240 msgr2=0x56123960cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58dda32640 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56123960f240 0x56123960cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58dda32640 1 -- 172.21.15.33:0/1726194961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561239278580 con 0x561239604060 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58dd231640 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604980 0x56123960c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58dda32640 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 0x56123960c0c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f58c8005c40 tx=0x7f58c809be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c266c364c7d96f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58bd7fa640 1 -- 172.21.15.33:0/1726194961 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c801d070 con 0x561239604060 2024-09-16T00:26:35.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58bd7fa640 1 -- 172.21.15.33:0/1726194961 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f58c800fc60 con 0x561239604060 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.030+0000 7f58e206f740 1 -- 172.21.15.33:0/1726194961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58cc040070 con 0x561239604060 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.031+0000 7f58bd7fa640 1 -- 172.21.15.33:0/1726194961 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f58c800fe60 con 0x561239604060 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.031+0000 7f58e206f740 10 client.17160 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.031+0000 7f58e206f740 1 -- 172.21.15.33:0/1726194961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561239342960 con 0x561239604060 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.031+0000 7f58bd7fa640 1 -- 172.21.15.33:0/1726194961 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58cc040070 con 0x561239604060 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.031+0000 7f58bd7fa640 1 -- 172.21.15.33:0/1726194961 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f58c809b3e0 con 0x561239604060 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.031+0000 7f58e206f740 10 client.17160 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.031+0000 7f58e206f740 10 client.17160 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.031+0000 7f58e206f740 20 client.17160 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.031+0000 7f58e206f740 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56123961ec80 0x5612396210f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.031+0000 7f58e206f740 4 client.17160 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.031+0000 7f58e206f740 1 -- 172.21.15.33:0/1726194961 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561239576b90 con 0x56123961ec80 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.031+0000 7f58dd231640 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56123961ec80 0x5612396210f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.032+0000 7f58dd231640 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56123961ec80 0x5612396210f0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6cd00b6dea2f4a9 server_cookie=f7a17b9cb6f2e287 in_seq=0 out_seq=0 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.032+0000 7f58bd7fa640 10 client.17160 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58bd7fa640 1 -- 172.21.15.33:0/1726194961 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561239576b90 con 0x56123961ec80 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58bd7fa640 10 client.17160 handle_command_reply: tid=0 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58e206f740 1 client.17160 shutdown 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58e206f740 1 -- 172.21.15.33:0/1726194961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 msgr2=0x56123960c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58e206f740 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 0x56123960c0c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f58c8005c40 tx=0x7f58c809be90 comp rx=0 tx=0).stop 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58e206f740 1 -- 172.21.15.33:0/1726194961 shutdown_connections 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58e206f740 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56123961ec80 0x5612396210f0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58e206f740 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56123960f240 0x56123960cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58e206f740 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604980 0x56123960c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58e206f740 1 --2- 172.21.15.33:0/1726194961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 0x56123960c0c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58bd7fa640 0 client.17160 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58e206f740 1 -- 172.21.15.33:0/1726194961 >> 172.21.15.33:0/1726194961 conn(0x5612396005f0 msgr2=0x5612396062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58e206f740 1 -- 172.21.15.33:0/1726194961 shutdown_connections 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58e206f740 1 -- 172.21.15.33:0/1726194961 wait complete. 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.033+0000 7f58e206f740 20 client.17160 trim_cache size 0 max 16384 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.050+0000 7f58e206f740 1 Processor -- start 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.050+0000 7f58e206f740 1 -- start start 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.050+0000 7f58e206f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604060 0x561239600c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.050+0000 7f58e206f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604980 0x5612396269c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.050+0000 7f58e206f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 0x561239626f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.050+0000 7f58e206f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81b3b80 con 0x56123960f240 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.050+0000 7f58e206f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81b1450 con 0x561239604980 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.050+0000 7f58e206f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81af2a0 con 0x561239604060 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.050+0000 7f58de233640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 0x561239626f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.050+0000 7f58de233640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 0x561239626f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34380/0 (socket says 172.21.15.33:34380) 2024-09-16T00:26:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.050+0000 7f58de233640 1 -- 172.21.15.33:0/213402554 learned_addr learned my addr 172.21.15.33:0/213402554 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.051+0000 7f58dda32640 1 --2- 172.21.15.33:0/213402554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604060 0x561239600c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.051+0000 7f58dd231640 1 --2- 172.21.15.33:0/213402554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604980 0x5612396269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.051+0000 7f58de233640 1 -- 172.21.15.33:0/213402554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604060 msgr2=0x561239600c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.051+0000 7f58de233640 1 --2- 172.21.15.33:0/213402554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604060 0x561239600c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.051+0000 7f58de233640 1 -- 172.21.15.33:0/213402554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604980 msgr2=0x5612396269c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.051+0000 7f58de233640 1 --2- 172.21.15.33:0/213402554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604980 0x5612396269c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.051+0000 7f58de233640 1 -- 172.21.15.33:0/213402554 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56123960ea30 con 0x56123960f240 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.051+0000 7f58de233640 1 --2- 172.21.15.33:0/213402554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 0x561239626f00 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f58d0003c60 tx=0x7f58d0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ae8783ebcf1fba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.051+0000 7f58bf7fe640 1 -- 172.21.15.33:0/213402554 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0012d50 con 0x56123960f240 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.051+0000 7f58bf7fe640 1 -- 172.21.15.33:0/213402554 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58d0004c80 con 0x56123960f240 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.051+0000 7f58bf7fe640 1 -- 172.21.15.33:0/213402554 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0005710 con 0x56123960f240 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.052+0000 7f58e206f740 1 -- 172.21.15.33:0/213402554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 msgr2=0x561239626f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.052+0000 7f58e206f740 1 --2- 172.21.15.33:0/213402554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 0x561239626f00 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f58d0003c60 tx=0x7f58d0010000 comp rx=0 tx=0).stop 2024-09-16T00:26:35.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.052+0000 7f58e206f740 1 -- 172.21.15.33:0/213402554 shutdown_connections 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.052+0000 7f58e206f740 1 --2- 172.21.15.33:0/213402554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56123960f240 0x561239626f00 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.052+0000 7f58e206f740 1 --2- 172.21.15.33:0/213402554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604980 0x5612396269c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.052+0000 7f58e206f740 1 --2- 172.21.15.33:0/213402554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561239604060 0x561239600c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.052+0000 7f58e206f740 1 -- 172.21.15.33:0/213402554 >> 172.21.15.33:0/213402554 conn(0x5612396005f0 msgr2=0x56123960e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.052+0000 7f58e206f740 1 -- 172.21.15.33:0/213402554 shutdown_connections 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.052+0000 7f58e206f740 1 -- 172.21.15.33:0/213402554 wait complete. 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.052+0000 7f58e206f740 1 -- start start 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.052+0000 7f58e206f740 1 Processor -- start 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58e206f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 0x561239617210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58e206f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604980 0x561239617750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58e206f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56123960f240 0x5612396141e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58e206f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81af910 con 0x561239604060 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58e206f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81b0d80 con 0x561239604980 2024-09-16T00:26:35.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58e206f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81affe0 con 0x56123960f240 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58dda32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 0x561239617210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58e206f740 10 client.0 _ll_register_callbacks cb 0x7f58c80982c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58dda32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 0x561239617210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34386/0 (socket says 172.21.15.33:34386) 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58dda32640 1 -- 172.21.15.33:0/83281658 learned_addr learned my addr 172.21.15.33:0/83281658 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58de233640 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56123960f240 0x5612396141e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58dd231640 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604980 0x561239617750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58dda32640 1 -- 172.21.15.33:0/83281658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56123960f240 msgr2=0x5612396141e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58dda32640 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56123960f240 0x5612396141e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58dda32640 1 -- 172.21.15.33:0/83281658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604980 msgr2=0x561239617750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58dda32640 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604980 0x561239617750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.053+0000 7f58dda32640 1 -- 172.21.15.33:0/83281658 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612396147d0 con 0x561239604060 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.054+0000 7f58dda32640 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 0x561239617210 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f58c8000f80 tx=0x7f58c807adf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f0f898eee1e954 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:35.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.054+0000 7f58bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.054+0000 7f58bd7fa640 1 -- 172.21.15.33:0/83281658 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c801d040 con 0x561239604060 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.054+0000 7f58bd7fa640 1 -- 172.21.15.33:0/83281658 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f58c80bab20 con 0x561239604060 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.054+0000 7f58e206f740 1 -- 172.21.15.33:0/83281658 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58d0004c80 con 0x561239604060 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.054+0000 7f58bd7fa640 1 -- 172.21.15.33:0/83281658 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f58c809b8f0 con 0x561239604060 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.054+0000 7f58e206f740 10 client.17166 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.054+0000 7f58e206f740 1 -- 172.21.15.33:0/83281658 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56123968a2c0 con 0x561239604060 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.054+0000 7f58bd7fa640 1 -- 172.21.15.33:0/83281658 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58d0004c80 con 0x561239604060 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.055+0000 7f58bd7fa640 1 -- 172.21.15.33:0/83281658 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f58c809ae50 con 0x561239604060 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.055+0000 7f58e206f740 10 client.17166 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.055+0000 7f58e206f740 10 client.17166 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.055+0000 7f58e206f740 20 client.17166 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.055+0000 7f58e206f740 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56123968a990 0x56123968cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.055+0000 7f58e206f740 4 client.17166 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "de406742", "release": true, "await": true}] 2024-09-16T00:26:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.055+0000 7f58e206f740 1 -- 172.21.15.33:0/83281658 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "de406742", "release": true, "await": true}) -- 0x5612395710a0 con 0x56123968a990 2024-09-16T00:26:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.056+0000 7f58dd231640 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56123968a990 0x56123968cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.056+0000 7f58dd231640 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56123968a990 0x56123968cd80 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94b4bb0877f5ce09 server_cookie=7338eeb9f09fee33 in_seq=0 out_seq=0 2024-09-16T00:26:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.056+0000 7f58bd7fa640 10 client.17166 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58bd7fa640 1 -- 172.21.15.33:0/83281658 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5612395710a0 con 0x56123968a990 2024-09-16T00:26:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58bd7fa640 10 client.17166 handle_command_reply: tid=0 2024-09-16T00:26:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58e206f740 1 client.17166 shutdown 2024-09-16T00:26:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58e206f740 1 -- 172.21.15.33:0/83281658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 msgr2=0x561239617210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58e206f740 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 0x561239617210 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f58c8000f80 tx=0x7f58c807adf0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58e206f740 1 -- 172.21.15.33:0/83281658 shutdown_connections 2024-09-16T00:26:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58e206f740 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56123968a990 0x56123968cd80 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58e206f740 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56123960f240 0x5612396141e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58bd7fa640 0 client.17166 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58e206f740 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561239604980 0x561239617750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58e206f740 1 --2- 172.21.15.33:0/83281658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561239604060 0x561239617210 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58e206f740 1 -- 172.21.15.33:0/83281658 >> 172.21.15.33:0/83281658 conn(0x5612396005f0 msgr2=0x56123960c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.064+0000 7f58e206f740 1 -- 172.21.15.33:0/83281658 shutdown_connections 2024-09-16T00:26:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.065+0000 7f58e206f740 1 -- 172.21.15.33:0/83281658 wait complete. 2024-09-16T00:26:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.065+0000 7f58e206f740 20 client.17166 trim_cache size 0 max 16384 2024-09-16T00:26:35.072 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:26:35.072 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:26:35.072 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:26:35.072 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 32, 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: "de406742": { 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 32, 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 142.3, 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:26:35.073 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:26:35.074 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:26:35.074 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:26:35.074 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:26:35.074 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:35.074 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:35.074 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:35.074 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:35.074 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:35.074 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:26:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.065+0000 7f58dfc96640 1 -- 172.21.15.33:0/1222350471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58c00a32c0 msgr2=0x7f58c00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.065+0000 7f58dfc96640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58c00a32c0 0x7f58c00a5780 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f58d000b650 tx=0x7f58d000b2e0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.065+0000 7f58dfc96640 1 -- 172.21.15.33:0/1222350471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58d8100e20 msgr2=0x7f58d80fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.065+0000 7f58dfc96640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58d8100e20 0x7f58d80fcc80 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f58c800cce0 tx=0x7f58c800a620 comp rx=0 tx=0).stop 2024-09-16T00:26:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.066+0000 7f58dfc96640 1 -- 172.21.15.33:0/1222350471 shutdown_connections 2024-09-16T00:26:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.066+0000 7f58dfc96640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58c00a32c0 0x7f58c00a5780 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.066+0000 7f58dfc96640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58d8102ef0 0x7f58d80fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.066+0000 7f58dfc96640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58d8102570 0x7f58d80fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.066+0000 7f58dfc96640 1 --2- 172.21.15.33:0/1222350471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58d8100e20 0x7f58d80fcc80 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.066+0000 7f58dfc96640 1 -- 172.21.15.33:0/1222350471 >> 172.21.15.33:0/1222350471 conn(0x7f58d80fc7d0 msgr2=0x7f58d806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.067+0000 7f58dfc96640 1 -- 172.21.15.33:0/1222350471 shutdown_connections 2024-09-16T00:26:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:35.067+0000 7f58dfc96640 1 -- 172.21.15.33:0/1222350471 wait complete. 2024-09-16T00:26:35.328 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:35.672 INFO:tasks.workunit.client.0.smithi033.stdout: 1 749839 9.97 MB/sec execute 343 sec latency 23331.973 ms 2024-09-16T00:26:35.768 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:35 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:26:35.768 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:26:35.769 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:35 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:26:35.769 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:26:35.769 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:35 smithi033 ceph-mon[31243]: pgmap v633: 129 pgs: 129 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 2.1 MiB/s wr, 85 op/s 2024-09-16T00:26:35.809 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'de406742', total seconds elapsed: 26.7 2024-09-16T00:26:35.809 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.35 2024-09-16T00:26:35.809 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:26:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:35 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:26:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:26:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:35 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:26:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:26:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:35 smithi083 ceph-mon[35242]: pgmap v633: 129 pgs: 129 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 2.1 MiB/s wr, 85 op/s 2024-09-16T00:26:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:35 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:26:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:26:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:35 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:26:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:26:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:35 smithi060 ceph-mon[35185]: pgmap v633: 129 pgs: 129 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 2.1 MiB/s wr, 85 op/s 2024-09-16T00:26:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.278+0000 7f618ac61640 1 -- 172.21.15.33:0/435107976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61841027a0 msgr2=0x7f61841067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.279+0000 7f618ac61640 1 --2- 172.21.15.33:0/435107976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61841027a0 0x7f61841067f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6178009900 tx=0x7f617802a410 comp rx=0 tx=0).stop 2024-09-16T00:26:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.279+0000 7f618ac61640 1 -- 172.21.15.33:0/435107976 shutdown_connections 2024-09-16T00:26:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.279+0000 7f618ac61640 1 --2- 172.21.15.33:0/435107976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6184106f20 0x7f618410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.279+0000 7f618ac61640 1 --2- 172.21.15.33:0/435107976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61841027a0 0x7f61841067f0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.279+0000 7f618ac61640 1 --2- 172.21.15.33:0/435107976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184101df0 0x7f61841021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.279+0000 7f618ac61640 1 -- 172.21.15.33:0/435107976 >> 172.21.15.33:0/435107976 conn(0x7f61840fd8f0 msgr2=0x7f61840ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.279+0000 7f618ac61640 1 -- 172.21.15.33:0/435107976 shutdown_connections 2024-09-16T00:26:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.279+0000 7f618ac61640 1 -- 172.21.15.33:0/435107976 wait complete. 2024-09-16T00:26:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.279+0000 7f618ac61640 1 Processor -- start 2024-09-16T00:26:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.279+0000 7f618ac61640 1 -- start start 2024-09-16T00:26:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f618ac61640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6184101df0 0x7f61840744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f618ac61640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61841027a0 0x7f6184074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f618ac61640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184106f20 0x7f6184074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f618ac61640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6184111fd0 con 0x7f6184101df0 2024-09-16T00:26:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f618ac61640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6184111e50 con 0x7f61841027a0 2024-09-16T00:26:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f618ac61640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6184112150 con 0x7f6184106f20 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f61889fd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6184101df0 0x7f61840744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f61889fd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6184101df0 0x7f61840744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34404/0 (socket says 172.21.15.33:34404) 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f61889fd640 1 -- 172.21.15.33:0/179208036 learned_addr learned my addr 172.21.15.33:0/179208036 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f6183fff640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61841027a0 0x7f6184074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f61891fe640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184106f20 0x7f6184074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f61889fd640 1 -- 172.21.15.33:0/179208036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184106f20 msgr2=0x7f6184074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f61889fd640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184106f20 0x7f6184074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f61889fd640 1 -- 172.21.15.33:0/179208036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61841027a0 msgr2=0x7f6184074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f61889fd640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61841027a0 0x7f6184074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f61889fd640 1 -- 172.21.15.33:0/179208036 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61841a6950 con 0x7f6184101df0 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f6183fff640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61841027a0 0x7f6184074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f61891fe640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184106f20 0x7f6184074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f61889fd640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6184101df0 0x7f61840744f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f617400eb90 tx=0x7f617400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.280+0000 7f6181ffb640 1 -- 172.21.15.33:0/179208036 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174004d20 con 0x7f6184101df0 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.281+0000 7f618ac61640 1 -- 172.21.15.33:0/179208036 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61841a6c40 con 0x7f6184101df0 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.281+0000 7f6181ffb640 1 -- 172.21.15.33:0/179208036 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f617400ee90 con 0x7f6184101df0 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.281+0000 7f6181ffb640 1 -- 172.21.15.33:0/179208036 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174010830 con 0x7f6184101df0 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.281+0000 7f618ac61640 1 -- 172.21.15.33:0/179208036 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61841a7150 con 0x7f6184101df0 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.283+0000 7f6181ffb640 1 -- 172.21.15.33:0/179208036 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6174005020 con 0x7f6184101df0 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f618d03a740 1 Processor -- start 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f618d03a740 1 -- start start 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f6181ffb640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f616c0a3240 0x7f616c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f618d03a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea7554a0 0x5629ea755880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f618d03a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea755dc0 0x5629ea760140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f618d03a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea760680 0x5629ea762a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f618d03a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5629ea6bb2e0 con 0x5629ea760680 2024-09-16T00:26:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f618d03a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5629ea662ce0 con 0x5629ea7554a0 2024-09-16T00:26:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f618d03a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5629ea6b72a0 con 0x5629ea755dc0 2024-09-16T00:26:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f61889fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea7554a0 0x5629ea755880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f61891fe640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f616c0a3240 0x7f616c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f61889fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea7554a0 0x5629ea755880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:48270/0 (socket says 172.21.15.33:48270) 2024-09-16T00:26:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f61889fd640 1 -- 172.21.15.33:0/867603295 learned_addr learned my addr 172.21.15.33:0/867603295 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f6183fff640 1 --2- 172.21.15.33:0/867603295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea755dc0 0x5629ea760140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f61891fe640 1 --2- 172.21.15.33:0/867603295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea760680 0x5629ea762a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.284+0000 7f6181ffb640 1 -- 172.21.15.33:0/179208036 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f61740d03c0 con 0x7f6184101df0 2024-09-16T00:26:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.285+0000 7f6183fff640 1 -- 172.21.15.33:0/867603295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea7554a0 msgr2=0x5629ea755880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.285+0000 7f6183fff640 1 --2- 172.21.15.33:0/867603295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea7554a0 0x5629ea755880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.285+0000 7f6183fff640 1 -- 172.21.15.33:0/867603295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea760680 msgr2=0x5629ea762a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.285+0000 7f6183fff640 1 --2- 172.21.15.33:0/867603295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea760680 0x5629ea762a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.285+0000 7f6183fff640 1 -- 172.21.15.33:0/867603295 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629ea647440 con 0x5629ea755dc0 2024-09-16T00:26:36.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.285+0000 7f61889fd640 1 --2- 172.21.15.33:0/867603295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea7554a0 0x5629ea755880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:36.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.285+0000 7f61891fe640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f616c0a3240 0x7f616c0a5700 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f617c005f60 tx=0x7f617c009ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:36.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.285+0000 7f6183fff640 1 --2- 172.21.15.33:0/867603295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea755dc0 0x5629ea760140 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f617802aa00 tx=0x7f6178005de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58434d43f400897 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:36.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.285+0000 7f616affd640 1 -- 172.21.15.33:0/867603295 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178035040 con 0x5629ea755dc0 2024-09-16T00:26:36.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.285+0000 7f616affd640 1 -- 172.21.15.33:0/867603295 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6178005520 con 0x5629ea755dc0 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.285+0000 7f616affd640 1 -- 172.21.15.33:0/867603295 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178003850 con 0x5629ea755dc0 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.286+0000 7f618d03a740 1 -- 172.21.15.33:0/867603295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea755dc0 msgr2=0x5629ea760140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.286+0000 7f618d03a740 1 --2- 172.21.15.33:0/867603295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea755dc0 0x5629ea760140 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f617802aa00 tx=0x7f6178005de0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.286+0000 7f618d03a740 1 -- 172.21.15.33:0/867603295 shutdown_connections 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.286+0000 7f618d03a740 1 --2- 172.21.15.33:0/867603295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea760680 0x5629ea762a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.286+0000 7f618d03a740 1 --2- 172.21.15.33:0/867603295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea755dc0 0x5629ea760140 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.286+0000 7f618d03a740 1 --2- 172.21.15.33:0/867603295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea7554a0 0x5629ea755880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.286+0000 7f618d03a740 1 -- 172.21.15.33:0/867603295 >> 172.21.15.33:0/867603295 conn(0x5629ea751a30 msgr2=0x5629ea751e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.286+0000 7f618d03a740 1 -- 172.21.15.33:0/867603295 shutdown_connections 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.286+0000 7f618d03a740 1 -- 172.21.15.33:0/867603295 wait complete. 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.286+0000 7f618d03a740 1 -- start start 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.286+0000 7f618d03a740 1 Processor -- start 2024-09-16T00:26:36.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f618d03a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea7554a0 0x5629ea75ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f618d03a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea755dc0 0x5629ea75d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f618d03a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea760680 0x5629ea75d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f618d03a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5629ea763250 con 0x5629ea760680 2024-09-16T00:26:36.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f618d03a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5629ea676050 con 0x5629ea755dc0 2024-09-16T00:26:36.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f618d03a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5629ea7633d0 con 0x5629ea7554a0 2024-09-16T00:26:36.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f618d03a740 10 client.0 _ll_register_callbacks cb 0x5629ea6b6160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:36.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f61891fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea760680 0x5629ea75d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f61889fd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea7554a0 0x5629ea75ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f61889fd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea7554a0 0x5629ea75ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37542/0 (socket says 172.21.15.33:37542) 2024-09-16T00:26:36.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f61889fd640 1 -- 172.21.15.33:0/3221960222 learned_addr learned my addr 172.21.15.33:0/3221960222 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:36.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f61891fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea760680 0x5629ea75d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34422/0 (socket says 172.21.15.33:34422) 2024-09-16T00:26:36.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f6183fff640 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea755dc0 0x5629ea75d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f61889fd640 1 -- 172.21.15.33:0/3221960222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea755dc0 msgr2=0x5629ea75d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:36.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f61889fd640 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea755dc0 0x5629ea75d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f61889fd640 1 -- 172.21.15.33:0/3221960222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea760680 msgr2=0x5629ea75d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:36.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f61889fd640 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea760680 0x5629ea75d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f61889fd640 1 -- 172.21.15.33:0/3221960222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629ea42c3c0 con 0x5629ea7554a0 2024-09-16T00:26:36.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.287+0000 7f61891fe640 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea760680 0x5629ea75d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:36.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f61889fd640 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea7554a0 0x5629ea75ff70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f617400cba0 tx=0x7f617406b2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=897505f79de3b895 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:36.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f6163fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:26:36.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f6163fff640 1 -- 172.21.15.33:0/3221960222 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617409c0a0 con 0x5629ea7554a0 2024-09-16T00:26:36.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f6163fff640 1 -- 172.21.15.33:0/3221960222 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f617409a860 con 0x5629ea7554a0 2024-09-16T00:26:36.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f618d03a740 1 -- 172.21.15.33:0/3221960222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6178005520 con 0x5629ea7554a0 2024-09-16T00:26:36.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f6163fff640 1 -- 172.21.15.33:0/3221960222 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f61740a9be0 con 0x5629ea7554a0 2024-09-16T00:26:36.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f618d03a740 10 client.26593 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:36.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f618d03a740 1 -- 172.21.15.33:0/3221960222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5629ea76ebc0 con 0x5629ea7554a0 2024-09-16T00:26:36.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f6163fff640 1 -- 172.21.15.33:0/3221960222 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6178005520 con 0x5629ea7554a0 2024-09-16T00:26:36.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f6163fff640 1 -- 172.21.15.33:0/3221960222 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f61740ac3d0 con 0x5629ea7554a0 2024-09-16T00:26:36.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f618d03a740 10 client.26593 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:36.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f618d03a740 10 client.26593 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:26:36.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f618d03a740 20 client.26593 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:36.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f618d03a740 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5629ea770070 0x5629ea7724e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f618d03a740 4 client.26593 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:26:36.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.288+0000 7f618d03a740 1 -- 172.21.15.33:0/3221960222 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629ea6c8850 con 0x5629ea770070 2024-09-16T00:26:36.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.289+0000 7f6183fff640 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5629ea770070 0x5629ea7724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.289+0000 7f6163fff640 10 client.26593 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:36.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.289+0000 7f6183fff640 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5629ea770070 0x5629ea7724e0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a076722d7eacfb70 server_cookie=1b788dd2a16c55d3 in_seq=0 out_seq=0 2024-09-16T00:26:36.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f6163fff640 1 -- 172.21.15.33:0/3221960222 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5629ea6c8850 con 0x5629ea770070 2024-09-16T00:26:36.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f6163fff640 10 client.26593 handle_command_reply: tid=0 2024-09-16T00:26:36.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f618d03a740 1 client.26593 shutdown 2024-09-16T00:26:36.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f618d03a740 1 -- 172.21.15.33:0/3221960222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea7554a0 msgr2=0x5629ea75ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:36.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f618d03a740 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea7554a0 0x5629ea75ff70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f617400cba0 tx=0x7f617406b2a0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f618d03a740 1 -- 172.21.15.33:0/3221960222 shutdown_connections 2024-09-16T00:26:36.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f618d03a740 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5629ea770070 0x5629ea7724e0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f618d03a740 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea760680 0x5629ea75d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f618d03a740 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea755dc0 0x5629ea75d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f618d03a740 1 --2- 172.21.15.33:0/3221960222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea7554a0 0x5629ea75ff70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f618d03a740 1 -- 172.21.15.33:0/3221960222 >> 172.21.15.33:0/3221960222 conn(0x5629ea751a30 msgr2=0x5629ea750b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:36.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f6163fff640 0 client.26593 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:36.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f618d03a740 1 -- 172.21.15.33:0/3221960222 shutdown_connections 2024-09-16T00:26:36.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f618d03a740 1 -- 172.21.15.33:0/3221960222 wait complete. 2024-09-16T00:26:36.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.290+0000 7f618d03a740 20 client.26593 trim_cache size 0 max 16384 2024-09-16T00:26:36.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.309+0000 7f618d03a740 1 Processor -- start 2024-09-16T00:26:36.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.309+0000 7f618d03a740 1 -- start start 2024-09-16T00:26:36.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.309+0000 7f618d03a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea751a30 0x5629ea76c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.309+0000 7f618d03a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea7554a0 0x5629ea757600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.309+0000 7f618d03a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea755dc0 0x5629ea777db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.309+0000 7f618d03a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f61841b3b70 con 0x5629ea751a30 2024-09-16T00:26:36.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.309+0000 7f618d03a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f61841b1440 con 0x5629ea755dc0 2024-09-16T00:26:36.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.309+0000 7f618d03a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f61841af290 con 0x5629ea7554a0 2024-09-16T00:26:36.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.309+0000 7f61889fd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea751a30 0x5629ea76c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.309+0000 7f61889fd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea751a30 0x5629ea76c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34432/0 (socket says 172.21.15.33:34432) 2024-09-16T00:26:36.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.309+0000 7f61889fd640 1 -- 172.21.15.33:0/665313953 learned_addr learned my addr 172.21.15.33:0/665313953 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:36.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.310+0000 7f6183fff640 1 --2- 172.21.15.33:0/665313953 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea7554a0 0x5629ea757600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.310+0000 7f61891fe640 1 --2- 172.21.15.33:0/665313953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea755dc0 0x5629ea777db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.310+0000 7f61889fd640 1 -- 172.21.15.33:0/665313953 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea7554a0 msgr2=0x5629ea757600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:36.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.310+0000 7f61889fd640 1 --2- 172.21.15.33:0/665313953 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea7554a0 0x5629ea757600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.310+0000 7f61889fd640 1 -- 172.21.15.33:0/665313953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea755dc0 msgr2=0x5629ea777db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:36.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.310+0000 7f61889fd640 1 --2- 172.21.15.33:0/665313953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea755dc0 0x5629ea777db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.310+0000 7f61889fd640 1 -- 172.21.15.33:0/665313953 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629ea75b770 con 0x5629ea751a30 2024-09-16T00:26:36.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.310+0000 7f61891fe640 1 --2- 172.21.15.33:0/665313953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea755dc0 0x5629ea777db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:26:36.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.310+0000 7f61889fd640 1 --2- 172.21.15.33:0/665313953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea751a30 0x5629ea76c4b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f61740989c0 tx=0x7f61740093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407f5fd352dbf4bf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:36.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.310+0000 7f616a7fc640 1 -- 172.21.15.33:0/665313953 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617409bec0 con 0x5629ea751a30 2024-09-16T00:26:36.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.310+0000 7f616a7fc640 1 -- 172.21.15.33:0/665313953 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61740b0020 con 0x5629ea751a30 2024-09-16T00:26:36.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.310+0000 7f616a7fc640 1 -- 172.21.15.33:0/665313953 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174005590 con 0x5629ea751a30 2024-09-16T00:26:36.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.311+0000 7f618d03a740 1 -- 172.21.15.33:0/665313953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea751a30 msgr2=0x5629ea76c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:36.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.311+0000 7f618d03a740 1 --2- 172.21.15.33:0/665313953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea751a30 0x5629ea76c4b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f61740989c0 tx=0x7f61740093e0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.311+0000 7f618d03a740 1 -- 172.21.15.33:0/665313953 shutdown_connections 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.311+0000 7f618d03a740 1 --2- 172.21.15.33:0/665313953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea755dc0 0x5629ea777db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.311+0000 7f618d03a740 1 --2- 172.21.15.33:0/665313953 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea7554a0 0x5629ea757600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.311+0000 7f618d03a740 1 --2- 172.21.15.33:0/665313953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea751a30 0x5629ea76c4b0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.311+0000 7f618d03a740 1 -- 172.21.15.33:0/665313953 >> 172.21.15.33:0/665313953 conn(0x5629ea75d8c0 msgr2=0x5629ea756ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.311+0000 7f618d03a740 1 -- 172.21.15.33:0/665313953 shutdown_connections 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.311+0000 7f618d03a740 1 -- 172.21.15.33:0/665313953 wait complete. 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.312+0000 7f618d03a740 1 -- start start 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.312+0000 7f618d03a740 1 Processor -- start 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.312+0000 7f618d03a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea751a30 0x5629ea768a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.312+0000 7f618d03a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea7554a0 0x5629ea763d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.312+0000 7f618d03a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea755dc0 0x5629ea7642d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.312+0000 7f618d03a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f61841af900 con 0x5629ea755dc0 2024-09-16T00:26:36.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.312+0000 7f618d03a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f61841b0d70 con 0x5629ea7554a0 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.312+0000 7f618d03a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f61841affd0 con 0x5629ea751a30 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.312+0000 7f61889fd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea751a30 0x5629ea768a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f61889fd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea751a30 0x5629ea768a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37562/0 (socket says 172.21.15.33:37562) 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f61889fd640 1 -- 172.21.15.33:0/657571414 learned_addr learned my addr 172.21.15.33:0/657571414 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f61891fe640 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea755dc0 0x5629ea7642d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f6183fff640 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea7554a0 0x5629ea763d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f618d03a740 10 client.0 _ll_register_callbacks cb 0x7f61740cd980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f61889fd640 1 -- 172.21.15.33:0/657571414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea7554a0 msgr2=0x5629ea763d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f61889fd640 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea7554a0 0x5629ea763d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f61889fd640 1 -- 172.21.15.33:0/657571414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea755dc0 msgr2=0x5629ea7642d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f61889fd640 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea755dc0 0x5629ea7642d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f61889fd640 1 -- 172.21.15.33:0/657571414 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629ea764810 con 0x5629ea751a30 2024-09-16T00:26:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f61889fd640 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea751a30 0x5629ea768a60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f617409a810 tx=0x7f61740a50d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b43b918f94b7480b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f6163fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f6163fff640 1 -- 172.21.15.33:0/657571414 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617409bec0 con 0x5629ea751a30 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.313+0000 7f6163fff640 1 -- 172.21.15.33:0/657571414 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f617409b9b0 con 0x5629ea751a30 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f618d03a740 1 -- 172.21.15.33:0/657571414 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61740b0020 con 0x5629ea751a30 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f6163fff640 1 -- 172.21.15.33:0/657571414 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f617409c0a0 con 0x5629ea751a30 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f618d03a740 10 client.26599 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f618d03a740 1 -- 172.21.15.33:0/657571414 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5629ea764810 con 0x5629ea751a30 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f6163fff640 1 -- 172.21.15.33:0/657571414 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61740b0020 con 0x5629ea751a30 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f6163fff640 1 -- 172.21.15.33:0/657571414 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f617409b420 con 0x5629ea751a30 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f618d03a740 10 client.26599 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f618d03a740 10 client.26599 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f618d03a740 20 client.26599 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:36.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f618d03a740 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5629ea760680 0x5629ea7dcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f618d03a740 4 client.26599 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f618d03a740 1 -- 172.21.15.33:0/657571414 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5629ea6c2d60 con 0x5629ea760680 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.314+0000 7f6183fff640 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5629ea760680 0x5629ea7dcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f6183fff640 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5629ea760680 0x5629ea7dcce0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13bf3fc592ead57f server_cookie=c2b86cc1a8f9cca8 in_seq=0 out_seq=0 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f6163fff640 10 client.26599 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f6163fff640 1 -- 172.21.15.33:0/657571414 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5629ea6c2d60 con 0x5629ea760680 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f6163fff640 10 client.26599 handle_command_reply: tid=0 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f618d03a740 1 client.26599 shutdown 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f618d03a740 1 -- 172.21.15.33:0/657571414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea751a30 msgr2=0x5629ea768a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f618d03a740 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea751a30 0x5629ea768a60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f617409a810 tx=0x7f61740a50d0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f618d03a740 1 -- 172.21.15.33:0/657571414 shutdown_connections 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f618d03a740 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5629ea760680 0x5629ea7dcce0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f618d03a740 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629ea755dc0 0x5629ea7642d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f618d03a740 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629ea7554a0 0x5629ea763d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f6163fff640 0 client.26599 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:36.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f618d03a740 1 --2- 172.21.15.33:0/657571414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629ea751a30 0x5629ea768a60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f618d03a740 1 -- 172.21.15.33:0/657571414 >> 172.21.15.33:0/657571414 conn(0x5629ea75d8c0 msgr2=0x5629ea757710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:36.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f618d03a740 1 -- 172.21.15.33:0/657571414 shutdown_connections 2024-09-16T00:26:36.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.315+0000 7f618d03a740 1 -- 172.21.15.33:0/657571414 wait complete. 2024-09-16T00:26:36.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618d03a740 20 client.26599 trim_cache size 0 max 16384 2024-09-16T00:26:36.320 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:26:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:26:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:26:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "6f95a3b6-ce5b-4e58-8c2d-0a5d9c7a385f": { 2024-09-16T00:26:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:26:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "6f95a3b6-ce5b-4e58-8c2d-0a5d9c7a385f", 2024-09-16T00:26:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:26:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:36.321 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:26:36.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618ac61640 1 -- 172.21.15.33:0/179208036 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f616c0a3240 msgr2=0x7f616c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:36.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618ac61640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f616c0a3240 0x7f616c0a5700 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f617c005f60 tx=0x7f617c009ed0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618ac61640 1 -- 172.21.15.33:0/179208036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6184101df0 msgr2=0x7f61840744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:36.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618ac61640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6184101df0 0x7f61840744f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f617400eb90 tx=0x7f617400c6a0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618ac61640 1 -- 172.21.15.33:0/179208036 shutdown_connections 2024-09-16T00:26:36.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618ac61640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f616c0a3240 0x7f616c0a5700 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618ac61640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184106f20 0x7f6184074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618ac61640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61841027a0 0x7f6184074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618ac61640 1 --2- 172.21.15.33:0/179208036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6184101df0 0x7f61840744f0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:36.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618ac61640 1 -- 172.21.15.33:0/179208036 >> 172.21.15.33:0/179208036 conn(0x7f61840fd8f0 msgr2=0x7f61840fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:36.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618ac61640 1 -- 172.21.15.33:0/179208036 shutdown_connections 2024-09-16T00:26:36.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:36.316+0000 7f618ac61640 1 -- 172.21.15.33:0/179208036 wait complete. 2024-09-16T00:26:36.673 INFO:tasks.workunit.client.0.smithi033.stdout: 1 752847 9.99 MB/sec execute 344 sec latency 14.425 ms 2024-09-16T00:26:37.095 INFO:tasks.cephfs.filesystem:scrub status for tag:6f95a3b6-ce5b-4e58-8c2d-0a5d9c7a385f - {'path': '/', 'tag': '6f95a3b6-ce5b-4e58-8c2d-0a5d9c7a385f', 'options': 'recursive,force'} 2024-09-16T00:26:37.135 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:37.574 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:37 smithi033 ceph-mon[31243]: pgmap v634: 129 pgs: 129 active+clean; 273 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 2.4 MiB/s wr, 101 op/s 2024-09-16T00:26:37.673 INFO:tasks.workunit.client.0.smithi033.stdout: 1 755380 9.99 MB/sec execute 345 sec latency 19.266 ms 2024-09-16T00:26:37.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.661+0000 7f8323445640 1 -- 172.21.15.33:0/3033619114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f831c102790 msgr2=0x7f831c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:37.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.661+0000 7f8323445640 1 --2- 172.21.15.33:0/3033619114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f831c102790 0x7f831c1067e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f830c009900 tx=0x7f830c02a410 comp rx=0 tx=0).stop 2024-09-16T00:26:37.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.661+0000 7f8323445640 1 -- 172.21.15.33:0/3033619114 shutdown_connections 2024-09-16T00:26:37.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.661+0000 7f8323445640 1 --2- 172.21.15.33:0/3033619114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f831c106f10 0x7f831c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:37.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.661+0000 7f8323445640 1 --2- 172.21.15.33:0/3033619114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f831c102790 0x7f831c1067e0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:37.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.661+0000 7f8323445640 1 --2- 172.21.15.33:0/3033619114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f831c101de0 0x7f831c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:37.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.661+0000 7f8323445640 1 -- 172.21.15.33:0/3033619114 >> 172.21.15.33:0/3033619114 conn(0x7f831c0fd8c0 msgr2=0x7f831c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:37.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.661+0000 7f8323445640 1 -- 172.21.15.33:0/3033619114 shutdown_connections 2024-09-16T00:26:37.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.661+0000 7f8323445640 1 -- 172.21.15.33:0/3033619114 wait complete. 2024-09-16T00:26:37.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.661+0000 7f8323445640 1 Processor -- start 2024-09-16T00:26:37.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f8323445640 1 -- start start 2024-09-16T00:26:37.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f8323445640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f831c101de0 0x7f831c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:37.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f8323445640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f831c102790 0x7f831c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:37.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f8323445640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f831c106f10 0x7f831c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:37.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f8323445640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f831c111f70 con 0x7f831c102790 2024-09-16T00:26:37.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f8323445640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f831c111df0 con 0x7f831c106f10 2024-09-16T00:26:37.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f8323445640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f831c1120f0 con 0x7f831c101de0 2024-09-16T00:26:37.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f83209e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f831c102790 0x7f831c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:37.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f83211e1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f831c101de0 0x7f831c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:37.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f83211e1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f831c101de0 0x7f831c198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37582/0 (socket says 172.21.15.33:37582) 2024-09-16T00:26:37.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f83209e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f831c102790 0x7f831c198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34458/0 (socket says 172.21.15.33:34458) 2024-09-16T00:26:37.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f83211e1640 1 -- 172.21.15.33:0/3831465745 learned_addr learned my addr 172.21.15.33:0/3831465745 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:37.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f83219e2640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f831c106f10 0x7f831c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:37.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f83211e1640 1 -- 172.21.15.33:0/3831465745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f831c106f10 msgr2=0x7f831c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:37.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f83211e1640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f831c106f10 0x7f831c19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:37.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f83211e1640 1 -- 172.21.15.33:0/3831465745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f831c102790 msgr2=0x7f831c198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:37.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f83211e1640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f831c102790 0x7f831c198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:37.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.662+0000 7f83211e1640 1 -- 172.21.15.33:0/3831465745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f831c19d400 con 0x7f831c101de0 2024-09-16T00:26:37.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.663+0000 7f83211e1640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f831c101de0 0x7f831c198450 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f831800e9f0 tx=0x7f831800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:37.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.663+0000 7f83127fc640 1 -- 172.21.15.33:0/3831465745 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831800cd80 con 0x7f831c101de0 2024-09-16T00:26:37.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.663+0000 7f83127fc640 1 -- 172.21.15.33:0/3831465745 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8318005500 con 0x7f831c101de0 2024-09-16T00:26:37.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.663+0000 7f83127fc640 1 -- 172.21.15.33:0/3831465745 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8318010690 con 0x7f831c101de0 2024-09-16T00:26:37.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.663+0000 7f8323445640 1 -- 172.21.15.33:0/3831465745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f831c19d6f0 con 0x7f831c101de0 2024-09-16T00:26:37.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.663+0000 7f8323445640 1 -- 172.21.15.33:0/3831465745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f831c1a4fd0 con 0x7f831c101de0 2024-09-16T00:26:37.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.664+0000 7f8323445640 1 -- 172.21.15.33:0/3831465745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f831c1021c0 con 0x7f831c101de0 2024-09-16T00:26:37.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.665+0000 7f83127fc640 1 -- 172.21.15.33:0/3831465745 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f83180046c0 con 0x7f831c101de0 2024-09-16T00:26:37.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.666+0000 7f83127fc640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f83080a32c0 0x7f83080a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:37.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.666+0000 7f83209e0640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f83080a32c0 0x7f83080a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:37.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.666+0000 7f83127fc640 1 -- 172.21.15.33:0/3831465745 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f83180d0430 con 0x7f831c101de0 2024-09-16T00:26:37.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.666+0000 7f83209e0640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f83080a32c0 0x7f83080a5780 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f830c009230 tx=0x7f830c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:37.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.670+0000 7f83127fc640 1 -- 172.21.15.33:0/3831465745 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f831809cc70 con 0x7f831c101de0 2024-09-16T00:26:37.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.900+0000 7f8323445640 1 -- 172.21.15.33:0/3831465745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f831c02d7c0 con 0x7f831c101de0 2024-09-16T00:26:37.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.901+0000 7f83127fc640 1 -- 172.21.15.33:0/3831465745 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f83180bc030 con 0x7f831c101de0 2024-09-16T00:26:37.901 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:26:37.901 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:26:37.902 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:26:37.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.903+0000 7f8323445640 1 -- 172.21.15.33:0/3831465745 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f83080a32c0 msgr2=0x7f83080a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:37.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.903+0000 7f8323445640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f83080a32c0 0x7f83080a5780 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f830c009230 tx=0x7f830c0058d0 comp rx=0 tx=0).stop 2024-09-16T00:26:37.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.903+0000 7f8323445640 1 -- 172.21.15.33:0/3831465745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f831c101de0 msgr2=0x7f831c198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:37.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.903+0000 7f8323445640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f831c101de0 0x7f831c198450 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f831800e9f0 tx=0x7f831800edf0 comp rx=0 tx=0).stop 2024-09-16T00:26:37.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.903+0000 7f8323445640 1 -- 172.21.15.33:0/3831465745 shutdown_connections 2024-09-16T00:26:37.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.903+0000 7f8323445640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f83080a32c0 0x7f83080a5780 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:37.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.903+0000 7f8323445640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f831c106f10 0x7f831c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:37.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.903+0000 7f8323445640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f831c102790 0x7f831c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:37.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.903+0000 7f8323445640 1 --2- 172.21.15.33:0/3831465745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f831c101de0 0x7f831c198450 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:37.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.903+0000 7f8323445640 1 -- 172.21.15.33:0/3831465745 >> 172.21.15.33:0/3831465745 conn(0x7f831c0fd8c0 msgr2=0x7f831c0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:37.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.903+0000 7f8323445640 1 -- 172.21.15.33:0/3831465745 shutdown_connections 2024-09-16T00:26:37.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:37.903+0000 7f8323445640 1 -- 172.21.15.33:0/3831465745 wait complete. 2024-09-16T00:26:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:37 smithi083 ceph-mon[35242]: pgmap v634: 129 pgs: 129 active+clean; 273 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 2.4 MiB/s wr, 101 op/s 2024-09-16T00:26:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:37 smithi060 ceph-mon[35185]: pgmap v634: 129 pgs: 129 active+clean; 273 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 2.4 MiB/s wr, 101 op/s 2024-09-16T00:26:38.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:38 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3831465745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:38.645 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0' 2024-09-16T00:26:38.645 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0 2024-09-16T00:26:38.673 INFO:tasks.workunit.client.0.smithi033.stdout: 1 757254 10.00 MB/sec execute 346 sec latency 47.337 ms 2024-09-16T00:26:38.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:38 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3831465745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:38.920 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:38 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3831465745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:39.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.481+0000 7fb192264640 1 -- 172.21.15.33:0/3296532521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb18c103660 msgr2=0x7fb18c10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:39.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.481+0000 7fb192264640 1 --2- 172.21.15.33:0/3296532521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb18c103660 0x7fb18c10bb50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb180009900 tx=0x7fb18002a480 comp rx=0 tx=0).stop 2024-09-16T00:26:39.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.481+0000 7fb192264640 1 -- 172.21.15.33:0/3296532521 shutdown_connections 2024-09-16T00:26:39.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.481+0000 7fb192264640 1 --2- 172.21.15.33:0/3296532521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb18c10c170 0x7fb18c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.481+0000 7fb192264640 1 --2- 172.21.15.33:0/3296532521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb18c103660 0x7fb18c10bb50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.481+0000 7fb192264640 1 --2- 172.21.15.33:0/3296532521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c102d40 0x7fb18c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.481+0000 7fb192264640 1 -- 172.21.15.33:0/3296532521 >> 172.21.15.33:0/3296532521 conn(0x7fb18c0fc7a0 msgr2=0x7fb18c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:39.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.481+0000 7fb192264640 1 -- 172.21.15.33:0/3296532521 shutdown_connections 2024-09-16T00:26:39.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.481+0000 7fb192264640 1 -- 172.21.15.33:0/3296532521 wait complete. 2024-09-16T00:26:39.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb192264640 1 Processor -- start 2024-09-16T00:26:39.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb192264640 1 -- start start 2024-09-16T00:26:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb192264640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c102d40 0x7fb18c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb192264640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb18c103660 0x7fb18c19acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb192264640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb18c10c170 0x7fb18c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb192264640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c111f70 con 0x7fb18c10c170 2024-09-16T00:26:39.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb192264640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c111df0 con 0x7fb18c102d40 2024-09-16T00:26:39.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb192264640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c1120f0 con 0x7fb18c103660 2024-09-16T00:26:39.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb18bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c102d40 0x7fb18c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb18bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c102d40 0x7fb18c19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:48328/0 (socket says 172.21.15.33:48328) 2024-09-16T00:26:39.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb18bfff640 1 -- 172.21.15.33:0/1404974008 learned_addr learned my addr 172.21.15.33:0/1404974008 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:39.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb18c103660 0x7fb18c19acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb190801640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb18c10c170 0x7fb18c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.482+0000 7fb18bfff640 1 -- 172.21.15.33:0/1404974008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb18c103660 msgr2=0x7fb18c19acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:39.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.483+0000 7fb18bfff640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb18c103660 0x7fb18c19acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.483+0000 7fb18bfff640 1 -- 172.21.15.33:0/1404974008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb18c10c170 msgr2=0x7fb18c19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:39.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.483+0000 7fb18bfff640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb18c10c170 0x7fb18c19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.483+0000 7fb18bfff640 1 -- 172.21.15.33:0/1404974008 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18c19f6a0 con 0x7fb18c102d40 2024-09-16T00:26:39.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.483+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb18c103660 0x7fb18c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:26:39.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.483+0000 7fb190801640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb18c10c170 0x7fb18c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:26:39.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.483+0000 7fb18bfff640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c102d40 0x7fb18c19a790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00ea60 tx=0x7fb17c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:39.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.483+0000 7fb1897fa640 1 -- 172.21.15.33:0/1404974008 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c00cde0 con 0x7fb18c102d40 2024-09-16T00:26:39.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.483+0000 7fb1897fa640 1 -- 172.21.15.33:0/1404974008 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb17c0035f0 con 0x7fb18c102d40 2024-09-16T00:26:39.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.483+0000 7fb192264640 1 -- 172.21.15.33:0/1404974008 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18c19f990 con 0x7fb18c102d40 2024-09-16T00:26:39.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.483+0000 7fb1897fa640 1 -- 172.21.15.33:0/1404974008 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c010670 con 0x7fb18c102d40 2024-09-16T00:26:39.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.483+0000 7fb192264640 1 -- 172.21.15.33:0/1404974008 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb18c1a7270 con 0x7fb18c102d40 2024-09-16T00:26:39.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.485+0000 7fb1897fa640 1 -- 172.21.15.33:0/1404974008 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb17c0037a0 con 0x7fb18c102d40 2024-09-16T00:26:39.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb19463d740 1 Processor -- start 2024-09-16T00:26:39.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb19463d740 1 -- start start 2024-09-16T00:26:39.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb19463d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be0a850 0x55b11be0ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb1897fa640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb1740a3240 0x7fb1740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb19463d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 0x55b11be154f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb19463d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be15a30 0x55b11be17e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb19463d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b11bd6f160 con 0x55b11be15a30 2024-09-16T00:26:39.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb19463d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55b11bd16720 con 0x55b11be0a850 2024-09-16T00:26:39.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb19463d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b11bd6b120 con 0x55b11be0b170 2024-09-16T00:26:39.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb18b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 0x55b11be154f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb18bfff640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb1740a3240 0x7fb1740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb190801640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be0a850 0x55b11be0ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb190801640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be0a850 0x55b11be0ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:48340/0 (socket says 172.21.15.33:48340) 2024-09-16T00:26:39.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb190801640 1 -- 172.21.15.33:0/1522232213 learned_addr learned my addr 172.21.15.33:0/1522232213 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:39.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb1897fa640 1 -- 172.21.15.33:0/1404974008 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb17c0d0230 con 0x7fb18c102d40 2024-09-16T00:26:39.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.486+0000 7fb190801640 1 --2- 172.21.15.33:0/1522232213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be15a30 0x55b11be17e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.487+0000 7fb18b7fe640 1 -- 172.21.15.33:0/1522232213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be0a850 msgr2=0x55b11be0ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:39.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.487+0000 7fb18bfff640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb1740a3240 0x7fb1740a5700 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb17c01f000 tx=0x7fb17c066d00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:39.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.487+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/1522232213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be0a850 0x55b11be0ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.487+0000 7fb18b7fe640 1 -- 172.21.15.33:0/1522232213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be15a30 msgr2=0x55b11be17e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.487+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/1522232213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be15a30 0x55b11be17e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.487+0000 7fb18b7fe640 1 -- 172.21.15.33:0/1522232213 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b11bae0ba0 con 0x55b11be0b170 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.487+0000 7fb190801640 1 --2- 172.21.15.33:0/1522232213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be0a850 0x55b11be0ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.487+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/1522232213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 0x55b11be154f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb180000c00 tx=0x7fb180032c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47d58bd10a47adb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.487+0000 7fb1727fc640 1 -- 172.21.15.33:0/1522232213 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18003f070 con 0x55b11be0b170 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.488+0000 7fb1727fc640 1 -- 172.21.15.33:0/1522232213 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb180032e20 con 0x55b11be0b170 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.488+0000 7fb1727fc640 1 -- 172.21.15.33:0/1522232213 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180035070 con 0x55b11be0b170 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.488+0000 7fb19463d740 1 -- 172.21.15.33:0/1522232213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 msgr2=0x55b11be154f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.488+0000 7fb19463d740 1 --2- 172.21.15.33:0/1522232213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 0x55b11be154f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb180000c00 tx=0x7fb180032c30 comp rx=0 tx=0).stop 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.488+0000 7fb19463d740 1 -- 172.21.15.33:0/1522232213 shutdown_connections 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.488+0000 7fb19463d740 1 --2- 172.21.15.33:0/1522232213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be15a30 0x55b11be17e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.488+0000 7fb19463d740 1 --2- 172.21.15.33:0/1522232213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 0x55b11be154f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.488+0000 7fb19463d740 1 --2- 172.21.15.33:0/1522232213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be0a850 0x55b11be0ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.488+0000 7fb19463d740 1 -- 172.21.15.33:0/1522232213 >> 172.21.15.33:0/1522232213 conn(0x55b11be06de0 msgr2=0x55b11be071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.488+0000 7fb19463d740 1 -- 172.21.15.33:0/1522232213 shutdown_connections 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.488+0000 7fb19463d740 1 -- 172.21.15.33:0/1522232213 wait complete. 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.489+0000 7fb19463d740 1 -- start start 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.489+0000 7fb19463d740 1 Processor -- start 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.489+0000 7fb19463d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 0x55b11be128c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.489+0000 7fb19463d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be0b170 0x55b11be12e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.489+0000 7fb19463d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be15a30 0x55b11be13340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.489+0000 7fb19463d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b11be18650 con 0x55b11be0b170 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.489+0000 7fb19463d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55b11bcf7900 con 0x55b11be15a30 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.489+0000 7fb19463d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b11be187d0 con 0x55b11be0a850 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.489+0000 7fb19463d740 10 client.0 _ll_register_callbacks cb 0x55b11bcf5780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:39.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.489+0000 7fb190801640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 0x55b11be128c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb18b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be0b170 0x55b11be12e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb190801640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 0x55b11be128c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37622/0 (socket says 172.21.15.33:37622) 2024-09-16T00:26:39.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb190801640 1 -- 172.21.15.33:0/533247697 learned_addr learned my addr 172.21.15.33:0/533247697 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:39.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb190801640 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be15a30 0x55b11be13340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb18b7fe640 1 -- 172.21.15.33:0/533247697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 msgr2=0x55b11be128c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:39.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 0x55b11be128c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb18b7fe640 1 -- 172.21.15.33:0/533247697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be15a30 msgr2=0x55b11be13340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:39.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be15a30 0x55b11be13340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb18b7fe640 1 -- 172.21.15.33:0/533247697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b11bd37c50 con 0x55b11be0b170 2024-09-16T00:26:39.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb190801640 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 0x55b11be128c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:26:39.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be0b170 0x55b11be12e00 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fb180009900 tx=0x7fb180005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62937a35762c6153 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:39.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb16b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:26:39.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb16b7fe640 1 -- 172.21.15.33:0/533247697 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18003f070 con 0x55b11be0b170 2024-09-16T00:26:39.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb16b7fe640 1 -- 172.21.15.33:0/533247697 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb180004030 con 0x55b11be0b170 2024-09-16T00:26:39.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb19463d740 1 -- 172.21.15.33:0/533247697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb180032e20 con 0x55b11be0b170 2024-09-16T00:26:39.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.490+0000 7fb16b7fe640 1 -- 172.21.15.33:0/533247697 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb180054050 con 0x55b11be0b170 2024-09-16T00:26:39.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.491+0000 7fb19463d740 10 client.17202 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:39.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.491+0000 7fb19463d740 1 -- 172.21.15.33:0/533247697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b11be24290 con 0x55b11be0b170 2024-09-16T00:26:39.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.491+0000 7fb16b7fe640 1 -- 172.21.15.33:0/533247697 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb180032e20 con 0x55b11be0b170 2024-09-16T00:26:39.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.491+0000 7fb16b7fe640 1 -- 172.21.15.33:0/533247697 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fb1800046c0 con 0x55b11be0b170 2024-09-16T00:26:39.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.491+0000 7fb19463d740 10 client.17202 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:39.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.491+0000 7fb19463d740 10 client.17202 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:26:39.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.491+0000 7fb19463d740 20 client.17202 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:39.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.491+0000 7fb19463d740 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b11be255e0 0x55b11be279d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.491+0000 7fb19463d740 4 client.17202 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:26:39.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.491+0000 7fb19463d740 1 -- 172.21.15.33:0/533247697 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b11bd7c890 con 0x55b11be255e0 2024-09-16T00:26:39.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.491+0000 7fb190801640 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b11be255e0 0x55b11be279d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb190801640 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b11be255e0 0x55b11be279d0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=986eb00ab416ec15 server_cookie=3c70ce7432420c79 in_seq=0 out_seq=0 2024-09-16T00:26:39.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb16b7fe640 10 client.17202 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:39.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb16b7fe640 1 -- 172.21.15.33:0/533247697 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b11bd7c890 con 0x55b11be255e0 2024-09-16T00:26:39.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb16b7fe640 10 client.17202 handle_command_reply: tid=0 2024-09-16T00:26:39.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb19463d740 1 client.17202 shutdown 2024-09-16T00:26:39.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb19463d740 1 -- 172.21.15.33:0/533247697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be0b170 msgr2=0x55b11be12e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:39.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb19463d740 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be0b170 0x55b11be12e00 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fb180009900 tx=0x7fb180005740 comp rx=0 tx=0).stop 2024-09-16T00:26:39.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb19463d740 1 -- 172.21.15.33:0/533247697 shutdown_connections 2024-09-16T00:26:39.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb19463d740 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b11be255e0 0x55b11be279d0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb19463d740 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be15a30 0x55b11be13340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb19463d740 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be0b170 0x55b11be12e00 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb19463d740 1 --2- 172.21.15.33:0/533247697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 0x55b11be128c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.492+0000 7fb19463d740 1 -- 172.21.15.33:0/533247697 >> 172.21.15.33:0/533247697 conn(0x55b11be06de0 msgr2=0x55b11be0cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:39.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.493+0000 7fb16b7fe640 0 client.17202 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:39.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.493+0000 7fb19463d740 1 -- 172.21.15.33:0/533247697 shutdown_connections 2024-09-16T00:26:39.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.493+0000 7fb19463d740 1 -- 172.21.15.33:0/533247697 wait complete. 2024-09-16T00:26:39.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.493+0000 7fb19463d740 20 client.17202 trim_cache size 0 max 16384 2024-09-16T00:26:39.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.511+0000 7fb19463d740 1 Processor -- start 2024-09-16T00:26:39.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.511+0000 7fb19463d740 1 -- start start 2024-09-16T00:26:39.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.511+0000 7fb19463d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be0a850 0x55b11be0d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.511+0000 7fb19463d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 0x55b11be13190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.511+0000 7fb19463d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be15a30 0x55b11be2a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.511+0000 7fb19463d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c1b3c60 con 0x55b11be15a30 2024-09-16T00:26:39.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.511+0000 7fb19463d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c1b1530 con 0x55b11be0a850 2024-09-16T00:26:39.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.511+0000 7fb19463d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c1af380 con 0x55b11be0b170 2024-09-16T00:26:39.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb190801640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be15a30 0x55b11be2a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb18b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 0x55b11be13190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb18b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 0x55b11be13190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37636/0 (socket says 172.21.15.33:37636) 2024-09-16T00:26:39.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb18b7fe640 1 -- 172.21.15.33:0/693572676 learned_addr learned my addr 172.21.15.33:0/693572676 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:39.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb190801640 1 --2- 172.21.15.33:0/693572676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be0a850 0x55b11be0d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb18b7fe640 1 -- 172.21.15.33:0/693572676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be0a850 msgr2=0x55b11be0d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:39.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/693572676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be0a850 0x55b11be0d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb18b7fe640 1 -- 172.21.15.33:0/693572676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be15a30 msgr2=0x55b11be2a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:39.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/693572676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be15a30 0x55b11be2a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb18b7fe640 1 -- 172.21.15.33:0/693572676 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b11be0a680 con 0x55b11be0b170 2024-09-16T00:26:39.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/693572676 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 0x55b11be13190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb1800092a0 tx=0x7fb180005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4a2b531d9c47177 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:39.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb171ffb640 1 -- 172.21.15.33:0/693572676 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18005d070 con 0x55b11be0b170 2024-09-16T00:26:39.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb171ffb640 1 -- 172.21.15.33:0/693572676 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb180032da0 con 0x55b11be0b170 2024-09-16T00:26:39.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.512+0000 7fb171ffb640 1 -- 172.21.15.33:0/693572676 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18003f040 con 0x55b11be0b170 2024-09-16T00:26:39.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.513+0000 7fb19463d740 1 -- 172.21.15.33:0/693572676 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 msgr2=0x55b11be13190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:39.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.513+0000 7fb19463d740 1 --2- 172.21.15.33:0/693572676 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 0x55b11be13190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb1800092a0 tx=0x7fb180005b50 comp rx=0 tx=0).stop 2024-09-16T00:26:39.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.513+0000 7fb19463d740 1 -- 172.21.15.33:0/693572676 shutdown_connections 2024-09-16T00:26:39.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.513+0000 7fb19463d740 1 --2- 172.21.15.33:0/693572676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be15a30 0x55b11be2a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.513+0000 7fb19463d740 1 --2- 172.21.15.33:0/693572676 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0b170 0x55b11be13190 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.513+0000 7fb19463d740 1 --2- 172.21.15.33:0/693572676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be0a850 0x55b11be0d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.513+0000 7fb19463d740 1 -- 172.21.15.33:0/693572676 >> 172.21.15.33:0/693572676 conn(0x55b11be06de0 msgr2=0x55b11be11190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:39.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.514+0000 7fb19463d740 1 -- 172.21.15.33:0/693572676 shutdown_connections 2024-09-16T00:26:39.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.514+0000 7fb19463d740 1 -- 172.21.15.33:0/693572676 wait complete. 2024-09-16T00:26:39.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.514+0000 7fb19463d740 1 -- start start 2024-09-16T00:26:39.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.514+0000 7fb19463d740 1 Processor -- start 2024-09-16T00:26:39.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.514+0000 7fb19463d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 0x55b11be2abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb19463d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be0b170 0x55b11be27f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb19463d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be15a30 0x55b11be28450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb19463d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c1af9f0 con 0x55b11be0b170 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb19463d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c1b0e60 con 0x55b11be15a30 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb19463d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c1b00c0 con 0x55b11be0a850 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb190801640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 0x55b11be2abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb19463d740 10 client.0 _ll_register_callbacks cb 0x7fb184012590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb190801640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 0x55b11be2abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:37642/0 (socket says 172.21.15.33:37642) 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb190801640 1 -- 172.21.15.33:0/1677226521 learned_addr learned my addr 172.21.15.33:0/1677226521 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be0b170 0x55b11be27f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb190801640 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be15a30 0x55b11be28450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb190801640 1 -- 172.21.15.33:0/1677226521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be15a30 msgr2=0x55b11be28450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb190801640 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be15a30 0x55b11be28450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb190801640 1 -- 172.21.15.33:0/1677226521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be0b170 msgr2=0x55b11be27f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:26:39.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb190801640 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be0b170 0x55b11be27f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb190801640 1 -- 172.21.15.33:0/1677226521 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b11be28a90 con 0x55b11be0a850 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb190801640 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 0x55b11be2abc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb184013ef0 tx=0x7fb184014040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=958065246ae14ef2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb16b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb16b7fe640 1 -- 172.21.15.33:0/1677226521 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb184012c20 con 0x55b11be0a850 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb19463d740 1 -- 172.21.15.33:0/1677226521 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb180032da0 con 0x55b11be0a850 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.515+0000 7fb16b7fe640 1 -- 172.21.15.33:0/1677226521 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb184020e80 con 0x55b11be0a850 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.516+0000 7fb16b7fe640 1 -- 172.21.15.33:0/1677226521 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb184004b20 con 0x55b11be0a850 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.516+0000 7fb19463d740 10 client.26620 fetch_fsmap learned FSMap version 30 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.516+0000 7fb19463d740 1 -- 172.21.15.33:0/1677226521 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b11be8fda0 con 0x55b11be0a850 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.516+0000 7fb16b7fe640 1 -- 172.21.15.33:0/1677226521 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb180032da0 con 0x55b11be0a850 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.516+0000 7fb16b7fe640 1 -- 172.21.15.33:0/1677226521 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fb184010a90 con 0x55b11be0a850 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.516+0000 7fb19463d740 10 client.26620 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:26:39.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.516+0000 7fb19463d740 10 client.26620 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:26:39.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.516+0000 7fb19463d740 20 client.26620 populate_metadata read hostname 'smithi033' 2024-09-16T00:26:39.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.516+0000 7fb19463d740 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b11be904f0 0x55b11be92910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:39.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.516+0000 7fb19463d740 4 client.26620 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}] 2024-09-16T00:26:39.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.516+0000 7fb19463d740 1 -- 172.21.15.33:0/1677226521 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}) -- 0x55b11be0e4e0 con 0x55b11be904f0 2024-09-16T00:26:39.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.516+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b11be904f0 0x55b11be92910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:39.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.517+0000 7fb16b7fe640 10 client.26620 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:39.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.517+0000 7fb18b7fe640 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b11be904f0 0x55b11be92910 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2c83f000622f0c2 server_cookie=d9ffc95067f7d2df in_seq=0 out_seq=0 2024-09-16T00:26:39.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb16b7fe640 1 -- 172.21.15.33:0/1677226521 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55b11be0e4e0 con 0x55b11be904f0 2024-09-16T00:26:39.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb16b7fe640 10 client.26620 handle_command_reply: tid=0 2024-09-16T00:26:39.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb19463d740 1 client.26620 shutdown 2024-09-16T00:26:39.611 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:39 smithi033 ceph-mon[31243]: pgmap v635: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-16T00:26:39.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb19463d740 1 -- 172.21.15.33:0/1677226521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 msgr2=0x55b11be2abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:39.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb19463d740 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 0x55b11be2abc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb184013ef0 tx=0x7fb184014040 comp rx=0 tx=0).stop 2024-09-16T00:26:39.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb19463d740 1 -- 172.21.15.33:0/1677226521 shutdown_connections 2024-09-16T00:26:39.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb19463d740 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55b11be904f0 0x55b11be92910 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb19463d740 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b11be15a30 0x55b11be28450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb19463d740 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b11be0b170 0x55b11be27f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb16b7fe640 0 client.26620 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:26:39.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb19463d740 1 --2- 172.21.15.33:0/1677226521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b11be0a850 0x55b11be2abc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb19463d740 1 -- 172.21.15.33:0/1677226521 >> 172.21.15.33:0/1677226521 conn(0x55b11be06de0 msgr2=0x55b11be20a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:39.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb19463d740 1 -- 172.21.15.33:0/1677226521 shutdown_connections 2024-09-16T00:26:39.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb19463d740 1 -- 172.21.15.33:0/1677226521 wait complete. 2024-09-16T00:26:39.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.610+0000 7fb19463d740 20 client.26620 trim_cache size 0 max 16384 2024-09-16T00:26:39.613 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:26:39.613 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:26:39.613 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:26:39.613 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 34, 2024-09-16T00:26:39.613 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:26:39.613 INFO:teuthology.orchestra.run.smithi033.stdout: "e0520840": { 2024-09-16T00:26:39.613 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 34, 2024-09-16T00:26:39.613 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:26:39.613 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:26:39.614 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:26:39.614 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:26:39.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:26:39.614 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:26:39.614 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 142.3, 2024-09-16T00:26:39.614 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:26:39.617 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:26:39.617 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:26:39.617 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:26:39.617 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:26:39.617 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:26:39.617 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:39.617 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:39.617 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:39.617 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:39.617 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:26:39.617 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:26:39.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.611+0000 7fb192264640 1 -- 172.21.15.33:0/1404974008 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb1740a3240 msgr2=0x7fb1740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:39.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.611+0000 7fb192264640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb1740a3240 0x7fb1740a5700 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb17c01f000 tx=0x7fb17c066d00 comp rx=0 tx=0).stop 2024-09-16T00:26:39.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.611+0000 7fb192264640 1 -- 172.21.15.33:0/1404974008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c102d40 msgr2=0x7fb18c19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:39.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.611+0000 7fb192264640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c102d40 0x7fb18c19a790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00ea60 tx=0x7fb17c00ef20 comp rx=0 tx=0).stop 2024-09-16T00:26:39.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.611+0000 7fb192264640 1 -- 172.21.15.33:0/1404974008 shutdown_connections 2024-09-16T00:26:39.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.611+0000 7fb192264640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb1740a3240 0x7fb1740a5700 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.611+0000 7fb192264640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb18c10c170 0x7fb18c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.611+0000 7fb192264640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb18c103660 0x7fb18c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.611+0000 7fb192264640 1 --2- 172.21.15.33:0/1404974008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c102d40 0x7fb18c19a790 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:39.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.611+0000 7fb192264640 1 -- 172.21.15.33:0/1404974008 >> 172.21.15.33:0/1404974008 conn(0x7fb18c0fc7a0 msgr2=0x7fb18c101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:39.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.611+0000 7fb192264640 1 -- 172.21.15.33:0/1404974008 shutdown_connections 2024-09-16T00:26:39.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:39.611+0000 7fb192264640 1 -- 172.21.15.33:0/1404974008 wait complete. 2024-09-16T00:26:39.673 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 10.02 MB/sec execute 347 sec latency 75.236 ms 2024-09-16T00:26:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:39 smithi083 ceph-mon[35242]: pgmap v635: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-16T00:26:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:39 smithi060 ceph-mon[35185]: pgmap v635: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-16T00:26:40.429 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'e0520840', quiesce took 4.6 seconds. Will release after: 17.75 2024-09-16T00:26:40.673 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.99 MB/sec execute 348 sec latency 1068.298 ms 2024-09-16T00:26:41.673 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.96 MB/sec execute 349 sec latency 2068.408 ms 2024-09-16T00:26:41.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:41 smithi033 ceph-mon[31243]: pgmap v636: 129 pgs: 129 active+clean; 269 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-16T00:26:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:41 smithi083 ceph-mon[35242]: pgmap v636: 129 pgs: 129 active+clean; 269 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-16T00:26:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:41 smithi060 ceph-mon[35185]: pgmap v636: 129 pgs: 129 active+clean; 269 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-16T00:26:42.673 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.93 MB/sec execute 350 sec latency 3068.521 ms 2024-09-16T00:26:42.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:42 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:26:42.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:42 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:26:42.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:42 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:26:42.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:42 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:26:42.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:42 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:26:42.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:42 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:26:43.673 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.90 MB/sec execute 351 sec latency 4068.579 ms 2024-09-16T00:26:43.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:43 smithi033 ceph-mon[31243]: pgmap v637: 129 pgs: 129 active+clean; 273 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 176 op/s 2024-09-16T00:26:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:43 smithi083 ceph-mon[35242]: pgmap v637: 129 pgs: 129 active+clean; 273 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 176 op/s 2024-09-16T00:26:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:43 smithi060 ceph-mon[35185]: pgmap v637: 129 pgs: 129 active+clean; 273 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 176 op/s 2024-09-16T00:26:44.673 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.87 MB/sec execute 352 sec latency 5068.695 ms 2024-09-16T00:26:45.674 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.85 MB/sec execute 353 sec latency 6068.772 ms 2024-09-16T00:26:45.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:45 smithi033 ceph-mon[31243]: pgmap v638: 129 pgs: 129 active+clean; 272 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 179 op/s 2024-09-16T00:26:45.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:45 smithi083 ceph-mon[35242]: pgmap v638: 129 pgs: 129 active+clean; 272 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 179 op/s 2024-09-16T00:26:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:45 smithi060 ceph-mon[35185]: pgmap v638: 129 pgs: 129 active+clean; 272 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 179 op/s 2024-09-16T00:26:46.674 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.82 MB/sec execute 354 sec latency 7068.873 ms 2024-09-16T00:26:47.674 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.79 MB/sec execute 355 sec latency 8068.969 ms 2024-09-16T00:26:47.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:47 smithi033 ceph-mon[31243]: pgmap v639: 129 pgs: 129 active+clean; 275 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 173 op/s 2024-09-16T00:26:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:47 smithi083 ceph-mon[35242]: pgmap v639: 129 pgs: 129 active+clean; 275 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 173 op/s 2024-09-16T00:26:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:47 smithi060 ceph-mon[35185]: pgmap v639: 129 pgs: 129 active+clean; 275 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 173 op/s 2024-09-16T00:26:48.674 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.76 MB/sec execute 356 sec latency 9069.072 ms 2024-09-16T00:26:49.674 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.74 MB/sec execute 357 sec latency 10069.150 ms 2024-09-16T00:26:49.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:49 smithi033 ceph-mon[31243]: pgmap v640: 129 pgs: 129 active+clean; 276 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 144 op/s 2024-09-16T00:26:49.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:49 smithi083 ceph-mon[35242]: pgmap v640: 129 pgs: 129 active+clean; 276 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 144 op/s 2024-09-16T00:26:49.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:49 smithi060 ceph-mon[35185]: pgmap v640: 129 pgs: 129 active+clean; 276 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 144 op/s 2024-09-16T00:26:50.674 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.71 MB/sec execute 358 sec latency 11069.222 ms 2024-09-16T00:26:51.674 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.68 MB/sec execute 359 sec latency 12069.320 ms 2024-09-16T00:26:51.782 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:51 smithi060 ceph-mon[35185]: pgmap v641: 129 pgs: 129 active+clean; 275 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 148 op/s 2024-09-16T00:26:51.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:51 smithi033 ceph-mon[31243]: pgmap v641: 129 pgs: 129 active+clean; 275 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 148 op/s 2024-09-16T00:26:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:51 smithi083 ceph-mon[35242]: pgmap v641: 129 pgs: 129 active+clean; 275 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 148 op/s 2024-09-16T00:26:52.674 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.65 MB/sec execute 360 sec latency 13069.436 ms 2024-09-16T00:26:53.674 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.63 MB/sec execute 361 sec latency 14069.542 ms 2024-09-16T00:26:53.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:53 smithi033 ceph-mon[31243]: pgmap v642: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 132 op/s 2024-09-16T00:26:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:53 smithi083 ceph-mon[35242]: pgmap v642: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 132 op/s 2024-09-16T00:26:53.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:53 smithi060 ceph-mon[35185]: pgmap v642: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 132 op/s 2024-09-16T00:26:54.674 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.60 MB/sec execute 362 sec latency 15069.625 ms 2024-09-16T00:26:55.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.58 MB/sec execute 363 sec latency 16069.728 ms 2024-09-16T00:26:55.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:55 smithi033 ceph-mon[31243]: pgmap v643: 129 pgs: 129 active+clean; 269 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 993 KiB/s wr, 50 op/s 2024-09-16T00:26:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:55 smithi083 ceph-mon[35242]: pgmap v643: 129 pgs: 129 active+clean; 269 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 993 KiB/s wr, 50 op/s 2024-09-16T00:26:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:55 smithi060 ceph-mon[35185]: pgmap v643: 129 pgs: 129 active+clean; 269 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 993 KiB/s wr, 50 op/s 2024-09-16T00:26:56.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.55 MB/sec execute 364 sec latency 17069.905 ms 2024-09-16T00:26:57.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.52 MB/sec execute 365 sec latency 18069.947 ms 2024-09-16T00:26:57.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:57 smithi033 ceph-mon[31243]: pgmap v644: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 620 KiB/s wr, 32 op/s 2024-09-16T00:26:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:57 smithi083 ceph-mon[35242]: pgmap v644: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 620 KiB/s wr, 32 op/s 2024-09-16T00:26:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:57 smithi060 ceph-mon[35185]: pgmap v644: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 620 KiB/s wr, 32 op/s 2024-09-16T00:26:58.180 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:26:58.452 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:26:58.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.50 MB/sec execute 366 sec latency 19070.005 ms 2024-09-16T00:26:58.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.949+0000 7fa4285e0640 1 -- 172.21.15.33:0/2445174708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa420102ef0 msgr2=0x7fa42010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:58.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.949+0000 7fa4285e0640 1 --2- 172.21.15.33:0/2445174708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa420102ef0 0x7fa42010da40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa418009ca0 tx=0x7fa41802a950 comp rx=0 tx=0).stop 2024-09-16T00:26:58.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 -- 172.21.15.33:0/2445174708 shutdown_connections 2024-09-16T00:26:58.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 --2- 172.21.15.33:0/2445174708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa420102ef0 0x7fa42010da40 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:58.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 --2- 172.21.15.33:0/2445174708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa420102570 0x7fa4201029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:58.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 --2- 172.21.15.33:0/2445174708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa420100e20 0x7fa420101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:58.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 -- 172.21.15.33:0/2445174708 >> 172.21.15.33:0/2445174708 conn(0x7fa4200fc7d0 msgr2=0x7fa4200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:58.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 -- 172.21.15.33:0/2445174708 shutdown_connections 2024-09-16T00:26:58.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 -- 172.21.15.33:0/2445174708 wait complete. 2024-09-16T00:26:58.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 Processor -- start 2024-09-16T00:26:58.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 -- start start 2024-09-16T00:26:58.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa420100e20 0x7fa4201984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:58.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa420102570 0x7fa4201989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:58.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa420102ef0 0x7fa42019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:58.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa4201120a0 con 0x7fa420100e20 2024-09-16T00:26:58.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa420111f20 con 0x7fa420102570 2024-09-16T00:26:58.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.950+0000 7fa4285e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa420112220 con 0x7fa420102ef0 2024-09-16T00:26:58.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa42637c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa420100e20 0x7fa4201984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa42637c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa420100e20 0x7fa4201984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48432/0 (socket says 172.21.15.33:48432) 2024-09-16T00:26:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa42637c640 1 -- 172.21.15.33:0/1562093482 learned_addr learned my addr 172.21.15.33:0/1562093482 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:26:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa425b7b640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa420102570 0x7fa4201989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa426b7d640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa420102ef0 0x7fa42019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa42637c640 1 -- 172.21.15.33:0/1562093482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa420102ef0 msgr2=0x7fa42019cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa42637c640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa420102ef0 0x7fa42019cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa42637c640 1 -- 172.21.15.33:0/1562093482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa420102570 msgr2=0x7fa4201989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa42637c640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa420102570 0x7fa4201989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa42637c640 1 -- 172.21.15.33:0/1562093482 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42019d450 con 0x7fa420100e20 2024-09-16T00:26:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa425b7b640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa420102570 0x7fa4201989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:26:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa42637c640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa420100e20 0x7fa4201984a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00d950 tx=0x7fa41c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa4177fe640 1 -- 172.21.15.33:0/1562093482 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c017030 con 0x7fa420100e20 2024-09-16T00:26:58.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa4177fe640 1 -- 172.21.15.33:0/1562093482 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa41c004020 con 0x7fa420100e20 2024-09-16T00:26:58.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa4285e0640 1 -- 172.21.15.33:0/1562093482 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42019d740 con 0x7fa420100e20 2024-09-16T00:26:58.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa4177fe640 1 -- 172.21.15.33:0/1562093482 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c0047a0 con 0x7fa420100e20 2024-09-16T00:26:58.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.951+0000 7fa4285e0640 1 -- 172.21.15.33:0/1562093482 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4201a5020 con 0x7fa420100e20 2024-09-16T00:26:58.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.952+0000 7fa4285e0640 1 -- 172.21.15.33:0/1562093482 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa420101200 con 0x7fa420100e20 2024-09-16T00:26:58.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.953+0000 7fa4177fe640 1 -- 172.21.15.33:0/1562093482 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa41c0041d0 con 0x7fa420100e20 2024-09-16T00:26:58.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.954+0000 7fa4177fe640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa40c0a3240 0x7fa40c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:26:58.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.954+0000 7fa425b7b640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa40c0a3240 0x7fa40c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:26:58.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.954+0000 7fa4177fe640 1 -- 172.21.15.33:0/1562093482 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa41c0cf910 con 0x7fa420100e20 2024-09-16T00:26:58.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.955+0000 7fa425b7b640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa40c0a3240 0x7fa40c0a5700 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa410005fd0 tx=0x7fa41000b5d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:26:58.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:58.958+0000 7fa4177fe640 1 -- 172.21.15.33:0/1562093482 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa41c09c150 con 0x7fa420100e20 2024-09-16T00:26:59.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.171+0000 7fa4285e0640 1 -- 172.21.15.33:0/1562093482 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4201a51f0 con 0x7fa420100e20 2024-09-16T00:26:59.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.172+0000 7fa4177fe640 1 -- 172.21.15.33:0/1562093482 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7fa41c09ba50 con 0x7fa420100e20 2024-09-16T00:26:59.173 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:26:59.174 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:26:59.174 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:26:59.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.175+0000 7fa4285e0640 1 -- 172.21.15.33:0/1562093482 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa40c0a3240 msgr2=0x7fa40c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:59.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.175+0000 7fa4285e0640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa40c0a3240 0x7fa40c0a5700 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa410005fd0 tx=0x7fa41000b5d0 comp rx=0 tx=0).stop 2024-09-16T00:26:59.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.175+0000 7fa4285e0640 1 -- 172.21.15.33:0/1562093482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa420100e20 msgr2=0x7fa4201984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:26:59.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.175+0000 7fa4285e0640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa420100e20 0x7fa4201984a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00d950 tx=0x7fa41c00de10 comp rx=0 tx=0).stop 2024-09-16T00:26:59.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.175+0000 7fa4285e0640 1 -- 172.21.15.33:0/1562093482 shutdown_connections 2024-09-16T00:26:59.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.175+0000 7fa4285e0640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa40c0a3240 0x7fa40c0a5700 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:59.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.175+0000 7fa4285e0640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa420102ef0 0x7fa42019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:59.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.175+0000 7fa4285e0640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa420102570 0x7fa4201989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:59.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.175+0000 7fa4285e0640 1 --2- 172.21.15.33:0/1562093482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa420100e20 0x7fa4201984a0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:26:59.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.175+0000 7fa4285e0640 1 -- 172.21.15.33:0/1562093482 >> 172.21.15.33:0/1562093482 conn(0x7fa4200fc7d0 msgr2=0x7fa420106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:26:59.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.175+0000 7fa4285e0640 1 -- 172.21.15.33:0/1562093482 shutdown_connections 2024-09-16T00:26:59.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:26:59.175+0000 7fa4285e0640 1 -- 172.21.15.33:0/1562093482 wait complete. 2024-09-16T00:26:59.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.47 MB/sec execute 367 sec latency 20070.104 ms 2024-09-16T00:26:59.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:59 smithi033 ceph-mon[31243]: pgmap v645: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 269 KiB/s wr, 22 op/s 2024-09-16T00:26:59.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:26:59 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1562093482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:59.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:59 smithi083 ceph-mon[35242]: pgmap v645: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 269 KiB/s wr, 22 op/s 2024-09-16T00:26:59.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:26:59 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1562093482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:26:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:59 smithi060 ceph-mon[35185]: pgmap v645: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 269 KiB/s wr, 22 op/s 2024-09-16T00:26:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:26:59 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1562093482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:27:00.016 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id e0520840 --release --await' 2024-09-16T00:27:00.016 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id e0520840 --release --await 2024-09-16T00:27:00.276 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:00.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 759985 9.45 MB/sec execute 368 sec latency 21070.154 ms 2024-09-16T00:27:00.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.758+0000 7f39f272a640 1 -- 172.21.15.33:0/1451683791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39ec10c180 msgr2=0x7f39ec10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:00.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.759+0000 7f39f272a640 1 --2- 172.21.15.33:0/1451683791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39ec10c180 0x7f39ec10e610 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f39e4009ca0 tx=0x7f39e402a950 comp rx=0 tx=0).stop 2024-09-16T00:27:00.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.759+0000 7f39f272a640 1 -- 172.21.15.33:0/1451683791 shutdown_connections 2024-09-16T00:27:00.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.759+0000 7f39f272a640 1 --2- 172.21.15.33:0/1451683791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39ec10c180 0x7f39ec10e610 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.759+0000 7f39f272a640 1 --2- 172.21.15.33:0/1451683791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec103700 0x7f39ec10bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.759+0000 7f39f272a640 1 --2- 172.21.15.33:0/1451683791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ec102de0 0x7f39ec1031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.759+0000 7f39f272a640 1 -- 172.21.15.33:0/1451683791 >> 172.21.15.33:0/1451683791 conn(0x7f39ec0fc7a0 msgr2=0x7f39ec0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:00.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.759+0000 7f39f272a640 1 -- 172.21.15.33:0/1451683791 shutdown_connections 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.759+0000 7f39f272a640 1 -- 172.21.15.33:0/1451683791 wait complete. 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.759+0000 7f39f272a640 1 Processor -- start 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.759+0000 7f39f272a640 1 -- start start 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39f272a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39ec102de0 0x7f39ec198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39f272a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ec103700 0x7f39ec198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39f272a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec10c180 0x7f39ec19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39f272a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec112030 con 0x7f39ec102de0 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39f272a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec111eb0 con 0x7f39ec10c180 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39f272a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec1121b0 con 0x7f39ec103700 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39ec102de0 0x7f39ec198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39ec102de0 0x7f39ec198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51542/0 (socket says 172.21.15.33:51542) 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39ebfff640 1 -- 172.21.15.33:0/1720119565 learned_addr learned my addr 172.21.15.33:0/1720119565 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:00.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39f0cc7640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec10c180 0x7f39ec19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39eb7fe640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ec103700 0x7f39ec198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39ebfff640 1 -- 172.21.15.33:0/1720119565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ec103700 msgr2=0x7f39ec198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39ebfff640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ec103700 0x7f39ec198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39ebfff640 1 -- 172.21.15.33:0/1720119565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec10c180 msgr2=0x7f39ec19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39ebfff640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec10c180 0x7f39ec19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39ebfff640 1 -- 172.21.15.33:0/1720119565 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ec19d500 con 0x7f39ec102de0 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39f0cc7640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec10c180 0x7f39ec19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39eb7fe640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ec103700 0x7f39ec198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39ebfff640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39ec102de0 0x7f39ec198550 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00b4a0 tx=0x7f39dc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.760+0000 7f39e97fa640 1 -- 172.21.15.33:0/1720119565 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc014070 con 0x7f39ec102de0 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.761+0000 7f39f272a640 1 -- 172.21.15.33:0/1720119565 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39ec19d7f0 con 0x7f39ec102de0 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.761+0000 7f39e97fa640 1 -- 172.21.15.33:0/1720119565 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39dc010af0 con 0x7f39ec102de0 2024-09-16T00:27:00.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.761+0000 7f39e97fa640 1 -- 172.21.15.33:0/1720119565 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc017d10 con 0x7f39ec102de0 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.761+0000 7f39f272a640 1 -- 172.21.15.33:0/1720119565 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39ec1a50d0 con 0x7f39ec102de0 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.763+0000 7f39e97fa640 1 -- 172.21.15.33:0/1720119565 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f39dc017070 con 0x7f39ec102de0 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.763+0000 7f39f4b03740 1 Processor -- start 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.763+0000 7f39f4b03740 1 -- start start 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.763+0000 7f39f4b03740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45060 0x55760ab45440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39f4b03740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab45980 0x55760ab4fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39f4b03740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab50240 0x55760ab52630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39f4b03740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55760aaaa620 con 0x55760ab50240 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39f4b03740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55760aa51e50 con 0x55760ab45060 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39f4b03740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55760aaa65e0 con 0x55760ab45980 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39eb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab50240 0x55760ab52630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39eb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab50240 0x55760ab52630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51546/0 (socket says 172.21.15.33:51546) 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39e97fa640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f39d40a3240 0x7f39d40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39eb7fe640 1 -- 172.21.15.33:0/1479046822 learned_addr learned my addr 172.21.15.33:0/1479046822 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39f0cc7640 1 --2- 172.21.15.33:0/1479046822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45060 0x55760ab45440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39f0cc7640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f39d40a3240 0x7f39d40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39e97fa640 1 -- 172.21.15.33:0/1720119565 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f39dc0cf280 con 0x7f39ec102de0 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39ebfff640 1 --2- 172.21.15.33:0/1479046822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab45980 0x55760ab4fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39eb7fe640 1 -- 172.21.15.33:0/1479046822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab45980 msgr2=0x55760ab4fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39eb7fe640 1 --2- 172.21.15.33:0/1479046822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab45980 0x55760ab4fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39eb7fe640 1 -- 172.21.15.33:0/1479046822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45060 msgr2=0x55760ab45440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39eb7fe640 1 --2- 172.21.15.33:0/1479046822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45060 0x55760ab45440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39eb7fe640 1 -- 172.21.15.33:0/1479046822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55760aa42d40 con 0x55760ab50240 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39f0cc7640 1 --2- 172.21.15.33:0/1479046822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45060 0x55760ab45440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39eb7fe640 1 --2- 172.21.15.33:0/1479046822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab50240 0x55760ab52630 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f39e000a9d0 tx=0x7f39e000ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=626d6c7f722806ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39f0cc7640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f39d40a3240 0x7f39d40a5700 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f39e4030040 tx=0x7f39e40095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:00.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.764+0000 7f39d27fc640 1 -- 172.21.15.33:0/1479046822 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e0015070 con 0x55760ab50240 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.765+0000 7f39d27fc640 1 -- 172.21.15.33:0/1479046822 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39e000cc60 con 0x55760ab50240 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.765+0000 7f39d27fc640 1 -- 172.21.15.33:0/1479046822 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e00185c0 con 0x55760ab50240 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.765+0000 7f39f4b03740 1 -- 172.21.15.33:0/1479046822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab50240 msgr2=0x55760ab52630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.765+0000 7f39f4b03740 1 --2- 172.21.15.33:0/1479046822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab50240 0x55760ab52630 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f39e000a9d0 tx=0x7f39e000ae90 comp rx=0 tx=0).stop 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.765+0000 7f39f4b03740 1 -- 172.21.15.33:0/1479046822 shutdown_connections 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.765+0000 7f39f4b03740 1 --2- 172.21.15.33:0/1479046822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab50240 0x55760ab52630 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.765+0000 7f39f4b03740 1 --2- 172.21.15.33:0/1479046822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab45980 0x55760ab4fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.765+0000 7f39f4b03740 1 --2- 172.21.15.33:0/1479046822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45060 0x55760ab45440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.765+0000 7f39f4b03740 1 -- 172.21.15.33:0/1479046822 >> 172.21.15.33:0/1479046822 conn(0x55760ab415f0 msgr2=0x55760ab419d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.765+0000 7f39f4b03740 1 -- 172.21.15.33:0/1479046822 shutdown_connections 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.765+0000 7f39f4b03740 1 -- 172.21.15.33:0/1479046822 wait complete. 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39f4b03740 1 -- start start 2024-09-16T00:27:00.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39f4b03740 1 Processor -- start 2024-09-16T00:27:00.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39f4b03740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab4d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39f4b03740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45980 0x55760ab4d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39f4b03740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab50240 0x55760ab4dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39f4b03740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55760ab52e10 con 0x55760ab45060 2024-09-16T00:27:00.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39f4b03740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55760aa32f10 con 0x55760ab45980 2024-09-16T00:27:00.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39f4b03740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55760ab52f90 con 0x55760ab50240 2024-09-16T00:27:00.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39f4b03740 10 client.0 _ll_register_callbacks cb 0x55760aa56db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:00.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab4d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab4d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51550/0 (socket says 172.21.15.33:51550) 2024-09-16T00:27:00.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39ebfff640 1 -- 172.21.15.33:0/4177323907 learned_addr learned my addr 172.21.15.33:0/4177323907 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:00.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.766+0000 7f39f0cc7640 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab50240 0x55760ab4dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39eb7fe640 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45980 0x55760ab4d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39ebfff640 1 -- 172.21.15.33:0/4177323907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab50240 msgr2=0x55760ab4dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39ebfff640 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab50240 0x55760ab4dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39ebfff640 1 -- 172.21.15.33:0/4177323907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45980 msgr2=0x55760ab4d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39ebfff640 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45980 0x55760ab4d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39ebfff640 1 -- 172.21.15.33:0/4177323907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55760a7b9580 con 0x55760ab45060 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39f0cc7640 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab50240 0x55760ab4dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39ebfff640 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab4d060 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009420 tx=0x7f39dc017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4cbc41fa84629f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39cb7fe640 1 -- 172.21.15.33:0/4177323907 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc01e070 con 0x55760ab45060 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39cb7fe640 1 -- 172.21.15.33:0/4177323907 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f39dc097a90 con 0x55760ab45060 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39f4b03740 1 -- 172.21.15.33:0/4177323907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39e000cc60 con 0x55760ab45060 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39cb7fe640 1 -- 172.21.15.33:0/4177323907 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f39dc09f7d0 con 0x55760ab45060 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39f4b03740 10 client.17226 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:00.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39f4b03740 1 -- 172.21.15.33:0/4177323907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55760a883960 con 0x55760ab45060 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.767+0000 7f39cb7fe640 1 -- 172.21.15.33:0/4177323907 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39e000cc60 con 0x55760ab45060 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.768+0000 7f39cb7fe640 1 -- 172.21.15.33:0/4177323907 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f39dc0b41e0 con 0x55760ab45060 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.768+0000 7f39f4b03740 10 client.17226 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.768+0000 7f39f4b03740 10 client.17226 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.768+0000 7f39f4b03740 20 client.17226 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.768+0000 7f39f4b03740 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55760ab5fb70 0x55760ab61fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.768+0000 7f39f4b03740 4 client.17226 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.768+0000 7f39f4b03740 1 -- 172.21.15.33:0/4177323907 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55760aab7b90 con 0x55760ab5fb70 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.768+0000 7f39eb7fe640 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55760ab5fb70 0x55760ab61fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.768+0000 7f39cb7fe640 10 client.17226 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.768+0000 7f39eb7fe640 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55760ab5fb70 0x55760ab61fe0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e99ca810b152efa server_cookie=c5ee4487a3f4812f in_seq=0 out_seq=0 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39cb7fe640 1 -- 172.21.15.33:0/4177323907 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55760aab7b90 con 0x55760ab5fb70 2024-09-16T00:27:00.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39cb7fe640 10 client.17226 handle_command_reply: tid=0 2024-09-16T00:27:00.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39f4b03740 1 client.17226 shutdown 2024-09-16T00:27:00.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39f4b03740 1 -- 172.21.15.33:0/4177323907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 msgr2=0x55760ab4d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:00.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39f4b03740 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab4d060 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009420 tx=0x7f39dc017730 comp rx=0 tx=0).stop 2024-09-16T00:27:00.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39f4b03740 1 -- 172.21.15.33:0/4177323907 shutdown_connections 2024-09-16T00:27:00.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39f4b03740 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55760ab5fb70 0x55760ab61fe0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39f4b03740 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab50240 0x55760ab4dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39f4b03740 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45980 0x55760ab4d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39f4b03740 1 --2- 172.21.15.33:0/4177323907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab4d060 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39cb7fe640 0 client.17226 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:00.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39f4b03740 1 -- 172.21.15.33:0/4177323907 >> 172.21.15.33:0/4177323907 conn(0x55760ab415f0 msgr2=0x55760ab472d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:00.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.769+0000 7f39f4b03740 1 -- 172.21.15.33:0/4177323907 shutdown_connections 2024-09-16T00:27:00.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.770+0000 7f39f4b03740 1 -- 172.21.15.33:0/4177323907 wait complete. 2024-09-16T00:27:00.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.770+0000 7f39f4b03740 20 client.17226 trim_cache size 0 max 16384 2024-09-16T00:27:00.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39f4b03740 1 Processor -- start 2024-09-16T00:27:00.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39f4b03740 1 -- start start 2024-09-16T00:27:00.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39f4b03740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab47ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39f4b03740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45980 0x55760ab40640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39f4b03740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab50240 0x55760ab67880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39f4b03740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec1b1a80 con 0x55760ab45060 2024-09-16T00:27:00.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39f4b03740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec1af350 con 0x55760ab45980 2024-09-16T00:27:00.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39f4b03740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec1ad1a0 con 0x55760ab50240 2024-09-16T00:27:00.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab47ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab47ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51562/0 (socket says 172.21.15.33:51562) 2024-09-16T00:27:00.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39ebfff640 1 -- 172.21.15.33:0/799965069 learned_addr learned my addr 172.21.15.33:0/799965069 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:00.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39eb7fe640 1 --2- 172.21.15.33:0/799965069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45980 0x55760ab40640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39f0cc7640 1 --2- 172.21.15.33:0/799965069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab50240 0x55760ab67880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39ebfff640 1 -- 172.21.15.33:0/799965069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab50240 msgr2=0x55760ab67880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:00.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39ebfff640 1 --2- 172.21.15.33:0/799965069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab50240 0x55760ab67880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39ebfff640 1 -- 172.21.15.33:0/799965069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45980 msgr2=0x55760ab40640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:00.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39ebfff640 1 --2- 172.21.15.33:0/799965069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45980 0x55760ab40640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39ebfff640 1 -- 172.21.15.33:0/799965069 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55760ab41420 con 0x55760ab45060 2024-09-16T00:27:00.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.787+0000 7f39ebfff640 1 --2- 172.21.15.33:0/799965069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab47ac0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0ccc70 tx=0x7f39dc09b250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295170691e10e708 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:00.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.788+0000 7f39d1ffb640 1 -- 172.21.15.33:0/799965069 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc01e070 con 0x55760ab45060 2024-09-16T00:27:00.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.788+0000 7f39d1ffb640 1 -- 172.21.15.33:0/799965069 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39dc0a4090 con 0x55760ab45060 2024-09-16T00:27:00.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.788+0000 7f39d1ffb640 1 -- 172.21.15.33:0/799965069 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc010db0 con 0x55760ab45060 2024-09-16T00:27:00.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.788+0000 7f39f4b03740 1 -- 172.21.15.33:0/799965069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 msgr2=0x55760ab47ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:00.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.788+0000 7f39f4b03740 1 --2- 172.21.15.33:0/799965069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab47ac0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0ccc70 tx=0x7f39dc09b250 comp rx=0 tx=0).stop 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.788+0000 7f39f4b03740 1 -- 172.21.15.33:0/799965069 shutdown_connections 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.788+0000 7f39f4b03740 1 --2- 172.21.15.33:0/799965069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab50240 0x55760ab67880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.788+0000 7f39f4b03740 1 --2- 172.21.15.33:0/799965069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab45980 0x55760ab40640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.788+0000 7f39f4b03740 1 --2- 172.21.15.33:0/799965069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab47ac0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.788+0000 7f39f4b03740 1 -- 172.21.15.33:0/799965069 >> 172.21.15.33:0/799965069 conn(0x55760ab415f0 msgr2=0x55760ab3ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.788+0000 7f39f4b03740 1 -- 172.21.15.33:0/799965069 shutdown_connections 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.788+0000 7f39f4b03740 1 -- 172.21.15.33:0/799965069 wait complete. 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.789+0000 7f39f4b03740 1 -- start start 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.789+0000 7f39f4b03740 1 Processor -- start 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.789+0000 7f39f4b03740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.789+0000 7f39f4b03740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab45980 0x55760ab58720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.789+0000 7f39f4b03740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab50240 0x55760ab551b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.789+0000 7f39f4b03740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec1ad810 con 0x55760ab45060 2024-09-16T00:27:00.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.789+0000 7f39f4b03740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec1aec80 con 0x55760ab50240 2024-09-16T00:27:00.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.789+0000 7f39f4b03740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec1adee0 con 0x55760ab45980 2024-09-16T00:27:00.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.789+0000 7f39f4b03740 10 client.0 _ll_register_callbacks cb 0x7f39dc09b050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:00.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.789+0000 7f39ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.789+0000 7f39ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51568/0 (socket says 172.21.15.33:51568) 2024-09-16T00:27:00.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39ebfff640 1 -- 172.21.15.33:0/2518751475 learned_addr learned my addr 172.21.15.33:0/2518751475 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:00.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39f0cc7640 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab50240 0x55760ab551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39eb7fe640 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab45980 0x55760ab58720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39ebfff640 1 -- 172.21.15.33:0/2518751475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab45980 msgr2=0x55760ab58720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39ebfff640 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab45980 0x55760ab58720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39ebfff640 1 -- 172.21.15.33:0/2518751475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab50240 msgr2=0x55760ab551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39ebfff640 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab50240 0x55760ab551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39ebfff640 1 -- 172.21.15.33:0/2518751475 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55760ab42280 con 0x55760ab45060 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39f0cc7640 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab50240 0x55760ab551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39ebfff640 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab581e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0996d0 tx=0x7f39dc017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c720193cffce01 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39cb7fe640 1 -- 172.21.15.33:0/2518751475 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc01e070 con 0x55760ab45060 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39cb7fe640 1 -- 172.21.15.33:0/2518751475 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f39dc0b4150 con 0x55760ab45060 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39f4b03740 1 -- 172.21.15.33:0/2518751475 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39dc0a4090 con 0x55760ab45060 2024-09-16T00:27:00.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39cb7fe640 1 -- 172.21.15.33:0/2518751475 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f39dc0b4350 con 0x55760ab45060 2024-09-16T00:27:00.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39f4b03740 10 client.17232 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:00.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.790+0000 7f39f4b03740 1 -- 172.21.15.33:0/2518751475 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55760abcb000 con 0x55760ab45060 2024-09-16T00:27:00.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.791+0000 7f39cb7fe640 1 -- 172.21.15.33:0/2518751475 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39dc0a4090 con 0x55760ab45060 2024-09-16T00:27:00.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.791+0000 7f39cb7fe640 1 -- 172.21.15.33:0/2518751475 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f39dc09a4a0 con 0x55760ab45060 2024-09-16T00:27:00.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.791+0000 7f39f4b03740 10 client.17232 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:00.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.791+0000 7f39f4b03740 10 client.17232 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:27:00.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.791+0000 7f39f4b03740 20 client.17232 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:00.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.791+0000 7f39f4b03740 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55760abcb4d0 0x55760abcd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:00.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.791+0000 7f39f4b03740 4 client.17232 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e0520840", "release": true, "await": true}] 2024-09-16T00:27:00.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.791+0000 7f39f4b03740 1 -- 172.21.15.33:0/2518751475 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e0520840", "release": true, "await": true}) -- 0x55760aab20a0 con 0x55760abcb4d0 2024-09-16T00:27:00.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.791+0000 7f39eb7fe640 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55760abcb4d0 0x55760abcd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:00.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.791+0000 7f39cb7fe640 10 client.17232 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:00.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.791+0000 7f39eb7fe640 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55760abcb4d0 0x55760abcd8c0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c58315cd2c24134 server_cookie=2304ba6f8ac599f3 in_seq=0 out_seq=0 2024-09-16T00:27:00.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.808+0000 7f39cb7fe640 1 -- 172.21.15.33:0/2518751475 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55760aab20a0 con 0x55760abcb4d0 2024-09-16T00:27:00.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.808+0000 7f39cb7fe640 10 client.17232 handle_command_reply: tid=0 2024-09-16T00:27:00.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.808+0000 7f39f4b03740 1 client.17232 shutdown 2024-09-16T00:27:00.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.808+0000 7f39f4b03740 1 -- 172.21.15.33:0/2518751475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 msgr2=0x55760ab581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:00.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.808+0000 7f39f4b03740 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab581e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0996d0 tx=0x7f39dc017730 comp rx=0 tx=0).stop 2024-09-16T00:27:00.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f4b03740 1 -- 172.21.15.33:0/2518751475 shutdown_connections 2024-09-16T00:27:00.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f4b03740 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55760abcb4d0 0x55760abcd8c0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f4b03740 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55760ab50240 0x55760ab551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f4b03740 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55760ab45980 0x55760ab58720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f4b03740 1 --2- 172.21.15.33:0/2518751475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55760ab45060 0x55760ab581e0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39cb7fe640 0 client.17232 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:00.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f4b03740 1 -- 172.21.15.33:0/2518751475 >> 172.21.15.33:0/2518751475 conn(0x55760ab415f0 msgr2=0x55760ab4d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:00.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f4b03740 1 -- 172.21.15.33:0/2518751475 shutdown_connections 2024-09-16T00:27:00.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f4b03740 1 -- 172.21.15.33:0/2518751475 wait complete. 2024-09-16T00:27:00.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f4b03740 20 client.17232 trim_cache size 0 max 16384 2024-09-16T00:27:00.811 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:27:00.812 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:27:00.812 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:27:00.812 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 36, 2024-09-16T00:27:00.812 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:27:00.812 INFO:teuthology.orchestra.run.smithi033.stdout: "e0520840": { 2024-09-16T00:27:00.812 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 36, 2024-09-16T00:27:00.812 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:27:00.812 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:27:00.812 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:27:00.812 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:27:00.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:27:00.812 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 142.3, 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:27:00.813 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:27:00.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f272a640 1 -- 172.21.15.33:0/1720119565 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f39d40a3240 msgr2=0x7f39d40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:00.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f272a640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f39d40a3240 0x7f39d40a5700 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f39e4030040 tx=0x7f39e40095c0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f272a640 1 -- 172.21.15.33:0/1720119565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39ec102de0 msgr2=0x7f39ec198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:00.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.809+0000 7f39f272a640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39ec102de0 0x7f39ec198550 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00b4a0 tx=0x7f39dc00b960 comp rx=0 tx=0).stop 2024-09-16T00:27:00.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.810+0000 7f39f272a640 1 -- 172.21.15.33:0/1720119565 shutdown_connections 2024-09-16T00:27:00.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.810+0000 7f39f272a640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f39d40a3240 0x7f39d40a5700 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.810+0000 7f39f272a640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec10c180 0x7f39ec19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.810+0000 7f39f272a640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ec103700 0x7f39ec198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.810+0000 7f39f272a640 1 --2- 172.21.15.33:0/1720119565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39ec102de0 0x7f39ec198550 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:00.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.810+0000 7f39f272a640 1 -- 172.21.15.33:0/1720119565 >> 172.21.15.33:0/1720119565 conn(0x7f39ec0fc7a0 msgr2=0x7f39ec0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:00.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.810+0000 7f39f272a640 1 -- 172.21.15.33:0/1720119565 shutdown_connections 2024-09-16T00:27:00.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:00.810+0000 7f39f272a640 1 -- 172.21.15.33:0/1720119565 wait complete. 2024-09-16T00:27:01.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 762243 9.44 MB/sec execute 369 sec latency 21205.333 ms 2024-09-16T00:27:01.733 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'e0520840', total seconds elapsed: 25.9 2024-09-16T00:27:01.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:01 smithi033 ceph-mon[31243]: pgmap v646: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 193 KiB/s wr, 22 op/s 2024-09-16T00:27:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:01 smithi083 ceph-mon[35242]: pgmap v646: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 193 KiB/s wr, 22 op/s 2024-09-16T00:27:01.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:01 smithi060 ceph-mon[35185]: pgmap v646: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 193 KiB/s wr, 22 op/s 2024-09-16T00:27:02.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 764482 9.45 MB/sec execute 370 sec latency 15.311 ms 2024-09-16T00:27:03.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 767549 9.48 MB/sec execute 371 sec latency 11.221 ms 2024-09-16T00:27:03.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:03 smithi033 ceph-mon[31243]: pgmap v647: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 13 op/s 2024-09-16T00:27:03.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:03 smithi083 ceph-mon[35242]: pgmap v647: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 13 op/s 2024-09-16T00:27:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:03 smithi060 ceph-mon[35185]: pgmap v647: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 13 op/s 2024-09-16T00:27:04.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 770728 9.50 MB/sec execute 372 sec latency 15.878 ms 2024-09-16T00:27:05.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 773479 9.51 MB/sec execute 373 sec latency 20.641 ms 2024-09-16T00:27:05.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:05 smithi033 ceph-mon[31243]: pgmap v648: 129 pgs: 129 active+clean; 272 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 240 KiB/s wr, 39 op/s 2024-09-16T00:27:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:05 smithi083 ceph-mon[35242]: pgmap v648: 129 pgs: 129 active+clean; 272 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 240 KiB/s wr, 39 op/s 2024-09-16T00:27:05.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:05 smithi060 ceph-mon[35185]: pgmap v648: 129 pgs: 129 active+clean; 272 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 240 KiB/s wr, 39 op/s 2024-09-16T00:27:06.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 775752 9.52 MB/sec execute 374 sec latency 12.303 ms 2024-09-16T00:27:07.096 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:27:07.384 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:07.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 778514 9.53 MB/sec execute 375 sec latency 15.868 ms 2024-09-16T00:27:07.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:07 smithi033 ceph-mon[31243]: pgmap v649: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 440 KiB/s wr, 58 op/s 2024-09-16T00:27:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:07 smithi083 ceph-mon[35242]: pgmap v649: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 440 KiB/s wr, 58 op/s 2024-09-16T00:27:07.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.928+0000 7f1a8ab83640 1 -- 172.21.15.33:0/542837691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a84100a80 msgr2=0x7f1a84100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:07.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.928+0000 7f1a8ab83640 1 --2- 172.21.15.33:0/542837691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a84100a80 0x7f1a84100ee0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009930 tx=0x7f1a7002a420 comp rx=0 tx=0).stop 2024-09-16T00:27:07.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.928+0000 7f1a8ab83640 1 -- 172.21.15.33:0/542837691 shutdown_connections 2024-09-16T00:27:07.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.928+0000 7f1a8ab83640 1 --2- 172.21.15.33:0/542837691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84101420 0x7f1a8410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.928+0000 7f1a8ab83640 1 --2- 172.21.15.33:0/542837691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a84100a80 0x7f1a84100ee0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.928+0000 7f1a8ab83640 1 --2- 172.21.15.33:0/542837691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a84106a80 0x7f1a84106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.928+0000 7f1a8ab83640 1 -- 172.21.15.33:0/542837691 >> 172.21.15.33:0/542837691 conn(0x7f1a840fc7a0 msgr2=0x7f1a840febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:07.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.928+0000 7f1a8ab83640 1 -- 172.21.15.33:0/542837691 shutdown_connections 2024-09-16T00:27:07.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.928+0000 7f1a8ab83640 1 -- 172.21.15.33:0/542837691 wait complete. 2024-09-16T00:27:07.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.929+0000 7f1a8ab83640 1 Processor -- start 2024-09-16T00:27:07.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.929+0000 7f1a8ab83640 1 -- start start 2024-09-16T00:27:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.929+0000 7f1a8ab83640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84100a80 0x7f1a841984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.929+0000 7f1a8ab83640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a84101420 0x7f1a841989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.929+0000 7f1a8ab83640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a84106a80 0x7f1a8419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.929+0000 7f1a8ab83640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1a84111fa0 con 0x7f1a84101420 2024-09-16T00:27:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.929+0000 7f1a8ab83640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1a84111e20 con 0x7f1a84100a80 2024-09-16T00:27:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.929+0000 7f1a8ab83640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1a84112120 con 0x7f1a84106a80 2024-09-16T00:27:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.929+0000 7f1a83fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a84101420 0x7f1a841989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.929+0000 7f1a83fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a84101420 0x7f1a841989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51598/0 (socket says 172.21.15.33:51598) 2024-09-16T00:27:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.929+0000 7f1a89120640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a84106a80 0x7f1a8419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.929+0000 7f1a83fff640 1 -- 172.21.15.33:0/1719420005 learned_addr learned my addr 172.21.15.33:0/1719420005 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.930+0000 7f1a8891f640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84100a80 0x7f1a841984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.930+0000 7f1a83fff640 1 -- 172.21.15.33:0/1719420005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a84106a80 msgr2=0x7f1a8419cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.930+0000 7f1a83fff640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a84106a80 0x7f1a8419cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.930+0000 7f1a83fff640 1 -- 172.21.15.33:0/1719420005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84100a80 msgr2=0x7f1a841984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.930+0000 7f1a83fff640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84100a80 0x7f1a841984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.930+0000 7f1a83fff640 1 -- 172.21.15.33:0/1719420005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8419d450 con 0x7f1a84101420 2024-09-16T00:27:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.930+0000 7f1a83fff640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a84101420 0x7f1a841989e0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009230 tx=0x7f1a70004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.930+0000 7f1a81ffb640 1 -- 172.21.15.33:0/1719420005 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7003f030 con 0x7f1a84101420 2024-09-16T00:27:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.930+0000 7f1a8ab83640 1 -- 172.21.15.33:0/1719420005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a8419d6e0 con 0x7f1a84101420 2024-09-16T00:27:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.930+0000 7f1a81ffb640 1 -- 172.21.15.33:0/1719420005 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a70004440 con 0x7f1a84101420 2024-09-16T00:27:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.930+0000 7f1a81ffb640 1 -- 172.21.15.33:0/1719420005 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70035070 con 0x7f1a84101420 2024-09-16T00:27:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.930+0000 7f1a8ab83640 1 -- 172.21.15.33:0/1719420005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a841a5020 con 0x7f1a84101420 2024-09-16T00:27:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.933+0000 7f1a81ffb640 1 -- 172.21.15.33:0/1719420005 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1a70042030 con 0x7f1a84101420 2024-09-16T00:27:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.933+0000 7f1a8cf5c740 1 Processor -- start 2024-09-16T00:27:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.933+0000 7f1a8cf5c740 1 -- start start 2024-09-16T00:27:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.933+0000 7f1a8cf5c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 0x55695070a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.933+0000 7f1a8cf5c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55695070adc0 0x556950715140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.933+0000 7f1a8cf5c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556950715680 0x556950717a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.933+0000 7f1a8cf5c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5569506702e0 con 0x556950715680 2024-09-16T00:27:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.933+0000 7f1a8cf5c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556950617ce0 con 0x55695070a4a0 2024-09-16T00:27:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.933+0000 7f1a8cf5c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55695066c2a0 con 0x55695070adc0 2024-09-16T00:27:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a83fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556950715680 0x556950717a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a89120640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 0x55695070a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a8891f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55695070adc0 0x556950715140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a8891f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55695070adc0 0x556950715140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:49578/0 (socket says 172.21.15.33:49578) 2024-09-16T00:27:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a83fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556950715680 0x556950717a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51614/0 (socket says 172.21.15.33:51614) 2024-09-16T00:27:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a89120640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 0x55695070a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:53788/0 (socket says 172.21.15.33:53788) 2024-09-16T00:27:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a8891f640 1 -- 172.21.15.33:0/1620121134 learned_addr learned my addr 172.21.15.33:0/1620121134 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a81ffb640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1a740a3240 0x7f1a740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a8891f640 1 -- 172.21.15.33:0/1620121134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 msgr2=0x55695070a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a8891f640 1 --2- 172.21.15.33:0/1620121134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 0x55695070a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a8891f640 1 -- 172.21.15.33:0/1620121134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556950715680 msgr2=0x556950717a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a8891f640 1 --2- 172.21.15.33:0/1620121134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556950715680 0x556950717a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a8891f640 1 -- 172.21.15.33:0/1620121134 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569505fc440 con 0x55695070adc0 2024-09-16T00:27:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a89120640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1a740a3240 0x7f1a740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a81ffb640 1 -- 172.21.15.33:0/1719420005 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1a700f0090 con 0x7f1a84101420 2024-09-16T00:27:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a8891f640 1 --2- 172.21.15.33:0/1620121134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55695070adc0 0x556950715140 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009ca0 tx=0x7f1a78011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d860dfff1ea3eccc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a6effd640 1 -- 172.21.15.33:0/1620121134 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7801e070 con 0x55695070adc0 2024-09-16T00:27:07.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a6effd640 1 -- 172.21.15.33:0/1620121134 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a78019b10 con 0x55695070adc0 2024-09-16T00:27:07.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.934+0000 7f1a6effd640 1 -- 172.21.15.33:0/1620121134 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7800e9c0 con 0x55695070adc0 2024-09-16T00:27:07.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.935+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/1620121134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55695070adc0 msgr2=0x556950715140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:07.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.935+0000 7f1a89120640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1a740a3240 0x7f1a740a5700 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c009fd0 tx=0x7f1a7c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:07.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.935+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/1620121134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55695070adc0 0x556950715140 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009ca0 tx=0x7f1a78011810 comp rx=0 tx=0).stop 2024-09-16T00:27:07.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.935+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/1620121134 shutdown_connections 2024-09-16T00:27:07.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.935+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/1620121134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556950715680 0x556950717a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.935+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/1620121134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55695070adc0 0x556950715140 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.935+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/1620121134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 0x55695070a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.935+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/1620121134 >> 172.21.15.33:0/1620121134 conn(0x556950706a30 msgr2=0x556950706e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.935+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/1620121134 shutdown_connections 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.935+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/1620121134 wait complete. 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a8cf5c740 1 -- start start 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a8cf5c740 1 Processor -- start 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a8cf5c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 0x556950714fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a8cf5c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070adc0 0x556950712280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a8cf5c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x5569507127e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a8cf5c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556950718250 con 0x55695070adc0 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a8cf5c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55695062b050 con 0x55695070a4a0 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a8cf5c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5569507183d0 con 0x556950715680 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a8cf5c740 10 client.0 _ll_register_callbacks cb 0x55695066b160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:07.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a83fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070adc0 0x556950712280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a8891f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 0x556950714fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a89120640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x5569507127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a83fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070adc0 0x556950712280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51626/0 (socket says 172.21.15.33:51626) 2024-09-16T00:27:07.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a8891f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 0x556950714fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:53804/0 (socket says 172.21.15.33:53804) 2024-09-16T00:27:07.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.936+0000 7f1a83fff640 1 -- 172.21.15.33:0/347495811 learned_addr learned my addr 172.21.15.33:0/347495811 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:07.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a8891f640 1 -- 172.21.15.33:0/347495811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 msgr2=0x5569507127e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:07.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a8891f640 1 --2- 172.21.15.33:0/347495811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x5569507127e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a8891f640 1 -- 172.21.15.33:0/347495811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070adc0 msgr2=0x556950712280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:07.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a8891f640 1 --2- 172.21.15.33:0/347495811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070adc0 0x556950712280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a8891f640 1 -- 172.21.15.33:0/347495811 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569503e13c0 con 0x55695070a4a0 2024-09-16T00:27:07.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a8891f640 1 --2- 172.21.15.33:0/347495811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 0x556950714fd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1a7801c040 tx=0x7f1a7800e430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7ee9fdf46cd658f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:07.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a63fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:27:07.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a63fff640 1 -- 172.21.15.33:0/347495811 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7801e040 con 0x55695070a4a0 2024-09-16T00:27:07.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/347495811 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a78019b10 con 0x55695070a4a0 2024-09-16T00:27:07.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a63fff640 1 -- 172.21.15.33:0/347495811 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1a78004020 con 0x55695070a4a0 2024-09-16T00:27:07.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a63fff640 1 -- 172.21.15.33:0/347495811 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1a78004220 con 0x55695070a4a0 2024-09-16T00:27:07.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a8cf5c740 10 client.26693 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:07.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.937+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/347495811 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556950723c80 con 0x55695070a4a0 2024-09-16T00:27:07.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.938+0000 7f1a63fff640 1 -- 172.21.15.33:0/347495811 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a78019b10 con 0x55695070a4a0 2024-09-16T00:27:07.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.938+0000 7f1a63fff640 1 -- 172.21.15.33:0/347495811 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f1a78004960 con 0x55695070a4a0 2024-09-16T00:27:07.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.938+0000 7f1a8cf5c740 10 client.26693 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:07.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.938+0000 7f1a8cf5c740 10 client.26693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:27:07.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.938+0000 7f1a8cf5c740 20 client.26693 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:07.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.938+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/347495811 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556950725090 0x556950727480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.938+0000 7f1a8cf5c740 4 client.26693 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:27:07.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.938+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/347495811 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55695067d850 con 0x556950725090 2024-09-16T00:27:07.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.938+0000 7f1a83fff640 1 --2- 172.21.15.33:0/347495811 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556950725090 0x556950727480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.938+0000 7f1a63fff640 10 client.26693 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:07.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.938+0000 7f1a83fff640 1 --2- 172.21.15.33:0/347495811 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556950725090 0x556950727480 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b38f87f75a13b96b server_cookie=ed26eb35f7a06501 in_seq=0 out_seq=0 2024-09-16T00:27:07.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a63fff640 1 -- 172.21.15.33:0/347495811 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55695067d850 con 0x556950725090 2024-09-16T00:27:07.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a63fff640 10 client.26693 handle_command_reply: tid=0 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a8cf5c740 1 client.26693 shutdown 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/347495811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 msgr2=0x556950714fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/347495811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 0x556950714fd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1a7801c040 tx=0x7f1a7800e430 comp rx=0 tx=0).stop 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/347495811 shutdown_connections 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/347495811 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556950725090 0x556950727480 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/347495811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x5569507127e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/347495811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070adc0 0x556950712280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a63fff640 0 client.26693 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/347495811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070a4a0 0x556950714fd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/347495811 >> 172.21.15.33:0/347495811 conn(0x556950706a30 msgr2=0x55695070c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.939+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/347495811 shutdown_connections 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.940+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/347495811 wait complete. 2024-09-16T00:27:07.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.940+0000 7f1a8cf5c740 20 client.26693 trim_cache size 0 max 16384 2024-09-16T00:27:07.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.957+0000 7f1a8cf5c740 1 Processor -- start 2024-09-16T00:27:07.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.957+0000 7f1a8cf5c740 1 -- start start 2024-09-16T00:27:07.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8cf5c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 0x5569507214b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8cf5c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070adc0 0x55695072cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8cf5c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x55695072d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8cf5c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1a841af7b0 con 0x55695070a4a0 2024-09-16T00:27:07.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8cf5c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1a841ad080 con 0x55695070adc0 2024-09-16T00:27:07.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8cf5c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1a841aaed0 con 0x556950715680 2024-09-16T00:27:07.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8891f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 0x5569507214b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8891f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 0x5569507214b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51630/0 (socket says 172.21.15.33:51630) 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8891f640 1 -- 172.21.15.33:0/1548053810 learned_addr learned my addr 172.21.15.33:0/1548053810 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a89120640 1 --2- 172.21.15.33:0/1548053810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x55695072d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a83fff640 1 --2- 172.21.15.33:0/1548053810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070adc0 0x55695072cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8891f640 1 -- 172.21.15.33:0/1548053810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 msgr2=0x55695072d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8891f640 1 --2- 172.21.15.33:0/1548053810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x55695072d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8891f640 1 -- 172.21.15.33:0/1548053810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070adc0 msgr2=0x55695072cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8891f640 1 --2- 172.21.15.33:0/1548053810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070adc0 0x55695072cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8891f640 1 -- 172.21.15.33:0/1548053810 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556950712b50 con 0x55695070a4a0 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.958+0000 7f1a8891f640 1 --2- 172.21.15.33:0/1548053810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 0x5569507214b0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1a78019fd0 tx=0x7f1a7800e6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83d66d08d602a20 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.959+0000 7f1a6e7fc640 1 -- 172.21.15.33:0/1548053810 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7801e070 con 0x55695070a4a0 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.959+0000 7f1a6e7fc640 1 -- 172.21.15.33:0/1548053810 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a7800edb0 con 0x55695070a4a0 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.959+0000 7f1a6e7fc640 1 -- 172.21.15.33:0/1548053810 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a780051d0 con 0x55695070a4a0 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.959+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/1548053810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 msgr2=0x5569507214b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:07.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.959+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/1548053810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 0x5569507214b0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1a78019fd0 tx=0x7f1a7800e6b0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.959+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/1548053810 shutdown_connections 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.959+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/1548053810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x55695072d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.959+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/1548053810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070adc0 0x55695072cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.959+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/1548053810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 0x5569507214b0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.959+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/1548053810 >> 172.21.15.33:0/1548053810 conn(0x556950706a30 msgr2=0x55695070d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.959+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/1548053810 shutdown_connections 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.960+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/1548053810 wait complete. 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.960+0000 7f1a8cf5c740 1 -- start start 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.960+0000 7f1a8cf5c740 1 Processor -- start 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.960+0000 7f1a8cf5c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 0x55695071d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.960+0000 7f1a8cf5c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070adc0 0x55695071dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.960+0000 7f1a8cf5c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x556950718bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.960+0000 7f1a8cf5c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1a841ab540 con 0x55695070a4a0 2024-09-16T00:27:07.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.960+0000 7f1a8cf5c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1a841ac9b0 con 0x55695070adc0 2024-09-16T00:27:07.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.960+0000 7f1a8cf5c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1a841abc10 con 0x556950715680 2024-09-16T00:27:07.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.960+0000 7f1a8cf5c740 10 client.0 _ll_register_callbacks cb 0x7f1a7801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:07.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.960+0000 7f1a8891f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 0x55695071d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a83fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070adc0 0x55695071dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a89120640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x556950718bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a8891f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 0x55695071d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51634/0 (socket says 172.21.15.33:51634) 2024-09-16T00:27:07.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a8891f640 1 -- 172.21.15.33:0/2125391291 learned_addr learned my addr 172.21.15.33:0/2125391291 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:07.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a83fff640 1 -- 172.21.15.33:0/2125391291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 msgr2=0x556950718bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:07.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a83fff640 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x556950718bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a83fff640 1 -- 172.21.15.33:0/2125391291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 msgr2=0x55695071d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a83fff640 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 0x55695071d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a83fff640 1 -- 172.21.15.33:0/2125391291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569507193f0 con 0x55695070adc0 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a89120640 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x556950718bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a8891f640 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 0x55695071d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a83fff640 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070adc0 0x55695071dd40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009900 tx=0x7f1a700f3b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68a0f9719e2ee2e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a63fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a63fff640 1 -- 172.21.15.33:0/2125391291 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70049040 con 0x55695070adc0 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a63fff640 1 -- 172.21.15.33:0/2125391291 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1a700b8950 con 0x55695070adc0 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.961+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/2125391291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a7800edb0 con 0x55695070adc0 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.962+0000 7f1a63fff640 1 -- 172.21.15.33:0/2125391291 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1a700bbc40 con 0x55695070adc0 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.962+0000 7f1a8cf5c740 10 client.26699 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:07.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.962+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/2125391291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55695078f760 con 0x55695070adc0 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.962+0000 7f1a63fff640 1 -- 172.21.15.33:0/2125391291 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a7800edb0 con 0x55695070adc0 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.962+0000 7f1a63fff640 1 -- 172.21.15.33:0/2125391291 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f1a700c01b0 con 0x55695070adc0 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.962+0000 7f1a8cf5c740 10 client.26699 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.962+0000 7f1a8cf5c740 10 client.26699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.962+0000 7f1a8cf5c740 20 client.26699 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.962+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55695078fee0 0x556950792300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.962+0000 7f1a8cf5c740 4 client.26699 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.962+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/2125391291 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556950677d60 con 0x55695078fee0 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.962+0000 7f1a8891f640 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55695078fee0 0x556950792300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a63fff640 10 client.26699 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a8891f640 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55695078fee0 0x556950792300 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d68717fb24717e2 server_cookie=56c35dbd8d7270df in_seq=0 out_seq=0 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a63fff640 1 -- 172.21.15.33:0/2125391291 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556950677d60 con 0x55695078fee0 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a63fff640 10 client.26699 handle_command_reply: tid=0 2024-09-16T00:27:07.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a8cf5c740 1 client.26699 shutdown 2024-09-16T00:27:07.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/2125391291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070adc0 msgr2=0x55695071dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:07.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070adc0 0x55695071dd40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009900 tx=0x7f1a700f3b70 comp rx=0 tx=0).stop 2024-09-16T00:27:07.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/2125391291 shutdown_connections 2024-09-16T00:27:07.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55695078fee0 0x556950792300 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a63fff640 0 client.26699 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:07.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556950715680 0x556950718bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55695070adc0 0x55695071dd40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a8cf5c740 1 --2- 172.21.15.33:0/2125391291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55695070a4a0 0x55695071d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/2125391291 >> 172.21.15.33:0/2125391291 conn(0x556950706a30 msgr2=0x556950705350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:07.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/2125391291 shutdown_connections 2024-09-16T00:27:07.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.963+0000 7f1a8cf5c740 1 -- 172.21.15.33:0/2125391291 wait complete. 2024-09-16T00:27:07.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8cf5c740 20 client.26699 trim_cache size 0 max 16384 2024-09-16T00:27:07.980 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:27:07.980 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:27:07.980 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:27:07.980 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:27:07.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:07 smithi060 ceph-mon[35185]: pgmap v649: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 440 KiB/s wr, 58 op/s 2024-09-16T00:27:07.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8ab83640 1 -- 172.21.15.33:0/1719420005 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1a740a3240 msgr2=0x7f1a740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:07.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8ab83640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1a740a3240 0x7f1a740a5700 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c009fd0 tx=0x7f1a7c004a40 comp rx=0 tx=0).stop 2024-09-16T00:27:07.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8ab83640 1 -- 172.21.15.33:0/1719420005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a84101420 msgr2=0x7f1a841989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:07.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8ab83640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a84101420 0x7f1a841989e0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009230 tx=0x7f1a70004040 comp rx=0 tx=0).stop 2024-09-16T00:27:07.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8ab83640 1 -- 172.21.15.33:0/1719420005 shutdown_connections 2024-09-16T00:27:07.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8ab83640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1a740a3240 0x7f1a740a5700 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8ab83640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a84106a80 0x7f1a8419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8ab83640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a84101420 0x7f1a841989e0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8ab83640 1 --2- 172.21.15.33:0/1719420005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84100a80 0x7f1a841984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:07.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8ab83640 1 -- 172.21.15.33:0/1719420005 >> 172.21.15.33:0/1719420005 conn(0x7f1a840fc7a0 msgr2=0x7f1a840fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:07.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8ab83640 1 -- 172.21.15.33:0/1719420005 shutdown_connections 2024-09-16T00:27:07.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:07.964+0000 7f1a8ab83640 1 -- 172.21.15.33:0/1719420005 wait complete. 2024-09-16T00:27:08.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 781739 9.55 MB/sec execute 376 sec latency 10.541 ms 2024-09-16T00:27:08.967 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:27:08.967 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:27:09.237 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:09.668 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:09 smithi033 ceph-mon[31243]: pgmap v650: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 479 KiB/s wr, 66 op/s 2024-09-16T00:27:09.675 INFO:tasks.workunit.client.0.smithi033.stdout: 1 784779 9.57 MB/sec execute 377 sec latency 59.442 ms 2024-09-16T00:27:09.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.756+0000 7f75259d3640 1 -- 172.21.15.33:0/1948605572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7520102d40 msgr2=0x7f7520103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:09.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.756+0000 7f75259d3640 1 --2- 172.21.15.33:0/1948605572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7520102d40 0x7f7520103120 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f7510009900 tx=0x7f751002a480 comp rx=0 tx=0).stop 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.756+0000 7f75259d3640 1 -- 172.21.15.33:0/1948605572 shutdown_connections 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.756+0000 7f75259d3640 1 --2- 172.21.15.33:0/1948605572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f752010c170 0x7f752010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.756+0000 7f75259d3640 1 --2- 172.21.15.33:0/1948605572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7520103660 0x7f752010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.756+0000 7f75259d3640 1 --2- 172.21.15.33:0/1948605572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7520102d40 0x7f7520103120 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.756+0000 7f75259d3640 1 -- 172.21.15.33:0/1948605572 >> 172.21.15.33:0/1948605572 conn(0x7f75200fc7a0 msgr2=0x7f75200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.757+0000 7f75259d3640 1 -- 172.21.15.33:0/1948605572 shutdown_connections 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.757+0000 7f75259d3640 1 -- 172.21.15.33:0/1948605572 wait complete. 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.757+0000 7f75259d3640 1 Processor -- start 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.757+0000 7f75259d3640 1 -- start start 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f75259d3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7520102d40 0x7f7520198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f75259d3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7520103660 0x7f7520198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f75259d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752010c170 0x7f752019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f75259d3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7520111f70 con 0x7f752010c170 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f75259d3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7520111df0 con 0x7f7520103660 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f75259d3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f75201120f0 con 0x7f7520102d40 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f751f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752010c170 0x7f752019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f751effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7520102d40 0x7f7520198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f751f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752010c170 0x7f752019cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51648/0 (socket says 172.21.15.33:51648) 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f751f7fe640 1 -- 172.21.15.33:0/156180975 learned_addr learned my addr 172.21.15.33:0/156180975 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f7517fff640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7520103660 0x7f7520198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f751f7fe640 1 -- 172.21.15.33:0/156180975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7520102d40 msgr2=0x7f7520198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f751f7fe640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7520102d40 0x7f7520198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f751f7fe640 1 -- 172.21.15.33:0/156180975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7520103660 msgr2=0x7f7520198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f751f7fe640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7520103660 0x7f7520198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f751f7fe640 1 -- 172.21.15.33:0/156180975 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752019d400 con 0x7f752010c170 2024-09-16T00:27:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f751f7fe640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752010c170 0x7f752019cd20 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f751800eb10 tx=0x7f751800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f75249d1640 1 -- 172.21.15.33:0/156180975 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518004b20 con 0x7f752010c170 2024-09-16T00:27:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f75249d1640 1 -- 172.21.15.33:0/156180975 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7518004cc0 con 0x7f752010c170 2024-09-16T00:27:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f75259d3640 1 -- 172.21.15.33:0/156180975 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f752019d6f0 con 0x7f752010c170 2024-09-16T00:27:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.758+0000 7f75249d1640 1 -- 172.21.15.33:0/156180975 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518010760 con 0x7f752010c170 2024-09-16T00:27:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.759+0000 7f75259d3640 1 -- 172.21.15.33:0/156180975 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75201a4fd0 con 0x7f752010c170 2024-09-16T00:27:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.760+0000 7f75259d3640 1 -- 172.21.15.33:0/156180975 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7520103120 con 0x7f752010c170 2024-09-16T00:27:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.761+0000 7f75249d1640 1 -- 172.21.15.33:0/156180975 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7518005020 con 0x7f752010c170 2024-09-16T00:27:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.761+0000 7f75249d1640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f750c0a32c0 0x7f750c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.762+0000 7f751effd640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f750c0a32c0 0x7f750c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.762+0000 7f75249d1640 1 -- 172.21.15.33:0/156180975 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f75180d03a0 con 0x7f752010c170 2024-09-16T00:27:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.762+0000 7f751effd640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f750c0a32c0 0x7f750c0a5780 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7510009230 tx=0x7f75100058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.766+0000 7f75249d1640 1 -- 172.21.15.33:0/156180975 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f751809cbe0 con 0x7f752010c170 2024-09-16T00:27:09.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:09 smithi083 ceph-mon[35242]: pgmap v650: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 479 KiB/s wr, 66 op/s 2024-09-16T00:27:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:09 smithi060 ceph-mon[35185]: pgmap v650: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 479 KiB/s wr, 66 op/s 2024-09-16T00:27:09.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.987+0000 7f75259d3640 1 -- 172.21.15.33:0/156180975 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7520107270 con 0x7f752010c170 2024-09-16T00:27:09.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.988+0000 7f75249d1640 1 -- 172.21.15.33:0/156180975 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f751809c4e0 con 0x7f752010c170 2024-09-16T00:27:09.990 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:27:09.990 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:27:09.990 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:27:09.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.991+0000 7f75259d3640 1 -- 172.21.15.33:0/156180975 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f750c0a32c0 msgr2=0x7f750c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:09.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.991+0000 7f75259d3640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f750c0a32c0 0x7f750c0a5780 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7510009230 tx=0x7f75100058d0 comp rx=0 tx=0).stop 2024-09-16T00:27:09.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.991+0000 7f75259d3640 1 -- 172.21.15.33:0/156180975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752010c170 msgr2=0x7f752019cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:09.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.991+0000 7f75259d3640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752010c170 0x7f752019cd20 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f751800eb10 tx=0x7f751800c2d0 comp rx=0 tx=0).stop 2024-09-16T00:27:09.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.991+0000 7f75259d3640 1 -- 172.21.15.33:0/156180975 shutdown_connections 2024-09-16T00:27:09.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.991+0000 7f75259d3640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f750c0a32c0 0x7f750c0a5780 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:09.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.991+0000 7f75259d3640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752010c170 0x7f752019cd20 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:09.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.991+0000 7f75259d3640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7520103660 0x7f7520198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:09.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.991+0000 7f75259d3640 1 --2- 172.21.15.33:0/156180975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7520102d40 0x7f7520198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:09.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.991+0000 7f75259d3640 1 -- 172.21.15.33:0/156180975 >> 172.21.15.33:0/156180975 conn(0x7f75200fc7a0 msgr2=0x7f75200fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:09.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.991+0000 7f75259d3640 1 -- 172.21.15.33:0/156180975 shutdown_connections 2024-09-16T00:27:09.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:09.991+0000 7f75259d3640 1 -- 172.21.15.33:0/156180975 wait complete. 2024-09-16T00:27:10.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 787125 9.57 MB/sec execute 378 sec latency 15.875 ms 2024-09-16T00:27:10.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:10 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/156180975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:27:10.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:10 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/156180975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:27:10.954 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:27:10.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:10 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/156180975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:27:11.216 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:11.638 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:11 smithi033 ceph-mon[31243]: pgmap v651: 129 pgs: 129 active+clean; 277 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 825 KiB/s wr, 115 op/s 2024-09-16T00:27:11.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 789219 9.58 MB/sec execute 379 sec latency 43.899 ms 2024-09-16T00:27:11.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.731+0000 7f7e2238f640 1 -- 172.21.15.33:0/2378070958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c108300 msgr2=0x7f7e1c10ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:11.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.731+0000 7f7e2238f640 1 --2- 172.21.15.33:0/2378070958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c108300 0x7f7e1c10ec70 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009930 tx=0x7f7e0c02a420 comp rx=0 tx=0).stop 2024-09-16T00:27:11.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.731+0000 7f7e2238f640 1 -- 172.21.15.33:0/2378070958 shutdown_connections 2024-09-16T00:27:11.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.731+0000 7f7e2238f640 1 --2- 172.21.15.33:0/2378070958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c108300 0x7f7e1c10ec70 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.731+0000 7f7e2238f640 1 --2- 172.21.15.33:0/2378070958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e1c103780 0x7f7e1c107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.731+0000 7f7e2238f640 1 --2- 172.21.15.33:0/2378070958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e1c102dd0 0x7f7e1c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.731+0000 7f7e2238f640 1 -- 172.21.15.33:0/2378070958 >> 172.21.15.33:0/2378070958 conn(0x7f7e1c0fc7d0 msgr2=0x7f7e1c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:11.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.731+0000 7f7e2238f640 1 -- 172.21.15.33:0/2378070958 shutdown_connections 2024-09-16T00:27:11.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.731+0000 7f7e2238f640 1 -- 172.21.15.33:0/2378070958 wait complete. 2024-09-16T00:27:11.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.732+0000 7f7e2238f640 1 Processor -- start 2024-09-16T00:27:11.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.732+0000 7f7e2238f640 1 -- start start 2024-09-16T00:27:11.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.732+0000 7f7e2238f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e1c102dd0 0x7f7e1c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.732+0000 7f7e2238f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c103780 0x7f7e1c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.732+0000 7f7e2238f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e1c108300 0x7f7e1c19ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.732+0000 7f7e2238f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c112120 con 0x7f7e1c103780 2024-09-16T00:27:11.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.732+0000 7f7e2238f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c111fa0 con 0x7f7e1c108300 2024-09-16T00:27:11.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.732+0000 7f7e1b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c103780 0x7f7e1c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.732+0000 7f7e2238f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1122a0 con 0x7f7e1c102dd0 2024-09-16T00:27:11.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.732+0000 7f7e1b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c103780 0x7f7e1c198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48334/0 (socket says 172.21.15.33:48334) 2024-09-16T00:27:11.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.732+0000 7f7e1b7fe640 1 -- 172.21.15.33:0/2336809030 learned_addr learned my addr 172.21.15.33:0/2336809030 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:11.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.732+0000 7f7e1bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e1c102dd0 0x7f7e1c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e2092c640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e1c108300 0x7f7e1c19ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e1bfff640 1 -- 172.21.15.33:0/2336809030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e1c108300 msgr2=0x7f7e1c19ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:11.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e1c108300 0x7f7e1c19ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e1bfff640 1 -- 172.21.15.33:0/2336809030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c103780 msgr2=0x7f7e1c198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:11.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c103780 0x7f7e1c198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e1bfff640 1 -- 172.21.15.33:0/2336809030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1c19d510 con 0x7f7e1c102dd0 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e2092c640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e1c108300 0x7f7e1c19ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e1c102dd0 0x7f7e1c198560 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009fd0 tx=0x7f7e1400ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e197fa640 1 -- 172.21.15.33:0/2336809030 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14004b20 con 0x7f7e1c102dd0 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e197fa640 1 -- 172.21.15.33:0/2336809030 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e14004cc0 con 0x7f7e1c102dd0 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e2238f640 1 -- 172.21.15.33:0/2336809030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e1c19d800 con 0x7f7e1c102dd0 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e197fa640 1 -- 172.21.15.33:0/2336809030 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e140107b0 con 0x7f7e1c102dd0 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.733+0000 7f7e2238f640 1 -- 172.21.15.33:0/2336809030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e1c100960 con 0x7f7e1c102dd0 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e24768740 1 Processor -- start 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e24768740 1 -- start start 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e197fa640 1 -- 172.21.15.33:0/2336809030 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7e14005020 con 0x7f7e1c102dd0 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e24768740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56289585fb40 0x56289585e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e24768740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 0x56289585ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e24768740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562895869370 0x56289586b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e24768740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5628957c42f0 con 0x56289585e5d0 2024-09-16T00:27:11.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e24768740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56289576bcf0 con 0x56289585fb40 2024-09-16T00:27:11.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e24768740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5628957c02b0 con 0x562895869370 2024-09-16T00:27:11.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 0x56289585ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 0x56289585ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48346/0 (socket says 172.21.15.33:48346) 2024-09-16T00:27:11.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e1bfff640 1 -- 172.21.15.33:0/3619369756 learned_addr learned my addr 172.21.15.33:0/3619369756 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:11.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e1b7fe640 1 --2- 172.21.15.33:0/3619369756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562895869370 0x56289586b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e2092c640 1 --2- 172.21.15.33:0/3619369756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56289585fb40 0x56289585e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.736+0000 7f7e1bfff640 1 -- 172.21.15.33:0/3619369756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562895869370 msgr2=0x56289586b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:11.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.737+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/3619369756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562895869370 0x56289586b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.737+0000 7f7e1bfff640 1 -- 172.21.15.33:0/3619369756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56289585fb40 msgr2=0x56289585e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:11.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.737+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/3619369756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56289585fb40 0x56289585e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.737+0000 7f7e1bfff640 1 -- 172.21.15.33:0/3619369756 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628957503a0 con 0x56289585e5d0 2024-09-16T00:27:11.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.737+0000 7f7e197fa640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7e080a3240 0x7f7e080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.737+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/3619369756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 0x56289585ea10 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f7e140d6720 tx=0x7f7e140d6810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ae8070c82e092d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:11.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.737+0000 7f7e2092c640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7e080a3240 0x7f7e080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.737+0000 7f7e197fa640 1 -- 172.21.15.33:0/2336809030 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7e140d0620 con 0x7f7e1c102dd0 2024-09-16T00:27:11.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.737+0000 7f7e127fc640 1 -- 172.21.15.33:0/3619369756 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14098a00 con 0x56289585e5d0 2024-09-16T00:27:11.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.737+0000 7f7e127fc640 1 -- 172.21.15.33:0/3619369756 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e1409ce60 con 0x56289585e5d0 2024-09-16T00:27:11.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.737+0000 7f7e127fc640 1 -- 172.21.15.33:0/3619369756 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1409c0c0 con 0x56289585e5d0 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.738+0000 7f7e24768740 1 -- 172.21.15.33:0/3619369756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 msgr2=0x56289585ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.738+0000 7f7e2092c640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7e080a3240 0x7f7e080a5700 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009230 tx=0x7f7e0c005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.738+0000 7f7e24768740 1 --2- 172.21.15.33:0/3619369756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 0x56289585ea10 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f7e140d6720 tx=0x7f7e140d6810 comp rx=0 tx=0).stop 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.738+0000 7f7e24768740 1 -- 172.21.15.33:0/3619369756 shutdown_connections 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.738+0000 7f7e24768740 1 --2- 172.21.15.33:0/3619369756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562895869370 0x56289586b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.738+0000 7f7e24768740 1 --2- 172.21.15.33:0/3619369756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 0x56289585ea10 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.738+0000 7f7e24768740 1 --2- 172.21.15.33:0/3619369756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56289585fb40 0x56289585e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.738+0000 7f7e24768740 1 -- 172.21.15.33:0/3619369756 >> 172.21.15.33:0/3619369756 conn(0x562895827310 msgr2=0x56289585ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.738+0000 7f7e24768740 1 -- 172.21.15.33:0/3619369756 shutdown_connections 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.738+0000 7f7e24768740 1 -- 172.21.15.33:0/3619369756 wait complete. 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.738+0000 7f7e24768740 1 -- start start 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e24768740 1 Processor -- start 2024-09-16T00:27:11.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e24768740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585e5d0 0x562895868dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e24768740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 0x562895866080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e24768740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562895869370 0x562895866810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e24768740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56289586c070 con 0x56289585fb40 2024-09-16T00:27:11.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e24768740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56289577f060 con 0x562895869370 2024-09-16T00:27:11.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e24768740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56289586c1f0 con 0x56289585e5d0 2024-09-16T00:27:11.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e24768740 10 client.0 _ll_register_callbacks cb 0x5628954d28c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:11.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e1bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585e5d0 0x562895868dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e1b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 0x562895866080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e1bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585e5d0 0x562895868dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59756/0 (socket says 172.21.15.33:59756) 2024-09-16T00:27:11.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e1bfff640 1 -- 172.21.15.33:0/2344422645 learned_addr learned my addr 172.21.15.33:0/2344422645 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:11.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.739+0000 7f7e2092c640 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562895869370 0x562895866810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.740+0000 7f7e1b7fe640 1 -- 172.21.15.33:0/2344422645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585e5d0 msgr2=0x562895868dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.740+0000 7f7e1b7fe640 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585e5d0 0x562895868dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.740+0000 7f7e1b7fe640 1 -- 172.21.15.33:0/2344422645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562895869370 msgr2=0x562895866810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.740+0000 7f7e1b7fe640 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562895869370 0x562895866810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.740+0000 7f7e1b7fe640 1 -- 172.21.15.33:0/2344422645 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628955353c0 con 0x56289585fb40 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.740+0000 7f7e2092c640 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562895869370 0x562895866810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.740+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585e5d0 0x562895868dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.740+0000 7f7e1b7fe640 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 0x562895866080 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0fed90 tx=0x7f7e04003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=481bb71486b3b9fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.740+0000 7f7dfb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.740+0000 7f7dfb7fe640 1 -- 172.21.15.33:0/2344422645 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0401f020 con 0x56289585fb40 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.740+0000 7f7dfb7fe640 1 -- 172.21.15.33:0/2344422645 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7e04009b80 con 0x56289585fb40 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.740+0000 7f7e24768740 1 -- 172.21.15.33:0/2344422645 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e1409ce60 con 0x56289585fb40 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.741+0000 7f7dfb7fe640 1 -- 172.21.15.33:0/2344422645 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7e0402a050 con 0x56289585fb40 2024-09-16T00:27:11.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.741+0000 7f7e24768740 10 client.17268 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.741+0000 7f7e24768740 1 -- 172.21.15.33:0/2344422645 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562895877bf0 con 0x56289585fb40 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.741+0000 7f7dfb7fe640 1 -- 172.21.15.33:0/2344422645 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e1409ce60 con 0x56289585fb40 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.741+0000 7f7dfb7fe640 1 -- 172.21.15.33:0/2344422645 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f7e0401a3b0 con 0x56289585fb40 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.741+0000 7f7e24768740 10 client.17268 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.741+0000 7f7e24768740 10 client.17268 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.741+0000 7f7e24768740 20 client.17268 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.741+0000 7f7e24768740 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5628958790b0 0x56289587b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.741+0000 7f7e24768740 4 client.17268 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.741+0000 7f7e24768740 1 -- 172.21.15.33:0/2344422645 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628957d1860 con 0x5628958790b0 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.741+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5628958790b0 0x56289587b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.742+0000 7f7dfb7fe640 10 client.17268 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.742+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5628958790b0 0x56289587b530 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=30e1f61e424375c2 server_cookie=2d45c0ecc7e37b99 in_seq=0 out_seq=0 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7dfb7fe640 1 -- 172.21.15.33:0/2344422645 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5628957d1860 con 0x5628958790b0 2024-09-16T00:27:11.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7dfb7fe640 10 client.17268 handle_command_reply: tid=0 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7e24768740 1 client.17268 shutdown 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7e24768740 1 -- 172.21.15.33:0/2344422645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 msgr2=0x562895866080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7e24768740 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 0x562895866080 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0fed90 tx=0x7f7e04003ce0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7e24768740 1 -- 172.21.15.33:0/2344422645 shutdown_connections 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7e24768740 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5628958790b0 0x56289587b530 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7e24768740 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562895869370 0x562895866810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7e24768740 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 0x562895866080 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7e24768740 1 --2- 172.21.15.33:0/2344422645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585e5d0 0x562895868dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7dfb7fe640 0 client.17268 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7e24768740 1 -- 172.21.15.33:0/2344422645 >> 172.21.15.33:0/2344422645 conn(0x562895827310 msgr2=0x562895860930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7e24768740 1 -- 172.21.15.33:0/2344422645 shutdown_connections 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7e24768740 1 -- 172.21.15.33:0/2344422645 wait complete. 2024-09-16T00:27:11.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.744+0000 7f7e24768740 20 client.17268 trim_cache size 0 max 16384 2024-09-16T00:27:11.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.759+0000 7f7e24768740 1 Processor -- start 2024-09-16T00:27:11.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e24768740 1 -- start start 2024-09-16T00:27:11.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e24768740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56289585e5d0 0x562895859bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e24768740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 0x562895861120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e24768740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562895869370 0x562895861660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e24768740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1af7d0 con 0x56289585fb40 2024-09-16T00:27:11.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e24768740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1ad0a0 con 0x56289585e5d0 2024-09-16T00:27:11.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e24768740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1aaef0 con 0x562895869370 2024-09-16T00:27:11.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e1b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 0x562895861120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e1b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 0x562895861120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48364/0 (socket says 172.21.15.33:48364) 2024-09-16T00:27:11.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e2092c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562895869370 0x562895861660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e1b7fe640 1 -- 172.21.15.33:0/3179357673 learned_addr learned my addr 172.21.15.33:0/3179357673 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/3179357673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56289585e5d0 0x562895859bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e2092c640 1 -- 172.21.15.33:0/3179357673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56289585e5d0 msgr2=0x562895859bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e2092c640 1 --2- 172.21.15.33:0/3179357673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56289585e5d0 0x562895859bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e2092c640 1 -- 172.21.15.33:0/3179357673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 msgr2=0x562895861120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e2092c640 1 --2- 172.21.15.33:0/3179357673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 0x562895861120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e2092c640 1 -- 172.21.15.33:0/3179357673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562895866b80 con 0x562895869370 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/3179357673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56289585e5d0 0x562895859bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.760+0000 7f7e1b7fe640 1 --2- 172.21.15.33:0/3179357673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 0x562895861120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.761+0000 7f7e2092c640 1 --2- 172.21.15.33:0/3179357673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562895869370 0x562895861660 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c005c40 tx=0x7f7e0c03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f019de23a42a292e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.761+0000 7f7e11ffb640 1 -- 172.21.15.33:0/3179357673 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c04e070 con 0x562895869370 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.761+0000 7f7e11ffb640 1 -- 172.21.15.33:0/3179357673 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e0c0325c0 con 0x562895869370 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.761+0000 7f7e11ffb640 1 -- 172.21.15.33:0/3179357673 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c035040 con 0x562895869370 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.762+0000 7f7e24768740 1 -- 172.21.15.33:0/3179357673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562895869370 msgr2=0x562895861660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:11.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.762+0000 7f7e24768740 1 --2- 172.21.15.33:0/3179357673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562895869370 0x562895861660 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c005c40 tx=0x7f7e0c03d000 comp rx=0 tx=0).stop 2024-09-16T00:27:11.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.762+0000 7f7e24768740 1 -- 172.21.15.33:0/3179357673 shutdown_connections 2024-09-16T00:27:11.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.762+0000 7f7e24768740 1 --2- 172.21.15.33:0/3179357673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562895869370 0x562895861660 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.762+0000 7f7e24768740 1 --2- 172.21.15.33:0/3179357673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585fb40 0x562895861120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.762+0000 7f7e24768740 1 --2- 172.21.15.33:0/3179357673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56289585e5d0 0x562895859bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.762+0000 7f7e24768740 1 -- 172.21.15.33:0/3179357673 >> 172.21.15.33:0/3179357673 conn(0x562895827310 msgr2=0x5628958594e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:11.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.762+0000 7f7e24768740 1 -- 172.21.15.33:0/3179357673 shutdown_connections 2024-09-16T00:27:11.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.762+0000 7f7e24768740 1 -- 172.21.15.33:0/3179357673 wait complete. 2024-09-16T00:27:11.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.762+0000 7f7e24768740 1 -- start start 2024-09-16T00:27:11.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.762+0000 7f7e24768740 1 Processor -- start 2024-09-16T00:27:11.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e24768740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 0x562895871da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e24768740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585fb40 0x562895862fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e24768740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562895869370 0x562895863750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e24768740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1ab560 con 0x56289585e5d0 2024-09-16T00:27:11.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e24768740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1ac9d0 con 0x562895869370 2024-09-16T00:27:11.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e24768740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1abc30 con 0x56289585fb40 2024-09-16T00:27:11.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e24768740 10 client.0 _ll_register_callbacks cb 0x7f7e0400fdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:11.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e1b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585fb40 0x562895862fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 0x562895871da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e1b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585fb40 0x562895862fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59776/0 (socket says 172.21.15.33:59776) 2024-09-16T00:27:11.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e1b7fe640 1 -- 172.21.15.33:0/1221397551 learned_addr learned my addr 172.21.15.33:0/1221397551 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:11.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e2092c640 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562895869370 0x562895863750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e1b7fe640 1 -- 172.21.15.33:0/1221397551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562895869370 msgr2=0x562895863750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e1b7fe640 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562895869370 0x562895863750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e1b7fe640 1 -- 172.21.15.33:0/1221397551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 msgr2=0x562895871da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e1b7fe640 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 0x562895871da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e1b7fe640 1 -- 172.21.15.33:0/1221397551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628958e2dd0 con 0x56289585fb40 2024-09-16T00:27:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.763+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 0x562895871da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.764+0000 7f7e1b7fe640 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585fb40 0x562895862fc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7e0401a820 tx=0x7f7e04003ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=352aa444308145 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.764+0000 7f7dfb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:27:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.764+0000 7f7dfb7fe640 1 -- 172.21.15.33:0/1221397551 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0401f070 con 0x56289585fb40 2024-09-16T00:27:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.764+0000 7f7dfb7fe640 1 -- 172.21.15.33:0/1221397551 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7e0401aa10 con 0x56289585fb40 2024-09-16T00:27:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.764+0000 7f7e24768740 1 -- 172.21.15.33:0/1221397551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e0c0325c0 con 0x56289585fb40 2024-09-16T00:27:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.764+0000 7f7dfb7fe640 1 -- 172.21.15.33:0/1221397551 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7e0403f050 con 0x56289585fb40 2024-09-16T00:27:11.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.764+0000 7f7e24768740 10 client.26674 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:11.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.764+0000 7f7e24768740 1 -- 172.21.15.33:0/1221397551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5628958e3fe0 con 0x56289585fb40 2024-09-16T00:27:11.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.764+0000 7f7dfb7fe640 1 -- 172.21.15.33:0/1221397551 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e0c0325c0 con 0x56289585fb40 2024-09-16T00:27:11.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.764+0000 7f7dfb7fe640 1 -- 172.21.15.33:0/1221397551 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f7e04005040 con 0x56289585fb40 2024-09-16T00:27:11.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.765+0000 7f7e24768740 10 client.26674 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:11.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.765+0000 7f7e24768740 10 client.26674 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:27:11.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.765+0000 7f7e24768740 20 client.26674 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:11.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.765+0000 7f7e24768740 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5628958e47c0 0x5628958e6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:11.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.765+0000 7f7e24768740 4 client.26674 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:27:11.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.765+0000 7f7e24768740 1 -- 172.21.15.33:0/1221397551 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628957cbd70 con 0x5628958e47c0 2024-09-16T00:27:11.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.765+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5628958e47c0 0x5628958e6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:11.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.765+0000 7f7dfb7fe640 10 client.26674 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:27:11.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.765+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5628958e47c0 0x5628958e6be0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9236182cf2046587 server_cookie=59bbe2dd95684f2b in_seq=0 out_seq=0 2024-09-16T00:27:11.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.765+0000 7f7dfb7fe640 1 -- 172.21.15.33:0/1221397551 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628957cbd70 con 0x5628958e47c0 2024-09-16T00:27:11.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.765+0000 7f7dfb7fe640 10 client.26674 handle_command_reply: tid=0 2024-09-16T00:27:11.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.765+0000 7f7e24768740 1 client.26674 shutdown 2024-09-16T00:27:11.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.766+0000 7f7e24768740 1 -- 172.21.15.33:0/1221397551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585fb40 msgr2=0x562895862fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:11.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.766+0000 7f7e24768740 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585fb40 0x562895862fc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7e0401a820 tx=0x7f7e04003ed0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.766+0000 7f7e24768740 1 -- 172.21.15.33:0/1221397551 shutdown_connections 2024-09-16T00:27:11.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.766+0000 7f7e24768740 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5628958e47c0 0x5628958e6be0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.766+0000 7f7e24768740 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562895869370 0x562895863750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.766+0000 7f7e24768740 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56289585fb40 0x562895862fc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.766+0000 7f7dfb7fe640 0 client.26674 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:27:11.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.766+0000 7f7e24768740 1 --2- 172.21.15.33:0/1221397551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56289585e5d0 0x562895871da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.766+0000 7f7e24768740 1 -- 172.21.15.33:0/1221397551 >> 172.21.15.33:0/1221397551 conn(0x562895827310 msgr2=0x562895868100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:11.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.766+0000 7f7e24768740 1 -- 172.21.15.33:0/1221397551 shutdown_connections 2024-09-16T00:27:11.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.766+0000 7f7e24768740 1 -- 172.21.15.33:0/1221397551 wait complete. 2024-09-16T00:27:11.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.766+0000 7f7e24768740 20 client.26674 trim_cache size 0 max 16384 2024-09-16T00:27:11.785 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:27:11.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.767+0000 7f7e2238f640 1 -- 172.21.15.33:0/2336809030 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7e080a3240 msgr2=0x7f7e080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:11.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.767+0000 7f7e2238f640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7e080a3240 0x7f7e080a5700 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009230 tx=0x7f7e0c005bd0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.767+0000 7f7e2238f640 1 -- 172.21.15.33:0/2336809030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e1c102dd0 msgr2=0x7f7e1c198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:11.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.767+0000 7f7e2238f640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e1c102dd0 0x7f7e1c198560 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009fd0 tx=0x7f7e1400ef90 comp rx=0 tx=0).stop 2024-09-16T00:27:11.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.767+0000 7f7e2238f640 1 -- 172.21.15.33:0/2336809030 shutdown_connections 2024-09-16T00:27:11.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.767+0000 7f7e2238f640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7e080a3240 0x7f7e080a5700 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.767+0000 7f7e2238f640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e1c108300 0x7f7e1c19ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.767+0000 7f7e2238f640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c103780 0x7f7e1c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.767+0000 7f7e2238f640 1 --2- 172.21.15.33:0/2336809030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e1c102dd0 0x7f7e1c198560 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:11.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.767+0000 7f7e2238f640 1 -- 172.21.15.33:0/2336809030 >> 172.21.15.33:0/2336809030 conn(0x7f7e1c0fc7d0 msgr2=0x7f7e1c1063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:11.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.767+0000 7f7e2238f640 1 -- 172.21.15.33:0/2336809030 shutdown_connections 2024-09-16T00:27:11.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:11.767+0000 7f7e2238f640 1 -- 172.21.15.33:0/2336809030 wait complete. 2024-09-16T00:27:11.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:11 smithi083 ceph-mon[35242]: pgmap v651: 129 pgs: 129 active+clean; 277 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 825 KiB/s wr, 115 op/s 2024-09-16T00:27:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:11 smithi060 ceph-mon[35185]: pgmap v651: 129 pgs: 129 active+clean; 277 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 825 KiB/s wr, 115 op/s 2024-09-16T00:27:12.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 792237 9.60 MB/sec execute 380 sec latency 24.404 ms 2024-09-16T00:27:13.303 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:27:13.546 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:13 smithi033 ceph-mon[31243]: pgmap v652: 129 pgs: 129 active+clean; 282 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-09-16T00:27:13.599 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:13.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 795269 9.62 MB/sec execute 381 sec latency 13.214 ms 2024-09-16T00:27:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:13 smithi083 ceph-mon[35242]: pgmap v652: 129 pgs: 129 active+clean; 282 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-09-16T00:27:13.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:13 smithi060 ceph-mon[35185]: pgmap v652: 129 pgs: 129 active+clean; 282 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-09-16T00:27:14.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.116+0000 7f6fd1344640 1 -- 172.21.15.33:0/3557344692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fcc102da0 msgr2=0x7f6fcc103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:14.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.116+0000 7f6fd1344640 1 --2- 172.21.15.33:0/3557344692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fcc102da0 0x7f6fcc103180 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009ca0 tx=0x7f6fc402a950 comp rx=0 tx=0).stop 2024-09-16T00:27:14.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.116+0000 7f6fd1344640 1 -- 172.21.15.33:0/3557344692 shutdown_connections 2024-09-16T00:27:14.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.116+0000 7f6fd1344640 1 --2- 172.21.15.33:0/3557344692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fcc10c180 0x7f6fcc10e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.116+0000 7f6fd1344640 1 --2- 172.21.15.33:0/3557344692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fcc1036c0 0x7f6fcc10bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.116+0000 7f6fd1344640 1 --2- 172.21.15.33:0/3557344692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fcc102da0 0x7f6fcc103180 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.116+0000 7f6fd1344640 1 -- 172.21.15.33:0/3557344692 >> 172.21.15.33:0/3557344692 conn(0x7f6fcc0fc7a0 msgr2=0x7f6fcc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:14.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.116+0000 7f6fd1344640 1 -- 172.21.15.33:0/3557344692 shutdown_connections 2024-09-16T00:27:14.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.116+0000 7f6fd1344640 1 -- 172.21.15.33:0/3557344692 wait complete. 2024-09-16T00:27:14.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.116+0000 7f6fd1344640 1 Processor -- start 2024-09-16T00:27:14.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fd1344640 1 -- start start 2024-09-16T00:27:14.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fd1344640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fcc102da0 0x7f6fcc198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fd1344640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fcc1036c0 0x7f6fcc198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fd1344640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fcc10c180 0x7f6fcc19ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fd1344640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc111f60 con 0x7f6fcc10c180 2024-09-16T00:27:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fd1344640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc111de0 con 0x7f6fcc102da0 2024-09-16T00:27:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fd1344640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc1120e0 con 0x7f6fcc1036c0 2024-09-16T00:27:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fcb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fcc10c180 0x7f6fcc19ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fcb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fcc10c180 0x7f6fcc19ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48378/0 (socket says 172.21.15.33:48378) 2024-09-16T00:27:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fcb7fe640 1 -- 172.21.15.33:0/3956685580 learned_addr learned my addr 172.21.15.33:0/3956685580 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fcc1036c0 0x7f6fcc198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fcaffd640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fcc102da0 0x7f6fcc198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fcb7fe640 1 -- 172.21.15.33:0/3956685580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fcc1036c0 msgr2=0x7f6fcc198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fcb7fe640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fcc1036c0 0x7f6fcc198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fcb7fe640 1 -- 172.21.15.33:0/3956685580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fcc102da0 msgr2=0x7f6fcc198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fcb7fe640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fcc102da0 0x7f6fcc198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fcb7fe640 1 -- 172.21.15.33:0/3956685580 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fcc19d3d0 con 0x7f6fcc10c180 2024-09-16T00:27:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fcc1036c0 0x7f6fcc198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fcaffd640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fcc102da0 0x7f6fcc198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:27:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.117+0000 7f6fcb7fe640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fcc10c180 0x7f6fcc19ccf0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00e970 tx=0x7f6fbc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.118+0000 7f6fc3fff640 1 -- 172.21.15.33:0/3956685580 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00ccc0 con 0x7f6fcc10c180 2024-09-16T00:27:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.118+0000 7f6fd1344640 1 -- 172.21.15.33:0/3956685580 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fcc19d6c0 con 0x7f6fcc10c180 2024-09-16T00:27:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.118+0000 7f6fc3fff640 1 -- 172.21.15.33:0/3956685580 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fbc00ce60 con 0x7f6fcc10c180 2024-09-16T00:27:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.118+0000 7f6fd1344640 1 -- 172.21.15.33:0/3956685580 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fcc1a4fa0 con 0x7f6fcc10c180 2024-09-16T00:27:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.118+0000 7f6fc3fff640 1 -- 172.21.15.33:0/3956685580 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc010450 con 0x7f6fcc10c180 2024-09-16T00:27:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.120+0000 7f6fc3fff640 1 -- 172.21.15.33:0/3956685580 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6fbc005020 con 0x7f6fcc10c180 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.121+0000 7f6fd371d740 1 Processor -- start 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.121+0000 7f6fd371d740 1 -- start start 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.121+0000 7f6fd371d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b21090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.121+0000 7f6fd371d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b215d0 0x560933b21a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.121+0000 7f6fd371d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b30780 0x560933b32b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.121+0000 7f6fd371d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560933a872f0 con 0x560933b30780 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.121+0000 7f6fd371d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560933a2ecf0 con 0x560933b215d0 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.121+0000 7f6fd371d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560933a832b0 con 0x560933b22b40 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.121+0000 7f6fc3fff640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6fb80a3240 0x7f6fb80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fca7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b21090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fca7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b21090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59804/0 (socket says 172.21.15.33:59804) 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fca7fc640 1 -- 172.21.15.33:0/3782266559 learned_addr learned my addr 172.21.15.33:0/3782266559 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fcb7fe640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6fb80a3240 0x7f6fb80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/3782266559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b30780 0x560933b32b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fcaffd640 1 --2- 172.21.15.33:0/3782266559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b215d0 0x560933b21a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fc3fff640 1 -- 172.21.15.33:0/3956685580 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6fbc0d01e0 con 0x7f6fcc10c180 2024-09-16T00:27:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fca7fc640 1 -- 172.21.15.33:0/3782266559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b215d0 msgr2=0x560933b21a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/3782266559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b215d0 0x560933b21a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fca7fc640 1 -- 172.21.15.33:0/3782266559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b30780 msgr2=0x560933b32b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/3782266559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b30780 0x560933b32b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fca7fc640 1 -- 172.21.15.33:0/3782266559 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560933a133a0 con 0x560933b22b40 2024-09-16T00:27:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/3782266559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b21090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009fd0 tx=0x7f6fb4009ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35d040919f3dd90e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fcb7fe640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6fb80a3240 0x7f6fb80a5700 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc09c310 tx=0x7f6fbc066d10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fc17fa640 1 -- 172.21.15.33:0/3782266559 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb400fc50 con 0x560933b22b40 2024-09-16T00:27:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fc17fa640 1 -- 172.21.15.33:0/3782266559 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fb400fdf0 con 0x560933b22b40 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.122+0000 7f6fc17fa640 1 -- 172.21.15.33:0/3782266559 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb400e6f0 con 0x560933b22b40 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.123+0000 7f6fd371d740 1 -- 172.21.15.33:0/3782266559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 msgr2=0x560933b21090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.123+0000 7f6fd371d740 1 --2- 172.21.15.33:0/3782266559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b21090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009fd0 tx=0x7f6fb4009ba0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.123+0000 7f6fd371d740 1 -- 172.21.15.33:0/3782266559 shutdown_connections 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.123+0000 7f6fd371d740 1 --2- 172.21.15.33:0/3782266559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b30780 0x560933b32b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.123+0000 7f6fd371d740 1 --2- 172.21.15.33:0/3782266559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b215d0 0x560933b21a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.123+0000 7f6fd371d740 1 --2- 172.21.15.33:0/3782266559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b21090 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.123+0000 7f6fd371d740 1 -- 172.21.15.33:0/3782266559 >> 172.21.15.33:0/3782266559 conn(0x560933aea310 msgr2=0x560933b1dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.123+0000 7f6fd371d740 1 -- 172.21.15.33:0/3782266559 shutdown_connections 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.123+0000 7f6fd371d740 1 -- 172.21.15.33:0/3782266559 wait complete. 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fd371d740 1 -- start start 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fd371d740 1 Processor -- start 2024-09-16T00:27:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fd371d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 0x560933b301f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fd371d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b2d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fd371d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b30780 0x560933b2dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fd371d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560933b33480 con 0x560933b215d0 2024-09-16T00:27:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fd371d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560933a42060 con 0x560933b30780 2024-09-16T00:27:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fd371d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560933b33600 con 0x560933b22b40 2024-09-16T00:27:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fd371d740 10 client.0 _ll_register_callbacks cb 0x5609337958c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fca7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 0x560933b301f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fca7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 0x560933b301f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48396/0 (socket says 172.21.15.33:48396) 2024-09-16T00:27:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fca7fc640 1 -- 172.21.15.33:0/1350844759 learned_addr learned my addr 172.21.15.33:0/1350844759 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fcaffd640 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b2d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.124+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b30780 0x560933b2dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fca7fc640 1 -- 172.21.15.33:0/1350844759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 msgr2=0x560933b2d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:14.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b2d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fca7fc640 1 -- 172.21.15.33:0/1350844759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b30780 msgr2=0x560933b2dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:14.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b30780 0x560933b2dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fca7fc640 1 -- 172.21.15.33:0/1350844759 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609337f83c0 con 0x560933b215d0 2024-09-16T00:27:14.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 0x560933b301f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400ffe0 tx=0x7f6fb401feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e6a34e643e2266 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:14.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fb27fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:27:14.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fb27fc640 1 -- 172.21.15.33:0/1350844759 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4005220 con 0x560933b215d0 2024-09-16T00:27:14.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fb27fc640 1 -- 172.21.15.33:0/1350844759 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6fb40053c0 con 0x560933b215d0 2024-09-16T00:27:14.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fd371d740 1 -- 172.21.15.33:0/1350844759 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fb400fdf0 con 0x560933b215d0 2024-09-16T00:27:14.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fb27fc640 1 -- 172.21.15.33:0/1350844759 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6fb4034050 con 0x560933b215d0 2024-09-16T00:27:14.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fd371d740 10 client.17286 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:14.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fd371d740 1 -- 172.21.15.33:0/1350844759 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560933b3efb0 con 0x560933b215d0 2024-09-16T00:27:14.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fb27fc640 1 -- 172.21.15.33:0/1350844759 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fb400fdf0 con 0x560933b215d0 2024-09-16T00:27:14.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.125+0000 7f6fb27fc640 1 -- 172.21.15.33:0/1350844759 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f6fb402a030 con 0x560933b215d0 2024-09-16T00:27:14.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.126+0000 7f6fd371d740 10 client.17286 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:14.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.126+0000 7f6fd371d740 10 client.17286 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:27:14.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.126+0000 7f6fd371d740 20 client.17286 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:14.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.126+0000 7f6fd371d740 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x560933b40300 0x560933b426f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.126+0000 7f6fd371d740 4 client.17286 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:27:14.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.126+0000 7f6fd371d740 1 -- 172.21.15.33:0/1350844759 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560933a94860 con 0x560933b40300 2024-09-16T00:27:14.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.126+0000 7f6fcaffd640 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x560933b40300 0x560933b426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.126+0000 7f6fb27fc640 10 client.17286 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:27:14.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.126+0000 7f6fcaffd640 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x560933b40300 0x560933b426f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b07d709b34e4a007 server_cookie=1dac885d5da7a39e in_seq=0 out_seq=0 2024-09-16T00:27:14.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.127+0000 7f6fb27fc640 1 -- 172.21.15.33:0/1350844759 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560933a94860 con 0x560933b40300 2024-09-16T00:27:14.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.127+0000 7f6fb27fc640 10 client.17286 handle_command_reply: tid=0 2024-09-16T00:27:14.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.127+0000 7f6fd371d740 1 client.17286 shutdown 2024-09-16T00:27:14.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.127+0000 7f6fd371d740 1 -- 172.21.15.33:0/1350844759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 msgr2=0x560933b301f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:14.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.127+0000 7f6fd371d740 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 0x560933b301f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400ffe0 tx=0x7f6fb401feb0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.127+0000 7f6fd371d740 1 -- 172.21.15.33:0/1350844759 shutdown_connections 2024-09-16T00:27:14.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.127+0000 7f6fd371d740 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x560933b40300 0x560933b426f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.127+0000 7f6fd371d740 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b30780 0x560933b2dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.127+0000 7f6fd371d740 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b2d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.127+0000 7f6fd371d740 1 --2- 172.21.15.33:0/1350844759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 0x560933b301f0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.127+0000 7f6fd371d740 1 -- 172.21.15.33:0/1350844759 >> 172.21.15.33:0/1350844759 conn(0x560933aea310 msgr2=0x560933b23960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:14.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.128+0000 7f6fb27fc640 0 client.17286 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:27:14.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.128+0000 7f6fd371d740 1 -- 172.21.15.33:0/1350844759 shutdown_connections 2024-09-16T00:27:14.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.128+0000 7f6fd371d740 1 -- 172.21.15.33:0/1350844759 wait complete. 2024-09-16T00:27:14.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.128+0000 7f6fd371d740 20 client.17286 trim_cache size 0 max 16384 2024-09-16T00:27:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.143+0000 7f6fd371d740 1 Processor -- start 2024-09-16T00:27:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.143+0000 7f6fd371d740 1 -- start start 2024-09-16T00:27:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.143+0000 7f6fd371d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b215d0 0x560933b3c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.143+0000 7f6fd371d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b22b40 0x560933b47d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.143+0000 7f6fd371d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b30780 0x560933b482c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.143+0000 7f6fd371d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc1af8c0 con 0x560933b22b40 2024-09-16T00:27:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.143+0000 7f6fd371d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc1ad190 con 0x560933b215d0 2024-09-16T00:27:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.143+0000 7f6fd371d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc1aafe0 con 0x560933b30780 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.143+0000 7f6fcaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b22b40 0x560933b47d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.143+0000 7f6fcaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b22b40 0x560933b47d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48410/0 (socket says 172.21.15.33:48410) 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.143+0000 7f6fcaffd640 1 -- 172.21.15.33:0/1773896823 learned_addr learned my addr 172.21.15.33:0/1773896823 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.144+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/1773896823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b215d0 0x560933b3c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.144+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/1773896823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b30780 0x560933b482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.144+0000 7f6fcaffd640 1 -- 172.21.15.33:0/1773896823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b30780 msgr2=0x560933b482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.144+0000 7f6fcaffd640 1 --2- 172.21.15.33:0/1773896823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b30780 0x560933b482c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.144+0000 7f6fcaffd640 1 -- 172.21.15.33:0/1773896823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b215d0 msgr2=0x560933b3c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.144+0000 7f6fcaffd640 1 --2- 172.21.15.33:0/1773896823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b215d0 0x560933b3c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.144+0000 7f6fcaffd640 1 -- 172.21.15.33:0/1773896823 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560933b3cc60 con 0x560933b22b40 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.144+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/1773896823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b215d0 0x560933b3c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.144+0000 7f6fcaffd640 1 --2- 172.21.15.33:0/1773896823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b22b40 0x560933b47d80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4035000 tx=0x7f6fc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f50212cce246240 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.144+0000 7f6fc0ff9640 1 -- 172.21.15.33:0/1773896823 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4034040 con 0x560933b22b40 2024-09-16T00:27:14.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.144+0000 7f6fc0ff9640 1 -- 172.21.15.33:0/1773896823 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc402fb40 con 0x560933b22b40 2024-09-16T00:27:14.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.144+0000 7f6fc0ff9640 1 -- 172.21.15.33:0/1773896823 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc40043c0 con 0x560933b22b40 2024-09-16T00:27:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.145+0000 7f6fd371d740 1 -- 172.21.15.33:0/1773896823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b22b40 msgr2=0x560933b47d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.145+0000 7f6fd371d740 1 --2- 172.21.15.33:0/1773896823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b22b40 0x560933b47d80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4035000 tx=0x7f6fc4004a40 comp rx=0 tx=0).stop 2024-09-16T00:27:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.145+0000 7f6fd371d740 1 -- 172.21.15.33:0/1773896823 shutdown_connections 2024-09-16T00:27:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.145+0000 7f6fd371d740 1 --2- 172.21.15.33:0/1773896823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b30780 0x560933b482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.145+0000 7f6fd371d740 1 --2- 172.21.15.33:0/1773896823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b22b40 0x560933b47d80 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.145+0000 7f6fd371d740 1 --2- 172.21.15.33:0/1773896823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b215d0 0x560933b3c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.145+0000 7f6fd371d740 1 -- 172.21.15.33:0/1773896823 >> 172.21.15.33:0/1773896823 conn(0x560933aea310 msgr2=0x560933b2c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.145+0000 7f6fd371d740 1 -- 172.21.15.33:0/1773896823 shutdown_connections 2024-09-16T00:27:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.145+0000 7f6fd371d740 1 -- 172.21.15.33:0/1773896823 wait complete. 2024-09-16T00:27:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.145+0000 7f6fd371d740 1 -- start start 2024-09-16T00:27:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fd371d740 1 Processor -- start 2024-09-16T00:27:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fd371d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 0x560933b34560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fd371d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b34aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fd371d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b30780 0x560933b34fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fd371d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc1ab650 con 0x560933b215d0 2024-09-16T00:27:14.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fd371d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc1acac0 con 0x560933b30780 2024-09-16T00:27:14.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fd371d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc1abd20 con 0x560933b22b40 2024-09-16T00:27:14.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fd371d740 10 client.0 _ll_register_callbacks cb 0x7f6fb40116b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:14.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fca7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 0x560933b34560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fcaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b34aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fca7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 0x560933b34560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48412/0 (socket says 172.21.15.33:48412) 2024-09-16T00:27:14.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fcaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b34aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59820/0 (socket says 172.21.15.33:59820) 2024-09-16T00:27:14.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fca7fc640 1 -- 172.21.15.33:0/2088320877 learned_addr learned my addr 172.21.15.33:0/2088320877 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:14.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fca7fc640 1 -- 172.21.15.33:0/2088320877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 msgr2=0x560933b34aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/2088320877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b34aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fca7fc640 1 -- 172.21.15.33:0/2088320877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b30780 msgr2=0x560933b34fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/2088320877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b30780 0x560933b34fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fca7fc640 1 -- 172.21.15.33:0/2088320877 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560933ba9d50 con 0x560933b215d0 2024-09-16T00:27:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.146+0000 7f6fcaffd640 1 --2- 172.21.15.33:0/2088320877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b34aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fca7fc640 1 --2- 172.21.15.33:0/2088320877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 0x560933b34560 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f6fb40115a0 tx=0x7f6fb4009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d989946e48c661 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fb27fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:27:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fb27fc640 1 -- 172.21.15.33:0/2088320877 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4005d30 con 0x560933b215d0 2024-09-16T00:27:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fb27fc640 1 -- 172.21.15.33:0/2088320877 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6fb4003800 con 0x560933b215d0 2024-09-16T00:27:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fd371d740 1 -- 172.21.15.33:0/2088320877 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fc402fb40 con 0x560933b215d0 2024-09-16T00:27:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fb27fc640 1 -- 172.21.15.33:0/2088320877 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6fb4003a00 con 0x560933b215d0 2024-09-16T00:27:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fd371d740 10 client.17292 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:14.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fd371d740 1 -- 172.21.15.33:0/2088320877 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560933baafb0 con 0x560933b215d0 2024-09-16T00:27:14.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fb27fc640 1 -- 172.21.15.33:0/2088320877 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc402fb40 con 0x560933b215d0 2024-09-16T00:27:14.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fb27fc640 1 -- 172.21.15.33:0/2088320877 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f6fb4037040 con 0x560933b215d0 2024-09-16T00:27:14.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fd371d740 10 client.17292 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:14.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fd371d740 10 client.17292 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:27:14.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fd371d740 20 client.17292 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:14.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fd371d740 1 --2- 172.21.15.33:0/2088320877 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x560933bab560 0x560933bad980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:14.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fd371d740 4 client.17292 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:27:14.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.147+0000 7f6fd371d740 1 -- 172.21.15.33:0/2088320877 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560933a8ed70 con 0x560933bab560 2024-09-16T00:27:14.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fcaffd640 1 --2- 172.21.15.33:0/2088320877 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x560933bab560 0x560933bad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:14.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fcaffd640 1 --2- 172.21.15.33:0/2088320877 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x560933bab560 0x560933bad980 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8fcc03267697fbac server_cookie=e47cbaaadf96f14a in_seq=0 out_seq=0 2024-09-16T00:27:14.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fb27fc640 10 client.17292 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fb27fc640 1 -- 172.21.15.33:0/2088320877 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560933a8ed70 con 0x560933bab560 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fb27fc640 10 client.17292 handle_command_reply: tid=0 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fd371d740 1 client.17292 shutdown 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fd371d740 1 -- 172.21.15.33:0/2088320877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 msgr2=0x560933b34560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fd371d740 1 --2- 172.21.15.33:0/2088320877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 0x560933b34560 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f6fb40115a0 tx=0x7f6fb4009f60 comp rx=0 tx=0).stop 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fd371d740 1 -- 172.21.15.33:0/2088320877 shutdown_connections 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fd371d740 1 --2- 172.21.15.33:0/2088320877 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x560933bab560 0x560933bad980 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fb27fc640 0 client.17292 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fd371d740 1 --2- 172.21.15.33:0/2088320877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560933b30780 0x560933b34fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fd371d740 1 --2- 172.21.15.33:0/2088320877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560933b22b40 0x560933b34aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fd371d740 1 --2- 172.21.15.33:0/2088320877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560933b215d0 0x560933b34560 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.148+0000 7f6fd371d740 1 -- 172.21.15.33:0/2088320877 >> 172.21.15.33:0/2088320877 conn(0x560933aea310 msgr2=0x560933b23110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.149+0000 7f6fd371d740 1 -- 172.21.15.33:0/2088320877 shutdown_connections 2024-09-16T00:27:14.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.149+0000 7f6fd371d740 1 -- 172.21.15.33:0/2088320877 wait complete. 2024-09-16T00:27:14.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.149+0000 7f6fd371d740 20 client.17292 trim_cache size 0 max 16384 2024-09-16T00:27:14.156 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:27:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.149+0000 7f6fd1344640 1 -- 172.21.15.33:0/3956685580 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6fb80a3240 msgr2=0x7f6fb80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.149+0000 7f6fd1344640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6fb80a3240 0x7f6fb80a5700 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc09c310 tx=0x7f6fbc066d10 comp rx=0 tx=0).stop 2024-09-16T00:27:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.149+0000 7f6fd1344640 1 -- 172.21.15.33:0/3956685580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fcc10c180 msgr2=0x7f6fcc19ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.149+0000 7f6fd1344640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fcc10c180 0x7f6fcc19ccf0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00e970 tx=0x7f6fbc00ed70 comp rx=0 tx=0).stop 2024-09-16T00:27:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.150+0000 7f6fd1344640 1 -- 172.21.15.33:0/3956685580 shutdown_connections 2024-09-16T00:27:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.150+0000 7f6fd1344640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6fb80a3240 0x7f6fb80a5700 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.150+0000 7f6fd1344640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fcc10c180 0x7f6fcc19ccf0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.150+0000 7f6fd1344640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fcc1036c0 0x7f6fcc198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.150+0000 7f6fd1344640 1 --2- 172.21.15.33:0/3956685580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fcc102da0 0x7f6fcc198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.150+0000 7f6fd1344640 1 -- 172.21.15.33:0/3956685580 >> 172.21.15.33:0/3956685580 conn(0x7f6fcc0fc7a0 msgr2=0x7f6fcc0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.150+0000 7f6fd1344640 1 -- 172.21.15.33:0/3956685580 shutdown_connections 2024-09-16T00:27:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:14.150+0000 7f6fd1344640 1 -- 172.21.15.33:0/3956685580 wait complete. 2024-09-16T00:27:14.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 796990 9.62 MB/sec execute 382 sec latency 32.440 ms 2024-09-16T00:27:14.895 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:27:15.178 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:15.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:15 smithi033 ceph-mon[31243]: pgmap v653: 129 pgs: 129 active+clean; 280 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 241 op/s 2024-09-16T00:27:15.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 799454 9.63 MB/sec execute 383 sec latency 44.243 ms 2024-09-16T00:27:15.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.704+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/520709287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbd41082a0 msgr2=0x7fdbd410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:15.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.704+0000 7fdbd9f8f640 1 --2- 172.21.15.33:0/520709287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbd41082a0 0x7fdbd410ec40 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009cd0 tx=0x7fdbcc02a8e0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.704+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/520709287 shutdown_connections 2024-09-16T00:27:15.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.704+0000 7fdbd9f8f640 1 --2- 172.21.15.33:0/520709287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbd41082a0 0x7fdbd410ec40 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.704+0000 7fdbd9f8f640 1 --2- 172.21.15.33:0/520709287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd41036e0 0x7fdbd4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.704+0000 7fdbd9f8f640 1 --2- 172.21.15.33:0/520709287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd4102d30 0x7fdbd4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.704+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/520709287 >> 172.21.15.33:0/520709287 conn(0x7fdbd40fc7d0 msgr2=0x7fdbd40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:15.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.704+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/520709287 shutdown_connections 2024-09-16T00:27:15.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.704+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/520709287 wait complete. 2024-09-16T00:27:15.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.704+0000 7fdbd9f8f640 1 Processor -- start 2024-09-16T00:27:15.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.704+0000 7fdbd9f8f640 1 -- start start 2024-09-16T00:27:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd9f8f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbd4102d30 0x7fdbd4198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd9f8f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd41036e0 0x7fdbd4198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd9f8f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd41082a0 0x7fdbd419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd9f8f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41120a0 con 0x7fdbd4102d30 2024-09-16T00:27:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd9f8f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd4111f20 con 0x7fdbd41082a0 2024-09-16T00:27:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd9f8f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd4112220 con 0x7fdbd41036e0 2024-09-16T00:27:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbd4102d30 0x7fdbd4198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbd4102d30 0x7fdbd4198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48440/0 (socket says 172.21.15.33:48440) 2024-09-16T00:27:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd37fe640 1 -- 172.21.15.33:0/1114950091 learned_addr learned my addr 172.21.15.33:0/1114950091 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbcaffd640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd41036e0 0x7fdbd4198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd3fff640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd41082a0 0x7fdbd419ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd37fe640 1 -- 172.21.15.33:0/1114950091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd41036e0 msgr2=0x7fdbd4198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:15.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd37fe640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd41036e0 0x7fdbd4198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd37fe640 1 -- 172.21.15.33:0/1114950091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd41082a0 msgr2=0x7fdbd419ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:15.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd37fe640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd41082a0 0x7fdbd419ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.705+0000 7fdbd37fe640 1 -- 172.21.15.33:0/1114950091 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd419d4f0 con 0x7fdbd4102d30 2024-09-16T00:27:15.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.706+0000 7fdbd37fe640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbd4102d30 0x7fdbd4198540 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400d950 tx=0x7fdbc400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:15.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.706+0000 7fdbd17fa640 1 -- 172.21.15.33:0/1114950091 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4017070 con 0x7fdbd4102d30 2024-09-16T00:27:15.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.706+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/1114950091 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd419d7e0 con 0x7fdbd4102d30 2024-09-16T00:27:15.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.706+0000 7fdbd17fa640 1 -- 172.21.15.33:0/1114950091 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbc4009c30 con 0x7fdbd4102d30 2024-09-16T00:27:15.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.706+0000 7fdbd17fa640 1 -- 172.21.15.33:0/1114950091 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc400f6b0 con 0x7fdbd4102d30 2024-09-16T00:27:15.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.706+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/1114950091 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbd41a50c0 con 0x7fdbd4102d30 2024-09-16T00:27:15.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.708+0000 7fdbd17fa640 1 -- 172.21.15.33:0/1114950091 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fdbc40109d0 con 0x7fdbd4102d30 2024-09-16T00:27:15.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.708+0000 7fdbdc368740 1 Processor -- start 2024-09-16T00:27:15.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.708+0000 7fdbdc368740 1 -- start start 2024-09-16T00:27:15.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbdc368740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e2af50 0x560d19e294a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbdc368740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 0x560d19e29e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbdc368740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e34780 0x560d19e36b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbdc368740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560d19d8b2f0 con 0x560d19e299e0 2024-09-16T00:27:15.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbdc368740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560d19d32cf0 con 0x560d19e2af50 2024-09-16T00:27:15.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbdc368740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560d19d872b0 con 0x560d19e34780 2024-09-16T00:27:15.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 0x560d19e29e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e2af50 0x560d19e294a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e2af50 0x560d19e294a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:37282/0 (socket says 172.21.15.33:37282) 2024-09-16T00:27:15.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 0x560d19e29e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48442/0 (socket says 172.21.15.33:48442) 2024-09-16T00:27:15.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd3fff640 1 -- 172.21.15.33:0/2406747400 learned_addr learned my addr 172.21.15.33:0/2406747400 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:15.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbcaffd640 1 --2- 172.21.15.33:0/2406747400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e34780 0x560d19e36b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd17fa640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdbc00a3240 0x7fdbc00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd3fff640 1 -- 172.21.15.33:0/2406747400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e34780 msgr2=0x560d19e36b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:15.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd3fff640 1 --2- 172.21.15.33:0/2406747400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e34780 0x560d19e36b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd3fff640 1 -- 172.21.15.33:0/2406747400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 msgr2=0x560d19e29e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:15.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd3fff640 1 --2- 172.21.15.33:0/2406747400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 0x560d19e29e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd3fff640 1 -- 172.21.15.33:0/2406747400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d19d173a0 con 0x560d19e2af50 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbcaffd640 1 --2- 172.21.15.33:0/2406747400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e34780 0x560d19e36b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd17fa640 1 -- 172.21.15.33:0/1114950091 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fdbc40cf950 con 0x7fdbd4102d30 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.709+0000 7fdbd3fff640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdbc00a3240 0x7fdbc00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.710+0000 7fdbd3fff640 1 --2- 172.21.15.33:0/2406747400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e2af50 0x560d19e294a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009cd0 tx=0x7fdbcc031500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f7fbae3b78f7657 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.710+0000 7fdbc9ffb640 1 -- 172.21.15.33:0/2406747400 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc041040 con 0x560d19e2af50 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.710+0000 7fdbc9ffb640 1 -- 172.21.15.33:0/2406747400 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbcc043070 con 0x560d19e2af50 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.710+0000 7fdbd3fff640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdbc00a3240 0x7fdbc00a5700 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc04c000 tx=0x7fdbcc04d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.710+0000 7fdbc9ffb640 1 -- 172.21.15.33:0/2406747400 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc030cc0 con 0x560d19e2af50 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.711+0000 7fdbdc368740 1 -- 172.21.15.33:0/2406747400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e2af50 msgr2=0x560d19e294a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.711+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2406747400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e2af50 0x560d19e294a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009cd0 tx=0x7fdbcc031500 comp rx=0 tx=0).stop 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.711+0000 7fdbdc368740 1 -- 172.21.15.33:0/2406747400 shutdown_connections 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.711+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2406747400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e34780 0x560d19e36b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.711+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2406747400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 0x560d19e29e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.711+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2406747400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e2af50 0x560d19e294a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.711+0000 7fdbdc368740 1 -- 172.21.15.33:0/2406747400 >> 172.21.15.33:0/2406747400 conn(0x560d19dee310 msgr2=0x560d19e26070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.711+0000 7fdbdc368740 1 -- 172.21.15.33:0/2406747400 shutdown_connections 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.711+0000 7fdbdc368740 1 -- 172.21.15.33:0/2406747400 wait complete. 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.711+0000 7fdbdc368740 1 -- start start 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbdc368740 1 Processor -- start 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbdc368740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e299e0 0x560d19e341a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbdc368740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e2af50 0x560d19e31450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbdc368740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e34780 0x560d19e31a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbdc368740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560d19e37480 con 0x560d19e2af50 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbdc368740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560d19d46060 con 0x560d19e34780 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbdc368740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560d19e37600 con 0x560d19e299e0 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbcaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e2af50 0x560d19e31450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbcaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e2af50 0x560d19e31450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48444/0 (socket says 172.21.15.33:48444) 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbcaffd640 1 -- 172.21.15.33:0/2807620055 learned_addr learned my addr 172.21.15.33:0/2807620055 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbdc368740 10 client.0 _ll_register_callbacks cb 0x560d19a998c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbd37fe640 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e299e0 0x560d19e341a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbd3fff640 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e34780 0x560d19e31a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbcaffd640 1 -- 172.21.15.33:0/2807620055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e299e0 msgr2=0x560d19e341a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbcaffd640 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e299e0 0x560d19e341a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbcaffd640 1 -- 172.21.15.33:0/2807620055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e34780 msgr2=0x560d19e31a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbcaffd640 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e34780 0x560d19e31a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbcaffd640 1 -- 172.21.15.33:0/2807620055 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d19afc3c0 con 0x560d19e2af50 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.712+0000 7fdbd37fe640 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e299e0 0x560d19e341a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.713+0000 7fdbcaffd640 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e2af50 0x560d19e31450 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc0126e0 tx=0x7fdbbc012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a294f9593aab52af server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.713+0000 7fdbb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:27:15.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.713+0000 7fdbb2ffd640 1 -- 172.21.15.33:0/2807620055 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc004050 con 0x560d19e2af50 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.713+0000 7fdbb2ffd640 1 -- 172.21.15.33:0/2807620055 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fdbbc0041f0 con 0x560d19e2af50 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.713+0000 7fdbdc368740 1 -- 172.21.15.33:0/2807620055 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbcc043070 con 0x560d19e2af50 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.713+0000 7fdbb2ffd640 1 -- 172.21.15.33:0/2807620055 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdbbc0043f0 con 0x560d19e2af50 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.713+0000 7fdbdc368740 10 client.17304 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.713+0000 7fdbdc368740 1 -- 172.21.15.33:0/2807620055 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560d19e43000 con 0x560d19e2af50 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.713+0000 7fdbb2ffd640 1 -- 172.21.15.33:0/2807620055 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbcc043070 con 0x560d19e2af50 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.713+0000 7fdbb2ffd640 1 -- 172.21.15.33:0/2807620055 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fdbbc0039c0 con 0x560d19e2af50 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.713+0000 7fdbdc368740 10 client.17304 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.713+0000 7fdbdc368740 10 client.17304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.714+0000 7fdbdc368740 20 client.17304 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.714+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560d19e444c0 0x560d19e468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.714+0000 7fdbdc368740 4 client.17304 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:27:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.714+0000 7fdbdc368740 1 -- 172.21.15.33:0/2807620055 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d19d98860 con 0x560d19e444c0 2024-09-16T00:27:15.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.714+0000 7fdbd37fe640 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560d19e444c0 0x560d19e468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.714+0000 7fdbb2ffd640 10 client.17304 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:15.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.714+0000 7fdbd37fe640 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560d19e444c0 0x560d19e468f0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=187cb188e2e4c75f server_cookie=640aa81a5f1b9460 in_seq=0 out_seq=0 2024-09-16T00:27:15.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbb2ffd640 1 -- 172.21.15.33:0/2807620055 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560d19d98860 con 0x560d19e444c0 2024-09-16T00:27:15.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbb2ffd640 10 client.17304 handle_command_reply: tid=0 2024-09-16T00:27:15.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbdc368740 1 client.17304 shutdown 2024-09-16T00:27:15.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbdc368740 1 -- 172.21.15.33:0/2807620055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e2af50 msgr2=0x560d19e31450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:15.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e2af50 0x560d19e31450 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc0126e0 tx=0x7fdbbc012ae0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbdc368740 1 -- 172.21.15.33:0/2807620055 shutdown_connections 2024-09-16T00:27:15.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560d19e444c0 0x560d19e468f0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e34780 0x560d19e31a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e2af50 0x560d19e31450 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2807620055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e299e0 0x560d19e341a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbdc368740 1 -- 172.21.15.33:0/2807620055 >> 172.21.15.33:0/2807620055 conn(0x560d19dee310 msgr2=0x560d19e2bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:15.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbb2ffd640 0 client.17304 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:15.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbdc368740 1 -- 172.21.15.33:0/2807620055 shutdown_connections 2024-09-16T00:27:15.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbdc368740 1 -- 172.21.15.33:0/2807620055 wait complete. 2024-09-16T00:27:15.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.715+0000 7fdbdc368740 20 client.17304 trim_cache size 0 max 16384 2024-09-16T00:27:15.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbdc368740 1 Processor -- start 2024-09-16T00:27:15.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbdc368740 1 -- start start 2024-09-16T00:27:15.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbdc368740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 0x560d19e333c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbdc368740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e2af50 0x560d19e2c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbdc368740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e34780 0x560d19e2ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbdc368740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41af760 con 0x560d19e299e0 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbdc368740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41ad030 con 0x560d19e34780 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbdc368740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41aae80 con 0x560d19e2af50 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 0x560d19e333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbcaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e2af50 0x560d19e2c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 0x560d19e333c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48454/0 (socket says 172.21.15.33:48454) 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbd37fe640 1 -- 172.21.15.33:0/2131798308 learned_addr learned my addr 172.21.15.33:0/2131798308 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.730+0000 7fdbd3fff640 1 --2- 172.21.15.33:0/2131798308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e34780 0x560d19e2ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.731+0000 7fdbd37fe640 1 -- 172.21.15.33:0/2131798308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e2af50 msgr2=0x560d19e2c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.731+0000 7fdbd37fe640 1 --2- 172.21.15.33:0/2131798308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e2af50 0x560d19e2c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.731+0000 7fdbd37fe640 1 -- 172.21.15.33:0/2131798308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e34780 msgr2=0x560d19e2ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.731+0000 7fdbd37fe640 1 --2- 172.21.15.33:0/2131798308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e34780 0x560d19e2ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.731+0000 7fdbd37fe640 1 -- 172.21.15.33:0/2131798308 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d19e34510 con 0x560d19e299e0 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.731+0000 7fdbcaffd640 1 --2- 172.21.15.33:0/2131798308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e2af50 0x560d19e2c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.731+0000 7fdbd37fe640 1 --2- 172.21.15.33:0/2131798308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 0x560d19e333c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40ccfd0 tx=0x7fdbc4086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892f60f0664f6f48 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.731+0000 7fdbc97fa640 1 -- 172.21.15.33:0/2131798308 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4020070 con 0x560d19e299e0 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.731+0000 7fdbc97fa640 1 -- 172.21.15.33:0/2131798308 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbc409fea0 con 0x560d19e299e0 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.731+0000 7fdbc97fa640 1 -- 172.21.15.33:0/2131798308 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc40860b0 con 0x560d19e299e0 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.732+0000 7fdbdc368740 1 -- 172.21.15.33:0/2131798308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 msgr2=0x560d19e333c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.732+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2131798308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 0x560d19e333c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40ccfd0 tx=0x7fdbc4086040 comp rx=0 tx=0).stop 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.732+0000 7fdbdc368740 1 -- 172.21.15.33:0/2131798308 shutdown_connections 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.732+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2131798308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e34780 0x560d19e2ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.732+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2131798308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e2af50 0x560d19e2c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.732+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2131798308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e299e0 0x560d19e333c0 secure :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40ccfd0 tx=0x7fdbc4086040 comp rx=0 tx=0).stop 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.732+0000 7fdbdc368740 1 -- 172.21.15.33:0/2131798308 >> 172.21.15.33:0/2131798308 conn(0x560d19dee310 msgr2=0x560d19e32cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.732+0000 7fdbdc368740 1 -- 172.21.15.33:0/2131798308 shutdown_connections 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.732+0000 7fdbdc368740 1 -- 172.21.15.33:0/2131798308 wait complete. 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.733+0000 7fdbdc368740 1 -- start start 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.733+0000 7fdbdc368740 1 Processor -- start 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.733+0000 7fdbdc368740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e2af50 0x560d19e38500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.733+0000 7fdbdc368740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e34780 0x560d19e38a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbdc368740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e38f80 0x560d19eae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbcaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e34780 0x560d19e38a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbcaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e34780 0x560d19e38a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48458/0 (socket says 172.21.15.33:48458) 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbcaffd640 1 -- 172.21.15.33:0/2371575108 learned_addr learned my addr 172.21.15.33:0/2371575108 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbdc368740 1 -- 172.21.15.33:0/2371575108 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41ab4f0 con 0x560d19e34780 2024-09-16T00:27:15.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbdc368740 1 -- 172.21.15.33:0/2371575108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41ac960 con 0x560d19e2af50 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbdc368740 1 -- 172.21.15.33:0/2371575108 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41abbc0 con 0x560d19e38f80 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbdc368740 10 client.0 _ll_register_callbacks cb 0x7fdbbc009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbd3fff640 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e38f80 0x560d19eae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbcaffd640 1 -- 172.21.15.33:0/2371575108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e38f80 msgr2=0x560d19eae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbcaffd640 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e38f80 0x560d19eae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbcaffd640 1 -- 172.21.15.33:0/2371575108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e2af50 msgr2=0x560d19e38500 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbcaffd640 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e2af50 0x560d19e38500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbcaffd640 1 -- 172.21.15.33:0/2371575108 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d19eae6d0 con 0x560d19e34780 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.734+0000 7fdbd3fff640 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e38f80 0x560d19eae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.735+0000 7fdbcaffd640 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e34780 0x560d19e38a40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc003e60 tx=0x7fdbbc00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d50bb00894f5d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.735+0000 7fdbb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.735+0000 7fdbb2ffd640 1 -- 172.21.15.33:0/2371575108 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc005030 con 0x560d19e34780 2024-09-16T00:27:15.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.735+0000 7fdbb2ffd640 1 -- 172.21.15.33:0/2371575108 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fdbbc0051d0 con 0x560d19e34780 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.735+0000 7fdbdc368740 1 -- 172.21.15.33:0/2371575108 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbc409fea0 con 0x560d19e34780 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.735+0000 7fdbb2ffd640 1 -- 172.21.15.33:0/2371575108 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdbbc0053d0 con 0x560d19e34780 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.735+0000 7fdbdc368740 10 client.17310 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.735+0000 7fdbdc368740 1 -- 172.21.15.33:0/2371575108 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560d19eaf880 con 0x560d19e34780 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.736+0000 7fdbb2ffd640 1 -- 172.21.15.33:0/2371575108 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbc409fea0 con 0x560d19e34780 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.736+0000 7fdbb2ffd640 1 -- 172.21.15.33:0/2371575108 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fdbbc010c10 con 0x560d19e34780 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.736+0000 7fdbdc368740 10 client.17310 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.736+0000 7fdbdc368740 10 client.17310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.736+0000 7fdbdc368740 20 client.17310 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.736+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560d19eb0030 0x560d19eb2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.736+0000 7fdbdc368740 4 client.17310 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.736+0000 7fdbdc368740 1 -- 172.21.15.33:0/2371575108 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d19d92d70 con 0x560d19eb0030 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.737+0000 7fdbd37fe640 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560d19eb0030 0x560d19eb2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:15.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.737+0000 7fdbd37fe640 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560d19eb0030 0x560d19eb2450 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70281c36f20a0830 server_cookie=e056c2c2bec18bf9 in_seq=0 out_seq=0 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.737+0000 7fdbb2ffd640 10 client.17310 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.737+0000 7fdbb2ffd640 1 -- 172.21.15.33:0/2371575108 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d19d92d70 con 0x560d19eb0030 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.737+0000 7fdbb2ffd640 10 client.17310 handle_command_reply: tid=0 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.737+0000 7fdbdc368740 1 client.17310 shutdown 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.738+0000 7fdbdc368740 1 -- 172.21.15.33:0/2371575108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e34780 msgr2=0x560d19e38a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.738+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e34780 0x560d19e38a40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc003e60 tx=0x7fdbbc00f3c0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.738+0000 7fdbdc368740 1 -- 172.21.15.33:0/2371575108 shutdown_connections 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.738+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x560d19eb0030 0x560d19eb2450 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.738+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560d19e38f80 0x560d19eae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.738+0000 7fdbb2ffd640 0 client.17310 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.738+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d19e34780 0x560d19e38a40 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.738+0000 7fdbdc368740 1 --2- 172.21.15.33:0/2371575108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560d19e2af50 0x560d19e38500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.738+0000 7fdbdc368740 1 -- 172.21.15.33:0/2371575108 >> 172.21.15.33:0/2371575108 conn(0x560d19dee310 msgr2=0x560d19e334d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:15.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.738+0000 7fdbdc368740 1 -- 172.21.15.33:0/2371575108 shutdown_connections 2024-09-16T00:27:15.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.738+0000 7fdbdc368740 1 -- 172.21.15.33:0/2371575108 wait complete. 2024-09-16T00:27:15.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.738+0000 7fdbdc368740 20 client.17310 trim_cache size 0 max 16384 2024-09-16T00:27:15.854 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:27:15.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.739+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/1114950091 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdbc00a3240 msgr2=0x7fdbc00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:15.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.739+0000 7fdbd9f8f640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdbc00a3240 0x7fdbc00a5700 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc04c000 tx=0x7fdbcc04d000 comp rx=0 tx=0).stop 2024-09-16T00:27:15.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.739+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/1114950091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbd4102d30 msgr2=0x7fdbd4198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:15.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.739+0000 7fdbd9f8f640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbd4102d30 0x7fdbd4198540 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400d950 tx=0x7fdbc400de10 comp rx=0 tx=0).stop 2024-09-16T00:27:15.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.739+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/1114950091 shutdown_connections 2024-09-16T00:27:15.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.739+0000 7fdbd9f8f640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdbc00a3240 0x7fdbc00a5700 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.739+0000 7fdbd9f8f640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd41082a0 0x7fdbd419ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.739+0000 7fdbd9f8f640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd41036e0 0x7fdbd4198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.739+0000 7fdbd9f8f640 1 --2- 172.21.15.33:0/1114950091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbd4102d30 0x7fdbd4198540 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:15.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.739+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/1114950091 >> 172.21.15.33:0/1114950091 conn(0x7fdbd40fc7d0 msgr2=0x7fdbd4105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:15.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.739+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/1114950091 shutdown_connections 2024-09-16T00:27:15.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:15.739+0000 7fdbd9f8f640 1 -- 172.21.15.33:0/1114950091 wait complete. 2024-09-16T00:27:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:15 smithi083 ceph-mon[35242]: pgmap v653: 129 pgs: 129 active+clean; 280 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 241 op/s 2024-09-16T00:27:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:15 smithi060 ceph-mon[35185]: pgmap v653: 129 pgs: 129 active+clean; 280 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 241 op/s 2024-09-16T00:27:16.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 802409 9.65 MB/sec execute 384 sec latency 64.923 ms 2024-09-16T00:27:16.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:16 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:27:16.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:16 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:27:16.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:16 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:27:17.645 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:27:17.645 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:27:17.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 803996 9.64 MB/sec execute 385 sec latency 70.069 ms 2024-09-16T00:27:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:17 smithi060 ceph-mon[35185]: pgmap v654: 129 pgs: 129 active+clean; 268 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 224 op/s 2024-09-16T00:27:17.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:17 smithi033 ceph-mon[31243]: pgmap v654: 129 pgs: 129 active+clean; 268 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 224 op/s 2024-09-16T00:27:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:17 smithi083 ceph-mon[35242]: pgmap v654: 129 pgs: 129 active+clean; 268 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 224 op/s 2024-09-16T00:27:17.947 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:18.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 806345 9.65 MB/sec execute 386 sec latency 105.497 ms 2024-09-16T00:27:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.170+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/3648667144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d88103730 msgr2=0x7f7d8810bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.170+0000 7f7d8f7ac640 1 --2- 172.21.15.33:0/3648667144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d88103730 0x7f7d8810bc20 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009900 tx=0x7f7d7802a4a0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.170+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/3648667144 shutdown_connections 2024-09-16T00:27:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.170+0000 7f7d8f7ac640 1 --2- 172.21.15.33:0/3648667144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d8810c1b0 0x7f7d8810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.170+0000 7f7d8f7ac640 1 --2- 172.21.15.33:0/3648667144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d88103730 0x7f7d8810bc20 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.170+0000 7f7d8f7ac640 1 --2- 172.21.15.33:0/3648667144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d88102e10 0x7f7d881031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.170+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/3648667144 >> 172.21.15.33:0/3648667144 conn(0x7f7d880fc7d0 msgr2=0x7f7d880febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.170+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/3648667144 shutdown_connections 2024-09-16T00:27:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.171+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/3648667144 wait complete. 2024-09-16T00:27:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8f7ac640 1 Processor -- start 2024-09-16T00:27:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8f7ac640 1 -- start start 2024-09-16T00:27:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8f7ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d88102e10 0x7f7d88100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8f7ac640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d88103730 0x7f7d88100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8f7ac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d8810c1b0 0x7f7d881014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8f7ac640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7d88111fd0 con 0x7f7d88103730 2024-09-16T00:27:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8f7ac640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7d88111e50 con 0x7f7d8810c1b0 2024-09-16T00:27:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8f7ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7d88112150 con 0x7f7d88102e10 2024-09-16T00:27:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8cd47640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d88103730 0x7f7d88100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8cd47640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d88103730 0x7f7d88100fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48474/0 (socket says 172.21.15.33:48474) 2024-09-16T00:27:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8cd47640 1 -- 172.21.15.33:0/1364327378 learned_addr learned my addr 172.21.15.33:0/1364327378 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8d548640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d88102e10 0x7f7d88100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.172+0000 7f7d8dd49640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d8810c1b0 0x7f7d881014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d8cd47640 1 -- 172.21.15.33:0/1364327378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d88102e10 msgr2=0x7f7d88100a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d8cd47640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d88102e10 0x7f7d88100a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d8cd47640 1 -- 172.21.15.33:0/1364327378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d8810c1b0 msgr2=0x7f7d881014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:19.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d8cd47640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d8810c1b0 0x7f7d881014f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d8cd47640 1 -- 172.21.15.33:0/1364327378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d881a68d0 con 0x7f7d88103730 2024-09-16T00:27:19.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d8d548640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d88102e10 0x7f7d88100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:19.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d8dd49640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d8810c1b0 0x7f7d881014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:19.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d8cd47640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d88103730 0x7f7d88100fb0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f7d78004770 tx=0x7f7d78003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:19.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d7e7fc640 1 -- 172.21.15.33:0/1364327378 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7803f070 con 0x7f7d88103730 2024-09-16T00:27:19.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/1364327378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d881a6b60 con 0x7f7d88103730 2024-09-16T00:27:19.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d7e7fc640 1 -- 172.21.15.33:0/1364327378 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d78031c20 con 0x7f7d88103730 2024-09-16T00:27:19.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d7e7fc640 1 -- 172.21.15.33:0/1364327378 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d78035030 con 0x7f7d88103730 2024-09-16T00:27:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.173+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/1364327378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d881a7040 con 0x7f7d88103730 2024-09-16T00:27:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.176+0000 7f7d7e7fc640 1 -- 172.21.15.33:0/1364327378 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7d78042070 con 0x7f7d88103730 2024-09-16T00:27:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.176+0000 7f7d91b85740 1 Processor -- start 2024-09-16T00:27:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.176+0000 7f7d91b85740 1 -- start start 2024-09-16T00:27:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d91b85740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f03b60 0x55c869f03f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d91b85740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 0x55c869f0e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d91b85740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f0ed40 0x55c869f11130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d91b85740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c869e69280 con 0x55c869f04480 2024-09-16T00:27:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d91b85740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c869e10b10 con 0x55c869f03b60 2024-09-16T00:27:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d91b85740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c869e65240 con 0x55c869f0ed40 2024-09-16T00:27:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d8d548640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 0x55c869f0e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d8dd49640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f03b60 0x55c869f03f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d7e7fc640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7d740a3240 0x7f7d740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d8dd49640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f03b60 0x55c869f03f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:37350/0 (socket says 172.21.15.33:37350) 2024-09-16T00:27:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d8dd49640 1 -- 172.21.15.33:0/2458717027 learned_addr learned my addr 172.21.15.33:0/2458717027 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d8cd47640 1 --2- 172.21.15.33:0/2458717027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f0ed40 0x55c869f11130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d8dd49640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7d740a3240 0x7f7d740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d8d548640 1 -- 172.21.15.33:0/2458717027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f0ed40 msgr2=0x55c869f11130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d8d548640 1 --2- 172.21.15.33:0/2458717027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f0ed40 0x55c869f11130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d8d548640 1 -- 172.21.15.33:0/2458717027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f03b60 msgr2=0x55c869f03f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d8d548640 1 --2- 172.21.15.33:0/2458717027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f03b60 0x55c869f03f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d8d548640 1 -- 172.21.15.33:0/2458717027 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c869e1d1f0 con 0x55c869f04480 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d8dd49640 1 --2- 172.21.15.33:0/2458717027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f03b60 0x55c869f03f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.177+0000 7f7d7e7fc640 1 -- 172.21.15.33:0/1364327378 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7d780f0430 con 0x7f7d88103730 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.178+0000 7f7d8d548640 1 --2- 172.21.15.33:0/2458717027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 0x55c869f0e800 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009fd0 tx=0x7f7d840118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dfe04dc67ec06b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.178+0000 7f7d8dd49640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7d740a3240 0x7f7d740a5700 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7d80007400 tx=0x7f7d80004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.178+0000 7f7d737fe640 1 -- 172.21.15.33:0/2458717027 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d84004020 con 0x55c869f04480 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.178+0000 7f7d737fe640 1 -- 172.21.15.33:0/2458717027 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d84005500 con 0x55c869f04480 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.178+0000 7f7d737fe640 1 -- 172.21.15.33:0/2458717027 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d84004e10 con 0x55c869f04480 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.179+0000 7f7d91b85740 1 -- 172.21.15.33:0/2458717027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 msgr2=0x55c869f0e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.179+0000 7f7d91b85740 1 --2- 172.21.15.33:0/2458717027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 0x55c869f0e800 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009fd0 tx=0x7f7d840118d0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.179+0000 7f7d91b85740 1 -- 172.21.15.33:0/2458717027 shutdown_connections 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.179+0000 7f7d91b85740 1 --2- 172.21.15.33:0/2458717027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f0ed40 0x55c869f11130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.179+0000 7f7d91b85740 1 --2- 172.21.15.33:0/2458717027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 0x55c869f0e800 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.179+0000 7f7d91b85740 1 --2- 172.21.15.33:0/2458717027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f03b60 0x55c869f03f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.179+0000 7f7d91b85740 1 -- 172.21.15.33:0/2458717027 >> 172.21.15.33:0/2458717027 conn(0x55c869f000f0 msgr2=0x55c869f004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:19.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.179+0000 7f7d91b85740 1 -- 172.21.15.33:0/2458717027 shutdown_connections 2024-09-16T00:27:19.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.179+0000 7f7d91b85740 1 -- 172.21.15.33:0/2458717027 wait complete. 2024-09-16T00:27:19.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.179+0000 7f7d91b85740 1 -- start start 2024-09-16T00:27:19.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.179+0000 7f7d91b85740 1 Processor -- start 2024-09-16T00:27:19.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d91b85740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f03b60 0x55c869f0e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d91b85740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f04480 0x55c869f0bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d91b85740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f0ed40 0x55c869f0c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d91b85740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c869f11910 con 0x55c869f0ed40 2024-09-16T00:27:19.285 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:27:19.285 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:27:19.285 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "67686f2d-4ec6-4901-843f-12c588d8ea61", 2024-09-16T00:27:19.285 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:27:19.285 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:27:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d91b85740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c869e23f10 con 0x55c869f03b60 2024-09-16T00:27:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d91b85740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c869f11a90 con 0x55c869f04480 2024-09-16T00:27:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d91b85740 10 client.0 _ll_register_callbacks cb 0x55c869deca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d8dd49640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f0ed40 0x55c869f0c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d8dd49640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f0ed40 0x55c869f0c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48500/0 (socket says 172.21.15.33:48500) 2024-09-16T00:27:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d8dd49640 1 -- 172.21.15.33:0/179340560 learned_addr learned my addr 172.21.15.33:0/179340560 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d8cd47640 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f04480 0x55c869f0bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d8d548640 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f03b60 0x55c869f0e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d8dd49640 1 -- 172.21.15.33:0/179340560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f04480 msgr2=0x55c869f0bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d8dd49640 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f04480 0x55c869f0bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d8dd49640 1 -- 172.21.15.33:0/179340560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f03b60 msgr2=0x55c869f0e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d8dd49640 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f03b60 0x55c869f0e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d8dd49640 1 -- 172.21.15.33:0/179340560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c869bda6a0 con 0x55c869f0ed40 2024-09-16T00:27:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d8dd49640 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f0ed40 0x55c869f0c2d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000fd80 tx=0x7f7d80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ba549f192a78db server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.180+0000 7f7d70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:27:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d70ff9640 1 -- 172.21.15.33:0/179340560 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80019070 con 0x55c869f0ed40 2024-09-16T00:27:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d70ff9640 1 -- 172.21.15.33:0/179340560 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7d80004b20 con 0x55c869f0ed40 2024-09-16T00:27:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d91b85740 1 -- 172.21.15.33:0/179340560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d84005500 con 0x55c869f0ed40 2024-09-16T00:27:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d70ff9640 1 -- 172.21.15.33:0/179340560 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7d80011a40 con 0x55c869f0ed40 2024-09-16T00:27:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d91b85740 10 client.17322 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d91b85740 1 -- 172.21.15.33:0/179340560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c869f1d4d0 con 0x55c869f0ed40 2024-09-16T00:27:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d70ff9640 1 -- 172.21.15.33:0/179340560 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d84005500 con 0x55c869f0ed40 2024-09-16T00:27:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d70ff9640 1 -- 172.21.15.33:0/179340560 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f7d80028400 con 0x55c869f0ed40 2024-09-16T00:27:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d91b85740 10 client.17322 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d91b85740 10 client.17322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:27:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d91b85740 20 client.17322 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d91b85740 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c869f1e8e0 0x55c869f20d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d91b85740 4 client.17322 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:27:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.181+0000 7f7d91b85740 1 -- 172.21.15.33:0/179340560 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c869e767f0 con 0x55c869f1e8e0 2024-09-16T00:27:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.182+0000 7f7d8d548640 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c869f1e8e0 0x55c869f20d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.182+0000 7f7d70ff9640 10 client.17322 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.182+0000 7f7d8d548640 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c869f1e8e0 0x55c869f20d50 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b37c9d0ca675b85 server_cookie=1ae47899afc207c1 in_seq=0 out_seq=0 2024-09-16T00:27:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d70ff9640 1 -- 172.21.15.33:0/179340560 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c869e767f0 con 0x55c869f1e8e0 2024-09-16T00:27:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d70ff9640 10 client.17322 handle_command_reply: tid=0 2024-09-16T00:27:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d91b85740 1 client.17322 shutdown 2024-09-16T00:27:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d91b85740 1 -- 172.21.15.33:0/179340560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f0ed40 msgr2=0x55c869f0c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d91b85740 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f0ed40 0x55c869f0c2d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000fd80 tx=0x7f7d80004a40 comp rx=0 tx=0).stop 2024-09-16T00:27:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d91b85740 1 -- 172.21.15.33:0/179340560 shutdown_connections 2024-09-16T00:27:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d91b85740 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c869f1e8e0 0x55c869f20d50 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d91b85740 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f0ed40 0x55c869f0c2d0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d91b85740 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f04480 0x55c869f0bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d91b85740 1 --2- 172.21.15.33:0/179340560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f03b60 0x55c869f0e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d91b85740 1 -- 172.21.15.33:0/179340560 >> 172.21.15.33:0/179340560 conn(0x55c869f000f0 msgr2=0x55c869eff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d70ff9640 0 client.17322 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.183+0000 7f7d91b85740 1 -- 172.21.15.33:0/179340560 shutdown_connections 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.184+0000 7f7d91b85740 1 -- 172.21.15.33:0/179340560 wait complete. 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.184+0000 7f7d91b85740 20 client.17322 trim_cache size 0 max 16384 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.204+0000 7f7d91b85740 1 Processor -- start 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.204+0000 7f7d91b85740 1 -- start start 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d91b85740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f03b60 0x55c869f0a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d91b85740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 0x55c869f0a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d91b85740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f0ed40 0x55c869f26670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d91b85740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881b3b10 con 0x55c869f04480 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d91b85740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881b13e0 con 0x55c869f0ed40 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d91b85740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881af230 con 0x55c869f03b60 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8cd47640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 0x55c869f0a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8d548640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f03b60 0x55c869f0a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8cd47640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 0x55c869f0a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48514/0 (socket says 172.21.15.33:48514) 2024-09-16T00:27:19.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8cd47640 1 -- 172.21.15.33:0/2384194423 learned_addr learned my addr 172.21.15.33:0/2384194423 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:19.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8dd49640 1 --2- 172.21.15.33:0/2384194423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f0ed40 0x55c869f26670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8cd47640 1 -- 172.21.15.33:0/2384194423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f03b60 msgr2=0x55c869f0a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:19.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8cd47640 1 --2- 172.21.15.33:0/2384194423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f03b60 0x55c869f0a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8cd47640 1 -- 172.21.15.33:0/2384194423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f0ed40 msgr2=0x55c869f26670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:19.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8cd47640 1 --2- 172.21.15.33:0/2384194423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f0ed40 0x55c869f26670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8cd47640 1 -- 172.21.15.33:0/2384194423 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c869f042b0 con 0x55c869f04480 2024-09-16T00:27:19.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8dd49640 1 --2- 172.21.15.33:0/2384194423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f0ed40 0x55c869f26670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:27:19.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8d548640 1 --2- 172.21.15.33:0/2384194423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f03b60 0x55c869f0a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:19.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.205+0000 7f7d8cd47640 1 --2- 172.21.15.33:0/2384194423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 0x55c869f0a950 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000c00 tx=0x7f7d7803b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907bad240d5bd827 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:19.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.206+0000 7f7d72ffd640 1 -- 172.21.15.33:0/2384194423 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d78049040 con 0x55c869f04480 2024-09-16T00:27:19.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.206+0000 7f7d72ffd640 1 -- 172.21.15.33:0/2384194423 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d780bc680 con 0x55c869f04480 2024-09-16T00:27:19.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.206+0000 7f7d72ffd640 1 -- 172.21.15.33:0/2384194423 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7803b460 con 0x55c869f04480 2024-09-16T00:27:19.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.207+0000 7f7d91b85740 1 -- 172.21.15.33:0/2384194423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 msgr2=0x55c869f0a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:19.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.207+0000 7f7d91b85740 1 --2- 172.21.15.33:0/2384194423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 0x55c869f0a950 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000c00 tx=0x7f7d7803b790 comp rx=0 tx=0).stop 2024-09-16T00:27:19.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.207+0000 7f7d91b85740 1 -- 172.21.15.33:0/2384194423 shutdown_connections 2024-09-16T00:27:19.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.207+0000 7f7d91b85740 1 --2- 172.21.15.33:0/2384194423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f0ed40 0x55c869f26670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.207+0000 7f7d91b85740 1 --2- 172.21.15.33:0/2384194423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f04480 0x55c869f0a950 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.207+0000 7f7d91b85740 1 --2- 172.21.15.33:0/2384194423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f03b60 0x55c869f0a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.207+0000 7f7d91b85740 1 -- 172.21.15.33:0/2384194423 >> 172.21.15.33:0/2384194423 conn(0x55c869f000f0 msgr2=0x55c869f06de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:19.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.207+0000 7f7d91b85740 1 -- 172.21.15.33:0/2384194423 shutdown_connections 2024-09-16T00:27:19.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.207+0000 7f7d91b85740 1 -- 172.21.15.33:0/2384194423 wait complete. 2024-09-16T00:27:19.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.207+0000 7f7d91b85740 1 -- start start 2024-09-16T00:27:19.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d91b85740 1 Processor -- start 2024-09-16T00:27:19.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d91b85740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f03b60 0x55c869f16d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d91b85740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f04480 0x55c869f17260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d91b85740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f0ed40 0x55c869f11cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d91b85740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881af8a0 con 0x55c869f03b60 2024-09-16T00:27:19.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d91b85740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881b0d10 con 0x55c869f0ed40 2024-09-16T00:27:19.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d91b85740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881aff70 con 0x55c869f04480 2024-09-16T00:27:19.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d91b85740 10 client.0 _ll_register_callbacks cb 0x7f7d8000fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:19.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d8d548640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f03b60 0x55c869f16d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d8d548640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f03b60 0x55c869f16d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48520/0 (socket says 172.21.15.33:48520) 2024-09-16T00:27:19.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d8cd47640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f04480 0x55c869f17260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d8d548640 1 -- 172.21.15.33:0/3554741911 learned_addr learned my addr 172.21.15.33:0/3554741911 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:19.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d8dd49640 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f0ed40 0x55c869f11cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d8cd47640 1 -- 172.21.15.33:0/3554741911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f0ed40 msgr2=0x55c869f11cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:19.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d8cd47640 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f0ed40 0x55c869f11cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d8cd47640 1 -- 172.21.15.33:0/3554741911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f03b60 msgr2=0x55c869f16d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:19.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d8cd47640 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f03b60 0x55c869f16d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d8cd47640 1 -- 172.21.15.33:0/3554741911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c869f17920 con 0x55c869f04480 2024-09-16T00:27:19.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.208+0000 7f7d8d548640 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f03b60 0x55c869f16d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.209+0000 7f7d8cd47640 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f04480 0x55c869f17260 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000c00 tx=0x7f7d7802a4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f37518f94bc0e5d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.209+0000 7f7d70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.209+0000 7f7d70ff9640 1 -- 172.21.15.33:0/3554741911 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d780d4030 con 0x55c869f04480 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.209+0000 7f7d91b85740 1 -- 172.21.15.33:0/3554741911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d780bc680 con 0x55c869f04480 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.209+0000 7f7d70ff9640 1 -- 172.21.15.33:0/3554741911 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7d7803b460 con 0x55c869f04480 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.209+0000 7f7d70ff9640 1 -- 172.21.15.33:0/3554741911 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7d78049070 con 0x55c869f04480 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.209+0000 7f7d91b85740 10 client.26728 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.209+0000 7f7d91b85740 1 -- 172.21.15.33:0/3554741911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c869f89c70 con 0x55c869f04480 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.209+0000 7f7d70ff9640 1 -- 172.21.15.33:0/3554741911 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d780bc680 con 0x55c869f04480 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.209+0000 7f7d70ff9640 1 -- 172.21.15.33:0/3554741911 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f7d780b8cf0 con 0x55c869f04480 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.209+0000 7f7d91b85740 10 client.26728 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.209+0000 7f7d91b85740 10 client.26728 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.210+0000 7f7d91b85740 20 client.26728 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:19.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.210+0000 7f7d91b85740 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c869f8a340 0x55c869f8c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:19.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.210+0000 7f7d91b85740 4 client.26728 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:27:19.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.210+0000 7f7d91b85740 1 -- 172.21.15.33:0/3554741911 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c869e70d00 con 0x55c869f8a340 2024-09-16T00:27:19.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.210+0000 7f7d8d548640 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c869f8a340 0x55c869f8c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:19.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.210+0000 7f7d70ff9640 10 client.26728 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:19.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.210+0000 7f7d8d548640 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c869f8a340 0x55c869f8c760 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a8f5adf28f1106a server_cookie=47757a5d459445c7 in_seq=0 out_seq=0 2024-09-16T00:27:19.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.211+0000 7f7d70ff9640 1 -- 172.21.15.33:0/3554741911 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c869e70d00 con 0x55c869f8a340 2024-09-16T00:27:19.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.211+0000 7f7d70ff9640 10 client.26728 handle_command_reply: tid=0 2024-09-16T00:27:19.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.211+0000 7f7d91b85740 1 client.26728 shutdown 2024-09-16T00:27:19.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.211+0000 7f7d91b85740 1 -- 172.21.15.33:0/3554741911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f04480 msgr2=0x55c869f17260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:19.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.211+0000 7f7d91b85740 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f04480 0x55c869f17260 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000c00 tx=0x7f7d7802a4a0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.211+0000 7f7d91b85740 1 -- 172.21.15.33:0/3554741911 shutdown_connections 2024-09-16T00:27:19.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.211+0000 7f7d91b85740 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c869f8a340 0x55c869f8c760 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.211+0000 7f7d91b85740 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c869f0ed40 0x55c869f11cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d91b85740 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c869f04480 0x55c869f17260 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d91b85740 1 --2- 172.21.15.33:0/3554741911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c869f03b60 0x55c869f16d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d70ff9640 0 client.26728 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:19.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d91b85740 1 -- 172.21.15.33:0/3554741911 >> 172.21.15.33:0/3554741911 conn(0x55c869f000f0 msgr2=0x55c869f09fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:19.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d91b85740 1 -- 172.21.15.33:0/3554741911 shutdown_connections 2024-09-16T00:27:19.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d91b85740 1 -- 172.21.15.33:0/3554741911 wait complete. 2024-09-16T00:27:19.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d91b85740 20 client.26728 trim_cache size 0 max 16384 2024-09-16T00:27:19.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/1364327378 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7d740a3240 msgr2=0x7f7d740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:19.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d8f7ac640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7d740a3240 0x7f7d740a5700 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7d80007400 tx=0x7f7d80004ab0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/1364327378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d88103730 msgr2=0x7f7d88100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:19.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d8f7ac640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d88103730 0x7f7d88100fb0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f7d78004770 tx=0x7f7d78003b90 comp rx=0 tx=0).stop 2024-09-16T00:27:19.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/1364327378 shutdown_connections 2024-09-16T00:27:19.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d8f7ac640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7d740a3240 0x7f7d740a5700 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d8f7ac640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d8810c1b0 0x7f7d881014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d8f7ac640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d88103730 0x7f7d88100fb0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d8f7ac640 1 --2- 172.21.15.33:0/1364327378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d88102e10 0x7f7d88100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:19.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.212+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/1364327378 >> 172.21.15.33:0/1364327378 conn(0x7f7d880fc7d0 msgr2=0x7f7d880fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:19.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.213+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/1364327378 shutdown_connections 2024-09-16T00:27:19.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:19.213+0000 7f7d8f7ac640 1 -- 172.21.15.33:0/1364327378 wait complete. 2024-09-16T00:27:19.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:19 smithi033 ceph-mon[31243]: pgmap v655: 129 pgs: 129 active+clean; 260 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 207 op/s 2024-09-16T00:27:19.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:19 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:19.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:19 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:19.380 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:19 smithi083 ceph-mon[35242]: pgmap v655: 129 pgs: 129 active+clean; 260 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 207 op/s 2024-09-16T00:27:19.380 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:19 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:19.380 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:19 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:19.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:19 smithi060 ceph-mon[35185]: pgmap v655: 129 pgs: 129 active+clean; 260 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 207 op/s 2024-09-16T00:27:19.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:19 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:19.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:19 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:19.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 808868 9.66 MB/sec execute 387 sec latency 54.362 ms 2024-09-16T00:27:20.039 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:27:20.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:20 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:27:20.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:20 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:27:20.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:20 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:27:20.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:20 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:27:20.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:20 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:27:20.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:20 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:27:20.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:20 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:27:20.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:20 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:27:20.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:20 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:27:20.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:20 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:27:20.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:20 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:27:20.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:20 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:27:20.419 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:20.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:20 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:27:20.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:20 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:27:20.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:20 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:27:20.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:20 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:27:20.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:20 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:27:20.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:20 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:27:20.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 810609 9.66 MB/sec execute 388 sec latency 29.718 ms 2024-09-16T00:27:20.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.949+0000 7f6db32aa640 1 -- 172.21.15.33:0/3525736646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac1036e0 msgr2=0x7f6dac107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:20.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.949+0000 7f6db32aa640 1 --2- 172.21.15.33:0/3525736646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac1036e0 0x7f6dac107b90 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009930 tx=0x7f6d9c02a420 comp rx=0 tx=0).stop 2024-09-16T00:27:20.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.950+0000 7f6db32aa640 1 -- 172.21.15.33:0/3525736646 shutdown_connections 2024-09-16T00:27:20.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.950+0000 7f6db32aa640 1 --2- 172.21.15.33:0/3525736646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dac1082a0 0x7f6dac10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.950+0000 7f6db32aa640 1 --2- 172.21.15.33:0/3525736646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac1036e0 0x7f6dac107b90 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.950+0000 7f6db32aa640 1 --2- 172.21.15.33:0/3525736646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dac102d30 0x7f6dac103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.950+0000 7f6db32aa640 1 -- 172.21.15.33:0/3525736646 >> 172.21.15.33:0/3525736646 conn(0x7f6dac0fc7d0 msgr2=0x7f6dac0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:20.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.950+0000 7f6db32aa640 1 -- 172.21.15.33:0/3525736646 shutdown_connections 2024-09-16T00:27:20.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.950+0000 7f6db32aa640 1 -- 172.21.15.33:0/3525736646 wait complete. 2024-09-16T00:27:20.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db32aa640 1 Processor -- start 2024-09-16T00:27:20.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db32aa640 1 -- start start 2024-09-16T00:27:20.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db32aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac102d30 0x7f6dac10ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db32aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dac1036e0 0x7f6dac10bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db32aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dac1082a0 0x7f6dac10c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db32aa640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac1021c0 con 0x7f6dac102d30 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db32aa640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac102040 con 0x7f6dac1036e0 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db32aa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac102340 con 0x7f6dac1082a0 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db1046640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac102d30 0x7f6dac10ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db1046640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac102d30 0x7f6dac10ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56182/0 (socket says 172.21.15.33:56182) 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db1046640 1 -- 172.21.15.33:0/728126627 learned_addr learned my addr 172.21.15.33:0/728126627 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db0845640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dac1036e0 0x7f6dac10bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db1847640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dac1082a0 0x7f6dac10c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db1046640 1 -- 172.21.15.33:0/728126627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dac1082a0 msgr2=0x7f6dac10c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db1046640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dac1082a0 0x7f6dac10c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db1046640 1 -- 172.21.15.33:0/728126627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dac1036e0 msgr2=0x7f6dac10bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db1046640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dac1036e0 0x7f6dac10bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.951+0000 7f6db1046640 1 -- 172.21.15.33:0/728126627 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dac10c910 con 0x7f6dac102d30 2024-09-16T00:27:20.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.952+0000 7f6db0845640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dac1036e0 0x7f6dac10bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:20.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.952+0000 7f6db1046640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac102d30 0x7f6dac10ea60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f6da800d950 tx=0x7f6da800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:20.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.952+0000 7f6da27fc640 1 -- 172.21.15.33:0/728126627 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8017070 con 0x7f6dac102d30 2024-09-16T00:27:20.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.952+0000 7f6da27fc640 1 -- 172.21.15.33:0/728126627 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da8009c30 con 0x7f6dac102d30 2024-09-16T00:27:20.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.952+0000 7f6da27fc640 1 -- 172.21.15.33:0/728126627 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da800f6b0 con 0x7f6dac102d30 2024-09-16T00:27:20.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.952+0000 7f6db32aa640 1 -- 172.21.15.33:0/728126627 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dac1ab1f0 con 0x7f6dac102d30 2024-09-16T00:27:20.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.952+0000 7f6db32aa640 1 -- 172.21.15.33:0/728126627 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dac1ab700 con 0x7f6dac102d30 2024-09-16T00:27:20.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.956+0000 7f6db5683740 1 Processor -- start 2024-09-16T00:27:20.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.956+0000 7f6db5683740 1 -- start start 2024-09-16T00:27:20.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.957+0000 7f6db5683740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170aded4a0 0x56170aded880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.957+0000 7f6db5683740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.957+0000 7f6db5683740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adf8680 0x56170adfaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.957+0000 7f6da27fc640 1 -- 172.21.15.33:0/728126627 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6da80109d0 con 0x7f6dac102d30 2024-09-16T00:27:20.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.957+0000 7f6db5683740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56170ad532e0 con 0x56170adf8680 2024-09-16T00:27:20.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.957+0000 7f6db5683740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56170acface0 con 0x56170adeddc0 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.957+0000 7f6db5683740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56170ad4f2a0 con 0x56170aded4a0 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.957+0000 7f6db1847640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170aded4a0 0x56170aded880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.957+0000 7f6db1847640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170aded4a0 0x56170aded880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:43482/0 (socket says 172.21.15.33:43482) 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.957+0000 7f6db1847640 1 -- 172.21.15.33:0/2696676352 learned_addr learned my addr 172.21.15.33:0/2696676352 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.957+0000 7f6db0845640 1 --2- 172.21.15.33:0/2696676352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adf8680 0x56170adfaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.957+0000 7f6db1046640 1 --2- 172.21.15.33:0/2696676352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.958+0000 7f6db1847640 1 -- 172.21.15.33:0/2696676352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 msgr2=0x56170adf8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.958+0000 7f6db1847640 1 --2- 172.21.15.33:0/2696676352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.958+0000 7f6db1847640 1 -- 172.21.15.33:0/2696676352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adf8680 msgr2=0x56170adfaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.958+0000 7f6db1847640 1 --2- 172.21.15.33:0/2696676352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adf8680 0x56170adfaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.958+0000 7f6db1847640 1 -- 172.21.15.33:0/2696676352 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56170acdf440 con 0x56170aded4a0 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.958+0000 7f6db1046640 1 --2- 172.21.15.33:0/2696676352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:27:20.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.958+0000 7f6db1847640 1 --2- 172.21.15.33:0/2696676352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170aded4a0 0x56170aded880 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009f30 tx=0x7f6da40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b87ff1fc4dd2f2ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:20.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.958+0000 7f6da27fc640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6d980a3390 0x7f6d980a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.959+0000 7f6d977fe640 1 -- 172.21.15.33:0/2696676352 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4004d20 con 0x56170aded4a0 2024-09-16T00:27:20.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.959+0000 7f6d977fe640 1 -- 172.21.15.33:0/2696676352 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da4004050 con 0x56170aded4a0 2024-09-16T00:27:20.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.959+0000 7f6d977fe640 1 -- 172.21.15.33:0/2696676352 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4005260 con 0x56170aded4a0 2024-09-16T00:27:20.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.959+0000 7f6da27fc640 1 -- 172.21.15.33:0/728126627 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6da80cf840 con 0x7f6dac102d30 2024-09-16T00:27:20.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.959+0000 7f6db1046640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6d980a3390 0x7f6d980a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.959+0000 7f6db5683740 1 -- 172.21.15.33:0/2696676352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170aded4a0 msgr2=0x56170aded880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:20.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.959+0000 7f6db5683740 1 --2- 172.21.15.33:0/2696676352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170aded4a0 0x56170aded880 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009f30 tx=0x7f6da40118d0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.959+0000 7f6db5683740 1 -- 172.21.15.33:0/2696676352 shutdown_connections 2024-09-16T00:27:20.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.959+0000 7f6db5683740 1 --2- 172.21.15.33:0/2696676352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adf8680 0x56170adfaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.959+0000 7f6db5683740 1 --2- 172.21.15.33:0/2696676352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.959+0000 7f6db5683740 1 --2- 172.21.15.33:0/2696676352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170aded4a0 0x56170aded880 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.959+0000 7f6db5683740 1 -- 172.21.15.33:0/2696676352 >> 172.21.15.33:0/2696676352 conn(0x56170ade9a30 msgr2=0x56170ade9e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:20.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.960+0000 7f6db5683740 1 -- 172.21.15.33:0/2696676352 shutdown_connections 2024-09-16T00:27:20.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.960+0000 7f6db1046640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6d980a3390 0x7f6d980a5850 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6da80cd230 tx=0x7f6da8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:20.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.960+0000 7f6db5683740 1 -- 172.21.15.33:0/2696676352 wait complete. 2024-09-16T00:27:20.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.960+0000 7f6db5683740 1 -- start start 2024-09-16T00:27:20.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.960+0000 7f6db5683740 1 Processor -- start 2024-09-16T00:27:20.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.960+0000 7f6db5683740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170aded4a0 0x56170adf7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.960+0000 7f6db5683740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.960+0000 7f6db5683740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170adf8680 0x56170adf5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.960+0000 7f6db5683740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56170adfb250 con 0x56170aded4a0 2024-09-16T00:27:20.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db5683740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56170ad0e050 con 0x56170adeddc0 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db5683740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56170adfb3d0 con 0x56170adf8680 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db1847640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170aded4a0 0x56170adf7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db5683740 10 client.0 _ll_register_callbacks cb 0x56170ad4e160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db1847640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170aded4a0 0x56170adf7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56194/0 (socket says 172.21.15.33:56194) 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db0845640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db1847640 1 -- 172.21.15.33:0/3846197108 learned_addr learned my addr 172.21.15.33:0/3846197108 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db1847640 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170adf8680 0x56170adf5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db1847640 1 -- 172.21.15.33:0/3846197108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170adf8680 msgr2=0x56170adf5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db1847640 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170adf8680 0x56170adf5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db1847640 1 -- 172.21.15.33:0/3846197108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 msgr2=0x56170adf5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db1847640 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db1847640 1 -- 172.21.15.33:0/3846197108 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56170aac43c0 con 0x56170aded4a0 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db0845640 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6db1847640 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170aded4a0 0x56170adf7f60 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f6da4004950 tx=0x7f6da4003cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef488da34bae442 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.961+0000 7f6d94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:27:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.962+0000 7f6d94ff9640 1 -- 172.21.15.33:0/3846197108 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da402c030 con 0x56170aded4a0 2024-09-16T00:27:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.962+0000 7f6d94ff9640 1 -- 172.21.15.33:0/3846197108 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6da4004c50 con 0x56170aded4a0 2024-09-16T00:27:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.962+0000 7f6db5683740 1 -- 172.21.15.33:0/3846197108 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6da4004050 con 0x56170aded4a0 2024-09-16T00:27:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.962+0000 7f6d94ff9640 1 -- 172.21.15.33:0/3846197108 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6da4036050 con 0x56170aded4a0 2024-09-16T00:27:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.962+0000 7f6db5683740 10 client.17340 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.962+0000 7f6db5683740 1 -- 172.21.15.33:0/3846197108 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56170ae06b30 con 0x56170aded4a0 2024-09-16T00:27:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.962+0000 7f6d94ff9640 1 -- 172.21.15.33:0/3846197108 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da4003d60 con 0x56170aded4a0 2024-09-16T00:27:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.962+0000 7f6d94ff9640 1 -- 172.21.15.33:0/3846197108 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f6da4028670 con 0x56170aded4a0 2024-09-16T00:27:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.963+0000 7f6db5683740 10 client.17340 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.963+0000 7f6db5683740 10 client.17340 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:27:20.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.963+0000 7f6db5683740 20 client.17340 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:20.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.963+0000 7f6db5683740 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56170ae08370 0x56170ae0a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.963+0000 7f6db5683740 4 client.17340 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:27:20.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.963+0000 7f6db0845640 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56170ae08370 0x56170ae0a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.963+0000 7f6db5683740 1 -- 172.21.15.33:0/3846197108 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6da4004050 con 0x56170ae08370 2024-09-16T00:27:20.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.963+0000 7f6d94ff9640 10 client.17340 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:20.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.963+0000 7f6db0845640 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56170ae08370 0x56170ae0a780 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4442113409e7e1a5 server_cookie=d0b434bc0b16d8e9 in_seq=0 out_seq=0 2024-09-16T00:27:20.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.964+0000 7f6d94ff9640 1 -- 172.21.15.33:0/3846197108 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f6da4004050 con 0x56170ae08370 2024-09-16T00:27:20.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.964+0000 7f6d94ff9640 10 client.17340 handle_command_reply: tid=0 2024-09-16T00:27:20.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.964+0000 7f6db5683740 1 client.17340 shutdown 2024-09-16T00:27:20.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.964+0000 7f6db5683740 1 -- 172.21.15.33:0/3846197108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170aded4a0 msgr2=0x56170adf7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:20.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.964+0000 7f6db5683740 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170aded4a0 0x56170adf7f60 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f6da4004950 tx=0x7f6da4003cf0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.964+0000 7f6db5683740 1 -- 172.21.15.33:0/3846197108 shutdown_connections 2024-09-16T00:27:20.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.964+0000 7f6db5683740 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56170ae08370 0x56170ae0a780 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.964+0000 7f6db5683740 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170adf8680 0x56170adf5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.964+0000 7f6db5683740 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.964+0000 7f6db5683740 1 --2- 172.21.15.33:0/3846197108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170aded4a0 0x56170adf7f60 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.964+0000 7f6db5683740 1 -- 172.21.15.33:0/3846197108 >> 172.21.15.33:0/3846197108 conn(0x56170ade9a30 msgr2=0x56170adef710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:20.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.964+0000 7f6d94ff9640 0 client.17340 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:20.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.965+0000 7f6db5683740 1 -- 172.21.15.33:0/3846197108 shutdown_connections 2024-09-16T00:27:20.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.965+0000 7f6db5683740 1 -- 172.21.15.33:0/3846197108 wait complete. 2024-09-16T00:27:20.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.965+0000 7f6db5683740 20 client.17340 trim_cache size 0 max 16384 2024-09-16T00:27:20.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.982+0000 7f6db5683740 1 Processor -- start 2024-09-16T00:27:20.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db5683740 1 -- start start 2024-09-16T00:27:20.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db5683740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170aded4a0 0x56170ade8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db5683740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db5683740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adf8680 0x56170adf29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db5683740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac1b8060 con 0x56170adf8680 2024-09-16T00:27:20.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db5683740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac1b5930 con 0x56170adeddc0 2024-09-16T00:27:20.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db5683740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac1b3780 con 0x56170aded4a0 2024-09-16T00:27:20.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db1847640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adf8680 0x56170adf29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db1847640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adf8680 0x56170adf29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56204/0 (socket says 172.21.15.33:56204) 2024-09-16T00:27:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db1847640 1 -- 172.21.15.33:0/3611777971 learned_addr learned my addr 172.21.15.33:0/3611777971 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db0845640 1 --2- 172.21.15.33:0/3611777971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db1847640 1 --2- 172.21.15.33:0/3611777971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170aded4a0 0x56170ade8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db1847640 1 -- 172.21.15.33:0/3611777971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170aded4a0 msgr2=0x56170ade8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db1847640 1 --2- 172.21.15.33:0/3611777971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170aded4a0 0x56170ade8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db1847640 1 -- 172.21.15.33:0/3611777971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 msgr2=0x56170adf2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db1847640 1 --2- 172.21.15.33:0/3611777971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.983+0000 7f6db1847640 1 -- 172.21.15.33:0/3611777971 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56170adfadb0 con 0x56170adf8680 2024-09-16T00:27:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.984+0000 7f6db1847640 1 --2- 172.21.15.33:0/3611777971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adf8680 0x56170adf29b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009560 tx=0x7f6da4009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1c52dcaa987e53 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.984+0000 7f6d96ffd640 1 -- 172.21.15.33:0/3611777971 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da402c040 con 0x56170adf8680 2024-09-16T00:27:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.984+0000 7f6d96ffd640 1 -- 172.21.15.33:0/3611777971 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da4004e50 con 0x56170adf8680 2024-09-16T00:27:20.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.984+0000 7f6d96ffd640 1 -- 172.21.15.33:0/3611777971 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4027740 con 0x56170adf8680 2024-09-16T00:27:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.985+0000 7f6db5683740 1 -- 172.21.15.33:0/3611777971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adf8680 msgr2=0x56170adf29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.985+0000 7f6db5683740 1 --2- 172.21.15.33:0/3611777971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adf8680 0x56170adf29b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009560 tx=0x7f6da4009fa0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.985+0000 7f6db5683740 1 -- 172.21.15.33:0/3611777971 shutdown_connections 2024-09-16T00:27:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.985+0000 7f6db5683740 1 --2- 172.21.15.33:0/3611777971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adf8680 0x56170adf29b0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.985+0000 7f6db5683740 1 --2- 172.21.15.33:0/3611777971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170adeddc0 0x56170adf2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.985+0000 7f6db5683740 1 --2- 172.21.15.33:0/3611777971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170aded4a0 0x56170ade8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.985+0000 7f6db5683740 1 -- 172.21.15.33:0/3611777971 >> 172.21.15.33:0/3611777971 conn(0x56170ade9a30 msgr2=0x56170adf52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.985+0000 7f6db5683740 1 -- 172.21.15.33:0/3611777971 shutdown_connections 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.985+0000 7f6db5683740 1 -- 172.21.15.33:0/3611777971 wait complete. 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.985+0000 7f6db5683740 1 -- start start 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.985+0000 7f6db5683740 1 Processor -- start 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db5683740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170aded4a0 0x56170ae0f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db5683740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adeddc0 0x56170ae0f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db5683740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170adf8680 0x56170ae70f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db5683740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac1b3df0 con 0x56170adeddc0 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db5683740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac1b5260 con 0x56170aded4a0 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db5683740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac1b44c0 con 0x56170adf8680 2024-09-16T00:27:20.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db0845640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adeddc0 0x56170ae0f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db0845640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adeddc0 0x56170ae0f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56212/0 (socket says 172.21.15.33:56212) 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db0845640 1 -- 172.21.15.33:0/4167721263 learned_addr learned my addr 172.21.15.33:0/4167721263 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db5683740 10 client.0 _ll_register_callbacks cb 0x7f6d9c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db1847640 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170aded4a0 0x56170ae0f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db1847640 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170adf8680 0x56170ae70f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db0845640 1 -- 172.21.15.33:0/4167721263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170adf8680 msgr2=0x56170ae70f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db0845640 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170adf8680 0x56170ae70f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db0845640 1 -- 172.21.15.33:0/4167721263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170aded4a0 msgr2=0x56170ae0f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db0845640 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170aded4a0 0x56170ae0f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db0845640 1 -- 172.21.15.33:0/4167721263 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56170ae71460 con 0x56170adeddc0 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.986+0000 7f6db1847640 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170adf8680 0x56170ae70f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.987+0000 7f6db0845640 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adeddc0 0x56170ae0f630 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c005940 tx=0x7f6d9c004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27137d5244c3db15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.987+0000 7f6d94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.987+0000 7f6d94ff9640 1 -- 172.21.15.33:0/4167721263 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c035040 con 0x56170adeddc0 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.987+0000 7f6d94ff9640 1 -- 172.21.15.33:0/4167721263 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6d9c005240 con 0x56170adeddc0 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.987+0000 7f6db5683740 1 -- 172.21.15.33:0/4167721263 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6da4004e50 con 0x56170adeddc0 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.987+0000 7f6d94ff9640 1 -- 172.21.15.33:0/4167721263 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6d9c054050 con 0x56170adeddc0 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.987+0000 7f6db5683740 10 client.17346 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.987+0000 7f6db5683740 1 -- 172.21.15.33:0/4167721263 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56170ae726c0 con 0x56170adeddc0 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.987+0000 7f6d94ff9640 1 -- 172.21.15.33:0/4167721263 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da4004e50 con 0x56170adeddc0 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.988+0000 7f6d94ff9640 1 -- 172.21.15.33:0/4167721263 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f6d9c004800 con 0x56170adeddc0 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.988+0000 7f6db5683740 10 client.17346 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.988+0000 7f6db5683740 10 client.17346 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.988+0000 7f6db5683740 20 client.17346 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.988+0000 7f6db5683740 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56170ae72e40 0x56170ae75260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:20.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.988+0000 7f6db5683740 4 client.17346 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:27:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.988+0000 7f6db5683740 1 -- 172.21.15.33:0/4167721263 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56170ad60850 con 0x56170ae72e40 2024-09-16T00:27:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.988+0000 7f6db1847640 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56170ae72e40 0x56170ae75260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.988+0000 7f6db1847640 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56170ae72e40 0x56170ae75260 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f128850371cb0704 server_cookie=6fdc069eecff01f2 in_seq=0 out_seq=0 2024-09-16T00:27:20.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.988+0000 7f6d94ff9640 10 client.17346 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6d94ff9640 1 -- 172.21.15.33:0/4167721263 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56170ad60850 con 0x56170ae72e40 2024-09-16T00:27:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6d94ff9640 10 client.17346 handle_command_reply: tid=0 2024-09-16T00:27:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6db5683740 1 client.17346 shutdown 2024-09-16T00:27:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6db5683740 1 -- 172.21.15.33:0/4167721263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adeddc0 msgr2=0x56170ae0f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6db5683740 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adeddc0 0x56170ae0f630 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c005940 tx=0x7f6d9c004230 comp rx=0 tx=0).stop 2024-09-16T00:27:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6db5683740 1 -- 172.21.15.33:0/4167721263 shutdown_connections 2024-09-16T00:27:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6db5683740 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56170ae72e40 0x56170ae75260 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6db5683740 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170adf8680 0x56170ae70f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6db5683740 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56170adeddc0 0x56170ae0f630 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6db5683740 1 --2- 172.21.15.33:0/4167721263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56170aded4a0 0x56170ae0f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6db5683740 1 -- 172.21.15.33:0/4167721263 >> 172.21.15.33:0/4167721263 conn(0x56170ade9a30 msgr2=0x56170adf52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6d94ff9640 0 client.17346 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6db5683740 1 -- 172.21.15.33:0/4167721263 shutdown_connections 2024-09-16T00:27:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6db5683740 1 -- 172.21.15.33:0/4167721263 wait complete. 2024-09-16T00:27:20.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.989+0000 7f6db5683740 20 client.17346 trim_cache size 0 max 16384 2024-09-16T00:27:20.995 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:27:20.995 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:27:20.995 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:27:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: "67686f2d-4ec6-4901-843f-12c588d8ea61": { 2024-09-16T00:27:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:27:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "67686f2d-4ec6-4901-843f-12c588d8ea61", 2024-09-16T00:27:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:27:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:27:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:27:20.996 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:27:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.990+0000 7f6db32aa640 1 -- 172.21.15.33:0/728126627 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6d980a3390 msgr2=0x7f6d980a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.990+0000 7f6db32aa640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6d980a3390 0x7f6d980a5850 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6da80cd230 tx=0x7f6da8000bc0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.990+0000 7f6db32aa640 1 -- 172.21.15.33:0/728126627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac102d30 msgr2=0x7f6dac10ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.990+0000 7f6db32aa640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac102d30 0x7f6dac10ea60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f6da800d950 tx=0x7f6da800de10 comp rx=0 tx=0).stop 2024-09-16T00:27:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.990+0000 7f6db32aa640 1 -- 172.21.15.33:0/728126627 shutdown_connections 2024-09-16T00:27:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.990+0000 7f6db32aa640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6d980a3390 0x7f6d980a5850 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.990+0000 7f6db32aa640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dac1082a0 0x7f6dac10c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.990+0000 7f6db32aa640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dac1036e0 0x7f6dac10bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.990+0000 7f6db32aa640 1 --2- 172.21.15.33:0/728126627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac102d30 0x7f6dac10ea60 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.990+0000 7f6db32aa640 1 -- 172.21.15.33:0/728126627 >> 172.21.15.33:0/728126627 conn(0x7f6dac0fc7d0 msgr2=0x7f6dac0fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.990+0000 7f6db32aa640 1 -- 172.21.15.33:0/728126627 shutdown_connections 2024-09-16T00:27:20.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:20.990+0000 7f6db32aa640 1 -- 172.21.15.33:0/728126627 wait complete. 2024-09-16T00:27:21.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:21 smithi083 ceph-mon[35242]: pgmap v656: 129 pgs: 129 active+clean; 265 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 258 op/s 2024-09-16T00:27:21.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:21 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:21 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:21 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:21 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:21 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:27:21.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:21 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:27:21.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:21 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:21 smithi060 ceph-mon[35185]: pgmap v656: 129 pgs: 129 active+clean; 265 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 258 op/s 2024-09-16T00:27:21.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:21 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:21 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:21 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:21 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:21 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:27:21.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:21 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:27:21.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:21 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:21 smithi033 ceph-mon[31243]: pgmap v656: 129 pgs: 129 active+clean; 265 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 258 op/s 2024-09-16T00:27:21.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:21 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:21 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:21 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:21 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:21 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:27:21.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:21 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:27:21.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:21 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:27:21.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 813509 9.68 MB/sec execute 389 sec latency 15.229 ms 2024-09-16T00:27:21.715 INFO:tasks.cephfs.filesystem:scrub status for tag:67686f2d-4ec6-4901-843f-12c588d8ea61 - {'path': '/', 'tag': '67686f2d-4ec6-4901-843f-12c588d8ea61', 'options': 'recursive,force'} 2024-09-16T00:27:22.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 816583 9.70 MB/sec execute 390 sec latency 19.674 ms 2024-09-16T00:27:23.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:23 smithi083 ceph-mon[35242]: pgmap v657: 129 pgs: 129 active+clean; 265 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 220 op/s 2024-09-16T00:27:23.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:23 smithi060 ceph-mon[35185]: pgmap v657: 129 pgs: 129 active+clean; 265 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 220 op/s 2024-09-16T00:27:23.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:23 smithi033 ceph-mon[31243]: pgmap v657: 129 pgs: 129 active+clean; 265 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 220 op/s 2024-09-16T00:27:23.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 819023 9.69 MB/sec execute 391 sec latency 17.185 ms 2024-09-16T00:27:24.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 821125 9.70 MB/sec execute 392 sec latency 19.137 ms 2024-09-16T00:27:25.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:25 smithi083 ceph-mon[35242]: pgmap v658: 129 pgs: 129 active+clean; 269 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 268 op/s 2024-09-16T00:27:25.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:25 smithi060 ceph-mon[35185]: pgmap v658: 129 pgs: 129 active+clean; 269 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 268 op/s 2024-09-16T00:27:25.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:25 smithi033 ceph-mon[31243]: pgmap v658: 129 pgs: 129 active+clean; 269 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 268 op/s 2024-09-16T00:27:25.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 824130 9.72 MB/sec execute 393 sec latency 84.895 ms 2024-09-16T00:27:26.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 827317 9.74 MB/sec execute 394 sec latency 17.644 ms 2024-09-16T00:27:27.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:27 smithi033 ceph-mon[31243]: pgmap v659: 129 pgs: 129 active+clean; 273 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.5 MiB/s wr, 208 op/s 2024-09-16T00:27:27.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:27 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:27:27.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:27 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:27:27.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:27 smithi083 ceph-mon[35242]: pgmap v659: 129 pgs: 129 active+clean; 273 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.5 MiB/s wr, 208 op/s 2024-09-16T00:27:27.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:27 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:27:27.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:27 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:27:27.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 829835 9.74 MB/sec execute 395 sec latency 13.958 ms 2024-09-16T00:27:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:27 smithi060 ceph-mon[35185]: pgmap v659: 129 pgs: 129 active+clean; 273 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.5 MiB/s wr, 208 op/s 2024-09-16T00:27:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:27 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:27:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:27 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:27:28.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 831901 9.75 MB/sec execute 396 sec latency 23.633 ms 2024-09-16T00:27:29.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:29 smithi083 ceph-mon[35242]: pgmap v660: 129 pgs: 129 active+clean; 273 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-09-16T00:27:29.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:29 smithi033 ceph-mon[31243]: pgmap v660: 129 pgs: 129 active+clean; 273 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-09-16T00:27:29.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 834910 9.77 MB/sec execute 397 sec latency 7.247 ms 2024-09-16T00:27:29.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:29 smithi060 ceph-mon[35185]: pgmap v660: 129 pgs: 129 active+clean; 273 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-09-16T00:27:30.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 838021 9.79 MB/sec execute 398 sec latency 18.222 ms 2024-09-16T00:27:31.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:31 smithi033 ceph-mon[31243]: pgmap v661: 129 pgs: 129 active+clean; 274 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.5 MiB/s wr, 213 op/s 2024-09-16T00:27:31.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:31 smithi083 ceph-mon[35242]: pgmap v661: 129 pgs: 129 active+clean; 274 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.5 MiB/s wr, 213 op/s 2024-09-16T00:27:31.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 840357 9.79 MB/sec execute 399 sec latency 19.486 ms 2024-09-16T00:27:31.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:31 smithi060 ceph-mon[35185]: pgmap v661: 129 pgs: 129 active+clean; 274 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.5 MiB/s wr, 213 op/s 2024-09-16T00:27:32.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 842487 9.80 MB/sec execute 400 sec latency 12.629 ms 2024-09-16T00:27:33.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:33 smithi033 ceph-mon[31243]: pgmap v662: 129 pgs: 129 active+clean; 277 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.4 MiB/s wr, 191 op/s 2024-09-16T00:27:33.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:33 smithi083 ceph-mon[35242]: pgmap v662: 129 pgs: 129 active+clean; 277 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.4 MiB/s wr, 191 op/s 2024-09-16T00:27:33.676 INFO:tasks.workunit.client.0.smithi033.stdout: 1 845640 9.82 MB/sec execute 401 sec latency 15.262 ms 2024-09-16T00:27:33.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:33 smithi060 ceph-mon[35185]: pgmap v662: 129 pgs: 129 active+clean; 277 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.4 MiB/s wr, 191 op/s 2024-09-16T00:27:34.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 848874 9.84 MB/sec execute 402 sec latency 11.639 ms 2024-09-16T00:27:35.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:35 smithi033 ceph-mon[31243]: pgmap v663: 129 pgs: 129 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 1.8 MiB/s wr, 252 op/s 2024-09-16T00:27:35.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:35 smithi083 ceph-mon[35242]: pgmap v663: 129 pgs: 129 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 1.8 MiB/s wr, 252 op/s 2024-09-16T00:27:35.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 850941 9.83 MB/sec execute 403 sec latency 31.736 ms 2024-09-16T00:27:35.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:35 smithi060 ceph-mon[35185]: pgmap v663: 129 pgs: 129 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 1.8 MiB/s wr, 252 op/s 2024-09-16T00:27:36.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 853037 9.84 MB/sec execute 404 sec latency 15.800 ms 2024-09-16T00:27:37.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:37 smithi033 ceph-mon[31243]: pgmap v664: 129 pgs: 129 active+clean; 288 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 229 op/s 2024-09-16T00:27:37.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:37 smithi083 ceph-mon[35242]: pgmap v664: 129 pgs: 129 active+clean; 288 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 229 op/s 2024-09-16T00:27:37.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 856032 9.86 MB/sec execute 405 sec latency 11.666 ms 2024-09-16T00:27:37.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:37 smithi060 ceph-mon[35185]: pgmap v664: 129 pgs: 129 active+clean; 288 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 229 op/s 2024-09-16T00:27:38.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 859177 9.88 MB/sec execute 406 sec latency 14.945 ms 2024-09-16T00:27:39.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:39 smithi033 ceph-mon[31243]: pgmap v665: 129 pgs: 129 active+clean; 288 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-09-16T00:27:39.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:39 smithi083 ceph-mon[35242]: pgmap v665: 129 pgs: 129 active+clean; 288 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-09-16T00:27:39.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 861657 9.88 MB/sec execute 407 sec latency 12.166 ms 2024-09-16T00:27:39.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:39 smithi060 ceph-mon[35185]: pgmap v665: 129 pgs: 129 active+clean; 288 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-09-16T00:27:40.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 863364 9.88 MB/sec execute 408 sec latency 93.349 ms 2024-09-16T00:27:41.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:41 smithi033 ceph-mon[31243]: pgmap v666: 129 pgs: 129 active+clean; 288 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 208 op/s 2024-09-16T00:27:41.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:41 smithi083 ceph-mon[35242]: pgmap v666: 129 pgs: 129 active+clean; 288 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 208 op/s 2024-09-16T00:27:41.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 866585 9.90 MB/sec execute 409 sec latency 20.697 ms 2024-09-16T00:27:41.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:41 smithi060 ceph-mon[35185]: pgmap v666: 129 pgs: 129 active+clean; 288 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 208 op/s 2024-09-16T00:27:42.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 869489 9.91 MB/sec execute 410 sec latency 14.197 ms 2024-09-16T00:27:43.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:43 smithi033 ceph-mon[31243]: pgmap v667: 129 pgs: 129 active+clean; 289 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 237 op/s 2024-09-16T00:27:43.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:43 smithi083 ceph-mon[35242]: pgmap v667: 129 pgs: 129 active+clean; 289 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 237 op/s 2024-09-16T00:27:43.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 871682 9.92 MB/sec execute 411 sec latency 20.870 ms 2024-09-16T00:27:43.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:43 smithi060 ceph-mon[35185]: pgmap v667: 129 pgs: 129 active+clean; 289 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 237 op/s 2024-09-16T00:27:44.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 874271 9.93 MB/sec execute 412 sec latency 15.246 ms 2024-09-16T00:27:45.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:45 smithi033 ceph-mon[31243]: pgmap v668: 129 pgs: 129 active+clean; 293 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.1 MiB/s wr, 283 op/s 2024-09-16T00:27:45.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:45 smithi083 ceph-mon[35242]: pgmap v668: 129 pgs: 129 active+clean; 293 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.1 MiB/s wr, 283 op/s 2024-09-16T00:27:45.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 876936 9.94 MB/sec execute 413 sec latency 86.130 ms 2024-09-16T00:27:45.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:45 smithi060 ceph-mon[35185]: pgmap v668: 129 pgs: 129 active+clean; 293 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.1 MiB/s wr, 283 op/s 2024-09-16T00:27:46.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 879368 9.94 MB/sec execute 414 sec latency 19.710 ms 2024-09-16T00:27:47.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:47 smithi033 ceph-mon[31243]: pgmap v669: 129 pgs: 129 active+clean; 295 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.6 MiB/s wr, 272 op/s 2024-09-16T00:27:47.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:47 smithi083 ceph-mon[35242]: pgmap v669: 129 pgs: 129 active+clean; 295 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.6 MiB/s wr, 272 op/s 2024-09-16T00:27:47.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 881655 9.95 MB/sec execute 415 sec latency 19.651 ms 2024-09-16T00:27:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:47 smithi060 ceph-mon[35185]: pgmap v669: 129 pgs: 129 active+clean; 295 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.6 MiB/s wr, 272 op/s 2024-09-16T00:27:48.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 884609 9.97 MB/sec execute 416 sec latency 16.574 ms 2024-09-16T00:27:49.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:49 smithi033 ceph-mon[31243]: pgmap v670: 129 pgs: 129 active+clean; 288 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 237 op/s 2024-09-16T00:27:49.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:49 smithi083 ceph-mon[35242]: pgmap v670: 129 pgs: 129 active+clean; 288 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 237 op/s 2024-09-16T00:27:49.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 887722 9.99 MB/sec execute 417 sec latency 15.928 ms 2024-09-16T00:27:49.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:49 smithi060 ceph-mon[35185]: pgmap v670: 129 pgs: 129 active+clean; 288 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 237 op/s 2024-09-16T00:27:50.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 889189 9.98 MB/sec execute 418 sec latency 49.526 ms 2024-09-16T00:27:51.532 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:51 smithi060 ceph-mon[35185]: pgmap v671: 129 pgs: 129 active+clean; 287 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 256 op/s 2024-09-16T00:27:51.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:51 smithi033 ceph-mon[31243]: pgmap v671: 129 pgs: 129 active+clean; 287 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 256 op/s 2024-09-16T00:27:51.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:51 smithi083 ceph-mon[35242]: pgmap v671: 129 pgs: 129 active+clean; 287 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 256 op/s 2024-09-16T00:27:51.677 INFO:tasks.workunit.client.0.smithi033.stdout: 1 891890 9.99 MB/sec execute 419 sec latency 16.990 ms 2024-09-16T00:27:51.716 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:27:51.986 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:52.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.575+0000 7fe290b1c640 1 -- 172.21.15.33:0/3399607194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe28c1055b0 msgr2=0x7fe28c10d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:52.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.575+0000 7fe290b1c640 1 --2- 172.21.15.33:0/3399607194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe28c1055b0 0x7fe28c10d130 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe28400adb0 tx=0x7fe28402b7a0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.575+0000 7fe290b1c640 1 -- 172.21.15.33:0/3399607194 shutdown_connections 2024-09-16T00:27:52.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.575+0000 7fe290b1c640 1 --2- 172.21.15.33:0/3399607194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe28c1055b0 0x7fe28c10d130 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.575+0000 7fe290b1c640 1 --2- 172.21.15.33:0/3399607194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28c100e30 0x7fe28c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.575+0000 7fe290b1c640 1 --2- 172.21.15.33:0/3399607194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe28c100480 0x7fe28c100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.575+0000 7fe290b1c640 1 -- 172.21.15.33:0/3399607194 >> 172.21.15.33:0/3399607194 conn(0x7fe28c077210 msgr2=0x7fe28c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:52.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.575+0000 7fe290b1c640 1 -- 172.21.15.33:0/3399607194 shutdown_connections 2024-09-16T00:27:52.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.575+0000 7fe290b1c640 1 -- 172.21.15.33:0/3399607194 wait complete. 2024-09-16T00:27:52.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.575+0000 7fe290b1c640 1 Processor -- start 2024-09-16T00:27:52.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe290b1c640 1 -- start start 2024-09-16T00:27:52.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe290b1c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28c100480 0x7fe28c1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe290b1c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe28c100e30 0x7fe28c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe290b1c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe28c1055b0 0x7fe28c19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe290b1c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c10fd90 con 0x7fe28c1055b0 2024-09-16T00:27:52.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe290b1c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c10fc10 con 0x7fe28c100480 2024-09-16T00:27:52.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe290b1c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c10ff10 con 0x7fe28c100e30 2024-09-16T00:27:52.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe28ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe28c1055b0 0x7fe28c19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe28ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe28c1055b0 0x7fe28c19cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54904/0 (socket says 172.21.15.33:54904) 2024-09-16T00:27:52.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe28ad9d640 1 -- 172.21.15.33:0/34503281 learned_addr learned my addr 172.21.15.33:0/34503281 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:52.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe28a59c640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28c100480 0x7fe28c1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe283fff640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe28c100e30 0x7fe28c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe28ad9d640 1 -- 172.21.15.33:0/34503281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe28c100e30 msgr2=0x7fe28c1988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:52.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe28ad9d640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe28c100e30 0x7fe28c1988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe28ad9d640 1 -- 172.21.15.33:0/34503281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28c100480 msgr2=0x7fe28c1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:52.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe28ad9d640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28c100480 0x7fe28c1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe28ad9d640 1 -- 172.21.15.33:0/34503281 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c19d350 con 0x7fe28c1055b0 2024-09-16T00:27:52.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe28a59c640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28c100480 0x7fe28c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:52.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.576+0000 7fe28ad9d640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe28c1055b0 0x7fe28c19cc70 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe284005cc0 tx=0x7fe284003c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:52.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.577+0000 7fe2837fe640 1 -- 172.21.15.33:0/34503281 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284040070 con 0x7fe28c1055b0 2024-09-16T00:27:52.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.577+0000 7fe290b1c640 1 -- 172.21.15.33:0/34503281 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28c19d5e0 con 0x7fe28c1055b0 2024-09-16T00:27:52.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.577+0000 7fe2837fe640 1 -- 172.21.15.33:0/34503281 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2840309f0 con 0x7fe28c1055b0 2024-09-16T00:27:52.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.577+0000 7fe2837fe640 1 -- 172.21.15.33:0/34503281 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2840044e0 con 0x7fe28c1055b0 2024-09-16T00:27:52.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.577+0000 7fe290b1c640 1 -- 172.21.15.33:0/34503281 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28c1a4f20 con 0x7fe28c1055b0 2024-09-16T00:27:52.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.579+0000 7fe2837fe640 1 -- 172.21.15.33:0/34503281 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe2840046a0 con 0x7fe28c1055b0 2024-09-16T00:27:52.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.579+0000 7fe292ef5740 1 Processor -- start 2024-09-16T00:27:52.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe292ef5740 1 -- start start 2024-09-16T00:27:52.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe292ef5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128bdbf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe292ef5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdca140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe292ef5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdca680 0x56128bdcca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe292ef5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56128bd252e0 con 0x56128bdca680 2024-09-16T00:27:52.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe292ef5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56128bcccce0 con 0x56128bdbf4a0 2024-09-16T00:27:52.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe292ef5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56128bd212a0 con 0x56128bdbfdc0 2024-09-16T00:27:52.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe28a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe283fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdca680 0x56128bdcca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe28a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42010/0 (socket says 172.21.15.33:42010) 2024-09-16T00:27:52.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe28a59c640 1 -- 172.21.15.33:0/1068560841 learned_addr learned my addr 172.21.15.33:0/1068560841 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:52.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe283fff640 1 --2- 172.21.15.33:0/1068560841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128bdbf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe2837fe640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe2780a3240 0x7fe2780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe28a59c640 1 -- 172.21.15.33:0/1068560841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 msgr2=0x56128bdbf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:52.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe28a59c640 1 --2- 172.21.15.33:0/1068560841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128bdbf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe28a59c640 1 -- 172.21.15.33:0/1068560841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdca680 msgr2=0x56128bdcca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:52.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe28a59c640 1 --2- 172.21.15.33:0/1068560841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdca680 0x56128bdcca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe28a59c640 1 -- 172.21.15.33:0/1068560841 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56128bcb1440 con 0x56128bdbfdc0 2024-09-16T00:27:52.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe2837fe640 1 -- 172.21.15.33:0/34503281 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe2840f10d0 con 0x7fe28c1055b0 2024-09-16T00:27:52.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.580+0000 7fe28ad9d640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe2780a3240 0x7fe2780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.581+0000 7fe28a59c640 1 --2- 172.21.15.33:0/1068560841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdca140 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0115f0 tx=0x7fe27c011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d796a69568ce1558 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:52.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.581+0000 7fe280ff9640 1 -- 172.21.15.33:0/1068560841 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c01e070 con 0x56128bdbfdc0 2024-09-16T00:27:52.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.581+0000 7fe280ff9640 1 -- 172.21.15.33:0/1068560841 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe27c00fac0 con 0x56128bdbfdc0 2024-09-16T00:27:52.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.581+0000 7fe280ff9640 1 -- 172.21.15.33:0/1068560841 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c019b60 con 0x56128bdbfdc0 2024-09-16T00:27:52.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.581+0000 7fe292ef5740 1 -- 172.21.15.33:0/1068560841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 msgr2=0x56128bdca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:52.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.581+0000 7fe28ad9d640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe2780a3240 0x7fe2780a5700 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe284032bc0 tx=0x7fe284032a60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:52.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.581+0000 7fe292ef5740 1 --2- 172.21.15.33:0/1068560841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdca140 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0115f0 tx=0x7fe27c011ab0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 -- 172.21.15.33:0/1068560841 shutdown_connections 2024-09-16T00:27:52.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 --2- 172.21.15.33:0/1068560841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdca680 0x56128bdcca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 --2- 172.21.15.33:0/1068560841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdca140 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 --2- 172.21.15.33:0/1068560841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128bdbf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 -- 172.21.15.33:0/1068560841 >> 172.21.15.33:0/1068560841 conn(0x56128bdbba30 msgr2=0x56128bdbbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:52.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 -- 172.21.15.33:0/1068560841 shutdown_connections 2024-09-16T00:27:52.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 -- 172.21.15.33:0/1068560841 wait complete. 2024-09-16T00:27:52.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 -- start start 2024-09-16T00:27:52.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 Processor -- start 2024-09-16T00:27:52.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128bdc9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdc71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdca680 0x56128bdc7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56128bdcd250 con 0x56128bdca680 2024-09-16T00:27:52.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56128bce0050 con 0x56128bdbf4a0 2024-09-16T00:27:52.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56128bdcd3d0 con 0x56128bdbfdc0 2024-09-16T00:27:52.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe292ef5740 10 client.0 _ll_register_callbacks cb 0x56128bd20160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:52.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe283fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdca680 0x56128bdc7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe283fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdca680 0x56128bdc7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54918/0 (socket says 172.21.15.33:54918) 2024-09-16T00:27:52.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe28a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdc71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.582+0000 7fe283fff640 1 -- 172.21.15.33:0/421395079 learned_addr learned my addr 172.21.15.33:0/421395079 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:52.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe283fff640 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128bdc9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe28a59c640 1 -- 172.21.15.33:0/421395079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 msgr2=0x56128bdc9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:52.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe28a59c640 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128bdc9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe28a59c640 1 -- 172.21.15.33:0/421395079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdca680 msgr2=0x56128bdc7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:52.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe28a59c640 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdca680 0x56128bdc7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe28a59c640 1 -- 172.21.15.33:0/421395079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56128ba963c0 con 0x56128bdbfdc0 2024-09-16T00:27:52.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe283fff640 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdca680 0x56128bdc7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:52.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe28a59c640 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdc71d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0115c0 tx=0x7fe27c00f890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=346539b1f1c97c7f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:52.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe271ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:27:52.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe271ffb640 1 -- 172.21.15.33:0/421395079 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c01e040 con 0x56128bdbfdc0 2024-09-16T00:27:52.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe271ffb640 1 -- 172.21.15.33:0/421395079 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe27c004040 con 0x56128bdbfdc0 2024-09-16T00:27:52.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe292ef5740 1 -- 172.21.15.33:0/421395079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe27c00fac0 con 0x56128bdbfdc0 2024-09-16T00:27:52.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.583+0000 7fe271ffb640 1 -- 172.21.15.33:0/421395079 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe27c02f050 con 0x56128bdbfdc0 2024-09-16T00:27:52.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.584+0000 7fe292ef5740 10 client.26755 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:52.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.584+0000 7fe292ef5740 1 -- 172.21.15.33:0/421395079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56128bdd8b70 con 0x56128bdbfdc0 2024-09-16T00:27:52.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.584+0000 7fe271ffb640 1 -- 172.21.15.33:0/421395079 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe27c00fac0 con 0x56128bdbfdc0 2024-09-16T00:27:52.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.584+0000 7fe271ffb640 1 -- 172.21.15.33:0/421395079 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fe27c02a030 con 0x56128bdbfdc0 2024-09-16T00:27:52.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.584+0000 7fe292ef5740 10 client.26755 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:52.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.584+0000 7fe292ef5740 10 client.26755 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:27:52.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.584+0000 7fe292ef5740 20 client.26755 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:52.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.584+0000 7fe292ef5740 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56128bdd9fd0 0x56128bddc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.584+0000 7fe292ef5740 4 client.26755 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:27:52.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.584+0000 7fe292ef5740 1 -- 172.21.15.33:0/421395079 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56128bd32850 con 0x56128bdd9fd0 2024-09-16T00:27:52.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.584+0000 7fe283fff640 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56128bdd9fd0 0x56128bddc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe283fff640 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56128bdd9fd0 0x56128bddc3c0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de9c25fe4097abe6 server_cookie=47ee297baa138094 in_seq=0 out_seq=0 2024-09-16T00:27:52.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe271ffb640 10 client.26755 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:52.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe271ffb640 1 -- 172.21.15.33:0/421395079 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56128bd32850 con 0x56128bdd9fd0 2024-09-16T00:27:52.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe271ffb640 10 client.26755 handle_command_reply: tid=0 2024-09-16T00:27:52.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe292ef5740 1 client.26755 shutdown 2024-09-16T00:27:52.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe292ef5740 1 -- 172.21.15.33:0/421395079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 msgr2=0x56128bdc71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:52.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe292ef5740 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdc71d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0115c0 tx=0x7fe27c00f890 comp rx=0 tx=0).stop 2024-09-16T00:27:52.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe292ef5740 1 -- 172.21.15.33:0/421395079 shutdown_connections 2024-09-16T00:27:52.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe292ef5740 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56128bdd9fd0 0x56128bddc3c0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe292ef5740 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdca680 0x56128bdc7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe292ef5740 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdc71d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe292ef5740 1 --2- 172.21.15.33:0/421395079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128bdc9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe271ffb640 0 client.26755 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:52.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.585+0000 7fe292ef5740 1 -- 172.21.15.33:0/421395079 >> 172.21.15.33:0/421395079 conn(0x56128bdbba30 msgr2=0x56128bdc1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:52.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.586+0000 7fe292ef5740 1 -- 172.21.15.33:0/421395079 shutdown_connections 2024-09-16T00:27:52.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.586+0000 7fe292ef5740 1 -- 172.21.15.33:0/421395079 wait complete. 2024-09-16T00:27:52.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.586+0000 7fe292ef5740 20 client.26755 trim_cache size 0 max 16384 2024-09-16T00:27:52.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.604+0000 7fe292ef5740 1 Processor -- start 2024-09-16T00:27:52.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.604+0000 7fe292ef5740 1 -- start start 2024-09-16T00:27:52.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.604+0000 7fe292ef5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbba30 0x56128bdd64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.604+0000 7fe292ef5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdbf4a0 0x56128bdc2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.604+0000 7fe292ef5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdc61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.604+0000 7fe292ef5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1b1930 con 0x56128bdbf4a0 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.604+0000 7fe292ef5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1af200 con 0x56128bdbba30 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.604+0000 7fe292ef5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1ad050 con 0x56128bdbfdc0 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.604+0000 7fe28a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdbf4a0 0x56128bdc2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.604+0000 7fe28a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdbf4a0 0x56128bdc2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54928/0 (socket says 172.21.15.33:54928) 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.604+0000 7fe28a59c640 1 -- 172.21.15.33:0/3471036437 learned_addr learned my addr 172.21.15.33:0/3471036437 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.604+0000 7fe283fff640 1 --2- 172.21.15.33:0/3471036437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbba30 0x56128bdd64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.605+0000 7fe283fff640 1 --2- 172.21.15.33:0/3471036437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdc61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.605+0000 7fe28a59c640 1 -- 172.21.15.33:0/3471036437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 msgr2=0x56128bdc61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.605+0000 7fe28a59c640 1 --2- 172.21.15.33:0/3471036437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdc61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.605+0000 7fe28a59c640 1 -- 172.21.15.33:0/3471036437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbba30 msgr2=0x56128bdd64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.605+0000 7fe28a59c640 1 --2- 172.21.15.33:0/3471036437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbba30 0x56128bdd64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.605+0000 7fe28a59c640 1 -- 172.21.15.33:0/3471036437 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56128bdc5720 con 0x56128bdbf4a0 2024-09-16T00:27:52.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.605+0000 7fe28a59c640 1 --2- 172.21.15.33:0/3471036437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdbf4a0 0x56128bdc2900 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009bd0 tx=0x7fe27c00f440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee274e87748fd017 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.605+0000 7fe273fff640 1 -- 172.21.15.33:0/3471036437 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c015030 con 0x56128bdbf4a0 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.605+0000 7fe273fff640 1 -- 172.21.15.33:0/3471036437 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe27c0052f0 con 0x56128bdbf4a0 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.605+0000 7fe273fff640 1 -- 172.21.15.33:0/3471036437 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c01e040 con 0x56128bdbf4a0 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.606+0000 7fe292ef5740 1 -- 172.21.15.33:0/3471036437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdbf4a0 msgr2=0x56128bdc2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.606+0000 7fe292ef5740 1 --2- 172.21.15.33:0/3471036437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdbf4a0 0x56128bdc2900 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009bd0 tx=0x7fe27c00f440 comp rx=0 tx=0).stop 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.606+0000 7fe292ef5740 1 -- 172.21.15.33:0/3471036437 shutdown_connections 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.606+0000 7fe292ef5740 1 --2- 172.21.15.33:0/3471036437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbfdc0 0x56128bdc61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.606+0000 7fe292ef5740 1 --2- 172.21.15.33:0/3471036437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdbf4a0 0x56128bdc2900 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.606+0000 7fe292ef5740 1 --2- 172.21.15.33:0/3471036437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbba30 0x56128bdd64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.606+0000 7fe292ef5740 1 -- 172.21.15.33:0/3471036437 >> 172.21.15.33:0/3471036437 conn(0x56128bdc7870 msgr2=0x56128bdc5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.606+0000 7fe292ef5740 1 -- 172.21.15.33:0/3471036437 shutdown_connections 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.606+0000 7fe292ef5740 1 -- 172.21.15.33:0/3471036437 wait complete. 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.606+0000 7fe292ef5740 1 -- start start 2024-09-16T00:27:52.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.606+0000 7fe292ef5740 1 Processor -- start 2024-09-16T00:27:52.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe292ef5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbba30 0x56128be41270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe292ef5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128be417b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe292ef5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdbfdc0 0x56128bdc1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe292ef5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1ad6c0 con 0x56128bdbfdc0 2024-09-16T00:27:52.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe292ef5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1aeb30 con 0x56128bdbf4a0 2024-09-16T00:27:52.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe292ef5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1add90 con 0x56128bdbba30 2024-09-16T00:27:52.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe292ef5740 10 client.0 _ll_register_callbacks cb 0x7fe27c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:52.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe283fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdbfdc0 0x56128bdc1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe28a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128be417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe283fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbba30 0x56128be41270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe28a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128be417b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:52782/0 (socket says 172.21.15.33:52782) 2024-09-16T00:27:52.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe28a59c640 1 -- 172.21.15.33:0/1662220919 learned_addr learned my addr 172.21.15.33:0/1662220919 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:52.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe28a59c640 1 -- 172.21.15.33:0/1662220919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbba30 msgr2=0x56128be41270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:52.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe28a59c640 1 --2- 172.21.15.33:0/1662220919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbba30 0x56128be41270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe28a59c640 1 -- 172.21.15.33:0/1662220919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdbfdc0 msgr2=0x56128bdc1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:52.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe28a59c640 1 --2- 172.21.15.33:0/1662220919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdbfdc0 0x56128bdc1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe28a59c640 1 -- 172.21.15.33:0/1662220919 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56128bdc2180 con 0x56128bdbf4a0 2024-09-16T00:27:52.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe28a59c640 1 --2- 172.21.15.33:0/1662220919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128be417b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe27c010010 tx=0x7fe27c019590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b09072927464d826 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:52.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe271ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:27:52.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.607+0000 7fe271ffb640 1 -- 172.21.15.33:0/1662220919 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c03a070 con 0x56128bdbf4a0 2024-09-16T00:27:52.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe292ef5740 1 -- 172.21.15.33:0/1662220919 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe27c0052f0 con 0x56128bdbf4a0 2024-09-16T00:27:52.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe271ffb640 1 -- 172.21.15.33:0/1662220919 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe27c00f7f0 con 0x56128bdbf4a0 2024-09-16T00:27:52.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe271ffb640 1 -- 172.21.15.33:0/1662220919 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe27c046050 con 0x56128bdbf4a0 2024-09-16T00:27:52.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe292ef5740 10 client.26795 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:52.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe292ef5740 1 -- 172.21.15.33:0/1662220919 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56128be48a50 con 0x56128bdbf4a0 2024-09-16T00:27:52.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe271ffb640 1 -- 172.21.15.33:0/1662220919 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe27c0052f0 con 0x56128bdbf4a0 2024-09-16T00:27:52.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe271ffb640 1 -- 172.21.15.33:0/1662220919 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fe27c02a040 con 0x56128bdbf4a0 2024-09-16T00:27:52.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe292ef5740 10 client.26795 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:52.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe292ef5740 10 client.26795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:27:52.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe292ef5740 20 client.26795 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:52.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe292ef5740 1 --2- 172.21.15.33:0/1662220919 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56128bdca680 0x56128be4b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:52.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe292ef5740 4 client.26795 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:27:52.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe292ef5740 1 -- 172.21.15.33:0/1662220919 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56128bd2cd60 con 0x56128bdca680 2024-09-16T00:27:52.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.608+0000 7fe283fff640 1 --2- 172.21.15.33:0/1662220919 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56128bdca680 0x56128be4b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:52.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe271ffb640 10 client.26795 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:52.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe283fff640 1 --2- 172.21.15.33:0/1662220919 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56128bdca680 0x56128be4b200 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b691c158dc153730 server_cookie=b3f1be8bb8657e3d in_seq=0 out_seq=0 2024-09-16T00:27:52.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe271ffb640 1 -- 172.21.15.33:0/1662220919 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56128bd2cd60 con 0x56128bdca680 2024-09-16T00:27:52.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe271ffb640 10 client.26795 handle_command_reply: tid=0 2024-09-16T00:27:52.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe292ef5740 1 client.26795 shutdown 2024-09-16T00:27:52.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe292ef5740 1 -- 172.21.15.33:0/1662220919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 msgr2=0x56128be417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:52.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe292ef5740 1 --2- 172.21.15.33:0/1662220919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128be417b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe27c010010 tx=0x7fe27c019590 comp rx=0 tx=0).stop 2024-09-16T00:27:52.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe292ef5740 1 -- 172.21.15.33:0/1662220919 shutdown_connections 2024-09-16T00:27:52.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe292ef5740 1 --2- 172.21.15.33:0/1662220919 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56128bdca680 0x56128be4b200 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe292ef5740 1 --2- 172.21.15.33:0/1662220919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56128bdbfdc0 0x56128bdc1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe292ef5740 1 --2- 172.21.15.33:0/1662220919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56128bdbf4a0 0x56128be417b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe271ffb640 0 client.26795 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:52.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe292ef5740 1 --2- 172.21.15.33:0/1662220919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56128bdbba30 0x56128be41270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe292ef5740 1 -- 172.21.15.33:0/1662220919 >> 172.21.15.33:0/1662220919 conn(0x56128bdc7870 msgr2=0x56128bdba350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:52.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe292ef5740 1 -- 172.21.15.33:0/1662220919 shutdown_connections 2024-09-16T00:27:52.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe292ef5740 1 -- 172.21.15.33:0/1662220919 wait complete. 2024-09-16T00:27:52.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.609+0000 7fe292ef5740 20 client.26795 trim_cache size 0 max 16384 2024-09-16T00:27:52.620 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:27:52.621 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:27:52.621 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:27:52.621 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:27:52.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.610+0000 7fe290b1c640 1 -- 172.21.15.33:0/34503281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe2780a3240 msgr2=0x7fe2780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:52.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.610+0000 7fe290b1c640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe2780a3240 0x7fe2780a5700 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe284032bc0 tx=0x7fe284032a60 comp rx=0 tx=0).stop 2024-09-16T00:27:52.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.610+0000 7fe290b1c640 1 -- 172.21.15.33:0/34503281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe28c1055b0 msgr2=0x7fe28c19cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:52.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.610+0000 7fe290b1c640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe28c1055b0 0x7fe28c19cc70 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe284005cc0 tx=0x7fe284003c90 comp rx=0 tx=0).stop 2024-09-16T00:27:52.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.610+0000 7fe290b1c640 1 -- 172.21.15.33:0/34503281 shutdown_connections 2024-09-16T00:27:52.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.610+0000 7fe290b1c640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe2780a3240 0x7fe2780a5700 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.610+0000 7fe290b1c640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe28c1055b0 0x7fe28c19cc70 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.610+0000 7fe290b1c640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe28c100e30 0x7fe28c1988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.610+0000 7fe290b1c640 1 --2- 172.21.15.33:0/34503281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28c100480 0x7fe28c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:52.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.610+0000 7fe290b1c640 1 -- 172.21.15.33:0/34503281 >> 172.21.15.33:0/34503281 conn(0x7fe28c077210 msgr2=0x7fe28c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:52.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.610+0000 7fe290b1c640 1 -- 172.21.15.33:0/34503281 shutdown_connections 2024-09-16T00:27:52.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:52.610+0000 7fe290b1c640 1 -- 172.21.15.33:0/34503281 wait complete. 2024-09-16T00:27:52.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 895065 10.01 MB/sec execute 420 sec latency 9.086 ms 2024-09-16T00:27:53.421 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:53 smithi033 ceph-mon[31243]: pgmap v672: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.9 MiB/s wr, 324 op/s 2024-09-16T00:27:53.470 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:27:53.470 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:27:53.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:53 smithi083 ceph-mon[35242]: pgmap v672: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.9 MiB/s wr, 324 op/s 2024-09-16T00:27:53.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 898219 10.03 MB/sec execute 421 sec latency 18.744 ms 2024-09-16T00:27:53.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:53 smithi060 ceph-mon[35185]: pgmap v672: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.9 MiB/s wr, 324 op/s 2024-09-16T00:27:53.739 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:54.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.256+0000 7f597b3d9640 1 -- 172.21.15.33:0/70287738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5974101de0 msgr2=0x7f59741021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:54.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.256+0000 7f597b3d9640 1 --2- 172.21.15.33:0/70287738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5974101de0 0x7f59741021c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f5970009900 tx=0x7f597002a410 comp rx=0 tx=0).stop 2024-09-16T00:27:54.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.256+0000 7f597b3d9640 1 -- 172.21.15.33:0/70287738 shutdown_connections 2024-09-16T00:27:54.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.256+0000 7f597b3d9640 1 --2- 172.21.15.33:0/70287738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5974106f10 0x7f597410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:54.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.256+0000 7f597b3d9640 1 --2- 172.21.15.33:0/70287738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5974102790 0x7f59741067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:54.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.256+0000 7f597b3d9640 1 --2- 172.21.15.33:0/70287738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5974101de0 0x7f59741021c0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:54.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.256+0000 7f597b3d9640 1 -- 172.21.15.33:0/70287738 >> 172.21.15.33:0/70287738 conn(0x7f59740fd8c0 msgr2=0x7f59740ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:54.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.256+0000 7f597b3d9640 1 -- 172.21.15.33:0/70287738 shutdown_connections 2024-09-16T00:27:54.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.256+0000 7f597b3d9640 1 -- 172.21.15.33:0/70287738 wait complete. 2024-09-16T00:27:54.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f597b3d9640 1 Processor -- start 2024-09-16T00:27:54.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f597b3d9640 1 -- start start 2024-09-16T00:27:54.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f597b3d9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5974101de0 0x7f5974198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f597b3d9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5974102790 0x7f5974198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f597b3d9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5974106f10 0x7f597419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f597b3d9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5974111f70 con 0x7f5974101de0 2024-09-16T00:27:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f597b3d9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5974111df0 con 0x7f5974102790 2024-09-16T00:27:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f597b3d9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f59741120f0 con 0x7f5974106f10 2024-09-16T00:27:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f5979175640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5974101de0 0x7f5974198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f5979175640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5974101de0 0x7f5974198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54936/0 (socket says 172.21.15.33:54936) 2024-09-16T00:27:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f5979175640 1 -- 172.21.15.33:0/728803489 learned_addr learned my addr 172.21.15.33:0/728803489 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f5978974640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5974102790 0x7f5974198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f5979976640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5974106f10 0x7f597419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f5979175640 1 -- 172.21.15.33:0/728803489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5974106f10 msgr2=0x7f597419cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:54.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f5979175640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5974106f10 0x7f597419cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f5979175640 1 -- 172.21.15.33:0/728803489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5974102790 msgr2=0x7f5974198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.257+0000 7f5979175640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5974102790 0x7f5974198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.258+0000 7f5979175640 1 -- 172.21.15.33:0/728803489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597419d3c0 con 0x7f5974101de0 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.258+0000 7f5979175640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5974101de0 0x7f5974198410 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5970006010 tx=0x7f59700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.258+0000 7f596a7fc640 1 -- 172.21.15.33:0/728803489 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597003f030 con 0x7f5974101de0 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.258+0000 7f596a7fc640 1 -- 172.21.15.33:0/728803489 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5970003ce0 con 0x7f5974101de0 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.258+0000 7f597b3d9640 1 -- 172.21.15.33:0/728803489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f597419d650 con 0x7f5974101de0 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.258+0000 7f596a7fc640 1 -- 172.21.15.33:0/728803489 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5970035070 con 0x7f5974101de0 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.258+0000 7f597b3d9640 1 -- 172.21.15.33:0/728803489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59741a4f90 con 0x7f5974101de0 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.259+0000 7f597b3d9640 1 -- 172.21.15.33:0/728803489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59741032c0 con 0x7f5974101de0 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.260+0000 7f596a7fc640 1 -- 172.21.15.33:0/728803489 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5970042030 con 0x7f5974101de0 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.261+0000 7f596a7fc640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f59600a3240 0x7f59600a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.261+0000 7f5978974640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f59600a3240 0x7f59600a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:54.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.261+0000 7f596a7fc640 1 -- 172.21.15.33:0/728803489 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f59700f02c0 con 0x7f5974101de0 2024-09-16T00:27:54.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.262+0000 7f5978974640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f59600a3240 0x7f59600a5700 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f59741088e0 tx=0x7f596400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:54.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.265+0000 7f596a7fc640 1 -- 172.21.15.33:0/728803489 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f59700bc5d0 con 0x7f5974101de0 2024-09-16T00:27:54.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.498+0000 7f597b3d9640 1 -- 172.21.15.33:0/728803489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f597410a0f0 con 0x7f5974101de0 2024-09-16T00:27:54.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.499+0000 7f596a7fc640 1 -- 172.21.15.33:0/728803489 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f59700c08d0 con 0x7f5974101de0 2024-09-16T00:27:54.501 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:27:54.501 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:27:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:27:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.501+0000 7f597b3d9640 1 -- 172.21.15.33:0/728803489 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f59600a3240 msgr2=0x7f59600a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.501+0000 7f597b3d9640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f59600a3240 0x7f59600a5700 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f59741088e0 tx=0x7f596400b410 comp rx=0 tx=0).stop 2024-09-16T00:27:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.501+0000 7f597b3d9640 1 -- 172.21.15.33:0/728803489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5974101de0 msgr2=0x7f5974198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.501+0000 7f597b3d9640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5974101de0 0x7f5974198410 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5970006010 tx=0x7f59700042f0 comp rx=0 tx=0).stop 2024-09-16T00:27:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.501+0000 7f597b3d9640 1 -- 172.21.15.33:0/728803489 shutdown_connections 2024-09-16T00:27:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.502+0000 7f597b3d9640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f59600a3240 0x7f59600a5700 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:54.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.502+0000 7f597b3d9640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5974106f10 0x7f597419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:54.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.502+0000 7f597b3d9640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5974102790 0x7f5974198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:54.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.502+0000 7f597b3d9640 1 --2- 172.21.15.33:0/728803489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5974101de0 0x7f5974198410 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:54.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.502+0000 7f597b3d9640 1 -- 172.21.15.33:0/728803489 >> 172.21.15.33:0/728803489 conn(0x7f59740fd8c0 msgr2=0x7f59740fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:54.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.502+0000 7f597b3d9640 1 -- 172.21.15.33:0/728803489 shutdown_connections 2024-09-16T00:27:54.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:54.502+0000 7f597b3d9640 1 -- 172.21.15.33:0/728803489 wait complete. 2024-09-16T00:27:54.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 900918 10.03 MB/sec execute 422 sec latency 12.687 ms 2024-09-16T00:27:55.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:55 smithi033 ceph-mon[31243]: pgmap v673: 129 pgs: 129 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 302 op/s 2024-09-16T00:27:55.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:55 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/728803489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:27:55.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:55 smithi083 ceph-mon[35242]: pgmap v673: 129 pgs: 129 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 302 op/s 2024-09-16T00:27:55.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:55 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/728803489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:27:55.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 902575 10.03 MB/sec execute 423 sec latency 117.879 ms 2024-09-16T00:27:55.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:55 smithi060 ceph-mon[35185]: pgmap v673: 129 pgs: 129 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 302 op/s 2024-09-16T00:27:55.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:55 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/728803489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:27:55.777 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:27:56.041 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:56.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.599+0000 7ff862187640 1 -- 172.21.15.33:0/3481833243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff85c108300 msgr2=0x7ff85c10ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:56.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.599+0000 7ff862187640 1 --2- 172.21.15.33:0/3481833243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff85c108300 0x7ff85c10ec70 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff854009cd0 tx=0x7ff85402a8d0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.599+0000 7ff862187640 1 -- 172.21.15.33:0/3481833243 shutdown_connections 2024-09-16T00:27:56.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.599+0000 7ff862187640 1 --2- 172.21.15.33:0/3481833243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff85c108300 0x7ff85c10ec70 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.599+0000 7ff862187640 1 --2- 172.21.15.33:0/3481833243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff85c103780 0x7ff85c107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.599+0000 7ff862187640 1 --2- 172.21.15.33:0/3481833243 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff85c102dd0 0x7ff85c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.599+0000 7ff862187640 1 -- 172.21.15.33:0/3481833243 >> 172.21.15.33:0/3481833243 conn(0x7ff85c0fc7d0 msgr2=0x7ff85c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:56.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.599+0000 7ff862187640 1 -- 172.21.15.33:0/3481833243 shutdown_connections 2024-09-16T00:27:56.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.599+0000 7ff862187640 1 -- 172.21.15.33:0/3481833243 wait complete. 2024-09-16T00:27:56.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.600+0000 7ff862187640 1 Processor -- start 2024-09-16T00:27:56.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.600+0000 7ff862187640 1 -- start start 2024-09-16T00:27:56.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff862187640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff85c102dd0 0x7ff85c1a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff862187640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff85c103780 0x7ff85c1a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff862187640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff85c108300 0x7ff85c10de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff862187640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c1120d0 con 0x7ff85c108300 2024-09-16T00:27:56.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff862187640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c111f50 con 0x7ff85c102dd0 2024-09-16T00:27:56.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff862187640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c112250 con 0x7ff85c103780 2024-09-16T00:27:56.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff85bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff85c108300 0x7ff85c10de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff85affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff85c103780 0x7ff85c1a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff85b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff85c102dd0 0x7ff85c1a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff85bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff85c108300 0x7ff85c10de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54948/0 (socket says 172.21.15.33:54948) 2024-09-16T00:27:56.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff85bfff640 1 -- 172.21.15.33:0/4069871882 learned_addr learned my addr 172.21.15.33:0/4069871882 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:56.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff85affd640 1 -- 172.21.15.33:0/4069871882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff85c102dd0 msgr2=0x7ff85c1a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:56.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff85affd640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff85c102dd0 0x7ff85c1a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff85affd640 1 -- 172.21.15.33:0/4069871882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff85c108300 msgr2=0x7ff85c10de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:56.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff85affd640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff85c108300 0x7ff85c10de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff85affd640 1 -- 172.21.15.33:0/4069871882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85c10e670 con 0x7ff85c103780 2024-09-16T00:27:56.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff85bfff640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff85c108300 0x7ff85c10de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:56.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.601+0000 7ff85affd640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff85c103780 0x7ff85c1a4e00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0697d0 tx=0x7ff85000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:56.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.602+0000 7ff858ff9640 1 -- 172.21.15.33:0/4069871882 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85000cc40 con 0x7ff85c103780 2024-09-16T00:27:56.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.602+0000 7ff862187640 1 -- 172.21.15.33:0/4069871882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff85c10e960 con 0x7ff85c103780 2024-09-16T00:27:56.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.602+0000 7ff858ff9640 1 -- 172.21.15.33:0/4069871882 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff85000cde0 con 0x7ff85c103780 2024-09-16T00:27:56.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.602+0000 7ff858ff9640 1 -- 172.21.15.33:0/4069871882 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850004a00 con 0x7ff85c103780 2024-09-16T00:27:56.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.602+0000 7ff862187640 1 -- 172.21.15.33:0/4069871882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff85c1ab6b0 con 0x7ff85c103780 2024-09-16T00:27:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.604+0000 7ff864560740 1 Processor -- start 2024-09-16T00:27:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.604+0000 7ff864560740 1 -- start start 2024-09-16T00:27:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.604+0000 7ff858ff9640 1 -- 172.21.15.33:0/4069871882 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff850004ba0 con 0x7ff85c103780 2024-09-16T00:27:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.604+0000 7ff864560740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559110540b40 0x55911053f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff864560740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55911053f5d0 0x55911053fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff864560740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911054a370 0x55911054c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff864560740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5591104a52f0 con 0x55911054a370 2024-09-16T00:27:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff864560740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55911044ccf0 con 0x55911053f5d0 2024-09-16T00:27:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff864560740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5591104a12b0 con 0x559110540b40 2024-09-16T00:27:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff85bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559110540b40 0x55911053f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff85bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559110540b40 0x55911053f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42090/0 (socket says 172.21.15.33:42090) 2024-09-16T00:27:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff85bfff640 1 -- 172.21.15.33:0/459743621 learned_addr learned my addr 172.21.15.33:0/459743621 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff85affd640 1 --2- 172.21.15.33:0/459743621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911054a370 0x55911054c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/459743621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55911053f5d0 0x55911053fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff85bfff640 1 -- 172.21.15.33:0/459743621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55911053f5d0 msgr2=0x55911053fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff85bfff640 1 --2- 172.21.15.33:0/459743621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55911053f5d0 0x55911053fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff85bfff640 1 -- 172.21.15.33:0/459743621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911054a370 msgr2=0x55911054c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff85bfff640 1 --2- 172.21.15.33:0/459743621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911054a370 0x55911054c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff85bfff640 1 -- 172.21.15.33:0/459743621 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591104313a0 con 0x559110540b40 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff85affd640 1 --2- 172.21.15.33:0/459743621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911054a370 0x55911054c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/459743621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55911053f5d0 0x55911053fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.605+0000 7ff858ff9640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8440a32c0 0x7ff8440a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.606+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8440a32c0 0x7ff8440a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.606+0000 7ff85bfff640 1 --2- 172.21.15.33:0/459743621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559110540b40 0x55911053f090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff854005f60 tx=0x7ff854005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a762c2caec3ce891 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.606+0000 7ff858ff9640 1 -- 172.21.15.33:0/4069871882 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff8500cfcb0 con 0x7ff85c103780 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.606+0000 7ff841ffb640 1 -- 172.21.15.33:0/459743621 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854031cb0 con 0x559110540b40 2024-09-16T00:27:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.606+0000 7ff841ffb640 1 -- 172.21.15.33:0/459743621 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff854036030 con 0x559110540b40 2024-09-16T00:27:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.606+0000 7ff841ffb640 1 -- 172.21.15.33:0/459743621 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854004240 con 0x559110540b40 2024-09-16T00:27:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.607+0000 7ff864560740 1 -- 172.21.15.33:0/459743621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559110540b40 msgr2=0x55911053f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.607+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8440a32c0 0x7ff8440a5780 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x55911053cac0 tx=0x7ff84c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.607+0000 7ff864560740 1 --2- 172.21.15.33:0/459743621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559110540b40 0x55911053f090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff854005f60 tx=0x7ff854005f90 comp rx=0 tx=0).stop 2024-09-16T00:27:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.607+0000 7ff864560740 1 -- 172.21.15.33:0/459743621 shutdown_connections 2024-09-16T00:27:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.607+0000 7ff864560740 1 --2- 172.21.15.33:0/459743621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911054a370 0x55911054c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.607+0000 7ff864560740 1 --2- 172.21.15.33:0/459743621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55911053f5d0 0x55911053fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.607+0000 7ff864560740 1 --2- 172.21.15.33:0/459743621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559110540b40 0x55911053f090 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.607+0000 7ff864560740 1 -- 172.21.15.33:0/459743621 >> 172.21.15.33:0/459743621 conn(0x559110508310 msgr2=0x55911053bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:56.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.607+0000 7ff864560740 1 -- 172.21.15.33:0/459743621 shutdown_connections 2024-09-16T00:27:56.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.607+0000 7ff864560740 1 -- 172.21.15.33:0/459743621 wait complete. 2024-09-16T00:27:56.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.607+0000 7ff864560740 1 -- start start 2024-09-16T00:27:56.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff864560740 1 Processor -- start 2024-09-16T00:27:56.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff864560740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55911053f5d0 0x559110549ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff864560740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559110540b40 0x559110546f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff864560740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911054a370 0x5591105474f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff864560740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55911054d010 con 0x559110540b40 2024-09-16T00:27:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff864560740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559110460060 con 0x55911053f5d0 2024-09-16T00:27:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff85affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559110540b40 0x559110546f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff85bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911054a370 0x5591105474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff85affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559110540b40 0x559110546f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54956/0 (socket says 172.21.15.33:54956) 2024-09-16T00:27:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff85affd640 1 -- 172.21.15.33:0/728392768 learned_addr learned my addr 172.21.15.33:0/728392768 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff85bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911054a370 0x5591105474f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42100/0 (socket says 172.21.15.33:42100) 2024-09-16T00:27:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff864560740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55911054d190 con 0x55911054a370 2024-09-16T00:27:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.608+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55911053f5d0 0x559110549ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.609+0000 7ff85b7fe640 1 -- 172.21.15.33:0/728392768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911054a370 msgr2=0x5591105474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.609+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911054a370 0x5591105474f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.609+0000 7ff85b7fe640 1 -- 172.21.15.33:0/728392768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559110540b40 msgr2=0x559110546f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.609+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559110540b40 0x559110546f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.609+0000 7ff85b7fe640 1 -- 172.21.15.33:0/728392768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591102163c0 con 0x55911053f5d0 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.609+0000 7ff85affd640 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559110540b40 0x559110546f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.609+0000 7ff864560740 10 client.0 _ll_register_callbacks cb 0x5591101b38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.609+0000 7ff864560740 1 -- 172.21.15.33:0/728392768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff854036030 con 0x55911053f5d0 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.609+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55911053f5d0 0x559110549ce0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff84c012cb0 tx=0x7ff84c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8157daed586e9e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.609+0000 7ff83affd640 10 client.26813 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.609+0000 7ff83affd640 1 -- 172.21.15.33:0/728392768 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c02e060 con 0x55911053f5d0 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.609+0000 7ff83affd640 1 -- 172.21.15.33:0/728392768 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff84c018030 con 0x55911053f5d0 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.610+0000 7ff83affd640 1 -- 172.21.15.33:0/728392768 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff84c009db0 con 0x55911053f5d0 2024-09-16T00:27:56.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.610+0000 7ff864560740 10 client.26813 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:56.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.610+0000 7ff864560740 1 -- 172.21.15.33:0/728392768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({fsmap=30}) -- 0x559110558880 con 0x55911053f5d0 2024-09-16T00:27:56.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.610+0000 7ff83affd640 1 -- 172.21.15.33:0/728392768 <== mon.2 v2:172.21.15.60:3300/0 4 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff84c00f3c0 con 0x55911053f5d0 2024-09-16T00:27:56.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.610+0000 7ff864560740 10 client.26813 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:56.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.610+0000 7ff864560740 10 client.26813 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:27:56.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.610+0000 7ff864560740 20 client.26813 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:56.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.610+0000 7ff864560740 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x559110559c20 0x55911055c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.610+0000 7ff864560740 4 client.26813 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:27:56.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.610+0000 7ff864560740 1 -- 172.21.15.33:0/728392768 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591104b2860 con 0x559110559c20 2024-09-16T00:27:56.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.610+0000 7ff85affd640 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x559110559c20 0x55911055c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.611+0000 7ff83affd640 10 client.26813 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:27:56.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.611+0000 7ff85affd640 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x559110559c20 0x55911055c070 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cbf2924c14fe8fa8 server_cookie=ddf599aa56a41ef3 in_seq=0 out_seq=0 2024-09-16T00:27:56.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.611+0000 7ff83affd640 1 -- 172.21.15.33:0/728392768 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5591104b2860 con 0x559110559c20 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.611+0000 7ff83affd640 10 client.26813 handle_command_reply: tid=0 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.611+0000 7ff864560740 1 client.26813 shutdown 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.612+0000 7ff864560740 1 -- 172.21.15.33:0/728392768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55911053f5d0 msgr2=0x559110549ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.612+0000 7ff864560740 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55911053f5d0 0x559110549ce0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff84c012cb0 tx=0x7ff84c004730 comp rx=0 tx=0).stop 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.612+0000 7ff864560740 1 -- 172.21.15.33:0/728392768 shutdown_connections 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.612+0000 7ff864560740 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x559110559c20 0x55911055c070 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.612+0000 7ff864560740 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911054a370 0x5591105474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.612+0000 7ff864560740 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559110540b40 0x559110546f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.612+0000 7ff864560740 1 --2- 172.21.15.33:0/728392768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55911053f5d0 0x559110549ce0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.612+0000 7ff864560740 1 -- 172.21.15.33:0/728392768 >> 172.21.15.33:0/728392768 conn(0x559110508310 msgr2=0x559110541960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.612+0000 7ff83affd640 0 client.26813 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.612+0000 7ff864560740 1 -- 172.21.15.33:0/728392768 shutdown_connections 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.612+0000 7ff864560740 1 -- 172.21.15.33:0/728392768 wait complete. 2024-09-16T00:27:56.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.612+0000 7ff864560740 20 client.26813 trim_cache size 0 max 16384 2024-09-16T00:27:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff864560740 1 Processor -- start 2024-09-16T00:27:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff864560740 1 -- start start 2024-09-16T00:27:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff864560740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911053f5d0 0x559110546850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff864560740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559110540b40 0x559110548f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff864560740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911054a370 0x559110542120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff864560740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c1b7f00 con 0x55911053f5d0 2024-09-16T00:27:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff864560740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c1b57d0 con 0x559110540b40 2024-09-16T00:27:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff864560740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c1b3620 con 0x55911054a370 2024-09-16T00:27:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff85b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911053f5d0 0x559110546850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff85b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911053f5d0 0x559110546850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54972/0 (socket says 172.21.15.33:54972) 2024-09-16T00:27:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff85b7fe640 1 -- 172.21.15.33:0/1846695994 learned_addr learned my addr 172.21.15.33:0/1846695994 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff85bfff640 1 --2- 172.21.15.33:0/1846695994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911054a370 0x559110542120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff85affd640 1 --2- 172.21.15.33:0/1846695994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559110540b40 0x559110548f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff85b7fe640 1 -- 172.21.15.33:0/1846695994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911054a370 msgr2=0x559110542120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/1846695994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911054a370 0x559110542120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff85b7fe640 1 -- 172.21.15.33:0/1846695994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559110540b40 msgr2=0x559110548f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/1846695994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559110540b40 0x559110548f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.626+0000 7ff85b7fe640 1 -- 172.21.15.33:0/1846695994 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559110545f40 con 0x55911053f5d0 2024-09-16T00:27:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.627+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/1846695994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911053f5d0 0x559110546850 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ff84c01eda0 tx=0x7ff84c023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5c227c9849b70 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.627+0000 7ff8417fa640 1 -- 172.21.15.33:0/1846695994 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c02e040 con 0x55911053f5d0 2024-09-16T00:27:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.627+0000 7ff8417fa640 1 -- 172.21.15.33:0/1846695994 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff854036030 con 0x55911053f5d0 2024-09-16T00:27:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.627+0000 7ff8417fa640 1 -- 172.21.15.33:0/1846695994 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c00fa50 con 0x55911053f5d0 2024-09-16T00:27:56.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.628+0000 7ff864560740 1 -- 172.21.15.33:0/1846695994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911053f5d0 msgr2=0x559110546850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:56.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.628+0000 7ff864560740 1 --2- 172.21.15.33:0/1846695994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911053f5d0 0x559110546850 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ff84c01eda0 tx=0x7ff84c023000 comp rx=0 tx=0).stop 2024-09-16T00:27:56.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.628+0000 7ff864560740 1 -- 172.21.15.33:0/1846695994 shutdown_connections 2024-09-16T00:27:56.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.628+0000 7ff864560740 1 --2- 172.21.15.33:0/1846695994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911054a370 0x559110542120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.628+0000 7ff864560740 1 --2- 172.21.15.33:0/1846695994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559110540b40 0x559110548f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.628+0000 7ff864560740 1 --2- 172.21.15.33:0/1846695994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911053f5d0 0x559110546850 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.628+0000 7ff864560740 1 -- 172.21.15.33:0/1846695994 >> 172.21.15.33:0/1846695994 conn(0x559110508310 msgr2=0x5591105487f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:56.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.628+0000 7ff864560740 1 -- 172.21.15.33:0/1846695994 shutdown_connections 2024-09-16T00:27:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.628+0000 7ff864560740 1 -- 172.21.15.33:0/1846695994 wait complete. 2024-09-16T00:27:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.628+0000 7ff864560740 1 -- start start 2024-09-16T00:27:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.628+0000 7ff864560740 1 Processor -- start 2024-09-16T00:27:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff864560740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911053f5d0 0x559110543e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff864560740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559110540b40 0x5591105443d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff864560740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911054a370 0x559110544910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff864560740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c1b3c90 con 0x55911054a370 2024-09-16T00:27:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff864560740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c1b5100 con 0x559110540b40 2024-09-16T00:27:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff864560740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c1b4360 con 0x55911053f5d0 2024-09-16T00:27:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff864560740 10 client.0 _ll_register_callbacks cb 0x7ff84c009680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff85bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911054a370 0x559110544910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff85b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911053f5d0 0x559110543e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff85affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559110540b40 0x5591105443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff85b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911053f5d0 0x559110543e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42120/0 (socket says 172.21.15.33:42120) 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff85b7fe640 1 -- 172.21.15.33:0/592342961 learned_addr learned my addr 172.21.15.33:0/592342961 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff85b7fe640 1 -- 172.21.15.33:0/592342961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559110540b40 msgr2=0x5591105443d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/592342961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559110540b40 0x5591105443d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff85b7fe640 1 -- 172.21.15.33:0/592342961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911054a370 msgr2=0x559110544910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/592342961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911054a370 0x559110544910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff85b7fe640 1 -- 172.21.15.33:0/592342961 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591105c3950 con 0x55911053f5d0 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff85b7fe640 1 --2- 172.21.15.33:0/592342961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911053f5d0 0x559110543e90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff84c01efd0 tx=0x7ff84c005730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4799356e18a20bd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.629+0000 7ff83affd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff83affd640 1 -- 172.21.15.33:0/592342961 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c040070 con 0x55911053f5d0 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff864560740 1 -- 172.21.15.33:0/592342961 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff854036030 con 0x55911053f5d0 2024-09-16T00:27:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff83affd640 1 -- 172.21.15.33:0/592342961 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff84c00f870 con 0x55911053f5d0 2024-09-16T00:27:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff83affd640 1 -- 172.21.15.33:0/592342961 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff84c00fab0 con 0x55911053f5d0 2024-09-16T00:27:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff864560740 10 client.26785 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff864560740 1 -- 172.21.15.33:0/592342961 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5591105c4b30 con 0x55911053f5d0 2024-09-16T00:27:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff83affd640 1 -- 172.21.15.33:0/592342961 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff854036030 con 0x55911053f5d0 2024-09-16T00:27:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff83affd640 1 -- 172.21.15.33:0/592342961 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff84c03a020 con 0x55911053f5d0 2024-09-16T00:27:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff864560740 10 client.26785 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff864560740 10 client.26785 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:27:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff864560740 20 client.26785 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff864560740 1 --2- 172.21.15.33:0/592342961 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5591105c5190 0x5591105c7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff864560740 4 client.26785 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:27:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff864560740 1 -- 172.21.15.33:0/592342961 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5591104acd70 con 0x5591105c5190 2024-09-16T00:27:56.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.630+0000 7ff85affd640 1 --2- 172.21.15.33:0/592342961 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5591105c5190 0x5591105c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.631+0000 7ff83affd640 10 client.26785 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:27:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.631+0000 7ff85affd640 1 --2- 172.21.15.33:0/592342961 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5591105c5190 0x5591105c7580 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cc69e51546dc6d1 server_cookie=fc9c63dd0c52ca22 in_seq=0 out_seq=0 2024-09-16T00:27:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.631+0000 7ff83affd640 1 -- 172.21.15.33:0/592342961 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5591104acd70 con 0x5591105c5190 2024-09-16T00:27:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.631+0000 7ff83affd640 10 client.26785 handle_command_reply: tid=0 2024-09-16T00:27:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.631+0000 7ff864560740 1 client.26785 shutdown 2024-09-16T00:27:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff864560740 1 -- 172.21.15.33:0/592342961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911053f5d0 msgr2=0x559110543e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff864560740 1 --2- 172.21.15.33:0/592342961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911053f5d0 0x559110543e90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff84c01efd0 tx=0x7ff84c005730 comp rx=0 tx=0).stop 2024-09-16T00:27:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff864560740 1 -- 172.21.15.33:0/592342961 shutdown_connections 2024-09-16T00:27:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff864560740 1 --2- 172.21.15.33:0/592342961 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5591105c5190 0x5591105c7580 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff864560740 1 --2- 172.21.15.33:0/592342961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55911054a370 0x559110544910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff864560740 1 --2- 172.21.15.33:0/592342961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559110540b40 0x5591105443d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff83affd640 0 client.26785 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:27:56.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff864560740 1 --2- 172.21.15.33:0/592342961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55911053f5d0 0x559110543e90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff864560740 1 -- 172.21.15.33:0/592342961 >> 172.21.15.33:0/592342961 conn(0x559110508310 msgr2=0x559110549010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:56.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff864560740 1 -- 172.21.15.33:0/592342961 shutdown_connections 2024-09-16T00:27:56.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff864560740 1 -- 172.21.15.33:0/592342961 wait complete. 2024-09-16T00:27:56.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff864560740 20 client.26785 trim_cache size 0 max 16384 2024-09-16T00:27:56.638 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:27:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff862187640 1 -- 172.21.15.33:0/4069871882 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8440a32c0 msgr2=0x7ff8440a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff862187640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8440a32c0 0x7ff8440a5780 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x55911053cac0 tx=0x7ff84c005f90 comp rx=0 tx=0).stop 2024-09-16T00:27:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff862187640 1 -- 172.21.15.33:0/4069871882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff85c103780 msgr2=0x7ff85c1a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff862187640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff85c103780 0x7ff85c1a4e00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0697d0 tx=0x7ff85000ee50 comp rx=0 tx=0).stop 2024-09-16T00:27:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff862187640 1 -- 172.21.15.33:0/4069871882 shutdown_connections 2024-09-16T00:27:56.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff862187640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8440a32c0 0x7ff8440a5780 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff862187640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff85c108300 0x7ff85c10de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff862187640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff85c103780 0x7ff85c1a4e00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff862187640 1 --2- 172.21.15.33:0/4069871882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff85c102dd0 0x7ff85c1a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:56.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.632+0000 7ff862187640 1 -- 172.21.15.33:0/4069871882 >> 172.21.15.33:0/4069871882 conn(0x7ff85c0fc7d0 msgr2=0x7ff85c103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:56.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.633+0000 7ff862187640 1 -- 172.21.15.33:0/4069871882 shutdown_connections 2024-09-16T00:27:56.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:56.633+0000 7ff862187640 1 -- 172.21.15.33:0/4069871882 wait complete. 2024-09-16T00:27:56.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 905566 10.05 MB/sec execute 424 sec latency 39.896 ms 2024-09-16T00:27:57.416 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:27:57.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:57 smithi033 ceph-mon[31243]: pgmap v674: 129 pgs: 129 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 296 op/s 2024-09-16T00:27:57.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:57 smithi083 ceph-mon[35242]: pgmap v674: 129 pgs: 129 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 296 op/s 2024-09-16T00:27:57.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 908288 10.06 MB/sec execute 425 sec latency 12.039 ms 2024-09-16T00:27:57.691 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:57.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:57 smithi060 ceph-mon[35185]: pgmap v674: 129 pgs: 129 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 296 op/s 2024-09-16T00:27:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.200+0000 7f5070085640 1 -- 172.21.15.33:0/580461540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068101e10 msgr2=0x7f50681021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.200+0000 7f5070085640 1 --2- 172.21.15.33:0/580461540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068101e10 0x7f50681021f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5064009900 tx=0x7f506402a410 comp rx=0 tx=0).stop 2024-09-16T00:27:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.200+0000 7f5070085640 1 -- 172.21.15.33:0/580461540 shutdown_connections 2024-09-16T00:27:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.200+0000 7f5070085640 1 --2- 172.21.15.33:0/580461540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5068106f40 0x7f506810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.200+0000 7f5070085640 1 --2- 172.21.15.33:0/580461540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50681027c0 0x7f5068106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.200+0000 7f5070085640 1 --2- 172.21.15.33:0/580461540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068101e10 0x7f50681021f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.200+0000 7f5070085640 1 -- 172.21.15.33:0/580461540 >> 172.21.15.33:0/580461540 conn(0x7f50680fd8d0 msgr2=0x7f50680ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.200+0000 7f5070085640 1 -- 172.21.15.33:0/580461540 shutdown_connections 2024-09-16T00:27:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.200+0000 7f5070085640 1 -- 172.21.15.33:0/580461540 wait complete. 2024-09-16T00:27:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f5070085640 1 Processor -- start 2024-09-16T00:27:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f5070085640 1 -- start start 2024-09-16T00:27:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f5070085640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068101e10 0x7f5068198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f5070085640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50681027c0 0x7f50681989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f5070085640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5068106f40 0x7f506819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f5070085640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5068111fa0 con 0x7f5068106f40 2024-09-16T00:27:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f5070085640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5068111e20 con 0x7f50681027c0 2024-09-16T00:27:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f5070085640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5068112120 con 0x7f5068101e10 2024-09-16T00:27:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f506de21640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068101e10 0x7f5068198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f506de21640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068101e10 0x7f5068198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42140/0 (socket says 172.21.15.33:42140) 2024-09-16T00:27:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f506de21640 1 -- 172.21.15.33:0/2499325811 learned_addr learned my addr 172.21.15.33:0/2499325811 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:58.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f506e622640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5068106f40 0x7f506819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.201+0000 7f506d620640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50681027c0 0x7f50681989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.202+0000 7f506de21640 1 -- 172.21.15.33:0/2499325811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50681027c0 msgr2=0x7f50681989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.202+0000 7f506de21640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50681027c0 0x7f50681989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.202+0000 7f506de21640 1 -- 172.21.15.33:0/2499325811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5068106f40 msgr2=0x7f506819cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.202+0000 7f506de21640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5068106f40 0x7f506819cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.202+0000 7f506de21640 1 -- 172.21.15.33:0/2499325811 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506819d410 con 0x7f5068101e10 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.202+0000 7f506d620640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50681027c0 0x7f50681989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.202+0000 7f506de21640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068101e10 0x7f5068198460 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5064006010 tx=0x7f5064004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.202+0000 7f505effd640 1 -- 172.21.15.33:0/2499325811 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506403f030 con 0x7f5068101e10 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.202+0000 7f505effd640 1 -- 172.21.15.33:0/2499325811 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5064004440 con 0x7f5068101e10 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.202+0000 7f5070085640 1 -- 172.21.15.33:0/2499325811 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f506819d6a0 con 0x7f5068101e10 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.202+0000 7f505effd640 1 -- 172.21.15.33:0/2499325811 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5064035070 con 0x7f5068101e10 2024-09-16T00:27:58.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.202+0000 7f5070085640 1 -- 172.21.15.33:0/2499325811 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50681a4fe0 con 0x7f5068101e10 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.205+0000 7f505effd640 1 -- 172.21.15.33:0/2499325811 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5064042030 con 0x7f5068101e10 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.205+0000 7f507245e740 1 Processor -- start 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.205+0000 7f507245e740 1 -- start start 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.205+0000 7f507245e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92afab40 0x564e92af9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.205+0000 7f507245e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92af95d0 0x564e92af9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.205+0000 7f507245e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92b04370 0x564e92b06760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.205+0000 7f507245e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564e92a5f2f0 con 0x564e92b04370 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.205+0000 7f507245e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564e92a06cf0 con 0x564e92afab40 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.205+0000 7f507245e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564e92a5b2b0 con 0x564e92af95d0 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f506de21640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92af95d0 0x564e92af9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f506e622640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92afab40 0x564e92af9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f506e622640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92afab40 0x564e92af9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:52868/0 (socket says 172.21.15.33:52868) 2024-09-16T00:27:58.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f506e622640 1 -- 172.21.15.33:0/4047115152 learned_addr learned my addr 172.21.15.33:0/4047115152 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f506d620640 1 --2- 172.21.15.33:0/4047115152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92b04370 0x564e92b06760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f505effd640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f50540a3240 0x7f50540a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f506d620640 1 -- 172.21.15.33:0/4047115152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92af95d0 msgr2=0x564e92af9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f506d620640 1 --2- 172.21.15.33:0/4047115152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92af95d0 0x564e92af9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f506d620640 1 -- 172.21.15.33:0/4047115152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92afab40 msgr2=0x564e92af9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f506d620640 1 --2- 172.21.15.33:0/4047115152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92afab40 0x564e92af9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f506d620640 1 -- 172.21.15.33:0/4047115152 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e929eb3a0 con 0x564e92b04370 2024-09-16T00:27:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f506e622640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f50540a3240 0x7f50540a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f505effd640 1 -- 172.21.15.33:0/2499325811 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f50640f00f0 con 0x7f5068101e10 2024-09-16T00:27:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f506d620640 1 --2- 172.21.15.33:0/4047115152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92b04370 0x564e92b06760 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5058009fa0 tx=0x7f50580118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c6d40429fb12fc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:58.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f5053fff640 1 -- 172.21.15.33:0/4047115152 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5058004030 con 0x564e92b04370 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f5053fff640 1 -- 172.21.15.33:0/4047115152 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5058005500 con 0x564e92b04370 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.206+0000 7f5053fff640 1 -- 172.21.15.33:0/4047115152 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5058004db0 con 0x564e92b04370 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.207+0000 7f507245e740 1 -- 172.21.15.33:0/4047115152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92b04370 msgr2=0x564e92b06760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.207+0000 7f507245e740 1 --2- 172.21.15.33:0/4047115152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92b04370 0x564e92b06760 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5058009fa0 tx=0x7f50580118d0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.207+0000 7f506e622640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f50540a3240 0x7f50540a5700 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5060010610 tx=0x7f5060004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.207+0000 7f507245e740 1 -- 172.21.15.33:0/4047115152 shutdown_connections 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.207+0000 7f507245e740 1 --2- 172.21.15.33:0/4047115152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92b04370 0x564e92b06760 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.207+0000 7f507245e740 1 --2- 172.21.15.33:0/4047115152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92af95d0 0x564e92af9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.207+0000 7f507245e740 1 --2- 172.21.15.33:0/4047115152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92afab40 0x564e92af9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.207+0000 7f507245e740 1 -- 172.21.15.33:0/4047115152 >> 172.21.15.33:0/4047115152 conn(0x564e92ac2310 msgr2=0x564e92af5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.207+0000 7f507245e740 1 -- 172.21.15.33:0/4047115152 shutdown_connections 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.207+0000 7f507245e740 1 -- 172.21.15.33:0/4047115152 wait complete. 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f507245e740 1 -- start start 2024-09-16T00:27:58.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f507245e740 1 Processor -- start 2024-09-16T00:27:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f507245e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 0x564e92b03d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f507245e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92afab40 0x564e92b01010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f507245e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92b04370 0x564e92b01600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f507245e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564e92b07070 con 0x564e92af95d0 2024-09-16T00:27:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f507245e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564e92a1a060 con 0x564e92b04370 2024-09-16T00:27:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f507245e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564e92b071f0 con 0x564e92afab40 2024-09-16T00:27:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f507245e740 10 client.0 _ll_register_callbacks cb 0x564e9276d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f506de21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 0x564e92b03d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f506de21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 0x564e92b03d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55022/0 (socket says 172.21.15.33:55022) 2024-09-16T00:27:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f506de21640 1 -- 172.21.15.33:0/210718874 learned_addr learned my addr 172.21.15.33:0/210718874 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.208+0000 7f506d620640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92afab40 0x564e92b01010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f506e622640 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92b04370 0x564e92b01600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f506de21640 1 -- 172.21.15.33:0/210718874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92afab40 msgr2=0x564e92b01010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f506de21640 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92afab40 0x564e92b01010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f506de21640 1 -- 172.21.15.33:0/210718874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92b04370 msgr2=0x564e92b01600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f506de21640 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92b04370 0x564e92b01600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f506de21640 1 -- 172.21.15.33:0/210718874 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e927d03c0 con 0x564e92af95d0 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f506d620640 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92afab40 0x564e92b01010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f506e622640 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92b04370 0x564e92b01600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f506de21640 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 0x564e92b03d60 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f50640b8d10 tx=0x7f5064005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166a3a64d696e8d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f50517fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f50517fa640 1 -- 172.21.15.33:0/210718874 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5064049050 con 0x564e92af95d0 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f50517fa640 1 -- 172.21.15.33:0/210718874 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5064003cd0 con 0x564e92af95d0 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f507245e740 1 -- 172.21.15.33:0/210718874 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5058005500 con 0x564e92af95d0 2024-09-16T00:27:58.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f50517fa640 1 -- 172.21.15.33:0/210718874 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f50640bbe80 con 0x564e92af95d0 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f507245e740 10 client.17400 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f507245e740 1 -- 172.21.15.33:0/210718874 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564e92b12b10 con 0x564e92af95d0 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.209+0000 7f50517fa640 1 -- 172.21.15.33:0/210718874 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5058005500 con 0x564e92af95d0 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.210+0000 7f50517fa640 1 -- 172.21.15.33:0/210718874 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f50640bc5a0 con 0x564e92af95d0 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.210+0000 7f507245e740 10 client.17400 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.210+0000 7f507245e740 10 client.17400 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.210+0000 7f507245e740 20 client.17400 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.210+0000 7f507245e740 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e92b13eb0 0x564e92b162c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.210+0000 7f507245e740 4 client.17400 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.210+0000 7f507245e740 1 -- 172.21.15.33:0/210718874 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e92a6c860 con 0x564e92b13eb0 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.210+0000 7f506d620640 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e92b13eb0 0x564e92b162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.210+0000 7f50517fa640 10 client.17400 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:27:58.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.210+0000 7f506d620640 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e92b13eb0 0x564e92b162c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c86e8ad78103c6e0 server_cookie=1b9dea124a6d7ed4 in_seq=0 out_seq=0 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f50517fa640 1 -- 172.21.15.33:0/210718874 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564e92a6c860 con 0x564e92b13eb0 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f50517fa640 10 client.17400 handle_command_reply: tid=0 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f507245e740 1 client.17400 shutdown 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f507245e740 1 -- 172.21.15.33:0/210718874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 msgr2=0x564e92b03d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f507245e740 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 0x564e92b03d60 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f50640b8d10 tx=0x7f5064005c90 comp rx=0 tx=0).stop 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f507245e740 1 -- 172.21.15.33:0/210718874 shutdown_connections 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f507245e740 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e92b13eb0 0x564e92b162c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f507245e740 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92b04370 0x564e92b01600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f507245e740 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92afab40 0x564e92b01010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f507245e740 1 --2- 172.21.15.33:0/210718874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 0x564e92b03d60 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f507245e740 1 -- 172.21.15.33:0/210718874 >> 172.21.15.33:0/210718874 conn(0x564e92ac2310 msgr2=0x564e92afb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f50517fa640 0 client.17400 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:27:58.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f507245e740 1 -- 172.21.15.33:0/210718874 shutdown_connections 2024-09-16T00:27:58.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f507245e740 1 -- 172.21.15.33:0/210718874 wait complete. 2024-09-16T00:27:58.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.211+0000 7f507245e740 20 client.17400 trim_cache size 0 max 16384 2024-09-16T00:27:58.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.225+0000 7f507245e740 1 Processor -- start 2024-09-16T00:27:58.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.225+0000 7f507245e740 1 -- start start 2024-09-16T00:27:58.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.225+0000 7f507245e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 0x564e92af5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.225+0000 7f507245e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92afab40 0x564e92b10310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.225+0000 7f507245e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92b04370 0x564e92afc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.225+0000 7f507245e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50681b1a00 con 0x564e92af95d0 2024-09-16T00:27:58.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.225+0000 7f507245e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f50681af2d0 con 0x564e92afab40 2024-09-16T00:27:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.225+0000 7f507245e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f50681ad120 con 0x564e92b04370 2024-09-16T00:27:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.225+0000 7f506de21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 0x564e92af5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.225+0000 7f506de21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 0x564e92af5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55038/0 (socket says 172.21.15.33:55038) 2024-09-16T00:27:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.225+0000 7f506de21640 1 -- 172.21.15.33:0/269061221 learned_addr learned my addr 172.21.15.33:0/269061221 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.225+0000 7f506e622640 1 --2- 172.21.15.33:0/269061221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92b04370 0x564e92afc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.226+0000 7f506d620640 1 --2- 172.21.15.33:0/269061221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92afab40 0x564e92b10310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.226+0000 7f506de21640 1 -- 172.21.15.33:0/269061221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92b04370 msgr2=0x564e92afc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.226+0000 7f506de21640 1 --2- 172.21.15.33:0/269061221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92b04370 0x564e92afc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.226+0000 7f506de21640 1 -- 172.21.15.33:0/269061221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92afab40 msgr2=0x564e92b10310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.226+0000 7f506de21640 1 --2- 172.21.15.33:0/269061221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92afab40 0x564e92b10310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.226+0000 7f506de21640 1 -- 172.21.15.33:0/269061221 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e92b04090 con 0x564e92af95d0 2024-09-16T00:27:58.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.226+0000 7f506de21640 1 --2- 172.21.15.33:0/269061221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 0x564e92af5e10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5064000c00 tx=0x7f506402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36017e5b601813a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.226+0000 7f50537fe640 1 -- 172.21.15.33:0/269061221 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5064049070 con 0x564e92af95d0 2024-09-16T00:27:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.226+0000 7f50537fe640 1 -- 172.21.15.33:0/269061221 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50640c0640 con 0x564e92af95d0 2024-09-16T00:27:58.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.226+0000 7f50537fe640 1 -- 172.21.15.33:0/269061221 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50640bbca0 con 0x564e92af95d0 2024-09-16T00:27:58.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.227+0000 7f507245e740 1 -- 172.21.15.33:0/269061221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 msgr2=0x564e92af5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:58.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.227+0000 7f507245e740 1 --2- 172.21.15.33:0/269061221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 0x564e92af5e10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5064000c00 tx=0x7f506402a410 comp rx=0 tx=0).stop 2024-09-16T00:27:58.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.227+0000 7f507245e740 1 -- 172.21.15.33:0/269061221 shutdown_connections 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.227+0000 7f507245e740 1 --2- 172.21.15.33:0/269061221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92b04370 0x564e92afc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.227+0000 7f507245e740 1 --2- 172.21.15.33:0/269061221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92afab40 0x564e92b10310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.227+0000 7f507245e740 1 --2- 172.21.15.33:0/269061221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92af95d0 0x564e92af5e10 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.227+0000 7f507245e740 1 -- 172.21.15.33:0/269061221 >> 172.21.15.33:0/269061221 conn(0x564e92ac2310 msgr2=0x564e92af5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.227+0000 7f507245e740 1 -- 172.21.15.33:0/269061221 shutdown_connections 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.227+0000 7f507245e740 1 -- 172.21.15.33:0/269061221 wait complete. 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.227+0000 7f507245e740 1 -- start start 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.227+0000 7f507245e740 1 Processor -- start 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f507245e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92af95d0 0x564e92b0c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f507245e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92afab40 0x564e92b0cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f507245e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92b04370 0x564e92b07de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f507245e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50681ad790 con 0x564e92afab40 2024-09-16T00:27:58.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f507245e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f50681aec00 con 0x564e92b04370 2024-09-16T00:27:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f507245e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f50681ade60 con 0x564e92af95d0 2024-09-16T00:27:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f507245e740 10 client.0 _ll_register_callbacks cb 0x7f50640f38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f506de21640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92af95d0 0x564e92b0c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f506d620640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92afab40 0x564e92b0cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f506de21640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92af95d0 0x564e92b0c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42150/0 (socket says 172.21.15.33:42150) 2024-09-16T00:27:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f506de21640 1 -- 172.21.15.33:0/2159122009 learned_addr learned my addr 172.21.15.33:0/2159122009 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f506d620640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92afab40 0x564e92b0cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55040/0 (socket says 172.21.15.33:55040) 2024-09-16T00:27:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f506e622640 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92b04370 0x564e92b07de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f506de21640 1 -- 172.21.15.33:0/2159122009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92b04370 msgr2=0x564e92b07de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f506de21640 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92b04370 0x564e92b07de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f506de21640 1 -- 172.21.15.33:0/2159122009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92afab40 msgr2=0x564e92b0cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:58.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f506de21640 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92afab40 0x564e92b0cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f506de21640 1 -- 172.21.15.33:0/2159122009 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e92b08610 con 0x564e92af95d0 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.228+0000 7f506e622640 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92b04370 0x564e92b07de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f506de21640 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92af95d0 0x564e92b0c860 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f50640bc040 tx=0x7f5064031e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96c3b5443a56d97e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f50517fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f50517fa640 1 -- 172.21.15.33:0/2159122009 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50640d5070 con 0x564e92af95d0 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f50517fa640 1 -- 172.21.15.33:0/2159122009 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f50640d5210 con 0x564e92af95d0 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f507245e740 1 -- 172.21.15.33:0/2159122009 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50640c0640 con 0x564e92af95d0 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f50517fa640 1 -- 172.21.15.33:0/2159122009 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5064049050 con 0x564e92af95d0 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f507245e740 10 client.26803 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f507245e740 1 -- 172.21.15.33:0/2159122009 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564e92b08610 con 0x564e92af95d0 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f50517fa640 1 -- 172.21.15.33:0/2159122009 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50640c0640 con 0x564e92af95d0 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f50517fa640 1 -- 172.21.15.33:0/2159122009 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f50640c0140 con 0x564e92af95d0 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f507245e740 10 client.26803 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:58.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f507245e740 10 client.26803 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f507245e740 20 client.26803 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f507245e740 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e92b82fb0 0x564e92b853a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f507245e740 4 client.26803 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.229+0000 7f507245e740 1 -- 172.21.15.33:0/2159122009 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e92a66d70 con 0x564e92b82fb0 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.230+0000 7f506d620640 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e92b82fb0 0x564e92b853a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.230+0000 7f506d620640 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e92b82fb0 0x564e92b853a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6ae1645cbb840e1d server_cookie=8a933eb467a8d75d in_seq=0 out_seq=0 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.230+0000 7f50517fa640 10 client.26803 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.230+0000 7f50517fa640 1 -- 172.21.15.33:0/2159122009 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e92a66d70 con 0x564e92b82fb0 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.230+0000 7f50517fa640 10 client.26803 handle_command_reply: tid=0 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.230+0000 7f507245e740 1 client.26803 shutdown 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.230+0000 7f507245e740 1 -- 172.21.15.33:0/2159122009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92af95d0 msgr2=0x564e92b0c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.230+0000 7f507245e740 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92af95d0 0x564e92b0c860 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f50640bc040 tx=0x7f5064031e60 comp rx=0 tx=0).stop 2024-09-16T00:27:58.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.230+0000 7f507245e740 1 -- 172.21.15.33:0/2159122009 shutdown_connections 2024-09-16T00:27:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.230+0000 7f507245e740 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564e92b82fb0 0x564e92b853a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.231+0000 7f507245e740 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564e92b04370 0x564e92b07de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.231+0000 7f50517fa640 0 client.26803 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:27:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.231+0000 7f507245e740 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e92afab40 0x564e92b0cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.231+0000 7f507245e740 1 --2- 172.21.15.33:0/2159122009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e92af95d0 0x564e92b0c860 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.231+0000 7f507245e740 1 -- 172.21.15.33:0/2159122009 >> 172.21.15.33:0/2159122009 conn(0x564e92ac2310 msgr2=0x564e92b03090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.231+0000 7f507245e740 1 -- 172.21.15.33:0/2159122009 shutdown_connections 2024-09-16T00:27:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.231+0000 7f507245e740 1 -- 172.21.15.33:0/2159122009 wait complete. 2024-09-16T00:27:58.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.231+0000 7f507245e740 20 client.26803 trim_cache size 0 max 16384 2024-09-16T00:27:58.237 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:27:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.231+0000 7f5070085640 1 -- 172.21.15.33:0/2499325811 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f50540a3240 msgr2=0x7f50540a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.231+0000 7f5070085640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f50540a3240 0x7f50540a5700 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5060010610 tx=0x7f5060004a40 comp rx=0 tx=0).stop 2024-09-16T00:27:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.231+0000 7f5070085640 1 -- 172.21.15.33:0/2499325811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068101e10 msgr2=0x7f5068198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:58.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.231+0000 7f5070085640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068101e10 0x7f5068198460 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5064006010 tx=0x7f5064004040 comp rx=0 tx=0).stop 2024-09-16T00:27:58.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.232+0000 7f5070085640 1 -- 172.21.15.33:0/2499325811 shutdown_connections 2024-09-16T00:27:58.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.232+0000 7f5070085640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f50540a3240 0x7f50540a5700 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.232+0000 7f5070085640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5068106f40 0x7f506819cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.232+0000 7f5070085640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50681027c0 0x7f50681989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.232+0000 7f5070085640 1 --2- 172.21.15.33:0/2499325811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068101e10 0x7f5068198460 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:58.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.232+0000 7f5070085640 1 -- 172.21.15.33:0/2499325811 >> 172.21.15.33:0/2499325811 conn(0x7f50680fd8d0 msgr2=0x7f50680fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:58.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.232+0000 7f5070085640 1 -- 172.21.15.33:0/2499325811 shutdown_connections 2024-09-16T00:27:58.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:58.232+0000 7f5070085640 1 -- 172.21.15.33:0/2499325811 wait complete. 2024-09-16T00:27:58.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 910498 10.06 MB/sec execute 426 sec latency 20.237 ms 2024-09-16T00:27:59.058 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:27:59.327 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:27:59.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:27:59 smithi033 ceph-mon[31243]: pgmap v675: 129 pgs: 129 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 243 op/s 2024-09-16T00:27:59.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:27:59 smithi083 ceph-mon[35242]: pgmap v675: 129 pgs: 129 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 243 op/s 2024-09-16T00:27:59.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 913104 10.07 MB/sec execute 427 sec latency 33.053 ms 2024-09-16T00:27:59.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:27:59 smithi060 ceph-mon[35185]: pgmap v675: 129 pgs: 129 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 243 op/s 2024-09-16T00:27:59.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.849+0000 7f100ca52640 1 -- 172.21.15.33:0/1136944619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1008101360 msgr2=0x7f100810df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:59.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.849+0000 7f100ca52640 1 --2- 172.21.15.33:0/1136944619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1008101360 0x7f100810df90 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009900 tx=0x7f0ff002a410 comp rx=0 tx=0).stop 2024-09-16T00:27:59.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.849+0000 7f100ca52640 1 -- 172.21.15.33:0/1136944619 shutdown_connections 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.849+0000 7f100ca52640 1 --2- 172.21.15.33:0/1136944619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f100810e4d0 0x7f10081108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.849+0000 7f100ca52640 1 --2- 172.21.15.33:0/1136944619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1008101360 0x7f100810df90 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.849+0000 7f100ca52640 1 --2- 172.21.15.33:0/1136944619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10081009b0 0x7f1008100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.849+0000 7f100ca52640 1 -- 172.21.15.33:0/1136944619 >> 172.21.15.33:0/1136944619 conn(0x7f10080fc7d0 msgr2=0x7f10080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.849+0000 7f100ca52640 1 -- 172.21.15.33:0/1136944619 shutdown_connections 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.849+0000 7f100ca52640 1 -- 172.21.15.33:0/1136944619 wait complete. 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.850+0000 7f100ca52640 1 Processor -- start 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.850+0000 7f100ca52640 1 -- start start 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.850+0000 7f100ca52640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10081009b0 0x7f10081a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.850+0000 7f100ca52640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1008101360 0x7f10081a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.850+0000 7f100ca52640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f100810e4d0 0x7f100819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.850+0000 7f100ca52640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1008114200 con 0x7f100810e4d0 2024-09-16T00:27:59.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.850+0000 7f100ca52640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1008114080 con 0x7f10081009b0 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.850+0000 7f100ca52640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1008114380 con 0x7f1008101360 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.850+0000 7f1006d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f100810e4d0 0x7f100819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f100659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10081009b0 0x7f10081a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f0ffffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1008101360 0x7f10081a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f100659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10081009b0 0x7f10081a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:52910/0 (socket says 172.21.15.33:52910) 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f1006d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f100810e4d0 0x7f100819a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55052/0 (socket says 172.21.15.33:55052) 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f100659c640 1 -- 172.21.15.33:0/3438875253 learned_addr learned my addr 172.21.15.33:0/3438875253 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f100659c640 1 -- 172.21.15.33:0/3438875253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1008101360 msgr2=0x7f10081a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f100659c640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1008101360 0x7f10081a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f100659c640 1 -- 172.21.15.33:0/3438875253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f100810e4d0 msgr2=0x7f100819a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f100659c640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f100810e4d0 0x7f100819a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f100659c640 1 -- 172.21.15.33:0/3438875253 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100819acc0 con 0x7f10081009b0 2024-09-16T00:27:59.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f1006d9d640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f100810e4d0 0x7f100819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f100659c640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10081009b0 0x7f10081a03d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ff800ea60 tx=0x7f0ff800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f0fff7fe640 1 -- 172.21.15.33:0/3438875253 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff800cde0 con 0x7f10081009b0 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f100ca52640 1 -- 172.21.15.33:0/3438875253 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f100819afb0 con 0x7f10081009b0 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f0fff7fe640 1 -- 172.21.15.33:0/3438875253 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ff80035f0 con 0x7f10081009b0 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f0fff7fe640 1 -- 172.21.15.33:0/3438875253 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8010670 con 0x7f10081009b0 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.851+0000 7f100ca52640 1 -- 172.21.15.33:0/3438875253 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10081a72a0 con 0x7f10081009b0 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.854+0000 7f0fff7fe640 1 -- 172.21.15.33:0/3438875253 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0ff80037a0 con 0x7f10081009b0 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.854+0000 7f100ee2b740 1 Processor -- start 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.854+0000 7f100ee2b740 1 -- start start 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.854+0000 7f100ee2b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38edb40 0x5565a38ec090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.854+0000 7f100ee2b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a38eca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.854+0000 7f100ee2b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38f7370 0x5565a38f9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.854+0000 7f100ee2b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565a38522f0 con 0x5565a38ec5d0 2024-09-16T00:27:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.855+0000 7f100ee2b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5565a37f9cf0 con 0x5565a38edb40 2024-09-16T00:27:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.855+0000 7f100ee2b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5565a384e2b0 con 0x5565a38f7370 2024-09-16T00:27:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.855+0000 7f1006d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38edb40 0x5565a38ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.855+0000 7f100659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a38eca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.855+0000 7f0fff7fe640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ff40a32c0 0x7f0ff40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.855+0000 7f1006d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38edb40 0x5565a38ec090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:52920/0 (socket says 172.21.15.33:52920) 2024-09-16T00:27:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.855+0000 7f1006d9d640 1 -- 172.21.15.33:0/894842224 learned_addr learned my addr 172.21.15.33:0/894842224 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.855+0000 7f0ffffff640 1 --2- 172.21.15.33:0/894842224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38f7370 0x5565a38f9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.855+0000 7f1006d9d640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ff40a32c0 0x7f0ff40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.855+0000 7f0fff7fe640 1 -- 172.21.15.33:0/3438875253 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0ff80d0080 con 0x7f10081009b0 2024-09-16T00:27:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.856+0000 7f1006d9d640 1 -- 172.21.15.33:0/894842224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38f7370 msgr2=0x5565a38f9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.856+0000 7f1006d9d640 1 --2- 172.21.15.33:0/894842224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38f7370 0x5565a38f9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.856+0000 7f1006d9d640 1 -- 172.21.15.33:0/894842224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 msgr2=0x5565a38eca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.856+0000 7f1006d9d640 1 --2- 172.21.15.33:0/894842224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a38eca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.856+0000 7f1006d9d640 1 -- 172.21.15.33:0/894842224 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565a37de3a0 con 0x5565a38edb40 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.856+0000 7f0ffffff640 1 --2- 172.21.15.33:0/894842224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38f7370 0x5565a38f9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.856+0000 7f100659c640 1 --2- 172.21.15.33:0/894842224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a38eca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.857+0000 7f1006d9d640 1 --2- 172.21.15.33:0/894842224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38edb40 0x5565a38ec090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1000012e60 tx=0x7f100000e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cae21e2bdad09d51 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.857+0000 7f0ffcff9640 1 -- 172.21.15.33:0/894842224 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000022070 con 0x5565a38edb40 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.857+0000 7f1006d9d640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ff40a32c0 0x7f0ff40a5780 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f100001b000 tx=0x7f100001d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.857+0000 7f0ffcff9640 1 -- 172.21.15.33:0/894842224 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1000003e50 con 0x5565a38edb40 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.857+0000 7f0ffcff9640 1 -- 172.21.15.33:0/894842224 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100001d400 con 0x5565a38edb40 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.858+0000 7f100ee2b740 1 -- 172.21.15.33:0/894842224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38edb40 msgr2=0x5565a38ec090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.858+0000 7f100ee2b740 1 --2- 172.21.15.33:0/894842224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38edb40 0x5565a38ec090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1000012e60 tx=0x7f100000e720 comp rx=0 tx=0).stop 2024-09-16T00:27:59.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.858+0000 7f100ee2b740 1 -- 172.21.15.33:0/894842224 shutdown_connections 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.858+0000 7f100ee2b740 1 --2- 172.21.15.33:0/894842224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38f7370 0x5565a38f9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.858+0000 7f100ee2b740 1 --2- 172.21.15.33:0/894842224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a38eca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.858+0000 7f100ee2b740 1 --2- 172.21.15.33:0/894842224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38edb40 0x5565a38ec090 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.858+0000 7f100ee2b740 1 -- 172.21.15.33:0/894842224 >> 172.21.15.33:0/894842224 conn(0x5565a38b5310 msgr2=0x5565a38e8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.858+0000 7f100ee2b740 1 -- 172.21.15.33:0/894842224 shutdown_connections 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.858+0000 7f100ee2b740 1 -- 172.21.15.33:0/894842224 wait complete. 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.858+0000 7f100ee2b740 1 -- start start 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100ee2b740 1 Processor -- start 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100ee2b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38ec5d0 0x5565a38f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100ee2b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38edb40 0x5565a38f4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100ee2b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38f7370 0x5565a38f4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100ee2b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565a38fa010 con 0x5565a38f7370 2024-09-16T00:27:59.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100ee2b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5565a380d060 con 0x5565a38edb40 2024-09-16T00:27:59.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100ee2b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5565a38fa190 con 0x5565a38ec5d0 2024-09-16T00:27:59.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38ec5d0 0x5565a38f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38ec5d0 0x5565a38f6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42200/0 (socket says 172.21.15.33:42200) 2024-09-16T00:27:59.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f0ffffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38edb40 0x5565a38f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100659c640 1 -- 172.21.15.33:0/3532954774 learned_addr learned my addr 172.21.15.33:0/3532954774 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:59.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f1006d9d640 1 --2- 172.21.15.33:0/3532954774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38f7370 0x5565a38f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100ee2b740 10 client.0 _ll_register_callbacks cb 0x5565a35608c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:59.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100659c640 1 -- 172.21.15.33:0/3532954774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38edb40 msgr2=0x5565a38f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:59.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100659c640 1 --2- 172.21.15.33:0/3532954774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38edb40 0x5565a38f4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100659c640 1 -- 172.21.15.33:0/3532954774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38f7370 msgr2=0x5565a38f4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:59.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100659c640 1 --2- 172.21.15.33:0/3532954774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38f7370 0x5565a38f4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100659c640 1 -- 172.21.15.33:0/3532954774 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565a35c33c0 con 0x5565a38ec5d0 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.859+0000 7f100ee2b740 1 -- 172.21.15.33:0/3532954774 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1000003e50 con 0x5565a38ec5d0 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f100659c640 1 --2- 172.21.15.33:0/3532954774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38ec5d0 0x5565a38f6dd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0ff801f000 tx=0x7f0ff8097e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29cb7b7bb502925e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f0fedffb640 10 client.26815 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f0fedffb640 1 -- 172.21.15.33:0/3532954774 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff809c8c0 con 0x5565a38ec5d0 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f0fedffb640 1 -- 172.21.15.33:0/3532954774 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0ff80cd7b0 con 0x5565a38ec5d0 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f0fedffb640 1 -- 172.21.15.33:0/3532954774 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0ff80a0570 con 0x5565a38ec5d0 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f100ee2b740 10 client.26815 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f100ee2b740 1 -- 172.21.15.33:0/3532954774 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({fsmap=30}) -- 0x5565a3905a50 con 0x5565a38ec5d0 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f0fedffb640 1 -- 172.21.15.33:0/3532954774 <== mon.1 v2:172.21.15.83:3300/0 4 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f0ff8098830 con 0x5565a38ec5d0 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f100ee2b740 10 client.26815 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f100ee2b740 10 client.26815 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f100ee2b740 20 client.26815 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:59.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f100ee2b740 1 --2- 172.21.15.33:0/3532954774 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5565a3906e00 0x5565a3909290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f100ee2b740 4 client.26815 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.860+0000 7f100ee2b740 1 -- 172.21.15.33:0/3532954774 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565a385f860 con 0x5565a3906e00 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.861+0000 7f0ffffff640 1 --2- 172.21.15.33:0/3532954774 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5565a3906e00 0x5565a3909290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.861+0000 7f0fedffb640 10 client.26815 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.861+0000 7f0ffffff640 1 --2- 172.21.15.33:0/3532954774 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5565a3906e00 0x5565a3909290 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=247b128e0612afc4 server_cookie=74dacd95be74ef78 in_seq=0 out_seq=0 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.861+0000 7f0fedffb640 1 -- 172.21.15.33:0/3532954774 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5565a385f860 con 0x5565a3906e00 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.861+0000 7f0fedffb640 10 client.26815 handle_command_reply: tid=0 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f100ee2b740 1 client.26815 shutdown 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f100ee2b740 1 -- 172.21.15.33:0/3532954774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38ec5d0 msgr2=0x5565a38f6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f100ee2b740 1 --2- 172.21.15.33:0/3532954774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38ec5d0 0x5565a38f6dd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0ff801f000 tx=0x7f0ff8097e10 comp rx=0 tx=0).stop 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f100ee2b740 1 -- 172.21.15.33:0/3532954774 shutdown_connections 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f100ee2b740 1 --2- 172.21.15.33:0/3532954774 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5565a3906e00 0x5565a3909290 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f100ee2b740 1 --2- 172.21.15.33:0/3532954774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38f7370 0x5565a38f4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f100ee2b740 1 --2- 172.21.15.33:0/3532954774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38edb40 0x5565a38f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f0fedffb640 0 client.26815 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:59.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f100ee2b740 1 --2- 172.21.15.33:0/3532954774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38ec5d0 0x5565a38f6dd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f100ee2b740 1 -- 172.21.15.33:0/3532954774 >> 172.21.15.33:0/3532954774 conn(0x5565a38b5310 msgr2=0x5565a38ee960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:59.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f100ee2b740 1 -- 172.21.15.33:0/3532954774 shutdown_connections 2024-09-16T00:27:59.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f100ee2b740 1 -- 172.21.15.33:0/3532954774 wait complete. 2024-09-16T00:27:59.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.862+0000 7f100ee2b740 20 client.26815 trim_cache size 0 max 16384 2024-09-16T00:27:59.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.876+0000 7f100ee2b740 1 Processor -- start 2024-09-16T00:27:59.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.876+0000 7f100ee2b740 1 -- start start 2024-09-16T00:27:59.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f100ee2b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a390ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f100ee2b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38edb40 0x5565a390efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f100ee2b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38f7370 0x5565a38f12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f100ee2b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10081af740 con 0x5565a38ec5d0 2024-09-16T00:27:59.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f100ee2b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f10081ad010 con 0x5565a38f7370 2024-09-16T00:27:59.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f100ee2b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f10081aae60 con 0x5565a38edb40 2024-09-16T00:27:59.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f100659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a390ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f1006d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38f7370 0x5565a38f12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f0ffffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38edb40 0x5565a390efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f1006d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38f7370 0x5565a38f12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:52942/0 (socket says 172.21.15.33:52942) 2024-09-16T00:27:59.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f1006d9d640 1 -- 172.21.15.33:0/2093089967 learned_addr learned my addr 172.21.15.33:0/2093089967 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:59.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f1006d9d640 1 -- 172.21.15.33:0/2093089967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38edb40 msgr2=0x5565a390efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:59.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f1006d9d640 1 --2- 172.21.15.33:0/2093089967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38edb40 0x5565a390efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f1006d9d640 1 -- 172.21.15.33:0/2093089967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 msgr2=0x5565a390ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:59.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f1006d9d640 1 --2- 172.21.15.33:0/2093089967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a390ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f1006d9d640 1 -- 172.21.15.33:0/2093089967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565a39032b0 con 0x5565a38f7370 2024-09-16T00:27:59.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.877+0000 7f1006d9d640 1 --2- 172.21.15.33:0/2093089967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38f7370 0x5565a38f12e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1000004680 tx=0x7f1000009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50d464b40cef983 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.878+0000 7f0feffff640 1 -- 172.21.15.33:0/2093089967 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100003b070 con 0x5565a38f7370 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.878+0000 7f0feffff640 1 -- 172.21.15.33:0/2093089967 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10000048d0 con 0x5565a38f7370 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.878+0000 7f0feffff640 1 -- 172.21.15.33:0/2093089967 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000022050 con 0x5565a38f7370 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 -- 172.21.15.33:0/2093089967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38f7370 msgr2=0x5565a38f12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 --2- 172.21.15.33:0/2093089967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38f7370 0x5565a38f12e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1000004680 tx=0x7f1000009920 comp rx=0 tx=0).stop 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 -- 172.21.15.33:0/2093089967 shutdown_connections 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 --2- 172.21.15.33:0/2093089967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38f7370 0x5565a38f12e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 --2- 172.21.15.33:0/2093089967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38edb40 0x5565a390efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 --2- 172.21.15.33:0/2093089967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a390ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 -- 172.21.15.33:0/2093089967 >> 172.21.15.33:0/2093089967 conn(0x5565a38b5310 msgr2=0x5565a38f2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 -- 172.21.15.33:0/2093089967 shutdown_connections 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 -- 172.21.15.33:0/2093089967 wait complete. 2024-09-16T00:27:59.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 -- start start 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 Processor -- start 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a38fb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38edb40 0x5565a38fbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38f7370 0x5565a3970a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10081ab4d0 con 0x5565a38ec5d0 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f10081ac940 con 0x5565a38f7370 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.879+0000 7f100ee2b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f10081abba0 con 0x5565a38edb40 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f100ee2b740 10 client.0 _ll_register_callbacks cb 0x7f0ff800e9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f100659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a38fb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f100659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a38fb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55090/0 (socket says 172.21.15.33:55090) 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f100659c640 1 -- 172.21.15.33:0/380351603 learned_addr learned my addr 172.21.15.33:0/380351603 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f0ffffff640 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38edb40 0x5565a38fbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f1006d9d640 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38f7370 0x5565a3970a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f0ffffff640 1 -- 172.21.15.33:0/380351603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38f7370 msgr2=0x5565a3970a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f0ffffff640 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38f7370 0x5565a3970a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f0ffffff640 1 -- 172.21.15.33:0/380351603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 msgr2=0x5565a38fb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f0ffffff640 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a38fb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f0ffffff640 1 -- 172.21.15.33:0/380351603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565a3970f80 con 0x5565a38edb40 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f100659c640 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a38fb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f0ffffff640 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38edb40 0x5565a38fbd10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0005940 tx=0x7f0ff0031b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25231c01b37b8653 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f0fedffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f0fedffb640 1 -- 172.21.15.33:0/380351603 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0035040 con 0x5565a38edb40 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f100ee2b740 1 -- 172.21.15.33:0/380351603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10000048d0 con 0x5565a38edb40 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.880+0000 7f0fedffb640 1 -- 172.21.15.33:0/380351603 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0ff0005020 con 0x5565a38edb40 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.881+0000 7f0fedffb640 1 -- 172.21.15.33:0/380351603 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0ff0057050 con 0x5565a38edb40 2024-09-16T00:27:59.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.881+0000 7f100ee2b740 10 client.26821 fetch_fsmap learned FSMap version 30 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.881+0000 7f100ee2b740 1 -- 172.21.15.33:0/380351603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5565a39721d0 con 0x5565a38edb40 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.881+0000 7f0fedffb640 1 -- 172.21.15.33:0/380351603 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10000048d0 con 0x5565a38edb40 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.881+0000 7f0fedffb640 1 -- 172.21.15.33:0/380351603 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f0ff00035f0 con 0x5565a38edb40 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.881+0000 7f100ee2b740 10 client.26821 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.881+0000 7f100ee2b740 10 client.26821 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.881+0000 7f100ee2b740 20 client.26821 populate_metadata read hostname 'smithi033' 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.881+0000 7f100ee2b740 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5565a3972940 0x5565a3974d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.881+0000 7f100ee2b740 4 client.26821 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.881+0000 7f100ee2b740 1 -- 172.21.15.33:0/380351603 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5565a3859d70 con 0x5565a3972940 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.881+0000 7f100659c640 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5565a3972940 0x5565a3974d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f0fedffb640 10 client.26821 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100659c640 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5565a3972940 0x5565a3974d30 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e55f513414232e8 server_cookie=fea0514da3f890c in_seq=0 out_seq=0 2024-09-16T00:27:59.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f0fedffb640 1 -- 172.21.15.33:0/380351603 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5565a3859d70 con 0x5565a3972940 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f0fedffb640 10 client.26821 handle_command_reply: tid=0 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100ee2b740 1 client.26821 shutdown 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100ee2b740 1 -- 172.21.15.33:0/380351603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38edb40 msgr2=0x5565a38fbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100ee2b740 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38edb40 0x5565a38fbd10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0005940 tx=0x7f0ff0031b50 comp rx=0 tx=0).stop 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100ee2b740 1 -- 172.21.15.33:0/380351603 shutdown_connections 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100ee2b740 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5565a3972940 0x5565a3974d30 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100ee2b740 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5565a38f7370 0x5565a3970a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100ee2b740 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a38edb40 0x5565a38fbd10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f0fedffb640 0 client.26821 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100ee2b740 1 --2- 172.21.15.33:0/380351603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565a38ec5d0 0x5565a38fb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100ee2b740 1 -- 172.21.15.33:0/380351603 >> 172.21.15.33:0/380351603 conn(0x5565a38b5310 msgr2=0x5565a39022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100ee2b740 1 -- 172.21.15.33:0/380351603 shutdown_connections 2024-09-16T00:27:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100ee2b740 1 -- 172.21.15.33:0/380351603 wait complete. 2024-09-16T00:27:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.882+0000 7f100ee2b740 20 client.26821 trim_cache size 0 max 16384 2024-09-16T00:27:59.898 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:27:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.883+0000 7f100ca52640 1 -- 172.21.15.33:0/3438875253 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ff40a32c0 msgr2=0x7f0ff40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.883+0000 7f100ca52640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ff40a32c0 0x7f0ff40a5780 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f100001b000 tx=0x7f100001d000 comp rx=0 tx=0).stop 2024-09-16T00:27:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.883+0000 7f100ca52640 1 -- 172.21.15.33:0/3438875253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10081009b0 msgr2=0x7f10081a03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:27:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.883+0000 7f100ca52640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10081009b0 0x7f10081a03d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ff800ea60 tx=0x7f0ff800ef20 comp rx=0 tx=0).stop 2024-09-16T00:27:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.883+0000 7f100ca52640 1 -- 172.21.15.33:0/3438875253 shutdown_connections 2024-09-16T00:27:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.883+0000 7f100ca52640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0ff40a32c0 0x7f0ff40a5780 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.883+0000 7f100ca52640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f100810e4d0 0x7f100819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.883+0000 7f100ca52640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1008101360 0x7f10081a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.883+0000 7f100ca52640 1 --2- 172.21.15.33:0/3438875253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10081009b0 0x7f10081a03d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:27:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.883+0000 7f100ca52640 1 -- 172.21.15.33:0/3438875253 >> 172.21.15.33:0/3438875253 conn(0x7f10080fc7d0 msgr2=0x7f10080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:27:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.883+0000 7f100ca52640 1 -- 172.21.15.33:0/3438875253 shutdown_connections 2024-09-16T00:27:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:27:59.883+0000 7f100ca52640 1 -- 172.21.15.33:0/3438875253 wait complete. 2024-09-16T00:28:00.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 916138 10.09 MB/sec execute 428 sec latency 47.350 ms 2024-09-16T00:28:01.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:01 smithi033 ceph-mon[31243]: pgmap v676: 129 pgs: 129 active+clean; 279 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 254 op/s 2024-09-16T00:28:01.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:01 smithi083 ceph-mon[35242]: pgmap v676: 129 pgs: 129 active+clean; 279 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 254 op/s 2024-09-16T00:28:01.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 917486 10.12 MB/sec execute 429 sec latency 25.943 ms 2024-09-16T00:28:01.696 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:28:01.696 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:28:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:01 smithi060 ceph-mon[35185]: pgmap v676: 129 pgs: 129 active+clean; 279 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 254 op/s 2024-09-16T00:28:01.971 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:02.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.522+0000 7f189fd6e640 1 -- 172.21.15.33:0/2667029786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18981036e0 msgr2=0x7f1898107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:02.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.523+0000 7f189fd6e640 1 --2- 172.21.15.33:0/2667029786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18981036e0 0x7f1898107b90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f188c009890 tx=0x7f188c02a360 comp rx=0 tx=0).stop 2024-09-16T00:28:02.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.523+0000 7f189fd6e640 1 -- 172.21.15.33:0/2667029786 shutdown_connections 2024-09-16T00:28:02.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.523+0000 7f189fd6e640 1 --2- 172.21.15.33:0/2667029786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18981082a0 0x7f189810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.523+0000 7f189fd6e640 1 --2- 172.21.15.33:0/2667029786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18981036e0 0x7f1898107b90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.523+0000 7f189fd6e640 1 --2- 172.21.15.33:0/2667029786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1898102d30 0x7f1898103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.523+0000 7f189fd6e640 1 -- 172.21.15.33:0/2667029786 >> 172.21.15.33:0/2667029786 conn(0x7f18980fc7d0 msgr2=0x7f18980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:02.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.523+0000 7f189fd6e640 1 -- 172.21.15.33:0/2667029786 shutdown_connections 2024-09-16T00:28:02.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.523+0000 7f189fd6e640 1 -- 172.21.15.33:0/2667029786 wait complete. 2024-09-16T00:28:02.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.524+0000 7f189fd6e640 1 Processor -- start 2024-09-16T00:28:02.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.524+0000 7f189fd6e640 1 -- start start 2024-09-16T00:28:02.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.524+0000 7f189fd6e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898102d30 0x7f189819f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.524+0000 7f189fd6e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18981036e0 0x7f189810de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.524+0000 7f189fd6e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18981082a0 0x7f189810e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.524+0000 7f189fd6e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1898112060 con 0x7f18981082a0 2024-09-16T00:28:02.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.524+0000 7f189fd6e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1898111ee0 con 0x7f1898102d30 2024-09-16T00:28:02.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.524+0000 7f189fd6e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f18981121e0 con 0x7f18981036e0 2024-09-16T00:28:02.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.525+0000 7f189db0a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898102d30 0x7f189819f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.525+0000 7f189d309640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18981036e0 0x7f189810de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.525+0000 7f189db0a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898102d30 0x7f189819f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51874/0 (socket says 172.21.15.33:51874) 2024-09-16T00:28:02.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.525+0000 7f189db0a640 1 -- 172.21.15.33:0/2573173527 learned_addr learned my addr 172.21.15.33:0/2573173527 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:02.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.526+0000 7f189e30b640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18981082a0 0x7f189810e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.526+0000 7f189db0a640 1 -- 172.21.15.33:0/2573173527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18981036e0 msgr2=0x7f189810de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:02.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.526+0000 7f189db0a640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18981036e0 0x7f189810de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.526+0000 7f189db0a640 1 -- 172.21.15.33:0/2573173527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18981082a0 msgr2=0x7f189810e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:02.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.526+0000 7f189db0a640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18981082a0 0x7f189810e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.526+0000 7f189db0a640 1 -- 172.21.15.33:0/2573173527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f189810e8f0 con 0x7f1898102d30 2024-09-16T00:28:02.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.526+0000 7f189db0a640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898102d30 0x7f189819f2b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f188800e9c0 tx=0x7f188800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:02.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.527+0000 7f1896ffd640 1 -- 172.21.15.33:0/2573173527 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188800ccc0 con 0x7f1898102d30 2024-09-16T00:28:02.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.527+0000 7f1896ffd640 1 -- 172.21.15.33:0/2573173527 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1888005500 con 0x7f1898102d30 2024-09-16T00:28:02.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.528+0000 7f1896ffd640 1 -- 172.21.15.33:0/2573173527 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1888010430 con 0x7f1898102d30 2024-09-16T00:28:02.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.529+0000 7f189fd6e640 1 -- 172.21.15.33:0/2573173527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f189810ebe0 con 0x7f1898102d30 2024-09-16T00:28:02.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.529+0000 7f189fd6e640 1 -- 172.21.15.33:0/2573173527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f189819ae60 con 0x7f1898102d30 2024-09-16T00:28:02.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.533+0000 7f18a2147740 1 Processor -- start 2024-09-16T00:28:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.533+0000 7f18a2147740 1 -- start start 2024-09-16T00:28:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.534+0000 7f18a2147740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c353fb60 0x5650c353ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.534+0000 7f18a2147740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c3540480 0x5650c354a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.534+0000 7f18a2147740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c354ad40 0x5650c354d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.534+0000 7f18a2147740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5650c34a5280 con 0x5650c353fb60 2024-09-16T00:28:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.534+0000 7f18a2147740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5650c344cb10 con 0x5650c354ad40 2024-09-16T00:28:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.534+0000 7f18a2147740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5650c34a1240 con 0x5650c3540480 2024-09-16T00:28:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.534+0000 7f189e30b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c353fb60 0x5650c353ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.534+0000 7f189e30b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c353fb60 0x5650c353ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47740/0 (socket says 172.21.15.33:47740) 2024-09-16T00:28:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.534+0000 7f189e30b640 1 -- 172.21.15.33:0/161022419 learned_addr learned my addr 172.21.15.33:0/161022419 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.534+0000 7f189d309640 1 --2- 172.21.15.33:0/161022419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c354ad40 0x5650c354d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.534+0000 7f1896ffd640 1 -- 172.21.15.33:0/2573173527 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1888004660 con 0x7f1898102d30 2024-09-16T00:28:02.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.535+0000 7f189d309640 1 -- 172.21.15.33:0/161022419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c3540480 msgr2=0x5650c354a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:02.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.535+0000 7f189d309640 1 --2- 172.21.15.33:0/161022419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c3540480 0x5650c354a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.535+0000 7f189d309640 1 -- 172.21.15.33:0/161022419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c353fb60 msgr2=0x5650c353ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:02.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.535+0000 7f189d309640 1 --2- 172.21.15.33:0/161022419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c353fb60 0x5650c353ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.535+0000 7f189d309640 1 -- 172.21.15.33:0/161022419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650c34591f0 con 0x5650c354ad40 2024-09-16T00:28:02.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.535+0000 7f1896ffd640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18800a31e0 0x7f18800a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.535+0000 7f189e30b640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18800a31e0 0x7f18800a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.535+0000 7f1896ffd640 1 -- 172.21.15.33:0/2573173527 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f18880d0060 con 0x7f1898102d30 2024-09-16T00:28:02.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.535+0000 7f189d309640 1 --2- 172.21.15.33:0/161022419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c354ad40 0x5650c354d130 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f188c035040 tx=0x7f188c030420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b24995a730dc7b57 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:02.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.536+0000 7f187ffff640 1 -- 172.21.15.33:0/161022419 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c041030 con 0x5650c354ad40 2024-09-16T00:28:02.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.536+0000 7f187ffff640 1 -- 172.21.15.33:0/161022419 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f188c0304f0 con 0x5650c354ad40 2024-09-16T00:28:02.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.536+0000 7f187ffff640 1 -- 172.21.15.33:0/161022419 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c02ae60 con 0x5650c354ad40 2024-09-16T00:28:02.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.536+0000 7f18a2147740 1 -- 172.21.15.33:0/161022419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c354ad40 msgr2=0x5650c354d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:02.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.536+0000 7f18a2147740 1 --2- 172.21.15.33:0/161022419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c354ad40 0x5650c354d130 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f188c035040 tx=0x7f188c030420 comp rx=0 tx=0).stop 2024-09-16T00:28:02.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.536+0000 7f189e30b640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18800a31e0 0x7f18800a56a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1890009fd0 tx=0x7f1890009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.537+0000 7f18a2147740 1 -- 172.21.15.33:0/161022419 shutdown_connections 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.537+0000 7f18a2147740 1 --2- 172.21.15.33:0/161022419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c354ad40 0x5650c354d130 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.537+0000 7f18a2147740 1 --2- 172.21.15.33:0/161022419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c3540480 0x5650c354a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.537+0000 7f18a2147740 1 --2- 172.21.15.33:0/161022419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c353fb60 0x5650c353ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.537+0000 7f18a2147740 1 -- 172.21.15.33:0/161022419 >> 172.21.15.33:0/161022419 conn(0x5650c353c0f0 msgr2=0x5650c353c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.537+0000 7f18a2147740 1 -- 172.21.15.33:0/161022419 shutdown_connections 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.537+0000 7f18a2147740 1 -- 172.21.15.33:0/161022419 wait complete. 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.537+0000 7f18a2147740 1 -- start start 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.537+0000 7f18a2147740 1 Processor -- start 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f18a2147740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c3547cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f18a2147740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c3540480 0x5650c3548200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f18a2147740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c354ad40 0x5650c3548740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f18a2147740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5650c354d910 con 0x5650c3540480 2024-09-16T00:28:02.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f18a2147740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5650c345ff10 con 0x5650c354ad40 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f18a2147740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5650c354da90 con 0x5650c353fb60 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f18a2147740 10 client.0 _ll_register_callbacks cb 0x5650c3428a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f189db0a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c3547cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f189e30b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c354ad40 0x5650c3548740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f189d309640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c3540480 0x5650c3548200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f189db0a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c3547cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46858/0 (socket says 172.21.15.33:46858) 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f189db0a640 1 -- 172.21.15.33:0/2123154257 learned_addr learned my addr 172.21.15.33:0/2123154257 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f189db0a640 1 -- 172.21.15.33:0/2123154257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c354ad40 msgr2=0x5650c3548740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f189db0a640 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c354ad40 0x5650c3548740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f189db0a640 1 -- 172.21.15.33:0/2123154257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c3540480 msgr2=0x5650c3548200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f189db0a640 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c3540480 0x5650c3548200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f189db0a640 1 -- 172.21.15.33:0/2123154257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650c32166a0 con 0x5650c353fb60 2024-09-16T00:28:02.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f189e30b640 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c354ad40 0x5650c3548740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.538+0000 7f189d309640 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c3540480 0x5650c3548200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.539+0000 7f189db0a640 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c3547cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f188809c190 tx=0x7f18880056b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72f0be5e6b0320f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.539+0000 7f187d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.539+0000 7f187d7fa640 1 -- 172.21.15.33:0/2123154257 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18880b5060 con 0x5650c353fb60 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.539+0000 7f187d7fa640 1 -- 172.21.15.33:0/2123154257 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f18880bc0f0 con 0x5650c353fb60 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.539+0000 7f18a2147740 1 -- 172.21.15.33:0/2123154257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f188c0304f0 con 0x5650c353fb60 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.539+0000 7f187d7fa640 1 -- 172.21.15.33:0/2123154257 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f18880b5200 con 0x5650c353fb60 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.539+0000 7f18a2147740 10 client.26830 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.539+0000 7f18a2147740 1 -- 172.21.15.33:0/2123154257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5650c3559620 con 0x5650c353fb60 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.539+0000 7f187d7fa640 1 -- 172.21.15.33:0/2123154257 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f188c0304f0 con 0x5650c353fb60 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.539+0000 7f187d7fa640 1 -- 172.21.15.33:0/2123154257 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f18880cdba0 con 0x5650c353fb60 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.539+0000 7f18a2147740 10 client.26830 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:02.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.539+0000 7f18a2147740 10 client.26830 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.540+0000 7f18a2147740 20 client.26830 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.540+0000 7f18a2147740 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5650c355aa30 0x5650c355ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.540+0000 7f18a2147740 4 client.26830 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.540+0000 7f18a2147740 1 -- 172.21.15.33:0/2123154257 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650c34b27f0 con 0x5650c355aa30 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.540+0000 7f189d309640 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5650c355aa30 0x5650c355ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.540+0000 7f189d309640 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5650c355aa30 0x5650c355ce20 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ffc30822bb9e438 server_cookie=b72d9b14c98d5c28 in_seq=0 out_seq=0 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.540+0000 7f187d7fa640 10 client.26830 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f187d7fa640 1 -- 172.21.15.33:0/2123154257 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5650c34b27f0 con 0x5650c355aa30 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f187d7fa640 10 client.26830 handle_command_reply: tid=0 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f18a2147740 1 client.26830 shutdown 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f18a2147740 1 -- 172.21.15.33:0/2123154257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 msgr2=0x5650c3547cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f18a2147740 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c3547cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f188809c190 tx=0x7f18880056b0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f18a2147740 1 -- 172.21.15.33:0/2123154257 shutdown_connections 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f18a2147740 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5650c355aa30 0x5650c355ce20 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f18a2147740 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c354ad40 0x5650c3548740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f18a2147740 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c3540480 0x5650c3548200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f18a2147740 1 --2- 172.21.15.33:0/2123154257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c3547cc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f18a2147740 1 -- 172.21.15.33:0/2123154257 >> 172.21.15.33:0/2123154257 conn(0x5650c353c0f0 msgr2=0x5650c3541e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:02.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f187d7fa640 0 client.26830 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:02.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f18a2147740 1 -- 172.21.15.33:0/2123154257 shutdown_connections 2024-09-16T00:28:02.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f18a2147740 1 -- 172.21.15.33:0/2123154257 wait complete. 2024-09-16T00:28:02.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.541+0000 7f18a2147740 20 client.26830 trim_cache size 0 max 16384 2024-09-16T00:28:02.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f18a2147740 1 Processor -- start 2024-09-16T00:28:02.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f18a2147740 1 -- start start 2024-09-16T00:28:02.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f18a2147740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c353b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f18a2147740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c3540480 0x5650c3562680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f18a2147740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c354ad40 0x5650c3562bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f18a2147740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18981c0410 con 0x5650c354ad40 2024-09-16T00:28:02.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f18a2147740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f18981bdce0 con 0x5650c3540480 2024-09-16T00:28:02.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f18a2147740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f18981bbb30 con 0x5650c353fb60 2024-09-16T00:28:02.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f189db0a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c353b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f189e30b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c354ad40 0x5650c3562bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f189d309640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c3540480 0x5650c3562680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f189db0a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c353b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46860/0 (socket says 172.21.15.33:46860) 2024-09-16T00:28:02.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.560+0000 7f189db0a640 1 -- 172.21.15.33:0/1877304786 learned_addr learned my addr 172.21.15.33:0/1877304786 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:02.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.561+0000 7f189d309640 1 -- 172.21.15.33:0/1877304786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 msgr2=0x5650c353b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:02.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.561+0000 7f189d309640 1 --2- 172.21.15.33:0/1877304786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c353b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.561+0000 7f189d309640 1 -- 172.21.15.33:0/1877304786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c354ad40 msgr2=0x5650c3562bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:02.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.561+0000 7f189d309640 1 --2- 172.21.15.33:0/1877304786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c354ad40 0x5650c3562bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.561+0000 7f189d309640 1 -- 172.21.15.33:0/1877304786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650c3546c70 con 0x5650c3540480 2024-09-16T00:28:02.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.561+0000 7f189d309640 1 --2- 172.21.15.33:0/1877304786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c3540480 0x5650c3562680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f188c035040 tx=0x7f188c02a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2f1c23ed04bdc7d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:02.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.561+0000 7f187f7fe640 1 -- 172.21.15.33:0/1877304786 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c041040 con 0x5650c3540480 2024-09-16T00:28:02.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.561+0000 7f187f7fe640 1 -- 172.21.15.33:0/1877304786 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f188c0044b0 con 0x5650c3540480 2024-09-16T00:28:02.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.561+0000 7f187f7fe640 1 -- 172.21.15.33:0/1877304786 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c037070 con 0x5650c3540480 2024-09-16T00:28:02.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.562+0000 7f18a2147740 1 -- 172.21.15.33:0/1877304786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c3540480 msgr2=0x5650c3562680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:02.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.562+0000 7f18a2147740 1 --2- 172.21.15.33:0/1877304786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c3540480 0x5650c3562680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f188c035040 tx=0x7f188c02a360 comp rx=0 tx=0).stop 2024-09-16T00:28:02.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.562+0000 7f18a2147740 1 -- 172.21.15.33:0/1877304786 shutdown_connections 2024-09-16T00:28:02.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.562+0000 7f18a2147740 1 --2- 172.21.15.33:0/1877304786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c354ad40 0x5650c3562bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.562+0000 7f18a2147740 1 --2- 172.21.15.33:0/1877304786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c3540480 0x5650c3562680 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.562+0000 7f18a2147740 1 --2- 172.21.15.33:0/1877304786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c353b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.562+0000 7f18a2147740 1 -- 172.21.15.33:0/1877304786 >> 172.21.15.33:0/1877304786 conn(0x5650c353c0f0 msgr2=0x5650c353aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:02.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.563+0000 7f18a2147740 1 -- 172.21.15.33:0/1877304786 shutdown_connections 2024-09-16T00:28:02.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.563+0000 7f18a2147740 1 -- 172.21.15.33:0/1877304786 wait complete. 2024-09-16T00:28:02.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.564+0000 7f18a2147740 1 -- start start 2024-09-16T00:28:02.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.564+0000 7f18a2147740 1 Processor -- start 2024-09-16T00:28:02.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f18a2147740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c354ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f18a2147740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c3540480 0x5650c354f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f18a2147740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c354ad40 0x5650c35c4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f18a2147740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18981bc1a0 con 0x5650c354ad40 2024-09-16T00:28:02.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f18a2147740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f18981bd610 con 0x5650c3540480 2024-09-16T00:28:02.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f18a2147740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f18981bc870 con 0x5650c353fb60 2024-09-16T00:28:02.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f18a2147740 10 client.0 _ll_register_callbacks cb 0x7f188800ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:02.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f189d309640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c3540480 0x5650c354f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f189db0a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c354ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f189e30b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c354ad40 0x5650c35c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f189db0a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c354ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46864/0 (socket says 172.21.15.33:46864) 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f189db0a640 1 -- 172.21.15.33:0/2946668320 learned_addr learned my addr 172.21.15.33:0/2946668320 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f189d309640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c3540480 0x5650c354f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51914/0 (socket says 172.21.15.33:51914) 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f189db0a640 1 -- 172.21.15.33:0/2946668320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c3540480 msgr2=0x5650c354f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f189db0a640 1 --2- 172.21.15.33:0/2946668320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c3540480 0x5650c354f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f189db0a640 1 -- 172.21.15.33:0/2946668320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c354ad40 msgr2=0x5650c35c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f189db0a640 1 --2- 172.21.15.33:0/2946668320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c354ad40 0x5650c35c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f189db0a640 1 -- 172.21.15.33:0/2946668320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650c3537830 con 0x5650c353fb60 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.565+0000 7f189db0a640 1 --2- 172.21.15.33:0/2946668320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c354ee60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1888004b70 tx=0x7f188809a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e21cc1bf9378b9ed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.566+0000 7f187d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.566+0000 7f187d7fa640 1 -- 172.21.15.33:0/2946668320 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18880b5060 con 0x5650c353fb60 2024-09-16T00:28:02.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.566+0000 7f187d7fa640 1 -- 172.21.15.33:0/2946668320 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f18880cdba0 con 0x5650c353fb60 2024-09-16T00:28:02.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.566+0000 7f18a2147740 1 -- 172.21.15.33:0/2946668320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f188c0044b0 con 0x5650c353fb60 2024-09-16T00:28:02.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.566+0000 7f187d7fa640 1 -- 172.21.15.33:0/2946668320 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f18880984a0 con 0x5650c353fb60 2024-09-16T00:28:02.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.566+0000 7f18a2147740 10 client.26836 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:02.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.566+0000 7f18a2147740 1 -- 172.21.15.33:0/2946668320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5650c35c5bc0 con 0x5650c353fb60 2024-09-16T00:28:02.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.566+0000 7f187d7fa640 1 -- 172.21.15.33:0/2946668320 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f188c0044b0 con 0x5650c353fb60 2024-09-16T00:28:02.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.566+0000 7f187d7fa640 1 -- 172.21.15.33:0/2946668320 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f18880bc0b0 con 0x5650c353fb60 2024-09-16T00:28:02.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.567+0000 7f18a2147740 10 client.26836 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:02.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.567+0000 7f18a2147740 10 client.26836 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:28:02.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.567+0000 7f18a2147740 20 client.26836 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:02.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.567+0000 7f18a2147740 1 --2- 172.21.15.33:0/2946668320 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5650c35c6330 0x5650c35c8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:02.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.567+0000 7f18a2147740 4 client.26836 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:28:02.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.567+0000 7f18a2147740 1 -- 172.21.15.33:0/2946668320 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5650c34acd00 con 0x5650c35c6330 2024-09-16T00:28:02.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.567+0000 7f189d309640 1 --2- 172.21.15.33:0/2946668320 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5650c35c6330 0x5650c35c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.567+0000 7f189d309640 1 --2- 172.21.15.33:0/2946668320 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5650c35c6330 0x5650c35c8750 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b96b6d99922c6f8 server_cookie=73255310e24d7445 in_seq=0 out_seq=0 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.567+0000 7f187d7fa640 10 client.26836 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.568+0000 7f187d7fa640 1 -- 172.21.15.33:0/2946668320 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5650c34acd00 con 0x5650c35c6330 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.568+0000 7f187d7fa640 10 client.26836 handle_command_reply: tid=0 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.568+0000 7f18a2147740 1 client.26836 shutdown 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.569+0000 7f18a2147740 1 -- 172.21.15.33:0/2946668320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 msgr2=0x5650c354ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.569+0000 7f18a2147740 1 --2- 172.21.15.33:0/2946668320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c354ee60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1888004b70 tx=0x7f188809a710 comp rx=0 tx=0).stop 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.569+0000 7f18a2147740 1 -- 172.21.15.33:0/2946668320 shutdown_connections 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.569+0000 7f18a2147740 1 --2- 172.21.15.33:0/2946668320 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5650c35c6330 0x5650c35c8750 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.569+0000 7f18a2147740 1 --2- 172.21.15.33:0/2946668320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5650c354ad40 0x5650c35c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.569+0000 7f18a2147740 1 --2- 172.21.15.33:0/2946668320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5650c3540480 0x5650c354f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.569+0000 7f18a2147740 1 --2- 172.21.15.33:0/2946668320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5650c353fb60 0x5650c354ee60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.569+0000 7f18a2147740 1 -- 172.21.15.33:0/2946668320 >> 172.21.15.33:0/2946668320 conn(0x5650c353c0f0 msgr2=0x5650c353b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:02.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.569+0000 7f187d7fa640 0 client.26836 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:02.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.569+0000 7f18a2147740 1 -- 172.21.15.33:0/2946668320 shutdown_connections 2024-09-16T00:28:02.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.569+0000 7f18a2147740 1 -- 172.21.15.33:0/2946668320 wait complete. 2024-09-16T00:28:02.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.569+0000 7f18a2147740 20 client.26836 trim_cache size 0 max 16384 2024-09-16T00:28:02.573 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:28:02.573 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:28:02.573 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "7be3f4b7-5a81-46b8-b1cd-d17cde3c2dca", 2024-09-16T00:28:02.573 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:28:02.573 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:28:02.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.570+0000 7f189fd6e640 1 -- 172.21.15.33:0/2573173527 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18800a31e0 msgr2=0x7f18800a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:02.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.570+0000 7f189fd6e640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18800a31e0 0x7f18800a56a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1890009fd0 tx=0x7f1890009500 comp rx=0 tx=0).stop 2024-09-16T00:28:02.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.570+0000 7f189fd6e640 1 -- 172.21.15.33:0/2573173527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898102d30 msgr2=0x7f189819f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:02.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.570+0000 7f189fd6e640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898102d30 0x7f189819f2b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f188800e9c0 tx=0x7f188800ee80 comp rx=0 tx=0).stop 2024-09-16T00:28:02.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.570+0000 7f189fd6e640 1 -- 172.21.15.33:0/2573173527 shutdown_connections 2024-09-16T00:28:02.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.570+0000 7f189fd6e640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18800a31e0 0x7f18800a56a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.570+0000 7f189fd6e640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18981082a0 0x7f189810e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.570+0000 7f189fd6e640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18981036e0 0x7f189810de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.570+0000 7f189fd6e640 1 --2- 172.21.15.33:0/2573173527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898102d30 0x7f189819f2b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:02.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.570+0000 7f189fd6e640 1 -- 172.21.15.33:0/2573173527 >> 172.21.15.33:0/2573173527 conn(0x7f18980fc7d0 msgr2=0x7f18981009c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:02.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.570+0000 7f189fd6e640 1 -- 172.21.15.33:0/2573173527 shutdown_connections 2024-09-16T00:28:02.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:02.570+0000 7f189fd6e640 1 -- 172.21.15.33:0/2573173527 wait complete. 2024-09-16T00:28:02.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 920632 10.14 MB/sec execute 430 sec latency 16.929 ms 2024-09-16T00:28:03.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:03 smithi033 ceph-mon[31243]: pgmap v677: 129 pgs: 129 active+clean; 283 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 313 op/s 2024-09-16T00:28:03.386 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:03 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:28:03.386 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:03 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:03.415 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:28:03.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:03 smithi083 ceph-mon[35242]: pgmap v677: 129 pgs: 129 active+clean; 283 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 313 op/s 2024-09-16T00:28:03.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:03 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:28:03.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:03 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:03.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 923884 10.15 MB/sec execute 431 sec latency 11.937 ms 2024-09-16T00:28:03.686 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:03.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:03 smithi060 ceph-mon[35185]: pgmap v677: 129 pgs: 129 active+clean; 283 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 313 op/s 2024-09-16T00:28:03.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:03 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:28:03.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:03 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:04.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.207+0000 7f194ad55640 1 -- 172.21.15.33:0/250123218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1944100e20 msgr2=0x7f1944101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.207+0000 7f194ad55640 1 --2- 172.21.15.33:0/250123218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1944100e20 0x7f1944101200 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1938009900 tx=0x7f193802a480 comp rx=0 tx=0).stop 2024-09-16T00:28:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.207+0000 7f194ad55640 1 -- 172.21.15.33:0/250123218 shutdown_connections 2024-09-16T00:28:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.207+0000 7f194ad55640 1 --2- 172.21.15.33:0/250123218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1944102ef0 0x7f194410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.207+0000 7f194ad55640 1 --2- 172.21.15.33:0/250123218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1944102570 0x7f19441029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.207+0000 7f194ad55640 1 --2- 172.21.15.33:0/250123218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1944100e20 0x7f1944101200 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.207+0000 7f194ad55640 1 -- 172.21.15.33:0/250123218 >> 172.21.15.33:0/250123218 conn(0x7f19440fc7d0 msgr2=0x7f19440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.207+0000 7f194ad55640 1 -- 172.21.15.33:0/250123218 shutdown_connections 2024-09-16T00:28:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.207+0000 7f194ad55640 1 -- 172.21.15.33:0/250123218 wait complete. 2024-09-16T00:28:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f194ad55640 1 Processor -- start 2024-09-16T00:28:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f194ad55640 1 -- start start 2024-09-16T00:28:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f194ad55640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1944100e20 0x7f1944199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f194ad55640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1944102570 0x7f1944199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f194ad55640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1944102ef0 0x7f19441a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f194ad55640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1944110f80 con 0x7f1944100e20 2024-09-16T00:28:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f194ad55640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1944110e00 con 0x7f1944102ef0 2024-09-16T00:28:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f194ad55640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1944111100 con 0x7f1944102570 2024-09-16T00:28:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f1948af1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1944100e20 0x7f1944199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f1948af1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1944100e20 0x7f1944199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47800/0 (socket says 172.21.15.33:47800) 2024-09-16T00:28:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f1943fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1944102570 0x7f1944199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f1948af1640 1 -- 172.21.15.33:0/2019767808 learned_addr learned my addr 172.21.15.33:0/2019767808 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f19492f2640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1944102ef0 0x7f19441a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f1943fff640 1 -- 172.21.15.33:0/2019767808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1944102ef0 msgr2=0x7f19441a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.208+0000 7f1943fff640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1944102ef0 0x7f19441a43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.209+0000 7f1943fff640 1 -- 172.21.15.33:0/2019767808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1944100e20 msgr2=0x7f1944199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.209+0000 7f1943fff640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1944100e20 0x7f1944199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.209+0000 7f1943fff640 1 -- 172.21.15.33:0/2019767808 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19441a48f0 con 0x7f1944102570 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.209+0000 7f19492f2640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1944102ef0 0x7f19441a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.209+0000 7f1948af1640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1944100e20 0x7f1944199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.209+0000 7f1943fff640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1944102570 0x7f1944199870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f193000e9f0 tx=0x7f193000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.209+0000 7f1941ffb640 1 -- 172.21.15.33:0/2019767808 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193000ccc0 con 0x7f1944102570 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.209+0000 7f1941ffb640 1 -- 172.21.15.33:0/2019767808 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1930010910 con 0x7f1944102570 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.209+0000 7f1941ffb640 1 -- 172.21.15.33:0/2019767808 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930018960 con 0x7f1944102570 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.209+0000 7f194ad55640 1 -- 172.21.15.33:0/2019767808 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19441a4ac0 con 0x7f1944102570 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.209+0000 7f194ad55640 1 -- 172.21.15.33:0/2019767808 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19441a5000 con 0x7f1944102570 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.212+0000 7f1941ffb640 1 -- 172.21.15.33:0/2019767808 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1930010ac0 con 0x7f1944102570 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.212+0000 7f194d12e740 1 Processor -- start 2024-09-16T00:28:04.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.212+0000 7f194d12e740 1 -- start start 2024-09-16T00:28:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f194d12e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 0x55e3c01bf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f194d12e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bfdc0 0x55e3c01ca140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f194d12e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01ca680 0x55e3c01cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f194d12e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e3c01252e0 con 0x55e3c01bf4a0 2024-09-16T00:28:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f194d12e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e3c00ccce0 con 0x55e3c01ca680 2024-09-16T00:28:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f194d12e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e3c01212a0 con 0x55e3c01bfdc0 2024-09-16T00:28:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f1941ffb640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f19340a32c0 0x7f19340a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f1948af1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bfdc0 0x55e3c01ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f1948af1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bfdc0 0x55e3c01ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46892/0 (socket says 172.21.15.33:46892) 2024-09-16T00:28:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f1948af1640 1 -- 172.21.15.33:0/2740153780 learned_addr learned my addr 172.21.15.33:0/2740153780 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f1943fff640 1 --2- 172.21.15.33:0/2740153780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01ca680 0x55e3c01cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f19492f2640 1 --2- 172.21.15.33:0/2740153780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 0x55e3c01bf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f19492f2640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f19340a32c0 0x7f19340a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f1941ffb640 1 -- 172.21.15.33:0/2019767808 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f19300d00a0 con 0x7f1944102570 2024-09-16T00:28:04.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f1948af1640 1 -- 172.21.15.33:0/2740153780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01ca680 msgr2=0x55e3c01cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:04.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f1948af1640 1 --2- 172.21.15.33:0/2740153780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01ca680 0x55e3c01cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f1948af1640 1 -- 172.21.15.33:0/2740153780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 msgr2=0x55e3c01bf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:04.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.213+0000 7f1948af1640 1 --2- 172.21.15.33:0/2740153780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 0x55e3c01bf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.214+0000 7f1948af1640 1 -- 172.21.15.33:0/2740153780 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3c00b1440 con 0x55e3c01bfdc0 2024-09-16T00:28:04.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.214+0000 7f19492f2640 1 --2- 172.21.15.33:0/2740153780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 0x55e3c01bf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:04.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.214+0000 7f1948af1640 1 --2- 172.21.15.33:0/2740153780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bfdc0 0x55e3c01ca140 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1938009a20 tx=0x7f1938005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c75660d65288dcd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:04.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.214+0000 7f19492f2640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f19340a32c0 0x7f19340a5780 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f193c0099e0 tx=0x7f193c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:04.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.214+0000 7f192effd640 1 -- 172.21.15.33:0/2740153780 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1938040070 con 0x55e3c01bfdc0 2024-09-16T00:28:04.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.214+0000 7f192effd640 1 -- 172.21.15.33:0/2740153780 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1938003cd0 con 0x55e3c01bfdc0 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.214+0000 7f192effd640 1 -- 172.21.15.33:0/2740153780 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1938035040 con 0x55e3c01bfdc0 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.215+0000 7f194d12e740 1 -- 172.21.15.33:0/2740153780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bfdc0 msgr2=0x55e3c01ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.215+0000 7f194d12e740 1 --2- 172.21.15.33:0/2740153780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bfdc0 0x55e3c01ca140 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1938009a20 tx=0x7f1938005e80 comp rx=0 tx=0).stop 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.215+0000 7f194d12e740 1 -- 172.21.15.33:0/2740153780 shutdown_connections 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.215+0000 7f194d12e740 1 --2- 172.21.15.33:0/2740153780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01ca680 0x55e3c01cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.215+0000 7f194d12e740 1 --2- 172.21.15.33:0/2740153780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bfdc0 0x55e3c01ca140 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.215+0000 7f194d12e740 1 --2- 172.21.15.33:0/2740153780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 0x55e3c01bf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.215+0000 7f194d12e740 1 -- 172.21.15.33:0/2740153780 >> 172.21.15.33:0/2740153780 conn(0x55e3c01bba30 msgr2=0x55e3c01bbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.215+0000 7f194d12e740 1 -- 172.21.15.33:0/2740153780 shutdown_connections 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.215+0000 7f194d12e740 1 -- 172.21.15.33:0/2740153780 wait complete. 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f194d12e740 1 -- start start 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f194d12e740 1 Processor -- start 2024-09-16T00:28:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f194d12e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 0x55e3c01c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f194d12e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bfdc0 0x55e3c01c71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f194d12e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01ca680 0x55e3c01c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f194d12e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e3c01cd250 con 0x55e3c01bf4a0 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f194d12e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e3c00e0050 con 0x55e3c01bfdc0 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f194d12e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e3c01cd3d0 con 0x55e3c01ca680 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f194d12e740 10 client.0 _ll_register_callbacks cb 0x55e3c0120160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f1948af1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 0x55e3c01c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f1948af1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 0x55e3c01c9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47812/0 (socket says 172.21.15.33:47812) 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f1943fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bfdc0 0x55e3c01c71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f19492f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01ca680 0x55e3c01c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f1948af1640 1 -- 172.21.15.33:0/1098654661 learned_addr learned my addr 172.21.15.33:0/1098654661 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f1948af1640 1 -- 172.21.15.33:0/1098654661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01ca680 msgr2=0x55e3c01c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.216+0000 7f1948af1640 1 --2- 172.21.15.33:0/1098654661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01ca680 0x55e3c01c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f1948af1640 1 -- 172.21.15.33:0/1098654661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bfdc0 msgr2=0x55e3c01c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f1948af1640 1 --2- 172.21.15.33:0/1098654661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bfdc0 0x55e3c01c71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f1948af1640 1 -- 172.21.15.33:0/1098654661 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3bfe963c0 con 0x55e3c01bf4a0 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f1948af1640 1 --2- 172.21.15.33:0/1098654661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 0x55e3c01c9f30 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f1938009310 tx=0x7f19380045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621c4947d5ae720c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f1923fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f1923fff640 1 -- 172.21.15.33:0/1098654661 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193804c020 con 0x55e3c01bf4a0 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f1923fff640 1 -- 172.21.15.33:0/1098654661 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f19380035f0 con 0x55e3c01bf4a0 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f194d12e740 1 -- 172.21.15.33:0/1098654661 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1938003cd0 con 0x55e3c01bf4a0 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f1923fff640 1 -- 172.21.15.33:0/1098654661 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1938057050 con 0x55e3c01bf4a0 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f194d12e740 10 client.17454 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f194d12e740 1 -- 172.21.15.33:0/1098654661 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e3c01dcf80 con 0x55e3c01bf4a0 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f1923fff640 1 -- 172.21.15.33:0/1098654661 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1938003cd0 con 0x55e3c01bf4a0 2024-09-16T00:28:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.217+0000 7f1923fff640 1 -- 172.21.15.33:0/1098654661 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f1938045aa0 con 0x55e3c01bf4a0 2024-09-16T00:28:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.218+0000 7f194d12e740 10 client.17454 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.218+0000 7f194d12e740 10 client.17454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:28:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.218+0000 7f194d12e740 20 client.17454 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.218+0000 7f194d12e740 1 --2- 172.21.15.33:0/1098654661 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3c01de2d0 0x55e3c01e06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.218+0000 7f194d12e740 4 client.17454 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:28:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.218+0000 7f194d12e740 1 -- 172.21.15.33:0/1098654661 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3c0132850 con 0x55e3c01de2d0 2024-09-16T00:28:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.218+0000 7f1943fff640 1 --2- 172.21.15.33:0/1098654661 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3c01de2d0 0x55e3c01e06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.218+0000 7f1943fff640 1 --2- 172.21.15.33:0/1098654661 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3c01de2d0 0x55e3c01e06c0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94dcba61fb306a91 server_cookie=78663ead1fa2aa66 in_seq=0 out_seq=0 2024-09-16T00:28:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.218+0000 7f1923fff640 10 client.17454 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.219+0000 7f1923fff640 1 -- 172.21.15.33:0/1098654661 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e3c0132850 con 0x55e3c01de2d0 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.219+0000 7f1923fff640 10 client.17454 handle_command_reply: tid=0 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.219+0000 7f194d12e740 1 client.17454 shutdown 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.219+0000 7f194d12e740 1 -- 172.21.15.33:0/1098654661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 msgr2=0x55e3c01c9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.219+0000 7f194d12e740 1 --2- 172.21.15.33:0/1098654661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 0x55e3c01c9f30 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f1938009310 tx=0x7f19380045e0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.219+0000 7f194d12e740 1 -- 172.21.15.33:0/1098654661 shutdown_connections 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.219+0000 7f194d12e740 1 --2- 172.21.15.33:0/1098654661 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3c01de2d0 0x55e3c01e06c0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.219+0000 7f194d12e740 1 --2- 172.21.15.33:0/1098654661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01ca680 0x55e3c01c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.219+0000 7f194d12e740 1 --2- 172.21.15.33:0/1098654661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bfdc0 0x55e3c01c71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.219+0000 7f194d12e740 1 --2- 172.21.15.33:0/1098654661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bf4a0 0x55e3c01c9f30 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.219+0000 7f194d12e740 1 -- 172.21.15.33:0/1098654661 >> 172.21.15.33:0/1098654661 conn(0x55e3c01bba30 msgr2=0x55e3c01c1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.220+0000 7f1923fff640 0 client.17454 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.220+0000 7f194d12e740 1 -- 172.21.15.33:0/1098654661 shutdown_connections 2024-09-16T00:28:04.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.220+0000 7f194d12e740 1 -- 172.21.15.33:0/1098654661 wait complete. 2024-09-16T00:28:04.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.220+0000 7f194d12e740 20 client.17454 trim_cache size 0 max 16384 2024-09-16T00:28:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.237+0000 7f194d12e740 1 Processor -- start 2024-09-16T00:28:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.237+0000 7f194d12e740 1 -- start start 2024-09-16T00:28:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f194d12e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bba30 0x55e3c01c2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f194d12e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bf4a0 0x55e3c01c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f194d12e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bfdc0 0x55e3c01e5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f194d12e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f19441af7a0 con 0x55e3c01bfdc0 2024-09-16T00:28:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f194d12e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f19441ad070 con 0x55e3c01bba30 2024-09-16T00:28:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f194d12e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f19441aaec0 con 0x55e3c01bf4a0 2024-09-16T00:28:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f1943fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bf4a0 0x55e3c01c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f1943fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bf4a0 0x55e3c01c61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46918/0 (socket says 172.21.15.33:46918) 2024-09-16T00:28:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f1948af1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bba30 0x55e3c01c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f1943fff640 1 -- 172.21.15.33:0/2755867743 learned_addr learned my addr 172.21.15.33:0/2755867743 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f19492f2640 1 --2- 172.21.15.33:0/2755867743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bfdc0 0x55e3c01e5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f1943fff640 1 -- 172.21.15.33:0/2755867743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bba30 msgr2=0x55e3c01c2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f1943fff640 1 --2- 172.21.15.33:0/2755867743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bba30 0x55e3c01c2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f1943fff640 1 -- 172.21.15.33:0/2755867743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bfdc0 msgr2=0x55e3c01e5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f1943fff640 1 --2- 172.21.15.33:0/2755867743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bfdc0 0x55e3c01e5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f1943fff640 1 -- 172.21.15.33:0/2755867743 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3c01bb860 con 0x55e3c01bf4a0 2024-09-16T00:28:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f1948af1640 1 --2- 172.21.15.33:0/2755867743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bba30 0x55e3c01c2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.238+0000 7f1943fff640 1 --2- 172.21.15.33:0/2755867743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bf4a0 0x55e3c01c61c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f19300cd770 tx=0x7f193006e0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c254fd8baa684d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.239+0000 7f192e7fc640 1 -- 172.21.15.33:0/2755867743 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930096e60 con 0x55e3c01bf4a0 2024-09-16T00:28:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.239+0000 7f192e7fc640 1 -- 172.21.15.33:0/2755867743 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19300a05f0 con 0x55e3c01bf4a0 2024-09-16T00:28:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.239+0000 7f192e7fc640 1 -- 172.21.15.33:0/2755867743 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19300a03d0 con 0x55e3c01bf4a0 2024-09-16T00:28:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.239+0000 7f194d12e740 1 -- 172.21.15.33:0/2755867743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bf4a0 msgr2=0x55e3c01c61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.239+0000 7f194d12e740 1 --2- 172.21.15.33:0/2755867743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bf4a0 0x55e3c01c61c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f19300cd770 tx=0x7f193006e0c0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.239+0000 7f194d12e740 1 -- 172.21.15.33:0/2755867743 shutdown_connections 2024-09-16T00:28:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.239+0000 7f194d12e740 1 --2- 172.21.15.33:0/2755867743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bfdc0 0x55e3c01e5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.239+0000 7f194d12e740 1 --2- 172.21.15.33:0/2755867743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bf4a0 0x55e3c01c61c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.239+0000 7f194d12e740 1 --2- 172.21.15.33:0/2755867743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bba30 0x55e3c01c2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.239+0000 7f194d12e740 1 -- 172.21.15.33:0/2755867743 >> 172.21.15.33:0/2755867743 conn(0x55e3c01c7880 msgr2=0x55e3c01c5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.239+0000 7f194d12e740 1 -- 172.21.15.33:0/2755867743 shutdown_connections 2024-09-16T00:28:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.239+0000 7f194d12e740 1 -- 172.21.15.33:0/2755867743 wait complete. 2024-09-16T00:28:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f194d12e740 1 -- start start 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f194d12e740 1 Processor -- start 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f194d12e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bba30 0x55e3c01d7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f194d12e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bf4a0 0x55e3c01d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f194d12e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bfdc0 0x55e3c01d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f194d12e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f19441ab530 con 0x55e3c01bfdc0 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f194d12e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f19441ac9a0 con 0x55e3c01bf4a0 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f194d12e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f19441abc00 con 0x55e3c01bba30 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f194d12e740 10 client.0 _ll_register_callbacks cb 0x7f193c0116e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f1948af1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bba30 0x55e3c01d7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f19492f2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bfdc0 0x55e3c01d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f1943fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bf4a0 0x55e3c01d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f1948af1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bba30 0x55e3c01d7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46926/0 (socket says 172.21.15.33:46926) 2024-09-16T00:28:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.240+0000 7f1948af1640 1 -- 172.21.15.33:0/3408436170 learned_addr learned my addr 172.21.15.33:0/3408436170 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.241+0000 7f19492f2640 1 -- 172.21.15.33:0/3408436170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bba30 msgr2=0x55e3c01d7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.241+0000 7f19492f2640 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bba30 0x55e3c01d7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.241+0000 7f19492f2640 1 -- 172.21.15.33:0/3408436170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bf4a0 msgr2=0x55e3c01d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.241+0000 7f19492f2640 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bf4a0 0x55e3c01d24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.241+0000 7f19492f2640 1 -- 172.21.15.33:0/3408436170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3c01d3230 con 0x55e3c01bfdc0 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.241+0000 7f1943fff640 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bf4a0 0x55e3c01d24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.241+0000 7f1948af1640 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bba30 0x55e3c01d7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.241+0000 7f19492f2640 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bfdc0 0x55e3c01d2a00 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f193c0116b0 tx=0x7f193c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f567d1d91084f89 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.241+0000 7f1923fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f1923fff640 1 -- 172.21.15.33:0/3408436170 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c003970 con 0x55e3c01bfdc0 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f1923fff640 1 -- 172.21.15.33:0/3408436170 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f193c003b10 con 0x55e3c01bfdc0 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f194d12e740 1 -- 172.21.15.33:0/3408436170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19300a05f0 con 0x55e3c01bfdc0 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f1923fff640 1 -- 172.21.15.33:0/3408436170 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f193c039050 con 0x55e3c01bfdc0 2024-09-16T00:28:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f194d12e740 10 client.17460 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:04.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f194d12e740 1 -- 172.21.15.33:0/3408436170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e3c0248c60 con 0x55e3c01bfdc0 2024-09-16T00:28:04.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f1923fff640 1 -- 172.21.15.33:0/3408436170 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f193c01dd40 con 0x55e3c01bfdc0 2024-09-16T00:28:04.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f1923fff640 1 -- 172.21.15.33:0/3408436170 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f193c01d070 con 0x55e3c01bfdc0 2024-09-16T00:28:04.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f194d12e740 10 client.17460 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:04.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f194d12e740 10 client.17460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:28:04.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f194d12e740 20 client.17460 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:04.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f194d12e740 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3c01ca680 0x55e3c024b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:04.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f194d12e740 4 client.17460 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:28:04.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.242+0000 7f194d12e740 1 -- 172.21.15.33:0/3408436170 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f19300a05f0 con 0x55e3c01ca680 2024-09-16T00:28:04.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.243+0000 7f1948af1640 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3c01ca680 0x55e3c024b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:04.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.243+0000 7f1923fff640 10 client.17460 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:04.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.243+0000 7f1948af1640 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3c01ca680 0x55e3c024b410 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e84b99ca559092cd server_cookie=7b1e28bb05913d96 in_seq=0 out_seq=0 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.243+0000 7f1923fff640 1 -- 172.21.15.33:0/3408436170 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f19300a05f0 con 0x55e3c01ca680 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.243+0000 7f1923fff640 10 client.17460 handle_command_reply: tid=0 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.243+0000 7f194d12e740 1 client.17460 shutdown 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194d12e740 1 -- 172.21.15.33:0/3408436170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bfdc0 msgr2=0x55e3c01d2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194d12e740 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bfdc0 0x55e3c01d2a00 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f193c0116b0 tx=0x7f193c004a40 comp rx=0 tx=0).stop 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194d12e740 1 -- 172.21.15.33:0/3408436170 shutdown_connections 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194d12e740 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e3c01ca680 0x55e3c024b410 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194d12e740 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e3c01bfdc0 0x55e3c01d2a00 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194d12e740 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e3c01bf4a0 0x55e3c01d24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194d12e740 1 --2- 172.21.15.33:0/3408436170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3c01bba30 0x55e3c01d7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194d12e740 1 -- 172.21.15.33:0/3408436170 >> 172.21.15.33:0/3408436170 conn(0x55e3c01c7880 msgr2=0x55e3c01ba350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f1923fff640 0 client.17460 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194d12e740 1 -- 172.21.15.33:0/3408436170 shutdown_connections 2024-09-16T00:28:04.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194d12e740 1 -- 172.21.15.33:0/3408436170 wait complete. 2024-09-16T00:28:04.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194d12e740 20 client.17460 trim_cache size 0 max 16384 2024-09-16T00:28:04.255 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:28:04.255 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:28:04.255 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:28:04.255 INFO:teuthology.orchestra.run.smithi033.stdout: "7be3f4b7-5a81-46b8-b1cd-d17cde3c2dca": { 2024-09-16T00:28:04.255 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:28:04.255 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "7be3f4b7-5a81-46b8-b1cd-d17cde3c2dca", 2024-09-16T00:28:04.255 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:28:04.256 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:04.256 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:04.256 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:28:04.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194ad55640 1 -- 172.21.15.33:0/2019767808 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f19340a32c0 msgr2=0x7f19340a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:04.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194ad55640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f19340a32c0 0x7f19340a5780 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f193c0099e0 tx=0x7f193c009500 comp rx=0 tx=0).stop 2024-09-16T00:28:04.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194ad55640 1 -- 172.21.15.33:0/2019767808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1944102570 msgr2=0x7f1944199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:04.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.244+0000 7f194ad55640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1944102570 0x7f1944199870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f193000e9f0 tx=0x7f193000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.245+0000 7f194ad55640 1 -- 172.21.15.33:0/2019767808 shutdown_connections 2024-09-16T00:28:04.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.245+0000 7f194ad55640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f19340a32c0 0x7f19340a5780 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.245+0000 7f194ad55640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1944102ef0 0x7f19441a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.245+0000 7f194ad55640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1944102570 0x7f1944199870 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.245+0000 7f194ad55640 1 --2- 172.21.15.33:0/2019767808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1944100e20 0x7f1944199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:04.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.245+0000 7f194ad55640 1 -- 172.21.15.33:0/2019767808 >> 172.21.15.33:0/2019767808 conn(0x7f19440fc7d0 msgr2=0x7f19440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:04.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.245+0000 7f194ad55640 1 -- 172.21.15.33:0/2019767808 shutdown_connections 2024-09-16T00:28:04.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:04.245+0000 7f194ad55640 1 -- 172.21.15.33:0/2019767808 wait complete. 2024-09-16T00:28:04.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:04 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:28:04.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:04 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:04.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:04 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:28:04.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:04 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:04.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:04 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:28:04.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:04 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:04.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:04 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:28:04.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:04 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:04.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 926344 10.15 MB/sec execute 432 sec latency 29.436 ms 2024-09-16T00:28:04.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:04 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:28:04.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:04 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:04.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:04 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:28:04.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:04 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:05.317 INFO:tasks.cephfs.filesystem:scrub status for tag:7be3f4b7-5a81-46b8-b1cd-d17cde3c2dca - {'path': '/', 'tag': '7be3f4b7-5a81-46b8-b1cd-d17cde3c2dca', 'options': 'recursive,force'} 2024-09-16T00:28:05.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:05 smithi033 ceph-mon[31243]: pgmap v678: 129 pgs: 129 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 282 op/s 2024-09-16T00:28:05.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:05 smithi083 ceph-mon[35242]: pgmap v678: 129 pgs: 129 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 282 op/s 2024-09-16T00:28:05.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 928396 10.16 MB/sec execute 433 sec latency 74.158 ms 2024-09-16T00:28:05.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:05 smithi060 ceph-mon[35185]: pgmap v678: 129 pgs: 129 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 282 op/s 2024-09-16T00:28:06.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 931266 10.18 MB/sec execute 434 sec latency 44.106 ms 2024-09-16T00:28:07.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:07 smithi033 ceph-mon[31243]: pgmap v679: 129 pgs: 129 active+clean; 272 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 291 op/s 2024-09-16T00:28:07.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:07 smithi083 ceph-mon[35242]: pgmap v679: 129 pgs: 129 active+clean; 272 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 291 op/s 2024-09-16T00:28:07.678 INFO:tasks.workunit.client.0.smithi033.stdout: 1 934500 10.19 MB/sec execute 435 sec latency 15.963 ms 2024-09-16T00:28:07.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:07 smithi060 ceph-mon[35185]: pgmap v679: 129 pgs: 129 active+clean; 272 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 291 op/s 2024-09-16T00:28:08.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 937230 10.20 MB/sec execute 436 sec latency 11.761 ms 2024-09-16T00:28:09.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:09 smithi033 ceph-mon[31243]: pgmap v680: 129 pgs: 129 active+clean; 271 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 219 op/s 2024-09-16T00:28:09.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:09 smithi083 ceph-mon[35242]: pgmap v680: 129 pgs: 129 active+clean; 271 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 219 op/s 2024-09-16T00:28:09.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 939460 10.20 MB/sec execute 437 sec latency 19.286 ms 2024-09-16T00:28:09.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:09 smithi060 ceph-mon[35185]: pgmap v680: 129 pgs: 129 active+clean; 271 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 219 op/s 2024-09-16T00:28:10.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 942285 10.21 MB/sec execute 438 sec latency 16.654 ms 2024-09-16T00:28:11.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:11 smithi033 ceph-mon[31243]: pgmap v681: 129 pgs: 129 active+clean; 276 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 290 op/s 2024-09-16T00:28:11.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:11 smithi083 ceph-mon[35242]: pgmap v681: 129 pgs: 129 active+clean; 276 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 290 op/s 2024-09-16T00:28:11.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 945048 10.23 MB/sec execute 439 sec latency 62.212 ms 2024-09-16T00:28:11.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:11 smithi060 ceph-mon[35185]: pgmap v681: 129 pgs: 129 active+clean; 276 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 290 op/s 2024-09-16T00:28:12.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:12 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:28:12.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:12 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:28:12.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:12 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:28:12.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:12 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:28:12.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 947696 10.23 MB/sec execute 440 sec latency 11.814 ms 2024-09-16T00:28:12.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:12 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:28:12.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:12 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:28:13.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:13 smithi033 ceph-mon[31243]: pgmap v682: 129 pgs: 129 active+clean; 276 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 292 op/s 2024-09-16T00:28:13.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:13 smithi083 ceph-mon[35242]: pgmap v682: 129 pgs: 129 active+clean; 276 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 292 op/s 2024-09-16T00:28:13.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 950311 10.24 MB/sec execute 441 sec latency 20.694 ms 2024-09-16T00:28:13.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:13 smithi060 ceph-mon[35185]: pgmap v682: 129 pgs: 129 active+clean; 276 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 292 op/s 2024-09-16T00:28:14.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 952930 10.25 MB/sec execute 442 sec latency 17.640 ms 2024-09-16T00:28:15.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:15 smithi033 ceph-mon[31243]: pgmap v683: 129 pgs: 129 active+clean; 277 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 271 op/s 2024-09-16T00:28:15.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:15 smithi083 ceph-mon[35242]: pgmap v683: 129 pgs: 129 active+clean; 277 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 271 op/s 2024-09-16T00:28:15.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 956002 10.27 MB/sec execute 443 sec latency 36.710 ms 2024-09-16T00:28:15.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:15 smithi060 ceph-mon[35185]: pgmap v683: 129 pgs: 129 active+clean; 277 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 271 op/s 2024-09-16T00:28:16.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 957808 10.26 MB/sec execute 444 sec latency 54.221 ms 2024-09-16T00:28:17.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:17 smithi033 ceph-mon[31243]: pgmap v684: 129 pgs: 129 active+clean; 274 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.8 MiB/s wr, 287 op/s 2024-09-16T00:28:17.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:17 smithi083 ceph-mon[35242]: pgmap v684: 129 pgs: 129 active+clean; 274 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.8 MiB/s wr, 287 op/s 2024-09-16T00:28:17.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 960497 10.27 MB/sec execute 445 sec latency 11.745 ms 2024-09-16T00:28:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:17 smithi060 ceph-mon[35185]: pgmap v684: 129 pgs: 129 active+clean; 274 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.8 MiB/s wr, 287 op/s 2024-09-16T00:28:18.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 963590 10.29 MB/sec execute 446 sec latency 11.665 ms 2024-09-16T00:28:19.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:19 smithi033 ceph-mon[31243]: pgmap v685: 129 pgs: 129 active+clean; 270 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 3.3 MiB/s wr, 255 op/s 2024-09-16T00:28:19.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:19 smithi083 ceph-mon[35242]: pgmap v685: 129 pgs: 129 active+clean; 270 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 3.3 MiB/s wr, 255 op/s 2024-09-16T00:28:19.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 966763 10.31 MB/sec execute 447 sec latency 20.955 ms 2024-09-16T00:28:19.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:19 smithi060 ceph-mon[35185]: pgmap v685: 129 pgs: 129 active+clean; 270 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 3.3 MiB/s wr, 255 op/s 2024-09-16T00:28:20.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 969727 10.32 MB/sec execute 448 sec latency 17.776 ms 2024-09-16T00:28:21.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:21 smithi033 ceph-mon[31243]: pgmap v686: 129 pgs: 129 active+clean; 259 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 4.0 MiB/s wr, 299 op/s 2024-09-16T00:28:21.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:21 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:28:21.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:21 smithi083 ceph-mon[35242]: pgmap v686: 129 pgs: 129 active+clean; 259 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 4.0 MiB/s wr, 299 op/s 2024-09-16T00:28:21.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:21 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:28:21.680 INFO:tasks.workunit.client.0.smithi033.stdout: 1 970705 10.30 MB/sec execute 449 sec latency 153.081 ms 2024-09-16T00:28:21.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:21 smithi060 ceph-mon[35185]: pgmap v686: 129 pgs: 129 active+clean; 259 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 4.0 MiB/s wr, 299 op/s 2024-09-16T00:28:21.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:21 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:28:22.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:28:22.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:28:22.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:28:22.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:28:22.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:28:22.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 973827 10.32 MB/sec execute 450 sec latency 22.194 ms 2024-09-16T00:28:22.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:28:22.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:28:22.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:28:22.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:28:22.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:28:22.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:28:22.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:28:22.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:28:22.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:28:22.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:28:23.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:23 smithi083 ceph-mon[35242]: pgmap v687: 129 pgs: 129 active+clean; 252 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 3.7 MiB/s wr, 255 op/s 2024-09-16T00:28:23.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 977001 10.34 MB/sec execute 451 sec latency 12.151 ms 2024-09-16T00:28:23.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:23 smithi060 ceph-mon[35185]: pgmap v687: 129 pgs: 129 active+clean; 252 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 3.7 MiB/s wr, 255 op/s 2024-09-16T00:28:23.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:23 smithi033 ceph-mon[31243]: pgmap v687: 129 pgs: 129 active+clean; 252 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 3.7 MiB/s wr, 255 op/s 2024-09-16T00:28:24.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 979218 10.33 MB/sec execute 452 sec latency 27.630 ms 2024-09-16T00:28:24.748 INFO:tasks.quiescer.fs.[cephfs]:Total duration (43.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T00:28:24.748 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.85 2024-09-16T00:28:24.749 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:28:25.017 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:25.410 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:25 smithi033 ceph-mon[31243]: pgmap v688: 129 pgs: 129 active+clean; 252 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 4.1 MiB/s wr, 296 op/s 2024-09-16T00:28:25.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:25 smithi083 ceph-mon[35242]: pgmap v688: 129 pgs: 129 active+clean; 252 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 4.1 MiB/s wr, 296 op/s 2024-09-16T00:28:25.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 981441 10.34 MB/sec execute 453 sec latency 36.722 ms 2024-09-16T00:28:25.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.495+0000 7fb5eae17640 1 -- 172.21.15.33:0/1583559771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5e41027c0 msgr2=0x7fb5e4106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:25.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.495+0000 7fb5eae17640 1 --2- 172.21.15.33:0/1583559771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5e41027c0 0x7fb5e4106810 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0009900 tx=0x7fb5d002a410 comp rx=0 tx=0).stop 2024-09-16T00:28:25.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.495+0000 7fb5eae17640 1 -- 172.21.15.33:0/1583559771 shutdown_connections 2024-09-16T00:28:25.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.495+0000 7fb5eae17640 1 --2- 172.21.15.33:0/1583559771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5e4106f40 0x7fb5e410eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:25.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.495+0000 7fb5eae17640 1 --2- 172.21.15.33:0/1583559771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5e41027c0 0x7fb5e4106810 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:25.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.495+0000 7fb5eae17640 1 --2- 172.21.15.33:0/1583559771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5e4101e10 0x7fb5e41021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:25.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.495+0000 7fb5eae17640 1 -- 172.21.15.33:0/1583559771 >> 172.21.15.33:0/1583559771 conn(0x7fb5e40fd8d0 msgr2=0x7fb5e40ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:25.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.495+0000 7fb5eae17640 1 -- 172.21.15.33:0/1583559771 shutdown_connections 2024-09-16T00:28:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.495+0000 7fb5eae17640 1 -- 172.21.15.33:0/1583559771 wait complete. 2024-09-16T00:28:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.496+0000 7fb5eae17640 1 Processor -- start 2024-09-16T00:28:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.496+0000 7fb5eae17640 1 -- start start 2024-09-16T00:28:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.496+0000 7fb5eae17640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5e4101e10 0x7fb5e4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.496+0000 7fb5eae17640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5e41027c0 0x7fb5e41989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5eae17640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5e4106f40 0x7fb5e419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5eae17640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e4111fa0 con 0x7fb5e41027c0 2024-09-16T00:28:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5eae17640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e4111e20 con 0x7fb5e4101e10 2024-09-16T00:28:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5eae17640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e4112120 con 0x7fb5e4106f40 2024-09-16T00:28:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5e41027c0 0x7fb5e41989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e8bb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5e4101e10 0x7fb5e4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:25.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5e41027c0 0x7fb5e41989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50492/0 (socket says 172.21.15.33:50492) 2024-09-16T00:28:25.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e3fff640 1 -- 172.21.15.33:0/3157525512 learned_addr learned my addr 172.21.15.33:0/3157525512 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:25.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e93b4640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5e4106f40 0x7fb5e419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:25.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e3fff640 1 -- 172.21.15.33:0/3157525512 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5e4106f40 msgr2=0x7fb5e419cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:25.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e3fff640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5e4106f40 0x7fb5e419cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:25.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e3fff640 1 -- 172.21.15.33:0/3157525512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5e4101e10 msgr2=0x7fb5e4198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:25.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e3fff640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5e4101e10 0x7fb5e4198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:25.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e3fff640 1 -- 172.21.15.33:0/3157525512 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e419d410 con 0x7fb5e41027c0 2024-09-16T00:28:25.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e8bb3640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5e4101e10 0x7fb5e4198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:25.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e93b4640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5e4106f40 0x7fb5e419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:25.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e3fff640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5e41027c0 0x7fb5e41989a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0006010 tx=0x7fb5d0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:25.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e1ffb640 1 -- 172.21.15.33:0/3157525512 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d003f030 con 0x7fb5e41027c0 2024-09-16T00:28:25.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e1ffb640 1 -- 172.21.15.33:0/3157525512 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5d0004440 con 0x7fb5e41027c0 2024-09-16T00:28:25.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5eae17640 1 -- 172.21.15.33:0/3157525512 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5e419d6a0 con 0x7fb5e41027c0 2024-09-16T00:28:25.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.497+0000 7fb5e1ffb640 1 -- 172.21.15.33:0/3157525512 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0035070 con 0x7fb5e41027c0 2024-09-16T00:28:25.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.498+0000 7fb5eae17640 1 -- 172.21.15.33:0/3157525512 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5e41a4fe0 con 0x7fb5e41027c0 2024-09-16T00:28:25.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.499+0000 7fb5eae17640 1 -- 172.21.15.33:0/3157525512 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5e41032f0 con 0x7fb5e41027c0 2024-09-16T00:28:25.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.500+0000 7fb5e1ffb640 1 -- 172.21.15.33:0/3157525512 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb5d0042030 con 0x7fb5e41027c0 2024-09-16T00:28:25.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.501+0000 7fb5e1ffb640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb5d40a3240 0x7fb5d40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:25.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.501+0000 7fb5e8bb3640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb5d40a3240 0x7fb5d40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:25.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.501+0000 7fb5e1ffb640 1 -- 172.21.15.33:0/3157525512 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb5d00f0370 con 0x7fb5e41027c0 2024-09-16T00:28:25.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.501+0000 7fb5e8bb3640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb5d40a3240 0x7fb5d40a5700 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009fc0 tx=0x7fb5d8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:25.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.504+0000 7fb5e1ffb640 1 -- 172.21.15.33:0/3157525512 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb5d00bc560 con 0x7fb5e41027c0 2024-09-16T00:28:25.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:25 smithi060 ceph-mon[35185]: pgmap v688: 129 pgs: 129 active+clean; 252 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 4.1 MiB/s wr, 296 op/s 2024-09-16T00:28:25.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.747+0000 7fb5eae17640 1 -- 172.21.15.33:0/3157525512 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5e410a120 con 0x7fb5e41027c0 2024-09-16T00:28:25.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.748+0000 7fb5e1ffb640 1 -- 172.21.15.33:0/3157525512 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7fb5d00c0860 con 0x7fb5e41027c0 2024-09-16T00:28:25.748 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:28:25.748 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:28:25.749 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:28:25.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.750+0000 7fb5eae17640 1 -- 172.21.15.33:0/3157525512 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb5d40a3240 msgr2=0x7fb5d40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:25.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.750+0000 7fb5eae17640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb5d40a3240 0x7fb5d40a5700 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009fc0 tx=0x7fb5d8005f90 comp rx=0 tx=0).stop 2024-09-16T00:28:25.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.750+0000 7fb5eae17640 1 -- 172.21.15.33:0/3157525512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5e41027c0 msgr2=0x7fb5e41989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:25.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.750+0000 7fb5eae17640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5e41027c0 0x7fb5e41989a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0006010 tx=0x7fb5d0004040 comp rx=0 tx=0).stop 2024-09-16T00:28:25.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.750+0000 7fb5eae17640 1 -- 172.21.15.33:0/3157525512 shutdown_connections 2024-09-16T00:28:25.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.750+0000 7fb5eae17640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb5d40a3240 0x7fb5d40a5700 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:25.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.750+0000 7fb5eae17640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5e4106f40 0x7fb5e419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:25.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.750+0000 7fb5eae17640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5e41027c0 0x7fb5e41989a0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:25.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.750+0000 7fb5eae17640 1 --2- 172.21.15.33:0/3157525512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5e4101e10 0x7fb5e4198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:25.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.750+0000 7fb5eae17640 1 -- 172.21.15.33:0/3157525512 >> 172.21.15.33:0/3157525512 conn(0x7fb5e40fd8d0 msgr2=0x7fb5e40fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:25.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.750+0000 7fb5eae17640 1 -- 172.21.15.33:0/3157525512 shutdown_connections 2024-09-16T00:28:25.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:25.750+0000 7fb5eae17640 1 -- 172.21.15.33:0/3157525512 wait complete. 2024-09-16T00:28:26.550 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:26 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3157525512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:28:26.599 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0' 2024-09-16T00:28:26.599 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0 2024-09-16T00:28:26.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:26 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3157525512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:28:26.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 984328 10.36 MB/sec execute 454 sec latency 80.960 ms 2024-09-16T00:28:26.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:26 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3157525512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:28:26.931 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.457+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/2892245109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44102790 msgr2=0x7f5d441067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.457+0000 7f5d4ab9a640 1 --2- 172.21.15.33:0/2892245109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44102790 0x7f5d441067e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009860 tx=0x7f5d3402a340 comp rx=0 tx=0).stop 2024-09-16T00:28:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.457+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/2892245109 shutdown_connections 2024-09-16T00:28:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.457+0000 7f5d4ab9a640 1 --2- 172.21.15.33:0/2892245109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d44106f10 0x7f5d4410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.457+0000 7f5d4ab9a640 1 --2- 172.21.15.33:0/2892245109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44102790 0x7f5d441067e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.457+0000 7f5d4ab9a640 1 --2- 172.21.15.33:0/2892245109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d44101de0 0x7f5d441021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.457+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/2892245109 >> 172.21.15.33:0/2892245109 conn(0x7f5d440fd8c0 msgr2=0x7f5d440ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.457+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/2892245109 shutdown_connections 2024-09-16T00:28:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.457+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/2892245109 wait complete. 2024-09-16T00:28:27.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d4ab9a640 1 Processor -- start 2024-09-16T00:28:27.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d4ab9a640 1 -- start start 2024-09-16T00:28:27.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d4ab9a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d44101de0 0x7f5d44198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d4ab9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44102790 0x7f5d44198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d4ab9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d44106f10 0x7f5d4419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d4ab9a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5d44111f70 con 0x7f5d44106f10 2024-09-16T00:28:27.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d4ab9a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5d44111df0 con 0x7f5d44101de0 2024-09-16T00:28:27.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d4ab9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441120f0 con 0x7f5d44102790 2024-09-16T00:28:27.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d3bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44102790 0x7f5d44198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d48936640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d44101de0 0x7f5d44198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d3bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44102790 0x7f5d44198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:35394/0 (socket says 172.21.15.33:35394) 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d3bfff640 1 -- 172.21.15.33:0/4025414052 learned_addr learned my addr 172.21.15.33:0/4025414052 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.458+0000 7f5d49137640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d44106f10 0x7f5d4419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.459+0000 7f5d3bfff640 1 -- 172.21.15.33:0/4025414052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d44101de0 msgr2=0x7f5d44198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.459+0000 7f5d3bfff640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d44101de0 0x7f5d44198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.459+0000 7f5d3bfff640 1 -- 172.21.15.33:0/4025414052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d44106f10 msgr2=0x7f5d4419ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.459+0000 7f5d3bfff640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d44106f10 0x7f5d4419ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.459+0000 7f5d3bfff640 1 -- 172.21.15.33:0/4025414052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4419d3d0 con 0x7f5d44102790 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.459+0000 7f5d48936640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d44101de0 0x7f5d44198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.459+0000 7f5d3bfff640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44102790 0x7f5d44198960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5d34035040 tx=0x7f5d34033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.459+0000 7f5d39ffb640 1 -- 172.21.15.33:0/4025414052 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3403f020 con 0x7f5d44102790 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.459+0000 7f5d39ffb640 1 -- 172.21.15.33:0/4025414052 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d34032770 con 0x7f5d44102790 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.459+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/4025414052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d4419d660 con 0x7f5d44102790 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.459+0000 7f5d39ffb640 1 -- 172.21.15.33:0/4025414052 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d34037040 con 0x7f5d44102790 2024-09-16T00:28:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.460+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/4025414052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d441a4fa0 con 0x7f5d44102790 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.462+0000 7f5d39ffb640 1 -- 172.21.15.33:0/4025414052 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5d34042020 con 0x7f5d44102790 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.462+0000 7f5d4cf73740 1 Processor -- start 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.462+0000 7f5d4cf73740 1 -- start start 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.462+0000 7f5d4cf73740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0be850 0x555c4a0bec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.462+0000 7f5d4cf73740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 0x555c4a0c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.462+0000 7f5d4cf73740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0c9a30 0x555c4a0cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.462+0000 7f5d4cf73740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555c4a023160 con 0x555c4a0bf170 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.462+0000 7f5d4cf73740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555c49fca720 con 0x555c4a0c9a30 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.462+0000 7f5d4cf73740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555c4a01f120 con 0x555c4a0be850 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d48936640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 0x555c4a0c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d49137640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0be850 0x555c4a0bec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d48936640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 0x555c4a0c94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50528/0 (socket says 172.21.15.33:50528) 2024-09-16T00:28:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d49137640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0be850 0x555c4a0bec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:35404/0 (socket says 172.21.15.33:35404) 2024-09-16T00:28:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d48936640 1 -- 172.21.15.33:0/4037166593 learned_addr learned my addr 172.21.15.33:0/4037166593 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d3bfff640 1 --2- 172.21.15.33:0/4037166593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0c9a30 0x555c4a0cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d39ffb640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5d2c0a32c0 0x7f5d2c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d49137640 1 -- 172.21.15.33:0/4037166593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0c9a30 msgr2=0x555c4a0cbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d49137640 1 --2- 172.21.15.33:0/4037166593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0c9a30 0x555c4a0cbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d49137640 1 -- 172.21.15.33:0/4037166593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 msgr2=0x555c4a0c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d49137640 1 --2- 172.21.15.33:0/4037166593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 0x555c4a0c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d49137640 1 -- 172.21.15.33:0/4037166593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c49d94ba0 con 0x555c4a0be850 2024-09-16T00:28:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d48936640 1 --2- 172.21.15.33:0/4037166593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 0x555c4a0c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d49137640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5d2c0a32c0 0x7f5d2c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d39ffb640 1 -- 172.21.15.33:0/4025414052 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5d340eff00 con 0x7f5d44102790 2024-09-16T00:28:27.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d49137640 1 --2- 172.21.15.33:0/4037166593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0be850 0x555c4a0bec30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c005fd0 tx=0x7f5d3c0138e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d4b90a822de69e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d2affd640 1 -- 172.21.15.33:0/4037166593 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c01f8e0 con 0x555c4a0be850 2024-09-16T00:28:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d2affd640 1 -- 172.21.15.33:0/4037166593 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d3c02a440 con 0x555c4a0be850 2024-09-16T00:28:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.464+0000 7f5d2affd640 1 -- 172.21.15.33:0/4037166593 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c00d640 con 0x555c4a0be850 2024-09-16T00:28:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.463+0000 7f5d49137640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5d2c0a32c0 0x7f5d2c0a5780 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c027000 tx=0x7f5d3c024000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.464+0000 7f5d4cf73740 1 -- 172.21.15.33:0/4037166593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0be850 msgr2=0x555c4a0bec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.464+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/4037166593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0be850 0x555c4a0bec30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c005fd0 tx=0x7f5d3c0138e0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.464+0000 7f5d4cf73740 1 -- 172.21.15.33:0/4037166593 shutdown_connections 2024-09-16T00:28:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.464+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/4037166593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0c9a30 0x555c4a0cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.464+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/4037166593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 0x555c4a0c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.464+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/4037166593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0be850 0x555c4a0bec30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.464+0000 7f5d4cf73740 1 -- 172.21.15.33:0/4037166593 >> 172.21.15.33:0/4037166593 conn(0x555c4a0bade0 msgr2=0x555c4a0bb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.464+0000 7f5d4cf73740 1 -- 172.21.15.33:0/4037166593 shutdown_connections 2024-09-16T00:28:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.464+0000 7f5d4cf73740 1 -- 172.21.15.33:0/4037166593 wait complete. 2024-09-16T00:28:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d4cf73740 1 -- start start 2024-09-16T00:28:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d4cf73740 1 Processor -- start 2024-09-16T00:28:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d4cf73740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0be850 0x555c4a0c6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d4cf73740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0bf170 0x555c4a0c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d4cf73740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0c9a30 0x555c4a0c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d4cf73740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555c4a0cc600 con 0x555c4a0be850 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d4cf73740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555c49fab900 con 0x555c4a0c9a30 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d4cf73740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555c4a0cc780 con 0x555c4a0bf170 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d4cf73740 10 client.0 _ll_register_callbacks cb 0x555c49fa9780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d48936640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0be850 0x555c4a0c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d48936640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0be850 0x555c4a0c6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50530/0 (socket says 172.21.15.33:50530) 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d48936640 1 -- 172.21.15.33:0/2510632638 learned_addr learned my addr 172.21.15.33:0/2510632638 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d3bfff640 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0bf170 0x555c4a0c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.465+0000 7f5d49137640 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0c9a30 0x555c4a0c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d48936640 1 -- 172.21.15.33:0/2510632638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0bf170 msgr2=0x555c4a0c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d48936640 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0bf170 0x555c4a0c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d48936640 1 -- 172.21.15.33:0/2510632638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0c9a30 msgr2=0x555c4a0c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d48936640 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0c9a30 0x555c4a0c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d48936640 1 -- 172.21.15.33:0/2510632638 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c49febc50 con 0x555c4a0be850 2024-09-16T00:28:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d48936640 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0be850 0x555c4a0c6900 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f5d40012d50 tx=0x7f5d4000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98692070d9ff9b8b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d23fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d23fff640 1 -- 172.21.15.33:0/2510632638 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d40004020 con 0x555c4a0be850 2024-09-16T00:28:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d4cf73740 1 -- 172.21.15.33:0/2510632638 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d3c02a440 con 0x555c4a0be850 2024-09-16T00:28:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d23fff640 1 -- 172.21.15.33:0/2510632638 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5d400041c0 con 0x555c4a0be850 2024-09-16T00:28:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d23fff640 1 -- 172.21.15.33:0/2510632638 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5d400043c0 con 0x555c4a0be850 2024-09-16T00:28:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d4cf73740 10 client.17478 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.466+0000 7f5d4cf73740 1 -- 172.21.15.33:0/2510632638 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555c4a0d8340 con 0x555c4a0be850 2024-09-16T00:28:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.467+0000 7f5d23fff640 1 -- 172.21.15.33:0/2510632638 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d3c02a440 con 0x555c4a0be850 2024-09-16T00:28:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.467+0000 7f5d23fff640 1 -- 172.21.15.33:0/2510632638 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f5d4001c500 con 0x555c4a0be850 2024-09-16T00:28:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.467+0000 7f5d4cf73740 10 client.17478 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.467+0000 7f5d4cf73740 10 client.17478 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:28:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.467+0000 7f5d4cf73740 20 client.17478 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.467+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c4a0d9750 0x555c4a0dbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.467+0000 7f5d4cf73740 4 client.17478 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:28:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.467+0000 7f5d4cf73740 1 -- 172.21.15.33:0/2510632638 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c4a030890 con 0x555c4a0d9750 2024-09-16T00:28:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.467+0000 7f5d3bfff640 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c4a0d9750 0x555c4a0dbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.467+0000 7f5d3bfff640 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c4a0d9750 0x555c4a0dbb80 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1123aa54b478cd28 server_cookie=784d7e6a6e18cdee in_seq=0 out_seq=0 2024-09-16T00:28:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.467+0000 7f5d23fff640 10 client.17478 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d23fff640 1 -- 172.21.15.33:0/2510632638 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555c4a030890 con 0x555c4a0d9750 2024-09-16T00:28:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d23fff640 10 client.17478 handle_command_reply: tid=0 2024-09-16T00:28:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d4cf73740 1 client.17478 shutdown 2024-09-16T00:28:27.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d4cf73740 1 -- 172.21.15.33:0/2510632638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0be850 msgr2=0x555c4a0c6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0be850 0x555c4a0c6900 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f5d40012d50 tx=0x7f5d4000f790 comp rx=0 tx=0).stop 2024-09-16T00:28:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d4cf73740 1 -- 172.21.15.33:0/2510632638 shutdown_connections 2024-09-16T00:28:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c4a0d9750 0x555c4a0dbb80 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0c9a30 0x555c4a0c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0bf170 0x555c4a0c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/2510632638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0be850 0x555c4a0c6900 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d23fff640 0 client.17478 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d4cf73740 1 -- 172.21.15.33:0/2510632638 >> 172.21.15.33:0/2510632638 conn(0x555c4a0bade0 msgr2=0x555c4a0c0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d4cf73740 1 -- 172.21.15.33:0/2510632638 shutdown_connections 2024-09-16T00:28:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.468+0000 7f5d4cf73740 1 -- 172.21.15.33:0/2510632638 wait complete. 2024-09-16T00:28:27.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.469+0000 7f5d4cf73740 20 client.17478 trim_cache size 0 max 16384 2024-09-16T00:28:27.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d4cf73740 1 Processor -- start 2024-09-16T00:28:27.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d4cf73740 1 -- start start 2024-09-16T00:28:27.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d4cf73740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 0x555c4a0c12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d4cf73740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 0x555c4a0c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d4cf73740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0c9a30 0x555c4a0c7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d4cf73740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441af730 con 0x555c4a0bf170 2024-09-16T00:28:27.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d4cf73740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441ad000 con 0x555c4a0be850 2024-09-16T00:28:27.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d4cf73740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441aae50 con 0x555c4a0c9a30 2024-09-16T00:28:27.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d3bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 0x555c4a0c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d3bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 0x555c4a0c6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50532/0 (socket says 172.21.15.33:50532) 2024-09-16T00:28:27.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d48936640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 0x555c4a0c12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d3bfff640 1 -- 172.21.15.33:0/3802939530 learned_addr learned my addr 172.21.15.33:0/3802939530 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:27.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.487+0000 7f5d49137640 1 --2- 172.21.15.33:0/3802939530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0c9a30 0x555c4a0c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:27 smithi033 ceph-mon[31243]: pgmap v689: 129 pgs: 129 active+clean; 255 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 4.1 MiB/s wr, 297 op/s 2024-09-16T00:28:27.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.488+0000 7f5d48936640 1 -- 172.21.15.33:0/3802939530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0c9a30 msgr2=0x555c4a0c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:27.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.488+0000 7f5d48936640 1 --2- 172.21.15.33:0/3802939530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0c9a30 0x555c4a0c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.488+0000 7f5d48936640 1 -- 172.21.15.33:0/3802939530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 msgr2=0x555c4a0c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.488+0000 7f5d48936640 1 --2- 172.21.15.33:0/3802939530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 0x555c4a0c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.488+0000 7f5d48936640 1 -- 172.21.15.33:0/3802939530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c4a0befa0 con 0x555c4a0be850 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.488+0000 7f5d48936640 1 --2- 172.21.15.33:0/3802939530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 0x555c4a0c12b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d4001c8f0 tx=0x7f5d4000fc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39deb88ab2d137d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.489+0000 7f5d2a7fc640 1 -- 172.21.15.33:0/3802939530 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4001cb60 con 0x555c4a0be850 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.489+0000 7f5d2a7fc640 1 -- 172.21.15.33:0/3802939530 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d4001cd00 con 0x555c4a0be850 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.489+0000 7f5d2a7fc640 1 -- 172.21.15.33:0/3802939530 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d400047c0 con 0x555c4a0be850 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.490+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3802939530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 msgr2=0x555c4a0c12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.490+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/3802939530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 0x555c4a0c12b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d4001c8f0 tx=0x7f5d4000fc90 comp rx=0 tx=0).stop 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.490+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3802939530 shutdown_connections 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.490+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/3802939530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0c9a30 0x555c4a0c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.490+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/3802939530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0bf170 0x555c4a0c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.490+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/3802939530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 0x555c4a0c12b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.490+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3802939530 >> 172.21.15.33:0/3802939530 conn(0x555c4a0bade0 msgr2=0x555c4a0c68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.490+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3802939530 shutdown_connections 2024-09-16T00:28:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.490+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3802939530 wait complete. 2024-09-16T00:28:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.490+0000 7f5d4cf73740 1 -- start start 2024-09-16T00:28:27.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.490+0000 7f5d4cf73740 1 Processor -- start 2024-09-16T00:28:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.491+0000 7f5d4cf73740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 0x555c4a0cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.491+0000 7f5d4cf73740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0bf170 0x555c4a0ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.491+0000 7f5d4cf73740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0c9a30 0x555c4a143420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.491+0000 7f5d48936640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 0x555c4a0cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.491+0000 7f5d48936640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 0x555c4a0cdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:53854/0 (socket says 172.21.15.33:53854) 2024-09-16T00:28:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.491+0000 7f5d48936640 1 -- 172.21.15.33:0/3263742206 learned_addr learned my addr 172.21.15.33:0/3263742206 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.491+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3263742206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441ab4c0 con 0x555c4a0c9a30 2024-09-16T00:28:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.491+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3263742206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441ac930 con 0x555c4a0be850 2024-09-16T00:28:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.491+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3263742206 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441abb90 con 0x555c4a0bf170 2024-09-16T00:28:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.491+0000 7f5d3bfff640 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0bf170 0x555c4a0ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.491+0000 7f5d4cf73740 10 client.0 _ll_register_callbacks cb 0x7f5d40009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.492+0000 7f5d49137640 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0c9a30 0x555c4a143420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.492+0000 7f5d48936640 1 -- 172.21.15.33:0/3263742206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0bf170 msgr2=0x555c4a0ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.492+0000 7f5d48936640 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0bf170 0x555c4a0ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.492+0000 7f5d48936640 1 -- 172.21.15.33:0/3263742206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0c9a30 msgr2=0x555c4a143420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.492+0000 7f5d48936640 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0c9a30 0x555c4a143420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.492+0000 7f5d48936640 1 -- 172.21.15.33:0/3263742206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c4a143a10 con 0x555c4a0be850 2024-09-16T00:28:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.492+0000 7f5d49137640 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0c9a30 0x555c4a143420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:28:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.492+0000 7f5d48936640 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 0x555c4a0cdf00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5d40005670 tx=0x7f5d400056a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6a6e8f79578b62d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.492+0000 7f5d23fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:28:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.492+0000 7f5d23fff640 1 -- 172.21.15.33:0/3263742206 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4001cae0 con 0x555c4a0be850 2024-09-16T00:28:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.492+0000 7f5d23fff640 1 -- 172.21.15.33:0/3263742206 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5d40004560 con 0x555c4a0be850 2024-09-16T00:28:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.493+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3263742206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d4001cd00 con 0x555c4a0be850 2024-09-16T00:28:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.493+0000 7f5d23fff640 1 -- 172.21.15.33:0/3263742206 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5d400047a0 con 0x555c4a0be850 2024-09-16T00:28:27.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.493+0000 7f5d4cf73740 10 client.26912 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.493+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3263742206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555c4a144cb0 con 0x555c4a0be850 2024-09-16T00:28:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.493+0000 7f5d23fff640 1 -- 172.21.15.33:0/3263742206 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d4001cd00 con 0x555c4a0be850 2024-09-16T00:28:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.493+0000 7f5d23fff640 1 -- 172.21.15.33:0/3263742206 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f5d40010aa0 con 0x555c4a0be850 2024-09-16T00:28:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.493+0000 7f5d4cf73740 10 client.26912 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.493+0000 7f5d4cf73740 10 client.26912 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:28:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.493+0000 7f5d4cf73740 20 client.26912 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.493+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c4a1453d0 0x555c4a147810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.493+0000 7f5d4cf73740 4 client.26912 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}] 2024-09-16T00:28:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.493+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3263742206 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}) -- 0x555c4a02abe0 con 0x555c4a1453d0 2024-09-16T00:28:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.494+0000 7f5d3bfff640 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c4a1453d0 0x555c4a147810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.494+0000 7f5d23fff640 10 client.26912 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:27.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.495+0000 7f5d3bfff640 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c4a1453d0 0x555c4a147810 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9304835dcebe078 server_cookie=a57326713ff7a690 in_seq=0 out_seq=0 2024-09-16T00:28:27.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d23fff640 1 -- 172.21.15.33:0/3263742206 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555c4a02abe0 con 0x555c4a1453d0 2024-09-16T00:28:27.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d23fff640 10 client.26912 handle_command_reply: tid=0 2024-09-16T00:28:27.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d4cf73740 1 client.26912 shutdown 2024-09-16T00:28:27.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3263742206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 msgr2=0x555c4a0cdf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:27.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 0x555c4a0cdf00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5d40005670 tx=0x7f5d400056a0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3263742206 shutdown_connections 2024-09-16T00:28:27.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c4a1453d0 0x555c4a147810 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c4a0c9a30 0x555c4a143420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d23fff640 0 client.26912 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:27.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c4a0bf170 0x555c4a0ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d4cf73740 1 --2- 172.21.15.33:0/3263742206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c4a0be850 0x555c4a0cdf00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3263742206 >> 172.21.15.33:0/3263742206 conn(0x555c4a0bade0 msgr2=0x555c4a0c70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:27.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3263742206 shutdown_connections 2024-09-16T00:28:27.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.615+0000 7f5d4cf73740 1 -- 172.21.15.33:0/3263742206 wait complete. 2024-09-16T00:28:27.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4cf73740 20 client.26912 trim_cache size 0 max 16384 2024-09-16T00:28:27.617 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:28:27.617 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:28:27.617 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:28:27.617 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 38, 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "b9e606eb": { 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 38, 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 141.8, 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:28:27.618 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:28:27.619 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:28:27.619 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:28:27.619 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:27.619 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:27.619 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:27.619 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:27.619 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:27.619 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:28:27.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/4025414052 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5d2c0a32c0 msgr2=0x7f5d2c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:27.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4ab9a640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5d2c0a32c0 0x7f5d2c0a5780 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c027000 tx=0x7f5d3c024000 comp rx=0 tx=0).stop 2024-09-16T00:28:27.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/4025414052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44102790 msgr2=0x7f5d44198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:27.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4ab9a640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44102790 0x7f5d44198960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5d34035040 tx=0x7f5d34033d90 comp rx=0 tx=0).stop 2024-09-16T00:28:27.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/4025414052 shutdown_connections 2024-09-16T00:28:27.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4ab9a640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5d2c0a32c0 0x7f5d2c0a5780 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4ab9a640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d44106f10 0x7f5d4419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4ab9a640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44102790 0x7f5d44198960 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4ab9a640 1 --2- 172.21.15.33:0/4025414052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d44101de0 0x7f5d44198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:27.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/4025414052 >> 172.21.15.33:0/4025414052 conn(0x7f5d440fd8c0 msgr2=0x7f5d440fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:27.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/4025414052 shutdown_connections 2024-09-16T00:28:27.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:27.616+0000 7f5d4ab9a640 1 -- 172.21.15.33:0/4025414052 wait complete. 2024-09-16T00:28:27.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:27 smithi083 ceph-mon[35242]: pgmap v689: 129 pgs: 129 active+clean; 255 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 4.1 MiB/s wr, 297 op/s 2024-09-16T00:28:27.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.35 MB/sec execute 455 sec latency 103.006 ms 2024-09-16T00:28:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:27 smithi060 ceph-mon[35185]: pgmap v689: 129 pgs: 129 active+clean; 255 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 4.1 MiB/s wr, 297 op/s 2024-09-16T00:28:28.444 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b9e606eb', quiesce took 3.7 seconds. Will release after: 18.150000000000002 2024-09-16T00:28:28.679 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.33 MB/sec execute 456 sec latency 1067.224 ms 2024-09-16T00:28:29.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:29 smithi083 ceph-mon[35242]: pgmap v690: 129 pgs: 129 active+clean; 255 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 3.1 MiB/s wr, 227 op/s 2024-09-16T00:28:29.680 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.31 MB/sec execute 457 sec latency 2067.368 ms 2024-09-16T00:28:29.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:29 smithi060 ceph-mon[35185]: pgmap v690: 129 pgs: 129 active+clean; 255 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 3.1 MiB/s wr, 227 op/s 2024-09-16T00:28:29.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:29 smithi033 ceph-mon[31243]: pgmap v690: 129 pgs: 129 active+clean; 255 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 3.1 MiB/s wr, 227 op/s 2024-09-16T00:28:30.680 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.28 MB/sec execute 458 sec latency 3067.493 ms 2024-09-16T00:28:31.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:31 smithi083 ceph-mon[35242]: pgmap v691: 129 pgs: 129 active+clean; 259 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 3.9 MiB/s wr, 263 op/s 2024-09-16T00:28:31.680 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.26 MB/sec execute 459 sec latency 4067.620 ms 2024-09-16T00:28:31.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:31 smithi060 ceph-mon[35185]: pgmap v691: 129 pgs: 129 active+clean; 259 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 3.9 MiB/s wr, 263 op/s 2024-09-16T00:28:31.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:31 smithi033 ceph-mon[31243]: pgmap v691: 129 pgs: 129 active+clean; 259 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 3.9 MiB/s wr, 263 op/s 2024-09-16T00:28:32.680 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.24 MB/sec execute 460 sec latency 5067.728 ms 2024-09-16T00:28:33.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:33 smithi083 ceph-mon[35242]: pgmap v692: 129 pgs: 129 active+clean; 261 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-09-16T00:28:33.680 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.22 MB/sec execute 461 sec latency 6067.783 ms 2024-09-16T00:28:33.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:33 smithi060 ceph-mon[35185]: pgmap v692: 129 pgs: 129 active+clean; 261 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-09-16T00:28:33.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:33 smithi033 ceph-mon[31243]: pgmap v692: 129 pgs: 129 active+clean; 261 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-09-16T00:28:34.680 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.19 MB/sec execute 462 sec latency 7067.888 ms 2024-09-16T00:28:35.359 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:28:35.630 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:35.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:35 smithi083 ceph-mon[35242]: pgmap v693: 129 pgs: 129 active+clean; 262 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 3.8 MiB/s wr, 230 op/s 2024-09-16T00:28:35.680 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.17 MB/sec execute 463 sec latency 8067.994 ms 2024-09-16T00:28:35.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:35 smithi060 ceph-mon[35185]: pgmap v693: 129 pgs: 129 active+clean; 262 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 3.8 MiB/s wr, 230 op/s 2024-09-16T00:28:35.751 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:35 smithi033 ceph-mon[31243]: pgmap v693: 129 pgs: 129 active+clean; 262 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 3.8 MiB/s wr, 230 op/s 2024-09-16T00:28:36.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/2171103070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b88106f10 msgr2=0x7f2b8810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:36.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 --2- 172.21.15.33:0/2171103070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b88106f10 0x7f2b8810ea90 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009ca0 tx=0x7f2b8002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/2171103070 shutdown_connections 2024-09-16T00:28:36.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 --2- 172.21.15.33:0/2171103070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b88106f10 0x7f2b8810ea90 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 --2- 172.21.15.33:0/2171103070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b88102790 0x7f2b881067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 --2- 172.21.15.33:0/2171103070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b88101de0 0x7f2b881021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/2171103070 >> 172.21.15.33:0/2171103070 conn(0x7f2b880fd8c0 msgr2=0x7f2b880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/2171103070 shutdown_connections 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/2171103070 wait complete. 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 Processor -- start 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 -- start start 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b88101de0 0x7f2b88198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b88102790 0x7f2b88198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b88106f10 0x7f2b8819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2b88111f70 con 0x7f2b88102790 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2b88111df0 con 0x7f2b88101de0 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.108+0000 7f2b8d4e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881120f0 con 0x7f2b88106f10 2024-09-16T00:28:36.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b7ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b88102790 0x7f2b88198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b7ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b88102790 0x7f2b88198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55528/0 (socket says 172.21.15.33:55528) 2024-09-16T00:28:36.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b7ffff640 1 -- 172.21.15.33:0/3037223135 learned_addr learned my addr 172.21.15.33:0/3037223135 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:36.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b88101de0 0x7f2b88198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b877fe640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b88106f10 0x7f2b8819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b7ffff640 1 -- 172.21.15.33:0/3037223135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b88106f10 msgr2=0x7f2b8819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b7ffff640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b88106f10 0x7f2b8819cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b7ffff640 1 -- 172.21.15.33:0/3037223135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b88101de0 msgr2=0x7f2b88198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b7ffff640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b88101de0 0x7f2b88198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b7ffff640 1 -- 172.21.15.33:0/3037223135 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8819d300 con 0x7f2b88102790 2024-09-16T00:28:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b877fe640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b88106f10 0x7f2b8819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:28:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b7ffff640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b88102790 0x7f2b88198890 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000c920 tx=0x7f2b7000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b84ff9640 1 -- 172.21.15.33:0/3037223135 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70007c70 con 0x7f2b88102790 2024-09-16T00:28:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/3037223135 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8819d5f0 con 0x7f2b88102790 2024-09-16T00:28:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b84ff9640 1 -- 172.21.15.33:0/3037223135 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b70007e10 con 0x7f2b88102790 2024-09-16T00:28:36.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b84ff9640 1 -- 172.21.15.33:0/3037223135 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70018890 con 0x7f2b88102790 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.109+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/3037223135 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b881a4ed0 con 0x7f2b88102790 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.112+0000 7f2b84ff9640 1 -- 172.21.15.33:0/3037223135 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2b70018a70 con 0x7f2b88102790 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.112+0000 7f2b8f8b9740 1 Processor -- start 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.112+0000 7f2b8f8b9740 1 -- start start 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.112+0000 7f2b8f8b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a16014a0 0x55c4a1601880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.112+0000 7f2b8f8b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a1601dc0 0x55c4a160c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b8f8b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a160c680 0x55c4a160ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b8f8b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c4a15672e0 con 0x55c4a1601dc0 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b8f8b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c4a150ece0 con 0x55c4a160c680 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b8f8b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c4a15632a0 con 0x55c4a16014a0 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b86ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a1601dc0 0x55c4a160c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b86ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a1601dc0 0x55c4a160c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55540/0 (socket says 172.21.15.33:55540) 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b86ffd640 1 -- 172.21.15.33:0/2910852131 learned_addr learned my addr 172.21.15.33:0/2910852131 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:36.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b877fe640 1 --2- 172.21.15.33:0/2910852131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a16014a0 0x55c4a1601880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b84ff9640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2b740a3240 0x7f2b740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b7ffff640 1 --2- 172.21.15.33:0/2910852131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a160c680 0x55c4a160ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b86ffd640 1 -- 172.21.15.33:0/2910852131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a16014a0 msgr2=0x55c4a1601880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/2910852131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a16014a0 0x55c4a1601880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b86ffd640 1 -- 172.21.15.33:0/2910852131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a160c680 msgr2=0x55c4a160ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/2910852131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a160c680 0x55c4a160ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b877fe640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2b740a3240 0x7f2b740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b86ffd640 1 -- 172.21.15.33:0/2910852131 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4a14f3440 con 0x55c4a1601dc0 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b84ff9640 1 -- 172.21.15.33:0/3037223135 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2b700cfe90 con 0x7f2b88102790 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/2910852131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a1601dc0 0x55c4a160c140 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2b7800bd10 tx=0x7f2b780107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5182fa7198c7ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.113+0000 7f2b7d7fa640 1 -- 172.21.15.33:0/2910852131 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78004030 con 0x55c4a1601dc0 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.114+0000 7f2b877fe640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2b740a3240 0x7f2b740a5700 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009430 tx=0x7f2b8002a8c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:36.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.114+0000 7f2b7d7fa640 1 -- 172.21.15.33:0/2910852131 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b780041d0 con 0x55c4a1601dc0 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.114+0000 7f2b7d7fa640 1 -- 172.21.15.33:0/2910852131 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78011820 con 0x55c4a1601dc0 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.114+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2910852131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a1601dc0 msgr2=0x55c4a160c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.114+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2910852131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a1601dc0 0x55c4a160c140 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2b7800bd10 tx=0x7f2b780107a0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.114+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2910852131 shutdown_connections 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.114+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2910852131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a160c680 0x55c4a160ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.114+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2910852131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a1601dc0 0x55c4a160c140 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.114+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2910852131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a16014a0 0x55c4a1601880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.114+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2910852131 >> 172.21.15.33:0/2910852131 conn(0x55c4a15fda30 msgr2=0x55c4a15fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.114+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2910852131 shutdown_connections 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.114+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2910852131 wait complete. 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b8f8b9740 1 -- start start 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b8f8b9740 1 Processor -- start 2024-09-16T00:28:36.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b8f8b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a160bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b8f8b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a1601dc0 0x55c4a1609220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b8f8b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a160c680 0x55c4a1609760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b8f8b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c4a160f250 con 0x55c4a16014a0 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b8f8b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c4a1522050 con 0x55c4a1601dc0 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b8f8b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c4a160f3d0 con 0x55c4a160c680 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b8f8b9740 10 client.0 _ll_register_callbacks cb 0x55c4a1562160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b86ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a160bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b86ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a160bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55554/0 (socket says 172.21.15.33:55554) 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b86ffd640 1 -- 172.21.15.33:0/561591658 learned_addr learned my addr 172.21.15.33:0/561591658 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b877fe640 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a160c680 0x55c4a1609760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b86ffd640 1 -- 172.21.15.33:0/561591658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a160c680 msgr2=0x55c4a1609760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a160c680 0x55c4a1609760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b86ffd640 1 -- 172.21.15.33:0/561591658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a1601dc0 msgr2=0x55c4a1609220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.115+0000 7f2b7ffff640 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a1601dc0 0x55c4a1609220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a1601dc0 0x55c4a1609220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b86ffd640 1 -- 172.21.15.33:0/561591658 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4a12d83c0 con 0x55c4a16014a0 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b877fe640 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a160c680 0x55c4a1609760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a160bf70 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f2b78010ca0 tx=0x7f2b780058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d036ad8e2c497 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b6e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b6e7fc640 1 -- 172.21.15.33:0/561591658 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78029060 con 0x55c4a16014a0 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b6e7fc640 1 -- 172.21.15.33:0/561591658 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2b780035f0 con 0x55c4a16014a0 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/561591658 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b780041d0 con 0x55c4a16014a0 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b6e7fc640 1 -- 172.21.15.33:0/561591658 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2b78030050 con 0x55c4a16014a0 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b8f8b9740 10 client.17493 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/561591658 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c4a161abc0 con 0x55c4a16014a0 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b6e7fc640 1 -- 172.21.15.33:0/561591658 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b780041d0 con 0x55c4a16014a0 2024-09-16T00:28:36.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b6e7fc640 1 -- 172.21.15.33:0/561591658 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f2b780047e0 con 0x55c4a16014a0 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b8f8b9740 10 client.17493 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.116+0000 7f2b8f8b9740 10 client.17493 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.117+0000 7f2b8f8b9740 20 client.17493 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.117+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c4a161c020 0x55c4a161e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.117+0000 7f2b8f8b9740 4 client.17493 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.117+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/561591658 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4a1574850 con 0x55c4a161c020 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.117+0000 7f2b7ffff640 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c4a161c020 0x55c4a161e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.117+0000 7f2b6e7fc640 10 client.17493 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.117+0000 7f2b7ffff640 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c4a161c020 0x55c4a161e410 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17ca719278a325e7 server_cookie=b718fc0c36360a1e in_seq=0 out_seq=0 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b6e7fc640 1 -- 172.21.15.33:0/561591658 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c4a1574850 con 0x55c4a161c020 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b6e7fc640 10 client.17493 handle_command_reply: tid=0 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b8f8b9740 1 client.17493 shutdown 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/561591658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 msgr2=0x55c4a160bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:36.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a160bf70 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f2b78010ca0 tx=0x7f2b780058e0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/561591658 shutdown_connections 2024-09-16T00:28:36.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c4a161c020 0x55c4a161e410 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a160c680 0x55c4a1609760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a1601dc0 0x55c4a1609220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/561591658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a160bf70 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b6e7fc640 0 client.17493 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:36.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/561591658 >> 172.21.15.33:0/561591658 conn(0x55c4a15fda30 msgr2=0x55c4a15fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:36.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/561591658 shutdown_connections 2024-09-16T00:28:36.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/561591658 wait complete. 2024-09-16T00:28:36.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.118+0000 7f2b8f8b9740 20 client.17493 trim_cache size 0 max 16384 2024-09-16T00:28:36.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.136+0000 7f2b8f8b9740 1 Processor -- start 2024-09-16T00:28:36.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.136+0000 7f2b8f8b9740 1 -- start start 2024-09-16T00:28:36.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.136+0000 7f2b8f8b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a1609fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.136+0000 7f2b8f8b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a1601dc0 0x55c4a1607af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.136+0000 7f2b8f8b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a160c680 0x55c4a16239e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.136+0000 7f2b8f8b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881b1910 con 0x55c4a16014a0 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.136+0000 7f2b8f8b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881af1e0 con 0x55c4a1601dc0 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.136+0000 7f2b8f8b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881ad030 con 0x55c4a160c680 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b86ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a1609fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b86ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a1609fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55560/0 (socket says 172.21.15.33:55560) 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b86ffd640 1 -- 172.21.15.33:0/2949513754 learned_addr learned my addr 172.21.15.33:0/2949513754 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b877fe640 1 --2- 172.21.15.33:0/2949513754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a160c680 0x55c4a16239e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b7ffff640 1 --2- 172.21.15.33:0/2949513754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a1601dc0 0x55c4a1607af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b86ffd640 1 -- 172.21.15.33:0/2949513754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a160c680 msgr2=0x55c4a16239e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/2949513754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a160c680 0x55c4a16239e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b86ffd640 1 -- 172.21.15.33:0/2949513754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a1601dc0 msgr2=0x55c4a1607af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:36.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/2949513754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a1601dc0 0x55c4a1607af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b86ffd640 1 -- 172.21.15.33:0/2949513754 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4a160f380 con 0x55c4a16014a0 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/2949513754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a1609fe0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2b780107e0 tx=0x7f2b7800bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a46c19acd88b199 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b7cff9640 1 -- 172.21.15.33:0/2949513754 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78029040 con 0x55c4a16014a0 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b7cff9640 1 -- 172.21.15.33:0/2949513754 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b7800b480 con 0x55c4a16014a0 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.137+0000 7f2b7cff9640 1 -- 172.21.15.33:0/2949513754 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b780039f0 con 0x55c4a16014a0 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.138+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2949513754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 msgr2=0x55c4a1609fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.138+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2949513754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a1609fe0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2b780107e0 tx=0x7f2b7800bbe0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.138+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2949513754 shutdown_connections 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.138+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2949513754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a160c680 0x55c4a16239e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.138+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2949513754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a1601dc0 0x55c4a1607af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.138+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2949513754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a1609fe0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.138+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2949513754 >> 172.21.15.33:0/2949513754 conn(0x55c4a15fda30 msgr2=0x55c4a15fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:36.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.138+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2949513754 shutdown_connections 2024-09-16T00:28:36.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.138+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2949513754 wait complete. 2024-09-16T00:28:36.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.138+0000 7f2b8f8b9740 1 -- start start 2024-09-16T00:28:36.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b8f8b9740 1 Processor -- start 2024-09-16T00:28:36.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b8f8b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a1614d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b8f8b9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a1601dc0 0x55c4a1612050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b8f8b9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a160c680 0x55c4a1612590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b8f8b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881ad6a0 con 0x55c4a16014a0 2024-09-16T00:28:36.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b8f8b9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881aeb10 con 0x55c4a160c680 2024-09-16T00:28:36.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b8f8b9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881add70 con 0x55c4a1601dc0 2024-09-16T00:28:36.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b8f8b9740 10 client.0 _ll_register_callbacks cb 0x7f2b80009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:36.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b86ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a1614d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b86ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a1614d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55562/0 (socket says 172.21.15.33:55562) 2024-09-16T00:28:36.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b86ffd640 1 -- 172.21.15.33:0/2310203083 learned_addr learned my addr 172.21.15.33:0/2310203083 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:36.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b7ffff640 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a1601dc0 0x55c4a1612050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b877fe640 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a160c680 0x55c4a1612590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b86ffd640 1 -- 172.21.15.33:0/2310203083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a1601dc0 msgr2=0x55c4a1612050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a1601dc0 0x55c4a1612050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b86ffd640 1 -- 172.21.15.33:0/2310203083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a160c680 msgr2=0x55c4a1612590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a160c680 0x55c4a1612590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b86ffd640 1 -- 172.21.15.33:0/2310203083 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4a1612dc0 con 0x55c4a16014a0 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.139+0000 7f2b7ffff640 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a1601dc0 0x55c4a1612050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b86ffd640 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a1614d00 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2b7800b7f0 tx=0x7f2b78010fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a6fa2b3c4bb0c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b6e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b6e7fc640 1 -- 172.21.15.33:0/2310203083 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78004040 con 0x55c4a16014a0 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b6e7fc640 1 -- 172.21.15.33:0/2310203083 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2b78004590 con 0x55c4a16014a0 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2310203083 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b7800b480 con 0x55c4a16014a0 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b6e7fc640 1 -- 172.21.15.33:0/2310203083 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2b7804b050 con 0x55c4a16014a0 2024-09-16T00:28:36.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b8f8b9740 10 client.17499 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:36.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2310203083 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c4a1686960 con 0x55c4a16014a0 2024-09-16T00:28:36.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b6e7fc640 1 -- 172.21.15.33:0/2310203083 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b7800b480 con 0x55c4a16014a0 2024-09-16T00:28:36.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b6e7fc640 1 -- 172.21.15.33:0/2310203083 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f2b7803d470 con 0x55c4a16014a0 2024-09-16T00:28:36.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b8f8b9740 10 client.17499 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:36.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b8f8b9740 10 client.17499 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:28:36.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b8f8b9740 20 client.17499 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:36.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c4a1686f70 0x55c4a1689360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:36.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b8f8b9740 4 client.17499 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:28:36.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.140+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2310203083 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c4a156ed60 con 0x55c4a1686f70 2024-09-16T00:28:36.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.141+0000 7f2b7ffff640 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c4a1686f70 0x55c4a1689360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:36.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.141+0000 7f2b7ffff640 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c4a1686f70 0x55c4a1689360 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd8777d19b7be84e server_cookie=a7adb9fd5e66f351 in_seq=0 out_seq=0 2024-09-16T00:28:36.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.141+0000 7f2b6e7fc640 10 client.17499 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.141+0000 7f2b6e7fc640 1 -- 172.21.15.33:0/2310203083 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c4a156ed60 con 0x55c4a1686f70 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.141+0000 7f2b6e7fc640 10 client.17499 handle_command_reply: tid=0 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.141+0000 7f2b8f8b9740 1 client.17499 shutdown 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.141+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2310203083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 msgr2=0x55c4a1614d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a1614d00 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2b7800b7f0 tx=0x7f2b78010fd0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2310203083 shutdown_connections 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c4a1686f70 0x55c4a1689360 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4a160c680 0x55c4a1612590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4a1601dc0 0x55c4a1612050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b6e7fc640 0 client.17499 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8f8b9740 1 --2- 172.21.15.33:0/2310203083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c4a16014a0 0x55c4a1614d00 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2310203083 >> 172.21.15.33:0/2310203083 conn(0x55c4a15fda30 msgr2=0x55c4a160aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2310203083 shutdown_connections 2024-09-16T00:28:36.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8f8b9740 1 -- 172.21.15.33:0/2310203083 wait complete. 2024-09-16T00:28:36.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8f8b9740 20 client.17499 trim_cache size 0 max 16384 2024-09-16T00:28:36.148 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:28:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:28:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:28:36.148 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:28:36.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/3037223135 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2b740a3240 msgr2=0x7f2b740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:36.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8d4e0640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2b740a3240 0x7f2b740a5700 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009430 tx=0x7f2b8002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/3037223135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b88102790 msgr2=0x7f2b88198890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:36.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8d4e0640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b88102790 0x7f2b88198890 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000c920 tx=0x7f2b7000cde0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/3037223135 shutdown_connections 2024-09-16T00:28:36.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8d4e0640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2b740a3240 0x7f2b740a5700 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8d4e0640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b88106f10 0x7f2b8819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8d4e0640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b88102790 0x7f2b88198890 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8d4e0640 1 --2- 172.21.15.33:0/3037223135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b88101de0 0x7f2b88198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:36.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.142+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/3037223135 >> 172.21.15.33:0/3037223135 conn(0x7f2b880fd8c0 msgr2=0x7f2b880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:36.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.143+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/3037223135 shutdown_connections 2024-09-16T00:28:36.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:36.143+0000 7f2b8d4e0640 1 -- 172.21.15.33:0/3037223135 wait complete. 2024-09-16T00:28:36.680 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.15 MB/sec execute 464 sec latency 9068.047 ms 2024-09-16T00:28:37.292 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:28:37.293 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:28:37.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:37 smithi033 ceph-mon[31243]: pgmap v694: 129 pgs: 129 active+clean; 261 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 3.3 MiB/s wr, 182 op/s 2024-09-16T00:28:37.588 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:37.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:37 smithi083 ceph-mon[35242]: pgmap v694: 129 pgs: 129 active+clean; 261 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 3.3 MiB/s wr, 182 op/s 2024-09-16T00:28:37.681 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.13 MB/sec execute 465 sec latency 10069.186 ms 2024-09-16T00:28:37.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:37 smithi060 ceph-mon[35185]: pgmap v694: 129 pgs: 129 active+clean; 261 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 3.3 MiB/s wr, 182 op/s 2024-09-16T00:28:38.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.159+0000 7fb2c94b6640 1 -- 172.21.15.33:0/1781087196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2c4100e20 msgr2=0x7fb2c4101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:38.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.159+0000 7fb2c94b6640 1 --2- 172.21.15.33:0/1781087196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2c4100e20 0x7fb2c4101200 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009900 tx=0x7fb2b402a480 comp rx=0 tx=0).stop 2024-09-16T00:28:38.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.159+0000 7fb2c94b6640 1 -- 172.21.15.33:0/1781087196 shutdown_connections 2024-09-16T00:28:38.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.159+0000 7fb2c94b6640 1 --2- 172.21.15.33:0/1781087196 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c4102ef0 0x7fb2c410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:38.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.159+0000 7fb2c94b6640 1 --2- 172.21.15.33:0/1781087196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2c4102570 0x7fb2c41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:38.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.159+0000 7fb2c94b6640 1 --2- 172.21.15.33:0/1781087196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2c4100e20 0x7fb2c4101200 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:38.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.159+0000 7fb2c94b6640 1 -- 172.21.15.33:0/1781087196 >> 172.21.15.33:0/1781087196 conn(0x7fb2c40fc7d0 msgr2=0x7fb2c40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:38.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.159+0000 7fb2c94b6640 1 -- 172.21.15.33:0/1781087196 shutdown_connections 2024-09-16T00:28:38.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.159+0000 7fb2c94b6640 1 -- 172.21.15.33:0/1781087196 wait complete. 2024-09-16T00:28:38.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2c94b6640 1 Processor -- start 2024-09-16T00:28:38.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2c94b6640 1 -- start start 2024-09-16T00:28:38.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2c94b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c4100e20 0x7fb2c4198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:38.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2c94b6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2c4102570 0x7fb2c41989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:38.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2c94b6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2c4102ef0 0x7fb2c419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:38.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2c94b6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c4110f80 con 0x7fb2c4102570 2024-09-16T00:28:38.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2c94b6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c4110e00 con 0x7fb2c4102ef0 2024-09-16T00:28:38.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2c94b6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c4111100 con 0x7fb2c4100e20 2024-09-16T00:28:38.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2bbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2c4102570 0x7fb2c41989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:38.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2bbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2c4102570 0x7fb2c41989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55588/0 (socket says 172.21.15.33:55588) 2024-09-16T00:28:38.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2bbfff640 1 -- 172.21.15.33:0/2270741407 learned_addr learned my addr 172.21.15.33:0/2270741407 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:38.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2c37fe640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2c4102ef0 0x7fb2c419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:38.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.160+0000 7fb2c2ffd640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c4100e20 0x7fb2c4198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:38.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.161+0000 7fb2bbfff640 1 -- 172.21.15.33:0/2270741407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c4100e20 msgr2=0x7fb2c4198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:38.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.161+0000 7fb2bbfff640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c4100e20 0x7fb2c4198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:38.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.161+0000 7fb2bbfff640 1 -- 172.21.15.33:0/2270741407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2c4102ef0 msgr2=0x7fb2c419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:38.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.161+0000 7fb2bbfff640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2c4102ef0 0x7fb2c419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:38.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.161+0000 7fb2bbfff640 1 -- 172.21.15.33:0/2270741407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c419d420 con 0x7fb2c4102570 2024-09-16T00:28:38.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.161+0000 7fb2c2ffd640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c4100e20 0x7fb2c4198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:28:38.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.161+0000 7fb2bbfff640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2c4102570 0x7fb2c41989b0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac00c920 tx=0x7fb2ac00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:38.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.161+0000 7fb2c0ff9640 1 -- 172.21.15.33:0/2270741407 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac007c70 con 0x7fb2c4102570 2024-09-16T00:28:38.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.161+0000 7fb2c94b6640 1 -- 172.21.15.33:0/2270741407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c419d710 con 0x7fb2c4102570 2024-09-16T00:28:38.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.161+0000 7fb2c0ff9640 1 -- 172.21.15.33:0/2270741407 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2ac007e10 con 0x7fb2c4102570 2024-09-16T00:28:38.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.161+0000 7fb2c0ff9640 1 -- 172.21.15.33:0/2270741407 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac018890 con 0x7fb2c4102570 2024-09-16T00:28:38.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.161+0000 7fb2c94b6640 1 -- 172.21.15.33:0/2270741407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c41a4ff0 con 0x7fb2c4102570 2024-09-16T00:28:38.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.162+0000 7fb2c94b6640 1 -- 172.21.15.33:0/2270741407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c4101200 con 0x7fb2c4102570 2024-09-16T00:28:38.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.163+0000 7fb2c0ff9640 1 -- 172.21.15.33:0/2270741407 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb2ac018a50 con 0x7fb2c4102570 2024-09-16T00:28:38.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.164+0000 7fb2c0ff9640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb2b00a3240 0x7fb2b00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:38.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.165+0000 7fb2c2ffd640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb2b00a3240 0x7fb2b00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:38.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.165+0000 7fb2c0ff9640 1 -- 172.21.15.33:0/2270741407 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb2ac0d0170 con 0x7fb2c4102570 2024-09-16T00:28:38.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.165+0000 7fb2c2ffd640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb2b00a3240 0x7fb2b00a5700 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4005ab0 tx=0x7fb2b4005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:38.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.168+0000 7fb2c0ff9640 1 -- 172.21.15.33:0/2270741407 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb2ac014030 con 0x7fb2c4102570 2024-09-16T00:28:38.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.379+0000 7fb2c94b6640 1 -- 172.21.15.33:0/2270741407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2c410a0a0 con 0x7fb2c4102570 2024-09-16T00:28:38.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.379+0000 7fb2c0ff9640 1 -- 172.21.15.33:0/2270741407 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7fb2ac09caa0 con 0x7fb2c4102570 2024-09-16T00:28:38.380 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:28:38.380 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:28:38.380 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:28:38.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.382+0000 7fb2c94b6640 1 -- 172.21.15.33:0/2270741407 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb2b00a3240 msgr2=0x7fb2b00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:38.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.382+0000 7fb2c94b6640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb2b00a3240 0x7fb2b00a5700 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4005ab0 tx=0x7fb2b4005a20 comp rx=0 tx=0).stop 2024-09-16T00:28:38.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.382+0000 7fb2c94b6640 1 -- 172.21.15.33:0/2270741407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2c4102570 msgr2=0x7fb2c41989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:38.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.382+0000 7fb2c94b6640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2c4102570 0x7fb2c41989b0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac00c920 tx=0x7fb2ac00cde0 comp rx=0 tx=0).stop 2024-09-16T00:28:38.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.382+0000 7fb2c94b6640 1 -- 172.21.15.33:0/2270741407 shutdown_connections 2024-09-16T00:28:38.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.382+0000 7fb2c94b6640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb2b00a3240 0x7fb2b00a5700 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:38.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.382+0000 7fb2c94b6640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2c4102ef0 0x7fb2c419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:38.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.382+0000 7fb2c94b6640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2c4102570 0x7fb2c41989b0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:38.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.382+0000 7fb2c94b6640 1 --2- 172.21.15.33:0/2270741407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c4100e20 0x7fb2c4198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:38.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.382+0000 7fb2c94b6640 1 -- 172.21.15.33:0/2270741407 >> 172.21.15.33:0/2270741407 conn(0x7fb2c40fc7d0 msgr2=0x7fb2c406a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:38.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.382+0000 7fb2c94b6640 1 -- 172.21.15.33:0/2270741407 shutdown_connections 2024-09-16T00:28:38.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:38.382+0000 7fb2c94b6640 1 -- 172.21.15.33:0/2270741407 wait complete. 2024-09-16T00:28:38.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:38 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2270741407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:28:38.682 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.11 MB/sec execute 466 sec latency 11069.292 ms 2024-09-16T00:28:38.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:38 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2270741407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:28:38.756 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:38 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2270741407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:28:39.420 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:28:39.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:39 smithi083 ceph-mon[35242]: pgmap v695: 129 pgs: 129 active+clean; 262 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-09-16T00:28:39.680 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:39.681 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.09 MB/sec execute 467 sec latency 12069.333 ms 2024-09-16T00:28:39.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:39 smithi060 ceph-mon[35185]: pgmap v695: 129 pgs: 129 active+clean; 262 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-09-16T00:28:39.801 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:39 smithi033 ceph-mon[31243]: pgmap v695: 129 pgs: 129 active+clean; 262 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-09-16T00:28:40.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.189+0000 7f3521e2c640 1 -- 172.21.15.33:0/2163987822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c100c10 msgr2=0x7f351c110840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:40.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.189+0000 7f3521e2c640 1 --2- 172.21.15.33:0/2163987822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c100c10 0x7f351c110840 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f3514009ca0 tx=0x7f351402a950 comp rx=0 tx=0).stop 2024-09-16T00:28:40.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.189+0000 7f3521e2c640 1 -- 172.21.15.33:0/2163987822 shutdown_connections 2024-09-16T00:28:40.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.189+0000 7f3521e2c640 1 --2- 172.21.15.33:0/2163987822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c100c10 0x7f351c110840 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.189+0000 7f3521e2c640 1 --2- 172.21.15.33:0/2163987822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f351c102ea0 0x7f351c1005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.189+0000 7f3521e2c640 1 --2- 172.21.15.33:0/2163987822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c102ac0 0x7f351c1000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.189+0000 7f3521e2c640 1 -- 172.21.15.33:0/2163987822 >> 172.21.15.33:0/2163987822 conn(0x7f351c077210 msgr2=0x7f351c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:40.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.189+0000 7f3521e2c640 1 -- 172.21.15.33:0/2163987822 shutdown_connections 2024-09-16T00:28:40.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.189+0000 7f3521e2c640 1 -- 172.21.15.33:0/2163987822 wait complete. 2024-09-16T00:28:40.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.190+0000 7f3521e2c640 1 Processor -- start 2024-09-16T00:28:40.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.190+0000 7f3521e2c640 1 -- start start 2024-09-16T00:28:40.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.190+0000 7f3521e2c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c100c10 0x7f351c19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.190+0000 7f3521e2c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c102ac0 0x7f351c19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.190+0000 7f3521e2c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f351c102ea0 0x7f351c19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.190+0000 7f3521e2c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f351c114160 con 0x7f351c102ac0 2024-09-16T00:28:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.190+0000 7f3521e2c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f351c113fe0 con 0x7f351c100c10 2024-09-16T00:28:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.190+0000 7f3521e2c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f351c1142e0 con 0x7f351c102ea0 2024-09-16T00:28:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.190+0000 7f351affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c102ac0 0x7f351c19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f351affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c102ac0 0x7f351c19ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55604/0 (socket says 172.21.15.33:55604) 2024-09-16T00:28:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f351b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c100c10 0x7f351c19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f351affd640 1 -- 172.21.15.33:0/1959331862 learned_addr learned my addr 172.21.15.33:0/1959331862 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f351bfff640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f351c102ea0 0x7f351c19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f351affd640 1 -- 172.21.15.33:0/1959331862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f351c102ea0 msgr2=0x7f351c19eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f351affd640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f351c102ea0 0x7f351c19eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f351affd640 1 -- 172.21.15.33:0/1959331862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c100c10 msgr2=0x7f351c19a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f351affd640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c100c10 0x7f351c19a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f351affd640 1 -- 172.21.15.33:0/1959331862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351c19f5d0 con 0x7f351c102ac0 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f351affd640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c102ac0 0x7f351c19ab60 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f351000c890 tx=0x7f351000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f3518ff9640 1 -- 172.21.15.33:0/1959331862 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510004220 con 0x7f351c102ac0 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f3521e2c640 1 -- 172.21.15.33:0/1959331862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f351c19f8c0 con 0x7f351c102ac0 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f3518ff9640 1 -- 172.21.15.33:0/1959331862 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35100043c0 con 0x7f351c102ac0 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f3518ff9640 1 -- 172.21.15.33:0/1959331862 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510004a00 con 0x7f351c102ac0 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.191+0000 7f3521e2c640 1 -- 172.21.15.33:0/1959331862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f351c1a71a0 con 0x7f351c102ac0 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.194+0000 7f3518ff9640 1 -- 172.21.15.33:0/1959331862 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f35100079b0 con 0x7f351c102ac0 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f3524205740 1 Processor -- start 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f3524205740 1 -- start start 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f3524205740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 0x55d725ecb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f3524205740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ecba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f3524205740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725ed8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f3524205740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d725e312f0 con 0x55d725eccb40 2024-09-16T00:28:40.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f3524205740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d725dd8cf0 con 0x55d725ecb5d0 2024-09-16T00:28:40.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f3524205740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d725e2d2b0 con 0x55d725ed6370 2024-09-16T00:28:40.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f351bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 0x55d725ecb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f351bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 0x55d725ecb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55610/0 (socket says 172.21.15.33:55610) 2024-09-16T00:28:40.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f351bfff640 1 -- 172.21.15.33:0/1277526215 learned_addr learned my addr 172.21.15.33:0/1277526215 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:40.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f3518ff9640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35040a3240 0x7f35040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f351b7fe640 1 --2- 172.21.15.33:0/1277526215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ecba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f351affd640 1 --2- 172.21.15.33:0/1277526215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725ed8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f351bfff640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35040a3240 0x7f35040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f3518ff9640 1 -- 172.21.15.33:0/1959331862 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f35100cfa80 con 0x7f351c102ac0 2024-09-16T00:28:40.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f351bfff640 1 -- 172.21.15.33:0/1277526215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 msgr2=0x55d725ed8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f351bfff640 1 --2- 172.21.15.33:0/1277526215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725ed8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f351bfff640 1 -- 172.21.15.33:0/1277526215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 msgr2=0x55d725ecba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f351bfff640 1 --2- 172.21.15.33:0/1277526215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ecba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.195+0000 7f351bfff640 1 -- 172.21.15.33:0/1277526215 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d725dbd3a0 con 0x55d725eccb40 2024-09-16T00:28:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.196+0000 7f351b7fe640 1 --2- 172.21.15.33:0/1277526215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ecba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:28:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.196+0000 7f351bfff640 1 --2- 172.21.15.33:0/1277526215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 0x55d725ecb090 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f351402f420 tx=0x7f351402fd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c527dabde2778dff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.196+0000 7f351bfff640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35040a3240 0x7f35040a5700 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3514047000 tx=0x7f3514049000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.196+0000 7f3501ffb640 1 -- 172.21.15.33:0/1277526215 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351404e070 con 0x55d725eccb40 2024-09-16T00:28:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.196+0000 7f3501ffb640 1 -- 172.21.15.33:0/1277526215 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3514004520 con 0x55d725eccb40 2024-09-16T00:28:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.196+0000 7f3501ffb640 1 -- 172.21.15.33:0/1277526215 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514034040 con 0x55d725eccb40 2024-09-16T00:28:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.197+0000 7f3524205740 1 -- 172.21.15.33:0/1277526215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 msgr2=0x55d725ecb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.197+0000 7f3524205740 1 --2- 172.21.15.33:0/1277526215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 0x55d725ecb090 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f351402f420 tx=0x7f351402fd50 comp rx=0 tx=0).stop 2024-09-16T00:28:40.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.197+0000 7f3524205740 1 -- 172.21.15.33:0/1277526215 shutdown_connections 2024-09-16T00:28:40.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.197+0000 7f3524205740 1 --2- 172.21.15.33:0/1277526215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725ed8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.197+0000 7f3524205740 1 --2- 172.21.15.33:0/1277526215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ecba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.197+0000 7f3524205740 1 --2- 172.21.15.33:0/1277526215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 0x55d725ecb090 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.197+0000 7f3524205740 1 -- 172.21.15.33:0/1277526215 >> 172.21.15.33:0/1277526215 conn(0x55d725e94310 msgr2=0x55d725ec7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:40.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.197+0000 7f3524205740 1 -- 172.21.15.33:0/1277526215 shutdown_connections 2024-09-16T00:28:40.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.197+0000 7f3524205740 1 -- 172.21.15.33:0/1277526215 wait complete. 2024-09-16T00:28:40.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.197+0000 7f3524205740 1 -- start start 2024-09-16T00:28:40.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f3524205740 1 Processor -- start 2024-09-16T00:28:40.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f3524205740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ed5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f3524205740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725eccb40 0x55d725ed3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f3524205740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ed6370 0x55d725ed3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f3524205740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d725ed9010 con 0x55d725ed6370 2024-09-16T00:28:40.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f3524205740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d725dec060 con 0x55d725ecb5d0 2024-09-16T00:28:40.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f3524205740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d725ed9190 con 0x55d725eccb40 2024-09-16T00:28:40.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f3524205740 10 client.0 _ll_register_callbacks cb 0x55d725b3f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:40.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f351bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ed6370 0x55d725ed3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f351bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ed6370 0x55d725ed3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55622/0 (socket says 172.21.15.33:55622) 2024-09-16T00:28:40.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f351bfff640 1 -- 172.21.15.33:0/3691957723 learned_addr learned my addr 172.21.15.33:0/3691957723 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:40.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f351b7fe640 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ed5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f351affd640 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725eccb40 0x55d725ed3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f351bfff640 1 -- 172.21.15.33:0/3691957723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725eccb40 msgr2=0x55d725ed3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:40.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f351bfff640 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725eccb40 0x55d725ed3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f351bfff640 1 -- 172.21.15.33:0/3691957723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 msgr2=0x55d725ed5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:40.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f351bfff640 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ed5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f351bfff640 1 -- 172.21.15.33:0/3691957723 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d725ba23c0 con 0x55d725ed6370 2024-09-16T00:28:40.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f351bfff640 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ed6370 0x55d725ed3640 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f351402a990 tx=0x7f351402fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4926cfa6817b8b43 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:40.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.198+0000 7f34faffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:40.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f34faffd640 1 -- 172.21.15.33:0/3691957723 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514055020 con 0x55d725ed6370 2024-09-16T00:28:40.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f34faffd640 1 -- 172.21.15.33:0/3691957723 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3514049730 con 0x55d725ed6370 2024-09-16T00:28:40.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f3524205740 1 -- 172.21.15.33:0/3691957723 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3514004520 con 0x55d725ed6370 2024-09-16T00:28:40.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f34faffd640 1 -- 172.21.15.33:0/3691957723 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3514060050 con 0x55d725ed6370 2024-09-16T00:28:40.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f3524205740 10 client.17517 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:40.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f3524205740 1 -- 172.21.15.33:0/3691957723 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d725ed9010 con 0x55d725ed6370 2024-09-16T00:28:40.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f34faffd640 1 -- 172.21.15.33:0/3691957723 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3514004520 con 0x55d725ed6370 2024-09-16T00:28:40.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f34faffd640 1 -- 172.21.15.33:0/3691957723 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f35140399b0 con 0x55d725ed6370 2024-09-16T00:28:40.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f3524205740 10 client.17517 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:40.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f3524205740 10 client.17517 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:28:40.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f3524205740 20 client.17517 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:40.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f3524205740 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d725ee5c60 0x55d725ee8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f3524205740 4 client.17517 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:28:40.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.199+0000 7f3524205740 1 -- 172.21.15.33:0/3691957723 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d725e3e860 con 0x55d725ee5c60 2024-09-16T00:28:40.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.200+0000 7f351b7fe640 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d725ee5c60 0x55d725ee8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.200+0000 7f351b7fe640 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d725ee5c60 0x55d725ee8050 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=99ebb6919c91e5a2 server_cookie=976209eaed6d4ac in_seq=0 out_seq=0 2024-09-16T00:28:40.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.200+0000 7f34faffd640 10 client.17517 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:28:40.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f34faffd640 1 -- 172.21.15.33:0/3691957723 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d725e3e860 con 0x55d725ee5c60 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f34faffd640 10 client.17517 handle_command_reply: tid=0 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f3524205740 1 client.17517 shutdown 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f3524205740 1 -- 172.21.15.33:0/3691957723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ed6370 msgr2=0x55d725ed3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f3524205740 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ed6370 0x55d725ed3640 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f351402a990 tx=0x7f351402fa00 comp rx=0 tx=0).stop 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f3524205740 1 -- 172.21.15.33:0/3691957723 shutdown_connections 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f3524205740 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d725ee5c60 0x55d725ee8050 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f3524205740 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ed6370 0x55d725ed3640 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f3524205740 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725eccb40 0x55d725ed3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f3524205740 1 --2- 172.21.15.33:0/3691957723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ed5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f34faffd640 0 client.17517 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f3524205740 1 -- 172.21.15.33:0/3691957723 >> 172.21.15.33:0/3691957723 conn(0x55d725e94310 msgr2=0x55d725ecd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f3524205740 1 -- 172.21.15.33:0/3691957723 shutdown_connections 2024-09-16T00:28:40.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f3524205740 1 -- 172.21.15.33:0/3691957723 wait complete. 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.201+0000 7f3524205740 20 client.17517 trim_cache size 0 max 16384 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.214+0000 7f3524205740 1 Processor -- start 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.214+0000 7f3524205740 1 -- start start 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.214+0000 7f3524205740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ee22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.214+0000 7f3524205740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 0x55d725ed1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.214+0000 7f3524205740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725eed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.214+0000 7f3524205740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f351c1b3ce0 con 0x55d725eccb40 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.214+0000 7f3524205740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f351c1b15b0 con 0x55d725ecb5d0 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f3524205740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f351c1af400 con 0x55d725ed6370 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f351affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 0x55d725ed1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f351affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 0x55d725ed1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55630/0 (socket says 172.21.15.33:55630) 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f351affd640 1 -- 172.21.15.33:0/3186385567 learned_addr learned my addr 172.21.15.33:0/3186385567 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:40.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f351b7fe640 1 --2- 172.21.15.33:0/3186385567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ee22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f351bfff640 1 --2- 172.21.15.33:0/3186385567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725eed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f351affd640 1 -- 172.21.15.33:0/3186385567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 msgr2=0x55d725eed7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f351affd640 1 --2- 172.21.15.33:0/3186385567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725eed7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f351affd640 1 -- 172.21.15.33:0/3186385567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 msgr2=0x55d725ee22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f351affd640 1 --2- 172.21.15.33:0/3186385567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ee22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f351affd640 1 -- 172.21.15.33:0/3186385567 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d725ece120 con 0x55d725eccb40 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f351bfff640 1 --2- 172.21.15.33:0/3186385567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725eed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f351affd640 1 --2- 172.21.15.33:0/3186385567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 0x55d725ed1900 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f3510007590 tx=0x7f3510013840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e303ebc320caaa5d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f35017fa640 1 -- 172.21.15.33:0/3186385567 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351009b520 con 0x55d725eccb40 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f35017fa640 1 -- 172.21.15.33:0/3186385567 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f351009c2c0 con 0x55d725eccb40 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.215+0000 7f35017fa640 1 -- 172.21.15.33:0/3186385567 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510098230 con 0x55d725eccb40 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.216+0000 7f3524205740 1 -- 172.21.15.33:0/3186385567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 msgr2=0x55d725ed1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.216+0000 7f3524205740 1 --2- 172.21.15.33:0/3186385567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 0x55d725ed1900 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f3510007590 tx=0x7f3510013840 comp rx=0 tx=0).stop 2024-09-16T00:28:40.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.216+0000 7f3524205740 1 -- 172.21.15.33:0/3186385567 shutdown_connections 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.216+0000 7f3524205740 1 --2- 172.21.15.33:0/3186385567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725eed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.216+0000 7f3524205740 1 --2- 172.21.15.33:0/3186385567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725eccb40 0x55d725ed1900 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.216+0000 7f3524205740 1 --2- 172.21.15.33:0/3186385567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725ecb5d0 0x55d725ee22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.216+0000 7f3524205740 1 -- 172.21.15.33:0/3186385567 >> 172.21.15.33:0/3186385567 conn(0x55d725e94310 msgr2=0x55d725ed50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.216+0000 7f3524205740 1 -- 172.21.15.33:0/3186385567 shutdown_connections 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.216+0000 7f3524205740 1 -- 172.21.15.33:0/3186385567 wait complete. 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.216+0000 7f3524205740 1 -- start start 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.216+0000 7f3524205740 1 Processor -- start 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f3524205740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ecb5d0 0x55d725eda330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f3524205740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725eccb40 0x55d725eda870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f3524205740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725edadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f3524205740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f351c1afa70 con 0x55d725ecb5d0 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f3524205740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f351c1b0ee0 con 0x55d725eccb40 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f3524205740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f351c1b0140 con 0x55d725ed6370 2024-09-16T00:28:40.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f3524205740 10 client.0 _ll_register_callbacks cb 0x7f351404c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:40.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f351b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ecb5d0 0x55d725eda330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f351b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ecb5d0 0x55d725eda330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55632/0 (socket says 172.21.15.33:55632) 2024-09-16T00:28:40.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f351b7fe640 1 -- 172.21.15.33:0/1506248005 learned_addr learned my addr 172.21.15.33:0/1506248005 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:40.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f351affd640 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725eccb40 0x55d725eda870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f351b7fe640 1 -- 172.21.15.33:0/1506248005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 msgr2=0x55d725edadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:40.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f351bfff640 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725edadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f351b7fe640 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725edadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f351b7fe640 1 -- 172.21.15.33:0/1506248005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725eccb40 msgr2=0x55d725eda870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:40.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f351b7fe640 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725eccb40 0x55d725eda870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f351b7fe640 1 -- 172.21.15.33:0/1506248005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d725f4f920 con 0x55d725ecb5d0 2024-09-16T00:28:40.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f351bfff640 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725edadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:28:40.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f351b7fe640 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ecb5d0 0x55d725eda330 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f350c005d90 tx=0x7f350c013b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bd94aac3e0deb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:40.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.217+0000 7f34faffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:40.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f34faffd640 1 -- 172.21.15.33:0/1506248005 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c00fa70 con 0x55d725ecb5d0 2024-09-16T00:28:40.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f34faffd640 1 -- 172.21.15.33:0/1506248005 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f350c00fc10 con 0x55d725ecb5d0 2024-09-16T00:28:40.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f3524205740 1 -- 172.21.15.33:0/1506248005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f351009c2c0 con 0x55d725ecb5d0 2024-09-16T00:28:40.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f34faffd640 1 -- 172.21.15.33:0/1506248005 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f350c02d050 con 0x55d725ecb5d0 2024-09-16T00:28:40.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f3524205740 10 client.17523 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:40.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f3524205740 1 -- 172.21.15.33:0/1506248005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d725f50b50 con 0x55d725ecb5d0 2024-09-16T00:28:40.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f34faffd640 1 -- 172.21.15.33:0/1506248005 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f351009c2c0 con 0x55d725ecb5d0 2024-09-16T00:28:40.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f34faffd640 1 -- 172.21.15.33:0/1506248005 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f350c004c80 con 0x55d725ecb5d0 2024-09-16T00:28:40.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f3524205740 10 client.17523 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:40.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f3524205740 10 client.17523 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f3524205740 20 client.17523 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f3524205740 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d725f512a0 0x55d725f536c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f3524205740 4 client.17523 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.218+0000 7f3524205740 1 -- 172.21.15.33:0/1506248005 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d725e38d70 con 0x55d725f512a0 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f351affd640 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d725f512a0 0x55d725f536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f351affd640 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d725f512a0 0x55d725f536c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d840f565c05d8fcc server_cookie=9bdba6e2f3e6d208 in_seq=0 out_seq=0 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f34faffd640 10 client.17523 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f34faffd640 1 -- 172.21.15.33:0/1506248005 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d725e38d70 con 0x55d725f512a0 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f34faffd640 10 client.17523 handle_command_reply: tid=0 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f3524205740 1 client.17523 shutdown 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f3524205740 1 -- 172.21.15.33:0/1506248005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ecb5d0 msgr2=0x55d725eda330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f3524205740 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ecb5d0 0x55d725eda330 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f350c005d90 tx=0x7f350c013b20 comp rx=0 tx=0).stop 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f3524205740 1 -- 172.21.15.33:0/1506248005 shutdown_connections 2024-09-16T00:28:40.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f3524205740 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d725f512a0 0x55d725f536c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f3524205740 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d725ed6370 0x55d725edadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f3524205740 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d725eccb40 0x55d725eda870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f3524205740 1 --2- 172.21.15.33:0/1506248005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d725ecb5d0 0x55d725eda330 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f34faffd640 0 client.17523 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:28:40.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f3524205740 1 -- 172.21.15.33:0/1506248005 >> 172.21.15.33:0/1506248005 conn(0x55d725e94310 msgr2=0x55d725ecd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:40.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f3524205740 1 -- 172.21.15.33:0/1506248005 shutdown_connections 2024-09-16T00:28:40.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f3524205740 1 -- 172.21.15.33:0/1506248005 wait complete. 2024-09-16T00:28:40.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.219+0000 7f3524205740 20 client.17523 trim_cache size 0 max 16384 2024-09-16T00:28:40.239 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:28:40.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.220+0000 7f3521e2c640 1 -- 172.21.15.33:0/1959331862 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35040a3240 msgr2=0x7f35040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:40.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.220+0000 7f3521e2c640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35040a3240 0x7f35040a5700 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3514047000 tx=0x7f3514049000 comp rx=0 tx=0).stop 2024-09-16T00:28:40.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.220+0000 7f3521e2c640 1 -- 172.21.15.33:0/1959331862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c102ac0 msgr2=0x7f351c19ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:40.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.220+0000 7f3521e2c640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c102ac0 0x7f351c19ab60 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f351000c890 tx=0x7f351000cd50 comp rx=0 tx=0).stop 2024-09-16T00:28:40.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.220+0000 7f3521e2c640 1 -- 172.21.15.33:0/1959331862 shutdown_connections 2024-09-16T00:28:40.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.220+0000 7f3521e2c640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35040a3240 0x7f35040a5700 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.220+0000 7f3521e2c640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f351c102ea0 0x7f351c19eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.220+0000 7f3521e2c640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f351c102ac0 0x7f351c19ab60 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.220+0000 7f3521e2c640 1 --2- 172.21.15.33:0/1959331862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c100c10 0x7f351c19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:40.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.220+0000 7f3521e2c640 1 -- 172.21.15.33:0/1959331862 >> 172.21.15.33:0/1959331862 conn(0x7f351c077210 msgr2=0x7f351c0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:40.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.220+0000 7f3521e2c640 1 -- 172.21.15.33:0/1959331862 shutdown_connections 2024-09-16T00:28:40.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:40.220+0000 7f3521e2c640 1 -- 172.21.15.33:0/1959331862 wait complete. 2024-09-16T00:28:40.682 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.06 MB/sec execute 468 sec latency 13069.380 ms 2024-09-16T00:28:40.980 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:28:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:41.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:41 smithi083 ceph-mon[35242]: pgmap v696: 129 pgs: 129 active+clean; 268 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 3.4 MiB/s wr, 131 op/s 2024-09-16T00:28:41.682 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.04 MB/sec execute 469 sec latency 14069.406 ms 2024-09-16T00:28:41.689 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:41 smithi033 ceph-mon[31243]: pgmap v696: 129 pgs: 129 active+clean; 268 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 3.4 MiB/s wr, 131 op/s 2024-09-16T00:28:41.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:41 smithi060 ceph-mon[35185]: pgmap v696: 129 pgs: 129 active+clean; 268 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 3.4 MiB/s wr, 131 op/s 2024-09-16T00:28:41.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.773+0000 7fdf04f9f640 1 -- 172.21.15.33:0/992591836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf00100e20 msgr2=0x7fdf00101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:41.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.773+0000 7fdf04f9f640 1 --2- 172.21.15.33:0/992591836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf00100e20 0x7fdf00101200 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fdef0009900 tx=0x7fdef002a480 comp rx=0 tx=0).stop 2024-09-16T00:28:41.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.773+0000 7fdf04f9f640 1 -- 172.21.15.33:0/992591836 shutdown_connections 2024-09-16T00:28:41.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.773+0000 7fdf04f9f640 1 --2- 172.21.15.33:0/992591836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf00102ef0 0x7fdf0010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.773+0000 7fdf04f9f640 1 --2- 172.21.15.33:0/992591836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf00102570 0x7fdf001029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.773+0000 7fdf04f9f640 1 --2- 172.21.15.33:0/992591836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf00100e20 0x7fdf00101200 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.773+0000 7fdf04f9f640 1 -- 172.21.15.33:0/992591836 >> 172.21.15.33:0/992591836 conn(0x7fdf000fc7d0 msgr2=0x7fdf000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:41.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.773+0000 7fdf04f9f640 1 -- 172.21.15.33:0/992591836 shutdown_connections 2024-09-16T00:28:41.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.773+0000 7fdf04f9f640 1 -- 172.21.15.33:0/992591836 wait complete. 2024-09-16T00:28:41.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdf04f9f640 1 Processor -- start 2024-09-16T00:28:41.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdf04f9f640 1 -- start start 2024-09-16T00:28:41.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdf04f9f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf00100e20 0x7fdf001984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdf04f9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf00102570 0x7fdf00198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdf04f9f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf00102ef0 0x7fdf0019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdf04f9f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdf00112060 con 0x7fdf00102570 2024-09-16T00:28:41.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdf04f9f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdf00111ee0 con 0x7fdf00100e20 2024-09-16T00:28:41.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdf04f9f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf001121e0 con 0x7fdf00102ef0 2024-09-16T00:28:41.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdef7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf00102570 0x7fdf00198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdef7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf00102570 0x7fdf00198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36106/0 (socket says 172.21.15.33:36106) 2024-09-16T00:28:41.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdef7fff640 1 -- 172.21.15.33:0/1679597686 learned_addr learned my addr 172.21.15.33:0/1679597686 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:41.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdefe59c640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf00100e20 0x7fdf001984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdefed9d640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf00102ef0 0x7fdf0019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdef7fff640 1 -- 172.21.15.33:0/1679597686 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf00102ef0 msgr2=0x7fdf0019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:41.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdef7fff640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf00102ef0 0x7fdf0019cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdef7fff640 1 -- 172.21.15.33:0/1679597686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf00100e20 msgr2=0x7fdf001984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:41.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdef7fff640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf00100e20 0x7fdf001984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdef7fff640 1 -- 172.21.15.33:0/1679597686 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf0019d490 con 0x7fdf00102570 2024-09-16T00:28:41.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.774+0000 7fdef7fff640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf00102570 0x7fdf00198a20 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fdee800c920 tx=0x7fdee800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:41.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.775+0000 7fdef77fe640 1 -- 172.21.15.33:0/1679597686 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee8007c70 con 0x7fdf00102570 2024-09-16T00:28:41.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.775+0000 7fdf04f9f640 1 -- 172.21.15.33:0/1679597686 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf0019d780 con 0x7fdf00102570 2024-09-16T00:28:41.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.775+0000 7fdef77fe640 1 -- 172.21.15.33:0/1679597686 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdee8007e10 con 0x7fdf00102570 2024-09-16T00:28:41.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.775+0000 7fdef77fe640 1 -- 172.21.15.33:0/1679597686 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee8018890 con 0x7fdf00102570 2024-09-16T00:28:41.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.775+0000 7fdf04f9f640 1 -- 172.21.15.33:0/1679597686 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf001a5060 con 0x7fdf00102570 2024-09-16T00:28:41.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.777+0000 7fdf07378740 1 Processor -- start 2024-09-16T00:28:41.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.777+0000 7fdef77fe640 1 -- 172.21.15.33:0/1679597686 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fdee8018a70 con 0x7fdf00102570 2024-09-16T00:28:41.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdf07378740 1 -- start start 2024-09-16T00:28:41.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdf07378740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba38b40 0x559feba37090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdf07378740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba375d0 0x559feba37a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdf07378740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba46780 0x559feba48b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdf07378740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559feb99d2f0 con 0x559feba375d0 2024-09-16T00:28:41.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdf07378740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559feb944cf0 con 0x559feba46780 2024-09-16T00:28:41.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdf07378740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559feb9992b0 con 0x559feba38b40 2024-09-16T00:28:41.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdefe59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba375d0 0x559feba37a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdefe59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba375d0 0x559feba37a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36120/0 (socket says 172.21.15.33:36120) 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdefe59c640 1 -- 172.21.15.33:0/4283702548 learned_addr learned my addr 172.21.15.33:0/4283702548 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdefed9d640 1 --2- 172.21.15.33:0/4283702548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba38b40 0x559feba37090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdef7fff640 1 --2- 172.21.15.33:0/4283702548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba46780 0x559feba48b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdefe59c640 1 -- 172.21.15.33:0/4283702548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba38b40 msgr2=0x559feba37090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdefe59c640 1 --2- 172.21.15.33:0/4283702548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba38b40 0x559feba37090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdef77fe640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdeec0a3240 0x7fdeec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdefe59c640 1 -- 172.21.15.33:0/4283702548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba46780 msgr2=0x559feba48b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdefe59c640 1 --2- 172.21.15.33:0/4283702548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba46780 0x559feba48b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdefe59c640 1 -- 172.21.15.33:0/4283702548 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559feb9293a0 con 0x559feba375d0 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.778+0000 7fdefed9d640 1 --2- 172.21.15.33:0/4283702548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba38b40 0x559feba37090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.779+0000 7fdefed9d640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdeec0a3240 0x7fdeec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.779+0000 7fdef77fe640 1 -- 172.21.15.33:0/1679597686 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fdee80cfee0 con 0x7fdf00102570 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.779+0000 7fdefe59c640 1 --2- 172.21.15.33:0/4283702548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba375d0 0x559feba37a10 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fdef0009900 tx=0x7fdef0004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c24ade707982a5a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.779+0000 7fdef4ff9640 1 -- 172.21.15.33:0/4283702548 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0035030 con 0x559feba375d0 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.779+0000 7fdef4ff9640 1 -- 172.21.15.33:0/4283702548 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdef0040070 con 0x559feba375d0 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.779+0000 7fdef4ff9640 1 -- 172.21.15.33:0/4283702548 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0031b70 con 0x559feba375d0 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.780+0000 7fdf07378740 1 -- 172.21.15.33:0/4283702548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba375d0 msgr2=0x559feba37a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.780+0000 7fdefed9d640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdeec0a3240 0x7fdeec0a5700 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fdef8005fd0 tx=0x7fdef8009760 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.780+0000 7fdf07378740 1 --2- 172.21.15.33:0/4283702548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba375d0 0x559feba37a10 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fdef0009900 tx=0x7fdef0004170 comp rx=0 tx=0).stop 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.780+0000 7fdf07378740 1 -- 172.21.15.33:0/4283702548 shutdown_connections 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.780+0000 7fdf07378740 1 --2- 172.21.15.33:0/4283702548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba46780 0x559feba48b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.780+0000 7fdf07378740 1 --2- 172.21.15.33:0/4283702548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba375d0 0x559feba37a10 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.780+0000 7fdf07378740 1 --2- 172.21.15.33:0/4283702548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba38b40 0x559feba37090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.780+0000 7fdf07378740 1 -- 172.21.15.33:0/4283702548 >> 172.21.15.33:0/4283702548 conn(0x559feba00310 msgr2=0x559feba33c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.780+0000 7fdf07378740 1 -- 172.21.15.33:0/4283702548 shutdown_connections 2024-09-16T00:28:41.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.780+0000 7fdf07378740 1 -- 172.21.15.33:0/4283702548 wait complete. 2024-09-16T00:28:41.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.780+0000 7fdf07378740 1 -- start start 2024-09-16T00:28:41.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.780+0000 7fdf07378740 1 Processor -- start 2024-09-16T00:28:41.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdf07378740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba461b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdf07378740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 0x559feba43460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdf07378740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba46780 0x559feba43bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdf07378740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559feba49420 con 0x559feba38b40 2024-09-16T00:28:41.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdf07378740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559feb958060 con 0x559feba375d0 2024-09-16T00:28:41.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdf07378740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559feba495a0 con 0x559feba46780 2024-09-16T00:28:41.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdf07378740 10 client.0 _ll_register_callbacks cb 0x559feb6ab8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:41.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdef7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 0x559feba43460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdefe59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdef7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 0x559feba43460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36124/0 (socket says 172.21.15.33:36124) 2024-09-16T00:28:41.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdef7fff640 1 -- 172.21.15.33:0/2033245035 learned_addr learned my addr 172.21.15.33:0/2033245035 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:41.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdefed9d640 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba46780 0x559feba43bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdef7fff640 1 -- 172.21.15.33:0/2033245035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba46780 msgr2=0x559feba43bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:41.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdef7fff640 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba46780 0x559feba43bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdef7fff640 1 -- 172.21.15.33:0/2033245035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 msgr2=0x559feba461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:41.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdef7fff640 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba461b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdef7fff640 1 -- 172.21.15.33:0/2033245035 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559feb70e3c0 con 0x559feba38b40 2024-09-16T00:28:41.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdefed9d640 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba46780 0x559feba43bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:28:41.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdef7fff640 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 0x559feba43460 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fdee800c8f0 tx=0x7fdee80983e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a26ced36b106d4d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:41.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdee5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:41.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.781+0000 7fdee5ffb640 1 -- 172.21.15.33:0/2033245035 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee809c780 con 0x559feba38b40 2024-09-16T00:28:41.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdee5ffb640 1 -- 172.21.15.33:0/2033245035 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fdee8018d60 con 0x559feba38b40 2024-09-16T00:28:41.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdf07378740 1 -- 172.21.15.33:0/2033245035 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdef0040070 con 0x559feba38b40 2024-09-16T00:28:41.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdee5ffb640 1 -- 172.21.15.33:0/2033245035 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdee809c080 con 0x559feba38b40 2024-09-16T00:28:41.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdf07378740 10 client.17535 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:41.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdf07378740 1 -- 172.21.15.33:0/2033245035 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559feba54fb0 con 0x559feba38b40 2024-09-16T00:28:41.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdee5ffb640 1 -- 172.21.15.33:0/2033245035 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdef0040070 con 0x559feba38b40 2024-09-16T00:28:41.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdee5ffb640 1 -- 172.21.15.33:0/2033245035 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fdee80a00b0 con 0x559feba38b40 2024-09-16T00:28:41.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdf07378740 10 client.17535 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:41.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdf07378740 10 client.17535 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:28:41.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdf07378740 20 client.17535 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:41.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdf07378740 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x559feba56470 0x559feba58860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdf07378740 4 client.17535 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdf07378740 1 -- 172.21.15.33:0/2033245035 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559feb9aa860 con 0x559feba56470 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.782+0000 7fdefe59c640 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x559feba56470 0x559feba58860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.783+0000 7fdefe59c640 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x559feba56470 0x559feba58860 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b5d9d07106a2db63 server_cookie=633112a448bafd65 in_seq=0 out_seq=0 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.783+0000 7fdee5ffb640 10 client.17535 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.783+0000 7fdee5ffb640 1 -- 172.21.15.33:0/2033245035 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559feb9aa860 con 0x559feba56470 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.783+0000 7fdee5ffb640 10 client.17535 handle_command_reply: tid=0 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdf07378740 1 client.17535 shutdown 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdf07378740 1 -- 172.21.15.33:0/2033245035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 msgr2=0x559feba43460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdf07378740 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 0x559feba43460 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fdee800c8f0 tx=0x7fdee80983e0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdf07378740 1 -- 172.21.15.33:0/2033245035 shutdown_connections 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdf07378740 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba46780 0x559feba43bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdf07378740 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 0x559feba43460 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdf07378740 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x559feba56470 0x559feba58860 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdf07378740 1 --2- 172.21.15.33:0/2033245035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdf07378740 1 -- 172.21.15.33:0/2033245035 >> 172.21.15.33:0/2033245035 conn(0x559feba00310 msgr2=0x559feba39960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:41.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdee5ffb640 0 client.17535 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:28:41.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdf07378740 1 -- 172.21.15.33:0/2033245035 shutdown_connections 2024-09-16T00:28:41.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdf07378740 1 -- 172.21.15.33:0/2033245035 wait complete. 2024-09-16T00:28:41.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.784+0000 7fdf07378740 20 client.17535 trim_cache size 0 max 16384 2024-09-16T00:28:41.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdf07378740 1 Processor -- start 2024-09-16T00:28:41.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdf07378740 1 -- start start 2024-09-16T00:28:41.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdf07378740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba42d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdf07378740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 0x559feba42440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdf07378740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba46780 0x559feba47b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdf07378740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdf001af6c0 con 0x559feba38b40 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdf07378740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdf001acf90 con 0x559feba375d0 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdf07378740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf001aade0 con 0x559feba46780 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdef7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 0x559feba42440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdef7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 0x559feba42440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36126/0 (socket says 172.21.15.33:36126) 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdef7fff640 1 -- 172.21.15.33:0/3233703749 learned_addr learned my addr 172.21.15.33:0/3233703749 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdefed9d640 1 --2- 172.21.15.33:0/3233703749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba46780 0x559feba47b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdefe59c640 1 --2- 172.21.15.33:0/3233703749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba42d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdef7fff640 1 -- 172.21.15.33:0/3233703749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba46780 msgr2=0x559feba47b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdef7fff640 1 --2- 172.21.15.33:0/3233703749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba46780 0x559feba47b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdef7fff640 1 -- 172.21.15.33:0/3233703749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 msgr2=0x559feba42d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdef7fff640 1 --2- 172.21.15.33:0/3233703749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba42d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.797+0000 7fdef7fff640 1 -- 172.21.15.33:0/3233703749 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559feba58e90 con 0x559feba38b40 2024-09-16T00:28:41.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.798+0000 7fdefed9d640 1 --2- 172.21.15.33:0/3233703749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba46780 0x559feba47b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:28:41.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.798+0000 7fdef7fff640 1 --2- 172.21.15.33:0/3233703749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 0x559feba42440 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fdee809a330 tx=0x7fdee8098450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393f6c8280dce33e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:41.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.798+0000 7fdee7fff640 1 -- 172.21.15.33:0/3233703749 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee80a4cf0 con 0x559feba38b40 2024-09-16T00:28:41.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.798+0000 7fdee7fff640 1 -- 172.21.15.33:0/3233703749 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdee809c780 con 0x559feba38b40 2024-09-16T00:28:41.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.798+0000 7fdee7fff640 1 -- 172.21.15.33:0/3233703749 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee80a0060 con 0x559feba38b40 2024-09-16T00:28:41.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.798+0000 7fdf07378740 1 -- 172.21.15.33:0/3233703749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 msgr2=0x559feba42440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:41.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.798+0000 7fdf07378740 1 --2- 172.21.15.33:0/3233703749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 0x559feba42440 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fdee809a330 tx=0x7fdee8098450 comp rx=0 tx=0).stop 2024-09-16T00:28:41.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.798+0000 7fdf07378740 1 -- 172.21.15.33:0/3233703749 shutdown_connections 2024-09-16T00:28:41.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.798+0000 7fdf07378740 1 --2- 172.21.15.33:0/3233703749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba46780 0x559feba47b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.798+0000 7fdf07378740 1 --2- 172.21.15.33:0/3233703749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba38b40 0x559feba42440 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.798+0000 7fdf07378740 1 --2- 172.21.15.33:0/3233703749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba42d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.798+0000 7fdf07378740 1 -- 172.21.15.33:0/3233703749 >> 172.21.15.33:0/3233703749 conn(0x559feba00310 msgr2=0x559feba41d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:41.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdf07378740 1 -- 172.21.15.33:0/3233703749 shutdown_connections 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdf07378740 1 -- 172.21.15.33:0/3233703749 wait complete. 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdf07378740 1 -- start start 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdf07378740 1 Processor -- start 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdf07378740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba5dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdf07378740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba38b40 0x559feba5e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdf07378740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba46780 0x559feba5e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdf07378740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdf001ab450 con 0x559feba46780 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdf07378740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdf001ac8c0 con 0x559feba375d0 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdf07378740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf001abb20 con 0x559feba38b40 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdf07378740 10 client.0 _ll_register_callbacks cb 0x7fdee80d36b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdefed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba46780 0x559feba5e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdefed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba46780 0x559feba5e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36132/0 (socket says 172.21.15.33:36132) 2024-09-16T00:28:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdefed9d640 1 -- 172.21.15.33:0/1744512696 learned_addr learned my addr 172.21.15.33:0/1744512696 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.799+0000 7fdefe59c640 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba5dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdef7fff640 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba38b40 0x559feba5e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdefed9d640 1 -- 172.21.15.33:0/1744512696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba38b40 msgr2=0x559feba5e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdefed9d640 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba38b40 0x559feba5e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdefed9d640 1 -- 172.21.15.33:0/1744512696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 msgr2=0x559feba5dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdefed9d640 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba5dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdefed9d640 1 -- 172.21.15.33:0/1744512696 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559feba4cd20 con 0x559feba46780 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdefe59c640 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba5dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdefed9d640 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba46780 0x559feba5e6d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fdef8010b10 tx=0x7fdef8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe6a5e1f791faf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdee5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdee5ffb640 1 -- 172.21.15.33:0/1744512696 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef802c060 con 0x559feba46780 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdee5ffb640 1 -- 172.21.15.33:0/1744512696 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fdef801cd80 con 0x559feba46780 2024-09-16T00:28:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdf07378740 1 -- 172.21.15.33:0/1744512696 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdee809c780 con 0x559feba46780 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdee5ffb640 1 -- 172.21.15.33:0/1744512696 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdef8033050 con 0x559feba46780 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdf07378740 10 client.17541 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.800+0000 7fdf07378740 1 -- 172.21.15.33:0/1744512696 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559febac13e0 con 0x559feba46780 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.801+0000 7fdee5ffb640 1 -- 172.21.15.33:0/1744512696 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdee809c780 con 0x559feba46780 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.801+0000 7fdee5ffb640 1 -- 172.21.15.33:0/1744512696 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fdef8026070 con 0x559feba46780 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.801+0000 7fdf07378740 10 client.17541 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.801+0000 7fdf07378740 10 client.17541 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.801+0000 7fdf07378740 20 client.17541 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.801+0000 7fdf07378740 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x559febac1b60 0x559febac3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.801+0000 7fdf07378740 4 client.17541 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.801+0000 7fdf07378740 1 -- 172.21.15.33:0/1744512696 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559feb9a4d70 con 0x559febac1b60 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.801+0000 7fdefe59c640 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x559febac1b60 0x559febac3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.801+0000 7fdefe59c640 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x559febac1b60 0x559febac3f80 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=988daf021985049a server_cookie=1d195139dde28844 in_seq=0 out_seq=0 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.801+0000 7fdee5ffb640 10 client.17541 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdee5ffb640 1 -- 172.21.15.33:0/1744512696 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559feb9a4d70 con 0x559febac1b60 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdee5ffb640 10 client.17541 handle_command_reply: tid=0 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdf07378740 1 client.17541 shutdown 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdf07378740 1 -- 172.21.15.33:0/1744512696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba46780 msgr2=0x559feba5e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdf07378740 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba46780 0x559feba5e6d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fdef8010b10 tx=0x7fdef8004a40 comp rx=0 tx=0).stop 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdf07378740 1 -- 172.21.15.33:0/1744512696 shutdown_connections 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdf07378740 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559feba46780 0x559feba5e6d0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdf07378740 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559feba38b40 0x559feba5e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdf07378740 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x559febac1b60 0x559febac3f80 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdee5ffb640 0 client.17541 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdf07378740 1 --2- 172.21.15.33:0/1744512696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559feba375d0 0x559feba5dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdf07378740 1 -- 172.21.15.33:0/1744512696 >> 172.21.15.33:0/1744512696 conn(0x559feba00310 msgr2=0x559feba39930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdf07378740 1 -- 172.21.15.33:0/1744512696 shutdown_connections 2024-09-16T00:28:41.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdf07378740 1 -- 172.21.15.33:0/1744512696 wait complete. 2024-09-16T00:28:41.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.802+0000 7fdf07378740 20 client.17541 trim_cache size 0 max 16384 2024-09-16T00:28:41.819 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:28:41.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.803+0000 7fdf04f9f640 1 -- 172.21.15.33:0/1679597686 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdeec0a3240 msgr2=0x7fdeec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:41.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.803+0000 7fdf04f9f640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdeec0a3240 0x7fdeec0a5700 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fdef8005fd0 tx=0x7fdef8009760 comp rx=0 tx=0).stop 2024-09-16T00:28:41.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.803+0000 7fdf04f9f640 1 -- 172.21.15.33:0/1679597686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf00102570 msgr2=0x7fdf00198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:41.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.803+0000 7fdf04f9f640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf00102570 0x7fdf00198a20 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fdee800c920 tx=0x7fdee800cde0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.803+0000 7fdf04f9f640 1 -- 172.21.15.33:0/1679597686 shutdown_connections 2024-09-16T00:28:41.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.803+0000 7fdf04f9f640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdeec0a3240 0x7fdeec0a5700 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.803+0000 7fdf04f9f640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf00102ef0 0x7fdf0019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.803+0000 7fdf04f9f640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf00102570 0x7fdf00198a20 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.803+0000 7fdf04f9f640 1 --2- 172.21.15.33:0/1679597686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf00100e20 0x7fdf001984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:41.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.803+0000 7fdf04f9f640 1 -- 172.21.15.33:0/1679597686 >> 172.21.15.33:0/1679597686 conn(0x7fdf000fc7d0 msgr2=0x7fdf001068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:41.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.803+0000 7fdf04f9f640 1 -- 172.21.15.33:0/1679597686 shutdown_connections 2024-09-16T00:28:41.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:41.803+0000 7fdf04f9f640 1 -- 172.21.15.33:0/1679597686 wait complete. 2024-09-16T00:28:42.633 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:28:42.682 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.02 MB/sec execute 470 sec latency 15069.437 ms 2024-09-16T00:28:42.911 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:43.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.451+0000 7f8cffa83640 1 -- 172.21.15.33:0/1131962092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8102780 msgr2=0x7f8cf81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:43.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.451+0000 7f8cffa83640 1 --2- 172.21.15.33:0/1131962092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8102780 0x7f8cf81067d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009900 tx=0x7f8cec02a410 comp rx=0 tx=0).stop 2024-09-16T00:28:43.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.451+0000 7f8cffa83640 1 -- 172.21.15.33:0/1131962092 shutdown_connections 2024-09-16T00:28:43.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.451+0000 7f8cffa83640 1 --2- 172.21.15.33:0/1131962092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf8106f00 0x7f8cf810ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.451+0000 7f8cffa83640 1 --2- 172.21.15.33:0/1131962092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8102780 0x7f8cf81067d0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.451+0000 7f8cffa83640 1 --2- 172.21.15.33:0/1131962092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cf8101dd0 0x7f8cf81021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.451+0000 7f8cffa83640 1 -- 172.21.15.33:0/1131962092 >> 172.21.15.33:0/1131962092 conn(0x7f8cf80fd8b0 msgr2=0x7f8cf80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:43.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.451+0000 7f8cffa83640 1 -- 172.21.15.33:0/1131962092 shutdown_connections 2024-09-16T00:28:43.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.451+0000 7f8cffa83640 1 -- 172.21.15.33:0/1131962092 wait complete. 2024-09-16T00:28:43.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cffa83640 1 Processor -- start 2024-09-16T00:28:43.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cffa83640 1 -- start start 2024-09-16T00:28:43.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cffa83640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cf8101dd0 0x7f8cf8198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cffa83640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8102780 0x7f8cf8198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cffa83640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf8106f00 0x7f8cf819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cffa83640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf8111f80 con 0x7f8cf8102780 2024-09-16T00:28:43.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cffa83640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf8111e00 con 0x7f8cf8106f00 2024-09-16T00:28:43.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cffa83640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf8112100 con 0x7f8cf8101dd0 2024-09-16T00:28:43.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cfd01e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8102780 0x7f8cf8198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cfd01e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8102780 0x7f8cf8198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36156/0 (socket says 172.21.15.33:36156) 2024-09-16T00:28:43.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cfd01e640 1 -- 172.21.15.33:0/99542288 learned_addr learned my addr 172.21.15.33:0/99542288 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:43.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cfd81f640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cf8101dd0 0x7f8cf8198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cfe020640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf8106f00 0x7f8cf819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cfd01e640 1 -- 172.21.15.33:0/99542288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cf8101dd0 msgr2=0x7f8cf8198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:43.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cfd01e640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cf8101dd0 0x7f8cf8198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cfd01e640 1 -- 172.21.15.33:0/99542288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf8106f00 msgr2=0x7f8cf819cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:43.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cfd01e640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf8106f00 0x7f8cf819cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cfd01e640 1 -- 172.21.15.33:0/99542288 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf819d3e0 con 0x7f8cf8102780 2024-09-16T00:28:43.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.452+0000 7f8cfe020640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf8106f00 0x7f8cf819cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:28:43.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.453+0000 7f8cfd01e640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8102780 0x7f8cf8198970 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f8cec005ab0 tx=0x7f8cec0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:43.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.453+0000 7f8cf6ffd640 1 -- 172.21.15.33:0/99542288 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec03f030 con 0x7f8cf8102780 2024-09-16T00:28:43.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.453+0000 7f8cffa83640 1 -- 172.21.15.33:0/99542288 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf819d670 con 0x7f8cf8102780 2024-09-16T00:28:43.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.453+0000 7f8cf6ffd640 1 -- 172.21.15.33:0/99542288 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cec003ce0 con 0x7f8cf8102780 2024-09-16T00:28:43.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.453+0000 7f8cf6ffd640 1 -- 172.21.15.33:0/99542288 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec035070 con 0x7f8cf8102780 2024-09-16T00:28:43.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.453+0000 7f8cffa83640 1 -- 172.21.15.33:0/99542288 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf81a4fb0 con 0x7f8cf8102780 2024-09-16T00:28:43.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.455+0000 7f8cf6ffd640 1 -- 172.21.15.33:0/99542288 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8cec042030 con 0x7f8cf8102780 2024-09-16T00:28:43.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.455+0000 7f8d01e5c740 1 Processor -- start 2024-09-16T00:28:43.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8d01e5c740 1 -- start start 2024-09-16T00:28:43.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8d01e5c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7e5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8d01e5c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7e55d0 0x55f90e7e5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8d01e5c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7f0370 0x55f90e7f2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8d01e5c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f90e74b2f0 con 0x55f90e7e6b40 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8d01e5c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f90e6f2cf0 con 0x55f90e7f0370 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8d01e5c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f90e7472b0 con 0x55f90e7e55d0 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8cfe020640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7e5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8cfe020640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7e5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36170/0 (socket says 172.21.15.33:36170) 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8cfe020640 1 -- 172.21.15.33:0/2857381902 learned_addr learned my addr 172.21.15.33:0/2857381902 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8cfd81f640 1 --2- 172.21.15.33:0/2857381902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7e55d0 0x55f90e7e5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8cfd01e640 1 --2- 172.21.15.33:0/2857381902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7f0370 0x55f90e7f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8cf6ffd640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ce00a3240 0x7f8ce00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8cfe020640 1 -- 172.21.15.33:0/2857381902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7e55d0 msgr2=0x55f90e7e5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:43.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8cfe020640 1 --2- 172.21.15.33:0/2857381902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7e55d0 0x55f90e7e5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8cfe020640 1 -- 172.21.15.33:0/2857381902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7f0370 msgr2=0x55f90e7f2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8cfe020640 1 --2- 172.21.15.33:0/2857381902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7f0370 0x55f90e7f2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.456+0000 7f8cfe020640 1 -- 172.21.15.33:0/2857381902 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f90e6d73a0 con 0x55f90e7e6b40 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.457+0000 7f8cfd81f640 1 --2- 172.21.15.33:0/2857381902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7e55d0 0x55f90e7e5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.457+0000 7f8cf6ffd640 1 -- 172.21.15.33:0/99542288 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8cec0f0680 con 0x7f8cf8102780 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.457+0000 7f8cfe020640 1 --2- 172.21.15.33:0/2857381902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7e5090 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000bb60 tx=0x7f8cf000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef44a183488b875 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.457+0000 7f8cfe020640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ce00a3240 0x7f8ce00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.457+0000 7f8cdffff640 1 -- 172.21.15.33:0/2857381902 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0017620 con 0x55f90e7e6b40 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.457+0000 7f8cdffff640 1 -- 172.21.15.33:0/2857381902 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cf000f420 con 0x55f90e7e6b40 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.457+0000 7f8cdffff640 1 -- 172.21.15.33:0/2857381902 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0016630 con 0x55f90e7e6b40 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.458+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2857381902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 msgr2=0x55f90e7e5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.458+0000 7f8cfe020640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ce00a3240 0x7f8ce00a5700 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0018000 tx=0x7f8cf0016da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:43.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.458+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/2857381902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7e5090 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000bb60 tx=0x7f8cf000d710 comp rx=0 tx=0).stop 2024-09-16T00:28:43.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.458+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2857381902 shutdown_connections 2024-09-16T00:28:43.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.458+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/2857381902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7f0370 0x55f90e7f2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.458+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/2857381902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7e55d0 0x55f90e7e5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.458+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/2857381902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7e5090 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.458+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2857381902 >> 172.21.15.33:0/2857381902 conn(0x55f90e7ae310 msgr2=0x55f90e7e1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:43.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.458+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2857381902 shutdown_connections 2024-09-16T00:28:43.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.458+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2857381902 wait complete. 2024-09-16T00:28:43.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.458+0000 7f8d01e5c740 1 -- start start 2024-09-16T00:28:43.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.458+0000 7f8d01e5c740 1 Processor -- start 2024-09-16T00:28:43.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8d01e5c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7efcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8d01e5c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7ecf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8d01e5c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7f0370 0x55f90e7ed4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8d01e5c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f90e7f3070 con 0x55f90e7e6b40 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8d01e5c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f90e706060 con 0x55f90e7e55d0 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8d01e5c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f90e7f31f0 con 0x55f90e7f0370 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8d01e5c740 10 client.0 _ll_register_callbacks cb 0x55f90e4598c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cfd01e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7ecf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cfd01e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7ecf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36182/0 (socket says 172.21.15.33:36182) 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cfd01e640 1 -- 172.21.15.33:0/3447316208 learned_addr learned my addr 172.21.15.33:0/3447316208 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cfd81f640 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7efcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cfe020640 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7f0370 0x55f90e7ed4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cfd01e640 1 -- 172.21.15.33:0/3447316208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7f0370 msgr2=0x55f90e7ed4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:43.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cfd01e640 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7f0370 0x55f90e7ed4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cfd01e640 1 -- 172.21.15.33:0/3447316208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 msgr2=0x55f90e7efcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:43.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cfd01e640 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7efcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cfd01e640 1 -- 172.21.15.33:0/3447316208 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f90e4bc3c0 con 0x55f90e7e6b40 2024-09-16T00:28:43.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cfd01e640 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7ecf60 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0bc520 tx=0x7f8cec02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a6943d16056ad6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:43.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cdd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:43.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.459+0000 7f8cdd7fa640 1 -- 172.21.15.33:0/3447316208 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec049070 con 0x55f90e7e6b40 2024-09-16T00:28:43.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8d01e5c740 1 -- 172.21.15.33:0/3447316208 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cf000f420 con 0x55f90e7e6b40 2024-09-16T00:28:43.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8cdd7fa640 1 -- 172.21.15.33:0/3447316208 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8cec03bd70 con 0x55f90e7e6b40 2024-09-16T00:28:43.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8cdd7fa640 1 -- 172.21.15.33:0/3447316208 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8cec0bc820 con 0x55f90e7e6b40 2024-09-16T00:28:43.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8d01e5c740 10 client.17553 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:43.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8d01e5c740 1 -- 172.21.15.33:0/3447316208 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f90e7fea60 con 0x55f90e7e6b40 2024-09-16T00:28:43.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8cdd7fa640 1 -- 172.21.15.33:0/3447316208 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cf000f420 con 0x55f90e7e6b40 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8cdd7fa640 1 -- 172.21.15.33:0/3447316208 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f8cec0f02b0 con 0x55f90e7e6b40 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8d01e5c740 10 client.17553 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8d01e5c740 10 client.17553 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8d01e5c740 20 client.17553 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f90e7ffec0 0x55f90e8022d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8d01e5c740 4 client.17553 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8d01e5c740 1 -- 172.21.15.33:0/3447316208 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f90e758860 con 0x55f90e7ffec0 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.460+0000 7f8cfd81f640 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f90e7ffec0 0x55f90e8022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.461+0000 7f8cfd81f640 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f90e7ffec0 0x55f90e8022d0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=265bfee42a088e8 server_cookie=70a53f251f15eb7c in_seq=0 out_seq=0 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.461+0000 7f8cdd7fa640 10 client.17553 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.461+0000 7f8cdd7fa640 1 -- 172.21.15.33:0/3447316208 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f90e758860 con 0x55f90e7ffec0 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.461+0000 7f8cdd7fa640 10 client.17553 handle_command_reply: tid=0 2024-09-16T00:28:43.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.461+0000 7f8d01e5c740 1 client.17553 shutdown 2024-09-16T00:28:43.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.462+0000 7f8d01e5c740 1 -- 172.21.15.33:0/3447316208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 msgr2=0x55f90e7ecf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:43.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.462+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7ecf60 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0bc520 tx=0x7f8cec02a410 comp rx=0 tx=0).stop 2024-09-16T00:28:43.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.462+0000 7f8d01e5c740 1 -- 172.21.15.33:0/3447316208 shutdown_connections 2024-09-16T00:28:43.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.462+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f90e7ffec0 0x55f90e8022d0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.462+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7f0370 0x55f90e7ed4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.462+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7ecf60 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.462+0000 7f8cdd7fa640 0 client.17553 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:43.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.462+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/3447316208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7efcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.462+0000 7f8d01e5c740 1 -- 172.21.15.33:0/3447316208 >> 172.21.15.33:0/3447316208 conn(0x55f90e7ae310 msgr2=0x55f90e7e7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:43.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.462+0000 7f8d01e5c740 1 -- 172.21.15.33:0/3447316208 shutdown_connections 2024-09-16T00:28:43.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.462+0000 7f8d01e5c740 1 -- 172.21.15.33:0/3447316208 wait complete. 2024-09-16T00:28:43.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.462+0000 7f8d01e5c740 20 client.17553 trim_cache size 0 max 16384 2024-09-16T00:28:43.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8d01e5c740 1 Processor -- start 2024-09-16T00:28:43.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8d01e5c740 1 -- start start 2024-09-16T00:28:43.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8d01e5c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7fc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8d01e5c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7e6b40 0x55f90e7e8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8d01e5c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7f0370 0x55f90e7e8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8d01e5c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81b1930 con 0x55f90e7f0370 2024-09-16T00:28:43.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8d01e5c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81af200 con 0x55f90e7e55d0 2024-09-16T00:28:43.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8d01e5c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81ad050 con 0x55f90e7e6b40 2024-09-16T00:28:43.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8cfe020640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7f0370 0x55f90e7e8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8cfe020640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7f0370 0x55f90e7e8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36188/0 (socket says 172.21.15.33:36188) 2024-09-16T00:28:43.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8cfe020640 1 -- 172.21.15.33:0/1690465234 learned_addr learned my addr 172.21.15.33:0/1690465234 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:43.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8cfd81f640 1 --2- 172.21.15.33:0/1690465234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.475+0000 7f8cfd01e640 1 --2- 172.21.15.33:0/1690465234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7e6b40 0x55f90e7e8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.476+0000 7f8cfe020640 1 -- 172.21.15.33:0/1690465234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7e6b40 msgr2=0x55f90e7e8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.476+0000 7f8cfe020640 1 --2- 172.21.15.33:0/1690465234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7e6b40 0x55f90e7e8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.476+0000 7f8cfe020640 1 -- 172.21.15.33:0/1690465234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 msgr2=0x55f90e7fc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.476+0000 7f8cfe020640 1 --2- 172.21.15.33:0/1690465234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7fc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.476+0000 7f8cfe020640 1 -- 172.21.15.33:0/1690465234 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f90e7e8ba0 con 0x55f90e7f0370 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.476+0000 7f8cfe020640 1 --2- 172.21.15.33:0/1690465234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7f0370 0x55f90e7e8660 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000dc10 tx=0x7f8cf0025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65824ff60697af7a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.476+0000 7f8cdf7fe640 1 -- 172.21.15.33:0/1690465234 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0017620 con 0x55f90e7f0370 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.476+0000 7f8cdf7fe640 1 -- 172.21.15.33:0/1690465234 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cf0028040 con 0x55f90e7f0370 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.476+0000 7f8cdf7fe640 1 -- 172.21.15.33:0/1690465234 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf000e950 con 0x55f90e7f0370 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.477+0000 7f8d01e5c740 1 -- 172.21.15.33:0/1690465234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7f0370 msgr2=0x55f90e7e8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.477+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/1690465234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7f0370 0x55f90e7e8660 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000dc10 tx=0x7f8cf0025000 comp rx=0 tx=0).stop 2024-09-16T00:28:43.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.477+0000 7f8d01e5c740 1 -- 172.21.15.33:0/1690465234 shutdown_connections 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.477+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/1690465234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7f0370 0x55f90e7e8660 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.477+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/1690465234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7e6b40 0x55f90e7e8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.477+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/1690465234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7fc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.477+0000 7f8d01e5c740 1 -- 172.21.15.33:0/1690465234 >> 172.21.15.33:0/1690465234 conn(0x55f90e7ae310 msgr2=0x55f90e7ecf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.477+0000 7f8d01e5c740 1 -- 172.21.15.33:0/1690465234 shutdown_connections 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.477+0000 7f8d01e5c740 1 -- 172.21.15.33:0/1690465234 wait complete. 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.477+0000 7f8d01e5c740 1 -- start start 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.477+0000 7f8d01e5c740 1 Processor -- start 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8d01e5c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7f4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8d01e5c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7f4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8d01e5c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7f0370 0x55f90e7f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8d01e5c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81ad6c0 con 0x55f90e7e6b40 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8d01e5c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81aeb30 con 0x55f90e7e55d0 2024-09-16T00:28:43.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8d01e5c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81add90 con 0x55f90e7f0370 2024-09-16T00:28:43.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8d01e5c740 10 client.0 _ll_register_callbacks cb 0x7f8cec009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:43.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfd01e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7f4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfd81f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfd01e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7f4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36194/0 (socket says 172.21.15.33:36194) 2024-09-16T00:28:43.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfd01e640 1 -- 172.21.15.33:0/2711069917 learned_addr learned my addr 172.21.15.33:0/2711069917 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:43.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfd81f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7f4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:57630/0 (socket says 172.21.15.33:57630) 2024-09-16T00:28:43.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfe020640 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7f0370 0x55f90e7f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfd01e640 1 -- 172.21.15.33:0/2711069917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7f0370 msgr2=0x55f90e7f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfd01e640 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7f0370 0x55f90e7f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfd01e640 1 -- 172.21.15.33:0/2711069917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 msgr2=0x55f90e7f4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfd01e640 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7f4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfd01e640 1 -- 172.21.15.33:0/2711069917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f90e869b70 con 0x55f90e7e6b40 2024-09-16T00:28:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfd81f640 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cfd01e640 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7f4650 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0a71a0 tx=0x7f8cec0b9160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390c73bb5cd32484 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.478+0000 7f8cdd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8cdd7fa640 1 -- 172.21.15.33:0/2711069917 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec049030 con 0x55f90e7e6b40 2024-09-16T00:28:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8cdd7fa640 1 -- 172.21.15.33:0/2711069917 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8cec03bd70 con 0x55f90e7e6b40 2024-09-16T00:28:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2711069917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cf0028040 con 0x55f90e7e6b40 2024-09-16T00:28:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8cdd7fa640 1 -- 172.21.15.33:0/2711069917 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8cec0bc180 con 0x55f90e7e6b40 2024-09-16T00:28:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8d01e5c740 10 client.17559 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2711069917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f90e86ad90 con 0x55f90e7e6b40 2024-09-16T00:28:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8cdd7fa640 1 -- 172.21.15.33:0/2711069917 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cf0028040 con 0x55f90e7e6b40 2024-09-16T00:28:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8cdd7fa640 1 -- 172.21.15.33:0/2711069917 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f8cec031ba0 con 0x55f90e7e6b40 2024-09-16T00:28:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8d01e5c740 10 client.17559 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8d01e5c740 10 client.17559 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:28:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8d01e5c740 20 client.17559 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f90e86b2f0 0x55f90e86d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8d01e5c740 4 client.17559 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:28:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2711069917 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f90e752d70 con 0x55f90e86b2f0 2024-09-16T00:28:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.479+0000 7f8cfd81f640 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f90e86b2f0 0x55f90e86d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8cfd81f640 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f90e86b2f0 0x55f90e86d710 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ee26febccba7f8d server_cookie=4cdb26345be73e1c in_seq=0 out_seq=0 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8cdd7fa640 10 client.17559 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8cdd7fa640 1 -- 172.21.15.33:0/2711069917 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f90e752d70 con 0x55f90e86b2f0 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8cdd7fa640 10 client.17559 handle_command_reply: tid=0 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8d01e5c740 1 client.17559 shutdown 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2711069917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 msgr2=0x55f90e7f4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7f4650 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0a71a0 tx=0x7f8cec0b9160 comp rx=0 tx=0).stop 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2711069917 shutdown_connections 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f90e86b2f0 0x55f90e86d710 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f90e7f0370 0x55f90e7f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f90e7e6b40 0x55f90e7f4650 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8d01e5c740 1 --2- 172.21.15.33:0/2711069917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f90e7e55d0 0x55f90e7f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2711069917 >> 172.21.15.33:0/2711069917 conn(0x55f90e7ae310 msgr2=0x55f90e7ee7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8cdd7fa640 0 client.17559 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2711069917 shutdown_connections 2024-09-16T00:28:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.480+0000 7f8d01e5c740 1 -- 172.21.15.33:0/2711069917 wait complete. 2024-09-16T00:28:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8d01e5c740 20 client.17559 trim_cache size 0 max 16384 2024-09-16T00:28:43.490 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:28:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8cffa83640 1 -- 172.21.15.33:0/99542288 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ce00a3240 msgr2=0x7f8ce00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8cffa83640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ce00a3240 0x7f8ce00a5700 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0018000 tx=0x7f8cf0016da0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8cffa83640 1 -- 172.21.15.33:0/99542288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8102780 msgr2=0x7f8cf8198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8cffa83640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8102780 0x7f8cf8198970 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f8cec005ab0 tx=0x7f8cec0042f0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8cffa83640 1 -- 172.21.15.33:0/99542288 shutdown_connections 2024-09-16T00:28:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8cffa83640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ce00a3240 0x7f8ce00a5700 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8cffa83640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf8106f00 0x7f8cf819cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8cffa83640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8102780 0x7f8cf8198970 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8cffa83640 1 --2- 172.21.15.33:0/99542288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cf8101dd0 0x7f8cf8198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8cffa83640 1 -- 172.21.15.33:0/99542288 >> 172.21.15.33:0/99542288 conn(0x7f8cf80fd8b0 msgr2=0x7f8cf80fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8cffa83640 1 -- 172.21.15.33:0/99542288 shutdown_connections 2024-09-16T00:28:43.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:43.481+0000 7f8cffa83640 1 -- 172.21.15.33:0/99542288 wait complete. 2024-09-16T00:28:43.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:43 smithi083 ceph-mon[35242]: pgmap v697: 129 pgs: 129 active+clean; 275 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 3.9 MiB/s wr, 109 op/s 2024-09-16T00:28:43.682 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 10.00 MB/sec execute 471 sec latency 16069.504 ms 2024-09-16T00:28:43.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:43 smithi060 ceph-mon[35185]: pgmap v697: 129 pgs: 129 active+clean; 275 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 3.9 MiB/s wr, 109 op/s 2024-09-16T00:28:43.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:43 smithi033 ceph-mon[31243]: pgmap v697: 129 pgs: 129 active+clean; 275 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 3.9 MiB/s wr, 109 op/s 2024-09-16T00:28:44.682 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 9.98 MB/sec execute 472 sec latency 17069.609 ms 2024-09-16T00:28:45.396 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:28:45.396 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:28:45.666 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:45.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:45 smithi083 ceph-mon[35242]: pgmap v698: 129 pgs: 129 active+clean; 274 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 3.7 MiB/s wr, 93 op/s 2024-09-16T00:28:45.682 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 9.96 MB/sec execute 473 sec latency 18069.653 ms 2024-09-16T00:28:45.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:45 smithi060 ceph-mon[35185]: pgmap v698: 129 pgs: 129 active+clean; 274 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 3.7 MiB/s wr, 93 op/s 2024-09-16T00:28:45.787 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:45 smithi033 ceph-mon[31243]: pgmap v698: 129 pgs: 129 active+clean; 274 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 3.7 MiB/s wr, 93 op/s 2024-09-16T00:28:46.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.190+0000 7fee65a60640 1 -- 172.21.15.33:0/835637889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee601027a0 msgr2=0x7fee601067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:46.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.190+0000 7fee65a60640 1 --2- 172.21.15.33:0/835637889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee601027a0 0x7fee601067f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fee54009900 tx=0x7fee5402a410 comp rx=0 tx=0).stop 2024-09-16T00:28:46.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.190+0000 7fee65a60640 1 -- 172.21.15.33:0/835637889 shutdown_connections 2024-09-16T00:28:46.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.190+0000 7fee65a60640 1 --2- 172.21.15.33:0/835637889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee60106f20 0x7fee6010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.190+0000 7fee65a60640 1 --2- 172.21.15.33:0/835637889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee601027a0 0x7fee601067f0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.190+0000 7fee65a60640 1 --2- 172.21.15.33:0/835637889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee60101df0 0x7fee601021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.190+0000 7fee65a60640 1 -- 172.21.15.33:0/835637889 >> 172.21.15.33:0/835637889 conn(0x7fee600fd8b0 msgr2=0x7fee600ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:46.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.190+0000 7fee65a60640 1 -- 172.21.15.33:0/835637889 shutdown_connections 2024-09-16T00:28:46.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.190+0000 7fee65a60640 1 -- 172.21.15.33:0/835637889 wait complete. 2024-09-16T00:28:46.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.191+0000 7fee65a60640 1 Processor -- start 2024-09-16T00:28:46.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.191+0000 7fee65a60640 1 -- start start 2024-09-16T00:28:46.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee65a60640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee60101df0 0x7fee60198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee65a60640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee601027a0 0x7fee60198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee65a60640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee60106f20 0x7fee6019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee65a60640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fee60111fa0 con 0x7fee60101df0 2024-09-16T00:28:46.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee65a60640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fee60111e20 con 0x7fee601027a0 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee65a60640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee60112120 con 0x7fee60106f20 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee5effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee60101df0 0x7fee60198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee5effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee60101df0 0x7fee60198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36224/0 (socket says 172.21.15.33:36224) 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee5effd640 1 -- 172.21.15.33:0/143849530 learned_addr learned my addr 172.21.15.33:0/143849530 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee5e7fc640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee601027a0 0x7fee60198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee5f7fe640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee60106f20 0x7fee6019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee5effd640 1 -- 172.21.15.33:0/143849530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee60106f20 msgr2=0x7fee6019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee5effd640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee60106f20 0x7fee6019cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee5effd640 1 -- 172.21.15.33:0/143849530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee601027a0 msgr2=0x7fee60198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee5effd640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee601027a0 0x7fee60198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee5effd640 1 -- 172.21.15.33:0/143849530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6019d4d0 con 0x7fee60101df0 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee5effd640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee60101df0 0x7fee60198520 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fee5000b4a0 tx=0x7fee5000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:46.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.192+0000 7fee64a5e640 1 -- 172.21.15.33:0/143849530 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50014070 con 0x7fee60101df0 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.193+0000 7fee64a5e640 1 -- 172.21.15.33:0/143849530 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee50010af0 con 0x7fee60101df0 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.193+0000 7fee65a60640 1 -- 172.21.15.33:0/143849530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee6019d7c0 con 0x7fee60101df0 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.193+0000 7fee64a5e640 1 -- 172.21.15.33:0/143849530 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50017d10 con 0x7fee60101df0 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.193+0000 7fee65a60640 1 -- 172.21.15.33:0/143849530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee601a50a0 con 0x7fee60101df0 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.195+0000 7fee64a5e640 1 -- 172.21.15.33:0/143849530 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fee50017070 con 0x7fee60101df0 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.195+0000 7fee67e39740 1 Processor -- start 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.195+0000 7fee67e39740 1 -- start start 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.195+0000 7fee67e39740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d78bb60 0x55d36d78bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.195+0000 7fee67e39740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78c480 0x55d36d796800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.195+0000 7fee67e39740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d796d40 0x55d36d799130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.195+0000 7fee67e39740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d36d6f1280 con 0x55d36d796d40 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.195+0000 7fee67e39740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d36d698b10 con 0x55d36d78bb60 2024-09-16T00:28:46.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.195+0000 7fee67e39740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d36d6ed240 con 0x55d36d78c480 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee64a5e640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee480a3240 0x7fee480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d796d40 0x55d36d799130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d796d40 0x55d36d799130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36228/0 (socket says 172.21.15.33:36228) 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5e7fc640 1 -- 172.21.15.33:0/3506333425 learned_addr learned my addr 172.21.15.33:0/3506333425 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5f7fe640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee480a3240 0x7fee480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee64a5e640 1 -- 172.21.15.33:0/143849530 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fee500cf270 con 0x7fee60101df0 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5effd640 1 --2- 172.21.15.33:0/3506333425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78c480 0x55d36d796800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5f7fe640 1 --2- 172.21.15.33:0/3506333425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d78bb60 0x55d36d78bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5e7fc640 1 -- 172.21.15.33:0/3506333425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78c480 msgr2=0x55d36d796800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5e7fc640 1 --2- 172.21.15.33:0/3506333425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78c480 0x55d36d796800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5e7fc640 1 -- 172.21.15.33:0/3506333425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d78bb60 msgr2=0x55d36d78bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5e7fc640 1 --2- 172.21.15.33:0/3506333425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d78bb60 0x55d36d78bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5e7fc640 1 -- 172.21.15.33:0/3506333425 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d36d6a51f0 con 0x55d36d796d40 2024-09-16T00:28:46.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5e7fc640 1 --2- 172.21.15.33:0/3506333425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d796d40 0x55d36d799130 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fee54005ab0 tx=0x7fee54031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c41145d3843aad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee5f7fe640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee480a3240 0x7fee480a5700 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fee58005c40 tx=0x7fee5800a920 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.196+0000 7fee45ffb640 1 -- 172.21.15.33:0/3506333425 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5403f070 con 0x55d36d796d40 2024-09-16T00:28:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.197+0000 7fee45ffb640 1 -- 172.21.15.33:0/3506333425 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee54031a20 con 0x55d36d796d40 2024-09-16T00:28:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.197+0000 7fee45ffb640 1 -- 172.21.15.33:0/3506333425 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54035070 con 0x55d36d796d40 2024-09-16T00:28:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.197+0000 7fee67e39740 1 -- 172.21.15.33:0/3506333425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d796d40 msgr2=0x55d36d799130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.197+0000 7fee67e39740 1 --2- 172.21.15.33:0/3506333425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d796d40 0x55d36d799130 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fee54005ab0 tx=0x7fee54031630 comp rx=0 tx=0).stop 2024-09-16T00:28:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.197+0000 7fee67e39740 1 -- 172.21.15.33:0/3506333425 shutdown_connections 2024-09-16T00:28:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.197+0000 7fee67e39740 1 --2- 172.21.15.33:0/3506333425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d796d40 0x55d36d799130 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.197+0000 7fee67e39740 1 --2- 172.21.15.33:0/3506333425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78c480 0x55d36d796800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.197+0000 7fee67e39740 1 --2- 172.21.15.33:0/3506333425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d78bb60 0x55d36d78bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.197+0000 7fee67e39740 1 -- 172.21.15.33:0/3506333425 >> 172.21.15.33:0/3506333425 conn(0x55d36d7880f0 msgr2=0x55d36d7884d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:46.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.197+0000 7fee67e39740 1 -- 172.21.15.33:0/3506333425 shutdown_connections 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.197+0000 7fee67e39740 1 -- 172.21.15.33:0/3506333425 wait complete. 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.198+0000 7fee67e39740 1 -- start start 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.198+0000 7fee67e39740 1 Processor -- start 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.198+0000 7fee67e39740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78bb60 0x55d36d793b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.198+0000 7fee67e39740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78c480 0x55d36d7940b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.198+0000 7fee67e39740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d796d40 0x55d36d7945f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.198+0000 7fee67e39740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d36d799910 con 0x55d36d78c480 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.198+0000 7fee67e39740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d36d6abf10 con 0x55d36d796d40 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.198+0000 7fee67e39740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d36d799a90 con 0x55d36d78bb60 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.198+0000 7fee67e39740 10 client.0 _ll_register_callbacks cb 0x55d36d674a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.198+0000 7fee5e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78c480 0x55d36d7940b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.198+0000 7fee5e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78c480 0x55d36d7940b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36230/0 (socket says 172.21.15.33:36230) 2024-09-16T00:28:46.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.198+0000 7fee5e7fc640 1 -- 172.21.15.33:0/66791299 learned_addr learned my addr 172.21.15.33:0/66791299 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee5effd640 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78bb60 0x55d36d793b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee5f7fe640 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d796d40 0x55d36d7945f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee5e7fc640 1 -- 172.21.15.33:0/66791299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78bb60 msgr2=0x55d36d793b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee5e7fc640 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78bb60 0x55d36d793b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee5e7fc640 1 -- 172.21.15.33:0/66791299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d796d40 msgr2=0x55d36d7945f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee5e7fc640 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d796d40 0x55d36d7945f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee5e7fc640 1 -- 172.21.15.33:0/66791299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d36d4626a0 con 0x55d36d78c480 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee5f7fe640 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d796d40 0x55d36d7945f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee5e7fc640 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78c480 0x55d36d7940b0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fee54031550 tx=0x7fee54004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421388ce18a83fec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee3effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee3effd640 1 -- 172.21.15.33:0/66791299 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54035070 con 0x55d36d78c480 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee3effd640 1 -- 172.21.15.33:0/66791299 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fee54004030 con 0x55d36d78c480 2024-09-16T00:28:46.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee67e39740 1 -- 172.21.15.33:0/66791299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee54031a20 con 0x55d36d78c480 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee3effd640 1 -- 172.21.15.33:0/66791299 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fee54055050 con 0x55d36d78c480 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee67e39740 10 client.17571 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee67e39740 1 -- 172.21.15.33:0/66791299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d36d7a5460 con 0x55d36d78c480 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.199+0000 7fee3effd640 1 -- 172.21.15.33:0/66791299 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee54031a20 con 0x55d36d78c480 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.200+0000 7fee3effd640 1 -- 172.21.15.33:0/66791299 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fee5403b430 con 0x55d36d78c480 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.200+0000 7fee67e39740 10 client.17571 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.200+0000 7fee67e39740 10 client.17571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.200+0000 7fee67e39740 20 client.17571 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.200+0000 7fee67e39740 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d36d7a67b0 0x55d36d7a8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.200+0000 7fee67e39740 4 client.17571 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.200+0000 7fee67e39740 1 -- 172.21.15.33:0/66791299 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d36d6fe7f0 con 0x55d36d7a67b0 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.200+0000 7fee5effd640 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d36d7a67b0 0x55d36d7a8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.200+0000 7fee5effd640 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d36d7a67b0 0x55d36d7a8ba0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2180d81f1dd6fba0 server_cookie=9e7827f04b2e0b1d in_seq=0 out_seq=0 2024-09-16T00:28:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.200+0000 7fee3effd640 10 client.17571 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.201+0000 7fee3effd640 1 -- 172.21.15.33:0/66791299 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d36d6fe7f0 con 0x55d36d7a67b0 2024-09-16T00:28:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.201+0000 7fee3effd640 10 client.17571 handle_command_reply: tid=0 2024-09-16T00:28:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.201+0000 7fee67e39740 1 client.17571 shutdown 2024-09-16T00:28:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.201+0000 7fee67e39740 1 -- 172.21.15.33:0/66791299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78c480 msgr2=0x55d36d7940b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.201+0000 7fee67e39740 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78c480 0x55d36d7940b0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fee54031550 tx=0x7fee54004a50 comp rx=0 tx=0).stop 2024-09-16T00:28:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.201+0000 7fee67e39740 1 -- 172.21.15.33:0/66791299 shutdown_connections 2024-09-16T00:28:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.201+0000 7fee67e39740 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d36d7a67b0 0x55d36d7a8ba0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.201+0000 7fee67e39740 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d796d40 0x55d36d7945f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.201+0000 7fee67e39740 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78c480 0x55d36d7940b0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.201+0000 7fee67e39740 1 --2- 172.21.15.33:0/66791299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78bb60 0x55d36d793b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.201+0000 7fee3effd640 0 client.17571 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.201+0000 7fee67e39740 1 -- 172.21.15.33:0/66791299 >> 172.21.15.33:0/66791299 conn(0x55d36d7880f0 msgr2=0x55d36d78ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.202+0000 7fee67e39740 1 -- 172.21.15.33:0/66791299 shutdown_connections 2024-09-16T00:28:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.202+0000 7fee67e39740 1 -- 172.21.15.33:0/66791299 wait complete. 2024-09-16T00:28:46.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.202+0000 7fee67e39740 20 client.17571 trim_cache size 0 max 16384 2024-09-16T00:28:46.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.219+0000 7fee67e39740 1 Processor -- start 2024-09-16T00:28:46.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.219+0000 7fee67e39740 1 -- start start 2024-09-16T00:28:46.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.219+0000 7fee67e39740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 0x55d36d78dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.219+0000 7fee67e39740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78c480 0x55d36d79f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.219+0000 7fee67e39740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d796d40 0x55d36d79f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.219+0000 7fee67e39740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fee601b19c0 con 0x55d36d78bb60 2024-09-16T00:28:46.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.219+0000 7fee67e39740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fee601af290 con 0x55d36d796d40 2024-09-16T00:28:46.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.219+0000 7fee67e39740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee601ad0e0 con 0x55d36d78c480 2024-09-16T00:28:46.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.219+0000 7fee5effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 0x55d36d78dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.219+0000 7fee5effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 0x55d36d78dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36240/0 (socket says 172.21.15.33:36240) 2024-09-16T00:28:46.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.219+0000 7fee5effd640 1 -- 172.21.15.33:0/430080637 learned_addr learned my addr 172.21.15.33:0/430080637 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:46.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.220+0000 7fee5e7fc640 1 --2- 172.21.15.33:0/430080637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78c480 0x55d36d79f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.220+0000 7fee5effd640 1 -- 172.21.15.33:0/430080637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78c480 msgr2=0x55d36d79f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:46.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.220+0000 7fee5f7fe640 1 --2- 172.21.15.33:0/430080637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d796d40 0x55d36d79f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.220+0000 7fee5effd640 1 --2- 172.21.15.33:0/430080637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78c480 0x55d36d79f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.220+0000 7fee5effd640 1 -- 172.21.15.33:0/430080637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d796d40 msgr2=0x55d36d79f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.220+0000 7fee5effd640 1 --2- 172.21.15.33:0/430080637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d796d40 0x55d36d79f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.220+0000 7fee5effd640 1 -- 172.21.15.33:0/430080637 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d36d7a8f10 con 0x55d36d78bb60 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.220+0000 7fee5effd640 1 --2- 172.21.15.33:0/430080637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 0x55d36d78dcc0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fee500cf510 tx=0x7fee50012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cca13917a6ee4ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.220+0000 7fee457fa640 1 -- 172.21.15.33:0/430080637 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5001e040 con 0x55d36d78bb60 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.220+0000 7fee457fa640 1 -- 172.21.15.33:0/430080637 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee500a4020 con 0x55d36d78bb60 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.220+0000 7fee457fa640 1 -- 172.21.15.33:0/430080637 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50097a20 con 0x55d36d78bb60 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.221+0000 7fee67e39740 1 -- 172.21.15.33:0/430080637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 msgr2=0x55d36d78dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.221+0000 7fee67e39740 1 --2- 172.21.15.33:0/430080637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 0x55d36d78dcc0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fee500cf510 tx=0x7fee50012040 comp rx=0 tx=0).stop 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.221+0000 7fee67e39740 1 -- 172.21.15.33:0/430080637 shutdown_connections 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.221+0000 7fee67e39740 1 --2- 172.21.15.33:0/430080637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d796d40 0x55d36d79f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.221+0000 7fee67e39740 1 --2- 172.21.15.33:0/430080637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d78c480 0x55d36d79f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.221+0000 7fee67e39740 1 --2- 172.21.15.33:0/430080637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 0x55d36d78dcc0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.221+0000 7fee67e39740 1 -- 172.21.15.33:0/430080637 >> 172.21.15.33:0/430080637 conn(0x55d36d7880f0 msgr2=0x55d36d78d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.221+0000 7fee67e39740 1 -- 172.21.15.33:0/430080637 shutdown_connections 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.221+0000 7fee67e39740 1 -- 172.21.15.33:0/430080637 wait complete. 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.221+0000 7fee67e39740 1 -- start start 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.221+0000 7fee67e39740 1 Processor -- start 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee67e39740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 0x55d36d7914a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee67e39740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d78c480 0x55d36d78fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee67e39740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d796d40 0x55d36d790340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee67e39740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fee601ad750 con 0x55d36d78bb60 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee67e39740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fee601aebc0 con 0x55d36d78c480 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee67e39740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee601ade20 con 0x55d36d796d40 2024-09-16T00:28:46.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee67e39740 10 client.0 _ll_register_callbacks cb 0x7fee50017f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:46.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee5effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 0x55d36d7914a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee5effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 0x55d36d7914a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36256/0 (socket says 172.21.15.33:36256) 2024-09-16T00:28:46.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee5effd640 1 -- 172.21.15.33:0/376066875 learned_addr learned my addr 172.21.15.33:0/376066875 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:46.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee5f7fe640 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d796d40 0x55d36d790340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee5e7fc640 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d78c480 0x55d36d78fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee5effd640 1 -- 172.21.15.33:0/376066875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d796d40 msgr2=0x55d36d790340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:46.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee5effd640 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d796d40 0x55d36d790340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee5effd640 1 -- 172.21.15.33:0/376066875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d78c480 msgr2=0x55d36d78fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:46.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee5effd640 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d78c480 0x55d36d78fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee5effd640 1 -- 172.21.15.33:0/376066875 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d36d790900 con 0x55d36d78bb60 2024-09-16T00:28:46.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee5effd640 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 0x55d36d7914a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fee50009420 tx=0x7fee5009f8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935636a7b4e6273d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:46.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee3effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee3effd640 1 -- 172.21.15.33:0/376066875 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5001e070 con 0x55d36d78bb60 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee3effd640 1 -- 172.21.15.33:0/376066875 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fee50099450 con 0x55d36d78bb60 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.222+0000 7fee67e39740 1 -- 172.21.15.33:0/376066875 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee500a4020 con 0x55d36d78bb60 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.223+0000 7fee3effd640 1 -- 172.21.15.33:0/376066875 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fee50010ca0 con 0x55d36d78bb60 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.223+0000 7fee67e39740 10 client.17577 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.223+0000 7fee67e39740 1 -- 172.21.15.33:0/376066875 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d36d815f60 con 0x55d36d78bb60 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.223+0000 7fee3effd640 1 -- 172.21.15.33:0/376066875 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee500a4020 con 0x55d36d78bb60 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.223+0000 7fee3effd640 1 -- 172.21.15.33:0/376066875 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fee50097a20 con 0x55d36d78bb60 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.223+0000 7fee67e39740 10 client.17577 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.223+0000 7fee67e39740 10 client.17577 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.223+0000 7fee67e39740 20 client.17577 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.223+0000 7fee67e39740 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d36d816620 0x55d36d818a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:46.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.223+0000 7fee67e39740 4 client.17577 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:28:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.223+0000 7fee67e39740 1 -- 172.21.15.33:0/376066875 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d36d6f8d00 con 0x55d36d816620 2024-09-16T00:28:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.223+0000 7fee5e7fc640 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d36d816620 0x55d36d818a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.224+0000 7fee5e7fc640 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d36d816620 0x55d36d818a40 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=451bd24636a5e7b8 server_cookie=95db518e8c23e3f3 in_seq=0 out_seq=0 2024-09-16T00:28:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.224+0000 7fee3effd640 10 client.17577 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee3effd640 1 -- 172.21.15.33:0/376066875 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d36d6f8d00 con 0x55d36d816620 2024-09-16T00:28:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee3effd640 10 client.17577 handle_command_reply: tid=0 2024-09-16T00:28:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee67e39740 1 client.17577 shutdown 2024-09-16T00:28:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee67e39740 1 -- 172.21.15.33:0/376066875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 msgr2=0x55d36d7914a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee67e39740 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 0x55d36d7914a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fee50009420 tx=0x7fee5009f8e0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee67e39740 1 -- 172.21.15.33:0/376066875 shutdown_connections 2024-09-16T00:28:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee67e39740 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d36d816620 0x55d36d818a40 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee67e39740 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36d796d40 0x55d36d790340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee67e39740 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d36d78c480 0x55d36d78fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee67e39740 1 --2- 172.21.15.33:0/376066875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d36d78bb60 0x55d36d7914a0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee67e39740 1 -- 172.21.15.33:0/376066875 >> 172.21.15.33:0/376066875 conn(0x55d36d7880f0 msgr2=0x55d36d794330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee3effd640 0 client.17577 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee67e39740 1 -- 172.21.15.33:0/376066875 shutdown_connections 2024-09-16T00:28:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee67e39740 1 -- 172.21.15.33:0/376066875 wait complete. 2024-09-16T00:28:46.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.225+0000 7fee67e39740 20 client.17577 trim_cache size 0 max 16384 2024-09-16T00:28:46.232 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:28:46.232 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:28:46.232 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "4b5f50d9-c90d-4ea4-b6d2-3484b2fe8a36", 2024-09-16T00:28:46.233 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:28:46.233 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:28:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.226+0000 7fee65a60640 1 -- 172.21.15.33:0/143849530 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee480a3240 msgr2=0x7fee480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.226+0000 7fee65a60640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee480a3240 0x7fee480a5700 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fee58005c40 tx=0x7fee5800a920 comp rx=0 tx=0).stop 2024-09-16T00:28:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.226+0000 7fee65a60640 1 -- 172.21.15.33:0/143849530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee60101df0 msgr2=0x7fee60198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.226+0000 7fee65a60640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee60101df0 0x7fee60198520 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fee5000b4a0 tx=0x7fee5000b960 comp rx=0 tx=0).stop 2024-09-16T00:28:46.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.226+0000 7fee65a60640 1 -- 172.21.15.33:0/143849530 shutdown_connections 2024-09-16T00:28:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.226+0000 7fee65a60640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee480a3240 0x7fee480a5700 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.226+0000 7fee65a60640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee60106f20 0x7fee6019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.226+0000 7fee65a60640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee601027a0 0x7fee60198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.226+0000 7fee65a60640 1 --2- 172.21.15.33:0/143849530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee60101df0 0x7fee60198520 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.226+0000 7fee65a60640 1 -- 172.21.15.33:0/143849530 >> 172.21.15.33:0/143849530 conn(0x7fee600fd8b0 msgr2=0x7fee600febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.226+0000 7fee65a60640 1 -- 172.21.15.33:0/143849530 shutdown_connections 2024-09-16T00:28:46.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:46.226+0000 7fee65a60640 1 -- 172.21.15.33:0/143849530 wait complete. 2024-09-16T00:28:46.595 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:28:46.682 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 9.94 MB/sec execute 474 sec latency 19069.683 ms 2024-09-16T00:28:47.029 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:28:47.106 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:47.521 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:47 smithi033 ceph-mon[31243]: pgmap v699: 129 pgs: 129 active+clean; 273 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 3.5 MiB/s wr, 93 op/s 2024-09-16T00:28:47.521 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:47 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:28:47.521 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:47 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:47.521 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:47 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:28:47.522 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:47 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:28:47.522 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:47 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:47.522 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:47 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:47.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.651+0000 7f25b959a640 1 -- 172.21.15.33:0/1967507041 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4035020 con 0x7f25bc114ab0 2024-09-16T00:28:47.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.651+0000 7f25c0afc640 1 -- 172.21.15.33:0/1967507041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25bc114ab0 msgr2=0x7f25bc116ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:47.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.651+0000 7f25c0afc640 1 --2- 172.21.15.33:0/1967507041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25bc114ab0 0x7f25bc116ea0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f25b4009900 tx=0x7f25b402a480 comp rx=0 tx=0).stop 2024-09-16T00:28:47.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.652+0000 7f25c0afc640 1 -- 172.21.15.33:0/1967507041 shutdown_connections 2024-09-16T00:28:47.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.652+0000 7f25c0afc640 1 --2- 172.21.15.33:0/1967507041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25bc114ab0 0x7f25bc116ea0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:47.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.652+0000 7f25c0afc640 1 --2- 172.21.15.33:0/1967507041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25bc109b10 0x7f25bc114570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:47.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.652+0000 7f25c0afc640 1 --2- 172.21.15.33:0/1967507041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc1091f0 0x7f25bc1095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:47.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.652+0000 7f25c0afc640 1 -- 172.21.15.33:0/1967507041 >> 172.21.15.33:0/1967507041 conn(0x7f25bc06e140 msgr2=0x7f25bc06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:47.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.652+0000 7f25c0afc640 1 -- 172.21.15.33:0/1967507041 shutdown_connections 2024-09-16T00:28:47.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.652+0000 7f25c0afc640 1 -- 172.21.15.33:0/1967507041 wait complete. 2024-09-16T00:28:47.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.653+0000 7f25c0afc640 1 Processor -- start 2024-09-16T00:28:47.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.653+0000 7f25c0afc640 1 -- start start 2024-09-16T00:28:47.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.654+0000 7f25c0afc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc1091f0 0x7f25bc1a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:47.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.654+0000 7f25c0afc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25bc109b10 0x7f25bc1a4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:47.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.654+0000 7f25c0afc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25bc114ab0 0x7f25bc19e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:47.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.654+0000 7f25c0afc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc118610 con 0x7f25bc109b10 2024-09-16T00:28:47.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.654+0000 7f25c0afc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc118490 con 0x7f25bc1091f0 2024-09-16T00:28:47.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.654+0000 7f25bad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25bc114ab0 0x7f25bc19e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:47.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.654+0000 7f25b9d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25bc109b10 0x7f25bc1a4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:47.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.654+0000 7f25b9d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25bc109b10 0x7f25bc1a4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36284/0 (socket says 172.21.15.33:36284) 2024-09-16T00:28:47.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.654+0000 7f25b9d9b640 1 -- 172.21.15.33:0/2939873940 learned_addr learned my addr 172.21.15.33:0/2939873940 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:47.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.654+0000 7f25bad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25bc114ab0 0x7f25bc19e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:38724/0 (socket says 172.21.15.33:38724) 2024-09-16T00:28:47.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.654+0000 7f25c0afc640 1 -- 172.21.15.33:0/2939873940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc118790 con 0x7f25bc114ab0 2024-09-16T00:28:47.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.655+0000 7f25ba59c640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc1091f0 0x7f25bc1a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:47.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.655+0000 7f25b9d9b640 1 -- 172.21.15.33:0/2939873940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25bc114ab0 msgr2=0x7f25bc19e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:47.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.655+0000 7f25b9d9b640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25bc114ab0 0x7f25bc19e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:47.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.655+0000 7f25b9d9b640 1 -- 172.21.15.33:0/2939873940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc1091f0 msgr2=0x7f25bc1a4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:47.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.655+0000 7f25b9d9b640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc1091f0 0x7f25bc1a4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:47.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.655+0000 7f25b9d9b640 1 -- 172.21.15.33:0/2939873940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25bc19ef80 con 0x7f25bc109b10 2024-09-16T00:28:47.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.655+0000 7f25b9d9b640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25bc109b10 0x7f25bc1a4ba0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f25b0011b20 tx=0x7f25b000f050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:47.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.656+0000 7f25ab7fe640 1 -- 172.21.15.33:0/2939873940 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b001e070 con 0x7f25bc109b10 2024-09-16T00:28:47.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.656+0000 7f25ab7fe640 1 -- 172.21.15.33:0/2939873940 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25b000fc90 con 0x7f25bc109b10 2024-09-16T00:28:47.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.656+0000 7f25ab7fe640 1 -- 172.21.15.33:0/2939873940 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b0018780 con 0x7f25bc109b10 2024-09-16T00:28:47.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.656+0000 7f25c0afc640 1 -- 172.21.15.33:0/2939873940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25bc19f270 con 0x7f25bc109b10 2024-09-16T00:28:47.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.656+0000 7f25c0afc640 1 -- 172.21.15.33:0/2939873940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25bc1ab4c0 con 0x7f25bc109b10 2024-09-16T00:28:47.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.658+0000 7f25a97fa640 1 -- 172.21.15.33:0/2939873940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25bc073a40 con 0x7f25bc109b10 2024-09-16T00:28:47.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.661+0000 7f25ab7fe640 1 -- 172.21.15.33:0/2939873940 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f25b0019400 con 0x7f25bc109b10 2024-09-16T00:28:47.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.663+0000 7f25ab7fe640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f25a40a3340 0x7f25a40a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:47.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.663+0000 7f25ab7fe640 1 -- 172.21.15.33:0/2939873940 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f25b00d3b50 con 0x7f25bc109b10 2024-09-16T00:28:47.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.663+0000 7f25ba59c640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f25a40a3340 0x7f25a40a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:47.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.664+0000 7f25ba59c640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f25a40a3340 0x7f25a40a5800 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f25ac007ec0 tx=0x7f25ac0075b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:47.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:47 smithi083 ceph-mon[35242]: pgmap v699: 129 pgs: 129 active+clean; 273 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 3.5 MiB/s wr, 93 op/s 2024-09-16T00:28:47.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:47 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:28:47.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:47 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:47.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:47 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:28:47.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:47 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:28:47.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:47 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:47.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:47 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:47.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.669+0000 7f25ab7fe640 1 -- 172.21.15.33:0/2939873940 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f25b0014070 con 0x7f25bc109b10 2024-09-16T00:28:47.682 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 9.92 MB/sec execute 475 sec latency 20070.252 ms 2024-09-16T00:28:47.693 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:47 smithi060 ceph-mon[35185]: pgmap v699: 129 pgs: 129 active+clean; 273 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 3.5 MiB/s wr, 93 op/s 2024-09-16T00:28:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:47 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:28:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:47 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:47 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:28:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:47 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:28:47.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:47 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:47.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:47 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:28:47.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.909+0000 7f25a97fa640 1 -- 172.21.15.33:0/2939873940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25bc1095d0 con 0x7f25bc109b10 2024-09-16T00:28:47.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.910+0000 7f25ab7fe640 1 -- 172.21.15.33:0/2939873940 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f25b00b7e90 con 0x7f25bc109b10 2024-09-16T00:28:47.910 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:28:47.910 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:28:47.911 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:28:47.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.912+0000 7f25a97fa640 1 -- 172.21.15.33:0/2939873940 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f25a40a3340 msgr2=0x7f25a40a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:47.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.912+0000 7f25a97fa640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f25a40a3340 0x7f25a40a5800 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f25ac007ec0 tx=0x7f25ac0075b0 comp rx=0 tx=0).stop 2024-09-16T00:28:47.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.912+0000 7f25a97fa640 1 -- 172.21.15.33:0/2939873940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25bc109b10 msgr2=0x7f25bc1a4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:47.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.912+0000 7f25a97fa640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25bc109b10 0x7f25bc1a4ba0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f25b0011b20 tx=0x7f25b000f050 comp rx=0 tx=0).stop 2024-09-16T00:28:47.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.912+0000 7f25a97fa640 1 -- 172.21.15.33:0/2939873940 shutdown_connections 2024-09-16T00:28:47.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.912+0000 7f25a97fa640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f25a40a3340 0x7f25a40a5800 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:47.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.912+0000 7f25a97fa640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25bc114ab0 0x7f25bc19e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:47.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.912+0000 7f25a97fa640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f25bc109b10 0x7f25bc1a4ba0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:47.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.912+0000 7f25a97fa640 1 --2- 172.21.15.33:0/2939873940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc1091f0 0x7f25bc1a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:47.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.912+0000 7f25a97fa640 1 -- 172.21.15.33:0/2939873940 >> 172.21.15.33:0/2939873940 conn(0x7f25bc06e140 msgr2=0x7f25bc070840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:47.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.912+0000 7f25a97fa640 1 -- 172.21.15.33:0/2939873940 shutdown_connections 2024-09-16T00:28:47.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:47.912+0000 7f25a97fa640 1 -- 172.21.15.33:0/2939873940 wait complete. 2024-09-16T00:28:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:48 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2939873940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:28:48.658 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id b9e606eb --release --await' 2024-09-16T00:28:48.658 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id b9e606eb --release --await 2024-09-16T00:28:48.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:48 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2939873940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:28:48.683 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 9.89 MB/sec execute 476 sec latency 21070.291 ms 2024-09-16T00:28:48.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:48 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2939873940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:28:48.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.824+0000 7fa002337640 1 -- 172.21.15.33:0/486254831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ffc102570 msgr2=0x7f9ffc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:48.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.824+0000 7fa002337640 1 --2- 172.21.15.33:0/486254831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ffc102570 0x7f9ffc1029b0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0009900 tx=0x7f9ff002a480 comp rx=0 tx=0).stop 2024-09-16T00:28:48.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.824+0000 7fa002337640 1 -- 172.21.15.33:0/486254831 shutdown_connections 2024-09-16T00:28:48.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.824+0000 7fa002337640 1 --2- 172.21.15.33:0/486254831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ffc102ef0 0x7f9ffc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.824+0000 7fa002337640 1 --2- 172.21.15.33:0/486254831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ffc102570 0x7f9ffc1029b0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.824+0000 7fa002337640 1 --2- 172.21.15.33:0/486254831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ffc100e20 0x7f9ffc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.824+0000 7fa002337640 1 -- 172.21.15.33:0/486254831 >> 172.21.15.33:0/486254831 conn(0x7f9ffc0fc7d0 msgr2=0x7f9ffc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:48.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.824+0000 7fa002337640 1 -- 172.21.15.33:0/486254831 shutdown_connections 2024-09-16T00:28:48.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.824+0000 7fa002337640 1 -- 172.21.15.33:0/486254831 wait complete. 2024-09-16T00:28:48.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.825+0000 7fa002337640 1 Processor -- start 2024-09-16T00:28:48.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.825+0000 7fa002337640 1 -- start start 2024-09-16T00:28:48.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.825+0000 7fa002337640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ffc100e20 0x7f9ffc0fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.825+0000 7fa002337640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ffc102570 0x7f9ffc0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.825+0000 7fa002337640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ffc102ef0 0x7f9ffc0fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.825+0000 7fa002337640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ffc110f80 con 0x7f9ffc102ef0 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.825+0000 7fa002337640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9ffc110e00 con 0x7f9ffc102570 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.825+0000 7fa002337640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9ffc111100 con 0x7f9ffc100e20 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.825+0000 7fa0008d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ffc102ef0 0x7f9ffc0fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.825+0000 7fa0008d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ffc102ef0 0x7f9ffc0fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36306/0 (socket says 172.21.15.33:36306) 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7fa0008d4640 1 -- 172.21.15.33:0/3582371931 learned_addr learned my addr 172.21.15.33:0/3582371931 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ffc102570 0x7f9ffc0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7f9ffbfff640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ffc100e20 0x7f9ffc0fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7fa0008d4640 1 -- 172.21.15.33:0/3582371931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ffc100e20 msgr2=0x7f9ffc0fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7fa0008d4640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ffc100e20 0x7f9ffc0fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7fa0008d4640 1 -- 172.21.15.33:0/3582371931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ffc102570 msgr2=0x7f9ffc0fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7fa0008d4640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ffc102570 0x7f9ffc0fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7fa0008d4640 1 -- 172.21.15.33:0/3582371931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ffc1a69a0 con 0x7f9ffc102ef0 2024-09-16T00:28:48.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7fa0008d4640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ffc102ef0 0x7f9ffc0fd700 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400eaa0 tx=0x7f9ff400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:48.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7f9ff97fa640 1 -- 172.21.15.33:0/3582371931 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff400cdb0 con 0x7f9ffc102ef0 2024-09-16T00:28:48.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7f9ff97fa640 1 -- 172.21.15.33:0/3582371931 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ff4005020 con 0x7f9ffc102ef0 2024-09-16T00:28:48.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7fa002337640 1 -- 172.21.15.33:0/3582371931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ffc1a6c90 con 0x7f9ffc102ef0 2024-09-16T00:28:48.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7f9ff97fa640 1 -- 172.21.15.33:0/3582371931 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff40116b0 con 0x7f9ffc102ef0 2024-09-16T00:28:48.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.826+0000 7fa002337640 1 -- 172.21.15.33:0/3582371931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ffc1a71a0 con 0x7f9ffc102ef0 2024-09-16T00:28:48.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.829+0000 7f9ff97fa640 1 -- 172.21.15.33:0/3582371931 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9ff4005560 con 0x7f9ffc102ef0 2024-09-16T00:28:48.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.829+0000 7f9ff97fa640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9fe40a32c0 0x7f9fe40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.829+0000 7f9ff97fa640 1 -- 172.21.15.33:0/3582371931 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9ff40cfe70 con 0x7f9ffc102ef0 2024-09-16T00:28:48.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.829+0000 7f9ffbfff640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9fe40a32c0 0x7f9fe40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.830+0000 7f9ffbfff640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9fe40a32c0 0x7f9fe40a5780 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009bd0 tx=0x7f9fec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:48.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.830+0000 7fa004710740 1 Processor -- start 2024-09-16T00:28:48.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.830+0000 7fa004710740 1 -- start start 2024-09-16T00:28:48.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.830+0000 7fa004710740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f999124a0 0x556f99912880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa004710740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f99912dc0 0x556f9991d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa004710740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 0x556f9991fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa004710740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556f998782e0 con 0x556f9991d680 2024-09-16T00:28:48.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa004710740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556f9981fce0 con 0x556f999124a0 2024-09-16T00:28:48.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa004710740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556f998742a0 con 0x556f99912dc0 2024-09-16T00:28:48.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa0008d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 0x556f9991fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa0008d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 0x556f9991fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36318/0 (socket says 172.21.15.33:36318) 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa0008d4640 1 -- 172.21.15.33:0/1363537441 learned_addr learned my addr 172.21.15.33:0/1363537441 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7f9ffbfff640 1 --2- 172.21.15.33:0/1363537441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f999124a0 0x556f99912880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/1363537441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f99912dc0 0x556f9991d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa0008d4640 1 -- 172.21.15.33:0/1363537441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f99912dc0 msgr2=0x556f9991d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa0008d4640 1 --2- 172.21.15.33:0/1363537441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f99912dc0 0x556f9991d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa0008d4640 1 -- 172.21.15.33:0/1363537441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f999124a0 msgr2=0x556f99912880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa0008d4640 1 --2- 172.21.15.33:0/1363537441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f999124a0 0x556f99912880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa0008d4640 1 -- 172.21.15.33:0/1363537441 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f99804440 con 0x556f9991d680 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/1363537441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f99912dc0 0x556f9991d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7fa0008d4640 1 --2- 172.21.15.33:0/1363537441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 0x556f9991fa70 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4005910 tx=0x7f9ff40a03c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa63886aaf96aa40 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7f9fe27fc640 1 -- 172.21.15.33:0/1363537441 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff401f030 con 0x556f9991d680 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7f9fe27fc640 1 -- 172.21.15.33:0/1363537441 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ff409a3c0 con 0x556f9991d680 2024-09-16T00:28:48.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.831+0000 7f9fe27fc640 1 -- 172.21.15.33:0/1363537441 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff40a47f0 con 0x556f9991d680 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.832+0000 7fa004710740 1 -- 172.21.15.33:0/1363537441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 msgr2=0x556f9991fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.832+0000 7fa004710740 1 --2- 172.21.15.33:0/1363537441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 0x556f9991fa70 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4005910 tx=0x7f9ff40a03c0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.832+0000 7fa004710740 1 -- 172.21.15.33:0/1363537441 shutdown_connections 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.832+0000 7fa004710740 1 --2- 172.21.15.33:0/1363537441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 0x556f9991fa70 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.832+0000 7fa004710740 1 --2- 172.21.15.33:0/1363537441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f99912dc0 0x556f9991d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.833+0000 7fa004710740 1 --2- 172.21.15.33:0/1363537441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f999124a0 0x556f99912880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.833+0000 7fa004710740 1 -- 172.21.15.33:0/1363537441 >> 172.21.15.33:0/1363537441 conn(0x556f9990ea30 msgr2=0x556f9990ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.833+0000 7fa004710740 1 -- 172.21.15.33:0/1363537441 shutdown_connections 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.833+0000 7fa004710740 1 -- 172.21.15.33:0/1363537441 wait complete. 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.833+0000 7fa004710740 1 -- start start 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.833+0000 7fa004710740 1 Processor -- start 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7fa004710740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f999124a0 0x556f9991cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7fa004710740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f99912dc0 0x556f9991a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7fa004710740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f9991d680 0x556f9991a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7fa004710740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556f99920250 con 0x556f999124a0 2024-09-16T00:28:48.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7fa004710740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556f99833050 con 0x556f99912dc0 2024-09-16T00:28:48.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7fa004710740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556f999203d0 con 0x556f9991d680 2024-09-16T00:28:48.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7fa004710740 10 client.0 _ll_register_callbacks cb 0x556f99873160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:48.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9ffbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f999124a0 0x556f9991cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9ffbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f999124a0 0x556f9991cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36324/0 (socket says 172.21.15.33:36324) 2024-09-16T00:28:48.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9ffbfff640 1 -- 172.21.15.33:0/4190442771 learned_addr learned my addr 172.21.15.33:0/4190442771 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:48.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f99912dc0 0x556f9991a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9ffbfff640 1 -- 172.21.15.33:0/4190442771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f9991d680 msgr2=0x556f9991a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:48.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9ffbfff640 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f9991d680 0x556f9991a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9ffbfff640 1 -- 172.21.15.33:0/4190442771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f99912dc0 msgr2=0x556f9991a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:48.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9ffbfff640 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f99912dc0 0x556f9991a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9ffbfff640 1 -- 172.21.15.33:0/4190442771 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f995e93c0 con 0x556f999124a0 2024-09-16T00:28:48.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f99912dc0 0x556f9991a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:28:48.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9ffbfff640 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f999124a0 0x556f9991cfe0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f9fec003950 tx=0x7f9fec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde97d90e0c35580 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:48.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9fdb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:48.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9fdb7fe640 1 -- 172.21.15.33:0/4190442771 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fec00e6a0 con 0x556f999124a0 2024-09-16T00:28:48.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7f9fdb7fe640 1 -- 172.21.15.33:0/4190442771 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9fec00e840 con 0x556f999124a0 2024-09-16T00:28:48.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.834+0000 7fa004710740 1 -- 172.21.15.33:0/4190442771 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ff409a3c0 con 0x556f999124a0 2024-09-16T00:28:48.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.835+0000 7f9fdb7fe640 1 -- 172.21.15.33:0/4190442771 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9fec00ea40 con 0x556f999124a0 2024-09-16T00:28:48.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.835+0000 7fa004710740 10 client.17595 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:48.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.835+0000 7fa004710740 1 -- 172.21.15.33:0/4190442771 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556f9992bbf0 con 0x556f999124a0 2024-09-16T00:28:48.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.835+0000 7f9fdb7fe640 1 -- 172.21.15.33:0/4190442771 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ff409a3c0 con 0x556f999124a0 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.835+0000 7f9fdb7fe640 1 -- 172.21.15.33:0/4190442771 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f9fec01d410 con 0x556f999124a0 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.835+0000 7fa004710740 10 client.17595 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.835+0000 7fa004710740 10 client.17595 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.835+0000 7fa004710740 20 client.17595 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.835+0000 7fa004710740 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556f9992d100 0x556f9992f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.835+0000 7fa004710740 4 client.17595 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.835+0000 7fa004710740 1 -- 172.21.15.33:0/4190442771 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f99885850 con 0x556f9992d100 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.835+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556f9992d100 0x556f9992f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.836+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556f9992d100 0x556f9992f590 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bb56c59d8e3be17 server_cookie=e24fc3d4bc481333 in_seq=0 out_seq=0 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.836+0000 7f9fdb7fe640 10 client.17595 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.836+0000 7f9fdb7fe640 1 -- 172.21.15.33:0/4190442771 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556f99885850 con 0x556f9992d100 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.836+0000 7f9fdb7fe640 10 client.17595 handle_command_reply: tid=0 2024-09-16T00:28:48.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.836+0000 7fa004710740 1 client.17595 shutdown 2024-09-16T00:28:48.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.837+0000 7fa004710740 1 -- 172.21.15.33:0/4190442771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f999124a0 msgr2=0x556f9991cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:48.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.837+0000 7fa004710740 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f999124a0 0x556f9991cfe0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f9fec003950 tx=0x7f9fec004730 comp rx=0 tx=0).stop 2024-09-16T00:28:48.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.837+0000 7fa004710740 1 -- 172.21.15.33:0/4190442771 shutdown_connections 2024-09-16T00:28:48.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.837+0000 7fa004710740 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556f9992d100 0x556f9992f590 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.837+0000 7f9fdb7fe640 0 client.17595 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:48.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.837+0000 7fa004710740 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f9991d680 0x556f9991a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.837+0000 7fa004710740 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f99912dc0 0x556f9991a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.837+0000 7fa004710740 1 --2- 172.21.15.33:0/4190442771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f999124a0 0x556f9991cfe0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.837+0000 7fa004710740 1 -- 172.21.15.33:0/4190442771 >> 172.21.15.33:0/4190442771 conn(0x556f9990ea30 msgr2=0x556f99914710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:48.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.837+0000 7fa004710740 1 -- 172.21.15.33:0/4190442771 shutdown_connections 2024-09-16T00:28:48.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.837+0000 7fa004710740 1 -- 172.21.15.33:0/4190442771 wait complete. 2024-09-16T00:28:48.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.837+0000 7fa004710740 20 client.17595 trim_cache size 0 max 16384 2024-09-16T00:28:48.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.856+0000 7fa004710740 1 Processor -- start 2024-09-16T00:28:48.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.856+0000 7fa004710740 1 -- start start 2024-09-16T00:28:48.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.856+0000 7fa004710740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f999124a0 0x556f999294b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.856+0000 7fa004710740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f99912dc0 0x556f99934e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.856+0000 7fa004710740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f9991d680 0x556f999353a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.856+0000 7fa004710740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ffc1b3c10 con 0x556f99912dc0 2024-09-16T00:28:48.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.856+0000 7fa004710740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9ffc1b14e0 con 0x556f9991d680 2024-09-16T00:28:48.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.856+0000 7fa004710740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9ffc1af330 con 0x556f999124a0 2024-09-16T00:28:48.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.856+0000 7f9ffb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f99912dc0 0x556f99934e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.856+0000 7f9ffb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f99912dc0 0x556f99934e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36334/0 (socket says 172.21.15.33:36334) 2024-09-16T00:28:48.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.856+0000 7f9ffb7fe640 1 -- 172.21.15.33:0/3838032195 learned_addr learned my addr 172.21.15.33:0/3838032195 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:48.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.857+0000 7f9ffbfff640 1 --2- 172.21.15.33:0/3838032195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f999124a0 0x556f999294b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.857+0000 7f9ffb7fe640 1 -- 172.21.15.33:0/3838032195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f999124a0 msgr2=0x556f999294b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:48.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.857+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/3838032195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f999124a0 0x556f999294b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.857+0000 7f9ffb7fe640 1 -- 172.21.15.33:0/3838032195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f9991d680 msgr2=0x556f999353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:48.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.857+0000 7fa0008d4640 1 --2- 172.21.15.33:0/3838032195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f9991d680 0x556f999353a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.857+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/3838032195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f9991d680 0x556f999353a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.857+0000 7f9ffb7fe640 1 -- 172.21.15.33:0/3838032195 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f99919240 con 0x556f99912dc0 2024-09-16T00:28:48.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.857+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/3838032195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f99912dc0 0x556f99934e60 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f9ff002a9a0 tx=0x7f9ff00058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a31e34219b81bd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:48.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.857+0000 7f9fe1ffb640 1 -- 172.21.15.33:0/3838032195 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff0035040 con 0x556f99912dc0 2024-09-16T00:28:48.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.857+0000 7f9fe1ffb640 1 -- 172.21.15.33:0/3838032195 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ff00047a0 con 0x556f99912dc0 2024-09-16T00:28:48.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.857+0000 7f9fe1ffb640 1 -- 172.21.15.33:0/3838032195 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00318a0 con 0x556f99912dc0 2024-09-16T00:28:48.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.858+0000 7fa004710740 1 -- 172.21.15.33:0/3838032195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f99912dc0 msgr2=0x556f99934e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:48.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.858+0000 7fa004710740 1 --2- 172.21.15.33:0/3838032195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f99912dc0 0x556f99934e60 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f9ff002a9a0 tx=0x7f9ff00058d0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.858+0000 7fa004710740 1 -- 172.21.15.33:0/3838032195 shutdown_connections 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.858+0000 7fa004710740 1 --2- 172.21.15.33:0/3838032195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f9991d680 0x556f999353a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.858+0000 7fa004710740 1 --2- 172.21.15.33:0/3838032195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f99912dc0 0x556f99934e60 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.858+0000 7fa004710740 1 --2- 172.21.15.33:0/3838032195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f999124a0 0x556f999294b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.858+0000 7fa004710740 1 -- 172.21.15.33:0/3838032195 >> 172.21.15.33:0/3838032195 conn(0x556f9990ea30 msgr2=0x556f999158b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.858+0000 7fa004710740 1 -- 172.21.15.33:0/3838032195 shutdown_connections 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.858+0000 7fa004710740 1 -- 172.21.15.33:0/3838032195 wait complete. 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.858+0000 7fa004710740 1 -- start start 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7fa004710740 1 Processor -- start 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7fa004710740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f999124a0 0x556f99925ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7fa004710740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f99912dc0 0x556f99921200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7fa004710740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 0x556f99921990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7fa004710740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ffc1af9a0 con 0x556f9991d680 2024-09-16T00:28:48.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7fa004710740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9ffc1b0e10 con 0x556f99912dc0 2024-09-16T00:28:48.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7fa004710740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9ffc1b0070 con 0x556f999124a0 2024-09-16T00:28:48.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7fa004710740 10 client.0 _ll_register_callbacks cb 0x7f9fec004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:48.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7fa0008d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 0x556f99921990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7fa0008d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 0x556f99921990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36350/0 (socket says 172.21.15.33:36350) 2024-09-16T00:28:48.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7fa0008d4640 1 -- 172.21.15.33:0/447107045 learned_addr learned my addr 172.21.15.33:0/447107045 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:48.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7f9ffbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f999124a0 0x556f99925ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.859+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f99912dc0 0x556f99921200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7f9ffbfff640 1 -- 172.21.15.33:0/447107045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f99912dc0 msgr2=0x556f99921200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:48.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7f9ffbfff640 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f99912dc0 0x556f99921200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7f9ffbfff640 1 -- 172.21.15.33:0/447107045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 msgr2=0x556f99921990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:48.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7f9ffbfff640 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 0x556f99921990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7f9ffbfff640 1 -- 172.21.15.33:0/447107045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f99996e30 con 0x556f999124a0 2024-09-16T00:28:48.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7fa0008d4640 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 0x556f99921990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:48.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7f9ffbfff640 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f999124a0 0x556f99925ed0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9fec01d900 tx=0x7f9fec00ebe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a003223fdb71162d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:48.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7f9fdb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:28:48.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7f9fdb7fe640 1 -- 172.21.15.33:0/447107045 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fec039020 con 0x556f999124a0 2024-09-16T00:28:48.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7f9fdb7fe640 1 -- 172.21.15.33:0/447107045 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9fec009040 con 0x556f999124a0 2024-09-16T00:28:48.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7fa004710740 1 -- 172.21.15.33:0/447107045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ff00047a0 con 0x556f999124a0 2024-09-16T00:28:48.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7f9fdb7fe640 1 -- 172.21.15.33:0/447107045 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9fec01d620 con 0x556f999124a0 2024-09-16T00:28:48.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7fa004710740 10 client.26977 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:48.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.860+0000 7fa004710740 1 -- 172.21.15.33:0/447107045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556f99998000 con 0x556f999124a0 2024-09-16T00:28:48.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.861+0000 7f9fdb7fe640 1 -- 172.21.15.33:0/447107045 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ff00047a0 con 0x556f999124a0 2024-09-16T00:28:48.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.861+0000 7f9fdb7fe640 1 -- 172.21.15.33:0/447107045 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f9fec027420 con 0x556f999124a0 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.861+0000 7fa004710740 10 client.26977 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.861+0000 7fa004710740 10 client.26977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.861+0000 7fa004710740 20 client.26977 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.861+0000 7fa004710740 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556f9992d100 0x556f9999a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.861+0000 7fa004710740 4 client.26977 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.861+0000 7fa004710740 1 -- 172.21.15.33:0/447107045 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556f9987fd60 con 0x556f9992d100 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.861+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556f9992d100 0x556f9999a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.861+0000 7f9fdb7fe640 10 client.26977 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.861+0000 7f9ffb7fe640 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556f9992d100 0x556f9999a7d0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57dca0b8cc068133 server_cookie=590a0e50e23cf858 in_seq=0 out_seq=0 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7f9fdb7fe640 1 -- 172.21.15.33:0/447107045 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556f9987fd60 con 0x556f9992d100 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7f9fdb7fe640 10 client.26977 handle_command_reply: tid=0 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7fa004710740 1 client.26977 shutdown 2024-09-16T00:28:48.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7fa004710740 1 -- 172.21.15.33:0/447107045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f999124a0 msgr2=0x556f99925ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:48.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7fa004710740 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f999124a0 0x556f99925ed0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9fec01d900 tx=0x7f9fec00ebe0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7fa004710740 1 -- 172.21.15.33:0/447107045 shutdown_connections 2024-09-16T00:28:48.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7fa004710740 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556f9992d100 0x556f9999a7d0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7fa004710740 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f9991d680 0x556f99921990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7fa004710740 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556f99912dc0 0x556f99921200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7f9fdb7fe640 0 client.26977 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:48.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7fa004710740 1 --2- 172.21.15.33:0/447107045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f999124a0 0x556f99925ed0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7fa004710740 1 -- 172.21.15.33:0/447107045 >> 172.21.15.33:0/447107045 conn(0x556f9990ea30 msgr2=0x556f9990d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:48.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7fa004710740 1 -- 172.21.15.33:0/447107045 shutdown_connections 2024-09-16T00:28:48.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7fa004710740 1 -- 172.21.15.33:0/447107045 wait complete. 2024-09-16T00:28:48.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.862+0000 7fa004710740 20 client.26977 trim_cache size 0 max 16384 2024-09-16T00:28:48.868 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:28:48.868 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:28:48.868 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:28:48.868 INFO:teuthology.orchestra.run.smithi033.stdout: "4b5f50d9-c90d-4ea4-b6d2-3484b2fe8a36": { 2024-09-16T00:28:48.868 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:28:48.868 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "4b5f50d9-c90d-4ea4-b6d2-3484b2fe8a36", 2024-09-16T00:28:48.868 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:28:48.868 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:48.868 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:48.868 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:28:48.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.863+0000 7fa002337640 1 -- 172.21.15.33:0/3582371931 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9fe40a32c0 msgr2=0x7f9fe40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:48.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.863+0000 7fa002337640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9fe40a32c0 0x7f9fe40a5780 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009bd0 tx=0x7f9fec005f90 comp rx=0 tx=0).stop 2024-09-16T00:28:48.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.863+0000 7fa002337640 1 -- 172.21.15.33:0/3582371931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ffc102ef0 msgr2=0x7f9ffc0fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:48.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.863+0000 7fa002337640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ffc102ef0 0x7f9ffc0fd700 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400eaa0 tx=0x7f9ff400ef60 comp rx=0 tx=0).stop 2024-09-16T00:28:48.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.863+0000 7fa002337640 1 -- 172.21.15.33:0/3582371931 shutdown_connections 2024-09-16T00:28:48.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.863+0000 7fa002337640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9fe40a32c0 0x7f9fe40a5780 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.863+0000 7fa002337640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ffc102ef0 0x7f9ffc0fd700 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.863+0000 7fa002337640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ffc102570 0x7f9ffc0fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.863+0000 7fa002337640 1 --2- 172.21.15.33:0/3582371931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ffc100e20 0x7f9ffc0fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:48.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.863+0000 7fa002337640 1 -- 172.21.15.33:0/3582371931 >> 172.21.15.33:0/3582371931 conn(0x7f9ffc0fc7d0 msgr2=0x7f9ffc06a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:48.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.863+0000 7fa002337640 1 -- 172.21.15.33:0/3582371931 shutdown_connections 2024-09-16T00:28:48.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:48.863+0000 7fa002337640 1 -- 172.21.15.33:0/3582371931 wait complete. 2024-09-16T00:28:49.571 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:49 smithi033 ceph-mon[31243]: pgmap v700: 129 pgs: 129 active+clean; 274 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 3.3 MiB/s wr, 80 op/s 2024-09-16T00:28:49.589 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:49.604 INFO:tasks.cephfs.filesystem:scrub status for tag:4b5f50d9-c90d-4ea4-b6d2-3484b2fe8a36 - {'path': '/', 'tag': '4b5f50d9-c90d-4ea4-b6d2-3484b2fe8a36', 'options': 'recursive,force'} 2024-09-16T00:28:49.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:49 smithi083 ceph-mon[35242]: pgmap v700: 129 pgs: 129 active+clean; 274 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 3.3 MiB/s wr, 80 op/s 2024-09-16T00:28:49.682 INFO:tasks.workunit.client.0.smithi033.stdout: 1 986505 9.87 MB/sec execute 477 sec latency 22070.316 ms 2024-09-16T00:28:49.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:49 smithi060 ceph-mon[35185]: pgmap v700: 129 pgs: 129 active+clean; 274 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 3.3 MiB/s wr, 80 op/s 2024-09-16T00:28:50.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.314+0000 7efe91629640 1 -- 172.21.15.33:0/1618396257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c102790 msgr2=0x7efe8c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:50.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.315+0000 7efe91629640 1 --2- 172.21.15.33:0/1618396257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c102790 0x7efe8c1067e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7efe74009900 tx=0x7efe7402a410 comp rx=0 tx=0).stop 2024-09-16T00:28:50.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.315+0000 7efe91629640 1 -- 172.21.15.33:0/1618396257 shutdown_connections 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.315+0000 7efe91629640 1 --2- 172.21.15.33:0/1618396257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe8c106f10 0x7efe8c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.315+0000 7efe91629640 1 --2- 172.21.15.33:0/1618396257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c102790 0x7efe8c1067e0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.315+0000 7efe91629640 1 --2- 172.21.15.33:0/1618396257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe8c101de0 0x7efe8c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.315+0000 7efe91629640 1 -- 172.21.15.33:0/1618396257 >> 172.21.15.33:0/1618396257 conn(0x7efe8c0fd8c0 msgr2=0x7efe8c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.315+0000 7efe91629640 1 -- 172.21.15.33:0/1618396257 shutdown_connections 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.315+0000 7efe91629640 1 -- 172.21.15.33:0/1618396257 wait complete. 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.315+0000 7efe91629640 1 Processor -- start 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.315+0000 7efe91629640 1 -- start start 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.315+0000 7efe91629640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe8c101de0 0x7efe8c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.315+0000 7efe91629640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe8c102790 0x7efe8c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe91629640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c106f10 0x7efe8c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe91629640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c111f30 con 0x7efe8c106f10 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe91629640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c111db0 con 0x7efe8c102790 2024-09-16T00:28:50.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe91629640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1120b0 con 0x7efe8c101de0 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe8b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c106f10 0x7efe8c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe8b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c106f10 0x7efe8c19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56192/0 (socket says 172.21.15.33:56192) 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe8b7fe640 1 -- 172.21.15.33:0/3551224081 learned_addr learned my addr 172.21.15.33:0/3551224081 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe8affd640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe8c101de0 0x7efe8c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe83fff640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe8c102790 0x7efe8c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe8b7fe640 1 -- 172.21.15.33:0/3551224081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe8c101de0 msgr2=0x7efe8c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe8b7fe640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe8c101de0 0x7efe8c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe8b7fe640 1 -- 172.21.15.33:0/3551224081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe8c102790 msgr2=0x7efe8c1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe8b7fe640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe8c102790 0x7efe8c1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe8b7fe640 1 -- 172.21.15.33:0/3551224081 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe8c19d440 con 0x7efe8c106f10 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe83fff640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe8c102790 0x7efe8c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe8b7fe640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c106f10 0x7efe8c19cd60 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7efe8400eaa0 tx=0x7efe8400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe88ff9640 1 -- 172.21.15.33:0/3551224081 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8400cdb0 con 0x7efe8c106f10 2024-09-16T00:28:50.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe88ff9640 1 -- 172.21.15.33:0/3551224081 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe84005020 con 0x7efe8c106f10 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe88ff9640 1 -- 172.21.15.33:0/3551224081 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe840116b0 con 0x7efe8c106f10 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe91629640 1 -- 172.21.15.33:0/3551224081 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe8c19d730 con 0x7efe8c106f10 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.316+0000 7efe91629640 1 -- 172.21.15.33:0/3551224081 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe8c1a5010 con 0x7efe8c106f10 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.319+0000 7efe88ff9640 1 -- 172.21.15.33:0/3551224081 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7efe84005560 con 0x7efe8c106f10 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.319+0000 7efe93a02740 1 Processor -- start 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.319+0000 7efe93a02740 1 -- start start 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.319+0000 7efe93a02740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313ee99320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe93a02740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe93a02740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313eeaaa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe93a02740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56313edfe620 con 0x56313ee99860 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe93a02740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56313eda5e50 con 0x56313ee98f40 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe93a02740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56313edfa5e0 con 0x56313eea8650 2024-09-16T00:28:50.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe88ff9640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efe780a32c0 0x7efe780a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe8affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe8affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56202/0 (socket says 172.21.15.33:56202) 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe8affd640 1 -- 172.21.15.33:0/3361778495 learned_addr learned my addr 172.21.15.33:0/3361778495 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe83fff640 1 --2- 172.21.15.33:0/3361778495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313ee99320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe8b7fe640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efe780a32c0 0x7efe780a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe88ff9640 1 -- 172.21.15.33:0/3551224081 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7efe840cff10 con 0x7efe8c106f10 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe83fff640 1 --2- 172.21.15.33:0/3361778495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313eeaaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe8affd640 1 -- 172.21.15.33:0/3361778495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 msgr2=0x56313eeaaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe8affd640 1 --2- 172.21.15.33:0/3361778495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313eeaaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe8affd640 1 -- 172.21.15.33:0/3361778495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 msgr2=0x56313ee99320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe8affd640 1 --2- 172.21.15.33:0/3361778495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313ee99320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe8affd640 1 -- 172.21.15.33:0/3361778495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56313ed96d40 con 0x56313ee99860 2024-09-16T00:28:50.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe83fff640 1 --2- 172.21.15.33:0/3361778495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313ee99320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe8affd640 1 --2- 172.21.15.33:0/3361778495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea8110 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00bd10 tx=0x7efe7c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a6301d36a8f7702 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe8b7fe640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efe780a32c0 0x7efe780a5780 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7efe84005910 tx=0x7efe84009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.320+0000 7efe817fa640 1 -- 172.21.15.33:0/3361778495 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c004030 con 0x56313ee99860 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.321+0000 7efe817fa640 1 -- 172.21.15.33:0/3361778495 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe7c0041d0 con 0x56313ee99860 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.321+0000 7efe817fa640 1 -- 172.21.15.33:0/3361778495 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c0117e0 con 0x56313ee99860 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.321+0000 7efe93a02740 1 -- 172.21.15.33:0/3361778495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 msgr2=0x56313eea8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.321+0000 7efe93a02740 1 --2- 172.21.15.33:0/3361778495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea8110 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00bd10 tx=0x7efe7c0107a0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.321+0000 7efe93a02740 1 -- 172.21.15.33:0/3361778495 shutdown_connections 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.321+0000 7efe93a02740 1 --2- 172.21.15.33:0/3361778495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313eeaaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.321+0000 7efe93a02740 1 --2- 172.21.15.33:0/3361778495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea8110 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.321+0000 7efe93a02740 1 --2- 172.21.15.33:0/3361778495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313ee99320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.321+0000 7efe93a02740 1 -- 172.21.15.33:0/3361778495 >> 172.21.15.33:0/3361778495 conn(0x56313ee955f0 msgr2=0x56313ee959d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.321+0000 7efe93a02740 1 -- 172.21.15.33:0/3361778495 shutdown_connections 2024-09-16T00:28:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.321+0000 7efe93a02740 1 -- 172.21.15.33:0/3361778495 wait complete. 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe93a02740 1 -- start start 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe93a02740 1 Processor -- start 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe93a02740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313eea55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe93a02740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe93a02740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313eea6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe93a02740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56313eeab220 con 0x56313ee99860 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe93a02740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56313ed86f10 con 0x56313ee98f40 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe93a02740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56313eeab3a0 con 0x56313eea8650 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe93a02740 10 client.0 _ll_register_callbacks cb 0x56313edaadb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe8affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe8affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56208/0 (socket says 172.21.15.33:56208) 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe8affd640 1 -- 172.21.15.33:0/526385961 learned_addr learned my addr 172.21.15.33:0/526385961 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:50.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe83fff640 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313eea6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe83fff640 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313eea55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe8affd640 1 -- 172.21.15.33:0/526385961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 msgr2=0x56313eea6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe8affd640 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313eea6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe8affd640 1 -- 172.21.15.33:0/526385961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 msgr2=0x56313eea55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe8affd640 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313eea55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe8affd640 1 -- 172.21.15.33:0/526385961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56313eb0d580 con 0x56313ee99860 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.322+0000 7efe83fff640 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313eea6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe8affd640 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea5af0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7efe7c003f30 tx=0x7efe7c00ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f33afef77fe6fb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe727fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe727fc640 1 -- 172.21.15.33:0/526385961 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c029060 con 0x56313ee99860 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe727fc640 1 -- 172.21.15.33:0/526385961 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7efe7c0035f0 con 0x56313ee99860 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe93a02740 1 -- 172.21.15.33:0/526385961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe7c0041d0 con 0x56313ee99860 2024-09-16T00:28:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe727fc640 1 -- 172.21.15.33:0/526385961 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efe7c030050 con 0x56313ee99860 2024-09-16T00:28:50.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe93a02740 10 client.17613 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:50.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe93a02740 1 -- 172.21.15.33:0/526385961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56313ebd7960 con 0x56313ee99860 2024-09-16T00:28:50.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe727fc640 1 -- 172.21.15.33:0/526385961 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe7c0041d0 con 0x56313ee99860 2024-09-16T00:28:50.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe727fc640 1 -- 172.21.15.33:0/526385961 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7efe7c004750 con 0x56313ee99860 2024-09-16T00:28:50.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe93a02740 10 client.17613 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:50.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe93a02740 10 client.17613 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:28:50.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe93a02740 20 client.17613 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:50.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe93a02740 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56313eeb8220 0x56313eeba610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe93a02740 4 client.17613 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.323+0000 7efe93a02740 1 -- 172.21.15.33:0/526385961 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56313ee0bb90 con 0x56313eeb8220 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.324+0000 7efe83fff640 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56313eeb8220 0x56313eeba610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.324+0000 7efe83fff640 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56313eeb8220 0x56313eeba610 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a892265783f975d server_cookie=5bc237b3e8a69ac0 in_seq=0 out_seq=0 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.324+0000 7efe727fc640 10 client.17613 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe727fc640 1 -- 172.21.15.33:0/526385961 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56313ee0bb90 con 0x56313eeb8220 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe727fc640 10 client.17613 handle_command_reply: tid=0 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe93a02740 1 client.17613 shutdown 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe93a02740 1 -- 172.21.15.33:0/526385961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 msgr2=0x56313eea5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe93a02740 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea5af0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7efe7c003f30 tx=0x7efe7c00ba20 comp rx=0 tx=0).stop 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe93a02740 1 -- 172.21.15.33:0/526385961 shutdown_connections 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe93a02740 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56313eeb8220 0x56313eeba610 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe93a02740 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313eea6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe93a02740 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea5af0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe727fc640 0 client.17613 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:50.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe93a02740 1 --2- 172.21.15.33:0/526385961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313eea55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe93a02740 1 -- 172.21.15.33:0/526385961 >> 172.21.15.33:0/526385961 conn(0x56313ee955f0 msgr2=0x56313ee9b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:50.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe93a02740 1 -- 172.21.15.33:0/526385961 shutdown_connections 2024-09-16T00:28:50.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe93a02740 1 -- 172.21.15.33:0/526385961 wait complete. 2024-09-16T00:28:50.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.325+0000 7efe93a02740 20 client.17613 trim_cache size 0 max 16384 2024-09-16T00:28:50.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.343+0000 7efe93a02740 1 Processor -- start 2024-09-16T00:28:50.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.343+0000 7efe93a02740 1 -- start start 2024-09-16T00:28:50.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.343+0000 7efe93a02740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313ee98f40 0x56313eea7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.343+0000 7efe93a02740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.343+0000 7efe93a02740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313eea8650 0x56313eea2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.343+0000 7efe93a02740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1af800 con 0x56313ee99860 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.343+0000 7efe93a02740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1ad0d0 con 0x56313eea8650 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.343+0000 7efe93a02740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1aaf20 con 0x56313ee98f40 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.343+0000 7efe8affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.343+0000 7efe8affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56224/0 (socket says 172.21.15.33:56224) 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.343+0000 7efe8affd640 1 -- 172.21.15.33:0/1409217717 learned_addr learned my addr 172.21.15.33:0/1409217717 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.343+0000 7efe83fff640 1 --2- 172.21.15.33:0/1409217717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313ee98f40 0x56313eea7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.344+0000 7efe83fff640 1 --2- 172.21.15.33:0/1409217717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313eea8650 0x56313eea2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.344+0000 7efe8affd640 1 -- 172.21.15.33:0/1409217717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313ee98f40 msgr2=0x56313eea7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.344+0000 7efe8affd640 1 --2- 172.21.15.33:0/1409217717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313ee98f40 0x56313eea7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.344+0000 7efe8affd640 1 -- 172.21.15.33:0/1409217717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313eea8650 msgr2=0x56313eea2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.344+0000 7efe8affd640 1 --2- 172.21.15.33:0/1409217717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313eea8650 0x56313eea2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.344+0000 7efe8affd640 1 -- 172.21.15.33:0/1409217717 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56313ee95420 con 0x56313ee99860 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.344+0000 7efe83fff640 1 --2- 172.21.15.33:0/1409217717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313eea8650 0x56313eea2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.344+0000 7efe8affd640 1 --2- 172.21.15.33:0/1409217717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea2810 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7efe7c004cc0 tx=0x7efe7c005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fef4343ea960289 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.344+0000 7efe80ff9640 1 -- 172.21.15.33:0/1409217717 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c039070 con 0x56313ee99860 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.344+0000 7efe80ff9640 1 -- 172.21.15.33:0/1409217717 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe7c004150 con 0x56313ee99860 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.344+0000 7efe80ff9640 1 -- 172.21.15.33:0/1409217717 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c029040 con 0x56313ee99860 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.345+0000 7efe93a02740 1 -- 172.21.15.33:0/1409217717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 msgr2=0x56313eea2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.345+0000 7efe93a02740 1 --2- 172.21.15.33:0/1409217717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea2810 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7efe7c004cc0 tx=0x7efe7c005c50 comp rx=0 tx=0).stop 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.345+0000 7efe93a02740 1 -- 172.21.15.33:0/1409217717 shutdown_connections 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.345+0000 7efe93a02740 1 --2- 172.21.15.33:0/1409217717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313eea8650 0x56313eea2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.345+0000 7efe93a02740 1 --2- 172.21.15.33:0/1409217717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eea2810 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.345+0000 7efe93a02740 1 --2- 172.21.15.33:0/1409217717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313ee98f40 0x56313eea7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.345+0000 7efe93a02740 1 -- 172.21.15.33:0/1409217717 >> 172.21.15.33:0/1409217717 conn(0x56313ee955f0 msgr2=0x56313eeb3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:50.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.345+0000 7efe93a02740 1 -- 172.21.15.33:0/1409217717 shutdown_connections 2024-09-16T00:28:50.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.345+0000 7efe93a02740 1 -- 172.21.15.33:0/1409217717 wait complete. 2024-09-16T00:28:50.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.345+0000 7efe93a02740 1 -- start start 2024-09-16T00:28:50.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe93a02740 1 Processor -- start 2024-09-16T00:28:50.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe93a02740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313eebbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe93a02740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eebc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe93a02740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313ef1bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe93a02740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1ab590 con 0x56313ee99860 2024-09-16T00:28:50.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe93a02740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1aca00 con 0x56313ee98f40 2024-09-16T00:28:50.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe93a02740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1abc60 con 0x56313eea8650 2024-09-16T00:28:50.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe93a02740 10 client.0 _ll_register_callbacks cb 0x7efe7c00b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:50.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe8affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eebc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe8affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eebc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56236/0 (socket says 172.21.15.33:56236) 2024-09-16T00:28:50.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe8affd640 1 -- 172.21.15.33:0/2880052236 learned_addr learned my addr 172.21.15.33:0/2880052236 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:50.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe83fff640 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313ef1bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe83fff640 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313eebbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe8affd640 1 -- 172.21.15.33:0/2880052236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 msgr2=0x56313ef1bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:50.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe8affd640 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313ef1bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe8affd640 1 -- 172.21.15.33:0/2880052236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 msgr2=0x56313eebbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:50.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe8affd640 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313eebbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.346+0000 7efe8affd640 1 -- 172.21.15.33:0/2880052236 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56313ef1c120 con 0x56313ee99860 2024-09-16T00:28:50.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe8affd640 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eebc150 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7efe7c001eb0 tx=0x7efe7c00b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee253afbe41af65 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:50.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe727fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:50.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe727fc640 1 -- 172.21.15.33:0/2880052236 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c039030 con 0x56313ee99860 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe727fc640 1 -- 172.21.15.33:0/2880052236 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7efe7c021040 con 0x56313ee99860 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe93a02740 1 -- 172.21.15.33:0/2880052236 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe7c004150 con 0x56313ee99860 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe727fc640 1 -- 172.21.15.33:0/2880052236 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efe7c029050 con 0x56313ee99860 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe93a02740 10 client.17619 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe93a02740 1 -- 172.21.15.33:0/2880052236 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56313ef234e0 con 0x56313ee99860 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe727fc640 1 -- 172.21.15.33:0/2880052236 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe7c004150 con 0x56313ee99860 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe727fc640 1 -- 172.21.15.33:0/2880052236 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7efe7c011940 con 0x56313ee99860 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe93a02740 10 client.17619 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe93a02740 10 client.17619 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe93a02740 20 client.17619 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe93a02740 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56313eeb8220 0x56313ef25b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe93a02740 4 client.17619 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b9e606eb", "release": true, "await": true}] 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.347+0000 7efe93a02740 1 -- 172.21.15.33:0/2880052236 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b9e606eb", "release": true, "await": true}) -- 0x56313ee060a0 con 0x56313eeb8220 2024-09-16T00:28:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.348+0000 7efe83fff640 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56313eeb8220 0x56313ef25b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:50.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.348+0000 7efe727fc640 10 client.17619 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:50.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.348+0000 7efe83fff640 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56313eeb8220 0x56313ef25b80 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=304ba402aacb650f server_cookie=9d7ed69e6e7f9269 in_seq=0 out_seq=0 2024-09-16T00:28:50.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.356+0000 7efe727fc640 1 -- 172.21.15.33:0/2880052236 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56313ee060a0 con 0x56313eeb8220 2024-09-16T00:28:50.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.356+0000 7efe727fc640 10 client.17619 handle_command_reply: tid=0 2024-09-16T00:28:50.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.356+0000 7efe93a02740 1 client.17619 shutdown 2024-09-16T00:28:50.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe93a02740 1 -- 172.21.15.33:0/2880052236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 msgr2=0x56313eebc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:50.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe93a02740 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eebc150 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7efe7c001eb0 tx=0x7efe7c00b770 comp rx=0 tx=0).stop 2024-09-16T00:28:50.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe93a02740 1 -- 172.21.15.33:0/2880052236 shutdown_connections 2024-09-16T00:28:50.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe93a02740 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56313eeb8220 0x56313ef25b80 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe93a02740 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56313eea8650 0x56313ef1bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe93a02740 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56313ee99860 0x56313eebc150 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe727fc640 0 client.17619 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:50.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe93a02740 1 --2- 172.21.15.33:0/2880052236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56313ee98f40 0x56313eebbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe93a02740 1 -- 172.21.15.33:0/2880052236 >> 172.21.15.33:0/2880052236 conn(0x56313ee955f0 msgr2=0x56313ee93f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:50.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe93a02740 1 -- 172.21.15.33:0/2880052236 shutdown_connections 2024-09-16T00:28:50.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe93a02740 1 -- 172.21.15.33:0/2880052236 wait complete. 2024-09-16T00:28:50.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe93a02740 20 client.17619 trim_cache size 0 max 16384 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 40, 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout: "b9e606eb": { 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 40, 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:28:50.359 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:28:50.360 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 141.8, 2024-09-16T00:28:50.360 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:28:50.360 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:28:50.360 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:28:50.360 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:28:50.360 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:28:50.360 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:28:50.360 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:50.360 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:50.360 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:50.360 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:50.360 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:50.361 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:28:50.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe91629640 1 -- 172.21.15.33:0/3551224081 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efe780a32c0 msgr2=0x7efe780a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:50.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe91629640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efe780a32c0 0x7efe780a5780 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7efe84005910 tx=0x7efe84009270 comp rx=0 tx=0).stop 2024-09-16T00:28:50.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe91629640 1 -- 172.21.15.33:0/3551224081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c106f10 msgr2=0x7efe8c19cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:50.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.357+0000 7efe91629640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c106f10 0x7efe8c19cd60 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7efe8400eaa0 tx=0x7efe8400ef60 comp rx=0 tx=0).stop 2024-09-16T00:28:50.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.358+0000 7efe91629640 1 -- 172.21.15.33:0/3551224081 shutdown_connections 2024-09-16T00:28:50.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.358+0000 7efe91629640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efe780a32c0 0x7efe780a5780 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.358+0000 7efe91629640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c106f10 0x7efe8c19cd60 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.358+0000 7efe91629640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe8c102790 0x7efe8c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.358+0000 7efe91629640 1 --2- 172.21.15.33:0/3551224081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe8c101de0 0x7efe8c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:50.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.358+0000 7efe91629640 1 -- 172.21.15.33:0/3551224081 >> 172.21.15.33:0/3551224081 conn(0x7efe8c0fd8c0 msgr2=0x7efe8c0fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:50.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.358+0000 7efe91629640 1 -- 172.21.15.33:0/3551224081 shutdown_connections 2024-09-16T00:28:50.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:50.358+0000 7efe91629640 1 -- 172.21.15.33:0/3551224081 wait complete. 2024-09-16T00:28:50.683 INFO:tasks.workunit.client.0.smithi033.stdout: 1 987470 9.87 MB/sec execute 478 sec latency 22739.046 ms 2024-09-16T00:28:51.117 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b9e606eb', total seconds elapsed: 26.4 2024-09-16T00:28:51.117 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.85 2024-09-16T00:28:51.117 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:28:51.388 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:51.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:51 smithi083 ceph-mon[35242]: pgmap v701: 129 pgs: 129 active+clean; 273 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 3.3 MiB/s wr, 90 op/s 2024-09-16T00:28:51.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:51 smithi033 ceph-mon[31243]: pgmap v701: 129 pgs: 129 active+clean; 273 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 3.3 MiB/s wr, 90 op/s 2024-09-16T00:28:51.682 INFO:tasks.workunit.client.0.smithi033.stdout: 1 989866 9.87 MB/sec execute 479 sec latency 15.104 ms 2024-09-16T00:28:51.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:51 smithi060 ceph-mon[35185]: pgmap v701: 129 pgs: 129 active+clean; 273 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 3.3 MiB/s wr, 90 op/s 2024-09-16T00:28:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.870+0000 7f18fae52640 1 -- 172.21.15.33:0/2643888251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f4103780 msgr2=0x7f18f4107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.871+0000 7f18fae52640 1 --2- 172.21.15.33:0/2643888251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f4103780 0x7f18f4107bc0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009930 tx=0x7f18e002a420 comp rx=0 tx=0).stop 2024-09-16T00:28:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.871+0000 7f18fae52640 1 -- 172.21.15.33:0/2643888251 shutdown_connections 2024-09-16T00:28:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.871+0000 7f18fae52640 1 --2- 172.21.15.33:0/2643888251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18f4108300 0x7f18f410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.871+0000 7f18fae52640 1 --2- 172.21.15.33:0/2643888251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f4103780 0x7f18f4107bc0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.871+0000 7f18fae52640 1 --2- 172.21.15.33:0/2643888251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18f4102dd0 0x7f18f41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.871+0000 7f18fae52640 1 -- 172.21.15.33:0/2643888251 >> 172.21.15.33:0/2643888251 conn(0x7f18f40fc7d0 msgr2=0x7f18f40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.871+0000 7f18fae52640 1 -- 172.21.15.33:0/2643888251 shutdown_connections 2024-09-16T00:28:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.871+0000 7f18fae52640 1 -- 172.21.15.33:0/2643888251 wait complete. 2024-09-16T00:28:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.871+0000 7f18fae52640 1 Processor -- start 2024-09-16T00:28:51.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.871+0000 7f18fae52640 1 -- start start 2024-09-16T00:28:51.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.871+0000 7f18fae52640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18f4102dd0 0x7f18f4100a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:51.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.871+0000 7f18fae52640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f4103780 0x7f18f4100f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:51.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18fae52640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18f4108300 0x7f18f41014b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:51.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18fae52640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18f4112070 con 0x7f18f4103780 2024-09-16T00:28:51.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18fae52640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f18f4111ef0 con 0x7f18f4102dd0 2024-09-16T00:28:51.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18fae52640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f18f41121f0 con 0x7f18f4108300 2024-09-16T00:28:51.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f4103780 0x7f18f4100f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:51.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f4103780 0x7f18f4100f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56246/0 (socket says 172.21.15.33:56246) 2024-09-16T00:28:51.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f3fff640 1 -- 172.21.15.33:0/929558355 learned_addr learned my addr 172.21.15.33:0/929558355 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:51.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f8bee640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18f4102dd0 0x7f18f4100a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:51.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f93ef640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18f4108300 0x7f18f41014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:51.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f3fff640 1 -- 172.21.15.33:0/929558355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18f4108300 msgr2=0x7f18f41014b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:51.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f3fff640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18f4108300 0x7f18f41014b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:51.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f3fff640 1 -- 172.21.15.33:0/929558355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18f4102dd0 msgr2=0x7f18f4100a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:51.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f3fff640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18f4102dd0 0x7f18f4100a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:51.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f3fff640 1 -- 172.21.15.33:0/929558355 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f41a6a90 con 0x7f18f4103780 2024-09-16T00:28:51.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f8bee640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18f4102dd0 0x7f18f4100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:51.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f3fff640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f4103780 0x7f18f4100f70 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f18e0005940 tx=0x7f18e0031b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:51.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f1ffb640 1 -- 172.21.15.33:0/929558355 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e003f030 con 0x7f18f4103780 2024-09-16T00:28:51.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f1ffb640 1 -- 172.21.15.33:0/929558355 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18e00043d0 con 0x7f18f4103780 2024-09-16T00:28:51.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18fae52640 1 -- 172.21.15.33:0/929558355 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f41a6d20 con 0x7f18f4103780 2024-09-16T00:28:51.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.872+0000 7f18f1ffb640 1 -- 172.21.15.33:0/929558355 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e0035070 con 0x7f18f4103780 2024-09-16T00:28:51.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.873+0000 7f18fae52640 1 -- 172.21.15.33:0/929558355 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f41a71b0 con 0x7f18f4103780 2024-09-16T00:28:51.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.874+0000 7f18fae52640 1 -- 172.21.15.33:0/929558355 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18f41031b0 con 0x7f18f4103780 2024-09-16T00:28:51.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.876+0000 7f18f1ffb640 1 -- 172.21.15.33:0/929558355 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f18e0042030 con 0x7f18f4103780 2024-09-16T00:28:51.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.876+0000 7f18f1ffb640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18e40a3240 0x7f18e40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:51.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.876+0000 7f18f1ffb640 1 -- 172.21.15.33:0/929558355 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f18e00f0270 con 0x7f18f4103780 2024-09-16T00:28:51.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.876+0000 7f18f8bee640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18e40a3240 0x7f18e40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:51.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.877+0000 7f18f8bee640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18e40a3240 0x7f18e40a5700 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009fc0 tx=0x7f18e8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:51.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:51.881+0000 7f18f1ffb640 1 -- 172.21.15.33:0/929558355 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f18e00bc4c0 con 0x7f18f4103780 2024-09-16T00:28:52.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.100+0000 7f18fae52640 1 -- 172.21.15.33:0/929558355 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18f410a290 con 0x7f18f4103780 2024-09-16T00:28:52.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.101+0000 7f18f1ffb640 1 -- 172.21.15.33:0/929558355 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f18e00c07c0 con 0x7f18f4103780 2024-09-16T00:28:52.101 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:28:52.102 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:28:52.102 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:28:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.103+0000 7f18fae52640 1 -- 172.21.15.33:0/929558355 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18e40a3240 msgr2=0x7f18e40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.103+0000 7f18fae52640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18e40a3240 0x7f18e40a5700 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009fc0 tx=0x7f18e8005f90 comp rx=0 tx=0).stop 2024-09-16T00:28:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.103+0000 7f18fae52640 1 -- 172.21.15.33:0/929558355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f4103780 msgr2=0x7f18f4100f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.103+0000 7f18fae52640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f4103780 0x7f18f4100f70 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f18e0005940 tx=0x7f18e0031b50 comp rx=0 tx=0).stop 2024-09-16T00:28:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.103+0000 7f18fae52640 1 -- 172.21.15.33:0/929558355 shutdown_connections 2024-09-16T00:28:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.103+0000 7f18fae52640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18e40a3240 0x7f18e40a5700 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.103+0000 7f18fae52640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18f4108300 0x7f18f41014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.103+0000 7f18fae52640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18f4103780 0x7f18f4100f70 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.103+0000 7f18fae52640 1 --2- 172.21.15.33:0/929558355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18f4102dd0 0x7f18f4100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.103+0000 7f18fae52640 1 -- 172.21.15.33:0/929558355 >> 172.21.15.33:0/929558355 conn(0x7f18f40fc7d0 msgr2=0x7f18f4105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.103+0000 7f18fae52640 1 -- 172.21.15.33:0/929558355 shutdown_connections 2024-09-16T00:28:52.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:52.103+0000 7f18fae52640 1 -- 172.21.15.33:0/929558355 wait complete. 2024-09-16T00:28:52.683 INFO:tasks.workunit.client.0.smithi033.stdout: 1 991917 9.88 MB/sec execute 480 sec latency 45.719 ms 2024-09-16T00:28:52.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:52 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:28:52.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:52 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:28:52.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:52 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/929558355' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:28:52.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:52 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:28:52.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:52 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:28:52.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:52 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/929558355' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:28:52.829 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0' 2024-09-16T00:28:52.830 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0 2024-09-16T00:28:52.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:52 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:28:52.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:52 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:28:52.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:52 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/929558355' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:28:53.098 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:28:53.504 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:53 smithi033 ceph-mon[31243]: pgmap v702: 129 pgs: 129 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 3.0 MiB/s wr, 90 op/s 2024-09-16T00:28:53.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.605+0000 7f99f7532640 1 -- 172.21.15.33:0/3786426808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0100a60 msgr2=0x7f99f0100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:53.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.605+0000 7f99f7532640 1 --2- 172.21.15.33:0/3786426808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0100a60 0x7f99f0100ec0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009930 tx=0x7f99e402a420 comp rx=0 tx=0).stop 2024-09-16T00:28:53.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.605+0000 7f99f7532640 1 -- 172.21.15.33:0/3786426808 shutdown_connections 2024-09-16T00:28:53.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.605+0000 7f99f7532640 1 --2- 172.21.15.33:0/3786426808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99f0101400 0x7f99f010d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.605+0000 7f99f7532640 1 --2- 172.21.15.33:0/3786426808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0100a60 0x7f99f0100ec0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.605+0000 7f99f7532640 1 --2- 172.21.15.33:0/3786426808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f0106a60 0x7f99f0106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.605+0000 7f99f7532640 1 -- 172.21.15.33:0/3786426808 >> 172.21.15.33:0/3786426808 conn(0x7f99f00fc780 msgr2=0x7f99f00feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:53.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.605+0000 7f99f7532640 1 -- 172.21.15.33:0/3786426808 shutdown_connections 2024-09-16T00:28:53.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.605+0000 7f99f7532640 1 -- 172.21.15.33:0/3786426808 wait complete. 2024-09-16T00:28:53.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.605+0000 7f99f7532640 1 Processor -- start 2024-09-16T00:28:53.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f7532640 1 -- start start 2024-09-16T00:28:53.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f7532640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99f0100a60 0x7f99f01a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f7532640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0101400 0x7f99f01a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f7532640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f0106a60 0x7f99f019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f7532640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f99f0110eb0 con 0x7f99f0101400 2024-09-16T00:28:53.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f7532640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f99f0110d30 con 0x7f99f0106a60 2024-09-16T00:28:53.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f7532640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f99f0111030 con 0x7f99f0100a60 2024-09-16T00:28:53.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f52ce640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99f0100a60 0x7f99f01a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f52ce640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99f0100a60 0x7f99f01a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:48568/0 (socket says 172.21.15.33:48568) 2024-09-16T00:28:53.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f52ce640 1 -- 172.21.15.33:0/2013295909 learned_addr learned my addr 172.21.15.33:0/2013295909 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:53.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f4acd640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0101400 0x7f99f01a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f5acf640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f0106a60 0x7f99f019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f52ce640 1 -- 172.21.15.33:0/2013295909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f0106a60 msgr2=0x7f99f019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:53.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f52ce640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f0106a60 0x7f99f019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f52ce640 1 -- 172.21.15.33:0/2013295909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0101400 msgr2=0x7f99f01a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:53.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.606+0000 7f99f52ce640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0101400 0x7f99f01a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.607+0000 7f99f52ce640 1 -- 172.21.15.33:0/2013295909 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f019f280 con 0x7f99f0100a60 2024-09-16T00:28:53.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.607+0000 7f99f52ce640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99f0100a60 0x7f99f01a46c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99e000ebc0 tx=0x7f99e000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:53.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.607+0000 7f99ee7fc640 1 -- 172.21.15.33:0/2013295909 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0010630 con 0x7f99f0100a60 2024-09-16T00:28:53.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.607+0000 7f99ee7fc640 1 -- 172.21.15.33:0/2013295909 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e0010cb0 con 0x7f99f0100a60 2024-09-16T00:28:53.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.607+0000 7f99f7532640 1 -- 172.21.15.33:0/2013295909 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99f019f570 con 0x7f99f0100a60 2024-09-16T00:28:53.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.607+0000 7f99ee7fc640 1 -- 172.21.15.33:0/2013295909 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0018bd0 con 0x7f99f0100a60 2024-09-16T00:28:53.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.607+0000 7f99f7532640 1 -- 172.21.15.33:0/2013295909 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99f01ab590 con 0x7f99f0100a60 2024-09-16T00:28:53.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.609+0000 7f99ee7fc640 1 -- 172.21.15.33:0/2013295909 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f99e00107d0 con 0x7f99f0100a60 2024-09-16T00:28:53.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.610+0000 7f99f990b740 1 Processor -- start 2024-09-16T00:28:53.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.610+0000 7f99f990b740 1 -- start start 2024-09-16T00:28:53.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.610+0000 7f99f990b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 0x55f5db8b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.610+0000 7f99f990b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b8170 0x55f5db8c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.610+0000 7f99f990b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8c2a30 0x55f5db8c4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.610+0000 7f99f990b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f5db81c160 con 0x55f5db8b7850 2024-09-16T00:28:53.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.610+0000 7f99f990b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f5db7c3720 con 0x55f5db8b8170 2024-09-16T00:28:53.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.610+0000 7f99f990b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f5db818120 con 0x55f5db8c2a30 2024-09-16T00:28:53.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99f5acf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 0x55f5db8b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99f5acf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 0x55f5db8b7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56286/0 (socket says 172.21.15.33:56286) 2024-09-16T00:28:53.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99f5acf640 1 -- 172.21.15.33:0/2586697947 learned_addr learned my addr 172.21.15.33:0/2586697947 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:53.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99f52ce640 1 --2- 172.21.15.33:0/2586697947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b8170 0x55f5db8c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99f4acd640 1 --2- 172.21.15.33:0/2586697947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8c2a30 0x55f5db8c4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99ee7fc640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f99d80a3240 0x7f99d80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99f5acf640 1 -- 172.21.15.33:0/2586697947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8c2a30 msgr2=0x55f5db8c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:53.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99f5acf640 1 --2- 172.21.15.33:0/2586697947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8c2a30 0x55f5db8c4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99f5acf640 1 -- 172.21.15.33:0/2586697947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b8170 msgr2=0x55f5db8c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:53.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99f5acf640 1 --2- 172.21.15.33:0/2586697947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b8170 0x55f5db8c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99f5acf640 1 -- 172.21.15.33:0/2586697947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5db58dba0 con 0x55f5db8b7850 2024-09-16T00:28:53.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99f5acf640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f99d80a3240 0x7f99d80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99ee7fc640 1 -- 172.21.15.33:0/2013295909 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f99e00d0350 con 0x7f99f0100a60 2024-09-16T00:28:53.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.611+0000 7f99f5acf640 1 --2- 172.21.15.33:0/2586697947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 0x55f5db8b7c30 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009e20 tx=0x7f99e8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9707899c839b434 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:53.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99d77fe640 1 -- 172.21.15.33:0/2586697947 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e8017050 con 0x55f5db8b7850 2024-09-16T00:28:53.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99d77fe640 1 -- 172.21.15.33:0/2586697947 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e8009e50 con 0x55f5db8b7850 2024-09-16T00:28:53.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99d77fe640 1 -- 172.21.15.33:0/2586697947 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e800f3c0 con 0x55f5db8b7850 2024-09-16T00:28:53.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99f990b740 1 -- 172.21.15.33:0/2586697947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 msgr2=0x55f5db8b7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:53.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99f5acf640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f99d80a3240 0x7f99d80a5700 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f99e8023000 tx=0x7f99e8024000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:53.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99f990b740 1 --2- 172.21.15.33:0/2586697947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 0x55f5db8b7c30 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009e20 tx=0x7f99e8011810 comp rx=0 tx=0).stop 2024-09-16T00:28:53.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99f990b740 1 -- 172.21.15.33:0/2586697947 shutdown_connections 2024-09-16T00:28:53.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99f990b740 1 --2- 172.21.15.33:0/2586697947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8c2a30 0x55f5db8c4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99f990b740 1 --2- 172.21.15.33:0/2586697947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b8170 0x55f5db8c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99f990b740 1 --2- 172.21.15.33:0/2586697947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 0x55f5db8b7c30 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99f990b740 1 -- 172.21.15.33:0/2586697947 >> 172.21.15.33:0/2586697947 conn(0x55f5db8b3de0 msgr2=0x55f5db8b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:53.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99f990b740 1 -- 172.21.15.33:0/2586697947 shutdown_connections 2024-09-16T00:28:53.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.612+0000 7f99f990b740 1 -- 172.21.15.33:0/2586697947 wait complete. 2024-09-16T00:28:53.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.613+0000 7f99f990b740 1 -- start start 2024-09-16T00:28:53.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.613+0000 7f99f990b740 1 Processor -- start 2024-09-16T00:28:53.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.613+0000 7f99f990b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 0x55f5db8bf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.613+0000 7f99f990b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 0x55f5db8bfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.613+0000 7f99f990b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8c2a30 0x55f5db8c03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.613+0000 7f99f990b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f5db8c5600 con 0x55f5db8b7850 2024-09-16T00:28:53.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.613+0000 7f99f990b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f5db7a4900 con 0x55f5db8c2a30 2024-09-16T00:28:53.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.613+0000 7f99f990b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f5db8c5780 con 0x55f5db8b8170 2024-09-16T00:28:53.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.613+0000 7f99f990b740 10 client.0 _ll_register_callbacks cb 0x55f5db7a2780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:53.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f52ce640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 0x55f5db8bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f52ce640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 0x55f5db8bf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56298/0 (socket says 172.21.15.33:56298) 2024-09-16T00:28:53.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f52ce640 1 -- 172.21.15.33:0/2856338266 learned_addr learned my addr 172.21.15.33:0/2856338266 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:53.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f4acd640 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 0x55f5db8bfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f5acf640 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8c2a30 0x55f5db8c03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f52ce640 1 -- 172.21.15.33:0/2856338266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 msgr2=0x55f5db8bfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:53.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f52ce640 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 0x55f5db8bfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f52ce640 1 -- 172.21.15.33:0/2856338266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8c2a30 msgr2=0x55f5db8c03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:53.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f52ce640 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8c2a30 0x55f5db8c03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f52ce640 1 -- 172.21.15.33:0/2856338266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5db7e4c50 con 0x55f5db8b7850 2024-09-16T00:28:53.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f5acf640 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8c2a30 0x55f5db8c03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:28:53.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f4acd640 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 0x55f5db8bfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:53.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f52ce640 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 0x55f5db8bf940 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f99e0098f10 tx=0x7f99e009c080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839885cb969021da server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:53.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:28:53.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99d4ff9640 1 -- 172.21.15.33:0/2856338266 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e00d0120 con 0x55f5db8b7850 2024-09-16T00:28:53.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99d4ff9640 1 -- 172.21.15.33:0/2856338266 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f99e009b990 con 0x55f5db8b7850 2024-09-16T00:28:53.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.614+0000 7f99f990b740 1 -- 172.21.15.33:0/2856338266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99e8009e50 con 0x55f5db8b7850 2024-09-16T00:28:53.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.615+0000 7f99d4ff9640 1 -- 172.21.15.33:0/2856338266 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f99e00a5100 con 0x55f5db8b7850 2024-09-16T00:28:53.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.615+0000 7f99f990b740 10 client.17637 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:53.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.615+0000 7f99f990b740 1 -- 172.21.15.33:0/2856338266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f5db8d12e0 con 0x55f5db8b7850 2024-09-16T00:28:53.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.615+0000 7f99d4ff9640 1 -- 172.21.15.33:0/2856338266 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e8009e50 con 0x55f5db8b7850 2024-09-16T00:28:53.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.615+0000 7f99d4ff9640 1 -- 172.21.15.33:0/2856338266 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f99e00b5180 con 0x55f5db8b7850 2024-09-16T00:28:53.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.615+0000 7f99f990b740 10 client.17637 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:53.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.615+0000 7f99f990b740 10 client.17637 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:28:53.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.615+0000 7f99f990b740 20 client.17637 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:53.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.615+0000 7f99f990b740 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f5db8d27a0 0x55f5db8d4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.615+0000 7f99f990b740 4 client.17637 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:28:53.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.615+0000 7f99f990b740 1 -- 172.21.15.33:0/2856338266 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5db829890 con 0x55f5db8d27a0 2024-09-16T00:28:53.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.615+0000 7f99f4acd640 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f5db8d27a0 0x55f5db8d4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.616+0000 7f99f4acd640 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f5db8d27a0 0x55f5db8d4bf0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90b50c7479b79727 server_cookie=9d87e84b952a771a in_seq=0 out_seq=0 2024-09-16T00:28:53.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.616+0000 7f99d4ff9640 10 client.17637 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:53.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.616+0000 7f99d4ff9640 1 -- 172.21.15.33:0/2856338266 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f5db829890 con 0x55f5db8d27a0 2024-09-16T00:28:53.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.616+0000 7f99d4ff9640 10 client.17637 handle_command_reply: tid=0 2024-09-16T00:28:53.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.616+0000 7f99f990b740 1 client.17637 shutdown 2024-09-16T00:28:53.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.616+0000 7f99f990b740 1 -- 172.21.15.33:0/2856338266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 msgr2=0x55f5db8bf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:53.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.616+0000 7f99f990b740 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 0x55f5db8bf940 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f99e0098f10 tx=0x7f99e009c080 comp rx=0 tx=0).stop 2024-09-16T00:28:53.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.617+0000 7f99f990b740 1 -- 172.21.15.33:0/2856338266 shutdown_connections 2024-09-16T00:28:53.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.617+0000 7f99f990b740 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f5db8d27a0 0x55f5db8d4bf0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.617+0000 7f99f990b740 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8c2a30 0x55f5db8c03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.617+0000 7f99f990b740 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 0x55f5db8bfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.617+0000 7f99f990b740 1 --2- 172.21.15.33:0/2856338266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b7850 0x55f5db8bf940 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.617+0000 7f99f990b740 1 -- 172.21.15.33:0/2856338266 >> 172.21.15.33:0/2856338266 conn(0x55f5db8b3de0 msgr2=0x55f5db8b9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:53.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.617+0000 7f99d4ff9640 0 client.17637 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:53.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.617+0000 7f99f990b740 1 -- 172.21.15.33:0/2856338266 shutdown_connections 2024-09-16T00:28:53.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.617+0000 7f99f990b740 1 -- 172.21.15.33:0/2856338266 wait complete. 2024-09-16T00:28:53.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.617+0000 7f99f990b740 20 client.17637 trim_cache size 0 max 16384 2024-09-16T00:28:53.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.635+0000 7f99f990b740 1 Processor -- start 2024-09-16T00:28:53.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.635+0000 7f99f990b740 1 -- start start 2024-09-16T00:28:53.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.635+0000 7f99f990b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b7850 0x55f5db8b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.635+0000 7f99f990b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b8170 0x55f5db8c0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.635+0000 7f99f990b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8c2a30 0x55f5db8b9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.635+0000 7f99f990b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01b8040 con 0x55f5db8b8170 2024-09-16T00:28:53.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.635+0000 7f99f990b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01b5910 con 0x55f5db8b7850 2024-09-16T00:28:53.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.635+0000 7f99f990b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01b3760 con 0x55f5db8c2a30 2024-09-16T00:28:53.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.635+0000 7f99f4acd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b8170 0x55f5db8c0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.635+0000 7f99f52ce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b7850 0x55f5db8b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.635+0000 7f99f4acd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b8170 0x55f5db8c0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56304/0 (socket says 172.21.15.33:56304) 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.635+0000 7f99f4acd640 1 -- 172.21.15.33:0/1003463233 learned_addr learned my addr 172.21.15.33:0/1003463233 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.636+0000 7f99f5acf640 1 --2- 172.21.15.33:0/1003463233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8c2a30 0x55f5db8b9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.636+0000 7f99f4acd640 1 -- 172.21.15.33:0/1003463233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8c2a30 msgr2=0x55f5db8b9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.636+0000 7f99f4acd640 1 --2- 172.21.15.33:0/1003463233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8c2a30 0x55f5db8b9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.636+0000 7f99f4acd640 1 -- 172.21.15.33:0/1003463233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b7850 msgr2=0x55f5db8b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.636+0000 7f99f4acd640 1 --2- 172.21.15.33:0/1003463233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b7850 0x55f5db8b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.636+0000 7f99f4acd640 1 -- 172.21.15.33:0/1003463233 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5db8c1880 con 0x55f5db8b8170 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.636+0000 7f99f52ce640 1 --2- 172.21.15.33:0/1003463233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b7850 0x55f5db8b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.636+0000 7f99f4acd640 1 --2- 172.21.15.33:0/1003463233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b8170 0x55f5db8c0210 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f99e402a9e0 tx=0x7f99e4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c85e803e498944 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.636+0000 7f99d6ffd640 1 -- 172.21.15.33:0/1003463233 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e4035040 con 0x55f5db8b8170 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.636+0000 7f99d6ffd640 1 -- 172.21.15.33:0/1003463233 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e404b070 con 0x55f5db8b8170 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.636+0000 7f99d6ffd640 1 -- 172.21.15.33:0/1003463233 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e40315b0 con 0x55f5db8b8170 2024-09-16T00:28:53.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.637+0000 7f99f990b740 1 -- 172.21.15.33:0/1003463233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b8170 msgr2=0x55f5db8c0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.637+0000 7f99f990b740 1 --2- 172.21.15.33:0/1003463233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b8170 0x55f5db8c0210 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f99e402a9e0 tx=0x7f99e4000bc0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.637+0000 7f99f990b740 1 -- 172.21.15.33:0/1003463233 shutdown_connections 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.637+0000 7f99f990b740 1 --2- 172.21.15.33:0/1003463233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8c2a30 0x55f5db8b9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.637+0000 7f99f990b740 1 --2- 172.21.15.33:0/1003463233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8b8170 0x55f5db8c0210 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.637+0000 7f99f990b740 1 --2- 172.21.15.33:0/1003463233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b7850 0x55f5db8b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.637+0000 7f99f990b740 1 -- 172.21.15.33:0/1003463233 >> 172.21.15.33:0/1003463233 conn(0x55f5db8b3de0 msgr2=0x55f5db8b92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.637+0000 7f99f990b740 1 -- 172.21.15.33:0/1003463233 shutdown_connections 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.637+0000 7f99f990b740 1 -- 172.21.15.33:0/1003463233 wait complete. 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f990b740 1 -- start start 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f990b740 1 Processor -- start 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f990b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b7850 0x55f5db8caf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f990b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 0x55f5db8c62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f990b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8c2a30 0x55f5db8c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f990b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01b3dd0 con 0x55f5db8c2a30 2024-09-16T00:28:53.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f990b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01b5240 con 0x55f5db8b7850 2024-09-16T00:28:53.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f990b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01b44a0 con 0x55f5db8b8170 2024-09-16T00:28:53.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f990b740 10 client.0 _ll_register_callbacks cb 0x7f99e000ef30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:28:53.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f5acf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8c2a30 0x55f5db8c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f4acd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 0x55f5db8c62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f4acd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 0x55f5db8c62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:48594/0 (socket says 172.21.15.33:48594) 2024-09-16T00:28:53.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.638+0000 7f99f4acd640 1 -- 172.21.15.33:0/413599861 learned_addr learned my addr 172.21.15.33:0/413599861 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:28:53.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99f52ce640 1 --2- 172.21.15.33:0/413599861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b7850 0x55f5db8caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99f4acd640 1 -- 172.21.15.33:0/413599861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b7850 msgr2=0x55f5db8caf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:53.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99f4acd640 1 --2- 172.21.15.33:0/413599861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b7850 0x55f5db8caf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99f4acd640 1 -- 172.21.15.33:0/413599861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8c2a30 msgr2=0x55f5db8c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:28:53.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99f4acd640 1 --2- 172.21.15.33:0/413599861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8c2a30 0x55f5db8c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99f4acd640 1 -- 172.21.15.33:0/413599861 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5db8b7fa0 con 0x55f5db8b8170 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99f4acd640 1 --2- 172.21.15.33:0/413599861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 0x55f5db8c62a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009850 tx=0x7f99e4005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8a3023bc9472a85 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99d4ff9640 1 -- 172.21.15.33:0/413599861 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e4035040 con 0x55f5db8b8170 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99f990b740 1 -- 172.21.15.33:0/413599861 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99e404b070 con 0x55f5db8b8170 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99d4ff9640 1 -- 172.21.15.33:0/413599861 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f99e40048c0 con 0x55f5db8b8170 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99d4ff9640 1 -- 172.21.15.33:0/413599861 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f99e4005040 con 0x55f5db8b8170 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99f990b740 10 client.27019 fetch_fsmap learned FSMap version 30 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.639+0000 7f99f990b740 1 -- 172.21.15.33:0/413599861 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f5db93d110 con 0x55f5db8b8170 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.640+0000 7f99d4ff9640 1 -- 172.21.15.33:0/413599861 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e404b070 con 0x55f5db8b8170 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.640+0000 7f99d4ff9640 1 -- 172.21.15.33:0/413599861 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f99e4031c90 con 0x55f5db8b8170 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.640+0000 7f99f990b740 10 client.27019 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:28:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.640+0000 7f99f990b740 10 client.27019 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:28:53.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.640+0000 7f99f990b740 20 client.27019 populate_metadata read hostname 'smithi033' 2024-09-16T00:28:53.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.640+0000 7f99f990b740 1 --2- 172.21.15.33:0/413599861 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f5db93d850 0x55f5db93fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:28:53.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.640+0000 7f99f990b740 4 client.27019 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}] 2024-09-16T00:28:53.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.640+0000 7f99f990b740 1 -- 172.21.15.33:0/413599861 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}) -- 0x55f5db823be0 con 0x55f5db93d850 2024-09-16T00:28:53.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.640+0000 7f99f52ce640 1 --2- 172.21.15.33:0/413599861 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f5db93d850 0x55f5db93fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:28:53.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.640+0000 7f99d4ff9640 10 client.27019 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:53.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.640+0000 7f99f52ce640 1 --2- 172.21.15.33:0/413599861 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f5db93d850 0x55f5db93fc70 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8389fd6aa29312c1 server_cookie=7025d0a444b43855 in_seq=0 out_seq=0 2024-09-16T00:28:53.683 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994896 9.89 MB/sec execute 481 sec latency 45.766 ms 2024-09-16T00:28:53.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:53 smithi060 ceph-mon[35185]: pgmap v702: 129 pgs: 129 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 3.0 MiB/s wr, 90 op/s 2024-09-16T00:28:53.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.739+0000 7f99d4ff9640 1 -- 172.21.15.33:0/413599861 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f5db823be0 con 0x55f5db93d850 2024-09-16T00:28:53.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.739+0000 7f99d4ff9640 10 client.27019 handle_command_reply: tid=0 2024-09-16T00:28:53.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.739+0000 7f99f990b740 1 client.27019 shutdown 2024-09-16T00:28:53.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.739+0000 7f99f990b740 1 -- 172.21.15.33:0/413599861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 msgr2=0x55f5db8c62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:53.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.739+0000 7f99f990b740 1 --2- 172.21.15.33:0/413599861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 0x55f5db8c62a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009850 tx=0x7f99e4005fd0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.739+0000 7f99f990b740 1 -- 172.21.15.33:0/413599861 shutdown_connections 2024-09-16T00:28:53.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.739+0000 7f99f990b740 1 --2- 172.21.15.33:0/413599861 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f5db93d850 0x55f5db93fc70 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f990b740 1 --2- 172.21.15.33:0/413599861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5db8c2a30 0x55f5db8c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99d4ff9640 0 client.27019 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:28:53.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f990b740 1 --2- 172.21.15.33:0/413599861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f5db8b8170 0x55f5db8c62a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f990b740 1 --2- 172.21.15.33:0/413599861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f5db8b7850 0x55f5db8caf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f990b740 1 -- 172.21.15.33:0/413599861 >> 172.21.15.33:0/413599861 conn(0x55f5db8b3de0 msgr2=0x55f5db8b9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:53.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f990b740 1 -- 172.21.15.33:0/413599861 shutdown_connections 2024-09-16T00:28:53.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f990b740 1 -- 172.21.15.33:0/413599861 wait complete. 2024-09-16T00:28:53.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f990b740 20 client.27019 trim_cache size 0 max 16384 2024-09-16T00:28:53.742 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:28:53.742 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:28:53.742 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:28:53.742 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 42, 2024-09-16T00:28:53.742 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: "26e2c7d4": { 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 42, 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 141.8, 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:28:53.743 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:28:53.744 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:28:53.744 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:28:53.744 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:53.744 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:53.744 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:53.744 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:53.744 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:28:53.744 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:28:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f7532640 1 -- 172.21.15.33:0/2013295909 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f99d80a3240 msgr2=0x7f99d80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f7532640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f99d80a3240 0x7f99d80a5700 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f99e8023000 tx=0x7f99e8024000 comp rx=0 tx=0).stop 2024-09-16T00:28:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f7532640 1 -- 172.21.15.33:0/2013295909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99f0100a60 msgr2=0x7f99f01a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:28:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f7532640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99f0100a60 0x7f99f01a46c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99e000ebc0 tx=0x7f99e000c6a0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f7532640 1 -- 172.21.15.33:0/2013295909 shutdown_connections 2024-09-16T00:28:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f7532640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f99d80a3240 0x7f99d80a5700 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f7532640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f0106a60 0x7f99f019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f7532640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0101400 0x7f99f01a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f7532640 1 --2- 172.21.15.33:0/2013295909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99f0100a60 0x7f99f01a46c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:28:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.740+0000 7f99f7532640 1 -- 172.21.15.33:0/2013295909 >> 172.21.15.33:0/2013295909 conn(0x7f99f00fc780 msgr2=0x7f99f006a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:28:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.741+0000 7f99f7532640 1 -- 172.21.15.33:0/2013295909 shutdown_connections 2024-09-16T00:28:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:28:53.741+0000 7f99f7532640 1 -- 172.21.15.33:0/2013295909 wait complete. 2024-09-16T00:28:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:53 smithi083 ceph-mon[35242]: pgmap v702: 129 pgs: 129 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 3.0 MiB/s wr, 90 op/s 2024-09-16T00:28:54.582 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '26e2c7d4', quiesce took 3.5 seconds. Will release after: 18.35 2024-09-16T00:28:54.683 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.87 MB/sec execute 482 sec latency 954.189 ms 2024-09-16T00:28:55.683 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.85 MB/sec execute 483 sec latency 1954.304 ms 2024-09-16T00:28:55.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:55 smithi060 ceph-mon[35185]: pgmap v703: 129 pgs: 129 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 2.5 MiB/s wr, 120 op/s 2024-09-16T00:28:55.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:55 smithi033 ceph-mon[31243]: pgmap v703: 129 pgs: 129 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 2.5 MiB/s wr, 120 op/s 2024-09-16T00:28:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:55 smithi083 ceph-mon[35242]: pgmap v703: 129 pgs: 129 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 2.5 MiB/s wr, 120 op/s 2024-09-16T00:28:56.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.83 MB/sec execute 484 sec latency 2954.410 ms 2024-09-16T00:28:57.683 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.81 MB/sec execute 485 sec latency 3954.451 ms 2024-09-16T00:28:57.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:57 smithi083 ceph-mon[35242]: pgmap v704: 129 pgs: 129 active+clean; 271 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 123 op/s 2024-09-16T00:28:57.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:57 smithi060 ceph-mon[35185]: pgmap v704: 129 pgs: 129 active+clean; 271 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 123 op/s 2024-09-16T00:28:57.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:57 smithi033 ceph-mon[31243]: pgmap v704: 129 pgs: 129 active+clean; 271 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 123 op/s 2024-09-16T00:28:58.683 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.79 MB/sec execute 486 sec latency 4954.544 ms 2024-09-16T00:28:59.683 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.77 MB/sec execute 487 sec latency 5954.648 ms 2024-09-16T00:28:59.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:28:59 smithi060 ceph-mon[35185]: pgmap v705: 129 pgs: 129 active+clean; 273 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 133 op/s 2024-09-16T00:28:59.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:28:59 smithi033 ceph-mon[31243]: pgmap v705: 129 pgs: 129 active+clean; 273 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 133 op/s 2024-09-16T00:28:59.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:28:59 smithi083 ceph-mon[35242]: pgmap v705: 129 pgs: 129 active+clean; 273 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 133 op/s 2024-09-16T00:29:00.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.75 MB/sec execute 488 sec latency 6954.769 ms 2024-09-16T00:29:01.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.73 MB/sec execute 489 sec latency 7954.844 ms 2024-09-16T00:29:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:01 smithi060 ceph-mon[35185]: pgmap v706: 129 pgs: 129 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 144 op/s 2024-09-16T00:29:01.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:01 smithi033 ceph-mon[31243]: pgmap v706: 129 pgs: 129 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 144 op/s 2024-09-16T00:29:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:01 smithi083 ceph-mon[35242]: pgmap v706: 129 pgs: 129 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 144 op/s 2024-09-16T00:29:02.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.71 MB/sec execute 490 sec latency 8954.920 ms 2024-09-16T00:29:03.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.69 MB/sec execute 491 sec latency 9955.024 ms 2024-09-16T00:29:03.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:03 smithi060 ceph-mon[35185]: pgmap v707: 129 pgs: 129 active+clean; 271 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 150 op/s 2024-09-16T00:29:03.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:03 smithi033 ceph-mon[31243]: pgmap v707: 129 pgs: 129 active+clean; 271 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 150 op/s 2024-09-16T00:29:03.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:03 smithi083 ceph-mon[35242]: pgmap v707: 129 pgs: 129 active+clean; 271 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 150 op/s 2024-09-16T00:29:04.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.67 MB/sec execute 492 sec latency 10955.149 ms 2024-09-16T00:29:05.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.65 MB/sec execute 493 sec latency 11955.236 ms 2024-09-16T00:29:05.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:05 smithi060 ceph-mon[35185]: pgmap v708: 129 pgs: 129 active+clean; 271 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 140 op/s 2024-09-16T00:29:05.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:05 smithi033 ceph-mon[31243]: pgmap v708: 129 pgs: 129 active+clean; 271 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 140 op/s 2024-09-16T00:29:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:05 smithi083 ceph-mon[35242]: pgmap v708: 129 pgs: 129 active+clean; 271 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 140 op/s 2024-09-16T00:29:06.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.63 MB/sec execute 494 sec latency 12955.326 ms 2024-09-16T00:29:07.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.61 MB/sec execute 495 sec latency 13955.387 ms 2024-09-16T00:29:07.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:07 smithi060 ceph-mon[35185]: pgmap v709: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 95 op/s 2024-09-16T00:29:07.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:07 smithi033 ceph-mon[31243]: pgmap v709: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 95 op/s 2024-09-16T00:29:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:07 smithi083 ceph-mon[35242]: pgmap v709: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 95 op/s 2024-09-16T00:29:08.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.59 MB/sec execute 496 sec latency 14955.509 ms 2024-09-16T00:29:09.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.58 MB/sec execute 497 sec latency 15955.610 ms 2024-09-16T00:29:09.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:09 smithi060 ceph-mon[35185]: pgmap v710: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 1.1 MiB/s wr, 74 op/s 2024-09-16T00:29:09.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:09 smithi033 ceph-mon[31243]: pgmap v710: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 1.1 MiB/s wr, 74 op/s 2024-09-16T00:29:09.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:09 smithi083 ceph-mon[35242]: pgmap v710: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 1.1 MiB/s wr, 74 op/s 2024-09-16T00:29:10.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.56 MB/sec execute 498 sec latency 16955.713 ms 2024-09-16T00:29:11.684 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.54 MB/sec execute 499 sec latency 17955.834 ms 2024-09-16T00:29:11.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:11 smithi060 ceph-mon[35185]: pgmap v711: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 922 KiB/s wr, 48 op/s 2024-09-16T00:29:11.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:11 smithi033 ceph-mon[31243]: pgmap v711: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 922 KiB/s wr, 48 op/s 2024-09-16T00:29:11.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:11 smithi083 ceph-mon[35242]: pgmap v711: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 922 KiB/s wr, 48 op/s 2024-09-16T00:29:12.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.52 MB/sec execute 500 sec latency 18955.925 ms 2024-09-16T00:29:12.934 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:29:13.205 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:29:13.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:13 smithi033 ceph-mon[31243]: pgmap v712: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 722 KiB/s wr, 34 op/s 2024-09-16T00:29:13.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.50 MB/sec execute 501 sec latency 19955.970 ms 2024-09-16T00:29:13.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:13 smithi060 ceph-mon[35185]: pgmap v712: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 722 KiB/s wr, 34 op/s 2024-09-16T00:29:13.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.689+0000 7f69ea343640 1 -- 172.21.15.33:0/246638538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69e4100a80 msgr2=0x7f69e4100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:13.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.689+0000 7f69ea343640 1 --2- 172.21.15.33:0/246638538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69e4100a80 0x7f69e4100ee0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f69cc009930 tx=0x7f69cc02a420 comp rx=0 tx=0).stop 2024-09-16T00:29:13.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.689+0000 7f69ea343640 1 -- 172.21.15.33:0/246638538 shutdown_connections 2024-09-16T00:29:13.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.689+0000 7f69ea343640 1 --2- 172.21.15.33:0/246638538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e4101420 0x7f69e410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:13.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.689+0000 7f69ea343640 1 --2- 172.21.15.33:0/246638538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69e4100a80 0x7f69e4100ee0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:13.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.689+0000 7f69ea343640 1 --2- 172.21.15.33:0/246638538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4106a80 0x7f69e4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:13.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.689+0000 7f69ea343640 1 -- 172.21.15.33:0/246638538 >> 172.21.15.33:0/246638538 conn(0x7f69e40fc7a0 msgr2=0x7f69e40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:13.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.689+0000 7f69ea343640 1 -- 172.21.15.33:0/246638538 shutdown_connections 2024-09-16T00:29:13.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.689+0000 7f69ea343640 1 -- 172.21.15.33:0/246638538 wait complete. 2024-09-16T00:29:13.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69ea343640 1 Processor -- start 2024-09-16T00:29:13.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69ea343640 1 -- start start 2024-09-16T00:29:13.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69ea343640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69e4100a80 0x7f69e40fce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:13.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69ea343640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e4101420 0x7f69e40fd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:13.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69ea343640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4106a80 0x7f69e40fd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:13.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69ea343640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f69e4110e80 con 0x7f69e4100a80 2024-09-16T00:29:13.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69ea343640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f69e4110d00 con 0x7f69e4101420 2024-09-16T00:29:13.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69ea343640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f69e4111000 con 0x7f69e4106a80 2024-09-16T00:29:13.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69e3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69e4100a80 0x7f69e40fce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:13.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69e3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69e4100a80 0x7f69e40fce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46366/0 (socket says 172.21.15.33:46366) 2024-09-16T00:29:13.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69e3fff640 1 -- 172.21.15.33:0/843375751 learned_addr learned my addr 172.21.15.33:0/843375751 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:13.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69e88e0640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4106a80 0x7f69e40fd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:13.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69dbfff640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e4101420 0x7f69e40fd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:13.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69e3fff640 1 -- 172.21.15.33:0/843375751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4106a80 msgr2=0x7f69e40fd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:13.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69e3fff640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4106a80 0x7f69e40fd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:13.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69e3fff640 1 -- 172.21.15.33:0/843375751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e4101420 msgr2=0x7f69e40fd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:13.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69e3fff640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e4101420 0x7f69e40fd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:13.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69e3fff640 1 -- 172.21.15.33:0/843375751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e41a6950 con 0x7f69e4100a80 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.690+0000 7f69dbfff640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e4101420 0x7f69e40fd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.691+0000 7f69e3fff640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69e4100a80 0x7f69e40fce30 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f69d400d950 tx=0x7f69d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.691+0000 7f69e1ffb640 1 -- 172.21.15.33:0/843375751 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d4017030 con 0x7f69e4100a80 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.691+0000 7f69ea343640 1 -- 172.21.15.33:0/843375751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69e41a6c40 con 0x7f69e4100a80 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.691+0000 7f69e1ffb640 1 -- 172.21.15.33:0/843375751 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69d4004020 con 0x7f69e4100a80 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.691+0000 7f69e1ffb640 1 -- 172.21.15.33:0/843375751 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d40047a0 con 0x7f69e4100a80 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.691+0000 7f69ea343640 1 -- 172.21.15.33:0/843375751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69e41a7180 con 0x7f69e4100a80 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.692+0000 7f69ea343640 1 -- 172.21.15.33:0/843375751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69e4100ee0 con 0x7f69e4100a80 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.693+0000 7f69e1ffb640 1 -- 172.21.15.33:0/843375751 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f69d40041d0 con 0x7f69e4100a80 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.694+0000 7f69e1ffb640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f69d00a3240 0x7f69d00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.694+0000 7f69dbfff640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f69d00a3240 0x7f69d00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.694+0000 7f69e1ffb640 1 -- 172.21.15.33:0/843375751 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f69d40cf6e0 con 0x7f69e4100a80 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.694+0000 7f69dbfff640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f69d00a3240 0x7f69d00a5700 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f69cc000c00 tx=0x7f69cc032c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:13.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.698+0000 7f69e1ffb640 1 -- 172.21.15.33:0/843375751 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f69d409bf80 con 0x7f69e4100a80 2024-09-16T00:29:13.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.910+0000 7f69ea343640 1 -- 172.21.15.33:0/843375751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69e40fe360 con 0x7f69e4100a80 2024-09-16T00:29:13.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.910+0000 7f69e1ffb640 1 -- 172.21.15.33:0/843375751 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f69d409b880 con 0x7f69e4100a80 2024-09-16T00:29:13.912 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:29:13.912 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:29:13.912 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:29:13.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.913+0000 7f69ea343640 1 -- 172.21.15.33:0/843375751 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f69d00a3240 msgr2=0x7f69d00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:13.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.913+0000 7f69ea343640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f69d00a3240 0x7f69d00a5700 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f69cc000c00 tx=0x7f69cc032c70 comp rx=0 tx=0).stop 2024-09-16T00:29:13.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.913+0000 7f69ea343640 1 -- 172.21.15.33:0/843375751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69e4100a80 msgr2=0x7f69e40fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:13.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.913+0000 7f69ea343640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69e4100a80 0x7f69e40fce30 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f69d400d950 tx=0x7f69d400de10 comp rx=0 tx=0).stop 2024-09-16T00:29:13.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.913+0000 7f69ea343640 1 -- 172.21.15.33:0/843375751 shutdown_connections 2024-09-16T00:29:13.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.913+0000 7f69ea343640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f69d00a3240 0x7f69d00a5700 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:13.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.913+0000 7f69ea343640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4106a80 0x7f69e40fd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:13.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.913+0000 7f69ea343640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e4101420 0x7f69e40fd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:13.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.913+0000 7f69ea343640 1 --2- 172.21.15.33:0/843375751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69e4100a80 0x7f69e40fce30 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:13.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.913+0000 7f69ea343640 1 -- 172.21.15.33:0/843375751 >> 172.21.15.33:0/843375751 conn(0x7f69e40fc7a0 msgr2=0x7f69e406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:13.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.913+0000 7f69ea343640 1 -- 172.21.15.33:0/843375751 shutdown_connections 2024-09-16T00:29:13.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:13.913+0000 7f69ea343640 1 -- 172.21.15.33:0/843375751 wait complete. 2024-09-16T00:29:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:13 smithi083 ceph-mon[35242]: pgmap v712: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 722 KiB/s wr, 34 op/s 2024-09-16T00:29:14.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 994897 9.48 MB/sec execute 502 sec latency 20956.075 ms 2024-09-16T00:29:14.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:14 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/843375751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:29:14.767 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:14 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/843375751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:29:14.803 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 26e2c7d4 --release --await' 2024-09-16T00:29:14.803 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 26e2c7d4 --release --await 2024-09-16T00:29:14.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:14 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/843375751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:29:15.065 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:29:15.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.562+0000 7f26e7300640 1 -- 172.21.15.33:0/183171134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e0106f10 msgr2=0x7f26e010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:15.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.562+0000 7f26e7300640 1 --2- 172.21.15.33:0/183171134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e0106f10 0x7f26e010ea90 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f26d8009ca0 tx=0x7f26d802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.562+0000 7f26e7300640 1 -- 172.21.15.33:0/183171134 shutdown_connections 2024-09-16T00:29:15.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.562+0000 7f26e7300640 1 --2- 172.21.15.33:0/183171134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e0106f10 0x7f26e010ea90 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.562+0000 7f26e7300640 1 --2- 172.21.15.33:0/183171134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e0102790 0x7f26e01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.562+0000 7f26e7300640 1 --2- 172.21.15.33:0/183171134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e0101de0 0x7f26e01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.562+0000 7f26e7300640 1 -- 172.21.15.33:0/183171134 >> 172.21.15.33:0/183171134 conn(0x7f26e00fd8c0 msgr2=0x7f26e00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:15.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.562+0000 7f26e7300640 1 -- 172.21.15.33:0/183171134 shutdown_connections 2024-09-16T00:29:15.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.562+0000 7f26e7300640 1 -- 172.21.15.33:0/183171134 wait complete. 2024-09-16T00:29:15.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.563+0000 7f26e7300640 1 Processor -- start 2024-09-16T00:29:15.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.563+0000 7f26e7300640 1 -- start start 2024-09-16T00:29:15.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.563+0000 7f26e7300640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e0101de0 0x7f26e0198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.563+0000 7f26e7300640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e0102790 0x7f26e0198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.563+0000 7f26e7300640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e0106f10 0x7f26e019cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.563+0000 7f26e7300640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26e0111f70 con 0x7f26e0106f10 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.563+0000 7f26e7300640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f26e0111df0 con 0x7f26e0102790 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.563+0000 7f26e7300640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f26e01120f0 con 0x7f26e0101de0 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.563+0000 7f26e589d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e0106f10 0x7f26e019cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.563+0000 7f26e589d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e0106f10 0x7f26e019cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46394/0 (socket says 172.21.15.33:46394) 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.563+0000 7f26e589d640 1 -- 172.21.15.33:0/2077248669 learned_addr learned my addr 172.21.15.33:0/2077248669 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.563+0000 7f26e509c640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e0101de0 0x7f26e0198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26e489b640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e0102790 0x7f26e0198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26e589d640 1 -- 172.21.15.33:0/2077248669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e0101de0 msgr2=0x7f26e0198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26e589d640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e0101de0 0x7f26e0198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26e589d640 1 -- 172.21.15.33:0/2077248669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e0102790 msgr2=0x7f26e0198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26e589d640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e0102790 0x7f26e0198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26e589d640 1 -- 172.21.15.33:0/2077248669 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e019d300 con 0x7f26e0106f10 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26e489b640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e0102790 0x7f26e0198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:29:15.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26e589d640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e0106f10 0x7f26e019cc20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f26d8031a20 tx=0x7f26d8031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26d67fc640 1 -- 172.21.15.33:0/2077248669 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d8003c30 con 0x7f26e0106f10 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26e7300640 1 -- 172.21.15.33:0/2077248669 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26e019d590 con 0x7f26e0106f10 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26d67fc640 1 -- 172.21.15.33:0/2077248669 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26d8003dd0 con 0x7f26e0106f10 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26d67fc640 1 -- 172.21.15.33:0/2077248669 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d802fcf0 con 0x7f26e0106f10 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.564+0000 7f26e7300640 1 -- 172.21.15.33:0/2077248669 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26e01a4ed0 con 0x7f26e0106f10 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.566+0000 7f26d67fc640 1 -- 172.21.15.33:0/2077248669 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f26d8004050 con 0x7f26e0106f10 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e96d9740 1 Processor -- start 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e96d9740 1 -- start start 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e96d9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45ef40 0x55c9af45f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e96d9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af45f860 0x55c9af46e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e96d9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af46e650 0x55c9af470a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e96d9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c9af3c4620 con 0x55c9af45f860 2024-09-16T00:29:15.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e96d9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c9af36be50 con 0x55c9af45ef40 2024-09-16T00:29:15.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e96d9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c9af3c05e0 con 0x55c9af46e650 2024-09-16T00:29:15.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e509c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af45f860 0x55c9af46e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e509c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af45f860 0x55c9af46e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46402/0 (socket says 172.21.15.33:46402) 2024-09-16T00:29:15.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e509c640 1 -- 172.21.15.33:0/2186210855 learned_addr learned my addr 172.21.15.33:0/2186210855 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:15.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e489b640 1 --2- 172.21.15.33:0/2186210855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af46e650 0x55c9af470a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e489b640 1 --2- 172.21.15.33:0/2186210855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45ef40 0x55c9af45f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e509c640 1 -- 172.21.15.33:0/2186210855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af46e650 msgr2=0x55c9af470a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:15.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26d67fc640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26cc0a3240 0x7f26cc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e509c640 1 --2- 172.21.15.33:0/2186210855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af46e650 0x55c9af470a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.567+0000 7f26e509c640 1 -- 172.21.15.33:0/2186210855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45ef40 msgr2=0x55c9af45f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:15.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.568+0000 7f26e509c640 1 --2- 172.21.15.33:0/2186210855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45ef40 0x55c9af45f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.568+0000 7f26e509c640 1 -- 172.21.15.33:0/2186210855 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9af35cd40 con 0x55c9af45f860 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.568+0000 7f26e489b640 1 --2- 172.21.15.33:0/2186210855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af46e650 0x55c9af470a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.568+0000 7f26e589d640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26cc0a3240 0x7f26cc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.568+0000 7f26e509c640 1 --2- 172.21.15.33:0/2186210855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af45f860 0x55c9af46e110 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f26dc00bcd0 tx=0x7f26dc0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd23ab5a8d3f014 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.568+0000 7f26d67fc640 1 -- 172.21.15.33:0/2077248669 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f26d80f1340 con 0x7f26e0106f10 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.568+0000 7f26cb7fe640 1 -- 172.21.15.33:0/2186210855 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc005220 con 0x55c9af45f860 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.568+0000 7f26cb7fe640 1 -- 172.21.15.33:0/2186210855 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26dc004c80 con 0x55c9af45f860 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.568+0000 7f26cb7fe640 1 -- 172.21.15.33:0/2186210855 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc011780 con 0x55c9af45f860 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.569+0000 7f26e96d9740 1 -- 172.21.15.33:0/2186210855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af45f860 msgr2=0x55c9af46e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.569+0000 7f26e96d9740 1 --2- 172.21.15.33:0/2186210855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af45f860 0x55c9af46e110 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f26dc00bcd0 tx=0x7f26dc0107a0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.569+0000 7f26e589d640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26cc0a3240 0x7f26cc0a5700 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f26d80b9ee0 tx=0x7f26d80e8250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.569+0000 7f26e96d9740 1 -- 172.21.15.33:0/2186210855 shutdown_connections 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.569+0000 7f26e96d9740 1 --2- 172.21.15.33:0/2186210855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af46e650 0x55c9af470a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.569+0000 7f26e96d9740 1 --2- 172.21.15.33:0/2186210855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af45f860 0x55c9af46e110 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.569+0000 7f26e96d9740 1 --2- 172.21.15.33:0/2186210855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45ef40 0x55c9af45f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.569+0000 7f26e96d9740 1 -- 172.21.15.33:0/2186210855 >> 172.21.15.33:0/2186210855 conn(0x55c9af45b5f0 msgr2=0x55c9af45b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.569+0000 7f26e96d9740 1 -- 172.21.15.33:0/2186210855 shutdown_connections 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.569+0000 7f26e96d9740 1 -- 172.21.15.33:0/2186210855 wait complete. 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.569+0000 7f26e96d9740 1 -- start start 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.569+0000 7f26e96d9740 1 Processor -- start 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e96d9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45ef40 0x55c9af46b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e96d9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45f860 0x55c9af46baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e96d9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af46c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e96d9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c9af471220 con 0x55c9af46e650 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e96d9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c9af34cf10 con 0x55c9af45f860 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e96d9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c9af4713a0 con 0x55c9af45ef40 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e96d9740 10 client.0 _ll_register_callbacks cb 0x55c9af370db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:15.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e489b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af46c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e489b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af46c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46406/0 (socket says 172.21.15.33:46406) 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e489b640 1 -- 172.21.15.33:0/4176688491 learned_addr learned my addr 172.21.15.33:0/4176688491 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e509c640 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45f860 0x55c9af46baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e489b640 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45ef40 0x55c9af46b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e489b640 1 -- 172.21.15.33:0/4176688491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45ef40 msgr2=0x55c9af46b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e489b640 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45ef40 0x55c9af46b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e489b640 1 -- 172.21.15.33:0/4176688491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45f860 msgr2=0x55c9af46baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e489b640 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45f860 0x55c9af46baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e489b640 1 -- 172.21.15.33:0/4176688491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9af0d3580 con 0x55c9af46e650 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26e489b640 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af46c030 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f26d0011ef0 tx=0x7f26d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce2d646dd386b5c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.570+0000 7f26c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26c8ff9640 1 -- 172.21.15.33:0/4176688491 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d0010430 con 0x55c9af46e650 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26e96d9740 1 -- 172.21.15.33:0/4176688491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26dc004c80 con 0x55c9af46e650 2024-09-16T00:29:15.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26c8ff9640 1 -- 172.21.15.33:0/4176688491 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f26d00105d0 con 0x55c9af46e650 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26c8ff9640 1 -- 172.21.15.33:0/4176688491 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f26d0026050 con 0x55c9af46e650 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26e96d9740 10 client.17661 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26e96d9740 1 -- 172.21.15.33:0/4176688491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c9af19d960 con 0x55c9af46e650 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26c8ff9640 1 -- 172.21.15.33:0/4176688491 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26dc004c80 con 0x55c9af46e650 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26c8ff9640 1 -- 172.21.15.33:0/4176688491 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f26d000f760 con 0x55c9af46e650 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26e96d9740 10 client.17661 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26e96d9740 10 client.17661 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26e96d9740 20 client.17661 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26e96d9740 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c9af47e1d0 0x55c9af4805e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26e96d9740 4 client.17661 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.571+0000 7f26e96d9740 1 -- 172.21.15.33:0/4176688491 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9af3d1b90 con 0x55c9af47e1d0 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.572+0000 7f26e509c640 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c9af47e1d0 0x55c9af4805e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.572+0000 7f26e509c640 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c9af47e1d0 0x55c9af4805e0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f0200d89d03bb99 server_cookie=545c5b8ba58c35f4 in_seq=0 out_seq=0 2024-09-16T00:29:15.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.572+0000 7f26c8ff9640 10 client.17661 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26c8ff9640 1 -- 172.21.15.33:0/4176688491 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c9af3d1b90 con 0x55c9af47e1d0 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26c8ff9640 10 client.17661 handle_command_reply: tid=0 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26e96d9740 1 client.17661 shutdown 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26e96d9740 1 -- 172.21.15.33:0/4176688491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 msgr2=0x55c9af46c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26e96d9740 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af46c030 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f26d0011ef0 tx=0x7f26d00046c0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26e96d9740 1 -- 172.21.15.33:0/4176688491 shutdown_connections 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26e96d9740 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c9af47e1d0 0x55c9af4805e0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26e96d9740 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af46c030 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26e96d9740 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45f860 0x55c9af46baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26e96d9740 1 --2- 172.21.15.33:0/4176688491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45ef40 0x55c9af46b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26e96d9740 1 -- 172.21.15.33:0/4176688491 >> 172.21.15.33:0/4176688491 conn(0x55c9af45b5f0 msgr2=0x55c9af461350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26c8ff9640 0 client.17661 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:15.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26e96d9740 1 -- 172.21.15.33:0/4176688491 shutdown_connections 2024-09-16T00:29:15.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.573+0000 7f26e96d9740 1 -- 172.21.15.33:0/4176688491 wait complete. 2024-09-16T00:29:15.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.574+0000 7f26e96d9740 20 client.17661 trim_cache size 0 max 16384 2024-09-16T00:29:15.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e96d9740 1 Processor -- start 2024-09-16T00:29:15.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e96d9740 1 -- start start 2024-09-16T00:29:15.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e96d9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45ef40 0x55c9af45a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e96d9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45f860 0x55c9af485eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e96d9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af4863f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e96d9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26e01af6d0 con 0x55c9af46e650 2024-09-16T00:29:15.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e96d9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f26e01acfa0 con 0x55c9af45f860 2024-09-16T00:29:15.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e96d9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f26e01aadf0 con 0x55c9af45ef40 2024-09-16T00:29:15.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e489b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af4863f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e489b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af4863f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46414/0 (socket says 172.21.15.33:46414) 2024-09-16T00:29:15.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e489b640 1 -- 172.21.15.33:0/1371562577 learned_addr learned my addr 172.21.15.33:0/1371562577 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:15.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e489b640 1 --2- 172.21.15.33:0/1371562577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45ef40 0x55c9af45a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e509c640 1 --2- 172.21.15.33:0/1371562577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45f860 0x55c9af485eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e489b640 1 -- 172.21.15.33:0/1371562577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45ef40 msgr2=0x55c9af45a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:15.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e489b640 1 --2- 172.21.15.33:0/1371562577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45ef40 0x55c9af45a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e489b640 1 -- 172.21.15.33:0/1371562577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45f860 msgr2=0x55c9af485eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:15.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e489b640 1 --2- 172.21.15.33:0/1371562577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45f860 0x55c9af485eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e489b640 1 -- 172.21.15.33:0/1371562577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9af45b420 con 0x55c9af46e650 2024-09-16T00:29:15.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e509c640 1 --2- 172.21.15.33:0/1371562577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45f860 0x55c9af485eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:29:15.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.591+0000 7f26e489b640 1 --2- 172.21.15.33:0/1371562577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af4863f0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b420 tx=0x7f26d0005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d76054d08fbdd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:15.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.592+0000 7f26caffd640 1 -- 172.21.15.33:0/1371562577 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d000bbd0 con 0x55c9af46e650 2024-09-16T00:29:15.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.592+0000 7f26caffd640 1 -- 172.21.15.33:0/1371562577 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26d000f8a0 con 0x55c9af46e650 2024-09-16T00:29:15.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.592+0000 7f26caffd640 1 -- 172.21.15.33:0/1371562577 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d000da00 con 0x55c9af46e650 2024-09-16T00:29:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.592+0000 7f26e96d9740 1 -- 172.21.15.33:0/1371562577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 msgr2=0x55c9af4863f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.592+0000 7f26e96d9740 1 --2- 172.21.15.33:0/1371562577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af4863f0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b420 tx=0x7f26d0005da0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.592+0000 7f26e96d9740 1 -- 172.21.15.33:0/1371562577 shutdown_connections 2024-09-16T00:29:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.592+0000 7f26e96d9740 1 --2- 172.21.15.33:0/1371562577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af4863f0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.592+0000 7f26e96d9740 1 --2- 172.21.15.33:0/1371562577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45f860 0x55c9af485eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.592+0000 7f26e96d9740 1 --2- 172.21.15.33:0/1371562577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45ef40 0x55c9af45a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.592+0000 7f26e96d9740 1 -- 172.21.15.33:0/1371562577 >> 172.21.15.33:0/1371562577 conn(0x55c9af45b5f0 msgr2=0x55c9af459f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.592+0000 7f26e96d9740 1 -- 172.21.15.33:0/1371562577 shutdown_connections 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.592+0000 7f26e96d9740 1 -- 172.21.15.33:0/1371562577 wait complete. 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e96d9740 1 -- start start 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e96d9740 1 Processor -- start 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e96d9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45ef40 0x55c9af472a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e96d9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45f860 0x55c9af472f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e96d9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af4e7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e96d9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26e01ab460 con 0x55c9af46e650 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e96d9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f26e01ac8d0 con 0x55c9af45ef40 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e96d9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f26e01abb30 con 0x55c9af45f860 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e96d9740 10 client.0 _ll_register_callbacks cb 0x7f26d0011b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e489b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af4e7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e489b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af4e7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46424/0 (socket says 172.21.15.33:46424) 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e489b640 1 -- 172.21.15.33:0/2419588004 learned_addr learned my addr 172.21.15.33:0/2419588004 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e509c640 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45f860 0x55c9af472f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.593+0000 7f26e489b640 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45ef40 0x55c9af472a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26e489b640 1 -- 172.21.15.33:0/2419588004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45f860 msgr2=0x55c9af472f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26e489b640 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45f860 0x55c9af472f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26e489b640 1 -- 172.21.15.33:0/2419588004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45ef40 msgr2=0x55c9af472a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26e489b640 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45ef40 0x55c9af472a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26e489b640 1 -- 172.21.15.33:0/2419588004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9af4e83c0 con 0x55c9af46e650 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26e509c640 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45f860 0x55c9af472f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26e489b640 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af4e7e80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b590 tx=0x7f26d000b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a873ff7e0d4c28d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:29:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26c8ff9640 1 -- 172.21.15.33:0/2419588004 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d000f3c0 con 0x55c9af46e650 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26c8ff9640 1 -- 172.21.15.33:0/2419588004 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f26d001a040 con 0x55c9af46e650 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26e96d9740 1 -- 172.21.15.33:0/2419588004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26d000f8a0 con 0x55c9af46e650 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26c8ff9640 1 -- 172.21.15.33:0/2419588004 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f26d000f560 con 0x55c9af46e650 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26e96d9740 10 client.17667 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.594+0000 7f26e96d9740 1 -- 172.21.15.33:0/2419588004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c9af4e95b0 con 0x55c9af46e650 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.595+0000 7f26c8ff9640 1 -- 172.21.15.33:0/2419588004 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26d000f8a0 con 0x55c9af46e650 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.595+0000 7f26c8ff9640 1 -- 172.21.15.33:0/2419588004 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f26d000da30 con 0x55c9af46e650 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.595+0000 7f26e96d9740 10 client.17667 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.595+0000 7f26e96d9740 10 client.17667 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.595+0000 7f26e96d9740 20 client.17667 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.595+0000 7f26e96d9740 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c9af4e9c00 0x55c9af4ebff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.595+0000 7f26e96d9740 4 client.17667 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "26e2c7d4", "release": true, "await": true}] 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.595+0000 7f26e96d9740 1 -- 172.21.15.33:0/2419588004 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "26e2c7d4", "release": true, "await": true}) -- 0x55c9af3cc0a0 con 0x55c9af4e9c00 2024-09-16T00:29:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.595+0000 7f26e509c640 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c9af4e9c00 0x55c9af4ebff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.595+0000 7f26e509c640 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c9af4e9c00 0x55c9af4ebff0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47add58a2c86a803 server_cookie=11b17f917196a85c in_seq=0 out_seq=0 2024-09-16T00:29:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.595+0000 7f26c8ff9640 10 client.17667 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:15.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.610+0000 7f26c8ff9640 1 -- 172.21.15.33:0/2419588004 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c9af3cc0a0 con 0x55c9af4e9c00 2024-09-16T00:29:15.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.610+0000 7f26c8ff9640 10 client.17667 handle_command_reply: tid=0 2024-09-16T00:29:15.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.610+0000 7f26e96d9740 1 client.17667 shutdown 2024-09-16T00:29:15.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e96d9740 1 -- 172.21.15.33:0/2419588004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 msgr2=0x55c9af4e7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:15.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e96d9740 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af4e7e80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b590 tx=0x7f26d000b7a0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e96d9740 1 -- 172.21.15.33:0/2419588004 shutdown_connections 2024-09-16T00:29:15.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e96d9740 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c9af4e9c00 0x55c9af4ebff0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e96d9740 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9af46e650 0x55c9af4e7e80 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26c8ff9640 0 client.17667 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:15.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e96d9740 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c9af45f860 0x55c9af472f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e96d9740 1 --2- 172.21.15.33:0/2419588004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c9af45ef40 0x55c9af472a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e96d9740 1 -- 172.21.15.33:0/2419588004 >> 172.21.15.33:0/2419588004 conn(0x55c9af45b5f0 msgr2=0x55c9af46bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:15.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e96d9740 1 -- 172.21.15.33:0/2419588004 shutdown_connections 2024-09-16T00:29:15.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e96d9740 1 -- 172.21.15.33:0/2419588004 wait complete. 2024-09-16T00:29:15.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e96d9740 20 client.17667 trim_cache size 0 max 16384 2024-09-16T00:29:15.613 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:29:15.613 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:29:15.613 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:29:15.613 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 44, 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "26e2c7d4": { 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 44, 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 141.8, 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:29:15.614 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:29:15.615 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:29:15.615 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:29:15.615 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:29:15.615 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:29:15.615 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:29:15.615 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:29:15.615 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:29:15.615 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:29:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e7300640 1 -- 172.21.15.33:0/2077248669 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26cc0a3240 msgr2=0x7f26cc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e7300640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26cc0a3240 0x7f26cc0a5700 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f26d80b9ee0 tx=0x7f26d80e8250 comp rx=0 tx=0).stop 2024-09-16T00:29:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e7300640 1 -- 172.21.15.33:0/2077248669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e0106f10 msgr2=0x7f26e019cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e7300640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e0106f10 0x7f26e019cc20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f26d8031a20 tx=0x7f26d8031a50 comp rx=0 tx=0).stop 2024-09-16T00:29:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.611+0000 7f26e7300640 1 -- 172.21.15.33:0/2077248669 shutdown_connections 2024-09-16T00:29:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.612+0000 7f26e7300640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26cc0a3240 0x7f26cc0a5700 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.612+0000 7f26e7300640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26e0106f10 0x7f26e019cc20 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.612+0000 7f26e7300640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e0102790 0x7f26e0198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.612+0000 7f26e7300640 1 --2- 172.21.15.33:0/2077248669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e0101de0 0x7f26e0198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.612+0000 7f26e7300640 1 -- 172.21.15.33:0/2077248669 >> 172.21.15.33:0/2077248669 conn(0x7f26e00fd8c0 msgr2=0x7f26e00fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.612+0000 7f26e7300640 1 -- 172.21.15.33:0/2077248669 shutdown_connections 2024-09-16T00:29:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:15.612+0000 7f26e7300640 1 -- 172.21.15.33:0/2077248669 wait complete. 2024-09-16T00:29:15.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 995010 9.46 MB/sec execute 503 sec latency 21885.141 ms 2024-09-16T00:29:15.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:15 smithi060 ceph-mon[35185]: pgmap v713: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 15 op/s 2024-09-16T00:29:15.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:15 smithi033 ceph-mon[31243]: pgmap v713: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 15 op/s 2024-09-16T00:29:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:15 smithi083 ceph-mon[35242]: pgmap v713: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 15 op/s 2024-09-16T00:29:16.434 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '26e2c7d4', total seconds elapsed: 25.3 2024-09-16T00:29:16.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 997667 9.47 MB/sec execute 504 sec latency 12.937 ms 2024-09-16T00:29:17.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 999613 9.47 MB/sec execute 505 sec latency 14.620 ms 2024-09-16T00:29:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:17 smithi060 ceph-mon[35185]: pgmap v714: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.4 KiB/s wr, 10 op/s 2024-09-16T00:29:17.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:17 smithi033 ceph-mon[31243]: pgmap v714: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.4 KiB/s wr, 10 op/s 2024-09-16T00:29:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:17 smithi083 ceph-mon[35242]: pgmap v714: 129 pgs: 129 active+clean; 270 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.4 KiB/s wr, 10 op/s 2024-09-16T00:29:18.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1002435 9.48 MB/sec execute 506 sec latency 14.154 ms 2024-09-16T00:29:19.605 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:29:19.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1005473 9.50 MB/sec execute 507 sec latency 11.623 ms 2024-09-16T00:29:19.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:19 smithi060 ceph-mon[35185]: pgmap v715: 129 pgs: 129 active+clean; 271 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 86 KiB/s wr, 20 op/s 2024-09-16T00:29:19.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:19 smithi033 ceph-mon[31243]: pgmap v715: 129 pgs: 129 active+clean; 271 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 86 KiB/s wr, 20 op/s 2024-09-16T00:29:19.879 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:29:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:19 smithi083 ceph-mon[35242]: pgmap v715: 129 pgs: 129 active+clean; 271 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 86 KiB/s wr, 20 op/s 2024-09-16T00:29:20.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.407+0000 7fc9e76ae640 1 -- 172.21.15.33:0/2547589022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9e01008c0 msgr2=0x7fc9e0100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:20.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.407+0000 7fc9e76ae640 1 --2- 172.21.15.33:0/2547589022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9e01008c0 0x7fc9e0100ca0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009cd0 tx=0x7fc9d002a8e0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.407+0000 7fc9e76ae640 1 -- 172.21.15.33:0/2547589022 shutdown_connections 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.407+0000 7fc9e76ae640 1 --2- 172.21.15.33:0/2547589022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9e0105ec0 0x7fc9e0106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.407+0000 7fc9e76ae640 1 --2- 172.21.15.33:0/2547589022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e01011e0 0x7fc9e0105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.407+0000 7fc9e76ae640 1 --2- 172.21.15.33:0/2547589022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9e01008c0 0x7fc9e0100ca0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.407+0000 7fc9e76ae640 1 -- 172.21.15.33:0/2547589022 >> 172.21.15.33:0/2547589022 conn(0x7fc9e00fc7d0 msgr2=0x7fc9e00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.407+0000 7fc9e76ae640 1 -- 172.21.15.33:0/2547589022 shutdown_connections 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.408+0000 7fc9e76ae640 1 -- 172.21.15.33:0/2547589022 wait complete. 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.408+0000 7fc9e76ae640 1 Processor -- start 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.408+0000 7fc9e76ae640 1 -- start start 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.408+0000 7fc9e76ae640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9e01008c0 0x7fc9e01a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.408+0000 7fc9e76ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e01011e0 0x7fc9e01a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.408+0000 7fc9e76ae640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9e0105ec0 0x7fc9e019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.408+0000 7fc9e76ae640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e0118610 con 0x7fc9e0105ec0 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.408+0000 7fc9e76ae640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e0118490 con 0x7fc9e01008c0 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.408+0000 7fc9e76ae640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e0118790 con 0x7fc9e01011e0 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9e5c4b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9e0105ec0 0x7fc9e019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9e5c4b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9e0105ec0 0x7fc9e019e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43394/0 (socket says 172.21.15.33:43394) 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9e5c4b640 1 -- 172.21.15.33:0/1121037210 learned_addr learned my addr 172.21.15.33:0/1121037210 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9e4c49640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e01011e0 0x7fc9e01a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9e544a640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9e01008c0 0x7fc9e01a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9e5c4b640 1 -- 172.21.15.33:0/1121037210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e01011e0 msgr2=0x7fc9e01a4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9e5c4b640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e01011e0 0x7fc9e01a4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9e5c4b640 1 -- 172.21.15.33:0/1121037210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9e01008c0 msgr2=0x7fc9e01a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9e5c4b640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9e01008c0 0x7fc9e01a47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9e5c4b640 1 -- 172.21.15.33:0/1121037210 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e019f0c0 con 0x7fc9e0105ec0 2024-09-16T00:29:20.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9e5c4b640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9e0105ec0 0x7fc9e019e990 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800e970 tx=0x7fc9d800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9de7fc640 1 -- 172.21.15.33:0/1121037210 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d800ccc0 con 0x7fc9e0105ec0 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9e76ae640 1 -- 172.21.15.33:0/1121037210 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e019f3b0 con 0x7fc9e0105ec0 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9de7fc640 1 -- 172.21.15.33:0/1121037210 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9d800ce60 con 0x7fc9e0105ec0 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.409+0000 7fc9de7fc640 1 -- 172.21.15.33:0/1121037210 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8010490 con 0x7fc9e0105ec0 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.410+0000 7fc9e76ae640 1 -- 172.21.15.33:0/1121037210 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e01ab6a0 con 0x7fc9e0105ec0 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.412+0000 7fc9de7fc640 1 -- 172.21.15.33:0/1121037210 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc9d8005020 con 0x7fc9e0105ec0 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9e9a87740 1 Processor -- start 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9e9a87740 1 -- start start 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9e9a87740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265aa4a0 0x55fb265aa880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9de7fc640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9c80a3240 0x7fc9c80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9e9a87740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aadc0 0x55fb265b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9e9a87740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265b5680 0x55fb265b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9e9a87740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fb265102e0 con 0x55fb265b5680 2024-09-16T00:29:20.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9e9a87740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55fb264b7ce0 con 0x55fb265aa4a0 2024-09-16T00:29:20.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9e9a87740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fb2650c2a0 con 0x55fb265aadc0 2024-09-16T00:29:20.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9e544a640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9c80a3240 0x7fc9c80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9e5c4b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265b5680 0x55fb265b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9e5c4b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265b5680 0x55fb265b7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43402/0 (socket says 172.21.15.33:43402) 2024-09-16T00:29:20.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9e5c4b640 1 -- 172.21.15.33:0/2454022445 learned_addr learned my addr 172.21.15.33:0/2454022445 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:20.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.413+0000 7fc9de7fc640 1 -- 172.21.15.33:0/1121037210 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc9d80d01e0 con 0x7fc9e0105ec0 2024-09-16T00:29:20.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.414+0000 7fc9e5c4b640 1 -- 172.21.15.33:0/2454022445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aadc0 msgr2=0x55fb265b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:20.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.414+0000 7fc9e5c4b640 1 --2- 172.21.15.33:0/2454022445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aadc0 0x55fb265b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.414+0000 7fc9e5c4b640 1 -- 172.21.15.33:0/2454022445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265aa4a0 msgr2=0x55fb265aa880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:20.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.414+0000 7fc9e5c4b640 1 --2- 172.21.15.33:0/2454022445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265aa4a0 0x55fb265aa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.414+0000 7fc9e5c4b640 1 -- 172.21.15.33:0/2454022445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb2649c440 con 0x55fb265b5680 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.414+0000 7fc9e5c4b640 1 --2- 172.21.15.33:0/2454022445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265b5680 0x55fb265b7a70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8004fd0 tx=0x7fc9d801bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd374ba83db68b9c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.414+0000 7fc9e544a640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9c80a3240 0x7fc9c80a5700 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009ca0 tx=0x7fc9d0009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.414+0000 7fc9c77fe640 1 -- 172.21.15.33:0/2454022445 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d80b5700 con 0x55fb265b5680 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.414+0000 7fc9c77fe640 1 -- 172.21.15.33:0/2454022445 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9d809ca20 con 0x55fb265b5680 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.414+0000 7fc9c77fe640 1 -- 172.21.15.33:0/2454022445 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d80b5520 con 0x55fb265b5680 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.415+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2454022445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265b5680 msgr2=0x55fb265b7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.415+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/2454022445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265b5680 0x55fb265b7a70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8004fd0 tx=0x7fc9d801bb30 comp rx=0 tx=0).stop 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.415+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2454022445 shutdown_connections 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.415+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/2454022445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265b5680 0x55fb265b7a70 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.415+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/2454022445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aadc0 0x55fb265b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.415+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/2454022445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265aa4a0 0x55fb265aa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.415+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2454022445 >> 172.21.15.33:0/2454022445 conn(0x55fb265a6a30 msgr2=0x55fb265a6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.415+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2454022445 shutdown_connections 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.415+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2454022445 wait complete. 2024-09-16T00:29:20.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e9a87740 1 -- start start 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e9a87740 1 Processor -- start 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e9a87740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265b5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e9a87740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265aadc0 0x55fb265b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e9a87740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265b2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e9a87740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fb265b8250 con 0x55fb265aadc0 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e9a87740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55fb264cb050 con 0x55fb265b5680 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e9a87740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fb265b83d0 con 0x55fb265aa4a0 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e9a87740 10 client.0 _ll_register_callbacks cb 0x55fb2650b160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e4c49640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265aadc0 0x55fb265b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e4c49640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265aadc0 0x55fb265b22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43408/0 (socket says 172.21.15.33:43408) 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e544a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.416+0000 7fc9e4c49640 1 -- 172.21.15.33:0/2444658991 learned_addr learned my addr 172.21.15.33:0/2444658991 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9e5c4b640 1 --2- 172.21.15.33:0/2444658991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9e4c49640 1 -- 172.21.15.33:0/2444658991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 msgr2=0x55fb265b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9e4c49640 1 --2- 172.21.15.33:0/2444658991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265b5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9e4c49640 1 -- 172.21.15.33:0/2444658991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 msgr2=0x55fb265b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9e4c49640 1 --2- 172.21.15.33:0/2444658991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9e4c49640 1 -- 172.21.15.33:0/2444658991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb262813c0 con 0x55fb265aadc0 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9e4c49640 1 --2- 172.21.15.33:0/2444658991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265aadc0 0x55fb265b22e0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4011670 tx=0x7fc9d4011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341ce4fed9896777 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9c4ff9640 1 -- 172.21.15.33:0/2444658991 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d401e060 con 0x55fb265aadc0 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9c4ff9640 1 -- 172.21.15.33:0/2444658991 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc9d400fd30 con 0x55fb265aadc0 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2444658991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9d809ca20 con 0x55fb265aadc0 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9c4ff9640 1 -- 172.21.15.33:0/2444658991 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc9d4025050 con 0x55fb265aadc0 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9e9a87740 10 client.17679 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.417+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2444658991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fb265c3cc0 con 0x55fb265aadc0 2024-09-16T00:29:20.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.418+0000 7fc9c4ff9640 1 -- 172.21.15.33:0/2444658991 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9d809ca20 con 0x55fb265aadc0 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.418+0000 7fc9c4ff9640 1 -- 172.21.15.33:0/2444658991 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fc9d4018420 con 0x55fb265aadc0 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.418+0000 7fc9e9a87740 10 client.17679 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.418+0000 7fc9e9a87740 10 client.17679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.418+0000 7fc9e9a87740 20 client.17679 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.418+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/2444658991 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb265c51d0 0x55fb265c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.418+0000 7fc9e9a87740 4 client.17679 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.418+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2444658991 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb2651d850 con 0x55fb265c51d0 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.418+0000 7fc9e544a640 1 --2- 172.21.15.33:0/2444658991 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb265c51d0 0x55fb265c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.418+0000 7fc9c4ff9640 10 client.17679 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.418+0000 7fc9e544a640 1 --2- 172.21.15.33:0/2444658991 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb265c51d0 0x55fb265c75f0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=956613d6ebab25f0 server_cookie=bfe7be128f437c88 in_seq=0 out_seq=0 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.419+0000 7fc9c4ff9640 1 -- 172.21.15.33:0/2444658991 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fb2651d850 con 0x55fb265c51d0 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.419+0000 7fc9c4ff9640 10 client.17679 handle_command_reply: tid=0 2024-09-16T00:29:20.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.419+0000 7fc9e9a87740 1 client.17679 shutdown 2024-09-16T00:29:20.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.419+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2444658991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265aadc0 msgr2=0x55fb265b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:20.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.419+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/2444658991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265aadc0 0x55fb265b22e0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4011670 tx=0x7fc9d4011b30 comp rx=0 tx=0).stop 2024-09-16T00:29:20.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.419+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2444658991 shutdown_connections 2024-09-16T00:29:20.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.419+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/2444658991 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb265c51d0 0x55fb265c75f0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.419+0000 7fc9c4ff9640 0 client.17679 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:20.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.419+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/2444658991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.419+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/2444658991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265aadc0 0x55fb265b22e0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.419+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/2444658991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265b5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.419+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2444658991 >> 172.21.15.33:0/2444658991 conn(0x55fb265a6a30 msgr2=0x55fb265ac760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:20.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.420+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2444658991 shutdown_connections 2024-09-16T00:29:20.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.420+0000 7fc9e9a87740 1 -- 172.21.15.33:0/2444658991 wait complete. 2024-09-16T00:29:20.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.420+0000 7fc9e9a87740 20 client.17679 trim_cache size 0 max 16384 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.439+0000 7fc9e9a87740 1 Processor -- start 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.439+0000 7fc9e9a87740 1 -- start start 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.439+0000 7fc9e9a87740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265c14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.439+0000 7fc9e9a87740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.439+0000 7fc9e9a87740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265cd400 0x55fb265b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e9a87740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e01b80f0 con 0x55fb265cd400 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e9a87740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e01b59c0 con 0x55fb265b5680 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e9a87740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e01b3810 con 0x55fb265aa4a0 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e5c4b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265cd400 0x55fb265b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e544a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265c14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e5c4b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265cd400 0x55fb265b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43424/0 (socket says 172.21.15.33:43424) 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e544a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265c14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34530/0 (socket says 172.21.15.33:34530) 2024-09-16T00:29:20.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e5c4b640 1 -- 172.21.15.33:0/611439863 learned_addr learned my addr 172.21.15.33:0/611439863 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:20.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e4c49640 1 --2- 172.21.15.33:0/611439863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e544a640 1 -- 172.21.15.33:0/611439863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 msgr2=0x55fb265ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:20.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e544a640 1 --2- 172.21.15.33:0/611439863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e544a640 1 -- 172.21.15.33:0/611439863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265cd400 msgr2=0x55fb265b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:20.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e544a640 1 --2- 172.21.15.33:0/611439863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265cd400 0x55fb265b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e544a640 1 -- 172.21.15.33:0/611439863 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb265c7c70 con 0x55fb265aa4a0 2024-09-16T00:29:20.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e5c4b640 1 --2- 172.21.15.33:0/611439863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265cd400 0x55fb265b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.440+0000 7fc9e544a640 1 --2- 172.21.15.33:0/611439863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265c14b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0005e30 tx=0x7fc9d003e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47fb7115798e0126 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.441+0000 7fc9c6ffd640 1 -- 172.21.15.33:0/611439863 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0055070 con 0x55fb265aa4a0 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.441+0000 7fc9c6ffd640 1 -- 172.21.15.33:0/611439863 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9d0003e50 con 0x55fb265aa4a0 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.441+0000 7fc9c6ffd640 1 -- 172.21.15.33:0/611439863 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0050630 con 0x55fb265aa4a0 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.442+0000 7fc9e9a87740 1 -- 172.21.15.33:0/611439863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 msgr2=0x55fb265c14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.442+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/611439863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265c14b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0005e30 tx=0x7fc9d003e000 comp rx=0 tx=0).stop 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.442+0000 7fc9e9a87740 1 -- 172.21.15.33:0/611439863 shutdown_connections 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.442+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/611439863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265cd400 0x55fb265b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.442+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/611439863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.442+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/611439863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265c14b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.442+0000 7fc9e9a87740 1 -- 172.21.15.33:0/611439863 >> 172.21.15.33:0/611439863 conn(0x55fb265a6a30 msgr2=0x55fb265b13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.442+0000 7fc9e9a87740 1 -- 172.21.15.33:0/611439863 shutdown_connections 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.442+0000 7fc9e9a87740 1 -- 172.21.15.33:0/611439863 wait complete. 2024-09-16T00:29:20.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.442+0000 7fc9e9a87740 1 -- start start 2024-09-16T00:29:20.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.442+0000 7fc9e9a87740 1 Processor -- start 2024-09-16T00:29:20.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e9a87740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265bdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e9a87740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265b9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e9a87740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265cd400 0x55fb265b9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e9a87740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e01b3e80 con 0x55fb265cd400 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e9a87740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e01b52f0 con 0x55fb265b5680 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e9a87740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e01b4550 con 0x55fb265aa4a0 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e9a87740 10 client.0 _ll_register_callbacks cb 0x7fc9d4009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e544a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265bdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e4c49640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265b9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e544a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265bdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34536/0 (socket says 172.21.15.33:34536) 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e4c49640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265b9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:41582/0 (socket says 172.21.15.33:41582) 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e544a640 1 -- 172.21.15.33:0/1851416065 learned_addr learned my addr 172.21.15.33:0/1851416065 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e5c4b640 1 --2- 172.21.15.33:0/1851416065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265cd400 0x55fb265b9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e544a640 1 -- 172.21.15.33:0/1851416065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 msgr2=0x55fb265b9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e544a640 1 --2- 172.21.15.33:0/1851416065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265b9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e544a640 1 -- 172.21.15.33:0/1851416065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265cd400 msgr2=0x55fb265b9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:20.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e544a640 1 --2- 172.21.15.33:0/1851416065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265cd400 0x55fb265b9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.443+0000 7fc9e544a640 1 -- 172.21.15.33:0/1851416065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb265b9d40 con 0x55fb265aa4a0 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.444+0000 7fc9e544a640 1 --2- 172.21.15.33:0/1851416065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265bdcd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0053040 tx=0x7fc9d0004760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=630e38e8693cd8e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.444+0000 7fc9c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.444+0000 7fc9c4ff9640 1 -- 172.21.15.33:0/1851416065 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d005e060 con 0x55fb265aa4a0 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.444+0000 7fc9e9a87740 1 -- 172.21.15.33:0/1851416065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9d0003e50 con 0x55fb265aa4a0 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.444+0000 7fc9c4ff9640 1 -- 172.21.15.33:0/1851416065 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc9d0004e00 con 0x55fb265aa4a0 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.444+0000 7fc9c4ff9640 1 -- 172.21.15.33:0/1851416065 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc9d0055070 con 0x55fb265aa4a0 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.444+0000 7fc9e9a87740 10 client.27055 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.444+0000 7fc9e9a87740 1 -- 172.21.15.33:0/1851416065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fb2662fe10 con 0x55fb265aa4a0 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.445+0000 7fc9c4ff9640 1 -- 172.21.15.33:0/1851416065 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9d0003e50 con 0x55fb265aa4a0 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.445+0000 7fc9c4ff9640 1 -- 172.21.15.33:0/1851416065 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fc9d0030070 con 0x55fb265aa4a0 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.445+0000 7fc9e9a87740 10 client.27055 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.445+0000 7fc9e9a87740 10 client.27055 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.445+0000 7fc9e9a87740 20 client.27055 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.445+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/1851416065 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb26630420 0x55fb26632840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.445+0000 7fc9e9a87740 4 client.27055 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.445+0000 7fc9e9a87740 1 -- 172.21.15.33:0/1851416065 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fb26517d60 con 0x55fb26630420 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.445+0000 7fc9e4c49640 1 --2- 172.21.15.33:0/1851416065 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb26630420 0x55fb26632840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.445+0000 7fc9e4c49640 1 --2- 172.21.15.33:0/1851416065 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb26630420 0x55fb26632840 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b1c3ac0c83a1c40 server_cookie=b299c4825004599d in_seq=0 out_seq=0 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.445+0000 7fc9c4ff9640 10 client.27055 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9c4ff9640 1 -- 172.21.15.33:0/1851416065 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fb26517d60 con 0x55fb26630420 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9c4ff9640 10 client.27055 handle_command_reply: tid=0 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9e9a87740 1 client.27055 shutdown 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9e9a87740 1 -- 172.21.15.33:0/1851416065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 msgr2=0x55fb265bdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/1851416065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265bdcd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0053040 tx=0x7fc9d0004760 comp rx=0 tx=0).stop 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9e9a87740 1 -- 172.21.15.33:0/1851416065 shutdown_connections 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/1851416065 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb26630420 0x55fb26632840 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/1851416065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb265cd400 0x55fb265b9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/1851416065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb265b5680 0x55fb265b9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9c4ff9640 0 client.27055 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:20.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9e9a87740 1 --2- 172.21.15.33:0/1851416065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb265aa4a0 0x55fb265bdcd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9e9a87740 1 -- 172.21.15.33:0/1851416065 >> 172.21.15.33:0/1851416065 conn(0x55fb265a6a30 msgr2=0x55fb265b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:20.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9e9a87740 1 -- 172.21.15.33:0/1851416065 shutdown_connections 2024-09-16T00:29:20.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9e9a87740 1 -- 172.21.15.33:0/1851416065 wait complete. 2024-09-16T00:29:20.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.446+0000 7fc9e9a87740 20 client.27055 trim_cache size 0 max 16384 2024-09-16T00:29:20.452 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:29:20.452 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:29:20.452 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:29:20.452 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:29:20.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.447+0000 7fc9e76ae640 1 -- 172.21.15.33:0/1121037210 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9c80a3240 msgr2=0x7fc9c80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:20.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.447+0000 7fc9e76ae640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9c80a3240 0x7fc9c80a5700 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009ca0 tx=0x7fc9d0009360 comp rx=0 tx=0).stop 2024-09-16T00:29:20.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.447+0000 7fc9e76ae640 1 -- 172.21.15.33:0/1121037210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9e0105ec0 msgr2=0x7fc9e019e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:20.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.447+0000 7fc9e76ae640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9e0105ec0 0x7fc9e019e990 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800e970 tx=0x7fc9d800ed70 comp rx=0 tx=0).stop 2024-09-16T00:29:20.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.447+0000 7fc9e76ae640 1 -- 172.21.15.33:0/1121037210 shutdown_connections 2024-09-16T00:29:20.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.447+0000 7fc9e76ae640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9c80a3240 0x7fc9c80a5700 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.447+0000 7fc9e76ae640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9e0105ec0 0x7fc9e019e990 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.447+0000 7fc9e76ae640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e01011e0 0x7fc9e01a4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.447+0000 7fc9e76ae640 1 --2- 172.21.15.33:0/1121037210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9e01008c0 0x7fc9e01a47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:20.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.447+0000 7fc9e76ae640 1 -- 172.21.15.33:0/1121037210 >> 172.21.15.33:0/1121037210 conn(0x7fc9e00fc7d0 msgr2=0x7fc9e010f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:20.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.447+0000 7fc9e76ae640 1 -- 172.21.15.33:0/1121037210 shutdown_connections 2024-09-16T00:29:20.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:20.447+0000 7fc9e76ae640 1 -- 172.21.15.33:0/1121037210 wait complete. 2024-09-16T00:29:20.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1007370 9.49 MB/sec execute 508 sec latency 28.155 ms 2024-09-16T00:29:21.530 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:29:21.530 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:29:21.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1009679 9.50 MB/sec execute 509 sec latency 29.844 ms 2024-09-16T00:29:21.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:21 smithi060 ceph-mon[35185]: pgmap v716: 129 pgs: 129 active+clean; 273 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 280 KiB/s wr, 49 op/s 2024-09-16T00:29:21.825 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:29:21.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:21 smithi033 ceph-mon[31243]: pgmap v716: 129 pgs: 129 active+clean; 273 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 280 KiB/s wr, 49 op/s 2024-09-16T00:29:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:21 smithi083 ceph-mon[35242]: pgmap v716: 129 pgs: 129 active+clean; 273 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 280 KiB/s wr, 49 op/s 2024-09-16T00:29:22.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.335+0000 7f185dcd0640 1 -- 172.21.15.33:0/277561965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1858101400 msgr2=0x7f185810d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:22.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.335+0000 7f185dcd0640 1 --2- 172.21.15.33:0/277561965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1858101400 0x7f185810d930 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f1850009cd0 tx=0x7f185002a8e0 comp rx=0 tx=0).stop 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.335+0000 7f185dcd0640 1 -- 172.21.15.33:0/277561965 shutdown_connections 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.335+0000 7f185dcd0640 1 --2- 172.21.15.33:0/277561965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1858101400 0x7f185810d930 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.335+0000 7f185dcd0640 1 --2- 172.21.15.33:0/277561965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1858100a60 0x7f1858100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.335+0000 7f185dcd0640 1 --2- 172.21.15.33:0/277561965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1858106a60 0x7f1858106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.335+0000 7f185dcd0640 1 -- 172.21.15.33:0/277561965 >> 172.21.15.33:0/277561965 conn(0x7f18580fc780 msgr2=0x7f18580feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.335+0000 7f185dcd0640 1 -- 172.21.15.33:0/277561965 shutdown_connections 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.335+0000 7f185dcd0640 1 -- 172.21.15.33:0/277561965 wait complete. 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.335+0000 7f185dcd0640 1 Processor -- start 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.335+0000 7f185dcd0640 1 -- start start 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f185dcd0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1858100a60 0x7f18581983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f185dcd0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1858101400 0x7f1858198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f185dcd0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1858106a60 0x7f185819cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:22.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f185dcd0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1858110e60 con 0x7f1858106a60 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f185dcd0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1858110ce0 con 0x7f1858101400 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f185dcd0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1858110fe0 con 0x7f1858100a60 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f1857fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1858106a60 0x7f185819cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f18577fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1858100a60 0x7f18581983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f184effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1858101400 0x7f1858198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f18577fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1858100a60 0x7f18581983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34548/0 (socket says 172.21.15.33:34548) 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f1857fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1858106a60 0x7f185819cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43442/0 (socket says 172.21.15.33:43442) 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f1857fff640 1 -- 172.21.15.33:0/1034444793 learned_addr learned my addr 172.21.15.33:0/1034444793 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f184effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1858101400 0x7f1858198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:41588/0 (socket says 172.21.15.33:41588) 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f1857fff640 1 -- 172.21.15.33:0/1034444793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1858100a60 msgr2=0x7f18581983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f1857fff640 1 --2- 172.21.15.33:0/1034444793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1858100a60 0x7f18581983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f1857fff640 1 -- 172.21.15.33:0/1034444793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1858101400 msgr2=0x7f1858198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:22.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f1857fff640 1 --2- 172.21.15.33:0/1034444793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1858101400 0x7f1858198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f1857fff640 1 -- 172.21.15.33:0/1034444793 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185819d370 con 0x7f1858106a60 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.336+0000 7f1857fff640 1 --2- 172.21.15.33:0/1034444793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1858106a60 0x7f185819cc90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f1850005cc0 tx=0x7f1850003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.337+0000 7f18557fa640 1 -- 172.21.15.33:0/1034444793 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1850004040 con 0x7f1858106a60 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.337+0000 7f185dcd0640 1 -- 172.21.15.33:0/1034444793 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f185819d600 con 0x7f1858106a60 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.337+0000 7f18557fa640 1 -- 172.21.15.33:0/1034444793 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18500046c0 con 0x7f1858106a60 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.337+0000 7f18557fa640 1 -- 172.21.15.33:0/1034444793 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1850046480 con 0x7f1858106a60 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.337+0000 7f185dcd0640 1 -- 172.21.15.33:0/1034444793 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18581a4f40 con 0x7f1858106a60 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.338+0000 7f185dcd0640 1 -- 172.21.15.33:0/1034444793 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1858100ec0 con 0x7f1858106a60 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.339+0000 7f18557fa640 1 -- 172.21.15.33:0/1034444793 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f18500305a0 con 0x7f1858106a60 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.340+0000 7f18557fa640 1 --2- 172.21.15.33:0/1034444793 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18440a3240 0x7f18440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.340+0000 7f18557fa640 1 -- 172.21.15.33:0/1034444793 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f185002e030 con 0x7f1858106a60 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.340+0000 7f18577fe640 1 --2- 172.21.15.33:0/1034444793 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18440a3240 0x7f18440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.341+0000 7f18577fe640 1 --2- 172.21.15.33:0/1034444793 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18440a3240 0x7f18440a5700 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f184800bb60 tx=0x7f184800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:22.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.343+0000 7f18557fa640 1 -- 172.21.15.33:0/1034444793 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f185004b060 con 0x7f1858106a60 2024-09-16T00:29:22.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:29:22.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:22.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:22.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:22.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:22 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:22.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.627+0000 7f185dcd0640 1 -- 172.21.15.33:0/1034444793 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1858199720 con 0x7f1858106a60 2024-09-16T00:29:22.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.628+0000 7f18557fa640 1 -- 172.21.15.33:0/1034444793 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f1858199720 con 0x7f1858106a60 2024-09-16T00:29:22.629 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:29:22.629 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:29:22.630 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:29:22.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.630+0000 7f185dcd0640 1 -- 172.21.15.33:0/1034444793 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18440a3240 msgr2=0x7f18440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:22.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.630+0000 7f185dcd0640 1 --2- 172.21.15.33:0/1034444793 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18440a3240 0x7f18440a5700 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f184800bb60 tx=0x7f184800b390 comp rx=0 tx=0).stop 2024-09-16T00:29:22.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.630+0000 7f185dcd0640 1 -- 172.21.15.33:0/1034444793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1858106a60 msgr2=0x7f185819cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:22.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.630+0000 7f185dcd0640 1 --2- 172.21.15.33:0/1034444793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1858106a60 0x7f185819cc90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f1850005cc0 tx=0x7f1850003970 comp rx=0 tx=0).stop 2024-09-16T00:29:22.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.630+0000 7f185dcd0640 1 -- 172.21.15.33:0/1034444793 shutdown_connections 2024-09-16T00:29:22.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.630+0000 7f185dcd0640 1 --2- 172.21.15.33:0/1034444793 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18440a3240 0x7f18440a5700 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:22.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.630+0000 7f185dcd0640 1 --2- 172.21.15.33:0/1034444793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1858106a60 0x7f185819cc90 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:22.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.630+0000 7f185dcd0640 1 --2- 172.21.15.33:0/1034444793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1858101400 0x7f1858198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:22.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.630+0000 7f185dcd0640 1 --2- 172.21.15.33:0/1034444793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1858100a60 0x7f18581983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:22.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.630+0000 7f185dcd0640 1 -- 172.21.15.33:0/1034444793 >> 172.21.15.33:0/1034444793 conn(0x7f18580fc780 msgr2=0x7f185806aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:22.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.631+0000 7f185dcd0640 1 -- 172.21.15.33:0/1034444793 shutdown_connections 2024-09-16T00:29:22.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:22.631+0000 7f185dcd0640 1 -- 172.21.15.33:0/1034444793 wait complete. 2024-09-16T00:29:22.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1011857 9.51 MB/sec execute 510 sec latency 67.760 ms 2024-09-16T00:29:22.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:29:22.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:22.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:22.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:22.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:22 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:29:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:22.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:22 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:23.340 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:29:23.622 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:29:23.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1013725 9.51 MB/sec execute 511 sec latency 12.349 ms 2024-09-16T00:29:23.743 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:23 smithi033 ceph-mon[31243]: pgmap v717: 129 pgs: 129 active+clean; 274 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 436 KiB/s wr, 61 op/s 2024-09-16T00:29:23.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:23 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1034444793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:29:23.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:29:23.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:29:23.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:23 smithi083 ceph-mon[35242]: pgmap v717: 129 pgs: 129 active+clean; 274 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 436 KiB/s wr, 61 op/s 2024-09-16T00:29:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:23 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1034444793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:29:23.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:29:23.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:29:23.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:23 smithi060 ceph-mon[35185]: pgmap v717: 129 pgs: 129 active+clean; 274 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 436 KiB/s wr, 61 op/s 2024-09-16T00:29:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:23 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1034444793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:29:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:29:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:29:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:29:24.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1016452 9.52 MB/sec execute 512 sec latency 24.880 ms 2024-09-16T00:29:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.689+0000 7fac3df48640 1 -- 172.21.15.33:0/3670485132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38103660 msgr2=0x7fac3810bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.689+0000 7fac3df48640 1 --2- 172.21.15.33:0/3670485132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38103660 0x7fac3810bb50 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fac20009900 tx=0x7fac2002a480 comp rx=0 tx=0).stop 2024-09-16T00:29:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.689+0000 7fac3df48640 1 -- 172.21.15.33:0/3670485132 shutdown_connections 2024-09-16T00:29:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.689+0000 7fac3df48640 1 --2- 172.21.15.33:0/3670485132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac3810c170 0x7fac3810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.689+0000 7fac3df48640 1 --2- 172.21.15.33:0/3670485132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38103660 0x7fac3810bb50 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.689+0000 7fac3df48640 1 --2- 172.21.15.33:0/3670485132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac38102d40 0x7fac38103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.689+0000 7fac3df48640 1 -- 172.21.15.33:0/3670485132 >> 172.21.15.33:0/3670485132 conn(0x7fac380fc7a0 msgr2=0x7fac380febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.689+0000 7fac3df48640 1 -- 172.21.15.33:0/3670485132 shutdown_connections 2024-09-16T00:29:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.689+0000 7fac3df48640 1 -- 172.21.15.33:0/3670485132 wait complete. 2024-09-16T00:29:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac3df48640 1 Processor -- start 2024-09-16T00:29:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac3df48640 1 -- start start 2024-09-16T00:29:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac3df48640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac38102d40 0x7fac38198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac3df48640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38103660 0x7fac381989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac3df48640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac3810c170 0x7fac3819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac3df48640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac38111f70 con 0x7fac38103660 2024-09-16T00:29:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac3df48640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fac38111df0 con 0x7fac3810c170 2024-09-16T00:29:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac3df48640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fac381120f0 con 0x7fac38102d40 2024-09-16T00:29:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac2effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38103660 0x7fac381989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac2effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38103660 0x7fac381989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43468/0 (socket says 172.21.15.33:43468) 2024-09-16T00:29:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac2effd640 1 -- 172.21.15.33:0/3169564289 learned_addr learned my addr 172.21.15.33:0/3169564289 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac37fff640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac3810c170 0x7fac3819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac377fe640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac38102d40 0x7fac38198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac2effd640 1 -- 172.21.15.33:0/3169564289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac38102d40 msgr2=0x7fac38198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac2effd640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac38102d40 0x7fac38198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac2effd640 1 -- 172.21.15.33:0/3169564289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac3810c170 msgr2=0x7fac3819cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac2effd640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac3810c170 0x7fac3819cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.690+0000 7fac2effd640 1 -- 172.21.15.33:0/3169564289 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3819d420 con 0x7fac38103660 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.691+0000 7fac37fff640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac3810c170 0x7fac3819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.691+0000 7fac2effd640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38103660 0x7fac381989b0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fac20006010 tx=0x7fac20031b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.691+0000 7fac357fa640 1 -- 172.21.15.33:0/3169564289 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2003f030 con 0x7fac38103660 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.691+0000 7fac357fa640 1 -- 172.21.15.33:0/3169564289 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac200043d0 con 0x7fac38103660 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.691+0000 7fac357fa640 1 -- 172.21.15.33:0/3169564289 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac20035070 con 0x7fac38103660 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.691+0000 7fac3df48640 1 -- 172.21.15.33:0/3169564289 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac3819d6b0 con 0x7fac38103660 2024-09-16T00:29:24.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.691+0000 7fac3df48640 1 -- 172.21.15.33:0/3169564289 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac381a4ff0 con 0x7fac38103660 2024-09-16T00:29:24.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.693+0000 7fac357fa640 1 -- 172.21.15.33:0/3169564289 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fac20042030 con 0x7fac38103660 2024-09-16T00:29:24.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac40321740 1 Processor -- start 2024-09-16T00:29:24.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac357fa640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac240a3240 0x7fac240a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac40321740 1 -- start start 2024-09-16T00:29:24.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac357fa640 1 -- 172.21.15.33:0/3169564289 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fac200f00d0 con 0x7fac38103660 2024-09-16T00:29:24.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac37fff640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac240a3240 0x7fac240a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac40321740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562bb40 0x56131562a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac40321740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131562a5d0 0x56131562aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac40321740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561315635370 0x561315637760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac40321740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5613155902f0 con 0x561315635370 2024-09-16T00:29:24.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac40321740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561315537cf0 con 0x56131562a5d0 2024-09-16T00:29:24.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac40321740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56131558c2b0 con 0x56131562bb40 2024-09-16T00:29:24.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac37fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561315635370 0x561315637760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.694+0000 7fac377fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562bb40 0x56131562a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac37fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561315635370 0x561315637760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43474/0 (socket says 172.21.15.33:43474) 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac37fff640 1 -- 172.21.15.33:0/3945255097 learned_addr learned my addr 172.21.15.33:0/3945255097 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac2effd640 1 --2- 172.21.15.33:0/3945255097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131562a5d0 0x56131562aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac37fff640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac240a3240 0x7fac240a5700 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fac30013b10 tx=0x7fac30013f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac2effd640 1 -- 172.21.15.33:0/3945255097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562bb40 msgr2=0x56131562a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac2effd640 1 --2- 172.21.15.33:0/3945255097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562bb40 0x56131562a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac2effd640 1 -- 172.21.15.33:0/3945255097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561315635370 msgr2=0x561315637760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac2effd640 1 --2- 172.21.15.33:0/3945255097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561315635370 0x561315637760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac2effd640 1 -- 172.21.15.33:0/3945255097 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56131551c3a0 con 0x56131562a5d0 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac37fff640 1 --2- 172.21.15.33:0/3945255097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561315635370 0x561315637760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac2effd640 1 --2- 172.21.15.33:0/3945255097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131562a5d0 0x56131562aa10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fac200ed6f0 tx=0x7fac200bbc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b3d9397e253cf88 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:24.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac2dffb640 1 -- 172.21.15.33:0/3945255097 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac20049040 con 0x56131562a5d0 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac2dffb640 1 -- 172.21.15.33:0/3945255097 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac200dae50 con 0x56131562a5d0 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.695+0000 7fac2dffb640 1 -- 172.21.15.33:0/3945255097 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac200c49f0 con 0x56131562a5d0 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.696+0000 7fac40321740 1 -- 172.21.15.33:0/3945255097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131562a5d0 msgr2=0x56131562aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.696+0000 7fac40321740 1 --2- 172.21.15.33:0/3945255097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131562a5d0 0x56131562aa10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fac200ed6f0 tx=0x7fac200bbc20 comp rx=0 tx=0).stop 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.696+0000 7fac40321740 1 -- 172.21.15.33:0/3945255097 shutdown_connections 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.696+0000 7fac40321740 1 --2- 172.21.15.33:0/3945255097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561315635370 0x561315637760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.696+0000 7fac40321740 1 --2- 172.21.15.33:0/3945255097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131562a5d0 0x56131562aa10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.696+0000 7fac40321740 1 --2- 172.21.15.33:0/3945255097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562bb40 0x56131562a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.696+0000 7fac40321740 1 -- 172.21.15.33:0/3945255097 >> 172.21.15.33:0/3945255097 conn(0x5613155f3310 msgr2=0x561315626c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.696+0000 7fac40321740 1 -- 172.21.15.33:0/3945255097 shutdown_connections 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.696+0000 7fac40321740 1 -- 172.21.15.33:0/3945255097 wait complete. 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac40321740 1 -- start start 2024-09-16T00:29:24.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac40321740 1 Processor -- start 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac40321740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131562a5d0 0x561315634de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac40321740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x561315632090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac40321740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561315635370 0x5613156327d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac40321740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561315537cf0 con 0x56131562bb40 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac40321740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56131554b060 con 0x56131562a5d0 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac40321740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561315637e30 con 0x561315635370 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac40321740 10 client.0 _ll_register_callbacks cb 0x56131529e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac2effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x561315632090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac37fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561315635370 0x5613156327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac377fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131562a5d0 0x561315634de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac37fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561315635370 0x5613156327d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34570/0 (socket says 172.21.15.33:34570) 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.697+0000 7fac37fff640 1 -- 172.21.15.33:0/2955371797 learned_addr learned my addr 172.21.15.33:0/2955371797 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:24.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac37fff640 1 -- 172.21.15.33:0/2955371797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131562a5d0 msgr2=0x561315634de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac37fff640 1 --2- 172.21.15.33:0/2955371797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131562a5d0 0x561315634de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac37fff640 1 -- 172.21.15.33:0/2955371797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 msgr2=0x561315632090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac37fff640 1 --2- 172.21.15.33:0/2955371797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x561315632090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac37fff640 1 -- 172.21.15.33:0/2955371797 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613153013c0 con 0x561315635370 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac37fff640 1 --2- 172.21.15.33:0/2955371797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561315635370 0x5613156327d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fac30013570 tx=0x7fac30004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c65dd36cda52fed3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac16ffd640 1 -- 172.21.15.33:0/2955371797 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac30029670 con 0x561315635370 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac40321740 1 -- 172.21.15.33:0/2955371797 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac200dae50 con 0x561315635370 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac16ffd640 1 -- 172.21.15.33:0/2955371797 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fac30029810 con 0x561315635370 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac16ffd640 1 -- 172.21.15.33:0/2955371797 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fac30032050 con 0x561315635370 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac40321740 10 client.27073 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac40321740 1 -- 172.21.15.33:0/2955371797 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561315643830 con 0x561315635370 2024-09-16T00:29:24.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.698+0000 7fac16ffd640 1 -- 172.21.15.33:0/2955371797 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac200dae50 con 0x561315635370 2024-09-16T00:29:24.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.699+0000 7fac16ffd640 1 -- 172.21.15.33:0/2955371797 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fac3001f070 con 0x561315635370 2024-09-16T00:29:24.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.699+0000 7fac40321740 10 client.27073 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:24.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.699+0000 7fac40321740 10 client.27073 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:29:24.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.699+0000 7fac40321740 20 client.27073 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:24.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.699+0000 7fac40321740 1 --2- 172.21.15.33:0/2955371797 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561315644c40 0x561315647050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.699+0000 7fac40321740 4 client.27073 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:29:24.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.699+0000 7fac40321740 1 -- 172.21.15.33:0/2955371797 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56131559d860 con 0x561315644c40 2024-09-16T00:29:24.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.699+0000 7fac377fe640 1 --2- 172.21.15.33:0/2955371797 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561315644c40 0x561315647050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.699+0000 7fac16ffd640 10 client.27073 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:29:24.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.699+0000 7fac377fe640 1 --2- 172.21.15.33:0/2955371797 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561315644c40 0x561315647050 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d1ab6e545506897c server_cookie=a90140bafe7f219a in_seq=0 out_seq=0 2024-09-16T00:29:24.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac16ffd640 1 -- 172.21.15.33:0/2955371797 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56131559d860 con 0x561315644c40 2024-09-16T00:29:24.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac16ffd640 10 client.27073 handle_command_reply: tid=0 2024-09-16T00:29:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac40321740 1 client.27073 shutdown 2024-09-16T00:29:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac40321740 1 -- 172.21.15.33:0/2955371797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561315635370 msgr2=0x5613156327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac40321740 1 --2- 172.21.15.33:0/2955371797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561315635370 0x5613156327d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fac30013570 tx=0x7fac30004ab0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac40321740 1 -- 172.21.15.33:0/2955371797 shutdown_connections 2024-09-16T00:29:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac40321740 1 --2- 172.21.15.33:0/2955371797 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x561315644c40 0x561315647050 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac40321740 1 --2- 172.21.15.33:0/2955371797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561315635370 0x5613156327d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac40321740 1 --2- 172.21.15.33:0/2955371797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x561315632090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac40321740 1 --2- 172.21.15.33:0/2955371797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131562a5d0 0x561315634de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac40321740 1 -- 172.21.15.33:0/2955371797 >> 172.21.15.33:0/2955371797 conn(0x5613155f3310 msgr2=0x56131562c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac16ffd640 0 client.27073 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:29:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac40321740 1 -- 172.21.15.33:0/2955371797 shutdown_connections 2024-09-16T00:29:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac40321740 1 -- 172.21.15.33:0/2955371797 wait complete. 2024-09-16T00:29:24.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.701+0000 7fac40321740 20 client.27073 trim_cache size 0 max 16384 2024-09-16T00:29:24.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac40321740 1 Processor -- start 2024-09-16T00:29:24.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac40321740 1 -- start start 2024-09-16T00:29:24.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac40321740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 0x5613156343c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac40321740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x56131562d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac40321740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561315635370 0x56131562d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac40321740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac381af790 con 0x56131562bb40 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac40321740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fac381ad060 con 0x561315635370 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac40321740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fac381aaeb0 con 0x56131562a5d0 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac2effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x56131562d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac377fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 0x5613156343c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac2effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x56131562d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43490/0 (socket says 172.21.15.33:43490) 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac2effd640 1 -- 172.21.15.33:0/1313150544 learned_addr learned my addr 172.21.15.33:0/1313150544 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac37fff640 1 --2- 172.21.15.33:0/1313150544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561315635370 0x56131562d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac2effd640 1 -- 172.21.15.33:0/1313150544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 msgr2=0x5613156343c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac2effd640 1 --2- 172.21.15.33:0/1313150544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 0x5613156343c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac2effd640 1 -- 172.21.15.33:0/1313150544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561315635370 msgr2=0x56131562d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:24.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac2effd640 1 --2- 172.21.15.33:0/1313150544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561315635370 0x56131562d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac2effd640 1 -- 172.21.15.33:0/1313150544 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561315637d40 con 0x56131562bb40 2024-09-16T00:29:24.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac377fe640 1 --2- 172.21.15.33:0/1313150544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 0x5613156343c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:24.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac2effd640 1 --2- 172.21.15.33:0/1313150544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x56131562d1b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fac2003d040 tx=0x7fac200b8b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167940888b12d9a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:24.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac2d7fa640 1 -- 172.21.15.33:0/1313150544 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac200d4070 con 0x56131562bb40 2024-09-16T00:29:24.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac2d7fa640 1 -- 172.21.15.33:0/1313150544 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac200c4e80 con 0x56131562bb40 2024-09-16T00:29:24.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.715+0000 7fac2d7fa640 1 -- 172.21.15.33:0/1313150544 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac20049040 con 0x56131562bb40 2024-09-16T00:29:24.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.716+0000 7fac40321740 1 -- 172.21.15.33:0/1313150544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 msgr2=0x56131562d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:24.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.716+0000 7fac40321740 1 --2- 172.21.15.33:0/1313150544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x56131562d1b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fac2003d040 tx=0x7fac200b8b40 comp rx=0 tx=0).stop 2024-09-16T00:29:24.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.716+0000 7fac40321740 1 -- 172.21.15.33:0/1313150544 shutdown_connections 2024-09-16T00:29:24.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.716+0000 7fac40321740 1 --2- 172.21.15.33:0/1313150544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561315635370 0x56131562d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.716+0000 7fac40321740 1 --2- 172.21.15.33:0/1313150544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x56131562d1b0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.716+0000 7fac40321740 1 --2- 172.21.15.33:0/1313150544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 0x5613156343c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.716+0000 7fac40321740 1 -- 172.21.15.33:0/1313150544 >> 172.21.15.33:0/1313150544 conn(0x5613155f3310 msgr2=0x561315631050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.716+0000 7fac40321740 1 -- 172.21.15.33:0/1313150544 shutdown_connections 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.716+0000 7fac40321740 1 -- 172.21.15.33:0/1313150544 wait complete. 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.716+0000 7fac40321740 1 -- start start 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac40321740 1 Processor -- start 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac40321740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 0x561315639590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac40321740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x561315639ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac40321740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561315635370 0x5613156ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac40321740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac381ab520 con 0x56131562bb40 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac40321740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fac381ac990 con 0x561315635370 2024-09-16T00:29:24.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac40321740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fac381abbf0 con 0x56131562a5d0 2024-09-16T00:29:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac40321740 10 client.0 _ll_register_callbacks cb 0x7fac30013840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac377fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 0x561315639590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac2effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x561315639ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac377fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 0x561315639590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34582/0 (socket says 172.21.15.33:34582) 2024-09-16T00:29:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac377fe640 1 -- 172.21.15.33:0/2522060290 learned_addr learned my addr 172.21.15.33:0/2522060290 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac37fff640 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561315635370 0x5613156ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac377fe640 1 -- 172.21.15.33:0/2522060290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561315635370 msgr2=0x5613156ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac377fe640 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561315635370 0x5613156ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac377fe640 1 -- 172.21.15.33:0/2522060290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 msgr2=0x561315639ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac377fe640 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x561315639ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac377fe640 1 -- 172.21.15.33:0/2522060290 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613156aeeb0 con 0x56131562a5d0 2024-09-16T00:29:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.717+0000 7fac2effd640 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x561315639ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac377fe640 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 0x561315639590 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fac28009fd0 tx=0x7fac2800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b78256ea3e29e8b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac16ffd640 1 -- 172.21.15.33:0/2522060290 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28026070 con 0x56131562a5d0 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac16ffd640 1 -- 172.21.15.33:0/2522060290 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fac28023600 con 0x56131562a5d0 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac40321740 1 -- 172.21.15.33:0/2522060290 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac200c4e80 con 0x56131562a5d0 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac16ffd640 1 -- 172.21.15.33:0/2522060290 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fac28023800 con 0x56131562a5d0 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac40321740 10 client.27079 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac40321740 1 -- 172.21.15.33:0/2522060290 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5613156b01d0 con 0x56131562a5d0 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac16ffd640 1 -- 172.21.15.33:0/2522060290 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac200c4e80 con 0x56131562a5d0 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac16ffd640 1 -- 172.21.15.33:0/2522060290 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fac3000fce0 con 0x56131562a5d0 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac40321740 10 client.27079 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac40321740 10 client.27079 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:29:24.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac40321740 20 client.27079 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.718+0000 7fac40321740 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5613156b08f0 0x5613156b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac40321740 4 client.27079 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac40321740 1 -- 172.21.15.33:0/2522060290 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561315597d70 con 0x5613156b08f0 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac2effd640 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5613156b08f0 0x5613156b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac16ffd640 10 client.27079 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac2effd640 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5613156b08f0 0x5613156b2ce0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=93d0cf3a1468f7ee server_cookie=1ccaa79c065b71e in_seq=0 out_seq=0 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac16ffd640 1 -- 172.21.15.33:0/2522060290 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561315597d70 con 0x5613156b08f0 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac16ffd640 10 client.27079 handle_command_reply: tid=0 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac40321740 1 client.27079 shutdown 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac40321740 1 -- 172.21.15.33:0/2522060290 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 msgr2=0x561315639590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac40321740 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 0x561315639590 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fac28009fd0 tx=0x7fac2800f790 comp rx=0 tx=0).stop 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac40321740 1 -- 172.21.15.33:0/2522060290 shutdown_connections 2024-09-16T00:29:24.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac40321740 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5613156b08f0 0x5613156b2ce0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac40321740 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561315635370 0x5613156ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac40321740 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56131562bb40 0x561315639ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac16ffd640 0 client.27079 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:29:24.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac40321740 1 --2- 172.21.15.33:0/2522060290 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56131562a5d0 0x561315639590 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.719+0000 7fac40321740 1 -- 172.21.15.33:0/2522060290 >> 172.21.15.33:0/2522060290 conn(0x5613155f3310 msgr2=0x561315630830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:24.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac40321740 1 -- 172.21.15.33:0/2522060290 shutdown_connections 2024-09-16T00:29:24.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac40321740 1 -- 172.21.15.33:0/2522060290 wait complete. 2024-09-16T00:29:24.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac40321740 20 client.27079 trim_cache size 0 max 16384 2024-09-16T00:29:24.726 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:29:24.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac3df48640 1 -- 172.21.15.33:0/3169564289 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac240a3240 msgr2=0x7fac240a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:24.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac3df48640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac240a3240 0x7fac240a5700 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fac30013b10 tx=0x7fac30013f10 comp rx=0 tx=0).stop 2024-09-16T00:29:24.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac3df48640 1 -- 172.21.15.33:0/3169564289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38103660 msgr2=0x7fac381989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:24.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac3df48640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38103660 0x7fac381989b0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fac20006010 tx=0x7fac20031b20 comp rx=0 tx=0).stop 2024-09-16T00:29:24.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac3df48640 1 -- 172.21.15.33:0/3169564289 shutdown_connections 2024-09-16T00:29:24.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac3df48640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac240a3240 0x7fac240a5700 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac3df48640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac3810c170 0x7fac3819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac3df48640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38103660 0x7fac381989b0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac3df48640 1 --2- 172.21.15.33:0/3169564289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac38102d40 0x7fac38198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:24.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac3df48640 1 -- 172.21.15.33:0/3169564289 >> 172.21.15.33:0/3169564289 conn(0x7fac380fc7a0 msgr2=0x7fac380fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:24.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac3df48640 1 -- 172.21.15.33:0/3169564289 shutdown_connections 2024-09-16T00:29:24.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:24.720+0000 7fac3df48640 1 -- 172.21.15.33:0/3169564289 wait complete. 2024-09-16T00:29:25.647 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:29:25.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1018988 9.53 MB/sec execute 513 sec latency 38.274 ms 2024-09-16T00:29:25.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:25 smithi033 ceph-mon[31243]: pgmap v718: 129 pgs: 129 active+clean; 278 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 982 KiB/s wr, 127 op/s 2024-09-16T00:29:25.913 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:29:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:25 smithi083 ceph-mon[35242]: pgmap v718: 129 pgs: 129 active+clean; 278 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 982 KiB/s wr, 127 op/s 2024-09-16T00:29:25.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:25 smithi060 ceph-mon[35185]: pgmap v718: 129 pgs: 129 active+clean; 278 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 982 KiB/s wr, 127 op/s 2024-09-16T00:29:26.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.501+0000 7fb346e20640 1 -- 172.21.15.33:0/354375042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb340101e10 msgr2=0x7fb3401021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:26.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.501+0000 7fb346e20640 1 --2- 172.21.15.33:0/354375042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb340101e10 0x7fb3401021f0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb334009900 tx=0x7fb33402a410 comp rx=0 tx=0).stop 2024-09-16T00:29:26.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.501+0000 7fb346e20640 1 -- 172.21.15.33:0/354375042 shutdown_connections 2024-09-16T00:29:26.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.501+0000 7fb346e20640 1 --2- 172.21.15.33:0/354375042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb340106f40 0x7fb34010eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.501+0000 7fb346e20640 1 --2- 172.21.15.33:0/354375042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3401027c0 0x7fb340106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.501+0000 7fb346e20640 1 --2- 172.21.15.33:0/354375042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb340101e10 0x7fb3401021f0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.501+0000 7fb346e20640 1 -- 172.21.15.33:0/354375042 >> 172.21.15.33:0/354375042 conn(0x7fb3400fd8d0 msgr2=0x7fb3400ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:26.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.501+0000 7fb346e20640 1 -- 172.21.15.33:0/354375042 shutdown_connections 2024-09-16T00:29:26.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.501+0000 7fb346e20640 1 -- 172.21.15.33:0/354375042 wait complete. 2024-09-16T00:29:26.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb346e20640 1 Processor -- start 2024-09-16T00:29:26.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb346e20640 1 -- start start 2024-09-16T00:29:26.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb346e20640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb340101e10 0x7fb3400744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb346e20640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3401027c0 0x7fb340074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb346e20640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb340106f40 0x7fb340074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb346e20640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb340111fa0 con 0x7fb340106f40 2024-09-16T00:29:26.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb346e20640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb340111e20 con 0x7fb340101e10 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb346e20640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb340112120 con 0x7fb3401027c0 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb3453bd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb340106f40 0x7fb340074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb33ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3401027c0 0x7fb340074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb344bbc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb340101e10 0x7fb3400744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb33ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3401027c0 0x7fb340074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34614/0 (socket says 172.21.15.33:34614) 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb344bbc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb340101e10 0x7fb3400744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:41672/0 (socket says 172.21.15.33:41672) 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb33ffff640 1 -- 172.21.15.33:0/4220705689 learned_addr learned my addr 172.21.15.33:0/4220705689 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb33ffff640 1 -- 172.21.15.33:0/4220705689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb340101e10 msgr2=0x7fb3400744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb33ffff640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb340101e10 0x7fb3400744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb33ffff640 1 -- 172.21.15.33:0/4220705689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb340106f40 msgr2=0x7fb340074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb33ffff640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb340106f40 0x7fb340074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb33ffff640 1 -- 172.21.15.33:0/4220705689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3401a68f0 con 0x7fb3401027c0 2024-09-16T00:29:26.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.502+0000 7fb3453bd640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb340106f40 0x7fb340074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:26.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.503+0000 7fb33ffff640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3401027c0 0x7fb340074a30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb32c00e9f0 tx=0x7fb32c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:26.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.503+0000 7fb33dffb640 1 -- 172.21.15.33:0/4220705689 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c00cd80 con 0x7fb3401027c0 2024-09-16T00:29:26.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.503+0000 7fb33dffb640 1 -- 172.21.15.33:0/4220705689 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb32c010910 con 0x7fb3401027c0 2024-09-16T00:29:26.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.503+0000 7fb346e20640 1 -- 172.21.15.33:0/4220705689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3401a6be0 con 0x7fb3401027c0 2024-09-16T00:29:26.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.503+0000 7fb33dffb640 1 -- 172.21.15.33:0/4220705689 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c018990 con 0x7fb3401027c0 2024-09-16T00:29:26.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.503+0000 7fb346e20640 1 -- 172.21.15.33:0/4220705689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3401a7170 con 0x7fb3401027c0 2024-09-16T00:29:26.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.505+0000 7fb33dffb640 1 -- 172.21.15.33:0/4220705689 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb32c010ac0 con 0x7fb3401027c0 2024-09-16T00:29:26.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.506+0000 7fb3491f9740 1 Processor -- start 2024-09-16T00:29:26.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.506+0000 7fb3491f9740 1 -- start start 2024-09-16T00:29:26.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.506+0000 7fb3491f9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 0x55c78c8fe090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.506+0000 7fb3491f9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c8fea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.506+0000 7fb3491f9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c909370 0x55c78c90b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.506+0000 7fb3491f9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c78c8642f0 con 0x55c78c8ffb40 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.506+0000 7fb3491f9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c78c80bcf0 con 0x55c78c8fe5d0 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.506+0000 7fb3491f9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c78c8602b0 con 0x55c78c909370 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb33dffb640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3300a32c0 0x7fb3300a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb3453bd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 0x55c78c8fe090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb3453bd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 0x55c78c8fe090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43520/0 (socket says 172.21.15.33:43520) 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb3453bd640 1 -- 172.21.15.33:0/1928448953 learned_addr learned my addr 172.21.15.33:0/1928448953 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb344bbc640 1 --2- 172.21.15.33:0/1928448953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c8fea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb3453bd640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3300a32c0 0x7fb3300a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb33ffff640 1 --2- 172.21.15.33:0/1928448953 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c909370 0x55c78c90b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb3453bd640 1 -- 172.21.15.33:0/1928448953 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c909370 msgr2=0x55c78c90b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:26.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb3453bd640 1 --2- 172.21.15.33:0/1928448953 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c909370 0x55c78c90b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb3453bd640 1 -- 172.21.15.33:0/1928448953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 msgr2=0x55c78c8fea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb3453bd640 1 --2- 172.21.15.33:0/1928448953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c8fea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb3453bd640 1 -- 172.21.15.33:0/1928448953 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c78c7f03a0 con 0x55c78c8ffb40 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb33dffb640 1 -- 172.21.15.33:0/4220705689 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb32c0d0060 con 0x7fb3401027c0 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb3453bd640 1 --2- 172.21.15.33:0/1928448953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 0x55c78c8fe090 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb33800f5c0 tx=0x7fb33800fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f3acbeabd6c640 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb3453bd640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3300a32c0 0x7fb3300a5780 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb3380099e0 tx=0x7fb33801e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.507+0000 7fb32affd640 1 -- 172.21.15.33:0/1928448953 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33802b070 con 0x55c78c8ffb40 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.508+0000 7fb32affd640 1 -- 172.21.15.33:0/1928448953 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3380268f0 con 0x55c78c8ffb40 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.508+0000 7fb32affd640 1 -- 172.21.15.33:0/1928448953 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb338022070 con 0x55c78c8ffb40 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.508+0000 7fb3491f9740 1 -- 172.21.15.33:0/1928448953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 msgr2=0x55c78c8fe090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.508+0000 7fb3491f9740 1 --2- 172.21.15.33:0/1928448953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 0x55c78c8fe090 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb33800f5c0 tx=0x7fb33800fa80 comp rx=0 tx=0).stop 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.508+0000 7fb3491f9740 1 -- 172.21.15.33:0/1928448953 shutdown_connections 2024-09-16T00:29:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.508+0000 7fb3491f9740 1 --2- 172.21.15.33:0/1928448953 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c909370 0x55c78c90b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.508+0000 7fb3491f9740 1 --2- 172.21.15.33:0/1928448953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c8fea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.508+0000 7fb3491f9740 1 --2- 172.21.15.33:0/1928448953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 0x55c78c8fe090 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.508+0000 7fb3491f9740 1 -- 172.21.15.33:0/1928448953 >> 172.21.15.33:0/1928448953 conn(0x55c78c8c7310 msgr2=0x55c78c8fac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.509+0000 7fb3491f9740 1 -- 172.21.15.33:0/1928448953 shutdown_connections 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.509+0000 7fb3491f9740 1 -- 172.21.15.33:0/1928448953 wait complete. 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.509+0000 7fb3491f9740 1 -- start start 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.509+0000 7fb3491f9740 1 Processor -- start 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.509+0000 7fb3491f9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c908d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.509+0000 7fb3491f9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 0x55c78c906000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.509+0000 7fb3491f9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c909370 0x55c78c9065d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.509+0000 7fb3491f9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c78c90c010 con 0x55c78c909370 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.509+0000 7fb3491f9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c78c81f060 con 0x55c78c8fe5d0 2024-09-16T00:29:26.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb3491f9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c78c90c190 con 0x55c78c8ffb40 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb3491f9740 10 client.0 _ll_register_callbacks cb 0x55c78c5728c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb33ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 0x55c78c906000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb33ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 0x55c78c906000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34626/0 (socket says 172.21.15.33:34626) 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb344bbc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c908d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb33ffff640 1 -- 172.21.15.33:0/288105676 learned_addr learned my addr 172.21.15.33:0/288105676 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb3453bd640 1 --2- 172.21.15.33:0/288105676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c909370 0x55c78c9065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb33ffff640 1 -- 172.21.15.33:0/288105676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 msgr2=0x55c78c908d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb33ffff640 1 --2- 172.21.15.33:0/288105676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c908d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb33ffff640 1 -- 172.21.15.33:0/288105676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c909370 msgr2=0x55c78c9065d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb33ffff640 1 --2- 172.21.15.33:0/288105676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c909370 0x55c78c9065d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb33ffff640 1 -- 172.21.15.33:0/288105676 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c78c5d53c0 con 0x55c78c8ffb40 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb33ffff640 1 --2- 172.21.15.33:0/288105676 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 0x55c78c906000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0986b0 tx=0x7fb32c06bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b21e8dc01f60fd65 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:26.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb31ffff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb31ffff640 1 -- 172.21.15.33:0/288105676 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c026040 con 0x55c78c8ffb40 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb31ffff640 1 -- 172.21.15.33:0/288105676 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb32c0bc0b0 con 0x55c78c8ffb40 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb3491f9740 1 -- 172.21.15.33:0/288105676 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3380268f0 con 0x55c78c8ffb40 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.510+0000 7fb31ffff640 1 -- 172.21.15.33:0/288105676 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb32c09c250 con 0x55c78c8ffb40 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb3491f9740 10 client.27091 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb3491f9740 1 -- 172.21.15.33:0/288105676 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c78c917af0 con 0x55c78c8ffb40 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb31ffff640 1 -- 172.21.15.33:0/288105676 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3380268f0 con 0x55c78c8ffb40 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb31ffff640 1 -- 172.21.15.33:0/288105676 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fb32c0a0100 con 0x55c78c8ffb40 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb3491f9740 10 client.27091 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb3491f9740 10 client.27091 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb3491f9740 20 client.27091 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb3491f9740 1 --2- 172.21.15.33:0/288105676 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c78c918fa0 0x55c78c91b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb3491f9740 4 client.27091 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb3491f9740 1 -- 172.21.15.33:0/288105676 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c78c871860 con 0x55c78c918fa0 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb344bbc640 1 --2- 172.21.15.33:0/288105676 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c78c918fa0 0x55c78c91b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb31ffff640 10 client.27091 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.511+0000 7fb344bbc640 1 --2- 172.21.15.33:0/288105676 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c78c918fa0 0x55c78c91b390 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=71a92fff6e795b30 server_cookie=c59f51bea6e577c4 in_seq=0 out_seq=0 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb31ffff640 1 -- 172.21.15.33:0/288105676 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c78c871860 con 0x55c78c918fa0 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb31ffff640 10 client.27091 handle_command_reply: tid=0 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb3491f9740 1 client.27091 shutdown 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb3491f9740 1 -- 172.21.15.33:0/288105676 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 msgr2=0x55c78c906000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb3491f9740 1 --2- 172.21.15.33:0/288105676 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 0x55c78c906000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0986b0 tx=0x7fb32c06bf80 comp rx=0 tx=0).stop 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb3491f9740 1 -- 172.21.15.33:0/288105676 shutdown_connections 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb3491f9740 1 --2- 172.21.15.33:0/288105676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c909370 0x55c78c9065d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb3491f9740 1 --2- 172.21.15.33:0/288105676 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 0x55c78c906000 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb3491f9740 1 --2- 172.21.15.33:0/288105676 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c78c918fa0 0x55c78c91b390 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb3491f9740 1 --2- 172.21.15.33:0/288105676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c908d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb31ffff640 0 client.27091 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:29:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb3491f9740 1 -- 172.21.15.33:0/288105676 >> 172.21.15.33:0/288105676 conn(0x55c78c8c7310 msgr2=0x55c78c900960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb3491f9740 1 -- 172.21.15.33:0/288105676 shutdown_connections 2024-09-16T00:29:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.512+0000 7fb3491f9740 1 -- 172.21.15.33:0/288105676 wait complete. 2024-09-16T00:29:26.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.513+0000 7fb3491f9740 20 client.27091 trim_cache size 0 max 16384 2024-09-16T00:29:26.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.526+0000 7fb3491f9740 1 Processor -- start 2024-09-16T00:29:26.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.526+0000 7fb3491f9740 1 -- start start 2024-09-16T00:29:26.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb3491f9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8fe5d0 0x55c78c9152b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb3491f9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 0x55c78c9058c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb3491f9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c909370 0x55c78c904fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb3491f9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb3401b3b70 con 0x55c78c8ffb40 2024-09-16T00:29:26.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb3491f9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb3401b1440 con 0x55c78c909370 2024-09-16T00:29:26.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb3491f9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb3401af290 con 0x55c78c8fe5d0 2024-09-16T00:29:26.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb33ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 0x55c78c9058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb344bbc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8fe5d0 0x55c78c9152b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb344bbc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8fe5d0 0x55c78c9152b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34634/0 (socket says 172.21.15.33:34634) 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb3453bd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c909370 0x55c78c904fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb344bbc640 1 -- 172.21.15.33:0/2877091120 learned_addr learned my addr 172.21.15.33:0/2877091120 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb344bbc640 1 -- 172.21.15.33:0/2877091120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c909370 msgr2=0x55c78c904fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb344bbc640 1 --2- 172.21.15.33:0/2877091120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c909370 0x55c78c904fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb344bbc640 1 -- 172.21.15.33:0/2877091120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 msgr2=0x55c78c9058c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb344bbc640 1 --2- 172.21.15.33:0/2877091120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 0x55c78c9058c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb344bbc640 1 -- 172.21.15.33:0/2877091120 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c78c8febe0 con 0x55c78c8fe5d0 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb33ffff640 1 --2- 172.21.15.33:0/2877091120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 0x55c78c9058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb3453bd640 1 --2- 172.21.15.33:0/2877091120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c909370 0x55c78c904fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.527+0000 7fb344bbc640 1 --2- 172.21.15.33:0/2877091120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8fe5d0 0x55c78c9152b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb33402a930 tx=0x7fb334031900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ffe4e3fc18fbdbf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.528+0000 7fb32a7fc640 1 -- 172.21.15.33:0/2877091120 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33404d030 con 0x55c78c8fe5d0 2024-09-16T00:29:26.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.528+0000 7fb32a7fc640 1 -- 172.21.15.33:0/2877091120 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb334031a70 con 0x55c78c8fe5d0 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.528+0000 7fb32a7fc640 1 -- 172.21.15.33:0/2877091120 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334035040 con 0x55c78c8fe5d0 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.528+0000 7fb3491f9740 1 -- 172.21.15.33:0/2877091120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8fe5d0 msgr2=0x55c78c9152b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.528+0000 7fb3491f9740 1 --2- 172.21.15.33:0/2877091120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8fe5d0 0x55c78c9152b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb33402a930 tx=0x7fb334031900 comp rx=0 tx=0).stop 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 -- 172.21.15.33:0/2877091120 shutdown_connections 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 --2- 172.21.15.33:0/2877091120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c909370 0x55c78c904fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 --2- 172.21.15.33:0/2877091120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c8ffb40 0x55c78c9058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 --2- 172.21.15.33:0/2877091120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8fe5d0 0x55c78c9152b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 -- 172.21.15.33:0/2877091120 >> 172.21.15.33:0/2877091120 conn(0x55c78c8c7310 msgr2=0x55c78c9048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 -- 172.21.15.33:0/2877091120 shutdown_connections 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 -- 172.21.15.33:0/2877091120 wait complete. 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 -- start start 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 Processor -- start 2024-09-16T00:29:26.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c911950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 0x55c78c90cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c909370 0x55c78c90d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb3401af900 con 0x55c78c909370 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb3401b0d70 con 0x55c78c8fe5d0 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.529+0000 7fb3491f9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb3401affd0 con 0x55c78c8ffb40 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb3491f9740 10 client.0 _ll_register_callbacks cb 0x7fb32c00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb3453bd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c909370 0x55c78c90d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb33ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 0x55c78c90cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb3453bd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c909370 0x55c78c90d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43542/0 (socket says 172.21.15.33:43542) 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb33ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 0x55c78c90cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34646/0 (socket says 172.21.15.33:34646) 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb3453bd640 1 -- 172.21.15.33:0/3055507932 learned_addr learned my addr 172.21.15.33:0/3055507932 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb344bbc640 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c911950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb33ffff640 1 -- 172.21.15.33:0/3055507932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 msgr2=0x55c78c911950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb33ffff640 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c911950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb33ffff640 1 -- 172.21.15.33:0/3055507932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c909370 msgr2=0x55c78c90d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb33ffff640 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c909370 0x55c78c90d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb33ffff640 1 -- 172.21.15.33:0/3055507932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c78c90d7b0 con 0x55c78c8ffb40 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb344bbc640 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c911950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb3453bd640 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c909370 0x55c78c90d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb33ffff640 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 0x55c78c90cc80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55c78c9075b0 tx=0x7fb32c018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d77b038c3c8888b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb31ffff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb31ffff640 1 -- 172.21.15.33:0/3055507932 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0a0550 con 0x55c78c8ffb40 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb31ffff640 1 -- 172.21.15.33:0/3055507932 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb32c018d40 con 0x55c78c8ffb40 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb3491f9740 1 -- 172.21.15.33:0/3055507932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb334031a70 con 0x55c78c8ffb40 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb31ffff640 1 -- 172.21.15.33:0/3055507932 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb32c0a4e10 con 0x55c78c8ffb40 2024-09-16T00:29:26.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb3491f9740 10 client.27097 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.530+0000 7fb3491f9740 1 -- 172.21.15.33:0/3055507932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c78c983a10 con 0x55c78c8ffb40 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.531+0000 7fb31ffff640 1 -- 172.21.15.33:0/3055507932 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb334031a70 con 0x55c78c8ffb40 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.531+0000 7fb31ffff640 1 -- 172.21.15.33:0/3055507932 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fb32c097fa0 con 0x55c78c8ffb40 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.531+0000 7fb3491f9740 10 client.27097 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.531+0000 7fb3491f9740 10 client.27097 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.531+0000 7fb3491f9740 20 client.27097 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.531+0000 7fb3491f9740 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c78c984130 0x55c78c986550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.531+0000 7fb3491f9740 4 client.27097 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.531+0000 7fb3491f9740 1 -- 172.21.15.33:0/3055507932 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c78c86bd70 con 0x55c78c984130 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.531+0000 7fb344bbc640 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c78c984130 0x55c78c986550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.531+0000 7fb344bbc640 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c78c984130 0x55c78c986550 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9a338bebf3829fef server_cookie=670f3aac156eabad in_seq=0 out_seq=0 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.531+0000 7fb31ffff640 10 client.27097 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:29:26.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb31ffff640 1 -- 172.21.15.33:0/3055507932 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c78c86bd70 con 0x55c78c984130 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb31ffff640 10 client.27097 handle_command_reply: tid=0 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb3491f9740 1 client.27097 shutdown 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb3491f9740 1 -- 172.21.15.33:0/3055507932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 msgr2=0x55c78c90cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb3491f9740 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 0x55c78c90cc80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55c78c9075b0 tx=0x7fb32c018420 comp rx=0 tx=0).stop 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb3491f9740 1 -- 172.21.15.33:0/3055507932 shutdown_connections 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb3491f9740 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c78c909370 0x55c78c90d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb3491f9740 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c78c8ffb40 0x55c78c90cc80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb3491f9740 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c78c984130 0x55c78c986550 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb3491f9740 1 --2- 172.21.15.33:0/3055507932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c78c8fe5d0 0x55c78c911950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb31ffff640 0 client.27097 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb3491f9740 1 -- 172.21.15.33:0/3055507932 >> 172.21.15.33:0/3055507932 conn(0x55c78c8c7310 msgr2=0x55c78c9050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb3491f9740 1 -- 172.21.15.33:0/3055507932 shutdown_connections 2024-09-16T00:29:26.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb3491f9740 1 -- 172.21.15.33:0/3055507932 wait complete. 2024-09-16T00:29:26.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.532+0000 7fb3491f9740 20 client.27097 trim_cache size 0 max 16384 2024-09-16T00:29:26.543 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:29:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.533+0000 7fb346e20640 1 -- 172.21.15.33:0/4220705689 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3300a32c0 msgr2=0x7fb3300a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.533+0000 7fb346e20640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3300a32c0 0x7fb3300a5780 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb3380099e0 tx=0x7fb33801e000 comp rx=0 tx=0).stop 2024-09-16T00:29:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.533+0000 7fb346e20640 1 -- 172.21.15.33:0/4220705689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3401027c0 msgr2=0x7fb340074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.533+0000 7fb346e20640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3401027c0 0x7fb340074a30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb32c00e9f0 tx=0x7fb32c00eeb0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.533+0000 7fb346e20640 1 -- 172.21.15.33:0/4220705689 shutdown_connections 2024-09-16T00:29:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.533+0000 7fb346e20640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3300a32c0 0x7fb3300a5780 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.533+0000 7fb346e20640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb340106f40 0x7fb340074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.533+0000 7fb346e20640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3401027c0 0x7fb340074a30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.533+0000 7fb346e20640 1 --2- 172.21.15.33:0/4220705689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb340101e10 0x7fb3400744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.533+0000 7fb346e20640 1 -- 172.21.15.33:0/4220705689 >> 172.21.15.33:0/4220705689 conn(0x7fb3400fd8d0 msgr2=0x7fb3400fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.533+0000 7fb346e20640 1 -- 172.21.15.33:0/4220705689 shutdown_connections 2024-09-16T00:29:26.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:26.533+0000 7fb346e20640 1 -- 172.21.15.33:0/4220705689 wait complete. 2024-09-16T00:29:26.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1020998 9.53 MB/sec execute 514 sec latency 15.649 ms 2024-09-16T00:29:27.338 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:29:27.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1023597 9.54 MB/sec execute 515 sec latency 23.744 ms 2024-09-16T00:29:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:29:27.809 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:27 smithi033 ceph-mon[31243]: pgmap v719: 129 pgs: 129 active+clean; 274 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-09-16T00:29:27.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:27 smithi083 ceph-mon[35242]: pgmap v719: 129 pgs: 129 active+clean; 274 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-09-16T00:29:27.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:27 smithi060 ceph-mon[35185]: pgmap v719: 129 pgs: 129 active+clean; 274 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-09-16T00:29:28.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.181+0000 7faf14118640 1 -- 172.21.15.33:0/4285297164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102790 msgr2=0x7faf0c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:28.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.181+0000 7faf14118640 1 --2- 172.21.15.33:0/4285297164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102790 0x7faf0c1067e0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7faefc009900 tx=0x7faefc02a3f0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.181+0000 7faf14118640 1 -- 172.21.15.33:0/4285297164 shutdown_connections 2024-09-16T00:29:28.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.181+0000 7faf14118640 1 --2- 172.21.15.33:0/4285297164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c106f10 0x7faf0c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.181+0000 7faf14118640 1 --2- 172.21.15.33:0/4285297164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102790 0x7faf0c1067e0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.181+0000 7faf14118640 1 --2- 172.21.15.33:0/4285297164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c101de0 0x7faf0c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.181+0000 7faf14118640 1 -- 172.21.15.33:0/4285297164 >> 172.21.15.33:0/4285297164 conn(0x7faf0c0fd8e0 msgr2=0x7faf0c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:28.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.182+0000 7faf14118640 1 -- 172.21.15.33:0/4285297164 shutdown_connections 2024-09-16T00:29:28.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.182+0000 7faf14118640 1 -- 172.21.15.33:0/4285297164 wait complete. 2024-09-16T00:29:28.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf14118640 1 Processor -- start 2024-09-16T00:29:28.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf14118640 1 -- start start 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf14118640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c101de0 0x7faf0c19a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf14118640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102790 0x7faf0c19abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf14118640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c106f10 0x7faf0c19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf14118640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c111f30 con 0x7faf0c102790 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf14118640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c111db0 con 0x7faf0c101de0 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf14118640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1120b0 con 0x7faf0c106f10 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf116b3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102790 0x7faf0c19abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf11eb4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c101de0 0x7faf0c19a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf11eb4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c101de0 0x7faf0c19a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:41742/0 (socket says 172.21.15.33:41742) 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf116b3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102790 0x7faf0c19abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43560/0 (socket says 172.21.15.33:43560) 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf126b5640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c106f10 0x7faf0c19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.183+0000 7faf11eb4640 1 -- 172.21.15.33:0/2707632352 learned_addr learned my addr 172.21.15.33:0/2707632352 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:28.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf11eb4640 1 -- 172.21.15.33:0/2707632352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c106f10 msgr2=0x7faf0c19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf11eb4640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c106f10 0x7faf0c19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf11eb4640 1 -- 172.21.15.33:0/2707632352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102790 msgr2=0x7faf0c19abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf11eb4640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102790 0x7faf0c19abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf11eb4640 1 -- 172.21.15.33:0/2707632352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0c19f630 con 0x7faf0c101de0 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf126b5640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c106f10 0x7faf0c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf116b3640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102790 0x7faf0c19abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf11eb4640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c101de0 0x7faf0c19a680 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faf0800e9f0 tx=0x7faf0800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf02ffd640 1 -- 172.21.15.33:0/2707632352 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0800cd80 con 0x7faf0c101de0 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf02ffd640 1 -- 172.21.15.33:0/2707632352 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf08010910 con 0x7faf0c101de0 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf14118640 1 -- 172.21.15.33:0/2707632352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0c19f920 con 0x7faf0c101de0 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf02ffd640 1 -- 172.21.15.33:0/2707632352 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf08018990 con 0x7faf0c101de0 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.184+0000 7faf14118640 1 -- 172.21.15.33:0/2707632352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf0c1a7200 con 0x7faf0c101de0 2024-09-16T00:29:28.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.187+0000 7faf02ffd640 1 -- 172.21.15.33:0/2707632352 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7faf08010ac0 con 0x7faf0c101de0 2024-09-16T00:29:28.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.187+0000 7faf164f1740 1 Processor -- start 2024-09-16T00:29:28.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.187+0000 7faf164f1740 1 -- start start 2024-09-16T00:29:28.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.187+0000 7faf164f1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1be6b40 0x5559a1be5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.187+0000 7faf164f1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be55d0 0x5559a1be5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.187+0000 7faf164f1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1bf0370 0x5559a1bf2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf164f1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5559a1b4b2f0 con 0x5559a1be55d0 2024-09-16T00:29:28.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf164f1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5559a1af2cf0 con 0x5559a1be6b40 2024-09-16T00:29:28.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf164f1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5559a1b472b0 con 0x5559a1bf0370 2024-09-16T00:29:28.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf02ffd640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faef80a3240 0x7faef80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf116b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1bf0370 0x5559a1bf2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf126b5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1be6b40 0x5559a1be5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf116b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1bf0370 0x5559a1bf2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34674/0 (socket says 172.21.15.33:34674) 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf116b3640 1 -- 172.21.15.33:0/654245913 learned_addr learned my addr 172.21.15.33:0/654245913 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf11eb4640 1 --2- 172.21.15.33:0/654245913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be55d0 0x5559a1be5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf126b5640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faef80a3240 0x7faef80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf02ffd640 1 -- 172.21.15.33:0/2707632352 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7faf080d00a0 con 0x7faf0c101de0 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf116b3640 1 -- 172.21.15.33:0/654245913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1be6b40 msgr2=0x5559a1be5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf116b3640 1 --2- 172.21.15.33:0/654245913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1be6b40 0x5559a1be5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf116b3640 1 -- 172.21.15.33:0/654245913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be55d0 msgr2=0x5559a1be5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf116b3640 1 --2- 172.21.15.33:0/654245913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be55d0 0x5559a1be5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf116b3640 1 -- 172.21.15.33:0/654245913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559a1ad73a0 con 0x5559a1bf0370 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf116b3640 1 --2- 172.21.15.33:0/654245913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1bf0370 0x5559a1bf2760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faefc009900 tx=0x7faefc005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f401d73779d1d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.188+0000 7faf126b5640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faef80a3240 0x7faef80a5700 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7faf040099e0 tx=0x7faf04009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.189+0000 7faef7fff640 1 -- 172.21.15.33:0/654245913 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc035040 con 0x5559a1bf0370 2024-09-16T00:29:28.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.189+0000 7faef7fff640 1 -- 172.21.15.33:0/654245913 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faefc031830 con 0x5559a1bf0370 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.189+0000 7faef7fff640 1 -- 172.21.15.33:0/654245913 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc0047a0 con 0x5559a1bf0370 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.189+0000 7faf164f1740 1 -- 172.21.15.33:0/654245913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1bf0370 msgr2=0x5559a1bf2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.189+0000 7faf164f1740 1 --2- 172.21.15.33:0/654245913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1bf0370 0x5559a1bf2760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faefc009900 tx=0x7faefc005d50 comp rx=0 tx=0).stop 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.189+0000 7faf164f1740 1 -- 172.21.15.33:0/654245913 shutdown_connections 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.189+0000 7faf164f1740 1 --2- 172.21.15.33:0/654245913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1bf0370 0x5559a1bf2760 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.189+0000 7faf164f1740 1 --2- 172.21.15.33:0/654245913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be55d0 0x5559a1be5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.189+0000 7faf164f1740 1 --2- 172.21.15.33:0/654245913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1be6b40 0x5559a1be5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.189+0000 7faf164f1740 1 -- 172.21.15.33:0/654245913 >> 172.21.15.33:0/654245913 conn(0x5559a1bae310 msgr2=0x5559a1be1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.190+0000 7faf164f1740 1 -- 172.21.15.33:0/654245913 shutdown_connections 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.190+0000 7faf164f1740 1 -- 172.21.15.33:0/654245913 wait complete. 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.190+0000 7faf164f1740 1 -- start start 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.190+0000 7faf164f1740 1 Processor -- start 2024-09-16T00:29:28.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.190+0000 7faf164f1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be55d0 0x5559a1befc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.190+0000 7faf164f1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be6b40 0x5559a1becf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.190+0000 7faf164f1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bf0370 0x5559a1bed480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf164f1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5559a1bf3010 con 0x5559a1be55d0 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf164f1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5559a1b06060 con 0x5559a1bf0370 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf164f1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5559a1bf3190 con 0x5559a1be6b40 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf11eb4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be55d0 0x5559a1befc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf164f1740 10 client.0 _ll_register_callbacks cb 0x5559a18598c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf116b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be6b40 0x5559a1becf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf11eb4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be55d0 0x5559a1befc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43590/0 (socket says 172.21.15.33:43590) 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf116b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be6b40 0x5559a1becf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34686/0 (socket says 172.21.15.33:34686) 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf11eb4640 1 -- 172.21.15.33:0/267243977 learned_addr learned my addr 172.21.15.33:0/267243977 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf126b5640 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bf0370 0x5559a1bed480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf11eb4640 1 -- 172.21.15.33:0/267243977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be6b40 msgr2=0x5559a1becf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:28.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf11eb4640 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be6b40 0x5559a1becf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf11eb4640 1 -- 172.21.15.33:0/267243977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bf0370 msgr2=0x5559a1bed480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:28.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf11eb4640 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bf0370 0x5559a1bed480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf11eb4640 1 -- 172.21.15.33:0/267243977 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559a18bc3c0 con 0x5559a1be55d0 2024-09-16T00:29:28.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf116b3640 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be6b40 0x5559a1becf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:28.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf126b5640 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bf0370 0x5559a1bed480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:28.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf11eb4640 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be55d0 0x5559a1befc40 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7faf08098cc0 tx=0x7faf0801d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90ba08d7088a2ff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:28.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faef57fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:29:28.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faef57fa640 1 -- 172.21.15.33:0/267243977 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf08096e60 con 0x5559a1be55d0 2024-09-16T00:29:28.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faef57fa640 1 -- 172.21.15.33:0/267243977 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7faf08014d20 con 0x5559a1be55d0 2024-09-16T00:29:28.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.191+0000 7faf164f1740 1 -- 172.21.15.33:0/267243977 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faefc031830 con 0x5559a1be55d0 2024-09-16T00:29:28.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.192+0000 7faef57fa640 1 -- 172.21.15.33:0/267243977 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faf0809c2f0 con 0x5559a1be55d0 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.192+0000 7faf164f1740 10 client.17739 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.192+0000 7faf164f1740 1 -- 172.21.15.33:0/267243977 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5559a1bfea30 con 0x5559a1be55d0 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.192+0000 7faef57fa640 1 -- 172.21.15.33:0/267243977 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faefc031830 con 0x5559a1be55d0 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.192+0000 7faef57fa640 1 -- 172.21.15.33:0/267243977 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7faf080bc0d0 con 0x5559a1be55d0 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.192+0000 7faf164f1740 10 client.17739 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.192+0000 7faf164f1740 10 client.17739 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.192+0000 7faf164f1740 20 client.17739 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.192+0000 7faf164f1740 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5559a1bffee0 0x5559a1c022f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.192+0000 7faf164f1740 4 client.17739 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.192+0000 7faf164f1740 1 -- 172.21.15.33:0/267243977 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559a1b58860 con 0x5559a1bffee0 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.192+0000 7faf116b3640 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5559a1bffee0 0x5559a1c022f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.193+0000 7faf116b3640 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5559a1bffee0 0x5559a1c022f0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35e57c3e02ea5e81 server_cookie=277b106b62fa6d5f in_seq=0 out_seq=0 2024-09-16T00:29:28.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.193+0000 7faef57fa640 10 client.17739 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:28.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.193+0000 7faef57fa640 1 -- 172.21.15.33:0/267243977 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5559a1b58860 con 0x5559a1bffee0 2024-09-16T00:29:28.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.193+0000 7faef57fa640 10 client.17739 handle_command_reply: tid=0 2024-09-16T00:29:28.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.193+0000 7faf164f1740 1 client.17739 shutdown 2024-09-16T00:29:28.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.193+0000 7faf164f1740 1 -- 172.21.15.33:0/267243977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be55d0 msgr2=0x5559a1befc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:28.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.193+0000 7faf164f1740 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be55d0 0x5559a1befc40 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7faf08098cc0 tx=0x7faf0801d9e0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.194+0000 7faf164f1740 1 -- 172.21.15.33:0/267243977 shutdown_connections 2024-09-16T00:29:28.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.194+0000 7faf164f1740 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5559a1bffee0 0x5559a1c022f0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.194+0000 7faf164f1740 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bf0370 0x5559a1bed480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.194+0000 7faf164f1740 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be6b40 0x5559a1becf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.194+0000 7faf164f1740 1 --2- 172.21.15.33:0/267243977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be55d0 0x5559a1befc40 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.194+0000 7faef57fa640 0 client.17739 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:28.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.194+0000 7faf164f1740 1 -- 172.21.15.33:0/267243977 >> 172.21.15.33:0/267243977 conn(0x5559a1bae310 msgr2=0x5559a1be7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.194+0000 7faf164f1740 1 -- 172.21.15.33:0/267243977 shutdown_connections 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.194+0000 7faf164f1740 1 -- 172.21.15.33:0/267243977 wait complete. 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.194+0000 7faf164f1740 20 client.17739 trim_cache size 0 max 16384 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.207+0000 7faf164f1740 1 Processor -- start 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.207+0000 7faf164f1740 1 -- start start 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.207+0000 7faf164f1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bae310 0x5559a1bfc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.207+0000 7faf164f1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be55d0 0x5559a1be8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.207+0000 7faf164f1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be6b40 0x5559a1c07bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.207+0000 7faf164f1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1b3c70 con 0x5559a1be6b40 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.207+0000 7faf164f1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1b1540 con 0x5559a1bae310 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.207+0000 7faf164f1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1af390 con 0x5559a1be55d0 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faf11eb4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bae310 0x5559a1bfc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faf11eb4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bae310 0x5559a1bfc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:41772/0 (socket says 172.21.15.33:41772) 2024-09-16T00:29:28.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faf11eb4640 1 -- 172.21.15.33:0/1744046887 learned_addr learned my addr 172.21.15.33:0/1744046887 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:28.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faf116b3640 1 --2- 172.21.15.33:0/1744046887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be55d0 0x5559a1be8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faf126b5640 1 --2- 172.21.15.33:0/1744046887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be6b40 0x5559a1c07bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faf11eb4640 1 -- 172.21.15.33:0/1744046887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be55d0 msgr2=0x5559a1be8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:28.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faf11eb4640 1 --2- 172.21.15.33:0/1744046887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be55d0 0x5559a1be8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faf11eb4640 1 -- 172.21.15.33:0/1744046887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be6b40 msgr2=0x5559a1c07bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:28.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faf11eb4640 1 --2- 172.21.15.33:0/1744046887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be6b40 0x5559a1c07bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faf11eb4640 1 -- 172.21.15.33:0/1744046887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559a1be2960 con 0x5559a1bae310 2024-09-16T00:29:28.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faf11eb4640 1 --2- 172.21.15.33:0/1744046887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bae310 0x5559a1bfc2b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faf080a4980 tx=0x7faf08000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45cc822ed3bcbb22 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:28.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faef77fe640 1 -- 172.21.15.33:0/1744046887 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf080a05f0 con 0x5559a1bae310 2024-09-16T00:29:28.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faef77fe640 1 -- 172.21.15.33:0/1744046887 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf080b50d0 con 0x5559a1bae310 2024-09-16T00:29:28.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.208+0000 7faef77fe640 1 -- 172.21.15.33:0/1744046887 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0809c7b0 con 0x5559a1bae310 2024-09-16T00:29:28.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.209+0000 7faf164f1740 1 -- 172.21.15.33:0/1744046887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bae310 msgr2=0x5559a1bfc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:28.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.209+0000 7faf164f1740 1 --2- 172.21.15.33:0/1744046887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bae310 0x5559a1bfc2b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faf080a4980 tx=0x7faf08000bc0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.209+0000 7faf164f1740 1 -- 172.21.15.33:0/1744046887 shutdown_connections 2024-09-16T00:29:28.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.209+0000 7faf164f1740 1 --2- 172.21.15.33:0/1744046887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1be6b40 0x5559a1c07bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.209+0000 7faf164f1740 1 --2- 172.21.15.33:0/1744046887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be55d0 0x5559a1be8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.209+0000 7faf164f1740 1 --2- 172.21.15.33:0/1744046887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1bae310 0x5559a1bfc2b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.209+0000 7faf164f1740 1 -- 172.21.15.33:0/1744046887 >> 172.21.15.33:0/1744046887 conn(0x5559a1bed5e0 msgr2=0x5559a1bec030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.209+0000 7faf164f1740 1 -- 172.21.15.33:0/1744046887 shutdown_connections 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.209+0000 7faf164f1740 1 -- 172.21.15.33:0/1744046887 wait complete. 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.209+0000 7faf164f1740 1 -- start start 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf164f1740 1 Processor -- start 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf164f1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1bae310 0x5559a1bf4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf164f1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be55d0 0x5559a1bf4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf164f1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1be6b40 0x5559a1bf4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf164f1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1afa00 con 0x5559a1bae310 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf164f1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1b0e70 con 0x5559a1be6b40 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf164f1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1b00d0 con 0x5559a1be55d0 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf164f1740 10 client.0 _ll_register_callbacks cb 0x7faf0800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:28.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf11eb4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1bae310 0x5559a1bf4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf116b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be55d0 0x5559a1bf4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf126b5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1be6b40 0x5559a1bf4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf11eb4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1bae310 0x5559a1bf4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43596/0 (socket says 172.21.15.33:43596) 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf126b5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1be6b40 0x5559a1bf4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:41780/0 (socket says 172.21.15.33:41780) 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf11eb4640 1 -- 172.21.15.33:0/947572380 learned_addr learned my addr 172.21.15.33:0/947572380 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf116b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be55d0 0x5559a1bf4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34704/0 (socket says 172.21.15.33:34704) 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf126b5640 1 -- 172.21.15.33:0/947572380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be55d0 msgr2=0x5559a1bf4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf126b5640 1 --2- 172.21.15.33:0/947572380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be55d0 0x5559a1bf4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf126b5640 1 -- 172.21.15.33:0/947572380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1bae310 msgr2=0x5559a1bf4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf126b5640 1 --2- 172.21.15.33:0/947572380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1bae310 0x5559a1bf4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf126b5640 1 -- 172.21.15.33:0/947572380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559a1c69de0 con 0x5559a1be6b40 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.210+0000 7faf11eb4640 1 --2- 172.21.15.33:0/947572380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1bae310 0x5559a1bf4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:28.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.211+0000 7faf126b5640 1 --2- 172.21.15.33:0/947572380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1be6b40 0x5559a1bf4e90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7faf04011c80 tx=0x7faf04004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d244f0ac0f18fde4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:28.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.211+0000 7faef57fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:29:28.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.211+0000 7faef57fa640 1 -- 172.21.15.33:0/947572380 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf04030070 con 0x5559a1be6b40 2024-09-16T00:29:28.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.211+0000 7faef57fa640 1 -- 172.21.15.33:0/947572380 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7faf04003970 con 0x5559a1be6b40 2024-09-16T00:29:28.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.211+0000 7faf164f1740 1 -- 172.21.15.33:0/947572380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf080b50d0 con 0x5559a1be6b40 2024-09-16T00:29:28.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.211+0000 7faef57fa640 1 -- 172.21.15.33:0/947572380 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faf0401fe60 con 0x5559a1be6b40 2024-09-16T00:29:28.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.211+0000 7faf164f1740 10 client.27122 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:28.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.211+0000 7faf164f1740 1 -- 172.21.15.33:0/947572380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5559a1c6b120 con 0x5559a1be6b40 2024-09-16T00:29:28.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.211+0000 7faef57fa640 1 -- 172.21.15.33:0/947572380 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf080b50d0 con 0x5559a1be6b40 2024-09-16T00:29:28.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.211+0000 7faef57fa640 1 -- 172.21.15.33:0/947572380 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7faf0401d070 con 0x5559a1be6b40 2024-09-16T00:29:28.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf164f1740 10 client.27122 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:28.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf164f1740 10 client.27122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:29:28.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf164f1740 20 client.27122 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf164f1740 1 --2- 172.21.15.33:0/947572380 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5559a1bf0370 0x5559a1c6d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf164f1740 4 client.27122 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf164f1740 1 -- 172.21.15.33:0/947572380 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5559a1b52d70 con 0x5559a1bf0370 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf11eb4640 1 --2- 172.21.15.33:0/947572380 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5559a1bf0370 0x5559a1c6d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faef57fa640 10 client.27122 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf11eb4640 1 --2- 172.21.15.33:0/947572380 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5559a1bf0370 0x5559a1c6d8a0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=590e7c3f27c5cace server_cookie=71be45883e96afe0 in_seq=0 out_seq=0 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faef57fa640 1 -- 172.21.15.33:0/947572380 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5559a1b52d70 con 0x5559a1bf0370 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faef57fa640 10 client.27122 handle_command_reply: tid=0 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf164f1740 1 client.27122 shutdown 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf164f1740 1 -- 172.21.15.33:0/947572380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1be6b40 msgr2=0x5559a1bf4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf164f1740 1 --2- 172.21.15.33:0/947572380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1be6b40 0x5559a1bf4e90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7faf04011c80 tx=0x7faf04004a40 comp rx=0 tx=0).stop 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf164f1740 1 -- 172.21.15.33:0/947572380 shutdown_connections 2024-09-16T00:29:28.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.212+0000 7faf164f1740 1 --2- 172.21.15.33:0/947572380 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5559a1bf0370 0x5559a1c6d8a0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf164f1740 1 --2- 172.21.15.33:0/947572380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5559a1be6b40 0x5559a1bf4e90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf164f1740 1 --2- 172.21.15.33:0/947572380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5559a1be55d0 0x5559a1bf4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faef57fa640 0 client.27122 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:28.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf164f1740 1 --2- 172.21.15.33:0/947572380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559a1bae310 0x5559a1bf4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf164f1740 1 -- 172.21.15.33:0/947572380 >> 172.21.15.33:0/947572380 conn(0x5559a1bed5e0 msgr2=0x5559a1bec030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:28.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf164f1740 1 -- 172.21.15.33:0/947572380 shutdown_connections 2024-09-16T00:29:28.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf164f1740 1 -- 172.21.15.33:0/947572380 wait complete. 2024-09-16T00:29:28.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf164f1740 20 client.27122 trim_cache size 0 max 16384 2024-09-16T00:29:28.229 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:29:28.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf14118640 1 -- 172.21.15.33:0/2707632352 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faef80a3240 msgr2=0x7faef80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:28.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf14118640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faef80a3240 0x7faef80a5700 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7faf040099e0 tx=0x7faf04009500 comp rx=0 tx=0).stop 2024-09-16T00:29:28.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf14118640 1 -- 172.21.15.33:0/2707632352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c101de0 msgr2=0x7faf0c19a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:28.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf14118640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c101de0 0x7faf0c19a680 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faf0800e9f0 tx=0x7faf0800eeb0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf14118640 1 -- 172.21.15.33:0/2707632352 shutdown_connections 2024-09-16T00:29:28.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf14118640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faef80a3240 0x7faef80a5700 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf14118640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c106f10 0x7faf0c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf14118640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102790 0x7faf0c19abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf14118640 1 --2- 172.21.15.33:0/2707632352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c101de0 0x7faf0c19a680 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:28.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf14118640 1 -- 172.21.15.33:0/2707632352 >> 172.21.15.33:0/2707632352 conn(0x7faf0c0fd8e0 msgr2=0x7faf0c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:28.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf14118640 1 -- 172.21.15.33:0/2707632352 shutdown_connections 2024-09-16T00:29:28.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:28.213+0000 7faf14118640 1 -- 172.21.15.33:0/2707632352 wait complete. 2024-09-16T00:29:28.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1026467 9.55 MB/sec execute 516 sec latency 97.757 ms 2024-09-16T00:29:29.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1029129 9.55 MB/sec execute 517 sec latency 16.900 ms 2024-09-16T00:29:29.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:29 smithi033 ceph-mon[31243]: pgmap v720: 129 pgs: 129 active+clean; 270 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-09-16T00:29:29.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:29 smithi083 ceph-mon[35242]: pgmap v720: 129 pgs: 129 active+clean; 270 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-09-16T00:29:29.963 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:29:29.963 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:29:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:29 smithi060 ceph-mon[35185]: pgmap v720: 129 pgs: 129 active+clean; 270 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-09-16T00:29:30.226 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:29:30.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1031221 9.56 MB/sec execute 518 sec latency 32.937 ms 2024-09-16T00:29:30.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.942+0000 7ff082d21640 1 -- 172.21.15.33:0/934221329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff07c102ef0 msgr2=0x7ff07c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:30.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.942+0000 7ff082d21640 1 --2- 172.21.15.33:0/934221329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff07c102ef0 0x7ff07c10da40 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff074009900 tx=0x7ff07402a480 comp rx=0 tx=0).stop 2024-09-16T00:29:30.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.942+0000 7ff082d21640 1 -- 172.21.15.33:0/934221329 shutdown_connections 2024-09-16T00:29:30.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.942+0000 7ff082d21640 1 --2- 172.21.15.33:0/934221329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff07c102ef0 0x7ff07c10da40 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.942+0000 7ff082d21640 1 --2- 172.21.15.33:0/934221329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff07c102570 0x7ff07c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.942+0000 7ff082d21640 1 --2- 172.21.15.33:0/934221329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff07c100e20 0x7ff07c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.942+0000 7ff082d21640 1 -- 172.21.15.33:0/934221329 >> 172.21.15.33:0/934221329 conn(0x7ff07c0fc7d0 msgr2=0x7ff07c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:30.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.942+0000 7ff082d21640 1 -- 172.21.15.33:0/934221329 shutdown_connections 2024-09-16T00:29:30.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.942+0000 7ff082d21640 1 -- 172.21.15.33:0/934221329 wait complete. 2024-09-16T00:29:30.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.943+0000 7ff082d21640 1 Processor -- start 2024-09-16T00:29:30.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.943+0000 7ff082d21640 1 -- start start 2024-09-16T00:29:30.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.943+0000 7ff082d21640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff07c100e20 0x7ff07c19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff082d21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff07c102570 0x7ff07c19ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff082d21640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff07c102ef0 0x7ff07c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff082d21640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c110f80 con 0x7ff07c102570 2024-09-16T00:29:30.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff082d21640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c110e00 con 0x7ff07c102ef0 2024-09-16T00:29:30.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff082d21640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c111100 con 0x7ff07c100e20 2024-09-16T00:29:30.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff080abd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff07c100e20 0x7ff07c19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff07bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff07c102570 0x7ff07c19ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff080abd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff07c100e20 0x7ff07c19a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:39960/0 (socket says 172.21.15.33:39960) 2024-09-16T00:29:30.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff080abd640 1 -- 172.21.15.33:0/2414016284 learned_addr learned my addr 172.21.15.33:0/2414016284 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:30.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff0812be640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff07c102ef0 0x7ff07c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff080abd640 1 -- 172.21.15.33:0/2414016284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff07c102ef0 msgr2=0x7ff07c19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:30.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff080abd640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff07c102ef0 0x7ff07c19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff080abd640 1 -- 172.21.15.33:0/2414016284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff07c102570 msgr2=0x7ff07c19ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:30.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff080abd640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff07c102570 0x7ff07c19ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff080abd640 1 -- 172.21.15.33:0/2414016284 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff07c19f670 con 0x7ff07c100e20 2024-09-16T00:29:30.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.944+0000 7ff07bfff640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff07c102570 0x7ff07c19ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:30.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.945+0000 7ff080abd640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff07c100e20 0x7ff07c19a6c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00e9f0 tx=0x7ff06c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:30.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.945+0000 7ff079ffb640 1 -- 172.21.15.33:0/2414016284 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c00cd80 con 0x7ff07c100e20 2024-09-16T00:29:30.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.945+0000 7ff079ffb640 1 -- 172.21.15.33:0/2414016284 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff06c005500 con 0x7ff07c100e20 2024-09-16T00:29:30.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.945+0000 7ff082d21640 1 -- 172.21.15.33:0/2414016284 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff07c19f960 con 0x7ff07c100e20 2024-09-16T00:29:30.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.945+0000 7ff079ffb640 1 -- 172.21.15.33:0/2414016284 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c010690 con 0x7ff07c100e20 2024-09-16T00:29:30.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.945+0000 7ff082d21640 1 -- 172.21.15.33:0/2414016284 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff07c107980 con 0x7ff07c100e20 2024-09-16T00:29:30.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff0850fa740 1 Processor -- start 2024-09-16T00:29:30.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff0850fa740 1 -- start start 2024-09-16T00:29:30.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff079ffb640 1 -- 172.21.15.33:0/2414016284 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff06c0046c0 con 0x7ff07c100e20 2024-09-16T00:29:30.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff0850fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 0x56492d852f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff0850fa740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d85d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff0850fa740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d860130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff0850fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56492d7b8280 con 0x56492d852b60 2024-09-16T00:29:30.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff0850fa740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56492d75fb10 con 0x56492d85dd40 2024-09-16T00:29:30.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff0850fa740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56492d7b4240 con 0x56492d853480 2024-09-16T00:29:30.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff0812be640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 0x56492d852f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff080abd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d85d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff080abd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d85d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:39964/0 (socket says 172.21.15.33:39964) 2024-09-16T00:29:30.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff0812be640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 0x56492d852f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44216/0 (socket says 172.21.15.33:44216) 2024-09-16T00:29:30.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff080abd640 1 -- 172.21.15.33:0/4005680721 learned_addr learned my addr 172.21.15.33:0/4005680721 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:30.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.948+0000 7ff07bfff640 1 --2- 172.21.15.33:0/4005680721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d860130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.949+0000 7ff080abd640 1 -- 172.21.15.33:0/4005680721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 msgr2=0x56492d860130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:30.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.949+0000 7ff080abd640 1 --2- 172.21.15.33:0/4005680721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d860130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.949+0000 7ff080abd640 1 -- 172.21.15.33:0/4005680721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 msgr2=0x56492d852f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:30.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.949+0000 7ff080abd640 1 --2- 172.21.15.33:0/4005680721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 0x56492d852f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.949+0000 7ff080abd640 1 -- 172.21.15.33:0/4005680721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56492d76c1f0 con 0x56492d853480 2024-09-16T00:29:30.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.949+0000 7ff079ffb640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0640a3240 0x7ff0640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.949+0000 7ff080abd640 1 --2- 172.21.15.33:0/4005680721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d85d800 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0d8260 tx=0x7ff06c0d8350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7a8c3f53312b32b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:30.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.949+0000 7ff0812be640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0640a3240 0x7ff0640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.949+0000 7ff079ffb640 1 -- 172.21.15.33:0/2414016284 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff06c0d0170 con 0x7ff07c100e20 2024-09-16T00:29:30.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.949+0000 7ff062ffd640 1 -- 172.21.15.33:0/4005680721 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c0985b0 con 0x56492d853480 2024-09-16T00:29:30.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.949+0000 7ff062ffd640 1 -- 172.21.15.33:0/4005680721 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff06c09ca10 con 0x56492d853480 2024-09-16T00:29:30.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.949+0000 7ff062ffd640 1 -- 172.21.15.33:0/4005680721 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c09bc70 con 0x56492d853480 2024-09-16T00:29:30.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.950+0000 7ff0850fa740 1 -- 172.21.15.33:0/4005680721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 msgr2=0x56492d85d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:30.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.950+0000 7ff0812be640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0640a3240 0x7ff0640a5700 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff0740314e0 tx=0x7ff074009620 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:30.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.950+0000 7ff0850fa740 1 --2- 172.21.15.33:0/4005680721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d85d800 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0d8260 tx=0x7ff06c0d8350 comp rx=0 tx=0).stop 2024-09-16T00:29:30.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.950+0000 7ff0850fa740 1 -- 172.21.15.33:0/4005680721 shutdown_connections 2024-09-16T00:29:30.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.950+0000 7ff0850fa740 1 --2- 172.21.15.33:0/4005680721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d860130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.950+0000 7ff0850fa740 1 --2- 172.21.15.33:0/4005680721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d85d800 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.950+0000 7ff0850fa740 1 --2- 172.21.15.33:0/4005680721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 0x56492d852f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.950+0000 7ff0850fa740 1 -- 172.21.15.33:0/4005680721 >> 172.21.15.33:0/4005680721 conn(0x56492d84f0f0 msgr2=0x56492d84f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:30.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.950+0000 7ff0850fa740 1 -- 172.21.15.33:0/4005680721 shutdown_connections 2024-09-16T00:29:30.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.950+0000 7ff0850fa740 1 -- 172.21.15.33:0/4005680721 wait complete. 2024-09-16T00:29:30.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff0850fa740 1 -- start start 2024-09-16T00:29:30.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff0850fa740 1 Processor -- start 2024-09-16T00:29:30.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff0850fa740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d852b60 0x56492d85abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff0850fa740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d85b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff0850fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d85dd40 0x56492d85b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff0850fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56492d8608b0 con 0x56492d85dd40 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff0850fa740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56492d772f10 con 0x56492d852b60 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff0850fa740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56492d860a30 con 0x56492d853480 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff0850fa740 10 client.0 _ll_register_callbacks cb 0x56492d73ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff0812be640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d85dd40 0x56492d85b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff080abd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d852b60 0x56492d85abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff0812be640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d85dd40 0x56492d85b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44230/0 (socket says 172.21.15.33:44230) 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff07bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d85b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.951+0000 7ff0812be640 1 -- 172.21.15.33:0/3820838803 learned_addr learned my addr 172.21.15.33:0/3820838803 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff080abd640 1 -- 172.21.15.33:0/3820838803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 msgr2=0x56492d85b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:30.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff080abd640 1 --2- 172.21.15.33:0/3820838803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d85b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff080abd640 1 -- 172.21.15.33:0/3820838803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d85dd40 msgr2=0x56492d85b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff080abd640 1 --2- 172.21.15.33:0/3820838803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d85dd40 0x56492d85b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff080abd640 1 -- 172.21.15.33:0/3820838803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56492d5296a0 con 0x56492d852b60 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff080abd640 1 --2- 172.21.15.33:0/3820838803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d852b60 0x56492d85abf0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x56492d84ff50 tx=0x7ff06c0d0440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfd3e411a0f89949 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff05bfff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff05bfff640 1 -- 172.21.15.33:0/3820838803 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c09be50 con 0x56492d852b60 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff0850fa740 1 -- 172.21.15.33:0/3820838803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff06c09ca10 con 0x56492d852b60 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff05bfff640 1 -- 172.21.15.33:0/3820838803 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff06c0a4af0 con 0x56492d852b60 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff05bfff640 1 -- 172.21.15.33:0/3820838803 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff06c09c0e0 con 0x56492d852b60 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff0850fa740 10 client.27134 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.952+0000 7ff0850fa740 1 -- 172.21.15.33:0/3820838803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56492d8608b0 con 0x56492d852b60 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.953+0000 7ff05bfff640 1 -- 172.21.15.33:0/3820838803 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff06c09ca10 con 0x56492d852b60 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.953+0000 7ff05bfff640 1 -- 172.21.15.33:0/3820838803 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff06c0b5100 con 0x56492d852b60 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.953+0000 7ff0850fa740 10 client.27134 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:30.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.953+0000 7ff0850fa740 10 client.27134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.953+0000 7ff0850fa740 20 client.27134 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.953+0000 7ff0850fa740 1 --2- 172.21.15.33:0/3820838803 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56492d86d600 0x56492d86f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.953+0000 7ff0850fa740 4 client.27134 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.953+0000 7ff0850fa740 1 -- 172.21.15.33:0/3820838803 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56492d7c57f0 con 0x56492d86d600 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.953+0000 7ff07bfff640 1 --2- 172.21.15.33:0/3820838803 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56492d86d600 0x56492d86f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.953+0000 7ff07bfff640 1 --2- 172.21.15.33:0/3820838803 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56492d86d600 0x56492d86f9f0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9e2f2165d7e49de server_cookie=6d514298ae9301f5 in_seq=0 out_seq=0 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.953+0000 7ff05bfff640 10 client.27134 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.954+0000 7ff05bfff640 1 -- 172.21.15.33:0/3820838803 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56492d7c57f0 con 0x56492d86d600 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.954+0000 7ff05bfff640 10 client.27134 handle_command_reply: tid=0 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.954+0000 7ff0850fa740 1 client.27134 shutdown 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.954+0000 7ff0850fa740 1 -- 172.21.15.33:0/3820838803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d852b60 msgr2=0x56492d85abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.954+0000 7ff0850fa740 1 --2- 172.21.15.33:0/3820838803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d852b60 0x56492d85abf0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x56492d84ff50 tx=0x7ff06c0d0440 comp rx=0 tx=0).stop 2024-09-16T00:29:30.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.954+0000 7ff0850fa740 1 -- 172.21.15.33:0/3820838803 shutdown_connections 2024-09-16T00:29:30.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.954+0000 7ff0850fa740 1 --2- 172.21.15.33:0/3820838803 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56492d86d600 0x56492d86f9f0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.954+0000 7ff0850fa740 1 --2- 172.21.15.33:0/3820838803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d85dd40 0x56492d85b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.954+0000 7ff0850fa740 1 --2- 172.21.15.33:0/3820838803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d85b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.954+0000 7ff0850fa740 1 --2- 172.21.15.33:0/3820838803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d852b60 0x56492d85abf0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.954+0000 7ff0850fa740 1 -- 172.21.15.33:0/3820838803 >> 172.21.15.33:0/3820838803 conn(0x56492d84f0f0 msgr2=0x56492d854e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:30.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.954+0000 7ff05bfff640 0 client.27134 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:30.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.955+0000 7ff0850fa740 1 -- 172.21.15.33:0/3820838803 shutdown_connections 2024-09-16T00:29:30.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.955+0000 7ff0850fa740 1 -- 172.21.15.33:0/3820838803 wait complete. 2024-09-16T00:29:30.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.955+0000 7ff0850fa740 20 client.27134 trim_cache size 0 max 16384 2024-09-16T00:29:30.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.973+0000 7ff0850fa740 1 Processor -- start 2024-09-16T00:29:30.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff0850fa740 1 -- start start 2024-09-16T00:29:30.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff0850fa740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d852b60 0x56492d8555c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff0850fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d853480 0x56492d859aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff0850fa740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d875110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff0850fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c1af780 con 0x56492d853480 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff0850fa740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c1ad050 con 0x56492d85dd40 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff0850fa740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c1aaf00 con 0x56492d852b60 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff07bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d853480 0x56492d859aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff07bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d853480 0x56492d859aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44240/0 (socket says 172.21.15.33:44240) 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff07bfff640 1 -- 172.21.15.33:0/1040330297 learned_addr learned my addr 172.21.15.33:0/1040330297 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff0812be640 1 --2- 172.21.15.33:0/1040330297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d875110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff080abd640 1 --2- 172.21.15.33:0/1040330297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d852b60 0x56492d8555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff07bfff640 1 -- 172.21.15.33:0/1040330297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d852b60 msgr2=0x56492d8555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff07bfff640 1 --2- 172.21.15.33:0/1040330297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d852b60 0x56492d8555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff07bfff640 1 -- 172.21.15.33:0/1040330297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 msgr2=0x56492d875110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff07bfff640 1 --2- 172.21.15.33:0/1040330297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d875110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff07bfff640 1 -- 172.21.15.33:0/1040330297 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56492d850280 con 0x56492d853480 2024-09-16T00:29:30.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff0812be640 1 --2- 172.21.15.33:0/1040330297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d875110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:29:30.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.974+0000 7ff07bfff640 1 --2- 172.21.15.33:0/1040330297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d853480 0x56492d859aa0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ff070009fd0 tx=0x7ff070009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ca4f00dceb21fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:30.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.975+0000 7ff0627fc640 1 -- 172.21.15.33:0/1040330297 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff070004300 con 0x56492d853480 2024-09-16T00:29:30.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.975+0000 7ff0627fc640 1 -- 172.21.15.33:0/1040330297 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff070010430 con 0x56492d853480 2024-09-16T00:29:30.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.975+0000 7ff0627fc640 1 -- 172.21.15.33:0/1040330297 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07001f660 con 0x56492d853480 2024-09-16T00:29:30.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.975+0000 7ff0850fa740 1 -- 172.21.15.33:0/1040330297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d853480 msgr2=0x56492d859aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:30.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.975+0000 7ff0850fa740 1 --2- 172.21.15.33:0/1040330297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d853480 0x56492d859aa0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ff070009fd0 tx=0x7ff070009500 comp rx=0 tx=0).stop 2024-09-16T00:29:30.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.975+0000 7ff0850fa740 1 -- 172.21.15.33:0/1040330297 shutdown_connections 2024-09-16T00:29:30.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.975+0000 7ff0850fa740 1 --2- 172.21.15.33:0/1040330297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d875110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.975+0000 7ff0850fa740 1 --2- 172.21.15.33:0/1040330297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d853480 0x56492d859aa0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.975+0000 7ff0850fa740 1 --2- 172.21.15.33:0/1040330297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d852b60 0x56492d8555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.975+0000 7ff0850fa740 1 -- 172.21.15.33:0/1040330297 >> 172.21.15.33:0/1040330297 conn(0x56492d84f0f0 msgr2=0x56492d859390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:30.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.975+0000 7ff0850fa740 1 -- 172.21.15.33:0/1040330297 shutdown_connections 2024-09-16T00:29:30.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.975+0000 7ff0850fa740 1 -- 172.21.15.33:0/1040330297 wait complete. 2024-09-16T00:29:30.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.976+0000 7ff0850fa740 1 -- start start 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.976+0000 7ff0850fa740 1 Processor -- start 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.976+0000 7ff0850fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 0x56492d861aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.976+0000 7ff0850fa740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d861fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.976+0000 7ff0850fa740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d862520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.976+0000 7ff0850fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c1ab550 con 0x56492d852b60 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.976+0000 7ff0850fa740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c1ac980 con 0x56492d85dd40 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.976+0000 7ff0850fa740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c1abbe0 con 0x56492d853480 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.976+0000 7ff0850fa740 10 client.0 _ll_register_callbacks cb 0x7ff06c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.976+0000 7ff080abd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 0x56492d861aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.976+0000 7ff080abd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 0x56492d861aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44244/0 (socket says 172.21.15.33:44244) 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.976+0000 7ff080abd640 1 -- 172.21.15.33:0/2119831882 learned_addr learned my addr 172.21.15.33:0/2119831882 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff07bfff640 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d861fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff0812be640 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d862520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff080abd640 1 -- 172.21.15.33:0/2119831882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 msgr2=0x56492d861fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff080abd640 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d861fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff080abd640 1 -- 172.21.15.33:0/2119831882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 msgr2=0x56492d862520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff080abd640 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d862520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff080abd640 1 -- 172.21.15.33:0/2119831882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56492d8d71b0 con 0x56492d852b60 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff080abd640 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 0x56492d861aa0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0a4fa0 tx=0x7ff06c0d04b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b521e59eb473ee1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff05bfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff05bfff640 1 -- 172.21.15.33:0/2119831882 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c09be50 con 0x56492d852b60 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff05bfff640 1 -- 172.21.15.33:0/2119831882 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff06c0d3ad0 con 0x56492d852b60 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff0850fa740 1 -- 172.21.15.33:0/2119831882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff070010430 con 0x56492d852b60 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff05bfff640 1 -- 172.21.15.33:0/2119831882 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff06c09bc70 con 0x56492d852b60 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff0850fa740 10 client.17763 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.977+0000 7ff0850fa740 1 -- 172.21.15.33:0/2119831882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56492d8d84d0 con 0x56492d852b60 2024-09-16T00:29:30.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.978+0000 7ff05bfff640 1 -- 172.21.15.33:0/2119831882 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff070010430 con 0x56492d852b60 2024-09-16T00:29:30.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.978+0000 7ff05bfff640 1 -- 172.21.15.33:0/2119831882 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff06c0b02f0 con 0x56492d852b60 2024-09-16T00:29:30.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.978+0000 7ff0850fa740 10 client.17763 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:30.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.978+0000 7ff0850fa740 10 client.17763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:29:30.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.978+0000 7ff0850fa740 20 client.17763 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:30.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.978+0000 7ff0850fa740 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56492d8d8a00 0x56492d8dadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:30.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.978+0000 7ff0850fa740 4 client.17763 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:29:30.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.978+0000 7ff0850fa740 1 -- 172.21.15.33:0/2119831882 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56492d7bfd00 con 0x56492d8d8a00 2024-09-16T00:29:30.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.978+0000 7ff07bfff640 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56492d8d8a00 0x56492d8dadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:30.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.978+0000 7ff05bfff640 10 client.17763 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:30.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.978+0000 7ff07bfff640 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56492d8d8a00 0x56492d8dadf0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=237c091a3a839736 server_cookie=bfc940e7ec664cfa in_seq=0 out_seq=0 2024-09-16T00:29:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.979+0000 7ff05bfff640 1 -- 172.21.15.33:0/2119831882 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56492d7bfd00 con 0x56492d8d8a00 2024-09-16T00:29:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.979+0000 7ff05bfff640 10 client.17763 handle_command_reply: tid=0 2024-09-16T00:29:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.979+0000 7ff0850fa740 1 client.17763 shutdown 2024-09-16T00:29:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.979+0000 7ff0850fa740 1 -- 172.21.15.33:0/2119831882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 msgr2=0x56492d861aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.979+0000 7ff0850fa740 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 0x56492d861aa0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0a4fa0 tx=0x7ff06c0d04b0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.980+0000 7ff0850fa740 1 -- 172.21.15.33:0/2119831882 shutdown_connections 2024-09-16T00:29:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.980+0000 7ff0850fa740 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56492d8d8a00 0x56492d8dadf0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.980+0000 7ff05bfff640 0 client.17763 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.980+0000 7ff0850fa740 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56492d85dd40 0x56492d862520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.980+0000 7ff0850fa740 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56492d853480 0x56492d861fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.980+0000 7ff0850fa740 1 --2- 172.21.15.33:0/2119831882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56492d852b60 0x56492d861aa0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.980+0000 7ff0850fa740 1 -- 172.21.15.33:0/2119831882 >> 172.21.15.33:0/2119831882 conn(0x56492d84f0f0 msgr2=0x56492d84e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:30.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.980+0000 7ff0850fa740 1 -- 172.21.15.33:0/2119831882 shutdown_connections 2024-09-16T00:29:30.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.980+0000 7ff0850fa740 1 -- 172.21.15.33:0/2119831882 wait complete. 2024-09-16T00:29:30.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.980+0000 7ff0850fa740 20 client.17763 trim_cache size 0 max 16384 2024-09-16T00:29:30.985 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:29:30.985 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:29:30.985 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "c4fe5a15-c238-44f6-a6b7-b0b965870679", 2024-09-16T00:29:30.985 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:29:30.985 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:29:30.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.981+0000 7ff082d21640 1 -- 172.21.15.33:0/2414016284 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0640a3240 msgr2=0x7ff0640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:30.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.981+0000 7ff082d21640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0640a3240 0x7ff0640a5700 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff0740314e0 tx=0x7ff074009620 comp rx=0 tx=0).stop 2024-09-16T00:29:30.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.981+0000 7ff082d21640 1 -- 172.21.15.33:0/2414016284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff07c100e20 msgr2=0x7ff07c19a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:30.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.981+0000 7ff082d21640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff07c100e20 0x7ff07c19a6c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00e9f0 tx=0x7ff06c00edf0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.981+0000 7ff082d21640 1 -- 172.21.15.33:0/2414016284 shutdown_connections 2024-09-16T00:29:30.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.981+0000 7ff082d21640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff0640a3240 0x7ff0640a5700 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.981+0000 7ff082d21640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff07c102ef0 0x7ff07c19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.981+0000 7ff082d21640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff07c102570 0x7ff07c19ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.981+0000 7ff082d21640 1 --2- 172.21.15.33:0/2414016284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff07c100e20 0x7ff07c19a6c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:30.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.981+0000 7ff082d21640 1 -- 172.21.15.33:0/2414016284 >> 172.21.15.33:0/2414016284 conn(0x7ff07c0fc7d0 msgr2=0x7ff07c06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:30.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.981+0000 7ff082d21640 1 -- 172.21.15.33:0/2414016284 shutdown_connections 2024-09-16T00:29:30.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:30.981+0000 7ff082d21640 1 -- 172.21.15.33:0/2414016284 wait complete. 2024-09-16T00:29:31.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1034217 9.58 MB/sec execute 519 sec latency 51.284 ms 2024-09-16T00:29:31.742 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:31 smithi033 ceph-mon[31243]: pgmap v721: 129 pgs: 129 active+clean; 262 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 195 op/s 2024-09-16T00:29:31.742 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:31 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:29:31.742 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:31 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:29:31.770 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:29:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:31 smithi083 ceph-mon[35242]: pgmap v721: 129 pgs: 129 active+clean; 262 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 195 op/s 2024-09-16T00:29:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:31 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:29:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:31 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:29:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:31 smithi060 ceph-mon[35185]: pgmap v721: 129 pgs: 129 active+clean; 262 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 195 op/s 2024-09-16T00:29:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:31 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:29:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:31 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:29:32.048 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:29:32.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.601+0000 7fafbfcd6640 1 -- 172.21.15.33:0/3426047934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafb8100e20 msgr2=0x7fafb8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:32.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.601+0000 7fafbfcd6640 1 --2- 172.21.15.33:0/3426047934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafb8100e20 0x7fafb8101200 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009900 tx=0x7fafb402a480 comp rx=0 tx=0).stop 2024-09-16T00:29:32.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.601+0000 7fafbfcd6640 1 -- 172.21.15.33:0/3426047934 shutdown_connections 2024-09-16T00:29:32.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.601+0000 7fafbfcd6640 1 --2- 172.21.15.33:0/3426047934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafb8102ef0 0x7fafb810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.601+0000 7fafbfcd6640 1 --2- 172.21.15.33:0/3426047934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb8102570 0x7fafb81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.601+0000 7fafbfcd6640 1 --2- 172.21.15.33:0/3426047934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafb8100e20 0x7fafb8101200 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.601+0000 7fafbfcd6640 1 -- 172.21.15.33:0/3426047934 >> 172.21.15.33:0/3426047934 conn(0x7fafb80fc7d0 msgr2=0x7fafb80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:32.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.601+0000 7fafbfcd6640 1 -- 172.21.15.33:0/3426047934 shutdown_connections 2024-09-16T00:29:32.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.601+0000 7fafbfcd6640 1 -- 172.21.15.33:0/3426047934 wait complete. 2024-09-16T00:29:32.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbfcd6640 1 Processor -- start 2024-09-16T00:29:32.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbfcd6640 1 -- start start 2024-09-16T00:29:32.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbfcd6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafb8100e20 0x7fafb8103c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbfcd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafb8102570 0x7fafb8104160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbfcd6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb8102ef0 0x7fafb81046a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbfcd6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fafb8110fd0 con 0x7fafb8102570 2024-09-16T00:29:32.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbfcd6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fafb8110e50 con 0x7fafb8100e20 2024-09-16T00:29:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbfcd6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fafb8111150 con 0x7fafb8102ef0 2024-09-16T00:29:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbd271640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafb8102570 0x7fafb8104160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbd271640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafb8102570 0x7fafb8104160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44252/0 (socket says 172.21.15.33:44252) 2024-09-16T00:29:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbd271640 1 -- 172.21.15.33:0/2090178889 learned_addr learned my addr 172.21.15.33:0/2090178889 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbe273640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb8102ef0 0x7fafb81046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbda72640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafb8100e20 0x7fafb8103c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbd271640 1 -- 172.21.15.33:0/2090178889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb8102ef0 msgr2=0x7fafb81046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbd271640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb8102ef0 0x7fafb81046a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbd271640 1 -- 172.21.15.33:0/2090178889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafb8100e20 msgr2=0x7fafb8103c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbd271640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafb8100e20 0x7fafb8103c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.602+0000 7fafbd271640 1 -- 172.21.15.33:0/2090178889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb81a6a40 con 0x7fafb8102570 2024-09-16T00:29:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.603+0000 7fafbd271640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafb8102570 0x7fafb8104160 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fafa800e9f0 tx=0x7fafa800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.603+0000 7fafaeffd640 1 -- 172.21.15.33:0/2090178889 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa800cd80 con 0x7fafb8102570 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.603+0000 7fafaeffd640 1 -- 172.21.15.33:0/2090178889 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafa8005500 con 0x7fafb8102570 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.603+0000 7fafbfcd6640 1 -- 172.21.15.33:0/2090178889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb81a6d30 con 0x7fafb8102570 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.603+0000 7fafaeffd640 1 -- 172.21.15.33:0/2090178889 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8010690 con 0x7fafb8102570 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.603+0000 7fafbfcd6640 1 -- 172.21.15.33:0/2090178889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb81a7210 con 0x7fafb8102570 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafaeffd640 1 -- 172.21.15.33:0/2090178889 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fafa80046c0 con 0x7fafb8102570 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafc20af740 1 Processor -- start 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafc20af740 1 -- start start 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafc20af740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561df4a0 0x5618561df880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafc20af740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561dfdc0 0x5618561ea140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafc20af740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561ea680 0x5618561eca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafc20af740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5618561452e0 con 0x5618561df4a0 2024-09-16T00:29:32.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafc20af740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5618560ecce0 con 0x5618561ea680 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafc20af740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5618561412a0 con 0x5618561dfdc0 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafbe273640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561df4a0 0x5618561df880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafbe273640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561df4a0 0x5618561df880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44268/0 (socket says 172.21.15.33:44268) 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafbe273640 1 -- 172.21.15.33:0/3781843337 learned_addr learned my addr 172.21.15.33:0/3781843337 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafbda72640 1 --2- 172.21.15.33:0/3781843337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561dfdc0 0x5618561ea140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.606+0000 7fafbd271640 1 --2- 172.21.15.33:0/3781843337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561ea680 0x5618561eca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafbe273640 1 -- 172.21.15.33:0/3781843337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561dfdc0 msgr2=0x5618561ea140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafaeffd640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fafa40a32c0 0x7fafa40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafbe273640 1 --2- 172.21.15.33:0/3781843337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561dfdc0 0x5618561ea140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafbe273640 1 -- 172.21.15.33:0/3781843337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561ea680 msgr2=0x5618561eca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafbe273640 1 --2- 172.21.15.33:0/3781843337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561ea680 0x5618561eca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafbe273640 1 -- 172.21.15.33:0/3781843337 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618560d1440 con 0x5618561df4a0 2024-09-16T00:29:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafbda72640 1 --2- 172.21.15.33:0/3781843337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561dfdc0 0x5618561ea140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafaeffd640 1 -- 172.21.15.33:0/2090178889 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fafa80d0110 con 0x7fafb8102570 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafbe273640 1 --2- 172.21.15.33:0/3781843337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561df4a0 0x5618561df880 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009f50 tx=0x7fafb00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2888ceb7b9e6c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafbe273640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fafa40a32c0 0x7fafa40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafa3fff640 1 -- 172.21.15.33:0/3781843337 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb0004250 con 0x5618561df4a0 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafa3fff640 1 -- 172.21.15.33:0/3781843337 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb000fe30 con 0x5618561df4a0 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.607+0000 7fafa3fff640 1 -- 172.21.15.33:0/3781843337 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb001bc60 con 0x5618561df4a0 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.608+0000 7fafc20af740 1 -- 172.21.15.33:0/3781843337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561df4a0 msgr2=0x5618561df880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.608+0000 7fafc20af740 1 --2- 172.21.15.33:0/3781843337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561df4a0 0x5618561df880 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009f50 tx=0x7fafb00118d0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.608+0000 7fafbe273640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fafa40a32c0 0x7fafa40a5780 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fafb0012fd0 tx=0x7fafb0021000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.608+0000 7fafc20af740 1 -- 172.21.15.33:0/3781843337 shutdown_connections 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.608+0000 7fafc20af740 1 --2- 172.21.15.33:0/3781843337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561ea680 0x5618561eca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.608+0000 7fafc20af740 1 --2- 172.21.15.33:0/3781843337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561dfdc0 0x5618561ea140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.608+0000 7fafc20af740 1 --2- 172.21.15.33:0/3781843337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561df4a0 0x5618561df880 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.608+0000 7fafc20af740 1 -- 172.21.15.33:0/3781843337 >> 172.21.15.33:0/3781843337 conn(0x5618561dba30 msgr2=0x5618561dbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.608+0000 7fafc20af740 1 -- 172.21.15.33:0/3781843337 shutdown_connections 2024-09-16T00:29:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.608+0000 7fafc20af740 1 -- 172.21.15.33:0/3781843337 wait complete. 2024-09-16T00:29:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafc20af740 1 -- start start 2024-09-16T00:29:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafc20af740 1 Processor -- start 2024-09-16T00:29:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafc20af740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561df4a0 0x5618561e9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafc20af740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561e7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafc20af740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561ea680 0x5618561e7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafc20af740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5618561ed250 con 0x5618561dfdc0 2024-09-16T00:29:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafc20af740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561856100050 con 0x5618561ea680 2024-09-16T00:29:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafc20af740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5618561ed3d0 con 0x5618561df4a0 2024-09-16T00:29:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafc20af740 10 client.0 _ll_register_callbacks cb 0x561856140160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafbd271640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561e7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafbda72640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561df4a0 0x5618561e9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafbd271640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561e7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44270/0 (socket says 172.21.15.33:44270) 2024-09-16T00:29:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafbd271640 1 -- 172.21.15.33:0/4109370626 learned_addr learned my addr 172.21.15.33:0/4109370626 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.609+0000 7fafbe273640 1 --2- 172.21.15.33:0/4109370626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561ea680 0x5618561e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafbd271640 1 -- 172.21.15.33:0/4109370626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561df4a0 msgr2=0x5618561e9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafbd271640 1 --2- 172.21.15.33:0/4109370626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561df4a0 0x5618561e9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafbd271640 1 -- 172.21.15.33:0/4109370626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561ea680 msgr2=0x5618561e7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafbd271640 1 --2- 172.21.15.33:0/4109370626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561ea680 0x5618561e7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafbd271640 1 -- 172.21.15.33:0/4109370626 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561855eb63c0 con 0x5618561dfdc0 2024-09-16T00:29:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafbd271640 1 --2- 172.21.15.33:0/4109370626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561e7220 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fafa800c6a0 tx=0x7fafa809c080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5780bbf42b9dfb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafa17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:29:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafa17fa640 1 -- 172.21.15.33:0/4109370626 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa80a4c90 con 0x5618561dfdc0 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafa17fa640 1 -- 172.21.15.33:0/4109370626 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fafa80bc0c0 con 0x5618561dfdc0 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafc20af740 1 -- 172.21.15.33:0/4109370626 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafb000fe30 con 0x5618561dfdc0 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafa17fa640 1 -- 172.21.15.33:0/4109370626 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fafa809c9b0 con 0x5618561dfdc0 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafc20af740 10 client.17775 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafc20af740 1 -- 172.21.15.33:0/4109370626 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5618561f8bc0 con 0x5618561dfdc0 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafa17fa640 1 -- 172.21.15.33:0/4109370626 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb000fe30 con 0x5618561dfdc0 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.610+0000 7fafa17fa640 1 -- 172.21.15.33:0/4109370626 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fafa80a0150 con 0x5618561dfdc0 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.611+0000 7fafc20af740 10 client.17775 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.611+0000 7fafc20af740 10 client.17775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.611+0000 7fafc20af740 20 client.17775 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.611+0000 7fafc20af740 1 --2- 172.21.15.33:0/4109370626 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5618561fa110 0x5618561fc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.611+0000 7fafc20af740 4 client.17775 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.611+0000 7fafc20af740 1 -- 172.21.15.33:0/4109370626 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561856152850 con 0x5618561fa110 2024-09-16T00:29:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.611+0000 7fafbda72640 1 --2- 172.21.15.33:0/4109370626 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5618561fa110 0x5618561fc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.611+0000 7fafbda72640 1 --2- 172.21.15.33:0/4109370626 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5618561fa110 0x5618561fc500 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5856aef3e496cfbe server_cookie=2d4bd742e716416 in_seq=0 out_seq=0 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.611+0000 7fafa17fa640 10 client.17775 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafa17fa640 1 -- 172.21.15.33:0/4109370626 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561856152850 con 0x5618561fa110 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafa17fa640 10 client.17775 handle_command_reply: tid=0 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafc20af740 1 client.17775 shutdown 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafc20af740 1 -- 172.21.15.33:0/4109370626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 msgr2=0x5618561e7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafc20af740 1 --2- 172.21.15.33:0/4109370626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561e7220 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fafa800c6a0 tx=0x7fafa809c080 comp rx=0 tx=0).stop 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafc20af740 1 -- 172.21.15.33:0/4109370626 shutdown_connections 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafc20af740 1 --2- 172.21.15.33:0/4109370626 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5618561fa110 0x5618561fc500 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafc20af740 1 --2- 172.21.15.33:0/4109370626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561ea680 0x5618561e7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafc20af740 1 --2- 172.21.15.33:0/4109370626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561e7220 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafa17fa640 0 client.17775 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafc20af740 1 --2- 172.21.15.33:0/4109370626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561df4a0 0x5618561e9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafc20af740 1 -- 172.21.15.33:0/4109370626 >> 172.21.15.33:0/4109370626 conn(0x5618561dba30 msgr2=0x5618561e1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:32.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafc20af740 1 -- 172.21.15.33:0/4109370626 shutdown_connections 2024-09-16T00:29:32.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafc20af740 1 -- 172.21.15.33:0/4109370626 wait complete. 2024-09-16T00:29:32.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.612+0000 7fafc20af740 20 client.17775 trim_cache size 0 max 16384 2024-09-16T00:29:32.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.633+0000 7fafc20af740 1 Processor -- start 2024-09-16T00:29:32.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafc20af740 1 -- start start 2024-09-16T00:29:32.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafc20af740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561df4a0 0x5618561f64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafc20af740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561e2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafc20af740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561ea680 0x5618561e6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafc20af740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fafb81b3bc0 con 0x5618561dfdc0 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafc20af740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fafb81b1490 con 0x5618561df4a0 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafc20af740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fafb81af2e0 con 0x5618561ea680 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafbd271640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561e2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafbd271640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561e2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44284/0 (socket says 172.21.15.33:44284) 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafbda72640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561df4a0 0x5618561f64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafbd271640 1 -- 172.21.15.33:0/1836343544 learned_addr learned my addr 172.21.15.33:0/1836343544 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafbe273640 1 --2- 172.21.15.33:0/1836343544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561ea680 0x5618561e6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafbda72640 1 -- 172.21.15.33:0/1836343544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561ea680 msgr2=0x5618561e6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafbda72640 1 --2- 172.21.15.33:0/1836343544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561ea680 0x5618561e6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafbda72640 1 -- 172.21.15.33:0/1836343544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 msgr2=0x5618561e2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:32.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafbda72640 1 --2- 172.21.15.33:0/1836343544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561e2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafbda72640 1 -- 172.21.15.33:0/1836343544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618561fcac0 con 0x5618561df4a0 2024-09-16T00:29:32.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.634+0000 7fafbd271640 1 --2- 172.21.15.33:0/1836343544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561e2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:29:32.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.635+0000 7fafbda72640 1 --2- 172.21.15.33:0/1836343544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561df4a0 0x5618561f64b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fafb402aa70 tx=0x7fafb4004550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=542e8d99a7ebd7d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:32.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.635+0000 7fafa37fe640 1 -- 172.21.15.33:0/1836343544 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb4035040 con 0x5618561df4a0 2024-09-16T00:29:32.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.635+0000 7fafa37fe640 1 -- 172.21.15.33:0/1836343544 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb404b070 con 0x5618561df4a0 2024-09-16T00:29:32.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.635+0000 7fafa37fe640 1 -- 172.21.15.33:0/1836343544 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb40035f0 con 0x5618561df4a0 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.636+0000 7fafc20af740 1 -- 172.21.15.33:0/1836343544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561df4a0 msgr2=0x5618561f64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.636+0000 7fafc20af740 1 --2- 172.21.15.33:0/1836343544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561df4a0 0x5618561f64b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fafb402aa70 tx=0x7fafb4004550 comp rx=0 tx=0).stop 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.636+0000 7fafc20af740 1 -- 172.21.15.33:0/1836343544 shutdown_connections 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.636+0000 7fafc20af740 1 --2- 172.21.15.33:0/1836343544 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561ea680 0x5618561e6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.636+0000 7fafc20af740 1 --2- 172.21.15.33:0/1836343544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561e2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.636+0000 7fafc20af740 1 --2- 172.21.15.33:0/1836343544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561df4a0 0x5618561f64b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.636+0000 7fafc20af740 1 -- 172.21.15.33:0/1836343544 >> 172.21.15.33:0/1836343544 conn(0x5618561dba30 msgr2=0x5618561e5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.636+0000 7fafc20af740 1 -- 172.21.15.33:0/1836343544 shutdown_connections 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.636+0000 7fafc20af740 1 -- 172.21.15.33:0/1836343544 wait complete. 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.636+0000 7fafc20af740 1 -- start start 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.636+0000 7fafc20af740 1 Processor -- start 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafc20af740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561df4a0 0x5618561f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafc20af740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561ede10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafc20af740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561ea680 0x5618561ee350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafc20af740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fafb81af950 con 0x5618561dfdc0 2024-09-16T00:29:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafc20af740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fafb81b0dc0 con 0x5618561df4a0 2024-09-16T00:29:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafc20af740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fafb81b0020 con 0x5618561ea680 2024-09-16T00:29:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafc20af740 10 client.0 _ll_register_callbacks cb 0x7fafa8009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:29:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafbd271640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561ede10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafbd271640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561ede10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44292/0 (socket says 172.21.15.33:44292) 2024-09-16T00:29:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafbd271640 1 -- 172.21.15.33:0/1532612988 learned_addr learned my addr 172.21.15.33:0/1532612988 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:29:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafbe273640 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561ea680 0x5618561ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafbda72640 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561df4a0 0x5618561f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafbd271640 1 -- 172.21.15.33:0/1532612988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561ea680 msgr2=0x5618561ee350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafbd271640 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561ea680 0x5618561ee350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafbd271640 1 -- 172.21.15.33:0/1532612988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561df4a0 msgr2=0x5618561f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafbd271640 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561df4a0 0x5618561f2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafbd271640 1 -- 172.21.15.33:0/1532612988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618561ee8c0 con 0x5618561dfdc0 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafbd271640 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561ede10 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fafa80cdb00 tx=0x7fafa80a4d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda71f59575cac19 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafa17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafa17fa640 1 -- 172.21.15.33:0/1532612988 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa809bdf0 con 0x5618561dfdc0 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafa17fa640 1 -- 172.21.15.33:0/1532612988 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fafa80bc0b0 con 0x5618561dfdc0 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.637+0000 7fafc20af740 1 -- 172.21.15.33:0/1532612988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafb404b070 con 0x5618561dfdc0 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.638+0000 7fafa17fa640 1 -- 172.21.15.33:0/1532612988 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fafa80a6020 con 0x5618561dfdc0 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.638+0000 7fafc20af740 10 client.17781 fetch_fsmap learned FSMap version 30 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.638+0000 7fafc20af740 1 -- 172.21.15.33:0/1532612988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561856264a50 con 0x5618561dfdc0 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.638+0000 7fafa17fa640 1 -- 172.21.15.33:0/1532612988 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb404b070 con 0x5618561dfdc0 2024-09-16T00:29:32.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.638+0000 7fafa17fa640 1 -- 172.21.15.33:0/1532612988 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fafa809b170 con 0x5618561dfdc0 2024-09-16T00:29:32.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.638+0000 7fafc20af740 10 client.17781 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:29:32.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.638+0000 7fafc20af740 10 client.17781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:29:32.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.638+0000 7fafc20af740 20 client.17781 populate_metadata read hostname 'smithi033' 2024-09-16T00:29:32.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.638+0000 7fafc20af740 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561856265180 0x5618562675a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:29:32.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.638+0000 7fafc20af740 4 client.17781 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:29:32.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.638+0000 7fafc20af740 1 -- 172.21.15.33:0/1532612988 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56185614cd60 con 0x561856265180 2024-09-16T00:29:32.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.638+0000 7fafbda72640 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561856265180 0x5618562675a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:29:32.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafa17fa640 10 client.17781 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:32.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafbda72640 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561856265180 0x5618562675a0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f08bd82d1832f21 server_cookie=3609fbcfbabef4db in_seq=0 out_seq=0 2024-09-16T00:29:32.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafa17fa640 1 -- 172.21.15.33:0/1532612988 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56185614cd60 con 0x561856265180 2024-09-16T00:29:32.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafa17fa640 10 client.17781 handle_command_reply: tid=0 2024-09-16T00:29:32.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafc20af740 1 client.17781 shutdown 2024-09-16T00:29:32.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafc20af740 1 -- 172.21.15.33:0/1532612988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 msgr2=0x5618561ede10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:32.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafc20af740 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561ede10 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fafa80cdb00 tx=0x7fafa80a4d40 comp rx=0 tx=0).stop 2024-09-16T00:29:32.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafc20af740 1 -- 172.21.15.33:0/1532612988 shutdown_connections 2024-09-16T00:29:32.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafc20af740 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561856265180 0x5618562675a0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafc20af740 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618561ea680 0x5618561ee350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafc20af740 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618561dfdc0 0x5618561ede10 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafc20af740 1 --2- 172.21.15.33:0/1532612988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5618561df4a0 0x5618561f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafc20af740 1 -- 172.21.15.33:0/1532612988 >> 172.21.15.33:0/1532612988 conn(0x5618561dba30 msgr2=0x5618561e8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:32.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafa17fa640 0 client.17781 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:29:32.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafc20af740 1 -- 172.21.15.33:0/1532612988 shutdown_connections 2024-09-16T00:29:32.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.639+0000 7fafc20af740 1 -- 172.21.15.33:0/1532612988 wait complete. 2024-09-16T00:29:32.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafc20af740 20 client.17781 trim_cache size 0 max 16384 2024-09-16T00:29:32.645 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:29:32.645 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:29:32.645 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:29:32.645 INFO:teuthology.orchestra.run.smithi033.stdout: "c4fe5a15-c238-44f6-a6b7-b0b965870679": { 2024-09-16T00:29:32.645 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:29:32.645 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "c4fe5a15-c238-44f6-a6b7-b0b965870679", 2024-09-16T00:29:32.645 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:29:32.645 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:29:32.645 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:29:32.645 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:29:32.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafbfcd6640 1 -- 172.21.15.33:0/2090178889 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fafa40a32c0 msgr2=0x7fafa40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:32.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafbfcd6640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fafa40a32c0 0x7fafa40a5780 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fafb0012fd0 tx=0x7fafb0021000 comp rx=0 tx=0).stop 2024-09-16T00:29:32.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafbfcd6640 1 -- 172.21.15.33:0/2090178889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafb8102570 msgr2=0x7fafb8104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:29:32.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafbfcd6640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafb8102570 0x7fafb8104160 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fafa800e9f0 tx=0x7fafa800edf0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafbfcd6640 1 -- 172.21.15.33:0/2090178889 shutdown_connections 2024-09-16T00:29:32.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafbfcd6640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fafa40a32c0 0x7fafa40a5780 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafbfcd6640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb8102ef0 0x7fafb81046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafbfcd6640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafb8102570 0x7fafb8104160 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafbfcd6640 1 --2- 172.21.15.33:0/2090178889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafb8100e20 0x7fafb8103c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:29:32.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafbfcd6640 1 -- 172.21.15.33:0/2090178889 >> 172.21.15.33:0/2090178889 conn(0x7fafb80fc7d0 msgr2=0x7fafb8105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:29:32.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafbfcd6640 1 -- 172.21.15.33:0/2090178889 shutdown_connections 2024-09-16T00:29:32.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:29:32.640+0000 7fafbfcd6640 1 -- 172.21.15.33:0/2090178889 wait complete. 2024-09-16T00:29:32.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1037320 9.59 MB/sec execute 520 sec latency 15.636 ms 2024-09-16T00:29:32.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:32 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:29:32.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:32 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:29:32.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:32 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:29:32.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:32 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:29:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:32 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:29:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:32 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:29:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:32 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:29:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:32 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:29:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:32 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:29:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:32 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:29:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:32 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:29:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:32 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:29:33.341 INFO:tasks.cephfs.filesystem:scrub status for tag:c4fe5a15-c238-44f6-a6b7-b0b965870679 - {'path': '/', 'tag': 'c4fe5a15-c238-44f6-a6b7-b0b965870679', 'options': 'recursive,force'} 2024-09-16T00:29:33.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:33 smithi033 ceph-mon[31243]: pgmap v722: 129 pgs: 129 active+clean; 261 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.1 MiB/s wr, 213 op/s 2024-09-16T00:29:33.685 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1039710 9.59 MB/sec execute 521 sec latency 58.670 ms 2024-09-16T00:29:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:33 smithi083 ceph-mon[35242]: pgmap v722: 129 pgs: 129 active+clean; 261 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.1 MiB/s wr, 213 op/s 2024-09-16T00:29:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:33 smithi060 ceph-mon[35185]: pgmap v722: 129 pgs: 129 active+clean; 261 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.1 MiB/s wr, 213 op/s 2024-09-16T00:29:34.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1041863 9.60 MB/sec execute 522 sec latency 13.976 ms 2024-09-16T00:29:35.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1044874 9.61 MB/sec execute 523 sec latency 28.121 ms 2024-09-16T00:29:35.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:35 smithi033 ceph-mon[31243]: pgmap v723: 129 pgs: 129 active+clean; 262 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 216 op/s 2024-09-16T00:29:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:35 smithi083 ceph-mon[35242]: pgmap v723: 129 pgs: 129 active+clean; 262 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 216 op/s 2024-09-16T00:29:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:35 smithi060 ceph-mon[35185]: pgmap v723: 129 pgs: 129 active+clean; 262 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 216 op/s 2024-09-16T00:29:36.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1048088 9.63 MB/sec execute 524 sec latency 11.933 ms 2024-09-16T00:29:37.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1050681 9.63 MB/sec execute 525 sec latency 19.275 ms 2024-09-16T00:29:37.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:37 smithi033 ceph-mon[31243]: pgmap v724: 129 pgs: 129 active+clean; 267 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 215 op/s 2024-09-16T00:29:37.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:37 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:29:37.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:37 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:29:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:37 smithi083 ceph-mon[35242]: pgmap v724: 129 pgs: 129 active+clean; 267 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 215 op/s 2024-09-16T00:29:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:37 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:29:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:37 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:29:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:37 smithi060 ceph-mon[35185]: pgmap v724: 129 pgs: 129 active+clean; 267 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 215 op/s 2024-09-16T00:29:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:37 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:29:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:37 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:29:38.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1052577 9.64 MB/sec execute 526 sec latency 20.439 ms 2024-09-16T00:29:39.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1055602 9.65 MB/sec execute 527 sec latency 14.102 ms 2024-09-16T00:29:39.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:39 smithi033 ceph-mon[31243]: pgmap v725: 129 pgs: 129 active+clean; 267 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 201 op/s 2024-09-16T00:29:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:39 smithi083 ceph-mon[35242]: pgmap v725: 129 pgs: 129 active+clean; 267 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 201 op/s 2024-09-16T00:29:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:39 smithi060 ceph-mon[35185]: pgmap v725: 129 pgs: 129 active+clean; 267 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 201 op/s 2024-09-16T00:29:40.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1058739 9.67 MB/sec execute 528 sec latency 12.656 ms 2024-09-16T00:29:41.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1061084 9.66 MB/sec execute 529 sec latency 21.023 ms 2024-09-16T00:29:41.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:41 smithi033 ceph-mon[31243]: pgmap v726: 129 pgs: 129 active+clean; 268 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 208 op/s 2024-09-16T00:29:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:41 smithi083 ceph-mon[35242]: pgmap v726: 129 pgs: 129 active+clean; 268 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 208 op/s 2024-09-16T00:29:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:41 smithi060 ceph-mon[35185]: pgmap v726: 129 pgs: 129 active+clean; 268 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 208 op/s 2024-09-16T00:29:42.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1063220 9.67 MB/sec execute 530 sec latency 15.698 ms 2024-09-16T00:29:43.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1066088 9.69 MB/sec execute 531 sec latency 43.456 ms 2024-09-16T00:29:43.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:43 smithi033 ceph-mon[31243]: pgmap v727: 129 pgs: 129 active+clean; 273 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 220 op/s 2024-09-16T00:29:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:43 smithi083 ceph-mon[35242]: pgmap v727: 129 pgs: 129 active+clean; 273 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 220 op/s 2024-09-16T00:29:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:43 smithi060 ceph-mon[35185]: pgmap v727: 129 pgs: 129 active+clean; 273 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 220 op/s 2024-09-16T00:29:44.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1069245 9.70 MB/sec execute 532 sec latency 12.904 ms 2024-09-16T00:29:45.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1071707 9.70 MB/sec execute 533 sec latency 20.017 ms 2024-09-16T00:29:45.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:45 smithi033 ceph-mon[31243]: pgmap v728: 129 pgs: 129 active+clean; 274 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.5 MiB/s wr, 186 op/s 2024-09-16T00:29:45.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:45 smithi083 ceph-mon[35242]: pgmap v728: 129 pgs: 129 active+clean; 274 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.5 MiB/s wr, 186 op/s 2024-09-16T00:29:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:45 smithi060 ceph-mon[35185]: pgmap v728: 129 pgs: 129 active+clean; 274 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.5 MiB/s wr, 186 op/s 2024-09-16T00:29:46.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1074002 9.71 MB/sec execute 534 sec latency 11.789 ms 2024-09-16T00:29:47.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1076927 9.72 MB/sec execute 535 sec latency 9.982 ms 2024-09-16T00:29:47.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:47 smithi033 ceph-mon[31243]: pgmap v729: 129 pgs: 129 active+clean; 279 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.7 MiB/s wr, 243 op/s 2024-09-16T00:29:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:47 smithi083 ceph-mon[35242]: pgmap v729: 129 pgs: 129 active+clean; 279 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.7 MiB/s wr, 243 op/s 2024-09-16T00:29:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:47 smithi060 ceph-mon[35185]: pgmap v729: 129 pgs: 129 active+clean; 279 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.7 MiB/s wr, 243 op/s 2024-09-16T00:29:48.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1079856 9.73 MB/sec execute 536 sec latency 42.320 ms 2024-09-16T00:29:49.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1081850 9.73 MB/sec execute 537 sec latency 58.528 ms 2024-09-16T00:29:49.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:49 smithi033 ceph-mon[31243]: pgmap v730: 129 pgs: 129 active+clean; 279 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.3 MiB/s wr, 179 op/s 2024-09-16T00:29:49.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:49 smithi083 ceph-mon[35242]: pgmap v730: 129 pgs: 129 active+clean; 279 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.3 MiB/s wr, 179 op/s 2024-09-16T00:29:49.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:49 smithi060 ceph-mon[35185]: pgmap v730: 129 pgs: 129 active+clean; 279 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.3 MiB/s wr, 179 op/s 2024-09-16T00:29:50.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1084294 9.74 MB/sec execute 538 sec latency 15.633 ms 2024-09-16T00:29:51.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1087372 9.75 MB/sec execute 539 sec latency 14.973 ms 2024-09-16T00:29:51.782 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:51 smithi060 ceph-mon[35185]: pgmap v731: 129 pgs: 129 active+clean; 280 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-09-16T00:29:51.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:51 smithi033 ceph-mon[31243]: pgmap v731: 129 pgs: 129 active+clean; 280 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-09-16T00:29:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:51 smithi083 ceph-mon[35242]: pgmap v731: 129 pgs: 129 active+clean; 280 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-09-16T00:29:52.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1090499 9.77 MB/sec execute 540 sec latency 16.461 ms 2024-09-16T00:29:53.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1092356 9.76 MB/sec execute 541 sec latency 76.302 ms 2024-09-16T00:29:53.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:53 smithi033 ceph-mon[31243]: pgmap v732: 129 pgs: 129 active+clean; 284 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 243 op/s 2024-09-16T00:29:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:53 smithi083 ceph-mon[35242]: pgmap v732: 129 pgs: 129 active+clean; 284 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 243 op/s 2024-09-16T00:29:53.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:53 smithi060 ceph-mon[35185]: pgmap v732: 129 pgs: 129 active+clean; 284 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 243 op/s 2024-09-16T00:29:54.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1094730 9.77 MB/sec execute 542 sec latency 21.818 ms 2024-09-16T00:29:55.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1097739 9.79 MB/sec execute 543 sec latency 21.643 ms 2024-09-16T00:29:55.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:55 smithi033 ceph-mon[31243]: pgmap v733: 129 pgs: 129 active+clean; 288 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.1 MiB/s wr, 248 op/s 2024-09-16T00:29:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:55 smithi083 ceph-mon[35242]: pgmap v733: 129 pgs: 129 active+clean; 288 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.1 MiB/s wr, 248 op/s 2024-09-16T00:29:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:55 smithi060 ceph-mon[35185]: pgmap v733: 129 pgs: 129 active+clean; 288 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.1 MiB/s wr, 248 op/s 2024-09-16T00:29:56.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1100361 9.79 MB/sec execute 544 sec latency 11.627 ms 2024-09-16T00:29:57.686 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1102376 9.79 MB/sec execute 545 sec latency 18.916 ms 2024-09-16T00:29:57.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:57 smithi033 ceph-mon[31243]: pgmap v734: 129 pgs: 129 active+clean; 290 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.9 MiB/s wr, 283 op/s 2024-09-16T00:29:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:57 smithi083 ceph-mon[35242]: pgmap v734: 129 pgs: 129 active+clean; 290 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.9 MiB/s wr, 283 op/s 2024-09-16T00:29:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:57 smithi060 ceph-mon[35185]: pgmap v734: 129 pgs: 129 active+clean; 290 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.9 MiB/s wr, 283 op/s 2024-09-16T00:29:58.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1105060 9.80 MB/sec execute 546 sec latency 94.068 ms 2024-09-16T00:29:59.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1107143 9.80 MB/sec execute 547 sec latency 76.520 ms 2024-09-16T00:29:59.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:29:59 smithi033 ceph-mon[31243]: pgmap v735: 129 pgs: 129 active+clean; 290 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 215 op/s 2024-09-16T00:29:59.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:29:59 smithi083 ceph-mon[35242]: pgmap v735: 129 pgs: 129 active+clean; 290 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 215 op/s 2024-09-16T00:29:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:29:59 smithi060 ceph-mon[35185]: pgmap v735: 129 pgs: 129 active+clean; 290 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 215 op/s 2024-09-16T00:30:00.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1109024 9.80 MB/sec execute 548 sec latency 139.487 ms 2024-09-16T00:30:00.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:00 smithi033 ceph-mon[31243]: overall HEALTH_OK 2024-09-16T00:30:00.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:00 smithi083 ceph-mon[35242]: overall HEALTH_OK 2024-09-16T00:30:00.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:00 smithi060 ceph-mon[35185]: overall HEALTH_OK 2024-09-16T00:30:01.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1112085 9.82 MB/sec execute 549 sec latency 17.582 ms 2024-09-16T00:30:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:01 smithi060 ceph-mon[35185]: pgmap v736: 129 pgs: 129 active+clean; 291 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 234 op/s 2024-09-16T00:30:01.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:01 smithi033 ceph-mon[31243]: pgmap v736: 129 pgs: 129 active+clean; 291 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 234 op/s 2024-09-16T00:30:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:01 smithi083 ceph-mon[35242]: pgmap v736: 129 pgs: 129 active+clean; 291 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 234 op/s 2024-09-16T00:30:02.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1114787 9.83 MB/sec execute 550 sec latency 14.033 ms 2024-09-16T00:30:03.343 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:30:03.667 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:03.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1116482 9.82 MB/sec execute 551 sec latency 101.351 ms 2024-09-16T00:30:03.788 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:03 smithi033 ceph-mon[31243]: pgmap v737: 129 pgs: 129 active+clean; 281 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.4 MiB/s wr, 288 op/s 2024-09-16T00:30:03.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:03 smithi083 ceph-mon[35242]: pgmap v737: 129 pgs: 129 active+clean; 281 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.4 MiB/s wr, 288 op/s 2024-09-16T00:30:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:03 smithi060 ceph-mon[35185]: pgmap v737: 129 pgs: 129 active+clean; 281 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.4 MiB/s wr, 288 op/s 2024-09-16T00:30:04.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.202+0000 7f8cf6330640 1 -- 172.21.15.33:0/1234297518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf01082a0 msgr2=0x7f8cf010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:04.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.202+0000 7f8cf6330640 1 --2- 172.21.15.33:0/1234297518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf01082a0 0x7f8cf010ec40 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009cd0 tx=0x7f8ce802a8e0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.202+0000 7f8cf6330640 1 -- 172.21.15.33:0/1234297518 shutdown_connections 2024-09-16T00:30:04.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.202+0000 7f8cf6330640 1 --2- 172.21.15.33:0/1234297518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf01082a0 0x7f8cf010ec40 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.202+0000 7f8cf6330640 1 --2- 172.21.15.33:0/1234297518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf01036e0 0x7f8cf0107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.202+0000 7f8cf6330640 1 --2- 172.21.15.33:0/1234297518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cf0102d30 0x7f8cf0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.202+0000 7f8cf6330640 1 -- 172.21.15.33:0/1234297518 >> 172.21.15.33:0/1234297518 conn(0x7f8cf00fc7d0 msgr2=0x7f8cf00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:04.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.202+0000 7f8cf6330640 1 -- 172.21.15.33:0/1234297518 shutdown_connections 2024-09-16T00:30:04.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.202+0000 7f8cf6330640 1 -- 172.21.15.33:0/1234297518 wait complete. 2024-09-16T00:30:04.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.202+0000 7f8cf6330640 1 Processor -- start 2024-09-16T00:30:04.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.202+0000 7f8cf6330640 1 -- start start 2024-09-16T00:30:04.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8cf6330640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf0102d30 0x7f8cf0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8cf6330640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cf01036e0 0x7f8cf0198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8cf6330640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf01082a0 0x7f8cf019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8cf6330640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf01120a0 con 0x7f8cf0102d30 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8cf6330640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf0111f20 con 0x7f8cf01082a0 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8cf6330640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf0112220 con 0x7f8cf01036e0 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf0102d30 0x7f8cf0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf0102d30 0x7f8cf0198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48716/0 (socket says 172.21.15.33:48716) 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8ceffff640 1 -- 172.21.15.33:0/1086849760 learned_addr learned my addr 172.21.15.33:0/1086849760 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8cf48cd640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf01082a0 0x7f8cf019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8cef7fe640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cf01036e0 0x7f8cf0198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8ceffff640 1 -- 172.21.15.33:0/1086849760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cf01036e0 msgr2=0x7f8cf0198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8ceffff640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cf01036e0 0x7f8cf0198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8ceffff640 1 -- 172.21.15.33:0/1086849760 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf01082a0 msgr2=0x7f8cf019ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8ceffff640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf01082a0 0x7f8cf019ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.203+0000 7f8ceffff640 1 -- 172.21.15.33:0/1086849760 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf019d4e0 con 0x7f8cf0102d30 2024-09-16T00:30:04.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.204+0000 7f8ceffff640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf0102d30 0x7f8cf0198530 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f8ce000e9f0 tx=0x7f8ce000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.204+0000 7f8ced7fa640 1 -- 172.21.15.33:0/1086849760 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce000cd80 con 0x7f8cf0102d30 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.204+0000 7f8cf6330640 1 -- 172.21.15.33:0/1086849760 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf019d7d0 con 0x7f8cf0102d30 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.204+0000 7f8ced7fa640 1 -- 172.21.15.33:0/1086849760 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce0005500 con 0x7f8cf0102d30 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.204+0000 7f8ced7fa640 1 -- 172.21.15.33:0/1086849760 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0010690 con 0x7f8cf0102d30 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.204+0000 7f8cf6330640 1 -- 172.21.15.33:0/1086849760 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf01a50b0 con 0x7f8cf0102d30 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.206+0000 7f8cf8709740 1 Processor -- start 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8ced7fa640 1 -- 172.21.15.33:0/1086849760 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8ce0010830 con 0x7f8cf0102d30 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cf8709740 1 -- start start 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cf8709740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52a9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cf8709740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52a9c80 0x5568a52b8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cf8709740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 0x5568a52bae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cf8709740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5568a520f2e0 con 0x5568a52a9360 2024-09-16T00:30:04.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cf8709740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5568a51b6ce0 con 0x5568a52a9c80 2024-09-16T00:30:04.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cf8709740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5568a520b2a0 con 0x5568a52b8a90 2024-09-16T00:30:04.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52a9c80 0x5568a52b8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cef7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 0x5568a52bae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cf48cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52a9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cef7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 0x5568a52bae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:43688/0 (socket says 172.21.15.33:43688) 2024-09-16T00:30:04.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cef7fe640 1 -- 172.21.15.33:0/1023953225 learned_addr learned my addr 172.21.15.33:0/1023953225 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:04.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8ceffff640 1 --2- 172.21.15.33:0/1023953225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52a9c80 0x5568a52b8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42336/0 (socket says 172.21.15.33:42336) 2024-09-16T00:30:04.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8ced7fa640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8cd80a3240 0x7f8cd80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cef7fe640 1 -- 172.21.15.33:0/1023953225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52a9c80 msgr2=0x5568a52b8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:04.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cef7fe640 1 --2- 172.21.15.33:0/1023953225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52a9c80 0x5568a52b8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cef7fe640 1 -- 172.21.15.33:0/1023953225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 msgr2=0x5568a52a9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cef7fe640 1 --2- 172.21.15.33:0/1023953225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52a9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.207+0000 7f8cef7fe640 1 -- 172.21.15.33:0/1023953225 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568a519b440 con 0x5568a52b8a90 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.208+0000 7f8cf48cd640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8cd80a3240 0x7f8cd80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.208+0000 7f8ced7fa640 1 -- 172.21.15.33:0/1086849760 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8ce00d01d0 con 0x7f8cf0102d30 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.208+0000 7f8cef7fe640 1 --2- 172.21.15.33:0/1023953225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 0x5568a52bae80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009ba0 tx=0x7f8ce4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54684ed4467ee41b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.208+0000 7f8cd67fc640 1 -- 172.21.15.33:0/1023953225 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4004230 con 0x5568a52b8a90 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.208+0000 7f8cd67fc640 1 -- 172.21.15.33:0/1023953225 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce4009e10 con 0x5568a52b8a90 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.208+0000 7f8cd67fc640 1 -- 172.21.15.33:0/1023953225 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4004db0 con 0x5568a52b8a90 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.209+0000 7f8cf8709740 1 -- 172.21.15.33:0/1023953225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 msgr2=0x5568a52bae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.209+0000 7f8cf8709740 1 --2- 172.21.15.33:0/1023953225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 0x5568a52bae80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009ba0 tx=0x7f8ce4011810 comp rx=0 tx=0).stop 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.209+0000 7f8cf48cd640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8cd80a3240 0x7f8cd80a5700 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8004ab0 tx=0x7f8ce80095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.209+0000 7f8cf8709740 1 -- 172.21.15.33:0/1023953225 shutdown_connections 2024-09-16T00:30:04.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.209+0000 7f8cf8709740 1 --2- 172.21.15.33:0/1023953225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 0x5568a52bae80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.209+0000 7f8cf8709740 1 --2- 172.21.15.33:0/1023953225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52a9c80 0x5568a52b8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.209+0000 7f8cf8709740 1 --2- 172.21.15.33:0/1023953225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52a9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.209+0000 7f8cf8709740 1 -- 172.21.15.33:0/1023953225 >> 172.21.15.33:0/1023953225 conn(0x5568a52a5a30 msgr2=0x5568a52a5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.209+0000 7f8cf8709740 1 -- 172.21.15.33:0/1023953225 shutdown_connections 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.209+0000 7f8cf8709740 1 -- 172.21.15.33:0/1023953225 wait complete. 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.209+0000 7f8cf8709740 1 -- start start 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.209+0000 7f8cf8709740 1 Processor -- start 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8cf8709740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52b8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8cf8709740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52a9c80 0x5568a52b5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8cf8709740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52b8a90 0x5568a52b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8cf8709740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5568a52bb660 con 0x5568a52a9360 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8cf8709740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5568a51ca050 con 0x5568a52b8a90 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8cf8709740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5568a52bb7e0 con 0x5568a52a9c80 2024-09-16T00:30:04.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8cf8709740 10 client.0 _ll_register_callbacks cb 0x5568a520a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52b8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52b8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48732/0 (socket says 172.21.15.33:48732) 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8cf48cd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52b8a90 0x5568a52b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8ceffff640 1 -- 172.21.15.33:0/3721939272 learned_addr learned my addr 172.21.15.33:0/3721939272 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8cef7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52a9c80 0x5568a52b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8ceffff640 1 -- 172.21.15.33:0/3721939272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52a9c80 msgr2=0x5568a52b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8ceffff640 1 --2- 172.21.15.33:0/3721939272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52a9c80 0x5568a52b5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8ceffff640 1 -- 172.21.15.33:0/3721939272 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52b8a90 msgr2=0x5568a52b5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8ceffff640 1 --2- 172.21.15.33:0/3721939272 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52b8a90 0x5568a52b5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8ceffff640 1 -- 172.21.15.33:0/3721939272 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568a4f803c0 con 0x5568a52a9360 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8ceffff640 1 --2- 172.21.15.33:0/3721939272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52b8460 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f8ce000e9c0 tx=0x7f8ce00d38a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40ec09c98d78f37 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8ccf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:30:04.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8ccf7fe640 1 -- 172.21.15.33:0/3721939272 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce009beb0 con 0x5568a52a9360 2024-09-16T00:30:04.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8ccf7fe640 1 -- 172.21.15.33:0/3721939272 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8ce00a4b50 con 0x5568a52a9360 2024-09-16T00:30:04.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.210+0000 7f8cf8709740 1 -- 172.21.15.33:0/3721939272 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ce4009e10 con 0x5568a52a9360 2024-09-16T00:30:04.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.211+0000 7f8ccf7fe640 1 -- 172.21.15.33:0/3721939272 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8ce00d3e60 con 0x5568a52a9360 2024-09-16T00:30:04.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.211+0000 7f8cf8709740 10 client.17793 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:04.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.211+0000 7f8cf8709740 1 -- 172.21.15.33:0/3721939272 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5568a52c7130 con 0x5568a52a9360 2024-09-16T00:30:04.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.211+0000 7f8ccf7fe640 1 -- 172.21.15.33:0/3721939272 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce4009e10 con 0x5568a52a9360 2024-09-16T00:30:04.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.211+0000 7f8ccf7fe640 1 -- 172.21.15.33:0/3721939272 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f8ce009b7c0 con 0x5568a52a9360 2024-09-16T00:30:04.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.211+0000 7f8cf8709740 10 client.17793 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:04.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.211+0000 7f8cf8709740 10 client.17793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:30:04.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.211+0000 7f8cf8709740 20 client.17793 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:04.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.211+0000 7f8cf8709740 1 --2- 172.21.15.33:0/3721939272 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568a52c8540 0x5568a52ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.211+0000 7f8cf8709740 4 client.17793 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.211+0000 7f8cf8709740 1 -- 172.21.15.33:0/3721939272 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568a521c850 con 0x5568a52c8540 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.211+0000 7f8cef7fe640 1 --2- 172.21.15.33:0/3721939272 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568a52c8540 0x5568a52ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.212+0000 7f8ccf7fe640 10 client.17793 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.212+0000 7f8cef7fe640 1 --2- 172.21.15.33:0/3721939272 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568a52c8540 0x5568a52ca950 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e62296f53942c004 server_cookie=67df92316e71499a in_seq=0 out_seq=0 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.212+0000 7f8ccf7fe640 1 -- 172.21.15.33:0/3721939272 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5568a521c850 con 0x5568a52c8540 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.212+0000 7f8ccf7fe640 10 client.17793 handle_command_reply: tid=0 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8cf8709740 1 client.17793 shutdown 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8cf8709740 1 -- 172.21.15.33:0/3721939272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 msgr2=0x5568a52b8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8cf8709740 1 --2- 172.21.15.33:0/3721939272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52b8460 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f8ce000e9c0 tx=0x7f8ce00d38a0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8cf8709740 1 -- 172.21.15.33:0/3721939272 shutdown_connections 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8cf8709740 1 --2- 172.21.15.33:0/3721939272 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568a52c8540 0x5568a52ca950 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8cf8709740 1 --2- 172.21.15.33:0/3721939272 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52b8a90 0x5568a52b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8cf8709740 1 --2- 172.21.15.33:0/3721939272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52a9c80 0x5568a52b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8cf8709740 1 --2- 172.21.15.33:0/3721939272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52b8460 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8cf8709740 1 -- 172.21.15.33:0/3721939272 >> 172.21.15.33:0/3721939272 conn(0x5568a52a5a30 msgr2=0x5568a52ab790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:04.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8ccf7fe640 0 client.17793 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:04.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8cf8709740 1 -- 172.21.15.33:0/3721939272 shutdown_connections 2024-09-16T00:30:04.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8cf8709740 1 -- 172.21.15.33:0/3721939272 wait complete. 2024-09-16T00:30:04.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.213+0000 7f8cf8709740 20 client.17793 trim_cache size 0 max 16384 2024-09-16T00:30:04.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8cf8709740 1 Processor -- start 2024-09-16T00:30:04.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8cf8709740 1 -- start start 2024-09-16T00:30:04.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8cf8709740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52c4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8cf8709740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52a9c80 0x5568a52b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8cf8709740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52b8a90 0x5568a52a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8cf8709740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf01af7f0 con 0x5568a52a9360 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8cf8709740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf01ad0c0 con 0x5568a52b8a90 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8cf8709740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf01aaf10 con 0x5568a52a9c80 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52c4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48748/0 (socket says 172.21.15.33:48748) 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8ceffff640 1 -- 172.21.15.33:0/3790447871 learned_addr learned my addr 172.21.15.33:0/3790447871 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8cef7fe640 1 --2- 172.21.15.33:0/3790447871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52a9c80 0x5568a52b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8cf48cd640 1 --2- 172.21.15.33:0/3790447871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52b8a90 0x5568a52a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8ceffff640 1 -- 172.21.15.33:0/3790447871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52a9c80 msgr2=0x5568a52b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8ceffff640 1 --2- 172.21.15.33:0/3790447871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52a9c80 0x5568a52b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8ceffff640 1 -- 172.21.15.33:0/3790447871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52b8a90 msgr2=0x5568a52a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8ceffff640 1 --2- 172.21.15.33:0/3790447871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52b8a90 0x5568a52a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8ceffff640 1 -- 172.21.15.33:0/3790447871 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568a52b46c0 con 0x5568a52a9360 2024-09-16T00:30:04.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.231+0000 7f8ceffff640 1 --2- 172.21.15.33:0/3790447871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52c4900 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f8ce00d3910 tx=0x7f8ce00d0160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acccbe8638b0632c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.232+0000 7f8cd5ffb640 1 -- 172.21.15.33:0/3790447871 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce009bcd0 con 0x5568a52a9360 2024-09-16T00:30:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.232+0000 7f8cd5ffb640 1 -- 172.21.15.33:0/3790447871 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce009ca70 con 0x5568a52a9360 2024-09-16T00:30:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.232+0000 7f8cd5ffb640 1 -- 172.21.15.33:0/3790447871 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce00046c0 con 0x5568a52a9360 2024-09-16T00:30:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.232+0000 7f8cf8709740 1 -- 172.21.15.33:0/3790447871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 msgr2=0x5568a52c4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.232+0000 7f8cf8709740 1 --2- 172.21.15.33:0/3790447871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52c4900 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f8ce00d3910 tx=0x7f8ce00d0160 comp rx=0 tx=0).stop 2024-09-16T00:30:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.233+0000 7f8cf8709740 1 -- 172.21.15.33:0/3790447871 shutdown_connections 2024-09-16T00:30:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.233+0000 7f8cf8709740 1 --2- 172.21.15.33:0/3790447871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52b8a90 0x5568a52a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.233+0000 7f8cf8709740 1 --2- 172.21.15.33:0/3790447871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52a9c80 0x5568a52b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.233+0000 7f8cf8709740 1 --2- 172.21.15.33:0/3790447871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a52a9360 0x5568a52c4900 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.233+0000 7f8cf8709740 1 -- 172.21.15.33:0/3790447871 >> 172.21.15.33:0/3790447871 conn(0x5568a52a5a30 msgr2=0x5568a52a4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:04.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.233+0000 7f8cf8709740 1 -- 172.21.15.33:0/3790447871 shutdown_connections 2024-09-16T00:30:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.233+0000 7f8cf8709740 1 -- 172.21.15.33:0/3790447871 wait complete. 2024-09-16T00:30:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.233+0000 7f8cf8709740 1 -- start start 2024-09-16T00:30:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.233+0000 7f8cf8709740 1 Processor -- start 2024-09-16T00:30:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf8709740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52a9c80 0x5568a52bcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf8709740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 0x5568a52bd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf8709740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a5330230 0x5568a5332620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf8709740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf01ab580 con 0x5568a5330230 2024-09-16T00:30:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf8709740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf01ac9f0 con 0x5568a52a9c80 2024-09-16T00:30:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf8709740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf01abc50 con 0x5568a52b8a90 2024-09-16T00:30:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf8709740 10 client.0 _ll_register_callbacks cb 0x7f8ce00d39a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf48cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a5330230 0x5568a5332620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cef7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 0x5568a52bd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf48cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a5330230 0x5568a5332620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48750/0 (socket says 172.21.15.33:48750) 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cef7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 0x5568a52bd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:43720/0 (socket says 172.21.15.33:43720) 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf48cd640 1 -- 172.21.15.33:0/1409692939 learned_addr learned my addr 172.21.15.33:0/1409692939 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8ceffff640 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52a9c80 0x5568a52bcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf48cd640 1 -- 172.21.15.33:0/1409692939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 msgr2=0x5568a52bd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf48cd640 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 0x5568a52bd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf48cd640 1 -- 172.21.15.33:0/1409692939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52a9c80 msgr2=0x5568a52bcd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf48cd640 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52a9c80 0x5568a52bcd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf48cd640 1 -- 172.21.15.33:0/1409692939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568a5332b60 con 0x5568a5330230 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8ceffff640 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52a9c80 0x5568a52bcd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8cf48cd640 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a5330230 0x5568a5332620 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009cd0 tx=0x7f8ce802a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7131e58b412204d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8ccf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:30:04.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.234+0000 7f8ccf7fe640 1 -- 172.21.15.33:0/1409692939 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce80305a0 con 0x5568a5330230 2024-09-16T00:30:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8ccf7fe640 1 -- 172.21.15.33:0/1409692939 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8ce8030740 con 0x5568a5330230 2024-09-16T00:30:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8cf8709740 1 -- 172.21.15.33:0/1409692939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ce009ca70 con 0x5568a5330230 2024-09-16T00:30:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8ccf7fe640 1 -- 172.21.15.33:0/1409692939 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8ce8061050 con 0x5568a5330230 2024-09-16T00:30:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8cf8709740 10 client.17799 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8cf8709740 1 -- 172.21.15.33:0/1409692939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5568a5333d70 con 0x5568a5330230 2024-09-16T00:30:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8ccf7fe640 1 -- 172.21.15.33:0/1409692939 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce009ca70 con 0x5568a5330230 2024-09-16T00:30:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8ccf7fe640 1 -- 172.21.15.33:0/1409692939 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f8ce8030070 con 0x5568a5330230 2024-09-16T00:30:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8cf8709740 10 client.17799 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8cf8709740 10 client.17799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:30:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8cf8709740 20 client.17799 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8cf8709740 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568a5334540 0x5568a5336960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:04.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8cf8709740 4 client.17799 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8cf8709740 1 -- 172.21.15.33:0/1409692939 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5568a5216d60 con 0x5568a5334540 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.235+0000 7f8ceffff640 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568a5334540 0x5568a5336960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8ceffff640 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568a5334540 0x5568a5336960 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f486a6f79672aa7 server_cookie=94568890a8e7cef8 in_seq=0 out_seq=0 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8ccf7fe640 10 client.17799 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8ccf7fe640 1 -- 172.21.15.33:0/1409692939 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5568a5216d60 con 0x5568a5334540 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8ccf7fe640 10 client.17799 handle_command_reply: tid=0 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8cf8709740 1 client.17799 shutdown 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8cf8709740 1 -- 172.21.15.33:0/1409692939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a5330230 msgr2=0x5568a5332620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8cf8709740 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a5330230 0x5568a5332620 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009cd0 tx=0x7f8ce802a8e0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8cf8709740 1 -- 172.21.15.33:0/1409692939 shutdown_connections 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8cf8709740 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568a5334540 0x5568a5336960 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8cf8709740 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568a5330230 0x5568a5332620 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8cf8709740 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568a52b8a90 0x5568a52bd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8cf8709740 1 --2- 172.21.15.33:0/1409692939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568a52a9c80 0x5568a52bcd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8cf8709740 1 -- 172.21.15.33:0/1409692939 >> 172.21.15.33:0/1409692939 conn(0x5568a52a5a30 msgr2=0x5568a52a4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:04.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8ccf7fe640 0 client.17799 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:04.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8cf8709740 1 -- 172.21.15.33:0/1409692939 shutdown_connections 2024-09-16T00:30:04.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8cf8709740 1 -- 172.21.15.33:0/1409692939 wait complete. 2024-09-16T00:30:04.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.236+0000 7f8cf8709740 20 client.17799 trim_cache size 0 max 16384 2024-09-16T00:30:04.243 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:30:04.243 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:30:04.243 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:30:04.243 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:30:04.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.237+0000 7f8cf6330640 1 -- 172.21.15.33:0/1086849760 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8cd80a3240 msgr2=0x7f8cd80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:04.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.237+0000 7f8cf6330640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8cd80a3240 0x7f8cd80a5700 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8004ab0 tx=0x7f8ce80095c0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.237+0000 7f8cf6330640 1 -- 172.21.15.33:0/1086849760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf0102d30 msgr2=0x7f8cf0198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:04.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.237+0000 7f8cf6330640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf0102d30 0x7f8cf0198530 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f8ce000e9f0 tx=0x7f8ce000edf0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.237+0000 7f8cf6330640 1 -- 172.21.15.33:0/1086849760 shutdown_connections 2024-09-16T00:30:04.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.237+0000 7f8cf6330640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8cd80a3240 0x7f8cd80a5700 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.237+0000 7f8cf6330640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cf01082a0 0x7f8cf019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.237+0000 7f8cf6330640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cf01036e0 0x7f8cf0198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.237+0000 7f8cf6330640 1 --2- 172.21.15.33:0/1086849760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf0102d30 0x7f8cf0198530 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:04.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.237+0000 7f8cf6330640 1 -- 172.21.15.33:0/1086849760 >> 172.21.15.33:0/1086849760 conn(0x7f8cf00fc7d0 msgr2=0x7f8cf0105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:04.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.237+0000 7f8cf6330640 1 -- 172.21.15.33:0/1086849760 shutdown_connections 2024-09-16T00:30:04.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:04.237+0000 7f8cf6330640 1 -- 172.21.15.33:0/1086849760 wait complete. 2024-09-16T00:30:04.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1119274 9.84 MB/sec execute 552 sec latency 17.747 ms 2024-09-16T00:30:05.117 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:30:05.118 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:30:05.382 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:05.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1122244 9.84 MB/sec execute 553 sec latency 66.068 ms 2024-09-16T00:30:05.793 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:05 smithi033 ceph-mon[31243]: pgmap v738: 129 pgs: 129 active+clean; 272 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.5 MiB/s wr, 311 op/s 2024-09-16T00:30:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.876+0000 7fea7b487640 1 -- 172.21.15.33:0/2895233535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea74108290 msgr2=0x7fea7410ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.876+0000 7fea7b487640 1 --2- 172.21.15.33:0/2895233535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea74108290 0x7fea7410ec30 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009930 tx=0x7fea6c02a420 comp rx=0 tx=0).stop 2024-09-16T00:30:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.876+0000 7fea7b487640 1 -- 172.21.15.33:0/2895233535 shutdown_connections 2024-09-16T00:30:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.876+0000 7fea7b487640 1 --2- 172.21.15.33:0/2895233535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea74108290 0x7fea7410ec30 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.876+0000 7fea7b487640 1 --2- 172.21.15.33:0/2895233535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74103740 0x7fea74107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.876+0000 7fea7b487640 1 --2- 172.21.15.33:0/2895233535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea74102d90 0x7fea74103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.876+0000 7fea7b487640 1 -- 172.21.15.33:0/2895233535 >> 172.21.15.33:0/2895233535 conn(0x7fea740fc7d0 msgr2=0x7fea740febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.876+0000 7fea7b487640 1 -- 172.21.15.33:0/2895233535 shutdown_connections 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.876+0000 7fea7b487640 1 -- 172.21.15.33:0/2895233535 wait complete. 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea7b487640 1 Processor -- start 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea7b487640 1 -- start start 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea7b487640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74102d90 0x7fea74198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea7b487640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea74103740 0x7fea74198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea7b487640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea74108290 0x7fea7419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea7b487640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fea74112090 con 0x7fea74103740 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea7b487640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fea74111f10 con 0x7fea74102d90 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea7b487640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fea74112210 con 0x7fea74108290 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea78a22640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea74103740 0x7fea74198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea78a22640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea74103740 0x7fea74198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48778/0 (socket says 172.21.15.33:48778) 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea78a22640 1 -- 172.21.15.33:0/1143168616 learned_addr learned my addr 172.21.15.33:0/1143168616 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea79a24640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea74108290 0x7fea7419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.877+0000 7fea79223640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74102d90 0x7fea74198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.878+0000 7fea78a22640 1 -- 172.21.15.33:0/1143168616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea74108290 msgr2=0x7fea7419ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.878+0000 7fea78a22640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea74108290 0x7fea7419ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.878+0000 7fea78a22640 1 -- 172.21.15.33:0/1143168616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74102d90 msgr2=0x7fea74198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.878+0000 7fea78a22640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74102d90 0x7fea74198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.878+0000 7fea78a22640 1 -- 172.21.15.33:0/1143168616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7419d500 con 0x7fea74103740 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.878+0000 7fea78a22640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea74103740 0x7fea74198a90 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fea6400e9a0 tx=0x7fea6400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.878+0000 7fea6a7fc640 1 -- 172.21.15.33:0/1143168616 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6400cd80 con 0x7fea74103740 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.878+0000 7fea6a7fc640 1 -- 172.21.15.33:0/1143168616 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea640035f0 con 0x7fea74103740 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.878+0000 7fea7b487640 1 -- 172.21.15.33:0/1143168616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea7419d7f0 con 0x7fea74103740 2024-09-16T00:30:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.878+0000 7fea6a7fc640 1 -- 172.21.15.33:0/1143168616 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea64010670 con 0x7fea74103740 2024-09-16T00:30:05.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.878+0000 7fea7b487640 1 -- 172.21.15.33:0/1143168616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea741a50d0 con 0x7fea74103740 2024-09-16T00:30:05.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.879+0000 7fea7b487640 1 -- 172.21.15.33:0/1143168616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea74103170 con 0x7fea74103740 2024-09-16T00:30:05.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.880+0000 7fea6a7fc640 1 -- 172.21.15.33:0/1143168616 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fea640037a0 con 0x7fea74103740 2024-09-16T00:30:05.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.881+0000 7fea6a7fc640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fea600a32c0 0x7fea600a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:05.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.881+0000 7fea79223640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fea600a32c0 0x7fea600a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:05.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.881+0000 7fea6a7fc640 1 -- 172.21.15.33:0/1143168616 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fea640d0000 con 0x7fea74103740 2024-09-16T00:30:05.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.882+0000 7fea79223640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fea600a32c0 0x7fea600a5780 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fea70005fd0 tx=0x7fea70009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:05.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:05.884+0000 7fea6a7fc640 1 -- 172.21.15.33:0/1143168616 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fea6409c8d0 con 0x7fea74103740 2024-09-16T00:30:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:05 smithi083 ceph-mon[35242]: pgmap v738: 129 pgs: 129 active+clean; 272 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.5 MiB/s wr, 311 op/s 2024-09-16T00:30:05.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:05 smithi060 ceph-mon[35185]: pgmap v738: 129 pgs: 129 active+clean; 272 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.5 MiB/s wr, 311 op/s 2024-09-16T00:30:06.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.103+0000 7fea7b487640 1 -- 172.21.15.33:0/1143168616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea741998b0 con 0x7fea74103740 2024-09-16T00:30:06.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.103+0000 7fea6a7fc640 1 -- 172.21.15.33:0/1143168616 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7fea741998b0 con 0x7fea74103740 2024-09-16T00:30:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:30:06.105 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:30:06.105 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:30:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.106+0000 7fea7b487640 1 -- 172.21.15.33:0/1143168616 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fea600a32c0 msgr2=0x7fea600a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.106+0000 7fea7b487640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fea600a32c0 0x7fea600a5780 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fea70005fd0 tx=0x7fea70009500 comp rx=0 tx=0).stop 2024-09-16T00:30:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.106+0000 7fea7b487640 1 -- 172.21.15.33:0/1143168616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea74103740 msgr2=0x7fea74198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.106+0000 7fea7b487640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea74103740 0x7fea74198a90 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fea6400e9a0 tx=0x7fea6400ee60 comp rx=0 tx=0).stop 2024-09-16T00:30:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.106+0000 7fea7b487640 1 -- 172.21.15.33:0/1143168616 shutdown_connections 2024-09-16T00:30:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.106+0000 7fea7b487640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fea600a32c0 0x7fea600a5780 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.106+0000 7fea7b487640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea74108290 0x7fea7419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.106+0000 7fea7b487640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea74103740 0x7fea74198a90 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.106+0000 7fea7b487640 1 --2- 172.21.15.33:0/1143168616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74102d90 0x7fea74198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:06.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.106+0000 7fea7b487640 1 -- 172.21.15.33:0/1143168616 >> 172.21.15.33:0/1143168616 conn(0x7fea740fc7d0 msgr2=0x7fea74105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:06.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.106+0000 7fea7b487640 1 -- 172.21.15.33:0/1143168616 shutdown_connections 2024-09-16T00:30:06.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:06.106+0000 7fea7b487640 1 -- 172.21.15.33:0/1143168616 wait complete. 2024-09-16T00:30:06.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1124751 9.85 MB/sec execute 554 sec latency 12.332 ms 2024-09-16T00:30:06.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:06 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1143168616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:30:06.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:06 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1143168616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:30:06.937 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:30:06.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:06 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1143168616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:30:07.206 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:07.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:07 smithi033 ceph-mon[31243]: pgmap v739: 129 pgs: 129 active+clean; 273 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.7 MiB/s wr, 273 op/s 2024-09-16T00:30:07.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1126772 9.85 MB/sec execute 555 sec latency 27.833 ms 2024-09-16T00:30:07.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.749+0000 7ff33299f640 1 -- 172.21.15.33:0/1462813884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff32c102570 msgr2=0x7ff32c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:07.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.749+0000 7ff33299f640 1 --2- 172.21.15.33:0/1462813884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff32c102570 0x7ff32c1029b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff320009900 tx=0x7ff32002a480 comp rx=0 tx=0).stop 2024-09-16T00:30:07.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.749+0000 7ff33299f640 1 -- 172.21.15.33:0/1462813884 shutdown_connections 2024-09-16T00:30:07.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.749+0000 7ff33299f640 1 --2- 172.21.15.33:0/1462813884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff32c102ef0 0x7ff32c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.749+0000 7ff33299f640 1 --2- 172.21.15.33:0/1462813884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff32c102570 0x7ff32c1029b0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.749+0000 7ff33299f640 1 --2- 172.21.15.33:0/1462813884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff32c100e20 0x7ff32c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.749+0000 7ff33299f640 1 -- 172.21.15.33:0/1462813884 >> 172.21.15.33:0/1462813884 conn(0x7ff32c0fc7d0 msgr2=0x7ff32c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:07.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.749+0000 7ff33299f640 1 -- 172.21.15.33:0/1462813884 shutdown_connections 2024-09-16T00:30:07.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.749+0000 7ff33299f640 1 -- 172.21.15.33:0/1462813884 wait complete. 2024-09-16T00:30:07.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.750+0000 7ff33299f640 1 Processor -- start 2024-09-16T00:30:07.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.750+0000 7ff33299f640 1 -- start start 2024-09-16T00:30:07.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff33299f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff32c100e20 0x7ff32c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff33299f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff32c102570 0x7ff32c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff33299f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff32c102ef0 0x7ff32c19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff33299f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c110f80 con 0x7ff32c102ef0 2024-09-16T00:30:07.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff33299f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c110e00 con 0x7ff32c102570 2024-09-16T00:30:07.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff33299f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c111100 con 0x7ff32c100e20 2024-09-16T00:30:07.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff330f3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff32c102ef0 0x7ff32c19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff32b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff32c102570 0x7ff32c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff32bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff32c100e20 0x7ff32c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff32b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff32c102570 0x7ff32c198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42400/0 (socket says 172.21.15.33:42400) 2024-09-16T00:30:07.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff32b7fe640 1 -- 172.21.15.33:0/2905891873 learned_addr learned my addr 172.21.15.33:0/2905891873 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:07.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff32b7fe640 1 -- 172.21.15.33:0/2905891873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff32c100e20 msgr2=0x7ff32c1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:07.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff32b7fe640 1 --2- 172.21.15.33:0/2905891873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff32c100e20 0x7ff32c1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff32b7fe640 1 -- 172.21.15.33:0/2905891873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff32c102ef0 msgr2=0x7ff32c19ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:07.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff32b7fe640 1 --2- 172.21.15.33:0/2905891873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff32c102ef0 0x7ff32c19ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff32b7fe640 1 -- 172.21.15.33:0/2905891873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32c19d560 con 0x7ff32c102570 2024-09-16T00:30:07.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.751+0000 7ff32b7fe640 1 --2- 172.21.15.33:0/2905891873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff32c102570 0x7ff32c198af0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff320006010 tx=0x7ff320003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:07.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.752+0000 7ff3297fa640 1 -- 172.21.15.33:0/2905891873 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32003f070 con 0x7ff32c102570 2024-09-16T00:30:07.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.752+0000 7ff3297fa640 1 -- 172.21.15.33:0/2905891873 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff320031c30 con 0x7ff32c102570 2024-09-16T00:30:07.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.752+0000 7ff3297fa640 1 -- 172.21.15.33:0/2905891873 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff320035030 con 0x7ff32c102570 2024-09-16T00:30:07.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.752+0000 7ff33299f640 1 -- 172.21.15.33:0/2905891873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff32c19d7f0 con 0x7ff32c102570 2024-09-16T00:30:07.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.752+0000 7ff33299f640 1 -- 172.21.15.33:0/2905891873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff32c19dc80 con 0x7ff32c102570 2024-09-16T00:30:07.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.754+0000 7ff3297fa640 1 -- 172.21.15.33:0/2905891873 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff320042070 con 0x7ff32c102570 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.754+0000 7ff334d78740 1 Processor -- start 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.754+0000 7ff334d78740 1 -- start start 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff334d78740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26bb40 0x55a0cf26a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff334d78740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26a5d0 0x55a0cf26aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff334d78740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf275370 0x55a0cf277760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff334d78740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a0cf1d02f0 con 0x55a0cf26a5d0 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff334d78740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a0cf177cf0 con 0x55a0cf26bb40 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff334d78740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a0cf1cc2b0 con 0x55a0cf275370 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff330f3c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26bb40 0x55a0cf26a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff32bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26a5d0 0x55a0cf26aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff32bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26a5d0 0x55a0cf26aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48798/0 (socket says 172.21.15.33:48798) 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff32b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf275370 0x55a0cf277760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff330f3c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26bb40 0x55a0cf26a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42410/0 (socket says 172.21.15.33:42410) 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff32bfff640 1 -- 172.21.15.33:0/2884197461 learned_addr learned my addr 172.21.15.33:0/2884197461 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff32bfff640 1 -- 172.21.15.33:0/2884197461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf275370 msgr2=0x55a0cf277760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff32bfff640 1 --2- 172.21.15.33:0/2884197461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf275370 0x55a0cf277760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff32bfff640 1 -- 172.21.15.33:0/2884197461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26bb40 msgr2=0x55a0cf26a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff32bfff640 1 --2- 172.21.15.33:0/2884197461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26bb40 0x55a0cf26a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff32bfff640 1 -- 172.21.15.33:0/2884197461 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0cf15c3a0 con 0x55a0cf26a5d0 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff32bfff640 1 --2- 172.21.15.33:0/2884197461 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26a5d0 0x55a0cf26aa10 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009d20 tx=0x7ff31c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a32bfc2caf0dd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.755+0000 7ff3297fa640 1 --2- 172.21.15.33:0/2905891873 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff3140a32c0 0x7ff3140a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.756+0000 7ff3127fc640 1 -- 172.21.15.33:0/2884197461 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c01e070 con 0x55a0cf26a5d0 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.756+0000 7ff3127fc640 1 -- 172.21.15.33:0/2884197461 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff31c019b10 con 0x55a0cf26a5d0 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.756+0000 7ff3127fc640 1 -- 172.21.15.33:0/2884197461 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c00ebf0 con 0x55a0cf26a5d0 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.756+0000 7ff330f3c640 1 --2- 172.21.15.33:0/2905891873 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff3140a32c0 0x7ff3140a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.756+0000 7ff3297fa640 1 -- 172.21.15.33:0/2905891873 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff3200f02a0 con 0x7ff32c102570 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.756+0000 7ff334d78740 1 -- 172.21.15.33:0/2884197461 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26a5d0 msgr2=0x55a0cf26aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.756+0000 7ff330f3c640 1 --2- 172.21.15.33:0/2905891873 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff3140a32c0 0x7ff3140a5780 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff324005fd0 tx=0x7ff324009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.756+0000 7ff334d78740 1 --2- 172.21.15.33:0/2884197461 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26a5d0 0x55a0cf26aa10 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009d20 tx=0x7ff31c011810 comp rx=0 tx=0).stop 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.757+0000 7ff334d78740 1 -- 172.21.15.33:0/2884197461 shutdown_connections 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.757+0000 7ff334d78740 1 --2- 172.21.15.33:0/2884197461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf275370 0x55a0cf277760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.757+0000 7ff334d78740 1 --2- 172.21.15.33:0/2884197461 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26a5d0 0x55a0cf26aa10 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.757+0000 7ff334d78740 1 --2- 172.21.15.33:0/2884197461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26bb40 0x55a0cf26a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.757+0000 7ff334d78740 1 -- 172.21.15.33:0/2884197461 >> 172.21.15.33:0/2884197461 conn(0x55a0cf233310 msgr2=0x55a0cf266c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.757+0000 7ff334d78740 1 -- 172.21.15.33:0/2884197461 shutdown_connections 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.757+0000 7ff334d78740 1 -- 172.21.15.33:0/2884197461 wait complete. 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.757+0000 7ff334d78740 1 -- start start 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.757+0000 7ff334d78740 1 Processor -- start 2024-09-16T00:30:07.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff334d78740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26a5d0 0x55a0cf274d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff334d78740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26bb40 0x55a0cf271ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff334d78740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf275370 0x55a0cf2725e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff334d78740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a0cf278070 con 0x55a0cf275370 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff334d78740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a0cf18b060 con 0x55a0cf26a5d0 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff334d78740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a0cf2781f0 con 0x55a0cf26bb40 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff334d78740 10 client.0 _ll_register_callbacks cb 0x55a0ceede8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff32b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26bb40 0x55a0cf271ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff32b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26bb40 0x55a0cf271ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:43760/0 (socket says 172.21.15.33:43760) 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff32b7fe640 1 -- 172.21.15.33:0/1129975433 learned_addr learned my addr 172.21.15.33:0/1129975433 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff32bfff640 1 --2- 172.21.15.33:0/1129975433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26a5d0 0x55a0cf274d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff32b7fe640 1 -- 172.21.15.33:0/1129975433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26a5d0 msgr2=0x55a0cf274d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff32b7fe640 1 --2- 172.21.15.33:0/1129975433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26a5d0 0x55a0cf274d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff32b7fe640 1 -- 172.21.15.33:0/1129975433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf275370 msgr2=0x55a0cf2725e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:30:07.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff32b7fe640 1 --2- 172.21.15.33:0/1129975433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf275370 0x55a0cf2725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff32b7fe640 1 -- 172.21.15.33:0/1129975433 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0cef413c0 con 0x55a0cf26bb40 2024-09-16T00:30:07.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff32b7fe640 1 --2- 172.21.15.33:0/1129975433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26bb40 0x55a0cf271ff0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff3200b8e10 tx=0x7ff320005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51ab9e66edabd925 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:07.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.758+0000 7ff30b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:07.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff30b7fe640 1 -- 172.21.15.33:0/1129975433 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff320049050 con 0x55a0cf26bb40 2024-09-16T00:30:07.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff334d78740 1 -- 172.21.15.33:0/1129975433 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff31c019b10 con 0x55a0cf26bb40 2024-09-16T00:30:07.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff30b7fe640 1 -- 172.21.15.33:0/1129975433 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff3200c4c20 con 0x55a0cf26bb40 2024-09-16T00:30:07.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff30b7fe640 1 -- 172.21.15.33:0/1129975433 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff3200bc210 con 0x55a0cf26bb40 2024-09-16T00:30:07.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff334d78740 10 client.27187 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:07.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff334d78740 1 -- 172.21.15.33:0/1129975433 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a0cf2780c0 con 0x55a0cf26bb40 2024-09-16T00:30:07.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff30b7fe640 1 -- 172.21.15.33:0/1129975433 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff31c019b10 con 0x55a0cf26bb40 2024-09-16T00:30:07.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff30b7fe640 1 -- 172.21.15.33:0/1129975433 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff3200d5240 con 0x55a0cf26bb40 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff334d78740 10 client.27187 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff334d78740 10 client.27187 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff334d78740 20 client.27187 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff334d78740 1 --2- 172.21.15.33:0/1129975433 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a0cf284f30 0x55a0cf287390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff334d78740 4 client.27187 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff334d78740 1 -- 172.21.15.33:0/1129975433 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0cf1dd860 con 0x55a0cf284f30 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.759+0000 7ff32bfff640 1 --2- 172.21.15.33:0/1129975433 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a0cf284f30 0x55a0cf287390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.760+0000 7ff32bfff640 1 --2- 172.21.15.33:0/1129975433 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a0cf284f30 0x55a0cf287390 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3a12b138da182458 server_cookie=e7988e4802ebb882 in_seq=0 out_seq=0 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.760+0000 7ff30b7fe640 10 client.27187 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.760+0000 7ff30b7fe640 1 -- 172.21.15.33:0/1129975433 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a0cf1dd860 con 0x55a0cf284f30 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.760+0000 7ff30b7fe640 10 client.27187 handle_command_reply: tid=0 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.760+0000 7ff334d78740 1 client.27187 shutdown 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.761+0000 7ff334d78740 1 -- 172.21.15.33:0/1129975433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26bb40 msgr2=0x55a0cf271ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:07.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.761+0000 7ff334d78740 1 --2- 172.21.15.33:0/1129975433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26bb40 0x55a0cf271ff0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff3200b8e10 tx=0x7ff320005b50 comp rx=0 tx=0).stop 2024-09-16T00:30:07.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.761+0000 7ff334d78740 1 -- 172.21.15.33:0/1129975433 shutdown_connections 2024-09-16T00:30:07.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.761+0000 7ff334d78740 1 --2- 172.21.15.33:0/1129975433 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a0cf284f30 0x55a0cf287390 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.761+0000 7ff334d78740 1 --2- 172.21.15.33:0/1129975433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf275370 0x55a0cf2725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.761+0000 7ff334d78740 1 --2- 172.21.15.33:0/1129975433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26bb40 0x55a0cf271ff0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.761+0000 7ff334d78740 1 --2- 172.21.15.33:0/1129975433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26a5d0 0x55a0cf274d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.761+0000 7ff334d78740 1 -- 172.21.15.33:0/1129975433 >> 172.21.15.33:0/1129975433 conn(0x55a0cf233310 msgr2=0x55a0cf26c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:07.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.761+0000 7ff30b7fe640 0 client.27187 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:30:07.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.761+0000 7ff334d78740 1 -- 172.21.15.33:0/1129975433 shutdown_connections 2024-09-16T00:30:07.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.761+0000 7ff334d78740 1 -- 172.21.15.33:0/1129975433 wait complete. 2024-09-16T00:30:07.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.761+0000 7ff334d78740 20 client.27187 trim_cache size 0 max 16384 2024-09-16T00:30:07.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.776+0000 7ff334d78740 1 Processor -- start 2024-09-16T00:30:07.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.776+0000 7ff334d78740 1 -- start start 2024-09-16T00:30:07.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.776+0000 7ff334d78740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26a5d0 0x55a0cf281340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.776+0000 7ff334d78740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26bb40 0x55a0cf26b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.776+0000 7ff334d78740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf275370 0x55a0cf266700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.776+0000 7ff334d78740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1b1970 con 0x55a0cf275370 2024-09-16T00:30:07.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.776+0000 7ff334d78740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1af240 con 0x55a0cf26bb40 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.776+0000 7ff334d78740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1ad090 con 0x55a0cf26a5d0 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.777+0000 7ff330f3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf275370 0x55a0cf266700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.777+0000 7ff32bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26a5d0 0x55a0cf281340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.777+0000 7ff32bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26a5d0 0x55a0cf281340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:43776/0 (socket says 172.21.15.33:43776) 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.777+0000 7ff32bfff640 1 -- 172.21.15.33:0/3688292949 learned_addr learned my addr 172.21.15.33:0/3688292949 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.777+0000 7ff32b7fe640 1 --2- 172.21.15.33:0/3688292949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26bb40 0x55a0cf26b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.777+0000 7ff32bfff640 1 -- 172.21.15.33:0/3688292949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26bb40 msgr2=0x55a0cf26b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.777+0000 7ff32bfff640 1 --2- 172.21.15.33:0/3688292949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26bb40 0x55a0cf26b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.777+0000 7ff32bfff640 1 -- 172.21.15.33:0/3688292949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf275370 msgr2=0x55a0cf266700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.777+0000 7ff32bfff640 1 --2- 172.21.15.33:0/3688292949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf275370 0x55a0cf266700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.777+0000 7ff32bfff640 1 -- 172.21.15.33:0/3688292949 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0cf2750b0 con 0x55a0cf26a5d0 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.777+0000 7ff32b7fe640 1 --2- 172.21.15.33:0/3688292949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26bb40 0x55a0cf26b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.777+0000 7ff32bfff640 1 --2- 172.21.15.33:0/3688292949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26a5d0 0x55a0cf281340 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009c40 tx=0x7ff31c00e450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0970c3e56f14e7d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:07.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.778+0000 7ff311ffb640 1 -- 172.21.15.33:0/3688292949 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c01e040 con 0x55a0cf26a5d0 2024-09-16T00:30:07.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.778+0000 7ff311ffb640 1 -- 172.21.15.33:0/3688292949 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff31c029040 con 0x55a0cf26a5d0 2024-09-16T00:30:07.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.778+0000 7ff311ffb640 1 -- 172.21.15.33:0/3688292949 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c00e650 con 0x55a0cf26a5d0 2024-09-16T00:30:07.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.778+0000 7ff334d78740 1 -- 172.21.15.33:0/3688292949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26a5d0 msgr2=0x55a0cf281340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:07.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.778+0000 7ff334d78740 1 --2- 172.21.15.33:0/3688292949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26a5d0 0x55a0cf281340 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009c40 tx=0x7ff31c00e450 comp rx=0 tx=0).stop 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.778+0000 7ff334d78740 1 -- 172.21.15.33:0/3688292949 shutdown_connections 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.778+0000 7ff334d78740 1 --2- 172.21.15.33:0/3688292949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf275370 0x55a0cf266700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.778+0000 7ff334d78740 1 --2- 172.21.15.33:0/3688292949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26bb40 0x55a0cf26b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.778+0000 7ff334d78740 1 --2- 172.21.15.33:0/3688292949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf26a5d0 0x55a0cf281340 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.778+0000 7ff334d78740 1 -- 172.21.15.33:0/3688292949 >> 172.21.15.33:0/3688292949 conn(0x55a0cf233310 msgr2=0x55a0cf261a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.778+0000 7ff334d78740 1 -- 172.21.15.33:0/3688292949 shutdown_connections 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.778+0000 7ff334d78740 1 -- 172.21.15.33:0/3688292949 wait complete. 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff334d78740 1 -- start start 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff334d78740 1 Processor -- start 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff334d78740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26a5d0 0x55a0cf27dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff334d78740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26bb40 0x55a0cf26ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff334d78740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf275370 0x55a0cf26f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff334d78740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1ad700 con 0x55a0cf26bb40 2024-09-16T00:30:07.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff334d78740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1aeb70 con 0x55a0cf26a5d0 2024-09-16T00:30:07.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff334d78740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1addd0 con 0x55a0cf275370 2024-09-16T00:30:07.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff334d78740 10 client.0 _ll_register_callbacks cb 0x7ff3200f3a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:07.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff32b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26bb40 0x55a0cf26ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff32b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26bb40 0x55a0cf26ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48830/0 (socket says 172.21.15.33:48830) 2024-09-16T00:30:07.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff330f3c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf275370 0x55a0cf26f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.779+0000 7ff32b7fe640 1 -- 172.21.15.33:0/72953627 learned_addr learned my addr 172.21.15.33:0/72953627 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:07.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.780+0000 7ff32bfff640 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26a5d0 0x55a0cf27dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.780+0000 7ff330f3c640 1 -- 172.21.15.33:0/72953627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26a5d0 msgr2=0x55a0cf27dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:07.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.780+0000 7ff330f3c640 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26a5d0 0x55a0cf27dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.780+0000 7ff330f3c640 1 -- 172.21.15.33:0/72953627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26bb40 msgr2=0x55a0cf26ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:07.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.780+0000 7ff330f3c640 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26bb40 0x55a0cf26ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.780+0000 7ff330f3c640 1 -- 172.21.15.33:0/72953627 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0cf26faf0 con 0x55a0cf275370 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.780+0000 7ff32b7fe640 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26bb40 0x55a0cf26ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.780+0000 7ff330f3c640 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf275370 0x55a0cf26f2c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff324009fd0 tx=0x7ff324004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bf47dff112843d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.780+0000 7ff30b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.780+0000 7ff30b7fe640 1 -- 172.21.15.33:0/72953627 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32402f060 con 0x55a0cf275370 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.780+0000 7ff30b7fe640 1 -- 172.21.15.33:0/72953627 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff32401d910 con 0x55a0cf275370 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.780+0000 7ff334d78740 1 -- 172.21.15.33:0/72953627 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff31c029040 con 0x55a0cf275370 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.781+0000 7ff30b7fe640 1 -- 172.21.15.33:0/72953627 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff32401db10 con 0x55a0cf275370 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.781+0000 7ff334d78740 10 client.27193 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.781+0000 7ff334d78740 1 -- 172.21.15.33:0/72953627 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a0cf2efd60 con 0x55a0cf275370 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.781+0000 7ff30b7fe640 1 -- 172.21.15.33:0/72953627 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff31c029040 con 0x55a0cf275370 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.781+0000 7ff30b7fe640 1 -- 172.21.15.33:0/72953627 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7ff32401c3a0 con 0x55a0cf275370 2024-09-16T00:30:07.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.781+0000 7ff334d78740 10 client.27193 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:07.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.781+0000 7ff334d78740 10 client.27193 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:30:07.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.781+0000 7ff334d78740 20 client.27193 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:07.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.781+0000 7ff334d78740 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a0cf2f0480 0x55a0cf2f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:07.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.781+0000 7ff334d78740 4 client.27193 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:30:07.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.781+0000 7ff334d78740 1 -- 172.21.15.33:0/72953627 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a0cf1d7d70 con 0x55a0cf2f0480 2024-09-16T00:30:07.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.781+0000 7ff32bfff640 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a0cf2f0480 0x55a0cf2f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:07.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.782+0000 7ff30b7fe640 10 client.27193 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:30:07.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.782+0000 7ff32bfff640 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a0cf2f0480 0x55a0cf2f28a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=44b96a1abee58840 server_cookie=851f145d62698c6 in_seq=0 out_seq=0 2024-09-16T00:30:07.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.782+0000 7ff30b7fe640 1 -- 172.21.15.33:0/72953627 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a0cf1d7d70 con 0x55a0cf2f0480 2024-09-16T00:30:07.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.782+0000 7ff30b7fe640 10 client.27193 handle_command_reply: tid=0 2024-09-16T00:30:07.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.782+0000 7ff334d78740 1 client.27193 shutdown 2024-09-16T00:30:07.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.782+0000 7ff334d78740 1 -- 172.21.15.33:0/72953627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf275370 msgr2=0x55a0cf26f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:07.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.782+0000 7ff334d78740 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf275370 0x55a0cf26f2c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff324009fd0 tx=0x7ff324004ab0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.782+0000 7ff334d78740 1 -- 172.21.15.33:0/72953627 shutdown_connections 2024-09-16T00:30:07.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.782+0000 7ff334d78740 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a0cf2f0480 0x55a0cf2f28a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.783+0000 7ff334d78740 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0cf275370 0x55a0cf26f2c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.783+0000 7ff334d78740 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0cf26bb40 0x55a0cf26ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.783+0000 7ff30b7fe640 0 client.27193 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:30:07.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.783+0000 7ff334d78740 1 --2- 172.21.15.33:0/72953627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a0cf26a5d0 0x55a0cf27dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.783+0000 7ff334d78740 1 -- 172.21.15.33:0/72953627 >> 172.21.15.33:0/72953627 conn(0x55a0cf233310 msgr2=0x55a0cf266700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:07.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.783+0000 7ff334d78740 1 -- 172.21.15.33:0/72953627 shutdown_connections 2024-09-16T00:30:07.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.783+0000 7ff334d78740 1 -- 172.21.15.33:0/72953627 wait complete. 2024-09-16T00:30:07.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.783+0000 7ff334d78740 20 client.27193 trim_cache size 0 max 16384 2024-09-16T00:30:07.789 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:30:07.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.783+0000 7ff33299f640 1 -- 172.21.15.33:0/2905891873 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff3140a32c0 msgr2=0x7ff3140a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:07.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.783+0000 7ff33299f640 1 --2- 172.21.15.33:0/2905891873 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff3140a32c0 0x7ff3140a5780 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff324005fd0 tx=0x7ff324009500 comp rx=0 tx=0).stop 2024-09-16T00:30:07.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.783+0000 7ff33299f640 1 -- 172.21.15.33:0/2905891873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff32c102570 msgr2=0x7ff32c198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:07.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.783+0000 7ff33299f640 1 --2- 172.21.15.33:0/2905891873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff32c102570 0x7ff32c198af0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff320006010 tx=0x7ff320003c10 comp rx=0 tx=0).stop 2024-09-16T00:30:07.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.784+0000 7ff33299f640 1 -- 172.21.15.33:0/2905891873 shutdown_connections 2024-09-16T00:30:07.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.784+0000 7ff33299f640 1 --2- 172.21.15.33:0/2905891873 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff3140a32c0 0x7ff3140a5780 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.784+0000 7ff33299f640 1 --2- 172.21.15.33:0/2905891873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff32c102ef0 0x7ff32c19ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.784+0000 7ff33299f640 1 --2- 172.21.15.33:0/2905891873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff32c102570 0x7ff32c198af0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.784+0000 7ff33299f640 1 --2- 172.21.15.33:0/2905891873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff32c100e20 0x7ff32c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:07.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.784+0000 7ff33299f640 1 -- 172.21.15.33:0/2905891873 >> 172.21.15.33:0/2905891873 conn(0x7ff32c0fc7d0 msgr2=0x7ff32c06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:07.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.784+0000 7ff33299f640 1 -- 172.21.15.33:0/2905891873 shutdown_connections 2024-09-16T00:30:07.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:07.784+0000 7ff33299f640 1 -- 172.21.15.33:0/2905891873 wait complete. 2024-09-16T00:30:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:07 smithi083 ceph-mon[35242]: pgmap v739: 129 pgs: 129 active+clean; 273 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.7 MiB/s wr, 273 op/s 2024-09-16T00:30:07.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:07 smithi060 ceph-mon[35185]: pgmap v739: 129 pgs: 129 active+clean; 273 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.7 MiB/s wr, 273 op/s 2024-09-16T00:30:08.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1129616 9.87 MB/sec execute 556 sec latency 62.663 ms 2024-09-16T00:30:09.325 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:30:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:09.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1131410 9.86 MB/sec execute 557 sec latency 16.122 ms 2024-09-16T00:30:09.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:09 smithi033 ceph-mon[31243]: pgmap v740: 129 pgs: 129 active+clean; 274 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 231 op/s 2024-09-16T00:30:09.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:09 smithi083 ceph-mon[35242]: pgmap v740: 129 pgs: 129 active+clean; 274 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 231 op/s 2024-09-16T00:30:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:09 smithi060 ceph-mon[35185]: pgmap v740: 129 pgs: 129 active+clean; 274 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 231 op/s 2024-09-16T00:30:10.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.096+0000 7f9663a9a640 1 -- 172.21.15.33:0/1485554684 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f965c102dd0 msgr2=0x7f965c1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:10.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.096+0000 7f9663a9a640 1 --2- 172.21.15.33:0/1485554684 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f965c102dd0 0x7f965c1031b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f9658009930 tx=0x7f965802a420 comp rx=0 tx=0).stop 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.096+0000 7f9663a9a640 1 -- 172.21.15.33:0/1485554684 shutdown_connections 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.096+0000 7f9663a9a640 1 --2- 172.21.15.33:0/1485554684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f965c108300 0x7f965c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.096+0000 7f9663a9a640 1 --2- 172.21.15.33:0/1485554684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f965c103780 0x7f965c107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.096+0000 7f9663a9a640 1 --2- 172.21.15.33:0/1485554684 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f965c102dd0 0x7f965c1031b0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.096+0000 7f9663a9a640 1 -- 172.21.15.33:0/1485554684 >> 172.21.15.33:0/1485554684 conn(0x7f965c0fc7d0 msgr2=0x7f965c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.096+0000 7f9663a9a640 1 -- 172.21.15.33:0/1485554684 shutdown_connections 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.096+0000 7f9663a9a640 1 -- 172.21.15.33:0/1485554684 wait complete. 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9663a9a640 1 Processor -- start 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9663a9a640 1 -- start start 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9663a9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f965c102dd0 0x7f965c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9663a9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f965c103780 0x7f965c198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9663a9a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f965c108300 0x7f965c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9663a9a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1120b0 con 0x7f965c102dd0 2024-09-16T00:30:10.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9663a9a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f965c111f30 con 0x7f965c108300 2024-09-16T00:30:10.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9663a9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f965c112230 con 0x7f965c103780 2024-09-16T00:30:10.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9661836640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f965c102dd0 0x7f965c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9661836640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f965c102dd0 0x7f965c198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48856/0 (socket says 172.21.15.33:48856) 2024-09-16T00:30:10.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9661836640 1 -- 172.21.15.33:0/709748131 learned_addr learned my addr 172.21.15.33:0/709748131 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:10.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9661035640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f965c103780 0x7f965c198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9662037640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f965c108300 0x7f965c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9661836640 1 -- 172.21.15.33:0/709748131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f965c103780 msgr2=0x7f965c198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:10.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.097+0000 7f9661836640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f965c103780 0x7f965c198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.098+0000 7f9661836640 1 -- 172.21.15.33:0/709748131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f965c108300 msgr2=0x7f965c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:10.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.098+0000 7f9661836640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f965c108300 0x7f965c19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.098+0000 7f9661836640 1 -- 172.21.15.33:0/709748131 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c19d620 con 0x7f965c102dd0 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.098+0000 7f9661836640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f965c102dd0 0x7f965c198670 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f9658009230 tx=0x7f9658004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.098+0000 7f9652ffd640 1 -- 172.21.15.33:0/709748131 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965803f030 con 0x7f965c102dd0 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.098+0000 7f9652ffd640 1 -- 172.21.15.33:0/709748131 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9658004440 con 0x7f965c102dd0 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.098+0000 7f9663a9a640 1 -- 172.21.15.33:0/709748131 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965c19d8b0 con 0x7f965c102dd0 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.098+0000 7f9652ffd640 1 -- 172.21.15.33:0/709748131 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658035070 con 0x7f965c102dd0 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.098+0000 7f9663a9a640 1 -- 172.21.15.33:0/709748131 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f965c1a51f0 con 0x7f965c102dd0 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.101+0000 7f9665e73740 1 Processor -- start 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.101+0000 7f9665e73740 1 -- start start 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9665e73740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e98b40 0x55d265e97090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9652ffd640 1 -- 172.21.15.33:0/709748131 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9658042030 con 0x7f965c102dd0 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9665e73740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e975d0 0x55d265e97a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9665e73740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265ea4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9665e73740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d265dfd2f0 con 0x55d265e98b40 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9665e73740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d265da4cf0 con 0x55d265e975d0 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9665e73740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d265df92b0 con 0x55d265ea2370 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9662037640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e98b40 0x55d265e97090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9661836640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e975d0 0x55d265e97a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9661035640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265ea4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9662037640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e98b40 0x55d265e97090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48860/0 (socket says 172.21.15.33:48860) 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9661836640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e975d0 0x55d265e97a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42454/0 (socket says 172.21.15.33:42454) 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9662037640 1 -- 172.21.15.33:0/267662841 learned_addr learned my addr 172.21.15.33:0/267662841 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9661035640 1 -- 172.21.15.33:0/267662841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e975d0 msgr2=0x55d265e97a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9661035640 1 --2- 172.21.15.33:0/267662841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e975d0 0x55d265e97a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9661035640 1 -- 172.21.15.33:0/267662841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e98b40 msgr2=0x55d265e97090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:10.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9661035640 1 --2- 172.21.15.33:0/267662841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e98b40 0x55d265e97090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9661035640 1 -- 172.21.15.33:0/267662841 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d265d893a0 con 0x55d265ea2370 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.102+0000 7f9661035640 1 --2- 172.21.15.33:0/267662841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265ea4760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f965c06a2d0 tx=0x7f964c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c450fbea30a570 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.103+0000 7f9652ffd640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f96480a32c0 0x7f96480a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.103+0000 7f9647fff640 1 -- 172.21.15.33:0/267662841 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c004050 con 0x55d265ea2370 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.103+0000 7f9662037640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f96480a32c0 0x7f96480a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.103+0000 7f9652ffd640 1 -- 172.21.15.33:0/709748131 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f96580f0060 con 0x7f965c102dd0 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.103+0000 7f9647fff640 1 -- 172.21.15.33:0/267662841 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f964c005500 con 0x55d265ea2370 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.103+0000 7f9647fff640 1 -- 172.21.15.33:0/267662841 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c004d90 con 0x55d265ea2370 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.104+0000 7f9665e73740 1 -- 172.21.15.33:0/267662841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 msgr2=0x55d265ea4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.104+0000 7f9662037640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f96480a32c0 0x7f96480a5780 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9654011630 tx=0x7f9654009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.104+0000 7f9665e73740 1 --2- 172.21.15.33:0/267662841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265ea4760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f965c06a2d0 tx=0x7f964c011810 comp rx=0 tx=0).stop 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.104+0000 7f9665e73740 1 -- 172.21.15.33:0/267662841 shutdown_connections 2024-09-16T00:30:10.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.104+0000 7f9665e73740 1 --2- 172.21.15.33:0/267662841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265ea4760 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.104+0000 7f9665e73740 1 --2- 172.21.15.33:0/267662841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e975d0 0x55d265e97a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.104+0000 7f9665e73740 1 --2- 172.21.15.33:0/267662841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e98b40 0x55d265e97090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.104+0000 7f9665e73740 1 -- 172.21.15.33:0/267662841 >> 172.21.15.33:0/267662841 conn(0x55d265e60310 msgr2=0x55d265e93c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.104+0000 7f9665e73740 1 -- 172.21.15.33:0/267662841 shutdown_connections 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.104+0000 7f9665e73740 1 -- 172.21.15.33:0/267662841 wait complete. 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.104+0000 7f9665e73740 1 -- start start 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.104+0000 7f9665e73740 1 Processor -- start 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9665e73740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e975d0 0x55d265ea1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9665e73740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265e9efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9665e73740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265e9f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9665e73740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d265ea50c0 con 0x55d265e975d0 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9665e73740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d265db8060 con 0x55d265e98b40 2024-09-16T00:30:10.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9665e73740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d265ea5240 con 0x55d265ea2370 2024-09-16T00:30:10.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9665e73740 10 client.0 _ll_register_callbacks cb 0x55d265b0b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:10.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9661836640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e975d0 0x55d265ea1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9661836640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e975d0 0x55d265ea1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48868/0 (socket says 172.21.15.33:48868) 2024-09-16T00:30:10.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9661836640 1 -- 172.21.15.33:0/3655516043 learned_addr learned my addr 172.21.15.33:0/3655516043 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:10.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9662037640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265e9f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9661035640 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265e9efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9662037640 1 -- 172.21.15.33:0/3655516043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 msgr2=0x55d265e9efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:10.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9662037640 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265e9efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9662037640 1 -- 172.21.15.33:0/3655516043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e975d0 msgr2=0x55d265ea1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:10.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9662037640 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e975d0 0x55d265ea1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9662037640 1 -- 172.21.15.33:0/3655516043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d265b6e3c0 con 0x55d265ea2370 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9661836640 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e975d0 0x55d265ea1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9661035640 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265e9efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f9662037640 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265e9f520 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9654011ec0 tx=0x7f9654004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4957f9c3387c5c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.105+0000 7f96457fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f96457fa640 1 -- 172.21.15.33:0/3655516043 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965400f8f0 con 0x55d265ea2370 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f96457fa640 1 -- 172.21.15.33:0/3655516043 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f965400fa90 con 0x55d265ea2370 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f9665e73740 1 -- 172.21.15.33:0/3655516043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f964c005500 con 0x55d265ea2370 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f96457fa640 1 -- 172.21.15.33:0/3655516043 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f965400fc90 con 0x55d265ea2370 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f9665e73740 10 client.27205 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f9665e73740 1 -- 172.21.15.33:0/3655516043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d265eb0b50 con 0x55d265ea2370 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f96457fa640 1 -- 172.21.15.33:0/3655516043 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f964c005500 con 0x55d265ea2370 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f96457fa640 1 -- 172.21.15.33:0/3655516043 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f9654026070 con 0x55d265ea2370 2024-09-16T00:30:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f9665e73740 10 client.27205 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:10.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f9665e73740 10 client.27205 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:30:10.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f9665e73740 20 client.27205 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:10.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f9665e73740 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55d265eb2010 0x55d265eb44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f9665e73740 4 client.27205 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:30:10.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.106+0000 7f9665e73740 1 -- 172.21.15.33:0/3655516043 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d265e0a860 con 0x55d265eb2010 2024-09-16T00:30:10.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.107+0000 7f9661836640 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55d265eb2010 0x55d265eb44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.108+0000 7f9661836640 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55d265eb2010 0x55d265eb44d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b9a1c6a1716adac9 server_cookie=44fd28dc7d17f29a in_seq=0 out_seq=0 2024-09-16T00:30:10.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.108+0000 7f96457fa640 10 client.27205 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:30:10.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.108+0000 7f96457fa640 1 -- 172.21.15.33:0/3655516043 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d265e0a860 con 0x55d265eb2010 2024-09-16T00:30:10.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.108+0000 7f96457fa640 10 client.27205 handle_command_reply: tid=0 2024-09-16T00:30:10.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f9665e73740 1 client.27205 shutdown 2024-09-16T00:30:10.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f9665e73740 1 -- 172.21.15.33:0/3655516043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 msgr2=0x55d265e9f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:10.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f9665e73740 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265e9f520 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9654011ec0 tx=0x7f9654004a40 comp rx=0 tx=0).stop 2024-09-16T00:30:10.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f9665e73740 1 -- 172.21.15.33:0/3655516043 shutdown_connections 2024-09-16T00:30:10.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f9665e73740 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265e9f520 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f9665e73740 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55d265eb2010 0x55d265eb44d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f9665e73740 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265e9efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f9665e73740 1 --2- 172.21.15.33:0/3655516043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e975d0 0x55d265ea1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f96457fa640 0 client.27205 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:30:10.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f9665e73740 1 -- 172.21.15.33:0/3655516043 >> 172.21.15.33:0/3655516043 conn(0x55d265e60310 msgr2=0x55d265e99930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:10.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f9665e73740 1 -- 172.21.15.33:0/3655516043 shutdown_connections 2024-09-16T00:30:10.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f9665e73740 1 -- 172.21.15.33:0/3655516043 wait complete. 2024-09-16T00:30:10.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.109+0000 7f9665e73740 20 client.27205 trim_cache size 0 max 16384 2024-09-16T00:30:10.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.124+0000 7f9665e73740 1 Processor -- start 2024-09-16T00:30:10.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.124+0000 7f9665e73740 1 -- start start 2024-09-16T00:30:10.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.124+0000 7f9665e73740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e975d0 0x55d265e9e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.124+0000 7f9665e73740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265eae360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.124+0000 7f9665e73740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265ea3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.124+0000 7f9665e73740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1b1990 con 0x55d265e975d0 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.124+0000 7f9665e73740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1af260 con 0x55d265e98b40 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.124+0000 7f9665e73740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1ad0b0 con 0x55d265ea2370 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.124+0000 7f9661035640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265eae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f9661035640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265eae360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42474/0 (socket says 172.21.15.33:42474) 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f9661035640 1 -- 172.21.15.33:0/939967883 learned_addr learned my addr 172.21.15.33:0/939967883 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f9662037640 1 --2- 172.21.15.33:0/939967883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265ea3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f9661836640 1 --2- 172.21.15.33:0/939967883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e975d0 0x55d265e9e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f9661035640 1 -- 172.21.15.33:0/939967883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 msgr2=0x55d265ea3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f9661035640 1 --2- 172.21.15.33:0/939967883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265ea3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f9661035640 1 -- 172.21.15.33:0/939967883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e975d0 msgr2=0x55d265e9e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f9661035640 1 --2- 172.21.15.33:0/939967883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e975d0 0x55d265e9e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f9661035640 1 -- 172.21.15.33:0/939967883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d265ea5160 con 0x55d265e98b40 2024-09-16T00:30:10.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f9661035640 1 --2- 172.21.15.33:0/939967883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265eae360 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55d265e98070 tx=0x7f964c009de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbbc98ba46ae632c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:10.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f96477fe640 1 -- 172.21.15.33:0/939967883 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c028070 con 0x55d265e98b40 2024-09-16T00:30:10.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f96477fe640 1 -- 172.21.15.33:0/939967883 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f964c004730 con 0x55d265e98b40 2024-09-16T00:30:10.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.125+0000 7f96477fe640 1 -- 172.21.15.33:0/939967883 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c01d040 con 0x55d265e98b40 2024-09-16T00:30:10.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.126+0000 7f9665e73740 1 -- 172.21.15.33:0/939967883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 msgr2=0x55d265eae360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:10.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.126+0000 7f9665e73740 1 --2- 172.21.15.33:0/939967883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265eae360 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55d265e98070 tx=0x7f964c009de0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.126+0000 7f9665e73740 1 -- 172.21.15.33:0/939967883 shutdown_connections 2024-09-16T00:30:10.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.126+0000 7f9665e73740 1 --2- 172.21.15.33:0/939967883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265ea2370 0x55d265ea3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.126+0000 7f9665e73740 1 --2- 172.21.15.33:0/939967883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265eae360 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.126+0000 7f9665e73740 1 --2- 172.21.15.33:0/939967883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265e975d0 0x55d265e9e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.126+0000 7f9665e73740 1 -- 172.21.15.33:0/939967883 >> 172.21.15.33:0/939967883 conn(0x55d265e60310 msgr2=0x55d265e9d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.127+0000 7f9665e73740 1 -- 172.21.15.33:0/939967883 shutdown_connections 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.127+0000 7f9665e73740 1 -- 172.21.15.33:0/939967883 wait complete. 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.127+0000 7f9665e73740 1 -- start start 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.127+0000 7f9665e73740 1 Processor -- start 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9665e73740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265e975d0 0x55d265eaa900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9665e73740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265eb9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9665e73740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265ea2370 0x55d265eb98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9665e73740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1ad720 con 0x55d265ea2370 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9665e73740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1aeb90 con 0x55d265e98b40 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9665e73740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1addf0 con 0x55d265e975d0 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9661836640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265e975d0 0x55d265eaa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9665e73740 10 client.0 _ll_register_callbacks cb 0x7f9654009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:10.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9661836640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265e975d0 0x55d265eaa900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:43820/0 (socket says 172.21.15.33:43820) 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9661836640 1 -- 172.21.15.33:0/1092476019 learned_addr learned my addr 172.21.15.33:0/1092476019 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9661035640 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265eb9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9662037640 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265ea2370 0x55d265eb98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9661836640 1 -- 172.21.15.33:0/1092476019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 msgr2=0x55d265eb9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9661836640 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265eb9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9661836640 1 -- 172.21.15.33:0/1092476019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265ea2370 msgr2=0x55d265eb98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9661836640 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265ea2370 0x55d265eb98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9661836640 1 -- 172.21.15.33:0/1092476019 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d265eb9e00 con 0x55d265e975d0 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9662037640 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265ea2370 0x55d265eb98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f9661836640 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265e975d0 0x55d265eaa900 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f965802a460 tx=0x7f96580dc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef013eeb39923839 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.128+0000 7f96457fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f96457fa640 1 -- 172.21.15.33:0/1092476019 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96580d5060 con 0x55d265e975d0 2024-09-16T00:30:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f96457fa640 1 -- 172.21.15.33:0/1092476019 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9658003d20 con 0x55d265e975d0 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f9665e73740 1 -- 172.21.15.33:0/1092476019 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f964c004730 con 0x55d265e975d0 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f96457fa640 1 -- 172.21.15.33:0/1092476019 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9658049040 con 0x55d265e975d0 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f9665e73740 10 client.27211 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f9665e73740 1 -- 172.21.15.33:0/1092476019 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d265eaa180 con 0x55d265e975d0 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f96457fa640 1 -- 172.21.15.33:0/1092476019 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f964c004730 con 0x55d265e975d0 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f96457fa640 1 -- 172.21.15.33:0/1092476019 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f96580b8920 con 0x55d265e975d0 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f9665e73740 10 client.27211 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f9665e73740 10 client.27211 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f9665e73740 20 client.27211 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f9665e73740 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55d265f21560 0x55d265f23950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f9665e73740 4 client.27211 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f9665e73740 1 -- 172.21.15.33:0/1092476019 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d265e04d70 con 0x55d265f21560 2024-09-16T00:30:10.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.129+0000 7f9661035640 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55d265f21560 0x55d265f23950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f96457fa640 10 client.27211 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9661035640 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55d265f21560 0x55d265f23950 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=98fbdcf31ae09fb5 server_cookie=ed3b805405e1d9c2 in_seq=0 out_seq=0 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f96457fa640 1 -- 172.21.15.33:0/1092476019 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d265e04d70 con 0x55d265f21560 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f96457fa640 10 client.27211 handle_command_reply: tid=0 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9665e73740 1 client.27211 shutdown 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9665e73740 1 -- 172.21.15.33:0/1092476019 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265e975d0 msgr2=0x55d265eaa900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9665e73740 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265e975d0 0x55d265eaa900 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f965802a460 tx=0x7f96580dc040 comp rx=0 tx=0).stop 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9665e73740 1 -- 172.21.15.33:0/1092476019 shutdown_connections 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9665e73740 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d265ea2370 0x55d265eb98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9665e73740 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55d265f21560 0x55d265f23950 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9665e73740 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d265e98b40 0x55d265eb9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9665e73740 1 --2- 172.21.15.33:0/1092476019 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d265e975d0 0x55d265eaa900 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9665e73740 1 -- 172.21.15.33:0/1092476019 >> 172.21.15.33:0/1092476019 conn(0x55d265e60310 msgr2=0x55d265ea0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:10.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f96457fa640 0 client.27211 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:30:10.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9665e73740 1 -- 172.21.15.33:0/1092476019 shutdown_connections 2024-09-16T00:30:10.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9665e73740 1 -- 172.21.15.33:0/1092476019 wait complete. 2024-09-16T00:30:10.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.130+0000 7f9665e73740 20 client.27211 trim_cache size 0 max 16384 2024-09-16T00:30:10.137 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:30:10.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.131+0000 7f9663a9a640 1 -- 172.21.15.33:0/709748131 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f96480a32c0 msgr2=0x7f96480a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:10.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.131+0000 7f9663a9a640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f96480a32c0 0x7f96480a5780 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9654011630 tx=0x7f9654009500 comp rx=0 tx=0).stop 2024-09-16T00:30:10.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.131+0000 7f9663a9a640 1 -- 172.21.15.33:0/709748131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f965c102dd0 msgr2=0x7f965c198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:10.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.131+0000 7f9663a9a640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f965c102dd0 0x7f965c198670 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f9658009230 tx=0x7f9658004040 comp rx=0 tx=0).stop 2024-09-16T00:30:10.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.131+0000 7f9663a9a640 1 -- 172.21.15.33:0/709748131 shutdown_connections 2024-09-16T00:30:10.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.131+0000 7f9663a9a640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f96480a32c0 0x7f96480a5780 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.131+0000 7f9663a9a640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f965c108300 0x7f965c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.131+0000 7f9663a9a640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f965c103780 0x7f965c198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.131+0000 7f9663a9a640 1 --2- 172.21.15.33:0/709748131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f965c102dd0 0x7f965c198670 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:10.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.131+0000 7f9663a9a640 1 -- 172.21.15.33:0/709748131 >> 172.21.15.33:0/709748131 conn(0x7f965c0fc7d0 msgr2=0x7f965c105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:10.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.131+0000 7f9663a9a640 1 -- 172.21.15.33:0/709748131 shutdown_connections 2024-09-16T00:30:10.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:10.131+0000 7f9663a9a640 1 -- 172.21.15.33:0/709748131 wait complete. 2024-09-16T00:30:10.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1133900 9.87 MB/sec execute 558 sec latency 40.232 ms 2024-09-16T00:30:10.844 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:30:11.117 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:11.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1136864 9.88 MB/sec execute 559 sec latency 39.697 ms 2024-09-16T00:30:11.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:11 smithi033 ceph-mon[31243]: pgmap v741: 129 pgs: 129 active+clean; 275 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 251 op/s 2024-09-16T00:30:11.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:11 smithi083 ceph-mon[35242]: pgmap v741: 129 pgs: 129 active+clean; 275 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 251 op/s 2024-09-16T00:30:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:11 smithi060 ceph-mon[35185]: pgmap v741: 129 pgs: 129 active+clean; 275 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 251 op/s 2024-09-16T00:30:12.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.989+0000 7fe3524c2640 1 -- 172.21.15.33:0/3691943148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe34c1082a0 msgr2=0x7fe34c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.989+0000 7fe3524c2640 1 --2- 172.21.15.33:0/3691943148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe34c1082a0 0x7fe34c10ec40 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fe344009cd0 tx=0x7fe34402a8e0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.989+0000 7fe3524c2640 1 -- 172.21.15.33:0/3691943148 shutdown_connections 2024-09-16T00:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.989+0000 7fe3524c2640 1 --2- 172.21.15.33:0/3691943148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe34c1082a0 0x7fe34c10ec40 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.989+0000 7fe3524c2640 1 --2- 172.21.15.33:0/3691943148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe34c1036e0 0x7fe34c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.989+0000 7fe3524c2640 1 --2- 172.21.15.33:0/3691943148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe34c102d30 0x7fe34c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.989+0000 7fe3524c2640 1 -- 172.21.15.33:0/3691943148 >> 172.21.15.33:0/3691943148 conn(0x7fe34c0fc7d0 msgr2=0x7fe34c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.989+0000 7fe3524c2640 1 -- 172.21.15.33:0/3691943148 shutdown_connections 2024-09-16T00:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.989+0000 7fe3524c2640 1 -- 172.21.15.33:0/3691943148 wait complete. 2024-09-16T00:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe3524c2640 1 Processor -- start 2024-09-16T00:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe3524c2640 1 -- start start 2024-09-16T00:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe3524c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe34c102d30 0x7fe34c100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe3524c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe34c1036e0 0x7fe34c100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe3524c2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe34c1082a0 0x7fe34c101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe3524c2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1120d0 con 0x7fe34c1036e0 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe3524c2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c111f50 con 0x7fe34c1082a0 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe3524c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c112250 con 0x7fe34c102d30 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe343fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe34c1036e0 0x7fe34c100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe34bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe34c102d30 0x7fe34c100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe34bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe34c102d30 0x7fe34c100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52880/0 (socket says 172.21.15.33:52880) 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe343fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe34c1036e0 0x7fe34c100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40642/0 (socket says 172.21.15.33:40642) 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe34bfff640 1 -- 172.21.15.33:0/1151492351 learned_addr learned my addr 172.21.15.33:0/1151492351 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe350a5f640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe34c1082a0 0x7fe34c101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe34bfff640 1 -- 172.21.15.33:0/1151492351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe34c1082a0 msgr2=0x7fe34c101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe34bfff640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe34c1082a0 0x7fe34c101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe34bfff640 1 -- 172.21.15.33:0/1151492351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe34c1036e0 msgr2=0x7fe34c100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe34bfff640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe34c1036e0 0x7fe34c100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe34bfff640 1 -- 172.21.15.33:0/1151492351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34c1a6ad0 con 0x7fe34c102d30 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.990+0000 7fe343fff640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe34c1036e0 0x7fe34c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.991+0000 7fe34bfff640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe34c102d30 0x7fe34c100990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00d950 tx=0x7fe33c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.991+0000 7fe349ffb640 1 -- 172.21.15.33:0/1151492351 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c017030 con 0x7fe34c102d30 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.991+0000 7fe3524c2640 1 -- 172.21.15.33:0/1151492351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe34c1a6dc0 con 0x7fe34c102d30 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.991+0000 7fe349ffb640 1 -- 172.21.15.33:0/1151492351 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe33c004020 con 0x7fe34c102d30 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.991+0000 7fe349ffb640 1 -- 172.21.15.33:0/1151492351 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0047a0 con 0x7fe34c102d30 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.991+0000 7fe3524c2640 1 -- 172.21.15.33:0/1151492351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe34c1a7280 con 0x7fe34c102d30 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.993+0000 7fe349ffb640 1 -- 172.21.15.33:0/1151492351 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe33c0041d0 con 0x7fe34c102d30 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe35489b740 1 Processor -- start 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe35489b740 1 -- start start 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe35489b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x55999538f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe35489b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999538f5d0 0x55999538fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe35489b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999539a370 0x55999539c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe35489b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5599952f52f0 con 0x55999538f5d0 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe35489b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55999529ccf0 con 0x55999539a370 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe35489b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5599952f12b0 con 0x559995390b40 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe349ffb640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3380a3240 0x7fe3380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe34bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999538f5d0 0x55999538fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe350a5f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x55999538f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe350a5f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x55999538f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52886/0 (socket says 172.21.15.33:52886) 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe350a5f640 1 -- 172.21.15.33:0/550076106 learned_addr learned my addr 172.21.15.33:0/550076106 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe343fff640 1 --2- 172.21.15.33:0/550076106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999539a370 0x55999539c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe350a5f640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3380a3240 0x7fe3380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe349ffb640 1 -- 172.21.15.33:0/1151492351 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe33c0cf730 con 0x7fe34c102d30 2024-09-16T00:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe350a5f640 1 -- 172.21.15.33:0/550076106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999539a370 msgr2=0x55999539c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe350a5f640 1 --2- 172.21.15.33:0/550076106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999539a370 0x55999539c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe350a5f640 1 -- 172.21.15.33:0/550076106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999538f5d0 msgr2=0x55999538fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe350a5f640 1 --2- 172.21.15.33:0/550076106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999538f5d0 0x55999538fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.994+0000 7fe350a5f640 1 -- 172.21.15.33:0/550076106 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599952813a0 con 0x559995390b40 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.995+0000 7fe350a5f640 1 --2- 172.21.15.33:0/550076106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x55999538f090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe344004ab0 tx=0x7fe3440319f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d32fadcc71d638d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.995+0000 7fe350a5f640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3380a3240 0x7fe3380a5700 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe344006010 tx=0x7fe34404b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.995+0000 7fe3427fc640 1 -- 172.21.15.33:0/550076106 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344051070 con 0x559995390b40 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.995+0000 7fe3427fc640 1 -- 172.21.15.33:0/550076106 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe344036040 con 0x559995390b40 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.995+0000 7fe3427fc640 1 -- 172.21.15.33:0/550076106 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344030d30 con 0x559995390b40 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.995+0000 7fe35489b740 1 -- 172.21.15.33:0/550076106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 msgr2=0x55999538f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.995+0000 7fe35489b740 1 --2- 172.21.15.33:0/550076106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x55999538f090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe344004ab0 tx=0x7fe3440319f0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.995+0000 7fe35489b740 1 -- 172.21.15.33:0/550076106 shutdown_connections 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 --2- 172.21.15.33:0/550076106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999539a370 0x55999539c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 --2- 172.21.15.33:0/550076106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999538f5d0 0x55999538fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 --2- 172.21.15.33:0/550076106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x55999538f090 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 -- 172.21.15.33:0/550076106 >> 172.21.15.33:0/550076106 conn(0x559995358310 msgr2=0x55999538bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 -- 172.21.15.33:0/550076106 shutdown_connections 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 -- 172.21.15.33:0/550076106 wait complete. 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 -- start start 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 Processor -- start 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999538f5d0 0x559995399d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x559995397000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 0x5599953975f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55999539d010 con 0x55999539a370 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5599952b0060 con 0x55999538f5d0 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55999539d190 con 0x559995390b40 2024-09-16T00:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.996+0000 7fe35489b740 10 client.0 _ll_register_callbacks cb 0x5599950038c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe350a5f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 0x5599953975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe343fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x559995397000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe350a5f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 0x5599953975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40658/0 (socket says 172.21.15.33:40658) 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe350a5f640 1 -- 172.21.15.33:0/1658474394 learned_addr learned my addr 172.21.15.33:0/1658474394 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe34bfff640 1 --2- 172.21.15.33:0/1658474394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999538f5d0 0x559995399d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe350a5f640 1 -- 172.21.15.33:0/1658474394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 msgr2=0x559995397000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe350a5f640 1 --2- 172.21.15.33:0/1658474394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x559995397000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe350a5f640 1 -- 172.21.15.33:0/1658474394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999538f5d0 msgr2=0x559995399d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe350a5f640 1 --2- 172.21.15.33:0/1658474394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999538f5d0 0x559995399d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe350a5f640 1 -- 172.21.15.33:0/1658474394 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599950663c0 con 0x55999539a370 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe350a5f640 1 --2- 172.21.15.33:0/1658474394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 0x5599953975f0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fe344009430 tx=0x7fe344004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81d2cd806653eaa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe32b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe32b7fe640 1 -- 172.21.15.33:0/1658474394 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344051040 con 0x55999539a370 2024-09-16T00:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe32b7fe640 1 -- 172.21.15.33:0/1658474394 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe344004610 con 0x55999539a370 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe35489b740 1 -- 172.21.15.33:0/1658474394 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe344036040 con 0x55999539a370 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe32b7fe640 1 -- 172.21.15.33:0/1658474394 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe3440315b0 con 0x55999539a370 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe35489b740 10 client.17850 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.997+0000 7fe35489b740 1 -- 172.21.15.33:0/1658474394 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55999539d010 con 0x55999539a370 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.998+0000 7fe32b7fe640 1 -- 172.21.15.33:0/1658474394 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe344036040 con 0x55999539a370 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.998+0000 7fe32b7fe640 1 -- 172.21.15.33:0/1658474394 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fe344030070 con 0x55999539a370 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.998+0000 7fe35489b740 10 client.17850 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.998+0000 7fe35489b740 10 client.17850 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.998+0000 7fe35489b740 20 client.17850 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.998+0000 7fe35489b740 1 --2- 172.21.15.33:0/1658474394 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5599953a9c00 0x5599953abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.998+0000 7fe35489b740 4 client.17850 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.998+0000 7fe35489b740 1 -- 172.21.15.33:0/1658474394 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559995302860 con 0x5599953a9c00 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.998+0000 7fe34bfff640 1 --2- 172.21.15.33:0/1658474394 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5599953a9c00 0x5599953abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.998+0000 7fe32b7fe640 10 client.17850 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.998+0000 7fe34bfff640 1 --2- 172.21.15.33:0/1658474394 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5599953a9c00 0x5599953abff0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aff4e942b02b3666 server_cookie=32eebd8063fc4030 in_seq=0 out_seq=0 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe32b7fe640 1 -- 172.21.15.33:0/1658474394 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559995302860 con 0x5599953a9c00 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe32b7fe640 10 client.17850 handle_command_reply: tid=0 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe35489b740 1 client.17850 shutdown 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe35489b740 1 -- 172.21.15.33:0/1658474394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 msgr2=0x5599953975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe35489b740 1 --2- 172.21.15.33:0/1658474394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 0x5599953975f0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fe344009430 tx=0x7fe344004a40 comp rx=0 tx=0).stop 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe35489b740 1 -- 172.21.15.33:0/1658474394 shutdown_connections 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe35489b740 1 --2- 172.21.15.33:0/1658474394 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5599953a9c00 0x5599953abff0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe35489b740 1 --2- 172.21.15.33:0/1658474394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 0x5599953975f0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe35489b740 1 --2- 172.21.15.33:0/1658474394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x559995397000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe35489b740 1 --2- 172.21.15.33:0/1658474394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999538f5d0 0x559995399d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe32b7fe640 0 client.17850 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe35489b740 1 -- 172.21.15.33:0/1658474394 >> 172.21.15.33:0/1658474394 conn(0x559995358310 msgr2=0x559995391960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe35489b740 1 -- 172.21.15.33:0/1658474394 shutdown_connections 2024-09-16T00:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:11.999+0000 7fe35489b740 1 -- 172.21.15.33:0/1658474394 wait complete. 2024-09-16T00:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.000+0000 7fe35489b740 20 client.17850 trim_cache size 0 max 16384 2024-09-16T00:30:12.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.013+0000 7fe35489b740 1 Processor -- start 2024-09-16T00:30:12.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.013+0000 7fe35489b740 1 -- start start 2024-09-16T00:30:12.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe35489b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999538f5d0 0x5599953a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe35489b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x5599953b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe35489b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 0x5599953b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe35489b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1b3cc0 con 0x55999539a370 2024-09-16T00:30:12.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe35489b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1b1590 con 0x55999538f5d0 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe35489b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1af3e0 con 0x559995390b40 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe350a5f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 0x5599953b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe343fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x5599953b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe350a5f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 0x5599953b1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40666/0 (socket says 172.21.15.33:40666) 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe343fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x5599953b17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52898/0 (socket says 172.21.15.33:52898) 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe350a5f640 1 -- 172.21.15.33:0/4042700465 learned_addr learned my addr 172.21.15.33:0/4042700465 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe34bfff640 1 --2- 172.21.15.33:0/4042700465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999538f5d0 0x5599953a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe343fff640 1 -- 172.21.15.33:0/4042700465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999538f5d0 msgr2=0x5599953a62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe343fff640 1 --2- 172.21.15.33:0/4042700465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999538f5d0 0x5599953a62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe343fff640 1 -- 172.21.15.33:0/4042700465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 msgr2=0x5599953b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe343fff640 1 --2- 172.21.15.33:0/4042700465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 0x5599953b1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe343fff640 1 -- 172.21.15.33:0/4042700465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559995392120 con 0x559995390b40 2024-09-16T00:30:12.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe350a5f640 1 --2- 172.21.15.33:0/4042700465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 0x5599953b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe34bfff640 1 --2- 172.21.15.33:0/4042700465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999538f5d0 0x5599953a62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.014+0000 7fe343fff640 1 --2- 172.21.15.33:0/4042700465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x5599953b17d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x559995390070 tx=0x7fe3340119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5df9392c324db35b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.015+0000 7fe341ffb640 1 -- 172.21.15.33:0/4042700465 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe334010870 con 0x559995390b40 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.015+0000 7fe341ffb640 1 -- 172.21.15.33:0/4042700465 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe334021070 con 0x559995390b40 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.015+0000 7fe341ffb640 1 -- 172.21.15.33:0/4042700465 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33401cdf0 con 0x559995390b40 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.015+0000 7fe35489b740 1 -- 172.21.15.33:0/4042700465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 msgr2=0x5599953b17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.015+0000 7fe35489b740 1 --2- 172.21.15.33:0/4042700465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x5599953b17d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x559995390070 tx=0x7fe3340119d0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.015+0000 7fe35489b740 1 -- 172.21.15.33:0/4042700465 shutdown_connections 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.015+0000 7fe35489b740 1 --2- 172.21.15.33:0/4042700465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999539a370 0x5599953b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.015+0000 7fe35489b740 1 --2- 172.21.15.33:0/4042700465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559995390b40 0x5599953b17d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.015+0000 7fe35489b740 1 --2- 172.21.15.33:0/4042700465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55999538f5d0 0x5599953a62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.015+0000 7fe35489b740 1 -- 172.21.15.33:0/4042700465 >> 172.21.15.33:0/4042700465 conn(0x559995358310 msgr2=0x5599953960f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:12.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.015+0000 7fe35489b740 1 -- 172.21.15.33:0/4042700465 shutdown_connections 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.016+0000 7fe35489b740 1 -- 172.21.15.33:0/4042700465 wait complete. 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.016+0000 7fe35489b740 1 -- start start 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.016+0000 7fe35489b740 1 Processor -- start 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.016+0000 7fe35489b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999538f5d0 0x5599953a2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.016+0000 7fe35489b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559995390b40 0x5599953a2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.016+0000 7fe35489b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55999539a370 0x559995393810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.016+0000 7fe35489b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1afa50 con 0x55999538f5d0 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.016+0000 7fe35489b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1b0ec0 con 0x559995390b40 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.016+0000 7fe35489b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1b0120 con 0x55999539a370 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.016+0000 7fe35489b740 10 client.0 _ll_register_callbacks cb 0x7fe34404f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe34bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999538f5d0 0x5599953a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe350a5f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55999539a370 0x559995393810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe350a5f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55999539a370 0x559995393810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52906/0 (socket says 172.21.15.33:52906) 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe350a5f640 1 -- 172.21.15.33:0/2421330082 learned_addr learned my addr 172.21.15.33:0/2421330082 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe343fff640 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559995390b40 0x5599953a2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe350a5f640 1 -- 172.21.15.33:0/2421330082 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559995390b40 msgr2=0x5599953a2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe350a5f640 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559995390b40 0x5599953a2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe350a5f640 1 -- 172.21.15.33:0/2421330082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999538f5d0 msgr2=0x5599953a2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe350a5f640 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999538f5d0 0x5599953a2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe350a5f640 1 -- 172.21.15.33:0/2421330082 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559995393f00 con 0x55999539a370 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe34bfff640 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999538f5d0 0x5599953a2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe350a5f640 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55999539a370 0x559995393810 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe344055ed0 tx=0x7fe344009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d834c67f80f170d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe32b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe32b7fe640 1 -- 172.21.15.33:0/2421330082 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344051040 con 0x55999539a370 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe32b7fe640 1 -- 172.21.15.33:0/2421330082 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe344004410 con 0x55999539a370 2024-09-16T00:30:12.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe35489b740 1 -- 172.21.15.33:0/2421330082 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe334021070 con 0x55999539a370 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.017+0000 7fe32b7fe640 1 -- 172.21.15.33:0/2421330082 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe344004610 con 0x55999539a370 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.018+0000 7fe35489b740 10 client.27229 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.018+0000 7fe35489b740 1 -- 172.21.15.33:0/2421330082 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559995414180 con 0x55999539a370 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.018+0000 7fe32b7fe640 1 -- 172.21.15.33:0/2421330082 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe334021070 con 0x55999539a370 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.018+0000 7fe32b7fe640 1 -- 172.21.15.33:0/2421330082 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fe344055610 con 0x55999539a370 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.018+0000 7fe35489b740 10 client.27229 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.018+0000 7fe35489b740 10 client.27229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.018+0000 7fe35489b740 20 client.27229 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.018+0000 7fe35489b740 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559995414730 0x559995416b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.018+0000 7fe35489b740 4 client.27229 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.018+0000 7fe35489b740 1 -- 172.21.15.33:0/2421330082 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5599952fcd70 con 0x559995414730 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.018+0000 7fe34bfff640 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559995414730 0x559995416b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:12.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe32b7fe640 10 client.27229 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe34bfff640 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559995414730 0x559995416b50 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=185f5799d5637b2b server_cookie=41c048fee6f09f1b in_seq=0 out_seq=0 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe32b7fe640 1 -- 172.21.15.33:0/2421330082 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5599952fcd70 con 0x559995414730 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe32b7fe640 10 client.27229 handle_command_reply: tid=0 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe35489b740 1 client.27229 shutdown 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe35489b740 1 -- 172.21.15.33:0/2421330082 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55999539a370 msgr2=0x559995393810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe35489b740 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55999539a370 0x559995393810 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe344055ed0 tx=0x7fe344009360 comp rx=0 tx=0).stop 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe35489b740 1 -- 172.21.15.33:0/2421330082 shutdown_connections 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe35489b740 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559995414730 0x559995416b50 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe35489b740 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55999539a370 0x559995393810 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe35489b740 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559995390b40 0x5599953a2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe32b7fe640 0 client.27229 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe35489b740 1 --2- 172.21.15.33:0/2421330082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55999538f5d0 0x5599953a2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe35489b740 1 -- 172.21.15.33:0/2421330082 >> 172.21.15.33:0/2421330082 conn(0x559995358310 msgr2=0x559995391110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:12.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe35489b740 1 -- 172.21.15.33:0/2421330082 shutdown_connections 2024-09-16T00:30:12.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe35489b740 1 -- 172.21.15.33:0/2421330082 wait complete. 2024-09-16T00:30:12.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.019+0000 7fe35489b740 20 client.27229 trim_cache size 0 max 16384 2024-09-16T00:30:12.031 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:30:12.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.020+0000 7fe3524c2640 1 -- 172.21.15.33:0/1151492351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3380a3240 msgr2=0x7fe3380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:12.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.020+0000 7fe3524c2640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3380a3240 0x7fe3380a5700 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe344006010 tx=0x7fe34404b000 comp rx=0 tx=0).stop 2024-09-16T00:30:12.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.020+0000 7fe3524c2640 1 -- 172.21.15.33:0/1151492351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe34c102d30 msgr2=0x7fe34c100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:12.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.020+0000 7fe3524c2640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe34c102d30 0x7fe34c100990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00d950 tx=0x7fe33c00de10 comp rx=0 tx=0).stop 2024-09-16T00:30:12.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.020+0000 7fe3524c2640 1 -- 172.21.15.33:0/1151492351 shutdown_connections 2024-09-16T00:30:12.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.020+0000 7fe3524c2640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3380a3240 0x7fe3380a5700 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.020+0000 7fe3524c2640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe34c1082a0 0x7fe34c101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.020+0000 7fe3524c2640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe34c1036e0 0x7fe34c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.020+0000 7fe3524c2640 1 --2- 172.21.15.33:0/1151492351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe34c102d30 0x7fe34c100990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:12.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.020+0000 7fe3524c2640 1 -- 172.21.15.33:0/1151492351 >> 172.21.15.33:0/1151492351 conn(0x7fe34c0fc7d0 msgr2=0x7fe34c0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:12.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.020+0000 7fe3524c2640 1 -- 172.21.15.33:0/1151492351 shutdown_connections 2024-09-16T00:30:12.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:12.020+0000 7fe3524c2640 1 -- 172.21.15.33:0/1151492351 wait complete. 2024-09-16T00:30:12.687 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1139249 9.88 MB/sec execute 560 sec latency 36.966 ms 2024-09-16T00:30:13.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1141104 9.88 MB/sec execute 561 sec latency 84.666 ms 2024-09-16T00:30:13.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:13 smithi033 ceph-mon[31243]: pgmap v742: 129 pgs: 129 active+clean; 277 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.7 MiB/s wr, 299 op/s 2024-09-16T00:30:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:13 smithi083 ceph-mon[35242]: pgmap v742: 129 pgs: 129 active+clean; 277 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.7 MiB/s wr, 299 op/s 2024-09-16T00:30:13.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:13 smithi060 ceph-mon[35185]: pgmap v742: 129 pgs: 129 active+clean; 277 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.7 MiB/s wr, 299 op/s 2024-09-16T00:30:14.147 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:30:14.147 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:30:14.436 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:14.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1143972 9.90 MB/sec execute 562 sec latency 103.036 ms 2024-09-16T00:30:14.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.989+0000 7f4aaefe4640 1 -- 172.21.15.33:0/3240684870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4aa8102d30 msgr2=0x7f4aa8103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:14.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.989+0000 7f4aaefe4640 1 --2- 172.21.15.33:0/3240684870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4aa8102d30 0x7f4aa8103110 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009930 tx=0x7f4a9c02a420 comp rx=0 tx=0).stop 2024-09-16T00:30:14.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.989+0000 7f4aaefe4640 1 -- 172.21.15.33:0/3240684870 shutdown_connections 2024-09-16T00:30:14.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.989+0000 7f4aaefe4640 1 --2- 172.21.15.33:0/3240684870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa81082a0 0x7f4aa810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:14.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.989+0000 7f4aaefe4640 1 --2- 172.21.15.33:0/3240684870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aa81036e0 0x7f4aa8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:14.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.989+0000 7f4aaefe4640 1 --2- 172.21.15.33:0/3240684870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4aa8102d30 0x7f4aa8103110 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:14.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.989+0000 7f4aaefe4640 1 -- 172.21.15.33:0/3240684870 >> 172.21.15.33:0/3240684870 conn(0x7f4aa80fc7d0 msgr2=0x7f4aa80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:14.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.989+0000 7f4aaefe4640 1 -- 172.21.15.33:0/3240684870 shutdown_connections 2024-09-16T00:30:14.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.989+0000 7f4aaefe4640 1 -- 172.21.15.33:0/3240684870 wait complete. 2024-09-16T00:30:14.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aaefe4640 1 Processor -- start 2024-09-16T00:30:14.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aaefe4640 1 -- start start 2024-09-16T00:30:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aaefe4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa8102d30 0x7f4aa81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aaefe4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aa81036e0 0x7f4aa8198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aaefe4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4aa81082a0 0x7f4aa819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aaefe4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa81120a0 con 0x7f4aa81082a0 2024-09-16T00:30:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aaefe4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa8111f20 con 0x7f4aa8102d30 2024-09-16T00:30:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aaefe4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa8112220 con 0x7f4aa81036e0 2024-09-16T00:30:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aad581640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4aa81082a0 0x7f4aa819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aad581640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4aa81082a0 0x7f4aa819ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40698/0 (socket says 172.21.15.33:40698) 2024-09-16T00:30:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aad581640 1 -- 172.21.15.33:0/561900914 learned_addr learned my addr 172.21.15.33:0/561900914 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aa81036e0 0x7f4aa8198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.990+0000 7f4aacd80640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa8102d30 0x7f4aa81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:14.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.991+0000 7f4aad581640 1 -- 172.21.15.33:0/561900914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aa81036e0 msgr2=0x7f4aa8198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:14.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.991+0000 7f4aad581640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aa81036e0 0x7f4aa8198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:14.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.991+0000 7f4aad581640 1 -- 172.21.15.33:0/561900914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa8102d30 msgr2=0x7f4aa81985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:14.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.991+0000 7f4aad581640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa8102d30 0x7f4aa81985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:14.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.991+0000 7f4aad581640 1 -- 172.21.15.33:0/561900914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa819d570 con 0x7f4aa81082a0 2024-09-16T00:30:14.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.991+0000 7f4aad581640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4aa81082a0 0x7f4aa819ce90 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000eaa0 tx=0x7f4aa000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:14.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.991+0000 7f4aa5ffb640 1 -- 172.21.15.33:0/561900914 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa000cdb0 con 0x7f4aa81082a0 2024-09-16T00:30:14.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.991+0000 7f4aa5ffb640 1 -- 172.21.15.33:0/561900914 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4aa0005020 con 0x7f4aa81082a0 2024-09-16T00:30:14.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.991+0000 7f4aaefe4640 1 -- 172.21.15.33:0/561900914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa819d860 con 0x7f4aa81082a0 2024-09-16T00:30:14.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.991+0000 7f4aa5ffb640 1 -- 172.21.15.33:0/561900914 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00116b0 con 0x7f4aa81082a0 2024-09-16T00:30:14.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.991+0000 7f4aaefe4640 1 -- 172.21.15.33:0/561900914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa81a5140 con 0x7f4aa81082a0 2024-09-16T00:30:14.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.994+0000 7f4aa5ffb640 1 -- 172.21.15.33:0/561900914 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4aa0005560 con 0x7f4aa81082a0 2024-09-16T00:30:14.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4ab13bd740 1 Processor -- start 2024-09-16T00:30:14.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4ab13bd740 1 -- start start 2024-09-16T00:30:14.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4ab13bd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb687b60 0x5585fb687f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4ab13bd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb692800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4ab13bd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb692d40 0x5585fb695130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4ab13bd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5585fb5ed280 con 0x5585fb688480 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4ab13bd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5585fb594b10 con 0x5585fb687b60 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4ab13bd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5585fb5e9240 con 0x5585fb692d40 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4aa5ffb640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4a980a3340 0x7f4a980a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4aacd80640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb692800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4aacd80640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb692800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40712/0 (socket says 172.21.15.33:40712) 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4aacd80640 1 -- 172.21.15.33:0/4120793177 learned_addr learned my addr 172.21.15.33:0/4120793177 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/4120793177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb692d40 0x5585fb695130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/4120793177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb687b60 0x5585fb687f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.995+0000 7f4aad581640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4a980a3340 0x7f4a980a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:14.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.996+0000 7f4aacd80640 1 -- 172.21.15.33:0/4120793177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb692d40 msgr2=0x5585fb695130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:14.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.996+0000 7f4aacd80640 1 --2- 172.21.15.33:0/4120793177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb692d40 0x5585fb695130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:14.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.996+0000 7f4aacd80640 1 -- 172.21.15.33:0/4120793177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb687b60 msgr2=0x5585fb687f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:14.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.996+0000 7f4aacd80640 1 --2- 172.21.15.33:0/4120793177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb687b60 0x5585fb687f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:14.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.996+0000 7f4aacd80640 1 -- 172.21.15.33:0/4120793177 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585fb5a11f0 con 0x5585fb688480 2024-09-16T00:30:14.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.996+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/4120793177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb687b60 0x5585fb687f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:30:14.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.996+0000 7f4aa5ffb640 1 -- 172.21.15.33:0/561900914 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4aa00cff70 con 0x7f4aa81082a0 2024-09-16T00:30:15.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.996+0000 7f4aacd80640 1 --2- 172.21.15.33:0/4120793177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb692800 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c02a940 tx=0x7f4a9c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=decb306260821ab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:15.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.996+0000 7f4aad581640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4a980a3340 0x7f4a980a5800 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4aa009bdb0 tx=0x7f4aa00c6e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:15.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.996+0000 7f4a92ffd640 1 -- 172.21.15.33:0/4120793177 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c03f070 con 0x5585fb688480 2024-09-16T00:30:15.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.996+0000 7f4a92ffd640 1 -- 172.21.15.33:0/4120793177 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c004030 con 0x5585fb688480 2024-09-16T00:30:15.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.996+0000 7f4a92ffd640 1 -- 172.21.15.33:0/4120793177 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c035070 con 0x5585fb688480 2024-09-16T00:30:15.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.997+0000 7f4ab13bd740 1 -- 172.21.15.33:0/4120793177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 msgr2=0x5585fb692800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:15.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.997+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/4120793177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb692800 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c02a940 tx=0x7f4a9c005dd0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.997+0000 7f4ab13bd740 1 -- 172.21.15.33:0/4120793177 shutdown_connections 2024-09-16T00:30:15.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.997+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/4120793177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb692d40 0x5585fb695130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.997+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/4120793177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb692800 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.997+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/4120793177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb687b60 0x5585fb687f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.997+0000 7f4ab13bd740 1 -- 172.21.15.33:0/4120793177 >> 172.21.15.33:0/4120793177 conn(0x5585fb6840f0 msgr2=0x5585fb6844d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.997+0000 7f4ab13bd740 1 -- 172.21.15.33:0/4120793177 shutdown_connections 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.997+0000 7f4ab13bd740 1 -- 172.21.15.33:0/4120793177 wait complete. 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.997+0000 7f4ab13bd740 1 -- start start 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.997+0000 7f4ab13bd740 1 Processor -- start 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4ab13bd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb687b60 0x5585fb68fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4ab13bd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb6900f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4ab13bd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb692d40 0x5585fb690630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4ab13bd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5585fb695910 con 0x5585fb688480 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4ab13bd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5585fb5a7f10 con 0x5585fb692d40 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4ab13bd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5585fb695a90 con 0x5585fb687b60 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4ab13bd740 10 client.0 _ll_register_callbacks cb 0x5585fb570a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4aa7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb687b60 0x5585fb68fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4aacd80640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb6900f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4aa7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb687b60 0x5585fb68fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52930/0 (socket says 172.21.15.33:52930) 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4aa7fff640 1 -- 172.21.15.33:0/394617940 learned_addr learned my addr 172.21.15.33:0/394617940 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/394617940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb692d40 0x5585fb690630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4aa7fff640 1 -- 172.21.15.33:0/394617940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb692d40 msgr2=0x5585fb690630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/394617940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb692d40 0x5585fb690630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4aa7fff640 1 -- 172.21.15.33:0/394617940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 msgr2=0x5585fb6900f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/394617940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb6900f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.998+0000 7f4aa7fff640 1 -- 172.21.15.33:0/394617940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585fb35e6a0 con 0x5585fb687b60 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/394617940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb687b60 0x5585fb68fbb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009dc0 tx=0x7f4a94009aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a1be43dd1dc105b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4a87fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4a87fff640 1 -- 172.21.15.33:0/394617940 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9400fc50 con 0x5585fb687b60 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4a87fff640 1 -- 172.21.15.33:0/394617940 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4a9400fdf0 con 0x5585fb687b60 2024-09-16T00:30:15.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4ab13bd740 1 -- 172.21.15.33:0/394617940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a9c004030 con 0x5585fb687b60 2024-09-16T00:30:15.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4a87fff640 1 -- 172.21.15.33:0/394617940 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4a94004020 con 0x5585fb687b60 2024-09-16T00:30:15.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4ab13bd740 10 client.27241 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:15.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4ab13bd740 1 -- 172.21.15.33:0/394617940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5585fb6a1500 con 0x5585fb687b60 2024-09-16T00:30:15.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4a87fff640 1 -- 172.21.15.33:0/394617940 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c004030 con 0x5585fb687b60 2024-09-16T00:30:15.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4a87fff640 1 -- 172.21.15.33:0/394617940 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f4a9401d070 con 0x5585fb687b60 2024-09-16T00:30:15.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4ab13bd740 10 client.27241 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:15.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4ab13bd740 10 client.27241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:30:15.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4ab13bd740 20 client.27241 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:15.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/394617940 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5585fb6a29c0 0x5585fb6a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:15.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4ab13bd740 4 client.27241 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:30:15.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:14.999+0000 7f4ab13bd740 1 -- 172.21.15.33:0/394617940 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585fb5fa7f0 con 0x5585fb6a29c0 2024-09-16T00:30:15.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.000+0000 7f4aacd80640 1 --2- 172.21.15.33:0/394617940 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5585fb6a29c0 0x5585fb6a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:15.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.000+0000 7f4a87fff640 10 client.27241 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:15.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.000+0000 7f4aacd80640 1 --2- 172.21.15.33:0/394617940 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5585fb6a29c0 0x5585fb6a4df0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c92ec21f14e06ca server_cookie=7e1447dbd2a4232a in_seq=0 out_seq=0 2024-09-16T00:30:15.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.001+0000 7f4a87fff640 1 -- 172.21.15.33:0/394617940 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5585fb5fa7f0 con 0x5585fb6a29c0 2024-09-16T00:30:15.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.001+0000 7f4a87fff640 10 client.27241 handle_command_reply: tid=0 2024-09-16T00:30:15.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.001+0000 7f4ab13bd740 1 client.27241 shutdown 2024-09-16T00:30:15.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.001+0000 7f4ab13bd740 1 -- 172.21.15.33:0/394617940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb687b60 msgr2=0x5585fb68fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:15.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.001+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/394617940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb687b60 0x5585fb68fbb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009dc0 tx=0x7f4a94009aa0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.001+0000 7f4ab13bd740 1 -- 172.21.15.33:0/394617940 shutdown_connections 2024-09-16T00:30:15.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.001+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/394617940 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5585fb6a29c0 0x5585fb6a4df0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.001+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/394617940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb692d40 0x5585fb690630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.001+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/394617940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb6900f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.001+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/394617940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb687b60 0x5585fb68fbb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.001+0000 7f4ab13bd740 1 -- 172.21.15.33:0/394617940 >> 172.21.15.33:0/394617940 conn(0x5585fb6840f0 msgr2=0x5585fb683270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:15.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.001+0000 7f4a87fff640 0 client.27241 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:15.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.002+0000 7f4ab13bd740 1 -- 172.21.15.33:0/394617940 shutdown_connections 2024-09-16T00:30:15.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.002+0000 7f4ab13bd740 1 -- 172.21.15.33:0/394617940 wait complete. 2024-09-16T00:30:15.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.002+0000 7f4ab13bd740 20 client.27241 trim_cache size 0 max 16384 2024-09-16T00:30:15.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.021+0000 7f4ab13bd740 1 Processor -- start 2024-09-16T00:30:15.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.021+0000 7f4ab13bd740 1 -- start start 2024-09-16T00:30:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.021+0000 7f4ab13bd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb687b60 0x5585fb68f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.021+0000 7f4ab13bd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb688480 0x5585fb6aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.021+0000 7f4ab13bd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb692d40 0x5585fb6aac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.021+0000 7f4ab13bd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa81b19f0 con 0x5585fb687b60 2024-09-16T00:30:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.021+0000 7f4ab13bd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa81af2c0 con 0x5585fb688480 2024-09-16T00:30:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.021+0000 7f4ab13bd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa81ad110 con 0x5585fb692d40 2024-09-16T00:30:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4aa7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb687b60 0x5585fb68f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4aa7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb687b60 0x5585fb68f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40734/0 (socket says 172.21.15.33:40734) 2024-09-16T00:30:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4aa7fff640 1 -- 172.21.15.33:0/3102173030 learned_addr learned my addr 172.21.15.33:0/3102173030 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/3102173030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb692d40 0x5585fb6aac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4aacd80640 1 --2- 172.21.15.33:0/3102173030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb688480 0x5585fb6aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4aa7fff640 1 -- 172.21.15.33:0/3102173030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb692d40 msgr2=0x5585fb6aac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/3102173030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb692d40 0x5585fb6aac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4aa7fff640 1 -- 172.21.15.33:0/3102173030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb688480 msgr2=0x5585fb6aa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/3102173030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb688480 0x5585fb6aa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4aa7fff640 1 -- 172.21.15.33:0/3102173030 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585fb683f20 con 0x5585fb687b60 2024-09-16T00:30:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/3102173030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb687b60 0x5585fb68f490 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f4a94011b10 tx=0x7f4a94009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac440dc79ebbd00f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4a927fc640 1 -- 172.21.15.33:0/3102173030 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94005470 con 0x5585fb687b60 2024-09-16T00:30:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4a927fc640 1 -- 172.21.15.33:0/3102173030 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9400fc50 con 0x5585fb687b60 2024-09-16T00:30:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.022+0000 7f4a927fc640 1 -- 172.21.15.33:0/3102173030 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9402c7d0 con 0x5585fb687b60 2024-09-16T00:30:15.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.023+0000 7f4ab13bd740 1 -- 172.21.15.33:0/3102173030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb687b60 msgr2=0x5585fb68f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:15.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.023+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/3102173030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb687b60 0x5585fb68f490 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f4a94011b10 tx=0x7f4a94009f70 comp rx=0 tx=0).stop 2024-09-16T00:30:15.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.023+0000 7f4ab13bd740 1 -- 172.21.15.33:0/3102173030 shutdown_connections 2024-09-16T00:30:15.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.023+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/3102173030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb692d40 0x5585fb6aac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.023+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/3102173030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb688480 0x5585fb6aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.023+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/3102173030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb687b60 0x5585fb68f490 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.023+0000 7f4ab13bd740 1 -- 172.21.15.33:0/3102173030 >> 172.21.15.33:0/3102173030 conn(0x5585fb6840f0 msgr2=0x5585fb689e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.023+0000 7f4ab13bd740 1 -- 172.21.15.33:0/3102173030 shutdown_connections 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.023+0000 7f4ab13bd740 1 -- 172.21.15.33:0/3102173030 wait complete. 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.024+0000 7f4ab13bd740 1 -- start start 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.024+0000 7f4ab13bd740 1 Processor -- start 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.024+0000 7f4ab13bd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb687b60 0x5585fb69ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.024+0000 7f4ab13bd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb69b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.024+0000 7f4ab13bd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb692d40 0x5585fb706520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.024+0000 7f4ab13bd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa81ad780 con 0x5585fb688480 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.024+0000 7f4ab13bd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa81aebf0 con 0x5585fb692d40 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.024+0000 7f4ab13bd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa81ade50 con 0x5585fb687b60 2024-09-16T00:30:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.024+0000 7f4ab13bd740 10 client.0 _ll_register_callbacks cb 0x7f4a94009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4aacd80640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb69b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4aacd80640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb69b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40738/0 (socket says 172.21.15.33:40738) 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4aacd80640 1 -- 172.21.15.33:0/1522344785 learned_addr learned my addr 172.21.15.33:0/1522344785 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb687b60 0x5585fb69ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb692d40 0x5585fb706520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4aacd80640 1 -- 172.21.15.33:0/1522344785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb687b60 msgr2=0x5585fb69ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4aacd80640 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb687b60 0x5585fb69ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4aacd80640 1 -- 172.21.15.33:0/1522344785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb692d40 msgr2=0x5585fb706520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4aacd80640 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb692d40 0x5585fb706520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4aacd80640 1 -- 172.21.15.33:0/1522344785 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585fb67f830 con 0x5585fb688480 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb692d40 0x5585fb706520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4aacd80640 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb69b220 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009850 tx=0x7f4a9c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63dcfa2ef1e80d44 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:15.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.025+0000 7f4a87fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.026+0000 7f4a87fff640 1 -- 172.21.15.33:0/1522344785 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c05e060 con 0x5585fb688480 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.026+0000 7f4a87fff640 1 -- 172.21.15.33:0/1522344785 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4a9c0315b0 con 0x5585fb688480 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.026+0000 7f4ab13bd740 1 -- 172.21.15.33:0/1522344785 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a9400fc50 con 0x5585fb688480 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.026+0000 7f4a87fff640 1 -- 172.21.15.33:0/1522344785 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4a9c065050 con 0x5585fb688480 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.026+0000 7f4ab13bd740 10 client.17874 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.026+0000 7f4ab13bd740 1 -- 172.21.15.33:0/1522344785 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5585fb709670 con 0x5585fb688480 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.026+0000 7f4a87fff640 1 -- 172.21.15.33:0/1522344785 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9400fc50 con 0x5585fb688480 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.026+0000 7f4a87fff640 1 -- 172.21.15.33:0/1522344785 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f4a9c003850 con 0x5585fb688480 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.027+0000 7f4ab13bd740 10 client.17874 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.027+0000 7f4ab13bd740 10 client.17874 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.027+0000 7f4ab13bd740 20 client.17874 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.027+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5585fb7099b0 0x5585fb70bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:15.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.027+0000 7f4ab13bd740 4 client.17874 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:30:15.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.027+0000 7f4ab13bd740 1 -- 172.21.15.33:0/1522344785 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5585fb5f4d00 con 0x5585fb7099b0 2024-09-16T00:30:15.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.027+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5585fb7099b0 0x5585fb70bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:15.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.027+0000 7f4aa7fff640 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5585fb7099b0 0x5585fb70bdd0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3672e9c48b0a604 server_cookie=6ecc0abe707c5076 in_seq=0 out_seq=0 2024-09-16T00:30:15.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.027+0000 7f4a87fff640 10 client.17874 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:15.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.028+0000 7f4a87fff640 1 -- 172.21.15.33:0/1522344785 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5585fb5f4d00 con 0x5585fb7099b0 2024-09-16T00:30:15.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.028+0000 7f4a87fff640 10 client.17874 handle_command_reply: tid=0 2024-09-16T00:30:15.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.028+0000 7f4ab13bd740 1 client.17874 shutdown 2024-09-16T00:30:15.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.028+0000 7f4ab13bd740 1 -- 172.21.15.33:0/1522344785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 msgr2=0x5585fb69b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:15.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.028+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb69b220 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009850 tx=0x7f4a9c005bd0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.028+0000 7f4ab13bd740 1 -- 172.21.15.33:0/1522344785 shutdown_connections 2024-09-16T00:30:15.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.028+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5585fb7099b0 0x5585fb70bdd0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.028+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5585fb692d40 0x5585fb706520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.029+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585fb688480 0x5585fb69b220 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.029+0000 7f4ab13bd740 1 --2- 172.21.15.33:0/1522344785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585fb687b60 0x5585fb69ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.029+0000 7f4a87fff640 0 client.17874 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:15.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.029+0000 7f4ab13bd740 1 -- 172.21.15.33:0/1522344785 >> 172.21.15.33:0/1522344785 conn(0x5585fb6840f0 msgr2=0x5585fb682a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:15.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.029+0000 7f4ab13bd740 1 -- 172.21.15.33:0/1522344785 shutdown_connections 2024-09-16T00:30:15.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.029+0000 7f4ab13bd740 1 -- 172.21.15.33:0/1522344785 wait complete. 2024-09-16T00:30:15.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.029+0000 7f4ab13bd740 20 client.17874 trim_cache size 0 max 16384 2024-09-16T00:30:15.034 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:30:15.035 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:30:15.035 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "00f4dea5-186d-4dea-b7d6-280d4e7ed423", 2024-09-16T00:30:15.035 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:30:15.035 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:30:15.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.030+0000 7f4aaefe4640 1 -- 172.21.15.33:0/561900914 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4a980a3340 msgr2=0x7f4a980a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:15.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.030+0000 7f4aaefe4640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4a980a3340 0x7f4a980a5800 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4aa009bdb0 tx=0x7f4aa00c6e80 comp rx=0 tx=0).stop 2024-09-16T00:30:15.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.030+0000 7f4aaefe4640 1 -- 172.21.15.33:0/561900914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4aa81082a0 msgr2=0x7f4aa819ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:15.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.030+0000 7f4aaefe4640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4aa81082a0 0x7f4aa819ce90 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000eaa0 tx=0x7f4aa000ef60 comp rx=0 tx=0).stop 2024-09-16T00:30:15.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.030+0000 7f4aaefe4640 1 -- 172.21.15.33:0/561900914 shutdown_connections 2024-09-16T00:30:15.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.030+0000 7f4aaefe4640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4a980a3340 0x7f4a980a5800 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.030+0000 7f4aaefe4640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4aa81082a0 0x7f4aa819ce90 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.030+0000 7f4aaefe4640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aa81036e0 0x7f4aa8198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.030+0000 7f4aaefe4640 1 --2- 172.21.15.33:0/561900914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa8102d30 0x7f4aa81985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:15.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.030+0000 7f4aaefe4640 1 -- 172.21.15.33:0/561900914 >> 172.21.15.33:0/561900914 conn(0x7f4aa80fc7d0 msgr2=0x7f4aa8105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:15.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.030+0000 7f4aaefe4640 1 -- 172.21.15.33:0/561900914 shutdown_connections 2024-09-16T00:30:15.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:15.030+0000 7f4aaefe4640 1 -- 172.21.15.33:0/561900914 wait complete. 2024-09-16T00:30:15.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1146133 9.89 MB/sec execute 563 sec latency 21.210 ms 2024-09-16T00:30:15.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:15 smithi033 ceph-mon[31243]: pgmap v743: 129 pgs: 129 active+clean; 280 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 310 op/s 2024-09-16T00:30:15.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:15 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:30:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:15 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:30:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:15 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:30:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:15 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:30:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:15 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:30:15.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:15 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:30:15.843 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:30:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:15 smithi083 ceph-mon[35242]: pgmap v743: 129 pgs: 129 active+clean; 280 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 310 op/s 2024-09-16T00:30:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:15 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:30:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:15 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:30:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:15 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:30:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:15 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:30:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:15 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:30:15.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:15 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:30:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:15 smithi060 ceph-mon[35185]: pgmap v743: 129 pgs: 129 active+clean; 280 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 310 op/s 2024-09-16T00:30:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:15 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:30:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:15 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:30:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:15 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:30:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:15 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:30:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:15 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:30:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:15 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:30:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:16.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.644+0000 7fc149c5e640 1 -- 172.21.15.33:0/3396595432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144101de0 msgr2=0x7fc1441021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:16.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.644+0000 7fc149c5e640 1 --2- 172.21.15.33:0/3396595432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144101de0 0x7fc1441021c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc134009be0 tx=0x7fc13402a6c0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.644+0000 7fc149c5e640 1 -- 172.21.15.33:0/3396595432 shutdown_connections 2024-09-16T00:30:16.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.644+0000 7fc149c5e640 1 --2- 172.21.15.33:0/3396595432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc144106f10 0x7fc14410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.644+0000 7fc149c5e640 1 --2- 172.21.15.33:0/3396595432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc144102790 0x7fc1441067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.644+0000 7fc149c5e640 1 --2- 172.21.15.33:0/3396595432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144101de0 0x7fc1441021c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.644+0000 7fc149c5e640 1 -- 172.21.15.33:0/3396595432 >> 172.21.15.33:0/3396595432 conn(0x7fc1440fd880 msgr2=0x7fc1440ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:16.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.644+0000 7fc149c5e640 1 -- 172.21.15.33:0/3396595432 shutdown_connections 2024-09-16T00:30:16.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.644+0000 7fc149c5e640 1 -- 172.21.15.33:0/3396595432 wait complete. 2024-09-16T00:30:16.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.645+0000 7fc149c5e640 1 Processor -- start 2024-09-16T00:30:16.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.645+0000 7fc149c5e640 1 -- start start 2024-09-16T00:30:16.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.645+0000 7fc149c5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc144101de0 0x7fc144198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc149c5e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc144102790 0x7fc144198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc149c5e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144106f10 0x7fc14419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc149c5e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc144111f20 con 0x7fc144102790 2024-09-16T00:30:16.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc149c5e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc144111da0 con 0x7fc144106f10 2024-09-16T00:30:16.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc149c5e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441120a0 con 0x7fc144101de0 2024-09-16T00:30:16.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc142ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc144102790 0x7fc144198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc142ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc144102790 0x7fc144198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40756/0 (socket says 172.21.15.33:40756) 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc142ffd640 1 -- 172.21.15.33:0/395273494 learned_addr learned my addr 172.21.15.33:0/395273494 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc1437fe640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc144101de0 0x7fc144198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc143fff640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144106f10 0x7fc14419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc143fff640 1 -- 172.21.15.33:0/395273494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc144101de0 msgr2=0x7fc144198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc143fff640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc144101de0 0x7fc144198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc143fff640 1 -- 172.21.15.33:0/395273494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc144102790 msgr2=0x7fc144198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc143fff640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc144102790 0x7fc144198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc143fff640 1 -- 172.21.15.33:0/395273494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14419d3e0 con 0x7fc144106f10 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.646+0000 7fc142ffd640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc144102790 0x7fc144198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.647+0000 7fc143fff640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144106f10 0x7fc14419cd00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009fd0 tx=0x7fc13c00eba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.647+0000 7fc140ff9640 1 -- 172.21.15.33:0/395273494 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c004220 con 0x7fc144106f10 2024-09-16T00:30:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.647+0000 7fc140ff9640 1 -- 172.21.15.33:0/395273494 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc13c00cd90 con 0x7fc144106f10 2024-09-16T00:30:16.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.647+0000 7fc140ff9640 1 -- 172.21.15.33:0/395273494 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c004a00 con 0x7fc144106f10 2024-09-16T00:30:16.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.647+0000 7fc149c5e640 1 -- 172.21.15.33:0/395273494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc14419d6d0 con 0x7fc144106f10 2024-09-16T00:30:16.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.647+0000 7fc149c5e640 1 -- 172.21.15.33:0/395273494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc144074420 con 0x7fc144106f10 2024-09-16T00:30:16.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.650+0000 7fc14c037740 1 Processor -- start 2024-09-16T00:30:16.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.650+0000 7fc140ff9640 1 -- 172.21.15.33:0/395273494 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc13c004ba0 con 0x7fc144106f10 2024-09-16T00:30:16.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.650+0000 7fc14c037740 1 -- start start 2024-09-16T00:30:16.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc14c037740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323a4a0 0x55c0a323a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc14c037740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323adc0 0x55c0a3245140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc14c037740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a3247a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc14c037740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c0a31a02e0 con 0x55c0a323adc0 2024-09-16T00:30:16.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc14c037740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c0a3147ce0 con 0x55c0a3245680 2024-09-16T00:30:16.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc14c037740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c0a319c2a0 con 0x55c0a323a4a0 2024-09-16T00:30:16.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323adc0 0x55c0a3245140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323adc0 0x55c0a3245140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40766/0 (socket says 172.21.15.33:40766) 2024-09-16T00:30:16.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc1437fe640 1 -- 172.21.15.33:0/2368493024 learned_addr learned my addr 172.21.15.33:0/2368493024 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:16.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc142ffd640 1 --2- 172.21.15.33:0/2368493024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a3247a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc142ffd640 1 --2- 172.21.15.33:0/2368493024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323a4a0 0x55c0a323a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc1437fe640 1 -- 172.21.15.33:0/2368493024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323a4a0 msgr2=0x55c0a323a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:16.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc1437fe640 1 --2- 172.21.15.33:0/2368493024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323a4a0 0x55c0a323a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc140ff9640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc12c0a76d0 0x7fc12c0a9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc1437fe640 1 -- 172.21.15.33:0/2368493024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 msgr2=0x55c0a3247a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:16.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc1437fe640 1 --2- 172.21.15.33:0/2368493024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a3247a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc1437fe640 1 -- 172.21.15.33:0/2368493024 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0a312c440 con 0x55c0a323adc0 2024-09-16T00:30:16.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc142ffd640 1 --2- 172.21.15.33:0/2368493024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323a4a0 0x55c0a323a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc143fff640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc12c0a76d0 0x7fc12c0a9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.651+0000 7fc1437fe640 1 --2- 172.21.15.33:0/2368493024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323adc0 0x55c0a3245140 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fc134009be0 tx=0x7fc134030a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151ffc8879793047 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.652+0000 7fc140ff9640 1 -- 172.21.15.33:0/395273494 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc13c0cfab0 con 0x7fc144106f10 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.652+0000 7fc129ffb640 1 -- 172.21.15.33:0/2368493024 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134042030 con 0x55c0a323adc0 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.652+0000 7fc129ffb640 1 -- 172.21.15.33:0/2368493024 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc134037040 con 0x55c0a323adc0 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.652+0000 7fc129ffb640 1 -- 172.21.15.33:0/2368493024 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134036070 con 0x55c0a323adc0 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.652+0000 7fc14c037740 1 -- 172.21.15.33:0/2368493024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323adc0 msgr2=0x55c0a3245140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.652+0000 7fc143fff640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc12c0a76d0 0x7fc12c0a9b90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0980b0 tx=0x7fc13c06a990 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.652+0000 7fc14c037740 1 --2- 172.21.15.33:0/2368493024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323adc0 0x55c0a3245140 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fc134009be0 tx=0x7fc134030a30 comp rx=0 tx=0).stop 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.653+0000 7fc14c037740 1 -- 172.21.15.33:0/2368493024 shutdown_connections 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.653+0000 7fc14c037740 1 --2- 172.21.15.33:0/2368493024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a3247a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.653+0000 7fc14c037740 1 --2- 172.21.15.33:0/2368493024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323adc0 0x55c0a3245140 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.653+0000 7fc14c037740 1 --2- 172.21.15.33:0/2368493024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323a4a0 0x55c0a323a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.653+0000 7fc14c037740 1 -- 172.21.15.33:0/2368493024 >> 172.21.15.33:0/2368493024 conn(0x55c0a3236a30 msgr2=0x55c0a3236e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.653+0000 7fc14c037740 1 -- 172.21.15.33:0/2368493024 shutdown_connections 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.653+0000 7fc14c037740 1 -- 172.21.15.33:0/2368493024 wait complete. 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.653+0000 7fc14c037740 1 -- start start 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.653+0000 7fc14c037740 1 Processor -- start 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc14c037740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a323a4a0 0x55c0a3244ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc14c037740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a3242190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc14c037740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a3245680 0x55c0a32426d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc14c037740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c0a3248250 con 0x55c0a3245680 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc14c037740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c0a315b050 con 0x55c0a323a4a0 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc14c037740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c0a32483d0 con 0x55c0a323adc0 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc14c037740 10 client.0 _ll_register_callbacks cb 0x55c0a319b160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:16.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a3242190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a3242190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52984/0 (socket says 172.21.15.33:52984) 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc1437fe640 1 -- 172.21.15.33:0/2013768552 learned_addr learned my addr 172.21.15.33:0/2013768552 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc142ffd640 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a323a4a0 0x55c0a3244ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc142ffd640 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a3245680 0x55c0a32426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc1437fe640 1 -- 172.21.15.33:0/2013768552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a323a4a0 msgr2=0x55c0a3244ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc1437fe640 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a323a4a0 0x55c0a3244ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc1437fe640 1 -- 172.21.15.33:0/2013768552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a3245680 msgr2=0x55c0a32426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc1437fe640 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a3245680 0x55c0a32426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc1437fe640 1 -- 172.21.15.33:0/2013768552 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0a2f113c0 con 0x55c0a323adc0 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc1437fe640 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a3242190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc134009370 tx=0x7fc13402a6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9e015dc45f8a108 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.654+0000 7fc122ffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.655+0000 7fc122ffd640 1 -- 172.21.15.33:0/2013768552 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134042040 con 0x55c0a323adc0 2024-09-16T00:30:16.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.655+0000 7fc122ffd640 1 -- 172.21.15.33:0/2013768552 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc1340305a0 con 0x55c0a323adc0 2024-09-16T00:30:16.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.655+0000 7fc14c037740 1 -- 172.21.15.33:0/2013768552 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc134037040 con 0x55c0a323adc0 2024-09-16T00:30:16.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.655+0000 7fc122ffd640 1 -- 172.21.15.33:0/2013768552 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc134058050 con 0x55c0a323adc0 2024-09-16T00:30:16.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.655+0000 7fc14c037740 10 client.27259 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:16.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.655+0000 7fc14c037740 1 -- 172.21.15.33:0/2013768552 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c0a3248250 con 0x55c0a323adc0 2024-09-16T00:30:16.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.655+0000 7fc122ffd640 1 -- 172.21.15.33:0/2013768552 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc134037040 con 0x55c0a323adc0 2024-09-16T00:30:16.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.656+0000 7fc122ffd640 1 -- 172.21.15.33:0/2013768552 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fc13403c040 con 0x55c0a323adc0 2024-09-16T00:30:16.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.656+0000 7fc14c037740 10 client.27259 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:16.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.656+0000 7fc14c037740 10 client.27259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:30:16.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.656+0000 7fc14c037740 20 client.27259 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:16.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.656+0000 7fc14c037740 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0a3254cd0 0x55c0a32570c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.656+0000 7fc14c037740 4 client.27259 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:30:16.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.656+0000 7fc14c037740 1 -- 172.21.15.33:0/2013768552 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0a31ad850 con 0x55c0a3254cd0 2024-09-16T00:30:16.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.656+0000 7fc142ffd640 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0a3254cd0 0x55c0a32570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.656+0000 7fc142ffd640 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0a3254cd0 0x55c0a32570c0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1649d31cb85f35e6 server_cookie=a5ddcefa730abb3d in_seq=0 out_seq=0 2024-09-16T00:30:16.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.656+0000 7fc122ffd640 10 client.27259 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:16.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc122ffd640 1 -- 172.21.15.33:0/2013768552 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c0a31ad850 con 0x55c0a3254cd0 2024-09-16T00:30:16.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc122ffd640 10 client.27259 handle_command_reply: tid=0 2024-09-16T00:30:16.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc14c037740 1 client.27259 shutdown 2024-09-16T00:30:16.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc14c037740 1 -- 172.21.15.33:0/2013768552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 msgr2=0x55c0a3242190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:16.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc14c037740 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a3242190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc134009370 tx=0x7fc13402a6c0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc14c037740 1 -- 172.21.15.33:0/2013768552 shutdown_connections 2024-09-16T00:30:16.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc14c037740 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0a3254cd0 0x55c0a32570c0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc14c037740 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a3245680 0x55c0a32426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc14c037740 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a3242190 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc14c037740 1 --2- 172.21.15.33:0/2013768552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a323a4a0 0x55c0a3244ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc14c037740 1 -- 172.21.15.33:0/2013768552 >> 172.21.15.33:0/2013768552 conn(0x55c0a3236a30 msgr2=0x55c0a323c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:16.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc122ffd640 0 client.27259 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:16.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc14c037740 1 -- 172.21.15.33:0/2013768552 shutdown_connections 2024-09-16T00:30:16.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.657+0000 7fc14c037740 1 -- 172.21.15.33:0/2013768552 wait complete. 2024-09-16T00:30:16.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.658+0000 7fc14c037740 20 client.27259 trim_cache size 0 max 16384 2024-09-16T00:30:16.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.678+0000 7fc14c037740 1 Processor -- start 2024-09-16T00:30:16.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.678+0000 7fc14c037740 1 -- start start 2024-09-16T00:30:16.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.678+0000 7fc14c037740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323a4a0 0x55c0a325c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.678+0000 7fc14c037740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a325cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.678+0000 7fc14c037740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a323f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.678+0000 7fc14c037740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441af750 con 0x55c0a323a4a0 2024-09-16T00:30:16.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.678+0000 7fc14c037740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441ad020 con 0x55c0a3245680 2024-09-16T00:30:16.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.678+0000 7fc14c037740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441aae70 con 0x55c0a323adc0 2024-09-16T00:30:16.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.678+0000 7fc142ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323a4a0 0x55c0a325c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.678+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a325cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.678+0000 7fc142ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323a4a0 0x55c0a325c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40784/0 (socket says 172.21.15.33:40784) 2024-09-16T00:30:16.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.678+0000 7fc142ffd640 1 -- 172.21.15.33:0/630006329 learned_addr learned my addr 172.21.15.33:0/630006329 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:16.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.679+0000 7fc1437fe640 1 -- 172.21.15.33:0/630006329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 msgr2=0x55c0a323f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:16.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.679+0000 7fc142ffd640 1 --2- 172.21.15.33:0/630006329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a323f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.679+0000 7fc1437fe640 1 --2- 172.21.15.33:0/630006329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a323f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.679+0000 7fc1437fe640 1 -- 172.21.15.33:0/630006329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323a4a0 msgr2=0x55c0a325c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:16.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.679+0000 7fc1437fe640 1 --2- 172.21.15.33:0/630006329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323a4a0 0x55c0a325c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.679+0000 7fc1437fe640 1 -- 172.21.15.33:0/630006329 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0a3236860 con 0x55c0a323adc0 2024-09-16T00:30:16.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.679+0000 7fc1437fe640 1 --2- 172.21.15.33:0/630006329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a325cdc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc134034040 tx=0x7fc134005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71cd1b19da29e07e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:16.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.679+0000 7fc1297fa640 1 -- 172.21.15.33:0/630006329 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134042070 con 0x55c0a323adc0 2024-09-16T00:30:16.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.679+0000 7fc1297fa640 1 -- 172.21.15.33:0/630006329 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc13403e8e0 con 0x55c0a323adc0 2024-09-16T00:30:16.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.679+0000 7fc1297fa640 1 -- 172.21.15.33:0/630006329 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134036040 con 0x55c0a323adc0 2024-09-16T00:30:16.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.680+0000 7fc14c037740 1 -- 172.21.15.33:0/630006329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 msgr2=0x55c0a325cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.680+0000 7fc14c037740 1 --2- 172.21.15.33:0/630006329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a325cdc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc134034040 tx=0x7fc134005fd0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.680+0000 7fc14c037740 1 -- 172.21.15.33:0/630006329 shutdown_connections 2024-09-16T00:30:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.680+0000 7fc14c037740 1 --2- 172.21.15.33:0/630006329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a323f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.680+0000 7fc14c037740 1 --2- 172.21.15.33:0/630006329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a325cdc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.680+0000 7fc14c037740 1 --2- 172.21.15.33:0/630006329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323a4a0 0x55c0a325c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.680+0000 7fc14c037740 1 -- 172.21.15.33:0/630006329 >> 172.21.15.33:0/630006329 conn(0x55c0a3236a30 msgr2=0x55c0a3244210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.680+0000 7fc14c037740 1 -- 172.21.15.33:0/630006329 shutdown_connections 2024-09-16T00:30:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.680+0000 7fc14c037740 1 -- 172.21.15.33:0/630006329 wait complete. 2024-09-16T00:30:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.681+0000 7fc14c037740 1 -- start start 2024-09-16T00:30:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.681+0000 7fc14c037740 1 Processor -- start 2024-09-16T00:30:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.681+0000 7fc14c037740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323a4a0 0x55c0a323f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.681+0000 7fc14c037740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a323f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.681+0000 7fc14c037740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a323fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.681+0000 7fc14c037740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441ab4e0 con 0x55c0a323a4a0 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.681+0000 7fc14c037740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441ac950 con 0x55c0a3245680 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.681+0000 7fc14c037740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441abbb0 con 0x55c0a323adc0 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.681+0000 7fc14c037740 10 client.0 _ll_register_callbacks cb 0x7fc138009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.681+0000 7fc142ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323a4a0 0x55c0a323f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.681+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a323f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.681+0000 7fc142ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323a4a0 0x55c0a323f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40786/0 (socket says 172.21.15.33:40786) 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc142ffd640 1 -- 172.21.15.33:0/3984034178 learned_addr learned my addr 172.21.15.33:0/3984034178 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a323f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:52992/0 (socket says 172.21.15.33:52992) 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc142ffd640 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a323fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc1437fe640 1 -- 172.21.15.33:0/3984034178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 msgr2=0x55c0a323fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:16.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc1437fe640 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a323fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc1437fe640 1 -- 172.21.15.33:0/3984034178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323a4a0 msgr2=0x55c0a323f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:16.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc1437fe640 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323a4a0 0x55c0a323f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc1437fe640 1 -- 172.21.15.33:0/3984034178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0a32be850 con 0x55c0a323adc0 2024-09-16T00:30:16.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc142ffd640 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a323fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:30:16.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc1437fe640 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a323f5d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55c0a3248220 tx=0x7fc13402a6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eae9694b585dd56c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:16.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc122ffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:16.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc122ffd640 1 -- 172.21.15.33:0/3984034178 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134036030 con 0x55c0a323adc0 2024-09-16T00:30:16.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc122ffd640 1 -- 172.21.15.33:0/3984034178 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc13403ec90 con 0x55c0a323adc0 2024-09-16T00:30:16.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.682+0000 7fc14c037740 1 -- 172.21.15.33:0/3984034178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc13403e8e0 con 0x55c0a323adc0 2024-09-16T00:30:16.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.683+0000 7fc122ffd640 1 -- 172.21.15.33:0/3984034178 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc13406d050 con 0x55c0a323adc0 2024-09-16T00:30:16.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.683+0000 7fc14c037740 10 client.27265 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:16.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.683+0000 7fc14c037740 1 -- 172.21.15.33:0/3984034178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c0a32bfa40 con 0x55c0a323adc0 2024-09-16T00:30:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.683+0000 7fc122ffd640 1 -- 172.21.15.33:0/3984034178 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc13403e8e0 con 0x55c0a323adc0 2024-09-16T00:30:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.683+0000 7fc122ffd640 1 -- 172.21.15.33:0/3984034178 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fc134031670 con 0x55c0a323adc0 2024-09-16T00:30:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.683+0000 7fc14c037740 10 client.27265 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.683+0000 7fc14c037740 10 client.27265 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:30:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.683+0000 7fc14c037740 20 client.27265 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.683+0000 7fc14c037740 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0a32bfff0 0x55c0a32c23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.683+0000 7fc14c037740 4 client.27265 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:30:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.683+0000 7fc14c037740 1 -- 172.21.15.33:0/3984034178 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c0a31a7d60 con 0x55c0a32bfff0 2024-09-16T00:30:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.683+0000 7fc142ffd640 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0a32bfff0 0x55c0a32c23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:16.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.684+0000 7fc122ffd640 10 client.27265 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:16.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.684+0000 7fc142ffd640 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0a32bfff0 0x55c0a32c23e0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=391d71ca93d930 server_cookie=c559520d40d1b9ad in_seq=0 out_seq=0 2024-09-16T00:30:16.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.684+0000 7fc122ffd640 1 -- 172.21.15.33:0/3984034178 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c0a31a7d60 con 0x55c0a32bfff0 2024-09-16T00:30:16.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.684+0000 7fc122ffd640 10 client.27265 handle_command_reply: tid=0 2024-09-16T00:30:16.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.684+0000 7fc14c037740 1 client.27265 shutdown 2024-09-16T00:30:16.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.685+0000 7fc14c037740 1 -- 172.21.15.33:0/3984034178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 msgr2=0x55c0a323f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:16.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.685+0000 7fc14c037740 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a323f5d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55c0a3248220 tx=0x7fc13402a6c0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.685+0000 7fc14c037740 1 -- 172.21.15.33:0/3984034178 shutdown_connections 2024-09-16T00:30:16.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.685+0000 7fc14c037740 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0a32bfff0 0x55c0a32c23e0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.685+0000 7fc14c037740 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0a3245680 0x55c0a323fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.685+0000 7fc122ffd640 0 client.27265 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:16.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.685+0000 7fc14c037740 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0a323adc0 0x55c0a323f5d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.685+0000 7fc14c037740 1 --2- 172.21.15.33:0/3984034178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0a323a4a0 0x55c0a323f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.685+0000 7fc14c037740 1 -- 172.21.15.33:0/3984034178 >> 172.21.15.33:0/3984034178 conn(0x55c0a3236a30 msgr2=0x55c0a3235350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:16.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.685+0000 7fc14c037740 1 -- 172.21.15.33:0/3984034178 shutdown_connections 2024-09-16T00:30:16.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.685+0000 7fc14c037740 1 -- 172.21.15.33:0/3984034178 wait complete. 2024-09-16T00:30:16.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.686+0000 7fc14c037740 20 client.27265 trim_cache size 0 max 16384 2024-09-16T00:30:16.692 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:30:16.692 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:30:16.692 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:30:16.692 INFO:teuthology.orchestra.run.smithi033.stdout: "00f4dea5-186d-4dea-b7d6-280d4e7ed423": { 2024-09-16T00:30:16.692 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:30:16.692 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "00f4dea5-186d-4dea-b7d6-280d4e7ed423", 2024-09-16T00:30:16.692 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:30:16.692 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:30:16.692 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:30:16.692 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:30:16.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.687+0000 7fc149c5e640 1 -- 172.21.15.33:0/395273494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc12c0a76d0 msgr2=0x7fc12c0a9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:16.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.687+0000 7fc149c5e640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc12c0a76d0 0x7fc12c0a9b90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0980b0 tx=0x7fc13c06a990 comp rx=0 tx=0).stop 2024-09-16T00:30:16.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.687+0000 7fc149c5e640 1 -- 172.21.15.33:0/395273494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144106f10 msgr2=0x7fc14419cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:16.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.687+0000 7fc149c5e640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144106f10 0x7fc14419cd00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009fd0 tx=0x7fc13c00eba0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.687+0000 7fc149c5e640 1 -- 172.21.15.33:0/395273494 shutdown_connections 2024-09-16T00:30:16.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.687+0000 7fc149c5e640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc12c0a76d0 0x7fc12c0a9b90 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.687+0000 7fc149c5e640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144106f10 0x7fc14419cd00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.687+0000 7fc149c5e640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc144102790 0x7fc144198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.687+0000 7fc149c5e640 1 --2- 172.21.15.33:0/395273494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc144101de0 0x7fc144198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:16.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.687+0000 7fc149c5e640 1 -- 172.21.15.33:0/395273494 >> 172.21.15.33:0/395273494 conn(0x7fc1440fd880 msgr2=0x7fc1440fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:16.694 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1148098 9.90 MB/sec execute 564 sec latency 37.601 ms 2024-09-16T00:30:16.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.687+0000 7fc149c5e640 1 -- 172.21.15.33:0/395273494 shutdown_connections 2024-09-16T00:30:16.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:16.687+0000 7fc149c5e640 1 -- 172.21.15.33:0/395273494 wait complete. 2024-09-16T00:30:17.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1150910 9.91 MB/sec execute 565 sec latency 54.884 ms 2024-09-16T00:30:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:17 smithi083 ceph-mon[35242]: pgmap v744: 129 pgs: 129 active+clean; 282 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 270 op/s 2024-09-16T00:30:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:17 smithi060 ceph-mon[35185]: pgmap v744: 129 pgs: 129 active+clean; 282 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 270 op/s 2024-09-16T00:30:18.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:17 smithi033 ceph-mon[31243]: pgmap v744: 129 pgs: 129 active+clean; 282 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 270 op/s 2024-09-16T00:30:18.451 INFO:tasks.cephfs.filesystem:scrub status for tag:00f4dea5-186d-4dea-b7d6-280d4e7ed423 - {'path': '/', 'tag': '00f4dea5-186d-4dea-b7d6-280d4e7ed423', 'options': 'recursive,force'} 2024-09-16T00:30:18.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1153286 9.91 MB/sec execute 566 sec latency 11.956 ms 2024-09-16T00:30:19.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1155175 9.91 MB/sec execute 567 sec latency 67.167 ms 2024-09-16T00:30:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:19 smithi083 ceph-mon[35242]: pgmap v745: 129 pgs: 129 active+clean; 282 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 234 op/s 2024-09-16T00:30:19.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:19 smithi060 ceph-mon[35185]: pgmap v745: 129 pgs: 129 active+clean; 282 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 234 op/s 2024-09-16T00:30:20.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:19 smithi033 ceph-mon[31243]: pgmap v745: 129 pgs: 129 active+clean; 282 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 234 op/s 2024-09-16T00:30:20.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1158149 9.93 MB/sec execute 568 sec latency 22.162 ms 2024-09-16T00:30:21.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1160867 9.93 MB/sec execute 569 sec latency 12.369 ms 2024-09-16T00:30:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:21 smithi083 ceph-mon[35242]: pgmap v746: 129 pgs: 129 active+clean; 278 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 253 op/s 2024-09-16T00:30:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:21 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:30:21.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:21 smithi060 ceph-mon[35185]: pgmap v746: 129 pgs: 129 active+clean; 278 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 253 op/s 2024-09-16T00:30:21.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:21 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:30:22.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:21 smithi033 ceph-mon[31243]: pgmap v746: 129 pgs: 129 active+clean; 278 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 253 op/s 2024-09-16T00:30:22.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:21 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:30:22.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1163123 9.94 MB/sec execute 570 sec latency 12.866 ms 2024-09-16T00:30:22.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:22 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:30:22.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:22 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:30:23.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:22 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:30:23.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1165684 9.94 MB/sec execute 571 sec latency 22.167 ms 2024-09-16T00:30:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:23 smithi083 ceph-mon[35242]: pgmap v747: 129 pgs: 129 active+clean; 282 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.9 MiB/s wr, 312 op/s 2024-09-16T00:30:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:23 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:30:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:23 smithi060 ceph-mon[35185]: pgmap v747: 129 pgs: 129 active+clean; 282 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.9 MiB/s wr, 312 op/s 2024-09-16T00:30:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:23 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:30:24.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:23 smithi033 ceph-mon[31243]: pgmap v747: 129 pgs: 129 active+clean; 282 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.9 MiB/s wr, 312 op/s 2024-09-16T00:30:24.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:23 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:30:24.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1167936 9.95 MB/sec execute 572 sec latency 137.557 ms 2024-09-16T00:30:24.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:24 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:30:24.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:24 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:30:24.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:24 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:30:24.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:24 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:30:24.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:24 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:30:24.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:24 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:30:25.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:24 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:30:25.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:24 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:30:25.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:24 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:30:25.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1169570 9.94 MB/sec execute 573 sec latency 155.636 ms 2024-09-16T00:30:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:25 smithi083 ceph-mon[35242]: pgmap v748: 129 pgs: 129 active+clean; 281 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 271 op/s 2024-09-16T00:30:25.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:25 smithi060 ceph-mon[35185]: pgmap v748: 129 pgs: 129 active+clean; 281 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 271 op/s 2024-09-16T00:30:26.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:25 smithi033 ceph-mon[31243]: pgmap v748: 129 pgs: 129 active+clean; 281 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 271 op/s 2024-09-16T00:30:26.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1172518 9.96 MB/sec execute 574 sec latency 21.176 ms 2024-09-16T00:30:27.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1175472 9.97 MB/sec execute 575 sec latency 14.685 ms 2024-09-16T00:30:27.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:27 smithi083 ceph-mon[35242]: pgmap v749: 129 pgs: 129 active+clean; 286 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.6 MiB/s wr, 276 op/s 2024-09-16T00:30:27.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:27 smithi060 ceph-mon[35185]: pgmap v749: 129 pgs: 129 active+clean; 286 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.6 MiB/s wr, 276 op/s 2024-09-16T00:30:28.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:27 smithi033 ceph-mon[31243]: pgmap v749: 129 pgs: 129 active+clean; 286 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.6 MiB/s wr, 276 op/s 2024-09-16T00:30:28.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1177635 9.97 MB/sec execute 576 sec latency 14.140 ms 2024-09-16T00:30:29.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1179669 9.97 MB/sec execute 577 sec latency 63.706 ms 2024-09-16T00:30:29.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:29 smithi083 ceph-mon[35242]: pgmap v750: 129 pgs: 129 active+clean; 286 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 235 op/s 2024-09-16T00:30:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:29 smithi060 ceph-mon[35185]: pgmap v750: 129 pgs: 129 active+clean; 286 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 235 op/s 2024-09-16T00:30:30.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:29 smithi033 ceph-mon[31243]: pgmap v750: 129 pgs: 129 active+clean; 286 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 235 op/s 2024-09-16T00:30:30.688 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1182221 9.98 MB/sec execute 578 sec latency 22.238 ms 2024-09-16T00:30:31.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1184173 9.98 MB/sec execute 579 sec latency 18.443 ms 2024-09-16T00:30:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:31 smithi083 ceph-mon[35242]: pgmap v751: 129 pgs: 129 active+clean; 286 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 248 op/s 2024-09-16T00:30:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:31 smithi060 ceph-mon[35185]: pgmap v751: 129 pgs: 129 active+clean; 286 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 248 op/s 2024-09-16T00:30:32.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:31 smithi033 ceph-mon[31243]: pgmap v751: 129 pgs: 129 active+clean; 286 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 248 op/s 2024-09-16T00:30:32.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1186729 9.99 MB/sec execute 580 sec latency 17.703 ms 2024-09-16T00:30:33.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1189487 9.99 MB/sec execute 581 sec latency 16.675 ms 2024-09-16T00:30:33.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:33 smithi033 ceph-mon[31243]: pgmap v752: 129 pgs: 129 active+clean; 288 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 290 op/s 2024-09-16T00:30:33.936 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:33 smithi083 ceph-mon[35242]: pgmap v752: 129 pgs: 129 active+clean; 288 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 290 op/s 2024-09-16T00:30:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:33 smithi060 ceph-mon[35185]: pgmap v752: 129 pgs: 129 active+clean; 288 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 290 op/s 2024-09-16T00:30:34.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1190714 9.99 MB/sec execute 582 sec latency 65.581 ms 2024-09-16T00:30:35.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1193598 10.00 MB/sec execute 583 sec latency 28.898 ms 2024-09-16T00:30:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:35 smithi083 ceph-mon[35242]: pgmap v753: 129 pgs: 129 active+clean; 286 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 282 op/s 2024-09-16T00:30:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:35 smithi060 ceph-mon[35185]: pgmap v753: 129 pgs: 129 active+clean; 286 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 282 op/s 2024-09-16T00:30:36.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:35 smithi033 ceph-mon[31243]: pgmap v753: 129 pgs: 129 active+clean; 286 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 282 op/s 2024-09-16T00:30:36.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1196159 10.00 MB/sec execute 584 sec latency 15.324 ms 2024-09-16T00:30:37.592 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 11.4 2024-09-16T00:30:37.592 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:30:37.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1198579 10.01 MB/sec execute 585 sec latency 15.401 ms 2024-09-16T00:30:37.888 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:37 smithi083 ceph-mon[35242]: pgmap v754: 129 pgs: 129 active+clean; 273 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.6 MiB/s wr, 288 op/s 2024-09-16T00:30:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:37 smithi060 ceph-mon[35185]: pgmap v754: 129 pgs: 129 active+clean; 273 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.6 MiB/s wr, 288 op/s 2024-09-16T00:30:38.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:37 smithi033 ceph-mon[31243]: pgmap v754: 129 pgs: 129 active+clean; 273 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.6 MiB/s wr, 288 op/s 2024-09-16T00:30:38.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1201092 10.02 MB/sec execute 586 sec latency 33.255 ms 2024-09-16T00:30:38.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.672+0000 7fd1ff207640 1 -- 172.21.15.33:0/1798575074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1f8106f10 msgr2=0x7fd1f810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:38.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.672+0000 7fd1ff207640 1 --2- 172.21.15.33:0/1798575074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1f8106f10 0x7fd1f810ea90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0009be0 tx=0x7fd1f002a6c0 comp rx=0 tx=0).stop 2024-09-16T00:30:38.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.672+0000 7fd1ff207640 1 -- 172.21.15.33:0/1798575074 shutdown_connections 2024-09-16T00:30:38.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.672+0000 7fd1ff207640 1 --2- 172.21.15.33:0/1798575074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1f8106f10 0x7fd1f810ea90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.672+0000 7fd1ff207640 1 --2- 172.21.15.33:0/1798575074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1f8102790 0x7fd1f81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.672+0000 7fd1ff207640 1 --2- 172.21.15.33:0/1798575074 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f8101de0 0x7fd1f81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.672+0000 7fd1ff207640 1 -- 172.21.15.33:0/1798575074 >> 172.21.15.33:0/1798575074 conn(0x7fd1f80fd8c0 msgr2=0x7fd1f80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.672+0000 7fd1ff207640 1 -- 172.21.15.33:0/1798575074 shutdown_connections 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.672+0000 7fd1ff207640 1 -- 172.21.15.33:0/1798575074 wait complete. 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1ff207640 1 Processor -- start 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1ff207640 1 -- start start 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1ff207640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f8101de0 0x7fd1f80744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1ff207640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1f8102790 0x7fd1f8074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1ff207640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1f8106f10 0x7fd1f8074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1ff207640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f8111f20 con 0x7fd1f8102790 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1ff207640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f8111da0 con 0x7fd1f8106f10 2024-09-16T00:30:38.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1ff207640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f81120a0 con 0x7fd1f8101de0 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1f7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1f8102790 0x7fd1f8074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1f7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1f8102790 0x7fd1f8074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37772/0 (socket says 172.21.15.33:37772) 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1f7fff640 1 -- 172.21.15.33:0/1554772070 learned_addr learned my addr 172.21.15.33:0/1554772070 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1fcfa3640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f8101de0 0x7fd1f80744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.673+0000 7fd1fd7a4640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1f8106f10 0x7fd1f8074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.674+0000 7fd1fcfa3640 1 -- 172.21.15.33:0/1554772070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1f8106f10 msgr2=0x7fd1f8074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.674+0000 7fd1fcfa3640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1f8106f10 0x7fd1f8074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.674+0000 7fd1fcfa3640 1 -- 172.21.15.33:0/1554772070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1f8102790 msgr2=0x7fd1f8074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.674+0000 7fd1fcfa3640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1f8102790 0x7fd1f8074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.674+0000 7fd1fcfa3640 1 -- 172.21.15.33:0/1554772070 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f81a6880 con 0x7fd1f8101de0 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.674+0000 7fd1f7fff640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1f8102790 0x7fd1f8074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.674+0000 7fd1fcfa3640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f8101de0 0x7fd1f80744f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00e9f0 tx=0x7fd1ec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:38.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.674+0000 7fd1f5ffb640 1 -- 172.21.15.33:0/1554772070 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec00cd80 con 0x7fd1f8101de0 2024-09-16T00:30:38.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.674+0000 7fd1f5ffb640 1 -- 172.21.15.33:0/1554772070 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1ec0035f0 con 0x7fd1f8101de0 2024-09-16T00:30:38.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.674+0000 7fd1f5ffb640 1 -- 172.21.15.33:0/1554772070 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec010690 con 0x7fd1f8101de0 2024-09-16T00:30:38.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.674+0000 7fd1ff207640 1 -- 172.21.15.33:0/1554772070 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f81a6b70 con 0x7fd1f8101de0 2024-09-16T00:30:38.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.674+0000 7fd1ff207640 1 -- 172.21.15.33:0/1554772070 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f81a7100 con 0x7fd1f8101de0 2024-09-16T00:30:38.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.675+0000 7fd1ff207640 1 -- 172.21.15.33:0/1554772070 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1f81021c0 con 0x7fd1f8101de0 2024-09-16T00:30:38.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.676+0000 7fd1f5ffb640 1 -- 172.21.15.33:0/1554772070 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd1ec003b30 con 0x7fd1f8101de0 2024-09-16T00:30:38.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.677+0000 7fd1f5ffb640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd1e80a32c0 0x7fd1e80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:38.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.677+0000 7fd1f7fff640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd1e80a32c0 0x7fd1e80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:38.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.677+0000 7fd1f5ffb640 1 -- 172.21.15.33:0/1554772070 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd1ec0cffe0 con 0x7fd1f8101de0 2024-09-16T00:30:38.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.678+0000 7fd1f7fff640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd1e80a32c0 0x7fd1e80a5780 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8075f80 tx=0x7fd1e4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:38.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.681+0000 7fd1f5ffb640 1 -- 172.21.15.33:0/1554772070 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd1ec09c880 con 0x7fd1f8101de0 2024-09-16T00:30:38.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.963+0000 7fd1ff207640 1 -- 172.21.15.33:0/1554772070 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1f8075a20 con 0x7fd1f8101de0 2024-09-16T00:30:38.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.964+0000 7fd1f5ffb640 1 -- 172.21.15.33:0/1554772070 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7fd1ec09c180 con 0x7fd1f8101de0 2024-09-16T00:30:38.965 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:30:38.965 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:30:38.965 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:30:38.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.966+0000 7fd1ff207640 1 -- 172.21.15.33:0/1554772070 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd1e80a32c0 msgr2=0x7fd1e80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:38.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.966+0000 7fd1ff207640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd1e80a32c0 0x7fd1e80a5780 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8075f80 tx=0x7fd1e4009260 comp rx=0 tx=0).stop 2024-09-16T00:30:38.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.966+0000 7fd1ff207640 1 -- 172.21.15.33:0/1554772070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f8101de0 msgr2=0x7fd1f80744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:38.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.966+0000 7fd1ff207640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f8101de0 0x7fd1f80744f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00e9f0 tx=0x7fd1ec00eeb0 comp rx=0 tx=0).stop 2024-09-16T00:30:38.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.966+0000 7fd1ff207640 1 -- 172.21.15.33:0/1554772070 shutdown_connections 2024-09-16T00:30:38.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.966+0000 7fd1ff207640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd1e80a32c0 0x7fd1e80a5780 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:38.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.966+0000 7fd1ff207640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1f8106f10 0x7fd1f8074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:38.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.966+0000 7fd1ff207640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1f8102790 0x7fd1f8074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:38.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.966+0000 7fd1ff207640 1 --2- 172.21.15.33:0/1554772070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f8101de0 0x7fd1f80744f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:38.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.966+0000 7fd1ff207640 1 -- 172.21.15.33:0/1554772070 >> 172.21.15.33:0/1554772070 conn(0x7fd1f80fd8c0 msgr2=0x7fd1f80fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:38.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.967+0000 7fd1ff207640 1 -- 172.21.15.33:0/1554772070 shutdown_connections 2024-09-16T00:30:38.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:38.967+0000 7fd1ff207640 1 -- 172.21.15.33:0/1554772070 wait complete. 2024-09-16T00:30:39.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1203835 10.02 MB/sec execute 587 sec latency 48.412 ms 2024-09-16T00:30:39.804 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 131.4 --await-for 45.0' 2024-09-16T00:30:39.805 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 131.4 --await-for 45.0 2024-09-16T00:30:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:39 smithi083 ceph-mon[35242]: pgmap v755: 129 pgs: 129 active+clean; 269 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 210 op/s 2024-09-16T00:30:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:39 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1554772070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:30:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:39 smithi060 ceph-mon[35185]: pgmap v755: 129 pgs: 129 active+clean; 269 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 210 op/s 2024-09-16T00:30:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:39 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1554772070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:30:40.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:39 smithi033 ceph-mon[31243]: pgmap v755: 129 pgs: 129 active+clean; 269 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 210 op/s 2024-09-16T00:30:40.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:39 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1554772070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:30:40.140 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:40.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.643+0000 7f6722270640 1 -- 172.21.15.33:0/2736159416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f671c102570 msgr2=0x7f671c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:40.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.643+0000 7f6722270640 1 --2- 172.21.15.33:0/2736159416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f671c102570 0x7f671c1029b0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6704009900 tx=0x7f670402a480 comp rx=0 tx=0).stop 2024-09-16T00:30:40.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.643+0000 7f6722270640 1 -- 172.21.15.33:0/2736159416 shutdown_connections 2024-09-16T00:30:40.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.643+0000 7f6722270640 1 --2- 172.21.15.33:0/2736159416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f671c102ef0 0x7f671c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.643+0000 7f6722270640 1 --2- 172.21.15.33:0/2736159416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f671c102570 0x7f671c1029b0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.643+0000 7f6722270640 1 --2- 172.21.15.33:0/2736159416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f671c100e20 0x7f671c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.643+0000 7f6722270640 1 -- 172.21.15.33:0/2736159416 >> 172.21.15.33:0/2736159416 conn(0x7f671c0fc7d0 msgr2=0x7f671c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:40.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.643+0000 7f6722270640 1 -- 172.21.15.33:0/2736159416 shutdown_connections 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.643+0000 7f6722270640 1 -- 172.21.15.33:0/2736159416 wait complete. 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.644+0000 7f6722270640 1 Processor -- start 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.644+0000 7f6722270640 1 -- start start 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f6722270640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f671c100e20 0x7f671c197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f6722270640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f671c102570 0x7f671c197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f6722270640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f671c102ef0 0x7f671c19bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f6722270640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f671c110f80 con 0x7f671c102ef0 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f6722270640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f671c110e00 con 0x7f671c102570 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f6722270640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f671c111100 con 0x7f671c100e20 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f672080d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f671c102ef0 0x7f671c19bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f671bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f671c100e20 0x7f671c197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f672080d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f671c102ef0 0x7f671c19bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43822/0 (socket says 172.21.15.33:43822) 2024-09-16T00:30:40.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f672080d640 1 -- 172.21.15.33:0/3944579549 learned_addr learned my addr 172.21.15.33:0/3944579549 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f6713fff640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f671c102570 0x7f671c197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f672080d640 1 -- 172.21.15.33:0/3944579549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f671c100e20 msgr2=0x7f671c197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f672080d640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f671c100e20 0x7f671c197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f672080d640 1 -- 172.21.15.33:0/3944579549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f671c102570 msgr2=0x7f671c197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f672080d640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f671c102570 0x7f671c197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f672080d640 1 -- 172.21.15.33:0/3944579549 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671c19c3d0 con 0x7f671c102ef0 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f671bfff640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f671c100e20 0x7f671c197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f6713fff640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f671c102570 0x7f671c197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f672080d640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f671c102ef0 0x7f671c19bcf0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f671400eb10 tx=0x7f671400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f6719ffb640 1 -- 172.21.15.33:0/3944579549 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6714004b20 con 0x7f671c102ef0 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.645+0000 7f6719ffb640 1 -- 172.21.15.33:0/3944579549 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6714004cc0 con 0x7f671c102ef0 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.646+0000 7f6722270640 1 -- 172.21.15.33:0/3944579549 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f671c19c6c0 con 0x7f671c102ef0 2024-09-16T00:30:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.646+0000 7f6719ffb640 1 -- 172.21.15.33:0/3944579549 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6714010760 con 0x7f671c102ef0 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.646+0000 7f6722270640 1 -- 172.21.15.33:0/3944579549 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f671c1a3fa0 con 0x7f671c102ef0 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.648+0000 7f6719ffb640 1 -- 172.21.15.33:0/3944579549 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6714005020 con 0x7f671c102ef0 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.648+0000 7f6724649740 1 Processor -- start 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6724649740 1 -- start start 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6724649740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3660 0x55efcbba3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6724649740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3f80 0x55efcbbb0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6724649740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 0x55efcbbb2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6724649740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55efcbb092a0 con 0x55efcbbb0b50 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6724649740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55efcbab0b10 con 0x55efcbba3f80 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6724649740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55efcbb05260 con 0x55efcbba3660 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6713fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3660 0x55efcbba3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6713fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 0x55efcbbb2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f671bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3f80 0x55efcbbb0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6713fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 0x55efcbbb2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43832/0 (socket says 172.21.15.33:43832) 2024-09-16T00:30:40.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6713fff640 1 -- 172.21.15.33:0/2141959823 learned_addr learned my addr 172.21.15.33:0/2141959823 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:40.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6719ffb640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f67080a3240 0x7f67080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f672080d640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f67080a3240 0x7f67080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f6719ffb640 1 -- 172.21.15.33:0/3944579549 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f67140d0330 con 0x7f671c102ef0 2024-09-16T00:30:40.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.649+0000 7f671bfff640 1 -- 172.21.15.33:0/2141959823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3660 msgr2=0x55efcbba3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:40.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.650+0000 7f671bfff640 1 --2- 172.21.15.33:0/2141959823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3660 0x55efcbba3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.650+0000 7f671bfff640 1 -- 172.21.15.33:0/2141959823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 msgr2=0x55efcbbb2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:40.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.650+0000 7f671bfff640 1 --2- 172.21.15.33:0/2141959823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 0x55efcbbb2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.650+0000 7f671bfff640 1 -- 172.21.15.33:0/2141959823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efcb87ab90 con 0x55efcbba3f80 2024-09-16T00:30:40.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.650+0000 7f671bfff640 1 --2- 172.21.15.33:0/2141959823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3f80 0x55efcbbb0610 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f670c009fd0 tx=0x7f670c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c368af770e32d69 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:40.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.650+0000 7f672080d640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f67080a3240 0x7f67080a5700 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6714098930 tx=0x7f67140919b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:40.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.650+0000 7f67127fc640 1 -- 172.21.15.33:0/2141959823 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c004310 con 0x55efcbba3f80 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.650+0000 7f67127fc640 1 -- 172.21.15.33:0/2141959823 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f670c0044b0 con 0x55efcbba3f80 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.650+0000 7f67127fc640 1 -- 172.21.15.33:0/2141959823 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c0035f0 con 0x55efcbba3f80 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.651+0000 7f6724649740 1 -- 172.21.15.33:0/2141959823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3f80 msgr2=0x55efcbbb0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.651+0000 7f6724649740 1 --2- 172.21.15.33:0/2141959823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3f80 0x55efcbbb0610 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f670c009fd0 tx=0x7f670c0118d0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.651+0000 7f6724649740 1 -- 172.21.15.33:0/2141959823 shutdown_connections 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.651+0000 7f6724649740 1 --2- 172.21.15.33:0/2141959823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 0x55efcbbb2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.651+0000 7f6724649740 1 --2- 172.21.15.33:0/2141959823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3f80 0x55efcbbb0610 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.651+0000 7f6724649740 1 --2- 172.21.15.33:0/2141959823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3660 0x55efcbba3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.651+0000 7f6724649740 1 -- 172.21.15.33:0/2141959823 >> 172.21.15.33:0/2141959823 conn(0x55efcbb9fcf0 msgr2=0x55efcbba00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.651+0000 7f6724649740 1 -- 172.21.15.33:0/2141959823 shutdown_connections 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.651+0000 7f6724649740 1 -- 172.21.15.33:0/2141959823 wait complete. 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.651+0000 7f6724649740 1 -- start start 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6724649740 1 Processor -- start 2024-09-16T00:30:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6724649740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3660 0x55efcbbada60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6724649740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3f80 0x55efcbbadfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6724649740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 0x55efcbbae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6724649740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55efcbbb3780 con 0x55efcbbb0b50 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6724649740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55efcba916e0 con 0x55efcbba3660 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6724649740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55efcbbb3900 con 0x55efcbba3f80 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6724649740 10 client.0 _ll_register_callbacks cb 0x55efcbab5a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6713fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 0x55efcbbae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6713fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 0x55efcbbae4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43848/0 (socket says 172.21.15.33:43848) 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6713fff640 1 -- 172.21.15.33:0/1402520315 learned_addr learned my addr 172.21.15.33:0/1402520315 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f671bfff640 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3f80 0x55efcbbadfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6713fff640 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3660 0x55efcbbada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f671bfff640 1 -- 172.21.15.33:0/1402520315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3660 msgr2=0x55efcbbada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f671bfff640 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3660 0x55efcbbada60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f671bfff640 1 -- 172.21.15.33:0/1402520315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 msgr2=0x55efcbbae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f671bfff640 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 0x55efcbbae4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f671bfff640 1 -- 172.21.15.33:0/1402520315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efcbbbcce0 con 0x55efcbba3f80 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.652+0000 7f6713fff640 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 0x55efcbbae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f671bfff640 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3f80 0x55efcbbadfa0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f670c012fd0 tx=0x7f670c0048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=118bc2b517093af8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f66fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f66fb7fe640 1 -- 172.21.15.33:0/1402520315 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c029070 con 0x55efcbba3f80 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f6724649740 1 -- 172.21.15.33:0/1402520315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f670c0044b0 con 0x55efcbba3f80 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f66fb7fe640 1 -- 172.21.15.33:0/1402520315 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f670c003a00 con 0x55efcbba3f80 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f66fb7fe640 1 -- 172.21.15.33:0/1402520315 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f670c035050 con 0x55efcbba3f80 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f6724649740 10 client.27283 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f6724649740 1 -- 172.21.15.33:0/1402520315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55efcbbbf670 con 0x55efcbba3f80 2024-09-16T00:30:40.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f66fb7fe640 1 -- 172.21.15.33:0/1402520315 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f670c0044b0 con 0x55efcbba3f80 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f66fb7fe640 1 -- 172.21.15.33:0/1402520315 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f670c00f480 con 0x55efcbba3f80 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f6724649740 10 client.27283 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f6724649740 10 client.27283 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f6724649740 20 client.27283 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.653+0000 7f6724649740 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55efcbbc0a20 0x55efcbbc2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.654+0000 7f6724649740 4 client.27283 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.654+0000 7f6724649740 1 -- 172.21.15.33:0/1402520315 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efcbb169d0 con 0x55efcbbc0a20 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.654+0000 7f6713fff640 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55efcbbc0a20 0x55efcbbc2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.654+0000 7f66fb7fe640 10 client.27283 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.654+0000 7f6713fff640 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55efcbbc0a20 0x55efcbbc2e10 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f618399b889a36c server_cookie=8bd6d6fddc207a68 in_seq=0 out_seq=0 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f66fb7fe640 1 -- 172.21.15.33:0/1402520315 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55efcbb169d0 con 0x55efcbbc0a20 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f66fb7fe640 10 client.27283 handle_command_reply: tid=0 2024-09-16T00:30:40.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f6724649740 1 client.27283 shutdown 2024-09-16T00:30:40.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f6724649740 1 -- 172.21.15.33:0/1402520315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3f80 msgr2=0x55efcbbadfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:40.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f6724649740 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3f80 0x55efcbbadfa0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f670c012fd0 tx=0x7f670c0048f0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f6724649740 1 -- 172.21.15.33:0/1402520315 shutdown_connections 2024-09-16T00:30:40.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f6724649740 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55efcbbc0a20 0x55efcbbc2e10 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f6724649740 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbbb0b50 0x55efcbbae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f6724649740 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3f80 0x55efcbbadfa0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f66fb7fe640 0 client.27283 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:40.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f6724649740 1 --2- 172.21.15.33:0/1402520315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3660 0x55efcbbada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f6724649740 1 -- 172.21.15.33:0/1402520315 >> 172.21.15.33:0/1402520315 conn(0x55efcbb9fcf0 msgr2=0x55efcbba5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:40.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f6724649740 1 -- 172.21.15.33:0/1402520315 shutdown_connections 2024-09-16T00:30:40.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f6724649740 1 -- 172.21.15.33:0/1402520315 wait complete. 2024-09-16T00:30:40.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.655+0000 7f6724649740 20 client.27283 trim_cache size 0 max 16384 2024-09-16T00:30:40.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f6724649740 1 Processor -- start 2024-09-16T00:30:40.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f6724649740 1 -- start start 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f6724649740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3660 0x55efcbbafd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f6724649740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 0x55efcbbc8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f6724649740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbbb0b50 0x55efcbbc8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f6724649740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f671c1af740 con 0x55efcbba3f80 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f6724649740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f671c1ad010 con 0x55efcbbb0b50 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f6724649740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f671c1aae60 con 0x55efcbba3660 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f671bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 0x55efcbbc8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f671bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 0x55efcbbc8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43854/0 (socket says 172.21.15.33:43854) 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f671bfff640 1 -- 172.21.15.33:0/1455532860 learned_addr learned my addr 172.21.15.33:0/1455532860 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f6713fff640 1 --2- 172.21.15.33:0/1455532860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3660 0x55efcbbafd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.674+0000 7f6713fff640 1 --2- 172.21.15.33:0/1455532860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbbb0b50 0x55efcbbc8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.675+0000 7f671bfff640 1 -- 172.21.15.33:0/1455532860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3660 msgr2=0x55efcbbafd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.675+0000 7f671bfff640 1 --2- 172.21.15.33:0/1455532860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3660 0x55efcbbafd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.675+0000 7f671bfff640 1 -- 172.21.15.33:0/1455532860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbbb0b50 msgr2=0x55efcbbc8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:40.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.675+0000 7f671bfff640 1 --2- 172.21.15.33:0/1455532860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbbb0b50 0x55efcbbc8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.675+0000 7f671bfff640 1 -- 172.21.15.33:0/1455532860 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efcbb9b260 con 0x55efcbba3f80 2024-09-16T00:30:40.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.675+0000 7f671bfff640 1 --2- 172.21.15.33:0/1455532860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 0x55efcbbc8580 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f670c00f880 tx=0x7f670c005990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67172feb2a3acceb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:40.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.675+0000 7f6711ffb640 1 -- 172.21.15.33:0/1455532860 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c03e070 con 0x55efcbba3f80 2024-09-16T00:30:40.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.675+0000 7f6711ffb640 1 -- 172.21.15.33:0/1455532860 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f670c004960 con 0x55efcbba3f80 2024-09-16T00:30:40.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.675+0000 7f6711ffb640 1 -- 172.21.15.33:0/1455532860 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c032030 con 0x55efcbba3f80 2024-09-16T00:30:40.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.676+0000 7f6724649740 1 -- 172.21.15.33:0/1455532860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 msgr2=0x55efcbbc8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.676+0000 7f6724649740 1 --2- 172.21.15.33:0/1455532860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 0x55efcbbc8580 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f670c00f880 tx=0x7f670c005990 comp rx=0 tx=0).stop 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.676+0000 7f6724649740 1 -- 172.21.15.33:0/1455532860 shutdown_connections 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.676+0000 7f6724649740 1 --2- 172.21.15.33:0/1455532860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbbb0b50 0x55efcbbc8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.676+0000 7f6724649740 1 --2- 172.21.15.33:0/1455532860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 0x55efcbbc8580 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.676+0000 7f6724649740 1 --2- 172.21.15.33:0/1455532860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbba3660 0x55efcbbafd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.676+0000 7f6724649740 1 -- 172.21.15.33:0/1455532860 >> 172.21.15.33:0/1455532860 conn(0x55efcbb9fcf0 msgr2=0x55efcbba5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.676+0000 7f6724649740 1 -- 172.21.15.33:0/1455532860 shutdown_connections 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.676+0000 7f6724649740 1 -- 172.21.15.33:0/1455532860 wait complete. 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f6724649740 1 -- start start 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f6724649740 1 Processor -- start 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f6724649740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3660 0x55efcbbb4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f6724649740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 0x55efcbbb5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f6724649740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbbb0b50 0x55efcbbb56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f6724649740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f671c1ab4d0 con 0x55efcbba3f80 2024-09-16T00:30:40.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f6724649740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f671c1ac940 con 0x55efcbba3660 2024-09-16T00:30:40.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f6724649740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f671c1abba0 con 0x55efcbbb0b50 2024-09-16T00:30:40.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f6724649740 10 client.0 _ll_register_callbacks cb 0x7f670c009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:40.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f671bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 0x55efcbbb5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f671bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 0x55efcbbb5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43858/0 (socket says 172.21.15.33:43858) 2024-09-16T00:30:40.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f6713fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3660 0x55efcbbb4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f671bfff640 1 -- 172.21.15.33:0/2436422314 learned_addr learned my addr 172.21.15.33:0/2436422314 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:40.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.677+0000 7f6713fff640 1 --2- 172.21.15.33:0/2436422314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbbb0b50 0x55efcbbb56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.678+0000 7f6713fff640 1 -- 172.21.15.33:0/2436422314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbbb0b50 msgr2=0x55efcbbb56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:40.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.678+0000 7f6713fff640 1 --2- 172.21.15.33:0/2436422314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbbb0b50 0x55efcbbb56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.678+0000 7f6713fff640 1 -- 172.21.15.33:0/2436422314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 msgr2=0x55efcbbb5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:40.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.678+0000 7f6713fff640 1 --2- 172.21.15.33:0/2436422314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 0x55efcbbb5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.678+0000 7f6713fff640 1 -- 172.21.15.33:0/2436422314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efcbba0e80 con 0x55efcbba3660 2024-09-16T00:30:40.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.678+0000 7f6713fff640 1 --2- 172.21.15.33:0/2436422314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3660 0x55efcbbb4bf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6704006010 tx=0x7f670c009be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9e2867f2650eb38 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:40.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.678+0000 7f66fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:30:40.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.678+0000 7f66fb7fe640 1 -- 172.21.15.33:0/2436422314 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6704035040 con 0x55efcbba3660 2024-09-16T00:30:40.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.678+0000 7f66fb7fe640 1 -- 172.21.15.33:0/2436422314 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f67040317c0 con 0x55efcbba3660 2024-09-16T00:30:40.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.678+0000 7f6724649740 1 -- 172.21.15.33:0/2436422314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f670c004960 con 0x55efcbba3660 2024-09-16T00:30:40.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.679+0000 7f66fb7fe640 1 -- 172.21.15.33:0/2436422314 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6704005580 con 0x55efcbba3660 2024-09-16T00:30:40.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.679+0000 7f6724649740 10 client.27290 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:40.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.679+0000 7f6724649740 1 -- 172.21.15.33:0/2436422314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55efcbba0980 con 0x55efcbba3660 2024-09-16T00:30:40.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.679+0000 7f66fb7fe640 1 -- 172.21.15.33:0/2436422314 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f670c004960 con 0x55efcbba3660 2024-09-16T00:30:40.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.679+0000 7f66fb7fe640 1 -- 172.21.15.33:0/2436422314 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f670404e430 con 0x55efcbba3660 2024-09-16T00:30:40.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.679+0000 7f6724649740 10 client.27290 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:40.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.679+0000 7f6724649740 10 client.27290 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:30:40.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.679+0000 7f6724649740 20 client.27290 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:40.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.679+0000 7f6724649740 1 --2- 172.21.15.33:0/2436422314 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55efcbc2bf10 0x55efcbc2e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:40.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.679+0000 7f6724649740 4 client.27290 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 131.4, "await_for": 45.0}] 2024-09-16T00:30:40.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.679+0000 7f6724649740 1 -- 172.21.15.33:0/2436422314 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 131.4, "await_for": 45.0}) -- 0x55efcbb10d20 con 0x55efcbc2bf10 2024-09-16T00:30:40.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.679+0000 7f671bfff640 1 --2- 172.21.15.33:0/2436422314 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55efcbc2bf10 0x55efcbc2e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:40.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.680+0000 7f66fb7fe640 10 client.27290 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:40.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.680+0000 7f671bfff640 1 --2- 172.21.15.33:0/2436422314 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55efcbc2bf10 0x55efcbc2e300 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f911484b570159f1 server_cookie=ae95343050d8a7ca in_seq=0 out_seq=0 2024-09-16T00:30:40.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1205614 10.02 MB/sec execute 588 sec latency 35.548 ms 2024-09-16T00:30:40.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f66fb7fe640 1 -- 172.21.15.33:0/2436422314 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55efcbb10d20 con 0x55efcbc2bf10 2024-09-16T00:30:40.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f66fb7fe640 10 client.27290 handle_command_reply: tid=0 2024-09-16T00:30:40.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f6724649740 1 client.27290 shutdown 2024-09-16T00:30:40.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f6724649740 1 -- 172.21.15.33:0/2436422314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3660 msgr2=0x55efcbbb4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:40.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f6724649740 1 --2- 172.21.15.33:0/2436422314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3660 0x55efcbbb4bf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6704006010 tx=0x7f670c009be0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f6724649740 1 -- 172.21.15.33:0/2436422314 shutdown_connections 2024-09-16T00:30:40.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f6724649740 1 --2- 172.21.15.33:0/2436422314 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55efcbc2bf10 0x55efcbc2e300 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f6724649740 1 --2- 172.21.15.33:0/2436422314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55efcbbb0b50 0x55efcbbb56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f6724649740 1 --2- 172.21.15.33:0/2436422314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55efcbba3f80 0x55efcbbb5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f66fb7fe640 0 client.27290 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:40.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f6724649740 1 --2- 172.21.15.33:0/2436422314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55efcbba3660 0x55efcbbb4bf0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f6724649740 1 -- 172.21.15.33:0/2436422314 >> 172.21.15.33:0/2436422314 conn(0x55efcbb9fcf0 msgr2=0x55efcbb9e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:40.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f6724649740 1 -- 172.21.15.33:0/2436422314 shutdown_connections 2024-09-16T00:30:40.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f6724649740 1 -- 172.21.15.33:0/2436422314 wait complete. 2024-09-16T00:30:40.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.756+0000 7f6724649740 20 client.27290 trim_cache size 0 max 16384 2024-09-16T00:30:40.758 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:30:40.758 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:30:40.758 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:30:40.758 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 46, 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: "e067ca76": { 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 46, 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 131.4, 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:30:40.759 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:30:40.760 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:30:40.760 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:30:40.760 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:30:40.760 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:30:40.760 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:30:40.760 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:30:40.760 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:30:40.760 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:30:40.760 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:30:40.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.757+0000 7f6722270640 1 -- 172.21.15.33:0/3944579549 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f67080a3240 msgr2=0x7f67080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:40.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.757+0000 7f6722270640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f67080a3240 0x7f67080a5700 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6714098930 tx=0x7f67140919b0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.757+0000 7f6722270640 1 -- 172.21.15.33:0/3944579549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f671c102ef0 msgr2=0x7f671c19bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:40.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.757+0000 7f6722270640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f671c102ef0 0x7f671c19bcf0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f671400eb10 tx=0x7f671400c2d0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.757+0000 7f6722270640 1 -- 172.21.15.33:0/3944579549 shutdown_connections 2024-09-16T00:30:40.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.757+0000 7f6722270640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f67080a3240 0x7f67080a5700 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.757+0000 7f6722270640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f671c102ef0 0x7f671c19bcf0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.757+0000 7f6722270640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f671c102570 0x7f671c197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.757+0000 7f6722270640 1 --2- 172.21.15.33:0/3944579549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f671c100e20 0x7f671c197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:40.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.757+0000 7f6722270640 1 -- 172.21.15.33:0/3944579549 >> 172.21.15.33:0/3944579549 conn(0x7f671c0fc7d0 msgr2=0x7f671c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:40.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.757+0000 7f6722270640 1 -- 172.21.15.33:0/3944579549 shutdown_connections 2024-09-16T00:30:40.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:40.757+0000 7f6722270640 1 -- 172.21.15.33:0/3944579549 wait complete. 2024-09-16T00:30:41.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1205642 10.01 MB/sec execute 589 sec latency 934.254 ms 2024-09-16T00:30:41.704 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'e067ca76', quiesce took 4.1 seconds. Will release after: 7.300000000000001 2024-09-16T00:30:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:41 smithi083 ceph-mon[35242]: pgmap v756: 129 pgs: 129 active+clean; 269 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 234 op/s 2024-09-16T00:30:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:41 smithi060 ceph-mon[35185]: pgmap v756: 129 pgs: 129 active+clean; 269 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 234 op/s 2024-09-16T00:30:42.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:41 smithi033 ceph-mon[31243]: pgmap v756: 129 pgs: 129 active+clean; 269 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 234 op/s 2024-09-16T00:30:42.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1205642 9.99 MB/sec execute 590 sec latency 1934.611 ms 2024-09-16T00:30:43.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1205642 9.97 MB/sec execute 591 sec latency 2934.655 ms 2024-09-16T00:30:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:43 smithi083 ceph-mon[35242]: pgmap v757: 129 pgs: 129 active+clean; 272 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.6 MiB/s wr, 288 op/s 2024-09-16T00:30:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:43 smithi060 ceph-mon[35185]: pgmap v757: 129 pgs: 129 active+clean; 272 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.6 MiB/s wr, 288 op/s 2024-09-16T00:30:44.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:43 smithi033 ceph-mon[31243]: pgmap v757: 129 pgs: 129 active+clean; 272 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.6 MiB/s wr, 288 op/s 2024-09-16T00:30:44.689 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1205642 9.96 MB/sec execute 592 sec latency 3934.697 ms 2024-09-16T00:30:45.690 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1205642 9.94 MB/sec execute 593 sec latency 4934.789 ms 2024-09-16T00:30:45.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:45 smithi083 ceph-mon[35242]: pgmap v758: 129 pgs: 129 active+clean; 272 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 263 op/s 2024-09-16T00:30:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:45 smithi060 ceph-mon[35185]: pgmap v758: 129 pgs: 129 active+clean; 272 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 263 op/s 2024-09-16T00:30:46.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:45 smithi033 ceph-mon[31243]: pgmap v758: 129 pgs: 129 active+clean; 272 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 263 op/s 2024-09-16T00:30:46.690 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1205642 9.92 MB/sec execute 594 sec latency 5934.850 ms 2024-09-16T00:30:47.690 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1205642 9.91 MB/sec execute 595 sec latency 6934.924 ms 2024-09-16T00:30:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:47 smithi083 ceph-mon[35242]: pgmap v759: 129 pgs: 129 active+clean; 276 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-09-16T00:30:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:47 smithi060 ceph-mon[35185]: pgmap v759: 129 pgs: 129 active+clean; 276 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-09-16T00:30:48.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:47 smithi033 ceph-mon[31243]: pgmap v759: 129 pgs: 129 active+clean; 276 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-09-16T00:30:48.453 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:30:48.690 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1205642 9.89 MB/sec execute 596 sec latency 7934.972 ms 2024-09-16T00:30:48.712 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:49.005 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:30:49.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.204+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2403307718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9df01036e0 msgr2=0x7f9df010bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:49.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.204+0000 7f9df7f3c640 1 --2- 172.21.15.33:0/2403307718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9df01036e0 0x7f9df010bbd0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009900 tx=0x7f9de002a480 comp rx=0 tx=0).stop 2024-09-16T00:30:49.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.204+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2403307718 shutdown_connections 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.204+0000 7f9df7f3c640 1 --2- 172.21.15.33:0/2403307718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df010c1a0 0x7f9df010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.204+0000 7f9df7f3c640 1 --2- 172.21.15.33:0/2403307718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9df01036e0 0x7f9df010bbd0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.204+0000 7f9df7f3c640 1 --2- 172.21.15.33:0/2403307718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df0102dc0 0x7f9df01031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.204+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2403307718 >> 172.21.15.33:0/2403307718 conn(0x7f9df00fc7a0 msgr2=0x7f9df00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.204+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2403307718 shutdown_connections 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.204+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2403307718 wait complete. 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.205+0000 7f9df7f3c640 1 Processor -- start 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.205+0000 7f9df7f3c640 1 -- start start 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.205+0000 7f9df7f3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9df0102dc0 0x7f9df0100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.205+0000 7f9df7f3c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df01036e0 0x7f9df0100f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.205+0000 7f9df7f3c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df010c1a0 0x7f9df01014a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.205+0000 7f9df7f3c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9df0111f70 con 0x7f9df0102dc0 2024-09-16T00:30:49.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.205+0000 7f9df7f3c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9df0111df0 con 0x7f9df010c1a0 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.205+0000 7f9df7f3c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01120f0 con 0x7f9df01036e0 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.205+0000 7f9df5cd8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9df0102dc0 0x7f9df0100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.205+0000 7f9df5cd8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9df0102dc0 0x7f9df0100a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43880/0 (socket says 172.21.15.33:43880) 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.205+0000 7f9df5cd8640 1 -- 172.21.15.33:0/2179106525 learned_addr learned my addr 172.21.15.33:0/2179106525 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.205+0000 7f9df54d7640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df01036e0 0x7f9df0100f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.206+0000 7f9df64d9640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df010c1a0 0x7f9df01014a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.206+0000 7f9df5cd8640 1 -- 172.21.15.33:0/2179106525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df01036e0 msgr2=0x7f9df0100f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.206+0000 7f9df5cd8640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df01036e0 0x7f9df0100f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.206+0000 7f9df5cd8640 1 -- 172.21.15.33:0/2179106525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df010c1a0 msgr2=0x7f9df01014a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.206+0000 7f9df5cd8640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df010c1a0 0x7f9df01014a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.206+0000 7f9df5cd8640 1 -- 172.21.15.33:0/2179106525 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df01a68e0 con 0x7f9df0102dc0 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.206+0000 7f9df5cd8640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9df0102dc0 0x7f9df0100a20 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00d950 tx=0x7f9dec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:49.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.206+0000 7f9de6ffd640 1 -- 172.21.15.33:0/2179106525 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec017070 con 0x7f9df0102dc0 2024-09-16T00:30:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.206+0000 7f9de6ffd640 1 -- 172.21.15.33:0/2179106525 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dec009c30 con 0x7f9df0102dc0 2024-09-16T00:30:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.206+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2179106525 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df01a6bd0 con 0x7f9df0102dc0 2024-09-16T00:30:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.206+0000 7f9de6ffd640 1 -- 172.21.15.33:0/2179106525 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec00f6b0 con 0x7f9df0102dc0 2024-09-16T00:30:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.206+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2179106525 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df01a70e0 con 0x7f9df0102dc0 2024-09-16T00:30:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.209+0000 7f9de6ffd640 1 -- 172.21.15.33:0/2179106525 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9dec0109d0 con 0x7f9df0102dc0 2024-09-16T00:30:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.209+0000 7f9dfa315740 1 Processor -- start 2024-09-16T00:30:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.209+0000 7f9dfa315740 1 -- start start 2024-09-16T00:30:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9dfa315740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 0x55bdaba08880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9dfa315740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba13140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9dfa315740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba13680 0x55bdaba15a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9dfa315740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bdab96e2e0 con 0x55bdaba084a0 2024-09-16T00:30:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9dfa315740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55bdab915ce0 con 0x55bdaba08dc0 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9dfa315740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bdab96a2a0 con 0x55bdaba13680 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9df64d9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 0x55bdaba08880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9df64d9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 0x55bdaba08880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43896/0 (socket says 172.21.15.33:43896) 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9df64d9640 1 -- 172.21.15.33:0/3544224239 learned_addr learned my addr 172.21.15.33:0/3544224239 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9df5cd8640 1 --2- 172.21.15.33:0/3544224239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba13140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9df54d7640 1 --2- 172.21.15.33:0/3544224239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba13680 0x55bdaba15a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9de6ffd640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9ddc0a3240 0x7f9ddc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9df64d9640 1 -- 172.21.15.33:0/3544224239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba13680 msgr2=0x55bdaba15a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9df64d9640 1 --2- 172.21.15.33:0/3544224239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba13680 0x55bdaba15a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9df64d9640 1 -- 172.21.15.33:0/3544224239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 msgr2=0x55bdaba13140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9df64d9640 1 --2- 172.21.15.33:0/3544224239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba13140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9df64d9640 1 -- 172.21.15.33:0/3544224239 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdab8fa440 con 0x55bdaba084a0 2024-09-16T00:30:49.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9de6ffd640 1 -- 172.21.15.33:0/2179106525 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9dec0cf9e0 con 0x7f9df0102dc0 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9df64d9640 1 --2- 172.21.15.33:0/3544224239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 0x55bdaba08880 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009f40 tx=0x7f9de8010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b5e7a0521edc73 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.210+0000 7f9df64d9640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9ddc0a3240 0x7f9ddc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.211+0000 7f9ddbfff640 1 -- 172.21.15.33:0/3544224239 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8004040 con 0x55bdaba084a0 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.211+0000 7f9ddbfff640 1 -- 172.21.15.33:0/3544224239 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9de8004740 con 0x55bdaba084a0 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.211+0000 7f9ddbfff640 1 -- 172.21.15.33:0/3544224239 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de801e680 con 0x55bdaba084a0 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.211+0000 7f9dfa315740 1 -- 172.21.15.33:0/3544224239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 msgr2=0x55bdaba08880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.211+0000 7f9dfa315740 1 --2- 172.21.15.33:0/3544224239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 0x55bdaba08880 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009f40 tx=0x7f9de8010860 comp rx=0 tx=0).stop 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.211+0000 7f9df64d9640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9ddc0a3240 0x7f9ddc0a5700 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009fd0 tx=0x7f9de801eeb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 -- 172.21.15.33:0/3544224239 shutdown_connections 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 --2- 172.21.15.33:0/3544224239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba13680 0x55bdaba15a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 --2- 172.21.15.33:0/3544224239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba13140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 --2- 172.21.15.33:0/3544224239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 0x55bdaba08880 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 -- 172.21.15.33:0/3544224239 >> 172.21.15.33:0/3544224239 conn(0x55bdaba04a30 msgr2=0x55bdaba04e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 -- 172.21.15.33:0/3544224239 shutdown_connections 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 -- 172.21.15.33:0/3544224239 wait complete. 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 -- start start 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 Processor -- start 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba084a0 0x55bdaba12f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba101d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 0x55bdaba10710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bdaba16250 con 0x55bdaba13680 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55bdab929050 con 0x55bdaba08dc0 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.212+0000 7f9dfa315740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bdaba163d0 con 0x55bdaba084a0 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9dfa315740 10 client.0 _ll_register_callbacks cb 0x55bdab969160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9df64d9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 0x55bdaba10710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9df64d9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 0x55bdaba10710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43900/0 (socket says 172.21.15.33:43900) 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9df64d9640 1 -- 172.21.15.33:0/3665672164 learned_addr learned my addr 172.21.15.33:0/3665672164 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9df5cd8640 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba084a0 0x55bdaba12f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9df54d7640 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9df64d9640 1 -- 172.21.15.33:0/3665672164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba084a0 msgr2=0x55bdaba12f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9df64d9640 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba084a0 0x55bdaba12f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9df64d9640 1 -- 172.21.15.33:0/3665672164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 msgr2=0x55bdaba101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9df64d9640 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9df64d9640 1 -- 172.21.15.33:0/3665672164 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdab6df3c0 con 0x55bdaba13680 2024-09-16T00:30:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9df64d9640 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 0x55bdaba10710 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009d40 tx=0x7f9de8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2574b0e50a40faea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9dd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:30:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9dd97fa640 1 -- 172.21.15.33:0/3665672164 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8037060 con 0x55bdaba13680 2024-09-16T00:30:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9dd97fa640 1 -- 172.21.15.33:0/3665672164 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9de8011500 con 0x55bdaba13680 2024-09-16T00:30:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9dfa315740 1 -- 172.21.15.33:0/3665672164 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9de8004740 con 0x55bdaba13680 2024-09-16T00:30:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.213+0000 7f9dd97fa640 1 -- 172.21.15.33:0/3665672164 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9de801e420 con 0x55bdaba13680 2024-09-16T00:30:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.214+0000 7f9dfa315740 10 client.17928 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.214+0000 7f9dfa315740 1 -- 172.21.15.33:0/3665672164 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55bdaba21bc0 con 0x55bdaba13680 2024-09-16T00:30:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.214+0000 7f9dd97fa640 1 -- 172.21.15.33:0/3665672164 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9de8004740 con 0x55bdaba13680 2024-09-16T00:30:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.214+0000 7f9dd97fa640 1 -- 172.21.15.33:0/3665672164 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f9de801d970 con 0x55bdaba13680 2024-09-16T00:30:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.214+0000 7f9dfa315740 10 client.17928 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.214+0000 7f9dfa315740 10 client.17928 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:30:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.214+0000 7f9dfa315740 20 client.17928 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.214+0000 7f9dfa315740 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bdaba23080 0x55bdaba25470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.214+0000 7f9dfa315740 4 client.17928 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:30:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.214+0000 7f9dfa315740 1 -- 172.21.15.33:0/3665672164 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdab97b850 con 0x55bdaba23080 2024-09-16T00:30:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.214+0000 7f9df5cd8640 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bdaba23080 0x55bdaba25470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.215+0000 7f9df5cd8640 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bdaba23080 0x55bdaba25470 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fab419cca1472f8e server_cookie=c0fe72b37e4258d7 in_seq=0 out_seq=0 2024-09-16T00:30:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.215+0000 7f9dd97fa640 10 client.17928 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.215+0000 7f9dd97fa640 1 -- 172.21.15.33:0/3665672164 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bdab97b850 con 0x55bdaba23080 2024-09-16T00:30:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.215+0000 7f9dd97fa640 10 client.17928 handle_command_reply: tid=0 2024-09-16T00:30:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.215+0000 7f9dfa315740 1 client.17928 shutdown 2024-09-16T00:30:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.215+0000 7f9dfa315740 1 -- 172.21.15.33:0/3665672164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 msgr2=0x55bdaba10710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.215+0000 7f9dfa315740 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 0x55bdaba10710 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009d40 tx=0x7f9de8017000 comp rx=0 tx=0).stop 2024-09-16T00:30:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.216+0000 7f9dfa315740 1 -- 172.21.15.33:0/3665672164 shutdown_connections 2024-09-16T00:30:49.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.216+0000 7f9dfa315740 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bdaba23080 0x55bdaba25470 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.216+0000 7f9dfa315740 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 0x55bdaba10710 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.216+0000 7f9dfa315740 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.216+0000 7f9dfa315740 1 --2- 172.21.15.33:0/3665672164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba084a0 0x55bdaba12f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.216+0000 7f9dd97fa640 0 client.17928 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:49.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.216+0000 7f9dfa315740 1 -- 172.21.15.33:0/3665672164 >> 172.21.15.33:0/3665672164 conn(0x55bdaba04a30 msgr2=0x55bdaba0a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:49.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.216+0000 7f9dfa315740 1 -- 172.21.15.33:0/3665672164 shutdown_connections 2024-09-16T00:30:49.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.216+0000 7f9dfa315740 1 -- 172.21.15.33:0/3665672164 wait complete. 2024-09-16T00:30:49.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.216+0000 7f9dfa315740 20 client.17928 trim_cache size 0 max 16384 2024-09-16T00:30:49.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.233+0000 7f9dfa315740 1 Processor -- start 2024-09-16T00:30:49.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.233+0000 7f9dfa315740 1 -- start start 2024-09-16T00:30:49.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.233+0000 7f9dfa315740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba084a0 0x55bdaba1f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.233+0000 7f9dfa315740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba08dc0 0x55bdaba2aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.233+0000 7f9dfa315740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 0x55bdaba2afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.233+0000 7f9dfa315740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01b3bc0 con 0x55bdaba13680 2024-09-16T00:30:49.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.233+0000 7f9dfa315740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01b1490 con 0x55bdaba084a0 2024-09-16T00:30:49.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.233+0000 7f9dfa315740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01af2e0 con 0x55bdaba08dc0 2024-09-16T00:30:49.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9df64d9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 0x55bdaba2afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9df64d9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 0x55bdaba2afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43904/0 (socket says 172.21.15.33:43904) 2024-09-16T00:30:49.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9df64d9640 1 -- 172.21.15.33:0/539764489 learned_addr learned my addr 172.21.15.33:0/539764489 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:49.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9df5cd8640 1 --2- 172.21.15.33:0/539764489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba084a0 0x55bdaba1f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9df54d7640 1 --2- 172.21.15.33:0/539764489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba08dc0 0x55bdaba2aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9df64d9640 1 -- 172.21.15.33:0/539764489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba08dc0 msgr2=0x55bdaba2aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9df64d9640 1 --2- 172.21.15.33:0/539764489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba08dc0 0x55bdaba2aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9df64d9640 1 -- 172.21.15.33:0/539764489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba084a0 msgr2=0x55bdaba1f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9df64d9640 1 --2- 172.21.15.33:0/539764489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba084a0 0x55bdaba1f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9df64d9640 1 -- 172.21.15.33:0/539764489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdaba04860 con 0x55bdaba13680 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9df64d9640 1 --2- 172.21.15.33:0/539764489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 0x55bdaba2afe0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f9de80108a0 tx=0x7f9de80099c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2b06b886e411e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9ddb7fe640 1 -- 172.21.15.33:0/539764489 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8047070 con 0x55bdaba13680 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.234+0000 7f9ddb7fe640 1 -- 172.21.15.33:0/539764489 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9de8011500 con 0x55bdaba13680 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.235+0000 7f9ddb7fe640 1 -- 172.21.15.33:0/539764489 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8037040 con 0x55bdaba13680 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.235+0000 7f9dfa315740 1 -- 172.21.15.33:0/539764489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 msgr2=0x55bdaba2afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.235+0000 7f9dfa315740 1 --2- 172.21.15.33:0/539764489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 0x55bdaba2afe0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f9de80108a0 tx=0x7f9de80099c0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.235+0000 7f9dfa315740 1 -- 172.21.15.33:0/539764489 shutdown_connections 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.235+0000 7f9dfa315740 1 --2- 172.21.15.33:0/539764489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba13680 0x55bdaba2afe0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.235+0000 7f9dfa315740 1 --2- 172.21.15.33:0/539764489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba08dc0 0x55bdaba2aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.235+0000 7f9dfa315740 1 --2- 172.21.15.33:0/539764489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba084a0 0x55bdaba1f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.235+0000 7f9dfa315740 1 -- 172.21.15.33:0/539764489 >> 172.21.15.33:0/539764489 conn(0x55bdaba04a30 msgr2=0x55bdaba0f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.235+0000 7f9dfa315740 1 -- 172.21.15.33:0/539764489 shutdown_connections 2024-09-16T00:30:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.235+0000 7f9dfa315740 1 -- 172.21.15.33:0/539764489 wait complete. 2024-09-16T00:30:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.236+0000 7f9dfa315740 1 -- start start 2024-09-16T00:30:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.236+0000 7f9dfa315740 1 Processor -- start 2024-09-16T00:30:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.236+0000 7f9dfa315740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 0x55bdaba1bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.236+0000 7f9dfa315740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba19060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.236+0000 7f9dfa315740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba13680 0x55bdaba195a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.236+0000 7f9dfa315740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01af950 con 0x55bdaba084a0 2024-09-16T00:30:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.236+0000 7f9dfa315740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01b0dc0 con 0x55bdaba08dc0 2024-09-16T00:30:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.236+0000 7f9dfa315740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01b0020 con 0x55bdaba13680 2024-09-16T00:30:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df5cd8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 0x55bdaba1bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9dfa315740 10 client.0 _ll_register_callbacks cb 0x7f9de8009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df5cd8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 0x55bdaba1bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43914/0 (socket says 172.21.15.33:43914) 2024-09-16T00:30:49.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df5cd8640 1 -- 172.21.15.33:0/1808730685 learned_addr learned my addr 172.21.15.33:0/1808730685 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df54d7640 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba19060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df64d9640 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba13680 0x55bdaba195a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df5cd8640 1 -- 172.21.15.33:0/1808730685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba13680 msgr2=0x55bdaba195a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df5cd8640 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba13680 0x55bdaba195a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df5cd8640 1 -- 172.21.15.33:0/1808730685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 msgr2=0x55bdaba19060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df5cd8640 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba19060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df5cd8640 1 -- 172.21.15.33:0/1808730685 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdaba19d80 con 0x55bdaba084a0 2024-09-16T00:30:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df54d7640 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba19060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:30:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df64d9640 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba13680 0x55bdaba195a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:30:49.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9df5cd8640 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 0x55bdaba1bd10 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9dec0cd3d0 tx=0x7f9dec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fd3769c68e9d57 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:49.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9dd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:30:49.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9dd97fa640 1 -- 172.21.15.33:0/1808730685 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec020040 con 0x55bdaba084a0 2024-09-16T00:30:49.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9dd97fa640 1 -- 172.21.15.33:0/1808730685 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9dec0981f0 con 0x55bdaba084a0 2024-09-16T00:30:49.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.237+0000 7f9dfa315740 1 -- 172.21.15.33:0/1808730685 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9de8011500 con 0x55bdaba084a0 2024-09-16T00:30:49.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.238+0000 7f9dd97fa640 1 -- 172.21.15.33:0/1808730685 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9dec09ff30 con 0x55bdaba084a0 2024-09-16T00:30:49.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.238+0000 7f9dfa315740 10 client.17934 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:49.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.238+0000 7f9dfa315740 1 -- 172.21.15.33:0/1808730685 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55bdaba8daa0 con 0x55bdaba084a0 2024-09-16T00:30:49.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.238+0000 7f9dd97fa640 1 -- 172.21.15.33:0/1808730685 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9de8011500 con 0x55bdaba084a0 2024-09-16T00:30:49.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.238+0000 7f9dd97fa640 1 -- 172.21.15.33:0/1808730685 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f9dec09b0f0 con 0x55bdaba084a0 2024-09-16T00:30:49.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.238+0000 7f9dfa315740 10 client.17934 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:49.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.238+0000 7f9dfa315740 10 client.17934 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:30:49.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.238+0000 7f9dfa315740 20 client.17934 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:49.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.238+0000 7f9dfa315740 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bdaba8e170 0x55bdaba90560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:49.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.238+0000 7f9dfa315740 4 client.17934 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:30:49.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.238+0000 7f9dfa315740 1 -- 172.21.15.33:0/1808730685 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bdab975d60 con 0x55bdaba8e170 2024-09-16T00:30:49.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.238+0000 7f9df54d7640 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bdaba8e170 0x55bdaba90560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:49.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dd97fa640 10 client.17934 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:49.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9df54d7640 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bdaba8e170 0x55bdaba90560 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1855bd99e8cef721 server_cookie=907a7ad070429183 in_seq=0 out_seq=0 2024-09-16T00:30:49.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dd97fa640 1 -- 172.21.15.33:0/1808730685 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bdab975d60 con 0x55bdaba8e170 2024-09-16T00:30:49.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dd97fa640 10 client.17934 handle_command_reply: tid=0 2024-09-16T00:30:49.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dfa315740 1 client.17934 shutdown 2024-09-16T00:30:49.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dfa315740 1 -- 172.21.15.33:0/1808730685 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 msgr2=0x55bdaba1bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:49.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dfa315740 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 0x55bdaba1bd10 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9dec0cd3d0 tx=0x7f9dec005880 comp rx=0 tx=0).stop 2024-09-16T00:30:49.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dfa315740 1 -- 172.21.15.33:0/1808730685 shutdown_connections 2024-09-16T00:30:49.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dfa315740 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bdaba8e170 0x55bdaba90560 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dfa315740 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdaba13680 0x55bdaba195a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dfa315740 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdaba08dc0 0x55bdaba19060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dfa315740 1 --2- 172.21.15.33:0/1808730685 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bdaba084a0 0x55bdaba1bd10 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dfa315740 1 -- 172.21.15.33:0/1808730685 >> 172.21.15.33:0/1808730685 conn(0x55bdaba04a30 msgr2=0x55bdaba03370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:49.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dd97fa640 0 client.17934 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:49.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dfa315740 1 -- 172.21.15.33:0/1808730685 shutdown_connections 2024-09-16T00:30:49.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.239+0000 7f9dfa315740 1 -- 172.21.15.33:0/1808730685 wait complete. 2024-09-16T00:30:49.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9dfa315740 20 client.17934 trim_cache size 0 max 16384 2024-09-16T00:30:49.249 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:30:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:30:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:30:49.249 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:30:49.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2179106525 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9ddc0a3240 msgr2=0x7f9ddc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:49.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9df7f3c640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9ddc0a3240 0x7f9ddc0a5700 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009fd0 tx=0x7f9de801eeb0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2179106525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9df0102dc0 msgr2=0x7f9df0100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:49.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9df7f3c640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9df0102dc0 0x7f9df0100a20 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00d950 tx=0x7f9dec00de10 comp rx=0 tx=0).stop 2024-09-16T00:30:49.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2179106525 shutdown_connections 2024-09-16T00:30:49.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9df7f3c640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9ddc0a3240 0x7f9ddc0a5700 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9df7f3c640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df010c1a0 0x7f9df01014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9df7f3c640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df01036e0 0x7f9df0100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9df7f3c640 1 --2- 172.21.15.33:0/2179106525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9df0102dc0 0x7f9df0100a20 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:49.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2179106525 >> 172.21.15.33:0/2179106525 conn(0x7f9df00fc7a0 msgr2=0x7f9df00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:49.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2179106525 shutdown_connections 2024-09-16T00:30:49.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:49.240+0000 7f9df7f3c640 1 -- 172.21.15.33:0/2179106525 wait complete. 2024-09-16T00:30:49.690 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1205642 9.87 MB/sec execute 597 sec latency 8935.018 ms 2024-09-16T00:30:49.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:49 smithi083 ceph-mon[35242]: pgmap v760: 129 pgs: 129 active+clean; 278 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 167 op/s 2024-09-16T00:30:49.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:49 smithi060 ceph-mon[35185]: pgmap v760: 129 pgs: 129 active+clean; 278 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 167 op/s 2024-09-16T00:30:50.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:49 smithi033 ceph-mon[31243]: pgmap v760: 129 pgs: 129 active+clean; 278 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 167 op/s 2024-09-16T00:30:50.325 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:30:50.325 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:30:50.376 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:50.690 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1205642 9.86 MB/sec execute 598 sec latency 9935.057 ms 2024-09-16T00:30:50.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.883+0000 7f3af3d28640 1 -- 172.21.15.33:0/785468727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3aec101420 msgr2=0x7f3aec10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:50.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.883+0000 7f3af3d28640 1 --2- 172.21.15.33:0/785468727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3aec101420 0x7f3aec10d950 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4009cd0 tx=0x7f3ae402a8e0 comp rx=0 tx=0).stop 2024-09-16T00:30:50.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.883+0000 7f3af3d28640 1 -- 172.21.15.33:0/785468727 shutdown_connections 2024-09-16T00:30:50.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.883+0000 7f3af3d28640 1 --2- 172.21.15.33:0/785468727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3aec101420 0x7f3aec10d950 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:50.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.883+0000 7f3af3d28640 1 --2- 172.21.15.33:0/785468727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aec100a80 0x7f3aec100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:50.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.883+0000 7f3af3d28640 1 --2- 172.21.15.33:0/785468727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3aec106a80 0x7f3aec106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:50.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.883+0000 7f3af3d28640 1 -- 172.21.15.33:0/785468727 >> 172.21.15.33:0/785468727 conn(0x7f3aec0fc7a0 msgr2=0x7f3aec0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:50.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.883+0000 7f3af3d28640 1 -- 172.21.15.33:0/785468727 shutdown_connections 2024-09-16T00:30:50.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.883+0000 7f3af3d28640 1 -- 172.21.15.33:0/785468727 wait complete. 2024-09-16T00:30:50.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.884+0000 7f3af3d28640 1 Processor -- start 2024-09-16T00:30:50.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.884+0000 7f3af3d28640 1 -- start start 2024-09-16T00:30:50.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.884+0000 7f3af3d28640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aec100a80 0x7f3aec1972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:50.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.884+0000 7f3af3d28640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3aec101420 0x7f3aec197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:50.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.884+0000 7f3af3d28640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3aec106a80 0x7f3aec19bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:50.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.884+0000 7f3af3d28640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3aec110e80 con 0x7f3aec101420 2024-09-16T00:30:50.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.884+0000 7f3af3d28640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3aec110d00 con 0x7f3aec106a80 2024-09-16T00:30:50.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.884+0000 7f3af3d28640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3aec111000 con 0x7f3aec100a80 2024-09-16T00:30:50.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af12c3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3aec101420 0x7f3aec197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:50.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af12c3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3aec101420 0x7f3aec197820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60406/0 (socket says 172.21.15.33:60406) 2024-09-16T00:30:50.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af12c3640 1 -- 172.21.15.33:0/1634086846 learned_addr learned my addr 172.21.15.33:0/1634086846 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af1ac4640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aec100a80 0x7f3aec1972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af22c5640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3aec106a80 0x7f3aec19bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af12c3640 1 -- 172.21.15.33:0/1634086846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aec100a80 msgr2=0x7f3aec1972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af12c3640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aec100a80 0x7f3aec1972e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af12c3640 1 -- 172.21.15.33:0/1634086846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3aec106a80 msgr2=0x7f3aec19bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af12c3640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3aec106a80 0x7f3aec19bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af12c3640 1 -- 172.21.15.33:0/1634086846 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aec19c290 con 0x7f3aec101420 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af22c5640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3aec106a80 0x7f3aec19bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af1ac4640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aec100a80 0x7f3aec1972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af12c3640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3aec101420 0x7f3aec197820 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f3adc00e9f0 tx=0x7f3adc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3ae2ffd640 1 -- 172.21.15.33:0/1634086846 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3adc00cd80 con 0x7f3aec101420 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3ae2ffd640 1 -- 172.21.15.33:0/1634086846 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3adc010910 con 0x7f3aec101420 2024-09-16T00:30:50.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af3d28640 1 -- 172.21.15.33:0/1634086846 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aec19c580 con 0x7f3aec101420 2024-09-16T00:30:50.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3ae2ffd640 1 -- 172.21.15.33:0/1634086846 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3adc018990 con 0x7f3aec101420 2024-09-16T00:30:50.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.885+0000 7f3af3d28640 1 -- 172.21.15.33:0/1634086846 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aec1a3e60 con 0x7f3aec101420 2024-09-16T00:30:50.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.886+0000 7f3af3d28640 1 -- 172.21.15.33:0/1634086846 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aec100ee0 con 0x7f3aec101420 2024-09-16T00:30:50.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.888+0000 7f3ae2ffd640 1 -- 172.21.15.33:0/1634086846 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3adc010ac0 con 0x7f3aec101420 2024-09-16T00:30:50.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.888+0000 7f3ae2ffd640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3ad80a3240 0x7f3ad80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:50.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.888+0000 7f3af1ac4640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3ad80a3240 0x7f3ad80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:50.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.889+0000 7f3ae2ffd640 1 -- 172.21.15.33:0/1634086846 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3adc0d0640 con 0x7f3aec101420 2024-09-16T00:30:50.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.889+0000 7f3af1ac4640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3ad80a3240 0x7f3ad80a5700 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009ab0 tx=0x7f3ae80094b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:50.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:50.891+0000 7f3ae2ffd640 1 -- 172.21.15.33:0/1634086846 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3adc09cee0 con 0x7f3aec101420 2024-09-16T00:30:50.948 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.124+0000 7f3af3d28640 1 -- 172.21.15.33:0/1634086846 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3aec1a4190 con 0x7f3aec101420 2024-09-16T00:30:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.125+0000 7f3ae2ffd640 1 -- 172.21.15.33:0/1634086846 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7f3adc09c7e0 con 0x7f3aec101420 2024-09-16T00:30:51.125 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:30:51.126 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:30:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:30:51.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.127+0000 7f3af3d28640 1 -- 172.21.15.33:0/1634086846 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3ad80a3240 msgr2=0x7f3ad80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:51.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.127+0000 7f3af3d28640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3ad80a3240 0x7f3ad80a5700 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009ab0 tx=0x7f3ae80094b0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.127+0000 7f3af3d28640 1 -- 172.21.15.33:0/1634086846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3aec101420 msgr2=0x7f3aec197820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:51.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.127+0000 7f3af3d28640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3aec101420 0x7f3aec197820 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f3adc00e9f0 tx=0x7f3adc00eeb0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.127+0000 7f3af3d28640 1 -- 172.21.15.33:0/1634086846 shutdown_connections 2024-09-16T00:30:51.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.127+0000 7f3af3d28640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3ad80a3240 0x7f3ad80a5700 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.127+0000 7f3af3d28640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3aec106a80 0x7f3aec19bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.127+0000 7f3af3d28640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3aec101420 0x7f3aec197820 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.127+0000 7f3af3d28640 1 --2- 172.21.15.33:0/1634086846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aec100a80 0x7f3aec1972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.127+0000 7f3af3d28640 1 -- 172.21.15.33:0/1634086846 >> 172.21.15.33:0/1634086846 conn(0x7f3aec0fc7a0 msgr2=0x7f3aec0fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:51.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.127+0000 7f3af3d28640 1 -- 172.21.15.33:0/1634086846 shutdown_connections 2024-09-16T00:30:51.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.127+0000 7f3af3d28640 1 -- 172.21.15.33:0/1634086846 wait complete. 2024-09-16T00:30:51.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.661+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1010789614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff888102d90 msgr2=0x7ff888103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:51.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.661+0000 7ff88cd8a640 1 --2- 172.21.15.33:0/1010789614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff888102d90 0x7ff888103170 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7ff878009930 tx=0x7ff87802a420 comp rx=0 tx=0).stop 2024-09-16T00:30:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.661+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1010789614 shutdown_connections 2024-09-16T00:30:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.661+0000 7ff88cd8a640 1 --2- 172.21.15.33:0/1010789614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff888108290 0x7ff88810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.661+0000 7ff88cd8a640 1 --2- 172.21.15.33:0/1010789614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff888103740 0x7ff888107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.661+0000 7ff88cd8a640 1 --2- 172.21.15.33:0/1010789614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff888102d90 0x7ff888103170 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.661+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1010789614 >> 172.21.15.33:0/1010789614 conn(0x7ff8880fc7d0 msgr2=0x7ff8880febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.661+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1010789614 shutdown_connections 2024-09-16T00:30:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.661+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1010789614 wait complete. 2024-09-16T00:30:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.661+0000 7ff88cd8a640 1 Processor -- start 2024-09-16T00:30:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff88cd8a640 1 -- start start 2024-09-16T00:30:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff88cd8a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff888102d90 0x7ff8881a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff88cd8a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff888103740 0x7ff8881a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff88cd8a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff888108290 0x7ff88819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff88cd8a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff888112090 con 0x7ff888103740 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff88cd8a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff888111f10 con 0x7ff888102d90 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff88cd8a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff888112210 con 0x7ff888108290 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff87dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff888103740 0x7ff8881a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff87dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff888103740 0x7ff8881a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60436/0 (socket says 172.21.15.33:60436) 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff87dd9b640 1 -- 172.21.15.33:0/1271002591 learned_addr learned my addr 172.21.15.33:0/1271002591 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff886d9d640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff888108290 0x7ff88819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff88659c640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff888102d90 0x7ff8881a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff87dd9b640 1 -- 172.21.15.33:0/1271002591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff888108290 msgr2=0x7ff88819a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff87dd9b640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff888108290 0x7ff88819a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff87dd9b640 1 -- 172.21.15.33:0/1271002591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff888102d90 msgr2=0x7ff8881a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff87dd9b640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff888102d90 0x7ff8881a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.662+0000 7ff87dd9b640 1 -- 172.21.15.33:0/1271002591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88819ad10 con 0x7ff888103740 2024-09-16T00:30:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.663+0000 7ff87dd9b640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff888103740 0x7ff8881a0960 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7ff87000d740 tx=0x7ff87000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.663+0000 7ff87ffff640 1 -- 172.21.15.33:0/1271002591 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870017070 con 0x7ff888103740 2024-09-16T00:30:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.663+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1271002591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff88819b000 con 0x7ff888103740 2024-09-16T00:30:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.663+0000 7ff87ffff640 1 -- 172.21.15.33:0/1271002591 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff87000de50 con 0x7ff888103740 2024-09-16T00:30:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.663+0000 7ff87ffff640 1 -- 172.21.15.33:0/1271002591 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870004e00 con 0x7ff888103740 2024-09-16T00:30:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.663+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1271002591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8881a72f0 con 0x7ff888103740 2024-09-16T00:30:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.664+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1271002591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff88806a7f0 con 0x7ff888103740 2024-09-16T00:30:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.665+0000 7ff87ffff640 1 -- 172.21.15.33:0/1271002591 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff870004fa0 con 0x7ff888103740 2024-09-16T00:30:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.666+0000 7ff87ffff640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8740a3240 0x7ff8740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.666+0000 7ff88659c640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8740a3240 0x7ff8740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.666+0000 7ff87ffff640 1 -- 172.21.15.33:0/1271002591 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff8700cfbd0 con 0x7ff888103740 2024-09-16T00:30:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.666+0000 7ff88659c640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8740a3240 0x7ff8740a5700 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff878000c00 tx=0x7ff878005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.669+0000 7ff87ffff640 1 -- 172.21.15.33:0/1271002591 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff87009c410 con 0x7ff888103740 2024-09-16T00:30:51.690 INFO:tasks.workunit.client.0.smithi033.stdout: 1 1205642 9.84 MB/sec execute 599 sec latency 10935.082 ms 2024-09-16T00:30:51.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.882+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1271002591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff88810c2b0 con 0x7ff888103740 2024-09-16T00:30:51.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:51 smithi033 ceph-mon[31243]: pgmap v761: 129 pgs: 129 active+clean; 278 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 175 op/s 2024-09-16T00:30:51.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:51 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1634086846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:30:51.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.883+0000 7ff87ffff640 1 -- 172.21.15.33:0/1271002591 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11903 (secure 0 0 0) 0x7ff87009bd10 con 0x7ff888103740 2024-09-16T00:30:51.883 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:30:51.884 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":30,"btime":"2024-09-16T00:21:22:042312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:21:21.463001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:30:51.884 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 30 2024-09-16T00:30:51.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.885+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1271002591 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8740a3240 msgr2=0x7ff8740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.885+0000 7ff88cd8a640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8740a3240 0x7ff8740a5700 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff878000c00 tx=0x7ff878005a20 comp rx=0 tx=0).stop 2024-09-16T00:30:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.885+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1271002591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff888103740 msgr2=0x7ff8881a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.885+0000 7ff88cd8a640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff888103740 0x7ff8881a0960 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7ff87000d740 tx=0x7ff87000dc00 comp rx=0 tx=0).stop 2024-09-16T00:30:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.885+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1271002591 shutdown_connections 2024-09-16T00:30:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.885+0000 7ff88cd8a640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff8740a3240 0x7ff8740a5700 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.885+0000 7ff88cd8a640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff888108290 0x7ff88819a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.885+0000 7ff88cd8a640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff888103740 0x7ff8881a0960 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.885+0000 7ff88cd8a640 1 --2- 172.21.15.33:0/1271002591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff888102d90 0x7ff8881a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.885+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1271002591 >> 172.21.15.33:0/1271002591 conn(0x7ff8880fc7d0 msgr2=0x7ff888100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.886+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1271002591 shutdown_connections 2024-09-16T00:30:51.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:51.886+0000 7ff88cd8a640 1 -- 172.21.15.33:0/1271002591 wait complete. 2024-09-16T00:30:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:51 smithi083 ceph-mon[35242]: pgmap v761: 129 pgs: 129 active+clean; 278 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 175 op/s 2024-09-16T00:30:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:51 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1634086846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:30:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:51 smithi060 ceph-mon[35185]: pgmap v761: 129 pgs: 129 active+clean; 278 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 175 op/s 2024-09-16T00:30:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1634086846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:30:52.550 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id e067ca76 --release --await' 2024-09-16T00:30:52.550 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id e067ca76 --release --await 2024-09-16T00:30:52.690 INFO:tasks.workunit.client.0.smithi033.stdout: 1 cleanup 600 sec 2024-09-16T00:30:52.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:52 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1271002591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:30:52.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:52 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1271002591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:30:52.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:52 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1271002591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:30:53.204 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:30:53.244 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:53.690 INFO:tasks.workunit.client.0.smithi033.stdout: 1 cleanup 601 sec 2024-09-16T00:30:53.739 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:53 smithi033 ceph-mon[31243]: pgmap v762: 129 pgs: 129 active+clean; 277 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-09-16T00:30:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.729+0000 7f58c14a6640 1 -- 172.21.15.33:0/552054194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58bc102570 msgr2=0x7f58bc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:53.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.729+0000 7f58c14a6640 1 --2- 172.21.15.33:0/552054194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58bc102570 0x7f58bc1029b0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009900 tx=0x7f58a402a480 comp rx=0 tx=0).stop 2024-09-16T00:30:53.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.730+0000 7f58c14a6640 1 -- 172.21.15.33:0/552054194 shutdown_connections 2024-09-16T00:30:53.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.730+0000 7f58c14a6640 1 --2- 172.21.15.33:0/552054194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58bc102ef0 0x7f58bc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.730+0000 7f58c14a6640 1 --2- 172.21.15.33:0/552054194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58bc102570 0x7f58bc1029b0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.730+0000 7f58c14a6640 1 --2- 172.21.15.33:0/552054194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58bc100e20 0x7f58bc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.730+0000 7f58c14a6640 1 -- 172.21.15.33:0/552054194 >> 172.21.15.33:0/552054194 conn(0x7f58bc0fc7d0 msgr2=0x7f58bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:53.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.730+0000 7f58c14a6640 1 -- 172.21.15.33:0/552054194 shutdown_connections 2024-09-16T00:30:53.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.730+0000 7f58c14a6640 1 -- 172.21.15.33:0/552054194 wait complete. 2024-09-16T00:30:53.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.730+0000 7f58c14a6640 1 Processor -- start 2024-09-16T00:30:53.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.730+0000 7f58c14a6640 1 -- start start 2024-09-16T00:30:53.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58c14a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58bc100e20 0x7f58bc19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58c14a6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58bc102570 0x7f58bc19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58c14a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58bc102ef0 0x7f58bc19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58c14a6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc110f80 con 0x7f58bc100e20 2024-09-16T00:30:53.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58c14a6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc110e00 con 0x7f58bc102ef0 2024-09-16T00:30:53.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58c14a6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc111100 con 0x7f58bc102570 2024-09-16T00:30:53.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58baffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58bc100e20 0x7f58bc19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58baffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58bc100e20 0x7f58bc19a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60450/0 (socket says 172.21.15.33:60450) 2024-09-16T00:30:53.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58baffd640 1 -- 172.21.15.33:0/2242566903 learned_addr learned my addr 172.21.15.33:0/2242566903 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:53.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58ba7fc640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58bc102570 0x7f58bc19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58bc102ef0 0x7f58bc19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58baffd640 1 -- 172.21.15.33:0/2242566903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58bc102570 msgr2=0x7f58bc19abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:53.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58baffd640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58bc102570 0x7f58bc19abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58baffd640 1 -- 172.21.15.33:0/2242566903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58bc102ef0 msgr2=0x7f58bc19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:53.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58baffd640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58bc102ef0 0x7f58bc19ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58baffd640 1 -- 172.21.15.33:0/2242566903 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58bc19f650 con 0x7f58bc100e20 2024-09-16T00:30:53.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58baffd640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58bc100e20 0x7f58bc19a6a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f58b400d950 tx=0x7f58b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:53.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.731+0000 7f58b3fff640 1 -- 172.21.15.33:0/2242566903 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b4017030 con 0x7f58bc100e20 2024-09-16T00:30:53.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.732+0000 7f58c14a6640 1 -- 172.21.15.33:0/2242566903 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58bc19f940 con 0x7f58bc100e20 2024-09-16T00:30:53.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.732+0000 7f58b3fff640 1 -- 172.21.15.33:0/2242566903 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58b4004020 con 0x7f58bc100e20 2024-09-16T00:30:53.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.732+0000 7f58b3fff640 1 -- 172.21.15.33:0/2242566903 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b40047a0 con 0x7f58bc100e20 2024-09-16T00:30:53.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.732+0000 7f58c14a6640 1 -- 172.21.15.33:0/2242566903 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58bc1a7220 con 0x7f58bc100e20 2024-09-16T00:30:53.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58b3fff640 1 -- 172.21.15.33:0/2242566903 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f58b40041d0 con 0x7f58bc100e20 2024-09-16T00:30:53.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58c387f740 1 Processor -- start 2024-09-16T00:30:53.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58c387f740 1 -- start start 2024-09-16T00:30:53.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58c387f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b87a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58c387f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b884d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58c387f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b887630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58c387f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556c1b7df620 con 0x556c1b885240 2024-09-16T00:30:53.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58c387f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556c1b786e50 con 0x556c1b87a980 2024-09-16T00:30:53.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58c387f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556c1b7db5e0 con 0x556c1b87a060 2024-09-16T00:30:53.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58ba7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b887630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58ba7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b887630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60458/0 (socket says 172.21.15.33:60458) 2024-09-16T00:30:53.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58bb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b87a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58ba7fc640 1 -- 172.21.15.33:0/4268388300 learned_addr learned my addr 172.21.15.33:0/4268388300 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:53.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.735+0000 7f58baffd640 1 --2- 172.21.15.33:0/4268388300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b884d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58b3fff640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58a80a3240 0x7f58a80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58bb7fe640 1 -- 172.21.15.33:0/4268388300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 msgr2=0x556c1b884d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:53.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/4268388300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b884d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58bb7fe640 1 -- 172.21.15.33:0/4268388300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 msgr2=0x556c1b887630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:53.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/4268388300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b887630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58bb7fe640 1 -- 172.21.15.33:0/4268388300 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c1b777d40 con 0x556c1b87a060 2024-09-16T00:30:53.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58ba7fc640 1 --2- 172.21.15.33:0/4268388300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b887630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:53.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58b3fff640 1 -- 172.21.15.33:0/2242566903 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f58b40cf770 con 0x7f58bc100e20 2024-09-16T00:30:53.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58a80a3240 0x7f58a80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/4268388300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b87a440 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00ae90 tx=0x7f58ac00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e5a3dde33c087ab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:53.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58b17fa640 1 -- 172.21.15.33:0/4268388300 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac01ab20 con 0x556c1b87a060 2024-09-16T00:30:53.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58b17fa640 1 -- 172.21.15.33:0/4268388300 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58ac004b40 con 0x556c1b87a060 2024-09-16T00:30:53.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.736+0000 7f58b17fa640 1 -- 172.21.15.33:0/4268388300 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac019370 con 0x556c1b87a060 2024-09-16T00:30:53.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.737+0000 7f58c387f740 1 -- 172.21.15.33:0/4268388300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 msgr2=0x556c1b87a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:53.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.737+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58a80a3240 0x7f58a80a5700 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f58ac021000 tx=0x7f58ac01e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:53.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.737+0000 7f58c387f740 1 --2- 172.21.15.33:0/4268388300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b87a440 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00ae90 tx=0x7f58ac00f790 comp rx=0 tx=0).stop 2024-09-16T00:30:53.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.737+0000 7f58c387f740 1 -- 172.21.15.33:0/4268388300 shutdown_connections 2024-09-16T00:30:53.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.737+0000 7f58c387f740 1 --2- 172.21.15.33:0/4268388300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b887630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.737+0000 7f58c387f740 1 --2- 172.21.15.33:0/4268388300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b884d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.737+0000 7f58c387f740 1 --2- 172.21.15.33:0/4268388300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b87a440 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.737+0000 7f58c387f740 1 -- 172.21.15.33:0/4268388300 >> 172.21.15.33:0/4268388300 conn(0x556c1b8765f0 msgr2=0x556c1b8769d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:53.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.737+0000 7f58c387f740 1 -- 172.21.15.33:0/4268388300 shutdown_connections 2024-09-16T00:30:53.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.737+0000 7f58c387f740 1 -- 172.21.15.33:0/4268388300 wait complete. 2024-09-16T00:30:53.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58c387f740 1 -- start start 2024-09-16T00:30:53.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58c387f740 1 Processor -- start 2024-09-16T00:30:53.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58c387f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b882100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58c387f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b882640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58c387f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b882b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58c387f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556c1b887e10 con 0x556c1b885240 2024-09-16T00:30:53.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58c387f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556c1b767f10 con 0x556c1b87a980 2024-09-16T00:30:53.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58c387f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556c1b887f90 con 0x556c1b87a060 2024-09-16T00:30:53.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58c387f740 10 client.0 _ll_register_callbacks cb 0x556c1b78bdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:53.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58bb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b882b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58bb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b882b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60468/0 (socket says 172.21.15.33:60468) 2024-09-16T00:30:53.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58bb7fe640 1 -- 172.21.15.33:0/859462755 learned_addr learned my addr 172.21.15.33:0/859462755 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:53.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58baffd640 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b882100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58ba7fc640 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b882640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58bb7fe640 1 -- 172.21.15.33:0/859462755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 msgr2=0x556c1b882100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:53.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b882100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58bb7fe640 1 -- 172.21.15.33:0/859462755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 msgr2=0x556c1b882640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:53.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b882640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58bb7fe640 1 -- 172.21.15.33:0/859462755 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c1b4ee580 con 0x556c1b885240 2024-09-16T00:30:53.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.738+0000 7f58ba7fc640 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b882640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:30:53.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.739+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b882b80 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f58ac000f80 tx=0x7f58ac00aa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc2071ad7bc4467 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.739+0000 7f58a27fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:30:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.739+0000 7f58a27fc640 1 -- 172.21.15.33:0/859462755 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac019370 con 0x556c1b885240 2024-09-16T00:30:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.739+0000 7f58a27fc640 1 -- 172.21.15.33:0/859462755 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f58ac005230 con 0x556c1b885240 2024-09-16T00:30:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.739+0000 7f58c387f740 1 -- 172.21.15.33:0/859462755 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58ac004b40 con 0x556c1b885240 2024-09-16T00:30:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.739+0000 7f58a27fc640 1 -- 172.21.15.33:0/859462755 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f58ac005430 con 0x556c1b885240 2024-09-16T00:30:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.739+0000 7f58c387f740 10 client.17958 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.739+0000 7f58c387f740 1 -- 172.21.15.33:0/859462755 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556c1b5b8960 con 0x556c1b885240 2024-09-16T00:30:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.740+0000 7f58a27fc640 1 -- 172.21.15.33:0/859462755 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58ac004b40 con 0x556c1b885240 2024-09-16T00:30:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.740+0000 7f58a27fc640 1 -- 172.21.15.33:0/859462755 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f58ac02c070 con 0x556c1b885240 2024-09-16T00:30:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.740+0000 7f58c387f740 10 client.17958 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.740+0000 7f58c387f740 10 client.17958 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:30:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.740+0000 7f58c387f740 20 client.17958 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.740+0000 7f58c387f740 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556c1b894c10 0x556c1b897000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.740+0000 7f58c387f740 4 client.17958 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:30:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.740+0000 7f58c387f740 1 -- 172.21.15.33:0/859462755 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c1b7ecb90 con 0x556c1b894c10 2024-09-16T00:30:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.740+0000 7f58baffd640 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556c1b894c10 0x556c1b897000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.741+0000 7f58a27fc640 10 client.17958 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.741+0000 7f58baffd640 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556c1b894c10 0x556c1b897000 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3c44d01ea290e2a server_cookie=3d0fbc0733c4f29a in_seq=0 out_seq=0 2024-09-16T00:30:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.741+0000 7f58a27fc640 1 -- 172.21.15.33:0/859462755 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556c1b7ecb90 con 0x556c1b894c10 2024-09-16T00:30:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.741+0000 7f58a27fc640 10 client.17958 handle_command_reply: tid=0 2024-09-16T00:30:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.742+0000 7f58c387f740 1 client.17958 shutdown 2024-09-16T00:30:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.742+0000 7f58c387f740 1 -- 172.21.15.33:0/859462755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 msgr2=0x556c1b882b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.742+0000 7f58c387f740 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b882b80 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f58ac000f80 tx=0x7f58ac00aa70 comp rx=0 tx=0).stop 2024-09-16T00:30:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.742+0000 7f58c387f740 1 -- 172.21.15.33:0/859462755 shutdown_connections 2024-09-16T00:30:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.742+0000 7f58c387f740 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556c1b894c10 0x556c1b897000 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.742+0000 7f58c387f740 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b882b80 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.742+0000 7f58c387f740 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b882640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.742+0000 7f58a27fc640 0 client.17958 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.742+0000 7f58c387f740 1 --2- 172.21.15.33:0/859462755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b882100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.742+0000 7f58c387f740 1 -- 172.21.15.33:0/859462755 >> 172.21.15.33:0/859462755 conn(0x556c1b8765f0 msgr2=0x556c1b87c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.742+0000 7f58c387f740 1 -- 172.21.15.33:0/859462755 shutdown_connections 2024-09-16T00:30:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.742+0000 7f58c387f740 1 -- 172.21.15.33:0/859462755 wait complete. 2024-09-16T00:30:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.743+0000 7f58c387f740 20 client.17958 trim_cache size 0 max 16384 2024-09-16T00:30:53.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.760+0000 7f58c387f740 1 Processor -- start 2024-09-16T00:30:53.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.760+0000 7f58c387f740 1 -- start start 2024-09-16T00:30:53.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.761+0000 7f58c387f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a060 0x556c1b87cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.761+0000 7f58c387f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b87a980 0x556c1b875640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.761+0000 7f58c387f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b885240 0x556c1b89c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.761+0000 7f58c387f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc1b3bd0 con 0x556c1b87a980 2024-09-16T00:30:53.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.761+0000 7f58c387f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc1b14a0 con 0x556c1b87a060 2024-09-16T00:30:53.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.761+0000 7f58c387f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc1af2f0 con 0x556c1b885240 2024-09-16T00:30:53.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.761+0000 7f58ba7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b87a980 0x556c1b875640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.761+0000 7f58ba7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b87a980 0x556c1b875640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60470/0 (socket says 172.21.15.33:60470) 2024-09-16T00:30:53.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.761+0000 7f58ba7fc640 1 -- 172.21.15.33:0/3472843233 learned_addr learned my addr 172.21.15.33:0/3472843233 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:53.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.761+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/3472843233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b885240 0x556c1b89c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.761+0000 7f58baffd640 1 --2- 172.21.15.33:0/3472843233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a060 0x556c1b87cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.762+0000 7f58bb7fe640 1 -- 172.21.15.33:0/3472843233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a060 msgr2=0x556c1b87cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:53.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.762+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/3472843233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a060 0x556c1b87cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.762+0000 7f58bb7fe640 1 -- 172.21.15.33:0/3472843233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b87a980 msgr2=0x556c1b875640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:53.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.762+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/3472843233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b87a980 0x556c1b875640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.762+0000 7f58bb7fe640 1 -- 172.21.15.33:0/3472843233 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c1b887cf0 con 0x556c1b885240 2024-09-16T00:30:53.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.762+0000 7f58ba7fc640 1 --2- 172.21.15.33:0/3472843233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b87a980 0x556c1b875640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:53.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.762+0000 7f58bb7fe640 1 --2- 172.21.15.33:0/3472843233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b885240 0x556c1b89c7e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0057a0 tx=0x7f58ac00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=616d4fe0197a62ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:53.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.762+0000 7f58b0ff9640 1 -- 172.21.15.33:0/3472843233 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac019370 con 0x556c1b885240 2024-09-16T00:30:53.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.762+0000 7f58b0ff9640 1 -- 172.21.15.33:0/3472843233 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58ac004b40 con 0x556c1b885240 2024-09-16T00:30:53.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.762+0000 7f58b0ff9640 1 -- 172.21.15.33:0/3472843233 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac01a070 con 0x556c1b885240 2024-09-16T00:30:53.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.763+0000 7f58c387f740 1 -- 172.21.15.33:0/3472843233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b885240 msgr2=0x556c1b89c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:53.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.763+0000 7f58c387f740 1 --2- 172.21.15.33:0/3472843233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b885240 0x556c1b89c7e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0057a0 tx=0x7f58ac00f790 comp rx=0 tx=0).stop 2024-09-16T00:30:53.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.763+0000 7f58c387f740 1 -- 172.21.15.33:0/3472843233 shutdown_connections 2024-09-16T00:30:53.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.763+0000 7f58c387f740 1 --2- 172.21.15.33:0/3472843233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b885240 0x556c1b89c7e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.763+0000 7f58c387f740 1 --2- 172.21.15.33:0/3472843233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b87a980 0x556c1b875640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.763+0000 7f58c387f740 1 --2- 172.21.15.33:0/3472843233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a060 0x556c1b87cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.763+0000 7f58c387f740 1 -- 172.21.15.33:0/3472843233 >> 172.21.15.33:0/3472843233 conn(0x556c1b8765f0 msgr2=0x556c1b874f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:53.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.763+0000 7f58c387f740 1 -- 172.21.15.33:0/3472843233 shutdown_connections 2024-09-16T00:30:53.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.763+0000 7f58c387f740 1 -- 172.21.15.33:0/3472843233 wait complete. 2024-09-16T00:30:53.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.763+0000 7f58c387f740 1 -- start start 2024-09-16T00:30:53.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.763+0000 7f58c387f740 1 Processor -- start 2024-09-16T00:30:53.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58c387f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b87f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58c387f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b87f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58c387f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b87fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58c387f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc1af960 con 0x556c1b885240 2024-09-16T00:30:53.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58c387f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc1b0dd0 con 0x556c1b87a980 2024-09-16T00:30:53.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58c387f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc1b0030 con 0x556c1b87a060 2024-09-16T00:30:53.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58c387f740 10 client.0 _ll_register_callbacks cb 0x7f58ac00acc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:53.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58bb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b87fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58baffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b87f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58baffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b87f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46934/0 (socket says 172.21.15.33:46934) 2024-09-16T00:30:53.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58bb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b87fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60476/0 (socket says 172.21.15.33:60476) 2024-09-16T00:30:53.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58baffd640 1 -- 172.21.15.33:0/171556810 learned_addr learned my addr 172.21.15.33:0/171556810 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:53.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.764+0000 7f58ba7fc640 1 --2- 172.21.15.33:0/171556810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b87f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58baffd640 1 -- 172.21.15.33:0/171556810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 msgr2=0x556c1b87f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:53.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58baffd640 1 --2- 172.21.15.33:0/171556810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b87f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58baffd640 1 -- 172.21.15.33:0/171556810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 msgr2=0x556c1b87fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:53.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58baffd640 1 --2- 172.21.15.33:0/171556810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b87fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58baffd640 1 -- 172.21.15.33:0/171556810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c1b8fe910 con 0x556c1b87a060 2024-09-16T00:30:53.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58baffd640 1 --2- 172.21.15.33:0/171556810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b87f210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f58b4097d70 tx=0x7f58b4097a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f153af5614ed7a89 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:53.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58a27fc640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58a27fc640 1 -- 172.21.15.33:0/171556810 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b409b910 con 0x556c1b87a060 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58c387f740 1 -- 172.21.15.33:0/171556810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58ac004b40 con 0x556c1b87a060 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58a27fc640 1 -- 172.21.15.33:0/171556810 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f58b4099c10 con 0x556c1b87a060 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58a27fc640 1 -- 172.21.15.33:0/171556810 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f58b409fcc0 con 0x556c1b87a060 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58c387f740 10 client.27328 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.765+0000 7f58c387f740 1 -- 172.21.15.33:0/171556810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556c1b8ffc30 con 0x556c1b87a060 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.766+0000 7f58a27fc640 1 -- 172.21.15.33:0/171556810 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58b40a4580 con 0x556c1b87a060 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.766+0000 7f58a27fc640 1 -- 172.21.15.33:0/171556810 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f58b40a40f0 con 0x556c1b87a060 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.766+0000 7f58c387f740 10 client.27328 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.766+0000 7f58c387f740 10 client.27328 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.766+0000 7f58c387f740 20 client.27328 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.766+0000 7f58c387f740 1 --2- 172.21.15.33:0/171556810 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556c1b9002b0 0x556c1b9026a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:53.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.766+0000 7f58c387f740 4 client.27328 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e067ca76", "release": true, "await": true}] 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.766+0000 7f58c387f740 1 -- 172.21.15.33:0/171556810 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e067ca76", "release": true, "await": true}) -- 0x7f58ac004b40 con 0x556c1b9002b0 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.766+0000 7f58ba7fc640 1 --2- 172.21.15.33:0/171556810 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556c1b9002b0 0x556c1b9026a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.767+0000 7f58ba7fc640 1 --2- 172.21.15.33:0/171556810 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556c1b9002b0 0x556c1b9026a0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aceb476a61e1ab server_cookie=1ce3052e05466894 in_seq=0 out_seq=0 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.767+0000 7f58a27fc640 10 client.27328 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.775+0000 7f58a27fc640 1 -- 172.21.15.33:0/171556810 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f58ac004b40 con 0x556c1b9002b0 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.775+0000 7f58a27fc640 10 client.27328 handle_command_reply: tid=0 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.775+0000 7f58c387f740 1 client.27328 shutdown 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.776+0000 7f58c387f740 1 -- 172.21.15.33:0/171556810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 msgr2=0x556c1b87f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.776+0000 7f58c387f740 1 --2- 172.21.15.33:0/171556810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b87f210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f58b4097d70 tx=0x7f58b4097a90 comp rx=0 tx=0).stop 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.776+0000 7f58c387f740 1 -- 172.21.15.33:0/171556810 shutdown_connections 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.776+0000 7f58c387f740 1 --2- 172.21.15.33:0/171556810 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556c1b9002b0 0x556c1b9026a0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.776+0000 7f58c387f740 1 --2- 172.21.15.33:0/171556810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c1b885240 0x556c1b87fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.776+0000 7f58c387f740 1 --2- 172.21.15.33:0/171556810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556c1b87a980 0x556c1b87f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.776+0000 7f58c387f740 1 --2- 172.21.15.33:0/171556810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c1b87a060 0x556c1b87f210 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.776+0000 7f58a27fc640 0 client.27328 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:53.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.776+0000 7f58c387f740 1 -- 172.21.15.33:0/171556810 >> 172.21.15.33:0/171556810 conn(0x556c1b8765f0 msgr2=0x556c1b8828c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:53.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.776+0000 7f58c387f740 1 -- 172.21.15.33:0/171556810 shutdown_connections 2024-09-16T00:30:53.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.776+0000 7f58c387f740 1 -- 172.21.15.33:0/171556810 wait complete. 2024-09-16T00:30:53.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.776+0000 7f58c387f740 20 client.27328 trim_cache size 0 max 16384 2024-09-16T00:30:53.789 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:30:53.789 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 30, 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 48, 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "e067ca76": { 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 48, 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 131.4, 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:30:53.790 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:30:53.791 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:30:53.791 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:30:53.791 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:30:53.791 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:30:53.791 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:30:53.791 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:30:53.791 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:30:53.791 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:30:53.791 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:30:53.791 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:30:53.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.777+0000 7f58c14a6640 1 -- 172.21.15.33:0/2242566903 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58a80a3240 msgr2=0x7f58a80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:53.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.777+0000 7f58c14a6640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58a80a3240 0x7f58a80a5700 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f58ac021000 tx=0x7f58ac01e000 comp rx=0 tx=0).stop 2024-09-16T00:30:53.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.777+0000 7f58c14a6640 1 -- 172.21.15.33:0/2242566903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58bc100e20 msgr2=0x7f58bc19a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:53.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.777+0000 7f58c14a6640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58bc100e20 0x7f58bc19a6a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f58b400d950 tx=0x7f58b400de10 comp rx=0 tx=0).stop 2024-09-16T00:30:53.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.777+0000 7f58c14a6640 1 -- 172.21.15.33:0/2242566903 shutdown_connections 2024-09-16T00:30:53.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.777+0000 7f58c14a6640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f58a80a3240 0x7f58a80a5700 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.777+0000 7f58c14a6640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58bc102ef0 0x7f58bc19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.777+0000 7f58c14a6640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58bc102570 0x7f58bc19abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.777+0000 7f58c14a6640 1 --2- 172.21.15.33:0/2242566903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58bc100e20 0x7f58bc19a6a0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:53.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.777+0000 7f58c14a6640 1 -- 172.21.15.33:0/2242566903 >> 172.21.15.33:0/2242566903 conn(0x7f58bc0fc7d0 msgr2=0x7f58bc06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:53.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.777+0000 7f58c14a6640 1 -- 172.21.15.33:0/2242566903 shutdown_connections 2024-09-16T00:30:53.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:53.778+0000 7f58c14a6640 1 -- 172.21.15.33:0/2242566903 wait complete. 2024-09-16T00:30:53.793 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:53 smithi083 ceph-mon[35242]: pgmap v762: 129 pgs: 129 active+clean; 277 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-09-16T00:30:53.960 INFO:tasks.workunit.client.0.smithi033.stdout: 0 cleanup 601 sec 2024-09-16T00:30:53.960 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-09-16T00:30:53.960 INFO:tasks.workunit.client.0.smithi033.stdout: Operation Count AvgLat MaxLat 2024-09-16T00:30:53.960 INFO:tasks.workunit.client.0.smithi033.stdout: ---------------------------------------- 2024-09-16T00:30:53.960 INFO:tasks.workunit.client.0.smithi033.stdout: NTCreateX 187956 1.443 26844.798 2024-09-16T00:30:53.960 INFO:tasks.workunit.client.0.smithi033.stdout: Close 138041 0.002 0.144 2024-09-16T00:30:53.960 INFO:tasks.workunit.client.0.smithi033.stdout: Rename 7958 6.809 23956.205 2024-09-16T00:30:53.960 INFO:tasks.workunit.client.0.smithi033.stdout: Unlink 37954 2.800 23331.962 2024-09-16T00:30:53.960 INFO:tasks.workunit.client.0.smithi033.stdout: Deltree 4 62.677 100.855 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout: Mkdir 2 0.005 0.005 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout: Qpathinfo 170370 0.554 105.481 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout: Qfileinfo 29833 0.001 0.189 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout: Qfsinfo 31232 0.151 5.740 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout: Sfileinfo 15308 0.018 13.747 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout: Find 65860 0.069 33.951 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout: WriteX 93566 0.014 25.030 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout: ReadX 294621 0.024 57.594 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout: LockX 612 0.717 3.041 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout: UnlockX 612 0.633 2.331 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout: Flush 13171 4.038 155.625 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-09-16T00:30:53.961 INFO:tasks.workunit.client.0.smithi033.stdout:Throughput 9.83932 MB/sec 1 clients 1 procs max_latency=26844.805 ms 2024-09-16T00:30:53.962 INFO:tasks.workunit.client.0.smithi033.stderr:+ dbench 10 2024-09-16T00:30:53.962 INFO:tasks.workunit.client.0.smithi033.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-16T00:30:53.962 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-09-16T00:30:53.962 INFO:tasks.workunit.client.0.smithi033.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-16T00:30:53.962 INFO:tasks.workunit.client.0.smithi033.stdout:8 of 10 processes prepared for launch 0 sec 2024-09-16T00:30:53.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:53 smithi060 ceph-mon[35185]: pgmap v762: 129 pgs: 129 active+clean; 277 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-09-16T00:30:54.062 INFO:tasks.workunit.client.0.smithi033.stdout:10 of 10 processes prepared for launch 0 sec 2024-09-16T00:30:54.062 INFO:tasks.workunit.client.0.smithi033.stdout:releasing clients 2024-09-16T00:30:54.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.697+0000 7fef08544640 1 -- 172.21.15.33:0/710129275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00074f40 msgr2=0x7fef0010de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:54.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.697+0000 7fef08544640 1 --2- 172.21.15.33:0/710129275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00074f40 0x7fef0010de80 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7feef0009900 tx=0x7feef002a480 comp rx=0 tx=0).stop 2024-09-16T00:30:54.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.697+0000 7fef08544640 1 -- 172.21.15.33:0/710129275 shutdown_connections 2024-09-16T00:30:54.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.697+0000 7fef08544640 1 --2- 172.21.15.33:0/710129275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef0010e3c0 0x7fef00110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.697+0000 7fef08544640 1 --2- 172.21.15.33:0/710129275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00074f40 0x7fef0010de80 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.697+0000 7fef08544640 1 --2- 172.21.15.33:0/710129275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef00074620 0x7fef00074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.697+0000 7fef08544640 1 -- 172.21.15.33:0/710129275 >> 172.21.15.33:0/710129275 conn(0x7fef000fd8c0 msgr2=0x7fef000ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:54.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.697+0000 7fef08544640 1 -- 172.21.15.33:0/710129275 shutdown_connections 2024-09-16T00:30:54.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.697+0000 7fef08544640 1 -- 172.21.15.33:0/710129275 wait complete. 2024-09-16T00:30:54.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.698+0000 7fef08544640 1 Processor -- start 2024-09-16T00:30:54.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.698+0000 7fef08544640 1 -- start start 2024-09-16T00:30:54.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef08544640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00074620 0x7fef0019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef08544640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00074f40 0x7fef0019ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef08544640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef0010e3c0 0x7fef0019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef08544640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fef00114140 con 0x7fef00074620 2024-09-16T00:30:54.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef08544640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fef00113fc0 con 0x7fef00074f40 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef08544640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fef001142c0 con 0x7fef0010e3c0 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef062e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00074620 0x7fef0019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef062e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00074620 0x7fef0019a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60480/0 (socket says 172.21.15.33:60480) 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef062e0640 1 -- 172.21.15.33:0/745741840 learned_addr learned my addr 172.21.15.33:0/745741840 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef05adf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00074f40 0x7fef0019ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef06ae1640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef0010e3c0 0x7fef0019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef062e0640 1 -- 172.21.15.33:0/745741840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef0010e3c0 msgr2=0x7fef0019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef062e0640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef0010e3c0 0x7fef0019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef062e0640 1 -- 172.21.15.33:0/745741840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00074f40 msgr2=0x7fef0019ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef062e0640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00074f40 0x7fef0019ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef062e0640 1 -- 172.21.15.33:0/745741840 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef0019f6a0 con 0x7fef00074620 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef05adf640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00074f40 0x7fef0019ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:54.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7fef062e0640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00074620 0x7fef0019a6f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7feefc00d950 tx=0x7feefc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:54.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.699+0000 7feef77fe640 1 -- 172.21.15.33:0/745741840 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc017030 con 0x7fef00074620 2024-09-16T00:30:54.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.700+0000 7fef08544640 1 -- 172.21.15.33:0/745741840 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef0019f990 con 0x7fef00074620 2024-09-16T00:30:54.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.700+0000 7feef77fe640 1 -- 172.21.15.33:0/745741840 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feefc004020 con 0x7fef00074620 2024-09-16T00:30:54.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.700+0000 7feef77fe640 1 -- 172.21.15.33:0/745741840 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc0047a0 con 0x7fef00074620 2024-09-16T00:30:54.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.700+0000 7fef08544640 1 -- 172.21.15.33:0/745741840 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef001a7270 con 0x7fef00074620 2024-09-16T00:30:54.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7fef0a91d740 1 Processor -- start 2024-09-16T00:30:54.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7feef77fe640 1 -- 172.21.15.33:0/745741840 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7feefc0041d0 con 0x7fef00074620 2024-09-16T00:30:54.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7fef0a91d740 1 -- start start 2024-09-16T00:30:54.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7fef0a91d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833d1b40 0x55d5833d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7fef0a91d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 0x55d5833d0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7fef0a91d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833db370 0x55d5833dd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7fef0a91d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d5833362f0 con 0x55d5833d05d0 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7fef0a91d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d5832ddcf0 con 0x55d5833d1b40 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7fef0a91d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d5833322b0 con 0x55d5833db370 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7fef062e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 0x55d5833d0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7fef062e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 0x55d5833d0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60494/0 (socket says 172.21.15.33:60494) 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7fef05adf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833db370 0x55d5833dd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.703+0000 7fef062e0640 1 -- 172.21.15.33:0/400540805 learned_addr learned my addr 172.21.15.33:0/400540805 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7fef06ae1640 1 --2- 172.21.15.33:0/400540805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833d1b40 0x55d5833d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7fef05adf640 1 -- 172.21.15.33:0/400540805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833d1b40 msgr2=0x55d5833d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7fef05adf640 1 --2- 172.21.15.33:0/400540805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833d1b40 0x55d5833d0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7fef05adf640 1 -- 172.21.15.33:0/400540805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 msgr2=0x55d5833d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:54.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7fef05adf640 1 --2- 172.21.15.33:0/400540805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 0x55d5833d0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7fef05adf640 1 -- 172.21.15.33:0/400540805 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5832c23a0 con 0x55d5833db370 2024-09-16T00:30:54.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7feef77fe640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7feeec0a3240 0x7feeec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7fef05adf640 1 --2- 172.21.15.33:0/400540805 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833db370 0x55d5833dd760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7feef0005940 tx=0x7feef0031630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ee89abd566e1169 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:54.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7fef06ae1640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7feeec0a3240 0x7feeec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7feef77fe640 1 -- 172.21.15.33:0/745741840 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7feefc0cf700 con 0x7fef00074620 2024-09-16T00:30:54.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7feef4ff9640 1 -- 172.21.15.33:0/400540805 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef0035040 con 0x55d5833db370 2024-09-16T00:30:54.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7feef4ff9640 1 -- 172.21.15.33:0/400540805 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feef0005020 con 0x55d5833db370 2024-09-16T00:30:54.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.704+0000 7feef4ff9640 1 -- 172.21.15.33:0/400540805 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef0004a00 con 0x55d5833db370 2024-09-16T00:30:54.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.705+0000 7fef0a91d740 1 -- 172.21.15.33:0/400540805 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833db370 msgr2=0x55d5833dd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:54.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.705+0000 7fef06ae1640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7feeec0a3240 0x7feeec0a5700 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7feef800ae90 tx=0x7feef800a4d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:54.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.705+0000 7fef0a91d740 1 --2- 172.21.15.33:0/400540805 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833db370 0x55d5833dd760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7feef0005940 tx=0x7feef0031630 comp rx=0 tx=0).stop 2024-09-16T00:30:54.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.705+0000 7fef0a91d740 1 -- 172.21.15.33:0/400540805 shutdown_connections 2024-09-16T00:30:54.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.705+0000 7fef0a91d740 1 --2- 172.21.15.33:0/400540805 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833db370 0x55d5833dd760 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.705+0000 7fef0a91d740 1 --2- 172.21.15.33:0/400540805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 0x55d5833d0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.705+0000 7fef0a91d740 1 --2- 172.21.15.33:0/400540805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833d1b40 0x55d5833d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.705+0000 7fef0a91d740 1 -- 172.21.15.33:0/400540805 >> 172.21.15.33:0/400540805 conn(0x55d583399310 msgr2=0x55d5833ccc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.705+0000 7fef0a91d740 1 -- 172.21.15.33:0/400540805 shutdown_connections 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.705+0000 7fef0a91d740 1 -- 172.21.15.33:0/400540805 wait complete. 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef0a91d740 1 -- start start 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef0a91d740 1 Processor -- start 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef0a91d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d05d0 0x55d5833dac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef0a91d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d1b40 0x55d5833d7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef0a91d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833db370 0x55d5833d8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef0a91d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d5833de010 con 0x55d5833d1b40 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef0a91d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d5832f1060 con 0x55d5833db370 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef0a91d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d5833de190 con 0x55d5833d05d0 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef0a91d740 10 client.0 _ll_register_callbacks cb 0x55d5830448c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:54.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef062e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d05d0 0x55d5833dac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef062e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d05d0 0x55d5833dac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46982/0 (socket says 172.21.15.33:46982) 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef062e0640 1 -- 172.21.15.33:0/3752656829 learned_addr learned my addr 172.21.15.33:0/3752656829 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef05adf640 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d1b40 0x55d5833d7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef06ae1640 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833db370 0x55d5833d8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef062e0640 1 -- 172.21.15.33:0/3752656829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833db370 msgr2=0x55d5833d8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef062e0640 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833db370 0x55d5833d8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.706+0000 7fef062e0640 1 -- 172.21.15.33:0/3752656829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d1b40 msgr2=0x55d5833d7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7fef062e0640 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d1b40 0x55d5833d7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7fef062e0640 1 -- 172.21.15.33:0/3752656829 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5830a73c0 con 0x55d5833d05d0 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7fef06ae1640 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833db370 0x55d5833d8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7fef05adf640 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d1b40 0x55d5833d7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7fef062e0640 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d05d0 0x55d5833dac90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feefc0ccdd0 tx=0x7feefc005130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6e39c1336ac9576 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:54.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7feee9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7feee9ffb640 1 -- 172.21.15.33:0/3752656829 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc09b3e0 con 0x55d5833d05d0 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7feee9ffb640 1 -- 172.21.15.33:0/3752656829 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7feefc0cf340 con 0x55d5833d05d0 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7fef0a91d740 1 -- 172.21.15.33:0/3752656829 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feef0005020 con 0x55d5833d05d0 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7feee9ffb640 1 -- 172.21.15.33:0/3752656829 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feefc09b200 con 0x55d5833d05d0 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7fef0a91d740 10 client.27340 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7fef0a91d740 1 -- 172.21.15.33:0/3752656829 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d5833e9a30 con 0x55d5833d05d0 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7feee9ffb640 1 -- 172.21.15.33:0/3752656829 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feefc0cf540 con 0x55d5833d05d0 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.707+0000 7feee9ffb640 1 -- 172.21.15.33:0/3752656829 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7feefc09cbd0 con 0x55d5833d05d0 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.708+0000 7fef0a91d740 10 client.27340 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.708+0000 7fef0a91d740 10 client.27340 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.708+0000 7fef0a91d740 20 client.27340 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.708+0000 7fef0a91d740 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d5833eae40 0x55d5833ed230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.708+0000 7fef0a91d740 4 client.27340 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.708+0000 7fef0a91d740 1 -- 172.21.15.33:0/3752656829 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7feef0005020 con 0x55d5833eae40 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.708+0000 7fef05adf640 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d5833eae40 0x55d5833ed230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.708+0000 7fef05adf640 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d5833eae40 0x55d5833ed230 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f937428b510a7098 server_cookie=d7ea52fe5237a708 in_seq=0 out_seq=0 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.708+0000 7feee9ffb640 10 client.27340 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7feee9ffb640 1 -- 172.21.15.33:0/3752656829 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7feef0005020 con 0x55d5833eae40 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7feee9ffb640 10 client.27340 handle_command_reply: tid=0 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7fef0a91d740 1 client.27340 shutdown 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7fef0a91d740 1 -- 172.21.15.33:0/3752656829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d05d0 msgr2=0x55d5833dac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7fef0a91d740 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d05d0 0x55d5833dac90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feefc0ccdd0 tx=0x7feefc005130 comp rx=0 tx=0).stop 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7fef0a91d740 1 -- 172.21.15.33:0/3752656829 shutdown_connections 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7fef0a91d740 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d5833eae40 0x55d5833ed230 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7fef0a91d740 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833db370 0x55d5833d8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7feee9ffb640 0 client.27340 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:30:54.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7fef0a91d740 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d1b40 0x55d5833d7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7fef0a91d740 1 --2- 172.21.15.33:0/3752656829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d05d0 0x55d5833dac90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7fef0a91d740 1 -- 172.21.15.33:0/3752656829 >> 172.21.15.33:0/3752656829 conn(0x55d583399310 msgr2=0x55d5833d2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:54.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7fef0a91d740 1 -- 172.21.15.33:0/3752656829 shutdown_connections 2024-09-16T00:30:54.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.709+0000 7fef0a91d740 1 -- 172.21.15.33:0/3752656829 wait complete. 2024-09-16T00:30:54.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.710+0000 7fef0a91d740 20 client.27340 trim_cache size 0 max 16384 2024-09-16T00:30:54.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.722+0000 7fef0a91d740 1 Processor -- start 2024-09-16T00:30:54.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.722+0000 7fef0a91d740 1 -- start start 2024-09-16T00:30:54.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef0a91d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833d05d0 0x55d5833e72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef0a91d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d1b40 0x55d5833d3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef0a91d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833db370 0x55d5833d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef0a91d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fef001b3c20 con 0x55d5833db370 2024-09-16T00:30:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef0a91d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fef001b14f0 con 0x55d5833d05d0 2024-09-16T00:30:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef0a91d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fef001af340 con 0x55d5833d1b40 2024-09-16T00:30:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef06ae1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833db370 0x55d5833d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef06ae1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833db370 0x55d5833d3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60512/0 (socket says 172.21.15.33:60512) 2024-09-16T00:30:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef06ae1640 1 -- 172.21.15.33:0/208565213 learned_addr learned my addr 172.21.15.33:0/208565213 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef062e0640 1 --2- 172.21.15.33:0/208565213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833d05d0 0x55d5833e72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef05adf640 1 --2- 172.21.15.33:0/208565213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d1b40 0x55d5833d3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef06ae1640 1 -- 172.21.15.33:0/208565213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d1b40 msgr2=0x55d5833d3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef06ae1640 1 --2- 172.21.15.33:0/208565213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d1b40 0x55d5833d3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef06ae1640 1 -- 172.21.15.33:0/208565213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833d05d0 msgr2=0x55d5833e72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef06ae1640 1 --2- 172.21.15.33:0/208565213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833d05d0 0x55d5833e72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef06ae1640 1 -- 172.21.15.33:0/208565213 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5833db000 con 0x55d5833db370 2024-09-16T00:30:54.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7fef06ae1640 1 --2- 172.21.15.33:0/208565213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833db370 0x55d5833d3660 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7feef801ebc0 tx=0x7feef8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d138950a715fb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:54.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7feeebfff640 1 -- 172.21.15.33:0/208565213 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef802e070 con 0x55d5833db370 2024-09-16T00:30:54.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.723+0000 7feeebfff640 1 -- 172.21.15.33:0/208565213 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feef8029430 con 0x55d5833db370 2024-09-16T00:30:54.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.724+0000 7feeebfff640 1 -- 172.21.15.33:0/208565213 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef800f680 con 0x55d5833db370 2024-09-16T00:30:54.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.724+0000 7fef0a91d740 1 -- 172.21.15.33:0/208565213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833db370 msgr2=0x55d5833d3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:54.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.724+0000 7fef0a91d740 1 --2- 172.21.15.33:0/208565213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833db370 0x55d5833d3660 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7feef801ebc0 tx=0x7feef8019000 comp rx=0 tx=0).stop 2024-09-16T00:30:54.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.724+0000 7fef0a91d740 1 -- 172.21.15.33:0/208565213 shutdown_connections 2024-09-16T00:30:54.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.724+0000 7fef0a91d740 1 --2- 172.21.15.33:0/208565213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833db370 0x55d5833d3660 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.724+0000 7fef0a91d740 1 --2- 172.21.15.33:0/208565213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d1b40 0x55d5833d3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.724+0000 7fef0a91d740 1 --2- 172.21.15.33:0/208565213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833d05d0 0x55d5833e72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.724+0000 7fef0a91d740 1 -- 172.21.15.33:0/208565213 >> 172.21.15.33:0/208565213 conn(0x55d583399310 msgr2=0x55d5833d9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:54.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.724+0000 7fef0a91d740 1 -- 172.21.15.33:0/208565213 shutdown_connections 2024-09-16T00:30:54.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.724+0000 7fef0a91d740 1 -- 172.21.15.33:0/208565213 wait complete. 2024-09-16T00:30:54.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.724+0000 7fef0a91d740 1 -- start start 2024-09-16T00:30:54.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef0a91d740 1 Processor -- start 2024-09-16T00:30:54.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef0a91d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 0x55d5833e3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef0a91d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d1b40 0x55d5833deec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef0a91d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833db370 0x55d5833df400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef0a91d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fef001af9b0 con 0x55d5833d05d0 2024-09-16T00:30:54.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef0a91d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fef001b0e20 con 0x55d5833db370 2024-09-16T00:30:54.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef0a91d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fef001b0080 con 0x55d5833d1b40 2024-09-16T00:30:54.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef062e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 0x55d5833e3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef0a91d740 10 client.0 _ll_register_callbacks cb 0x7feefc0d5240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:54.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef062e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 0x55d5833e3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60514/0 (socket says 172.21.15.33:60514) 2024-09-16T00:30:54.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef062e0640 1 -- 172.21.15.33:0/380030373 learned_addr learned my addr 172.21.15.33:0/380030373 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:54.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef06ae1640 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833db370 0x55d5833df400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef05adf640 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d1b40 0x55d5833deec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef062e0640 1 -- 172.21.15.33:0/380030373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d1b40 msgr2=0x55d5833deec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef062e0640 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d1b40 0x55d5833deec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef062e0640 1 -- 172.21.15.33:0/380030373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833db370 msgr2=0x55d5833df400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef062e0640 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833db370 0x55d5833df400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.725+0000 7fef062e0640 1 -- 172.21.15.33:0/380030373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5833dfc50 con 0x55d5833d05d0 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7fef062e0640 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 0x55d5833e3b90 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7feefc005fd0 tx=0x7feefc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5370228e81d731 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7feee9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7feee9ffb640 1 -- 172.21.15.33:0/380030373 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc09b3e0 con 0x55d5833d05d0 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7feee9ffb640 1 -- 172.21.15.33:0/380030373 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7feefc0cf340 con 0x55d5833d05d0 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7fef0a91d740 1 -- 172.21.15.33:0/380030373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feef8029430 con 0x55d5833d05d0 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7feee9ffb640 1 -- 172.21.15.33:0/380030373 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feefc0860b0 con 0x55d5833d05d0 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7fef0a91d740 10 client.17982 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7fef0a91d740 1 -- 172.21.15.33:0/380030373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d583455ad0 con 0x55d5833d05d0 2024-09-16T00:30:54.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7feee9ffb640 1 -- 172.21.15.33:0/380030373 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feef8029430 con 0x55d5833d05d0 2024-09-16T00:30:54.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7feee9ffb640 1 -- 172.21.15.33:0/380030373 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7feefc0ba0c0 con 0x55d5833d05d0 2024-09-16T00:30:54.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7fef0a91d740 10 client.17982 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:54.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7fef0a91d740 10 client.17982 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:30:54.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7fef0a91d740 20 client.17982 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:54.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7fef0a91d740 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d5833eae40 0x55d583458070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:54.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7fef0a91d740 4 client.17982 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:30:54.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7fef0a91d740 1 -- 172.21.15.33:0/380030373 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d583343860 con 0x55d5833eae40 2024-09-16T00:30:54.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.726+0000 7fef05adf640 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d5833eae40 0x55d583458070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:54.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.727+0000 7feee9ffb640 10 client.17982 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:30:54.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.727+0000 7fef05adf640 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d5833eae40 0x55d583458070 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=215ec42c23c7de3f server_cookie=400ccacc8f7a8ce9 in_seq=0 out_seq=0 2024-09-16T00:30:54.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.731+0000 7feee9ffb640 1 -- 172.21.15.33:0/380030373 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d583343860 con 0x55d5833eae40 2024-09-16T00:30:54.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.731+0000 7feee9ffb640 10 client.17982 handle_command_reply: tid=0 2024-09-16T00:30:54.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.731+0000 7fef0a91d740 1 client.17982 shutdown 2024-09-16T00:30:54.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef0a91d740 1 -- 172.21.15.33:0/380030373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 msgr2=0x55d5833e3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:54.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef0a91d740 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 0x55d5833e3b90 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7feefc005fd0 tx=0x7feefc0057b0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef0a91d740 1 -- 172.21.15.33:0/380030373 shutdown_connections 2024-09-16T00:30:54.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef0a91d740 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d5833eae40 0x55d583458070 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef0a91d740 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d5833db370 0x55d5833df400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7feee9ffb640 0 client.17982 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:30:54.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef0a91d740 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d5833d1b40 0x55d5833deec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef0a91d740 1 --2- 172.21.15.33:0/380030373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d5833d05d0 0x55d5833e3b90 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef0a91d740 1 -- 172.21.15.33:0/380030373 >> 172.21.15.33:0/380030373 conn(0x55d583399310 msgr2=0x55d5833cb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:54.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef0a91d740 1 -- 172.21.15.33:0/380030373 shutdown_connections 2024-09-16T00:30:54.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef0a91d740 1 -- 172.21.15.33:0/380030373 wait complete. 2024-09-16T00:30:54.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef0a91d740 20 client.17982 trim_cache size 0 max 16384 2024-09-16T00:30:54.734 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:30:54.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef08544640 1 -- 172.21.15.33:0/745741840 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7feeec0a3240 msgr2=0x7feeec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:54.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef08544640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7feeec0a3240 0x7feeec0a5700 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7feef800ae90 tx=0x7feef800a4d0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef08544640 1 -- 172.21.15.33:0/745741840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00074620 msgr2=0x7fef0019a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:54.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.732+0000 7fef08544640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00074620 0x7fef0019a6f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7feefc00d950 tx=0x7feefc00de10 comp rx=0 tx=0).stop 2024-09-16T00:30:54.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.733+0000 7fef08544640 1 -- 172.21.15.33:0/745741840 shutdown_connections 2024-09-16T00:30:54.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.733+0000 7fef08544640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7feeec0a3240 0x7feeec0a5700 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.733+0000 7fef08544640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef0010e3c0 0x7fef0019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.733+0000 7fef08544640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00074f40 0x7fef0019ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.733+0000 7fef08544640 1 --2- 172.21.15.33:0/745741840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00074620 0x7fef0019a6f0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:54.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.733+0000 7fef08544640 1 -- 172.21.15.33:0/745741840 >> 172.21.15.33:0/745741840 conn(0x7fef000fd8c0 msgr2=0x7fef000ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:54.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.733+0000 7fef08544640 1 -- 172.21.15.33:0/745741840 shutdown_connections 2024-09-16T00:30:54.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:54.733+0000 7fef08544640 1 -- 172.21.15.33:0/745741840 wait complete. 2024-09-16T00:30:55.062 INFO:tasks.workunit.client.0.smithi033.stdout: 10 357 128.08 MB/sec warmup 1 sec latency 147.204 ms 2024-09-16T00:30:55.508 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'e067ca76', total seconds elapsed: 17.9 2024-09-16T00:30:55.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:55 smithi033 ceph-mon[31243]: pgmap v763: 129 pgs: 129 active+clean; 277 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 2.5 MiB/s wr, 105 op/s 2024-09-16T00:30:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:55 smithi083 ceph-mon[35242]: pgmap v763: 129 pgs: 129 active+clean; 277 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 2.5 MiB/s wr, 105 op/s 2024-09-16T00:30:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:55 smithi060 ceph-mon[35185]: pgmap v763: 129 pgs: 129 active+clean; 277 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 2.5 MiB/s wr, 105 op/s 2024-09-16T00:30:56.040 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:30:56.062 INFO:tasks.workunit.client.0.smithi033.stdout: 10 800 107.25 MB/sec warmup 2 sec latency 50.125 ms 2024-09-16T00:30:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:56.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.964+0000 7f9ced8b6640 1 -- 172.21.15.33:0/3051784402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ce8100e20 msgr2=0x7f9ce8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:56.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.964+0000 7f9ced8b6640 1 --2- 172.21.15.33:0/3051784402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ce8100e20 0x7f9ce8101200 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009900 tx=0x7f9cd802a480 comp rx=0 tx=0).stop 2024-09-16T00:30:56.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.964+0000 7f9ced8b6640 1 -- 172.21.15.33:0/3051784402 shutdown_connections 2024-09-16T00:30:56.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.964+0000 7f9ced8b6640 1 --2- 172.21.15.33:0/3051784402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ce8102ef0 0x7f9ce810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.964+0000 7f9ced8b6640 1 --2- 172.21.15.33:0/3051784402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce8102570 0x7f9ce81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.964+0000 7f9ced8b6640 1 --2- 172.21.15.33:0/3051784402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ce8100e20 0x7f9ce8101200 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.964+0000 7f9ced8b6640 1 -- 172.21.15.33:0/3051784402 >> 172.21.15.33:0/3051784402 conn(0x7f9ce80fc7d0 msgr2=0x7f9ce80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.965+0000 7f9ced8b6640 1 -- 172.21.15.33:0/3051784402 shutdown_connections 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.965+0000 7f9ced8b6640 1 -- 172.21.15.33:0/3051784402 wait complete. 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.965+0000 7f9ced8b6640 1 Processor -- start 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ced8b6640 1 -- start start 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ced8b6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ce8100e20 0x7f9ce80fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ced8b6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce8102570 0x7f9ce80fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ced8b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ce8102ef0 0x7f9ce80fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ced8b6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce8110f50 con 0x7f9ce8100e20 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ced8b6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce8110dd0 con 0x7f9ce8102570 2024-09-16T00:30:56.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ced8b6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce81110d0 con 0x7f9ce8102ef0 2024-09-16T00:30:56.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ce6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ce8100e20 0x7f9ce80fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ce6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ce8100e20 0x7f9ce80fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60544/0 (socket says 172.21.15.33:60544) 2024-09-16T00:30:56.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ce6ffd640 1 -- 172.21.15.33:0/115426280 learned_addr learned my addr 172.21.15.33:0/115426280 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:56.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ce8102ef0 0x7f9ce80fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ce67fc640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce8102570 0x7f9ce80fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ce6ffd640 1 -- 172.21.15.33:0/115426280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ce8102ef0 msgr2=0x7f9ce80fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:56.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ce6ffd640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ce8102ef0 0x7f9ce80fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ce6ffd640 1 -- 172.21.15.33:0/115426280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce8102570 msgr2=0x7f9ce80fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:56.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ce6ffd640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce8102570 0x7f9ce80fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ce6ffd640 1 -- 172.21.15.33:0/115426280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce81a69e0 con 0x7f9ce8100e20 2024-09-16T00:30:56.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ce8102ef0 0x7f9ce80fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:56.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.966+0000 7f9ce67fc640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce8102570 0x7f9ce80fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.967+0000 7f9ce6ffd640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ce8100e20 0x7f9ce80fcc80 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8006010 tx=0x7f9cd8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.967+0000 7f9cec8b4640 1 -- 172.21.15.33:0/115426280 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd803f070 con 0x7f9ce8100e20 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.967+0000 7f9cec8b4640 1 -- 172.21.15.33:0/115426280 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cd8031c30 con 0x7f9ce8100e20 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.967+0000 7f9ced8b6640 1 -- 172.21.15.33:0/115426280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ce81a6c70 con 0x7f9ce8100e20 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.967+0000 7f9cec8b4640 1 -- 172.21.15.33:0/115426280 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8035030 con 0x7f9ce8100e20 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.967+0000 7f9ced8b6640 1 -- 172.21.15.33:0/115426280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ce81a7150 con 0x7f9ce8100e20 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9cec8b4640 1 -- 172.21.15.33:0/115426280 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9cd8042070 con 0x7f9ce8100e20 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9cefc8f740 1 Processor -- start 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9cefc8f740 1 -- start start 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9cefc8f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 0x561ff5e00090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9cefc8f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e005d0 0x561ff5e00a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9cefc8f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e0b370 0x561ff5e0d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9cefc8f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561ff5d662f0 con 0x561ff5e01b40 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9cefc8f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561ff5d0dcf0 con 0x561ff5e0b370 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9cefc8f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561ff5d622b0 con 0x561ff5e005d0 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9ce77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 0x561ff5e00090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9ce77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 0x561ff5e00090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60550/0 (socket says 172.21.15.33:60550) 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9ce77fe640 1 -- 172.21.15.33:0/703402110 learned_addr learned my addr 172.21.15.33:0/703402110 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9ce67fc640 1 --2- 172.21.15.33:0/703402110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e0b370 0x561ff5e0d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.970+0000 7f9ce6ffd640 1 --2- 172.21.15.33:0/703402110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e005d0 0x561ff5e00a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.971+0000 7f9ce77fe640 1 -- 172.21.15.33:0/703402110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e005d0 msgr2=0x561ff5e00a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.971+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/703402110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e005d0 0x561ff5e00a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.971+0000 7f9ce77fe640 1 -- 172.21.15.33:0/703402110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e0b370 msgr2=0x561ff5e0d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.971+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/703402110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e0b370 0x561ff5e0d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.971+0000 7f9ce77fe640 1 -- 172.21.15.33:0/703402110 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ff5cf23a0 con 0x561ff5e01b40 2024-09-16T00:30:56.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.971+0000 7f9cec8b4640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9cd00a3240 0x7f9cd00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.971+0000 7f9ce6ffd640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9cd00a3240 0x7f9cd00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.971+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/703402110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 0x561ff5e00090 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0011790 tx=0x7f9ce0011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ff7a1163535572 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:56.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.971+0000 7f9cec8b4640 1 -- 172.21.15.33:0/115426280 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9cd80f03f0 con 0x7f9ce8100e20 2024-09-16T00:30:56.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.971+0000 7f9ccdffb640 1 -- 172.21.15.33:0/703402110 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0003c30 con 0x561ff5e01b40 2024-09-16T00:30:56.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.971+0000 7f9ccdffb640 1 -- 172.21.15.33:0/703402110 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce000fe80 con 0x561ff5e01b40 2024-09-16T00:30:56.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.971+0000 7f9ccdffb640 1 -- 172.21.15.33:0/703402110 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce00193e0 con 0x561ff5e01b40 2024-09-16T00:30:56.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.972+0000 7f9cefc8f740 1 -- 172.21.15.33:0/703402110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 msgr2=0x561ff5e00090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:56.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.972+0000 7f9ce6ffd640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9cd00a3240 0x7f9cd00a5700 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8000c00 tx=0x7f9cd80f5040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:56.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.972+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/703402110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 0x561ff5e00090 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0011790 tx=0x7f9ce0011c50 comp rx=0 tx=0).stop 2024-09-16T00:30:56.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.972+0000 7f9cefc8f740 1 -- 172.21.15.33:0/703402110 shutdown_connections 2024-09-16T00:30:56.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.972+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/703402110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e0b370 0x561ff5e0d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.972+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/703402110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e005d0 0x561ff5e00a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.972+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/703402110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 0x561ff5e00090 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.972+0000 7f9cefc8f740 1 -- 172.21.15.33:0/703402110 >> 172.21.15.33:0/703402110 conn(0x561ff5dc9310 msgr2=0x561ff5dfcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.972+0000 7f9cefc8f740 1 -- 172.21.15.33:0/703402110 shutdown_connections 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.973+0000 7f9cefc8f740 1 -- 172.21.15.33:0/703402110 wait complete. 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.973+0000 7f9cefc8f740 1 -- start start 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.973+0000 7f9cefc8f740 1 Processor -- start 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.973+0000 7f9cefc8f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e005d0 0x561ff5e0ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.973+0000 7f9cefc8f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e01b40 0x561ff5e08020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.973+0000 7f9cefc8f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 0x561ff5e08610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.973+0000 7f9cefc8f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561ff5e0e070 con 0x561ff5e0b370 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.973+0000 7f9cefc8f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561ff5d21060 con 0x561ff5e005d0 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.973+0000 7f9cefc8f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561ff5e0e1f0 con 0x561ff5e01b40 2024-09-16T00:30:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.973+0000 7f9cefc8f740 10 client.0 _ll_register_callbacks cb 0x561ff5a748c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9ce77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e005d0 0x561ff5e0ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9ce77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 0x561ff5e08610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9ce67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e01b40 0x561ff5e08020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9ce77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 0x561ff5e08610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60560/0 (socket says 172.21.15.33:60560) 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9ce77fe640 1 -- 172.21.15.33:0/3489953794 learned_addr learned my addr 172.21.15.33:0/3489953794 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9ce77fe640 1 -- 172.21.15.33:0/3489953794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e01b40 msgr2=0x561ff5e08020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/3489953794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e01b40 0x561ff5e08020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9ce77fe640 1 -- 172.21.15.33:0/3489953794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e005d0 msgr2=0x561ff5e0ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/3489953794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e005d0 0x561ff5e0ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9ce77fe640 1 -- 172.21.15.33:0/3489953794 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ff5ad73c0 con 0x561ff5e0b370 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/3489953794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 0x561ff5e08610 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00046c0 tx=0x7f9ce0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4d032634e16774 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9cc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:30:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9cc6ffd640 1 -- 172.21.15.33:0/3489953794 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0011050 con 0x561ff5e0b370 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9cc6ffd640 1 -- 172.21.15.33:0/3489953794 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9ce00111f0 con 0x561ff5e0b370 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.974+0000 7f9cefc8f740 1 -- 172.21.15.33:0/3489953794 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561ff5d73860 con 0x561ff5e0b370 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.975+0000 7f9cc6ffd640 1 -- 172.21.15.33:0/3489953794 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9ce0004a40 con 0x561ff5e0b370 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.975+0000 7f9cefc8f740 10 client.17994 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.975+0000 7f9cefc8f740 1 -- 172.21.15.33:0/3489953794 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561ff5e19b40 con 0x561ff5e0b370 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.975+0000 7f9cc6ffd640 1 -- 172.21.15.33:0/3489953794 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561ff5d73860 con 0x561ff5e0b370 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.975+0000 7f9cc6ffd640 1 -- 172.21.15.33:0/3489953794 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f9ce0018070 con 0x561ff5e0b370 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.975+0000 7f9cefc8f740 10 client.17994 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.975+0000 7f9cefc8f740 10 client.17994 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.975+0000 7f9cefc8f740 20 client.17994 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.975+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/3489953794 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561ff5e1afa0 0x561ff5e1d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.975+0000 7f9cefc8f740 4 client.17994 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:30:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.975+0000 7f9cefc8f740 1 -- 172.21.15.33:0/3489953794 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ff5d6dd70 con 0x561ff5e1afa0 2024-09-16T00:30:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.975+0000 7f9ce67fc640 1 --2- 172.21.15.33:0/3489953794 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561ff5e1afa0 0x561ff5e1d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.976+0000 7f9cc6ffd640 10 client.17994 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:30:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.976+0000 7f9ce67fc640 1 --2- 172.21.15.33:0/3489953794 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561ff5e1afa0 0x561ff5e1d390 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3b0a2509c1e392c0 server_cookie=54e7798afe1ca0de in_seq=0 out_seq=0 2024-09-16T00:30:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cc6ffd640 1 -- 172.21.15.33:0/3489953794 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561ff5d6dd70 con 0x561ff5e1afa0 2024-09-16T00:30:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cc6ffd640 10 client.17994 handle_command_reply: tid=0 2024-09-16T00:30:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cefc8f740 1 client.17994 shutdown 2024-09-16T00:30:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cefc8f740 1 -- 172.21.15.33:0/3489953794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 msgr2=0x561ff5e08610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/3489953794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 0x561ff5e08610 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00046c0 tx=0x7f9ce0005fd0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cefc8f740 1 -- 172.21.15.33:0/3489953794 shutdown_connections 2024-09-16T00:30:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/3489953794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 0x561ff5e08610 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/3489953794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e01b40 0x561ff5e08020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/3489953794 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561ff5e1afa0 0x561ff5e1d390 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/3489953794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e005d0 0x561ff5e0ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cefc8f740 1 -- 172.21.15.33:0/3489953794 >> 172.21.15.33:0/3489953794 conn(0x561ff5dc9310 msgr2=0x561ff5e02930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cc6ffd640 0 client.17994 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:30:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cefc8f740 1 -- 172.21.15.33:0/3489953794 shutdown_connections 2024-09-16T00:30:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cefc8f740 1 -- 172.21.15.33:0/3489953794 wait complete. 2024-09-16T00:30:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.977+0000 7f9cefc8f740 20 client.17994 trim_cache size 0 max 16384 2024-09-16T00:30:56.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9cefc8f740 1 Processor -- start 2024-09-16T00:30:56.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9cefc8f740 1 -- start start 2024-09-16T00:30:56.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9cefc8f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e005d0 0x561ff5dfce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9cefc8f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e01b40 0x561ff5e17340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9cefc8f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 0x561ff5e03120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9cefc8f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce81b3be0 con 0x561ff5e0b370 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9cefc8f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce81b14b0 con 0x561ff5e005d0 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9cefc8f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce81af300 con 0x561ff5e01b40 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9ce67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e01b40 0x561ff5e17340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9ce67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e01b40 0x561ff5e17340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:47038/0 (socket says 172.21.15.33:47038) 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9ce67fc640 1 -- 172.21.15.33:0/2969270044 learned_addr learned my addr 172.21.15.33:0/2969270044 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/2969270044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 0x561ff5e03120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.992+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/2969270044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e005d0 0x561ff5dfce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.993+0000 7f9ce67fc640 1 -- 172.21.15.33:0/2969270044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e005d0 msgr2=0x561ff5dfce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.993+0000 7f9ce67fc640 1 --2- 172.21.15.33:0/2969270044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e005d0 0x561ff5dfce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.993+0000 7f9ce67fc640 1 -- 172.21.15.33:0/2969270044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 msgr2=0x561ff5e03120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.993+0000 7f9ce67fc640 1 --2- 172.21.15.33:0/2969270044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 0x561ff5e03120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.993+0000 7f9ce67fc640 1 -- 172.21.15.33:0/2969270044 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ff5dfd730 con 0x561ff5e01b40 2024-09-16T00:30:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.993+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/2969270044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 0x561ff5e03120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:56.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.993+0000 7f9ce67fc640 1 --2- 172.21.15.33:0/2969270044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e01b40 0x561ff5e17340 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x561ff5e078b0 tx=0x7f9cdc005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2a7b010d3940a4d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:56.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.993+0000 7f9ccd7fa640 1 -- 172.21.15.33:0/2969270044 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc005220 con 0x561ff5e01b40 2024-09-16T00:30:56.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.993+0000 7f9ccd7fa640 1 -- 172.21.15.33:0/2969270044 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cdc004c80 con 0x561ff5e01b40 2024-09-16T00:30:56.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.993+0000 7f9ccd7fa640 1 -- 172.21.15.33:0/2969270044 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc0279e0 con 0x561ff5e01b40 2024-09-16T00:30:56.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.994+0000 7f9cefc8f740 1 -- 172.21.15.33:0/2969270044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e01b40 msgr2=0x561ff5e17340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:56.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.994+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/2969270044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e01b40 0x561ff5e17340 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x561ff5e078b0 tx=0x7f9cdc005f90 comp rx=0 tx=0).stop 2024-09-16T00:30:56.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.994+0000 7f9cefc8f740 1 -- 172.21.15.33:0/2969270044 shutdown_connections 2024-09-16T00:30:56.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.994+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/2969270044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e0b370 0x561ff5e03120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.994+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/2969270044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e01b40 0x561ff5e17340 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.994+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/2969270044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e005d0 0x561ff5dfce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.994+0000 7f9cefc8f740 1 -- 172.21.15.33:0/2969270044 >> 172.21.15.33:0/2969270044 conn(0x561ff5dc9310 msgr2=0x561ff5dfc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.994+0000 7f9cefc8f740 1 -- 172.21.15.33:0/2969270044 shutdown_connections 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.994+0000 7f9cefc8f740 1 -- 172.21.15.33:0/2969270044 wait complete. 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.994+0000 7f9cefc8f740 1 -- start start 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9cefc8f740 1 Processor -- start 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9cefc8f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e005d0 0x561ff5e0e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9cefc8f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 0x561ff5e0e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9cefc8f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e0b370 0x561ff5e0eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9cefc8f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce81af970 con 0x561ff5e01b40 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9cefc8f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce81b0de0 con 0x561ff5e0b370 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9cefc8f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce81b0040 con 0x561ff5e005d0 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9cefc8f740 10 client.0 _ll_register_callbacks cb 0x7f9cdc009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:56.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9ce67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 0x561ff5e0e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9ce67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 0x561ff5e0e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60582/0 (socket says 172.21.15.33:60582) 2024-09-16T00:30:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9ce67fc640 1 -- 172.21.15.33:0/4276591782 learned_addr learned my addr 172.21.15.33:0/4276591782 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e005d0 0x561ff5e0e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.995+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e0b370 0x561ff5e0eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9ce77fe640 1 -- 172.21.15.33:0/4276591782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e0b370 msgr2=0x561ff5e0eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e0b370 0x561ff5e0eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9ce77fe640 1 -- 172.21.15.33:0/4276591782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 msgr2=0x561ff5e0e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 0x561ff5e0e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9ce77fe640 1 -- 172.21.15.33:0/4276591782 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ff5e83b40 con 0x561ff5e005d0 2024-09-16T00:30:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9ce67fc640 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 0x561ff5e0e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9ce77fe640 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e005d0 0x561ff5e0e470 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0002230 tx=0x7f9ce0009590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ef22c8c5ecd2782 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9cc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9cc6ffd640 1 -- 172.21.15.33:0/4276591782 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce000f790 con 0x561ff5e005d0 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9cc6ffd640 1 -- 172.21.15.33:0/4276591782 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9ce0005850 con 0x561ff5e005d0 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9cefc8f740 1 -- 172.21.15.33:0/4276591782 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cdc004c80 con 0x561ff5e005d0 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9cc6ffd640 1 -- 172.21.15.33:0/4276591782 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9ce0005a70 con 0x561ff5e005d0 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9cefc8f740 10 client.27364 fetch_fsmap learned FSMap version 30 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9cefc8f740 1 -- 172.21.15.33:0/4276591782 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561ff5e84e40 con 0x561ff5e005d0 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.996+0000 7f9cc6ffd640 1 -- 172.21.15.33:0/4276591782 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cdc004c80 con 0x561ff5e005d0 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.997+0000 7f9cc6ffd640 1 -- 172.21.15.33:0/4276591782 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f9ce0018040 con 0x561ff5e005d0 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.997+0000 7f9cefc8f740 10 client.27364 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.997+0000 7f9cefc8f740 10 client.27364 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.997+0000 7f9cefc8f740 20 client.27364 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.997+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561ff5e854d0 0x561ff5e878f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:57.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.997+0000 7f9cefc8f740 4 client.27364 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.997+0000 7f9cefc8f740 1 -- 172.21.15.33:0/4276591782 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ff5cea170 con 0x561ff5e854d0 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.997+0000 7f9ce67fc640 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561ff5e854d0 0x561ff5e878f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.997+0000 7f9ce67fc640 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561ff5e854d0 0x561ff5e878f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=77da9a2c0216d7e7 server_cookie=41d335439b503a21 in_seq=0 out_seq=0 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.997+0000 7f9cc6ffd640 10 client.27364 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.997+0000 7f9cc6ffd640 1 -- 172.21.15.33:0/4276591782 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ff5cea170 con 0x561ff5e854d0 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.997+0000 7f9cc6ffd640 10 client.27364 handle_command_reply: tid=0 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cefc8f740 1 client.27364 shutdown 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cefc8f740 1 -- 172.21.15.33:0/4276591782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e005d0 msgr2=0x561ff5e0e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e005d0 0x561ff5e0e470 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0002230 tx=0x7f9ce0009590 comp rx=0 tx=0).stop 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cefc8f740 1 -- 172.21.15.33:0/4276591782 shutdown_connections 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561ff5e854d0 0x561ff5e878f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ff5e0b370 0x561ff5e0eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ff5e01b40 0x561ff5e0e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:57.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cefc8f740 1 --2- 172.21.15.33:0/4276591782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff5e005d0 0x561ff5e0e470 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:57.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cc6ffd640 0 client.27364 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:30:57.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cefc8f740 1 -- 172.21.15.33:0/4276591782 >> 172.21.15.33:0/4276591782 conn(0x561ff5dc9310 msgr2=0x561ff5e09880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:57.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cefc8f740 1 -- 172.21.15.33:0/4276591782 shutdown_connections 2024-09-16T00:30:57.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cefc8f740 1 -- 172.21.15.33:0/4276591782 wait complete. 2024-09-16T00:30:57.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.998+0000 7f9cefc8f740 20 client.27364 trim_cache size 0 max 16384 2024-09-16T00:30:57.003 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:30:57.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.999+0000 7f9ced8b6640 1 -- 172.21.15.33:0/115426280 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9cd00a3240 msgr2=0x7f9cd00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:57.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.999+0000 7f9ced8b6640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9cd00a3240 0x7f9cd00a5700 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8000c00 tx=0x7f9cd80f5040 comp rx=0 tx=0).stop 2024-09-16T00:30:57.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.999+0000 7f9ced8b6640 1 -- 172.21.15.33:0/115426280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ce8100e20 msgr2=0x7f9ce80fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:57.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.999+0000 7f9ced8b6640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ce8100e20 0x7f9ce80fcc80 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8006010 tx=0x7f9cd8003c10 comp rx=0 tx=0).stop 2024-09-16T00:30:57.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.999+0000 7f9ced8b6640 1 -- 172.21.15.33:0/115426280 shutdown_connections 2024-09-16T00:30:57.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.999+0000 7f9ced8b6640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9cd00a3240 0x7f9cd00a5700 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:57.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.999+0000 7f9ced8b6640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ce8102ef0 0x7f9ce80fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:57.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.999+0000 7f9ced8b6640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce8102570 0x7f9ce80fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:57.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.999+0000 7f9ced8b6640 1 --2- 172.21.15.33:0/115426280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ce8100e20 0x7f9ce80fcc80 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:57.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.999+0000 7f9ced8b6640 1 -- 172.21.15.33:0/115426280 >> 172.21.15.33:0/115426280 conn(0x7f9ce80fc7d0 msgr2=0x7f9ce806a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:57.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.999+0000 7f9ced8b6640 1 -- 172.21.15.33:0/115426280 shutdown_connections 2024-09-16T00:30:57.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:56.999+0000 7f9ced8b6640 1 -- 172.21.15.33:0/115426280 wait complete. 2024-09-16T00:30:57.062 INFO:tasks.workunit.client.0.smithi033.stdout: 10 1395 83.85 MB/sec warmup 3 sec latency 49.541 ms 2024-09-16T00:30:57.696 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:30:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:57 smithi083 ceph-mon[35242]: pgmap v764: 129 pgs: 129 active+clean; 271 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 2.3 MiB/s wr, 80 op/s 2024-09-16T00:30:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:57 smithi060 ceph-mon[35185]: pgmap v764: 129 pgs: 129 active+clean; 271 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 2.3 MiB/s wr, 80 op/s 2024-09-16T00:30:58.047 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:30:58.062 INFO:tasks.workunit.client.0.smithi033.stdout: 10 2316 70.16 MB/sec warmup 4 sec latency 217.026 ms 2024-09-16T00:30:58.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:57 smithi033 ceph-mon[31243]: pgmap v764: 129 pgs: 129 active+clean; 271 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 2.3 MiB/s wr, 80 op/s 2024-09-16T00:30:58.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.558+0000 7f36ffb11640 1 -- 172.21.15.33:0/1814234350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8102ef0 msgr2=0x7f36f810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:58.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.558+0000 7f36ffb11640 1 --2- 172.21.15.33:0/1814234350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8102ef0 0x7f36f810da40 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009ca0 tx=0x7f36e802a950 comp rx=0 tx=0).stop 2024-09-16T00:30:58.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.558+0000 7f36ffb11640 1 -- 172.21.15.33:0/1814234350 shutdown_connections 2024-09-16T00:30:58.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.558+0000 7f36ffb11640 1 --2- 172.21.15.33:0/1814234350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8102ef0 0x7f36f810da40 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.558+0000 7f36ffb11640 1 --2- 172.21.15.33:0/1814234350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36f8102570 0x7f36f81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.558+0000 7f36ffb11640 1 --2- 172.21.15.33:0/1814234350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36f8100e20 0x7f36f8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.558+0000 7f36ffb11640 1 -- 172.21.15.33:0/1814234350 >> 172.21.15.33:0/1814234350 conn(0x7f36f80fc7d0 msgr2=0x7f36f80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:58.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.558+0000 7f36ffb11640 1 -- 172.21.15.33:0/1814234350 shutdown_connections 2024-09-16T00:30:58.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.558+0000 7f36ffb11640 1 -- 172.21.15.33:0/1814234350 wait complete. 2024-09-16T00:30:58.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36ffb11640 1 Processor -- start 2024-09-16T00:30:58.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36ffb11640 1 -- start start 2024-09-16T00:30:58.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36ffb11640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8100e20 0x7f36f81993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36ffb11640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36f8102570 0x7f36f8199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36ffb11640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36f8102ef0 0x7f36f8199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36ffb11640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36f8110f80 con 0x7f36f8100e20 2024-09-16T00:30:58.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36ffb11640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f36f8110e00 con 0x7f36f8102570 2024-09-16T00:30:58.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36ffb11640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f36f8111100 con 0x7f36f8102ef0 2024-09-16T00:30:58.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36fd8ad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8100e20 0x7f36f81993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36fd8ad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8100e20 0x7f36f81993d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60598/0 (socket says 172.21.15.33:60598) 2024-09-16T00:30:58.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36fe0ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36f8102ef0 0x7f36f8199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36fd8ad640 1 -- 172.21.15.33:0/4162664795 learned_addr learned my addr 172.21.15.33:0/4162664795 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:58.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.559+0000 7f36fd0ac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36f8102570 0x7f36f8199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.560+0000 7f36fd8ad640 1 -- 172.21.15.33:0/4162664795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36f8102ef0 msgr2=0x7f36f8199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:58.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.560+0000 7f36fd8ad640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36f8102ef0 0x7f36f8199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.560+0000 7f36fd8ad640 1 -- 172.21.15.33:0/4162664795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36f8102570 msgr2=0x7f36f8199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:58.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.560+0000 7f36fd8ad640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36f8102570 0x7f36f8199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.560+0000 7f36fd8ad640 1 -- 172.21.15.33:0/4162664795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f81a47d0 con 0x7f36f8100e20 2024-09-16T00:30:58.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.560+0000 7f36fd0ac640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36f8102570 0x7f36f8199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:58.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.560+0000 7f36fd8ad640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8100e20 0x7f36f81993d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f36f000e9f0 tx=0x7f36f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:58.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.560+0000 7f36f6ffd640 1 -- 172.21.15.33:0/4162664795 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f000cd80 con 0x7f36f8100e20 2024-09-16T00:30:58.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.560+0000 7f36f6ffd640 1 -- 172.21.15.33:0/4162664795 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36f0005500 con 0x7f36f8100e20 2024-09-16T00:30:58.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.560+0000 7f36ffb11640 1 -- 172.21.15.33:0/4162664795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f81a4ac0 con 0x7f36f8100e20 2024-09-16T00:30:58.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.560+0000 7f36f6ffd640 1 -- 172.21.15.33:0/4162664795 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f0010690 con 0x7f36f8100e20 2024-09-16T00:30:58.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.560+0000 7f36ffb11640 1 -- 172.21.15.33:0/4162664795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f81a4fb0 con 0x7f36f8100e20 2024-09-16T00:30:58.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f36f6ffd640 1 -- 172.21.15.33:0/4162664795 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f36f00046c0 con 0x7f36f8100e20 2024-09-16T00:30:58.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f3701eea740 1 Processor -- start 2024-09-16T00:30:58.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f3701eea740 1 -- start start 2024-09-16T00:30:58.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f3701eea740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b68b40 0x558df3b67090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f3701eea740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b675d0 0x558df3b67a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f3701eea740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b72370 0x558df3b74760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f3701eea740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558df3acd2f0 con 0x558df3b72370 2024-09-16T00:30:58.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f3701eea740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558df3a74cf0 con 0x558df3b68b40 2024-09-16T00:30:58.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f3701eea740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558df3ac92b0 con 0x558df3b675d0 2024-09-16T00:30:58.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f36fe0ae640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b68b40 0x558df3b67090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f36fd0ac640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b72370 0x558df3b74760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f36fe0ae640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b68b40 0x558df3b67090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:50516/0 (socket says 172.21.15.33:50516) 2024-09-16T00:30:58.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f36fe0ae640 1 -- 172.21.15.33:0/3447141017 learned_addr learned my addr 172.21.15.33:0/3447141017 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:58.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.563+0000 7f36fd8ad640 1 --2- 172.21.15.33:0/3447141017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b675d0 0x558df3b67a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.564+0000 7f36fe0ae640 1 -- 172.21.15.33:0/3447141017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b675d0 msgr2=0x558df3b67a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:58.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.564+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/3447141017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b675d0 0x558df3b67a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.564+0000 7f36fe0ae640 1 -- 172.21.15.33:0/3447141017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b72370 msgr2=0x558df3b74760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:58.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.564+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/3447141017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b72370 0x558df3b74760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.564+0000 7f36fe0ae640 1 -- 172.21.15.33:0/3447141017 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558df3a593a0 con 0x558df3b68b40 2024-09-16T00:30:58.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.564+0000 7f36f6ffd640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f36e00a3240 0x7f36e00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.564+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/3447141017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b68b40 0x558df3b67090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009ca0 tx=0x7f36e80093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb8e72163b919da5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:58.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.564+0000 7f36fd8ad640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f36e00a3240 0x7f36e00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.564+0000 7f36dffff640 1 -- 172.21.15.33:0/3447141017 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e8048070 con 0x558df3b68b40 2024-09-16T00:30:58.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.564+0000 7f36dffff640 1 -- 172.21.15.33:0/3447141017 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e802f930 con 0x558df3b68b40 2024-09-16T00:30:58.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.564+0000 7f36dffff640 1 -- 172.21.15.33:0/3447141017 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e8034040 con 0x558df3b68b40 2024-09-16T00:30:58.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.564+0000 7f36f6ffd640 1 -- 172.21.15.33:0/4162664795 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f36f00d01b0 con 0x7f36f8100e20 2024-09-16T00:30:58.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.565+0000 7f3701eea740 1 -- 172.21.15.33:0/3447141017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b68b40 msgr2=0x558df3b67090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:58.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.565+0000 7f36fd8ad640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f36e00a3240 0x7f36e00a5700 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f36f00cdba0 tx=0x7f36f00baf90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:58.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.565+0000 7f3701eea740 1 --2- 172.21.15.33:0/3447141017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b68b40 0x558df3b67090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009ca0 tx=0x7f36e80093e0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.565+0000 7f3701eea740 1 -- 172.21.15.33:0/3447141017 shutdown_connections 2024-09-16T00:30:58.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.565+0000 7f3701eea740 1 --2- 172.21.15.33:0/3447141017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b72370 0x558df3b74760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.565+0000 7f3701eea740 1 --2- 172.21.15.33:0/3447141017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b675d0 0x558df3b67a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.565+0000 7f3701eea740 1 --2- 172.21.15.33:0/3447141017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b68b40 0x558df3b67090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.565+0000 7f3701eea740 1 -- 172.21.15.33:0/3447141017 >> 172.21.15.33:0/3447141017 conn(0x558df3b30310 msgr2=0x558df3b63c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:58.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.565+0000 7f3701eea740 1 -- 172.21.15.33:0/3447141017 shutdown_connections 2024-09-16T00:30:58.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.565+0000 7f3701eea740 1 -- 172.21.15.33:0/3447141017 wait complete. 2024-09-16T00:30:58.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f3701eea740 1 -- start start 2024-09-16T00:30:58.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f3701eea740 1 Processor -- start 2024-09-16T00:30:58.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f3701eea740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 0x558df3b71dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f3701eea740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b6f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f3701eea740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b72370 0x558df3b6f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f3701eea740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558df3b75010 con 0x558df3b675d0 2024-09-16T00:30:58.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f3701eea740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558df3a88060 con 0x558df3b72370 2024-09-16T00:30:58.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f3701eea740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558df3b75190 con 0x558df3b68b40 2024-09-16T00:30:58.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f3701eea740 10 client.0 _ll_register_callbacks cb 0x558df37db8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:58.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f36fe0ae640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 0x558df3b71dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f36fe0ae640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 0x558df3b71dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60616/0 (socket says 172.21.15.33:60616) 2024-09-16T00:30:58.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f36fe0ae640 1 -- 172.21.15.33:0/3296781568 learned_addr learned my addr 172.21.15.33:0/3296781568 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:58.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.566+0000 7f36fd0ac640 1 --2- 172.21.15.33:0/3296781568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b6f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.567+0000 7f36fe0ae640 1 -- 172.21.15.33:0/3296781568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 msgr2=0x558df3b6f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:58.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.567+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/3296781568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b6f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.567+0000 7f36fe0ae640 1 -- 172.21.15.33:0/3296781568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b72370 msgr2=0x558df3b6f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:58.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.567+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/3296781568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b72370 0x558df3b6f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.567+0000 7f36fe0ae640 1 -- 172.21.15.33:0/3296781568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558df383e3c0 con 0x558df3b675d0 2024-09-16T00:30:58.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.567+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/3296781568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 0x558df3b71dc0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f36e8005cc0 tx=0x7f36e802fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a832a34c8abf20 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:58.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.567+0000 7f36dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:30:58.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.567+0000 7f36dd7fa640 1 -- 172.21.15.33:0/3296781568 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e8051060 con 0x558df3b675d0 2024-09-16T00:30:58.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.567+0000 7f3701eea740 1 -- 172.21.15.33:0/3296781568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36e802f930 con 0x558df3b675d0 2024-09-16T00:30:58.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.567+0000 7f36dd7fa640 1 -- 172.21.15.33:0/3296781568 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f36e8030650 con 0x558df3b675d0 2024-09-16T00:30:58.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.568+0000 7f36dd7fa640 1 -- 172.21.15.33:0/3296781568 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f36e8058050 con 0x558df3b675d0 2024-09-16T00:30:58.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.568+0000 7f3701eea740 10 client.18012 fetch_fsmap learned FSMap version 31 2024-09-16T00:30:58.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.568+0000 7f3701eea740 1 -- 172.21.15.33:0/3296781568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558df3b80b50 con 0x558df3b675d0 2024-09-16T00:30:58.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.568+0000 7f36dd7fa640 1 -- 172.21.15.33:0/3296781568 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e802f930 con 0x558df3b675d0 2024-09-16T00:30:58.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.569+0000 7f36dd7fa640 1 -- 172.21.15.33:0/3296781568 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f36e804c790 con 0x558df3b675d0 2024-09-16T00:30:58.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.569+0000 7f3701eea740 10 client.18012 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:30:58.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.569+0000 7f3701eea740 10 client.18012 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:30:58.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.570+0000 7f3701eea740 20 client.18012 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:58.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.570+0000 7f3701eea740 1 --2- 172.21.15.33:0/3296781568 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558df3b81ea0 0x558df3b84290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.570+0000 7f3701eea740 4 client.18012 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:30:58.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.570+0000 7f3701eea740 1 -- 172.21.15.33:0/3296781568 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558df3ada860 con 0x558df3b81ea0 2024-09-16T00:30:58.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.570+0000 7f36fd0ac640 1 --2- 172.21.15.33:0/3296781568 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558df3b81ea0 0x558df3b84290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.570+0000 7f36fd0ac640 1 --2- 172.21.15.33:0/3296781568 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558df3b81ea0 0x558df3b84290 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb03f5c715ecef31 server_cookie=bdfbd9db4b4b1c04 in_seq=0 out_seq=0 2024-09-16T00:30:58.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.571+0000 7f36dd7fa640 10 client.18012 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:58.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.571+0000 7f36dd7fa640 1 -- 172.21.15.33:0/3296781568 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558df3ada860 con 0x558df3b81ea0 2024-09-16T00:30:58.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.571+0000 7f36dd7fa640 10 client.18012 handle_command_reply: tid=0 2024-09-16T00:30:58.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.571+0000 7f3701eea740 1 client.18012 shutdown 2024-09-16T00:30:58.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.571+0000 7f3701eea740 1 -- 172.21.15.33:0/3296781568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 msgr2=0x558df3b71dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:58.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.571+0000 7f3701eea740 1 --2- 172.21.15.33:0/3296781568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 0x558df3b71dc0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f36e8005cc0 tx=0x7f36e802fd10 comp rx=0 tx=0).stop 2024-09-16T00:30:58.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.572+0000 7f3701eea740 1 -- 172.21.15.33:0/3296781568 shutdown_connections 2024-09-16T00:30:58.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.572+0000 7f3701eea740 1 --2- 172.21.15.33:0/3296781568 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558df3b81ea0 0x558df3b84290 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.572+0000 7f3701eea740 1 --2- 172.21.15.33:0/3296781568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b72370 0x558df3b6f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.572+0000 7f3701eea740 1 --2- 172.21.15.33:0/3296781568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b6f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.572+0000 7f3701eea740 1 --2- 172.21.15.33:0/3296781568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 0x558df3b71dc0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.572+0000 7f36dd7fa640 0 client.18012 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:58.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.572+0000 7f3701eea740 1 -- 172.21.15.33:0/3296781568 >> 172.21.15.33:0/3296781568 conn(0x558df3b30310 msgr2=0x558df3b69960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:58.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.572+0000 7f3701eea740 1 -- 172.21.15.33:0/3296781568 shutdown_connections 2024-09-16T00:30:58.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.572+0000 7f3701eea740 1 -- 172.21.15.33:0/3296781568 wait complete. 2024-09-16T00:30:58.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.572+0000 7f3701eea740 20 client.18012 trim_cache size 0 max 16384 2024-09-16T00:30:58.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.586+0000 7f3701eea740 1 Processor -- start 2024-09-16T00:30:58.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.586+0000 7f3701eea740 1 -- start start 2024-09-16T00:30:58.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f3701eea740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b675d0 0x558df3b6e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f3701eea740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b89a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f3701eea740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b72370 0x558df3b89f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f3701eea740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36f81af770 con 0x558df3b72370 2024-09-16T00:30:58.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f3701eea740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f36f81acfe0 con 0x558df3b675d0 2024-09-16T00:30:58.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f3701eea740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f36f81aae30 con 0x558df3b68b40 2024-09-16T00:30:58.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fe0ae640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b72370 0x558df3b89f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fd0ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b89a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fd0ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b89a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:47106/0 (socket says 172.21.15.33:47106) 2024-09-16T00:30:58.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fd0ac640 1 -- 172.21.15.33:0/3400815888 learned_addr learned my addr 172.21.15.33:0/3400815888 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:58.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/3400815888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b72370 0x558df3b89f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60630/0 (socket says 172.21.15.33:60630) 2024-09-16T00:30:58.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/3400815888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b675d0 0x558df3b6e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fd0ac640 1 -- 172.21.15.33:0/3400815888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b675d0 msgr2=0x558df3b6e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:58.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fd0ac640 1 --2- 172.21.15.33:0/3400815888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b675d0 0x558df3b6e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fd0ac640 1 -- 172.21.15.33:0/3400815888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b72370 msgr2=0x558df3b89f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:58.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fd0ac640 1 --2- 172.21.15.33:0/3400815888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b72370 0x558df3b89f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fd0ac640 1 -- 172.21.15.33:0/3400815888 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558df3b6a120 con 0x558df3b68b40 2024-09-16T00:30:58.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/3400815888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b72370 0x558df3b89f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:58.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.587+0000 7f36fd0ac640 1 --2- 172.21.15.33:0/3400815888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b89a00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x558df3b6e900 tx=0x7f36ec016eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c90d43a7e790004 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:58.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.588+0000 7f36df7fe640 1 -- 172.21.15.33:0/3400815888 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec0150d0 con 0x558df3b68b40 2024-09-16T00:30:58.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.588+0000 7f36df7fe640 1 -- 172.21.15.33:0/3400815888 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36ec0137d0 con 0x558df3b68b40 2024-09-16T00:30:58.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.588+0000 7f36df7fe640 1 -- 172.21.15.33:0/3400815888 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec003970 con 0x558df3b68b40 2024-09-16T00:30:58.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.588+0000 7f3701eea740 1 -- 172.21.15.33:0/3400815888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 msgr2=0x558df3b89a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:58.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.588+0000 7f3701eea740 1 --2- 172.21.15.33:0/3400815888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b89a00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x558df3b6e900 tx=0x7f36ec016eb0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.589+0000 7f3701eea740 1 -- 172.21.15.33:0/3400815888 shutdown_connections 2024-09-16T00:30:58.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.589+0000 7f3701eea740 1 --2- 172.21.15.33:0/3400815888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b72370 0x558df3b89f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.589+0000 7f3701eea740 1 --2- 172.21.15.33:0/3400815888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b89a00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.589+0000 7f3701eea740 1 --2- 172.21.15.33:0/3400815888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b675d0 0x558df3b6e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.589+0000 7f3701eea740 1 -- 172.21.15.33:0/3400815888 >> 172.21.15.33:0/3400815888 conn(0x558df3b30310 msgr2=0x558df3b5ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:58.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.589+0000 7f3701eea740 1 -- 172.21.15.33:0/3400815888 shutdown_connections 2024-09-16T00:30:58.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.589+0000 7f3701eea740 1 -- 172.21.15.33:0/3400815888 wait complete. 2024-09-16T00:30:58.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.589+0000 7f3701eea740 1 -- start start 2024-09-16T00:30:58.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.589+0000 7f3701eea740 1 Processor -- start 2024-09-16T00:30:58.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f3701eea740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 0x558df3b75b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f3701eea740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b760b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f3701eea740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b72370 0x558df3be4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f3701eea740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36f81ab4a0 con 0x558df3b675d0 2024-09-16T00:30:58.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f3701eea740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f36f81ac910 con 0x558df3b72370 2024-09-16T00:30:58.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f3701eea740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f36f81abb70 con 0x558df3b68b40 2024-09-16T00:30:58.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f3701eea740 10 client.0 _ll_register_callbacks cb 0x7f36e8046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:30:58.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f36fd0ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f36fd0ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b760b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:47110/0 (socket says 172.21.15.33:47110) 2024-09-16T00:30:58.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f36fd0ac640 1 -- 172.21.15.33:0/2624600436 learned_addr learned my addr 172.21.15.33:0/2624600436 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:30:58.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 0x558df3b75b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b72370 0x558df3be4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f36fd0ac640 1 -- 172.21.15.33:0/2624600436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b72370 msgr2=0x558df3be4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:58.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f36fd0ac640 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b72370 0x558df3be4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f36fd0ac640 1 -- 172.21.15.33:0/2624600436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 msgr2=0x558df3b75b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:30:58.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f36fd0ac640 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 0x558df3b75b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f36fd0ac640 1 -- 172.21.15.33:0/2624600436 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558df3be5040 con 0x558df3b68b40 2024-09-16T00:30:58.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.590+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 0x558df3b75b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:30:58.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.591+0000 7f36fd0ac640 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b760b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009fd0 tx=0x7f36ec005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=316d4bb3b7557262 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:30:58.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.591+0000 7f36dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:30:58.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.591+0000 7f36dd7fa640 1 -- 172.21.15.33:0/2624600436 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec0153a0 con 0x558df3b68b40 2024-09-16T00:30:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.591+0000 7f3701eea740 1 -- 172.21.15.33:0/2624600436 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36ec0137d0 con 0x558df3b68b40 2024-09-16T00:30:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.591+0000 7f36dd7fa640 1 -- 172.21.15.33:0/2624600436 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f36ec015540 con 0x558df3b68b40 2024-09-16T00:30:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.591+0000 7f36dd7fa640 1 -- 172.21.15.33:0/2624600436 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f36ec03c050 con 0x558df3b68b40 2024-09-16T00:30:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.591+0000 7f3701eea740 10 client.27382 fetch_fsmap learned FSMap version 31 2024-09-16T00:30:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.591+0000 7f3701eea740 1 -- 172.21.15.33:0/2624600436 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558df3bec3f0 con 0x558df3b68b40 2024-09-16T00:30:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.591+0000 7f36dd7fa640 1 -- 172.21.15.33:0/2624600436 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36ec0137d0 con 0x558df3b68b40 2024-09-16T00:30:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.592+0000 7f36dd7fa640 1 -- 172.21.15.33:0/2624600436 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f36ec032030 con 0x558df3b68b40 2024-09-16T00:30:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.592+0000 7f3701eea740 10 client.27382 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:30:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.592+0000 7f3701eea740 10 client.27382 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:30:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.592+0000 7f3701eea740 20 client.27382 populate_metadata read hostname 'smithi033' 2024-09-16T00:30:58.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.592+0000 7f3701eea740 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558df3beca90 0x558df3beeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:30:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.592+0000 7f3701eea740 4 client.27382 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:30:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.592+0000 7f3701eea740 1 -- 172.21.15.33:0/2624600436 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558df3ad4d70 con 0x558df3beca90 2024-09-16T00:30:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.592+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558df3beca90 0x558df3beeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:30:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.592+0000 7f36dd7fa640 10 client.27382 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f36fe0ae640 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558df3beca90 0x558df3beeeb0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e94a282561fe8cc6 server_cookie=9f2fda7580a1b54b in_seq=0 out_seq=0 2024-09-16T00:30:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f36dd7fa640 1 -- 172.21.15.33:0/2624600436 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558df3ad4d70 con 0x558df3beca90 2024-09-16T00:30:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f36dd7fa640 10 client.27382 handle_command_reply: tid=0 2024-09-16T00:30:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f3701eea740 1 client.27382 shutdown 2024-09-16T00:30:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f3701eea740 1 -- 172.21.15.33:0/2624600436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 msgr2=0x558df3b760b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f3701eea740 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b760b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009fd0 tx=0x7f36ec005780 comp rx=0 tx=0).stop 2024-09-16T00:30:58.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f3701eea740 1 -- 172.21.15.33:0/2624600436 shutdown_connections 2024-09-16T00:30:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f3701eea740 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558df3beca90 0x558df3beeeb0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f3701eea740 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558df3b72370 0x558df3be4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f3701eea740 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558df3b68b40 0x558df3b760b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f3701eea740 1 --2- 172.21.15.33:0/2624600436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558df3b675d0 0x558df3b75b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f3701eea740 1 -- 172.21.15.33:0/2624600436 >> 172.21.15.33:0/2624600436 conn(0x558df3b30310 msgr2=0x558df3b690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f36dd7fa640 0 client.27382 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:30:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f3701eea740 1 -- 172.21.15.33:0/2624600436 shutdown_connections 2024-09-16T00:30:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f3701eea740 1 -- 172.21.15.33:0/2624600436 wait complete. 2024-09-16T00:30:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.593+0000 7f3701eea740 20 client.27382 trim_cache size 0 max 16384 2024-09-16T00:30:58.607 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:30:58.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.594+0000 7f36ffb11640 1 -- 172.21.15.33:0/4162664795 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f36e00a3240 msgr2=0x7f36e00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:58.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.594+0000 7f36ffb11640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f36e00a3240 0x7f36e00a5700 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f36f00cdba0 tx=0x7f36f00baf90 comp rx=0 tx=0).stop 2024-09-16T00:30:58.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.594+0000 7f36ffb11640 1 -- 172.21.15.33:0/4162664795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8100e20 msgr2=0x7f36f81993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:30:58.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.594+0000 7f36ffb11640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8100e20 0x7f36f81993d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f36f000e9f0 tx=0x7f36f000edf0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.594+0000 7f36ffb11640 1 -- 172.21.15.33:0/4162664795 shutdown_connections 2024-09-16T00:30:58.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.594+0000 7f36ffb11640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f36e00a3240 0x7f36e00a5700 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.594+0000 7f36ffb11640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36f8102ef0 0x7f36f8199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.594+0000 7f36ffb11640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36f8102570 0x7f36f8199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.594+0000 7f36ffb11640 1 --2- 172.21.15.33:0/4162664795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8100e20 0x7f36f81993d0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:30:58.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.594+0000 7f36ffb11640 1 -- 172.21.15.33:0/4162664795 >> 172.21.15.33:0/4162664795 conn(0x7f36f80fc7d0 msgr2=0x7f36f81068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:30:58.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.594+0000 7f36ffb11640 1 -- 172.21.15.33:0/4162664795 shutdown_connections 2024-09-16T00:30:58.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:30:58.594+0000 7f36ffb11640 1 -- 172.21.15.33:0/4162664795 wait complete. 2024-09-16T00:30:58.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:58 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:30:58.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:58 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:30:58.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:58 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:30:59.062 INFO:tasks.workunit.client.0.smithi033.stdout: 10 3065 66.50 MB/sec warmup 5 sec latency 808.215 ms 2024-09-16T00:30:59.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:30:59 smithi083 ceph-mon[35242]: pgmap v765: 129 pgs: 129 active+clean; 268 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.0 MiB/s wr, 75 op/s 2024-09-16T00:31:00.137 INFO:tasks.workunit.client.0.smithi033.stdout: 10 3758 64.33 MB/sec warmup 6 sec latency 84.834 ms 2024-09-16T00:31:00.137 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:30:59 smithi060 ceph-mon[35185]: pgmap v765: 129 pgs: 129 active+clean; 268 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.0 MiB/s wr, 75 op/s 2024-09-16T00:31:00.137 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:30:59 smithi033 ceph-mon[31243]: pgmap v765: 129 pgs: 129 active+clean; 268 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.0 MiB/s wr, 75 op/s 2024-09-16T00:31:00.368 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:31:00.369 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:31:00.643 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:01.062 INFO:tasks.workunit.client.0.smithi033.stdout: 10 4270 59.81 MB/sec warmup 7 sec latency 87.864 ms 2024-09-16T00:31:01.683 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:01 smithi033 ceph-mon[31243]: pgmap v766: 129 pgs: 129 active+clean; 264 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 2.5 MiB/s wr, 112 op/s 2024-09-16T00:31:01.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.795+0000 7ff70f888640 1 -- 172.21.15.33:0/647820807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff708102d30 msgr2=0x7ff708103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:01.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.795+0000 7ff70f888640 1 --2- 172.21.15.33:0/647820807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff708102d30 0x7ff708103110 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7ff704009930 tx=0x7ff70402a420 comp rx=0 tx=0).stop 2024-09-16T00:31:01.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.795+0000 7ff70f888640 1 -- 172.21.15.33:0/647820807 shutdown_connections 2024-09-16T00:31:01.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.795+0000 7ff70f888640 1 --2- 172.21.15.33:0/647820807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7081082a0 0x7ff70810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.795+0000 7ff70f888640 1 --2- 172.21.15.33:0/647820807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7081036e0 0x7ff708107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.795+0000 7ff70f888640 1 --2- 172.21.15.33:0/647820807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff708102d30 0x7ff708103110 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.795+0000 7ff70f888640 1 -- 172.21.15.33:0/647820807 >> 172.21.15.33:0/647820807 conn(0x7ff7080fc7d0 msgr2=0x7ff7080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:01.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.796+0000 7ff70f888640 1 -- 172.21.15.33:0/647820807 shutdown_connections 2024-09-16T00:31:01.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.796+0000 7ff70f888640 1 -- 172.21.15.33:0/647820807 wait complete. 2024-09-16T00:31:01.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.796+0000 7ff70f888640 1 Processor -- start 2024-09-16T00:31:01.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.796+0000 7ff70f888640 1 -- start start 2024-09-16T00:31:01.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70f888640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff708102d30 0x7ff708100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70f888640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7081036e0 0x7ff708100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70f888640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7081082a0 0x7ff708101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70f888640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081120a0 con 0x7ff7081036e0 2024-09-16T00:31:01.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70f888640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff708111f20 con 0x7ff708102d30 2024-09-16T00:31:01.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70f888640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff708112220 con 0x7ff7081082a0 2024-09-16T00:31:01.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70d624640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff708102d30 0x7ff708100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70d624640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff708102d30 0x7ff708100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:37916/0 (socket says 172.21.15.33:37916) 2024-09-16T00:31:01.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70d624640 1 -- 172.21.15.33:0/2911643547 learned_addr learned my addr 172.21.15.33:0/2911643547 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:01.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70ce23640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7081036e0 0x7ff708100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70de25640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7081082a0 0x7ff708101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70d624640 1 -- 172.21.15.33:0/2911643547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7081082a0 msgr2=0x7ff708101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:01.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70d624640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7081082a0 0x7ff708101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70d624640 1 -- 172.21.15.33:0/2911643547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7081036e0 msgr2=0x7ff708100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:01.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70d624640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7081036e0 0x7ff708100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70d624640 1 -- 172.21.15.33:0/2911643547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7081a6a80 con 0x7ff708102d30 2024-09-16T00:31:01.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.797+0000 7ff70ce23640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7081036e0 0x7ff708100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:01.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.798+0000 7ff70d624640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff708102d30 0x7ff708100990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff704009230 tx=0x7ff704004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:01.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.798+0000 7ff6fe7fc640 1 -- 172.21.15.33:0/2911643547 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70403f030 con 0x7ff708102d30 2024-09-16T00:31:01.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.798+0000 7ff70f888640 1 -- 172.21.15.33:0/2911643547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7081a6d10 con 0x7ff708102d30 2024-09-16T00:31:01.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.798+0000 7ff70f888640 1 -- 172.21.15.33:0/2911643547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7081a71e0 con 0x7ff708102d30 2024-09-16T00:31:01.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.798+0000 7ff6fe7fc640 1 -- 172.21.15.33:0/2911643547 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff704004440 con 0x7ff708102d30 2024-09-16T00:31:01.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.798+0000 7ff6fe7fc640 1 -- 172.21.15.33:0/2911643547 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff704035070 con 0x7ff708102d30 2024-09-16T00:31:01.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.801+0000 7ff6fe7fc640 1 -- 172.21.15.33:0/2911643547 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff704042030 con 0x7ff708102d30 2024-09-16T00:31:01.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.801+0000 7ff711c61740 1 Processor -- start 2024-09-16T00:31:01.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.801+0000 7ff711c61740 1 -- start start 2024-09-16T00:31:01.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff711c61740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406521f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff711c61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e406522480 0x55e40652c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff711c61740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e40652cd40 0x55e40652f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff711c61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e406487280 con 0x55e406522480 2024-09-16T00:31:01.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff711c61740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e40642eb10 con 0x55e406521b60 2024-09-16T00:31:01.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff711c61740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e406483240 con 0x55e40652cd40 2024-09-16T00:31:01.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff70de25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406521f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff70de25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406521f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:37920/0 (socket says 172.21.15.33:37920) 2024-09-16T00:31:01.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff70de25640 1 -- 172.21.15.33:0/2185882981 learned_addr learned my addr 172.21.15.33:0/2185882981 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:01.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff70ce23640 1 --2- 172.21.15.33:0/2185882981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e40652cd40 0x55e40652f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff70d624640 1 --2- 172.21.15.33:0/2185882981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e406522480 0x55e40652c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff6fe7fc640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff6f40a32c0 0x7ff6f40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff70ce23640 1 -- 172.21.15.33:0/2185882981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 msgr2=0x55e406521f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:01.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff70ce23640 1 --2- 172.21.15.33:0/2185882981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406521f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff70ce23640 1 -- 172.21.15.33:0/2185882981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e406522480 msgr2=0x55e40652c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:01.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff70ce23640 1 --2- 172.21.15.33:0/2185882981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e406522480 0x55e40652c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.802+0000 7ff70ce23640 1 -- 172.21.15.33:0/2185882981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e40643b1f0 con 0x55e40652cd40 2024-09-16T00:31:01.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.803+0000 7ff70de25640 1 --2- 172.21.15.33:0/2185882981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406521f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:01.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.803+0000 7ff6fe7fc640 1 -- 172.21.15.33:0/2911643547 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff7040f0010 con 0x7ff708102d30 2024-09-16T00:31:01.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.803+0000 7ff70de25640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff6f40a32c0 0x7ff6f40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.803+0000 7ff70ce23640 1 --2- 172.21.15.33:0/2185882981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e40652cd40 0x55e40652f130 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff70806a370 tx=0x7ff6f80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bb5de3eb34ce9b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:01.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.803+0000 7ff70de25640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff6f40a32c0 0x7ff6f40a5780 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff7000107d0 tx=0x7ff700004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:01.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.803+0000 7ff6f37fe640 1 -- 172.21.15.33:0/2185882981 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8004310 con 0x55e40652cd40 2024-09-16T00:31:01.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.803+0000 7ff6f37fe640 1 -- 172.21.15.33:0/2185882981 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6f80044b0 con 0x55e40652cd40 2024-09-16T00:31:01.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.803+0000 7ff6f37fe640 1 -- 172.21.15.33:0/2185882981 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f800fae0 con 0x55e40652cd40 2024-09-16T00:31:01.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.804+0000 7ff711c61740 1 -- 172.21.15.33:0/2185882981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e40652cd40 msgr2=0x55e40652f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:01.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.804+0000 7ff711c61740 1 --2- 172.21.15.33:0/2185882981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e40652cd40 0x55e40652f130 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff70806a370 tx=0x7ff6f80118d0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.804+0000 7ff711c61740 1 -- 172.21.15.33:0/2185882981 shutdown_connections 2024-09-16T00:31:01.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.804+0000 7ff711c61740 1 --2- 172.21.15.33:0/2185882981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e40652cd40 0x55e40652f130 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.804+0000 7ff711c61740 1 --2- 172.21.15.33:0/2185882981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e406522480 0x55e40652c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.804+0000 7ff711c61740 1 --2- 172.21.15.33:0/2185882981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406521f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.804+0000 7ff711c61740 1 -- 172.21.15.33:0/2185882981 >> 172.21.15.33:0/2185882981 conn(0x55e40651e0f0 msgr2=0x55e40651e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:01.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.804+0000 7ff711c61740 1 -- 172.21.15.33:0/2185882981 shutdown_connections 2024-09-16T00:31:01.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.804+0000 7ff711c61740 1 -- 172.21.15.33:0/2185882981 wait complete. 2024-09-16T00:31:01.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.804+0000 7ff711c61740 1 -- start start 2024-09-16T00:31:01.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.804+0000 7ff711c61740 1 Processor -- start 2024-09-16T00:31:01.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.805+0000 7ff711c61740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406529be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.805+0000 7ff711c61740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406522480 0x55e40652a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.805+0000 7ff711c61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 0x55e40652a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.805+0000 7ff711c61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e40652f910 con 0x55e40652cd40 2024-09-16T00:31:01.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.805+0000 7ff711c61740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e406441f10 con 0x55e406521b60 2024-09-16T00:31:01.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.805+0000 7ff711c61740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e40652fa90 con 0x55e406522480 2024-09-16T00:31:01.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.805+0000 7ff70de25640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 0x55e40652a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.805+0000 7ff70de25640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 0x55e40652a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51756/0 (socket says 172.21.15.33:51756) 2024-09-16T00:31:01.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.805+0000 7ff711c61740 10 client.0 _ll_register_callbacks cb 0x55e40640aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:01.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.805+0000 7ff70de25640 1 -- 172.21.15.33:0/1363449961 learned_addr learned my addr 172.21.15.33:0/1363449961 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:01.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.805+0000 7ff70ce23640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406522480 0x55e40652a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.805+0000 7ff70d624640 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406529be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.806+0000 7ff70de25640 1 -- 172.21.15.33:0/1363449961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406522480 msgr2=0x55e40652a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:01.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.806+0000 7ff70de25640 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406522480 0x55e40652a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.806+0000 7ff70de25640 1 -- 172.21.15.33:0/1363449961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 msgr2=0x55e406529be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:01.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.806+0000 7ff70de25640 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406529be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.806+0000 7ff70de25640 1 -- 172.21.15.33:0/1363449961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4061f86a0 con 0x55e40652cd40 2024-09-16T00:31:01.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.806+0000 7ff70ce23640 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406522480 0x55e40652a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:01.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.806+0000 7ff70d624640 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406529be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:01.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.807+0000 7ff70de25640 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 0x55e40652a660 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ff70000ff10 tx=0x7ff700004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f4080e00a9a914b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:01.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.807+0000 7ff6f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:01.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.807+0000 7ff6f0ff9640 1 -- 172.21.15.33:0/1363449961 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff700004d20 con 0x55e40652cd40 2024-09-16T00:31:01.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.807+0000 7ff6f0ff9640 1 -- 172.21.15.33:0/1363449961 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff700005020 con 0x55e40652cd40 2024-09-16T00:31:01.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.807+0000 7ff711c61740 1 -- 172.21.15.33:0/1363449961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6f80044b0 con 0x55e40652cd40 2024-09-16T00:31:01.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.807+0000 7ff6f0ff9640 1 -- 172.21.15.33:0/1363449961 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff700030050 con 0x55e40652cd40 2024-09-16T00:31:01.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.807+0000 7ff711c61740 10 client.18030 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:01.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.807+0000 7ff711c61740 1 -- 172.21.15.33:0/1363449961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e40653b540 con 0x55e40652cd40 2024-09-16T00:31:01.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.808+0000 7ff6f0ff9640 1 -- 172.21.15.33:0/1363449961 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6f80044b0 con 0x55e40652cd40 2024-09-16T00:31:01.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.808+0000 7ff6f0ff9640 1 -- 172.21.15.33:0/1363449961 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff700026030 con 0x55e40652cd40 2024-09-16T00:31:01.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.808+0000 7ff711c61740 10 client.18030 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:01.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.808+0000 7ff711c61740 10 client.18030 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:31:01.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.808+0000 7ff711c61740 20 client.18030 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:01.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.808+0000 7ff711c61740 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e40653ca00 0x55e40653edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.808+0000 7ff711c61740 4 client.18030 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:31:01.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.808+0000 7ff711c61740 1 -- 172.21.15.33:0/1363449961 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4064947f0 con 0x55e40653ca00 2024-09-16T00:31:01.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.808+0000 7ff70d624640 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e40653ca00 0x55e40653edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.809+0000 7ff70d624640 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e40653ca00 0x55e40653edf0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e94753656e4ecd43 server_cookie=1571b10be076a91e in_seq=0 out_seq=0 2024-09-16T00:31:01.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.809+0000 7ff6f0ff9640 10 client.18030 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:01.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.809+0000 7ff6f0ff9640 1 -- 172.21.15.33:0/1363449961 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e4064947f0 con 0x55e40653ca00 2024-09-16T00:31:01.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.809+0000 7ff6f0ff9640 10 client.18030 handle_command_reply: tid=0 2024-09-16T00:31:01.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.809+0000 7ff711c61740 1 client.18030 shutdown 2024-09-16T00:31:01.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.809+0000 7ff711c61740 1 -- 172.21.15.33:0/1363449961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 msgr2=0x55e40652a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:01.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.809+0000 7ff711c61740 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 0x55e40652a660 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ff70000ff10 tx=0x7ff700004ab0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.810+0000 7ff711c61740 1 -- 172.21.15.33:0/1363449961 shutdown_connections 2024-09-16T00:31:01.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.810+0000 7ff711c61740 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e40653ca00 0x55e40653edf0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.810+0000 7ff6f0ff9640 0 client.18030 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:01.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.810+0000 7ff711c61740 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 0x55e40652a660 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.810+0000 7ff711c61740 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406522480 0x55e40652a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.810+0000 7ff711c61740 1 --2- 172.21.15.33:0/1363449961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406529be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.810+0000 7ff711c61740 1 -- 172.21.15.33:0/1363449961 >> 172.21.15.33:0/1363449961 conn(0x55e40651e0f0 msgr2=0x55e406523e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:01.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.810+0000 7ff711c61740 1 -- 172.21.15.33:0/1363449961 shutdown_connections 2024-09-16T00:31:01.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.810+0000 7ff711c61740 1 -- 172.21.15.33:0/1363449961 wait complete. 2024-09-16T00:31:01.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.810+0000 7ff711c61740 20 client.18030 trim_cache size 0 max 16384 2024-09-16T00:31:01.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff711c61740 1 Processor -- start 2024-09-16T00:31:01.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff711c61740 1 -- start start 2024-09-16T00:31:01.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff711c61740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406521b60 0x55e406528be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff711c61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e406522480 0x55e406544620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff711c61740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e40652cd40 0x55e406544b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff711c61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081b3d20 con 0x55e406522480 2024-09-16T00:31:01.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff711c61740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081b1590 con 0x55e40652cd40 2024-09-16T00:31:01.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff711c61740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081af3e0 con 0x55e406521b60 2024-09-16T00:31:01.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff70ce23640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e406522480 0x55e406544620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff70ce23640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e406522480 0x55e406544620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51772/0 (socket says 172.21.15.33:51772) 2024-09-16T00:31:01.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff70ce23640 1 -- 172.21.15.33:0/3463802105 learned_addr learned my addr 172.21.15.33:0/3463802105 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:01.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff70de25640 1 --2- 172.21.15.33:0/3463802105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e40652cd40 0x55e406544b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.829+0000 7ff70d624640 1 --2- 172.21.15.33:0/3463802105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406521b60 0x55e406528be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.830+0000 7ff70de25640 1 -- 172.21.15.33:0/3463802105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406521b60 msgr2=0x55e406528be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:01.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.830+0000 7ff70de25640 1 --2- 172.21.15.33:0/3463802105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406521b60 0x55e406528be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.830+0000 7ff70de25640 1 -- 172.21.15.33:0/3463802105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e406522480 msgr2=0x55e406544620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:01.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.830+0000 7ff70de25640 1 --2- 172.21.15.33:0/3463802105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e406522480 0x55e406544620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.830+0000 7ff70de25640 1 -- 172.21.15.33:0/3463802105 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e40652c4b0 con 0x55e40652cd40 2024-09-16T00:31:01.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.830+0000 7ff70ce23640 1 --2- 172.21.15.33:0/3463802105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e406522480 0x55e406544620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:01.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.830+0000 7ff70de25640 1 --2- 172.21.15.33:0/3463802105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e40652cd40 0x55e406544b60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff70000f830 tx=0x7ff70001fc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=305d7d8952445b28 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:01.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.830+0000 7ff6f2ffd640 1 -- 172.21.15.33:0/3463802105 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70001cd30 con 0x55e40652cd40 2024-09-16T00:31:01.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.830+0000 7ff6f2ffd640 1 -- 172.21.15.33:0/3463802105 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7000094e0 con 0x55e40652cd40 2024-09-16T00:31:01.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.830+0000 7ff6f2ffd640 1 -- 172.21.15.33:0/3463802105 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7000273f0 con 0x55e40652cd40 2024-09-16T00:31:01.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.831+0000 7ff711c61740 1 -- 172.21.15.33:0/3463802105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e40652cd40 msgr2=0x55e406544b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:01.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.831+0000 7ff711c61740 1 --2- 172.21.15.33:0/3463802105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e40652cd40 0x55e406544b60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff70000f830 tx=0x7ff70001fc90 comp rx=0 tx=0).stop 2024-09-16T00:31:01.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.831+0000 7ff711c61740 1 -- 172.21.15.33:0/3463802105 shutdown_connections 2024-09-16T00:31:01.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.831+0000 7ff711c61740 1 --2- 172.21.15.33:0/3463802105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e40652cd40 0x55e406544b60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.831+0000 7ff711c61740 1 --2- 172.21.15.33:0/3463802105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e406522480 0x55e406544620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.831+0000 7ff711c61740 1 --2- 172.21.15.33:0/3463802105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406521b60 0x55e406528be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.831+0000 7ff711c61740 1 -- 172.21.15.33:0/3463802105 >> 172.21.15.33:0/3463802105 conn(0x55e40651e0f0 msgr2=0x55e4065284d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:01.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.831+0000 7ff711c61740 1 -- 172.21.15.33:0/3463802105 shutdown_connections 2024-09-16T00:31:01.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.831+0000 7ff711c61740 1 -- 172.21.15.33:0/3463802105 wait complete. 2024-09-16T00:31:01.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.831+0000 7ff711c61740 1 -- start start 2024-09-16T00:31:01.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.831+0000 7ff711c61740 1 Processor -- start 2024-09-16T00:31:01.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.832+0000 7ff711c61740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406535260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.832+0000 7ff711c61740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406522480 0x55e406526710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.832+0000 7ff711c61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 0x55e406526c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.832+0000 7ff711c61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081afa50 con 0x55e40652cd40 2024-09-16T00:31:01.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.832+0000 7ff711c61740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081b0ec0 con 0x55e406521b60 2024-09-16T00:31:01.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.832+0000 7ff711c61740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081b0120 con 0x55e406522480 2024-09-16T00:31:01.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.832+0000 7ff711c61740 10 client.0 _ll_register_callbacks cb 0x7ff700009c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:01.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.832+0000 7ff70d624640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406535260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.832+0000 7ff70de25640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 0x55e406526c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.832+0000 7ff70ce23640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406522480 0x55e406526710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.832+0000 7ff70d624640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406535260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:37946/0 (socket says 172.21.15.33:37946) 2024-09-16T00:31:01.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.832+0000 7ff70d624640 1 -- 172.21.15.33:0/2650595002 learned_addr learned my addr 172.21.15.33:0/2650595002 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:01.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.833+0000 7ff70d624640 1 -- 172.21.15.33:0/2650595002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406522480 msgr2=0x55e406526710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:01.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.833+0000 7ff70d624640 1 --2- 172.21.15.33:0/2650595002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406522480 0x55e406526710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.833+0000 7ff70d624640 1 -- 172.21.15.33:0/2650595002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 msgr2=0x55e406526c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:01.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.833+0000 7ff70d624640 1 --2- 172.21.15.33:0/2650595002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 0x55e406526c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.833+0000 7ff70d624640 1 -- 172.21.15.33:0/2650595002 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e406527240 con 0x55e406521b60 2024-09-16T00:31:01.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.833+0000 7ff70de25640 1 --2- 172.21.15.33:0/2650595002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 0x55e406526c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:01.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.833+0000 7ff70d624640 1 --2- 172.21.15.33:0/2650595002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406535260 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff7040ed690 tx=0x7ff7040dc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a50399938aa39597 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:01.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.833+0000 7ff6f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:31:01.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.833+0000 7ff6f0ff9640 1 -- 172.21.15.33:0/2650595002 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff704049040 con 0x55e406521b60 2024-09-16T00:31:01.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.833+0000 7ff6f0ff9640 1 -- 172.21.15.33:0/2650595002 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff7040ba560 con 0x55e406521b60 2024-09-16T00:31:01.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.833+0000 7ff711c61740 1 -- 172.21.15.33:0/2650595002 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7000094e0 con 0x55e406521b60 2024-09-16T00:31:01.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.833+0000 7ff6f0ff9640 1 -- 172.21.15.33:0/2650595002 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff7040bc260 con 0x55e406521b60 2024-09-16T00:31:01.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.834+0000 7ff711c61740 10 client.27395 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:01.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.834+0000 7ff711c61740 1 -- 172.21.15.33:0/2650595002 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e4065a7440 con 0x55e406521b60 2024-09-16T00:31:01.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.834+0000 7ff6f0ff9640 1 -- 172.21.15.33:0/2650595002 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7000094e0 con 0x55e406521b60 2024-09-16T00:31:01.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.834+0000 7ff6f0ff9640 1 -- 172.21.15.33:0/2650595002 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff7040b7fb0 con 0x55e406521b60 2024-09-16T00:31:01.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.834+0000 7ff711c61740 10 client.27395 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:01.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.834+0000 7ff711c61740 10 client.27395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:31:01.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.834+0000 7ff711c61740 20 client.27395 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:01.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.834+0000 7ff711c61740 1 --2- 172.21.15.33:0/2650595002 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e40653ca00 0x55e4065a9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:01.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.834+0000 7ff711c61740 4 client.27395 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:31:01.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.834+0000 7ff711c61740 1 -- 172.21.15.33:0/2650595002 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e40648ed00 con 0x55e40653ca00 2024-09-16T00:31:01.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.834+0000 7ff70ce23640 1 --2- 172.21.15.33:0/2650595002 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e40653ca00 0x55e4065a9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:01.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.835+0000 7ff70ce23640 1 --2- 172.21.15.33:0/2650595002 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e40653ca00 0x55e4065a9bb0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adfb62056dd5e5be server_cookie=aa1d12268cb3ae50 in_seq=0 out_seq=0 2024-09-16T00:31:01.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.835+0000 7ff6f0ff9640 10 client.27395 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:01.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.835+0000 7ff6f0ff9640 1 -- 172.21.15.33:0/2650595002 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e40648ed00 con 0x55e40653ca00 2024-09-16T00:31:01.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.835+0000 7ff6f0ff9640 10 client.27395 handle_command_reply: tid=0 2024-09-16T00:31:01.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.835+0000 7ff711c61740 1 client.27395 shutdown 2024-09-16T00:31:01.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.836+0000 7ff711c61740 1 -- 172.21.15.33:0/2650595002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 msgr2=0x55e406535260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:01.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.836+0000 7ff711c61740 1 --2- 172.21.15.33:0/2650595002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406535260 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff7040ed690 tx=0x7ff7040dc040 comp rx=0 tx=0).stop 2024-09-16T00:31:01.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.836+0000 7ff711c61740 1 -- 172.21.15.33:0/2650595002 shutdown_connections 2024-09-16T00:31:01.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.836+0000 7ff711c61740 1 --2- 172.21.15.33:0/2650595002 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e40653ca00 0x55e4065a9bb0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.836+0000 7ff6f0ff9640 0 client.27395 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:01.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.836+0000 7ff711c61740 1 --2- 172.21.15.33:0/2650595002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e40652cd40 0x55e406526c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.836+0000 7ff711c61740 1 --2- 172.21.15.33:0/2650595002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e406522480 0x55e406526710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.836+0000 7ff711c61740 1 --2- 172.21.15.33:0/2650595002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e406521b60 0x55e406535260 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.836+0000 7ff711c61740 1 -- 172.21.15.33:0/2650595002 >> 172.21.15.33:0/2650595002 conn(0x55e40651e0f0 msgr2=0x55e406528cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:01.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.836+0000 7ff711c61740 1 -- 172.21.15.33:0/2650595002 shutdown_connections 2024-09-16T00:31:01.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.836+0000 7ff711c61740 1 -- 172.21.15.33:0/2650595002 wait complete. 2024-09-16T00:31:01.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.836+0000 7ff711c61740 20 client.27395 trim_cache size 0 max 16384 2024-09-16T00:31:01.868 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:31:01.868 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:31:01.869 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "c51dced2-1906-49b4-9e82-30a664a7239c", 2024-09-16T00:31:01.869 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:31:01.869 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:31:01.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.837+0000 7ff70f888640 1 -- 172.21.15.33:0/2911643547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff6f40a32c0 msgr2=0x7ff6f40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:01.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.837+0000 7ff70f888640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff6f40a32c0 0x7ff6f40a5780 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff7000107d0 tx=0x7ff700004a40 comp rx=0 tx=0).stop 2024-09-16T00:31:01.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.837+0000 7ff70f888640 1 -- 172.21.15.33:0/2911643547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff708102d30 msgr2=0x7ff708100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:01.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.837+0000 7ff70f888640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff708102d30 0x7ff708100990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff704009230 tx=0x7ff704004040 comp rx=0 tx=0).stop 2024-09-16T00:31:01.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.837+0000 7ff70f888640 1 -- 172.21.15.33:0/2911643547 shutdown_connections 2024-09-16T00:31:01.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.837+0000 7ff70f888640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff6f40a32c0 0x7ff6f40a5780 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.837+0000 7ff70f888640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7081082a0 0x7ff708101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.837+0000 7ff70f888640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff7081036e0 0x7ff708100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.837+0000 7ff70f888640 1 --2- 172.21.15.33:0/2911643547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff708102d30 0x7ff708100990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:01.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.837+0000 7ff70f888640 1 -- 172.21.15.33:0/2911643547 >> 172.21.15.33:0/2911643547 conn(0x7ff7080fc7d0 msgr2=0x7ff708105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:01.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.837+0000 7ff70f888640 1 -- 172.21.15.33:0/2911643547 shutdown_connections 2024-09-16T00:31:01.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:01.837+0000 7ff70f888640 1 -- 172.21.15.33:0/2911643547 wait complete. 2024-09-16T00:31:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:01 smithi083 ceph-mon[35242]: pgmap v766: 129 pgs: 129 active+clean; 264 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 2.5 MiB/s wr, 112 op/s 2024-09-16T00:31:01.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:01 smithi060 ceph-mon[35185]: pgmap v766: 129 pgs: 129 active+clean; 264 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 2.5 MiB/s wr, 112 op/s 2024-09-16T00:31:02.062 INFO:tasks.workunit.client.0.smithi033.stdout: 10 4572 53.37 MB/sec warmup 8 sec latency 128.943 ms 2024-09-16T00:31:02.571 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:31:02.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:31:02.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.664 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:02.664 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:02.664 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:02.664 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.664 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.664 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:02.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:02.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:02.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:02.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:02.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:02.666 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.666 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:02.666 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.666 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:02.666 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.864 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:02.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:31:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:31:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:03.062 INFO:tasks.workunit.client.0.smithi033.stdout: 10 4907 48.08 MB/sec warmup 9 sec latency 168.945 ms 2024-09-16T00:31:03.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.383+0000 7f40ace25640 1 -- 172.21.15.33:0/1219905115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a8106f10 msgr2=0x7f40a810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:03.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.383+0000 7f40ace25640 1 --2- 172.21.15.33:0/1219905115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a8106f10 0x7f40a810ea90 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f40a0009ca0 tx=0x7f40a002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.383+0000 7f40ace25640 1 -- 172.21.15.33:0/1219905115 shutdown_connections 2024-09-16T00:31:03.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.383+0000 7f40ace25640 1 --2- 172.21.15.33:0/1219905115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a8106f10 0x7f40a810ea90 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.383+0000 7f40ace25640 1 --2- 172.21.15.33:0/1219905115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40a8102790 0x7f40a81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.383+0000 7f40ace25640 1 --2- 172.21.15.33:0/1219905115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a8101de0 0x7f40a81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.383+0000 7f40ace25640 1 -- 172.21.15.33:0/1219905115 >> 172.21.15.33:0/1219905115 conn(0x7f40a80fd8c0 msgr2=0x7f40a80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:03.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.383+0000 7f40ace25640 1 -- 172.21.15.33:0/1219905115 shutdown_connections 2024-09-16T00:31:03.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.383+0000 7f40ace25640 1 -- 172.21.15.33:0/1219905115 wait complete. 2024-09-16T00:31:03.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.384+0000 7f40ace25640 1 Processor -- start 2024-09-16T00:31:03.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.384+0000 7f40ace25640 1 -- start start 2024-09-16T00:31:03.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.384+0000 7f40ace25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40a8101de0 0x7f40a8198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.384+0000 7f40ace25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a8102790 0x7f40a8198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.384+0000 7f40ace25640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a8106f10 0x7f40a819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.384+0000 7f40ace25640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f40a8111f70 con 0x7f40a8106f10 2024-09-16T00:31:03.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.384+0000 7f40ace25640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f40a8111df0 con 0x7f40a8102790 2024-09-16T00:31:03.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.384+0000 7f40ace25640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f40a81120f0 con 0x7f40a8101de0 2024-09-16T00:31:03.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.384+0000 7f40a6d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a8106f10 0x7f40a819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40a659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40a8101de0 0x7f40a8198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40a6d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a8106f10 0x7f40a819cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51792/0 (socket says 172.21.15.33:51792) 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40a6d9d640 1 -- 172.21.15.33:0/2577255473 learned_addr learned my addr 172.21.15.33:0/2577255473 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a8102790 0x7f40a8198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40a6d9d640 1 -- 172.21.15.33:0/2577255473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40a8101de0 msgr2=0x7f40a8198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40a6d9d640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40a8101de0 0x7f40a8198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40a6d9d640 1 -- 172.21.15.33:0/2577255473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a8102790 msgr2=0x7f40a8198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40a6d9d640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a8102790 0x7f40a8198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40a6d9d640 1 -- 172.21.15.33:0/2577255473 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a819d3c0 con 0x7f40a8106f10 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40a659c640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40a8101de0 0x7f40a8198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a8102790 0x7f40a8198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40a6d9d640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a8106f10 0x7f40a819cce0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f40a0031a20 tx=0x7f40a0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40977fe640 1 -- 172.21.15.33:0/2577255473 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a0003b70 con 0x7f40a8106f10 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40977fe640 1 -- 172.21.15.33:0/2577255473 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40a0003d10 con 0x7f40a8106f10 2024-09-16T00:31:03.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.385+0000 7f40ace25640 1 -- 172.21.15.33:0/2577255473 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a819d650 con 0x7f40a8106f10 2024-09-16T00:31:03.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.386+0000 7f40977fe640 1 -- 172.21.15.33:0/2577255473 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a002fc40 con 0x7f40a8106f10 2024-09-16T00:31:03.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.386+0000 7f40ace25640 1 -- 172.21.15.33:0/2577255473 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a81a4f90 con 0x7f40a8106f10 2024-09-16T00:31:03.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.388+0000 7f40977fe640 1 -- 172.21.15.33:0/2577255473 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f40a0004050 con 0x7f40a8106f10 2024-09-16T00:31:03.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.388+0000 7f40af1fe740 1 Processor -- start 2024-09-16T00:31:03.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.388+0000 7f40af1fe740 1 -- start start 2024-09-16T00:31:03.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40af1fe740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82e4a0 0x55895f82e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40af1fe740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 0x55895f839140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40af1fe740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f83ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40af1fe740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55895f7942e0 con 0x55895f839680 2024-09-16T00:31:03.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40af1fe740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55895f73bce0 con 0x55895f82edc0 2024-09-16T00:31:03.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40af1fe740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55895f7902a0 con 0x55895f82e4a0 2024-09-16T00:31:03.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f83ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 0x55895f839140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 0x55895f839140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:37974/0 (socket says 172.21.15.33:37974) 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f83ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51794/0 (socket says 172.21.15.33:51794) 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a659c640 1 -- 172.21.15.33:0/3629578236 learned_addr learned my addr 172.21.15.33:0/3629578236 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/3629578236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82e4a0 0x55895f82e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40977fe640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40900a32c0 0x7f40900a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a659c640 1 -- 172.21.15.33:0/3629578236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82e4a0 msgr2=0x55895f82e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a659c640 1 --2- 172.21.15.33:0/3629578236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82e4a0 0x55895f82e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a659c640 1 -- 172.21.15.33:0/3629578236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 msgr2=0x55895f83ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a659c640 1 --2- 172.21.15.33:0/3629578236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f83ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a659c640 1 -- 172.21.15.33:0/3629578236 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55895f720440 con 0x55895f82edc0 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/3629578236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f83ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:03.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a6d9d640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40900a32c0 0x7f40900a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40a659c640 1 --2- 172.21.15.33:0/3629578236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 0x55895f839140 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f40980115f0 tx=0x7f4098011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=871407224b72cbed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.389+0000 7f40977fe640 1 -- 172.21.15.33:0/2577255473 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f40a00f0e30 con 0x7f40a8106f10 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.390+0000 7f4094ff9640 1 -- 172.21.15.33:0/3629578236 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409801e070 con 0x55895f82edc0 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.390+0000 7f4094ff9640 1 -- 172.21.15.33:0/3629578236 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f409800fb20 con 0x55895f82edc0 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.390+0000 7f4094ff9640 1 -- 172.21.15.33:0/3629578236 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098019bf0 con 0x55895f82edc0 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.390+0000 7f40af1fe740 1 -- 172.21.15.33:0/3629578236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 msgr2=0x55895f839140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.390+0000 7f40a6d9d640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40900a32c0 0x7f40900a5780 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f40a00b9af0 tx=0x7f40a003d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.390+0000 7f40af1fe740 1 --2- 172.21.15.33:0/3629578236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 0x55895f839140 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f40980115f0 tx=0x7f4098011ab0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.390+0000 7f40af1fe740 1 -- 172.21.15.33:0/3629578236 shutdown_connections 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.390+0000 7f40af1fe740 1 --2- 172.21.15.33:0/3629578236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f83ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.390+0000 7f40af1fe740 1 --2- 172.21.15.33:0/3629578236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 0x55895f839140 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.391+0000 7f40af1fe740 1 --2- 172.21.15.33:0/3629578236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82e4a0 0x55895f82e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.391+0000 7f40af1fe740 1 -- 172.21.15.33:0/3629578236 >> 172.21.15.33:0/3629578236 conn(0x55895f82aa30 msgr2=0x55895f82ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:03.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.391+0000 7f40af1fe740 1 -- 172.21.15.33:0/3629578236 shutdown_connections 2024-09-16T00:31:03.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.391+0000 7f40af1fe740 1 -- 172.21.15.33:0/3629578236 wait complete. 2024-09-16T00:31:03.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.391+0000 7f40af1fe740 1 -- start start 2024-09-16T00:31:03.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.391+0000 7f40af1fe740 1 Processor -- start 2024-09-16T00:31:03.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.391+0000 7f40af1fe740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82e4a0 0x55895f838f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.391+0000 7f40af1fe740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f82edc0 0x55895f836210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.391+0000 7f40af1fe740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f839680 0x55895f836750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.391+0000 7f40af1fe740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55895f83c250 con 0x55895f82edc0 2024-09-16T00:31:03.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.391+0000 7f40af1fe740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55895f74f050 con 0x55895f82e4a0 2024-09-16T00:31:03.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.391+0000 7f40af1fe740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55895f83c3d0 con 0x55895f839680 2024-09-16T00:31:03.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40af1fe740 10 client.0 _ll_register_callbacks cb 0x55895f78f160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40a5d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82e4a0 0x55895f838f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40a5d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82e4a0 0x55895f838f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:37978/0 (socket says 172.21.15.33:37978) 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40a5d9b640 1 -- 172.21.15.33:0/2529831049 learned_addr learned my addr 172.21.15.33:0/2529831049 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40a659c640 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f82edc0 0x55895f836210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f839680 0x55895f836750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40a5d9b640 1 -- 172.21.15.33:0/2529831049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f839680 msgr2=0x55895f836750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f839680 0x55895f836750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40a5d9b640 1 -- 172.21.15.33:0/2529831049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f82edc0 msgr2=0x55895f836210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f82edc0 0x55895f836210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40a5d9b640 1 -- 172.21.15.33:0/2529831049 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55895f5053c0 con 0x55895f82e4a0 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82e4a0 0x55895f838f60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f409c011760 tx=0x7f409c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b69a0fbb9eb43ae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f408dffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:31:03.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f408dffb640 1 -- 172.21.15.33:0/2529831049 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c004240 con 0x55895f82e4a0 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40af1fe740 1 -- 172.21.15.33:0/2529831049 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f409800fb20 con 0x55895f82e4a0 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f408dffb640 1 -- 172.21.15.33:0/2529831049 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f409c0043e0 con 0x55895f82e4a0 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f408dffb640 1 -- 172.21.15.33:0/2529831049 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f409c005020 con 0x55895f82e4a0 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40af1fe740 10 client.27407 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.392+0000 7f40af1fe740 1 -- 172.21.15.33:0/2529831049 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55895f847ca0 con 0x55895f82e4a0 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.393+0000 7f408dffb640 1 -- 172.21.15.33:0/2529831049 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f409800fb20 con 0x55895f82e4a0 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.393+0000 7f408dffb640 1 -- 172.21.15.33:0/2529831049 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f409c00c6d0 con 0x55895f82e4a0 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.393+0000 7f40af1fe740 10 client.27407 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.393+0000 7f40af1fe740 10 client.27407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.393+0000 7f40af1fe740 20 client.27407 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.393+0000 7f40af1fe740 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55895f8490b0 0x55895f84b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.393+0000 7f40af1fe740 4 client.27407 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:31:03.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.393+0000 7f40af1fe740 1 -- 172.21.15.33:0/2529831049 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55895f7a1850 con 0x55895f8490b0 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.393+0000 7f40a659c640 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55895f8490b0 0x55895f84b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.393+0000 7f408dffb640 10 client.27407 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.393+0000 7f40a659c640 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55895f8490b0 0x55895f84b4c0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=383021ac5f25be4c server_cookie=e7c45ce77a51114a in_seq=0 out_seq=0 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f408dffb640 1 -- 172.21.15.33:0/2529831049 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55895f7a1850 con 0x55895f8490b0 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f408dffb640 10 client.27407 handle_command_reply: tid=0 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f40af1fe740 1 client.27407 shutdown 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f40af1fe740 1 -- 172.21.15.33:0/2529831049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82e4a0 msgr2=0x55895f838f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f40af1fe740 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82e4a0 0x55895f838f60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f409c011760 tx=0x7f409c004730 comp rx=0 tx=0).stop 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f40af1fe740 1 -- 172.21.15.33:0/2529831049 shutdown_connections 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f40af1fe740 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55895f8490b0 0x55895f84b4c0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f40af1fe740 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f839680 0x55895f836750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f40af1fe740 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f82edc0 0x55895f836210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f408dffb640 0 client.27407 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:03.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f40af1fe740 1 --2- 172.21.15.33:0/2529831049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82e4a0 0x55895f838f60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f40af1fe740 1 -- 172.21.15.33:0/2529831049 >> 172.21.15.33:0/2529831049 conn(0x55895f82aa30 msgr2=0x55895f830710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:03.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.394+0000 7f40af1fe740 1 -- 172.21.15.33:0/2529831049 shutdown_connections 2024-09-16T00:31:03.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.395+0000 7f40af1fe740 1 -- 172.21.15.33:0/2529831049 wait complete. 2024-09-16T00:31:03.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.395+0000 7f40af1fe740 20 client.27407 trim_cache size 0 max 16384 2024-09-16T00:31:03.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40af1fe740 1 Processor -- start 2024-09-16T00:31:03.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40af1fe740 1 -- start start 2024-09-16T00:31:03.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40af1fe740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82e4a0 0x55895f850d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40af1fe740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 0x55895f8512d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40af1fe740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f833470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40af1fe740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f40a81b19f0 con 0x55895f839680 2024-09-16T00:31:03.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40af1fe740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f40a81af260 con 0x55895f82edc0 2024-09-16T00:31:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40af1fe740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f40a81ad0b0 con 0x55895f82e4a0 2024-09-16T00:31:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40a5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f833470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40a659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 0x55895f8512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40a5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f833470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51804/0 (socket says 172.21.15.33:51804) 2024-09-16T00:31:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40a5d9b640 1 -- 172.21.15.33:0/1313890470 learned_addr learned my addr 172.21.15.33:0/1313890470 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.415+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/1313890470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82e4a0 0x55895f850d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.416+0000 7f40a5d9b640 1 -- 172.21.15.33:0/1313890470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82e4a0 msgr2=0x55895f850d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.416+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/1313890470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82e4a0 0x55895f850d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.416+0000 7f40a5d9b640 1 -- 172.21.15.33:0/1313890470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 msgr2=0x55895f8512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.416+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/1313890470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 0x55895f8512d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.416+0000 7f40a5d9b640 1 -- 172.21.15.33:0/1313890470 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55895f82a2a0 con 0x55895f839680 2024-09-16T00:31:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.416+0000 7f40a659c640 1 --2- 172.21.15.33:0/1313890470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 0x55895f8512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:03.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.416+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/1313890470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f833470 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f409c001eb0 tx=0x7f409c013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa01ebfdc0550401 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:03.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.416+0000 7f408ffff640 1 -- 172.21.15.33:0/1313890470 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c0051c0 con 0x55895f839680 2024-09-16T00:31:03.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.416+0000 7f408ffff640 1 -- 172.21.15.33:0/1313890470 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f409c005360 con 0x55895f839680 2024-09-16T00:31:03.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.416+0000 7f408ffff640 1 -- 172.21.15.33:0/1313890470 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c004c20 con 0x55895f839680 2024-09-16T00:31:03.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.417+0000 7f40af1fe740 1 -- 172.21.15.33:0/1313890470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 msgr2=0x55895f833470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:03.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.417+0000 7f40af1fe740 1 --2- 172.21.15.33:0/1313890470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f833470 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f409c001eb0 tx=0x7f409c013040 comp rx=0 tx=0).stop 2024-09-16T00:31:03.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.417+0000 7f40af1fe740 1 -- 172.21.15.33:0/1313890470 shutdown_connections 2024-09-16T00:31:03.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.417+0000 7f40af1fe740 1 --2- 172.21.15.33:0/1313890470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f833470 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.417+0000 7f40af1fe740 1 --2- 172.21.15.33:0/1313890470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82edc0 0x55895f8512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.417+0000 7f40af1fe740 1 --2- 172.21.15.33:0/1313890470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82e4a0 0x55895f850d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.417+0000 7f40af1fe740 1 -- 172.21.15.33:0/1313890470 >> 172.21.15.33:0/1313890470 conn(0x55895f82aa30 msgr2=0x55895f829b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:03.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.417+0000 7f40af1fe740 1 -- 172.21.15.33:0/1313890470 shutdown_connections 2024-09-16T00:31:03.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.417+0000 7f40af1fe740 1 -- 172.21.15.33:0/1313890470 wait complete. 2024-09-16T00:31:03.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40af1fe740 1 -- start start 2024-09-16T00:31:03.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40af1fe740 1 Processor -- start 2024-09-16T00:31:03.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40af1fe740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82e4a0 0x55895f841ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40af1fe740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82edc0 0x55895f8acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40af1fe740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f8ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40af1fe740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f40a81ad720 con 0x55895f839680 2024-09-16T00:31:03.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40af1fe740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f40a81aeb90 con 0x55895f82e4a0 2024-09-16T00:31:03.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40af1fe740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f40a81addf0 con 0x55895f82edc0 2024-09-16T00:31:03.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40af1fe740 10 client.0 _ll_register_callbacks cb 0x7f409c00ff70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:03.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40a659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82edc0 0x55895f8acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40a5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f8ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40a659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82edc0 0x55895f8acdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:45784/0 (socket says 172.21.15.33:45784) 2024-09-16T00:31:03.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40a5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f8ad4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51812/0 (socket says 172.21.15.33:51812) 2024-09-16T00:31:03.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.418+0000 7f40a659c640 1 -- 172.21.15.33:0/3795619747 learned_addr learned my addr 172.21.15.33:0/3795619747 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:03.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.419+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82e4a0 0x55895f841ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.419+0000 7f40a659c640 1 -- 172.21.15.33:0/3795619747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82e4a0 msgr2=0x55895f841ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:03.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.419+0000 7f40a659c640 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82e4a0 0x55895f841ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.419+0000 7f40a659c640 1 -- 172.21.15.33:0/3795619747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 msgr2=0x55895f8ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:03.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.419+0000 7f40a659c640 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f8ad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.419+0000 7f40a659c640 1 -- 172.21.15.33:0/3795619747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55895f83fa10 con 0x55895f82edc0 2024-09-16T00:31:03.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.419+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f8ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:03.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.419+0000 7f40a659c640 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82edc0 0x55895f8acdb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f40980115f0 tx=0x7f4098019860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31a341a6e3ca257d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:03.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.419+0000 7f408dffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:31:03.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.419+0000 7f408dffb640 1 -- 172.21.15.33:0/3795619747 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098028030 con 0x55895f82edc0 2024-09-16T00:31:03.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.419+0000 7f408dffb640 1 -- 172.21.15.33:0/3795619747 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4098016040 con 0x55895f82edc0 2024-09-16T00:31:03.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.419+0000 7f40af1fe740 1 -- 172.21.15.33:0/3795619747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f409c005360 con 0x55895f82edc0 2024-09-16T00:31:03.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.420+0000 7f408dffb640 1 -- 172.21.15.33:0/3795619747 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f409801e040 con 0x55895f82edc0 2024-09-16T00:31:03.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.420+0000 7f40af1fe740 10 client.27418 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:03.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.420+0000 7f40af1fe740 1 -- 172.21.15.33:0/3795619747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55895f8bcbc0 con 0x55895f82edc0 2024-09-16T00:31:03.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.420+0000 7f408dffb640 1 -- 172.21.15.33:0/3795619747 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f409c005360 con 0x55895f82edc0 2024-09-16T00:31:03.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.420+0000 7f408dffb640 1 -- 172.21.15.33:0/3795619747 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f4098019ac0 con 0x55895f82edc0 2024-09-16T00:31:03.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.420+0000 7f40af1fe740 10 client.27418 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:03.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.420+0000 7f40af1fe740 10 client.27418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:31:03.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.420+0000 7f40af1fe740 20 client.27418 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:03.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.420+0000 7f40af1fe740 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55895f8bd190 0x55895f8bf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:03.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.420+0000 7f40af1fe740 4 client.27418 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:31:03.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.420+0000 7f40af1fe740 1 -- 172.21.15.33:0/3795619747 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55895f79bd60 con 0x55895f8bd190 2024-09-16T00:31:03.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.420+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55895f8bd190 0x55895f8bf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f408dffb640 10 client.27418 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40a5d9b640 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55895f8bd190 0x55895f8bf5b0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbaa31f696c74836 server_cookie=206cda4c932ce42c in_seq=0 out_seq=0 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f408dffb640 1 -- 172.21.15.33:0/3795619747 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55895f79bd60 con 0x55895f8bd190 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f408dffb640 10 client.27418 handle_command_reply: tid=0 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40af1fe740 1 client.27418 shutdown 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40af1fe740 1 -- 172.21.15.33:0/3795619747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82edc0 msgr2=0x55895f8acdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40af1fe740 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82edc0 0x55895f8acdb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f40980115f0 tx=0x7f4098019860 comp rx=0 tx=0).stop 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40af1fe740 1 -- 172.21.15.33:0/3795619747 shutdown_connections 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40af1fe740 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55895f8bd190 0x55895f8bf5b0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40af1fe740 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895f839680 0x55895f8ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f408dffb640 0 client.27418 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40af1fe740 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55895f82edc0 0x55895f8acdb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40af1fe740 1 --2- 172.21.15.33:0/3795619747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55895f82e4a0 0x55895f841ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40af1fe740 1 -- 172.21.15.33:0/3795619747 >> 172.21.15.33:0/3795619747 conn(0x55895f82aa30 msgr2=0x55895f837a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:03.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40af1fe740 1 -- 172.21.15.33:0/3795619747 shutdown_connections 2024-09-16T00:31:03.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40af1fe740 1 -- 172.21.15.33:0/3795619747 wait complete. 2024-09-16T00:31:03.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.421+0000 7f40af1fe740 20 client.27418 trim_cache size 0 max 16384 2024-09-16T00:31:03.428 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:31:03.428 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:31:03.428 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:31:03.428 INFO:teuthology.orchestra.run.smithi033.stdout: "c51dced2-1906-49b4-9e82-30a664a7239c": { 2024-09-16T00:31:03.428 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:31:03.429 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "c51dced2-1906-49b4-9e82-30a664a7239c", 2024-09-16T00:31:03.429 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:31:03.429 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:03.429 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:03.429 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:31:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.422+0000 7f40ace25640 1 -- 172.21.15.33:0/2577255473 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40900a32c0 msgr2=0x7f40900a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.422+0000 7f40ace25640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40900a32c0 0x7f40900a5780 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f40a00b9af0 tx=0x7f40a003d040 comp rx=0 tx=0).stop 2024-09-16T00:31:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.422+0000 7f40ace25640 1 -- 172.21.15.33:0/2577255473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a8106f10 msgr2=0x7f40a819cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.422+0000 7f40ace25640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a8106f10 0x7f40a819cce0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f40a0031a20 tx=0x7f40a0031a50 comp rx=0 tx=0).stop 2024-09-16T00:31:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.422+0000 7f40ace25640 1 -- 172.21.15.33:0/2577255473 shutdown_connections 2024-09-16T00:31:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.422+0000 7f40ace25640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f40900a32c0 0x7f40900a5780 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.422+0000 7f40ace25640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a8106f10 0x7f40a819cce0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.422+0000 7f40ace25640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a8102790 0x7f40a8198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.422+0000 7f40ace25640 1 --2- 172.21.15.33:0/2577255473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40a8101de0 0x7f40a8198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.422+0000 7f40ace25640 1 -- 172.21.15.33:0/2577255473 >> 172.21.15.33:0/2577255473 conn(0x7f40a80fd8c0 msgr2=0x7f40a80feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.422+0000 7f40ace25640 1 -- 172.21.15.33:0/2577255473 shutdown_connections 2024-09-16T00:31:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:03.423+0000 7f40ace25640 1 -- 172.21.15.33:0/2577255473 wait complete. 2024-09-16T00:31:03.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:03 smithi083 ceph-mon[35242]: pgmap v767: 129 pgs: 129 active+clean; 271 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 3.9 MiB/s wr, 226 op/s 2024-09-16T00:31:03.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:03 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:03 smithi060 ceph-mon[35185]: pgmap v767: 129 pgs: 129 active+clean; 271 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 3.9 MiB/s wr, 226 op/s 2024-09-16T00:31:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:03 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:04.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:03 smithi033 ceph-mon[31243]: pgmap v767: 129 pgs: 129 active+clean; 271 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 3.9 MiB/s wr, 226 op/s 2024-09-16T00:31:04.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:03 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:04.062 INFO:tasks.workunit.client.0.smithi033.stdout: 10 5679 47.75 MB/sec warmup 10 sec latency 59.330 ms 2024-09-16T00:31:04.457 INFO:tasks.cephfs.filesystem:scrub status for tag:c51dced2-1906-49b4-9e82-30a664a7239c - {'path': '/', 'tag': 'c51dced2-1906-49b4-9e82-30a664a7239c', 'options': 'recursive,force'} 2024-09-16T00:31:05.063 INFO:tasks.workunit.client.0.smithi033.stdout: 10 6353 46.10 MB/sec warmup 11 sec latency 118.704 ms 2024-09-16T00:31:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:05 smithi083 ceph-mon[35242]: pgmap v768: 129 pgs: 129 active+clean; 274 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 4.2 MiB/s wr, 247 op/s 2024-09-16T00:31:05.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:05 smithi060 ceph-mon[35185]: pgmap v768: 129 pgs: 129 active+clean; 274 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 4.2 MiB/s wr, 247 op/s 2024-09-16T00:31:06.063 INFO:tasks.workunit.client.0.smithi033.stdout: 10 7068 45.44 MB/sec warmup 12 sec latency 65.480 ms 2024-09-16T00:31:06.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:05 smithi033 ceph-mon[31243]: pgmap v768: 129 pgs: 129 active+clean; 274 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 4.2 MiB/s wr, 247 op/s 2024-09-16T00:31:07.063 INFO:tasks.workunit.client.0.smithi033.stdout: 10 7863 46.27 MB/sec warmup 13 sec latency 76.148 ms 2024-09-16T00:31:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:07 smithi083 ceph-mon[35242]: pgmap v769: 129 pgs: 129 active+clean; 280 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 330 op/s 2024-09-16T00:31:07.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:07 smithi060 ceph-mon[35185]: pgmap v769: 129 pgs: 129 active+clean; 280 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 330 op/s 2024-09-16T00:31:08.063 INFO:tasks.workunit.client.0.smithi033.stdout: 10 8387 45.80 MB/sec warmup 14 sec latency 145.838 ms 2024-09-16T00:31:08.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:07 smithi033 ceph-mon[31243]: pgmap v769: 129 pgs: 129 active+clean; 280 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 330 op/s 2024-09-16T00:31:09.063 INFO:tasks.workunit.client.0.smithi033.stdout: 10 8698 43.37 MB/sec warmup 15 sec latency 61.697 ms 2024-09-16T00:31:09.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:09 smithi083 ceph-mon[35242]: pgmap v770: 129 pgs: 129 active+clean; 285 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 373 op/s 2024-09-16T00:31:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:09 smithi060 ceph-mon[35185]: pgmap v770: 129 pgs: 129 active+clean; 285 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 373 op/s 2024-09-16T00:31:10.063 INFO:tasks.workunit.client.0.smithi033.stdout: 10 9098 41.50 MB/sec warmup 16 sec latency 165.130 ms 2024-09-16T00:31:10.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:09 smithi033 ceph-mon[31243]: pgmap v770: 129 pgs: 129 active+clean; 285 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 373 op/s 2024-09-16T00:31:11.063 INFO:tasks.workunit.client.0.smithi033.stdout: 10 9391 39.94 MB/sec warmup 17 sec latency 526.768 ms 2024-09-16T00:31:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:11 smithi060 ceph-mon[35185]: pgmap v771: 129 pgs: 129 active+clean; 270 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 391 op/s 2024-09-16T00:31:12.063 INFO:tasks.workunit.client.0.smithi033.stdout: 10 9899 39.27 MB/sec warmup 18 sec latency 100.392 ms 2024-09-16T00:31:12.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:11 smithi033 ceph-mon[31243]: pgmap v771: 129 pgs: 129 active+clean; 270 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 391 op/s 2024-09-16T00:31:12.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:11 smithi083 ceph-mon[35242]: pgmap v771: 129 pgs: 129 active+clean; 270 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 391 op/s 2024-09-16T00:31:13.063 INFO:tasks.workunit.client.0.smithi033.stdout: 10 10614 38.72 MB/sec warmup 19 sec latency 63.692 ms 2024-09-16T00:31:13.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:13 smithi060 ceph-mon[35185]: pgmap v772: 129 pgs: 129 active+clean; 298 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 8.2 MiB/s wr, 449 op/s 2024-09-16T00:31:14.063 INFO:tasks.workunit.client.0.smithi033.stdout: 10 11483 39.16 MB/sec warmup 20 sec latency 48.615 ms 2024-09-16T00:31:14.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:13 smithi033 ceph-mon[31243]: pgmap v772: 129 pgs: 129 active+clean; 298 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 8.2 MiB/s wr, 449 op/s 2024-09-16T00:31:14.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:13 smithi083 ceph-mon[35242]: pgmap v772: 129 pgs: 129 active+clean; 298 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 8.2 MiB/s wr, 449 op/s 2024-09-16T00:31:15.063 INFO:tasks.workunit.client.0.smithi033.stdout: 10 12198 39.58 MB/sec warmup 21 sec latency 76.860 ms 2024-09-16T00:31:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:15 smithi060 ceph-mon[35185]: pgmap v773: 129 pgs: 129 active+clean; 383 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 14 MiB/s wr, 412 op/s 2024-09-16T00:31:16.063 INFO:tasks.workunit.client.0.smithi033.stdout: 10 12251 38.04 MB/sec warmup 22 sec latency 659.848 ms 2024-09-16T00:31:16.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:15 smithi033 ceph-mon[31243]: pgmap v773: 129 pgs: 129 active+clean; 383 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 14 MiB/s wr, 412 op/s 2024-09-16T00:31:16.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:15 smithi083 ceph-mon[35242]: pgmap v773: 129 pgs: 129 active+clean; 383 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 14 MiB/s wr, 412 op/s 2024-09-16T00:31:16.680 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 26.5 2024-09-16T00:31:16.680 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:31:16.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:16 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:31:17.065 INFO:tasks.workunit.client.0.smithi033.stdout: 10 12582 37.47 MB/sec warmup 23 sec latency 614.823 ms 2024-09-16T00:31:17.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:16 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:31:17.136 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:17.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:16 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:31:17.822 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:17 smithi033 ceph-mon[31243]: pgmap v774: 129 pgs: 129 active+clean; 447 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 20 MiB/s wr, 517 op/s 2024-09-16T00:31:17.822 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:17 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:31:17.823 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:17 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:31:17.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.863+0000 7f4cf6974640 1 -- 172.21.15.33:0/985899446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cf01012a0 msgr2=0x7f4cf0105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:17.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.863+0000 7f4cf6974640 1 --2- 172.21.15.33:0/985899446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cf01012a0 0x7f4cf0105770 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009ca0 tx=0x7f4ce802a960 comp rx=0 tx=0).stop 2024-09-16T00:31:17.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.865+0000 7f4cf6974640 1 -- 172.21.15.33:0/985899446 shutdown_connections 2024-09-16T00:31:17.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.865+0000 7f4cf6974640 1 --2- 172.21.15.33:0/985899446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cf0105d90 0x7f4cf0106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:17.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.865+0000 7f4cf6974640 1 --2- 172.21.15.33:0/985899446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cf01012a0 0x7f4cf0105770 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:17.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.865+0000 7f4cf6974640 1 --2- 172.21.15.33:0/985899446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf0100980 0x7f4cf0100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:17.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.865+0000 7f4cf6974640 1 -- 172.21.15.33:0/985899446 >> 172.21.15.33:0/985899446 conn(0x7f4cf00fc610 msgr2=0x7f4cf00fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:17.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.866+0000 7f4cf6974640 1 -- 172.21.15.33:0/985899446 shutdown_connections 2024-09-16T00:31:17.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.866+0000 7f4cf6974640 1 -- 172.21.15.33:0/985899446 wait complete. 2024-09-16T00:31:17.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.867+0000 7f4cf6974640 1 Processor -- start 2024-09-16T00:31:17.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.867+0000 7f4cf6974640 1 -- start start 2024-09-16T00:31:17.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.867+0000 7f4cf6974640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cf0100980 0x7f4cf00739e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:17.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.867+0000 7f4cf6974640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cf01012a0 0x7f4cf006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:17.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.867+0000 7f4cf6974640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf0105d90 0x7f4cf006cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:17.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.867+0000 7f4cf6974640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf0118410 con 0x7f4cf01012a0 2024-09-16T00:31:17.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.867+0000 7f4cf6974640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf0118290 con 0x7f4cf0105d90 2024-09-16T00:31:17.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.867+0000 7f4cf6974640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf0118590 con 0x7f4cf0100980 2024-09-16T00:31:17.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.868+0000 7f4cf5171640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cf01012a0 0x7f4cf006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:17.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.868+0000 7f4cf5171640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cf01012a0 0x7f4cf006ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41704/0 (socket says 172.21.15.33:41704) 2024-09-16T00:31:17.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.868+0000 7f4cf5171640 1 -- 172.21.15.33:0/2175621762 learned_addr learned my addr 172.21.15.33:0/2175621762 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:17.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.868+0000 7f4cf5972640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cf0100980 0x7f4cf00739e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:17.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.869+0000 7f4cf6173640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf0105d90 0x7f4cf006cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:17.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.869+0000 7f4cf5171640 1 -- 172.21.15.33:0/2175621762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cf0100980 msgr2=0x7f4cf00739e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:17.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.869+0000 7f4cf5171640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cf0100980 0x7f4cf00739e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:17.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.869+0000 7f4cf5171640 1 -- 172.21.15.33:0/2175621762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf0105d90 msgr2=0x7f4cf006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:17.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.869+0000 7f4cf5171640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf0105d90 0x7f4cf006cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:17.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.869+0000 7f4cf5171640 1 -- 172.21.15.33:0/2175621762 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf006d570 con 0x7f4cf01012a0 2024-09-16T00:31:17.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.869+0000 7f4cf5171640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cf01012a0 0x7f4cf006ca70 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8006010 tx=0x7f4ce802ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:17.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.870+0000 7f4ce6ffd640 1 -- 172.21.15.33:0/2175621762 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8034030 con 0x7f4cf01012a0 2024-09-16T00:31:17.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.870+0000 7f4cf6974640 1 -- 172.21.15.33:0/2175621762 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf006d800 con 0x7f4cf01012a0 2024-09-16T00:31:17.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.870+0000 7f4cf6974640 1 -- 172.21.15.33:0/2175621762 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf00717f0 con 0x7f4cf01012a0 2024-09-16T00:31:17.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.871+0000 7f4ce6ffd640 1 -- 172.21.15.33:0/2175621762 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ce8003d80 con 0x7f4cf01012a0 2024-09-16T00:31:17.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.871+0000 7f4ce6ffd640 1 -- 172.21.15.33:0/2175621762 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8004710 con 0x7f4cf01012a0 2024-09-16T00:31:17.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.875+0000 7f4cf6974640 1 -- 172.21.15.33:0/2175621762 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cd4005290 con 0x7f4cf01012a0 2024-09-16T00:31:17.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.876+0000 7f4ce6ffd640 1 -- 172.21.15.33:0/2175621762 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4ce8037020 con 0x7f4cf01012a0 2024-09-16T00:31:17.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.877+0000 7f4ce6ffd640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4cdc0a3300 0x7f4cdc0a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:17.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.877+0000 7f4ce6ffd640 1 -- 172.21.15.33:0/2175621762 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4ce80f0770 con 0x7f4cf01012a0 2024-09-16T00:31:17.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.877+0000 7f4cf5972640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4cdc0a3300 0x7f4cdc0a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:17.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.878+0000 7f4cf5972640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4cdc0a3300 0x7f4cdc0a57c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009aa0 tx=0x7f4cec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:17.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:17.885+0000 7f4ce6ffd640 1 -- 172.21.15.33:0/2175621762 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4ce80bd010 con 0x7f4cf01012a0 2024-09-16T00:31:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:17 smithi060 ceph-mon[35185]: pgmap v774: 129 pgs: 129 active+clean; 447 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 20 MiB/s wr, 517 op/s 2024-09-16T00:31:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:17 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:31:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:17 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:31:18.065 INFO:tasks.workunit.client.0.smithi033.stdout: 10 12860 36.66 MB/sec warmup 24 sec latency 231.572 ms 2024-09-16T00:31:18.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:17 smithi083 ceph-mon[35242]: pgmap v774: 129 pgs: 129 active+clean; 447 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 20 MiB/s wr, 517 op/s 2024-09-16T00:31:18.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:17 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:31:18.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:17 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:31:18.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.238+0000 7f4cf6974640 1 -- 172.21.15.33:0/2175621762 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cd4005580 con 0x7f4cf01012a0 2024-09-16T00:31:18.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.239+0000 7f4ce6ffd640 1 -- 172.21.15.33:0/2175621762 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11906 (secure 0 0 0) 0x7f4ce80bc910 con 0x7f4cf01012a0 2024-09-16T00:31:18.241 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:31:18.241 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 31 2024-09-16T00:31:18.242 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":31,"btime":"2024-09-16T00:30:57:641642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:30:57.027530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:31:18.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.242+0000 7f4cf6974640 1 -- 172.21.15.33:0/2175621762 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4cdc0a3300 msgr2=0x7f4cdc0a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:18.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.242+0000 7f4cf6974640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4cdc0a3300 0x7f4cdc0a57c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009aa0 tx=0x7f4cec005f90 comp rx=0 tx=0).stop 2024-09-16T00:31:18.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.242+0000 7f4cf6974640 1 -- 172.21.15.33:0/2175621762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cf01012a0 msgr2=0x7f4cf006ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:18.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.242+0000 7f4cf6974640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cf01012a0 0x7f4cf006ca70 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8006010 tx=0x7f4ce802ff90 comp rx=0 tx=0).stop 2024-09-16T00:31:18.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.242+0000 7f4cf6974640 1 -- 172.21.15.33:0/2175621762 shutdown_connections 2024-09-16T00:31:18.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.242+0000 7f4cf6974640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4cdc0a3300 0x7f4cdc0a57c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:18.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.242+0000 7f4cf6974640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf0105d90 0x7f4cf006cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:18.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.242+0000 7f4cf6974640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cf01012a0 0x7f4cf006ca70 secure :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8006010 tx=0x7f4ce802ff90 comp rx=0 tx=0).stop 2024-09-16T00:31:18.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.242+0000 7f4cf6974640 1 --2- 172.21.15.33:0/2175621762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cf0100980 0x7f4cf00739e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:18.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.242+0000 7f4cf6974640 1 -- 172.21.15.33:0/2175621762 >> 172.21.15.33:0/2175621762 conn(0x7f4cf00fc610 msgr2=0x7f4cf0103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:18.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.243+0000 7f4cf6974640 1 -- 172.21.15.33:0/2175621762 shutdown_connections 2024-09-16T00:31:18.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:18.243+0000 7f4cf6974640 1 -- 172.21.15.33:0/2175621762 wait complete. 2024-09-16T00:31:18.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:18 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2175621762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:31:18.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:18 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2175621762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:31:19.065 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13272 35.89 MB/sec warmup 25 sec latency 98.995 ms 2024-09-16T00:31:19.074 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 146.5 --await-for 45.0' 2024-09-16T00:31:19.075 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 146.5 --await-for 45.0 2024-09-16T00:31:19.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:18 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2175621762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:31:19.442 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:19.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:19 smithi033 ceph-mon[31243]: pgmap v775: 129 pgs: 129 active+clean; 457 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 20 MiB/s wr, 438 op/s 2024-09-16T00:31:19.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:19 smithi060 ceph-mon[35185]: pgmap v775: 129 pgs: 129 active+clean; 457 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 20 MiB/s wr, 438 op/s 2024-09-16T00:31:20.065 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13649 34.92 MB/sec warmup 26 sec latency 174.270 ms 2024-09-16T00:31:20.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:19 smithi083 ceph-mon[35242]: pgmap v775: 129 pgs: 129 active+clean; 457 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 20 MiB/s wr, 438 op/s 2024-09-16T00:31:20.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.298+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2481152360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2e0100e20 msgr2=0x7fc2e0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:20.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.298+0000 7fc2e72f2640 1 --2- 172.21.15.33:0/2481152360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2e0100e20 0x7fc2e0101200 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc009900 tx=0x7fc2dc02a480 comp rx=0 tx=0).stop 2024-09-16T00:31:20.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.298+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2481152360 shutdown_connections 2024-09-16T00:31:20.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.298+0000 7fc2e72f2640 1 --2- 172.21.15.33:0/2481152360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2e0102ef0 0x7fc2e010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.298+0000 7fc2e72f2640 1 --2- 172.21.15.33:0/2481152360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e0102570 0x7fc2e01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.298+0000 7fc2e72f2640 1 --2- 172.21.15.33:0/2481152360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2e0100e20 0x7fc2e0101200 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.298+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2481152360 >> 172.21.15.33:0/2481152360 conn(0x7fc2e00fc7d0 msgr2=0x7fc2e00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:20.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.298+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2481152360 shutdown_connections 2024-09-16T00:31:20.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.298+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2481152360 wait complete. 2024-09-16T00:31:20.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.299+0000 7fc2e72f2640 1 Processor -- start 2024-09-16T00:31:20.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.299+0000 7fc2e72f2640 1 -- start start 2024-09-16T00:31:20.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.299+0000 7fc2e72f2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2e0100e20 0x7fc2e01962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.299+0000 7fc2e72f2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e0102570 0x7fc2e01967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.299+0000 7fc2e72f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2e0102ef0 0x7fc2e019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.299+0000 7fc2e72f2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e010fe80 con 0x7fc2e0100e20 2024-09-16T00:31:20.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.299+0000 7fc2e72f2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e010fd00 con 0x7fc2e0102570 2024-09-16T00:31:20.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.299+0000 7fc2e72f2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e0110000 con 0x7fc2e0102ef0 2024-09-16T00:31:20.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.299+0000 7fc2e588f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2e0102ef0 0x7fc2e019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.300+0000 7fc2e588f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2e0102ef0 0x7fc2e019ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:48234/0 (socket says 172.21.15.33:48234) 2024-09-16T00:31:20.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.300+0000 7fc2e588f640 1 -- 172.21.15.33:0/2369812238 learned_addr learned my addr 172.21.15.33:0/2369812238 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:20.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.300+0000 7fc2e508e640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2e0100e20 0x7fc2e01962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.300+0000 7fc2e488d640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e0102570 0x7fc2e01967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.300+0000 7fc2e588f640 1 -- 172.21.15.33:0/2369812238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e0102570 msgr2=0x7fc2e01967f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:20.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.300+0000 7fc2e588f640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e0102570 0x7fc2e01967f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.300+0000 7fc2e588f640 1 -- 172.21.15.33:0/2369812238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2e0100e20 msgr2=0x7fc2e01962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:20.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.300+0000 7fc2e588f640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2e0100e20 0x7fc2e01962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.300+0000 7fc2e588f640 1 -- 172.21.15.33:0/2369812238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e019b260 con 0x7fc2e0102ef0 2024-09-16T00:31:20.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.300+0000 7fc2e508e640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2e0100e20 0x7fc2e01962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:20.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.301+0000 7fc2e588f640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2e0102ef0 0x7fc2e019ab80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800eaa0 tx=0x7fc2d800ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:20.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.301+0000 7fc2d67fc640 1 -- 172.21.15.33:0/2369812238 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d800cdb0 con 0x7fc2e0102ef0 2024-09-16T00:31:20.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.301+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2369812238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2e019b550 con 0x7fc2e0102ef0 2024-09-16T00:31:20.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.301+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2369812238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2e01a7240 con 0x7fc2e0102ef0 2024-09-16T00:31:20.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.301+0000 7fc2d67fc640 1 -- 172.21.15.33:0/2369812238 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2d8011950 con 0x7fc2e0102ef0 2024-09-16T00:31:20.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.301+0000 7fc2d67fc640 1 -- 172.21.15.33:0/2369812238 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d801f060 con 0x7fc2e0102ef0 2024-09-16T00:31:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.304+0000 7fc2e96cb740 1 Processor -- start 2024-09-16T00:31:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.304+0000 7fc2d67fc640 1 -- 172.21.15.33:0/2369812238 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc2d8011470 con 0x7fc2e0102ef0 2024-09-16T00:31:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e96cb740 1 -- start start 2024-09-16T00:31:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e96cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9a9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e96cb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9aa080 0x55770a9b4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e96cb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9b4940 0x55770a9b6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e96cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55770a90f2a0 con 0x55770a9a9760 2024-09-16T00:31:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e96cb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55770a8b6b10 con 0x55770a9b4940 2024-09-16T00:31:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e96cb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55770a90b260 con 0x55770a9aa080 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e488d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9a9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2d67fc640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc2cc0a3440 0x7fc2cc0a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e488d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9a9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33898/0 (socket says 172.21.15.33:33898) 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e488d640 1 -- 172.21.15.33:0/3582330189 learned_addr learned my addr 172.21.15.33:0/3582330189 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e508e640 1 --2- 172.21.15.33:0/3582330189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9aa080 0x55770a9b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e488d640 1 --2- 172.21.15.33:0/3582330189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9b4940 0x55770a9b6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e588f640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc2cc0a3440 0x7fc2cc0a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2d67fc640 1 -- 172.21.15.33:0/2369812238 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc2d8010840 con 0x7fc2e0102ef0 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e488d640 1 -- 172.21.15.33:0/3582330189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9aa080 msgr2=0x55770a9b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e488d640 1 --2- 172.21.15.33:0/3582330189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9aa080 0x55770a9b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e488d640 1 -- 172.21.15.33:0/3582330189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9b4940 msgr2=0x55770a9b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e488d640 1 --2- 172.21.15.33:0/3582330189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9b4940 0x55770a9b6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.305+0000 7fc2e488d640 1 -- 172.21.15.33:0/3582330189 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55770a680b90 con 0x55770a9a9760 2024-09-16T00:31:20.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.306+0000 7fc2e488d640 1 --2- 172.21.15.33:0/3582330189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9a9b40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0004650 tx=0x7fc2d0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad1bba833a8ec60 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:20.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.306+0000 7fc2e588f640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc2cc0a3440 0x7fc2cc0a5900 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc2d809bef0 tx=0x7fc2d80668f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:20.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.306+0000 7fc2cb7fe640 1 -- 172.21.15.33:0/3582330189 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0020070 con 0x55770a9a9760 2024-09-16T00:31:20.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.306+0000 7fc2cb7fe640 1 -- 172.21.15.33:0/3582330189 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2d00198f0 con 0x55770a9a9760 2024-09-16T00:31:20.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.306+0000 7fc2cb7fe640 1 -- 172.21.15.33:0/3582330189 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d001b580 con 0x55770a9a9760 2024-09-16T00:31:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.307+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3582330189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 msgr2=0x55770a9a9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.307+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3582330189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9a9b40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0004650 tx=0x7fc2d0005f90 comp rx=0 tx=0).stop 2024-09-16T00:31:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.307+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3582330189 shutdown_connections 2024-09-16T00:31:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.307+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3582330189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9b4940 0x55770a9b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.307+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3582330189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9aa080 0x55770a9b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.307+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3582330189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9a9b40 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.307+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3582330189 >> 172.21.15.33:0/3582330189 conn(0x55770a9a5cf0 msgr2=0x55770a9a60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.307+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3582330189 shutdown_connections 2024-09-16T00:31:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.307+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3582330189 wait complete. 2024-09-16T00:31:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.308+0000 7fc2e96cb740 1 -- start start 2024-09-16T00:31:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.308+0000 7fc2e96cb740 1 Processor -- start 2024-09-16T00:31:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.309+0000 7fc2e96cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.309+0000 7fc2e96cb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 0x55770a9b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.309+0000 7fc2e96cb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9b4940 0x55770a9b2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.309+0000 7fc2e96cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55770a9b7510 con 0x55770a9a9760 2024-09-16T00:31:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.309+0000 7fc2e96cb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55770a8976e0 con 0x55770a9aa080 2024-09-16T00:31:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.309+0000 7fc2e96cb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55770a9b7690 con 0x55770a9b4940 2024-09-16T00:31:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.309+0000 7fc2e96cb740 10 client.0 _ll_register_callbacks cb 0x55770a8bba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.309+0000 7fc2e508e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 0x55770a9b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.309+0000 7fc2e508e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 0x55770a9b1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:57816/0 (socket says 172.21.15.33:57816) 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.309+0000 7fc2e508e640 1 -- 172.21.15.33:0/3390133919 learned_addr learned my addr 172.21.15.33:0/3390133919 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.309+0000 7fc2e488d640 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.309+0000 7fc2e488d640 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9b4940 0x55770a9b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.310+0000 7fc2e488d640 1 -- 172.21.15.33:0/3390133919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9b4940 msgr2=0x55770a9b2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.310+0000 7fc2e488d640 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9b4940 0x55770a9b2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.310+0000 7fc2e488d640 1 -- 172.21.15.33:0/3390133919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 msgr2=0x55770a9b1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.310+0000 7fc2e488d640 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 0x55770a9b1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.310+0000 7fc2e488d640 1 -- 172.21.15.33:0/3390133919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55770a9c9220 con 0x55770a9a9760 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.310+0000 7fc2e508e640 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 0x55770a9b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.310+0000 7fc2e488d640 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9b17a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000bd10 tx=0x7fc2d000b2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28bfc6d06a2cd6de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.310+0000 7fc2c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:20.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.310+0000 7fc2c8ff9640 1 -- 172.21.15.33:0/3390133919 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0020040 con 0x55770a9a9760 2024-09-16T00:31:20.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.310+0000 7fc2c8ff9640 1 -- 172.21.15.33:0/3390133919 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc2d0005050 con 0x55770a9a9760 2024-09-16T00:31:20.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.310+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3390133919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2d00198f0 con 0x55770a9a9760 2024-09-16T00:31:20.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.310+0000 7fc2c8ff9640 1 -- 172.21.15.33:0/3390133919 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc2d0030050 con 0x55770a9a9760 2024-09-16T00:31:20.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.311+0000 7fc2e96cb740 10 client.18069 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:20.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.311+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3390133919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55770a9cbbb0 con 0x55770a9a9760 2024-09-16T00:31:20.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.311+0000 7fc2c8ff9640 1 -- 172.21.15.33:0/3390133919 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2d00198f0 con 0x55770a9a9760 2024-09-16T00:31:20.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.311+0000 7fc2c8ff9640 1 -- 172.21.15.33:0/3390133919 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc2d0026040 con 0x55770a9a9760 2024-09-16T00:31:20.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.311+0000 7fc2e96cb740 10 client.18069 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:20.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.311+0000 7fc2e96cb740 10 client.18069 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:31:20.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.311+0000 7fc2e96cb740 20 client.18069 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.311+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55770a9ccf00 0x55770a9cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.311+0000 7fc2e96cb740 4 client.18069 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:31:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.311+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3390133919 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55770a91c9d0 con 0x55770a9ccf00 2024-09-16T00:31:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.311+0000 7fc2e508e640 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55770a9ccf00 0x55770a9cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.312+0000 7fc2e508e640 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55770a9ccf00 0x55770a9cf2f0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=958eabb33d4253 server_cookie=14d7c8d4722521f5 in_seq=0 out_seq=0 2024-09-16T00:31:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.312+0000 7fc2c8ff9640 10 client.18069 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.312+0000 7fc2c8ff9640 1 -- 172.21.15.33:0/3390133919 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55770a91c9d0 con 0x55770a9ccf00 2024-09-16T00:31:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.312+0000 7fc2c8ff9640 10 client.18069 handle_command_reply: tid=0 2024-09-16T00:31:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.312+0000 7fc2e96cb740 1 client.18069 shutdown 2024-09-16T00:31:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.313+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3390133919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 msgr2=0x55770a9b17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.313+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9b17a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000bd10 tx=0x7fc2d000b2c0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.313+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3390133919 shutdown_connections 2024-09-16T00:31:20.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.313+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55770a9ccf00 0x55770a9cf2f0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.313+0000 7fc2c8ff9640 0 client.18069 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:20.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.313+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9b4940 0x55770a9b2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.313+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 0x55770a9b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.313+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3390133919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9b17a0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.313+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3390133919 >> 172.21.15.33:0/3390133919 conn(0x55770a9a5cf0 msgr2=0x55770a9aba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:20.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.313+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3390133919 shutdown_connections 2024-09-16T00:31:20.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.313+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3390133919 wait complete. 2024-09-16T00:31:20.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.313+0000 7fc2e96cb740 20 client.18069 trim_cache size 0 max 16384 2024-09-16T00:31:20.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e96cb740 1 Processor -- start 2024-09-16T00:31:20.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e96cb740 1 -- start start 2024-09-16T00:31:20.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e96cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e96cb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9aa080 0x55770a9d49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e96cb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9b4940 0x55770a9d4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e96cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c40128c0 con 0x55770a9a9760 2024-09-16T00:31:20.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e96cb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c4010130 con 0x55770a9b4940 2024-09-16T00:31:20.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e96cb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c400df80 con 0x55770a9aa080 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e488d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e508e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9aa080 0x55770a9d49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e488d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9b06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33924/0 (socket says 172.21.15.33:33924) 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e488d640 1 -- 172.21.15.33:0/3948464665 learned_addr learned my addr 172.21.15.33:0/3948464665 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.336+0000 7fc2e488d640 1 --2- 172.21.15.33:0/3948464665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9b4940 0x55770a9d4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.337+0000 7fc2e508e640 1 -- 172.21.15.33:0/3948464665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9b4940 msgr2=0x55770a9d4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.337+0000 7fc2e508e640 1 --2- 172.21.15.33:0/3948464665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9b4940 0x55770a9d4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.337+0000 7fc2e508e640 1 -- 172.21.15.33:0/3948464665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 msgr2=0x55770a9b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.337+0000 7fc2e508e640 1 --2- 172.21.15.33:0/3948464665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.337+0000 7fc2e508e640 1 -- 172.21.15.33:0/3948464665 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55770a9a5b20 con 0x55770a9aa080 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.337+0000 7fc2e508e640 1 --2- 172.21.15.33:0/3948464665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9aa080 0x55770a9d49a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x55770a9b1030 tx=0x7fc2dc031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ea9221ea95a7d89 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.337+0000 7fc2caffd640 1 -- 172.21.15.33:0/3948464665 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc035040 con 0x55770a9aa080 2024-09-16T00:31:20.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.337+0000 7fc2caffd640 1 -- 172.21.15.33:0/3948464665 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2dc04a070 con 0x55770a9aa080 2024-09-16T00:31:20.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.337+0000 7fc2caffd640 1 -- 172.21.15.33:0/3948464665 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0038e0 con 0x55770a9aa080 2024-09-16T00:31:20.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.338+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3948464665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9aa080 msgr2=0x55770a9d49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:20.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.338+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3948464665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9aa080 0x55770a9d49a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x55770a9b1030 tx=0x7fc2dc031400 comp rx=0 tx=0).stop 2024-09-16T00:31:20.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.338+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3948464665 shutdown_connections 2024-09-16T00:31:20.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.338+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3948464665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9b4940 0x55770a9d4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.338+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3948464665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9aa080 0x55770a9d49a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.338+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/3948464665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.338+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3948464665 >> 172.21.15.33:0/3948464665 conn(0x55770a9a5cf0 msgr2=0x55770a9b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:20.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.338+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3948464665 shutdown_connections 2024-09-16T00:31:20.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.338+0000 7fc2e96cb740 1 -- 172.21.15.33:0/3948464665 wait complete. 2024-09-16T00:31:20.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.338+0000 7fc2e96cb740 1 -- start start 2024-09-16T00:31:20.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.338+0000 7fc2e96cb740 1 Processor -- start 2024-09-16T00:31:20.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e96cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9c5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e96cb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 0x55770a9c5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e96cb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9b4940 0x55770a9c01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e96cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c400e5f0 con 0x55770a9a9760 2024-09-16T00:31:20.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e96cb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c400fa60 con 0x55770a9aa080 2024-09-16T00:31:20.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e96cb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c400ecc0 con 0x55770a9b4940 2024-09-16T00:31:20.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e96cb740 10 client.0 _ll_register_callbacks cb 0x7fc2d00174c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:20.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e488d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e508e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 0x55770a9c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e508e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 0x55770a9c5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:57828/0 (socket says 172.21.15.33:57828) 2024-09-16T00:31:20.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e508e640 1 -- 172.21.15.33:0/2231126223 learned_addr learned my addr 172.21.15.33:0/2231126223 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:20.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e488d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9c5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33926/0 (socket says 172.21.15.33:33926) 2024-09-16T00:31:20.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e488d640 1 --2- 172.21.15.33:0/2231126223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9b4940 0x55770a9c01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e508e640 1 -- 172.21.15.33:0/2231126223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9b4940 msgr2=0x55770a9c01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e508e640 1 --2- 172.21.15.33:0/2231126223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9b4940 0x55770a9c01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e508e640 1 -- 172.21.15.33:0/2231126223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 msgr2=0x55770a9c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e508e640 1 --2- 172.21.15.33:0/2231126223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9c5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.339+0000 7fc2e508e640 1 -- 172.21.15.33:0/2231126223 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55770a9c5dc0 con 0x55770a9aa080 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2e508e640 1 --2- 172.21.15.33:0/2231126223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 0x55770a9c5750 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc006010 tx=0x7fc2dc000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95aa4379e3c15454 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2c8ff9640 1 -- 172.21.15.33:0/2231126223 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc052030 con 0x55770a9aa080 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2e96cb740 1 -- 172.21.15.33:0/2231126223 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2dc04a070 con 0x55770a9aa080 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2c8ff9640 1 -- 172.21.15.33:0/2231126223 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc2dc0055b0 con 0x55770a9aa080 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2c8ff9640 1 -- 172.21.15.33:0/2231126223 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc2dc035050 con 0x55770a9aa080 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2e96cb740 10 client.27434 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2e96cb740 1 -- 172.21.15.33:0/2231126223 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55770aa37ee0 con 0x55770a9aa080 2024-09-16T00:31:20.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2c8ff9640 1 -- 172.21.15.33:0/2231126223 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2dc04a070 con 0x55770a9aa080 2024-09-16T00:31:20.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2c8ff9640 1 -- 172.21.15.33:0/2231126223 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc2dc003bc0 con 0x55770a9aa080 2024-09-16T00:31:20.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2e96cb740 10 client.27434 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:20.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2e96cb740 10 client.27434 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:31:20.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2e96cb740 20 client.27434 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:20.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/2231126223 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55770a9ccf00 0x55770aa3a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:20.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2e96cb740 4 client.27434 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.5, "await_for": 45.0}] 2024-09-16T00:31:20.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.340+0000 7fc2e96cb740 1 -- 172.21.15.33:0/2231126223 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.5, "await_for": 45.0}) -- 0x55770a916d20 con 0x55770a9ccf00 2024-09-16T00:31:20.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.341+0000 7fc2e488d640 1 --2- 172.21.15.33:0/2231126223 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55770a9ccf00 0x55770aa3a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:20.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.341+0000 7fc2c8ff9640 10 client.27434 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:20.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:20.341+0000 7fc2e488d640 1 --2- 172.21.15.33:0/2231126223 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55770a9ccf00 0x55770aa3a6f0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fea8d0e3ffc9aad server_cookie=6603fa5486414ad0 in_seq=0 out_seq=0 2024-09-16T00:31:21.065 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13831 33.89 MB/sec warmup 27 sec latency 729.789 ms 2024-09-16T00:31:21.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:20 smithi033 ceph-mon[31243]: pgmap v776: 129 pgs: 129 active+clean; 469 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 23 MiB/s wr, 516 op/s 2024-09-16T00:31:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.140+0000 7fc2c8ff9640 1 -- 172.21.15.33:0/2231126223 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55770a916d20 con 0x55770a9ccf00 2024-09-16T00:31:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.140+0000 7fc2c8ff9640 10 client.27434 handle_command_reply: tid=0 2024-09-16T00:31:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.140+0000 7fc2e96cb740 1 client.27434 shutdown 2024-09-16T00:31:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.140+0000 7fc2e96cb740 1 -- 172.21.15.33:0/2231126223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 msgr2=0x55770a9c5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.140+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/2231126223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 0x55770a9c5750 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc006010 tx=0x7fc2dc000bc0 comp rx=0 tx=0).stop 2024-09-16T00:31:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.140+0000 7fc2e96cb740 1 -- 172.21.15.33:0/2231126223 shutdown_connections 2024-09-16T00:31:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.140+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/2231126223 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55770a9ccf00 0x55770aa3a6f0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.140+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/2231126223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55770a9b4940 0x55770a9c01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:21.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.140+0000 7fc2c8ff9640 0 client.27434 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:21.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.140+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/2231126223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55770a9aa080 0x55770a9c5750 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:21.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.140+0000 7fc2e96cb740 1 --2- 172.21.15.33:0/2231126223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55770a9a9760 0x55770a9c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:21.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.140+0000 7fc2e96cb740 1 -- 172.21.15.33:0/2231126223 >> 172.21.15.33:0/2231126223 conn(0x55770a9a5cf0 msgr2=0x55770a9c8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:21.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.141+0000 7fc2e96cb740 1 -- 172.21.15.33:0/2231126223 shutdown_connections 2024-09-16T00:31:21.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.141+0000 7fc2e96cb740 1 -- 172.21.15.33:0/2231126223 wait complete. 2024-09-16T00:31:21.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.141+0000 7fc2e96cb740 20 client.27434 trim_cache size 0 max 16384 2024-09-16T00:31:21.143 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:31:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 31, 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 50, 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "8f702410": { 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 50, 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 146.5, 2024-09-16T00:31:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:31:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:31:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:31:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:31:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:31:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:31:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:21.145 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:31:21.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.141+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2369812238 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc2cc0a3440 msgr2=0x7fc2cc0a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:21.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.141+0000 7fc2e72f2640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc2cc0a3440 0x7fc2cc0a5900 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc2d809bef0 tx=0x7fc2d80668f0 comp rx=0 tx=0).stop 2024-09-16T00:31:21.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.141+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2369812238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2e0102ef0 msgr2=0x7fc2e019ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:21.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.141+0000 7fc2e72f2640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2e0102ef0 0x7fc2e019ab80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800eaa0 tx=0x7fc2d800ef60 comp rx=0 tx=0).stop 2024-09-16T00:31:21.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.142+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2369812238 shutdown_connections 2024-09-16T00:31:21.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.142+0000 7fc2e72f2640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc2cc0a3440 0x7fc2cc0a5900 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:21.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.142+0000 7fc2e72f2640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2e0102ef0 0x7fc2e019ab80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:21.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.142+0000 7fc2e72f2640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e0102570 0x7fc2e01967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:21.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.142+0000 7fc2e72f2640 1 --2- 172.21.15.33:0/2369812238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2e0100e20 0x7fc2e01962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:21.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.142+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2369812238 >> 172.21.15.33:0/2369812238 conn(0x7fc2e00fc7d0 msgr2=0x7fc2e00fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:21.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.142+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2369812238 shutdown_connections 2024-09-16T00:31:21.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:21.142+0000 7fc2e72f2640 1 -- 172.21.15.33:0/2369812238 wait complete. 2024-09-16T00:31:21.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:20 smithi083 ceph-mon[35242]: pgmap v776: 129 pgs: 129 active+clean; 469 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 23 MiB/s wr, 516 op/s 2024-09-16T00:31:21.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:20 smithi060 ceph-mon[35185]: pgmap v776: 129 pgs: 129 active+clean; 469 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 23 MiB/s wr, 516 op/s 2024-09-16T00:31:21.875 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '8f702410', quiesce took 5.2 seconds. Will release after: 21.3 2024-09-16T00:31:22.065 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 32.68 MB/sec warmup 28 sec latency 1729.822 ms 2024-09-16T00:31:23.065 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 31.56 MB/sec warmup 29 sec latency 2729.938 ms 2024-09-16T00:31:23.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:22 smithi060 ceph-mon[35185]: pgmap v777: 129 pgs: 129 active+clean; 475 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 26 MiB/s wr, 630 op/s 2024-09-16T00:31:23.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:22 smithi033 ceph-mon[31243]: pgmap v777: 129 pgs: 129 active+clean; 475 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 26 MiB/s wr, 630 op/s 2024-09-16T00:31:23.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:22 smithi083 ceph-mon[35242]: pgmap v777: 129 pgs: 129 active+clean; 475 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 26 MiB/s wr, 630 op/s 2024-09-16T00:31:24.065 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 30.50 MB/sec warmup 30 sec latency 3730.063 ms 2024-09-16T00:31:25.066 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 29.52 MB/sec warmup 31 sec latency 4730.187 ms 2024-09-16T00:31:25.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:24 smithi060 ceph-mon[35185]: pgmap v778: 129 pgs: 129 active+clean; 473 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 23 MiB/s wr, 609 op/s 2024-09-16T00:31:25.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:24 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:31:25.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:24 smithi033 ceph-mon[31243]: pgmap v778: 129 pgs: 129 active+clean; 473 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 23 MiB/s wr, 609 op/s 2024-09-16T00:31:25.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:24 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:31:25.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:24 smithi083 ceph-mon[35242]: pgmap v778: 129 pgs: 129 active+clean; 473 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 23 MiB/s wr, 609 op/s 2024-09-16T00:31:25.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:24 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:31:26.066 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 28.60 MB/sec warmup 32 sec latency 5730.307 ms 2024-09-16T00:31:26.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:31:26.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:31:26.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:31:26.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:31:26.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:31:26.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:31:26.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:31:26.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:31:26.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:31:27.066 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 27.73 MB/sec warmup 33 sec latency 6730.420 ms 2024-09-16T00:31:27.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:27 smithi033 ceph-mon[31243]: pgmap v779: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 15 MiB/s wr, 536 op/s 2024-09-16T00:31:27.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:27 smithi083 ceph-mon[35242]: pgmap v779: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 15 MiB/s wr, 536 op/s 2024-09-16T00:31:27.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:27 smithi060 ceph-mon[35185]: pgmap v779: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 15 MiB/s wr, 536 op/s 2024-09-16T00:31:28.066 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 26.92 MB/sec warmup 34 sec latency 7730.506 ms 2024-09-16T00:31:29.066 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 26.15 MB/sec warmup 35 sec latency 8730.600 ms 2024-09-16T00:31:29.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:29 smithi033 ceph-mon[31243]: pgmap v780: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.1 MiB/s wr, 394 op/s 2024-09-16T00:31:29.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:29 smithi083 ceph-mon[35242]: pgmap v780: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.1 MiB/s wr, 394 op/s 2024-09-16T00:31:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:29 smithi060 ceph-mon[35185]: pgmap v780: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.1 MiB/s wr, 394 op/s 2024-09-16T00:31:30.066 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 25.42 MB/sec warmup 36 sec latency 9730.715 ms 2024-09-16T00:31:31.066 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 24.73 MB/sec warmup 37 sec latency 10730.850 ms 2024-09-16T00:31:31.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:31 smithi033 ceph-mon[31243]: pgmap v781: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.0 MiB/s wr, 376 op/s 2024-09-16T00:31:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:31 smithi083 ceph-mon[35242]: pgmap v781: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.0 MiB/s wr, 376 op/s 2024-09-16T00:31:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:31 smithi060 ceph-mon[35185]: pgmap v781: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.0 MiB/s wr, 376 op/s 2024-09-16T00:31:32.066 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 24.08 MB/sec warmup 38 sec latency 11730.974 ms 2024-09-16T00:31:33.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 23.47 MB/sec warmup 39 sec latency 12731.112 ms 2024-09-16T00:31:33.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:33 smithi033 ceph-mon[31243]: pgmap v782: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 238 op/s 2024-09-16T00:31:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:33 smithi083 ceph-mon[35242]: pgmap v782: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 238 op/s 2024-09-16T00:31:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:33 smithi060 ceph-mon[35185]: pgmap v782: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 238 op/s 2024-09-16T00:31:34.066 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 22.88 MB/sec warmup 40 sec latency 13731.163 ms 2024-09-16T00:31:34.459 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:31:34.719 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:35.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 22.32 MB/sec warmup 41 sec latency 14731.210 ms 2024-09-16T00:31:35.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.204+0000 7f2bb6b84640 1 -- 172.21.15.33:0/2629656949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bb0100e20 msgr2=0x7f2bb0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:35.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.204+0000 7f2bb6b84640 1 --2- 172.21.15.33:0/2629656949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bb0100e20 0x7f2bb0101200 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009900 tx=0x7f2ba002a480 comp rx=0 tx=0).stop 2024-09-16T00:31:35.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.204+0000 7f2bb6b84640 1 -- 172.21.15.33:0/2629656949 shutdown_connections 2024-09-16T00:31:35.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.204+0000 7f2bb6b84640 1 --2- 172.21.15.33:0/2629656949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb0102ef0 0x7f2bb010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.204+0000 7f2bb6b84640 1 --2- 172.21.15.33:0/2629656949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2bb0102570 0x7f2bb01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.204+0000 7f2bb6b84640 1 --2- 172.21.15.33:0/2629656949 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bb0100e20 0x7f2bb0101200 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.204+0000 7f2bb6b84640 1 -- 172.21.15.33:0/2629656949 >> 172.21.15.33:0/2629656949 conn(0x7f2bb00fc7d0 msgr2=0x7f2bb00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:35.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.204+0000 7f2bb6b84640 1 -- 172.21.15.33:0/2629656949 shutdown_connections 2024-09-16T00:31:35.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.204+0000 7f2bb6b84640 1 -- 172.21.15.33:0/2629656949 wait complete. 2024-09-16T00:31:35.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.204+0000 7f2bb6b84640 1 Processor -- start 2024-09-16T00:31:35.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.204+0000 7f2bb6b84640 1 -- start start 2024-09-16T00:31:35.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb6b84640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb0100e20 0x7f2bb0197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb6b84640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2bb0102570 0x7f2bb01976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb6b84640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bb0102ef0 0x7f2bb0197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb6b84640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0110fd0 con 0x7f2bb0102ef0 2024-09-16T00:31:35.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb6b84640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0110e50 con 0x7f2bb0100e20 2024-09-16T00:31:35.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb6b84640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0111150 con 0x7f2bb0102570 2024-09-16T00:31:35.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb5121640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bb0102ef0 0x7f2bb0197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb5121640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bb0102ef0 0x7f2bb0197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52012/0 (socket says 172.21.15.33:52012) 2024-09-16T00:31:35.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb5121640 1 -- 172.21.15.33:0/1455824533 learned_addr learned my addr 172.21.15.33:0/1455824533 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:35.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb4920640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb0100e20 0x7f2bb0197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2baffff640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2bb0102570 0x7f2bb01976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb5121640 1 -- 172.21.15.33:0/1455824533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2bb0102570 msgr2=0x7f2bb01976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:35.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb5121640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2bb0102570 0x7f2bb01976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb5121640 1 -- 172.21.15.33:0/1455824533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb0100e20 msgr2=0x7f2bb0197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:35.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb5121640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb0100e20 0x7f2bb0197190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb5121640 1 -- 172.21.15.33:0/1455824533 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb01a6a00 con 0x7f2bb0102ef0 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.205+0000 7f2bb5121640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bb0102ef0 0x7f2bb0197c10 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009fd0 tx=0x7f2ba800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.206+0000 7f2badffb640 1 -- 172.21.15.33:0/1455824533 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8017070 con 0x7f2bb0102ef0 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.206+0000 7f2badffb640 1 -- 172.21.15.33:0/1455824533 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ba8003e50 con 0x7f2bb0102ef0 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.206+0000 7f2bb6b84640 1 -- 172.21.15.33:0/1455824533 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb01a6cf0 con 0x7f2bb0102ef0 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.206+0000 7f2badffb640 1 -- 172.21.15.33:0/1455824533 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80106c0 con 0x7f2bb0102ef0 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.206+0000 7f2bb6b84640 1 -- 172.21.15.33:0/1455824533 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb01a71e0 con 0x7f2bb0102ef0 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.208+0000 7f2badffb640 1 -- 172.21.15.33:0/1455824533 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2ba8003970 con 0x7f2bb0102ef0 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.208+0000 7f2bb8f5d740 1 Processor -- start 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.208+0000 7f2bb8f5d740 1 -- start start 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.208+0000 7f2badffb640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2b980a3240 0x7f2b980a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb8f5d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 0x558b1f370880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb8f5d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f370dc0 0x558b1f37b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb8f5d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f37b680 0x558b1f37da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb8f5d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558b1f2d62e0 con 0x558b1f37b680 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb8f5d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558b1f27dce0 con 0x558b1f3704a0 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb8f5d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558b1f2d22a0 con 0x558b1f370dc0 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2baffff640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2b980a3240 0x7f2b980a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2badffb640 1 -- 172.21.15.33:0/1455824533 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2ba80cfb20 con 0x7f2bb0102ef0 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb5121640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f37b680 0x558b1f37da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb5121640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f37b680 0x558b1f37da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52016/0 (socket says 172.21.15.33:52016) 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb5121640 1 -- 172.21.15.33:0/911108693 learned_addr learned my addr 172.21.15.33:0/911108693 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb4920640 1 --2- 172.21.15.33:0/911108693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 0x558b1f370880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2baffff640 1 --2- 172.21.15.33:0/911108693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f370dc0 0x558b1f37b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb5121640 1 -- 172.21.15.33:0/911108693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f370dc0 msgr2=0x558b1f37b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb5121640 1 --2- 172.21.15.33:0/911108693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f370dc0 0x558b1f37b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb5121640 1 -- 172.21.15.33:0/911108693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 msgr2=0x558b1f370880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb5121640 1 --2- 172.21.15.33:0/911108693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 0x558b1f370880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb5121640 1 -- 172.21.15.33:0/911108693 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b1f262440 con 0x558b1f37b680 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2baffff640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2b980a3240 0x7f2b980a5700 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4005e10 tx=0x7f2ba4012810 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.209+0000 7f2bb5121640 1 --2- 172.21.15.33:0/911108693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f37b680 0x558b1f37da70 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2ba809bbf0 tx=0x7f2ba809b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b78c935200e7cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.210+0000 7f2b96ffd640 1 -- 172.21.15.33:0/911108693 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8020070 con 0x558b1f37b680 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.210+0000 7f2b96ffd640 1 -- 172.21.15.33:0/911108693 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ba809bc60 con 0x558b1f37b680 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.210+0000 7f2b96ffd640 1 -- 172.21.15.33:0/911108693 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80982d0 con 0x558b1f37b680 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.210+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/911108693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f37b680 msgr2=0x558b1f37da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.210+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/911108693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f37b680 0x558b1f37da70 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2ba809bbf0 tx=0x7f2ba809b7a0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.210+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/911108693 shutdown_connections 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.210+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/911108693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f37b680 0x558b1f37da70 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.210+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/911108693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f370dc0 0x558b1f37b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.210+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/911108693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 0x558b1f370880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.210+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/911108693 >> 172.21.15.33:0/911108693 conn(0x558b1f36ca30 msgr2=0x558b1f36ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.210+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/911108693 shutdown_connections 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.210+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/911108693 wait complete. 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.211+0000 7f2bb8f5d740 1 -- start start 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.211+0000 7f2bb8f5d740 1 Processor -- start 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.211+0000 7f2bb8f5d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f3704a0 0x558b1f37af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.211+0000 7f2bb8f5d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f370dc0 0x558b1f378200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.211+0000 7f2bb8f5d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f37b680 0x558b1f378740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.211+0000 7f2bb8f5d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558b1f37e1f0 con 0x558b1f3704a0 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.211+0000 7f2bb8f5d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558b1f291050 con 0x558b1f370dc0 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.211+0000 7f2bb8f5d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558b1f37e370 con 0x558b1f37b680 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.211+0000 7f2bb8f5d740 10 client.0 _ll_register_callbacks cb 0x558b1f2d1160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:35.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.211+0000 7f2bb4920640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f3704a0 0x558b1f37af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2bb4920640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f3704a0 0x558b1f37af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52022/0 (socket says 172.21.15.33:52022) 2024-09-16T00:31:35.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2bb4920640 1 -- 172.21.15.33:0/38117772 learned_addr learned my addr 172.21.15.33:0/38117772 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:35.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2bb5121640 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f37b680 0x558b1f378740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2baffff640 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f370dc0 0x558b1f378200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2bb4920640 1 -- 172.21.15.33:0/38117772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f37b680 msgr2=0x558b1f378740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:35.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2bb4920640 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f37b680 0x558b1f378740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2bb4920640 1 -- 172.21.15.33:0/38117772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f370dc0 msgr2=0x558b1f378200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:35.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2bb4920640 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f370dc0 0x558b1f378200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2bb4920640 1 -- 172.21.15.33:0/38117772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b1f0473c0 con 0x558b1f3704a0 2024-09-16T00:31:35.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2baffff640 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f370dc0 0x558b1f378200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:31:35.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2bb4920640 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f3704a0 0x558b1f37af50 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009900 tx=0x7f2ba0005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a572cfd193ad194 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2b8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2b8ffff640 1 -- 172.21.15.33:0/38117772 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0035040 con 0x558b1f3704a0 2024-09-16T00:31:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2b8ffff640 1 -- 172.21.15.33:0/38117772 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2ba00046c0 con 0x558b1f3704a0 2024-09-16T00:31:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/38117772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ba809bc60 con 0x558b1f3704a0 2024-09-16T00:31:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2b8ffff640 1 -- 172.21.15.33:0/38117772 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2ba004f050 con 0x558b1f3704a0 2024-09-16T00:31:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2bb8f5d740 10 client.18087 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.212+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/38117772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558b1f389ba0 con 0x558b1f3704a0 2024-09-16T00:31:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.213+0000 7f2b8ffff640 1 -- 172.21.15.33:0/38117772 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ba809bc60 con 0x558b1f3704a0 2024-09-16T00:31:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.213+0000 7f2b8ffff640 1 -- 172.21.15.33:0/38117772 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2ba0003bc0 con 0x558b1f3704a0 2024-09-16T00:31:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.213+0000 7f2bb8f5d740 10 client.18087 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.213+0000 7f2bb8f5d740 10 client.18087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:31:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.213+0000 7f2bb8f5d740 20 client.18087 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:35.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.213+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558b1f38b060 0x558b1f38d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.213+0000 7f2bb8f5d740 4 client.18087 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:31:35.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.213+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/38117772 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b1f2e3850 con 0x558b1f38b060 2024-09-16T00:31:35.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.213+0000 7f2baffff640 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558b1f38b060 0x558b1f38d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.214+0000 7f2b8ffff640 10 client.18087 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:35.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.214+0000 7f2baffff640 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558b1f38b060 0x558b1f38d4e0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=144305b5a74bdd66 server_cookie=c6d806ec86006251 in_seq=0 out_seq=0 2024-09-16T00:31:35.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.214+0000 7f2b8ffff640 1 -- 172.21.15.33:0/38117772 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558b1f2e3850 con 0x558b1f38b060 2024-09-16T00:31:35.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.214+0000 7f2b8ffff640 10 client.18087 handle_command_reply: tid=0 2024-09-16T00:31:35.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.214+0000 7f2bb8f5d740 1 client.18087 shutdown 2024-09-16T00:31:35.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.215+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/38117772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f3704a0 msgr2=0x558b1f37af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:35.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.215+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f3704a0 0x558b1f37af50 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009900 tx=0x7f2ba0005040 comp rx=0 tx=0).stop 2024-09-16T00:31:35.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.215+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/38117772 shutdown_connections 2024-09-16T00:31:35.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.215+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558b1f38b060 0x558b1f38d4e0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.215+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f37b680 0x558b1f378740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.215+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f370dc0 0x558b1f378200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.215+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/38117772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f3704a0 0x558b1f37af50 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.215+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/38117772 >> 172.21.15.33:0/38117772 conn(0x558b1f36ca30 msgr2=0x558b1f372760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:35.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.215+0000 7f2b8ffff640 0 client.18087 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:35.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.215+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/38117772 shutdown_connections 2024-09-16T00:31:35.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.215+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/38117772 wait complete. 2024-09-16T00:31:35.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.215+0000 7f2bb8f5d740 20 client.18087 trim_cache size 0 max 16384 2024-09-16T00:31:35.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.232+0000 7f2bb8f5d740 1 Processor -- start 2024-09-16T00:31:35.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.232+0000 7f2bb8f5d740 1 -- start start 2024-09-16T00:31:35.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.232+0000 7f2bb8f5d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f36ca30 0x558b1f3874b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.232+0000 7f2bb8f5d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 0x558b1f392db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.232+0000 7f2bb8f5d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 0x558b1f3932f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.232+0000 7f2bb8f5d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01b3c10 con 0x558b1f370dc0 2024-09-16T00:31:35.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.232+0000 7f2bb8f5d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01b1480 con 0x558b1f3704a0 2024-09-16T00:31:35.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.232+0000 7f2bb8f5d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01af2d0 con 0x558b1f36ca30 2024-09-16T00:31:35.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2bb5121640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 0x558b1f3932f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2bb5121640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 0x558b1f3932f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52026/0 (socket says 172.21.15.33:52026) 2024-09-16T00:31:35.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2bb5121640 1 -- 172.21.15.33:0/3517173408 learned_addr learned my addr 172.21.15.33:0/3517173408 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:35.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2baffff640 1 --2- 172.21.15.33:0/3517173408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 0x558b1f392db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2bb4920640 1 --2- 172.21.15.33:0/3517173408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f36ca30 0x558b1f3874b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2bb5121640 1 -- 172.21.15.33:0/3517173408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f36ca30 msgr2=0x558b1f3874b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:35.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2bb5121640 1 --2- 172.21.15.33:0/3517173408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f36ca30 0x558b1f3874b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2bb5121640 1 -- 172.21.15.33:0/3517173408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 msgr2=0x558b1f392db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:35.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2bb5121640 1 --2- 172.21.15.33:0/3517173408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 0x558b1f392db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2bb5121640 1 -- 172.21.15.33:0/3517173408 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b1f37e130 con 0x558b1f370dc0 2024-09-16T00:31:35.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2bb4920640 1 --2- 172.21.15.33:0/3517173408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f36ca30 0x558b1f3874b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:31:35.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2bb5121640 1 --2- 172.21.15.33:0/3517173408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 0x558b1f3932f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80980c0 tx=0x7f2ba800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1225f412ab07e43 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:35.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2b967fc640 1 -- 172.21.15.33:0/3517173408 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8020040 con 0x558b1f370dc0 2024-09-16T00:31:35.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2b967fc640 1 -- 172.21.15.33:0/3517173408 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ba80a48d0 con 0x558b1f370dc0 2024-09-16T00:31:35.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.233+0000 7f2b967fc640 1 -- 172.21.15.33:0/3517173408 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80982d0 con 0x558b1f370dc0 2024-09-16T00:31:35.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.234+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3517173408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 msgr2=0x558b1f3932f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:35.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.234+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/3517173408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 0x558b1f3932f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80980c0 tx=0x7f2ba800bf10 comp rx=0 tx=0).stop 2024-09-16T00:31:35.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.234+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3517173408 shutdown_connections 2024-09-16T00:31:35.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.234+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/3517173408 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 0x558b1f3932f0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.234+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/3517173408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 0x558b1f392db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.234+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/3517173408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f36ca30 0x558b1f3874b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.234+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3517173408 >> 172.21.15.33:0/3517173408 conn(0x558b1f3788a0 msgr2=0x558b1f376ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:35.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.234+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3517173408 shutdown_connections 2024-09-16T00:31:35.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.234+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3517173408 wait complete. 2024-09-16T00:31:35.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.234+0000 7f2bb8f5d740 1 -- start start 2024-09-16T00:31:35.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.234+0000 7f2bb8f5d740 1 Processor -- start 2024-09-16T00:31:35.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb8f5d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f36ca30 0x558b1f3752d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb8f5d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 0x558b1f375810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb8f5d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 0x558b1f375d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb8f5d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01af940 con 0x558b1f370dc0 2024-09-16T00:31:35.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb8f5d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01b0db0 con 0x558b1f3704a0 2024-09-16T00:31:35.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb8f5d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01b0010 con 0x558b1f36ca30 2024-09-16T00:31:35.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb8f5d740 10 client.0 _ll_register_callbacks cb 0x7f2ba0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:35.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb5121640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 0x558b1f375d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb5121640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 0x558b1f375d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52028/0 (socket says 172.21.15.33:52028) 2024-09-16T00:31:35.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb5121640 1 -- 172.21.15.33:0/3903194177 learned_addr learned my addr 172.21.15.33:0/3903194177 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:35.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb4920640 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f36ca30 0x558b1f3752d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2baffff640 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 0x558b1f375810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb5121640 1 -- 172.21.15.33:0/3903194177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f36ca30 msgr2=0x558b1f3752d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:35.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb5121640 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f36ca30 0x558b1f3752d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb5121640 1 -- 172.21.15.33:0/3903194177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 msgr2=0x558b1f375810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:35.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb5121640 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 0x558b1f375810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb5121640 1 -- 172.21.15.33:0/3903194177 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b1f3f4e30 con 0x558b1f370dc0 2024-09-16T00:31:35.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2bb5121640 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 0x558b1f375d50 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800dfd0 tx=0x7f2ba8004890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e155c54407309c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:35.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2b8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:35.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2b8ffff640 1 -- 172.21.15.33:0/3903194177 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80b4e60 con 0x558b1f370dc0 2024-09-16T00:31:35.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.235+0000 7f2b8ffff640 1 -- 172.21.15.33:0/3903194177 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2ba80982d0 con 0x558b1f370dc0 2024-09-16T00:31:35.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3903194177 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ba80a48d0 con 0x558b1f370dc0 2024-09-16T00:31:35.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2b8ffff640 1 -- 172.21.15.33:0/3903194177 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2ba8020070 con 0x558b1f370dc0 2024-09-16T00:31:35.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2bb8f5d740 10 client.18093 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:35.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3903194177 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558b1f3f6090 con 0x558b1f370dc0 2024-09-16T00:31:35.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2b8ffff640 1 -- 172.21.15.33:0/3903194177 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ba80a48d0 con 0x558b1f370dc0 2024-09-16T00:31:35.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2b8ffff640 1 -- 172.21.15.33:0/3903194177 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2ba809b1d0 con 0x558b1f370dc0 2024-09-16T00:31:35.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2bb8f5d740 10 client.18093 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:35.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2bb8f5d740 10 client.18093 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:31:35.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2bb8f5d740 20 client.18093 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:35.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558b1f37b680 0x558b1f3f8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:35.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2bb8f5d740 4 client.18093 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:31:35.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3903194177 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558b1f2ddd60 con 0x558b1f37b680 2024-09-16T00:31:35.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.236+0000 7f2bb4920640 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558b1f37b680 0x558b1f3f8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:35.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2b8ffff640 10 client.18093 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:35.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb4920640 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558b1f37b680 0x558b1f3f8320 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52974804c996bbc3 server_cookie=8ea9ff44bac6352e in_seq=0 out_seq=0 2024-09-16T00:31:35.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2b8ffff640 1 -- 172.21.15.33:0/3903194177 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558b1f2ddd60 con 0x558b1f37b680 2024-09-16T00:31:35.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2b8ffff640 10 client.18093 handle_command_reply: tid=0 2024-09-16T00:31:35.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb8f5d740 1 client.18093 shutdown 2024-09-16T00:31:35.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3903194177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 msgr2=0x558b1f375d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:35.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 0x558b1f375d50 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800dfd0 tx=0x7f2ba8004890 comp rx=0 tx=0).stop 2024-09-16T00:31:35.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3903194177 shutdown_connections 2024-09-16T00:31:35.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558b1f37b680 0x558b1f3f8320 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b1f370dc0 0x558b1f375d50 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558b1f3704a0 0x558b1f375810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2b8ffff640 0 client.18093 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:35.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb8f5d740 1 --2- 172.21.15.33:0/3903194177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558b1f36ca30 0x558b1f3752d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3903194177 >> 172.21.15.33:0/3903194177 conn(0x558b1f3788a0 msgr2=0x558b1f372710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:35.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3903194177 shutdown_connections 2024-09-16T00:31:35.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb8f5d740 1 -- 172.21.15.33:0/3903194177 wait complete. 2024-09-16T00:31:35.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.237+0000 7f2bb8f5d740 20 client.18093 trim_cache size 0 max 16384 2024-09-16T00:31:35.265 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:31:35.266 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:31:35.266 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:31:35.266 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:31:35.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.238+0000 7f2bb6b84640 1 -- 172.21.15.33:0/1455824533 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2b980a3240 msgr2=0x7f2b980a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:35.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.238+0000 7f2bb6b84640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2b980a3240 0x7f2b980a5700 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4005e10 tx=0x7f2ba4012810 comp rx=0 tx=0).stop 2024-09-16T00:31:35.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.238+0000 7f2bb6b84640 1 -- 172.21.15.33:0/1455824533 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bb0102ef0 msgr2=0x7f2bb0197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:35.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.238+0000 7f2bb6b84640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bb0102ef0 0x7f2bb0197c10 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009fd0 tx=0x7f2ba800b260 comp rx=0 tx=0).stop 2024-09-16T00:31:35.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.238+0000 7f2bb6b84640 1 -- 172.21.15.33:0/1455824533 shutdown_connections 2024-09-16T00:31:35.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.238+0000 7f2bb6b84640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2b980a3240 0x7f2b980a5700 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.238+0000 7f2bb6b84640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bb0102ef0 0x7f2bb0197c10 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.238+0000 7f2bb6b84640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2bb0102570 0x7f2bb01976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.238+0000 7f2bb6b84640 1 --2- 172.21.15.33:0/1455824533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb0100e20 0x7f2bb0197190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:35.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.238+0000 7f2bb6b84640 1 -- 172.21.15.33:0/1455824533 >> 172.21.15.33:0/1455824533 conn(0x7f2bb00fc7d0 msgr2=0x7f2bb00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:35.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.238+0000 7f2bb6b84640 1 -- 172.21.15.33:0/1455824533 shutdown_connections 2024-09-16T00:31:35.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:35.238+0000 7f2bb6b84640 1 -- 172.21.15.33:0/1455824533 wait complete. 2024-09-16T00:31:35.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:35 smithi033 ceph-mon[31243]: pgmap v783: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 613 KiB/s wr, 102 op/s 2024-09-16T00:31:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:35 smithi083 ceph-mon[35242]: pgmap v783: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 613 KiB/s wr, 102 op/s 2024-09-16T00:31:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:35 smithi060 ceph-mon[35185]: pgmap v783: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 613 KiB/s wr, 102 op/s 2024-09-16T00:31:36.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 21.79 MB/sec warmup 42 sec latency 15731.257 ms 2024-09-16T00:31:36.095 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:31:36.096 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:31:36.356 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:36.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.887+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/2029888831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb8102d40 msgr2=0x7f8bb8103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:36.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.887+0000 7f8bbdc0a640 1 --2- 172.21.15.33:0/2029888831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb8102d40 0x7f8bb8103120 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009900 tx=0x7f8ba802a480 comp rx=0 tx=0).stop 2024-09-16T00:31:36.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.888+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/2029888831 shutdown_connections 2024-09-16T00:31:36.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.888+0000 7f8bbdc0a640 1 --2- 172.21.15.33:0/2029888831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bb810c170 0x7f8bb810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:36.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.888+0000 7f8bbdc0a640 1 --2- 172.21.15.33:0/2029888831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8103660 0x7f8bb810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:36.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.888+0000 7f8bbdc0a640 1 --2- 172.21.15.33:0/2029888831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb8102d40 0x7f8bb8103120 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.888+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/2029888831 >> 172.21.15.33:0/2029888831 conn(0x7f8bb80fc7a0 msgr2=0x7f8bb80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.888+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/2029888831 shutdown_connections 2024-09-16T00:31:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.888+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/2029888831 wait complete. 2024-09-16T00:31:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.888+0000 7f8bbdc0a640 1 Processor -- start 2024-09-16T00:31:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.888+0000 7f8bbdc0a640 1 -- start start 2024-09-16T00:31:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bbdc0a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb8102d40 0x7f8bb8198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bbdc0a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8103660 0x7f8bb8198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bbdc0a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bb810c170 0x7f8bb819ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bbdc0a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb8111f70 con 0x7f8bb8102d40 2024-09-16T00:31:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bbdc0a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb8111df0 con 0x7f8bb8103660 2024-09-16T00:31:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bbdc0a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb81120f0 con 0x7f8bb810c170 2024-09-16T00:31:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bb77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb8102d40 0x7f8bb8198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bb77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb8102d40 0x7f8bb8198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52044/0 (socket says 172.21.15.33:52044) 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bb77fe640 1 -- 172.21.15.33:0/3509465922 learned_addr learned my addr 172.21.15.33:0/3509465922 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bb6ffd640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8103660 0x7f8bb8198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bb7fff640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bb810c170 0x7f8bb819ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bb77fe640 1 -- 172.21.15.33:0/3509465922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bb810c170 msgr2=0x7f8bb819ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bb77fe640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bb810c170 0x7f8bb819ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bb77fe640 1 -- 172.21.15.33:0/3509465922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8103660 msgr2=0x7f8bb8198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bb77fe640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8103660 0x7f8bb8198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bb77fe640 1 -- 172.21.15.33:0/3509465922 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb819d3d0 con 0x7f8bb8102d40 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.889+0000 7f8bb77fe640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb8102d40 0x7f8bb8198420 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8006010 tx=0x7f8ba80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.890+0000 7f8bb4ff9640 1 -- 172.21.15.33:0/3509465922 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba803f030 con 0x7f8bb8102d40 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.890+0000 7f8bb4ff9640 1 -- 172.21.15.33:0/3509465922 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ba8003ce0 con 0x7f8bb8102d40 2024-09-16T00:31:36.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.890+0000 7f8bb4ff9640 1 -- 172.21.15.33:0/3509465922 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8035070 con 0x7f8bb8102d40 2024-09-16T00:31:36.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.890+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/3509465922 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb819d660 con 0x7f8bb8102d40 2024-09-16T00:31:36.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.890+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/3509465922 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb819daf0 con 0x7f8bb8102d40 2024-09-16T00:31:36.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.891+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/3509465922 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bb8103120 con 0x7f8bb8102d40 2024-09-16T00:31:36.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.892+0000 7f8bb4ff9640 1 -- 172.21.15.33:0/3509465922 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8ba8042030 con 0x7f8bb8102d40 2024-09-16T00:31:36.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.893+0000 7f8bb4ff9640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ba00a3240 0x7f8ba00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:36.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.893+0000 7f8bb4ff9640 1 -- 172.21.15.33:0/3509465922 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8ba80f05f0 con 0x7f8bb8102d40 2024-09-16T00:31:36.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.893+0000 7f8bb6ffd640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ba00a3240 0x7f8ba00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:36.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.893+0000 7f8bb6ffd640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ba00a3240 0x7f8ba00a5700 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8bb810d5b0 tx=0x7f8bac00b2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:36.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:36.896+0000 7f8bb4ff9640 1 -- 172.21.15.33:0/3509465922 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8ba80bcf50 con 0x7f8bb8102d40 2024-09-16T00:31:37.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 21.28 MB/sec warmup 43 sec latency 16731.297 ms 2024-09-16T00:31:37.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.111+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/3509465922 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bb802d7c0 con 0x7f8bb8102d40 2024-09-16T00:31:37.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.112+0000 7f8bb4ff9640 1 -- 172.21.15.33:0/3509465922 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11906 (secure 0 0 0) 0x7f8ba80bc850 con 0x7f8bb8102d40 2024-09-16T00:31:37.112 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:31:37.113 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":31,"btime":"2024-09-16T00:30:57:641642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:30:57.027530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:31:37.113 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 31 2024-09-16T00:31:37.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.115+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/3509465922 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ba00a3240 msgr2=0x7f8ba00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:37.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.115+0000 7f8bbdc0a640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ba00a3240 0x7f8ba00a5700 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8bb810d5b0 tx=0x7f8bac00b2e0 comp rx=0 tx=0).stop 2024-09-16T00:31:37.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.115+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/3509465922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb8102d40 msgr2=0x7f8bb8198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:37.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.115+0000 7f8bbdc0a640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb8102d40 0x7f8bb8198420 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8006010 tx=0x7f8ba80042f0 comp rx=0 tx=0).stop 2024-09-16T00:31:37.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.115+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/3509465922 shutdown_connections 2024-09-16T00:31:37.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.115+0000 7f8bbdc0a640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8ba00a3240 0x7f8ba00a5700 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:37.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.115+0000 7f8bbdc0a640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bb810c170 0x7f8bb819ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:37.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.115+0000 7f8bbdc0a640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8103660 0x7f8bb8198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:37.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.115+0000 7f8bbdc0a640 1 --2- 172.21.15.33:0/3509465922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb8102d40 0x7f8bb8198420 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:37.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.115+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/3509465922 >> 172.21.15.33:0/3509465922 conn(0x7f8bb80fc7a0 msgr2=0x7f8bb80fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:37.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.115+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/3509465922 shutdown_connections 2024-09-16T00:31:37.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:37.115+0000 7f8bbdc0a640 1 -- 172.21.15.33:0/3509465922 wait complete. 2024-09-16T00:31:37.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:37 smithi033 ceph-mon[31243]: pgmap v784: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 121 KiB/s wr, 27 op/s 2024-09-16T00:31:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:37 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3509465922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:31:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:37 smithi083 ceph-mon[35242]: pgmap v784: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 121 KiB/s wr, 27 op/s 2024-09-16T00:31:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:37 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3509465922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:31:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:37 smithi060 ceph-mon[35185]: pgmap v784: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 121 KiB/s wr, 27 op/s 2024-09-16T00:31:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:37 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3509465922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:31:38.064 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:31:38.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 20.80 MB/sec warmup 44 sec latency 17731.334 ms 2024-09-16T00:31:38.328 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:38.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.806+0000 7fc6e0269640 1 -- 172.21.15.33:0/3645692532 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6d81036c0 msgr2=0x7fc6d810bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:38.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.806+0000 7fc6e0269640 1 --2- 172.21.15.33:0/3645692532 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6d81036c0 0x7fc6d810bbb0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8009900 tx=0x7fc6c802a480 comp rx=0 tx=0).stop 2024-09-16T00:31:38.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.806+0000 7fc6e0269640 1 -- 172.21.15.33:0/3645692532 shutdown_connections 2024-09-16T00:31:38.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.806+0000 7fc6e0269640 1 --2- 172.21.15.33:0/3645692532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d810c180 0x7fc6d810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.806+0000 7fc6e0269640 1 --2- 172.21.15.33:0/3645692532 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6d81036c0 0x7fc6d810bbb0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.806+0000 7fc6e0269640 1 --2- 172.21.15.33:0/3645692532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d8102da0 0x7fc6d8103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.806+0000 7fc6e0269640 1 -- 172.21.15.33:0/3645692532 >> 172.21.15.33:0/3645692532 conn(0x7fc6d80fc7a0 msgr2=0x7fc6d80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:38.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.806+0000 7fc6e0269640 1 -- 172.21.15.33:0/3645692532 shutdown_connections 2024-09-16T00:31:38.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.806+0000 7fc6e0269640 1 -- 172.21.15.33:0/3645692532 wait complete. 2024-09-16T00:31:38.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.806+0000 7fc6e0269640 1 Processor -- start 2024-09-16T00:31:38.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6e0269640 1 -- start start 2024-09-16T00:31:38.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6e0269640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d8102da0 0x7fc6d8100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6e0269640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6d81036c0 0x7fc6d8100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6e0269640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d810c180 0x7fc6d8101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6e0269640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d8111f60 con 0x7fc6d81036c0 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6e0269640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d8111de0 con 0x7fc6d8102da0 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6e0269640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d81120e0 con 0x7fc6d810c180 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6dd804640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6d81036c0 0x7fc6d8100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6dd804640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6d81036c0 0x7fc6d8100f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52056/0 (socket says 172.21.15.33:52056) 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6dd804640 1 -- 172.21.15.33:0/2622171835 learned_addr learned my addr 172.21.15.33:0/2622171835 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6de005640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d8102da0 0x7fc6d8100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6de806640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d810c180 0x7fc6d8101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6dd804640 1 -- 172.21.15.33:0/2622171835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d810c180 msgr2=0x7fc6d8101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6dd804640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d810c180 0x7fc6d8101480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6dd804640 1 -- 172.21.15.33:0/2622171835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d8102da0 msgr2=0x7fc6d8100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6dd804640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d8102da0 0x7fc6d8100a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6dd804640 1 -- 172.21.15.33:0/2622171835 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d81a6940 con 0x7fc6d81036c0 2024-09-16T00:31:38.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6de806640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d810c180 0x7fc6d8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:38.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.807+0000 7fc6de005640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d8102da0 0x7fc6d8100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:38.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.808+0000 7fc6dd804640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6d81036c0 0x7fc6d8100f40 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8006010 tx=0x7fc6c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:38.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.808+0000 7fc6cf7fe640 1 -- 172.21.15.33:0/2622171835 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c803f030 con 0x7fc6d81036c0 2024-09-16T00:31:38.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.808+0000 7fc6cf7fe640 1 -- 172.21.15.33:0/2622171835 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6c8003ce0 con 0x7fc6d81036c0 2024-09-16T00:31:38.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.808+0000 7fc6e0269640 1 -- 172.21.15.33:0/2622171835 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d81a6bd0 con 0x7fc6d81036c0 2024-09-16T00:31:38.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.808+0000 7fc6cf7fe640 1 -- 172.21.15.33:0/2622171835 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8035070 con 0x7fc6d81036c0 2024-09-16T00:31:38.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.808+0000 7fc6e0269640 1 -- 172.21.15.33:0/2622171835 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d81a70b0 con 0x7fc6d81036c0 2024-09-16T00:31:38.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.810+0000 7fc6cf7fe640 1 -- 172.21.15.33:0/2622171835 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc6c8042030 con 0x7fc6d81036c0 2024-09-16T00:31:38.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.811+0000 7fc6cf7fe640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6c40a3240 0x7fc6c40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.811+0000 7fc6e2642740 1 Processor -- start 2024-09-16T00:31:38.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.811+0000 7fc6de005640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6c40a3240 0x7fc6c40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.811+0000 7fc6cf7fe640 1 -- 172.21.15.33:0/2622171835 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc6c80f0350 con 0x7fc6d81036c0 2024-09-16T00:31:38.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.811+0000 7fc6e2642740 1 -- start start 2024-09-16T00:31:38.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6e2642740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 0x55d872113090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6e2642740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d8721135d0 0x55d872113a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6e2642740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d87211e370 0x55d872120760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6e2642740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d8720792f0 con 0x55d872114b40 2024-09-16T00:31:38.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6e2642740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d872020cf0 con 0x55d87211e370 2024-09-16T00:31:38.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6e2642740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d8720752b0 con 0x55d8721135d0 2024-09-16T00:31:38.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6de005640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 0x55d872113090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6de005640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 0x55d872113090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52058/0 (socket says 172.21.15.33:52058) 2024-09-16T00:31:38.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6de005640 1 -- 172.21.15.33:0/644033790 learned_addr learned my addr 172.21.15.33:0/644033790 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:38.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6de005640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6c40a3240 0x7fc6c40a5700 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40058c0 tx=0x7fc6d4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6de806640 1 --2- 172.21.15.33:0/644033790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d87211e370 0x55d872120760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6dd804640 1 --2- 172.21.15.33:0/644033790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d8721135d0 0x55d872113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6de005640 1 -- 172.21.15.33:0/644033790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d8721135d0 msgr2=0x55d872113a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6de005640 1 --2- 172.21.15.33:0/644033790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d8721135d0 0x55d872113a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6de005640 1 -- 172.21.15.33:0/644033790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d87211e370 msgr2=0x55d872120760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6de005640 1 --2- 172.21.15.33:0/644033790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d87211e370 0x55d872120760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6de005640 1 -- 172.21.15.33:0/644033790 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8720053a0 con 0x55d872114b40 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6de806640 1 --2- 172.21.15.33:0/644033790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d87211e370 0x55d872120760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6de005640 1 --2- 172.21.15.33:0/644033790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 0x55d872113090 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400b450 tx=0x7fc6d4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6431b21064a82af server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6ccdea640 1 -- 172.21.15.33:0/644033790 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4022070 con 0x55d872114b40 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6ccdea640 1 -- 172.21.15.33:0/644033790 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6d401eb30 con 0x55d872114b40 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.812+0000 7fc6ccdea640 1 -- 172.21.15.33:0/644033790 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d401dae0 con 0x55d872114b40 2024-09-16T00:31:38.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.813+0000 7fc6e2642740 1 -- 172.21.15.33:0/644033790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 msgr2=0x55d872113090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.813+0000 7fc6e2642740 1 --2- 172.21.15.33:0/644033790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 0x55d872113090 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400b450 tx=0x7fc6d4012000 comp rx=0 tx=0).stop 2024-09-16T00:31:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.813+0000 7fc6e2642740 1 -- 172.21.15.33:0/644033790 shutdown_connections 2024-09-16T00:31:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.813+0000 7fc6e2642740 1 --2- 172.21.15.33:0/644033790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d87211e370 0x55d872120760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.813+0000 7fc6e2642740 1 --2- 172.21.15.33:0/644033790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d8721135d0 0x55d872113a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.813+0000 7fc6e2642740 1 --2- 172.21.15.33:0/644033790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 0x55d872113090 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.813+0000 7fc6e2642740 1 -- 172.21.15.33:0/644033790 >> 172.21.15.33:0/644033790 conn(0x55d8720dc310 msgr2=0x55d87210fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.813+0000 7fc6e2642740 1 -- 172.21.15.33:0/644033790 shutdown_connections 2024-09-16T00:31:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.813+0000 7fc6e2642740 1 -- 172.21.15.33:0/644033790 wait complete. 2024-09-16T00:31:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.814+0000 7fc6e2642740 1 -- start start 2024-09-16T00:31:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.814+0000 7fc6e2642740 1 Processor -- start 2024-09-16T00:31:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.814+0000 7fc6e2642740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8721135d0 0x55d87211dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.814+0000 7fc6e2642740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d872114b40 0x55d87211afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.814+0000 7fc6e2642740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d87211e370 0x55d87211b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.814+0000 7fc6e2642740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d872121070 con 0x55d8721135d0 2024-09-16T00:31:38.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.814+0000 7fc6e2642740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d872034060 con 0x55d87211e370 2024-09-16T00:31:38.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.814+0000 7fc6e2642740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d8721211f0 con 0x55d872114b40 2024-09-16T00:31:38.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.814+0000 7fc6e2642740 10 client.0 _ll_register_callbacks cb 0x55d871d878c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:38.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.814+0000 7fc6de005640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8721135d0 0x55d87211dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.814+0000 7fc6de005640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8721135d0 0x55d87211dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52064/0 (socket says 172.21.15.33:52064) 2024-09-16T00:31:38.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.814+0000 7fc6de005640 1 -- 172.21.15.33:0/3906195150 learned_addr learned my addr 172.21.15.33:0/3906195150 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:38.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6dd804640 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d872114b40 0x55d87211afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6de806640 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d87211e370 0x55d87211b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6de005640 1 -- 172.21.15.33:0/3906195150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d872114b40 msgr2=0x55d87211afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:38.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6de005640 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d872114b40 0x55d87211afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6de005640 1 -- 172.21.15.33:0/3906195150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d87211e370 msgr2=0x55d87211b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:38.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6de005640 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d87211e370 0x55d87211b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6de005640 1 -- 172.21.15.33:0/3906195150 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d871dea3c0 con 0x55d8721135d0 2024-09-16T00:31:38.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6de005640 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8721135d0 0x55d87211dd10 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400fbc0 tx=0x7fc6d401e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e051277ddeae5b01 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:38.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:38.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6c1ffb640 1 -- 172.21.15.33:0/3906195150 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4022050 con 0x55d8721135d0 2024-09-16T00:31:38.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6e2642740 1 -- 172.21.15.33:0/3906195150 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6d401eb30 con 0x55d8721135d0 2024-09-16T00:31:38.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6c1ffb640 1 -- 172.21.15.33:0/3906195150 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc6d4004020 con 0x55d8721135d0 2024-09-16T00:31:38.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6c1ffb640 1 -- 172.21.15.33:0/3906195150 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc6d403b050 con 0x55d8721135d0 2024-09-16T00:31:38.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6e2642740 10 client.18111 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:38.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6e2642740 1 -- 172.21.15.33:0/3906195150 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d87212cac0 con 0x55d8721135d0 2024-09-16T00:31:38.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.815+0000 7fc6c1ffb640 1 -- 172.21.15.33:0/3906195150 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6d401eb30 con 0x55d8721135d0 2024-09-16T00:31:38.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.816+0000 7fc6c1ffb640 1 -- 172.21.15.33:0/3906195150 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc6d401ece0 con 0x55d8721135d0 2024-09-16T00:31:38.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.816+0000 7fc6e2642740 10 client.18111 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:38.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.816+0000 7fc6e2642740 10 client.18111 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:31:38.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.816+0000 7fc6e2642740 20 client.18111 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:38.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.816+0000 7fc6e2642740 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d87212dec0 0x55d8721302b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.816+0000 7fc6e2642740 4 client.18111 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:31:38.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.816+0000 7fc6e2642740 1 -- 172.21.15.33:0/3906195150 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d872086860 con 0x55d87212dec0 2024-09-16T00:31:38.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.816+0000 7fc6dd804640 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d87212dec0 0x55d8721302b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.816+0000 7fc6dd804640 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d87212dec0 0x55d8721302b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=48648ab06009b960 server_cookie=fd1da7e5d7811041 in_seq=0 out_seq=0 2024-09-16T00:31:38.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.816+0000 7fc6c1ffb640 10 client.18111 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:31:38.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6c1ffb640 1 -- 172.21.15.33:0/3906195150 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d872086860 con 0x55d87212dec0 2024-09-16T00:31:38.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6c1ffb640 10 client.18111 handle_command_reply: tid=0 2024-09-16T00:31:38.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6e2642740 1 client.18111 shutdown 2024-09-16T00:31:38.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6e2642740 1 -- 172.21.15.33:0/3906195150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8721135d0 msgr2=0x55d87211dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:38.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6e2642740 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8721135d0 0x55d87211dd10 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400fbc0 tx=0x7fc6d401e680 comp rx=0 tx=0).stop 2024-09-16T00:31:38.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6e2642740 1 -- 172.21.15.33:0/3906195150 shutdown_connections 2024-09-16T00:31:38.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6e2642740 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d87212dec0 0x55d8721302b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6e2642740 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d87211e370 0x55d87211b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6e2642740 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d872114b40 0x55d87211afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6c1ffb640 0 client.18111 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:31:38.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6e2642740 1 --2- 172.21.15.33:0/3906195150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8721135d0 0x55d87211dd10 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6e2642740 1 -- 172.21.15.33:0/3906195150 >> 172.21.15.33:0/3906195150 conn(0x55d8720dc310 msgr2=0x55d872115960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.817+0000 7fc6e2642740 1 -- 172.21.15.33:0/3906195150 shutdown_connections 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.818+0000 7fc6e2642740 1 -- 172.21.15.33:0/3906195150 wait complete. 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.818+0000 7fc6e2642740 20 client.18111 trim_cache size 0 max 16384 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6e2642740 1 Processor -- start 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6e2642740 1 -- start start 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6e2642740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8721135d0 0x55d87212a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6e2642740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 0x55d872116120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6e2642740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d87211e370 0x55d872116660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6e2642740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d81b3bf0 con 0x55d872114b40 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6e2642740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d81b1460 con 0x55d8721135d0 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6e2642740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d81af2b0 con 0x55d87211e370 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6dd804640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 0x55d872116120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6dd804640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 0x55d872116120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52068/0 (socket says 172.21.15.33:52068) 2024-09-16T00:31:38.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6dd804640 1 -- 172.21.15.33:0/3351722904 learned_addr learned my addr 172.21.15.33:0/3351722904 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:38.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6dd804640 1 -- 172.21.15.33:0/3351722904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d87211e370 msgr2=0x55d872116660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:38.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6dd804640 1 --2- 172.21.15.33:0/3351722904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d87211e370 0x55d872116660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6dd804640 1 -- 172.21.15.33:0/3351722904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8721135d0 msgr2=0x55d87212a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:38.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6dd804640 1 --2- 172.21.15.33:0/3351722904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8721135d0 0x55d87212a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6dd804640 1 -- 172.21.15.33:0/3351722904 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d87211d930 con 0x55d872114b40 2024-09-16T00:31:38.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.831+0000 7fc6dd804640 1 --2- 172.21.15.33:0/3351722904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 0x55d872116120 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8000c00 tx=0x7fc6c80ed8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16737a5194f968bd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:38.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.832+0000 7fc6c3fff640 1 -- 172.21.15.33:0/3351722904 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80d9070 con 0x55d872114b40 2024-09-16T00:31:38.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.832+0000 7fc6c3fff640 1 -- 172.21.15.33:0/3351722904 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6c803b580 con 0x55d872114b40 2024-09-16T00:31:38.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.832+0000 7fc6c3fff640 1 -- 172.21.15.33:0/3351722904 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8049040 con 0x55d872114b40 2024-09-16T00:31:38.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.832+0000 7fc6e2642740 1 -- 172.21.15.33:0/3351722904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 msgr2=0x55d872116120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:38.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.832+0000 7fc6e2642740 1 --2- 172.21.15.33:0/3351722904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 0x55d872116120 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8000c00 tx=0x7fc6c80ed8a0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.832+0000 7fc6e2642740 1 -- 172.21.15.33:0/3351722904 shutdown_connections 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.832+0000 7fc6e2642740 1 --2- 172.21.15.33:0/3351722904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d87211e370 0x55d872116660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.832+0000 7fc6e2642740 1 --2- 172.21.15.33:0/3351722904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d872114b40 0x55d872116120 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.832+0000 7fc6e2642740 1 --2- 172.21.15.33:0/3351722904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8721135d0 0x55d87212a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.832+0000 7fc6e2642740 1 -- 172.21.15.33:0/3351722904 >> 172.21.15.33:0/3351722904 conn(0x55d8720dc310 msgr2=0x55d87210ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.832+0000 7fc6e2642740 1 -- 172.21.15.33:0/3351722904 shutdown_connections 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.832+0000 7fc6e2642740 1 -- 172.21.15.33:0/3351722904 wait complete. 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6e2642740 1 -- start start 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6e2642740 1 Processor -- start 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6e2642740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d8721135d0 0x55d872126480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6e2642740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d872114b40 0x55d8721269c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6e2642740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d87211e370 0x55d872123450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6e2642740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d81af920 con 0x55d87211e370 2024-09-16T00:31:38.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6e2642740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d81b0d90 con 0x55d872114b40 2024-09-16T00:31:38.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6e2642740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d81afff0 con 0x55d8721135d0 2024-09-16T00:31:38.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6e2642740 10 client.0 _ll_register_callbacks cb 0x7fc6d400bec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:38.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6de806640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d87211e370 0x55d872123450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6de806640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d87211e370 0x55d872123450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52078/0 (socket says 172.21.15.33:52078) 2024-09-16T00:31:38.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6de806640 1 -- 172.21.15.33:0/2297031813 learned_addr learned my addr 172.21.15.33:0/2297031813 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:38.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6de005640 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d8721135d0 0x55d872126480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.833+0000 7fc6dd804640 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d872114b40 0x55d8721269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6de806640 1 -- 172.21.15.33:0/2297031813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d8721135d0 msgr2=0x55d872126480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:38.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6de806640 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d8721135d0 0x55d872126480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6de806640 1 -- 172.21.15.33:0/2297031813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d872114b40 msgr2=0x55d8721269c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:38.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6de806640 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d872114b40 0x55d8721269c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6de806640 1 -- 172.21.15.33:0/2297031813 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d872123a10 con 0x55d87211e370 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6de806640 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d87211e370 0x55d872123450 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00134b0 tx=0x7fc6d0013970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42f65ecd6de8386 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6c1ffb640 1 -- 172.21.15.33:0/2297031813 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0004d20 con 0x55d87211e370 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6c1ffb640 1 -- 172.21.15.33:0/2297031813 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc6d0011790 con 0x55d87211e370 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6e2642740 1 -- 172.21.15.33:0/2297031813 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d872080d70 con 0x55d87211e370 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6c1ffb640 1 -- 172.21.15.33:0/2297031813 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc6d0011990 con 0x55d87211e370 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6e2642740 10 client.18117 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6e2642740 1 -- 172.21.15.33:0/2297031813 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d872199210 con 0x55d87211e370 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.834+0000 7fc6c1ffb640 1 -- 172.21.15.33:0/2297031813 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d872080d70 con 0x55d87211e370 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.835+0000 7fc6c1ffb640 1 -- 172.21.15.33:0/2297031813 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc6d001c3a0 con 0x55d87211e370 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.835+0000 7fc6e2642740 10 client.18117 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.835+0000 7fc6e2642740 10 client.18117 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:31:38.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.835+0000 7fc6e2642740 20 client.18117 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.835+0000 7fc6e2642740 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d872199a40 0x55d87219be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.835+0000 7fc6e2642740 4 client.18117 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:31:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.835+0000 7fc6e2642740 1 -- 172.21.15.33:0/2297031813 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d871ffd170 con 0x55d872199a40 2024-09-16T00:31:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.835+0000 7fc6de005640 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d872199a40 0x55d87219be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.835+0000 7fc6de005640 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d872199a40 0x55d87219be60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d3c3033201056f8 server_cookie=dcac312ce6f3b0da in_seq=0 out_seq=0 2024-09-16T00:31:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.835+0000 7fc6c1ffb640 10 client.18117 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:31:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6c1ffb640 1 -- 172.21.15.33:0/2297031813 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d871ffd170 con 0x55d872199a40 2024-09-16T00:31:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6c1ffb640 10 client.18117 handle_command_reply: tid=0 2024-09-16T00:31:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e2642740 1 client.18117 shutdown 2024-09-16T00:31:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e2642740 1 -- 172.21.15.33:0/2297031813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d87211e370 msgr2=0x55d872123450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e2642740 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d87211e370 0x55d872123450 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00134b0 tx=0x7fc6d0013970 comp rx=0 tx=0).stop 2024-09-16T00:31:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e2642740 1 -- 172.21.15.33:0/2297031813 shutdown_connections 2024-09-16T00:31:38.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e2642740 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55d872199a40 0x55d87219be60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e2642740 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d87211e370 0x55d872123450 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e2642740 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d872114b40 0x55d8721269c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e2642740 1 --2- 172.21.15.33:0/2297031813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d8721135d0 0x55d872126480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e2642740 1 -- 172.21.15.33:0/2297031813 >> 172.21.15.33:0/2297031813 conn(0x55d8720dc310 msgr2=0x55d872129300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:38.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6c1ffb640 0 client.18117 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:31:38.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e2642740 1 -- 172.21.15.33:0/2297031813 shutdown_connections 2024-09-16T00:31:38.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e2642740 1 -- 172.21.15.33:0/2297031813 wait complete. 2024-09-16T00:31:38.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e2642740 20 client.18117 trim_cache size 0 max 16384 2024-09-16T00:31:38.853 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:31:38.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e0269640 1 -- 172.21.15.33:0/2622171835 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6c40a3240 msgr2=0x7fc6c40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:38.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e0269640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6c40a3240 0x7fc6c40a5700 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40058c0 tx=0x7fc6d4005850 comp rx=0 tx=0).stop 2024-09-16T00:31:38.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e0269640 1 -- 172.21.15.33:0/2622171835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6d81036c0 msgr2=0x7fc6d8100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:38.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.836+0000 7fc6e0269640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6d81036c0 0x7fc6d8100f40 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8006010 tx=0x7fc6c80042f0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.837+0000 7fc6e0269640 1 -- 172.21.15.33:0/2622171835 shutdown_connections 2024-09-16T00:31:38.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.837+0000 7fc6e0269640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6c40a3240 0x7fc6c40a5700 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.837+0000 7fc6e0269640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d810c180 0x7fc6d8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.837+0000 7fc6e0269640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc6d81036c0 0x7fc6d8100f40 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.837+0000 7fc6e0269640 1 --2- 172.21.15.33:0/2622171835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d8102da0 0x7fc6d8100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:38.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.837+0000 7fc6e0269640 1 -- 172.21.15.33:0/2622171835 >> 172.21.15.33:0/2622171835 conn(0x7fc6d80fc7a0 msgr2=0x7fc6d80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:38.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.837+0000 7fc6e0269640 1 -- 172.21.15.33:0/2622171835 shutdown_connections 2024-09-16T00:31:38.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:38.837+0000 7fc6e0269640 1 -- 172.21.15.33:0/2622171835 wait complete. 2024-09-16T00:31:39.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 20.34 MB/sec warmup 45 sec latency 18731.433 ms 2024-09-16T00:31:39.661 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:31:39.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:39 smithi033 ceph-mon[31243]: pgmap v785: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-09-16T00:31:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:39 smithi083 ceph-mon[35242]: pgmap v785: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-09-16T00:31:39.919 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:39 smithi060 ceph-mon[35185]: pgmap v785: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-09-16T00:31:40.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 19.89 MB/sec warmup 46 sec latency 19731.479 ms 2024-09-16T00:31:40.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.438+0000 7f6222d82640 1 -- 172.21.15.33:0/65991861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f621c106a80 msgr2=0x7f621c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:40.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.438+0000 7f6222d82640 1 --2- 172.21.15.33:0/65991861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f621c106a80 0x7f621c106e60 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f620c009930 tx=0x7f620c02a420 comp rx=0 tx=0).stop 2024-09-16T00:31:40.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.438+0000 7f6222d82640 1 -- 172.21.15.33:0/65991861 shutdown_connections 2024-09-16T00:31:40.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.438+0000 7f6222d82640 1 --2- 172.21.15.33:0/65991861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f621c101420 0x7f621c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.438+0000 7f6222d82640 1 --2- 172.21.15.33:0/65991861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f621c100a80 0x7f621c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.438+0000 7f6222d82640 1 --2- 172.21.15.33:0/65991861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f621c106a80 0x7f621c106e60 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.438+0000 7f6222d82640 1 -- 172.21.15.33:0/65991861 >> 172.21.15.33:0/65991861 conn(0x7f621c0fc7a0 msgr2=0x7f621c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:40.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.439+0000 7f6222d82640 1 -- 172.21.15.33:0/65991861 shutdown_connections 2024-09-16T00:31:40.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.439+0000 7f6222d82640 1 -- 172.21.15.33:0/65991861 wait complete. 2024-09-16T00:31:40.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.439+0000 7f6222d82640 1 Processor -- start 2024-09-16T00:31:40.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.439+0000 7f6222d82640 1 -- start start 2024-09-16T00:31:40.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.439+0000 7f6222d82640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f621c100a80 0x7f621c102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f6222d82640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f621c101420 0x7f621c102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f6222d82640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f621c106a80 0x7f621c1030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f6222d82640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f621c110ea0 con 0x7f621c106a80 2024-09-16T00:31:40.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f6222d82640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f621c110d20 con 0x7f621c101420 2024-09-16T00:31:40.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f6222d82640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f621c111020 con 0x7f621c100a80 2024-09-16T00:31:40.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f622131f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f621c106a80 0x7f621c1030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f622131f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f621c106a80 0x7f621c1030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37746/0 (socket says 172.21.15.33:37746) 2024-09-16T00:31:40.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f622131f640 1 -- 172.21.15.33:0/3420312133 learned_addr learned my addr 172.21.15.33:0/3420312133 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:40.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f6220b1e640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f621c100a80 0x7f621c102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f621bfff640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f621c101420 0x7f621c102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f622131f640 1 -- 172.21.15.33:0/3420312133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f621c100a80 msgr2=0x7f621c102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:40.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f622131f640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f621c100a80 0x7f621c102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f622131f640 1 -- 172.21.15.33:0/3420312133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f621c101420 msgr2=0x7f621c102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:40.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f622131f640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f621c101420 0x7f621c102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f622131f640 1 -- 172.21.15.33:0/3420312133 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621c1a6980 con 0x7f621c106a80 2024-09-16T00:31:40.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f622131f640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f621c106a80 0x7f621c1030d0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f621400b970 tx=0x7f621400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:40.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f6219ffb640 1 -- 172.21.15.33:0/3420312133 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214019070 con 0x7f621c106a80 2024-09-16T00:31:40.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f6219ffb640 1 -- 172.21.15.33:0/3420312133 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f621400cb40 con 0x7f621c106a80 2024-09-16T00:31:40.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f6219ffb640 1 -- 172.21.15.33:0/3420312133 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214007650 con 0x7f621c106a80 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.440+0000 7f6222d82640 1 -- 172.21.15.33:0/3420312133 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f621c1a6c70 con 0x7f621c106a80 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.441+0000 7f6222d82640 1 -- 172.21.15.33:0/3420312133 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f621c1a71b0 con 0x7f621c106a80 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f6219ffb640 1 -- 172.21.15.33:0/3420312133 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f621401c070 con 0x7f621c106a80 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f622515b740 1 Processor -- start 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f622515b740 1 -- start start 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f622515b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561525327b40 0x561525326090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f622515b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 0x561525326a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f622515b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x561525333760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f622515b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56152528c2f0 con 0x5615253265d0 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f622515b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561525233cf0 con 0x561525327b40 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f622515b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5615252882b0 con 0x561525331370 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f6220b1e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 0x561525326a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f6220b1e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 0x561525326a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37750/0 (socket says 172.21.15.33:37750) 2024-09-16T00:31:40.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f6220b1e640 1 -- 172.21.15.33:0/3045001239 learned_addr learned my addr 172.21.15.33:0/3045001239 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:40.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.444+0000 7f6219ffb640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62040a32c0 0x7f62040a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f6220b1e640 1 -- 172.21.15.33:0/3045001239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 msgr2=0x561525333760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:40.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f6220b1e640 1 --2- 172.21.15.33:0/3045001239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x561525333760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f6220b1e640 1 -- 172.21.15.33:0/3045001239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561525327b40 msgr2=0x561525326090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:40.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f621bfff640 1 --2- 172.21.15.33:0/3045001239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561525327b40 0x561525326090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f6220b1e640 1 --2- 172.21.15.33:0/3045001239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561525327b40 0x561525326090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f622131f640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62040a32c0 0x7f62040a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f6220b1e640 1 -- 172.21.15.33:0/3045001239 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615252183a0 con 0x5615253265d0 2024-09-16T00:31:40.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f621bfff640 1 --2- 172.21.15.33:0/3045001239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561525327b40 0x561525326090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:31:40.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f6219ffb640 1 -- 172.21.15.33:0/3420312133 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f62140cf390 con 0x7f621c106a80 2024-09-16T00:31:40.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f6220b1e640 1 --2- 172.21.15.33:0/3045001239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 0x561525326a10 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f620c005ab0 tx=0x7f620c031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59840623ef4f94d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f6202ffd640 1 -- 172.21.15.33:0/3045001239 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c03f070 con 0x5615253265d0 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f622131f640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62040a32c0 0x7f62040a5780 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f621409b230 tx=0x7f621400f300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f6202ffd640 1 -- 172.21.15.33:0/3045001239 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f620c031a20 con 0x5615253265d0 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.445+0000 7f6202ffd640 1 -- 172.21.15.33:0/3045001239 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c035070 con 0x5615253265d0 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.446+0000 7f622515b740 1 -- 172.21.15.33:0/3045001239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 msgr2=0x561525326a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.446+0000 7f622515b740 1 --2- 172.21.15.33:0/3045001239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 0x561525326a10 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f620c005ab0 tx=0x7f620c031630 comp rx=0 tx=0).stop 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.446+0000 7f622515b740 1 -- 172.21.15.33:0/3045001239 shutdown_connections 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.446+0000 7f622515b740 1 --2- 172.21.15.33:0/3045001239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x561525333760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.446+0000 7f622515b740 1 --2- 172.21.15.33:0/3045001239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 0x561525326a10 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.446+0000 7f622515b740 1 --2- 172.21.15.33:0/3045001239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561525327b40 0x561525326090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.446+0000 7f622515b740 1 -- 172.21.15.33:0/3045001239 >> 172.21.15.33:0/3045001239 conn(0x5615252ef310 msgr2=0x561525322c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.446+0000 7f622515b740 1 -- 172.21.15.33:0/3045001239 shutdown_connections 2024-09-16T00:31:40.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.446+0000 7f622515b740 1 -- 172.21.15.33:0/3045001239 wait complete. 2024-09-16T00:31:40.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.446+0000 7f622515b740 1 -- start start 2024-09-16T00:31:40.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.446+0000 7f622515b740 1 Processor -- start 2024-09-16T00:31:40.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f622515b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615253265d0 0x561525330d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f622515b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 0x56152532dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f622515b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x56152532e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f622515b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561525334010 con 0x561525327b40 2024-09-16T00:31:40.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f622515b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561525247060 con 0x5615253265d0 2024-09-16T00:31:40.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f622515b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561525334190 con 0x561525331370 2024-09-16T00:31:40.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f622515b740 10 client.0 _ll_register_callbacks cb 0x561524f9a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:40.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f6220b1e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 0x56152532dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f6220b1e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 0x56152532dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37762/0 (socket says 172.21.15.33:37762) 2024-09-16T00:31:40.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f6220b1e640 1 -- 172.21.15.33:0/4272704594 learned_addr learned my addr 172.21.15.33:0/4272704594 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f621bfff640 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615253265d0 0x561525330d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f621bfff640 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x56152532e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f6220b1e640 1 -- 172.21.15.33:0/4272704594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 msgr2=0x56152532e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f6220b1e640 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x56152532e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f6220b1e640 1 -- 172.21.15.33:0/4272704594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615253265d0 msgr2=0x561525330d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f6220b1e640 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615253265d0 0x561525330d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f6220b1e640 1 -- 172.21.15.33:0/4272704594 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561524ffd3c0 con 0x561525327b40 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f6220b1e640 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 0x56152532dff0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f620c031550 tx=0x7f620c004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856c77ee92bc0ad8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f61fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.447+0000 7f61fbfff640 1 -- 172.21.15.33:0/4272704594 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c04b020 con 0x561525327b40 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f61fbfff640 1 -- 172.21.15.33:0/4272704594 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f620c03b430 con 0x561525327b40 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f622515b740 1 -- 172.21.15.33:0/4272704594 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f620c031a20 con 0x561525327b40 2024-09-16T00:31:40.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f61fbfff640 1 -- 172.21.15.33:0/4272704594 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f620c056050 con 0x561525327b40 2024-09-16T00:31:40.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f622515b740 10 client.18129 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:40.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f622515b740 1 -- 172.21.15.33:0/4272704594 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56152533fa20 con 0x561525327b40 2024-09-16T00:31:40.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f61fbfff640 1 -- 172.21.15.33:0/4272704594 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f620c031a20 con 0x561525327b40 2024-09-16T00:31:40.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f61fbfff640 1 -- 172.21.15.33:0/4272704594 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f620c043840 con 0x561525327b40 2024-09-16T00:31:40.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f622515b740 10 client.18129 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:40.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f622515b740 10 client.18129 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:31:40.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f622515b740 20 client.18129 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:40.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f622515b740 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561525340d70 0x561525343160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f622515b740 4 client.18129 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:31:40.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f622515b740 1 -- 172.21.15.33:0/4272704594 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561525299860 con 0x561525340d70 2024-09-16T00:31:40.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.448+0000 7f621bfff640 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561525340d70 0x561525343160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.449+0000 7f621bfff640 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561525340d70 0x561525343160 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d415bf9aef591dd7 server_cookie=e035c1d1d1bbf32d in_seq=0 out_seq=0 2024-09-16T00:31:40.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.449+0000 7f61fbfff640 10 client.18129 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:31:40.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f61fbfff640 1 -- 172.21.15.33:0/4272704594 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561525299860 con 0x561525340d70 2024-09-16T00:31:40.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f61fbfff640 10 client.18129 handle_command_reply: tid=0 2024-09-16T00:31:40.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f622515b740 1 client.18129 shutdown 2024-09-16T00:31:40.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f622515b740 1 -- 172.21.15.33:0/4272704594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 msgr2=0x56152532dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:40.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f622515b740 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 0x56152532dff0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f620c031550 tx=0x7f620c004a50 comp rx=0 tx=0).stop 2024-09-16T00:31:40.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f622515b740 1 -- 172.21.15.33:0/4272704594 shutdown_connections 2024-09-16T00:31:40.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f622515b740 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x56152532e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f622515b740 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 0x56152532dff0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f622515b740 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x561525340d70 0x561525343160 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f622515b740 1 --2- 172.21.15.33:0/4272704594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615253265d0 0x561525330d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f61fbfff640 0 client.18129 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:31:40.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f622515b740 1 -- 172.21.15.33:0/4272704594 >> 172.21.15.33:0/4272704594 conn(0x5615252ef310 msgr2=0x561525328930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:40.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f622515b740 1 -- 172.21.15.33:0/4272704594 shutdown_connections 2024-09-16T00:31:40.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f622515b740 1 -- 172.21.15.33:0/4272704594 wait complete. 2024-09-16T00:31:40.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.450+0000 7f622515b740 20 client.18129 trim_cache size 0 max 16384 2024-09-16T00:31:40.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f622515b740 1 Processor -- start 2024-09-16T00:31:40.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f622515b740 1 -- start start 2024-09-16T00:31:40.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f622515b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615253265d0 0x56152532dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f622515b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 0x56152533d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f622515b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x561525348700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f622515b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f621c1b3b30 con 0x561525327b40 2024-09-16T00:31:40.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f622515b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f621c1b13a0 con 0x5615253265d0 2024-09-16T00:31:40.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f622515b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f621c1af1f0 con 0x561525331370 2024-09-16T00:31:40.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f6220b1e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 0x56152533d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f6220b1e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 0x56152533d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37774/0 (socket says 172.21.15.33:37774) 2024-09-16T00:31:40.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f6220b1e640 1 -- 172.21.15.33:0/272199282 learned_addr learned my addr 172.21.15.33:0/272199282 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:40.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f621bfff640 1 --2- 172.21.15.33:0/272199282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615253265d0 0x56152532dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f621bfff640 1 --2- 172.21.15.33:0/272199282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x561525348700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f6220b1e640 1 -- 172.21.15.33:0/272199282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 msgr2=0x561525348700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:40.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f6220b1e640 1 --2- 172.21.15.33:0/272199282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x561525348700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f6220b1e640 1 -- 172.21.15.33:0/272199282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615253265d0 msgr2=0x56152532dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:40.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f6220b1e640 1 --2- 172.21.15.33:0/272199282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615253265d0 0x56152532dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f6220b1e640 1 -- 172.21.15.33:0/272199282 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615253262d0 con 0x561525327b40 2024-09-16T00:31:40.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.464+0000 7f621bfff640 1 --2- 172.21.15.33:0/272199282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615253265d0 0x56152532dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:40.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f6220b1e640 1 --2- 172.21.15.33:0/272199282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 0x56152533d2e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f620c0096c0 tx=0x7f620c031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc4a86c110a002b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:40.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f62027fc640 1 -- 172.21.15.33:0/272199282 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c04b070 con 0x561525327b40 2024-09-16T00:31:40.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f62027fc640 1 -- 172.21.15.33:0/272199282 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f620c004ac0 con 0x561525327b40 2024-09-16T00:31:40.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f62027fc640 1 -- 172.21.15.33:0/272199282 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c035070 con 0x561525327b40 2024-09-16T00:31:40.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f622515b740 1 -- 172.21.15.33:0/272199282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 msgr2=0x56152533d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:40.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f622515b740 1 --2- 172.21.15.33:0/272199282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 0x56152533d2e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f620c0096c0 tx=0x7f620c031630 comp rx=0 tx=0).stop 2024-09-16T00:31:40.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f622515b740 1 -- 172.21.15.33:0/272199282 shutdown_connections 2024-09-16T00:31:40.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f622515b740 1 --2- 172.21.15.33:0/272199282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x561525348700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f622515b740 1 --2- 172.21.15.33:0/272199282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561525327b40 0x56152533d2e0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f622515b740 1 --2- 172.21.15.33:0/272199282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615253265d0 0x56152532dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f622515b740 1 -- 172.21.15.33:0/272199282 >> 172.21.15.33:0/272199282 conn(0x5615252ef310 msgr2=0x56152532d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:40.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f622515b740 1 -- 172.21.15.33:0/272199282 shutdown_connections 2024-09-16T00:31:40.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.465+0000 7f622515b740 1 -- 172.21.15.33:0/272199282 wait complete. 2024-09-16T00:31:40.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.466+0000 7f622515b740 1 -- start start 2024-09-16T00:31:40.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.466+0000 7f622515b740 1 Processor -- start 2024-09-16T00:31:40.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.466+0000 7f622515b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 0x5615253396b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.466+0000 7f622515b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561525327b40 0x561525339bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.466+0000 7f622515b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x56152532a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.466+0000 7f622515b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f621c1af860 con 0x5615253265d0 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.466+0000 7f622515b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f621c1b0cd0 con 0x561525327b40 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.466+0000 7f622515b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f621c1aff30 con 0x561525331370 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.466+0000 7f622515b740 10 client.0 _ll_register_callbacks cb 0x7f620c009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.466+0000 7f621bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 0x5615253396b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.466+0000 7f621bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 0x5615253396b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37788/0 (socket says 172.21.15.33:37788) 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.466+0000 7f621bfff640 1 -- 172.21.15.33:0/925955287 learned_addr learned my addr 172.21.15.33:0/925955287 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f6220b1e640 1 --2- 172.21.15.33:0/925955287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561525327b40 0x561525339bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f621bfff640 1 -- 172.21.15.33:0/925955287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 msgr2=0x56152532a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f621bfff640 1 --2- 172.21.15.33:0/925955287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x56152532a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f621bfff640 1 -- 172.21.15.33:0/925955287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561525327b40 msgr2=0x561525339bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f621bfff640 1 --2- 172.21.15.33:0/925955287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561525327b40 0x561525339bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f621bfff640 1 -- 172.21.15.33:0/925955287 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615253224f0 con 0x5615253265d0 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f621bfff640 1 --2- 172.21.15.33:0/925955287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 0x5615253396b0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f6210005d90 tx=0x7f621000cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25c126b8c0d094e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f61fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f61fbfff640 1 -- 172.21.15.33:0/925955287 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6210024040 con 0x5615253265d0 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f61fbfff640 1 -- 172.21.15.33:0/925955287 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f621001fe80 con 0x5615253265d0 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f622515b740 1 -- 172.21.15.33:0/925955287 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f620c004ac0 con 0x5615253265d0 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f61fbfff640 1 -- 172.21.15.33:0/925955287 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6210004020 con 0x5615253265d0 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f622515b740 10 client.18135 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.467+0000 7f622515b740 1 -- 172.21.15.33:0/925955287 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561525323960 con 0x5615253265d0 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.468+0000 7f61fbfff640 1 -- 172.21.15.33:0/925955287 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62100041c0 con 0x5615253265d0 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.468+0000 7f61fbfff640 1 -- 172.21.15.33:0/925955287 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6210003680 con 0x5615253265d0 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.468+0000 7f622515b740 10 client.18135 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.468+0000 7f622515b740 10 client.18135 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:31:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.468+0000 7f622515b740 20 client.18135 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:40.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.468+0000 7f622515b740 1 --2- 172.21.15.33:0/925955287 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5615253ab680 0x5615253adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:40.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.468+0000 7f622515b740 4 client.18135 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:31:40.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.468+0000 7f622515b740 1 -- 172.21.15.33:0/925955287 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f620c004ac0 con 0x5615253ab680 2024-09-16T00:31:40.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.468+0000 7f6220b1e640 1 --2- 172.21.15.33:0/925955287 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5615253ab680 0x5615253adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:40.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.468+0000 7f6220b1e640 1 --2- 172.21.15.33:0/925955287 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5615253ab680 0x5615253adaa0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b12c6182ecd9312 server_cookie=45e8d03ed7dd7bdf in_seq=0 out_seq=0 2024-09-16T00:31:40.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.468+0000 7f61fbfff640 10 client.18135 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:31:40.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f61fbfff640 1 -- 172.21.15.33:0/925955287 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f620c004ac0 con 0x5615253ab680 2024-09-16T00:31:40.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f61fbfff640 10 client.18135 handle_command_reply: tid=0 2024-09-16T00:31:40.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f622515b740 1 client.18135 shutdown 2024-09-16T00:31:40.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f622515b740 1 -- 172.21.15.33:0/925955287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 msgr2=0x5615253396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:40.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f622515b740 1 --2- 172.21.15.33:0/925955287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 0x5615253396b0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f6210005d90 tx=0x7f621000cab0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f622515b740 1 -- 172.21.15.33:0/925955287 shutdown_connections 2024-09-16T00:31:40.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f622515b740 1 --2- 172.21.15.33:0/925955287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561525331370 0x56152532a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f622515b740 1 --2- 172.21.15.33:0/925955287 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5615253ab680 0x5615253adaa0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f622515b740 1 --2- 172.21.15.33:0/925955287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561525327b40 0x561525339bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f622515b740 1 --2- 172.21.15.33:0/925955287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615253265d0 0x5615253396b0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f61fbfff640 0 client.18135 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:31:40.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f622515b740 1 -- 172.21.15.33:0/925955287 >> 172.21.15.33:0/925955287 conn(0x5615252ef310 msgr2=0x56152532c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:40.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f622515b740 1 -- 172.21.15.33:0/925955287 shutdown_connections 2024-09-16T00:31:40.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f622515b740 1 -- 172.21.15.33:0/925955287 wait complete. 2024-09-16T00:31:40.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.469+0000 7f622515b740 20 client.18135 trim_cache size 0 max 16384 2024-09-16T00:31:40.475 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:31:40.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.470+0000 7f6222d82640 1 -- 172.21.15.33:0/3420312133 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62040a32c0 msgr2=0x7f62040a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:40.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.470+0000 7f6222d82640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62040a32c0 0x7f62040a5780 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f621409b230 tx=0x7f621400f300 comp rx=0 tx=0).stop 2024-09-16T00:31:40.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.470+0000 7f6222d82640 1 -- 172.21.15.33:0/3420312133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f621c106a80 msgr2=0x7f621c1030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:40.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.470+0000 7f6222d82640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f621c106a80 0x7f621c1030d0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f621400b970 tx=0x7f621400be30 comp rx=0 tx=0).stop 2024-09-16T00:31:40.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.470+0000 7f6222d82640 1 -- 172.21.15.33:0/3420312133 shutdown_connections 2024-09-16T00:31:40.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.470+0000 7f6222d82640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62040a32c0 0x7f62040a5780 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.470+0000 7f6222d82640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f621c106a80 0x7f621c1030d0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.470+0000 7f6222d82640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f621c101420 0x7f621c102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.470+0000 7f6222d82640 1 --2- 172.21.15.33:0/3420312133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f621c100a80 0x7f621c102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:40.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.470+0000 7f6222d82640 1 -- 172.21.15.33:0/3420312133 >> 172.21.15.33:0/3420312133 conn(0x7f621c0fc7a0 msgr2=0x7f621c06a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:40.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.470+0000 7f6222d82640 1 -- 172.21.15.33:0/3420312133 shutdown_connections 2024-09-16T00:31:40.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:40.470+0000 7f6222d82640 1 -- 172.21.15.33:0/3420312133 wait complete. 2024-09-16T00:31:41.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 19.47 MB/sec warmup 47 sec latency 20731.578 ms 2024-09-16T00:31:41.306 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:31:41.567 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:41.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:41 smithi033 ceph-mon[31243]: pgmap v786: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.4 KiB/s wr, 9 op/s 2024-09-16T00:31:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:41 smithi083 ceph-mon[35242]: pgmap v786: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.4 KiB/s wr, 9 op/s 2024-09-16T00:31:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:41 smithi060 ceph-mon[35185]: pgmap v786: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.4 KiB/s wr, 9 op/s 2024-09-16T00:31:42.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 19.07 MB/sec warmup 48 sec latency 21731.626 ms 2024-09-16T00:31:42.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.065+0000 7fb282392640 1 -- 172.21.15.33:0/1548407280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c10c170 msgr2=0x7fb27c10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:42.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.065+0000 7fb282392640 1 --2- 172.21.15.33:0/1548407280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c10c170 0x7fb27c10e610 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fb274009ca0 tx=0x7fb27402a950 comp rx=0 tx=0).stop 2024-09-16T00:31:42.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.065+0000 7fb282392640 1 -- 172.21.15.33:0/1548407280 shutdown_connections 2024-09-16T00:31:42.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.065+0000 7fb282392640 1 --2- 172.21.15.33:0/1548407280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c10c170 0x7fb27c10e610 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.065+0000 7fb282392640 1 --2- 172.21.15.33:0/1548407280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb27c103660 0x7fb27c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.065+0000 7fb282392640 1 --2- 172.21.15.33:0/1548407280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb27c102d40 0x7fb27c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.065+0000 7fb282392640 1 -- 172.21.15.33:0/1548407280 >> 172.21.15.33:0/1548407280 conn(0x7fb27c0fc7a0 msgr2=0x7fb27c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:42.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.065+0000 7fb282392640 1 -- 172.21.15.33:0/1548407280 shutdown_connections 2024-09-16T00:31:42.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.065+0000 7fb282392640 1 -- 172.21.15.33:0/1548407280 wait complete. 2024-09-16T00:31:42.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.066+0000 7fb282392640 1 Processor -- start 2024-09-16T00:31:42.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.066+0000 7fb282392640 1 -- start start 2024-09-16T00:31:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.066+0000 7fb282392640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c102d40 0x7fb27c19a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.066+0000 7fb282392640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb27c103660 0x7fb27c19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.066+0000 7fb282392640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb27c10c170 0x7fb27c19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.066+0000 7fb282392640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c111f30 con 0x7fb27c102d40 2024-09-16T00:31:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.066+0000 7fb282392640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c111db0 con 0x7fb27c10c170 2024-09-16T00:31:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.066+0000 7fb282392640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1120b0 con 0x7fb27c103660 2024-09-16T00:31:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb27bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c102d40 0x7fb27c19a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb27bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c102d40 0x7fb27c19a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37804/0 (socket says 172.21.15.33:37804) 2024-09-16T00:31:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb27bfff640 1 -- 172.21.15.33:0/2539022296 learned_addr learned my addr 172.21.15.33:0/2539022296 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb27b7fe640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb27c103660 0x7fb27c19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb28092f640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb27c10c170 0x7fb27c19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb27bfff640 1 -- 172.21.15.33:0/2539022296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb27c103660 msgr2=0x7fb27c19aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb27bfff640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb27c103660 0x7fb27c19aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb27bfff640 1 -- 172.21.15.33:0/2539022296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb27c10c170 msgr2=0x7fb27c19ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb27bfff640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb27c10c170 0x7fb27c19ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb27bfff640 1 -- 172.21.15.33:0/2539022296 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27c19f530 con 0x7fb27c102d40 2024-09-16T00:31:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb27bfff640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c102d40 0x7fb27c19a580 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00c920 tx=0x7fb26c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb2797fa640 1 -- 172.21.15.33:0/2539022296 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c007c70 con 0x7fb27c102d40 2024-09-16T00:31:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb2797fa640 1 -- 172.21.15.33:0/2539022296 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb26c007e10 con 0x7fb27c102d40 2024-09-16T00:31:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb282392640 1 -- 172.21.15.33:0/2539022296 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27c19f820 con 0x7fb27c102d40 2024-09-16T00:31:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb2797fa640 1 -- 172.21.15.33:0/2539022296 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c018890 con 0x7fb27c102d40 2024-09-16T00:31:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.067+0000 7fb282392640 1 -- 172.21.15.33:0/2539022296 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb27c1a7100 con 0x7fb27c102d40 2024-09-16T00:31:42.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.070+0000 7fb2797fa640 1 -- 172.21.15.33:0/2539022296 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb26c018a70 con 0x7fb27c102d40 2024-09-16T00:31:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.070+0000 7fb28476b740 1 Processor -- start 2024-09-16T00:31:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.070+0000 7fb28476b740 1 -- start start 2024-09-16T00:31:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.070+0000 7fb28476b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 0x5634be6c9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.070+0000 7fb28476b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6c95d0 0x5634be6c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.070+0000 7fb28476b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6d4370 0x5634be6d6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.070+0000 7fb28476b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5634be62f2f0 con 0x5634be6cab40 2024-09-16T00:31:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.070+0000 7fb28476b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5634be5d6cf0 con 0x5634be6d4370 2024-09-16T00:31:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.070+0000 7fb28476b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5634be62b2b0 con 0x5634be6c95d0 2024-09-16T00:31:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.070+0000 7fb28092f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 0x5634be6c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.070+0000 7fb28092f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 0x5634be6c9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37818/0 (socket says 172.21.15.33:37818) 2024-09-16T00:31:42.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.070+0000 7fb28092f640 1 -- 172.21.15.33:0/1848336477 learned_addr learned my addr 172.21.15.33:0/1848336477 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb27bfff640 1 --2- 172.21.15.33:0/1848336477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6c95d0 0x5634be6c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb27b7fe640 1 --2- 172.21.15.33:0/1848336477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6d4370 0x5634be6d6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb2797fa640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb2640a3240 0x7fb2640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb28092f640 1 -- 172.21.15.33:0/1848336477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6c95d0 msgr2=0x5634be6c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb28092f640 1 --2- 172.21.15.33:0/1848336477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6c95d0 0x5634be6c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb28092f640 1 -- 172.21.15.33:0/1848336477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6d4370 msgr2=0x5634be6d6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb28092f640 1 --2- 172.21.15.33:0/1848336477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6d4370 0x5634be6d6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb28092f640 1 -- 172.21.15.33:0/1848336477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634be5bb3a0 con 0x5634be6cab40 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb27b7fe640 1 --2- 172.21.15.33:0/1848336477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6d4370 0x5634be6d6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb2797fa640 1 -- 172.21.15.33:0/2539022296 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb26c0cfee0 con 0x7fb27c102d40 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb28092f640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb2640a3240 0x7fb2640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb28092f640 1 --2- 172.21.15.33:0/1848336477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 0x5634be6c9090 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb274030040 tx=0x7fb27402f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b9b864b90bbab5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:42.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.071+0000 7fb2627fc640 1 -- 172.21.15.33:0/1848336477 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274041040 con 0x5634be6cab40 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.072+0000 7fb2627fc640 1 -- 172.21.15.33:0/1848336477 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb274004520 con 0x5634be6cab40 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.072+0000 7fb2627fc640 1 -- 172.21.15.33:0/1848336477 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274034040 con 0x5634be6cab40 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.072+0000 7fb28092f640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb2640a3240 0x7fb2640a5700 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb27404c000 tx=0x7fb27404d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.072+0000 7fb28476b740 1 -- 172.21.15.33:0/1848336477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 msgr2=0x5634be6c9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.072+0000 7fb28476b740 1 --2- 172.21.15.33:0/1848336477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 0x5634be6c9090 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb274030040 tx=0x7fb27402f9c0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.072+0000 7fb28476b740 1 -- 172.21.15.33:0/1848336477 shutdown_connections 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.072+0000 7fb28476b740 1 --2- 172.21.15.33:0/1848336477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6d4370 0x5634be6d6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.072+0000 7fb28476b740 1 --2- 172.21.15.33:0/1848336477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6c95d0 0x5634be6c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.072+0000 7fb28476b740 1 --2- 172.21.15.33:0/1848336477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 0x5634be6c9090 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.072+0000 7fb28476b740 1 -- 172.21.15.33:0/1848336477 >> 172.21.15.33:0/1848336477 conn(0x5634be692310 msgr2=0x5634be6c5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.072+0000 7fb28476b740 1 -- 172.21.15.33:0/1848336477 shutdown_connections 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.072+0000 7fb28476b740 1 -- 172.21.15.33:0/1848336477 wait complete. 2024-09-16T00:31:42.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb28476b740 1 -- start start 2024-09-16T00:31:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb28476b740 1 Processor -- start 2024-09-16T00:31:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb28476b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6c95d0 0x5634be6d3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb28476b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 0x5634be6d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb28476b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6d4370 0x5634be6d1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb28476b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5634be6d7070 con 0x5634be6cab40 2024-09-16T00:31:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb28476b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5634be5ea060 con 0x5634be6c95d0 2024-09-16T00:31:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb28476b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5634be6d71f0 con 0x5634be6d4370 2024-09-16T00:31:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb28476b740 10 client.0 _ll_register_callbacks cb 0x5634be33d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb27b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 0x5634be6d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb27b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 0x5634be6d1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37826/0 (socket says 172.21.15.33:37826) 2024-09-16T00:31:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb27b7fe640 1 -- 172.21.15.33:0/3687673907 learned_addr learned my addr 172.21.15.33:0/3687673907 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.073+0000 7fb28092f640 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6d4370 0x5634be6d1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb27bfff640 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6c95d0 0x5634be6d3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb27b7fe640 1 -- 172.21.15.33:0/3687673907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6d4370 msgr2=0x5634be6d1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb27b7fe640 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6d4370 0x5634be6d1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb27b7fe640 1 -- 172.21.15.33:0/3687673907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6c95d0 msgr2=0x5634be6d3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb27b7fe640 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6c95d0 0x5634be6d3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb27b7fe640 1 -- 172.21.15.33:0/3687673907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634be3a03c0 con 0x5634be6cab40 2024-09-16T00:31:42.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb27b7fe640 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 0x5634be6d1010 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fb270010750 tx=0x7fb270010c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77478b9a97202cc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:42.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb25b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:42.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb25b7fe640 1 -- 172.21.15.33:0/3687673907 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27001e060 con 0x5634be6cab40 2024-09-16T00:31:42.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb25b7fe640 1 -- 172.21.15.33:0/3687673907 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb27000fbd0 con 0x5634be6cab40 2024-09-16T00:31:42.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb28476b740 1 -- 172.21.15.33:0/3687673907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb274004520 con 0x5634be6cab40 2024-09-16T00:31:42.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb25b7fe640 1 -- 172.21.15.33:0/3687673907 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb270025050 con 0x5634be6cab40 2024-09-16T00:31:42.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb28476b740 10 client.18147 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:42.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb28476b740 1 -- 172.21.15.33:0/3687673907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5634be6e2b70 con 0x5634be6cab40 2024-09-16T00:31:42.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.074+0000 7fb25b7fe640 1 -- 172.21.15.33:0/3687673907 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb274004520 con 0x5634be6cab40 2024-09-16T00:31:42.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.075+0000 7fb25b7fe640 1 -- 172.21.15.33:0/3687673907 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb270018370 con 0x5634be6cab40 2024-09-16T00:31:42.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.075+0000 7fb28476b740 10 client.18147 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:42.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.075+0000 7fb28476b740 10 client.18147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:31:42.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.075+0000 7fb28476b740 20 client.18147 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:42.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.075+0000 7fb28476b740 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634be6e4080 0x5634be6e6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.075+0000 7fb28476b740 4 client.18147 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:31:42.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.075+0000 7fb28476b740 1 -- 172.21.15.33:0/3687673907 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634be63c860 con 0x5634be6e4080 2024-09-16T00:31:42.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.075+0000 7fb27bfff640 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634be6e4080 0x5634be6e6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.075+0000 7fb27bfff640 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634be6e4080 0x5634be6e6490 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55ce269ccb5adeff server_cookie=de1fb2fa6dc3a1f in_seq=0 out_seq=0 2024-09-16T00:31:42.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.075+0000 7fb25b7fe640 10 client.18147 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:42.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.076+0000 7fb25b7fe640 1 -- 172.21.15.33:0/3687673907 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5634be63c860 con 0x5634be6e4080 2024-09-16T00:31:42.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.076+0000 7fb25b7fe640 10 client.18147 handle_command_reply: tid=0 2024-09-16T00:31:42.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.076+0000 7fb28476b740 1 client.18147 shutdown 2024-09-16T00:31:42.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.076+0000 7fb28476b740 1 -- 172.21.15.33:0/3687673907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 msgr2=0x5634be6d1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:42.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.076+0000 7fb28476b740 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 0x5634be6d1010 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fb270010750 tx=0x7fb270010c10 comp rx=0 tx=0).stop 2024-09-16T00:31:42.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.076+0000 7fb28476b740 1 -- 172.21.15.33:0/3687673907 shutdown_connections 2024-09-16T00:31:42.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.076+0000 7fb28476b740 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634be6e4080 0x5634be6e6490 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.076+0000 7fb28476b740 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6d4370 0x5634be6d1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.076+0000 7fb28476b740 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6cab40 0x5634be6d1010 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.076+0000 7fb28476b740 1 --2- 172.21.15.33:0/3687673907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6c95d0 0x5634be6d3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.076+0000 7fb28476b740 1 -- 172.21.15.33:0/3687673907 >> 172.21.15.33:0/3687673907 conn(0x5634be692310 msgr2=0x5634be6cb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:42.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.076+0000 7fb25b7fe640 0 client.18147 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:42.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.077+0000 7fb28476b740 1 -- 172.21.15.33:0/3687673907 shutdown_connections 2024-09-16T00:31:42.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.077+0000 7fb28476b740 1 -- 172.21.15.33:0/3687673907 wait complete. 2024-09-16T00:31:42.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.077+0000 7fb28476b740 20 client.18147 trim_cache size 0 max 16384 2024-09-16T00:31:42.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.090+0000 7fb28476b740 1 Processor -- start 2024-09-16T00:31:42.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.090+0000 7fb28476b740 1 -- start start 2024-09-16T00:31:42.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.090+0000 7fb28476b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 0x5634be6cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.090+0000 7fb28476b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6cab40 0x5634be6d2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.090+0000 7fb28476b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6d4370 0x5634be6cc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.090+0000 7fb28476b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1b3b30 con 0x5634be6c95d0 2024-09-16T00:31:42.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.090+0000 7fb28476b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1b13a0 con 0x5634be6cab40 2024-09-16T00:31:42.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.090+0000 7fb28476b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1af1f0 con 0x5634be6d4370 2024-09-16T00:31:42.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.090+0000 7fb27bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 0x5634be6cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.090+0000 7fb27bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 0x5634be6cffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37832/0 (socket says 172.21.15.33:37832) 2024-09-16T00:31:42.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.090+0000 7fb27bfff640 1 -- 172.21.15.33:0/3492948103 learned_addr learned my addr 172.21.15.33:0/3492948103 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:42.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.090+0000 7fb27b7fe640 1 --2- 172.21.15.33:0/3492948103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6cab40 0x5634be6d2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.091+0000 7fb28092f640 1 --2- 172.21.15.33:0/3492948103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6d4370 0x5634be6cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.091+0000 7fb27bfff640 1 -- 172.21.15.33:0/3492948103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6d4370 msgr2=0x5634be6cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:42.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.091+0000 7fb27bfff640 1 --2- 172.21.15.33:0/3492948103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6d4370 0x5634be6cc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.091+0000 7fb27bfff640 1 -- 172.21.15.33:0/3492948103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6cab40 msgr2=0x5634be6d2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:42.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.091+0000 7fb27bfff640 1 --2- 172.21.15.33:0/3492948103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6cab40 0x5634be6d2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.091+0000 7fb27bfff640 1 -- 172.21.15.33:0/3492948103 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634be6d1970 con 0x5634be6c95d0 2024-09-16T00:31:42.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.091+0000 7fb28092f640 1 --2- 172.21.15.33:0/3492948103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6d4370 0x5634be6cc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:31:42.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.091+0000 7fb27b7fe640 1 --2- 172.21.15.33:0/3492948103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6cab40 0x5634be6d2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:42.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.091+0000 7fb27bfff640 1 --2- 172.21.15.33:0/3492948103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 0x5634be6cffb0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0984e0 tx=0x7fb26c0d23f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beabb4b97b1ffae4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:42.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.091+0000 7fb261ffb640 1 -- 172.21.15.33:0/3492948103 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c09c780 con 0x5634be6c95d0 2024-09-16T00:31:42.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.091+0000 7fb261ffb640 1 -- 172.21.15.33:0/3492948103 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb26c09c080 con 0x5634be6c95d0 2024-09-16T00:31:42.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.091+0000 7fb261ffb640 1 -- 172.21.15.33:0/3492948103 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c018d60 con 0x5634be6c95d0 2024-09-16T00:31:42.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.092+0000 7fb28476b740 1 -- 172.21.15.33:0/3492948103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 msgr2=0x5634be6cffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:42.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.092+0000 7fb28476b740 1 --2- 172.21.15.33:0/3492948103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 0x5634be6cffb0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0984e0 tx=0x7fb26c0d23f0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.092+0000 7fb28476b740 1 -- 172.21.15.33:0/3492948103 shutdown_connections 2024-09-16T00:31:42.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.092+0000 7fb28476b740 1 --2- 172.21.15.33:0/3492948103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6d4370 0x5634be6cc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.092+0000 7fb28476b740 1 --2- 172.21.15.33:0/3492948103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6cab40 0x5634be6d2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.092+0000 7fb28476b740 1 --2- 172.21.15.33:0/3492948103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 0x5634be6cffb0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.092+0000 7fb28476b740 1 -- 172.21.15.33:0/3492948103 >> 172.21.15.33:0/3492948103 conn(0x5634be692310 msgr2=0x5634be6d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:42.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.092+0000 7fb28476b740 1 -- 172.21.15.33:0/3492948103 shutdown_connections 2024-09-16T00:31:42.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.092+0000 7fb28476b740 1 -- 172.21.15.33:0/3492948103 wait complete. 2024-09-16T00:31:42.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.093+0000 7fb28476b740 1 -- start start 2024-09-16T00:31:42.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.093+0000 7fb28476b740 1 Processor -- start 2024-09-16T00:31:42.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.093+0000 7fb28476b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 0x5634be6dcae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.093+0000 7fb28476b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6cab40 0x5634be6cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb28476b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6d4370 0x5634be6ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb28476b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1af860 con 0x5634be6c95d0 2024-09-16T00:31:42.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb28476b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1b0cd0 con 0x5634be6d4370 2024-09-16T00:31:42.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb28476b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1aff30 con 0x5634be6cab40 2024-09-16T00:31:42.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb28476b740 10 client.0 _ll_register_callbacks cb 0x7fb27000a710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb27bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 0x5634be6dcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb27bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 0x5634be6dcae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37846/0 (socket says 172.21.15.33:37846) 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb27bfff640 1 -- 172.21.15.33:0/2282310587 learned_addr learned my addr 172.21.15.33:0/2282310587 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb28092f640 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6d4370 0x5634be6ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb27b7fe640 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6cab40 0x5634be6cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb27bfff640 1 -- 172.21.15.33:0/2282310587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6cab40 msgr2=0x5634be6cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb27bfff640 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6cab40 0x5634be6cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb27bfff640 1 -- 172.21.15.33:0/2282310587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6d4370 msgr2=0x5634be6ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb27bfff640 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6d4370 0x5634be6ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb27bfff640 1 -- 172.21.15.33:0/2282310587 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634be6ce840 con 0x5634be6c95d0 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb27b7fe640 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6cab40 0x5634be6cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb27bfff640 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 0x5634be6dcae0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00cf30 tx=0x7fb26c09c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3f5884089d7f9c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:42.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb25b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:42.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb25b7fe640 1 -- 172.21.15.33:0/2282310587 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c08b060 con 0x5634be6c95d0 2024-09-16T00:31:42.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb25b7fe640 1 -- 172.21.15.33:0/2282310587 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb26c09a380 con 0x5634be6c95d0 2024-09-16T00:31:42.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.094+0000 7fb28476b740 1 -- 172.21.15.33:0/2282310587 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb26c09c080 con 0x5634be6c95d0 2024-09-16T00:31:42.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.095+0000 7fb25b7fe640 1 -- 172.21.15.33:0/2282310587 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb26c0a0430 con 0x5634be6c95d0 2024-09-16T00:31:42.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.095+0000 7fb28476b740 10 client.18153 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:42.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.095+0000 7fb28476b740 1 -- 172.21.15.33:0/2282310587 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5634be74eb70 con 0x5634be6c95d0 2024-09-16T00:31:42.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.095+0000 7fb25b7fe640 1 -- 172.21.15.33:0/2282310587 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb26c09c080 con 0x5634be6c95d0 2024-09-16T00:31:42.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.095+0000 7fb25b7fe640 1 -- 172.21.15.33:0/2282310587 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb26c086b50 con 0x5634be6c95d0 2024-09-16T00:31:42.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.095+0000 7fb28476b740 10 client.18153 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:42.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.095+0000 7fb28476b740 10 client.18153 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:31:42.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.095+0000 7fb28476b740 20 client.18153 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:42.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.095+0000 7fb28476b740 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634be74f1e0 0x5634be751600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:42.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.095+0000 7fb28476b740 4 client.18153 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:31:42.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.095+0000 7fb28476b740 1 -- 172.21.15.33:0/2282310587 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5634be636d70 con 0x5634be74f1e0 2024-09-16T00:31:42.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.095+0000 7fb27b7fe640 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634be74f1e0 0x5634be751600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:42.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb27b7fe640 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634be74f1e0 0x5634be751600 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74cea62402c23277 server_cookie=331f93e6efd0ad83 in_seq=0 out_seq=0 2024-09-16T00:31:42.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb25b7fe640 10 client.18153 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:42.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb25b7fe640 1 -- 172.21.15.33:0/2282310587 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5634be636d70 con 0x5634be74f1e0 2024-09-16T00:31:42.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb25b7fe640 10 client.18153 handle_command_reply: tid=0 2024-09-16T00:31:42.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb28476b740 1 client.18153 shutdown 2024-09-16T00:31:42.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb28476b740 1 -- 172.21.15.33:0/2282310587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 msgr2=0x5634be6dcae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:42.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb28476b740 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 0x5634be6dcae0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00cf30 tx=0x7fb26c09c780 comp rx=0 tx=0).stop 2024-09-16T00:31:42.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb28476b740 1 -- 172.21.15.33:0/2282310587 shutdown_connections 2024-09-16T00:31:42.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb28476b740 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634be74f1e0 0x5634be751600 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb28476b740 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634be6d4370 0x5634be6ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb28476b740 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634be6cab40 0x5634be6cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb28476b740 1 --2- 172.21.15.33:0/2282310587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634be6c95d0 0x5634be6dcae0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb25b7fe640 0 client.18153 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:42.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.096+0000 7fb28476b740 1 -- 172.21.15.33:0/2282310587 >> 172.21.15.33:0/2282310587 conn(0x5634be692310 msgr2=0x5634be6d3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:42.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb28476b740 1 -- 172.21.15.33:0/2282310587 shutdown_connections 2024-09-16T00:31:42.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb28476b740 1 -- 172.21.15.33:0/2282310587 wait complete. 2024-09-16T00:31:42.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb28476b740 20 client.18153 trim_cache size 0 max 16384 2024-09-16T00:31:42.102 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:31:42.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb282392640 1 -- 172.21.15.33:0/2539022296 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb2640a3240 msgr2=0x7fb2640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:42.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb282392640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb2640a3240 0x7fb2640a5700 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb27404c000 tx=0x7fb27404d000 comp rx=0 tx=0).stop 2024-09-16T00:31:42.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb282392640 1 -- 172.21.15.33:0/2539022296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c102d40 msgr2=0x7fb27c19a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:42.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb282392640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c102d40 0x7fb27c19a580 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00c920 tx=0x7fb26c00cde0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb282392640 1 -- 172.21.15.33:0/2539022296 shutdown_connections 2024-09-16T00:31:42.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb282392640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb2640a3240 0x7fb2640a5700 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb282392640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb27c10c170 0x7fb27c19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb282392640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb27c103660 0x7fb27c19aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb282392640 1 --2- 172.21.15.33:0/2539022296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c102d40 0x7fb27c19a580 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:42.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.097+0000 7fb282392640 1 -- 172.21.15.33:0/2539022296 >> 172.21.15.33:0/2539022296 conn(0x7fb27c0fc7a0 msgr2=0x7fb27c10d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:42.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.098+0000 7fb282392640 1 -- 172.21.15.33:0/2539022296 shutdown_connections 2024-09-16T00:31:42.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:42.098+0000 7fb282392640 1 -- 172.21.15.33:0/2539022296 wait complete. 2024-09-16T00:31:43.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 18.68 MB/sec warmup 49 sec latency 22731.721 ms 2024-09-16T00:31:43.176 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:31:43.442 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:43.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:43 smithi033 ceph-mon[31243]: pgmap v787: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.4 KiB/s wr, 9 op/s 2024-09-16T00:31:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:43 smithi083 ceph-mon[35242]: pgmap v787: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.4 KiB/s wr, 9 op/s 2024-09-16T00:31:43.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.911+0000 7fad3c20d640 1 -- 172.21.15.33:0/4278284967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad34101de0 msgr2=0x7fad341021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:43.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.911+0000 7fad3c20d640 1 --2- 172.21.15.33:0/4278284967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad34101de0 0x7fad341021c0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fad30009900 tx=0x7fad3002a410 comp rx=0 tx=0).stop 2024-09-16T00:31:43.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.911+0000 7fad3c20d640 1 -- 172.21.15.33:0/4278284967 shutdown_connections 2024-09-16T00:31:43.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.911+0000 7fad3c20d640 1 --2- 172.21.15.33:0/4278284967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad34106f10 0x7fad3410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:43.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.911+0000 7fad3c20d640 1 --2- 172.21.15.33:0/4278284967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad34102790 0x7fad341067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:43.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.911+0000 7fad3c20d640 1 --2- 172.21.15.33:0/4278284967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad34101de0 0x7fad341021c0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:43.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.911+0000 7fad3c20d640 1 -- 172.21.15.33:0/4278284967 >> 172.21.15.33:0/4278284967 conn(0x7fad340fd8c0 msgr2=0x7fad340ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:43.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.911+0000 7fad3c20d640 1 -- 172.21.15.33:0/4278284967 shutdown_connections 2024-09-16T00:31:43.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.911+0000 7fad3c20d640 1 -- 172.21.15.33:0/4278284967 wait complete. 2024-09-16T00:31:43.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad3c20d640 1 Processor -- start 2024-09-16T00:31:43.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad3c20d640 1 -- start start 2024-09-16T00:31:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad3c20d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad34101de0 0x7fad34198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad3c20d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad34102790 0x7fad34198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad3c20d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad34106f10 0x7fad3419cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad3c20d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fad34111f30 con 0x7fad34106f10 2024-09-16T00:31:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad3c20d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fad34111db0 con 0x7fad34102790 2024-09-16T00:31:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad3c20d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fad341120b0 con 0x7fad34101de0 2024-09-16T00:31:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad3a7aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad34106f10 0x7fad3419cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad3a7aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad34106f10 0x7fad3419cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37858/0 (socket says 172.21.15.33:37858) 2024-09-16T00:31:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad3a7aa640 1 -- 172.21.15.33:0/3434974421 learned_addr learned my addr 172.21.15.33:0/3434974421 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad39fa9640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad34101de0 0x7fad34198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.912+0000 7fad397a8640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad34102790 0x7fad34198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.913+0000 7fad3a7aa640 1 -- 172.21.15.33:0/3434974421 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad34101de0 msgr2=0x7fad34198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.913+0000 7fad3a7aa640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad34101de0 0x7fad34198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.913+0000 7fad3a7aa640 1 -- 172.21.15.33:0/3434974421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad34102790 msgr2=0x7fad34198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.913+0000 7fad3a7aa640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad34102790 0x7fad34198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.913+0000 7fad3a7aa640 1 -- 172.21.15.33:0/3434974421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad3419d2c0 con 0x7fad34106f10 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.913+0000 7fad3a7aa640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad34106f10 0x7fad3419cbe0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00cce0 tx=0x7fad2c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.913+0000 7fad2affd640 1 -- 172.21.15.33:0/3434974421 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2c00f450 con 0x7fad34106f10 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.913+0000 7fad2affd640 1 -- 172.21.15.33:0/3434974421 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad2c00fad0 con 0x7fad34106f10 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.913+0000 7fad3c20d640 1 -- 172.21.15.33:0/3434974421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad3419d5b0 con 0x7fad34106f10 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.913+0000 7fad2affd640 1 -- 172.21.15.33:0/3434974421 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2c018b60 con 0x7fad34106f10 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.913+0000 7fad3c20d640 1 -- 172.21.15.33:0/3434974421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad341a4e90 con 0x7fad34106f10 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.914+0000 7fad3c20d640 1 -- 172.21.15.33:0/3434974421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad341021c0 con 0x7fad34106f10 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.915+0000 7fad2affd640 1 -- 172.21.15.33:0/3434974421 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fad2c00fc80 con 0x7fad34106f10 2024-09-16T00:31:43.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.916+0000 7fad2affd640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fad200a3240 0x7fad200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:43.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.916+0000 7fad39fa9640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fad200a3240 0x7fad200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:43.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.916+0000 7fad2affd640 1 -- 172.21.15.33:0/3434974421 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fad2c0d05a0 con 0x7fad34106f10 2024-09-16T00:31:43.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.917+0000 7fad39fa9640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fad200a3240 0x7fad200a5700 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fad30000c00 tx=0x7fad300058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:43.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:43.919+0000 7fad2affd640 1 -- 172.21.15.33:0/3434974421 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fad2c014030 con 0x7fad34106f10 2024-09-16T00:31:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:43 smithi060 ceph-mon[35185]: pgmap v787: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.4 KiB/s wr, 9 op/s 2024-09-16T00:31:44.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 18.30 MB/sec warmup 50 sec latency 23731.790 ms 2024-09-16T00:31:44.129 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:31:44.129 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:31:44.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.134+0000 7fad3c20d640 1 -- 172.21.15.33:0/3434974421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad341a50c0 con 0x7fad34106f10 2024-09-16T00:31:44.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.135+0000 7fad2affd640 1 -- 172.21.15.33:0/3434974421 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11906 (secure 0 0 0) 0x7fad2c09ce40 con 0x7fad34106f10 2024-09-16T00:31:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:31:44.135 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":31,"btime":"2024-09-16T00:30:57:641642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:30:57.027530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:31:44.136 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 31 2024-09-16T00:31:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.137+0000 7fad3c20d640 1 -- 172.21.15.33:0/3434974421 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fad200a3240 msgr2=0x7fad200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.137+0000 7fad3c20d640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fad200a3240 0x7fad200a5700 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fad30000c00 tx=0x7fad300058d0 comp rx=0 tx=0).stop 2024-09-16T00:31:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.137+0000 7fad3c20d640 1 -- 172.21.15.33:0/3434974421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad34106f10 msgr2=0x7fad3419cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.137+0000 7fad3c20d640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad34106f10 0x7fad3419cbe0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00cce0 tx=0x7fad2c00a620 comp rx=0 tx=0).stop 2024-09-16T00:31:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.137+0000 7fad3c20d640 1 -- 172.21.15.33:0/3434974421 shutdown_connections 2024-09-16T00:31:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.137+0000 7fad3c20d640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fad200a3240 0x7fad200a5700 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.137+0000 7fad3c20d640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad34106f10 0x7fad3419cbe0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.137+0000 7fad3c20d640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad34102790 0x7fad34198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.137+0000 7fad3c20d640 1 --2- 172.21.15.33:0/3434974421 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad34101de0 0x7fad34198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.137+0000 7fad3c20d640 1 -- 172.21.15.33:0/3434974421 >> 172.21.15.33:0/3434974421 conn(0x7fad340fd8c0 msgr2=0x7fad340ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.138+0000 7fad3c20d640 1 -- 172.21.15.33:0/3434974421 shutdown_connections 2024-09-16T00:31:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:44.138+0000 7fad3c20d640 1 -- 172.21.15.33:0/3434974421 wait complete. 2024-09-16T00:31:44.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:44 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3434974421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:31:44.886 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 8f702410 --release --await' 2024-09-16T00:31:44.886 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 8f702410 --release --await 2024-09-16T00:31:44.904 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:44.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:44 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3434974421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:31:44.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:44 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3434974421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:31:45.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 17.94 MB/sec warmup 51 sec latency 24731.835 ms 2024-09-16T00:31:45.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.433+0000 7f6667566640 1 -- 172.21.15.33:0/2231120978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6660106a80 msgr2=0x7f6660106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:45.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.433+0000 7f6667566640 1 --2- 172.21.15.33:0/2231120978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6660106a80 0x7f6660106e60 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f665c009930 tx=0x7f665c02a420 comp rx=0 tx=0).stop 2024-09-16T00:31:45.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.433+0000 7f6667566640 1 -- 172.21.15.33:0/2231120978 shutdown_connections 2024-09-16T00:31:45.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.433+0000 7f6667566640 1 --2- 172.21.15.33:0/2231120978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6660101420 0x7f666010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.433+0000 7f6667566640 1 --2- 172.21.15.33:0/2231120978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6660100a80 0x7f6660100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.433+0000 7f6667566640 1 --2- 172.21.15.33:0/2231120978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6660106a80 0x7f6660106e60 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.433+0000 7f6667566640 1 -- 172.21.15.33:0/2231120978 >> 172.21.15.33:0/2231120978 conn(0x7f66600fc7a0 msgr2=0x7f66600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:45.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.433+0000 7f6667566640 1 -- 172.21.15.33:0/2231120978 shutdown_connections 2024-09-16T00:31:45.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.433+0000 7f6667566640 1 -- 172.21.15.33:0/2231120978 wait complete. 2024-09-16T00:31:45.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6667566640 1 Processor -- start 2024-09-16T00:31:45.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6667566640 1 -- start start 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6667566640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6660100a80 0x7f6660197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6667566640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6660101420 0x7f6660197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6667566640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6660106a80 0x7f6660197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6667566640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6660110e80 con 0x7f6660100a80 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6667566640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6660110d00 con 0x7f6660106a80 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6667566640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6660111000 con 0x7f6660101420 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6665302640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6660100a80 0x7f6660197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6665302640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6660100a80 0x7f6660197110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37872/0 (socket says 172.21.15.33:37872) 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6665302640 1 -- 172.21.15.33:0/3121850260 learned_addr learned my addr 172.21.15.33:0/3121850260 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6664b01640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6660101420 0x7f6660197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6665b03640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6660106a80 0x7f6660197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6665302640 1 -- 172.21.15.33:0/3121850260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6660101420 msgr2=0x7f6660197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:45.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6665302640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6660101420 0x7f6660197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6665302640 1 -- 172.21.15.33:0/3121850260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6660106a80 msgr2=0x7f6660197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:45.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6665302640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6660106a80 0x7f6660197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.434+0000 7f6665302640 1 -- 172.21.15.33:0/3121850260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66601a6980 con 0x7f6660100a80 2024-09-16T00:31:45.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.435+0000 7f6665b03640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6660106a80 0x7f6660197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:45.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.435+0000 7f6665302640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6660100a80 0x7f6660197110 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f665c009230 tx=0x7f665c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:45.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.435+0000 7f66567fc640 1 -- 172.21.15.33:0/3121850260 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c03f030 con 0x7f6660100a80 2024-09-16T00:31:45.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.435+0000 7f66567fc640 1 -- 172.21.15.33:0/3121850260 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f665c004440 con 0x7f6660100a80 2024-09-16T00:31:45.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.435+0000 7f6667566640 1 -- 172.21.15.33:0/3121850260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66601a6c10 con 0x7f6660100a80 2024-09-16T00:31:45.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.435+0000 7f66567fc640 1 -- 172.21.15.33:0/3121850260 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c035070 con 0x7f6660100a80 2024-09-16T00:31:45.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.435+0000 7f6667566640 1 -- 172.21.15.33:0/3121850260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66601a7140 con 0x7f6660100a80 2024-09-16T00:31:45.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.437+0000 7f66567fc640 1 -- 172.21.15.33:0/3121850260 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f665c042030 con 0x7f6660100a80 2024-09-16T00:31:45.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.438+0000 7f666993f740 1 Processor -- start 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.438+0000 7f666993f740 1 -- start start 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.438+0000 7f666993f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d77b60 0x562533d77f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.438+0000 7f666993f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d78480 0x562533d82800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.438+0000 7f666993f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d82d40 0x562533d85130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.438+0000 7f66567fc640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f664c0a32c0 0x7f664c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.438+0000 7f666993f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562533cdd280 con 0x562533d82d40 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.438+0000 7f666993f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562533c84b10 con 0x562533d78480 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.438+0000 7f666993f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562533cd9240 con 0x562533d77b60 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.439+0000 7f6664b01640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d82d40 0x562533d85130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.439+0000 7f6664b01640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d82d40 0x562533d85130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37878/0 (socket says 172.21.15.33:37878) 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.439+0000 7f6664b01640 1 -- 172.21.15.33:0/3214905206 learned_addr learned my addr 172.21.15.33:0/3214905206 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.439+0000 7f66567fc640 1 -- 172.21.15.33:0/3121850260 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f665c0efe60 con 0x7f6660100a80 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.439+0000 7f6665b03640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f664c0a32c0 0x7f664c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.439+0000 7f6665b03640 1 --2- 172.21.15.33:0/3214905206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d77b60 0x562533d77f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.439+0000 7f6665302640 1 --2- 172.21.15.33:0/3214905206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d78480 0x562533d82800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.439+0000 7f6664b01640 1 -- 172.21.15.33:0/3214905206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d77b60 msgr2=0x562533d77f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:45.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.439+0000 7f6664b01640 1 --2- 172.21.15.33:0/3214905206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d77b60 0x562533d77f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.439+0000 7f6664b01640 1 -- 172.21.15.33:0/3214905206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d78480 msgr2=0x562533d82800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:45.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.439+0000 7f6664b01640 1 --2- 172.21.15.33:0/3214905206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d78480 0x562533d82800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.439+0000 7f6664b01640 1 -- 172.21.15.33:0/3214905206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562533c911f0 con 0x562533d82d40 2024-09-16T00:31:45.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.440+0000 7f6664b01640 1 --2- 172.21.15.33:0/3214905206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d82d40 0x562533d85130 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f6650009c30 tx=0x7f66500117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11a20d5105d3f6c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:45.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.440+0000 7f664b7fe640 1 -- 172.21.15.33:0/3214905206 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6650012450 con 0x562533d82d40 2024-09-16T00:31:45.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.440+0000 7f664b7fe640 1 -- 172.21.15.33:0/3214905206 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6650012ad0 con 0x562533d82d40 2024-09-16T00:31:45.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.440+0000 7f664b7fe640 1 -- 172.21.15.33:0/3214905206 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665000fbd0 con 0x562533d82d40 2024-09-16T00:31:45.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.441+0000 7f6665b03640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f664c0a32c0 0x7f664c0a5780 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6658005fd0 tx=0x7f6658009db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:45.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.441+0000 7f666993f740 1 -- 172.21.15.33:0/3214905206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d82d40 msgr2=0x562533d85130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.441+0000 7f666993f740 1 --2- 172.21.15.33:0/3214905206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d82d40 0x562533d85130 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f6650009c30 tx=0x7f66500117b0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.441+0000 7f666993f740 1 -- 172.21.15.33:0/3214905206 shutdown_connections 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.441+0000 7f666993f740 1 --2- 172.21.15.33:0/3214905206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d82d40 0x562533d85130 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.441+0000 7f666993f740 1 --2- 172.21.15.33:0/3214905206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d78480 0x562533d82800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.441+0000 7f666993f740 1 --2- 172.21.15.33:0/3214905206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d77b60 0x562533d77f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.441+0000 7f666993f740 1 -- 172.21.15.33:0/3214905206 >> 172.21.15.33:0/3214905206 conn(0x562533d740f0 msgr2=0x562533d744d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.441+0000 7f666993f740 1 -- 172.21.15.33:0/3214905206 shutdown_connections 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.441+0000 7f666993f740 1 -- 172.21.15.33:0/3214905206 wait complete. 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.442+0000 7f666993f740 1 -- start start 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.442+0000 7f666993f740 1 Processor -- start 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f666993f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d7fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f666993f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 0x562533d800c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f666993f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d80600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f6665302640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d7fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f6665302640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d7fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37888/0 (socket says 172.21.15.33:37888) 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f6665302640 1 -- 172.21.15.33:0/324511061 learned_addr learned my addr 172.21.15.33:0/324511061 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f666993f740 1 -- 172.21.15.33:0/324511061 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562533d85960 con 0x562533d77b60 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f666993f740 1 -- 172.21.15.33:0/324511061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562533c97f10 con 0x562533d82d40 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f666993f740 1 -- 172.21.15.33:0/324511061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562533d85ae0 con 0x562533d78480 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f6665b03640 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d80600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f666993f740 10 client.0 _ll_register_callbacks cb 0x562533c60a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f6665b03640 1 -- 172.21.15.33:0/324511061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 msgr2=0x562533d800c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f6665b03640 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 0x562533d800c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f6665b03640 1 -- 172.21.15.33:0/324511061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 msgr2=0x562533d7fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f6665b03640 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d7fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f6665b03640 1 -- 172.21.15.33:0/324511061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562533a4e6a0 con 0x562533d82d40 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.443+0000 7f6665302640 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d7fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.444+0000 7f6665b03640 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d80600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6658012fd0 tx=0x7f6658009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=226efa57be2fca69 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.444+0000 7f6648ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.444+0000 7f6648ff9640 1 -- 172.21.15.33:0/324511061 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6658011910 con 0x562533d82d40 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.444+0000 7f6648ff9640 1 -- 172.21.15.33:0/324511061 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6658011ab0 con 0x562533d82d40 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.444+0000 7f666993f740 1 -- 172.21.15.33:0/324511061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6650012ad0 con 0x562533d82d40 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.444+0000 7f6648ff9640 1 -- 172.21.15.33:0/324511061 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6658011cb0 con 0x562533d82d40 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.444+0000 7f666993f740 10 client.27512 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.444+0000 7f666993f740 1 -- 172.21.15.33:0/324511061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562533d914d0 con 0x562533d82d40 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.444+0000 7f6648ff9640 1 -- 172.21.15.33:0/324511061 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6650012ad0 con 0x562533d82d40 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.445+0000 7f6648ff9640 1 -- 172.21.15.33:0/324511061 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f665801f050 con 0x562533d82d40 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.445+0000 7f666993f740 10 client.27512 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:45.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.445+0000 7f666993f740 10 client.27512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.445+0000 7f666993f740 20 client.27512 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.445+0000 7f666993f740 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562533d92a30 0x562533d94ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.445+0000 7f666993f740 4 client.27512 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.445+0000 7f666993f740 1 -- 172.21.15.33:0/324511061 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562533cea7f0 con 0x562533d92a30 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.445+0000 7f6665302640 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562533d92a30 0x562533d94ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.446+0000 7f6665302640 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562533d92a30 0x562533d94ec0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cc26567aa169bc0 server_cookie=ee9d4abd76ba5a2c in_seq=0 out_seq=0 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.446+0000 7f6648ff9640 10 client.27512 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.446+0000 7f6648ff9640 1 -- 172.21.15.33:0/324511061 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562533cea7f0 con 0x562533d92a30 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.446+0000 7f6648ff9640 10 client.27512 handle_command_reply: tid=0 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.446+0000 7f666993f740 1 client.27512 shutdown 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.447+0000 7f666993f740 1 -- 172.21.15.33:0/324511061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 msgr2=0x562533d80600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.447+0000 7f666993f740 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d80600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6658012fd0 tx=0x7f6658009500 comp rx=0 tx=0).stop 2024-09-16T00:31:45.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.447+0000 7f666993f740 1 -- 172.21.15.33:0/324511061 shutdown_connections 2024-09-16T00:31:45.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.447+0000 7f666993f740 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562533d92a30 0x562533d94ec0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.447+0000 7f666993f740 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d80600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.447+0000 7f666993f740 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 0x562533d800c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.447+0000 7f666993f740 1 --2- 172.21.15.33:0/324511061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d7fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.447+0000 7f666993f740 1 -- 172.21.15.33:0/324511061 >> 172.21.15.33:0/324511061 conn(0x562533d740f0 msgr2=0x562533d79dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:45.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.447+0000 7f6648ff9640 0 client.27512 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:45.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.447+0000 7f666993f740 1 -- 172.21.15.33:0/324511061 shutdown_connections 2024-09-16T00:31:45.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.447+0000 7f666993f740 1 -- 172.21.15.33:0/324511061 wait complete. 2024-09-16T00:31:45.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.447+0000 7f666993f740 20 client.27512 trim_cache size 0 max 16384 2024-09-16T00:31:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.465+0000 7f666993f740 1 Processor -- start 2024-09-16T00:31:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.465+0000 7f666993f740 1 -- start start 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.465+0000 7f666993f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d9a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.465+0000 7f666993f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 0x562533d9acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.465+0000 7f666993f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d7cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.465+0000 7f666993f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f66601b3c00 con 0x562533d77b60 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.465+0000 7f666993f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f66601b1470 con 0x562533d82d40 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.465+0000 7f666993f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f66601af2c0 con 0x562533d78480 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.466+0000 7f6665302640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d9a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.466+0000 7f6665302640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d9a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37894/0 (socket says 172.21.15.33:37894) 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.466+0000 7f6665302640 1 -- 172.21.15.33:0/2677276158 learned_addr learned my addr 172.21.15.33:0/2677276158 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.466+0000 7f6664b01640 1 --2- 172.21.15.33:0/2677276158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 0x562533d9acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.466+0000 7f6665b03640 1 --2- 172.21.15.33:0/2677276158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d7cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.466+0000 7f6665302640 1 -- 172.21.15.33:0/2677276158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 msgr2=0x562533d9acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:45.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.466+0000 7f6665302640 1 --2- 172.21.15.33:0/2677276158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 0x562533d9acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.466+0000 7f6665302640 1 -- 172.21.15.33:0/2677276158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 msgr2=0x562533d7cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.466+0000 7f6665302640 1 --2- 172.21.15.33:0/2677276158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d7cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.466+0000 7f6665302640 1 -- 172.21.15.33:0/2677276158 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562533d73f20 con 0x562533d77b60 2024-09-16T00:31:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.466+0000 7f6665302640 1 --2- 172.21.15.33:0/2677276158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d9a790 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f665c0bbf30 tx=0x7f665c0ef040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aecca14f3dc8cef server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.466+0000 7f664affd640 1 -- 172.21.15.33:0/2677276158 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c049040 con 0x562533d77b60 2024-09-16T00:31:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.467+0000 7f664affd640 1 -- 172.21.15.33:0/2677276158 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f665c0c0350 con 0x562533d77b60 2024-09-16T00:31:45.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.467+0000 7f664affd640 1 -- 172.21.15.33:0/2677276158 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c0bc200 con 0x562533d77b60 2024-09-16T00:31:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.467+0000 7f666993f740 1 -- 172.21.15.33:0/2677276158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 msgr2=0x562533d9a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.467+0000 7f666993f740 1 --2- 172.21.15.33:0/2677276158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d9a790 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f665c0bbf30 tx=0x7f665c0ef040 comp rx=0 tx=0).stop 2024-09-16T00:31:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.468+0000 7f666993f740 1 -- 172.21.15.33:0/2677276158 shutdown_connections 2024-09-16T00:31:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.468+0000 7f666993f740 1 --2- 172.21.15.33:0/2677276158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d7cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.468+0000 7f666993f740 1 --2- 172.21.15.33:0/2677276158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 0x562533d9acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.468+0000 7f666993f740 1 --2- 172.21.15.33:0/2677276158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d9a790 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.468+0000 7f666993f740 1 -- 172.21.15.33:0/2677276158 >> 172.21.15.33:0/2677276158 conn(0x562533d740f0 msgr2=0x562533d79dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.468+0000 7f666993f740 1 -- 172.21.15.33:0/2677276158 shutdown_connections 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.468+0000 7f666993f740 1 -- 172.21.15.33:0/2677276158 wait complete. 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.468+0000 7f666993f740 1 -- start start 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.468+0000 7f666993f740 1 Processor -- start 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f666993f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d8afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f666993f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 0x562533d8b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f666993f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d85d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f666993f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f66601af930 con 0x562533d77b60 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f6665302640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d8afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f6665302640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d8afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37910/0 (socket says 172.21.15.33:37910) 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f6665302640 1 -- 172.21.15.33:0/1463367149 learned_addr learned my addr 172.21.15.33:0/1463367149 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f666993f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f66601b0da0 con 0x562533d82d40 2024-09-16T00:31:45.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f666993f740 1 -- 172.21.15.33:0/1463367149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f66601b0000 con 0x562533d78480 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f666993f740 10 client.0 _ll_register_callbacks cb 0x7f66580098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f6664b01640 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 0x562533d8b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f6665b03640 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d85d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f6665302640 1 -- 172.21.15.33:0/1463367149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 msgr2=0x562533d8b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f6665302640 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 0x562533d8b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f6665302640 1 -- 172.21.15.33:0/1463367149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 msgr2=0x562533d85d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f6665302640 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d85d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f6665302640 1 -- 172.21.15.33:0/1463367149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562533d86580 con 0x562533d77b60 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.469+0000 7f6665302640 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d8afb0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f665c000c00 tx=0x7f665c0ed3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18822a99b1a1880 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f6648ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f6648ff9640 1 -- 172.21.15.33:0/1463367149 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c0c4e60 con 0x562533d77b60 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f6648ff9640 1 -- 172.21.15.33:0/1463367149 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f665c0b8610 con 0x562533d77b60 2024-09-16T00:31:45.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f666993f740 1 -- 172.21.15.33:0/1463367149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f665c0c0350 con 0x562533d77b60 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f6648ff9640 1 -- 172.21.15.33:0/1463367149 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f665c049030 con 0x562533d77b60 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f666993f740 10 client.18177 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f666993f740 1 -- 172.21.15.33:0/1463367149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562533d86580 con 0x562533d77b60 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f6648ff9640 1 -- 172.21.15.33:0/1463367149 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f665c0c0350 con 0x562533d77b60 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f6648ff9640 1 -- 172.21.15.33:0/1463367149 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f665c0a3070 con 0x562533d77b60 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f666993f740 10 client.18177 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f666993f740 10 client.18177 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f666993f740 20 client.18177 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f666993f740 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562533dfe610 0x562533e00a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f666993f740 4 client.18177 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.470+0000 7f666993f740 1 -- 172.21.15.33:0/1463367149 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562533ce4d00 con 0x562533dfe610 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.471+0000 7f6664b01640 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562533dfe610 0x562533e00a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.471+0000 7f6648ff9640 10 client.18177 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:45.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.471+0000 7f6664b01640 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562533dfe610 0x562533e00a00 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b22776d7313b2314 server_cookie=7740c2741c3b5419 in_seq=0 out_seq=0 2024-09-16T00:31:45.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.472+0000 7f6648ff9640 1 -- 172.21.15.33:0/1463367149 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562533ce4d00 con 0x562533dfe610 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.472+0000 7f6648ff9640 10 client.18177 handle_command_reply: tid=0 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.472+0000 7f666993f740 1 client.18177 shutdown 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.473+0000 7f666993f740 1 -- 172.21.15.33:0/1463367149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 msgr2=0x562533d8afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.473+0000 7f666993f740 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d8afb0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f665c000c00 tx=0x7f665c0ed3c0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.473+0000 7f666993f740 1 -- 172.21.15.33:0/1463367149 shutdown_connections 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.473+0000 7f666993f740 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562533dfe610 0x562533e00a00 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.473+0000 7f666993f740 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562533d82d40 0x562533d85d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.473+0000 7f666993f740 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562533d78480 0x562533d8b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.473+0000 7f666993f740 1 --2- 172.21.15.33:0/1463367149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562533d77b60 0x562533d8afb0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.473+0000 7f666993f740 1 -- 172.21.15.33:0/1463367149 >> 172.21.15.33:0/1463367149 conn(0x562533d740f0 msgr2=0x562533d795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.473+0000 7f6648ff9640 0 client.18177 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.473+0000 7f666993f740 1 -- 172.21.15.33:0/1463367149 shutdown_connections 2024-09-16T00:31:45.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.473+0000 7f666993f740 1 -- 172.21.15.33:0/1463367149 wait complete. 2024-09-16T00:31:45.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.473+0000 7f666993f740 20 client.18177 trim_cache size 0 max 16384 2024-09-16T00:31:45.478 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:31:45.478 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:31:45.478 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "5a1869ce-cd48-42b9-99a8-63e821b640b8", 2024-09-16T00:31:45.478 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:31:45.478 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:31:45.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.474+0000 7f6667566640 1 -- 172.21.15.33:0/3121850260 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f664c0a32c0 msgr2=0x7f664c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:45.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.474+0000 7f6667566640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f664c0a32c0 0x7f664c0a5780 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6658005fd0 tx=0x7f6658009db0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.474+0000 7f6667566640 1 -- 172.21.15.33:0/3121850260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6660100a80 msgr2=0x7f6660197110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:45.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.474+0000 7f6667566640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6660100a80 0x7f6660197110 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f665c009230 tx=0x7f665c004040 comp rx=0 tx=0).stop 2024-09-16T00:31:45.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.474+0000 7f6667566640 1 -- 172.21.15.33:0/3121850260 shutdown_connections 2024-09-16T00:31:45.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.474+0000 7f6667566640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f664c0a32c0 0x7f664c0a5780 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.474+0000 7f6667566640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6660106a80 0x7f6660197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.474+0000 7f6667566640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6660101420 0x7f6660197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.474+0000 7f6667566640 1 --2- 172.21.15.33:0/3121850260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6660100a80 0x7f6660197110 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:45.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.474+0000 7f6667566640 1 -- 172.21.15.33:0/3121850260 >> 172.21.15.33:0/3121850260 conn(0x7f66600fc7a0 msgr2=0x7f66600fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:45.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.474+0000 7f6667566640 1 -- 172.21.15.33:0/3121850260 shutdown_connections 2024-09-16T00:31:45.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:45.474+0000 7f6667566640 1 -- 172.21.15.33:0/3121850260 wait complete. 2024-09-16T00:31:45.480 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:45.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:45 smithi033 ceph-mon[31243]: pgmap v788: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 61 KiB/s wr, 10 op/s 2024-09-16T00:31:45.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:45 smithi083 ceph-mon[35242]: pgmap v788: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 61 KiB/s wr, 10 op/s 2024-09-16T00:31:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:45 smithi060 ceph-mon[35185]: pgmap v788: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 61 KiB/s wr, 10 op/s 2024-09-16T00:31:46.067 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13834 17.60 MB/sec warmup 52 sec latency 25731.921 ms 2024-09-16T00:31:46.223 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:31:46.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:31:46.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:46.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:46.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:46.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:46.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:46.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:46.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:46.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:46.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:31:46.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.781+0000 7fcbba18c640 1 -- 172.21.15.33:0/4186084230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb41002f0 msgr2=0x7fcbb41006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:46.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.781+0000 7fcbba18c640 1 --2- 172.21.15.33:0/4186084230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb41002f0 0x7fcbb41006d0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009930 tx=0x7fcba402a420 comp rx=0 tx=0).stop 2024-09-16T00:31:46.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.781+0000 7fcbba18c640 1 -- 172.21.15.33:0/4186084230 shutdown_connections 2024-09-16T00:31:46.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.781+0000 7fcbba18c640 1 --2- 172.21.15.33:0/4186084230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb4106f50 0x7fcbb410eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.781+0000 7fcbba18c640 1 --2- 172.21.15.33:0/4186084230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbb4100ca0 0x7fcbb4106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.781+0000 7fcbba18c640 1 --2- 172.21.15.33:0/4186084230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb41002f0 0x7fcbb41006d0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.781+0000 7fcbba18c640 1 -- 172.21.15.33:0/4186084230 >> 172.21.15.33:0/4186084230 conn(0x7fcbb4077210 msgr2=0x7fcbb40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:46.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.781+0000 7fcbba18c640 1 -- 172.21.15.33:0/4186084230 shutdown_connections 2024-09-16T00:31:46.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.781+0000 7fcbba18c640 1 -- 172.21.15.33:0/4186084230 wait complete. 2024-09-16T00:31:46.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.781+0000 7fcbba18c640 1 Processor -- start 2024-09-16T00:31:46.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbba18c640 1 -- start start 2024-09-16T00:31:46.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbba18c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbb41002f0 0x7fcbb4198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbba18c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb4100ca0 0x7fcbb4198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbba18c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb4106f50 0x7fcbb419ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbba18c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb4112000 con 0x7fcbb4100ca0 2024-09-16T00:31:46.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbba18c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb4111e80 con 0x7fcbb4106f50 2024-09-16T00:31:46.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbba18c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb4112180 con 0x7fcbb41002f0 2024-09-16T00:31:46.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbb2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb4100ca0 0x7fcbb4198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbb2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb4100ca0 0x7fcbb4198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37934/0 (socket says 172.21.15.33:37934) 2024-09-16T00:31:46.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbb2ffd640 1 -- 172.21.15.33:0/3566280225 learned_addr learned my addr 172.21.15.33:0/3566280225 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:46.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbb37fe640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbb41002f0 0x7fcbb4198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb4106f50 0x7fcbb419ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbb2ffd640 1 -- 172.21.15.33:0/3566280225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbb41002f0 msgr2=0x7fcbb4198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:46.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbb2ffd640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbb41002f0 0x7fcbb4198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbb2ffd640 1 -- 172.21.15.33:0/3566280225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb4106f50 msgr2=0x7fcbb419ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:46.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbb2ffd640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb4106f50 0x7fcbb419ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.782+0000 7fcbb2ffd640 1 -- 172.21.15.33:0/3566280225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb419d510 con 0x7fcbb4100ca0 2024-09-16T00:31:46.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.783+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb4106f50 0x7fcbb419ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:46.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.783+0000 7fcbb2ffd640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb4100ca0 0x7fcbb4198aa0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fcba800b9c0 tx=0x7fcba800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:46.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.783+0000 7fcbb0ff9640 1 -- 172.21.15.33:0/3566280225 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba8019070 con 0x7fcbb4100ca0 2024-09-16T00:31:46.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.783+0000 7fcbba18c640 1 -- 172.21.15.33:0/3566280225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb419d800 con 0x7fcbb4100ca0 2024-09-16T00:31:46.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.783+0000 7fcbb0ff9640 1 -- 172.21.15.33:0/3566280225 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba800cba0 con 0x7fcbb4100ca0 2024-09-16T00:31:46.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.783+0000 7fcbba18c640 1 -- 172.21.15.33:0/3566280225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb41a50e0 con 0x7fcbb4100ca0 2024-09-16T00:31:46.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.783+0000 7fcbb0ff9640 1 -- 172.21.15.33:0/3566280225 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba8023060 con 0x7fcbb4100ca0 2024-09-16T00:31:46.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.785+0000 7fcbb0ff9640 1 -- 172.21.15.33:0/3566280225 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fcba801c070 con 0x7fcbb4100ca0 2024-09-16T00:31:46.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.785+0000 7fcbbc565740 1 Processor -- start 2024-09-16T00:31:46.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.785+0000 7fcbbc565740 1 -- start start 2024-09-16T00:31:46.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbbc565740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9856f40 0x557aa9857320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbbc565740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9857860 0x557aa9866110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbbc565740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9866650 0x557aa9868a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbbc565740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557aa97bc620 con 0x557aa9856f40 2024-09-16T00:31:46.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbbc565740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x557aa9763e50 con 0x557aa9866650 2024-09-16T00:31:46.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbbc565740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557aa97b85e0 con 0x557aa9857860 2024-09-16T00:31:46.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9856f40 0x557aa9857320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9856f40 0x557aa9857320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37938/0 (socket says 172.21.15.33:37938) 2024-09-16T00:31:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb3fff640 1 -- 172.21.15.33:0/2557542767 learned_addr learned my addr 172.21.15.33:0/2557542767 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb0ff9640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcb9c0a3240 0x7fcb9c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb37fe640 1 --2- 172.21.15.33:0/2557542767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9857860 0x557aa9866110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb2ffd640 1 --2- 172.21.15.33:0/2557542767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9866650 0x557aa9868a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb3fff640 1 -- 172.21.15.33:0/2557542767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9857860 msgr2=0x557aa9866110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/2557542767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9857860 0x557aa9866110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb3fff640 1 -- 172.21.15.33:0/2557542767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9866650 msgr2=0x557aa9868a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/2557542767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9866650 0x557aa9868a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb3fff640 1 -- 172.21.15.33:0/2557542767 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557aa9754d40 con 0x557aa9856f40 2024-09-16T00:31:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb2ffd640 1 --2- 172.21.15.33:0/2557542767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9866650 0x557aa9868a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:46.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb0ff9640 1 -- 172.21.15.33:0/3566280225 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fcba80cf400 con 0x7fcbb4100ca0 2024-09-16T00:31:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.786+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcb9c0a3240 0x7fcb9c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.787+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/2557542767 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9856f40 0x557aa9857320 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00cd90 tx=0x7fcbac00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58eb6e692d8f48ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.787+0000 7fcb99ffb640 1 -- 172.21.15.33:0/2557542767 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac022070 con 0x557aa9856f40 2024-09-16T00:31:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.787+0000 7fcb99ffb640 1 -- 172.21.15.33:0/2557542767 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbac01c470 con 0x557aa9856f40 2024-09-16T00:31:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.787+0000 7fcb99ffb640 1 -- 172.21.15.33:0/2557542767 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac01a660 con 0x557aa9856f40 2024-09-16T00:31:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.787+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcb9c0a3240 0x7fcb9c0a5700 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcbac01f000 tx=0x7fcbac020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.787+0000 7fcbbc565740 1 -- 172.21.15.33:0/2557542767 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9856f40 msgr2=0x557aa9857320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.787+0000 7fcbbc565740 1 --2- 172.21.15.33:0/2557542767 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9856f40 0x557aa9857320 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00cd90 tx=0x7fcbac00e720 comp rx=0 tx=0).stop 2024-09-16T00:31:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 -- 172.21.15.33:0/2557542767 shutdown_connections 2024-09-16T00:31:46.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 --2- 172.21.15.33:0/2557542767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9866650 0x557aa9868a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 --2- 172.21.15.33:0/2557542767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9857860 0x557aa9866110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 --2- 172.21.15.33:0/2557542767 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9856f40 0x557aa9857320 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 -- 172.21.15.33:0/2557542767 >> 172.21.15.33:0/2557542767 conn(0x557aa98535f0 msgr2=0x557aa98539d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 -- 172.21.15.33:0/2557542767 shutdown_connections 2024-09-16T00:31:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 -- 172.21.15.33:0/2557542767 wait complete. 2024-09-16T00:31:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 -- start start 2024-09-16T00:31:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 Processor -- start 2024-09-16T00:31:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9856f40 0x557aa98635c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9857860 0x557aa9863b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9866650 0x557aa9864040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557aa9869220 con 0x557aa9866650 2024-09-16T00:31:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x557aa9744f10 con 0x557aa9857860 2024-09-16T00:31:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557aa98693a0 con 0x557aa9856f40 2024-09-16T00:31:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.788+0000 7fcbbc565740 10 client.0 _ll_register_callbacks cb 0x557aa9768db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbb3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9866650 0x557aa9864040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbb3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9866650 0x557aa9864040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37954/0 (socket says 172.21.15.33:37954) 2024-09-16T00:31:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbb3fff640 1 -- 172.21.15.33:0/1998359317 learned_addr learned my addr 172.21.15.33:0/1998359317 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbb37fe640 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9856f40 0x557aa98635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbb2ffd640 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9857860 0x557aa9863b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbb3fff640 1 -- 172.21.15.33:0/1998359317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9856f40 msgr2=0x557aa98635c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9856f40 0x557aa98635c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbb3fff640 1 -- 172.21.15.33:0/1998359317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9857860 msgr2=0x557aa9863b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9857860 0x557aa9863b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbb3fff640 1 -- 172.21.15.33:0/1998359317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557aa94cb580 con 0x557aa9866650 2024-09-16T00:31:46.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9866650 0x557aa9864040 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fcbac01c880 tx=0x7fcbac01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff312259e8e1a01 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:46.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcb92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:46.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcb92ffd640 1 -- 172.21.15.33:0/1998359317 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac022040 con 0x557aa9866650 2024-09-16T00:31:46.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcb92ffd640 1 -- 172.21.15.33:0/1998359317 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fcbac01a8c0 con 0x557aa9866650 2024-09-16T00:31:46.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbbc565740 1 -- 172.21.15.33:0/1998359317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbac01c470 con 0x557aa9866650 2024-09-16T00:31:46.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcb92ffd640 1 -- 172.21.15.33:0/1998359317 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcbac03a050 con 0x557aa9866650 2024-09-16T00:31:46.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbbc565740 10 client.18189 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:46.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.789+0000 7fcbbc565740 1 -- 172.21.15.33:0/1998359317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557aa9595960 con 0x557aa9866650 2024-09-16T00:31:46.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.790+0000 7fcb92ffd640 1 -- 172.21.15.33:0/1998359317 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbac01c470 con 0x557aa9866650 2024-09-16T00:31:46.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.790+0000 7fcb92ffd640 1 -- 172.21.15.33:0/1998359317 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fcbac019040 con 0x557aa9866650 2024-09-16T00:31:46.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.790+0000 7fcbbc565740 10 client.18189 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:46.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.790+0000 7fcbbc565740 10 client.18189 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:31:46.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.790+0000 7fcbbc565740 20 client.18189 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:46.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.790+0000 7fcbbc565740 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557aa98761c0 0x557aa98785b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.790+0000 7fcbbc565740 4 client.18189 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:31:46.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.790+0000 7fcbbc565740 1 -- 172.21.15.33:0/1998359317 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557aa97c9b90 con 0x557aa98761c0 2024-09-16T00:31:46.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.790+0000 7fcbb37fe640 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557aa98761c0 0x557aa98785b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.791+0000 7fcb92ffd640 10 client.18189 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.791+0000 7fcbb37fe640 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557aa98761c0 0x557aa98785b0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=743344a652f4cb11 server_cookie=cc1caeb82db60b0b in_seq=0 out_seq=0 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.791+0000 7fcb92ffd640 1 -- 172.21.15.33:0/1998359317 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557aa97c9b90 con 0x557aa98761c0 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.791+0000 7fcb92ffd640 10 client.18189 handle_command_reply: tid=0 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.791+0000 7fcbbc565740 1 client.18189 shutdown 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.791+0000 7fcbbc565740 1 -- 172.21.15.33:0/1998359317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9866650 msgr2=0x557aa9864040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.791+0000 7fcbbc565740 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9866650 0x557aa9864040 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fcbac01c880 tx=0x7fcbac01c8b0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.792+0000 7fcbbc565740 1 -- 172.21.15.33:0/1998359317 shutdown_connections 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.792+0000 7fcbbc565740 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557aa98761c0 0x557aa98785b0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.792+0000 7fcbbc565740 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9866650 0x557aa9864040 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.792+0000 7fcbbc565740 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9857860 0x557aa9863b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.792+0000 7fcbbc565740 1 --2- 172.21.15.33:0/1998359317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9856f40 0x557aa98635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.792+0000 7fcbbc565740 1 -- 172.21.15.33:0/1998359317 >> 172.21.15.33:0/1998359317 conn(0x557aa98535f0 msgr2=0x557aa9859350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:46.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.792+0000 7fcb92ffd640 0 client.18189 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:46.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.792+0000 7fcbbc565740 1 -- 172.21.15.33:0/1998359317 shutdown_connections 2024-09-16T00:31:46.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.792+0000 7fcbbc565740 1 -- 172.21.15.33:0/1998359317 wait complete. 2024-09-16T00:31:46.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.792+0000 7fcbbc565740 20 client.18189 trim_cache size 0 max 16384 2024-09-16T00:31:46.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.809+0000 7fcbbc565740 1 Processor -- start 2024-09-16T00:31:46.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.809+0000 7fcbbc565740 1 -- start start 2024-09-16T00:31:46.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbbc565740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9856f40 0x557aa9852640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbbc565740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9857860 0x557aa987dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbbc565740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9866650 0x557aa987e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbbc565740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41af8b0 con 0x557aa9857860 2024-09-16T00:31:46.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbbc565740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41ad120 con 0x557aa9856f40 2024-09-16T00:31:46.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbbc565740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41aaf70 con 0x557aa9866650 2024-09-16T00:31:46.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbb37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9856f40 0x557aa9852640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbb37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9856f40 0x557aa9852640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55738/0 (socket says 172.21.15.33:55738) 2024-09-16T00:31:46.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbb37fe640 1 -- 172.21.15.33:0/2302147351 learned_addr learned my addr 172.21.15.33:0/2302147351 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:46.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbb2ffd640 1 --2- 172.21.15.33:0/2302147351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9857860 0x557aa987dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/2302147351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9866650 0x557aa987e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbb37fe640 1 -- 172.21.15.33:0/2302147351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9866650 msgr2=0x557aa987e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbb37fe640 1 --2- 172.21.15.33:0/2302147351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9866650 0x557aa987e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbb37fe640 1 -- 172.21.15.33:0/2302147351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9857860 msgr2=0x557aa987dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbb37fe640 1 --2- 172.21.15.33:0/2302147351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9857860 0x557aa987dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.810+0000 7fcbb37fe640 1 -- 172.21.15.33:0/2302147351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557aa9865f30 con 0x557aa9856f40 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.811+0000 7fcbb37fe640 1 --2- 172.21.15.33:0/2302147351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9856f40 0x557aa9852640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcba402a940 tx=0x7fcba4005e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1ecfe0a148446c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.811+0000 7fcb997fa640 1 -- 172.21.15.33:0/2302147351 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba4035040 con 0x557aa9856f40 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.811+0000 7fcb997fa640 1 -- 172.21.15.33:0/2302147351 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba4005500 con 0x557aa9856f40 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.811+0000 7fcb997fa640 1 -- 172.21.15.33:0/2302147351 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba4004dd0 con 0x557aa9856f40 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.812+0000 7fcbbc565740 1 -- 172.21.15.33:0/2302147351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9856f40 msgr2=0x557aa9852640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.812+0000 7fcbbc565740 1 --2- 172.21.15.33:0/2302147351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9856f40 0x557aa9852640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcba402a940 tx=0x7fcba4005e20 comp rx=0 tx=0).stop 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.812+0000 7fcbbc565740 1 -- 172.21.15.33:0/2302147351 shutdown_connections 2024-09-16T00:31:46.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.812+0000 7fcbbc565740 1 --2- 172.21.15.33:0/2302147351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9866650 0x557aa987e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.812+0000 7fcbbc565740 1 --2- 172.21.15.33:0/2302147351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9857860 0x557aa987dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.812+0000 7fcbbc565740 1 --2- 172.21.15.33:0/2302147351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9856f40 0x557aa9852640 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.812+0000 7fcbbc565740 1 -- 172.21.15.33:0/2302147351 >> 172.21.15.33:0/2302147351 conn(0x557aa98535f0 msgr2=0x557aa9851f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.812+0000 7fcbbc565740 1 -- 172.21.15.33:0/2302147351 shutdown_connections 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.812+0000 7fcbbc565740 1 -- 172.21.15.33:0/2302147351 wait complete. 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.812+0000 7fcbbc565740 1 -- start start 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbbc565740 1 Processor -- start 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbbc565740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9856f40 0x557aa986a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbbc565740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9857860 0x557aa986ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbbc565740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9866650 0x557aa986b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbbc565740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41ab5e0 con 0x557aa9857860 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbbc565740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41aca50 con 0x557aa9866650 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbbc565740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41abcb0 con 0x557aa9856f40 2024-09-16T00:31:46.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbbc565740 10 client.0 _ll_register_callbacks cb 0x7fcbac00cbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbb2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9857860 0x557aa986ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbb2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9857860 0x557aa986ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37976/0 (socket says 172.21.15.33:37976) 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbb2ffd640 1 -- 172.21.15.33:0/1296135223 learned_addr learned my addr 172.21.15.33:0/1296135223 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbb37fe640 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9856f40 0x557aa986a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9866650 0x557aa986b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbb2ffd640 1 -- 172.21.15.33:0/1296135223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9856f40 msgr2=0x557aa986a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbb2ffd640 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9856f40 0x557aa986a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbb2ffd640 1 -- 172.21.15.33:0/1296135223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9866650 msgr2=0x557aa986b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbb2ffd640 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9866650 0x557aa986b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.813+0000 7fcbb2ffd640 1 -- 172.21.15.33:0/1296135223 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557aa98dff40 con 0x557aa9857860 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.814+0000 7fcbb3fff640 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9866650 0x557aa986b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.814+0000 7fcbb2ffd640 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9857860 0x557aa986ab10 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fcba8097a50 tx=0x7fcba80cc8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fad005c6f3234da server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:46.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.814+0000 7fcb92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:46.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.814+0000 7fcb92ffd640 1 -- 172.21.15.33:0/1296135223 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba80ccb30 con 0x557aa9857860 2024-09-16T00:31:46.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.814+0000 7fcb92ffd640 1 -- 172.21.15.33:0/1296135223 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fcba80a3d20 con 0x557aa9857860 2024-09-16T00:31:46.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.814+0000 7fcbbc565740 1 -- 172.21.15.33:0/1296135223 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcba4005500 con 0x557aa9857860 2024-09-16T00:31:46.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.814+0000 7fcb92ffd640 1 -- 172.21.15.33:0/1296135223 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcba80a3f20 con 0x557aa9857860 2024-09-16T00:31:46.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.814+0000 7fcbbc565740 10 client.18195 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:46.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.814+0000 7fcbbc565740 1 -- 172.21.15.33:0/1296135223 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557aa98e1310 con 0x557aa9857860 2024-09-16T00:31:46.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.814+0000 7fcb92ffd640 1 -- 172.21.15.33:0/1296135223 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba4005500 con 0x557aa9857860 2024-09-16T00:31:46.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.815+0000 7fcb92ffd640 1 -- 172.21.15.33:0/1296135223 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fcba809a4d0 con 0x557aa9857860 2024-09-16T00:31:46.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.815+0000 7fcbbc565740 10 client.18195 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:46.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.815+0000 7fcbbc565740 10 client.18195 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:31:46.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.815+0000 7fcbbc565740 20 client.18195 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:46.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.815+0000 7fcbbc565740 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557aa98e1990 0x557aa98e3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:46.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.815+0000 7fcbbc565740 4 client.18195 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "8f702410", "release": true, "await": true}] 2024-09-16T00:31:46.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.815+0000 7fcbbc565740 1 -- 172.21.15.33:0/1296135223 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "8f702410", "release": true, "await": true}) -- 0x557aa97c40a0 con 0x557aa98e1990 2024-09-16T00:31:46.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.815+0000 7fcbb37fe640 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557aa98e1990 0x557aa98e3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:46.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.815+0000 7fcbb37fe640 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557aa98e1990 0x557aa98e3d80 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9549fef405042f7 server_cookie=ad0c81e0ec166cbc in_seq=0 out_seq=0 2024-09-16T00:31:46.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.815+0000 7fcb92ffd640 10 client.18195 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:46.841 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:31:46.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.870+0000 7fcb92ffd640 1 -- 172.21.15.33:0/1296135223 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x557aa97c40a0 con 0x557aa98e1990 2024-09-16T00:31:46.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.870+0000 7fcb92ffd640 10 client.18195 handle_command_reply: tid=0 2024-09-16T00:31:46.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.870+0000 7fcbbc565740 1 client.18195 shutdown 2024-09-16T00:31:46.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.871+0000 7fcbbc565740 1 -- 172.21.15.33:0/1296135223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9857860 msgr2=0x557aa986ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:46.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.871+0000 7fcbbc565740 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9857860 0x557aa986ab10 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fcba8097a50 tx=0x7fcba80cc8f0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.871+0000 7fcbbc565740 1 -- 172.21.15.33:0/1296135223 shutdown_connections 2024-09-16T00:31:46.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.871+0000 7fcbbc565740 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557aa98e1990 0x557aa98e3d80 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.871+0000 7fcbbc565740 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557aa9866650 0x557aa986b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.871+0000 7fcbbc565740 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557aa9857860 0x557aa986ab10 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.871+0000 7fcbbc565740 1 --2- 172.21.15.33:0/1296135223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa9856f40 0x557aa986a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.871+0000 7fcbbc565740 1 -- 172.21.15.33:0/1296135223 >> 172.21.15.33:0/1296135223 conn(0x557aa98535f0 msgr2=0x557aa9852750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:46.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.871+0000 7fcb92ffd640 0 client.18195 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:46.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.871+0000 7fcbbc565740 1 -- 172.21.15.33:0/1296135223 shutdown_connections 2024-09-16T00:31:46.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.871+0000 7fcbbc565740 1 -- 172.21.15.33:0/1296135223 wait complete. 2024-09-16T00:31:46.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.871+0000 7fcbbc565740 20 client.18195 trim_cache size 0 max 16384 2024-09-16T00:31:46.873 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:31:46.874 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 31, 2024-09-16T00:31:46.874 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:31:46.874 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 52, 2024-09-16T00:31:46.874 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:31:46.874 INFO:teuthology.orchestra.run.smithi033.stdout: "8f702410": { 2024-09-16T00:31:46.874 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 52, 2024-09-16T00:31:46.874 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:31:46.874 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:31:46.875 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:31:46.875 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:31:46.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:31:46.875 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:31:46.875 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 146.5, 2024-09-16T00:31:46.875 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:31:46.875 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:31:46.875 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:31:46.875 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:31:46.876 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:31:46.876 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:31:46.876 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:46.876 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:46.876 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:46.876 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:46.876 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:46.876 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:31:46.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.872+0000 7fcbba18c640 1 -- 172.21.15.33:0/3566280225 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcb9c0a3240 msgr2=0x7fcb9c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:46.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.872+0000 7fcbba18c640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcb9c0a3240 0x7fcb9c0a5700 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcbac01f000 tx=0x7fcbac020000 comp rx=0 tx=0).stop 2024-09-16T00:31:46.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.872+0000 7fcbba18c640 1 -- 172.21.15.33:0/3566280225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb4100ca0 msgr2=0x7fcbb4198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:46.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.872+0000 7fcbba18c640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb4100ca0 0x7fcbb4198aa0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fcba800b9c0 tx=0x7fcba800be80 comp rx=0 tx=0).stop 2024-09-16T00:31:46.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.872+0000 7fcbba18c640 1 -- 172.21.15.33:0/3566280225 shutdown_connections 2024-09-16T00:31:46.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.872+0000 7fcbba18c640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcb9c0a3240 0x7fcb9c0a5700 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.872+0000 7fcbba18c640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb4106f50 0x7fcbb419ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.872+0000 7fcbba18c640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb4100ca0 0x7fcbb4198aa0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.872+0000 7fcbba18c640 1 --2- 172.21.15.33:0/3566280225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbb41002f0 0x7fcbb4198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:46.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.872+0000 7fcbba18c640 1 -- 172.21.15.33:0/3566280225 >> 172.21.15.33:0/3566280225 conn(0x7fcbb4077210 msgr2=0x7fcbb41051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:46.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.872+0000 7fcbba18c640 1 -- 172.21.15.33:0/3566280225 shutdown_connections 2024-09-16T00:31:46.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:46.872+0000 7fcbba18c640 1 -- 172.21.15.33:0/3566280225 wait complete. 2024-09-16T00:31:46.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:31:46.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:46.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:31:46.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:31:46.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:46.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:46.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:31:46.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:46.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:31:47.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 13912 17.41 MB/sec warmup 53 sec latency 26536.492 ms 2024-09-16T00:31:47.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.552+0000 7fc94effe640 1 -- 172.21.15.33:0/4017415782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9481082e0 msgr2=0x7fc94810ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:47.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.552+0000 7fc94effe640 1 --2- 172.21.15.33:0/4017415782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9481082e0 0x7fc94810ec50 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fc940009cd0 tx=0x7fc94002a8e0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.552+0000 7fc94effe640 1 -- 172.21.15.33:0/4017415782 shutdown_connections 2024-09-16T00:31:47.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.552+0000 7fc94effe640 1 --2- 172.21.15.33:0/4017415782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9481082e0 0x7fc94810ec50 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.552+0000 7fc94effe640 1 --2- 172.21.15.33:0/4017415782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc948103760 0x7fc948107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.552+0000 7fc94effe640 1 --2- 172.21.15.33:0/4017415782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc948102db0 0x7fc948103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.552+0000 7fc94effe640 1 -- 172.21.15.33:0/4017415782 >> 172.21.15.33:0/4017415782 conn(0x7fc9480fc7d0 msgr2=0x7fc9480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:47.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.552+0000 7fc94effe640 1 -- 172.21.15.33:0/4017415782 shutdown_connections 2024-09-16T00:31:47.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.552+0000 7fc94effe640 1 -- 172.21.15.33:0/4017415782 wait complete. 2024-09-16T00:31:47.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc94effe640 1 Processor -- start 2024-09-16T00:31:47.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc94effe640 1 -- start start 2024-09-16T00:31:47.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc94effe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc948102db0 0x7fc948100a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc94effe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc948103760 0x7fc948100f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc94effe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9481082e0 0x7fc948101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc94effe640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc9481120b0 con 0x7fc948103760 2024-09-16T00:31:47.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc94effe640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc948111f30 con 0x7fc948102db0 2024-09-16T00:31:47.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc94effe640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc948112230 con 0x7fc9481082e0 2024-09-16T00:31:47.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc947fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc948103760 0x7fc948100f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc947fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc948103760 0x7fc948100f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37996/0 (socket says 172.21.15.33:37996) 2024-09-16T00:31:47.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc947fff640 1 -- 172.21.15.33:0/2180644107 learned_addr learned my addr 172.21.15.33:0/2180644107 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:47.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc94cd9a640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc948102db0 0x7fc948100a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.553+0000 7fc94d59b640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9481082e0 0x7fc948101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.554+0000 7fc947fff640 1 -- 172.21.15.33:0/2180644107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9481082e0 msgr2=0x7fc948101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:47.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.554+0000 7fc947fff640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9481082e0 0x7fc948101490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.554+0000 7fc947fff640 1 -- 172.21.15.33:0/2180644107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc948102db0 msgr2=0x7fc948100a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:47.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.554+0000 7fc947fff640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc948102db0 0x7fc948100a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.554+0000 7fc947fff640 1 -- 172.21.15.33:0/2180644107 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9481a6a70 con 0x7fc948103760 2024-09-16T00:31:47.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.554+0000 7fc947fff640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc948103760 0x7fc948100f50 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fc93400e9c0 tx=0x7fc93400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:47.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.554+0000 7fc945ffb640 1 -- 172.21.15.33:0/2180644107 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93400ccc0 con 0x7fc948103760 2024-09-16T00:31:47.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.554+0000 7fc94effe640 1 -- 172.21.15.33:0/2180644107 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9481a6d60 con 0x7fc948103760 2024-09-16T00:31:47.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.554+0000 7fc945ffb640 1 -- 172.21.15.33:0/2180644107 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc934005500 con 0x7fc948103760 2024-09-16T00:31:47.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.554+0000 7fc945ffb640 1 -- 172.21.15.33:0/2180644107 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934010690 con 0x7fc948103760 2024-09-16T00:31:47.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.555+0000 7fc94effe640 1 -- 172.21.15.33:0/2180644107 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9481a71f0 con 0x7fc948103760 2024-09-16T00:31:47.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.558+0000 7fc945ffb640 1 -- 172.21.15.33:0/2180644107 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc934004e80 con 0x7fc948103760 2024-09-16T00:31:47.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.558+0000 7fc9513d7740 1 Processor -- start 2024-09-16T00:31:47.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.558+0000 7fc9513d7740 1 -- start start 2024-09-16T00:31:47.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.558+0000 7fc9513d7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e774a0 0x555799e77880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.558+0000 7fc9513d7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e77dc0 0x555799e82140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc9513d7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e82680 0x555799e84a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc9513d7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555799ddd2e0 con 0x555799e82680 2024-09-16T00:31:47.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc9513d7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555799d84ce0 con 0x555799e774a0 2024-09-16T00:31:47.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc9513d7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555799dd92a0 con 0x555799e77dc0 2024-09-16T00:31:47.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc94d59b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e774a0 0x555799e77880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc94d59b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e774a0 0x555799e77880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55774/0 (socket says 172.21.15.33:55774) 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc94d59b640 1 -- 172.21.15.33:0/3546706262 learned_addr learned my addr 172.21.15.33:0/3546706262 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc947fff640 1 --2- 172.21.15.33:0/3546706262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e82680 0x555799e84a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc94cd9a640 1 --2- 172.21.15.33:0/3546706262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e77dc0 0x555799e82140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc945ffb640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9380a3240 0x7fc9380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc94d59b640 1 -- 172.21.15.33:0/3546706262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e77dc0 msgr2=0x555799e82140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc94d59b640 1 --2- 172.21.15.33:0/3546706262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e77dc0 0x555799e82140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc94d59b640 1 -- 172.21.15.33:0/3546706262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e82680 msgr2=0x555799e84a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc94d59b640 1 --2- 172.21.15.33:0/3546706262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e82680 0x555799e84a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc94d59b640 1 -- 172.21.15.33:0/3546706262 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555799d69440 con 0x555799e774a0 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc947fff640 1 --2- 172.21.15.33:0/3546706262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e82680 0x555799e84a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc945ffb640 1 -- 172.21.15.33:0/2180644107 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc9340d00d0 con 0x7fc948103760 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc94d59b640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9380a3240 0x7fc9380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.559+0000 7fc94d59b640 1 --2- 172.21.15.33:0/3546706262 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e774a0 0x555799e77880 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc940009cd0 tx=0x7fc940009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=247e400b2744cce9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:47.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.560+0000 7fc932ffd640 1 -- 172.21.15.33:0/3546706262 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940031910 con 0x555799e774a0 2024-09-16T00:31:47.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.560+0000 7fc932ffd640 1 -- 172.21.15.33:0/3546706262 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc940036070 con 0x555799e774a0 2024-09-16T00:31:47.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.560+0000 7fc932ffd640 1 -- 172.21.15.33:0/3546706262 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9400309c0 con 0x555799e774a0 2024-09-16T00:31:47.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.560+0000 7fc94d59b640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9380a3240 0x7fc9380a5700 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc9400318b0 tx=0x7fc94004a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:47.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.560+0000 7fc9513d7740 1 -- 172.21.15.33:0/3546706262 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e774a0 msgr2=0x555799e77880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.560+0000 7fc9513d7740 1 --2- 172.21.15.33:0/3546706262 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e774a0 0x555799e77880 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc940009cd0 tx=0x7fc940009a20 comp rx=0 tx=0).stop 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.561+0000 7fc9513d7740 1 -- 172.21.15.33:0/3546706262 shutdown_connections 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.561+0000 7fc9513d7740 1 --2- 172.21.15.33:0/3546706262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e82680 0x555799e84a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.561+0000 7fc9513d7740 1 --2- 172.21.15.33:0/3546706262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e77dc0 0x555799e82140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.561+0000 7fc9513d7740 1 --2- 172.21.15.33:0/3546706262 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e774a0 0x555799e77880 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.561+0000 7fc9513d7740 1 -- 172.21.15.33:0/3546706262 >> 172.21.15.33:0/3546706262 conn(0x555799e73a30 msgr2=0x555799e73e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.561+0000 7fc9513d7740 1 -- 172.21.15.33:0/3546706262 shutdown_connections 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.561+0000 7fc9513d7740 1 -- 172.21.15.33:0/3546706262 wait complete. 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.561+0000 7fc9513d7740 1 -- start start 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.561+0000 7fc9513d7740 1 Processor -- start 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc9513d7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e774a0 0x555799e81f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc9513d7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e77dc0 0x555799e7f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc9513d7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e82680 0x555799e7f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc9513d7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555799e85250 con 0x555799e77dc0 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc9513d7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555799d98050 con 0x555799e82680 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc9513d7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555799e853d0 con 0x555799e774a0 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc9513d7740 10 client.0 _ll_register_callbacks cb 0x555799dd8160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc94cd9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e774a0 0x555799e81f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc947fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e77dc0 0x555799e7f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc94cd9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e774a0 0x555799e81f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:49310/0 (socket says 172.21.15.33:49310) 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc94d59b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e82680 0x555799e7f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc94cd9a640 1 -- 172.21.15.33:0/3832848914 learned_addr learned my addr 172.21.15.33:0/3832848914 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc947fff640 1 -- 172.21.15.33:0/3832848914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e774a0 msgr2=0x555799e81f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc947fff640 1 --2- 172.21.15.33:0/3832848914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e774a0 0x555799e81f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc947fff640 1 -- 172.21.15.33:0/3832848914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e82680 msgr2=0x555799e7f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc947fff640 1 --2- 172.21.15.33:0/3832848914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e82680 0x555799e7f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.562+0000 7fc947fff640 1 -- 172.21.15.33:0/3832848914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555799b4e3c0 con 0x555799e77dc0 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.563+0000 7fc947fff640 1 --2- 172.21.15.33:0/3832848914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e77dc0 0x555799e7f200 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fc9340cd7a0 tx=0x7fc9340d3990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2713213816be39 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.563+0000 7fc927fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.563+0000 7fc927fff640 1 -- 172.21.15.33:0/3832848914 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93400ce60 con 0x555799e77dc0 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.563+0000 7fc927fff640 1 -- 172.21.15.33:0/3832848914 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc934005170 con 0x555799e77dc0 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.563+0000 7fc9513d7740 1 -- 172.21.15.33:0/3832848914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc940036070 con 0x555799e77dc0 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.563+0000 7fc927fff640 1 -- 172.21.15.33:0/3832848914 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc93409c970 con 0x555799e77dc0 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.563+0000 7fc9513d7740 10 client.18207 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.563+0000 7fc9513d7740 1 -- 172.21.15.33:0/3832848914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555799e90c30 con 0x555799e77dc0 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.564+0000 7fc927fff640 1 -- 172.21.15.33:0/3832848914 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc940036070 con 0x555799e77dc0 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.564+0000 7fc927fff640 1 -- 172.21.15.33:0/3832848914 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc9340b50d0 con 0x555799e77dc0 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.564+0000 7fc9513d7740 10 client.18207 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.564+0000 7fc9513d7740 10 client.18207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:31:47.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.564+0000 7fc9513d7740 20 client.18207 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:47.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.564+0000 7fc9513d7740 1 --2- 172.21.15.33:0/3832848914 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555799e920f0 0x555799e94500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.564+0000 7fc9513d7740 4 client.18207 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:31:47.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.564+0000 7fc9513d7740 1 -- 172.21.15.33:0/3832848914 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555799dea850 con 0x555799e920f0 2024-09-16T00:31:47.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.565+0000 7fc94cd9a640 1 --2- 172.21.15.33:0/3832848914 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555799e920f0 0x555799e94500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.566+0000 7fc927fff640 10 client.18207 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:47.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.566+0000 7fc94cd9a640 1 --2- 172.21.15.33:0/3832848914 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555799e920f0 0x555799e94500 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67828332b4de7684 server_cookie=1e3b852cb25ae6ac in_seq=0 out_seq=0 2024-09-16T00:31:47.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.566+0000 7fc927fff640 1 -- 172.21.15.33:0/3832848914 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555799dea850 con 0x555799e920f0 2024-09-16T00:31:47.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.567+0000 7fc927fff640 10 client.18207 handle_command_reply: tid=0 2024-09-16T00:31:47.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.567+0000 7fc9513d7740 1 client.18207 shutdown 2024-09-16T00:31:47.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.569+0000 7fc9513d7740 1 -- 172.21.15.33:0/3832848914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e77dc0 msgr2=0x555799e7f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:47.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.569+0000 7fc9513d7740 1 --2- 172.21.15.33:0/3832848914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e77dc0 0x555799e7f200 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fc9340cd7a0 tx=0x7fc9340d3990 comp rx=0 tx=0).stop 2024-09-16T00:31:47.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.569+0000 7fc9513d7740 1 -- 172.21.15.33:0/3832848914 shutdown_connections 2024-09-16T00:31:47.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.569+0000 7fc9513d7740 1 --2- 172.21.15.33:0/3832848914 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555799e920f0 0x555799e94500 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.569+0000 7fc9513d7740 1 --2- 172.21.15.33:0/3832848914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e82680 0x555799e7f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.569+0000 7fc9513d7740 1 --2- 172.21.15.33:0/3832848914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e77dc0 0x555799e7f200 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.569+0000 7fc9513d7740 1 --2- 172.21.15.33:0/3832848914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e774a0 0x555799e81f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.569+0000 7fc927fff640 0 client.18207 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:47.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.569+0000 7fc9513d7740 1 -- 172.21.15.33:0/3832848914 >> 172.21.15.33:0/3832848914 conn(0x555799e73a30 msgr2=0x555799e79760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:47.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.570+0000 7fc9513d7740 1 -- 172.21.15.33:0/3832848914 shutdown_connections 2024-09-16T00:31:47.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.570+0000 7fc9513d7740 1 -- 172.21.15.33:0/3832848914 wait complete. 2024-09-16T00:31:47.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.570+0000 7fc9513d7740 20 client.18207 trim_cache size 0 max 16384 2024-09-16T00:31:47.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.589+0000 7fc9513d7740 1 Processor -- start 2024-09-16T00:31:47.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.589+0000 7fc9513d7740 1 -- start start 2024-09-16T00:31:47.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.589+0000 7fc9513d7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 0x555799e8e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.589+0000 7fc9513d7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e77dc0 0x555799e7a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.589+0000 7fc9513d7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e82680 0x555799e81170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.589+0000 7fc9513d7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc9481b3c50 con 0x555799e774a0 2024-09-16T00:31:47.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.589+0000 7fc9513d7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc9481b14c0 con 0x555799e77dc0 2024-09-16T00:31:47.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.589+0000 7fc9513d7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc9481af310 con 0x555799e82680 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc94cd9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 0x555799e8e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc94cd9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 0x555799e8e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38026/0 (socket says 172.21.15.33:38026) 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc94cd9a640 1 -- 172.21.15.33:0/1117957094 learned_addr learned my addr 172.21.15.33:0/1117957094 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc94d59b640 1 --2- 172.21.15.33:0/1117957094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e82680 0x555799e81170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc947fff640 1 --2- 172.21.15.33:0/1117957094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e77dc0 0x555799e7a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc94cd9a640 1 -- 172.21.15.33:0/1117957094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e82680 msgr2=0x555799e81170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc94cd9a640 1 --2- 172.21.15.33:0/1117957094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e82680 0x555799e81170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc94cd9a640 1 -- 172.21.15.33:0/1117957094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e77dc0 msgr2=0x555799e7a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc94cd9a640 1 --2- 172.21.15.33:0/1117957094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e77dc0 0x555799e7a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc94cd9a640 1 -- 172.21.15.33:0/1117957094 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555799e734d0 con 0x555799e774a0 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc947fff640 1 --2- 172.21.15.33:0/1117957094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e77dc0 0x555799e7a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc94cd9a640 1 --2- 172.21.15.33:0/1117957094 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 0x555799e8e4e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009e00 tx=0x7fc93c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3799bb243d348262 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:47.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc9327fc640 1 -- 172.21.15.33:0/1117957094 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c003cf0 con 0x555799e774a0 2024-09-16T00:31:47.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.590+0000 7fc9327fc640 1 -- 172.21.15.33:0/1117957094 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc93c010910 con 0x555799e774a0 2024-09-16T00:31:47.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.591+0000 7fc9327fc640 1 -- 172.21.15.33:0/1117957094 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c01f870 con 0x555799e774a0 2024-09-16T00:31:47.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.591+0000 7fc9513d7740 1 -- 172.21.15.33:0/1117957094 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 msgr2=0x555799e8e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:47.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.591+0000 7fc9513d7740 1 --2- 172.21.15.33:0/1117957094 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 0x555799e8e4e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009e00 tx=0x7fc93c005f90 comp rx=0 tx=0).stop 2024-09-16T00:31:47.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.591+0000 7fc9513d7740 1 -- 172.21.15.33:0/1117957094 shutdown_connections 2024-09-16T00:31:47.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.591+0000 7fc9513d7740 1 --2- 172.21.15.33:0/1117957094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e82680 0x555799e81170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.591+0000 7fc9513d7740 1 --2- 172.21.15.33:0/1117957094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e77dc0 0x555799e7a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.591+0000 7fc9513d7740 1 --2- 172.21.15.33:0/1117957094 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 0x555799e8e4e0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.591+0000 7fc9513d7740 1 -- 172.21.15.33:0/1117957094 >> 172.21.15.33:0/1117957094 conn(0x555799e73a30 msgr2=0x555799e80a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:47.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.591+0000 7fc9513d7740 1 -- 172.21.15.33:0/1117957094 shutdown_connections 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.591+0000 7fc9513d7740 1 -- 172.21.15.33:0/1117957094 wait complete. 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.592+0000 7fc9513d7740 1 -- start start 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.592+0000 7fc9513d7740 1 Processor -- start 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.592+0000 7fc9513d7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 0x555799e8aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.592+0000 7fc9513d7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e77dc0 0x555799e861f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.592+0000 7fc9513d7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e82680 0x555799e867e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.592+0000 7fc9513d7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc9481af980 con 0x555799e774a0 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.592+0000 7fc94cd9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 0x555799e8aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.592+0000 7fc9513d7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc9481b0df0 con 0x555799e82680 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.592+0000 7fc9513d7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc9481b0050 con 0x555799e77dc0 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.592+0000 7fc94cd9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 0x555799e8aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38036/0 (socket says 172.21.15.33:38036) 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.592+0000 7fc94cd9a640 1 -- 172.21.15.33:0/1933668386 learned_addr learned my addr 172.21.15.33:0/1933668386 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:31:47.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.592+0000 7fc947fff640 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e77dc0 0x555799e861f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc9513d7740 10 client.0 _ll_register_callbacks cb 0x7fc9340096c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:31:47.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc94d59b640 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e82680 0x555799e867e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc94cd9a640 1 -- 172.21.15.33:0/1933668386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e77dc0 msgr2=0x555799e861f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:47.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc94cd9a640 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e77dc0 0x555799e861f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc94cd9a640 1 -- 172.21.15.33:0/1933668386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e82680 msgr2=0x555799e867e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:31:47.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc94cd9a640 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e82680 0x555799e867e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc94cd9a640 1 -- 172.21.15.33:0/1933668386 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555799e87010 con 0x555799e774a0 2024-09-16T00:31:47.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc94cd9a640 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 0x555799e8aec0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc93c005e00 tx=0x7fc93c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a80b13aa41ecde server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:31:47.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc927fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:31:47.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc927fff640 1 -- 172.21.15.33:0/1933668386 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c010c00 con 0x555799e774a0 2024-09-16T00:31:47.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc9513d7740 1 -- 172.21.15.33:0/1933668386 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc93c010910 con 0x555799e774a0 2024-09-16T00:31:47.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc927fff640 1 -- 172.21.15.33:0/1933668386 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc93c010da0 con 0x555799e774a0 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc927fff640 1 -- 172.21.15.33:0/1933668386 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc93c03c050 con 0x555799e774a0 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc9513d7740 10 client.18213 fetch_fsmap learned FSMap version 31 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.593+0000 7fc9513d7740 1 -- 172.21.15.33:0/1933668386 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555799efce30 con 0x555799e774a0 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.594+0000 7fc927fff640 1 -- 172.21.15.33:0/1933668386 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc93c010910 con 0x555799e774a0 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.594+0000 7fc927fff640 1 -- 172.21.15.33:0/1933668386 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc93c01f9e0 con 0x555799e774a0 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.594+0000 7fc9513d7740 10 client.18213 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.594+0000 7fc9513d7740 10 client.18213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.594+0000 7fc9513d7740 20 client.18213 populate_metadata read hostname 'smithi033' 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.594+0000 7fc9513d7740 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555799efd440 0x555799eff830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.594+0000 7fc9513d7740 4 client.18213 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.594+0000 7fc9513d7740 1 -- 172.21.15.33:0/1933668386 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555799de4d60 con 0x555799efd440 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.594+0000 7fc947fff640 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555799efd440 0x555799eff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.594+0000 7fc927fff640 10 client.18213 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:47.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.594+0000 7fc947fff640 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555799efd440 0x555799eff830 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c4f25bdef32677 server_cookie=b84cfb59aae28041 in_seq=0 out_seq=0 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc927fff640 1 -- 172.21.15.33:0/1933668386 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555799de4d60 con 0x555799efd440 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc927fff640 10 client.18213 handle_command_reply: tid=0 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc9513d7740 1 client.18213 shutdown 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc9513d7740 1 -- 172.21.15.33:0/1933668386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 msgr2=0x555799e8aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc9513d7740 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 0x555799e8aec0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc93c005e00 tx=0x7fc93c009b30 comp rx=0 tx=0).stop 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc9513d7740 1 -- 172.21.15.33:0/1933668386 shutdown_connections 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc9513d7740 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555799efd440 0x555799eff830 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc9513d7740 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555799e82680 0x555799e867e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc927fff640 0 client.18213 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc9513d7740 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555799e77dc0 0x555799e861f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc9513d7740 1 --2- 172.21.15.33:0/1933668386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555799e774a0 0x555799e8aec0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc9513d7740 1 -- 172.21.15.33:0/1933668386 >> 172.21.15.33:0/1933668386 conn(0x555799e73a30 msgr2=0x555799e7a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:47.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc9513d7740 1 -- 172.21.15.33:0/1933668386 shutdown_connections 2024-09-16T00:31:47.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc9513d7740 1 -- 172.21.15.33:0/1933668386 wait complete. 2024-09-16T00:31:47.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.595+0000 7fc9513d7740 20 client.18213 trim_cache size 0 max 16384 2024-09-16T00:31:47.601 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:31:47.601 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:31:47.601 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:31:47.601 INFO:teuthology.orchestra.run.smithi033.stdout: "5a1869ce-cd48-42b9-99a8-63e821b640b8": { 2024-09-16T00:31:47.601 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:31:47.601 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "5a1869ce-cd48-42b9-99a8-63e821b640b8", 2024-09-16T00:31:47.601 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:31:47.601 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:47.602 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:31:47.602 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:31:47.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.596+0000 7fc94effe640 1 -- 172.21.15.33:0/2180644107 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9380a3240 msgr2=0x7fc9380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:47.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.596+0000 7fc94effe640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9380a3240 0x7fc9380a5700 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc9400318b0 tx=0x7fc94004a000 comp rx=0 tx=0).stop 2024-09-16T00:31:47.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.596+0000 7fc94effe640 1 -- 172.21.15.33:0/2180644107 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc948103760 msgr2=0x7fc948100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:31:47.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.596+0000 7fc94effe640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc948103760 0x7fc948100f50 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fc93400e9c0 tx=0x7fc93400ee80 comp rx=0 tx=0).stop 2024-09-16T00:31:47.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.596+0000 7fc94effe640 1 -- 172.21.15.33:0/2180644107 shutdown_connections 2024-09-16T00:31:47.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.596+0000 7fc94effe640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc9380a3240 0x7fc9380a5700 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.596+0000 7fc94effe640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9481082e0 0x7fc948101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.596+0000 7fc94effe640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc948103760 0x7fc948100f50 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.596+0000 7fc94effe640 1 --2- 172.21.15.33:0/2180644107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc948102db0 0x7fc948100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:31:47.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.596+0000 7fc94effe640 1 -- 172.21.15.33:0/2180644107 >> 172.21.15.33:0/2180644107 conn(0x7fc9480fc7d0 msgr2=0x7fc948105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:31:47.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.596+0000 7fc94effe640 1 -- 172.21.15.33:0/2180644107 shutdown_connections 2024-09-16T00:31:47.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:31:47.596+0000 7fc94effe640 1 -- 172.21.15.33:0/2180644107 wait complete. 2024-09-16T00:31:47.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:47 smithi033 ceph-mon[31243]: pgmap v789: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 69 KiB/s wr, 12 op/s 2024-09-16T00:31:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:47 smithi083 ceph-mon[35242]: pgmap v789: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 69 KiB/s wr, 12 op/s 2024-09-16T00:31:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:47 smithi060 ceph-mon[35185]: pgmap v789: 129 pgs: 129 active+clean; 474 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 69 KiB/s wr, 12 op/s 2024-09-16T00:31:48.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 14493 17.47 MB/sec warmup 54 sec latency 110.552 ms 2024-09-16T00:31:48.351 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '8f702410', total seconds elapsed: 31.7 2024-09-16T00:31:49.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 15367 18.05 MB/sec warmup 55 sec latency 123.773 ms 2024-09-16T00:31:49.197 INFO:tasks.cephfs.filesystem:scrub status for tag:5a1869ce-cd48-42b9-99a8-63e821b640b8 - {'path': '/', 'tag': '5a1869ce-cd48-42b9-99a8-63e821b640b8', 'options': 'recursive,force'} 2024-09-16T00:31:49.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:49 smithi083 ceph-mon[35242]: pgmap v790: 129 pgs: 129 active+clean; 476 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 314 KiB/s wr, 19 op/s 2024-09-16T00:31:49.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:49 smithi033 ceph-mon[31243]: pgmap v790: 129 pgs: 129 active+clean; 476 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 314 KiB/s wr, 19 op/s 2024-09-16T00:31:49.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:49 smithi060 ceph-mon[35185]: pgmap v790: 129 pgs: 129 active+clean; 476 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 314 KiB/s wr, 19 op/s 2024-09-16T00:31:50.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 16050 18.63 MB/sec warmup 56 sec latency 70.400 ms 2024-09-16T00:31:51.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 16515 18.74 MB/sec warmup 57 sec latency 145.342 ms 2024-09-16T00:31:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:51 smithi083 ceph-mon[35242]: pgmap v791: 129 pgs: 129 active+clean; 475 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.0 MiB/s wr, 57 op/s 2024-09-16T00:31:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:51 smithi060 ceph-mon[35185]: pgmap v791: 129 pgs: 129 active+clean; 475 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.0 MiB/s wr, 57 op/s 2024-09-16T00:31:52.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 17062 19.00 MB/sec warmup 58 sec latency 108.357 ms 2024-09-16T00:31:52.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:51 smithi033 ceph-mon[31243]: pgmap v791: 129 pgs: 129 active+clean; 475 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.0 MiB/s wr, 57 op/s 2024-09-16T00:31:53.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 17497 19.07 MB/sec warmup 59 sec latency 67.450 ms 2024-09-16T00:31:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:53 smithi083 ceph-mon[35242]: pgmap v792: 129 pgs: 129 active+clean; 479 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 180 op/s 2024-09-16T00:31:53.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:53 smithi060 ceph-mon[35185]: pgmap v792: 129 pgs: 129 active+clean; 479 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 180 op/s 2024-09-16T00:31:54.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 17885 18.96 MB/sec warmup 60 sec latency 60.005 ms 2024-09-16T00:31:54.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:53 smithi033 ceph-mon[31243]: pgmap v792: 129 pgs: 129 active+clean; 479 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 180 op/s 2024-09-16T00:31:55.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 18583 19.20 MB/sec warmup 61 sec latency 87.615 ms 2024-09-16T00:31:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:55 smithi083 ceph-mon[35242]: pgmap v793: 129 pgs: 129 active+clean; 477 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 256 op/s 2024-09-16T00:31:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:55 smithi060 ceph-mon[35185]: pgmap v793: 129 pgs: 129 active+clean; 477 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 256 op/s 2024-09-16T00:31:56.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 19271 19.33 MB/sec warmup 62 sec latency 74.359 ms 2024-09-16T00:31:56.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:55 smithi033 ceph-mon[31243]: pgmap v793: 129 pgs: 129 active+clean; 477 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 256 op/s 2024-09-16T00:31:56.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:56 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:31:56.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:56 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:31:57.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 20019 19.76 MB/sec warmup 63 sec latency 65.437 ms 2024-09-16T00:31:57.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:56 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:31:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:57 smithi083 ceph-mon[35242]: pgmap v794: 129 pgs: 129 active+clean; 463 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.8 MiB/s wr, 366 op/s 2024-09-16T00:31:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:57 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:31:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:57 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:31:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:57 smithi060 ceph-mon[35185]: pgmap v794: 129 pgs: 129 active+clean; 463 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.8 MiB/s wr, 366 op/s 2024-09-16T00:31:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:57 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:31:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:57 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:31:58.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 20782 20.30 MB/sec warmup 64 sec latency 39.451 ms 2024-09-16T00:31:58.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:57 smithi033 ceph-mon[31243]: pgmap v794: 129 pgs: 129 active+clean; 463 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.8 MiB/s wr, 366 op/s 2024-09-16T00:31:58.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:57 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:31:58.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:57 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:31:59.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 21307 20.49 MB/sec warmup 65 sec latency 51.412 ms 2024-09-16T00:31:59.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:31:59 smithi083 ceph-mon[35242]: pgmap v795: 129 pgs: 129 active+clean; 463 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.8 MiB/s wr, 373 op/s 2024-09-16T00:31:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:31:59 smithi060 ceph-mon[35185]: pgmap v795: 129 pgs: 129 active+clean; 463 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.8 MiB/s wr, 373 op/s 2024-09-16T00:32:00.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:31:59 smithi033 ceph-mon[31243]: pgmap v795: 129 pgs: 129 active+clean; 463 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.8 MiB/s wr, 373 op/s 2024-09-16T00:32:00.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 21766 20.53 MB/sec warmup 66 sec latency 52.508 ms 2024-09-16T00:32:01.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 22180 20.54 MB/sec warmup 67 sec latency 200.404 ms 2024-09-16T00:32:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:01 smithi083 ceph-mon[35242]: pgmap v796: 129 pgs: 129 active+clean; 470 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.3 MiB/s wr, 444 op/s 2024-09-16T00:32:01.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:01 smithi060 ceph-mon[35185]: pgmap v796: 129 pgs: 129 active+clean; 470 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.3 MiB/s wr, 444 op/s 2024-09-16T00:32:02.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 22644 20.52 MB/sec warmup 68 sec latency 56.013 ms 2024-09-16T00:32:02.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:01 smithi033 ceph-mon[31243]: pgmap v796: 129 pgs: 129 active+clean; 470 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.3 MiB/s wr, 444 op/s 2024-09-16T00:32:03.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 23291 20.59 MB/sec warmup 69 sec latency 82.579 ms 2024-09-16T00:32:03.881 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:03 smithi033 ceph-mon[31243]: pgmap v797: 129 pgs: 129 active+clean; 477 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 6.3 MiB/s wr, 484 op/s 2024-09-16T00:32:03.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:03 smithi083 ceph-mon[35242]: pgmap v797: 129 pgs: 129 active+clean; 477 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 6.3 MiB/s wr, 484 op/s 2024-09-16T00:32:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:03 smithi060 ceph-mon[35185]: pgmap v797: 129 pgs: 129 active+clean; 477 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 6.3 MiB/s wr, 484 op/s 2024-09-16T00:32:04.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 24191 20.99 MB/sec warmup 70 sec latency 50.562 ms 2024-09-16T00:32:05.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 24904 21.38 MB/sec warmup 71 sec latency 73.292 ms 2024-09-16T00:32:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:05 smithi083 ceph-mon[35242]: pgmap v798: 129 pgs: 129 active+clean; 481 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.9 MiB/s wr, 432 op/s 2024-09-16T00:32:05.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:05 smithi060 ceph-mon[35185]: pgmap v798: 129 pgs: 129 active+clean; 481 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.9 MiB/s wr, 432 op/s 2024-09-16T00:32:06.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 25419 21.50 MB/sec warmup 72 sec latency 72.148 ms 2024-09-16T00:32:06.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:05 smithi033 ceph-mon[31243]: pgmap v798: 129 pgs: 129 active+clean; 481 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.9 MiB/s wr, 432 op/s 2024-09-16T00:32:07.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 26034 21.67 MB/sec warmup 73 sec latency 48.775 ms 2024-09-16T00:32:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:07 smithi083 ceph-mon[35242]: pgmap v799: 129 pgs: 129 active+clean; 489 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 4.7 MiB/s wr, 444 op/s 2024-09-16T00:32:07.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:07 smithi060 ceph-mon[35185]: pgmap v799: 129 pgs: 129 active+clean; 489 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 4.7 MiB/s wr, 444 op/s 2024-09-16T00:32:08.068 INFO:tasks.workunit.client.0.smithi033.stdout: 10 26492 21.66 MB/sec warmup 74 sec latency 110.208 ms 2024-09-16T00:32:08.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:07 smithi033 ceph-mon[31243]: pgmap v799: 129 pgs: 129 active+clean; 489 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 4.7 MiB/s wr, 444 op/s 2024-09-16T00:32:09.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 27001 21.76 MB/sec warmup 75 sec latency 63.923 ms 2024-09-16T00:32:09.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:09 smithi083 ceph-mon[35242]: pgmap v800: 129 pgs: 129 active+clean; 490 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 349 op/s 2024-09-16T00:32:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:09 smithi060 ceph-mon[35185]: pgmap v800: 129 pgs: 129 active+clean; 490 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 349 op/s 2024-09-16T00:32:10.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 27691 21.82 MB/sec warmup 76 sec latency 192.188 ms 2024-09-16T00:32:10.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:09 smithi033 ceph-mon[31243]: pgmap v800: 129 pgs: 129 active+clean; 490 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 349 op/s 2024-09-16T00:32:11.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 28373 22.02 MB/sec warmup 77 sec latency 112.037 ms 2024-09-16T00:32:11.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:11 smithi083 ceph-mon[35242]: pgmap v801: 129 pgs: 129 active+clean; 494 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.3 MiB/s wr, 383 op/s 2024-09-16T00:32:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:11 smithi060 ceph-mon[35185]: pgmap v801: 129 pgs: 129 active+clean; 494 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.3 MiB/s wr, 383 op/s 2024-09-16T00:32:12.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 29128 22.38 MB/sec warmup 78 sec latency 73.003 ms 2024-09-16T00:32:12.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:11 smithi033 ceph-mon[31243]: pgmap v801: 129 pgs: 129 active+clean; 494 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.3 MiB/s wr, 383 op/s 2024-09-16T00:32:13.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 29740 22.55 MB/sec warmup 79 sec latency 65.487 ms 2024-09-16T00:32:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:13 smithi083 ceph-mon[35242]: pgmap v802: 129 pgs: 129 active+clean; 504 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.6 MiB/s wr, 416 op/s 2024-09-16T00:32:13.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:13 smithi060 ceph-mon[35185]: pgmap v802: 129 pgs: 129 active+clean; 504 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.6 MiB/s wr, 416 op/s 2024-09-16T00:32:14.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 30253 22.56 MB/sec warmup 80 sec latency 42.690 ms 2024-09-16T00:32:14.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:13 smithi033 ceph-mon[31243]: pgmap v802: 129 pgs: 129 active+clean; 504 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.6 MiB/s wr, 416 op/s 2024-09-16T00:32:15.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 30760 22.63 MB/sec warmup 81 sec latency 120.308 ms 2024-09-16T00:32:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:15 smithi083 ceph-mon[35242]: pgmap v803: 129 pgs: 129 active+clean; 512 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.7 MiB/s wr, 429 op/s 2024-09-16T00:32:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:15 smithi060 ceph-mon[35185]: pgmap v803: 129 pgs: 129 active+clean; 512 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.7 MiB/s wr, 429 op/s 2024-09-16T00:32:16.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 31388 22.83 MB/sec warmup 82 sec latency 60.522 ms 2024-09-16T00:32:16.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:15 smithi033 ceph-mon[31243]: pgmap v803: 129 pgs: 129 active+clean; 512 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.7 MiB/s wr, 429 op/s 2024-09-16T00:32:17.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 31666 22.63 MB/sec warmup 83 sec latency 513.689 ms 2024-09-16T00:32:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:17 smithi083 ceph-mon[35242]: pgmap v804: 129 pgs: 129 active+clean; 518 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.9 MiB/s wr, 426 op/s 2024-09-16T00:32:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:17 smithi060 ceph-mon[35185]: pgmap v804: 129 pgs: 129 active+clean; 518 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.9 MiB/s wr, 426 op/s 2024-09-16T00:32:18.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 31804 22.40 MB/sec warmup 84 sec latency 924.209 ms 2024-09-16T00:32:18.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:17 smithi033 ceph-mon[31243]: pgmap v804: 129 pgs: 129 active+clean; 518 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.9 MiB/s wr, 426 op/s 2024-09-16T00:32:19.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 32124 22.32 MB/sec warmup 85 sec latency 182.215 ms 2024-09-16T00:32:19.199 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:32:19.601 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:32:19.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:19 smithi033 ceph-mon[31243]: pgmap v805: 129 pgs: 129 active+clean; 521 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-09-16T00:32:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:19 smithi083 ceph-mon[35242]: pgmap v805: 129 pgs: 129 active+clean; 521 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-09-16T00:32:19.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:19 smithi060 ceph-mon[35185]: pgmap v805: 129 pgs: 129 active+clean; 521 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-09-16T00:32:20.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 32622 22.42 MB/sec warmup 86 sec latency 75.860 ms 2024-09-16T00:32:20.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.155+0000 7fb598ee2640 1 -- 172.21.15.33:0/2517022367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb594106b80 msgr2=0x7fb594106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:20.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.156+0000 7fb598ee2640 1 --2- 172.21.15.33:0/2517022367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb594106b80 0x7fb594106f60 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb584009900 tx=0x7fb58402a410 comp rx=0 tx=0).stop 2024-09-16T00:32:20.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.156+0000 7fb598ee2640 1 -- 172.21.15.33:0/2517022367 shutdown_connections 2024-09-16T00:32:20.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.156+0000 7fb598ee2640 1 --2- 172.21.15.33:0/2517022367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb594100aa0 0x7fb59410ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.156+0000 7fb598ee2640 1 --2- 172.21.15.33:0/2517022367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb594100100 0x7fb594100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.156+0000 7fb598ee2640 1 --2- 172.21.15.33:0/2517022367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb594106b80 0x7fb594106f60 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.156+0000 7fb598ee2640 1 -- 172.21.15.33:0/2517022367 >> 172.21.15.33:0/2517022367 conn(0x7fb594077230 msgr2=0x7fb5940fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:20.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.156+0000 7fb598ee2640 1 -- 172.21.15.33:0/2517022367 shutdown_connections 2024-09-16T00:32:20.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.156+0000 7fb598ee2640 1 -- 172.21.15.33:0/2517022367 wait complete. 2024-09-16T00:32:20.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb598ee2640 1 Processor -- start 2024-09-16T00:32:20.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb598ee2640 1 -- start start 2024-09-16T00:32:20.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb598ee2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb594100100 0x7fb5941984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb598ee2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb594100aa0 0x7fb594198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb598ee2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb594106b80 0x7fb59419cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb598ee2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb594112050 con 0x7fb594100100 2024-09-16T00:32:20.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb598ee2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb594111ed0 con 0x7fb594100aa0 2024-09-16T00:32:20.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb598ee2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb5941121d0 con 0x7fb594106b80 2024-09-16T00:32:20.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb59259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb594100100 0x7fb5941984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb59259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb594100100 0x7fb5941984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35186/0 (socket says 172.21.15.33:35186) 2024-09-16T00:32:20.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb59259c640 1 -- 172.21.15.33:0/22904337 learned_addr learned my addr 172.21.15.33:0/22904337 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:20.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb591d9b640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb594100aa0 0x7fb594198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb592d9d640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb594106b80 0x7fb59419cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb59259c640 1 -- 172.21.15.33:0/22904337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb594106b80 msgr2=0x7fb59419cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:20.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.157+0000 7fb59259c640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb594106b80 0x7fb59419cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.158+0000 7fb59259c640 1 -- 172.21.15.33:0/22904337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb594100aa0 msgr2=0x7fb594198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:20.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.158+0000 7fb59259c640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb594100aa0 0x7fb594198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.158+0000 7fb59259c640 1 -- 172.21.15.33:0/22904337 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59419d4a0 con 0x7fb594100100 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.158+0000 7fb591d9b640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb594100aa0 0x7fb594198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.158+0000 7fb59259c640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb594100100 0x7fb5941984f0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fb584006010 tx=0x7fb5840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.158+0000 7fb5837fe640 1 -- 172.21.15.33:0/22904337 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58403f030 con 0x7fb594100100 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.158+0000 7fb5837fe640 1 -- 172.21.15.33:0/22904337 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb584003ce0 con 0x7fb594100100 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.158+0000 7fb5837fe640 1 -- 172.21.15.33:0/22904337 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb584035070 con 0x7fb594100100 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.158+0000 7fb598ee2640 1 -- 172.21.15.33:0/22904337 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb59419d730 con 0x7fb594100100 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.158+0000 7fb598ee2640 1 -- 172.21.15.33:0/22904337 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5941a5070 con 0x7fb594100100 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.161+0000 7fb5837fe640 1 -- 172.21.15.33:0/22904337 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb584042030 con 0x7fb594100100 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.161+0000 7fb59b2bb740 1 Processor -- start 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.161+0000 7fb59b2bb740 1 -- start start 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb59b2bb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d54a0 0x561afc9d5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb59b2bb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d5dc0 0x561afc9e0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb59b2bb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9e0680 0x561afc9e2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb59b2bb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561afc93b2e0 con 0x561afc9d54a0 2024-09-16T00:32:20.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb59b2bb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561afc8e2ce0 con 0x561afc9d5dc0 2024-09-16T00:32:20.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb59b2bb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561afc9372a0 con 0x561afc9e0680 2024-09-16T00:32:20.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb592d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d54a0 0x561afc9d5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb592d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d54a0 0x561afc9d5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35198/0 (socket says 172.21.15.33:35198) 2024-09-16T00:32:20.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb592d9d640 1 -- 172.21.15.33:0/4178518259 learned_addr learned my addr 172.21.15.33:0/4178518259 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:20.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb59259c640 1 --2- 172.21.15.33:0/4178518259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d5dc0 0x561afc9e0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb591d9b640 1 --2- 172.21.15.33:0/4178518259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9e0680 0x561afc9e2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb5837fe640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb57c0a3240 0x7fb57c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb592d9d640 1 -- 172.21.15.33:0/4178518259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9e0680 msgr2=0x561afc9e2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:20.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb592d9d640 1 --2- 172.21.15.33:0/4178518259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9e0680 0x561afc9e2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb592d9d640 1 -- 172.21.15.33:0/4178518259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d5dc0 msgr2=0x561afc9e0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:20.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb592d9d640 1 --2- 172.21.15.33:0/4178518259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d5dc0 0x561afc9e0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.162+0000 7fb592d9d640 1 -- 172.21.15.33:0/4178518259 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561afc8c7440 con 0x561afc9d54a0 2024-09-16T00:32:20.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.163+0000 7fb5837fe640 1 -- 172.21.15.33:0/22904337 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb5840f0230 con 0x7fb594100100 2024-09-16T00:32:20.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.163+0000 7fb592d9d640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb57c0a3240 0x7fb57c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.163+0000 7fb592d9d640 1 --2- 172.21.15.33:0/4178518259 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d54a0 0x561afc9d5880 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb58c003970 tx=0x7fb58c003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d760afa9bbda328e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:20.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.163+0000 7fb59259c640 1 --2- 172.21.15.33:0/4178518259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d5dc0 0x561afc9e0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:20.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.163+0000 7fb580ff9640 1 -- 172.21.15.33:0/4178518259 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c022070 con 0x561afc9d54a0 2024-09-16T00:32:20.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.163+0000 7fb580ff9640 1 -- 172.21.15.33:0/4178518259 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb58c015070 con 0x561afc9d54a0 2024-09-16T00:32:20.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.163+0000 7fb580ff9640 1 -- 172.21.15.33:0/4178518259 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c010bf0 con 0x561afc9d54a0 2024-09-16T00:32:20.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.164+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4178518259 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d54a0 msgr2=0x561afc9d5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:20.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.164+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/4178518259 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d54a0 0x561afc9d5880 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb58c003970 tx=0x7fb58c003e30 comp rx=0 tx=0).stop 2024-09-16T00:32:20.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.164+0000 7fb592d9d640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb57c0a3240 0x7fb57c0a5700 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb58c010e20 tx=0x7fb58c010d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:20.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.164+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4178518259 shutdown_connections 2024-09-16T00:32:20.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.164+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/4178518259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9e0680 0x561afc9e2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.164+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/4178518259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d5dc0 0x561afc9e0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.164+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/4178518259 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d54a0 0x561afc9d5880 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.165+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4178518259 >> 172.21.15.33:0/4178518259 conn(0x561afc9d1a30 msgr2=0x561afc9d1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:20.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.165+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4178518259 shutdown_connections 2024-09-16T00:32:20.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.165+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4178518259 wait complete. 2024-09-16T00:32:20.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.165+0000 7fb59b2bb740 1 -- start start 2024-09-16T00:32:20.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.165+0000 7fb59b2bb740 1 Processor -- start 2024-09-16T00:32:20.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59b2bb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d54a0 0x561afc9dff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59b2bb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d5dc0 0x561afc9dd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d54a0 0x561afc9dff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d54a0 0x561afc9dff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42510/0 (socket says 172.21.15.33:42510) 2024-09-16T00:32:20.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb591d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d5dc0 0x561afc9dd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59259c640 1 -- 172.21.15.33:0/119066094 learned_addr learned my addr 172.21.15.33:0/119066094 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:20.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9e0680 0x561afc9dd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59b2bb740 1 -- 172.21.15.33:0/119066094 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561afc9e3250 con 0x561afc9d5dc0 2024-09-16T00:32:20.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59b2bb740 1 -- 172.21.15.33:0/119066094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561afc8f6050 con 0x561afc9d54a0 2024-09-16T00:32:20.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59b2bb740 1 -- 172.21.15.33:0/119066094 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561afc9e33d0 con 0x561afc9e0680 2024-09-16T00:32:20.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb592d9d640 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9e0680 0x561afc9dd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59259c640 1 -- 172.21.15.33:0/119066094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9e0680 msgr2=0x561afc9dd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:20.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59b2bb740 10 client.0 _ll_register_callbacks cb 0x561afc936160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:20.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59259c640 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9e0680 0x561afc9dd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59259c640 1 -- 172.21.15.33:0/119066094 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d5dc0 msgr2=0x561afc9dd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:20.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59259c640 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d5dc0 0x561afc9dd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59259c640 1 -- 172.21.15.33:0/119066094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561afc6ac3c0 con 0x561afc9d54a0 2024-09-16T00:32:20.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.166+0000 7fb59b2bb740 1 -- 172.21.15.33:0/119066094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb58c015070 con 0x561afc9d54a0 2024-09-16T00:32:20.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.167+0000 7fb59259c640 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d54a0 0x561afc9dff90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb5840b8df0 tx=0x7fb58402a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0f30a4a64889044 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:20.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.167+0000 7fb579ffb640 10 client.27560 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:32:20.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.167+0000 7fb579ffb640 1 -- 172.21.15.33:0/119066094 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb584049070 con 0x561afc9d54a0 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.167+0000 7fb579ffb640 1 -- 172.21.15.33:0/119066094 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb5840b8a90 con 0x561afc9d54a0 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.167+0000 7fb579ffb640 1 -- 172.21.15.33:0/119066094 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb5840bbf60 con 0x561afc9d54a0 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.167+0000 7fb59b2bb740 10 client.27560 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.167+0000 7fb59b2bb740 1 -- 172.21.15.33:0/119066094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({fsmap=31}) -- 0x561afc9eea70 con 0x561afc9d54a0 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.167+0000 7fb579ffb640 1 -- 172.21.15.33:0/119066094 <== mon.2 v2:172.21.15.60:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb5840bc680 con 0x561afc9d54a0 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.167+0000 7fb59b2bb740 10 client.27560 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.167+0000 7fb59b2bb740 10 client.27560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.167+0000 7fb59b2bb740 20 client.27560 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.167+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561afc9efe10 0x561afc9f2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.168+0000 7fb59b2bb740 4 client.27560 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.168+0000 7fb59b2bb740 1 -- 172.21.15.33:0/119066094 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561afc948850 con 0x561afc9efe10 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.168+0000 7fb591d9b640 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561afc9efe10 0x561afc9f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.168+0000 7fb579ffb640 10 client.27560 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.168+0000 7fb591d9b640 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561afc9efe10 0x561afc9f2280 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a27c96e9393f8e36 server_cookie=2d986e23ae6cb2d4 in_seq=0 out_seq=0 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb579ffb640 1 -- 172.21.15.33:0/119066094 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561afc948850 con 0x561afc9efe10 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb579ffb640 10 client.27560 handle_command_reply: tid=0 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb59b2bb740 1 client.27560 shutdown 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb59b2bb740 1 -- 172.21.15.33:0/119066094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d54a0 msgr2=0x561afc9dff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d54a0 0x561afc9dff90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb5840b8df0 tx=0x7fb58402a410 comp rx=0 tx=0).stop 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb59b2bb740 1 -- 172.21.15.33:0/119066094 shutdown_connections 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561afc9efe10 0x561afc9f2280 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9e0680 0x561afc9dd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb579ffb640 0 client.27560 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d5dc0 0x561afc9dd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/119066094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9d54a0 0x561afc9dff90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb59b2bb740 1 -- 172.21.15.33:0/119066094 >> 172.21.15.33:0/119066094 conn(0x561afc9d1a30 msgr2=0x561afc9d7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:20.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb59b2bb740 1 -- 172.21.15.33:0/119066094 shutdown_connections 2024-09-16T00:32:20.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb59b2bb740 1 -- 172.21.15.33:0/119066094 wait complete. 2024-09-16T00:32:20.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.169+0000 7fb59b2bb740 20 client.27560 trim_cache size 0 max 16384 2024-09-16T00:32:20.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.189+0000 7fb59b2bb740 1 Processor -- start 2024-09-16T00:32:20.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.189+0000 7fb59b2bb740 1 -- start start 2024-09-16T00:32:20.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.189+0000 7fb59b2bb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 0x561afc9ec4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.189+0000 7fb59b2bb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d5dc0 0x561afc9f7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.189+0000 7fb59b2bb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9e0680 0x561afc9f8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.189+0000 7fb59b2bb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb5941af730 con 0x561afc9d5dc0 2024-09-16T00:32:20.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.189+0000 7fb59b2bb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb5941acfa0 con 0x561afc9e0680 2024-09-16T00:32:20.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.189+0000 7fb59b2bb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb5941aadf0 con 0x561afc9d54a0 2024-09-16T00:32:20.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb59259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 0x561afc9ec4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb591d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d5dc0 0x561afc9f7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb59259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 0x561afc9ec4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:53972/0 (socket says 172.21.15.33:53972) 2024-09-16T00:32:20.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb591d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d5dc0 0x561afc9f7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35216/0 (socket says 172.21.15.33:35216) 2024-09-16T00:32:20.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb59259c640 1 -- 172.21.15.33:0/3786702783 learned_addr learned my addr 172.21.15.33:0/3786702783 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:20.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb592d9d640 1 --2- 172.21.15.33:0/3786702783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9e0680 0x561afc9f8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb591d9b640 1 -- 172.21.15.33:0/3786702783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 msgr2=0x561afc9ec4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:20.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb591d9b640 1 --2- 172.21.15.33:0/3786702783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 0x561afc9ec4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb591d9b640 1 -- 172.21.15.33:0/3786702783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9e0680 msgr2=0x561afc9f8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:20.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb591d9b640 1 --2- 172.21.15.33:0/3786702783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9e0680 0x561afc9f8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb591d9b640 1 -- 172.21.15.33:0/3786702783 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561afc9d1860 con 0x561afc9d5dc0 2024-09-16T00:32:20.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb59259c640 1 --2- 172.21.15.33:0/3786702783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 0x561afc9ec4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:20.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb592d9d640 1 --2- 172.21.15.33:0/3786702783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9e0680 0x561afc9f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:20.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.190+0000 7fb591d9b640 1 --2- 172.21.15.33:0/3786702783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d5dc0 0x561afc9f7b50 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x561afc9de880 tx=0x7fb588018b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22fdafbc6e42a788 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:20.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.191+0000 7fb57bfff640 1 -- 172.21.15.33:0/3786702783 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb588004020 con 0x561afc9d5dc0 2024-09-16T00:32:20.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.191+0000 7fb57bfff640 1 -- 172.21.15.33:0/3786702783 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb588011cf0 con 0x561afc9d5dc0 2024-09-16T00:32:20.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.191+0000 7fb57bfff640 1 -- 172.21.15.33:0/3786702783 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb588003980 con 0x561afc9d5dc0 2024-09-16T00:32:20.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.191+0000 7fb59b2bb740 1 -- 172.21.15.33:0/3786702783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d5dc0 msgr2=0x561afc9f7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:20.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.191+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/3786702783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d5dc0 0x561afc9f7b50 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x561afc9de880 tx=0x7fb588018b90 comp rx=0 tx=0).stop 2024-09-16T00:32:20.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.191+0000 7fb59b2bb740 1 -- 172.21.15.33:0/3786702783 shutdown_connections 2024-09-16T00:32:20.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.191+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/3786702783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9e0680 0x561afc9f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.191+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/3786702783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9d5dc0 0x561afc9f7b50 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.191+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/3786702783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 0x561afc9ec4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.191+0000 7fb59b2bb740 1 -- 172.21.15.33:0/3786702783 >> 172.21.15.33:0/3786702783 conn(0x561afc9d1a30 msgr2=0x561afc9dbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:20.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.192+0000 7fb59b2bb740 1 -- 172.21.15.33:0/3786702783 shutdown_connections 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.192+0000 7fb59b2bb740 1 -- 172.21.15.33:0/3786702783 wait complete. 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.192+0000 7fb59b2bb740 1 -- start start 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.192+0000 7fb59b2bb740 1 Processor -- start 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.192+0000 7fb59b2bb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 0x561afc9e8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.192+0000 7fb59b2bb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9e0680 0x561afc9e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59b2bb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9efe10 0x561afc9e55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 0x561afc9e8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 0x561afc9e8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:53986/0 (socket says 172.21.15.33:53986) 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59259c640 1 -- 172.21.15.33:0/4019900535 learned_addr learned my addr 172.21.15.33:0/4019900535 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4019900535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb5941ab460 con 0x561afc9efe10 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4019900535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb5941ac8d0 con 0x561afc9e0680 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4019900535 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb5941abb30 con 0x561afc9d54a0 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb592d9d640 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9efe10 0x561afc9e55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59b2bb740 10 client.0 _ll_register_callbacks cb 0x7fb584005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb591d9b640 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9e0680 0x561afc9e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59259c640 1 -- 172.21.15.33:0/4019900535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9e0680 msgr2=0x561afc9e8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59259c640 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9e0680 0x561afc9e8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59259c640 1 -- 172.21.15.33:0/4019900535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9efe10 msgr2=0x561afc9e55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59259c640 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9efe10 0x561afc9e55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59259c640 1 -- 172.21.15.33:0/4019900535 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561afc9e5df0 con 0x561afc9d54a0 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb592d9d640 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9efe10 0x561afc9e55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb59259c640 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 0x561afc9e8800 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb5840bc120 tx=0x7fb5840ed720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15c576b0d6d13a26 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.193+0000 7fb579ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb579ffb640 1 -- 172.21.15.33:0/4019900535 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb584049070 con 0x561afc9d54a0 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb579ffb640 1 -- 172.21.15.33:0/4019900535 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb58403bbb0 con 0x561afc9d54a0 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4019900535 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb588011cf0 con 0x561afc9d54a0 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb579ffb640 1 -- 172.21.15.33:0/4019900535 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb5840044e0 con 0x561afc9d54a0 2024-09-16T00:32:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb59b2bb740 10 client.27562 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:20.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4019900535 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561afca63dd0 con 0x561afc9d54a0 2024-09-16T00:32:20.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb579ffb640 1 -- 172.21.15.33:0/4019900535 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb58c015070 con 0x561afc9d54a0 2024-09-16T00:32:20.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb579ffb640 1 -- 172.21.15.33:0/4019900535 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb5840bb870 con 0x561afc9d54a0 2024-09-16T00:32:20.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb59b2bb740 10 client.27562 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:20.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb59b2bb740 10 client.27562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:32:20.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb59b2bb740 20 client.27562 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:20.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561afca642a0 0x561afca66690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:20.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb59b2bb740 4 client.27562 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:32:20.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4019900535 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fb588011cf0 con 0x561afca642a0 2024-09-16T00:32:20.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.194+0000 7fb591d9b640 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561afca642a0 0x561afca66690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:20.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb579ffb640 10 client.27562 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:20.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb591d9b640 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561afca642a0 0x561afca66690 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fef9bcaf7fc9e26 server_cookie=5ec12308b8c32365 in_seq=0 out_seq=0 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb579ffb640 1 -- 172.21.15.33:0/4019900535 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fb588011cf0 con 0x561afca642a0 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb579ffb640 10 client.27562 handle_command_reply: tid=0 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb59b2bb740 1 client.27562 shutdown 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4019900535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 msgr2=0x561afc9e8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 0x561afc9e8800 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb5840bc120 tx=0x7fb5840ed720 comp rx=0 tx=0).stop 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4019900535 shutdown_connections 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561afca642a0 0x561afca66690 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb579ffb640 0 client.27562 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561afc9efe10 0x561afc9e55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561afc9e0680 0x561afc9e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb59b2bb740 1 --2- 172.21.15.33:0/4019900535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561afc9d54a0 0x561afc9e8800 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.195+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4019900535 >> 172.21.15.33:0/4019900535 conn(0x561afc9d1a30 msgr2=0x561afc9d6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:20.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4019900535 shutdown_connections 2024-09-16T00:32:20.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb59b2bb740 1 -- 172.21.15.33:0/4019900535 wait complete. 2024-09-16T00:32:20.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb59b2bb740 20 client.27562 trim_cache size 0 max 16384 2024-09-16T00:32:20.202 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:32:20.202 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:32:20.202 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:32:20.202 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:32:20.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb598ee2640 1 -- 172.21.15.33:0/22904337 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb57c0a3240 msgr2=0x7fb57c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:20.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb598ee2640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb57c0a3240 0x7fb57c0a5700 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb58c010e20 tx=0x7fb58c010d90 comp rx=0 tx=0).stop 2024-09-16T00:32:20.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb598ee2640 1 -- 172.21.15.33:0/22904337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb594100100 msgr2=0x7fb5941984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:20.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb598ee2640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb594100100 0x7fb5941984f0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fb584006010 tx=0x7fb5840042f0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb598ee2640 1 -- 172.21.15.33:0/22904337 shutdown_connections 2024-09-16T00:32:20.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb598ee2640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb57c0a3240 0x7fb57c0a5700 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb598ee2640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb594106b80 0x7fb59419cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb598ee2640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb594100aa0 0x7fb594198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb598ee2640 1 --2- 172.21.15.33:0/22904337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb594100100 0x7fb5941984f0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:20.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.196+0000 7fb598ee2640 1 -- 172.21.15.33:0/22904337 >> 172.21.15.33:0/22904337 conn(0x7fb594077230 msgr2=0x7fb594104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:20.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.197+0000 7fb598ee2640 1 -- 172.21.15.33:0/22904337 shutdown_connections 2024-09-16T00:32:20.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:20.197+0000 7fb598ee2640 1 -- 172.21.15.33:0/22904337 wait complete. 2024-09-16T00:32:21.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 33435 22.77 MB/sec warmup 87 sec latency 56.127 ms 2024-09-16T00:32:21.083 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:32:21.084 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:32:21.442 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:32:21.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:21 smithi033 ceph-mon[31243]: pgmap v806: 129 pgs: 129 active+clean; 531 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.2 MiB/s wr, 463 op/s 2024-09-16T00:32:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:21 smithi083 ceph-mon[35242]: pgmap v806: 129 pgs: 129 active+clean; 531 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.2 MiB/s wr, 463 op/s 2024-09-16T00:32:21.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:21 smithi060 ceph-mon[35185]: pgmap v806: 129 pgs: 129 active+clean; 531 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.2 MiB/s wr, 463 op/s 2024-09-16T00:32:22.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 33691 22.68 MB/sec warmup 88 sec latency 460.469 ms 2024-09-16T00:32:22.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.071+0000 7f72adb26640 1 -- 172.21.15.33:0/958087893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72a8109e00 msgr2=0x7f72a810e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:22.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.071+0000 7f72adb26640 1 --2- 172.21.15.33:0/958087893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72a8109e00 0x7f72a810e350 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f72a0005fd0 tx=0x7f72a002c7b0 comp rx=0 tx=0).stop 2024-09-16T00:32:22.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.072+0000 7f72adb26640 1 -- 172.21.15.33:0/958087893 shutdown_connections 2024-09-16T00:32:22.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.072+0000 7f72adb26640 1 --2- 172.21.15.33:0/958087893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a810ea80 0x7f72a81154f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:22.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.072+0000 7f72adb26640 1 --2- 172.21.15.33:0/958087893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72a8109e00 0x7f72a810e350 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:22.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.072+0000 7f72adb26640 1 --2- 172.21.15.33:0/958087893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72a8109450 0x7f72a8109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:22.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.072+0000 7f72adb26640 1 -- 172.21.15.33:0/958087893 >> 172.21.15.33:0/958087893 conn(0x7f72a806e2d0 msgr2=0x7f72a806e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:22.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.073+0000 7f72adb26640 1 -- 172.21.15.33:0/958087893 shutdown_connections 2024-09-16T00:32:22.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.073+0000 7f72adb26640 1 -- 172.21.15.33:0/958087893 wait complete. 2024-09-16T00:32:22.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.074+0000 7f72adb26640 1 Processor -- start 2024-09-16T00:32:22.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.074+0000 7f72adb26640 1 -- start start 2024-09-16T00:32:22.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72adb26640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a8109450 0x7f72a8113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:22.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72adb26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72a8109e00 0x7f72a8114510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:22.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72adb26640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72a810ea80 0x7f72a81ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:22.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72adb26640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f72a8117bf0 con 0x7f72a8109450 2024-09-16T00:32:22.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72adb26640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f72a8117a70 con 0x7f72a8109e00 2024-09-16T00:32:22.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72adb26640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f72a8117d70 con 0x7f72a810ea80 2024-09-16T00:32:22.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72a6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72a8109e00 0x7f72a8114510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:22.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72a6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72a8109e00 0x7f72a8114510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:43734/0 (socket says 172.21.15.33:43734) 2024-09-16T00:32:22.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72a6ffd640 1 -- 172.21.15.33:0/2847177208 learned_addr learned my addr 172.21.15.33:0/2847177208 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:22.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72a7fff640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72a810ea80 0x7f72a81ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:22.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72a6ffd640 1 -- 172.21.15.33:0/2847177208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72a810ea80 msgr2=0x7f72a81ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:22.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72a6ffd640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72a810ea80 0x7f72a81ace10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:22.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72a6ffd640 1 -- 172.21.15.33:0/2847177208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a8109450 msgr2=0x7f72a8113fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:22.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72a6ffd640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a8109450 0x7f72a8113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:22.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.075+0000 7f72a6ffd640 1 -- 172.21.15.33:0/2847177208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a81ad5e0 con 0x7f72a8109e00 2024-09-16T00:32:22.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.076+0000 7f72a6ffd640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72a8109e00 0x7f72a8114510 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f72a000a2e0 tx=0x7f72a000a310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:22.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.077+0000 7f72a4ff9640 1 -- 172.21.15.33:0/2847177208 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0038070 con 0x7f72a8109e00 2024-09-16T00:32:22.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.078+0000 7f72a4ff9640 1 -- 172.21.15.33:0/2847177208 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72a003a050 con 0x7f72a8109e00 2024-09-16T00:32:22.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.078+0000 7f72adb26640 1 -- 172.21.15.33:0/2847177208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a81ad870 con 0x7f72a8109e00 2024-09-16T00:32:22.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.078+0000 7f72adb26640 1 -- 172.21.15.33:0/2847177208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a811d850 con 0x7f72a8109e00 2024-09-16T00:32:22.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.078+0000 7f72a4ff9640 1 -- 172.21.15.33:0/2847177208 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0004660 con 0x7f72a8109e00 2024-09-16T00:32:22.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.083+0000 7f72a4ff9640 1 -- 172.21.15.33:0/2847177208 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f72a0047430 con 0x7f72a8109e00 2024-09-16T00:32:22.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.084+0000 7f72a4ff9640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f72900a3450 0x7f72900a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:22.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.084+0000 7f72a4ff9640 1 -- 172.21.15.33:0/2847177208 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f72a00f09b0 con 0x7f72a8109e00 2024-09-16T00:32:22.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.085+0000 7f72adb26640 1 -- 172.21.15.33:0/2847177208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7294005290 con 0x7f72a8109e00 2024-09-16T00:32:22.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.086+0000 7f72a77fe640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f72900a3450 0x7f72900a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:22.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.088+0000 7f72a77fe640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f72900a3450 0x7f72900a5910 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7298009710 tx=0x7f7298006c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:22.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.093+0000 7f72a4ff9640 1 -- 172.21.15.33:0/2847177208 <== mon.2 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f72a0033c10 con 0x7f72a8109e00 2024-09-16T00:32:22.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.518+0000 7f72adb26640 1 -- 172.21.15.33:0/2847177208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7294005580 con 0x7f72a8109e00 2024-09-16T00:32:22.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.520+0000 7f72a4ff9640 1 -- 172.21.15.33:0/2847177208 <== mon.2 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11906 (secure 0 0 0) 0x7f72a00bde20 con 0x7f72a8109e00 2024-09-16T00:32:22.522 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:32:22.522 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":31,"btime":"2024-09-16T00:30:57:641642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:30:57.027530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:32:22.522 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 31 2024-09-16T00:32:22.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.523+0000 7f72adb26640 1 -- 172.21.15.33:0/2847177208 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f72900a3450 msgr2=0x7f72900a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:22.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.523+0000 7f72adb26640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f72900a3450 0x7f72900a5910 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7298009710 tx=0x7f7298006c90 comp rx=0 tx=0).stop 2024-09-16T00:32:22.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.523+0000 7f72adb26640 1 -- 172.21.15.33:0/2847177208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72a8109e00 msgr2=0x7f72a8114510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:22.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.523+0000 7f72adb26640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72a8109e00 0x7f72a8114510 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f72a000a2e0 tx=0x7f72a000a310 comp rx=0 tx=0).stop 2024-09-16T00:32:22.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.523+0000 7f72adb26640 1 -- 172.21.15.33:0/2847177208 shutdown_connections 2024-09-16T00:32:22.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.523+0000 7f72adb26640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f72900a3450 0x7f72900a5910 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:22.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.523+0000 7f72adb26640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72a810ea80 0x7f72a81ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:22.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.523+0000 7f72adb26640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72a8109e00 0x7f72a8114510 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:22.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.523+0000 7f72adb26640 1 --2- 172.21.15.33:0/2847177208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a8109450 0x7f72a8113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:22.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.523+0000 7f72adb26640 1 -- 172.21.15.33:0/2847177208 >> 172.21.15.33:0/2847177208 conn(0x7f72a806e2d0 msgr2=0x7f72a8073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:22.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.524+0000 7f72adb26640 1 -- 172.21.15.33:0/2847177208 shutdown_connections 2024-09-16T00:32:22.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:22.524+0000 7f72adb26640 1 -- 172.21.15.33:0/2847177208 wait complete. 2024-09-16T00:32:22.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:22 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2847177208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:32:22.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:22 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2847177208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:32:22.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:22 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2847177208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:32:23.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 33987 22.56 MB/sec warmup 89 sec latency 787.559 ms 2024-09-16T00:32:23.697 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:32:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:23 smithi083 ceph-mon[35242]: pgmap v807: 129 pgs: 129 active+clean; 525 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 10 MiB/s wr, 602 op/s 2024-09-16T00:32:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:23 smithi060 ceph-mon[35185]: pgmap v807: 129 pgs: 129 active+clean; 525 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 10 MiB/s wr, 602 op/s 2024-09-16T00:32:24.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 34266 22.46 MB/sec warmup 90 sec latency 112.047 ms 2024-09-16T00:32:24.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:23 smithi033 ceph-mon[31243]: pgmap v807: 129 pgs: 129 active+clean; 525 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 10 MiB/s wr, 602 op/s 2024-09-16T00:32:24.117 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:32:24.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.713+0000 7fbb2f701640 1 -- 172.21.15.33:0/2483839229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb28102d90 msgr2=0x7fbb28103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:24.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.713+0000 7fbb2f701640 1 --2- 172.21.15.33:0/2483839229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb28102d90 0x7fbb28103170 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009890 tx=0x7fbb2402a380 comp rx=0 tx=0).stop 2024-09-16T00:32:24.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.713+0000 7fbb2f701640 1 -- 172.21.15.33:0/2483839229 shutdown_connections 2024-09-16T00:32:24.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.713+0000 7fbb2f701640 1 --2- 172.21.15.33:0/2483839229 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb28108290 0x7fbb2810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.713+0000 7fbb2f701640 1 --2- 172.21.15.33:0/2483839229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb28103740 0x7fbb28107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.713+0000 7fbb2f701640 1 --2- 172.21.15.33:0/2483839229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb28102d90 0x7fbb28103170 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.713+0000 7fbb2f701640 1 -- 172.21.15.33:0/2483839229 >> 172.21.15.33:0/2483839229 conn(0x7fbb280fc7d0 msgr2=0x7fbb280febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:24.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.714+0000 7fbb2f701640 1 -- 172.21.15.33:0/2483839229 shutdown_connections 2024-09-16T00:32:24.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.714+0000 7fbb2f701640 1 -- 172.21.15.33:0/2483839229 wait complete. 2024-09-16T00:32:24.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.714+0000 7fbb2f701640 1 Processor -- start 2024-09-16T00:32:24.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.714+0000 7fbb2f701640 1 -- start start 2024-09-16T00:32:24.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2f701640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb28102d90 0x7fbb28198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2f701640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb28103740 0x7fbb28198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2f701640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb28108290 0x7fbb2819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2f701640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbb28112090 con 0x7fbb28108290 2024-09-16T00:32:24.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2f701640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbb28111f10 con 0x7fbb28103740 2024-09-16T00:32:24.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2f701640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbb28112210 con 0x7fbb28102d90 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2d49d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb28102d90 0x7fbb28198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2dc9e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb28108290 0x7fbb2819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2d49d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb28102d90 0x7fbb28198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60092/0 (socket says 172.21.15.33:60092) 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2d49d640 1 -- 172.21.15.33:0/3382561814 learned_addr learned my addr 172.21.15.33:0/3382561814 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2cc9c640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb28103740 0x7fbb28198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2d49d640 1 -- 172.21.15.33:0/3382561814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb28103740 msgr2=0x7fbb28198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2d49d640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb28103740 0x7fbb28198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2d49d640 1 -- 172.21.15.33:0/3382561814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb28108290 msgr2=0x7fbb2819cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2d49d640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb28108290 0x7fbb2819cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.715+0000 7fbb2d49d640 1 -- 172.21.15.33:0/3382561814 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2819d400 con 0x7fbb28102d90 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.716+0000 7fbb2d49d640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb28102d90 0x7fbb28198450 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbb2402a970 tx=0x7fbb24033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.716+0000 7fbb1e7fc640 1 -- 172.21.15.33:0/3382561814 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24037040 con 0x7fbb28102d90 2024-09-16T00:32:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.716+0000 7fbb1e7fc640 1 -- 172.21.15.33:0/3382561814 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb24042070 con 0x7fbb28102d90 2024-09-16T00:32:24.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.716+0000 7fbb1e7fc640 1 -- 172.21.15.33:0/3382561814 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24003b10 con 0x7fbb28102d90 2024-09-16T00:32:24.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.716+0000 7fbb2f701640 1 -- 172.21.15.33:0/3382561814 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb2819d690 con 0x7fbb28102d90 2024-09-16T00:32:24.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.716+0000 7fbb2f701640 1 -- 172.21.15.33:0/3382561814 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb2819db20 con 0x7fbb28102d90 2024-09-16T00:32:24.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.719+0000 7fbb31ada740 1 Processor -- start 2024-09-16T00:32:24.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.719+0000 7fbb1e7fc640 1 -- 172.21.15.33:0/3382561814 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbb24003cb0 con 0x7fbb28102d90 2024-09-16T00:32:24.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.719+0000 7fbb31ada740 1 -- start start 2024-09-16T00:32:24.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb31ada740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee81f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb31ada740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee81f5d0 0x562bee81fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb31ada740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee82c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb31ada740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562bee7852f0 con 0x562bee820b40 2024-09-16T00:32:24.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb31ada740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562bee72ccf0 con 0x562bee82a370 2024-09-16T00:32:24.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb31ada740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562bee7812b0 con 0x562bee81f5d0 2024-09-16T00:32:24.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb2dc9e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee81f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb2dc9e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee81f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39564/0 (socket says 172.21.15.33:39564) 2024-09-16T00:32:24.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb2d49d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee81f5d0 0x562bee81fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb2dc9e640 1 -- 172.21.15.33:0/4207650780 learned_addr learned my addr 172.21.15.33:0/4207650780 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:24.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb2cc9c640 1 --2- 172.21.15.33:0/4207650780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee82c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb2dc9e640 1 -- 172.21.15.33:0/4207650780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee81f5d0 msgr2=0x562bee81fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:24.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/4207650780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee81f5d0 0x562bee81fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb2dc9e640 1 -- 172.21.15.33:0/4207650780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 msgr2=0x562bee82c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:24.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/4207650780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee82c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.720+0000 7fbb2dc9e640 1 -- 172.21.15.33:0/4207650780 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bee7113a0 con 0x562bee820b40 2024-09-16T00:32:24.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.721+0000 7fbb2cc9c640 1 --2- 172.21.15.33:0/4207650780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee82c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:24.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.721+0000 7fbb1e7fc640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbb140a3240 0x7fbb140a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.721+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/4207650780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee81f090 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009f50 tx=0x7fbb20009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d52e68325ee7a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:24.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.721+0000 7fbb2d49d640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbb140a3240 0x7fbb140a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.721+0000 7fbb137fe640 1 -- 172.21.15.33:0/4207650780 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb20003c30 con 0x562bee820b40 2024-09-16T00:32:24.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.721+0000 7fbb1e7fc640 1 -- 172.21.15.33:0/3382561814 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fbb240efd60 con 0x7fbb28102d90 2024-09-16T00:32:24.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.721+0000 7fbb137fe640 1 -- 172.21.15.33:0/4207650780 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb2001f070 con 0x562bee820b40 2024-09-16T00:32:24.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.721+0000 7fbb137fe640 1 -- 172.21.15.33:0/4207650780 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2000ba80 con 0x562bee820b40 2024-09-16T00:32:24.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.722+0000 7fbb31ada740 1 -- 172.21.15.33:0/4207650780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 msgr2=0x562bee81f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:24.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.722+0000 7fbb2d49d640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbb140a3240 0x7fbb140a5700 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbb24004650 tx=0x7fbb240dc040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:24.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.722+0000 7fbb31ada740 1 --2- 172.21.15.33:0/4207650780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee81f090 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009f50 tx=0x7fbb20009500 comp rx=0 tx=0).stop 2024-09-16T00:32:24.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.722+0000 7fbb31ada740 1 -- 172.21.15.33:0/4207650780 shutdown_connections 2024-09-16T00:32:24.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.722+0000 7fbb31ada740 1 --2- 172.21.15.33:0/4207650780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee82c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.722+0000 7fbb31ada740 1 --2- 172.21.15.33:0/4207650780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee81f5d0 0x562bee81fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.722+0000 7fbb31ada740 1 --2- 172.21.15.33:0/4207650780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee81f090 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.722+0000 7fbb31ada740 1 -- 172.21.15.33:0/4207650780 >> 172.21.15.33:0/4207650780 conn(0x562bee7e8310 msgr2=0x562bee81bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:24.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.722+0000 7fbb31ada740 1 -- 172.21.15.33:0/4207650780 shutdown_connections 2024-09-16T00:32:24.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.722+0000 7fbb31ada740 1 -- 172.21.15.33:0/4207650780 wait complete. 2024-09-16T00:32:24.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb31ada740 1 -- start start 2024-09-16T00:32:24.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb31ada740 1 Processor -- start 2024-09-16T00:32:24.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb31ada740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee81f5d0 0x562bee829d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb31ada740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee827020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb31ada740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee827610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb31ada740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562bee82d070 con 0x562bee820b40 2024-09-16T00:32:24.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb31ada740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562bee740060 con 0x562bee82a370 2024-09-16T00:32:24.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb31ada740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562bee82d1f0 con 0x562bee81f5d0 2024-09-16T00:32:24.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb31ada740 10 client.0 _ll_register_callbacks cb 0x562bee4938c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:24.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb2cc9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee827020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb2cc9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee827020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39580/0 (socket says 172.21.15.33:39580) 2024-09-16T00:32:24.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb2cc9c640 1 -- 172.21.15.33:0/2708492906 learned_addr learned my addr 172.21.15.33:0/2708492906 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:24.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee81f5d0 0x562bee829d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.723+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee827610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb2cc9c640 1 -- 172.21.15.33:0/2708492906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee81f5d0 msgr2=0x562bee829d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:24.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb2cc9c640 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee81f5d0 0x562bee829d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb2cc9c640 1 -- 172.21.15.33:0/2708492906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 msgr2=0x562bee827610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:24.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb2cc9c640 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee827610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb2cc9c640 1 -- 172.21.15.33:0/2708492906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bee4f63c0 con 0x562bee820b40 2024-09-16T00:32:24.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb2cc9c640 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee827020 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fbb180116a0 tx=0x7fbb18011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f3f38d62298f77 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:24.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb10ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:32:24.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb10ff9640 1 -- 172.21.15.33:0/2708492906 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1801e020 con 0x562bee820b40 2024-09-16T00:32:24.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb10ff9640 1 -- 172.21.15.33:0/2708492906 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fbb1800fbf0 con 0x562bee820b40 2024-09-16T00:32:24.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb31ada740 1 -- 172.21.15.33:0/2708492906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb2001f070 con 0x562bee820b40 2024-09-16T00:32:24.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb10ff9640 1 -- 172.21.15.33:0/2708492906 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbb18026050 con 0x562bee820b40 2024-09-16T00:32:24.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb31ada740 10 client.18246 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:24.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.724+0000 7fbb31ada740 1 -- 172.21.15.33:0/2708492906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562bee838ba0 con 0x562bee820b40 2024-09-16T00:32:24.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.725+0000 7fbb10ff9640 1 -- 172.21.15.33:0/2708492906 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb2001f070 con 0x562bee820b40 2024-09-16T00:32:24.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.725+0000 7fbb10ff9640 1 -- 172.21.15.33:0/2708492906 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbb18019070 con 0x562bee820b40 2024-09-16T00:32:24.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.725+0000 7fbb31ada740 10 client.18246 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:24.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.725+0000 7fbb31ada740 10 client.18246 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:32:24.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.725+0000 7fbb31ada740 20 client.18246 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:24.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.725+0000 7fbb31ada740 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x562bee83a120 0x562bee83c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.725+0000 7fbb31ada740 4 client.18246 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:32:24.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.725+0000 7fbb31ada740 1 -- 172.21.15.33:0/2708492906 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bee792860 con 0x562bee83a120 2024-09-16T00:32:24.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.725+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x562bee83a120 0x562bee83c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.726+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x562bee83a120 0x562bee83c510 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5914e302f03589d7 server_cookie=907f186fce40c023 in_seq=0 out_seq=0 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.726+0000 7fbb10ff9640 10 client.18246 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb10ff9640 1 -- 172.21.15.33:0/2708492906 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562bee792860 con 0x562bee83a120 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb10ff9640 10 client.18246 handle_command_reply: tid=0 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb31ada740 1 client.18246 shutdown 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb31ada740 1 -- 172.21.15.33:0/2708492906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 msgr2=0x562bee827020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb31ada740 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee827020 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fbb180116a0 tx=0x7fbb18011b60 comp rx=0 tx=0).stop 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb31ada740 1 -- 172.21.15.33:0/2708492906 shutdown_connections 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb31ada740 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x562bee83a120 0x562bee83c510 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb31ada740 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee827610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb31ada740 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee827020 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb10ff9640 0 client.18246 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb31ada740 1 --2- 172.21.15.33:0/2708492906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee81f5d0 0x562bee829d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb31ada740 1 -- 172.21.15.33:0/2708492906 >> 172.21.15.33:0/2708492906 conn(0x562bee7e8310 msgr2=0x562bee821930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:24.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb31ada740 1 -- 172.21.15.33:0/2708492906 shutdown_connections 2024-09-16T00:32:24.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb31ada740 1 -- 172.21.15.33:0/2708492906 wait complete. 2024-09-16T00:32:24.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.728+0000 7fbb31ada740 20 client.18246 trim_cache size 0 max 16384 2024-09-16T00:32:24.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.745+0000 7fbb31ada740 1 Processor -- start 2024-09-16T00:32:24.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.745+0000 7fbb31ada740 1 -- start start 2024-09-16T00:32:24.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.745+0000 7fbb31ada740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee81f5d0 0x562bee836340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.745+0000 7fbb31ada740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee820b40 0x562bee826000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.745+0000 7fbb31ada740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee841e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.745+0000 7fbb31ada740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbb281af910 con 0x562bee81f5d0 2024-09-16T00:32:24.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.745+0000 7fbb31ada740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbb281ad180 con 0x562bee82a370 2024-09-16T00:32:24.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.745+0000 7fbb31ada740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbb281aafd0 con 0x562bee820b40 2024-09-16T00:32:24.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.746+0000 7fbb2dc9e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee81f5d0 0x562bee836340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.746+0000 7fbb2dc9e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee81f5d0 0x562bee836340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39596/0 (socket says 172.21.15.33:39596) 2024-09-16T00:32:24.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.746+0000 7fbb2dc9e640 1 -- 172.21.15.33:0/31232276 learned_addr learned my addr 172.21.15.33:0/31232276 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:24.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.746+0000 7fbb2cc9c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee820b40 0x562bee826000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.746+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/31232276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee841e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.746+0000 7fbb2cc9c640 1 -- 172.21.15.33:0/31232276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 msgr2=0x562bee841e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.746+0000 7fbb2cc9c640 1 --2- 172.21.15.33:0/31232276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee841e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.746+0000 7fbb2cc9c640 1 -- 172.21.15.33:0/31232276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee81f5d0 msgr2=0x562bee836340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.746+0000 7fbb2cc9c640 1 --2- 172.21.15.33:0/31232276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee81f5d0 0x562bee836340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.746+0000 7fbb2cc9c640 1 -- 172.21.15.33:0/31232276 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bee81bde0 con 0x562bee820b40 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.746+0000 7fbb2cc9c640 1 --2- 172.21.15.33:0/31232276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee820b40 0x562bee826000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x562bee828660 tx=0x7fbb18009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=703d42024a7e1799 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.747+0000 7fbb12ffd640 1 -- 172.21.15.33:0/31232276 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1802f070 con 0x562bee820b40 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.747+0000 7fbb12ffd640 1 -- 172.21.15.33:0/31232276 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb18005510 con 0x562bee820b40 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.747+0000 7fbb12ffd640 1 -- 172.21.15.33:0/31232276 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1801e040 con 0x562bee820b40 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.747+0000 7fbb31ada740 1 -- 172.21.15.33:0/31232276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee820b40 msgr2=0x562bee826000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.747+0000 7fbb31ada740 1 --2- 172.21.15.33:0/31232276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee820b40 0x562bee826000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x562bee828660 tx=0x7fbb18009f70 comp rx=0 tx=0).stop 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.747+0000 7fbb31ada740 1 -- 172.21.15.33:0/31232276 shutdown_connections 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.747+0000 7fbb31ada740 1 --2- 172.21.15.33:0/31232276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee82a370 0x562bee841e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.747+0000 7fbb31ada740 1 --2- 172.21.15.33:0/31232276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee820b40 0x562bee826000 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.747+0000 7fbb31ada740 1 --2- 172.21.15.33:0/31232276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee81f5d0 0x562bee836340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.747+0000 7fbb31ada740 1 -- 172.21.15.33:0/31232276 >> 172.21.15.33:0/31232276 conn(0x562bee7e8310 msgr2=0x562bee8258f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.747+0000 7fbb31ada740 1 -- 172.21.15.33:0/31232276 shutdown_connections 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.747+0000 7fbb31ada740 1 -- 172.21.15.33:0/31232276 wait complete. 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb31ada740 1 -- start start 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb31ada740 1 Processor -- start 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb31ada740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee81f5d0 0x562bee832930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb31ada740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee82dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb31ada740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee82a370 0x562bee82e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb31ada740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbb281ab640 con 0x562bee820b40 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb31ada740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbb281acab0 con 0x562bee81f5d0 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb31ada740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbb281abd10 con 0x562bee82a370 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb31ada740 10 client.0 _ll_register_callbacks cb 0x7fbb18009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:24.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb2cc9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee82dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb2cc9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee82dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39612/0 (socket says 172.21.15.33:39612) 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb2cc9c640 1 -- 172.21.15.33:0/888652330 learned_addr learned my addr 172.21.15.33:0/888652330 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.748+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee81f5d0 0x562bee832930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee82a370 0x562bee82e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb2cc9c640 1 -- 172.21.15.33:0/888652330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee82a370 msgr2=0x562bee82e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb2cc9c640 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee82a370 0x562bee82e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb2cc9c640 1 -- 172.21.15.33:0/888652330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee81f5d0 msgr2=0x562bee832930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb2cc9c640 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee81f5d0 0x562bee832930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb2cc9c640 1 -- 172.21.15.33:0/888652330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bee82e6e0 con 0x562bee820b40 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee82a370 0x562bee82e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb2cc9c640 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee82dc60 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800fed0 tx=0x7fbb1800ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc4aca855367e25 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb10ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:32:24.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb10ff9640 1 -- 172.21.15.33:0/888652330 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1802f040 con 0x562bee820b40 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb10ff9640 1 -- 172.21.15.33:0/888652330 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fbb18019620 con 0x562bee820b40 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb31ada740 1 -- 172.21.15.33:0/888652330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb18005510 con 0x562bee820b40 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb10ff9640 1 -- 172.21.15.33:0/888652330 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbb1801e040 con 0x562bee820b40 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb31ada740 10 client.18252 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.749+0000 7fbb31ada740 1 -- 172.21.15.33:0/888652330 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562bee8ad330 con 0x562bee820b40 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.750+0000 7fbb10ff9640 1 -- 172.21.15.33:0/888652330 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb18005510 con 0x562bee820b40 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.750+0000 7fbb10ff9640 1 -- 172.21.15.33:0/888652330 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbb1802a8f0 con 0x562bee820b40 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.750+0000 7fbb31ada740 10 client.18252 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.750+0000 7fbb31ada740 10 client.18252 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.750+0000 7fbb31ada740 20 client.18252 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.750+0000 7fbb31ada740 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x562bee83a120 0x562bee8afa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.750+0000 7fbb31ada740 4 client.18252 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:32:24.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.750+0000 7fbb31ada740 1 -- 172.21.15.33:0/888652330 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562bee78cd70 con 0x562bee83a120 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.750+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x562bee83a120 0x562bee8afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.750+0000 7fbb2dc9e640 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x562bee83a120 0x562bee8afa00 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bac1ca1e2674d513 server_cookie=57e87f10e23bceaf in_seq=0 out_seq=0 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.750+0000 7fbb10ff9640 10 client.18252 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.754+0000 7fbb10ff9640 1 -- 172.21.15.33:0/888652330 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562bee78cd70 con 0x562bee83a120 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.754+0000 7fbb10ff9640 10 client.18252 handle_command_reply: tid=0 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.754+0000 7fbb31ada740 1 client.18252 shutdown 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.755+0000 7fbb31ada740 1 -- 172.21.15.33:0/888652330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 msgr2=0x562bee82dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.755+0000 7fbb31ada740 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee82dc60 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800fed0 tx=0x7fbb1800ff00 comp rx=0 tx=0).stop 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.755+0000 7fbb31ada740 1 -- 172.21.15.33:0/888652330 shutdown_connections 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.755+0000 7fbb31ada740 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x562bee83a120 0x562bee8afa00 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.755+0000 7fbb31ada740 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bee82a370 0x562bee82e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.755+0000 7fbb31ada740 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562bee820b40 0x562bee82dc60 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.755+0000 7fbb31ada740 1 --2- 172.21.15.33:0/888652330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562bee81f5d0 0x562bee832930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.755+0000 7fbb31ada740 1 -- 172.21.15.33:0/888652330 >> 172.21.15.33:0/888652330 conn(0x562bee7e8310 msgr2=0x562bee826110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:24.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.755+0000 7fbb10ff9640 0 client.18252 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:32:24.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.755+0000 7fbb31ada740 1 -- 172.21.15.33:0/888652330 shutdown_connections 2024-09-16T00:32:24.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.755+0000 7fbb31ada740 1 -- 172.21.15.33:0/888652330 wait complete. 2024-09-16T00:32:24.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.755+0000 7fbb31ada740 20 client.18252 trim_cache size 0 max 16384 2024-09-16T00:32:24.766 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:32:24.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.756+0000 7fbb2f701640 1 -- 172.21.15.33:0/3382561814 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbb140a3240 msgr2=0x7fbb140a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:24.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.756+0000 7fbb2f701640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbb140a3240 0x7fbb140a5700 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbb24004650 tx=0x7fbb240dc040 comp rx=0 tx=0).stop 2024-09-16T00:32:24.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.756+0000 7fbb2f701640 1 -- 172.21.15.33:0/3382561814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb28102d90 msgr2=0x7fbb28198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:24.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.756+0000 7fbb2f701640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb28102d90 0x7fbb28198450 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbb2402a970 tx=0x7fbb24033d90 comp rx=0 tx=0).stop 2024-09-16T00:32:24.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.756+0000 7fbb2f701640 1 -- 172.21.15.33:0/3382561814 shutdown_connections 2024-09-16T00:32:24.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.756+0000 7fbb2f701640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fbb140a3240 0x7fbb140a5700 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.756+0000 7fbb2f701640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbb28108290 0x7fbb2819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.756+0000 7fbb2f701640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb28103740 0x7fbb28198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.756+0000 7fbb2f701640 1 --2- 172.21.15.33:0/3382561814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb28102d90 0x7fbb28198450 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:24.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.756+0000 7fbb2f701640 1 -- 172.21.15.33:0/3382561814 >> 172.21.15.33:0/3382561814 conn(0x7fbb280fc7d0 msgr2=0x7fbb280fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:24.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.756+0000 7fbb2f701640 1 -- 172.21.15.33:0/3382561814 shutdown_connections 2024-09-16T00:32:24.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:24.756+0000 7fbb2f701640 1 -- 172.21.15.33:0/3382561814 wait complete. 2024-09-16T00:32:25.072 INFO:tasks.workunit.client.0.smithi033.stdout: 10 34762 22.43 MB/sec warmup 91 sec latency 94.485 ms 2024-09-16T00:32:25.573 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:32:25.882 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:32:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:25 smithi083 ceph-mon[35242]: pgmap v808: 129 pgs: 129 active+clean; 520 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 13 MiB/s wr, 649 op/s 2024-09-16T00:32:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:25 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:32:25.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:25 smithi060 ceph-mon[35185]: pgmap v808: 129 pgs: 129 active+clean; 520 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 13 MiB/s wr, 649 op/s 2024-09-16T00:32:25.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:25 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:32:26.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:25 smithi033 ceph-mon[31243]: pgmap v808: 129 pgs: 129 active+clean; 520 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 13 MiB/s wr, 649 op/s 2024-09-16T00:32:26.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:25 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:32:26.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 35417 22.69 MB/sec warmup 92 sec latency 77.615 ms 2024-09-16T00:32:26.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:26 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:32:26.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:26 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:32:26.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:26 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:32:27.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:26 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:32:27.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:26 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:32:27.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:26 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:32:27.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 36000 22.70 MB/sec warmup 93 sec latency 68.542 ms 2024-09-16T00:32:27.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:26 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:32:27.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:26 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:32:27.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:26 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:32:27.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.414+0000 7fd8f42d2640 1 -- 172.21.15.33:0/2720715716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd8ec101e10 msgr2=0x7fd8ec1021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:27.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.414+0000 7fd8f42d2640 1 --2- 172.21.15.33:0/2720715716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd8ec101e10 0x7fd8ec1021f0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009900 tx=0x7fd8e802a410 comp rx=0 tx=0).stop 2024-09-16T00:32:27.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.414+0000 7fd8f42d2640 1 -- 172.21.15.33:0/2720715716 shutdown_connections 2024-09-16T00:32:27.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.414+0000 7fd8f42d2640 1 --2- 172.21.15.33:0/2720715716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8ec106f40 0x7fd8ec10eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.414+0000 7fd8f42d2640 1 --2- 172.21.15.33:0/2720715716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8ec1027c0 0x7fd8ec106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.414+0000 7fd8f42d2640 1 --2- 172.21.15.33:0/2720715716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd8ec101e10 0x7fd8ec1021f0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.414+0000 7fd8f42d2640 1 -- 172.21.15.33:0/2720715716 >> 172.21.15.33:0/2720715716 conn(0x7fd8ec0fd8d0 msgr2=0x7fd8ec0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:27.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.414+0000 7fd8f42d2640 1 -- 172.21.15.33:0/2720715716 shutdown_connections 2024-09-16T00:32:27.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.414+0000 7fd8f42d2640 1 -- 172.21.15.33:0/2720715716 wait complete. 2024-09-16T00:32:27.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.415+0000 7fd8f42d2640 1 Processor -- start 2024-09-16T00:32:27.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.415+0000 7fd8f42d2640 1 -- start start 2024-09-16T00:32:27.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.415+0000 7fd8f42d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8ec101e10 0x7fd8ec198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.415+0000 7fd8f42d2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8ec1027c0 0x7fd8ec1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.415+0000 7fd8f42d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd8ec106f40 0x7fd8ec19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.415+0000 7fd8f42d2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec111fa0 con 0x7fd8ec106f40 2024-09-16T00:32:27.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.415+0000 7fd8f42d2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec111e20 con 0x7fd8ec101e10 2024-09-16T00:32:27.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.415+0000 7fd8f42d2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec112120 con 0x7fd8ec1027c0 2024-09-16T00:32:27.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f206e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8ec101e10 0x7fd8ec198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f286f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd8ec106f40 0x7fd8ec19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f186d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8ec1027c0 0x7fd8ec1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f206e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8ec101e10 0x7fd8ec198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:43800/0 (socket says 172.21.15.33:43800) 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f286f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd8ec106f40 0x7fd8ec19cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39618/0 (socket says 172.21.15.33:39618) 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f206e640 1 -- 172.21.15.33:0/3555456791 learned_addr learned my addr 172.21.15.33:0/3555456791 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f186d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8ec1027c0 0x7fd8ec1989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60148/0 (socket says 172.21.15.33:60148) 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f286f640 1 -- 172.21.15.33:0/3555456791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8ec1027c0 msgr2=0x7fd8ec1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f286f640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8ec1027c0 0x7fd8ec1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f286f640 1 -- 172.21.15.33:0/3555456791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8ec101e10 msgr2=0x7fd8ec198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f286f640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8ec101e10 0x7fd8ec198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f286f640 1 -- 172.21.15.33:0/3555456791 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ec19d410 con 0x7fd8ec106f40 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f186d640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8ec1027c0 0x7fd8ec1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f286f640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd8ec106f40 0x7fd8ec19cd30 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4009fd0 tx=0x7fd8e400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8e37fe640 1 -- 172.21.15.33:0/3555456791 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e400ce20 con 0x7fd8ec106f40 2024-09-16T00:32:27.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8e37fe640 1 -- 172.21.15.33:0/3555456791 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8e4005020 con 0x7fd8ec106f40 2024-09-16T00:32:27.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8f42d2640 1 -- 172.21.15.33:0/3555456791 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8ec19d700 con 0x7fd8ec106f40 2024-09-16T00:32:27.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.416+0000 7fd8e37fe640 1 -- 172.21.15.33:0/3555456791 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4010720 con 0x7fd8ec106f40 2024-09-16T00:32:27.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.417+0000 7fd8f42d2640 1 -- 172.21.15.33:0/3555456791 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8ec1a4fe0 con 0x7fd8ec106f40 2024-09-16T00:32:27.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.419+0000 7fd8e37fe640 1 -- 172.21.15.33:0/3555456791 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd8e4005540 con 0x7fd8ec106f40 2024-09-16T00:32:27.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.419+0000 7fd8f66ab740 1 Processor -- start 2024-09-16T00:32:27.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.419+0000 7fd8f66ab740 1 -- start start 2024-09-16T00:32:27.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f66ab740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea8bb40 0x55f7bea8a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f66ab740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8a5d0 0x55f7bea8aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f66ab740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea95370 0x55f7bea97760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f66ab740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f7be9f02f0 con 0x55f7bea8bb40 2024-09-16T00:32:27.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f66ab740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f7be997cf0 con 0x55f7bea8a5d0 2024-09-16T00:32:27.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f66ab740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f7be9ec2b0 con 0x55f7bea95370 2024-09-16T00:32:27.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f186d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea8bb40 0x55f7bea8a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f206e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8a5d0 0x55f7bea8aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f206e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8a5d0 0x55f7bea8aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:43812/0 (socket says 172.21.15.33:43812) 2024-09-16T00:32:27.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f206e640 1 -- 172.21.15.33:0/1523083252 learned_addr learned my addr 172.21.15.33:0/1523083252 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:27.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f186d640 1 --2- 172.21.15.33:0/1523083252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea95370 0x55f7bea97760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f206e640 1 -- 172.21.15.33:0/1523083252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea95370 msgr2=0x55f7bea97760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:27.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f206e640 1 --2- 172.21.15.33:0/1523083252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea95370 0x55f7bea97760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f206e640 1 -- 172.21.15.33:0/1523083252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea8bb40 msgr2=0x55f7bea8a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:27.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f206e640 1 --2- 172.21.15.33:0/1523083252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea8bb40 0x55f7bea8a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f206e640 1 -- 172.21.15.33:0/1523083252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7be97c3a0 con 0x55f7bea8a5d0 2024-09-16T00:32:27.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8e37fe640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd8d80a3240 0x7fd8d80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f206e640 1 --2- 172.21.15.33:0/1523083252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8a5d0 0x55f7bea8aa10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009a20 tx=0x7fd8e8005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5922a4d3fe5d834f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:27.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8f186d640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd8d80a3240 0x7fd8d80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.420+0000 7fd8e37fe640 1 -- 172.21.15.33:0/3555456791 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd8e40d0410 con 0x7fd8ec106f40 2024-09-16T00:32:27.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.421+0000 7fd8e0dea640 1 -- 172.21.15.33:0/1523083252 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8040070 con 0x55f7bea8a5d0 2024-09-16T00:32:27.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.421+0000 7fd8e0dea640 1 -- 172.21.15.33:0/1523083252 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8e8004030 con 0x55f7bea8a5d0 2024-09-16T00:32:27.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.421+0000 7fd8e0dea640 1 -- 172.21.15.33:0/1523083252 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8035040 con 0x55f7bea8a5d0 2024-09-16T00:32:27.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.421+0000 7fd8f66ab740 1 -- 172.21.15.33:0/1523083252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8a5d0 msgr2=0x55f7bea8aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:27.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.421+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/1523083252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8a5d0 0x55f7bea8aa10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009a20 tx=0x7fd8e8005dd0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.421+0000 7fd8f186d640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd8d80a3240 0x7fd8d80a5700 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009710 tx=0x7fd8dc015040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:27.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.421+0000 7fd8f66ab740 1 -- 172.21.15.33:0/1523083252 shutdown_connections 2024-09-16T00:32:27.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.421+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/1523083252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea95370 0x55f7bea97760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.421+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/1523083252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8a5d0 0x55f7bea8aa10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.421+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/1523083252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea8bb40 0x55f7bea8a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.421+0000 7fd8f66ab740 1 -- 172.21.15.33:0/1523083252 >> 172.21.15.33:0/1523083252 conn(0x55f7bea53310 msgr2=0x55f7bea86c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:27.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f66ab740 1 -- 172.21.15.33:0/1523083252 shutdown_connections 2024-09-16T00:32:27.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f66ab740 1 -- 172.21.15.33:0/1523083252 wait complete. 2024-09-16T00:32:27.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f66ab740 1 -- start start 2024-09-16T00:32:27.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f66ab740 1 Processor -- start 2024-09-16T00:32:27.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f66ab740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7bea94d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f66ab740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8bb40 0x55f7bea92010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f66ab740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 0x55f7bea92600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f66ab740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f7bea98010 con 0x55f7bea95370 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f66ab740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f7be9ab060 con 0x55f7bea8bb40 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f66ab740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f7bea98190 con 0x55f7bea8a5d0 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f66ab740 10 client.0 _ll_register_callbacks cb 0x55f7be6fe8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f286f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 0x55f7bea92600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f206e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7bea94d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f286f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 0x55f7bea92600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39640/0 (socket says 172.21.15.33:39640) 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f206e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7bea94d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60170/0 (socket says 172.21.15.33:60170) 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.422+0000 7fd8f286f640 1 -- 172.21.15.33:0/194353755 learned_addr learned my addr 172.21.15.33:0/194353755 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8f186d640 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8bb40 0x55f7bea92010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8f286f640 1 -- 172.21.15.33:0/194353755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 msgr2=0x55f7bea94d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8f286f640 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7bea94d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8f286f640 1 -- 172.21.15.33:0/194353755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8bb40 msgr2=0x55f7bea92010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:27.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8f286f640 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8bb40 0x55f7bea92010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8f286f640 1 -- 172.21.15.33:0/194353755 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7be7613c0 con 0x55f7bea95370 2024-09-16T00:32:27.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8f186d640 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8bb40 0x55f7bea92010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:32:27.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8f286f640 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 0x55f7bea92600 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fd8e40a4cf0 tx=0x7fd8e409cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9271ddd819720a32 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:27.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:32:27.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8d5ffb640 1 -- 172.21.15.33:0/194353755 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e40a5220 con 0x55f7bea95370 2024-09-16T00:32:27.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8f66ab740 1 -- 172.21.15.33:0/194353755 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8e8004030 con 0x55f7bea95370 2024-09-16T00:32:27.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8d5ffb640 1 -- 172.21.15.33:0/194353755 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd8e40051d0 con 0x55f7bea95370 2024-09-16T00:32:27.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8d5ffb640 1 -- 172.21.15.33:0/194353755 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd8e40a4d90 con 0x55f7bea95370 2024-09-16T00:32:27.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8f66ab740 10 client.18264 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:27.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8f66ab740 1 -- 172.21.15.33:0/194353755 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f7beaa3b10 con 0x55f7bea95370 2024-09-16T00:32:27.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.423+0000 7fd8d5ffb640 1 -- 172.21.15.33:0/194353755 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8e8004030 con 0x55f7bea95370 2024-09-16T00:32:27.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.424+0000 7fd8d5ffb640 1 -- 172.21.15.33:0/194353755 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd8e40b0ad0 con 0x55f7bea95370 2024-09-16T00:32:27.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.424+0000 7fd8f66ab740 10 client.18264 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:27.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.424+0000 7fd8f66ab740 10 client.18264 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:32:27.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.424+0000 7fd8f66ab740 20 client.18264 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:27.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.424+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f7beaa5020 0x55f7beaa7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.424+0000 7fd8f66ab740 4 client.18264 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:32:27.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.424+0000 7fd8f66ab740 1 -- 172.21.15.33:0/194353755 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7be9fd860 con 0x55f7beaa5020 2024-09-16T00:32:27.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.424+0000 7fd8f206e640 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f7beaa5020 0x55f7beaa7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.424+0000 7fd8f206e640 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f7beaa5020 0x55f7beaa7470 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fc1bd7fc9f96badf server_cookie=90af32387959cdc9 in_seq=0 out_seq=0 2024-09-16T00:32:27.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.424+0000 7fd8d5ffb640 10 client.18264 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:32:27.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8d5ffb640 1 -- 172.21.15.33:0/194353755 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f7be9fd860 con 0x55f7beaa5020 2024-09-16T00:32:27.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8d5ffb640 10 client.18264 handle_command_reply: tid=0 2024-09-16T00:32:27.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8f66ab740 1 client.18264 shutdown 2024-09-16T00:32:27.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8f66ab740 1 -- 172.21.15.33:0/194353755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 msgr2=0x55f7bea92600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:27.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 0x55f7bea92600 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fd8e40a4cf0 tx=0x7fd8e409cd70 comp rx=0 tx=0).stop 2024-09-16T00:32:27.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8f66ab740 1 -- 172.21.15.33:0/194353755 shutdown_connections 2024-09-16T00:32:27.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 0x55f7bea92600 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f7beaa5020 0x55f7beaa7470 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8bb40 0x55f7bea92010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/194353755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7bea94d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8d5ffb640 0 client.18264 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:32:27.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8f66ab740 1 -- 172.21.15.33:0/194353755 >> 172.21.15.33:0/194353755 conn(0x55f7bea53310 msgr2=0x55f7bea8c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:27.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8f66ab740 1 -- 172.21.15.33:0/194353755 shutdown_connections 2024-09-16T00:32:27.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.425+0000 7fd8f66ab740 1 -- 172.21.15.33:0/194353755 wait complete. 2024-09-16T00:32:27.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.426+0000 7fd8f66ab740 20 client.18264 trim_cache size 0 max 16384 2024-09-16T00:32:27.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.439+0000 7fd8f66ab740 1 Processor -- start 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.439+0000 7fd8f66ab740 1 -- start start 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.439+0000 7fd8f66ab740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7beaa12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.439+0000 7fd8f66ab740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea8bb40 0x55f7bea86700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.439+0000 7fd8f66ab740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea95370 0x55f7bea86c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.439+0000 7fd8f66ab740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1b1960 con 0x55f7bea8bb40 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.439+0000 7fd8f66ab740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1af1d0 con 0x55f7bea95370 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.439+0000 7fd8f66ab740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1ad020 con 0x55f7bea8a5d0 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8f186d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea8bb40 0x55f7bea86700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8f186d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea8bb40 0x55f7bea86700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39648/0 (socket says 172.21.15.33:39648) 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8f186d640 1 -- 172.21.15.33:0/1956323739 learned_addr learned my addr 172.21.15.33:0/1956323739 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8f206e640 1 --2- 172.21.15.33:0/1956323739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7beaa12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8f286f640 1 --2- 172.21.15.33:0/1956323739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea95370 0x55f7bea86c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8f186d640 1 -- 172.21.15.33:0/1956323739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 msgr2=0x55f7beaa12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:27.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8f186d640 1 --2- 172.21.15.33:0/1956323739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7beaa12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8f186d640 1 -- 172.21.15.33:0/1956323739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea95370 msgr2=0x55f7bea86c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:27.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8f186d640 1 --2- 172.21.15.33:0/1956323739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea95370 0x55f7bea86c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8f186d640 1 -- 172.21.15.33:0/1956323739 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7bea98180 con 0x55f7bea8bb40 2024-09-16T00:32:27.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8f186d640 1 --2- 172.21.15.33:0/1956323739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea8bb40 0x55f7bea86700 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc0058c0 tx=0x7fd8dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b46be5a56055f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:27.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8d7fff640 1 -- 172.21.15.33:0/1956323739 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc010ca0 con 0x55f7bea8bb40 2024-09-16T00:32:27.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8d7fff640 1 -- 172.21.15.33:0/1956323739 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8dc025030 con 0x55f7bea8bb40 2024-09-16T00:32:27.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.440+0000 7fd8d7fff640 1 -- 172.21.15.33:0/1956323739 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc0218d0 con 0x55f7bea8bb40 2024-09-16T00:32:27.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.441+0000 7fd8f66ab740 1 -- 172.21.15.33:0/1956323739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea8bb40 msgr2=0x55f7bea86700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:27.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.441+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/1956323739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea8bb40 0x55f7bea86700 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc0058c0 tx=0x7fd8dc004730 comp rx=0 tx=0).stop 2024-09-16T00:32:27.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.441+0000 7fd8f66ab740 1 -- 172.21.15.33:0/1956323739 shutdown_connections 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.441+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/1956323739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea95370 0x55f7bea86c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.441+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/1956323739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea8bb40 0x55f7bea86700 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.441+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/1956323739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7beaa12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.441+0000 7fd8f66ab740 1 -- 172.21.15.33:0/1956323739 >> 172.21.15.33:0/1956323739 conn(0x55f7bea53310 msgr2=0x55f7bea91fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.441+0000 7fd8f66ab740 1 -- 172.21.15.33:0/1956323739 shutdown_connections 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.441+0000 7fd8f66ab740 1 -- 172.21.15.33:0/1956323739 wait complete. 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.441+0000 7fd8f66ab740 1 -- start start 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f66ab740 1 Processor -- start 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f66ab740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7bea8efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f66ab740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8bb40 0x55f7bea8f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f66ab740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 0x55f7bea8fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f66ab740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1ad690 con 0x55f7bea95370 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f66ab740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1aeb00 con 0x55f7bea8bb40 2024-09-16T00:32:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f66ab740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1add60 con 0x55f7bea8a5d0 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f66ab740 10 client.0 _ll_register_callbacks cb 0x7fd8e4005830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f206e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7bea8efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f186d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8bb40 0x55f7bea8f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f206e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7bea8efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60182/0 (socket says 172.21.15.33:60182) 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f206e640 1 -- 172.21.15.33:0/3942028013 learned_addr learned my addr 172.21.15.33:0/3942028013 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f286f640 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 0x55f7bea8fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f186d640 1 -- 172.21.15.33:0/3942028013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 msgr2=0x55f7bea8efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f186d640 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7bea8efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f186d640 1 -- 172.21.15.33:0/3942028013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 msgr2=0x55f7bea8fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f186d640 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 0x55f7bea8fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f186d640 1 -- 172.21.15.33:0/3942028013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7beaa17f0 con 0x55f7bea8bb40 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f286f640 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 0x55f7bea8fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.442+0000 7fd8f206e640 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7bea8efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.443+0000 7fd8f186d640 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8bb40 0x55f7bea8f520 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55f7bea931d0 tx=0x7fd8dc010e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be2a6145a809c07d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.443+0000 7fd8d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.443+0000 7fd8d5ffb640 1 -- 172.21.15.33:0/3942028013 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc0035f0 con 0x55f7bea8bb40 2024-09-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.443+0000 7fd8f66ab740 1 -- 172.21.15.33:0/3942028013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8dc025030 con 0x55f7bea8bb40 2024-09-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.443+0000 7fd8d5ffb640 1 -- 172.21.15.33:0/3942028013 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd8dc003790 con 0x55f7bea8bb40 2024-09-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.443+0000 7fd8d5ffb640 1 -- 172.21.15.33:0/3942028013 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd8dc0039d0 con 0x55f7bea8bb40 2024-09-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.443+0000 7fd8f66ab740 10 client.27605 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.443+0000 7fd8f66ab740 1 -- 172.21.15.33:0/3942028013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f7beb14170 con 0x55f7bea8bb40 2024-09-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.443+0000 7fd8d5ffb640 1 -- 172.21.15.33:0/3942028013 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8dc025030 con 0x55f7bea8bb40 2024-09-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.443+0000 7fd8d5ffb640 1 -- 172.21.15.33:0/3942028013 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd8dc021bb0 con 0x55f7bea8bb40 2024-09-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.444+0000 7fd8f66ab740 10 client.27605 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.444+0000 7fd8f66ab740 10 client.27605 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:32:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.444+0000 7fd8f66ab740 20 client.27605 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.444+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f7beb14780 0x55f7beb16ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.444+0000 7fd8f66ab740 4 client.27605 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:32:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.444+0000 7fd8f66ab740 1 -- 172.21.15.33:0/3942028013 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f7be9f7d70 con 0x55f7beb14780 2024-09-16T00:32:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.444+0000 7fd8f206e640 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f7beb14780 0x55f7beb16ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.444+0000 7fd8f206e640 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f7beb14780 0x55f7beb16ba0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=27d12e030d038202 server_cookie=5071aba46bb2242a in_seq=0 out_seq=0 2024-09-16T00:32:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.444+0000 7fd8d5ffb640 10 client.27605 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:32:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.444+0000 7fd8d5ffb640 1 -- 172.21.15.33:0/3942028013 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f7be9f7d70 con 0x55f7beb14780 2024-09-16T00:32:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.444+0000 7fd8d5ffb640 10 client.27605 handle_command_reply: tid=0 2024-09-16T00:32:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.444+0000 7fd8f66ab740 1 client.27605 shutdown 2024-09-16T00:32:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f66ab740 1 -- 172.21.15.33:0/3942028013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8bb40 msgr2=0x55f7bea8f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:27.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8bb40 0x55f7bea8f520 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55f7bea931d0 tx=0x7fd8dc010e80 comp rx=0 tx=0).stop 2024-09-16T00:32:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f66ab740 1 -- 172.21.15.33:0/3942028013 shutdown_connections 2024-09-16T00:32:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7bea95370 0x55f7bea8fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f7beb14780 0x55f7beb16ba0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f7bea8bb40 0x55f7bea8f520 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f66ab740 1 --2- 172.21.15.33:0/3942028013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7bea8a5d0 0x55f7bea8efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8d5ffb640 0 client.27605 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:32:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f66ab740 1 -- 172.21.15.33:0/3942028013 >> 172.21.15.33:0/3942028013 conn(0x55f7bea53310 msgr2=0x55f7bea91fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f66ab740 1 -- 172.21.15.33:0/3942028013 shutdown_connections 2024-09-16T00:32:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f66ab740 1 -- 172.21.15.33:0/3942028013 wait complete. 2024-09-16T00:32:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f66ab740 20 client.27605 trim_cache size 0 max 16384 2024-09-16T00:32:27.451 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:32:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f42d2640 1 -- 172.21.15.33:0/3555456791 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd8d80a3240 msgr2=0x7fd8d80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f42d2640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd8d80a3240 0x7fd8d80a5700 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009710 tx=0x7fd8dc015040 comp rx=0 tx=0).stop 2024-09-16T00:32:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f42d2640 1 -- 172.21.15.33:0/3555456791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd8ec106f40 msgr2=0x7fd8ec19cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:27.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f42d2640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd8ec106f40 0x7fd8ec19cd30 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4009fd0 tx=0x7fd8e400ef10 comp rx=0 tx=0).stop 2024-09-16T00:32:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f42d2640 1 -- 172.21.15.33:0/3555456791 shutdown_connections 2024-09-16T00:32:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f42d2640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd8d80a3240 0x7fd8d80a5700 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f42d2640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd8ec106f40 0x7fd8ec19cd30 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f42d2640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8ec1027c0 0x7fd8ec1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f42d2640 1 --2- 172.21.15.33:0/3555456791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8ec101e10 0x7fd8ec198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.445+0000 7fd8f42d2640 1 -- 172.21.15.33:0/3555456791 >> 172.21.15.33:0/3555456791 conn(0x7fd8ec0fd8d0 msgr2=0x7fd8ec0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.446+0000 7fd8f42d2640 1 -- 172.21.15.33:0/3555456791 shutdown_connections 2024-09-16T00:32:27.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:27.446+0000 7fd8f42d2640 1 -- 172.21.15.33:0/3555456791 wait complete. 2024-09-16T00:32:27.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:27 smithi060 ceph-mon[35185]: pgmap v809: 129 pgs: 129 active+clean; 519 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 15 MiB/s wr, 713 op/s 2024-09-16T00:32:28.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:27 smithi033 ceph-mon[31243]: pgmap v809: 129 pgs: 129 active+clean; 519 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 15 MiB/s wr, 713 op/s 2024-09-16T00:32:28.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:27 smithi083 ceph-mon[35242]: pgmap v809: 129 pgs: 129 active+clean; 519 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 15 MiB/s wr, 713 op/s 2024-09-16T00:32:28.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 36629 22.84 MB/sec warmup 94 sec latency 88.867 ms 2024-09-16T00:32:28.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:32:28.499 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:32:29.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.052+0000 7f898c3c2640 1 -- 172.21.15.33:0/2944876294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8984103650 msgr2=0x7f898410bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:29.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.052+0000 7f898c3c2640 1 --2- 172.21.15.33:0/2944876294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8984103650 0x7f898410bb40 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f8974009900 tx=0x7f897402a490 comp rx=0 tx=0).stop 2024-09-16T00:32:29.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.053+0000 7f898c3c2640 1 -- 172.21.15.33:0/2944876294 shutdown_connections 2024-09-16T00:32:29.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.053+0000 7f898c3c2640 1 --2- 172.21.15.33:0/2944876294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f898410c160 0x7f898410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.053+0000 7f898c3c2640 1 --2- 172.21.15.33:0/2944876294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8984103650 0x7f898410bb40 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.053+0000 7f898c3c2640 1 --2- 172.21.15.33:0/2944876294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8984102d30 0x7f8984103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.053+0000 7f898c3c2640 1 -- 172.21.15.33:0/2944876294 >> 172.21.15.33:0/2944876294 conn(0x7f89840fc7d0 msgr2=0x7f89840febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:29.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.055+0000 7f898c3c2640 1 -- 172.21.15.33:0/2944876294 shutdown_connections 2024-09-16T00:32:29.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.055+0000 7f898c3c2640 1 -- 172.21.15.33:0/2944876294 wait complete. 2024-09-16T00:32:29.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.056+0000 7f898c3c2640 1 Processor -- start 2024-09-16T00:32:29.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.056+0000 7f898c3c2640 1 -- start start 2024-09-16T00:32:29.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.057+0000 7f898c3c2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8984102d30 0x7f8984198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.057+0000 7f898c3c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8984103650 0x7f89841989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.057+0000 7f898c3c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f898410c160 0x7f898419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.057+0000 7f898c3c2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8984111fa0 con 0x7f8984103650 2024-09-16T00:32:29.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.057+0000 7f898c3c2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8984111e20 con 0x7f8984102d30 2024-09-16T00:32:29.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.057+0000 7f898c3c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8984112120 con 0x7f898410c160 2024-09-16T00:32:29.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.057+0000 7f898995d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8984103650 0x7f89841989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.058+0000 7f898995d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8984103650 0x7f89841989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39674/0 (socket says 172.21.15.33:39674) 2024-09-16T00:32:29.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.058+0000 7f898995d640 1 -- 172.21.15.33:0/2998600519 learned_addr learned my addr 172.21.15.33:0/2998600519 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:29.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.058+0000 7f898a95f640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f898410c160 0x7f898419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.058+0000 7f898a15e640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8984102d30 0x7f8984198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.058+0000 7f898995d640 1 -- 172.21.15.33:0/2998600519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f898410c160 msgr2=0x7f898419cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:29.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.058+0000 7f898995d640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f898410c160 0x7f898419cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.058+0000 7f898995d640 1 -- 172.21.15.33:0/2998600519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8984102d30 msgr2=0x7f8984198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:29.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.058+0000 7f898995d640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8984102d30 0x7f8984198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.058+0000 7f898995d640 1 -- 172.21.15.33:0/2998600519 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898419d410 con 0x7f8984103650 2024-09-16T00:32:29.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.058+0000 7f898a15e640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8984102d30 0x7f8984198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:32:29.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.059+0000 7f898995d640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8984103650 0x7f89841989a0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f8974006010 tx=0x7f89740045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:29.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.060+0000 7f897b7fe640 1 -- 172.21.15.33:0/2998600519 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8974040070 con 0x7f8984103650 2024-09-16T00:32:29.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.060+0000 7f897b7fe640 1 -- 172.21.15.33:0/2998600519 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8974031c80 con 0x7f8984103650 2024-09-16T00:32:29.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.060+0000 7f898c3c2640 1 -- 172.21.15.33:0/2998600519 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f898419d6a0 con 0x7f8984103650 2024-09-16T00:32:29.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.060+0000 7f897b7fe640 1 -- 172.21.15.33:0/2998600519 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8974035020 con 0x7f8984103650 2024-09-16T00:32:29.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.061+0000 7f898c3c2640 1 -- 172.21.15.33:0/2998600519 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89841a4fe0 con 0x7f8984103650 2024-09-16T00:32:29.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 37373 23.05 MB/sec warmup 95 sec latency 128.666 ms 2024-09-16T00:32:29.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.066+0000 7f898e79b740 1 Processor -- start 2024-09-16T00:32:29.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.067+0000 7f898e79b740 1 -- start start 2024-09-16T00:32:29.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.067+0000 7f898e79b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e530b30 0x55a06e530f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.067+0000 7f898e79b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e531450 0x55a06e52fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.067+0000 7f898e79b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 0x55a06e530440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.067+0000 7f898e79b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a06e4902f0 con 0x55a06e52ffc0 2024-09-16T00:32:29.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.067+0000 7f898e79b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a06e437cf0 con 0x55a06e530b30 2024-09-16T00:32:29.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.067+0000 7f898e79b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a06e48c2b0 con 0x55a06e531450 2024-09-16T00:32:29.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.067+0000 7f898a95f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e530b30 0x55a06e530f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.067+0000 7f898a95f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e530b30 0x55a06e530f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:43852/0 (socket says 172.21.15.33:43852) 2024-09-16T00:32:29.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.067+0000 7f898a95f640 1 -- 172.21.15.33:0/3064621725 learned_addr learned my addr 172.21.15.33:0/3064621725 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:29.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.069+0000 7f898a15e640 1 --2- 172.21.15.33:0/3064621725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e531450 0x55a06e52fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.071+0000 7f898a95f640 1 -- 172.21.15.33:0/3064621725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e531450 msgr2=0x55a06e52fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:29.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.071+0000 7f898a95f640 1 --2- 172.21.15.33:0/3064621725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e531450 0x55a06e52fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.071+0000 7f898a95f640 1 -- 172.21.15.33:0/3064621725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 msgr2=0x55a06e530440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:32:29.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.071+0000 7f898a95f640 1 --2- 172.21.15.33:0/3064621725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 0x55a06e530440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.071+0000 7f898a95f640 1 -- 172.21.15.33:0/3064621725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a06e41c3a0 con 0x55a06e530b30 2024-09-16T00:32:29.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.072+0000 7f898a95f640 1 --2- 172.21.15.33:0/3064621725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e530b30 0x55a06e530f10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f897c009f20 tx=0x7f897c02e8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=627f2de38d405b8a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:29.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.072+0000 7f8978dea640 1 -- 172.21.15.33:0/3064621725 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c03c070 con 0x55a06e530b30 2024-09-16T00:32:29.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.072+0000 7f8978dea640 1 -- 172.21.15.33:0/3064621725 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f897c0378f0 con 0x55a06e530b30 2024-09-16T00:32:29.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.072+0000 7f8978dea640 1 -- 172.21.15.33:0/3064621725 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c02c950 con 0x55a06e530b30 2024-09-16T00:32:29.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.073+0000 7f897b7fe640 1 -- 172.21.15.33:0/2998600519 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8974043070 con 0x7f8984103650 2024-09-16T00:32:29.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.073+0000 7f898e79b740 1 -- 172.21.15.33:0/3064621725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e530b30 msgr2=0x55a06e530f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:29.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.073+0000 7f898e79b740 1 --2- 172.21.15.33:0/3064621725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e530b30 0x55a06e530f10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f897c009f20 tx=0x7f897c02e8c0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.073+0000 7f898e79b740 1 -- 172.21.15.33:0/3064621725 shutdown_connections 2024-09-16T00:32:29.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.073+0000 7f898e79b740 1 --2- 172.21.15.33:0/3064621725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 0x55a06e530440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.073+0000 7f898e79b740 1 --2- 172.21.15.33:0/3064621725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e531450 0x55a06e52fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.073+0000 7f898e79b740 1 --2- 172.21.15.33:0/3064621725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e530b30 0x55a06e530f10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.073+0000 7f898e79b740 1 -- 172.21.15.33:0/3064621725 >> 172.21.15.33:0/3064621725 conn(0x55a06e4f3310 msgr2=0x55a06e5266e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:29.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.074+0000 7f898e79b740 1 -- 172.21.15.33:0/3064621725 shutdown_connections 2024-09-16T00:32:29.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.074+0000 7f898e79b740 1 -- 172.21.15.33:0/3064621725 wait complete. 2024-09-16T00:32:29.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.074+0000 7f897b7fe640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89700a3330 0x7f89700a57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.074+0000 7f898e79b740 1 -- start start 2024-09-16T00:32:29.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.074+0000 7f897b7fe640 1 -- 172.21.15.33:0/2998600519 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f89740f0a60 con 0x7f8984103650 2024-09-16T00:32:29.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.074+0000 7f898e79b740 1 Processor -- start 2024-09-16T00:32:29.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.075+0000 7f898e79b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e52ffc0 0x55a06e535390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.075+0000 7f898e79b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e530b30 0x55a06e5358d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.075+0000 7f898e79b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e531450 0x55a06e539a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.075+0000 7f898e79b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a06e534260 con 0x55a06e530b30 2024-09-16T00:32:29.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.075+0000 7f898e79b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a06e44b060 con 0x55a06e52ffc0 2024-09-16T00:32:29.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.075+0000 7f898e79b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a06e5343e0 con 0x55a06e531450 2024-09-16T00:32:29.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.075+0000 7f898e79b740 10 client.0 _ll_register_callbacks cb 0x55a06e19e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:29.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.075+0000 7f898a15e640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89700a3330 0x7f89700a57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.075+0000 7f898a95f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e531450 0x55a06e539a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.075+0000 7f898a95f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e531450 0x55a06e539a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60230/0 (socket says 172.21.15.33:60230) 2024-09-16T00:32:29.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.075+0000 7f898a95f640 1 -- 172.21.15.33:0/1364266869 learned_addr learned my addr 172.21.15.33:0/1364266869 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:29.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.076+0000 7f898a15e640 1 --2- 172.21.15.33:0/1364266869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e52ffc0 0x55a06e535390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.076+0000 7f898a95f640 1 -- 172.21.15.33:0/1364266869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e52ffc0 msgr2=0x55a06e535390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:29.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.076+0000 7f898a95f640 1 --2- 172.21.15.33:0/1364266869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e52ffc0 0x55a06e535390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.076+0000 7f898a95f640 1 -- 172.21.15.33:0/1364266869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e530b30 msgr2=0x55a06e5358d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:29.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.076+0000 7f898a95f640 1 --2- 172.21.15.33:0/1364266869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e530b30 0x55a06e5358d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.076+0000 7f898a95f640 1 -- 172.21.15.33:0/1364266869 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a06e2013c0 con 0x55a06e531450 2024-09-16T00:32:29.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.076+0000 7f898a95f640 1 --2- 172.21.15.33:0/1364266869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e531450 0x55a06e539a40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f897c02efd0 tx=0x7f897c02c550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1166d00aa42dd609 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:29.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.077+0000 7f898a15e640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89700a3330 0x7f89700a57f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x55a06e527a40 tx=0x7f898000aaa0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.078+0000 7f896dffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:32:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.078+0000 7f896dffb640 1 -- 172.21.15.33:0/1364266869 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c03c040 con 0x55a06e531450 2024-09-16T00:32:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.078+0000 7f898e79b740 1 -- 172.21.15.33:0/1364266869 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f897c0378f0 con 0x55a06e531450 2024-09-16T00:32:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.078+0000 7f896dffb640 1 -- 172.21.15.33:0/1364266869 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f897c02c6f0 con 0x55a06e531450 2024-09-16T00:32:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.079+0000 7f896dffb640 1 -- 172.21.15.33:0/1364266869 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f897c050050 con 0x55a06e531450 2024-09-16T00:32:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.079+0000 7f898e79b740 10 client.27607 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.079+0000 7f898e79b740 1 -- 172.21.15.33:0/1364266869 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a06e543b40 con 0x55a06e531450 2024-09-16T00:32:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.080+0000 7f896dffb640 1 -- 172.21.15.33:0/1364266869 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f897c0378f0 con 0x55a06e531450 2024-09-16T00:32:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.080+0000 7f896dffb640 1 -- 172.21.15.33:0/1364266869 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f897c042700 con 0x55a06e531450 2024-09-16T00:32:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.080+0000 7f898e79b740 10 client.27607 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.080+0000 7f898e79b740 10 client.27607 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:32:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.080+0000 7f898e79b740 20 client.27607 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.080+0000 7f898e79b740 1 --2- 172.21.15.33:0/1364266869 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a06e5453b0 0x55a06e5477d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.080+0000 7f898e79b740 4 client.27607 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:32:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.080+0000 7f898e79b740 1 -- 172.21.15.33:0/1364266869 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a06e49d860 con 0x55a06e5453b0 2024-09-16T00:32:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.080+0000 7f898a15e640 1 --2- 172.21.15.33:0/1364266869 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a06e5453b0 0x55a06e5477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.082+0000 7f898a15e640 1 --2- 172.21.15.33:0/1364266869 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a06e5453b0 0x55a06e5477d0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8339a7586f886c8 server_cookie=1b710fbac9bde9c in_seq=0 out_seq=0 2024-09-16T00:32:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.082+0000 7f896dffb640 10 client.27607 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.084+0000 7f896dffb640 1 -- 172.21.15.33:0/1364266869 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a06e49d860 con 0x55a06e5453b0 2024-09-16T00:32:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.084+0000 7f896dffb640 10 client.27607 handle_command_reply: tid=0 2024-09-16T00:32:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.084+0000 7f898e79b740 1 client.27607 shutdown 2024-09-16T00:32:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.085+0000 7f898e79b740 1 -- 172.21.15.33:0/1364266869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e531450 msgr2=0x55a06e539a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.085+0000 7f898e79b740 1 --2- 172.21.15.33:0/1364266869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e531450 0x55a06e539a40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f897c02efd0 tx=0x7f897c02c550 comp rx=0 tx=0).stop 2024-09-16T00:32:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.085+0000 7f898e79b740 1 -- 172.21.15.33:0/1364266869 shutdown_connections 2024-09-16T00:32:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.085+0000 7f898e79b740 1 --2- 172.21.15.33:0/1364266869 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a06e5453b0 0x55a06e5477d0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.085+0000 7f898e79b740 1 --2- 172.21.15.33:0/1364266869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e531450 0x55a06e539a40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.085+0000 7f898e79b740 1 --2- 172.21.15.33:0/1364266869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e530b30 0x55a06e5358d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.085+0000 7f898e79b740 1 --2- 172.21.15.33:0/1364266869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e52ffc0 0x55a06e535390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.085+0000 7f896dffb640 0 client.27607 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.085+0000 7f898e79b740 1 -- 172.21.15.33:0/1364266869 >> 172.21.15.33:0/1364266869 conn(0x55a06e4f3310 msgr2=0x55a06e5323d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.085+0000 7f898e79b740 1 -- 172.21.15.33:0/1364266869 shutdown_connections 2024-09-16T00:32:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.085+0000 7f898e79b740 1 -- 172.21.15.33:0/1364266869 wait complete. 2024-09-16T00:32:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.085+0000 7f898e79b740 20 client.27607 trim_cache size 0 max 16384 2024-09-16T00:32:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.108+0000 7f898e79b740 1 Processor -- start 2024-09-16T00:32:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.108+0000 7f898e79b740 1 -- start start 2024-09-16T00:32:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.109+0000 7f898e79b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 0x55a06e539a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.109+0000 7f898e79b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e530b30 0x55a06e54d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.109+0000 7f898e79b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 0x55a06e54d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.109+0000 7f898e79b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f89841af860 con 0x55a06e52ffc0 2024-09-16T00:32:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.109+0000 7f898e79b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f89841ad0d0 con 0x55a06e531450 2024-09-16T00:32:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.109+0000 7f898e79b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89841aaf20 con 0x55a06e530b30 2024-09-16T00:32:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.110+0000 7f898a15e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 0x55a06e539a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.110+0000 7f898a95f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 0x55a06e54d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.110+0000 7f898a95f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 0x55a06e54d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:43880/0 (socket says 172.21.15.33:43880) 2024-09-16T00:32:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.110+0000 7f898a95f640 1 -- 172.21.15.33:0/1929666977 learned_addr learned my addr 172.21.15.33:0/1929666977 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.110+0000 7f898995d640 1 --2- 172.21.15.33:0/1929666977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e530b30 0x55a06e54d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.111+0000 7f898a95f640 1 -- 172.21.15.33:0/1929666977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e530b30 msgr2=0x55a06e54d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.111+0000 7f898a95f640 1 --2- 172.21.15.33:0/1929666977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e530b30 0x55a06e54d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.111+0000 7f898a95f640 1 -- 172.21.15.33:0/1929666977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 msgr2=0x55a06e539a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.111+0000 7f898a95f640 1 --2- 172.21.15.33:0/1929666977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 0x55a06e539a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.111+0000 7f898a95f640 1 -- 172.21.15.33:0/1929666977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a06e526c20 con 0x55a06e531450 2024-09-16T00:32:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.111+0000 7f898a95f640 1 --2- 172.21.15.33:0/1929666977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 0x55a06e54d620 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f897c02e900 tx=0x7f897c009390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd8d52a49c272252 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.112+0000 7f896ffff640 1 -- 172.21.15.33:0/1929666977 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c03c040 con 0x55a06e531450 2024-09-16T00:32:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.112+0000 7f896ffff640 1 -- 172.21.15.33:0/1929666977 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f897c047070 con 0x55a06e531450 2024-09-16T00:32:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.112+0000 7f896ffff640 1 -- 172.21.15.33:0/1929666977 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c009550 con 0x55a06e531450 2024-09-16T00:32:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.113+0000 7f898e79b740 1 -- 172.21.15.33:0/1929666977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 msgr2=0x55a06e54d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.113+0000 7f898e79b740 1 --2- 172.21.15.33:0/1929666977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 0x55a06e54d620 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f897c02e900 tx=0x7f897c009390 comp rx=0 tx=0).stop 2024-09-16T00:32:29.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.113+0000 7f898e79b740 1 -- 172.21.15.33:0/1929666977 shutdown_connections 2024-09-16T00:32:29.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.113+0000 7f898e79b740 1 --2- 172.21.15.33:0/1929666977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 0x55a06e54d620 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.113+0000 7f898e79b740 1 --2- 172.21.15.33:0/1929666977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e530b30 0x55a06e54d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.113+0000 7f898e79b740 1 --2- 172.21.15.33:0/1929666977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 0x55a06e539a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.113+0000 7f898e79b740 1 -- 172.21.15.33:0/1929666977 >> 172.21.15.33:0/1929666977 conn(0x55a06e4f3310 msgr2=0x55a06e525d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:29.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.114+0000 7f898e79b740 1 -- 172.21.15.33:0/1929666977 shutdown_connections 2024-09-16T00:32:29.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.114+0000 7f898e79b740 1 -- 172.21.15.33:0/1929666977 wait complete. 2024-09-16T00:32:29.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.114+0000 7f898e79b740 1 -- start start 2024-09-16T00:32:29.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.114+0000 7f898e79b740 1 Processor -- start 2024-09-16T00:32:29.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898e79b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 0x55a06e5414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898e79b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e530b30 0x55a06e5419f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898e79b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 0x55a06e52b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898e79b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f89841ab590 con 0x55a06e52ffc0 2024-09-16T00:32:29.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898e79b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f89841aca00 con 0x55a06e531450 2024-09-16T00:32:29.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898e79b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89841abc60 con 0x55a06e530b30 2024-09-16T00:32:29.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898e79b740 10 client.0 _ll_register_callbacks cb 0x7f897c03a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:29.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898995d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e530b30 0x55a06e5419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898a95f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 0x55a06e52b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898a15e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 0x55a06e5414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898a95f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 0x55a06e52b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:43890/0 (socket says 172.21.15.33:43890) 2024-09-16T00:32:29.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898a15e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 0x55a06e5414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39704/0 (socket says 172.21.15.33:39704) 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.115+0000 7f898a95f640 1 -- 172.21.15.33:0/3379156213 learned_addr learned my addr 172.21.15.33:0/3379156213 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.116+0000 7f898995d640 1 -- 172.21.15.33:0/3379156213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 msgr2=0x55a06e52b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.116+0000 7f898995d640 1 --2- 172.21.15.33:0/3379156213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 0x55a06e52b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.116+0000 7f898995d640 1 -- 172.21.15.33:0/3379156213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 msgr2=0x55a06e5414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.116+0000 7f898995d640 1 --2- 172.21.15.33:0/3379156213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 0x55a06e5414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.116+0000 7f898995d640 1 -- 172.21.15.33:0/3379156213 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a06e531280 con 0x55a06e530b30 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.116+0000 7f898a95f640 1 --2- 172.21.15.33:0/3379156213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 0x55a06e52b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.116+0000 7f898995d640 1 --2- 172.21.15.33:0/3379156213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e530b30 0x55a06e5419f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x55a06e534a40 tx=0x7f89740bca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21f499497b2591e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.116+0000 7f896dffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.116+0000 7f896dffb640 1 -- 172.21.15.33:0/3379156213 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897404a040 con 0x55a06e530b30 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.116+0000 7f896dffb640 1 -- 172.21.15.33:0/3379156213 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f89740b9210 con 0x55a06e530b30 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.116+0000 7f898e79b740 1 -- 172.21.15.33:0/3379156213 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f897c047070 con 0x55a06e530b30 2024-09-16T00:32:29.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.117+0000 7f896dffb640 1 -- 172.21.15.33:0/3379156213 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f89740bd2a0 con 0x55a06e530b30 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.117+0000 7f898e79b740 10 client.27613 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.117+0000 7f898e79b740 1 -- 172.21.15.33:0/3379156213 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a06e53f960 con 0x55a06e530b30 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.117+0000 7f896dffb640 1 -- 172.21.15.33:0/3379156213 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89740bcba0 con 0x55a06e530b30 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.117+0000 7f896dffb640 1 -- 172.21.15.33:0/3379156213 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f897403a040 con 0x55a06e530b30 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.117+0000 7f898e79b740 10 client.27613 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.117+0000 7f898e79b740 10 client.27613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.117+0000 7f898e79b740 20 client.27613 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.117+0000 7f898e79b740 1 --2- 172.21.15.33:0/3379156213 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a06e5b01c0 0x55a06e5b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.117+0000 7f898e79b740 4 client.27613 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.117+0000 7f898e79b740 1 -- 172.21.15.33:0/3379156213 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f897c047070 con 0x55a06e5b01c0 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.117+0000 7f898a15e640 1 --2- 172.21.15.33:0/3379156213 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a06e5b01c0 0x55a06e5b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f898a15e640 1 --2- 172.21.15.33:0/3379156213 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a06e5b01c0 0x55a06e5b25b0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b9d3c62b15ceb5 server_cookie=190678638b7a1295 in_seq=0 out_seq=0 2024-09-16T00:32:29.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f896dffb640 10 client.27613 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f896dffb640 1 -- 172.21.15.33:0/3379156213 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f897c047070 con 0x55a06e5b01c0 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f896dffb640 10 client.27613 handle_command_reply: tid=0 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f898e79b740 1 client.27613 shutdown 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f898e79b740 1 -- 172.21.15.33:0/3379156213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e530b30 msgr2=0x55a06e5419f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f898e79b740 1 --2- 172.21.15.33:0/3379156213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e530b30 0x55a06e5419f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x55a06e534a40 tx=0x7f89740bca30 comp rx=0 tx=0).stop 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f898e79b740 1 -- 172.21.15.33:0/3379156213 shutdown_connections 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f898e79b740 1 --2- 172.21.15.33:0/3379156213 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a06e5b01c0 0x55a06e5b25b0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f896dffb640 0 client.27613 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f898e79b740 1 --2- 172.21.15.33:0/3379156213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a06e531450 0x55a06e52b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f898e79b740 1 --2- 172.21.15.33:0/3379156213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a06e530b30 0x55a06e5419f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f898e79b740 1 --2- 172.21.15.33:0/3379156213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a06e52ffc0 0x55a06e5414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.118+0000 7f898e79b740 1 -- 172.21.15.33:0/3379156213 >> 172.21.15.33:0/3379156213 conn(0x55a06e4f3310 msgr2=0x55a06e531ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:29.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.119+0000 7f898e79b740 1 -- 172.21.15.33:0/3379156213 shutdown_connections 2024-09-16T00:32:29.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.119+0000 7f898e79b740 1 -- 172.21.15.33:0/3379156213 wait complete. 2024-09-16T00:32:29.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.119+0000 7f898e79b740 20 client.27613 trim_cache size 0 max 16384 2024-09-16T00:32:29.124 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:32:29.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.119+0000 7f898c3c2640 1 -- 172.21.15.33:0/2998600519 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89700a3330 msgr2=0x7f89700a57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:29.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.119+0000 7f898c3c2640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89700a3330 0x7f89700a57f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x55a06e527a40 tx=0x7f898000aaa0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.119+0000 7f898c3c2640 1 -- 172.21.15.33:0/2998600519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8984103650 msgr2=0x7f89841989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:29.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.119+0000 7f898c3c2640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8984103650 0x7f89841989a0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f8974006010 tx=0x7f89740045e0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.120+0000 7f898c3c2640 1 -- 172.21.15.33:0/2998600519 shutdown_connections 2024-09-16T00:32:29.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.120+0000 7f898c3c2640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89700a3330 0x7f89700a57f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.120+0000 7f898c3c2640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f898410c160 0x7f898419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.120+0000 7f898c3c2640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8984103650 0x7f89841989a0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.120+0000 7f898c3c2640 1 --2- 172.21.15.33:0/2998600519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8984102d30 0x7f8984198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:29.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.120+0000 7f898c3c2640 1 -- 172.21.15.33:0/2998600519 >> 172.21.15.33:0/2998600519 conn(0x7f89840fc7d0 msgr2=0x7f89840fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:29.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.120+0000 7f898c3c2640 1 -- 172.21.15.33:0/2998600519 shutdown_connections 2024-09-16T00:32:29.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:29.120+0000 7f898c3c2640 1 -- 172.21.15.33:0/2998600519 wait complete. 2024-09-16T00:32:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:29 smithi060 ceph-mon[35185]: pgmap v810: 129 pgs: 129 active+clean; 512 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 15 MiB/s wr, 668 op/s 2024-09-16T00:32:30.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 37935 23.15 MB/sec warmup 96 sec latency 171.437 ms 2024-09-16T00:32:30.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:29 smithi033 ceph-mon[31243]: pgmap v810: 129 pgs: 129 active+clean; 512 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 15 MiB/s wr, 668 op/s 2024-09-16T00:32:30.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:29 smithi083 ceph-mon[35242]: pgmap v810: 129 pgs: 129 active+clean; 512 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 15 MiB/s wr, 668 op/s 2024-09-16T00:32:31.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 38328 23.08 MB/sec warmup 97 sec latency 211.887 ms 2024-09-16T00:32:31.660 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:32:31.661 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:32:31.969 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:32:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:31 smithi060 ceph-mon[35185]: pgmap v811: 129 pgs: 129 active+clean; 495 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 15 MiB/s wr, 752 op/s 2024-09-16T00:32:32.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 38934 23.15 MB/sec warmup 98 sec latency 159.365 ms 2024-09-16T00:32:32.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:31 smithi033 ceph-mon[31243]: pgmap v811: 129 pgs: 129 active+clean; 495 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 15 MiB/s wr, 752 op/s 2024-09-16T00:32:32.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:31 smithi083 ceph-mon[35242]: pgmap v811: 129 pgs: 129 active+clean; 495 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 15 MiB/s wr, 752 op/s 2024-09-16T00:32:33.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 39565 23.31 MB/sec warmup 99 sec latency 150.571 ms 2024-09-16T00:32:33.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.345+0000 7fe890757640 1 -- 172.21.15.33:0/3303078956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe888106b80 msgr2=0x7fe888106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:33.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.345+0000 7fe890757640 1 --2- 172.21.15.33:0/3303078956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe888106b80 0x7fe888106f60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe884009900 tx=0x7fe88402a410 comp rx=0 tx=0).stop 2024-09-16T00:32:33.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.345+0000 7fe890757640 1 -- 172.21.15.33:0/3303078956 shutdown_connections 2024-09-16T00:32:33.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.345+0000 7fe890757640 1 --2- 172.21.15.33:0/3303078956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe888100aa0 0x7fe88810ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.346+0000 7fe890757640 1 --2- 172.21.15.33:0/3303078956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe888100100 0x7fe888100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.346+0000 7fe890757640 1 --2- 172.21.15.33:0/3303078956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe888106b80 0x7fe888106f60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.346+0000 7fe890757640 1 -- 172.21.15.33:0/3303078956 >> 172.21.15.33:0/3303078956 conn(0x7fe888077230 msgr2=0x7fe8880fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:33.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.346+0000 7fe890757640 1 -- 172.21.15.33:0/3303078956 shutdown_connections 2024-09-16T00:32:33.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.346+0000 7fe890757640 1 -- 172.21.15.33:0/3303078956 wait complete. 2024-09-16T00:32:33.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.346+0000 7fe890757640 1 Processor -- start 2024-09-16T00:32:33.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.346+0000 7fe890757640 1 -- start start 2024-09-16T00:32:33.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe890757640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe888100100 0x7fe8881984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe890757640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe888100aa0 0x7fe888198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe890757640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe888106b80 0x7fe88819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe890757640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe888112050 con 0x7fe888106b80 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe890757640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe888111ed0 con 0x7fe888100100 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe890757640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881121d0 con 0x7fe888100aa0 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe88e4f3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe888100100 0x7fe8881984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe88e4f3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe888100100 0x7fe8881984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:36558/0 (socket says 172.21.15.33:36558) 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe88e4f3640 1 -- 172.21.15.33:0/3161859238 learned_addr learned my addr 172.21.15.33:0/3161859238 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe88ecf4640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe888106b80 0x7fe88819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe88dcf2640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe888100aa0 0x7fe888198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe88e4f3640 1 -- 172.21.15.33:0/3161859238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe888100aa0 msgr2=0x7fe888198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe888100aa0 0x7fe888198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe88e4f3640 1 -- 172.21.15.33:0/3161859238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe888106b80 msgr2=0x7fe88819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:33.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe888106b80 0x7fe88819cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.347+0000 7fe88e4f3640 1 -- 172.21.15.33:0/3161859238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88819d490 con 0x7fe888100100 2024-09-16T00:32:33.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.348+0000 7fe88dcf2640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe888100aa0 0x7fe888198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:32:33.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.348+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe888100100 0x7fe8881984e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe884006010 tx=0x7fe8840042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:33.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.348+0000 7fe87f7fe640 1 -- 172.21.15.33:0/3161859238 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88403f030 con 0x7fe888100100 2024-09-16T00:32:33.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.348+0000 7fe890757640 1 -- 172.21.15.33:0/3161859238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88819d720 con 0x7fe888100100 2024-09-16T00:32:33.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.348+0000 7fe87f7fe640 1 -- 172.21.15.33:0/3161859238 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe884003ce0 con 0x7fe888100100 2024-09-16T00:32:33.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.348+0000 7fe87f7fe640 1 -- 172.21.15.33:0/3161859238 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884035070 con 0x7fe888100100 2024-09-16T00:32:33.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.348+0000 7fe890757640 1 -- 172.21.15.33:0/3161859238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8881a5060 con 0x7fe888100100 2024-09-16T00:32:33.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.351+0000 7fe87f7fe640 1 -- 172.21.15.33:0/3161859238 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe884042030 con 0x7fe888100100 2024-09-16T00:32:33.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.351+0000 7fe892b30740 1 Processor -- start 2024-09-16T00:32:33.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.352+0000 7fe892b30740 1 -- start start 2024-09-16T00:32:33.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.352+0000 7fe892b30740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 0x55a08fb23e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.352+0000 7fe892b30740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb243a0 0x55a08fb32c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.352+0000 7fe892b30740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb33150 0x55a08fb35540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.352+0000 7fe892b30740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a08fa89280 con 0x55a08fb33150 2024-09-16T00:32:33.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.352+0000 7fe892b30740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a08fa30b10 con 0x55a08fb23a80 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.352+0000 7fe892b30740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a08fa85240 con 0x55a08fb243a0 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.352+0000 7fe88ecf4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 0x55a08fb23e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.352+0000 7fe88dcf2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb33150 0x55a08fb35540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.352+0000 7fe88ecf4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 0x55a08fb23e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:36574/0 (socket says 172.21.15.33:36574) 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.352+0000 7fe88ecf4640 1 -- 172.21.15.33:0/3718532234 learned_addr learned my addr 172.21.15.33:0/3718532234 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.352+0000 7fe88dcf2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb33150 0x55a08fb35540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48674/0 (socket says 172.21.15.33:48674) 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.353+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/3718532234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb243a0 0x55a08fb32c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.353+0000 7fe87f7fe640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe8740a3240 0x7fe8740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.353+0000 7fe88dcf2640 1 -- 172.21.15.33:0/3718532234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb243a0 msgr2=0x55a08fb32c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.353+0000 7fe88dcf2640 1 --2- 172.21.15.33:0/3718532234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb243a0 0x55a08fb32c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.353+0000 7fe88dcf2640 1 -- 172.21.15.33:0/3718532234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 msgr2=0x55a08fb23e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.353+0000 7fe88dcf2640 1 --2- 172.21.15.33:0/3718532234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 0x55a08fb23e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.353+0000 7fe88dcf2640 1 -- 172.21.15.33:0/3718532234 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a08fa3d1f0 con 0x55a08fb33150 2024-09-16T00:32:33.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.353+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/3718532234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb243a0 0x55a08fb32c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:32:33.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.353+0000 7fe88ecf4640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe8740a3240 0x7fe8740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.353+0000 7fe87f7fe640 1 -- 172.21.15.33:0/3161859238 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe8840f02c0 con 0x7fe888100100 2024-09-16T00:32:33.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.353+0000 7fe88dcf2640 1 --2- 172.21.15.33:0/3718532234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb33150 0x55a08fb35540 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe878009b60 tx=0x7fe87800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7327627370c54e40 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:33.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.354+0000 7fe87cff9640 1 -- 172.21.15.33:0/3718532234 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878016070 con 0x55a08fb33150 2024-09-16T00:32:33.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.354+0000 7fe88ecf4640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe8740a3240 0x7fe8740a5700 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe880010650 tx=0x7fe880004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:33.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.354+0000 7fe87cff9640 1 -- 172.21.15.33:0/3718532234 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe878009e50 con 0x55a08fb33150 2024-09-16T00:32:33.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.354+0000 7fe87cff9640 1 -- 172.21.15.33:0/3718532234 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878010790 con 0x55a08fb33150 2024-09-16T00:32:33.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.354+0000 7fe892b30740 1 -- 172.21.15.33:0/3718532234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb33150 msgr2=0x55a08fb35540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:33.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.354+0000 7fe892b30740 1 --2- 172.21.15.33:0/3718532234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb33150 0x55a08fb35540 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe878009b60 tx=0x7fe87800e6c0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.354+0000 7fe892b30740 1 -- 172.21.15.33:0/3718532234 shutdown_connections 2024-09-16T00:32:33.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.354+0000 7fe892b30740 1 --2- 172.21.15.33:0/3718532234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb33150 0x55a08fb35540 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.354+0000 7fe892b30740 1 --2- 172.21.15.33:0/3718532234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb243a0 0x55a08fb32c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.354+0000 7fe892b30740 1 --2- 172.21.15.33:0/3718532234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 0x55a08fb23e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.354+0000 7fe892b30740 1 -- 172.21.15.33:0/3718532234 >> 172.21.15.33:0/3718532234 conn(0x55a08fb200f0 msgr2=0x55a08fb204d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.355+0000 7fe892b30740 1 -- 172.21.15.33:0/3718532234 shutdown_connections 2024-09-16T00:32:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.355+0000 7fe892b30740 1 -- 172.21.15.33:0/3718532234 wait complete. 2024-09-16T00:32:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.355+0000 7fe892b30740 1 -- start start 2024-09-16T00:32:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.355+0000 7fe892b30740 1 Processor -- start 2024-09-16T00:32:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.355+0000 7fe892b30740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 0x55a08fb32930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.355+0000 7fe892b30740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb243a0 0x55a08fb2bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.355+0000 7fe892b30740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb33150 0x55a08fb2c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.355+0000 7fe892b30740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a08fb35cc0 con 0x55a08fb243a0 2024-09-16T00:32:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe892b30740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a08fa43f10 con 0x55a08fb33150 2024-09-16T00:32:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe892b30740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a08fb35e40 con 0x55a08fb23a80 2024-09-16T00:32:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe88e4f3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 0x55a08fb32930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe892b30740 10 client.0 _ll_register_callbacks cb 0x55a08fa0ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe88e4f3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 0x55a08fb32930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34258/0 (socket says 172.21.15.33:34258) 2024-09-16T00:32:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe88dcf2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb243a0 0x55a08fb2bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe88e4f3640 1 -- 172.21.15.33:0/446963225 learned_addr learned my addr 172.21.15.33:0/446963225 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe88ecf4640 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb33150 0x55a08fb2c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe88dcf2640 1 -- 172.21.15.33:0/446963225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 msgr2=0x55a08fb32930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe88dcf2640 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 0x55a08fb32930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe88dcf2640 1 -- 172.21.15.33:0/446963225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb33150 msgr2=0x55a08fb2c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe88dcf2640 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb33150 0x55a08fb2c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe88dcf2640 1 -- 172.21.15.33:0/446963225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a08f7fa6a0 con 0x55a08fb243a0 2024-09-16T00:32:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe88ecf4640 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb33150 0x55a08fb2c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe88dcf2640 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb243a0 0x55a08fb2bc60 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe878011f30 tx=0x7fe87800ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509830632cc1fee2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe871ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.356+0000 7fe871ffb640 1 -- 172.21.15.33:0/446963225 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878028060 con 0x55a08fb243a0 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe871ffb640 1 -- 172.21.15.33:0/446963225 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe878011680 con 0x55a08fb243a0 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe892b30740 1 -- 172.21.15.33:0/446963225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe878009e50 con 0x55a08fb243a0 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe871ffb640 1 -- 172.21.15.33:0/446963225 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe87802f050 con 0x55a08fb243a0 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe892b30740 10 client.18294 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe892b30740 1 -- 172.21.15.33:0/446963225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a08fb311e0 con 0x55a08fb243a0 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe871ffb640 1 -- 172.21.15.33:0/446963225 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe878009e50 con 0x55a08fb243a0 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe871ffb640 1 -- 172.21.15.33:0/446963225 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe87800f040 con 0x55a08fb243a0 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe892b30740 10 client.18294 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe892b30740 10 client.18294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe892b30740 20 client.18294 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe892b30740 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08fb37720 0x55a08fb39b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe892b30740 4 client.18294 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:32:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe892b30740 1 -- 172.21.15.33:0/446963225 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a08fa967f0 con 0x55a08fb37720 2024-09-16T00:32:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.357+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08fb37720 0x55a08fb39b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.358+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08fb37720 0x55a08fb39b60 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=946d6059dfee4bcf server_cookie=f1c400132eb5889e in_seq=0 out_seq=0 2024-09-16T00:32:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.358+0000 7fe871ffb640 10 client.18294 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.359+0000 7fe871ffb640 1 -- 172.21.15.33:0/446963225 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a08fa967f0 con 0x55a08fb37720 2024-09-16T00:32:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.359+0000 7fe871ffb640 10 client.18294 handle_command_reply: tid=0 2024-09-16T00:32:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.359+0000 7fe892b30740 1 client.18294 shutdown 2024-09-16T00:32:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.359+0000 7fe892b30740 1 -- 172.21.15.33:0/446963225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb243a0 msgr2=0x55a08fb2bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.359+0000 7fe892b30740 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb243a0 0x55a08fb2bc60 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe878011f30 tx=0x7fe87800ef30 comp rx=0 tx=0).stop 2024-09-16T00:32:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.361+0000 7fe892b30740 1 -- 172.21.15.33:0/446963225 shutdown_connections 2024-09-16T00:32:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.361+0000 7fe892b30740 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08fb37720 0x55a08fb39b60 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.361+0000 7fe892b30740 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb33150 0x55a08fb2c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.361+0000 7fe892b30740 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb243a0 0x55a08fb2bc60 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.361+0000 7fe892b30740 1 --2- 172.21.15.33:0/446963225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 0x55a08fb32930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.361+0000 7fe892b30740 1 -- 172.21.15.33:0/446963225 >> 172.21.15.33:0/446963225 conn(0x55a08fb200f0 msgr2=0x55a08fb1f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:33.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.361+0000 7fe871ffb640 0 client.18294 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:33.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.361+0000 7fe892b30740 1 -- 172.21.15.33:0/446963225 shutdown_connections 2024-09-16T00:32:33.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.361+0000 7fe892b30740 1 -- 172.21.15.33:0/446963225 wait complete. 2024-09-16T00:32:33.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.361+0000 7fe892b30740 20 client.18294 trim_cache size 0 max 16384 2024-09-16T00:32:33.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.396+0000 7fe892b30740 1 Processor -- start 2024-09-16T00:32:33.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.397+0000 7fe892b30740 1 -- start start 2024-09-16T00:32:33.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.397+0000 7fe892b30740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 0x55a08fb5bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.397+0000 7fe892b30740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb243a0 0x55a08fb5c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.397+0000 7fe892b30740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb33150 0x55a08fb28ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.397+0000 7fe892b30740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881c8f60 con 0x55a08fb243a0 2024-09-16T00:32:33.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.397+0000 7fe892b30740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe8880739b0 con 0x55a08fb33150 2024-09-16T00:32:33.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.397+0000 7fe892b30740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe888071800 con 0x55a08fb23a80 2024-09-16T00:32:33.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.397+0000 7fe88e4f3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 0x55a08fb5bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.397+0000 7fe88e4f3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 0x55a08fb5bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34268/0 (socket says 172.21.15.33:34268) 2024-09-16T00:32:33.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.397+0000 7fe88e4f3640 1 -- 172.21.15.33:0/2366149447 learned_addr learned my addr 172.21.15.33:0/2366149447 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:33.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.398+0000 7fe88dcf2640 1 --2- 172.21.15.33:0/2366149447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb243a0 0x55a08fb5c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.398+0000 7fe88ecf4640 1 --2- 172.21.15.33:0/2366149447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb33150 0x55a08fb28ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.398+0000 7fe88e4f3640 1 -- 172.21.15.33:0/2366149447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb33150 msgr2=0x55a08fb28ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:33.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.399+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/2366149447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb33150 0x55a08fb28ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.399+0000 7fe88e4f3640 1 -- 172.21.15.33:0/2366149447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb243a0 msgr2=0x55a08fb5c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:33.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.399+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/2366149447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb243a0 0x55a08fb5c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.399+0000 7fe88e4f3640 1 -- 172.21.15.33:0/2366149447 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a08fb32ca0 con 0x55a08fb23a80 2024-09-16T00:32:33.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.399+0000 7fe88ecf4640 1 --2- 172.21.15.33:0/2366149447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb33150 0x55a08fb28ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:32:33.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.399+0000 7fe88dcf2640 1 --2- 172.21.15.33:0/2366149447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb243a0 0x55a08fb5c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:32:33.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.399+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/2366149447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 0x55a08fb5bfe0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe884004650 tx=0x7fe88402a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e4f7a486e8cd333 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:33.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.399+0000 7fe873fff640 1 -- 172.21.15.33:0/2366149447 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884049040 con 0x55a08fb23a80 2024-09-16T00:32:33.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.400+0000 7fe873fff640 1 -- 172.21.15.33:0/2366149447 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8840bcc20 con 0x55a08fb23a80 2024-09-16T00:32:33.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.401+0000 7fe892b30740 1 -- 172.21.15.33:0/2366149447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 msgr2=0x55a08fb5bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:33.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.401+0000 7fe892b30740 1 --2- 172.21.15.33:0/2366149447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 0x55a08fb5bfe0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe884004650 tx=0x7fe88402a410 comp rx=0 tx=0).stop 2024-09-16T00:32:33.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.401+0000 7fe892b30740 1 -- 172.21.15.33:0/2366149447 shutdown_connections 2024-09-16T00:32:33.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.401+0000 7fe892b30740 1 --2- 172.21.15.33:0/2366149447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb33150 0x55a08fb28ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.401+0000 7fe892b30740 1 --2- 172.21.15.33:0/2366149447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb243a0 0x55a08fb5c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.401+0000 7fe892b30740 1 --2- 172.21.15.33:0/2366149447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb23a80 0x55a08fb5bfe0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.401+0000 7fe892b30740 1 -- 172.21.15.33:0/2366149447 >> 172.21.15.33:0/2366149447 conn(0x55a08fb200f0 msgr2=0x55a08fb1f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:33.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.402+0000 7fe892b30740 1 -- 172.21.15.33:0/2366149447 shutdown_connections 2024-09-16T00:32:33.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.402+0000 7fe892b30740 1 -- 172.21.15.33:0/2366149447 wait complete. 2024-09-16T00:32:33.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.402+0000 7fe892b30740 1 -- start start 2024-09-16T00:32:33.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.402+0000 7fe892b30740 1 Processor -- start 2024-09-16T00:32:33.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.403+0000 7fe892b30740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 0x55a08fb54ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.403+0000 7fe892b30740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb243a0 0x55a08fb55420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.403+0000 7fe892b30740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb33150 0x55a08fb50460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.403+0000 7fe892b30740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe888071e70 con 0x55a08fb33150 2024-09-16T00:32:33.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.403+0000 7fe892b30740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe8880732e0 con 0x55a08fb23a80 2024-09-16T00:32:33.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.403+0000 7fe892b30740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe888072540 con 0x55a08fb243a0 2024-09-16T00:32:33.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.403+0000 7fe892b30740 10 client.0 _ll_register_callbacks cb 0x7fe880009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:33.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.403+0000 7fe88e4f3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 0x55a08fb54ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.403+0000 7fe88e4f3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 0x55a08fb54ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:36600/0 (socket says 172.21.15.33:36600) 2024-09-16T00:32:33.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.403+0000 7fe88dcf2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb243a0 0x55a08fb55420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.403+0000 7fe88e4f3640 1 -- 172.21.15.33:0/1612382380 learned_addr learned my addr 172.21.15.33:0/1612382380 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:33.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.404+0000 7fe88ecf4640 1 --2- 172.21.15.33:0/1612382380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb33150 0x55a08fb50460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.404+0000 7fe88e4f3640 1 -- 172.21.15.33:0/1612382380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb243a0 msgr2=0x55a08fb55420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:33.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.404+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/1612382380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb243a0 0x55a08fb55420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.404+0000 7fe88e4f3640 1 -- 172.21.15.33:0/1612382380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb33150 msgr2=0x55a08fb50460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:33.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.404+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/1612382380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb33150 0x55a08fb50460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.404+0000 7fe88e4f3640 1 -- 172.21.15.33:0/1612382380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a08fb50c90 con 0x55a08fb23a80 2024-09-16T00:32:33.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.404+0000 7fe88e4f3640 1 --2- 172.21.15.33:0/1612382380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 0x55a08fb54ee0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe8840bc4b0 tx=0x7fe8840baa30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c92b54c95309749 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:33.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.404+0000 7fe871ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:32:33.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.405+0000 7fe871ffb640 1 -- 172.21.15.33:0/1612382380 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884049040 con 0x55a08fb23a80 2024-09-16T00:32:33.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.405+0000 7fe892b30740 1 -- 172.21.15.33:0/1612382380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8840bcc20 con 0x55a08fb23a80 2024-09-16T00:32:33.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.405+0000 7fe871ffb640 1 -- 172.21.15.33:0/1612382380 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe8840bc780 con 0x55a08fb23a80 2024-09-16T00:32:33.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.406+0000 7fe871ffb640 1 -- 172.21.15.33:0/1612382380 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe8840c08d0 con 0x55a08fb23a80 2024-09-16T00:32:33.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.406+0000 7fe892b30740 10 client.27635 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:33.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.406+0000 7fe892b30740 1 -- 172.21.15.33:0/1612382380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a08fbbabf0 con 0x55a08fb23a80 2024-09-16T00:32:33.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.406+0000 7fe871ffb640 1 -- 172.21.15.33:0/1612382380 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8840bcc20 con 0x55a08fb23a80 2024-09-16T00:32:33.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.407+0000 7fe871ffb640 1 -- 172.21.15.33:0/1612382380 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe88403bb50 con 0x55a08fb23a80 2024-09-16T00:32:33.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.407+0000 7fe892b30740 10 client.27635 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:33.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.407+0000 7fe892b30740 10 client.27635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:32:33.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.407+0000 7fe892b30740 20 client.27635 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:33.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.407+0000 7fe892b30740 1 --2- 172.21.15.33:0/1612382380 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08fbbb2c0 0x55a08fbbd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:33.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.407+0000 7fe892b30740 4 client.27635 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:32:33.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.407+0000 7fe892b30740 1 -- 172.21.15.33:0/1612382380 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a08fa90d00 con 0x55a08fbbb2c0 2024-09-16T00:32:33.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.407+0000 7fe88dcf2640 1 --2- 172.21.15.33:0/1612382380 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08fbbb2c0 0x55a08fbbd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:33.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.408+0000 7fe88dcf2640 1 --2- 172.21.15.33:0/1612382380 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08fbbb2c0 0x55a08fbbd6b0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d90e50db11acd37 server_cookie=e918179bb50f26ae in_seq=0 out_seq=0 2024-09-16T00:32:33.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.408+0000 7fe871ffb640 10 client.27635 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:33.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.409+0000 7fe871ffb640 1 -- 172.21.15.33:0/1612382380 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a08fa90d00 con 0x55a08fbbb2c0 2024-09-16T00:32:33.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.409+0000 7fe871ffb640 10 client.27635 handle_command_reply: tid=0 2024-09-16T00:32:33.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.409+0000 7fe892b30740 1 client.27635 shutdown 2024-09-16T00:32:33.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.410+0000 7fe892b30740 1 -- 172.21.15.33:0/1612382380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 msgr2=0x55a08fb54ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:33.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.410+0000 7fe892b30740 1 --2- 172.21.15.33:0/1612382380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 0x55a08fb54ee0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe8840bc4b0 tx=0x7fe8840baa30 comp rx=0 tx=0).stop 2024-09-16T00:32:33.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.413+0000 7fe892b30740 1 -- 172.21.15.33:0/1612382380 shutdown_connections 2024-09-16T00:32:33.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.413+0000 7fe892b30740 1 --2- 172.21.15.33:0/1612382380 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55a08fbbb2c0 0x55a08fbbd6b0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.413+0000 7fe892b30740 1 --2- 172.21.15.33:0/1612382380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a08fb33150 0x55a08fb50460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.413+0000 7fe892b30740 1 --2- 172.21.15.33:0/1612382380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a08fb243a0 0x55a08fb55420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.413+0000 7fe892b30740 1 --2- 172.21.15.33:0/1612382380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08fb23a80 0x55a08fb54ee0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.413+0000 7fe892b30740 1 -- 172.21.15.33:0/1612382380 >> 172.21.15.33:0/1612382380 conn(0x55a08fb200f0 msgr2=0x55a08fb254d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:33.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.413+0000 7fe871ffb640 0 client.27635 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:33.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.414+0000 7fe892b30740 1 -- 172.21.15.33:0/1612382380 shutdown_connections 2024-09-16T00:32:33.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.414+0000 7fe892b30740 1 -- 172.21.15.33:0/1612382380 wait complete. 2024-09-16T00:32:33.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.416+0000 7fe892b30740 20 client.27635 trim_cache size 0 max 16384 2024-09-16T00:32:33.416 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:32:33.416 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:32:33.416 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "4d3d12a5-80b7-49b0-be33-ed29059dab8a", 2024-09-16T00:32:33.416 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:32:33.417 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:32:33.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.417+0000 7fe890757640 1 -- 172.21.15.33:0/3161859238 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe8740a3240 msgr2=0x7fe8740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:33.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.417+0000 7fe890757640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe8740a3240 0x7fe8740a5700 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe880010650 tx=0x7fe880004a40 comp rx=0 tx=0).stop 2024-09-16T00:32:33.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.417+0000 7fe890757640 1 -- 172.21.15.33:0/3161859238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe888100100 msgr2=0x7fe8881984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:33.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.417+0000 7fe890757640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe888100100 0x7fe8881984e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe884006010 tx=0x7fe8840042f0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.417+0000 7fe890757640 1 -- 172.21.15.33:0/3161859238 shutdown_connections 2024-09-16T00:32:33.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.417+0000 7fe890757640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe8740a3240 0x7fe8740a5700 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.418+0000 7fe890757640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe888106b80 0x7fe88819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.418+0000 7fe890757640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe888100aa0 0x7fe888198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.418+0000 7fe890757640 1 --2- 172.21.15.33:0/3161859238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe888100100 0x7fe8881984e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:33.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.418+0000 7fe890757640 1 -- 172.21.15.33:0/3161859238 >> 172.21.15.33:0/3161859238 conn(0x7fe888077230 msgr2=0x7fe888104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:33.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.418+0000 7fe890757640 1 -- 172.21.15.33:0/3161859238 shutdown_connections 2024-09-16T00:32:33.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:33.418+0000 7fe890757640 1 -- 172.21.15.33:0/3161859238 wait complete. 2024-09-16T00:32:33.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: pgmap v812: 129 pgs: 129 active+clean; 500 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 12 MiB/s wr, 700 op/s 2024-09-16T00:32:33.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:32:33.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:33.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:32:33.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:33.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:32:33.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:33.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:32:33.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:33.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:32:33.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:33.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:32:33.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:33 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: pgmap v812: 129 pgs: 129 active+clean; 500 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 12 MiB/s wr, 700 op/s 2024-09-16T00:32:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:32:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:32:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:32:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:32:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:32:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:32:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:33 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 40125 23.32 MB/sec warmup 100 sec latency 76.438 ms 2024-09-16T00:32:34.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: pgmap v812: 129 pgs: 129 active+clean; 500 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 12 MiB/s wr, 700 op/s 2024-09-16T00:32:34.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:32:34.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:32:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:32:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:32:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:32:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:32:34.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:33 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.181 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:32:34.489 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:32:34.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:34 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:32:34.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:34 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:32:34.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:34 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:34 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:32:34.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:34 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:34 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:32:34.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:34 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:34 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:32:34.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:34 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:34 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:34 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:32:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:34 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:32:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:34 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:34 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:32:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:34 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:34 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:32:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:34 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:34 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:32:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:34 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:34.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:34 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:35.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 40726 23.42 MB/sec warmup 101 sec latency 51.593 ms 2024-09-16T00:32:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.063+0000 7f69666a4640 1 -- 172.21.15.33:0/892082239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6960100e20 msgr2=0x7f6960101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.063+0000 7f69666a4640 1 --2- 172.21.15.33:0/892082239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6960100e20 0x7f6960101200 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6950009890 tx=0x7f695002a410 comp rx=0 tx=0).stop 2024-09-16T00:32:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.063+0000 7f69666a4640 1 -- 172.21.15.33:0/892082239 shutdown_connections 2024-09-16T00:32:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.063+0000 7f69666a4640 1 --2- 172.21.15.33:0/892082239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6960102ef0 0x7f696010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.063+0000 7f69666a4640 1 --2- 172.21.15.33:0/892082239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6960102570 0x7f69601029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.063+0000 7f69666a4640 1 --2- 172.21.15.33:0/892082239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6960100e20 0x7f6960101200 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.063+0000 7f69666a4640 1 -- 172.21.15.33:0/892082239 >> 172.21.15.33:0/892082239 conn(0x7f69600fc7d0 msgr2=0x7f69600febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.063+0000 7f69666a4640 1 -- 172.21.15.33:0/892082239 shutdown_connections 2024-09-16T00:32:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.063+0000 7f69666a4640 1 -- 172.21.15.33:0/892082239 wait complete. 2024-09-16T00:32:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.063+0000 7f69666a4640 1 Processor -- start 2024-09-16T00:32:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.063+0000 7f69666a4640 1 -- start start 2024-09-16T00:32:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f69666a4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6960100e20 0x7f69600fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f69666a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6960102570 0x7f69600fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f69666a4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6960102ef0 0x7f69600fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f69666a4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6960110f50 con 0x7f6960100e20 2024-09-16T00:32:35.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f69666a4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6960110dd0 con 0x7f6960102ef0 2024-09-16T00:32:35.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f69666a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f69601110d0 con 0x7f6960102570 2024-09-16T00:32:35.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f695ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6960100e20 0x7f69600fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f6957fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6960102570 0x7f69600fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f695ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6960100e20 0x7f69600fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48716/0 (socket says 172.21.15.33:48716) 2024-09-16T00:32:35.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f695ffff640 1 -- 172.21.15.33:0/2341679962 learned_addr learned my addr 172.21.15.33:0/2341679962 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:35.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f6964c41640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6960102ef0 0x7f69600fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f6964c41640 1 -- 172.21.15.33:0/2341679962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6960102570 msgr2=0x7f69600fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:35.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f6964c41640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6960102570 0x7f69600fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f6964c41640 1 -- 172.21.15.33:0/2341679962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6960100e20 msgr2=0x7f69600fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:35.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f6964c41640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6960100e20 0x7f69600fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f6964c41640 1 -- 172.21.15.33:0/2341679962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69601a6a10 con 0x7f6960102ef0 2024-09-16T00:32:35.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.064+0000 7f6957fff640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6960102570 0x7f69600fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:35.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.065+0000 7f6964c41640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6960102ef0 0x7f69600fd700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6958009fd0 tx=0x7f695800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:35.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.065+0000 7f695dffb640 1 -- 172.21.15.33:0/2341679962 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695800cd60 con 0x7f6960102ef0 2024-09-16T00:32:35.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.065+0000 7f69666a4640 1 -- 172.21.15.33:0/2341679962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69601a6d00 con 0x7f6960102ef0 2024-09-16T00:32:35.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.065+0000 7f695dffb640 1 -- 172.21.15.33:0/2341679962 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6958005020 con 0x7f6960102ef0 2024-09-16T00:32:35.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.065+0000 7f695dffb640 1 -- 172.21.15.33:0/2341679962 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6958010690 con 0x7f6960102ef0 2024-09-16T00:32:35.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.065+0000 7f69666a4640 1 -- 172.21.15.33:0/2341679962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69601a71f0 con 0x7f6960102ef0 2024-09-16T00:32:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.068+0000 7f695dffb640 1 -- 172.21.15.33:0/2341679962 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6958005540 con 0x7f6960102ef0 2024-09-16T00:32:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.068+0000 7f6968a7d740 1 Processor -- start 2024-09-16T00:32:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.068+0000 7f6968a7d740 1 -- start start 2024-09-16T00:32:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.068+0000 7f6968a7d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d826496090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.068+0000 7f6968a7d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264a0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.068+0000 7f6968a7d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264a3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.068+0000 7f6968a7d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d8263f32e0 con 0x55d8264965d0 2024-09-16T00:32:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.068+0000 7f6968a7d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d82639ace0 con 0x55d8264a0e90 2024-09-16T00:32:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.068+0000 7f6968a7d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d8263ef2a0 con 0x55d826495cb0 2024-09-16T00:32:35.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.069+0000 7f695ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.069+0000 7f6957fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d826496090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.069+0000 7f695ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264a0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48718/0 (socket says 172.21.15.33:48718) 2024-09-16T00:32:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.069+0000 7f695ffff640 1 -- 172.21.15.33:0/2259555420 learned_addr learned my addr 172.21.15.33:0/2259555420 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.069+0000 7f6957fff640 1 --2- 172.21.15.33:0/2259555420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264a3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.069+0000 7f695ffff640 1 -- 172.21.15.33:0/2259555420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 msgr2=0x55d826496090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.069+0000 7f695ffff640 1 --2- 172.21.15.33:0/2259555420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d826496090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.069+0000 7f695ffff640 1 -- 172.21.15.33:0/2259555420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 msgr2=0x55d8264a3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.069+0000 7f695ffff640 1 --2- 172.21.15.33:0/2259555420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264a3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.069+0000 7f695ffff640 1 -- 172.21.15.33:0/2259555420 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d82637f440 con 0x55d8264965d0 2024-09-16T00:32:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.069+0000 7f695dffb640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f694c0a3240 0x7f694c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.070+0000 7f695dffb640 1 -- 172.21.15.33:0/2341679962 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f69580d03b0 con 0x7f6960102ef0 2024-09-16T00:32:35.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.070+0000 7f6957fff640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f694c0a3240 0x7f694c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.070+0000 7f695ffff640 1 --2- 172.21.15.33:0/2259555420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264a0950 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f6950032f50 tx=0x7f6950032f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ef204972a64328 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:35.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.070+0000 7f6957fff640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f694c0a3240 0x7f694c0a5700 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x55d8264935c0 tx=0x7f6948009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:35.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.070+0000 7f69567fc640 1 -- 172.21.15.33:0/2259555420 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6950030cb0 con 0x55d8264965d0 2024-09-16T00:32:35.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.070+0000 7f69567fc640 1 -- 172.21.15.33:0/2259555420 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6950030e50 con 0x55d8264965d0 2024-09-16T00:32:35.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.071+0000 7f69567fc640 1 -- 172.21.15.33:0/2259555420 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6950030cb0 con 0x55d8264965d0 2024-09-16T00:32:35.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.071+0000 7f6968a7d740 1 -- 172.21.15.33:0/2259555420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 msgr2=0x55d8264a0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:35.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.071+0000 7f6968a7d740 1 --2- 172.21.15.33:0/2259555420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264a0950 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f6950032f50 tx=0x7f6950032f80 comp rx=0 tx=0).stop 2024-09-16T00:32:35.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.072+0000 7f6968a7d740 1 -- 172.21.15.33:0/2259555420 shutdown_connections 2024-09-16T00:32:35.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.072+0000 7f6968a7d740 1 --2- 172.21.15.33:0/2259555420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264a3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.072+0000 7f6968a7d740 1 --2- 172.21.15.33:0/2259555420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264a0950 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.072+0000 7f6968a7d740 1 --2- 172.21.15.33:0/2259555420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d826496090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.072+0000 7f6968a7d740 1 -- 172.21.15.33:0/2259555420 >> 172.21.15.33:0/2259555420 conn(0x55d826492240 msgr2=0x55d826492620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.072+0000 7f6968a7d740 1 -- 172.21.15.33:0/2259555420 shutdown_connections 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.072+0000 7f6968a7d740 1 -- 172.21.15.33:0/2259555420 wait complete. 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.072+0000 7f6968a7d740 1 -- start start 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.072+0000 7f6968a7d740 1 Processor -- start 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6968a7d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d8264a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6968a7d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d82649d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6968a7d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d82649ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6968a7d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d8264a3a60 con 0x55d8264965d0 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6968a7d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d8263ae050 con 0x55d8264a0e90 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6968a7d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d8264a3be0 con 0x55d826495cb0 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6968a7d740 10 client.0 _ll_register_callbacks cb 0x55d8263ee160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6957fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d82649d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6964c41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d82649ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6957fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d82649d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48732/0 (socket says 172.21.15.33:48732) 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6964c41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d82649ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:36628/0 (socket says 172.21.15.33:36628) 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6964c41640 1 -- 172.21.15.33:0/1268261770 learned_addr learned my addr 172.21.15.33:0/1268261770 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f695ffff640 1 --2- 172.21.15.33:0/1268261770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d8264a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6964c41640 1 -- 172.21.15.33:0/1268261770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 msgr2=0x55d8264a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6964c41640 1 --2- 172.21.15.33:0/1268261770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d8264a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6964c41640 1 -- 172.21.15.33:0/1268261770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 msgr2=0x55d82649d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6964c41640 1 --2- 172.21.15.33:0/1268261770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d82649d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.073+0000 7f6964c41640 1 -- 172.21.15.33:0/1268261770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8261643c0 con 0x55d8264a0e90 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.074+0000 7f6964c41640 1 --2- 172.21.15.33:0/1268261770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d82649ded0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f69580a4c90 tx=0x7f695809cc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a9559e95c2f3963 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.074+0000 7f693f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:32:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.074+0000 7f693f7fe640 1 -- 172.21.15.33:0/1268261770 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69580a0900 con 0x55d8264a0e90 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.074+0000 7f693f7fe640 1 -- 172.21.15.33:0/1268261770 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f69580051d0 con 0x55d8264a0e90 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.074+0000 7f6968a7d740 1 -- 172.21.15.33:0/1268261770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6950030e50 con 0x55d8264a0e90 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.074+0000 7f693f7fe640 1 -- 172.21.15.33:0/1268261770 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f69580a51c0 con 0x55d8264a0e90 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.074+0000 7f6968a7d740 10 client.27647 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.074+0000 7f6968a7d740 1 -- 172.21.15.33:0/1268261770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d8264af3c0 con 0x55d8264a0e90 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.074+0000 7f693f7fe640 1 -- 172.21.15.33:0/1268261770 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6950030e50 con 0x55d8264a0e90 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.075+0000 7f693f7fe640 1 -- 172.21.15.33:0/1268261770 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f69580982a0 con 0x55d8264a0e90 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.075+0000 7f6968a7d740 10 client.27647 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.075+0000 7f6968a7d740 10 client.27647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.075+0000 7f6968a7d740 20 client.27647 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.075+0000 7f6968a7d740 1 --2- 172.21.15.33:0/1268261770 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d8264b0820 0x55d8264b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.075+0000 7f6968a7d740 4 client.27647 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:32:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.075+0000 7f6968a7d740 1 -- 172.21.15.33:0/1268261770 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d826400850 con 0x55d8264b0820 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.075+0000 7f695ffff640 1 --2- 172.21.15.33:0/1268261770 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d8264b0820 0x55d8264b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.075+0000 7f695ffff640 1 --2- 172.21.15.33:0/1268261770 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d8264b0820 0x55d8264b2cc0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b00ebce5eeef0e44 server_cookie=7983c0650b8b1115 in_seq=0 out_seq=0 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.075+0000 7f693f7fe640 10 client.27647 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f693f7fe640 1 -- 172.21.15.33:0/1268261770 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d826400850 con 0x55d8264b0820 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f693f7fe640 10 client.27647 handle_command_reply: tid=0 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f6968a7d740 1 client.27647 shutdown 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f6968a7d740 1 -- 172.21.15.33:0/1268261770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 msgr2=0x55d82649ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f6968a7d740 1 --2- 172.21.15.33:0/1268261770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d82649ded0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f69580a4c90 tx=0x7f695809cc50 comp rx=0 tx=0).stop 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f6968a7d740 1 -- 172.21.15.33:0/1268261770 shutdown_connections 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f6968a7d740 1 --2- 172.21.15.33:0/1268261770 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d8264b0820 0x55d8264b2cc0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f6968a7d740 1 --2- 172.21.15.33:0/1268261770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d82649ded0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f6968a7d740 1 --2- 172.21.15.33:0/1268261770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d82649d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f6968a7d740 1 --2- 172.21.15.33:0/1268261770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d8264a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f693f7fe640 0 client.27647 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:35.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f6968a7d740 1 -- 172.21.15.33:0/1268261770 >> 172.21.15.33:0/1268261770 conn(0x55d826492240 msgr2=0x55d826488b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:35.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f6968a7d740 1 -- 172.21.15.33:0/1268261770 shutdown_connections 2024-09-16T00:32:35.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.077+0000 7f6968a7d740 1 -- 172.21.15.33:0/1268261770 wait complete. 2024-09-16T00:32:35.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.078+0000 7f6968a7d740 20 client.27647 trim_cache size 0 max 16384 2024-09-16T00:32:35.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f6968a7d740 1 Processor -- start 2024-09-16T00:32:35.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f6968a7d740 1 -- start start 2024-09-16T00:32:35.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f6968a7d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d82649c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f6968a7d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264b8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f6968a7d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f6968a7d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f69601af760 con 0x55d8264965d0 2024-09-16T00:32:35.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f6968a7d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f69601acfd0 con 0x55d8264a0e90 2024-09-16T00:32:35.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f6968a7d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f69601aae20 con 0x55d826495cb0 2024-09-16T00:32:35.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f6957fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264b8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f6957fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264b8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48736/0 (socket says 172.21.15.33:48736) 2024-09-16T00:32:35.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f6957fff640 1 -- 172.21.15.33:0/79101672 learned_addr learned my addr 172.21.15.33:0/79101672 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:35.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f695ffff640 1 --2- 172.21.15.33:0/79101672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d82649c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.105+0000 7f6964c41640 1 --2- 172.21.15.33:0/79101672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.106+0000 7f6957fff640 1 -- 172.21.15.33:0/79101672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 msgr2=0x55d82649c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.106+0000 7f6957fff640 1 --2- 172.21.15.33:0/79101672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d82649c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.106+0000 7f6957fff640 1 -- 172.21.15.33:0/79101672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 msgr2=0x55d8264b8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.106+0000 7f6957fff640 1 --2- 172.21.15.33:0/79101672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264b8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.106+0000 7f6957fff640 1 -- 172.21.15.33:0/79101672 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d82649c940 con 0x55d8264965d0 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.106+0000 7f6957fff640 1 --2- 172.21.15.33:0/79101672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264b8590 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f69600fe710 tx=0x7f6948016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5139fb1429459c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.107+0000 7f6955ffb640 1 -- 172.21.15.33:0/79101672 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6948020410 con 0x55d8264965d0 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.107+0000 7f6955ffb640 1 -- 172.21.15.33:0/79101672 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6948031070 con 0x55d8264965d0 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.107+0000 7f6955ffb640 1 -- 172.21.15.33:0/79101672 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694802c860 con 0x55d8264965d0 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.108+0000 7f6968a7d740 1 -- 172.21.15.33:0/79101672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 msgr2=0x55d8264b8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.108+0000 7f6968a7d740 1 --2- 172.21.15.33:0/79101672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264b8590 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f69600fe710 tx=0x7f6948016000 comp rx=0 tx=0).stop 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.108+0000 7f6968a7d740 1 -- 172.21.15.33:0/79101672 shutdown_connections 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.108+0000 7f6968a7d740 1 --2- 172.21.15.33:0/79101672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264b8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.108+0000 7f6968a7d740 1 --2- 172.21.15.33:0/79101672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264b8590 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.108+0000 7f6968a7d740 1 --2- 172.21.15.33:0/79101672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d82649c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.108+0000 7f6968a7d740 1 -- 172.21.15.33:0/79101672 >> 172.21.15.33:0/79101672 conn(0x55d826492240 msgr2=0x55d82649fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:35.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.108+0000 7f6968a7d740 1 -- 172.21.15.33:0/79101672 shutdown_connections 2024-09-16T00:32:35.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.108+0000 7f6968a7d740 1 -- 172.21.15.33:0/79101672 wait complete. 2024-09-16T00:32:35.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.109+0000 7f6968a7d740 1 -- start start 2024-09-16T00:32:35.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.109+0000 7f6968a7d740 1 Processor -- start 2024-09-16T00:32:35.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.110+0000 7f6968a7d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d8264a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.110+0000 7f6968a7d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264a5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.110+0000 7f6968a7d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.110+0000 7f6968a7d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f69601ab490 con 0x55d8264965d0 2024-09-16T00:32:35.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.110+0000 7f6968a7d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f69601ac900 con 0x55d8264a0e90 2024-09-16T00:32:35.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.110+0000 7f6968a7d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f69601abb60 con 0x55d826495cb0 2024-09-16T00:32:35.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.111+0000 7f6964c41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.111+0000 7f6968a7d740 10 client.0 _ll_register_callbacks cb 0x7f695809c090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:35.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.111+0000 7f695ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d8264a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.111+0000 7f695ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d8264a4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34334/0 (socket says 172.21.15.33:34334) 2024-09-16T00:32:35.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.111+0000 7f695ffff640 1 -- 172.21.15.33:0/3869062497 learned_addr learned my addr 172.21.15.33:0/3869062497 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:35.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.112+0000 7f6957fff640 1 --2- 172.21.15.33:0/3869062497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.112+0000 7f6964c41640 1 -- 172.21.15.33:0/3869062497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 msgr2=0x55d8264a4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:35.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.112+0000 7f6964c41640 1 --2- 172.21.15.33:0/3869062497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d8264a4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.112+0000 7f6964c41640 1 -- 172.21.15.33:0/3869062497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 msgr2=0x55d8264a5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:35.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.112+0000 7f6964c41640 1 --2- 172.21.15.33:0/3869062497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264a5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.112+0000 7f6964c41640 1 -- 172.21.15.33:0/3869062497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d82651a560 con 0x55d8264a0e90 2024-09-16T00:32:35.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.112+0000 7f6964c41640 1 --2- 172.21.15.33:0/3869062497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264a56c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f695809a800 tx=0x7f695809c550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=292f2329069ed265 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:35.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.112+0000 7f693f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:32:35.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.113+0000 7f693f7fe640 1 -- 172.21.15.33:0/3869062497 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69580a0900 con 0x55d8264a0e90 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.113+0000 7f693f7fe640 1 -- 172.21.15.33:0/3869062497 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f695809b4e0 con 0x55d8264a0e90 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.113+0000 7f6968a7d740 1 -- 172.21.15.33:0/3869062497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6948031070 con 0x55d8264a0e90 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.113+0000 7f693f7fe640 1 -- 172.21.15.33:0/3869062497 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6958082050 con 0x55d8264a0e90 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.113+0000 7f6968a7d740 10 client.27653 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.113+0000 7f6968a7d740 1 -- 172.21.15.33:0/3869062497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d82651b890 con 0x55d8264a0e90 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.113+0000 7f693f7fe640 1 -- 172.21.15.33:0/3869062497 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f695807c020 con 0x55d8264a0e90 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.114+0000 7f693f7fe640 1 -- 172.21.15.33:0/3869062497 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6958085390 con 0x55d8264a0e90 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.114+0000 7f6968a7d740 10 client.27653 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.114+0000 7f6968a7d740 10 client.27653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.114+0000 7f6968a7d740 20 client.27653 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.114+0000 7f6968a7d740 1 --2- 172.21.15.33:0/3869062497 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d82651bfb0 0x55d82651e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.114+0000 7f6968a7d740 4 client.27653 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:32:35.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.114+0000 7f6968a7d740 1 -- 172.21.15.33:0/3869062497 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f6948031070 con 0x55d82651bfb0 2024-09-16T00:32:35.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.115+0000 7f695ffff640 1 --2- 172.21.15.33:0/3869062497 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d82651bfb0 0x55d82651e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:35.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.118+0000 7f695ffff640 1 --2- 172.21.15.33:0/3869062497 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d82651bfb0 0x55d82651e3d0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8a5fc9db0425835 server_cookie=bbc150783211ba1d in_seq=0 out_seq=0 2024-09-16T00:32:35.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.118+0000 7f693f7fe640 10 client.27653 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:35.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.119+0000 7f693f7fe640 1 -- 172.21.15.33:0/3869062497 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f6948031070 con 0x55d82651bfb0 2024-09-16T00:32:35.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.119+0000 7f693f7fe640 10 client.27653 handle_command_reply: tid=0 2024-09-16T00:32:35.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.119+0000 7f6968a7d740 1 client.27653 shutdown 2024-09-16T00:32:35.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.119+0000 7f6968a7d740 1 -- 172.21.15.33:0/3869062497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 msgr2=0x55d8264a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:35.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.119+0000 7f6968a7d740 1 --2- 172.21.15.33:0/3869062497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264a56c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f695809a800 tx=0x7f695809c550 comp rx=0 tx=0).stop 2024-09-16T00:32:35.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.119+0000 7f6968a7d740 1 -- 172.21.15.33:0/3869062497 shutdown_connections 2024-09-16T00:32:35.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.120+0000 7f6968a7d740 1 --2- 172.21.15.33:0/3869062497 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d82651bfb0 0x55d82651e3d0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.120+0000 7f6968a7d740 1 --2- 172.21.15.33:0/3869062497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d8264a0e90 0x55d8264a56c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.120+0000 7f693f7fe640 0 client.27653 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:35.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.120+0000 7f6968a7d740 1 --2- 172.21.15.33:0/3869062497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8264965d0 0x55d8264a5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.120+0000 7f6968a7d740 1 --2- 172.21.15.33:0/3869062497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d826495cb0 0x55d8264a4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.120+0000 7f6968a7d740 1 -- 172.21.15.33:0/3869062497 >> 172.21.15.33:0/3869062497 conn(0x55d826492240 msgr2=0x55d826488350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:35.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.121+0000 7f6968a7d740 1 -- 172.21.15.33:0/3869062497 shutdown_connections 2024-09-16T00:32:35.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.122+0000 7f6968a7d740 1 -- 172.21.15.33:0/3869062497 wait complete. 2024-09-16T00:32:35.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.122+0000 7f6968a7d740 20 client.27653 trim_cache size 0 max 16384 2024-09-16T00:32:35.122 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:32:35.123 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:32:35.123 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:32:35.123 INFO:teuthology.orchestra.run.smithi033.stdout: "4d3d12a5-80b7-49b0-be33-ed29059dab8a": { 2024-09-16T00:32:35.123 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:32:35.123 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "4d3d12a5-80b7-49b0-be33-ed29059dab8a", 2024-09-16T00:32:35.123 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:32:35.123 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:32:35.123 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:32:35.123 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:32:35.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.123+0000 7f69666a4640 1 -- 172.21.15.33:0/2341679962 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f694c0a3240 msgr2=0x7f694c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:35.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.123+0000 7f69666a4640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f694c0a3240 0x7f694c0a5700 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x55d8264935c0 tx=0x7f6948009260 comp rx=0 tx=0).stop 2024-09-16T00:32:35.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.123+0000 7f69666a4640 1 -- 172.21.15.33:0/2341679962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6960102ef0 msgr2=0x7f69600fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:35.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.123+0000 7f69666a4640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6960102ef0 0x7f69600fd700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6958009fd0 tx=0x7f695800ee50 comp rx=0 tx=0).stop 2024-09-16T00:32:35.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.124+0000 7f69666a4640 1 -- 172.21.15.33:0/2341679962 shutdown_connections 2024-09-16T00:32:35.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.124+0000 7f69666a4640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f694c0a3240 0x7f694c0a5700 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.124+0000 7f69666a4640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6960102ef0 0x7f69600fd700 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.124+0000 7f69666a4640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6960102570 0x7f69600fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.124+0000 7f69666a4640 1 --2- 172.21.15.33:0/2341679962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6960100e20 0x7f69600fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:35.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.124+0000 7f69666a4640 1 -- 172.21.15.33:0/2341679962 >> 172.21.15.33:0/2341679962 conn(0x7f69600fc7d0 msgr2=0x7f696006a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:35.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.124+0000 7f69666a4640 1 -- 172.21.15.33:0/2341679962 shutdown_connections 2024-09-16T00:32:35.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:35.124+0000 7f69666a4640 1 -- 172.21.15.33:0/2341679962 wait complete. 2024-09-16T00:32:35.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:34 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:32:35.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:34 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:32:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:34 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:34 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:32:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:34 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:34 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:32:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:34 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:34 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:32:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:34 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:34 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:32:35.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:35 smithi033 ceph-mon[31243]: pgmap v813: 129 pgs: 129 active+clean; 508 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.0 MiB/s wr, 600 op/s 2024-09-16T00:32:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:35 smithi060 ceph-mon[35185]: pgmap v813: 129 pgs: 129 active+clean; 508 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.0 MiB/s wr, 600 op/s 2024-09-16T00:32:35.996 INFO:tasks.cephfs.filesystem:scrub status for tag:4d3d12a5-80b7-49b0-be33-ed29059dab8a - {'path': '/', 'tag': '4d3d12a5-80b7-49b0-be33-ed29059dab8a', 'options': 'recursive,force'} 2024-09-16T00:32:36.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 41463 23.66 MB/sec warmup 102 sec latency 57.037 ms 2024-09-16T00:32:36.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:35 smithi083 ceph-mon[35242]: pgmap v813: 129 pgs: 129 active+clean; 508 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.0 MiB/s wr, 600 op/s 2024-09-16T00:32:37.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 41953 23.70 MB/sec warmup 103 sec latency 103.463 ms 2024-09-16T00:32:37.567 INFO:tasks.quiescer.fs.[cephfs]:Total duration (35.4) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T00:32:37.567 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.7 2024-09-16T00:32:37.567 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:32:37.870 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:32:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:37 smithi060 ceph-mon[35185]: pgmap v814: 129 pgs: 129 active+clean; 512 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 6.2 MiB/s wr, 491 op/s 2024-09-16T00:32:37.994 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:37 smithi033 ceph-mon[31243]: pgmap v814: 129 pgs: 129 active+clean; 512 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 6.2 MiB/s wr, 491 op/s 2024-09-16T00:32:38.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 42557 23.70 MB/sec warmup 104 sec latency 87.424 ms 2024-09-16T00:32:38.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:37 smithi083 ceph-mon[35242]: pgmap v814: 129 pgs: 129 active+clean; 512 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 6.2 MiB/s wr, 491 op/s 2024-09-16T00:32:38.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.425+0000 7ff7439c5640 1 -- 172.21.15.33:0/3756914098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff73c10c180 msgr2=0x7ff73c10e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:38.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.425+0000 7ff7439c5640 1 --2- 172.21.15.33:0/3756914098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff73c10c180 0x7ff73c10e600 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7ff734009ca0 tx=0x7ff73402a950 comp rx=0 tx=0).stop 2024-09-16T00:32:38.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.425+0000 7ff7439c5640 1 -- 172.21.15.33:0/3756914098 shutdown_connections 2024-09-16T00:32:38.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.425+0000 7ff7439c5640 1 --2- 172.21.15.33:0/3756914098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff73c10c180 0x7ff73c10e600 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:38.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.425+0000 7ff7439c5640 1 --2- 172.21.15.33:0/3756914098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73c1036c0 0x7ff73c10bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:38.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.425+0000 7ff7439c5640 1 --2- 172.21.15.33:0/3756914098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff73c102da0 0x7ff73c103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:38.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.425+0000 7ff7439c5640 1 -- 172.21.15.33:0/3756914098 >> 172.21.15.33:0/3756914098 conn(0x7ff73c0fc7a0 msgr2=0x7ff73c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:38.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.425+0000 7ff7439c5640 1 -- 172.21.15.33:0/3756914098 shutdown_connections 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.425+0000 7ff7439c5640 1 -- 172.21.15.33:0/3756914098 wait complete. 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.426+0000 7ff7439c5640 1 Processor -- start 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.426+0000 7ff7439c5640 1 -- start start 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.426+0000 7ff7439c5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff73c102da0 0x7ff73c1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.426+0000 7ff7439c5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73c1036c0 0x7ff73c198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.426+0000 7ff7439c5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff73c10c180 0x7ff73c19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.426+0000 7ff7439c5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c111f60 con 0x7ff73c10c180 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.426+0000 7ff7439c5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c111de0 con 0x7ff73c1036c0 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.426+0000 7ff7439c5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c1120e0 con 0x7ff73c102da0 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.426+0000 7ff740f60640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73c1036c0 0x7ff73c198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.426+0000 7ff740f60640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73c1036c0 0x7ff73c198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:36676/0 (socket says 172.21.15.33:36676) 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.427+0000 7ff740f60640 1 -- 172.21.15.33:0/1703136504 learned_addr learned my addr 172.21.15.33:0/1703136504 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:38.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.427+0000 7ff741761640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff73c102da0 0x7ff73c1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.427+0000 7ff740f60640 1 -- 172.21.15.33:0/1703136504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff73c102da0 msgr2=0x7ff73c1983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.427+0000 7ff740f60640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff73c102da0 0x7ff73c1983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.428+0000 7ff740f60640 1 -- 172.21.15.33:0/1703136504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff73c10c180 msgr2=0x7ff73c19ccb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.428+0000 7ff740f60640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff73c10c180 0x7ff73c19ccb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.428+0000 7ff740f60640 1 -- 172.21.15.33:0/1703136504 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73c19d390 con 0x7ff73c1036c0 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.429+0000 7ff740f60640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73c1036c0 0x7ff73c198920 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff73000e9f0 tx=0x7ff73000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.432+0000 7ff72e7fc640 1 -- 172.21.15.33:0/1703136504 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73000cd80 con 0x7ff73c1036c0 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.432+0000 7ff72e7fc640 1 -- 172.21.15.33:0/1703136504 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff730010910 con 0x7ff73c1036c0 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.434+0000 7ff7439c5640 1 -- 172.21.15.33:0/1703136504 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff73c19d680 con 0x7ff73c1036c0 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.434+0000 7ff72e7fc640 1 -- 172.21.15.33:0/1703136504 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7300188f0 con 0x7ff73c1036c0 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.434+0000 7ff7439c5640 1 -- 172.21.15.33:0/1703136504 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff73c1a4f60 con 0x7ff73c1036c0 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.436+0000 7ff7439c5640 1 -- 172.21.15.33:0/1703136504 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff73c06b010 con 0x7ff73c1036c0 2024-09-16T00:32:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.438+0000 7ff72e7fc640 1 -- 172.21.15.33:0/1703136504 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff730010430 con 0x7ff73c1036c0 2024-09-16T00:32:38.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.439+0000 7ff72e7fc640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7280a3260 0x7ff7280a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:38.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.440+0000 7ff72e7fc640 1 -- 172.21.15.33:0/1703136504 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff7300d0040 con 0x7ff73c1036c0 2024-09-16T00:32:38.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.443+0000 7ff741761640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7280a3260 0x7ff7280a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:38.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.444+0000 7ff741761640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7280a3260 0x7ff7280a5720 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff738009b40 tx=0x7ff738005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:38.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.446+0000 7ff72e7fc640 1 -- 172.21.15.33:0/1703136504 <== mon.2 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff73009c8e0 con 0x7ff73c1036c0 2024-09-16T00:32:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.765+0000 7ff7439c5640 1 -- 172.21.15.33:0/1703136504 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff73c107220 con 0x7ff73c1036c0 2024-09-16T00:32:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.766+0000 7ff72e7fc640 1 -- 172.21.15.33:0/1703136504 <== mon.2 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11906 (secure 0 0 0) 0x7ff7300bae20 con 0x7ff73c1036c0 2024-09-16T00:32:38.766 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:32:38.766 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":31,"btime":"2024-09-16T00:30:57:641642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:30:57.027530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:32:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 31 2024-09-16T00:32:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.768+0000 7ff7439c5640 1 -- 172.21.15.33:0/1703136504 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7280a3260 msgr2=0x7ff7280a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.768+0000 7ff7439c5640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7280a3260 0x7ff7280a5720 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff738009b40 tx=0x7ff738005f90 comp rx=0 tx=0).stop 2024-09-16T00:32:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.768+0000 7ff7439c5640 1 -- 172.21.15.33:0/1703136504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73c1036c0 msgr2=0x7ff73c198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.768+0000 7ff7439c5640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73c1036c0 0x7ff73c198920 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff73000e9f0 tx=0x7ff73000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:32:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.768+0000 7ff7439c5640 1 -- 172.21.15.33:0/1703136504 shutdown_connections 2024-09-16T00:32:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.768+0000 7ff7439c5640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7280a3260 0x7ff7280a5720 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.768+0000 7ff7439c5640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff73c10c180 0x7ff73c19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.768+0000 7ff7439c5640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73c1036c0 0x7ff73c198920 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.768+0000 7ff7439c5640 1 --2- 172.21.15.33:0/1703136504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff73c102da0 0x7ff73c1983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.768+0000 7ff7439c5640 1 -- 172.21.15.33:0/1703136504 >> 172.21.15.33:0/1703136504 conn(0x7ff73c0fc7a0 msgr2=0x7ff73c0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.768+0000 7ff7439c5640 1 -- 172.21.15.33:0/1703136504 shutdown_connections 2024-09-16T00:32:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:38.768+0000 7ff7439c5640 1 -- 172.21.15.33:0/1703136504 wait complete. 2024-09-16T00:32:39.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 43089 23.75 MB/sec warmup 105 sec latency 79.288 ms 2024-09-16T00:32:39.453 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0' 2024-09-16T00:32:39.453 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0 2024-09-16T00:32:39.787 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:32:39.947 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:39 smithi033 ceph-mon[31243]: pgmap v815: 129 pgs: 129 active+clean; 516 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.0 MiB/s wr, 378 op/s 2024-09-16T00:32:39.947 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:39 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1703136504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:32:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:39 smithi060 ceph-mon[35185]: pgmap v815: 129 pgs: 129 active+clean; 516 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.0 MiB/s wr, 378 op/s 2024-09-16T00:32:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:39 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1703136504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:32:40.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 43802 23.84 MB/sec warmup 106 sec latency 61.776 ms 2024-09-16T00:32:40.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:39 smithi083 ceph-mon[35242]: pgmap v815: 129 pgs: 129 active+clean; 516 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.0 MiB/s wr, 378 op/s 2024-09-16T00:32:40.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:39 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1703136504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:32:40.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.328+0000 7f3369e4d640 1 -- 172.21.15.33:0/4188066889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3364102d50 msgr2=0x7f3364103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:40.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.328+0000 7f3369e4d640 1 --2- 172.21.15.33:0/4188066889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3364102d50 0x7f3364103130 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f3354009900 tx=0x7f335402a480 comp rx=0 tx=0).stop 2024-09-16T00:32:40.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.328+0000 7f3369e4d640 1 -- 172.21.15.33:0/4188066889 shutdown_connections 2024-09-16T00:32:40.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.328+0000 7f3369e4d640 1 --2- 172.21.15.33:0/4188066889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f336410c180 0x7f336410e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.328+0000 7f3369e4d640 1 --2- 172.21.15.33:0/4188066889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3364103670 0x7f336410bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.328+0000 7f3369e4d640 1 --2- 172.21.15.33:0/4188066889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3364102d50 0x7f3364103130 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.328+0000 7f3369e4d640 1 -- 172.21.15.33:0/4188066889 >> 172.21.15.33:0/4188066889 conn(0x7f33640fc7d0 msgr2=0x7f33640febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:40.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.328+0000 7f3369e4d640 1 -- 172.21.15.33:0/4188066889 shutdown_connections 2024-09-16T00:32:40.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.328+0000 7f3369e4d640 1 -- 172.21.15.33:0/4188066889 wait complete. 2024-09-16T00:32:40.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.329+0000 7f3369e4d640 1 Processor -- start 2024-09-16T00:32:40.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.329+0000 7f3369e4d640 1 -- start start 2024-09-16T00:32:40.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.329+0000 7f3369e4d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3364102d50 0x7f33641009b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.329+0000 7f3369e4d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3364103670 0x7f3364100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.329+0000 7f3369e4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f336410c180 0x7f3364101430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.329+0000 7f3369e4d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3364111f80 con 0x7f3364103670 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.329+0000 7f3369e4d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3364111e00 con 0x7f336410c180 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.329+0000 7f3369e4d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3364112100 con 0x7f3364102d50 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.329+0000 7f3362ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3364103670 0x7f3364100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.329+0000 7f3362ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3364103670 0x7f3364100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60460/0 (socket says 172.21.15.33:60460) 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.329+0000 7f3362ffd640 1 -- 172.21.15.33:0/2457343106 learned_addr learned my addr 172.21.15.33:0/2457343106 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.329+0000 7f33637fe640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3364102d50 0x7f33641009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.330+0000 7f3363fff640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f336410c180 0x7f3364101430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.330+0000 7f3362ffd640 1 -- 172.21.15.33:0/2457343106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3364102d50 msgr2=0x7f33641009b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.330+0000 7f3362ffd640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3364102d50 0x7f33641009b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.330+0000 7f3362ffd640 1 -- 172.21.15.33:0/2457343106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f336410c180 msgr2=0x7f3364101430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:40.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.330+0000 7f3362ffd640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f336410c180 0x7f3364101430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.330+0000 7f3362ffd640 1 -- 172.21.15.33:0/2457343106 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33641a6900 con 0x7f3364103670 2024-09-16T00:32:40.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.330+0000 7f3362ffd640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3364103670 0x7f3364100ef0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f335800e9f0 tx=0x7f335800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:40.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.330+0000 7f3360ff9640 1 -- 172.21.15.33:0/2457343106 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335800ccc0 con 0x7f3364103670 2024-09-16T00:32:40.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.330+0000 7f3369e4d640 1 -- 172.21.15.33:0/2457343106 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33641a6bf0 con 0x7f3364103670 2024-09-16T00:32:40.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.330+0000 7f3360ff9640 1 -- 172.21.15.33:0/2457343106 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3358010910 con 0x7f3364103670 2024-09-16T00:32:40.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.330+0000 7f3360ff9640 1 -- 172.21.15.33:0/2457343106 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3358018920 con 0x7f3364103670 2024-09-16T00:32:40.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.330+0000 7f3369e4d640 1 -- 172.21.15.33:0/2457343106 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33641a7130 con 0x7f3364103670 2024-09-16T00:32:40.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.333+0000 7f3360ff9640 1 -- 172.21.15.33:0/2457343106 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3358010430 con 0x7f3364103670 2024-09-16T00:32:40.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.334+0000 7f336c226740 1 Processor -- start 2024-09-16T00:32:40.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.334+0000 7f336c226740 1 -- start start 2024-09-16T00:32:40.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.334+0000 7f336c226740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.334+0000 7f336c226740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 0x5615816d3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.334+0000 7f336c226740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816d3940 0x5615816d5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.334+0000 7f336c226740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56158162e2a0 con 0x5615816c9080 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.334+0000 7f336c226740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5615815d5b10 con 0x5615816d3940 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.334+0000 7f336c226740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56158162a260 con 0x5615816c8760 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.334+0000 7f3363fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.334+0000 7f33637fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 0x5615816d3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.334+0000 7f3363fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816c8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60744/0 (socket says 172.21.15.33:60744) 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.334+0000 7f3363fff640 1 -- 172.21.15.33:0/2462363440 learned_addr learned my addr 172.21.15.33:0/2462363440 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.335+0000 7f3363fff640 1 -- 172.21.15.33:0/2462363440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816d3940 msgr2=0x5615816d5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.335+0000 7f3363fff640 1 --2- 172.21.15.33:0/2462363440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816d3940 0x5615816d5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.335+0000 7f3363fff640 1 -- 172.21.15.33:0/2462363440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 msgr2=0x5615816d3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:40.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.335+0000 7f3363fff640 1 --2- 172.21.15.33:0/2462363440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 0x5615816d3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.335+0000 7f3363fff640 1 -- 172.21.15.33:0/2462363440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56158139fb90 con 0x5615816c8760 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.335+0000 7f33637fe640 1 --2- 172.21.15.33:0/2462363440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 0x5615816d3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.335+0000 7f3360ff9640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f334c0a3240 0x7f334c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.335+0000 7f3363fff640 1 --2- 172.21.15.33:0/2462363440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816c8b40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f335c009f50 tx=0x7f335c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13c5073489a4489b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.335+0000 7f3360ff9640 1 -- 172.21.15.33:0/2457343106 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f33580cffb0 con 0x7f3364103670 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.335+0000 7f33637fe640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f334c0a3240 0x7f334c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.336+0000 7f3349ffb640 1 -- 172.21.15.33:0/2462363440 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c004d20 con 0x5615816c8760 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.336+0000 7f3349ffb640 1 -- 172.21.15.33:0/2462363440 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f335c004050 con 0x5615816c8760 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.336+0000 7f3349ffb640 1 -- 172.21.15.33:0/2462363440 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c005280 con 0x5615816c8760 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.336+0000 7f336c226740 1 -- 172.21.15.33:0/2462363440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 msgr2=0x5615816c8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.336+0000 7f336c226740 1 --2- 172.21.15.33:0/2462363440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816c8b40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f335c009f50 tx=0x7f335c0118d0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.337+0000 7f33637fe640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f334c0a3240 0x7f334c0a5700 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3354009230 tx=0x7f33540058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:40.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.337+0000 7f336c226740 1 -- 172.21.15.33:0/2462363440 shutdown_connections 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.337+0000 7f336c226740 1 --2- 172.21.15.33:0/2462363440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816d3940 0x5615816d5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.337+0000 7f336c226740 1 --2- 172.21.15.33:0/2462363440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 0x5615816d3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.337+0000 7f336c226740 1 --2- 172.21.15.33:0/2462363440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816c8b40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.337+0000 7f336c226740 1 -- 172.21.15.33:0/2462363440 >> 172.21.15.33:0/2462363440 conn(0x5615816c4cf0 msgr2=0x5615816c50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.337+0000 7f336c226740 1 -- 172.21.15.33:0/2462363440 shutdown_connections 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.337+0000 7f336c226740 1 -- 172.21.15.33:0/2462363440 wait complete. 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.338+0000 7f336c226740 1 -- start start 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.338+0000 7f336c226740 1 Processor -- start 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.338+0000 7f336c226740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c8760 0x5615816d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.338+0000 7f336c226740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c9080 0x5615816d0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f336c226740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816d3940 0x5615816d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f336c226740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5615816d6510 con 0x5615816d3940 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f336c226740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5615815b66e0 con 0x5615816c8760 2024-09-16T00:32:40.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f336c226740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5615816d6690 con 0x5615816c9080 2024-09-16T00:32:40.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f33637fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c8760 0x5615816d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f33637fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c8760 0x5615816d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:54994/0 (socket says 172.21.15.33:54994) 2024-09-16T00:32:40.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f33637fe640 1 -- 172.21.15.33:0/280916259 learned_addr learned my addr 172.21.15.33:0/280916259 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:40.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f336c226740 10 client.0 _ll_register_callbacks cb 0x5615815daa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:40.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f3363fff640 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816d3940 0x5615816d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f3362ffd640 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c9080 0x5615816d0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f33637fe640 1 -- 172.21.15.33:0/280916259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c9080 msgr2=0x5615816d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:40.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f33637fe640 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c9080 0x5615816d0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f33637fe640 1 -- 172.21.15.33:0/280916259 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816d3940 msgr2=0x5615816d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:40.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f33637fe640 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816d3940 0x5615816d1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f33637fe640 1 -- 172.21.15.33:0/280916259 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615816dfa80 con 0x5615816c8760 2024-09-16T00:32:40.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f33637fe640 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c8760 0x5615816d08a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3354033000 tx=0x7f3354009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dadc3f5878180a11 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.339+0000 7f3342ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.340+0000 7f3342ffd640 1 -- 172.21.15.33:0/280916259 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335404e070 con 0x5615816c8760 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.340+0000 7f3342ffd640 1 -- 172.21.15.33:0/280916259 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3354003b80 con 0x5615816c8760 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.340+0000 7f336c226740 1 -- 172.21.15.33:0/280916259 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f335c004050 con 0x5615816c8760 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.340+0000 7f3342ffd640 1 -- 172.21.15.33:0/280916259 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f335405a050 con 0x5615816c8760 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.340+0000 7f336c226740 10 client.27668 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.340+0000 7f336c226740 1 -- 172.21.15.33:0/280916259 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5615816eac70 con 0x5615816c8760 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.340+0000 7f3342ffd640 1 -- 172.21.15.33:0/280916259 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f335c004050 con 0x5615816c8760 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.341+0000 7f3342ffd640 1 -- 172.21.15.33:0/280916259 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f335403d450 con 0x5615816c8760 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.341+0000 7f336c226740 10 client.27668 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.341+0000 7f336c226740 10 client.27668 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.341+0000 7f336c226740 20 client.27668 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:40.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.341+0000 7f336c226740 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5615816ec1e0 0x5615816ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.341+0000 7f336c226740 4 client.27668 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.341+0000 7f336c226740 1 -- 172.21.15.33:0/280916259 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56158163b9d0 con 0x5615816ec1e0 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.341+0000 7f3362ffd640 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5615816ec1e0 0x5615816ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.341+0000 7f3342ffd640 10 client.27668 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.341+0000 7f3362ffd640 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5615816ec1e0 0x5615816ee5f0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40657d760c8aeec3 server_cookie=668ffea6d4f08113 in_seq=0 out_seq=0 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.342+0000 7f3342ffd640 1 -- 172.21.15.33:0/280916259 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56158163b9d0 con 0x5615816ec1e0 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.342+0000 7f3342ffd640 10 client.27668 handle_command_reply: tid=0 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.342+0000 7f336c226740 1 client.27668 shutdown 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.342+0000 7f336c226740 1 -- 172.21.15.33:0/280916259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c8760 msgr2=0x5615816d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.342+0000 7f336c226740 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c8760 0x5615816d08a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3354033000 tx=0x7f3354009260 comp rx=0 tx=0).stop 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.343+0000 7f336c226740 1 -- 172.21.15.33:0/280916259 shutdown_connections 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.343+0000 7f336c226740 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5615816ec1e0 0x5615816ee5f0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.343+0000 7f336c226740 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816d3940 0x5615816d1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.343+0000 7f3342ffd640 0 client.27668 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.343+0000 7f336c226740 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c9080 0x5615816d0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.343+0000 7f336c226740 1 --2- 172.21.15.33:0/280916259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c8760 0x5615816d08a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.343+0000 7f336c226740 1 -- 172.21.15.33:0/280916259 >> 172.21.15.33:0/280916259 conn(0x5615816c4cf0 msgr2=0x5615816caa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.343+0000 7f336c226740 1 -- 172.21.15.33:0/280916259 shutdown_connections 2024-09-16T00:32:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.343+0000 7f336c226740 1 -- 172.21.15.33:0/280916259 wait complete. 2024-09-16T00:32:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.343+0000 7f336c226740 20 client.27668 trim_cache size 0 max 16384 2024-09-16T00:32:40.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f336c226740 1 Processor -- start 2024-09-16T00:32:40.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f336c226740 1 -- start start 2024-09-16T00:32:40.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f336c226740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816d1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f336c226740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 0x5615816f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f336c226740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816d3940 0x5615816f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f336c226740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f33641af7e0 con 0x5615816c9080 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f336c226740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f33641ad050 con 0x5615816d3940 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f336c226740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f33641aaec0 con 0x5615816c8760 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f3362ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 0x5615816f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f3362ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 0x5615816f3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60490/0 (socket says 172.21.15.33:60490) 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f3362ffd640 1 -- 172.21.15.33:0/1842297170 learned_addr learned my addr 172.21.15.33:0/1842297170 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f33637fe640 1 --2- 172.21.15.33:0/1842297170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816d1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f3363fff640 1 --2- 172.21.15.33:0/1842297170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816d3940 0x5615816f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f3362ffd640 1 -- 172.21.15.33:0/1842297170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 msgr2=0x5615816d1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f3362ffd640 1 --2- 172.21.15.33:0/1842297170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816d1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f3362ffd640 1 -- 172.21.15.33:0/1842297170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816d3940 msgr2=0x5615816f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f3362ffd640 1 --2- 172.21.15.33:0/1842297170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816d3940 0x5615816f42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.365+0000 7f3362ffd640 1 -- 172.21.15.33:0/1842297170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615816c0260 con 0x5615816c9080 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.366+0000 7f3362ffd640 1 --2- 172.21.15.33:0/1842297170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 0x5615816f3d60 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f33580cd630 tx=0x7f335800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b0bdc04e30485d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:40.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.366+0000 7f33497fa640 1 -- 172.21.15.33:0/1842297170 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33580b5070 con 0x5615816c9080 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.366+0000 7f33497fa640 1 -- 172.21.15.33:0/1842297170 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3358010720 con 0x5615816c9080 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.366+0000 7f33497fa640 1 -- 172.21.15.33:0/1842297170 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33580bc0b0 con 0x5615816c9080 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.367+0000 7f336c226740 1 -- 172.21.15.33:0/1842297170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 msgr2=0x5615816f3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.367+0000 7f336c226740 1 --2- 172.21.15.33:0/1842297170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 0x5615816f3d60 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f33580cd630 tx=0x7f335800cf60 comp rx=0 tx=0).stop 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.367+0000 7f336c226740 1 -- 172.21.15.33:0/1842297170 shutdown_connections 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.367+0000 7f336c226740 1 --2- 172.21.15.33:0/1842297170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816d3940 0x5615816f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.367+0000 7f336c226740 1 --2- 172.21.15.33:0/1842297170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816c9080 0x5615816f3d60 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.367+0000 7f336c226740 1 --2- 172.21.15.33:0/1842297170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816d1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.367+0000 7f336c226740 1 -- 172.21.15.33:0/1842297170 >> 172.21.15.33:0/1842297170 conn(0x5615816c4cf0 msgr2=0x5615816ca9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.367+0000 7f336c226740 1 -- 172.21.15.33:0/1842297170 shutdown_connections 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.367+0000 7f336c226740 1 -- 172.21.15.33:0/1842297170 wait complete. 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.367+0000 7f336c226740 1 -- start start 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.367+0000 7f336c226740 1 Processor -- start 2024-09-16T00:32:40.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f336c226740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816dc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f336c226740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c9080 0x5615816cd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f336c226740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816d3940 0x5615816cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f336c226740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f33641ab510 con 0x5615816d3940 2024-09-16T00:32:40.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f336c226740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f33641ac980 con 0x5615816c9080 2024-09-16T00:32:40.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f336c226740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f33641abbe0 con 0x5615816c8760 2024-09-16T00:32:40.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f336c226740 10 client.0 _ll_register_callbacks cb 0x7f3354009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:40.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f3363fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816d3940 0x5615816cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f33637fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816dc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f3362ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c9080 0x5615816cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f33637fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816dc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:60760/0 (socket says 172.21.15.33:60760) 2024-09-16T00:32:40.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f3363fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816d3940 0x5615816cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60496/0 (socket says 172.21.15.33:60496) 2024-09-16T00:32:40.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f33637fe640 1 -- 172.21.15.33:0/393453311 learned_addr learned my addr 172.21.15.33:0/393453311 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f3362ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c9080 0x5615816cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55012/0 (socket says 172.21.15.33:55012) 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f33637fe640 1 -- 172.21.15.33:0/393453311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c9080 msgr2=0x5615816cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f33637fe640 1 --2- 172.21.15.33:0/393453311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c9080 0x5615816cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f33637fe640 1 -- 172.21.15.33:0/393453311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816d3940 msgr2=0x5615816cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f33637fe640 1 --2- 172.21.15.33:0/393453311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816d3940 0x5615816cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f33637fe640 1 -- 172.21.15.33:0/393453311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561581755d30 con 0x5615816c8760 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f3363fff640 1 --2- 172.21.15.33:0/393453311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816d3940 0x5615816cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f33637fe640 1 --2- 172.21.15.33:0/393453311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816dc1f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f335403d8a0 tx=0x7f335403dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ab30979559b4d41 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.368+0000 7f3342ffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f3342ffd640 1 -- 172.21.15.33:0/393453311 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354062070 con 0x5615816c8760 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f336c226740 1 -- 172.21.15.33:0/393453311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3358010720 con 0x5615816c8760 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f3342ffd640 1 -- 172.21.15.33:0/393453311 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3354003d90 con 0x5615816c8760 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f3342ffd640 1 -- 172.21.15.33:0/393453311 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f335406f050 con 0x5615816c8760 2024-09-16T00:32:40.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f336c226740 10 client.27670 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:40.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f336c226740 1 -- 172.21.15.33:0/393453311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561581756e70 con 0x5615816c8760 2024-09-16T00:32:40.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f3342ffd640 1 -- 172.21.15.33:0/393453311 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3358010720 con 0x5615816c8760 2024-09-16T00:32:40.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f3342ffd640 1 -- 172.21.15.33:0/393453311 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f335404aca0 con 0x5615816c8760 2024-09-16T00:32:40.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f336c226740 10 client.27670 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:40.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f336c226740 10 client.27670 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:32:40.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f336c226740 20 client.27670 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:40.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f336c226740 1 --2- 172.21.15.33:0/393453311 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561581757630 0x561581759a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:40.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f336c226740 4 client.27670 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}] 2024-09-16T00:32:40.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.369+0000 7f336c226740 1 -- 172.21.15.33:0/393453311 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}) -- 0x561581635d20 con 0x561581757630 2024-09-16T00:32:40.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.370+0000 7f3362ffd640 1 --2- 172.21.15.33:0/393453311 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561581757630 0x561581759a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:40.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.370+0000 7f3362ffd640 1 --2- 172.21.15.33:0/393453311 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561581757630 0x561581759a50 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42ad9e339e256061 server_cookie=e77694969653b690 in_seq=0 out_seq=0 2024-09-16T00:32:40.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:40.370+0000 7f3342ffd640 10 client.27670 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:41.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44012 23.74 MB/sec warmup 107 sec latency 724.479 ms 2024-09-16T00:32:41.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.179+0000 7f3342ffd640 1 -- 172.21.15.33:0/393453311 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x561581635d20 con 0x561581757630 2024-09-16T00:32:41.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.179+0000 7f3342ffd640 10 client.27670 handle_command_reply: tid=0 2024-09-16T00:32:41.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.179+0000 7f336c226740 1 client.27670 shutdown 2024-09-16T00:32:41.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.179+0000 7f336c226740 1 -- 172.21.15.33:0/393453311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 msgr2=0x5615816dc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:41.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.179+0000 7f336c226740 1 --2- 172.21.15.33:0/393453311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816dc1f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f335403d8a0 tx=0x7f335403dcc0 comp rx=0 tx=0).stop 2024-09-16T00:32:41.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.180+0000 7f336c226740 1 -- 172.21.15.33:0/393453311 shutdown_connections 2024-09-16T00:32:41.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.180+0000 7f336c226740 1 --2- 172.21.15.33:0/393453311 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561581757630 0x561581759a50 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:41.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.180+0000 7f336c226740 1 --2- 172.21.15.33:0/393453311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615816d3940 0x5615816cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:41.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.180+0000 7f3342ffd640 0 client.27670 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:41.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.180+0000 7f336c226740 1 --2- 172.21.15.33:0/393453311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5615816c9080 0x5615816cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:41.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.180+0000 7f336c226740 1 --2- 172.21.15.33:0/393453311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5615816c8760 0x5615816dc1f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:41.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.180+0000 7f336c226740 1 -- 172.21.15.33:0/393453311 >> 172.21.15.33:0/393453311 conn(0x5615816c4cf0 msgr2=0x5615816ca1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:41.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.180+0000 7f336c226740 1 -- 172.21.15.33:0/393453311 shutdown_connections 2024-09-16T00:32:41.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.180+0000 7f336c226740 1 -- 172.21.15.33:0/393453311 wait complete. 2024-09-16T00:32:41.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.181+0000 7f336c226740 20 client.27670 trim_cache size 0 max 16384 2024-09-16T00:32:41.182 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:32:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 31, 2024-09-16T00:32:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:32:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 54, 2024-09-16T00:32:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:32:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "113b4b23": { 2024-09-16T00:32:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 54, 2024-09-16T00:32:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:32:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:32:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:32:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:32:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:32:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 137.7, 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:32:41.184 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:32:41.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.182+0000 7f3369e4d640 1 -- 172.21.15.33:0/2457343106 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f334c0a3240 msgr2=0x7f334c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:41.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.182+0000 7f3369e4d640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f334c0a3240 0x7f334c0a5700 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3354009230 tx=0x7f33540058d0 comp rx=0 tx=0).stop 2024-09-16T00:32:41.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.182+0000 7f3369e4d640 1 -- 172.21.15.33:0/2457343106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3364103670 msgr2=0x7f3364100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:41.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.182+0000 7f3369e4d640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3364103670 0x7f3364100ef0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f335800e9f0 tx=0x7f335800eeb0 comp rx=0 tx=0).stop 2024-09-16T00:32:41.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.182+0000 7f3369e4d640 1 -- 172.21.15.33:0/2457343106 shutdown_connections 2024-09-16T00:32:41.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.182+0000 7f3369e4d640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f334c0a3240 0x7f334c0a5700 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:41.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.182+0000 7f3369e4d640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f336410c180 0x7f3364101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:41.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.182+0000 7f3369e4d640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3364103670 0x7f3364100ef0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:41.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.182+0000 7f3369e4d640 1 --2- 172.21.15.33:0/2457343106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3364102d50 0x7f33641009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:41.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.182+0000 7f3369e4d640 1 -- 172.21.15.33:0/2457343106 >> 172.21.15.33:0/2457343106 conn(0x7f33640fc7d0 msgr2=0x7f33640fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:41.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.182+0000 7f3369e4d640 1 -- 172.21.15.33:0/2457343106 shutdown_connections 2024-09-16T00:32:41.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:41.182+0000 7f3369e4d640 1 -- 172.21.15.33:0/2457343106 wait complete. 2024-09-16T00:32:41.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:41 smithi033 ceph-mon[31243]: pgmap v816: 129 pgs: 129 active+clean; 512 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.4 MiB/s wr, 429 op/s 2024-09-16T00:32:41.920 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '113b4b23', quiesce took 4.4 seconds. Will release after: 13.299999999999999 2024-09-16T00:32:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:41 smithi060 ceph-mon[35185]: pgmap v816: 129 pgs: 129 active+clean; 512 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.4 MiB/s wr, 429 op/s 2024-09-16T00:32:42.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 23.52 MB/sec warmup 108 sec latency 1724.523 ms 2024-09-16T00:32:42.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:41 smithi083 ceph-mon[35242]: pgmap v816: 129 pgs: 129 active+clean; 512 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.4 MiB/s wr, 429 op/s 2024-09-16T00:32:43.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 23.30 MB/sec warmup 109 sec latency 2724.627 ms 2024-09-16T00:32:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:43 smithi060 ceph-mon[35185]: pgmap v817: 129 pgs: 129 active+clean; 509 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.6 MiB/s wr, 404 op/s 2024-09-16T00:32:44.070 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 23.09 MB/sec warmup 110 sec latency 3724.743 ms 2024-09-16T00:32:44.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:43 smithi033 ceph-mon[31243]: pgmap v817: 129 pgs: 129 active+clean; 509 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.6 MiB/s wr, 404 op/s 2024-09-16T00:32:44.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:43 smithi083 ceph-mon[35242]: pgmap v817: 129 pgs: 129 active+clean; 509 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.6 MiB/s wr, 404 op/s 2024-09-16T00:32:45.071 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 22.88 MB/sec warmup 111 sec latency 4724.854 ms 2024-09-16T00:32:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:45 smithi060 ceph-mon[35185]: pgmap v818: 129 pgs: 129 active+clean; 508 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.3 MiB/s wr, 387 op/s 2024-09-16T00:32:46.071 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 22.68 MB/sec warmup 112 sec latency 5724.959 ms 2024-09-16T00:32:46.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:45 smithi033 ceph-mon[31243]: pgmap v818: 129 pgs: 129 active+clean; 508 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.3 MiB/s wr, 387 op/s 2024-09-16T00:32:46.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:45 smithi083 ceph-mon[35242]: pgmap v818: 129 pgs: 129 active+clean; 508 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.3 MiB/s wr, 387 op/s 2024-09-16T00:32:46.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:46 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:32:47.071 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 22.48 MB/sec warmup 113 sec latency 6725.061 ms 2024-09-16T00:32:47.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:46 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:32:47.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:46 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:32:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:47 smithi060 ceph-mon[35185]: pgmap v819: 129 pgs: 129 active+clean; 508 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 2.6 MiB/s wr, 335 op/s 2024-09-16T00:32:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:47 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:32:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:47 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:32:48.071 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 22.28 MB/sec warmup 114 sec latency 7725.101 ms 2024-09-16T00:32:48.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:47 smithi033 ceph-mon[31243]: pgmap v819: 129 pgs: 129 active+clean; 508 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 2.6 MiB/s wr, 335 op/s 2024-09-16T00:32:48.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:47 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:32:48.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:47 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:32:48.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:47 smithi083 ceph-mon[35242]: pgmap v819: 129 pgs: 129 active+clean; 508 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 2.6 MiB/s wr, 335 op/s 2024-09-16T00:32:48.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:47 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:32:48.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:47 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:32:49.071 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 22.09 MB/sec warmup 115 sec latency 8725.181 ms 2024-09-16T00:32:49.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:48 smithi033 ceph-mon[31243]: pgmap v820: 129 pgs: 129 active+clean; 508 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.2 MiB/s wr, 284 op/s 2024-09-16T00:32:49.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:48 smithi083 ceph-mon[35242]: pgmap v820: 129 pgs: 129 active+clean; 508 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.2 MiB/s wr, 284 op/s 2024-09-16T00:32:49.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:48 smithi060 ceph-mon[35185]: pgmap v820: 129 pgs: 129 active+clean; 508 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.2 MiB/s wr, 284 op/s 2024-09-16T00:32:50.071 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 21.90 MB/sec warmup 116 sec latency 9725.220 ms 2024-09-16T00:32:51.071 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 21.71 MB/sec warmup 117 sec latency 10725.337 ms 2024-09-16T00:32:51.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:51 smithi033 ceph-mon[31243]: pgmap v821: 129 pgs: 129 active+clean; 546 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.4 MiB/s wr, 310 op/s 2024-09-16T00:32:51.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:51 smithi083 ceph-mon[35242]: pgmap v821: 129 pgs: 129 active+clean; 546 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.4 MiB/s wr, 310 op/s 2024-09-16T00:32:51.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:51 smithi060 ceph-mon[35185]: pgmap v821: 129 pgs: 129 active+clean; 546 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.4 MiB/s wr, 310 op/s 2024-09-16T00:32:52.071 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 21.53 MB/sec warmup 118 sec latency 11725.394 ms 2024-09-16T00:32:53.071 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 21.34 MB/sec warmup 119 sec latency 12725.512 ms 2024-09-16T00:32:53.282 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:53 smithi060 ceph-mon[35185]: pgmap v822: 129 pgs: 129 active+clean; 563 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 9.1 MiB/s wr, 355 op/s 2024-09-16T00:32:53.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:53 smithi033 ceph-mon[31243]: pgmap v822: 129 pgs: 129 active+clean; 563 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 9.1 MiB/s wr, 355 op/s 2024-09-16T00:32:53.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:53 smithi083 ceph-mon[35242]: pgmap v822: 129 pgs: 129 active+clean; 563 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 9.1 MiB/s wr, 355 op/s 2024-09-16T00:32:55.071 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 0.00 MB/sec execute 1 sec latency 14725.723 ms 2024-09-16T00:32:55.221 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:32:55.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:55 smithi033 ceph-mon[31243]: pgmap v823: 129 pgs: 129 active+clean; 545 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 9.5 MiB/s wr, 338 op/s 2024-09-16T00:32:55.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:55 smithi083 ceph-mon[35242]: pgmap v823: 129 pgs: 129 active+clean; 545 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 9.5 MiB/s wr, 338 op/s 2024-09-16T00:32:55.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:55 smithi060 ceph-mon[35185]: pgmap v823: 129 pgs: 129 active+clean; 545 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 9.5 MiB/s wr, 338 op/s 2024-09-16T00:32:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:32:56.071 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 0.00 MB/sec execute 2 sec latency 15725.790 ms 2024-09-16T00:32:56.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.100+0000 7ffa51f7b640 1 -- 172.21.15.33:0/3792315022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa4c100a80 msgr2=0x7ffa4c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:56.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.100+0000 7ffa51f7b640 1 --2- 172.21.15.33:0/3792315022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa4c100a80 0x7ffa4c100ee0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7ffa34009930 tx=0x7ffa3402a420 comp rx=0 tx=0).stop 2024-09-16T00:32:56.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.101+0000 7ffa51f7b640 1 -- 172.21.15.33:0/3792315022 shutdown_connections 2024-09-16T00:32:56.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.101+0000 7ffa51f7b640 1 --2- 172.21.15.33:0/3792315022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa4c101420 0x7ffa4c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:56.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.101+0000 7ffa51f7b640 1 --2- 172.21.15.33:0/3792315022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa4c100a80 0x7ffa4c100ee0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:56.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.101+0000 7ffa51f7b640 1 --2- 172.21.15.33:0/3792315022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa4c106a80 0x7ffa4c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:56.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.101+0000 7ffa51f7b640 1 -- 172.21.15.33:0/3792315022 >> 172.21.15.33:0/3792315022 conn(0x7ffa4c0fc7a0 msgr2=0x7ffa4c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:56.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.101+0000 7ffa51f7b640 1 -- 172.21.15.33:0/3792315022 shutdown_connections 2024-09-16T00:32:56.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.101+0000 7ffa51f7b640 1 -- 172.21.15.33:0/3792315022 wait complete. 2024-09-16T00:32:56.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.101+0000 7ffa51f7b640 1 Processor -- start 2024-09-16T00:32:56.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.101+0000 7ffa51f7b640 1 -- start start 2024-09-16T00:32:56.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa51f7b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa4c100a80 0x7ffa4c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:56.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa51f7b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa4c101420 0x7ffa4c198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:56.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa51f7b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa4c106a80 0x7ffa4c19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:56.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa51f7b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c110ea0 con 0x7ffa4c106a80 2024-09-16T00:32:56.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa51f7b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c110d20 con 0x7ffa4c101420 2024-09-16T00:32:56.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa51f7b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c111020 con 0x7ffa4c100a80 2024-09-16T00:32:56.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa4bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa4c106a80 0x7ffa4c19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:56.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa4bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa4c106a80 0x7ffa4c19cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35724/0 (socket says 172.21.15.33:35724) 2024-09-16T00:32:56.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa4bfff640 1 -- 172.21.15.33:0/2139131357 learned_addr learned my addr 172.21.15.33:0/2139131357 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:56.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa4affd640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa4c101420 0x7ffa4c198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:56.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa4b7fe640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa4c100a80 0x7ffa4c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:56.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa4bfff640 1 -- 172.21.15.33:0/2139131357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa4c100a80 msgr2=0x7ffa4c1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:56.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa4bfff640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa4c100a80 0x7ffa4c1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:56.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa4bfff640 1 -- 172.21.15.33:0/2139131357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa4c101420 msgr2=0x7ffa4c198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa4bfff640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa4c101420 0x7ffa4c198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa4bfff640 1 -- 172.21.15.33:0/2139131357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4c19d480 con 0x7ffa4c106a80 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa4bfff640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa4c106a80 0x7ffa4c19cda0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7ffa4400c970 tx=0x7ffa4400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa48ff9640 1 -- 172.21.15.33:0/2139131357 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa44007bb0 con 0x7ffa4c106a80 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa51f7b640 1 -- 172.21.15.33:0/2139131357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4c19d770 con 0x7ffa4c106a80 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.102+0000 7ffa51f7b640 1 -- 172.21.15.33:0/2139131357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa4c1a5050 con 0x7ffa4c106a80 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.103+0000 7ffa48ff9640 1 -- 172.21.15.33:0/2139131357 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa44007d50 con 0x7ffa4c106a80 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.103+0000 7ffa48ff9640 1 -- 172.21.15.33:0/2139131357 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4401d060 con 0x7ffa4c106a80 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.103+0000 7ffa51f7b640 1 -- 172.21.15.33:0/2139131357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa2c0052c0 con 0x7ffa4c106a80 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.105+0000 7ffa48ff9640 1 -- 172.21.15.33:0/2139131357 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ffa440046c0 con 0x7ffa4c106a80 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.105+0000 7ffa48ff9640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffa380a3240 0x7ffa380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.106+0000 7ffa4b7fe640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffa380a3240 0x7ffa380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:56.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.106+0000 7ffa48ff9640 1 -- 172.21.15.33:0/2139131357 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ffa440cfd50 con 0x7ffa4c106a80 2024-09-16T00:32:56.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.106+0000 7ffa4b7fe640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffa380a3240 0x7ffa380a5700 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009f20 tx=0x7ffa3c0094f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:56.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.109+0000 7ffa48ff9640 1 -- 172.21.15.33:0/2139131357 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffa44014030 con 0x7ffa4c106a80 2024-09-16T00:32:56.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.320+0000 7ffa51f7b640 1 -- 172.21.15.33:0/2139131357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa2c0055b0 con 0x7ffa4c106a80 2024-09-16T00:32:56.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.321+0000 7ffa48ff9640 1 -- 172.21.15.33:0/2139131357 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11906 (secure 0 0 0) 0x7ffa4409c590 con 0x7ffa4c106a80 2024-09-16T00:32:56.321 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:32:56.322 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":31,"btime":"2024-09-16T00:30:57:641642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:30:57.027530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:32:56.322 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 31 2024-09-16T00:32:56.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.323+0000 7ffa51f7b640 1 -- 172.21.15.33:0/2139131357 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffa380a3240 msgr2=0x7ffa380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:56.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.323+0000 7ffa51f7b640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffa380a3240 0x7ffa380a5700 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009f20 tx=0x7ffa3c0094f0 comp rx=0 tx=0).stop 2024-09-16T00:32:56.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.323+0000 7ffa51f7b640 1 -- 172.21.15.33:0/2139131357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa4c106a80 msgr2=0x7ffa4c19cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:56.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.323+0000 7ffa51f7b640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa4c106a80 0x7ffa4c19cda0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7ffa4400c970 tx=0x7ffa4400ce30 comp rx=0 tx=0).stop 2024-09-16T00:32:56.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.323+0000 7ffa51f7b640 1 -- 172.21.15.33:0/2139131357 shutdown_connections 2024-09-16T00:32:56.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.323+0000 7ffa51f7b640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffa380a3240 0x7ffa380a5700 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:56.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.323+0000 7ffa51f7b640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa4c106a80 0x7ffa4c19cda0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:56.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.323+0000 7ffa51f7b640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa4c101420 0x7ffa4c198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:56.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.323+0000 7ffa51f7b640 1 --2- 172.21.15.33:0/2139131357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa4c100a80 0x7ffa4c1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:56.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.323+0000 7ffa51f7b640 1 -- 172.21.15.33:0/2139131357 >> 172.21.15.33:0/2139131357 conn(0x7ffa4c0fc7a0 msgr2=0x7ffa4c06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:56.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.323+0000 7ffa51f7b640 1 -- 172.21.15.33:0/2139131357 shutdown_connections 2024-09-16T00:32:56.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:56.323+0000 7ffa51f7b640 1 -- 172.21.15.33:0/2139131357 wait complete. 2024-09-16T00:32:57.072 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 0.00 MB/sec execute 3 sec latency 16725.844 ms 2024-09-16T00:32:57.199 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:57 smithi033 ceph-mon[31243]: pgmap v824: 129 pgs: 129 active+clean; 515 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 9.1 MiB/s wr, 329 op/s 2024-09-16T00:32:57.199 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:57 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2139131357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:32:57.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:57 smithi083 ceph-mon[35242]: pgmap v824: 129 pgs: 129 active+clean; 515 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 9.1 MiB/s wr, 329 op/s 2024-09-16T00:32:57.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:57 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2139131357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:32:57.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:57 smithi060 ceph-mon[35185]: pgmap v824: 129 pgs: 129 active+clean; 515 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 9.1 MiB/s wr, 329 op/s 2024-09-16T00:32:57.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:57 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2139131357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:32:57.647 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 113b4b23 --release --await' 2024-09-16T00:32:57.647 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 113b4b23 --release --await 2024-09-16T00:32:57.904 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:32:58.072 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44013 0.00 MB/sec execute 4 sec latency 17725.905 ms 2024-09-16T00:32:58.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.386+0000 7f45e36c7640 1 -- 172.21.15.33:0/2865535519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45dc101de0 msgr2=0x7f45dc1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:58.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.386+0000 7f45e36c7640 1 --2- 172.21.15.33:0/2865535519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45dc101de0 0x7f45dc1021c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009900 tx=0x7f45d802a410 comp rx=0 tx=0).stop 2024-09-16T00:32:58.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.386+0000 7f45e36c7640 1 -- 172.21.15.33:0/2865535519 shutdown_connections 2024-09-16T00:32:58.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.386+0000 7f45e36c7640 1 --2- 172.21.15.33:0/2865535519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc106f10 0x7f45dc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.386+0000 7f45e36c7640 1 --2- 172.21.15.33:0/2865535519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45dc102790 0x7f45dc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.386+0000 7f45e36c7640 1 --2- 172.21.15.33:0/2865535519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45dc101de0 0x7f45dc1021c0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.386+0000 7f45e36c7640 1 -- 172.21.15.33:0/2865535519 >> 172.21.15.33:0/2865535519 conn(0x7f45dc0fd8c0 msgr2=0x7f45dc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:58.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.386+0000 7f45e36c7640 1 -- 172.21.15.33:0/2865535519 shutdown_connections 2024-09-16T00:32:58.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.386+0000 7f45e36c7640 1 -- 172.21.15.33:0/2865535519 wait complete. 2024-09-16T00:32:58.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.386+0000 7f45e36c7640 1 Processor -- start 2024-09-16T00:32:58.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.386+0000 7f45e36c7640 1 -- start start 2024-09-16T00:32:58.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e36c7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc101de0 0x7f45dc198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e36c7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45dc102790 0x7f45dc1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e36c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45dc106f10 0x7f45dc19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e36c7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc111f70 con 0x7f45dc102790 2024-09-16T00:32:58.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e36c7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc111df0 con 0x7f45dc101de0 2024-09-16T00:32:58.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e36c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1120f0 con 0x7f45dc106f10 2024-09-16T00:32:58.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e0c62640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45dc102790 0x7f45dc1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e0c62640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45dc102790 0x7f45dc1989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35744/0 (socket says 172.21.15.33:35744) 2024-09-16T00:32:58.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e0c62640 1 -- 172.21.15.33:0/3926654085 learned_addr learned my addr 172.21.15.33:0/3926654085 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:58.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e1463640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc101de0 0x7f45dc198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e1c64640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45dc106f10 0x7f45dc19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e0c62640 1 -- 172.21.15.33:0/3926654085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45dc106f10 msgr2=0x7f45dc19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:58.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e0c62640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45dc106f10 0x7f45dc19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e0c62640 1 -- 172.21.15.33:0/3926654085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc101de0 msgr2=0x7f45dc198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e0c62640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc101de0 0x7f45dc198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e0c62640 1 -- 172.21.15.33:0/3926654085 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45dc19d440 con 0x7f45dc102790 2024-09-16T00:32:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e1c64640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45dc106f10 0x7f45dc19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:32:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45e0c62640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45dc102790 0x7f45dc1989d0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00d770 tx=0x7f45cc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.387+0000 7f45d27fc640 1 -- 172.21.15.33:0/3926654085 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc017070 con 0x7f45dc102790 2024-09-16T00:32:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.388+0000 7f45d27fc640 1 -- 172.21.15.33:0/3926654085 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45cc005500 con 0x7f45dc102790 2024-09-16T00:32:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.388+0000 7f45d27fc640 1 -- 172.21.15.33:0/3926654085 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc004e00 con 0x7f45dc102790 2024-09-16T00:32:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.388+0000 7f45e36c7640 1 -- 172.21.15.33:0/3926654085 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45dc19d730 con 0x7f45dc102790 2024-09-16T00:32:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.388+0000 7f45e36c7640 1 -- 172.21.15.33:0/3926654085 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45dc1a5010 con 0x7f45dc102790 2024-09-16T00:32:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.390+0000 7f45d27fc640 1 -- 172.21.15.33:0/3926654085 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f45cc004fa0 con 0x7f45dc102790 2024-09-16T00:32:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.390+0000 7f45e5aa0740 1 Processor -- start 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.390+0000 7f45e5aa0740 1 -- start start 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e5aa0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9269f60 0x55e8b926a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e5aa0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b926a880 0x55e8b9279110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e5aa0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b9279650 0x55e8b927ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e5aa0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e8b91cf620 con 0x55e8b926a880 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e5aa0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e8b9176e50 con 0x55e8b9279650 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e5aa0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e8b91cb5e0 con 0x55e8b9269f60 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45d27fc640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f45c80a3240 0x7f45c80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1463640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b926a880 0x55e8b9279110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1463640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b926a880 0x55e8b9279110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35748/0 (socket says 172.21.15.33:35748) 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1463640 1 -- 172.21.15.33:0/1163585305 learned_addr learned my addr 172.21.15.33:0/1163585305 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e0c62640 1 --2- 172.21.15.33:0/1163585305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b9279650 0x55e8b927ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1c64640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f45c80a3240 0x7f45c80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1c64640 1 --2- 172.21.15.33:0/1163585305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9269f60 0x55e8b926a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45d27fc640 1 -- 172.21.15.33:0/3926654085 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f45cc0cf870 con 0x7f45dc102790 2024-09-16T00:32:58.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1463640 1 -- 172.21.15.33:0/1163585305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9269f60 msgr2=0x55e8b926a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:58.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1463640 1 --2- 172.21.15.33:0/1163585305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9269f60 0x55e8b926a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1463640 1 -- 172.21.15.33:0/1163585305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b9279650 msgr2=0x55e8b927ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:58.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1463640 1 --2- 172.21.15.33:0/1163585305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b9279650 0x55e8b927ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1463640 1 -- 172.21.15.33:0/1163585305 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8b9167d40 con 0x55e8b926a880 2024-09-16T00:32:58.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1c64640 1 --2- 172.21.15.33:0/1163585305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9269f60 0x55e8b926a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:32:58.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e0c62640 1 --2- 172.21.15.33:0/1163585305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b9279650 0x55e8b927ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:32:58.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1463640 1 --2- 172.21.15.33:0/1163585305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b926a880 0x55e8b9279110 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009900 tx=0x7f45d8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded3f46e5f5ca87f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:58.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.391+0000 7f45e1c64640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f45c80a3240 0x7f45c80a5700 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f45d4005c40 tx=0x7f45d40094a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:58.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.392+0000 7f45c77fe640 1 -- 172.21.15.33:0/1163585305 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8040070 con 0x55e8b926a880 2024-09-16T00:32:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.392+0000 7f45c77fe640 1 -- 172.21.15.33:0/1163585305 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45d80319f0 con 0x55e8b926a880 2024-09-16T00:32:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.392+0000 7f45c77fe640 1 -- 172.21.15.33:0/1163585305 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8035040 con 0x55e8b926a880 2024-09-16T00:32:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.392+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1163585305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b926a880 msgr2=0x55e8b9279110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.392+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/1163585305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b926a880 0x55e8b9279110 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009900 tx=0x7f45d8005e80 comp rx=0 tx=0).stop 2024-09-16T00:32:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.392+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1163585305 shutdown_connections 2024-09-16T00:32:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.392+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/1163585305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b9279650 0x55e8b927ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.392+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/1163585305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b926a880 0x55e8b9279110 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.392+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/1163585305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9269f60 0x55e8b926a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.392+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1163585305 >> 172.21.15.33:0/1163585305 conn(0x55e8b92665f0 msgr2=0x55e8b92669d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.392+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1163585305 shutdown_connections 2024-09-16T00:32:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.393+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1163585305 wait complete. 2024-09-16T00:32:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.393+0000 7f45e5aa0740 1 -- start start 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.393+0000 7f45e5aa0740 1 Processor -- start 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.393+0000 7f45e5aa0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b9269f60 0x55e8b92765c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.393+0000 7f45e5aa0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b926a880 0x55e8b9276b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.393+0000 7f45e5aa0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 0x55e8b9277040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.393+0000 7f45e5aa0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e8b927c220 con 0x55e8b9279650 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.393+0000 7f45e5aa0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e8b9157f10 con 0x55e8b9269f60 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.393+0000 7f45e5aa0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e8b927c3a0 con 0x55e8b926a880 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.393+0000 7f45e5aa0740 10 client.0 _ll_register_callbacks cb 0x55e8b917bdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e1c64640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 0x55e8b9277040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e1c64640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 0x55e8b9277040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35764/0 (socket says 172.21.15.33:35764) 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e1c64640 1 -- 172.21.15.33:0/2188532894 learned_addr learned my addr 172.21.15.33:0/2188532894 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e0c62640 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b926a880 0x55e8b9276b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e1463640 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b9269f60 0x55e8b92765c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e1c64640 1 -- 172.21.15.33:0/2188532894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b926a880 msgr2=0x55e8b9276b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e1c64640 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b926a880 0x55e8b9276b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e1c64640 1 -- 172.21.15.33:0/2188532894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b9269f60 msgr2=0x55e8b92765c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e1c64640 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b9269f60 0x55e8b92765c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e1c64640 1 -- 172.21.15.33:0/2188532894 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8b8ede580 con 0x55e8b9279650 2024-09-16T00:32:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e1c64640 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 0x55e8b9277040 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f45d4014fc0 tx=0x7f45d4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d787441c84fd15f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:32:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45c4ff9640 1 -- 172.21.15.33:0/2188532894 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d4024030 con 0x55e8b9279650 2024-09-16T00:32:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45c4ff9640 1 -- 172.21.15.33:0/2188532894 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f45d4012970 con 0x55e8b9279650 2024-09-16T00:32:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e5aa0740 1 -- 172.21.15.33:0/2188532894 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45d80319f0 con 0x55e8b9279650 2024-09-16T00:32:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45c4ff9640 1 -- 172.21.15.33:0/2188532894 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f45d4031050 con 0x55e8b9279650 2024-09-16T00:32:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e5aa0740 10 client.18357 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.394+0000 7f45e5aa0740 1 -- 172.21.15.33:0/2188532894 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e8b8fa8960 con 0x55e8b9279650 2024-09-16T00:32:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.395+0000 7f45c4ff9640 1 -- 172.21.15.33:0/2188532894 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45d80319f0 con 0x55e8b9279650 2024-09-16T00:32:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.395+0000 7f45c4ff9640 1 -- 172.21.15.33:0/2188532894 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f45d401d050 con 0x55e8b9279650 2024-09-16T00:32:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.395+0000 7f45e5aa0740 10 client.18357 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.395+0000 7f45e5aa0740 10 client.18357 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:32:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.395+0000 7f45e5aa0740 20 client.18357 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.395+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b9289330 0x55e8b928b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.395+0000 7f45e5aa0740 4 client.18357 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:32:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.395+0000 7f45e5aa0740 1 -- 172.21.15.33:0/2188532894 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8b91dcb90 con 0x55e8b9289330 2024-09-16T00:32:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.395+0000 7f45e1463640 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b9289330 0x55e8b928b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.395+0000 7f45c4ff9640 10 client.18357 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.395+0000 7f45e1463640 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b9289330 0x55e8b928b7d0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a33599959b3dc96 server_cookie=89c5dbc612aa3219 in_seq=0 out_seq=0 2024-09-16T00:32:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.396+0000 7f45c4ff9640 1 -- 172.21.15.33:0/2188532894 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e8b91dcb90 con 0x55e8b9289330 2024-09-16T00:32:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.396+0000 7f45c4ff9640 10 client.18357 handle_command_reply: tid=0 2024-09-16T00:32:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.396+0000 7f45e5aa0740 1 client.18357 shutdown 2024-09-16T00:32:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.397+0000 7f45e5aa0740 1 -- 172.21.15.33:0/2188532894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 msgr2=0x55e8b9277040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.397+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 0x55e8b9277040 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f45d4014fc0 tx=0x7f45d4004ab0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.397+0000 7f45e5aa0740 1 -- 172.21.15.33:0/2188532894 shutdown_connections 2024-09-16T00:32:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.397+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b9289330 0x55e8b928b7d0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.397+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 0x55e8b9277040 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.397+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b926a880 0x55e8b9276b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.397+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/2188532894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b9269f60 0x55e8b92765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.397+0000 7f45c4ff9640 0 client.18357 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:58.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.397+0000 7f45e5aa0740 1 -- 172.21.15.33:0/2188532894 >> 172.21.15.33:0/2188532894 conn(0x55e8b92665f0 msgr2=0x55e8b926c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:58.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.397+0000 7f45e5aa0740 1 -- 172.21.15.33:0/2188532894 shutdown_connections 2024-09-16T00:32:58.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.397+0000 7f45e5aa0740 1 -- 172.21.15.33:0/2188532894 wait complete. 2024-09-16T00:32:58.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.397+0000 7f45e5aa0740 20 client.18357 trim_cache size 0 max 16384 2024-09-16T00:32:58.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.414+0000 7f45e5aa0740 1 Processor -- start 2024-09-16T00:32:58.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.414+0000 7f45e5aa0740 1 -- start start 2024-09-16T00:32:58.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.414+0000 7f45e5aa0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9269f60 0x55e8b9278530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.414+0000 7f45e5aa0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b926a880 0x55e8b92910f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.414+0000 7f45e5aa0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 0x55e8b9291630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.414+0000 7f45e5aa0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1af750 con 0x55e8b9279650 2024-09-16T00:32:58.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.414+0000 7f45e5aa0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1acfc0 con 0x55e8b926a880 2024-09-16T00:32:58.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.414+0000 7f45e5aa0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1aae10 con 0x55e8b9269f60 2024-09-16T00:32:58.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45e1c64640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 0x55e8b9291630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45e1c64640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 0x55e8b9291630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35772/0 (socket says 172.21.15.33:35772) 2024-09-16T00:32:58.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45e1c64640 1 -- 172.21.15.33:0/3423877923 learned_addr learned my addr 172.21.15.33:0/3423877923 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:58.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45e0c62640 1 --2- 172.21.15.33:0/3423877923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b926a880 0x55e8b92910f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45e1463640 1 --2- 172.21.15.33:0/3423877923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9269f60 0x55e8b9278530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45e1c64640 1 -- 172.21.15.33:0/3423877923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9269f60 msgr2=0x55e8b9278530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:58.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45e1c64640 1 --2- 172.21.15.33:0/3423877923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9269f60 0x55e8b9278530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45e1c64640 1 -- 172.21.15.33:0/3423877923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b926a880 msgr2=0x55e8b92910f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:58.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45e1c64640 1 --2- 172.21.15.33:0/3423877923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b926a880 0x55e8b92910f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45e1c64640 1 -- 172.21.15.33:0/3423877923 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8b9275570 con 0x55e8b9279650 2024-09-16T00:32:58.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45e1c64640 1 --2- 172.21.15.33:0/3423877923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 0x55e8b9291630 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f45d4004a40 tx=0x7f45d4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da641efa5ce4609 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:58.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45c6ffd640 1 -- 172.21.15.33:0/3423877923 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d402e070 con 0x55e8b9279650 2024-09-16T00:32:58.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45c6ffd640 1 -- 172.21.15.33:0/3423877923 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45d4014530 con 0x55e8b9279650 2024-09-16T00:32:58.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.415+0000 7f45c6ffd640 1 -- 172.21.15.33:0/3423877923 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d4024070 con 0x55e8b9279650 2024-09-16T00:32:58.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.416+0000 7f45e5aa0740 1 -- 172.21.15.33:0/3423877923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 msgr2=0x55e8b9291630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.416+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/3423877923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 0x55e8b9291630 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f45d4004a40 tx=0x7f45d4005fd0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.416+0000 7f45e5aa0740 1 -- 172.21.15.33:0/3423877923 shutdown_connections 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.416+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/3423877923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9279650 0x55e8b9291630 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.416+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/3423877923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b926a880 0x55e8b92910f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.416+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/3423877923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9269f60 0x55e8b9278530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.416+0000 7f45e5aa0740 1 -- 172.21.15.33:0/3423877923 >> 172.21.15.33:0/3423877923 conn(0x55e8b92665f0 msgr2=0x55e8b9277e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.416+0000 7f45e5aa0740 1 -- 172.21.15.33:0/3423877923 shutdown_connections 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.416+0000 7f45e5aa0740 1 -- 172.21.15.33:0/3423877923 wait complete. 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.416+0000 7f45e5aa0740 1 -- start start 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.416+0000 7f45e5aa0740 1 Processor -- start 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e5aa0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9269f60 0x55e8b9281b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e5aa0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b926a880 0x55e8b927de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e5aa0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9279650 0x55e8b927e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e5aa0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1ab480 con 0x55e8b9269f60 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e5aa0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1ac8f0 con 0x55e8b926a880 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e5aa0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1abb50 con 0x55e8b9279650 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e5aa0740 10 client.0 _ll_register_callbacks cb 0x7f45d4009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e1463640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9269f60 0x55e8b9281b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e1463640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9269f60 0x55e8b9281b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35778/0 (socket says 172.21.15.33:35778) 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e1463640 1 -- 172.21.15.33:0/1403736033 learned_addr learned my addr 172.21.15.33:0/1403736033 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e1c64640 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9279650 0x55e8b927e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e0c62640 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b926a880 0x55e8b927de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e1463640 1 -- 172.21.15.33:0/1403736033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9279650 msgr2=0x55e8b927e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e1463640 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9279650 0x55e8b927e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e1463640 1 -- 172.21.15.33:0/1403736033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b926a880 msgr2=0x55e8b927de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:32:58.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e1463640 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b926a880 0x55e8b927de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.417+0000 7f45e1463640 1 -- 172.21.15.33:0/1403736033 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8b926a6b0 con 0x55e8b9269f60 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45e1463640 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9269f60 0x55e8b9281b20 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009900 tx=0x7f45d802ab30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9df32343138d0a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45c4ff9640 1 -- 172.21.15.33:0/1403736033 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8040050 con 0x55e8b9269f60 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45c4ff9640 1 -- 172.21.15.33:0/1403736033 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f45d8004450 con 0x55e8b9269f60 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1403736033 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45d4014530 con 0x55e8b9269f60 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45c4ff9640 1 -- 172.21.15.33:0/1403736033 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f45d805a050 con 0x55e8b9269f60 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45e5aa0740 10 client.18363 fetch_fsmap learned FSMap version 31 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1403736033 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e8b92f4cd0 con 0x55e8b9269f60 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45c4ff9640 1 -- 172.21.15.33:0/1403736033 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45d4014530 con 0x55e8b9269f60 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45c4ff9640 1 -- 172.21.15.33:0/1403736033 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f45d40140f0 con 0x55e8b9269f60 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45e5aa0740 10 client.18363 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:32:58.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45e5aa0740 10 client.18363 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:32:58.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45e5aa0740 20 client.18363 populate_metadata read hostname 'smithi033' 2024-09-16T00:32:58.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.418+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b92f50d0 0x55e8b92f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:32:58.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.419+0000 7f45e5aa0740 4 client.18363 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "113b4b23", "release": true, "await": true}] 2024-09-16T00:32:58.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.419+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1403736033 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "113b4b23", "release": true, "await": true}) -- 0x55e8b91d70a0 con 0x55e8b92f50d0 2024-09-16T00:32:58.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.419+0000 7f45e0c62640 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b92f50d0 0x55e8b92f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:32:58.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.419+0000 7f45c4ff9640 10 client.18363 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:58.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.419+0000 7f45e0c62640 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b92f50d0 0x55e8b92f74c0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f98c9a261589d318 server_cookie=cd87478aa2b9f0d8 in_seq=0 out_seq=0 2024-09-16T00:32:58.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45c4ff9640 1 -- 172.21.15.33:0/1403736033 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e8b91d70a0 con 0x55e8b92f50d0 2024-09-16T00:32:58.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45c4ff9640 10 client.18363 handle_command_reply: tid=0 2024-09-16T00:32:58.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45e5aa0740 1 client.18363 shutdown 2024-09-16T00:32:58.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1403736033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9269f60 msgr2=0x55e8b9281b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:58.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9269f60 0x55e8b9281b20 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009900 tx=0x7f45d802ab30 comp rx=0 tx=0).stop 2024-09-16T00:32:58.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1403736033 shutdown_connections 2024-09-16T00:32:58.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e8b92f50d0 0x55e8b92f74c0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e8b9279650 0x55e8b927e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e8b926a880 0x55e8b927de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45e5aa0740 1 --2- 172.21.15.33:0/1403736033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e8b9269f60 0x55e8b9281b20 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1403736033 >> 172.21.15.33:0/1403736033 conn(0x55e8b92665f0 msgr2=0x55e8b9265750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:58.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45c4ff9640 0 client.18363 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:32:58.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1403736033 shutdown_connections 2024-09-16T00:32:58.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45e5aa0740 1 -- 172.21.15.33:0/1403736033 wait complete. 2024-09-16T00:32:58.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.487+0000 7f45e5aa0740 20 client.18363 trim_cache size 0 max 16384 2024-09-16T00:32:58.489 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:32:58.489 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 31, 2024-09-16T00:32:58.489 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:32:58.489 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 56, 2024-09-16T00:32:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:32:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "113b4b23": { 2024-09-16T00:32:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 56, 2024-09-16T00:32:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:32:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:32:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:32:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:32:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:32:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:32:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 137.7, 2024-09-16T00:32:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:32:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:32:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:32:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:32:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:32:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:32:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:32:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:32:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:32:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:32:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:32:58.491 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:32:58.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.488+0000 7f45e36c7640 1 -- 172.21.15.33:0/3926654085 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f45c80a3240 msgr2=0x7f45c80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:58.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.488+0000 7f45e36c7640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f45c80a3240 0x7f45c80a5700 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f45d4005c40 tx=0x7f45d40094a0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.488+0000 7f45e36c7640 1 -- 172.21.15.33:0/3926654085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45dc102790 msgr2=0x7f45dc1989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:32:58.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.488+0000 7f45e36c7640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45dc102790 0x7f45dc1989d0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00d770 tx=0x7f45cc00dc30 comp rx=0 tx=0).stop 2024-09-16T00:32:58.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.488+0000 7f45e36c7640 1 -- 172.21.15.33:0/3926654085 shutdown_connections 2024-09-16T00:32:58.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.488+0000 7f45e36c7640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f45c80a3240 0x7f45c80a5700 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.488+0000 7f45e36c7640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45dc106f10 0x7f45dc19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.488+0000 7f45e36c7640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45dc102790 0x7f45dc1989d0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.488+0000 7f45e36c7640 1 --2- 172.21.15.33:0/3926654085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc101de0 0x7f45dc198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:32:58.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.488+0000 7f45e36c7640 1 -- 172.21.15.33:0/3926654085 >> 172.21.15.33:0/3926654085 conn(0x7f45dc0fd8c0 msgr2=0x7f45dc0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:32:58.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.488+0000 7f45e36c7640 1 -- 172.21.15.33:0/3926654085 shutdown_connections 2024-09-16T00:32:58.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:32:58.488+0000 7f45e36c7640 1 -- 172.21.15.33:0/3926654085 wait complete. 2024-09-16T00:32:59.071 INFO:tasks.workunit.client.0.smithi033.stdout: 10 44452 4.71 MB/sec execute 5 sec latency 18158.570 ms 2024-09-16T00:32:59.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:32:59 smithi033 ceph-mon[31243]: pgmap v825: 129 pgs: 129 active+clean; 510 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 9.1 MiB/s wr, 301 op/s 2024-09-16T00:32:59.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:32:59 smithi083 ceph-mon[35242]: pgmap v825: 129 pgs: 129 active+clean; 510 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 9.1 MiB/s wr, 301 op/s 2024-09-16T00:32:59.426 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '113b4b23', total seconds elapsed: 21.9 2024-09-16T00:32:59.426 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.7 2024-09-16T00:32:59.427 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:32:59.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:32:59 smithi060 ceph-mon[35185]: pgmap v825: 129 pgs: 129 active+clean; 510 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 9.1 MiB/s wr, 301 op/s 2024-09-16T00:32:59.751 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:00.072 INFO:tasks.workunit.client.0.smithi033.stdout: 10 45102 10.55 MB/sec execute 6 sec latency 42.008 ms 2024-09-16T00:33:00.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.285+0000 7f8019f4e640 1 -- 172.21.15.33:0/3943253989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f801410e420 msgr2=0x7f8014110810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:00.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.285+0000 7f8019f4e640 1 --2- 172.21.15.33:0/3943253989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f801410e420 0x7f8014110810 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f800c009ca0 tx=0x7f800c02a950 comp rx=0 tx=0).stop 2024-09-16T00:33:00.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.285+0000 7f8019f4e640 1 -- 172.21.15.33:0/3943253989 shutdown_connections 2024-09-16T00:33:00.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.285+0000 7f8019f4e640 1 --2- 172.21.15.33:0/3943253989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f801410e420 0x7f8014110810 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:00.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.285+0000 7f8019f4e640 1 --2- 172.21.15.33:0/3943253989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8014074ee0 0x7f801410dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:00.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.285+0000 7f8019f4e640 1 --2- 172.21.15.33:0/3943253989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80140745c0 0x7f80140749a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:00.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.285+0000 7f8019f4e640 1 -- 172.21.15.33:0/3943253989 >> 172.21.15.33:0/3943253989 conn(0x7f80140fd8e0 msgr2=0x7f80140ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:00.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.286+0000 7f8019f4e640 1 -- 172.21.15.33:0/3943253989 shutdown_connections 2024-09-16T00:33:00.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.286+0000 7f8019f4e640 1 -- 172.21.15.33:0/3943253989 wait complete. 2024-09-16T00:33:00.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.286+0000 7f8019f4e640 1 Processor -- start 2024-09-16T00:33:00.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8019f4e640 1 -- start start 2024-09-16T00:33:00.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8019f4e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80140745c0 0x7f801419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:00.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8019f4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8014074ee0 0x7f801419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:00.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8019f4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f801410e420 0x7f801419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:00.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8019f4e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8014114160 con 0x7f8014074ee0 2024-09-16T00:33:00.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8019f4e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8014113fe0 con 0x7f80140745c0 2024-09-16T00:33:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8019f4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f80141142e0 con 0x7f801410e420 2024-09-16T00:33:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8013fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f801410e420 0x7f801419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8013fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f801410e420 0x7f801419eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36012/0 (socket says 172.21.15.33:36012) 2024-09-16T00:33:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8013fff640 1 -- 172.21.15.33:0/1119086325 learned_addr learned my addr 172.21.15.33:0/1119086325 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f80137fe640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80140745c0 0x7f801419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8012ffd640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8014074ee0 0x7f801419ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8013fff640 1 -- 172.21.15.33:0/1119086325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80140745c0 msgr2=0x7f801419a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:00.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8013fff640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80140745c0 0x7f801419a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8013fff640 1 -- 172.21.15.33:0/1119086325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8014074ee0 msgr2=0x7f801419ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8013fff640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8014074ee0 0x7f801419ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.287+0000 7f8013fff640 1 -- 172.21.15.33:0/1119086325 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f801419f5d0 con 0x7f801410e420 2024-09-16T00:33:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.288+0000 7f8013fff640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f801410e420 0x7f801419eef0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f800c004af0 tx=0x7f800c02f3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.288+0000 7f8010ff9640 1 -- 172.21.15.33:0/1119086325 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800c034040 con 0x7f801410e420 2024-09-16T00:33:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.288+0000 7f8010ff9640 1 -- 172.21.15.33:0/1119086325 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f800c030920 con 0x7f801410e420 2024-09-16T00:33:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.288+0000 7f8019f4e640 1 -- 172.21.15.33:0/1119086325 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f801419f860 con 0x7f801410e420 2024-09-16T00:33:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.288+0000 7f8010ff9640 1 -- 172.21.15.33:0/1119086325 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800c0046a0 con 0x7f801410e420 2024-09-16T00:33:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.288+0000 7f8019f4e640 1 -- 172.21.15.33:0/1119086325 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80141a71a0 con 0x7f801410e420 2024-09-16T00:33:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.289+0000 7f8019f4e640 1 -- 172.21.15.33:0/1119086325 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80140749a0 con 0x7f801410e420 2024-09-16T00:33:00.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.291+0000 7f8010ff9640 1 -- 172.21.15.33:0/1119086325 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f800c037020 con 0x7f801410e420 2024-09-16T00:33:00.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.292+0000 7f8010ff9640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ffc0a32c0 0x7f7ffc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:00.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.292+0000 7f8010ff9640 1 -- 172.21.15.33:0/1119086325 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f800c0f0c30 con 0x7f801410e420 2024-09-16T00:33:00.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.292+0000 7f80137fe640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ffc0a32c0 0x7f7ffc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:00.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.292+0000 7f80137fe640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ffc0a32c0 0x7f7ffc0a5780 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8004009b60 tx=0x7f8004005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:00.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.299+0000 7f8010ff9640 1 -- 172.21.15.33:0/1119086325 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f800c0bd560 con 0x7f801410e420 2024-09-16T00:33:00.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.594+0000 7f8019f4e640 1 -- 172.21.15.33:0/1119086325 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f801419b980 con 0x7f801410e420 2024-09-16T00:33:00.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.595+0000 7f8010ff9640 1 -- 172.21.15.33:0/1119086325 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11906 (secure 0 0 0) 0x7f801419b980 con 0x7f801410e420 2024-09-16T00:33:00.597 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:33:00.597 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":31,"btime":"2024-09-16T00:30:57:641642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:30:57.027530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:33:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 31 2024-09-16T00:33:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.599+0000 7f8019f4e640 1 -- 172.21.15.33:0/1119086325 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ffc0a32c0 msgr2=0x7f7ffc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.599+0000 7f8019f4e640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ffc0a32c0 0x7f7ffc0a5780 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8004009b60 tx=0x7f8004005f90 comp rx=0 tx=0).stop 2024-09-16T00:33:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.599+0000 7f8019f4e640 1 -- 172.21.15.33:0/1119086325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f801410e420 msgr2=0x7f801419eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.599+0000 7f8019f4e640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f801410e420 0x7f801419eef0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f800c004af0 tx=0x7f800c02f3a0 comp rx=0 tx=0).stop 2024-09-16T00:33:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.599+0000 7f8019f4e640 1 -- 172.21.15.33:0/1119086325 shutdown_connections 2024-09-16T00:33:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.599+0000 7f8019f4e640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7ffc0a32c0 0x7f7ffc0a5780 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.599+0000 7f8019f4e640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f801410e420 0x7f801419eef0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.599+0000 7f8019f4e640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8014074ee0 0x7f801419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.599+0000 7f8019f4e640 1 --2- 172.21.15.33:0/1119086325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80140745c0 0x7f801419a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.599+0000 7f8019f4e640 1 -- 172.21.15.33:0/1119086325 >> 172.21.15.33:0/1119086325 conn(0x7f80140fd8e0 msgr2=0x7f80140fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.599+0000 7f8019f4e640 1 -- 172.21.15.33:0/1119086325 shutdown_connections 2024-09-16T00:33:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:00.599+0000 7f8019f4e640 1 -- 172.21.15.33:0/1119086325 wait complete. 2024-09-16T00:33:01.072 INFO:tasks.workunit.client.0.smithi033.stdout: 10 45692 14.09 MB/sec execute 7 sec latency 66.809 ms 2024-09-16T00:33:01.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:01 smithi033 ceph-mon[31243]: pgmap v826: 129 pgs: 129 active+clean; 501 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 9.1 MiB/s wr, 316 op/s 2024-09-16T00:33:01.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:01 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1119086325' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:33:01.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:01 smithi083 ceph-mon[35242]: pgmap v826: 129 pgs: 129 active+clean; 501 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 9.1 MiB/s wr, 316 op/s 2024-09-16T00:33:01.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:01 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1119086325' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:33:01.445 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0' 2024-09-16T00:33:01.445 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0 2024-09-16T00:33:01.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:01 smithi060 ceph-mon[35185]: pgmap v826: 129 pgs: 129 active+clean; 501 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 9.1 MiB/s wr, 316 op/s 2024-09-16T00:33:01.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:01 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1119086325' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:33:01.787 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:02.072 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46224 15.58 MB/sec execute 8 sec latency 97.304 ms 2024-09-16T00:33:02.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.312+0000 7fd3553bb640 1 -- 172.21.15.33:0/4147149803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd350102790 msgr2=0x7fd3501067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:02.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.312+0000 7fd3553bb640 1 --2- 172.21.15.33:0/4147149803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd350102790 0x7fd3501067e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd338009860 tx=0x7fd33802a340 comp rx=0 tx=0).stop 2024-09-16T00:33:02.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.313+0000 7fd3553bb640 1 -- 172.21.15.33:0/4147149803 shutdown_connections 2024-09-16T00:33:02.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.313+0000 7fd3553bb640 1 --2- 172.21.15.33:0/4147149803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd350106f10 0x7fd35010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.313+0000 7fd3553bb640 1 --2- 172.21.15.33:0/4147149803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd350102790 0x7fd3501067e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.313+0000 7fd3553bb640 1 --2- 172.21.15.33:0/4147149803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd350101de0 0x7fd3501021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.313+0000 7fd3553bb640 1 -- 172.21.15.33:0/4147149803 >> 172.21.15.33:0/4147149803 conn(0x7fd3500fd8c0 msgr2=0x7fd3500ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:02.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.313+0000 7fd3553bb640 1 -- 172.21.15.33:0/4147149803 shutdown_connections 2024-09-16T00:33:02.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.313+0000 7fd3553bb640 1 -- 172.21.15.33:0/4147149803 wait complete. 2024-09-16T00:33:02.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.313+0000 7fd3553bb640 1 Processor -- start 2024-09-16T00:33:02.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.313+0000 7fd3553bb640 1 -- start start 2024-09-16T00:33:02.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.314+0000 7fd3553bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd350101de0 0x7fd350198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.314+0000 7fd3553bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd350102790 0x7fd3501989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.314+0000 7fd3553bb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd350106f10 0x7fd35019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.314+0000 7fd3553bb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd350111f30 con 0x7fd350106f10 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.314+0000 7fd3553bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd350111db0 con 0x7fd350101de0 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.314+0000 7fd3553bb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501120b0 con 0x7fd350102790 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.314+0000 7fd347fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd350102790 0x7fd3501989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.314+0000 7fd347fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd350102790 0x7fd3501989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36044/0 (socket says 172.21.15.33:36044) 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.314+0000 7fd347fff640 1 -- 172.21.15.33:0/90282369 learned_addr learned my addr 172.21.15.33:0/90282369 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.315+0000 7fd347fff640 1 -- 172.21.15.33:0/90282369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd350101de0 msgr2=0x7fd350198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.315+0000 7fd34effd640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd350101de0 0x7fd350198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.315+0000 7fd347fff640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd350101de0 0x7fd350198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.315+0000 7fd347fff640 1 -- 172.21.15.33:0/90282369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd350106f10 msgr2=0x7fd35019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.315+0000 7fd347fff640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd350106f10 0x7fd35019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.315+0000 7fd347fff640 1 -- 172.21.15.33:0/90282369 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35019d420 con 0x7fd350102790 2024-09-16T00:33:02.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.315+0000 7fd34effd640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd350101de0 0x7fd350198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:33:02.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.315+0000 7fd347fff640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd350102790 0x7fd3501989b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd338035040 tx=0x7fd338033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:02.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.315+0000 7fd34cff9640 1 -- 172.21.15.33:0/90282369 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33803f020 con 0x7fd350102790 2024-09-16T00:33:02.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.316+0000 7fd34cff9640 1 -- 172.21.15.33:0/90282369 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd338032770 con 0x7fd350102790 2024-09-16T00:33:02.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.316+0000 7fd3553bb640 1 -- 172.21.15.33:0/90282369 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35019d6b0 con 0x7fd350102790 2024-09-16T00:33:02.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.316+0000 7fd34cff9640 1 -- 172.21.15.33:0/90282369 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd338037040 con 0x7fd350102790 2024-09-16T00:33:02.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.316+0000 7fd3553bb640 1 -- 172.21.15.33:0/90282369 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd35019db40 con 0x7fd350102790 2024-09-16T00:33:02.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.319+0000 7fd34cff9640 1 -- 172.21.15.33:0/90282369 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd338042020 con 0x7fd350102790 2024-09-16T00:33:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.320+0000 7fd357794740 1 Processor -- start 2024-09-16T00:33:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.320+0000 7fd357794740 1 -- start start 2024-09-16T00:33:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.320+0000 7fd34cff9640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd33c0a32c0 0x7fd33c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.320+0000 7fd34cff9640 1 -- 172.21.15.33:0/90282369 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd3380f00b0 con 0x7fd350102790 2024-09-16T00:33:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.320+0000 7fd34f7fe640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd33c0a32c0 0x7fd33c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.320+0000 7fd357794740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5614760 0x5651d5614b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.320+0000 7fd357794740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5615080 0x5651d561f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd357794740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d5621d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd357794740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5651d557a2a0 con 0x5651d5614760 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd357794740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5651d5521b10 con 0x5651d561f940 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd357794740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5651d5576260 con 0x5651d5615080 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd347fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5615080 0x5651d561f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd34effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5614760 0x5651d5614b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd347fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5615080 0x5651d561f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36046/0 (socket says 172.21.15.33:36046) 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd34effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5614760 0x5651d5614b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46204/0 (socket says 172.21.15.33:46204) 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd347fff640 1 -- 172.21.15.33:0/420307500 learned_addr learned my addr 172.21.15.33:0/420307500 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd34f7fe640 1 --2- 172.21.15.33:0/420307500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d5621d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd34f7fe640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd33c0a32c0 0x7fd33c0a5780 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd348005fd0 tx=0x7fd34800e720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd34f7fe640 1 -- 172.21.15.33:0/420307500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5615080 msgr2=0x5651d561f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd34f7fe640 1 --2- 172.21.15.33:0/420307500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5615080 0x5651d561f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd34f7fe640 1 -- 172.21.15.33:0/420307500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5614760 msgr2=0x5651d5614b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:02.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd34f7fe640 1 --2- 172.21.15.33:0/420307500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5614760 0x5651d5614b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd34f7fe640 1 -- 172.21.15.33:0/420307500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651d52ebb90 con 0x5651d561f940 2024-09-16T00:33:02.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.321+0000 7fd34f7fe640 1 --2- 172.21.15.33:0/420307500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d5621d30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd34800ec20 tx=0x7fd348019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55c1cd1575f8a747 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:02.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.322+0000 7fd3457fa640 1 -- 172.21.15.33:0/420307500 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348010420 con 0x5651d561f940 2024-09-16T00:33:02.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.322+0000 7fd3457fa640 1 -- 172.21.15.33:0/420307500 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd348010aa0 con 0x5651d561f940 2024-09-16T00:33:02.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.322+0000 7fd3457fa640 1 -- 172.21.15.33:0/420307500 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348024ca0 con 0x5651d561f940 2024-09-16T00:33:02.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.322+0000 7fd357794740 1 -- 172.21.15.33:0/420307500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 msgr2=0x5651d5621d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:02.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.322+0000 7fd357794740 1 --2- 172.21.15.33:0/420307500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d5621d30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd34800ec20 tx=0x7fd348019000 comp rx=0 tx=0).stop 2024-09-16T00:33:02.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.322+0000 7fd357794740 1 -- 172.21.15.33:0/420307500 shutdown_connections 2024-09-16T00:33:02.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.322+0000 7fd357794740 1 --2- 172.21.15.33:0/420307500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d5621d30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.322+0000 7fd357794740 1 --2- 172.21.15.33:0/420307500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5615080 0x5651d561f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.322+0000 7fd357794740 1 --2- 172.21.15.33:0/420307500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5614760 0x5651d5614b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.322+0000 7fd357794740 1 -- 172.21.15.33:0/420307500 >> 172.21.15.33:0/420307500 conn(0x5651d5610cf0 msgr2=0x5651d56110d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:02.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.322+0000 7fd357794740 1 -- 172.21.15.33:0/420307500 shutdown_connections 2024-09-16T00:33:02.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd357794740 1 -- 172.21.15.33:0/420307500 wait complete. 2024-09-16T00:33:02.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd357794740 1 -- start start 2024-09-16T00:33:02.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd357794740 1 Processor -- start 2024-09-16T00:33:02.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd357794740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 0x5651d561c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd357794740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d561cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd357794740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d561d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd357794740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5651d5622510 con 0x5651d5615080 2024-09-16T00:33:02.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd357794740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5651d55026e0 con 0x5651d561f940 2024-09-16T00:33:02.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd357794740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5651d5622690 con 0x5651d5614760 2024-09-16T00:33:02.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd357794740 10 client.0 _ll_register_callbacks cb 0x5651d5526a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:02.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd347fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d561cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd34effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 0x5651d561c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.323+0000 7fd34effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 0x5651d561c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36060/0 (socket says 172.21.15.33:36060) 2024-09-16T00:33:02.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd34effd640 1 -- 172.21.15.33:0/3108006451 learned_addr learned my addr 172.21.15.33:0/3108006451 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:02.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd34f7fe640 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d561d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd34effd640 1 -- 172.21.15.33:0/3108006451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 msgr2=0x5651d561d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:02.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd34effd640 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d561d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd34effd640 1 -- 172.21.15.33:0/3108006451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 msgr2=0x5651d561cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:02.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd34effd640 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d561cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd34effd640 1 -- 172.21.15.33:0/3108006451 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651d562ba30 con 0x5651d5614760 2024-09-16T00:33:02.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd34f7fe640 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d561d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:02.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd347fff640 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d561cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:02.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd34effd640 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 0x5651d561c850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3400136f0 tx=0x7fd340013bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6adb16187fa4a26f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:02.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd3367fc640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:33:02.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd3367fc640 1 -- 172.21.15.33:0/3108006451 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340011bd0 con 0x5651d5614760 2024-09-16T00:33:02.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd357794740 1 -- 172.21.15.33:0/3108006451 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd348010aa0 con 0x5651d5614760 2024-09-16T00:33:02.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.324+0000 7fd3367fc640 1 -- 172.21.15.33:0/3108006451 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd340011d70 con 0x5651d5614760 2024-09-16T00:33:02.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.325+0000 7fd3367fc640 1 -- 172.21.15.33:0/3108006451 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd34001c440 con 0x5651d5614760 2024-09-16T00:33:02.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.325+0000 7fd357794740 10 client.27706 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:02.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.325+0000 7fd357794740 1 -- 172.21.15.33:0/3108006451 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5651d562e470 con 0x5651d5614760 2024-09-16T00:33:02.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.325+0000 7fd3367fc640 1 -- 172.21.15.33:0/3108006451 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd348010aa0 con 0x5651d5614760 2024-09-16T00:33:02.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.325+0000 7fd3367fc640 1 -- 172.21.15.33:0/3108006451 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd34001a070 con 0x5651d5614760 2024-09-16T00:33:02.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.325+0000 7fd357794740 10 client.27706 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:02.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.325+0000 7fd357794740 10 client.27706 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:33:02.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.325+0000 7fd357794740 20 client.27706 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:02.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.325+0000 7fd357794740 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5651d562f880 0x5651d5631d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.325+0000 7fd357794740 4 client.27706 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:33:02.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.325+0000 7fd357794740 1 -- 172.21.15.33:0/3108006451 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651d55879d0 con 0x5651d562f880 2024-09-16T00:33:02.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.325+0000 7fd347fff640 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5651d562f880 0x5651d5631d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.326+0000 7fd3367fc640 10 client.27706 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:02.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.326+0000 7fd347fff640 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5651d562f880 0x5651d5631d20 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d324fffef7be212 server_cookie=2ea0f54158081c90 in_seq=0 out_seq=0 2024-09-16T00:33:02.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.326+0000 7fd3367fc640 1 -- 172.21.15.33:0/3108006451 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5651d55879d0 con 0x5651d562f880 2024-09-16T00:33:02.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd3367fc640 10 client.27706 handle_command_reply: tid=0 2024-09-16T00:33:02.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd357794740 1 client.27706 shutdown 2024-09-16T00:33:02.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd357794740 1 -- 172.21.15.33:0/3108006451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 msgr2=0x5651d561c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:02.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd357794740 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 0x5651d561c850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3400136f0 tx=0x7fd340013bb0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd357794740 1 -- 172.21.15.33:0/3108006451 shutdown_connections 2024-09-16T00:33:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd357794740 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5651d562f880 0x5651d5631d20 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd357794740 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d561d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd3367fc640 0 client.27706 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd357794740 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d561cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd357794740 1 --2- 172.21.15.33:0/3108006451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 0x5651d561c850 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd357794740 1 -- 172.21.15.33:0/3108006451 >> 172.21.15.33:0/3108006451 conn(0x5651d5610cf0 msgr2=0x5651d5616a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd357794740 1 -- 172.21.15.33:0/3108006451 shutdown_connections 2024-09-16T00:33:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd357794740 1 -- 172.21.15.33:0/3108006451 wait complete. 2024-09-16T00:33:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.327+0000 7fd357794740 20 client.27706 trim_cache size 0 max 16384 2024-09-16T00:33:02.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.355+0000 7fd357794740 1 Processor -- start 2024-09-16T00:33:02.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.355+0000 7fd357794740 1 -- start start 2024-09-16T00:33:02.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd357794740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 0x5651d561b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd357794740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d5637640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd357794740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d5637b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd357794740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501af860 con 0x5651d5615080 2024-09-16T00:33:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd357794740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501ad0d0 con 0x5651d561f940 2024-09-16T00:33:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd357794740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501aaf20 con 0x5651d5614760 2024-09-16T00:33:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd347fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d5637640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd347fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d5637640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46218/0 (socket says 172.21.15.33:46218) 2024-09-16T00:33:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd347fff640 1 -- 172.21.15.33:0/1652865482 learned_addr learned my addr 172.21.15.33:0/1652865482 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd34effd640 1 --2- 172.21.15.33:0/1652865482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 0x5651d561b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd34f7fe640 1 --2- 172.21.15.33:0/1652865482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d5637b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd347fff640 1 -- 172.21.15.33:0/1652865482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 msgr2=0x5651d561b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd347fff640 1 --2- 172.21.15.33:0/1652865482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 0x5651d561b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd347fff640 1 -- 172.21.15.33:0/1652865482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 msgr2=0x5651d5637b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd347fff640 1 --2- 172.21.15.33:0/1652865482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d5637b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd347fff640 1 -- 172.21.15.33:0/1652865482 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651d560c260 con 0x5651d5615080 2024-09-16T00:33:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.356+0000 7fd34effd640 1 --2- 172.21.15.33:0/1652865482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 0x5651d561b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.357+0000 7fd347fff640 1 --2- 172.21.15.33:0/1652865482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d5637640 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fd3380095f0 tx=0x7fd338004240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2bb884ae118026 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.357+0000 7fd344ff9640 1 -- 172.21.15.33:0/1652865482 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3380ba4f0 con 0x5651d5615080 2024-09-16T00:33:02.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.357+0000 7fd344ff9640 1 -- 172.21.15.33:0/1652865482 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3380ed7e0 con 0x5651d5615080 2024-09-16T00:33:02.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.357+0000 7fd344ff9640 1 -- 172.21.15.33:0/1652865482 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3380bc6b0 con 0x5651d5615080 2024-09-16T00:33:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.358+0000 7fd357794740 1 -- 172.21.15.33:0/1652865482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 msgr2=0x5651d5637640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.358+0000 7fd357794740 1 --2- 172.21.15.33:0/1652865482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d5637640 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fd3380095f0 tx=0x7fd338004240 comp rx=0 tx=0).stop 2024-09-16T00:33:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.358+0000 7fd357794740 1 -- 172.21.15.33:0/1652865482 shutdown_connections 2024-09-16T00:33:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.358+0000 7fd357794740 1 --2- 172.21.15.33:0/1652865482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d561f940 0x5651d5637b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.358+0000 7fd357794740 1 --2- 172.21.15.33:0/1652865482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d5637640 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.358+0000 7fd357794740 1 --2- 172.21.15.33:0/1652865482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d5614760 0x5651d561b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.358+0000 7fd357794740 1 -- 172.21.15.33:0/1652865482 >> 172.21.15.33:0/1652865482 conn(0x5651d5610cf0 msgr2=0x5651d561e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.358+0000 7fd357794740 1 -- 172.21.15.33:0/1652865482 shutdown_connections 2024-09-16T00:33:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.358+0000 7fd357794740 1 -- 172.21.15.33:0/1652865482 wait complete. 2024-09-16T00:33:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd357794740 1 -- start start 2024-09-16T00:33:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd357794740 1 Processor -- start 2024-09-16T00:33:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd357794740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d5614760 0x5651d5623a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd357794740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d5623f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd357794740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d561f940 0x5651d5699610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd357794740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501ab590 con 0x5651d5615080 2024-09-16T00:33:02.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd357794740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501aca00 con 0x5651d5614760 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd357794740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501abc60 con 0x5651d561f940 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd357794740 10 client.0 _ll_register_callbacks cb 0x7fd340009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd34effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d5614760 0x5651d5623a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd347fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d5623f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd34effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d5614760 0x5651d5623a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51028/0 (socket says 172.21.15.33:51028) 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd34effd640 1 -- 172.21.15.33:0/1433584829 learned_addr learned my addr 172.21.15.33:0/1433584829 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.359+0000 7fd34f7fe640 1 --2- 172.21.15.33:0/1433584829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d561f940 0x5651d5699610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd34effd640 1 -- 172.21.15.33:0/1433584829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d561f940 msgr2=0x5651d5699610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd34effd640 1 --2- 172.21.15.33:0/1433584829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d561f940 0x5651d5699610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd34effd640 1 -- 172.21.15.33:0/1433584829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 msgr2=0x5651d5623f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd34effd640 1 --2- 172.21.15.33:0/1433584829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d5623f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd34effd640 1 -- 172.21.15.33:0/1433584829 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651d5611980 con 0x5651d5614760 2024-09-16T00:33:02.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd34effd640 1 --2- 172.21.15.33:0/1433584829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d5614760 0x5651d5623a50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd34001b890 tx=0x7fd34001b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1de24378ebeecc48 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd3367fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd3367fc640 1 -- 172.21.15.33:0/1433584829 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34002e070 con 0x5651d5614760 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd3367fc640 1 -- 172.21.15.33:0/1433584829 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd340011de0 con 0x5651d5614760 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd357794740 1 -- 172.21.15.33:0/1433584829 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3380ed7e0 con 0x5651d5614760 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd3367fc640 1 -- 172.21.15.33:0/1433584829 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd34003a050 con 0x5651d5614760 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd357794740 10 client.27716 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.360+0000 7fd357794740 1 -- 172.21.15.33:0/1433584829 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5651d569ab00 con 0x5651d5614760 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.361+0000 7fd3367fc640 1 -- 172.21.15.33:0/1433584829 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3380ed7e0 con 0x5651d5614760 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.361+0000 7fd3367fc640 1 -- 172.21.15.33:0/1433584829 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd34001a050 con 0x5651d5614760 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.361+0000 7fd357794740 10 client.27716 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.361+0000 7fd357794740 10 client.27716 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.361+0000 7fd357794740 20 client.27716 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.361+0000 7fd357794740 1 --2- 172.21.15.33:0/1433584829 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5651d562f880 0x5651d569d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:02.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.361+0000 7fd357794740 4 client.27716 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}] 2024-09-16T00:33:02.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.361+0000 7fd357794740 1 -- 172.21.15.33:0/1433584829 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}) -- 0x5651d5581d20 con 0x5651d562f880 2024-09-16T00:33:02.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.361+0000 7fd347fff640 1 --2- 172.21.15.33:0/1433584829 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5651d562f880 0x5651d569d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:02.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.361+0000 7fd3367fc640 10 client.27716 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:02.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:02.361+0000 7fd347fff640 1 --2- 172.21.15.33:0/1433584829 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5651d562f880 0x5651d569d300 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d76519eeff823984 server_cookie=50fdb463bca6aa95 in_seq=0 out_seq=0 2024-09-16T00:33:03.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd3367fc640 1 -- 172.21.15.33:0/1433584829 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5651d5581d20 con 0x5651d562f880 2024-09-16T00:33:03.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd3367fc640 10 client.27716 handle_command_reply: tid=0 2024-09-16T00:33:03.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd357794740 1 client.27716 shutdown 2024-09-16T00:33:03.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd357794740 1 -- 172.21.15.33:0/1433584829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d5614760 msgr2=0x5651d5623a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:03.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd357794740 1 --2- 172.21.15.33:0/1433584829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d5614760 0x5651d5623a50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd34001b890 tx=0x7fd34001b690 comp rx=0 tx=0).stop 2024-09-16T00:33:03.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd357794740 1 -- 172.21.15.33:0/1433584829 shutdown_connections 2024-09-16T00:33:03.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd357794740 1 --2- 172.21.15.33:0/1433584829 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5651d562f880 0x5651d569d300 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:03.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd357794740 1 --2- 172.21.15.33:0/1433584829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5651d561f940 0x5651d5699610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:03.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd3367fc640 0 client.27716 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:03.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd357794740 1 --2- 172.21.15.33:0/1433584829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5651d5615080 0x5651d5623f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:03.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd357794740 1 --2- 172.21.15.33:0/1433584829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5651d5614760 0x5651d5623a50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:03.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd357794740 1 -- 172.21.15.33:0/1433584829 >> 172.21.15.33:0/1433584829 conn(0x5651d5610cf0 msgr2=0x5651d560f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:03.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd357794740 1 -- 172.21.15.33:0/1433584829 shutdown_connections 2024-09-16T00:33:03.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd357794740 1 -- 172.21.15.33:0/1433584829 wait complete. 2024-09-16T00:33:03.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.039+0000 7fd357794740 20 client.27716 trim_cache size 0 max 16384 2024-09-16T00:33:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:33:03.042 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 31, 2024-09-16T00:33:03.042 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:33:03.042 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 58, 2024-09-16T00:33:03.042 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:33:03.042 INFO:teuthology.orchestra.run.smithi033.stdout: "be63bf18": { 2024-09-16T00:33:03.042 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 58, 2024-09-16T00:33:03.042 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:33:03.042 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:33:03.042 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 137.7, 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:33:03.043 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:33:03.044 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:33:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:33:03.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.040+0000 7fd3553bb640 1 -- 172.21.15.33:0/90282369 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd33c0a32c0 msgr2=0x7fd33c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:03.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.040+0000 7fd3553bb640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd33c0a32c0 0x7fd33c0a5780 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd348005fd0 tx=0x7fd34800e720 comp rx=0 tx=0).stop 2024-09-16T00:33:03.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.040+0000 7fd3553bb640 1 -- 172.21.15.33:0/90282369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd350102790 msgr2=0x7fd3501989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:03.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.040+0000 7fd3553bb640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd350102790 0x7fd3501989b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd338035040 tx=0x7fd338033d90 comp rx=0 tx=0).stop 2024-09-16T00:33:03.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.040+0000 7fd3553bb640 1 -- 172.21.15.33:0/90282369 shutdown_connections 2024-09-16T00:33:03.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.040+0000 7fd3553bb640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd33c0a32c0 0x7fd33c0a5780 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:03.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.040+0000 7fd3553bb640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd350106f10 0x7fd35019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:03.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.040+0000 7fd3553bb640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd350102790 0x7fd3501989b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:03.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.040+0000 7fd3553bb640 1 --2- 172.21.15.33:0/90282369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd350101de0 0x7fd350198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:03.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.040+0000 7fd3553bb640 1 -- 172.21.15.33:0/90282369 >> 172.21.15.33:0/90282369 conn(0x7fd3500fd8c0 msgr2=0x7fd3500fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:03.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.040+0000 7fd3553bb640 1 -- 172.21.15.33:0/90282369 shutdown_connections 2024-09-16T00:33:03.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:03.040+0000 7fd3553bb640 1 -- 172.21.15.33:0/90282369 wait complete. 2024-09-16T00:33:03.072 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 15.75 MB/sec execute 9 sec latency 701.947 ms 2024-09-16T00:33:03.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:03 smithi033 ceph-mon[31243]: pgmap v827: 129 pgs: 129 active+clean; 501 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 257 op/s 2024-09-16T00:33:03.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:03 smithi083 ceph-mon[35242]: pgmap v827: 129 pgs: 129 active+clean; 501 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 257 op/s 2024-09-16T00:33:03.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:03 smithi060 ceph-mon[35185]: pgmap v827: 129 pgs: 129 active+clean; 501 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 257 op/s 2024-09-16T00:33:03.906 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'be63bf18', quiesce took 4.5 seconds. Will release after: 13.2 2024-09-16T00:33:04.072 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 14.18 MB/sec execute 10 sec latency 1701.979 ms 2024-09-16T00:33:05.072 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 12.89 MB/sec execute 11 sec latency 2702.089 ms 2024-09-16T00:33:05.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:05 smithi033 ceph-mon[31243]: pgmap v828: 129 pgs: 129 active+clean; 504 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-09-16T00:33:05.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:05 smithi083 ceph-mon[35242]: pgmap v828: 129 pgs: 129 active+clean; 504 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-09-16T00:33:05.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:05 smithi060 ceph-mon[35185]: pgmap v828: 129 pgs: 129 active+clean; 504 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-09-16T00:33:05.997 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:33:06.072 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 11.81 MB/sec execute 12 sec latency 3702.147 ms 2024-09-16T00:33:06.267 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:06.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.748+0000 7f6853e78640 1 -- 172.21.15.33:0/595011425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f684c102570 msgr2=0x7f684c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:06.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.748+0000 7f6853e78640 1 --2- 172.21.15.33:0/595011425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f684c102570 0x7f684c1029b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f683c009900 tx=0x7f683c02a480 comp rx=0 tx=0).stop 2024-09-16T00:33:06.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.748+0000 7f6853e78640 1 -- 172.21.15.33:0/595011425 shutdown_connections 2024-09-16T00:33:06.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.748+0000 7f6853e78640 1 --2- 172.21.15.33:0/595011425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f684c102ef0 0x7f684c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.748+0000 7f6853e78640 1 --2- 172.21.15.33:0/595011425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f684c102570 0x7f684c1029b0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.748+0000 7f6853e78640 1 --2- 172.21.15.33:0/595011425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f684c100e20 0x7f684c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.748+0000 7f6853e78640 1 -- 172.21.15.33:0/595011425 >> 172.21.15.33:0/595011425 conn(0x7f684c0fc7d0 msgr2=0x7f684c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:06.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.748+0000 7f6853e78640 1 -- 172.21.15.33:0/595011425 shutdown_connections 2024-09-16T00:33:06.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.748+0000 7f6853e78640 1 -- 172.21.15.33:0/595011425 wait complete. 2024-09-16T00:33:06.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6853e78640 1 Processor -- start 2024-09-16T00:33:06.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6853e78640 1 -- start start 2024-09-16T00:33:06.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6853e78640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f684c100e20 0x7f684c1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6853e78640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f684c102570 0x7f684c198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6853e78640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f684c102ef0 0x7f684c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6853e78640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f684c110f50 con 0x7f684c100e20 2024-09-16T00:33:06.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6853e78640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f684c110dd0 con 0x7f684c102ef0 2024-09-16T00:33:06.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6853e78640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f684c1110d0 con 0x7f684c102570 2024-09-16T00:33:06.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6851c14640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f684c100e20 0x7f684c1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6851c14640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f684c100e20 0x7f684c1984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46238/0 (socket says 172.21.15.33:46238) 2024-09-16T00:33:06.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6851c14640 1 -- 172.21.15.33:0/2436147458 learned_addr learned my addr 172.21.15.33:0/2436147458 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:06.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6851413640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f684c102570 0x7f684c198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6852415640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f684c102ef0 0x7f684c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6851c14640 1 -- 172.21.15.33:0/2436147458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f684c102570 msgr2=0x7f684c198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6851c14640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f684c102570 0x7f684c198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6851c14640 1 -- 172.21.15.33:0/2436147458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f684c102ef0 msgr2=0x7f684c19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6851c14640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f684c102ef0 0x7f684c19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.749+0000 7f6851c14640 1 -- 172.21.15.33:0/2436147458 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684c19d490 con 0x7f684c100e20 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.750+0000 7f6851c14640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f684c100e20 0x7f684c1984e0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f684800c890 tx=0x7f684800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.750+0000 7f6842ffd640 1 -- 172.21.15.33:0/2436147458 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6848004220 con 0x7f684c100e20 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.750+0000 7f6842ffd640 1 -- 172.21.15.33:0/2436147458 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68480043c0 con 0x7f684c100e20 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.750+0000 7f6853e78640 1 -- 172.21.15.33:0/2436147458 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f684c19d780 con 0x7f684c100e20 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.750+0000 7f6842ffd640 1 -- 172.21.15.33:0/2436147458 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68480049e0 con 0x7f684c100e20 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.750+0000 7f6853e78640 1 -- 172.21.15.33:0/2436147458 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f684c1a5060 con 0x7f684c100e20 2024-09-16T00:33:06.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.753+0000 7f6842ffd640 1 -- 172.21.15.33:0/2436147458 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f68480079b0 con 0x7f684c100e20 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.753+0000 7f6856251740 1 Processor -- start 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.753+0000 7f6856251740 1 -- start start 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.753+0000 7f6856251740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c34a0 0x55c0f50c3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.753+0000 7f6856251740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50c3dc0 0x55c0f50ce140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.753+0000 7f6856251740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50ce680 0x55c0f50d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.753+0000 7f6856251740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c0f50292e0 con 0x55c0f50ce680 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.753+0000 7f6856251740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c0f4fd0ce0 con 0x55c0f50c3dc0 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.753+0000 7f6856251740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c0f50252a0 con 0x55c0f50c34a0 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.753+0000 7f6851413640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50ce680 0x55c0f50d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6851413640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50ce680 0x55c0f50d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46254/0 (socket says 172.21.15.33:46254) 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6851413640 1 -- 172.21.15.33:0/4008091636 learned_addr learned my addr 172.21.15.33:0/4008091636 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6851c14640 1 --2- 172.21.15.33:0/4008091636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50c3dc0 0x55c0f50ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6852415640 1 --2- 172.21.15.33:0/4008091636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c34a0 0x55c0f50c3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6842ffd640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68380a32c0 0x7f68380a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6851413640 1 -- 172.21.15.33:0/4008091636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c34a0 msgr2=0x55c0f50c3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:06.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6851413640 1 --2- 172.21.15.33:0/4008091636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c34a0 0x55c0f50c3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6851413640 1 -- 172.21.15.33:0/4008091636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50c3dc0 msgr2=0x55c0f50ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:06.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6851413640 1 --2- 172.21.15.33:0/4008091636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50c3dc0 0x55c0f50ce140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6851413640 1 -- 172.21.15.33:0/4008091636 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0f4fb5440 con 0x55c0f50ce680 2024-09-16T00:33:06.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6842ffd640 1 -- 172.21.15.33:0/2436147458 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f68480cfdb0 con 0x7f684c100e20 2024-09-16T00:33:06.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6851413640 1 --2- 172.21.15.33:0/4008091636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50ce680 0x55c0f50d0a70 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f683c009900 tx=0x7f683c004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477e7e17e5180adf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:06.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6837fff640 1 -- 172.21.15.33:0/4008091636 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c035030 con 0x55c0f50ce680 2024-09-16T00:33:06.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6837fff640 1 -- 172.21.15.33:0/4008091636 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c040070 con 0x55c0f50ce680 2024-09-16T00:33:06.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.754+0000 7f6837fff640 1 -- 172.21.15.33:0/4008091636 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c031b70 con 0x55c0f50ce680 2024-09-16T00:33:06.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.755+0000 7f6852415640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68380a32c0 0x7f68380a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.755+0000 7f6856251740 1 -- 172.21.15.33:0/4008091636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50ce680 msgr2=0x55c0f50d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:06.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.755+0000 7f6852415640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68380a32c0 0x7f68380a5780 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6844005fd0 tx=0x7f6844004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:06.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.755+0000 7f6856251740 1 --2- 172.21.15.33:0/4008091636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50ce680 0x55c0f50d0a70 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f683c009900 tx=0x7f683c004170 comp rx=0 tx=0).stop 2024-09-16T00:33:06.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.755+0000 7f6856251740 1 -- 172.21.15.33:0/4008091636 shutdown_connections 2024-09-16T00:33:06.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.755+0000 7f6856251740 1 --2- 172.21.15.33:0/4008091636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50ce680 0x55c0f50d0a70 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.755+0000 7f6856251740 1 --2- 172.21.15.33:0/4008091636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50c3dc0 0x55c0f50ce140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.755+0000 7f6856251740 1 --2- 172.21.15.33:0/4008091636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c34a0 0x55c0f50c3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.755+0000 7f6856251740 1 -- 172.21.15.33:0/4008091636 >> 172.21.15.33:0/4008091636 conn(0x55c0f50bfa30 msgr2=0x55c0f50bfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.755+0000 7f6856251740 1 -- 172.21.15.33:0/4008091636 shutdown_connections 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.755+0000 7f6856251740 1 -- 172.21.15.33:0/4008091636 wait complete. 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.756+0000 7f6856251740 1 -- start start 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.756+0000 7f6856251740 1 Processor -- start 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.756+0000 7f6856251740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c34a0 0x55c0f50cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.756+0000 7f6856251740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c3dc0 0x55c0f50cb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.756+0000 7f6856251740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50ce680 0x55c0f50cb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.756+0000 7f6856251740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c0f50d1250 con 0x55c0f50c3dc0 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.756+0000 7f6856251740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c0f4fe4050 con 0x55c0f50ce680 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.756+0000 7f6856251740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c0f50d13d0 con 0x55c0f50c34a0 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.756+0000 7f6856251740 10 client.0 _ll_register_callbacks cb 0x55c0f5024160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.756+0000 7f6851413640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c3dc0 0x55c0f50cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.756+0000 7f6851413640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c3dc0 0x55c0f50cb240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46256/0 (socket says 172.21.15.33:46256) 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.756+0000 7f6851413640 1 -- 172.21.15.33:0/3640713426 learned_addr learned my addr 172.21.15.33:0/3640713426 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f6851c14640 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c34a0 0x55c0f50cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f6852415640 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50ce680 0x55c0f50cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f6851413640 1 -- 172.21.15.33:0/3640713426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c34a0 msgr2=0x55c0f50cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f6851413640 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c34a0 0x55c0f50cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f6851413640 1 -- 172.21.15.33:0/3640713426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50ce680 msgr2=0x55c0f50cb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f6851413640 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50ce680 0x55c0f50cb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f6851413640 1 -- 172.21.15.33:0/3640713426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0f4d9a3c0 con 0x55c0f50c3dc0 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f6851413640 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c3dc0 0x55c0f50cb240 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f683c000c00 tx=0x7f683c02abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7827ff6b37151a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f68357fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f68357fa640 1 -- 172.21.15.33:0/3640713426 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c035070 con 0x55c0f50c3dc0 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f6856251740 1 -- 172.21.15.33:0/3640713426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f683c040070 con 0x55c0f50c3dc0 2024-09-16T00:33:06.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f68357fa640 1 -- 172.21.15.33:0/3640713426 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f683c031bc0 con 0x55c0f50c3dc0 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f68357fa640 1 -- 172.21.15.33:0/3640713426 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f683c056050 con 0x55c0f50c3dc0 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f6856251740 10 client.18396 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.757+0000 7f6856251740 1 -- 172.21.15.33:0/3640713426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c0f50dcbd0 con 0x55c0f50c3dc0 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.758+0000 7f68357fa640 1 -- 172.21.15.33:0/3640713426 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c040070 con 0x55c0f50c3dc0 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.758+0000 7f68357fa640 1 -- 172.21.15.33:0/3640713426 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f683c0456d0 con 0x55c0f50c3dc0 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.758+0000 7f6856251740 10 client.18396 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.758+0000 7f6856251740 10 client.18396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.758+0000 7f6856251740 20 client.18396 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.758+0000 7f6856251740 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0f50ddf20 0x55c0f50e0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.758+0000 7f6856251740 4 client.18396 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.758+0000 7f6856251740 1 -- 172.21.15.33:0/3640713426 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0f5036850 con 0x55c0f50ddf20 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.758+0000 7f6851c14640 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0f50ddf20 0x55c0f50e0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.759+0000 7f6851c14640 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0f50ddf20 0x55c0f50e0310 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ece8417dd4aa2d91 server_cookie=ce5ef1d2bd340285 in_seq=0 out_seq=0 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.759+0000 7f68357fa640 10 client.18396 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.759+0000 7f68357fa640 1 -- 172.21.15.33:0/3640713426 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c0f5036850 con 0x55c0f50ddf20 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.759+0000 7f68357fa640 10 client.18396 handle_command_reply: tid=0 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.759+0000 7f6856251740 1 client.18396 shutdown 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.759+0000 7f6856251740 1 -- 172.21.15.33:0/3640713426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c3dc0 msgr2=0x55c0f50cb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.759+0000 7f6856251740 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c3dc0 0x55c0f50cb240 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f683c000c00 tx=0x7f683c02abf0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.760+0000 7f6856251740 1 -- 172.21.15.33:0/3640713426 shutdown_connections 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.760+0000 7f6856251740 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0f50ddf20 0x55c0f50e0310 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.760+0000 7f6856251740 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50ce680 0x55c0f50cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.760+0000 7f68357fa640 0 client.18396 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.760+0000 7f6856251740 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c3dc0 0x55c0f50cb240 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.760+0000 7f6856251740 1 --2- 172.21.15.33:0/3640713426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c34a0 0x55c0f50cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.760+0000 7f6856251740 1 -- 172.21.15.33:0/3640713426 >> 172.21.15.33:0/3640713426 conn(0x55c0f50bfa30 msgr2=0x55c0f50c5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:06.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.760+0000 7f6856251740 1 -- 172.21.15.33:0/3640713426 shutdown_connections 2024-09-16T00:33:06.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.760+0000 7f6856251740 1 -- 172.21.15.33:0/3640713426 wait complete. 2024-09-16T00:33:06.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.760+0000 7f6856251740 20 client.18396 trim_cache size 0 max 16384 2024-09-16T00:33:06.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.777+0000 7f6856251740 1 Processor -- start 2024-09-16T00:33:06.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.777+0000 7f6856251740 1 -- start start 2024-09-16T00:33:06.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.777+0000 7f6856251740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50c34a0 0x55c0f50e3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.777+0000 7f6856251740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c3dc0 0x55c0f50e3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.777+0000 7f6856251740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50ce680 0x55c0f50c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.777+0000 7f6856251740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f684c1b19b0 con 0x55c0f50ce680 2024-09-16T00:33:06.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.777+0000 7f6856251740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f684c1af220 con 0x55c0f50c34a0 2024-09-16T00:33:06.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.777+0000 7f6856251740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f684c1ad070 con 0x55c0f50c3dc0 2024-09-16T00:33:06.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.777+0000 7f6851c14640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50c34a0 0x55c0f50e3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.777+0000 7f6851c14640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50c34a0 0x55c0f50e3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51066/0 (socket says 172.21.15.33:51066) 2024-09-16T00:33:06.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.777+0000 7f6851413640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c3dc0 0x55c0f50e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.777+0000 7f6851c14640 1 -- 172.21.15.33:0/4162754091 learned_addr learned my addr 172.21.15.33:0/4162754091 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.778+0000 7f6852415640 1 --2- 172.21.15.33:0/4162754091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50ce680 0x55c0f50c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.778+0000 7f6851413640 1 -- 172.21.15.33:0/4162754091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50c34a0 msgr2=0x55c0f50e3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.778+0000 7f6851413640 1 --2- 172.21.15.33:0/4162754091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50c34a0 0x55c0f50e3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.778+0000 7f6851413640 1 -- 172.21.15.33:0/4162754091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50ce680 msgr2=0x55c0f50c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.778+0000 7f6851413640 1 --2- 172.21.15.33:0/4162754091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50ce680 0x55c0f50c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.778+0000 7f6851413640 1 -- 172.21.15.33:0/4162754091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0f50c3bf0 con 0x55c0f50c3dc0 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.778+0000 7f6851413640 1 --2- 172.21.15.33:0/4162754091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c3dc0 0x55c0f50e3870 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f683c02a9a0 tx=0x7f683c004170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70fc526c471f1501 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.778+0000 7f68377fe640 1 -- 172.21.15.33:0/4162754091 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c035040 con 0x55c0f50c3dc0 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.778+0000 7f68377fe640 1 -- 172.21.15.33:0/4162754091 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c040040 con 0x55c0f50c3dc0 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.778+0000 7f68377fe640 1 -- 172.21.15.33:0/4162754091 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c003bc0 con 0x55c0f50c3dc0 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.779+0000 7f6856251740 1 -- 172.21.15.33:0/4162754091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c3dc0 msgr2=0x55c0f50e3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:06.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.779+0000 7f6856251740 1 --2- 172.21.15.33:0/4162754091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c3dc0 0x55c0f50e3870 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f683c02a9a0 tx=0x7f683c004170 comp rx=0 tx=0).stop 2024-09-16T00:33:06.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.779+0000 7f6856251740 1 -- 172.21.15.33:0/4162754091 shutdown_connections 2024-09-16T00:33:06.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.779+0000 7f6856251740 1 --2- 172.21.15.33:0/4162754091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50ce680 0x55c0f50c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.779+0000 7f6856251740 1 --2- 172.21.15.33:0/4162754091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c3dc0 0x55c0f50e3870 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.779+0000 7f6856251740 1 --2- 172.21.15.33:0/4162754091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50c34a0 0x55c0f50e3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.779+0000 7f6856251740 1 -- 172.21.15.33:0/4162754091 >> 172.21.15.33:0/4162754091 conn(0x55c0f50bfa30 msgr2=0x55c0f50c6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:06.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.779+0000 7f6856251740 1 -- 172.21.15.33:0/4162754091 shutdown_connections 2024-09-16T00:33:06.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.779+0000 7f6856251740 1 -- 172.21.15.33:0/4162754091 wait complete. 2024-09-16T00:33:06.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.779+0000 7f6856251740 1 -- start start 2024-09-16T00:33:06.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.779+0000 7f6856251740 1 Processor -- start 2024-09-16T00:33:06.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6856251740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c34a0 0x55c0f50e3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6856251740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c3dc0 0x55c0f50e3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6856251740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50ce680 0x55c0f50e08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6856251740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f684c1ad6e0 con 0x55c0f50c34a0 2024-09-16T00:33:06.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6856251740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f684c1aeb50 con 0x55c0f50ce680 2024-09-16T00:33:06.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6856251740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f684c1addb0 con 0x55c0f50c3dc0 2024-09-16T00:33:06.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6851c14640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c34a0 0x55c0f50e3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6851c14640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c34a0 0x55c0f50e3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46266/0 (socket says 172.21.15.33:46266) 2024-09-16T00:33:06.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6851c14640 1 -- 172.21.15.33:0/2055585799 learned_addr learned my addr 172.21.15.33:0/2055585799 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:06.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6856251740 10 client.0 _ll_register_callbacks cb 0x7f68480d3760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:06.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6851413640 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c3dc0 0x55c0f50e3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6852415640 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50ce680 0x55c0f50e08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6851c14640 1 -- 172.21.15.33:0/2055585799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c3dc0 msgr2=0x55c0f50e3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:06.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6851c14640 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c3dc0 0x55c0f50e3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6851c14640 1 -- 172.21.15.33:0/2055585799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50ce680 msgr2=0x55c0f50e08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:06.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6851c14640 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50ce680 0x55c0f50e08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6851c14640 1 -- 172.21.15.33:0/2055585799 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0f50e0fe0 con 0x55c0f50c34a0 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.780+0000 7f6852415640 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50ce680 0x55c0f50e08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f6851c14640 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c34a0 0x55c0f50e3330 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f6848020040 tx=0x7f68480d3890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a290ee50a7dac8b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f68357fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f68357fa640 1 -- 172.21.15.33:0/2055585799 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684801d040 con 0x55c0f50c34a0 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f68357fa640 1 -- 172.21.15.33:0/2055585799 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f68480985c0 con 0x55c0f50c34a0 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f6856251740 1 -- 172.21.15.33:0/2055585799 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f683c040040 con 0x55c0f50c34a0 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f68357fa640 1 -- 172.21.15.33:0/2055585799 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f684809ba90 con 0x55c0f50c34a0 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f6856251740 10 client.18402 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f6856251740 1 -- 172.21.15.33:0/2055585799 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c0f5148410 con 0x55c0f50c34a0 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f68357fa640 1 -- 172.21.15.33:0/2055585799 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c040040 con 0x55c0f50c34a0 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f68357fa640 1 -- 172.21.15.33:0/2055585799 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6848007ca0 con 0x55c0f50c34a0 2024-09-16T00:33:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f6856251740 10 client.18402 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:06.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f6856251740 10 client.18402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:33:06.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f6856251740 20 client.18402 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:06.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f6856251740 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0f5148ae0 0x55c0f514af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:06.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f6856251740 4 client.18402 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:33:06.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.781+0000 7f6856251740 1 -- 172.21.15.33:0/2055585799 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c0f5030d60 con 0x55c0f5148ae0 2024-09-16T00:33:06.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.782+0000 7f6851413640 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0f5148ae0 0x55c0f514af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:06.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.782+0000 7f6851413640 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0f5148ae0 0x55c0f514af00 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3e64c24198b550 server_cookie=5f3a179640ddfbb in_seq=0 out_seq=0 2024-09-16T00:33:06.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.782+0000 7f68357fa640 10 client.18402 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:06.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.782+0000 7f68357fa640 1 -- 172.21.15.33:0/2055585799 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c0f5030d60 con 0x55c0f5148ae0 2024-09-16T00:33:06.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.782+0000 7f68357fa640 10 client.18402 handle_command_reply: tid=0 2024-09-16T00:33:06.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.782+0000 7f6856251740 1 client.18402 shutdown 2024-09-16T00:33:06.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6856251740 1 -- 172.21.15.33:0/2055585799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c34a0 msgr2=0x55c0f50e3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:06.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6856251740 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c34a0 0x55c0f50e3330 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f6848020040 tx=0x7f68480d3890 comp rx=0 tx=0).stop 2024-09-16T00:33:06.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6856251740 1 -- 172.21.15.33:0/2055585799 shutdown_connections 2024-09-16T00:33:06.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6856251740 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0f5148ae0 0x55c0f514af00 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6856251740 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0f50ce680 0x55c0f50e08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6856251740 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0f50c3dc0 0x55c0f50e3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f68357fa640 0 client.18402 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:06.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6856251740 1 --2- 172.21.15.33:0/2055585799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0f50c34a0 0x55c0f50e3330 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6856251740 1 -- 172.21.15.33:0/2055585799 >> 172.21.15.33:0/2055585799 conn(0x55c0f50bfa30 msgr2=0x55c0f50ccaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:06.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6856251740 1 -- 172.21.15.33:0/2055585799 shutdown_connections 2024-09-16T00:33:06.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6856251740 1 -- 172.21.15.33:0/2055585799 wait complete. 2024-09-16T00:33:06.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6856251740 20 client.18402 trim_cache size 0 max 16384 2024-09-16T00:33:06.793 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:33:06.793 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:33:06.793 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:33:06.793 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:33:06.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6853e78640 1 -- 172.21.15.33:0/2436147458 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68380a32c0 msgr2=0x7f68380a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:06.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6853e78640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68380a32c0 0x7f68380a5780 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6844005fd0 tx=0x7f6844004a40 comp rx=0 tx=0).stop 2024-09-16T00:33:06.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6853e78640 1 -- 172.21.15.33:0/2436147458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f684c100e20 msgr2=0x7f684c1984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:06.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.783+0000 7f6853e78640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f684c100e20 0x7f684c1984e0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f684800c890 tx=0x7f684800cd50 comp rx=0 tx=0).stop 2024-09-16T00:33:06.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.784+0000 7f6853e78640 1 -- 172.21.15.33:0/2436147458 shutdown_connections 2024-09-16T00:33:06.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.784+0000 7f6853e78640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f68380a32c0 0x7f68380a5780 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.784+0000 7f6853e78640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f684c102ef0 0x7f684c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.784+0000 7f6853e78640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f684c102570 0x7f684c198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.784+0000 7f6853e78640 1 --2- 172.21.15.33:0/2436147458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f684c100e20 0x7f684c1984e0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:06.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.784+0000 7f6853e78640 1 -- 172.21.15.33:0/2436147458 >> 172.21.15.33:0/2436147458 conn(0x7f684c0fc7d0 msgr2=0x7f684c069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:06.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.784+0000 7f6853e78640 1 -- 172.21.15.33:0/2436147458 shutdown_connections 2024-09-16T00:33:06.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:06.784+0000 7f6853e78640 1 -- 172.21.15.33:0/2436147458 wait complete. 2024-09-16T00:33:07.073 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 10.91 MB/sec execute 13 sec latency 4702.256 ms 2024-09-16T00:33:07.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:07 smithi033 ceph-mon[31243]: pgmap v829: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 249 op/s 2024-09-16T00:33:07.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:07 smithi083 ceph-mon[35242]: pgmap v829: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 249 op/s 2024-09-16T00:33:07.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:07 smithi060 ceph-mon[35185]: pgmap v829: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 249 op/s 2024-09-16T00:33:07.642 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:33:07.643 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:33:07.902 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:08.073 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 10.13 MB/sec execute 14 sec latency 5702.761 ms 2024-09-16T00:33:08.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.396+0000 7fe9a73bc640 1 -- 172.21.15.33:0/1192511283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9a0101de0 msgr2=0x7fe9a01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:08.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.396+0000 7fe9a73bc640 1 --2- 172.21.15.33:0/1192511283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9a0101de0 0x7fe9a01021c0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009900 tx=0x7fe99c02a3f0 comp rx=0 tx=0).stop 2024-09-16T00:33:08.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.396+0000 7fe9a73bc640 1 -- 172.21.15.33:0/1192511283 shutdown_connections 2024-09-16T00:33:08.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.396+0000 7fe9a73bc640 1 --2- 172.21.15.33:0/1192511283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9a0106f10 0x7fe9a010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:08.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.396+0000 7fe9a73bc640 1 --2- 172.21.15.33:0/1192511283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9a0102790 0x7fe9a01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:08.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.396+0000 7fe9a73bc640 1 --2- 172.21.15.33:0/1192511283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9a0101de0 0x7fe9a01021c0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:08.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.396+0000 7fe9a73bc640 1 -- 172.21.15.33:0/1192511283 >> 172.21.15.33:0/1192511283 conn(0x7fe9a00fd8c0 msgr2=0x7fe9a00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:08.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.396+0000 7fe9a73bc640 1 -- 172.21.15.33:0/1192511283 shutdown_connections 2024-09-16T00:33:08.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.396+0000 7fe9a73bc640 1 -- 172.21.15.33:0/1192511283 wait complete. 2024-09-16T00:33:08.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a73bc640 1 Processor -- start 2024-09-16T00:33:08.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a73bc640 1 -- start start 2024-09-16T00:33:08.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a73bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9a0101de0 0x7fe9a01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:08.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a73bc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9a0102790 0x7fe9a0198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:08.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a73bc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9a0106f10 0x7fe9a019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:08.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a73bc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a0111f70 con 0x7fe9a0102790 2024-09-16T00:33:08.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a73bc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a0111df0 con 0x7fe9a0106f10 2024-09-16T00:33:08.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a73bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a01120f0 con 0x7fe9a0101de0 2024-09-16T00:33:08.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a4957640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9a0102790 0x7fe9a0198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:08.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a5158640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9a0101de0 0x7fe9a01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:08.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a4957640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9a0102790 0x7fe9a0198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46282/0 (socket says 172.21.15.33:46282) 2024-09-16T00:33:08.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a4957640 1 -- 172.21.15.33:0/4081870149 learned_addr learned my addr 172.21.15.33:0/4081870149 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:08.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a5959640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9a0106f10 0x7fe9a019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:08.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a4957640 1 -- 172.21.15.33:0/4081870149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9a0101de0 msgr2=0x7fe9a01983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:08.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a4957640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9a0101de0 0x7fe9a01983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:08.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a4957640 1 -- 172.21.15.33:0/4081870149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9a0106f10 msgr2=0x7fe9a019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:08.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a4957640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9a0106f10 0x7fe9a019cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:08.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.397+0000 7fe9a4957640 1 -- 172.21.15.33:0/4081870149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a019d380 con 0x7fe9a0102790 2024-09-16T00:33:08.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.398+0000 7fe9a4957640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9a0102790 0x7fe9a0198910 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fe99000c890 tx=0x7fe99000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:08.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.398+0000 7fe9967fc640 1 -- 172.21.15.33:0/4081870149 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990004220 con 0x7fe9a0102790 2024-09-16T00:33:08.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.398+0000 7fe9a73bc640 1 -- 172.21.15.33:0/4081870149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a019d670 con 0x7fe9a0102790 2024-09-16T00:33:08.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.398+0000 7fe9967fc640 1 -- 172.21.15.33:0/4081870149 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9900043c0 con 0x7fe9a0102790 2024-09-16T00:33:08.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.398+0000 7fe9967fc640 1 -- 172.21.15.33:0/4081870149 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990004a00 con 0x7fe9a0102790 2024-09-16T00:33:08.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.398+0000 7fe9a73bc640 1 -- 172.21.15.33:0/4081870149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a01a4f50 con 0x7fe9a0102790 2024-09-16T00:33:08.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.399+0000 7fe9a73bc640 1 -- 172.21.15.33:0/4081870149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a01021c0 con 0x7fe9a0102790 2024-09-16T00:33:08.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.400+0000 7fe9967fc640 1 -- 172.21.15.33:0/4081870149 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe9900079b0 con 0x7fe9a0102790 2024-09-16T00:33:08.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.401+0000 7fe9967fc640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe98c0a3240 0x7fe98c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:08.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.401+0000 7fe9967fc640 1 -- 172.21.15.33:0/4081870149 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe9900cfa40 con 0x7fe9a0102790 2024-09-16T00:33:08.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.401+0000 7fe9a5158640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe98c0a3240 0x7fe98c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:08.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.401+0000 7fe9a5158640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe98c0a3240 0x7fe98c0a5700 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe99c000c00 tx=0x7fe99c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:08.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.405+0000 7fe9967fc640 1 -- 172.21.15.33:0/4081870149 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe990014030 con 0x7fe9a0102790 2024-09-16T00:33:08.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.620+0000 7fe9a73bc640 1 -- 172.21.15.33:0/4081870149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9a0199730 con 0x7fe9a0102790 2024-09-16T00:33:08.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.620+0000 7fe9967fc640 1 -- 172.21.15.33:0/4081870149 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11906 (secure 0 0 0) 0x7fe9a0199730 con 0x7fe9a0102790 2024-09-16T00:33:08.625 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:33:08.625 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":31,"btime":"2024-09-16T00:30:57:641642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:30:57.027530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:33:08.626 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 31 2024-09-16T00:33:08.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.623+0000 7fe9a73bc640 1 -- 172.21.15.33:0/4081870149 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe98c0a3240 msgr2=0x7fe98c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:08.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.623+0000 7fe9a73bc640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe98c0a3240 0x7fe98c0a5700 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe99c000c00 tx=0x7fe99c0058d0 comp rx=0 tx=0).stop 2024-09-16T00:33:08.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.623+0000 7fe9a73bc640 1 -- 172.21.15.33:0/4081870149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9a0102790 msgr2=0x7fe9a0198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:08.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.623+0000 7fe9a73bc640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9a0102790 0x7fe9a0198910 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fe99000c890 tx=0x7fe99000cd50 comp rx=0 tx=0).stop 2024-09-16T00:33:08.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.623+0000 7fe9a73bc640 1 -- 172.21.15.33:0/4081870149 shutdown_connections 2024-09-16T00:33:08.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.623+0000 7fe9a73bc640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe98c0a3240 0x7fe98c0a5700 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:08.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.623+0000 7fe9a73bc640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9a0106f10 0x7fe9a019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:08.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.623+0000 7fe9a73bc640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9a0102790 0x7fe9a0198910 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:08.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.623+0000 7fe9a73bc640 1 --2- 172.21.15.33:0/4081870149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9a0101de0 0x7fe9a01983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:08.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.623+0000 7fe9a73bc640 1 -- 172.21.15.33:0/4081870149 >> 172.21.15.33:0/4081870149 conn(0x7fe9a00fd8c0 msgr2=0x7fe9a00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:08.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.623+0000 7fe9a73bc640 1 -- 172.21.15.33:0/4081870149 shutdown_connections 2024-09-16T00:33:08.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:08.623+0000 7fe9a73bc640 1 -- 172.21.15.33:0/4081870149 wait complete. 2024-09-16T00:33:09.073 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 9.45 MB/sec execute 15 sec latency 6702.858 ms 2024-09-16T00:33:09.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:09 smithi033 ceph-mon[31243]: pgmap v830: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 210 op/s 2024-09-16T00:33:09.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:09 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4081870149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:33:09.361 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:33:09.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:09 smithi083 ceph-mon[35242]: pgmap v830: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 210 op/s 2024-09-16T00:33:09.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:09 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4081870149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:33:09.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:09 smithi060 ceph-mon[35185]: pgmap v830: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 210 op/s 2024-09-16T00:33:09.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:09 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4081870149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:33:09.620 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:10.073 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 8.86 MB/sec execute 16 sec latency 7702.943 ms 2024-09-16T00:33:10.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.375+0000 7f6ea8b37640 1 -- 172.21.15.33:0/127884132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea4100a80 msgr2=0x7f6ea4100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:10.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.375+0000 7f6ea8b37640 1 --2- 172.21.15.33:0/127884132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea4100a80 0x7f6ea4100ee0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009930 tx=0x7f6e9802a420 comp rx=0 tx=0).stop 2024-09-16T00:33:10.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.375+0000 7f6ea8b37640 1 -- 172.21.15.33:0/127884132 shutdown_connections 2024-09-16T00:33:10.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.375+0000 7f6ea8b37640 1 --2- 172.21.15.33:0/127884132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ea4101420 0x7f6ea410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.375+0000 7f6ea8b37640 1 --2- 172.21.15.33:0/127884132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea4100a80 0x7f6ea4100ee0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.375+0000 7f6ea8b37640 1 --2- 172.21.15.33:0/127884132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea4106a80 0x7f6ea4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.375+0000 7f6ea8b37640 1 -- 172.21.15.33:0/127884132 >> 172.21.15.33:0/127884132 conn(0x7f6ea40fc7a0 msgr2=0x7f6ea40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.376+0000 7f6ea8b37640 1 -- 172.21.15.33:0/127884132 shutdown_connections 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.376+0000 7f6ea8b37640 1 -- 172.21.15.33:0/127884132 wait complete. 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.376+0000 7f6ea8b37640 1 Processor -- start 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea8b37640 1 -- start start 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea8b37640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea4100a80 0x7f6ea4197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea8b37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ea4101420 0x7f6ea4197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea8b37640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea4106a80 0x7f6ea4197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea8b37640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4110e80 con 0x7f6ea4100a80 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea8b37640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4110d00 con 0x7f6ea4106a80 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea8b37640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4111000 con 0x7f6ea4101420 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea4100a80 0x7f6ea4197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea4100a80 0x7f6ea4197100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60664/0 (socket says 172.21.15.33:60664) 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea259c640 1 -- 172.21.15.33:0/541770440 learned_addr learned my addr 172.21.15.33:0/541770440 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea2d9d640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea4106a80 0x7f6ea4197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea1d9b640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ea4101420 0x7f6ea4197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea259c640 1 -- 172.21.15.33:0/541770440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ea4101420 msgr2=0x7f6ea4197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea259c640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ea4101420 0x7f6ea4197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea259c640 1 -- 172.21.15.33:0/541770440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea4106a80 msgr2=0x7f6ea4197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea259c640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea4106a80 0x7f6ea4197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea259c640 1 -- 172.21.15.33:0/541770440 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea41a6970 con 0x7f6ea4100a80 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea2d9d640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea4106a80 0x7f6ea4197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.377+0000 7f6ea259c640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea4100a80 0x7f6ea4197100 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f6e9400e9f0 tx=0x7f6e9400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.378+0000 7f6e937fe640 1 -- 172.21.15.33:0/541770440 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9400cd80 con 0x7f6ea4100a80 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.378+0000 7f6e937fe640 1 -- 172.21.15.33:0/541770440 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e94010910 con 0x7f6ea4100a80 2024-09-16T00:33:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.378+0000 7f6ea8b37640 1 -- 172.21.15.33:0/541770440 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea41a6c60 con 0x7f6ea4100a80 2024-09-16T00:33:10.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.378+0000 7f6e937fe640 1 -- 172.21.15.33:0/541770440 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94018990 con 0x7f6ea4100a80 2024-09-16T00:33:10.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.378+0000 7f6ea8b37640 1 -- 172.21.15.33:0/541770440 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea41a7170 con 0x7f6ea4100a80 2024-09-16T00:33:10.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.380+0000 7f6e937fe640 1 -- 172.21.15.33:0/541770440 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6e94010ac0 con 0x7f6ea4100a80 2024-09-16T00:33:10.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.380+0000 7f6eaaf10740 1 Processor -- start 2024-09-16T00:33:10.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6eaaf10740 1 -- start start 2024-09-16T00:33:10.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6eaaf10740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a274b40 0x55cc9a273090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6eaaf10740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a2735d0 0x55cc9a273a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6eaaf10740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a27e370 0x55cc9a280760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6eaaf10740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cc9a1d92f0 con 0x55cc9a2735d0 2024-09-16T00:33:10.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6eaaf10740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55cc9a180cf0 con 0x55cc9a27e370 2024-09-16T00:33:10.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6eaaf10740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cc9a1d52b0 con 0x55cc9a274b40 2024-09-16T00:33:10.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6ea259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a2735d0 0x55cc9a273a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6ea259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a2735d0 0x55cc9a273a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60668/0 (socket says 172.21.15.33:60668) 2024-09-16T00:33:10.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6ea259c640 1 -- 172.21.15.33:0/3204803476 learned_addr learned my addr 172.21.15.33:0/3204803476 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:10.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6ea2d9d640 1 --2- 172.21.15.33:0/3204803476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a274b40 0x55cc9a273090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6e937fe640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6e8c0a3240 0x7f6e8c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6ea1d9b640 1 --2- 172.21.15.33:0/3204803476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a27e370 0x55cc9a280760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6ea2d9d640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6e8c0a3240 0x7f6e8c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6ea259c640 1 -- 172.21.15.33:0/3204803476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a274b40 msgr2=0x55cc9a273090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:10.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6ea259c640 1 --2- 172.21.15.33:0/3204803476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a274b40 0x55cc9a273090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6ea259c640 1 -- 172.21.15.33:0/3204803476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a27e370 msgr2=0x55cc9a280760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6ea259c640 1 --2- 172.21.15.33:0/3204803476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a27e370 0x55cc9a280760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6e937fe640 1 -- 172.21.15.33:0/541770440 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6e940d0180 con 0x7f6ea4100a80 2024-09-16T00:33:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6ea259c640 1 -- 172.21.15.33:0/3204803476 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc9a1653a0 con 0x55cc9a2735d0 2024-09-16T00:33:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.381+0000 7f6ea1d9b640 1 --2- 172.21.15.33:0/3204803476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a27e370 0x55cc9a280760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.382+0000 7f6ea259c640 1 --2- 172.21.15.33:0/3204803476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a2735d0 0x55cc9a273a10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f6e94098cf0 tx=0x7f6e94018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9bdda49d7ca064 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.382+0000 7f6e90ff9640 1 -- 172.21.15.33:0/3204803476 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9409c0f0 con 0x55cc9a2735d0 2024-09-16T00:33:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.382+0000 7f6ea2d9d640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6e8c0a3240 0x7f6e8c0a5700 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c012590 tx=0x7f6e9c004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.382+0000 7f6e90ff9640 1 -- 172.21.15.33:0/3204803476 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9409ca20 con 0x55cc9a2735d0 2024-09-16T00:33:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.382+0000 7f6e90ff9640 1 -- 172.21.15.33:0/3204803476 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9409bc80 con 0x55cc9a2735d0 2024-09-16T00:33:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3204803476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a2735d0 msgr2=0x55cc9a273a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/3204803476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a2735d0 0x55cc9a273a10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f6e94098cf0 tx=0x7f6e94018420 comp rx=0 tx=0).stop 2024-09-16T00:33:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3204803476 shutdown_connections 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/3204803476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a27e370 0x55cc9a280760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/3204803476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a2735d0 0x55cc9a273a10 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/3204803476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a274b40 0x55cc9a273090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3204803476 >> 172.21.15.33:0/3204803476 conn(0x55cc9a23c310 msgr2=0x55cc9a26fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3204803476 shutdown_connections 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3204803476 wait complete. 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 -- start start 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 Processor -- start 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a27dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a274b40 0x55cc9a27afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a27e370 0x55cc9a27b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cc9a280fb0 con 0x55cc9a274b40 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55cc9a194060 con 0x55cc9a27e370 2024-09-16T00:33:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cc9a281130 con 0x55cc9a2735d0 2024-09-16T00:33:10.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.383+0000 7f6eaaf10740 10 client.0 _ll_register_callbacks cb 0x55cc99ee78c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:10.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6ea1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a274b40 0x55cc9a27afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6ea1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a274b40 0x55cc9a27afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60672/0 (socket says 172.21.15.33:60672) 2024-09-16T00:33:10.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6ea1d9b640 1 -- 172.21.15.33:0/3314709297 learned_addr learned my addr 172.21.15.33:0/3314709297 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:10.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6ea259c640 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a27dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6ea2d9d640 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a27e370 0x55cc9a27b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6ea1d9b640 1 -- 172.21.15.33:0/3314709297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 msgr2=0x55cc9a27dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:10.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6ea1d9b640 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a27dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6ea1d9b640 1 -- 172.21.15.33:0/3314709297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a27e370 msgr2=0x55cc9a27b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:10.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6ea1d9b640 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a27e370 0x55cc9a27b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6ea1d9b640 1 -- 172.21.15.33:0/3314709297 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc99f4a3c0 con 0x55cc9a274b40 2024-09-16T00:33:10.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6ea259c640 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a27dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6ea1d9b640 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a274b40 0x55cc9a27afb0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f6e98005ab0 tx=0x7f6e980317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e11d088cef1056 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6e89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6e89ffb640 1 -- 172.21.15.33:0/3314709297 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98035070 con 0x55cc9a274b40 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6e89ffb640 1 -- 172.21.15.33:0/3314709297 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6e98004040 con 0x55cc9a274b40 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3314709297 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e9409ca20 con 0x55cc9a274b40 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6e89ffb640 1 -- 172.21.15.33:0/3314709297 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6e9804d050 con 0x55cc9a274b40 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6eaaf10740 10 client.18420 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.384+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3314709297 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cc9a28ca10 con 0x55cc9a274b40 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.385+0000 7f6e89ffb640 1 -- 172.21.15.33:0/3314709297 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9409ca20 con 0x55cc9a274b40 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.385+0000 7f6e89ffb640 1 -- 172.21.15.33:0/3314709297 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6e980049a0 con 0x55cc9a274b40 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.385+0000 7f6eaaf10740 10 client.18420 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.385+0000 7f6eaaf10740 10 client.18420 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.385+0000 7f6eaaf10740 20 client.18420 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.385+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cc9a28df70 0x55cc9a2903d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.385+0000 7f6eaaf10740 4 client.18420 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.385+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3314709297 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc9a1e6860 con 0x55cc9a28df70 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.385+0000 7f6ea259c640 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cc9a28df70 0x55cc9a2903d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.386+0000 7f6ea259c640 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cc9a28df70 0x55cc9a2903d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f0861de128128ee4 server_cookie=b275ff7f77dcdd9a in_seq=0 out_seq=0 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.386+0000 7f6e89ffb640 10 client.18420 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.386+0000 7f6e89ffb640 1 -- 172.21.15.33:0/3314709297 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cc9a1e6860 con 0x55cc9a28df70 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.386+0000 7f6e89ffb640 10 client.18420 handle_command_reply: tid=0 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.386+0000 7f6eaaf10740 1 client.18420 shutdown 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.387+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3314709297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a274b40 msgr2=0x55cc9a27afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.387+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a274b40 0x55cc9a27afb0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f6e98005ab0 tx=0x7f6e980317e0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.387+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3314709297 shutdown_connections 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.387+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cc9a28df70 0x55cc9a2903d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.387+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a27e370 0x55cc9a27b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.387+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a274b40 0x55cc9a27afb0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.387+0000 7f6e89ffb640 0 client.18420 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.387+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/3314709297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a27dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.387+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3314709297 >> 172.21.15.33:0/3314709297 conn(0x55cc9a23c310 msgr2=0x55cc9a275990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.387+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3314709297 shutdown_connections 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.387+0000 7f6eaaf10740 1 -- 172.21.15.33:0/3314709297 wait complete. 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.387+0000 7f6eaaf10740 20 client.18420 trim_cache size 0 max 16384 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6eaaf10740 1 Processor -- start 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6eaaf10740 1 -- start start 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6eaaf10740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a28a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6eaaf10740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a274b40 0x55cc9a295ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6eaaf10740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 0x55cc9a2961e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6eaaf10740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41b3c40 con 0x55cc9a27e370 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6eaaf10740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41b14b0 con 0x55cc9a274b40 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6eaaf10740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41af300 con 0x55cc9a2735d0 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6ea2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 0x55cc9a2961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6ea2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 0x55cc9a2961e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60678/0 (socket says 172.21.15.33:60678) 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6ea2d9d640 1 -- 172.21.15.33:0/2165688411 learned_addr learned my addr 172.21.15.33:0/2165688411 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6ea259c640 1 --2- 172.21.15.33:0/2165688411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a28a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6ea1d9b640 1 --2- 172.21.15.33:0/2165688411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a274b40 0x55cc9a295ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6ea2d9d640 1 -- 172.21.15.33:0/2165688411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 msgr2=0x55cc9a28a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.400+0000 7f6ea2d9d640 1 --2- 172.21.15.33:0/2165688411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a28a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.401+0000 7f6ea2d9d640 1 -- 172.21.15.33:0/2165688411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a274b40 msgr2=0x55cc9a295ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.401+0000 7f6ea2d9d640 1 --2- 172.21.15.33:0/2165688411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a274b40 0x55cc9a295ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.401+0000 7f6ea2d9d640 1 -- 172.21.15.33:0/2165688411 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc9a279500 con 0x55cc9a27e370 2024-09-16T00:33:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.401+0000 7f6ea259c640 1 --2- 172.21.15.33:0/2165688411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a28a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:10.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.401+0000 7f6ea1d9b640 1 --2- 172.21.15.33:0/2165688411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a274b40 0x55cc9a295ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:10.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.401+0000 7f6ea2d9d640 1 --2- 172.21.15.33:0/2165688411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 0x55cc9a2961e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c010800 tx=0x7f6e9c014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35c1abefa5343f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:10.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.401+0000 7f6e8bfff640 1 -- 172.21.15.33:0/2165688411 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c00f650 con 0x55cc9a27e370 2024-09-16T00:33:10.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.401+0000 7f6e8bfff640 1 -- 172.21.15.33:0/2165688411 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c00fcd0 con 0x55cc9a27e370 2024-09-16T00:33:10.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.401+0000 7f6e8bfff640 1 -- 172.21.15.33:0/2165688411 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c025c20 con 0x55cc9a27e370 2024-09-16T00:33:10.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2165688411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 msgr2=0x55cc9a2961e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:10.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/2165688411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 0x55cc9a2961e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c010800 tx=0x7f6e9c014040 comp rx=0 tx=0).stop 2024-09-16T00:33:10.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2165688411 shutdown_connections 2024-09-16T00:33:10.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/2165688411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 0x55cc9a2961e0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/2165688411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a274b40 0x55cc9a295ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/2165688411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a28a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2165688411 >> 172.21.15.33:0/2165688411 conn(0x55cc9a23c310 msgr2=0x55cc9a276230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:10.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2165688411 shutdown_connections 2024-09-16T00:33:10.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2165688411 wait complete. 2024-09-16T00:33:10.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 -- start start 2024-09-16T00:33:10.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 Processor -- start 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a282090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a274b40 0x55cc9a2825d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 0x55cc9a282b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41af970 con 0x55cc9a27e370 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41b0de0 con 0x55cc9a274b40 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41b0040 con 0x55cc9a2735d0 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6eaaf10740 10 client.0 _ll_register_callbacks cb 0x7f6e94005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.402+0000 7f6ea2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 0x55cc9a282b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6ea2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 0x55cc9a282b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60682/0 (socket says 172.21.15.33:60682) 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6ea2d9d640 1 -- 172.21.15.33:0/2793636956 learned_addr learned my addr 172.21.15.33:0/2793636956 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6ea259c640 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a282090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6ea1d9b640 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a274b40 0x55cc9a2825d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6ea2d9d640 1 -- 172.21.15.33:0/2793636956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 msgr2=0x55cc9a282090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6ea2d9d640 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a282090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6ea2d9d640 1 -- 172.21.15.33:0/2793636956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a274b40 msgr2=0x55cc9a2825d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6ea2d9d640 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a274b40 0x55cc9a2825d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6ea2d9d640 1 -- 172.21.15.33:0/2793636956 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc9a2f7c70 con 0x55cc9a27e370 2024-09-16T00:33:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6ea259c640 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a282090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6ea1d9b640 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a274b40 0x55cc9a2825d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:33:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6ea2d9d640 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 0x55cc9a282b10 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c012e20 tx=0x7f6e9c00aea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4f4f3924786659 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6e89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6e89ffb640 1 -- 172.21.15.33:0/2793636956 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c00fe80 con 0x55cc9a27e370 2024-09-16T00:33:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6e89ffb640 1 -- 172.21.15.33:0/2793636956 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6e9c021050 con 0x55cc9a27e370 2024-09-16T00:33:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2793636956 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e9c00fcd0 con 0x55cc9a27e370 2024-09-16T00:33:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6e89ffb640 1 -- 172.21.15.33:0/2793636956 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6e9c00f860 con 0x55cc9a27e370 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6eaaf10740 10 client.18426 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.403+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2793636956 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cc9a2f8eb0 con 0x55cc9a27e370 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.404+0000 7f6e89ffb640 1 -- 172.21.15.33:0/2793636956 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c00fcd0 con 0x55cc9a27e370 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.404+0000 7f6e89ffb640 1 -- 172.21.15.33:0/2793636956 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6e9c01a3f0 con 0x55cc9a27e370 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.404+0000 7f6eaaf10740 10 client.18426 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.404+0000 7f6eaaf10740 10 client.18426 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.404+0000 7f6eaaf10740 20 client.18426 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.404+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cc9a2f95c0 0x55cc9a2fb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.404+0000 7f6eaaf10740 4 client.18426 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.404+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2793636956 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cc9a1e0d70 con 0x55cc9a2f95c0 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.404+0000 7f6ea259c640 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cc9a2f95c0 0x55cc9a2fb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6ea259c640 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cc9a2f95c0 0x55cc9a2fb9e0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ad97c3fdd72c60ad server_cookie=8b8b05c6fa26db34 in_seq=0 out_seq=0 2024-09-16T00:33:10.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6e89ffb640 10 client.18426 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6e89ffb640 1 -- 172.21.15.33:0/2793636956 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cc9a1e0d70 con 0x55cc9a2f95c0 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6e89ffb640 10 client.18426 handle_command_reply: tid=0 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6eaaf10740 1 client.18426 shutdown 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2793636956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 msgr2=0x55cc9a282b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 0x55cc9a282b10 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c012e20 tx=0x7f6e9c00aea0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2793636956 shutdown_connections 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cc9a2f95c0 0x55cc9a2fb9e0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc9a27e370 0x55cc9a282b10 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc9a274b40 0x55cc9a2825d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6eaaf10740 1 --2- 172.21.15.33:0/2793636956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cc9a2735d0 0x55cc9a282090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2793636956 >> 172.21.15.33:0/2793636956 conn(0x55cc9a23c310 msgr2=0x55cc9a279880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6e89ffb640 0 client.18426 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:33:10.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2793636956 shutdown_connections 2024-09-16T00:33:10.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6eaaf10740 1 -- 172.21.15.33:0/2793636956 wait complete. 2024-09-16T00:33:10.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.405+0000 7f6eaaf10740 20 client.18426 trim_cache size 0 max 16384 2024-09-16T00:33:10.423 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:33:10.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.406+0000 7f6ea8b37640 1 -- 172.21.15.33:0/541770440 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6e8c0a3240 msgr2=0x7f6e8c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:10.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.406+0000 7f6ea8b37640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6e8c0a3240 0x7f6e8c0a5700 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c012590 tx=0x7f6e9c004ab0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.406+0000 7f6ea8b37640 1 -- 172.21.15.33:0/541770440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea4100a80 msgr2=0x7f6ea4197100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:10.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.406+0000 7f6ea8b37640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea4100a80 0x7f6ea4197100 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f6e9400e9f0 tx=0x7f6e9400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.406+0000 7f6ea8b37640 1 -- 172.21.15.33:0/541770440 shutdown_connections 2024-09-16T00:33:10.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.406+0000 7f6ea8b37640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6e8c0a3240 0x7f6e8c0a5700 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.406+0000 7f6ea8b37640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea4106a80 0x7f6ea4197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.406+0000 7f6ea8b37640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ea4101420 0x7f6ea4197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.406+0000 7f6ea8b37640 1 --2- 172.21.15.33:0/541770440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea4100a80 0x7f6ea4197100 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:10.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.406+0000 7f6ea8b37640 1 -- 172.21.15.33:0/541770440 >> 172.21.15.33:0/541770440 conn(0x7f6ea40fc7a0 msgr2=0x7f6ea40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:10.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.406+0000 7f6ea8b37640 1 -- 172.21.15.33:0/541770440 shutdown_connections 2024-09-16T00:33:10.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:10.406+0000 7f6ea8b37640 1 -- 172.21.15.33:0/541770440 wait complete. 2024-09-16T00:33:11.073 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 8.34 MB/sec execute 17 sec latency 8703.053 ms 2024-09-16T00:33:11.197 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:11 smithi033 ceph-mon[31243]: pgmap v831: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 209 op/s 2024-09-16T00:33:11.228 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:33:11.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:11 smithi083 ceph-mon[35242]: pgmap v831: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 209 op/s 2024-09-16T00:33:11.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:11 smithi060 ceph-mon[35185]: pgmap v831: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 209 op/s 2024-09-16T00:33:11.486 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:12.073 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 7.88 MB/sec execute 18 sec latency 9703.105 ms 2024-09-16T00:33:12.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.087+0000 7fab29b5c640 1 -- 172.21.15.33:0/3742994189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab24102ef0 msgr2=0x7fab2410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:12.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.087+0000 7fab29b5c640 1 --2- 172.21.15.33:0/3742994189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab24102ef0 0x7fab2410da40 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fab1c009ca0 tx=0x7fab1c02a950 comp rx=0 tx=0).stop 2024-09-16T00:33:12.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.087+0000 7fab29b5c640 1 -- 172.21.15.33:0/3742994189 shutdown_connections 2024-09-16T00:33:12.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.087+0000 7fab29b5c640 1 --2- 172.21.15.33:0/3742994189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab24102ef0 0x7fab2410da40 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.087+0000 7fab29b5c640 1 --2- 172.21.15.33:0/3742994189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab24102570 0x7fab241029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.087+0000 7fab29b5c640 1 --2- 172.21.15.33:0/3742994189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab24100e20 0x7fab24101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.087+0000 7fab29b5c640 1 -- 172.21.15.33:0/3742994189 >> 172.21.15.33:0/3742994189 conn(0x7fab240fc7d0 msgr2=0x7fab240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:12.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.087+0000 7fab29b5c640 1 -- 172.21.15.33:0/3742994189 shutdown_connections 2024-09-16T00:33:12.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.087+0000 7fab29b5c640 1 -- 172.21.15.33:0/3742994189 wait complete. 2024-09-16T00:33:12.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab29b5c640 1 Processor -- start 2024-09-16T00:33:12.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab29b5c640 1 -- start start 2024-09-16T00:33:12.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab29b5c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab24100e20 0x7fab24199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab29b5c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab24102570 0x7fab24199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab29b5c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab24102ef0 0x7fab241a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab29b5c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab24110fc0 con 0x7fab24100e20 2024-09-16T00:33:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab29b5c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fab24110e40 con 0x7fab24102570 2024-09-16T00:33:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab29b5c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fab24111140 con 0x7fab24102ef0 2024-09-16T00:33:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab237fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab24100e20 0x7fab24199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab237fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab24100e20 0x7fab24199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60692/0 (socket says 172.21.15.33:60692) 2024-09-16T00:33:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab237fe640 1 -- 172.21.15.33:0/1853048998 learned_addr learned my addr 172.21.15.33:0/1853048998 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab23fff640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab24102ef0 0x7fab241a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab1bfff640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab24102570 0x7fab24199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab237fe640 1 -- 172.21.15.33:0/1853048998 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab24102ef0 msgr2=0x7fab241a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab237fe640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab24102ef0 0x7fab241a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab237fe640 1 -- 172.21.15.33:0/1853048998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab24102570 msgr2=0x7fab24199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab237fe640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab24102570 0x7fab24199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab237fe640 1 -- 172.21.15.33:0/1853048998 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab241a4910 con 0x7fab24100e20 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.088+0000 7fab1bfff640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab24102570 0x7fab24199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.089+0000 7fab237fe640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab24100e20 0x7fab24199350 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fab1400b4a0 tx=0x7fab1400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.089+0000 7fab217fa640 1 -- 172.21.15.33:0/1853048998 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14014070 con 0x7fab24100e20 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.089+0000 7fab29b5c640 1 -- 172.21.15.33:0/1853048998 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab241a4ae0 con 0x7fab24100e20 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.089+0000 7fab217fa640 1 -- 172.21.15.33:0/1853048998 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab14010af0 con 0x7fab24100e20 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.089+0000 7fab217fa640 1 -- 172.21.15.33:0/1853048998 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14017d10 con 0x7fab24100e20 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.089+0000 7fab29b5c640 1 -- 172.21.15.33:0/1853048998 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab241a4ff0 con 0x7fab24100e20 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.091+0000 7fab217fa640 1 -- 172.21.15.33:0/1853048998 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fab14017070 con 0x7fab24100e20 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.091+0000 7fab2bf35740 1 Processor -- start 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab2bf35740 1 -- start start 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab2bf35740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e89b40 0x563623e88090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab2bf35740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e885d0 0x563623e88a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab2bf35740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e99b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab2bf35740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563623dee2f0 con 0x563623e97780 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab2bf35740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x563623d95cf0 con 0x563623e885d0 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab2bf35740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563623dea2b0 con 0x563623e89b40 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e99b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e99b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60700/0 (socket says 172.21.15.33:60700) 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab1bfff640 1 -- 172.21.15.33:0/2528892283 learned_addr learned my addr 172.21.15.33:0/2528892283 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab217fa640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab100a3240 0x7fab100a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab237fe640 1 --2- 172.21.15.33:0/2528892283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e885d0 0x563623e88a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab23fff640 1 --2- 172.21.15.33:0/2528892283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e89b40 0x563623e88090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab23fff640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab100a3240 0x7fab100a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab1bfff640 1 -- 172.21.15.33:0/2528892283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e89b40 msgr2=0x563623e88090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab1bfff640 1 --2- 172.21.15.33:0/2528892283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e89b40 0x563623e88090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab1bfff640 1 -- 172.21.15.33:0/2528892283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e885d0 msgr2=0x563623e88a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:12.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab1bfff640 1 --2- 172.21.15.33:0/2528892283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e885d0 0x563623e88a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab1bfff640 1 -- 172.21.15.33:0/2528892283 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563623d7a3a0 con 0x563623e97780 2024-09-16T00:33:12.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab217fa640 1 -- 172.21.15.33:0/1853048998 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fab140cf1c0 con 0x7fab24100e20 2024-09-16T00:33:12.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.092+0000 7fab237fe640 1 --2- 172.21.15.33:0/2528892283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e885d0 0x563623e88a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:33:12.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.093+0000 7fab1bfff640 1 --2- 172.21.15.33:0/2528892283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e99b70 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00bd30 tx=0x7fab0c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee14fb958efb8b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:12.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.093+0000 7fab23fff640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab100a3240 0x7fab100a5700 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fab1c005e30 tx=0x7fab1c005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:12.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.093+0000 7fab19ffb640 1 -- 172.21.15.33:0/2528892283 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c015070 con 0x563623e97780 2024-09-16T00:33:12.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.093+0000 7fab19ffb640 1 -- 172.21.15.33:0/2528892283 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab0c00be20 con 0x563623e97780 2024-09-16T00:33:12.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.093+0000 7fab19ffb640 1 -- 172.21.15.33:0/2528892283 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c011a10 con 0x563623e97780 2024-09-16T00:33:12.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.093+0000 7fab2bf35740 1 -- 172.21.15.33:0/2528892283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 msgr2=0x563623e99b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:12.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.093+0000 7fab2bf35740 1 --2- 172.21.15.33:0/2528892283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e99b70 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00bd30 tx=0x7fab0c00f730 comp rx=0 tx=0).stop 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.093+0000 7fab2bf35740 1 -- 172.21.15.33:0/2528892283 shutdown_connections 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.093+0000 7fab2bf35740 1 --2- 172.21.15.33:0/2528892283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e99b70 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.093+0000 7fab2bf35740 1 --2- 172.21.15.33:0/2528892283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e885d0 0x563623e88a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.093+0000 7fab2bf35740 1 --2- 172.21.15.33:0/2528892283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e89b40 0x563623e88090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.093+0000 7fab2bf35740 1 -- 172.21.15.33:0/2528892283 >> 172.21.15.33:0/2528892283 conn(0x563623e51310 msgr2=0x563623e84c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.094+0000 7fab2bf35740 1 -- 172.21.15.33:0/2528892283 shutdown_connections 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.094+0000 7fab2bf35740 1 -- 172.21.15.33:0/2528892283 wait complete. 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.094+0000 7fab2bf35740 1 -- start start 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.094+0000 7fab2bf35740 1 Processor -- start 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.094+0000 7fab2bf35740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e885d0 0x563623e970e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.094+0000 7fab2bf35740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e89b40 0x563623e94390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.094+0000 7fab2bf35740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e948f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.094+0000 7fab2bf35740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563623e9a480 con 0x563623e97780 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.094+0000 7fab2bf35740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x563623da9060 con 0x563623e885d0 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.094+0000 7fab2bf35740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563623e9a600 con 0x563623e89b40 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab2bf35740 10 client.0 _ll_register_callbacks cb 0x563623afc8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab23fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab23fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e948f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60704/0 (socket says 172.21.15.33:60704) 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab23fff640 1 -- 172.21.15.33:0/7539593 learned_addr learned my addr 172.21.15.33:0/7539593 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab1bfff640 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e89b40 0x563623e94390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab237fe640 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e885d0 0x563623e970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab23fff640 1 -- 172.21.15.33:0/7539593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e89b40 msgr2=0x563623e94390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab23fff640 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e89b40 0x563623e94390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab23fff640 1 -- 172.21.15.33:0/7539593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e885d0 msgr2=0x563623e970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab23fff640 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e885d0 0x563623e970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab23fff640 1 -- 172.21.15.33:0/7539593 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563623b5f3c0 con 0x563623e97780 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab23fff640 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e948f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fab1c02f8c0 tx=0x7fab1c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1eb9baa1ec6548b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab02ffd640 1 -- 172.21.15.33:0/7539593 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c04d030 con 0x563623e97780 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab02ffd640 1 -- 172.21.15.33:0/7539593 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fab1c004310 con 0x563623e97780 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.095+0000 7fab2bf35740 1 -- 172.21.15.33:0/7539593 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab0c00be20 con 0x563623e97780 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.096+0000 7fab02ffd640 1 -- 172.21.15.33:0/7539593 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fab1c034040 con 0x563623e97780 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.096+0000 7fab2bf35740 10 client.18438 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.096+0000 7fab2bf35740 1 -- 172.21.15.33:0/7539593 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563623ea5e30 con 0x563623e97780 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.096+0000 7fab02ffd640 1 -- 172.21.15.33:0/7539593 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab0c00be20 con 0x563623e97780 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.096+0000 7fab02ffd640 1 -- 172.21.15.33:0/7539593 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fab1c048040 con 0x563623e97780 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.096+0000 7fab2bf35740 10 client.18438 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.096+0000 7fab2bf35740 10 client.18438 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:33:12.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.096+0000 7fab2bf35740 20 client.18438 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.096+0000 7fab2bf35740 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x563623ea72f0 0x563623ea9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.096+0000 7fab2bf35740 4 client.18438 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:33:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.096+0000 7fab2bf35740 1 -- 172.21.15.33:0/7539593 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563623dfb860 con 0x563623ea72f0 2024-09-16T00:33:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.096+0000 7fab237fe640 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x563623ea72f0 0x563623ea9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.097+0000 7fab02ffd640 10 client.18438 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:33:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.097+0000 7fab237fe640 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x563623ea72f0 0x563623ea9790 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4027e1a4f3ffaf74 server_cookie=f07c8dfcc0ad4af3 in_seq=0 out_seq=0 2024-09-16T00:33:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.097+0000 7fab02ffd640 1 -- 172.21.15.33:0/7539593 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563623dfb860 con 0x563623ea72f0 2024-09-16T00:33:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.097+0000 7fab02ffd640 10 client.18438 handle_command_reply: tid=0 2024-09-16T00:33:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.097+0000 7fab2bf35740 1 client.18438 shutdown 2024-09-16T00:33:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.098+0000 7fab2bf35740 1 -- 172.21.15.33:0/7539593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 msgr2=0x563623e948f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.098+0000 7fab2bf35740 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e948f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fab1c02f8c0 tx=0x7fab1c0095c0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.098+0000 7fab2bf35740 1 -- 172.21.15.33:0/7539593 shutdown_connections 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.098+0000 7fab2bf35740 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e948f0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.098+0000 7fab2bf35740 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e89b40 0x563623e94390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.098+0000 7fab2bf35740 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x563623ea72f0 0x563623ea9790 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.098+0000 7fab2bf35740 1 --2- 172.21.15.33:0/7539593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e885d0 0x563623e970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.098+0000 7fab2bf35740 1 -- 172.21.15.33:0/7539593 >> 172.21.15.33:0/7539593 conn(0x563623e51310 msgr2=0x563623e8a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.098+0000 7fab02ffd640 0 client.18438 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.098+0000 7fab2bf35740 1 -- 172.21.15.33:0/7539593 shutdown_connections 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.098+0000 7fab2bf35740 1 -- 172.21.15.33:0/7539593 wait complete. 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.098+0000 7fab2bf35740 20 client.18438 trim_cache size 0 max 16384 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab2bf35740 1 Processor -- start 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab2bf35740 1 -- start start 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab2bf35740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e885d0 0x563623ea3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab2bf35740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e89b40 0x563623e92c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab2bf35740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e97780 0x563623e931a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab2bf35740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab241af680 con 0x563623e89b40 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab2bf35740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fab241acef0 con 0x563623e97780 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab2bf35740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fab241aad40 con 0x563623e885d0 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e89b40 0x563623e92c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e89b40 0x563623e92c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60710/0 (socket says 172.21.15.33:60710) 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab1bfff640 1 -- 172.21.15.33:0/138195917 learned_addr learned my addr 172.21.15.33:0/138195917 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab237fe640 1 --2- 172.21.15.33:0/138195917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e885d0 0x563623ea3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab1bfff640 1 -- 172.21.15.33:0/138195917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e885d0 msgr2=0x563623ea3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab1bfff640 1 --2- 172.21.15.33:0/138195917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e885d0 0x563623ea3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab1bfff640 1 -- 172.21.15.33:0/138195917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e97780 msgr2=0x563623e931a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab1bfff640 1 --2- 172.21.15.33:0/138195917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e97780 0x563623e931a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.111+0000 7fab1bfff640 1 -- 172.21.15.33:0/138195917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563623e8b120 con 0x563623e89b40 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.112+0000 7fab1bfff640 1 --2- 172.21.15.33:0/138195917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e89b40 0x563623e92c60 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00b410 tx=0x7fab0c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b347caf70f9fd57c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.112+0000 7fab197fa640 1 -- 172.21.15.33:0/138195917 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c029070 con 0x563623e89b40 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.112+0000 7fab197fa640 1 -- 172.21.15.33:0/138195917 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab0c004680 con 0x563623e89b40 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.112+0000 7fab197fa640 1 -- 172.21.15.33:0/138195917 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c015040 con 0x563623e89b40 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.112+0000 7fab2bf35740 1 -- 172.21.15.33:0/138195917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e89b40 msgr2=0x563623e92c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.112+0000 7fab2bf35740 1 --2- 172.21.15.33:0/138195917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e89b40 0x563623e92c60 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00b410 tx=0x7fab0c00b8f0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.112+0000 7fab2bf35740 1 -- 172.21.15.33:0/138195917 shutdown_connections 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.112+0000 7fab2bf35740 1 --2- 172.21.15.33:0/138195917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e97780 0x563623e931a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.112+0000 7fab2bf35740 1 --2- 172.21.15.33:0/138195917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e89b40 0x563623e92c60 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.112+0000 7fab2bf35740 1 --2- 172.21.15.33:0/138195917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e885d0 0x563623ea3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.112+0000 7fab2bf35740 1 -- 172.21.15.33:0/138195917 >> 172.21.15.33:0/138195917 conn(0x563623e51310 msgr2=0x563623e94350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.113+0000 7fab2bf35740 1 -- 172.21.15.33:0/138195917 shutdown_connections 2024-09-16T00:33:12.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.113+0000 7fab2bf35740 1 -- 172.21.15.33:0/138195917 wait complete. 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.113+0000 7fab2bf35740 1 -- start start 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.113+0000 7fab2bf35740 1 Processor -- start 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.113+0000 7fab2bf35740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e885d0 0x563623e9fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.113+0000 7fab2bf35740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e89b40 0x563623ea0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.113+0000 7fab2bf35740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e9b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.113+0000 7fab2bf35740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab241ab3b0 con 0x563623e97780 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.113+0000 7fab2bf35740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fab241ac820 con 0x563623e89b40 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.113+0000 7fab2bf35740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fab241aba80 con 0x563623e885d0 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.113+0000 7fab2bf35740 10 client.0 _ll_register_callbacks cb 0x7fab1c02f7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab23fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e9b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab23fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e9b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60714/0 (socket says 172.21.15.33:60714) 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab23fff640 1 -- 172.21.15.33:0/3940138247 learned_addr learned my addr 172.21.15.33:0/3940138247 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:12.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab237fe640 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e885d0 0x563623e9fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab1bfff640 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e89b40 0x563623ea0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab23fff640 1 -- 172.21.15.33:0/3940138247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e885d0 msgr2=0x563623e9fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab23fff640 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e885d0 0x563623e9fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab23fff640 1 -- 172.21.15.33:0/3940138247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e89b40 msgr2=0x563623ea0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab23fff640 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e89b40 0x563623ea0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab23fff640 1 -- 172.21.15.33:0/3940138247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563623e9ba40 con 0x563623e97780 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab1bfff640 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e89b40 0x563623ea0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab23fff640 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e9b210 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fab1c02ae70 tx=0x7fab1c03ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573493439c8f7c0f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab02ffd640 1 -- 172.21.15.33:0/3940138247 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c04d070 con 0x563623e97780 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab02ffd640 1 -- 172.21.15.33:0/3940138247 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fab1c004510 con 0x563623e97780 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab2bf35740 1 -- 172.21.15.33:0/3940138247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab0c004680 con 0x563623e97780 2024-09-16T00:33:12.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab02ffd640 1 -- 172.21.15.33:0/3940138247 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fab1c02f930 con 0x563623e97780 2024-09-16T00:33:12.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab2bf35740 10 client.18444 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:12.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.114+0000 7fab2bf35740 1 -- 172.21.15.33:0/3940138247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563623f11ce0 con 0x563623e97780 2024-09-16T00:33:12.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.115+0000 7fab02ffd640 1 -- 172.21.15.33:0/3940138247 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab0c004680 con 0x563623e97780 2024-09-16T00:33:12.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.115+0000 7fab02ffd640 1 -- 172.21.15.33:0/3940138247 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fab1c048050 con 0x563623e97780 2024-09-16T00:33:12.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.115+0000 7fab2bf35740 10 client.18444 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:12.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.115+0000 7fab2bf35740 10 client.18444 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:33:12.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.115+0000 7fab2bf35740 20 client.18444 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:12.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.115+0000 7fab2bf35740 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x563623f12480 0x563623f148a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:12.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.115+0000 7fab2bf35740 4 client.18444 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:33:12.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.115+0000 7fab2bf35740 1 -- 172.21.15.33:0/3940138247 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563623df5d70 con 0x563623f12480 2024-09-16T00:33:12.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.115+0000 7fab237fe640 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x563623f12480 0x563623f148a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:12.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab02ffd640 10 client.18444 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:33:12.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab237fe640 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x563623f12480 0x563623f148a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a3433bf2191ede9c server_cookie=b27e0f6e9a58c5b8 in_seq=0 out_seq=0 2024-09-16T00:33:12.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab02ffd640 1 -- 172.21.15.33:0/3940138247 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563623df5d70 con 0x563623f12480 2024-09-16T00:33:12.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab02ffd640 10 client.18444 handle_command_reply: tid=0 2024-09-16T00:33:12.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab2bf35740 1 client.18444 shutdown 2024-09-16T00:33:12.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab2bf35740 1 -- 172.21.15.33:0/3940138247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 msgr2=0x563623e9b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:12.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab2bf35740 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e9b210 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fab1c02ae70 tx=0x7fab1c03ab60 comp rx=0 tx=0).stop 2024-09-16T00:33:12.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab2bf35740 1 -- 172.21.15.33:0/3940138247 shutdown_connections 2024-09-16T00:33:12.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab2bf35740 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563623e97780 0x563623e9b210 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab2bf35740 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x563623f12480 0x563623f148a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab2bf35740 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563623e89b40 0x563623ea0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab2bf35740 1 --2- 172.21.15.33:0/3940138247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563623e885d0 0x563623e9fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab2bf35740 1 -- 172.21.15.33:0/3940138247 >> 172.21.15.33:0/3940138247 conn(0x563623e51310 msgr2=0x563623e92ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab02ffd640 0 client.18444 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:33:12.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab2bf35740 1 -- 172.21.15.33:0/3940138247 shutdown_connections 2024-09-16T00:33:12.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab2bf35740 1 -- 172.21.15.33:0/3940138247 wait complete. 2024-09-16T00:33:12.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.116+0000 7fab2bf35740 20 client.18444 trim_cache size 0 max 16384 2024-09-16T00:33:12.138 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:33:12.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.117+0000 7fab29b5c640 1 -- 172.21.15.33:0/1853048998 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab100a3240 msgr2=0x7fab100a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:12.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.117+0000 7fab29b5c640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab100a3240 0x7fab100a5700 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fab1c005e30 tx=0x7fab1c005c50 comp rx=0 tx=0).stop 2024-09-16T00:33:12.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.117+0000 7fab29b5c640 1 -- 172.21.15.33:0/1853048998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab24100e20 msgr2=0x7fab24199350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:12.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.117+0000 7fab29b5c640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab24100e20 0x7fab24199350 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fab1400b4a0 tx=0x7fab1400b960 comp rx=0 tx=0).stop 2024-09-16T00:33:12.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.117+0000 7fab29b5c640 1 -- 172.21.15.33:0/1853048998 shutdown_connections 2024-09-16T00:33:12.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.117+0000 7fab29b5c640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fab100a3240 0x7fab100a5700 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.117+0000 7fab29b5c640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab24102ef0 0x7fab241a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.117+0000 7fab29b5c640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab24102570 0x7fab24199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.117+0000 7fab29b5c640 1 --2- 172.21.15.33:0/1853048998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab24100e20 0x7fab24199350 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:12.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.117+0000 7fab29b5c640 1 -- 172.21.15.33:0/1853048998 >> 172.21.15.33:0/1853048998 conn(0x7fab240fc7d0 msgr2=0x7fab24106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:12.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.117+0000 7fab29b5c640 1 -- 172.21.15.33:0/1853048998 shutdown_connections 2024-09-16T00:33:12.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:12.117+0000 7fab29b5c640 1 -- 172.21.15.33:0/1853048998 wait complete. 2024-09-16T00:33:12.840 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:33:13.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:13 smithi033 ceph-mon[31243]: pgmap v832: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 198 op/s 2024-09-16T00:33:13.073 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 7.46 MB/sec execute 19 sec latency 10703.158 ms 2024-09-16T00:33:13.098 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:13.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:13 smithi083 ceph-mon[35242]: pgmap v832: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 198 op/s 2024-09-16T00:33:13.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:13 smithi060 ceph-mon[35185]: pgmap v832: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 198 op/s 2024-09-16T00:33:13.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.605+0000 7f82f025a640 1 -- 172.21.15.33:0/2324757136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82e8106f20 msgr2=0x7f82e810eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:13.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.605+0000 7f82f025a640 1 --2- 172.21.15.33:0/2324757136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82e8106f20 0x7f82e810eaa0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009ca0 tx=0x7f82e002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.605+0000 7f82f025a640 1 -- 172.21.15.33:0/2324757136 shutdown_connections 2024-09-16T00:33:13.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.605+0000 7f82f025a640 1 --2- 172.21.15.33:0/2324757136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82e8106f20 0x7f82e810eaa0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.605+0000 7f82f025a640 1 --2- 172.21.15.33:0/2324757136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82e81027a0 0x7f82e81067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.605+0000 7f82f025a640 1 --2- 172.21.15.33:0/2324757136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82e8101df0 0x7f82e81021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.605+0000 7f82f025a640 1 -- 172.21.15.33:0/2324757136 >> 172.21.15.33:0/2324757136 conn(0x7f82e80fd8b0 msgr2=0x7f82e80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:13.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.605+0000 7f82f025a640 1 -- 172.21.15.33:0/2324757136 shutdown_connections 2024-09-16T00:33:13.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.606+0000 7f82f025a640 1 -- 172.21.15.33:0/2324757136 wait complete. 2024-09-16T00:33:13.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.606+0000 7f82f025a640 1 Processor -- start 2024-09-16T00:33:13.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.606+0000 7f82f025a640 1 -- start start 2024-09-16T00:33:13.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.606+0000 7f82f025a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82e8101df0 0x7f82e8198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.606+0000 7f82f025a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82e81027a0 0x7f82e8198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.606+0000 7f82f025a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82e8106f20 0x7f82e819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.606+0000 7f82f025a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f82e8111f80 con 0x7f82e8101df0 2024-09-16T00:33:13.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.606+0000 7f82f025a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f82e8111e00 con 0x7f82e8106f20 2024-09-16T00:33:13.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.606+0000 7f82f025a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f82e8112100 con 0x7f82e81027a0 2024-09-16T00:33:13.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.606+0000 7f82edff6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82e8101df0 0x7f82e8198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.606+0000 7f82edff6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82e8101df0 0x7f82e8198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60740/0 (socket says 172.21.15.33:60740) 2024-09-16T00:33:13.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82edff6640 1 -- 172.21.15.33:0/864673258 learned_addr learned my addr 172.21.15.33:0/864673258 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:13.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82ed7f5640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82e81027a0 0x7f82e8198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82ee7f7640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82e8106f20 0x7f82e819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82edff6640 1 -- 172.21.15.33:0/864673258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82e81027a0 msgr2=0x7f82e8198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:13.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82edff6640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82e81027a0 0x7f82e8198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82edff6640 1 -- 172.21.15.33:0/864673258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82e8106f20 msgr2=0x7f82e819cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:13.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82edff6640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82e8106f20 0x7f82e819cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82edff6640 1 -- 172.21.15.33:0/864673258 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e819d3f0 con 0x7f82e8101df0 2024-09-16T00:33:13.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82ed7f5640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82e81027a0 0x7f82e8198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:33:13.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82edff6640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82e8101df0 0x7f82e8198440 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f82e400d950 tx=0x7f82e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:13.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82deffd640 1 -- 172.21.15.33:0/864673258 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4017030 con 0x7f82e8101df0 2024-09-16T00:33:13.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82deffd640 1 -- 172.21.15.33:0/864673258 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e4004020 con 0x7f82e8101df0 2024-09-16T00:33:13.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82f025a640 1 -- 172.21.15.33:0/864673258 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e819d6e0 con 0x7f82e8101df0 2024-09-16T00:33:13.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82deffd640 1 -- 172.21.15.33:0/864673258 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e40047a0 con 0x7f82e8101df0 2024-09-16T00:33:13.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.607+0000 7f82f025a640 1 -- 172.21.15.33:0/864673258 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e81a4fc0 con 0x7f82e8101df0 2024-09-16T00:33:13.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.610+0000 7f82deffd640 1 -- 172.21.15.33:0/864673258 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f82e40041d0 con 0x7f82e8101df0 2024-09-16T00:33:13.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.610+0000 7f82f2633740 1 Processor -- start 2024-09-16T00:33:13.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.610+0000 7f82f2633740 1 -- start start 2024-09-16T00:33:13.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.610+0000 7f82f2633740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d87b40 0x562be4d86090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.610+0000 7f82f2633740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4d86a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.610+0000 7f82f2633740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4d93760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.610+0000 7f82f2633740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562be4cec2f0 con 0x562be4d865d0 2024-09-16T00:33:13.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.610+0000 7f82f2633740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562be4c93cf0 con 0x562be4d91370 2024-09-16T00:33:13.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.610+0000 7f82f2633740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562be4ce82b0 con 0x562be4d87b40 2024-09-16T00:33:13.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.610+0000 7f82edff6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4d86a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82edff6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4d86a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60742/0 (socket says 172.21.15.33:60742) 2024-09-16T00:33:13.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82edff6640 1 -- 172.21.15.33:0/1879024750 learned_addr learned my addr 172.21.15.33:0/1879024750 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:13.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82ee7f7640 1 --2- 172.21.15.33:0/1879024750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d87b40 0x562be4d86090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82deffd640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f82d40a32c0 0x7f82d40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82ed7f5640 1 --2- 172.21.15.33:0/1879024750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4d93760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82deffd640 1 -- 172.21.15.33:0/864673258 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f82e40cf820 con 0x7f82e8101df0 2024-09-16T00:33:13.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82edff6640 1 -- 172.21.15.33:0/1879024750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d87b40 msgr2=0x562be4d86090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:13.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82edff6640 1 --2- 172.21.15.33:0/1879024750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d87b40 0x562be4d86090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82edff6640 1 -- 172.21.15.33:0/1879024750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 msgr2=0x562be4d93760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:13.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82edff6640 1 --2- 172.21.15.33:0/1879024750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4d93760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82edff6640 1 -- 172.21.15.33:0/1879024750 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562be4c783a0 con 0x562be4d865d0 2024-09-16T00:33:13.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82ed7f5640 1 --2- 172.21.15.33:0/1879024750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4d93760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:33:13.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82edff6640 1 --2- 172.21.15.33:0/1879024750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4d86a10 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f82e409b660 tx=0x7f82e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293fc9e890c30451 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:13.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.611+0000 7f82ee7f7640 1 --2- 172.21.15.33:0/1879024750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d87b40 0x562be4d86090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:13.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.612+0000 7f82ee7f7640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f82d40a32c0 0x7f82d40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.612+0000 7f82d3fff640 1 -- 172.21.15.33:0/1879024750 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4014070 con 0x562be4d865d0 2024-09-16T00:33:13.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.612+0000 7f82d3fff640 1 -- 172.21.15.33:0/1879024750 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e409c060 con 0x562be4d865d0 2024-09-16T00:33:13.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.612+0000 7f82d3fff640 1 -- 172.21.15.33:0/1879024750 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e40a45d0 con 0x562be4d865d0 2024-09-16T00:33:13.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.613+0000 7f82f2633740 1 -- 172.21.15.33:0/1879024750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 msgr2=0x562be4d86a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:13.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.613+0000 7f82ee7f7640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f82d40a32c0 0x7f82d40a5780 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f82e0005cc0 tx=0x7f82e00095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:13.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.613+0000 7f82f2633740 1 --2- 172.21.15.33:0/1879024750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4d86a10 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f82e409b660 tx=0x7f82e40046c0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.613+0000 7f82f2633740 1 -- 172.21.15.33:0/1879024750 shutdown_connections 2024-09-16T00:33:13.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.613+0000 7f82f2633740 1 --2- 172.21.15.33:0/1879024750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4d93760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.613+0000 7f82f2633740 1 --2- 172.21.15.33:0/1879024750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4d86a10 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.613+0000 7f82f2633740 1 --2- 172.21.15.33:0/1879024750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d87b40 0x562be4d86090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.613+0000 7f82f2633740 1 -- 172.21.15.33:0/1879024750 >> 172.21.15.33:0/1879024750 conn(0x562be4d4f310 msgr2=0x562be4d82c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.613+0000 7f82f2633740 1 -- 172.21.15.33:0/1879024750 shutdown_connections 2024-09-16T00:33:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.613+0000 7f82f2633740 1 -- 172.21.15.33:0/1879024750 wait complete. 2024-09-16T00:33:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.613+0000 7f82f2633740 1 -- start start 2024-09-16T00:33:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.613+0000 7f82f2633740 1 Processor -- start 2024-09-16T00:33:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82f2633740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4d90d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82f2633740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d87b40 0x562be4d8e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82f2633740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d91370 0x562be4d8e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82f2633740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562be4d98420 con 0x562be4d865d0 2024-09-16T00:33:13.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82f2633740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562be4ca7060 con 0x562be4d87b40 2024-09-16T00:33:13.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82f2633740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562be4d985a0 con 0x562be4d91370 2024-09-16T00:33:13.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82f2633740 10 client.0 _ll_register_callbacks cb 0x562be49fa8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:13.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82edff6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4d90d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82edff6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4d90d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60752/0 (socket says 172.21.15.33:60752) 2024-09-16T00:33:13.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82edff6640 1 -- 172.21.15.33:0/3336813096 learned_addr learned my addr 172.21.15.33:0/3336813096 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:13.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82ee7f7640 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d91370 0x562be4d8e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82ed7f5640 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d87b40 0x562be4d8e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82edff6640 1 -- 172.21.15.33:0/3336813096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d91370 msgr2=0x562be4d8e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82edff6640 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d91370 0x562be4d8e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82edff6640 1 -- 172.21.15.33:0/3336813096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d87b40 msgr2=0x562be4d8e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82edff6640 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d87b40 0x562be4d8e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.614+0000 7f82edff6640 1 -- 172.21.15.33:0/3336813096 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562be4a5d3c0 con 0x562be4d865d0 2024-09-16T00:33:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82edff6640 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4d90d20 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f82e40860b0 tx=0x7f82e40d5240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a47bf12cf8557b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82d17fa640 1 -- 172.21.15.33:0/3336813096 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4014070 con 0x562be4d865d0 2024-09-16T00:33:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82d17fa640 1 -- 172.21.15.33:0/3336813096 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f82e4097fd0 con 0x562be4d865d0 2024-09-16T00:33:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82f2633740 1 -- 172.21.15.33:0/3336813096 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82e409c060 con 0x562be4d865d0 2024-09-16T00:33:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82d17fa640 1 -- 172.21.15.33:0/3336813096 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f82e40a45d0 con 0x562be4d865d0 2024-09-16T00:33:13.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82f2633740 10 client.18456 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:13.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82f2633740 1 -- 172.21.15.33:0/3336813096 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562be4da3eb0 con 0x562be4d865d0 2024-09-16T00:33:13.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82d17fa640 1 -- 172.21.15.33:0/3336813096 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e40cf3f0 con 0x562be4d865d0 2024-09-16T00:33:13.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82d17fa640 1 -- 172.21.15.33:0/3336813096 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f82e40d1ce0 con 0x562be4d865d0 2024-09-16T00:33:13.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82f2633740 10 client.18456 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:13.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82f2633740 10 client.18456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:33:13.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82f2633740 20 client.18456 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:13.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82f2633740 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562be4da5200 0x562be4da75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82f2633740 4 client.18456 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:33:13.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.615+0000 7f82f2633740 1 -- 172.21.15.33:0/3336813096 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f82e409c060 con 0x562be4da5200 2024-09-16T00:33:13.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.616+0000 7f82ed7f5640 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562be4da5200 0x562be4da75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.616+0000 7f82ed7f5640 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562be4da5200 0x562be4da75f0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cde4897e766eddc0 server_cookie=be5b456d3aebea9a in_seq=0 out_seq=0 2024-09-16T00:33:13.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.616+0000 7f82d17fa640 10 client.18456 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:13.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82d17fa640 1 -- 172.21.15.33:0/3336813096 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f82e409c060 con 0x562be4da5200 2024-09-16T00:33:13.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82d17fa640 10 client.18456 handle_command_reply: tid=0 2024-09-16T00:33:13.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82f2633740 1 client.18456 shutdown 2024-09-16T00:33:13.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82f2633740 1 -- 172.21.15.33:0/3336813096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 msgr2=0x562be4d90d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:13.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82f2633740 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4d90d20 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f82e40860b0 tx=0x7f82e40d5240 comp rx=0 tx=0).stop 2024-09-16T00:33:13.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82f2633740 1 -- 172.21.15.33:0/3336813096 shutdown_connections 2024-09-16T00:33:13.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82f2633740 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562be4da5200 0x562be4da75f0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82f2633740 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d91370 0x562be4d8e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82d17fa640 0 client.18456 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:13.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82f2633740 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d87b40 0x562be4d8e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82f2633740 1 --2- 172.21.15.33:0/3336813096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4d90d20 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82f2633740 1 -- 172.21.15.33:0/3336813096 >> 172.21.15.33:0/3336813096 conn(0x562be4d4f310 msgr2=0x562be4d88990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:13.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82f2633740 1 -- 172.21.15.33:0/3336813096 shutdown_connections 2024-09-16T00:33:13.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.617+0000 7f82f2633740 1 -- 172.21.15.33:0/3336813096 wait complete. 2024-09-16T00:33:13.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.618+0000 7f82f2633740 20 client.18456 trim_cache size 0 max 16384 2024-09-16T00:33:13.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.631+0000 7f82f2633740 1 Processor -- start 2024-09-16T00:33:13.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.631+0000 7f82f2633740 1 -- start start 2024-09-16T00:33:13.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.631+0000 7f82f2633740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4da1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.631+0000 7f82f2633740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d87b40 0x562be4d898c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.631+0000 7f82f2633740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4d88820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.631+0000 7f82f2633740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81af7b0 con 0x562be4d865d0 2024-09-16T00:33:13.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82f2633740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81ad020 con 0x562be4d91370 2024-09-16T00:33:13.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82f2633740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81aae70 con 0x562be4d87b40 2024-09-16T00:33:13.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82edff6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4da1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82edff6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4da1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60756/0 (socket says 172.21.15.33:60756) 2024-09-16T00:33:13.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82edff6640 1 -- 172.21.15.33:0/402216703 learned_addr learned my addr 172.21.15.33:0/402216703 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82ed7f5640 1 --2- 172.21.15.33:0/402216703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d87b40 0x562be4d898c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82ee7f7640 1 --2- 172.21.15.33:0/402216703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4d88820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82edff6640 1 -- 172.21.15.33:0/402216703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d87b40 msgr2=0x562be4d898c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82edff6640 1 --2- 172.21.15.33:0/402216703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d87b40 0x562be4d898c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82edff6640 1 -- 172.21.15.33:0/402216703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 msgr2=0x562be4d88820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82edff6640 1 --2- 172.21.15.33:0/402216703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4d88820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82edff6640 1 -- 172.21.15.33:0/402216703 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562be4d824f0 con 0x562be4d865d0 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82ee7f7640 1 --2- 172.21.15.33:0/402216703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4d88820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82ed7f5640 1 --2- 172.21.15.33:0/402216703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d87b40 0x562be4d898c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82edff6640 1 --2- 172.21.15.33:0/402216703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4da1840 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f82e409b8f0 tx=0x7f82e400b530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31abfc8d5df41fa7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82d37fe640 1 -- 172.21.15.33:0/402216703 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4014070 con 0x562be4d865d0 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82d37fe640 1 -- 172.21.15.33:0/402216703 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e409c060 con 0x562be4d865d0 2024-09-16T00:33:13.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.632+0000 7f82d37fe640 1 -- 172.21.15.33:0/402216703 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e409af00 con 0x562be4d865d0 2024-09-16T00:33:13.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.633+0000 7f82f2633740 1 -- 172.21.15.33:0/402216703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 msgr2=0x562be4da1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:13.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.633+0000 7f82f2633740 1 --2- 172.21.15.33:0/402216703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4da1840 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f82e409b8f0 tx=0x7f82e400b530 comp rx=0 tx=0).stop 2024-09-16T00:33:13.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.633+0000 7f82f2633740 1 -- 172.21.15.33:0/402216703 shutdown_connections 2024-09-16T00:33:13.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.633+0000 7f82f2633740 1 --2- 172.21.15.33:0/402216703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4d88820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.633+0000 7f82f2633740 1 --2- 172.21.15.33:0/402216703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d87b40 0x562be4d898c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.633+0000 7f82f2633740 1 --2- 172.21.15.33:0/402216703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d865d0 0x562be4da1840 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.633+0000 7f82f2633740 1 -- 172.21.15.33:0/402216703 >> 172.21.15.33:0/402216703 conn(0x562be4d4f310 msgr2=0x562be4d88110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:13.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.633+0000 7f82f2633740 1 -- 172.21.15.33:0/402216703 shutdown_connections 2024-09-16T00:33:13.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.633+0000 7f82f2633740 1 -- 172.21.15.33:0/402216703 wait complete. 2024-09-16T00:33:13.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.633+0000 7f82f2633740 1 -- start start 2024-09-16T00:33:13.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.633+0000 7f82f2633740 1 Processor -- start 2024-09-16T00:33:13.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82f2633740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d865d0 0x562be4da8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82f2633740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d87b40 0x562be4da8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82f2633740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4da9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82f2633740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81ab4e0 con 0x562be4d87b40 2024-09-16T00:33:13.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82f2633740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81ac950 con 0x562be4d91370 2024-09-16T00:33:13.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82f2633740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81abbb0 con 0x562be4d865d0 2024-09-16T00:33:13.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82f2633740 10 client.0 _ll_register_callbacks cb 0x7f82e409b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:13.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82ed7f5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d87b40 0x562be4da8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82ed7f5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d87b40 0x562be4da8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60762/0 (socket says 172.21.15.33:60762) 2024-09-16T00:33:13.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82ed7f5640 1 -- 172.21.15.33:0/1946663365 learned_addr learned my addr 172.21.15.33:0/1946663365 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:13.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82edff6640 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d865d0 0x562be4da8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82ee7f7640 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4da9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82ed7f5640 1 -- 172.21.15.33:0/1946663365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d865d0 msgr2=0x562be4da8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:13.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82ed7f5640 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d865d0 0x562be4da8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82ed7f5640 1 -- 172.21.15.33:0/1946663365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 msgr2=0x562be4da9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:13.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82ed7f5640 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4da9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.634+0000 7f82ed7f5640 1 -- 172.21.15.33:0/1946663365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562be4e0ed90 con 0x562be4d87b40 2024-09-16T00:33:13.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82ed7f5640 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d87b40 0x562be4da8b40 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f82d8005d80 tx=0x7f82d8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfcf6e6c977a8473 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:13.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:13.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82d17fa640 1 -- 172.21.15.33:0/1946663365 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d801b710 con 0x562be4d87b40 2024-09-16T00:33:13.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82f2633740 1 -- 172.21.15.33:0/1946663365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82e409c060 con 0x562be4d87b40 2024-09-16T00:33:13.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82d17fa640 1 -- 172.21.15.33:0/1946663365 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f82d801b8b0 con 0x562be4d87b40 2024-09-16T00:33:13.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82d17fa640 1 -- 172.21.15.33:0/1946663365 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f82d802d050 con 0x562be4d87b40 2024-09-16T00:33:13.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82f2633740 10 client.18462 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:13.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82f2633740 1 -- 172.21.15.33:0/1946663365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562be4e10060 con 0x562be4d87b40 2024-09-16T00:33:13.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82d17fa640 1 -- 172.21.15.33:0/1946663365 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e409c060 con 0x562be4d87b40 2024-09-16T00:33:13.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82d17fa640 1 -- 172.21.15.33:0/1946663365 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f82d8024c50 con 0x562be4d87b40 2024-09-16T00:33:13.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82f2633740 10 client.18462 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:13.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82f2633740 10 client.18462 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:33:13.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82f2633740 20 client.18462 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:13.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82f2633740 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562be4e10810 0x562be4e12c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:13.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82f2633740 4 client.18462 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:33:13.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.635+0000 7f82f2633740 1 -- 172.21.15.33:0/1946663365 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562be4cf9860 con 0x562be4e10810 2024-09-16T00:33:13.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82edff6640 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562be4e10810 0x562be4e12c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:13.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82edff6640 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562be4e10810 0x562be4e12c30 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40b687231beb4218 server_cookie=b1fe3ad10b98b72e in_seq=0 out_seq=0 2024-09-16T00:33:13.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82d17fa640 10 client.18462 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:13.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82d17fa640 1 -- 172.21.15.33:0/1946663365 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562be4cf9860 con 0x562be4e10810 2024-09-16T00:33:13.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82d17fa640 10 client.18462 handle_command_reply: tid=0 2024-09-16T00:33:13.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82f2633740 1 client.18462 shutdown 2024-09-16T00:33:13.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82f2633740 1 -- 172.21.15.33:0/1946663365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d87b40 msgr2=0x562be4da8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:13.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82f2633740 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d87b40 0x562be4da8b40 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f82d8005d80 tx=0x7f82d8011810 comp rx=0 tx=0).stop 2024-09-16T00:33:13.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82f2633740 1 -- 172.21.15.33:0/1946663365 shutdown_connections 2024-09-16T00:33:13.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82f2633740 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562be4e10810 0x562be4e12c30 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82f2633740 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562be4d91370 0x562be4da9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82d17fa640 0 client.18462 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:13.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82f2633740 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562be4d87b40 0x562be4da8b40 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.636+0000 7f82f2633740 1 --2- 172.21.15.33:0/1946663365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562be4d865d0 0x562be4da8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f2633740 1 -- 172.21.15.33:0/1946663365 >> 172.21.15.33:0/1946663365 conn(0x562be4d4f310 msgr2=0x562be4d8f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:13.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f2633740 1 -- 172.21.15.33:0/1946663365 shutdown_connections 2024-09-16T00:33:13.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f2633740 1 -- 172.21.15.33:0/1946663365 wait complete. 2024-09-16T00:33:13.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f2633740 20 client.18462 trim_cache size 0 max 16384 2024-09-16T00:33:13.650 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:33:13.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f025a640 1 -- 172.21.15.33:0/864673258 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f82d40a32c0 msgr2=0x7f82d40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:13.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f025a640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f82d40a32c0 0x7f82d40a5780 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f82e0005cc0 tx=0x7f82e00095c0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f025a640 1 -- 172.21.15.33:0/864673258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82e8101df0 msgr2=0x7f82e8198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:13.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f025a640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82e8101df0 0x7f82e8198440 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f82e400d950 tx=0x7f82e400de10 comp rx=0 tx=0).stop 2024-09-16T00:33:13.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f025a640 1 -- 172.21.15.33:0/864673258 shutdown_connections 2024-09-16T00:33:13.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f025a640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f82d40a32c0 0x7f82d40a5780 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f025a640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82e8106f20 0x7f82e819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f025a640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82e81027a0 0x7f82e8198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f025a640 1 --2- 172.21.15.33:0/864673258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82e8101df0 0x7f82e8198440 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:13.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f025a640 1 -- 172.21.15.33:0/864673258 >> 172.21.15.33:0/864673258 conn(0x7f82e80fd8b0 msgr2=0x7f82e80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:13.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f025a640 1 -- 172.21.15.33:0/864673258 shutdown_connections 2024-09-16T00:33:13.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:13.637+0000 7f82f025a640 1 -- 172.21.15.33:0/864673258 wait complete. 2024-09-16T00:33:14.073 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 7.09 MB/sec execute 20 sec latency 11703.236 ms 2024-09-16T00:33:15.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 6.75 MB/sec execute 21 sec latency 12703.342 ms 2024-09-16T00:33:15.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:15 smithi033 ceph-mon[31243]: pgmap v833: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.2 MiB/s wr, 190 op/s 2024-09-16T00:33:15.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:15 smithi083 ceph-mon[35242]: pgmap v833: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.2 MiB/s wr, 190 op/s 2024-09-16T00:33:15.447 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:33:15.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:33:15.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:15 smithi060 ceph-mon[35185]: pgmap v833: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.2 MiB/s wr, 190 op/s 2024-09-16T00:33:15.718 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:16.073 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 6.44 MB/sec execute 22 sec latency 13703.387 ms 2024-09-16T00:33:16.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.234+0000 7fa04424c640 1 -- 172.21.15.33:0/3114151555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa03c1027a0 msgr2=0x7fa03c1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:16.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.234+0000 7fa04424c640 1 --2- 172.21.15.33:0/3114151555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa03c1027a0 0x7fa03c1067f0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fa02c009900 tx=0x7fa02c02a410 comp rx=0 tx=0).stop 2024-09-16T00:33:16.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.234+0000 7fa04424c640 1 -- 172.21.15.33:0/3114151555 shutdown_connections 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.234+0000 7fa04424c640 1 --2- 172.21.15.33:0/3114151555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa03c106f20 0x7fa03c10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.234+0000 7fa04424c640 1 --2- 172.21.15.33:0/3114151555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa03c1027a0 0x7fa03c1067f0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.234+0000 7fa04424c640 1 --2- 172.21.15.33:0/3114151555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa03c101df0 0x7fa03c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.234+0000 7fa04424c640 1 -- 172.21.15.33:0/3114151555 >> 172.21.15.33:0/3114151555 conn(0x7fa03c0fd8b0 msgr2=0x7fa03c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.234+0000 7fa04424c640 1 -- 172.21.15.33:0/3114151555 shutdown_connections 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.234+0000 7fa04424c640 1 -- 172.21.15.33:0/3114151555 wait complete. 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa04424c640 1 Processor -- start 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa04424c640 1 -- start start 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa04424c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa03c101df0 0x7fa03c19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa04424c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa03c1027a0 0x7fa03c19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa04424c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa03c106f20 0x7fa03c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa04424c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c111f80 con 0x7fa03c101df0 2024-09-16T00:33:16.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa04424c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c111e00 con 0x7fa03c106f20 2024-09-16T00:33:16.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa04424c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c112100 con 0x7fa03c1027a0 2024-09-16T00:33:16.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa041fe8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa03c101df0 0x7fa03c19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa041fe8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa03c101df0 0x7fa03c19a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60778/0 (socket says 172.21.15.33:60778) 2024-09-16T00:33:16.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa041fe8640 1 -- 172.21.15.33:0/3368817942 learned_addr learned my addr 172.21.15.33:0/3368817942 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:16.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa0417e7640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa03c1027a0 0x7fa03c19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.235+0000 7fa0427e9640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa03c106f20 0x7fa03c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.236+0000 7fa041fe8640 1 -- 172.21.15.33:0/3368817942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa03c1027a0 msgr2=0x7fa03c19ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:16.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.236+0000 7fa041fe8640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa03c1027a0 0x7fa03c19ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.236+0000 7fa041fe8640 1 -- 172.21.15.33:0/3368817942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa03c106f20 msgr2=0x7fa03c19ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:16.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.236+0000 7fa041fe8640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa03c106f20 0x7fa03c19ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.236+0000 7fa041fe8640 1 -- 172.21.15.33:0/3368817942 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa03c19f5f0 con 0x7fa03c101df0 2024-09-16T00:33:16.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.236+0000 7fa041fe8640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa03c101df0 0x7fa03c19a640 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fa03800d770 tx=0x7fa03800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:16.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.236+0000 7fa032ffd640 1 -- 172.21.15.33:0/3368817942 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038017070 con 0x7fa03c101df0 2024-09-16T00:33:16.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.236+0000 7fa04424c640 1 -- 172.21.15.33:0/3368817942 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa03c19f8e0 con 0x7fa03c101df0 2024-09-16T00:33:16.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.236+0000 7fa032ffd640 1 -- 172.21.15.33:0/3368817942 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa038005500 con 0x7fa03c101df0 2024-09-16T00:33:16.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.236+0000 7fa032ffd640 1 -- 172.21.15.33:0/3368817942 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038004e00 con 0x7fa03c101df0 2024-09-16T00:33:16.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.236+0000 7fa04424c640 1 -- 172.21.15.33:0/3368817942 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa03c1a71c0 con 0x7fa03c101df0 2024-09-16T00:33:16.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.239+0000 7fa046625740 1 Processor -- start 2024-09-16T00:33:16.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.239+0000 7fa032ffd640 1 -- 172.21.15.33:0/3368817942 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa038004fa0 con 0x7fa03c101df0 2024-09-16T00:33:16.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.239+0000 7fa046625740 1 -- start start 2024-09-16T00:33:16.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa046625740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574d0b60 0x5561574d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa046625740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 0x5561574db800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa046625740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574dbd40 0x5561574de130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa046625740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556157436280 con 0x5561574d1480 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa046625740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5561573ddb10 con 0x5561574dbd40 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa046625740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556157432240 con 0x5561574d0b60 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa041fe8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 0x5561574db800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa041fe8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 0x5561574db800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60786/0 (socket says 172.21.15.33:60786) 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa041fe8640 1 -- 172.21.15.33:0/2246271202 learned_addr learned my addr 172.21.15.33:0/2246271202 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa0417e7640 1 --2- 172.21.15.33:0/2246271202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574dbd40 0x5561574de130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa0427e9640 1 --2- 172.21.15.33:0/2246271202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574d0b60 0x5561574d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa041fe8640 1 -- 172.21.15.33:0/2246271202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574d0b60 msgr2=0x5561574d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa041fe8640 1 --2- 172.21.15.33:0/2246271202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574d0b60 0x5561574d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa041fe8640 1 -- 172.21.15.33:0/2246271202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574dbd40 msgr2=0x5561574de130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa041fe8640 1 --2- 172.21.15.33:0/2246271202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574dbd40 0x5561574de130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa041fe8640 1 -- 172.21.15.33:0/2246271202 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561573ea1f0 con 0x5561574d1480 2024-09-16T00:33:16.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa032ffd640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa0280a3240 0x7fa0280a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.240+0000 7fa041fe8640 1 --2- 172.21.15.33:0/2246271202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 0x5561574db800 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fa0380d5c10 tx=0x7fa0380d7690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0f7755a48b4b70 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:16.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa0427e9640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa0280a3240 0x7fa0280a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa032ffd640 1 -- 172.21.15.33:0/3368817942 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa0380cf710 con 0x7fa03c101df0 2024-09-16T00:33:16.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa027fff640 1 -- 172.21.15.33:0/2246271202 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038097c00 con 0x5561574d1480 2024-09-16T00:33:16.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa027fff640 1 -- 172.21.15.33:0/2246271202 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0380046c0 con 0x5561574d1480 2024-09-16T00:33:16.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa027fff640 1 -- 172.21.15.33:0/2246271202 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03809b2c0 con 0x5561574d1480 2024-09-16T00:33:16.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa046625740 1 -- 172.21.15.33:0/2246271202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 msgr2=0x5561574db800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:16.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa0427e9640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa0280a3240 0x7fa0280a5700 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa034005fd0 tx=0x7fa0340096a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:16.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa046625740 1 --2- 172.21.15.33:0/2246271202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 0x5561574db800 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fa0380d5c10 tx=0x7fa0380d7690 comp rx=0 tx=0).stop 2024-09-16T00:33:16.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa046625740 1 -- 172.21.15.33:0/2246271202 shutdown_connections 2024-09-16T00:33:16.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa046625740 1 --2- 172.21.15.33:0/2246271202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574dbd40 0x5561574de130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa046625740 1 --2- 172.21.15.33:0/2246271202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 0x5561574db800 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa046625740 1 --2- 172.21.15.33:0/2246271202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574d0b60 0x5561574d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.241+0000 7fa046625740 1 -- 172.21.15.33:0/2246271202 >> 172.21.15.33:0/2246271202 conn(0x5561574cd0f0 msgr2=0x5561574cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.242+0000 7fa046625740 1 -- 172.21.15.33:0/2246271202 shutdown_connections 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.242+0000 7fa046625740 1 -- 172.21.15.33:0/2246271202 wait complete. 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.242+0000 7fa046625740 1 -- start start 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.242+0000 7fa046625740 1 Processor -- start 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.242+0000 7fa046625740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d0b60 0x5561574d8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.242+0000 7fa046625740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 0x5561574d9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.242+0000 7fa046625740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574dbd40 0x5561574d96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.242+0000 7fa046625740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5561574de910 con 0x5561574d1480 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.242+0000 7fa046625740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5561573f0f10 con 0x5561574d0b60 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.242+0000 7fa046625740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5561574dea90 con 0x5561574dbd40 2024-09-16T00:33:16.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.242+0000 7fa046625740 10 client.0 _ll_register_callbacks cb 0x5561573b9a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:16.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0417e7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 0x5561574d9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0417e7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 0x5561574d9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60798/0 (socket says 172.21.15.33:60798) 2024-09-16T00:33:16.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0417e7640 1 -- 172.21.15.33:0/27865795 learned_addr learned my addr 172.21.15.33:0/27865795 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:16.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0427e9640 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574dbd40 0x5561574d96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa041fe8640 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d0b60 0x5561574d8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0417e7640 1 -- 172.21.15.33:0/27865795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574dbd40 msgr2=0x5561574d96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:16.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0417e7640 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574dbd40 0x5561574d96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0417e7640 1 -- 172.21.15.33:0/27865795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d0b60 msgr2=0x5561574d8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:16.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0417e7640 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d0b60 0x5561574d8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0417e7640 1 -- 172.21.15.33:0/27865795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561571a76a0 con 0x5561574d1480 2024-09-16T00:33:16.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa041fe8640 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d0b60 0x5561574d8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:33:16.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0417e7640 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 0x5561574d9190 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fa02c02a930 tx=0x7fa02c0319b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3953f9e08b24da7f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:16.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0257fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:16.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0257fa640 1 -- 172.21.15.33:0/27865795 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c035040 con 0x5561574d1480 2024-09-16T00:33:16.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa046625740 1 -- 172.21.15.33:0/27865795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5561574437f0 con 0x5561574d1480 2024-09-16T00:33:16.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0257fa640 1 -- 172.21.15.33:0/27865795 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa02c005020 con 0x5561574d1480 2024-09-16T00:33:16.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa0257fa640 1 -- 172.21.15.33:0/27865795 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa02c04e050 con 0x5561574d1480 2024-09-16T00:33:16.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa046625740 10 client.18474 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:16.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.243+0000 7fa046625740 1 -- 172.21.15.33:0/27865795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5561574ea590 con 0x5561574d1480 2024-09-16T00:33:16.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.244+0000 7fa0257fa640 1 -- 172.21.15.33:0/27865795 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5561574437f0 con 0x5561574d1480 2024-09-16T00:33:16.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.244+0000 7fa0257fa640 1 -- 172.21.15.33:0/27865795 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa02c0035f0 con 0x5561574d1480 2024-09-16T00:33:16.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.244+0000 7fa046625740 10 client.18474 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:16.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.244+0000 7fa046625740 10 client.18474 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:33:16.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.244+0000 7fa046625740 20 client.18474 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:16.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.244+0000 7fa046625740 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5561574eba50 0x5561574ede80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.244+0000 7fa046625740 4 client.18474 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:33:16.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.244+0000 7fa046625740 1 -- 172.21.15.33:0/27865795 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55615743dd00 con 0x5561574eba50 2024-09-16T00:33:16.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.244+0000 7fa041fe8640 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5561574eba50 0x5561574ede80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.245+0000 7fa0257fa640 10 client.18474 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:16.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.245+0000 7fa041fe8640 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5561574eba50 0x5561574ede80 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36f487a9041b6d85 server_cookie=14c154ebee57a729 in_seq=0 out_seq=0 2024-09-16T00:33:16.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.245+0000 7fa0257fa640 1 -- 172.21.15.33:0/27865795 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55615743dd00 con 0x5561574eba50 2024-09-16T00:33:16.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.245+0000 7fa0257fa640 10 client.18474 handle_command_reply: tid=0 2024-09-16T00:33:16.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.245+0000 7fa046625740 1 client.18474 shutdown 2024-09-16T00:33:16.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.246+0000 7fa046625740 1 -- 172.21.15.33:0/27865795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 msgr2=0x5561574d9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:16.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.246+0000 7fa046625740 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 0x5561574d9190 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fa02c02a930 tx=0x7fa02c0319b0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.246+0000 7fa046625740 1 -- 172.21.15.33:0/27865795 shutdown_connections 2024-09-16T00:33:16.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.246+0000 7fa046625740 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5561574eba50 0x5561574ede80 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.246+0000 7fa046625740 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574dbd40 0x5561574d96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.246+0000 7fa0257fa640 0 client.18474 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:16.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.246+0000 7fa046625740 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d1480 0x5561574d9190 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.246+0000 7fa046625740 1 --2- 172.21.15.33:0/27865795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d0b60 0x5561574d8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.246+0000 7fa046625740 1 -- 172.21.15.33:0/27865795 >> 172.21.15.33:0/27865795 conn(0x5561574cd0f0 msgr2=0x5561574d2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:16.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.246+0000 7fa046625740 1 -- 172.21.15.33:0/27865795 shutdown_connections 2024-09-16T00:33:16.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.246+0000 7fa046625740 1 -- 172.21.15.33:0/27865795 wait complete. 2024-09-16T00:33:16.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.246+0000 7fa046625740 20 client.18474 trim_cache size 0 max 16384 2024-09-16T00:33:16.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.264+0000 7fa046625740 1 Processor -- start 2024-09-16T00:33:16.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.264+0000 7fa046625740 1 -- start start 2024-09-16T00:33:16.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.264+0000 7fa046625740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574d0b60 0x5561574d2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.264+0000 7fa046625740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d1480 0x5561574f3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.264+0000 7fa046625740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574dbd40 0x5561574f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.264+0000 7fa046625740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c1b3b00 con 0x5561574dbd40 2024-09-16T00:33:16.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.264+0000 7fa046625740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c1b1370 con 0x5561574d1480 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.264+0000 7fa046625740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c1af1c0 con 0x5561574d0b60 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0427e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574dbd40 0x5561574f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0427e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574dbd40 0x5561574f3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60806/0 (socket says 172.21.15.33:60806) 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0427e9640 1 -- 172.21.15.33:0/2692064939 learned_addr learned my addr 172.21.15.33:0/2692064939 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0417e7640 1 --2- 172.21.15.33:0/2692064939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d1480 0x5561574f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa041fe8640 1 --2- 172.21.15.33:0/2692064939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574d0b60 0x5561574d2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0427e9640 1 -- 172.21.15.33:0/2692064939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574d0b60 msgr2=0x5561574d2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0427e9640 1 --2- 172.21.15.33:0/2692064939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574d0b60 0x5561574d2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0427e9640 1 -- 172.21.15.33:0/2692064939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d1480 msgr2=0x5561574f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0427e9640 1 --2- 172.21.15.33:0/2692064939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d1480 0x5561574f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0427e9640 1 -- 172.21.15.33:0/2692064939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561574d12b0 con 0x5561574dbd40 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0417e7640 1 --2- 172.21.15.33:0/2692064939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d1480 0x5561574f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:16.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0427e9640 1 --2- 172.21.15.33:0/2692064939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574dbd40 0x5561574f3c90 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fa0340109a0 tx=0x7fa034016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56228caf93fbd82e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0277fe640 1 -- 172.21.15.33:0/2692064939 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa034003970 con 0x5561574dbd40 2024-09-16T00:33:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0277fe640 1 -- 172.21.15.33:0/2692064939 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa03401f930 con 0x5561574dbd40 2024-09-16T00:33:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.265+0000 7fa0277fe640 1 -- 172.21.15.33:0/2692064939 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa034004510 con 0x5561574dbd40 2024-09-16T00:33:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.266+0000 7fa046625740 1 -- 172.21.15.33:0/2692064939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574dbd40 msgr2=0x5561574f3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.266+0000 7fa046625740 1 --2- 172.21.15.33:0/2692064939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574dbd40 0x5561574f3c90 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fa0340109a0 tx=0x7fa034016000 comp rx=0 tx=0).stop 2024-09-16T00:33:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.266+0000 7fa046625740 1 -- 172.21.15.33:0/2692064939 shutdown_connections 2024-09-16T00:33:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.266+0000 7fa046625740 1 --2- 172.21.15.33:0/2692064939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574dbd40 0x5561574f3c90 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.266+0000 7fa046625740 1 --2- 172.21.15.33:0/2692064939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d1480 0x5561574f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.266+0000 7fa046625740 1 --2- 172.21.15.33:0/2692064939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574d0b60 0x5561574d2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.266+0000 7fa046625740 1 -- 172.21.15.33:0/2692064939 >> 172.21.15.33:0/2692064939 conn(0x5561574cd0f0 msgr2=0x5561574d25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.266+0000 7fa046625740 1 -- 172.21.15.33:0/2692064939 shutdown_connections 2024-09-16T00:33:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.266+0000 7fa046625740 1 -- 172.21.15.33:0/2692064939 wait complete. 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa046625740 1 -- start start 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa046625740 1 Processor -- start 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa046625740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d0b60 0x5561574dfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa046625740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d1480 0x5561574e0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa046625740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574dbd40 0x5561574e0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa046625740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c1af830 con 0x5561574d0b60 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa046625740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c1b0ca0 con 0x5561574d1480 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa046625740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c1aff00 con 0x5561574dbd40 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa046625740 10 client.0 _ll_register_callbacks cb 0x7fa02c009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa041fe8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d0b60 0x5561574dfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa041fe8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d0b60 0x5561574dfbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60816/0 (socket says 172.21.15.33:60816) 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa041fe8640 1 -- 172.21.15.33:0/3394368889 learned_addr learned my addr 172.21.15.33:0/3394368889 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:16.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa0427e9640 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574dbd40 0x5561574e0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.267+0000 7fa0417e7640 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d1480 0x5561574e0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa041fe8640 1 -- 172.21.15.33:0/3394368889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574dbd40 msgr2=0x5561574e0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa041fe8640 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574dbd40 0x5561574e0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa041fe8640 1 -- 172.21.15.33:0/3394368889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d1480 msgr2=0x5561574e0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa041fe8640 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d1480 0x5561574e0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa041fe8640 1 -- 172.21.15.33:0/3394368889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561575558c0 con 0x5561574d0b60 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa0417e7640 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d1480 0x5561574e0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa041fe8640 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d0b60 0x5561574dfbd0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fa038009610 tx=0x7fa0380d5f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc952fda5aa079e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa0257fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa0257fa640 1 -- 172.21.15.33:0/3394368889 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038020040 con 0x5561574d0b60 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa0257fa640 1 -- 172.21.15.33:0/3394368889 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa0380860b0 con 0x5561574d0b60 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa046625740 1 -- 172.21.15.33:0/3394368889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa03401f930 con 0x5561574d0b60 2024-09-16T00:33:16.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa0257fa640 1 -- 172.21.15.33:0/3394368889 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa03809b4a0 con 0x5561574d0b60 2024-09-16T00:33:16.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa046625740 10 client.18480 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:16.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa046625740 1 -- 172.21.15.33:0/3394368889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556157556be0 con 0x5561574d0b60 2024-09-16T00:33:16.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa0257fa640 1 -- 172.21.15.33:0/3394368889 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa03401f930 con 0x5561574d0b60 2024-09-16T00:33:16.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa0257fa640 1 -- 172.21.15.33:0/3394368889 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa0380cf030 con 0x5561574d0b60 2024-09-16T00:33:16.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.268+0000 7fa046625740 10 client.18480 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:16.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.269+0000 7fa046625740 10 client.18480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:33:16.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.269+0000 7fa046625740 20 client.18480 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:16.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.269+0000 7fa046625740 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556157556e70 0x556157559260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:16.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.269+0000 7fa046625740 4 client.18480 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:33:16.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.269+0000 7fa046625740 1 -- 172.21.15.33:0/3394368889 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5561573ba110 con 0x556157556e70 2024-09-16T00:33:16.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.269+0000 7fa0417e7640 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556157556e70 0x556157559260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:16.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.269+0000 7fa0257fa640 10 client.18480 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.269+0000 7fa0417e7640 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556157556e70 0x556157559260 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83bd5bd2fabbd6ab server_cookie=b1ae2a92fc244af0 in_seq=0 out_seq=0 2024-09-16T00:33:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.270+0000 7fa0257fa640 1 -- 172.21.15.33:0/3394368889 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5561573ba110 con 0x556157556e70 2024-09-16T00:33:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.270+0000 7fa0257fa640 10 client.18480 handle_command_reply: tid=0 2024-09-16T00:33:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.270+0000 7fa046625740 1 client.18480 shutdown 2024-09-16T00:33:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.270+0000 7fa046625740 1 -- 172.21.15.33:0/3394368889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d0b60 msgr2=0x5561574dfbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.270+0000 7fa046625740 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d0b60 0x5561574dfbd0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fa038009610 tx=0x7fa0380d5f10 comp rx=0 tx=0).stop 2024-09-16T00:33:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa046625740 1 -- 172.21.15.33:0/3394368889 shutdown_connections 2024-09-16T00:33:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa046625740 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556157556e70 0x556157559260 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa0257fa640 0 client.18480 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa046625740 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5561574dbd40 0x5561574e0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa046625740 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5561574d1480 0x5561574e0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa046625740 1 --2- 172.21.15.33:0/3394368889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5561574d0b60 0x5561574dfbd0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa046625740 1 -- 172.21.15.33:0/3394368889 >> 172.21.15.33:0/3394368889 conn(0x5561574cd0f0 msgr2=0x5561574cc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:16.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa046625740 1 -- 172.21.15.33:0/3394368889 shutdown_connections 2024-09-16T00:33:16.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa046625740 1 -- 172.21.15.33:0/3394368889 wait complete. 2024-09-16T00:33:16.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa046625740 20 client.18480 trim_cache size 0 max 16384 2024-09-16T00:33:16.281 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:33:16.281 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:33:16.281 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "a8f025d3-5f23-4059-b96f-482c0aed9788", 2024-09-16T00:33:16.281 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:33:16.281 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:33:16.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa04424c640 1 -- 172.21.15.33:0/3368817942 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa0280a3240 msgr2=0x7fa0280a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:16.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa04424c640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa0280a3240 0x7fa0280a5700 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa034005fd0 tx=0x7fa0340096a0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa04424c640 1 -- 172.21.15.33:0/3368817942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa03c101df0 msgr2=0x7fa03c19a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:16.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa04424c640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa03c101df0 0x7fa03c19a640 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fa03800d770 tx=0x7fa03800dc30 comp rx=0 tx=0).stop 2024-09-16T00:33:16.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa04424c640 1 -- 172.21.15.33:0/3368817942 shutdown_connections 2024-09-16T00:33:16.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa04424c640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa0280a3240 0x7fa0280a5700 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa04424c640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa03c106f20 0x7fa03c19ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa04424c640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa03c1027a0 0x7fa03c19ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa04424c640 1 --2- 172.21.15.33:0/3368817942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa03c101df0 0x7fa03c19a640 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:16.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.271+0000 7fa04424c640 1 -- 172.21.15.33:0/3368817942 >> 172.21.15.33:0/3368817942 conn(0x7fa03c0fd8b0 msgr2=0x7fa03c0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:16.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.272+0000 7fa04424c640 1 -- 172.21.15.33:0/3368817942 shutdown_connections 2024-09-16T00:33:16.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:16.272+0000 7fa04424c640 1 -- 172.21.15.33:0/3368817942 wait complete. 2024-09-16T00:33:17.073 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 6.16 MB/sec execute 23 sec latency 14703.415 ms 2024-09-16T00:33:17.101 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:33:17.107 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:33:17.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: pgmap v834: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 543 KiB/s wr, 96 op/s 2024-09-16T00:33:17.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:33:17.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:33:17.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:33:17.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:33:17.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.365 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:17.391 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:17.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: pgmap v834: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 543 KiB/s wr, 96 op/s 2024-09-16T00:33:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:33:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:33:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:33:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:33:17.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:33:17.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:33:17.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:33:17.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:33:17.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:33:17.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:33:17.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: pgmap v834: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 543 KiB/s wr, 96 op/s 2024-09-16T00:33:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:33:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:33:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:33:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:33:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:33:17.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:33:17.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:33:17.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:33:17.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:33:17.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:17.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:33:17.483 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:18.073 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 5.91 MB/sec execute 24 sec latency 15703.444 ms 2024-09-16T00:33:18.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.079+0000 7fed99881640 1 -- 172.21.15.33:0/1375984522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed9410c180 msgr2=0x7fed9410e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:18.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.079+0000 7fed99881640 1 --2- 172.21.15.33:0/1375984522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed9410c180 0x7fed9410e600 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009ca0 tx=0x7fed8c02a950 comp rx=0 tx=0).stop 2024-09-16T00:33:18.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.079+0000 7fed99881640 1 -- 172.21.15.33:0/1375984522 shutdown_connections 2024-09-16T00:33:18.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.079+0000 7fed99881640 1 --2- 172.21.15.33:0/1375984522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed9410c180 0x7fed9410e600 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.079+0000 7fed99881640 1 --2- 172.21.15.33:0/1375984522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed941036c0 0x7fed9410bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.079+0000 7fed99881640 1 --2- 172.21.15.33:0/1375984522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed94102da0 0x7fed94103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.079+0000 7fed99881640 1 -- 172.21.15.33:0/1375984522 >> 172.21.15.33:0/1375984522 conn(0x7fed940fc7a0 msgr2=0x7fed940febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.079+0000 7fed99881640 1 -- 172.21.15.33:0/1375984522 shutdown_connections 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.079+0000 7fed99881640 1 -- 172.21.15.33:0/1375984522 wait complete. 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.080+0000 7fed99881640 1 Processor -- start 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.080+0000 7fed99881640 1 -- start start 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.080+0000 7fed99881640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed94102da0 0x7fed94198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.080+0000 7fed99881640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed941036c0 0x7fed94198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.080+0000 7fed99881640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed9410c180 0x7fed9419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.080+0000 7fed99881640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fed94111f60 con 0x7fed941036c0 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.080+0000 7fed99881640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fed94111de0 con 0x7fed9410c180 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.080+0000 7fed99881640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fed941120e0 con 0x7fed94102da0 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.080+0000 7fed927fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed941036c0 0x7fed94198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed927fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed941036c0 0x7fed94198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60838/0 (socket says 172.21.15.33:60838) 2024-09-16T00:33:18.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed927fc640 1 -- 172.21.15.33:0/1700702972 learned_addr learned my addr 172.21.15.33:0/1700702972 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:18.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed937fe640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed9410c180 0x7fed9419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed92ffd640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed94102da0 0x7fed94198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed927fc640 1 -- 172.21.15.33:0/1700702972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed94102da0 msgr2=0x7fed94198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:18.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed927fc640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed94102da0 0x7fed94198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed927fc640 1 -- 172.21.15.33:0/1700702972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed9410c180 msgr2=0x7fed9419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:18.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed927fc640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed9410c180 0x7fed9419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed927fc640 1 -- 172.21.15.33:0/1700702972 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed9419d3f0 con 0x7fed941036c0 2024-09-16T00:33:18.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed927fc640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed941036c0 0x7fed94198980 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fed8800d950 tx=0x7fed8800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:18.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed9887f640 1 -- 172.21.15.33:0/1700702972 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88017070 con 0x7fed941036c0 2024-09-16T00:33:18.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed9887f640 1 -- 172.21.15.33:0/1700702972 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed88009c30 con 0x7fed941036c0 2024-09-16T00:33:18.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed99881640 1 -- 172.21.15.33:0/1700702972 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed9419d6e0 con 0x7fed941036c0 2024-09-16T00:33:18.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed9887f640 1 -- 172.21.15.33:0/1700702972 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8800f6b0 con 0x7fed941036c0 2024-09-16T00:33:18.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.081+0000 7fed99881640 1 -- 172.21.15.33:0/1700702972 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed941a4fc0 con 0x7fed941036c0 2024-09-16T00:33:18.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.084+0000 7fed9887f640 1 -- 172.21.15.33:0/1700702972 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fed880109d0 con 0x7fed941036c0 2024-09-16T00:33:18.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.084+0000 7fed9bc5a740 1 Processor -- start 2024-09-16T00:33:18.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.084+0000 7fed9bc5a740 1 -- start start 2024-09-16T00:33:18.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.084+0000 7fed9bc5a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550380 0x565344550760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.084+0000 7fed9bc5a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550ca0 0x56534455f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.084+0000 7fed9bc5a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56534455fa90 0x565344561e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.084+0000 7fed9bc5a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5653444b62e0 con 0x565344550ca0 2024-09-16T00:33:18.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.084+0000 7fed9bc5a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56534445dce0 con 0x56534455fa90 2024-09-16T00:33:18.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.084+0000 7fed9bc5a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5653444b22a0 con 0x565344550380 2024-09-16T00:33:18.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.084+0000 7fed9887f640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fed7c0a3240 0x7fed7c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.084+0000 7fed92ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550ca0 0x56534455f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed92ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550ca0 0x56534455f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60854/0 (socket says 172.21.15.33:60854) 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed92ffd640 1 -- 172.21.15.33:0/1285408449 learned_addr learned my addr 172.21.15.33:0/1285408449 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed937fe640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fed7c0a3240 0x7fed7c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed9887f640 1 -- 172.21.15.33:0/1700702972 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fed880cf920 con 0x7fed941036c0 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed937fe640 1 --2- 172.21.15.33:0/1285408449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550380 0x565344550760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed927fc640 1 --2- 172.21.15.33:0/1285408449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56534455fa90 0x565344561e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed92ffd640 1 -- 172.21.15.33:0/1285408449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550380 msgr2=0x565344550760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed92ffd640 1 --2- 172.21.15.33:0/1285408449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550380 0x565344550760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed92ffd640 1 -- 172.21.15.33:0/1285408449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56534455fa90 msgr2=0x565344561e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed92ffd640 1 --2- 172.21.15.33:0/1285408449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56534455fa90 0x565344561e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed92ffd640 1 -- 172.21.15.33:0/1285408449 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565344442440 con 0x565344550ca0 2024-09-16T00:33:18.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed937fe640 1 --2- 172.21.15.33:0/1285408449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550380 0x565344550760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed937fe640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fed7c0a3240 0x7fed7c0a5700 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fed8c030040 tx=0x7fed8c02fe10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed92ffd640 1 --2- 172.21.15.33:0/1285408449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550ca0 0x56534455f550 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fed8400bcf0 tx=0x7fed8400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbc76c069433f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed79ffb640 1 -- 172.21.15.33:0/1285408449 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed84015070 con 0x565344550ca0 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed79ffb640 1 -- 172.21.15.33:0/1285408449 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed8401a940 con 0x565344550ca0 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.085+0000 7fed79ffb640 1 -- 172.21.15.33:0/1285408449 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed840119d0 con 0x565344550ca0 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.086+0000 7fed9bc5a740 1 -- 172.21.15.33:0/1285408449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550ca0 msgr2=0x56534455f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.086+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/1285408449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550ca0 0x56534455f550 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fed8400bcf0 tx=0x7fed8400f730 comp rx=0 tx=0).stop 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.086+0000 7fed9bc5a740 1 -- 172.21.15.33:0/1285408449 shutdown_connections 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.086+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/1285408449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56534455fa90 0x565344561e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.086+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/1285408449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550ca0 0x56534455f550 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.086+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/1285408449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550380 0x565344550760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.086+0000 7fed9bc5a740 1 -- 172.21.15.33:0/1285408449 >> 172.21.15.33:0/1285408449 conn(0x56534454ca30 msgr2=0x56534454ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:18.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.086+0000 7fed9bc5a740 1 -- 172.21.15.33:0/1285408449 shutdown_connections 2024-09-16T00:33:18.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.086+0000 7fed9bc5a740 1 -- 172.21.15.33:0/1285408449 wait complete. 2024-09-16T00:33:18.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed9bc5a740 1 -- start start 2024-09-16T00:33:18.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed9bc5a740 1 Processor -- start 2024-09-16T00:33:18.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed9bc5a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565344550380 0x56534455f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed9bc5a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 0x56534455c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed9bc5a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534455fa90 0x56534455cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed9bc5a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x565344562660 con 0x56534455fa90 2024-09-16T00:33:18.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed9bc5a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x565344471050 con 0x565344550380 2024-09-16T00:33:18.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed9bc5a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5653445627e0 con 0x565344550ca0 2024-09-16T00:33:18.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed9bc5a740 10 client.0 _ll_register_callbacks cb 0x5653444b1160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:18.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed937fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534455fa90 0x56534455cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed937fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534455fa90 0x56534455cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60864/0 (socket says 172.21.15.33:60864) 2024-09-16T00:33:18.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed937fe640 1 -- 172.21.15.33:0/2082255296 learned_addr learned my addr 172.21.15.33:0/2082255296 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:18.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed927fc640 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 0x56534455c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.087+0000 7fed92ffd640 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565344550380 0x56534455f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed937fe640 1 -- 172.21.15.33:0/2082255296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 msgr2=0x56534455c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:18.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed937fe640 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 0x56534455c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed937fe640 1 -- 172.21.15.33:0/2082255296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565344550380 msgr2=0x56534455f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:18.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed937fe640 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565344550380 0x56534455f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed937fe640 1 -- 172.21.15.33:0/2082255296 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653442273c0 con 0x56534455fa90 2024-09-16T00:33:18.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed937fe640 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534455fa90 0x56534455cba0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fed8c035000 tx=0x7fed8c02a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ed7223ae0ddf20 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:18.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:18.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed72ffd640 1 -- 172.21.15.33:0/2082255296 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8c04d070 con 0x56534455fa90 2024-09-16T00:33:18.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed72ffd640 1 -- 172.21.15.33:0/2082255296 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fed8c004250 con 0x56534455fa90 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed9bc5a740 1 -- 172.21.15.33:0/2082255296 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed8401a940 con 0x56534455fa90 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed72ffd640 1 -- 172.21.15.33:0/2082255296 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fed8c034040 con 0x56534455fa90 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed9bc5a740 10 client.18492 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed9bc5a740 1 -- 172.21.15.33:0/2082255296 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56534456e020 con 0x56534455fa90 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.088+0000 7fed72ffd640 1 -- 172.21.15.33:0/2082255296 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed8401a940 con 0x56534455fa90 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.089+0000 7fed72ffd640 1 -- 172.21.15.33:0/2082255296 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fed8c048040 con 0x56534455fa90 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.089+0000 7fed9bc5a740 10 client.18492 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.089+0000 7fed9bc5a740 10 client.18492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.089+0000 7fed9bc5a740 20 client.18492 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.089+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56534456f480 0x5653445718d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.089+0000 7fed9bc5a740 4 client.18492 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.089+0000 7fed9bc5a740 1 -- 172.21.15.33:0/2082255296 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653444c3850 con 0x56534456f480 2024-09-16T00:33:18.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.089+0000 7fed92ffd640 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56534456f480 0x5653445718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.089+0000 7fed92ffd640 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56534456f480 0x5653445718d0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b189fcb1ee34e4c1 server_cookie=b97224dc03de0701 in_seq=0 out_seq=0 2024-09-16T00:33:18.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.089+0000 7fed72ffd640 10 client.18492 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:18.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed72ffd640 1 -- 172.21.15.33:0/2082255296 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5653444c3850 con 0x56534456f480 2024-09-16T00:33:18.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed72ffd640 10 client.18492 handle_command_reply: tid=0 2024-09-16T00:33:18.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed9bc5a740 1 client.18492 shutdown 2024-09-16T00:33:18.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed9bc5a740 1 -- 172.21.15.33:0/2082255296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534455fa90 msgr2=0x56534455cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:18.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534455fa90 0x56534455cba0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fed8c035000 tx=0x7fed8c02a950 comp rx=0 tx=0).stop 2024-09-16T00:33:18.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed9bc5a740 1 -- 172.21.15.33:0/2082255296 shutdown_connections 2024-09-16T00:33:18.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56534456f480 0x5653445718d0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534455fa90 0x56534455cba0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 0x56534455c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/2082255296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565344550380 0x56534455f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed9bc5a740 1 -- 172.21.15.33:0/2082255296 >> 172.21.15.33:0/2082255296 conn(0x56534454ca30 msgr2=0x565344552790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:18.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed72ffd640 0 client.18492 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:18.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed9bc5a740 1 -- 172.21.15.33:0/2082255296 shutdown_connections 2024-09-16T00:33:18.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.090+0000 7fed9bc5a740 1 -- 172.21.15.33:0/2082255296 wait complete. 2024-09-16T00:33:18.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.091+0000 7fed9bc5a740 20 client.18492 trim_cache size 0 max 16384 2024-09-16T00:33:18.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.108+0000 7fed9bc5a740 1 Processor -- start 2024-09-16T00:33:18.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.108+0000 7fed9bc5a740 1 -- start start 2024-09-16T00:33:18.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.108+0000 7fed9bc5a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56534454ca30 0x56534456b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.108+0000 7fed9bc5a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550380 0x56534455af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.108+0000 7fed9bc5a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 0x5653445771a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.108+0000 7fed9bc5a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fed941af7c0 con 0x565344550380 2024-09-16T00:33:18.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.108+0000 7fed9bc5a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fed941ad030 con 0x56534454ca30 2024-09-16T00:33:18.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.108+0000 7fed9bc5a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fed941aae80 con 0x565344550ca0 2024-09-16T00:33:18.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.108+0000 7fed927fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550380 0x56534455af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.108+0000 7fed927fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550380 0x56534455af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60880/0 (socket says 172.21.15.33:60880) 2024-09-16T00:33:18.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.108+0000 7fed927fc640 1 -- 172.21.15.33:0/1201685287 learned_addr learned my addr 172.21.15.33:0/1201685287 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:18.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.108+0000 7fed92ffd640 1 --2- 172.21.15.33:0/1201685287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56534454ca30 0x56534456b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.109+0000 7fed937fe640 1 --2- 172.21.15.33:0/1201685287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 0x5653445771a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.109+0000 7fed927fc640 1 -- 172.21.15.33:0/1201685287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 msgr2=0x5653445771a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.109+0000 7fed927fc640 1 --2- 172.21.15.33:0/1201685287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 0x5653445771a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.109+0000 7fed927fc640 1 -- 172.21.15.33:0/1201685287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56534454ca30 msgr2=0x56534456b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.109+0000 7fed927fc640 1 --2- 172.21.15.33:0/1201685287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56534454ca30 0x56534456b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.109+0000 7fed927fc640 1 -- 172.21.15.33:0/1201685287 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56534455b610 con 0x565344550380 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.109+0000 7fed92ffd640 1 --2- 172.21.15.33:0/1201685287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56534454ca30 0x56534456b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.109+0000 7fed927fc640 1 --2- 172.21.15.33:0/1201685287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550380 0x56534455af30 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fed8809b7c0 tx=0x7fed88010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27cdd11f5381fc16 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.109+0000 7fed797fa640 1 -- 172.21.15.33:0/1201685287 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88020040 con 0x565344550380 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.109+0000 7fed797fa640 1 -- 172.21.15.33:0/1201685287 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed8809bac0 con 0x565344550380 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.109+0000 7fed797fa640 1 -- 172.21.15.33:0/1201685287 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed880860b0 con 0x565344550380 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 -- 172.21.15.33:0/1201685287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550380 msgr2=0x56534455af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/1201685287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550380 0x56534455af30 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fed8809b7c0 tx=0x7fed88010d70 comp rx=0 tx=0).stop 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 -- 172.21.15.33:0/1201685287 shutdown_connections 2024-09-16T00:33:18.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/1201685287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 0x5653445771a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/1201685287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565344550380 0x56534455af30 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/1201685287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56534454ca30 0x56534456b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 -- 172.21.15.33:0/1201685287 >> 172.21.15.33:0/1201685287 conn(0x56534455cd00 msgr2=0x565344557d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:18.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 -- 172.21.15.33:0/1201685287 shutdown_connections 2024-09-16T00:33:18.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 -- 172.21.15.33:0/1201685287 wait complete. 2024-09-16T00:33:18.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 -- start start 2024-09-16T00:33:18.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 Processor -- start 2024-09-16T00:33:18.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534454ca30 0x565344563b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.110+0000 7fed9bc5a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565344550380 0x5653445640c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed9bc5a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 0x565344564640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed9bc5a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fed941ab4f0 con 0x56534454ca30 2024-09-16T00:33:18.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed9bc5a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fed941ac960 con 0x565344550380 2024-09-16T00:33:18.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed9bc5a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fed941abbc0 con 0x565344550ca0 2024-09-16T00:33:18.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed9bc5a740 10 client.0 _ll_register_callbacks cb 0x7fed8c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:18.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed92ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534454ca30 0x565344563b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed92ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534454ca30 0x565344563b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60884/0 (socket says 172.21.15.33:60884) 2024-09-16T00:33:18.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed92ffd640 1 -- 172.21.15.33:0/3613440523 learned_addr learned my addr 172.21.15.33:0/3613440523 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:18.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed927fc640 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565344550380 0x5653445640c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed937fe640 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 0x565344564640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed92ffd640 1 -- 172.21.15.33:0/3613440523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 msgr2=0x565344564640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:18.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed92ffd640 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 0x565344564640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed92ffd640 1 -- 172.21.15.33:0/3613440523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565344550380 msgr2=0x5653445640c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:18.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed92ffd640 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565344550380 0x5653445640c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed92ffd640 1 -- 172.21.15.33:0/3613440523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56534455f6d0 con 0x56534454ca30 2024-09-16T00:33:18.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed92ffd640 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534454ca30 0x565344563b80 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fed840116a0 tx=0x7fed8400bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0241b5ee4744854 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:18.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:18.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed72ffd640 1 -- 172.21.15.33:0/3613440523 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed84015040 con 0x56534454ca30 2024-09-16T00:33:18.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed72ffd640 1 -- 172.21.15.33:0/3613440523 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fed84005450 con 0x56534454ca30 2024-09-16T00:33:18.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.111+0000 7fed9bc5a740 1 -- 172.21.15.33:0/3613440523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed8809bac0 con 0x56534454ca30 2024-09-16T00:33:18.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.112+0000 7fed72ffd640 1 -- 172.21.15.33:0/3613440523 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fed84035050 con 0x56534454ca30 2024-09-16T00:33:18.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.112+0000 7fed9bc5a740 10 client.18498 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:18.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.112+0000 7fed9bc5a740 1 -- 172.21.15.33:0/3613440523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5653445da5a0 con 0x56534454ca30 2024-09-16T00:33:18.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.112+0000 7fed72ffd640 1 -- 172.21.15.33:0/3613440523 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed8809bac0 con 0x56534454ca30 2024-09-16T00:33:18.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.112+0000 7fed72ffd640 1 -- 172.21.15.33:0/3613440523 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fed84004680 con 0x56534454ca30 2024-09-16T00:33:18.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.112+0000 7fed9bc5a740 10 client.18498 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:18.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.112+0000 7fed9bc5a740 10 client.18498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:33:18.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.112+0000 7fed9bc5a740 20 client.18498 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:18.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.112+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56534455fa90 0x5653445dca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:18.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.112+0000 7fed9bc5a740 4 client.18498 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:33:18.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.112+0000 7fed9bc5a740 1 -- 172.21.15.33:0/3613440523 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5653444bdd60 con 0x56534455fa90 2024-09-16T00:33:18.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.112+0000 7fed927fc640 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56534455fa90 0x5653445dca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:18.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed72ffd640 10 client.18498 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:18.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed927fc640 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56534455fa90 0x5653445dca50 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67351f9204848850 server_cookie=8ac7792130a7406f in_seq=0 out_seq=0 2024-09-16T00:33:18.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed72ffd640 1 -- 172.21.15.33:0/3613440523 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5653444bdd60 con 0x56534455fa90 2024-09-16T00:33:18.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed72ffd640 10 client.18498 handle_command_reply: tid=0 2024-09-16T00:33:18.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed9bc5a740 1 client.18498 shutdown 2024-09-16T00:33:18.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed9bc5a740 1 -- 172.21.15.33:0/3613440523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534454ca30 msgr2=0x565344563b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:18.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534454ca30 0x565344563b80 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fed840116a0 tx=0x7fed8400bee0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed9bc5a740 1 -- 172.21.15.33:0/3613440523 shutdown_connections 2024-09-16T00:33:18.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56534455fa90 0x5653445dca50 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565344550ca0 0x565344564640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565344550380 0x5653445640c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed9bc5a740 1 --2- 172.21.15.33:0/3613440523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56534454ca30 0x565344563b80 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed9bc5a740 1 -- 172.21.15.33:0/3613440523 >> 172.21.15.33:0/3613440523 conn(0x56534455cd00 msgr2=0x56534454b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:18.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed72ffd640 0 client.18498 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:18.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed9bc5a740 1 -- 172.21.15.33:0/3613440523 shutdown_connections 2024-09-16T00:33:18.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.113+0000 7fed9bc5a740 1 -- 172.21.15.33:0/3613440523 wait complete. 2024-09-16T00:33:18.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed9bc5a740 20 client.18498 trim_cache size 0 max 16384 2024-09-16T00:33:18.128 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:33:18.128 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:33:18.128 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:33:18.128 INFO:teuthology.orchestra.run.smithi033.stdout: "a8f025d3-5f23-4059-b96f-482c0aed9788": { 2024-09-16T00:33:18.128 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:33:18.128 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "a8f025d3-5f23-4059-b96f-482c0aed9788", 2024-09-16T00:33:18.128 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:33:18.128 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:33:18.128 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:33:18.128 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:33:18.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed99881640 1 -- 172.21.15.33:0/1700702972 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fed7c0a3240 msgr2=0x7fed7c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:18.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed99881640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fed7c0a3240 0x7fed7c0a5700 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fed8c030040 tx=0x7fed8c02fe10 comp rx=0 tx=0).stop 2024-09-16T00:33:18.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed99881640 1 -- 172.21.15.33:0/1700702972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed941036c0 msgr2=0x7fed94198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:18.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed99881640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed941036c0 0x7fed94198980 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fed8800d950 tx=0x7fed8800de10 comp rx=0 tx=0).stop 2024-09-16T00:33:18.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed99881640 1 -- 172.21.15.33:0/1700702972 shutdown_connections 2024-09-16T00:33:18.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed99881640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fed7c0a3240 0x7fed7c0a5700 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed99881640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed9410c180 0x7fed9419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed99881640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed941036c0 0x7fed94198980 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed99881640 1 --2- 172.21.15.33:0/1700702972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed94102da0 0x7fed94198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:18.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed99881640 1 -- 172.21.15.33:0/1700702972 >> 172.21.15.33:0/1700702972 conn(0x7fed940fc7a0 msgr2=0x7fed940fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:18.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed99881640 1 -- 172.21.15.33:0/1700702972 shutdown_connections 2024-09-16T00:33:18.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.114+0000 7fed99881640 1 -- 172.21.15.33:0/1700702972 wait complete. 2024-09-16T00:33:18.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:33:18.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:18.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:33:18.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:18.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:33:18.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:33:19.067 INFO:tasks.cephfs.filesystem:scrub status for tag:a8f025d3-5f23-4059-b96f-482c0aed9788 - {'path': '/', 'tag': 'a8f025d3-5f23-4059-b96f-482c0aed9788', 'options': 'recursive,force'} 2024-09-16T00:33:19.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 5.67 MB/sec execute 25 sec latency 16703.481 ms 2024-09-16T00:33:19.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.664+0000 7f6e5931e640 1 -- 172.21.15.33:0/4171047041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e54102790 msgr2=0x7f6e541067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:19.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.664+0000 7f6e5931e640 1 --2- 172.21.15.33:0/4171047041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e54102790 0x7f6e541067e0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c009900 tx=0x7f6e3c02a410 comp rx=0 tx=0).stop 2024-09-16T00:33:19.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.664+0000 7f6e5931e640 1 -- 172.21.15.33:0/4171047041 shutdown_connections 2024-09-16T00:33:19.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.664+0000 7f6e5931e640 1 --2- 172.21.15.33:0/4171047041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e54106f10 0x7f6e5410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:19.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.664+0000 7f6e5931e640 1 --2- 172.21.15.33:0/4171047041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e54102790 0x7f6e541067e0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:19.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.664+0000 7f6e5931e640 1 --2- 172.21.15.33:0/4171047041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e54101de0 0x7f6e541021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:19.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.664+0000 7f6e5931e640 1 -- 172.21.15.33:0/4171047041 >> 172.21.15.33:0/4171047041 conn(0x7f6e540fd8c0 msgr2=0x7f6e540ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:19.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.664+0000 7f6e5931e640 1 -- 172.21.15.33:0/4171047041 shutdown_connections 2024-09-16T00:33:19.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.664+0000 7f6e5931e640 1 -- 172.21.15.33:0/4171047041 wait complete. 2024-09-16T00:33:19.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.665+0000 7f6e5931e640 1 Processor -- start 2024-09-16T00:33:19.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.665+0000 7f6e5931e640 1 -- start start 2024-09-16T00:33:19.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.665+0000 7f6e5931e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e54101de0 0x7f6e540744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:19.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.665+0000 7f6e5931e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e54102790 0x7f6e54074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.665+0000 7f6e5931e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e54106f10 0x7f6e54074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.665+0000 7f6e5931e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6e54111f70 con 0x7f6e54106f10 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.665+0000 7f6e5931e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6e54111df0 con 0x7f6e54102790 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.665+0000 7f6e5931e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6e541120f0 con 0x7f6e54101de0 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.665+0000 7f6e537fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e54106f10 0x7f6e54074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.665+0000 7f6e537fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e54106f10 0x7f6e54074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60914/0 (socket says 172.21.15.33:60914) 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.665+0000 7f6e537fe640 1 -- 172.21.15.33:0/34805810 learned_addr learned my addr 172.21.15.33:0/34805810 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.665+0000 7f6e4bfff640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e54102790 0x7f6e54074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e52ffd640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e54101de0 0x7f6e540744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e537fe640 1 -- 172.21.15.33:0/34805810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e54101de0 msgr2=0x7f6e540744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e537fe640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e54101de0 0x7f6e540744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e537fe640 1 -- 172.21.15.33:0/34805810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e54102790 msgr2=0x7f6e54074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:19.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e537fe640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e54102790 0x7f6e54074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e537fe640 1 -- 172.21.15.33:0/34805810 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e541a68f0 con 0x7f6e54106f10 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e52ffd640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e54101de0 0x7f6e540744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e537fe640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e54106f10 0x7f6e54074f70 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00eaa0 tx=0x7f6e4c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e50ff9640 1 -- 172.21.15.33:0/34805810 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c00cdb0 con 0x7f6e54106f10 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e50ff9640 1 -- 172.21.15.33:0/34805810 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e4c005020 con 0x7f6e54106f10 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e5931e640 1 -- 172.21.15.33:0/34805810 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e541a6be0 con 0x7f6e54106f10 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e50ff9640 1 -- 172.21.15.33:0/34805810 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c0116b0 con 0x7f6e54106f10 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.666+0000 7f6e5931e640 1 -- 172.21.15.33:0/34805810 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e541a70f0 con 0x7f6e54106f10 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.667+0000 7f6e5931e640 1 -- 172.21.15.33:0/34805810 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e541021c0 con 0x7f6e54106f10 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.668+0000 7f6e50ff9640 1 -- 172.21.15.33:0/34805810 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6e4c005560 con 0x7f6e54106f10 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.669+0000 7f6e50ff9640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6e400a32c0 0x7f6e400a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.669+0000 7f6e52ffd640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6e400a32c0 0x7f6e400a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.669+0000 7f6e50ff9640 1 -- 172.21.15.33:0/34805810 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6e4c0d02a0 con 0x7f6e54106f10 2024-09-16T00:33:19.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.670+0000 7f6e52ffd640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6e400a32c0 0x7f6e400a5780 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6e440058c0 tx=0x7f6e44005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:19.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.672+0000 7f6e50ff9640 1 -- 172.21.15.33:0/34805810 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6e4c016030 con 0x7f6e54106f10 2024-09-16T00:33:19.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.883+0000 7f6e5931e640 1 -- 172.21.15.33:0/34805810 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e5402d7c0 con 0x7f6e54106f10 2024-09-16T00:33:19.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.884+0000 7f6e50ff9640 1 -- 172.21.15.33:0/34805810 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11906 (secure 0 0 0) 0x7f6e4c09cb60 con 0x7f6e54106f10 2024-09-16T00:33:19.120 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:33:19.120 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":31,"btime":"2024-09-16T00:30:57:641642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:30:57.027530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:33:19.121 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 31 2024-09-16T00:33:19.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.886+0000 7f6e5931e640 1 -- 172.21.15.33:0/34805810 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6e400a32c0 msgr2=0x7f6e400a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:19.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.886+0000 7f6e5931e640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6e400a32c0 0x7f6e400a5780 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6e440058c0 tx=0x7f6e44005850 comp rx=0 tx=0).stop 2024-09-16T00:33:19.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.886+0000 7f6e5931e640 1 -- 172.21.15.33:0/34805810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e54106f10 msgr2=0x7f6e54074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:19.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.886+0000 7f6e5931e640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e54106f10 0x7f6e54074f70 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00eaa0 tx=0x7f6e4c00ef60 comp rx=0 tx=0).stop 2024-09-16T00:33:19.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.886+0000 7f6e5931e640 1 -- 172.21.15.33:0/34805810 shutdown_connections 2024-09-16T00:33:19.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.886+0000 7f6e5931e640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6e400a32c0 0x7f6e400a5780 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:19.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.886+0000 7f6e5931e640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e54106f10 0x7f6e54074f70 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:19.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.886+0000 7f6e5931e640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e54102790 0x7f6e54074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:19.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.886+0000 7f6e5931e640 1 --2- 172.21.15.33:0/34805810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e54101de0 0x7f6e540744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:19.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.886+0000 7f6e5931e640 1 -- 172.21.15.33:0/34805810 >> 172.21.15.33:0/34805810 conn(0x7f6e540fd8c0 msgr2=0x7f6e540fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:19.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.886+0000 7f6e5931e640 1 -- 172.21.15.33:0/34805810 shutdown_connections 2024-09-16T00:33:19.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:18.886+0000 7f6e5931e640 1 -- 172.21.15.33:0/34805810 wait complete. 2024-09-16T00:33:19.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:19 smithi033 ceph-mon[31243]: pgmap v835: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 13 op/s 2024-09-16T00:33:19.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:19 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/34805810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:33:19.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:19 smithi083 ceph-mon[35242]: pgmap v835: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 13 op/s 2024-09-16T00:33:19.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:19 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/34805810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:33:19.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:19 smithi060 ceph-mon[35185]: pgmap v835: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 13 op/s 2024-09-16T00:33:19.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:19 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/34805810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:33:19.809 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id be63bf18 --release --await' 2024-09-16T00:33:19.810 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id be63bf18 --release --await 2024-09-16T00:33:20.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46453 5.45 MB/sec execute 26 sec latency 17703.518 ms 2024-09-16T00:33:20.111 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:20.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.661+0000 7ff564345640 1 -- 172.21.15.33:0/1775756798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff55c102ef0 msgr2=0x7ff55c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:20.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.661+0000 7ff564345640 1 --2- 172.21.15.33:0/1775756798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff55c102ef0 0x7ff55c10da40 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7ff554009ca0 tx=0x7ff55402a950 comp rx=0 tx=0).stop 2024-09-16T00:33:20.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.661+0000 7ff564345640 1 -- 172.21.15.33:0/1775756798 shutdown_connections 2024-09-16T00:33:20.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.661+0000 7ff564345640 1 --2- 172.21.15.33:0/1775756798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff55c102ef0 0x7ff55c10da40 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.661+0000 7ff564345640 1 --2- 172.21.15.33:0/1775756798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff55c102570 0x7ff55c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.661+0000 7ff564345640 1 --2- 172.21.15.33:0/1775756798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff55c100e20 0x7ff55c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.661+0000 7ff564345640 1 -- 172.21.15.33:0/1775756798 >> 172.21.15.33:0/1775756798 conn(0x7ff55c0fc7d0 msgr2=0x7ff55c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:20.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.661+0000 7ff564345640 1 -- 172.21.15.33:0/1775756798 shutdown_connections 2024-09-16T00:33:20.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.661+0000 7ff564345640 1 -- 172.21.15.33:0/1775756798 wait complete. 2024-09-16T00:33:20.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.661+0000 7ff564345640 1 Processor -- start 2024-09-16T00:33:20.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.661+0000 7ff564345640 1 -- start start 2024-09-16T00:33:20.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff564345640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff55c100e20 0x7ff55c0fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff564345640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff55c102570 0x7ff55c0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff564345640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff55c102ef0 0x7ff55c0fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff564345640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c111020 con 0x7ff55c100e20 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff564345640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c110ea0 con 0x7ff55c102570 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff564345640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1111a0 con 0x7ff55c102ef0 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5620e1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff55c100e20 0x7ff55c0fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5620e1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff55c100e20 0x7ff55c0fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54070/0 (socket says 172.21.15.33:54070) 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5620e1640 1 -- 172.21.15.33:0/1963521666 learned_addr learned my addr 172.21.15.33:0/1963521666 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5628e2640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff55c102ef0 0x7ff55c0fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5618e0640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff55c102570 0x7ff55c0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5620e1640 1 -- 172.21.15.33:0/1963521666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff55c102ef0 msgr2=0x7ff55c0fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5620e1640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff55c102ef0 0x7ff55c0fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5620e1640 1 -- 172.21.15.33:0/1963521666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff55c102570 msgr2=0x7ff55c0fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5620e1640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff55c102570 0x7ff55c0fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5620e1640 1 -- 172.21.15.33:0/1963521666 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55c1a6aa0 con 0x7ff55c100e20 2024-09-16T00:33:20.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5628e2640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff55c102ef0 0x7ff55c0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5618e0640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff55c102570 0x7ff55c0fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5620e1640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff55c100e20 0x7ff55c0fcc80 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7ff55800b4a0 tx=0x7ff55800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.662+0000 7ff5537fe640 1 -- 172.21.15.33:0/1963521666 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558014070 con 0x7ff55c100e20 2024-09-16T00:33:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.663+0000 7ff564345640 1 -- 172.21.15.33:0/1963521666 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff55c1a6d90 con 0x7ff55c100e20 2024-09-16T00:33:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.663+0000 7ff5537fe640 1 -- 172.21.15.33:0/1963521666 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff558010af0 con 0x7ff55c100e20 2024-09-16T00:33:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.663+0000 7ff5537fe640 1 -- 172.21.15.33:0/1963521666 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558017d10 con 0x7ff55c100e20 2024-09-16T00:33:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.663+0000 7ff564345640 1 -- 172.21.15.33:0/1963521666 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff55c1a7250 con 0x7ff55c100e20 2024-09-16T00:33:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff5537fe640 1 -- 172.21.15.33:0/1963521666 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff558017070 con 0x7ff55c100e20 2024-09-16T00:33:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff56671e740 1 Processor -- start 2024-09-16T00:33:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff56671e740 1 -- start start 2024-09-16T00:33:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff56671e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3060 0x55e58a2c3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff56671e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3980 0x55e58a2cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff56671e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2ce240 0x55e58a2d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff56671e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e58a228620 con 0x55e58a2c3060 2024-09-16T00:33:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff56671e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e58a1cfe50 con 0x55e58a2c3980 2024-09-16T00:33:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff56671e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e58a2245e0 con 0x55e58a2ce240 2024-09-16T00:33:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff5628e2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3060 0x55e58a2c3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff5628e2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3060 0x55e58a2c3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54086/0 (socket says 172.21.15.33:54086) 2024-09-16T00:33:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff5628e2640 1 -- 172.21.15.33:0/783485092 learned_addr learned my addr 172.21.15.33:0/783485092 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff5620e1640 1 --2- 172.21.15.33:0/783485092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3980 0x55e58a2cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.666+0000 7ff5618e0640 1 --2- 172.21.15.33:0/783485092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2ce240 0x55e58a2d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.667+0000 7ff5537fe640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5480a3240 0x7ff5480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.667+0000 7ff5628e2640 1 -- 172.21.15.33:0/783485092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2ce240 msgr2=0x55e58a2d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.667+0000 7ff5628e2640 1 --2- 172.21.15.33:0/783485092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2ce240 0x55e58a2d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.667+0000 7ff5628e2640 1 -- 172.21.15.33:0/783485092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3980 msgr2=0x55e58a2cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:20.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.667+0000 7ff5628e2640 1 --2- 172.21.15.33:0/783485092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3980 0x55e58a2cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.667+0000 7ff5628e2640 1 -- 172.21.15.33:0/783485092 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e58a1c0d40 con 0x55e58a2c3060 2024-09-16T00:33:20.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.667+0000 7ff5537fe640 1 -- 172.21.15.33:0/1963521666 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff5580cf1b0 con 0x7ff55c100e20 2024-09-16T00:33:20.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.667+0000 7ff5628e2640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5480a3240 0x7ff5480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.667+0000 7ff5628e2640 1 --2- 172.21.15.33:0/783485092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3060 0x55e58a2c3440 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7ff554009a20 tx=0x7ff55402f930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835cbb3c6ca025a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:20.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.667+0000 7ff550dea640 1 -- 172.21.15.33:0/783485092 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff554034040 con 0x55e58a2c3060 2024-09-16T00:33:20.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.667+0000 7ff550dea640 1 -- 172.21.15.33:0/783485092 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff55402f6e0 con 0x55e58a2c3060 2024-09-16T00:33:20.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.667+0000 7ff550dea640 1 -- 172.21.15.33:0/783485092 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5540307c0 con 0x55e58a2c3060 2024-09-16T00:33:20.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.668+0000 7ff56671e740 1 -- 172.21.15.33:0/783485092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3060 msgr2=0x55e58a2c3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:20.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.668+0000 7ff5628e2640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5480a3240 0x7ff5480a5700 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff554049000 tx=0x7ff55404a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:20.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.668+0000 7ff56671e740 1 --2- 172.21.15.33:0/783485092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3060 0x55e58a2c3440 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7ff554009a20 tx=0x7ff55402f930 comp rx=0 tx=0).stop 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.668+0000 7ff56671e740 1 -- 172.21.15.33:0/783485092 shutdown_connections 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.668+0000 7ff56671e740 1 --2- 172.21.15.33:0/783485092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2ce240 0x55e58a2d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.668+0000 7ff56671e740 1 --2- 172.21.15.33:0/783485092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3980 0x55e58a2cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.668+0000 7ff56671e740 1 --2- 172.21.15.33:0/783485092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3060 0x55e58a2c3440 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.668+0000 7ff56671e740 1 -- 172.21.15.33:0/783485092 >> 172.21.15.33:0/783485092 conn(0x55e58a2bf5f0 msgr2=0x55e58a2bf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.668+0000 7ff56671e740 1 -- 172.21.15.33:0/783485092 shutdown_connections 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.668+0000 7ff56671e740 1 -- 172.21.15.33:0/783485092 wait complete. 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.668+0000 7ff56671e740 1 -- start start 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff56671e740 1 Processor -- start 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff56671e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2c3060 0x55e58a2cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff56671e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 0x55e58a2cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff56671e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2ce240 0x55e58a2cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff56671e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e58a2d0e10 con 0x55e58a2c3980 2024-09-16T00:33:20.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff56671e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e58a1b0f10 con 0x55e58a2ce240 2024-09-16T00:33:20.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff56671e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e58a2d0f90 con 0x55e58a2c3060 2024-09-16T00:33:20.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff56671e740 10 client.0 _ll_register_callbacks cb 0x55e58a1d4db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:20.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff5618e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 0x55e58a2cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff5618e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 0x55e58a2cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54090/0 (socket says 172.21.15.33:54090) 2024-09-16T00:33:20.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff5618e0640 1 -- 172.21.15.33:0/1235269859 learned_addr learned my addr 172.21.15.33:0/1235269859 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:20.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff5620e1640 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2c3060 0x55e58a2cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff5628e2640 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2ce240 0x55e58a2cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff5618e0640 1 -- 172.21.15.33:0/1235269859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2c3060 msgr2=0x55e58a2cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:20.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff5618e0640 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2c3060 0x55e58a2cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff5618e0640 1 -- 172.21.15.33:0/1235269859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2ce240 msgr2=0x55e58a2cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:20.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff5618e0640 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2ce240 0x55e58a2cbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.669+0000 7ff5618e0640 1 -- 172.21.15.33:0/1235269859 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e589f37580 con 0x55e58a2c3980 2024-09-16T00:33:20.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff5618e0640 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 0x55e58a2cb600 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7ff54c010500 tx=0x7ff54c0109c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cdefc1a93782714 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:20.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff545ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:20.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff545ffb640 1 -- 172.21.15.33:0/1235269859 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c018070 con 0x55e58a2c3980 2024-09-16T00:33:20.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff545ffb640 1 -- 172.21.15.33:0/1235269859 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff54c014640 con 0x55e58a2c3980 2024-09-16T00:33:20.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff56671e740 1 -- 172.21.15.33:0/1235269859 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff55402f6e0 con 0x55e58a2c3980 2024-09-16T00:33:20.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff545ffb640 1 -- 172.21.15.33:0/1235269859 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff54c024050 con 0x55e58a2c3980 2024-09-16T00:33:20.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff56671e740 10 client.18516 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:20.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff56671e740 1 -- 172.21.15.33:0/1235269859 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e58a001960 con 0x55e58a2c3980 2024-09-16T00:33:20.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff545ffb640 1 -- 172.21.15.33:0/1235269859 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff55402f6e0 con 0x55e58a2c3980 2024-09-16T00:33:20.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff545ffb640 1 -- 172.21.15.33:0/1235269859 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff54c012a80 con 0x55e58a2c3980 2024-09-16T00:33:20.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff56671e740 10 client.18516 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff56671e740 10 client.18516 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff56671e740 20 client.18516 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff56671e740 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e58a2ddd90 0x55e58a2e01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff56671e740 4 client.18516 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.670+0000 7ff56671e740 1 -- 172.21.15.33:0/1235269859 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e58a235b90 con 0x55e58a2ddd90 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.671+0000 7ff5620e1640 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e58a2ddd90 0x55e58a2e01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.671+0000 7ff545ffb640 10 client.18516 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.671+0000 7ff5620e1640 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e58a2ddd90 0x55e58a2e01d0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e8fa9e69979153f server_cookie=3cd0fbb60537a64a in_seq=0 out_seq=0 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff545ffb640 1 -- 172.21.15.33:0/1235269859 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e58a235b90 con 0x55e58a2ddd90 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff545ffb640 10 client.18516 handle_command_reply: tid=0 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff56671e740 1 client.18516 shutdown 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff56671e740 1 -- 172.21.15.33:0/1235269859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 msgr2=0x55e58a2cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:20.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff56671e740 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 0x55e58a2cb600 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7ff54c010500 tx=0x7ff54c0109c0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff56671e740 1 -- 172.21.15.33:0/1235269859 shutdown_connections 2024-09-16T00:33:20.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff56671e740 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e58a2ddd90 0x55e58a2e01d0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff56671e740 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2ce240 0x55e58a2cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff545ffb640 0 client.18516 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:20.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff56671e740 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 0x55e58a2cb600 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff56671e740 1 --2- 172.21.15.33:0/1235269859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2c3060 0x55e58a2cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff56671e740 1 -- 172.21.15.33:0/1235269859 >> 172.21.15.33:0/1235269859 conn(0x55e58a2bf5f0 msgr2=0x55e58a2be750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:20.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff56671e740 1 -- 172.21.15.33:0/1235269859 shutdown_connections 2024-09-16T00:33:20.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff56671e740 1 -- 172.21.15.33:0/1235269859 wait complete. 2024-09-16T00:33:20.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.672+0000 7ff56671e740 20 client.18516 trim_cache size 0 max 16384 2024-09-16T00:33:20.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.689+0000 7ff56671e740 1 Processor -- start 2024-09-16T00:33:20.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.689+0000 7ff56671e740 1 -- start start 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.689+0000 7ff56671e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3060 0x55e58a2be640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.689+0000 7ff56671e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2c3980 0x55e58a2e5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.689+0000 7ff56671e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2ce240 0x55e58a2e5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.689+0000 7ff56671e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1b3d60 con 0x55e58a2ce240 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff56671e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1b15d0 con 0x55e58a2c3060 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff56671e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1af420 con 0x55e58a2c3980 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff5628e2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2ce240 0x55e58a2e5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff5628e2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2ce240 0x55e58a2e5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54106/0 (socket says 172.21.15.33:54106) 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff5628e2640 1 -- 172.21.15.33:0/2735628598 learned_addr learned my addr 172.21.15.33:0/2735628598 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff5620e1640 1 --2- 172.21.15.33:0/2735628598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3060 0x55e58a2be640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff5618e0640 1 --2- 172.21.15.33:0/2735628598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2c3980 0x55e58a2e5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff5628e2640 1 -- 172.21.15.33:0/2735628598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2c3980 msgr2=0x55e58a2e5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:20.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff5628e2640 1 --2- 172.21.15.33:0/2735628598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2c3980 0x55e58a2e5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff5628e2640 1 -- 172.21.15.33:0/2735628598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3060 msgr2=0x55e58a2be640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff5628e2640 1 --2- 172.21.15.33:0/2735628598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3060 0x55e58a2be640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff5628e2640 1 -- 172.21.15.33:0/2735628598 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e58a2c0280 con 0x55e58a2ce240 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff5620e1640 1 --2- 172.21.15.33:0/2735628598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3060 0x55e58a2be640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff5628e2640 1 --2- 172.21.15.33:0/2735628598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2ce240 0x55e58a2e5fe0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7ff554009a20 tx=0x7ff554052000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9291e5610a7521 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff547fff640 1 -- 172.21.15.33:0/2735628598 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55405f070 con 0x55e58a2ce240 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff547fff640 1 -- 172.21.15.33:0/2735628598 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff55402f6e0 con 0x55e58a2ce240 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.690+0000 7ff547fff640 1 -- 172.21.15.33:0/2735628598 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff554034040 con 0x55e58a2ce240 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.691+0000 7ff56671e740 1 -- 172.21.15.33:0/2735628598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2ce240 msgr2=0x55e58a2e5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.691+0000 7ff56671e740 1 --2- 172.21.15.33:0/2735628598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2ce240 0x55e58a2e5fe0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7ff554009a20 tx=0x7ff554052000 comp rx=0 tx=0).stop 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.691+0000 7ff56671e740 1 -- 172.21.15.33:0/2735628598 shutdown_connections 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.691+0000 7ff56671e740 1 --2- 172.21.15.33:0/2735628598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2ce240 0x55e58a2e5fe0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.691+0000 7ff56671e740 1 --2- 172.21.15.33:0/2735628598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2c3980 0x55e58a2e5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.691+0000 7ff56671e740 1 --2- 172.21.15.33:0/2735628598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3060 0x55e58a2be640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.691+0000 7ff56671e740 1 -- 172.21.15.33:0/2735628598 >> 172.21.15.33:0/2735628598 conn(0x55e58a2bf5f0 msgr2=0x55e58a2bdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:20.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.691+0000 7ff56671e740 1 -- 172.21.15.33:0/2735628598 shutdown_connections 2024-09-16T00:33:20.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.691+0000 7ff56671e740 1 -- 172.21.15.33:0/2735628598 wait complete. 2024-09-16T00:33:20.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.691+0000 7ff56671e740 1 -- start start 2024-09-16T00:33:20.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff56671e740 1 Processor -- start 2024-09-16T00:33:20.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff56671e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3060 0x55e58a2d63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff56671e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 0x55e58a2d6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff56671e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2ce240 0x55e58a2d1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff56671e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1afa90 con 0x55e58a2c3980 2024-09-16T00:33:20.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff56671e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1b0f00 con 0x55e58a2c3060 2024-09-16T00:33:20.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff56671e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1b0160 con 0x55e58a2ce240 2024-09-16T00:33:20.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff56671e740 10 client.0 _ll_register_callbacks cb 0x7ff54c007e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:20.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff5618e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 0x55e58a2d6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff5618e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 0x55e58a2d6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54122/0 (socket says 172.21.15.33:54122) 2024-09-16T00:33:20.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff5618e0640 1 -- 172.21.15.33:0/4098472063 learned_addr learned my addr 172.21.15.33:0/4098472063 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:20.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff5628e2640 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2ce240 0x55e58a2d1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff5620e1640 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3060 0x55e58a2d63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff5618e0640 1 -- 172.21.15.33:0/4098472063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2ce240 msgr2=0x55e58a2d1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:20.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff5618e0640 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2ce240 0x55e58a2d1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff5618e0640 1 -- 172.21.15.33:0/4098472063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3060 msgr2=0x55e58a2d63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:20.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff5618e0640 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3060 0x55e58a2d63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff5618e0640 1 -- 172.21.15.33:0/4098472063 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e58a2d1a50 con 0x55e58a2c3980 2024-09-16T00:33:20.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff5618e0640 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 0x55e58a2d6930 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ff54c004810 tx=0x7ff54c0136d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6a445361b0e7a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:20.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.692+0000 7ff545ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:33:20.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff545ffb640 1 -- 172.21.15.33:0/4098472063 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c018070 con 0x55e58a2c3980 2024-09-16T00:33:20.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff56671e740 1 -- 172.21.15.33:0/4098472063 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff55402f6e0 con 0x55e58a2c3980 2024-09-16T00:33:20.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff545ffb640 1 -- 172.21.15.33:0/4098472063 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff54c013c80 con 0x55e58a2c3980 2024-09-16T00:33:20.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff545ffb640 1 -- 172.21.15.33:0/4098472063 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff54c013e80 con 0x55e58a2c3980 2024-09-16T00:33:20.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff56671e740 10 client.18522 fetch_fsmap learned FSMap version 31 2024-09-16T00:33:20.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff56671e740 1 -- 172.21.15.33:0/4098472063 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e58a349650 con 0x55e58a2c3980 2024-09-16T00:33:20.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff545ffb640 1 -- 172.21.15.33:0/4098472063 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff55402f6e0 con 0x55e58a2c3980 2024-09-16T00:33:20.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff545ffb640 1 -- 172.21.15.33:0/4098472063 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff54c0123f0 con 0x55e58a2c3980 2024-09-16T00:33:20.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff56671e740 10 client.18522 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T00:33:20.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff56671e740 10 client.18522 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:33:20.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff56671e740 20 client.18522 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:20.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff56671e740 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e58a349dc0 0x55e58a34c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:20.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff56671e740 4 client.18522 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "be63bf18", "release": true, "await": true}] 2024-09-16T00:33:20.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.693+0000 7ff56671e740 1 -- 172.21.15.33:0/4098472063 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "be63bf18", "release": true, "await": true}) -- 0x55e58a2300a0 con 0x55e58a349dc0 2024-09-16T00:33:20.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.694+0000 7ff5620e1640 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e58a349dc0 0x55e58a34c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:20.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.694+0000 7ff5620e1640 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e58a349dc0 0x55e58a34c1e0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=320c6af4d2530075 server_cookie=707d48d5207dc833 in_seq=0 out_seq=0 2024-09-16T00:33:20.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.694+0000 7ff545ffb640 10 client.18522 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:20.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.758+0000 7ff545ffb640 1 -- 172.21.15.33:0/4098472063 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e58a2300a0 con 0x55e58a349dc0 2024-09-16T00:33:20.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.758+0000 7ff545ffb640 10 client.18522 handle_command_reply: tid=0 2024-09-16T00:33:20.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.758+0000 7ff56671e740 1 client.18522 shutdown 2024-09-16T00:33:20.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.758+0000 7ff56671e740 1 -- 172.21.15.33:0/4098472063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 msgr2=0x55e58a2d6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:20.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.758+0000 7ff56671e740 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 0x55e58a2d6930 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ff54c004810 tx=0x7ff54c0136d0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.758+0000 7ff56671e740 1 -- 172.21.15.33:0/4098472063 shutdown_connections 2024-09-16T00:33:20.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.758+0000 7ff56671e740 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e58a349dc0 0x55e58a34c1e0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.758+0000 7ff56671e740 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e58a2ce240 0x55e58a2d1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.758+0000 7ff56671e740 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e58a2c3980 0x55e58a2d6930 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.758+0000 7ff545ffb640 0 client.18522 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:20.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.758+0000 7ff56671e740 1 --2- 172.21.15.33:0/4098472063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e58a2c3060 0x55e58a2d63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.758+0000 7ff56671e740 1 -- 172.21.15.33:0/4098472063 >> 172.21.15.33:0/4098472063 conn(0x55e58a2bf5f0 msgr2=0x55e58a2be750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:20.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff56671e740 1 -- 172.21.15.33:0/4098472063 shutdown_connections 2024-09-16T00:33:20.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff56671e740 1 -- 172.21.15.33:0/4098472063 wait complete. 2024-09-16T00:33:20.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff56671e740 20 client.18522 trim_cache size 0 max 16384 2024-09-16T00:33:20.761 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:33:20.761 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 31, 2024-09-16T00:33:20.761 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:33:20.761 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 60, 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: "be63bf18": { 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 60, 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 137.7, 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:33:20.762 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:33:20.763 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:33:20.763 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:33:20.763 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:33:20.763 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:33:20.763 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:33:20.763 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:33:20.763 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:33:20.763 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:33:20.763 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:33:20.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff564345640 1 -- 172.21.15.33:0/1963521666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5480a3240 msgr2=0x7ff5480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:20.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff564345640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5480a3240 0x7ff5480a5700 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff554049000 tx=0x7ff55404a000 comp rx=0 tx=0).stop 2024-09-16T00:33:20.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff564345640 1 -- 172.21.15.33:0/1963521666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff55c100e20 msgr2=0x7ff55c0fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:20.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff564345640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff55c100e20 0x7ff55c0fcc80 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7ff55800b4a0 tx=0x7ff55800b960 comp rx=0 tx=0).stop 2024-09-16T00:33:20.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff564345640 1 -- 172.21.15.33:0/1963521666 shutdown_connections 2024-09-16T00:33:20.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff564345640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5480a3240 0x7ff5480a5700 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff564345640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff55c102ef0 0x7ff55c0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff564345640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff55c102570 0x7ff55c0fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff564345640 1 --2- 172.21.15.33:0/1963521666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff55c100e20 0x7ff55c0fcc80 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:20.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.759+0000 7ff564345640 1 -- 172.21.15.33:0/1963521666 >> 172.21.15.33:0/1963521666 conn(0x7ff55c0fc7d0 msgr2=0x7ff55c06a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:20.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.760+0000 7ff564345640 1 -- 172.21.15.33:0/1963521666 shutdown_connections 2024-09-16T00:33:20.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:20.760+0000 7ff564345640 1 -- 172.21.15.33:0/1963521666 wait complete. 2024-09-16T00:33:21.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 46610 5.64 MB/sec execute 27 sec latency 18393.483 ms 2024-09-16T00:33:21.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:21 smithi033 ceph-mon[31243]: pgmap v836: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 83 KiB/s wr, 15 op/s 2024-09-16T00:33:21.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:21 smithi083 ceph-mon[35242]: pgmap v836: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 83 KiB/s wr, 15 op/s 2024-09-16T00:33:21.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:21 smithi060 ceph-mon[35185]: pgmap v836: 129 pgs: 129 active+clean; 505 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 83 KiB/s wr, 15 op/s 2024-09-16T00:33:21.811 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'be63bf18', total seconds elapsed: 22.4 2024-09-16T00:33:22.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 47029 6.07 MB/sec execute 28 sec latency 63.807 ms 2024-09-16T00:33:23.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 47508 6.60 MB/sec execute 29 sec latency 106.362 ms 2024-09-16T00:33:23.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:23 smithi033 ceph-mon[31243]: pgmap v837: 129 pgs: 129 active+clean; 515 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.3 MiB/s wr, 33 op/s 2024-09-16T00:33:23.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:23 smithi083 ceph-mon[35242]: pgmap v837: 129 pgs: 129 active+clean; 515 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.3 MiB/s wr, 33 op/s 2024-09-16T00:33:23.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:23 smithi060 ceph-mon[35185]: pgmap v837: 129 pgs: 129 active+clean; 515 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.3 MiB/s wr, 33 op/s 2024-09-16T00:33:24.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 48389 8.02 MB/sec execute 30 sec latency 36.388 ms 2024-09-16T00:33:25.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 49034 8.73 MB/sec execute 31 sec latency 62.614 ms 2024-09-16T00:33:25.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:25 smithi033 ceph-mon[31243]: pgmap v838: 129 pgs: 129 active+clean; 521 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 107 op/s 2024-09-16T00:33:25.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:25 smithi083 ceph-mon[35242]: pgmap v838: 129 pgs: 129 active+clean; 521 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 107 op/s 2024-09-16T00:33:25.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:25 smithi060 ceph-mon[35185]: pgmap v838: 129 pgs: 129 active+clean; 521 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 107 op/s 2024-09-16T00:33:26.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 49561 9.66 MB/sec execute 32 sec latency 52.618 ms 2024-09-16T00:33:27.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 50237 10.50 MB/sec execute 33 sec latency 50.068 ms 2024-09-16T00:33:27.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:27 smithi033 ceph-mon[31243]: pgmap v839: 129 pgs: 129 active+clean; 518 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 210 op/s 2024-09-16T00:33:27.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:33:27.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:27 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:33:27.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:33:27.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:33:27.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:33:27.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:27 smithi083 ceph-mon[35242]: pgmap v839: 129 pgs: 129 active+clean; 518 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 210 op/s 2024-09-16T00:33:27.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:33:27.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:27 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:33:27.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:33:27.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:33:27.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:33:27.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:27 smithi060 ceph-mon[35185]: pgmap v839: 129 pgs: 129 active+clean; 518 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 210 op/s 2024-09-16T00:33:27.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:33:27.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:27 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:33:27.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:33:27.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:33:27.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:33:28.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 50704 10.95 MB/sec execute 34 sec latency 74.397 ms 2024-09-16T00:33:28.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:28 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:33:28.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:28 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:33:28.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:28 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:33:28.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:28 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:33:28.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:28 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:33:28.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:28 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:33:28.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:28 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:33:28.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:28 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:33:28.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:28 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:33:29.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 51178 11.20 MB/sec execute 35 sec latency 45.991 ms 2024-09-16T00:33:29.319 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:29 smithi083 ceph-mon[35242]: pgmap v840: 129 pgs: 129 active+clean; 518 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.8 MiB/s wr, 215 op/s 2024-09-16T00:33:29.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:29 smithi033 ceph-mon[31243]: pgmap v840: 129 pgs: 129 active+clean; 518 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.8 MiB/s wr, 215 op/s 2024-09-16T00:33:29.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:29 smithi060 ceph-mon[35185]: pgmap v840: 129 pgs: 129 active+clean; 518 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.8 MiB/s wr, 215 op/s 2024-09-16T00:33:30.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 51661 11.41 MB/sec execute 36 sec latency 93.865 ms 2024-09-16T00:33:31.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 52401 12.38 MB/sec execute 37 sec latency 70.926 ms 2024-09-16T00:33:31.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:31 smithi033 ceph-mon[31243]: pgmap v841: 129 pgs: 129 active+clean; 520 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.4 MiB/s wr, 300 op/s 2024-09-16T00:33:31.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:31 smithi083 ceph-mon[35242]: pgmap v841: 129 pgs: 129 active+clean; 520 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.4 MiB/s wr, 300 op/s 2024-09-16T00:33:31.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:31 smithi060 ceph-mon[35185]: pgmap v841: 129 pgs: 129 active+clean; 520 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.4 MiB/s wr, 300 op/s 2024-09-16T00:33:32.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 53171 12.92 MB/sec execute 38 sec latency 51.542 ms 2024-09-16T00:33:33.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 53851 13.78 MB/sec execute 39 sec latency 55.143 ms 2024-09-16T00:33:33.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:33 smithi033 ceph-mon[31243]: pgmap v842: 129 pgs: 129 active+clean; 519 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 5.2 MiB/s wr, 385 op/s 2024-09-16T00:33:33.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:33 smithi083 ceph-mon[35242]: pgmap v842: 129 pgs: 129 active+clean; 519 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 5.2 MiB/s wr, 385 op/s 2024-09-16T00:33:33.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:33 smithi060 ceph-mon[35185]: pgmap v842: 129 pgs: 129 active+clean; 519 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 5.2 MiB/s wr, 385 op/s 2024-09-16T00:33:34.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 54500 14.42 MB/sec execute 40 sec latency 75.427 ms 2024-09-16T00:33:35.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 55006 14.71 MB/sec execute 41 sec latency 79.120 ms 2024-09-16T00:33:35.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:35 smithi033 ceph-mon[31243]: pgmap v843: 129 pgs: 129 active+clean; 510 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.6 MiB/s wr, 453 op/s 2024-09-16T00:33:35.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:35 smithi083 ceph-mon[35242]: pgmap v843: 129 pgs: 129 active+clean; 510 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.6 MiB/s wr, 453 op/s 2024-09-16T00:33:35.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:35 smithi060 ceph-mon[35185]: pgmap v843: 129 pgs: 129 active+clean; 510 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.6 MiB/s wr, 453 op/s 2024-09-16T00:33:36.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 55569 14.96 MB/sec execute 42 sec latency 75.260 ms 2024-09-16T00:33:37.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 56059 15.23 MB/sec execute 43 sec latency 84.991 ms 2024-09-16T00:33:37.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:37 smithi033 ceph-mon[31243]: pgmap v844: 129 pgs: 129 active+clean; 515 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.6 MiB/s wr, 427 op/s 2024-09-16T00:33:37.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:37 smithi083 ceph-mon[35242]: pgmap v844: 129 pgs: 129 active+clean; 515 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.6 MiB/s wr, 427 op/s 2024-09-16T00:33:37.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:37 smithi060 ceph-mon[35185]: pgmap v844: 129 pgs: 129 active+clean; 515 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.6 MiB/s wr, 427 op/s 2024-09-16T00:33:38.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 56717 15.57 MB/sec execute 44 sec latency 114.138 ms 2024-09-16T00:33:39.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 57417 15.98 MB/sec execute 45 sec latency 72.433 ms 2024-09-16T00:33:39.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:39 smithi033 ceph-mon[31243]: pgmap v845: 129 pgs: 129 active+clean; 517 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 348 op/s 2024-09-16T00:33:39.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:39 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:33:39.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:39 smithi083 ceph-mon[35242]: pgmap v845: 129 pgs: 129 active+clean; 517 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 348 op/s 2024-09-16T00:33:39.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:39 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:33:39.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:39 smithi060 ceph-mon[35185]: pgmap v845: 129 pgs: 129 active+clean; 517 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 348 op/s 2024-09-16T00:33:39.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:39 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:33:40.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 58250 16.98 MB/sec execute 46 sec latency 84.253 ms 2024-09-16T00:33:41.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 58774 17.11 MB/sec execute 47 sec latency 77.024 ms 2024-09-16T00:33:41.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:41 smithi033 ceph-mon[31243]: pgmap v846: 129 pgs: 129 active+clean; 525 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.8 MiB/s wr, 447 op/s 2024-09-16T00:33:41.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:41 smithi083 ceph-mon[35242]: pgmap v846: 129 pgs: 129 active+clean; 525 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.8 MiB/s wr, 447 op/s 2024-09-16T00:33:41.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:41 smithi060 ceph-mon[35185]: pgmap v846: 129 pgs: 129 active+clean; 525 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.8 MiB/s wr, 447 op/s 2024-09-16T00:33:42.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 59287 17.35 MB/sec execute 48 sec latency 58.153 ms 2024-09-16T00:33:43.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 59882 17.62 MB/sec execute 49 sec latency 82.685 ms 2024-09-16T00:33:43.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:43 smithi033 ceph-mon[31243]: pgmap v847: 129 pgs: 129 active+clean; 528 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.5 MiB/s wr, 391 op/s 2024-09-16T00:33:43.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:43 smithi083 ceph-mon[35242]: pgmap v847: 129 pgs: 129 active+clean; 528 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.5 MiB/s wr, 391 op/s 2024-09-16T00:33:43.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:43 smithi060 ceph-mon[35185]: pgmap v847: 129 pgs: 129 active+clean; 528 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.5 MiB/s wr, 391 op/s 2024-09-16T00:33:44.074 INFO:tasks.workunit.client.0.smithi033.stdout: 10 60394 17.79 MB/sec execute 50 sec latency 78.423 ms 2024-09-16T00:33:45.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 61121 18.07 MB/sec execute 51 sec latency 86.797 ms 2024-09-16T00:33:45.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:45 smithi033 ceph-mon[31243]: pgmap v848: 129 pgs: 129 active+clean; 540 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.8 MiB/s wr, 445 op/s 2024-09-16T00:33:45.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:45 smithi083 ceph-mon[35242]: pgmap v848: 129 pgs: 129 active+clean; 540 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.8 MiB/s wr, 445 op/s 2024-09-16T00:33:45.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:45 smithi060 ceph-mon[35185]: pgmap v848: 129 pgs: 129 active+clean; 540 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.8 MiB/s wr, 445 op/s 2024-09-16T00:33:46.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 61915 18.60 MB/sec execute 52 sec latency 48.261 ms 2024-09-16T00:33:47.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 62599 19.10 MB/sec execute 53 sec latency 244.444 ms 2024-09-16T00:33:47.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:47 smithi083 ceph-mon[35242]: pgmap v849: 129 pgs: 129 active+clean; 545 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.7 MiB/s wr, 416 op/s 2024-09-16T00:33:47.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:47 smithi060 ceph-mon[35185]: pgmap v849: 129 pgs: 129 active+clean; 545 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.7 MiB/s wr, 416 op/s 2024-09-16T00:33:47.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:47 smithi033 ceph-mon[31243]: pgmap v849: 129 pgs: 129 active+clean; 545 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.7 MiB/s wr, 416 op/s 2024-09-16T00:33:48.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 63006 19.13 MB/sec execute 54 sec latency 202.944 ms 2024-09-16T00:33:49.068 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:33:49.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 63566 19.46 MB/sec execute 55 sec latency 123.519 ms 2024-09-16T00:33:49.399 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:49.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:49 smithi083 ceph-mon[35242]: pgmap v850: 129 pgs: 129 active+clean; 545 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.2 MiB/s wr, 370 op/s 2024-09-16T00:33:49.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:49 smithi060 ceph-mon[35185]: pgmap v850: 129 pgs: 129 active+clean; 545 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.2 MiB/s wr, 370 op/s 2024-09-16T00:33:49.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:49 smithi033 ceph-mon[31243]: pgmap v850: 129 pgs: 129 active+clean; 545 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.2 MiB/s wr, 370 op/s 2024-09-16T00:33:50.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.987+0000 7f626e01d640 1 -- 172.21.15.33:0/2769800496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6268103780 msgr2=0x7f6268107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:50.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.987+0000 7f626e01d640 1 --2- 172.21.15.33:0/2769800496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6268103780 0x7f6268107bc0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f6258009930 tx=0x7f625802a420 comp rx=0 tx=0).stop 2024-09-16T00:33:50.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.987+0000 7f626e01d640 1 -- 172.21.15.33:0/2769800496 shutdown_connections 2024-09-16T00:33:50.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.987+0000 7f626e01d640 1 --2- 172.21.15.33:0/2769800496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6268108300 0x7f626810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.987+0000 7f626e01d640 1 --2- 172.21.15.33:0/2769800496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6268103780 0x7f6268107bc0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.987+0000 7f626e01d640 1 --2- 172.21.15.33:0/2769800496 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6268102dd0 0x7f62681031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.987+0000 7f626e01d640 1 -- 172.21.15.33:0/2769800496 >> 172.21.15.33:0/2769800496 conn(0x7f62680fc7d0 msgr2=0x7f62680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:50.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.987+0000 7f626e01d640 1 -- 172.21.15.33:0/2769800496 shutdown_connections 2024-09-16T00:33:50.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.987+0000 7f626e01d640 1 -- 172.21.15.33:0/2769800496 wait complete. 2024-09-16T00:33:50.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.988+0000 7f626e01d640 1 Processor -- start 2024-09-16T00:33:50.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.988+0000 7f626e01d640 1 -- start start 2024-09-16T00:33:50.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.989+0000 7f626e01d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6268102dd0 0x7f626819f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.989+0000 7f626e01d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6268103780 0x7f626819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.989+0000 7f62677fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6268102dd0 0x7f626819f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.989+0000 7f62677fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6268102dd0 0x7f626819f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:51090/0 (socket says 172.21.15.33:51090) 2024-09-16T00:33:50.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.989+0000 7f626e01d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6268108300 0x7f626810e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.989+0000 7f62677fe640 1 -- 172.21.15.33:0/3103688339 learned_addr learned my addr 172.21.15.33:0/3103688339 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:50.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.989+0000 7f626e01d640 1 -- 172.21.15.33:0/3103688339 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6268112050 con 0x7f6268108300 2024-09-16T00:33:50.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.989+0000 7f626e01d640 1 -- 172.21.15.33:0/3103688339 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6268111ed0 con 0x7f6268103780 2024-09-16T00:33:50.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.989+0000 7f626e01d640 1 -- 172.21.15.33:0/3103688339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f62681121d0 con 0x7f6268102dd0 2024-09-16T00:33:50.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.990+0000 7f6267fff640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6268108300 0x7f626810e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.990+0000 7f6266ffd640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6268103780 0x7f626819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.990+0000 7f62677fe640 1 -- 172.21.15.33:0/3103688339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6268103780 msgr2=0x7f626819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:50.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.990+0000 7f62677fe640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6268103780 0x7f626819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.990+0000 7f62677fe640 1 -- 172.21.15.33:0/3103688339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6268108300 msgr2=0x7f626810e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.990+0000 7f62677fe640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6268108300 0x7f626810e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.990+0000 7f62677fe640 1 -- 172.21.15.33:0/3103688339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626810e8f0 con 0x7f6268102dd0 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.990+0000 7f62677fe640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6268102dd0 0x7f626819f2a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f625c00e9f0 tx=0x7f625c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.990+0000 7f6264ff9640 1 -- 172.21.15.33:0/3103688339 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c00ccc0 con 0x7f6268102dd0 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.990+0000 7f6264ff9640 1 -- 172.21.15.33:0/3103688339 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f625c010910 con 0x7f6268102dd0 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.990+0000 7f626e01d640 1 -- 172.21.15.33:0/3103688339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f626810ebe0 con 0x7f6268102dd0 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.991+0000 7f6264ff9640 1 -- 172.21.15.33:0/3103688339 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c0188a0 con 0x7f6268102dd0 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.991+0000 7f626e01d640 1 -- 172.21.15.33:0/3103688339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f626819aea0 con 0x7f6268102dd0 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.993+0000 7f6264ff9640 1 -- 172.21.15.33:0/3103688339 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f625c010ac0 con 0x7f6268102dd0 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.993+0000 7f62703f6740 1 Processor -- start 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.993+0000 7f62703f6740 1 -- start start 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f62703f6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6225880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f62703f6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 0x5593d6230140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f62703f6740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d6232a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f62703f6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5593d618b2e0 con 0x5593d6225dc0 2024-09-16T00:33:50.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f62703f6740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5593d6132ce0 con 0x5593d6230680 2024-09-16T00:33:50.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f62703f6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5593d61872a0 con 0x5593d62254a0 2024-09-16T00:33:50.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f6264ff9640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62500a32c0 0x7f62500a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f62677fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 0x5593d6230140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f6267fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6225880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f62677fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 0x5593d6230140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46096/0 (socket says 172.21.15.33:46096) 2024-09-16T00:33:50.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f62677fe640 1 -- 172.21.15.33:0/4223938681 learned_addr learned my addr 172.21.15.33:0/4223938681 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:50.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f6267fff640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62500a32c0 0x7f62500a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f6266ffd640 1 --2- 172.21.15.33:0/4223938681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d6232a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f6264ff9640 1 -- 172.21.15.33:0/3103688339 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f625c0d01a0 con 0x7f6268102dd0 2024-09-16T00:33:50.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f6267fff640 1 -- 172.21.15.33:0/4223938681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 msgr2=0x5593d6232a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f6267fff640 1 --2- 172.21.15.33:0/4223938681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d6232a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f6267fff640 1 -- 172.21.15.33:0/4223938681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 msgr2=0x5593d6230140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f6267fff640 1 --2- 172.21.15.33:0/4223938681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 0x5593d6230140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f6267fff640 1 -- 172.21.15.33:0/4223938681 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593d6117440 con 0x5593d62254a0 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f6266ffd640 1 --2- 172.21.15.33:0/4223938681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d6232a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.994+0000 7f62677fe640 1 --2- 172.21.15.33:0/4223938681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 0x5593d6230140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.995+0000 7f6267fff640 1 --2- 172.21.15.33:0/4223938681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6225880 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6260011cd0 tx=0x7f626000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=687e490ed3e8d44 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.995+0000 7f6267fff640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62500a32c0 0x7f62500a5780 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f626000fc90 tx=0x7f626001e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.995+0000 7f624dffb640 1 -- 172.21.15.33:0/4223938681 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626001dbf0 con 0x5593d62254a0 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.995+0000 7f624dffb640 1 -- 172.21.15.33:0/4223938681 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6260009cf0 con 0x5593d62254a0 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.995+0000 7f624dffb640 1 -- 172.21.15.33:0/4223938681 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260005760 con 0x5593d62254a0 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.995+0000 7f62703f6740 1 -- 172.21.15.33:0/4223938681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 msgr2=0x5593d6225880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:50.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.996+0000 7f62703f6740 1 --2- 172.21.15.33:0/4223938681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6225880 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6260011cd0 tx=0x7f626000f790 comp rx=0 tx=0).stop 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.996+0000 7f62703f6740 1 -- 172.21.15.33:0/4223938681 shutdown_connections 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.996+0000 7f62703f6740 1 --2- 172.21.15.33:0/4223938681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d6232a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.996+0000 7f62703f6740 1 --2- 172.21.15.33:0/4223938681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 0x5593d6230140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.996+0000 7f62703f6740 1 --2- 172.21.15.33:0/4223938681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6225880 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.996+0000 7f62703f6740 1 -- 172.21.15.33:0/4223938681 >> 172.21.15.33:0/4223938681 conn(0x5593d6221a30 msgr2=0x5593d6221e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.996+0000 7f62703f6740 1 -- 172.21.15.33:0/4223938681 shutdown_connections 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.996+0000 7f62703f6740 1 -- 172.21.15.33:0/4223938681 wait complete. 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.996+0000 7f62703f6740 1 -- start start 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.996+0000 7f62703f6740 1 Processor -- start 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.996+0000 7f62703f6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d62254a0 0x5593d622ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.996+0000 7f62703f6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d6225dc0 0x5593d622d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f62703f6740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d622d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f62703f6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5593d62331f0 con 0x5593d62254a0 2024-09-16T00:33:50.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f62703f6740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5593d6146050 con 0x5593d6230680 2024-09-16T00:33:50.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f62703f6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5593d6233370 con 0x5593d6225dc0 2024-09-16T00:33:50.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f62703f6740 10 client.0 _ll_register_callbacks cb 0x5593d6186160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:50.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f62677fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d62254a0 0x5593d622ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f6266ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d6225dc0 0x5593d622d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f62677fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d62254a0 0x5593d622ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46110/0 (socket says 172.21.15.33:46110) 2024-09-16T00:33:50.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f6266ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d6225dc0 0x5593d622d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:51104/0 (socket says 172.21.15.33:51104) 2024-09-16T00:33:50.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f62677fe640 1 -- 172.21.15.33:0/2026586687 learned_addr learned my addr 172.21.15.33:0/2026586687 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:50.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f6267fff640 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d622d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f6266ffd640 1 -- 172.21.15.33:0/2026586687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 msgr2=0x5593d622d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:50.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f6266ffd640 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d622d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f6266ffd640 1 -- 172.21.15.33:0/2026586687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d62254a0 msgr2=0x5593d622ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:50.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f6266ffd640 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d62254a0 0x5593d622ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f6266ffd640 1 -- 172.21.15.33:0/2026586687 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593d5efc3c0 con 0x5593d6225dc0 2024-09-16T00:33:50.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f62677fe640 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d62254a0 0x5593d622ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:50.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f6266ffd640 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d6225dc0 0x5593d622d290 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f625802a940 tx=0x7f6258032e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdb198f1369da0ac server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:50.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f6246ffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:33:50.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.997+0000 7f6246ffd640 1 -- 172.21.15.33:0/2026586687 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258035040 con 0x5593d6225dc0 2024-09-16T00:33:50.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f62703f6740 1 -- 172.21.15.33:0/2026586687 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6260009cf0 con 0x5593d6225dc0 2024-09-16T00:33:50.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f6246ffd640 1 -- 172.21.15.33:0/2026586687 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f62580317b0 con 0x5593d6225dc0 2024-09-16T00:33:50.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f6246ffd640 1 -- 172.21.15.33:0/2026586687 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f625804f050 con 0x5593d6225dc0 2024-09-16T00:33:50.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f62703f6740 10 client.27844 fetch_fsmap learned FSMap version 33 2024-09-16T00:33:50.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f62703f6740 1 -- 172.21.15.33:0/2026586687 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5593d623ec10 con 0x5593d6225dc0 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f6246ffd640 1 -- 172.21.15.33:0/2026586687 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6260009cf0 con 0x5593d6225dc0 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f6246ffd640 1 -- 172.21.15.33:0/2026586687 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f6258004770 con 0x5593d6225dc0 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f62703f6740 10 client.27844 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f62703f6740 10 client.27844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f62703f6740 20 client.27844 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f62703f6740 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5593d6240020 0x5593d6242430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f62703f6740 4 client.27844 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.998+0000 7f62703f6740 1 -- 172.21.15.33:0/2026586687 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593d6198850 con 0x5593d6240020 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.999+0000 7f62677fe640 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5593d6240020 0x5593d6242430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.999+0000 7f62677fe640 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5593d6240020 0x5593d6242430 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d4fe8b01d2c56b6 server_cookie=99dec9d87bb28b75 in_seq=0 out_seq=0 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:49.999+0000 7f6246ffd640 10 client.27844 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f6246ffd640 1 -- 172.21.15.33:0/2026586687 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5593d6198850 con 0x5593d6240020 2024-09-16T00:33:50.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f6246ffd640 10 client.27844 handle_command_reply: tid=0 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f62703f6740 1 client.27844 shutdown 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f62703f6740 1 -- 172.21.15.33:0/2026586687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d6225dc0 msgr2=0x5593d622d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f62703f6740 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d6225dc0 0x5593d622d290 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f625802a940 tx=0x7f6258032e60 comp rx=0 tx=0).stop 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f62703f6740 1 -- 172.21.15.33:0/2026586687 shutdown_connections 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f62703f6740 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5593d6240020 0x5593d6242430 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f62703f6740 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d622d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f62703f6740 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d6225dc0 0x5593d622d290 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f62703f6740 1 --2- 172.21.15.33:0/2026586687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d62254a0 0x5593d622ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f6246ffd640 0 client.27844 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f62703f6740 1 -- 172.21.15.33:0/2026586687 >> 172.21.15.33:0/2026586687 conn(0x5593d6221a30 msgr2=0x5593d6220b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f62703f6740 1 -- 172.21.15.33:0/2026586687 shutdown_connections 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f62703f6740 1 -- 172.21.15.33:0/2026586687 wait complete. 2024-09-16T00:33:50.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.000+0000 7f62703f6740 20 client.27844 trim_cache size 0 max 16384 2024-09-16T00:33:50.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f62703f6740 1 Processor -- start 2024-09-16T00:33:50.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f62703f6740 1 -- start start 2024-09-16T00:33:50.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f62703f6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6247d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f62703f6740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6225dc0 0x5593d6248240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f62703f6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6230680 0x5593d622a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f62703f6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f62681c0820 con 0x5593d6230680 2024-09-16T00:33:50.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f62703f6740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f62681be0f0 con 0x5593d6225dc0 2024-09-16T00:33:50.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f62703f6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f62681bbf40 con 0x5593d62254a0 2024-09-16T00:33:50.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f6266ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6225dc0 0x5593d6248240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f62677fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6247d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f6266ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6225dc0 0x5593d6248240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:50772/0 (socket says 172.21.15.33:50772) 2024-09-16T00:33:50.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f62677fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6247d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:51118/0 (socket says 172.21.15.33:51118) 2024-09-16T00:33:50.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.020+0000 7f6266ffd640 1 -- 172.21.15.33:0/2213511770 learned_addr learned my addr 172.21.15.33:0/2213511770 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:50.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.021+0000 7f62677fe640 1 -- 172.21.15.33:0/2213511770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6225dc0 msgr2=0x5593d6248240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:50.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.021+0000 7f62677fe640 1 --2- 172.21.15.33:0/2213511770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6225dc0 0x5593d6248240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.021+0000 7f62677fe640 1 -- 172.21.15.33:0/2213511770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6230680 msgr2=0x5593d622a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:50.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.021+0000 7f62677fe640 1 --2- 172.21.15.33:0/2213511770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6230680 0x5593d622a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.021+0000 7f62677fe640 1 -- 172.21.15.33:0/2213511770 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593d6222ba0 con 0x5593d62254a0 2024-09-16T00:33:50.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.021+0000 7f62677fe640 1 --2- 172.21.15.33:0/2213511770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6247d00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f625c00c6a0 tx=0x7f625c009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15de03b36e124478 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:50.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.021+0000 7f624d7fa640 1 -- 172.21.15.33:0/2213511770 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c0a4f50 con 0x5593d62254a0 2024-09-16T00:33:50.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.021+0000 7f624d7fa640 1 -- 172.21.15.33:0/2213511770 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f625c096e50 con 0x5593d62254a0 2024-09-16T00:33:50.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.021+0000 7f624d7fa640 1 -- 172.21.15.33:0/2213511770 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c0bc310 con 0x5593d62254a0 2024-09-16T00:33:50.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.022+0000 7f62703f6740 1 -- 172.21.15.33:0/2213511770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 msgr2=0x5593d6247d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:50.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.022+0000 7f62703f6740 1 --2- 172.21.15.33:0/2213511770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6247d00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f625c00c6a0 tx=0x7f625c009700 comp rx=0 tx=0).stop 2024-09-16T00:33:50.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.022+0000 7f62703f6740 1 -- 172.21.15.33:0/2213511770 shutdown_connections 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.022+0000 7f62703f6740 1 --2- 172.21.15.33:0/2213511770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6230680 0x5593d622a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.022+0000 7f62703f6740 1 --2- 172.21.15.33:0/2213511770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6225dc0 0x5593d6248240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.022+0000 7f62703f6740 1 --2- 172.21.15.33:0/2213511770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6247d00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.022+0000 7f62703f6740 1 -- 172.21.15.33:0/2213511770 >> 172.21.15.33:0/2213511770 conn(0x5593d6221a30 msgr2=0x5593d622f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.022+0000 7f62703f6740 1 -- 172.21.15.33:0/2213511770 shutdown_connections 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.022+0000 7f62703f6740 1 -- 172.21.15.33:0/2213511770 wait complete. 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.023+0000 7f62703f6740 1 -- start start 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.023+0000 7f62703f6740 1 Processor -- start 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.023+0000 7f62703f6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6238dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.023+0000 7f62703f6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 0x5593d6236160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.023+0000 7f62703f6740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d62366e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.023+0000 7f62703f6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f62681bc5b0 con 0x5593d6225dc0 2024-09-16T00:33:50.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.023+0000 7f62703f6740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f62681bda20 con 0x5593d6230680 2024-09-16T00:33:50.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.023+0000 7f62703f6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f62681bcc80 con 0x5593d62254a0 2024-09-16T00:33:50.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.023+0000 7f62703f6740 10 client.0 _ll_register_callbacks cb 0x7f6258009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:50.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.024+0000 7f6266ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 0x5593d6236160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.024+0000 7f6266ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 0x5593d6236160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46116/0 (socket says 172.21.15.33:46116) 2024-09-16T00:33:50.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.024+0000 7f6266ffd640 1 -- 172.21.15.33:0/1483293491 learned_addr learned my addr 172.21.15.33:0/1483293491 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:50.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.024+0000 7f62677fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6238dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.024+0000 7f6267fff640 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d62366e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.024+0000 7f6267fff640 1 -- 172.21.15.33:0/1483293491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 msgr2=0x5593d6238dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:50.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.024+0000 7f6267fff640 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6238dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.024+0000 7f6267fff640 1 -- 172.21.15.33:0/1483293491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 msgr2=0x5593d6236160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:50.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.024+0000 7f6267fff640 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 0x5593d6236160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.024+0000 7f6267fff640 1 -- 172.21.15.33:0/1483293491 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593d6236f10 con 0x5593d6230680 2024-09-16T00:33:50.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.024+0000 7f6266ffd640 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 0x5593d6236160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:50.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.025+0000 7f6267fff640 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d62366e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6260009ab0 tx=0x7f6260011b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8413be8748367b0b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:50.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.025+0000 7f6246ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:33:50.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.025+0000 7f62677fe640 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6238dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:50.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.025+0000 7f6246ffd640 1 -- 172.21.15.33:0/1483293491 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260034070 con 0x5593d6230680 2024-09-16T00:33:50.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.025+0000 7f62703f6740 1 -- 172.21.15.33:0/1483293491 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f625c096e50 con 0x5593d6230680 2024-09-16T00:33:50.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.025+0000 7f6246ffd640 1 -- 172.21.15.33:0/1483293491 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6260011250 con 0x5593d6230680 2024-09-16T00:33:50.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.025+0000 7f6246ffd640 1 -- 172.21.15.33:0/1483293491 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6260040050 con 0x5593d6230680 2024-09-16T00:33:50.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.025+0000 7f62703f6740 10 client.27836 fetch_fsmap learned FSMap version 33 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.025+0000 7f62703f6740 1 -- 172.21.15.33:0/1483293491 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5593d62aad90 con 0x5593d6230680 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.025+0000 7f6246ffd640 1 -- 172.21.15.33:0/1483293491 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f625c096e50 con 0x5593d6230680 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.026+0000 7f6246ffd640 1 -- 172.21.15.33:0/1483293491 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f6260004e30 con 0x5593d6230680 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.026+0000 7f62703f6740 10 client.27836 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.026+0000 7f62703f6740 10 client.27836 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.026+0000 7f62703f6740 20 client.27836 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.026+0000 7f62703f6740 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5593d62ab370 0x5593d62ad760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.026+0000 7f62703f6740 4 client.27836 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.026+0000 7f62703f6740 1 -- 172.21.15.33:0/1483293491 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5593d6192d60 con 0x5593d62ab370 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.026+0000 7f62677fe640 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5593d62ab370 0x5593d62ad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f6246ffd640 10 client.27836 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f62677fe640 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5593d62ab370 0x5593d62ad760 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97454bf7afea4466 server_cookie=de00cf279640cdc6 in_seq=0 out_seq=0 2024-09-16T00:33:50.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f6246ffd640 1 -- 172.21.15.33:0/1483293491 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5593d6192d60 con 0x5593d62ab370 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f6246ffd640 10 client.27836 handle_command_reply: tid=0 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f62703f6740 1 client.27836 shutdown 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f62703f6740 1 -- 172.21.15.33:0/1483293491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 msgr2=0x5593d62366e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f62703f6740 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d62366e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6260009ab0 tx=0x7f6260011b80 comp rx=0 tx=0).stop 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f62703f6740 1 -- 172.21.15.33:0/1483293491 shutdown_connections 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f62703f6740 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5593d62ab370 0x5593d62ad760 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f62703f6740 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5593d6230680 0x5593d62366e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f6246ffd640 0 client.27836 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f62703f6740 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593d6225dc0 0x5593d6236160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.027+0000 7f62703f6740 1 --2- 172.21.15.33:0/1483293491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5593d62254a0 0x5593d6238dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.028+0000 7f62703f6740 1 -- 172.21.15.33:0/1483293491 >> 172.21.15.33:0/1483293491 conn(0x5593d6221a30 msgr2=0x5593d6220370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.028+0000 7f62703f6740 1 -- 172.21.15.33:0/1483293491 shutdown_connections 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.028+0000 7f62703f6740 1 -- 172.21.15.33:0/1483293491 wait complete. 2024-09-16T00:33:50.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.028+0000 7f62703f6740 20 client.27836 trim_cache size 0 max 16384 2024-09-16T00:33:50.032 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:33:50.032 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:33:50.032 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:33:50.032 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:33:50.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.029+0000 7f626e01d640 1 -- 172.21.15.33:0/3103688339 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62500a32c0 msgr2=0x7f62500a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:50.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.029+0000 7f626e01d640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62500a32c0 0x7f62500a5780 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f626000fc90 tx=0x7f626001e000 comp rx=0 tx=0).stop 2024-09-16T00:33:50.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.029+0000 7f626e01d640 1 -- 172.21.15.33:0/3103688339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6268102dd0 msgr2=0x7f626819f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:50.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.029+0000 7f626e01d640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6268102dd0 0x7f626819f2a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f625c00e9f0 tx=0x7f625c00edf0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.029+0000 7f626e01d640 1 -- 172.21.15.33:0/3103688339 shutdown_connections 2024-09-16T00:33:50.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.029+0000 7f626e01d640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f62500a32c0 0x7f62500a5780 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.029+0000 7f626e01d640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6268108300 0x7f626810e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.029+0000 7f626e01d640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6268103780 0x7f626819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.029+0000 7f626e01d640 1 --2- 172.21.15.33:0/3103688339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6268102dd0 0x7f626819f2a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:50.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.029+0000 7f626e01d640 1 -- 172.21.15.33:0/3103688339 >> 172.21.15.33:0/3103688339 conn(0x7f62680fc7d0 msgr2=0x7f6268100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:50.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.029+0000 7f626e01d640 1 -- 172.21.15.33:0/3103688339 shutdown_connections 2024-09-16T00:33:50.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:50.029+0000 7f626e01d640 1 -- 172.21.15.33:0/3103688339 wait complete. 2024-09-16T00:33:50.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 64036 19.37 MB/sec execute 56 sec latency 94.570 ms 2024-09-16T00:33:51.008 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:33:51.009 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:33:51.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 64569 19.55 MB/sec execute 57 sec latency 61.863 ms 2024-09-16T00:33:51.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:51 smithi033 ceph-mon[31243]: pgmap v851: 129 pgs: 129 active+clean; 551 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.7 MiB/s wr, 429 op/s 2024-09-16T00:33:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:51.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:51 smithi083 ceph-mon[35242]: pgmap v851: 129 pgs: 129 active+clean; 551 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.7 MiB/s wr, 429 op/s 2024-09-16T00:33:51.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:51 smithi060 ceph-mon[35185]: pgmap v851: 129 pgs: 129 active+clean; 551 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.7 MiB/s wr, 429 op/s 2024-09-16T00:33:52.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 65006 19.62 MB/sec execute 58 sec latency 375.337 ms 2024-09-16T00:33:52.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.261+0000 7f26831ba640 1 -- 172.21.15.33:0/1192538529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f267c109450 msgr2=0x7f267c110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:52.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.261+0000 7f26831ba640 1 --2- 172.21.15.33:0/1192538529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f267c109450 0x7f267c110250 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f267001a910 tx=0x7f267003b430 comp rx=0 tx=0).stop 2024-09-16T00:33:52.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.263+0000 7f26831ba640 1 -- 172.21.15.33:0/1192538529 shutdown_connections 2024-09-16T00:33:52.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.263+0000 7f26831ba640 1 --2- 172.21.15.33:0/1192538529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f267c11b140 0x7f267c11d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:52.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.263+0000 7f26831ba640 1 --2- 172.21.15.33:0/1192538529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f267c109830 0x7f267c109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:52.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.263+0000 7f26831ba640 1 --2- 172.21.15.33:0/1192538529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f267c109450 0x7f267c110250 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:52.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.263+0000 7f26831ba640 1 -- 172.21.15.33:0/1192538529 >> 172.21.15.33:0/1192538529 conn(0x7f267c06e140 msgr2=0x7f267c06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:52.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.268+0000 7f26831ba640 1 -- 172.21.15.33:0/1192538529 shutdown_connections 2024-09-16T00:33:52.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.272+0000 7f26831ba640 1 -- 172.21.15.33:0/1192538529 wait complete. 2024-09-16T00:33:52.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.273+0000 7f26831ba640 1 Processor -- start 2024-09-16T00:33:52.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.281+0000 7f26831ba640 1 -- start start 2024-09-16T00:33:52.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.281+0000 7f26831ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f267c109450 0x7f267c11a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:52.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.281+0000 7f26831ba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f267c109830 0x7f267c116930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:52.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.281+0000 7f26831ba640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f267c11b140 0x7f267c116e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:52.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.281+0000 7f26831ba640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f267c113320 con 0x7f267c11b140 2024-09-16T00:33:52.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.281+0000 7f26831ba640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f267c1131a0 con 0x7f267c109450 2024-09-16T00:33:52.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.281+0000 7f26831ba640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f267c1134a0 con 0x7f267c109830 2024-09-16T00:33:52.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.283+0000 7f267bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f267c109830 0x7f267c116930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:52.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.283+0000 7f267bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f267c109830 0x7f267c116930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:58840/0 (socket says 172.21.15.33:58840) 2024-09-16T00:33:52.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.283+0000 7f267bfff640 1 -- 172.21.15.33:0/4164112232 learned_addr learned my addr 172.21.15.33:0/4164112232 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:52.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.283+0000 7f2680f56640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f267c109450 0x7f267c11a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:52.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.284+0000 7f267bfff640 1 -- 172.21.15.33:0/4164112232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f267c109450 msgr2=0x7f267c11a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:52.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.284+0000 7f267bfff640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f267c109450 0x7f267c11a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:52.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.284+0000 7f267bfff640 1 -- 172.21.15.33:0/4164112232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f267c11b140 msgr2=0x7f267c116e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:52.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.284+0000 7f267bfff640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f267c11b140 0x7f267c116e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:52.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.284+0000 7f267bfff640 1 -- 172.21.15.33:0/4164112232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f267c1175a0 con 0x7f267c109830 2024-09-16T00:33:52.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.285+0000 7f267bfff640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f267c109830 0x7f267c116930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f267400ba50 tx=0x7f267400bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:52.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.287+0000 7f2679ffb640 1 -- 172.21.15.33:0/4164112232 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2674019070 con 0x7f267c109830 2024-09-16T00:33:52.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.287+0000 7f2679ffb640 1 -- 172.21.15.33:0/4164112232 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f267400cbe0 con 0x7f267c109830 2024-09-16T00:33:52.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.287+0000 7f2679ffb640 1 -- 172.21.15.33:0/4164112232 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2674007730 con 0x7f267c109830 2024-09-16T00:33:52.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.287+0000 7f26831ba640 1 -- 172.21.15.33:0/4164112232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f267c1ab170 con 0x7f267c109830 2024-09-16T00:33:52.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.287+0000 7f26831ba640 1 -- 172.21.15.33:0/4164112232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f267c1ab570 con 0x7f267c109830 2024-09-16T00:33:52.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.292+0000 7f26831ba640 1 -- 172.21.15.33:0/4164112232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f267c073b00 con 0x7f267c109830 2024-09-16T00:33:52.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.296+0000 7f2679ffb640 1 -- 172.21.15.33:0/4164112232 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2674014a40 con 0x7f267c109830 2024-09-16T00:33:52.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.297+0000 7f2679ffb640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26640a3450 0x7f26640a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:52.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.298+0000 7f2679ffb640 1 -- 172.21.15.33:0/4164112232 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f26740cf550 con 0x7f267c109830 2024-09-16T00:33:52.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.303+0000 7f2680f56640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26640a3450 0x7f26640a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:52.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.304+0000 7f2679ffb640 1 -- 172.21.15.33:0/4164112232 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f267409be10 con 0x7f267c109830 2024-09-16T00:33:52.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.310+0000 7f2680f56640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26640a3450 0x7f26640a5910 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2670005ab0 tx=0x7f2670000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:52.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.737+0000 7f26831ba640 1 -- 172.21.15.33:0/4164112232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f267c109c90 con 0x7f267c109830 2024-09-16T00:33:52.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.739+0000 7f2679ffb640 1 -- 172.21.15.33:0/4164112232 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f267409b710 con 0x7f267c109830 2024-09-16T00:33:52.739 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:33:52.739 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:33:52.740 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:33:52.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.745+0000 7f26831ba640 1 -- 172.21.15.33:0/4164112232 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26640a3450 msgr2=0x7f26640a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:52.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.745+0000 7f26831ba640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26640a3450 0x7f26640a5910 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2670005ab0 tx=0x7f2670000bc0 comp rx=0 tx=0).stop 2024-09-16T00:33:52.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.745+0000 7f26831ba640 1 -- 172.21.15.33:0/4164112232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f267c109830 msgr2=0x7f267c116930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:52.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.745+0000 7f26831ba640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f267c109830 0x7f267c116930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f267400ba50 tx=0x7f267400bf10 comp rx=0 tx=0).stop 2024-09-16T00:33:52.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.745+0000 7f26831ba640 1 -- 172.21.15.33:0/4164112232 shutdown_connections 2024-09-16T00:33:52.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.745+0000 7f26831ba640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f26640a3450 0x7f26640a5910 secure :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2670005ab0 tx=0x7f2670000bc0 comp rx=0 tx=0).stop 2024-09-16T00:33:52.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.745+0000 7f26831ba640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f267c11b140 0x7f267c116e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:52.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.745+0000 7f26831ba640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f267c109830 0x7f267c116930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:52.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.745+0000 7f26831ba640 1 --2- 172.21.15.33:0/4164112232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f267c109450 0x7f267c11a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:52.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.745+0000 7f26831ba640 1 -- 172.21.15.33:0/4164112232 >> 172.21.15.33:0/4164112232 conn(0x7f267c06e140 msgr2=0x7f267c072300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:52.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.746+0000 7f26831ba640 1 -- 172.21.15.33:0/4164112232 shutdown_connections 2024-09-16T00:33:52.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:52.747+0000 7f26831ba640 1 -- 172.21.15.33:0/4164112232 wait complete. 2024-09-16T00:33:53.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 65042 19.32 MB/sec execute 59 sec latency 1193.339 ms 2024-09-16T00:33:53.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:53 smithi083 ceph-mon[35242]: pgmap v852: 129 pgs: 129 active+clean; 559 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.6 MiB/s wr, 402 op/s 2024-09-16T00:33:53.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:53 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4164112232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:33:53.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:53 smithi060 ceph-mon[35185]: pgmap v852: 129 pgs: 129 active+clean; 559 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.6 MiB/s wr, 402 op/s 2024-09-16T00:33:53.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:53 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4164112232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:33:53.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:53 smithi033 ceph-mon[31243]: pgmap v852: 129 pgs: 129 active+clean; 559 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.6 MiB/s wr, 402 op/s 2024-09-16T00:33:53.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:53 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4164112232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:33:54.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 65272 19.16 MB/sec execute 60 sec latency 590.762 ms 2024-09-16T00:33:54.226 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:33:54.691 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:55.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 65668 19.12 MB/sec execute 61 sec latency 135.746 ms 2024-09-16T00:33:55.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:55 smithi033 ceph-mon[31243]: pgmap v853: 129 pgs: 129 active+clean; 582 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 6.7 MiB/s wr, 518 op/s 2024-09-16T00:33:55.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:55 smithi083 ceph-mon[35242]: pgmap v853: 129 pgs: 129 active+clean; 582 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 6.7 MiB/s wr, 518 op/s 2024-09-16T00:33:55.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:55 smithi060 ceph-mon[35185]: pgmap v853: 129 pgs: 129 active+clean; 582 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 6.7 MiB/s wr, 518 op/s 2024-09-16T00:33:56.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 66191 19.29 MB/sec execute 62 sec latency 81.642 ms 2024-09-16T00:33:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.528+0000 7f016b0eb640 1 -- 172.21.15.33:0/1435516564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0164106f40 msgr2=0x7f016410eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.528+0000 7f016b0eb640 1 --2- 172.21.15.33:0/1435516564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0164106f40 0x7f016410eac0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f015c009ca0 tx=0x7f015c02a8c0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.528+0000 7f016b0eb640 1 -- 172.21.15.33:0/1435516564 shutdown_connections 2024-09-16T00:33:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.528+0000 7f016b0eb640 1 --2- 172.21.15.33:0/1435516564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0164106f40 0x7f016410eac0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.528+0000 7f016b0eb640 1 --2- 172.21.15.33:0/1435516564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01641027c0 0x7f0164106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.528+0000 7f016b0eb640 1 --2- 172.21.15.33:0/1435516564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0164101e10 0x7f01641021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.528+0000 7f016b0eb640 1 -- 172.21.15.33:0/1435516564 >> 172.21.15.33:0/1435516564 conn(0x7f01640fd8d0 msgr2=0x7f01640ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.528+0000 7f016b0eb640 1 -- 172.21.15.33:0/1435516564 shutdown_connections 2024-09-16T00:33:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.528+0000 7f016b0eb640 1 -- 172.21.15.33:0/1435516564 wait complete. 2024-09-16T00:33:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.529+0000 7f016b0eb640 1 Processor -- start 2024-09-16T00:33:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.529+0000 7f016b0eb640 1 -- start start 2024-09-16T00:33:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.529+0000 7f016b0eb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0164101e10 0x7f01641a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.529+0000 7f016b0eb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01641027c0 0x7f01641a3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.529+0000 7f016b0eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0164106f40 0x7f016410dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.529+0000 7f016b0eb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0164111fa0 con 0x7f01641027c0 2024-09-16T00:33:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.529+0000 7f016b0eb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0164111e20 con 0x7f0164101e10 2024-09-16T00:33:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f016b0eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0164112120 con 0x7f0164106f40 2024-09-16T00:33:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0163fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01641027c0 0x7f01641a3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0163fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01641027c0 0x7f01641a3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45900/0 (socket says 172.21.15.33:45900) 2024-09-16T00:33:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0163fff640 1 -- 172.21.15.33:0/2746744388 learned_addr learned my addr 172.21.15.33:0/2746744388 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0168e87640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0164101e10 0x7f01641a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0169688640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0164106f40 0x7f016410dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0163fff640 1 -- 172.21.15.33:0/2746744388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0164106f40 msgr2=0x7f016410dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0163fff640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0164106f40 0x7f016410dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0163fff640 1 -- 172.21.15.33:0/2746744388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0164101e10 msgr2=0x7f01641a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0163fff640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0164101e10 0x7f01641a36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0163fff640 1 -- 172.21.15.33:0/2746744388 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f016410e420 con 0x7f01641027c0 2024-09-16T00:33:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0168e87640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0164101e10 0x7f01641a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0163fff640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01641027c0 0x7f01641a3be0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f015400e9f0 tx=0x7f015400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0161ffb640 1 -- 172.21.15.33:0/2746744388 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015400cd80 con 0x7f01641027c0 2024-09-16T00:33:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0161ffb640 1 -- 172.21.15.33:0/2746744388 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0154010910 con 0x7f01641027c0 2024-09-16T00:33:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f016b0eb640 1 -- 172.21.15.33:0/2746744388 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f016410e710 con 0x7f01641027c0 2024-09-16T00:33:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.530+0000 7f0161ffb640 1 -- 172.21.15.33:0/2746744388 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0154018990 con 0x7f01641027c0 2024-09-16T00:33:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.531+0000 7f016b0eb640 1 -- 172.21.15.33:0/2746744388 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01641ab5f0 con 0x7f01641027c0 2024-09-16T00:33:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f0161ffb640 1 -- 172.21.15.33:0/2746744388 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0154010ac0 con 0x7f01641027c0 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f016d4c4740 1 Processor -- start 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f016d4c4740 1 -- start start 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f016d4c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x564095194090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f016d4c4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x564095194a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f016d4c4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56409519f370 0x5640951a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f016d4c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5640950fa2f0 con 0x564095195b40 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f016d4c4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5640950a1cf0 con 0x5640951945d0 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f016d4c4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5640950f62b0 con 0x56409519f370 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f0169688640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x564095194090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f0168e87640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x564095194a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f0169688640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x564095194090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45902/0 (socket says 172.21.15.33:45902) 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f0169688640 1 -- 172.21.15.33:0/3776173520 learned_addr learned my addr 172.21.15.33:0/3776173520 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f0163fff640 1 --2- 172.21.15.33:0/3776173520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56409519f370 0x5640951a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f0169688640 1 -- 172.21.15.33:0/3776173520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56409519f370 msgr2=0x5640951a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.534+0000 7f0169688640 1 --2- 172.21.15.33:0/3776173520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56409519f370 0x5640951a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.535+0000 7f0169688640 1 -- 172.21.15.33:0/3776173520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 msgr2=0x564095194a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.535+0000 7f0169688640 1 --2- 172.21.15.33:0/3776173520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x564095194a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.535+0000 7f0169688640 1 -- 172.21.15.33:0/3776173520 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640950863a0 con 0x564095195b40 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.535+0000 7f0163fff640 1 --2- 172.21.15.33:0/3776173520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56409519f370 0x5640951a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.535+0000 7f0161ffb640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01500a3240 0x7f01500a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.535+0000 7f0169688640 1 --2- 172.21.15.33:0/3776173520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x564095194090 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f015c009ca0 tx=0x7f015c005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45bce35bd1255de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.535+0000 7f0168e87640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01500a3240 0x7f01500a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.535+0000 7f0161ffb640 1 -- 172.21.15.33:0/2746744388 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f01540d0020 con 0x7f01641027c0 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.535+0000 7f015affd640 1 -- 172.21.15.33:0/3776173520 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c031db0 con 0x564095195b40 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.535+0000 7f015affd640 1 -- 172.21.15.33:0/3776173520 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f015c036040 con 0x564095195b40 2024-09-16T00:33:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.535+0000 7f015affd640 1 -- 172.21.15.33:0/3776173520 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c0042a0 con 0x564095195b40 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.536+0000 7f016d4c4740 1 -- 172.21.15.33:0/3776173520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 msgr2=0x564095194090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.536+0000 7f016d4c4740 1 --2- 172.21.15.33:0/3776173520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x564095194090 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f015c009ca0 tx=0x7f015c005f10 comp rx=0 tx=0).stop 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.536+0000 7f0168e87640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01500a3240 0x7f01500a5700 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x564095191ac0 tx=0x7f014c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.536+0000 7f016d4c4740 1 -- 172.21.15.33:0/3776173520 shutdown_connections 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.536+0000 7f016d4c4740 1 --2- 172.21.15.33:0/3776173520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56409519f370 0x5640951a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.536+0000 7f016d4c4740 1 --2- 172.21.15.33:0/3776173520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x564095194a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.536+0000 7f016d4c4740 1 --2- 172.21.15.33:0/3776173520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x564095194090 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.536+0000 7f016d4c4740 1 -- 172.21.15.33:0/3776173520 >> 172.21.15.33:0/3776173520 conn(0x56409515d310 msgr2=0x564095190c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.536+0000 7f016d4c4740 1 -- 172.21.15.33:0/3776173520 shutdown_connections 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.536+0000 7f016d4c4740 1 -- 172.21.15.33:0/3776173520 wait complete. 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f016d4c4740 1 -- start start 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f016d4c4740 1 Processor -- start 2024-09-16T00:33:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f016d4c4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x56409519ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f016d4c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x56409519bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f016d4c4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56409519f370 0x56409519c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f016d4c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5640951a1e90 con 0x564095195b40 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f016d4c4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5640950b5060 con 0x5640951945d0 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f016d4c4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5640951a2010 con 0x56409519f370 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f016d4c4740 10 client.0 _ll_register_callbacks cb 0x564094e088c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f0163fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x56409519bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f0168e87640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x56409519ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f0168e87640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x56409519ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:52536/0 (socket says 172.21.15.33:52536) 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f0163fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x56409519bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45918/0 (socket says 172.21.15.33:45918) 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f0169688640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56409519f370 0x56409519c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.537+0000 7f0168e87640 1 -- 172.21.15.33:0/1953937216 learned_addr learned my addr 172.21.15.33:0/1953937216 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.538+0000 7f0168e87640 1 -- 172.21.15.33:0/1953937216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56409519f370 msgr2=0x56409519c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.538+0000 7f0168e87640 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56409519f370 0x56409519c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.538+0000 7f0168e87640 1 -- 172.21.15.33:0/1953937216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 msgr2=0x56409519bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.538+0000 7f0168e87640 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x56409519bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.538+0000 7f0168e87640 1 -- 172.21.15.33:0/1953937216 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564094e6b3c0 con 0x5640951945d0 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.538+0000 7f0169688640 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56409519f370 0x56409519c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.538+0000 7f0163fff640 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x56409519bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.538+0000 7f0168e87640 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x56409519ecd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f014c005bc0 tx=0x7f014c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43357df5ccec5500 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.538+0000 7f0143fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.538+0000 7f0143fff640 1 -- 172.21.15.33:0/1953937216 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c00f770 con 0x5640951945d0 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.538+0000 7f016d4c4740 1 -- 172.21.15.33:0/1953937216 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f015c036040 con 0x5640951945d0 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.538+0000 7f0143fff640 1 -- 172.21.15.33:0/1953937216 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f014c00f910 con 0x5640951945d0 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.539+0000 7f0143fff640 1 -- 172.21.15.33:0/1953937216 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f014c00fb10 con 0x5640951945d0 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.539+0000 7f016d4c4740 10 client.27851 fetch_fsmap learned FSMap version 33 2024-09-16T00:33:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.539+0000 7f016d4c4740 1 -- 172.21.15.33:0/1953937216 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5640951ad930 con 0x5640951945d0 2024-09-16T00:33:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.539+0000 7f0143fff640 1 -- 172.21.15.33:0/1953937216 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f015c036040 con 0x5640951945d0 2024-09-16T00:33:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.539+0000 7f0143fff640 1 -- 172.21.15.33:0/1953937216 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f014c027070 con 0x5640951945d0 2024-09-16T00:33:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.539+0000 7f016d4c4740 10 client.27851 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:33:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.539+0000 7f016d4c4740 10 client.27851 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:33:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.539+0000 7f016d4c4740 20 client.27851 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.539+0000 7f016d4c4740 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5640951aedf0 0x5640951b1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.539+0000 7f016d4c4740 4 client.27851 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:33:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.539+0000 7f016d4c4740 1 -- 172.21.15.33:0/1953937216 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564095107860 con 0x5640951aedf0 2024-09-16T00:33:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.539+0000 7f0163fff640 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5640951aedf0 0x5640951b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.540+0000 7f0163fff640 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5640951aedf0 0x5640951b1240 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ba2644356414d13 server_cookie=ededf8298ccabeb0 in_seq=0 out_seq=0 2024-09-16T00:33:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.540+0000 7f0143fff640 10 client.27851 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:33:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f0143fff640 1 -- 172.21.15.33:0/1953937216 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564095107860 con 0x5640951aedf0 2024-09-16T00:33:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f0143fff640 10 client.27851 handle_command_reply: tid=0 2024-09-16T00:33:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f016d4c4740 1 client.27851 shutdown 2024-09-16T00:33:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f016d4c4740 1 -- 172.21.15.33:0/1953937216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 msgr2=0x56409519ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f016d4c4740 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x56409519ecd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f014c005bc0 tx=0x7f014c004730 comp rx=0 tx=0).stop 2024-09-16T00:33:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f016d4c4740 1 -- 172.21.15.33:0/1953937216 shutdown_connections 2024-09-16T00:33:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f016d4c4740 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5640951aedf0 0x5640951b1240 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f016d4c4740 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56409519f370 0x56409519c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f0143fff640 0 client.27851 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:33:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f016d4c4740 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x56409519bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f016d4c4740 1 --2- 172.21.15.33:0/1953937216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x56409519ecd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f016d4c4740 1 -- 172.21.15.33:0/1953937216 >> 172.21.15.33:0/1953937216 conn(0x56409515d310 msgr2=0x564095196930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f016d4c4740 1 -- 172.21.15.33:0/1953937216 shutdown_connections 2024-09-16T00:33:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f016d4c4740 1 -- 172.21.15.33:0/1953937216 wait complete. 2024-09-16T00:33:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.542+0000 7f016d4c4740 20 client.27851 trim_cache size 0 max 16384 2024-09-16T00:33:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.556+0000 7f016d4c4740 1 Processor -- start 2024-09-16T00:33:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.556+0000 7f016d4c4740 1 -- start start 2024-09-16T00:33:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f016d4c4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5640951945d0 0x5640951ab160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f016d4c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x5640951b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f016d4c4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56409519f370 0x5640951b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f016d4c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01641b7f80 con 0x564095195b40 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f016d4c4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f01641b5850 con 0x56409519f370 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f016d4c4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f01641b36a0 con 0x5640951945d0 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f0163fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x5640951b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f0163fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x5640951b6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45932/0 (socket says 172.21.15.33:45932) 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f0163fff640 1 -- 172.21.15.33:0/2337520943 learned_addr learned my addr 172.21.15.33:0/2337520943 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f0168e87640 1 --2- 172.21.15.33:0/2337520943 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5640951945d0 0x5640951ab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f0169688640 1 --2- 172.21.15.33:0/2337520943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56409519f370 0x5640951b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f0163fff640 1 -- 172.21.15.33:0/2337520943 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5640951945d0 msgr2=0x5640951ab160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f0163fff640 1 --2- 172.21.15.33:0/2337520943 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5640951945d0 0x5640951ab160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f0163fff640 1 -- 172.21.15.33:0/2337520943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56409519f370 msgr2=0x5640951b70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f0163fff640 1 --2- 172.21.15.33:0/2337520943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56409519f370 0x5640951b70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f0163fff640 1 -- 172.21.15.33:0/2337520943 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56409519c850 con 0x564095195b40 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.557+0000 7f0168e87640 1 --2- 172.21.15.33:0/2337520943 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5640951945d0 0x5640951ab160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.558+0000 7f0163fff640 1 --2- 172.21.15.33:0/2337520943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x5640951b6b60 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f015409c200 tx=0x7f01540cd570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf8178613c47600 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.558+0000 7f015a7fc640 1 -- 172.21.15.33:0/2337520943 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01540a4e30 con 0x564095195b40 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.558+0000 7f015a7fc640 1 -- 172.21.15.33:0/2337520943 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f015409a310 con 0x564095195b40 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.558+0000 7f015a7fc640 1 -- 172.21.15.33:0/2337520943 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01540bc2b0 con 0x564095195b40 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.559+0000 7f016d4c4740 1 -- 172.21.15.33:0/2337520943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 msgr2=0x5640951b6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.559+0000 7f016d4c4740 1 --2- 172.21.15.33:0/2337520943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x5640951b6b60 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f015409c200 tx=0x7f01540cd570 comp rx=0 tx=0).stop 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.559+0000 7f016d4c4740 1 -- 172.21.15.33:0/2337520943 shutdown_connections 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.559+0000 7f016d4c4740 1 --2- 172.21.15.33:0/2337520943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56409519f370 0x5640951b70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.559+0000 7f016d4c4740 1 --2- 172.21.15.33:0/2337520943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564095195b40 0x5640951b6b60 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.559+0000 7f016d4c4740 1 --2- 172.21.15.33:0/2337520943 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5640951945d0 0x5640951ab160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.559+0000 7f016d4c4740 1 -- 172.21.15.33:0/2337520943 >> 172.21.15.33:0/2337520943 conn(0x56409515d310 msgr2=0x56409519a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.559+0000 7f016d4c4740 1 -- 172.21.15.33:0/2337520943 shutdown_connections 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.559+0000 7f016d4c4740 1 -- 172.21.15.33:0/2337520943 wait complete. 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.559+0000 7f016d4c4740 1 -- start start 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.559+0000 7f016d4c4740 1 Processor -- start 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.560+0000 7f016d4c4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x5640951a75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.560+0000 7f016d4c4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564095195b40 0x5640951a7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.560+0000 7f016d4c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56409519f370 0x564095198a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.560+0000 7f016d4c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01641b3d10 con 0x56409519f370 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.560+0000 7f016d4c4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f01641b5180 con 0x5640951945d0 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.560+0000 7f016d4c4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f01641b43e0 con 0x564095195b40 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.560+0000 7f016d4c4740 10 client.0 _ll_register_callbacks cb 0x7f014c009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.560+0000 7f0168e87640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x5640951a75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.560+0000 7f0168e87640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x5640951a75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:52544/0 (socket says 172.21.15.33:52544) 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.560+0000 7f0168e87640 1 -- 172.21.15.33:0/2425935401 learned_addr learned my addr 172.21.15.33:0/2425935401 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.560+0000 7f0163fff640 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564095195b40 0x5640951a7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.560+0000 7f0169688640 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56409519f370 0x564095198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f0163fff640 1 -- 172.21.15.33:0/2425935401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 msgr2=0x5640951a75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f0163fff640 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x5640951a75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f0163fff640 1 -- 172.21.15.33:0/2425935401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56409519f370 msgr2=0x564095198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f0163fff640 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56409519f370 0x564095198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f0163fff640 1 -- 172.21.15.33:0/2425935401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564095199280 con 0x564095195b40 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f0163fff640 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564095195b40 0x5640951a7af0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x564095191f30 tx=0x7f015409a4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a8b6e8845de8c4f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f0143fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f0143fff640 1 -- 172.21.15.33:0/2425935401 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01540bc290 con 0x564095195b40 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f0143fff640 1 -- 172.21.15.33:0/2425935401 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f01540d2110 con 0x564095195b40 2024-09-16T00:33:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f016d4c4740 1 -- 172.21.15.33:0/2425935401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f015409a310 con 0x564095195b40 2024-09-16T00:33:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f0143fff640 1 -- 172.21.15.33:0/2425935401 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f01540d2310 con 0x564095195b40 2024-09-16T00:33:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f016d4c4740 10 client.27871 fetch_fsmap learned FSMap version 33 2024-09-16T00:33:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.561+0000 7f016d4c4740 1 -- 172.21.15.33:0/2425935401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564095219750 con 0x564095195b40 2024-09-16T00:33:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.562+0000 7f0143fff640 1 -- 172.21.15.33:0/2425935401 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f015409a310 con 0x564095195b40 2024-09-16T00:33:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.562+0000 7f0143fff640 1 -- 172.21.15.33:0/2425935401 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f01540cdbf0 con 0x564095195b40 2024-09-16T00:33:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.562+0000 7f016d4c4740 10 client.27871 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.562+0000 7f016d4c4740 10 client.27871 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.562+0000 7f016d4c4740 20 client.27871 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.562+0000 7f016d4c4740 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x564095219d90 0x56409521c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.562+0000 7f016d4c4740 4 client.27871 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.562+0000 7f016d4c4740 1 -- 172.21.15.33:0/2425935401 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564095101d70 con 0x564095219d90 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.562+0000 7f0168e87640 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x564095219d90 0x56409521c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.562+0000 7f0143fff640 10 client.27871 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.562+0000 7f0168e87640 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x564095219d90 0x56409521c1b0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a39379473fb31880 server_cookie=ced7d9793eb12ca2 in_seq=0 out_seq=0 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f0143fff640 1 -- 172.21.15.33:0/2425935401 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564095101d70 con 0x564095219d90 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f0143fff640 10 client.27871 handle_command_reply: tid=0 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f016d4c4740 1 client.27871 shutdown 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f016d4c4740 1 -- 172.21.15.33:0/2425935401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564095195b40 msgr2=0x5640951a7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f016d4c4740 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564095195b40 0x5640951a7af0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x564095191f30 tx=0x7f015409a4c0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f016d4c4740 1 -- 172.21.15.33:0/2425935401 shutdown_connections 2024-09-16T00:33:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f016d4c4740 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x564095219d90 0x56409521c1b0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f016d4c4740 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56409519f370 0x564095198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f016d4c4740 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564095195b40 0x5640951a7af0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f016d4c4740 1 --2- 172.21.15.33:0/2425935401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5640951945d0 0x5640951a75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f0143fff640 0 client.27871 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:33:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f016d4c4740 1 -- 172.21.15.33:0/2425935401 >> 172.21.15.33:0/2425935401 conn(0x56409515d310 msgr2=0x564095196110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f016d4c4740 1 -- 172.21.15.33:0/2425935401 shutdown_connections 2024-09-16T00:33:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f016d4c4740 1 -- 172.21.15.33:0/2425935401 wait complete. 2024-09-16T00:33:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.566+0000 7f016d4c4740 20 client.27871 trim_cache size 0 max 16384 2024-09-16T00:33:56.569 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:33:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.567+0000 7f016b0eb640 1 -- 172.21.15.33:0/2746744388 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01500a3240 msgr2=0x7f01500a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.567+0000 7f016b0eb640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01500a3240 0x7f01500a5700 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x564095191ac0 tx=0x7f014c005f90 comp rx=0 tx=0).stop 2024-09-16T00:33:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.567+0000 7f016b0eb640 1 -- 172.21.15.33:0/2746744388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01641027c0 msgr2=0x7f01641a3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.567+0000 7f016b0eb640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01641027c0 0x7f01641a3be0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f015400e9f0 tx=0x7f015400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.567+0000 7f016b0eb640 1 -- 172.21.15.33:0/2746744388 shutdown_connections 2024-09-16T00:33:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.567+0000 7f016b0eb640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f01500a3240 0x7f01500a5700 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.567+0000 7f016b0eb640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0164106f40 0x7f016410dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.567+0000 7f016b0eb640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01641027c0 0x7f01641a3be0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.567+0000 7f016b0eb640 1 --2- 172.21.15.33:0/2746744388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0164101e10 0x7f01641a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.567+0000 7f016b0eb640 1 -- 172.21.15.33:0/2746744388 >> 172.21.15.33:0/2746744388 conn(0x7f01640fd8d0 msgr2=0x7f01640ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.567+0000 7f016b0eb640 1 -- 172.21.15.33:0/2746744388 shutdown_connections 2024-09-16T00:33:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:56.567+0000 7f016b0eb640 1 -- 172.21.15.33:0/2746744388 wait complete. 2024-09-16T00:33:57.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 66676 19.51 MB/sec execute 63 sec latency 266.315 ms 2024-09-16T00:33:57.345 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:33:57.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:57 smithi033 ceph-mon[31243]: pgmap v854: 129 pgs: 129 active+clean; 573 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 11 MiB/s wr, 573 op/s 2024-09-16T00:33:57.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:57 smithi083 ceph-mon[35242]: pgmap v854: 129 pgs: 129 active+clean; 573 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 11 MiB/s wr, 573 op/s 2024-09-16T00:33:57.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:57 smithi060 ceph-mon[35185]: pgmap v854: 129 pgs: 129 active+clean; 573 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 11 MiB/s wr, 573 op/s 2024-09-16T00:33:57.777 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:33:58.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 66949 19.40 MB/sec execute 64 sec latency 557.275 ms 2024-09-16T00:33:58.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.810+0000 7f19363ec640 1 -- 172.21.15.33:0/729355675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19301011e0 msgr2=0x7f1930105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:58.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.810+0000 7f19363ec640 1 --2- 172.21.15.33:0/729355675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19301011e0 0x7f1930105980 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1924009890 tx=0x7f192402a370 comp rx=0 tx=0).stop 2024-09-16T00:33:58.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.811+0000 7f19363ec640 1 -- 172.21.15.33:0/729355675 shutdown_connections 2024-09-16T00:33:58.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.811+0000 7f19363ec640 1 --2- 172.21.15.33:0/729355675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1930105ec0 0x7f1930106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.811+0000 7f19363ec640 1 --2- 172.21.15.33:0/729355675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19301011e0 0x7f1930105980 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.811+0000 7f19363ec640 1 --2- 172.21.15.33:0/729355675 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19301008c0 0x7f1930100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.811+0000 7f19363ec640 1 -- 172.21.15.33:0/729355675 >> 172.21.15.33:0/729355675 conn(0x7f19300fc7d0 msgr2=0x7f19300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:58.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.812+0000 7f19363ec640 1 -- 172.21.15.33:0/729355675 shutdown_connections 2024-09-16T00:33:58.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.812+0000 7f19363ec640 1 -- 172.21.15.33:0/729355675 wait complete. 2024-09-16T00:33:58.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.813+0000 7f19363ec640 1 Processor -- start 2024-09-16T00:33:58.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.813+0000 7f19363ec640 1 -- start start 2024-09-16T00:33:58.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.814+0000 7f19363ec640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f19301008c0 0x7f19301a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.814+0000 7f19363ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19301011e0 0x7f19301a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.814+0000 7f19363ec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1930105ec0 0x7f193019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.814+0000 7f19363ec640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1930118610 con 0x7f19301008c0 2024-09-16T00:33:58.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.814+0000 7f19363ec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1930118490 con 0x7f19301011e0 2024-09-16T00:33:58.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.814+0000 7f19363ec640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1930118790 con 0x7f1930105ec0 2024-09-16T00:33:58.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.814+0000 7f1934989640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1930105ec0 0x7f193019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.814+0000 7f1934989640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1930105ec0 0x7f193019e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:58914/0 (socket says 172.21.15.33:58914) 2024-09-16T00:33:58.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.814+0000 7f1934989640 1 -- 172.21.15.33:0/1055471890 learned_addr learned my addr 172.21.15.33:0/1055471890 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:58.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.814+0000 7f192ffff640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f19301008c0 0x7f19301a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.815+0000 7f192f7fe640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19301011e0 0x7f19301a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.815+0000 7f192ffff640 1 -- 172.21.15.33:0/1055471890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1930105ec0 msgr2=0x7f193019e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:58.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.815+0000 7f192ffff640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1930105ec0 0x7f193019e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.815+0000 7f192ffff640 1 -- 172.21.15.33:0/1055471890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19301011e0 msgr2=0x7f19301a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:58.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.815+0000 7f192ffff640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19301011e0 0x7f19301a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.815+0000 7f192ffff640 1 -- 172.21.15.33:0/1055471890 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193019f070 con 0x7f19301008c0 2024-09-16T00:33:58.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.815+0000 7f192ffff640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f19301008c0 0x7f19301a4780 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f192800eb00 tx=0x7f192800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:58.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.816+0000 7f192d7fa640 1 -- 172.21.15.33:0/1055471890 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1928010430 con 0x7f19301008c0 2024-09-16T00:33:58.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.816+0000 7f19363ec640 1 -- 172.21.15.33:0/1055471890 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f193019f360 con 0x7f19301008c0 2024-09-16T00:33:58.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.816+0000 7f19363ec640 1 -- 172.21.15.33:0/1055471890 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19301ab650 con 0x7f19301008c0 2024-09-16T00:33:58.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.816+0000 7f192d7fa640 1 -- 172.21.15.33:0/1055471890 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1928010ab0 con 0x7f19301008c0 2024-09-16T00:33:58.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.816+0000 7f192d7fa640 1 -- 172.21.15.33:0/1055471890 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1928017bb0 con 0x7f19301008c0 2024-09-16T00:33:58.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.819+0000 7f19387c5740 1 Processor -- start 2024-09-16T00:33:58.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.820+0000 7f19387c5740 1 -- start start 2024-09-16T00:33:58.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.820+0000 7f19387c5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4bb40 0x5622c5c4a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.820+0000 7f19387c5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c4a5d0 0x5622c5c4aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.820+0000 7f19387c5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c55370 0x5622c5c57760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.820+0000 7f19387c5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5622c5bb02f0 con 0x5622c5c4a5d0 2024-09-16T00:33:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.820+0000 7f19387c5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5622c5b57cf0 con 0x5622c5c4bb40 2024-09-16T00:33:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.820+0000 7f19387c5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5622c5bac2b0 con 0x5622c5c55370 2024-09-16T00:33:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.820+0000 7f1934989640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4bb40 0x5622c5c4a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.820+0000 7f1934989640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4bb40 0x5622c5c4a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:52582/0 (socket says 172.21.15.33:52582) 2024-09-16T00:33:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.820+0000 7f1934989640 1 -- 172.21.15.33:0/207342872 learned_addr learned my addr 172.21.15.33:0/207342872 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.821+0000 7f192f7fe640 1 --2- 172.21.15.33:0/207342872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c55370 0x5622c5c57760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.821+0000 7f1934989640 1 -- 172.21.15.33:0/207342872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c55370 msgr2=0x5622c5c57760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:58.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.821+0000 7f1934989640 1 --2- 172.21.15.33:0/207342872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c55370 0x5622c5c57760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.821+0000 7f1934989640 1 -- 172.21.15.33:0/207342872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c4a5d0 msgr2=0x5622c5c4aa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:33:58.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.821+0000 7f1934989640 1 --2- 172.21.15.33:0/207342872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c4a5d0 0x5622c5c4aa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.821+0000 7f1934989640 1 -- 172.21.15.33:0/207342872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622c5b3c3a0 con 0x5622c5c4bb40 2024-09-16T00:33:58.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.821+0000 7f192d7fa640 1 -- 172.21.15.33:0/1055471890 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f19280105d0 con 0x7f19301008c0 2024-09-16T00:33:58.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.821+0000 7f1934989640 1 --2- 172.21.15.33:0/207342872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4bb40 0x5622c5c4a090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1920009cc0 tx=0x7f19200107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33743640575b6839 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:58.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.822+0000 7f19167fc640 1 -- 172.21.15.33:0/207342872 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192001c480 con 0x5622c5c4bb40 2024-09-16T00:33:58.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.822+0000 7f19167fc640 1 -- 172.21.15.33:0/207342872 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f192001cb00 con 0x5622c5c4bb40 2024-09-16T00:33:58.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.822+0000 7f19167fc640 1 -- 172.21.15.33:0/207342872 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920011ca0 con 0x5622c5c4bb40 2024-09-16T00:33:58.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.822+0000 7f192d7fa640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f19180a32c0 0x7f19180a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.823+0000 7f192d7fa640 1 -- 172.21.15.33:0/1055471890 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f19280d06a0 con 0x7f19301008c0 2024-09-16T00:33:58.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.823+0000 7f192ffff640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f19180a32c0 0x7f19180a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.823+0000 7f19387c5740 1 -- 172.21.15.33:0/207342872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4bb40 msgr2=0x5622c5c4a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:58.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.823+0000 7f19387c5740 1 --2- 172.21.15.33:0/207342872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4bb40 0x5622c5c4a090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1920009cc0 tx=0x7f19200107a0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.823+0000 7f19387c5740 1 -- 172.21.15.33:0/207342872 shutdown_connections 2024-09-16T00:33:58.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.824+0000 7f19387c5740 1 --2- 172.21.15.33:0/207342872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c55370 0x5622c5c57760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.824+0000 7f19387c5740 1 --2- 172.21.15.33:0/207342872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c4a5d0 0x5622c5c4aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.824+0000 7f19387c5740 1 --2- 172.21.15.33:0/207342872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4bb40 0x5622c5c4a090 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.824+0000 7f19387c5740 1 -- 172.21.15.33:0/207342872 >> 172.21.15.33:0/207342872 conn(0x5622c5c13310 msgr2=0x5622c5c46c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:58.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.825+0000 7f192ffff640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f19180a32c0 0x7f19180a5780 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f19280176e0 tx=0x7f1928010f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:58.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.825+0000 7f19387c5740 1 -- 172.21.15.33:0/207342872 shutdown_connections 2024-09-16T00:33:58.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.826+0000 7f19387c5740 1 -- 172.21.15.33:0/207342872 wait complete. 2024-09-16T00:33:58.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.826+0000 7f19387c5740 1 -- start start 2024-09-16T00:33:58.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.827+0000 7f19387c5740 1 Processor -- start 2024-09-16T00:33:58.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.827+0000 7f19387c5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4a5d0 0x5622c5c54db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.827+0000 7f19387c5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4bb40 0x5622c5c52060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.827+0000 7f19387c5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c55370 0x5622c5c527f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.827+0000 7f19387c5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5622c5c58070 con 0x5622c5c55370 2024-09-16T00:33:58.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.827+0000 7f19387c5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5622c5b6b060 con 0x5622c5c4a5d0 2024-09-16T00:33:58.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.827+0000 7f19387c5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5622c5c581f0 con 0x5622c5c4bb40 2024-09-16T00:33:58.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.827+0000 7f19387c5740 10 client.0 _ll_register_callbacks cb 0x5622c58be8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:58.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.827+0000 7f1934989640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c55370 0x5622c5c527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.827+0000 7f1934989640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c55370 0x5622c5c527f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45958/0 (socket says 172.21.15.33:45958) 2024-09-16T00:33:58.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.827+0000 7f1934989640 1 -- 172.21.15.33:0/1404032743 learned_addr learned my addr 172.21.15.33:0/1404032743 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:58.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.827+0000 7f1934989640 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4a5d0 0x5622c5c54db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.828+0000 7f192f7fe640 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4bb40 0x5622c5c52060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.828+0000 7f1934989640 1 -- 172.21.15.33:0/1404032743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4bb40 msgr2=0x5622c5c52060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:58.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.828+0000 7f1934989640 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4bb40 0x5622c5c52060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.828+0000 7f1934989640 1 -- 172.21.15.33:0/1404032743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c55370 msgr2=0x5622c5c527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:58.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.828+0000 7f1934989640 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c55370 0x5622c5c527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.828+0000 7f1934989640 1 -- 172.21.15.33:0/1404032743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622c59213c0 con 0x5622c5c4a5d0 2024-09-16T00:33:58.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.829+0000 7f1934989640 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4a5d0 0x5622c5c54db0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1920005ab0 tx=0x7f1920005a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eeb6da1c928ee61 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:58.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.829+0000 7f190f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:33:58.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.829+0000 7f190f7fe640 1 -- 172.21.15.33:0/1404032743 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192001c480 con 0x5622c5c4a5d0 2024-09-16T00:33:58.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.829+0000 7f190f7fe640 1 -- 172.21.15.33:0/1404032743 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f192001c620 con 0x5622c5c4a5d0 2024-09-16T00:33:58.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.830+0000 7f19387c5740 1 -- 172.21.15.33:0/1404032743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f192001cb00 con 0x5622c5c4a5d0 2024-09-16T00:33:58.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.832+0000 7f190f7fe640 1 -- 172.21.15.33:0/1404032743 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f192001c820 con 0x5622c5c4a5d0 2024-09-16T00:33:58.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.833+0000 7f19387c5740 10 client.27869 fetch_fsmap learned FSMap version 33 2024-09-16T00:33:58.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.833+0000 7f19387c5740 1 -- 172.21.15.33:0/1404032743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5622c5c74bc0 con 0x5622c5c4a5d0 2024-09-16T00:33:58.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.835+0000 7f190f7fe640 1 -- 172.21.15.33:0/1404032743 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1920005510 con 0x5622c5c4a5d0 2024-09-16T00:33:58.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.835+0000 7f190f7fe640 1 -- 172.21.15.33:0/1404032743 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f192002a020 con 0x5622c5c4a5d0 2024-09-16T00:33:58.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.835+0000 7f19387c5740 10 client.27869 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:33:58.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.835+0000 7f19387c5740 10 client.27869 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:33:58.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.835+0000 7f19387c5740 20 client.27869 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:58.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.835+0000 7f19387c5740 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5622c5c75f10 0x5622c5c78300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.835+0000 7f19387c5740 4 client.27869 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:33:58.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.835+0000 7f19387c5740 1 -- 172.21.15.33:0/1404032743 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f192001cb00 con 0x5622c5c75f10 2024-09-16T00:33:58.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.837+0000 7f192f7fe640 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5622c5c75f10 0x5622c5c78300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.840+0000 7f192f7fe640 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5622c5c75f10 0x5622c5c78300 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b3ab710d2a6b1a3e server_cookie=f432e73ca32e67 in_seq=0 out_seq=0 2024-09-16T00:33:58.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.840+0000 7f190f7fe640 10 client.27869 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:33:58.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.841+0000 7f190f7fe640 1 -- 172.21.15.33:0/1404032743 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f192001cb00 con 0x5622c5c75f10 2024-09-16T00:33:58.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.841+0000 7f190f7fe640 10 client.27869 handle_command_reply: tid=0 2024-09-16T00:33:58.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.841+0000 7f19387c5740 1 client.27869 shutdown 2024-09-16T00:33:58.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.842+0000 7f19387c5740 1 -- 172.21.15.33:0/1404032743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4a5d0 msgr2=0x5622c5c54db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:58.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.842+0000 7f19387c5740 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4a5d0 0x5622c5c54db0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1920005ab0 tx=0x7f1920005a00 comp rx=0 tx=0).stop 2024-09-16T00:33:58.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.843+0000 7f19387c5740 1 -- 172.21.15.33:0/1404032743 shutdown_connections 2024-09-16T00:33:58.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.843+0000 7f19387c5740 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c55370 0x5622c5c527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.843+0000 7f19387c5740 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4bb40 0x5622c5c52060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.843+0000 7f19387c5740 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5622c5c75f10 0x5622c5c78300 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.843+0000 7f190f7fe640 0 client.27869 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:33:58.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.843+0000 7f19387c5740 1 --2- 172.21.15.33:0/1404032743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4a5d0 0x5622c5c54db0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.843+0000 7f19387c5740 1 -- 172.21.15.33:0/1404032743 >> 172.21.15.33:0/1404032743 conn(0x5622c5c13310 msgr2=0x5622c5c4c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:58.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.843+0000 7f19387c5740 1 -- 172.21.15.33:0/1404032743 shutdown_connections 2024-09-16T00:33:58.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.843+0000 7f19387c5740 1 -- 172.21.15.33:0/1404032743 wait complete. 2024-09-16T00:33:58.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.843+0000 7f19387c5740 20 client.27869 trim_cache size 0 max 16384 2024-09-16T00:33:58.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.868+0000 7f19387c5740 1 Processor -- start 2024-09-16T00:33:58.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.869+0000 7f19387c5740 1 -- start start 2024-09-16T00:33:58.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.869+0000 7f19387c5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4a5d0 0x5622c5c518f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.869+0000 7f19387c5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c4bb40 0x5622c5c7d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.870+0000 7f1934989640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4a5d0 0x5622c5c518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.870+0000 7f1934989640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4a5d0 0x5622c5c518f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:58944/0 (socket says 172.21.15.33:58944) 2024-09-16T00:33:58.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.870+0000 7f192f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c4bb40 0x5622c5c7d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.870+0000 7f1934989640 1 -- 172.21.15.33:0/3416298728 learned_addr learned my addr 172.21.15.33:0/3416298728 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:58.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.870+0000 7f19387c5740 1 --2- 172.21.15.33:0/3416298728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c55370 0x5622c5c7ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.870+0000 7f19387c5740 1 -- 172.21.15.33:0/3416298728 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f191c012980 con 0x5622c5c4bb40 2024-09-16T00:33:58.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.870+0000 7f19387c5740 1 -- 172.21.15.33:0/3416298728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f191c010250 con 0x5622c5c55370 2024-09-16T00:33:58.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.870+0000 7f19387c5740 1 -- 172.21.15.33:0/3416298728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f191c00e0a0 con 0x5622c5c4a5d0 2024-09-16T00:33:58.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.871+0000 7f192f7fe640 1 -- 172.21.15.33:0/3416298728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4a5d0 msgr2=0x5622c5c518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:58.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.871+0000 7f192f7fe640 1 --2- 172.21.15.33:0/3416298728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4a5d0 0x5622c5c518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.871+0000 7f192f7fe640 1 -- 172.21.15.33:0/3416298728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c55370 msgr2=0x5622c5c7ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:58.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.871+0000 7f192f7fe640 1 --2- 172.21.15.33:0/3416298728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c55370 0x5622c5c7ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.871+0000 7f192f7fe640 1 -- 172.21.15.33:0/3416298728 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622c5c51010 con 0x5622c5c4bb40 2024-09-16T00:33:58.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.871+0000 7f1934989640 1 --2- 172.21.15.33:0/3416298728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4a5d0 0x5622c5c518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:33:58.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.871+0000 7f192f7fe640 1 --2- 172.21.15.33:0/3416298728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c4bb40 0x5622c5c7d990 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f192402a960 tx=0x7f19240306b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45f5ef239277991f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:58.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.871+0000 7f1915ffb640 1 -- 172.21.15.33:0/3416298728 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1924037070 con 0x5622c5c4bb40 2024-09-16T00:33:58.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.872+0000 7f1915ffb640 1 -- 172.21.15.33:0/3416298728 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1924030810 con 0x5622c5c4bb40 2024-09-16T00:33:58.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.872+0000 7f1915ffb640 1 -- 172.21.15.33:0/3416298728 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19240035f0 con 0x5622c5c4bb40 2024-09-16T00:33:58.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.873+0000 7f19387c5740 1 -- 172.21.15.33:0/3416298728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c4bb40 msgr2=0x5622c5c7d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:58.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.873+0000 7f19387c5740 1 --2- 172.21.15.33:0/3416298728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c4bb40 0x5622c5c7d990 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f192402a960 tx=0x7f19240306b0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.874+0000 7f19387c5740 1 -- 172.21.15.33:0/3416298728 shutdown_connections 2024-09-16T00:33:58.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.874+0000 7f19387c5740 1 --2- 172.21.15.33:0/3416298728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c55370 0x5622c5c7ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.874+0000 7f19387c5740 1 --2- 172.21.15.33:0/3416298728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c4bb40 0x5622c5c7d990 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.874+0000 7f19387c5740 1 --2- 172.21.15.33:0/3416298728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4a5d0 0x5622c5c518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.874+0000 7f19387c5740 1 -- 172.21.15.33:0/3416298728 >> 172.21.15.33:0/3416298728 conn(0x5622c5c13310 msgr2=0x5622c5c4c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:58.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.874+0000 7f19387c5740 1 -- 172.21.15.33:0/3416298728 shutdown_connections 2024-09-16T00:33:58.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.874+0000 7f19387c5740 1 -- 172.21.15.33:0/3416298728 wait complete. 2024-09-16T00:33:58.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.875+0000 7f19387c5740 1 -- start start 2024-09-16T00:33:58.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.875+0000 7f19387c5740 1 Processor -- start 2024-09-16T00:33:58.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.876+0000 7f19387c5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4a5d0 0x5622c5c59580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.876+0000 7f19387c5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4bb40 0x5622c5c59ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.876+0000 7f19387c5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c55370 0x5622c5cdf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.876+0000 7f19387c5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f191c00e710 con 0x5622c5c55370 2024-09-16T00:33:58.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.876+0000 7f19387c5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f191c00fb80 con 0x5622c5c4a5d0 2024-09-16T00:33:58.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.876+0000 7f19387c5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f191c00ede0 con 0x5622c5c4bb40 2024-09-16T00:33:58.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.876+0000 7f19387c5740 10 client.0 _ll_register_callbacks cb 0x7f1920005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:33:58.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.877+0000 7f1934989640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c55370 0x5622c5cdf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.877+0000 7f1934989640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c55370 0x5622c5cdf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45974/0 (socket says 172.21.15.33:45974) 2024-09-16T00:33:58.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.877+0000 7f1934989640 1 -- 172.21.15.33:0/1708720370 learned_addr learned my addr 172.21.15.33:0/1708720370 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:33:58.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.877+0000 7f1934989640 1 --2- 172.21.15.33:0/1708720370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4a5d0 0x5622c5c59580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.878+0000 7f1934989640 1 -- 172.21.15.33:0/1708720370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4bb40 msgr2=0x5622c5c59ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:33:58.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.878+0000 7f1934989640 1 --2- 172.21.15.33:0/1708720370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4bb40 0x5622c5c59ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.878+0000 7f1934989640 1 -- 172.21.15.33:0/1708720370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c55370 msgr2=0x5622c5cdf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:33:58.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.878+0000 7f1934989640 1 --2- 172.21.15.33:0/1708720370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c55370 0x5622c5cdf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.878+0000 7f1934989640 1 -- 172.21.15.33:0/1708720370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622c5c5a3a0 con 0x5622c5c4a5d0 2024-09-16T00:33:58.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.878+0000 7f1934989640 1 --2- 172.21.15.33:0/1708720370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4a5d0 0x5622c5c59580 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1920004e10 tx=0x7f19200107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59fb66ffa86228e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:33:58.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.878+0000 7f190f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:33:58.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.878+0000 7f190f7fe640 1 -- 172.21.15.33:0/1708720370 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920009940 con 0x5622c5c4a5d0 2024-09-16T00:33:58.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.879+0000 7f190f7fe640 1 -- 172.21.15.33:0/1708720370 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f192001e480 con 0x5622c5c4a5d0 2024-09-16T00:33:58.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.879+0000 7f19387c5740 1 -- 172.21.15.33:0/1708720370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1924030810 con 0x5622c5c4a5d0 2024-09-16T00:33:58.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.879+0000 7f190f7fe640 1 -- 172.21.15.33:0/1708720370 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1920048050 con 0x5622c5c4a5d0 2024-09-16T00:33:58.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.879+0000 7f19387c5740 10 client.27872 fetch_fsmap learned FSMap version 33 2024-09-16T00:33:58.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.879+0000 7f19387c5740 1 -- 172.21.15.33:0/1708720370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5622c5ce0f80 con 0x5622c5c4a5d0 2024-09-16T00:33:58.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.880+0000 7f190f7fe640 1 -- 172.21.15.33:0/1708720370 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f192003b030 con 0x5622c5c4a5d0 2024-09-16T00:33:58.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.880+0000 7f190f7fe640 1 -- 172.21.15.33:0/1708720370 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f192002a070 con 0x5622c5c4a5d0 2024-09-16T00:33:58.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.880+0000 7f19387c5740 10 client.27872 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:33:58.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.880+0000 7f19387c5740 10 client.27872 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:33:58.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.880+0000 7f19387c5740 20 client.27872 populate_metadata read hostname 'smithi033' 2024-09-16T00:33:58.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.880+0000 7f19387c5740 1 --2- 172.21.15.33:0/1708720370 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5622c5c75f10 0x5622c5ce3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:33:58.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.880+0000 7f19387c5740 4 client.27872 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:33:58.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.880+0000 7f19387c5740 1 -- 172.21.15.33:0/1708720370 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f1924030810 con 0x5622c5c75f10 2024-09-16T00:33:58.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.880+0000 7f192f7fe640 1 --2- 172.21.15.33:0/1708720370 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5622c5c75f10 0x5622c5ce3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:33:58.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.885+0000 7f192f7fe640 1 --2- 172.21.15.33:0/1708720370 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5622c5c75f10 0x5622c5ce3540 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=82d04036e110c4fa server_cookie=936eff346ab0058 in_seq=0 out_seq=0 2024-09-16T00:33:58.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.885+0000 7f190f7fe640 10 client.27872 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:33:58.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.887+0000 7f190f7fe640 1 -- 172.21.15.33:0/1708720370 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f1924030810 con 0x5622c5c75f10 2024-09-16T00:33:58.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.887+0000 7f190f7fe640 10 client.27872 handle_command_reply: tid=0 2024-09-16T00:33:58.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.887+0000 7f19387c5740 1 client.27872 shutdown 2024-09-16T00:33:58.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.891+0000 7f19387c5740 1 -- 172.21.15.33:0/1708720370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4a5d0 msgr2=0x5622c5c59580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:58.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.891+0000 7f19387c5740 1 --2- 172.21.15.33:0/1708720370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4a5d0 0x5622c5c59580 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1920004e10 tx=0x7f19200107a0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.891+0000 7f19387c5740 1 -- 172.21.15.33:0/1708720370 shutdown_connections 2024-09-16T00:33:58.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.891+0000 7f19387c5740 1 --2- 172.21.15.33:0/1708720370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5622c5c55370 0x5622c5cdf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.891+0000 7f19387c5740 1 --2- 172.21.15.33:0/1708720370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622c5c4bb40 0x5622c5c59ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.891+0000 7f19387c5740 1 --2- 172.21.15.33:0/1708720370 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5622c5c75f10 0x5622c5ce3540 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.891+0000 7f19387c5740 1 --2- 172.21.15.33:0/1708720370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5622c5c4a5d0 0x5622c5c59580 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.891+0000 7f19387c5740 1 -- 172.21.15.33:0/1708720370 >> 172.21.15.33:0/1708720370 conn(0x5622c5c13310 msgr2=0x5622c5c4c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:58.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.891+0000 7f190f7fe640 0 client.27872 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:33:58.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.891+0000 7f19387c5740 1 -- 172.21.15.33:0/1708720370 shutdown_connections 2024-09-16T00:33:58.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.891+0000 7f19387c5740 1 -- 172.21.15.33:0/1708720370 wait complete. 2024-09-16T00:33:58.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.891+0000 7f19387c5740 20 client.27872 trim_cache size 0 max 16384 2024-09-16T00:33:58.895 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:33:58.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.893+0000 7f19363ec640 1 -- 172.21.15.33:0/1055471890 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f19180a32c0 msgr2=0x7f19180a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:58.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.893+0000 7f19363ec640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f19180a32c0 0x7f19180a5780 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f19280176e0 tx=0x7f1928010f90 comp rx=0 tx=0).stop 2024-09-16T00:33:58.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.893+0000 7f19363ec640 1 -- 172.21.15.33:0/1055471890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f19301008c0 msgr2=0x7f19301a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:33:58.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.893+0000 7f19363ec640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f19301008c0 0x7f19301a4780 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f192800eb00 tx=0x7f192800c2d0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.893+0000 7f19363ec640 1 -- 172.21.15.33:0/1055471890 shutdown_connections 2024-09-16T00:33:58.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.893+0000 7f19363ec640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f19180a32c0 0x7f19180a5780 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.893+0000 7f19363ec640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1930105ec0 0x7f193019e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.893+0000 7f19363ec640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19301011e0 0x7f19301a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.893+0000 7f19363ec640 1 --2- 172.21.15.33:0/1055471890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f19301008c0 0x7f19301a4780 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:33:58.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.893+0000 7f19363ec640 1 -- 172.21.15.33:0/1055471890 >> 172.21.15.33:0/1055471890 conn(0x7f19300fc7d0 msgr2=0x7f1930103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:33:58.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.896+0000 7f19363ec640 1 -- 172.21.15.33:0/1055471890 shutdown_connections 2024-09-16T00:33:58.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:33:58.897+0000 7f19363ec640 1 -- 172.21.15.33:0/1055471890 wait complete. 2024-09-16T00:33:59.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 67426 19.57 MB/sec execute 65 sec latency 110.773 ms 2024-09-16T00:33:59.649 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:33:59 smithi033 ceph-mon[31243]: pgmap v855: 129 pgs: 129 active+clean; 573 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 11 MiB/s wr, 538 op/s 2024-09-16T00:33:59.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:33:59 smithi083 ceph-mon[35242]: pgmap v855: 129 pgs: 129 active+clean; 573 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 11 MiB/s wr, 538 op/s 2024-09-16T00:33:59.684 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:33:59.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:33:59 smithi060 ceph-mon[35185]: pgmap v855: 129 pgs: 129 active+clean; 573 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 11 MiB/s wr, 538 op/s 2024-09-16T00:33:59.983 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:00.075 INFO:tasks.workunit.client.0.smithi033.stdout: 10 67841 19.63 MB/sec execute 66 sec latency 59.143 ms 2024-09-16T00:34:00.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.522+0000 7f8e2eb49640 1 -- 172.21.15.33:0/1050583056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e28102ef0 msgr2=0x7f8e2810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:00.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.522+0000 7f8e2eb49640 1 --2- 172.21.15.33:0/1050583056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e28102ef0 0x7f8e2810da40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009ca0 tx=0x7f8e2002a950 comp rx=0 tx=0).stop 2024-09-16T00:34:00.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.522+0000 7f8e2eb49640 1 -- 172.21.15.33:0/1050583056 shutdown_connections 2024-09-16T00:34:00.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.522+0000 7f8e2eb49640 1 --2- 172.21.15.33:0/1050583056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e28102ef0 0x7f8e2810da40 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.522+0000 7f8e2eb49640 1 --2- 172.21.15.33:0/1050583056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e28102570 0x7f8e281029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.522+0000 7f8e2eb49640 1 --2- 172.21.15.33:0/1050583056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e28100e20 0x7f8e28101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.522+0000 7f8e2eb49640 1 -- 172.21.15.33:0/1050583056 >> 172.21.15.33:0/1050583056 conn(0x7f8e280fc7d0 msgr2=0x7f8e280febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:00.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.522+0000 7f8e2eb49640 1 -- 172.21.15.33:0/1050583056 shutdown_connections 2024-09-16T00:34:00.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.522+0000 7f8e2eb49640 1 -- 172.21.15.33:0/1050583056 wait complete. 2024-09-16T00:34:00.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e2eb49640 1 Processor -- start 2024-09-16T00:34:00.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e2eb49640 1 -- start start 2024-09-16T00:34:00.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e2eb49640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e28100e20 0x7f8e28198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e2eb49640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e28102570 0x7f8e281989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e2eb49640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e28102ef0 0x7f8e2819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e2eb49640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e28110fd0 con 0x7f8e28102570 2024-09-16T00:34:00.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e2eb49640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8e28110e50 con 0x7f8e28100e20 2024-09-16T00:34:00.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e2eb49640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8e28111150 con 0x7f8e28102ef0 2024-09-16T00:34:00.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e27fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e28102570 0x7f8e281989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e27fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e28102570 0x7f8e281989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54992/0 (socket says 172.21.15.33:54992) 2024-09-16T00:34:00.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e27fff640 1 -- 172.21.15.33:0/3695979867 learned_addr learned my addr 172.21.15.33:0/3695979867 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:00.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e2c8e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e28100e20 0x7f8e28198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e2d0e6640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e28102ef0 0x7f8e2819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e27fff640 1 -- 172.21.15.33:0/3695979867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e28102ef0 msgr2=0x7f8e2819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:00.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e27fff640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e28102ef0 0x7f8e2819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e27fff640 1 -- 172.21.15.33:0/3695979867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e28100e20 msgr2=0x7f8e28198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:00.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e27fff640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e28100e20 0x7f8e28198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.523+0000 7f8e27fff640 1 -- 172.21.15.33:0/3695979867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e2819d440 con 0x7f8e28102570 2024-09-16T00:34:00.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.524+0000 7f8e27fff640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e28102570 0x7f8e281989d0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400e9f0 tx=0x7f8e1400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:00.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.524+0000 7f8e25ffb640 1 -- 172.21.15.33:0/3695979867 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1400cd80 con 0x7f8e28102570 2024-09-16T00:34:00.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.524+0000 7f8e25ffb640 1 -- 172.21.15.33:0/3695979867 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e14010910 con 0x7f8e28102570 2024-09-16T00:34:00.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.524+0000 7f8e2eb49640 1 -- 172.21.15.33:0/3695979867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e2819d730 con 0x7f8e28102570 2024-09-16T00:34:00.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.524+0000 7f8e25ffb640 1 -- 172.21.15.33:0/3695979867 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e14018990 con 0x7f8e28102570 2024-09-16T00:34:00.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.524+0000 7f8e2eb49640 1 -- 172.21.15.33:0/3695979867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e281a5010 con 0x7f8e28102570 2024-09-16T00:34:00.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.527+0000 7f8e25ffb640 1 -- 172.21.15.33:0/3695979867 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8e14010ac0 con 0x7f8e28102570 2024-09-16T00:34:00.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.527+0000 7f8e30f22740 1 Processor -- start 2024-09-16T00:34:00.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.527+0000 7f8e30f22740 1 -- start start 2024-09-16T00:34:00.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e30f22740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 0x563a6ce68090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e30f22740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 0x563a6ce68a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e30f22740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce77780 0x563a6ce79b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e30f22740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563a6cdce2f0 con 0x563a6ce69b40 2024-09-16T00:34:00.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e30f22740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x563a6cd75cf0 con 0x563a6ce685d0 2024-09-16T00:34:00.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e30f22740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563a6cdca2b0 con 0x563a6ce77780 2024-09-16T00:34:00.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e2d0e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 0x563a6ce68090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e2c8e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 0x563a6ce68a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e2d0e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 0x563a6ce68090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55000/0 (socket says 172.21.15.33:55000) 2024-09-16T00:34:00.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e2d0e6640 1 -- 172.21.15.33:0/35566489 learned_addr learned my addr 172.21.15.33:0/35566489 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:00.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e2c8e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 0x563a6ce68a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:56922/0 (socket says 172.21.15.33:56922) 2024-09-16T00:34:00.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e27fff640 1 --2- 172.21.15.33:0/35566489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce77780 0x563a6ce79b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e25ffb640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e180a32c0 0x7f8e180a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e2c8e5640 1 -- 172.21.15.33:0/35566489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce77780 msgr2=0x563a6ce79b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:00.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e2c8e5640 1 --2- 172.21.15.33:0/35566489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce77780 0x563a6ce79b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e2c8e5640 1 -- 172.21.15.33:0/35566489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 msgr2=0x563a6ce68090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:00.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e2c8e5640 1 --2- 172.21.15.33:0/35566489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 0x563a6ce68090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e2c8e5640 1 -- 172.21.15.33:0/35566489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a6cd5a3a0 con 0x563a6ce685d0 2024-09-16T00:34:00.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e2d0e6640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e180a32c0 0x7f8e180a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e25ffb640 1 -- 172.21.15.33:0/3695979867 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8e140d0000 con 0x7f8e28102570 2024-09-16T00:34:00.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.528+0000 7f8e2c8e5640 1 --2- 172.21.15.33:0/35566489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 0x563a6ce68a10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c009c20 tx=0x7f8e1c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe3287c45618a32e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:00.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.529+0000 7f8e12ffd640 1 -- 172.21.15.33:0/35566489 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c004050 con 0x563a6ce685d0 2024-09-16T00:34:00.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.529+0000 7f8e12ffd640 1 -- 172.21.15.33:0/35566489 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e1c005500 con 0x563a6ce685d0 2024-09-16T00:34:00.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.529+0000 7f8e12ffd640 1 -- 172.21.15.33:0/35566489 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c004dd0 con 0x563a6ce685d0 2024-09-16T00:34:00.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.529+0000 7f8e2d0e6640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e180a32c0 0x7f8e180a5780 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009a60 tx=0x7f8e2003c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:00.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.530+0000 7f8e30f22740 1 -- 172.21.15.33:0/35566489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 msgr2=0x563a6ce68a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:00.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.530+0000 7f8e30f22740 1 --2- 172.21.15.33:0/35566489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 0x563a6ce68a10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c009c20 tx=0x7f8e1c011810 comp rx=0 tx=0).stop 2024-09-16T00:34:00.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.530+0000 7f8e30f22740 1 -- 172.21.15.33:0/35566489 shutdown_connections 2024-09-16T00:34:00.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.530+0000 7f8e30f22740 1 --2- 172.21.15.33:0/35566489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce77780 0x563a6ce79b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.530+0000 7f8e30f22740 1 --2- 172.21.15.33:0/35566489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 0x563a6ce68a10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.530+0000 7f8e30f22740 1 --2- 172.21.15.33:0/35566489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 0x563a6ce68090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.530+0000 7f8e30f22740 1 -- 172.21.15.33:0/35566489 >> 172.21.15.33:0/35566489 conn(0x563a6ce31310 msgr2=0x563a6ce64c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:00.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.530+0000 7f8e30f22740 1 -- 172.21.15.33:0/35566489 shutdown_connections 2024-09-16T00:34:00.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.530+0000 7f8e30f22740 1 -- 172.21.15.33:0/35566489 wait complete. 2024-09-16T00:34:00.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.530+0000 7f8e30f22740 1 -- start start 2024-09-16T00:34:00.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.530+0000 7f8e30f22740 1 Processor -- start 2024-09-16T00:34:00.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e30f22740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 0x563a6ce77160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e30f22740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce69b40 0x563a6ce74410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e30f22740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce77780 0x563a6ce749e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.657 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:34:00.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e30f22740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563a6ce7a480 con 0x563a6ce77780 2024-09-16T00:34:00.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e30f22740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x563a6cd89060 con 0x563a6ce685d0 2024-09-16T00:34:00.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e30f22740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563a6ce7a600 con 0x563a6ce69b40 2024-09-16T00:34:00.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e30f22740 10 client.0 _ll_register_callbacks cb 0x563a6cadc8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:00.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e27fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce69b40 0x563a6ce74410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e2c8e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 0x563a6ce77160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e27fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce69b40 0x563a6ce74410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:44732/0 (socket says 172.21.15.33:44732) 2024-09-16T00:34:00.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e27fff640 1 -- 172.21.15.33:0/1341287769 learned_addr learned my addr 172.21.15.33:0/1341287769 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:00.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e2c8e5640 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 0x563a6ce77160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:56928/0 (socket says 172.21.15.33:56928) 2024-09-16T00:34:00.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e2d0e6640 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce77780 0x563a6ce749e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e27fff640 1 -- 172.21.15.33:0/1341287769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 msgr2=0x563a6ce77160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:00.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e27fff640 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 0x563a6ce77160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e27fff640 1 -- 172.21.15.33:0/1341287769 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce77780 msgr2=0x563a6ce749e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:00.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.531+0000 7f8e27fff640 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce77780 0x563a6ce749e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.532+0000 7f8e27fff640 1 -- 172.21.15.33:0/1341287769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6cb3f3c0 con 0x563a6ce69b40 2024-09-16T00:34:00.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.532+0000 7f8e27fff640 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce69b40 0x563a6ce74410 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400e9c0 tx=0x7f8e1409c020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d720f6128d760601 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:00.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.532+0000 7f8e07fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:34:00.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.532+0000 7f8e07fff640 1 -- 172.21.15.33:0/1341287769 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e140a4e10 con 0x563a6ce69b40 2024-09-16T00:34:00.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.532+0000 7f8e07fff640 1 -- 172.21.15.33:0/1341287769 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8e140d20d0 con 0x563a6ce69b40 2024-09-16T00:34:00.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.532+0000 7f8e30f22740 1 -- 172.21.15.33:0/1341287769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e1c005500 con 0x563a6ce69b40 2024-09-16T00:34:00.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.532+0000 7f8e07fff640 1 -- 172.21.15.33:0/1341287769 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8e1409c250 con 0x563a6ce69b40 2024-09-16T00:34:00.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.532+0000 7f8e30f22740 10 client.27895 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:00.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.532+0000 7f8e30f22740 1 -- 172.21.15.33:0/1341287769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563a6ce85fb0 con 0x563a6ce69b40 2024-09-16T00:34:00.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.533+0000 7f8e07fff640 1 -- 172.21.15.33:0/1341287769 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e1c005500 con 0x563a6ce69b40 2024-09-16T00:34:00.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.533+0000 7f8e07fff640 1 -- 172.21.15.33:0/1341287769 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f8e140a4980 con 0x563a6ce69b40 2024-09-16T00:34:00.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.533+0000 7f8e30f22740 10 client.27895 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:00.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.533+0000 7f8e30f22740 10 client.27895 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:00.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.533+0000 7f8e30f22740 20 client.27895 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:00.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.533+0000 7f8e30f22740 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563a6ce876c0 0x563a6ce89b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.533+0000 7f8e30f22740 4 client.27895 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:34:00.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.533+0000 7f8e30f22740 1 -- 172.21.15.33:0/1341287769 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a6cddb860 con 0x563a6ce876c0 2024-09-16T00:34:00.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.533+0000 7f8e2c8e5640 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563a6ce876c0 0x563a6ce89b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.534+0000 7f8e2c8e5640 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563a6ce876c0 0x563a6ce89b00 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d80c4596bb4fb79 server_cookie=1acc0578a5c6dd1e in_seq=0 out_seq=0 2024-09-16T00:34:00.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.534+0000 7f8e07fff640 10 client.27895 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:00.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.535+0000 7f8e07fff640 1 -- 172.21.15.33:0/1341287769 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563a6cddb860 con 0x563a6ce876c0 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.535+0000 7f8e07fff640 10 client.27895 handle_command_reply: tid=0 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.535+0000 7f8e30f22740 1 client.27895 shutdown 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.535+0000 7f8e30f22740 1 -- 172.21.15.33:0/1341287769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce69b40 msgr2=0x563a6ce74410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.535+0000 7f8e30f22740 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce69b40 0x563a6ce74410 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400e9c0 tx=0x7f8e1409c020 comp rx=0 tx=0).stop 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.535+0000 7f8e30f22740 1 -- 172.21.15.33:0/1341287769 shutdown_connections 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.536+0000 7f8e30f22740 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563a6ce876c0 0x563a6ce89b00 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.536+0000 7f8e30f22740 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce77780 0x563a6ce749e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.536+0000 7f8e30f22740 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce69b40 0x563a6ce74410 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.536+0000 7f8e30f22740 1 --2- 172.21.15.33:0/1341287769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce685d0 0x563a6ce77160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.536+0000 7f8e30f22740 1 -- 172.21.15.33:0/1341287769 >> 172.21.15.33:0/1341287769 conn(0x563a6ce31310 msgr2=0x563a6ce6a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.536+0000 7f8e07fff640 0 client.27895 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.536+0000 7f8e30f22740 1 -- 172.21.15.33:0/1341287769 shutdown_connections 2024-09-16T00:34:00.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.536+0000 7f8e30f22740 1 -- 172.21.15.33:0/1341287769 wait complete. 2024-09-16T00:34:00.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.536+0000 7f8e30f22740 20 client.27895 trim_cache size 0 max 16384 2024-09-16T00:34:00.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e30f22740 1 Processor -- start 2024-09-16T00:34:00.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e30f22740 1 -- start start 2024-09-16T00:34:00.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e30f22740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce685d0 0x563a6ce69300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e30f22740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 0x563a6ce83720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e30f22740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce77780 0x563a6ce6f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e30f22740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e281af700 con 0x563a6ce69b40 2024-09-16T00:34:00.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e30f22740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8e281acfd0 con 0x563a6ce77780 2024-09-16T00:34:00.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e30f22740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8e281aae20 con 0x563a6ce685d0 2024-09-16T00:34:00.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e27fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 0x563a6ce83720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e27fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 0x563a6ce83720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55020/0 (socket says 172.21.15.33:55020) 2024-09-16T00:34:00.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e27fff640 1 -- 172.21.15.33:0/2733806363 learned_addr learned my addr 172.21.15.33:0/2733806363 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:00.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e2c8e5640 1 --2- 172.21.15.33:0/2733806363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce685d0 0x563a6ce69300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e2d0e6640 1 --2- 172.21.15.33:0/2733806363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce77780 0x563a6ce6f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e27fff640 1 -- 172.21.15.33:0/2733806363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce685d0 msgr2=0x563a6ce69300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:00.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e27fff640 1 --2- 172.21.15.33:0/2733806363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce685d0 0x563a6ce69300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e27fff640 1 -- 172.21.15.33:0/2733806363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce77780 msgr2=0x563a6ce6f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:00.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e27fff640 1 --2- 172.21.15.33:0/2733806363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce77780 0x563a6ce6f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.551+0000 7f8e27fff640 1 -- 172.21.15.33:0/2733806363 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a6ce6b120 con 0x563a6ce69b40 2024-09-16T00:34:00.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.552+0000 7f8e27fff640 1 --2- 172.21.15.33:0/2733806363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 0x563a6ce83720 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f8e1409bf50 tx=0x7f8e14000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f8006c3fea1dba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:00.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.552+0000 7f8e127fc640 1 -- 172.21.15.33:0/2733806363 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e140a4e10 con 0x563a6ce69b40 2024-09-16T00:34:00.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.552+0000 7f8e127fc640 1 -- 172.21.15.33:0/2733806363 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e140a0550 con 0x563a6ce69b40 2024-09-16T00:34:00.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.552+0000 7f8e127fc640 1 -- 172.21.15.33:0/2733806363 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e140d33a0 con 0x563a6ce69b40 2024-09-16T00:34:00.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.553+0000 7f8e30f22740 1 -- 172.21.15.33:0/2733806363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 msgr2=0x563a6ce83720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:00.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.553+0000 7f8e30f22740 1 --2- 172.21.15.33:0/2733806363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 0x563a6ce83720 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f8e1409bf50 tx=0x7f8e14000bc0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.553+0000 7f8e30f22740 1 -- 172.21.15.33:0/2733806363 shutdown_connections 2024-09-16T00:34:00.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.553+0000 7f8e30f22740 1 --2- 172.21.15.33:0/2733806363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce77780 0x563a6ce6f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.553+0000 7f8e30f22740 1 --2- 172.21.15.33:0/2733806363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce69b40 0x563a6ce83720 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.553+0000 7f8e30f22740 1 --2- 172.21.15.33:0/2733806363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce685d0 0x563a6ce69300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.553+0000 7f8e30f22740 1 -- 172.21.15.33:0/2733806363 >> 172.21.15.33:0/2733806363 conn(0x563a6ce31310 msgr2=0x563a6ce5fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:00.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.553+0000 7f8e30f22740 1 -- 172.21.15.33:0/2733806363 shutdown_connections 2024-09-16T00:34:00.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.553+0000 7f8e30f22740 1 -- 172.21.15.33:0/2733806363 wait complete. 2024-09-16T00:34:00.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.553+0000 7f8e30f22740 1 -- start start 2024-09-16T00:34:00.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.553+0000 7f8e30f22740 1 Processor -- start 2024-09-16T00:34:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e30f22740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce685d0 0x563a6ce7f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e30f22740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce69b40 0x563a6ce7fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e30f22740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce77780 0x563a6ce70a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e30f22740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e281ab490 con 0x563a6ce685d0 2024-09-16T00:34:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e30f22740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8e281ac900 con 0x563a6ce77780 2024-09-16T00:34:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e30f22740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8e281abb60 con 0x563a6ce69b40 2024-09-16T00:34:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e30f22740 10 client.0 _ll_register_callbacks cb 0x7f8e1400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e2c8e5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce685d0 0x563a6ce7f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e2d0e6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce77780 0x563a6ce70a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e2c8e5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce685d0 0x563a6ce7f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55028/0 (socket says 172.21.15.33:55028) 2024-09-16T00:34:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e2c8e5640 1 -- 172.21.15.33:0/1002082438 learned_addr learned my addr 172.21.15.33:0/1002082438 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e27fff640 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce69b40 0x563a6ce7fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e2c8e5640 1 -- 172.21.15.33:0/1002082438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce69b40 msgr2=0x563a6ce7fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e2c8e5640 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce69b40 0x563a6ce7fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e2c8e5640 1 -- 172.21.15.33:0/1002082438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce77780 msgr2=0x563a6ce70a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e2c8e5640 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce77780 0x563a6ce70a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e2c8e5640 1 -- 172.21.15.33:0/1002082438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6ce70f80 con 0x563a6ce685d0 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e2d0e6640 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce77780 0x563a6ce70a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e2c8e5640 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce685d0 0x563a6ce7f890 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c0099b0 tx=0x7f8e1c009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae415d65d1d6d84 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e07fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.554+0000 7f8e07fff640 1 -- 172.21.15.33:0/1002082438 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c004450 con 0x563a6ce685d0 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e30f22740 1 -- 172.21.15.33:0/1002082438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e140a0550 con 0x563a6ce685d0 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e07fff640 1 -- 172.21.15.33:0/1002082438 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8e1c005040 con 0x563a6ce685d0 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e07fff640 1 -- 172.21.15.33:0/1002082438 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8e1c005240 con 0x563a6ce685d0 2024-09-16T00:34:00.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e30f22740 10 client.18591 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e30f22740 1 -- 172.21.15.33:0/1002082438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563a6cef1b40 con 0x563a6ce685d0 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e07fff640 1 -- 172.21.15.33:0/1002082438 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e140a0550 con 0x563a6ce685d0 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e07fff640 1 -- 172.21.15.33:0/1002082438 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f8e1c003660 con 0x563a6ce685d0 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e30f22740 10 client.18591 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e30f22740 10 client.18591 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e30f22740 20 client.18591 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e30f22740 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563a6cef2070 0x563a6cef4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e30f22740 4 client.18591 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.555+0000 7f8e30f22740 1 -- 172.21.15.33:0/1002082438 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a6cdd5d70 con 0x563a6cef2070 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.556+0000 7f8e27fff640 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563a6cef2070 0x563a6cef4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.556+0000 7f8e07fff640 10 client.18591 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.556+0000 7f8e27fff640 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563a6cef2070 0x563a6cef4460 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14f9584d2843446c server_cookie=98b83df36cf7eb15 in_seq=0 out_seq=0 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.556+0000 7f8e07fff640 1 -- 172.21.15.33:0/1002082438 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a6cdd5d70 con 0x563a6cef2070 2024-09-16T00:34:00.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.556+0000 7f8e07fff640 10 client.18591 handle_command_reply: tid=0 2024-09-16T00:34:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.556+0000 7f8e30f22740 1 client.18591 shutdown 2024-09-16T00:34:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.556+0000 7f8e30f22740 1 -- 172.21.15.33:0/1002082438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce685d0 msgr2=0x563a6ce7f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.556+0000 7f8e30f22740 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce685d0 0x563a6ce7f890 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c0099b0 tx=0x7f8e1c009e20 comp rx=0 tx=0).stop 2024-09-16T00:34:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.557+0000 7f8e30f22740 1 -- 172.21.15.33:0/1002082438 shutdown_connections 2024-09-16T00:34:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.557+0000 7f8e30f22740 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563a6cef2070 0x563a6cef4460 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.557+0000 7f8e30f22740 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563a6ce77780 0x563a6ce70a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.557+0000 7f8e30f22740 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563a6ce69b40 0x563a6ce7fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.557+0000 7f8e30f22740 1 --2- 172.21.15.33:0/1002082438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563a6ce685d0 0x563a6ce7f890 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.557+0000 7f8e30f22740 1 -- 172.21.15.33:0/1002082438 >> 172.21.15.33:0/1002082438 conn(0x563a6ce31310 msgr2=0x563a6ce6f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.557+0000 7f8e07fff640 0 client.18591 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.557+0000 7f8e30f22740 1 -- 172.21.15.33:0/1002082438 shutdown_connections 2024-09-16T00:34:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.557+0000 7f8e30f22740 1 -- 172.21.15.33:0/1002082438 wait complete. 2024-09-16T00:34:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.557+0000 7f8e30f22740 20 client.18591 trim_cache size 0 max 16384 2024-09-16T00:34:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.558+0000 7f8e2eb49640 1 -- 172.21.15.33:0/3695979867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e180a32c0 msgr2=0x7f8e180a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.558+0000 7f8e2eb49640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e180a32c0 0x7f8e180a5780 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009a60 tx=0x7f8e2003c040 comp rx=0 tx=0).stop 2024-09-16T00:34:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.558+0000 7f8e2eb49640 1 -- 172.21.15.33:0/3695979867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e28102570 msgr2=0x7f8e281989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.558+0000 7f8e2eb49640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e28102570 0x7f8e281989d0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400e9f0 tx=0x7f8e1400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.558+0000 7f8e2eb49640 1 -- 172.21.15.33:0/3695979867 shutdown_connections 2024-09-16T00:34:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.558+0000 7f8e2eb49640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e180a32c0 0x7f8e180a5780 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.558+0000 7f8e2eb49640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e28102ef0 0x7f8e2819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.558+0000 7f8e2eb49640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e28102570 0x7f8e281989d0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.558+0000 7f8e2eb49640 1 --2- 172.21.15.33:0/3695979867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e28100e20 0x7f8e28198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:00.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.558+0000 7f8e2eb49640 1 -- 172.21.15.33:0/3695979867 >> 172.21.15.33:0/3695979867 conn(0x7f8e280fc7d0 msgr2=0x7f8e2806a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:00.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.558+0000 7f8e2eb49640 1 -- 172.21.15.33:0/3695979867 shutdown_connections 2024-09-16T00:34:00.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:00.558+0000 7f8e2eb49640 1 -- 172.21.15.33:0/3695979867 wait complete. 2024-09-16T00:34:01.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 68278 19.63 MB/sec execute 67 sec latency 79.863 ms 2024-09-16T00:34:01.469 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:01 smithi033 ceph-mon[31243]: pgmap v856: 129 pgs: 129 active+clean; 555 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 13 MiB/s wr, 644 op/s 2024-09-16T00:34:01.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:01 smithi083 ceph-mon[35242]: pgmap v856: 129 pgs: 129 active+clean; 555 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 13 MiB/s wr, 644 op/s 2024-09-16T00:34:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:01 smithi060 ceph-mon[35185]: pgmap v856: 129 pgs: 129 active+clean; 555 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 13 MiB/s wr, 644 op/s 2024-09-16T00:34:02.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 68827 19.71 MB/sec execute 68 sec latency 92.695 ms 2024-09-16T00:34:02.505 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:34:02.505 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:34:02.824 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:03.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 69378 19.93 MB/sec execute 69 sec latency 212.623 ms 2024-09-16T00:34:03.662 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:03 smithi033 ceph-mon[31243]: pgmap v857: 129 pgs: 129 active+clean; 554 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 16 MiB/s wr, 737 op/s 2024-09-16T00:34:03.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:03 smithi083 ceph-mon[35242]: pgmap v857: 129 pgs: 129 active+clean; 554 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 16 MiB/s wr, 737 op/s 2024-09-16T00:34:03.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:03 smithi060 ceph-mon[35185]: pgmap v857: 129 pgs: 129 active+clean; 554 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 16 MiB/s wr, 737 op/s 2024-09-16T00:34:03.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.980+0000 7fb446872640 1 -- 172.21.15.33:0/298168735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4401002f0 msgr2=0x7fb4401006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.980+0000 7fb446872640 1 --2- 172.21.15.33:0/298168735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4401002f0 0x7fb4401006d0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fb430009930 tx=0x7fb43002a420 comp rx=0 tx=0).stop 2024-09-16T00:34:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.980+0000 7fb446872640 1 -- 172.21.15.33:0/298168735 shutdown_connections 2024-09-16T00:34:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.980+0000 7fb446872640 1 --2- 172.21.15.33:0/298168735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb440106f50 0x7fb44010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.980+0000 7fb446872640 1 --2- 172.21.15.33:0/298168735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb440100ca0 0x7fb440106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.980+0000 7fb446872640 1 --2- 172.21.15.33:0/298168735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4401002f0 0x7fb4401006d0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.980+0000 7fb446872640 1 -- 172.21.15.33:0/298168735 >> 172.21.15.33:0/298168735 conn(0x7fb440077210 msgr2=0x7fb4400fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.980+0000 7fb446872640 1 -- 172.21.15.33:0/298168735 shutdown_connections 2024-09-16T00:34:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.980+0000 7fb446872640 1 -- 172.21.15.33:0/298168735 wait complete. 2024-09-16T00:34:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.980+0000 7fb446872640 1 Processor -- start 2024-09-16T00:34:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.980+0000 7fb446872640 1 -- start start 2024-09-16T00:34:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.981+0000 7fb446872640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4401002f0 0x7fb4401984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.981+0000 7fb446872640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb440100ca0 0x7fb4401989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.981+0000 7fb446872640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb440106f50 0x7fb44019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.981+0000 7fb446872640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb440111ff0 con 0x7fb440106f50 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.981+0000 7fb446872640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb440111e70 con 0x7fb4401002f0 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.981+0000 7fb446872640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb440112170 con 0x7fb440100ca0 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.981+0000 7fb444e0f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb440106f50 0x7fb44019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.981+0000 7fb43f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb440100ca0 0x7fb4401989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.981+0000 7fb444e0f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb440106f50 0x7fb44019cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55048/0 (socket says 172.21.15.33:55048) 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.981+0000 7fb444e0f640 1 -- 172.21.15.33:0/2497731147 learned_addr learned my addr 172.21.15.33:0/2497731147 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.981+0000 7fb43ffff640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4401002f0 0x7fb4401984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.982+0000 7fb43f7fe640 1 -- 172.21.15.33:0/2497731147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4401002f0 msgr2=0x7fb4401984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.982+0000 7fb43f7fe640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4401002f0 0x7fb4401984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.982+0000 7fb43f7fe640 1 -- 172.21.15.33:0/2497731147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb440106f50 msgr2=0x7fb44019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.982+0000 7fb43f7fe640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb440106f50 0x7fb44019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:03.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.982+0000 7fb43f7fe640 1 -- 172.21.15.33:0/2497731147 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44019d460 con 0x7fb440100ca0 2024-09-16T00:34:03.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.982+0000 7fb444e0f640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb440106f50 0x7fb44019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:03.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.982+0000 7fb43f7fe640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb440100ca0 0x7fb4401989f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb43400e9c0 tx=0x7fb43400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:03.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.982+0000 7fb43d7fa640 1 -- 172.21.15.33:0/2497731147 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43400cd80 con 0x7fb440100ca0 2024-09-16T00:34:03.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.982+0000 7fb43d7fa640 1 -- 172.21.15.33:0/2497731147 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4340035f0 con 0x7fb440100ca0 2024-09-16T00:34:03.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.982+0000 7fb446872640 1 -- 172.21.15.33:0/2497731147 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb44019d750 con 0x7fb440100ca0 2024-09-16T00:34:03.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.982+0000 7fb43d7fa640 1 -- 172.21.15.33:0/2497731147 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb434010690 con 0x7fb440100ca0 2024-09-16T00:34:03.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.982+0000 7fb446872640 1 -- 172.21.15.33:0/2497731147 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4401a5030 con 0x7fb440100ca0 2024-09-16T00:34:03.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.985+0000 7fb43d7fa640 1 -- 172.21.15.33:0/2497731147 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb4340037a0 con 0x7fb440100ca0 2024-09-16T00:34:03.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.986+0000 7fb43d7fa640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4280a3240 0x7fb4280a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:03.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.986+0000 7fb448c4b740 1 Processor -- start 2024-09-16T00:34:03.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.986+0000 7fb43ffff640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4280a3240 0x7fb4280a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:03.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.986+0000 7fb43d7fa640 1 -- 172.21.15.33:0/2497731147 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb4340d00e0 con 0x7fb440100ca0 2024-09-16T00:34:03.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.986+0000 7fb448c4b740 1 -- start start 2024-09-16T00:34:03.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.986+0000 7fb448c4b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d46bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:03.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb448c4b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 0x56338d46c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:03.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb448c4b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d47b400 0x56338d47d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:03.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb448c4b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56338d3cd280 con 0x56338d46d4a0 2024-09-16T00:34:03.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb448c4b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56338d374b10 con 0x56338d46c4b0 2024-09-16T00:34:03.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb448c4b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56338d3c9240 con 0x56338d47b400 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb43ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d46bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb43f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 0x56338d46c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb43f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 0x56338d46c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:56968/0 (socket says 172.21.15.33:56968) 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb43f7fe640 1 -- 172.21.15.33:0/1006487178 learned_addr learned my addr 172.21.15.33:0/1006487178 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb43ffff640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4280a3240 0x7fb4280a5700 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb430006010 tx=0x7fb430000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb444e0f640 1 --2- 172.21.15.33:0/1006487178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d47b400 0x56338d47d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb43ffff640 1 -- 172.21.15.33:0/1006487178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d47b400 msgr2=0x56338d47d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb43ffff640 1 --2- 172.21.15.33:0/1006487178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d47b400 0x56338d47d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb43ffff640 1 -- 172.21.15.33:0/1006487178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 msgr2=0x56338d46c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb43ffff640 1 --2- 172.21.15.33:0/1006487178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 0x56338d46c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb43ffff640 1 -- 172.21.15.33:0/1006487178 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56338d3811f0 con 0x56338d46d4a0 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.987+0000 7fb444e0f640 1 --2- 172.21.15.33:0/1006487178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d47b400 0x56338d47d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:34:03.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.988+0000 7fb43ffff640 1 --2- 172.21.15.33:0/1006487178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d46bf70 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fb430009230 tx=0x7fb43003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c899d0f0b3168c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:03.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.988+0000 7fb43f7fe640 1 --2- 172.21.15.33:0/1006487178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 0x56338d46c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:03.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.988+0000 7fb4267fc640 1 -- 172.21.15.33:0/1006487178 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43004d070 con 0x56338d46d4a0 2024-09-16T00:34:03.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.988+0000 7fb4267fc640 1 -- 172.21.15.33:0/1006487178 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb430004ba0 con 0x56338d46d4a0 2024-09-16T00:34:03.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.988+0000 7fb4267fc640 1 -- 172.21.15.33:0/1006487178 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430035040 con 0x56338d46d4a0 2024-09-16T00:34:03.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.989+0000 7fb448c4b740 1 -- 172.21.15.33:0/1006487178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 msgr2=0x56338d46bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:03.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.989+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1006487178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d46bf70 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fb430009230 tx=0x7fb43003d000 comp rx=0 tx=0).stop 2024-09-16T00:34:03.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.989+0000 7fb448c4b740 1 -- 172.21.15.33:0/1006487178 shutdown_connections 2024-09-16T00:34:03.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.989+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1006487178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d47b400 0x56338d47d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:03.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.989+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1006487178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 0x56338d46c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:03.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.989+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1006487178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d46bf70 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:03.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.989+0000 7fb448c4b740 1 -- 172.21.15.33:0/1006487178 >> 172.21.15.33:0/1006487178 conn(0x56338d468440 msgr2=0x56338d468820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:03.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.989+0000 7fb448c4b740 1 -- 172.21.15.33:0/1006487178 shutdown_connections 2024-09-16T00:34:03.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.989+0000 7fb448c4b740 1 -- 172.21.15.33:0/1006487178 wait complete. 2024-09-16T00:34:03.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.990+0000 7fb448c4b740 1 -- start start 2024-09-16T00:34:03.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.990+0000 7fb448c4b740 1 Processor -- start 2024-09-16T00:34:03.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.990+0000 7fb448c4b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 0x56338d464ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb448c4b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d465010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb43ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 0x56338d464ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb43ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 0x56338d464ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:44786/0 (socket says 172.21.15.33:44786) 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb43f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d465010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb43f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d465010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55070/0 (socket says 172.21.15.33:55070) 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb448c4b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d47b400 0x56338d465550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb43ffff640 1 -- 172.21.15.33:0/1325881877 learned_addr learned my addr 172.21.15.33:0/1325881877 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb448c4b740 1 -- 172.21.15.33:0/1325881877 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56338d47e0a0 con 0x56338d46d4a0 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb448c4b740 1 -- 172.21.15.33:0/1325881877 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56338d387f10 con 0x56338d47b400 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb448c4b740 1 -- 172.21.15.33:0/1325881877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56338d47e220 con 0x56338d46c4b0 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb448c4b740 10 client.0 _ll_register_callbacks cb 0x56338d350a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb444e0f640 1 --2- 172.21.15.33:0/1325881877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d47b400 0x56338d465550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb43ffff640 1 -- 172.21.15.33:0/1325881877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d47b400 msgr2=0x56338d465550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:03.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb43ffff640 1 --2- 172.21.15.33:0/1325881877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d47b400 0x56338d465550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:03.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb43ffff640 1 -- 172.21.15.33:0/1325881877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 msgr2=0x56338d465010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:03.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb43ffff640 1 --2- 172.21.15.33:0/1325881877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d465010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:03.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.991+0000 7fb43ffff640 1 -- 172.21.15.33:0/1325881877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56338d13e6a0 con 0x56338d46c4b0 2024-09-16T00:34:04.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.993+0000 7fb43ffff640 1 --2- 172.21.15.33:0/1325881877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 0x56338d464ad0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb430003a00 tx=0x7fb430004f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb8f10cc0cf6a042 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:04.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.993+0000 7fb41f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.993+0000 7fb41f7fe640 1 -- 172.21.15.33:0/1325881877 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43004d070 con 0x56338d46c4b0 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.993+0000 7fb448c4b740 1 -- 172.21.15.33:0/1325881877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb430004ba0 con 0x56338d46c4b0 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.993+0000 7fb41f7fe640 1 -- 172.21.15.33:0/1325881877 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb430003d60 con 0x56338d46c4b0 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.993+0000 7fb41f7fe640 1 -- 172.21.15.33:0/1325881877 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb430035040 con 0x56338d46c4b0 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.993+0000 7fb448c4b740 10 client.27910 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.993+0000 7fb448c4b740 1 -- 172.21.15.33:0/1325881877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56338d47e0a0 con 0x56338d46c4b0 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.993+0000 7fb41f7fe640 1 -- 172.21.15.33:0/1325881877 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb430004ba0 con 0x56338d46c4b0 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.994+0000 7fb41f7fe640 1 -- 172.21.15.33:0/1325881877 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb430032cc0 con 0x56338d46c4b0 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.994+0000 7fb448c4b740 10 client.27910 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.994+0000 7fb448c4b740 10 client.27910 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.994+0000 7fb448c4b740 20 client.27910 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.994+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1325881877 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56338d46fec0 0x56338d472330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.994+0000 7fb448c4b740 4 client.27910 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:34:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.994+0000 7fb448c4b740 1 -- 172.21.15.33:0/1325881877 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56338d3da7f0 con 0x56338d46fec0 2024-09-16T00:34:04.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.994+0000 7fb43f7fe640 1 --2- 172.21.15.33:0/1325881877 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56338d46fec0 0x56338d472330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:04.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.995+0000 7fb43f7fe640 1 --2- 172.21.15.33:0/1325881877 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56338d46fec0 0x56338d472330 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2d519ec7de0ec9d server_cookie=5cc4ce79868d4160 in_seq=0 out_seq=0 2024-09-16T00:34:04.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.995+0000 7fb41f7fe640 10 client.27910 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:04.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.995+0000 7fb41f7fe640 1 -- 172.21.15.33:0/1325881877 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56338d3da7f0 con 0x56338d46fec0 2024-09-16T00:34:04.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.995+0000 7fb41f7fe640 10 client.27910 handle_command_reply: tid=0 2024-09-16T00:34:04.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.995+0000 7fb448c4b740 1 client.27910 shutdown 2024-09-16T00:34:04.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.995+0000 7fb448c4b740 1 -- 172.21.15.33:0/1325881877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 msgr2=0x56338d464ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:04.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.995+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1325881877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 0x56338d464ad0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb430003a00 tx=0x7fb430004f80 comp rx=0 tx=0).stop 2024-09-16T00:34:04.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.995+0000 7fb448c4b740 1 -- 172.21.15.33:0/1325881877 shutdown_connections 2024-09-16T00:34:04.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.995+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1325881877 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56338d46fec0 0x56338d472330 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.996+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1325881877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d47b400 0x56338d465550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.996+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1325881877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d465010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.996+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1325881877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 0x56338d464ad0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.996+0000 7fb448c4b740 1 -- 172.21.15.33:0/1325881877 >> 172.21.15.33:0/1325881877 conn(0x56338d468440 msgr2=0x56338d46e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:04.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.996+0000 7fb41f7fe640 0 client.27910 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:04.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.996+0000 7fb448c4b740 1 -- 172.21.15.33:0/1325881877 shutdown_connections 2024-09-16T00:34:04.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.996+0000 7fb448c4b740 1 -- 172.21.15.33:0/1325881877 wait complete. 2024-09-16T00:34:04.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:03.996+0000 7fb448c4b740 20 client.27910 trim_cache size 0 max 16384 2024-09-16T00:34:04.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb448c4b740 1 Processor -- start 2024-09-16T00:34:04.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb448c4b740 1 -- start start 2024-09-16T00:34:04.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb448c4b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 0x56338d47a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:04.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb448c4b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d47ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:04.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb448c4b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d47b400 0x56338d463270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:04.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb448c4b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb4401af830 con 0x56338d46d4a0 2024-09-16T00:34:04.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb448c4b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb4401ad100 con 0x56338d46c4b0 2024-09-16T00:34:04.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb448c4b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb4401aaf50 con 0x56338d47b400 2024-09-16T00:34:04.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb43f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d47ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:04.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb43f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d47ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55084/0 (socket says 172.21.15.33:55084) 2024-09-16T00:34:04.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb43f7fe640 1 -- 172.21.15.33:0/1232447653 learned_addr learned my addr 172.21.15.33:0/1232447653 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:04.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb43ffff640 1 --2- 172.21.15.33:0/1232447653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 0x56338d47a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:04.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.015+0000 7fb444e0f640 1 --2- 172.21.15.33:0/1232447653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d47b400 0x56338d463270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:04.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.016+0000 7fb43ffff640 1 -- 172.21.15.33:0/1232447653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d47b400 msgr2=0x56338d463270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:04.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.016+0000 7fb43ffff640 1 --2- 172.21.15.33:0/1232447653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d47b400 0x56338d463270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.016+0000 7fb43ffff640 1 -- 172.21.15.33:0/1232447653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 msgr2=0x56338d47ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:04.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.016+0000 7fb43ffff640 1 --2- 172.21.15.33:0/1232447653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d47ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.016+0000 7fb43ffff640 1 -- 172.21.15.33:0/1232447653 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56338d463020 con 0x56338d46c4b0 2024-09-16T00:34:04.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.016+0000 7fb43f7fe640 1 --2- 172.21.15.33:0/1232447653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d47ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:04.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.016+0000 7fb43ffff640 1 --2- 172.21.15.33:0/1232447653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 0x56338d47a6e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb43002a940 tx=0x7fb430009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98f4f9819281565 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:04.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.016+0000 7fb425ffb640 1 -- 172.21.15.33:0/1232447653 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43004d070 con 0x56338d46c4b0 2024-09-16T00:34:04.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.016+0000 7fb425ffb640 1 -- 172.21.15.33:0/1232447653 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb430005690 con 0x56338d46c4b0 2024-09-16T00:34:04.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.016+0000 7fb425ffb640 1 -- 172.21.15.33:0/1232447653 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430040040 con 0x56338d46c4b0 2024-09-16T00:34:04.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.017+0000 7fb448c4b740 1 -- 172.21.15.33:0/1232447653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 msgr2=0x56338d47a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:04.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.017+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1232447653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 0x56338d47a6e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb43002a940 tx=0x7fb430009780 comp rx=0 tx=0).stop 2024-09-16T00:34:04.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.017+0000 7fb448c4b740 1 -- 172.21.15.33:0/1232447653 shutdown_connections 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.017+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1232447653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d47b400 0x56338d463270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.017+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1232447653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d47ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.017+0000 7fb448c4b740 1 --2- 172.21.15.33:0/1232447653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d46c4b0 0x56338d47a6e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.017+0000 7fb448c4b740 1 -- 172.21.15.33:0/1232447653 >> 172.21.15.33:0/1232447653 conn(0x56338d468440 msgr2=0x56338d477260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.017+0000 7fb448c4b740 1 -- 172.21.15.33:0/1232447653 shutdown_connections 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.017+0000 7fb448c4b740 1 -- 172.21.15.33:0/1232447653 wait complete. 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.017+0000 7fb448c4b740 1 -- start start 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb448c4b740 1 Processor -- start 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb448c4b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 0x56338d483ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb448c4b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d48e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb448c4b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d47b400 0x56338d48e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb448c4b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb4401ab5c0 con 0x56338d46d4a0 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb448c4b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb4401aca30 con 0x56338d47b400 2024-09-16T00:34:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb448c4b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb4401abc90 con 0x56338d46c4b0 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb448c4b740 10 client.0 _ll_register_callbacks cb 0x7fb434097e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb43f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d48e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb43ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 0x56338d483ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb43ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 0x56338d483ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:44796/0 (socket says 172.21.15.33:44796) 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb43f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d48e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55098/0 (socket says 172.21.15.33:55098) 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb43ffff640 1 -- 172.21.15.33:0/3054509589 learned_addr learned my addr 172.21.15.33:0/3054509589 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb444e0f640 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d47b400 0x56338d48e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb43f7fe640 1 -- 172.21.15.33:0/3054509589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 msgr2=0x56338d483ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb43f7fe640 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 0x56338d483ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb43f7fe640 1 -- 172.21.15.33:0/3054509589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d47b400 msgr2=0x56338d48e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb43f7fe640 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d47b400 0x56338d48e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb43f7fe640 1 -- 172.21.15.33:0/3054509589 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56338d48f020 con 0x56338d46d4a0 2024-09-16T00:34:04.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.018+0000 7fb444e0f640 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d47b400 0x56338d48e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.019+0000 7fb43ffff640 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 0x56338d483ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.019+0000 7fb43f7fe640 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d48e3d0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fb43409bf20 tx=0x7fb43409c220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb0a97457c7b17a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.019+0000 7fb41f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.019+0000 7fb41f7fe640 1 -- 172.21.15.33:0/3054509589 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4340bae60 con 0x56338d46d4a0 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.019+0000 7fb448c4b740 1 -- 172.21.15.33:0/3054509589 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56338d3d4d00 con 0x56338d46d4a0 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.019+0000 7fb41f7fe640 1 -- 172.21.15.33:0/3054509589 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb434003b30 con 0x56338d46d4a0 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.019+0000 7fb41f7fe640 1 -- 172.21.15.33:0/3054509589 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb434098890 con 0x56338d46d4a0 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.019+0000 7fb448c4b740 10 client.18609 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.019+0000 7fb448c4b740 1 -- 172.21.15.33:0/3054509589 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56338d4edb60 con 0x56338d46d4a0 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.019+0000 7fb41f7fe640 1 -- 172.21.15.33:0/3054509589 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb43407ae50 con 0x56338d46d4a0 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.020+0000 7fb41f7fe640 1 -- 172.21.15.33:0/3054509589 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb4340cdbc0 con 0x56338d46d4a0 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.020+0000 7fb448c4b740 10 client.18609 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:04.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.020+0000 7fb448c4b740 10 client.18609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:04.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.020+0000 7fb448c4b740 20 client.18609 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:04.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.020+0000 7fb448c4b740 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56338d4ee2c0 0x56338d4f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:04.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.020+0000 7fb448c4b740 4 client.18609 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:34:04.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.020+0000 7fb448c4b740 1 -- 172.21.15.33:0/3054509589 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56338d351110 con 0x56338d4ee2c0 2024-09-16T00:34:04.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.020+0000 7fb43ffff640 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56338d4ee2c0 0x56338d4f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:04.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.020+0000 7fb43ffff640 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56338d4ee2c0 0x56338d4f06e0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f68e899a35b20ec server_cookie=def3a30f1ee351c in_seq=0 out_seq=0 2024-09-16T00:34:04.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.020+0000 7fb41f7fe640 10 client.18609 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:04.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.021+0000 7fb41f7fe640 1 -- 172.21.15.33:0/3054509589 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56338d351110 con 0x56338d4ee2c0 2024-09-16T00:34:04.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.021+0000 7fb41f7fe640 10 client.18609 handle_command_reply: tid=0 2024-09-16T00:34:04.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.021+0000 7fb448c4b740 1 client.18609 shutdown 2024-09-16T00:34:04.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.021+0000 7fb448c4b740 1 -- 172.21.15.33:0/3054509589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 msgr2=0x56338d48e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:04.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.021+0000 7fb448c4b740 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d48e3d0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fb43409bf20 tx=0x7fb43409c220 comp rx=0 tx=0).stop 2024-09-16T00:34:04.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.022+0000 7fb448c4b740 1 -- 172.21.15.33:0/3054509589 shutdown_connections 2024-09-16T00:34:04.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.022+0000 7fb448c4b740 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56338d4ee2c0 0x56338d4f06e0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.022+0000 7fb448c4b740 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56338d47b400 0x56338d48e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.022+0000 7fb41f7fe640 0 client.18609 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:04.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.022+0000 7fb448c4b740 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56338d46d4a0 0x56338d48e3d0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.022+0000 7fb448c4b740 1 --2- 172.21.15.33:0/3054509589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56338d46c4b0 0x56338d483ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.022+0000 7fb448c4b740 1 -- 172.21.15.33:0/3054509589 >> 172.21.15.33:0/3054509589 conn(0x56338d468440 msgr2=0x56338d463270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:04.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.022+0000 7fb448c4b740 1 -- 172.21.15.33:0/3054509589 shutdown_connections 2024-09-16T00:34:04.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.022+0000 7fb448c4b740 1 -- 172.21.15.33:0/3054509589 wait complete. 2024-09-16T00:34:04.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.022+0000 7fb448c4b740 20 client.18609 trim_cache size 0 max 16384 2024-09-16T00:34:04.027 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:34:04.028 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:34:04.028 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "4c61ae00-83c8-4627-9b85-171aff43ab66", 2024-09-16T00:34:04.028 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:34:04.028 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:34:04.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.022+0000 7fb446872640 1 -- 172.21.15.33:0/2497731147 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4280a3240 msgr2=0x7fb4280a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:04.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.023+0000 7fb446872640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4280a3240 0x7fb4280a5700 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb430006010 tx=0x7fb430000bc0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.023+0000 7fb446872640 1 -- 172.21.15.33:0/2497731147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb440100ca0 msgr2=0x7fb4401989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:04.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.023+0000 7fb446872640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb440100ca0 0x7fb4401989f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb43400e9c0 tx=0x7fb43400ee80 comp rx=0 tx=0).stop 2024-09-16T00:34:04.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.023+0000 7fb446872640 1 -- 172.21.15.33:0/2497731147 shutdown_connections 2024-09-16T00:34:04.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.023+0000 7fb446872640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb4280a3240 0x7fb4280a5700 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.023+0000 7fb446872640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb440106f50 0x7fb44019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.023+0000 7fb446872640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb440100ca0 0x7fb4401989f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.023+0000 7fb446872640 1 --2- 172.21.15.33:0/2497731147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4401002f0 0x7fb4401984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:04.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.023+0000 7fb446872640 1 -- 172.21.15.33:0/2497731147 >> 172.21.15.33:0/2497731147 conn(0x7fb440077210 msgr2=0x7fb440105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:04.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.023+0000 7fb446872640 1 -- 172.21.15.33:0/2497731147 shutdown_connections 2024-09-16T00:34:04.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:04.023+0000 7fb446872640 1 -- 172.21.15.33:0/2497731147 wait complete. 2024-09-16T00:34:04.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 69857 19.92 MB/sec execute 70 sec latency 67.768 ms 2024-09-16T00:34:04.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:04 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:04.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:04 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:04.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:04 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:04.967 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:34:05.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 70401 19.98 MB/sec execute 71 sec latency 189.925 ms 2024-09-16T00:34:05.268 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:05.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: pgmap v858: 129 pgs: 129 active+clean; 536 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 16 MiB/s wr, 759 op/s 2024-09-16T00:34:05.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:34:05.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:05.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:05.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:05.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.636 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:05 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: pgmap v858: 129 pgs: 129 active+clean; 536 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 16 MiB/s wr, 759 op/s 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:05 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: pgmap v858: 129 pgs: 129 active+clean; 536 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 16 MiB/s wr, 759 op/s 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:05.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:05.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:05.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:05.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:05.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:05 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:05.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.880+0000 7f0d3f195640 1 -- 172.21.15.33:0/945348905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d38102570 msgr2=0x7f0d381029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:05.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.880+0000 7f0d3f195640 1 --2- 172.21.15.33:0/945348905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d38102570 0x7f0d381029b0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009900 tx=0x7f0d2402a480 comp rx=0 tx=0).stop 2024-09-16T00:34:05.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.880+0000 7f0d3f195640 1 -- 172.21.15.33:0/945348905 shutdown_connections 2024-09-16T00:34:05.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.880+0000 7f0d3f195640 1 --2- 172.21.15.33:0/945348905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d38102ef0 0x7f0d3810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.880+0000 7f0d3f195640 1 --2- 172.21.15.33:0/945348905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d38102570 0x7f0d381029b0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.880+0000 7f0d3f195640 1 --2- 172.21.15.33:0/945348905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d38100e20 0x7f0d38101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.880+0000 7f0d3f195640 1 -- 172.21.15.33:0/945348905 >> 172.21.15.33:0/945348905 conn(0x7f0d380fc7d0 msgr2=0x7f0d380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:05.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.880+0000 7f0d3f195640 1 -- 172.21.15.33:0/945348905 shutdown_connections 2024-09-16T00:34:05.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.880+0000 7f0d3f195640 1 -- 172.21.15.33:0/945348905 wait complete. 2024-09-16T00:34:05.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d3f195640 1 Processor -- start 2024-09-16T00:34:05.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d3f195640 1 -- start start 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d3f195640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d38100e20 0x7f0d38199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d3f195640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d38102570 0x7f0d38199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d3f195640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d38102ef0 0x7f0d381a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d3f195640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d38110f80 con 0x7f0d38100e20 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d3f195640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0d38110e00 con 0x7f0d38102570 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d3f195640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0d38111100 con 0x7f0d38102ef0 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d3cf31640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d38100e20 0x7f0d38199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d3cf31640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d38100e20 0x7f0d38199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55122/0 (socket says 172.21.15.33:55122) 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d3cf31640 1 -- 172.21.15.33:0/892526256 learned_addr learned my addr 172.21.15.33:0/892526256 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d3d732640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d38102ef0 0x7f0d381a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.881+0000 7f0d37fff640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d38102570 0x7f0d38199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.882+0000 7f0d3cf31640 1 -- 172.21.15.33:0/892526256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d38102ef0 msgr2=0x7f0d381a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.882+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d38102ef0 0x7f0d381a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.882+0000 7f0d3cf31640 1 -- 172.21.15.33:0/892526256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d38102570 msgr2=0x7f0d38199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.882+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d38102570 0x7f0d38199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.882+0000 7f0d3cf31640 1 -- 172.21.15.33:0/892526256 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d381a4910 con 0x7f0d38100e20 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.882+0000 7f0d37fff640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d38102570 0x7f0d38199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.882+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d38100e20 0x7f0d38199350 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00e9f0 tx=0x7f0d2c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.882+0000 7f0d35ffb640 1 -- 172.21.15.33:0/892526256 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c00cd80 con 0x7f0d38100e20 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.882+0000 7f0d35ffb640 1 -- 172.21.15.33:0/892526256 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d2c010910 con 0x7f0d38100e20 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.882+0000 7f0d3f195640 1 -- 172.21.15.33:0/892526256 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d381a4ae0 con 0x7f0d38100e20 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.882+0000 7f0d35ffb640 1 -- 172.21.15.33:0/892526256 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c018990 con 0x7f0d38100e20 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.884+0000 7f0d3f195640 1 -- 172.21.15.33:0/892526256 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d381a4ff0 con 0x7f0d38100e20 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.886+0000 7f0d35ffb640 1 -- 172.21.15.33:0/892526256 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0d2c010ac0 con 0x7f0d38100e20 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.888+0000 7f0d35ffb640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d280a32c0 0x7f0d280a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.888+0000 7f0d37fff640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d280a32c0 0x7f0d280a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.888+0000 7f0d35ffb640 1 -- 172.21.15.33:0/892526256 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0d2c0d0000 con 0x7f0d38100e20 2024-09-16T00:34:05.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.889+0000 7f0d37fff640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d280a32c0 0x7f0d280a5780 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009230 tx=0x7f0d240058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:05.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.889+0000 7f0d4156e740 1 Processor -- start 2024-09-16T00:34:05.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.889+0000 7f0d4156e740 1 -- start start 2024-09-16T00:34:05.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.890+0000 7f0d4156e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 0x561889055880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.890+0000 7f0d4156e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561889055dc0 0x561889060140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.890+0000 7f0d4156e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x561889062a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.890+0000 7f0d4156e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561888fbb2e0 con 0x5618890554a0 2024-09-16T00:34:05.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.890+0000 7f0d4156e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561888f62ce0 con 0x561889060680 2024-09-16T00:34:05.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.890+0000 7f0d4156e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561888fb72a0 con 0x561889055dc0 2024-09-16T00:34:05.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.890+0000 7f0d3d732640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x561889062a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.890+0000 7f0d3d732640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x561889062a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:57022/0 (socket says 172.21.15.33:57022) 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.890+0000 7f0d3d732640 1 -- 172.21.15.33:0/898681836 learned_addr learned my addr 172.21.15.33:0/898681836 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.890+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/898681836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 0x561889055880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.891+0000 7f0d3d732640 1 -- 172.21.15.33:0/898681836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561889055dc0 msgr2=0x561889060140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.891+0000 7f0d3d732640 1 --2- 172.21.15.33:0/898681836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561889055dc0 0x561889060140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.891+0000 7f0d3d732640 1 -- 172.21.15.33:0/898681836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 msgr2=0x561889055880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.891+0000 7f0d3d732640 1 --2- 172.21.15.33:0/898681836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 0x561889055880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.891+0000 7f0d3d732640 1 -- 172.21.15.33:0/898681836 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561888f47440 con 0x561889060680 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.891+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/898681836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 0x561889055880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.891+0000 7f0d3d732640 1 --2- 172.21.15.33:0/898681836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x561889062a70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0d30009f50 tx=0x7f0d300118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1676c5834567bb35 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.892+0000 7f0d22ffd640 1 -- 172.21.15.33:0/898681836 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30004d20 con 0x561889060680 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.892+0000 7f0d22ffd640 1 -- 172.21.15.33:0/898681836 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d30004050 con 0x561889060680 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.892+0000 7f0d22ffd640 1 -- 172.21.15.33:0/898681836 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30005280 con 0x561889060680 2024-09-16T00:34:05.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.893+0000 7f0d4156e740 1 -- 172.21.15.33:0/898681836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 msgr2=0x561889062a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.893+0000 7f0d4156e740 1 --2- 172.21.15.33:0/898681836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x561889062a70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0d30009f50 tx=0x7f0d300118d0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.893+0000 7f0d4156e740 1 -- 172.21.15.33:0/898681836 shutdown_connections 2024-09-16T00:34:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.893+0000 7f0d4156e740 1 --2- 172.21.15.33:0/898681836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x561889062a70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.893+0000 7f0d4156e740 1 --2- 172.21.15.33:0/898681836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561889055dc0 0x561889060140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.893+0000 7f0d4156e740 1 --2- 172.21.15.33:0/898681836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 0x561889055880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.893+0000 7f0d4156e740 1 -- 172.21.15.33:0/898681836 >> 172.21.15.33:0/898681836 conn(0x561889051a30 msgr2=0x561889051e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:05.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.893+0000 7f0d4156e740 1 -- 172.21.15.33:0/898681836 shutdown_connections 2024-09-16T00:34:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.893+0000 7f0d4156e740 1 -- 172.21.15.33:0/898681836 wait complete. 2024-09-16T00:34:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.894+0000 7f0d4156e740 1 -- start start 2024-09-16T00:34:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.894+0000 7f0d4156e740 1 Processor -- start 2024-09-16T00:34:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.894+0000 7f0d4156e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618890554a0 0x561889060030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.894+0000 7f0d4156e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561889055dc0 0x56188905d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.894+0000 7f0d4156e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x56188905d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.895+0000 7f0d4156e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561889063250 con 0x561889055dc0 2024-09-16T00:34:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.895+0000 7f0d3d732640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x56188905d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.895+0000 7f0d4156e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561888f76050 con 0x561889060680 2024-09-16T00:34:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.895+0000 7f0d3d732640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x56188905d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:57024/0 (socket says 172.21.15.33:57024) 2024-09-16T00:34:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.895+0000 7f0d3d732640 1 -- 172.21.15.33:0/1472209114 learned_addr learned my addr 172.21.15.33:0/1472209114 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:05.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.895+0000 7f0d37fff640 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561889055dc0 0x56188905d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.895+0000 7f0d4156e740 1 -- 172.21.15.33:0/1472209114 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5618890633d0 con 0x5618890554a0 2024-09-16T00:34:05.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.895+0000 7f0d4156e740 10 client.0 _ll_register_callbacks cb 0x561888fb6160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:05.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.896+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618890554a0 0x561889060030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.896+0000 7f0d3cf31640 1 -- 172.21.15.33:0/1472209114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 msgr2=0x56188905d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:05.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.896+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x56188905d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.896+0000 7f0d3cf31640 1 -- 172.21.15.33:0/1472209114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561889055dc0 msgr2=0x56188905d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:05.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.896+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561889055dc0 0x56188905d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.896+0000 7f0d3cf31640 1 -- 172.21.15.33:0/1472209114 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561888d2c3c0 con 0x5618890554a0 2024-09-16T00:34:05.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.897+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618890554a0 0x561889060030 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c0cd9f0 tx=0x7f0d2c0a0610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faf4a9cb3da00ae1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:05.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.897+0000 7f0d17fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:34:05.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.897+0000 7f0d17fff640 1 -- 172.21.15.33:0/1472209114 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c0cdb40 con 0x5618890554a0 2024-09-16T00:34:05.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.897+0000 7f0d17fff640 1 -- 172.21.15.33:0/1472209114 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0d2c018b30 con 0x5618890554a0 2024-09-16T00:34:05.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.897+0000 7f0d4156e740 1 -- 172.21.15.33:0/1472209114 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d30004050 con 0x5618890554a0 2024-09-16T00:34:05.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.898+0000 7f0d17fff640 1 -- 172.21.15.33:0/1472209114 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0d2c018d30 con 0x5618890554a0 2024-09-16T00:34:05.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.898+0000 7f0d4156e740 10 client.27925 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:05.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.898+0000 7f0d4156e740 1 -- 172.21.15.33:0/1472209114 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561889077500 con 0x5618890554a0 2024-09-16T00:34:05.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.898+0000 7f0d17fff640 1 -- 172.21.15.33:0/1472209114 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d30004050 con 0x5618890554a0 2024-09-16T00:34:05.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.899+0000 7f0d17fff640 1 -- 172.21.15.33:0/1472209114 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0d2c0cdce0 con 0x5618890554a0 2024-09-16T00:34:05.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.899+0000 7f0d4156e740 10 client.27925 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:05.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.899+0000 7f0d4156e740 10 client.27925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:05.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.899+0000 7f0d4156e740 20 client.27925 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:05.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.899+0000 7f0d4156e740 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5618890789c0 0x56188907add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.899+0000 7f0d4156e740 4 client.27925 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:34:05.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.899+0000 7f0d4156e740 1 -- 172.21.15.33:0/1472209114 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561888fc8850 con 0x5618890789c0 2024-09-16T00:34:05.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.900+0000 7f0d37fff640 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5618890789c0 0x56188907add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.900+0000 7f0d37fff640 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5618890789c0 0x56188907add0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bce8e493c84c9dbf server_cookie=7ecb806d498cf275 in_seq=0 out_seq=0 2024-09-16T00:34:05.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.900+0000 7f0d17fff640 10 client.27925 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:05.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.901+0000 7f0d17fff640 1 -- 172.21.15.33:0/1472209114 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561888fc8850 con 0x5618890789c0 2024-09-16T00:34:05.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.901+0000 7f0d17fff640 10 client.27925 handle_command_reply: tid=0 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.901+0000 7f0d4156e740 1 client.27925 shutdown 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.901+0000 7f0d4156e740 1 -- 172.21.15.33:0/1472209114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618890554a0 msgr2=0x561889060030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.901+0000 7f0d4156e740 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618890554a0 0x561889060030 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c0cd9f0 tx=0x7f0d2c0a0610 comp rx=0 tx=0).stop 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.901+0000 7f0d4156e740 1 -- 172.21.15.33:0/1472209114 shutdown_connections 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.901+0000 7f0d4156e740 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5618890789c0 0x56188907add0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.902+0000 7f0d4156e740 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x56188905d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.902+0000 7f0d17fff640 0 client.27925 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.902+0000 7f0d4156e740 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561889055dc0 0x56188905d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.902+0000 7f0d4156e740 1 --2- 172.21.15.33:0/1472209114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618890554a0 0x561889060030 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.902+0000 7f0d4156e740 1 -- 172.21.15.33:0/1472209114 >> 172.21.15.33:0/1472209114 conn(0x561889051a30 msgr2=0x561889057760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.902+0000 7f0d4156e740 1 -- 172.21.15.33:0/1472209114 shutdown_connections 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.902+0000 7f0d4156e740 1 -- 172.21.15.33:0/1472209114 wait complete. 2024-09-16T00:34:05.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.902+0000 7f0d4156e740 20 client.27925 trim_cache size 0 max 16384 2024-09-16T00:34:05.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d4156e740 1 Processor -- start 2024-09-16T00:34:05.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d4156e740 1 -- start start 2024-09-16T00:34:05.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d4156e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 0x561889074d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d4156e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889055dc0 0x561889080690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d4156e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561889060680 0x561889080bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d4156e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d381af810 con 0x5618890554a0 2024-09-16T00:34:05.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d4156e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0d381ad0e0 con 0x561889055dc0 2024-09-16T00:34:05.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d4156e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0d381aaf30 con 0x561889060680 2024-09-16T00:34:05.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d3cf31640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 0x561889074d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d3cf31640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 0x561889074d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55148/0 (socket says 172.21.15.33:55148) 2024-09-16T00:34:05.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d3cf31640 1 -- 172.21.15.33:0/834946958 learned_addr learned my addr 172.21.15.33:0/834946958 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:05.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d37fff640 1 --2- 172.21.15.33:0/834946958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889055dc0 0x561889080690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.923+0000 7f0d3d732640 1 --2- 172.21.15.33:0/834946958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561889060680 0x561889080bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.924+0000 7f0d3cf31640 1 -- 172.21.15.33:0/834946958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561889060680 msgr2=0x561889080bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:05.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.924+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/834946958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561889060680 0x561889080bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.924+0000 7f0d3cf31640 1 -- 172.21.15.33:0/834946958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889055dc0 msgr2=0x561889080690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:05.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.924+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/834946958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889055dc0 0x561889080690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.924+0000 7f0d3cf31640 1 -- 172.21.15.33:0/834946958 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618890552d0 con 0x5618890554a0 2024-09-16T00:34:05.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.924+0000 7f0d37fff640 1 --2- 172.21.15.33:0/834946958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889055dc0 0x561889080690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:05.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.924+0000 7f0d3d732640 1 --2- 172.21.15.33:0/834946958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561889060680 0x561889080bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:34:05.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.924+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/834946958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 0x561889074d70 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00c6a0 tx=0x7f0d2c0981b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5999ad198a2bc86 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:05.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.924+0000 7f0d227fc640 1 -- 172.21.15.33:0/834946958 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c09c250 con 0x5618890554a0 2024-09-16T00:34:05.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.924+0000 7f0d227fc640 1 -- 172.21.15.33:0/834946958 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d2c09b6a0 con 0x5618890554a0 2024-09-16T00:34:05.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.924+0000 7f0d227fc640 1 -- 172.21.15.33:0/834946958 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c0cdda0 con 0x5618890554a0 2024-09-16T00:34:05.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.925+0000 7f0d4156e740 1 -- 172.21.15.33:0/834946958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 msgr2=0x561889074d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:05.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.925+0000 7f0d4156e740 1 --2- 172.21.15.33:0/834946958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 0x561889074d70 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00c6a0 tx=0x7f0d2c0981b0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.925+0000 7f0d4156e740 1 -- 172.21.15.33:0/834946958 shutdown_connections 2024-09-16T00:34:05.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.925+0000 7f0d4156e740 1 --2- 172.21.15.33:0/834946958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561889060680 0x561889080bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.925+0000 7f0d4156e740 1 --2- 172.21.15.33:0/834946958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889055dc0 0x561889080690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.925+0000 7f0d4156e740 1 --2- 172.21.15.33:0/834946958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618890554a0 0x561889074d70 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.925+0000 7f0d4156e740 1 -- 172.21.15.33:0/834946958 >> 172.21.15.33:0/834946958 conn(0x561889051a30 msgr2=0x5618890588e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.925+0000 7f0d4156e740 1 -- 172.21.15.33:0/834946958 shutdown_connections 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.925+0000 7f0d4156e740 1 -- 172.21.15.33:0/834946958 wait complete. 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.925+0000 7f0d4156e740 1 -- start start 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.925+0000 7f0d4156e740 1 Processor -- start 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d4156e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618890554a0 0x56188906d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d4156e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561889055dc0 0x56188906d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d4156e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x5618890e2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d4156e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d381ab5a0 con 0x561889055dc0 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d4156e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0d381aca10 con 0x561889060680 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d4156e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0d381abc70 con 0x5618890554a0 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d4156e740 10 client.0 _ll_register_callbacks cb 0x7f0d2c09c0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:05.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d37fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561889055dc0 0x56188906d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d37fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561889055dc0 0x56188906d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55164/0 (socket says 172.21.15.33:55164) 2024-09-16T00:34:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d3d732640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x5618890e2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d37fff640 1 -- 172.21.15.33:0/842852314 learned_addr learned my addr 172.21.15.33:0/842852314 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618890554a0 0x56188906d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d37fff640 1 -- 172.21.15.33:0/842852314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618890554a0 msgr2=0x56188906d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d37fff640 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618890554a0 0x56188906d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d37fff640 1 -- 172.21.15.33:0/842852314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 msgr2=0x5618890e2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d37fff640 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x5618890e2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d37fff640 1 -- 172.21.15.33:0/842852314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618890e2ba0 con 0x561889055dc0 2024-09-16T00:34:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d3d732640 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x5618890e2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d37fff640 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561889055dc0 0x56188906d580 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009900 tx=0x7f0d2403e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff871d3a0106f902 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.926+0000 7f0d17fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d17fff640 1 -- 172.21.15.33:0/842852314 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24035050 con 0x561889055dc0 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d17fff640 1 -- 172.21.15.33:0/842852314 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0d24031a10 con 0x561889055dc0 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d4156e740 1 -- 172.21.15.33:0/842852314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d2c09b6a0 con 0x561889055dc0 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d17fff640 1 -- 172.21.15.33:0/842852314 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0d24062050 con 0x561889055dc0 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d4156e740 10 client.18627 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d4156e740 1 -- 172.21.15.33:0/842852314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5618890e3d50 con 0x561889055dc0 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d17fff640 1 -- 172.21.15.33:0/842852314 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d2c09b6a0 con 0x561889055dc0 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d17fff640 1 -- 172.21.15.33:0/842852314 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0d2404a430 con 0x561889055dc0 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d4156e740 10 client.18627 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d4156e740 10 client.18627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d4156e740 20 client.18627 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d4156e740 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5618890789c0 0x5618890e6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:05.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d4156e740 4 client.18627 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.927+0000 7f0d4156e740 1 -- 172.21.15.33:0/842852314 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561888fc2d60 con 0x5618890789c0 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5618890789c0 0x5618890e6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d17fff640 10 client.18627 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d3cf31640 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5618890789c0 0x5618890e6540 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dea2690cf6b7799d server_cookie=e4a9b1676ed8fb1b in_seq=0 out_seq=0 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d17fff640 1 -- 172.21.15.33:0/842852314 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561888fc2d60 con 0x5618890789c0 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d17fff640 10 client.18627 handle_command_reply: tid=0 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d4156e740 1 client.18627 shutdown 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d4156e740 1 -- 172.21.15.33:0/842852314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561889055dc0 msgr2=0x56188906d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d4156e740 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561889055dc0 0x56188906d580 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009900 tx=0x7f0d2403e070 comp rx=0 tx=0).stop 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d4156e740 1 -- 172.21.15.33:0/842852314 shutdown_connections 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d4156e740 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5618890789c0 0x5618890e6540 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d4156e740 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561889060680 0x5618890e2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d4156e740 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561889055dc0 0x56188906d580 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d17fff640 0 client.18627 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:05.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d4156e740 1 --2- 172.21.15.33:0/842852314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618890554a0 0x56188906d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.928+0000 7f0d4156e740 1 -- 172.21.15.33:0/842852314 >> 172.21.15.33:0/842852314 conn(0x561889051a30 msgr2=0x561889050350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:05.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d4156e740 1 -- 172.21.15.33:0/842852314 shutdown_connections 2024-09-16T00:34:05.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d4156e740 1 -- 172.21.15.33:0/842852314 wait complete. 2024-09-16T00:34:05.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d4156e740 20 client.18627 trim_cache size 0 max 16384 2024-09-16T00:34:05.935 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:34:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:34:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:34:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "4c61ae00-83c8-4627-9b85-171aff43ab66": { 2024-09-16T00:34:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:34:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "4c61ae00-83c8-4627-9b85-171aff43ab66", 2024-09-16T00:34:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:34:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:05.936 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:34:05.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d3f195640 1 -- 172.21.15.33:0/892526256 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d280a32c0 msgr2=0x7f0d280a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:05.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d3f195640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d280a32c0 0x7f0d280a5780 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009230 tx=0x7f0d240058d0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d3f195640 1 -- 172.21.15.33:0/892526256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d38100e20 msgr2=0x7f0d38199350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:05.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d3f195640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d38100e20 0x7f0d38199350 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00e9f0 tx=0x7f0d2c00eeb0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d3f195640 1 -- 172.21.15.33:0/892526256 shutdown_connections 2024-09-16T00:34:05.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d3f195640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d280a32c0 0x7f0d280a5780 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d3f195640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d38102ef0 0x7f0d381a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d3f195640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d38102570 0x7f0d38199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d3f195640 1 --2- 172.21.15.33:0/892526256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d38100e20 0x7f0d38199350 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:05.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.929+0000 7f0d3f195640 1 -- 172.21.15.33:0/892526256 >> 172.21.15.33:0/892526256 conn(0x7f0d380fc7d0 msgr2=0x7f0d381068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:05.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.930+0000 7f0d3f195640 1 -- 172.21.15.33:0/892526256 shutdown_connections 2024-09-16T00:34:05.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:05.930+0000 7f0d3f195640 1 -- 172.21.15.33:0/892526256 wait complete. 2024-09-16T00:34:06.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 71211 20.45 MB/sec execute 72 sec latency 66.776 ms 2024-09-16T00:34:06.754 INFO:tasks.cephfs.filesystem:scrub status for tag:4c61ae00-83c8-4627-9b85-171aff43ab66 - {'path': '/', 'tag': '4c61ae00-83c8-4627-9b85-171aff43ab66', 'options': 'recursive,force'} 2024-09-16T00:34:07.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 71805 20.66 MB/sec execute 73 sec latency 59.502 ms 2024-09-16T00:34:07.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:07 smithi083 ceph-mon[35242]: pgmap v859: 129 pgs: 129 active+clean; 539 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 13 MiB/s wr, 657 op/s 2024-09-16T00:34:07.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:07 smithi060 ceph-mon[35185]: pgmap v859: 129 pgs: 129 active+clean; 539 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 13 MiB/s wr, 657 op/s 2024-09-16T00:34:07.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:07 smithi033 ceph-mon[31243]: pgmap v859: 129 pgs: 129 active+clean; 539 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 13 MiB/s wr, 657 op/s 2024-09-16T00:34:08.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 72175 20.63 MB/sec execute 74 sec latency 95.274 ms 2024-09-16T00:34:09.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 72684 20.74 MB/sec execute 75 sec latency 78.581 ms 2024-09-16T00:34:09.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:09 smithi083 ceph-mon[35242]: pgmap v860: 129 pgs: 129 active+clean; 544 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 7.7 MiB/s wr, 519 op/s 2024-09-16T00:34:09.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:09 smithi060 ceph-mon[35185]: pgmap v860: 129 pgs: 129 active+clean; 544 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 7.7 MiB/s wr, 519 op/s 2024-09-16T00:34:09.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:09 smithi033 ceph-mon[31243]: pgmap v860: 129 pgs: 129 active+clean; 544 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 7.7 MiB/s wr, 519 op/s 2024-09-16T00:34:10.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 73232 20.88 MB/sec execute 76 sec latency 80.854 ms 2024-09-16T00:34:11.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 73840 20.95 MB/sec execute 77 sec latency 62.184 ms 2024-09-16T00:34:11.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:11 smithi083 ceph-mon[35242]: pgmap v861: 129 pgs: 129 active+clean; 548 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.5 MiB/s wr, 554 op/s 2024-09-16T00:34:11.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:11 smithi060 ceph-mon[35185]: pgmap v861: 129 pgs: 129 active+clean; 548 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.5 MiB/s wr, 554 op/s 2024-09-16T00:34:11.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:11 smithi033 ceph-mon[31243]: pgmap v861: 129 pgs: 129 active+clean; 548 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.5 MiB/s wr, 554 op/s 2024-09-16T00:34:12.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 74432 21.02 MB/sec execute 78 sec latency 62.352 ms 2024-09-16T00:34:13.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 75112 21.30 MB/sec execute 79 sec latency 76.821 ms 2024-09-16T00:34:13.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:13 smithi083 ceph-mon[35242]: pgmap v862: 129 pgs: 129 active+clean; 552 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.9 MiB/s wr, 494 op/s 2024-09-16T00:34:13.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:13 smithi060 ceph-mon[35185]: pgmap v862: 129 pgs: 129 active+clean; 552 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.9 MiB/s wr, 494 op/s 2024-09-16T00:34:13.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:13 smithi033 ceph-mon[31243]: pgmap v862: 129 pgs: 129 active+clean; 552 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.9 MiB/s wr, 494 op/s 2024-09-16T00:34:14.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 75728 21.44 MB/sec execute 80 sec latency 66.628 ms 2024-09-16T00:34:15.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 76265 21.60 MB/sec execute 81 sec latency 68.591 ms 2024-09-16T00:34:15.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:15 smithi083 ceph-mon[35242]: pgmap v863: 129 pgs: 129 active+clean; 549 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.3 MiB/s wr, 417 op/s 2024-09-16T00:34:15.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:15 smithi060 ceph-mon[35185]: pgmap v863: 129 pgs: 129 active+clean; 549 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.3 MiB/s wr, 417 op/s 2024-09-16T00:34:15.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:15 smithi033 ceph-mon[31243]: pgmap v863: 129 pgs: 129 active+clean; 549 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.3 MiB/s wr, 417 op/s 2024-09-16T00:34:16.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 76801 21.67 MB/sec execute 82 sec latency 106.999 ms 2024-09-16T00:34:17.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 77338 21.78 MB/sec execute 83 sec latency 58.309 ms 2024-09-16T00:34:17.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:17 smithi083 ceph-mon[35242]: pgmap v864: 129 pgs: 129 active+clean; 546 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.3 MiB/s wr, 383 op/s 2024-09-16T00:34:17.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:17 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:34:17.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:17 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:34:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:17 smithi060 ceph-mon[35185]: pgmap v864: 129 pgs: 129 active+clean; 546 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.3 MiB/s wr, 383 op/s 2024-09-16T00:34:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:17 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:34:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:17 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:34:17.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:17 smithi033 ceph-mon[31243]: pgmap v864: 129 pgs: 129 active+clean; 546 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.3 MiB/s wr, 383 op/s 2024-09-16T00:34:17.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:17 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:34:17.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:17 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:34:18.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 77950 21.84 MB/sec execute 84 sec latency 75.084 ms 2024-09-16T00:34:18.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:18 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:34:18.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:18 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:34:18.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:18 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:34:19.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 78666 22.06 MB/sec execute 85 sec latency 65.724 ms 2024-09-16T00:34:19.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:19 smithi083 ceph-mon[35242]: pgmap v865: 129 pgs: 129 active+clean; 546 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 383 op/s 2024-09-16T00:34:19.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:19 smithi060 ceph-mon[35185]: pgmap v865: 129 pgs: 129 active+clean; 546 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 383 op/s 2024-09-16T00:34:19.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:19 smithi033 ceph-mon[31243]: pgmap v865: 129 pgs: 129 active+clean; 546 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 383 op/s 2024-09-16T00:34:20.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 79284 22.17 MB/sec execute 86 sec latency 70.241 ms 2024-09-16T00:34:21.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 79917 22.31 MB/sec execute 87 sec latency 60.748 ms 2024-09-16T00:34:21.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:21 smithi083 ceph-mon[35242]: pgmap v866: 129 pgs: 129 active+clean; 549 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.0 MiB/s wr, 354 op/s 2024-09-16T00:34:21.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:21 smithi060 ceph-mon[35185]: pgmap v866: 129 pgs: 129 active+clean; 549 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.0 MiB/s wr, 354 op/s 2024-09-16T00:34:21.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:21 smithi033 ceph-mon[31243]: pgmap v866: 129 pgs: 129 active+clean; 549 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.0 MiB/s wr, 354 op/s 2024-09-16T00:34:22.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 80614 22.59 MB/sec execute 88 sec latency 49.244 ms 2024-09-16T00:34:23.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 80824 22.41 MB/sec execute 89 sec latency 591.584 ms 2024-09-16T00:34:23.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:23 smithi083 ceph-mon[35242]: pgmap v867: 129 pgs: 129 active+clean; 559 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.6 MiB/s wr, 412 op/s 2024-09-16T00:34:23.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:23 smithi060 ceph-mon[35185]: pgmap v867: 129 pgs: 129 active+clean; 559 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.6 MiB/s wr, 412 op/s 2024-09-16T00:34:23.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:23 smithi033 ceph-mon[31243]: pgmap v867: 129 pgs: 129 active+clean; 559 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.6 MiB/s wr, 412 op/s 2024-09-16T00:34:24.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 80844 22.16 MB/sec execute 90 sec latency 1591.628 ms 2024-09-16T00:34:25.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 81030 21.98 MB/sec execute 91 sec latency 1602.026 ms 2024-09-16T00:34:25.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:25 smithi083 ceph-mon[35242]: pgmap v868: 129 pgs: 129 active+clean; 582 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 5.5 MiB/s wr, 437 op/s 2024-09-16T00:34:25.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:25 smithi060 ceph-mon[35185]: pgmap v868: 129 pgs: 129 active+clean; 582 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 5.5 MiB/s wr, 437 op/s 2024-09-16T00:34:25.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:25 smithi033 ceph-mon[31243]: pgmap v868: 129 pgs: 129 active+clean; 582 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 5.5 MiB/s wr, 437 op/s 2024-09-16T00:34:26.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 81326 21.92 MB/sec execute 92 sec latency 152.293 ms 2024-09-16T00:34:27.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 81753 21.94 MB/sec execute 93 sec latency 83.583 ms 2024-09-16T00:34:27.555 INFO:tasks.quiescer.fs.[cephfs]:Total duration (45.9) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T00:34:27.555 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.95 2024-09-16T00:34:27.555 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:34:27.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:27 smithi083 ceph-mon[35242]: pgmap v869: 129 pgs: 129 active+clean; 627 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 11 MiB/s wr, 520 op/s 2024-09-16T00:34:27.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:27 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:34:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:27 smithi060 ceph-mon[35185]: pgmap v869: 129 pgs: 129 active+clean; 627 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 11 MiB/s wr, 520 op/s 2024-09-16T00:34:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:27 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:34:27.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:27 smithi033 ceph-mon[31243]: pgmap v869: 129 pgs: 129 active+clean; 627 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 11 MiB/s wr, 520 op/s 2024-09-16T00:34:27.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:27 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:34:28.062 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:28.076 INFO:tasks.workunit.client.0.smithi033.stdout: 10 82075 21.86 MB/sec execute 94 sec latency 290.070 ms 2024-09-16T00:34:28.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi060 to 177.0M 2024-09-16T00:34:28.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: Unable to set osd_memory_target on smithi060 to 185699225: error parsing value: Value '185699225' is below minimum 939524096 2024-09-16T00:34:28.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.459 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.459 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.459 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.459 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.459 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.459 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:34:28.459 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:34:28.459 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:34:28.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.548+0000 7fe70ba50640 1 -- 172.21.15.33:0/373157380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe704102d90 msgr2=0x7fe704103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:28.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.548+0000 7fe70ba50640 1 --2- 172.21.15.33:0/373157380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe704102d90 0x7fe704103170 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fe700009930 tx=0x7fe70002a420 comp rx=0 tx=0).stop 2024-09-16T00:34:28.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.548+0000 7fe70ba50640 1 -- 172.21.15.33:0/373157380 shutdown_connections 2024-09-16T00:34:28.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.548+0000 7fe70ba50640 1 --2- 172.21.15.33:0/373157380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe704108290 0x7fe70410ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:28.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.548+0000 7fe70ba50640 1 --2- 172.21.15.33:0/373157380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe704103740 0x7fe704107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:28.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.548+0000 7fe70ba50640 1 --2- 172.21.15.33:0/373157380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe704102d90 0x7fe704103170 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:28.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.548+0000 7fe70ba50640 1 -- 172.21.15.33:0/373157380 >> 172.21.15.33:0/373157380 conn(0x7fe7040fc7d0 msgr2=0x7fe7040febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:28.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.548+0000 7fe70ba50640 1 -- 172.21.15.33:0/373157380 shutdown_connections 2024-09-16T00:34:28.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.549+0000 7fe70ba50640 1 -- 172.21.15.33:0/373157380 wait complete. 2024-09-16T00:34:28.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.549+0000 7fe70ba50640 1 Processor -- start 2024-09-16T00:34:28.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.549+0000 7fe70ba50640 1 -- start start 2024-09-16T00:34:28.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe70ba50640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe704102d90 0x7fe7041009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:28.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe70ba50640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe704103740 0x7fe704100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:28.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe70ba50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe704108290 0x7fe704101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:28.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe70ba50640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe704112090 con 0x7fe704102d90 2024-09-16T00:34:28.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe70ba50640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe704111f10 con 0x7fe704103740 2024-09-16T00:34:28.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe70ba50640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe704112210 con 0x7fe704108290 2024-09-16T00:34:28.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe7097ec640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe704102d90 0x7fe7041009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:28.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe7097ec640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe704102d90 0x7fe7041009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57804/0 (socket says 172.21.15.33:57804) 2024-09-16T00:34:28.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe7097ec640 1 -- 172.21.15.33:0/3888497006 learned_addr learned my addr 172.21.15.33:0/3888497006 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:28.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe708feb640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe704103740 0x7fe704100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:28.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe7097ec640 1 -- 172.21.15.33:0/3888497006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe704108290 msgr2=0x7fe704101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:28.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe7097ec640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe704108290 0x7fe704101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe7097ec640 1 -- 172.21.15.33:0/3888497006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe704103740 msgr2=0x7fe704100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe7097ec640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe704103740 0x7fe704100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe7097ec640 1 -- 172.21.15.33:0/3888497006 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7041a6a50 con 0x7fe704102d90 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.550+0000 7fe7097ec640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe704102d90 0x7fe7041009f0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fe700009230 tx=0x7fe700004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.551+0000 7fe6fa7fc640 1 -- 172.21.15.33:0/3888497006 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70003f030 con 0x7fe704102d90 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.551+0000 7fe70ba50640 1 -- 172.21.15.33:0/3888497006 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7041a6ce0 con 0x7fe704102d90 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.551+0000 7fe6fa7fc640 1 -- 172.21.15.33:0/3888497006 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe700003ad0 con 0x7fe704102d90 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.551+0000 7fe70ba50640 1 -- 172.21.15.33:0/3888497006 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7041a71c0 con 0x7fe704102d90 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.551+0000 7fe6fa7fc640 1 -- 172.21.15.33:0/3888497006 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe700049060 con 0x7fe704102d90 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.552+0000 7fe70ba50640 1 -- 172.21.15.33:0/3888497006 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe70406a7f0 con 0x7fe704102d90 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.553+0000 7fe6fa7fc640 1 -- 172.21.15.33:0/3888497006 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe700042030 con 0x7fe704102d90 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.554+0000 7fe6fa7fc640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6f00a3240 0x7fe6f00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:28.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.554+0000 7fe6fa7fc640 1 -- 172.21.15.33:0/3888497006 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe7000f0840 con 0x7fe704102d90 2024-09-16T00:34:28.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.554+0000 7fe708feb640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6f00a3240 0x7fe6f00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:28.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.555+0000 7fe708feb640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6f00a3240 0x7fe6f00a5700 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe704102480 tx=0x7fe6f4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:28.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.558+0000 7fe6fa7fc640 1 -- 172.21.15.33:0/3888497006 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe7000bd080 con 0x7fe704102d90 2024-09-16T00:34:28.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi060 to 177.0M 2024-09-16T00:34:28.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi060 to 177.0M 2024-09-16T00:34:28.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: Unable to set osd_memory_target on smithi060 to 185699225: error parsing value: Value '185699225' is below minimum 939524096 2024-09-16T00:34:28.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.733 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:28.734 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:34:28.734 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:34:28.734 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:34:28.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.815+0000 7fe70ba50640 1 -- 172.21.15.33:0/3888497006 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe70410a2a0 con 0x7fe704102d90 2024-09-16T00:34:28.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.818+0000 7fe6fa7fc640 1 -- 172.21.15.33:0/3888497006 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7fe7000bc980 con 0x7fe704102d90 2024-09-16T00:34:28.819 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:34:28.819 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:34:28.820 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:34:28.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.823+0000 7fe70ba50640 1 -- 172.21.15.33:0/3888497006 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6f00a3240 msgr2=0x7fe6f00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:28.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.823+0000 7fe70ba50640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6f00a3240 0x7fe6f00a5700 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe704102480 tx=0x7fe6f4005f90 comp rx=0 tx=0).stop 2024-09-16T00:34:28.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.823+0000 7fe70ba50640 1 -- 172.21.15.33:0/3888497006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe704102d90 msgr2=0x7fe7041009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.823+0000 7fe70ba50640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe704102d90 0x7fe7041009f0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fe700009230 tx=0x7fe700004270 comp rx=0 tx=0).stop 2024-09-16T00:34:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.825+0000 7fe70ba50640 1 -- 172.21.15.33:0/3888497006 shutdown_connections 2024-09-16T00:34:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.825+0000 7fe70ba50640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe6f00a3240 0x7fe6f00a5700 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.825+0000 7fe70ba50640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe704108290 0x7fe704101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.825+0000 7fe70ba50640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe704103740 0x7fe704100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.825+0000 7fe70ba50640 1 --2- 172.21.15.33:0/3888497006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe704102d90 0x7fe7041009f0 secure :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fe700009230 tx=0x7fe700004270 comp rx=0 tx=0).stop 2024-09-16T00:34:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.825+0000 7fe70ba50640 1 -- 172.21.15.33:0/3888497006 >> 172.21.15.33:0/3888497006 conn(0x7fe7040fc7d0 msgr2=0x7fe704105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.825+0000 7fe70ba50640 1 -- 172.21.15.33:0/3888497006 shutdown_connections 2024-09-16T00:34:28.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:28.825+0000 7fe70ba50640 1 -- 172.21.15.33:0/3888497006 wait complete. 2024-09-16T00:34:29.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 82603 21.96 MB/sec execute 95 sec latency 129.956 ms 2024-09-16T00:34:29.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: Unable to set osd_memory_target on smithi060 to 185699225: error parsing value: Value '185699225' is below minimum 939524096 2024-09-16T00:34:29.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T00:34:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:34:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:34:29.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:34:29.573 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:29 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi083 to 176.0M 2024-09-16T00:34:29.573 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:29 smithi033 ceph-mon[31243]: Adjusting osd_memory_target on smithi033 to 177.0M 2024-09-16T00:34:29.573 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:29 smithi033 ceph-mon[31243]: Unable to set osd_memory_target on smithi083 to 184589619: error parsing value: Value '184589619' is below minimum 939524096 2024-09-16T00:34:29.574 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:29 smithi033 ceph-mon[31243]: Unable to set osd_memory_target on smithi033 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-16T00:34:29.574 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:29 smithi033 ceph-mon[31243]: pgmap v870: 129 pgs: 129 active+clean; 627 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 10 MiB/s wr, 450 op/s 2024-09-16T00:34:29.574 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:29 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3888497006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:34:29.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:29 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi083 to 176.0M 2024-09-16T00:34:29.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:29 smithi083 ceph-mon[35242]: Adjusting osd_memory_target on smithi033 to 177.0M 2024-09-16T00:34:29.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:29 smithi083 ceph-mon[35242]: Unable to set osd_memory_target on smithi083 to 184589619: error parsing value: Value '184589619' is below minimum 939524096 2024-09-16T00:34:29.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:29 smithi083 ceph-mon[35242]: Unable to set osd_memory_target on smithi033 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-16T00:34:29.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:29 smithi083 ceph-mon[35242]: pgmap v870: 129 pgs: 129 active+clean; 627 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 10 MiB/s wr, 450 op/s 2024-09-16T00:34:29.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:29 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3888497006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:34:29.692 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 142.95 --await-for 45.0' 2024-09-16T00:34:29.692 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 142.95 --await-for 45.0 2024-09-16T00:34:29.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:29 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi083 to 176.0M 2024-09-16T00:34:29.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:29 smithi060 ceph-mon[35185]: Adjusting osd_memory_target on smithi033 to 177.0M 2024-09-16T00:34:29.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:29 smithi060 ceph-mon[35185]: Unable to set osd_memory_target on smithi083 to 184589619: error parsing value: Value '184589619' is below minimum 939524096 2024-09-16T00:34:29.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:29 smithi060 ceph-mon[35185]: Unable to set osd_memory_target on smithi033 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-16T00:34:29.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:29 smithi060 ceph-mon[35185]: pgmap v870: 129 pgs: 129 active+clean; 627 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 10 MiB/s wr, 450 op/s 2024-09-16T00:34:29.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:29 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3888497006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:34:30.066 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:30.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83151 22.03 MB/sec execute 96 sec latency 109.791 ms 2024-09-16T00:34:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.583+0000 7fd0f5492640 1 -- 172.21.15.33:0/2842646097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102570 msgr2=0x7fd0f01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:30.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.583+0000 7fd0f5492640 1 --2- 172.21.15.33:0/2842646097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102570 0x7fd0f01029b0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4009900 tx=0x7fd0e402a480 comp rx=0 tx=0).stop 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.584+0000 7fd0f5492640 1 -- 172.21.15.33:0/2842646097 shutdown_connections 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.584+0000 7fd0f5492640 1 --2- 172.21.15.33:0/2842646097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0102ef0 0x7fd0f010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.584+0000 7fd0f5492640 1 --2- 172.21.15.33:0/2842646097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102570 0x7fd0f01029b0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.584+0000 7fd0f5492640 1 --2- 172.21.15.33:0/2842646097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f0100e20 0x7fd0f0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.584+0000 7fd0f5492640 1 -- 172.21.15.33:0/2842646097 >> 172.21.15.33:0/2842646097 conn(0x7fd0f00fc7d0 msgr2=0x7fd0f00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.584+0000 7fd0f5492640 1 -- 172.21.15.33:0/2842646097 shutdown_connections 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.584+0000 7fd0f5492640 1 -- 172.21.15.33:0/2842646097 wait complete. 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.584+0000 7fd0f5492640 1 Processor -- start 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0f5492640 1 -- start start 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0f5492640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0100e20 0x7fd0f00fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0f5492640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102570 0x7fd0f00fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0f5492640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f0102ef0 0x7fd0f00fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0f5492640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f0110f80 con 0x7fd0f0102570 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0f5492640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f0110e00 con 0x7fd0f0102ef0 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0f5492640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f0111100 con 0x7fd0f0100e20 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0ee7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102570 0x7fd0f00fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0eeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0100e20 0x7fd0f00fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0ef7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f0102ef0 0x7fd0f00fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0ee7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102570 0x7fd0f00fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33394/0 (socket says 172.21.15.33:33394) 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0eeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0100e20 0x7fd0f00fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:33080/0 (socket says 172.21.15.33:33080) 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0ee7fc640 1 -- 172.21.15.33:0/1527940727 learned_addr learned my addr 172.21.15.33:0/1527940727 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0ef7fe640 1 -- 172.21.15.33:0/1527940727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0100e20 msgr2=0x7fd0f00fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0ef7fe640 1 --2- 172.21.15.33:0/1527940727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0100e20 0x7fd0f00fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0ef7fe640 1 -- 172.21.15.33:0/1527940727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102570 msgr2=0x7fd0f00fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0ef7fe640 1 --2- 172.21.15.33:0/1527940727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102570 0x7fd0f00fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.585+0000 7fd0ef7fe640 1 -- 172.21.15.33:0/1527940727 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f01a69f0 con 0x7fd0f0102ef0 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.586+0000 7fd0ef7fe640 1 --2- 172.21.15.33:0/1527940727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f0102ef0 0x7fd0f00fd700 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8009fd0 tx=0x7fd0e800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.586+0000 7fd0d7fff640 1 -- 172.21.15.33:0/1527940727 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e800ce20 con 0x7fd0f0102ef0 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.586+0000 7fd0f5492640 1 -- 172.21.15.33:0/1527940727 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f01a6ce0 con 0x7fd0f0102ef0 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.586+0000 7fd0d7fff640 1 -- 172.21.15.33:0/1527940727 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e8005020 con 0x7fd0f0102ef0 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.586+0000 7fd0d7fff640 1 -- 172.21.15.33:0/1527940727 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8010720 con 0x7fd0f0102ef0 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.586+0000 7fd0f5492640 1 -- 172.21.15.33:0/1527940727 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f01a7220 con 0x7fd0f0102ef0 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.589+0000 7fd0d7fff640 1 -- 172.21.15.33:0/1527940727 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd0e8005540 con 0x7fd0f0102ef0 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.589+0000 7fd0f786b740 1 Processor -- start 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.589+0000 7fd0f786b740 1 -- start start 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.589+0000 7fd0f786b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f0a850 0x5568f0f0ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0f786b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f154f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0f786b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f15a30 0x5568f0f17e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0f786b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5568f0e6f160 con 0x5568f0f0a850 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0f786b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5568f0e16720 con 0x5568f0f0b170 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0f786b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5568f0e6b120 con 0x5568f0f15a30 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0eeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f154f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0eeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f154f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:36116/0 (socket says 172.21.15.33:36116) 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0eeffd640 1 -- 172.21.15.33:0/2069740241 learned_addr learned my addr 172.21.15.33:0/2069740241 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0ee7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f0a850 0x5568f0f0ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0d7fff640 1 --2- 172.21.15.33:0/1527940727 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd0d80a3240 0x7fd0d80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0ee7fc640 1 --2- 172.21.15.33:0/2069740241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f15a30 0x5568f0f17e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0ef7fe640 1 --2- 172.21.15.33:0/1527940727 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd0d80a3240 0x7fd0d80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0eeffd640 1 -- 172.21.15.33:0/2069740241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f15a30 msgr2=0x5568f0f17e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0eeffd640 1 --2- 172.21.15.33:0/2069740241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f15a30 0x5568f0f17e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0eeffd640 1 -- 172.21.15.33:0/2069740241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f0a850 msgr2=0x5568f0f0ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:30.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0eeffd640 1 --2- 172.21.15.33:0/2069740241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f0a850 0x5568f0f0ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0eeffd640 1 -- 172.21.15.33:0/2069740241 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568f0be0ba0 con 0x5568f0f0b170 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0d7fff640 1 -- 172.21.15.33:0/1527940727 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd0e80d0380 con 0x7fd0f0102ef0 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0ee7fc640 1 --2- 172.21.15.33:0/2069740241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f15a30 0x5568f0f17e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.590+0000 7fd0eeffd640 1 --2- 172.21.15.33:0/2069740241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f154f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0009cf0 tx=0x7fd0e0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=247ebe5ee80e0b8c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.591+0000 7fd0ef7fe640 1 --2- 172.21.15.33:0/1527940727 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd0d80a3240 0x7fd0d80a5700 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8098980 tx=0x7fd0e8066f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.591+0000 7fd0d57fa640 1 -- 172.21.15.33:0/2069740241 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0004230 con 0x5568f0f0b170 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.591+0000 7fd0d57fa640 1 -- 172.21.15.33:0/2069740241 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e00043d0 con 0x5568f0f0b170 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.591+0000 7fd0d57fa640 1 -- 172.21.15.33:0/2069740241 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0019830 con 0x5568f0f0b170 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.592+0000 7fd0f786b740 1 -- 172.21.15.33:0/2069740241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 msgr2=0x5568f0f154f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.592+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2069740241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f154f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0009cf0 tx=0x7fd0e0011810 comp rx=0 tx=0).stop 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.592+0000 7fd0f786b740 1 -- 172.21.15.33:0/2069740241 shutdown_connections 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.592+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2069740241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f15a30 0x5568f0f17e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.592+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2069740241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f154f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.592+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2069740241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f0a850 0x5568f0f0ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.592+0000 7fd0f786b740 1 -- 172.21.15.33:0/2069740241 >> 172.21.15.33:0/2069740241 conn(0x5568f0f06de0 msgr2=0x5568f0f071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.592+0000 7fd0f786b740 1 -- 172.21.15.33:0/2069740241 shutdown_connections 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.592+0000 7fd0f786b740 1 -- 172.21.15.33:0/2069740241 wait complete. 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.592+0000 7fd0f786b740 1 -- start start 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.592+0000 7fd0f786b740 1 Processor -- start 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0f786b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0a850 0x5568f0f12850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0f786b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f12d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0f786b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f15a30 0x5568f0f132d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0f786b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5568f0f18600 con 0x5568f0f15a30 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0f786b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5568f0df7900 con 0x5568f0f0b170 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0f786b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5568f0f18780 con 0x5568f0f0a850 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0f786b740 10 client.0 _ll_register_callbacks cb 0x5568f0df5780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:30.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0ee7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f15a30 0x5568f0f132d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0eeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f12d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0ee7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f15a30 0x5568f0f132d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33402/0 (socket says 172.21.15.33:33402) 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0ee7fc640 1 -- 172.21.15.33:0/2372491619 learned_addr learned my addr 172.21.15.33:0/2372491619 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0ee7fc640 1 --2- 172.21.15.33:0/2372491619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0a850 0x5568f0f12850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0eeffd640 1 -- 172.21.15.33:0/2372491619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0a850 msgr2=0x5568f0f12850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0eeffd640 1 --2- 172.21.15.33:0/2372491619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0a850 0x5568f0f12850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0eeffd640 1 -- 172.21.15.33:0/2372491619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f15a30 msgr2=0x5568f0f132d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0eeffd640 1 --2- 172.21.15.33:0/2372491619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f15a30 0x5568f0f132d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0eeffd640 1 -- 172.21.15.33:0/2372491619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568f0e37c50 con 0x5568f0f0b170 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0eeffd640 1 --2- 172.21.15.33:0/2372491619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f12d90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0000c00 tx=0x7fd0e00038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=215cbb9198ca5214 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.593+0000 7fd0d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.594+0000 7fd0d2ff5640 1 -- 172.21.15.33:0/2372491619 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0009910 con 0x5568f0f0b170 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.594+0000 7fd0d2ff5640 1 -- 172.21.15.33:0/2372491619 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd0e0005260 con 0x5568f0f0b170 2024-09-16T00:34:30.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.594+0000 7fd0f786b740 1 -- 172.21.15.33:0/2372491619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0e00043d0 con 0x5568f0f0b170 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.594+0000 7fd0d2ff5640 1 -- 172.21.15.33:0/2372491619 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd0e0005c50 con 0x5568f0f0b170 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.594+0000 7fd0f786b740 10 client.27944 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.594+0000 7fd0f786b740 1 -- 172.21.15.33:0/2372491619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5568f0f24150 con 0x5568f0f0b170 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.595+0000 7fd0d2ff5640 1 -- 172.21.15.33:0/2372491619 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e00043d0 con 0x5568f0f0b170 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.595+0000 7fd0d2ff5640 1 -- 172.21.15.33:0/2372491619 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fd0e0019590 con 0x5568f0f0b170 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.595+0000 7fd0f786b740 10 client.27944 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.595+0000 7fd0f786b740 10 client.27944 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.595+0000 7fd0f786b740 20 client.27944 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.595+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2372491619 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568f0f255b0 0x5568f0f279a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.595+0000 7fd0f786b740 4 client.27944 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.595+0000 7fd0f786b740 1 -- 172.21.15.33:0/2372491619 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568f0e7c890 con 0x5568f0f255b0 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.596+0000 7fd0ee7fc640 1 --2- 172.21.15.33:0/2372491619 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568f0f255b0 0x5568f0f279a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.596+0000 7fd0ee7fc640 1 --2- 172.21.15.33:0/2372491619 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568f0f255b0 0x5568f0f279a0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e39e660bda4641 server_cookie=fb0fb6519cdabf1 in_seq=0 out_seq=0 2024-09-16T00:34:30.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.596+0000 7fd0d2ff5640 10 client.27944 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:30.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.597+0000 7fd0d2ff5640 1 -- 172.21.15.33:0/2372491619 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5568f0e7c890 con 0x5568f0f255b0 2024-09-16T00:34:30.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.597+0000 7fd0d2ff5640 10 client.27944 handle_command_reply: tid=0 2024-09-16T00:34:30.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.597+0000 7fd0f786b740 1 client.27944 shutdown 2024-09-16T00:34:30.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.597+0000 7fd0f786b740 1 -- 172.21.15.33:0/2372491619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 msgr2=0x5568f0f12d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:30.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.597+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2372491619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f12d90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0000c00 tx=0x7fd0e00038a0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.597+0000 7fd0f786b740 1 -- 172.21.15.33:0/2372491619 shutdown_connections 2024-09-16T00:34:30.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.597+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2372491619 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568f0f255b0 0x5568f0f279a0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.598+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2372491619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f15a30 0x5568f0f132d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.598+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2372491619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f12d90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.598+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2372491619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0a850 0x5568f0f12850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.598+0000 7fd0f786b740 1 -- 172.21.15.33:0/2372491619 >> 172.21.15.33:0/2372491619 conn(0x5568f0f06de0 msgr2=0x5568f0f0cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:30.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.598+0000 7fd0d2ff5640 0 client.27944 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:30.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.598+0000 7fd0f786b740 1 -- 172.21.15.33:0/2372491619 shutdown_connections 2024-09-16T00:34:30.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.598+0000 7fd0f786b740 1 -- 172.21.15.33:0/2372491619 wait complete. 2024-09-16T00:34:30.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.598+0000 7fd0f786b740 20 client.27944 trim_cache size 0 max 16384 2024-09-16T00:34:30.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.616+0000 7fd0f786b740 1 Processor -- start 2024-09-16T00:34:30.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.616+0000 7fd0f786b740 1 -- start start 2024-09-16T00:34:30.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.616+0000 7fd0f786b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f0a850 0x5568f0f14720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.616+0000 7fd0f786b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0b170 0x5568f0f2d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.616+0000 7fd0f786b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f15a30 0x5568f0f2d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.616+0000 7fd0f786b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01b3c50 con 0x5568f0f0a850 2024-09-16T00:34:30.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.616+0000 7fd0f786b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01b1520 con 0x5568f0f15a30 2024-09-16T00:34:30.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.616+0000 7fd0f786b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01af370 con 0x5568f0f0b170 2024-09-16T00:34:30.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0eeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0b170 0x5568f0f2d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0eeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0b170 0x5568f0f2d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:33112/0 (socket says 172.21.15.33:33112) 2024-09-16T00:34:30.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0eeffd640 1 -- 172.21.15.33:0/1911848351 learned_addr learned my addr 172.21.15.33:0/1911848351 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:30.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0ee7fc640 1 --2- 172.21.15.33:0/1911848351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f0a850 0x5568f0f14720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0ee7fc640 1 --2- 172.21.15.33:0/1911848351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f15a30 0x5568f0f2d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0eeffd640 1 -- 172.21.15.33:0/1911848351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f15a30 msgr2=0x5568f0f2d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:30.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0eeffd640 1 --2- 172.21.15.33:0/1911848351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f15a30 0x5568f0f2d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0eeffd640 1 -- 172.21.15.33:0/1911848351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f0a850 msgr2=0x5568f0f14720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:30.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0eeffd640 1 --2- 172.21.15.33:0/1911848351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f0a850 0x5568f0f14720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0eeffd640 1 -- 172.21.15.33:0/1911848351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568f0f0afa0 con 0x5568f0f0b170 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0ee7fc640 1 --2- 172.21.15.33:0/1911848351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f15a30 0x5568f0f2d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0eeffd640 1 --2- 172.21.15.33:0/1911848351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0b170 0x5568f0f2d180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0000c00 tx=0x7fd0e0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baea5dd575c62d22 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.617+0000 7fd0d4ff9640 1 -- 172.21.15.33:0/1911848351 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0009b70 con 0x5568f0f0b170 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.618+0000 7fd0d4ff9640 1 -- 172.21.15.33:0/1911848351 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e0009910 con 0x5568f0f0b170 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.618+0000 7fd0d4ff9640 1 -- 172.21.15.33:0/1911848351 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0003bb0 con 0x5568f0f0b170 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.618+0000 7fd0f786b740 1 -- 172.21.15.33:0/1911848351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0b170 msgr2=0x5568f0f2d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.618+0000 7fd0f786b740 1 --2- 172.21.15.33:0/1911848351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0b170 0x5568f0f2d180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0000c00 tx=0x7fd0e0011810 comp rx=0 tx=0).stop 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.618+0000 7fd0f786b740 1 -- 172.21.15.33:0/1911848351 shutdown_connections 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.618+0000 7fd0f786b740 1 --2- 172.21.15.33:0/1911848351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f15a30 0x5568f0f2d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.618+0000 7fd0f786b740 1 --2- 172.21.15.33:0/1911848351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0b170 0x5568f0f2d180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.618+0000 7fd0f786b740 1 --2- 172.21.15.33:0/1911848351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f0a850 0x5568f0f14720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.618+0000 7fd0f786b740 1 -- 172.21.15.33:0/1911848351 >> 172.21.15.33:0/1911848351 conn(0x5568f0f06de0 msgr2=0x5568f0f05ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:30.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.618+0000 7fd0f786b740 1 -- 172.21.15.33:0/1911848351 shutdown_connections 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.618+0000 7fd0f786b740 1 -- 172.21.15.33:0/1911848351 wait complete. 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0f786b740 1 -- start start 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0f786b740 1 Processor -- start 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0f786b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0a850 0x5568f0f16c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0f786b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f171a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0f786b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f15a30 0x5568f0f176e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0f786b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01af9e0 con 0x5568f0f15a30 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0f786b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01b0e50 con 0x5568f0f0b170 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0f786b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01b00b0 con 0x5568f0f0a850 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0f786b740 10 client.0 _ll_register_callbacks cb 0x7fd0e80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0ee7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0a850 0x5568f0f16c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0ee7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0a850 0x5568f0f16c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:33126/0 (socket says 172.21.15.33:33126) 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0ee7fc640 1 -- 172.21.15.33:0/2005190166 learned_addr learned my addr 172.21.15.33:0/2005190166 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:30.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0ee7fc640 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f15a30 0x5568f0f176e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.619+0000 7fd0eeffd640 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f171a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0ee7fc640 1 -- 172.21.15.33:0/2005190166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 msgr2=0x5568f0f171a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0ee7fc640 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f171a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0ee7fc640 1 -- 172.21.15.33:0/2005190166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f15a30 msgr2=0x5568f0f176e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0ee7fc640 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f15a30 0x5568f0f176e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0ee7fc640 1 -- 172.21.15.33:0/2005190166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568f0f1c010 con 0x5568f0f0a850 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0ee7fc640 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0a850 0x5568f0f16c60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4006010 tx=0x7fd0e4005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf00323d4fc6cf63 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0d2ff5640 1 -- 172.21.15.33:0/2005190166 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4051070 con 0x5568f0f0a850 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0d2ff5640 1 -- 172.21.15.33:0/2005190166 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd0e404ce80 con 0x5568f0f0a850 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0f786b740 1 -- 172.21.15.33:0/2005190166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0e0009910 con 0x5568f0f0a850 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0d2ff5640 1 -- 172.21.15.33:0/2005190166 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd0e4035040 con 0x5568f0f0a850 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0f786b740 10 client.27946 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:30.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.620+0000 7fd0f786b740 1 -- 172.21.15.33:0/2005190166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5568f0f904e0 con 0x5568f0f0a850 2024-09-16T00:34:30.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.621+0000 7fd0d2ff5640 1 -- 172.21.15.33:0/2005190166 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e0009910 con 0x5568f0f0a850 2024-09-16T00:34:30.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.621+0000 7fd0d2ff5640 1 -- 172.21.15.33:0/2005190166 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fd0e40047e0 con 0x5568f0f0a850 2024-09-16T00:34:30.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.621+0000 7fd0f786b740 10 client.27946 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:30.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.621+0000 7fd0f786b740 10 client.27946 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:34:30.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.621+0000 7fd0f786b740 20 client.27946 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:30.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.621+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568f0f90c30 0x5568f0f93050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:30.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.621+0000 7fd0f786b740 4 client.27946 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.95, "await_for": 45.0}] 2024-09-16T00:34:30.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.621+0000 7fd0f786b740 1 -- 172.21.15.33:0/2005190166 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.95, "await_for": 45.0}) -- 0x5568f0e76be0 con 0x5568f0f90c30 2024-09-16T00:34:30.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.621+0000 7fd0eeffd640 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568f0f90c30 0x5568f0f93050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:30.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.621+0000 7fd0eeffd640 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568f0f90c30 0x5568f0f93050 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5493037dad865ed server_cookie=87570bc92a339978 in_seq=0 out_seq=0 2024-09-16T00:34:30.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:30.622+0000 7fd0d2ff5640 10 client.27946 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:31.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83498 22.01 MB/sec execute 97 sec latency 409.761 ms 2024-09-16T00:34:31.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0d2ff5640 1 -- 172.21.15.33:0/2005190166 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5568f0e76be0 con 0x5568f0f90c30 2024-09-16T00:34:31.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0d2ff5640 10 client.27946 handle_command_reply: tid=0 2024-09-16T00:34:31.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0f786b740 1 client.27946 shutdown 2024-09-16T00:34:31.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0f786b740 1 -- 172.21.15.33:0/2005190166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0a850 msgr2=0x5568f0f16c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:31.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0a850 0x5568f0f16c60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4006010 tx=0x7fd0e4005a20 comp rx=0 tx=0).stop 2024-09-16T00:34:31.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0f786b740 1 -- 172.21.15.33:0/2005190166 shutdown_connections 2024-09-16T00:34:31.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568f0f90c30 0x5568f0f93050 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:31.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f0f15a30 0x5568f0f176e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:31.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568f0f0b170 0x5568f0f171a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:31.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0f786b740 1 --2- 172.21.15.33:0/2005190166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568f0f0a850 0x5568f0f16c60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:31.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0d2ff5640 0 client.27946 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:31.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0f786b740 1 -- 172.21.15.33:0/2005190166 >> 172.21.15.33:0/2005190166 conn(0x5568f0f06de0 msgr2=0x5568f0f056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:31.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0f786b740 1 -- 172.21.15.33:0/2005190166 shutdown_connections 2024-09-16T00:34:31.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.465+0000 7fd0f786b740 1 -- 172.21.15.33:0/2005190166 wait complete. 2024-09-16T00:34:31.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.466+0000 7fd0f786b740 20 client.27946 trim_cache size 0 max 16384 2024-09-16T00:34:31.468 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:34:31.468 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:34:31.469 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:34:31.469 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 62, 2024-09-16T00:34:31.469 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:34:31.469 INFO:teuthology.orchestra.run.smithi033.stdout: "17f371aa": { 2024-09-16T00:34:31.469 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 62, 2024-09-16T00:34:31.469 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:34:31.469 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:34:31.469 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:34:31.470 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:34:31.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:34:31.470 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:34:31.470 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 142.9, 2024-09-16T00:34:31.470 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:34:31.470 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:34:31.470 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:34:31.470 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:34:31.471 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:34:31.471 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:34:31.471 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:31.471 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:31.471 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:31.471 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:31.471 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:31.471 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:34:31.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.466+0000 7fd0f5492640 1 -- 172.21.15.33:0/1527940727 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd0d80a3240 msgr2=0x7fd0d80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:31.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.466+0000 7fd0f5492640 1 --2- 172.21.15.33:0/1527940727 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd0d80a3240 0x7fd0d80a5700 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8098980 tx=0x7fd0e8066f60 comp rx=0 tx=0).stop 2024-09-16T00:34:31.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.466+0000 7fd0f5492640 1 -- 172.21.15.33:0/1527940727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f0102ef0 msgr2=0x7fd0f00fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:31.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.466+0000 7fd0f5492640 1 --2- 172.21.15.33:0/1527940727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f0102ef0 0x7fd0f00fd700 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8009fd0 tx=0x7fd0e800ef10 comp rx=0 tx=0).stop 2024-09-16T00:34:31.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.466+0000 7fd0f5492640 1 -- 172.21.15.33:0/1527940727 shutdown_connections 2024-09-16T00:34:31.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.466+0000 7fd0f5492640 1 --2- 172.21.15.33:0/1527940727 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd0d80a3240 0x7fd0d80a5700 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:31.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.466+0000 7fd0f5492640 1 --2- 172.21.15.33:0/1527940727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f0102ef0 0x7fd0f00fd700 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:31.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.466+0000 7fd0f5492640 1 --2- 172.21.15.33:0/1527940727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102570 0x7fd0f00fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:31.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.466+0000 7fd0f5492640 1 --2- 172.21.15.33:0/1527940727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0100e20 0x7fd0f00fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:31.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.466+0000 7fd0f5492640 1 -- 172.21.15.33:0/1527940727 >> 172.21.15.33:0/1527940727 conn(0x7fd0f00fc7d0 msgr2=0x7fd0f006a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:31.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.467+0000 7fd0f5492640 1 -- 172.21.15.33:0/1527940727 shutdown_connections 2024-09-16T00:34:31.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:31.467+0000 7fd0f5492640 1 -- 172.21.15.33:0/1527940727 wait complete. 2024-09-16T00:34:31.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:31 smithi083 ceph-mon[35242]: pgmap v871: 129 pgs: 129 active+clean; 605 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 585 op/s 2024-09-16T00:34:31.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:31 smithi060 ceph-mon[35185]: pgmap v871: 129 pgs: 129 active+clean; 605 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 585 op/s 2024-09-16T00:34:31.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:31 smithi033 ceph-mon[31243]: pgmap v871: 129 pgs: 129 active+clean; 605 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 585 op/s 2024-09-16T00:34:32.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 21.79 MB/sec execute 98 sec latency 1409.805 ms 2024-09-16T00:34:32.322 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '17f371aa', quiesce took 4.8 seconds. Will release after: 18.15 2024-09-16T00:34:33.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 21.57 MB/sec execute 99 sec latency 2409.894 ms 2024-09-16T00:34:33.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:33 smithi083 ceph-mon[35242]: pgmap v872: 129 pgs: 129 active+clean; 568 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 14 MiB/s wr, 687 op/s 2024-09-16T00:34:33.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:33 smithi060 ceph-mon[35185]: pgmap v872: 129 pgs: 129 active+clean; 568 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 14 MiB/s wr, 687 op/s 2024-09-16T00:34:33.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:33 smithi033 ceph-mon[31243]: pgmap v872: 129 pgs: 129 active+clean; 568 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 14 MiB/s wr, 687 op/s 2024-09-16T00:34:34.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 21.35 MB/sec execute 100 sec latency 3409.960 ms 2024-09-16T00:34:35.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 21.14 MB/sec execute 101 sec latency 4410.084 ms 2024-09-16T00:34:35.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:35 smithi083 ceph-mon[35242]: pgmap v873: 129 pgs: 129 active+clean; 562 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 14 MiB/s wr, 682 op/s 2024-09-16T00:34:35.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:35 smithi060 ceph-mon[35185]: pgmap v873: 129 pgs: 129 active+clean; 562 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 14 MiB/s wr, 682 op/s 2024-09-16T00:34:35.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:35 smithi033 ceph-mon[31243]: pgmap v873: 129 pgs: 129 active+clean; 562 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 14 MiB/s wr, 682 op/s 2024-09-16T00:34:36.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 20.93 MB/sec execute 102 sec latency 5410.189 ms 2024-09-16T00:34:36.756 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:34:37.029 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:37.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 20.73 MB/sec execute 103 sec latency 6410.278 ms 2024-09-16T00:34:37.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.533+0000 7ff2adff9640 1 -- 172.21.15.33:0/1863839538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2a8106f10 msgr2=0x7ff2a810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:37.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.533+0000 7ff2adff9640 1 --2- 172.21.15.33:0/1863839538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2a8106f10 0x7ff2a810ea90 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009ca0 tx=0x7ff2a002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.533+0000 7ff2adff9640 1 -- 172.21.15.33:0/1863839538 shutdown_connections 2024-09-16T00:34:37.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.533+0000 7ff2adff9640 1 --2- 172.21.15.33:0/1863839538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2a8106f10 0x7ff2a810ea90 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.533+0000 7ff2adff9640 1 --2- 172.21.15.33:0/1863839538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a8102790 0x7ff2a81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.533+0000 7ff2adff9640 1 --2- 172.21.15.33:0/1863839538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2a8101de0 0x7ff2a81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.533+0000 7ff2adff9640 1 -- 172.21.15.33:0/1863839538 >> 172.21.15.33:0/1863839538 conn(0x7ff2a80fd8c0 msgr2=0x7ff2a80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:37.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.533+0000 7ff2adff9640 1 -- 172.21.15.33:0/1863839538 shutdown_connections 2024-09-16T00:34:37.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.533+0000 7ff2adff9640 1 -- 172.21.15.33:0/1863839538 wait complete. 2024-09-16T00:34:37.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.534+0000 7ff2adff9640 1 Processor -- start 2024-09-16T00:34:37.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.534+0000 7ff2adff9640 1 -- start start 2024-09-16T00:34:37.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.534+0000 7ff2adff9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2a8101de0 0x7ff2a8108ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.534+0000 7ff2adff9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a8102790 0x7ff2a8108ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.534+0000 7ff2adff9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2a8106f10 0x7ff2a8109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.534+0000 7ff2adff9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a8111f70 con 0x7ff2a8101de0 2024-09-16T00:34:37.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.534+0000 7ff2adff9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a8111df0 con 0x7ff2a8106f10 2024-09-16T00:34:37.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.534+0000 7ff2adff9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a81120f0 con 0x7ff2a8102790 2024-09-16T00:34:37.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.534+0000 7ff2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2a8101de0 0x7ff2a8108ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.534+0000 7ff2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2a8101de0 0x7ff2a8108ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33422/0 (socket says 172.21.15.33:33422) 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.534+0000 7ff2a77fe640 1 -- 172.21.15.33:0/388893136 learned_addr learned my addr 172.21.15.33:0/388893136 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2a6ffd640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a8102790 0x7ff2a8108ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2a7fff640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2a8106f10 0x7ff2a8109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2a77fe640 1 -- 172.21.15.33:0/388893136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a8102790 msgr2=0x7ff2a8108ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a8102790 0x7ff2a8108ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2a77fe640 1 -- 172.21.15.33:0/388893136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2a8106f10 msgr2=0x7ff2a8109530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2a8106f10 0x7ff2a8109530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2a77fe640 1 -- 172.21.15.33:0/388893136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a81aac20 con 0x7ff2a8101de0 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2a8101de0 0x7ff2a8108ab0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c950 tx=0x7ff29800ce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2a4ff9640 1 -- 172.21.15.33:0/388893136 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff298007c70 con 0x7ff2a8101de0 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2adff9640 1 -- 172.21.15.33:0/388893136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2a81aaf10 con 0x7ff2a8101de0 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2a4ff9640 1 -- 172.21.15.33:0/388893136 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff298007e10 con 0x7ff2a8101de0 2024-09-16T00:34:37.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2a4ff9640 1 -- 172.21.15.33:0/388893136 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff298018960 con 0x7ff2a8101de0 2024-09-16T00:34:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.535+0000 7ff2adff9640 1 -- 172.21.15.33:0/388893136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2a81ab3c0 con 0x7ff2a8101de0 2024-09-16T00:34:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.537+0000 7ff2a4ff9640 1 -- 172.21.15.33:0/388893136 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff29800fbc0 con 0x7ff2a8101de0 2024-09-16T00:34:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.538+0000 7ff2b03d2740 1 Processor -- start 2024-09-16T00:34:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.538+0000 7ff2b03d2740 1 -- start start 2024-09-16T00:34:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.538+0000 7ff2b03d2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da424a0 0x55592da42880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.538+0000 7ff2b03d2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 0x55592da4d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.538+0000 7ff2b03d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da4d680 0x55592da4fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.538+0000 7ff2b03d2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55592d9a82e0 con 0x55592da42dc0 2024-09-16T00:34:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.538+0000 7ff2b03d2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55592d94fce0 con 0x55592da424a0 2024-09-16T00:34:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.538+0000 7ff2b03d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55592d9a42a0 con 0x55592da4d680 2024-09-16T00:34:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.538+0000 7ff2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 0x55592da4d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.538+0000 7ff2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 0x55592da4d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33428/0 (socket says 172.21.15.33:33428) 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.538+0000 7ff2a77fe640 1 -- 172.21.15.33:0/363738812 learned_addr learned my addr 172.21.15.33:0/363738812 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.538+0000 7ff2a7fff640 1 --2- 172.21.15.33:0/363738812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da424a0 0x55592da42880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff2a6ffd640 1 --2- 172.21.15.33:0/363738812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da4d680 0x55592da4fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff2a77fe640 1 -- 172.21.15.33:0/363738812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da4d680 msgr2=0x55592da4fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/363738812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da4d680 0x55592da4fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff2a4ff9640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff2940a3240 0x7ff2940a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff2a77fe640 1 -- 172.21.15.33:0/363738812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da424a0 msgr2=0x55592da42880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/363738812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da424a0 0x55592da42880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff2a77fe640 1 -- 172.21.15.33:0/363738812 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55592d934440 con 0x55592da42dc0 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff2a7fff640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff2940a3240 0x7ff2940a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff2a4ff9640 1 -- 172.21.15.33:0/388893136 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff2980cfe00 con 0x7ff2a8101de0 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/363738812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 0x55592da4d140 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ff29809e4b0 tx=0x7ff2980d3f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca80cac2712098a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:37.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff29dffb640 1 -- 172.21.15.33:0/363738812 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29809bd10 con 0x55592da42dc0 2024-09-16T00:34:37.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff29dffb640 1 -- 172.21.15.33:0/363738812 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2980a00e0 con 0x55592da42dc0 2024-09-16T00:34:37.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.539+0000 7ff29dffb640 1 -- 172.21.15.33:0/363738812 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29809e280 con 0x55592da42dc0 2024-09-16T00:34:37.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.540+0000 7ff2b03d2740 1 -- 172.21.15.33:0/363738812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 msgr2=0x55592da4d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:37.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.540+0000 7ff2a7fff640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff2940a3240 0x7ff2940a5700 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0005e30 tx=0x7ff2a00095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:37.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.540+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/363738812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 0x55592da4d140 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ff29809e4b0 tx=0x7ff2980d3f40 comp rx=0 tx=0).stop 2024-09-16T00:34:37.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.540+0000 7ff2b03d2740 1 -- 172.21.15.33:0/363738812 shutdown_connections 2024-09-16T00:34:37.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.540+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/363738812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da4d680 0x55592da4fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.540+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/363738812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 0x55592da4d140 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.540+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/363738812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da424a0 0x55592da42880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.540+0000 7ff2b03d2740 1 -- 172.21.15.33:0/363738812 >> 172.21.15.33:0/363738812 conn(0x55592da3ea30 msgr2=0x55592da3ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:37.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.540+0000 7ff2b03d2740 1 -- 172.21.15.33:0/363738812 shutdown_connections 2024-09-16T00:34:37.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.540+0000 7ff2b03d2740 1 -- 172.21.15.33:0/363738812 wait complete. 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.540+0000 7ff2b03d2740 1 -- start start 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2b03d2740 1 Processor -- start 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2b03d2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 0x55592da4cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2b03d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da42dc0 0x55592da4a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2b03d2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da4d680 0x55592da4a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2b03d2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55592da50250 con 0x55592da424a0 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2b03d2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55592d963050 con 0x55592da4d680 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2b03d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55592da503d0 con 0x55592da42dc0 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2b03d2740 10 client.0 _ll_register_callbacks cb 0x55592d9a3160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 0x55592da4cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 0x55592da4cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33432/0 (socket says 172.21.15.33:33432) 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2a77fe640 1 -- 172.21.15.33:0/177919671 learned_addr learned my addr 172.21.15.33:0/177919671 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:37.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2a6ffd640 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da42dc0 0x55592da4a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2a7fff640 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da4d680 0x55592da4a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2a77fe640 1 -- 172.21.15.33:0/177919671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da42dc0 msgr2=0x55592da4a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da42dc0 0x55592da4a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2a77fe640 1 -- 172.21.15.33:0/177919671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da4d680 msgr2=0x55592da4a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da4d680 0x55592da4a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.541+0000 7ff2a77fe640 1 -- 172.21.15.33:0/177919671 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55592d7193c0 con 0x55592da424a0 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 0x55592da4cfd0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7ff2980a4680 tx=0x7ff29800fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b00a72823bc75bf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff286ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff286ffd640 1 -- 172.21.15.33:0/177919671 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29809b8a0 con 0x55592da424a0 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff286ffd640 1 -- 172.21.15.33:0/177919671 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff2980d3a40 con 0x55592da424a0 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff2b03d2740 1 -- 172.21.15.33:0/177919671 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2980a00e0 con 0x55592da424a0 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff286ffd640 1 -- 172.21.15.33:0/177919671 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff29809ba80 con 0x55592da424a0 2024-09-16T00:34:37.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff2b03d2740 10 client.18663 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff2b03d2740 1 -- 172.21.15.33:0/177919671 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55592da5bc50 con 0x55592da424a0 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff286ffd640 1 -- 172.21.15.33:0/177919671 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2980a00e0 con 0x55592da424a0 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff286ffd640 1 -- 172.21.15.33:0/177919671 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ff298097c90 con 0x55592da424a0 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff2b03d2740 10 client.18663 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff2b03d2740 10 client.18663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff2b03d2740 20 client.18663 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55592da5cfa0 0x55592da5f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff2b03d2740 4 client.18663 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.542+0000 7ff2b03d2740 1 -- 172.21.15.33:0/177919671 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55592d9b5850 con 0x55592da5cfa0 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.543+0000 7ff2a6ffd640 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55592da5cfa0 0x55592da5f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.543+0000 7ff2a6ffd640 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55592da5cfa0 0x55592da5f390 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=599c6a48f19b769e server_cookie=7a2964647e58381a in_seq=0 out_seq=0 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.543+0000 7ff286ffd640 10 client.18663 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:37.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff286ffd640 1 -- 172.21.15.33:0/177919671 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55592d9b5850 con 0x55592da5cfa0 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff286ffd640 10 client.18663 handle_command_reply: tid=0 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff2b03d2740 1 client.18663 shutdown 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff2b03d2740 1 -- 172.21.15.33:0/177919671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 msgr2=0x55592da4cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 0x55592da4cfd0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7ff2980a4680 tx=0x7ff29800fb30 comp rx=0 tx=0).stop 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff2b03d2740 1 -- 172.21.15.33:0/177919671 shutdown_connections 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55592da5cfa0 0x55592da5f390 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da4d680 0x55592da4a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da42dc0 0x55592da4a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff286ffd640 0 client.18663 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/177919671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 0x55592da4cfd0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff2b03d2740 1 -- 172.21.15.33:0/177919671 >> 172.21.15.33:0/177919671 conn(0x55592da3ea30 msgr2=0x55592da44760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff2b03d2740 1 -- 172.21.15.33:0/177919671 shutdown_connections 2024-09-16T00:34:37.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff2b03d2740 1 -- 172.21.15.33:0/177919671 wait complete. 2024-09-16T00:34:37.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.544+0000 7ff2b03d2740 20 client.18663 trim_cache size 0 max 16384 2024-09-16T00:34:37.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2b03d2740 1 Processor -- start 2024-09-16T00:34:37.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2b03d2740 1 -- start start 2024-09-16T00:34:37.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2b03d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da424a0 0x55592da48b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2b03d2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 0x55592da623c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2b03d2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da4d680 0x55592da62900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2b03d2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a81b7fb0 con 0x55592da42dc0 2024-09-16T00:34:37.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2b03d2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a81b5880 con 0x55592da4d680 2024-09-16T00:34:37.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2b03d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a81b36d0 con 0x55592da424a0 2024-09-16T00:34:37.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2a6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 0x55592da623c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2a6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 0x55592da623c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33444/0 (socket says 172.21.15.33:33444) 2024-09-16T00:34:37.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2a6ffd640 1 -- 172.21.15.33:0/3859366092 learned_addr learned my addr 172.21.15.33:0/3859366092 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:37.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2a7fff640 1 --2- 172.21.15.33:0/3859366092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da4d680 0x55592da62900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/3859366092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da424a0 0x55592da48b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2a6ffd640 1 -- 172.21.15.33:0/3859366092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da424a0 msgr2=0x55592da48b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:37.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2a6ffd640 1 --2- 172.21.15.33:0/3859366092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da424a0 0x55592da48b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2a6ffd640 1 -- 172.21.15.33:0/3859366092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da4d680 msgr2=0x55592da62900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:37.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2a6ffd640 1 --2- 172.21.15.33:0/3859366092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da4d680 0x55592da62900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.562+0000 7ff2a6ffd640 1 -- 172.21.15.33:0/3859366092 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55592da49230 con 0x55592da42dc0 2024-09-16T00:34:37.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.563+0000 7ff2a6ffd640 1 --2- 172.21.15.33:0/3859366092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 0x55592da623c0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x55592da3fdb0 tx=0x7ff29000bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a17ec5688db5364 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:37.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.563+0000 7ff29d7fa640 1 -- 172.21.15.33:0/3859366092 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff290022070 con 0x55592da42dc0 2024-09-16T00:34:37.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.563+0000 7ff29d7fa640 1 -- 172.21.15.33:0/3859366092 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff290003d20 con 0x55592da42dc0 2024-09-16T00:34:37.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.563+0000 7ff29d7fa640 1 -- 172.21.15.33:0/3859366092 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff290015070 con 0x55592da42dc0 2024-09-16T00:34:37.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.564+0000 7ff2b03d2740 1 -- 172.21.15.33:0/3859366092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 msgr2=0x55592da623c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:37.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.564+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/3859366092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 0x55592da623c0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x55592da3fdb0 tx=0x7ff29000bdf0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.564+0000 7ff2b03d2740 1 -- 172.21.15.33:0/3859366092 shutdown_connections 2024-09-16T00:34:37.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.564+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/3859366092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da4d680 0x55592da62900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.564+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/3859366092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da42dc0 0x55592da623c0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.564+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/3859366092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da424a0 0x55592da48b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.564+0000 7ff2b03d2740 1 -- 172.21.15.33:0/3859366092 >> 172.21.15.33:0/3859366092 conn(0x55592da3ea30 msgr2=0x55592da3db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:37.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.564+0000 7ff2b03d2740 1 -- 172.21.15.33:0/3859366092 shutdown_connections 2024-09-16T00:34:37.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.564+0000 7ff2b03d2740 1 -- 172.21.15.33:0/3859366092 wait complete. 2024-09-16T00:34:37.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.564+0000 7ff2b03d2740 1 -- start start 2024-09-16T00:34:37.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.564+0000 7ff2b03d2740 1 Processor -- start 2024-09-16T00:34:37.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2b03d2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 0x55592da62220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2b03d2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da42dc0 0x55592da62760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2b03d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da4d680 0x55592dabee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2b03d2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a81b3d40 con 0x55592da424a0 2024-09-16T00:34:37.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2b03d2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a81b51b0 con 0x55592da42dc0 2024-09-16T00:34:37.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2b03d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a81b4410 con 0x55592da4d680 2024-09-16T00:34:37.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2b03d2740 10 client.0 _ll_register_callbacks cb 0x7ff2a0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:37.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 0x55592da62220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 0x55592da62220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33452/0 (socket says 172.21.15.33:33452) 2024-09-16T00:34:37.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2a77fe640 1 -- 172.21.15.33:0/172156067 learned_addr learned my addr 172.21.15.33:0/172156067 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:37.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2a6ffd640 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da42dc0 0x55592da62760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2a7fff640 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da4d680 0x55592dabee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2a77fe640 1 -- 172.21.15.33:0/172156067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da4d680 msgr2=0x55592dabee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:37.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da4d680 0x55592dabee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2a77fe640 1 -- 172.21.15.33:0/172156067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da42dc0 msgr2=0x55592da62760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:37.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da42dc0 0x55592da62760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2a77fe640 1 -- 172.21.15.33:0/172156067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55592da42bf0 con 0x55592da424a0 2024-09-16T00:34:37.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2a6ffd640 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da42dc0 0x55592da62760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:37.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff2a77fe640 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 0x55592da62220 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff2980b0370 tx=0x7ff29809c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d683e2e1826057 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:37.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff286ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.565+0000 7ff286ffd640 1 -- 172.21.15.33:0/172156067 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29809b8a0 con 0x55592da424a0 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff286ffd640 1 -- 172.21.15.33:0/172156067 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff2980d3a40 con 0x55592da424a0 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff2b03d2740 1 -- 172.21.15.33:0/172156067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff290003d20 con 0x55592da424a0 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff286ffd640 1 -- 172.21.15.33:0/172156067 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff29809bd10 con 0x55592da424a0 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff2b03d2740 10 client.18669 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff2b03d2740 1 -- 172.21.15.33:0/172156067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55592da42bf0 con 0x55592da424a0 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff286ffd640 1 -- 172.21.15.33:0/172156067 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff290003d20 con 0x55592da424a0 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff286ffd640 1 -- 172.21.15.33:0/172156067 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ff2980a4720 con 0x55592da424a0 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff2b03d2740 10 client.18669 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff2b03d2740 10 client.18669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff2b03d2740 20 client.18669 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55592dac7b70 0x55592dac9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:37.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff2b03d2740 4 client.18669 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff2b03d2740 1 -- 172.21.15.33:0/172156067 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55592d9afd60 con 0x55592dac7b70 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.566+0000 7ff2a6ffd640 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55592dac7b70 0x55592dac9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2a6ffd640 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55592dac7b70 0x55592dac9f60 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ce4b37741f31ade server_cookie=5e7aa6be12c91614 in_seq=0 out_seq=0 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff286ffd640 10 client.18669 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff286ffd640 1 -- 172.21.15.33:0/172156067 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55592d9afd60 con 0x55592dac7b70 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff286ffd640 10 client.18669 handle_command_reply: tid=0 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2b03d2740 1 client.18669 shutdown 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2b03d2740 1 -- 172.21.15.33:0/172156067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 msgr2=0x55592da62220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 0x55592da62220 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff2980b0370 tx=0x7ff29809c510 comp rx=0 tx=0).stop 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2b03d2740 1 -- 172.21.15.33:0/172156067 shutdown_connections 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55592dac7b70 0x55592dac9f60 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55592da4d680 0x55592dabee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55592da42dc0 0x55592da62760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2b03d2740 1 --2- 172.21.15.33:0/172156067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55592da424a0 0x55592da62220 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2b03d2740 1 -- 172.21.15.33:0/172156067 >> 172.21.15.33:0/172156067 conn(0x55592da3ea30 msgr2=0x55592da3db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:37.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff286ffd640 0 client.18669 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:37.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2b03d2740 1 -- 172.21.15.33:0/172156067 shutdown_connections 2024-09-16T00:34:37.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2b03d2740 1 -- 172.21.15.33:0/172156067 wait complete. 2024-09-16T00:34:37.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.567+0000 7ff2b03d2740 20 client.18669 trim_cache size 0 max 16384 2024-09-16T00:34:37.574 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:34:37.574 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:34:37.574 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:34:37.574 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:34:37.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.568+0000 7ff2adff9640 1 -- 172.21.15.33:0/388893136 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff2940a3240 msgr2=0x7ff2940a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:37.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.568+0000 7ff2adff9640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff2940a3240 0x7ff2940a5700 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0005e30 tx=0x7ff2a00095c0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.568+0000 7ff2adff9640 1 -- 172.21.15.33:0/388893136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2a8101de0 msgr2=0x7ff2a8108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:37.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.568+0000 7ff2adff9640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2a8101de0 0x7ff2a8108ab0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c950 tx=0x7ff29800ce10 comp rx=0 tx=0).stop 2024-09-16T00:34:37.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.568+0000 7ff2adff9640 1 -- 172.21.15.33:0/388893136 shutdown_connections 2024-09-16T00:34:37.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.568+0000 7ff2adff9640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff2940a3240 0x7ff2940a5700 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.568+0000 7ff2adff9640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2a8106f10 0x7ff2a8109530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.568+0000 7ff2adff9640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a8102790 0x7ff2a8108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.568+0000 7ff2adff9640 1 --2- 172.21.15.33:0/388893136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2a8101de0 0x7ff2a8108ab0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:37.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.568+0000 7ff2adff9640 1 -- 172.21.15.33:0/388893136 >> 172.21.15.33:0/388893136 conn(0x7ff2a80fd8c0 msgr2=0x7ff2a80fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:37.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.568+0000 7ff2adff9640 1 -- 172.21.15.33:0/388893136 shutdown_connections 2024-09-16T00:34:37.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:37.568+0000 7ff2adff9640 1 -- 172.21.15.33:0/388893136 wait complete. 2024-09-16T00:34:37.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:37 smithi083 ceph-mon[35242]: pgmap v874: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 11 MiB/s wr, 611 op/s 2024-09-16T00:34:37.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:37 smithi060 ceph-mon[35185]: pgmap v874: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 11 MiB/s wr, 611 op/s 2024-09-16T00:34:37.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:37 smithi033 ceph-mon[31243]: pgmap v874: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 11 MiB/s wr, 611 op/s 2024-09-16T00:34:38.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 20.53 MB/sec execute 104 sec latency 7410.377 ms 2024-09-16T00:34:38.580 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:34:38.580 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:34:38.842 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:39.078 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 20.33 MB/sec execute 105 sec latency 8410.467 ms 2024-09-16T00:34:39.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.332+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2933010829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ebc106a80 msgr2=0x7f5ebc106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:39.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.332+0000 7f5ec15c2640 1 --2- 172.21.15.33:0/2933010829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ebc106a80 0x7f5ebc106e60 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f5eac009cd0 tx=0x7f5eac02a8e0 comp rx=0 tx=0).stop 2024-09-16T00:34:39.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.332+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2933010829 shutdown_connections 2024-09-16T00:34:39.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.332+0000 7f5ec15c2640 1 --2- 172.21.15.33:0/2933010829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ebc101420 0x7f5ebc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:39.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.332+0000 7f5ec15c2640 1 --2- 172.21.15.33:0/2933010829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ebc100a80 0x7f5ebc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.332+0000 7f5ec15c2640 1 --2- 172.21.15.33:0/2933010829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ebc106a80 0x7f5ebc106e60 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.332+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2933010829 >> 172.21.15.33:0/2933010829 conn(0x7f5ebc0fc7a0 msgr2=0x7f5ebc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.332+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2933010829 shutdown_connections 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.332+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2933010829 wait complete. 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.333+0000 7f5ec15c2640 1 Processor -- start 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.333+0000 7f5ec15c2640 1 -- start start 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.333+0000 7f5ec15c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ebc100a80 0x7f5ebc0fcd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.333+0000 7f5ec15c2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ebc101420 0x7f5ebc0fd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.333+0000 7f5ec15c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ebc106a80 0x7f5ebc0fd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.333+0000 7f5ec15c2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5ebc110e80 con 0x7f5ebc106a80 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.333+0000 7f5ec15c2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5ebc110d00 con 0x7f5ebc101420 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.333+0000 7f5ec15c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5ebc111000 con 0x7f5ebc100a80 2024-09-16T00:34:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.333+0000 7f5ebb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ebc106a80 0x7f5ebc0fd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.333+0000 7f5ebb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ebc106a80 0x7f5ebc0fd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33472/0 (socket says 172.21.15.33:33472) 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.333+0000 7f5ebaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ebc100a80 0x7f5ebc0fcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.333+0000 7f5ebb7fe640 1 -- 172.21.15.33:0/2536716226 learned_addr learned my addr 172.21.15.33:0/2536716226 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.334+0000 7f5eba7fc640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ebc101420 0x7f5ebc0fd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.334+0000 7f5ebb7fe640 1 -- 172.21.15.33:0/2536716226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ebc100a80 msgr2=0x7f5ebc0fcd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.334+0000 7f5ebb7fe640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ebc100a80 0x7f5ebc0fcd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.334+0000 7f5ebb7fe640 1 -- 172.21.15.33:0/2536716226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ebc101420 msgr2=0x7f5ebc0fd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.334+0000 7f5ebb7fe640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ebc101420 0x7f5ebc0fd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.334+0000 7f5ebb7fe640 1 -- 172.21.15.33:0/2536716226 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ebc1a6930 con 0x7f5ebc106a80 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.334+0000 7f5ebb7fe640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ebc106a80 0x7f5ebc0fd800 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400c970 tx=0x7f5eb400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.334+0000 7f5ea3fff640 1 -- 172.21.15.33:0/2536716226 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4007bb0 con 0x7f5ebc106a80 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.334+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2536716226 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ebc1a6c20 con 0x7f5ebc106a80 2024-09-16T00:34:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.334+0000 7f5ea3fff640 1 -- 172.21.15.33:0/2536716226 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5eb4007d50 con 0x7f5ebc106a80 2024-09-16T00:34:39.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.334+0000 7f5ea3fff640 1 -- 172.21.15.33:0/2536716226 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4018800 con 0x7f5ebc106a80 2024-09-16T00:34:39.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.334+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2536716226 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ebc1a7110 con 0x7f5ebc106a80 2024-09-16T00:34:39.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.335+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2536716226 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ebc100ee0 con 0x7f5ebc106a80 2024-09-16T00:34:39.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.336+0000 7f5ea3fff640 1 -- 172.21.15.33:0/2536716226 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5eb400fa50 con 0x7f5ebc106a80 2024-09-16T00:34:39.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.337+0000 7f5ea3fff640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5ea40a3240 0x7f5ea40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:39.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.337+0000 7f5ea3fff640 1 -- 172.21.15.33:0/2536716226 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5eb40cffd0 con 0x7f5ebc106a80 2024-09-16T00:34:39.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.337+0000 7f5ebaffd640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5ea40a3240 0x7f5ea40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:39.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.338+0000 7f5ebaffd640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5ea40a3240 0x7f5ea40a5700 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5eac006010 tx=0x7f5eac009900 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:39.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.340+0000 7f5ea3fff640 1 -- 172.21.15.33:0/2536716226 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5eb4014030 con 0x7f5ebc106a80 2024-09-16T00:34:39.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.552+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2536716226 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ebc0fe2b0 con 0x7f5ebc106a80 2024-09-16T00:34:39.553 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:39 smithi033 ceph-mon[31243]: pgmap v875: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.6 MiB/s wr, 430 op/s 2024-09-16T00:34:39.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.553+0000 7f5ea3fff640 1 -- 172.21.15.33:0/2536716226 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f5ebc0fe2b0 con 0x7f5ebc106a80 2024-09-16T00:34:39.553 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:34:39.554 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:34:39.554 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:34:39.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.556+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2536716226 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5ea40a3240 msgr2=0x7f5ea40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:39.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.556+0000 7f5ec15c2640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5ea40a3240 0x7f5ea40a5700 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5eac006010 tx=0x7f5eac009900 comp rx=0 tx=0).stop 2024-09-16T00:34:39.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.556+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2536716226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ebc106a80 msgr2=0x7f5ebc0fd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:39.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.556+0000 7f5ec15c2640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ebc106a80 0x7f5ebc0fd800 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400c970 tx=0x7f5eb400ce30 comp rx=0 tx=0).stop 2024-09-16T00:34:39.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.556+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2536716226 shutdown_connections 2024-09-16T00:34:39.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.556+0000 7f5ec15c2640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5ea40a3240 0x7f5ea40a5700 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:39.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.556+0000 7f5ec15c2640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ebc106a80 0x7f5ebc0fd800 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:39.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.556+0000 7f5ec15c2640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ebc101420 0x7f5ebc0fd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:39.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.556+0000 7f5ec15c2640 1 --2- 172.21.15.33:0/2536716226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ebc100a80 0x7f5ebc0fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:39.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.556+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2536716226 >> 172.21.15.33:0/2536716226 conn(0x7f5ebc0fc7a0 msgr2=0x7f5ebc06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:39.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.556+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2536716226 shutdown_connections 2024-09-16T00:34:39.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:39.556+0000 7f5ec15c2640 1 -- 172.21.15.33:0/2536716226 wait complete. 2024-09-16T00:34:39.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:39 smithi060 ceph-mon[35185]: pgmap v875: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.6 MiB/s wr, 430 op/s 2024-09-16T00:34:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:39 smithi083 ceph-mon[35242]: pgmap v875: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.6 MiB/s wr, 430 op/s 2024-09-16T00:34:40.078 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 20.14 MB/sec execute 106 sec latency 9410.578 ms 2024-09-16T00:34:40.271 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:34:40.533 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:40.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:40 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2536716226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:34:40.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:40 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2536716226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:34:40.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:40 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2536716226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:34:41.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.027+0000 7fafdd74f640 1 -- 172.21.15.33:0/3000489816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd8101de0 msgr2=0x7fafd81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:41.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.027+0000 7fafdd74f640 1 --2- 172.21.15.33:0/3000489816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd8101de0 0x7fafd81021c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009ca0 tx=0x7fafc802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.027+0000 7fafdd74f640 1 -- 172.21.15.33:0/3000489816 shutdown_connections 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.027+0000 7fafdd74f640 1 --2- 172.21.15.33:0/3000489816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafd8106f10 0x7fafd810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.027+0000 7fafdd74f640 1 --2- 172.21.15.33:0/3000489816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafd8102790 0x7fafd81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.027+0000 7fafdd74f640 1 --2- 172.21.15.33:0/3000489816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd8101de0 0x7fafd81021c0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.027+0000 7fafdd74f640 1 -- 172.21.15.33:0/3000489816 >> 172.21.15.33:0/3000489816 conn(0x7fafd80fd8c0 msgr2=0x7fafd80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.027+0000 7fafdd74f640 1 -- 172.21.15.33:0/3000489816 shutdown_connections 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.027+0000 7fafdd74f640 1 -- 172.21.15.33:0/3000489816 wait complete. 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.028+0000 7fafdd74f640 1 Processor -- start 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.028+0000 7fafdd74f640 1 -- start start 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafdd74f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafd8101de0 0x7fafd81984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafdd74f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd8102790 0x7fafd81989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafdd74f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafd8106f10 0x7fafd819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafdd74f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8111fc0 con 0x7fafd8102790 2024-09-16T00:34:41.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafdd74f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8111e40 con 0x7fafd8106f10 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafdd74f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8112140 con 0x7fafd8101de0 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafd67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd8102790 0x7fafd81989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafd67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd8102790 0x7fafd81989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49908/0 (socket says 172.21.15.33:49908) 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafd67fc640 1 -- 172.21.15.33:0/1873376067 learned_addr learned my addr 172.21.15.33:0/1873376067 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafd6ffd640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafd8101de0 0x7fafd81984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafd77fe640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafd8106f10 0x7fafd819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafd67fc640 1 -- 172.21.15.33:0/1873376067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafd8101de0 msgr2=0x7fafd81984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafd67fc640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafd8101de0 0x7fafd81984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafd67fc640 1 -- 172.21.15.33:0/1873376067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafd8106f10 msgr2=0x7fafd819cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafd67fc640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafd8106f10 0x7fafd819cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafd67fc640 1 -- 172.21.15.33:0/1873376067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd819d450 con 0x7fafd8102790 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafd67fc640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd8102790 0x7fafd81989e0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fafc000d950 tx=0x7fafc000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:41.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafcffff640 1 -- 172.21.15.33:0/1873376067 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0017070 con 0x7fafd8102790 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafcffff640 1 -- 172.21.15.33:0/1873376067 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc0009c30 con 0x7fafd8102790 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.029+0000 7fafdd74f640 1 -- 172.21.15.33:0/1873376067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd819d740 con 0x7fafd8102790 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.030+0000 7fafcffff640 1 -- 172.21.15.33:0/1873376067 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc000f6b0 con 0x7fafd8102790 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.030+0000 7fafdd74f640 1 -- 172.21.15.33:0/1873376067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd81a5020 con 0x7fafd8102790 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.032+0000 7fafcffff640 1 -- 172.21.15.33:0/1873376067 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fafc00109d0 con 0x7fafd8102790 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.032+0000 7fafdfb28740 1 Processor -- start 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.032+0000 7fafdfb28740 1 -- start start 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafdfb28740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 0x55e77e0e0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafdfb28740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 0x55e77e0e0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafdfb28740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e0ed760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafdfb28740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e77e0462f0 con 0x55e77e0e05d0 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafdfb28740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e77dfedcf0 con 0x55e77e0eb370 2024-09-16T00:34:41.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafdfb28740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e77e0422b0 con 0x55e77e0e1b40 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 0x55e77e0e0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 0x55e77e0e0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49924/0 (socket says 172.21.15.33:49924) 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd6ffd640 1 -- 172.21.15.33:0/3877492830 learned_addr learned my addr 172.21.15.33:0/3877492830 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd77fe640 1 --2- 172.21.15.33:0/3877492830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 0x55e77e0e0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd67fc640 1 --2- 172.21.15.33:0/3877492830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e0ed760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafcffff640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fafc40a3240 0x7fafc40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd6ffd640 1 -- 172.21.15.33:0/3877492830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 msgr2=0x55e77e0e0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd6ffd640 1 --2- 172.21.15.33:0/3877492830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 0x55e77e0e0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd6ffd640 1 -- 172.21.15.33:0/3877492830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 msgr2=0x55e77e0ed760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd6ffd640 1 --2- 172.21.15.33:0/3877492830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e0ed760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd6ffd640 1 -- 172.21.15.33:0/3877492830 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e77dfd23a0 con 0x55e77e0e05d0 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd67fc640 1 --2- 172.21.15.33:0/3877492830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e0ed760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:34:41.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd77fe640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fafc40a3240 0x7fafc40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafcffff640 1 -- 172.21.15.33:0/1873376067 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fafc00cfa20 con 0x7fafd8102790 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafd6ffd640 1 --2- 172.21.15.33:0/3877492830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 0x55e77e0e0a10 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009ca0 tx=0x7fafc8009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cabef73c53dffc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.033+0000 7fafcd7fa640 1 -- 172.21.15.33:0/3877492830 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8003970 con 0x55e77e0e05d0 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.034+0000 7fafcd7fa640 1 -- 172.21.15.33:0/3877492830 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc8036030 con 0x55e77e0e05d0 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.034+0000 7fafcd7fa640 1 -- 172.21.15.33:0/3877492830 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8004480 con 0x55e77e0e05d0 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.034+0000 7fafdfb28740 1 -- 172.21.15.33:0/3877492830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 msgr2=0x55e77e0e0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.034+0000 7fafd77fe640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fafc40a3240 0x7fafc40a5700 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fafd0010df0 tx=0x7fafd0004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.034+0000 7fafdfb28740 1 --2- 172.21.15.33:0/3877492830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 0x55e77e0e0a10 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009ca0 tx=0x7fafc8009a10 comp rx=0 tx=0).stop 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.034+0000 7fafdfb28740 1 -- 172.21.15.33:0/3877492830 shutdown_connections 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.034+0000 7fafdfb28740 1 --2- 172.21.15.33:0/3877492830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e0ed760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.034+0000 7fafdfb28740 1 --2- 172.21.15.33:0/3877492830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 0x55e77e0e0a10 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.034+0000 7fafdfb28740 1 --2- 172.21.15.33:0/3877492830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 0x55e77e0e0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.034+0000 7fafdfb28740 1 -- 172.21.15.33:0/3877492830 >> 172.21.15.33:0/3877492830 conn(0x55e77e0a9310 msgr2=0x55e77e0dcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:41.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.034+0000 7fafdfb28740 1 -- 172.21.15.33:0/3877492830 shutdown_connections 2024-09-16T00:34:41.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.034+0000 7fafdfb28740 1 -- 172.21.15.33:0/3877492830 wait complete. 2024-09-16T00:34:41.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.035+0000 7fafdfb28740 1 -- start start 2024-09-16T00:34:41.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.035+0000 7fafdfb28740 1 Processor -- start 2024-09-16T00:34:41.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.035+0000 7fafdfb28740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0e05d0 0x55e77e0eace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.035+0000 7fafdfb28740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 0x55e77e0e7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.035+0000 7fafdfb28740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0eb370 0x55e77e0e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.035+0000 7fafdfb28740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e77e0ede90 con 0x55e77e0eb370 2024-09-16T00:34:41.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.035+0000 7fafdfb28740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e77e001060 con 0x55e77e0e05d0 2024-09-16T00:34:41.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.035+0000 7fafdfb28740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e77e0ee010 con 0x55e77e0e1b40 2024-09-16T00:34:41.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.035+0000 7fafdfb28740 10 client.0 _ll_register_callbacks cb 0x55e77dd548c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:41.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.035+0000 7fafd77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0eb370 0x55e77e0e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.035+0000 7fafd77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0eb370 0x55e77e0e84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49938/0 (socket says 172.21.15.33:49938) 2024-09-16T00:34:41.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.035+0000 7fafd77fe640 1 -- 172.21.15.33:0/1786371283 learned_addr learned my addr 172.21.15.33:0/1786371283 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:41.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafd6ffd640 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0e05d0 0x55e77e0eace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafd67fc640 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 0x55e77e0e7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafd77fe640 1 -- 172.21.15.33:0/1786371283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 msgr2=0x55e77e0e7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:41.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafd77fe640 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 0x55e77e0e7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafd77fe640 1 -- 172.21.15.33:0/1786371283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0e05d0 msgr2=0x55e77e0eace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:41.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafd77fe640 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0e05d0 0x55e77e0eace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafd77fe640 1 -- 172.21.15.33:0/1786371283 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e77ddb73c0 con 0x55e77e0eb370 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafd77fe640 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0eb370 0x55e77e0e84f0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fafd000dcb0 tx=0x7fafd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6f9957c9bd28e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafbe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafbe7fc640 1 -- 172.21.15.33:0/1786371283 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd000eb20 con 0x55e77e0eb370 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafbe7fc640 1 -- 172.21.15.33:0/1786371283 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fafd000ecc0 con 0x55e77e0eb370 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafdfb28740 1 -- 172.21.15.33:0/1786371283 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc8036030 con 0x55e77e0eb370 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafbe7fc640 1 -- 172.21.15.33:0/1786371283 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fafd000eec0 con 0x55e77e0eb370 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafdfb28740 10 client.18687 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafdfb28740 1 -- 172.21.15.33:0/1786371283 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e77e0f98c0 con 0x55e77e0eb370 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.036+0000 7fafbe7fc640 1 -- 172.21.15.33:0/1786371283 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc8036030 con 0x55e77e0eb370 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.037+0000 7fafbe7fc640 1 -- 172.21.15.33:0/1786371283 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fafd001a070 con 0x55e77e0eb370 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.037+0000 7fafdfb28740 10 client.18687 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:41.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.037+0000 7fafdfb28740 10 client.18687 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.037+0000 7fafdfb28740 20 client.18687 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.037+0000 7fafdfb28740 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e77e0fad80 0x55e77e0fd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.037+0000 7fafdfb28740 4 client.18687 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.037+0000 7fafdfb28740 1 -- 172.21.15.33:0/1786371283 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e77e053860 con 0x55e77e0fad80 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.037+0000 7fafd6ffd640 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e77e0fad80 0x55e77e0fd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.037+0000 7fafbe7fc640 10 client.18687 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.037+0000 7fafd6ffd640 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e77e0fad80 0x55e77e0fd1d0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a159d8921dce02e6 server_cookie=f074b6d03b4d10e8 in_seq=0 out_seq=0 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.038+0000 7fafbe7fc640 1 -- 172.21.15.33:0/1786371283 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e77e053860 con 0x55e77e0fad80 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.038+0000 7fafbe7fc640 10 client.18687 handle_command_reply: tid=0 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.038+0000 7fafdfb28740 1 client.18687 shutdown 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.038+0000 7fafdfb28740 1 -- 172.21.15.33:0/1786371283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0eb370 msgr2=0x55e77e0e84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.038+0000 7fafdfb28740 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0eb370 0x55e77e0e84f0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fafd000dcb0 tx=0x7fafd00046c0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.038+0000 7fafdfb28740 1 -- 172.21.15.33:0/1786371283 shutdown_connections 2024-09-16T00:34:41.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.038+0000 7fafdfb28740 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e77e0fad80 0x55e77e0fd1d0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.038+0000 7fafdfb28740 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0eb370 0x55e77e0e84f0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.038+0000 7fafdfb28740 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 0x55e77e0e7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.038+0000 7fafdfb28740 1 --2- 172.21.15.33:0/1786371283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0e05d0 0x55e77e0eace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.038+0000 7fafdfb28740 1 -- 172.21.15.33:0/1786371283 >> 172.21.15.33:0/1786371283 conn(0x55e77e0a9310 msgr2=0x55e77e0e2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:41.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.038+0000 7fafbe7fc640 0 client.18687 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:34:41.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.039+0000 7fafdfb28740 1 -- 172.21.15.33:0/1786371283 shutdown_connections 2024-09-16T00:34:41.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.039+0000 7fafdfb28740 1 -- 172.21.15.33:0/1786371283 wait complete. 2024-09-16T00:34:41.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.039+0000 7fafdfb28740 20 client.18687 trim_cache size 0 max 16384 2024-09-16T00:34:41.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.052+0000 7fafdfb28740 1 Processor -- start 2024-09-16T00:34:41.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.052+0000 7fafdfb28740 1 -- start start 2024-09-16T00:34:41.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.052+0000 7fafdfb28740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e05d0 0x55e77e0dce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.052+0000 7fafdfb28740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e1b40 0x55e77e0f7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.052+0000 7fafdfb28740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.052+0000 7fafdfb28740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81b18f0 con 0x55e77e0e1b40 2024-09-16T00:34:41.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.052+0000 7fafdfb28740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81af1c0 con 0x55e77e0eb370 2024-09-16T00:34:41.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.052+0000 7fafdfb28740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81ad010 con 0x55e77e0e05d0 2024-09-16T00:34:41.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.052+0000 7fafd67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e1b40 0x55e77e0f7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.052+0000 7fafd67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e1b40 0x55e77e0f7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49952/0 (socket says 172.21.15.33:49952) 2024-09-16T00:34:41.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.052+0000 7fafd67fc640 1 -- 172.21.15.33:0/3692909395 learned_addr learned my addr 172.21.15.33:0/3692909395 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:41.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.053+0000 7fafd6ffd640 1 --2- 172.21.15.33:0/3692909395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e05d0 0x55e77e0dce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.053+0000 7fafd77fe640 1 --2- 172.21.15.33:0/3692909395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.053+0000 7fafd67fc640 1 -- 172.21.15.33:0/3692909395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e05d0 msgr2=0x55e77e0dce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:41.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.053+0000 7fafd67fc640 1 --2- 172.21.15.33:0/3692909395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e05d0 0x55e77e0dce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.053+0000 7fafd67fc640 1 -- 172.21.15.33:0/3692909395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 msgr2=0x55e77e102aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:41.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.053+0000 7fafd67fc640 1 --2- 172.21.15.33:0/3692909395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e102aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.053+0000 7fafd67fc640 1 -- 172.21.15.33:0/3692909395 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e77e0dd730 con 0x55e77e0e1b40 2024-09-16T00:34:41.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.053+0000 7fafd77fe640 1 --2- 172.21.15.33:0/3692909395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:34:41.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.053+0000 7fafd67fc640 1 --2- 172.21.15.33:0/3692909395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e1b40 0x55e77e0f7130 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fafc0098590 tx=0x7fafc009b520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ec756c9672fd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.053+0000 7fafccff9640 1 -- 172.21.15.33:0/3692909395 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0020040 con 0x55e77e0e1b40 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.053+0000 7fafccff9640 1 -- 172.21.15.33:0/3692909395 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc009bbc0 con 0x55e77e0e1b40 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.053+0000 7fafccff9640 1 -- 172.21.15.33:0/3692909395 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc00a43a0 con 0x55e77e0e1b40 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.054+0000 7fafdfb28740 1 -- 172.21.15.33:0/3692909395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e1b40 msgr2=0x55e77e0f7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.054+0000 7fafdfb28740 1 --2- 172.21.15.33:0/3692909395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e1b40 0x55e77e0f7130 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fafc0098590 tx=0x7fafc009b520 comp rx=0 tx=0).stop 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.054+0000 7fafdfb28740 1 -- 172.21.15.33:0/3692909395 shutdown_connections 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.054+0000 7fafdfb28740 1 --2- 172.21.15.33:0/3692909395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.054+0000 7fafdfb28740 1 --2- 172.21.15.33:0/3692909395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e1b40 0x55e77e0f7130 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.054+0000 7fafdfb28740 1 --2- 172.21.15.33:0/3692909395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e05d0 0x55e77e0dce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.054+0000 7fafdfb28740 1 -- 172.21.15.33:0/3692909395 >> 172.21.15.33:0/3692909395 conn(0x55e77e0a9310 msgr2=0x55e77e0dc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.054+0000 7fafdfb28740 1 -- 172.21.15.33:0/3692909395 shutdown_connections 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.054+0000 7fafdfb28740 1 -- 172.21.15.33:0/3692909395 wait complete. 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.054+0000 7fafdfb28740 1 -- start start 2024-09-16T00:34:41.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.054+0000 7fafdfb28740 1 Processor -- start 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafdfb28740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 0x55e77e0f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafdfb28740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 0x55e77e0e4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafdfb28740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e0e4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafdfb28740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81ad680 con 0x55e77e0e05d0 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafdfb28740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81aeaf0 con 0x55e77e0eb370 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafdfb28740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81add50 con 0x55e77e0e1b40 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafdfb28740 10 client.0 _ll_register_callbacks cb 0x7fafd0010750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafd6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 0x55e77e0f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafd6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 0x55e77e0f36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49962/0 (socket says 172.21.15.33:49962) 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafd6ffd640 1 -- 172.21.15.33:0/1976495529 learned_addr learned my addr 172.21.15.33:0/1976495529 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafd67fc640 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 0x55e77e0e4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafd77fe640 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e0e4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafd6ffd640 1 -- 172.21.15.33:0/1976495529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 msgr2=0x55e77e0e4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:41.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafd6ffd640 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 0x55e77e0e4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafd6ffd640 1 -- 172.21.15.33:0/1976495529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 msgr2=0x55e77e0e4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:41.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafd6ffd640 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e0e4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafd6ffd640 1 -- 172.21.15.33:0/1976495529 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e77e0e53e0 con 0x55e77e0e05d0 2024-09-16T00:34:41.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafd6ffd640 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 0x55e77e0f36a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009bf0 tx=0x7fafc8009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614422e8fb90fe5e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:41.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafbe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:41.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafbe7fc640 1 -- 172.21.15.33:0/1976495529 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8060060 con 0x55e77e0e05d0 2024-09-16T00:34:41.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafbe7fc640 1 -- 172.21.15.33:0/1976495529 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fafc8003970 con 0x55e77e0e05d0 2024-09-16T00:34:41.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.055+0000 7fafdfb28740 1 -- 172.21.15.33:0/1976495529 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc009bbc0 con 0x55e77e0e05d0 2024-09-16T00:34:41.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.056+0000 7fafbe7fc640 1 -- 172.21.15.33:0/1976495529 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fafc8067050 con 0x55e77e0e05d0 2024-09-16T00:34:41.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.056+0000 7fafdfb28740 10 client.18693 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:41.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.056+0000 7fafdfb28740 1 -- 172.21.15.33:0/1976495529 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e77e1656d0 con 0x55e77e0e05d0 2024-09-16T00:34:41.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.056+0000 7fafbe7fc640 1 -- 172.21.15.33:0/1976495529 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc009bbc0 con 0x55e77e0e05d0 2024-09-16T00:34:41.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.056+0000 7fafbe7fc640 1 -- 172.21.15.33:0/1976495529 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fafc8043070 con 0x55e77e0e05d0 2024-09-16T00:34:41.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.056+0000 7fafdfb28740 10 client.18693 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:41.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.056+0000 7fafdfb28740 10 client.18693 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:34:41.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.056+0000 7fafdfb28740 20 client.18693 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:41.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.056+0000 7fafdfb28740 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e77e165b20 0x55e77e167f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:41.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.056+0000 7fafdfb28740 4 client.18693 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.056+0000 7fafdfb28740 1 -- 172.21.15.33:0/1976495529 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e77e04dd70 con 0x55e77e165b20 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.056+0000 7fafd67fc640 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e77e165b20 0x55e77e167f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafbe7fc640 10 client.18693 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafd67fc640 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e77e165b20 0x55e77e167f40 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4a5329710a402080 server_cookie=b61581a7a392f8e1 in_seq=0 out_seq=0 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafbe7fc640 1 -- 172.21.15.33:0/1976495529 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e77e04dd70 con 0x55e77e165b20 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafbe7fc640 10 client.18693 handle_command_reply: tid=0 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafdfb28740 1 client.18693 shutdown 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafdfb28740 1 -- 172.21.15.33:0/1976495529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 msgr2=0x55e77e0f36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafdfb28740 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 0x55e77e0f36a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009bf0 tx=0x7fafc8009800 comp rx=0 tx=0).stop 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafdfb28740 1 -- 172.21.15.33:0/1976495529 shutdown_connections 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafdfb28740 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55e77e165b20 0x55e77e167f40 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafdfb28740 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e77e0eb370 0x55e77e0e4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafdfb28740 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e77e0e1b40 0x55e77e0e4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafbe7fc640 0 client.18693 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:34:41.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafdfb28740 1 --2- 172.21.15.33:0/1976495529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e77e0e05d0 0x55e77e0f36a0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafdfb28740 1 -- 172.21.15.33:0/1976495529 >> 172.21.15.33:0/1976495529 conn(0x55e77e0a9310 msgr2=0x55e77e0dbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:41.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafdfb28740 1 -- 172.21.15.33:0/1976495529 shutdown_connections 2024-09-16T00:34:41.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafdfb28740 1 -- 172.21.15.33:0/1976495529 wait complete. 2024-09-16T00:34:41.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.057+0000 7fafdfb28740 20 client.18693 trim_cache size 0 max 16384 2024-09-16T00:34:41.066 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:34:41.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.058+0000 7fafdd74f640 1 -- 172.21.15.33:0/1873376067 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fafc40a3240 msgr2=0x7fafc40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:41.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.058+0000 7fafdd74f640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fafc40a3240 0x7fafc40a5700 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fafd0010df0 tx=0x7fafd0004730 comp rx=0 tx=0).stop 2024-09-16T00:34:41.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.058+0000 7fafdd74f640 1 -- 172.21.15.33:0/1873376067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd8102790 msgr2=0x7fafd81989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:41.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.058+0000 7fafdd74f640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd8102790 0x7fafd81989e0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fafc000d950 tx=0x7fafc000de10 comp rx=0 tx=0).stop 2024-09-16T00:34:41.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.058+0000 7fafdd74f640 1 -- 172.21.15.33:0/1873376067 shutdown_connections 2024-09-16T00:34:41.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.058+0000 7fafdd74f640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fafc40a3240 0x7fafc40a5700 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.058+0000 7fafdd74f640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafd8106f10 0x7fafd819cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.058+0000 7fafdd74f640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd8102790 0x7fafd81989e0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.058+0000 7fafdd74f640 1 --2- 172.21.15.33:0/1873376067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafd8101de0 0x7fafd81984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:41.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.058+0000 7fafdd74f640 1 -- 172.21.15.33:0/1873376067 >> 172.21.15.33:0/1873376067 conn(0x7fafd80fd8c0 msgr2=0x7fafd80fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:41.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.058+0000 7fafdd74f640 1 -- 172.21.15.33:0/1873376067 shutdown_connections 2024-09-16T00:34:41.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:41.058+0000 7fafdd74f640 1 -- 172.21.15.33:0/1873376067 wait complete. 2024-09-16T00:34:41.077 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 19.95 MB/sec execute 107 sec latency 10410.637 ms 2024-09-16T00:34:41.685 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:41 smithi033 ceph-mon[31243]: pgmap v876: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.6 MiB/s wr, 436 op/s 2024-09-16T00:34:41.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:41 smithi060 ceph-mon[35185]: pgmap v876: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.6 MiB/s wr, 436 op/s 2024-09-16T00:34:41.840 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:34:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:41 smithi083 ceph-mon[35242]: pgmap v876: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.6 MiB/s wr, 436 op/s 2024-09-16T00:34:42.078 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 19.77 MB/sec execute 108 sec latency 11410.677 ms 2024-09-16T00:34:42.101 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:42.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.580+0000 7f8919c72640 1 -- 172.21.15.33:0/1608040466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8914101430 msgr2=0x7f891410d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:42.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.580+0000 7f8919c72640 1 --2- 172.21.15.33:0/1608040466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8914101430 0x7f891410d960 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f890c009cd0 tx=0x7f890c02c7c0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.580+0000 7f8919c72640 1 -- 172.21.15.33:0/1608040466 shutdown_connections 2024-09-16T00:34:42.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.580+0000 7f8919c72640 1 --2- 172.21.15.33:0/1608040466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8914101430 0x7f891410d960 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.580+0000 7f8919c72640 1 --2- 172.21.15.33:0/1608040466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8914100a90 0x7f8914100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.580+0000 7f8919c72640 1 --2- 172.21.15.33:0/1608040466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8914106a90 0x7f8914106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.580+0000 7f8919c72640 1 -- 172.21.15.33:0/1608040466 >> 172.21.15.33:0/1608040466 conn(0x7f89140fc7d0 msgr2=0x7f89140febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:42.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.581+0000 7f8919c72640 1 -- 172.21.15.33:0/1608040466 shutdown_connections 2024-09-16T00:34:42.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.581+0000 7f8919c72640 1 -- 172.21.15.33:0/1608040466 wait complete. 2024-09-16T00:34:42.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.581+0000 7f8919c72640 1 Processor -- start 2024-09-16T00:34:42.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.581+0000 7f8919c72640 1 -- start start 2024-09-16T00:34:42.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f8919c72640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8914100a90 0x7f89141961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f8919c72640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8914101430 0x7f8914196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f8919c72640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8914106a90 0x7f891419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f8919c72640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f891410fd80 con 0x7f8914101430 2024-09-16T00:34:42.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f8919c72640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f891410fc00 con 0x7f8914106a90 2024-09-16T00:34:42.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f8919c72640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f891410ff00 con 0x7f8914100a90 2024-09-16T00:34:42.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f8912ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8914101430 0x7f8914196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f8912ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8914101430 0x7f8914196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49972/0 (socket says 172.21.15.33:49972) 2024-09-16T00:34:42.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f8912ffd640 1 -- 172.21.15.33:0/3466325254 learned_addr learned my addr 172.21.15.33:0/3466325254 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:42.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f89137fe640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8914100a90 0x7f89141961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f8913fff640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8914106a90 0x7f891419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f89137fe640 1 -- 172.21.15.33:0/3466325254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8914106a90 msgr2=0x7f891419aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:42.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f89137fe640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8914106a90 0x7f891419aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f89137fe640 1 -- 172.21.15.33:0/3466325254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8914101430 msgr2=0x7f8914196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:42.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f89137fe640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8914101430 0x7f8914196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f89137fe640 1 -- 172.21.15.33:0/3466325254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891419b170 con 0x7f8914100a90 2024-09-16T00:34:42.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f8912ffd640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8914101430 0x7f8914196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:42.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.582+0000 7f89137fe640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8914100a90 0x7f89141961c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f890400e9f0 tx=0x7f890400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:42.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.583+0000 7f8910ff9640 1 -- 172.21.15.33:0/3466325254 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890400cd80 con 0x7f8914100a90 2024-09-16T00:34:42.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.583+0000 7f8910ff9640 1 -- 172.21.15.33:0/3466325254 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89040035f0 con 0x7f8914100a90 2024-09-16T00:34:42.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.583+0000 7f8919c72640 1 -- 172.21.15.33:0/3466325254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f891419b460 con 0x7f8914100a90 2024-09-16T00:34:42.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.583+0000 7f8910ff9640 1 -- 172.21.15.33:0/3466325254 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904010670 con 0x7f8914100a90 2024-09-16T00:34:42.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.583+0000 7f8919c72640 1 -- 172.21.15.33:0/3466325254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89141a2d40 con 0x7f8914100a90 2024-09-16T00:34:42.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f891c04b740 1 Processor -- start 2024-09-16T00:34:42.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f891c04b740 1 -- start start 2024-09-16T00:34:42.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f8910ff9640 1 -- 172.21.15.33:0/3466325254 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f89040037a0 con 0x7f8914100a90 2024-09-16T00:34:42.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f891c04b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55deb40 0x5575c55dd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f891c04b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55dd5d0 0x5575c55dda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f891c04b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 0x5575c55eeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f891c04b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5575c55432f0 con 0x5575c55ec780 2024-09-16T00:34:42.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f891c04b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5575c54eacf0 con 0x5575c55dd5d0 2024-09-16T00:34:42.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f891c04b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5575c553f2b0 con 0x5575c55deb40 2024-09-16T00:34:42.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f8912ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 0x5575c55eeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f89137fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55dd5d0 0x5575c55dda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f8912ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 0x5575c55eeb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49986/0 (socket says 172.21.15.33:49986) 2024-09-16T00:34:42.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f89137fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55dd5d0 0x5575c55dda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:60940/0 (socket says 172.21.15.33:60940) 2024-09-16T00:34:42.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f8912ffd640 1 -- 172.21.15.33:0/800860845 learned_addr learned my addr 172.21.15.33:0/800860845 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:42.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.586+0000 7f8913fff640 1 --2- 172.21.15.33:0/800860845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55deb40 0x5575c55dd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.587+0000 7f8912ffd640 1 -- 172.21.15.33:0/800860845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55deb40 msgr2=0x5575c55dd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:42.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.587+0000 7f8912ffd640 1 --2- 172.21.15.33:0/800860845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55deb40 0x5575c55dd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.587+0000 7f8912ffd640 1 -- 172.21.15.33:0/800860845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55dd5d0 msgr2=0x5575c55dda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.587+0000 7f8912ffd640 1 --2- 172.21.15.33:0/800860845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55dd5d0 0x5575c55dda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.587+0000 7f8912ffd640 1 -- 172.21.15.33:0/800860845 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575c54cf3a0 con 0x5575c55ec780 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.587+0000 7f8912ffd640 1 --2- 172.21.15.33:0/800860845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 0x5575c55eeb70 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f89080115f0 tx=0x7f8908011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a8a1784561653a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.587+0000 7f8910ff9640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f88fc0a3240 0x7f88fc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.587+0000 7f8913fff640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f88fc0a3240 0x7f88fc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.587+0000 7f88f9ffb640 1 -- 172.21.15.33:0/800860845 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890801e070 con 0x5575c55ec780 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.587+0000 7f88f9ffb640 1 -- 172.21.15.33:0/800860845 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f890800f660 con 0x5575c55ec780 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.587+0000 7f8910ff9640 1 -- 172.21.15.33:0/3466325254 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f89040d0010 con 0x7f8914100a90 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.587+0000 7f88f9ffb640 1 -- 172.21.15.33:0/800860845 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8908003870 con 0x5575c55ec780 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.588+0000 7f891c04b740 1 -- 172.21.15.33:0/800860845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 msgr2=0x5575c55eeb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.588+0000 7f8913fff640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f88fc0a3240 0x7f88fc0a5700 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f890c02cce0 tx=0x7f890c032340 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.588+0000 7f891c04b740 1 --2- 172.21.15.33:0/800860845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 0x5575c55eeb70 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f89080115f0 tx=0x7f8908011ab0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.588+0000 7f891c04b740 1 -- 172.21.15.33:0/800860845 shutdown_connections 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.588+0000 7f891c04b740 1 --2- 172.21.15.33:0/800860845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 0x5575c55eeb70 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.588+0000 7f891c04b740 1 --2- 172.21.15.33:0/800860845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55dd5d0 0x5575c55dda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.588+0000 7f891c04b740 1 --2- 172.21.15.33:0/800860845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55deb40 0x5575c55dd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.588+0000 7f891c04b740 1 -- 172.21.15.33:0/800860845 >> 172.21.15.33:0/800860845 conn(0x5575c55a6310 msgr2=0x5575c55d9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.588+0000 7f891c04b740 1 -- 172.21.15.33:0/800860845 shutdown_connections 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.588+0000 7f891c04b740 1 -- 172.21.15.33:0/800860845 wait complete. 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.588+0000 7f891c04b740 1 -- start start 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.588+0000 7f891c04b740 1 Processor -- start 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f891c04b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55dd5d0 0x5575c55ec160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f891c04b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55deb40 0x5575c55e9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f891c04b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55ec780 0x5575c55e99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f891c04b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5575c55ef480 con 0x5575c55dd5d0 2024-09-16T00:34:42.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f891c04b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5575c54fe060 con 0x5575c55ec780 2024-09-16T00:34:42.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f891c04b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5575c55ef600 con 0x5575c55deb40 2024-09-16T00:34:42.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f891c04b740 10 client.0 _ll_register_callbacks cb 0x5575c52518c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:42.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f89137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55dd5d0 0x5575c55ec160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f89137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55dd5d0 0x5575c55ec160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49992/0 (socket says 172.21.15.33:49992) 2024-09-16T00:34:42.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f89137fe640 1 -- 172.21.15.33:0/89303022 learned_addr learned my addr 172.21.15.33:0/89303022 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:42.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f8912ffd640 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55deb40 0x5575c55e9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f8913fff640 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55ec780 0x5575c55e99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f89137fe640 1 -- 172.21.15.33:0/89303022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55deb40 msgr2=0x5575c55e9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:42.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f89137fe640 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55deb40 0x5575c55e9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f89137fe640 1 -- 172.21.15.33:0/89303022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55ec780 msgr2=0x5575c55e99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:42.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f89137fe640 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55ec780 0x5575c55e99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f89137fe640 1 -- 172.21.15.33:0/89303022 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575c52b43c0 con 0x5575c55dd5d0 2024-09-16T00:34:42.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f8913fff640 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55ec780 0x5575c55e99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:34:42.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f89137fe640 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55dd5d0 0x5575c55ec160 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f890400e9c0 tx=0x7f89040d36e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2963df5d122bd5a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:42.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.589+0000 7f88f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:42.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f88f2ffd640 1 -- 172.21.15.33:0/89303022 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890409bb10 con 0x5575c55dd5d0 2024-09-16T00:34:42.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f891c04b740 1 -- 172.21.15.33:0/89303022 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f890800f660 con 0x5575c55dd5d0 2024-09-16T00:34:42.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f88f2ffd640 1 -- 172.21.15.33:0/89303022 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f890409a4b0 con 0x5575c55dd5d0 2024-09-16T00:34:42.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f88f2ffd640 1 -- 172.21.15.33:0/89303022 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f890409c8b0 con 0x5575c55dd5d0 2024-09-16T00:34:42.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f891c04b740 10 client.18705 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:42.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f891c04b740 1 -- 172.21.15.33:0/89303022 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5575c55faf00 con 0x5575c55dd5d0 2024-09-16T00:34:42.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f88f2ffd640 1 -- 172.21.15.33:0/89303022 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f890800f660 con 0x5575c55dd5d0 2024-09-16T00:34:42.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f88f2ffd640 1 -- 172.21.15.33:0/89303022 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f8904098820 con 0x5575c55dd5d0 2024-09-16T00:34:42.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f891c04b740 10 client.18705 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:42.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f891c04b740 10 client.18705 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:34:42.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f891c04b740 20 client.18705 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:42.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f891c04b740 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5575c55fc310 0x5575c55fe780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f891c04b740 4 client.18705 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:34:42.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f891c04b740 1 -- 172.21.15.33:0/89303022 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575c5550860 con 0x5575c55fc310 2024-09-16T00:34:42.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.590+0000 7f8912ffd640 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5575c55fc310 0x5575c55fe780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.591+0000 7f8912ffd640 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5575c55fc310 0x5575c55fe780 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8c8a284fd292f2d4 server_cookie=d6cfec53fe0cb91d in_seq=0 out_seq=0 2024-09-16T00:34:42.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.591+0000 7f88f2ffd640 10 client.18705 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:34:42.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.591+0000 7f88f2ffd640 1 -- 172.21.15.33:0/89303022 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5575c5550860 con 0x5575c55fc310 2024-09-16T00:34:42.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.591+0000 7f88f2ffd640 10 client.18705 handle_command_reply: tid=0 2024-09-16T00:34:42.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f891c04b740 1 client.18705 shutdown 2024-09-16T00:34:42.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f891c04b740 1 -- 172.21.15.33:0/89303022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55dd5d0 msgr2=0x5575c55ec160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:42.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f891c04b740 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55dd5d0 0x5575c55ec160 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f890400e9c0 tx=0x7f89040d36e0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f891c04b740 1 -- 172.21.15.33:0/89303022 shutdown_connections 2024-09-16T00:34:42.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f891c04b740 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5575c55fc310 0x5575c55fe780 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f891c04b740 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55ec780 0x5575c55e99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f891c04b740 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55deb40 0x5575c55e9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f891c04b740 1 --2- 172.21.15.33:0/89303022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55dd5d0 0x5575c55ec160 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f88f2ffd640 0 client.18705 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:34:42.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f891c04b740 1 -- 172.21.15.33:0/89303022 >> 172.21.15.33:0/89303022 conn(0x5575c55a6310 msgr2=0x5575c55df960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:42.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f891c04b740 1 -- 172.21.15.33:0/89303022 shutdown_connections 2024-09-16T00:34:42.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f891c04b740 1 -- 172.21.15.33:0/89303022 wait complete. 2024-09-16T00:34:42.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.592+0000 7f891c04b740 20 client.18705 trim_cache size 0 max 16384 2024-09-16T00:34:42.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.605+0000 7f891c04b740 1 Processor -- start 2024-09-16T00:34:42.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.605+0000 7f891c04b740 1 -- start start 2024-09-16T00:34:42.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.605+0000 7f891c04b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55dd5d0 0x5575c55f8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.605+0000 7f891c04b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55deb40 0x5575c5604040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.605+0000 7f891c04b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55ec780 0x5575c5604580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.605+0000 7f891c04b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f89141af800 con 0x5575c55deb40 2024-09-16T00:34:42.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.605+0000 7f891c04b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f89141ad0d0 con 0x5575c55ec780 2024-09-16T00:34:42.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.605+0000 7f891c04b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89141aaf20 con 0x5575c55dd5d0 2024-09-16T00:34:42.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.605+0000 7f8912ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55deb40 0x5575c5604040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.605+0000 7f8912ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55deb40 0x5575c5604040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50002/0 (socket says 172.21.15.33:50002) 2024-09-16T00:34:42.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.606+0000 7f8912ffd640 1 -- 172.21.15.33:0/1907890944 learned_addr learned my addr 172.21.15.33:0/1907890944 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:42.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.606+0000 7f89137fe640 1 --2- 172.21.15.33:0/1907890944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55dd5d0 0x5575c55f8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.606+0000 7f8913fff640 1 --2- 172.21.15.33:0/1907890944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55ec780 0x5575c5604580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.606+0000 7f8912ffd640 1 -- 172.21.15.33:0/1907890944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55dd5d0 msgr2=0x5575c55f8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:42.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.606+0000 7f8912ffd640 1 --2- 172.21.15.33:0/1907890944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55dd5d0 0x5575c55f8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.606+0000 7f8912ffd640 1 -- 172.21.15.33:0/1907890944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55ec780 msgr2=0x5575c5604580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:42.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.606+0000 7f8912ffd640 1 --2- 172.21.15.33:0/1907890944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55ec780 0x5575c5604580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.606+0000 7f8912ffd640 1 -- 172.21.15.33:0/1907890944 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575c55d9ed0 con 0x5575c55deb40 2024-09-16T00:34:42.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.606+0000 7f8912ffd640 1 --2- 172.21.15.33:0/1907890944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55deb40 0x5575c5604040 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f89080115f0 tx=0x7f8908005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65015bbc64576831 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:42.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.606+0000 7f88f97fa640 1 -- 172.21.15.33:0/1907890944 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8908028070 con 0x5575c55deb40 2024-09-16T00:34:42.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.606+0000 7f88f97fa640 1 -- 172.21.15.33:0/1907890944 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f890800f660 con 0x5575c55deb40 2024-09-16T00:34:42.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.606+0000 7f88f97fa640 1 -- 172.21.15.33:0/1907890944 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890801e040 con 0x5575c55deb40 2024-09-16T00:34:42.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.607+0000 7f891c04b740 1 -- 172.21.15.33:0/1907890944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55deb40 msgr2=0x5575c5604040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:42.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.607+0000 7f891c04b740 1 --2- 172.21.15.33:0/1907890944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55deb40 0x5575c5604040 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f89080115f0 tx=0x7f8908005f90 comp rx=0 tx=0).stop 2024-09-16T00:34:42.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.607+0000 7f891c04b740 1 -- 172.21.15.33:0/1907890944 shutdown_connections 2024-09-16T00:34:42.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.607+0000 7f891c04b740 1 --2- 172.21.15.33:0/1907890944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55ec780 0x5575c5604580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.607+0000 7f891c04b740 1 --2- 172.21.15.33:0/1907890944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55deb40 0x5575c5604040 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.607+0000 7f891c04b740 1 --2- 172.21.15.33:0/1907890944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55dd5d0 0x5575c55f8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.607+0000 7f891c04b740 1 -- 172.21.15.33:0/1907890944 >> 172.21.15.33:0/1907890944 conn(0x5575c55a6310 msgr2=0x5575c55e4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:42.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.607+0000 7f891c04b740 1 -- 172.21.15.33:0/1907890944 shutdown_connections 2024-09-16T00:34:42.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.607+0000 7f891c04b740 1 -- 172.21.15.33:0/1907890944 wait complete. 2024-09-16T00:34:42.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.607+0000 7f891c04b740 1 -- start start 2024-09-16T00:34:42.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f891c04b740 1 Processor -- start 2024-09-16T00:34:42.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f891c04b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55dd5d0 0x5575c55f0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f891c04b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55deb40 0x5575c55f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f891c04b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 0x5575c55f11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f891c04b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f89141ab590 con 0x5575c55ec780 2024-09-16T00:34:42.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f891c04b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f89141aca00 con 0x5575c55deb40 2024-09-16T00:34:42.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f891c04b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89141abc60 con 0x5575c55dd5d0 2024-09-16T00:34:42.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f891c04b740 10 client.0 _ll_register_callbacks cb 0x7f89040d37e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:42.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f8913fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 0x5575c55f11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f8913fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 0x5575c55f11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50018/0 (socket says 172.21.15.33:50018) 2024-09-16T00:34:42.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f8913fff640 1 -- 172.21.15.33:0/4123756894 learned_addr learned my addr 172.21.15.33:0/4123756894 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:42.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f89137fe640 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55dd5d0 0x5575c55f0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f8912ffd640 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55deb40 0x5575c55f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f8913fff640 1 -- 172.21.15.33:0/4123756894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55dd5d0 msgr2=0x5575c55f0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:42.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f8913fff640 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55dd5d0 0x5575c55f0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f8913fff640 1 -- 172.21.15.33:0/4123756894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55deb40 msgr2=0x5575c55f0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:42.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f8913fff640 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55deb40 0x5575c55f0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.608+0000 7f8913fff640 1 -- 172.21.15.33:0/4123756894 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575c56661b0 con 0x5575c55ec780 2024-09-16T00:34:42.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f8913fff640 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 0x5575c55f11f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f890c0323d0 tx=0x7f890c02c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca3b51cad02ae03 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:42.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f88f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:42.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f88f2ffd640 1 -- 172.21.15.33:0/4123756894 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c039050 con 0x5575c55ec780 2024-09-16T00:34:42.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f88f2ffd640 1 -- 172.21.15.33:0/4123756894 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f890c004040 con 0x5575c55ec780 2024-09-16T00:34:42.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f891c04b740 1 -- 172.21.15.33:0/4123756894 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f890800f660 con 0x5575c55ec780 2024-09-16T00:34:42.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f88f2ffd640 1 -- 172.21.15.33:0/4123756894 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f890c05c050 con 0x5575c55ec780 2024-09-16T00:34:42.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f891c04b740 10 client.18711 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:42.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f891c04b740 1 -- 172.21.15.33:0/4123756894 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5575c5667470 con 0x5575c55ec780 2024-09-16T00:34:42.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f88f2ffd640 1 -- 172.21.15.33:0/4123756894 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f890800f660 con 0x5575c55ec780 2024-09-16T00:34:42.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f88f2ffd640 1 -- 172.21.15.33:0/4123756894 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f890c032b10 con 0x5575c55ec780 2024-09-16T00:34:42.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f891c04b740 10 client.18711 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:42.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f891c04b740 10 client.18711 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:34:42.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f891c04b740 20 client.18711 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:42.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f891c04b740 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5575c5667c10 0x5575c566a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:42.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f891c04b740 4 client.18711 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:34:42.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.609+0000 7f891c04b740 1 -- 172.21.15.33:0/4123756894 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5575c554ad70 con 0x5575c5667c10 2024-09-16T00:34:42.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f89137fe640 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5575c5667c10 0x5575c566a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:42.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f89137fe640 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5575c5667c10 0x5575c566a030 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dbdd90eaef3796f2 server_cookie=7efa5408cd5eb018 in_seq=0 out_seq=0 2024-09-16T00:34:42.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f88f2ffd640 10 client.18711 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:34:42.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f88f2ffd640 1 -- 172.21.15.33:0/4123756894 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5575c554ad70 con 0x5575c5667c10 2024-09-16T00:34:42.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f88f2ffd640 10 client.18711 handle_command_reply: tid=0 2024-09-16T00:34:42.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f891c04b740 1 client.18711 shutdown 2024-09-16T00:34:42.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f891c04b740 1 -- 172.21.15.33:0/4123756894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 msgr2=0x5575c55f11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:42.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f891c04b740 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 0x5575c55f11f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f890c0323d0 tx=0x7f890c02c7c0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f891c04b740 1 -- 172.21.15.33:0/4123756894 shutdown_connections 2024-09-16T00:34:42.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f891c04b740 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575c55ec780 0x5575c55f11f0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f891c04b740 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5575c5667c10 0x5575c566a030 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f891c04b740 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5575c55deb40 0x5575c55f0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f891c04b740 1 --2- 172.21.15.33:0/4123756894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575c55dd5d0 0x5575c55f0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f88f2ffd640 0 client.18711 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:34:42.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.610+0000 7f891c04b740 1 -- 172.21.15.33:0/4123756894 >> 172.21.15.33:0/4123756894 conn(0x5575c55a6310 msgr2=0x5575c55d84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:42.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f891c04b740 1 -- 172.21.15.33:0/4123756894 shutdown_connections 2024-09-16T00:34:42.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f891c04b740 1 -- 172.21.15.33:0/4123756894 wait complete. 2024-09-16T00:34:42.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f891c04b740 20 client.18711 trim_cache size 0 max 16384 2024-09-16T00:34:42.635 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:34:42.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f8919c72640 1 -- 172.21.15.33:0/3466325254 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f88fc0a3240 msgr2=0x7f88fc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:42.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f8919c72640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f88fc0a3240 0x7f88fc0a5700 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f890c02cce0 tx=0x7f890c032340 comp rx=0 tx=0).stop 2024-09-16T00:34:42.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f8919c72640 1 -- 172.21.15.33:0/3466325254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8914100a90 msgr2=0x7f89141961c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:42.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f8919c72640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8914100a90 0x7f89141961c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f890400e9f0 tx=0x7f890400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f8919c72640 1 -- 172.21.15.33:0/3466325254 shutdown_connections 2024-09-16T00:34:42.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f8919c72640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f88fc0a3240 0x7f88fc0a5700 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f8919c72640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8914106a90 0x7f891419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f8919c72640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8914101430 0x7f8914196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f8919c72640 1 --2- 172.21.15.33:0/3466325254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8914100a90 0x7f89141961c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:42.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f8919c72640 1 -- 172.21.15.33:0/3466325254 >> 172.21.15.33:0/3466325254 conn(0x7f89140fc7d0 msgr2=0x7f89140fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:42.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f8919c72640 1 -- 172.21.15.33:0/3466325254 shutdown_connections 2024-09-16T00:34:42.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:42.611+0000 7f8919c72640 1 -- 172.21.15.33:0/3466325254 wait complete. 2024-09-16T00:34:43.078 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 19.59 MB/sec execute 109 sec latency 12410.753 ms 2024-09-16T00:34:43.348 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:34:43.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:43 smithi033 ceph-mon[31243]: pgmap v877: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 259 op/s 2024-09-16T00:34:43.612 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:43.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:43 smithi060 ceph-mon[35185]: pgmap v877: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 259 op/s 2024-09-16T00:34:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:43 smithi083 ceph-mon[35242]: pgmap v877: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 259 op/s 2024-09-16T00:34:44.078 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 19.41 MB/sec execute 110 sec latency 13410.805 ms 2024-09-16T00:34:44.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.107+0000 7f003bfb9640 1 -- 172.21.15.33:0/1244687896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034102d30 msgr2=0x7f0034103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:44.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.107+0000 7f003bfb9640 1 --2- 172.21.15.33:0/1244687896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034102d30 0x7f0034103110 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f0030009cd0 tx=0x7f003002a8e0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.107+0000 7f003bfb9640 1 -- 172.21.15.33:0/1244687896 shutdown_connections 2024-09-16T00:34:44.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.107+0000 7f003bfb9640 1 --2- 172.21.15.33:0/1244687896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00341082a0 0x7f003410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.107+0000 7f003bfb9640 1 --2- 172.21.15.33:0/1244687896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00341036e0 0x7f0034107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.107+0000 7f003bfb9640 1 --2- 172.21.15.33:0/1244687896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034102d30 0x7f0034103110 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.107+0000 7f003bfb9640 1 -- 172.21.15.33:0/1244687896 >> 172.21.15.33:0/1244687896 conn(0x7f00340fc7d0 msgr2=0x7f00340febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:44.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.107+0000 7f003bfb9640 1 -- 172.21.15.33:0/1244687896 shutdown_connections 2024-09-16T00:34:44.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.107+0000 7f003bfb9640 1 -- 172.21.15.33:0/1244687896 wait complete. 2024-09-16T00:34:44.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.108+0000 7f003bfb9640 1 Processor -- start 2024-09-16T00:34:44.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.108+0000 7f003bfb9640 1 -- start start 2024-09-16T00:34:44.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.108+0000 7f003bfb9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034102d30 0x7f0034100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.108+0000 7f003bfb9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00341036e0 0x7f0034100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.108+0000 7f003bfb9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00341082a0 0x7f0034101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.108+0000 7f003bfb9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00341120a0 con 0x7f0034102d30 2024-09-16T00:34:44.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.108+0000 7f003bfb9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0034111f20 con 0x7f00341082a0 2024-09-16T00:34:44.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.108+0000 7f003bfb9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0034112220 con 0x7f00341036e0 2024-09-16T00:34:44.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.108+0000 7f0039d55640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034102d30 0x7f0034100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0039d55640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034102d30 0x7f0034100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50036/0 (socket says 172.21.15.33:50036) 2024-09-16T00:34:44.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0039d55640 1 -- 172.21.15.33:0/3054842834 learned_addr learned my addr 172.21.15.33:0/3054842834 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:44.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f003a556640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00341082a0 0x7f0034101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0039554640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00341036e0 0x7f0034100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0039d55640 1 -- 172.21.15.33:0/3054842834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00341036e0 msgr2=0x7f0034100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:44.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0039d55640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00341036e0 0x7f0034100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0039d55640 1 -- 172.21.15.33:0/3054842834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00341082a0 msgr2=0x7f0034101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:44.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0039d55640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00341082a0 0x7f0034101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0039d55640 1 -- 172.21.15.33:0/3054842834 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00341aaef0 con 0x7f0034102d30 2024-09-16T00:34:44.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0039554640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00341036e0 0x7f0034100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:34:44.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f003a556640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00341082a0 0x7f0034101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:34:44.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0039d55640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034102d30 0x7f0034100990 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f003002ae00 tx=0x7f0030003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:44.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0026ffd640 1 -- 172.21.15.33:0/3054842834 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0030004040 con 0x7f0034102d30 2024-09-16T00:34:44.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f003bfb9640 1 -- 172.21.15.33:0/3054842834 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00341ab120 con 0x7f0034102d30 2024-09-16T00:34:44.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0026ffd640 1 -- 172.21.15.33:0/3054842834 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00300046c0 con 0x7f0034102d30 2024-09-16T00:34:44.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f0026ffd640 1 -- 172.21.15.33:0/3054842834 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0030046480 con 0x7f0034102d30 2024-09-16T00:34:44.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.109+0000 7f003bfb9640 1 -- 172.21.15.33:0/3054842834 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00341ab660 con 0x7f0034102d30 2024-09-16T00:34:44.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.112+0000 7f0026ffd640 1 -- 172.21.15.33:0/3054842834 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f00300305a0 con 0x7f0034102d30 2024-09-16T00:34:44.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.112+0000 7f003e392740 1 Processor -- start 2024-09-16T00:34:44.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.112+0000 7f003e392740 1 -- start start 2024-09-16T00:34:44.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.112+0000 7f003e392740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9fb40 0x563c00b9e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.112+0000 7f003e392740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00b9ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.112+0000 7f003e392740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 0x563c00bab760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.112+0000 7f003e392740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563c00b042f0 con 0x563c00ba9370 2024-09-16T00:34:44.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.112+0000 7f003e392740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x563c00aabcf0 con 0x563c00b9e5d0 2024-09-16T00:34:44.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.112+0000 7f003e392740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563c00b002b0 con 0x563c00b9fb40 2024-09-16T00:34:44.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f0039554640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 0x563c00bab760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f0039554640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 0x563c00bab760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50038/0 (socket says 172.21.15.33:50038) 2024-09-16T00:34:44.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f0039554640 1 -- 172.21.15.33:0/1829412373 learned_addr learned my addr 172.21.15.33:0/1829412373 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:44.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f003a556640 1 --2- 172.21.15.33:0/1829412373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9fb40 0x563c00b9e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f0026ffd640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00200a3240 0x7f00200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f0039d55640 1 --2- 172.21.15.33:0/1829412373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00b9ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f003a556640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00200a3240 0x7f00200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f0026ffd640 1 -- 172.21.15.33:0/3054842834 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f003002e030 con 0x7f0034102d30 2024-09-16T00:34:44.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f0039554640 1 -- 172.21.15.33:0/1829412373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9fb40 msgr2=0x563c00b9e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:44.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f0039554640 1 --2- 172.21.15.33:0/1829412373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9fb40 0x563c00b9e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f0039554640 1 -- 172.21.15.33:0/1829412373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 msgr2=0x563c00b9ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:44.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f0039554640 1 --2- 172.21.15.33:0/1829412373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00b9ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f0039554640 1 -- 172.21.15.33:0/1829412373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c00a903a0 con 0x563c00ba9370 2024-09-16T00:34:44.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.113+0000 7f0039554640 1 --2- 172.21.15.33:0/1829412373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 0x563c00bab760 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f002800bb50 tx=0x7f002800d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95eea526f581179c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.114+0000 7f003a556640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00200a3240 0x7f00200a5700 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f002c005fd0 tx=0x7f002c004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.114+0000 7f001ffff640 1 -- 172.21.15.33:0/1829412373 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028005220 con 0x563c00ba9370 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.114+0000 7f001ffff640 1 -- 172.21.15.33:0/1829412373 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f002800be50 con 0x563c00ba9370 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.114+0000 7f001ffff640 1 -- 172.21.15.33:0/1829412373 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002800f7a0 con 0x563c00ba9370 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.114+0000 7f003e392740 1 -- 172.21.15.33:0/1829412373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 msgr2=0x563c00bab760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.114+0000 7f003e392740 1 --2- 172.21.15.33:0/1829412373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 0x563c00bab760 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f002800bb50 tx=0x7f002800d710 comp rx=0 tx=0).stop 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.114+0000 7f003e392740 1 -- 172.21.15.33:0/1829412373 shutdown_connections 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.114+0000 7f003e392740 1 --2- 172.21.15.33:0/1829412373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 0x563c00bab760 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.114+0000 7f003e392740 1 --2- 172.21.15.33:0/1829412373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00b9ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.114+0000 7f003e392740 1 --2- 172.21.15.33:0/1829412373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9fb40 0x563c00b9e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.114+0000 7f003e392740 1 -- 172.21.15.33:0/1829412373 >> 172.21.15.33:0/1829412373 conn(0x563c00b67310 msgr2=0x563c00b9ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.115+0000 7f003e392740 1 -- 172.21.15.33:0/1829412373 shutdown_connections 2024-09-16T00:34:44.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.115+0000 7f003e392740 1 -- 172.21.15.33:0/1829412373 wait complete. 2024-09-16T00:34:44.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.115+0000 7f003e392740 1 -- start start 2024-09-16T00:34:44.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.115+0000 7f003e392740 1 Processor -- start 2024-09-16T00:34:44.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.115+0000 7f003e392740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00ba8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.115+0000 7f003e392740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 0x563c00ba5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.115+0000 7f003e392740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00ba9370 0x563c00ba65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f003e392740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563c00bac070 con 0x563c00b9fb40 2024-09-16T00:34:44.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f003e392740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x563c00abf060 con 0x563c00b9e5d0 2024-09-16T00:34:44.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f003e392740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563c00bac1f0 con 0x563c00ba9370 2024-09-16T00:34:44.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f003e392740 10 client.0 _ll_register_callbacks cb 0x563c008128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:44.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f0039554640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 0x563c00ba5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f0039554640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 0x563c00ba5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50050/0 (socket says 172.21.15.33:50050) 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f0039554640 1 -- 172.21.15.33:0/416967190 learned_addr learned my addr 172.21.15.33:0/416967190 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f0039d55640 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00ba8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f003a556640 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00ba9370 0x563c00ba65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f0039554640 1 -- 172.21.15.33:0/416967190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00ba9370 msgr2=0x563c00ba65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f0039554640 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00ba9370 0x563c00ba65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f0039554640 1 -- 172.21.15.33:0/416967190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 msgr2=0x563c00ba8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f0039554640 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00ba8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f0039554640 1 -- 172.21.15.33:0/416967190 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c008753c0 con 0x563c00b9fb40 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f0039554640 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 0x563c00ba5fe0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f0028005720 tx=0x7f0028005540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ea75c3be0c6b62 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f001d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f001d7fa640 1 -- 172.21.15.33:0/416967190 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028025020 con 0x563c00b9fb40 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f001d7fa640 1 -- 172.21.15.33:0/416967190 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f00280046c0 con 0x563c00b9fb40 2024-09-16T00:34:44.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.116+0000 7f003e392740 1 -- 172.21.15.33:0/416967190 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f002800be50 con 0x563c00b9fb40 2024-09-16T00:34:44.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.117+0000 7f001d7fa640 1 -- 172.21.15.33:0/416967190 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0028030050 con 0x563c00b9fb40 2024-09-16T00:34:44.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.117+0000 7f003e392740 10 client.18723 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:44.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.117+0000 7f003e392740 1 -- 172.21.15.33:0/416967190 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563c00bb7ad0 con 0x563c00b9fb40 2024-09-16T00:34:44.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.117+0000 7f001d7fa640 1 -- 172.21.15.33:0/416967190 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f002800be50 con 0x563c00b9fb40 2024-09-16T00:34:44.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.117+0000 7f001d7fa640 1 -- 172.21.15.33:0/416967190 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f002800fc90 con 0x563c00b9fb40 2024-09-16T00:34:44.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.117+0000 7f003e392740 10 client.18723 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:44.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.117+0000 7f003e392740 10 client.18723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:44.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.117+0000 7f003e392740 20 client.18723 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:44.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.117+0000 7f003e392740 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563c00bb8ed0 0x563c00bbb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.117+0000 7f003e392740 4 client.18723 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:34:44.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.117+0000 7f003e392740 1 -- 172.21.15.33:0/416967190 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c00b11860 con 0x563c00bb8ed0 2024-09-16T00:34:44.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.117+0000 7f0039d55640 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563c00bb8ed0 0x563c00bbb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.118+0000 7f0039d55640 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563c00bb8ed0 0x563c00bbb2c0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b63f084def7b18ad server_cookie=e6aaa4377f5f9d49 in_seq=0 out_seq=0 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.118+0000 7f001d7fa640 10 client.18723 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.118+0000 7f001d7fa640 1 -- 172.21.15.33:0/416967190 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563c00b11860 con 0x563c00bb8ed0 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.118+0000 7f001d7fa640 10 client.18723 handle_command_reply: tid=0 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.118+0000 7f003e392740 1 client.18723 shutdown 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.119+0000 7f003e392740 1 -- 172.21.15.33:0/416967190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 msgr2=0x563c00ba5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.119+0000 7f003e392740 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 0x563c00ba5fe0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f0028005720 tx=0x7f0028005540 comp rx=0 tx=0).stop 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.119+0000 7f003e392740 1 -- 172.21.15.33:0/416967190 shutdown_connections 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.119+0000 7f003e392740 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563c00bb8ed0 0x563c00bbb2c0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.119+0000 7f003e392740 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00ba9370 0x563c00ba65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.119+0000 7f003e392740 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 0x563c00ba5fe0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.119+0000 7f001d7fa640 0 client.18723 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:44.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.119+0000 7f003e392740 1 --2- 172.21.15.33:0/416967190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00ba8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.119+0000 7f003e392740 1 -- 172.21.15.33:0/416967190 >> 172.21.15.33:0/416967190 conn(0x563c00b67310 msgr2=0x563c00ba0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:44.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.119+0000 7f003e392740 1 -- 172.21.15.33:0/416967190 shutdown_connections 2024-09-16T00:34:44.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.119+0000 7f003e392740 1 -- 172.21.15.33:0/416967190 wait complete. 2024-09-16T00:34:44.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.119+0000 7f003e392740 20 client.18723 trim_cache size 0 max 16384 2024-09-16T00:34:44.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f003e392740 1 Processor -- start 2024-09-16T00:34:44.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f003e392740 1 -- start start 2024-09-16T00:34:44.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f003e392740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00bb5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f003e392740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9fb40 0x563c00bc0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f003e392740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 0x563c00bc0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f003e392740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00341b7f40 con 0x563c00ba9370 2024-09-16T00:34:44.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f003e392740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f00341b5810 con 0x563c00b9e5d0 2024-09-16T00:34:44.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f003e392740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00341b3660 con 0x563c00b9fb40 2024-09-16T00:34:44.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f003a556640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 0x563c00bc0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f003a556640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 0x563c00bc0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50066/0 (socket says 172.21.15.33:50066) 2024-09-16T00:34:44.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f003a556640 1 -- 172.21.15.33:0/2933963308 learned_addr learned my addr 172.21.15.33:0/2933963308 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:44.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f0039d55640 1 --2- 172.21.15.33:0/2933963308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00bb5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.132+0000 7f0039554640 1 --2- 172.21.15.33:0/2933963308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9fb40 0x563c00bc0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.133+0000 7f003a556640 1 -- 172.21.15.33:0/2933963308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9fb40 msgr2=0x563c00bc0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:44.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.133+0000 7f003a556640 1 --2- 172.21.15.33:0/2933963308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9fb40 0x563c00bc0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.133+0000 7f003a556640 1 -- 172.21.15.33:0/2933963308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 msgr2=0x563c00bb5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:44.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.133+0000 7f003a556640 1 --2- 172.21.15.33:0/2933963308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00bb5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.133+0000 7f003a556640 1 -- 172.21.15.33:0/2933963308 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c00b9bdf0 con 0x563c00ba9370 2024-09-16T00:34:44.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.133+0000 7f0039d55640 1 --2- 172.21.15.33:0/2933963308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00bb5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.133+0000 7f003a556640 1 --2- 172.21.15.33:0/2933963308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 0x563c00bc0e90 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f002c011730 tx=0x7f002c00d5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd95e7b1c3258b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.133+0000 7f001f7fe640 1 -- 172.21.15.33:0/2933963308 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c019070 con 0x563c00ba9370 2024-09-16T00:34:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.133+0000 7f001f7fe640 1 -- 172.21.15.33:0/2933963308 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f002c01d8f0 con 0x563c00ba9370 2024-09-16T00:34:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.133+0000 7f001f7fe640 1 -- 172.21.15.33:0/2933963308 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c01e780 con 0x563c00ba9370 2024-09-16T00:34:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.134+0000 7f003e392740 1 -- 172.21.15.33:0/2933963308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 msgr2=0x563c00bc0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.134+0000 7f003e392740 1 --2- 172.21.15.33:0/2933963308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 0x563c00bc0e90 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f002c011730 tx=0x7f002c00d5c0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.134+0000 7f003e392740 1 -- 172.21.15.33:0/2933963308 shutdown_connections 2024-09-16T00:34:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.134+0000 7f003e392740 1 --2- 172.21.15.33:0/2933963308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00ba9370 0x563c00bc0e90 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.134+0000 7f003e392740 1 --2- 172.21.15.33:0/2933963308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9fb40 0x563c00bc0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.134+0000 7f003e392740 1 --2- 172.21.15.33:0/2933963308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00b9e5d0 0x563c00bb5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.134+0000 7f003e392740 1 -- 172.21.15.33:0/2933963308 >> 172.21.15.33:0/2933963308 conn(0x563c00b67310 msgr2=0x563c00ba12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.134+0000 7f003e392740 1 -- 172.21.15.33:0/2933963308 shutdown_connections 2024-09-16T00:34:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.134+0000 7f003e392740 1 -- 172.21.15.33:0/2933963308 wait complete. 2024-09-16T00:34:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.134+0000 7f003e392740 1 -- start start 2024-09-16T00:34:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.134+0000 7f003e392740 1 Processor -- start 2024-09-16T00:34:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f003e392740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9e5d0 0x563c00badb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f003e392740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 0x563c00bae080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f003e392740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00ba9370 0x563c00bb1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f003e392740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00341b3cd0 con 0x563c00b9fb40 2024-09-16T00:34:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f003e392740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f00341b5140 con 0x563c00ba9370 2024-09-16T00:34:44.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f003e392740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00341b43a0 con 0x563c00b9e5d0 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f0039554640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 0x563c00bae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f003e392740 10 client.0 _ll_register_callbacks cb 0x7f00300b9410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f0039554640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 0x563c00bae080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50070/0 (socket says 172.21.15.33:50070) 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f0039554640 1 -- 172.21.15.33:0/3398353459 learned_addr learned my addr 172.21.15.33:0/3398353459 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f0039d55640 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9e5d0 0x563c00badb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f003a556640 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00ba9370 0x563c00bb1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f0039d55640 1 -- 172.21.15.33:0/3398353459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00ba9370 msgr2=0x563c00bb1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f0039d55640 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00ba9370 0x563c00bb1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f0039d55640 1 -- 172.21.15.33:0/3398353459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 msgr2=0x563c00bae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f0039d55640 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 0x563c00bae080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f0039d55640 1 -- 172.21.15.33:0/3398353459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c00bb17e0 con 0x563c00b9e5d0 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.135+0000 7f0039554640 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 0x563c00bae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:44.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.136+0000 7f0039d55640 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9e5d0 0x563c00badb40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0030009ca0 tx=0x7f00300095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32a8d401094a828d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.136+0000 7f001d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.136+0000 7f001d7fa640 1 -- 172.21.15.33:0/3398353459 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0030030890 con 0x563c00b9e5d0 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.136+0000 7f001d7fa640 1 -- 172.21.15.33:0/3398353459 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f00300b9e20 con 0x563c00b9e5d0 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.136+0000 7f003e392740 1 -- 172.21.15.33:0/3398353459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f002c01d8f0 con 0x563c00b9e5d0 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.136+0000 7f001d7fa640 1 -- 172.21.15.33:0/3398353459 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f00300bd7b0 con 0x563c00b9e5d0 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.136+0000 7f003e392740 10 client.28015 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.136+0000 7f003e392740 1 -- 172.21.15.33:0/3398353459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563c00c243e0 con 0x563c00b9e5d0 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.136+0000 7f001d7fa640 1 -- 172.21.15.33:0/3398353459 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f002c01d8f0 con 0x563c00b9e5d0 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.136+0000 7f001d7fa640 1 -- 172.21.15.33:0/3398353459 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f003004e070 con 0x563c00b9e5d0 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.136+0000 7f003e392740 10 client.28015 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.137+0000 7f003e392740 10 client.28015 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.137+0000 7f003e392740 20 client.28015 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.137+0000 7f003e392740 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563c00c24a60 0x563c00c26e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.137+0000 7f003e392740 4 client.28015 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:34:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.137+0000 7f003e392740 1 -- 172.21.15.33:0/3398353459 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563c00b0bd70 con 0x563c00c24a60 2024-09-16T00:34:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.137+0000 7f0039554640 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563c00c24a60 0x563c00c26e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.137+0000 7f0039554640 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563c00c24a60 0x563c00c26e80 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b45a2e6e47d6bda1 server_cookie=22e42e5659ac0df2 in_seq=0 out_seq=0 2024-09-16T00:34:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.137+0000 7f001d7fa640 10 client.28015 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.137+0000 7f001d7fa640 1 -- 172.21.15.33:0/3398353459 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563c00b0bd70 con 0x563c00c24a60 2024-09-16T00:34:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.137+0000 7f001d7fa640 10 client.28015 handle_command_reply: tid=0 2024-09-16T00:34:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.137+0000 7f003e392740 1 client.28015 shutdown 2024-09-16T00:34:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003e392740 1 -- 172.21.15.33:0/3398353459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9e5d0 msgr2=0x563c00badb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003e392740 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9e5d0 0x563c00badb40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0030009ca0 tx=0x7f00300095c0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003e392740 1 -- 172.21.15.33:0/3398353459 shutdown_connections 2024-09-16T00:34:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003e392740 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x563c00c24a60 0x563c00c26e80 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003e392740 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c00ba9370 0x563c00bb1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003e392740 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c00b9fb40 0x563c00bae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003e392740 1 --2- 172.21.15.33:0/3398353459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c00b9e5d0 0x563c00badb40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f001d7fa640 0 client.28015 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:44.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003e392740 1 -- 172.21.15.33:0/3398353459 >> 172.21.15.33:0/3398353459 conn(0x563c00b67310 msgr2=0x563c00ba7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:44.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003e392740 1 -- 172.21.15.33:0/3398353459 shutdown_connections 2024-09-16T00:34:44.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003e392740 1 -- 172.21.15.33:0/3398353459 wait complete. 2024-09-16T00:34:44.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003e392740 20 client.28015 trim_cache size 0 max 16384 2024-09-16T00:34:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:34:44.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003bfb9640 1 -- 172.21.15.33:0/3054842834 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00200a3240 msgr2=0x7f00200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:44.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003bfb9640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00200a3240 0x7f00200a5700 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f002c005fd0 tx=0x7f002c004730 comp rx=0 tx=0).stop 2024-09-16T00:34:44.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003bfb9640 1 -- 172.21.15.33:0/3054842834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034102d30 msgr2=0x7f0034100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:44.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.138+0000 7f003bfb9640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034102d30 0x7f0034100990 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f003002ae00 tx=0x7f0030003970 comp rx=0 tx=0).stop 2024-09-16T00:34:44.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.139+0000 7f003bfb9640 1 -- 172.21.15.33:0/3054842834 shutdown_connections 2024-09-16T00:34:44.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.139+0000 7f003bfb9640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f00200a3240 0x7f00200a5700 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.139+0000 7f003bfb9640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00341082a0 0x7f0034101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.139+0000 7f003bfb9640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00341036e0 0x7f0034100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.139+0000 7f003bfb9640 1 --2- 172.21.15.33:0/3054842834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034102d30 0x7f0034100990 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:44.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.139+0000 7f003bfb9640 1 -- 172.21.15.33:0/3054842834 >> 172.21.15.33:0/3054842834 conn(0x7f00340fc7d0 msgr2=0x7f0034105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:44.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.139+0000 7f003bfb9640 1 -- 172.21.15.33:0/3054842834 shutdown_connections 2024-09-16T00:34:44.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:44.139+0000 7f003bfb9640 1 -- 172.21.15.33:0/3054842834 wait complete. 2024-09-16T00:34:45.078 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 19.23 MB/sec execute 111 sec latency 14410.886 ms 2024-09-16T00:34:45.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:45 smithi060 ceph-mon[35185]: pgmap v878: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 689 KiB/s wr, 131 op/s 2024-09-16T00:34:45.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:45 smithi033 ceph-mon[31243]: pgmap v878: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 689 KiB/s wr, 131 op/s 2024-09-16T00:34:45.858 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:34:45.859 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:34:45.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:45 smithi083 ceph-mon[35242]: pgmap v878: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 689 KiB/s wr, 131 op/s 2024-09-16T00:34:46.078 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 19.06 MB/sec execute 112 sec latency 15410.943 ms 2024-09-16T00:34:46.130 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:46.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.626+0000 7fb7cfd34640 1 -- 172.21.15.33:0/515634113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c810c180 msgr2=0x7fb7c810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:46.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.626+0000 7fb7cfd34640 1 --2- 172.21.15.33:0/515634113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c810c180 0x7fb7c810e610 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0009ca0 tx=0x7fb7c002a950 comp rx=0 tx=0).stop 2024-09-16T00:34:46.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.626+0000 7fb7cfd34640 1 -- 172.21.15.33:0/515634113 shutdown_connections 2024-09-16T00:34:46.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.626+0000 7fb7cfd34640 1 --2- 172.21.15.33:0/515634113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c810c180 0x7fb7c810e610 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.626+0000 7fb7cfd34640 1 --2- 172.21.15.33:0/515634113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7c8103700 0x7fb7c810bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.626+0000 7fb7cfd34640 1 --2- 172.21.15.33:0/515634113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7c8102de0 0x7fb7c81031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.626+0000 7fb7cfd34640 1 -- 172.21.15.33:0/515634113 >> 172.21.15.33:0/515634113 conn(0x7fb7c80fc7a0 msgr2=0x7fb7c80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:46.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.626+0000 7fb7cfd34640 1 -- 172.21.15.33:0/515634113 shutdown_connections 2024-09-16T00:34:46.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.626+0000 7fb7cfd34640 1 -- 172.21.15.33:0/515634113 wait complete. 2024-09-16T00:34:46.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.627+0000 7fb7cfd34640 1 Processor -- start 2024-09-16T00:34:46.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.627+0000 7fb7cfd34640 1 -- start start 2024-09-16T00:34:46.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.627+0000 7fb7cfd34640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7c8102de0 0x7fb7c81983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.627+0000 7fb7cfd34640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103700 0x7fb7c8198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.627+0000 7fb7cfd34640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7c810c180 0x7fb7c819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.627+0000 7fb7cfd34640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c8111f70 con 0x7fb7c8103700 2024-09-16T00:34:46.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.627+0000 7fb7cfd34640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c8111df0 con 0x7fb7c8102de0 2024-09-16T00:34:46.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.627+0000 7fb7cfd34640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81120f0 con 0x7fb7c810c180 2024-09-16T00:34:46.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7cd2cf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103700 0x7fb7c8198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7cd2cf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103700 0x7fb7c8198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50090/0 (socket says 172.21.15.33:50090) 2024-09-16T00:34:46.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/3386376690 learned_addr learned my addr 172.21.15.33:0/3386376690 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:46.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7ce2d1640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7c810c180 0x7fb7c819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7cdad0640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7c8102de0 0x7fb7c81983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/3386376690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7c810c180 msgr2=0x7fb7c819ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:46.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7c810c180 0x7fb7c819ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/3386376690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7c8102de0 msgr2=0x7fb7c81983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:46.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7c8102de0 0x7fb7c81983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/3386376690 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c819d3a0 con 0x7fb7c8103700 2024-09-16T00:34:46.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103700 0x7fb7c8198930 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800d950 tx=0x7fb7b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:46.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7beffd640 1 -- 172.21.15.33:0/3386376690 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8017070 con 0x7fb7c8103700 2024-09-16T00:34:46.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7cfd34640 1 -- 172.21.15.33:0/3386376690 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c819d690 con 0x7fb7c8103700 2024-09-16T00:34:46.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7beffd640 1 -- 172.21.15.33:0/3386376690 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b8009c30 con 0x7fb7c8103700 2024-09-16T00:34:46.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7beffd640 1 -- 172.21.15.33:0/3386376690 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b800f6b0 con 0x7fb7c8103700 2024-09-16T00:34:46.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.628+0000 7fb7cfd34640 1 -- 172.21.15.33:0/3386376690 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c81a4f70 con 0x7fb7c8103700 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.631+0000 7fb7beffd640 1 -- 172.21.15.33:0/3386376690 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb7b80109d0 con 0x7fb7c8103700 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.631+0000 7fb7d210d740 1 Processor -- start 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.631+0000 7fb7d210d740 1 -- start start 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7d210d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4f9fa20 0x5646e4f9fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7d210d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4fa0340 0x5646e4faec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7d210d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 0x5646e4fb1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7d210d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5646e4f05280 con 0x5646e4faf150 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7d210d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5646e4eacb10 con 0x5646e4fa0340 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7d210d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5646e4f01240 con 0x5646e4f9fa20 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7cd2cf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 0x5646e4fb1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7cdad0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4fa0340 0x5646e4faec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7cd2cf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 0x5646e4fb1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50094/0 (socket says 172.21.15.33:50094) 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/332333780 learned_addr learned my addr 172.21.15.33:0/332333780 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:46.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7ce2d1640 1 --2- 172.21.15.33:0/332333780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4f9fa20 0x5646e4f9fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7beffd640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb7b40a3240 0x7fb7b40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7ce2d1640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb7b40a3240 0x7fb7b40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/332333780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4f9fa20 msgr2=0x5646e4f9fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:46.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/332333780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4f9fa20 0x5646e4f9fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/332333780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4fa0340 msgr2=0x5646e4faec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:46.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/332333780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4fa0340 0x5646e4faec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/332333780 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646e4eb91f0 con 0x5646e4faf150 2024-09-16T00:34:46.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7cdad0640 1 --2- 172.21.15.33:0/332333780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4fa0340 0x5646e4faec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:46.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7beffd640 1 -- 172.21.15.33:0/3386376690 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb7b80cf940 con 0x7fb7c8103700 2024-09-16T00:34:46.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.632+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/332333780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 0x5646e4fb1540 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fb7b809b830 tx=0x7fb7b800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffdfc1f9f6f4d8dd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:46.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.633+0000 7fb7b3fff640 1 -- 172.21.15.33:0/332333780 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8020070 con 0x5646e4faf150 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.633+0000 7fb7ce2d1640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb7b40a3240 0x7fb7b40a5700 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0004ad0 tx=0x7fb7c002a950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.633+0000 7fb7b3fff640 1 -- 172.21.15.33:0/332333780 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b80a46f0 con 0x5646e4faf150 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.633+0000 7fb7b3fff640 1 -- 172.21.15.33:0/332333780 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8017d40 con 0x5646e4faf150 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.633+0000 7fb7d210d740 1 -- 172.21.15.33:0/332333780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 msgr2=0x5646e4fb1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.633+0000 7fb7d210d740 1 --2- 172.21.15.33:0/332333780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 0x5646e4fb1540 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fb7b809b830 tx=0x7fb7b800bf60 comp rx=0 tx=0).stop 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.633+0000 7fb7d210d740 1 -- 172.21.15.33:0/332333780 shutdown_connections 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.633+0000 7fb7d210d740 1 --2- 172.21.15.33:0/332333780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 0x5646e4fb1540 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.633+0000 7fb7d210d740 1 --2- 172.21.15.33:0/332333780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4fa0340 0x5646e4faec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.633+0000 7fb7d210d740 1 --2- 172.21.15.33:0/332333780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4f9fa20 0x5646e4f9fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.633+0000 7fb7d210d740 1 -- 172.21.15.33:0/332333780 >> 172.21.15.33:0/332333780 conn(0x5646e4f9c0f0 msgr2=0x5646e4f9c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.634+0000 7fb7d210d740 1 -- 172.21.15.33:0/332333780 shutdown_connections 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.634+0000 7fb7d210d740 1 -- 172.21.15.33:0/332333780 wait complete. 2024-09-16T00:34:46.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.634+0000 7fb7d210d740 1 -- start start 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.634+0000 7fb7d210d740 1 Processor -- start 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.634+0000 7fb7d210d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4f9fa20 0x5646e4fac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.634+0000 7fb7d210d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4fa0340 0x5646e4fac610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.634+0000 7fb7d210d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 0x5646e4facb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.634+0000 7fb7d210d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5646e4fb1cc0 con 0x5646e4faf150 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.634+0000 7fb7d210d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5646e4ebff10 con 0x5646e4f9fa20 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.634+0000 7fb7d210d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5646e4fb1e40 con 0x5646e4fa0340 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.634+0000 7fb7d210d740 10 client.0 _ll_register_callbacks cb 0x5646e4e88a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7ce2d1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 0x5646e4facb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7ce2d1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 0x5646e4facb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50100/0 (socket says 172.21.15.33:50100) 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7ce2d1640 1 -- 172.21.15.33:0/2437751893 learned_addr learned my addr 172.21.15.33:0/2437751893 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4fa0340 0x5646e4fac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7cdad0640 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4f9fa20 0x5646e4fac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7ce2d1640 1 -- 172.21.15.33:0/2437751893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4fa0340 msgr2=0x5646e4fac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:46.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7ce2d1640 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4fa0340 0x5646e4fac610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7ce2d1640 1 -- 172.21.15.33:0/2437751893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4f9fa20 msgr2=0x5646e4fac0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:46.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7ce2d1640 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4f9fa20 0x5646e4fac0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7ce2d1640 1 -- 172.21.15.33:0/2437751893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646e4c766a0 con 0x5646e4faf150 2024-09-16T00:34:46.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7ce2d1640 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 0x5646e4facb50 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fb7c002f620 tx=0x7fb7c0005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79de0a0cd9555402 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:46.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:46.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7b17fa640 1 -- 172.21.15.33:0/2437751893 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0034040 con 0x5646e4faf150 2024-09-16T00:34:46.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7d210d740 1 -- 172.21.15.33:0/2437751893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7b80a46f0 con 0x5646e4faf150 2024-09-16T00:34:46.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7b17fa640 1 -- 172.21.15.33:0/2437751893 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb7c0043070 con 0x5646e4faf150 2024-09-16T00:34:46.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7b17fa640 1 -- 172.21.15.33:0/2437751893 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb7c0003970 con 0x5646e4faf150 2024-09-16T00:34:46.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.635+0000 7fb7d210d740 10 client.18741 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.636+0000 7fb7d210d740 1 -- 172.21.15.33:0/2437751893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5646e4fbda00 con 0x5646e4faf150 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.636+0000 7fb7b17fa640 1 -- 172.21.15.33:0/2437751893 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b80a46f0 con 0x5646e4faf150 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.636+0000 7fb7b17fa640 1 -- 172.21.15.33:0/2437751893 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb7c00309d0 con 0x5646e4faf150 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.636+0000 7fb7d210d740 10 client.18741 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.636+0000 7fb7d210d740 10 client.18741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.636+0000 7fb7d210d740 20 client.18741 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.636+0000 7fb7d210d740 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646e4fbeec0 0x5646e4fc1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.636+0000 7fb7d210d740 4 client.18741 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.636+0000 7fb7d210d740 1 -- 172.21.15.33:0/2437751893 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646e4f127f0 con 0x5646e4fbeec0 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.636+0000 7fb7cdad0640 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646e4fbeec0 0x5646e4fc1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.637+0000 7fb7cdad0640 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646e4fbeec0 0x5646e4fc1300 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47a837144ad077e1 server_cookie=d482b2f2cf5d7db7 in_seq=0 out_seq=0 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.637+0000 7fb7b17fa640 10 client.18741 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:46.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.637+0000 7fb7b17fa640 1 -- 172.21.15.33:0/2437751893 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5646e4f127f0 con 0x5646e4fbeec0 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.637+0000 7fb7b17fa640 10 client.18741 handle_command_reply: tid=0 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.637+0000 7fb7d210d740 1 client.18741 shutdown 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.638+0000 7fb7d210d740 1 -- 172.21.15.33:0/2437751893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 msgr2=0x5646e4facb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.638+0000 7fb7d210d740 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 0x5646e4facb50 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fb7c002f620 tx=0x7fb7c0005c50 comp rx=0 tx=0).stop 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.638+0000 7fb7d210d740 1 -- 172.21.15.33:0/2437751893 shutdown_connections 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.638+0000 7fb7d210d740 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646e4fbeec0 0x5646e4fc1300 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.638+0000 7fb7d210d740 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4faf150 0x5646e4facb50 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.638+0000 7fb7d210d740 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4fa0340 0x5646e4fac610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.638+0000 7fb7d210d740 1 --2- 172.21.15.33:0/2437751893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4f9fa20 0x5646e4fac0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.638+0000 7fb7b17fa640 0 client.18741 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.638+0000 7fb7d210d740 1 -- 172.21.15.33:0/2437751893 >> 172.21.15.33:0/2437751893 conn(0x5646e4f9c0f0 msgr2=0x5646e4fa1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.638+0000 7fb7d210d740 1 -- 172.21.15.33:0/2437751893 shutdown_connections 2024-09-16T00:34:46.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.638+0000 7fb7d210d740 1 -- 172.21.15.33:0/2437751893 wait complete. 2024-09-16T00:34:46.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.638+0000 7fb7d210d740 20 client.18741 trim_cache size 0 max 16384 2024-09-16T00:34:46.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7d210d740 1 Processor -- start 2024-09-16T00:34:46.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7d210d740 1 -- start start 2024-09-16T00:34:46.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7d210d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4f9fa20 0x5646e4fac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7d210d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 0x5646e4fc6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7d210d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4faf150 0x5646e4fc7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7d210d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81af770 con 0x5646e4fa0340 2024-09-16T00:34:46.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7d210d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81ad040 con 0x5646e4f9fa20 2024-09-16T00:34:46.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7d210d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81aae90 con 0x5646e4faf150 2024-09-16T00:34:46.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7cd2cf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 0x5646e4fc6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7cd2cf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 0x5646e4fc6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50110/0 (socket says 172.21.15.33:50110) 2024-09-16T00:34:46.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/2068762610 learned_addr learned my addr 172.21.15.33:0/2068762610 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:46.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7ce2d1640 1 --2- 172.21.15.33:0/2068762610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4faf150 0x5646e4fc7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7cdad0640 1 --2- 172.21.15.33:0/2068762610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4f9fa20 0x5646e4fac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/2068762610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4faf150 msgr2=0x5646e4fc7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:46.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/2068762610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4faf150 0x5646e4fc7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.656+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/2068762610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4f9fa20 msgr2=0x5646e4fac9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:46.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.657+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/2068762610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4f9fa20 0x5646e4fac9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.657+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/2068762610 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646e4faa620 con 0x5646e4fa0340 2024-09-16T00:34:46.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.657+0000 7fb7cdad0640 1 --2- 172.21.15.33:0/2068762610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4f9fa20 0x5646e4fac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.657+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/2068762610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 0x5646e4fc6c20 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009610 tx=0x7fb7b8097720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3484a5fa8f47e314 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.657+0000 7fb7b37fe640 1 -- 172.21.15.33:0/2068762610 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8020070 con 0x5646e4fa0340 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.657+0000 7fb7b37fe640 1 -- 172.21.15.33:0/2068762610 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b80cd070 con 0x5646e4fa0340 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.657+0000 7fb7b37fe640 1 -- 172.21.15.33:0/2068762610 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b80a4260 con 0x5646e4fa0340 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.658+0000 7fb7d210d740 1 -- 172.21.15.33:0/2068762610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 msgr2=0x5646e4fc6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.658+0000 7fb7d210d740 1 --2- 172.21.15.33:0/2068762610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 0x5646e4fc6c20 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009610 tx=0x7fb7b8097720 comp rx=0 tx=0).stop 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.658+0000 7fb7d210d740 1 -- 172.21.15.33:0/2068762610 shutdown_connections 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.658+0000 7fb7d210d740 1 --2- 172.21.15.33:0/2068762610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4faf150 0x5646e4fc7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.658+0000 7fb7d210d740 1 --2- 172.21.15.33:0/2068762610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 0x5646e4fc6c20 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.658+0000 7fb7d210d740 1 --2- 172.21.15.33:0/2068762610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4f9fa20 0x5646e4fac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.658+0000 7fb7d210d740 1 -- 172.21.15.33:0/2068762610 >> 172.21.15.33:0/2068762610 conn(0x5646e4f9c0f0 msgr2=0x5646e4f9b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.658+0000 7fb7d210d740 1 -- 172.21.15.33:0/2068762610 shutdown_connections 2024-09-16T00:34:46.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.658+0000 7fb7d210d740 1 -- 172.21.15.33:0/2068762610 wait complete. 2024-09-16T00:34:46.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.658+0000 7fb7d210d740 1 -- start start 2024-09-16T00:34:46.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.658+0000 7fb7d210d740 1 Processor -- start 2024-09-16T00:34:46.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7d210d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4f9fa20 0x5646e4fb34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7d210d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 0x5646e4fb3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7d210d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4faf150 0x5646e5028bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7d210d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81ab500 con 0x5646e4fa0340 2024-09-16T00:34:46.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7d210d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81ac970 con 0x5646e4faf150 2024-09-16T00:34:46.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7d210d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81abbd0 con 0x5646e4f9fa20 2024-09-16T00:34:46.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7d210d740 10 client.0 _ll_register_callbacks cb 0x7fb7c002fef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:46.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7cd2cf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 0x5646e4fb3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7cd2cf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 0x5646e4fb3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50116/0 (socket says 172.21.15.33:50116) 2024-09-16T00:34:46.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/4070798697 learned_addr learned my addr 172.21.15.33:0/4070798697 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7ce2d1640 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4faf150 0x5646e5028bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7cdad0640 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4f9fa20 0x5646e4fb34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/4070798697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4f9fa20 msgr2=0x5646e4fb34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4f9fa20 0x5646e4fb34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/4070798697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4faf150 msgr2=0x5646e5028bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4faf150 0x5646e5028bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7cd2cf640 1 -- 172.21.15.33:0/4070798697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646e5029130 con 0x5646e4fa0340 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7cd2cf640 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 0x5646e4fb3a20 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80a41c0 tx=0x7fb7b8005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20131ff78ee3883e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7b17fa640 1 -- 172.21.15.33:0/4070798697 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8020070 con 0x5646e4fa0340 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7b17fa640 1 -- 172.21.15.33:0/4070798697 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb7b80a4260 con 0x5646e4fa0340 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.659+0000 7fb7d210d740 1 -- 172.21.15.33:0/4070798697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7b80cd070 con 0x5646e4fa0340 2024-09-16T00:34:46.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.660+0000 7fb7b17fa640 1 -- 172.21.15.33:0/4070798697 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb7b80a44e0 con 0x5646e4fa0340 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.660+0000 7fb7d210d740 10 client.18747 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.660+0000 7fb7d210d740 1 -- 172.21.15.33:0/4070798697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5646e502a340 con 0x5646e4fa0340 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.660+0000 7fb7b17fa640 1 -- 172.21.15.33:0/4070798697 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b80cd070 con 0x5646e4fa0340 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.660+0000 7fb7b17fa640 1 -- 172.21.15.33:0/4070798697 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb7b8097880 con 0x5646e4fa0340 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.660+0000 7fb7d210d740 10 client.18747 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.660+0000 7fb7d210d740 10 client.18747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.660+0000 7fb7d210d740 20 client.18747 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.660+0000 7fb7d210d740 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646e502a5c0 0x5646e502c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.660+0000 7fb7d210d740 4 client.18747 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.660+0000 7fb7d210d740 1 -- 172.21.15.33:0/4070798697 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5646e4f0cd00 con 0x5646e502a5c0 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.660+0000 7fb7cdad0640 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646e502a5c0 0x5646e502c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.661+0000 7fb7cdad0640 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646e502a5c0 0x5646e502c9b0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc8867924c71bd05 server_cookie=82e34c42a5bc6486 in_seq=0 out_seq=0 2024-09-16T00:34:46.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.661+0000 7fb7b17fa640 10 client.18747 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:46.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7b17fa640 1 -- 172.21.15.33:0/4070798697 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5646e4f0cd00 con 0x5646e502a5c0 2024-09-16T00:34:46.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7b17fa640 10 client.18747 handle_command_reply: tid=0 2024-09-16T00:34:46.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7d210d740 1 client.18747 shutdown 2024-09-16T00:34:46.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7d210d740 1 -- 172.21.15.33:0/4070798697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 msgr2=0x5646e4fb3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:46.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7d210d740 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 0x5646e4fb3a20 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80a41c0 tx=0x7fb7b8005f70 comp rx=0 tx=0).stop 2024-09-16T00:34:46.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7d210d740 1 -- 172.21.15.33:0/4070798697 shutdown_connections 2024-09-16T00:34:46.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7d210d740 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5646e502a5c0 0x5646e502c9b0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7d210d740 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5646e4faf150 0x5646e5028bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7d210d740 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5646e4fa0340 0x5646e4fb3a20 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7b17fa640 0 client.18747 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:46.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7d210d740 1 --2- 172.21.15.33:0/4070798697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5646e4f9fa20 0x5646e4fb34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7d210d740 1 -- 172.21.15.33:0/4070798697 >> 172.21.15.33:0/4070798697 conn(0x5646e4f9c0f0 msgr2=0x5646e4fa14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:46.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7d210d740 1 -- 172.21.15.33:0/4070798697 shutdown_connections 2024-09-16T00:34:46.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.662+0000 7fb7d210d740 1 -- 172.21.15.33:0/4070798697 wait complete. 2024-09-16T00:34:46.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7d210d740 20 client.18747 trim_cache size 0 max 16384 2024-09-16T00:34:46.673 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:34:46.673 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:34:46.673 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "783a5370-2649-4307-8ec1-62caca7967b8", 2024-09-16T00:34:46.673 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:34:46.673 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:34:46.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7cfd34640 1 -- 172.21.15.33:0/3386376690 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb7b40a3240 msgr2=0x7fb7b40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:46.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7cfd34640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb7b40a3240 0x7fb7b40a5700 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0004ad0 tx=0x7fb7c002a950 comp rx=0 tx=0).stop 2024-09-16T00:34:46.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7cfd34640 1 -- 172.21.15.33:0/3386376690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103700 msgr2=0x7fb7c8198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:46.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7cfd34640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103700 0x7fb7c8198930 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800d950 tx=0x7fb7b800de10 comp rx=0 tx=0).stop 2024-09-16T00:34:46.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7cfd34640 1 -- 172.21.15.33:0/3386376690 shutdown_connections 2024-09-16T00:34:46.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7cfd34640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb7b40a3240 0x7fb7b40a5700 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7cfd34640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7c810c180 0x7fb7c819ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7cfd34640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103700 0x7fb7c8198930 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7cfd34640 1 --2- 172.21.15.33:0/3386376690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7c8102de0 0x7fb7c81983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:46.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7cfd34640 1 -- 172.21.15.33:0/3386376690 >> 172.21.15.33:0/3386376690 conn(0x7fb7c80fc7a0 msgr2=0x7fb7c80fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:46.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7cfd34640 1 -- 172.21.15.33:0/3386376690 shutdown_connections 2024-09-16T00:34:46.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:46.663+0000 7fb7cfd34640 1 -- 172.21.15.33:0/3386376690 wait complete. 2024-09-16T00:34:47.078 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 18.89 MB/sec execute 113 sec latency 16410.993 ms 2024-09-16T00:34:47.383 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:34:47.640 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:47 smithi060 ceph-mon[35185]: pgmap v879: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 62 KiB/s wr, 18 op/s 2024-09-16T00:34:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:47 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:47 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:47.761 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:47 smithi033 ceph-mon[31243]: pgmap v879: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 62 KiB/s wr, 18 op/s 2024-09-16T00:34:47.761 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:47 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:47.761 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:47 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:47 smithi083 ceph-mon[35242]: pgmap v879: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 62 KiB/s wr, 18 op/s 2024-09-16T00:34:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:47 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:47 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.078 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 18.73 MB/sec execute 114 sec latency 17411.088 ms 2024-09-16T00:34:48.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.175+0000 7fba147ad640 1 -- 172.21.15.33:0/4063423105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c106f10 msgr2=0x7fba0c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:48.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.175+0000 7fba147ad640 1 --2- 172.21.15.33:0/4063423105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c106f10 0x7fba0c10ea90 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fba04009ca0 tx=0x7fba0402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.175+0000 7fba147ad640 1 -- 172.21.15.33:0/4063423105 shutdown_connections 2024-09-16T00:34:48.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.175+0000 7fba147ad640 1 --2- 172.21.15.33:0/4063423105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c106f10 0x7fba0c10ea90 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.175+0000 7fba147ad640 1 --2- 172.21.15.33:0/4063423105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba0c102790 0x7fba0c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.175+0000 7fba147ad640 1 --2- 172.21.15.33:0/4063423105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba0c101de0 0x7fba0c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.175+0000 7fba147ad640 1 -- 172.21.15.33:0/4063423105 >> 172.21.15.33:0/4063423105 conn(0x7fba0c0fd8c0 msgr2=0x7fba0c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:48.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.175+0000 7fba147ad640 1 -- 172.21.15.33:0/4063423105 shutdown_connections 2024-09-16T00:34:48.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.175+0000 7fba147ad640 1 -- 172.21.15.33:0/4063423105 wait complete. 2024-09-16T00:34:48.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba147ad640 1 Processor -- start 2024-09-16T00:34:48.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba147ad640 1 -- start start 2024-09-16T00:34:48.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba147ad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba0c101de0 0x7fba0c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba147ad640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba0c102790 0x7fba0c198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba147ad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c106f10 0x7fba0c19ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba147ad640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c111f70 con 0x7fba0c106f10 2024-09-16T00:34:48.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba147ad640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c111df0 con 0x7fba0c101de0 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba147ad640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1120f0 con 0x7fba0c102790 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba12d4a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c106f10 0x7fba0c19ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba12d4a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c106f10 0x7fba0c19ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50132/0 (socket says 172.21.15.33:50132) 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba12d4a640 1 -- 172.21.15.33:0/2435649112 learned_addr learned my addr 172.21.15.33:0/2435649112 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba12549640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba0c101de0 0x7fba0c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba11d48640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba0c102790 0x7fba0c198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba12d4a640 1 -- 172.21.15.33:0/2435649112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba0c102790 msgr2=0x7fba0c198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba12d4a640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba0c102790 0x7fba0c198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba12d4a640 1 -- 172.21.15.33:0/2435649112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba0c101de0 msgr2=0x7fba0c1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.176+0000 7fba12d4a640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba0c101de0 0x7fba0c1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.177+0000 7fba12d4a640 1 -- 172.21.15.33:0/2435649112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0c19d3a0 con 0x7fba0c106f10 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.177+0000 7fba12d4a640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c106f10 0x7fba0c19ccc0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fba04031a20 tx=0x7fba04031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.177+0000 7fba037fe640 1 -- 172.21.15.33:0/2435649112 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba04003c30 con 0x7fba0c106f10 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.177+0000 7fba147ad640 1 -- 172.21.15.33:0/2435649112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0c19d630 con 0x7fba0c106f10 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.177+0000 7fba037fe640 1 -- 172.21.15.33:0/2435649112 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba04003dd0 con 0x7fba0c106f10 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.177+0000 7fba037fe640 1 -- 172.21.15.33:0/2435649112 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0402fcf0 con 0x7fba0c106f10 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.177+0000 7fba147ad640 1 -- 172.21.15.33:0/2435649112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba0c1a4f70 con 0x7fba0c106f10 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.179+0000 7fba037fe640 1 -- 172.21.15.33:0/2435649112 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fba04004050 con 0x7fba0c106f10 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba16b86740 1 Processor -- start 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba16b86740 1 -- start start 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba16b86740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbd54a0 0x55789cbd5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba16b86740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbe0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba16b86740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 0x55789cbe2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba16b86740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55789cb3b2e0 con 0x55789cbe0680 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba16b86740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55789cae2ce0 con 0x55789cbd5dc0 2024-09-16T00:34:48.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba16b86740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55789cb372a0 con 0x55789cbd54a0 2024-09-16T00:34:48.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba037fe640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb9f80a3240 0x7fb9f80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba11d48640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 0x55789cbe2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba11d48640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 0x55789cbe2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50146/0 (socket says 172.21.15.33:50146) 2024-09-16T00:34:48.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba11d48640 1 -- 172.21.15.33:0/2145866626 learned_addr learned my addr 172.21.15.33:0/2145866626 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:48.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba11d48640 1 --2- 172.21.15.33:0/2145866626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbd54a0 0x55789cbd5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba12d4a640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb9f80a3240 0x7fb9f80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba12549640 1 --2- 172.21.15.33:0/2145866626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbe0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.180+0000 7fba037fe640 1 -- 172.21.15.33:0/2435649112 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fba040f1030 con 0x7fba0c106f10 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.181+0000 7fba11d48640 1 -- 172.21.15.33:0/2145866626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbd54a0 msgr2=0x55789cbd5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.181+0000 7fba11d48640 1 --2- 172.21.15.33:0/2145866626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbd54a0 0x55789cbd5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.181+0000 7fba11d48640 1 -- 172.21.15.33:0/2145866626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 msgr2=0x55789cbe0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.181+0000 7fba11d48640 1 --2- 172.21.15.33:0/2145866626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbe0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.181+0000 7fba11d48640 1 -- 172.21.15.33:0/2145866626 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55789cac7440 con 0x55789cbe0680 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.181+0000 7fba11d48640 1 --2- 172.21.15.33:0/2145866626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 0x55789cbe2a70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc012720 tx=0x7fb9fc00b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb754fcdb7f6db0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.181+0000 7fba12d4a640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb9f80a3240 0x7fb9f80a5700 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fba040b96c0 tx=0x7fba040b2740 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.181+0000 7fba00ff9640 1 -- 172.21.15.33:0/2145866626 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc012ca0 con 0x55789cbe0680 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.181+0000 7fba00ff9640 1 -- 172.21.15.33:0/2145866626 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9fc012e40 con 0x55789cbe0680 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.181+0000 7fba00ff9640 1 -- 172.21.15.33:0/2145866626 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc0049e0 con 0x55789cbe0680 2024-09-16T00:34:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.182+0000 7fba16b86740 1 -- 172.21.15.33:0/2145866626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 msgr2=0x55789cbe2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:48.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.182+0000 7fba16b86740 1 --2- 172.21.15.33:0/2145866626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 0x55789cbe2a70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc012720 tx=0x7fb9fc00b6e0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.182+0000 7fba16b86740 1 -- 172.21.15.33:0/2145866626 shutdown_connections 2024-09-16T00:34:48.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.182+0000 7fba16b86740 1 --2- 172.21.15.33:0/2145866626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 0x55789cbe2a70 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.182+0000 7fba16b86740 1 --2- 172.21.15.33:0/2145866626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbe0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.182+0000 7fba16b86740 1 --2- 172.21.15.33:0/2145866626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbd54a0 0x55789cbd5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.182+0000 7fba16b86740 1 -- 172.21.15.33:0/2145866626 >> 172.21.15.33:0/2145866626 conn(0x55789cbd1a30 msgr2=0x55789cbd1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:48.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.182+0000 7fba16b86740 1 -- 172.21.15.33:0/2145866626 shutdown_connections 2024-09-16T00:34:48.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.182+0000 7fba16b86740 1 -- 172.21.15.33:0/2145866626 wait complete. 2024-09-16T00:34:48.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.182+0000 7fba16b86740 1 -- start start 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.182+0000 7fba16b86740 1 Processor -- start 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba16b86740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd54a0 0x55789cbdffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba16b86740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbd5dc0 0x55789cbdd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba16b86740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 0x55789cbdd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba16b86740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55789cbe3250 con 0x55789cbe0680 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba16b86740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55789caf6050 con 0x55789cbd54a0 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba16b86740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55789cbe33d0 con 0x55789cbd5dc0 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba16b86740 10 client.0 _ll_register_callbacks cb 0x55789cb36160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba11d48640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 0x55789cbdd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba11d48640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 0x55789cbdd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50160/0 (socket says 172.21.15.33:50160) 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba11d48640 1 -- 172.21.15.33:0/2227072854 learned_addr learned my addr 172.21.15.33:0/2227072854 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba11d48640 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd54a0 0x55789cbdffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba12549640 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbd5dc0 0x55789cbdd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba11d48640 1 -- 172.21.15.33:0/2227072854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbd5dc0 msgr2=0x55789cbdd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba11d48640 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbd5dc0 0x55789cbdd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba11d48640 1 -- 172.21.15.33:0/2227072854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd54a0 msgr2=0x55789cbdffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba11d48640 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd54a0 0x55789cbdffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba11d48640 1 -- 172.21.15.33:0/2227072854 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55789c8ac3c0 con 0x55789cbe0680 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba12549640 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbd5dc0 0x55789cbdd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fba11d48640 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 0x55789cbdd7a0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc001eb0 tx=0x7fb9fc014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92557c940b78622 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fb9f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fb9f5ffb640 1 -- 172.21.15.33:0/2227072854 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc010740 con 0x55789cbe0680 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.183+0000 7fb9f5ffb640 1 -- 172.21.15.33:0/2227072854 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb9fc0108e0 con 0x55789cbe0680 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fba16b86740 1 -- 172.21.15.33:0/2227072854 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55789cb48850 con 0x55789cbe0680 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fb9f5ffb640 1 -- 172.21.15.33:0/2227072854 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb9fc033050 con 0x55789cbe0680 2024-09-16T00:34:48.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fba16b86740 10 client.18759 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:48.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fba16b86740 1 -- 172.21.15.33:0/2227072854 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55789cbeec30 con 0x55789cbe0680 2024-09-16T00:34:48.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fb9f5ffb640 1 -- 172.21.15.33:0/2227072854 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55789cb48850 con 0x55789cbe0680 2024-09-16T00:34:48.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fb9f5ffb640 1 -- 172.21.15.33:0/2227072854 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb9fc02d4d0 con 0x55789cbe0680 2024-09-16T00:34:48.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fba16b86740 10 client.18759 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:48.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fba16b86740 10 client.18759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:48.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fba16b86740 20 client.18759 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:48.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fba16b86740 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55789cbeff80 0x55789cbf2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fba16b86740 4 client.18759 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:34:48.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fba16b86740 1 -- 172.21.15.33:0/2227072854 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55789cb42d60 con 0x55789cbeff80 2024-09-16T00:34:48.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.184+0000 7fba12549640 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55789cbeff80 0x55789cbf2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.185+0000 7fb9f5ffb640 10 client.18759 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:48.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.185+0000 7fba12549640 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55789cbeff80 0x55789cbf2370 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad10e1222a18388a server_cookie=d2dafa3b015beed4 in_seq=0 out_seq=0 2024-09-16T00:34:48.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.185+0000 7fb9f5ffb640 1 -- 172.21.15.33:0/2227072854 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55789cb42d60 con 0x55789cbeff80 2024-09-16T00:34:48.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.185+0000 7fb9f5ffb640 10 client.18759 handle_command_reply: tid=0 2024-09-16T00:34:48.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.185+0000 7fba16b86740 1 client.18759 shutdown 2024-09-16T00:34:48.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.186+0000 7fba16b86740 1 -- 172.21.15.33:0/2227072854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 msgr2=0x55789cbdd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:48.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.186+0000 7fba16b86740 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 0x55789cbdd7a0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc001eb0 tx=0x7fb9fc014040 comp rx=0 tx=0).stop 2024-09-16T00:34:48.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.186+0000 7fba16b86740 1 -- 172.21.15.33:0/2227072854 shutdown_connections 2024-09-16T00:34:48.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.186+0000 7fba16b86740 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55789cbeff80 0x55789cbf2370 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.186+0000 7fba16b86740 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbe0680 0x55789cbdd7a0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.186+0000 7fba16b86740 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbd5dc0 0x55789cbdd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.186+0000 7fba16b86740 1 --2- 172.21.15.33:0/2227072854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd54a0 0x55789cbdffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.186+0000 7fba16b86740 1 -- 172.21.15.33:0/2227072854 >> 172.21.15.33:0/2227072854 conn(0x55789cbd1a30 msgr2=0x55789cbd7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:48.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.186+0000 7fb9f5ffb640 0 client.18759 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:48.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.186+0000 7fba16b86740 1 -- 172.21.15.33:0/2227072854 shutdown_connections 2024-09-16T00:34:48.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.186+0000 7fba16b86740 1 -- 172.21.15.33:0/2227072854 wait complete. 2024-09-16T00:34:48.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.186+0000 7fba16b86740 20 client.18759 trim_cache size 0 max 16384 2024-09-16T00:34:48.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.203+0000 7fba16b86740 1 Processor -- start 2024-09-16T00:34:48.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.203+0000 7fba16b86740 1 -- start start 2024-09-16T00:34:48.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.203+0000 7fba16b86740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 0x55789cbec4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.203+0000 7fba16b86740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbd8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.203+0000 7fba16b86740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbe0680 0x55789cbf7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.203+0000 7fba16b86740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1b1990 con 0x55789cbd54a0 2024-09-16T00:34:48.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.203+0000 7fba16b86740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1af260 con 0x55789cbd5dc0 2024-09-16T00:34:48.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.203+0000 7fba16b86740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1ad0b0 con 0x55789cbe0680 2024-09-16T00:34:48.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.203+0000 7fba11d48640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 0x55789cbec4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.203+0000 7fba11d48640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 0x55789cbec4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50174/0 (socket says 172.21.15.33:50174) 2024-09-16T00:34:48.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.203+0000 7fba11d48640 1 -- 172.21.15.33:0/4004735453 learned_addr learned my addr 172.21.15.33:0/4004735453 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:48.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.204+0000 7fba12549640 1 --2- 172.21.15.33:0/4004735453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbd8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.204+0000 7fba11d48640 1 -- 172.21.15.33:0/4004735453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbe0680 msgr2=0x55789cbf7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:48.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.204+0000 7fba11d48640 1 --2- 172.21.15.33:0/4004735453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbe0680 0x55789cbf7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.204+0000 7fba11d48640 1 -- 172.21.15.33:0/4004735453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 msgr2=0x55789cbd8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:48.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.204+0000 7fba11d48640 1 --2- 172.21.15.33:0/4004735453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbd8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.204+0000 7fba11d48640 1 -- 172.21.15.33:0/4004735453 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55789cbe2ed0 con 0x55789cbd54a0 2024-09-16T00:34:48.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.204+0000 7fba12549640 1 --2- 172.21.15.33:0/4004735453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbd8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:34:48.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.204+0000 7fba11d48640 1 --2- 172.21.15.33:0/4004735453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 0x55789cbec4e0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc005630 tx=0x7fb9fc00bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca7d4fb8a83a59d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:48.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.204+0000 7fb9f7fff640 1 -- 172.21.15.33:0/4004735453 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc00ba30 con 0x55789cbd54a0 2024-09-16T00:34:48.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.204+0000 7fb9f7fff640 1 -- 172.21.15.33:0/4004735453 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9fc016070 con 0x55789cbd54a0 2024-09-16T00:34:48.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.204+0000 7fb9f7fff640 1 -- 172.21.15.33:0/4004735453 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc010740 con 0x55789cbd54a0 2024-09-16T00:34:48.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 -- 172.21.15.33:0/4004735453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 msgr2=0x55789cbec4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:48.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 --2- 172.21.15.33:0/4004735453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 0x55789cbec4e0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc005630 tx=0x7fb9fc00bf50 comp rx=0 tx=0).stop 2024-09-16T00:34:48.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 -- 172.21.15.33:0/4004735453 shutdown_connections 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 --2- 172.21.15.33:0/4004735453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbe0680 0x55789cbf7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 --2- 172.21.15.33:0/4004735453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbd8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 --2- 172.21.15.33:0/4004735453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 0x55789cbec4e0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 -- 172.21.15.33:0/4004735453 >> 172.21.15.33:0/4004735453 conn(0x55789cbd1a30 msgr2=0x55789cbdf2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 -- 172.21.15.33:0/4004735453 shutdown_connections 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 -- 172.21.15.33:0/4004735453 wait complete. 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 -- start start 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 Processor -- start 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 0x55789cbe8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbd9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbe0680 0x55789cbda250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1ad720 con 0x55789cbd54a0 2024-09-16T00:34:48.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1aeb90 con 0x55789cbd5dc0 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.205+0000 7fba16b86740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1addf0 con 0x55789cbe0680 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba16b86740 10 client.0 _ll_register_callbacks cb 0x7fb9fc00bdf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba11d48640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 0x55789cbe8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba11d48640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 0x55789cbe8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50180/0 (socket says 172.21.15.33:50180) 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba11d48640 1 -- 172.21.15.33:0/3632388788 learned_addr learned my addr 172.21.15.33:0/3632388788 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba12549640 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbd9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba11d48640 1 -- 172.21.15.33:0/3632388788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbe0680 msgr2=0x55789cbda250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba11d48640 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbe0680 0x55789cbda250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba11d48640 1 -- 172.21.15.33:0/3632388788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 msgr2=0x55789cbd9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba11d48640 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbd9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba11d48640 1 -- 172.21.15.33:0/3632388788 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55789cbda7c0 con 0x55789cbd54a0 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba12549640 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbd9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:48.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba11d48640 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 0x55789cbe8ad0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc02dd40 tx=0x7fb9fc0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ceb5916ca9ee4f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fb9f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fb9f5ffb640 1 -- 172.21.15.33:0/3632388788 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc00ba30 con 0x55789cbd54a0 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fb9f5ffb640 1 -- 172.21.15.33:0/3632388788 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb9fc02d2c0 con 0x55789cbd54a0 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba16b86740 1 -- 172.21.15.33:0/3632388788 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9fc016070 con 0x55789cbd54a0 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fb9f5ffb640 1 -- 172.21.15.33:0/3632388788 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb9fc045050 con 0x55789cbd54a0 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.206+0000 7fba16b86740 10 client.18765 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.207+0000 7fba16b86740 1 -- 172.21.15.33:0/3632388788 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55789cc5a7d0 con 0x55789cbd54a0 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.207+0000 7fb9f5ffb640 1 -- 172.21.15.33:0/3632388788 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9fc016070 con 0x55789cbd54a0 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.207+0000 7fb9f5ffb640 1 -- 172.21.15.33:0/3632388788 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb9fc0035f0 con 0x55789cbd54a0 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.207+0000 7fba16b86740 10 client.18765 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.207+0000 7fba16b86740 10 client.18765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.207+0000 7fba16b86740 20 client.18765 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:48.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.207+0000 7fba16b86740 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55789cc5ae00 0x55789cc5d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:48.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.207+0000 7fba16b86740 4 client.18765 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:34:48.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.207+0000 7fba16b86740 1 -- 172.21.15.33:0/3632388788 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55789cbddb30 con 0x55789cc5ae00 2024-09-16T00:34:48.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.207+0000 7fba12549640 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55789cc5ae00 0x55789cc5d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:48.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba12549640 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55789cc5ae00 0x55789cc5d220 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59b3c52eebdf6f32 server_cookie=ded3edb7dbeb543e in_seq=0 out_seq=0 2024-09-16T00:34:48.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fb9f5ffb640 10 client.18765 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:48.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fb9f5ffb640 1 -- 172.21.15.33:0/3632388788 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55789cbddb30 con 0x55789cc5ae00 2024-09-16T00:34:48.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fb9f5ffb640 10 client.18765 handle_command_reply: tid=0 2024-09-16T00:34:48.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba16b86740 1 client.18765 shutdown 2024-09-16T00:34:48.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba16b86740 1 -- 172.21.15.33:0/3632388788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 msgr2=0x55789cbe8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:48.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba16b86740 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 0x55789cbe8ad0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc02dd40 tx=0x7fb9fc0056d0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba16b86740 1 -- 172.21.15.33:0/3632388788 shutdown_connections 2024-09-16T00:34:48.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba16b86740 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55789cc5ae00 0x55789cc5d220 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fb9f5ffb640 0 client.18765 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:48.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba16b86740 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55789cbe0680 0x55789cbda250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba16b86740 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55789cbd5dc0 0x55789cbd9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba16b86740 1 --2- 172.21.15.33:0/3632388788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55789cbd54a0 0x55789cbe8ad0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba16b86740 1 -- 172.21.15.33:0/3632388788 >> 172.21.15.33:0/3632388788 conn(0x55789cbd1a30 msgr2=0x55789cbdeac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:48.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba16b86740 1 -- 172.21.15.33:0/3632388788 shutdown_connections 2024-09-16T00:34:48.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba16b86740 1 -- 172.21.15.33:0/3632388788 wait complete. 2024-09-16T00:34:48.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.208+0000 7fba16b86740 20 client.18765 trim_cache size 0 max 16384 2024-09-16T00:34:48.217 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:34:48.217 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:34:48.218 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:34:48.218 INFO:teuthology.orchestra.run.smithi033.stdout: "783a5370-2649-4307-8ec1-62caca7967b8": { 2024-09-16T00:34:48.218 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:34:48.218 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "783a5370-2649-4307-8ec1-62caca7967b8", 2024-09-16T00:34:48.218 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:34:48.218 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:48.218 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:48.218 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:34:48.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.209+0000 7fba147ad640 1 -- 172.21.15.33:0/2435649112 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb9f80a3240 msgr2=0x7fb9f80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:48.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.209+0000 7fba147ad640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb9f80a3240 0x7fb9f80a5700 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fba040b96c0 tx=0x7fba040b2740 comp rx=0 tx=0).stop 2024-09-16T00:34:48.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.209+0000 7fba147ad640 1 -- 172.21.15.33:0/2435649112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c106f10 msgr2=0x7fba0c19ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:48.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.209+0000 7fba147ad640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c106f10 0x7fba0c19ccc0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fba04031a20 tx=0x7fba04031a50 comp rx=0 tx=0).stop 2024-09-16T00:34:48.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.209+0000 7fba147ad640 1 -- 172.21.15.33:0/2435649112 shutdown_connections 2024-09-16T00:34:48.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.209+0000 7fba147ad640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb9f80a3240 0x7fb9f80a5700 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.209+0000 7fba147ad640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c106f10 0x7fba0c19ccc0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.209+0000 7fba147ad640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba0c102790 0x7fba0c198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.209+0000 7fba147ad640 1 --2- 172.21.15.33:0/2435649112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba0c101de0 0x7fba0c1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:48.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.209+0000 7fba147ad640 1 -- 172.21.15.33:0/2435649112 >> 172.21.15.33:0/2435649112 conn(0x7fba0c0fd8c0 msgr2=0x7fba0c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:48.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.209+0000 7fba147ad640 1 -- 172.21.15.33:0/2435649112 shutdown_connections 2024-09-16T00:34:48.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:48.209+0000 7fba147ad640 1 -- 172.21.15.33:0/2435649112 wait complete. 2024-09-16T00:34:48.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:34:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:48.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:34:48.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:34:48.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:48.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:48.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:48.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:48.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:48.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:48.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:34:48.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:34:48.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:48.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:48.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:48.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:48.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:48.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:48.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:48.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:34:48.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:34:49.078 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 18.56 MB/sec execute 115 sec latency 18411.182 ms 2024-09-16T00:34:49.250 INFO:tasks.cephfs.filesystem:scrub status for tag:783a5370-2649-4307-8ec1-62caca7967b8 - {'path': '/', 'tag': '783a5370-2649-4307-8ec1-62caca7967b8', 'options': 'recursive,force'} 2024-09-16T00:34:49.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:49 smithi033 ceph-mon[31243]: pgmap v880: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.9 KiB/s wr, 11 op/s 2024-09-16T00:34:49.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:49 smithi060 ceph-mon[35185]: pgmap v880: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.9 KiB/s wr, 11 op/s 2024-09-16T00:34:49.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:49 smithi083 ceph-mon[35242]: pgmap v880: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.9 KiB/s wr, 11 op/s 2024-09-16T00:34:50.078 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 18.40 MB/sec execute 116 sec latency 19411.261 ms 2024-09-16T00:34:50.474 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:34:50.750 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:51.078 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 18.25 MB/sec execute 117 sec latency 20411.320 ms 2024-09-16T00:34:51.714 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:51 smithi033 ceph-mon[31243]: pgmap v881: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-09-16T00:34:51.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.713+0000 7fe0c0aca640 1 -- 172.21.15.33:0/532108161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0bc101de0 msgr2=0x7fe0bc1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:51.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.713+0000 7fe0c0aca640 1 --2- 172.21.15.33:0/532108161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0bc101de0 0x7fe0bc1021c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac009900 tx=0x7fe0ac02a410 comp rx=0 tx=0).stop 2024-09-16T00:34:51.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.713+0000 7fe0c0aca640 1 -- 172.21.15.33:0/532108161 shutdown_connections 2024-09-16T00:34:51.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.713+0000 7fe0c0aca640 1 --2- 172.21.15.33:0/532108161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0bc106f10 0x7fe0bc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:51.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.713+0000 7fe0c0aca640 1 --2- 172.21.15.33:0/532108161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0bc102790 0x7fe0bc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:51.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.713+0000 7fe0c0aca640 1 --2- 172.21.15.33:0/532108161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0bc101de0 0x7fe0bc1021c0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:51.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.713+0000 7fe0c0aca640 1 -- 172.21.15.33:0/532108161 >> 172.21.15.33:0/532108161 conn(0x7fe0bc0fd8c0 msgr2=0x7fe0bc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:51.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.714+0000 7fe0c0aca640 1 -- 172.21.15.33:0/532108161 shutdown_connections 2024-09-16T00:34:51.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.714+0000 7fe0c0aca640 1 -- 172.21.15.33:0/532108161 wait complete. 2024-09-16T00:34:51.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.714+0000 7fe0c0aca640 1 Processor -- start 2024-09-16T00:34:51.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0c0aca640 1 -- start start 2024-09-16T00:34:51.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0c0aca640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0bc101de0 0x7fe0bc1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:51.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0c0aca640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0bc102790 0x7fe0bc198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:51.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0c0aca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0bc106f10 0x7fe0bc19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:51.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0c0aca640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe0bc111f70 con 0x7fe0bc101de0 2024-09-16T00:34:51.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0c0aca640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe0bc111df0 con 0x7fe0bc106f10 2024-09-16T00:34:51.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0c0aca640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe0bc1120f0 con 0x7fe0bc102790 2024-09-16T00:34:51.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0ba59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0bc101de0 0x7fe0bc1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:51.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0ba59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0bc101de0 0x7fe0bc1984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33504/0 (socket says 172.21.15.33:33504) 2024-09-16T00:34:51.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0ba59c640 1 -- 172.21.15.33:0/3022953731 learned_addr learned my addr 172.21.15.33:0/3022953731 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:51.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0bad9d640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0bc106f10 0x7fe0bc19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:51.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0b9d9b640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0bc102790 0x7fe0bc198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:51.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0ba59c640 1 -- 172.21.15.33:0/3022953731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0bc102790 msgr2=0x7fe0bc198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:51.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0ba59c640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0bc102790 0x7fe0bc198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:51.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0ba59c640 1 -- 172.21.15.33:0/3022953731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0bc106f10 msgr2=0x7fe0bc19cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:51.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0ba59c640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0bc106f10 0x7fe0bc19cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:51.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0ba59c640 1 -- 172.21.15.33:0/3022953731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0bc19d480 con 0x7fe0bc101de0 2024-09-16T00:34:51.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.715+0000 7fe0ba59c640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0bc101de0 0x7fe0bc1984d0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac005ab0 tx=0x7fe0ac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:51.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.716+0000 7fe0ab7fe640 1 -- 172.21.15.33:0/3022953731 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac03f030 con 0x7fe0bc101de0 2024-09-16T00:34:51.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.716+0000 7fe0c0aca640 1 -- 172.21.15.33:0/3022953731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0bc19d710 con 0x7fe0bc101de0 2024-09-16T00:34:51.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.716+0000 7fe0ab7fe640 1 -- 172.21.15.33:0/3022953731 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0ac003ce0 con 0x7fe0bc101de0 2024-09-16T00:34:51.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.716+0000 7fe0ab7fe640 1 -- 172.21.15.33:0/3022953731 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac035070 con 0x7fe0bc101de0 2024-09-16T00:34:51.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.716+0000 7fe0c0aca640 1 -- 172.21.15.33:0/3022953731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0bc1a5050 con 0x7fe0bc101de0 2024-09-16T00:34:51.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.717+0000 7fe0c0aca640 1 -- 172.21.15.33:0/3022953731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0bc1021c0 con 0x7fe0bc101de0 2024-09-16T00:34:51.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.718+0000 7fe0ab7fe640 1 -- 172.21.15.33:0/3022953731 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe0ac042030 con 0x7fe0bc101de0 2024-09-16T00:34:51.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.719+0000 7fe0ab7fe640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0a40a32c0 0x7fe0a40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:51.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.719+0000 7fe0b9d9b640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0a40a32c0 0x7fe0a40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:51.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.719+0000 7fe0ab7fe640 1 -- 172.21.15.33:0/3022953731 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe0ac0f0680 con 0x7fe0bc101de0 2024-09-16T00:34:51.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.719+0000 7fe0b9d9b640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0a40a32c0 0x7fe0a40a5780 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0ff840 tx=0x7fe0b000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:51.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.723+0000 7fe0ab7fe640 1 -- 172.21.15.33:0/3022953731 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe0ac0bcec0 con 0x7fe0bc101de0 2024-09-16T00:34:51.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:51 smithi060 ceph-mon[35185]: pgmap v881: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-09-16T00:34:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:51 smithi083 ceph-mon[35242]: pgmap v881: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-09-16T00:34:51.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.934+0000 7fe0c0aca640 1 -- 172.21.15.33:0/3022953731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0bc10a0f0 con 0x7fe0bc101de0 2024-09-16T00:34:51.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.934+0000 7fe0ab7fe640 1 -- 172.21.15.33:0/3022953731 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7fe0ac0bc7c0 con 0x7fe0bc101de0 2024-09-16T00:34:51.935 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:34:51.935 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:34:51.935 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:34:51.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.936+0000 7fe0c0aca640 1 -- 172.21.15.33:0/3022953731 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0a40a32c0 msgr2=0x7fe0a40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:51.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.936+0000 7fe0c0aca640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0a40a32c0 0x7fe0a40a5780 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0ff840 tx=0x7fe0b000a4c0 comp rx=0 tx=0).stop 2024-09-16T00:34:51.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.936+0000 7fe0c0aca640 1 -- 172.21.15.33:0/3022953731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0bc101de0 msgr2=0x7fe0bc1984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:51.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.936+0000 7fe0c0aca640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0bc101de0 0x7fe0bc1984d0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac005ab0 tx=0x7fe0ac0042f0 comp rx=0 tx=0).stop 2024-09-16T00:34:51.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.936+0000 7fe0c0aca640 1 -- 172.21.15.33:0/3022953731 shutdown_connections 2024-09-16T00:34:51.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.936+0000 7fe0c0aca640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe0a40a32c0 0x7fe0a40a5780 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:51.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.936+0000 7fe0c0aca640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0bc106f10 0x7fe0bc19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:51.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.936+0000 7fe0c0aca640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0bc102790 0x7fe0bc198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:51.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.936+0000 7fe0c0aca640 1 --2- 172.21.15.33:0/3022953731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0bc101de0 0x7fe0bc1984d0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:51.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.936+0000 7fe0c0aca640 1 -- 172.21.15.33:0/3022953731 >> 172.21.15.33:0/3022953731 conn(0x7fe0bc0fd8c0 msgr2=0x7fe0bc0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:51.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.937+0000 7fe0c0aca640 1 -- 172.21.15.33:0/3022953731 shutdown_connections 2024-09-16T00:34:51.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:51.937+0000 7fe0c0aca640 1 -- 172.21.15.33:0/3022953731 wait complete. 2024-09-16T00:34:52.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 18.09 MB/sec execute 118 sec latency 21411.424 ms 2024-09-16T00:34:52.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:52 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3022953731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:34:52.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:52 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3022953731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:34:52.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:52 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3022953731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:34:52.978 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 17f371aa --release --await' 2024-09-16T00:34:52.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 17f371aa --release --await 2024-09-16T00:34:53.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83503 17.94 MB/sec execute 119 sec latency 22411.769 ms 2024-09-16T00:34:53.458 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:53.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:53 smithi060 ceph-mon[35185]: pgmap v882: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-09-16T00:34:53.740 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:53 smithi033 ceph-mon[31243]: pgmap v882: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-09-16T00:34:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:53 smithi083 ceph-mon[35242]: pgmap v882: 129 pgs: 129 active+clean; 561 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-09-16T00:34:53.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.979+0000 7f5848ff9640 1 -- 172.21.15.33:0/3187233810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5844106a80 msgr2=0x7f5844106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:53.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.979+0000 7f5848ff9640 1 --2- 172.21.15.33:0/3187233810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5844106a80 0x7f5844106e60 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f5834009930 tx=0x7f583402a420 comp rx=0 tx=0).stop 2024-09-16T00:34:53.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.979+0000 7f5848ff9640 1 -- 172.21.15.33:0/3187233810 shutdown_connections 2024-09-16T00:34:53.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.979+0000 7f5848ff9640 1 --2- 172.21.15.33:0/3187233810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5844101420 0x7f584410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:53.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.979+0000 7f5848ff9640 1 --2- 172.21.15.33:0/3187233810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5844100a80 0x7f5844100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:53.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.979+0000 7f5848ff9640 1 --2- 172.21.15.33:0/3187233810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5844106a80 0x7f5844106e60 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:53.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.979+0000 7f5848ff9640 1 -- 172.21.15.33:0/3187233810 >> 172.21.15.33:0/3187233810 conn(0x7f58440fc7a0 msgr2=0x7f58440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:53.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.979+0000 7f5848ff9640 1 -- 172.21.15.33:0/3187233810 shutdown_connections 2024-09-16T00:34:53.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.979+0000 7f5848ff9640 1 -- 172.21.15.33:0/3187233810 wait complete. 2024-09-16T00:34:53.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.980+0000 7f5848ff9640 1 Processor -- start 2024-09-16T00:34:53.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.980+0000 7f5848ff9640 1 -- start start 2024-09-16T00:34:53.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.980+0000 7f5848ff9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5844100a80 0x7f5844198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:53.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.980+0000 7f5848ff9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5844101420 0x7f5844198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:53.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.980+0000 7f5848ff9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5844106a80 0x7f584419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:53.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.980+0000 7f5848ff9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5844110e40 con 0x7f5844100a80 2024-09-16T00:34:53.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.980+0000 7f5848ff9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5844110cc0 con 0x7f5844101420 2024-09-16T00:34:53.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.980+0000 7f5848ff9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5844110fc0 con 0x7f5844106a80 2024-09-16T00:34:53.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f584259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5844100a80 0x7f5844198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:53.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f584259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5844100a80 0x7f5844198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33518/0 (socket says 172.21.15.33:33518) 2024-09-16T00:34:53.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f584259c640 1 -- 172.21.15.33:0/3541495500 learned_addr learned my addr 172.21.15.33:0/3541495500 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:53.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f5841d9b640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5844101420 0x7f5844198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:53.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f5842d9d640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5844106a80 0x7f584419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:53.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f584259c640 1 -- 172.21.15.33:0/3541495500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5844106a80 msgr2=0x7f584419cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:53.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f584259c640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5844106a80 0x7f584419cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:53.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f584259c640 1 -- 172.21.15.33:0/3541495500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5844101420 msgr2=0x7f5844198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:53.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f584259c640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5844101420 0x7f5844198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:53.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f584259c640 1 -- 172.21.15.33:0/3541495500 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584419d400 con 0x7f5844100a80 2024-09-16T00:34:53.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f584259c640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5844100a80 0x7f5844198450 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f5834009230 tx=0x7f5834004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:53.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f58337fe640 1 -- 172.21.15.33:0/3541495500 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583403f030 con 0x7f5844100a80 2024-09-16T00:34:53.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f58337fe640 1 -- 172.21.15.33:0/3541495500 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5834004440 con 0x7f5844100a80 2024-09-16T00:34:53.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f5848ff9640 1 -- 172.21.15.33:0/3541495500 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584419d690 con 0x7f5844100a80 2024-09-16T00:34:53.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f58337fe640 1 -- 172.21.15.33:0/3541495500 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5834035070 con 0x7f5844100a80 2024-09-16T00:34:53.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.981+0000 7f5848ff9640 1 -- 172.21.15.33:0/3541495500 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58441a4fd0 con 0x7f5844100a80 2024-09-16T00:34:53.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.984+0000 7f58337fe640 1 -- 172.21.15.33:0/3541495500 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5834042030 con 0x7f5844100a80 2024-09-16T00:34:53.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.984+0000 7f584b3d2740 1 Processor -- start 2024-09-16T00:34:53.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.984+0000 7f584b3d2740 1 -- start start 2024-09-16T00:34:53.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.984+0000 7f584b3d2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb89440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:53.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.984+0000 7f584b3d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fb93d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.984+0000 7f584b3d2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fb96630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.984+0000 7f584b3d2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56273faee620 con 0x56273fb94240 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.984+0000 7f584b3d2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56273fa95e50 con 0x56273fb89060 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.984+0000 7f584b3d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56273faea5e0 con 0x56273fb89980 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f58337fe640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f582c0a3240 0x7f582c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5841d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fb96630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5841d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fb96630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33520/0 (socket says 172.21.15.33:33520) 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5841d9b640 1 -- 172.21.15.33:0/2359690113 learned_addr learned my addr 172.21.15.33:0/2359690113 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5842d9d640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f582c0a3240 0x7f582c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f58337fe640 1 -- 172.21.15.33:0/3541495500 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f58340f01c0 con 0x7f5844100a80 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f584259c640 1 --2- 172.21.15.33:0/2359690113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fb93d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5842d9d640 1 --2- 172.21.15.33:0/2359690113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb89440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:53.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5841d9b640 1 -- 172.21.15.33:0/2359690113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 msgr2=0x56273fb93d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5841d9b640 1 --2- 172.21.15.33:0/2359690113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fb93d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5841d9b640 1 -- 172.21.15.33:0/2359690113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 msgr2=0x56273fb89440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5841d9b640 1 --2- 172.21.15.33:0/2359690113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb89440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5841d9b640 1 -- 172.21.15.33:0/2359690113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56273fa86d40 con 0x56273fb94240 2024-09-16T00:34:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f584259c640 1 --2- 172.21.15.33:0/2359690113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fb93d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:34:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5842d9d640 1 --2- 172.21.15.33:0/2359690113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb89440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:34:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5841d9b640 1 --2- 172.21.15.33:0/2359690113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fb96630 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f5838009c20 tx=0x7f58380117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbccad34a5cf5036 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5842d9d640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f582c0a3240 0x7f582c0a5700 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f583c009fd0 tx=0x7f583c0091e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5830ff9640 1 -- 172.21.15.33:0/2359690113 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5838012450 con 0x56273fb94240 2024-09-16T00:34:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5830ff9640 1 -- 172.21.15.33:0/2359690113 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5838012ad0 con 0x56273fb94240 2024-09-16T00:34:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.985+0000 7f5830ff9640 1 -- 172.21.15.33:0/2359690113 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583800fbf0 con 0x56273fb94240 2024-09-16T00:34:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.986+0000 7f584b3d2740 1 -- 172.21.15.33:0/2359690113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 msgr2=0x56273fb96630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.986+0000 7f584b3d2740 1 --2- 172.21.15.33:0/2359690113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fb96630 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f5838009c20 tx=0x7f58380117b0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.986+0000 7f584b3d2740 1 -- 172.21.15.33:0/2359690113 shutdown_connections 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.986+0000 7f584b3d2740 1 --2- 172.21.15.33:0/2359690113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fb96630 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.986+0000 7f584b3d2740 1 --2- 172.21.15.33:0/2359690113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fb93d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.986+0000 7f584b3d2740 1 --2- 172.21.15.33:0/2359690113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb89440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.986+0000 7f584b3d2740 1 -- 172.21.15.33:0/2359690113 >> 172.21.15.33:0/2359690113 conn(0x56273fb855f0 msgr2=0x56273fb859d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.986+0000 7f584b3d2740 1 -- 172.21.15.33:0/2359690113 shutdown_connections 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.986+0000 7f584b3d2740 1 -- 172.21.15.33:0/2359690113 wait complete. 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f584b3d2740 1 -- start start 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f584b3d2740 1 Processor -- start 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f584b3d2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb91060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f584b3d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fb915a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:54.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f584b3d2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fb91ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f584b3d2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56273fb96e10 con 0x56273fb94240 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f584b3d2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56273fa76f10 con 0x56273fb89060 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f584b3d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56273fb96f90 con 0x56273fb89980 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f584b3d2740 10 client.0 _ll_register_callbacks cb 0x56273fa9adb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f5842d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fb91ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f5842d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fb91ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33536/0 (socket says 172.21.15.33:33536) 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f5842d9d640 1 -- 172.21.15.33:0/3870067947 learned_addr learned my addr 172.21.15.33:0/3870067947 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f5841d9b640 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fb915a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f584259c640 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb91060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f5842d9d640 1 -- 172.21.15.33:0/3870067947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 msgr2=0x56273fb915a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f5842d9d640 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fb915a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f5842d9d640 1 -- 172.21.15.33:0/3870067947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 msgr2=0x56273fb91060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:54.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f5842d9d640 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb91060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f5842d9d640 1 -- 172.21.15.33:0/3870067947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56273f7fd580 con 0x56273fb94240 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.987+0000 7f5841d9b640 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fb915a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f5842d9d640 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fb91ae0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f583c012670 tx=0x7f583c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b560a160e09e2b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f5829ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f5829ffb640 1 -- 172.21.15.33:0/3870067947 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c02c060 con 0x56273fb94240 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f5829ffb640 1 -- 172.21.15.33:0/3870067947 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f583c0045c0 con 0x56273fb94240 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f584b3d2740 1 -- 172.21.15.33:0/3870067947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5838012ad0 con 0x56273fb94240 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f5829ffb640 1 -- 172.21.15.33:0/3870067947 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f583c033050 con 0x56273fb94240 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f584b3d2740 10 client.18783 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f584b3d2740 1 -- 172.21.15.33:0/3870067947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56273f8c7960 con 0x56273fb94240 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f5829ffb640 1 -- 172.21.15.33:0/3870067947 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5838012ad0 con 0x56273fb94240 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f5829ffb640 1 -- 172.21.15.33:0/3870067947 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f583c026030 con 0x56273fb94240 2024-09-16T00:34:54.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f584b3d2740 10 client.18783 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f584b3d2740 10 client.18783 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.988+0000 7f584b3d2740 20 client.18783 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.989+0000 7f584b3d2740 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56273fba3cb0 0x56273fba6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.989+0000 7f584b3d2740 4 client.18783 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.989+0000 7f584b3d2740 1 -- 172.21.15.33:0/3870067947 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56273fafbb90 con 0x56273fba3cb0 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.989+0000 7f584259c640 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56273fba3cb0 0x56273fba6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.989+0000 7f584259c640 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56273fba3cb0 0x56273fba6100 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8145ac4a057d5f5 server_cookie=ffd933609a17559d in_seq=0 out_seq=0 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.989+0000 7f5829ffb640 10 client.18783 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.990+0000 7f5829ffb640 1 -- 172.21.15.33:0/3870067947 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56273fafbb90 con 0x56273fba3cb0 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.990+0000 7f5829ffb640 10 client.18783 handle_command_reply: tid=0 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.990+0000 7f584b3d2740 1 client.18783 shutdown 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.991+0000 7f584b3d2740 1 -- 172.21.15.33:0/3870067947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 msgr2=0x56273fb91ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.991+0000 7f584b3d2740 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fb91ae0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f583c012670 tx=0x7f583c004a40 comp rx=0 tx=0).stop 2024-09-16T00:34:54.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.991+0000 7f584b3d2740 1 -- 172.21.15.33:0/3870067947 shutdown_connections 2024-09-16T00:34:54.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.991+0000 7f584b3d2740 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56273fba3cb0 0x56273fba6100 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.991+0000 7f584b3d2740 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fb91ae0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.991+0000 7f584b3d2740 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fb915a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.991+0000 7f5829ffb640 0 client.18783 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:54.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.991+0000 7f584b3d2740 1 --2- 172.21.15.33:0/3870067947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb91060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.991+0000 7f584b3d2740 1 -- 172.21.15.33:0/3870067947 >> 172.21.15.33:0/3870067947 conn(0x56273fb855f0 msgr2=0x56273fb8b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:54.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.991+0000 7f584b3d2740 1 -- 172.21.15.33:0/3870067947 shutdown_connections 2024-09-16T00:34:54.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.991+0000 7f584b3d2740 1 -- 172.21.15.33:0/3870067947 wait complete. 2024-09-16T00:34:54.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:53.991+0000 7f584b3d2740 20 client.18783 trim_cache size 0 max 16384 2024-09-16T00:34:54.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.010+0000 7f584b3d2740 1 Processor -- start 2024-09-16T00:34:54.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.010+0000 7f584b3d2740 1 -- start start 2024-09-16T00:34:54.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.010+0000 7f584b3d2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb8b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:54.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.010+0000 7f584b3d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fbab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:54.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.010+0000 7f584b3d2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fbabf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:54.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.010+0000 7f584b3d2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58441b1990 con 0x56273fb94240 2024-09-16T00:34:54.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.010+0000 7f584b3d2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f58441af260 con 0x56273fb89060 2024-09-16T00:34:54.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.010+0000 7f584b3d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f58441ad0b0 con 0x56273fb89980 2024-09-16T00:34:54.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f5842d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fbabf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:54.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f5842d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fbabf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33544/0 (socket says 172.21.15.33:33544) 2024-09-16T00:34:54.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f5842d9d640 1 -- 172.21.15.33:0/3694463270 learned_addr learned my addr 172.21.15.33:0/3694463270 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:54.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f5841d9b640 1 --2- 172.21.15.33:0/3694463270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fbab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:54.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f584259c640 1 --2- 172.21.15.33:0/3694463270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb8b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:54.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f5842d9d640 1 -- 172.21.15.33:0/3694463270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 msgr2=0x56273fbab9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:54.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f5842d9d640 1 --2- 172.21.15.33:0/3694463270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fbab9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f5842d9d640 1 -- 172.21.15.33:0/3694463270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 msgr2=0x56273fb8b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:54.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f5842d9d640 1 --2- 172.21.15.33:0/3694463270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb8b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f5842d9d640 1 -- 172.21.15.33:0/3694463270 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56273fb85420 con 0x56273fb94240 2024-09-16T00:34:54.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f5842d9d640 1 --2- 172.21.15.33:0/3694463270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fbabf10 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f583c005c60 tx=0x7f583c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e451d121e15342 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:54.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f582bfff640 1 -- 172.21.15.33:0/3694463270 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c004cb0 con 0x56273fb94240 2024-09-16T00:34:54.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f582bfff640 1 -- 172.21.15.33:0/3694463270 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f583c004e50 con 0x56273fb94240 2024-09-16T00:34:54.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.011+0000 7f582bfff640 1 -- 172.21.15.33:0/3694463270 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c01d450 con 0x56273fb94240 2024-09-16T00:34:54.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.012+0000 7f584b3d2740 1 -- 172.21.15.33:0/3694463270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 msgr2=0x56273fbabf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:54.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.012+0000 7f584b3d2740 1 --2- 172.21.15.33:0/3694463270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fbabf10 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f583c005c60 tx=0x7f583c009da0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.012+0000 7f584b3d2740 1 -- 172.21.15.33:0/3694463270 shutdown_connections 2024-09-16T00:34:54.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.012+0000 7f584b3d2740 1 --2- 172.21.15.33:0/3694463270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb94240 0x56273fbabf10 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.012+0000 7f584b3d2740 1 --2- 172.21.15.33:0/3694463270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb89980 0x56273fbab9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.012+0000 7f584b3d2740 1 --2- 172.21.15.33:0/3694463270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb8b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.012+0000 7f584b3d2740 1 -- 172.21.15.33:0/3694463270 >> 172.21.15.33:0/3694463270 conn(0x56273fb855f0 msgr2=0x56273fb8aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.012+0000 7f584b3d2740 1 -- 172.21.15.33:0/3694463270 shutdown_connections 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.012+0000 7f584b3d2740 1 -- 172.21.15.33:0/3694463270 wait complete. 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.012+0000 7f584b3d2740 1 -- start start 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.012+0000 7f584b3d2740 1 Processor -- start 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f584b3d2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb9c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f584b3d2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb89980 0x56273fb9c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f584b3d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb94240 0x56273fb991b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f584b3d2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58441ad720 con 0x56273fb89980 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f584b3d2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f58441aeb90 con 0x56273fb89060 2024-09-16T00:34:54.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f584b3d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f58441addf0 con 0x56273fb94240 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f584b3d2740 10 client.0 _ll_register_callbacks cb 0x7f583c003c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f5841d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb89980 0x56273fb9c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f5841d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb89980 0x56273fb9c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33554/0 (socket says 172.21.15.33:33554) 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f5841d9b640 1 -- 172.21.15.33:0/2527220568 learned_addr learned my addr 172.21.15.33:0/2527220568 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f5842d9d640 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb94240 0x56273fb991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f584259c640 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb9c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f5841d9b640 1 -- 172.21.15.33:0/2527220568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb94240 msgr2=0x56273fb991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f5841d9b640 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb94240 0x56273fb991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f5841d9b640 1 -- 172.21.15.33:0/2527220568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 msgr2=0x56273fb9c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f5841d9b640 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb9c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f5841d9b640 1 -- 172.21.15.33:0/2527220568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56273fb86280 con 0x56273fb89980 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f584259c640 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb9c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:34:54.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f5841d9b640 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb89980 0x56273fb9c720 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x56273fb86970 tx=0x7f58380053b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafa528fa49f6134 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:54.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f5829ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:54.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.013+0000 7f5829ffb640 1 -- 172.21.15.33:0/2527220568 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5838029070 con 0x56273fb89980 2024-09-16T00:34:54.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f5829ffb640 1 -- 172.21.15.33:0/2527220568 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f583800fa00 con 0x56273fb89980 2024-09-16T00:34:54.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f584b3d2740 1 -- 172.21.15.33:0/2527220568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f583c004e50 con 0x56273fb89980 2024-09-16T00:34:54.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f5829ffb640 1 -- 172.21.15.33:0/2527220568 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5838037050 con 0x56273fb89980 2024-09-16T00:34:54.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f584b3d2740 10 client.18789 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:54.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f584b3d2740 1 -- 172.21.15.33:0/2527220568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56273fc0f0f0 con 0x56273fb89980 2024-09-16T00:34:54.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f5829ffb640 1 -- 172.21.15.33:0/2527220568 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f583c004e50 con 0x56273fb89980 2024-09-16T00:34:54.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f5829ffb640 1 -- 172.21.15.33:0/2527220568 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f583802e460 con 0x56273fb89980 2024-09-16T00:34:54.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f584b3d2740 10 client.18789 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:54.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f584b3d2740 10 client.18789 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:34:54.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f584b3d2740 20 client.18789 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:54.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f584b3d2740 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56273fc0f5f0 0x56273fc11a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:54.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f584b3d2740 4 client.18789 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "17f371aa", "release": true, "await": true}] 2024-09-16T00:34:54.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.014+0000 7f584b3d2740 1 -- 172.21.15.33:0/2527220568 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "17f371aa", "release": true, "await": true}) -- 0x56273faf60a0 con 0x56273fc0f5f0 2024-09-16T00:34:54.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.015+0000 7f584259c640 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56273fc0f5f0 0x56273fc11a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:54.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.015+0000 7f584259c640 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56273fc0f5f0 0x56273fc11a10 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e01e0a0b27d64034 server_cookie=7d3cf87266c4314e in_seq=0 out_seq=0 2024-09-16T00:34:54.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.015+0000 7f5829ffb640 10 client.18789 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:54.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83506 17.80 MB/sec execute 120 sec latency 23411.811 ms 2024-09-16T00:34:54.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.080+0000 7f5829ffb640 1 -- 172.21.15.33:0/2527220568 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56273faf60a0 con 0x56273fc0f5f0 2024-09-16T00:34:54.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.080+0000 7f5829ffb640 10 client.18789 handle_command_reply: tid=0 2024-09-16T00:34:54.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f584b3d2740 1 client.18789 shutdown 2024-09-16T00:34:54.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f584b3d2740 1 -- 172.21.15.33:0/2527220568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb89980 msgr2=0x56273fb9c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:54.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f584b3d2740 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb89980 0x56273fb9c720 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x56273fb86970 tx=0x7f58380053b0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f584b3d2740 1 -- 172.21.15.33:0/2527220568 shutdown_connections 2024-09-16T00:34:54.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f584b3d2740 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56273fc0f5f0 0x56273fc11a10 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f584b3d2740 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56273fb94240 0x56273fb991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f584b3d2740 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56273fb89980 0x56273fb9c720 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f584b3d2740 1 --2- 172.21.15.33:0/2527220568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56273fb89060 0x56273fb9c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f584b3d2740 1 -- 172.21.15.33:0/2527220568 >> 172.21.15.33:0/2527220568 conn(0x56273fb855f0 msgr2=0x56273fb91840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:54.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f5829ffb640 0 client.18789 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:54.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f584b3d2740 1 -- 172.21.15.33:0/2527220568 shutdown_connections 2024-09-16T00:34:54.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f584b3d2740 1 -- 172.21.15.33:0/2527220568 wait complete. 2024-09-16T00:34:54.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.081+0000 7f584b3d2740 20 client.18789 trim_cache size 0 max 16384 2024-09-16T00:34:54.084 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:34:54.085 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:34:54.085 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:34:54.085 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 64, 2024-09-16T00:34:54.085 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:34:54.085 INFO:teuthology.orchestra.run.smithi033.stdout: "17f371aa": { 2024-09-16T00:34:54.085 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 64, 2024-09-16T00:34:54.085 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:34:54.085 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:34:54.085 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:34:54.085 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:34:54.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 142.9, 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:54.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:54.087 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:34:54.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.082+0000 7f5848ff9640 1 -- 172.21.15.33:0/3541495500 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f582c0a3240 msgr2=0x7f582c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:54.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.082+0000 7f5848ff9640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f582c0a3240 0x7f582c0a5700 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f583c009fd0 tx=0x7f583c0091e0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.082+0000 7f5848ff9640 1 -- 172.21.15.33:0/3541495500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5844100a80 msgr2=0x7f5844198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.082+0000 7f5848ff9640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5844100a80 0x7f5844198450 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f5834009230 tx=0x7f5834004040 comp rx=0 tx=0).stop 2024-09-16T00:34:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.082+0000 7f5848ff9640 1 -- 172.21.15.33:0/3541495500 shutdown_connections 2024-09-16T00:34:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.082+0000 7f5848ff9640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f582c0a3240 0x7f582c0a5700 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.082+0000 7f5848ff9640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5844106a80 0x7f584419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.082+0000 7f5848ff9640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5844101420 0x7f5844198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.082+0000 7f5848ff9640 1 --2- 172.21.15.33:0/3541495500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5844100a80 0x7f5844198450 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.082+0000 7f5848ff9640 1 -- 172.21.15.33:0/3541495500 >> 172.21.15.33:0/3541495500 conn(0x7f58440fc7a0 msgr2=0x7f584406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.082+0000 7f5848ff9640 1 -- 172.21.15.33:0/3541495500 shutdown_connections 2024-09-16T00:34:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:54.082+0000 7f5848ff9640 1 -- 172.21.15.33:0/3541495500 wait complete. 2024-09-16T00:34:55.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 83979 17.82 MB/sec execute 121 sec latency 23412.397 ms 2024-09-16T00:34:55.250 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '17f371aa', total seconds elapsed: 27.7 2024-09-16T00:34:55.250 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.95 2024-09-16T00:34:55.251 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:34:55.578 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:55.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:55 smithi060 ceph-mon[35185]: pgmap v883: 129 pgs: 129 active+clean; 567 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 558 KiB/s wr, 19 op/s 2024-09-16T00:34:55.749 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:55 smithi033 ceph-mon[31243]: pgmap v883: 129 pgs: 129 active+clean; 567 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 558 KiB/s wr, 19 op/s 2024-09-16T00:34:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:55 smithi083 ceph-mon[35242]: pgmap v883: 129 pgs: 129 active+clean; 567 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 558 KiB/s wr, 19 op/s 2024-09-16T00:34:56.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 84551 17.99 MB/sec execute 122 sec latency 76.378 ms 2024-09-16T00:34:56.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.113+0000 7f9fe8366640 1 -- 172.21.15.33:0/3113767409 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fe0100e30 msgr2=0x7f9fe0104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:56.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.113+0000 7f9fe8366640 1 --2- 172.21.15.33:0/3113767409 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fe0100e30 0x7f9fe0104e80 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009900 tx=0x7f9fd402a410 comp rx=0 tx=0).stop 2024-09-16T00:34:56.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.114+0000 7f9fe8366640 1 -- 172.21.15.33:0/3113767409 shutdown_connections 2024-09-16T00:34:56.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.114+0000 7f9fe8366640 1 --2- 172.21.15.33:0/3113767409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fe01055b0 0x7f9fe010d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:56.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.114+0000 7f9fe8366640 1 --2- 172.21.15.33:0/3113767409 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fe0100e30 0x7f9fe0104e80 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:56.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.114+0000 7f9fe8366640 1 --2- 172.21.15.33:0/3113767409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fe0100480 0x7f9fe0100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:56.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.114+0000 7f9fe8366640 1 -- 172.21.15.33:0/3113767409 >> 172.21.15.33:0/3113767409 conn(0x7f9fe0077210 msgr2=0x7f9fe00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:56.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.114+0000 7f9fe8366640 1 -- 172.21.15.33:0/3113767409 shutdown_connections 2024-09-16T00:34:56.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.114+0000 7f9fe8366640 1 -- 172.21.15.33:0/3113767409 wait complete. 2024-09-16T00:34:56.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.114+0000 7f9fe8366640 1 Processor -- start 2024-09-16T00:34:56.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.114+0000 7f9fe8366640 1 -- start start 2024-09-16T00:34:56.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe8366640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fe0100480 0x7f9fe0198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:56.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe8366640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fe0100e30 0x7f9fe01989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:56.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe8366640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fe01055b0 0x7f9fe019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:56.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe8366640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe010fd90 con 0x7f9fe01055b0 2024-09-16T00:34:56.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe8366640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe010fc10 con 0x7f9fe0100480 2024-09-16T00:34:56.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe8366640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe010ff10 con 0x7f9fe0100e30 2024-09-16T00:34:56.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe6903640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fe01055b0 0x7f9fe019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe6102640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fe0100480 0x7f9fe0198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe6903640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fe01055b0 0x7f9fe019cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33576/0 (socket says 172.21.15.33:33576) 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe5901640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fe0100e30 0x7f9fe01989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe6903640 1 -- 172.21.15.33:0/2804418378 learned_addr learned my addr 172.21.15.33:0/2804418378 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe6102640 1 -- 172.21.15.33:0/2804418378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fe0100e30 msgr2=0x7f9fe01989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe6102640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fe0100e30 0x7f9fe01989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe6102640 1 -- 172.21.15.33:0/2804418378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fe01055b0 msgr2=0x7f9fe019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe6102640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fe01055b0 0x7f9fe019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe6102640 1 -- 172.21.15.33:0/2804418378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe019d420 con 0x7f9fe0100480 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe6903640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fe01055b0 0x7f9fe019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.115+0000 7f9fe6102640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fe0100480 0x7f9fe0198470 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00e9f0 tx=0x7f9fdc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.116+0000 7f9fd37fe640 1 -- 172.21.15.33:0/2804418378 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc00ccc0 con 0x7f9fe0100480 2024-09-16T00:34:56.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.116+0000 7f9fd37fe640 1 -- 172.21.15.33:0/2804418378 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fdc010910 con 0x7f9fe0100480 2024-09-16T00:34:56.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.116+0000 7f9fe8366640 1 -- 172.21.15.33:0/2804418378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe019d710 con 0x7f9fe0100480 2024-09-16T00:34:56.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.116+0000 7f9fd37fe640 1 -- 172.21.15.33:0/2804418378 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc018960 con 0x7f9fe0100480 2024-09-16T00:34:56.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.116+0000 7f9fe8366640 1 -- 172.21.15.33:0/2804418378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe01a4ff0 con 0x7f9fe0100480 2024-09-16T00:34:56.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.117+0000 7f9fe8366640 1 -- 172.21.15.33:0/2804418378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe0100860 con 0x7f9fe0100480 2024-09-16T00:34:56.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.118+0000 7f9fd37fe640 1 -- 172.21.15.33:0/2804418378 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9fdc010ac0 con 0x7f9fe0100480 2024-09-16T00:34:56.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.119+0000 7f9fd37fe640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9fcc0a32c0 0x7f9fcc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:56.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.119+0000 7f9fe5901640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9fcc0a32c0 0x7f9fcc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:56.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.119+0000 7f9fd37fe640 1 -- 172.21.15.33:0/2804418378 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9fdc0d0100 con 0x7f9fe0100480 2024-09-16T00:34:56.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.119+0000 7f9fe5901640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9fcc0a32c0 0x7f9fcc0a5780 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009230 tx=0x7f9fd40058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:56.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.123+0000 7f9fd37fe640 1 -- 172.21.15.33:0/2804418378 <== mon.2 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9fdc09c2f0 con 0x7f9fe0100480 2024-09-16T00:34:56.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.353+0000 7f9fe8366640 1 -- 172.21.15.33:0/2804418378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fe010a800 con 0x7f9fe0100480 2024-09-16T00:34:56.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.354+0000 7f9fd37fe640 1 -- 172.21.15.33:0/2804418378 <== mon.2 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f9fdc0a05f0 con 0x7f9fe0100480 2024-09-16T00:34:56.355 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:34:56.355 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:34:56.355 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:34:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.356+0000 7f9fe8366640 1 -- 172.21.15.33:0/2804418378 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9fcc0a32c0 msgr2=0x7f9fcc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.356+0000 7f9fe8366640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9fcc0a32c0 0x7f9fcc0a5780 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009230 tx=0x7f9fd40058d0 comp rx=0 tx=0).stop 2024-09-16T00:34:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.356+0000 7f9fe8366640 1 -- 172.21.15.33:0/2804418378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fe0100480 msgr2=0x7f9fe0198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.356+0000 7f9fe8366640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fe0100480 0x7f9fe0198470 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00e9f0 tx=0x7f9fdc00eeb0 comp rx=0 tx=0).stop 2024-09-16T00:34:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.356+0000 7f9fe8366640 1 -- 172.21.15.33:0/2804418378 shutdown_connections 2024-09-16T00:34:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.356+0000 7f9fe8366640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9fcc0a32c0 0x7f9fcc0a5780 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.356+0000 7f9fe8366640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fe01055b0 0x7f9fe019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.356+0000 7f9fe8366640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fe0100e30 0x7f9fe01989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.356+0000 7f9fe8366640 1 --2- 172.21.15.33:0/2804418378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fe0100480 0x7f9fe0198470 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.356+0000 7f9fe8366640 1 -- 172.21.15.33:0/2804418378 >> 172.21.15.33:0/2804418378 conn(0x7f9fe0077210 msgr2=0x7f9fe0103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.357+0000 7f9fe8366640 1 -- 172.21.15.33:0/2804418378 shutdown_connections 2024-09-16T00:34:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:56.357+0000 7f9fe8366640 1 -- 172.21.15.33:0/2804418378 wait complete. 2024-09-16T00:34:56.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:56 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2804418378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:34:56.800 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:56 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2804418378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:34:56.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:56 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2804418378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:34:57.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85033 18.05 MB/sec execute 123 sec latency 74.211 ms 2024-09-16T00:34:57.237 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 142.95 --await-for 45.0' 2024-09-16T00:34:57.237 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 142.95 --await-for 45.0 2024-09-16T00:34:57.531 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:34:57.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:57 smithi060 ceph-mon[35185]: pgmap v884: 129 pgs: 129 active+clean; 570 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.9 MiB/s wr, 55 op/s 2024-09-16T00:34:57.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:57 smithi033 ceph-mon[31243]: pgmap v884: 129 pgs: 129 active+clean; 570 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.9 MiB/s wr, 55 op/s 2024-09-16T00:34:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:57 smithi083 ceph-mon[35242]: pgmap v884: 129 pgs: 129 active+clean; 570 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.9 MiB/s wr, 55 op/s 2024-09-16T00:34:58.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85466 18.09 MB/sec execute 124 sec latency 59.955 ms 2024-09-16T00:34:58.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.252+0000 7f7f4248b640 1 -- 172.21.15.33:0/1202877439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f3c100e20 msgr2=0x7f7f3c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.252+0000 7f7f4248b640 1 --2- 172.21.15.33:0/1202877439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f3c100e20 0x7f7f3c101200 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009900 tx=0x7f7f2c02a480 comp rx=0 tx=0).stop 2024-09-16T00:34:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.252+0000 7f7f4248b640 1 -- 172.21.15.33:0/1202877439 shutdown_connections 2024-09-16T00:34:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.252+0000 7f7f4248b640 1 --2- 172.21.15.33:0/1202877439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f3c102ef0 0x7f7f3c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.252+0000 7f7f4248b640 1 --2- 172.21.15.33:0/1202877439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f3c102570 0x7f7f3c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.252+0000 7f7f4248b640 1 --2- 172.21.15.33:0/1202877439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f3c100e20 0x7f7f3c101200 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.252+0000 7f7f4248b640 1 -- 172.21.15.33:0/1202877439 >> 172.21.15.33:0/1202877439 conn(0x7f7f3c0fc7d0 msgr2=0x7f7f3c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.252+0000 7f7f4248b640 1 -- 172.21.15.33:0/1202877439 shutdown_connections 2024-09-16T00:34:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.252+0000 7f7f4248b640 1 -- 172.21.15.33:0/1202877439 wait complete. 2024-09-16T00:34:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.253+0000 7f7f4248b640 1 Processor -- start 2024-09-16T00:34:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.253+0000 7f7f4248b640 1 -- start start 2024-09-16T00:34:58.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.253+0000 7f7f4248b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f3c100e20 0x7f7f3c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.253+0000 7f7f4248b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f3c102570 0x7f7f3c198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.253+0000 7f7f4248b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f3c102ef0 0x7f7f3c19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.253+0000 7f7f4248b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c1120d0 con 0x7f7f3c102ef0 2024-09-16T00:34:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.253+0000 7f7f4248b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c111f50 con 0x7f7f3c100e20 2024-09-16T00:34:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.253+0000 7f7f4248b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c112250 con 0x7f7f3c102570 2024-09-16T00:34:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.253+0000 7f7f40a28640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f3c102ef0 0x7f7f3c19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f3bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f3c100e20 0x7f7f3c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f40a28640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f3c102ef0 0x7f7f3c19ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33604/0 (socket says 172.21.15.33:33604) 2024-09-16T00:34:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f3bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f3c100e20 0x7f7f3c198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55392/0 (socket says 172.21.15.33:55392) 2024-09-16T00:34:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f40a28640 1 -- 172.21.15.33:0/1122385764 learned_addr learned my addr 172.21.15.33:0/1122385764 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f3c102570 0x7f7f3c198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f40a28640 1 -- 172.21.15.33:0/1122385764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f3c102570 msgr2=0x7f7f3c198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f40a28640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f3c102570 0x7f7f3c198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f40a28640 1 -- 172.21.15.33:0/1122385764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f3c100e20 msgr2=0x7f7f3c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f40a28640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f3c100e20 0x7f7f3c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f40a28640 1 -- 172.21.15.33:0/1122385764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3c19d520 con 0x7f7f3c102ef0 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f3c102570 0x7f7f3c198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f3c100e20 0x7f7f3c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f40a28640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f3c102ef0 0x7f7f3c19ce40 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400eb10 tx=0x7f7f3400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f397fa640 1 -- 172.21.15.33:0/1122385764 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f34004b20 con 0x7f7f3c102ef0 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f4248b640 1 -- 172.21.15.33:0/1122385764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f3c19d810 con 0x7f7f3c102ef0 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f397fa640 1 -- 172.21.15.33:0/1122385764 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f34004cc0 con 0x7f7f3c102ef0 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f397fa640 1 -- 172.21.15.33:0/1122385764 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f34010760 con 0x7f7f3c102ef0 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.254+0000 7f7f4248b640 1 -- 172.21.15.33:0/1122385764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f3c1a50f0 con 0x7f7f3c102ef0 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.257+0000 7f7f44864740 1 Processor -- start 2024-09-16T00:34:58.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.257+0000 7f7f397fa640 1 -- 172.21.15.33:0/1122385764 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7f34005020 con 0x7f7f3c102ef0 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f44864740 1 -- start start 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f44864740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220aa8730 0x556220aa8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f44864740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220ab7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f44864740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220ab7e40 0x556220aba230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f44864740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556220a0d160 con 0x556220ab7e40 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f44864740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5562209b4720 con 0x556220aa9050 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f44864740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556220a09120 con 0x556220aa8730 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f3b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220ab7e40 0x556220aba230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f3bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220ab7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f3bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220ab7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55398/0 (socket says 172.21.15.33:55398) 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f3bfff640 1 -- 172.21.15.33:0/3212576121 learned_addr learned my addr 172.21.15.33:0/3212576121 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/3212576121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220aa8730 0x556220aa8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f397fa640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7f240a3240 0x7f7f240a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f3bfff640 1 -- 172.21.15.33:0/3212576121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220aa8730 msgr2=0x556220aa8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:58.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/3212576121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220aa8730 0x556220aa8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f3bfff640 1 -- 172.21.15.33:0/3212576121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220ab7e40 msgr2=0x556220aba230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:58.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.258+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/3212576121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220ab7e40 0x556220aba230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.259+0000 7f7f3bfff640 1 -- 172.21.15.33:0/3212576121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55622077eba0 con 0x556220aa9050 2024-09-16T00:34:58.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.259+0000 7f7f40a28640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7f240a3240 0x7f7f240a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.259+0000 7f7f397fa640 1 -- 172.21.15.33:0/1122385764 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7f340d0330 con 0x7f7f3c102ef0 2024-09-16T00:34:58.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.259+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/3212576121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220ab7900 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009900 tx=0x7f7f2c009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9792ddb477e9dab2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:58.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.259+0000 7f7f227fc640 1 -- 172.21.15.33:0/3212576121 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c035040 con 0x556220aa9050 2024-09-16T00:34:58.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.259+0000 7f7f227fc640 1 -- 172.21.15.33:0/3212576121 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f2c004760 con 0x556220aa9050 2024-09-16T00:34:58.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.259+0000 7f7f227fc640 1 -- 172.21.15.33:0/3212576121 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c003950 con 0x556220aa9050 2024-09-16T00:34:58.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.260+0000 7f7f44864740 1 -- 172.21.15.33:0/3212576121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 msgr2=0x556220ab7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:58.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.260+0000 7f7f40a28640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7f240a3240 0x7f7f240a5700 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7f34098930 tx=0x7f7f340919b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:58.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.260+0000 7f7f44864740 1 --2- 172.21.15.33:0/3212576121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220ab7900 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009900 tx=0x7f7f2c009650 comp rx=0 tx=0).stop 2024-09-16T00:34:58.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.260+0000 7f7f44864740 1 -- 172.21.15.33:0/3212576121 shutdown_connections 2024-09-16T00:34:58.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.260+0000 7f7f44864740 1 --2- 172.21.15.33:0/3212576121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220ab7e40 0x556220aba230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.260+0000 7f7f44864740 1 --2- 172.21.15.33:0/3212576121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220ab7900 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.260+0000 7f7f44864740 1 --2- 172.21.15.33:0/3212576121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220aa8730 0x556220aa8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.260+0000 7f7f44864740 1 -- 172.21.15.33:0/3212576121 >> 172.21.15.33:0/3212576121 conn(0x556220aa4de0 msgr2=0x556220aa51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:58.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.260+0000 7f7f44864740 1 -- 172.21.15.33:0/3212576121 shutdown_connections 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.260+0000 7f7f44864740 1 -- 172.21.15.33:0/3212576121 wait complete. 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.261+0000 7f7f44864740 1 -- start start 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.261+0000 7f7f44864740 1 Processor -- start 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.261+0000 7f7f44864740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 0x556220ab4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.261+0000 7f7f44864740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220ab52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.261+0000 7f7f44864740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 0x556220ab5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.261+0000 7f7f44864740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556220abaa10 con 0x556220aa8730 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.261+0000 7f7f44864740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556220995900 con 0x556220aa9050 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.261+0000 7f7f44864740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556220abab90 con 0x556220ab7e40 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.261+0000 7f7f44864740 10 client.0 _ll_register_callbacks cb 0x556220993780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.261+0000 7f7f3b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 0x556220ab4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.261+0000 7f7f3b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 0x556220ab4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33634/0 (socket says 172.21.15.33:33634) 2024-09-16T00:34:58.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.261+0000 7f7f3b7fe640 1 -- 172.21.15.33:0/4164899523 learned_addr learned my addr 172.21.15.33:0/4164899523 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 0x556220ab5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220ab52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f3b7fe640 1 -- 172.21.15.33:0/4164899523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 msgr2=0x556220ab5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 0x556220ab5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f3b7fe640 1 -- 172.21.15.33:0/4164899523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 msgr2=0x556220ab52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220ab52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f3b7fe640 1 -- 172.21.15.33:0/4164899523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562209d5c50 con 0x556220aa8730 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220ab52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 0x556220ab4da0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f7f30012ee0 tx=0x7f7f30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820c8dec84435cce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f1b7fe640 1 -- 172.21.15.33:0/4164899523 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3000de70 con 0x556220aa8730 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f1b7fe640 1 -- 172.21.15.33:0/4164899523 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7f30005020 con 0x556220aa8730 2024-09-16T00:34:58.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f44864740 1 -- 172.21.15.33:0/4164899523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f2c004760 con 0x556220aa8730 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f1b7fe640 1 -- 172.21.15.33:0/4164899523 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7f3002d050 con 0x556220aa8730 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f44864740 10 client.18807 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.262+0000 7f7f44864740 1 -- 172.21.15.33:0/4164899523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556220ac6770 con 0x556220aa8730 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.263+0000 7f7f1b7fe640 1 -- 172.21.15.33:0/4164899523 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f2c004760 con 0x556220aa8730 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.263+0000 7f7f1b7fe640 1 -- 172.21.15.33:0/4164899523 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7f3001e030 con 0x556220aa8730 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.263+0000 7f7f44864740 10 client.18807 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.263+0000 7f7f44864740 10 client.18807 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.263+0000 7f7f44864740 20 client.18807 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.263+0000 7f7f44864740 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556220ac7c80 0x556220aca120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.263+0000 7f7f44864740 4 client.18807 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.263+0000 7f7f44864740 1 -- 172.21.15.33:0/4164899523 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556220a1a890 con 0x556220ac7c80 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.263+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556220ac7c80 0x556220aca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.264+0000 7f7f1b7fe640 10 client.18807 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.264+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556220ac7c80 0x556220aca120 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=998537e58c1f01db server_cookie=39e1a7cd34196893 in_seq=0 out_seq=0 2024-09-16T00:34:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.264+0000 7f7f1b7fe640 1 -- 172.21.15.33:0/4164899523 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556220a1a890 con 0x556220ac7c80 2024-09-16T00:34:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.264+0000 7f7f1b7fe640 10 client.18807 handle_command_reply: tid=0 2024-09-16T00:34:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.264+0000 7f7f44864740 1 client.18807 shutdown 2024-09-16T00:34:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.264+0000 7f7f44864740 1 -- 172.21.15.33:0/4164899523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 msgr2=0x556220ab4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.264+0000 7f7f44864740 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 0x556220ab4da0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f7f30012ee0 tx=0x7f7f30004730 comp rx=0 tx=0).stop 2024-09-16T00:34:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.265+0000 7f7f44864740 1 -- 172.21.15.33:0/4164899523 shutdown_connections 2024-09-16T00:34:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.265+0000 7f7f44864740 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556220ac7c80 0x556220aca120 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.265+0000 7f7f1b7fe640 0 client.18807 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.265+0000 7f7f44864740 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 0x556220ab5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.265+0000 7f7f44864740 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220ab52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.265+0000 7f7f44864740 1 --2- 172.21.15.33:0/4164899523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 0x556220ab4da0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.265+0000 7f7f44864740 1 -- 172.21.15.33:0/4164899523 >> 172.21.15.33:0/4164899523 conn(0x556220aa4de0 msgr2=0x556220aaab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.265+0000 7f7f44864740 1 -- 172.21.15.33:0/4164899523 shutdown_connections 2024-09-16T00:34:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.265+0000 7f7f44864740 1 -- 172.21.15.33:0/4164899523 wait complete. 2024-09-16T00:34:58.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.265+0000 7f7f44864740 20 client.18807 trim_cache size 0 max 16384 2024-09-16T00:34:58.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f44864740 1 Processor -- start 2024-09-16T00:34:58.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f44864740 1 -- start start 2024-09-16T00:34:58.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f44864740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa8730 0x556220ab5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f44864740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa9050 0x556220ab59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f44864740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 0x556220acf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f44864740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c1af810 con 0x556220aa9050 2024-09-16T00:34:58.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f44864740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c1ad0e0 con 0x556220aa8730 2024-09-16T00:34:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f44864740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c1aaf30 con 0x556220ab7e40 2024-09-16T00:34:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f3bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa9050 0x556220ab59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f3bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa9050 0x556220ab59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33640/0 (socket says 172.21.15.33:33640) 2024-09-16T00:34:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f3bfff640 1 -- 172.21.15.33:0/3692682397 learned_addr learned my addr 172.21.15.33:0/3692682397 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/3692682397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa8730 0x556220ab5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/3692682397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 0x556220acf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f3bfff640 1 -- 172.21.15.33:0/3692682397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 msgr2=0x556220acf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/3692682397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 0x556220acf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f3bfff640 1 -- 172.21.15.33:0/3692682397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa8730 msgr2=0x556220ab5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/3692682397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa8730 0x556220ab5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.284+0000 7f7f3bfff640 1 -- 172.21.15.33:0/3692682397 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556220aa58a0 con 0x556220aa9050 2024-09-16T00:34:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.285+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/3692682397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa9050 0x556220ab59b0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009900 tx=0x7f7f2c005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec374e3c952d3e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.285+0000 7f7f21ffb640 1 -- 172.21.15.33:0/3692682397 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c035040 con 0x556220aa9050 2024-09-16T00:34:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.285+0000 7f7f21ffb640 1 -- 172.21.15.33:0/3692682397 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f2c005530 con 0x556220aa9050 2024-09-16T00:34:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.285+0000 7f7f21ffb640 1 -- 172.21.15.33:0/3692682397 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c004bf0 con 0x556220aa9050 2024-09-16T00:34:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.285+0000 7f7f44864740 1 -- 172.21.15.33:0/3692682397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa9050 msgr2=0x556220ab59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.285+0000 7f7f44864740 1 --2- 172.21.15.33:0/3692682397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa9050 0x556220ab59b0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009900 tx=0x7f7f2c005b50 comp rx=0 tx=0).stop 2024-09-16T00:34:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.286+0000 7f7f44864740 1 -- 172.21.15.33:0/3692682397 shutdown_connections 2024-09-16T00:34:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.286+0000 7f7f44864740 1 --2- 172.21.15.33:0/3692682397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 0x556220acf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.286+0000 7f7f44864740 1 --2- 172.21.15.33:0/3692682397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa9050 0x556220ab59b0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.286+0000 7f7f44864740 1 --2- 172.21.15.33:0/3692682397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa8730 0x556220ab5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.286+0000 7f7f44864740 1 -- 172.21.15.33:0/3692682397 >> 172.21.15.33:0/3692682397 conn(0x556220aa4de0 msgr2=0x556220ab4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.286+0000 7f7f44864740 1 -- 172.21.15.33:0/3692682397 shutdown_connections 2024-09-16T00:34:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.286+0000 7f7f44864740 1 -- 172.21.15.33:0/3692682397 wait complete. 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.286+0000 7f7f44864740 1 -- start start 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.287+0000 7f7f44864740 1 Processor -- start 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.287+0000 7f7f44864740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 0x556220ac0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.287+0000 7f7f44864740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220abb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.287+0000 7f7f44864740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 0x556220abbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.287+0000 7f7f44864740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c1ab5a0 con 0x556220aa8730 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.287+0000 7f7f44864740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c1aca10 con 0x556220aa9050 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.287+0000 7f7f44864740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c1abc70 con 0x556220ab7e40 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.287+0000 7f7f44864740 10 client.0 _ll_register_callbacks cb 0x7f7f30012cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.287+0000 7f7f3b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 0x556220ac0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f3b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 0x556220ac0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33646/0 (socket says 172.21.15.33:33646) 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f3b7fe640 1 -- 172.21.15.33:0/2227915620 learned_addr learned my addr 172.21.15.33:0/2227915620 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:34:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 0x556220abbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220abb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f3b7fe640 1 -- 172.21.15.33:0/2227915620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 msgr2=0x556220abbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 0x556220abbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f3b7fe640 1 -- 172.21.15.33:0/2227915620 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 msgr2=0x556220abb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:34:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220abb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f3b7fe640 1 -- 172.21.15.33:0/2227915620 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556220abc3b0 con 0x556220aa8730 2024-09-16T00:34:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f3b7fe640 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 0x556220ac0450 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f7f30000c00 tx=0x7f7f30012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238377930a317138 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:34:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:34:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f1b7fe640 1 -- 172.21.15.33:0/2227915620 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3000fbc0 con 0x556220aa8730 2024-09-16T00:34:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.288+0000 7f7f1b7fe640 1 -- 172.21.15.33:0/2227915620 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7f3000fd60 con 0x556220aa8730 2024-09-16T00:34:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.289+0000 7f7f44864740 1 -- 172.21.15.33:0/2227915620 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f2c005530 con 0x556220aa8730 2024-09-16T00:34:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.289+0000 7f7f1b7fe640 1 -- 172.21.15.33:0/2227915620 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7f3003f050 con 0x556220aa8730 2024-09-16T00:34:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.289+0000 7f7f44864740 10 client.18813 fetch_fsmap learned FSMap version 33 2024-09-16T00:34:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.289+0000 7f7f44864740 1 -- 172.21.15.33:0/2227915620 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556220b32880 con 0x556220aa8730 2024-09-16T00:34:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.289+0000 7f7f1b7fe640 1 -- 172.21.15.33:0/2227915620 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f30004c80 con 0x556220aa8730 2024-09-16T00:34:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.289+0000 7f7f1b7fe640 1 -- 172.21.15.33:0/2227915620 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7f3001e040 con 0x556220aa8730 2024-09-16T00:34:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.289+0000 7f7f44864740 10 client.18813 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:34:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.289+0000 7f7f44864740 10 client.18813 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:34:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.289+0000 7f7f44864740 20 client.18813 populate_metadata read hostname 'smithi033' 2024-09-16T00:34:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.289+0000 7f7f44864740 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556220ac7c80 0x556220b34f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:34:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.289+0000 7f7f44864740 4 client.18813 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.95, "await_for": 45.0}] 2024-09-16T00:34:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.289+0000 7f7f44864740 1 -- 172.21.15.33:0/2227915620 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.95, "await_for": 45.0}) -- 0x7f7f2c005530 con 0x556220ac7c80 2024-09-16T00:34:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.290+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556220ac7c80 0x556220b34f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:34:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.290+0000 7f7f3bfff640 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556220ac7c80 0x556220b34f00 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8efde9714405720 server_cookie=7d5fc8a941608658 in_seq=0 out_seq=0 2024-09-16T00:34:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:58.290+0000 7f7f1b7fe640 10 client.18813 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:59.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f1b7fe640 1 -- 172.21.15.33:0/2227915620 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f7f2c005530 con 0x556220ac7c80 2024-09-16T00:34:59.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f1b7fe640 10 client.18813 handle_command_reply: tid=0 2024-09-16T00:34:59.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f44864740 1 client.18813 shutdown 2024-09-16T00:34:59.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f44864740 1 -- 172.21.15.33:0/2227915620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 msgr2=0x556220ac0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:59.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f44864740 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 0x556220ac0450 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f7f30000c00 tx=0x7f7f30012b90 comp rx=0 tx=0).stop 2024-09-16T00:34:59.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f44864740 1 -- 172.21.15.33:0/2227915620 shutdown_connections 2024-09-16T00:34:59.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f44864740 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556220ac7c80 0x556220b34f00 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:59.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f44864740 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556220ab7e40 0x556220abbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:59.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f44864740 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556220aa9050 0x556220abb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:59.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f44864740 1 --2- 172.21.15.33:0/2227915620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556220aa8730 0x556220ac0450 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:59.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f44864740 1 -- 172.21.15.33:0/2227915620 >> 172.21.15.33:0/2227915620 conn(0x556220aa4de0 msgr2=0x556220aa3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:59.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f1b7fe640 0 client.18813 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:34:59.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f44864740 1 -- 172.21.15.33:0/2227915620 shutdown_connections 2024-09-16T00:34:59.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f44864740 1 -- 172.21.15.33:0/2227915620 wait complete. 2024-09-16T00:34:59.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.056+0000 7f7f44864740 20 client.18813 trim_cache size 0 max 16384 2024-09-16T00:34:59.058 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:34:59.058 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:34:59.058 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:34:59.058 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 66, 2024-09-16T00:34:59.058 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:34:59.058 INFO:teuthology.orchestra.run.smithi033.stdout: "3ec5e571": { 2024-09-16T00:34:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 66, 2024-09-16T00:34:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:34:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:34:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:34:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:34:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:34:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:34:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 142.9, 2024-09-16T00:34:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:34:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:34:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:34:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:34:59.060 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:34:59.060 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:34:59.060 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:59.060 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:59.060 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:59.060 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:59.060 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:34:59.060 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:34:59.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.057+0000 7f7f4248b640 1 -- 172.21.15.33:0/1122385764 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7f240a3240 msgr2=0x7f7f240a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:59.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.057+0000 7f7f4248b640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7f240a3240 0x7f7f240a5700 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7f34098930 tx=0x7f7f340919b0 comp rx=0 tx=0).stop 2024-09-16T00:34:59.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.057+0000 7f7f4248b640 1 -- 172.21.15.33:0/1122385764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f3c102ef0 msgr2=0x7f7f3c19ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:34:59.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.057+0000 7f7f4248b640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f3c102ef0 0x7f7f3c19ce40 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400eb10 tx=0x7f7f3400c2d0 comp rx=0 tx=0).stop 2024-09-16T00:34:59.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.057+0000 7f7f4248b640 1 -- 172.21.15.33:0/1122385764 shutdown_connections 2024-09-16T00:34:59.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.057+0000 7f7f4248b640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7f240a3240 0x7f7f240a5700 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:59.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.057+0000 7f7f4248b640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f3c102ef0 0x7f7f3c19ce40 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:59.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.057+0000 7f7f4248b640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f3c102570 0x7f7f3c198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:59.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.057+0000 7f7f4248b640 1 --2- 172.21.15.33:0/1122385764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f3c100e20 0x7f7f3c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:34:59.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.057+0000 7f7f4248b640 1 -- 172.21.15.33:0/1122385764 >> 172.21.15.33:0/1122385764 conn(0x7f7f3c0fc7d0 msgr2=0x7f7f3c0fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:34:59.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.057+0000 7f7f4248b640 1 -- 172.21.15.33:0/1122385764 shutdown_connections 2024-09-16T00:34:59.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:34:59.057+0000 7f7f4248b640 1 -- 172.21.15.33:0/1122385764 wait complete. 2024-09-16T00:34:59.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 17.96 MB/sec execute 125 sec latency 785.347 ms 2024-09-16T00:34:59.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:34:59 smithi060 ceph-mon[35185]: pgmap v885: 129 pgs: 129 active+clean; 570 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.9 MiB/s wr, 54 op/s 2024-09-16T00:34:59.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:34:59 smithi033 ceph-mon[31243]: pgmap v885: 129 pgs: 129 active+clean; 570 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.9 MiB/s wr, 54 op/s 2024-09-16T00:34:59.808 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '3ec5e571', quiesce took 4.6 seconds. Will release after: 18.35 2024-09-16T00:34:59.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:34:59 smithi083 ceph-mon[35242]: pgmap v885: 129 pgs: 129 active+clean; 570 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.9 MiB/s wr, 54 op/s 2024-09-16T00:35:00.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 17.82 MB/sec execute 126 sec latency 1785.399 ms 2024-09-16T00:35:01.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 17.68 MB/sec execute 127 sec latency 2785.484 ms 2024-09-16T00:35:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:01 smithi060 ceph-mon[35185]: pgmap v886: 129 pgs: 129 active+clean; 564 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 188 op/s 2024-09-16T00:35:01.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:01 smithi033 ceph-mon[31243]: pgmap v886: 129 pgs: 129 active+clean; 564 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 188 op/s 2024-09-16T00:35:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:01 smithi083 ceph-mon[35242]: pgmap v886: 129 pgs: 129 active+clean; 564 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 188 op/s 2024-09-16T00:35:02.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 17.54 MB/sec execute 128 sec latency 3785.545 ms 2024-09-16T00:35:02.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:02 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:35:02.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:02 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:35:02.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:02 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:35:02.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:02 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:35:02.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:02 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:35:02.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:02 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:35:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:02 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:35:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:02 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:35:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:02 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:35:03.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 17.40 MB/sec execute 129 sec latency 4785.640 ms 2024-09-16T00:35:03.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:03 smithi033 ceph-mon[31243]: pgmap v887: 129 pgs: 129 active+clean; 541 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 207 op/s 2024-09-16T00:35:03.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:03 smithi083 ceph-mon[35242]: pgmap v887: 129 pgs: 129 active+clean; 541 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 207 op/s 2024-09-16T00:35:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:03 smithi060 ceph-mon[35185]: pgmap v887: 129 pgs: 129 active+clean; 541 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.4 MiB/s wr, 207 op/s 2024-09-16T00:35:04.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 17.27 MB/sec execute 130 sec latency 5785.755 ms 2024-09-16T00:35:05.080 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 17.14 MB/sec execute 131 sec latency 6785.839 ms 2024-09-16T00:35:05.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:05 smithi033 ceph-mon[31243]: pgmap v888: 129 pgs: 129 active+clean; 542 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-09-16T00:35:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:05 smithi083 ceph-mon[35242]: pgmap v888: 129 pgs: 129 active+clean; 542 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-09-16T00:35:05.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:05 smithi060 ceph-mon[35185]: pgmap v888: 129 pgs: 129 active+clean; 542 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-09-16T00:35:06.079 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 17.01 MB/sec execute 132 sec latency 7785.882 ms 2024-09-16T00:35:07.080 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 16.88 MB/sec execute 133 sec latency 8785.967 ms 2024-09-16T00:35:07.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:07 smithi033 ceph-mon[31243]: pgmap v889: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 233 op/s 2024-09-16T00:35:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:07 smithi083 ceph-mon[35242]: pgmap v889: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 233 op/s 2024-09-16T00:35:07.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:07 smithi060 ceph-mon[35185]: pgmap v889: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 233 op/s 2024-09-16T00:35:08.080 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 16.75 MB/sec execute 134 sec latency 9786.010 ms 2024-09-16T00:35:09.080 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 16.63 MB/sec execute 135 sec latency 10786.063 ms 2024-09-16T00:35:09.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:09 smithi033 ceph-mon[31243]: pgmap v890: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-09-16T00:35:09.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:09 smithi083 ceph-mon[35242]: pgmap v890: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-09-16T00:35:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:09 smithi060 ceph-mon[35185]: pgmap v890: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-09-16T00:35:10.080 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 16.51 MB/sec execute 136 sec latency 11786.150 ms 2024-09-16T00:35:11.080 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 16.38 MB/sec execute 137 sec latency 12786.316 ms 2024-09-16T00:35:11.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:11 smithi033 ceph-mon[31243]: pgmap v891: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 199 op/s 2024-09-16T00:35:11.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:11 smithi083 ceph-mon[35242]: pgmap v891: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 199 op/s 2024-09-16T00:35:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:11 smithi060 ceph-mon[35185]: pgmap v891: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 199 op/s 2024-09-16T00:35:12.080 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 16.27 MB/sec execute 138 sec latency 13786.411 ms 2024-09-16T00:35:13.080 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 16.15 MB/sec execute 139 sec latency 14786.517 ms 2024-09-16T00:35:13.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:13 smithi033 ceph-mon[31243]: pgmap v892: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 272 KiB/s wr, 66 op/s 2024-09-16T00:35:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:13 smithi083 ceph-mon[35242]: pgmap v892: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 272 KiB/s wr, 66 op/s 2024-09-16T00:35:13.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:13 smithi060 ceph-mon[35185]: pgmap v892: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 272 KiB/s wr, 66 op/s 2024-09-16T00:35:14.080 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 16.03 MB/sec execute 140 sec latency 15786.639 ms 2024-09-16T00:35:15.080 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 15.92 MB/sec execute 141 sec latency 16786.728 ms 2024-09-16T00:35:15.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:15 smithi033 ceph-mon[31243]: pgmap v893: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 140 KiB/s wr, 46 op/s 2024-09-16T00:35:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:15 smithi083 ceph-mon[35242]: pgmap v893: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 140 KiB/s wr, 46 op/s 2024-09-16T00:35:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:15 smithi060 ceph-mon[35185]: pgmap v893: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 140 KiB/s wr, 46 op/s 2024-09-16T00:35:16.080 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 15.81 MB/sec execute 142 sec latency 17786.847 ms 2024-09-16T00:35:17.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 15.70 MB/sec execute 143 sec latency 18786.954 ms 2024-09-16T00:35:17.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:17 smithi033 ceph-mon[31243]: pgmap v894: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-09-16T00:35:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:17 smithi083 ceph-mon[35242]: pgmap v894: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-09-16T00:35:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:17 smithi060 ceph-mon[35185]: pgmap v894: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-09-16T00:35:18.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 15.59 MB/sec execute 144 sec latency 19787.050 ms 2024-09-16T00:35:18.159 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:35:18.435 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:35:18.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.912+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/58181266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a78104e80 msgr2=0x7f6a78105330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:18.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.912+0000 7f6a7d6ab640 1 --2- 172.21.15.33:0/58181266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a78104e80 0x7f6a78105330 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009ca0 tx=0x7f6a7002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:35:18.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.913+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/58181266 shutdown_connections 2024-09-16T00:35:18.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.913+0000 7f6a7d6ab640 1 --2- 172.21.15.33:0/58181266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a78104e80 0x7f6a78105330 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:18.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.913+0000 7f6a7d6ab640 1 --2- 172.21.15.33:0/58181266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a7806a370 0x7f6a781009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:18.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.913+0000 7f6a7d6ab640 1 --2- 172.21.15.33:0/58181266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a78069f00 0x7f6a78100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:18.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.913+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/58181266 >> 172.21.15.33:0/58181266 conn(0x7f6a78077210 msgr2=0x7f6a780fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:18.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.913+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/58181266 shutdown_connections 2024-09-16T00:35:18.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.913+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/58181266 wait complete. 2024-09-16T00:35:18.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.913+0000 7f6a7d6ab640 1 Processor -- start 2024-09-16T00:35:18.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.913+0000 7f6a7d6ab640 1 -- start start 2024-09-16T00:35:18.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a7d6ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a78069f00 0x7f6a781993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:18.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a7d6ab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a7806a370 0x7f6a78199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:18.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a7d6ab640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a78104e80 0x7f6a781a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:18.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a7d6ab640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6a78110f30 con 0x7f6a78104e80 2024-09-16T00:35:18.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a7d6ab640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6a78110db0 con 0x7f6a78069f00 2024-09-16T00:35:18.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a7d6ab640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6a781110b0 con 0x7f6a7806a370 2024-09-16T00:35:18.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a78104e80 0x7f6a781a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:18.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a78104e80 0x7f6a781a4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33460/0 (socket says 172.21.15.33:33460) 2024-09-16T00:35:18.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a777fe640 1 -- 172.21.15.33:0/1568595699 learned_addr learned my addr 172.21.15.33:0/1568595699 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:18.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a76ffd640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a78069f00 0x7f6a781993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a767fc640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a7806a370 0x7f6a78199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a777fe640 1 -- 172.21.15.33:0/1568595699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a7806a370 msgr2=0x7f6a78199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a777fe640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a7806a370 0x7f6a78199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a777fe640 1 -- 172.21.15.33:0/1568595699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a78069f00 msgr2=0x7f6a781993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a777fe640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a78069f00 0x7f6a781993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a777fe640 1 -- 172.21.15.33:0/1568595699 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a781a4980 con 0x7f6a78104e80 2024-09-16T00:35:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a767fc640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a7806a370 0x7f6a78199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:35:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a777fe640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a78104e80 0x7f6a781a4440 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f6a70031a60 tx=0x7f6a70031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a5ffff640 1 -- 172.21.15.33:0/1568595699 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a70003b70 con 0x7f6a78104e80 2024-09-16T00:35:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.914+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/1568595699 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a781a4b50 con 0x7f6a78104e80 2024-09-16T00:35:18.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.915+0000 7f6a5ffff640 1 -- 172.21.15.33:0/1568595699 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a70003d10 con 0x7f6a78104e80 2024-09-16T00:35:18.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.915+0000 7f6a5ffff640 1 -- 172.21.15.33:0/1568595699 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a70004770 con 0x7f6a78104e80 2024-09-16T00:35:18.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.915+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/1568595699 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a781a5030 con 0x7f6a78104e80 2024-09-16T00:35:18.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.916+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/1568595699 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a781053b0 con 0x7f6a78104e80 2024-09-16T00:35:18.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.918+0000 7f6a5ffff640 1 -- 172.21.15.33:0/1568595699 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6a70004b40 con 0x7f6a78104e80 2024-09-16T00:35:18.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.918+0000 7f6a5ffff640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6a600a3240 0x7f6a600a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:18.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.918+0000 7f6a76ffd640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6a600a3240 0x7f6a600a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:18.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.918+0000 7f6a5ffff640 1 -- 172.21.15.33:0/1568595699 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f6a700f1260 con 0x7f6a78104e80 2024-09-16T00:35:18.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.919+0000 7f6a76ffd640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6a600a3240 0x7f6a600a5700 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009990 tx=0x7f6a68009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:18.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:18.922+0000 7f6a5ffff640 1 -- 172.21.15.33:0/1568595699 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6a700bdb00 con 0x7f6a78104e80 2024-09-16T00:35:19.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 15.48 MB/sec execute 145 sec latency 20787.101 ms 2024-09-16T00:35:19.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.136+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/1568595699 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a781a5260 con 0x7f6a78104e80 2024-09-16T00:35:19.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.137+0000 7f6a5ffff640 1 -- 172.21.15.33:0/1568595699 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f6a700bd400 con 0x7f6a78104e80 2024-09-16T00:35:19.139 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:35:19.139 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:35:19.139 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:35:19.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.139+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/1568595699 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6a600a3240 msgr2=0x7f6a600a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:19.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.139+0000 7f6a7d6ab640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6a600a3240 0x7f6a600a5700 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009990 tx=0x7f6a68009110 comp rx=0 tx=0).stop 2024-09-16T00:35:19.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.139+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/1568595699 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a78104e80 msgr2=0x7f6a781a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:19.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.139+0000 7f6a7d6ab640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a78104e80 0x7f6a781a4440 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f6a70031a60 tx=0x7f6a70031a90 comp rx=0 tx=0).stop 2024-09-16T00:35:19.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.139+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/1568595699 shutdown_connections 2024-09-16T00:35:19.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.139+0000 7f6a7d6ab640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f6a600a3240 0x7f6a600a5700 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:19.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.139+0000 7f6a7d6ab640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a78104e80 0x7f6a781a4440 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:19.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.139+0000 7f6a7d6ab640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a7806a370 0x7f6a78199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:19.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.139+0000 7f6a7d6ab640 1 --2- 172.21.15.33:0/1568595699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a78069f00 0x7f6a781993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:19.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.139+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/1568595699 >> 172.21.15.33:0/1568595699 conn(0x7f6a78077210 msgr2=0x7f6a78102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:19.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.140+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/1568595699 shutdown_connections 2024-09-16T00:35:19.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:19.140+0000 7f6a7d6ab640 1 -- 172.21.15.33:0/1568595699 wait complete. 2024-09-16T00:35:19.251 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:35:19.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:19 smithi033 ceph-mon[31243]: pgmap v895: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-09-16T00:35:19.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:19 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1568595699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:35:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:19 smithi083 ceph-mon[35242]: pgmap v895: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-09-16T00:35:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:19 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1568595699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:35:19.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:19 smithi060 ceph-mon[35185]: pgmap v895: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-09-16T00:35:19.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:19 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1568595699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:35:20.037 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 3ec5e571 --release --await' 2024-09-16T00:35:20.037 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 3ec5e571 --release --await 2024-09-16T00:35:20.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 15.37 MB/sec execute 146 sec latency 21787.137 ms 2024-09-16T00:35:20.109 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:35:20.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.607+0000 7f223e62e640 1 -- 172.21.15.33:0/3751415830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2238100e20 msgr2=0x7f2238101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:20.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.607+0000 7f223e62e640 1 --2- 172.21.15.33:0/3751415830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2238100e20 0x7f2238101200 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f2228009900 tx=0x7f222802a480 comp rx=0 tx=0).stop 2024-09-16T00:35:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.607+0000 7f223e62e640 1 -- 172.21.15.33:0/3751415830 shutdown_connections 2024-09-16T00:35:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.607+0000 7f223e62e640 1 --2- 172.21.15.33:0/3751415830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2238102ef0 0x7f223810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.607+0000 7f223e62e640 1 --2- 172.21.15.33:0/3751415830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2238102570 0x7f22381029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.607+0000 7f223e62e640 1 --2- 172.21.15.33:0/3751415830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2238100e20 0x7f2238101200 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.607+0000 7f223e62e640 1 -- 172.21.15.33:0/3751415830 >> 172.21.15.33:0/3751415830 conn(0x7f22380fc7d0 msgr2=0x7f22380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.607+0000 7f223e62e640 1 -- 172.21.15.33:0/3751415830 shutdown_connections 2024-09-16T00:35:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.607+0000 7f223e62e640 1 -- 172.21.15.33:0/3751415830 wait complete. 2024-09-16T00:35:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.608+0000 7f223e62e640 1 Processor -- start 2024-09-16T00:35:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.608+0000 7f223e62e640 1 -- start start 2024-09-16T00:35:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.608+0000 7f223e62e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2238100e20 0x7f22381992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.608+0000 7f223e62e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2238102570 0x7f2238199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.608+0000 7f223e62e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2238102ef0 0x7f22381a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.608+0000 7f223e62e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2238110f80 con 0x7f2238100e20 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.608+0000 7f223e62e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2238110e00 con 0x7f2238102570 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.608+0000 7f223e62e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2238111100 con 0x7f2238102ef0 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.608+0000 7f2237fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2238100e20 0x7f22381992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.608+0000 7f2237fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2238100e20 0x7f22381992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56236/0 (socket says 172.21.15.33:56236) 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.608+0000 7f2237fff640 1 -- 172.21.15.33:0/1868000292 learned_addr learned my addr 172.21.15.33:0/1868000292 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.608+0000 7f223cbcb640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2238102ef0 0x7f22381a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f22377fe640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2238102570 0x7f2238199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f2237fff640 1 -- 172.21.15.33:0/1868000292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2238102ef0 msgr2=0x7f22381a4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f2237fff640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2238102ef0 0x7f22381a4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f2237fff640 1 -- 172.21.15.33:0/1868000292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2238102570 msgr2=0x7f2238199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f2237fff640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2238102570 0x7f2238199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f2237fff640 1 -- 172.21.15.33:0/1868000292 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22381a4890 con 0x7f2238100e20 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f22377fe640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2238102570 0x7f2238199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f2237fff640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2238100e20 0x7f22381992d0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f2228006010 tx=0x7f2228003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f22357fa640 1 -- 172.21.15.33:0/1868000292 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222803f070 con 0x7f2238100e20 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f223e62e640 1 -- 172.21.15.33:0/1868000292 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22381a4a60 con 0x7f2238100e20 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f22357fa640 1 -- 172.21.15.33:0/1868000292 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2228031e30 con 0x7f2238100e20 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f22357fa640 1 -- 172.21.15.33:0/1868000292 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228035030 con 0x7f2238100e20 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.609+0000 7f223e62e640 1 -- 172.21.15.33:0/1868000292 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22381a4f10 con 0x7f2238100e20 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f22357fa640 1 -- 172.21.15.33:0/1868000292 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2228042070 con 0x7f2238100e20 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f2240a07740 1 Processor -- start 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f2240a07740 1 -- start start 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f2240a07740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591948534a0 0x559194853880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f2240a07740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559194853dc0 0x55919485e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f2240a07740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55919485e680 0x559194860a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f2240a07740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5591947b92e0 con 0x5591948534a0 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f2240a07740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559194760ce0 con 0x55919485e680 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f2240a07740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5591947b52a0 con 0x559194853dc0 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f223cbcb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591948534a0 0x559194853880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f223cbcb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591948534a0 0x559194853880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56246/0 (socket says 172.21.15.33:56246) 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f223cbcb640 1 -- 172.21.15.33:0/2682668836 learned_addr learned my addr 172.21.15.33:0/2682668836 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f2237fff640 1 --2- 172.21.15.33:0/2682668836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559194853dc0 0x55919485e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.612+0000 7f22377fe640 1 --2- 172.21.15.33:0/2682668836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55919485e680 0x559194860a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f223cbcb640 1 -- 172.21.15.33:0/2682668836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559194853dc0 msgr2=0x55919485e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f223cbcb640 1 --2- 172.21.15.33:0/2682668836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559194853dc0 0x55919485e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f22357fa640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f22200a3240 0x7f22200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f223cbcb640 1 -- 172.21.15.33:0/2682668836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55919485e680 msgr2=0x559194860a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f223cbcb640 1 --2- 172.21.15.33:0/2682668836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55919485e680 0x559194860a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f223cbcb640 1 -- 172.21.15.33:0/2682668836 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559194745440 con 0x5591948534a0 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f2237fff640 1 --2- 172.21.15.33:0/2682668836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559194853dc0 0x55919485e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f22357fa640 1 -- 172.21.15.33:0/1868000292 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f22280f06e0 con 0x7f2238100e20 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f223cbcb640 1 --2- 172.21.15.33:0/2682668836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591948534a0 0x559194853880 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f2230007ec0 tx=0x7f223000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec8749b39a01856 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f223cbcb640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f22200a3240 0x7f22200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f221e7fc640 1 -- 172.21.15.33:0/2682668836 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2230010940 con 0x5591948534a0 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f221e7fc640 1 -- 172.21.15.33:0/2682668836 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2230010e40 con 0x5591948534a0 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.613+0000 7f221e7fc640 1 -- 172.21.15.33:0/2682668836 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2230017a30 con 0x5591948534a0 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.614+0000 7f2240a07740 1 -- 172.21.15.33:0/2682668836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591948534a0 msgr2=0x559194853880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.614+0000 7f223cbcb640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f22200a3240 0x7f22200a5700 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2230007fd0 tx=0x7f2230017dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.614+0000 7f2240a07740 1 --2- 172.21.15.33:0/2682668836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591948534a0 0x559194853880 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f2230007ec0 tx=0x7f223000f790 comp rx=0 tx=0).stop 2024-09-16T00:35:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.614+0000 7f2240a07740 1 -- 172.21.15.33:0/2682668836 shutdown_connections 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.614+0000 7f2240a07740 1 --2- 172.21.15.33:0/2682668836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55919485e680 0x559194860a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.614+0000 7f2240a07740 1 --2- 172.21.15.33:0/2682668836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559194853dc0 0x55919485e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.614+0000 7f2240a07740 1 --2- 172.21.15.33:0/2682668836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5591948534a0 0x559194853880 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.614+0000 7f2240a07740 1 -- 172.21.15.33:0/2682668836 >> 172.21.15.33:0/2682668836 conn(0x55919484fa30 msgr2=0x55919484fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.614+0000 7f2240a07740 1 -- 172.21.15.33:0/2682668836 shutdown_connections 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.614+0000 7f2240a07740 1 -- 172.21.15.33:0/2682668836 wait complete. 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.614+0000 7f2240a07740 1 -- start start 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.614+0000 7f2240a07740 1 Processor -- start 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f2240a07740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x55919485df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f2240a07740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x55919485b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f2240a07740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x55919485b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f2240a07740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559194861250 con 0x55919485e680 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f2240a07740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559194774050 con 0x559194853dc0 2024-09-16T00:35:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f2240a07740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5591948613d0 con 0x5591948534a0 2024-09-16T00:35:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f2240a07740 10 client.0 _ll_register_callbacks cb 0x5591947b4160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f223cbcb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x55919485b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f223cbcb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x55919485b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56248/0 (socket says 172.21.15.33:56248) 2024-09-16T00:35:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f223cbcb640 1 -- 172.21.15.33:0/3462391183 learned_addr learned my addr 172.21.15.33:0/3462391183 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f22377fe640 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x55919485b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f2237fff640 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x55919485df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f223cbcb640 1 -- 172.21.15.33:0/3462391183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 msgr2=0x55919485df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f223cbcb640 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x55919485df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f223cbcb640 1 -- 172.21.15.33:0/3462391183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 msgr2=0x55919485b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f223cbcb640 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x55919485b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f223cbcb640 1 -- 172.21.15.33:0/3462391183 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55919452a3c0 con 0x55919485e680 2024-09-16T00:35:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f2237fff640 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x55919485df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f22377fe640 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x55919485b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f223cbcb640 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x55919485b710 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f2230003ea0 tx=0x7f2230017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=176e009b901c1922 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.615+0000 7f22177fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f22177fe640 1 -- 172.21.15.33:0/3462391183 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2230034060 con 0x55919485e680 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f2240a07740 1 -- 172.21.15.33:0/3462391183 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5591947c6850 con 0x55919485e680 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f22177fe640 1 -- 172.21.15.33:0/3462391183 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2230017a00 con 0x55919485e680 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f22177fe640 1 -- 172.21.15.33:0/3462391183 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f223003b050 con 0x55919485e680 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f2240a07740 10 client.18831 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f2240a07740 1 -- 172.21.15.33:0/3462391183 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55919486cbf0 con 0x55919485e680 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f22177fe640 1 -- 172.21.15.33:0/3462391183 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5591947c6850 con 0x55919485e680 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f22177fe640 1 -- 172.21.15.33:0/3462391183 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f22300050e0 con 0x55919485e680 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f2240a07740 10 client.18831 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f2240a07740 10 client.18831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:35:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f2240a07740 20 client.18831 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f2240a07740 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55919486e000 0x5591948703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f2240a07740 4 client.18831 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:35:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.616+0000 7f2240a07740 1 -- 172.21.15.33:0/3462391183 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591947c0d60 con 0x55919486e000 2024-09-16T00:35:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.617+0000 7f2237fff640 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55919486e000 0x5591948703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.617+0000 7f2237fff640 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55919486e000 0x5591948703f0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e59febc035e1232c server_cookie=32b4688a79a5ced8 in_seq=0 out_seq=0 2024-09-16T00:35:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.617+0000 7f22177fe640 10 client.18831 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f22177fe640 1 -- 172.21.15.33:0/3462391183 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5591947c0d60 con 0x55919486e000 2024-09-16T00:35:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f22177fe640 10 client.18831 handle_command_reply: tid=0 2024-09-16T00:35:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f2240a07740 1 client.18831 shutdown 2024-09-16T00:35:20.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f2240a07740 1 -- 172.21.15.33:0/3462391183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 msgr2=0x55919485b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:20.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f2240a07740 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x55919485b710 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f2230003ea0 tx=0x7f2230017000 comp rx=0 tx=0).stop 2024-09-16T00:35:20.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f2240a07740 1 -- 172.21.15.33:0/3462391183 shutdown_connections 2024-09-16T00:35:20.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f2240a07740 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55919486e000 0x5591948703f0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f22177fe640 0 client.18831 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:20.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f2240a07740 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x55919485b710 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f2240a07740 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x55919485b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f2240a07740 1 --2- 172.21.15.33:0/3462391183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x55919485df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f2240a07740 1 -- 172.21.15.33:0/3462391183 >> 172.21.15.33:0/3462391183 conn(0x55919484fa30 msgr2=0x55919484eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:20.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f2240a07740 1 -- 172.21.15.33:0/3462391183 shutdown_connections 2024-09-16T00:35:20.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f2240a07740 1 -- 172.21.15.33:0/3462391183 wait complete. 2024-09-16T00:35:20.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.618+0000 7f2240a07740 20 client.18831 trim_cache size 0 max 16384 2024-09-16T00:35:20.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.636+0000 7f2240a07740 1 Processor -- start 2024-09-16T00:35:20.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.636+0000 7f2240a07740 1 -- start start 2024-09-16T00:35:20.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.636+0000 7f2240a07740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x55919486a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.636+0000 7f2240a07740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x559194875a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.636+0000 7f2240a07740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x559194875fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.636+0000 7f2240a07740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22381b18d0 con 0x55919485e680 2024-09-16T00:35:20.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.636+0000 7f2240a07740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f22381af1a0 con 0x559194853dc0 2024-09-16T00:35:20.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.636+0000 7f2240a07740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f22381acff0 con 0x5591948534a0 2024-09-16T00:35:20.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.636+0000 7f223cbcb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x559194875fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.636+0000 7f223cbcb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x559194875fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56262/0 (socket says 172.21.15.33:56262) 2024-09-16T00:35:20.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.636+0000 7f223cbcb640 1 -- 172.21.15.33:0/2853347321 learned_addr learned my addr 172.21.15.33:0/2853347321 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:20.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.636+0000 7f2237fff640 1 --2- 172.21.15.33:0/2853347321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x55919486a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.637+0000 7f22377fe640 1 --2- 172.21.15.33:0/2853347321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x559194875a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.637+0000 7f223cbcb640 1 -- 172.21.15.33:0/2853347321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 msgr2=0x55919486a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:20.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.637+0000 7f223cbcb640 1 --2- 172.21.15.33:0/2853347321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x55919486a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.637+0000 7f223cbcb640 1 -- 172.21.15.33:0/2853347321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 msgr2=0x559194875a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:20.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.637+0000 7f223cbcb640 1 --2- 172.21.15.33:0/2853347321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x559194875a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.637+0000 7f223cbcb640 1 -- 172.21.15.33:0/2853347321 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559194853bf0 con 0x55919485e680 2024-09-16T00:35:20.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.637+0000 7f223cbcb640 1 --2- 172.21.15.33:0/2853347321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x559194875fc0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f2230000f80 tx=0x7f2230005a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933ea3319d9fe363 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:20.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.637+0000 7f221dffb640 1 -- 172.21.15.33:0/2853347321 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2230034040 con 0x55919485e680 2024-09-16T00:35:20.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.637+0000 7f221dffb640 1 -- 172.21.15.33:0/2853347321 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f223000fc90 con 0x55919485e680 2024-09-16T00:35:20.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.637+0000 7f221dffb640 1 -- 172.21.15.33:0/2853347321 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22300052d0 con 0x55919485e680 2024-09-16T00:35:20.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.638+0000 7f2240a07740 1 -- 172.21.15.33:0/2853347321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 msgr2=0x559194875fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:20.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.638+0000 7f2240a07740 1 --2- 172.21.15.33:0/2853347321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x559194875fc0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f2230000f80 tx=0x7f2230005a80 comp rx=0 tx=0).stop 2024-09-16T00:35:20.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.638+0000 7f2240a07740 1 -- 172.21.15.33:0/2853347321 shutdown_connections 2024-09-16T00:35:20.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.638+0000 7f2240a07740 1 --2- 172.21.15.33:0/2853347321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x559194875fc0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.638+0000 7f2240a07740 1 --2- 172.21.15.33:0/2853347321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x559194875a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.638+0000 7f2240a07740 1 --2- 172.21.15.33:0/2853347321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x55919486a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.638+0000 7f2240a07740 1 -- 172.21.15.33:0/2853347321 >> 172.21.15.33:0/2853347321 conn(0x55919484fa30 msgr2=0x55919485a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:20.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.638+0000 7f2240a07740 1 -- 172.21.15.33:0/2853347321 shutdown_connections 2024-09-16T00:35:20.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.638+0000 7f2240a07740 1 -- 172.21.15.33:0/2853347321 wait complete. 2024-09-16T00:35:20.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.638+0000 7f2240a07740 1 -- start start 2024-09-16T00:35:20.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f2240a07740 1 Processor -- start 2024-09-16T00:35:20.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f2240a07740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x559194861610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f2240a07740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x559194861b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f2240a07740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x559194862090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f2240a07740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22381ad660 con 0x55919485e680 2024-09-16T00:35:20.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f2240a07740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f22381aead0 con 0x559194853dc0 2024-09-16T00:35:20.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f2240a07740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f22381add30 con 0x5591948534a0 2024-09-16T00:35:20.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f2240a07740 10 client.0 _ll_register_callbacks cb 0x7f222c00c390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:20.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f223cbcb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x559194862090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f223cbcb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x559194862090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56264/0 (socket says 172.21.15.33:56264) 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f223cbcb640 1 -- 172.21.15.33:0/1184818122 learned_addr learned my addr 172.21.15.33:0/1184818122 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f2237fff640 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x559194861610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f22377fe640 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x559194861b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f223cbcb640 1 -- 172.21.15.33:0/1184818122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 msgr2=0x559194861610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f223cbcb640 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x559194861610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f223cbcb640 1 -- 172.21.15.33:0/1184818122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 msgr2=0x559194861b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f223cbcb640 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x559194861b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f223cbcb640 1 -- 172.21.15.33:0/1184818122 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591948d6ba0 con 0x55919485e680 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.639+0000 7f2237fff640 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x559194861610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.640+0000 7f223cbcb640 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x559194862090 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f2230007580 tx=0x7f2230007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6639b16315f773 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.640+0000 7f22177fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:35:20.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.640+0000 7f22177fe640 1 -- 172.21.15.33:0/1184818122 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2230045040 con 0x55919485e680 2024-09-16T00:35:20.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.640+0000 7f22177fe640 1 -- 172.21.15.33:0/1184818122 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2230017800 con 0x55919485e680 2024-09-16T00:35:20.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.640+0000 7f2240a07740 1 -- 172.21.15.33:0/1184818122 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f223000fc90 con 0x55919485e680 2024-09-16T00:35:20.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.640+0000 7f22177fe640 1 -- 172.21.15.33:0/1184818122 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2230007a90 con 0x55919485e680 2024-09-16T00:35:20.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.640+0000 7f2240a07740 10 client.18837 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:20.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.640+0000 7f2240a07740 1 -- 172.21.15.33:0/1184818122 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5591948d7e90 con 0x55919485e680 2024-09-16T00:35:20.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.640+0000 7f22177fe640 1 -- 172.21.15.33:0/1184818122 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f223000fc90 con 0x55919485e680 2024-09-16T00:35:20.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.641+0000 7f22177fe640 1 -- 172.21.15.33:0/1184818122 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2230048020 con 0x55919485e680 2024-09-16T00:35:20.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.641+0000 7f2240a07740 10 client.18837 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:20.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.641+0000 7f2240a07740 10 client.18837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:35:20.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.641+0000 7f2240a07740 20 client.18837 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:20.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.641+0000 7f2240a07740 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55919486e000 0x5591948da2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.641+0000 7f2240a07740 4 client.18837 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.641+0000 7f2240a07740 1 -- 172.21.15.33:0/1184818122 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55919473d210 con 0x55919486e000 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.641+0000 7f2237fff640 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55919486e000 0x5591948da2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.641+0000 7f22177fe640 10 client.18837 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.641+0000 7f2237fff640 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55919486e000 0x5591948da2e0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccbe9a9f383b7177 server_cookie=a20fd89df46fe973 in_seq=0 out_seq=0 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f22177fe640 1 -- 172.21.15.33:0/1184818122 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55919473d210 con 0x55919486e000 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f22177fe640 10 client.18837 handle_command_reply: tid=0 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f2240a07740 1 client.18837 shutdown 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f2240a07740 1 -- 172.21.15.33:0/1184818122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 msgr2=0x559194862090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f2240a07740 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x559194862090 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f2230007580 tx=0x7f2230007d10 comp rx=0 tx=0).stop 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f2240a07740 1 -- 172.21.15.33:0/1184818122 shutdown_connections 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f2240a07740 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55919486e000 0x5591948da2e0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f2240a07740 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55919485e680 0x559194862090 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f2240a07740 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559194853dc0 0x559194861b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f2240a07740 1 --2- 172.21.15.33:0/1184818122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591948534a0 0x559194861610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f22177fe640 0 client.18837 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:20.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f2240a07740 1 -- 172.21.15.33:0/1184818122 >> 172.21.15.33:0/1184818122 conn(0x55919484fa30 msgr2=0x55919485ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:20.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f2240a07740 1 -- 172.21.15.33:0/1184818122 shutdown_connections 2024-09-16T00:35:20.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f2240a07740 1 -- 172.21.15.33:0/1184818122 wait complete. 2024-09-16T00:35:20.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.642+0000 7f2240a07740 20 client.18837 trim_cache size 0 max 16384 2024-09-16T00:35:20.660 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:35:20.660 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:35:20.661 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:35:20.661 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:35:20.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.643+0000 7f223e62e640 1 -- 172.21.15.33:0/1868000292 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f22200a3240 msgr2=0x7f22200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:20.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.643+0000 7f223e62e640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f22200a3240 0x7f22200a5700 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2230007fd0 tx=0x7f2230017dd0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.643+0000 7f223e62e640 1 -- 172.21.15.33:0/1868000292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2238100e20 msgr2=0x7f22381992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:20.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.643+0000 7f223e62e640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2238100e20 0x7f22381992d0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f2228006010 tx=0x7f2228003c80 comp rx=0 tx=0).stop 2024-09-16T00:35:20.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.643+0000 7f223e62e640 1 -- 172.21.15.33:0/1868000292 shutdown_connections 2024-09-16T00:35:20.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.643+0000 7f223e62e640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f22200a3240 0x7f22200a5700 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.643+0000 7f223e62e640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2238102ef0 0x7f22381a4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.643+0000 7f223e62e640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2238102570 0x7f2238199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.643+0000 7f223e62e640 1 --2- 172.21.15.33:0/1868000292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2238100e20 0x7f22381992d0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:20.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.643+0000 7f223e62e640 1 -- 172.21.15.33:0/1868000292 >> 172.21.15.33:0/1868000292 conn(0x7f22380fc7d0 msgr2=0x7f22380fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:20.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.643+0000 7f223e62e640 1 -- 172.21.15.33:0/1868000292 shutdown_connections 2024-09-16T00:35:20.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:20.643+0000 7f223e62e640 1 -- 172.21.15.33:0/1868000292 wait complete. 2024-09-16T00:35:20.904 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:35:21.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85596 15.27 MB/sec execute 147 sec latency 22787.177 ms 2024-09-16T00:35:21.350 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:35:21.350 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:35:21.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:21 smithi033 ceph-mon[31243]: pgmap v896: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-09-16T00:35:21.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.820+0000 7f75bd27a640 1 -- 172.21.15.33:0/1959890525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75b8100e20 msgr2=0x7f75b8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:21.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.820+0000 7f75bd27a640 1 --2- 172.21.15.33:0/1959890525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75b8100e20 0x7f75b8101200 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f75b0009900 tx=0x7f75b002a480 comp rx=0 tx=0).stop 2024-09-16T00:35:21.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.820+0000 7f75bd27a640 1 -- 172.21.15.33:0/1959890525 shutdown_connections 2024-09-16T00:35:21.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.820+0000 7f75bd27a640 1 --2- 172.21.15.33:0/1959890525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b8102ef0 0x7f75b810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.820+0000 7f75bd27a640 1 --2- 172.21.15.33:0/1959890525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75b8102570 0x7f75b81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.820+0000 7f75bd27a640 1 --2- 172.21.15.33:0/1959890525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75b8100e20 0x7f75b8101200 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.820+0000 7f75bd27a640 1 -- 172.21.15.33:0/1959890525 >> 172.21.15.33:0/1959890525 conn(0x7f75b80fc7d0 msgr2=0x7f75b80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:21.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.820+0000 7f75bd27a640 1 -- 172.21.15.33:0/1959890525 shutdown_connections 2024-09-16T00:35:21.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.821+0000 7f75bd27a640 1 -- 172.21.15.33:0/1959890525 wait complete. 2024-09-16T00:35:21.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.821+0000 7f75bd27a640 1 Processor -- start 2024-09-16T00:35:21.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.821+0000 7f75bd27a640 1 -- start start 2024-09-16T00:35:21.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75bd27a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75b8100e20 0x7f75b8197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75bd27a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b8102570 0x7f75b81978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75bd27a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75b8102ef0 0x7f75b819bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75bd27a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f75b8110f80 con 0x7f75b8100e20 2024-09-16T00:35:21.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75bd27a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f75b8110e00 con 0x7f75b8102570 2024-09-16T00:35:21.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75bd27a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f75b8111100 con 0x7f75b8102ef0 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75b6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75b8100e20 0x7f75b8197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75b6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75b8100e20 0x7f75b8197380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56286/0 (socket says 172.21.15.33:56286) 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75b6ffd640 1 -- 172.21.15.33:0/2262512769 learned_addr learned my addr 172.21.15.33:0/2262512769 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75b67fc640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b8102570 0x7f75b81978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75b77fe640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75b8102ef0 0x7f75b819bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75b6ffd640 1 -- 172.21.15.33:0/2262512769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75b8102ef0 msgr2=0x7f75b819bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75b6ffd640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75b8102ef0 0x7f75b819bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75b6ffd640 1 -- 172.21.15.33:0/2262512769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b8102570 msgr2=0x7f75b81978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75b6ffd640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b8102570 0x7f75b81978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75b6ffd640 1 -- 172.21.15.33:0/2262512769 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b819c330 con 0x7f75b8100e20 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75b6ffd640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75b8100e20 0x7f75b8197380 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f75b0006010 tx=0x7f75b0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.822+0000 7f75affff640 1 -- 172.21.15.33:0/2262512769 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b003f070 con 0x7f75b8100e20 2024-09-16T00:35:21.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.823+0000 7f75bd27a640 1 -- 172.21.15.33:0/2262512769 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b819c5c0 con 0x7f75b8100e20 2024-09-16T00:35:21.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.823+0000 7f75affff640 1 -- 172.21.15.33:0/2262512769 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b0031c30 con 0x7f75b8100e20 2024-09-16T00:35:21.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.823+0000 7f75affff640 1 -- 172.21.15.33:0/2262512769 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0035030 con 0x7f75b8100e20 2024-09-16T00:35:21.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.823+0000 7f75bd27a640 1 -- 172.21.15.33:0/2262512769 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b81a3f00 con 0x7f75b8100e20 2024-09-16T00:35:21.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.825+0000 7f75affff640 1 -- 172.21.15.33:0/2262512769 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f75b0042070 con 0x7f75b8100e20 2024-09-16T00:35:21.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.825+0000 7f75bf653740 1 Processor -- start 2024-09-16T00:35:21.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.825+0000 7f75bf653740 1 -- start start 2024-09-16T00:35:21.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75bf653740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17270 0x55c65bf17650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75bf653740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf17b90 0x55c65bf21f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75bf653740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf22450 0x55c65bf24840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75bf653740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c65be7a620 con 0x55c65bf17b90 2024-09-16T00:35:21.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75bf653740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c65be21e50 con 0x55c65bf17270 2024-09-16T00:35:21.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75bf653740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c65be765e0 con 0x55c65bf22450 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75b6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf17b90 0x55c65bf21f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75b77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17270 0x55c65bf17650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75b67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf22450 0x55c65bf24840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75b6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf17b90 0x55c65bf21f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56294/0 (socket says 172.21.15.33:56294) 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75b6ffd640 1 -- 172.21.15.33:0/3595801784 learned_addr learned my addr 172.21.15.33:0/3595801784 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75b67fc640 1 -- 172.21.15.33:0/3595801784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17270 msgr2=0x55c65bf17650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75b67fc640 1 --2- 172.21.15.33:0/3595801784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17270 0x55c65bf17650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75b67fc640 1 -- 172.21.15.33:0/3595801784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf17b90 msgr2=0x55c65bf21f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.826+0000 7f75b67fc640 1 --2- 172.21.15.33:0/3595801784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf17b90 0x55c65bf21f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.827+0000 7f75b67fc640 1 -- 172.21.15.33:0/3595801784 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c65be12d40 con 0x55c65bf22450 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.827+0000 7f75affff640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f75a40a3240 0x7f75a40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.827+0000 7f75b67fc640 1 --2- 172.21.15.33:0/3595801784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf22450 0x55c65bf24840 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f75a0005fd0 tx=0x7f75a000bb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=129c9c1f85299459 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:21.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.827+0000 7f75b77fe640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f75a40a3240 0x7f75a40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.827+0000 7f75affff640 1 -- 172.21.15.33:0/2262512769 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f75b00f0660 con 0x7f75b8100e20 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.827+0000 7f75ad7fa640 1 -- 172.21.15.33:0/3595801784 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0003bd0 con 0x55c65bf22450 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.827+0000 7f75ad7fa640 1 -- 172.21.15.33:0/3595801784 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a0015070 con 0x55c65bf22450 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.827+0000 7f75ad7fa640 1 -- 172.21.15.33:0/3595801784 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0004d20 con 0x55c65bf22450 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.828+0000 7f75bf653740 1 -- 172.21.15.33:0/3595801784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf22450 msgr2=0x55c65bf24840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.828+0000 7f75bf653740 1 --2- 172.21.15.33:0/3595801784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf22450 0x55c65bf24840 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f75a0005fd0 tx=0x7f75a000bb20 comp rx=0 tx=0).stop 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.828+0000 7f75b77fe640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f75a40a3240 0x7f75a40a5700 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f75a8008e80 tx=0x7f75a80085d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.828+0000 7f75bf653740 1 -- 172.21.15.33:0/3595801784 shutdown_connections 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.828+0000 7f75bf653740 1 --2- 172.21.15.33:0/3595801784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf22450 0x55c65bf24840 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.828+0000 7f75bf653740 1 --2- 172.21.15.33:0/3595801784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf17b90 0x55c65bf21f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.828+0000 7f75bf653740 1 --2- 172.21.15.33:0/3595801784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17270 0x55c65bf17650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.828+0000 7f75bf653740 1 -- 172.21.15.33:0/3595801784 >> 172.21.15.33:0/3595801784 conn(0x55c65bf13800 msgr2=0x55c65bf13be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:21.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.828+0000 7f75bf653740 1 -- 172.21.15.33:0/3595801784 shutdown_connections 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.828+0000 7f75bf653740 1 -- 172.21.15.33:0/3595801784 wait complete. 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.828+0000 7f75bf653740 1 -- start start 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.828+0000 7f75bf653740 1 Processor -- start 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75bf653740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf1d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75bf653740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf1de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75bf653740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf30230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75bf653740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c65bf11220 con 0x55c65bf22450 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75bf653740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c65be02f10 con 0x55c65bf17b90 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75bf653740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c65bf113a0 con 0x55c65bf17270 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75bf653740 10 client.0 _ll_register_callbacks cb 0x55c65be26db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf30230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf30230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56302/0 (socket says 172.21.15.33:56302) 2024-09-16T00:35:21.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75b77fe640 1 -- 172.21.15.33:0/4279380985 learned_addr learned my addr 172.21.15.33:0/4279380985 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:21.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75b6ffd640 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf1d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75b67fc640 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf1de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75b77fe640 1 -- 172.21.15.33:0/4279380985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 msgr2=0x55c65bf1d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:21.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75b77fe640 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf1d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75b77fe640 1 -- 172.21.15.33:0/4279380985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 msgr2=0x55c65bf1de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:21.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75b77fe640 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf1de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75b77fe640 1 -- 172.21.15.33:0/4279380985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c65bb89580 con 0x55c65bf22450 2024-09-16T00:35:21.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75b67fc640 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf1de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:35:21.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f75b77fe640 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf30230 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f75a800ed80 tx=0x7f75a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89138acd9fafa00d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:21.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.829+0000 7f759e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:35:21.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f759e7fc640 1 -- 172.21.15.33:0/4279380985 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a8025600 con 0x55c65bf22450 2024-09-16T00:35:21.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f75bf653740 1 -- 172.21.15.33:0/4279380985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75a0015070 con 0x55c65bf22450 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f759e7fc640 1 -- 172.21.15.33:0/4279380985 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f75a80257a0 con 0x55c65bf22450 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f759e7fc640 1 -- 172.21.15.33:0/4279380985 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f75a802e050 con 0x55c65bf22450 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f75bf653740 10 client.18849 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f75bf653740 1 -- 172.21.15.33:0/4279380985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c65bc53960 con 0x55c65bf22450 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f759e7fc640 1 -- 172.21.15.33:0/4279380985 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a0015070 con 0x55c65bf22450 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f759e7fc640 1 -- 172.21.15.33:0/4279380985 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f75a801d970 con 0x55c65bf22450 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f75bf653740 10 client.18849 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f75bf653740 10 client.18849 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f75bf653740 20 client.18849 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f75bf653740 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c65bf34230 0x55c65bf36680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f75bf653740 4 client.18849 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.830+0000 7f75bf653740 1 -- 172.21.15.33:0/4279380985 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c65be87b90 con 0x55c65bf34230 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.831+0000 7f75b6ffd640 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c65bf34230 0x55c65bf36680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.831+0000 7f759e7fc640 10 client.18849 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:21.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.831+0000 7f75b6ffd640 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c65bf34230 0x55c65bf36680 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=542000d631af5e65 server_cookie=eae212d81501bd81 in_seq=0 out_seq=0 2024-09-16T00:35:21.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f759e7fc640 1 -- 172.21.15.33:0/4279380985 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c65be87b90 con 0x55c65bf34230 2024-09-16T00:35:21.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f759e7fc640 10 client.18849 handle_command_reply: tid=0 2024-09-16T00:35:21.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f75bf653740 1 client.18849 shutdown 2024-09-16T00:35:21.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f75bf653740 1 -- 172.21.15.33:0/4279380985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 msgr2=0x55c65bf30230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:21.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f75bf653740 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf30230 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f75a800ed80 tx=0x7f75a8004a40 comp rx=0 tx=0).stop 2024-09-16T00:35:21.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f75bf653740 1 -- 172.21.15.33:0/4279380985 shutdown_connections 2024-09-16T00:35:21.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f75bf653740 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c65bf34230 0x55c65bf36680 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f75bf653740 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf30230 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f75bf653740 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf1de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f75bf653740 1 --2- 172.21.15.33:0/4279380985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf1d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f75bf653740 1 -- 172.21.15.33:0/4279380985 >> 172.21.15.33:0/4279380985 conn(0x55c65bf13800 msgr2=0x55c65bf19530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:21.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f759e7fc640 0 client.18849 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:21.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f75bf653740 1 -- 172.21.15.33:0/4279380985 shutdown_connections 2024-09-16T00:35:21.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f75bf653740 1 -- 172.21.15.33:0/4279380985 wait complete. 2024-09-16T00:35:21.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.832+0000 7f75bf653740 20 client.18849 trim_cache size 0 max 16384 2024-09-16T00:35:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.852+0000 7f75bf653740 1 Processor -- start 2024-09-16T00:35:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.852+0000 7f75bf653740 1 -- start start 2024-09-16T00:35:21.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.852+0000 7f75bf653740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf19420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.852+0000 7f75bf653740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf1ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.852+0000 7f75bf653740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf1b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75bf653740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f75b81af5d0 con 0x55c65bf22450 2024-09-16T00:35:21.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75bf653740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f75b81acea0 con 0x55c65bf17b90 2024-09-16T00:35:21.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75bf653740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f75b81aacf0 con 0x55c65bf17270 2024-09-16T00:35:21.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf1b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf1b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56314/0 (socket says 172.21.15.33:56314) 2024-09-16T00:35:21.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75b77fe640 1 -- 172.21.15.33:0/1357518528 learned_addr learned my addr 172.21.15.33:0/1357518528 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:21.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75b67fc640 1 --2- 172.21.15.33:0/1357518528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf1ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75b6ffd640 1 --2- 172.21.15.33:0/1357518528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf19420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75b77fe640 1 -- 172.21.15.33:0/1357518528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 msgr2=0x55c65bf19420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:21.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75b77fe640 1 --2- 172.21.15.33:0/1357518528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf19420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75b77fe640 1 -- 172.21.15.33:0/1357518528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 msgr2=0x55c65bf1ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:21.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75b77fe640 1 --2- 172.21.15.33:0/1357518528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf1ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75b77fe640 1 -- 172.21.15.33:0/1357518528 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c65bf32dd0 con 0x55c65bf22450 2024-09-16T00:35:21.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75b77fe640 1 --2- 172.21.15.33:0/1357518528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf1b370 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f75a800e7d0 tx=0x7f75a80086c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6640ad46a79adcc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:21.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.853+0000 7f75acff9640 1 -- 172.21.15.33:0/1357518528 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a802b040 con 0x55c65bf22450 2024-09-16T00:35:21.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.854+0000 7f75acff9640 1 -- 172.21.15.33:0/1357518528 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a800ee70 con 0x55c65bf22450 2024-09-16T00:35:21.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.854+0000 7f75acff9640 1 -- 172.21.15.33:0/1357518528 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a8025b20 con 0x55c65bf22450 2024-09-16T00:35:21.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.854+0000 7f75bf653740 1 -- 172.21.15.33:0/1357518528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 msgr2=0x55c65bf1b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:21.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.854+0000 7f75bf653740 1 --2- 172.21.15.33:0/1357518528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf1b370 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f75a800e7d0 tx=0x7f75a80086c0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.854+0000 7f75bf653740 1 -- 172.21.15.33:0/1357518528 shutdown_connections 2024-09-16T00:35:21.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.854+0000 7f75bf653740 1 --2- 172.21.15.33:0/1357518528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf1b370 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.854+0000 7f75bf653740 1 --2- 172.21.15.33:0/1357518528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf1ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.854+0000 7f75bf653740 1 --2- 172.21.15.33:0/1357518528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf19420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.854+0000 7f75bf653740 1 -- 172.21.15.33:0/1357518528 >> 172.21.15.33:0/1357518528 conn(0x55c65bf13800 msgr2=0x55c65bf18d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:21.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.855+0000 7f75bf653740 1 -- 172.21.15.33:0/1357518528 shutdown_connections 2024-09-16T00:35:21.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.855+0000 7f75bf653740 1 -- 172.21.15.33:0/1357518528 wait complete. 2024-09-16T00:35:21.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.855+0000 7f75bf653740 1 -- start start 2024-09-16T00:35:21.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.855+0000 7f75bf653740 1 Processor -- start 2024-09-16T00:35:21.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75bf653740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf30250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75bf653740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf30790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75bf653740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf2b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75bf653740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f75b81ab360 con 0x55c65bf22450 2024-09-16T00:35:21.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75bf653740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f75b81ac7d0 con 0x55c65bf17b90 2024-09-16T00:35:21.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75bf653740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f75b81aba30 con 0x55c65bf17270 2024-09-16T00:35:21.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75bf653740 10 client.0 _ll_register_callbacks cb 0x7f75a800ea30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:21.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf2b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b67fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf30790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf2b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56320/0 (socket says 172.21.15.33:56320) 2024-09-16T00:35:21.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b77fe640 1 -- 172.21.15.33:0/1225616420 learned_addr learned my addr 172.21.15.33:0/1225616420 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:21.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b6ffd640 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf30250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b77fe640 1 -- 172.21.15.33:0/1225616420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 msgr2=0x55c65bf30250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:21.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b77fe640 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf30250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b77fe640 1 -- 172.21.15.33:0/1225616420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 msgr2=0x55c65bf30790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:21.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b77fe640 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf30790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b77fe640 1 -- 172.21.15.33:0/1225616420 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c65bf2ba30 con 0x55c65bf22450 2024-09-16T00:35:21.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b6ffd640 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf30250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:35:21.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b67fc640 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf30790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:21.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f75b77fe640 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf2b200 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f75a801deb0 tx=0x7f75a8008b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be75327e06ba249f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:21.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.856+0000 7f759e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:35:21.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f759e7fc640 1 -- 172.21.15.33:0/1225616420 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a80257a0 con 0x55c65bf22450 2024-09-16T00:35:21.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f759e7fc640 1 -- 172.21.15.33:0/1225616420 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f75a8025940 con 0x55c65bf22450 2024-09-16T00:35:21.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f75bf653740 1 -- 172.21.15.33:0/1225616420 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75a800ee70 con 0x55c65bf22450 2024-09-16T00:35:21.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f759e7fc640 1 -- 172.21.15.33:0/1225616420 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f75a804b050 con 0x55c65bf22450 2024-09-16T00:35:21.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f75bf653740 10 client.18855 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:21.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f75bf653740 1 -- 172.21.15.33:0/1225616420 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c65bf9fa40 con 0x55c65bf22450 2024-09-16T00:35:21.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f759e7fc640 1 -- 172.21.15.33:0/1225616420 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a800ee70 con 0x55c65bf22450 2024-09-16T00:35:21.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f759e7fc640 1 -- 172.21.15.33:0/1225616420 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f75a8005040 con 0x55c65bf22450 2024-09-16T00:35:21.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f75bf653740 10 client.18855 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:21.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f75bf653740 10 client.18855 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:35:21.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f75bf653740 20 client.18855 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:21.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f75bf653740 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c65bfa0020 0x55c65bfa2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:21.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f75bf653740 4 client.18855 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3ec5e571", "release": true, "await": true}] 2024-09-16T00:35:21.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.857+0000 7f75bf653740 1 -- 172.21.15.33:0/1225616420 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3ec5e571", "release": true, "await": true}) -- 0x55c65be820a0 con 0x55c65bfa0020 2024-09-16T00:35:21.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.858+0000 7f75b6ffd640 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c65bfa0020 0x55c65bfa2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:21.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.858+0000 7f75b6ffd640 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c65bfa0020 0x55c65bfa2440 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b11d5ebab9f99446 server_cookie=7c1fea4a361fa09a in_seq=0 out_seq=0 2024-09-16T00:35:21.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.858+0000 7f759e7fc640 10 client.18855 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:21.872 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:35:21.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f759e7fc640 1 -- 172.21.15.33:0/1225616420 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c65be820a0 con 0x55c65bfa0020 2024-09-16T00:35:21.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f759e7fc640 10 client.18855 handle_command_reply: tid=0 2024-09-16T00:35:21.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f75bf653740 1 client.18855 shutdown 2024-09-16T00:35:21.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f75bf653740 1 -- 172.21.15.33:0/1225616420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 msgr2=0x55c65bf2b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:21.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f75bf653740 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf2b200 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f75a801deb0 tx=0x7f75a8008b90 comp rx=0 tx=0).stop 2024-09-16T00:35:21.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f75bf653740 1 -- 172.21.15.33:0/1225616420 shutdown_connections 2024-09-16T00:35:21.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f75bf653740 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c65bfa0020 0x55c65bfa2440 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f75bf653740 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65bf22450 0x55c65bf2b200 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f75bf653740 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c65bf17b90 0x55c65bf30790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f759e7fc640 0 client.18855 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:21.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f75bf653740 1 --2- 172.21.15.33:0/1225616420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c65bf17270 0x55c65bf30250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f75bf653740 1 -- 172.21.15.33:0/1225616420 >> 172.21.15.33:0/1225616420 conn(0x55c65bf13800 msgr2=0x55c65bf217d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:21.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f75bf653740 1 -- 172.21.15.33:0/1225616420 shutdown_connections 2024-09-16T00:35:21.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f75bf653740 1 -- 172.21.15.33:0/1225616420 wait complete. 2024-09-16T00:35:21.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.915+0000 7f75bf653740 20 client.18855 trim_cache size 0 max 16384 2024-09-16T00:35:21.918 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:35:21.918 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:35:21.918 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:35:21.918 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 68, 2024-09-16T00:35:21.918 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:35:21.918 INFO:teuthology.orchestra.run.smithi033.stdout: "3ec5e571": { 2024-09-16T00:35:21.918 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 68, 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 142.9, 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:35:21.919 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:35:21.920 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:35:21.920 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:35:21.920 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:35:21.920 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:35:21.920 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:35:21.920 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:35:21.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.916+0000 7f75bd27a640 1 -- 172.21.15.33:0/2262512769 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f75a40a3240 msgr2=0x7f75a40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.916+0000 7f75bd27a640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f75a40a3240 0x7f75a40a5700 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f75a8008e80 tx=0x7f75a80085d0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.916+0000 7f75bd27a640 1 -- 172.21.15.33:0/2262512769 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75b8100e20 msgr2=0x7f75b8197380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.916+0000 7f75bd27a640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75b8100e20 0x7f75b8197380 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f75b0006010 tx=0x7f75b0003c10 comp rx=0 tx=0).stop 2024-09-16T00:35:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.916+0000 7f75bd27a640 1 -- 172.21.15.33:0/2262512769 shutdown_connections 2024-09-16T00:35:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.916+0000 7f75bd27a640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f75a40a3240 0x7f75a40a5700 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.916+0000 7f75bd27a640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75b8102ef0 0x7f75b819bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.916+0000 7f75bd27a640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b8102570 0x7f75b81978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.916+0000 7f75bd27a640 1 --2- 172.21.15.33:0/2262512769 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75b8100e20 0x7f75b8197380 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.916+0000 7f75bd27a640 1 -- 172.21.15.33:0/2262512769 >> 172.21.15.33:0/2262512769 conn(0x7f75b80fc7d0 msgr2=0x7f75b80fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.916+0000 7f75bd27a640 1 -- 172.21.15.33:0/2262512769 shutdown_connections 2024-09-16T00:35:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:21.916+0000 7f75bd27a640 1 -- 172.21.15.33:0/2262512769 wait complete. 2024-09-16T00:35:21.922 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:21 smithi083 ceph-mon[35242]: pgmap v896: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-09-16T00:35:21.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:21 smithi060 ceph-mon[35185]: pgmap v896: 129 pgs: 129 active+clean; 541 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-09-16T00:35:22.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 85671 15.17 MB/sec execute 148 sec latency 23647.949 ms 2024-09-16T00:35:22.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.738+0000 7fd2c37c2640 1 -- 172.21.15.33:0/406581336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2bc1036e0 msgr2=0x7fd2bc107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:22.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.738+0000 7fd2c37c2640 1 --2- 172.21.15.33:0/406581336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2bc1036e0 0x7fd2bc107b90 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009930 tx=0x7fd2ac02a420 comp rx=0 tx=0).stop 2024-09-16T00:35:22.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.738+0000 7fd2c37c2640 1 -- 172.21.15.33:0/406581336 shutdown_connections 2024-09-16T00:35:22.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.738+0000 7fd2c37c2640 1 --2- 172.21.15.33:0/406581336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2bc1082a0 0x7fd2bc10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:22.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.738+0000 7fd2c37c2640 1 --2- 172.21.15.33:0/406581336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2bc1036e0 0x7fd2bc107b90 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:22.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.738+0000 7fd2c37c2640 1 --2- 172.21.15.33:0/406581336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2bc102d30 0x7fd2bc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:22.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.738+0000 7fd2c37c2640 1 -- 172.21.15.33:0/406581336 >> 172.21.15.33:0/406581336 conn(0x7fd2bc0fc7d0 msgr2=0x7fd2bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:22.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.739+0000 7fd2c37c2640 1 -- 172.21.15.33:0/406581336 shutdown_connections 2024-09-16T00:35:22.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.739+0000 7fd2c37c2640 1 -- 172.21.15.33:0/406581336 wait complete. 2024-09-16T00:35:22.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.739+0000 7fd2c37c2640 1 Processor -- start 2024-09-16T00:35:22.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.739+0000 7fd2c37c2640 1 -- start start 2024-09-16T00:35:22.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c37c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2bc102d30 0x7fd2bc1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:22.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c37c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2bc1036e0 0x7fd2bc198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:22.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c37c2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2bc1082a0 0x7fd2bc19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:22.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c37c2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1120a0 con 0x7fd2bc1036e0 2024-09-16T00:35:22.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c37c2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc111f20 con 0x7fd2bc1082a0 2024-09-16T00:35:22.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c37c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc112220 con 0x7fd2bc102d30 2024-09-16T00:35:22.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c0d5d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2bc1036e0 0x7fd2bc198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:22.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c0d5d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2bc1036e0 0x7fd2bc198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56342/0 (socket says 172.21.15.33:56342) 2024-09-16T00:35:22.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c0d5d640 1 -- 172.21.15.33:0/1307260173 learned_addr learned my addr 172.21.15.33:0/1307260173 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:22.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c1d5f640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2bc1082a0 0x7fd2bc19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:22.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c155e640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2bc102d30 0x7fd2bc1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:22.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c0d5d640 1 -- 172.21.15.33:0/1307260173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2bc102d30 msgr2=0x7fd2bc1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:22.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c0d5d640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2bc102d30 0x7fd2bc1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:22.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c0d5d640 1 -- 172.21.15.33:0/1307260173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2bc1082a0 msgr2=0x7fd2bc19ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:22.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c0d5d640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2bc1082a0 0x7fd2bc19ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:22.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c0d5d640 1 -- 172.21.15.33:0/1307260173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2bc19d570 con 0x7fd2bc1036e0 2024-09-16T00:35:22.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.740+0000 7fd2c1d5f640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2bc1082a0 0x7fd2bc19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:22.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.741+0000 7fd2c0d5d640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2bc1036e0 0x7fd2bc198b00 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac005940 tx=0x7fd2ac0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:22.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.741+0000 7fd2b27fc640 1 -- 172.21.15.33:0/1307260173 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac03f030 con 0x7fd2bc1036e0 2024-09-16T00:35:22.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.741+0000 7fd2c37c2640 1 -- 172.21.15.33:0/1307260173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2bc19d800 con 0x7fd2bc1036e0 2024-09-16T00:35:22.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.741+0000 7fd2b27fc640 1 -- 172.21.15.33:0/1307260173 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2ac0044c0 con 0x7fd2bc1036e0 2024-09-16T00:35:22.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.741+0000 7fd2b27fc640 1 -- 172.21.15.33:0/1307260173 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac035070 con 0x7fd2bc1036e0 2024-09-16T00:35:22.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.741+0000 7fd2c37c2640 1 -- 172.21.15.33:0/1307260173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2bc1a5140 con 0x7fd2bc1036e0 2024-09-16T00:35:22.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.742+0000 7fd2c37c2640 1 -- 172.21.15.33:0/1307260173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2bc103110 con 0x7fd2bc1036e0 2024-09-16T00:35:22.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.743+0000 7fd2b27fc640 1 -- 172.21.15.33:0/1307260173 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd2ac042030 con 0x7fd2bc1036e0 2024-09-16T00:35:22.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.744+0000 7fd2b27fc640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2a80a3240 0x7fd2a80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:22.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.744+0000 7fd2c155e640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2a80a3240 0x7fd2a80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:22.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.744+0000 7fd2b27fc640 1 -- 172.21.15.33:0/1307260173 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd2ac0efda0 con 0x7fd2bc1036e0 2024-09-16T00:35:22.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.745+0000 7fd2c155e640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2a80a3240 0x7fd2a80a5700 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009bd0 tx=0x7fd2b8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:22.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:22.749+0000 7fd2b27fc640 1 -- 172.21.15.33:0/1307260173 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd2ac0bbff0 con 0x7fd2bc1036e0 2024-09-16T00:35:23.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.058+0000 7fd2c37c2640 1 -- 172.21.15.33:0/1307260173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2bc10a2b0 con 0x7fd2bc1036e0 2024-09-16T00:35:23.062 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:35:23.062 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:35:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.060+0000 7fd2b27fc640 1 -- 172.21.15.33:0/1307260173 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7fd2ac0c02f0 con 0x7fd2bc1036e0 2024-09-16T00:35:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:35:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.064+0000 7fd2c37c2640 1 -- 172.21.15.33:0/1307260173 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2a80a3240 msgr2=0x7fd2a80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.064+0000 7fd2c37c2640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2a80a3240 0x7fd2a80a5700 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009bd0 tx=0x7fd2b8005f90 comp rx=0 tx=0).stop 2024-09-16T00:35:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.064+0000 7fd2c37c2640 1 -- 172.21.15.33:0/1307260173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2bc1036e0 msgr2=0x7fd2bc198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.064+0000 7fd2c37c2640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2bc1036e0 0x7fd2bc198b00 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac005940 tx=0x7fd2ac0040c0 comp rx=0 tx=0).stop 2024-09-16T00:35:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.065+0000 7fd2c37c2640 1 -- 172.21.15.33:0/1307260173 shutdown_connections 2024-09-16T00:35:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.065+0000 7fd2c37c2640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd2a80a3240 0x7fd2a80a5700 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.065+0000 7fd2c37c2640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2bc1082a0 0x7fd2bc19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.065+0000 7fd2c37c2640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2bc1036e0 0x7fd2bc198b00 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.065+0000 7fd2c37c2640 1 --2- 172.21.15.33:0/1307260173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2bc102d30 0x7fd2bc1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.065+0000 7fd2c37c2640 1 -- 172.21.15.33:0/1307260173 >> 172.21.15.33:0/1307260173 conn(0x7fd2bc0fc7d0 msgr2=0x7fd2bc105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.066+0000 7fd2c37c2640 1 -- 172.21.15.33:0/1307260173 shutdown_connections 2024-09-16T00:35:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:23.066+0000 7fd2c37c2640 1 -- 172.21.15.33:0/1307260173 wait complete. 2024-09-16T00:35:23.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 86227 15.32 MB/sec execute 149 sec latency 140.795 ms 2024-09-16T00:35:23.406 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '3ec5e571', total seconds elapsed: 28.2 2024-09-16T00:35:23.759 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:23 smithi033 ceph-mon[31243]: pgmap v897: 129 pgs: 129 active+clean; 550 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.2 MiB/s wr, 26 op/s 2024-09-16T00:35:23.759 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:23 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1307260173' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:35:23.782 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:23 smithi060 ceph-mon[35185]: pgmap v897: 129 pgs: 129 active+clean; 550 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.2 MiB/s wr, 26 op/s 2024-09-16T00:35:23.782 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:23 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1307260173' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:35:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:23 smithi083 ceph-mon[35242]: pgmap v897: 129 pgs: 129 active+clean; 550 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.2 MiB/s wr, 26 op/s 2024-09-16T00:35:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:23 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1307260173' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:35:24.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 86799 15.38 MB/sec execute 150 sec latency 64.858 ms 2024-09-16T00:35:24.240 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:35:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:35:25.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 87414 15.52 MB/sec execute 151 sec latency 94.211 ms 2024-09-16T00:35:25.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.105+0000 7f7a2c549640 1 -- 172.21.15.33:0/3670403320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a241082a0 msgr2=0x7f7a2410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:25.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.105+0000 7f7a2c549640 1 --2- 172.21.15.33:0/3670403320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a241082a0 0x7f7a2410ec40 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009cd0 tx=0x7f7a1c02a8e0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.105+0000 7f7a2c549640 1 -- 172.21.15.33:0/3670403320 shutdown_connections 2024-09-16T00:35:25.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.105+0000 7f7a2c549640 1 --2- 172.21.15.33:0/3670403320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a241082a0 0x7f7a2410ec40 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.105+0000 7f7a2c549640 1 --2- 172.21.15.33:0/3670403320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a241036e0 0x7f7a24107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.105+0000 7f7a2c549640 1 --2- 172.21.15.33:0/3670403320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a24102d30 0x7f7a24103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.105+0000 7f7a2c549640 1 -- 172.21.15.33:0/3670403320 >> 172.21.15.33:0/3670403320 conn(0x7f7a240fc7d0 msgr2=0x7f7a240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:25.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.105+0000 7f7a2c549640 1 -- 172.21.15.33:0/3670403320 shutdown_connections 2024-09-16T00:35:25.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.105+0000 7f7a2c549640 1 -- 172.21.15.33:0/3670403320 wait complete. 2024-09-16T00:35:25.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.106+0000 7f7a2c549640 1 Processor -- start 2024-09-16T00:35:25.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.106+0000 7f7a2c549640 1 -- start start 2024-09-16T00:35:25.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.107+0000 7f7a2c549640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a24102d30 0x7f7a24100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.107+0000 7f7a2c549640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a241036e0 0x7f7a24100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.107+0000 7f7a2c549640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a241082a0 0x7f7a24101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.107+0000 7f7a2c549640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7a241120d0 con 0x7f7a241082a0 2024-09-16T00:35:25.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.107+0000 7f7a2c549640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7a24111f50 con 0x7f7a241036e0 2024-09-16T00:35:25.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.107+0000 7f7a2c549640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7a24112250 con 0x7f7a24102d30 2024-09-16T00:35:25.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.107+0000 7f7a29ae4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a241036e0 0x7f7a24100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.107+0000 7f7a2a2e5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a24102d30 0x7f7a24100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.107+0000 7f7a29ae4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a241036e0 0x7f7a24100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:34058/0 (socket says 172.21.15.33:34058) 2024-09-16T00:35:25.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.107+0000 7f7a29ae4640 1 -- 172.21.15.33:0/1895500281 learned_addr learned my addr 172.21.15.33:0/1895500281 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:25.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.108+0000 7f7a2a2e5640 1 -- 172.21.15.33:0/1895500281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a241036e0 msgr2=0x7f7a24100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:25.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.108+0000 7f7a2a2e5640 1 --2- 172.21.15.33:0/1895500281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a241036e0 0x7f7a24100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.108+0000 7f7a2a2e5640 1 -- 172.21.15.33:0/1895500281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a241082a0 msgr2=0x7f7a24101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:25.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.108+0000 7f7a2a2e5640 1 --2- 172.21.15.33:0/1895500281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a241082a0 0x7f7a24101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.108+0000 7f7a2a2e5640 1 -- 172.21.15.33:0/1895500281 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a241a6a70 con 0x7f7a24102d30 2024-09-16T00:35:25.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.108+0000 7f7a2a2e5640 1 --2- 172.21.15.33:0/1895500281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a24102d30 0x7f7a24100990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7a2000d6d0 tx=0x7f7a2000db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:25.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.109+0000 7f7a1b7fe640 1 -- 172.21.15.33:0/1895500281 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20017070 con 0x7f7a24102d30 2024-09-16T00:35:25.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.109+0000 7f7a1b7fe640 1 -- 172.21.15.33:0/1895500281 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a20009820 con 0x7f7a24102d30 2024-09-16T00:35:25.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.109+0000 7f7a2c549640 1 -- 172.21.15.33:0/1895500281 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a24198370 con 0x7f7a24102d30 2024-09-16T00:35:25.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.109+0000 7f7a1b7fe640 1 -- 172.21.15.33:0/1895500281 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20004ce0 con 0x7f7a24102d30 2024-09-16T00:35:25.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.109+0000 7f7a2c549640 1 -- 172.21.15.33:0/1895500281 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a24198900 con 0x7f7a24102d30 2024-09-16T00:35:25.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.112+0000 7f7a2e922740 1 Processor -- start 2024-09-16T00:35:25.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a2e922740 1 -- start start 2024-09-16T00:35:25.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a1b7fe640 1 -- 172.21.15.33:0/1895500281 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7a200099d0 con 0x7f7a24102d30 2024-09-16T00:35:25.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a2e922740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da4f7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a2e922740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da4f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a2e922740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da502370 0x55c7da504760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a2e922740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c7da45d2f0 con 0x55c7da502370 2024-09-16T00:35:25.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a2e922740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c7da404cf0 con 0x55c7da4f75d0 2024-09-16T00:35:25.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a2e922740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c7da4592b0 con 0x55c7da4f8b40 2024-09-16T00:35:25.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a2aae6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da4f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a2aae6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da4f7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:45558/0 (socket says 172.21.15.33:45558) 2024-09-16T00:35:25.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a2aae6640 1 -- 172.21.15.33:0/1324364754 learned_addr learned my addr 172.21.15.33:0/1324364754 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:25.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a2a2e5640 1 --2- 172.21.15.33:0/1324364754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da4f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.113+0000 7f7a29ae4640 1 --2- 172.21.15.33:0/1324364754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da502370 0x55c7da504760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.114+0000 7f7a2aae6640 1 -- 172.21.15.33:0/1324364754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 msgr2=0x55c7da4f7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:25.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.114+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/1324364754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da4f7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.114+0000 7f7a2aae6640 1 -- 172.21.15.33:0/1324364754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da502370 msgr2=0x55c7da504760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:25.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.114+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/1324364754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da502370 0x55c7da504760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.114+0000 7f7a2aae6640 1 -- 172.21.15.33:0/1324364754 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7da3e93a0 con 0x55c7da4f8b40 2024-09-16T00:35:25.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.114+0000 7f7a29ae4640 1 --2- 172.21.15.33:0/1324364754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da502370 0x55c7da504760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:35:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.114+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/1324364754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da4f7090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009ca0 tx=0x7f7a1c009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f078276afefb16a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.115+0000 7f7a18ff9640 1 -- 172.21.15.33:0/1324364754 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c03f070 con 0x55c7da4f8b40 2024-09-16T00:35:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.115+0000 7f7a1b7fe640 1 --2- 172.21.15.33:0/1895500281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7a100a32c0 0x7f7a100a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.115+0000 7f7a18ff9640 1 -- 172.21.15.33:0/1324364754 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a1c036040 con 0x55c7da4f8b40 2024-09-16T00:35:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.115+0000 7f7a18ff9640 1 -- 172.21.15.33:0/1324364754 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c0042a0 con 0x55c7da4f8b40 2024-09-16T00:35:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.115+0000 7f7a1b7fe640 1 -- 172.21.15.33:0/1895500281 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7a200cf6b0 con 0x7f7a24102d30 2024-09-16T00:35:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.115+0000 7f7a2a2e5640 1 --2- 172.21.15.33:0/1895500281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7a100a32c0 0x7f7a100a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.116+0000 7f7a2e922740 1 -- 172.21.15.33:0/1324364754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 msgr2=0x55c7da4f7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.116+0000 7f7a2e922740 1 --2- 172.21.15.33:0/1324364754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da4f7090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009ca0 tx=0x7f7a1c009a20 comp rx=0 tx=0).stop 2024-09-16T00:35:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.116+0000 7f7a2e922740 1 -- 172.21.15.33:0/1324364754 shutdown_connections 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.116+0000 7f7a2e922740 1 --2- 172.21.15.33:0/1324364754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da502370 0x55c7da504760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.116+0000 7f7a2e922740 1 --2- 172.21.15.33:0/1324364754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da4f7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.116+0000 7f7a2e922740 1 --2- 172.21.15.33:0/1324364754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da4f7090 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.116+0000 7f7a2e922740 1 -- 172.21.15.33:0/1324364754 >> 172.21.15.33:0/1324364754 conn(0x55c7da4c0310 msgr2=0x55c7da4f3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.116+0000 7f7a2e922740 1 -- 172.21.15.33:0/1324364754 shutdown_connections 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.116+0000 7f7a2e922740 1 -- 172.21.15.33:0/1324364754 wait complete. 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.117+0000 7f7a2a2e5640 1 --2- 172.21.15.33:0/1895500281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7a100a32c0 0x7f7a100a5780 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7a200cd0a0 tx=0x7f7a200d5550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.117+0000 7f7a2e922740 1 -- start start 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.118+0000 7f7a2e922740 1 Processor -- start 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.118+0000 7f7a2e922740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da501dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.118+0000 7f7a2e922740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f8b40 0x55c7da4ff070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.118+0000 7f7a29ae4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f8b40 0x55c7da4ff070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.118+0000 7f7a29ae4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f8b40 0x55c7da4ff070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56376/0 (socket says 172.21.15.33:56376) 2024-09-16T00:35:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.118+0000 7f7a2aae6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da501dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a2aae6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da501dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:34062/0 (socket says 172.21.15.33:34062) 2024-09-16T00:35:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a29ae4640 1 -- 172.21.15.33:0/2825736495 learned_addr learned my addr 172.21.15.33:0/2825736495 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a2e922740 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da502370 0x55c7da4ff800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a2e922740 1 -- 172.21.15.33:0/2825736495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c7da505010 con 0x55c7da4f8b40 2024-09-16T00:35:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a2e922740 1 -- 172.21.15.33:0/2825736495 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c7da418060 con 0x55c7da4f75d0 2024-09-16T00:35:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a2e922740 1 -- 172.21.15.33:0/2825736495 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c7da505190 con 0x55c7da502370 2024-09-16T00:35:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a29ae4640 1 -- 172.21.15.33:0/2825736495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da502370 msgr2=0x55c7da4ff800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:35:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a29ae4640 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da502370 0x55c7da4ff800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a29ae4640 1 -- 172.21.15.33:0/2825736495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 msgr2=0x55c7da501dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a29ae4640 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da501dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a29ae4640 1 -- 172.21.15.33:0/2825736495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7da1ce3c0 con 0x55c7da4f8b40 2024-09-16T00:35:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da501dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:35:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a29ae4640 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f8b40 0x55c7da4ff070 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f7a14009990 tx=0x7f7a14009e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a2b5964df5e078 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.119+0000 7f7a2e922740 10 client.0 _ll_register_callbacks cb 0x55c7da16b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.120+0000 7f7a2e922740 1 -- 172.21.15.33:0/2825736495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a1c036040 con 0x55c7da4f8b40 2024-09-16T00:35:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.120+0000 7f7a0dffb640 10 client.18867 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:35:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.120+0000 7f7a0dffb640 1 -- 172.21.15.33:0/2825736495 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a14026060 con 0x55c7da4f8b40 2024-09-16T00:35:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.120+0000 7f7a0dffb640 1 -- 172.21.15.33:0/2825736495 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7a14003e10 con 0x55c7da4f8b40 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.120+0000 7f7a0dffb640 1 -- 172.21.15.33:0/2825736495 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7a14021400 con 0x55c7da4f8b40 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.120+0000 7f7a2e922740 10 client.18867 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.120+0000 7f7a2e922740 1 -- 172.21.15.33:0/2825736495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({fsmap=33}) -- 0x55c7da5109a0 con 0x55c7da4f8b40 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.121+0000 7f7a0dffb640 1 -- 172.21.15.33:0/2825736495 <== mon.0 v2:172.21.15.33:3300/0 4 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7a1402a3d0 con 0x55c7da4f8b40 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.121+0000 7f7a2e922740 10 client.18867 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.121+0000 7f7a2e922740 10 client.18867 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.121+0000 7f7a2e922740 20 client.18867 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.121+0000 7f7a2e922740 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c7da512230 0x55c7da5146f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.121+0000 7f7a2e922740 4 client.18867 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.121+0000 7f7a2e922740 1 -- 172.21.15.33:0/2825736495 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7da46a860 con 0x55c7da512230 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.121+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c7da512230 0x55c7da5146f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.121+0000 7f7a0dffb640 10 client.18867 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:35:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.121+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c7da512230 0x55c7da5146f0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=253f942d4d348583 server_cookie=25fcf3d0ebe317ae in_seq=0 out_seq=0 2024-09-16T00:35:25.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.126+0000 7f7a0dffb640 1 -- 172.21.15.33:0/2825736495 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c7da46a860 con 0x55c7da512230 2024-09-16T00:35:25.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.126+0000 7f7a0dffb640 10 client.18867 handle_command_reply: tid=0 2024-09-16T00:35:25.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.126+0000 7f7a2e922740 1 client.18867 shutdown 2024-09-16T00:35:25.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.127+0000 7f7a2e922740 1 -- 172.21.15.33:0/2825736495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f8b40 msgr2=0x55c7da4ff070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:25.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.127+0000 7f7a2e922740 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f8b40 0x55c7da4ff070 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f7a14009990 tx=0x7f7a14009e50 comp rx=0 tx=0).stop 2024-09-16T00:35:25.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.127+0000 7f7a2e922740 1 -- 172.21.15.33:0/2825736495 shutdown_connections 2024-09-16T00:35:25.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.127+0000 7f7a2e922740 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c7da512230 0x55c7da5146f0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.127+0000 7f7a2e922740 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da502370 0x55c7da4ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.127+0000 7f7a2e922740 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f8b40 0x55c7da4ff070 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.127+0000 7f7a2e922740 1 --2- 172.21.15.33:0/2825736495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da501dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.127+0000 7f7a0dffb640 0 client.18867 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:35:25.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.127+0000 7f7a2e922740 1 -- 172.21.15.33:0/2825736495 >> 172.21.15.33:0/2825736495 conn(0x55c7da4c0310 msgr2=0x55c7da4f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:25.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.127+0000 7f7a2e922740 1 -- 172.21.15.33:0/2825736495 shutdown_connections 2024-09-16T00:35:25.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.127+0000 7f7a2e922740 1 -- 172.21.15.33:0/2825736495 wait complete. 2024-09-16T00:35:25.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.127+0000 7f7a2e922740 20 client.18867 trim_cache size 0 max 16384 2024-09-16T00:35:25.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.145+0000 7f7a2e922740 1 Processor -- start 2024-09-16T00:35:25.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.145+0000 7f7a2e922740 1 -- start start 2024-09-16T00:35:25.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.145+0000 7f7a2e922740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da4f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.145+0000 7f7a2e922740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da50d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.145+0000 7f7a2e922740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da502370 0x55c7da50def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.145+0000 7f7a2e922740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7a241af960 con 0x55c7da502370 2024-09-16T00:35:25.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.145+0000 7f7a2e922740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7a241ad230 con 0x55c7da4f75d0 2024-09-16T00:35:25.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.145+0000 7f7a2e922740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7a241ab080 con 0x55c7da4f8b40 2024-09-16T00:35:25.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.146+0000 7f7a2aae6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da4f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.146+0000 7f7a29ae4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da50d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.146+0000 7f7a2aae6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da4f3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:34070/0 (socket says 172.21.15.33:34070) 2024-09-16T00:35:25.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.146+0000 7f7a2aae6640 1 -- 172.21.15.33:0/1402422240 learned_addr learned my addr 172.21.15.33:0/1402422240 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:25.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.146+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/1402422240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da502370 0x55c7da50def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.146+0000 7f7a2aae6640 1 -- 172.21.15.33:0/1402422240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 msgr2=0x55c7da50d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:25.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.146+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/1402422240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da50d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.146+0000 7f7a2aae6640 1 -- 172.21.15.33:0/1402422240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da502370 msgr2=0x55c7da50def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:25.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.146+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/1402422240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da502370 0x55c7da50def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.146+0000 7f7a2aae6640 1 -- 172.21.15.33:0/1402422240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7da505180 con 0x55c7da4f75d0 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.146+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/1402422240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da4f3d70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c006010 tx=0x7f7a1c003db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5f835599d0f0037 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.146+0000 7f7a0ffff640 1 -- 172.21.15.33:0/1402422240 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c03f040 con 0x55c7da4f75d0 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.147+0000 7f7a0ffff640 1 -- 172.21.15.33:0/1402422240 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a1c0314e0 con 0x55c7da4f75d0 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.147+0000 7f7a0ffff640 1 -- 172.21.15.33:0/1402422240 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c004420 con 0x55c7da4f75d0 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.147+0000 7f7a2e922740 1 -- 172.21.15.33:0/1402422240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 msgr2=0x55c7da4f3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.147+0000 7f7a2e922740 1 --2- 172.21.15.33:0/1402422240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da4f3d70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c006010 tx=0x7f7a1c003db0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.147+0000 7f7a2e922740 1 -- 172.21.15.33:0/1402422240 shutdown_connections 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.147+0000 7f7a2e922740 1 --2- 172.21.15.33:0/1402422240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da502370 0x55c7da50def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.147+0000 7f7a2e922740 1 --2- 172.21.15.33:0/1402422240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da50d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.147+0000 7f7a2e922740 1 --2- 172.21.15.33:0/1402422240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da4f75d0 0x55c7da4f3d70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.147+0000 7f7a2e922740 1 -- 172.21.15.33:0/1402422240 >> 172.21.15.33:0/1402422240 conn(0x55c7da4c0310 msgr2=0x55c7da50d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.147+0000 7f7a2e922740 1 -- 172.21.15.33:0/1402422240 shutdown_connections 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a2e922740 1 -- 172.21.15.33:0/1402422240 wait complete. 2024-09-16T00:35:25.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a2e922740 1 -- start start 2024-09-16T00:35:25.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a2e922740 1 Processor -- start 2024-09-16T00:35:25.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a2e922740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f75d0 0x55c7da50a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a2e922740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da505bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a2e922740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da502370 0x55c7da5060f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a2e922740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7a241ab6f0 con 0x55c7da4f75d0 2024-09-16T00:35:25.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a2e922740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7a241acb60 con 0x55c7da502370 2024-09-16T00:35:25.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a2e922740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7a241abdc0 con 0x55c7da4f8b40 2024-09-16T00:35:25.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a2e922740 10 client.0 _ll_register_callbacks cb 0x7f7a14004550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:25.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a2aae6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f75d0 0x55c7da50a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a29ae4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da505bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a29ae4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da505bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:45568/0 (socket says 172.21.15.33:45568) 2024-09-16T00:35:25.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a2aae6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f75d0 0x55c7da50a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56396/0 (socket says 172.21.15.33:56396) 2024-09-16T00:35:25.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.148+0000 7f7a29ae4640 1 -- 172.21.15.33:0/471250178 learned_addr learned my addr 172.21.15.33:0/471250178 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:25.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.149+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/471250178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da502370 0x55c7da5060f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.149+0000 7f7a2aae6640 1 -- 172.21.15.33:0/471250178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 msgr2=0x55c7da505bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:25.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.149+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/471250178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da505bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.149+0000 7f7a2aae6640 1 -- 172.21.15.33:0/471250178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da502370 msgr2=0x55c7da5060f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:25.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.149+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/471250178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da502370 0x55c7da5060f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.149+0000 7f7a2aae6640 1 -- 172.21.15.33:0/471250178 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7da506630 con 0x55c7da4f75d0 2024-09-16T00:35:25.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.149+0000 7f7a2aae6640 1 --2- 172.21.15.33:0/471250178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f75d0 0x55c7da50a880 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009bf0 tx=0x7f7a1c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29645a61514270cb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:25.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.149+0000 7f7a0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:35:25.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.149+0000 7f7a0dffb640 1 -- 172.21.15.33:0/471250178 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c05e060 con 0x55c7da4f75d0 2024-09-16T00:35:25.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.149+0000 7f7a2e922740 1 -- 172.21.15.33:0/471250178 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a1c0314e0 con 0x55c7da4f75d0 2024-09-16T00:35:25.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.149+0000 7f7a0dffb640 1 -- 172.21.15.33:0/471250178 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7a1c02e070 con 0x55c7da4f75d0 2024-09-16T00:35:25.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.150+0000 7f7a0dffb640 1 -- 172.21.15.33:0/471250178 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7a1c03f040 con 0x55c7da4f75d0 2024-09-16T00:35:25.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.150+0000 7f7a2e922740 10 client.18873 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.150+0000 7f7a2e922740 1 -- 172.21.15.33:0/471250178 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c7da57cce0 con 0x55c7da4f75d0 2024-09-16T00:35:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.150+0000 7f7a0dffb640 1 -- 172.21.15.33:0/471250178 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a1c0314e0 con 0x55c7da4f75d0 2024-09-16T00:35:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.150+0000 7f7a0dffb640 1 -- 172.21.15.33:0/471250178 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7a1c030070 con 0x55c7da4f75d0 2024-09-16T00:35:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.150+0000 7f7a2e922740 10 client.18873 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.150+0000 7f7a2e922740 10 client.18873 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:35:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.150+0000 7f7a2e922740 20 client.18873 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.150+0000 7f7a2e922740 1 --2- 172.21.15.33:0/471250178 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c7da57cf10 0x55c7da57f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.150+0000 7f7a2e922740 4 client.18873 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:35:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.150+0000 7f7a2e922740 1 -- 172.21.15.33:0/471250178 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c7da464d70 con 0x55c7da57cf10 2024-09-16T00:35:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.150+0000 7f7a29ae4640 1 --2- 172.21.15.33:0/471250178 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c7da57cf10 0x55c7da57f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.152+0000 7f7a29ae4640 1 --2- 172.21.15.33:0/471250178 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c7da57cf10 0x55c7da57f300 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6187f14b41c6ccf7 server_cookie=f0dfed003706938 in_seq=0 out_seq=0 2024-09-16T00:35:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.152+0000 7f7a0dffb640 10 client.18873 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:35:25.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.154+0000 7f7a0dffb640 1 -- 172.21.15.33:0/471250178 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c7da464d70 con 0x55c7da57cf10 2024-09-16T00:35:25.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.154+0000 7f7a0dffb640 10 client.18873 handle_command_reply: tid=0 2024-09-16T00:35:25.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.154+0000 7f7a2e922740 1 client.18873 shutdown 2024-09-16T00:35:25.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.154+0000 7f7a2e922740 1 -- 172.21.15.33:0/471250178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f75d0 msgr2=0x55c7da50a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:25.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.155+0000 7f7a2e922740 1 --2- 172.21.15.33:0/471250178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f75d0 0x55c7da50a880 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009bf0 tx=0x7f7a1c005bd0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.155+0000 7f7a2e922740 1 -- 172.21.15.33:0/471250178 shutdown_connections 2024-09-16T00:35:25.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.155+0000 7f7a2e922740 1 --2- 172.21.15.33:0/471250178 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c7da57cf10 0x55c7da57f300 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.155+0000 7f7a2e922740 1 --2- 172.21.15.33:0/471250178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7da502370 0x55c7da5060f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.155+0000 7f7a0dffb640 0 client.18873 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:35:25.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.155+0000 7f7a2e922740 1 --2- 172.21.15.33:0/471250178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7da4f8b40 0x55c7da505bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.155+0000 7f7a2e922740 1 --2- 172.21.15.33:0/471250178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7da4f75d0 0x55c7da50a880 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.155+0000 7f7a2e922740 1 -- 172.21.15.33:0/471250178 >> 172.21.15.33:0/471250178 conn(0x55c7da4c0310 msgr2=0x55c7da50dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:25.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.155+0000 7f7a2e922740 1 -- 172.21.15.33:0/471250178 shutdown_connections 2024-09-16T00:35:25.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.155+0000 7f7a2e922740 1 -- 172.21.15.33:0/471250178 wait complete. 2024-09-16T00:35:25.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.155+0000 7f7a2e922740 20 client.18873 trim_cache size 0 max 16384 2024-09-16T00:35:25.159 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:35:25.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.156+0000 7f7a2c549640 1 -- 172.21.15.33:0/1895500281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7a100a32c0 msgr2=0x7f7a100a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:25.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.156+0000 7f7a2c549640 1 --2- 172.21.15.33:0/1895500281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7a100a32c0 0x7f7a100a5780 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7a200cd0a0 tx=0x7f7a200d5550 comp rx=0 tx=0).stop 2024-09-16T00:35:25.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.156+0000 7f7a2c549640 1 -- 172.21.15.33:0/1895500281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a24102d30 msgr2=0x7f7a24100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:25.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.156+0000 7f7a2c549640 1 --2- 172.21.15.33:0/1895500281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a24102d30 0x7f7a24100990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7a2000d6d0 tx=0x7f7a2000db90 comp rx=0 tx=0).stop 2024-09-16T00:35:25.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.156+0000 7f7a2c549640 1 -- 172.21.15.33:0/1895500281 shutdown_connections 2024-09-16T00:35:25.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.156+0000 7f7a2c549640 1 --2- 172.21.15.33:0/1895500281 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7a100a32c0 0x7f7a100a5780 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.156+0000 7f7a2c549640 1 --2- 172.21.15.33:0/1895500281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a241082a0 0x7f7a24101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.156+0000 7f7a2c549640 1 --2- 172.21.15.33:0/1895500281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a241036e0 0x7f7a24100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.156+0000 7f7a2c549640 1 --2- 172.21.15.33:0/1895500281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a24102d30 0x7f7a24100990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:25.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.156+0000 7f7a2c549640 1 -- 172.21.15.33:0/1895500281 >> 172.21.15.33:0/1895500281 conn(0x7f7a240fc7d0 msgr2=0x7f7a24106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:25.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.157+0000 7f7a2c549640 1 -- 172.21.15.33:0/1895500281 shutdown_connections 2024-09-16T00:35:25.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:25.157+0000 7f7a2c549640 1 -- 172.21.15.33:0/1895500281 wait complete. 2024-09-16T00:35:25.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:25 smithi033 ceph-mon[31243]: pgmap v898: 129 pgs: 129 active+clean; 558 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 100 op/s 2024-09-16T00:35:25.898 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:35:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:25 smithi083 ceph-mon[35242]: pgmap v898: 129 pgs: 129 active+clean; 558 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 100 op/s 2024-09-16T00:35:25.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:25 smithi060 ceph-mon[35185]: pgmap v898: 129 pgs: 129 active+clean; 558 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 100 op/s 2024-09-16T00:35:26.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 88134 15.71 MB/sec execute 152 sec latency 50.844 ms 2024-09-16T00:35:26.234 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:35:26.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.809+0000 7f9677888640 1 -- 172.21.15.33:0/1965942313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9670106f20 msgr2=0x7f967010eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:26.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.809+0000 7f9677888640 1 --2- 172.21.15.33:0/1965942313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9670106f20 0x7f967010eaa0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f9668009ca0 tx=0x7f966802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.809+0000 7f9677888640 1 -- 172.21.15.33:0/1965942313 shutdown_connections 2024-09-16T00:35:26.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.809+0000 7f9677888640 1 --2- 172.21.15.33:0/1965942313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9670106f20 0x7f967010eaa0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.809+0000 7f9677888640 1 --2- 172.21.15.33:0/1965942313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96701027a0 0x7f96701067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.809+0000 7f9677888640 1 --2- 172.21.15.33:0/1965942313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9670101df0 0x7f96701021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.809+0000 7f9677888640 1 -- 172.21.15.33:0/1965942313 >> 172.21.15.33:0/1965942313 conn(0x7f96700fd8f0 msgr2=0x7f96700ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:26.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.810+0000 7f9677888640 1 -- 172.21.15.33:0/1965942313 shutdown_connections 2024-09-16T00:35:26.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.810+0000 7f9677888640 1 -- 172.21.15.33:0/1965942313 wait complete. 2024-09-16T00:35:26.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.810+0000 7f9677888640 1 Processor -- start 2024-09-16T00:35:26.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.810+0000 7f9677888640 1 -- start start 2024-09-16T00:35:26.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9677888640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9670101df0 0x7f96700744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9677888640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96701027a0 0x7f9670074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9677888640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9670106f20 0x7f9670074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9677888640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9670111f80 con 0x7f9670101df0 2024-09-16T00:35:26.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9677888640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9670111e00 con 0x7f9670106f20 2024-09-16T00:35:26.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9677888640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9670112100 con 0x7f96701027a0 2024-09-16T00:35:26.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9675624640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9670101df0 0x7f96700744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9675624640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9670101df0 0x7f96700744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56412/0 (socket says 172.21.15.33:56412) 2024-09-16T00:35:26.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9675624640 1 -- 172.21.15.33:0/4172916390 learned_addr learned my addr 172.21.15.33:0/4172916390 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:26.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9675e25640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9670106f20 0x7f9670074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9674e23640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96701027a0 0x7f9670074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9675624640 1 -- 172.21.15.33:0/4172916390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96701027a0 msgr2=0x7f9670074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9675624640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96701027a0 0x7f9670074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9675624640 1 -- 172.21.15.33:0/4172916390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9670106f20 msgr2=0x7f9670074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9675624640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9670106f20 0x7f9670074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9675624640 1 -- 172.21.15.33:0/4172916390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96701a68e0 con 0x7f9670101df0 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9675e25640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9670106f20 0x7f9670074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.811+0000 7f9675624640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9670101df0 0x7f96700744f0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f966c00e9f0 tx=0x7f966c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.812+0000 7f96667fc640 1 -- 172.21.15.33:0/4172916390 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c00cd80 con 0x7f9670101df0 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.812+0000 7f9677888640 1 -- 172.21.15.33:0/4172916390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96701a6bd0 con 0x7f9670101df0 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.812+0000 7f96667fc640 1 -- 172.21.15.33:0/4172916390 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f966c010910 con 0x7f9670101df0 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.812+0000 7f96667fc640 1 -- 172.21.15.33:0/4172916390 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c018990 con 0x7f9670101df0 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.812+0000 7f9677888640 1 -- 172.21.15.33:0/4172916390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96701a7160 con 0x7f9670101df0 2024-09-16T00:35:26.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.815+0000 7f9679c61740 1 Processor -- start 2024-09-16T00:35:26.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.815+0000 7f9679c61740 1 -- start start 2024-09-16T00:35:26.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.815+0000 7f9679c61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c39b40 0x5649b8c38090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.815+0000 7f9679c61740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c385d0 0x5649b8c38a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.815+0000 7f9679c61740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c4bb80 0x5649b8c4df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.815+0000 7f9679c61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5649b8b9e2f0 con 0x5649b8c39b40 2024-09-16T00:35:26.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.815+0000 7f9679c61740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5649b8b45cf0 con 0x5649b8c385d0 2024-09-16T00:35:26.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.815+0000 7f9679c61740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5649b8b9a2b0 con 0x5649b8c4bb80 2024-09-16T00:35:26.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.816+0000 7f9675e25640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c39b40 0x5649b8c38090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.816+0000 7f96667fc640 1 -- 172.21.15.33:0/4172916390 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f966c010ac0 con 0x7f9670101df0 2024-09-16T00:35:26.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.816+0000 7f9675e25640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c39b40 0x5649b8c38090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56414/0 (socket says 172.21.15.33:56414) 2024-09-16T00:35:26.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.816+0000 7f9675e25640 1 -- 172.21.15.33:0/3709172143 learned_addr learned my addr 172.21.15.33:0/3709172143 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:26.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.816+0000 7f9675624640 1 --2- 172.21.15.33:0/3709172143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c385d0 0x5649b8c38a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.818+0000 7f96667fc640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f965c0a32c0 0x7f965c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.818+0000 7f96667fc640 1 -- 172.21.15.33:0/4172916390 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f966c0d00e0 con 0x7f9670101df0 2024-09-16T00:35:26.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.818+0000 7f9675e25640 1 -- 172.21.15.33:0/3709172143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c4bb80 msgr2=0x5649b8c4df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:26.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.818+0000 7f9675e25640 1 --2- 172.21.15.33:0/3709172143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c4bb80 0x5649b8c4df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.818+0000 7f9675e25640 1 -- 172.21.15.33:0/3709172143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c385d0 msgr2=0x5649b8c38a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:26.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.818+0000 7f9675e25640 1 --2- 172.21.15.33:0/3709172143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c385d0 0x5649b8c38a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.818+0000 7f9675e25640 1 -- 172.21.15.33:0/3709172143 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649b8b2a3a0 con 0x5649b8c39b40 2024-09-16T00:35:26.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.819+0000 7f9675e25640 1 --2- 172.21.15.33:0/3709172143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c39b40 0x5649b8c38090 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f9668031f80 tx=0x7f9668005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2367db62e1497996 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:26.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.819+0000 7f9675e25640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f965c0a32c0 0x7f965c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.819+0000 7f965b7fe640 1 -- 172.21.15.33:0/3709172143 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96680319a0 con 0x5649b8c39b40 2024-09-16T00:35:26.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.819+0000 7f965b7fe640 1 -- 172.21.15.33:0/3709172143 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9668036040 con 0x5649b8c39b40 2024-09-16T00:35:26.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.819+0000 7f965b7fe640 1 -- 172.21.15.33:0/3709172143 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966802f870 con 0x5649b8c39b40 2024-09-16T00:35:26.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.820+0000 7f9675e25640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f965c0a32c0 0x7f965c0a5780 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f966804a000 tx=0x7f966804b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:26.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.821+0000 7f9679c61740 1 -- 172.21.15.33:0/3709172143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c39b40 msgr2=0x5649b8c38090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:26.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.821+0000 7f9679c61740 1 --2- 172.21.15.33:0/3709172143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c39b40 0x5649b8c38090 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f9668031f80 tx=0x7f9668005ef0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.821+0000 7f9679c61740 1 -- 172.21.15.33:0/3709172143 shutdown_connections 2024-09-16T00:35:26.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.821+0000 7f9679c61740 1 --2- 172.21.15.33:0/3709172143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c4bb80 0x5649b8c4df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.821+0000 7f9679c61740 1 --2- 172.21.15.33:0/3709172143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c385d0 0x5649b8c38a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.821+0000 7f9679c61740 1 --2- 172.21.15.33:0/3709172143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c39b40 0x5649b8c38090 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.821+0000 7f9679c61740 1 -- 172.21.15.33:0/3709172143 >> 172.21.15.33:0/3709172143 conn(0x5649b8c01310 msgr2=0x5649b8c34c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:26.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.822+0000 7f9679c61740 1 -- 172.21.15.33:0/3709172143 shutdown_connections 2024-09-16T00:35:26.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.823+0000 7f9679c61740 1 -- 172.21.15.33:0/3709172143 wait complete. 2024-09-16T00:35:26.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.823+0000 7f9679c61740 1 -- start start 2024-09-16T00:35:26.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.823+0000 7f9679c61740 1 Processor -- start 2024-09-16T00:35:26.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.824+0000 7f9679c61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 0x5649b8c4b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.824+0000 7f9679c61740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c39b40 0x5649b8c48840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.824+0000 7f9679c61740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 0x5649b8c48ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.824+0000 7f9679c61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5649b8c4e700 con 0x5649b8c385d0 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.824+0000 7f9679c61740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5649b8b59060 con 0x5649b8c4bb80 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.824+0000 7f9679c61740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5649b8c4e880 con 0x5649b8c39b40 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.825+0000 7f9679c61740 10 client.0 _ll_register_callbacks cb 0x5649b88ac8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.825+0000 7f9675624640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 0x5649b8c4b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.825+0000 7f9675e25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 0x5649b8c48ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.825+0000 7f9675624640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 0x5649b8c4b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56430/0 (socket says 172.21.15.33:56430) 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.825+0000 7f9675624640 1 -- 172.21.15.33:0/825950366 learned_addr learned my addr 172.21.15.33:0/825950366 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.825+0000 7f9675e25640 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 0x5649b8c48ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:34108/0 (socket says 172.21.15.33:34108) 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.825+0000 7f9675e25640 1 -- 172.21.15.33:0/825950366 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c39b40 msgr2=0x5649b8c48840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.825+0000 7f9675e25640 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c39b40 0x5649b8c48840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.825+0000 7f9675e25640 1 -- 172.21.15.33:0/825950366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 msgr2=0x5649b8c4b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:26.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.825+0000 7f9675e25640 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 0x5649b8c4b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.825+0000 7f9675e25640 1 -- 172.21.15.33:0/825950366 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649b890f3c0 con 0x5649b8c4bb80 2024-09-16T00:35:26.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.825+0000 7f9675624640 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 0x5649b8c4b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:26.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.826+0000 7f9675e25640 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 0x5649b8c48ea0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9668005e30 tx=0x7f9668003e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cccce2d80b73d9f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:26.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.826+0000 7f9658ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:35:26.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.826+0000 7f9658ff9640 1 -- 172.21.15.33:0/825950366 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96680319a0 con 0x5649b8c4bb80 2024-09-16T00:35:26.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.826+0000 7f9679c61740 1 -- 172.21.15.33:0/825950366 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9668036040 con 0x5649b8c4bb80 2024-09-16T00:35:26.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.826+0000 7f9658ff9640 1 -- 172.21.15.33:0/825950366 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f966802fda0 con 0x5649b8c4bb80 2024-09-16T00:35:26.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.826+0000 7f9658ff9640 1 -- 172.21.15.33:0/825950366 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9668004510 con 0x5649b8c4bb80 2024-09-16T00:35:26.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.826+0000 7f9679c61740 10 client.28154 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:26.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.826+0000 7f9679c61740 1 -- 172.21.15.33:0/825950366 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5649b8c4e700 con 0x5649b8c4bb80 2024-09-16T00:35:26.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.826+0000 7f9658ff9640 1 -- 172.21.15.33:0/825950366 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9668036040 con 0x5649b8c4bb80 2024-09-16T00:35:26.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.826+0000 7f9658ff9640 1 -- 172.21.15.33:0/825950366 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f9668030aa0 con 0x5649b8c4bb80 2024-09-16T00:35:26.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.827+0000 7f9679c61740 10 client.28154 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:26.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.827+0000 7f9679c61740 10 client.28154 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:35:26.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.827+0000 7f9679c61740 20 client.28154 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:26.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.827+0000 7f9679c61740 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5649b8c5b380 0x5649b8c5d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.827+0000 7f9679c61740 4 client.28154 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:35:26.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.827+0000 7f9679c61740 1 -- 172.21.15.33:0/825950366 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649b8bab860 con 0x5649b8c5b380 2024-09-16T00:35:26.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.827+0000 7f9675624640 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5649b8c5b380 0x5649b8c5d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.830+0000 7f9675624640 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5649b8c5b380 0x5649b8c5d7b0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7a639461d5b8d01d server_cookie=16d36c23c9740779 in_seq=0 out_seq=0 2024-09-16T00:35:26.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.830+0000 7f9658ff9640 10 client.28154 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:35:26.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.832+0000 7f9658ff9640 1 -- 172.21.15.33:0/825950366 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5649b8bab860 con 0x5649b8c5b380 2024-09-16T00:35:26.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.832+0000 7f9658ff9640 10 client.28154 handle_command_reply: tid=0 2024-09-16T00:35:26.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.832+0000 7f9679c61740 1 client.28154 shutdown 2024-09-16T00:35:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.835+0000 7f9679c61740 1 -- 172.21.15.33:0/825950366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 msgr2=0x5649b8c48ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.835+0000 7f9679c61740 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 0x5649b8c48ea0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9668005e30 tx=0x7f9668003e50 comp rx=0 tx=0).stop 2024-09-16T00:35:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.836+0000 7f9679c61740 1 -- 172.21.15.33:0/825950366 shutdown_connections 2024-09-16T00:35:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.836+0000 7f9679c61740 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5649b8c5b380 0x5649b8c5d7b0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.836+0000 7f9679c61740 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 0x5649b8c48ea0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.836+0000 7f9658ff9640 0 client.28154 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:35:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.836+0000 7f9679c61740 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c39b40 0x5649b8c48840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.836+0000 7f9679c61740 1 --2- 172.21.15.33:0/825950366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 0x5649b8c4b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.836+0000 7f9679c61740 1 -- 172.21.15.33:0/825950366 >> 172.21.15.33:0/825950366 conn(0x5649b8c01310 msgr2=0x5649b8c3a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.836+0000 7f9679c61740 1 -- 172.21.15.33:0/825950366 shutdown_connections 2024-09-16T00:35:26.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.836+0000 7f9679c61740 1 -- 172.21.15.33:0/825950366 wait complete. 2024-09-16T00:35:26.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.836+0000 7f9679c61740 20 client.28154 trim_cache size 0 max 16384 2024-09-16T00:35:26.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.860+0000 7f9679c61740 1 Processor -- start 2024-09-16T00:35:26.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.860+0000 7f9679c61740 1 -- start start 2024-09-16T00:35:26.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.860+0000 7f9679c61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 0x5649b8c57a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.860+0000 7f9679c61740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c39b40 0x5649b8c63080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.860+0000 7f9679c61740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c4bb80 0x5649b8c635c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.860+0000 7f9679c61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f96701af680 con 0x5649b8c385d0 2024-09-16T00:35:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.861+0000 7f9679c61740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f96701acf50 con 0x5649b8c39b40 2024-09-16T00:35:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.861+0000 7f9679c61740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f96701aadf0 con 0x5649b8c4bb80 2024-09-16T00:35:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.861+0000 7f9675e25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c4bb80 0x5649b8c635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.861+0000 7f9675e25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c4bb80 0x5649b8c635c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:45600/0 (socket says 172.21.15.33:45600) 2024-09-16T00:35:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.861+0000 7f9675e25640 1 -- 172.21.15.33:0/4047799841 learned_addr learned my addr 172.21.15.33:0/4047799841 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.861+0000 7f9675624640 1 --2- 172.21.15.33:0/4047799841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 0x5649b8c57a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.861+0000 7f9674e23640 1 --2- 172.21.15.33:0/4047799841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c39b40 0x5649b8c63080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.861+0000 7f9675624640 1 -- 172.21.15.33:0/4047799841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c4bb80 msgr2=0x5649b8c635c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.861+0000 7f9675624640 1 --2- 172.21.15.33:0/4047799841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c4bb80 0x5649b8c635c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.861+0000 7f9675624640 1 -- 172.21.15.33:0/4047799841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c39b40 msgr2=0x5649b8c63080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.861+0000 7f9675624640 1 --2- 172.21.15.33:0/4047799841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c39b40 0x5649b8c63080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.861+0000 7f9675624640 1 -- 172.21.15.33:0/4047799841 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649b8c5de30 con 0x5649b8c385d0 2024-09-16T00:35:26.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.862+0000 7f9675624640 1 --2- 172.21.15.33:0/4047799841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 0x5649b8c57a00 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f966c0a4960 tx=0x7f966c018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd50d9ce18d7772b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:26.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.865+0000 7f965affd640 1 -- 172.21.15.33:0/4047799841 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c0bae60 con 0x5649b8c385d0 2024-09-16T00:35:26.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.865+0000 7f965affd640 1 -- 172.21.15.33:0/4047799841 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f966c0a4e90 con 0x5649b8c385d0 2024-09-16T00:35:26.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.865+0000 7f965affd640 1 -- 172.21.15.33:0/4047799841 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c0a03c0 con 0x5649b8c385d0 2024-09-16T00:35:26.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.867+0000 7f9679c61740 1 -- 172.21.15.33:0/4047799841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 msgr2=0x5649b8c57a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:26.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.867+0000 7f9679c61740 1 --2- 172.21.15.33:0/4047799841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 0x5649b8c57a00 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f966c0a4960 tx=0x7f966c018420 comp rx=0 tx=0).stop 2024-09-16T00:35:26.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.867+0000 7f9679c61740 1 -- 172.21.15.33:0/4047799841 shutdown_connections 2024-09-16T00:35:26.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.867+0000 7f9679c61740 1 --2- 172.21.15.33:0/4047799841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c4bb80 0x5649b8c635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.867+0000 7f9679c61740 1 --2- 172.21.15.33:0/4047799841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c39b40 0x5649b8c63080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.867+0000 7f9679c61740 1 --2- 172.21.15.33:0/4047799841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c385d0 0x5649b8c57a00 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.867+0000 7f9679c61740 1 -- 172.21.15.33:0/4047799841 >> 172.21.15.33:0/4047799841 conn(0x5649b8c01310 msgr2=0x5649b8c47110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:26.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.868+0000 7f9679c61740 1 -- 172.21.15.33:0/4047799841 shutdown_connections 2024-09-16T00:35:26.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.869+0000 7f9679c61740 1 -- 172.21.15.33:0/4047799841 wait complete. 2024-09-16T00:35:26.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.869+0000 7f9679c61740 1 -- start start 2024-09-16T00:35:26.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.869+0000 7f9679c61740 1 Processor -- start 2024-09-16T00:35:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.870+0000 7f9679c61740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c385d0 0x5649b8c542d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.870+0000 7f9679c61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c39b40 0x5649b8c4f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.870+0000 7f9679c61740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 0x5649b8c4fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.870+0000 7f9679c61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f96701ab410 con 0x5649b8c39b40 2024-09-16T00:35:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.870+0000 7f9679c61740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f96701ac880 con 0x5649b8c4bb80 2024-09-16T00:35:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.870+0000 7f9679c61740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f96701abae0 con 0x5649b8c385d0 2024-09-16T00:35:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.870+0000 7f9679c61740 10 client.0 _ll_register_callbacks cb 0x7f9668049020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.870+0000 7f9675e25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 0x5649b8c4fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.870+0000 7f9675e25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 0x5649b8c4fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:34136/0 (socket says 172.21.15.33:34136) 2024-09-16T00:35:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.870+0000 7f9675e25640 1 -- 172.21.15.33:0/1659990480 learned_addr learned my addr 172.21.15.33:0/1659990480 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.870+0000 7f9675624640 1 --2- 172.21.15.33:0/1659990480 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c385d0 0x5649b8c542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.873+0000 7f9675e25640 1 -- 172.21.15.33:0/1659990480 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c385d0 msgr2=0x5649b8c542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.873+0000 7f9675e25640 1 --2- 172.21.15.33:0/1659990480 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c385d0 0x5649b8c542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.873+0000 7f9675e25640 1 -- 172.21.15.33:0/1659990480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c39b40 msgr2=0x5649b8c4f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.873+0000 7f9675e25640 1 --2- 172.21.15.33:0/1659990480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c39b40 0x5649b8c4f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.873+0000 7f9675e25640 1 -- 172.21.15.33:0/1659990480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649b8c50320 con 0x5649b8c4bb80 2024-09-16T00:35:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.874+0000 7f9675e25640 1 --2- 172.21.15.33:0/1659990480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 0x5649b8c4fb40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9668009a60 tx=0x7f9668003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c19d1122d72ef28f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.874+0000 7f9658ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:35:26.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.878+0000 7f9658ff9640 1 -- 172.21.15.33:0/1659990480 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9668004e50 con 0x5649b8c4bb80 2024-09-16T00:35:26.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.878+0000 7f9658ff9640 1 -- 172.21.15.33:0/1659990480 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9668003af0 con 0x5649b8c4bb80 2024-09-16T00:35:26.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.879+0000 7f9679c61740 1 -- 172.21.15.33:0/1659990480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f966c0a4e90 con 0x5649b8c4bb80 2024-09-16T00:35:26.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.882+0000 7f9658ff9640 1 -- 172.21.15.33:0/1659990480 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f96680319a0 con 0x5649b8c4bb80 2024-09-16T00:35:26.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.882+0000 7f9679c61740 10 client.28160 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:26.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.882+0000 7f9679c61740 1 -- 172.21.15.33:0/1659990480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5649b8cc6060 con 0x5649b8c4bb80 2024-09-16T00:35:26.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.883+0000 7f9658ff9640 1 -- 172.21.15.33:0/1659990480 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f966c0a4e90 con 0x5649b8c4bb80 2024-09-16T00:35:26.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.883+0000 7f9658ff9640 1 -- 172.21.15.33:0/1659990480 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f9668030760 con 0x5649b8c4bb80 2024-09-16T00:35:26.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.883+0000 7f9679c61740 10 client.28160 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:26.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.883+0000 7f9679c61740 10 client.28160 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:35:26.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.883+0000 7f9679c61740 20 client.28160 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:26.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.883+0000 7f9679c61740 1 --2- 172.21.15.33:0/1659990480 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5649b8cc6750 0x5649b8cc8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:26.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.884+0000 7f9679c61740 4 client.28160 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:35:26.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.884+0000 7f9679c61740 1 -- 172.21.15.33:0/1659990480 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5649b8ba5d70 con 0x5649b8cc6750 2024-09-16T00:35:26.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.884+0000 7f9675624640 1 --2- 172.21.15.33:0/1659990480 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5649b8cc6750 0x5649b8cc8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:26.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.885+0000 7f9675624640 1 --2- 172.21.15.33:0/1659990480 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5649b8cc6750 0x5649b8cc8b70 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac7c2d1a66fc5747 server_cookie=75360320053807b9 in_seq=0 out_seq=0 2024-09-16T00:35:26.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.885+0000 7f9658ff9640 10 client.28160 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:35:26.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.886+0000 7f9658ff9640 1 -- 172.21.15.33:0/1659990480 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5649b8ba5d70 con 0x5649b8cc6750 2024-09-16T00:35:26.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.886+0000 7f9658ff9640 10 client.28160 handle_command_reply: tid=0 2024-09-16T00:35:26.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.886+0000 7f9679c61740 1 client.28160 shutdown 2024-09-16T00:35:26.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.886+0000 7f9679c61740 1 -- 172.21.15.33:0/1659990480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 msgr2=0x5649b8c4fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:26.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.886+0000 7f9679c61740 1 --2- 172.21.15.33:0/1659990480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 0x5649b8c4fb40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9668009a60 tx=0x7f9668003f90 comp rx=0 tx=0).stop 2024-09-16T00:35:26.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.886+0000 7f9679c61740 1 -- 172.21.15.33:0/1659990480 shutdown_connections 2024-09-16T00:35:26.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.886+0000 7f9679c61740 1 --2- 172.21.15.33:0/1659990480 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5649b8cc6750 0x5649b8cc8b70 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.886+0000 7f9679c61740 1 --2- 172.21.15.33:0/1659990480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5649b8c4bb80 0x5649b8c4fb40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.886+0000 7f9679c61740 1 --2- 172.21.15.33:0/1659990480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5649b8c39b40 0x5649b8c4f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.886+0000 7f9679c61740 1 --2- 172.21.15.33:0/1659990480 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5649b8c385d0 0x5649b8c542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.886+0000 7f9679c61740 1 -- 172.21.15.33:0/1659990480 >> 172.21.15.33:0/1659990480 conn(0x5649b8c01310 msgr2=0x5649b8c4a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.886+0000 7f9658ff9640 0 client.28160 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:35:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.887+0000 7f9679c61740 1 -- 172.21.15.33:0/1659990480 shutdown_connections 2024-09-16T00:35:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.888+0000 7f9679c61740 1 -- 172.21.15.33:0/1659990480 wait complete. 2024-09-16T00:35:26.890 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:35:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.889+0000 7f9679c61740 20 client.28160 trim_cache size 0 max 16384 2024-09-16T00:35:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.891+0000 7f9677888640 1 -- 172.21.15.33:0/4172916390 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f965c0a32c0 msgr2=0x7f965c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.891+0000 7f9677888640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f965c0a32c0 0x7f965c0a5780 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f966804a000 tx=0x7f966804b000 comp rx=0 tx=0).stop 2024-09-16T00:35:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.891+0000 7f9677888640 1 -- 172.21.15.33:0/4172916390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9670101df0 msgr2=0x7f96700744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.891+0000 7f9677888640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9670101df0 0x7f96700744f0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f966c00e9f0 tx=0x7f966c00eeb0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.891+0000 7f9677888640 1 -- 172.21.15.33:0/4172916390 shutdown_connections 2024-09-16T00:35:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.891+0000 7f9677888640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f965c0a32c0 0x7f965c0a5780 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.891+0000 7f9677888640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9670106f20 0x7f9670074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.891+0000 7f9677888640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96701027a0 0x7f9670074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.891+0000 7f9677888640 1 --2- 172.21.15.33:0/4172916390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9670101df0 0x7f96700744f0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.891+0000 7f9677888640 1 -- 172.21.15.33:0/4172916390 >> 172.21.15.33:0/4172916390 conn(0x7f96700fd8f0 msgr2=0x7f96700febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.891+0000 7f9677888640 1 -- 172.21.15.33:0/4172916390 shutdown_connections 2024-09-16T00:35:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:26.891+0000 7f9677888640 1 -- 172.21.15.33:0/4172916390 wait complete. 2024-09-16T00:35:27.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 88651 15.78 MB/sec execute 153 sec latency 61.334 ms 2024-09-16T00:35:27.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:27 smithi033 ceph-mon[31243]: pgmap v899: 129 pgs: 129 active+clean; 551 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.9 MiB/s wr, 162 op/s 2024-09-16T00:35:27.713 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:35:27.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:27 smithi083 ceph-mon[35242]: pgmap v899: 129 pgs: 129 active+clean; 551 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.9 MiB/s wr, 162 op/s 2024-09-16T00:35:27.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:27 smithi060 ceph-mon[35185]: pgmap v899: 129 pgs: 129 active+clean; 551 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.9 MiB/s wr, 162 op/s 2024-09-16T00:35:28.068 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:35:28.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 89151 15.86 MB/sec execute 154 sec latency 58.319 ms 2024-09-16T00:35:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.642+0000 7f7bceea1640 1 -- 172.21.15.33:0/2305371604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc8103780 msgr2=0x7f7bc8107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.642+0000 7f7bceea1640 1 --2- 172.21.15.33:0/2305371604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc8103780 0x7f7bc8107bc0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8009890 tx=0x7f7bb802a370 comp rx=0 tx=0).stop 2024-09-16T00:35:28.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.642+0000 7f7bceea1640 1 -- 172.21.15.33:0/2305371604 shutdown_connections 2024-09-16T00:35:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.642+0000 7f7bceea1640 1 --2- 172.21.15.33:0/2305371604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc8108300 0x7f7bc810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.642+0000 7f7bceea1640 1 --2- 172.21.15.33:0/2305371604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc8103780 0x7f7bc8107bc0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.642+0000 7f7bceea1640 1 --2- 172.21.15.33:0/2305371604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bc8102dd0 0x7f7bc81031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.642+0000 7f7bceea1640 1 -- 172.21.15.33:0/2305371604 >> 172.21.15.33:0/2305371604 conn(0x7f7bc80fc7d0 msgr2=0x7f7bc80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.642+0000 7f7bceea1640 1 -- 172.21.15.33:0/2305371604 shutdown_connections 2024-09-16T00:35:28.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.642+0000 7f7bceea1640 1 -- 172.21.15.33:0/2305371604 wait complete. 2024-09-16T00:35:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.643+0000 7f7bceea1640 1 Processor -- start 2024-09-16T00:35:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.643+0000 7f7bceea1640 1 -- start start 2024-09-16T00:35:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.644+0000 7f7bceea1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bc8102dd0 0x7f7bc8073a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.644+0000 7f7bceea1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc8103780 0x7f7bc806caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.644+0000 7f7bceea1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc8108300 0x7f7bc806d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.644+0000 7f7bceea1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc8112050 con 0x7f7bc8103780 2024-09-16T00:35:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.644+0000 7f7bceea1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc8111ed0 con 0x7f7bc8108300 2024-09-16T00:35:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.644+0000 7f7bceea1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc81121d0 con 0x7f7bc8102dd0 2024-09-16T00:35:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.644+0000 7f7bc7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc8103780 0x7f7bc806caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.644+0000 7f7bcd43e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc8108300 0x7f7bc806d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.644+0000 7f7bcd43e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc8108300 0x7f7bc806d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:34152/0 (socket says 172.21.15.33:34152) 2024-09-16T00:35:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.644+0000 7f7bcd43e640 1 -- 172.21.15.33:0/136681606 learned_addr learned my addr 172.21.15.33:0/136681606 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:28.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.644+0000 7f7bccc3d640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bc8102dd0 0x7f7bc8073a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.645+0000 7f7bcd43e640 1 -- 172.21.15.33:0/136681606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bc8102dd0 msgr2=0x7f7bc8073a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.645+0000 7f7bcd43e640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bc8102dd0 0x7f7bc8073a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.645+0000 7f7bcd43e640 1 -- 172.21.15.33:0/136681606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc8103780 msgr2=0x7f7bc806caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.645+0000 7f7bcd43e640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc8103780 0x7f7bc806caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.645+0000 7f7bcd43e640 1 -- 172.21.15.33:0/136681606 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc806d670 con 0x7f7bc8108300 2024-09-16T00:35:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.645+0000 7f7bcd43e640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc8108300 0x7f7bc806d030 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000eb10 tx=0x7f7bc000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.645+0000 7f7bc5ffb640 1 -- 172.21.15.33:0/136681606 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0004b20 con 0x7f7bc8108300 2024-09-16T00:35:28.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.645+0000 7f7bc5ffb640 1 -- 172.21.15.33:0/136681606 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bc0004cc0 con 0x7f7bc8108300 2024-09-16T00:35:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.645+0000 7f7bceea1640 1 -- 172.21.15.33:0/136681606 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bc8071590 con 0x7f7bc8108300 2024-09-16T00:35:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.646+0000 7f7bc5ffb640 1 -- 172.21.15.33:0/136681606 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0010760 con 0x7f7bc8108300 2024-09-16T00:35:28.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.646+0000 7f7bceea1640 1 -- 172.21.15.33:0/136681606 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bc8071aa0 con 0x7f7bc8108300 2024-09-16T00:35:28.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.649+0000 7f7bc5ffb640 1 -- 172.21.15.33:0/136681606 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7bc0005020 con 0x7f7bc8108300 2024-09-16T00:35:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.651+0000 7f7bc5ffb640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7bb00a3340 0x7f7bb00a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.651+0000 7f7bd127a740 1 Processor -- start 2024-09-16T00:35:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.651+0000 7f7bd127a740 1 -- start start 2024-09-16T00:35:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.651+0000 7f7bc5ffb640 1 -- 172.21.15.33:0/136681606 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7bc00d02d0 con 0x7f7bc8108300 2024-09-16T00:35:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.651+0000 7f7bd127a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 0x55f16d1eb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.652+0000 7f7bd127a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1ebdd0 0x55f16d1f03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.652+0000 7f7bd127a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1f09b0 0x55f16d1f0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.652+0000 7f7bd127a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f16d1472f0 con 0x55f16d1eb420 2024-09-16T00:35:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.652+0000 7f7bd127a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f16d0eecf0 con 0x55f16d1ebdd0 2024-09-16T00:35:28.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.652+0000 7f7bd127a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f16d1432b0 con 0x55f16d1f09b0 2024-09-16T00:35:28.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.652+0000 7f7bccc3d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 0x55f16d1eb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.652+0000 7f7bccc3d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 0x55f16d1eb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56470/0 (socket says 172.21.15.33:56470) 2024-09-16T00:35:28.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.652+0000 7f7bccc3d640 1 -- 172.21.15.33:0/1954087403 learned_addr learned my addr 172.21.15.33:0/1954087403 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:28.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.652+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/1954087403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1ebdd0 0x55f16d1f03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.652+0000 7f7bcd43e640 1 --2- 172.21.15.33:0/1954087403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1f09b0 0x55f16d1f0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.652+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7bb00a3340 0x7f7bb00a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.653+0000 7f7bcd43e640 1 -- 172.21.15.33:0/1954087403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1ebdd0 msgr2=0x55f16d1f03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.653+0000 7f7bcd43e640 1 --2- 172.21.15.33:0/1954087403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1ebdd0 0x55f16d1f03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.653+0000 7f7bcd43e640 1 -- 172.21.15.33:0/1954087403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 msgr2=0x55f16d1eb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.653+0000 7f7bcd43e640 1 --2- 172.21.15.33:0/1954087403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 0x55f16d1eb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.653+0000 7f7bcd43e640 1 -- 172.21.15.33:0/1954087403 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f16d0d33a0 con 0x55f16d1f09b0 2024-09-16T00:35:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.653+0000 7f7bccc3d640 1 --2- 172.21.15.33:0/1954087403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 0x55f16d1eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.653+0000 7f7bcd43e640 1 --2- 172.21.15.33:0/1954087403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1f09b0 0x55f16d1f0e40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0018000 tx=0x7f7bc000cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a268578456f48bef server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.653+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/1954087403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1ebdd0 0x55f16d1f03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:35:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.653+0000 7f7baeffd640 1 -- 172.21.15.33:0/1954087403 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc009bd70 con 0x55f16d1f09b0 2024-09-16T00:35:28.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.653+0000 7f7baeffd640 1 -- 172.21.15.33:0/1954087403 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bc009cb10 con 0x55f16d1f09b0 2024-09-16T00:35:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.654+0000 7f7bd127a740 1 -- 172.21.15.33:0/1954087403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1f09b0 msgr2=0x55f16d1f0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.654+0000 7f7bd127a740 1 --2- 172.21.15.33:0/1954087403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1f09b0 0x55f16d1f0e40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0018000 tx=0x7f7bc000cf70 comp rx=0 tx=0).stop 2024-09-16T00:35:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.655+0000 7f7bd127a740 1 -- 172.21.15.33:0/1954087403 shutdown_connections 2024-09-16T00:35:28.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.655+0000 7f7bd127a740 1 --2- 172.21.15.33:0/1954087403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1f09b0 0x55f16d1f0e40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.655+0000 7f7bd127a740 1 --2- 172.21.15.33:0/1954087403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1ebdd0 0x55f16d1f03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.655+0000 7f7bd127a740 1 --2- 172.21.15.33:0/1954087403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 0x55f16d1eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.655+0000 7f7bd127a740 1 -- 172.21.15.33:0/1954087403 >> 172.21.15.33:0/1954087403 conn(0x55f16d1aa310 msgr2=0x55f16d1dd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.655+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7bb00a3340 0x7f7bb00a5800 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8000c00 tx=0x7f7bb8005870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.655+0000 7f7bd127a740 1 -- 172.21.15.33:0/1954087403 shutdown_connections 2024-09-16T00:35:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.655+0000 7f7bd127a740 1 -- 172.21.15.33:0/1954087403 wait complete. 2024-09-16T00:35:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.656+0000 7f7bd127a740 1 -- start start 2024-09-16T00:35:28.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.656+0000 7f7bd127a740 1 Processor -- start 2024-09-16T00:35:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.657+0000 7f7bd127a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 0x55f16d1f6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.657+0000 7f7bd127a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1ebdd0 0x55f16d1f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.657+0000 7f7bccc3d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 0x55f16d1f6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.657+0000 7f7bccc3d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 0x55f16d1f6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56472/0 (socket says 172.21.15.33:56472) 2024-09-16T00:35:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.657+0000 7f7bccc3d640 1 -- 172.21.15.33:0/3779417634 learned_addr learned my addr 172.21.15.33:0/3779417634 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.657+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1f09b0 0x55f16d1f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.657+0000 7f7bd127a740 1 -- 172.21.15.33:0/3779417634 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f16d1f78f0 con 0x55f16d1eb420 2024-09-16T00:35:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.657+0000 7f7bd127a740 1 -- 172.21.15.33:0/3779417634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f16d102060 con 0x55f16d1f09b0 2024-09-16T00:35:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.657+0000 7f7bd127a740 1 -- 172.21.15.33:0/3779417634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f16d1f7a70 con 0x55f16d1ebdd0 2024-09-16T00:35:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.658+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1ebdd0 0x55f16d1f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.657+0000 7f7bcd43e640 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1f09b0 0x55f16d1f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.658+0000 7f7bd127a740 10 client.0 _ll_register_callbacks cb 0x55f16ce558c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.658+0000 7f7bccc3d640 1 -- 172.21.15.33:0/3779417634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1ebdd0 msgr2=0x55f16d1f3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.658+0000 7f7bccc3d640 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1ebdd0 0x55f16d1f3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.658+0000 7f7bccc3d640 1 -- 172.21.15.33:0/3779417634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1f09b0 msgr2=0x55f16d1f43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.658+0000 7f7bccc3d640 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1f09b0 0x55f16d1f43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.658+0000 7f7bccc3d640 1 -- 172.21.15.33:0/3779417634 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f16ceb83c0 con 0x55f16d1eb420 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.658+0000 7f7bd127a740 1 -- 172.21.15.33:0/3779417634 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bc009cb10 con 0x55f16d1eb420 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.658+0000 7f7bccc3d640 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 0x55f16d1f6c00 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc009fd0 tx=0x7f7bbc012ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562a50930f77e279 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.658+0000 7f7ba7fff640 10 client.18900 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.658+0000 7f7ba7fff640 1 -- 172.21.15.33:0/3779417634 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00f920 con 0x55f16d1eb420 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.659+0000 7f7ba7fff640 1 -- 172.21.15.33:0/3779417634 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7bbc00fac0 con 0x55f16d1eb420 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.659+0000 7f7ba7fff640 1 -- 172.21.15.33:0/3779417634 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7bbc00fd00 con 0x55f16d1eb420 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.659+0000 7f7bd127a740 10 client.18900 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.659+0000 7f7bd127a740 1 -- 172.21.15.33:0/3779417634 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({fsmap=33}) -- 0x55f16d203050 con 0x55f16d1eb420 2024-09-16T00:35:28.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.659+0000 7f7ba7fff640 1 -- 172.21.15.33:0/3779417634 <== mon.0 v2:172.21.15.33:3300/0 4 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7bbc026470 con 0x55f16d1eb420 2024-09-16T00:35:28.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.659+0000 7f7bd127a740 10 client.18900 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:28.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.659+0000 7f7bd127a740 10 client.18900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:35:28.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.659+0000 7f7bd127a740 20 client.18900 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:28.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.659+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f16d204450 0x55f16d206880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.659+0000 7f7bd127a740 4 client.18900 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:35:28.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.659+0000 7f7bd127a740 1 -- 172.21.15.33:0/3779417634 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f16d154860 con 0x55f16d204450 2024-09-16T00:35:28.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.660+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f16d204450 0x55f16d206880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.660+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f16d204450 0x55f16d206880 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a66319d4cedfecc9 server_cookie=634e643ad1f49684 in_seq=0 out_seq=0 2024-09-16T00:35:28.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.660+0000 7f7ba7fff640 10 client.18900 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:28.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.661+0000 7f7ba7fff640 1 -- 172.21.15.33:0/3779417634 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f16d154860 con 0x55f16d204450 2024-09-16T00:35:28.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.661+0000 7f7ba7fff640 10 client.18900 handle_command_reply: tid=0 2024-09-16T00:35:28.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.661+0000 7f7bd127a740 1 client.18900 shutdown 2024-09-16T00:35:28.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.661+0000 7f7bd127a740 1 -- 172.21.15.33:0/3779417634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 msgr2=0x55f16d1f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:28.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.661+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 0x55f16d1f6c00 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc009fd0 tx=0x7f7bbc012ca0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.661+0000 7f7bd127a740 1 -- 172.21.15.33:0/3779417634 shutdown_connections 2024-09-16T00:35:28.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.661+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f16d204450 0x55f16d206880 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.661+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1f09b0 0x55f16d1f43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.661+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1ebdd0 0x55f16d1f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.661+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3779417634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1eb420 0x55f16d1f6c00 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.661+0000 7f7bd127a740 1 -- 172.21.15.33:0/3779417634 >> 172.21.15.33:0/3779417634 conn(0x55f16d1aa310 msgr2=0x55f16d1eaa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:28.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.661+0000 7f7ba7fff640 0 client.18900 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:28.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.662+0000 7f7bd127a740 1 -- 172.21.15.33:0/3779417634 shutdown_connections 2024-09-16T00:35:28.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.662+0000 7f7bd127a740 1 -- 172.21.15.33:0/3779417634 wait complete. 2024-09-16T00:35:28.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.662+0000 7f7bd127a740 20 client.18900 trim_cache size 0 max 16384 2024-09-16T00:35:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.678+0000 7f7bd127a740 1 Processor -- start 2024-09-16T00:35:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.678+0000 7f7bd127a740 1 -- start start 2024-09-16T00:35:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bd127a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1eb420 0x55f16d1ea940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bd127a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1ebdd0 0x55f16d20c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bd127a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1f09b0 0x55f16d20c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bd127a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc81b7f80 con 0x55f16d1f09b0 2024-09-16T00:35:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bd127a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc81b5850 con 0x55f16d1ebdd0 2024-09-16T00:35:28.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bd127a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc81b36a0 con 0x55f16d1eb420 2024-09-16T00:35:28.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bccc3d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1eb420 0x55f16d1ea940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bc7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1ebdd0 0x55f16d20c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bccc3d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1eb420 0x55f16d1ea940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:45668/0 (socket says 172.21.15.33:45668) 2024-09-16T00:35:28.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bccc3d640 1 -- 172.21.15.33:0/1153090330 learned_addr learned my addr 172.21.15.33:0/1153090330 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:28.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bcd43e640 1 --2- 172.21.15.33:0/1153090330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1f09b0 0x55f16d20c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bc7fff640 1 -- 172.21.15.33:0/1153090330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1eb420 msgr2=0x55f16d1ea940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/1153090330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1eb420 0x55f16d1ea940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bc7fff640 1 -- 172.21.15.33:0/1153090330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1f09b0 msgr2=0x55f16d20c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/1153090330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1f09b0 0x55f16d20c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bc7fff640 1 -- 172.21.15.33:0/1153090330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f16d1e1580 con 0x55f16d1ebdd0 2024-09-16T00:35:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.679+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/1153090330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1ebdd0 0x55f16d20c150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8035040 tx=0x7f7bb802c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ef024d13443f593 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.680+0000 7f7bae7fc640 1 -- 172.21.15.33:0/1153090330 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8056070 con 0x55f16d1ebdd0 2024-09-16T00:35:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.680+0000 7f7bae7fc640 1 -- 172.21.15.33:0/1153090330 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bb8043040 con 0x55f16d1ebdd0 2024-09-16T00:35:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.680+0000 7f7bae7fc640 1 -- 172.21.15.33:0/1153090330 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8042040 con 0x55f16d1ebdd0 2024-09-16T00:35:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.681+0000 7f7bd127a740 1 -- 172.21.15.33:0/1153090330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1ebdd0 msgr2=0x55f16d20c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.681+0000 7f7bd127a740 1 --2- 172.21.15.33:0/1153090330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1ebdd0 0x55f16d20c150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8035040 tx=0x7f7bb802c040 comp rx=0 tx=0).stop 2024-09-16T00:35:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.681+0000 7f7bd127a740 1 -- 172.21.15.33:0/1153090330 shutdown_connections 2024-09-16T00:35:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.681+0000 7f7bd127a740 1 --2- 172.21.15.33:0/1153090330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1f09b0 0x55f16d20c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.681+0000 7f7bd127a740 1 --2- 172.21.15.33:0/1153090330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1ebdd0 0x55f16d20c150 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.681+0000 7f7bd127a740 1 --2- 172.21.15.33:0/1153090330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1eb420 0x55f16d1ea940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.681+0000 7f7bd127a740 1 -- 172.21.15.33:0/1153090330 >> 172.21.15.33:0/1153090330 conn(0x55f16d1aa310 msgr2=0x55f16d1ea230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.681+0000 7f7bd127a740 1 -- 172.21.15.33:0/1153090330 shutdown_connections 2024-09-16T00:35:28.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.681+0000 7f7bd127a740 1 -- 172.21.15.33:0/1153090330 wait complete. 2024-09-16T00:35:28.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.681+0000 7f7bd127a740 1 -- start start 2024-09-16T00:35:28.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bd127a740 1 Processor -- start 2024-09-16T00:35:28.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bd127a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1eb420 0x55f16d1fd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bd127a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1ebdd0 0x55f16d1fd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bd127a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1f09b0 0x55f16d1f9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bd127a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc81b3d10 con 0x55f16d1ebdd0 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bd127a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc81b5180 con 0x55f16d1eb420 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bd127a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc81b43e0 con 0x55f16d1f09b0 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bd127a740 10 client.0 _ll_register_callbacks cb 0x7f7bbc009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bccc3d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1eb420 0x55f16d1fd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bccc3d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1eb420 0x55f16d1fd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:34188/0 (socket says 172.21.15.33:34188) 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bccc3d640 1 -- 172.21.15.33:0/3214263339 learned_addr learned my addr 172.21.15.33:0/3214263339 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bcd43e640 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1f09b0 0x55f16d1f9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.682+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1ebdd0 0x55f16d1fd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7bccc3d640 1 -- 172.21.15.33:0/3214263339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1f09b0 msgr2=0x55f16d1f9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:28.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7bccc3d640 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1f09b0 0x55f16d1f9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7bccc3d640 1 -- 172.21.15.33:0/3214263339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1ebdd0 msgr2=0x55f16d1fd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7bccc3d640 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1ebdd0 0x55f16d1fd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7bccc3d640 1 -- 172.21.15.33:0/3214263339 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f16d1fa700 con 0x55f16d1eb420 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7bcd43e640 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1f09b0 0x55f16d1f9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7bccc3d640 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1eb420 0x55f16d1fd070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc001eb0 tx=0x7f7bbc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57d594462e9e5093 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7ba7fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7ba7fff640 1 -- 172.21.15.33:0/3214263339 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc026bc0 con 0x55f16d1eb420 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7bd127a740 1 -- 172.21.15.33:0/3214263339 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bb8043040 con 0x55f16d1eb420 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7ba7fff640 1 -- 172.21.15.33:0/3214263339 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7bbc026d60 con 0x55f16d1eb420 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7ba7fff640 1 -- 172.21.15.33:0/3214263339 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7bbc03a050 con 0x55f16d1eb420 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7bd127a740 10 client.28178 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7bd127a740 1 -- 172.21.15.33:0/3214263339 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f16d26fd90 con 0x55f16d1eb420 2024-09-16T00:35:28.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.683+0000 7f7ba7fff640 1 -- 172.21.15.33:0/3214263339 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bb8043040 con 0x55f16d1eb420 2024-09-16T00:35:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.684+0000 7f7ba7fff640 1 -- 172.21.15.33:0/3214263339 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7bbc010430 con 0x55f16d1eb420 2024-09-16T00:35:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.684+0000 7f7bd127a740 10 client.28178 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.684+0000 7f7bd127a740 10 client.28178 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:35:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.684+0000 7f7bd127a740 20 client.28178 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.684+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f16d2704b0 0x55f16d2728a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.684+0000 7f7bd127a740 4 client.28178 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:35:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.684+0000 7f7bd127a740 1 -- 172.21.15.33:0/3214263339 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f16d14ed70 con 0x55f16d2704b0 2024-09-16T00:35:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.684+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f16d2704b0 0x55f16d2728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.684+0000 7f7ba7fff640 10 client.28178 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bc7fff640 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f16d2704b0 0x55f16d2728a0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b236b61503002510 server_cookie=eb703f08feb7973d in_seq=0 out_seq=0 2024-09-16T00:35:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7ba7fff640 1 -- 172.21.15.33:0/3214263339 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f16d14ed70 con 0x55f16d2704b0 2024-09-16T00:35:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7ba7fff640 10 client.28178 handle_command_reply: tid=0 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bd127a740 1 client.28178 shutdown 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bd127a740 1 -- 172.21.15.33:0/3214263339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1eb420 msgr2=0x55f16d1fd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1eb420 0x55f16d1fd070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc001eb0 tx=0x7f7bbc0046c0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bd127a740 1 -- 172.21.15.33:0/3214263339 shutdown_connections 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f16d2704b0 0x55f16d2728a0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7ba7fff640 0 client.28178 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f16d1f09b0 0x55f16d1f9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f16d1ebdd0 0x55f16d1fd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bd127a740 1 --2- 172.21.15.33:0/3214263339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f16d1eb420 0x55f16d1fd070 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bd127a740 1 -- 172.21.15.33:0/3214263339 >> 172.21.15.33:0/3214263339 conn(0x55f16d1aa310 msgr2=0x55f16d1eaa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bd127a740 1 -- 172.21.15.33:0/3214263339 shutdown_connections 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bd127a740 1 -- 172.21.15.33:0/3214263339 wait complete. 2024-09-16T00:35:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.685+0000 7f7bd127a740 20 client.28178 trim_cache size 0 max 16384 2024-09-16T00:35:28.691 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:35:28.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.686+0000 7f7bceea1640 1 -- 172.21.15.33:0/136681606 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7bb00a3340 msgr2=0x7f7bb00a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:28.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.686+0000 7f7bceea1640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7bb00a3340 0x7f7bb00a5800 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8000c00 tx=0x7f7bb8005870 comp rx=0 tx=0).stop 2024-09-16T00:35:28.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.686+0000 7f7bceea1640 1 -- 172.21.15.33:0/136681606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc8108300 msgr2=0x7f7bc806d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:28.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.686+0000 7f7bceea1640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc8108300 0x7f7bc806d030 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000eb10 tx=0x7f7bc000c2d0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.686+0000 7f7bceea1640 1 -- 172.21.15.33:0/136681606 shutdown_connections 2024-09-16T00:35:28.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.686+0000 7f7bceea1640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7bb00a3340 0x7f7bb00a5800 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.686+0000 7f7bceea1640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc8108300 0x7f7bc806d030 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.686+0000 7f7bceea1640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc8103780 0x7f7bc806caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.686+0000 7f7bceea1640 1 --2- 172.21.15.33:0/136681606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7bc8102dd0 0x7f7bc8073a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:28.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.686+0000 7f7bceea1640 1 -- 172.21.15.33:0/136681606 >> 172.21.15.33:0/136681606 conn(0x7f7bc80fc7d0 msgr2=0x7f7bc8105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:28.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.686+0000 7f7bceea1640 1 -- 172.21.15.33:0/136681606 shutdown_connections 2024-09-16T00:35:28.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:28.686+0000 7f7bceea1640 1 -- 172.21.15.33:0/136681606 wait complete. 2024-09-16T00:35:28.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:28 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:35:28.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:28 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:35:28.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:28 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:35:29.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 89777 15.98 MB/sec execute 155 sec latency 69.672 ms 2024-09-16T00:35:29.820 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:29 smithi083 ceph-mon[35242]: pgmap v900: 129 pgs: 129 active+clean; 551 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.9 MiB/s wr, 161 op/s 2024-09-16T00:35:29.820 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:35:29.821 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:35:29.821 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:35:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:29 smithi033 ceph-mon[31243]: pgmap v900: 129 pgs: 129 active+clean; 551 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.9 MiB/s wr, 161 op/s 2024-09-16T00:35:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:35:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:35:29.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:35:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:29 smithi060 ceph-mon[35185]: pgmap v900: 129 pgs: 129 active+clean; 551 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.9 MiB/s wr, 161 op/s 2024-09-16T00:35:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:35:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:35:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:35:30.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 90207 15.98 MB/sec execute 156 sec latency 183.332 ms 2024-09-16T00:35:30.438 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:35:30.439 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:35:30.747 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:35:31.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 90781 16.07 MB/sec execute 157 sec latency 56.757 ms 2024-09-16T00:35:31.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.292+0000 7fae9d941640 1 -- 172.21.15.33:0/1103933160 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae98102790 msgr2=0x7fae981067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:31.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.292+0000 7fae9d941640 1 --2- 172.21.15.33:0/1103933160 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae98102790 0x7fae981067e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009860 tx=0x7fae8c02a340 comp rx=0 tx=0).stop 2024-09-16T00:35:31.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.293+0000 7fae9d941640 1 -- 172.21.15.33:0/1103933160 shutdown_connections 2024-09-16T00:35:31.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.293+0000 7fae9d941640 1 --2- 172.21.15.33:0/1103933160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae98106f10 0x7fae9810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.293+0000 7fae9d941640 1 --2- 172.21.15.33:0/1103933160 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae98102790 0x7fae981067e0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.293+0000 7fae9d941640 1 --2- 172.21.15.33:0/1103933160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae98101de0 0x7fae981021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.293+0000 7fae9d941640 1 -- 172.21.15.33:0/1103933160 >> 172.21.15.33:0/1103933160 conn(0x7fae980fd8c0 msgr2=0x7fae980ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:31.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.293+0000 7fae9d941640 1 -- 172.21.15.33:0/1103933160 shutdown_connections 2024-09-16T00:35:31.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.293+0000 7fae9d941640 1 -- 172.21.15.33:0/1103933160 wait complete. 2024-09-16T00:35:31.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae9d941640 1 Processor -- start 2024-09-16T00:35:31.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae9d941640 1 -- start start 2024-09-16T00:35:31.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae9d941640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae98101de0 0x7fae98198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae9d941640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae98102790 0x7fae981989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae9d941640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae98106f10 0x7fae9819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae9d941640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae98111f70 con 0x7fae98101de0 2024-09-16T00:35:31.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae9d941640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fae98111df0 con 0x7fae98102790 2024-09-16T00:35:31.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae9d941640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae981120f0 con 0x7fae98106f10 2024-09-16T00:35:31.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae96ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae98101de0 0x7fae98198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae96ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae98101de0 0x7fae98198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46358/0 (socket says 172.21.15.33:46358) 2024-09-16T00:35:31.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae96ffd640 1 -- 172.21.15.33:0/3767424872 learned_addr learned my addr 172.21.15.33:0/3767424872 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:31.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae967fc640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae98102790 0x7fae981989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.294+0000 7fae977fe640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae98106f10 0x7fae9819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.295+0000 7fae96ffd640 1 -- 172.21.15.33:0/3767424872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae98106f10 msgr2=0x7fae9819cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:31.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.295+0000 7fae96ffd640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae98106f10 0x7fae9819cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.295+0000 7fae96ffd640 1 -- 172.21.15.33:0/3767424872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae98102790 msgr2=0x7fae981989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:31.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.295+0000 7fae96ffd640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae98102790 0x7fae981989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.295+0000 7fae96ffd640 1 -- 172.21.15.33:0/3767424872 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9819d420 con 0x7fae98101de0 2024-09-16T00:35:31.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.295+0000 7fae967fc640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae98102790 0x7fae981989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:31.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.295+0000 7fae96ffd640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae98101de0 0x7fae98198470 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fae8800e9f0 tx=0x7fae8800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:31.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.295+0000 7fae9c93f640 1 -- 172.21.15.33:0/3767424872 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8800cd80 con 0x7fae98101de0 2024-09-16T00:35:31.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.295+0000 7fae9d941640 1 -- 172.21.15.33:0/3767424872 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae9819d710 con 0x7fae98101de0 2024-09-16T00:35:31.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.295+0000 7fae9c93f640 1 -- 172.21.15.33:0/3767424872 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae88010910 con 0x7fae98101de0 2024-09-16T00:35:31.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.295+0000 7fae9d941640 1 -- 172.21.15.33:0/3767424872 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae98074420 con 0x7fae98101de0 2024-09-16T00:35:31.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.295+0000 7fae9c93f640 1 -- 172.21.15.33:0/3767424872 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae88026060 con 0x7fae98101de0 2024-09-16T00:35:31.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.299+0000 7fae9fd1a740 1 Processor -- start 2024-09-16T00:35:31.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.299+0000 7fae9c93f640 1 -- 172.21.15.33:0/3767424872 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fae88010ac0 con 0x7fae98101de0 2024-09-16T00:35:31.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.299+0000 7fae9fd1a740 1 -- start start 2024-09-16T00:35:31.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.299+0000 7fae9fd1a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d530750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.299+0000 7fae9fd1a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d53b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.299+0000 7fae9fd1a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d53d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.299+0000 7fae9fd1a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55820d48d280 con 0x55820d53b550 2024-09-16T00:35:31.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.299+0000 7fae9fd1a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55820d434b10 con 0x55820d530c90 2024-09-16T00:35:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.299+0000 7fae9fd1a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55820d489240 con 0x55820d530370 2024-09-16T00:35:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.300+0000 7fae967fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d53d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.300+0000 7fae967fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d53d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46362/0 (socket says 172.21.15.33:46362) 2024-09-16T00:35:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.300+0000 7fae967fc640 1 -- 172.21.15.33:0/1659869279 learned_addr learned my addr 172.21.15.33:0/1659869279 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.300+0000 7fae96ffd640 1 --2- 172.21.15.33:0/1659869279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d53b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.300+0000 7fae96ffd640 1 -- 172.21.15.33:0/1659869279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 msgr2=0x55820d530750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.300+0000 7fae96ffd640 1 --2- 172.21.15.33:0/1659869279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d530750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.300+0000 7fae96ffd640 1 -- 172.21.15.33:0/1659869279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 msgr2=0x55820d53d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.300+0000 7fae96ffd640 1 --2- 172.21.15.33:0/1659869279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d53d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.300+0000 7fae96ffd640 1 -- 172.21.15.33:0/1659869279 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55820d4411f0 con 0x55820d530c90 2024-09-16T00:35:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.300+0000 7fae9c93f640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae800a3240 0x7fae800a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.301+0000 7fae96ffd640 1 --2- 172.21.15.33:0/1659869279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d53b010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fae880cd700 tx=0x7fae880d8780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20111fee084aa8fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:31.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.301+0000 7fae977fe640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae800a3240 0x7fae800a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.301+0000 7fae9c93f640 1 -- 172.21.15.33:0/3767424872 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fae880d0140 con 0x7fae98101de0 2024-09-16T00:35:31.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.301+0000 7fae7dffb640 1 -- 172.21.15.33:0/1659869279 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae880985d0 con 0x55820d530c90 2024-09-16T00:35:31.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.301+0000 7fae7dffb640 1 -- 172.21.15.33:0/1659869279 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae8809c330 con 0x55820d530c90 2024-09-16T00:35:31.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.301+0000 7fae7dffb640 1 -- 172.21.15.33:0/1659869279 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8809bc90 con 0x55820d530c90 2024-09-16T00:35:31.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.302+0000 7fae9fd1a740 1 -- 172.21.15.33:0/1659869279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 msgr2=0x55820d53b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:31.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.302+0000 7fae977fe640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae800a3240 0x7fae800a5700 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fae90005fd0 tx=0x7fae90010040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:31.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.302+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/1659869279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d53b010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fae880cd700 tx=0x7fae880d8780 comp rx=0 tx=0).stop 2024-09-16T00:35:31.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.302+0000 7fae9fd1a740 1 -- 172.21.15.33:0/1659869279 shutdown_connections 2024-09-16T00:35:31.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.302+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/1659869279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d53d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.302+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/1659869279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d53b010 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.302+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/1659869279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d530750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.302+0000 7fae9fd1a740 1 -- 172.21.15.33:0/1659869279 >> 172.21.15.33:0/1659869279 conn(0x55820d52c900 msgr2=0x55820d52cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:31.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.302+0000 7fae9fd1a740 1 -- 172.21.15.33:0/1659869279 shutdown_connections 2024-09-16T00:35:31.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.302+0000 7fae9fd1a740 1 -- 172.21.15.33:0/1659869279 wait complete. 2024-09-16T00:35:31.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.303+0000 7fae9fd1a740 1 -- start start 2024-09-16T00:35:31.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.303+0000 7fae9fd1a740 1 Processor -- start 2024-09-16T00:35:31.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.303+0000 7fae9fd1a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d538490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.303+0000 7fae9fd1a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d5389d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.303+0000 7fae9fd1a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d538f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.303+0000 7fae9fd1a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55820d53e120 con 0x55820d53b550 2024-09-16T00:35:31.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.303+0000 7fae9fd1a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55820d447f10 con 0x55820d530c90 2024-09-16T00:35:31.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.303+0000 7fae9fd1a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55820d53e2a0 con 0x55820d530370 2024-09-16T00:35:31.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.303+0000 7fae9fd1a740 10 client.0 _ll_register_callbacks cb 0x55820d410a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:31.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.304+0000 7fae967fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d5389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.304+0000 7fae967fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d5389d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51008/0 (socket says 172.21.15.33:51008) 2024-09-16T00:35:31.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.304+0000 7fae967fc640 1 -- 172.21.15.33:0/56591936 learned_addr learned my addr 172.21.15.33:0/56591936 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:31.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.304+0000 7fae977fe640 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d538f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.304+0000 7fae96ffd640 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d538490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.304+0000 7fae967fc640 1 -- 172.21.15.33:0/56591936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 msgr2=0x55820d538490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:31.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.304+0000 7fae967fc640 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d538490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.304+0000 7fae967fc640 1 -- 172.21.15.33:0/56591936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 msgr2=0x55820d538f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:31.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.304+0000 7fae967fc640 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d538f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.304+0000 7fae967fc640 1 -- 172.21.15.33:0/56591936 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55820d1fe6a0 con 0x55820d530c90 2024-09-16T00:35:31.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.304+0000 7fae76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:35:31.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.304+0000 7fae967fc640 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d5389d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009830 tx=0x7fae8c0045f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63bbb6f362e2e9c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:31.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.305+0000 7fae76ffd640 1 -- 172.21.15.33:0/56591936 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c037040 con 0x55820d530c90 2024-09-16T00:35:31.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.305+0000 7fae9fd1a740 1 -- 172.21.15.33:0/56591936 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae8809c330 con 0x55820d530c90 2024-09-16T00:35:31.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.305+0000 7fae76ffd640 1 -- 172.21.15.33:0/56591936 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fae8c030960 con 0x55820d530c90 2024-09-16T00:35:31.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.305+0000 7fae76ffd640 1 -- 172.21.15.33:0/56591936 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fae8c04c050 con 0x55820d530c90 2024-09-16T00:35:31.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.305+0000 7fae9fd1a740 10 client.28190 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:31.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.305+0000 7fae9fd1a740 1 -- 172.21.15.33:0/56591936 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55820d549dd0 con 0x55820d530c90 2024-09-16T00:35:31.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.306+0000 7fae76ffd640 1 -- 172.21.15.33:0/56591936 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae8809c330 con 0x55820d530c90 2024-09-16T00:35:31.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.306+0000 7fae76ffd640 1 -- 172.21.15.33:0/56591936 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fae8c03d7c0 con 0x55820d530c90 2024-09-16T00:35:31.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.306+0000 7fae9fd1a740 10 client.28190 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:31.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.306+0000 7fae9fd1a740 10 client.28190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:35:31.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.306+0000 7fae9fd1a740 20 client.28190 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:31.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.306+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55820d54b230 0x55820d54d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.306+0000 7fae9fd1a740 4 client.28190 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:35:31.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.306+0000 7fae9fd1a740 1 -- 172.21.15.33:0/56591936 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55820d49a7f0 con 0x55820d54b230 2024-09-16T00:35:31.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.306+0000 7fae96ffd640 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55820d54b230 0x55820d54d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.307+0000 7fae96ffd640 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55820d54b230 0x55820d54d670 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=705a9dd8d8f96738 server_cookie=71c6b8fb941be826 in_seq=0 out_seq=0 2024-09-16T00:35:31.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.307+0000 7fae76ffd640 10 client.28190 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:31.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.308+0000 7fae76ffd640 1 -- 172.21.15.33:0/56591936 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55820d49a7f0 con 0x55820d54b230 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.308+0000 7fae76ffd640 10 client.28190 handle_command_reply: tid=0 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.308+0000 7fae9fd1a740 1 client.28190 shutdown 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.308+0000 7fae9fd1a740 1 -- 172.21.15.33:0/56591936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 msgr2=0x55820d5389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.308+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d5389d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009830 tx=0x7fae8c0045f0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.308+0000 7fae9fd1a740 1 -- 172.21.15.33:0/56591936 shutdown_connections 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.308+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55820d54b230 0x55820d54d670 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.309+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d538f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.309+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d5389d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.309+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/56591936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d538490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.309+0000 7fae9fd1a740 1 -- 172.21.15.33:0/56591936 >> 172.21.15.33:0/56591936 conn(0x55820d52c900 msgr2=0x55820d523270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.309+0000 7fae76ffd640 0 client.28190 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.309+0000 7fae9fd1a740 1 -- 172.21.15.33:0/56591936 shutdown_connections 2024-09-16T00:35:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.309+0000 7fae9fd1a740 1 -- 172.21.15.33:0/56591936 wait complete. 2024-09-16T00:35:31.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.309+0000 7fae9fd1a740 20 client.28190 trim_cache size 0 max 16384 2024-09-16T00:35:31.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.330+0000 7fae9fd1a740 1 Processor -- start 2024-09-16T00:35:31.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.330+0000 7fae9fd1a740 1 -- start start 2024-09-16T00:35:31.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.330+0000 7fae9fd1a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d538d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.330+0000 7fae9fd1a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d523160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae9fd1a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d552f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae9fd1a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae981af850 con 0x55820d53b550 2024-09-16T00:35:31.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae9fd1a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fae981ad120 con 0x55820d530c90 2024-09-16T00:35:31.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae9fd1a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae981aaf70 con 0x55820d530370 2024-09-16T00:35:31.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae96ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d538d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae96ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d538d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40010/0 (socket says 172.21.15.33:40010) 2024-09-16T00:35:31.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae96ffd640 1 -- 172.21.15.33:0/980210736 learned_addr learned my addr 172.21.15.33:0/980210736 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:31.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae977fe640 1 --2- 172.21.15.33:0/980210736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d552f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae967fc640 1 --2- 172.21.15.33:0/980210736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d523160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae96ffd640 1 -- 172.21.15.33:0/980210736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 msgr2=0x55820d523160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:31.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae96ffd640 1 --2- 172.21.15.33:0/980210736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d523160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae96ffd640 1 -- 172.21.15.33:0/980210736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 msgr2=0x55820d552f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:31.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae96ffd640 1 --2- 172.21.15.33:0/980210736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d552f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae96ffd640 1 -- 172.21.15.33:0/980210736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55820d530160 con 0x55820d530370 2024-09-16T00:35:31.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae977fe640 1 --2- 172.21.15.33:0/980210736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d552f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:31.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae967fc640 1 --2- 172.21.15.33:0/980210736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d523160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:31.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.331+0000 7fae96ffd640 1 --2- 172.21.15.33:0/980210736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d538d60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fae8809a5e0 tx=0x7fae880bc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ef71549a352cfdf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:31.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.332+0000 7fae7d7fa640 1 -- 172.21.15.33:0/980210736 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8809be70 con 0x55820d530370 2024-09-16T00:35:31.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.332+0000 7fae7d7fa640 1 -- 172.21.15.33:0/980210736 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae880a0630 con 0x55820d530370 2024-09-16T00:35:31.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.332+0000 7fae7d7fa640 1 -- 172.21.15.33:0/980210736 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8809c100 con 0x55820d530370 2024-09-16T00:35:31.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.333+0000 7fae9fd1a740 1 -- 172.21.15.33:0/980210736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 msgr2=0x55820d538d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:31.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.333+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/980210736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d538d60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fae8809a5e0 tx=0x7fae880bc040 comp rx=0 tx=0).stop 2024-09-16T00:35:31.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.333+0000 7fae9fd1a740 1 -- 172.21.15.33:0/980210736 shutdown_connections 2024-09-16T00:35:31.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.333+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/980210736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d53b550 0x55820d552f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.333+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/980210736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530c90 0x55820d523160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.333+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/980210736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d530370 0x55820d538d60 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.333+0000 7fae9fd1a740 1 -- 172.21.15.33:0/980210736 >> 172.21.15.33:0/980210736 conn(0x55820d52c900 msgr2=0x55820d522a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:31.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.333+0000 7fae9fd1a740 1 -- 172.21.15.33:0/980210736 shutdown_connections 2024-09-16T00:35:31.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.333+0000 7fae9fd1a740 1 -- 172.21.15.33:0/980210736 wait complete. 2024-09-16T00:35:31.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.333+0000 7fae9fd1a740 1 -- start start 2024-09-16T00:35:31.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.333+0000 7fae9fd1a740 1 Processor -- start 2024-09-16T00:35:31.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae9fd1a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530370 0x55820d53f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae9fd1a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d530c90 0x55820d53fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae9fd1a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d53b550 0x55820d53ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae9fd1a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae981ab5e0 con 0x55820d530c90 2024-09-16T00:35:31.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae9fd1a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fae981aca50 con 0x55820d530370 2024-09-16T00:35:31.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae9fd1a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae981abcb0 con 0x55820d53b550 2024-09-16T00:35:31.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae9fd1a740 10 client.0 _ll_register_callbacks cb 0x7fae90009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:31.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae967fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d530c90 0x55820d53fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae977fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d53b550 0x55820d53ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae96ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530370 0x55820d53f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae96ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530370 0x55820d53f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51024/0 (socket says 172.21.15.33:51024) 2024-09-16T00:35:31.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae977fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d53b550 0x55820d53ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40020/0 (socket says 172.21.15.33:40020) 2024-09-16T00:35:31.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae967fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d530c90 0x55820d53fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46378/0 (socket says 172.21.15.33:46378) 2024-09-16T00:35:31.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae96ffd640 1 -- 172.21.15.33:0/2563800759 learned_addr learned my addr 172.21.15.33:0/2563800759 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:31.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae96ffd640 1 -- 172.21.15.33:0/2563800759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d53b550 msgr2=0x55820d53ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:31.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae96ffd640 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d53b550 0x55820d53ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae96ffd640 1 -- 172.21.15.33:0/2563800759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d530c90 msgr2=0x55820d53fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:31.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae96ffd640 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d530c90 0x55820d53fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae96ffd640 1 -- 172.21.15.33:0/2563800759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55820d5b5160 con 0x55820d530370 2024-09-16T00:35:31.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae977fe640 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d53b550 0x55820d53ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:31.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.334+0000 7fae967fc640 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d530c90 0x55820d53fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:31.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.335+0000 7fae96ffd640 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530370 0x55820d53f520 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fae880bc010 tx=0x7fae880d37b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12c3a204887577b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:31.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.335+0000 7fae76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:35:31.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.335+0000 7fae76ffd640 1 -- 172.21.15.33:0/2563800759 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8809c100 con 0x55820d530370 2024-09-16T00:35:31.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.335+0000 7fae76ffd640 1 -- 172.21.15.33:0/2563800759 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fae880d3c70 con 0x55820d530370 2024-09-16T00:35:31.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.335+0000 7fae9fd1a740 1 -- 172.21.15.33:0/2563800759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae880a0630 con 0x55820d530370 2024-09-16T00:35:31.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.335+0000 7fae76ffd640 1 -- 172.21.15.33:0/2563800759 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fae880985d0 con 0x55820d530370 2024-09-16T00:35:31.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.335+0000 7fae9fd1a740 10 client.28196 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:31.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.335+0000 7fae9fd1a740 1 -- 172.21.15.33:0/2563800759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55820d5b6460 con 0x55820d530370 2024-09-16T00:35:31.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.335+0000 7fae76ffd640 1 -- 172.21.15.33:0/2563800759 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae880a0630 con 0x55820d530370 2024-09-16T00:35:31.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.336+0000 7fae76ffd640 1 -- 172.21.15.33:0/2563800759 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fae880989a0 con 0x55820d530370 2024-09-16T00:35:31.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.336+0000 7fae9fd1a740 10 client.28196 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:31.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.336+0000 7fae9fd1a740 10 client.28196 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:35:31.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.336+0000 7fae9fd1a740 20 client.28196 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:31.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.336+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55820d5b66e0 0x55820d5b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:31.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.336+0000 7fae9fd1a740 4 client.28196 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:35:31.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.336+0000 7fae9fd1a740 1 -- 172.21.15.33:0/2563800759 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55820d531910 con 0x55820d5b66e0 2024-09-16T00:35:31.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.336+0000 7fae967fc640 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55820d5b66e0 0x55820d5b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:31.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.336+0000 7fae967fc640 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55820d5b66e0 0x55820d5b8ad0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88eedcf2d01e4d99 server_cookie=326346725dbfab95 in_seq=0 out_seq=0 2024-09-16T00:35:31.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.336+0000 7fae76ffd640 10 client.28196 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:31.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.337+0000 7fae76ffd640 1 -- 172.21.15.33:0/2563800759 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55820d531910 con 0x55820d5b66e0 2024-09-16T00:35:31.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.337+0000 7fae76ffd640 10 client.28196 handle_command_reply: tid=0 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.337+0000 7fae9fd1a740 1 client.28196 shutdown 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.338+0000 7fae9fd1a740 1 -- 172.21.15.33:0/2563800759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530370 msgr2=0x55820d53f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.338+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530370 0x55820d53f520 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fae880bc010 tx=0x7fae880d37b0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.338+0000 7fae9fd1a740 1 -- 172.21.15.33:0/2563800759 shutdown_connections 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.338+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55820d5b66e0 0x55820d5b8ad0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.338+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55820d53b550 0x55820d53ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.338+0000 7fae76ffd640 0 client.28196 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.338+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55820d530c90 0x55820d53fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.338+0000 7fae9fd1a740 1 --2- 172.21.15.33:0/2563800759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55820d530370 0x55820d53f520 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.338+0000 7fae9fd1a740 1 -- 172.21.15.33:0/2563800759 >> 172.21.15.33:0/2563800759 conn(0x55820d52c900 msgr2=0x55820d523270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.338+0000 7fae9fd1a740 1 -- 172.21.15.33:0/2563800759 shutdown_connections 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.338+0000 7fae9fd1a740 1 -- 172.21.15.33:0/2563800759 wait complete. 2024-09-16T00:35:31.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.338+0000 7fae9fd1a740 20 client.28196 trim_cache size 0 max 16384 2024-09-16T00:35:31.346 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:35:31.346 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:35:31.346 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "8a3190e5-bb0f-42a2-bbc5-addabec36d51", 2024-09-16T00:35:31.346 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:35:31.346 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:35:31.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.339+0000 7fae9d941640 1 -- 172.21.15.33:0/3767424872 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae800a3240 msgr2=0x7fae800a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:31.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.339+0000 7fae9d941640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae800a3240 0x7fae800a5700 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fae90005fd0 tx=0x7fae90010040 comp rx=0 tx=0).stop 2024-09-16T00:35:31.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.339+0000 7fae9d941640 1 -- 172.21.15.33:0/3767424872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae98101de0 msgr2=0x7fae98198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:31.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.339+0000 7fae9d941640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae98101de0 0x7fae98198470 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fae8800e9f0 tx=0x7fae8800eeb0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.339+0000 7fae9d941640 1 -- 172.21.15.33:0/3767424872 shutdown_connections 2024-09-16T00:35:31.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.339+0000 7fae9d941640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae800a3240 0x7fae800a5700 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.339+0000 7fae9d941640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae98106f10 0x7fae9819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.339+0000 7fae9d941640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae98102790 0x7fae981989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.339+0000 7fae9d941640 1 --2- 172.21.15.33:0/3767424872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae98101de0 0x7fae98198470 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:31.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.339+0000 7fae9d941640 1 -- 172.21.15.33:0/3767424872 >> 172.21.15.33:0/3767424872 conn(0x7fae980fd8c0 msgr2=0x7fae980febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:31.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.339+0000 7fae9d941640 1 -- 172.21.15.33:0/3767424872 shutdown_connections 2024-09-16T00:35:31.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:31.339+0000 7fae9d941640 1 -- 172.21.15.33:0/3767424872 wait complete. 2024-09-16T00:35:31.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:31 smithi033 ceph-mon[31243]: pgmap v901: 129 pgs: 129 active+clean; 553 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 3.7 MiB/s wr, 275 op/s 2024-09-16T00:35:31.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:31 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:35:31.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:31 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:31.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:31 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:35:31.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:31 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:31.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:31 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:35:31.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:31 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:31.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:31 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:35:31.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:31 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:31 smithi083 ceph-mon[35242]: pgmap v901: 129 pgs: 129 active+clean; 553 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 3.7 MiB/s wr, 275 op/s 2024-09-16T00:35:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:31 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:35:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:31 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:31 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:35:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:31 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:31 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:35:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:31 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:31 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:35:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:31 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:31 smithi060 ceph-mon[35185]: pgmap v901: 129 pgs: 129 active+clean; 553 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 3.7 MiB/s wr, 275 op/s 2024-09-16T00:35:31.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:31 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:35:31.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:31 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:31.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:31 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:35:31.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:31 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:31.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:31 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:35:31.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:31 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:31.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:31 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:35:31.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:31 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 91442 16.19 MB/sec execute 158 sec latency 80.764 ms 2024-09-16T00:35:32.546 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:35:32.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:35:32.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:35:32.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:35:32.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:35:32.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:35:32.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:35:32.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:35:32.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:35:32.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:32 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.886 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:35:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:35:32.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:32 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:35:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:35:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:35:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:35:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:35:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:35:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:35:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:35:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:32.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:32 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:35:33.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 92130 16.36 MB/sec execute 159 sec latency 55.346 ms 2024-09-16T00:35:33.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:33 smithi033 ceph-mon[31243]: pgmap v902: 129 pgs: 129 active+clean; 558 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.3 MiB/s wr, 334 op/s 2024-09-16T00:35:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:33 smithi083 ceph-mon[35242]: pgmap v902: 129 pgs: 129 active+clean; 558 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.3 MiB/s wr, 334 op/s 2024-09-16T00:35:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:33 smithi060 ceph-mon[35185]: pgmap v902: 129 pgs: 129 active+clean; 558 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.3 MiB/s wr, 334 op/s 2024-09-16T00:35:33.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:33.998+0000 7fee5f87f640 1 -- 172.21.15.33:0/97498192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee58102ef0 msgr2=0x7fee5810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:33.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:33.998+0000 7fee5f87f640 1 --2- 172.21.15.33:0/97498192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee58102ef0 0x7fee5810da40 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fee50009ca0 tx=0x7fee5002ea50 comp rx=0 tx=0).stop 2024-09-16T00:35:33.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:33.998+0000 7fee5f87f640 1 -- 172.21.15.33:0/97498192 shutdown_connections 2024-09-16T00:35:33.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:33.998+0000 7fee5f87f640 1 --2- 172.21.15.33:0/97498192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee58102ef0 0x7fee5810da40 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:33.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:33.998+0000 7fee5f87f640 1 --2- 172.21.15.33:0/97498192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee58102570 0x7fee581029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:33.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:33.998+0000 7fee5f87f640 1 --2- 172.21.15.33:0/97498192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee58100e20 0x7fee58101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:33.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:33.998+0000 7fee5f87f640 1 -- 172.21.15.33:0/97498192 >> 172.21.15.33:0/97498192 conn(0x7fee580fc7d0 msgr2=0x7fee580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:33.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:33.999+0000 7fee5f87f640 1 -- 172.21.15.33:0/97498192 shutdown_connections 2024-09-16T00:35:34.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:33.999+0000 7fee5f87f640 1 -- 172.21.15.33:0/97498192 wait complete. 2024-09-16T00:35:34.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.000+0000 7fee5f87f640 1 Processor -- start 2024-09-16T00:35:34.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.000+0000 7fee5f87f640 1 -- start start 2024-09-16T00:35:34.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.000+0000 7fee5f87f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee58100e20 0x7fee58073950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.000+0000 7fee5f87f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee58102570 0x7fee58073e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.000+0000 7fee5f87f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee58102ef0 0x7fee5806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.000+0000 7fee5f87f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fee5810fe40 con 0x7fee58102ef0 2024-09-16T00:35:34.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.001+0000 7fee5f87f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fee5810fcc0 con 0x7fee58102570 2024-09-16T00:35:34.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.001+0000 7fee5f87f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee5810ffc0 con 0x7fee58100e20 2024-09-16T00:35:34.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.001+0000 7fee5ce1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee58102570 0x7fee58073e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.001+0000 7fee5ce1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee58102570 0x7fee58073e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51044/0 (socket says 172.21.15.33:51044) 2024-09-16T00:35:34.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.001+0000 7fee5ce1a640 1 -- 172.21.15.33:0/2723830828 learned_addr learned my addr 172.21.15.33:0/2723830828 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:34.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.002+0000 7fee5d61b640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee58100e20 0x7fee58073950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.002+0000 7fee5de1c640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee58102ef0 0x7fee5806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.002+0000 7fee5ce1a640 1 -- 172.21.15.33:0/2723830828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee58100e20 msgr2=0x7fee58073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:34.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.002+0000 7fee5ce1a640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee58100e20 0x7fee58073950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.002+0000 7fee5ce1a640 1 -- 172.21.15.33:0/2723830828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee58102ef0 msgr2=0x7fee5806cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:34.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.002+0000 7fee5ce1a640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee58102ef0 0x7fee5806cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.002+0000 7fee5ce1a640 1 -- 172.21.15.33:0/2723830828 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5806d430 con 0x7fee58102570 2024-09-16T00:35:34.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.003+0000 7fee5ce1a640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee58102570 0x7fee58073e90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fee4800b4a0 tx=0x7fee4800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:34.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.004+0000 7fee4e7fc640 1 -- 172.21.15.33:0/2723830828 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48014070 con 0x7fee58102570 2024-09-16T00:35:34.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.004+0000 7fee4e7fc640 1 -- 172.21.15.33:0/2723830828 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee48010af0 con 0x7fee58102570 2024-09-16T00:35:34.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.004+0000 7fee4e7fc640 1 -- 172.21.15.33:0/2723830828 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48017d10 con 0x7fee58102570 2024-09-16T00:35:34.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.004+0000 7fee5f87f640 1 -- 172.21.15.33:0/2723830828 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee5806d720 con 0x7fee58102570 2024-09-16T00:35:34.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.004+0000 7fee5f87f640 1 -- 172.21.15.33:0/2723830828 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee580716b0 con 0x7fee58102570 2024-09-16T00:35:34.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.007+0000 7fee4e7fc640 1 -- 172.21.15.33:0/2723830828 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fee48017070 con 0x7fee58102570 2024-09-16T00:35:34.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee61c58740 1 Processor -- start 2024-09-16T00:35:34.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee61c58740 1 -- start start 2024-09-16T00:35:34.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee61c58740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0789384a0 0x55c078938880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee61c58740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078943140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee61c58740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078943680 0x55c078945a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee61c58740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c07889e2e0 con 0x55c078943680 2024-09-16T00:35:34.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee61c58740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c078845ce0 con 0x55c0789384a0 2024-09-16T00:35:34.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee61c58740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c07889a2a0 con 0x55c078938dc0 2024-09-16T00:35:34.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee4e7fc640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee440a32c0 0x7fee440a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee5de1c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0789384a0 0x55c078938880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee5de1c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0789384a0 0x55c078938880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:51058/0 (socket says 172.21.15.33:51058) 2024-09-16T00:35:34.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee5de1c640 1 -- 172.21.15.33:0/1405787133 learned_addr learned my addr 172.21.15.33:0/1405787133 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:34.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.008+0000 7fee5d61b640 1 --2- 172.21.15.33:0/1405787133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078943140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.009+0000 7fee5de1c640 1 -- 172.21.15.33:0/1405787133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 msgr2=0x55c078943140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:34.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.009+0000 7fee5de1c640 1 --2- 172.21.15.33:0/1405787133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078943140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.009+0000 7fee5de1c640 1 -- 172.21.15.33:0/1405787133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078943680 msgr2=0x55c078945a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:34.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.009+0000 7fee5ce1a640 1 --2- 172.21.15.33:0/1405787133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078943680 0x55c078945a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.009+0000 7fee5de1c640 1 --2- 172.21.15.33:0/1405787133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078943680 0x55c078945a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.009+0000 7fee5de1c640 1 -- 172.21.15.33:0/1405787133 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c07882a440 con 0x55c0789384a0 2024-09-16T00:35:34.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.009+0000 7fee4e7fc640 1 -- 172.21.15.33:0/2723830828 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fee480cefd0 con 0x7fee58102570 2024-09-16T00:35:34.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.009+0000 7fee5d61b640 1 --2- 172.21.15.33:0/1405787133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078943140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:35:34.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.010+0000 7fee5de1c640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee440a32c0 0x7fee440a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.010+0000 7fee5de1c640 1 --2- 172.21.15.33:0/1405787133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0789384a0 0x55c078938880 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fee5002f040 tx=0x7fee5000f360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e06f51d1d730e9c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:34.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.011+0000 7fee437fe640 1 -- 172.21.15.33:0/1405787133 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee500098c0 con 0x55c0789384a0 2024-09-16T00:35:34.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.011+0000 7fee437fe640 1 -- 172.21.15.33:0/1405787133 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee5003a8e0 con 0x55c0789384a0 2024-09-16T00:35:34.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.011+0000 7fee437fe640 1 -- 172.21.15.33:0/1405787133 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50047940 con 0x55c0789384a0 2024-09-16T00:35:34.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.012+0000 7fee61c58740 1 -- 172.21.15.33:0/1405787133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0789384a0 msgr2=0x55c078938880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:34.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.012+0000 7fee61c58740 1 --2- 172.21.15.33:0/1405787133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0789384a0 0x55c078938880 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fee5002f040 tx=0x7fee5000f360 comp rx=0 tx=0).stop 2024-09-16T00:35:34.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.012+0000 7fee5de1c640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee440a32c0 0x7fee440a5780 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fee5004c000 tx=0x7fee50049000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:34.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.012+0000 7fee61c58740 1 -- 172.21.15.33:0/1405787133 shutdown_connections 2024-09-16T00:35:34.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.012+0000 7fee61c58740 1 --2- 172.21.15.33:0/1405787133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078943680 0x55c078945a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.012+0000 7fee61c58740 1 --2- 172.21.15.33:0/1405787133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078943140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.012+0000 7fee61c58740 1 --2- 172.21.15.33:0/1405787133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0789384a0 0x55c078938880 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.012+0000 7fee61c58740 1 -- 172.21.15.33:0/1405787133 >> 172.21.15.33:0/1405787133 conn(0x55c078934a30 msgr2=0x55c078934e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:34.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.012+0000 7fee61c58740 1 -- 172.21.15.33:0/1405787133 shutdown_connections 2024-09-16T00:35:34.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.012+0000 7fee61c58740 1 -- 172.21.15.33:0/1405787133 wait complete. 2024-09-16T00:35:34.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.013+0000 7fee61c58740 1 -- start start 2024-09-16T00:35:34.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.013+0000 7fee61c58740 1 Processor -- start 2024-09-16T00:35:34.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.013+0000 7fee61c58740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0789384a0 0x55c078943070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee61c58740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078940320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee61c58740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078943680 0x55c078940910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee61c58740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c078946250 con 0x55c078943680 2024-09-16T00:35:34.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee61c58740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c078859050 con 0x55c0789384a0 2024-09-16T00:35:34.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee61c58740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c0789463d0 con 0x55c078938dc0 2024-09-16T00:35:34.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee61c58740 10 client.0 _ll_register_callbacks cb 0x55c078899160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:34.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee5de1c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078943680 0x55c078940910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee5de1c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078943680 0x55c078940910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46408/0 (socket says 172.21.15.33:46408) 2024-09-16T00:35:34.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee5de1c640 1 -- 172.21.15.33:0/3892296312 learned_addr learned my addr 172.21.15.33:0/3892296312 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:34.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee5d61b640 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0789384a0 0x55c078943070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee5ce1a640 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078940320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee5de1c640 1 -- 172.21.15.33:0/3892296312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 msgr2=0x55c078940320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:34.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee5de1c640 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078940320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee5de1c640 1 -- 172.21.15.33:0/3892296312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0789384a0 msgr2=0x55c078943070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:34.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee5de1c640 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0789384a0 0x55c078943070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.014+0000 7fee5de1c640 1 -- 172.21.15.33:0/3892296312 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c07860f3c0 con 0x55c078943680 2024-09-16T00:35:34.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.015+0000 7fee5ce1a640 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078940320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:35:34.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.015+0000 7fee5de1c640 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078943680 0x55c078940910 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fee50004aa0 tx=0x7fee5000f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce094f11d22a7509 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:34.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.015+0000 7fee40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:35:34.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.015+0000 7fee40ff9640 1 -- 172.21.15.33:0/3892296312 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5005f060 con 0x55c078943680 2024-09-16T00:35:34.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.015+0000 7fee40ff9640 1 -- 172.21.15.33:0/3892296312 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fee50047740 con 0x55c078943680 2024-09-16T00:35:34.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.015+0000 7fee61c58740 1 -- 172.21.15.33:0/3892296312 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee5003a8e0 con 0x55c078943680 2024-09-16T00:35:34.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.015+0000 7fee40ff9640 1 -- 172.21.15.33:0/3892296312 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fee50066050 con 0x55c078943680 2024-09-16T00:35:34.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.015+0000 7fee61c58740 10 client.18933 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:34.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.015+0000 7fee61c58740 1 -- 172.21.15.33:0/3892296312 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c078946250 con 0x55c078943680 2024-09-16T00:35:34.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.016+0000 7fee40ff9640 1 -- 172.21.15.33:0/3892296312 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee5003a8e0 con 0x55c078943680 2024-09-16T00:35:34.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.016+0000 7fee40ff9640 1 -- 172.21.15.33:0/3892296312 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fee5000b040 con 0x55c078943680 2024-09-16T00:35:34.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.016+0000 7fee61c58740 10 client.18933 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:34.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.016+0000 7fee61c58740 10 client.18933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:35:34.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.016+0000 7fee61c58740 20 client.18933 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:34.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.016+0000 7fee61c58740 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c078952e30 0x55c078955220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.016+0000 7fee61c58740 4 client.18933 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:35:34.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.016+0000 7fee61c58740 1 -- 172.21.15.33:0/3892296312 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0788ab850 con 0x55c078952e30 2024-09-16T00:35:34.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.016+0000 7fee5d61b640 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c078952e30 0x55c078955220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.017+0000 7fee5d61b640 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c078952e30 0x55c078955220 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36bd4cadb0f5efbf server_cookie=7c1f94b48a3b213d in_seq=0 out_seq=0 2024-09-16T00:35:34.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.017+0000 7fee40ff9640 10 client.18933 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:34.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.018+0000 7fee40ff9640 1 -- 172.21.15.33:0/3892296312 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c0788ab850 con 0x55c078952e30 2024-09-16T00:35:34.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.018+0000 7fee40ff9640 10 client.18933 handle_command_reply: tid=0 2024-09-16T00:35:34.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.019+0000 7fee61c58740 1 client.18933 shutdown 2024-09-16T00:35:34.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.019+0000 7fee61c58740 1 -- 172.21.15.33:0/3892296312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078943680 msgr2=0x55c078940910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:34.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.019+0000 7fee61c58740 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078943680 0x55c078940910 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fee50004aa0 tx=0x7fee5000f3d0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.019+0000 7fee61c58740 1 -- 172.21.15.33:0/3892296312 shutdown_connections 2024-09-16T00:35:34.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.019+0000 7fee61c58740 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c078952e30 0x55c078955220 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.019+0000 7fee40ff9640 0 client.18933 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:34.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.019+0000 7fee61c58740 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078943680 0x55c078940910 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.019+0000 7fee61c58740 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078940320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.019+0000 7fee61c58740 1 --2- 172.21.15.33:0/3892296312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c0789384a0 0x55c078943070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.019+0000 7fee61c58740 1 -- 172.21.15.33:0/3892296312 >> 172.21.15.33:0/3892296312 conn(0x55c078934a30 msgr2=0x55c07893a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:34.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.020+0000 7fee61c58740 1 -- 172.21.15.33:0/3892296312 shutdown_connections 2024-09-16T00:35:34.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.020+0000 7fee61c58740 1 -- 172.21.15.33:0/3892296312 wait complete. 2024-09-16T00:35:34.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.020+0000 7fee61c58740 20 client.18933 trim_cache size 0 max 16384 2024-09-16T00:35:34.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.055+0000 7fee61c58740 1 Processor -- start 2024-09-16T00:35:34.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.055+0000 7fee61c58740 1 -- start start 2024-09-16T00:35:34.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.055+0000 7fee61c58740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0789384a0 0x55c07894f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.055+0000 7fee61c58740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078938dc0 0x55c07895a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.056+0000 7fee61c58740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c078943680 0x55c07895ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.056+0000 7fee61c58740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fee581c0840 con 0x55c078938dc0 2024-09-16T00:35:34.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.056+0000 7fee61c58740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fee581be110 con 0x55c078943680 2024-09-16T00:35:34.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.056+0000 7fee61c58740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee581bbf60 con 0x55c0789384a0 2024-09-16T00:35:34.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.056+0000 7fee5d61b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0789384a0 0x55c07894f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.056+0000 7fee5ce1a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078938dc0 0x55c07895a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.056+0000 7fee5ce1a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078938dc0 0x55c07895a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46424/0 (socket says 172.21.15.33:46424) 2024-09-16T00:35:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.056+0000 7fee5ce1a640 1 -- 172.21.15.33:0/3378463680 learned_addr learned my addr 172.21.15.33:0/3378463680 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.058+0000 7fee5ce1a640 1 -- 172.21.15.33:0/3378463680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0789384a0 msgr2=0x55c07894f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.058+0000 7fee5ce1a640 1 --2- 172.21.15.33:0/3378463680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0789384a0 0x55c07894f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.058+0000 7fee5ce1a640 1 -- 172.21.15.33:0/3378463680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c078943680 msgr2=0x55c07895ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.058+0000 7fee5ce1a640 1 --2- 172.21.15.33:0/3378463680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c078943680 0x55c07895ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.058+0000 7fee5ce1a640 1 -- 172.21.15.33:0/3378463680 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0789382d0 con 0x55c078938dc0 2024-09-16T00:35:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.059+0000 7fee5ce1a640 1 --2- 172.21.15.33:0/3378463680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078938dc0 0x55c07895a940 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fee4800be60 tx=0x7fee48017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceeafb918eab914b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.061+0000 7fee42ffd640 1 -- 172.21.15.33:0/3378463680 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4801e040 con 0x55c078938dc0 2024-09-16T00:35:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.061+0000 7fee42ffd640 1 -- 172.21.15.33:0/3378463680 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee480992c0 con 0x55c078938dc0 2024-09-16T00:35:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.062+0000 7fee61c58740 1 -- 172.21.15.33:0/3378463680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078938dc0 msgr2=0x55c07895a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.062+0000 7fee61c58740 1 --2- 172.21.15.33:0/3378463680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078938dc0 0x55c07895a940 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fee4800be60 tx=0x7fee48017730 comp rx=0 tx=0).stop 2024-09-16T00:35:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.063+0000 7fee61c58740 1 -- 172.21.15.33:0/3378463680 shutdown_connections 2024-09-16T00:35:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.063+0000 7fee61c58740 1 --2- 172.21.15.33:0/3378463680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c078943680 0x55c07895ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.063+0000 7fee61c58740 1 --2- 172.21.15.33:0/3378463680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c078938dc0 0x55c07895a940 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.063+0000 7fee61c58740 1 --2- 172.21.15.33:0/3378463680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0789384a0 0x55c07894f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.063+0000 7fee61c58740 1 -- 172.21.15.33:0/3378463680 >> 172.21.15.33:0/3378463680 conn(0x55c078934a30 msgr2=0x55c0789423a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.063+0000 7fee61c58740 1 -- 172.21.15.33:0/3378463680 shutdown_connections 2024-09-16T00:35:34.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.064+0000 7fee61c58740 1 -- 172.21.15.33:0/3378463680 wait complete. 2024-09-16T00:35:34.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.064+0000 7fee61c58740 1 -- start start 2024-09-16T00:35:34.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.064+0000 7fee61c58740 1 Processor -- start 2024-09-16T00:35:34.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.065+0000 7fee61c58740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0789384a0 0x55c07894bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.065+0000 7fee61c58740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078949120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.065+0000 7fee61c58740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c078943680 0x55c078949660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.065+0000 7fee61c58740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fee581bc5d0 con 0x55c0789384a0 2024-09-16T00:35:34.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.065+0000 7fee61c58740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fee581bda40 con 0x55c078943680 2024-09-16T00:35:34.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.065+0000 7fee61c58740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee581bcca0 con 0x55c078938dc0 2024-09-16T00:35:34.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.065+0000 7fee61c58740 10 client.0 _ll_register_callbacks cb 0x7fee5400b7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:35:34.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.065+0000 7fee5ce1a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078949120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.065+0000 7fee5d61b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0789384a0 0x55c07894bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.065+0000 7fee5d61b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0789384a0 0x55c07894bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46440/0 (socket says 172.21.15.33:46440) 2024-09-16T00:35:34.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.065+0000 7fee5d61b640 1 -- 172.21.15.33:0/298412388 learned_addr learned my addr 172.21.15.33:0/298412388 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:35:34.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.065+0000 7fee5de1c640 1 --2- 172.21.15.33:0/298412388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c078943680 0x55c078949660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.066+0000 7fee5d61b640 1 -- 172.21.15.33:0/298412388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 msgr2=0x55c078949120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:34.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.066+0000 7fee5d61b640 1 --2- 172.21.15.33:0/298412388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078949120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.066+0000 7fee5d61b640 1 -- 172.21.15.33:0/298412388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c078943680 msgr2=0x55c078949660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:35:34.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.066+0000 7fee5d61b640 1 --2- 172.21.15.33:0/298412388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c078943680 0x55c078949660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.066+0000 7fee5d61b640 1 -- 172.21.15.33:0/298412388 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c078949e90 con 0x55c0789384a0 2024-09-16T00:35:34.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.067+0000 7fee5d61b640 1 --2- 172.21.15.33:0/298412388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0789384a0 0x55c07894bdd0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x55c0789558b0 tx=0x7fee5400b3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f5910b90362977 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:35:34.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.069+0000 7fee40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:35:34.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.069+0000 7fee40ff9640 1 -- 172.21.15.33:0/298412388 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54004230 con 0x55c0789384a0 2024-09-16T00:35:34.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.069+0000 7fee61c58740 1 -- 172.21.15.33:0/298412388 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee480992c0 con 0x55c0789384a0 2024-09-16T00:35:34.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.069+0000 7fee40ff9640 1 -- 172.21.15.33:0/298412388 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fee540043d0 con 0x55c0789384a0 2024-09-16T00:35:34.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.071+0000 7fee40ff9640 1 -- 172.21.15.33:0/298412388 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fee54017780 con 0x55c0789384a0 2024-09-16T00:35:34.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.071+0000 7fee61c58740 10 client.18939 fetch_fsmap learned FSMap version 33 2024-09-16T00:35:34.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.071+0000 7fee61c58740 1 -- 172.21.15.33:0/298412388 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c0789cea00 con 0x55c0789384a0 2024-09-16T00:35:34.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.071+0000 7fee40ff9640 1 -- 172.21.15.33:0/298412388 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee480992c0 con 0x55c0789384a0 2024-09-16T00:35:34.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.072+0000 7fee40ff9640 1 -- 172.21.15.33:0/298412388 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fee54030420 con 0x55c0789384a0 2024-09-16T00:35:34.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.072+0000 7fee61c58740 10 client.18939 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:35:34.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.072+0000 7fee61c58740 10 client.18939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:35:34.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.072+0000 7fee61c58740 20 client.18939 populate_metadata read hostname 'smithi033' 2024-09-16T00:35:34.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.072+0000 7fee61c58740 1 --2- 172.21.15.33:0/298412388 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0789cf080 0x55c0789d14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:35:34.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.072+0000 7fee61c58740 4 client.18939 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:35:34.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.072+0000 7fee61c58740 1 -- 172.21.15.33:0/298412388 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c0788a5d60 con 0x55c0789cf080 2024-09-16T00:35:34.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.072+0000 7fee5ce1a640 1 --2- 172.21.15.33:0/298412388 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0789cf080 0x55c0789d14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:35:34.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.075+0000 7fee5ce1a640 1 --2- 172.21.15.33:0/298412388 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0789cf080 0x55c0789d14a0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8aa83248265bba95 server_cookie=7b4acf68ebc47610 in_seq=0 out_seq=0 2024-09-16T00:35:34.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.075+0000 7fee40ff9640 10 client.18939 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:34.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.076+0000 7fee40ff9640 1 -- 172.21.15.33:0/298412388 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c0788a5d60 con 0x55c0789cf080 2024-09-16T00:35:34.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.076+0000 7fee40ff9640 10 client.18939 handle_command_reply: tid=0 2024-09-16T00:35:34.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.076+0000 7fee61c58740 1 client.18939 shutdown 2024-09-16T00:35:34.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.077+0000 7fee61c58740 1 -- 172.21.15.33:0/298412388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0789384a0 msgr2=0x55c07894bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:34.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.077+0000 7fee61c58740 1 --2- 172.21.15.33:0/298412388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0789384a0 0x55c07894bdd0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x55c0789558b0 tx=0x7fee5400b3f0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.077+0000 7fee61c58740 1 -- 172.21.15.33:0/298412388 shutdown_connections 2024-09-16T00:35:34.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.077+0000 7fee61c58740 1 --2- 172.21.15.33:0/298412388 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c0789cf080 0x55c0789d14a0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.077+0000 7fee61c58740 1 --2- 172.21.15.33:0/298412388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c078943680 0x55c078949660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.077+0000 7fee61c58740 1 --2- 172.21.15.33:0/298412388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c078938dc0 0x55c078949120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.077+0000 7fee61c58740 1 --2- 172.21.15.33:0/298412388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0789384a0 0x55c07894bdd0 secure :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x55c0789558b0 tx=0x7fee5400b3f0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.077+0000 7fee61c58740 1 -- 172.21.15.33:0/298412388 >> 172.21.15.33:0/298412388 conn(0x55c078934a30 msgr2=0x55c078933370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:34.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.078+0000 7fee40ff9640 0 client.18939 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:35:34.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.079+0000 7fee61c58740 1 -- 172.21.15.33:0/298412388 shutdown_connections 2024-09-16T00:35:34.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.079+0000 7fee61c58740 1 -- 172.21.15.33:0/298412388 wait complete. 2024-09-16T00:35:34.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.079+0000 7fee61c58740 20 client.18939 trim_cache size 0 max 16384 2024-09-16T00:35:34.079 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:35:34.080 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:35:34.080 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:35:34.080 INFO:teuthology.orchestra.run.smithi033.stdout: "8a3190e5-bb0f-42a2-bbc5-addabec36d51": { 2024-09-16T00:35:34.080 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:35:34.080 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "8a3190e5-bb0f-42a2-bbc5-addabec36d51", 2024-09-16T00:35:34.080 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:35:34.080 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:35:34.080 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:35:34.080 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:35:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.080+0000 7fee5f87f640 1 -- 172.21.15.33:0/2723830828 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee440a32c0 msgr2=0x7fee440a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.080+0000 7fee5f87f640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee440a32c0 0x7fee440a5780 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fee5004c000 tx=0x7fee50049000 comp rx=0 tx=0).stop 2024-09-16T00:35:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.080+0000 7fee5f87f640 1 -- 172.21.15.33:0/2723830828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee58102570 msgr2=0x7fee58073e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:35:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.080+0000 7fee5f87f640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee58102570 0x7fee58073e90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fee4800b4a0 tx=0x7fee4800b960 comp rx=0 tx=0).stop 2024-09-16T00:35:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.080+0000 7fee5f87f640 1 -- 172.21.15.33:0/2723830828 shutdown_connections 2024-09-16T00:35:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.080+0000 7fee5f87f640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fee440a32c0 0x7fee440a5780 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.080+0000 7fee5f87f640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee58102ef0 0x7fee5806cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.081+0000 7fee5f87f640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee58102570 0x7fee58073e90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.081+0000 7fee5f87f640 1 --2- 172.21.15.33:0/2723830828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee58100e20 0x7fee58073950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:35:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.081+0000 7fee5f87f640 1 -- 172.21.15.33:0/2723830828 >> 172.21.15.33:0/2723830828 conn(0x7fee580fc7d0 msgr2=0x7fee580febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:35:34.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 92628 16.41 MB/sec execute 160 sec latency 115.178 ms 2024-09-16T00:35:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.081+0000 7fee5f87f640 1 -- 172.21.15.33:0/2723830828 shutdown_connections 2024-09-16T00:35:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:35:34.083+0000 7fee5f87f640 1 -- 172.21.15.33:0/2723830828 wait complete. 2024-09-16T00:35:35.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 93209 16.54 MB/sec execute 161 sec latency 108.712 ms 2024-09-16T00:35:35.223 INFO:tasks.cephfs.filesystem:scrub status for tag:8a3190e5-bb0f-42a2-bbc5-addabec36d51 - {'path': '/', 'tag': '8a3190e5-bb0f-42a2-bbc5-addabec36d51', 'options': 'recursive,force'} 2024-09-16T00:35:35.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:35 smithi033 ceph-mon[31243]: pgmap v903: 129 pgs: 129 active+clean; 565 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 411 op/s 2024-09-16T00:35:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:35 smithi083 ceph-mon[35242]: pgmap v903: 129 pgs: 129 active+clean; 565 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 411 op/s 2024-09-16T00:35:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:35 smithi060 ceph-mon[35185]: pgmap v903: 129 pgs: 129 active+clean; 565 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 411 op/s 2024-09-16T00:35:36.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 93808 16.60 MB/sec execute 162 sec latency 48.001 ms 2024-09-16T00:35:37.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 94309 16.68 MB/sec execute 163 sec latency 59.059 ms 2024-09-16T00:35:37.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:37 smithi033 ceph-mon[31243]: pgmap v904: 129 pgs: 129 active+clean; 571 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.2 MiB/s wr, 410 op/s 2024-09-16T00:35:37.961 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:37 smithi083 ceph-mon[35242]: pgmap v904: 129 pgs: 129 active+clean; 571 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.2 MiB/s wr, 410 op/s 2024-09-16T00:35:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:37 smithi060 ceph-mon[35185]: pgmap v904: 129 pgs: 129 active+clean; 571 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.2 MiB/s wr, 410 op/s 2024-09-16T00:35:38.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 94845 16.71 MB/sec execute 164 sec latency 180.815 ms 2024-09-16T00:35:39.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 95461 16.84 MB/sec execute 165 sec latency 194.046 ms 2024-09-16T00:35:39.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:39 smithi033 ceph-mon[31243]: pgmap v905: 129 pgs: 129 active+clean; 571 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 2.8 MiB/s wr, 344 op/s 2024-09-16T00:35:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:39 smithi083 ceph-mon[35242]: pgmap v905: 129 pgs: 129 active+clean; 571 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 2.8 MiB/s wr, 344 op/s 2024-09-16T00:35:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:39 smithi060 ceph-mon[35185]: pgmap v905: 129 pgs: 129 active+clean; 571 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 2.8 MiB/s wr, 344 op/s 2024-09-16T00:35:40.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 96032 16.89 MB/sec execute 166 sec latency 65.337 ms 2024-09-16T00:35:41.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 96741 17.07 MB/sec execute 167 sec latency 46.432 ms 2024-09-16T00:35:41.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:41 smithi033 ceph-mon[31243]: pgmap v906: 129 pgs: 129 active+clean; 580 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.7 MiB/s wr, 446 op/s 2024-09-16T00:35:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:41 smithi083 ceph-mon[35242]: pgmap v906: 129 pgs: 129 active+clean; 580 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.7 MiB/s wr, 446 op/s 2024-09-16T00:35:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:41 smithi060 ceph-mon[35185]: pgmap v906: 129 pgs: 129 active+clean; 580 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.7 MiB/s wr, 446 op/s 2024-09-16T00:35:42.081 INFO:tasks.workunit.client.0.smithi033.stdout: 10 97332 17.16 MB/sec execute 168 sec latency 96.165 ms 2024-09-16T00:35:42.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:42 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:35:42.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:42 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:35:42.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:42 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:35:42.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:42 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:35:42.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:42 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:35:42.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:42 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:35:42.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:42 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:35:42.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:42 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:35:42.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:42 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:35:43.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 97772 17.20 MB/sec execute 169 sec latency 87.847 ms 2024-09-16T00:35:43.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:43 smithi033 ceph-mon[31243]: pgmap v907: 129 pgs: 129 active+clean; 569 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.4 MiB/s wr, 385 op/s 2024-09-16T00:35:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:43 smithi083 ceph-mon[35242]: pgmap v907: 129 pgs: 129 active+clean; 569 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.4 MiB/s wr, 385 op/s 2024-09-16T00:35:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:43 smithi060 ceph-mon[35185]: pgmap v907: 129 pgs: 129 active+clean; 569 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.4 MiB/s wr, 385 op/s 2024-09-16T00:35:44.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 98318 17.27 MB/sec execute 170 sec latency 122.110 ms 2024-09-16T00:35:45.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 98953 17.38 MB/sec execute 171 sec latency 83.617 ms 2024-09-16T00:35:45.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:45 smithi033 ceph-mon[31243]: pgmap v908: 129 pgs: 129 active+clean; 567 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.5 MiB/s wr, 408 op/s 2024-09-16T00:35:45.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:45 smithi083 ceph-mon[35242]: pgmap v908: 129 pgs: 129 active+clean; 567 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.5 MiB/s wr, 408 op/s 2024-09-16T00:35:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:45 smithi060 ceph-mon[35185]: pgmap v908: 129 pgs: 129 active+clean; 567 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.5 MiB/s wr, 408 op/s 2024-09-16T00:35:46.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 99452 17.41 MB/sec execute 172 sec latency 100.018 ms 2024-09-16T00:35:47.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 100024 17.46 MB/sec execute 173 sec latency 82.726 ms 2024-09-16T00:35:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:47 smithi033 ceph-mon[31243]: pgmap v909: 129 pgs: 129 active+clean; 573 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 386 op/s 2024-09-16T00:35:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:47 smithi083 ceph-mon[35242]: pgmap v909: 129 pgs: 129 active+clean; 573 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 386 op/s 2024-09-16T00:35:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:47 smithi060 ceph-mon[35185]: pgmap v909: 129 pgs: 129 active+clean; 573 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 386 op/s 2024-09-16T00:35:48.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 100675 17.56 MB/sec execute 174 sec latency 114.907 ms 2024-09-16T00:35:49.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 101234 17.67 MB/sec execute 175 sec latency 253.629 ms 2024-09-16T00:35:49.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:49 smithi033 ceph-mon[31243]: pgmap v910: 129 pgs: 129 active+clean; 573 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 2.7 MiB/s wr, 315 op/s 2024-09-16T00:35:49.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:49 smithi083 ceph-mon[35242]: pgmap v910: 129 pgs: 129 active+clean; 573 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 2.7 MiB/s wr, 315 op/s 2024-09-16T00:35:49.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:49 smithi060 ceph-mon[35185]: pgmap v910: 129 pgs: 129 active+clean; 573 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 2.7 MiB/s wr, 315 op/s 2024-09-16T00:35:50.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 101723 17.74 MB/sec execute 176 sec latency 110.171 ms 2024-09-16T00:35:51.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 102093 17.72 MB/sec execute 177 sec latency 242.890 ms 2024-09-16T00:35:51.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:51 smithi033 ceph-mon[31243]: pgmap v911: 129 pgs: 129 active+clean; 579 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 3.4 MiB/s wr, 379 op/s 2024-09-16T00:35:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:51 smithi083 ceph-mon[35242]: pgmap v911: 129 pgs: 129 active+clean; 579 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 3.4 MiB/s wr, 379 op/s 2024-09-16T00:35:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:51 smithi060 ceph-mon[35185]: pgmap v911: 129 pgs: 129 active+clean; 579 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 3.4 MiB/s wr, 379 op/s 2024-09-16T00:35:52.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 102575 17.77 MB/sec execute 178 sec latency 61.406 ms 2024-09-16T00:35:53.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 103134 17.83 MB/sec execute 179 sec latency 277.438 ms 2024-09-16T00:35:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:53 smithi083 ceph-mon[35242]: pgmap v912: 129 pgs: 129 active+clean; 588 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.4 MiB/s wr, 394 op/s 2024-09-16T00:35:53.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:53 smithi060 ceph-mon[35185]: pgmap v912: 129 pgs: 129 active+clean; 588 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.4 MiB/s wr, 394 op/s 2024-09-16T00:35:54.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:53 smithi033 ceph-mon[31243]: pgmap v912: 129 pgs: 129 active+clean; 588 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.4 MiB/s wr, 394 op/s 2024-09-16T00:35:54.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 103674 17.90 MB/sec execute 180 sec latency 178.228 ms 2024-09-16T00:35:55.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 104230 17.93 MB/sec execute 181 sec latency 75.473 ms 2024-09-16T00:35:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:55 smithi083 ceph-mon[35242]: pgmap v913: 129 pgs: 129 active+clean; 589 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.2 MiB/s wr, 453 op/s 2024-09-16T00:35:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:55 smithi060 ceph-mon[35185]: pgmap v913: 129 pgs: 129 active+clean; 589 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.2 MiB/s wr, 453 op/s 2024-09-16T00:35:56.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:55 smithi033 ceph-mon[31243]: pgmap v913: 129 pgs: 129 active+clean; 589 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.2 MiB/s wr, 453 op/s 2024-09-16T00:35:56.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 104925 18.08 MB/sec execute 182 sec latency 48.593 ms 2024-09-16T00:35:57.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 105565 18.18 MB/sec execute 183 sec latency 88.039 ms 2024-09-16T00:35:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:57 smithi083 ceph-mon[35242]: pgmap v914: 129 pgs: 129 active+clean; 600 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.8 MiB/s wr, 551 op/s 2024-09-16T00:35:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:57 smithi060 ceph-mon[35185]: pgmap v914: 129 pgs: 129 active+clean; 600 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.8 MiB/s wr, 551 op/s 2024-09-16T00:35:58.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:57 smithi033 ceph-mon[31243]: pgmap v914: 129 pgs: 129 active+clean; 600 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.8 MiB/s wr, 551 op/s 2024-09-16T00:35:58.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 105905 18.18 MB/sec execute 184 sec latency 447.062 ms 2024-09-16T00:35:59.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 105922 18.09 MB/sec execute 185 sec latency 1359.944 ms 2024-09-16T00:35:59.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:35:59 smithi083 ceph-mon[35242]: pgmap v915: 129 pgs: 129 active+clean; 600 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.2 MiB/s wr, 477 op/s 2024-09-16T00:35:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:35:59 smithi060 ceph-mon[35185]: pgmap v915: 129 pgs: 129 active+clean; 600 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.2 MiB/s wr, 477 op/s 2024-09-16T00:36:00.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:35:59 smithi033 ceph-mon[31243]: pgmap v915: 129 pgs: 129 active+clean; 600 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.2 MiB/s wr, 477 op/s 2024-09-16T00:36:00.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 105944 18.00 MB/sec execute 186 sec latency 1691.847 ms 2024-09-16T00:36:01.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 106101 17.95 MB/sec execute 187 sec latency 221.237 ms 2024-09-16T00:36:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:01 smithi083 ceph-mon[35242]: pgmap v916: 129 pgs: 129 active+clean; 640 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 7.0 MiB/s wr, 573 op/s 2024-09-16T00:36:01.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:01 smithi060 ceph-mon[35185]: pgmap v916: 129 pgs: 129 active+clean; 640 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 7.0 MiB/s wr, 573 op/s 2024-09-16T00:36:02.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:01 smithi033 ceph-mon[31243]: pgmap v916: 129 pgs: 129 active+clean; 640 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 7.0 MiB/s wr, 573 op/s 2024-09-16T00:36:02.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 106432 17.93 MB/sec execute 188 sec latency 86.107 ms 2024-09-16T00:36:03.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 106727 17.89 MB/sec execute 189 sec latency 305.730 ms 2024-09-16T00:36:03.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:03 smithi033 ceph-mon[31243]: pgmap v917: 129 pgs: 129 active+clean; 682 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 12 MiB/s wr, 695 op/s 2024-09-16T00:36:03.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:03 smithi083 ceph-mon[35242]: pgmap v917: 129 pgs: 129 active+clean; 682 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 12 MiB/s wr, 695 op/s 2024-09-16T00:36:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:03 smithi060 ceph-mon[35185]: pgmap v917: 129 pgs: 129 active+clean; 682 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 12 MiB/s wr, 695 op/s 2024-09-16T00:36:04.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 107026 17.92 MB/sec execute 190 sec latency 398.577 ms 2024-09-16T00:36:05.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 107412 17.91 MB/sec execute 191 sec latency 113.774 ms 2024-09-16T00:36:05.224 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:36:05.519 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:05 smithi083 ceph-mon[35242]: pgmap v918: 129 pgs: 129 active+clean; 649 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 15 MiB/s wr, 743 op/s 2024-09-16T00:36:05.919 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:05 smithi033 ceph-mon[31243]: pgmap v918: 129 pgs: 129 active+clean; 649 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 15 MiB/s wr, 743 op/s 2024-09-16T00:36:05.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:05 smithi060 ceph-mon[35185]: pgmap v918: 129 pgs: 129 active+clean; 649 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 15 MiB/s wr, 743 op/s 2024-09-16T00:36:06.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.028+0000 7f56841b9640 1 -- 172.21.15.33:0/168280492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f567c101420 msgr2=0x7f567c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:06.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.028+0000 7f56841b9640 1 --2- 172.21.15.33:0/168280492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f567c101420 0x7f567c10d950 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f5674009cd0 tx=0x7f567402a8e0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.028+0000 7f56841b9640 1 -- 172.21.15.33:0/168280492 shutdown_connections 2024-09-16T00:36:06.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.028+0000 7f56841b9640 1 --2- 172.21.15.33:0/168280492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f567c101420 0x7f567c10d950 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.028+0000 7f56841b9640 1 --2- 172.21.15.33:0/168280492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f567c100a80 0x7f567c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.028+0000 7f56841b9640 1 --2- 172.21.15.33:0/168280492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f567c106a80 0x7f567c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.028+0000 7f56841b9640 1 -- 172.21.15.33:0/168280492 >> 172.21.15.33:0/168280492 conn(0x7f567c0fc7a0 msgr2=0x7f567c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:06.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.028+0000 7f56841b9640 1 -- 172.21.15.33:0/168280492 shutdown_connections 2024-09-16T00:36:06.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.028+0000 7f56841b9640 1 -- 172.21.15.33:0/168280492 wait complete. 2024-09-16T00:36:06.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.029+0000 7f56841b9640 1 Processor -- start 2024-09-16T00:36:06.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.029+0000 7f56841b9640 1 -- start start 2024-09-16T00:36:06.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.029+0000 7f56841b9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f567c100a80 0x7f567c1970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.029+0000 7f56841b9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f567c101420 0x7f567c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.029+0000 7f56841b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f567c106a80 0x7f567c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f56841b9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f567c110e80 con 0x7f567c101420 2024-09-16T00:36:06.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f56841b9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f567c110d00 con 0x7f567c100a80 2024-09-16T00:36:06.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f56841b9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f567c111000 con 0x7f567c106a80 2024-09-16T00:36:06.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5681754640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f567c101420 0x7f567c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5681f55640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f567c100a80 0x7f567c1970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5681754640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f567c101420 0x7f567c1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33314/0 (socket says 172.21.15.33:33314) 2024-09-16T00:36:06.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5681f55640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f567c100a80 0x7f567c1970a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40164/0 (socket says 172.21.15.33:40164) 2024-09-16T00:36:06.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5681754640 1 -- 172.21.15.33:0/1858836229 learned_addr learned my addr 172.21.15.33:0/1858836229 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:06.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5682756640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f567c106a80 0x7f567c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5681f55640 1 -- 172.21.15.33:0/1858836229 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f567c106a80 msgr2=0x7f567c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:06.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5681f55640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f567c106a80 0x7f567c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5681f55640 1 -- 172.21.15.33:0/1858836229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f567c101420 msgr2=0x7f567c1975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:06.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5681f55640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f567c101420 0x7f567c1975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5681f55640 1 -- 172.21.15.33:0/1858836229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567c1a6910 con 0x7f567c100a80 2024-09-16T00:36:06.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5681754640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f567c101420 0x7f567c1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:06.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f5681f55640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f567c100a80 0x7f567c1970a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5678005fd0 tx=0x7f567800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:06.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.030+0000 7f566effd640 1 -- 172.21.15.33:0/1858836229 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678009c30 con 0x7f567c100a80 2024-09-16T00:36:06.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.031+0000 7f56841b9640 1 -- 172.21.15.33:0/1858836229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f567c1a6c00 con 0x7f567c100a80 2024-09-16T00:36:06.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.031+0000 7f566effd640 1 -- 172.21.15.33:0/1858836229 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5678009dd0 con 0x7f567c100a80 2024-09-16T00:36:06.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.031+0000 7f566effd640 1 -- 172.21.15.33:0/1858836229 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678010460 con 0x7f567c100a80 2024-09-16T00:36:06.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.031+0000 7f56841b9640 1 -- 172.21.15.33:0/1858836229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f567c1a7140 con 0x7f567c100a80 2024-09-16T00:36:06.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.033+0000 7f566effd640 1 -- 172.21.15.33:0/1858836229 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5678003ef0 con 0x7f567c100a80 2024-09-16T00:36:06.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.033+0000 7f5686592740 1 Processor -- start 2024-09-16T00:36:06.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.033+0000 7f5686592740 1 -- start start 2024-09-16T00:36:06.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.033+0000 7f5686592740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37f4a0 0x55d24d37f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5686592740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d37fdc0 0x55d24d38a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5686592740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d38a680 0x55d24d38ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5686592740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d24d2e52e0 con 0x55d24d37fdc0 2024-09-16T00:36:06.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5686592740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d24d28cce0 con 0x55d24d37f4a0 2024-09-16T00:36:06.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5686592740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d24d2e12a0 con 0x55d24d38a680 2024-09-16T00:36:06.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5682756640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37f4a0 0x55d24d37f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5681f55640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d37fdc0 0x55d24d38a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5681754640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d38a680 0x55d24d38ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5682756640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37f4a0 0x55d24d37f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40172/0 (socket says 172.21.15.33:40172) 2024-09-16T00:36:06.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5682756640 1 -- 172.21.15.33:0/1460907641 learned_addr learned my addr 172.21.15.33:0/1460907641 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:06.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f566effd640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f56680a3240 0x7f56680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5681754640 1 -- 172.21.15.33:0/1460907641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37f4a0 msgr2=0x55d24d37f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:06.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5681754640 1 --2- 172.21.15.33:0/1460907641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37f4a0 0x55d24d37f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5681754640 1 -- 172.21.15.33:0/1460907641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d37fdc0 msgr2=0x55d24d38a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:06.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5681754640 1 --2- 172.21.15.33:0/1460907641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d37fdc0 0x55d24d38a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5681754640 1 -- 172.21.15.33:0/1460907641 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d24d271440 con 0x55d24d38a680 2024-09-16T00:36:06.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5682756640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f56680a3240 0x7f56680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f566effd640 1 -- 172.21.15.33:0/1858836229 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f56780cf7e0 con 0x7f567c100a80 2024-09-16T00:36:06.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.034+0000 7f5681754640 1 --2- 172.21.15.33:0/1460907641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d38a680 0x55d24d38ca70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f567c199200 tx=0x7f56700119a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7062898d68e0e6ed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:06.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.035+0000 7f5667fff640 1 -- 172.21.15.33:0/1460907641 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567001e070 con 0x55d24d38a680 2024-09-16T00:36:06.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.035+0000 7f5667fff640 1 -- 172.21.15.33:0/1460907641 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f567000faa0 con 0x55d24d38a680 2024-09-16T00:36:06.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.035+0000 7f5667fff640 1 -- 172.21.15.33:0/1460907641 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5670019bd0 con 0x55d24d38a680 2024-09-16T00:36:06.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.035+0000 7f5686592740 1 -- 172.21.15.33:0/1460907641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d38a680 msgr2=0x55d24d38ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:06.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.035+0000 7f5682756640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f56680a3240 0x7f56680a5700 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5674004ab0 tx=0x7f56740095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:06.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.035+0000 7f5686592740 1 --2- 172.21.15.33:0/1460907641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d38a680 0x55d24d38ca70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f567c199200 tx=0x7f56700119a0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.036+0000 7f5686592740 1 -- 172.21.15.33:0/1460907641 shutdown_connections 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.036+0000 7f5686592740 1 --2- 172.21.15.33:0/1460907641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d38a680 0x55d24d38ca70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.036+0000 7f5686592740 1 --2- 172.21.15.33:0/1460907641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d37fdc0 0x55d24d38a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.036+0000 7f5686592740 1 --2- 172.21.15.33:0/1460907641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37f4a0 0x55d24d37f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.036+0000 7f5686592740 1 -- 172.21.15.33:0/1460907641 >> 172.21.15.33:0/1460907641 conn(0x55d24d37ba30 msgr2=0x55d24d37be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.036+0000 7f5686592740 1 -- 172.21.15.33:0/1460907641 shutdown_connections 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.036+0000 7f5686592740 1 -- 172.21.15.33:0/1460907641 wait complete. 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.036+0000 7f5686592740 1 -- start start 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.036+0000 7f5686592740 1 Processor -- start 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5686592740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37f4a0 0x55d24d389ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5686592740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d37fdc0 0x55d24d3871a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5686592740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d38a680 0x55d24d3876e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5686592740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d24d38d250 con 0x55d24d37fdc0 2024-09-16T00:36:06.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5686592740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d24d2a0050 con 0x55d24d37f4a0 2024-09-16T00:36:06.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5686592740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d24d38d3d0 con 0x55d24d38a680 2024-09-16T00:36:06.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5686592740 10 client.0 _ll_register_callbacks cb 0x55d24d2e0160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:06.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5681754640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d37fdc0 0x55d24d3871a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5682756640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d38a680 0x55d24d3876e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5681754640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d37fdc0 0x55d24d3871a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33324/0 (socket says 172.21.15.33:33324) 2024-09-16T00:36:06.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5681754640 1 -- 172.21.15.33:0/946849382 learned_addr learned my addr 172.21.15.33:0/946849382 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:06.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5681f55640 1 --2- 172.21.15.33:0/946849382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37f4a0 0x55d24d389ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5681754640 1 -- 172.21.15.33:0/946849382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d38a680 msgr2=0x55d24d3876e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:06.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5681754640 1 --2- 172.21.15.33:0/946849382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d38a680 0x55d24d3876e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5681754640 1 -- 172.21.15.33:0/946849382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37f4a0 msgr2=0x55d24d389ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:06.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5681754640 1 --2- 172.21.15.33:0/946849382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37f4a0 0x55d24d389ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5681754640 1 -- 172.21.15.33:0/946849382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d24d0563c0 con 0x55d24d37fdc0 2024-09-16T00:36:06.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f5681754640 1 --2- 172.21.15.33:0/946849382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d37fdc0 0x55d24d3871a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f567001c040 tx=0x7f567000f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=481bab4b00a8353e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:06.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.037+0000 7f56657fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:06.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f56657fa640 1 -- 172.21.15.33:0/946849382 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567001e040 con 0x55d24d37fdc0 2024-09-16T00:36:06.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f5686592740 1 -- 172.21.15.33:0/946849382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f567000faa0 con 0x55d24d37fdc0 2024-09-16T00:36:06.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f56657fa640 1 -- 172.21.15.33:0/946849382 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5670019930 con 0x55d24d37fdc0 2024-09-16T00:36:06.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f56657fa640 1 -- 172.21.15.33:0/946849382 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5670032050 con 0x55d24d37fdc0 2024-09-16T00:36:06.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f5686592740 10 client.18951 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:06.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f5686592740 1 -- 172.21.15.33:0/946849382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d24d398b70 con 0x55d24d37fdc0 2024-09-16T00:36:06.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f56657fa640 1 -- 172.21.15.33:0/946849382 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f567000faa0 con 0x55d24d37fdc0 2024-09-16T00:36:06.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f56657fa640 1 -- 172.21.15.33:0/946849382 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f5670019cb0 con 0x55d24d37fdc0 2024-09-16T00:36:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f5686592740 10 client.18951 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f5686592740 10 client.18951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:36:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f5686592740 20 client.18951 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f5686592740 1 --2- 172.21.15.33:0/946849382 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d24d399f80 0x55d24d39c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f5686592740 4 client.18951 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:36:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f5686592740 1 -- 172.21.15.33:0/946849382 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d24d2f2850 con 0x55d24d399f80 2024-09-16T00:36:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.038+0000 7f5681f55640 1 --2- 172.21.15.33:0/946849382 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d24d399f80 0x55d24d39c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.039+0000 7f56657fa640 10 client.18951 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.039+0000 7f5681f55640 1 --2- 172.21.15.33:0/946849382 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d24d399f80 0x55d24d39c370 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e8d3622d787592c server_cookie=f8c7931b89fb69c5 in_seq=0 out_seq=0 2024-09-16T00:36:06.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.039+0000 7f56657fa640 1 -- 172.21.15.33:0/946849382 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d24d2f2850 con 0x55d24d399f80 2024-09-16T00:36:06.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.039+0000 7f56657fa640 10 client.18951 handle_command_reply: tid=0 2024-09-16T00:36:06.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.039+0000 7f5686592740 1 client.18951 shutdown 2024-09-16T00:36:06.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.040+0000 7f5686592740 1 -- 172.21.15.33:0/946849382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d37fdc0 msgr2=0x55d24d3871a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:06.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.040+0000 7f5686592740 1 --2- 172.21.15.33:0/946849382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d37fdc0 0x55d24d3871a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f567001c040 tx=0x7f567000f870 comp rx=0 tx=0).stop 2024-09-16T00:36:06.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.040+0000 7f5686592740 1 -- 172.21.15.33:0/946849382 shutdown_connections 2024-09-16T00:36:06.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.040+0000 7f5686592740 1 --2- 172.21.15.33:0/946849382 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d24d399f80 0x55d24d39c370 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.040+0000 7f5686592740 1 --2- 172.21.15.33:0/946849382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d38a680 0x55d24d3876e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.040+0000 7f5686592740 1 --2- 172.21.15.33:0/946849382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d37fdc0 0x55d24d3871a0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.040+0000 7f5686592740 1 --2- 172.21.15.33:0/946849382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37f4a0 0x55d24d389ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.040+0000 7f5686592740 1 -- 172.21.15.33:0/946849382 >> 172.21.15.33:0/946849382 conn(0x55d24d37ba30 msgr2=0x55d24d381760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:06.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.040+0000 7f56657fa640 0 client.18951 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:06.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.040+0000 7f5686592740 1 -- 172.21.15.33:0/946849382 shutdown_connections 2024-09-16T00:36:06.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.040+0000 7f5686592740 1 -- 172.21.15.33:0/946849382 wait complete. 2024-09-16T00:36:06.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.040+0000 7f5686592740 20 client.18951 trim_cache size 0 max 16384 2024-09-16T00:36:06.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.064+0000 7f5686592740 1 Processor -- start 2024-09-16T00:36:06.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.064+0000 7f5686592740 1 -- start start 2024-09-16T00:36:06.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.064+0000 7f5686592740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d37f4a0 0x55d24d382900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.064+0000 7f5686592740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 0x55d24d389110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.064+0000 7f5686592740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d38a680 0x55d24d3a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.064+0000 7f5686592740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f567c1b3c50 con 0x55d24d38a680 2024-09-16T00:36:06.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.064+0000 7f5686592740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f567c1b1520 con 0x55d24d37fdc0 2024-09-16T00:36:06.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.064+0000 7f5686592740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f567c1af370 con 0x55d24d37f4a0 2024-09-16T00:36:06.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.065+0000 7f5681f55640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d37f4a0 0x55d24d382900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.065+0000 7f5681f55640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d37f4a0 0x55d24d382900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:45624/0 (socket says 172.21.15.33:45624) 2024-09-16T00:36:06.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.065+0000 7f5682756640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d38a680 0x55d24d3a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.065+0000 7f5681f55640 1 -- 172.21.15.33:0/1391525431 learned_addr learned my addr 172.21.15.33:0/1391525431 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:06.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.065+0000 7f5681754640 1 --2- 172.21.15.33:0/1391525431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 0x55d24d389110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.065+0000 7f5681754640 1 -- 172.21.15.33:0/1391525431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d37f4a0 msgr2=0x55d24d382900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:06.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.065+0000 7f5681754640 1 --2- 172.21.15.33:0/1391525431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d37f4a0 0x55d24d382900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.065+0000 7f5681754640 1 -- 172.21.15.33:0/1391525431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d38a680 msgr2=0x55d24d3a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:06.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.065+0000 7f5681754640 1 --2- 172.21.15.33:0/1391525431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d38a680 0x55d24d3a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.065+0000 7f5681754640 1 -- 172.21.15.33:0/1391525431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d24d39c900 con 0x55d24d37fdc0 2024-09-16T00:36:06.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.065+0000 7f5681f55640 1 --2- 172.21.15.33:0/1391525431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d37f4a0 0x55d24d382900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:06.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.065+0000 7f5681754640 1 --2- 172.21.15.33:0/1391525431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 0x55d24d389110 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5670004790 tx=0x7f5670005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2515502ba0301869 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:06.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.066+0000 7f56677fe640 1 -- 172.21.15.33:0/1391525431 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567003b070 con 0x55d24d37fdc0 2024-09-16T00:36:06.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.066+0000 7f56677fe640 1 -- 172.21.15.33:0/1391525431 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5670029070 con 0x55d24d37fdc0 2024-09-16T00:36:06.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.066+0000 7f56677fe640 1 -- 172.21.15.33:0/1391525431 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567001e070 con 0x55d24d37fdc0 2024-09-16T00:36:06.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.066+0000 7f5686592740 1 -- 172.21.15.33:0/1391525431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 msgr2=0x55d24d389110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:06.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.066+0000 7f5686592740 1 --2- 172.21.15.33:0/1391525431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 0x55d24d389110 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5670004790 tx=0x7f5670005d30 comp rx=0 tx=0).stop 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.066+0000 7f5686592740 1 -- 172.21.15.33:0/1391525431 shutdown_connections 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.066+0000 7f5686592740 1 --2- 172.21.15.33:0/1391525431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d38a680 0x55d24d3a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.066+0000 7f5686592740 1 --2- 172.21.15.33:0/1391525431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 0x55d24d389110 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.066+0000 7f5686592740 1 --2- 172.21.15.33:0/1391525431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d37f4a0 0x55d24d382900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.066+0000 7f5686592740 1 -- 172.21.15.33:0/1391525431 >> 172.21.15.33:0/1391525431 conn(0x55d24d37ba30 msgr2=0x55d24d388a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.066+0000 7f5686592740 1 -- 172.21.15.33:0/1391525431 shutdown_connections 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.066+0000 7f5686592740 1 -- 172.21.15.33:0/1391525431 wait complete. 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.067+0000 7f5686592740 1 -- start start 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.067+0000 7f5686592740 1 Processor -- start 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.067+0000 7f5686592740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d37f4a0 0x55d24d392650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.067+0000 7f5686592740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 0x55d24d392b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.067+0000 7f5686592740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d38a680 0x55d24d38f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.067+0000 7f5686592740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f567c1af9e0 con 0x55d24d38a680 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.067+0000 7f5686592740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f567c1b0e50 con 0x55d24d37fdc0 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.067+0000 7f5686592740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f567c1b00b0 con 0x55d24d37f4a0 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.067+0000 7f5686592740 10 client.0 _ll_register_callbacks cb 0x7f567001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.067+0000 7f5681754640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 0x55d24d392b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.067+0000 7f5682756640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d38a680 0x55d24d38f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f5681f55640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d37f4a0 0x55d24d392650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f5681754640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 0x55d24d392b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40202/0 (socket says 172.21.15.33:40202) 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f5681754640 1 -- 172.21.15.33:0/2313450329 learned_addr learned my addr 172.21.15.33:0/2313450329 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f5681754640 1 -- 172.21.15.33:0/2313450329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d37f4a0 msgr2=0x55d24d392650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f5681754640 1 --2- 172.21.15.33:0/2313450329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d37f4a0 0x55d24d392650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f5681754640 1 -- 172.21.15.33:0/2313450329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d38a680 msgr2=0x55d24d38f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f5681754640 1 --2- 172.21.15.33:0/2313450329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d38a680 0x55d24d38f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f5681754640 1 -- 172.21.15.33:0/2313450329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d24d37b860 con 0x55d24d37fdc0 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f5681754640 1 --2- 172.21.15.33:0/2313450329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 0x55d24d392b90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5670011460 tx=0x7f5670019ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66162dacdbca2a52 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f56657fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f56657fa640 1 -- 172.21.15.33:0/2313450329 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567001e070 con 0x55d24d37fdc0 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f56657fa640 1 -- 172.21.15.33:0/2313450329 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5670019610 con 0x55d24d37fdc0 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f5686592740 1 -- 172.21.15.33:0/2313450329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5670029070 con 0x55d24d37fdc0 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f56657fa640 1 -- 172.21.15.33:0/2313450329 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f567003b070 con 0x55d24d37fdc0 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f5686592740 10 client.28226 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.068+0000 7f5686592740 1 -- 172.21.15.33:0/2313450329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d24d404f50 con 0x55d24d37fdc0 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.069+0000 7f56657fa640 1 -- 172.21.15.33:0/2313450329 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5670029070 con 0x55d24d37fdc0 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.069+0000 7f56657fa640 1 -- 172.21.15.33:0/2313450329 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f5670024cd0 con 0x55d24d37fdc0 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.069+0000 7f5686592740 10 client.28226 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.069+0000 7f5686592740 10 client.28226 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.069+0000 7f5686592740 20 client.28226 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:06.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.069+0000 7f5686592740 1 --2- 172.21.15.33:0/2313450329 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d24d405670 0x55d24d407a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.069+0000 7f5686592740 4 client.28226 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:36:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.069+0000 7f5686592740 1 -- 172.21.15.33:0/2313450329 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d24d2ecd60 con 0x55d24d405670 2024-09-16T00:36:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.069+0000 7f5681f55640 1 --2- 172.21.15.33:0/2313450329 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d24d405670 0x55d24d407a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.069+0000 7f56657fa640 10 client.28226 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.069+0000 7f5681f55640 1 --2- 172.21.15.33:0/2313450329 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d24d405670 0x55d24d407a90 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a075dd7c326b2ba0 server_cookie=40978e9739c7c580 in_seq=0 out_seq=0 2024-09-16T00:36:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f56657fa640 1 -- 172.21.15.33:0/2313450329 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d24d2ecd60 con 0x55d24d405670 2024-09-16T00:36:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f56657fa640 10 client.28226 handle_command_reply: tid=0 2024-09-16T00:36:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f5686592740 1 client.28226 shutdown 2024-09-16T00:36:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f5686592740 1 -- 172.21.15.33:0/2313450329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 msgr2=0x55d24d392b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f5686592740 1 --2- 172.21.15.33:0/2313450329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 0x55d24d392b90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5670011460 tx=0x7f5670019ec0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f5686592740 1 -- 172.21.15.33:0/2313450329 shutdown_connections 2024-09-16T00:36:06.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f5686592740 1 --2- 172.21.15.33:0/2313450329 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d24d405670 0x55d24d407a90 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f5686592740 1 --2- 172.21.15.33:0/2313450329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d24d38a680 0x55d24d38f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f5686592740 1 --2- 172.21.15.33:0/2313450329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d24d37fdc0 0x55d24d392b90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f5686592740 1 --2- 172.21.15.33:0/2313450329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24d37f4a0 0x55d24d392650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f5686592740 1 -- 172.21.15.33:0/2313450329 >> 172.21.15.33:0/2313450329 conn(0x55d24d37ba30 msgr2=0x55d24d389220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:06.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f56657fa640 0 client.28226 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:06.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f5686592740 1 -- 172.21.15.33:0/2313450329 shutdown_connections 2024-09-16T00:36:06.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f5686592740 1 -- 172.21.15.33:0/2313450329 wait complete. 2024-09-16T00:36:06.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.070+0000 7f5686592740 20 client.28226 trim_cache size 0 max 16384 2024-09-16T00:36:06.077 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:36:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:36:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:36:06.078 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:36:06.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.071+0000 7f56841b9640 1 -- 172.21.15.33:0/1858836229 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f56680a3240 msgr2=0x7f56680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:06.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.071+0000 7f56841b9640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f56680a3240 0x7f56680a5700 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5674004ab0 tx=0x7f56740095c0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.071+0000 7f56841b9640 1 -- 172.21.15.33:0/1858836229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f567c100a80 msgr2=0x7f567c1970a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:06.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.071+0000 7f56841b9640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f567c100a80 0x7f567c1970a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5678005fd0 tx=0x7f567800dde0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.071+0000 7f56841b9640 1 -- 172.21.15.33:0/1858836229 shutdown_connections 2024-09-16T00:36:06.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.071+0000 7f56841b9640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f56680a3240 0x7f56680a5700 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.071+0000 7f56841b9640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f567c106a80 0x7f567c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.071+0000 7f56841b9640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f567c101420 0x7f567c1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.071+0000 7f56841b9640 1 --2- 172.21.15.33:0/1858836229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f567c100a80 0x7f567c1970a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:06.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.071+0000 7f56841b9640 1 -- 172.21.15.33:0/1858836229 >> 172.21.15.33:0/1858836229 conn(0x7f567c0fc7a0 msgr2=0x7f567c0fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:06.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.071+0000 7f56841b9640 1 -- 172.21.15.33:0/1858836229 shutdown_connections 2024-09-16T00:36:06.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:06.071+0000 7f56841b9640 1 -- 172.21.15.33:0/1858836229 wait complete. 2024-09-16T00:36:06.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 108046 18.01 MB/sec execute 192 sec latency 56.757 ms 2024-09-16T00:36:06.940 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:36:06.941 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:36:07.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 108627 18.03 MB/sec execute 193 sec latency 64.684 ms 2024-09-16T00:36:07.257 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:07.731 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:07 smithi033 ceph-mon[31243]: pgmap v919: 129 pgs: 129 active+clean; 589 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 17 MiB/s wr, 828 op/s 2024-09-16T00:36:07.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.836+0000 7ff5bc627640 1 -- 172.21.15.33:0/21292539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5b4101de0 msgr2=0x7ff5b41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:07.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.836+0000 7ff5bc627640 1 --2- 172.21.15.33:0/21292539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5b4101de0 0x7ff5b41021c0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009900 tx=0x7ff5b002a410 comp rx=0 tx=0).stop 2024-09-16T00:36:07.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.836+0000 7ff5bc627640 1 -- 172.21.15.33:0/21292539 shutdown_connections 2024-09-16T00:36:07.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.836+0000 7ff5bc627640 1 --2- 172.21.15.33:0/21292539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5b4106f10 0x7ff5b410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:07.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.836+0000 7ff5bc627640 1 --2- 172.21.15.33:0/21292539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b4102790 0x7ff5b41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:07.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.836+0000 7ff5bc627640 1 --2- 172.21.15.33:0/21292539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5b4101de0 0x7ff5b41021c0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:07.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.836+0000 7ff5bc627640 1 -- 172.21.15.33:0/21292539 >> 172.21.15.33:0/21292539 conn(0x7ff5b40fd8c0 msgr2=0x7ff5b40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:07.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.836+0000 7ff5bc627640 1 -- 172.21.15.33:0/21292539 shutdown_connections 2024-09-16T00:36:07.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.836+0000 7ff5bc627640 1 -- 172.21.15.33:0/21292539 wait complete. 2024-09-16T00:36:07.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.837+0000 7ff5bc627640 1 Processor -- start 2024-09-16T00:36:07.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.837+0000 7ff5bc627640 1 -- start start 2024-09-16T00:36:07.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.837+0000 7ff5bc627640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5b4101de0 0x7ff5b4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:07.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.837+0000 7ff5bc627640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5b4102790 0x7ff5b4198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:07.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.837+0000 7ff5bc627640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b4106f10 0x7ff5b419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:07.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.837+0000 7ff5bc627640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b4111fc0 con 0x7ff5b4102790 2024-09-16T00:36:07.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.837+0000 7ff5bc627640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b4111e40 con 0x7ff5b4101de0 2024-09-16T00:36:07.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.837+0000 7ff5bc627640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b4112140 con 0x7ff5b4106f10 2024-09-16T00:36:07.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.838+0000 7ff5b9bc2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5b4102790 0x7ff5b4198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:07.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.838+0000 7ff5ba3c3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5b4101de0 0x7ff5b4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:07.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.838+0000 7ff5b9bc2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5b4102790 0x7ff5b4198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33368/0 (socket says 172.21.15.33:33368) 2024-09-16T00:36:07.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.838+0000 7ff5b9bc2640 1 -- 172.21.15.33:0/2176300099 learned_addr learned my addr 172.21.15.33:0/2176300099 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:07.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.838+0000 7ff5babc4640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b4106f10 0x7ff5b419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:07.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.838+0000 7ff5b9bc2640 1 -- 172.21.15.33:0/2176300099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b4106f10 msgr2=0x7ff5b419ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:07.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.838+0000 7ff5b9bc2640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b4106f10 0x7ff5b419ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:07.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.838+0000 7ff5b9bc2640 1 -- 172.21.15.33:0/2176300099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5b4101de0 msgr2=0x7ff5b4198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:07.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.838+0000 7ff5b9bc2640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5b4101de0 0x7ff5b4198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:07.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.838+0000 7ff5b9bc2640 1 -- 172.21.15.33:0/2176300099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b419d3d0 con 0x7ff5b4102790 2024-09-16T00:36:07.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.838+0000 7ff5ba3c3640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5b4101de0 0x7ff5b4198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:07.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.838+0000 7ff5b9bc2640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5b4102790 0x7ff5b4198960 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400e9f0 tx=0x7ff5a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:07.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.839+0000 7ff5ab7fe640 1 -- 172.21.15.33:0/2176300099 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a400cd80 con 0x7ff5b4102790 2024-09-16T00:36:07.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.839+0000 7ff5ab7fe640 1 -- 172.21.15.33:0/2176300099 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5a4010910 con 0x7ff5b4102790 2024-09-16T00:36:07.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.839+0000 7ff5ab7fe640 1 -- 172.21.15.33:0/2176300099 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4018990 con 0x7ff5b4102790 2024-09-16T00:36:07.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.839+0000 7ff5bc627640 1 -- 172.21.15.33:0/2176300099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b419d6c0 con 0x7ff5b4102790 2024-09-16T00:36:07.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.839+0000 7ff5bc627640 1 -- 172.21.15.33:0/2176300099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b419db20 con 0x7ff5b4102790 2024-09-16T00:36:07.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.840+0000 7ff5bc627640 1 -- 172.21.15.33:0/2176300099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5b41021c0 con 0x7ff5b4102790 2024-09-16T00:36:07.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.843+0000 7ff5ab7fe640 1 -- 172.21.15.33:0/2176300099 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff5a4010ac0 con 0x7ff5b4102790 2024-09-16T00:36:07.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.844+0000 7ff5ab7fe640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5a00a32b0 0x7ff5a00a5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:07.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.844+0000 7ff5ab7fe640 1 -- 172.21.15.33:0/2176300099 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff5a40d0000 con 0x7ff5b4102790 2024-09-16T00:36:07.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.844+0000 7ff5ba3c3640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5a00a32b0 0x7ff5a00a5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:07.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.845+0000 7ff5ba3c3640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5a00a32b0 0x7ff5a00a5770 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0000c00 tx=0x7ff5b0031510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:07.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:07.851+0000 7ff5ab7fe640 1 -- 172.21.15.33:0/2176300099 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff5b419db20 con 0x7ff5b4102790 2024-09-16T00:36:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:07 smithi083 ceph-mon[35242]: pgmap v919: 129 pgs: 129 active+clean; 589 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 17 MiB/s wr, 828 op/s 2024-09-16T00:36:07.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:07 smithi060 ceph-mon[35185]: pgmap v919: 129 pgs: 129 active+clean; 589 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 17 MiB/s wr, 828 op/s 2024-09-16T00:36:08.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 109191 18.17 MB/sec execute 194 sec latency 118.632 ms 2024-09-16T00:36:08.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.096+0000 7ff5bc627640 1 -- 172.21.15.33:0/2176300099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5b410a0f0 con 0x7ff5b4102790 2024-09-16T00:36:08.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.097+0000 7ff5ab7fe640 1 -- 172.21.15.33:0/2176300099 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7ff5a409c250 con 0x7ff5b4102790 2024-09-16T00:36:08.099 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:36:08.099 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:36:08.099 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:36:08.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.100+0000 7ff5bc627640 1 -- 172.21.15.33:0/2176300099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5a00a32b0 msgr2=0x7ff5a00a5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:08.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.100+0000 7ff5bc627640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5a00a32b0 0x7ff5a00a5770 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0000c00 tx=0x7ff5b0031510 comp rx=0 tx=0).stop 2024-09-16T00:36:08.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.100+0000 7ff5bc627640 1 -- 172.21.15.33:0/2176300099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5b4102790 msgr2=0x7ff5b4198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:08.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.100+0000 7ff5bc627640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5b4102790 0x7ff5b4198960 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400e9f0 tx=0x7ff5a400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:36:08.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.100+0000 7ff5bc627640 1 -- 172.21.15.33:0/2176300099 shutdown_connections 2024-09-16T00:36:08.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.100+0000 7ff5bc627640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5a00a32b0 0x7ff5a00a5770 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:08.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.100+0000 7ff5bc627640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b4106f10 0x7ff5b419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:08.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.100+0000 7ff5bc627640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5b4102790 0x7ff5b4198960 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:08.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.100+0000 7ff5bc627640 1 --2- 172.21.15.33:0/2176300099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5b4101de0 0x7ff5b4198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:08.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.100+0000 7ff5bc627640 1 -- 172.21.15.33:0/2176300099 >> 172.21.15.33:0/2176300099 conn(0x7ff5b40fd8c0 msgr2=0x7ff5b40fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:08.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.100+0000 7ff5bc627640 1 -- 172.21.15.33:0/2176300099 shutdown_connections 2024-09-16T00:36:08.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:08.100+0000 7ff5bc627640 1 -- 172.21.15.33:0/2176300099 wait complete. 2024-09-16T00:36:08.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:08 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2176300099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:36:08.876 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:36:08.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:08 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2176300099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:36:08.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:08 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2176300099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:36:09.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 109756 18.22 MB/sec execute 195 sec latency 67.683 ms 2024-09-16T00:36:09.237 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:09.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.770+0000 7fa396707640 1 -- 172.21.15.33:0/2175227437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa390102ef0 msgr2=0x7fa39010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:09.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.770+0000 7fa396707640 1 --2- 172.21.15.33:0/2175227437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa390102ef0 0x7fa39010da40 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fa388009ca0 tx=0x7fa38802a950 comp rx=0 tx=0).stop 2024-09-16T00:36:09.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.770+0000 7fa396707640 1 -- 172.21.15.33:0/2175227437 shutdown_connections 2024-09-16T00:36:09.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.770+0000 7fa396707640 1 --2- 172.21.15.33:0/2175227437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa390102ef0 0x7fa39010da40 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.770+0000 7fa396707640 1 --2- 172.21.15.33:0/2175227437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa390102570 0x7fa3901029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.770+0000 7fa396707640 1 --2- 172.21.15.33:0/2175227437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa390100e20 0x7fa390101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.770+0000 7fa396707640 1 -- 172.21.15.33:0/2175227437 >> 172.21.15.33:0/2175227437 conn(0x7fa3900fc7d0 msgr2=0x7fa3900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:09.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.770+0000 7fa396707640 1 -- 172.21.15.33:0/2175227437 shutdown_connections 2024-09-16T00:36:09.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.770+0000 7fa396707640 1 -- 172.21.15.33:0/2175227437 wait complete. 2024-09-16T00:36:09.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.771+0000 7fa396707640 1 Processor -- start 2024-09-16T00:36:09.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.771+0000 7fa396707640 1 -- start start 2024-09-16T00:36:09.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.771+0000 7fa396707640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa390100e20 0x7fa390197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.771+0000 7fa396707640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa390102570 0x7fa390197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.771+0000 7fa396707640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa390102ef0 0x7fa39019bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.771+0000 7fa396707640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa390110f80 con 0x7fa390102ef0 2024-09-16T00:36:09.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.771+0000 7fa396707640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa390110e00 con 0x7fa390100e20 2024-09-16T00:36:09.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.771+0000 7fa396707640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa390111100 con 0x7fa390102570 2024-09-16T00:36:09.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa394ca4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa390102ef0 0x7fa39019bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa38ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa390100e20 0x7fa390197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa38f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa390102570 0x7fa390197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa394ca4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa390102ef0 0x7fa39019bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33386/0 (socket says 172.21.15.33:33386) 2024-09-16T00:36:09.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa394ca4640 1 -- 172.21.15.33:0/4017191784 learned_addr learned my addr 172.21.15.33:0/4017191784 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:09.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa38f7fe640 1 -- 172.21.15.33:0/4017191784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa390100e20 msgr2=0x7fa390197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:09.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/4017191784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa390100e20 0x7fa390197440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa38f7fe640 1 -- 172.21.15.33:0/4017191784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa390102ef0 msgr2=0x7fa39019bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:09.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/4017191784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa390102ef0 0x7fa39019bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa38f7fe640 1 -- 172.21.15.33:0/4017191784 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa39019c3f0 con 0x7fa390102570 2024-09-16T00:36:09.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/4017191784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa390102570 0x7fa390197980 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa38400e9f0 tx=0x7fa38400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:09.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa38d7fa640 1 -- 172.21.15.33:0/4017191784 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38400ccc0 con 0x7fa390102570 2024-09-16T00:36:09.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa38d7fa640 1 -- 172.21.15.33:0/4017191784 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa384010910 con 0x7fa390102570 2024-09-16T00:36:09.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa396707640 1 -- 172.21.15.33:0/4017191784 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa39019c6e0 con 0x7fa390102570 2024-09-16T00:36:09.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa38d7fa640 1 -- 172.21.15.33:0/4017191784 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa384018960 con 0x7fa390102570 2024-09-16T00:36:09.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.772+0000 7fa396707640 1 -- 172.21.15.33:0/4017191784 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3901a3fc0 con 0x7fa390102570 2024-09-16T00:36:09.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa38d7fa640 1 -- 172.21.15.33:0/4017191784 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa384010ac0 con 0x7fa390102570 2024-09-16T00:36:09.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa398ae0740 1 Processor -- start 2024-09-16T00:36:09.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa398ae0740 1 -- start start 2024-09-16T00:36:09.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa398ae0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acfac30 0x56299acf92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa398ae0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acf97e0 0x56299acf9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa398ae0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299ad06970 0x56299ad08d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa398ae0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56299ac5d2f0 con 0x56299ad06970 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa398ae0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56299ac04cf0 con 0x56299acf97e0 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa398ae0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56299ac592b0 con 0x56299acfac30 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa394ca4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acfac30 0x56299acf92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa38ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acf97e0 0x56299acf9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa394ca4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acfac30 0x56299acf92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:45670/0 (socket says 172.21.15.33:45670) 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa38ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acf97e0 0x56299acf9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40254/0 (socket says 172.21.15.33:40254) 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.775+0000 7fa394ca4640 1 -- 172.21.15.33:0/2216138348 learned_addr learned my addr 172.21.15.33:0/2216138348 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.776+0000 7fa38d7fa640 1 --2- 172.21.15.33:0/4017191784 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3780a3240 0x7fa3780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.776+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/2216138348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299ad06970 0x56299ad08d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.776+0000 7fa38d7fa640 1 -- 172.21.15.33:0/4017191784 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa3840d0110 con 0x7fa390102570 2024-09-16T00:36:09.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.776+0000 7fa394ca4640 1 --2- 172.21.15.33:0/4017191784 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3780a3240 0x7fa3780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.776+0000 7fa38f7fe640 1 -- 172.21.15.33:0/2216138348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acfac30 msgr2=0x56299acf92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:09.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.776+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/2216138348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acfac30 0x56299acf92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.776+0000 7fa38f7fe640 1 -- 172.21.15.33:0/2216138348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acf97e0 msgr2=0x56299acf9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:09.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.776+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/2216138348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acf97e0 0x56299acf9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.776+0000 7fa38f7fe640 1 -- 172.21.15.33:0/2216138348 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56299abe93a0 con 0x56299ad06970 2024-09-16T00:36:09.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.777+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/2216138348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299ad06970 0x56299ad08d60 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fa38400e9c0 tx=0x7fa38406f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446930939a44d1e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:09.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.777+0000 7fa394ca4640 1 --2- 172.21.15.33:0/4017191784 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3780a3240 0x7fa3780a5700 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa388009430 tx=0x7fa38802a950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:09.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.777+0000 7fa3767fc640 1 -- 172.21.15.33:0/2216138348 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38409a1e0 con 0x56299ad06970 2024-09-16T00:36:09.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.777+0000 7fa3767fc640 1 -- 172.21.15.33:0/2216138348 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa38409a6a0 con 0x56299ad06970 2024-09-16T00:36:09.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.777+0000 7fa3767fc640 1 -- 172.21.15.33:0/2216138348 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa384099950 con 0x56299ad06970 2024-09-16T00:36:09.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.778+0000 7fa398ae0740 1 -- 172.21.15.33:0/2216138348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299ad06970 msgr2=0x56299ad08d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:09.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.778+0000 7fa398ae0740 1 --2- 172.21.15.33:0/2216138348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299ad06970 0x56299ad08d60 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fa38400e9c0 tx=0x7fa38406f180 comp rx=0 tx=0).stop 2024-09-16T00:36:09.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.778+0000 7fa398ae0740 1 -- 172.21.15.33:0/2216138348 shutdown_connections 2024-09-16T00:36:09.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.778+0000 7fa398ae0740 1 --2- 172.21.15.33:0/2216138348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299ad06970 0x56299ad08d60 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.778+0000 7fa398ae0740 1 --2- 172.21.15.33:0/2216138348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acf97e0 0x56299acf9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.778+0000 7fa398ae0740 1 --2- 172.21.15.33:0/2216138348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acfac30 0x56299acf92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.778+0000 7fa398ae0740 1 -- 172.21.15.33:0/2216138348 >> 172.21.15.33:0/2216138348 conn(0x56299acc0310 msgr2=0x56299acf5e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:09.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.778+0000 7fa398ae0740 1 -- 172.21.15.33:0/2216138348 shutdown_connections 2024-09-16T00:36:09.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.778+0000 7fa398ae0740 1 -- 172.21.15.33:0/2216138348 wait complete. 2024-09-16T00:36:09.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.779+0000 7fa398ae0740 1 -- start start 2024-09-16T00:36:09.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.779+0000 7fa398ae0740 1 Processor -- start 2024-09-16T00:36:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa398ae0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 0x56299ad06300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa398ae0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acfac30 0x56299ad035b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa398ae0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299ad06970 0x56299ad03b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa398ae0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56299ad094e0 con 0x56299acf97e0 2024-09-16T00:36:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa398ae0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56299ac18060 con 0x56299ad06970 2024-09-16T00:36:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa398ae0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56299ad09660 con 0x56299acfac30 2024-09-16T00:36:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa398ae0740 10 client.0 _ll_register_callbacks cb 0x56299a96b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa38ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 0x56299ad06300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa38ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 0x56299ad06300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33400/0 (socket says 172.21.15.33:33400) 2024-09-16T00:36:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa38ffff640 1 -- 172.21.15.33:0/1961201147 learned_addr learned my addr 172.21.15.33:0/1961201147 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:09.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acfac30 0x56299ad035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa394ca4640 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299ad06970 0x56299ad03b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa38ffff640 1 -- 172.21.15.33:0/1961201147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acfac30 msgr2=0x56299ad035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa38ffff640 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acfac30 0x56299ad035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa38ffff640 1 -- 172.21.15.33:0/1961201147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299ad06970 msgr2=0x56299ad03b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa38ffff640 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299ad06970 0x56299ad03b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa38ffff640 1 -- 172.21.15.33:0/1961201147 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56299a9ce3c0 con 0x56299acf97e0 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa38ffff640 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 0x56299ad06300 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fa3800126e0 tx=0x7fa380012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb8e7ad96042429 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.780+0000 7fa36f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa36f7fe640 1 -- 172.21.15.33:0/1961201147 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa380012190 con 0x56299acf97e0 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa398ae0740 1 -- 172.21.15.33:0/1961201147 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa38409a6a0 con 0x56299acf97e0 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa36f7fe640 1 -- 172.21.15.33:0/1961201147 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa38000f920 con 0x56299acf97e0 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa36f7fe640 1 -- 172.21.15.33:0/1961201147 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa38000fb20 con 0x56299acf97e0 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa398ae0740 10 client.18975 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:09.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa398ae0740 1 -- 172.21.15.33:0/1961201147 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56299ad14fc0 con 0x56299acf97e0 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa36f7fe640 1 -- 172.21.15.33:0/1961201147 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa38409a6a0 con 0x56299acf97e0 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa36f7fe640 1 -- 172.21.15.33:0/1961201147 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fa38001c420 con 0x56299acf97e0 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa398ae0740 10 client.18975 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa398ae0740 10 client.18975 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa398ae0740 20 client.18975 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa398ae0740 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56299ad163d0 0x56299ad18830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa398ae0740 4 client.18975 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.781+0000 7fa398ae0740 1 -- 172.21.15.33:0/1961201147 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56299ac6a860 con 0x56299ad163d0 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.782+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56299ad163d0 0x56299ad18830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.782+0000 7fa36f7fe640 10 client.18975 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.782+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56299ad163d0 0x56299ad18830 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=78dc4041a48e1d4c server_cookie=8c7cae2ae92cb514 in_seq=0 out_seq=0 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa36f7fe640 1 -- 172.21.15.33:0/1961201147 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fa38406de80 con 0x56299ad163d0 2024-09-16T00:36:09.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa36f7fe640 10 client.18975 handle_command_reply: tid=0 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa398ae0740 1 client.18975 shutdown 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa398ae0740 1 -- 172.21.15.33:0/1961201147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 msgr2=0x56299ad06300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa398ae0740 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 0x56299ad06300 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fa3800126e0 tx=0x7fa380012ba0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa398ae0740 1 -- 172.21.15.33:0/1961201147 shutdown_connections 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa398ae0740 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56299ad163d0 0x56299ad18830 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa36f7fe640 0 client.18975 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa398ae0740 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299ad06970 0x56299ad03b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa398ae0740 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acfac30 0x56299ad035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa398ae0740 1 --2- 172.21.15.33:0/1961201147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 0x56299ad06300 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa398ae0740 1 -- 172.21.15.33:0/1961201147 >> 172.21.15.33:0/1961201147 conn(0x56299acc0310 msgr2=0x56299acfbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa398ae0740 1 -- 172.21.15.33:0/1961201147 shutdown_connections 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa398ae0740 1 -- 172.21.15.33:0/1961201147 wait complete. 2024-09-16T00:36:09.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.784+0000 7fa398ae0740 20 client.18975 trim_cache size 0 max 16384 2024-09-16T00:36:09.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.798+0000 7fa398ae0740 1 Processor -- start 2024-09-16T00:36:09.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.798+0000 7fa398ae0740 1 -- start start 2024-09-16T00:36:09.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.798+0000 7fa398ae0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acc0310 0x56299ad12740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa398ae0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acf97e0 0x56299ad00930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa398ae0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acfac30 0x56299ad00e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa398ae0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3901af820 con 0x56299acc0310 2024-09-16T00:36:09.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa398ae0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa3901ad0f0 con 0x56299acfac30 2024-09-16T00:36:09.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa398ae0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa3901aaf40 con 0x56299acf97e0 2024-09-16T00:36:09.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa38ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acc0310 0x56299ad12740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa38ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acc0310 0x56299ad12740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33402/0 (socket says 172.21.15.33:33402) 2024-09-16T00:36:09.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa38ffff640 1 -- 172.21.15.33:0/2746559589 learned_addr learned my addr 172.21.15.33:0/2746559589 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:09.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/2746559589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acf97e0 0x56299ad00930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa394ca4640 1 --2- 172.21.15.33:0/2746559589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acfac30 0x56299ad00e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa38ffff640 1 -- 172.21.15.33:0/2746559589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acf97e0 msgr2=0x56299ad00930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:09.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa38ffff640 1 --2- 172.21.15.33:0/2746559589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acf97e0 0x56299ad00930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa38ffff640 1 -- 172.21.15.33:0/2746559589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acfac30 msgr2=0x56299ad00e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:09.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa38ffff640 1 --2- 172.21.15.33:0/2746559589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acfac30 0x56299ad00e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa38ffff640 1 -- 172.21.15.33:0/2746559589 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56299acf6940 con 0x56299acc0310 2024-09-16T00:36:09.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa394ca4640 1 --2- 172.21.15.33:0/2746559589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acfac30 0x56299ad00e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:36:09.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa38ffff640 1 --2- 172.21.15.33:0/2746559589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acc0310 0x56299ad12740 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fa380009e00 tx=0x7fa38000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efd7b6c0b6f48d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:09.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa375ffb640 1 -- 172.21.15.33:0/2746559589 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa380012190 con 0x56299acc0310 2024-09-16T00:36:09.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa375ffb640 1 -- 172.21.15.33:0/2746559589 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa38000fcc0 con 0x56299acc0310 2024-09-16T00:36:09.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.799+0000 7fa375ffb640 1 -- 172.21.15.33:0/2746559589 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38001c680 con 0x56299acc0310 2024-09-16T00:36:09.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.800+0000 7fa398ae0740 1 -- 172.21.15.33:0/2746559589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acc0310 msgr2=0x56299ad12740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:09.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.800+0000 7fa398ae0740 1 --2- 172.21.15.33:0/2746559589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acc0310 0x56299ad12740 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fa380009e00 tx=0x7fa38000fab0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.800+0000 7fa398ae0740 1 -- 172.21.15.33:0/2746559589 shutdown_connections 2024-09-16T00:36:09.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.800+0000 7fa398ae0740 1 --2- 172.21.15.33:0/2746559589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acfac30 0x56299ad00e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.800+0000 7fa398ae0740 1 --2- 172.21.15.33:0/2746559589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acf97e0 0x56299ad00930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.800+0000 7fa398ae0740 1 --2- 172.21.15.33:0/2746559589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acc0310 0x56299ad12740 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.800+0000 7fa398ae0740 1 -- 172.21.15.33:0/2746559589 >> 172.21.15.33:0/2746559589 conn(0x56299acfa670 msgr2=0x56299acf24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:09.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.800+0000 7fa398ae0740 1 -- 172.21.15.33:0/2746559589 shutdown_connections 2024-09-16T00:36:09.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.800+0000 7fa398ae0740 1 -- 172.21.15.33:0/2746559589 wait complete. 2024-09-16T00:36:09.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.801+0000 7fa398ae0740 1 -- start start 2024-09-16T00:36:09.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.801+0000 7fa398ae0740 1 Processor -- start 2024-09-16T00:36:09.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.801+0000 7fa398ae0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acc0310 0x56299ad0ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.801+0000 7fa398ae0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 0x56299ad0efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.801+0000 7fa398ae0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acfac30 0x56299ad0b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.801+0000 7fa398ae0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3901ab5b0 con 0x56299acf97e0 2024-09-16T00:36:09.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.801+0000 7fa398ae0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa3901aca20 con 0x56299acfac30 2024-09-16T00:36:09.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.801+0000 7fa398ae0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa3901abc80 con 0x56299acc0310 2024-09-16T00:36:09.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.801+0000 7fa398ae0740 10 client.0 _ll_register_callbacks cb 0x7fa380009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:09.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.801+0000 7fa38f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 0x56299ad0efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.801+0000 7fa38ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acc0310 0x56299ad0ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa38f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 0x56299ad0efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33404/0 (socket says 172.21.15.33:33404) 2024-09-16T00:36:09.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa38f7fe640 1 -- 172.21.15.33:0/1804150855 learned_addr learned my addr 172.21.15.33:0/1804150855 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa38ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acc0310 0x56299ad0ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:45706/0 (socket says 172.21.15.33:45706) 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa394ca4640 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acfac30 0x56299ad0b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa38f7fe640 1 -- 172.21.15.33:0/1804150855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acc0310 msgr2=0x56299ad0ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acc0310 0x56299ad0ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa38f7fe640 1 -- 172.21.15.33:0/1804150855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acfac30 msgr2=0x56299ad0b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acfac30 0x56299ad0b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa38f7fe640 1 -- 172.21.15.33:0/1804150855 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56299ad0c080 con 0x56299acf97e0 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa394ca4640 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acfac30 0x56299ad0b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa38f7fe640 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 0x56299ad0efd0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fa38400c6a0 tx=0x7fa384009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f48bd37e3db0dfd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa36f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa36f7fe640 1 -- 172.21.15.33:0/1804150855 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa384066060 con 0x56299acf97e0 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa36f7fe640 1 -- 172.21.15.33:0/1804150855 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa3840a4a30 con 0x56299acf97e0 2024-09-16T00:36:09.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.802+0000 7fa398ae0740 1 -- 172.21.15.33:0/1804150855 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa38000fcc0 con 0x56299acf97e0 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.803+0000 7fa36f7fe640 1 -- 172.21.15.33:0/1804150855 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa384099b30 con 0x56299acf97e0 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.803+0000 7fa398ae0740 10 client.18981 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.803+0000 7fa398ae0740 1 -- 172.21.15.33:0/1804150855 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56299ad81bb0 con 0x56299acf97e0 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.803+0000 7fa36f7fe640 1 -- 172.21.15.33:0/1804150855 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa38000fcc0 con 0x56299acf97e0 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.803+0000 7fa36f7fe640 1 -- 172.21.15.33:0/1804150855 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fa3840bc0b0 con 0x56299acf97e0 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.803+0000 7fa398ae0740 10 client.18981 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.803+0000 7fa398ae0740 10 client.18981 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.803+0000 7fa398ae0740 20 client.18981 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.803+0000 7fa398ae0740 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56299ad06970 0x56299ad83f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.803+0000 7fa398ae0740 4 client.18981 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.803+0000 7fa398ae0740 1 -- 172.21.15.33:0/1804150855 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56299ac64d70 con 0x56299ad06970 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.804+0000 7fa38ffff640 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56299ad06970 0x56299ad83f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:09.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.804+0000 7fa36f7fe640 10 client.18981 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:36:09.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.804+0000 7fa38ffff640 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56299ad06970 0x56299ad83f30 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4f03c7d24659d1ff server_cookie=781b7e01496ef461 in_seq=0 out_seq=0 2024-09-16T00:36:09.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.805+0000 7fa36f7fe640 1 -- 172.21.15.33:0/1804150855 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56299ac64d70 con 0x56299ad06970 2024-09-16T00:36:09.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.805+0000 7fa36f7fe640 10 client.18981 handle_command_reply: tid=0 2024-09-16T00:36:09.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.805+0000 7fa398ae0740 1 client.18981 shutdown 2024-09-16T00:36:09.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.806+0000 7fa398ae0740 1 -- 172.21.15.33:0/1804150855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 msgr2=0x56299ad0efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:09.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.806+0000 7fa398ae0740 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 0x56299ad0efd0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fa38400c6a0 tx=0x7fa384009700 comp rx=0 tx=0).stop 2024-09-16T00:36:09.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.806+0000 7fa398ae0740 1 -- 172.21.15.33:0/1804150855 shutdown_connections 2024-09-16T00:36:09.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.806+0000 7fa398ae0740 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x56299ad06970 0x56299ad83f30 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.806+0000 7fa398ae0740 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56299acfac30 0x56299ad0b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.806+0000 7fa398ae0740 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56299acf97e0 0x56299ad0efd0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.806+0000 7fa36f7fe640 0 client.18981 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:36:09.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.806+0000 7fa398ae0740 1 --2- 172.21.15.33:0/1804150855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56299acc0310 0x56299ad0ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.806+0000 7fa398ae0740 1 -- 172.21.15.33:0/1804150855 >> 172.21.15.33:0/1804150855 conn(0x56299acfa670 msgr2=0x56299ad04e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:09.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.806+0000 7fa398ae0740 1 -- 172.21.15.33:0/1804150855 shutdown_connections 2024-09-16T00:36:09.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.806+0000 7fa398ae0740 1 -- 172.21.15.33:0/1804150855 wait complete. 2024-09-16T00:36:09.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.806+0000 7fa398ae0740 20 client.18981 trim_cache size 0 max 16384 2024-09-16T00:36:09.812 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:36:09.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.807+0000 7fa396707640 1 -- 172.21.15.33:0/4017191784 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3780a3240 msgr2=0x7fa3780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:09.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.807+0000 7fa396707640 1 --2- 172.21.15.33:0/4017191784 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3780a3240 0x7fa3780a5700 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa388009430 tx=0x7fa38802a950 comp rx=0 tx=0).stop 2024-09-16T00:36:09.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.807+0000 7fa396707640 1 -- 172.21.15.33:0/4017191784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa390102570 msgr2=0x7fa390197980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:09.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.807+0000 7fa396707640 1 --2- 172.21.15.33:0/4017191784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa390102570 0x7fa390197980 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa38400e9f0 tx=0x7fa38400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.807+0000 7fa396707640 1 -- 172.21.15.33:0/4017191784 shutdown_connections 2024-09-16T00:36:09.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.807+0000 7fa396707640 1 --2- 172.21.15.33:0/4017191784 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3780a3240 0x7fa3780a5700 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.807+0000 7fa396707640 1 --2- 172.21.15.33:0/4017191784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa390102ef0 0x7fa39019bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.807+0000 7fa396707640 1 --2- 172.21.15.33:0/4017191784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa390102570 0x7fa390197980 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.807+0000 7fa396707640 1 --2- 172.21.15.33:0/4017191784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa390100e20 0x7fa390197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:09.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.807+0000 7fa396707640 1 -- 172.21.15.33:0/4017191784 >> 172.21.15.33:0/4017191784 conn(0x7fa3900fc7d0 msgr2=0x7fa3901068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:09.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.807+0000 7fa396707640 1 -- 172.21.15.33:0/4017191784 shutdown_connections 2024-09-16T00:36:09.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:09.807+0000 7fa396707640 1 -- 172.21.15.33:0/4017191784 wait complete. 2024-09-16T00:36:09.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:09 smithi083 ceph-mon[35242]: pgmap v920: 129 pgs: 129 active+clean; 589 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 16 MiB/s wr, 644 op/s 2024-09-16T00:36:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:09 smithi060 ceph-mon[35185]: pgmap v920: 129 pgs: 129 active+clean; 589 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 16 MiB/s wr, 644 op/s 2024-09-16T00:36:10.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:09 smithi033 ceph-mon[31243]: pgmap v920: 129 pgs: 129 active+clean; 589 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 16 MiB/s wr, 644 op/s 2024-09-16T00:36:10.082 INFO:tasks.workunit.client.0.smithi033.stdout: 10 110234 18.26 MB/sec execute 196 sec latency 77.412 ms 2024-09-16T00:36:10.654 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:36:10.982 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:11.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 110772 18.31 MB/sec execute 197 sec latency 61.736 ms 2024-09-16T00:36:11.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:11 smithi033 ceph-mon[31243]: pgmap v921: 129 pgs: 129 active+clean; 574 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 17 MiB/s wr, 780 op/s 2024-09-16T00:36:11.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:11 smithi083 ceph-mon[35242]: pgmap v921: 129 pgs: 129 active+clean; 574 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 17 MiB/s wr, 780 op/s 2024-09-16T00:36:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:11 smithi060 ceph-mon[35185]: pgmap v921: 129 pgs: 129 active+clean; 574 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 17 MiB/s wr, 780 op/s 2024-09-16T00:36:12.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.016+0000 7f0b29ce3640 1 -- 172.21.15.33:0/1385817468 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b24101df0 msgr2=0x7f0b241021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:12.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.016+0000 7f0b29ce3640 1 --2- 172.21.15.33:0/1385817468 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b24101df0 0x7f0b241021d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009900 tx=0x7f0b1802a400 comp rx=0 tx=0).stop 2024-09-16T00:36:12.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.016+0000 7f0b29ce3640 1 -- 172.21.15.33:0/1385817468 shutdown_connections 2024-09-16T00:36:12.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.016+0000 7f0b29ce3640 1 --2- 172.21.15.33:0/1385817468 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b24106f20 0x7f0b2410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.016+0000 7f0b29ce3640 1 --2- 172.21.15.33:0/1385817468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b241027a0 0x7f0b241067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.016+0000 7f0b29ce3640 1 --2- 172.21.15.33:0/1385817468 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b24101df0 0x7f0b241021d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.016+0000 7f0b29ce3640 1 -- 172.21.15.33:0/1385817468 >> 172.21.15.33:0/1385817468 conn(0x7f0b240fd8f0 msgr2=0x7f0b240ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:12.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.016+0000 7f0b29ce3640 1 -- 172.21.15.33:0/1385817468 shutdown_connections 2024-09-16T00:36:12.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.016+0000 7f0b29ce3640 1 -- 172.21.15.33:0/1385817468 wait complete. 2024-09-16T00:36:12.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.017+0000 7f0b29ce3640 1 Processor -- start 2024-09-16T00:36:12.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.018+0000 7f0b29ce3640 1 -- start start 2024-09-16T00:36:12.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.018+0000 7f0b29ce3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b24101df0 0x7f0b241984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.019+0000 7f0b29ce3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b241027a0 0x7f0b24198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.019+0000 7f0b29ce3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b24106f20 0x7f0b2419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.019+0000 7f0b22ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b241027a0 0x7f0b24198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.019+0000 7f0b22ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b241027a0 0x7f0b24198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:39418/0 (socket says 172.21.15.33:39418) 2024-09-16T00:36:12.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.019+0000 7f0b22ffd640 1 -- 172.21.15.33:0/3936703982 learned_addr learned my addr 172.21.15.33:0/3936703982 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:12.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.019+0000 7f0b29ce3640 1 -- 172.21.15.33:0/3936703982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0b24111fa0 con 0x7f0b24101df0 2024-09-16T00:36:12.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.019+0000 7f0b29ce3640 1 -- 172.21.15.33:0/3936703982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0b24111e20 con 0x7f0b241027a0 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.019+0000 7f0b29ce3640 1 -- 172.21.15.33:0/3936703982 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0b24112120 con 0x7f0b24106f20 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.020+0000 7f0b237fe640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b24101df0 0x7f0b241984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.020+0000 7f0b237fe640 1 -- 172.21.15.33:0/3936703982 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b24106f20 msgr2=0x7f0b2419cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.020+0000 7f0b237fe640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b24106f20 0x7f0b2419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.020+0000 7f0b237fe640 1 -- 172.21.15.33:0/3936703982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b241027a0 msgr2=0x7f0b24198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.020+0000 7f0b237fe640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b241027a0 0x7f0b24198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.020+0000 7f0b237fe640 1 -- 172.21.15.33:0/3936703982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b2419d470 con 0x7f0b24101df0 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.021+0000 7f0b22ffd640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b241027a0 0x7f0b24198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.021+0000 7f0b237fe640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b24101df0 0x7f0b241984c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f0b1802a920 tx=0x7f0b180045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.022+0000 7f0b20ff9640 1 -- 172.21.15.33:0/3936703982 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b18040070 con 0x7f0b24101df0 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.022+0000 7f0b20ff9640 1 -- 172.21.15.33:0/3936703982 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b18031c80 con 0x7f0b24101df0 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.022+0000 7f0b20ff9640 1 -- 172.21.15.33:0/3936703982 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b18035020 con 0x7f0b24101df0 2024-09-16T00:36:12.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.023+0000 7f0b29ce3640 1 -- 172.21.15.33:0/3936703982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b2419d700 con 0x7f0b24101df0 2024-09-16T00:36:12.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.023+0000 7f0b29ce3640 1 -- 172.21.15.33:0/3936703982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b2419db90 con 0x7f0b24101df0 2024-09-16T00:36:12.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.026+0000 7f0b2c0bc740 1 Processor -- start 2024-09-16T00:36:12.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.028+0000 7f0b20ff9640 1 -- 172.21.15.33:0/3936703982 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0b18043070 con 0x7f0b24101df0 2024-09-16T00:36:12.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.029+0000 7f0b2c0bc740 1 -- start start 2024-09-16T00:36:12.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.029+0000 7f0b20ff9640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b0c0a3410 0x7f0b0c0a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.030+0000 7f0b20ff9640 1 -- 172.21.15.33:0/3936703982 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0b180f08d0 con 0x7f0b24101df0 2024-09-16T00:36:12.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.031+0000 7f0b2c0bc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 0x55dd7b3b88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.031+0000 7f0b2c0bc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3b9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.031+0000 7f0b2c0bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.031+0000 7f0b2c0bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dd7b3162f0 con 0x55dd7b3c3b80 2024-09-16T00:36:12.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.031+0000 7f0b2c0bc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55dd7b2bdcf0 con 0x55dd7b3ba350 2024-09-16T00:36:12.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.031+0000 7f0b2c0bc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dd7b3122b0 con 0x55dd7b3b8de0 2024-09-16T00:36:12.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.031+0000 7f0b237fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 0x55dd7b3b88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.031+0000 7f0b237fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 0x55dd7b3b88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:39422/0 (socket says 172.21.15.33:39422) 2024-09-16T00:36:12.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.031+0000 7f0b237fe640 1 -- 172.21.15.33:0/642449792 learned_addr learned my addr 172.21.15.33:0/642449792 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:12.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.032+0000 7f0b23fff640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b0c0a3410 0x7f0b0c0a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.032+0000 7f0b23fff640 1 --2- 172.21.15.33:0/642449792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.032+0000 7f0b237fe640 1 -- 172.21.15.33:0/642449792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3b8de0 msgr2=0x55dd7b3b9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:12.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.032+0000 7f0b237fe640 1 --2- 172.21.15.33:0/642449792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.032+0000 7f0b237fe640 1 -- 172.21.15.33:0/642449792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 msgr2=0x55dd7b3c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:12.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.032+0000 7f0b237fe640 1 --2- 172.21.15.33:0/642449792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.032+0000 7f0b237fe640 1 -- 172.21.15.33:0/642449792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd7b2a23a0 con 0x55dd7b3ba350 2024-09-16T00:36:12.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.034+0000 7f0b23fff640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b0c0a3410 0x7f0b0c0a58d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x55dd7b3b9880 tx=0x7f0b1c009780 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:12.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.035+0000 7f0b237fe640 1 --2- 172.21.15.33:0/642449792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 0x55dd7b3b88a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0b180b8e70 tx=0x7f0b18004b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b884afa93c42964 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:12.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.037+0000 7f0b09ffb640 1 -- 172.21.15.33:0/642449792 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1804a040 con 0x55dd7b3ba350 2024-09-16T00:36:12.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.037+0000 7f0b09ffb640 1 -- 172.21.15.33:0/642449792 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b180bca10 con 0x55dd7b3ba350 2024-09-16T00:36:12.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.037+0000 7f0b09ffb640 1 -- 172.21.15.33:0/642449792 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b18004bf0 con 0x55dd7b3ba350 2024-09-16T00:36:12.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.038+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/642449792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 msgr2=0x55dd7b3b88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:12.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.038+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/642449792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 0x55dd7b3b88a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0b180b8e70 tx=0x7f0b18004b60 comp rx=0 tx=0).stop 2024-09-16T00:36:12.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.038+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/642449792 shutdown_connections 2024-09-16T00:36:12.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.038+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/642449792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.038+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/642449792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3b9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.039+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/642449792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 0x55dd7b3b88a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.039+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/642449792 >> 172.21.15.33:0/642449792 conn(0x55dd7b379310 msgr2=0x55dd7b3b5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:12.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.039+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/642449792 shutdown_connections 2024-09-16T00:36:12.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.039+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/642449792 wait complete. 2024-09-16T00:36:12.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.040+0000 7f0b2c0bc740 1 -- start start 2024-09-16T00:36:12.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.040+0000 7f0b2c0bc740 1 Processor -- start 2024-09-16T00:36:12.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.041+0000 7f0b2c0bc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3c3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.041+0000 7f0b2c0bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3ba350 0x55dd7b3c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.041+0000 7f0b237fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3c3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.041+0000 7f0b237fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3c3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:39434/0 (socket says 172.21.15.33:39434) 2024-09-16T00:36:12.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.041+0000 7f0b237fe640 1 -- 172.21.15.33:0/4152994261 learned_addr learned my addr 172.21.15.33:0/4152994261 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:12.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.041+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3c1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.041+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4152994261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dd7b3c6640 con 0x55dd7b3ba350 2024-09-16T00:36:12.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.042+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4152994261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55dd7b2d1060 con 0x55dd7b3b8de0 2024-09-16T00:36:12.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.042+0000 7f0b23fff640 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3c1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.042+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4152994261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dd7b3c67c0 con 0x55dd7b3c3b80 2024-09-16T00:36:12.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.042+0000 7f0b2c0bc740 10 client.0 _ll_register_callbacks cb 0x55dd7b0248c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:12.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.042+0000 7f0b237fe640 1 -- 172.21.15.33:0/4152994261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3c3b80 msgr2=0x55dd7b3c1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:12.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.042+0000 7f0b237fe640 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3c1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.042+0000 7f0b237fe640 1 -- 172.21.15.33:0/4152994261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3ba350 msgr2=0x55dd7b3c08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:12.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.042+0000 7f0b237fe640 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3ba350 0x55dd7b3c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.042+0000 7f0b237fe640 1 -- 172.21.15.33:0/4152994261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd7b0873c0 con 0x55dd7b3b8de0 2024-09-16T00:36:12.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.043+0000 7f0b237fe640 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3c3620 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0b180c5150 tx=0x7f0b180c56f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4abcc096311dc09e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:12.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.043+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4152994261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b180bca10 con 0x55dd7b3b8de0 2024-09-16T00:36:12.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.043+0000 7f0b02ffd640 10 client.28259 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:36:12.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.043+0000 7f0b02ffd640 1 -- 172.21.15.33:0/4152994261 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1804a070 con 0x55dd7b3b8de0 2024-09-16T00:36:12.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.044+0000 7f0b02ffd640 1 -- 172.21.15.33:0/4152994261 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0b180db270 con 0x55dd7b3b8de0 2024-09-16T00:36:12.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.044+0000 7f0b02ffd640 1 -- 172.21.15.33:0/4152994261 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0b180cd050 con 0x55dd7b3b8de0 2024-09-16T00:36:12.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.044+0000 7f0b23fff640 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3c1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:36:12.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.044+0000 7f0b2c0bc740 10 client.28259 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:12.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.044+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4152994261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({fsmap=33}) -- 0x55dd7b3c6640 con 0x55dd7b3b8de0 2024-09-16T00:36:12.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.045+0000 7f0b02ffd640 1 -- 172.21.15.33:0/4152994261 <== mon.2 v2:172.21.15.60:3300/0 4 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0b1803a040 con 0x55dd7b3b8de0 2024-09-16T00:36:12.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.045+0000 7f0b2c0bc740 10 client.28259 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:12.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.045+0000 7f0b2c0bc740 10 client.28259 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:36:12.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.045+0000 7f0b2c0bc740 20 client.28259 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:12.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.045+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55dd7b3d3110 0x55dd7b3d5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.045+0000 7f0b2c0bc740 4 client.28259 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.045+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4152994261 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0b180bca10 con 0x55dd7b3d3110 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.048+0000 7f0b22ffd640 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55dd7b3d3110 0x55dd7b3d5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.050+0000 7f0b22ffd640 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55dd7b3d3110 0x55dd7b3d5500 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4916e8f4d9d5c5be server_cookie=eb1e0a1155d9a0de in_seq=0 out_seq=0 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.050+0000 7f0b02ffd640 10 client.28259 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.054+0000 7f0b02ffd640 1 -- 172.21.15.33:0/4152994261 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f0b180bca10 con 0x55dd7b3d3110 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.054+0000 7f0b02ffd640 10 client.28259 handle_command_reply: tid=0 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.054+0000 7f0b2c0bc740 1 client.28259 shutdown 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.055+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4152994261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3b8de0 msgr2=0x55dd7b3c3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.055+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3c3620 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0b180c5150 tx=0x7f0b180c56f0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.055+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4152994261 shutdown_connections 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.055+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3c1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.055+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3ba350 0x55dd7b3c08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.055+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55dd7b3d3110 0x55dd7b3d5500 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.055+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4152994261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3c3620 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.055+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4152994261 >> 172.21.15.33:0/4152994261 conn(0x55dd7b379310 msgr2=0x55dd7b3bb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:12.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.055+0000 7f0b02ffd640 0 client.28259 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:36:12.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.056+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4152994261 shutdown_connections 2024-09-16T00:36:12.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.056+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4152994261 wait complete. 2024-09-16T00:36:12.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.056+0000 7f0b2c0bc740 20 client.28259 trim_cache size 0 max 16384 2024-09-16T00:36:12.093 INFO:tasks.workunit.client.0.smithi033.stdout: 10 111220 18.33 MB/sec execute 198 sec latency 97.192 ms 2024-09-16T00:36:12.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.082+0000 7f0b2c0bc740 1 Processor -- start 2024-09-16T00:36:12.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.082+0000 7f0b2c0bc740 1 -- start start 2024-09-16T00:36:12.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.083+0000 7f0b2c0bc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3bb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.083+0000 7f0b2c0bc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3ba350 0x55dd7b3bbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.083+0000 7f0b2c0bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3dae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.083+0000 7f0b2c0bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0b241af8c0 con 0x55dd7b3c3b80 2024-09-16T00:36:12.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.083+0000 7f0b2c0bc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0b241ad190 con 0x55dd7b3b8de0 2024-09-16T00:36:12.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.083+0000 7f0b2c0bc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0b241aafe0 con 0x55dd7b3ba350 2024-09-16T00:36:12.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.083+0000 7f0b22ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3ba350 0x55dd7b3bbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.083+0000 7f0b22ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3ba350 0x55dd7b3bbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:45964/0 (socket says 172.21.15.33:45964) 2024-09-16T00:36:12.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.083+0000 7f0b22ffd640 1 -- 172.21.15.33:0/923535444 learned_addr learned my addr 172.21.15.33:0/923535444 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:12.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.083+0000 7f0b23fff640 1 --2- 172.21.15.33:0/923535444 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3dae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.083+0000 7f0b237fe640 1 --2- 172.21.15.33:0/923535444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3bb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.084+0000 7f0b22ffd640 1 -- 172.21.15.33:0/923535444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3b8de0 msgr2=0x55dd7b3bb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:12.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.084+0000 7f0b22ffd640 1 --2- 172.21.15.33:0/923535444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3bb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.084+0000 7f0b22ffd640 1 -- 172.21.15.33:0/923535444 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 msgr2=0x55dd7b3dae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:12.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.084+0000 7f0b22ffd640 1 --2- 172.21.15.33:0/923535444 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3dae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.084+0000 7f0b22ffd640 1 -- 172.21.15.33:0/923535444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd7b3ab310 con 0x55dd7b3ba350 2024-09-16T00:36:12.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.084+0000 7f0b22ffd640 1 --2- 172.21.15.33:0/923535444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3ba350 0x55dd7b3bbe40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55dd7b3b62d0 tx=0x7f0b1400af30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d91e099521b3182 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:12.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.085+0000 7f0b097fa640 1 -- 172.21.15.33:0/923535444 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b14022070 con 0x55dd7b3ba350 2024-09-16T00:36:12.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.085+0000 7f0b097fa640 1 -- 172.21.15.33:0/923535444 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b14014bf0 con 0x55dd7b3ba350 2024-09-16T00:36:12.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.085+0000 7f0b097fa640 1 -- 172.21.15.33:0/923535444 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1401d660 con 0x55dd7b3ba350 2024-09-16T00:36:12.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.086+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/923535444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3ba350 msgr2=0x55dd7b3bbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:12.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.086+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/923535444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3ba350 0x55dd7b3bbe40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55dd7b3b62d0 tx=0x7f0b1400af30 comp rx=0 tx=0).stop 2024-09-16T00:36:12.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.086+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/923535444 shutdown_connections 2024-09-16T00:36:12.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.086+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/923535444 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3dae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.086+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/923535444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3ba350 0x55dd7b3bbe40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.086+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/923535444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3bb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.087+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/923535444 >> 172.21.15.33:0/923535444 conn(0x55dd7b379310 msgr2=0x55dd7b3bf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:12.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.087+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/923535444 shutdown_connections 2024-09-16T00:36:12.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.087+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/923535444 wait complete. 2024-09-16T00:36:12.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.087+0000 7f0b2c0bc740 1 -- start start 2024-09-16T00:36:12.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.087+0000 7f0b2c0bc740 1 Processor -- start 2024-09-16T00:36:12.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b2c0bc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3bda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b2c0bc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 0x55dd7b3bdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b2c0bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3be510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b237fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3bda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b2c0bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0b241ab650 con 0x55dd7b3c3b80 2024-09-16T00:36:12.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b2c0bc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0b241acac0 con 0x55dd7b3ba350 2024-09-16T00:36:12.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b2c0bc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0b241abd20 con 0x55dd7b3b8de0 2024-09-16T00:36:12.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b237fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3bda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:45976/0 (socket says 172.21.15.33:45976) 2024-09-16T00:36:12.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b237fe640 1 -- 172.21.15.33:0/4223129128 learned_addr learned my addr 172.21.15.33:0/4223129128 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:12.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b22ffd640 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 0x55dd7b3bdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b2c0bc740 10 client.0 _ll_register_callbacks cb 0x7f0b1c009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:12.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b23fff640 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3be510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b22ffd640 1 -- 172.21.15.33:0/4223129128 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3b8de0 msgr2=0x55dd7b3bda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:12.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b22ffd640 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3bda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b22ffd640 1 -- 172.21.15.33:0/4223129128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 msgr2=0x55dd7b3be510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:12.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b22ffd640 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3be510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.088+0000 7f0b22ffd640 1 -- 172.21.15.33:0/4223129128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd7b43cf10 con 0x55dd7b3ba350 2024-09-16T00:36:12.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.089+0000 7f0b22ffd640 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 0x55dd7b3bdfd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55dd7b3c0130 tx=0x7f0b1400a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ee2191ce036e3c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:12.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.089+0000 7f0b02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:36:12.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.089+0000 7f0b02ffd640 1 -- 172.21.15.33:0/4223129128 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1403d070 con 0x55dd7b3ba350 2024-09-16T00:36:12.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.089+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4223129128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b14014bf0 con 0x55dd7b3ba350 2024-09-16T00:36:12.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.089+0000 7f0b02ffd640 1 -- 172.21.15.33:0/4223129128 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0b14016960 con 0x55dd7b3ba350 2024-09-16T00:36:12.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.090+0000 7f0b02ffd640 1 -- 172.21.15.33:0/4223129128 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0b14049050 con 0x55dd7b3ba350 2024-09-16T00:36:12.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.090+0000 7f0b2c0bc740 10 client.28265 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:12.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.090+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4223129128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dd7b43e1d0 con 0x55dd7b3ba350 2024-09-16T00:36:12.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.090+0000 7f0b02ffd640 1 -- 172.21.15.33:0/4223129128 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b14033040 con 0x55dd7b3ba350 2024-09-16T00:36:12.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.090+0000 7f0b02ffd640 1 -- 172.21.15.33:0/4223129128 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0b14039490 con 0x55dd7b3ba350 2024-09-16T00:36:12.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.090+0000 7f0b2c0bc740 10 client.28265 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:12.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.090+0000 7f0b2c0bc740 10 client.28265 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:36:12.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.090+0000 7f0b2c0bc740 20 client.28265 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:12.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.090+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55dd7b43e900 0x55dd7b440d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:12.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.090+0000 7f0b2c0bc740 4 client.28265 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:36:12.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.090+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4223129128 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f0b14014bf0 con 0x55dd7b43e900 2024-09-16T00:36:12.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.090+0000 7f0b237fe640 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55dd7b43e900 0x55dd7b440d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:12.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.091+0000 7f0b02ffd640 10 client.28265 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:36:12.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.091+0000 7f0b237fe640 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55dd7b43e900 0x55dd7b440d20 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=49cf1f1a52ca9dd2 server_cookie=8356eb7ef1a42054 in_seq=0 out_seq=0 2024-09-16T00:36:12.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b02ffd640 1 -- 172.21.15.33:0/4223129128 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f0b14014bf0 con 0x55dd7b43e900 2024-09-16T00:36:12.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b02ffd640 10 client.28265 handle_command_reply: tid=0 2024-09-16T00:36:12.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b2c0bc740 1 client.28265 shutdown 2024-09-16T00:36:12.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4223129128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 msgr2=0x55dd7b3bdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:12.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 0x55dd7b3bdfd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55dd7b3c0130 tx=0x7f0b1400a7a0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4223129128 shutdown_connections 2024-09-16T00:36:12.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd7b3c3b80 0x55dd7b3be510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55dd7b43e900 0x55dd7b440d20 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd7b3ba350 0x55dd7b3bdfd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b2c0bc740 1 --2- 172.21.15.33:0/4223129128 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd7b3b8de0 0x55dd7b3bda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4223129128 >> 172.21.15.33:0/4223129128 conn(0x55dd7b379310 msgr2=0x55dd7b3ab500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:12.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b02ffd640 0 client.28265 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:36:12.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4223129128 shutdown_connections 2024-09-16T00:36:12.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.094+0000 7f0b2c0bc740 1 -- 172.21.15.33:0/4223129128 wait complete. 2024-09-16T00:36:12.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.095+0000 7f0b2c0bc740 20 client.28265 trim_cache size 0 max 16384 2024-09-16T00:36:12.108 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:36:12.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.096+0000 7f0b29ce3640 1 -- 172.21.15.33:0/3936703982 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b0c0a3410 msgr2=0x7f0b0c0a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:12.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.096+0000 7f0b29ce3640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b0c0a3410 0x7f0b0c0a58d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x55dd7b3b9880 tx=0x7f0b1c009780 comp rx=0 tx=0).stop 2024-09-16T00:36:12.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.096+0000 7f0b29ce3640 1 -- 172.21.15.33:0/3936703982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b24101df0 msgr2=0x7f0b241984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:12.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.096+0000 7f0b29ce3640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b24101df0 0x7f0b241984c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f0b1802a920 tx=0x7f0b180045e0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.096+0000 7f0b29ce3640 1 -- 172.21.15.33:0/3936703982 shutdown_connections 2024-09-16T00:36:12.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.096+0000 7f0b29ce3640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b0c0a3410 0x7f0b0c0a58d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.096+0000 7f0b29ce3640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b24106f20 0x7f0b2419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.096+0000 7f0b29ce3640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b241027a0 0x7f0b24198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.096+0000 7f0b29ce3640 1 --2- 172.21.15.33:0/3936703982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b24101df0 0x7f0b241984c0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:12.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.096+0000 7f0b29ce3640 1 -- 172.21.15.33:0/3936703982 >> 172.21.15.33:0/3936703982 conn(0x7f0b240fd8f0 msgr2=0x7f0b240fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:12.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.096+0000 7f0b29ce3640 1 -- 172.21.15.33:0/3936703982 shutdown_connections 2024-09-16T00:36:12.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:12.096+0000 7f0b29ce3640 1 -- 172.21.15.33:0/3936703982 wait complete. 2024-09-16T00:36:13.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 111734 18.36 MB/sec execute 199 sec latency 89.454 ms 2024-09-16T00:36:13.192 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:36:13.509 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:13 smithi083 ceph-mon[35242]: pgmap v922: 129 pgs: 129 active+clean; 577 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 13 MiB/s wr, 726 op/s 2024-09-16T00:36:13.961 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:13 smithi033 ceph-mon[31243]: pgmap v922: 129 pgs: 129 active+clean; 577 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 13 MiB/s wr, 726 op/s 2024-09-16T00:36:13.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:13 smithi060 ceph-mon[35185]: pgmap v922: 129 pgs: 129 active+clean; 577 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 13 MiB/s wr, 726 op/s 2024-09-16T00:36:14.084 INFO:tasks.workunit.client.0.smithi033.stdout: 10 112341 18.42 MB/sec execute 200 sec latency 82.920 ms 2024-09-16T00:36:14.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.064+0000 7f21654a5640 1 -- 172.21.15.33:0/1665012196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2160100e20 msgr2=0x7f2160101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:14.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.064+0000 7f21654a5640 1 --2- 172.21.15.33:0/1665012196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2160100e20 0x7f2160101200 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f2150009900 tx=0x7f215002a480 comp rx=0 tx=0).stop 2024-09-16T00:36:14.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.064+0000 7f21654a5640 1 -- 172.21.15.33:0/1665012196 shutdown_connections 2024-09-16T00:36:14.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.064+0000 7f21654a5640 1 --2- 172.21.15.33:0/1665012196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102ef0 0x7f216010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.064+0000 7f21654a5640 1 --2- 172.21.15.33:0/1665012196 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2160102570 0x7f21601029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.064+0000 7f21654a5640 1 --2- 172.21.15.33:0/1665012196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2160100e20 0x7f2160101200 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.064+0000 7f21654a5640 1 -- 172.21.15.33:0/1665012196 >> 172.21.15.33:0/1665012196 conn(0x7f21600fc7d0 msgr2=0x7f21600febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:14.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.064+0000 7f21654a5640 1 -- 172.21.15.33:0/1665012196 shutdown_connections 2024-09-16T00:36:14.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.064+0000 7f21654a5640 1 -- 172.21.15.33:0/1665012196 wait complete. 2024-09-16T00:36:14.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.064+0000 7f21654a5640 1 Processor -- start 2024-09-16T00:36:14.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.064+0000 7f21654a5640 1 -- start start 2024-09-16T00:36:14.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f21654a5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2160100e20 0x7f2160198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f21654a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2160102570 0x7f2160198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f21654a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102ef0 0x7f216019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f21654a5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2160110f80 con 0x7f2160100e20 2024-09-16T00:36:14.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f21654a5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2160110e00 con 0x7f2160102ef0 2024-09-16T00:36:14.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f21654a5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2160111100 con 0x7f2160102570 2024-09-16T00:36:14.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2160100e20 0x7f2160198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f2157fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2160102570 0x7f2160198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2160100e20 0x7f2160198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38098/0 (socket says 172.21.15.33:38098) 2024-09-16T00:36:14.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215effd640 1 -- 172.21.15.33:0/691892437 learned_addr learned my addr 172.21.15.33:0/691892437 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:14.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215f7fe640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102ef0 0x7f216019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215effd640 1 -- 172.21.15.33:0/691892437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2160102570 msgr2=0x7f2160198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:14.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215effd640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2160102570 0x7f2160198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215effd640 1 -- 172.21.15.33:0/691892437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102ef0 msgr2=0x7f216019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215effd640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102ef0 0x7f216019cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215effd640 1 -- 172.21.15.33:0/691892437 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f216019d4d0 con 0x7f2160100e20 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215f7fe640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102ef0 0x7f216019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215effd640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2160100e20 0x7f2160198520 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f2150006010 tx=0x7f2150003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215cff9640 1 -- 172.21.15.33:0/691892437 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215003f070 con 0x7f2160100e20 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.065+0000 7f215cff9640 1 -- 172.21.15.33:0/691892437 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2150031c30 con 0x7f2160100e20 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.066+0000 7f21654a5640 1 -- 172.21.15.33:0/691892437 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f216019d760 con 0x7f2160100e20 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.066+0000 7f215cff9640 1 -- 172.21.15.33:0/691892437 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150035030 con 0x7f2160100e20 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.066+0000 7f21654a5640 1 -- 172.21.15.33:0/691892437 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21601a50a0 con 0x7f2160100e20 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.069+0000 7f215cff9640 1 -- 172.21.15.33:0/691892437 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2150042070 con 0x7f2160100e20 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.069+0000 7f216787e740 1 Processor -- start 2024-09-16T00:36:14.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.069+0000 7f216787e740 1 -- start start 2024-09-16T00:36:14.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.069+0000 7f216787e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 0x5566a0fa84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.069+0000 7f216787e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fa89e0 0x5566a0fa8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.069+0000 7f216787e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fb3780 0x5566a0fb5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.069+0000 7f216787e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5566a0f0a2f0 con 0x5566a0fa9f50 2024-09-16T00:36:14.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.069+0000 7f216787e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5566a0eb1cf0 con 0x5566a0fa89e0 2024-09-16T00:36:14.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.069+0000 7f216787e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5566a0f062b0 con 0x5566a0fb3780 2024-09-16T00:36:14.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.070+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 0x5566a0fa84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.070+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 0x5566a0fa84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38110/0 (socket says 172.21.15.33:38110) 2024-09-16T00:36:14.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.070+0000 7f215f7fe640 1 -- 172.21.15.33:0/2404262799 learned_addr learned my addr 172.21.15.33:0/2404262799 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:14.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.070+0000 7f2157fff640 1 --2- 172.21.15.33:0/2404262799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fb3780 0x5566a0fb5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.070+0000 7f215effd640 1 --2- 172.21.15.33:0/2404262799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fa89e0 0x5566a0fa8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.070+0000 7f215f7fe640 1 -- 172.21.15.33:0/2404262799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fb3780 msgr2=0x5566a0fb5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:14.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.070+0000 7f215f7fe640 1 --2- 172.21.15.33:0/2404262799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fb3780 0x5566a0fb5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.070+0000 7f215f7fe640 1 -- 172.21.15.33:0/2404262799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fa89e0 msgr2=0x5566a0fa8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:14.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.070+0000 7f215f7fe640 1 --2- 172.21.15.33:0/2404262799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fa89e0 0x5566a0fa8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.070+0000 7f215f7fe640 1 -- 172.21.15.33:0/2404262799 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566a0e963a0 con 0x5566a0fa9f50 2024-09-16T00:36:14.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.070+0000 7f215f7fe640 1 --2- 172.21.15.33:0/2404262799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 0x5566a0fa84a0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f2158009fd0 tx=0x7f21580108e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8174f6f2466d559 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:14.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.071+0000 7f215cff9640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f214c0a32c0 0x7f214c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.071+0000 7f21557fa640 1 -- 172.21.15.33:0/2404262799 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2158011760 con 0x5566a0fa9f50 2024-09-16T00:36:14.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.071+0000 7f215effd640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f214c0a32c0 0x7f214c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.071+0000 7f21557fa640 1 -- 172.21.15.33:0/2404262799 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2158011de0 con 0x5566a0fa9f50 2024-09-16T00:36:14.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.071+0000 7f21557fa640 1 -- 172.21.15.33:0/2404262799 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215800e3c0 con 0x5566a0fa9f50 2024-09-16T00:36:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.071+0000 7f215cff9640 1 -- 172.21.15.33:0/691892437 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f21500f0500 con 0x7f2160100e20 2024-09-16T00:36:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.072+0000 7f215effd640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f214c0a32c0 0x7f214c0a5780 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f21500c4d80 tx=0x7f21500f3ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.072+0000 7f216787e740 1 -- 172.21.15.33:0/2404262799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 msgr2=0x5566a0fa84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.072+0000 7f216787e740 1 --2- 172.21.15.33:0/2404262799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 0x5566a0fa84a0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f2158009fd0 tx=0x7f21580108e0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.072+0000 7f216787e740 1 -- 172.21.15.33:0/2404262799 shutdown_connections 2024-09-16T00:36:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.072+0000 7f216787e740 1 --2- 172.21.15.33:0/2404262799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fb3780 0x5566a0fb5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.072+0000 7f216787e740 1 --2- 172.21.15.33:0/2404262799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fa89e0 0x5566a0fa8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.072+0000 7f216787e740 1 --2- 172.21.15.33:0/2404262799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 0x5566a0fa84a0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.072+0000 7f216787e740 1 -- 172.21.15.33:0/2404262799 >> 172.21.15.33:0/2404262799 conn(0x5566a0f6d310 msgr2=0x5566a0fa5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.072+0000 7f216787e740 1 -- 172.21.15.33:0/2404262799 shutdown_connections 2024-09-16T00:36:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.072+0000 7f216787e740 1 -- 172.21.15.33:0/2404262799 wait complete. 2024-09-16T00:36:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.072+0000 7f216787e740 1 -- start start 2024-09-16T00:36:14.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f216787e740 1 Processor -- start 2024-09-16T00:36:14.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f216787e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fa89e0 0x5566a0fb3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f216787e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 0x5566a0fb0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f216787e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb3780 0x5566a0fb0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f216787e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5566a0fb6520 con 0x5566a0fa9f50 2024-09-16T00:36:14.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f216787e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5566a0ec5060 con 0x5566a0fb3780 2024-09-16T00:36:14.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fa89e0 0x5566a0fb3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f216787e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5566a0fb66a0 con 0x5566a0fa89e0 2024-09-16T00:36:14.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fa89e0 0x5566a0fb3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46006/0 (socket says 172.21.15.33:46006) 2024-09-16T00:36:14.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f215f7fe640 1 -- 172.21.15.33:0/3102192450 learned_addr learned my addr 172.21.15.33:0/3102192450 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:14.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f2157fff640 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 0x5566a0fb0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f215f7fe640 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb3780 0x5566a0fb0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.073+0000 7f216787e740 10 client.0 _ll_register_callbacks cb 0x5566a0c188c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f2157fff640 1 -- 172.21.15.33:0/3102192450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fa89e0 msgr2=0x5566a0fb3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f2157fff640 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fa89e0 0x5566a0fb3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f2157fff640 1 -- 172.21.15.33:0/3102192450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb3780 msgr2=0x5566a0fb0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f2157fff640 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb3780 0x5566a0fb0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f2157fff640 1 -- 172.21.15.33:0/3102192450 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566a0c7b3c0 con 0x5566a0fa9f50 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f2157fff640 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 0x5566a0fb0420 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f21600fff40 tx=0x7f2148012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5babf1f5fc8637bd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f21467fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f21467fc640 1 -- 172.21.15.33:0/3102192450 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2148004020 con 0x5566a0fa9f50 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f21467fc640 1 -- 172.21.15.33:0/3102192450 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f21480041c0 con 0x5566a0fa9f50 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f216787e740 1 -- 172.21.15.33:0/3102192450 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2158011de0 con 0x5566a0fa9f50 2024-09-16T00:36:14.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f21467fc640 1 -- 172.21.15.33:0/3102192450 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2148025050 con 0x5566a0fa9f50 2024-09-16T00:36:14.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f216787e740 10 client.19008 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:14.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f216787e740 1 -- 172.21.15.33:0/3102192450 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5566a0fc2050 con 0x5566a0fa9f50 2024-09-16T00:36:14.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f21467fc640 1 -- 172.21.15.33:0/3102192450 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2158011de0 con 0x5566a0fa9f50 2024-09-16T00:36:14.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.074+0000 7f21467fc640 1 -- 172.21.15.33:0/3102192450 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2148003a60 con 0x5566a0fa9f50 2024-09-16T00:36:14.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.075+0000 7f216787e740 10 client.19008 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:14.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.075+0000 7f216787e740 10 client.19008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:36:14.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.075+0000 7f216787e740 20 client.19008 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:14.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.075+0000 7f216787e740 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566a0fc3560 0x5566a0fc5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.075+0000 7f216787e740 4 client.19008 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:36:14.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.075+0000 7f216787e740 1 -- 172.21.15.33:0/3102192450 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566a0f17860 con 0x5566a0fc3560 2024-09-16T00:36:14.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.075+0000 7f215f7fe640 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566a0fc3560 0x5566a0fc5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.075+0000 7f21467fc640 10 client.19008 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:14.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.075+0000 7f215f7fe640 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566a0fc3560 0x5566a0fc5980 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd375da006edc61f server_cookie=d474068448e31a37 in_seq=0 out_seq=0 2024-09-16T00:36:14.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f21467fc640 1 -- 172.21.15.33:0/3102192450 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5566a0f17860 con 0x5566a0fc3560 2024-09-16T00:36:14.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f21467fc640 10 client.19008 handle_command_reply: tid=0 2024-09-16T00:36:14.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f216787e740 1 client.19008 shutdown 2024-09-16T00:36:14.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f216787e740 1 -- 172.21.15.33:0/3102192450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 msgr2=0x5566a0fb0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:14.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f216787e740 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 0x5566a0fb0420 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f21600fff40 tx=0x7f2148012b60 comp rx=0 tx=0).stop 2024-09-16T00:36:14.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f216787e740 1 -- 172.21.15.33:0/3102192450 shutdown_connections 2024-09-16T00:36:14.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f216787e740 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566a0fc3560 0x5566a0fc5980 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f216787e740 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb3780 0x5566a0fb0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f216787e740 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa9f50 0x5566a0fb0420 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f216787e740 1 --2- 172.21.15.33:0/3102192450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fa89e0 0x5566a0fb3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f21467fc640 0 client.19008 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f216787e740 1 -- 172.21.15.33:0/3102192450 >> 172.21.15.33:0/3102192450 conn(0x5566a0f6d310 msgr2=0x5566a0faad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f216787e740 1 -- 172.21.15.33:0/3102192450 shutdown_connections 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f216787e740 1 -- 172.21.15.33:0/3102192450 wait complete. 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.076+0000 7f216787e740 20 client.19008 trim_cache size 0 max 16384 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.092+0000 7f216787e740 1 Processor -- start 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.092+0000 7f216787e740 1 -- start start 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.092+0000 7f216787e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa89e0 0x5566a0fbf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.092+0000 7f216787e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb3780 0x5566a0fcb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.092+0000 7f216787e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fb04b0 0x5566a0fcb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.092+0000 7f216787e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21601af780 con 0x5566a0fa89e0 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.092+0000 7f216787e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f21601ad050 con 0x5566a0fb3780 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.092+0000 7f216787e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21601aaea0 con 0x5566a0fb04b0 2024-09-16T00:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa89e0 0x5566a0fbf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa89e0 0x5566a0fbf7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38140/0 (socket says 172.21.15.33:38140) 2024-09-16T00:36:14.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f215f7fe640 1 -- 172.21.15.33:0/294626787 learned_addr learned my addr 172.21.15.33:0/294626787 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:14.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f2157fff640 1 --2- 172.21.15.33:0/294626787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb3780 0x5566a0fcb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f215f7fe640 1 --2- 172.21.15.33:0/294626787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fb04b0 0x5566a0fcb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f215f7fe640 1 -- 172.21.15.33:0/294626787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fb04b0 msgr2=0x5566a0fcb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:14.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f215f7fe640 1 --2- 172.21.15.33:0/294626787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fb04b0 0x5566a0fcb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f215f7fe640 1 -- 172.21.15.33:0/294626787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb3780 msgr2=0x5566a0fcb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:14.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f215f7fe640 1 --2- 172.21.15.33:0/294626787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb3780 0x5566a0fcb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f215f7fe640 1 -- 172.21.15.33:0/294626787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566a0fa9940 con 0x5566a0fa89e0 2024-09-16T00:36:14.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f215f7fe640 1 --2- 172.21.15.33:0/294626787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa89e0 0x5566a0fbf7f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f2158009fd0 tx=0x7f2158013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c52c42b4a50bed8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:14.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f2154ff9640 1 -- 172.21.15.33:0/294626787 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2158025040 con 0x5566a0fa89e0 2024-09-16T00:36:14.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f2154ff9640 1 -- 172.21.15.33:0/294626787 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2158026040 con 0x5566a0fa89e0 2024-09-16T00:36:14.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.093+0000 7f2154ff9640 1 -- 172.21.15.33:0/294626787 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215802e390 con 0x5566a0fa89e0 2024-09-16T00:36:14.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.094+0000 7f216787e740 1 -- 172.21.15.33:0/294626787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa89e0 msgr2=0x5566a0fbf7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:14.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.094+0000 7f216787e740 1 --2- 172.21.15.33:0/294626787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa89e0 0x5566a0fbf7f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f2158009fd0 tx=0x7f2158013040 comp rx=0 tx=0).stop 2024-09-16T00:36:14.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 -- 172.21.15.33:0/294626787 shutdown_connections 2024-09-16T00:36:14.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 --2- 172.21.15.33:0/294626787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fb04b0 0x5566a0fcb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 --2- 172.21.15.33:0/294626787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb3780 0x5566a0fcb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 --2- 172.21.15.33:0/294626787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fa89e0 0x5566a0fbf7f0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 -- 172.21.15.33:0/294626787 >> 172.21.15.33:0/294626787 conn(0x5566a0f6d310 msgr2=0x5566a0faf510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:14.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 -- 172.21.15.33:0/294626787 shutdown_connections 2024-09-16T00:36:14.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 -- 172.21.15.33:0/294626787 wait complete. 2024-09-16T00:36:14.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 -- start start 2024-09-16T00:36:14.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 Processor -- start 2024-09-16T00:36:14.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fa89e0 0x5566a0fb7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb04b0 0x5566a0fb7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fb3780 0x5566a102d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21601ab510 con 0x5566a0fb3780 2024-09-16T00:36:14.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f21601ac980 con 0x5566a0fb04b0 2024-09-16T00:36:14.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21601abbe0 con 0x5566a0fa89e0 2024-09-16T00:36:14.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.095+0000 7f216787e740 10 client.0 _ll_register_callbacks cb 0x7f2148009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:14.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fb3780 0x5566a102d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fa89e0 0x5566a0fb7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fa89e0 0x5566a0fb7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46010/0 (socket says 172.21.15.33:46010) 2024-09-16T00:36:14.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f215f7fe640 1 -- 172.21.15.33:0/3211676131 learned_addr learned my addr 172.21.15.33:0/3211676131 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:14.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f2157fff640 1 --2- 172.21.15.33:0/3211676131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb04b0 0x5566a0fb7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f215f7fe640 1 -- 172.21.15.33:0/3211676131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fa89e0 msgr2=0x5566a0fb7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:14.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f215f7fe640 1 --2- 172.21.15.33:0/3211676131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fa89e0 0x5566a0fb7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f215f7fe640 1 -- 172.21.15.33:0/3211676131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb04b0 msgr2=0x5566a0fb7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:14.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f215f7fe640 1 --2- 172.21.15.33:0/3211676131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb04b0 0x5566a0fb7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f215f7fe640 1 -- 172.21.15.33:0/3211676131 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566a102d740 con 0x5566a0fb3780 2024-09-16T00:36:14.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f215f7fe640 1 --2- 172.21.15.33:0/3211676131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fb3780 0x5566a102d200 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f2158009e10 tx=0x7f2158011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=559cc6f5f922c386 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:14.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f21467fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:14.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f21467fc640 1 -- 172.21.15.33:0/3211676131 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2158025040 con 0x5566a0fb3780 2024-09-16T00:36:14.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f21467fc640 1 -- 172.21.15.33:0/3211676131 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2158011d80 con 0x5566a0fb3780 2024-09-16T00:36:14.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.096+0000 7f216787e740 1 -- 172.21.15.33:0/3211676131 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2158026040 con 0x5566a0fb3780 2024-09-16T00:36:14.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.097+0000 7f21467fc640 1 -- 172.21.15.33:0/3211676131 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2158011560 con 0x5566a0fb3780 2024-09-16T00:36:14.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.097+0000 7f216787e740 10 client.19014 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:14.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.097+0000 7f216787e740 1 -- 172.21.15.33:0/3211676131 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5566a102e8d0 con 0x5566a0fb3780 2024-09-16T00:36:14.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.097+0000 7f21467fc640 1 -- 172.21.15.33:0/3211676131 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2158026040 con 0x5566a0fb3780 2024-09-16T00:36:14.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.097+0000 7f21467fc640 1 -- 172.21.15.33:0/3211676131 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f215801cce0 con 0x5566a0fb3780 2024-09-16T00:36:14.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.097+0000 7f216787e740 10 client.19014 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:14.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.097+0000 7f216787e740 10 client.19014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:36:14.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.097+0000 7f216787e740 20 client.19014 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:14.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.097+0000 7f216787e740 1 --2- 172.21.15.33:0/3211676131 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566a102ec50 0x5566a1031040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:14.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.097+0000 7f216787e740 4 client.19014 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:36:14.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.097+0000 7f216787e740 1 -- 172.21.15.33:0/3211676131 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5566a0f11d70 con 0x5566a102ec50 2024-09-16T00:36:14.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.097+0000 7f2157fff640 1 --2- 172.21.15.33:0/3211676131 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566a102ec50 0x5566a1031040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:14.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f21467fc640 10 client.19014 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:14.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f2157fff640 1 --2- 172.21.15.33:0/3211676131 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566a102ec50 0x5566a1031040 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1a8cc115b3c0db server_cookie=cb1944c6a382d5b2 in_seq=0 out_seq=0 2024-09-16T00:36:14.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f21467fc640 1 -- 172.21.15.33:0/3211676131 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5566a0f11d70 con 0x5566a102ec50 2024-09-16T00:36:14.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f21467fc640 10 client.19014 handle_command_reply: tid=0 2024-09-16T00:36:14.383 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:36:14.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f216787e740 1 client.19014 shutdown 2024-09-16T00:36:14.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f216787e740 1 -- 172.21.15.33:0/3211676131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fb3780 msgr2=0x5566a102d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:14.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f216787e740 1 --2- 172.21.15.33:0/3211676131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fb3780 0x5566a102d200 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f2158009e10 tx=0x7f2158011ad0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f216787e740 1 -- 172.21.15.33:0/3211676131 shutdown_connections 2024-09-16T00:36:14.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f216787e740 1 --2- 172.21.15.33:0/3211676131 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566a102ec50 0x5566a1031040 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f21467fc640 0 client.19014 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:14.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f216787e740 1 --2- 172.21.15.33:0/3211676131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566a0fb3780 0x5566a102d200 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f216787e740 1 --2- 172.21.15.33:0/3211676131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566a0fb04b0 0x5566a0fb7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f216787e740 1 --2- 172.21.15.33:0/3211676131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566a0fa89e0 0x5566a0fb7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f216787e740 1 -- 172.21.15.33:0/3211676131 >> 172.21.15.33:0/3211676131 conn(0x5566a0f6d310 msgr2=0x5566a0f9f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:14.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f216787e740 1 -- 172.21.15.33:0/3211676131 shutdown_connections 2024-09-16T00:36:14.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f216787e740 1 -- 172.21.15.33:0/3211676131 wait complete. 2024-09-16T00:36:14.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.098+0000 7f216787e740 20 client.19014 trim_cache size 0 max 16384 2024-09-16T00:36:14.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.099+0000 7f21654a5640 1 -- 172.21.15.33:0/691892437 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f214c0a32c0 msgr2=0x7f214c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:14.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.099+0000 7f21654a5640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f214c0a32c0 0x7f214c0a5780 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f21500c4d80 tx=0x7f21500f3ec0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.099+0000 7f21654a5640 1 -- 172.21.15.33:0/691892437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2160100e20 msgr2=0x7f2160198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.099+0000 7f21654a5640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2160100e20 0x7f2160198520 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f2150006010 tx=0x7f2150003c10 comp rx=0 tx=0).stop 2024-09-16T00:36:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.099+0000 7f21654a5640 1 -- 172.21.15.33:0/691892437 shutdown_connections 2024-09-16T00:36:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.099+0000 7f21654a5640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f214c0a32c0 0x7f214c0a5780 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.099+0000 7f21654a5640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102ef0 0x7f216019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.099+0000 7f21654a5640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2160102570 0x7f2160198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.099+0000 7f21654a5640 1 --2- 172.21.15.33:0/691892437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2160100e20 0x7f2160198520 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.099+0000 7f21654a5640 1 -- 172.21.15.33:0/691892437 >> 172.21.15.33:0/691892437 conn(0x7f21600fc7d0 msgr2=0x7f216006a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.100+0000 7f21654a5640 1 -- 172.21.15.33:0/691892437 shutdown_connections 2024-09-16T00:36:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:14.100+0000 7f21654a5640 1 -- 172.21.15.33:0/691892437 wait complete. 2024-09-16T00:36:15.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 113090 18.55 MB/sec execute 201 sec latency 72.501 ms 2024-09-16T00:36:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:15 smithi083 ceph-mon[35242]: pgmap v923: 129 pgs: 129 active+clean; 583 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 8.1 MiB/s wr, 629 op/s 2024-09-16T00:36:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:15 smithi060 ceph-mon[35185]: pgmap v923: 129 pgs: 129 active+clean; 583 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 8.1 MiB/s wr, 629 op/s 2024-09-16T00:36:16.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:15 smithi033 ceph-mon[31243]: pgmap v923: 129 pgs: 129 active+clean; 583 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 8.1 MiB/s wr, 629 op/s 2024-09-16T00:36:16.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 113665 18.62 MB/sec execute 202 sec latency 54.153 ms 2024-09-16T00:36:16.158 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:36:16.159 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:36:16.497 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:17.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 114149 18.66 MB/sec execute 203 sec latency 74.703 ms 2024-09-16T00:36:17.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.325+0000 7fd505c98640 1 -- 172.21.15.33:0/2555458156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd500100e20 msgr2=0x7fd500101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:17.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.325+0000 7fd505c98640 1 --2- 172.21.15.33:0/2555458156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd500100e20 0x7fd500101200 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0009930 tx=0x7fd4f002a4d0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.325+0000 7fd505c98640 1 -- 172.21.15.33:0/2555458156 shutdown_connections 2024-09-16T00:36:17.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.325+0000 7fd505c98640 1 --2- 172.21.15.33:0/2555458156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd500102ef0 0x7fd50010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.325+0000 7fd505c98640 1 --2- 172.21.15.33:0/2555458156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd500102570 0x7fd5001029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.325+0000 7fd505c98640 1 --2- 172.21.15.33:0/2555458156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd500100e20 0x7fd500101200 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.325+0000 7fd505c98640 1 -- 172.21.15.33:0/2555458156 >> 172.21.15.33:0/2555458156 conn(0x7fd5000fc7d0 msgr2=0x7fd5000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:17.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.325+0000 7fd505c98640 1 -- 172.21.15.33:0/2555458156 shutdown_connections 2024-09-16T00:36:17.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.325+0000 7fd505c98640 1 -- 172.21.15.33:0/2555458156 wait complete. 2024-09-16T00:36:17.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.326+0000 7fd505c98640 1 Processor -- start 2024-09-16T00:36:17.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.326+0000 7fd505c98640 1 -- start start 2024-09-16T00:36:17.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.326+0000 7fd505c98640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd500100e20 0x7fd50010d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.326+0000 7fd505c98640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd500102570 0x7fd50010aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.326+0000 7fd505c98640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd500102ef0 0x7fd50010b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.326+0000 7fd505c98640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001142a0 con 0x7fd500102570 2024-09-16T00:36:17.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.326+0000 7fd505c98640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd500114120 con 0x7fd500102ef0 2024-09-16T00:36:17.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.326+0000 7fd505c98640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd500114420 con 0x7fd500100e20 2024-09-16T00:36:17.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.327+0000 7fd4ff7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd500100e20 0x7fd50010d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.327+0000 7fd4ff7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd500100e20 0x7fd50010d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46032/0 (socket says 172.21.15.33:46032) 2024-09-16T00:36:17.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.327+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/2117903471 learned_addr learned my addr 172.21.15.33:0/2117903471 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:17.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.327+0000 7fd4feffd640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd500102570 0x7fd50010aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.327+0000 7fd4fffff640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd500102ef0 0x7fd50010b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.327+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/2117903471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd500102ef0 msgr2=0x7fd50010b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.327+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd500102ef0 0x7fd50010b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.327+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/2117903471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd500102570 msgr2=0x7fd50010aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.327+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd500102570 0x7fd50010aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.327+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/2117903471 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5001aafc0 con 0x7fd500100e20 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.327+0000 7fd4feffd640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd500102570 0x7fd50010aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.327+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd500100e20 0x7fd50010d970 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd4f002a9f0 tx=0x7fd4f0003b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.328+0000 7fd4fcff9640 1 -- 172.21.15.33:0/2117903471 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f003f070 con 0x7fd500100e20 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.328+0000 7fd4fcff9640 1 -- 172.21.15.33:0/2117903471 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4f0031c90 con 0x7fd500100e20 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.328+0000 7fd4fcff9640 1 -- 172.21.15.33:0/2117903471 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0035030 con 0x7fd500100e20 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.328+0000 7fd505c98640 1 -- 172.21.15.33:0/2117903471 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5001ab190 con 0x7fd500100e20 2024-09-16T00:36:17.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.328+0000 7fd505c98640 1 -- 172.21.15.33:0/2117903471 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5001ab5f0 con 0x7fd500100e20 2024-09-16T00:36:17.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.331+0000 7fd4fcff9640 1 -- 172.21.15.33:0/2117903471 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd4f0042070 con 0x7fd500100e20 2024-09-16T00:36:17.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd508071740 1 Processor -- start 2024-09-16T00:36:17.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd508071740 1 -- start start 2024-09-16T00:36:17.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd4fcff9640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd4e80a32c0 0x7fd4e80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd508071740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 0x55681053ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd508071740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 0x556810549800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd508071740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810549d40 0x55681054c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd4fffff640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd4e80a32c0 0x7fd4e80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd508071740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5568104a4280 con 0x55681053eb60 2024-09-16T00:36:17.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd508071740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55681044bb10 con 0x55681053f480 2024-09-16T00:36:17.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd508071740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5568104a0240 con 0x556810549d40 2024-09-16T00:36:17.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd4fcff9640 1 -- 172.21.15.33:0/2117903471 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd4f00f03d0 con 0x7fd500100e20 2024-09-16T00:36:17.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd4ff7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 0x55681053ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd4ff7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 0x55681053ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38178/0 (socket says 172.21.15.33:38178) 2024-09-16T00:36:17.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/415138374 learned_addr learned my addr 172.21.15.33:0/415138374 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:17.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.332+0000 7fd4feffd640 1 --2- 172.21.15.33:0/415138374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 0x556810549800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.333+0000 7fd4fffff640 1 --2- 172.21.15.33:0/415138374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810549d40 0x55681054c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.333+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/415138374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810549d40 msgr2=0x55681054c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:17.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.333+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/415138374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810549d40 0x55681054c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.333+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/415138374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 msgr2=0x556810549800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:17.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.333+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/415138374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 0x556810549800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.333+0000 7fd4fffff640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd4e80a32c0 0x7fd4e80a5780 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8005fd0 tx=0x7fd4f800b710 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:17.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.333+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/415138374 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568104581f0 con 0x55681053eb60 2024-09-16T00:36:17.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.333+0000 7fd4feffd640 1 --2- 172.21.15.33:0/415138374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 0x556810549800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:17.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.333+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/415138374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 0x55681053ef40 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fd4f002aa90 tx=0x7fd4f003b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f422a4dc4e436 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:17.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.334+0000 7fd4e5ffb640 1 -- 172.21.15.33:0/415138374 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0049070 con 0x55681053eb60 2024-09-16T00:36:17.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.334+0000 7fd4e5ffb640 1 -- 172.21.15.33:0/415138374 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4f002bd00 con 0x55681053eb60 2024-09-16T00:36:17.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.334+0000 7fd4e5ffb640 1 -- 172.21.15.33:0/415138374 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f00bb870 con 0x55681053eb60 2024-09-16T00:36:17.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.335+0000 7fd508071740 1 -- 172.21.15.33:0/415138374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 msgr2=0x55681053ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:17.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.335+0000 7fd508071740 1 --2- 172.21.15.33:0/415138374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 0x55681053ef40 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fd4f002aa90 tx=0x7fd4f003b720 comp rx=0 tx=0).stop 2024-09-16T00:36:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.335+0000 7fd508071740 1 -- 172.21.15.33:0/415138374 shutdown_connections 2024-09-16T00:36:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.335+0000 7fd508071740 1 --2- 172.21.15.33:0/415138374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810549d40 0x55681054c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.335+0000 7fd508071740 1 --2- 172.21.15.33:0/415138374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 0x556810549800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.335+0000 7fd508071740 1 --2- 172.21.15.33:0/415138374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 0x55681053ef40 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.335+0000 7fd508071740 1 -- 172.21.15.33:0/415138374 >> 172.21.15.33:0/415138374 conn(0x55681053b0f0 msgr2=0x55681053b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.336+0000 7fd508071740 1 -- 172.21.15.33:0/415138374 shutdown_connections 2024-09-16T00:36:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.336+0000 7fd508071740 1 -- 172.21.15.33:0/415138374 wait complete. 2024-09-16T00:36:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.336+0000 7fd508071740 1 -- start start 2024-09-16T00:36:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.336+0000 7fd508071740 1 Processor -- start 2024-09-16T00:36:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd508071740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053eb60 0x556810546bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd508071740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 0x556810547110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd508071740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556810549d40 0x556810547650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd4ff7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053eb60 0x556810546bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd4ff7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053eb60 0x556810546bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46038/0 (socket says 172.21.15.33:46038) 2024-09-16T00:36:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd4fffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556810549d40 0x556810547650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd4fffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556810549d40 0x556810547650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38188/0 (socket says 172.21.15.33:38188) 2024-09-16T00:36:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd4feffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 0x556810547110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/2712954095 learned_addr learned my addr 172.21.15.33:0/2712954095 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd4feffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 0x556810547110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:39586/0 (socket says 172.21.15.33:39586) 2024-09-16T00:36:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd508071740 1 -- 172.21.15.33:0/2712954095 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55681054c6d0 con 0x556810549d40 2024-09-16T00:36:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd508071740 1 -- 172.21.15.33:0/2712954095 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55681045ef10 con 0x55681053f480 2024-09-16T00:36:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.337+0000 7fd508071740 1 -- 172.21.15.33:0/2712954095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55681054c850 con 0x55681053eb60 2024-09-16T00:36:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.338+0000 7fd508071740 10 client.0 _ll_register_callbacks cb 0x556810427a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.338+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/2712954095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 msgr2=0x556810547110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.338+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/2712954095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 0x556810547110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.338+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/2712954095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556810549d40 msgr2=0x556810547650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.338+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/2712954095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556810549d40 0x556810547650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.338+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/2712954095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568102156a0 con 0x55681053eb60 2024-09-16T00:36:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.338+0000 7fd4feffd640 1 --2- 172.21.15.33:0/2712954095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 0x556810547110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:36:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.339+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/2712954095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053eb60 0x556810546bd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00b9980 tx=0x7fd4f0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89e89b3d64afb148 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.339+0000 7fd4deffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:36:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.339+0000 7fd4deffd640 1 -- 172.21.15.33:0/2712954095 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0049040 con 0x55681053eb60 2024-09-16T00:36:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.339+0000 7fd4deffd640 1 -- 172.21.15.33:0/2712954095 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd4f003b460 con 0x55681053eb60 2024-09-16T00:36:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.339+0000 7fd508071740 1 -- 172.21.15.33:0/2712954095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4f002bd00 con 0x55681053eb60 2024-09-16T00:36:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.339+0000 7fd4deffd640 1 -- 172.21.15.33:0/2712954095 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd4f00bd570 con 0x55681053eb60 2024-09-16T00:36:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.339+0000 7fd508071740 10 client.28279 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.339+0000 7fd508071740 1 -- 172.21.15.33:0/2712954095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55681054c6d0 con 0x55681053eb60 2024-09-16T00:36:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.340+0000 7fd4deffd640 1 -- 172.21.15.33:0/2712954095 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4f002bd00 con 0x55681053eb60 2024-09-16T00:36:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.340+0000 7fd4deffd640 1 -- 172.21.15.33:0/2712954095 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fd4f00d6260 con 0x55681053eb60 2024-09-16T00:36:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.340+0000 7fd508071740 10 client.28279 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.340+0000 7fd508071740 10 client.28279 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:36:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.340+0000 7fd508071740 20 client.28279 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.340+0000 7fd508071740 1 --2- 172.21.15.33:0/2712954095 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556810559370 0x55681055b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.340+0000 7fd508071740 4 client.28279 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:36:17.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.340+0000 7fd508071740 1 -- 172.21.15.33:0/2712954095 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568104b17f0 con 0x556810559370 2024-09-16T00:36:17.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.340+0000 7fd4feffd640 1 --2- 172.21.15.33:0/2712954095 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556810559370 0x55681055b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.341+0000 7fd4feffd640 1 --2- 172.21.15.33:0/2712954095 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556810559370 0x55681055b760 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18188705842f3275 server_cookie=3992ee7a54825d44 in_seq=0 out_seq=0 2024-09-16T00:36:17.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.341+0000 7fd4deffd640 10 client.28279 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:17.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.343+0000 7fd4deffd640 1 -- 172.21.15.33:0/2712954095 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5568104b17f0 con 0x556810559370 2024-09-16T00:36:17.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.343+0000 7fd4deffd640 10 client.28279 handle_command_reply: tid=0 2024-09-16T00:36:17.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.343+0000 7fd508071740 1 client.28279 shutdown 2024-09-16T00:36:17.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.344+0000 7fd508071740 1 -- 172.21.15.33:0/2712954095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053eb60 msgr2=0x556810546bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:17.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.344+0000 7fd508071740 1 --2- 172.21.15.33:0/2712954095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053eb60 0x556810546bd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00b9980 tx=0x7fd4f0009260 comp rx=0 tx=0).stop 2024-09-16T00:36:17.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.344+0000 7fd508071740 1 -- 172.21.15.33:0/2712954095 shutdown_connections 2024-09-16T00:36:17.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.344+0000 7fd508071740 1 --2- 172.21.15.33:0/2712954095 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556810559370 0x55681055b760 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.344+0000 7fd508071740 1 --2- 172.21.15.33:0/2712954095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556810549d40 0x556810547650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.344+0000 7fd508071740 1 --2- 172.21.15.33:0/2712954095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053f480 0x556810547110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.344+0000 7fd4deffd640 0 client.28279 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:17.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.344+0000 7fd508071740 1 --2- 172.21.15.33:0/2712954095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053eb60 0x556810546bd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.344+0000 7fd508071740 1 -- 172.21.15.33:0/2712954095 >> 172.21.15.33:0/2712954095 conn(0x55681053b0f0 msgr2=0x556810540e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:17.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.344+0000 7fd508071740 1 -- 172.21.15.33:0/2712954095 shutdown_connections 2024-09-16T00:36:17.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.344+0000 7fd508071740 1 -- 172.21.15.33:0/2712954095 wait complete. 2024-09-16T00:36:17.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.344+0000 7fd508071740 20 client.28279 trim_cache size 0 max 16384 2024-09-16T00:36:17.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.365+0000 7fd508071740 1 Processor -- start 2024-09-16T00:36:17.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.365+0000 7fd508071740 1 -- start start 2024-09-16T00:36:17.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd508071740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 0x5568105415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd508071740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053f480 0x556810560f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd508071740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556810549d40 0x5568105614b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd508071740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001b8160 con 0x55681053eb60 2024-09-16T00:36:17.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd508071740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001b5a30 con 0x556810549d40 2024-09-16T00:36:17.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd508071740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001b3880 con 0x55681053f480 2024-09-16T00:36:17.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd4ff7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 0x5568105415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd4feffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053f480 0x556810560f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd4ff7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 0x5568105415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38192/0 (socket says 172.21.15.33:38192) 2024-09-16T00:36:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd4ff7fe640 1 -- 172.21.15.33:0/3262409598 learned_addr learned my addr 172.21.15.33:0/3262409598 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd4feffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053f480 0x556810560f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46052/0 (socket says 172.21.15.33:46052) 2024-09-16T00:36:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd4feffd640 1 -- 172.21.15.33:0/3262409598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556810549d40 msgr2=0x5568105614b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd4feffd640 1 --2- 172.21.15.33:0/3262409598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556810549d40 0x5568105614b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd4feffd640 1 -- 172.21.15.33:0/3262409598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 msgr2=0x5568105415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd4feffd640 1 --2- 172.21.15.33:0/3262409598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 0x5568105415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd4feffd640 1 -- 172.21.15.33:0/3262409598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568105491d0 con 0x55681053f480 2024-09-16T00:36:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/3262409598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 0x5568105415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.366+0000 7fd4feffd640 1 --2- 172.21.15.33:0/3262409598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053f480 0x556810560f70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x556810548210 tx=0x7fd4f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41da628fc9913620 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.367+0000 7fd4e57fa640 1 -- 172.21.15.33:0/3262409598 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f40037f0 con 0x55681053f480 2024-09-16T00:36:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.367+0000 7fd4e57fa640 1 -- 172.21.15.33:0/3262409598 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4f4004490 con 0x55681053f480 2024-09-16T00:36:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.367+0000 7fd4e57fa640 1 -- 172.21.15.33:0/3262409598 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4026e40 con 0x55681053f480 2024-09-16T00:36:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.368+0000 7fd508071740 1 -- 172.21.15.33:0/3262409598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053f480 msgr2=0x556810560f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.368+0000 7fd508071740 1 --2- 172.21.15.33:0/3262409598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053f480 0x556810560f70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x556810548210 tx=0x7fd4f40046c0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.368+0000 7fd508071740 1 -- 172.21.15.33:0/3262409598 shutdown_connections 2024-09-16T00:36:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.368+0000 7fd508071740 1 --2- 172.21.15.33:0/3262409598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556810549d40 0x5568105614b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.368+0000 7fd508071740 1 --2- 172.21.15.33:0/3262409598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55681053f480 0x556810560f70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.368+0000 7fd508071740 1 --2- 172.21.15.33:0/3262409598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053eb60 0x5568105415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.368+0000 7fd508071740 1 -- 172.21.15.33:0/3262409598 >> 172.21.15.33:0/3262409598 conn(0x55681053b0f0 msgr2=0x55681053a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.368+0000 7fd508071740 1 -- 172.21.15.33:0/3262409598 shutdown_connections 2024-09-16T00:36:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.368+0000 7fd508071740 1 -- 172.21.15.33:0/3262409598 wait complete. 2024-09-16T00:36:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.368+0000 7fd508071740 1 -- start start 2024-09-16T00:36:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd508071740 1 Processor -- start 2024-09-16T00:36:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd508071740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053eb60 0x556810551b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd508071740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053f480 0x556810552040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd508071740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810549d40 0x55681054e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd508071740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001b3ef0 con 0x55681053f480 2024-09-16T00:36:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd508071740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001b5360 con 0x55681053eb60 2024-09-16T00:36:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd508071740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001b45c0 con 0x556810549d40 2024-09-16T00:36:17.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd508071740 10 client.0 _ll_register_callbacks cb 0x7fd4f00b9810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:17.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd4fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810549d40 0x55681054e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd4ff7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053eb60 0x556810551b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd4feffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053f480 0x556810552040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd4ff7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053eb60 0x556810551b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:39602/0 (socket says 172.21.15.33:39602) 2024-09-16T00:36:17.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd4fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810549d40 0x55681054e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:46058/0 (socket says 172.21.15.33:46058) 2024-09-16T00:36:17.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd4fffff640 1 -- 172.21.15.33:0/1086053732 learned_addr learned my addr 172.21.15.33:0/1086053732 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:17.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.369+0000 7fd4feffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053f480 0x556810552040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38206/0 (socket says 172.21.15.33:38206) 2024-09-16T00:36:17.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.370+0000 7fd4fffff640 1 -- 172.21.15.33:0/1086053732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053eb60 msgr2=0x556810551b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:17.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.370+0000 7fd4fffff640 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053eb60 0x556810551b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.370+0000 7fd4fffff640 1 -- 172.21.15.33:0/1086053732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053f480 msgr2=0x556810552040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:17.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.370+0000 7fd4fffff640 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053f480 0x556810552040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.370+0000 7fd4fffff640 1 -- 172.21.15.33:0/1086053732 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55681055bad0 con 0x556810549d40 2024-09-16T00:36:17.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.370+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053eb60 0x556810551b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:17.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.370+0000 7fd4feffd640 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053f480 0x556810552040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:17.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.370+0000 7fd4fffff640 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810549d40 0x55681054e8f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800bfd0 tx=0x7fd4f8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77737e5d210de0a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:17.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.370+0000 7fd4deffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:36:17.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.370+0000 7fd4deffd640 1 -- 172.21.15.33:0/1086053732 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8028070 con 0x556810549d40 2024-09-16T00:36:17.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.370+0000 7fd4deffd640 1 -- 172.21.15.33:0/1086053732 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd4f8004b20 con 0x556810549d40 2024-09-16T00:36:17.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.370+0000 7fd508071740 1 -- 172.21.15.33:0/1086053732 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4f4004490 con 0x556810549d40 2024-09-16T00:36:17.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.371+0000 7fd4deffd640 1 -- 172.21.15.33:0/1086053732 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd4f8033050 con 0x556810549d40 2024-09-16T00:36:17.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.371+0000 7fd508071740 10 client.28285 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:17.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.371+0000 7fd508071740 1 -- 172.21.15.33:0/1086053732 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5568105c4790 con 0x556810549d40 2024-09-16T00:36:17.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.371+0000 7fd4deffd640 1 -- 172.21.15.33:0/1086053732 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4f4004490 con 0x556810549d40 2024-09-16T00:36:17.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.371+0000 7fd4deffd640 1 -- 172.21.15.33:0/1086053732 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fd4f8024b90 con 0x556810549d40 2024-09-16T00:36:17.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.371+0000 7fd508071740 10 client.28285 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:17.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.371+0000 7fd508071740 10 client.28285 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:36:17.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.371+0000 7fd508071740 20 client.28285 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:17.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.371+0000 7fd508071740 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568105c4eb0 0x5568105c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:17.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.371+0000 7fd508071740 4 client.28285 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:36:17.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.371+0000 7fd508071740 1 -- 172.21.15.33:0/1086053732 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5568104abd00 con 0x5568105c4eb0 2024-09-16T00:36:17.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.373+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568105c4eb0 0x5568105c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:17.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.374+0000 7fd4deffd640 10 client.28285 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:17.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.374+0000 7fd4ff7fe640 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568105c4eb0 0x5568105c72d0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ec2dbd29ad1ebcb server_cookie=7b907158a1c315b8 in_seq=0 out_seq=0 2024-09-16T00:36:17.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.376+0000 7fd4deffd640 1 -- 172.21.15.33:0/1086053732 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5568104abd00 con 0x5568105c4eb0 2024-09-16T00:36:17.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.376+0000 7fd4deffd640 10 client.28285 handle_command_reply: tid=0 2024-09-16T00:36:17.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.376+0000 7fd508071740 1 client.28285 shutdown 2024-09-16T00:36:17.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.376+0000 7fd508071740 1 -- 172.21.15.33:0/1086053732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810549d40 msgr2=0x55681054e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:17.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.376+0000 7fd508071740 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810549d40 0x55681054e8f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800bfd0 tx=0x7fd4f8004a40 comp rx=0 tx=0).stop 2024-09-16T00:36:17.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.376+0000 7fd508071740 1 -- 172.21.15.33:0/1086053732 shutdown_connections 2024-09-16T00:36:17.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.376+0000 7fd508071740 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5568105c4eb0 0x5568105c72d0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.376+0000 7fd508071740 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810549d40 0x55681054e8f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.376+0000 7fd508071740 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55681053f480 0x556810552040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.376+0000 7fd508071740 1 --2- 172.21.15.33:0/1086053732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55681053eb60 0x556810551b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.376+0000 7fd508071740 1 -- 172.21.15.33:0/1086053732 >> 172.21.15.33:0/1086053732 conn(0x55681053b0f0 msgr2=0x556810554870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:17.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.376+0000 7fd4deffd640 0 client.28285 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:17.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.377+0000 7fd508071740 1 -- 172.21.15.33:0/1086053732 shutdown_connections 2024-09-16T00:36:17.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.377+0000 7fd508071740 1 -- 172.21.15.33:0/1086053732 wait complete. 2024-09-16T00:36:17.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.377+0000 7fd508071740 20 client.28285 trim_cache size 0 max 16384 2024-09-16T00:36:17.381 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:36:17.381 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:36:17.382 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "670f9148-f975-45ae-b1d5-d26fec546026", 2024-09-16T00:36:17.382 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:36:17.382 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:36:17.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.378+0000 7fd505c98640 1 -- 172.21.15.33:0/2117903471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd4e80a32c0 msgr2=0x7fd4e80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:17.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.378+0000 7fd505c98640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd4e80a32c0 0x7fd4e80a5780 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8005fd0 tx=0x7fd4f800b710 comp rx=0 tx=0).stop 2024-09-16T00:36:17.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.378+0000 7fd505c98640 1 -- 172.21.15.33:0/2117903471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd500100e20 msgr2=0x7fd50010d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:17.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.378+0000 7fd505c98640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd500100e20 0x7fd50010d970 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd4f002a9f0 tx=0x7fd4f0003b20 comp rx=0 tx=0).stop 2024-09-16T00:36:17.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.378+0000 7fd505c98640 1 -- 172.21.15.33:0/2117903471 shutdown_connections 2024-09-16T00:36:17.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.378+0000 7fd505c98640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd4e80a32c0 0x7fd4e80a5780 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.378+0000 7fd505c98640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd500102ef0 0x7fd50010b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.378+0000 7fd505c98640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd500102570 0x7fd50010aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.378+0000 7fd505c98640 1 --2- 172.21.15.33:0/2117903471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd500100e20 0x7fd50010d970 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:17.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.378+0000 7fd505c98640 1 -- 172.21.15.33:0/2117903471 >> 172.21.15.33:0/2117903471 conn(0x7fd5000fc7d0 msgr2=0x7fd5001048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:17.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.378+0000 7fd505c98640 1 -- 172.21.15.33:0/2117903471 shutdown_connections 2024-09-16T00:36:17.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:17.378+0000 7fd505c98640 1 -- 172.21.15.33:0/2117903471 wait complete. 2024-09-16T00:36:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: pgmap v924: 129 pgs: 129 active+clean; 589 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 5.2 MiB/s wr, 531 op/s 2024-09-16T00:36:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:36:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:36:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:36:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:36:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:36:17.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:36:17.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:36:17.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:36:17.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:36:17.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:36:17.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:36:17.922 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.922 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:36:17.922 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:17 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: pgmap v924: 129 pgs: 129 active+clean; 589 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 5.2 MiB/s wr, 531 op/s 2024-09-16T00:36:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:36:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:36:17.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:36:17.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:36:17.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:17.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:36:17.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:17 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:18.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: pgmap v924: 129 pgs: 129 active+clean; 589 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 5.2 MiB/s wr, 531 op/s 2024-09-16T00:36:18.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:36:18.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:18.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:36:18.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:36:18.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:36:18.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:18.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:36:18.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:18.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:36:18.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:17 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:18.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 114642 18.68 MB/sec execute 204 sec latency 65.815 ms 2024-09-16T00:36:19.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 115147 18.76 MB/sec execute 205 sec latency 278.405 ms 2024-09-16T00:36:19.298 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:36:19.609 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:19 smithi083 ceph-mon[35242]: pgmap v925: 129 pgs: 129 active+clean; 589 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.6 MiB/s wr, 334 op/s 2024-09-16T00:36:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:19 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:19.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:19 smithi060 ceph-mon[35185]: pgmap v925: 129 pgs: 129 active+clean; 589 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.6 MiB/s wr, 334 op/s 2024-09-16T00:36:19.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:19 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:20.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:19 smithi033 ceph-mon[31243]: pgmap v925: 129 pgs: 129 active+clean; 589 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.6 MiB/s wr, 334 op/s 2024-09-16T00:36:20.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:19 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:36:20.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 115712 18.78 MB/sec execute 206 sec latency 77.787 ms 2024-09-16T00:36:20.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.812+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4161584409 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7ec100930 msgr2=0x7fe7ec100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:20.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.812+0000 7fe7f44d9640 1 --2- 172.21.15.33:0/4161584409 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7ec100930 0x7fe7ec100d10 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009900 tx=0x7fe7e802a410 comp rx=0 tx=0).stop 2024-09-16T00:36:20.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.812+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4161584409 shutdown_connections 2024-09-16T00:36:20.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.812+0000 7fe7f44d9640 1 --2- 172.21.15.33:0/4161584409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec10e4e0 0x7fe7ec1108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.812+0000 7fe7f44d9640 1 --2- 172.21.15.33:0/4161584409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7ec1012e0 0x7fe7ec10dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.812+0000 7fe7f44d9640 1 --2- 172.21.15.33:0/4161584409 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7ec100930 0x7fe7ec100d10 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.812+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4161584409 >> 172.21.15.33:0/4161584409 conn(0x7fe7ec0fc7a0 msgr2=0x7fe7ec0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:20.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.812+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4161584409 shutdown_connections 2024-09-16T00:36:20.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.812+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4161584409 wait complete. 2024-09-16T00:36:20.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f44d9640 1 Processor -- start 2024-09-16T00:36:20.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f44d9640 1 -- start start 2024-09-16T00:36:20.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f44d9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7ec100930 0x7fe7ec19a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f44d9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec1012e0 0x7fe7ec19abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f44d9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7ec10e4e0 0x7fe7ec19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f44d9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec114180 con 0x7fe7ec10e4e0 2024-09-16T00:36:20.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f44d9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec114000 con 0x7fe7ec100930 2024-09-16T00:36:20.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f44d9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec114300 con 0x7fe7ec1012e0 2024-09-16T00:36:20.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f1a74640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec1012e0 0x7fe7ec19abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f2275640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7ec100930 0x7fe7ec19a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f1a74640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec1012e0 0x7fe7ec19abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59530/0 (socket says 172.21.15.33:59530) 2024-09-16T00:36:20.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f1a74640 1 -- 172.21.15.33:0/4064356311 learned_addr learned my addr 172.21.15.33:0/4064356311 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:20.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.813+0000 7fe7f2a76640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7ec10e4e0 0x7fe7ec19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.814+0000 7fe7f1a74640 1 -- 172.21.15.33:0/4064356311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7ec100930 msgr2=0x7fe7ec19a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:20.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.814+0000 7fe7f1a74640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7ec100930 0x7fe7ec19a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.814+0000 7fe7f1a74640 1 -- 172.21.15.33:0/4064356311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7ec10e4e0 msgr2=0x7fe7ec19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:20.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.814+0000 7fe7f1a74640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7ec10e4e0 0x7fe7ec19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.814+0000 7fe7f1a74640 1 -- 172.21.15.33:0/4064356311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ec19f630 con 0x7fe7ec1012e0 2024-09-16T00:36:20.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.814+0000 7fe7f1a74640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec1012e0 0x7fe7ec19abc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00d950 tx=0x7fe7dc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:20.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.814+0000 7fe7e37fe640 1 -- 172.21.15.33:0/4064356311 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc017030 con 0x7fe7ec1012e0 2024-09-16T00:36:20.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.814+0000 7fe7e37fe640 1 -- 172.21.15.33:0/4064356311 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7dc004020 con 0x7fe7ec1012e0 2024-09-16T00:36:20.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.814+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4064356311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7ec19f920 con 0x7fe7ec1012e0 2024-09-16T00:36:20.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.814+0000 7fe7e37fe640 1 -- 172.21.15.33:0/4064356311 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc0047a0 con 0x7fe7ec1012e0 2024-09-16T00:36:20.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.814+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4064356311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7ec1a7200 con 0x7fe7ec1012e0 2024-09-16T00:36:20.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.817+0000 7fe7f68b2740 1 Processor -- start 2024-09-16T00:36:20.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.817+0000 7fe7f68b2740 1 -- start start 2024-09-16T00:36:20.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f68b2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd6f4a0 0x55c56dd6f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f68b2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 0x55c56dd7a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f68b2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd7a680 0x55c56dd7ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f68b2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c56dcd52e0 con 0x55c56dd6f4a0 2024-09-16T00:36:20.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f68b2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c56dc7cce0 con 0x55c56dd7a680 2024-09-16T00:36:20.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f68b2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c56dcd12a0 con 0x55c56dd6fdc0 2024-09-16T00:36:20.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f2275640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 0x55c56dd7a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7e37fe640 1 -- 172.21.15.33:0/4064356311 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe7dc0041d0 con 0x7fe7ec1012e0 2024-09-16T00:36:20.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f2275640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 0x55c56dd7a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59532/0 (socket says 172.21.15.33:59532) 2024-09-16T00:36:20.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f2275640 1 -- 172.21.15.33:0/1524066678 learned_addr learned my addr 172.21.15.33:0/1524066678 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:20.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f1a74640 1 --2- 172.21.15.33:0/1524066678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd7a680 0x55c56dd7ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f2275640 1 -- 172.21.15.33:0/1524066678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd7a680 msgr2=0x55c56dd7ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:20.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f2275640 1 --2- 172.21.15.33:0/1524066678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd7a680 0x55c56dd7ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f2275640 1 -- 172.21.15.33:0/1524066678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd6f4a0 msgr2=0x55c56dd6f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:36:20.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f2275640 1 --2- 172.21.15.33:0/1524066678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd6f4a0 0x55c56dd6f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.818+0000 7fe7f2275640 1 -- 172.21.15.33:0/1524066678 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c56dc61440 con 0x55c56dd6fdc0 2024-09-16T00:36:20.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.819+0000 7fe7f2275640 1 --2- 172.21.15.33:0/1524066678 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 0x55c56dd7a140 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009900 tx=0x7fe7e8005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7649496731c0efff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:20.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.819+0000 7fe7e0ff9640 1 -- 172.21.15.33:0/1524066678 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8040070 con 0x55c56dd6fdc0 2024-09-16T00:36:20.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.819+0000 7fe7e37fe640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe7d80a3240 0x7fe7d80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.819+0000 7fe7e0ff9640 1 -- 172.21.15.33:0/1524066678 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e80318f0 con 0x55c56dd6fdc0 2024-09-16T00:36:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.819+0000 7fe7e0ff9640 1 -- 172.21.15.33:0/1524066678 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8035040 con 0x55c56dd6fdc0 2024-09-16T00:36:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.819+0000 7fe7e37fe640 1 -- 172.21.15.33:0/4064356311 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe7dc0cf6c0 con 0x7fe7ec1012e0 2024-09-16T00:36:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.819+0000 7fe7f2a76640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe7d80a3240 0x7fe7d80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.820+0000 7fe7f68b2740 1 -- 172.21.15.33:0/1524066678 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 msgr2=0x55c56dd7a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.820+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/1524066678 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 0x55c56dd7a140 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009900 tx=0x7fe7e8005e00 comp rx=0 tx=0).stop 2024-09-16T00:36:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.820+0000 7fe7f2a76640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe7d80a3240 0x7fe7d80a5700 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec102b10 tx=0x7fe7e4009a90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.820+0000 7fe7f68b2740 1 -- 172.21.15.33:0/1524066678 shutdown_connections 2024-09-16T00:36:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.820+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/1524066678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd7a680 0x55c56dd7ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.820+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/1524066678 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 0x55c56dd7a140 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.820+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/1524066678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd6f4a0 0x55c56dd6f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.820+0000 7fe7f68b2740 1 -- 172.21.15.33:0/1524066678 >> 172.21.15.33:0/1524066678 conn(0x55c56dd6ba30 msgr2=0x55c56dd6be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.820+0000 7fe7f68b2740 1 -- 172.21.15.33:0/1524066678 shutdown_connections 2024-09-16T00:36:20.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.820+0000 7fe7f68b2740 1 -- 172.21.15.33:0/1524066678 wait complete. 2024-09-16T00:36:20.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f68b2740 1 -- start start 2024-09-16T00:36:20.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f68b2740 1 Processor -- start 2024-09-16T00:36:20.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f68b2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 0x55c56dd79f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f68b2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd6fdc0 0x55c56dd77230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f68b2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd7a680 0x55c56dd77770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f68b2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c56dd7d250 con 0x55c56dd7a680 2024-09-16T00:36:20.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f68b2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c56dc90050 con 0x55c56dd6fdc0 2024-09-16T00:36:20.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f68b2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c56dd7d3d0 con 0x55c56dd6f4a0 2024-09-16T00:36:20.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f68b2740 10 client.0 _ll_register_callbacks cb 0x55c56dcd0160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:20.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f2275640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 0x55c56dd79f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f2275640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 0x55c56dd79f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59544/0 (socket says 172.21.15.33:59544) 2024-09-16T00:36:20.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f2275640 1 -- 172.21.15.33:0/4172183046 learned_addr learned my addr 172.21.15.33:0/4172183046 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:20.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f2a76640 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd7a680 0x55c56dd77770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.821+0000 7fe7f1a74640 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd6fdc0 0x55c56dd77230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.822+0000 7fe7f1a74640 1 -- 172.21.15.33:0/4172183046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 msgr2=0x55c56dd79f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:20.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.822+0000 7fe7f1a74640 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 0x55c56dd79f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.822+0000 7fe7f1a74640 1 -- 172.21.15.33:0/4172183046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd7a680 msgr2=0x55c56dd77770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:20.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.822+0000 7fe7f1a74640 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd7a680 0x55c56dd77770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.822+0000 7fe7f1a74640 1 -- 172.21.15.33:0/4172183046 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c56da463c0 con 0x55c56dd6fdc0 2024-09-16T00:36:20.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.822+0000 7fe7f1a74640 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd6fdc0 0x55c56dd77230 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0ccd90 tx=0x7fe7dc09b3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=879f6e9a49b11d2c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:20.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.822+0000 7fe7d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:36:20.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.822+0000 7fe7d5ffb640 1 -- 172.21.15.33:0/4172183046 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc086070 con 0x55c56dd6fdc0 2024-09-16T00:36:20.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.822+0000 7fe7d5ffb640 1 -- 172.21.15.33:0/4172183046 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe7dc0cf300 con 0x55c56dd6fdc0 2024-09-16T00:36:20.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.822+0000 7fe7f68b2740 1 -- 172.21.15.33:0/4172183046 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7e80318f0 con 0x55c56dd6fdc0 2024-09-16T00:36:20.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.823+0000 7fe7d5ffb640 1 -- 172.21.15.33:0/4172183046 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe7dc0cf500 con 0x55c56dd6fdc0 2024-09-16T00:36:20.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.823+0000 7fe7f68b2740 10 client.28307 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:20.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.823+0000 7fe7f68b2740 1 -- 172.21.15.33:0/4172183046 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c56dd913d0 con 0x55c56dd6fdc0 2024-09-16T00:36:20.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.823+0000 7fe7d5ffb640 1 -- 172.21.15.33:0/4172183046 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7dc09b860 con 0x55c56dd6fdc0 2024-09-16T00:36:20.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.823+0000 7fe7d5ffb640 1 -- 172.21.15.33:0/4172183046 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fe7dc09cb90 con 0x55c56dd6fdc0 2024-09-16T00:36:20.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.823+0000 7fe7f68b2740 10 client.28307 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:20.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.823+0000 7fe7f68b2740 10 client.28307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:36:20.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.823+0000 7fe7f68b2740 20 client.28307 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:20.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.823+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c56dd927e0 0x55c56dd94bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.823+0000 7fe7f68b2740 4 client.28307 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:36:20.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.823+0000 7fe7f68b2740 1 -- 172.21.15.33:0/4172183046 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe7e80318f0 con 0x55c56dd927e0 2024-09-16T00:36:20.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.823+0000 7fe7f2275640 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c56dd927e0 0x55c56dd94bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.824+0000 7fe7d5ffb640 10 client.28307 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:20.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.824+0000 7fe7f2275640 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c56dd927e0 0x55c56dd94bd0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc27fc74d9c2ad9b server_cookie=7f5fb4eecb4264fc in_seq=0 out_seq=0 2024-09-16T00:36:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.825+0000 7fe7d5ffb640 1 -- 172.21.15.33:0/4172183046 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fe7e80318f0 con 0x55c56dd927e0 2024-09-16T00:36:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.825+0000 7fe7d5ffb640 10 client.28307 handle_command_reply: tid=0 2024-09-16T00:36:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.825+0000 7fe7f68b2740 1 client.28307 shutdown 2024-09-16T00:36:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.825+0000 7fe7f68b2740 1 -- 172.21.15.33:0/4172183046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd6fdc0 msgr2=0x55c56dd77230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.825+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd6fdc0 0x55c56dd77230 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0ccd90 tx=0x7fe7dc09b3a0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.825+0000 7fe7f68b2740 1 -- 172.21.15.33:0/4172183046 shutdown_connections 2024-09-16T00:36:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.825+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c56dd927e0 0x55c56dd94bd0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.825+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd7a680 0x55c56dd77770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.825+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd6fdc0 0x55c56dd77230 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.825+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/4172183046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 0x55c56dd79f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.825+0000 7fe7f68b2740 1 -- 172.21.15.33:0/4172183046 >> 172.21.15.33:0/4172183046 conn(0x55c56dd6ba30 msgr2=0x55c56dd71760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.825+0000 7fe7d5ffb640 0 client.28307 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:20.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.826+0000 7fe7f68b2740 1 -- 172.21.15.33:0/4172183046 shutdown_connections 2024-09-16T00:36:20.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.826+0000 7fe7f68b2740 1 -- 172.21.15.33:0/4172183046 wait complete. 2024-09-16T00:36:20.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.826+0000 7fe7f68b2740 20 client.28307 trim_cache size 0 max 16384 2024-09-16T00:36:20.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.846+0000 7fe7f68b2740 1 Processor -- start 2024-09-16T00:36:20.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.846+0000 7fe7f68b2740 1 -- start start 2024-09-16T00:36:20.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.846+0000 7fe7f68b2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd6f4a0 0x55c56dd6aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.846+0000 7fe7f68b2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 0x55c56dd9a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f68b2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd7a680 0x55c56dd9a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f68b2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec1af840 con 0x55c56dd7a680 2024-09-16T00:36:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f68b2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec1ad110 con 0x55c56dd6f4a0 2024-09-16T00:36:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f68b2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec1aaf60 con 0x55c56dd6fdc0 2024-09-16T00:36:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f1a74640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 0x55c56dd9a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f2a76640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd7a680 0x55c56dd9a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f1a74640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 0x55c56dd9a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59554/0 (socket says 172.21.15.33:59554) 2024-09-16T00:36:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f1a74640 1 -- 172.21.15.33:0/3092908777 learned_addr learned my addr 172.21.15.33:0/3092908777 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f2275640 1 --2- 172.21.15.33:0/3092908777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd6f4a0 0x55c56dd6aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f2a76640 1 -- 172.21.15.33:0/3092908777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 msgr2=0x55c56dd9a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f2a76640 1 --2- 172.21.15.33:0/3092908777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 0x55c56dd9a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f2a76640 1 -- 172.21.15.33:0/3092908777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd6f4a0 msgr2=0x55c56dd6aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f2a76640 1 --2- 172.21.15.33:0/3092908777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd6f4a0 0x55c56dd6aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f2a76640 1 -- 172.21.15.33:0/3092908777 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c56dd6f2d0 con 0x55c56dd7a680 2024-09-16T00:36:20.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f1a74640 1 --2- 172.21.15.33:0/3092908777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 0x55c56dd9a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:20.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7f2a76640 1 --2- 172.21.15.33:0/3092908777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd7a680 0x55c56dd9a9e0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4004a80 tx=0x7fe7e4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8243538f1b3aa57f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:20.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7d7fff640 1 -- 172.21.15.33:0/3092908777 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4004d20 con 0x55c56dd7a680 2024-09-16T00:36:20.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7d7fff640 1 -- 172.21.15.33:0/3092908777 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e4003e40 con 0x55c56dd7a680 2024-09-16T00:36:20.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.847+0000 7fe7d7fff640 1 -- 172.21.15.33:0/3092908777 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4027750 con 0x55c56dd7a680 2024-09-16T00:36:20.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.848+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3092908777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd7a680 msgr2=0x55c56dd9a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:20.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.848+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/3092908777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd7a680 0x55c56dd9a9e0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4004a80 tx=0x7fe7e4014000 comp rx=0 tx=0).stop 2024-09-16T00:36:20.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.848+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3092908777 shutdown_connections 2024-09-16T00:36:20.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.848+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/3092908777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd7a680 0x55c56dd9a9e0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.848+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/3092908777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6fdc0 0x55c56dd9a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.848+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/3092908777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd6f4a0 0x55c56dd6aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.848+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3092908777 >> 172.21.15.33:0/3092908777 conn(0x55c56dd6ba30 msgr2=0x55c56dd6a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.848+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3092908777 shutdown_connections 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.848+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3092908777 wait complete. 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f68b2740 1 -- start start 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f68b2740 1 Processor -- start 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f68b2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 0x55c56dd8b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f68b2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd6fdc0 0x55c56dd88b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f68b2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd7a680 0x55c56dd892d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f68b2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec1ab5d0 con 0x55c56dd6fdc0 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f68b2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec1aca40 con 0x55c56dd7a680 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f68b2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec1abca0 con 0x55c56dd6f4a0 2024-09-16T00:36:20.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f68b2740 10 client.0 _ll_register_callbacks cb 0x7fe7dc097b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:20.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f2275640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 0x55c56dd8b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f1a74640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd6fdc0 0x55c56dd88b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f2275640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 0x55c56dd8b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59564/0 (socket says 172.21.15.33:59564) 2024-09-16T00:36:20.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f2275640 1 -- 172.21.15.33:0/3073691730 learned_addr learned my addr 172.21.15.33:0/3073691730 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:20.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f2a76640 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd7a680 0x55c56dd892d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f2275640 1 -- 172.21.15.33:0/3073691730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd7a680 msgr2=0x55c56dd892d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:20.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f2275640 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd7a680 0x55c56dd892d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f2275640 1 -- 172.21.15.33:0/3073691730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd6fdc0 msgr2=0x55c56dd88b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:20.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f2275640 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd6fdc0 0x55c56dd88b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f2275640 1 -- 172.21.15.33:0/3073691730 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c56ddfc480 con 0x55c56dd6f4a0 2024-09-16T00:36:20.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.849+0000 7fe7f1a74640 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd6fdc0 0x55c56dd88b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:20.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.850+0000 7fe7f2275640 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 0x55c56dd8b7a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009900 tx=0x7fe7e8005470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1063003665ba45b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.850+0000 7fe7d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.850+0000 7fe7d5ffb640 1 -- 172.21.15.33:0/3073691730 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8040040 con 0x55c56dd6f4a0 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.850+0000 7fe7d5ffb640 1 -- 172.21.15.33:0/3073691730 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe7e8004240 con 0x55c56dd6f4a0 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.850+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3073691730 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c56dce2850 con 0x55c56dd6f4a0 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.850+0000 7fe7d5ffb640 1 -- 172.21.15.33:0/3073691730 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe7e805f050 con 0x55c56dd6f4a0 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.850+0000 7fe7f68b2740 10 client.28303 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.850+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3073691730 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c56ddfd690 con 0x55c56dd6f4a0 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.850+0000 7fe7d5ffb640 1 -- 172.21.15.33:0/3073691730 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c56dce2850 con 0x55c56dd6f4a0 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.851+0000 7fe7d5ffb640 1 -- 172.21.15.33:0/3073691730 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fe7e804e070 con 0x55c56dd6f4a0 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.851+0000 7fe7f68b2740 10 client.28303 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.851+0000 7fe7f68b2740 10 client.28303 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.851+0000 7fe7f68b2740 20 client.28303 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:20.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.851+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c56ddfd9f0 0x55c56ddffe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.851+0000 7fe7f68b2740 4 client.28303 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:36:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.851+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3073691730 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c56dcdcd60 con 0x55c56ddfd9f0 2024-09-16T00:36:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.851+0000 7fe7f1a74640 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c56ddfd9f0 0x55c56ddffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f1a74640 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c56ddfd9f0 0x55c56ddffe10 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88aa26c61308def6 server_cookie=e93b13fca654d1b in_seq=0 out_seq=0 2024-09-16T00:36:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7d5ffb640 10 client.28303 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7d5ffb640 1 -- 172.21.15.33:0/3073691730 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c56dcdcd60 con 0x55c56ddfd9f0 2024-09-16T00:36:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7d5ffb640 10 client.28303 handle_command_reply: tid=0 2024-09-16T00:36:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f68b2740 1 client.28303 shutdown 2024-09-16T00:36:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3073691730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 msgr2=0x55c56dd8b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 0x55c56dd8b7a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009900 tx=0x7fe7e8005470 comp rx=0 tx=0).stop 2024-09-16T00:36:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3073691730 shutdown_connections 2024-09-16T00:36:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c56ddfd9f0 0x55c56ddffe10 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c56dd7a680 0x55c56dd892d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c56dd6fdc0 0x55c56dd88b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f68b2740 1 --2- 172.21.15.33:0/3073691730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c56dd6f4a0 0x55c56dd8b7a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7d5ffb640 0 client.28303 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:20.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3073691730 >> 172.21.15.33:0/3073691730 conn(0x55c56dd6ba30 msgr2=0x55c56dd6ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:20.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3073691730 shutdown_connections 2024-09-16T00:36:20.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f68b2740 1 -- 172.21.15.33:0/3073691730 wait complete. 2024-09-16T00:36:20.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.852+0000 7fe7f68b2740 20 client.28303 trim_cache size 0 max 16384 2024-09-16T00:36:20.860 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:36:20.860 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:36:20.860 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:36:20.860 INFO:teuthology.orchestra.run.smithi033.stdout: "670f9148-f975-45ae-b1d5-d26fec546026": { 2024-09-16T00:36:20.860 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:36:20.860 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "670f9148-f975-45ae-b1d5-d26fec546026", 2024-09-16T00:36:20.861 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:36:20.861 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:36:20.861 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:36:20.861 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:36:20.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.853+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4064356311 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe7d80a3240 msgr2=0x7fe7d80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:20.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.853+0000 7fe7f44d9640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe7d80a3240 0x7fe7d80a5700 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec102b10 tx=0x7fe7e4009a90 comp rx=0 tx=0).stop 2024-09-16T00:36:20.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.853+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4064356311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec1012e0 msgr2=0x7fe7ec19abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:20.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.853+0000 7fe7f44d9640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec1012e0 0x7fe7ec19abc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00d950 tx=0x7fe7dc00de10 comp rx=0 tx=0).stop 2024-09-16T00:36:20.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.853+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4064356311 shutdown_connections 2024-09-16T00:36:20.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.853+0000 7fe7f44d9640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe7d80a3240 0x7fe7d80a5700 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.853+0000 7fe7f44d9640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7ec10e4e0 0x7fe7ec19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.853+0000 7fe7f44d9640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec1012e0 0x7fe7ec19abc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.853+0000 7fe7f44d9640 1 --2- 172.21.15.33:0/4064356311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7ec100930 0x7fe7ec19a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:20.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.853+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4064356311 >> 172.21.15.33:0/4064356311 conn(0x7fe7ec0fc7a0 msgr2=0x7fe7ec0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:20.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.853+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4064356311 shutdown_connections 2024-09-16T00:36:20.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:20.853+0000 7fe7f44d9640 1 -- 172.21.15.33:0/4064356311 wait complete. 2024-09-16T00:36:21.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 116302 18.83 MB/sec execute 207 sec latency 136.409 ms 2024-09-16T00:36:21.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:21 smithi033 ceph-mon[31243]: pgmap v926: 129 pgs: 129 active+clean; 597 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.4 MiB/s wr, 445 op/s 2024-09-16T00:36:21.688 INFO:tasks.cephfs.filesystem:scrub status for tag:670f9148-f975-45ae-b1d5-d26fec546026 - {'path': '/', 'tag': '670f9148-f975-45ae-b1d5-d26fec546026', 'options': 'recursive,force'} 2024-09-16T00:36:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:21 smithi083 ceph-mon[35242]: pgmap v926: 129 pgs: 129 active+clean; 597 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.4 MiB/s wr, 445 op/s 2024-09-16T00:36:21.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:21 smithi060 ceph-mon[35185]: pgmap v926: 129 pgs: 129 active+clean; 597 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.4 MiB/s wr, 445 op/s 2024-09-16T00:36:22.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 116966 18.91 MB/sec execute 208 sec latency 69.852 ms 2024-09-16T00:36:23.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 117636 19.04 MB/sec execute 209 sec latency 61.397 ms 2024-09-16T00:36:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:23 smithi083 ceph-mon[35242]: pgmap v927: 129 pgs: 129 active+clean; 602 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2024-09-16T00:36:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:23 smithi060 ceph-mon[35185]: pgmap v927: 129 pgs: 129 active+clean; 602 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2024-09-16T00:36:24.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:23 smithi033 ceph-mon[31243]: pgmap v927: 129 pgs: 129 active+clean; 602 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2024-09-16T00:36:24.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 117778 18.98 MB/sec execute 210 sec latency 627.402 ms 2024-09-16T00:36:25.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 117862 18.90 MB/sec execute 211 sec latency 563.908 ms 2024-09-16T00:36:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:25 smithi083 ceph-mon[35242]: pgmap v928: 129 pgs: 129 active+clean; 617 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 4.3 MiB/s wr, 430 op/s 2024-09-16T00:36:25.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:25 smithi060 ceph-mon[35185]: pgmap v928: 129 pgs: 129 active+clean; 617 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 4.3 MiB/s wr, 430 op/s 2024-09-16T00:36:26.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:25 smithi033 ceph-mon[31243]: pgmap v928: 129 pgs: 129 active+clean; 617 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 4.3 MiB/s wr, 430 op/s 2024-09-16T00:36:26.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 118143 18.90 MB/sec execute 212 sec latency 130.856 ms 2024-09-16T00:36:27.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 118499 18.90 MB/sec execute 213 sec latency 91.058 ms 2024-09-16T00:36:27.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:27 smithi083 ceph-mon[35242]: pgmap v929: 129 pgs: 129 active+clean; 656 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.4 MiB/s wr, 509 op/s 2024-09-16T00:36:27.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:27 smithi060 ceph-mon[35185]: pgmap v929: 129 pgs: 129 active+clean; 656 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.4 MiB/s wr, 509 op/s 2024-09-16T00:36:28.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:27 smithi033 ceph-mon[31243]: pgmap v929: 129 pgs: 129 active+clean; 656 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.4 MiB/s wr, 509 op/s 2024-09-16T00:36:28.083 INFO:tasks.workunit.client.0.smithi033.stdout: 10 119022 18.92 MB/sec execute 214 sec latency 80.424 ms 2024-09-16T00:36:29.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 119370 18.93 MB/sec execute 215 sec latency 328.376 ms 2024-09-16T00:36:29.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:29 smithi083 ceph-mon[35242]: pgmap v930: 129 pgs: 129 active+clean; 656 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.8 MiB/s wr, 442 op/s 2024-09-16T00:36:29.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:36:29.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:36:29.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:36:29.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:29 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:36:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:29 smithi060 ceph-mon[35185]: pgmap v930: 129 pgs: 129 active+clean; 656 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.8 MiB/s wr, 442 op/s 2024-09-16T00:36:29.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:36:29.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:36:29.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:36:29.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:29 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:36:30.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:29 smithi033 ceph-mon[31243]: pgmap v930: 129 pgs: 129 active+clean; 656 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.8 MiB/s wr, 442 op/s 2024-09-16T00:36:30.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:36:30.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:36:30.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:36:30.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:29 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:36:30.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 119844 18.96 MB/sec execute 216 sec latency 147.428 ms 2024-09-16T00:36:31.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 120392 18.98 MB/sec execute 217 sec latency 72.322 ms 2024-09-16T00:36:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:31 smithi083 ceph-mon[35242]: pgmap v931: 129 pgs: 129 active+clean; 629 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 610 op/s 2024-09-16T00:36:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:31 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:36:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:31 smithi060 ceph-mon[35185]: pgmap v931: 129 pgs: 129 active+clean; 629 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 610 op/s 2024-09-16T00:36:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:31 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:36:32.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:31 smithi033 ceph-mon[31243]: pgmap v931: 129 pgs: 129 active+clean; 629 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 610 op/s 2024-09-16T00:36:32.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:31 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:36:32.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 121092 19.07 MB/sec execute 218 sec latency 58.497 ms 2024-09-16T00:36:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:32 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:36:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:32 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:36:33.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:32 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:36:33.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:32 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:36:33.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 121778 19.17 MB/sec execute 219 sec latency 61.216 ms 2024-09-16T00:36:33.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:32 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:36:33.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:32 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:36:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:33 smithi060 ceph-mon[35185]: pgmap v932: 129 pgs: 129 active+clean; 589 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 11 MiB/s wr, 613 op/s 2024-09-16T00:36:33.980 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:33 smithi033 ceph-mon[31243]: pgmap v932: 129 pgs: 129 active+clean; 589 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 11 MiB/s wr, 613 op/s 2024-09-16T00:36:34.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 122252 19.21 MB/sec execute 220 sec latency 99.490 ms 2024-09-16T00:36:34.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:33 smithi083 ceph-mon[35242]: pgmap v932: 129 pgs: 129 active+clean; 589 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 11 MiB/s wr, 613 op/s 2024-09-16T00:36:35.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 122689 19.23 MB/sec execute 221 sec latency 64.868 ms 2024-09-16T00:36:35.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:35 smithi033 ceph-mon[31243]: pgmap v933: 129 pgs: 129 active+clean; 592 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 12 MiB/s wr, 645 op/s 2024-09-16T00:36:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:35 smithi060 ceph-mon[35185]: pgmap v933: 129 pgs: 129 active+clean; 592 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 12 MiB/s wr, 645 op/s 2024-09-16T00:36:36.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 123282 19.29 MB/sec execute 222 sec latency 93.702 ms 2024-09-16T00:36:36.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:35 smithi083 ceph-mon[35242]: pgmap v933: 129 pgs: 129 active+clean; 592 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 12 MiB/s wr, 645 op/s 2024-09-16T00:36:37.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 123885 19.31 MB/sec execute 223 sec latency 54.049 ms 2024-09-16T00:36:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:37 smithi060 ceph-mon[35185]: pgmap v934: 129 pgs: 129 active+clean; 596 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 11 MiB/s wr, 628 op/s 2024-09-16T00:36:38.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:37 smithi033 ceph-mon[31243]: pgmap v934: 129 pgs: 129 active+clean; 596 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 11 MiB/s wr, 628 op/s 2024-09-16T00:36:38.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 124506 19.39 MB/sec execute 224 sec latency 73.295 ms 2024-09-16T00:36:38.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:37 smithi083 ceph-mon[35242]: pgmap v934: 129 pgs: 129 active+clean; 596 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 11 MiB/s wr, 628 op/s 2024-09-16T00:36:39.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 125167 19.44 MB/sec execute 225 sec latency 62.215 ms 2024-09-16T00:36:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:39 smithi060 ceph-mon[35185]: pgmap v935: 129 pgs: 129 active+clean; 596 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.8 MiB/s wr, 460 op/s 2024-09-16T00:36:40.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:39 smithi033 ceph-mon[31243]: pgmap v935: 129 pgs: 129 active+clean; 596 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.8 MiB/s wr, 460 op/s 2024-09-16T00:36:40.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 125806 19.55 MB/sec execute 226 sec latency 86.137 ms 2024-09-16T00:36:40.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:39 smithi083 ceph-mon[35242]: pgmap v935: 129 pgs: 129 active+clean; 596 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.8 MiB/s wr, 460 op/s 2024-09-16T00:36:41.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 126365 19.58 MB/sec execute 227 sec latency 62.741 ms 2024-09-16T00:36:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:41 smithi060 ceph-mon[35185]: pgmap v936: 129 pgs: 129 active+clean; 604 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 6.6 MiB/s wr, 560 op/s 2024-09-16T00:36:42.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:41 smithi033 ceph-mon[31243]: pgmap v936: 129 pgs: 129 active+clean; 604 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 6.6 MiB/s wr, 560 op/s 2024-09-16T00:36:42.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 126913 19.65 MB/sec execute 228 sec latency 58.396 ms 2024-09-16T00:36:42.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:41 smithi083 ceph-mon[35242]: pgmap v936: 129 pgs: 129 active+clean; 604 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 6.6 MiB/s wr, 560 op/s 2024-09-16T00:36:43.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 127416 19.67 MB/sec execute 229 sec latency 82.452 ms 2024-09-16T00:36:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:43 smithi060 ceph-mon[35185]: pgmap v937: 129 pgs: 129 active+clean; 610 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.3 MiB/s wr, 462 op/s 2024-09-16T00:36:44.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:43 smithi033 ceph-mon[31243]: pgmap v937: 129 pgs: 129 active+clean; 610 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.3 MiB/s wr, 462 op/s 2024-09-16T00:36:44.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 127970 19.70 MB/sec execute 230 sec latency 83.191 ms 2024-09-16T00:36:44.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:43 smithi083 ceph-mon[35242]: pgmap v937: 129 pgs: 129 active+clean; 610 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.3 MiB/s wr, 462 op/s 2024-09-16T00:36:45.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 128758 19.79 MB/sec execute 231 sec latency 43.347 ms 2024-09-16T00:36:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:45 smithi060 ceph-mon[35185]: pgmap v938: 129 pgs: 129 active+clean; 614 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.5 MiB/s wr, 393 op/s 2024-09-16T00:36:46.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:45 smithi033 ceph-mon[31243]: pgmap v938: 129 pgs: 129 active+clean; 614 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.5 MiB/s wr, 393 op/s 2024-09-16T00:36:46.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 129395 19.86 MB/sec execute 232 sec latency 59.813 ms 2024-09-16T00:36:46.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:45 smithi083 ceph-mon[35242]: pgmap v938: 129 pgs: 129 active+clean; 614 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.5 MiB/s wr, 393 op/s 2024-09-16T00:36:47.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 129959 19.92 MB/sec execute 233 sec latency 74.842 ms 2024-09-16T00:36:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:47 smithi060 ceph-mon[35185]: pgmap v939: 129 pgs: 129 active+clean; 622 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.4 MiB/s wr, 377 op/s 2024-09-16T00:36:48.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:47 smithi033 ceph-mon[31243]: pgmap v939: 129 pgs: 129 active+clean; 622 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.4 MiB/s wr, 377 op/s 2024-09-16T00:36:48.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 130422 19.95 MB/sec execute 234 sec latency 96.401 ms 2024-09-16T00:36:48.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:47 smithi083 ceph-mon[35242]: pgmap v939: 129 pgs: 129 active+clean; 622 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.4 MiB/s wr, 377 op/s 2024-09-16T00:36:48.650 INFO:tasks.quiescer.fs.[cephfs]:Total duration (56.3) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T00:36:48.650 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 28.15 2024-09-16T00:36:48.650 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:36:48.949 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:49.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 130987 19.98 MB/sec execute 235 sec latency 151.207 ms 2024-09-16T00:36:49.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.473+0000 7fb070e11640 1 -- 172.21.15.33:0/3476124981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06c1082a0 msgr2=0x7fb06c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:49.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.473+0000 7fb070e11640 1 --2- 172.21.15.33:0/3476124981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06c1082a0 0x7fb06c10ec40 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb064009cd0 tx=0x7fb06402a8e0 comp rx=0 tx=0).stop 2024-09-16T00:36:49.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.473+0000 7fb070e11640 1 -- 172.21.15.33:0/3476124981 shutdown_connections 2024-09-16T00:36:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.473+0000 7fb070e11640 1 --2- 172.21.15.33:0/3476124981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06c1082a0 0x7fb06c10ec40 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.473+0000 7fb070e11640 1 --2- 172.21.15.33:0/3476124981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb06c1036e0 0x7fb06c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.473+0000 7fb070e11640 1 --2- 172.21.15.33:0/3476124981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb06c102d30 0x7fb06c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.473+0000 7fb070e11640 1 -- 172.21.15.33:0/3476124981 >> 172.21.15.33:0/3476124981 conn(0x7fb06c0fc7d0 msgr2=0x7fb06c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.473+0000 7fb070e11640 1 -- 172.21.15.33:0/3476124981 shutdown_connections 2024-09-16T00:36:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.473+0000 7fb070e11640 1 -- 172.21.15.33:0/3476124981 wait complete. 2024-09-16T00:36:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.474+0000 7fb070e11640 1 Processor -- start 2024-09-16T00:36:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.474+0000 7fb070e11640 1 -- start start 2024-09-16T00:36:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.474+0000 7fb070e11640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb06c102d30 0x7fb06c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.474+0000 7fb070e11640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb06c1036e0 0x7fb06c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.474+0000 7fb070e11640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06c1082a0 0x7fb06c19ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:49.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.474+0000 7fb070e11640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c1120a0 con 0x7fb06c1082a0 2024-09-16T00:36:49.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.474+0000 7fb070e11640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c111f20 con 0x7fb06c102d30 2024-09-16T00:36:49.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.474+0000 7fb070e11640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c112220 con 0x7fb06c1036e0 2024-09-16T00:36:49.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb06ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06c1082a0 0x7fb06c19ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:49.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb06a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb06c102d30 0x7fb06c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:49.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb069d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb06c1036e0 0x7fb06c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:49.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb06a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb06c102d30 0x7fb06c198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:48674/0 (socket says 172.21.15.33:48674) 2024-09-16T00:36:49.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb06ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06c1082a0 0x7fb06c19ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58712/0 (socket says 172.21.15.33:58712) 2024-09-16T00:36:49.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb06a59c640 1 -- 172.21.15.33:0/4164107451 learned_addr learned my addr 172.21.15.33:0/4164107451 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:49.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb069d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb06c1036e0 0x7fb06c198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:36730/0 (socket says 172.21.15.33:36730) 2024-09-16T00:36:49.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb06a59c640 1 -- 172.21.15.33:0/4164107451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb06c1036e0 msgr2=0x7fb06c198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:49.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb06a59c640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb06c1036e0 0x7fb06c198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:49.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb06a59c640 1 -- 172.21.15.33:0/4164107451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06c1082a0 msgr2=0x7fb06c19ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:49.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb06a59c640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06c1082a0 0x7fb06c19ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:49.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb06a59c640 1 -- 172.21.15.33:0/4164107451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb06c19d510 con 0x7fb06c102d30 2024-09-16T00:36:49.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb069d9b640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb06c1036e0 0x7fb06c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:49.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb06a59c640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb06c102d30 0x7fb06c198560 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009c20 tx=0x7fb05c00b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:49.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb05b7fe640 1 -- 172.21.15.33:0/4164107451 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c01e070 con 0x7fb06c102d30 2024-09-16T00:36:49.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb05b7fe640 1 -- 172.21.15.33:0/4164107451 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb05c0198f0 con 0x7fb06c102d30 2024-09-16T00:36:49.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb070e11640 1 -- 172.21.15.33:0/4164107451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb06c19d800 con 0x7fb06c102d30 2024-09-16T00:36:49.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.475+0000 7fb05b7fe640 1 -- 172.21.15.33:0/4164107451 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0187f0 con 0x7fb06c102d30 2024-09-16T00:36:49.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.476+0000 7fb070e11640 1 -- 172.21.15.33:0/4164107451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb06c1008c0 con 0x7fb06c102d30 2024-09-16T00:36:49.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.476+0000 7fb070e11640 1 -- 172.21.15.33:0/4164107451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb06c103110 con 0x7fb06c102d30 2024-09-16T00:36:49.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.478+0000 7fb05b7fe640 1 -- 172.21.15.33:0/4164107451 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb05c021070 con 0x7fb06c102d30 2024-09-16T00:36:49.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.479+0000 7fb05b7fe640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb0540a3240 0x7fb0540a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:49.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.479+0000 7fb069d9b640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb0540a3240 0x7fb0540a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:49.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.479+0000 7fb05b7fe640 1 -- 172.21.15.33:0/4164107451 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb05c0d3b50 con 0x7fb06c102d30 2024-09-16T00:36:49.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.479+0000 7fb069d9b640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb0540a3240 0x7fb0540a5700 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0fee00 tx=0x7fb060005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:49.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.483+0000 7fb05b7fe640 1 -- 172.21.15.33:0/4164107451 <== mon.2 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb05c09fd40 con 0x7fb06c102d30 2024-09-16T00:36:49.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.721+0000 7fb070e11640 1 -- 172.21.15.33:0/4164107451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb06c1998c0 con 0x7fb06c102d30 2024-09-16T00:36:49.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.721+0000 7fb05b7fe640 1 -- 172.21.15.33:0/4164107451 <== mon.2 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7fb05c0b7e90 con 0x7fb06c102d30 2024-09-16T00:36:49.722 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:36:49.722 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:36:49.722 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:36:49.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.724+0000 7fb070e11640 1 -- 172.21.15.33:0/4164107451 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb0540a3240 msgr2=0x7fb0540a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:49.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.724+0000 7fb070e11640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb0540a3240 0x7fb0540a5700 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0fee00 tx=0x7fb060005f90 comp rx=0 tx=0).stop 2024-09-16T00:36:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.724+0000 7fb070e11640 1 -- 172.21.15.33:0/4164107451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb06c102d30 msgr2=0x7fb06c198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.724+0000 7fb070e11640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb06c102d30 0x7fb06c198560 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009c20 tx=0x7fb05c00b690 comp rx=0 tx=0).stop 2024-09-16T00:36:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.724+0000 7fb070e11640 1 -- 172.21.15.33:0/4164107451 shutdown_connections 2024-09-16T00:36:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.724+0000 7fb070e11640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb0540a3240 0x7fb0540a5700 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.724+0000 7fb070e11640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb06c1082a0 0x7fb06c19ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.724+0000 7fb070e11640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb06c1036e0 0x7fb06c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.724+0000 7fb070e11640 1 --2- 172.21.15.33:0/4164107451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb06c102d30 0x7fb06c198560 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.724+0000 7fb070e11640 1 -- 172.21.15.33:0/4164107451 >> 172.21.15.33:0/4164107451 conn(0x7fb06c0fc7d0 msgr2=0x7fb06c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.724+0000 7fb070e11640 1 -- 172.21.15.33:0/4164107451 shutdown_connections 2024-09-16T00:36:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:49.724+0000 7fb070e11640 1 -- 172.21.15.33:0/4164107451 wait complete. 2024-09-16T00:36:49.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:49 smithi060 ceph-mon[35185]: pgmap v940: 129 pgs: 129 active+clean; 622 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 2.8 MiB/s wr, 310 op/s 2024-09-16T00:36:50.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:49 smithi033 ceph-mon[31243]: pgmap v940: 129 pgs: 129 active+clean; 622 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 2.8 MiB/s wr, 310 op/s 2024-09-16T00:36:50.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 131491 20.00 MB/sec execute 236 sec latency 77.695 ms 2024-09-16T00:36:50.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:49 smithi083 ceph-mon[35242]: pgmap v940: 129 pgs: 129 active+clean; 622 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 2.8 MiB/s wr, 310 op/s 2024-09-16T00:36:50.770 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 148.15 --await-for 45.0' 2024-09-16T00:36:50.770 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 148.15 --await-for 45.0 2024-09-16T00:36:50.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:50 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4164107451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:36:51.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:50 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4164107451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:36:51.085 INFO:tasks.workunit.client.0.smithi033.stdout: 10 131995 20.02 MB/sec execute 237 sec latency 63.289 ms 2024-09-16T00:36:51.097 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:51.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:50 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4164107451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:36:51.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.606+0000 7f1ee1278640 1 -- 172.21.15.33:0/652834904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1edc1082a0 msgr2=0x7f1edc10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:51.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.606+0000 7f1ee1278640 1 --2- 172.21.15.33:0/652834904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1edc1082a0 0x7f1edc10ec40 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4009930 tx=0x7f1ed402a420 comp rx=0 tx=0).stop 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.606+0000 7f1ee1278640 1 -- 172.21.15.33:0/652834904 shutdown_connections 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.606+0000 7f1ee1278640 1 --2- 172.21.15.33:0/652834904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1edc1082a0 0x7f1edc10ec40 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.606+0000 7f1ee1278640 1 --2- 172.21.15.33:0/652834904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1edc1036e0 0x7f1edc107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.606+0000 7f1ee1278640 1 --2- 172.21.15.33:0/652834904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1edc102d30 0x7f1edc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.606+0000 7f1ee1278640 1 -- 172.21.15.33:0/652834904 >> 172.21.15.33:0/652834904 conn(0x7f1edc0fc7d0 msgr2=0x7f1edc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.606+0000 7f1ee1278640 1 -- 172.21.15.33:0/652834904 shutdown_connections 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.606+0000 7f1ee1278640 1 -- 172.21.15.33:0/652834904 wait complete. 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1ee1278640 1 Processor -- start 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1ee1278640 1 -- start start 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1ee1278640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1edc102d30 0x7f1edc100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1ee1278640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1edc1036e0 0x7f1edc100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1ee1278640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1edc1082a0 0x7f1edc101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1ee1278640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1120d0 con 0x7f1edc1036e0 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1ee1278640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc111f50 con 0x7f1edc1082a0 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1ee1278640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc112250 con 0x7f1edc102d30 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1eda7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1edc1036e0 0x7f1edc100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1edaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1edc102d30 0x7f1edc100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1edaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1edc102d30 0x7f1edc100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40580/0 (socket says 172.21.15.33:40580) 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1edaffd640 1 -- 172.21.15.33:0/3528323703 learned_addr learned my addr 172.21.15.33:0/3528323703 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.607+0000 7f1eda7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1edc1036e0 0x7f1edc100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47458/0 (socket says 172.21.15.33:47458) 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1edb7fe640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1edc1082a0 0x7f1edc101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1edaffd640 1 -- 172.21.15.33:0/3528323703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1edc1082a0 msgr2=0x7f1edc101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1edaffd640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1edc1082a0 0x7f1edc101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1edaffd640 1 -- 172.21.15.33:0/3528323703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1edc1036e0 msgr2=0x7f1edc100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1edaffd640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1edc1036e0 0x7f1edc100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1edaffd640 1 -- 172.21.15.33:0/3528323703 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1edc1a6a30 con 0x7f1edc102d30 2024-09-16T00:36:51.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1eda7fc640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1edc1036e0 0x7f1edc100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:51.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1edaffd640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1edc102d30 0x7f1edc100990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00ebc0 tx=0x7f1ecc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:51.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1ec3fff640 1 -- 172.21.15.33:0/3528323703 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc010630 con 0x7f1edc102d30 2024-09-16T00:36:51.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1ee1278640 1 -- 172.21.15.33:0/3528323703 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1edc1a6d20 con 0x7f1edc102d30 2024-09-16T00:36:51.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1ec3fff640 1 -- 172.21.15.33:0/3528323703 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ecc010cb0 con 0x7f1edc102d30 2024-09-16T00:36:51.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1ec3fff640 1 -- 172.21.15.33:0/3528323703 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc018bd0 con 0x7f1edc102d30 2024-09-16T00:36:51.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.608+0000 7f1ee1278640 1 -- 172.21.15.33:0/3528323703 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1edc1a71f0 con 0x7f1edc102d30 2024-09-16T00:36:51.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.611+0000 7f1ee3651740 1 Processor -- start 2024-09-16T00:36:51.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1ee3651740 1 -- start start 2024-09-16T00:36:51.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1ec3fff640 1 -- 172.21.15.33:0/3528323703 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1ecc0107d0 con 0x7f1edc102d30 2024-09-16T00:36:51.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1ee3651740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302edd850 0x565302eddc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1ee3651740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 0x565302ee84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1ee3651740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302ee8a30 0x565302eeae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1ee3651740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x565302e42160 con 0x565302ee8a30 2024-09-16T00:36:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1ee3651740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x565302de9720 con 0x565302edd850 2024-09-16T00:36:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1ee3651740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x565302e3e120 con 0x565302ede170 2024-09-16T00:36:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1edaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 0x565302ee84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1eda7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302ee8a30 0x565302eeae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1edaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 0x565302ee84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40592/0 (socket says 172.21.15.33:40592) 2024-09-16T00:36:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1edaffd640 1 -- 172.21.15.33:0/887183821 learned_addr learned my addr 172.21.15.33:0/887183821 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1eda7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302ee8a30 0x565302eeae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47472/0 (socket says 172.21.15.33:47472) 2024-09-16T00:36:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1edaffd640 1 -- 172.21.15.33:0/887183821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302edd850 msgr2=0x565302eddc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1edaffd640 1 --2- 172.21.15.33:0/887183821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302edd850 0x565302eddc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1edaffd640 1 -- 172.21.15.33:0/887183821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302ee8a30 msgr2=0x565302eeae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:51.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1edaffd640 1 --2- 172.21.15.33:0/887183821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302ee8a30 0x565302eeae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1edaffd640 1 -- 172.21.15.33:0/887183821 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565302bb3ba0 con 0x565302ede170 2024-09-16T00:36:51.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.612+0000 7f1eda7fc640 1 --2- 172.21.15.33:0/887183821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302ee8a30 0x565302eeae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:51.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.613+0000 7f1edaffd640 1 --2- 172.21.15.33:0/887183821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 0x565302ee84f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0cda20 tx=0x7f1ecc0d6a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca1be2f6b1396d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:51.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.613+0000 7f1ec3fff640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1ec40a3240 0x7f1ec40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.613+0000 7f1edb7fe640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1ec40a3240 0x7f1ec40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.613+0000 7f1ec17fa640 1 -- 172.21.15.33:0/887183821 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc0e3070 con 0x565302ede170 2024-09-16T00:36:51.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.613+0000 7f1ec3fff640 1 -- 172.21.15.33:0/3528323703 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1ecc0d0460 con 0x7f1edc102d30 2024-09-16T00:36:51.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.613+0000 7f1ec17fa640 1 -- 172.21.15.33:0/887183821 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ecc09cd00 con 0x565302ede170 2024-09-16T00:36:51.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.613+0000 7f1ec17fa640 1 -- 172.21.15.33:0/887183821 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc0988a0 con 0x565302ede170 2024-09-16T00:36:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.614+0000 7f1ee3651740 1 -- 172.21.15.33:0/887183821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 msgr2=0x565302ee84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.614+0000 7f1ee3651740 1 --2- 172.21.15.33:0/887183821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 0x565302ee84f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0cda20 tx=0x7f1ecc0d6a90 comp rx=0 tx=0).stop 2024-09-16T00:36:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.614+0000 7f1edb7fe640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1ec40a3240 0x7f1ec40a5700 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4006010 tx=0x7f1ed4005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.614+0000 7f1ee3651740 1 -- 172.21.15.33:0/887183821 shutdown_connections 2024-09-16T00:36:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.614+0000 7f1ee3651740 1 --2- 172.21.15.33:0/887183821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302ee8a30 0x565302eeae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.614+0000 7f1ee3651740 1 --2- 172.21.15.33:0/887183821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 0x565302ee84f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.614+0000 7f1ee3651740 1 --2- 172.21.15.33:0/887183821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302edd850 0x565302eddc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.614+0000 7f1ee3651740 1 -- 172.21.15.33:0/887183821 >> 172.21.15.33:0/887183821 conn(0x565302ed9de0 msgr2=0x565302eda1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.614+0000 7f1ee3651740 1 -- 172.21.15.33:0/887183821 shutdown_connections 2024-09-16T00:36:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.614+0000 7f1ee3651740 1 -- 172.21.15.33:0/887183821 wait complete. 2024-09-16T00:36:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1ee3651740 1 -- start start 2024-09-16T00:36:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1ee3651740 1 Processor -- start 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1ee3651740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302ee59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1ee3651740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ede170 0x565302ee5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1ee3651740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ee8a30 0x565302ee6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1ee3651740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x565302eeb600 con 0x565302edd850 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1ee3651740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x565302dca900 con 0x565302ede170 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1ee3651740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x565302eeb780 con 0x565302ee8a30 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1edaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302ee59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1edaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302ee59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47488/0 (socket says 172.21.15.33:47488) 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1edaffd640 1 -- 172.21.15.33:0/862676487 learned_addr learned my addr 172.21.15.33:0/862676487 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1eda7fc640 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ede170 0x565302ee5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1ee3651740 10 client.0 _ll_register_callbacks cb 0x565302dc8780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.615+0000 7f1edb7fe640 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ee8a30 0x565302ee6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1edaffd640 1 -- 172.21.15.33:0/862676487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ee8a30 msgr2=0x565302ee6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1edaffd640 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ee8a30 0x565302ee6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1edaffd640 1 -- 172.21.15.33:0/862676487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ede170 msgr2=0x565302ee5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1edaffd640 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ede170 0x565302ee5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1edaffd640 1 -- 172.21.15.33:0/862676487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565302e0ac50 con 0x565302edd850 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1edaffd640 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302ee59c0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0a4d40 tx=0x7f1ecc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62d82e2932518be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1ebeff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1ebeff5640 1 -- 172.21.15.33:0/862676487 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc098050 con 0x565302edd850 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1ebeff5640 1 -- 172.21.15.33:0/862676487 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1ecc09a900 con 0x565302edd850 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1ee3651740 1 -- 172.21.15.33:0/862676487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ecc09cd00 con 0x565302edd850 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1ebeff5640 1 -- 172.21.15.33:0/862676487 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1ecc010e60 con 0x565302edd850 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1ee3651740 10 client.19065 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1ee3651740 1 -- 172.21.15.33:0/862676487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x565302eeb600 con 0x565302edd850 2024-09-16T00:36:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.616+0000 7f1ebeff5640 1 -- 172.21.15.33:0/862676487 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ecc09cd00 con 0x565302edd850 2024-09-16T00:36:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.617+0000 7f1ebeff5640 1 -- 172.21.15.33:0/862676487 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f1ecc09c860 con 0x565302edd850 2024-09-16T00:36:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.617+0000 7f1ee3651740 10 client.19065 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.617+0000 7f1ee3651740 10 client.19065 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:36:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.617+0000 7f1ee3651740 20 client.19065 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.617+0000 7f1ee3651740 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x565302ef84b0 0x565302efa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.617+0000 7f1ee3651740 4 client.19065 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:36:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.617+0000 7f1ee3651740 1 -- 172.21.15.33:0/862676487 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565302e4f890 con 0x565302ef84b0 2024-09-16T00:36:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.617+0000 7f1eda7fc640 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x565302ef84b0 0x565302efa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.617+0000 7f1eda7fc640 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x565302ef84b0 0x565302efa8a0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee5071c88d7e440b server_cookie=8667c56ef409347f in_seq=0 out_seq=0 2024-09-16T00:36:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.617+0000 7f1ebeff5640 10 client.19065 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.619+0000 7f1ebeff5640 1 -- 172.21.15.33:0/862676487 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x565302e4f890 con 0x565302ef84b0 2024-09-16T00:36:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.619+0000 7f1ebeff5640 10 client.19065 handle_command_reply: tid=0 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.619+0000 7f1ee3651740 1 client.19065 shutdown 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.619+0000 7f1ee3651740 1 -- 172.21.15.33:0/862676487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 msgr2=0x565302ee59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.619+0000 7f1ee3651740 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302ee59c0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0a4d40 tx=0x7f1ecc00cd50 comp rx=0 tx=0).stop 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.619+0000 7f1ee3651740 1 -- 172.21.15.33:0/862676487 shutdown_connections 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.619+0000 7f1ee3651740 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x565302ef84b0 0x565302efa8a0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.620+0000 7f1ee3651740 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ee8a30 0x565302ee6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.620+0000 7f1ee3651740 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ede170 0x565302ee5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.620+0000 7f1ee3651740 1 --2- 172.21.15.33:0/862676487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302ee59c0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.620+0000 7f1ee3651740 1 -- 172.21.15.33:0/862676487 >> 172.21.15.33:0/862676487 conn(0x565302ed9de0 msgr2=0x565302edfac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.620+0000 7f1ebeff5640 0 client.19065 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.620+0000 7f1ee3651740 1 -- 172.21.15.33:0/862676487 shutdown_connections 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.620+0000 7f1ee3651740 1 -- 172.21.15.33:0/862676487 wait complete. 2024-09-16T00:36:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.620+0000 7f1ee3651740 20 client.19065 trim_cache size 0 max 16384 2024-09-16T00:36:51.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.640+0000 7f1ee3651740 1 Processor -- start 2024-09-16T00:36:51.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.640+0000 7f1ee3651740 1 -- start start 2024-09-16T00:36:51.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.640+0000 7f1ee3651740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302ee4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.640+0000 7f1ee3651740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 0x565302efff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.640+0000 7f1ee3651740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ee8a30 0x565302f004d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.640+0000 7f1ee3651740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1b3c30 con 0x565302edd850 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.640+0000 7f1ee3651740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1b1500 con 0x565302ee8a30 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.640+0000 7f1ee3651740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1af350 con 0x565302ede170 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.640+0000 7f1edaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302ee4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.640+0000 7f1edaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302ee4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47492/0 (socket says 172.21.15.33:47492) 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.640+0000 7f1edaffd640 1 -- 172.21.15.33:0/282797368 learned_addr learned my addr 172.21.15.33:0/282797368 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.641+0000 7f1edb7fe640 1 --2- 172.21.15.33:0/282797368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ee8a30 0x565302f004d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.641+0000 7f1eda7fc640 1 --2- 172.21.15.33:0/282797368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 0x565302efff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.641+0000 7f1edaffd640 1 -- 172.21.15.33:0/282797368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 msgr2=0x565302efff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.641+0000 7f1edaffd640 1 --2- 172.21.15.33:0/282797368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 0x565302efff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.641+0000 7f1edaffd640 1 -- 172.21.15.33:0/282797368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ee8a30 msgr2=0x565302f004d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:51.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.641+0000 7f1edaffd640 1 --2- 172.21.15.33:0/282797368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ee8a30 0x565302f004d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.641+0000 7f1edaffd640 1 -- 172.21.15.33:0/282797368 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565302edd5d0 con 0x565302edd850 2024-09-16T00:36:51.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.641+0000 7f1eda7fc640 1 --2- 172.21.15.33:0/282797368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 0x565302efff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:36:51.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.641+0000 7f1edaffd640 1 --2- 172.21.15.33:0/282797368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302ee4900 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00cba0 tx=0x7f1ecc0d6a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d8f95764524d67 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:51.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.641+0000 7f1ec0ff9640 1 -- 172.21.15.33:0/282797368 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc098040 con 0x565302edd850 2024-09-16T00:36:51.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.641+0000 7f1ec0ff9640 1 -- 172.21.15.33:0/282797368 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ecc09cd00 con 0x565302edd850 2024-09-16T00:36:51.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.641+0000 7f1ec0ff9640 1 -- 172.21.15.33:0/282797368 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc024120 con 0x565302edd850 2024-09-16T00:36:51.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.642+0000 7f1ee3651740 1 -- 172.21.15.33:0/282797368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 msgr2=0x565302ee4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:51.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.642+0000 7f1ee3651740 1 --2- 172.21.15.33:0/282797368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302ee4900 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00cba0 tx=0x7f1ecc0d6a90 comp rx=0 tx=0).stop 2024-09-16T00:36:51.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.642+0000 7f1ee3651740 1 -- 172.21.15.33:0/282797368 shutdown_connections 2024-09-16T00:36:51.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.642+0000 7f1ee3651740 1 --2- 172.21.15.33:0/282797368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ee8a30 0x565302f004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.642+0000 7f1ee3651740 1 --2- 172.21.15.33:0/282797368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ede170 0x565302efff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.642+0000 7f1ee3651740 1 --2- 172.21.15.33:0/282797368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302ee4900 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.642+0000 7f1ee3651740 1 -- 172.21.15.33:0/282797368 >> 172.21.15.33:0/282797368 conn(0x565302ed9de0 msgr2=0x565302edfac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:51.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.642+0000 7f1ee3651740 1 -- 172.21.15.33:0/282797368 shutdown_connections 2024-09-16T00:36:51.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.642+0000 7f1ee3651740 1 -- 172.21.15.33:0/282797368 wait complete. 2024-09-16T00:36:51.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.642+0000 7f1ee3651740 1 -- start start 2024-09-16T00:36:51.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1ee3651740 1 Processor -- start 2024-09-16T00:36:51.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1ee3651740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302eec800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1ee3651740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ede170 0x565302eecd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1ee3651740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ee8a30 0x565302eed280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1ee3651740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1af9c0 con 0x565302edd850 2024-09-16T00:36:51.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1ee3651740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1b0e30 con 0x565302ede170 2024-09-16T00:36:51.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1ee3651740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1b0090 con 0x565302ee8a30 2024-09-16T00:36:51.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1ee3651740 10 client.0 _ll_register_callbacks cb 0x7f1ecc0e1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:51.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1edaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302eec800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1edb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ee8a30 0x565302eed280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1edaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302eec800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47504/0 (socket says 172.21.15.33:47504) 2024-09-16T00:36:51.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1edaffd640 1 -- 172.21.15.33:0/3654775483 learned_addr learned my addr 172.21.15.33:0/3654775483 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:51.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1eda7fc640 1 --2- 172.21.15.33:0/3654775483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ede170 0x565302eecd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1edaffd640 1 -- 172.21.15.33:0/3654775483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ee8a30 msgr2=0x565302eed280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1edaffd640 1 --2- 172.21.15.33:0/3654775483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ee8a30 0x565302eed280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1edaffd640 1 -- 172.21.15.33:0/3654775483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ede170 msgr2=0x565302eecd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1edaffd640 1 --2- 172.21.15.33:0/3654775483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ede170 0x565302eecd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.643+0000 7f1edaffd640 1 -- 172.21.15.33:0/3654775483 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565302f62030 con 0x565302edd850 2024-09-16T00:36:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1edaffd640 1 --2- 172.21.15.33:0/3654775483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302eec800 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc098f80 tx=0x7f1ecc0d0760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35dc89dbec4c2824 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ebeff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ebeff5640 1 -- 172.21.15.33:0/3654775483 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc098070 con 0x565302edd850 2024-09-16T00:36:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ebeff5640 1 -- 172.21.15.33:0/3654775483 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1ecc09a900 con 0x565302edd850 2024-09-16T00:36:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ee3651740 1 -- 172.21.15.33:0/3654775483 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ecc09cd00 con 0x565302edd850 2024-09-16T00:36:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ebeff5640 1 -- 172.21.15.33:0/3654775483 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1ecc09bf60 con 0x565302edd850 2024-09-16T00:36:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ee3651740 10 client.19071 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:51.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ee3651740 1 -- 172.21.15.33:0/3654775483 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x565302f62030 con 0x565302edd850 2024-09-16T00:36:51.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ebeff5640 1 -- 172.21.15.33:0/3654775483 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ecc09cd00 con 0x565302edd850 2024-09-16T00:36:51.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ebeff5640 1 -- 172.21.15.33:0/3654775483 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f1ecc0a4de0 con 0x565302edd850 2024-09-16T00:36:51.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ee3651740 10 client.19071 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:51.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ee3651740 10 client.19071 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:36:51.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ee3651740 20 client.19071 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:51.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ee3651740 1 --2- 172.21.15.33:0/3654775483 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x565302f635b0 0x565302f659a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:51.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ee3651740 4 client.19071 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 148.15, "await_for": 45.0}] 2024-09-16T00:36:51.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.644+0000 7f1ee3651740 1 -- 172.21.15.33:0/3654775483 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 148.15, "await_for": 45.0}) -- 0x565302e49be0 con 0x565302f635b0 2024-09-16T00:36:51.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.645+0000 7f1eda7fc640 1 --2- 172.21.15.33:0/3654775483 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x565302f635b0 0x565302f659a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:51.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.645+0000 7f1ebeff5640 10 client.19071 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:51.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:51.645+0000 7f1eda7fc640 1 --2- 172.21.15.33:0/3654775483 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x565302f635b0 0x565302f659a0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2adfa36055404aac server_cookie=61077dfde1a4cf8c in_seq=0 out_seq=0 2024-09-16T00:36:51.689 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:36:51.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:51 smithi033 ceph-mon[31243]: pgmap v941: 129 pgs: 129 active+clean; 632 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.8 MiB/s wr, 436 op/s 2024-09-16T00:36:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:51 smithi060 ceph-mon[35185]: pgmap v941: 129 pgs: 129 active+clean; 632 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.8 MiB/s wr, 436 op/s 2024-09-16T00:36:51.996 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:52.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132371 20.03 MB/sec execute 238 sec latency 402.278 ms 2024-09-16T00:36:52.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:51 smithi083 ceph-mon[35242]: pgmap v941: 129 pgs: 129 active+clean; 632 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.8 MiB/s wr, 436 op/s 2024-09-16T00:36:52.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.496+0000 7ff614afe640 1 -- 172.21.15.33:0/2013365575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff610100e20 msgr2=0x7ff610101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:52.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.496+0000 7ff614afe640 1 --2- 172.21.15.33:0/2013365575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff610100e20 0x7ff610101200 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff600009900 tx=0x7ff60002a480 comp rx=0 tx=0).stop 2024-09-16T00:36:52.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.496+0000 7ff614afe640 1 -- 172.21.15.33:0/2013365575 shutdown_connections 2024-09-16T00:36:52.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.496+0000 7ff614afe640 1 --2- 172.21.15.33:0/2013365575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff610102ef0 0x7ff61010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.496+0000 7ff614afe640 1 --2- 172.21.15.33:0/2013365575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff610102570 0x7ff6101029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.496+0000 7ff614afe640 1 --2- 172.21.15.33:0/2013365575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff610100e20 0x7ff610101200 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.496+0000 7ff614afe640 1 -- 172.21.15.33:0/2013365575 >> 172.21.15.33:0/2013365575 conn(0x7ff6100fc7d0 msgr2=0x7ff6100febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:52.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.496+0000 7ff614afe640 1 -- 172.21.15.33:0/2013365575 shutdown_connections 2024-09-16T00:36:52.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.496+0000 7ff614afe640 1 -- 172.21.15.33:0/2013365575 wait complete. 2024-09-16T00:36:52.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.496+0000 7ff614afe640 1 Processor -- start 2024-09-16T00:36:52.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff614afe640 1 -- start start 2024-09-16T00:36:52.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff614afe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff610100e20 0x7ff610198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff614afe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff610102570 0x7ff6101989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff614afe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff610102ef0 0x7ff61019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff614afe640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff610110f80 con 0x7ff610100e20 2024-09-16T00:36:52.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff614afe640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff610110e00 con 0x7ff610102ef0 2024-09-16T00:36:52.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff614afe640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff610111100 con 0x7ff610102570 2024-09-16T00:36:52.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff60e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff610100e20 0x7ff610198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff60dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff610102570 0x7ff6101989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff60e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff610100e20 0x7ff610198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47530/0 (socket says 172.21.15.33:47530) 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff60e59c640 1 -- 172.21.15.33:0/1417733291 learned_addr learned my addr 172.21.15.33:0/1417733291 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff60ed9d640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff610102ef0 0x7ff61019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff60e59c640 1 -- 172.21.15.33:0/1417733291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff610102570 msgr2=0x7ff6101989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff60e59c640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff610102570 0x7ff6101989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff60e59c640 1 -- 172.21.15.33:0/1417733291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff610102ef0 msgr2=0x7ff61019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff60e59c640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff610102ef0 0x7ff61019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff60e59c640 1 -- 172.21.15.33:0/1417733291 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61019d420 con 0x7ff610100e20 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff610102570 0x7ff6101989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.497+0000 7ff60e59c640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff610100e20 0x7ff610198470 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff600006010 tx=0x7ff600003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.498+0000 7ff5ff7fe640 1 -- 172.21.15.33:0/1417733291 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60003f070 con 0x7ff610100e20 2024-09-16T00:36:52.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.498+0000 7ff5ff7fe640 1 -- 172.21.15.33:0/1417733291 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff600031c30 con 0x7ff610100e20 2024-09-16T00:36:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.498+0000 7ff614afe640 1 -- 172.21.15.33:0/1417733291 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff61019d6b0 con 0x7ff610100e20 2024-09-16T00:36:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.498+0000 7ff5ff7fe640 1 -- 172.21.15.33:0/1417733291 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600035030 con 0x7ff610100e20 2024-09-16T00:36:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.498+0000 7ff614afe640 1 -- 172.21.15.33:0/1417733291 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6101a4ff0 con 0x7ff610100e20 2024-09-16T00:36:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.500+0000 7ff5ff7fe640 1 -- 172.21.15.33:0/1417733291 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff600042070 con 0x7ff610100e20 2024-09-16T00:36:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.501+0000 7ff616ed7740 1 Processor -- start 2024-09-16T00:36:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.501+0000 7ff616ed7740 1 -- start start 2024-09-16T00:36:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.501+0000 7ff5ff7fe640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5f80a32c0 0x7ff5f80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff616ed7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f754a0 0x561ba8f75880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff616ed7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f80140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff616ed7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f80680 0x561ba8f82a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff616ed7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561ba8edb2e0 con 0x561ba8f754a0 2024-09-16T00:36:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff616ed7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561ba8e82ce0 con 0x561ba8f80680 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff616ed7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561ba8ed72a0 con 0x561ba8f75dc0 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff60ed9d640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5f80a32c0 0x7ff5f80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff5ff7fe640 1 -- 172.21.15.33:0/1417733291 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff6000f04c0 con 0x7ff610100e20 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff60dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f80140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff60e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f754a0 0x561ba8f75880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff60dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f80140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40654/0 (socket says 172.21.15.33:40654) 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff60dd9b640 1 -- 172.21.15.33:0/2497275601 learned_addr learned my addr 172.21.15.33:0/2497275601 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff60ed9d640 1 --2- 172.21.15.33:0/2497275601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f80680 0x561ba8f82a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff60dd9b640 1 -- 172.21.15.33:0/2497275601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f80680 msgr2=0x561ba8f82a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/2497275601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f80680 0x561ba8f82a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff60dd9b640 1 -- 172.21.15.33:0/2497275601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f754a0 msgr2=0x561ba8f75880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/2497275601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f754a0 0x561ba8f75880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff60dd9b640 1 -- 172.21.15.33:0/2497275601 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ba8e67440 con 0x561ba8f75dc0 2024-09-16T00:36:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.502+0000 7ff60e59c640 1 --2- 172.21.15.33:0/2497275601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f754a0 0x561ba8f75880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.503+0000 7ff60ed9d640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5f80a32c0 0x7ff5f80a5780 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff6080136d0 tx=0x7ff608013ad0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.503+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/2497275601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f80140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff6040104a0 tx=0x7ff604010960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b79eced557e702d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.503+0000 7ff5fcff9640 1 -- 172.21.15.33:0/2497275601 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604005220 con 0x561ba8f75dc0 2024-09-16T00:36:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.503+0000 7ff5fcff9640 1 -- 172.21.15.33:0/2497275601 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff604004c80 con 0x561ba8f75dc0 2024-09-16T00:36:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.503+0000 7ff5fcff9640 1 -- 172.21.15.33:0/2497275601 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604011c60 con 0x561ba8f75dc0 2024-09-16T00:36:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.504+0000 7ff616ed7740 1 -- 172.21.15.33:0/2497275601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 msgr2=0x561ba8f80140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.504+0000 7ff616ed7740 1 --2- 172.21.15.33:0/2497275601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f80140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff6040104a0 tx=0x7ff604010960 comp rx=0 tx=0).stop 2024-09-16T00:36:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.504+0000 7ff616ed7740 1 -- 172.21.15.33:0/2497275601 shutdown_connections 2024-09-16T00:36:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.504+0000 7ff616ed7740 1 --2- 172.21.15.33:0/2497275601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f80680 0x561ba8f82a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.504+0000 7ff616ed7740 1 --2- 172.21.15.33:0/2497275601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f80140 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.504+0000 7ff616ed7740 1 --2- 172.21.15.33:0/2497275601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f754a0 0x561ba8f75880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.504+0000 7ff616ed7740 1 -- 172.21.15.33:0/2497275601 >> 172.21.15.33:0/2497275601 conn(0x561ba8f71a30 msgr2=0x561ba8f71e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.504+0000 7ff616ed7740 1 -- 172.21.15.33:0/2497275601 shutdown_connections 2024-09-16T00:36:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.504+0000 7ff616ed7740 1 -- 172.21.15.33:0/2497275601 wait complete. 2024-09-16T00:36:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff616ed7740 1 -- start start 2024-09-16T00:36:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff616ed7740 1 Processor -- start 2024-09-16T00:36:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff616ed7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 0x561ba8f79540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff616ed7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f79a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff616ed7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8f79fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff616ed7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561ba8f831f0 con 0x561ba8f80680 2024-09-16T00:36:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff616ed7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561ba8e96050 con 0x561ba8f754a0 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff616ed7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561ba8f83370 con 0x561ba8f75dc0 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff616ed7740 10 client.0 _ll_register_callbacks cb 0x561ba8ed6160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff60dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f79a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff60dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f79a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40668/0 (socket says 172.21.15.33:40668) 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff60dd9b640 1 -- 172.21.15.33:0/1271717486 learned_addr learned my addr 172.21.15.33:0/1271717486 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff60e59c640 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 0x561ba8f79540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.505+0000 7ff60ed9d640 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8f79fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff60dd9b640 1 -- 172.21.15.33:0/1271717486 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 msgr2=0x561ba8f79540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 0x561ba8f79540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff60dd9b640 1 -- 172.21.15.33:0/1271717486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 msgr2=0x561ba8f79fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8f79fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff60dd9b640 1 -- 172.21.15.33:0/1271717486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ba8c4c3c0 con 0x561ba8f75dc0 2024-09-16T00:36:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff60ed9d640 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8f79fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f79a80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff604000c00 tx=0x7ff604011670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f6eaf90b9d861ff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff5f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff5f5ffb640 1 -- 172.21.15.33:0/1271717486 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604004a90 con 0x561ba8f75dc0 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff616ed7740 1 -- 172.21.15.33:0/1271717486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff604004c80 con 0x561ba8f75dc0 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff5f5ffb640 1 -- 172.21.15.33:0/1271717486 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff604004020 con 0x561ba8f75dc0 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff5f5ffb640 1 -- 172.21.15.33:0/1271717486 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff60402f050 con 0x561ba8f75dc0 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff616ed7740 10 client.28336 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.506+0000 7ff616ed7740 1 -- 172.21.15.33:0/1271717486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561ba8f97510 con 0x561ba8f75dc0 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.507+0000 7ff5f5ffb640 1 -- 172.21.15.33:0/1271717486 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff604004c80 con 0x561ba8f75dc0 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.507+0000 7ff5f5ffb640 1 -- 172.21.15.33:0/1271717486 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ff604005200 con 0x561ba8f75dc0 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.507+0000 7ff616ed7740 10 client.28336 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.507+0000 7ff616ed7740 10 client.28336 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.507+0000 7ff616ed7740 20 client.28336 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.507+0000 7ff616ed7740 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561ba8f98970 0x561ba8f9ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.507+0000 7ff616ed7740 4 client.28336 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.507+0000 7ff616ed7740 1 -- 172.21.15.33:0/1271717486 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ba8ee8850 con 0x561ba8f98970 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.507+0000 7ff60e59c640 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561ba8f98970 0x561ba8f9ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.507+0000 7ff60e59c640 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561ba8f98970 0x561ba8f9ad60 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ce72c03665a4be0 server_cookie=4271f74c067eaab9 in_seq=0 out_seq=0 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.507+0000 7ff5f5ffb640 10 client.28336 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.508+0000 7ff5f5ffb640 1 -- 172.21.15.33:0/1271717486 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561ba8ee8850 con 0x561ba8f98970 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.508+0000 7ff5f5ffb640 10 client.28336 handle_command_reply: tid=0 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.508+0000 7ff616ed7740 1 client.28336 shutdown 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.509+0000 7ff616ed7740 1 -- 172.21.15.33:0/1271717486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 msgr2=0x561ba8f79a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.509+0000 7ff616ed7740 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f79a80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff604000c00 tx=0x7ff604011670 comp rx=0 tx=0).stop 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.509+0000 7ff616ed7740 1 -- 172.21.15.33:0/1271717486 shutdown_connections 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.509+0000 7ff616ed7740 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561ba8f98970 0x561ba8f9ad60 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.509+0000 7ff616ed7740 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8f79fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.509+0000 7ff616ed7740 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f79a80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.509+0000 7ff5f5ffb640 0 client.28336 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:52.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.509+0000 7ff616ed7740 1 --2- 172.21.15.33:0/1271717486 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 0x561ba8f79540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.509+0000 7ff616ed7740 1 -- 172.21.15.33:0/1271717486 >> 172.21.15.33:0/1271717486 conn(0x561ba8f71a30 msgr2=0x561ba8f70b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:52.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.509+0000 7ff616ed7740 1 -- 172.21.15.33:0/1271717486 shutdown_connections 2024-09-16T00:36:52.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.509+0000 7ff616ed7740 1 -- 172.21.15.33:0/1271717486 wait complete. 2024-09-16T00:36:52.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.509+0000 7ff616ed7740 20 client.28336 trim_cache size 0 max 16384 2024-09-16T00:36:52.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff616ed7740 1 Processor -- start 2024-09-16T00:36:52.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff616ed7740 1 -- start start 2024-09-16T00:36:52.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff616ed7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 0x561ba8f79e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff616ed7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8fa0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff616ed7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8fa08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff616ed7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101af5e0 con 0x561ba8f80680 2024-09-16T00:36:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff616ed7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101aceb0 con 0x561ba8f754a0 2024-09-16T00:36:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff616ed7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101aad00 con 0x561ba8f75dc0 2024-09-16T00:36:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff60ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8fa08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff60ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8fa08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47560/0 (socket says 172.21.15.33:47560) 2024-09-16T00:36:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff60dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8fa0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff60e59c640 1 --2- 172.21.15.33:0/3129520660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 0x561ba8f79e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff60ed9d640 1 -- 172.21.15.33:0/3129520660 learned_addr learned my addr 172.21.15.33:0/3129520660 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff60dd9b640 1 -- 172.21.15.33:0/3129520660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 msgr2=0x561ba8f79e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/3129520660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 0x561ba8f79e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff60dd9b640 1 -- 172.21.15.33:0/3129520660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 msgr2=0x561ba8fa08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:52.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/3129520660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8fa08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff60dd9b640 1 -- 172.21.15.33:0/3129520660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ba8f7fe80 con 0x561ba8f75dc0 2024-09-16T00:36:52.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.527+0000 7ff60ed9d640 1 --2- 172.21.15.33:0/3129520660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8fa08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:52.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.528+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/3129520660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8fa0390 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff6040055d0 tx=0x7ff604011a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b31094423eaba797 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:52.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.528+0000 7ff5f7fff640 1 -- 172.21.15.33:0/3129520660 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60402c040 con 0x561ba8f75dc0 2024-09-16T00:36:52.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.528+0000 7ff5f7fff640 1 -- 172.21.15.33:0/3129520660 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff604005870 con 0x561ba8f75dc0 2024-09-16T00:36:52.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.528+0000 7ff5f7fff640 1 -- 172.21.15.33:0/3129520660 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604004f70 con 0x561ba8f75dc0 2024-09-16T00:36:52.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.529+0000 7ff616ed7740 1 -- 172.21.15.33:0/3129520660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 msgr2=0x561ba8fa0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:52.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.529+0000 7ff616ed7740 1 --2- 172.21.15.33:0/3129520660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8fa0390 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff6040055d0 tx=0x7ff604011a30 comp rx=0 tx=0).stop 2024-09-16T00:36:52.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.529+0000 7ff616ed7740 1 -- 172.21.15.33:0/3129520660 shutdown_connections 2024-09-16T00:36:52.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.529+0000 7ff616ed7740 1 --2- 172.21.15.33:0/3129520660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8fa08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.529+0000 7ff616ed7740 1 --2- 172.21.15.33:0/3129520660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8fa0390 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.529+0000 7ff616ed7740 1 --2- 172.21.15.33:0/3129520660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 0x561ba8f79e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.529+0000 7ff616ed7740 1 -- 172.21.15.33:0/3129520660 >> 172.21.15.33:0/3129520660 conn(0x561ba8f71a30 msgr2=0x561ba8f83dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:52.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.529+0000 7ff616ed7740 1 -- 172.21.15.33:0/3129520660 shutdown_connections 2024-09-16T00:36:52.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.529+0000 7ff616ed7740 1 -- 172.21.15.33:0/3129520660 wait complete. 2024-09-16T00:36:52.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.529+0000 7ff616ed7740 1 -- start start 2024-09-16T00:36:52.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.530+0000 7ff616ed7740 1 Processor -- start 2024-09-16T00:36:52.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.530+0000 7ff616ed7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 0x561ba8f904b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.530+0000 7ff616ed7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f909f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.530+0000 7ff616ed7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8f94ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.530+0000 7ff60dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f909f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.530+0000 7ff60dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f909f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40690/0 (socket says 172.21.15.33:40690) 2024-09-16T00:36:52.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.530+0000 7ff60dd9b640 1 -- 172.21.15.33:0/603943705 learned_addr learned my addr 172.21.15.33:0/603943705 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:52.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.530+0000 7ff616ed7740 1 -- 172.21.15.33:0/603943705 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101ab370 con 0x561ba8f80680 2024-09-16T00:36:52.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.530+0000 7ff616ed7740 1 -- 172.21.15.33:0/603943705 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101ac7e0 con 0x561ba8f754a0 2024-09-16T00:36:52.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.530+0000 7ff616ed7740 1 -- 172.21.15.33:0/603943705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101aba40 con 0x561ba8f75dc0 2024-09-16T00:36:52.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.530+0000 7ff616ed7740 10 client.0 _ll_register_callbacks cb 0x7ff600005750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:52.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.530+0000 7ff60e59c640 1 --2- 172.21.15.33:0/603943705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 0x561ba8f904b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.531+0000 7ff60dd9b640 1 -- 172.21.15.33:0/603943705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 msgr2=0x561ba8f904b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:52.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.531+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/603943705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 0x561ba8f904b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.531+0000 7ff60dd9b640 1 -- 172.21.15.33:0/603943705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 msgr2=0x561ba8f94ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:52.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.531+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/603943705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8f94ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.531+0000 7ff60dd9b640 1 -- 172.21.15.33:0/603943705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ba8f95160 con 0x561ba8f75dc0 2024-09-16T00:36:52.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.532+0000 7ff60dd9b640 1 --2- 172.21.15.33:0/603943705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f909f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff6040109a0 tx=0x7ff60400bb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a141759d1c22a6b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:52.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.532+0000 7ff5f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:36:52.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.532+0000 7ff5f5ffb640 1 -- 172.21.15.33:0/603943705 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604004a90 con 0x561ba8f75dc0 2024-09-16T00:36:52.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.532+0000 7ff5f5ffb640 1 -- 172.21.15.33:0/603943705 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff604004c30 con 0x561ba8f75dc0 2024-09-16T00:36:52.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.532+0000 7ff616ed7740 1 -- 172.21.15.33:0/603943705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff604005870 con 0x561ba8f75dc0 2024-09-16T00:36:52.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.533+0000 7ff5f5ffb640 1 -- 172.21.15.33:0/603943705 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff604004e30 con 0x561ba8f75dc0 2024-09-16T00:36:52.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.533+0000 7ff616ed7740 10 client.28342 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:52.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.533+0000 7ff616ed7740 1 -- 172.21.15.33:0/603943705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561ba8f87420 con 0x561ba8f75dc0 2024-09-16T00:36:52.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.533+0000 7ff5f5ffb640 1 -- 172.21.15.33:0/603943705 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff604005870 con 0x561ba8f75dc0 2024-09-16T00:36:52.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.533+0000 7ff5f5ffb640 1 -- 172.21.15.33:0/603943705 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ff6040268d0 con 0x561ba8f75dc0 2024-09-16T00:36:52.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.533+0000 7ff616ed7740 10 client.28342 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:52.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.533+0000 7ff616ed7740 10 client.28342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:36:52.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.533+0000 7ff616ed7740 20 client.28342 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:52.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.533+0000 7ff616ed7740 1 --2- 172.21.15.33:0/603943705 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561ba8f87b70 0x561ba8f89f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:52.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.534+0000 7ff616ed7740 4 client.28342 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:36:52.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.534+0000 7ff616ed7740 1 -- 172.21.15.33:0/603943705 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561ba8f78c20 con 0x561ba8f87b70 2024-09-16T00:36:52.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.534+0000 7ff60e59c640 1 --2- 172.21.15.33:0/603943705 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561ba8f87b70 0x561ba8f89f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:52.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.534+0000 7ff60e59c640 1 --2- 172.21.15.33:0/603943705 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561ba8f87b70 0x561ba8f89f90 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8812b4d8e7f5a8d9 server_cookie=1fb9db4146ec47a1 in_seq=0 out_seq=0 2024-09-16T00:36:52.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.534+0000 7ff5f5ffb640 10 client.28342 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:52.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.535+0000 7ff5f5ffb640 1 -- 172.21.15.33:0/603943705 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561ba8f78c20 con 0x561ba8f87b70 2024-09-16T00:36:52.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.535+0000 7ff5f5ffb640 10 client.28342 handle_command_reply: tid=0 2024-09-16T00:36:52.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.535+0000 7ff616ed7740 1 client.28342 shutdown 2024-09-16T00:36:52.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.535+0000 7ff616ed7740 1 -- 172.21.15.33:0/603943705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 msgr2=0x561ba8f909f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:52.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.535+0000 7ff616ed7740 1 --2- 172.21.15.33:0/603943705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f909f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff6040109a0 tx=0x7ff60400bb50 comp rx=0 tx=0).stop 2024-09-16T00:36:52.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.535+0000 7ff616ed7740 1 -- 172.21.15.33:0/603943705 shutdown_connections 2024-09-16T00:36:52.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.535+0000 7ff616ed7740 1 --2- 172.21.15.33:0/603943705 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561ba8f87b70 0x561ba8f89f90 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.535+0000 7ff5f5ffb640 0 client.28342 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:52.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.535+0000 7ff616ed7740 1 --2- 172.21.15.33:0/603943705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ba8f80680 0x561ba8f94ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.535+0000 7ff616ed7740 1 --2- 172.21.15.33:0/603943705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ba8f75dc0 0x561ba8f909f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.535+0000 7ff616ed7740 1 --2- 172.21.15.33:0/603943705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561ba8f754a0 0x561ba8f904b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.536+0000 7ff616ed7740 1 -- 172.21.15.33:0/603943705 >> 172.21.15.33:0/603943705 conn(0x561ba8f71a30 msgr2=0x561ba8f7ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:52.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.536+0000 7ff616ed7740 1 -- 172.21.15.33:0/603943705 shutdown_connections 2024-09-16T00:36:52.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.536+0000 7ff616ed7740 1 -- 172.21.15.33:0/603943705 wait complete. 2024-09-16T00:36:52.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.536+0000 7ff616ed7740 20 client.28342 trim_cache size 0 max 16384 2024-09-16T00:36:52.541 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:36:52.541 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:36:52.542 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:36:52.542 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:36:52.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.536+0000 7ff614afe640 1 -- 172.21.15.33:0/1417733291 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5f80a32c0 msgr2=0x7ff5f80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:52.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.536+0000 7ff614afe640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5f80a32c0 0x7ff5f80a5780 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff6080136d0 tx=0x7ff608013ad0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.536+0000 7ff614afe640 1 -- 172.21.15.33:0/1417733291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff610100e20 msgr2=0x7ff610198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:52.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.536+0000 7ff614afe640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff610100e20 0x7ff610198470 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff600006010 tx=0x7ff600003c10 comp rx=0 tx=0).stop 2024-09-16T00:36:52.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.537+0000 7ff614afe640 1 -- 172.21.15.33:0/1417733291 shutdown_connections 2024-09-16T00:36:52.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.537+0000 7ff614afe640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff5f80a32c0 0x7ff5f80a5780 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.537+0000 7ff614afe640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff610102ef0 0x7ff61019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.537+0000 7ff614afe640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff610102570 0x7ff6101989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.537+0000 7ff614afe640 1 --2- 172.21.15.33:0/1417733291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff610100e20 0x7ff610198470 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.537+0000 7ff614afe640 1 -- 172.21.15.33:0/1417733291 >> 172.21.15.33:0/1417733291 conn(0x7ff6100fc7d0 msgr2=0x7ff61006a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:52.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.537+0000 7ff614afe640 1 -- 172.21.15.33:0/1417733291 shutdown_connections 2024-09-16T00:36:52.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.537+0000 7ff614afe640 1 -- 172.21.15.33:0/1417733291 wait complete. 2024-09-16T00:36:52.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.753+0000 7f1ebeff5640 1 -- 172.21.15.33:0/3654775483 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x565302e49be0 con 0x565302f635b0 2024-09-16T00:36:52.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.753+0000 7f1ebeff5640 10 client.19071 handle_command_reply: tid=0 2024-09-16T00:36:52.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.753+0000 7f1ee3651740 1 client.19071 shutdown 2024-09-16T00:36:52.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.754+0000 7f1ee3651740 1 -- 172.21.15.33:0/3654775483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 msgr2=0x565302eec800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:52.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.754+0000 7f1ee3651740 1 --2- 172.21.15.33:0/3654775483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302eec800 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc098f80 tx=0x7f1ecc0d0760 comp rx=0 tx=0).stop 2024-09-16T00:36:52.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.754+0000 7f1ee3651740 1 -- 172.21.15.33:0/3654775483 shutdown_connections 2024-09-16T00:36:52.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.754+0000 7f1ee3651740 1 --2- 172.21.15.33:0/3654775483 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x565302f635b0 0x565302f659a0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.754+0000 7f1ee3651740 1 --2- 172.21.15.33:0/3654775483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x565302ee8a30 0x565302eed280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.754+0000 7f1ee3651740 1 --2- 172.21.15.33:0/3654775483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x565302ede170 0x565302eecd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.754+0000 7f1ee3651740 1 --2- 172.21.15.33:0/3654775483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565302edd850 0x565302eec800 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.754+0000 7f1ee3651740 1 -- 172.21.15.33:0/3654775483 >> 172.21.15.33:0/3654775483 conn(0x565302ed9de0 msgr2=0x565302ed86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:52.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.754+0000 7f1ebeff5640 0 client.19071 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:52.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.754+0000 7f1ee3651740 1 -- 172.21.15.33:0/3654775483 shutdown_connections 2024-09-16T00:36:52.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.754+0000 7f1ee3651740 1 -- 172.21.15.33:0/3654775483 wait complete. 2024-09-16T00:36:52.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.754+0000 7f1ee3651740 20 client.19071 trim_cache size 0 max 16384 2024-09-16T00:36:52.756 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:36:52.756 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:36:52.756 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:36:52.756 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 70, 2024-09-16T00:36:52.756 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:36:52.756 INFO:teuthology.orchestra.run.smithi033.stdout: "e0f4c493": { 2024-09-16T00:36:52.756 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 70, 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 148.2, 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:36:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:36:52.758 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:36:52.758 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:36:52.758 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:36:52.758 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:36:52.758 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:36:52.758 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:36:52.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.755+0000 7f1ee1278640 1 -- 172.21.15.33:0/3528323703 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1ec40a3240 msgr2=0x7f1ec40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:52.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.755+0000 7f1ee1278640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1ec40a3240 0x7f1ec40a5700 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4006010 tx=0x7f1ed4005b50 comp rx=0 tx=0).stop 2024-09-16T00:36:52.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.755+0000 7f1ee1278640 1 -- 172.21.15.33:0/3528323703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1edc102d30 msgr2=0x7f1edc100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:52.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.755+0000 7f1ee1278640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1edc102d30 0x7f1edc100990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00ebc0 tx=0x7f1ecc00c6a0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.755+0000 7f1ee1278640 1 -- 172.21.15.33:0/3528323703 shutdown_connections 2024-09-16T00:36:52.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.755+0000 7f1ee1278640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1ec40a3240 0x7f1ec40a5700 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.755+0000 7f1ee1278640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1edc1082a0 0x7f1edc101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.755+0000 7f1ee1278640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1edc1036e0 0x7f1edc100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.755+0000 7f1ee1278640 1 --2- 172.21.15.33:0/3528323703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1edc102d30 0x7f1edc100990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:52.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.755+0000 7f1ee1278640 1 -- 172.21.15.33:0/3528323703 >> 172.21.15.33:0/3528323703 conn(0x7f1edc0fc7d0 msgr2=0x7f1edc106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:52.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.755+0000 7f1ee1278640 1 -- 172.21.15.33:0/3528323703 shutdown_connections 2024-09-16T00:36:52.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:52.755+0000 7f1ee1278640 1 -- 172.21.15.33:0/3528323703 wait complete. 2024-09-16T00:36:53.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 19.95 MB/sec execute 239 sec latency 1400.306 ms 2024-09-16T00:36:53.252 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:36:53.252 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:36:53.938 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:53 smithi033 ceph-mon[31243]: pgmap v942: 129 pgs: 129 active+clean; 622 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.3 MiB/s wr, 363 op/s 2024-09-16T00:36:53.972 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'e0f4c493', quiesce took 5.3 seconds. Will release after: 22.849999999999998 2024-09-16T00:36:53.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:53 smithi060 ceph-mon[35185]: pgmap v942: 129 pgs: 129 active+clean; 622 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.3 MiB/s wr, 363 op/s 2024-09-16T00:36:54.040 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:54.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 19.87 MB/sec execute 240 sec latency 2400.392 ms 2024-09-16T00:36:54.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:53 smithi083 ceph-mon[35242]: pgmap v942: 129 pgs: 129 active+clean; 622 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.3 MiB/s wr, 363 op/s 2024-09-16T00:36:54.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.624+0000 7f1561c51640 1 -- 172.21.15.33:0/1272484997 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f155c106f10 msgr2=0x7f155c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:54.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.624+0000 7f1561c51640 1 --2- 172.21.15.33:0/1272484997 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f155c106f10 0x7f155c10ea90 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f1554009ca0 tx=0x7f155402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:36:54.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.624+0000 7f1561c51640 1 -- 172.21.15.33:0/1272484997 shutdown_connections 2024-09-16T00:36:54.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.624+0000 7f1561c51640 1 --2- 172.21.15.33:0/1272484997 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f155c106f10 0x7f155c10ea90 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:54.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.624+0000 7f1561c51640 1 --2- 172.21.15.33:0/1272484997 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f155c102790 0x7f155c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:54.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.624+0000 7f1561c51640 1 --2- 172.21.15.33:0/1272484997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c101de0 0x7f155c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:54.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.624+0000 7f1561c51640 1 -- 172.21.15.33:0/1272484997 >> 172.21.15.33:0/1272484997 conn(0x7f155c0fd8c0 msgr2=0x7f155c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:54.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.624+0000 7f1561c51640 1 -- 172.21.15.33:0/1272484997 shutdown_connections 2024-09-16T00:36:54.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.624+0000 7f1561c51640 1 -- 172.21.15.33:0/1272484997 wait complete. 2024-09-16T00:36:54.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.625+0000 7f1561c51640 1 Processor -- start 2024-09-16T00:36:54.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.625+0000 7f1561c51640 1 -- start start 2024-09-16T00:36:54.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.625+0000 7f1561c51640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c101de0 0x7f155c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:54.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.625+0000 7f1561c51640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f155c102790 0x7f155c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:54.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.625+0000 7f1561c51640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f155c106f10 0x7f155c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:54.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.625+0000 7f1561c51640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f155c111f70 con 0x7f155c102790 2024-09-16T00:36:54.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.625+0000 7f1561c51640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f155c111df0 con 0x7f155c101de0 2024-09-16T00:36:54.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.625+0000 7f1561c51640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f155c1120f0 con 0x7f155c106f10 2024-09-16T00:36:54.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.625+0000 7f155affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f155c102790 0x7f155c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:54.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f155b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c101de0 0x7f155c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:54.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f155affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f155c102790 0x7f155c198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47586/0 (socket says 172.21.15.33:47586) 2024-09-16T00:36:54.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f155affd640 1 -- 172.21.15.33:0/1092239985 learned_addr learned my addr 172.21.15.33:0/1092239985 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f155bfff640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f155c106f10 0x7f155c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f155affd640 1 -- 172.21.15.33:0/1092239985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f155c106f10 msgr2=0x7f155c19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f155affd640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f155c106f10 0x7f155c19cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f155affd640 1 -- 172.21.15.33:0/1092239985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c101de0 msgr2=0x7f155c198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f155affd640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c101de0 0x7f155c198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f155affd640 1 -- 172.21.15.33:0/1092239985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f155c19d3c0 con 0x7f155c102790 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f155affd640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f155c102790 0x7f155c198950 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f154c00d770 tx=0x7f154c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f1558ff9640 1 -- 172.21.15.33:0/1092239985 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c017070 con 0x7f155c102790 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f1558ff9640 1 -- 172.21.15.33:0/1092239985 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f154c005500 con 0x7f155c102790 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f1561c51640 1 -- 172.21.15.33:0/1092239985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f155c19d6b0 con 0x7f155c102790 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f1558ff9640 1 -- 172.21.15.33:0/1092239985 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c004e00 con 0x7f155c102790 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.626+0000 7f1561c51640 1 -- 172.21.15.33:0/1092239985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f155c1a4f90 con 0x7f155c102790 2024-09-16T00:36:54.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.627+0000 7f1561c51640 1 -- 172.21.15.33:0/1092239985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f155c1032c0 con 0x7f155c102790 2024-09-16T00:36:54.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.628+0000 7f1558ff9640 1 -- 172.21.15.33:0/1092239985 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f154c004fa0 con 0x7f155c102790 2024-09-16T00:36:54.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.629+0000 7f1558ff9640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15480a3240 0x7f15480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:54.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.629+0000 7f155b7fe640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15480a3240 0x7f15480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:54.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.629+0000 7f1558ff9640 1 -- 172.21.15.33:0/1092239985 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f154c0cfbf0 con 0x7f155c102790 2024-09-16T00:36:54.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.630+0000 7f155b7fe640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15480a3240 0x7f15480a5700 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f154400bbe0 tx=0x7f154400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:54.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.632+0000 7f1558ff9640 1 -- 172.21.15.33:0/1092239985 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f154c09c430 con 0x7f155c102790 2024-09-16T00:36:54.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.844+0000 7f1561c51640 1 -- 172.21.15.33:0/1092239985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f155c1021c0 con 0x7f155c102790 2024-09-16T00:36:54.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.844+0000 7f1558ff9640 1 -- 172.21.15.33:0/1092239985 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f154c09bd30 con 0x7f155c102790 2024-09-16T00:36:54.845 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:36:54.845 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:36:54.845 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:36:54.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.846+0000 7f1561c51640 1 -- 172.21.15.33:0/1092239985 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15480a3240 msgr2=0x7f15480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:54.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.846+0000 7f1561c51640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15480a3240 0x7f15480a5700 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f154400bbe0 tx=0x7f154400b390 comp rx=0 tx=0).stop 2024-09-16T00:36:54.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.846+0000 7f1561c51640 1 -- 172.21.15.33:0/1092239985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f155c102790 msgr2=0x7f155c198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:54.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.846+0000 7f1561c51640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f155c102790 0x7f155c198950 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f154c00d770 tx=0x7f154c00dc30 comp rx=0 tx=0).stop 2024-09-16T00:36:54.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.846+0000 7f1561c51640 1 -- 172.21.15.33:0/1092239985 shutdown_connections 2024-09-16T00:36:54.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.846+0000 7f1561c51640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f15480a3240 0x7f15480a5700 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:54.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.846+0000 7f1561c51640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f155c106f10 0x7f155c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:54.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.846+0000 7f1561c51640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f155c102790 0x7f155c198950 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:54.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.846+0000 7f1561c51640 1 --2- 172.21.15.33:0/1092239985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c101de0 0x7f155c198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:54.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.846+0000 7f1561c51640 1 -- 172.21.15.33:0/1092239985 >> 172.21.15.33:0/1092239985 conn(0x7f155c0fd8c0 msgr2=0x7f155c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:54.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.847+0000 7f1561c51640 1 -- 172.21.15.33:0/1092239985 shutdown_connections 2024-09-16T00:36:54.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:54.847+0000 7f1561c51640 1 -- 172.21.15.33:0/1092239985 wait complete. 2024-09-16T00:36:55.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 19.78 MB/sec execute 241 sec latency 3400.499 ms 2024-09-16T00:36:55.784 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:36:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:55 smithi060 ceph-mon[35185]: pgmap v943: 129 pgs: 129 active+clean; 625 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.3 MiB/s wr, 405 op/s 2024-09-16T00:36:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:55 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1092239985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:36:56.046 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:56.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:55 smithi033 ceph-mon[31243]: pgmap v943: 129 pgs: 129 active+clean; 625 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.3 MiB/s wr, 405 op/s 2024-09-16T00:36:56.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:55 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1092239985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:36:56.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 19.70 MB/sec execute 242 sec latency 4400.544 ms 2024-09-16T00:36:56.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:55 smithi083 ceph-mon[35242]: pgmap v943: 129 pgs: 129 active+clean; 625 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.3 MiB/s wr, 405 op/s 2024-09-16T00:36:56.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:55 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1092239985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:36:56.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.513+0000 7f355605c640 1 -- 172.21.15.33:0/384066587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35501027a0 msgr2=0x7f35501067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:56.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.513+0000 7f355605c640 1 --2- 172.21.15.33:0/384066587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35501027a0 0x7f35501067f0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f3544009900 tx=0x7f354402a410 comp rx=0 tx=0).stop 2024-09-16T00:36:56.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.513+0000 7f355605c640 1 -- 172.21.15.33:0/384066587 shutdown_connections 2024-09-16T00:36:56.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.513+0000 7f355605c640 1 --2- 172.21.15.33:0/384066587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3550106f20 0x7f355010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.513+0000 7f355605c640 1 --2- 172.21.15.33:0/384066587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35501027a0 0x7f35501067f0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.513+0000 7f355605c640 1 --2- 172.21.15.33:0/384066587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3550101df0 0x7f35501021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.513+0000 7f355605c640 1 -- 172.21.15.33:0/384066587 >> 172.21.15.33:0/384066587 conn(0x7f35500fd8b0 msgr2=0x7f35500ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:56.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.513+0000 7f355605c640 1 -- 172.21.15.33:0/384066587 shutdown_connections 2024-09-16T00:36:56.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.513+0000 7f355605c640 1 -- 172.21.15.33:0/384066587 wait complete. 2024-09-16T00:36:56.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f355605c640 1 Processor -- start 2024-09-16T00:36:56.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f355605c640 1 -- start start 2024-09-16T00:36:56.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f355605c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3550101df0 0x7f35501983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f355605c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35501027a0 0x7f35501988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f355605c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3550106f20 0x7f355019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f355605c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3550111f40 con 0x7f3550106f20 2024-09-16T00:36:56.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f355605c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3550111dc0 con 0x7f3550101df0 2024-09-16T00:36:56.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f355605c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f35501120c0 con 0x7f35501027a0 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f354ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3550106f20 0x7f355019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f354ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3550106f20 0x7f355019cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47610/0 (socket says 172.21.15.33:47610) 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f354ffff640 1 -- 172.21.15.33:0/1413964528 learned_addr learned my addr 172.21.15.33:0/1413964528 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f354f7fe640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3550101df0 0x7f35501983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.514+0000 7f354effd640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35501027a0 0x7f35501988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.515+0000 7f354ffff640 1 -- 172.21.15.33:0/1413964528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35501027a0 msgr2=0x7f35501988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.515+0000 7f354ffff640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35501027a0 0x7f35501988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.515+0000 7f354ffff640 1 -- 172.21.15.33:0/1413964528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3550101df0 msgr2=0x7f35501983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.515+0000 7f354ffff640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3550101df0 0x7f35501983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.515+0000 7f354ffff640 1 -- 172.21.15.33:0/1413964528 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355019d350 con 0x7f3550106f20 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.515+0000 7f354f7fe640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3550101df0 0x7f35501983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.515+0000 7f354ffff640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3550106f20 0x7f355019cc70 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f354800eaa0 tx=0x7f354800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.515+0000 7f354cff9640 1 -- 172.21.15.33:0/1413964528 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354800cdb0 con 0x7f3550106f20 2024-09-16T00:36:56.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.515+0000 7f354cff9640 1 -- 172.21.15.33:0/1413964528 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3548005020 con 0x7f3550106f20 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.515+0000 7f355605c640 1 -- 172.21.15.33:0/1413964528 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f355019d640 con 0x7f3550106f20 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.515+0000 7f354cff9640 1 -- 172.21.15.33:0/1413964528 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35480116b0 con 0x7f3550106f20 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.515+0000 7f355605c640 1 -- 172.21.15.33:0/1413964528 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35501a4f20 con 0x7f3550106f20 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.517+0000 7f354cff9640 1 -- 172.21.15.33:0/1413964528 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3548005560 con 0x7f3550106f20 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.517+0000 7f3558435740 1 Processor -- start 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f3558435740 1 -- start start 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f3558435740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4ce7b40 0x5643b4ce6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f3558435740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce65d0 0x5643b4ce6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f3558435740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4cf1370 0x5643b4cf3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f3558435740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5643b4c4c2f0 con 0x5643b4cf1370 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f3558435740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5643b4bf3cf0 con 0x5643b4ce7b40 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f3558435740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5643b4c482b0 con 0x5643b4ce65d0 2024-09-16T00:36:56.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f354effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4cf1370 0x5643b4cf3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f354effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4cf1370 0x5643b4cf3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47618/0 (socket says 172.21.15.33:47618) 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f354effd640 1 -- 172.21.15.33:0/4263817205 learned_addr learned my addr 172.21.15.33:0/4263817205 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f354f7fe640 1 --2- 172.21.15.33:0/4263817205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce65d0 0x5643b4ce6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f354effd640 1 --2- 172.21.15.33:0/4263817205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4ce7b40 0x5643b4ce6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f354effd640 1 -- 172.21.15.33:0/4263817205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce65d0 msgr2=0x5643b4ce6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f354effd640 1 --2- 172.21.15.33:0/4263817205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce65d0 0x5643b4ce6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f354effd640 1 -- 172.21.15.33:0/4263817205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4ce7b40 msgr2=0x5643b4ce6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f354effd640 1 --2- 172.21.15.33:0/4263817205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4ce7b40 0x5643b4ce6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f354effd640 1 -- 172.21.15.33:0/4263817205 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643b4bd83a0 con 0x5643b4cf1370 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f354cff9640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35380a32c0 0x7f35380a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.518+0000 7f354effd640 1 --2- 172.21.15.33:0/4263817205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4cf1370 0x5643b4cf3760 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f3544009900 tx=0x7f35440056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a985340aaa25259c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.519+0000 7f354f7fe640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35380a32c0 0x7f35380a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.519+0000 7f354cff9640 1 -- 172.21.15.33:0/1413964528 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f35480cffd0 con 0x7f3550106f20 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.519+0000 7f3535ffb640 1 -- 172.21.15.33:0/4263817205 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544035040 con 0x5643b4cf1370 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.519+0000 7f3535ffb640 1 -- 172.21.15.33:0/4263817205 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3544004030 con 0x5643b4cf1370 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.519+0000 7f3535ffb640 1 -- 172.21.15.33:0/4263817205 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544003be0 con 0x5643b4cf1370 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.519+0000 7f3558435740 1 -- 172.21.15.33:0/4263817205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4cf1370 msgr2=0x5643b4cf3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.519+0000 7f3558435740 1 --2- 172.21.15.33:0/4263817205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4cf1370 0x5643b4cf3760 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f3544009900 tx=0x7f35440056d0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.519+0000 7f354f7fe640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35380a32c0 0x7f35380a5780 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x5643b4ce3ac0 tx=0x7f35400084c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 -- 172.21.15.33:0/4263817205 shutdown_connections 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 --2- 172.21.15.33:0/4263817205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4cf1370 0x5643b4cf3760 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 --2- 172.21.15.33:0/4263817205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce65d0 0x5643b4ce6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 --2- 172.21.15.33:0/4263817205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4ce7b40 0x5643b4ce6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 -- 172.21.15.33:0/4263817205 >> 172.21.15.33:0/4263817205 conn(0x5643b4caf310 msgr2=0x5643b4ce2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 -- 172.21.15.33:0/4263817205 shutdown_connections 2024-09-16T00:36:56.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 -- 172.21.15.33:0/4263817205 wait complete. 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 -- start start 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 Processor -- start 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4ce65d0 0x5643b4cf0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 0x5643b4cee060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4cf1370 0x5643b4cee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5643b4cf4010 con 0x5643b4ce7b40 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5643b4c07060 con 0x5643b4ce65d0 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5643b4cf4190 con 0x5643b4cf1370 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.520+0000 7f3558435740 10 client.0 _ll_register_callbacks cb 0x5643b495a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 0x5643b4cee060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 0x5643b4cee060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47626/0 (socket says 172.21.15.33:47626) 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354effd640 1 -- 172.21.15.33:0/3501839761 learned_addr learned my addr 172.21.15.33:0/3501839761 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:56.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354f7fe640 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4ce65d0 0x5643b4cf0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354ffff640 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4cf1370 0x5643b4cee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354effd640 1 -- 172.21.15.33:0/3501839761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4cf1370 msgr2=0x5643b4cee7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354effd640 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4cf1370 0x5643b4cee7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354effd640 1 -- 172.21.15.33:0/3501839761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4ce65d0 msgr2=0x5643b4cf0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354effd640 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4ce65d0 0x5643b4cf0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354effd640 1 -- 172.21.15.33:0/3501839761 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643b49bd3c0 con 0x5643b4ce7b40 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354f7fe640 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4ce65d0 0x5643b4cf0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354ffff640 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4cf1370 0x5643b4cee7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f354effd640 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 0x5643b4cee060 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f3544006010 tx=0x7f35440058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440b7d660c7ebde1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f352effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f352effd640 1 -- 172.21.15.33:0/3501839761 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544050060 con 0x5643b4ce7b40 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f352effd640 1 -- 172.21.15.33:0/3501839761 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3544003b70 con 0x5643b4ce7b40 2024-09-16T00:36:56.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f3558435740 1 -- 172.21.15.33:0/3501839761 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3544004030 con 0x5643b4ce7b40 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f352effd640 1 -- 172.21.15.33:0/3501839761 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3544057050 con 0x5643b4ce7b40 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f3558435740 10 client.19101 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.521+0000 7f3558435740 1 -- 172.21.15.33:0/3501839761 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5643b4cffb00 con 0x5643b4ce7b40 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.522+0000 7f352effd640 1 -- 172.21.15.33:0/3501839761 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3544004030 con 0x5643b4ce7b40 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.522+0000 7f352effd640 1 -- 172.21.15.33:0/3501839761 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f354403a040 con 0x5643b4ce7b40 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.522+0000 7f3558435740 10 client.19101 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.522+0000 7f3558435740 10 client.19101 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.522+0000 7f3558435740 20 client.19101 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.522+0000 7f3558435740 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5643b4d00e50 0x5643b4d03240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.522+0000 7f3558435740 4 client.19101 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.522+0000 7f3558435740 1 -- 172.21.15.33:0/3501839761 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643b4c59860 con 0x5643b4d00e50 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.522+0000 7f354f7fe640 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5643b4d00e50 0x5643b4d03240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.523+0000 7f354f7fe640 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5643b4d00e50 0x5643b4d03240 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7998bdfb4f60622c server_cookie=2b760411f435a3a in_seq=0 out_seq=0 2024-09-16T00:36:56.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.523+0000 7f352effd640 10 client.19101 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.523+0000 7f352effd640 1 -- 172.21.15.33:0/3501839761 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5643b4c59860 con 0x5643b4d00e50 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.523+0000 7f352effd640 10 client.19101 handle_command_reply: tid=0 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f3558435740 1 client.19101 shutdown 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f3558435740 1 -- 172.21.15.33:0/3501839761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 msgr2=0x5643b4cee060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f3558435740 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 0x5643b4cee060 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f3544006010 tx=0x7f35440058b0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f3558435740 1 -- 172.21.15.33:0/3501839761 shutdown_connections 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f3558435740 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5643b4d00e50 0x5643b4d03240 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f3558435740 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4cf1370 0x5643b4cee7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f3558435740 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 0x5643b4cee060 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f3558435740 1 --2- 172.21.15.33:0/3501839761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4ce65d0 0x5643b4cf0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f352effd640 0 client.19101 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f3558435740 1 -- 172.21.15.33:0/3501839761 >> 172.21.15.33:0/3501839761 conn(0x5643b4caf310 msgr2=0x5643b4ce8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f3558435740 1 -- 172.21.15.33:0/3501839761 shutdown_connections 2024-09-16T00:36:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f3558435740 1 -- 172.21.15.33:0/3501839761 wait complete. 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.524+0000 7f3558435740 20 client.19101 trim_cache size 0 max 16384 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f3558435740 1 Processor -- start 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f3558435740 1 -- start start 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f3558435740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce65d0 0x5643b4ce2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f3558435740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 0x5643b4ced040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f3558435740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4cf1370 0x5643b4d06260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f3558435740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f35501b1880 con 0x5643b4ce7b40 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f3558435740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f35501af150 con 0x5643b4cf1370 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f3558435740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f35501acfa0 con 0x5643b4ce65d0 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f354effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 0x5643b4ced040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f354effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 0x5643b4ced040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47642/0 (socket says 172.21.15.33:47642) 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f354effd640 1 -- 172.21.15.33:0/1232923213 learned_addr learned my addr 172.21.15.33:0/1232923213 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:56.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f354ffff640 1 --2- 172.21.15.33:0/1232923213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4cf1370 0x5643b4d06260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.537+0000 7f354f7fe640 1 --2- 172.21.15.33:0/1232923213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce65d0 0x5643b4ce2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f354effd640 1 -- 172.21.15.33:0/1232923213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce65d0 msgr2=0x5643b4ce2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f354effd640 1 --2- 172.21.15.33:0/1232923213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce65d0 0x5643b4ce2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f354effd640 1 -- 172.21.15.33:0/1232923213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4cf1370 msgr2=0x5643b4d06260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f354effd640 1 --2- 172.21.15.33:0/1232923213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4cf1370 0x5643b4d06260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f354effd640 1 -- 172.21.15.33:0/1232923213 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643b4ceeb20 con 0x5643b4ce7b40 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f354f7fe640 1 --2- 172.21.15.33:0/1232923213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce65d0 0x5643b4ce2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f354ffff640 1 --2- 172.21.15.33:0/1232923213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4cf1370 0x5643b4d06260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f354effd640 1 --2- 172.21.15.33:0/1232923213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 0x5643b4ced040 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f3544009900 tx=0x7f3544003880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=111ab1861a8f35ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f35357fa640 1 -- 172.21.15.33:0/1232923213 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544050040 con 0x5643b4ce7b40 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f35357fa640 1 -- 172.21.15.33:0/1232923213 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35440483f0 con 0x5643b4ce7b40 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f35357fa640 1 -- 172.21.15.33:0/1232923213 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544035030 con 0x5643b4ce7b40 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f3558435740 1 -- 172.21.15.33:0/1232923213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 msgr2=0x5643b4ced040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:56.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.538+0000 7f3558435740 1 --2- 172.21.15.33:0/1232923213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 0x5643b4ced040 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f3544009900 tx=0x7f3544003880 comp rx=0 tx=0).stop 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 -- 172.21.15.33:0/1232923213 shutdown_connections 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 --2- 172.21.15.33:0/1232923213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4cf1370 0x5643b4d06260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 --2- 172.21.15.33:0/1232923213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce7b40 0x5643b4ced040 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 --2- 172.21.15.33:0/1232923213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce65d0 0x5643b4ce2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 -- 172.21.15.33:0/1232923213 >> 172.21.15.33:0/1232923213 conn(0x5643b4caf310 msgr2=0x5643b4cec930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 -- 172.21.15.33:0/1232923213 shutdown_connections 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 -- 172.21.15.33:0/1232923213 wait complete. 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 -- start start 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 Processor -- start 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce65d0 0x5643b4d62ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce7b40 0x5643b4d63020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4cf1370 0x5643b4d03810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f35501ad610 con 0x5643b4ce65d0 2024-09-16T00:36:56.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f35501aea80 con 0x5643b4cf1370 2024-09-16T00:36:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f35501adce0 con 0x5643b4ce7b40 2024-09-16T00:36:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.539+0000 7f3558435740 10 client.0 _ll_register_callbacks cb 0x7f3540008390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.540+0000 7f354f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce65d0 0x5643b4d62ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.540+0000 7f354f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce65d0 0x5643b4d62ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47646/0 (socket says 172.21.15.33:47646) 2024-09-16T00:36:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.540+0000 7f354f7fe640 1 -- 172.21.15.33:0/2251000560 learned_addr learned my addr 172.21.15.33:0/2251000560 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.540+0000 7f354ffff640 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4cf1370 0x5643b4d03810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.540+0000 7f354effd640 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce7b40 0x5643b4d63020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.540+0000 7f354f7fe640 1 -- 172.21.15.33:0/2251000560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce7b40 msgr2=0x5643b4d63020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.540+0000 7f354f7fe640 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce7b40 0x5643b4d63020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.540+0000 7f354f7fe640 1 -- 172.21.15.33:0/2251000560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4cf1370 msgr2=0x5643b4d03810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.540+0000 7f354f7fe640 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4cf1370 0x5643b4d03810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.540+0000 7f354f7fe640 1 -- 172.21.15.33:0/2251000560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643b4d04010 con 0x5643b4ce65d0 2024-09-16T00:36:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.540+0000 7f354f7fe640 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce65d0 0x5643b4d62ae0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f3540004140 tx=0x7f35400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b2a6fd6518a9cc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.540+0000 7f352effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f352effd640 1 -- 172.21.15.33:0/2251000560 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354002b060 con 0x5643b4ce65d0 2024-09-16T00:36:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f352effd640 1 -- 172.21.15.33:0/2251000560 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f35400262e0 con 0x5643b4ce65d0 2024-09-16T00:36:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f3558435740 1 -- 172.21.15.33:0/2251000560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35440483f0 con 0x5643b4ce65d0 2024-09-16T00:36:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f352effd640 1 -- 172.21.15.33:0/2251000560 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3540032050 con 0x5643b4ce65d0 2024-09-16T00:36:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f3558435740 10 client.19107 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f3558435740 1 -- 172.21.15.33:0/2251000560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5643b4d6b270 con 0x5643b4ce65d0 2024-09-16T00:36:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f352effd640 1 -- 172.21.15.33:0/2251000560 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35440483f0 con 0x5643b4ce65d0 2024-09-16T00:36:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f352effd640 1 -- 172.21.15.33:0/2251000560 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f354001d860 con 0x5643b4ce65d0 2024-09-16T00:36:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f3558435740 10 client.19107 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:56.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f3558435740 10 client.19107 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:36:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f3558435740 20 client.19107 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f3558435740 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5643b4d6b940 0x5643b4d6dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f3558435740 4 client.19107 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:36:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.541+0000 7f3558435740 1 -- 172.21.15.33:0/2251000560 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5643b4c53d70 con 0x5643b4d6b940 2024-09-16T00:36:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f354effd640 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5643b4d6b940 0x5643b4d6dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f354effd640 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5643b4d6b940 0x5643b4d6dda0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=42509ed7ec977b69 server_cookie=e2fc8786565c79b3 in_seq=0 out_seq=0 2024-09-16T00:36:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f352effd640 10 client.19107 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:36:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f352effd640 1 -- 172.21.15.33:0/2251000560 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5643b4c53d70 con 0x5643b4d6b940 2024-09-16T00:36:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f352effd640 10 client.19107 handle_command_reply: tid=0 2024-09-16T00:36:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f3558435740 1 client.19107 shutdown 2024-09-16T00:36:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f3558435740 1 -- 172.21.15.33:0/2251000560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce65d0 msgr2=0x5643b4d62ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:56.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f3558435740 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce65d0 0x5643b4d62ae0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f3540004140 tx=0x7f35400046c0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f3558435740 1 -- 172.21.15.33:0/2251000560 shutdown_connections 2024-09-16T00:36:56.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f3558435740 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5643b4d6b940 0x5643b4d6dda0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f3558435740 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643b4cf1370 0x5643b4d03810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f3558435740 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643b4ce7b40 0x5643b4d63020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f3558435740 1 --2- 172.21.15.33:0/2251000560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5643b4ce65d0 0x5643b4d62ae0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f352effd640 0 client.19107 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:36:56.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.542+0000 7f3558435740 1 -- 172.21.15.33:0/2251000560 >> 172.21.15.33:0/2251000560 conn(0x5643b4caf310 msgr2=0x5643b4ce8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:56.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f3558435740 1 -- 172.21.15.33:0/2251000560 shutdown_connections 2024-09-16T00:36:56.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f3558435740 1 -- 172.21.15.33:0/2251000560 wait complete. 2024-09-16T00:36:56.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f3558435740 20 client.19107 trim_cache size 0 max 16384 2024-09-16T00:36:56.559 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:36:56.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f355605c640 1 -- 172.21.15.33:0/1413964528 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35380a32c0 msgr2=0x7f35380a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:56.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f355605c640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35380a32c0 0x7f35380a5780 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x5643b4ce3ac0 tx=0x7f35400084c0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f355605c640 1 -- 172.21.15.33:0/1413964528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3550106f20 msgr2=0x7f355019cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:56.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f355605c640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3550106f20 0x7f355019cc70 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f354800eaa0 tx=0x7f354800ef60 comp rx=0 tx=0).stop 2024-09-16T00:36:56.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f355605c640 1 -- 172.21.15.33:0/1413964528 shutdown_connections 2024-09-16T00:36:56.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f355605c640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f35380a32c0 0x7f35380a5780 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f355605c640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3550106f20 0x7f355019cc70 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f355605c640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35501027a0 0x7f35501988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f355605c640 1 --2- 172.21.15.33:0/1413964528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3550101df0 0x7f35501983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.543+0000 7f355605c640 1 -- 172.21.15.33:0/1413964528 >> 172.21.15.33:0/1413964528 conn(0x7f35500fd8b0 msgr2=0x7f35500fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.544+0000 7f355605c640 1 -- 172.21.15.33:0/1413964528 shutdown_connections 2024-09-16T00:36:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:56.544+0000 7f355605c640 1 -- 172.21.15.33:0/1413964528 wait complete. 2024-09-16T00:36:57.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 19.62 MB/sec execute 243 sec latency 5400.651 ms 2024-09-16T00:36:57.321 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:36:57.577 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:57 smithi060 ceph-mon[35185]: pgmap v944: 129 pgs: 129 active+clean; 619 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.2 MiB/s wr, 379 op/s 2024-09-16T00:36:57.981 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:57 smithi033 ceph-mon[31243]: pgmap v944: 129 pgs: 129 active+clean; 619 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.2 MiB/s wr, 379 op/s 2024-09-16T00:36:58.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 19.54 MB/sec execute 244 sec latency 6400.685 ms 2024-09-16T00:36:58.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.065+0000 7f9036cb6640 1 -- 172.21.15.33:0/4179882213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903010c170 msgr2=0x7f903010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:58.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.065+0000 7f9036cb6640 1 --2- 172.21.15.33:0/4179882213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903010c170 0x7f903010e610 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f9028009ca0 tx=0x7f902802a950 comp rx=0 tx=0).stop 2024-09-16T00:36:58.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.065+0000 7f9036cb6640 1 -- 172.21.15.33:0/4179882213 shutdown_connections 2024-09-16T00:36:58.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.065+0000 7f9036cb6640 1 --2- 172.21.15.33:0/4179882213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903010c170 0x7f903010e610 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.065+0000 7f9036cb6640 1 --2- 172.21.15.33:0/4179882213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9030103660 0x7f903010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.065+0000 7f9036cb6640 1 --2- 172.21.15.33:0/4179882213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9030102d40 0x7f9030103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.065+0000 7f9036cb6640 1 -- 172.21.15.33:0/4179882213 >> 172.21.15.33:0/4179882213 conn(0x7f90300fc7a0 msgr2=0x7f90300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.065+0000 7f9036cb6640 1 -- 172.21.15.33:0/4179882213 shutdown_connections 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.065+0000 7f9036cb6640 1 -- 172.21.15.33:0/4179882213 wait complete. 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9036cb6640 1 Processor -- start 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9036cb6640 1 -- start start 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9036cb6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9030102d40 0x7f90301983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9036cb6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9030103660 0x7f9030198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9036cb6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f903010c170 0x7f903019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9036cb6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9030111fb0 con 0x7f9030102d40 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9036cb6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9030111e30 con 0x7f9030103660 2024-09-16T00:36:58.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9036cb6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9030112130 con 0x7f903010c170 2024-09-16T00:36:58.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9034a52640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9030102d40 0x7f90301983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9034a52640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9030102d40 0x7f90301983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47660/0 (socket says 172.21.15.33:47660) 2024-09-16T00:36:58.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9034a52640 1 -- 172.21.15.33:0/2199748120 learned_addr learned my addr 172.21.15.33:0/2199748120 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:58.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f902ffff640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9030103660 0x7f9030198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9035253640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f903010c170 0x7f903019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9034a52640 1 -- 172.21.15.33:0/2199748120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f903010c170 msgr2=0x7f903019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:58.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9034a52640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f903010c170 0x7f903019cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9034a52640 1 -- 172.21.15.33:0/2199748120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9030103660 msgr2=0x7f9030198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:58.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9034a52640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9030103660 0x7f9030198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.066+0000 7f9034a52640 1 -- 172.21.15.33:0/2199748120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903019d380 con 0x7f9030102d40 2024-09-16T00:36:58.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.067+0000 7f9034a52640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9030102d40 0x7f90301983d0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f902000b4a0 tx=0x7f902000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:58.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.067+0000 7f902dffb640 1 -- 172.21.15.33:0/2199748120 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020014070 con 0x7f9030102d40 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.067+0000 7f902dffb640 1 -- 172.21.15.33:0/2199748120 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9020010af0 con 0x7f9030102d40 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.067+0000 7f9036cb6640 1 -- 172.21.15.33:0/2199748120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f903019d670 con 0x7f9030102d40 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.067+0000 7f902dffb640 1 -- 172.21.15.33:0/2199748120 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020017d10 con 0x7f9030102d40 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.067+0000 7f9036cb6640 1 -- 172.21.15.33:0/2199748120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90301a4f50 con 0x7f9030102d40 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.069+0000 7f902dffb640 1 -- 172.21.15.33:0/2199748120 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9020017070 con 0x7f9030102d40 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.069+0000 7f903908f740 1 Processor -- start 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f903908f740 1 -- start start 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f903908f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 0x5568db1f6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f903908f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db1f65d0 0x5568db1f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f903908f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db201370 0x5568db203760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f903908f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5568db15c2f0 con 0x5568db1f7b40 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f903908f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5568db103cf0 con 0x5568db201370 2024-09-16T00:36:58.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f903908f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5568db1582b0 con 0x5568db1f65d0 2024-09-16T00:36:58.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f902dffb640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f90180a3240 0x7f90180a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f9035253640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 0x5568db1f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f9035253640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 0x5568db1f6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47674/0 (socket says 172.21.15.33:47674) 2024-09-16T00:36:58.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f9035253640 1 -- 172.21.15.33:0/2951920546 learned_addr learned my addr 172.21.15.33:0/2951920546 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:58.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f9034a52640 1 --2- 172.21.15.33:0/2951920546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db1f65d0 0x5568db1f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f902ffff640 1 --2- 172.21.15.33:0/2951920546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db201370 0x5568db203760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f9035253640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f90180a3240 0x7f90180a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f902dffb640 1 -- 172.21.15.33:0/2199748120 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f90200cf110 con 0x7f9030102d40 2024-09-16T00:36:58.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f9035253640 1 -- 172.21.15.33:0/2951920546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db1f65d0 msgr2=0x5568db1f6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:58.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f9035253640 1 --2- 172.21.15.33:0/2951920546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db1f65d0 0x5568db1f6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f9035253640 1 -- 172.21.15.33:0/2951920546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db201370 msgr2=0x5568db203760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:58.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f9035253640 1 --2- 172.21.15.33:0/2951920546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db201370 0x5568db203760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.070+0000 7f9035253640 1 -- 172.21.15.33:0/2951920546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568db0e83a0 con 0x5568db1f7b40 2024-09-16T00:36:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.071+0000 7f9035253640 1 --2- 172.21.15.33:0/2951920546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 0x5568db1f6090 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f9028009ca0 tx=0x7f902802f4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21b04246ef0f5ff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.071+0000 7f9016ffd640 1 -- 172.21.15.33:0/2951920546 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9028043050 con 0x5568db1f7b40 2024-09-16T00:36:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.071+0000 7f9016ffd640 1 -- 172.21.15.33:0/2951920546 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9028044040 con 0x5568db1f7b40 2024-09-16T00:36:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.071+0000 7f9016ffd640 1 -- 172.21.15.33:0/2951920546 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902804d020 con 0x5568db1f7b40 2024-09-16T00:36:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.071+0000 7f903908f740 1 -- 172.21.15.33:0/2951920546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 msgr2=0x5568db1f6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.071+0000 7f9035253640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f90180a3240 0x7f90180a5700 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f902804e000 tx=0x7f902804c000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.071+0000 7f903908f740 1 --2- 172.21.15.33:0/2951920546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 0x5568db1f6090 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f9028009ca0 tx=0x7f902802f4d0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.071+0000 7f903908f740 1 -- 172.21.15.33:0/2951920546 shutdown_connections 2024-09-16T00:36:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.071+0000 7f903908f740 1 --2- 172.21.15.33:0/2951920546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db201370 0x5568db203760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.071+0000 7f903908f740 1 --2- 172.21.15.33:0/2951920546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db1f65d0 0x5568db1f6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.071+0000 7f903908f740 1 --2- 172.21.15.33:0/2951920546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 0x5568db1f6090 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.072+0000 7f903908f740 1 -- 172.21.15.33:0/2951920546 >> 172.21.15.33:0/2951920546 conn(0x5568db1bf310 msgr2=0x5568db1f2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.072+0000 7f903908f740 1 -- 172.21.15.33:0/2951920546 shutdown_connections 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.072+0000 7f903908f740 1 -- 172.21.15.33:0/2951920546 wait complete. 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.072+0000 7f903908f740 1 -- start start 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.072+0000 7f903908f740 1 Processor -- start 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.072+0000 7f903908f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f65d0 0x5568db200d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.072+0000 7f903908f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db1f7b40 0x5568db1fe010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.072+0000 7f903908f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db201370 0x5568db1fe600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.072+0000 7f903908f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5568db204070 con 0x5568db201370 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.072+0000 7f903908f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5568db117060 con 0x5568db1f65d0 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.072+0000 7f903908f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5568db2041f0 con 0x5568db1f7b40 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.072+0000 7f903908f740 10 client.0 _ll_register_callbacks cb 0x5568dae6a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:58.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f9035253640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db201370 0x5568db1fe600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f9035253640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db201370 0x5568db1fe600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47686/0 (socket says 172.21.15.33:47686) 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f9035253640 1 -- 172.21.15.33:0/2024668764 learned_addr learned my addr 172.21.15.33:0/2024668764 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f9034a52640 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f65d0 0x5568db200d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f902ffff640 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db1f7b40 0x5568db1fe010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f9035253640 1 -- 172.21.15.33:0/2024668764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db1f7b40 msgr2=0x5568db1fe010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f9035253640 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db1f7b40 0x5568db1fe010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f9035253640 1 -- 172.21.15.33:0/2024668764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f65d0 msgr2=0x5568db200d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f9035253640 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f65d0 0x5568db200d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f9035253640 1 -- 172.21.15.33:0/2024668764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568daecd3c0 con 0x5568db201370 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f9034a52640 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f65d0 0x5568db200d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f9035253640 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db201370 0x5568db1fe600 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f902802a990 tx=0x7f9028004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c660bf4cd9ee723 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f900ffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:58.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f900ffff640 1 -- 172.21.15.33:0/2024668764 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902804d030 con 0x5568db201370 2024-09-16T00:36:58.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f900ffff640 1 -- 172.21.15.33:0/2024668764 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f90280498b0 con 0x5568db201370 2024-09-16T00:36:58.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f903908f740 1 -- 172.21.15.33:0/2024668764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9028044040 con 0x5568db201370 2024-09-16T00:36:58.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f900ffff640 1 -- 172.21.15.33:0/2024668764 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9028063050 con 0x5568db201370 2024-09-16T00:36:58.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f903908f740 10 client.19119 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:58.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.073+0000 7f903908f740 1 -- 172.21.15.33:0/2024668764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5568db213f20 con 0x5568db201370 2024-09-16T00:36:58.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.074+0000 7f900ffff640 1 -- 172.21.15.33:0/2024668764 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9028044040 con 0x5568db201370 2024-09-16T00:36:58.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.074+0000 7f900ffff640 1 -- 172.21.15.33:0/2024668764 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f90280307b0 con 0x5568db201370 2024-09-16T00:36:58.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.074+0000 7f903908f740 10 client.19119 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:58.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.074+0000 7f903908f740 10 client.19119 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:36:58.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.074+0000 7f903908f740 20 client.19119 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:58.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.074+0000 7f903908f740 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5568db215270 0x5568db217660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.074+0000 7f903908f740 4 client.19119 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.074+0000 7f903908f740 1 -- 172.21.15.33:0/2024668764 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568db169860 con 0x5568db215270 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.074+0000 7f9034a52640 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5568db215270 0x5568db217660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.075+0000 7f9034a52640 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5568db215270 0x5568db217660 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9b781cbd676fb24b server_cookie=b814a16ec4954119 in_seq=0 out_seq=0 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.075+0000 7f900ffff640 10 client.19119 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.075+0000 7f900ffff640 1 -- 172.21.15.33:0/2024668764 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5568db169860 con 0x5568db215270 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.075+0000 7f900ffff640 10 client.19119 handle_command_reply: tid=0 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.075+0000 7f903908f740 1 client.19119 shutdown 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.076+0000 7f903908f740 1 -- 172.21.15.33:0/2024668764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db201370 msgr2=0x5568db1fe600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.076+0000 7f903908f740 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db201370 0x5568db1fe600 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f902802a990 tx=0x7f9028004690 comp rx=0 tx=0).stop 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.076+0000 7f903908f740 1 -- 172.21.15.33:0/2024668764 shutdown_connections 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.076+0000 7f903908f740 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db201370 0x5568db1fe600 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.076+0000 7f903908f740 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db1f7b40 0x5568db1fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.076+0000 7f903908f740 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5568db215270 0x5568db217660 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.076+0000 7f903908f740 1 --2- 172.21.15.33:0/2024668764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f65d0 0x5568db200d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.076+0000 7f900ffff640 0 client.19119 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:36:58.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.076+0000 7f903908f740 1 -- 172.21.15.33:0/2024668764 >> 172.21.15.33:0/2024668764 conn(0x5568db1bf310 msgr2=0x5568db1f8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:58.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.076+0000 7f903908f740 1 -- 172.21.15.33:0/2024668764 shutdown_connections 2024-09-16T00:36:58.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.076+0000 7f903908f740 1 -- 172.21.15.33:0/2024668764 wait complete. 2024-09-16T00:36:58.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.076+0000 7f903908f740 20 client.19119 trim_cache size 0 max 16384 2024-09-16T00:36:58.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.089+0000 7f903908f740 1 Processor -- start 2024-09-16T00:36:58.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.089+0000 7f903908f740 1 -- start start 2024-09-16T00:36:58.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.089+0000 7f903908f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f65d0 0x5568db20d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.089+0000 7f903908f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 0x5568db1f9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.089+0000 7f903908f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db201370 0x5568db1f9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f903908f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f90301af6b0 con 0x5568db1f7b40 2024-09-16T00:36:58.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f903908f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f90301acf80 con 0x5568db1f65d0 2024-09-16T00:36:58.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f903908f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f90301aadd0 con 0x5568db201370 2024-09-16T00:36:58.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f902ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 0x5568db1f9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f902ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 0x5568db1f9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47694/0 (socket says 172.21.15.33:47694) 2024-09-16T00:36:58.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f902ffff640 1 -- 172.21.15.33:0/3111233832 learned_addr learned my addr 172.21.15.33:0/3111233832 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:58.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f9035253640 1 --2- 172.21.15.33:0/3111233832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db201370 0x5568db1f9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f9034a52640 1 --2- 172.21.15.33:0/3111233832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f65d0 0x5568db20d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f902ffff640 1 -- 172.21.15.33:0/3111233832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db201370 msgr2=0x5568db1f9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f902ffff640 1 --2- 172.21.15.33:0/3111233832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db201370 0x5568db1f9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f902ffff640 1 -- 172.21.15.33:0/3111233832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f65d0 msgr2=0x5568db20d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f902ffff640 1 --2- 172.21.15.33:0/3111233832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f65d0 0x5568db20d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f902ffff640 1 -- 172.21.15.33:0/3111233832 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568db1fc5c0 con 0x5568db1f7b40 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.090+0000 7f9034a52640 1 --2- 172.21.15.33:0/3111233832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f65d0 0x5568db20d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.091+0000 7f902ffff640 1 --2- 172.21.15.33:0/3111233832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 0x5568db1f9120 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x5568db1f7070 tx=0x7f902400af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9804b1bc58513d77 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.091+0000 7f90167fc640 1 -- 172.21.15.33:0/3111233832 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9024012d50 con 0x5568db1f7b40 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.091+0000 7f90167fc640 1 -- 172.21.15.33:0/3111233832 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9024005500 con 0x5568db1f7b40 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.091+0000 7f90167fc640 1 -- 172.21.15.33:0/3111233832 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902400f680 con 0x5568db1f7b40 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.091+0000 7f903908f740 1 -- 172.21.15.33:0/3111233832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 msgr2=0x5568db1f9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.091+0000 7f903908f740 1 --2- 172.21.15.33:0/3111233832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 0x5568db1f9120 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x5568db1f7070 tx=0x7f902400af00 comp rx=0 tx=0).stop 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 -- 172.21.15.33:0/3111233832 shutdown_connections 2024-09-16T00:36:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 --2- 172.21.15.33:0/3111233832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db201370 0x5568db1f9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 --2- 172.21.15.33:0/3111233832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f7b40 0x5568db1f9120 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 --2- 172.21.15.33:0/3111233832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f65d0 0x5568db20d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 -- 172.21.15.33:0/3111233832 >> 172.21.15.33:0/3111233832 conn(0x5568db1bf310 msgr2=0x5568db200090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 -- 172.21.15.33:0/3111233832 shutdown_connections 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 -- 172.21.15.33:0/3111233832 wait complete. 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 -- start start 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 Processor -- start 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f65d0 0x5568db209c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f7b40 0x5568db1fae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db201370 0x5568db1fb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f90301ab440 con 0x5568db1f65d0 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f90301ac8b0 con 0x5568db1f7b40 2024-09-16T00:36:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f90301abb10 con 0x5568db201370 2024-09-16T00:36:58.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.092+0000 7f903908f740 10 client.0 _ll_register_callbacks cb 0x7f9020017eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:58.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f9034a52640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f65d0 0x5568db209c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f9034a52640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f65d0 0x5568db209c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47710/0 (socket says 172.21.15.33:47710) 2024-09-16T00:36:58.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f9034a52640 1 -- 172.21.15.33:0/1590832991 learned_addr learned my addr 172.21.15.33:0/1590832991 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:58.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f902ffff640 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f7b40 0x5568db1fae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f9035253640 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db201370 0x5568db1fb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f9034a52640 1 -- 172.21.15.33:0/1590832991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db201370 msgr2=0x5568db1fb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:58.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f9034a52640 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db201370 0x5568db1fb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f9034a52640 1 -- 172.21.15.33:0/1590832991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f7b40 msgr2=0x5568db1fae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:58.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f9034a52640 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f7b40 0x5568db1fae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f9034a52640 1 -- 172.21.15.33:0/1590832991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568db1fbbe0 con 0x5568db1f65d0 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f9034a52640 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f65d0 0x5568db209c70 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f9020009420 tx=0x7f902001ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c886083d0cf8f6cf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f900ffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f900ffff640 1 -- 172.21.15.33:0/1590832991 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902001e040 con 0x5568db1f65d0 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f900ffff640 1 -- 172.21.15.33:0/1590832991 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f90200cc840 con 0x5568db1f65d0 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f903908f740 1 -- 172.21.15.33:0/1590832991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9024005500 con 0x5568db1f65d0 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f900ffff640 1 -- 172.21.15.33:0/1590832991 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9020017360 con 0x5568db1f65d0 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f903908f740 10 client.19125 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.093+0000 7f903908f740 1 -- 172.21.15.33:0/1590832991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5568db27fd60 con 0x5568db1f65d0 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.094+0000 7f900ffff640 1 -- 172.21.15.33:0/1590832991 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9024005500 con 0x5568db1f65d0 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.094+0000 7f900ffff640 1 -- 172.21.15.33:0/1590832991 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f9020097970 con 0x5568db1f65d0 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.094+0000 7f903908f740 10 client.19125 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:58.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.094+0000 7f903908f740 10 client.19125 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.094+0000 7f903908f740 20 client.19125 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.094+0000 7f903908f740 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5568db2803e0 0x5568db282810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.094+0000 7f903908f740 4 client.19125 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.094+0000 7f903908f740 1 -- 172.21.15.33:0/1590832991 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5568db163d70 con 0x5568db2803e0 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.094+0000 7f902ffff640 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5568db2803e0 0x5568db282810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.094+0000 7f902ffff640 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5568db2803e0 0x5568db282810 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4685550c9d114a0c server_cookie=e4ddf8ce3e7f1271 in_seq=0 out_seq=0 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.094+0000 7f900ffff640 10 client.19125 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f900ffff640 1 -- 172.21.15.33:0/1590832991 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5568db163d70 con 0x5568db2803e0 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f900ffff640 10 client.19125 handle_command_reply: tid=0 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f903908f740 1 client.19125 shutdown 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f903908f740 1 -- 172.21.15.33:0/1590832991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f65d0 msgr2=0x5568db209c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f903908f740 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f65d0 0x5568db209c70 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f9020009420 tx=0x7f902001ec80 comp rx=0 tx=0).stop 2024-09-16T00:36:58.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f903908f740 1 -- 172.21.15.33:0/1590832991 shutdown_connections 2024-09-16T00:36:58.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f903908f740 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568db201370 0x5568db1fb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f903908f740 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5568db2803e0 0x5568db282810 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f903908f740 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5568db1f7b40 0x5568db1fae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f903908f740 1 --2- 172.21.15.33:0/1590832991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568db1f65d0 0x5568db209c70 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f903908f740 1 -- 172.21.15.33:0/1590832991 >> 172.21.15.33:0/1590832991 conn(0x5568db1bf310 msgr2=0x5568db1f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:58.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f900ffff640 0 client.19125 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:36:58.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f903908f740 1 -- 172.21.15.33:0/1590832991 shutdown_connections 2024-09-16T00:36:58.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f903908f740 1 -- 172.21.15.33:0/1590832991 wait complete. 2024-09-16T00:36:58.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.095+0000 7f903908f740 20 client.19125 trim_cache size 0 max 16384 2024-09-16T00:36:58.117 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:36:58.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.096+0000 7f9036cb6640 1 -- 172.21.15.33:0/2199748120 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f90180a3240 msgr2=0x7f90180a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:58.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.096+0000 7f9036cb6640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f90180a3240 0x7f90180a5700 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f902804e000 tx=0x7f902804c000 comp rx=0 tx=0).stop 2024-09-16T00:36:58.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.096+0000 7f9036cb6640 1 -- 172.21.15.33:0/2199748120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9030102d40 msgr2=0x7f90301983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:58.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.096+0000 7f9036cb6640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9030102d40 0x7f90301983d0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f902000b4a0 tx=0x7f902000b960 comp rx=0 tx=0).stop 2024-09-16T00:36:58.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.096+0000 7f9036cb6640 1 -- 172.21.15.33:0/2199748120 shutdown_connections 2024-09-16T00:36:58.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.096+0000 7f9036cb6640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f90180a3240 0x7f90180a5700 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.096+0000 7f9036cb6640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f903010c170 0x7f903019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.096+0000 7f9036cb6640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9030103660 0x7f9030198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.096+0000 7f9036cb6640 1 --2- 172.21.15.33:0/2199748120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9030102d40 0x7f90301983d0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:58.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.096+0000 7f9036cb6640 1 -- 172.21.15.33:0/2199748120 >> 172.21.15.33:0/2199748120 conn(0x7f90300fc7a0 msgr2=0x7f90300fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:58.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.096+0000 7f9036cb6640 1 -- 172.21.15.33:0/2199748120 shutdown_connections 2024-09-16T00:36:58.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:58.096+0000 7f9036cb6640 1 -- 172.21.15.33:0/2199748120 wait complete. 2024-09-16T00:36:58.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:57 smithi083 ceph-mon[35242]: pgmap v944: 129 pgs: 129 active+clean; 619 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.2 MiB/s wr, 379 op/s 2024-09-16T00:36:58.782 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:36:59.038 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:36:59.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 19.46 MB/sec execute 245 sec latency 7400.949 ms 2024-09-16T00:36:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.584+0000 7f39bb02e640 1 -- 172.21.15.33:0/1719585803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39b41055b0 msgr2=0x7f39b410d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.584+0000 7f39bb02e640 1 --2- 172.21.15.33:0/1719585803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39b41055b0 0x7f39b410d130 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f39ac009ca0 tx=0x7f39ac02a8c0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.584+0000 7f39bb02e640 1 -- 172.21.15.33:0/1719585803 shutdown_connections 2024-09-16T00:36:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.584+0000 7f39bb02e640 1 --2- 172.21.15.33:0/1719585803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39b41055b0 0x7f39b410d130 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.584+0000 7f39bb02e640 1 --2- 172.21.15.33:0/1719585803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39b4100e30 0x7f39b4104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.584+0000 7f39bb02e640 1 --2- 172.21.15.33:0/1719585803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b4100480 0x7f39b4100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.585+0000 7f39bb02e640 1 -- 172.21.15.33:0/1719585803 >> 172.21.15.33:0/1719585803 conn(0x7f39b4077210 msgr2=0x7f39b40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.585+0000 7f39bb02e640 1 -- 172.21.15.33:0/1719585803 shutdown_connections 2024-09-16T00:36:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.585+0000 7f39bb02e640 1 -- 172.21.15.33:0/1719585803 wait complete. 2024-09-16T00:36:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.585+0000 7f39bb02e640 1 Processor -- start 2024-09-16T00:36:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.585+0000 7f39bb02e640 1 -- start start 2024-09-16T00:36:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.585+0000 7f39bb02e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b4100480 0x7f39b419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.585+0000 7f39bb02e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39b4100e30 0x7f39b419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.585+0000 7f39bb02e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39b41055b0 0x7f39b419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.585+0000 7f39bb02e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4111f90 con 0x7f39b4100e30 2024-09-16T00:36:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.585+0000 7f39bb02e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4111e10 con 0x7f39b4100480 2024-09-16T00:36:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.585+0000 7f39bb02e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4112110 con 0x7f39b41055b0 2024-09-16T00:36:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39b4100e30 0x7f39b419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39b4100e30 0x7f39b419aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47738/0 (socket says 172.21.15.33:47738) 2024-09-16T00:36:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b3fff640 1 -- 172.21.15.33:0/824075604 learned_addr learned my addr 172.21.15.33:0/824075604 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b95cb640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39b41055b0 0x7f39b419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b8dca640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b4100480 0x7f39b419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b3fff640 1 -- 172.21.15.33:0/824075604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39b41055b0 msgr2=0x7f39b419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b3fff640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39b41055b0 0x7f39b419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b3fff640 1 -- 172.21.15.33:0/824075604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b4100480 msgr2=0x7f39b419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b3fff640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b4100480 0x7f39b419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b3fff640 1 -- 172.21.15.33:0/824075604 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b419f610 con 0x7f39b4100e30 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b3fff640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39b4100e30 0x7f39b419aba0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f39a800e9f0 tx=0x7f39a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b1ffb640 1 -- 172.21.15.33:0/824075604 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a800cd80 con 0x7f39b4100e30 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39bb02e640 1 -- 172.21.15.33:0/824075604 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b419f900 con 0x7f39b4100e30 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b1ffb640 1 -- 172.21.15.33:0/824075604 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39a8005500 con 0x7f39b4100e30 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39b1ffb640 1 -- 172.21.15.33:0/824075604 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a8010690 con 0x7f39b4100e30 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.586+0000 7f39bb02e640 1 -- 172.21.15.33:0/824075604 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b41a71e0 con 0x7f39b4100e30 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.588+0000 7f39b1ffb640 1 -- 172.21.15.33:0/824075604 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f39a80046c0 con 0x7f39b4100e30 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.589+0000 7f39bd407740 1 Processor -- start 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.589+0000 7f39bd407740 1 -- start start 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.589+0000 7f39bd407740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 0x5610c624e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.589+0000 7f39bd407740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624e5d0 0x5610c624ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.589+0000 7f39bd407740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c6259370 0x5610c625b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.589+0000 7f39bd407740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5610c61b42f0 con 0x5610c624fb40 2024-09-16T00:36:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.589+0000 7f39bd407740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5610c615bcf0 con 0x5610c624e5d0 2024-09-16T00:36:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.589+0000 7f39bd407740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5610c61b02b0 con 0x5610c6259370 2024-09-16T00:36:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b95cb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 0x5610c624e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b95cb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 0x5610c624e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47746/0 (socket says 172.21.15.33:47746) 2024-09-16T00:36:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b95cb640 1 -- 172.21.15.33:0/1610250658 learned_addr learned my addr 172.21.15.33:0/1610250658 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b1ffb640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f399c0a3240 0x7f399c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b3fff640 1 --2- 172.21.15.33:0/1610250658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c6259370 0x5610c625b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b95cb640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f399c0a3240 0x7f399c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b8dca640 1 --2- 172.21.15.33:0/1610250658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624e5d0 0x5610c624ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b95cb640 1 -- 172.21.15.33:0/1610250658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c6259370 msgr2=0x5610c625b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b95cb640 1 --2- 172.21.15.33:0/1610250658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c6259370 0x5610c625b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b95cb640 1 -- 172.21.15.33:0/1610250658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624e5d0 msgr2=0x5610c624ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b95cb640 1 --2- 172.21.15.33:0/1610250658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624e5d0 0x5610c624ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b95cb640 1 -- 172.21.15.33:0/1610250658 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610c61403a0 con 0x5610c624fb40 2024-09-16T00:36:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b8dca640 1 --2- 172.21.15.33:0/1610250658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624e5d0 0x5610c624ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:36:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b1ffb640 1 -- 172.21.15.33:0/824075604 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f39a80d0190 con 0x7f39b4100e30 2024-09-16T00:36:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b95cb640 1 --2- 172.21.15.33:0/1610250658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 0x5610c624e090 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f39ac009430 tx=0x7f39ac031f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e07ae8f3c8d5b71 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f399affd640 1 -- 172.21.15.33:0/1610250658 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac003c50 con 0x5610c624fb40 2024-09-16T00:36:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f399affd640 1 -- 172.21.15.33:0/1610250658 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39ac04f070 con 0x5610c624fb40 2024-09-16T00:36:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f399affd640 1 -- 172.21.15.33:0/1610250658 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac02fae0 con 0x5610c624fb40 2024-09-16T00:36:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.590+0000 7f39b95cb640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f399c0a3240 0x7f399c0a5700 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f39ac04e000 tx=0x7f39ac04c000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.591+0000 7f39bd407740 1 -- 172.21.15.33:0/1610250658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 msgr2=0x5610c624e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.591+0000 7f39bd407740 1 --2- 172.21.15.33:0/1610250658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 0x5610c624e090 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f39ac009430 tx=0x7f39ac031f20 comp rx=0 tx=0).stop 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.591+0000 7f39bd407740 1 -- 172.21.15.33:0/1610250658 shutdown_connections 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.591+0000 7f39bd407740 1 --2- 172.21.15.33:0/1610250658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c6259370 0x5610c625b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.591+0000 7f39bd407740 1 --2- 172.21.15.33:0/1610250658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624e5d0 0x5610c624ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.591+0000 7f39bd407740 1 --2- 172.21.15.33:0/1610250658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 0x5610c624e090 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.591+0000 7f39bd407740 1 -- 172.21.15.33:0/1610250658 >> 172.21.15.33:0/1610250658 conn(0x5610c6217310 msgr2=0x5610c624ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.591+0000 7f39bd407740 1 -- 172.21.15.33:0/1610250658 shutdown_connections 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.591+0000 7f39bd407740 1 -- 172.21.15.33:0/1610250658 wait complete. 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39bd407740 1 -- start start 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39bd407740 1 Processor -- start 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39bd407740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c624e5d0 0x5610c6258d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39bd407740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 0x5610c6255ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39bd407740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c6259370 0x5610c62565e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39bd407740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5610c625bef0 con 0x5610c624fb40 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39bd407740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5610c616f060 con 0x5610c6259370 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39bd407740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5610c625c070 con 0x5610c624e5d0 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39bd407740 10 client.0 _ll_register_callbacks cb 0x5610c5ec28c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39b3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 0x5610c6255ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39b3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 0x5610c6255ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47750/0 (socket says 172.21.15.33:47750) 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39b3fff640 1 -- 172.21.15.33:0/1496368836 learned_addr learned my addr 172.21.15.33:0/1496368836 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39b8dca640 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c624e5d0 0x5610c6258d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.592+0000 7f39b95cb640 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c6259370 0x5610c62565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f39b3fff640 1 -- 172.21.15.33:0/1496368836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c624e5d0 msgr2=0x5610c6258d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f39b3fff640 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c624e5d0 0x5610c6258d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f39b3fff640 1 -- 172.21.15.33:0/1496368836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c6259370 msgr2=0x5610c62565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f39b3fff640 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c6259370 0x5610c62565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f39b3fff640 1 -- 172.21.15.33:0/1496368836 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610c5f253c0 con 0x5610c624fb40 2024-09-16T00:36:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f39b95cb640 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c6259370 0x5610c62565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:36:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f39b3fff640 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 0x5610c6255ff0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f39a809c260 tx=0x7f39a8021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a7ce3cc10e50b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f3993fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f3993fff640 1 -- 172.21.15.33:0/1496368836 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a80a4cb0 con 0x5610c624fb40 2024-09-16T00:36:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f39bd407740 1 -- 172.21.15.33:0/1496368836 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39ac04f070 con 0x5610c624fb40 2024-09-16T00:36:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f3993fff640 1 -- 172.21.15.33:0/1496368836 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f39a80bc0b0 con 0x5610c624fb40 2024-09-16T00:36:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f3993fff640 1 -- 172.21.15.33:0/1496368836 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f39a809c2d0 con 0x5610c624fb40 2024-09-16T00:36:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f39bd407740 10 client.19137 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f39bd407740 1 -- 172.21.15.33:0/1496368836 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5610c6267920 con 0x5610c624fb40 2024-09-16T00:36:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f3993fff640 1 -- 172.21.15.33:0/1496368836 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39ac04f070 con 0x5610c624fb40 2024-09-16T00:36:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f3993fff640 1 -- 172.21.15.33:0/1496368836 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f39a80d20f0 con 0x5610c624fb40 2024-09-16T00:36:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f39bd407740 10 client.19137 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.593+0000 7f39bd407740 10 client.19137 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:36:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.594+0000 7f39bd407740 20 client.19137 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.594+0000 7f39bd407740 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5610c6268d20 0x5610c626b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.594+0000 7f39bd407740 4 client.19137 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:36:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.594+0000 7f39bd407740 1 -- 172.21.15.33:0/1496368836 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610c61c1860 con 0x5610c6268d20 2024-09-16T00:36:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.594+0000 7f39b8dca640 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5610c6268d20 0x5610c626b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.594+0000 7f3993fff640 10 client.19137 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.594+0000 7f39b8dca640 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5610c6268d20 0x5610c626b130 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab3aa377009be23a server_cookie=e352a83601fbfcec in_seq=0 out_seq=0 2024-09-16T00:36:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f3993fff640 1 -- 172.21.15.33:0/1496368836 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5610c61c1860 con 0x5610c6268d20 2024-09-16T00:36:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f3993fff640 10 client.19137 handle_command_reply: tid=0 2024-09-16T00:36:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f39bd407740 1 client.19137 shutdown 2024-09-16T00:36:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f39bd407740 1 -- 172.21.15.33:0/1496368836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 msgr2=0x5610c6255ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:59.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f39bd407740 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 0x5610c6255ff0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f39a809c260 tx=0x7f39a8021000 comp rx=0 tx=0).stop 2024-09-16T00:36:59.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f39bd407740 1 -- 172.21.15.33:0/1496368836 shutdown_connections 2024-09-16T00:36:59.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f39bd407740 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5610c6268d20 0x5610c626b130 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f39bd407740 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c6259370 0x5610c62565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f39bd407740 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624fb40 0x5610c6255ff0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f3993fff640 0 client.19137 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:59.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f39bd407740 1 --2- 172.21.15.33:0/1496368836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c624e5d0 0x5610c6258d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f39bd407740 1 -- 172.21.15.33:0/1496368836 >> 172.21.15.33:0/1496368836 conn(0x5610c6217310 msgr2=0x5610c6250960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:59.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f39bd407740 1 -- 172.21.15.33:0/1496368836 shutdown_connections 2024-09-16T00:36:59.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f39bd407740 1 -- 172.21.15.33:0/1496368836 wait complete. 2024-09-16T00:36:59.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.595+0000 7f39bd407740 20 client.19137 trim_cache size 0 max 16384 2024-09-16T00:36:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.608+0000 7f39bd407740 1 Processor -- start 2024-09-16T00:36:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.608+0000 7f39bd407740 1 -- start start 2024-09-16T00:36:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.608+0000 7f39bd407740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624e5d0 0x5610c6265190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.608+0000 7f39bd407740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624fb40 0x5610c6251120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.608+0000 7f39bd407740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c6259370 0x5610c6251660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.608+0000 7f39bd407740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41b3a60 con 0x5610c624e5d0 2024-09-16T00:36:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.608+0000 7f39bd407740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41b1330 con 0x5610c624fb40 2024-09-16T00:36:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.608+0000 7f39bd407740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41af180 con 0x5610c6259370 2024-09-16T00:36:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b8dca640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624e5d0 0x5610c6265190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b8dca640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624e5d0 0x5610c6265190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47764/0 (socket says 172.21.15.33:47764) 2024-09-16T00:36:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b8dca640 1 -- 172.21.15.33:0/3587071269 learned_addr learned my addr 172.21.15.33:0/3587071269 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b3fff640 1 --2- 172.21.15.33:0/3587071269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624fb40 0x5610c6251120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b95cb640 1 --2- 172.21.15.33:0/3587071269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c6259370 0x5610c6251660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b8dca640 1 -- 172.21.15.33:0/3587071269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c6259370 msgr2=0x5610c6251660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b8dca640 1 --2- 172.21.15.33:0/3587071269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c6259370 0x5610c6251660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b8dca640 1 -- 172.21.15.33:0/3587071269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624fb40 msgr2=0x5610c6251120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b8dca640 1 --2- 172.21.15.33:0/3587071269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624fb40 0x5610c6251120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b8dca640 1 -- 172.21.15.33:0/3587071269 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610c624aed0 con 0x5610c624e5d0 2024-09-16T00:36:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b3fff640 1 --2- 172.21.15.33:0/3587071269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624fb40 0x5610c6251120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b95cb640 1 --2- 172.21.15.33:0/3587071269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c6259370 0x5610c6251660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f39b8dca640 1 --2- 172.21.15.33:0/3587071269 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624e5d0 0x5610c6265190 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f39a4000c00 tx=0x7f39a4012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e366753f29a0dc9a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f399a7fc640 1 -- 172.21.15.33:0/3587071269 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4023030 con 0x5610c624e5d0 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f399a7fc640 1 -- 172.21.15.33:0/3587071269 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39a4005500 con 0x5610c624e5d0 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.609+0000 7f399a7fc640 1 -- 172.21.15.33:0/3587071269 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a401e680 con 0x5610c624e5d0 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.610+0000 7f39bd407740 1 -- 172.21.15.33:0/3587071269 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624e5d0 msgr2=0x5610c6265190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.610+0000 7f39bd407740 1 --2- 172.21.15.33:0/3587071269 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624e5d0 0x5610c6265190 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f39a4000c00 tx=0x7f39a4012820 comp rx=0 tx=0).stop 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.610+0000 7f39bd407740 1 -- 172.21.15.33:0/3587071269 shutdown_connections 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.610+0000 7f39bd407740 1 --2- 172.21.15.33:0/3587071269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c6259370 0x5610c6251660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.610+0000 7f39bd407740 1 --2- 172.21.15.33:0/3587071269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624fb40 0x5610c6251120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.610+0000 7f39bd407740 1 --2- 172.21.15.33:0/3587071269 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c624e5d0 0x5610c6265190 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.610+0000 7f39bd407740 1 -- 172.21.15.33:0/3587071269 >> 172.21.15.33:0/3587071269 conn(0x5610c6217310 msgr2=0x5610c62550e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.610+0000 7f39bd407740 1 -- 172.21.15.33:0/3587071269 shutdown_connections 2024-09-16T00:36:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.610+0000 7f39bd407740 1 -- 172.21.15.33:0/3587071269 wait complete. 2024-09-16T00:36:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.610+0000 7f39bd407740 1 -- start start 2024-09-16T00:36:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.610+0000 7f39bd407740 1 Processor -- start 2024-09-16T00:36:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39bd407740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c624e5d0 0x5610c6261930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39bd407740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624fb40 0x5610c625cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39bd407740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c6259370 0x5610c625d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39bd407740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41af7f0 con 0x5610c6259370 2024-09-16T00:36:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39bd407740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41b0c60 con 0x5610c624fb40 2024-09-16T00:36:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39bd407740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41afec0 con 0x5610c624e5d0 2024-09-16T00:36:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39bd407740 10 client.0 _ll_register_callbacks cb 0x7f39a800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:36:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39b95cb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c6259370 0x5610c625d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39b95cb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c6259370 0x5610c625d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47772/0 (socket says 172.21.15.33:47772) 2024-09-16T00:36:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39b95cb640 1 -- 172.21.15.33:0/3613089147 learned_addr learned my addr 172.21.15.33:0/3613089147 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:36:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39b3fff640 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624fb40 0x5610c625cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39b8dca640 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c624e5d0 0x5610c6261930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39b95cb640 1 -- 172.21.15.33:0/3613089147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c624e5d0 msgr2=0x5610c6261930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39b95cb640 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c624e5d0 0x5610c6261930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39b95cb640 1 -- 172.21.15.33:0/3613089147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624fb40 msgr2=0x5610c625cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:36:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39b95cb640 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624fb40 0x5610c625cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39b95cb640 1 -- 172.21.15.33:0/3613089147 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610c625d890 con 0x5610c6259370 2024-09-16T00:36:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39b8dca640 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c624e5d0 0x5610c6261930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:36:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.611+0000 7f39b3fff640 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624fb40 0x5610c625cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:36:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f39b95cb640 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c6259370 0x5610c625d1a0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f39ac005e30 tx=0x7f39ac0039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da28db427805d8fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:36:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f3993fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f3993fff640 1 -- 172.21.15.33:0/3613089147 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac009800 con 0x5610c6259370 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f3993fff640 1 -- 172.21.15.33:0/3613089147 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f39ac004610 con 0x5610c6259370 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f39bd407740 1 -- 172.21.15.33:0/3613089147 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39a4005500 con 0x5610c6259370 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f3993fff640 1 -- 172.21.15.33:0/3613089147 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f39ac004810 con 0x5610c6259370 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f39bd407740 10 client.19143 fetch_fsmap learned FSMap version 33 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f39bd407740 1 -- 172.21.15.33:0/3613089147 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5610c62d3820 con 0x5610c6259370 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f3993fff640 1 -- 172.21.15.33:0/3613089147 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39a4005500 con 0x5610c6259370 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f3993fff640 1 -- 172.21.15.33:0/3613089147 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f39ac030af0 con 0x5610c6259370 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f39bd407740 10 client.19143 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f39bd407740 10 client.19143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f39bd407740 20 client.19143 populate_metadata read hostname 'smithi033' 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f39bd407740 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5610c6268d20 0x5610c62d5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f39bd407740 4 client.19143 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:36:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.612+0000 7f39bd407740 1 -- 172.21.15.33:0/3613089147 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5610c61bbd70 con 0x5610c6268d20 2024-09-16T00:36:59.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f39b8dca640 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5610c6268d20 0x5610c62d5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:36:59.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f3993fff640 10 client.19143 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:59.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f39b8dca640 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5610c6268d20 0x5610c62d5d00 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d51ec5cbda1ffc11 server_cookie=f86545ba9ff8a9fc in_seq=0 out_seq=0 2024-09-16T00:36:59.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f3993fff640 1 -- 172.21.15.33:0/3613089147 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5610c61bbd70 con 0x5610c6268d20 2024-09-16T00:36:59.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f3993fff640 10 client.19143 handle_command_reply: tid=0 2024-09-16T00:36:59.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f39bd407740 1 client.19143 shutdown 2024-09-16T00:36:59.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f39bd407740 1 -- 172.21.15.33:0/3613089147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c6259370 msgr2=0x5610c625d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:59.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f39bd407740 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c6259370 0x5610c625d1a0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f39ac005e30 tx=0x7f39ac0039f0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f39bd407740 1 -- 172.21.15.33:0/3613089147 shutdown_connections 2024-09-16T00:36:59.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f39bd407740 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5610c6268d20 0x5610c62d5d00 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f39bd407740 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5610c6259370 0x5610c625d1a0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f3993fff640 0 client.19143 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:36:59.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f39bd407740 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5610c624fb40 0x5610c625cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f39bd407740 1 --2- 172.21.15.33:0/3613089147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5610c624e5d0 0x5610c6261930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.613+0000 7f39bd407740 1 -- 172.21.15.33:0/3613089147 >> 172.21.15.33:0/3613089147 conn(0x5610c6217310 msgr2=0x5610c6257850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:59.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bd407740 1 -- 172.21.15.33:0/3613089147 shutdown_connections 2024-09-16T00:36:59.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bd407740 1 -- 172.21.15.33:0/3613089147 wait complete. 2024-09-16T00:36:59.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bd407740 20 client.19143 trim_cache size 0 max 16384 2024-09-16T00:36:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:36:59.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bb02e640 1 -- 172.21.15.33:0/824075604 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f399c0a3240 msgr2=0x7f399c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:59.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bb02e640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f399c0a3240 0x7f399c0a5700 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f39ac04e000 tx=0x7f39ac04c000 comp rx=0 tx=0).stop 2024-09-16T00:36:59.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bb02e640 1 -- 172.21.15.33:0/824075604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39b4100e30 msgr2=0x7f39b419aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:36:59.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bb02e640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39b4100e30 0x7f39b419aba0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f39a800e9f0 tx=0x7f39a800edf0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bb02e640 1 -- 172.21.15.33:0/824075604 shutdown_connections 2024-09-16T00:36:59.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bb02e640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f399c0a3240 0x7f399c0a5700 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bb02e640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39b41055b0 0x7f39b419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bb02e640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39b4100e30 0x7f39b419aba0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bb02e640 1 --2- 172.21.15.33:0/824075604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b4100480 0x7f39b419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:36:59.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bb02e640 1 -- 172.21.15.33:0/824075604 >> 172.21.15.33:0/824075604 conn(0x7f39b4077210 msgr2=0x7f39b4103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:36:59.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bb02e640 1 -- 172.21.15.33:0/824075604 shutdown_connections 2024-09-16T00:36:59.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:36:59.614+0000 7f39bb02e640 1 -- 172.21.15.33:0/824075604 wait complete. 2024-09-16T00:36:59.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:36:59 smithi033 ceph-mon[31243]: pgmap v945: 129 pgs: 129 active+clean; 619 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.3 MiB/s wr, 283 op/s 2024-09-16T00:36:59.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:36:59 smithi083 ceph-mon[35242]: pgmap v945: 129 pgs: 129 active+clean; 619 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.3 MiB/s wr, 283 op/s 2024-09-16T00:36:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:36:59 smithi060 ceph-mon[35185]: pgmap v945: 129 pgs: 129 active+clean; 619 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.3 MiB/s wr, 283 op/s 2024-09-16T00:37:00.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 19.38 MB/sec execute 246 sec latency 8401.044 ms 2024-09-16T00:37:01.087 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 19.30 MB/sec execute 247 sec latency 9401.146 ms 2024-09-16T00:37:01.343 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:37:01.343 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:37:01.614 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:01.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:01 smithi060 ceph-mon[35185]: pgmap v946: 129 pgs: 129 active+clean; 619 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.4 MiB/s wr, 292 op/s 2024-09-16T00:37:02.051 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:01 smithi033 ceph-mon[31243]: pgmap v946: 129 pgs: 129 active+clean; 619 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.4 MiB/s wr, 292 op/s 2024-09-16T00:37:02.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 19.22 MB/sec execute 248 sec latency 10401.222 ms 2024-09-16T00:37:02.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:01 smithi083 ceph-mon[35242]: pgmap v946: 129 pgs: 129 active+clean; 619 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.4 MiB/s wr, 292 op/s 2024-09-16T00:37:02.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.135+0000 7fc66ecea640 1 -- 172.21.15.33:0/3588805559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc668101420 msgr2=0x7fc66810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:02.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.135+0000 7fc66ecea640 1 --2- 172.21.15.33:0/3588805559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc668101420 0x7fc66810d950 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fc660009cd0 tx=0x7fc66002a8e0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.135+0000 7fc66ecea640 1 -- 172.21.15.33:0/3588805559 shutdown_connections 2024-09-16T00:37:02.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.135+0000 7fc66ecea640 1 --2- 172.21.15.33:0/3588805559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc668101420 0x7fc66810d950 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.135+0000 7fc66ecea640 1 --2- 172.21.15.33:0/3588805559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc668100a80 0x7fc668100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.135+0000 7fc66ecea640 1 --2- 172.21.15.33:0/3588805559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668106a80 0x7fc668106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.135+0000 7fc66ecea640 1 -- 172.21.15.33:0/3588805559 >> 172.21.15.33:0/3588805559 conn(0x7fc6680fc7a0 msgr2=0x7fc6680febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:02.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.135+0000 7fc66ecea640 1 -- 172.21.15.33:0/3588805559 shutdown_connections 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.135+0000 7fc66ecea640 1 -- 172.21.15.33:0/3588805559 wait complete. 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc66ecea640 1 Processor -- start 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc66ecea640 1 -- start start 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc66ecea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668100a80 0x7fc668198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc66ecea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc668101420 0x7fc6681988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc66ecea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc668106a80 0x7fc66819cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc66ecea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc668110e80 con 0x7fc668101420 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc66ecea640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc668110d00 con 0x7fc668106a80 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc66ecea640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc668111000 con 0x7fc668100a80 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc667fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc668101420 0x7fc6681988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc667fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc668101420 0x7fc6681988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47348/0 (socket says 172.21.15.33:47348) 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc667fff640 1 -- 172.21.15.33:0/3561985495 learned_addr learned my addr 172.21.15.33:0/3561985495 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:02.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc66ca86640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668100a80 0x7fc668198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.136+0000 7fc66d287640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc668106a80 0x7fc66819cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.137+0000 7fc667fff640 1 -- 172.21.15.33:0/3561985495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668100a80 msgr2=0x7fc668198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:02.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.137+0000 7fc667fff640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668100a80 0x7fc668198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.137+0000 7fc667fff640 1 -- 172.21.15.33:0/3561985495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc668106a80 msgr2=0x7fc66819cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:02.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.137+0000 7fc667fff640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc668106a80 0x7fc66819cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.137+0000 7fc667fff640 1 -- 172.21.15.33:0/3561985495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66819d340 con 0x7fc668101420 2024-09-16T00:37:02.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.137+0000 7fc667fff640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc668101420 0x7fc6681988d0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fc65400c920 tx=0x7fc65400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:02.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.137+0000 7fc665ffb640 1 -- 172.21.15.33:0/3561985495 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc654007c70 con 0x7fc668101420 2024-09-16T00:37:02.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.137+0000 7fc665ffb640 1 -- 172.21.15.33:0/3561985495 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc654007e10 con 0x7fc668101420 2024-09-16T00:37:02.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.137+0000 7fc66ecea640 1 -- 172.21.15.33:0/3561985495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc66819d630 con 0x7fc668101420 2024-09-16T00:37:02.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.137+0000 7fc665ffb640 1 -- 172.21.15.33:0/3561985495 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc654018890 con 0x7fc668101420 2024-09-16T00:37:02.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.137+0000 7fc66ecea640 1 -- 172.21.15.33:0/3561985495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6681a4f10 con 0x7fc668101420 2024-09-16T00:37:02.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.139+0000 7fc665ffb640 1 -- 172.21.15.33:0/3561985495 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc654018a70 con 0x7fc668101420 2024-09-16T00:37:02.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc6710c3740 1 Processor -- start 2024-09-16T00:37:02.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc6710c3740 1 -- start start 2024-09-16T00:37:02.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc6710c3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f17e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc6710c3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f26c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc6710c3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f29540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc665ffb640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6580a32c0 0x7fc6580a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc6710c3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fb96e7d280 con 0x55fb96f27150 2024-09-16T00:37:02.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc6710c3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55fb96e24b10 con 0x55fb96f17a60 2024-09-16T00:37:02.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc6710c3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fb96e79240 con 0x55fb96f18380 2024-09-16T00:37:02.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc667fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f29540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc66d287640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6580a32c0 0x7fc6580a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc667fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f29540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47360/0 (socket says 172.21.15.33:47360) 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc667fff640 1 -- 172.21.15.33:0/845870422 learned_addr learned my addr 172.21.15.33:0/845870422 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc66ca86640 1 --2- 172.21.15.33:0/845870422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f26c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc66d287640 1 --2- 172.21.15.33:0/845870422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f17e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.140+0000 7fc665ffb640 1 -- 172.21.15.33:0/3561985495 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc6540cfe30 con 0x7fc668101420 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.141+0000 7fc667fff640 1 -- 172.21.15.33:0/845870422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 msgr2=0x55fb96f26c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.141+0000 7fc667fff640 1 --2- 172.21.15.33:0/845870422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f26c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.141+0000 7fc667fff640 1 -- 172.21.15.33:0/845870422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 msgr2=0x55fb96f17e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.141+0000 7fc667fff640 1 --2- 172.21.15.33:0/845870422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f17e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.141+0000 7fc667fff640 1 -- 172.21.15.33:0/845870422 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb96e311f0 con 0x55fb96f27150 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.141+0000 7fc66d287640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6580a32c0 0x7fc6580a5780 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc660009430 tx=0x7fc660032fc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.141+0000 7fc667fff640 1 --2- 172.21.15.33:0/845870422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f29540 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fc6540983d0 tx=0x7fc65406bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641d92ad129c9bf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.141+0000 7fc652ffd640 1 -- 172.21.15.33:0/845870422 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65409c670 con 0x55fb96f27150 2024-09-16T00:37:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.141+0000 7fc652ffd640 1 -- 172.21.15.33:0/845870422 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc65409bf70 con 0x55fb96f27150 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.141+0000 7fc652ffd640 1 -- 172.21.15.33:0/845870422 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65400fb30 con 0x55fb96f27150 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.142+0000 7fc6710c3740 1 -- 172.21.15.33:0/845870422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 msgr2=0x55fb96f29540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.142+0000 7fc6710c3740 1 --2- 172.21.15.33:0/845870422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f29540 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fc6540983d0 tx=0x7fc65406bbe0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.142+0000 7fc6710c3740 1 -- 172.21.15.33:0/845870422 shutdown_connections 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.142+0000 7fc6710c3740 1 --2- 172.21.15.33:0/845870422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f29540 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.142+0000 7fc6710c3740 1 --2- 172.21.15.33:0/845870422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f26c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.142+0000 7fc6710c3740 1 --2- 172.21.15.33:0/845870422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f17e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.142+0000 7fc6710c3740 1 -- 172.21.15.33:0/845870422 >> 172.21.15.33:0/845870422 conn(0x55fb96f140f0 msgr2=0x55fb96f144d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.142+0000 7fc6710c3740 1 -- 172.21.15.33:0/845870422 shutdown_connections 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.142+0000 7fc6710c3740 1 -- 172.21.15.33:0/845870422 wait complete. 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.142+0000 7fc6710c3740 1 -- start start 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.142+0000 7fc6710c3740 1 Processor -- start 2024-09-16T00:37:02.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc6710c3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f24090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc6710c3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f245d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc6710c3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f24b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc6710c3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fb96f29d80 con 0x55fb96f27150 2024-09-16T00:37:02.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc6710c3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55fb96e37f10 con 0x55fb96f17a60 2024-09-16T00:37:02.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc6710c3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fb96f29f00 con 0x55fb96f18380 2024-09-16T00:37:02.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc6710c3740 10 client.0 _ll_register_callbacks cb 0x55fb96e00a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:02.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc66d287640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f24b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc66d287640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f24b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47366/0 (socket says 172.21.15.33:47366) 2024-09-16T00:37:02.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc66d287640 1 -- 172.21.15.33:0/369548032 learned_addr learned my addr 172.21.15.33:0/369548032 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:02.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc667fff640 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f245d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc66ca86640 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f24090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc66d287640 1 -- 172.21.15.33:0/369548032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 msgr2=0x55fb96f245d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc66d287640 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f245d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc66d287640 1 -- 172.21.15.33:0/369548032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 msgr2=0x55fb96f24090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc66d287640 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f24090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc66d287640 1 -- 172.21.15.33:0/369548032 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb96bee6a0 con 0x55fb96f27150 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc66d287640 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f24b10 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fc660009cd0 tx=0x7fc660004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7201254b8a6c2782 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.143+0000 7fc647fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc647fff640 1 -- 172.21.15.33:0/369548032 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66004f070 con 0x55fb96f27150 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc647fff640 1 -- 172.21.15.33:0/369548032 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc6600047a0 con 0x55fb96f27150 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc6710c3740 1 -- 172.21.15.33:0/369548032 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc65409bf70 con 0x55fb96f27150 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc647fff640 1 -- 172.21.15.33:0/369548032 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc660004b40 con 0x55fb96f27150 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc6710c3740 10 client.19155 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc6710c3740 1 -- 172.21.15.33:0/369548032 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fb96f35a20 con 0x55fb96f27150 2024-09-16T00:37:02.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc647fff640 1 -- 172.21.15.33:0/369548032 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc65409bf70 con 0x55fb96f27150 2024-09-16T00:37:02.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc647fff640 1 -- 172.21.15.33:0/369548032 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fc660048040 con 0x55fb96f27150 2024-09-16T00:37:02.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc6710c3740 10 client.19155 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:02.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc6710c3740 10 client.19155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:37:02.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc6710c3740 20 client.19155 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:02.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc6710c3740 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb96f36ee0 0x55fb96f392f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc6710c3740 4 client.19155 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:37:02.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.144+0000 7fc6710c3740 1 -- 172.21.15.33:0/369548032 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb96e8a7f0 con 0x55fb96f36ee0 2024-09-16T00:37:02.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.145+0000 7fc66ca86640 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb96f36ee0 0x55fb96f392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.145+0000 7fc66ca86640 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb96f36ee0 0x55fb96f392f0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c1377d673f4bc23 server_cookie=21015ab43bcef92f in_seq=0 out_seq=0 2024-09-16T00:37:02.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.145+0000 7fc647fff640 10 client.19155 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:02.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc647fff640 1 -- 172.21.15.33:0/369548032 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fb96e8a7f0 con 0x55fb96f36ee0 2024-09-16T00:37:02.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc647fff640 10 client.19155 handle_command_reply: tid=0 2024-09-16T00:37:02.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc6710c3740 1 client.19155 shutdown 2024-09-16T00:37:02.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc6710c3740 1 -- 172.21.15.33:0/369548032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 msgr2=0x55fb96f24b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:02.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc6710c3740 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f24b10 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fc660009cd0 tx=0x7fc660004a40 comp rx=0 tx=0).stop 2024-09-16T00:37:02.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc6710c3740 1 -- 172.21.15.33:0/369548032 shutdown_connections 2024-09-16T00:37:02.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc6710c3740 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb96f36ee0 0x55fb96f392f0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc6710c3740 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f24b10 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc6710c3740 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f245d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc6710c3740 1 --2- 172.21.15.33:0/369548032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f24090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc6710c3740 1 -- 172.21.15.33:0/369548032 >> 172.21.15.33:0/369548032 conn(0x55fb96f140f0 msgr2=0x55fb96f19e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:02.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc647fff640 0 client.19155 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:02.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc6710c3740 1 -- 172.21.15.33:0/369548032 shutdown_connections 2024-09-16T00:37:02.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc6710c3740 1 -- 172.21.15.33:0/369548032 wait complete. 2024-09-16T00:37:02.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.146+0000 7fc6710c3740 20 client.19155 trim_cache size 0 max 16384 2024-09-16T00:37:02.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.163+0000 7fc6710c3740 1 Processor -- start 2024-09-16T00:37:02.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.163+0000 7fc6710c3740 1 -- start start 2024-09-16T00:37:02.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc6710c3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f17a60 0x55fb96f19d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc6710c3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f18380 0x55fb96f3ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc6710c3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f27150 0x55fb96f3f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc6710c3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681af6d0 con 0x55fb96f18380 2024-09-16T00:37:02.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc6710c3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681acfa0 con 0x55fb96f27150 2024-09-16T00:37:02.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc6710c3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681aadf0 con 0x55fb96f17a60 2024-09-16T00:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc667fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f18380 0x55fb96f3ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc667fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f18380 0x55fb96f3ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47378/0 (socket says 172.21.15.33:47378) 2024-09-16T00:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc667fff640 1 -- 172.21.15.33:0/2737375519 learned_addr learned my addr 172.21.15.33:0/2737375519 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc66ca86640 1 --2- 172.21.15.33:0/2737375519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f17a60 0x55fb96f19d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc66d287640 1 --2- 172.21.15.33:0/2737375519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f27150 0x55fb96f3f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc667fff640 1 -- 172.21.15.33:0/2737375519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f17a60 msgr2=0x55fb96f19d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc667fff640 1 --2- 172.21.15.33:0/2737375519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f17a60 0x55fb96f19d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc667fff640 1 -- 172.21.15.33:0/2737375519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f27150 msgr2=0x55fb96f3f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc667fff640 1 --2- 172.21.15.33:0/2737375519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f27150 0x55fb96f3f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc667fff640 1 -- 172.21.15.33:0/2737375519 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb96f23040 con 0x55fb96f18380 2024-09-16T00:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc667fff640 1 --2- 172.21.15.33:0/2737375519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f18380 0x55fb96f3ebc0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fc65409bf00 tx=0x7fc654097c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574430161e528878 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc6527fc640 1 -- 172.21.15.33:0/2737375519 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6540a4be0 con 0x55fb96f18380 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc6527fc640 1 -- 172.21.15.33:0/2737375519 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6540a0320 con 0x55fb96f18380 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.164+0000 7fc6527fc640 1 -- 172.21.15.33:0/2737375519 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6540985e0 con 0x55fb96f18380 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.165+0000 7fc6710c3740 1 -- 172.21.15.33:0/2737375519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f18380 msgr2=0x55fb96f3ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.165+0000 7fc6710c3740 1 --2- 172.21.15.33:0/2737375519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f18380 0x55fb96f3ebc0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fc65409bf00 tx=0x7fc654097c10 comp rx=0 tx=0).stop 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.165+0000 7fc6710c3740 1 -- 172.21.15.33:0/2737375519 shutdown_connections 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.165+0000 7fc6710c3740 1 --2- 172.21.15.33:0/2737375519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f27150 0x55fb96f3f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.165+0000 7fc6710c3740 1 --2- 172.21.15.33:0/2737375519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f18380 0x55fb96f3ebc0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.165+0000 7fc6710c3740 1 --2- 172.21.15.33:0/2737375519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f17a60 0x55fb96f19d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.165+0000 7fc6710c3740 1 -- 172.21.15.33:0/2737375519 >> 172.21.15.33:0/2737375519 conn(0x55fb96f140f0 msgr2=0x55fb96f19630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.165+0000 7fc6710c3740 1 -- 172.21.15.33:0/2737375519 shutdown_connections 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.165+0000 7fc6710c3740 1 -- 172.21.15.33:0/2737375519 wait complete. 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.165+0000 7fc6710c3740 1 -- start start 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc6710c3740 1 Processor -- start 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc6710c3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f2aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc6710c3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f2b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc6710c3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f2b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc6710c3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681ab460 con 0x55fb96f27150 2024-09-16T00:37:02.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc6710c3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681ac8d0 con 0x55fb96f17a60 2024-09-16T00:37:02.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc6710c3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681abb30 con 0x55fb96f18380 2024-09-16T00:37:02.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc6710c3740 10 client.0 _ll_register_callbacks cb 0x7fc660005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:02.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc66d287640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f2b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc66d287640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f2b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47390/0 (socket says 172.21.15.33:47390) 2024-09-16T00:37:02.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc66d287640 1 -- 172.21.15.33:0/3769252647 learned_addr learned my addr 172.21.15.33:0/3769252647 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:02.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc667fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f2b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc66ca86640 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f2aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc66d287640 1 -- 172.21.15.33:0/3769252647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 msgr2=0x55fb96f2b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc66d287640 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f2b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc66d287640 1 -- 172.21.15.33:0/3769252647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 msgr2=0x55fb96f2aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc66d287640 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f2aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc66d287640 1 -- 172.21.15.33:0/3769252647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb96fa0b90 con 0x55fb96f27150 2024-09-16T00:37:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc667fff640 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f2b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.166+0000 7fc66ca86640 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f2aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:37:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc66d287640 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f2b930 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fc660004ab0 tx=0x7fc6600044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c79d8561cecccfed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc647fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc647fff640 1 -- 172.21.15.33:0/3769252647 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc660066060 con 0x55fb96f27150 2024-09-16T00:37:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc647fff640 1 -- 172.21.15.33:0/3769252647 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc66002e070 con 0x55fb96f27150 2024-09-16T00:37:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc6710c3740 1 -- 172.21.15.33:0/3769252647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6540a0320 con 0x55fb96f27150 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc647fff640 1 -- 172.21.15.33:0/3769252647 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc66006d050 con 0x55fb96f27150 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc6710c3740 10 client.19161 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc6710c3740 1 -- 172.21.15.33:0/3769252647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fb96fa1e80 con 0x55fb96f27150 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc647fff640 1 -- 172.21.15.33:0/3769252647 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6540a0320 con 0x55fb96f27150 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc647fff640 1 -- 172.21.15.33:0/3769252647 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fc660030070 con 0x55fb96f27150 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc6710c3740 10 client.19161 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc6710c3740 10 client.19161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.167+0000 7fc6710c3740 20 client.19161 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.168+0000 7fc6710c3740 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb96fa2540 0x55fb96fa4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.168+0000 7fc6710c3740 4 client.19161 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.168+0000 7fc6710c3740 1 -- 172.21.15.33:0/3769252647 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fb96e84d00 con 0x55fb96fa2540 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.168+0000 7fc66ca86640 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb96fa2540 0x55fb96fa4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:02.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.168+0000 7fc647fff640 10 client.19161 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.168+0000 7fc66ca86640 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb96fa2540 0x55fb96fa4960 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0ff5346916e8cbf server_cookie=336adff50d7fd3dc in_seq=0 out_seq=0 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.169+0000 7fc647fff640 1 -- 172.21.15.33:0/3769252647 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fb96e84d00 con 0x55fb96fa2540 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.169+0000 7fc647fff640 10 client.19161 handle_command_reply: tid=0 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.169+0000 7fc6710c3740 1 client.19161 shutdown 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.169+0000 7fc6710c3740 1 -- 172.21.15.33:0/3769252647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 msgr2=0x55fb96f2b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.169+0000 7fc6710c3740 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f2b930 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fc660004ab0 tx=0x7fc6600044b0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.169+0000 7fc6710c3740 1 -- 172.21.15.33:0/3769252647 shutdown_connections 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.169+0000 7fc6710c3740 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fb96fa2540 0x55fb96fa4960 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.169+0000 7fc6710c3740 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb96f27150 0x55fb96f2b930 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.169+0000 7fc6710c3740 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb96f18380 0x55fb96f2b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.169+0000 7fc647fff640 0 client.19161 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.169+0000 7fc6710c3740 1 --2- 172.21.15.33:0/3769252647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fb96f17a60 0x55fb96f2aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.169+0000 7fc6710c3740 1 -- 172.21.15.33:0/3769252647 >> 172.21.15.33:0/3769252647 conn(0x55fb96f140f0 msgr2=0x55fb96f19db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc6710c3740 1 -- 172.21.15.33:0/3769252647 shutdown_connections 2024-09-16T00:37:02.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc6710c3740 1 -- 172.21.15.33:0/3769252647 wait complete. 2024-09-16T00:37:02.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc6710c3740 20 client.19161 trim_cache size 0 max 16384 2024-09-16T00:37:02.188 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:37:02.188 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:37:02.188 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "1269db95-cf2b-434a-9bbe-bc0ccff8b574", 2024-09-16T00:37:02.188 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:37:02.188 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:37:02.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc66ecea640 1 -- 172.21.15.33:0/3561985495 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6580a32c0 msgr2=0x7fc6580a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:02.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc66ecea640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6580a32c0 0x7fc6580a5780 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc660009430 tx=0x7fc660032fc0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc66ecea640 1 -- 172.21.15.33:0/3561985495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc668101420 msgr2=0x7fc6681988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:02.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc66ecea640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc668101420 0x7fc6681988d0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fc65400c920 tx=0x7fc65400cde0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc66ecea640 1 -- 172.21.15.33:0/3561985495 shutdown_connections 2024-09-16T00:37:02.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc66ecea640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6580a32c0 0x7fc6580a5780 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc66ecea640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc668106a80 0x7fc66819cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc66ecea640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc668101420 0x7fc6681988d0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc66ecea640 1 --2- 172.21.15.33:0/3561985495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668100a80 0x7fc668198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:02.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.170+0000 7fc66ecea640 1 -- 172.21.15.33:0/3561985495 >> 172.21.15.33:0/3561985495 conn(0x7fc6680fc7a0 msgr2=0x7fc668104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:02.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.171+0000 7fc66ecea640 1 -- 172.21.15.33:0/3561985495 shutdown_connections 2024-09-16T00:37:02.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:02.171+0000 7fc66ecea640 1 -- 172.21.15.33:0/3561985495 wait complete. 2024-09-16T00:37:02.918 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:37:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:37:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:37:03.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:03.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:03.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:03.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:03.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:03.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:03.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:03.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:03.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 19.15 MB/sec execute 249 sec latency 11401.258 ms 2024-09-16T00:37:03.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:37:03.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:03.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:03.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:03.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:03.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:03.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:03.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:03.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:03.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:03.265 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:03.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.777+0000 7f1ddebcf640 1 -- 172.21.15.33:0/3724646052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dd8101de0 msgr2=0x7f1dd81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:03.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.777+0000 7f1ddebcf640 1 --2- 172.21.15.33:0/3724646052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dd8101de0 0x7f1dd81021c0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc009900 tx=0x7f1dcc02a410 comp rx=0 tx=0).stop 2024-09-16T00:37:03.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.777+0000 7f1ddebcf640 1 -- 172.21.15.33:0/3724646052 shutdown_connections 2024-09-16T00:37:03.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.777+0000 7f1ddebcf640 1 --2- 172.21.15.33:0/3724646052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dd8106f10 0x7f1dd810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.777+0000 7f1ddebcf640 1 --2- 172.21.15.33:0/3724646052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dd8102790 0x7f1dd81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.777+0000 7f1ddebcf640 1 --2- 172.21.15.33:0/3724646052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dd8101de0 0x7f1dd81021c0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.777+0000 7f1ddebcf640 1 -- 172.21.15.33:0/3724646052 >> 172.21.15.33:0/3724646052 conn(0x7f1dd80fd8c0 msgr2=0x7f1dd80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:03.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.777+0000 7f1ddebcf640 1 -- 172.21.15.33:0/3724646052 shutdown_connections 2024-09-16T00:37:03.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.777+0000 7f1ddebcf640 1 -- 172.21.15.33:0/3724646052 wait complete. 2024-09-16T00:37:03.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.778+0000 7f1ddebcf640 1 Processor -- start 2024-09-16T00:37:03.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.778+0000 7f1ddebcf640 1 -- start start 2024-09-16T00:37:03.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.778+0000 7f1ddebcf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dd8101de0 0x7f1dd819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.778+0000 7f1ddebcf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dd8102790 0x7f1dd819ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.778+0000 7f1ddebcf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dd8106f10 0x7f1dd819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.778+0000 7f1ddebcf640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd8111f70 con 0x7f1dd8102790 2024-09-16T00:37:03.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.778+0000 7f1ddebcf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd8111df0 con 0x7f1dd8101de0 2024-09-16T00:37:03.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.778+0000 7f1ddebcf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81120f0 con 0x7f1dd8106f10 2024-09-16T00:37:03.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.778+0000 7f1dd7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dd8102790 0x7f1dd819ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.778+0000 7f1dd7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dd8102790 0x7f1dd819ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47412/0 (socket says 172.21.15.33:47412) 2024-09-16T00:37:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.778+0000 7f1dd7fff640 1 -- 172.21.15.33:0/4020647606 learned_addr learned my addr 172.21.15.33:0/4020647606 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.778+0000 7f1ddc96b640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dd8101de0 0x7f1dd819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.779+0000 7f1ddd16c640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dd8106f10 0x7f1dd819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.779+0000 7f1dd7fff640 1 -- 172.21.15.33:0/4020647606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dd8106f10 msgr2=0x7f1dd819ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.779+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dd8106f10 0x7f1dd819ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.779+0000 7f1dd7fff640 1 -- 172.21.15.33:0/4020647606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dd8101de0 msgr2=0x7f1dd819a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.779+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dd8101de0 0x7f1dd819a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.779+0000 7f1dd7fff640 1 -- 172.21.15.33:0/4020647606 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd819f5e0 con 0x7f1dd8102790 2024-09-16T00:37:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.779+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dd8102790 0x7f1dd819ab70 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c970 tx=0x7f1dc400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.779+0000 7f1dd5ffb640 1 -- 172.21.15.33:0/4020647606 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4007bb0 con 0x7f1dd8102790 2024-09-16T00:37:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.779+0000 7f1dd5ffb640 1 -- 172.21.15.33:0/4020647606 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dc4007d50 con 0x7f1dd8102790 2024-09-16T00:37:03.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.779+0000 7f1dd5ffb640 1 -- 172.21.15.33:0/4020647606 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc400f450 con 0x7f1dd8102790 2024-09-16T00:37:03.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.779+0000 7f1ddebcf640 1 -- 172.21.15.33:0/4020647606 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd819f8d0 con 0x7f1dd8102790 2024-09-16T00:37:03.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.779+0000 7f1ddebcf640 1 -- 172.21.15.33:0/4020647606 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd81a71b0 con 0x7f1dd8102790 2024-09-16T00:37:03.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.781+0000 7f1dd5ffb640 1 -- 172.21.15.33:0/4020647606 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1dc4005020 con 0x7f1dd8102790 2024-09-16T00:37:03.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1de0fa8740 1 Processor -- start 2024-09-16T00:37:03.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1de0fa8740 1 -- start start 2024-09-16T00:37:03.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1de0fa8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74de2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1de0fa8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74ded140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1de0fa8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ded680 0x558a74defa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1de0fa8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558a74d482e0 con 0x558a74de24a0 2024-09-16T00:37:03.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1de0fa8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558a74cefce0 con 0x558a74de2dc0 2024-09-16T00:37:03.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1de0fa8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558a74d442a0 con 0x558a74ded680 2024-09-16T00:37:03.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1ddd16c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74de2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1ddd16c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74de2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47420/0 (socket says 172.21.15.33:47420) 2024-09-16T00:37:03.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1dd5ffb640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1dc80a3240 0x7f1dc80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1ddd16c640 1 -- 172.21.15.33:0/350480446 learned_addr learned my addr 172.21.15.33:0/350480446 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:03.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1ddc96b640 1 --2- 172.21.15.33:0/350480446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74ded140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.782+0000 7f1ddd16c640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1dc80a3240 0x7f1dc80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.783+0000 7f1dd5ffb640 1 -- 172.21.15.33:0/4020647606 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1dc40cfbd0 con 0x7f1dd8102790 2024-09-16T00:37:03.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.783+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/350480446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ded680 0x558a74defa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.783+0000 7f1ddd16c640 1 -- 172.21.15.33:0/350480446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ded680 msgr2=0x558a74defa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.783+0000 7f1ddd16c640 1 --2- 172.21.15.33:0/350480446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ded680 0x558a74defa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.783+0000 7f1ddd16c640 1 -- 172.21.15.33:0/350480446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 msgr2=0x558a74ded140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.783+0000 7f1ddd16c640 1 --2- 172.21.15.33:0/350480446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74ded140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.783+0000 7f1ddd16c640 1 -- 172.21.15.33:0/350480446 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a74cd4440 con 0x558a74de24a0 2024-09-16T00:37:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.783+0000 7f1ddd16c640 1 --2- 172.21.15.33:0/350480446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74de2880 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00129f0 tx=0x7f1dd0012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b556e37de2e4b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.783+0000 7f1dc2ffd640 1 -- 172.21.15.33:0/350480446 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0023070 con 0x558a74de24a0 2024-09-16T00:37:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.783+0000 7f1dc2ffd640 1 -- 172.21.15.33:0/350480446 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dd0004710 con 0x558a74de24a0 2024-09-16T00:37:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.783+0000 7f1dc2ffd640 1 -- 172.21.15.33:0/350480446 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd001fa90 con 0x558a74de24a0 2024-09-16T00:37:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.784+0000 7f1de0fa8740 1 -- 172.21.15.33:0/350480446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 msgr2=0x558a74de2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.784+0000 7f1ddd16c640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1dc80a3240 0x7f1dc80a5700 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0022000 tx=0x7f1dd0021000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.784+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/350480446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74de2880 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00129f0 tx=0x7f1dd0012eb0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.784+0000 7f1de0fa8740 1 -- 172.21.15.33:0/350480446 shutdown_connections 2024-09-16T00:37:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.784+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/350480446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ded680 0x558a74defa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.784+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/350480446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74ded140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.784+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/350480446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74de2880 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.784+0000 7f1de0fa8740 1 -- 172.21.15.33:0/350480446 >> 172.21.15.33:0/350480446 conn(0x558a74ddea30 msgr2=0x558a74ddee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.784+0000 7f1de0fa8740 1 -- 172.21.15.33:0/350480446 shutdown_connections 2024-09-16T00:37:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.784+0000 7f1de0fa8740 1 -- 172.21.15.33:0/350480446 wait complete. 2024-09-16T00:37:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.784+0000 7f1de0fa8740 1 -- start start 2024-09-16T00:37:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.784+0000 7f1de0fa8740 1 Processor -- start 2024-09-16T00:37:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1de0fa8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74decfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1de0fa8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74dea250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1de0fa8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ded680 0x558a74dea790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1de0fa8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558a74df0250 con 0x558a74de24a0 2024-09-16T00:37:03.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1de0fa8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558a74d03050 con 0x558a74de2dc0 2024-09-16T00:37:03.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1de0fa8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558a74df03d0 con 0x558a74ded680 2024-09-16T00:37:03.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1de0fa8740 10 client.0 _ll_register_callbacks cb 0x558a74d43160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:03.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1ddc96b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74decfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1ddc96b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74decfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47428/0 (socket says 172.21.15.33:47428) 2024-09-16T00:37:03.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1ddc96b640 1 -- 172.21.15.33:0/3221267898 learned_addr learned my addr 172.21.15.33:0/3221267898 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:03.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1ddd16c640 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ded680 0x558a74dea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74dea250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1ddc96b640 1 -- 172.21.15.33:0/3221267898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ded680 msgr2=0x558a74dea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:03.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1ddc96b640 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ded680 0x558a74dea790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1ddc96b640 1 -- 172.21.15.33:0/3221267898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 msgr2=0x558a74dea250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:03.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1ddc96b640 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74dea250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1ddc96b640 1 -- 172.21.15.33:0/3221267898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a74ab93c0 con 0x558a74de24a0 2024-09-16T00:37:03.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1ddc96b640 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74decfa0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc02a930 tx=0x7f1dcc031920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5472afc7191b01be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:03.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.785+0000 7f1db7fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:03.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1db7fff640 1 -- 172.21.15.33:0/3221267898 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc035040 con 0x558a74de24a0 2024-09-16T00:37:03.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1de0fa8740 1 -- 172.21.15.33:0/3221267898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dd0004710 con 0x558a74de24a0 2024-09-16T00:37:03.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1db7fff640 1 -- 172.21.15.33:0/3221267898 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1dcc004240 con 0x558a74de24a0 2024-09-16T00:37:03.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1db7fff640 1 -- 172.21.15.33:0/3221267898 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1dcc04e050 con 0x558a74de24a0 2024-09-16T00:37:03.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1de0fa8740 10 client.19173 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:03.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1de0fa8740 1 -- 172.21.15.33:0/3221267898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558a74dfbc20 con 0x558a74de24a0 2024-09-16T00:37:03.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1db7fff640 1 -- 172.21.15.33:0/3221267898 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dd0004710 con 0x558a74de24a0 2024-09-16T00:37:03.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1db7fff640 1 -- 172.21.15.33:0/3221267898 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f1dcc0455a0 con 0x558a74de24a0 2024-09-16T00:37:03.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1de0fa8740 10 client.19173 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:03.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1de0fa8740 10 client.19173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:37:03.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1de0fa8740 20 client.19173 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:03.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a74dfd030 0x558a74dff420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1de0fa8740 4 client.19173 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:37:03.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.786+0000 7f1de0fa8740 1 -- 172.21.15.33:0/3221267898 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a74d55850 con 0x558a74dfd030 2024-09-16T00:37:03.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.787+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a74dfd030 0x558a74dff420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.787+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a74dfd030 0x558a74dff420 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c695bda00766ffe server_cookie=93d95482f313da11 in_seq=0 out_seq=0 2024-09-16T00:37:03.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.787+0000 7f1db7fff640 10 client.19173 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:03.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1db7fff640 1 -- 172.21.15.33:0/3221267898 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558a74d55850 con 0x558a74dfd030 2024-09-16T00:37:03.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1db7fff640 10 client.19173 handle_command_reply: tid=0 2024-09-16T00:37:03.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1de0fa8740 1 client.19173 shutdown 2024-09-16T00:37:03.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1de0fa8740 1 -- 172.21.15.33:0/3221267898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 msgr2=0x558a74decfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:03.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74decfa0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc02a930 tx=0x7f1dcc031920 comp rx=0 tx=0).stop 2024-09-16T00:37:03.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1de0fa8740 1 -- 172.21.15.33:0/3221267898 shutdown_connections 2024-09-16T00:37:03.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a74dfd030 0x558a74dff420 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ded680 0x558a74dea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74dea250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1db7fff640 0 client.19173 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:03.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/3221267898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74decfa0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1de0fa8740 1 -- 172.21.15.33:0/3221267898 >> 172.21.15.33:0/3221267898 conn(0x558a74ddea30 msgr2=0x558a74de4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:03.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1de0fa8740 1 -- 172.21.15.33:0/3221267898 shutdown_connections 2024-09-16T00:37:03.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1de0fa8740 1 -- 172.21.15.33:0/3221267898 wait complete. 2024-09-16T00:37:03.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.788+0000 7f1de0fa8740 20 client.19173 trim_cache size 0 max 16384 2024-09-16T00:37:03.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.805+0000 7f1de0fa8740 1 Processor -- start 2024-09-16T00:37:03.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1de0fa8740 1 -- start start 2024-09-16T00:37:03.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1de0fa8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74ddea30 0x558a74ddda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1de0fa8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74e04cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1de0fa8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74de2dc0 0x558a74e05230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1de0fa8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81b3b60 con 0x558a74de24a0 2024-09-16T00:37:03.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1de0fa8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81b1430 con 0x558a74ddea30 2024-09-16T00:37:03.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1de0fa8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81af280 con 0x558a74de2dc0 2024-09-16T00:37:03.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1dd7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74e04cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1dd7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74e04cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47436/0 (socket says 172.21.15.33:47436) 2024-09-16T00:37:03.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1dd7fff640 1 -- 172.21.15.33:0/931142015 learned_addr learned my addr 172.21.15.33:0/931142015 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:03.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1ddc96b640 1 --2- 172.21.15.33:0/931142015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74ddea30 0x558a74ddda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1ddd16c640 1 --2- 172.21.15.33:0/931142015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74de2dc0 0x558a74e05230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1dd7fff640 1 -- 172.21.15.33:0/931142015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74de2dc0 msgr2=0x558a74e05230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/931142015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74de2dc0 0x558a74e05230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1dd7fff640 1 -- 172.21.15.33:0/931142015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74ddea30 msgr2=0x558a74ddda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/931142015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74ddea30 0x558a74ddda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1dd7fff640 1 -- 172.21.15.33:0/931142015 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a74de22d0 con 0x558a74de24a0 2024-09-16T00:37:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/931142015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74e04cf0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40cd1f0 tx=0x7f1dc4004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326c3f81f7ca9899 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.806+0000 7f1dc27fc640 1 -- 172.21.15.33:0/931142015 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc401d040 con 0x558a74de24a0 2024-09-16T00:37:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.807+0000 7f1dc27fc640 1 -- 172.21.15.33:0/931142015 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dc40a00c0 con 0x558a74de24a0 2024-09-16T00:37:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.807+0000 7f1dc27fc640 1 -- 172.21.15.33:0/931142015 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4005310 con 0x558a74de24a0 2024-09-16T00:37:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.807+0000 7f1de0fa8740 1 -- 172.21.15.33:0/931142015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 msgr2=0x558a74e04cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.807+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/931142015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74e04cf0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40cd1f0 tx=0x7f1dc4004a20 comp rx=0 tx=0).stop 2024-09-16T00:37:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.807+0000 7f1de0fa8740 1 -- 172.21.15.33:0/931142015 shutdown_connections 2024-09-16T00:37:03.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.807+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/931142015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74de2dc0 0x558a74e05230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.807+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/931142015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74e04cf0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.807+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/931142015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74ddea30 0x558a74ddda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.807+0000 7f1de0fa8740 1 -- 172.21.15.33:0/931142015 >> 172.21.15.33:0/931142015 conn(0x558a74dea8f0 msgr2=0x558a74ddd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.807+0000 7f1de0fa8740 1 -- 172.21.15.33:0/931142015 shutdown_connections 2024-09-16T00:37:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.807+0000 7f1de0fa8740 1 -- 172.21.15.33:0/931142015 wait complete. 2024-09-16T00:37:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.807+0000 7f1de0fa8740 1 -- start start 2024-09-16T00:37:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1de0fa8740 1 Processor -- start 2024-09-16T00:37:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1de0fa8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ddea30 0x558a74df5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1de0fa8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74df3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1de0fa8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74df35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1de0fa8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81af8f0 con 0x558a74de24a0 2024-09-16T00:37:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1de0fa8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81b0d60 con 0x558a74de2dc0 2024-09-16T00:37:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1de0fa8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81affc0 con 0x558a74ddea30 2024-09-16T00:37:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1de0fa8740 10 client.0 _ll_register_callbacks cb 0x7f1dcc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1dd7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74df3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1dd7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74df3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47442/0 (socket says 172.21.15.33:47442) 2024-09-16T00:37:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1dd7fff640 1 -- 172.21.15.33:0/673143231 learned_addr learned my addr 172.21.15.33:0/673143231 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1ddc96b640 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ddea30 0x558a74df5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1ddd16c640 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74df35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1dd7fff640 1 -- 172.21.15.33:0/673143231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ddea30 msgr2=0x558a74df5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ddea30 0x558a74df5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1dd7fff640 1 -- 172.21.15.33:0/673143231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 msgr2=0x558a74df35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.808+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74df35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1dd7fff640 1 -- 172.21.15.33:0/673143231 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a74df3dd0 con 0x558a74de24a0 2024-09-16T00:37:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1ddd16c640 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74df35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:37:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1dd7fff640 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74df3090 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4007590 tx=0x7f1dc4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec38d7d95eed20b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1db7fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1db7fff640 1 -- 172.21.15.33:0/673143231 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc401d070 con 0x558a74de24a0 2024-09-16T00:37:03.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1db7fff640 1 -- 172.21.15.33:0/673143231 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1dc4098380 con 0x558a74de24a0 2024-09-16T00:37:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1de0fa8740 1 -- 172.21.15.33:0/673143231 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dc40a00c0 con 0x558a74de24a0 2024-09-16T00:37:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1db7fff640 1 -- 172.21.15.33:0/673143231 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1dc4099db0 con 0x558a74de24a0 2024-09-16T00:37:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1de0fa8740 10 client.19179 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1de0fa8740 1 -- 172.21.15.33:0/673143231 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558a74e67cb0 con 0x558a74de24a0 2024-09-16T00:37:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1db7fff640 1 -- 172.21.15.33:0/673143231 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dc40a00c0 con 0x558a74de24a0 2024-09-16T00:37:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1db7fff640 1 -- 172.21.15.33:0/673143231 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f1dc4082030 con 0x558a74de24a0 2024-09-16T00:37:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1de0fa8740 10 client.19179 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1de0fa8740 10 client.19179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:37:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1de0fa8740 20 client.19179 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a74ded680 0x558a74e6a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:03.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1de0fa8740 4 client.19179 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:37:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.809+0000 7f1de0fa8740 1 -- 172.21.15.33:0/673143231 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558a74d4fd60 con 0x558a74ded680 2024-09-16T00:37:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1ddc96b640 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a74ded680 0x558a74e6a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1ddc96b640 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a74ded680 0x558a74e6a4f0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4372679c0a1812e0 server_cookie=63dfabc2d38ddc69 in_seq=0 out_seq=0 2024-09-16T00:37:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1db7fff640 10 client.19179 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1db7fff640 1 -- 172.21.15.33:0/673143231 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558a74d4fd60 con 0x558a74ded680 2024-09-16T00:37:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1db7fff640 10 client.19179 handle_command_reply: tid=0 2024-09-16T00:37:03.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1de0fa8740 1 client.19179 shutdown 2024-09-16T00:37:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1de0fa8740 1 -- 172.21.15.33:0/673143231 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 msgr2=0x558a74df3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:03.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74df3090 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4007590 tx=0x7f1dc4007f90 comp rx=0 tx=0).stop 2024-09-16T00:37:03.840 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:37:03.840 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:37:03.840 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:37:03.840 INFO:teuthology.orchestra.run.smithi033.stdout: "1269db95-cf2b-434a-9bbe-bc0ccff8b574": { 2024-09-16T00:37:03.841 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:37:03.841 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "1269db95-cf2b-434a-9bbe-bc0ccff8b574", 2024-09-16T00:37:03.841 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:37:03.841 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:03.841 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:03.841 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:37:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1de0fa8740 1 -- 172.21.15.33:0/673143231 shutdown_connections 2024-09-16T00:37:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a74ded680 0x558a74e6a4f0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a74de2dc0 0x558a74df35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a74de24a0 0x558a74df3090 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1de0fa8740 1 --2- 172.21.15.33:0/673143231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a74ddea30 0x558a74df5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1de0fa8740 1 -- 172.21.15.33:0/673143231 >> 172.21.15.33:0/673143231 conn(0x558a74dea8f0 msgr2=0x558a74ddd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.810+0000 7f1db7fff640 0 client.19179 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:03.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1de0fa8740 1 -- 172.21.15.33:0/673143231 shutdown_connections 2024-09-16T00:37:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1de0fa8740 1 -- 172.21.15.33:0/673143231 wait complete. 2024-09-16T00:37:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1de0fa8740 20 client.19179 trim_cache size 0 max 16384 2024-09-16T00:37:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1ddebcf640 1 -- 172.21.15.33:0/4020647606 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1dc80a3240 msgr2=0x7f1dc80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1ddebcf640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1dc80a3240 0x7f1dc80a5700 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0022000 tx=0x7f1dd0021000 comp rx=0 tx=0).stop 2024-09-16T00:37:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1ddebcf640 1 -- 172.21.15.33:0/4020647606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dd8102790 msgr2=0x7f1dd819ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1ddebcf640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dd8102790 0x7f1dd819ab70 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c970 tx=0x7f1dc400ce30 comp rx=0 tx=0).stop 2024-09-16T00:37:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1ddebcf640 1 -- 172.21.15.33:0/4020647606 shutdown_connections 2024-09-16T00:37:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1ddebcf640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1dc80a3240 0x7f1dc80a5700 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1ddebcf640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dd8106f10 0x7f1dd819ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1ddebcf640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dd8102790 0x7f1dd819ab70 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1ddebcf640 1 --2- 172.21.15.33:0/4020647606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dd8101de0 0x7f1dd819a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1ddebcf640 1 -- 172.21.15.33:0/4020647606 >> 172.21.15.33:0/4020647606 conn(0x7f1dd80fd8c0 msgr2=0x7f1dd80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1ddebcf640 1 -- 172.21.15.33:0/4020647606 shutdown_connections 2024-09-16T00:37:03.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:03.811+0000 7f1ddebcf640 1 -- 172.21.15.33:0/4020647606 wait complete. 2024-09-16T00:37:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:03 smithi060 ceph-mon[35185]: pgmap v947: 129 pgs: 129 active+clean; 627 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 179 op/s 2024-09-16T00:37:04.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:03 smithi033 ceph-mon[31243]: pgmap v947: 129 pgs: 129 active+clean; 627 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 179 op/s 2024-09-16T00:37:04.086 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 19.07 MB/sec execute 250 sec latency 12401.295 ms 2024-09-16T00:37:04.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:03 smithi083 ceph-mon[35242]: pgmap v947: 129 pgs: 129 active+clean; 627 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 179 op/s 2024-09-16T00:37:04.644 INFO:tasks.cephfs.filesystem:scrub status for tag:1269db95-cf2b-434a-9bbe-bc0ccff8b574 - {'path': '/', 'tag': '1269db95-cf2b-434a-9bbe-bc0ccff8b574', 'options': 'recursive,force'} 2024-09-16T00:37:05.087 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.99 MB/sec execute 251 sec latency 13401.325 ms 2024-09-16T00:37:05.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:05 smithi060 ceph-mon[35185]: pgmap v948: 129 pgs: 129 active+clean; 627 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 193 op/s 2024-09-16T00:37:06.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:05 smithi033 ceph-mon[31243]: pgmap v948: 129 pgs: 129 active+clean; 627 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 193 op/s 2024-09-16T00:37:06.087 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.92 MB/sec execute 252 sec latency 14401.672 ms 2024-09-16T00:37:06.173 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:05 smithi083 ceph-mon[35242]: pgmap v948: 129 pgs: 129 active+clean; 627 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 193 op/s 2024-09-16T00:37:07.087 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.84 MB/sec execute 253 sec latency 15401.781 ms 2024-09-16T00:37:07.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:07 smithi060 ceph-mon[35185]: pgmap v949: 129 pgs: 129 active+clean; 628 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 3.1 MiB/s wr, 141 op/s 2024-09-16T00:37:08.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:07 smithi033 ceph-mon[31243]: pgmap v949: 129 pgs: 129 active+clean; 628 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 3.1 MiB/s wr, 141 op/s 2024-09-16T00:37:08.087 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.77 MB/sec execute 254 sec latency 16401.835 ms 2024-09-16T00:37:08.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:07 smithi083 ceph-mon[35242]: pgmap v949: 129 pgs: 129 active+clean; 628 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 3.1 MiB/s wr, 141 op/s 2024-09-16T00:37:09.087 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.70 MB/sec execute 255 sec latency 17401.939 ms 2024-09-16T00:37:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:09 smithi060 ceph-mon[35185]: pgmap v950: 129 pgs: 129 active+clean; 628 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 2.8 MiB/s wr, 123 op/s 2024-09-16T00:37:10.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:09 smithi033 ceph-mon[31243]: pgmap v950: 129 pgs: 129 active+clean; 628 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 2.8 MiB/s wr, 123 op/s 2024-09-16T00:37:10.087 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.62 MB/sec execute 256 sec latency 18402.046 ms 2024-09-16T00:37:10.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:09 smithi083 ceph-mon[35242]: pgmap v950: 129 pgs: 129 active+clean; 628 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 2.8 MiB/s wr, 123 op/s 2024-09-16T00:37:11.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.55 MB/sec execute 257 sec latency 19402.169 ms 2024-09-16T00:37:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:11 smithi060 ceph-mon[35185]: pgmap v951: 129 pgs: 129 active+clean; 627 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 3.7 MiB/s wr, 162 op/s 2024-09-16T00:37:12.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:11 smithi033 ceph-mon[31243]: pgmap v951: 129 pgs: 129 active+clean; 627 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 3.7 MiB/s wr, 162 op/s 2024-09-16T00:37:12.087 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.48 MB/sec execute 258 sec latency 20402.229 ms 2024-09-16T00:37:12.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:11 smithi083 ceph-mon[35242]: pgmap v951: 129 pgs: 129 active+clean; 627 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 3.7 MiB/s wr, 162 op/s 2024-09-16T00:37:13.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.41 MB/sec execute 259 sec latency 21402.278 ms 2024-09-16T00:37:14.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:13 smithi033 ceph-mon[31243]: pgmap v952: 129 pgs: 129 active+clean; 620 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-09-16T00:37:14.087 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.34 MB/sec execute 260 sec latency 22402.338 ms 2024-09-16T00:37:14.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:13 smithi083 ceph-mon[35242]: pgmap v952: 129 pgs: 129 active+clean; 620 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-09-16T00:37:14.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:13 smithi060 ceph-mon[35185]: pgmap v952: 129 pgs: 129 active+clean; 620 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-09-16T00:37:15.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.27 MB/sec execute 261 sec latency 23402.459 ms 2024-09-16T00:37:16.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:15 smithi033 ceph-mon[31243]: pgmap v953: 129 pgs: 129 active+clean; 620 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-09-16T00:37:16.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.20 MB/sec execute 262 sec latency 24402.507 ms 2024-09-16T00:37:16.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:15 smithi083 ceph-mon[35242]: pgmap v953: 129 pgs: 129 active+clean; 620 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-09-16T00:37:16.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:15 smithi060 ceph-mon[35185]: pgmap v953: 129 pgs: 129 active+clean; 620 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-09-16T00:37:16.823 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:37:17.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:16 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:37:17.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:16 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:37:17.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:16 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:37:17.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.13 MB/sec execute 263 sec latency 25402.541 ms 2024-09-16T00:37:17.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:16 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:37:17.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:16 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:37:17.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:16 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:37:17.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:16 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:37:17.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:16 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:37:17.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:16 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:37:17.259 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:17.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.804+0000 7f8212924640 1 -- 172.21.15.33:0/2865566619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f820c102790 msgr2=0x7f820c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:17.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.804+0000 7f8212924640 1 --2- 172.21.15.33:0/2865566619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f820c102790 0x7f820c1067e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f8200009900 tx=0x7f820002a410 comp rx=0 tx=0).stop 2024-09-16T00:37:17.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.804+0000 7f8212924640 1 -- 172.21.15.33:0/2865566619 shutdown_connections 2024-09-16T00:37:17.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.804+0000 7f8212924640 1 --2- 172.21.15.33:0/2865566619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f820c106f10 0x7f820c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:17.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.804+0000 7f8212924640 1 --2- 172.21.15.33:0/2865566619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f820c102790 0x7f820c1067e0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:17.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.804+0000 7f8212924640 1 --2- 172.21.15.33:0/2865566619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f820c101de0 0x7f820c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:17.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.804+0000 7f8212924640 1 -- 172.21.15.33:0/2865566619 >> 172.21.15.33:0/2865566619 conn(0x7f820c0fd8c0 msgr2=0x7f820c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.804+0000 7f8212924640 1 -- 172.21.15.33:0/2865566619 shutdown_connections 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.804+0000 7f8212924640 1 -- 172.21.15.33:0/2865566619 wait complete. 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.805+0000 7f8212924640 1 Processor -- start 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.805+0000 7f8212924640 1 -- start start 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.805+0000 7f8212924640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f820c101de0 0x7f820c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.805+0000 7f8212924640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f820c102790 0x7f820c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.805+0000 7f8212924640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f820c106f10 0x7f820c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.805+0000 7f8212924640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f820c111f70 con 0x7f820c102790 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.805+0000 7f8212924640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f820c111df0 con 0x7f820c106f10 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.805+0000 7f8212924640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f820c1120f0 con 0x7f820c101de0 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f820b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f820c102790 0x7f820c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f820b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f820c102790 0x7f820c1989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42736/0 (socket says 172.21.15.33:42736) 2024-09-16T00:37:17.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f820b7fe640 1 -- 172.21.15.33:0/282586888 learned_addr learned my addr 172.21.15.33:0/282586888 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f820bfff640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f820c101de0 0x7f820c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f8210ec1640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f820c106f10 0x7f820c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f820b7fe640 1 -- 172.21.15.33:0/282586888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f820c101de0 msgr2=0x7f820c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f820b7fe640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f820c101de0 0x7f820c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f820b7fe640 1 -- 172.21.15.33:0/282586888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f820c106f10 msgr2=0x7f820c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f820b7fe640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f820c106f10 0x7f820c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f820b7fe640 1 -- 172.21.15.33:0/282586888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c19d440 con 0x7f820c102790 2024-09-16T00:37:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f820b7fe640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f820c102790 0x7f820c1989d0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f8200006010 tx=0x7f8200004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f82097fa640 1 -- 172.21.15.33:0/282586888 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820003f030 con 0x7f820c102790 2024-09-16T00:37:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f82097fa640 1 -- 172.21.15.33:0/282586888 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8200004440 con 0x7f820c102790 2024-09-16T00:37:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f8212924640 1 -- 172.21.15.33:0/282586888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820c19d6d0 con 0x7f820c102790 2024-09-16T00:37:17.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f82097fa640 1 -- 172.21.15.33:0/282586888 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200035070 con 0x7f820c102790 2024-09-16T00:37:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.806+0000 7f8212924640 1 -- 172.21.15.33:0/282586888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f820c1a5010 con 0x7f820c102790 2024-09-16T00:37:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.807+0000 7f8212924640 1 -- 172.21.15.33:0/282586888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f820c1032c0 con 0x7f820c102790 2024-09-16T00:37:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.808+0000 7f82097fa640 1 -- 172.21.15.33:0/282586888 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8200042030 con 0x7f820c102790 2024-09-16T00:37:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.809+0000 7f82097fa640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81f40a3240 0x7f81f40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.809+0000 7f820bfff640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81f40a3240 0x7f81f40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.809+0000 7f82097fa640 1 -- 172.21.15.33:0/282586888 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f82000f0140 con 0x7f820c102790 2024-09-16T00:37:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.810+0000 7f820bfff640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81f40a3240 0x7f81f40a5700 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f81fc009bd0 tx=0x7f81fc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:17.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:17.813+0000 7f82097fa640 1 -- 172.21.15.33:0/282586888 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f82000bc330 con 0x7f820c102790 2024-09-16T00:37:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.023+0000 7f8212924640 1 -- 172.21.15.33:0/282586888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f820c10a0f0 con 0x7f820c102790 2024-09-16T00:37:18.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:17 smithi033 ceph-mon[31243]: pgmap v954: 129 pgs: 129 active+clean; 619 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 4.9 MiB/s wr, 214 op/s 2024-09-16T00:37:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.024+0000 7f82097fa640 1 -- 172.21.15.33:0/282586888 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f82000c0630 con 0x7f820c102790 2024-09-16T00:37:18.026 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:37:18.026 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:37:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:37:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.027+0000 7f8212924640 1 -- 172.21.15.33:0/282586888 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81f40a3240 msgr2=0x7f81f40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.027+0000 7f8212924640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81f40a3240 0x7f81f40a5700 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f81fc009bd0 tx=0x7f81fc005f90 comp rx=0 tx=0).stop 2024-09-16T00:37:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.027+0000 7f8212924640 1 -- 172.21.15.33:0/282586888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f820c102790 msgr2=0x7f820c1989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.027+0000 7f8212924640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f820c102790 0x7f820c1989d0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f8200006010 tx=0x7f8200004040 comp rx=0 tx=0).stop 2024-09-16T00:37:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.028+0000 7f8212924640 1 -- 172.21.15.33:0/282586888 shutdown_connections 2024-09-16T00:37:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.028+0000 7f8212924640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81f40a3240 0x7f81f40a5700 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.028+0000 7f8212924640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f820c106f10 0x7f820c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.028+0000 7f8212924640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f820c102790 0x7f820c1989d0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.028+0000 7f8212924640 1 --2- 172.21.15.33:0/282586888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f820c101de0 0x7f820c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.028+0000 7f8212924640 1 -- 172.21.15.33:0/282586888 >> 172.21.15.33:0/282586888 conn(0x7f820c0fd8c0 msgr2=0x7f820c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.028+0000 7f8212924640 1 -- 172.21.15.33:0/282586888 shutdown_connections 2024-09-16T00:37:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:18.028+0000 7f8212924640 1 -- 172.21.15.33:0/282586888 wait complete. 2024-09-16T00:37:18.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 18.06 MB/sec execute 264 sec latency 26402.579 ms 2024-09-16T00:37:18.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:17 smithi083 ceph-mon[35242]: pgmap v954: 129 pgs: 129 active+clean; 619 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 4.9 MiB/s wr, 214 op/s 2024-09-16T00:37:18.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:17 smithi060 ceph-mon[35185]: pgmap v954: 129 pgs: 129 active+clean; 619 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 4.9 MiB/s wr, 214 op/s 2024-09-16T00:37:18.768 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id e0f4c493 --release --await' 2024-09-16T00:37:18.768 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id e0f4c493 --release --await 2024-09-16T00:37:19.025 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:19.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:18 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/282586888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:37:19.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132376 17.99 MB/sec execute 265 sec latency 27402.657 ms 2024-09-16T00:37:19.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:18 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/282586888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:37:19.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:18 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/282586888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:37:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.511+0000 7ff1c76bd640 1 -- 172.21.15.33:0/1890453987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1c0106f10 msgr2=0x7ff1c010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.511+0000 7ff1c76bd640 1 --2- 172.21.15.33:0/1890453987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1c0106f10 0x7ff1c010ea90 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8009ca0 tx=0x7ff1b802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.511+0000 7ff1c76bd640 1 -- 172.21.15.33:0/1890453987 shutdown_connections 2024-09-16T00:37:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.511+0000 7ff1c76bd640 1 --2- 172.21.15.33:0/1890453987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1c0106f10 0x7ff1c010ea90 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.511+0000 7ff1c76bd640 1 --2- 172.21.15.33:0/1890453987 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1c0102790 0x7ff1c01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.511+0000 7ff1c76bd640 1 --2- 172.21.15.33:0/1890453987 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c0101de0 0x7ff1c01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.511+0000 7ff1c76bd640 1 -- 172.21.15.33:0/1890453987 >> 172.21.15.33:0/1890453987 conn(0x7ff1c00fd8c0 msgr2=0x7ff1c00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.511+0000 7ff1c76bd640 1 -- 172.21.15.33:0/1890453987 shutdown_connections 2024-09-16T00:37:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.511+0000 7ff1c76bd640 1 -- 172.21.15.33:0/1890453987 wait complete. 2024-09-16T00:37:19.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c76bd640 1 Processor -- start 2024-09-16T00:37:19.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c76bd640 1 -- start start 2024-09-16T00:37:19.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c76bd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1c0101de0 0x7ff1c01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c76bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c0102790 0x7ff1c01989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c76bd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1c0106f10 0x7ff1c019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c76bd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c0111f90 con 0x7ff1c0106f10 2024-09-16T00:37:19.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c76bd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c0111e10 con 0x7ff1c0101de0 2024-09-16T00:37:19.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c76bd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c0112110 con 0x7ff1c0102790 2024-09-16T00:37:19.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c5c5a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1c0106f10 0x7ff1c019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c5c5a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1c0106f10 0x7ff1c019cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42758/0 (socket says 172.21.15.33:42758) 2024-09-16T00:37:19.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c5c5a640 1 -- 172.21.15.33:0/533373216 learned_addr learned my addr 172.21.15.33:0/533373216 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:19.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c0102790 0x7ff1c01989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.512+0000 7ff1c5459640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1c0101de0 0x7ff1c01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.513+0000 7ff1c5c5a640 1 -- 172.21.15.33:0/533373216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c0102790 msgr2=0x7ff1c01989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:19.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.513+0000 7ff1c5c5a640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c0102790 0x7ff1c01989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.513+0000 7ff1c5c5a640 1 -- 172.21.15.33:0/533373216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1c0101de0 msgr2=0x7ff1c01984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:19.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.513+0000 7ff1c5c5a640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1c0101de0 0x7ff1c01984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.513+0000 7ff1c5c5a640 1 -- 172.21.15.33:0/533373216 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c019d460 con 0x7ff1c0106f10 2024-09-16T00:37:19.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.513+0000 7ff1c5459640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1c0101de0 0x7ff1c01984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:37:19.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.513+0000 7ff1c5c5a640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1c0106f10 0x7ff1c019cd80 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8031a20 tx=0x7ff1b8031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:19.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.513+0000 7ff1b67fc640 1 -- 172.21.15.33:0/533373216 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b8003c30 con 0x7ff1c0106f10 2024-09-16T00:37:19.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.513+0000 7ff1c76bd640 1 -- 172.21.15.33:0/533373216 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1c019d6f0 con 0x7ff1c0106f10 2024-09-16T00:37:19.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.513+0000 7ff1b67fc640 1 -- 172.21.15.33:0/533373216 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1b8003dd0 con 0x7ff1c0106f10 2024-09-16T00:37:19.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.513+0000 7ff1b67fc640 1 -- 172.21.15.33:0/533373216 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b802fcf0 con 0x7ff1c0106f10 2024-09-16T00:37:19.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.513+0000 7ff1c76bd640 1 -- 172.21.15.33:0/533373216 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1c01a5030 con 0x7ff1c0106f10 2024-09-16T00:37:19.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.515+0000 7ff1b67fc640 1 -- 172.21.15.33:0/533373216 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff1b8004050 con 0x7ff1c0106f10 2024-09-16T00:37:19.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.516+0000 7ff1c9a96740 1 Processor -- start 2024-09-16T00:37:19.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.516+0000 7ff1c9a96740 1 -- start start 2024-09-16T00:37:19.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.516+0000 7ff1c9a96740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0ef60 0x5558c8d0f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.516+0000 7ff1c9a96740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0f880 0x5558c8d1e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.516+0000 7ff1c9a96740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d1e650 0x5558c8d20a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.516+0000 7ff1c9a96740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5558c8c74620 con 0x5558c8d0f880 2024-09-16T00:37:19.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.516+0000 7ff1c9a96740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5558c8c1be50 con 0x5558c8d0ef60 2024-09-16T00:37:19.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.516+0000 7ff1c9a96740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5558c8c705e0 con 0x5558c8d1e650 2024-09-16T00:37:19.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.516+0000 7ff1c5459640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0f880 0x5558c8d1e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c5459640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0f880 0x5558c8d1e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42774/0 (socket says 172.21.15.33:42774) 2024-09-16T00:37:19.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1b67fc640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff1ac0a3240 0x7ff1ac0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c5459640 1 -- 172.21.15.33:0/2061247538 learned_addr learned my addr 172.21.15.33:0/2061247538 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:19.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/2061247538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0ef60 0x5558c8d0f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c5c5a640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff1ac0a3240 0x7ff1ac0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/2061247538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d1e650 0x5558c8d20a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1b67fc640 1 -- 172.21.15.33:0/533373216 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff1b80f1170 con 0x7ff1c0106f10 2024-09-16T00:37:19.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c5459640 1 -- 172.21.15.33:0/2061247538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d1e650 msgr2=0x5558c8d20a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:19.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c5459640 1 --2- 172.21.15.33:0/2061247538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d1e650 0x5558c8d20a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c5459640 1 -- 172.21.15.33:0/2061247538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0ef60 msgr2=0x5558c8d0f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:19.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c5459640 1 --2- 172.21.15.33:0/2061247538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0ef60 0x5558c8d0f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c5459640 1 -- 172.21.15.33:0/2061247538 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558c8c0cd40 con 0x5558c8d0f880 2024-09-16T00:37:19.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/2061247538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d1e650 0x5558c8d20a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:37:19.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c5459640 1 --2- 172.21.15.33:0/2061247538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0f880 0x5558c8d1e110 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc007850 tx=0x7ff1bc007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4ca51be4a8d5b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:19.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1c5c5a640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff1ac0a3240 0x7ff1ac0a5700 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff1b80b98c0 tx=0x7ff1b8090400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:19.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1ab7fe640 1 -- 172.21.15.33:0/2061247538 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc00d950 con 0x5558c8d0f880 2024-09-16T00:37:19.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1ab7fe640 1 -- 172.21.15.33:0/2061247538 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1bc018070 con 0x5558c8d0f880 2024-09-16T00:37:19.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.517+0000 7ff1ab7fe640 1 -- 172.21.15.33:0/2061247538 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0123e0 con 0x5558c8d0f880 2024-09-16T00:37:19.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.518+0000 7ff1c9a96740 1 -- 172.21.15.33:0/2061247538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0f880 msgr2=0x5558c8d1e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:19.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.518+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/2061247538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0f880 0x5558c8d1e110 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc007850 tx=0x7ff1bc007d10 comp rx=0 tx=0).stop 2024-09-16T00:37:19.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.518+0000 7ff1c9a96740 1 -- 172.21.15.33:0/2061247538 shutdown_connections 2024-09-16T00:37:19.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.518+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/2061247538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d1e650 0x5558c8d20a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.518+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/2061247538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0f880 0x5558c8d1e110 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.518+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/2061247538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0ef60 0x5558c8d0f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.518+0000 7ff1c9a96740 1 -- 172.21.15.33:0/2061247538 >> 172.21.15.33:0/2061247538 conn(0x5558c8d0b5f0 msgr2=0x5558c8d0b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:19.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.518+0000 7ff1c9a96740 1 -- 172.21.15.33:0/2061247538 shutdown_connections 2024-09-16T00:37:19.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.518+0000 7ff1c9a96740 1 -- 172.21.15.33:0/2061247538 wait complete. 2024-09-16T00:37:19.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c9a96740 1 -- start start 2024-09-16T00:37:19.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c9a96740 1 Processor -- start 2024-09-16T00:37:19.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c9a96740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d0ef60 0x5558c8d1b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c9a96740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0f880 0x5558c8d1ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c9a96740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d1e650 0x5558c8d1bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c9a96740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5558c8d21280 con 0x5558c8d1e650 2024-09-16T00:37:19.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c9a96740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5558c8bfcf10 con 0x5558c8d0f880 2024-09-16T00:37:19.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c9a96740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5558c8d21400 con 0x5558c8d0ef60 2024-09-16T00:37:19.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c9a96740 10 client.0 _ll_register_callbacks cb 0x5558c8c20db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:19.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c4c58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d1e650 0x5558c8d1bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c4c58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d1e650 0x5558c8d1bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42786/0 (socket says 172.21.15.33:42786) 2024-09-16T00:37:19.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c4c58640 1 -- 172.21.15.33:0/1685713977 learned_addr learned my addr 172.21.15.33:0/1685713977 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:19.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.519+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d0ef60 0x5558c8d1b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1c5459640 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0f880 0x5558c8d1ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1c4c58640 1 -- 172.21.15.33:0/1685713977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d0ef60 msgr2=0x5558c8d1b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d0ef60 0x5558c8d1b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1c4c58640 1 -- 172.21.15.33:0/1685713977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0f880 msgr2=0x5558c8d1ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0f880 0x5558c8d1ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1c4c58640 1 -- 172.21.15.33:0/1685713977 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558c8983580 con 0x5558c8d1e650 2024-09-16T00:37:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d1e650 0x5558c8d1bfc0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0012920 tx=0x7ff1b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c29c50ec3f08bf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1a8ff9640 1 -- 172.21.15.33:0/1685713977 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0010ae0 con 0x5558c8d1e650 2024-09-16T00:37:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1c9a96740 1 -- 172.21.15.33:0/1685713977 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1bc018070 con 0x5558c8d1e650 2024-09-16T00:37:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1a8ff9640 1 -- 172.21.15.33:0/1685713977 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff1b0010c80 con 0x5558c8d1e650 2024-09-16T00:37:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1a8ff9640 1 -- 172.21.15.33:0/1685713977 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff1b0029050 con 0x5558c8d1e650 2024-09-16T00:37:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1c9a96740 10 client.19197 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1c9a96740 1 -- 172.21.15.33:0/1685713977 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5558c8a4d960 con 0x5558c8d1e650 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.520+0000 7ff1a8ff9640 1 -- 172.21.15.33:0/1685713977 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1bc018070 con 0x5558c8d1e650 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.521+0000 7ff1a8ff9640 1 -- 172.21.15.33:0/1685713977 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ff1b0003bc0 con 0x5558c8d1e650 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.521+0000 7ff1c9a96740 10 client.19197 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.521+0000 7ff1c9a96740 10 client.19197 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.521+0000 7ff1c9a96740 20 client.19197 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.521+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5558c8d2e2c0 0x5558c8d306b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.521+0000 7ff1c9a96740 4 client.19197 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.521+0000 7ff1c9a96740 1 -- 172.21.15.33:0/1685713977 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558c8c81b90 con 0x5558c8d2e2c0 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.521+0000 7ff1c5459640 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5558c8d2e2c0 0x5558c8d306b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.522+0000 7ff1c5459640 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5558c8d2e2c0 0x5558c8d306b0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6add9f3a4cb530c server_cookie=a54b0771ac81f20c in_seq=0 out_seq=0 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.522+0000 7ff1a8ff9640 10 client.19197 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.522+0000 7ff1a8ff9640 1 -- 172.21.15.33:0/1685713977 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5558c8c81b90 con 0x5558c8d2e2c0 2024-09-16T00:37:19.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.522+0000 7ff1a8ff9640 10 client.19197 handle_command_reply: tid=0 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.522+0000 7ff1c9a96740 1 client.19197 shutdown 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.523+0000 7ff1c9a96740 1 -- 172.21.15.33:0/1685713977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d1e650 msgr2=0x5558c8d1bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.523+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d1e650 0x5558c8d1bfc0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0012920 tx=0x7ff1b0004730 comp rx=0 tx=0).stop 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.523+0000 7ff1c9a96740 1 -- 172.21.15.33:0/1685713977 shutdown_connections 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.523+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5558c8d2e2c0 0x5558c8d306b0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.523+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d1e650 0x5558c8d1bfc0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.523+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0f880 0x5558c8d1ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.523+0000 7ff1a8ff9640 0 client.19197 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.523+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/1685713977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d0ef60 0x5558c8d1b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.523+0000 7ff1c9a96740 1 -- 172.21.15.33:0/1685713977 >> 172.21.15.33:0/1685713977 conn(0x5558c8d0b5f0 msgr2=0x5558c8d11350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.523+0000 7ff1c9a96740 1 -- 172.21.15.33:0/1685713977 shutdown_connections 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.523+0000 7ff1c9a96740 1 -- 172.21.15.33:0/1685713977 wait complete. 2024-09-16T00:37:19.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.523+0000 7ff1c9a96740 20 client.19197 trim_cache size 0 max 16384 2024-09-16T00:37:19.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c9a96740 1 Processor -- start 2024-09-16T00:37:19.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c9a96740 1 -- start start 2024-09-16T00:37:19.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c9a96740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 0x5558c8d111a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c9a96740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0f880 0x5558c8d35ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c9a96740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d1e650 0x5558c8d36420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c9a96740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c01af800 con 0x5558c8d0ef60 2024-09-16T00:37:19.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c9a96740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c01ad0d0 con 0x5558c8d0f880 2024-09-16T00:37:19.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c9a96740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c01aaf20 con 0x5558c8d1e650 2024-09-16T00:37:19.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c4c58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 0x5558c8d111a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c4c58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 0x5558c8d111a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42792/0 (socket says 172.21.15.33:42792) 2024-09-16T00:37:19.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c4c58640 1 -- 172.21.15.33:0/2595331039 learned_addr learned my addr 172.21.15.33:0/2595331039 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:19.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c5459640 1 --2- 172.21.15.33:0/2595331039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0f880 0x5558c8d35ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c4c58640 1 -- 172.21.15.33:0/2595331039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d1e650 msgr2=0x5558c8d36420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:19.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/2595331039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d1e650 0x5558c8d36420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c4c58640 1 -- 172.21.15.33:0/2595331039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0f880 msgr2=0x5558c8d35ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:19.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/2595331039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0f880 0x5558c8d35ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c4c58640 1 -- 172.21.15.33:0/2595331039 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558c8d0c780 con 0x5558c8d0ef60 2024-09-16T00:37:19.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c5459640 1 --2- 172.21.15.33:0/2595331039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0f880 0x5558c8d35ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:37:19.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.540+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/2595331039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 0x5558c8d111a0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0000c00 tx=0x7ff1b0005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e503161aca80d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:19.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.541+0000 7ff1aaffd640 1 -- 172.21.15.33:0/2595331039 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b00120d0 con 0x5558c8d0ef60 2024-09-16T00:37:19.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.541+0000 7ff1aaffd640 1 -- 172.21.15.33:0/2595331039 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1b00205c0 con 0x5558c8d0ef60 2024-09-16T00:37:19.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.541+0000 7ff1aaffd640 1 -- 172.21.15.33:0/2595331039 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0003f90 con 0x5558c8d0ef60 2024-09-16T00:37:19.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.541+0000 7ff1c9a96740 1 -- 172.21.15.33:0/2595331039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 msgr2=0x5558c8d111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:19.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.541+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/2595331039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 0x5558c8d111a0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0000c00 tx=0x7ff1b0005900 comp rx=0 tx=0).stop 2024-09-16T00:37:19.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.541+0000 7ff1c9a96740 1 -- 172.21.15.33:0/2595331039 shutdown_connections 2024-09-16T00:37:19.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.541+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/2595331039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d1e650 0x5558c8d36420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.541+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/2595331039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d0f880 0x5558c8d35ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.541+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/2595331039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 0x5558c8d111a0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.541+0000 7ff1c9a96740 1 -- 172.21.15.33:0/2595331039 >> 172.21.15.33:0/2595331039 conn(0x5558c8d0b5f0 msgr2=0x5558c8d10a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:19.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.541+0000 7ff1c9a96740 1 -- 172.21.15.33:0/2595331039 shutdown_connections 2024-09-16T00:37:19.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.541+0000 7ff1c9a96740 1 -- 172.21.15.33:0/2595331039 wait complete. 2024-09-16T00:37:19.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.542+0000 7ff1c9a96740 1 -- start start 2024-09-16T00:37:19.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.542+0000 7ff1c9a96740 1 Processor -- start 2024-09-16T00:37:19.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.542+0000 7ff1c9a96740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 0x5558c8d26690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.542+0000 7ff1c9a96740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d0f880 0x5558c8d26bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.542+0000 7ff1c9a96740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d1e650 0x5558c8d23660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.542+0000 7ff1c9a96740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c01ab590 con 0x5558c8d0ef60 2024-09-16T00:37:19.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.542+0000 7ff1c9a96740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c01aca00 con 0x5558c8d1e650 2024-09-16T00:37:19.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.542+0000 7ff1c9a96740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c01abc60 con 0x5558c8d0f880 2024-09-16T00:37:19.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.542+0000 7ff1c9a96740 10 client.0 _ll_register_callbacks cb 0x7ff1b00123f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:19.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c4c58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 0x5558c8d26690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c4c58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 0x5558c8d26690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42798/0 (socket says 172.21.15.33:42798) 2024-09-16T00:37:19.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c4c58640 1 -- 172.21.15.33:0/747089890 learned_addr learned my addr 172.21.15.33:0/747089890 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:19.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c5459640 1 --2- 172.21.15.33:0/747089890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d0f880 0x5558c8d26bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c4c58640 1 -- 172.21.15.33:0/747089890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d0f880 msgr2=0x5558c8d26bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:19.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/747089890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d0f880 0x5558c8d26bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c4c58640 1 -- 172.21.15.33:0/747089890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d1e650 msgr2=0x5558c8d23660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:19.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/747089890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d1e650 0x5558c8d23660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c4c58640 1 -- 172.21.15.33:0/747089890 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558c8d23ba0 con 0x5558c8d0ef60 2024-09-16T00:37:19.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c4c58640 1 --2- 172.21.15.33:0/747089890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 0x5558c8d26690 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0020dd0 tx=0x7ff1b000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894e0f07a243870d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:19.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:19.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1a8ff9640 1 -- 172.21.15.33:0/747089890 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0010ae0 con 0x5558c8d0ef60 2024-09-16T00:37:19.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1a8ff9640 1 -- 172.21.15.33:0/747089890 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff1b0010c80 con 0x5558c8d0ef60 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c9a96740 1 -- 172.21.15.33:0/747089890 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1b00205c0 con 0x5558c8d0ef60 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1a8ff9640 1 -- 172.21.15.33:0/747089890 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff1b003c050 con 0x5558c8d0ef60 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c9a96740 10 client.19203 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.543+0000 7ff1c9a96740 1 -- 172.21.15.33:0/747089890 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5558c8d99550 con 0x5558c8d0ef60 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.544+0000 7ff1a8ff9640 1 -- 172.21.15.33:0/747089890 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1b00205c0 con 0x5558c8d0ef60 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.544+0000 7ff1a8ff9640 1 -- 172.21.15.33:0/747089890 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ff1b0003a30 con 0x5558c8d0ef60 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.544+0000 7ff1c9a96740 10 client.19203 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.544+0000 7ff1c9a96740 10 client.19203 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.544+0000 7ff1c9a96740 20 client.19203 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.544+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/747089890 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5558c8d99c60 0x5558c8d9c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.544+0000 7ff1c9a96740 4 client.19203 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e0f4c493", "release": true, "await": true}] 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.544+0000 7ff1c9a96740 1 -- 172.21.15.33:0/747089890 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e0f4c493", "release": true, "await": true}) -- 0x5558c8c7c0a0 con 0x5558c8d99c60 2024-09-16T00:37:19.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.544+0000 7ff1c5459640 1 --2- 172.21.15.33:0/747089890 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5558c8d99c60 0x5558c8d9c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:19.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.545+0000 7ff1a8ff9640 10 client.19203 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:19.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.545+0000 7ff1c5459640 1 --2- 172.21.15.33:0/747089890 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5558c8d99c60 0x5558c8d9c050 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdb485885686d673 server_cookie=27803aab0917223f in_seq=0 out_seq=0 2024-09-16T00:37:19.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.658+0000 7ff1a8ff9640 1 -- 172.21.15.33:0/747089890 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5558c8c7c0a0 con 0x5558c8d99c60 2024-09-16T00:37:19.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.658+0000 7ff1a8ff9640 10 client.19203 handle_command_reply: tid=0 2024-09-16T00:37:19.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.658+0000 7ff1c9a96740 1 client.19203 shutdown 2024-09-16T00:37:19.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.658+0000 7ff1c9a96740 1 -- 172.21.15.33:0/747089890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 msgr2=0x5558c8d26690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:19.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.658+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/747089890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 0x5558c8d26690 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0020dd0 tx=0x7ff1b000b480 comp rx=0 tx=0).stop 2024-09-16T00:37:19.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.659+0000 7ff1c9a96740 1 -- 172.21.15.33:0/747089890 shutdown_connections 2024-09-16T00:37:19.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.659+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/747089890 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5558c8d99c60 0x5558c8d9c050 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.659+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/747089890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5558c8d1e650 0x5558c8d23660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.659+0000 7ff1a8ff9640 0 client.19203 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:19.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.659+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/747089890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558c8d0f880 0x5558c8d26bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.659+0000 7ff1c9a96740 1 --2- 172.21.15.33:0/747089890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5558c8d0ef60 0x5558c8d26690 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.659+0000 7ff1c9a96740 1 -- 172.21.15.33:0/747089890 >> 172.21.15.33:0/747089890 conn(0x5558c8d0b5f0 msgr2=0x5558c8d1bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:19.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.659+0000 7ff1c9a96740 1 -- 172.21.15.33:0/747089890 shutdown_connections 2024-09-16T00:37:19.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.659+0000 7ff1c9a96740 1 -- 172.21.15.33:0/747089890 wait complete. 2024-09-16T00:37:19.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.659+0000 7ff1c9a96740 20 client.19203 trim_cache size 0 max 16384 2024-09-16T00:37:19.662 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:37:19.662 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:37:19.662 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:37:19.662 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 72, 2024-09-16T00:37:19.662 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:37:19.662 INFO:teuthology.orchestra.run.smithi033.stdout: "e0f4c493": { 2024-09-16T00:37:19.662 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 72, 2024-09-16T00:37:19.662 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:37:19.662 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:37:19.662 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:37:19.662 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:37:19.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 148.2, 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:19.663 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:19.664 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:37:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.660+0000 7ff1c76bd640 1 -- 172.21.15.33:0/533373216 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff1ac0a3240 msgr2=0x7ff1ac0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.660+0000 7ff1c76bd640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff1ac0a3240 0x7ff1ac0a5700 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff1b80b98c0 tx=0x7ff1b8090400 comp rx=0 tx=0).stop 2024-09-16T00:37:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.660+0000 7ff1c76bd640 1 -- 172.21.15.33:0/533373216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1c0106f10 msgr2=0x7ff1c019cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.660+0000 7ff1c76bd640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1c0106f10 0x7ff1c019cd80 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8031a20 tx=0x7ff1b8031a50 comp rx=0 tx=0).stop 2024-09-16T00:37:19.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.660+0000 7ff1c76bd640 1 -- 172.21.15.33:0/533373216 shutdown_connections 2024-09-16T00:37:19.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.660+0000 7ff1c76bd640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff1ac0a3240 0x7ff1ac0a5700 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.660+0000 7ff1c76bd640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1c0106f10 0x7ff1c019cd80 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.660+0000 7ff1c76bd640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c0102790 0x7ff1c01989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.660+0000 7ff1c76bd640 1 --2- 172.21.15.33:0/533373216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1c0101de0 0x7ff1c01984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:19.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.660+0000 7ff1c76bd640 1 -- 172.21.15.33:0/533373216 >> 172.21.15.33:0/533373216 conn(0x7ff1c00fd8c0 msgr2=0x7ff1c00fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:19.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.660+0000 7ff1c76bd640 1 -- 172.21.15.33:0/533373216 shutdown_connections 2024-09-16T00:37:19.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:19.660+0000 7ff1c76bd640 1 -- 172.21.15.33:0/533373216 wait complete. 2024-09-16T00:37:20.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:19 smithi033 ceph-mon[31243]: pgmap v955: 129 pgs: 129 active+clean; 619 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 3.8 MiB/s wr, 154 op/s 2024-09-16T00:37:20.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 132428 17.94 MB/sec execute 266 sec latency 27985.452 ms 2024-09-16T00:37:20.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:19 smithi083 ceph-mon[35242]: pgmap v955: 129 pgs: 129 active+clean; 619 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 3.8 MiB/s wr, 154 op/s 2024-09-16T00:37:20.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:19 smithi060 ceph-mon[35185]: pgmap v955: 129 pgs: 129 active+clean; 619 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 3.8 MiB/s wr, 154 op/s 2024-09-16T00:37:20.906 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'e0f4c493', total seconds elapsed: 32.3 2024-09-16T00:37:20.906 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 28.15 2024-09-16T00:37:20.907 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:37:21.055 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:20 smithi033 ceph-mon[31243]: pgmap v956: 129 pgs: 129 active+clean; 618 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 4.8 MiB/s wr, 193 op/s 2024-09-16T00:37:21.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 133049 17.99 MB/sec execute 267 sec latency 109.791 ms 2024-09-16T00:37:21.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:20 smithi083 ceph-mon[35242]: pgmap v956: 129 pgs: 129 active+clean; 618 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 4.8 MiB/s wr, 193 op/s 2024-09-16T00:37:21.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:20 smithi060 ceph-mon[35185]: pgmap v956: 129 pgs: 129 active+clean; 618 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 4.8 MiB/s wr, 193 op/s 2024-09-16T00:37:21.360 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:21.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.901+0000 7efe2c2df640 1 -- 172.21.15.33:0/57459363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe24106f20 msgr2=0x7efe2410eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:21.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.901+0000 7efe2c2df640 1 --2- 172.21.15.33:0/57459363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe24106f20 0x7efe2410eaa0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efe1c009ca0 tx=0x7efe1c02a8c0 comp rx=0 tx=0).stop 2024-09-16T00:37:21.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.901+0000 7efe2c2df640 1 -- 172.21.15.33:0/57459363 shutdown_connections 2024-09-16T00:37:21.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.901+0000 7efe2c2df640 1 --2- 172.21.15.33:0/57459363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe24106f20 0x7efe2410eaa0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:21.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.901+0000 7efe2c2df640 1 --2- 172.21.15.33:0/57459363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe241027a0 0x7efe241067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:21.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.901+0000 7efe2c2df640 1 --2- 172.21.15.33:0/57459363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe24101df0 0x7efe241021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:21.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.901+0000 7efe2c2df640 1 -- 172.21.15.33:0/57459363 >> 172.21.15.33:0/57459363 conn(0x7efe240fd8f0 msgr2=0x7efe240ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:21.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.901+0000 7efe2c2df640 1 -- 172.21.15.33:0/57459363 shutdown_connections 2024-09-16T00:37:21.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.901+0000 7efe2c2df640 1 -- 172.21.15.33:0/57459363 wait complete. 2024-09-16T00:37:21.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2c2df640 1 Processor -- start 2024-09-16T00:37:21.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2c2df640 1 -- start start 2024-09-16T00:37:21.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2c2df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe24101df0 0x7efe2410e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:21.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2c2df640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe241027a0 0x7efe2410baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:21.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2c2df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe24106f20 0x7efe2410c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:21.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2c2df640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efe24075fc0 con 0x7efe241027a0 2024-09-16T00:37:21.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2c2df640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efe24075e40 con 0x7efe24101df0 2024-09-16T00:37:21.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2c2df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efe24076140 con 0x7efe24106f20 2024-09-16T00:37:21.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2987a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe241027a0 0x7efe2410baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:21.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2a07b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe24101df0 0x7efe2410e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:21.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2987a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe241027a0 0x7efe2410baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52544/0 (socket says 172.21.15.33:52544) 2024-09-16T00:37:21.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2987a640 1 -- 172.21.15.33:0/1534827536 learned_addr learned my addr 172.21.15.33:0/1534827536 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:21.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.902+0000 7efe2a87c640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe24106f20 0x7efe2410c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:21.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.903+0000 7efe2a07b640 1 -- 172.21.15.33:0/1534827536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe24106f20 msgr2=0x7efe2410c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:21.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.903+0000 7efe2a07b640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe24106f20 0x7efe2410c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:21.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.903+0000 7efe2a07b640 1 -- 172.21.15.33:0/1534827536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe241027a0 msgr2=0x7efe2410baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:21.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.903+0000 7efe2a07b640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe241027a0 0x7efe2410baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:21.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.903+0000 7efe2a07b640 1 -- 172.21.15.33:0/1534827536 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe241aadc0 con 0x7efe24101df0 2024-09-16T00:37:21.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.903+0000 7efe2a07b640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe24101df0 0x7efe2410e9f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efe1400e990 tx=0x7efe1400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:21.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.903+0000 7efe1b7fe640 1 -- 172.21.15.33:0/1534827536 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1400cbe0 con 0x7efe24101df0 2024-09-16T00:37:21.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.903+0000 7efe1b7fe640 1 -- 172.21.15.33:0/1534827536 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe1400cd80 con 0x7efe24101df0 2024-09-16T00:37:21.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.903+0000 7efe2c2df640 1 -- 172.21.15.33:0/1534827536 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe241aaf90 con 0x7efe24101df0 2024-09-16T00:37:21.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.903+0000 7efe1b7fe640 1 -- 172.21.15.33:0/1534827536 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1401a780 con 0x7efe24101df0 2024-09-16T00:37:21.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.903+0000 7efe2c2df640 1 -- 172.21.15.33:0/1534827536 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe241ab440 con 0x7efe24101df0 2024-09-16T00:37:21.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.905+0000 7efe2c2df640 1 -- 172.21.15.33:0/1534827536 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe241032d0 con 0x7efe24101df0 2024-09-16T00:37:21.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.906+0000 7efe1b7fe640 1 -- 172.21.15.33:0/1534827536 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7efe14011470 con 0x7efe24101df0 2024-09-16T00:37:21.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.907+0000 7efe1b7fe640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efe100a3240 0x7efe100a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:21.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.907+0000 7efe1b7fe640 1 -- 172.21.15.33:0/1534827536 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7efe140cff60 con 0x7efe24101df0 2024-09-16T00:37:21.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.907+0000 7efe2987a640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efe100a3240 0x7efe100a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:21.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.908+0000 7efe2987a640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efe100a3240 0x7efe100a5700 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7efe2410d260 tx=0x7efe20005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:21.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:21.913+0000 7efe1b7fe640 1 -- 172.21.15.33:0/1534827536 <== mon.2 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7efe14016030 con 0x7efe24101df0 2024-09-16T00:37:22.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 133501 18.00 MB/sec execute 268 sec latency 90.715 ms 2024-09-16T00:37:22.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.229+0000 7efe2c2df640 1 -- 172.21.15.33:0/1534827536 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe241116c0 con 0x7efe24101df0 2024-09-16T00:37:22.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.230+0000 7efe1b7fe640 1 -- 172.21.15.33:0/1534827536 <== mon.2 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7efe1409c7a0 con 0x7efe24101df0 2024-09-16T00:37:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:37:22.232 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:37:22.232 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:37:22.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.233+0000 7efe2c2df640 1 -- 172.21.15.33:0/1534827536 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efe100a3240 msgr2=0x7efe100a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:22.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.233+0000 7efe2c2df640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efe100a3240 0x7efe100a5700 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7efe2410d260 tx=0x7efe20005f90 comp rx=0 tx=0).stop 2024-09-16T00:37:22.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.233+0000 7efe2c2df640 1 -- 172.21.15.33:0/1534827536 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe24101df0 msgr2=0x7efe2410e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:22.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.233+0000 7efe2c2df640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe24101df0 0x7efe2410e9f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efe1400e990 tx=0x7efe1400ee50 comp rx=0 tx=0).stop 2024-09-16T00:37:22.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.233+0000 7efe2c2df640 1 -- 172.21.15.33:0/1534827536 shutdown_connections 2024-09-16T00:37:22.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.233+0000 7efe2c2df640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efe100a3240 0x7efe100a5700 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:22.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.233+0000 7efe2c2df640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe24106f20 0x7efe2410c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:22.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.233+0000 7efe2c2df640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe241027a0 0x7efe2410baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:22.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.233+0000 7efe2c2df640 1 --2- 172.21.15.33:0/1534827536 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe24101df0 0x7efe2410e9f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:22.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.233+0000 7efe2c2df640 1 -- 172.21.15.33:0/1534827536 >> 172.21.15.33:0/1534827536 conn(0x7efe240fd8f0 msgr2=0x7efe240febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:22.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.233+0000 7efe2c2df640 1 -- 172.21.15.33:0/1534827536 shutdown_connections 2024-09-16T00:37:22.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:22.233+0000 7efe2c2df640 1 -- 172.21.15.33:0/1534827536 wait complete. 2024-09-16T00:37:23.057 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 148.15 --await-for 45.0' 2024-09-16T00:37:23.057 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 148.15 --await-for 45.0 2024-09-16T00:37:23.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134029 18.07 MB/sec execute 269 sec latency 68.723 ms 2024-09-16T00:37:23.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:23 smithi033 ceph-mon[31243]: pgmap v957: 129 pgs: 129 active+clean; 614 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 4.8 MiB/s wr, 197 op/s 2024-09-16T00:37:23.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:23 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1534827536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:37:23.390 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:23.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:23 smithi083 ceph-mon[35242]: pgmap v957: 129 pgs: 129 active+clean; 614 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 4.8 MiB/s wr, 197 op/s 2024-09-16T00:37:23.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:23 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1534827536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:37:23.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:23 smithi060 ceph-mon[35185]: pgmap v957: 129 pgs: 129 active+clean; 614 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 4.8 MiB/s wr, 197 op/s 2024-09-16T00:37:23.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:23 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1534827536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:37:23.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.923+0000 7f9ab0243640 1 -- 172.21.15.33:0/3647896862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa8102790 msgr2=0x7f9aa81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:23.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.923+0000 7f9ab0243640 1 --2- 172.21.15.33:0/3647896862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa8102790 0x7f9aa81067e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009900 tx=0x7f9a9802a410 comp rx=0 tx=0).stop 2024-09-16T00:37:23.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.923+0000 7f9ab0243640 1 -- 172.21.15.33:0/3647896862 shutdown_connections 2024-09-16T00:37:23.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.923+0000 7f9ab0243640 1 --2- 172.21.15.33:0/3647896862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aa8106f10 0x7f9aa810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.923+0000 7f9ab0243640 1 --2- 172.21.15.33:0/3647896862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa8102790 0x7f9aa81067e0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.923+0000 7f9ab0243640 1 --2- 172.21.15.33:0/3647896862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aa8101de0 0x7f9aa81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.923+0000 7f9ab0243640 1 -- 172.21.15.33:0/3647896862 >> 172.21.15.33:0/3647896862 conn(0x7f9aa80fd880 msgr2=0x7f9aa80ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.923+0000 7f9ab0243640 1 -- 172.21.15.33:0/3647896862 shutdown_connections 2024-09-16T00:37:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.923+0000 7f9ab0243640 1 -- 172.21.15.33:0/3647896862 wait complete. 2024-09-16T00:37:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9ab0243640 1 Processor -- start 2024-09-16T00:37:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9ab0243640 1 -- start start 2024-09-16T00:37:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9ab0243640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa8101de0 0x7f9aa81983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9ab0243640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aa8102790 0x7f9aa8198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9ab0243640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aa8106f10 0x7f9aa819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9ab0243640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa8111f70 con 0x7f9aa8101de0 2024-09-16T00:37:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9ab0243640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa8111df0 con 0x7f9aa8106f10 2024-09-16T00:37:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9ab0243640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa81120f0 con 0x7f9aa8102790 2024-09-16T00:37:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9aadfdf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa8101de0 0x7f9aa81983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9aadfdf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa8101de0 0x7f9aa81983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52566/0 (socket says 172.21.15.33:52566) 2024-09-16T00:37:23.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9aadfdf640 1 -- 172.21.15.33:0/3176009554 learned_addr learned my addr 172.21.15.33:0/3176009554 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:23.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9aad7de640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aa8102790 0x7f9aa8198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9aae7e0640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aa8106f10 0x7f9aa819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9aadfdf640 1 -- 172.21.15.33:0/3176009554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aa8102790 msgr2=0x7f9aa8198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:23.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9aadfdf640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aa8102790 0x7f9aa8198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9aadfdf640 1 -- 172.21.15.33:0/3176009554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aa8106f10 msgr2=0x7f9aa819cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:23.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9aadfdf640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aa8106f10 0x7f9aa819cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9aadfdf640 1 -- 172.21.15.33:0/3176009554 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa819d380 con 0x7f9aa8101de0 2024-09-16T00:37:23.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.924+0000 7f9aae7e0640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aa8106f10 0x7f9aa819cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:23.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.925+0000 7f9aadfdf640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa8101de0 0x7f9aa81983d0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400ea60 tx=0x7f9aa400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:23.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.925+0000 7f9a9effd640 1 -- 172.21.15.33:0/3176009554 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa400cde0 con 0x7f9aa8101de0 2024-09-16T00:37:23.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.925+0000 7f9a9effd640 1 -- 172.21.15.33:0/3176009554 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa40035f0 con 0x7f9aa8101de0 2024-09-16T00:37:23.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.925+0000 7f9ab0243640 1 -- 172.21.15.33:0/3176009554 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa819d670 con 0x7f9aa8101de0 2024-09-16T00:37:23.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.925+0000 7f9a9effd640 1 -- 172.21.15.33:0/3176009554 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4010670 con 0x7f9aa8101de0 2024-09-16T00:37:23.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.925+0000 7f9ab0243640 1 -- 172.21.15.33:0/3176009554 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa81a4f50 con 0x7f9aa8101de0 2024-09-16T00:37:23.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.929+0000 7f9a9effd640 1 -- 172.21.15.33:0/3176009554 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9aa40037a0 con 0x7f9aa8101de0 2024-09-16T00:37:23.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.929+0000 7f9ab261c740 1 Processor -- start 2024-09-16T00:37:23.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.929+0000 7f9ab261c740 1 -- start start 2024-09-16T00:37:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.929+0000 7f9ab261c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e736850 0x55ba8e736c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.929+0000 7f9ab261c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e737170 0x55ba8e7414f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.929+0000 7f9ab261c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 0x55ba8e743e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.929+0000 7f9ab261c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ba8e69b160 con 0x55ba8e736850 2024-09-16T00:37:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.929+0000 7f9ab261c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ba8e642720 con 0x55ba8e741a30 2024-09-16T00:37:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.929+0000 7f9ab261c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ba8e697120 con 0x55ba8e737170 2024-09-16T00:37:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aae7e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e736850 0x55ba8e736c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aae7e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e736850 0x55ba8e736c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52580/0 (socket says 172.21.15.33:52580) 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aae7e0640 1 -- 172.21.15.33:0/3297725351 learned_addr learned my addr 172.21.15.33:0/3297725351 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aadfdf640 1 --2- 172.21.15.33:0/3297725351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e737170 0x55ba8e7414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aad7de640 1 --2- 172.21.15.33:0/3297725351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 0x55ba8e743e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9a9effd640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a940a3240 0x7f9a940a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aae7e0640 1 -- 172.21.15.33:0/3297725351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e737170 msgr2=0x55ba8e7414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9a9effd640 1 -- 172.21.15.33:0/3176009554 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9aa40d0060 con 0x7f9aa8101de0 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aae7e0640 1 --2- 172.21.15.33:0/3297725351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e737170 0x55ba8e7414f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aae7e0640 1 -- 172.21.15.33:0/3297725351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 msgr2=0x55ba8e743e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aae7e0640 1 --2- 172.21.15.33:0/3297725351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 0x55ba8e743e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aae7e0640 1 -- 172.21.15.33:0/3297725351 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba8e40cba0 con 0x55ba8e736850 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aad7de640 1 --2- 172.21.15.33:0/3297725351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 0x55ba8e743e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aadfdf640 1 --2- 172.21.15.33:0/3297725351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e737170 0x55ba8e7414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:37:23.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.930+0000 7f9aae7e0640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a940a3240 0x7f9a940a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.931+0000 7f9aae7e0640 1 --2- 172.21.15.33:0/3297725351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e736850 0x55ba8e736c30 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00118d0 tx=0x7f9aa0011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40800411a7176a51 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.931+0000 7f9a93fff640 1 -- 172.21.15.33:0/3297725351 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0017050 con 0x55ba8e736850 2024-09-16T00:37:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.931+0000 7f9a93fff640 1 -- 172.21.15.33:0/3297725351 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa0026440 con 0x55ba8e736850 2024-09-16T00:37:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.931+0000 7f9a93fff640 1 -- 172.21.15.33:0/3297725351 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa000e5b0 con 0x55ba8e736850 2024-09-16T00:37:23.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.931+0000 7f9aae7e0640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a940a3240 0x7f9a940a5700 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0023000 tx=0x7f9aa0024000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:23.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.932+0000 7f9ab261c740 1 -- 172.21.15.33:0/3297725351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e736850 msgr2=0x55ba8e736c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.932+0000 7f9ab261c740 1 --2- 172.21.15.33:0/3297725351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e736850 0x55ba8e736c30 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00118d0 tx=0x7f9aa0011d90 comp rx=0 tx=0).stop 2024-09-16T00:37:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.932+0000 7f9ab261c740 1 -- 172.21.15.33:0/3297725351 shutdown_connections 2024-09-16T00:37:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.932+0000 7f9ab261c740 1 --2- 172.21.15.33:0/3297725351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 0x55ba8e743e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.932+0000 7f9ab261c740 1 --2- 172.21.15.33:0/3297725351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e737170 0x55ba8e7414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.932+0000 7f9ab261c740 1 --2- 172.21.15.33:0/3297725351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e736850 0x55ba8e736c30 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.932+0000 7f9ab261c740 1 -- 172.21.15.33:0/3297725351 >> 172.21.15.33:0/3297725351 conn(0x55ba8e732de0 msgr2=0x55ba8e7331c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.932+0000 7f9ab261c740 1 -- 172.21.15.33:0/3297725351 shutdown_connections 2024-09-16T00:37:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.932+0000 7f9ab261c740 1 -- 172.21.15.33:0/3297725351 wait complete. 2024-09-16T00:37:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.932+0000 7f9ab261c740 1 -- start start 2024-09-16T00:37:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9ab261c740 1 Processor -- start 2024-09-16T00:37:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9ab261c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e73e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9ab261c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e737170 0x55ba8e73ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9ab261c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 0x55ba8e73f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9ab261c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ba8e744600 con 0x55ba8e741a30 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9ab261c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ba8e623900 con 0x55ba8e737170 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9ab261c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ba8e744780 con 0x55ba8e736850 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9ab261c740 10 client.0 _ll_register_callbacks cb 0x55ba8e621780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9aae7e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 0x55ba8e73f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9aae7e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 0x55ba8e73f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52586/0 (socket says 172.21.15.33:52586) 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9aae7e0640 1 -- 172.21.15.33:0/88960575 learned_addr learned my addr 172.21.15.33:0/88960575 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9aadfdf640 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e73e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.933+0000 7f9aad7de640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e737170 0x55ba8e73ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.934+0000 7f9aad7de640 1 -- 172.21.15.33:0/88960575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 msgr2=0x55ba8e73e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.934+0000 7f9aad7de640 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e73e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.934+0000 7f9aad7de640 1 -- 172.21.15.33:0/88960575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 msgr2=0x55ba8e73f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.934+0000 7f9aad7de640 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 0x55ba8e73f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.934+0000 7f9aad7de640 1 -- 172.21.15.33:0/88960575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba8e663c50 con 0x55ba8e737170 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.934+0000 7f9aae7e0640 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 0x55ba8e73f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.934+0000 7f9aad7de640 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e737170 0x55ba8e73ee70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9a98006010 tx=0x7f9a980317e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b9b366e87b8f778 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.934+0000 7f9a917fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.934+0000 7f9a917fa640 1 -- 172.21.15.33:0/88960575 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a98035040 con 0x55ba8e737170 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9ab261c740 1 -- 172.21.15.33:0/88960575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9aa0026440 con 0x55ba8e737170 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9a917fa640 1 -- 172.21.15.33:0/88960575 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9a98004460 con 0x55ba8e737170 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9a917fa640 1 -- 172.21.15.33:0/88960575 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9a9804e050 con 0x55ba8e737170 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9ab261c740 10 client.28463 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9ab261c740 1 -- 172.21.15.33:0/88960575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ba8e754640 con 0x55ba8e737170 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9a917fa640 1 -- 172.21.15.33:0/88960575 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a98041070 con 0x55ba8e737170 2024-09-16T00:37:23.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9a917fa640 1 -- 172.21.15.33:0/88960575 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f9a98004cc0 con 0x55ba8e737170 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9ab261c740 10 client.28463 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9ab261c740 10 client.28463 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9ab261c740 20 client.28463 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9ab261c740 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ba8e755b00 0x55ba8e757ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9ab261c740 4 client.28463 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.935+0000 7f9ab261c740 1 -- 172.21.15.33:0/88960575 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9aa0026440 con 0x55ba8e755b00 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.936+0000 7f9aadfdf640 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ba8e755b00 0x55ba8e757ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.936+0000 7f9aadfdf640 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ba8e755b00 0x55ba8e757ef0 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b999ce715b22f62 server_cookie=71ea2674bf5eb9ea in_seq=0 out_seq=0 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.936+0000 7f9a917fa640 10 client.28463 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9a917fa640 1 -- 172.21.15.33:0/88960575 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f9aa0026440 con 0x55ba8e755b00 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9a917fa640 10 client.28463 handle_command_reply: tid=0 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9ab261c740 1 client.28463 shutdown 2024-09-16T00:37:23.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9ab261c740 1 -- 172.21.15.33:0/88960575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e737170 msgr2=0x55ba8e73ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:23.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9ab261c740 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e737170 0x55ba8e73ee70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9a98006010 tx=0x7f9a980317e0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9ab261c740 1 -- 172.21.15.33:0/88960575 shutdown_connections 2024-09-16T00:37:23.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9ab261c740 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ba8e755b00 0x55ba8e757ef0 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9ab261c740 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 0x55ba8e73f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9ab261c740 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e737170 0x55ba8e73ee70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9ab261c740 1 --2- 172.21.15.33:0/88960575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e73e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9ab261c740 1 -- 172.21.15.33:0/88960575 >> 172.21.15.33:0/88960575 conn(0x55ba8e732de0 msgr2=0x55ba8e738ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:23.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9a917fa640 0 client.28463 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:23.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9ab261c740 1 -- 172.21.15.33:0/88960575 shutdown_connections 2024-09-16T00:37:23.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.937+0000 7f9ab261c740 1 -- 172.21.15.33:0/88960575 wait complete. 2024-09-16T00:37:23.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.938+0000 7f9ab261c740 20 client.28463 trim_cache size 0 max 16384 2024-09-16T00:37:23.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.959+0000 7f9ab261c740 1 Processor -- start 2024-09-16T00:37:23.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.959+0000 7f9ab261c740 1 -- start start 2024-09-16T00:37:23.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9ab261c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e7392b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9ab261c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e737170 0x55ba8e731dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9ab261c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 0x55ba8e732310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9ab261c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa81af8d0 con 0x55ba8e737170 2024-09-16T00:37:23.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9ab261c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa81ad1a0 con 0x55ba8e741a30 2024-09-16T00:37:23.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9ab261c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa81aaff0 con 0x55ba8e736850 2024-09-16T00:37:23.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aad7de640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e737170 0x55ba8e731dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aae7e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 0x55ba8e732310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aad7de640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e737170 0x55ba8e731dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52592/0 (socket says 172.21.15.33:52592) 2024-09-16T00:37:23.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aae7e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 0x55ba8e732310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:44912/0 (socket says 172.21.15.33:44912) 2024-09-16T00:37:23.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aadfdf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e7392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aad7de640 1 -- 172.21.15.33:0/2533869862 learned_addr learned my addr 172.21.15.33:0/2533869862 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:23.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aae7e0640 1 -- 172.21.15.33:0/2533869862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 msgr2=0x55ba8e7392b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:23.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aae7e0640 1 --2- 172.21.15.33:0/2533869862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e7392b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aae7e0640 1 -- 172.21.15.33:0/2533869862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e737170 msgr2=0x55ba8e731dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:23.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aae7e0640 1 --2- 172.21.15.33:0/2533869862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e737170 0x55ba8e731dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aae7e0640 1 -- 172.21.15.33:0/2533869862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba8e758520 con 0x55ba8e741a30 2024-09-16T00:37:23.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aad7de640 1 --2- 172.21.15.33:0/2533869862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e737170 0x55ba8e731dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:23.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.960+0000 7f9aae7e0640 1 --2- 172.21.15.33:0/2533869862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 0x55ba8e732310 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0000f80 tx=0x7f9aa0014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b993b32ab96396fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:23.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.961+0000 7f9a937fe640 1 -- 172.21.15.33:0/2533869862 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0037030 con 0x55ba8e741a30 2024-09-16T00:37:23.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.961+0000 7f9a937fe640 1 -- 172.21.15.33:0/2533869862 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa002c040 con 0x55ba8e741a30 2024-09-16T00:37:23.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.961+0000 7f9a937fe640 1 -- 172.21.15.33:0/2533869862 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa002b040 con 0x55ba8e741a30 2024-09-16T00:37:23.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.961+0000 7f9ab261c740 1 -- 172.21.15.33:0/2533869862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 msgr2=0x55ba8e732310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:23.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.961+0000 7f9ab261c740 1 --2- 172.21.15.33:0/2533869862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 0x55ba8e732310 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0000f80 tx=0x7f9aa0014000 comp rx=0 tx=0).stop 2024-09-16T00:37:23.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.961+0000 7f9ab261c740 1 -- 172.21.15.33:0/2533869862 shutdown_connections 2024-09-16T00:37:23.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.961+0000 7f9ab261c740 1 --2- 172.21.15.33:0/2533869862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e741a30 0x55ba8e732310 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.961+0000 7f9ab261c740 1 --2- 172.21.15.33:0/2533869862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e737170 0x55ba8e731dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.961+0000 7f9ab261c740 1 --2- 172.21.15.33:0/2533869862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e7392b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.961+0000 7f9ab261c740 1 -- 172.21.15.33:0/2533869862 >> 172.21.15.33:0/2533869862 conn(0x55ba8e732de0 msgr2=0x55ba8e7316c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:23.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.962+0000 7f9ab261c740 1 -- 172.21.15.33:0/2533869862 shutdown_connections 2024-09-16T00:37:23.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.962+0000 7f9ab261c740 1 -- 172.21.15.33:0/2533869862 wait complete. 2024-09-16T00:37:23.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.962+0000 7f9ab261c740 1 -- start start 2024-09-16T00:37:23.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.962+0000 7f9ab261c740 1 Processor -- start 2024-09-16T00:37:23.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.962+0000 7f9ab261c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e749bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.962+0000 7f9ab261c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e737170 0x55ba8e74a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.962+0000 7f9ab261c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 0x55ba8e7469a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.962+0000 7f9ab261c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa81ab660 con 0x55ba8e741a30 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.962+0000 7f9ab261c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa81acad0 con 0x55ba8e737170 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.962+0000 7f9ab261c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa81abd30 con 0x55ba8e736850 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.963+0000 7f9ab261c740 10 client.0 _ll_register_callbacks cb 0x7f9a98009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.963+0000 7f9aae7e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 0x55ba8e7469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.963+0000 7f9aae7e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 0x55ba8e7469a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52604/0 (socket says 172.21.15.33:52604) 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.963+0000 7f9aadfdf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e749bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.963+0000 7f9aae7e0640 1 -- 172.21.15.33:0/1891832460 learned_addr learned my addr 172.21.15.33:0/1891832460 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.963+0000 7f9aad7de640 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e737170 0x55ba8e74a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.963+0000 7f9aadfdf640 1 -- 172.21.15.33:0/1891832460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e737170 msgr2=0x55ba8e74a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.963+0000 7f9aadfdf640 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e737170 0x55ba8e74a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.963+0000 7f9aadfdf640 1 -- 172.21.15.33:0/1891832460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 msgr2=0x55ba8e7469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.963+0000 7f9aadfdf640 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 0x55ba8e7469a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.963+0000 7f9aadfdf640 1 -- 172.21.15.33:0/1891832460 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba8e7471d0 con 0x55ba8e736850 2024-09-16T00:37:23.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.963+0000 7f9aae7e0640 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 0x55ba8e7469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:23.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.966+0000 7f9aadfdf640 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e749bb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9aa40cd730 tx=0x7f9aa400ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6e4971877689316 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:23.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.968+0000 7f9a917fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:37:23.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.969+0000 7f9a917fa640 1 -- 172.21.15.33:0/1891832460 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa40b4030 con 0x55ba8e736850 2024-09-16T00:37:23.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.970+0000 7f9a917fa640 1 -- 172.21.15.33:0/1891832460 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9aa409a500 con 0x55ba8e736850 2024-09-16T00:37:23.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.970+0000 7f9ab261c740 1 -- 172.21.15.33:0/1891832460 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9aa002c040 con 0x55ba8e736850 2024-09-16T00:37:23.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.970+0000 7f9a917fa640 1 -- 172.21.15.33:0/1891832460 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9aa4003b30 con 0x55ba8e736850 2024-09-16T00:37:23.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.970+0000 7f9ab261c740 10 client.28453 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:23.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.970+0000 7f9ab261c740 1 -- 172.21.15.33:0/1891832460 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ba8e7c1230 con 0x55ba8e736850 2024-09-16T00:37:23.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.970+0000 7f9a917fa640 1 -- 172.21.15.33:0/1891832460 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa002c040 con 0x55ba8e736850 2024-09-16T00:37:23.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.970+0000 7f9a917fa640 1 -- 172.21.15.33:0/1891832460 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f9aa40a49e0 con 0x55ba8e736850 2024-09-16T00:37:23.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.971+0000 7f9ab261c740 10 client.28453 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:23.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.971+0000 7f9ab261c740 10 client.28453 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:37:23.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.971+0000 7f9ab261c740 20 client.28453 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:23.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.971+0000 7f9ab261c740 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ba8e7c18d0 0x55ba8e7c3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:23.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.971+0000 7f9ab261c740 4 client.28453 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 148.15, "await_for": 45.0}] 2024-09-16T00:37:23.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.971+0000 7f9ab261c740 1 -- 172.21.15.33:0/1891832460 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 148.15, "await_for": 45.0}) -- 0x55ba8e72e410 con 0x55ba8e7c18d0 2024-09-16T00:37:23.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.971+0000 7f9aad7de640 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ba8e7c18d0 0x55ba8e7c3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:23.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.971+0000 7f9aad7de640 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ba8e7c18d0 0x55ba8e7c3cc0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e248a3ad14c49141 server_cookie=91f7188da2ffe2ab in_seq=0 out_seq=0 2024-09-16T00:37:23.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:23.971+0000 7f9a917fa640 10 client.28453 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:24.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134516 18.10 MB/sec execute 270 sec latency 125.571 ms 2024-09-16T00:37:24.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9a917fa640 1 -- 172.21.15.33:0/1891832460 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ba8e72e410 con 0x55ba8e7c18d0 2024-09-16T00:37:24.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9a917fa640 10 client.28453 handle_command_reply: tid=0 2024-09-16T00:37:24.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9ab261c740 1 client.28453 shutdown 2024-09-16T00:37:24.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9ab261c740 1 -- 172.21.15.33:0/1891832460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 msgr2=0x55ba8e749bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:24.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9ab261c740 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e749bb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9aa40cd730 tx=0x7f9aa400ef90 comp rx=0 tx=0).stop 2024-09-16T00:37:24.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9ab261c740 1 -- 172.21.15.33:0/1891832460 shutdown_connections 2024-09-16T00:37:24.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9ab261c740 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ba8e7c18d0 0x55ba8e7c3cc0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:24.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9ab261c740 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ba8e741a30 0x55ba8e7469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:24.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9ab261c740 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ba8e737170 0x55ba8e74a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:24.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9ab261c740 1 --2- 172.21.15.33:0/1891832460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ba8e736850 0x55ba8e749bb0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:24.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9ab261c740 1 -- 172.21.15.33:0/1891832460 >> 172.21.15.33:0/1891832460 conn(0x55ba8e732de0 msgr2=0x55ba8e73f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:24.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9a917fa640 0 client.28453 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:24.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9ab261c740 1 -- 172.21.15.33:0/1891832460 shutdown_connections 2024-09-16T00:37:24.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9ab261c740 1 -- 172.21.15.33:0/1891832460 wait complete. 2024-09-16T00:37:24.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.939+0000 7f9ab261c740 20 client.28453 trim_cache size 0 max 16384 2024-09-16T00:37:24.942 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:37:24.942 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:37:24.942 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:37:24.943 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 74, 2024-09-16T00:37:24.943 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:37:24.943 INFO:teuthology.orchestra.run.smithi033.stdout: "50781c66": { 2024-09-16T00:37:24.943 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 74, 2024-09-16T00:37:24.943 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:37:24.943 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:37:24.943 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:37:24.943 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:37:24.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:37:24.943 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 148.2, 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:24.944 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:37:24.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.940+0000 7f9ab0243640 1 -- 172.21.15.33:0/3176009554 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a940a3240 msgr2=0x7f9a940a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:24.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.940+0000 7f9ab0243640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a940a3240 0x7f9a940a5700 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0023000 tx=0x7f9aa0024000 comp rx=0 tx=0).stop 2024-09-16T00:37:24.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.940+0000 7f9ab0243640 1 -- 172.21.15.33:0/3176009554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa8101de0 msgr2=0x7f9aa81983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:24.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.940+0000 7f9ab0243640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa8101de0 0x7f9aa81983d0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400ea60 tx=0x7f9aa400ef20 comp rx=0 tx=0).stop 2024-09-16T00:37:24.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.940+0000 7f9ab0243640 1 -- 172.21.15.33:0/3176009554 shutdown_connections 2024-09-16T00:37:24.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.940+0000 7f9ab0243640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9a940a3240 0x7f9a940a5700 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:24.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.940+0000 7f9ab0243640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aa8106f10 0x7f9aa819cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:24.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.940+0000 7f9ab0243640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aa8102790 0x7f9aa8198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:24.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.940+0000 7f9ab0243640 1 --2- 172.21.15.33:0/3176009554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa8101de0 0x7f9aa81983d0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:24.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.940+0000 7f9ab0243640 1 -- 172.21.15.33:0/3176009554 >> 172.21.15.33:0/3176009554 conn(0x7f9aa80fd880 msgr2=0x7f9aa80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:24.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.940+0000 7f9ab0243640 1 -- 172.21.15.33:0/3176009554 shutdown_connections 2024-09-16T00:37:24.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:24.940+0000 7f9ab0243640 1 -- 172.21.15.33:0/3176009554 wait complete. 2024-09-16T00:37:25.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 18.04 MB/sec execute 271 sec latency 1125.608 ms 2024-09-16T00:37:25.258 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:25 smithi033 ceph-mon[31243]: pgmap v958: 129 pgs: 129 active+clean; 616 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 245 op/s 2024-09-16T00:37:25.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:25 smithi083 ceph-mon[35242]: pgmap v958: 129 pgs: 129 active+clean; 616 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 245 op/s 2024-09-16T00:37:25.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:25 smithi060 ceph-mon[35185]: pgmap v958: 129 pgs: 129 active+clean; 616 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 245 op/s 2024-09-16T00:37:25.724 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '50781c66', quiesce took 4.8 seconds. Will release after: 23.349999999999998 2024-09-16T00:37:26.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.97 MB/sec execute 272 sec latency 2125.654 ms 2024-09-16T00:37:27.088 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.91 MB/sec execute 273 sec latency 3125.758 ms 2024-09-16T00:37:27.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:27 smithi083 ceph-mon[35242]: pgmap v959: 129 pgs: 129 active+clean; 615 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.8 MiB/s wr, 340 op/s 2024-09-16T00:37:27.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:27 smithi060 ceph-mon[35185]: pgmap v959: 129 pgs: 129 active+clean; 615 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.8 MiB/s wr, 340 op/s 2024-09-16T00:37:27.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:27 smithi033 ceph-mon[31243]: pgmap v959: 129 pgs: 129 active+clean; 615 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.8 MiB/s wr, 340 op/s 2024-09-16T00:37:28.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.84 MB/sec execute 274 sec latency 4125.882 ms 2024-09-16T00:37:29.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.78 MB/sec execute 275 sec latency 5125.994 ms 2024-09-16T00:37:29.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:29 smithi083 ceph-mon[35242]: pgmap v960: 129 pgs: 129 active+clean; 615 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.6 MiB/s wr, 295 op/s 2024-09-16T00:37:29.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:29 smithi060 ceph-mon[35185]: pgmap v960: 129 pgs: 129 active+clean; 615 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.6 MiB/s wr, 295 op/s 2024-09-16T00:37:29.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:29 smithi033 ceph-mon[31243]: pgmap v960: 129 pgs: 129 active+clean; 615 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.6 MiB/s wr, 295 op/s 2024-09-16T00:37:30.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.71 MB/sec execute 276 sec latency 6126.042 ms 2024-09-16T00:37:30.362 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:30 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:37:30.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:30 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:37:30.467 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:30 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:37:31.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.65 MB/sec execute 277 sec latency 7126.071 ms 2024-09-16T00:37:31.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:31 smithi083 ceph-mon[35242]: pgmap v961: 129 pgs: 129 active+clean; 615 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.9 MiB/s wr, 324 op/s 2024-09-16T00:37:31.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:31.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:31 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:31.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:31 smithi060 ceph-mon[35185]: pgmap v961: 129 pgs: 129 active+clean; 615 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.9 MiB/s wr, 324 op/s 2024-09-16T00:37:31.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:31.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:31 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:31.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:31 smithi033 ceph-mon[31243]: pgmap v961: 129 pgs: 129 active+clean; 615 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.9 MiB/s wr, 324 op/s 2024-09-16T00:37:31.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:31.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:31 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:32.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.58 MB/sec execute 278 sec latency 8126.129 ms 2024-09-16T00:37:32.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:32 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:32.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:32 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:32.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:32.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:32 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:32 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:32 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:33.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.52 MB/sec execute 279 sec latency 9126.227 ms 2024-09-16T00:37:33.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:33 smithi033 ceph-mon[31243]: pgmap v962: 129 pgs: 129 active+clean; 610 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 295 op/s 2024-09-16T00:37:33.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:33.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:33.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:37:33.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:37:33.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:33 smithi083 ceph-mon[35242]: pgmap v962: 129 pgs: 129 active+clean; 610 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 295 op/s 2024-09-16T00:37:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:37:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:37:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:33 smithi060 ceph-mon[35185]: pgmap v962: 129 pgs: 129 active+clean; 610 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 295 op/s 2024-09-16T00:37:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:37:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:37:33.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:37:34.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.46 MB/sec execute 280 sec latency 10126.315 ms 2024-09-16T00:37:34.645 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:37:34.916 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:35.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.40 MB/sec execute 281 sec latency 11126.429 ms 2024-09-16T00:37:35.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.395+0000 7fd268391640 1 -- 172.21.15.33:0/2013054478 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd260106a80 msgr2=0x7fd260106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:35.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.395+0000 7fd268391640 1 --2- 172.21.15.33:0/2013054478 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd260106a80 0x7fd260106e60 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009930 tx=0x7fd25c02a420 comp rx=0 tx=0).stop 2024-09-16T00:37:35.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.395+0000 7fd268391640 1 -- 172.21.15.33:0/2013054478 shutdown_connections 2024-09-16T00:37:35.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.396+0000 7fd268391640 1 --2- 172.21.15.33:0/2013054478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd260101420 0x7fd26010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.396+0000 7fd268391640 1 --2- 172.21.15.33:0/2013054478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd260100a80 0x7fd260100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.396+0000 7fd268391640 1 --2- 172.21.15.33:0/2013054478 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd260106a80 0x7fd260106e60 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.396+0000 7fd268391640 1 -- 172.21.15.33:0/2013054478 >> 172.21.15.33:0/2013054478 conn(0x7fd2600fc7a0 msgr2=0x7fd2600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:35.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.396+0000 7fd268391640 1 -- 172.21.15.33:0/2013054478 shutdown_connections 2024-09-16T00:37:35.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.396+0000 7fd268391640 1 -- 172.21.15.33:0/2013054478 wait complete. 2024-09-16T00:37:35.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.396+0000 7fd268391640 1 Processor -- start 2024-09-16T00:37:35.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.396+0000 7fd268391640 1 -- start start 2024-09-16T00:37:35.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd268391640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd260100a80 0x7fd2601973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd268391640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd260101420 0x7fd2601978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd268391640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd260106a80 0x7fd26019bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd268391640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd260110e80 con 0x7fd260101420 2024-09-16T00:37:35.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd268391640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd260110d00 con 0x7fd260100a80 2024-09-16T00:37:35.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd268391640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd260111000 con 0x7fd260106a80 2024-09-16T00:37:35.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd26592c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd260101420 0x7fd2601978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd26592c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd260101420 0x7fd2601978f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36708/0 (socket says 172.21.15.33:36708) 2024-09-16T00:37:35.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd26592c640 1 -- 172.21.15.33:0/4057221979 learned_addr learned my addr 172.21.15.33:0/4057221979 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:35.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd26612d640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd260100a80 0x7fd2601973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd26692e640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd260106a80 0x7fd26019bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd26592c640 1 -- 172.21.15.33:0/4057221979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd260106a80 msgr2=0x7fd26019bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd26592c640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd260106a80 0x7fd26019bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd26592c640 1 -- 172.21.15.33:0/4057221979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd260100a80 msgr2=0x7fd2601973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd26592c640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd260100a80 0x7fd2601973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd26592c640 1 -- 172.21.15.33:0/4057221979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26019c360 con 0x7fd260101420 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd26592c640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd260101420 0x7fd2601978f0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fd25400c970 tx=0x7fd25400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd2537fe640 1 -- 172.21.15.33:0/4057221979 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254007bb0 con 0x7fd260101420 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd2537fe640 1 -- 172.21.15.33:0/4057221979 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd254007d50 con 0x7fd260101420 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd268391640 1 -- 172.21.15.33:0/4057221979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd26019c650 con 0x7fd260101420 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.397+0000 7fd2537fe640 1 -- 172.21.15.33:0/4057221979 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25400f450 con 0x7fd260101420 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.398+0000 7fd268391640 1 -- 172.21.15.33:0/4057221979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2601a3f30 con 0x7fd260101420 2024-09-16T00:37:35.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.400+0000 7fd2537fe640 1 -- 172.21.15.33:0/4057221979 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd254005020 con 0x7fd260101420 2024-09-16T00:37:35.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.400+0000 7fd26a76a740 1 Processor -- start 2024-09-16T00:37:35.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.400+0000 7fd26a76a740 1 -- start start 2024-09-16T00:37:35.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.400+0000 7fd26a76a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d4486b0 0x55fe8d448a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.400+0000 7fd26a76a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d453350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26a76a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 0x55fe8d455c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26a76a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fe8d3ac2e0 con 0x55fe8d453890 2024-09-16T00:37:35.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26a76a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55fe8d353ce0 con 0x55fe8d448fd0 2024-09-16T00:37:35.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26a76a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fe8d3a82a0 con 0x55fe8d4486b0 2024-09-16T00:37:35.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd2537fe640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd24c0a32c0 0x7fd24c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26592c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 0x55fe8d455c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26592c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 0x55fe8d455c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36718/0 (socket says 172.21.15.33:36718) 2024-09-16T00:37:35.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26592c640 1 -- 172.21.15.33:0/919584703 learned_addr learned my addr 172.21.15.33:0/919584703 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26692e640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd24c0a32c0 0x7fd24c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26612d640 1 --2- 172.21.15.33:0/919584703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d453350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26692e640 1 --2- 172.21.15.33:0/919584703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d4486b0 0x55fe8d448a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd2537fe640 1 -- 172.21.15.33:0/4057221979 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd2540cfb50 con 0x7fd260101420 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26592c640 1 -- 172.21.15.33:0/919584703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d4486b0 msgr2=0x55fe8d448a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26592c640 1 --2- 172.21.15.33:0/919584703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d4486b0 0x55fe8d448a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26592c640 1 -- 172.21.15.33:0/919584703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 msgr2=0x55fe8d453350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26592c640 1 --2- 172.21.15.33:0/919584703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d453350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26592c640 1 -- 172.21.15.33:0/919584703 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe8d338440 con 0x55fe8d453890 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26592c640 1 --2- 172.21.15.33:0/919584703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 0x55fe8d455c80 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fd2540058c0 tx=0x7fd25401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9fb80457559708d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.401+0000 7fd26692e640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd24c0a32c0 0x7fd24c0a5780 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd258009fd0 tx=0x7fd258009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.402+0000 7fd250dea640 1 -- 172.21.15.33:0/919584703 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25401d070 con 0x55fe8d453890 2024-09-16T00:37:35.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.402+0000 7fd250dea640 1 -- 172.21.15.33:0/919584703 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd25409c390 con 0x55fe8d453890 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.402+0000 7fd250dea640 1 -- 172.21.15.33:0/919584703 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254098300 con 0x55fe8d453890 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.402+0000 7fd26a76a740 1 -- 172.21.15.33:0/919584703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 msgr2=0x55fe8d455c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.402+0000 7fd26a76a740 1 --2- 172.21.15.33:0/919584703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 0x55fe8d455c80 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fd2540058c0 tx=0x7fd25401b000 comp rx=0 tx=0).stop 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.402+0000 7fd26a76a740 1 -- 172.21.15.33:0/919584703 shutdown_connections 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.402+0000 7fd26a76a740 1 --2- 172.21.15.33:0/919584703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 0x55fe8d455c80 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.402+0000 7fd26a76a740 1 --2- 172.21.15.33:0/919584703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d453350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.402+0000 7fd26a76a740 1 --2- 172.21.15.33:0/919584703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d4486b0 0x55fe8d448a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.402+0000 7fd26a76a740 1 -- 172.21.15.33:0/919584703 >> 172.21.15.33:0/919584703 conn(0x55fe8d444c40 msgr2=0x55fe8d445020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.402+0000 7fd26a76a740 1 -- 172.21.15.33:0/919584703 shutdown_connections 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.402+0000 7fd26a76a740 1 -- 172.21.15.33:0/919584703 wait complete. 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26a76a740 1 -- start start 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26a76a740 1 Processor -- start 2024-09-16T00:37:35.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26a76a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d4486b0 0x55fe8d4531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26a76a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d448fd0 0x55fe8d450480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26a76a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d453890 0x55fe8d4509c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26a76a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fe8d456460 con 0x55fe8d448fd0 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26a76a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55fe8d367050 con 0x55fe8d4486b0 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26a76a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fe8d4565e0 con 0x55fe8d453890 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26a76a740 10 client.0 _ll_register_callbacks cb 0x55fe8d3a7160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26592c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d448fd0 0x55fe8d450480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26592c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d448fd0 0x55fe8d450480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36720/0 (socket says 172.21.15.33:36720) 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26592c640 1 -- 172.21.15.33:0/2662634306 learned_addr learned my addr 172.21.15.33:0/2662634306 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.403+0000 7fd26612d640 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d4486b0 0x55fe8d4531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd26692e640 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d453890 0x55fe8d4509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd26592c640 1 -- 172.21.15.33:0/2662634306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d453890 msgr2=0x55fe8d4509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:35.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd26592c640 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d453890 0x55fe8d4509c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd26592c640 1 -- 172.21.15.33:0/2662634306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d4486b0 msgr2=0x55fe8d4531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd26592c640 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d4486b0 0x55fe8d4531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd26592c640 1 -- 172.21.15.33:0/2662634306 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe8d11d3c0 con 0x55fe8d448fd0 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd26692e640 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d453890 0x55fe8d4509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd26592c640 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d448fd0 0x55fe8d450480 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd254007590 tx=0x7fd2540a00c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6612ddda28fc5ff6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd249ffb640 1 -- 172.21.15.33:0/2662634306 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25401d040 con 0x55fe8d448fd0 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd249ffb640 1 -- 172.21.15.33:0/2662634306 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd25409b0e0 con 0x55fe8d448fd0 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd26a76a740 1 -- 172.21.15.33:0/2662634306 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd25409c390 con 0x55fe8d448fd0 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd249ffb640 1 -- 172.21.15.33:0/2662634306 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd254099f90 con 0x55fe8d448fd0 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd26a76a740 10 client.19239 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd26a76a740 1 -- 172.21.15.33:0/2662634306 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fe8d461e10 con 0x55fe8d448fd0 2024-09-16T00:37:35.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd249ffb640 1 -- 172.21.15.33:0/2662634306 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd25409c390 con 0x55fe8d448fd0 2024-09-16T00:37:35.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.404+0000 7fd249ffb640 1 -- 172.21.15.33:0/2662634306 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fd2540a11d0 con 0x55fe8d448fd0 2024-09-16T00:37:35.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.405+0000 7fd26a76a740 10 client.19239 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:35.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.405+0000 7fd26a76a740 10 client.19239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:37:35.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.405+0000 7fd26a76a740 20 client.19239 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:35.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.405+0000 7fd26a76a740 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fe8d463160 0x55fe8d465550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.405+0000 7fd26a76a740 4 client.19239 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:37:35.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.405+0000 7fd26a76a740 1 -- 172.21.15.33:0/2662634306 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe8d3b9850 con 0x55fe8d463160 2024-09-16T00:37:35.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.405+0000 7fd26612d640 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fe8d463160 0x55fe8d465550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.405+0000 7fd26612d640 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fe8d463160 0x55fe8d465550 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20049d9c1dfd073c server_cookie=5ce705b412b46992 in_seq=0 out_seq=0 2024-09-16T00:37:35.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.405+0000 7fd249ffb640 10 client.19239 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:35.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd249ffb640 1 -- 172.21.15.33:0/2662634306 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fe8d3b9850 con 0x55fe8d463160 2024-09-16T00:37:35.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd249ffb640 10 client.19239 handle_command_reply: tid=0 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd26a76a740 1 client.19239 shutdown 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd26a76a740 1 -- 172.21.15.33:0/2662634306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d448fd0 msgr2=0x55fe8d450480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd26a76a740 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d448fd0 0x55fe8d450480 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd254007590 tx=0x7fd2540a00c0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd26a76a740 1 -- 172.21.15.33:0/2662634306 shutdown_connections 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd26a76a740 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fe8d463160 0x55fe8d465550 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd26a76a740 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d453890 0x55fe8d4509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd26a76a740 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d448fd0 0x55fe8d450480 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd249ffb640 0 client.19239 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd26a76a740 1 --2- 172.21.15.33:0/2662634306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d4486b0 0x55fe8d4531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd26a76a740 1 -- 172.21.15.33:0/2662634306 >> 172.21.15.33:0/2662634306 conn(0x55fe8d444c40 msgr2=0x55fe8d441b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.406+0000 7fd26a76a740 1 -- 172.21.15.33:0/2662634306 shutdown_connections 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.407+0000 7fd26a76a740 1 -- 172.21.15.33:0/2662634306 wait complete. 2024-09-16T00:37:35.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.407+0000 7fd26a76a740 20 client.19239 trim_cache size 0 max 16384 2024-09-16T00:37:35.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26a76a740 1 Processor -- start 2024-09-16T00:37:35.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26a76a740 1 -- start start 2024-09-16T00:37:35.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26a76a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d4486b0 0x55fe8d44d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26a76a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d44dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26a76a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d453890 0x55fe8d44a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26a76a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd2601af740 con 0x55fe8d4486b0 2024-09-16T00:37:35.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26a76a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd2601ad010 con 0x55fe8d448fd0 2024-09-16T00:37:35.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26a76a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd2601aae60 con 0x55fe8d453890 2024-09-16T00:37:35.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26612d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d4486b0 0x55fe8d44d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26612d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d4486b0 0x55fe8d44d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36722/0 (socket says 172.21.15.33:36722) 2024-09-16T00:37:35.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26612d640 1 -- 172.21.15.33:0/298600653 learned_addr learned my addr 172.21.15.33:0/298600653 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:35.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26692e640 1 --2- 172.21.15.33:0/298600653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d453890 0x55fe8d44a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.424+0000 7fd26592c640 1 --2- 172.21.15.33:0/298600653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d44dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.425+0000 7fd26612d640 1 -- 172.21.15.33:0/298600653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d453890 msgr2=0x55fe8d44a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:35.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.425+0000 7fd26612d640 1 --2- 172.21.15.33:0/298600653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d453890 0x55fe8d44a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.425+0000 7fd26612d640 1 -- 172.21.15.33:0/298600653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 msgr2=0x55fe8d44dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:35.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.425+0000 7fd26612d640 1 --2- 172.21.15.33:0/298600653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d44dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.425+0000 7fd26612d640 1 -- 172.21.15.33:0/298600653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe8d450ce0 con 0x55fe8d4486b0 2024-09-16T00:37:35.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.425+0000 7fd26592c640 1 --2- 172.21.15.33:0/298600653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d44dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:37:35.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.425+0000 7fd26612d640 1 --2- 172.21.15.33:0/298600653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d4486b0 0x55fe8d44d6e0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009900 tx=0x7fd25c031980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4c4b69d6a6d576 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:35.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.425+0000 7fd24bfff640 1 -- 172.21.15.33:0/298600653 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25c04d070 con 0x55fe8d4486b0 2024-09-16T00:37:35.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.425+0000 7fd24bfff640 1 -- 172.21.15.33:0/298600653 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd25c005500 con 0x55fe8d4486b0 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.425+0000 7fd24bfff640 1 -- 172.21.15.33:0/298600653 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25c035040 con 0x55fe8d4486b0 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.426+0000 7fd26a76a740 1 -- 172.21.15.33:0/298600653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d4486b0 msgr2=0x55fe8d44d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.426+0000 7fd26a76a740 1 --2- 172.21.15.33:0/298600653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d4486b0 0x55fe8d44d6e0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009900 tx=0x7fd25c031980 comp rx=0 tx=0).stop 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.426+0000 7fd26a76a740 1 -- 172.21.15.33:0/298600653 shutdown_connections 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.426+0000 7fd26a76a740 1 --2- 172.21.15.33:0/298600653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d453890 0x55fe8d44a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.426+0000 7fd26a76a740 1 --2- 172.21.15.33:0/298600653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d44dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.426+0000 7fd26a76a740 1 --2- 172.21.15.33:0/298600653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d4486b0 0x55fe8d44d6e0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.426+0000 7fd26a76a740 1 -- 172.21.15.33:0/298600653 >> 172.21.15.33:0/298600653 conn(0x55fe8d444c40 msgr2=0x55fe8d44bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.426+0000 7fd26a76a740 1 -- 172.21.15.33:0/298600653 shutdown_connections 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.426+0000 7fd26a76a740 1 -- 172.21.15.33:0/298600653 wait complete. 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.426+0000 7fd26a76a740 1 -- start start 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.426+0000 7fd26a76a740 1 Processor -- start 2024-09-16T00:37:35.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26a76a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d4486b0 0x55fe8d468970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26a76a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d468eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26a76a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 0x55fe8d463ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26a76a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd2601ab4d0 con 0x55fe8d453890 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26a76a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd2601ac940 con 0x55fe8d448fd0 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26a76a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd2601abba0 con 0x55fe8d4486b0 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26a76a740 10 client.0 _ll_register_callbacks cb 0x7fd258012610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26692e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 0x55fe8d463ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26692e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 0x55fe8d463ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36726/0 (socket says 172.21.15.33:36726) 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26692e640 1 -- 172.21.15.33:0/3902545625 learned_addr learned my addr 172.21.15.33:0/3902545625 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26612d640 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d4486b0 0x55fe8d468970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26592c640 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d468eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26692e640 1 -- 172.21.15.33:0/3902545625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d4486b0 msgr2=0x55fe8d468970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:35.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26692e640 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d4486b0 0x55fe8d468970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26692e640 1 -- 172.21.15.33:0/3902545625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 msgr2=0x55fe8d468eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:35.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26692e640 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d468eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26692e640 1 -- 172.21.15.33:0/3902545625 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe8d464720 con 0x55fe8d453890 2024-09-16T00:37:35.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26692e640 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 0x55fe8d463ef0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fd258010c20 tx=0x7fd258004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb46c32bd856bc02 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:35.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:35.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd249ffb640 1 -- 172.21.15.33:0/3902545625 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25801edc0 con 0x55fe8d453890 2024-09-16T00:37:35.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd249ffb640 1 -- 172.21.15.33:0/3902545625 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd25801b070 con 0x55fe8d453890 2024-09-16T00:37:35.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.427+0000 7fd26a76a740 1 -- 172.21.15.33:0/3902545625 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd25c005500 con 0x55fe8d453890 2024-09-16T00:37:35.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.428+0000 7fd249ffb640 1 -- 172.21.15.33:0/3902545625 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd258004b20 con 0x55fe8d453890 2024-09-16T00:37:35.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.428+0000 7fd26a76a740 10 client.19245 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:35.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.428+0000 7fd26a76a740 1 -- 172.21.15.33:0/3902545625 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fe8d4cfb50 con 0x55fe8d453890 2024-09-16T00:37:35.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.428+0000 7fd249ffb640 1 -- 172.21.15.33:0/3902545625 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd25c005500 con 0x55fe8d453890 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.428+0000 7fd249ffb640 1 -- 172.21.15.33:0/3902545625 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fd258029400 con 0x55fe8d453890 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.428+0000 7fd26a76a740 10 client.19245 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.428+0000 7fd26a76a740 10 client.19245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.428+0000 7fd26a76a740 20 client.19245 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.428+0000 7fd26a76a740 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fe8d4d0320 0x55fe8d4d2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.428+0000 7fd26a76a740 4 client.19245 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.428+0000 7fd26a76a740 1 -- 172.21.15.33:0/3902545625 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fe8d3b3d60 con 0x55fe8d4d0320 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.428+0000 7fd26612d640 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fe8d4d0320 0x55fe8d4d2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26612d640 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fe8d4d0320 0x55fe8d4d2740 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dd33141f141c5cb server_cookie=35289b9b256012e in_seq=0 out_seq=0 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd249ffb640 10 client.19245 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd249ffb640 1 -- 172.21.15.33:0/3902545625 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fe8d3b3d60 con 0x55fe8d4d0320 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd249ffb640 10 client.19245 handle_command_reply: tid=0 2024-09-16T00:37:35.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26a76a740 1 client.19245 shutdown 2024-09-16T00:37:35.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26a76a740 1 -- 172.21.15.33:0/3902545625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 msgr2=0x55fe8d463ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:35.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26a76a740 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 0x55fe8d463ef0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fd258010c20 tx=0x7fd258004ab0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26a76a740 1 -- 172.21.15.33:0/3902545625 shutdown_connections 2024-09-16T00:37:35.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26a76a740 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55fe8d4d0320 0x55fe8d4d2740 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26a76a740 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe8d453890 0x55fe8d463ef0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26a76a740 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe8d448fd0 0x55fe8d468eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd249ffb640 0 client.19245 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:35.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26a76a740 1 --2- 172.21.15.33:0/3902545625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe8d4486b0 0x55fe8d468970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26a76a740 1 -- 172.21.15.33:0/3902545625 >> 172.21.15.33:0/3902545625 conn(0x55fe8d444c40 msgr2=0x55fe8d44a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:35.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26a76a740 1 -- 172.21.15.33:0/3902545625 shutdown_connections 2024-09-16T00:37:35.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26a76a740 1 -- 172.21.15.33:0/3902545625 wait complete. 2024-09-16T00:37:35.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.429+0000 7fd26a76a740 20 client.19245 trim_cache size 0 max 16384 2024-09-16T00:37:35.441 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:37:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:37:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:37:35.441 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:37:35.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.430+0000 7fd268391640 1 -- 172.21.15.33:0/4057221979 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd24c0a32c0 msgr2=0x7fd24c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:35.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.430+0000 7fd268391640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd24c0a32c0 0x7fd24c0a5780 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd258009fd0 tx=0x7fd258009500 comp rx=0 tx=0).stop 2024-09-16T00:37:35.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.430+0000 7fd268391640 1 -- 172.21.15.33:0/4057221979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd260101420 msgr2=0x7fd2601978f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:35.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.430+0000 7fd268391640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd260101420 0x7fd2601978f0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fd25400c970 tx=0x7fd25400ce30 comp rx=0 tx=0).stop 2024-09-16T00:37:35.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.430+0000 7fd268391640 1 -- 172.21.15.33:0/4057221979 shutdown_connections 2024-09-16T00:37:35.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.430+0000 7fd268391640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd24c0a32c0 0x7fd24c0a5780 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.430+0000 7fd268391640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd260106a80 0x7fd26019bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.430+0000 7fd268391640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd260101420 0x7fd2601978f0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.430+0000 7fd268391640 1 --2- 172.21.15.33:0/4057221979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd260100a80 0x7fd2601973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:35.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.430+0000 7fd268391640 1 -- 172.21.15.33:0/4057221979 >> 172.21.15.33:0/4057221979 conn(0x7fd2600fc7a0 msgr2=0x7fd2600fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:35.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.430+0000 7fd268391640 1 -- 172.21.15.33:0/4057221979 shutdown_connections 2024-09-16T00:37:35.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:35.430+0000 7fd268391640 1 -- 172.21.15.33:0/4057221979 wait complete. 2024-09-16T00:37:35.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:35 smithi033 ceph-mon[31243]: pgmap v963: 129 pgs: 129 active+clean; 610 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.0 MiB/s wr, 255 op/s 2024-09-16T00:37:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:35 smithi083 ceph-mon[35242]: pgmap v963: 129 pgs: 129 active+clean; 610 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.0 MiB/s wr, 255 op/s 2024-09-16T00:37:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:35 smithi060 ceph-mon[35185]: pgmap v963: 129 pgs: 129 active+clean; 610 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.0 MiB/s wr, 255 op/s 2024-09-16T00:37:36.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.33 MB/sec execute 282 sec latency 12126.542 ms 2024-09-16T00:37:36.291 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:37:36.291 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:37:36.555 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:37.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.044+0000 7fae53f03640 1 -- 172.21.15.33:0/2189097608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae4c106f10 msgr2=0x7fae4c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:37.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.044+0000 7fae53f03640 1 --2- 172.21.15.33:0/2189097608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae4c106f10 0x7fae4c10ea90 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009900 tx=0x7fae3c02a410 comp rx=0 tx=0).stop 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.044+0000 7fae53f03640 1 -- 172.21.15.33:0/2189097608 shutdown_connections 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.044+0000 7fae53f03640 1 --2- 172.21.15.33:0/2189097608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae4c106f10 0x7fae4c10ea90 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.044+0000 7fae53f03640 1 --2- 172.21.15.33:0/2189097608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae4c102790 0x7fae4c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.044+0000 7fae53f03640 1 --2- 172.21.15.33:0/2189097608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae4c101de0 0x7fae4c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.044+0000 7fae53f03640 1 -- 172.21.15.33:0/2189097608 >> 172.21.15.33:0/2189097608 conn(0x7fae4c0fd8c0 msgr2=0x7fae4c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.044+0000 7fae53f03640 1 -- 172.21.15.33:0/2189097608 shutdown_connections 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.044+0000 7fae53f03640 1 -- 172.21.15.33:0/2189097608 wait complete. 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.045+0000 7fae53f03640 1 Processor -- start 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.045+0000 7fae53f03640 1 -- start start 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.045+0000 7fae53f03640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae4c101de0 0x7fae4c198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.045+0000 7fae53f03640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae4c102790 0x7fae4c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.045+0000 7fae53f03640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae4c106f10 0x7fae4c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.045+0000 7fae53f03640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c111f70 con 0x7fae4c101de0 2024-09-16T00:37:37.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.045+0000 7fae53f03640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c111df0 con 0x7fae4c106f10 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.045+0000 7fae53f03640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1120f0 con 0x7fae4c102790 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.045+0000 7fae51c9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae4c101de0 0x7fae4c198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.045+0000 7fae51c9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae4c101de0 0x7fae4c198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36748/0 (socket says 172.21.15.33:36748) 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.045+0000 7fae51c9f640 1 -- 172.21.15.33:0/2532544960 learned_addr learned my addr 172.21.15.33:0/2532544960 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.045+0000 7fae5149e640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae4c102790 0x7fae4c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.046+0000 7fae524a0640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae4c106f10 0x7fae4c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.046+0000 7fae51c9f640 1 -- 172.21.15.33:0/2532544960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae4c102790 msgr2=0x7fae4c198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.046+0000 7fae51c9f640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae4c102790 0x7fae4c198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.046+0000 7fae51c9f640 1 -- 172.21.15.33:0/2532544960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae4c106f10 msgr2=0x7fae4c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.046+0000 7fae51c9f640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae4c106f10 0x7fae4c19cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.046+0000 7fae51c9f640 1 -- 172.21.15.33:0/2532544960 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae4c19d2e0 con 0x7fae4c101de0 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.046+0000 7fae51c9f640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae4c101de0 0x7fae4c198330 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fae4400cce0 tx=0x7fae4400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:37.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.046+0000 7fae4affd640 1 -- 172.21.15.33:0/2532544960 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4400f650 con 0x7fae4c101de0 2024-09-16T00:37:37.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.046+0000 7fae4affd640 1 -- 172.21.15.33:0/2532544960 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae4400fcd0 con 0x7fae4c101de0 2024-09-16T00:37:37.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.046+0000 7fae53f03640 1 -- 172.21.15.33:0/2532544960 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae4c19d5d0 con 0x7fae4c101de0 2024-09-16T00:37:37.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.046+0000 7fae4affd640 1 -- 172.21.15.33:0/2532544960 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae44018c70 con 0x7fae4c101de0 2024-09-16T00:37:37.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.046+0000 7fae53f03640 1 -- 172.21.15.33:0/2532544960 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae4c1a4eb0 con 0x7fae4c101de0 2024-09-16T00:37:37.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.047+0000 7fae53f03640 1 -- 172.21.15.33:0/2532544960 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae4c1021c0 con 0x7fae4c101de0 2024-09-16T00:37:37.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.048+0000 7fae4affd640 1 -- 172.21.15.33:0/2532544960 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fae4400f7f0 con 0x7fae4c101de0 2024-09-16T00:37:37.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.049+0000 7fae4affd640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae340a3240 0x7fae340a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:37.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.049+0000 7fae5149e640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae340a3240 0x7fae340a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:37.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.049+0000 7fae4affd640 1 -- 172.21.15.33:0/2532544960 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fae440d0290 con 0x7fae4c101de0 2024-09-16T00:37:37.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.049+0000 7fae5149e640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae340a3240 0x7fae340a5700 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fae4c108910 tx=0x7fae4000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:37.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.053+0000 7fae4affd640 1 -- 172.21.15.33:0/2532544960 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fae44014030 con 0x7fae4c101de0 2024-09-16T00:37:37.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.27 MB/sec execute 283 sec latency 13126.606 ms 2024-09-16T00:37:37.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.269+0000 7fae53f03640 1 -- 172.21.15.33:0/2532544960 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae4c199690 con 0x7fae4c101de0 2024-09-16T00:37:37.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.270+0000 7fae4affd640 1 -- 172.21.15.33:0/2532544960 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7fae4409cad0 con 0x7fae4c101de0 2024-09-16T00:37:37.271 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:37:37.271 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:37:37.271 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:37:37.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.273+0000 7fae53f03640 1 -- 172.21.15.33:0/2532544960 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae340a3240 msgr2=0x7fae340a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:37.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.273+0000 7fae53f03640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae340a3240 0x7fae340a5700 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fae4c108910 tx=0x7fae4000b390 comp rx=0 tx=0).stop 2024-09-16T00:37:37.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.273+0000 7fae53f03640 1 -- 172.21.15.33:0/2532544960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae4c101de0 msgr2=0x7fae4c198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:37.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.273+0000 7fae53f03640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae4c101de0 0x7fae4c198330 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fae4400cce0 tx=0x7fae4400a620 comp rx=0 tx=0).stop 2024-09-16T00:37:37.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.273+0000 7fae53f03640 1 -- 172.21.15.33:0/2532544960 shutdown_connections 2024-09-16T00:37:37.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.273+0000 7fae53f03640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fae340a3240 0x7fae340a5700 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:37.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.273+0000 7fae53f03640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae4c106f10 0x7fae4c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:37.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.273+0000 7fae53f03640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae4c102790 0x7fae4c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:37.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.273+0000 7fae53f03640 1 --2- 172.21.15.33:0/2532544960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae4c101de0 0x7fae4c198330 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:37.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.273+0000 7fae53f03640 1 -- 172.21.15.33:0/2532544960 >> 172.21.15.33:0/2532544960 conn(0x7fae4c0fd8c0 msgr2=0x7fae4c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:37.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.273+0000 7fae53f03640 1 -- 172.21.15.33:0/2532544960 shutdown_connections 2024-09-16T00:37:37.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:37.273+0000 7fae53f03640 1 -- 172.21.15.33:0/2532544960 wait complete. 2024-09-16T00:37:37.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:37 smithi033 ceph-mon[31243]: pgmap v964: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-09-16T00:37:37.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:37 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2532544960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:37:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:37 smithi083 ceph-mon[35242]: pgmap v964: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-09-16T00:37:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:37 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2532544960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:37:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:37 smithi060 ceph-mon[35185]: pgmap v964: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-09-16T00:37:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:37 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2532544960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:37:37.987 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:37:38.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.21 MB/sec execute 284 sec latency 14126.657 ms 2024-09-16T00:37:38.246 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.738+0000 7fb40e21f640 1 -- 172.21.15.33:0/289906419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb408102d40 msgr2=0x7fb408103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.738+0000 7fb40e21f640 1 --2- 172.21.15.33:0/289906419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb408102d40 0x7fb408103120 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009900 tx=0x7fb3f802a480 comp rx=0 tx=0).stop 2024-09-16T00:37:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.739+0000 7fb40e21f640 1 -- 172.21.15.33:0/289906419 shutdown_connections 2024-09-16T00:37:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.739+0000 7fb40e21f640 1 --2- 172.21.15.33:0/289906419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb40810c170 0x7fb40810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.739+0000 7fb40e21f640 1 --2- 172.21.15.33:0/289906419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb408103660 0x7fb40810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.739+0000 7fb40e21f640 1 --2- 172.21.15.33:0/289906419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb408102d40 0x7fb408103120 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.739+0000 7fb40e21f640 1 -- 172.21.15.33:0/289906419 >> 172.21.15.33:0/289906419 conn(0x7fb4080fc7a0 msgr2=0x7fb4080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.739+0000 7fb40e21f640 1 -- 172.21.15.33:0/289906419 shutdown_connections 2024-09-16T00:37:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.739+0000 7fb40e21f640 1 -- 172.21.15.33:0/289906419 wait complete. 2024-09-16T00:37:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.739+0000 7fb40e21f640 1 Processor -- start 2024-09-16T00:37:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.739+0000 7fb40e21f640 1 -- start start 2024-09-16T00:37:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb40e21f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb408102d40 0x7fb4081984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb40e21f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb408103660 0x7fb4081989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb40e21f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb40810c170 0x7fb40819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb40e21f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb408112000 con 0x7fb408103660 2024-09-16T00:37:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb40e21f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb408111e80 con 0x7fb40810c170 2024-09-16T00:37:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb40e21f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb408112180 con 0x7fb408102d40 2024-09-16T00:37:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb406ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb408103660 0x7fb4081989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb4077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb408102d40 0x7fb4081984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb406ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb408103660 0x7fb4081989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36766/0 (socket says 172.21.15.33:36766) 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb406ffd640 1 -- 172.21.15.33:0/3390554733 learned_addr learned my addr 172.21.15.33:0/3390554733 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb407fff640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb40810c170 0x7fb40819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb406ffd640 1 -- 172.21.15.33:0/3390554733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb408102d40 msgr2=0x7fb4081984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb406ffd640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb408102d40 0x7fb4081984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb406ffd640 1 -- 172.21.15.33:0/3390554733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb40810c170 msgr2=0x7fb40819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb406ffd640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb40810c170 0x7fb40819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb406ffd640 1 -- 172.21.15.33:0/3390554733 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40819d460 con 0x7fb408103660 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb406ffd640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb408103660 0x7fb4081989f0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00b3f0 tx=0x7fb3fc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb404ff9640 1 -- 172.21.15.33:0/3390554733 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc015020 con 0x7fb408103660 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb40e21f640 1 -- 172.21.15.33:0/3390554733 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40819d750 con 0x7fb408103660 2024-09-16T00:37:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb404ff9640 1 -- 172.21.15.33:0/3390554733 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3fc011d10 con 0x7fb408103660 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb404ff9640 1 -- 172.21.15.33:0/3390554733 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc00f2e0 con 0x7fb408103660 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.740+0000 7fb40e21f640 1 -- 172.21.15.33:0/3390554733 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4081a5030 con 0x7fb408103660 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb404ff9640 1 -- 172.21.15.33:0/3390554733 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb3fc018020 con 0x7fb408103660 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb4105f8740 1 Processor -- start 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb4105f8740 1 -- start start 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb4105f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e11e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb4105f8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e11e5d0 0x55c52e11ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb4105f8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e12d780 0x55c52e12fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb4105f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c52e0842f0 con 0x55c52e11fb40 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb4105f8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c52e02bcf0 con 0x55c52e12d780 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb4105f8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c52e0802b0 con 0x55c52e11e5d0 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb407fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e11e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb404ff9640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3f00a3240 0x7fb3f00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb4077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e11e5d0 0x55c52e11ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb407fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e11e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36780/0 (socket says 172.21.15.33:36780) 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.743+0000 7fb407fff640 1 -- 172.21.15.33:0/1056963330 learned_addr learned my addr 172.21.15.33:0/1056963330 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb404ff9640 1 -- 172.21.15.33:0/3390554733 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb3fc0cf5e0 con 0x7fb408103660 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb407fff640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3f00a3240 0x7fb3f00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb406ffd640 1 --2- 172.21.15.33:0/1056963330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e12d780 0x55c52e12fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb4077fe640 1 -- 172.21.15.33:0/1056963330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e12d780 msgr2=0x55c52e12fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb4077fe640 1 --2- 172.21.15.33:0/1056963330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e12d780 0x55c52e12fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb4077fe640 1 -- 172.21.15.33:0/1056963330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 msgr2=0x55c52e11e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb4077fe640 1 --2- 172.21.15.33:0/1056963330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e11e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb4077fe640 1 -- 172.21.15.33:0/1056963330 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c52e0103a0 con 0x55c52e11e5d0 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb406ffd640 1 --2- 172.21.15.33:0/1056963330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e12d780 0x55c52e12fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:37:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb4077fe640 1 --2- 172.21.15.33:0/1056963330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e11e5d0 0x55c52e11ea10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb3f802aa70 tx=0x7fb3f8004180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2da29dd767fcae3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb3edffb640 1 -- 172.21.15.33:0/1056963330 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8040070 con 0x55c52e11e5d0 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb407fff640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3f00a3240 0x7fb3f00a5700 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb4000127f0 tx=0x7fb400004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb3edffb640 1 -- 172.21.15.33:0/1056963330 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f8005500 con 0x55c52e11e5d0 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.744+0000 7fb3edffb640 1 -- 172.21.15.33:0/1056963330 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8035040 con 0x55c52e11e5d0 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.745+0000 7fb4105f8740 1 -- 172.21.15.33:0/1056963330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e11e5d0 msgr2=0x55c52e11ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.745+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1056963330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e11e5d0 0x55c52e11ea10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb3f802aa70 tx=0x7fb3f8004180 comp rx=0 tx=0).stop 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.745+0000 7fb4105f8740 1 -- 172.21.15.33:0/1056963330 shutdown_connections 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.745+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1056963330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e12d780 0x55c52e12fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.745+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1056963330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e11e5d0 0x55c52e11ea10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.745+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1056963330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e11e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.745+0000 7fb4105f8740 1 -- 172.21.15.33:0/1056963330 >> 172.21.15.33:0/1056963330 conn(0x55c52e0e7310 msgr2=0x55c52e11ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.745+0000 7fb4105f8740 1 -- 172.21.15.33:0/1056963330 shutdown_connections 2024-09-16T00:37:38.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.745+0000 7fb4105f8740 1 -- 172.21.15.33:0/1056963330 wait complete. 2024-09-16T00:37:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.745+0000 7fb4105f8740 1 -- start start 2024-09-16T00:37:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb4105f8740 1 Processor -- start 2024-09-16T00:37:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb4105f8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11e5d0 0x55c52e12d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb4105f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e12a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb4105f8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e12d780 0x55c52e12a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb4105f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c52e130420 con 0x55c52e11fb40 2024-09-16T00:37:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb4105f8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c52e03f060 con 0x55c52e11e5d0 2024-09-16T00:37:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb4105f8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c52e1305a0 con 0x55c52e12d780 2024-09-16T00:37:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb4105f8740 10 client.0 _ll_register_callbacks cb 0x55c52dd928c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb406ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e12a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb406ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e12a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36786/0 (socket says 172.21.15.33:36786) 2024-09-16T00:37:38.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb406ffd640 1 -- 172.21.15.33:0/2006846497 learned_addr learned my addr 172.21.15.33:0/2006846497 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb407fff640 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e12d780 0x55c52e12a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb4077fe640 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11e5d0 0x55c52e12d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb406ffd640 1 -- 172.21.15.33:0/2006846497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e12d780 msgr2=0x55c52e12a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb406ffd640 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e12d780 0x55c52e12a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb406ffd640 1 -- 172.21.15.33:0/2006846497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11e5d0 msgr2=0x55c52e12d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb406ffd640 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11e5d0 0x55c52e12d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.746+0000 7fb406ffd640 1 -- 172.21.15.33:0/2006846497 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c52ddf53c0 con 0x55c52e11fb40 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb406ffd640 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e12a400 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0ccfd0 tx=0x7fb3fc010eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5066d5b05d146f97 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb3e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb3e6ffd640 1 -- 172.21.15.33:0/2006846497 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc0b4060 con 0x55c52e11fb40 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb3e6ffd640 1 -- 172.21.15.33:0/2006846497 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb3fc010510 con 0x55c52e11fb40 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb4105f8740 1 -- 172.21.15.33:0/2006846497 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3f8005500 con 0x55c52e11fb40 2024-09-16T00:37:38.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb3e6ffd640 1 -- 172.21.15.33:0/2006846497 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb3fc0ba170 con 0x55c52e11fb40 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb4105f8740 10 client.19263 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb4105f8740 1 -- 172.21.15.33:0/2006846497 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c52e13be60 con 0x55c52e11fb40 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb3e6ffd640 1 -- 172.21.15.33:0/2006846497 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f8005500 con 0x55c52e11fb40 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb3e6ffd640 1 -- 172.21.15.33:0/2006846497 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb3fc097040 con 0x55c52e11fb40 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb4105f8740 10 client.19263 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb4105f8740 10 client.19263 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb4105f8740 20 client.19263 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb4105f8740 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c52e13d270 0x55c52e13f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb4105f8740 4 client.19263 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.747+0000 7fb4105f8740 1 -- 172.21.15.33:0/2006846497 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c52e091860 con 0x55c52e13d270 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.748+0000 7fb4077fe640 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c52e13d270 0x55c52e13f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.748+0000 7fb3e6ffd640 10 client.19263 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:37:38.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.748+0000 7fb4077fe640 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c52e13d270 0x55c52e13f6a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b336696ba5af4097 server_cookie=335c3c760b9a9417 in_seq=0 out_seq=0 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb3e6ffd640 1 -- 172.21.15.33:0/2006846497 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c52e091860 con 0x55c52e13d270 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb3e6ffd640 10 client.19263 handle_command_reply: tid=0 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb4105f8740 1 client.19263 shutdown 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb4105f8740 1 -- 172.21.15.33:0/2006846497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 msgr2=0x55c52e12a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb4105f8740 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e12a400 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0ccfd0 tx=0x7fb3fc010eb0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb4105f8740 1 -- 172.21.15.33:0/2006846497 shutdown_connections 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb4105f8740 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c52e13d270 0x55c52e13f6a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb4105f8740 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e12d780 0x55c52e12a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb4105f8740 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e12a400 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb3e6ffd640 0 client.19263 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb4105f8740 1 --2- 172.21.15.33:0/2006846497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11e5d0 0x55c52e12d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb4105f8740 1 -- 172.21.15.33:0/2006846497 >> 172.21.15.33:0/2006846497 conn(0x55c52e0e7310 msgr2=0x55c52e120960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb4105f8740 1 -- 172.21.15.33:0/2006846497 shutdown_connections 2024-09-16T00:37:38.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb4105f8740 1 -- 172.21.15.33:0/2006846497 wait complete. 2024-09-16T00:37:38.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.749+0000 7fb4105f8740 20 client.19263 trim_cache size 0 max 16384 2024-09-16T00:37:38.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.762+0000 7fb4105f8740 1 Processor -- start 2024-09-16T00:37:38.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.762+0000 7fb4105f8740 1 -- start start 2024-09-16T00:37:38.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.763+0000 7fb4105f8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11e5d0 0x55c52e1396c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.763+0000 7fb4105f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e129cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.763+0000 7fb4105f8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e12d780 0x55c52e129420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.763+0000 7fb4105f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081af770 con 0x55c52e11fb40 2024-09-16T00:37:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.763+0000 7fb4105f8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081ad040 con 0x55c52e11e5d0 2024-09-16T00:37:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.763+0000 7fb4105f8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081aae90 con 0x55c52e12d780 2024-09-16T00:37:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.763+0000 7fb407fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e12d780 0x55c52e129420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.763+0000 7fb407fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e12d780 0x55c52e129420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:58916/0 (socket says 172.21.15.33:58916) 2024-09-16T00:37:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.763+0000 7fb407fff640 1 -- 172.21.15.33:0/1395864889 learned_addr learned my addr 172.21.15.33:0/1395864889 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.763+0000 7fb406ffd640 1 --2- 172.21.15.33:0/1395864889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e129cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.763+0000 7fb4077fe640 1 --2- 172.21.15.33:0/1395864889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11e5d0 0x55c52e1396c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.764+0000 7fb407fff640 1 -- 172.21.15.33:0/1395864889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11e5d0 msgr2=0x55c52e1396c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.764+0000 7fb407fff640 1 --2- 172.21.15.33:0/1395864889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11e5d0 0x55c52e1396c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.764+0000 7fb407fff640 1 -- 172.21.15.33:0/1395864889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 msgr2=0x55c52e129cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.764+0000 7fb407fff640 1 --2- 172.21.15.33:0/1395864889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e129cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.764+0000 7fb407fff640 1 -- 172.21.15.33:0/1395864889 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c52e11f300 con 0x55c52e12d780 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.764+0000 7fb406ffd640 1 --2- 172.21.15.33:0/1395864889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e129cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.764+0000 7fb4077fe640 1 --2- 172.21.15.33:0/1395864889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11e5d0 0x55c52e1396c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.764+0000 7fb407fff640 1 --2- 172.21.15.33:0/1395864889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e12d780 0x55c52e129420 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb400012690 tx=0x7fb4000087b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73ae5ca129cee871 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.764+0000 7fb3ed7fa640 1 -- 172.21.15.33:0/1395864889 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400018040 con 0x55c52e12d780 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.764+0000 7fb3ed7fa640 1 -- 172.21.15.33:0/1395864889 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb400025030 con 0x55c52e12d780 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.764+0000 7fb3ed7fa640 1 -- 172.21.15.33:0/1395864889 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400021d60 con 0x55c52e12d780 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.765+0000 7fb4105f8740 1 -- 172.21.15.33:0/1395864889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e12d780 msgr2=0x55c52e129420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.765+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1395864889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e12d780 0x55c52e129420 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb400012690 tx=0x7fb4000087b0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.765+0000 7fb4105f8740 1 -- 172.21.15.33:0/1395864889 shutdown_connections 2024-09-16T00:37:38.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.765+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1395864889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e12d780 0x55c52e129420 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.765+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1395864889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e11fb40 0x55c52e129cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.765+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1395864889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11e5d0 0x55c52e1396c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.765+0000 7fb4105f8740 1 -- 172.21.15.33:0/1395864889 >> 172.21.15.33:0/1395864889 conn(0x55c52e0e7310 msgr2=0x55c52e128d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.765+0000 7fb4105f8740 1 -- 172.21.15.33:0/1395864889 shutdown_connections 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.765+0000 7fb4105f8740 1 -- 172.21.15.33:0/1395864889 wait complete. 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.765+0000 7fb4105f8740 1 -- start start 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb4105f8740 1 Processor -- start 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb4105f8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e11e5d0 0x55c52e126d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb4105f8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11fb40 0x55c52e1272b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb4105f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e12d780 0x55c52e134820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb4105f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081ab500 con 0x55c52e12d780 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb4105f8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081ac970 con 0x55c52e11fb40 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb4105f8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081abbd0 con 0x55c52e11e5d0 2024-09-16T00:37:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb4105f8740 10 client.0 _ll_register_callbacks cb 0x7fb3fc097360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb407fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e12d780 0x55c52e134820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb407fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e12d780 0x55c52e134820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36812/0 (socket says 172.21.15.33:36812) 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb407fff640 1 -- 172.21.15.33:0/1417226163 learned_addr learned my addr 172.21.15.33:0/1417226163 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb4077fe640 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e11e5d0 0x55c52e126d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb406ffd640 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11fb40 0x55c52e1272b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb407fff640 1 -- 172.21.15.33:0/1417226163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e11e5d0 msgr2=0x55c52e126d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb407fff640 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e11e5d0 0x55c52e126d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb407fff640 1 -- 172.21.15.33:0/1417226163 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11fb40 msgr2=0x55c52e1272b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb407fff640 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11fb40 0x55c52e1272b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.766+0000 7fb407fff640 1 -- 172.21.15.33:0/1417226163 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c52e134d60 con 0x55c52e12d780 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb407fff640 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e12d780 0x55c52e134820 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fb4000107a0 tx=0x7fb400021ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244f0e5ef3356d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb3e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb3e6ffd640 1 -- 172.21.15.33:0/1417226163 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400038060 con 0x55c52e12d780 2024-09-16T00:37:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb3e6ffd640 1 -- 172.21.15.33:0/1417226163 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb40000f7d0 con 0x55c52e12d780 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb4105f8740 1 -- 172.21.15.33:0/1417226163 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb400025030 con 0x55c52e12d780 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb3e6ffd640 1 -- 172.21.15.33:0/1417226163 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb40003f050 con 0x55c52e12d780 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb4105f8740 10 client.19269 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb4105f8740 1 -- 172.21.15.33:0/1417226163 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c52e135f70 con 0x55c52e12d780 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb3e6ffd640 1 -- 172.21.15.33:0/1417226163 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb400025030 con 0x55c52e12d780 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb3e6ffd640 1 -- 172.21.15.33:0/1417226163 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb40002a420 con 0x55c52e12d780 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb4105f8740 10 client.19269 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb4105f8740 10 client.19269 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb4105f8740 20 client.19269 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c52e1a3f20 0x55c52e1a6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb4105f8740 4 client.19269 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.767+0000 7fb4105f8740 1 -- 172.21.15.33:0/1417226163 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c52e08bd70 con 0x55c52e1a3f20 2024-09-16T00:37:38.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb4077fe640 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c52e1a3f20 0x55c52e1a6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb4077fe640 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c52e1a3f20 0x55c52e1a6340 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7918444b6bf824c8 server_cookie=1af157194958ab0 in_seq=0 out_seq=0 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb3e6ffd640 10 client.19269 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb3e6ffd640 1 -- 172.21.15.33:0/1417226163 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c52e08bd70 con 0x55c52e1a3f20 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb3e6ffd640 10 client.19269 handle_command_reply: tid=0 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb4105f8740 1 client.19269 shutdown 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb4105f8740 1 -- 172.21.15.33:0/1417226163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e12d780 msgr2=0x55c52e134820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e12d780 0x55c52e134820 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fb4000107a0 tx=0x7fb400021ec0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb4105f8740 1 -- 172.21.15.33:0/1417226163 shutdown_connections 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55c52e1a3f20 0x55c52e1a6340 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c52e12d780 0x55c52e134820 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c52e11fb40 0x55c52e1272b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb4105f8740 1 --2- 172.21.15.33:0/1417226163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c52e11e5d0 0x55c52e126d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.768+0000 7fb4105f8740 1 -- 172.21.15.33:0/1417226163 >> 172.21.15.33:0/1417226163 conn(0x55c52e0e7310 msgr2=0x55c52e129530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:38.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb3e6ffd640 0 client.19269 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:37:38.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb4105f8740 1 -- 172.21.15.33:0/1417226163 shutdown_connections 2024-09-16T00:37:38.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb4105f8740 1 -- 172.21.15.33:0/1417226163 wait complete. 2024-09-16T00:37:38.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb4105f8740 20 client.19269 trim_cache size 0 max 16384 2024-09-16T00:37:38.783 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:37:38.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb40e21f640 1 -- 172.21.15.33:0/3390554733 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3f00a3240 msgr2=0x7fb3f00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:38.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb40e21f640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3f00a3240 0x7fb3f00a5700 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb4000127f0 tx=0x7fb400004ab0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb40e21f640 1 -- 172.21.15.33:0/3390554733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb408103660 msgr2=0x7fb4081989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:38.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb40e21f640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb408103660 0x7fb4081989f0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00b3f0 tx=0x7fb3fc00b8b0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb40e21f640 1 -- 172.21.15.33:0/3390554733 shutdown_connections 2024-09-16T00:37:38.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb40e21f640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb3f00a3240 0x7fb3f00a5700 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb40e21f640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb40810c170 0x7fb40819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb40e21f640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb408103660 0x7fb4081989f0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb40e21f640 1 --2- 172.21.15.33:0/3390554733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb408102d40 0x7fb4081984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:38.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb40e21f640 1 -- 172.21.15.33:0/3390554733 >> 172.21.15.33:0/3390554733 conn(0x7fb4080fc7a0 msgr2=0x7fb4080fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:38.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.769+0000 7fb40e21f640 1 -- 172.21.15.33:0/3390554733 shutdown_connections 2024-09-16T00:37:38.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:38.770+0000 7fb40e21f640 1 -- 172.21.15.33:0/3390554733 wait complete. 2024-09-16T00:37:39.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.15 MB/sec execute 285 sec latency 15126.740 ms 2024-09-16T00:37:39.638 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:37:39.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:39 smithi033 ceph-mon[31243]: pgmap v965: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 398 KiB/s wr, 45 op/s 2024-09-16T00:37:39.902 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:39 smithi083 ceph-mon[35242]: pgmap v965: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 398 KiB/s wr, 45 op/s 2024-09-16T00:37:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:39 smithi060 ceph-mon[35185]: pgmap v965: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 398 KiB/s wr, 45 op/s 2024-09-16T00:37:40.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.09 MB/sec execute 286 sec latency 16126.785 ms 2024-09-16T00:37:40.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.787+0000 7f42bff3a640 1 -- 172.21.15.33:0/2375303268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42b8102790 msgr2=0x7f42b81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:40.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.787+0000 7f42bff3a640 1 --2- 172.21.15.33:0/2375303268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42b8102790 0x7f42b81067e0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f42a8009900 tx=0x7f42a802a410 comp rx=0 tx=0).stop 2024-09-16T00:37:40.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.787+0000 7f42bff3a640 1 -- 172.21.15.33:0/2375303268 shutdown_connections 2024-09-16T00:37:40.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.787+0000 7f42bff3a640 1 --2- 172.21.15.33:0/2375303268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42b8106f10 0x7f42b810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.787+0000 7f42bff3a640 1 --2- 172.21.15.33:0/2375303268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42b8102790 0x7f42b81067e0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.787+0000 7f42bff3a640 1 --2- 172.21.15.33:0/2375303268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42b8101de0 0x7f42b81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.787+0000 7f42bff3a640 1 -- 172.21.15.33:0/2375303268 >> 172.21.15.33:0/2375303268 conn(0x7f42b80fd8c0 msgr2=0x7f42b80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:40.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.787+0000 7f42bff3a640 1 -- 172.21.15.33:0/2375303268 shutdown_connections 2024-09-16T00:37:40.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.787+0000 7f42bff3a640 1 -- 172.21.15.33:0/2375303268 wait complete. 2024-09-16T00:37:40.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42bff3a640 1 Processor -- start 2024-09-16T00:37:40.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42bff3a640 1 -- start start 2024-09-16T00:37:40.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42bff3a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42b8101de0 0x7f42b81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42bff3a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42b8102790 0x7f42b8198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42bff3a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42b8106f10 0x7f42b819cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42bff3a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f42b8111fc0 con 0x7f42b8102790 2024-09-16T00:37:40.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42bff3a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f42b8111e40 con 0x7f42b8106f10 2024-09-16T00:37:40.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42bff3a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f42b8112140 con 0x7f42b8101de0 2024-09-16T00:37:40.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42bd4d5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42b8102790 0x7f42b8198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42bd4d5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42b8102790 0x7f42b8198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34730/0 (socket says 172.21.15.33:34730) 2024-09-16T00:37:40.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42bd4d5640 1 -- 172.21.15.33:0/3423998234 learned_addr learned my addr 172.21.15.33:0/3423998234 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:40.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42bdcd6640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42b8101de0 0x7f42b81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.788+0000 7f42be4d7640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42b8106f10 0x7f42b819cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.789+0000 7f42bd4d5640 1 -- 172.21.15.33:0/3423998234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42b8101de0 msgr2=0x7f42b81983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:40.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.789+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42b8101de0 0x7f42b81983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.789+0000 7f42bd4d5640 1 -- 172.21.15.33:0/3423998234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42b8106f10 msgr2=0x7f42b819cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:40.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.789+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42b8106f10 0x7f42b819cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.789+0000 7f42bd4d5640 1 -- 172.21.15.33:0/3423998234 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b819d370 con 0x7f42b8102790 2024-09-16T00:37:40.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.789+0000 7f42bdcd6640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42b8101de0 0x7f42b81983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:40.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.789+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42b8102790 0x7f42b8198900 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f42a8004770 tx=0x7f42a8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:40.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.789+0000 7f42aeffd640 1 -- 172.21.15.33:0/3423998234 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a803f070 con 0x7f42b8102790 2024-09-16T00:37:40.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.789+0000 7f42aeffd640 1 -- 172.21.15.33:0/3423998234 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42a8031ca0 con 0x7f42b8102790 2024-09-16T00:37:40.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.789+0000 7f42bff3a640 1 -- 172.21.15.33:0/3423998234 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42b819d600 con 0x7f42b8102790 2024-09-16T00:37:40.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.789+0000 7f42aeffd640 1 -- 172.21.15.33:0/3423998234 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a8035030 con 0x7f42b8102790 2024-09-16T00:37:40.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.789+0000 7f42bff3a640 1 -- 172.21.15.33:0/3423998234 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42b81a4f40 con 0x7f42b8102790 2024-09-16T00:37:40.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.791+0000 7f42aeffd640 1 -- 172.21.15.33:0/3423998234 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f42a8042070 con 0x7f42b8102790 2024-09-16T00:37:40.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42c2313740 1 Processor -- start 2024-09-16T00:37:40.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42c2313740 1 -- start start 2024-09-16T00:37:40.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42c2313740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d14b40 0x55f6e2d13090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42c2313740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 0x55f6e2d13a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42c2313740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d1e370 0x55f6e2d20760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42c2313740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f6e2c792f0 con 0x55f6e2d135d0 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42c2313740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f6e2c20cf0 con 0x55f6e2d14b40 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42c2313740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f6e2c752b0 con 0x55f6e2d1e370 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42aeffd640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42a40a32c0 0x7f42a40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42bdcd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 0x55f6e2d13a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42bdcd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 0x55f6e2d13a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34740/0 (socket says 172.21.15.33:34740) 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42bdcd6640 1 -- 172.21.15.33:0/1833869860 learned_addr learned my addr 172.21.15.33:0/1833869860 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.792+0000 7f42be4d7640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42a40a32c0 0x7f42a40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42aeffd640 1 -- 172.21.15.33:0/3423998234 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f42a80f0030 con 0x7f42b8102790 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42be4d7640 1 --2- 172.21.15.33:0/1833869860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d14b40 0x55f6e2d13090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/1833869860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d1e370 0x55f6e2d20760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42bdcd6640 1 -- 172.21.15.33:0/1833869860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d1e370 msgr2=0x55f6e2d20760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42bdcd6640 1 --2- 172.21.15.33:0/1833869860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d1e370 0x55f6e2d20760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42bdcd6640 1 -- 172.21.15.33:0/1833869860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d14b40 msgr2=0x55f6e2d13090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:40.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42bdcd6640 1 --2- 172.21.15.33:0/1833869860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d14b40 0x55f6e2d13090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42bdcd6640 1 -- 172.21.15.33:0/1833869860 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6e2c053a0 con 0x55f6e2d135d0 2024-09-16T00:37:40.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42be4d7640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42a40a32c0 0x7f42a40a5780 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f42b0007fd0 tx=0x7f42b00072a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:40.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42bdcd6640 1 --2- 172.21.15.33:0/1833869860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 0x55f6e2d13a10 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f42b40115f0 tx=0x7f42b4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a506dfe638abc5d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:40.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42a3fff640 1 -- 172.21.15.33:0/1833869860 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b401e070 con 0x55f6e2d135d0 2024-09-16T00:37:40.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42a3fff640 1 -- 172.21.15.33:0/1833869860 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42b400fb20 con 0x55f6e2d135d0 2024-09-16T00:37:40.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.793+0000 7f42a3fff640 1 -- 172.21.15.33:0/1833869860 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4019bf0 con 0x55f6e2d135d0 2024-09-16T00:37:40.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.794+0000 7f42c2313740 1 -- 172.21.15.33:0/1833869860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 msgr2=0x55f6e2d13a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:40.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.794+0000 7f42c2313740 1 --2- 172.21.15.33:0/1833869860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 0x55f6e2d13a10 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f42b40115f0 tx=0x7f42b4011ab0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.794+0000 7f42c2313740 1 -- 172.21.15.33:0/1833869860 shutdown_connections 2024-09-16T00:37:40.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.794+0000 7f42c2313740 1 --2- 172.21.15.33:0/1833869860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d1e370 0x55f6e2d20760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.794+0000 7f42c2313740 1 --2- 172.21.15.33:0/1833869860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 0x55f6e2d13a10 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.794+0000 7f42c2313740 1 --2- 172.21.15.33:0/1833869860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d14b40 0x55f6e2d13090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.794+0000 7f42c2313740 1 -- 172.21.15.33:0/1833869860 >> 172.21.15.33:0/1833869860 conn(0x55f6e2cdc310 msgr2=0x55f6e2d0fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:40.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.794+0000 7f42c2313740 1 -- 172.21.15.33:0/1833869860 shutdown_connections 2024-09-16T00:37:40.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.794+0000 7f42c2313740 1 -- 172.21.15.33:0/1833869860 wait complete. 2024-09-16T00:37:40.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.794+0000 7f42c2313740 1 -- start start 2024-09-16T00:37:40.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.794+0000 7f42c2313740 1 Processor -- start 2024-09-16T00:37:40.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42c2313740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d135d0 0x55f6e2d1dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42c2313740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d14b40 0x55f6e2d1aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42c2313740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d1e370 0x55f6e2d1b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42c2313740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f6e2d21070 con 0x55f6e2d1e370 2024-09-16T00:37:40.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42c2313740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f6e2c34060 con 0x55f6e2d14b40 2024-09-16T00:37:40.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42c2313740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f6e2d211f0 con 0x55f6e2d135d0 2024-09-16T00:37:40.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42c2313740 10 client.0 _ll_register_callbacks cb 0x55f6e29878c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:40.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42be4d7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d1e370 0x55f6e2d1b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42be4d7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d1e370 0x55f6e2d1b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34754/0 (socket says 172.21.15.33:34754) 2024-09-16T00:37:40.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42be4d7640 1 -- 172.21.15.33:0/3974103116 learned_addr learned my addr 172.21.15.33:0/3974103116 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:40.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42bdcd6640 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d135d0 0x55f6e2d1dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d14b40 0x55f6e2d1aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42be4d7640 1 -- 172.21.15.33:0/3974103116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d135d0 msgr2=0x55f6e2d1dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:40.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42be4d7640 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d135d0 0x55f6e2d1dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42be4d7640 1 -- 172.21.15.33:0/3974103116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d14b40 msgr2=0x55f6e2d1aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42be4d7640 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d14b40 0x55f6e2d1aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42be4d7640 1 -- 172.21.15.33:0/3974103116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6e29ea3c0 con 0x55f6e2d1e370 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d14b40 0x55f6e2d1aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42be4d7640 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d1e370 0x55f6e2d1b5e0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f42b0007fa0 tx=0x7f42b0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3760eee81c46832a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.795+0000 7f42a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42a17fa640 1 -- 172.21.15.33:0/3974103116 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0024030 con 0x55f6e2d1e370 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42a17fa640 1 -- 172.21.15.33:0/3974103116 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f42b0012ca0 con 0x55f6e2d1e370 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42c2313740 1 -- 172.21.15.33:0/3974103116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42b400fb20 con 0x55f6e2d1e370 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42a17fa640 1 -- 172.21.15.33:0/3974103116 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f42b0031050 con 0x55f6e2d1e370 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42c2313740 10 client.19281 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42c2313740 1 -- 172.21.15.33:0/3974103116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f6e2d2cae0 con 0x55f6e2d1e370 2024-09-16T00:37:40.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42a17fa640 1 -- 172.21.15.33:0/3974103116 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42b400fb20 con 0x55f6e2d1e370 2024-09-16T00:37:40.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42a17fa640 1 -- 172.21.15.33:0/3974103116 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f42b001d050 con 0x55f6e2d1e370 2024-09-16T00:37:40.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42c2313740 10 client.19281 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:40.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42c2313740 10 client.19281 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:37:40.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42c2313740 20 client.19281 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:40.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42c2313740 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f6e2d2dfa0 0x55f6e2d30440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42c2313740 4 client.19281 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:37:40.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.796+0000 7f42c2313740 1 -- 172.21.15.33:0/3974103116 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6e2c86860 con 0x55f6e2d2dfa0 2024-09-16T00:37:40.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.797+0000 7f42bdcd6640 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f6e2d2dfa0 0x55f6e2d30440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.797+0000 7f42bdcd6640 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f6e2d2dfa0 0x55f6e2d30440 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=13d01453b48d3242 server_cookie=7f756a4bed5768f4 in_seq=0 out_seq=0 2024-09-16T00:37:40.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.797+0000 7f42a17fa640 10 client.19281 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:37:40.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42a17fa640 1 -- 172.21.15.33:0/3974103116 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f6e2c86860 con 0x55f6e2d2dfa0 2024-09-16T00:37:40.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42a17fa640 10 client.19281 handle_command_reply: tid=0 2024-09-16T00:37:40.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42c2313740 1 client.19281 shutdown 2024-09-16T00:37:40.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42c2313740 1 -- 172.21.15.33:0/3974103116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d1e370 msgr2=0x55f6e2d1b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:40.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42c2313740 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d1e370 0x55f6e2d1b5e0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f42b0007fa0 tx=0x7f42b0004ab0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42c2313740 1 -- 172.21.15.33:0/3974103116 shutdown_connections 2024-09-16T00:37:40.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42c2313740 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d1e370 0x55f6e2d1b5e0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42c2313740 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f6e2d2dfa0 0x55f6e2d30440 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42c2313740 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d14b40 0x55f6e2d1aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42c2313740 1 --2- 172.21.15.33:0/3974103116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d135d0 0x55f6e2d1dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42c2313740 1 -- 172.21.15.33:0/3974103116 >> 172.21.15.33:0/3974103116 conn(0x55f6e2cdc310 msgr2=0x55f6e2d15960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:40.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42a17fa640 0 client.19281 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:37:40.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42c2313740 1 -- 172.21.15.33:0/3974103116 shutdown_connections 2024-09-16T00:37:40.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42c2313740 1 -- 172.21.15.33:0/3974103116 wait complete. 2024-09-16T00:37:40.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.798+0000 7f42c2313740 20 client.19281 trim_cache size 0 max 16384 2024-09-16T00:37:40.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.811+0000 7f42c2313740 1 Processor -- start 2024-09-16T00:37:40.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.811+0000 7f42c2313740 1 -- start start 2024-09-16T00:37:40.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42c2313740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d135d0 0x55f6e2d2a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42c2313740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d14b40 0x55f6e2d0f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42c2313740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d1e370 0x55f6e2d0fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42c2313740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f42b81b1900 con 0x55f6e2d14b40 2024-09-16T00:37:40.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42c2313740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f42b81af1d0 con 0x55f6e2d135d0 2024-09-16T00:37:40.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42c2313740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f42b81ad020 con 0x55f6e2d1e370 2024-09-16T00:37:40.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42bd4d5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d14b40 0x55f6e2d0f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42bd4d5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d14b40 0x55f6e2d0f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34766/0 (socket says 172.21.15.33:34766) 2024-09-16T00:37:40.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42bd4d5640 1 -- 172.21.15.33:0/669543212 learned_addr learned my addr 172.21.15.33:0/669543212 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:40.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42be4d7640 1 --2- 172.21.15.33:0/669543212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d1e370 0x55f6e2d0fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42bdcd6640 1 --2- 172.21.15.33:0/669543212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d135d0 0x55f6e2d2a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42bd4d5640 1 -- 172.21.15.33:0/669543212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d1e370 msgr2=0x55f6e2d0fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:40.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/669543212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d1e370 0x55f6e2d0fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42bd4d5640 1 -- 172.21.15.33:0/669543212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d135d0 msgr2=0x55f6e2d2a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:40.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/669543212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d135d0 0x55f6e2d2a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42bd4d5640 1 -- 172.21.15.33:0/669543212 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6e2d10180 con 0x55f6e2d14b40 2024-09-16T00:37:40.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.812+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/669543212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d14b40 0x55f6e2d0f700 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f42a80ed700 tx=0x7f42a802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad81220f3bccfc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:40.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.813+0000 7f42a37fe640 1 -- 172.21.15.33:0/669543212 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a8049040 con 0x55f6e2d14b40 2024-09-16T00:37:40.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.813+0000 7f42a37fe640 1 -- 172.21.15.33:0/669543212 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42a80bc280 con 0x55f6e2d14b40 2024-09-16T00:37:40.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.813+0000 7f42a37fe640 1 -- 172.21.15.33:0/669543212 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a803b4f0 con 0x55f6e2d14b40 2024-09-16T00:37:40.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.813+0000 7f42c2313740 1 -- 172.21.15.33:0/669543212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d14b40 msgr2=0x55f6e2d0f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:40.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.813+0000 7f42c2313740 1 --2- 172.21.15.33:0/669543212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d14b40 0x55f6e2d0f700 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f42a80ed700 tx=0x7f42a802a410 comp rx=0 tx=0).stop 2024-09-16T00:37:40.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.813+0000 7f42c2313740 1 -- 172.21.15.33:0/669543212 shutdown_connections 2024-09-16T00:37:40.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.813+0000 7f42c2313740 1 --2- 172.21.15.33:0/669543212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d1e370 0x55f6e2d0fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.813+0000 7f42c2313740 1 --2- 172.21.15.33:0/669543212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d14b40 0x55f6e2d0f700 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.813+0000 7f42c2313740 1 --2- 172.21.15.33:0/669543212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d135d0 0x55f6e2d2a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.813+0000 7f42c2313740 1 -- 172.21.15.33:0/669543212 >> 172.21.15.33:0/669543212 conn(0x55f6e2cdc310 msgr2=0x55f6e2d1afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:40.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.813+0000 7f42c2313740 1 -- 172.21.15.33:0/669543212 shutdown_connections 2024-09-16T00:37:40.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.813+0000 7f42c2313740 1 -- 172.21.15.33:0/669543212 wait complete. 2024-09-16T00:37:40.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42c2313740 1 -- start start 2024-09-16T00:37:40.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42c2313740 1 Processor -- start 2024-09-16T00:37:40.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42c2313740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 0x55f6e2d22360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42c2313740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d14b40 0x55f6e2d228a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42c2313740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d1e370 0x55f6e2d22de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42c2313740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f42b81ad690 con 0x55f6e2d135d0 2024-09-16T00:37:40.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42c2313740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f42b81aeb00 con 0x55f6e2d1e370 2024-09-16T00:37:40.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42c2313740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f42b81add60 con 0x55f6e2d14b40 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42c2313740 10 client.0 _ll_register_callbacks cb 0x7f42b0007b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42bdcd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 0x55f6e2d22360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42bdcd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 0x55f6e2d22360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34774/0 (socket says 172.21.15.33:34774) 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42bdcd6640 1 -- 172.21.15.33:0/2490955346 learned_addr learned my addr 172.21.15.33:0/2490955346 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d14b40 0x55f6e2d228a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.814+0000 7f42be4d7640 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d1e370 0x55f6e2d22de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42bdcd6640 1 -- 172.21.15.33:0/2490955346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d14b40 msgr2=0x55f6e2d228a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42bdcd6640 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d14b40 0x55f6e2d228a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42bdcd6640 1 -- 172.21.15.33:0/2490955346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d1e370 msgr2=0x55f6e2d22de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42bdcd6640 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d1e370 0x55f6e2d22de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42bdcd6640 1 -- 172.21.15.33:0/2490955346 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6e2d97ed0 con 0x55f6e2d135d0 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d14b40 0x55f6e2d228a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:40.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42bdcd6640 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 0x55f6e2d22360 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f42b4000c00 tx=0x7f42b40093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdbc79aacad6cbb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42a17fa640 1 -- 172.21.15.33:0/2490955346 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4028030 con 0x55f6e2d135d0 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42a17fa640 1 -- 172.21.15.33:0/2490955346 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f42b4016040 con 0x55f6e2d135d0 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42c2313740 1 -- 172.21.15.33:0/2490955346 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42a80bc280 con 0x55f6e2d135d0 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42a17fa640 1 -- 172.21.15.33:0/2490955346 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f42b4037050 con 0x55f6e2d135d0 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42c2313740 10 client.19287 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42c2313740 1 -- 172.21.15.33:0/2490955346 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f6e2d99190 con 0x55f6e2d135d0 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.815+0000 7f42a17fa640 1 -- 172.21.15.33:0/2490955346 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42a80bc280 con 0x55f6e2d135d0 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.816+0000 7f42a17fa640 1 -- 172.21.15.33:0/2490955346 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f42b4019ac0 con 0x55f6e2d135d0 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.816+0000 7f42c2313740 10 client.19287 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.816+0000 7f42c2313740 10 client.19287 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.816+0000 7f42c2313740 20 client.19287 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.816+0000 7f42c2313740 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f6e2d99750 0x55f6e2d9bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.816+0000 7f42c2313740 4 client.19287 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:37:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.816+0000 7f42c2313740 1 -- 172.21.15.33:0/2490955346 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f6e2c80d70 con 0x55f6e2d99750 2024-09-16T00:37:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.816+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f6e2d99750 0x55f6e2d9bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.816+0000 7f42a17fa640 10 client.19287 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:37:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.816+0000 7f42bd4d5640 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f6e2d99750 0x55f6e2d9bb70 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=666cade03865940e server_cookie=fc60503a3ce8f56c in_seq=0 out_seq=0 2024-09-16T00:37:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42a17fa640 1 -- 172.21.15.33:0/2490955346 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f6e2c80d70 con 0x55f6e2d99750 2024-09-16T00:37:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42a17fa640 10 client.19287 handle_command_reply: tid=0 2024-09-16T00:37:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42c2313740 1 client.19287 shutdown 2024-09-16T00:37:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42c2313740 1 -- 172.21.15.33:0/2490955346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 msgr2=0x55f6e2d22360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42c2313740 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 0x55f6e2d22360 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f42b4000c00 tx=0x7f42b40093b0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42c2313740 1 -- 172.21.15.33:0/2490955346 shutdown_connections 2024-09-16T00:37:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42c2313740 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55f6e2d99750 0x55f6e2d9bb70 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42c2313740 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f6e2d1e370 0x55f6e2d22de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42c2313740 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6e2d14b40 0x55f6e2d228a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42c2313740 1 --2- 172.21.15.33:0/2490955346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6e2d135d0 0x55f6e2d22360 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42a17fa640 0 client.19287 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:37:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42c2313740 1 -- 172.21.15.33:0/2490955346 >> 172.21.15.33:0/2490955346 conn(0x55f6e2cdc310 msgr2=0x55f6e2d15110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42c2313740 1 -- 172.21.15.33:0/2490955346 shutdown_connections 2024-09-16T00:37:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42c2313740 1 -- 172.21.15.33:0/2490955346 wait complete. 2024-09-16T00:37:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42c2313740 20 client.19287 trim_cache size 0 max 16384 2024-09-16T00:37:40.823 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:37:40.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42bff3a640 1 -- 172.21.15.33:0/3423998234 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42a40a32c0 msgr2=0x7f42a40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:40.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42bff3a640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42a40a32c0 0x7f42a40a5780 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f42b0007fd0 tx=0x7f42b00072a0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42bff3a640 1 -- 172.21.15.33:0/3423998234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42b8102790 msgr2=0x7f42b8198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:40.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.817+0000 7f42bff3a640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42b8102790 0x7f42b8198900 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f42a8004770 tx=0x7f42a8003c10 comp rx=0 tx=0).stop 2024-09-16T00:37:40.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.818+0000 7f42bff3a640 1 -- 172.21.15.33:0/3423998234 shutdown_connections 2024-09-16T00:37:40.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.818+0000 7f42bff3a640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42a40a32c0 0x7f42a40a5780 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.818+0000 7f42bff3a640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42b8106f10 0x7f42b819cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.818+0000 7f42bff3a640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42b8102790 0x7f42b8198900 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.818+0000 7f42bff3a640 1 --2- 172.21.15.33:0/3423998234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42b8101de0 0x7f42b81983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:40.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.818+0000 7f42bff3a640 1 -- 172.21.15.33:0/3423998234 >> 172.21.15.33:0/3423998234 conn(0x7f42b80fd8c0 msgr2=0x7f42b80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:40.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.818+0000 7f42bff3a640 1 -- 172.21.15.33:0/3423998234 shutdown_connections 2024-09-16T00:37:40.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:40.818+0000 7f42bff3a640 1 -- 172.21.15.33:0/3423998234 wait complete. 2024-09-16T00:37:41.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 17.03 MB/sec execute 287 sec latency 17126.840 ms 2024-09-16T00:37:41.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:41 smithi033 ceph-mon[31243]: pgmap v966: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 398 KiB/s wr, 47 op/s 2024-09-16T00:37:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:41 smithi083 ceph-mon[35242]: pgmap v966: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 398 KiB/s wr, 47 op/s 2024-09-16T00:37:41.963 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:37:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:41 smithi060 ceph-mon[35185]: pgmap v966: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 398 KiB/s wr, 47 op/s 2024-09-16T00:37:42.089 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 16.97 MB/sec execute 288 sec latency 18126.903 ms 2024-09-16T00:37:42.224 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:42.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.816+0000 7f7b8d392640 1 -- 172.21.15.33:0/1627196315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8810c170 msgr2=0x7f7b8810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:42.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.816+0000 7f7b8d392640 1 --2- 172.21.15.33:0/1627196315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8810c170 0x7f7b8810e610 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f7b80009ca0 tx=0x7f7b8002a950 comp rx=0 tx=0).stop 2024-09-16T00:37:42.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.816+0000 7f7b8d392640 1 -- 172.21.15.33:0/1627196315 shutdown_connections 2024-09-16T00:37:42.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.816+0000 7f7b8d392640 1 --2- 172.21.15.33:0/1627196315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8810c170 0x7f7b8810e610 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.817+0000 7f7b8d392640 1 --2- 172.21.15.33:0/1627196315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b88103660 0x7f7b8810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.817+0000 7f7b8d392640 1 --2- 172.21.15.33:0/1627196315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b88102d40 0x7f7b88103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.817+0000 7f7b8d392640 1 -- 172.21.15.33:0/1627196315 >> 172.21.15.33:0/1627196315 conn(0x7f7b880fc7a0 msgr2=0x7f7b880febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:42.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.817+0000 7f7b8d392640 1 -- 172.21.15.33:0/1627196315 shutdown_connections 2024-09-16T00:37:42.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.817+0000 7f7b8d392640 1 -- 172.21.15.33:0/1627196315 wait complete. 2024-09-16T00:37:42.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.817+0000 7f7b8d392640 1 Processor -- start 2024-09-16T00:37:42.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.817+0000 7f7b8d392640 1 -- start start 2024-09-16T00:37:42.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.817+0000 7f7b8d392640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b88102d40 0x7f7b88198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b8d392640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b88103660 0x7f7b88198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b8d392640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8810c170 0x7f7b8819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b8d392640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b88111f70 con 0x7f7b8810c170 2024-09-16T00:37:42.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b8d392640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7b88111df0 con 0x7f7b88102d40 2024-09-16T00:37:42.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b8d392640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881120f0 con 0x7f7b88103660 2024-09-16T00:37:42.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b877fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8810c170 0x7f7b8819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b877fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8810c170 0x7f7b8819cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34782/0 (socket says 172.21.15.33:34782) 2024-09-16T00:37:42.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b877fe640 1 -- 172.21.15.33:0/868179155 learned_addr learned my addr 172.21.15.33:0/868179155 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:42.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b867fc640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b88103660 0x7f7b88198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b86ffd640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b88102d40 0x7f7b88198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b877fe640 1 -- 172.21.15.33:0/868179155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b88103660 msgr2=0x7f7b88198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b877fe640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b88103660 0x7f7b88198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b877fe640 1 -- 172.21.15.33:0/868179155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b88102d40 msgr2=0x7f7b88198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b877fe640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b88102d40 0x7f7b88198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b877fe640 1 -- 172.21.15.33:0/868179155 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8819d3c0 con 0x7f7b8810c170 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b877fe640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8810c170 0x7f7b8819cce0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f7b80006010 tx=0x7f7b8002f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.818+0000 7f7b6ffff640 1 -- 172.21.15.33:0/868179155 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b80034040 con 0x7f7b8810c170 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.819+0000 7f7b6ffff640 1 -- 172.21.15.33:0/868179155 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b80003e00 con 0x7f7b8810c170 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.819+0000 7f7b8d392640 1 -- 172.21.15.33:0/868179155 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b8819d650 con 0x7f7b8810c170 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.819+0000 7f7b6ffff640 1 -- 172.21.15.33:0/868179155 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b800456e0 con 0x7f7b8810c170 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.819+0000 7f7b8d392640 1 -- 172.21.15.33:0/868179155 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b881a4f90 con 0x7f7b8810c170 2024-09-16T00:37:42.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b6ffff640 1 -- 172.21.15.33:0/868179155 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7b80037020 con 0x7f7b8810c170 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b8f76b740 1 Processor -- start 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b8f76b740 1 -- start start 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b8f76b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 0x55f48056d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b8f76b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056d5d0 0x55f48056da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b8f76b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f48057a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b8f76b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f4804d32f0 con 0x55f480578370 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b8f76b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f48047acf0 con 0x55f48056d5d0 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b8f76b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f4804cf2b0 con 0x55f48056eb40 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f48057a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f48057a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34798/0 (socket says 172.21.15.33:34798) 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b867fc640 1 -- 172.21.15.33:0/2841186581 learned_addr learned my addr 172.21.15.33:0/2841186581 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.821+0000 7f7b867fc640 1 --2- 172.21.15.33:0/2841186581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 0x55f48056d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b86ffd640 1 --2- 172.21.15.33:0/2841186581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056d5d0 0x55f48056da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b6ffff640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b700a3240 0x7f7b700a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b867fc640 1 -- 172.21.15.33:0/2841186581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 msgr2=0x55f48056d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:42.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b867fc640 1 --2- 172.21.15.33:0/2841186581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 0x55f48056d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b867fc640 1 -- 172.21.15.33:0/2841186581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056d5d0 msgr2=0x55f48056da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:42.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b867fc640 1 --2- 172.21.15.33:0/2841186581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056d5d0 0x55f48056da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b867fc640 1 -- 172.21.15.33:0/2841186581 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f48045f3a0 con 0x55f480578370 2024-09-16T00:37:42.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b877fe640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b700a3240 0x7f7b700a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b6ffff640 1 -- 172.21.15.33:0/868179155 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7b800f0e60 con 0x7f7b8810c170 2024-09-16T00:37:42.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b867fc640 1 --2- 172.21.15.33:0/2841186581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f48057a760 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0116a0 tx=0x7f7b7c00a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a01f2438389f4ef server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:42.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b6d7fa640 1 -- 172.21.15.33:0/2841186581 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c011250 con 0x55f480578370 2024-09-16T00:37:42.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b6d7fa640 1 -- 172.21.15.33:0/2841186581 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b7c005020 con 0x55f480578370 2024-09-16T00:37:42.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.822+0000 7f7b6d7fa640 1 -- 172.21.15.33:0/2841186581 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c004ae0 con 0x55f480578370 2024-09-16T00:37:42.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.823+0000 7f7b8f76b740 1 -- 172.21.15.33:0/2841186581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 msgr2=0x55f48057a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:42.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.823+0000 7f7b877fe640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b700a3240 0x7f7b700a5700 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7b800bcd00 tx=0x7f7b8008bd40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:42.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.823+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/2841186581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f48057a760 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0116a0 tx=0x7f7b7c00a610 comp rx=0 tx=0).stop 2024-09-16T00:37:42.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.823+0000 7f7b8f76b740 1 -- 172.21.15.33:0/2841186581 shutdown_connections 2024-09-16T00:37:42.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.823+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/2841186581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f48057a760 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.823+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/2841186581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056d5d0 0x55f48056da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.823+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/2841186581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 0x55f48056d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.823+0000 7f7b8f76b740 1 -- 172.21.15.33:0/2841186581 >> 172.21.15.33:0/2841186581 conn(0x55f480536310 msgr2=0x55f480569c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:42.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.823+0000 7f7b8f76b740 1 -- 172.21.15.33:0/2841186581 shutdown_connections 2024-09-16T00:37:42.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.823+0000 7f7b8f76b740 1 -- 172.21.15.33:0/2841186581 wait complete. 2024-09-16T00:37:42.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.823+0000 7f7b8f76b740 1 -- start start 2024-09-16T00:37:42.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.823+0000 7f7b8f76b740 1 Processor -- start 2024-09-16T00:37:42.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b8f76b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056d5d0 0x55f480577da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b8f76b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 0x55f480575050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b8f76b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f4805757e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b8f76b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f48057b070 con 0x55f480578370 2024-09-16T00:37:42.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b8f76b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f48048e060 con 0x55f48056d5d0 2024-09-16T00:37:42.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b8f76b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f48057b1f0 con 0x55f48056eb40 2024-09-16T00:37:42.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b8f76b740 10 client.0 _ll_register_callbacks cb 0x55f4801e18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:42.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f4805757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f4805757e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34800/0 (socket says 172.21.15.33:34800) 2024-09-16T00:37:42.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b867fc640 1 -- 172.21.15.33:0/3475981546 learned_addr learned my addr 172.21.15.33:0/3475981546 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:42.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b867fc640 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056d5d0 0x55f480577da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b86ffd640 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 0x55f480575050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b867fc640 1 -- 172.21.15.33:0/3475981546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 msgr2=0x55f480575050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:42.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b867fc640 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 0x55f480575050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b867fc640 1 -- 172.21.15.33:0/3475981546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056d5d0 msgr2=0x55f480577da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b867fc640 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056d5d0 0x55f480577da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b867fc640 1 -- 172.21.15.33:0/3475981546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4802443c0 con 0x55f480578370 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b867fc640 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f4805757e0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0047e0 tx=0x7f7b7c01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f819fa5e02a449cc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.824+0000 7f7b6aff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b6aff5640 1 -- 172.21.15.33:0/3475981546 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c01e470 con 0x55f480578370 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b6aff5640 1 -- 172.21.15.33:0/3475981546 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7b7c028040 con 0x55f480578370 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3475981546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b7c005020 con 0x55f480578370 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b6aff5640 1 -- 172.21.15.33:0/3475981546 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7b7c031050 con 0x55f480578370 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b8f76b740 10 client.19299 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3475981546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f480586b50 con 0x55f480578370 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b6aff5640 1 -- 172.21.15.33:0/3475981546 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b7c005020 con 0x55f480578370 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b6aff5640 1 -- 172.21.15.33:0/3475981546 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7b7c010640 con 0x55f480578370 2024-09-16T00:37:42.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b8f76b740 10 client.19299 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b8f76b740 10 client.19299 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b8f76b740 20 client.19299 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f480587ea0 0x55f48058a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b8f76b740 4 client.19299 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.825+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3475981546 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4804e0860 con 0x55f480587ea0 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.826+0000 7f7b86ffd640 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f480587ea0 0x55f48058a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.826+0000 7f7b86ffd640 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f480587ea0 0x55f48058a290 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=662ae6f8fc99e005 server_cookie=7bf8626f702d4a23 in_seq=0 out_seq=0 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.826+0000 7f7b6aff5640 10 client.19299 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b6aff5640 1 -- 172.21.15.33:0/3475981546 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f4804e0860 con 0x55f480587ea0 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b6aff5640 10 client.19299 handle_command_reply: tid=0 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b8f76b740 1 client.19299 shutdown 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3475981546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 msgr2=0x55f4805757e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:42.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f4805757e0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0047e0 tx=0x7f7b7c01a040 comp rx=0 tx=0).stop 2024-09-16T00:37:42.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3475981546 shutdown_connections 2024-09-16T00:37:42.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f480587ea0 0x55f48058a290 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f4805757e0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 0x55f480575050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b6aff5640 0 client.19299 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:42.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3475981546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056d5d0 0x55f480577da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3475981546 >> 172.21.15.33:0/3475981546 conn(0x55f480536310 msgr2=0x55f48056f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:42.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3475981546 shutdown_connections 2024-09-16T00:37:42.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3475981546 wait complete. 2024-09-16T00:37:42.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.827+0000 7f7b8f76b740 20 client.19299 trim_cache size 0 max 16384 2024-09-16T00:37:42.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.840+0000 7f7b8f76b740 1 Processor -- start 2024-09-16T00:37:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.840+0000 7f7b8f76b740 1 -- start start 2024-09-16T00:37:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.840+0000 7f7b8f76b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056d5d0 0x55f480574050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.840+0000 7f7b8f76b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056eb40 0x55f48058f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.840+0000 7f7b8f76b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f48058fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.840+0000 7f7b8f76b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881b1930 con 0x55f480578370 2024-09-16T00:37:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.840+0000 7f7b8f76b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881af200 con 0x55f48056eb40 2024-09-16T00:37:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.840+0000 7f7b8f76b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881ad050 con 0x55f48056d5d0 2024-09-16T00:37:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.840+0000 7f7b867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f48058fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.840+0000 7f7b867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f48058fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34816/0 (socket says 172.21.15.33:34816) 2024-09-16T00:37:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.840+0000 7f7b867fc640 1 -- 172.21.15.33:0/3645699599 learned_addr learned my addr 172.21.15.33:0/3645699599 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:42.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.840+0000 7f7b86ffd640 1 --2- 172.21.15.33:0/3645699599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056eb40 0x55f48058f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.841+0000 7f7b867fc640 1 --2- 172.21.15.33:0/3645699599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056d5d0 0x55f480574050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.841+0000 7f7b867fc640 1 -- 172.21.15.33:0/3645699599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056d5d0 msgr2=0x55f480574050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.841+0000 7f7b867fc640 1 --2- 172.21.15.33:0/3645699599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056d5d0 0x55f480574050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.841+0000 7f7b867fc640 1 -- 172.21.15.33:0/3645699599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056eb40 msgr2=0x55f48058f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.841+0000 7f7b867fc640 1 --2- 172.21.15.33:0/3645699599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056eb40 0x55f48058f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.841+0000 7f7b867fc640 1 -- 172.21.15.33:0/3645699599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4805694f0 con 0x55f480578370 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.841+0000 7f7b86ffd640 1 --2- 172.21.15.33:0/3645699599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056eb40 0x55f48058f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.841+0000 7f7b867fc640 1 --2- 172.21.15.33:0/3645699599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f48058fec0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c006010 tx=0x7f7b7c004810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2c50c25e26a210 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.841+0000 7f7b6cff9640 1 -- 172.21.15.33:0/3645699599 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c00a940 con 0x55f480578370 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.841+0000 7f7b6cff9640 1 -- 172.21.15.33:0/3645699599 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b7c005620 con 0x55f480578370 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.841+0000 7f7b6cff9640 1 -- 172.21.15.33:0/3645699599 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c004310 con 0x55f480578370 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3645699599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 msgr2=0x55f48058fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3645699599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f48058fec0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c006010 tx=0x7f7b7c004810 comp rx=0 tx=0).stop 2024-09-16T00:37:42.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3645699599 shutdown_connections 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3645699599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f480578370 0x55f48058fec0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3645699599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f48056eb40 0x55f48058f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3645699599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056d5d0 0x55f480574050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3645699599 >> 172.21.15.33:0/3645699599 conn(0x55f480536310 msgr2=0x55f480573940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3645699599 shutdown_connections 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3645699599 wait complete. 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 -- start start 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 Processor -- start 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f48056d5d0 0x55f480580960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 0x55f480571b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f480578370 0x55f480572060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881ad6c0 con 0x55f48056d5d0 2024-09-16T00:37:42.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881aeb30 con 0x55f480578370 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881add90 con 0x55f48056eb40 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b8f76b740 10 client.0 _ll_register_callbacks cb 0x7f7b7c00ad20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.842+0000 7f7b867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f48056d5d0 0x55f480580960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f48056d5d0 0x55f480580960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34828/0 (socket says 172.21.15.33:34828) 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b867fc640 1 -- 172.21.15.33:0/3425379243 learned_addr learned my addr 172.21.15.33:0/3425379243 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b86ffd640 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 0x55f480571b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b867fc640 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f480578370 0x55f480572060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b867fc640 1 -- 172.21.15.33:0/3425379243 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 msgr2=0x55f480571b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b867fc640 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 0x55f480571b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b867fc640 1 -- 172.21.15.33:0/3425379243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f480578370 msgr2=0x55f480572060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b867fc640 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f480578370 0x55f480572060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b867fc640 1 -- 172.21.15.33:0/3425379243 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f480572620 con 0x55f48056d5d0 2024-09-16T00:37:42.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b867fc640 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f48056d5d0 0x55f480580960 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00aa90 tx=0x7f7b7c00a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75fa83a54c00e3b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b6aff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b6aff5640 1 -- 172.21.15.33:0/3425379243 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c03c020 con 0x55f48056d5d0 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b6aff5640 1 -- 172.21.15.33:0/3425379243 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7b7c010640 con 0x55f48056d5d0 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3425379243 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b7c005620 con 0x55f48056d5d0 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b6aff5640 1 -- 172.21.15.33:0/3425379243 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7b7c010e50 con 0x55f48056d5d0 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b8f76b740 10 client.19305 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.843+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3425379243 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f4805f2650 con 0x55f48056d5d0 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.844+0000 7f7b6aff5640 1 -- 172.21.15.33:0/3425379243 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b7c005620 con 0x55f48056d5d0 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.844+0000 7f7b6aff5640 1 -- 172.21.15.33:0/3425379243 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7b7c035430 con 0x55f48056d5d0 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.844+0000 7f7b8f76b740 10 client.19305 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.844+0000 7f7b8f76b740 10 client.19305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.844+0000 7f7b8f76b740 20 client.19305 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:42.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.844+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f480587ea0 0x55f4805f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.844+0000 7f7b8f76b740 4 client.19305 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.844+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3425379243 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f4804dad70 con 0x55f480587ea0 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.844+0000 7f7b86ffd640 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f480587ea0 0x55f4805f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b6aff5640 10 client.19305 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b86ffd640 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f480587ea0 0x55f4805f4ca0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c7a7aa5e82ac4d9 server_cookie=a45f96fcede79048 in_seq=0 out_seq=0 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b6aff5640 1 -- 172.21.15.33:0/3425379243 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f4804dad70 con 0x55f480587ea0 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b6aff5640 10 client.19305 handle_command_reply: tid=0 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b8f76b740 1 client.19305 shutdown 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3425379243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f48056d5d0 msgr2=0x55f480580960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f48056d5d0 0x55f480580960 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00aa90 tx=0x7f7b7c00a400 comp rx=0 tx=0).stop 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3425379243 shutdown_connections 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f480587ea0 0x55f4805f4ca0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f480578370 0x55f480572060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f48056eb40 0x55f480571b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b8f76b740 1 --2- 172.21.15.33:0/3425379243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f48056d5d0 0x55f480580960 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3425379243 >> 172.21.15.33:0/3425379243 conn(0x55f480536310 msgr2=0x55f480574160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:42.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b6aff5640 0 client.19305 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:42.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3425379243 shutdown_connections 2024-09-16T00:37:42.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b8f76b740 1 -- 172.21.15.33:0/3425379243 wait complete. 2024-09-16T00:37:42.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.845+0000 7f7b8f76b740 20 client.19305 trim_cache size 0 max 16384 2024-09-16T00:37:42.855 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:37:42.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.846+0000 7f7b8d392640 1 -- 172.21.15.33:0/868179155 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b700a3240 msgr2=0x7f7b700a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:42.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.846+0000 7f7b8d392640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b700a3240 0x7f7b700a5700 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7b800bcd00 tx=0x7f7b8008bd40 comp rx=0 tx=0).stop 2024-09-16T00:37:42.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.846+0000 7f7b8d392640 1 -- 172.21.15.33:0/868179155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8810c170 msgr2=0x7f7b8819cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:42.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.846+0000 7f7b8d392640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8810c170 0x7f7b8819cce0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f7b80006010 tx=0x7f7b8002f710 comp rx=0 tx=0).stop 2024-09-16T00:37:42.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.846+0000 7f7b8d392640 1 -- 172.21.15.33:0/868179155 shutdown_connections 2024-09-16T00:37:42.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.846+0000 7f7b8d392640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b700a3240 0x7f7b700a5700 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.846+0000 7f7b8d392640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8810c170 0x7f7b8819cce0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.846+0000 7f7b8d392640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b88103660 0x7f7b88198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.846+0000 7f7b8d392640 1 --2- 172.21.15.33:0/868179155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b88102d40 0x7f7b88198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:42.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.846+0000 7f7b8d392640 1 -- 172.21.15.33:0/868179155 >> 172.21.15.33:0/868179155 conn(0x7f7b880fc7a0 msgr2=0x7f7b880fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:42.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.846+0000 7f7b8d392640 1 -- 172.21.15.33:0/868179155 shutdown_connections 2024-09-16T00:37:42.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:42.846+0000 7f7b8d392640 1 -- 172.21.15.33:0/868179155 wait complete. 2024-09-16T00:37:43.090 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 16.92 MB/sec execute 289 sec latency 19126.990 ms 2024-09-16T00:37:43.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:43 smithi033 ceph-mon[31243]: pgmap v967: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 44 KiB/s wr, 21 op/s 2024-09-16T00:37:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:43 smithi083 ceph-mon[35242]: pgmap v967: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 44 KiB/s wr, 21 op/s 2024-09-16T00:37:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:43 smithi060 ceph-mon[35185]: pgmap v967: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 44 KiB/s wr, 21 op/s 2024-09-16T00:37:44.090 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 16.86 MB/sec execute 290 sec latency 20127.093 ms 2024-09-16T00:37:44.669 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:37:44.669 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:37:44.941 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:45.090 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 16.80 MB/sec execute 291 sec latency 21127.216 ms 2024-09-16T00:37:45.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.568+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2114347178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1bc102570 msgr2=0x7fe1bc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:45.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.568+0000 7fe1c3cb2640 1 --2- 172.21.15.33:0/2114347178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1bc102570 0x7fe1bc1029b0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009900 tx=0x7fe1ac02a480 comp rx=0 tx=0).stop 2024-09-16T00:37:45.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.568+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2114347178 shutdown_connections 2024-09-16T00:37:45.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.568+0000 7fe1c3cb2640 1 --2- 172.21.15.33:0/2114347178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1bc102ef0 0x7fe1bc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.568+0000 7fe1c3cb2640 1 --2- 172.21.15.33:0/2114347178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1bc102570 0x7fe1bc1029b0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.568+0000 7fe1c3cb2640 1 --2- 172.21.15.33:0/2114347178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc100e20 0x7fe1bc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.568+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2114347178 >> 172.21.15.33:0/2114347178 conn(0x7fe1bc0fc7d0 msgr2=0x7fe1bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:45.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.568+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2114347178 shutdown_connections 2024-09-16T00:37:45.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.568+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2114347178 wait complete. 2024-09-16T00:37:45.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c3cb2640 1 Processor -- start 2024-09-16T00:37:45.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c3cb2640 1 -- start start 2024-09-16T00:37:45.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c3cb2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1bc100e20 0x7fe1bc1973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c3cb2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1bc102570 0x7fe1bc197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c3cb2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc102ef0 0x7fe1bc19bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c3cb2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc110f50 con 0x7fe1bc100e20 2024-09-16T00:37:45.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c3cb2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc110dd0 con 0x7fe1bc102570 2024-09-16T00:37:45.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c3cb2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1110d0 con 0x7fe1bc102ef0 2024-09-16T00:37:45.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c1a4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1bc100e20 0x7fe1bc1973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c124d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1bc102570 0x7fe1bc197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c1a4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1bc100e20 0x7fe1bc1973d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34842/0 (socket says 172.21.15.33:34842) 2024-09-16T00:37:45.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c124d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1bc102570 0x7fe1bc197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:52888/0 (socket says 172.21.15.33:52888) 2024-09-16T00:37:45.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c1a4e640 1 -- 172.21.15.33:0/2429270742 learned_addr learned my addr 172.21.15.33:0/2429270742 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:45.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.569+0000 7fe1c224f640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc102ef0 0x7fe1bc19bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1c124d640 1 -- 172.21.15.33:0/2429270742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc102ef0 msgr2=0x7fe1bc19bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:45.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1c124d640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc102ef0 0x7fe1bc19bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1c124d640 1 -- 172.21.15.33:0/2429270742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1bc100e20 msgr2=0x7fe1bc1973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:45.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1c124d640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1bc100e20 0x7fe1bc1973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1c124d640 1 -- 172.21.15.33:0/2429270742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1bc19c380 con 0x7fe1bc102570 2024-09-16T00:37:45.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1bc100e20 0x7fe1bc1973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:45.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1c224f640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc102ef0 0x7fe1bc19bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:37:45.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1c124d640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1bc102570 0x7fe1bc197910 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac005940 tx=0x7fe1ac003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:45.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1b2ffd640 1 -- 172.21.15.33:0/2429270742 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac03f070 con 0x7fe1bc102570 2024-09-16T00:37:45.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1b2ffd640 1 -- 172.21.15.33:0/2429270742 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1ac031c30 con 0x7fe1bc102570 2024-09-16T00:37:45.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2429270742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1bc19c610 con 0x7fe1bc102570 2024-09-16T00:37:45.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1b2ffd640 1 -- 172.21.15.33:0/2429270742 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac035030 con 0x7fe1bc102570 2024-09-16T00:37:45.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.570+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2429270742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1bc1a3f50 con 0x7fe1bc102570 2024-09-16T00:37:45.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.572+0000 7fe1b2ffd640 1 -- 172.21.15.33:0/2429270742 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe1ac042070 con 0x7fe1bc102570 2024-09-16T00:37:45.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1b2ffd640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1a80a3240 0x7fe1a80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c608b740 1 Processor -- start 2024-09-16T00:37:45.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1b2ffd640 1 -- 172.21.15.33:0/2429270742 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe1ac0f04d0 con 0x7fe1bc102570 2024-09-16T00:37:45.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1a80a3240 0x7fe1a80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c608b740 1 -- start start 2024-09-16T00:37:45.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c608b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc65253c0 0x55dfc6523c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c608b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6524190 0x55dfc65245f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c608b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6530e00 0x55dfc65331f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c608b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dfc6487280 con 0x55dfc6530e00 2024-09-16T00:37:45.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c608b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55dfc642eb10 con 0x55dfc6524190 2024-09-16T00:37:45.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c608b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dfc6483240 con 0x55dfc65253c0 2024-09-16T00:37:45.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c124d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6524190 0x55dfc65245f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1a80a3240 0x7fe1a80a5700 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8005fd0 tx=0x7fe1b800b4a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:45.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c124d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6524190 0x55dfc65245f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:52894/0 (socket says 172.21.15.33:52894) 2024-09-16T00:37:45.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c224f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6530e00 0x55dfc65331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.573+0000 7fe1c124d640 1 -- 172.21.15.33:0/3151161757 learned_addr learned my addr 172.21.15.33:0/3151161757 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:45.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.574+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/3151161757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc65253c0 0x55dfc6523c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.574+0000 7fe1c124d640 1 -- 172.21.15.33:0/3151161757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc65253c0 msgr2=0x55dfc6523c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:45.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.574+0000 7fe1c124d640 1 --2- 172.21.15.33:0/3151161757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc65253c0 0x55dfc6523c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.574+0000 7fe1c124d640 1 -- 172.21.15.33:0/3151161757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6530e00 msgr2=0x55dfc65331f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:45.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.574+0000 7fe1c124d640 1 --2- 172.21.15.33:0/3151161757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6530e00 0x55dfc65331f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.574+0000 7fe1c124d640 1 -- 172.21.15.33:0/3151161757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfc643b1f0 con 0x55dfc6524190 2024-09-16T00:37:45.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.574+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/3151161757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc65253c0 0x55dfc6523c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:37:45.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.574+0000 7fe1c224f640 1 --2- 172.21.15.33:0/3151161757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6530e00 0x55dfc65331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:45.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.574+0000 7fe1c124d640 1 --2- 172.21.15.33:0/3151161757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6524190 0x55dfc65245f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0b8a70 tx=0x7fe1ac03b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f90dd3f0118fc4e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:45.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.574+0000 7fe1a7fff640 1 -- 172.21.15.33:0/3151161757 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac049070 con 0x55dfc6524190 2024-09-16T00:37:45.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.574+0000 7fe1a7fff640 1 -- 172.21.15.33:0/3151161757 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1ac0c09c0 con 0x55dfc6524190 2024-09-16T00:37:45.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.574+0000 7fe1a7fff640 1 -- 172.21.15.33:0/3151161757 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac0b8c80 con 0x55dfc6524190 2024-09-16T00:37:45.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.575+0000 7fe1c608b740 1 -- 172.21.15.33:0/3151161757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6524190 msgr2=0x55dfc65245f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:45.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.575+0000 7fe1c608b740 1 --2- 172.21.15.33:0/3151161757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6524190 0x55dfc65245f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0b8a70 tx=0x7fe1ac03b790 comp rx=0 tx=0).stop 2024-09-16T00:37:45.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.575+0000 7fe1c608b740 1 -- 172.21.15.33:0/3151161757 shutdown_connections 2024-09-16T00:37:45.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.575+0000 7fe1c608b740 1 --2- 172.21.15.33:0/3151161757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6530e00 0x55dfc65331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.575+0000 7fe1c608b740 1 --2- 172.21.15.33:0/3151161757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6524190 0x55dfc65245f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.575+0000 7fe1c608b740 1 --2- 172.21.15.33:0/3151161757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc65253c0 0x55dfc6523c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.575+0000 7fe1c608b740 1 -- 172.21.15.33:0/3151161757 >> 172.21.15.33:0/3151161757 conn(0x55dfc6520300 msgr2=0x55dfc65206e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:45.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.575+0000 7fe1c608b740 1 -- 172.21.15.33:0/3151161757 shutdown_connections 2024-09-16T00:37:45.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.575+0000 7fe1c608b740 1 -- 172.21.15.33:0/3151161757 wait complete. 2024-09-16T00:37:45.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.575+0000 7fe1c608b740 1 -- start start 2024-09-16T00:37:45.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.575+0000 7fe1c608b740 1 Processor -- start 2024-09-16T00:37:45.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c608b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6524190 0x55dfc652dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c608b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 0x55dfc652e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c608b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6530e00 0x55dfc652e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c124d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 0x55dfc652e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c124d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 0x55dfc652e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34858/0 (socket says 172.21.15.33:34858) 2024-09-16T00:37:45.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c124d640 1 -- 172.21.15.33:0/232309393 learned_addr learned my addr 172.21.15.33:0/232309393 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:45.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6524190 0x55dfc652dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c608b740 1 -- 172.21.15.33:0/232309393 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dfc6533ad0 con 0x55dfc65253c0 2024-09-16T00:37:45.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c608b740 1 -- 172.21.15.33:0/232309393 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55dfc6441f10 con 0x55dfc6530e00 2024-09-16T00:37:45.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c608b740 1 -- 172.21.15.33:0/232309393 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dfc6533c50 con 0x55dfc6524190 2024-09-16T00:37:45.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c608b740 10 client.0 _ll_register_callbacks cb 0x55dfc640aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:45.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c224f640 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6530e00 0x55dfc652e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c124d640 1 -- 172.21.15.33:0/232309393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6524190 msgr2=0x55dfc652dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:45.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c124d640 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6524190 0x55dfc652dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c124d640 1 -- 172.21.15.33:0/232309393 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6530e00 msgr2=0x55dfc652e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:45.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c124d640 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6530e00 0x55dfc652e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c124d640 1 -- 172.21.15.33:0/232309393 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfc61f86a0 con 0x55dfc65253c0 2024-09-16T00:37:45.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6524190 0x55dfc652dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:45.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.576+0000 7fe1c124d640 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 0x55dfc652e400 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009900 tx=0x7fe1ac0f0350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0e2763504113bb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:45.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:45.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1a57fa640 1 -- 172.21.15.33:0/232309393 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac0d4030 con 0x55dfc65253c0 2024-09-16T00:37:45.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1a57fa640 1 -- 172.21.15.33:0/232309393 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe1ac031de0 con 0x55dfc65253c0 2024-09-16T00:37:45.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1c608b740 1 -- 172.21.15.33:0/232309393 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1ac0c09c0 con 0x55dfc65253c0 2024-09-16T00:37:45.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1a57fa640 1 -- 172.21.15.33:0/232309393 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe1ac049040 con 0x55dfc65253c0 2024-09-16T00:37:45.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1c608b740 10 client.19317 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:45.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1c608b740 1 -- 172.21.15.33:0/232309393 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dfc6533ad0 con 0x55dfc65253c0 2024-09-16T00:37:45.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1a57fa640 1 -- 172.21.15.33:0/232309393 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1ac0c09c0 con 0x55dfc65253c0 2024-09-16T00:37:45.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1a57fa640 1 -- 172.21.15.33:0/232309393 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fe1ac0c4e30 con 0x55dfc65253c0 2024-09-16T00:37:45.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1c608b740 10 client.19317 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:45.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1c608b740 10 client.19317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:37:45.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1c608b740 20 client.19317 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:45.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1c608b740 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dfc65409b0 0x55dfc6542de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1c608b740 4 client.19317 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:37:45.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.577+0000 7fe1c608b740 1 -- 172.21.15.33:0/232309393 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfc64947f0 con 0x55dfc65409b0 2024-09-16T00:37:45.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.578+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dfc65409b0 0x55dfc6542de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.578+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dfc65409b0 0x55dfc6542de0 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28511e2129de4531 server_cookie=ae272a33e1ed8edc in_seq=0 out_seq=0 2024-09-16T00:37:45.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.578+0000 7fe1a57fa640 10 client.19317 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:45.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.578+0000 7fe1a57fa640 1 -- 172.21.15.33:0/232309393 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dfc64947f0 con 0x55dfc65409b0 2024-09-16T00:37:45.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1a57fa640 10 client.19317 handle_command_reply: tid=0 2024-09-16T00:37:45.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1c608b740 1 client.19317 shutdown 2024-09-16T00:37:45.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1c608b740 1 -- 172.21.15.33:0/232309393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 msgr2=0x55dfc652e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:45.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1c608b740 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 0x55dfc652e400 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009900 tx=0x7fe1ac0f0350 comp rx=0 tx=0).stop 2024-09-16T00:37:45.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1c608b740 1 -- 172.21.15.33:0/232309393 shutdown_connections 2024-09-16T00:37:45.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1c608b740 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dfc65409b0 0x55dfc6542de0 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1c608b740 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6530e00 0x55dfc652e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1a57fa640 0 client.19317 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:45.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1c608b740 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 0x55dfc652e400 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1c608b740 1 --2- 172.21.15.33:0/232309393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6524190 0x55dfc652dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1c608b740 1 -- 172.21.15.33:0/232309393 >> 172.21.15.33:0/232309393 conn(0x55dfc6520300 msgr2=0x55dfc6526060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:45.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1c608b740 1 -- 172.21.15.33:0/232309393 shutdown_connections 2024-09-16T00:37:45.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1c608b740 1 -- 172.21.15.33:0/232309393 wait complete. 2024-09-16T00:37:45.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.579+0000 7fe1c608b740 20 client.19317 trim_cache size 0 max 16384 2024-09-16T00:37:45.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.596+0000 7fe1c608b740 1 Processor -- start 2024-09-16T00:37:45.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.596+0000 7fe1c608b740 1 -- start start 2024-09-16T00:37:45.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c608b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6524190 0x55dfc651edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c608b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc65253c0 0x55dfc651f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c608b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6530e00 0x55dfc65486f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c608b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1af6c0 con 0x55dfc6524190 2024-09-16T00:37:45.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c608b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1acf90 con 0x55dfc65253c0 2024-09-16T00:37:45.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c608b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1aade0 con 0x55dfc6530e00 2024-09-16T00:37:45.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c1a4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6524190 0x55dfc651edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c1a4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6524190 0x55dfc651edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34874/0 (socket says 172.21.15.33:34874) 2024-09-16T00:37:45.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c1a4e640 1 -- 172.21.15.33:0/505704381 learned_addr learned my addr 172.21.15.33:0/505704381 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:45.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c224f640 1 --2- 172.21.15.33:0/505704381 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6530e00 0x55dfc65486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c124d640 1 --2- 172.21.15.33:0/505704381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc65253c0 0x55dfc651f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c1a4e640 1 -- 172.21.15.33:0/505704381 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6530e00 msgr2=0x55dfc65486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:45.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/505704381 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6530e00 0x55dfc65486f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c1a4e640 1 -- 172.21.15.33:0/505704381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc65253c0 msgr2=0x55dfc651f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:45.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/505704381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc65253c0 0x55dfc651f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c1a4e640 1 -- 172.21.15.33:0/505704381 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfc65267b0 con 0x55dfc6524190 2024-09-16T00:37:45.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c124d640 1 --2- 172.21.15.33:0/505704381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc65253c0 0x55dfc651f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:37:45.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/505704381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6524190 0x55dfc651edd0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8011e40 tx=0x7fe1b8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a865dbacce1513 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:45.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.597+0000 7fe1a77fe640 1 -- 172.21.15.33:0/505704381 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8020940 con 0x55dfc6524190 2024-09-16T00:37:45.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.598+0000 7fe1a77fe640 1 -- 172.21.15.33:0/505704381 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1b8029420 con 0x55dfc6524190 2024-09-16T00:37:45.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.598+0000 7fe1a77fe640 1 -- 172.21.15.33:0/505704381 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b801e680 con 0x55dfc6524190 2024-09-16T00:37:45.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.598+0000 7fe1c608b740 1 -- 172.21.15.33:0/505704381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6524190 msgr2=0x55dfc651edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:45.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.598+0000 7fe1c608b740 1 --2- 172.21.15.33:0/505704381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6524190 0x55dfc651edd0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8011e40 tx=0x7fe1b8013000 comp rx=0 tx=0).stop 2024-09-16T00:37:45.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.598+0000 7fe1c608b740 1 -- 172.21.15.33:0/505704381 shutdown_connections 2024-09-16T00:37:45.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.598+0000 7fe1c608b740 1 --2- 172.21.15.33:0/505704381 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6530e00 0x55dfc65486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.598+0000 7fe1c608b740 1 --2- 172.21.15.33:0/505704381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc65253c0 0x55dfc651f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.598+0000 7fe1c608b740 1 --2- 172.21.15.33:0/505704381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc6524190 0x55dfc651edd0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.598+0000 7fe1c608b740 1 -- 172.21.15.33:0/505704381 >> 172.21.15.33:0/505704381 conn(0x55dfc6520300 msgr2=0x55dfc652c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:45.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.598+0000 7fe1c608b740 1 -- 172.21.15.33:0/505704381 shutdown_connections 2024-09-16T00:37:45.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.598+0000 7fe1c608b740 1 -- 172.21.15.33:0/505704381 wait complete. 2024-09-16T00:37:45.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c608b740 1 -- start start 2024-09-16T00:37:45.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c608b740 1 Processor -- start 2024-09-16T00:37:45.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c608b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6524190 0x55dfc65393b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c608b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 0x55dfc6535e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c608b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6530e00 0x55dfc65363b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c608b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1ab450 con 0x55dfc65253c0 2024-09-16T00:37:45.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c608b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1ac8c0 con 0x55dfc6530e00 2024-09-16T00:37:45.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c608b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1abb20 con 0x55dfc6524190 2024-09-16T00:37:45.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c608b740 10 client.0 _ll_register_callbacks cb 0x7fe1ac0c0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:45.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c124d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 0x55dfc6535e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c124d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 0x55dfc6535e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34888/0 (socket says 172.21.15.33:34888) 2024-09-16T00:37:45.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c124d640 1 -- 172.21.15.33:0/2298958430 learned_addr learned my addr 172.21.15.33:0/2298958430 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6524190 0x55dfc65393b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.599+0000 7fe1c224f640 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6530e00 0x55dfc65363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1c124d640 1 -- 172.21.15.33:0/2298958430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6524190 msgr2=0x55dfc65393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1c124d640 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6524190 0x55dfc65393b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1c124d640 1 -- 172.21.15.33:0/2298958430 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6530e00 msgr2=0x55dfc65363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1c124d640 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6530e00 0x55dfc65363b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1c124d640 1 -- 172.21.15.33:0/2298958430 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfc65368f0 con 0x55dfc65253c0 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1c224f640 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6530e00 0x55dfc65363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1c124d640 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 0x55dfc6535e70 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0b8a70 tx=0x7fe1ac03b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad35609561dd2a93 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1a57fa640 1 -- 172.21.15.33:0/2298958430 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac0d4070 con 0x55dfc65253c0 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1c608b740 1 -- 172.21.15.33:0/2298958430 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1b8029420 con 0x55dfc65253c0 2024-09-16T00:37:45.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1a57fa640 1 -- 172.21.15.33:0/2298958430 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe1ac0c4df0 con 0x55dfc65253c0 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1a57fa640 1 -- 172.21.15.33:0/2298958430 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe1ac005d50 con 0x55dfc65253c0 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1c608b740 10 client.19323 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1c608b740 1 -- 172.21.15.33:0/2298958430 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dfc65368f0 con 0x55dfc65253c0 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.600+0000 7fe1a57fa640 1 -- 172.21.15.33:0/2298958430 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1ac0a4020 con 0x55dfc65253c0 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.601+0000 7fe1a57fa640 1 -- 172.21.15.33:0/2298958430 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fe1ac0d5290 con 0x55dfc65253c0 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.601+0000 7fe1c608b740 10 client.19323 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.601+0000 7fe1c608b740 10 client.19323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.601+0000 7fe1c608b740 20 client.19323 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.601+0000 7fe1c608b740 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dfc65409b0 0x55dfc65ae740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.601+0000 7fe1c608b740 4 client.19323 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.601+0000 7fe1c608b740 1 -- 172.21.15.33:0/2298958430 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dfc648ed00 con 0x55dfc65409b0 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.601+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dfc65409b0 0x55dfc65ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.601+0000 7fe1c1a4e640 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dfc65409b0 0x55dfc65ae740 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=536c3a047efa6ef8 server_cookie=91eb8955b572d6 in_seq=0 out_seq=0 2024-09-16T00:37:45.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.601+0000 7fe1a57fa640 10 client.19323 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:45.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.602+0000 7fe1a57fa640 1 -- 172.21.15.33:0/2298958430 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dfc648ed00 con 0x55dfc65409b0 2024-09-16T00:37:45.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.602+0000 7fe1a57fa640 10 client.19323 handle_command_reply: tid=0 2024-09-16T00:37:45.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.602+0000 7fe1c608b740 1 client.19323 shutdown 2024-09-16T00:37:45.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.602+0000 7fe1c608b740 1 -- 172.21.15.33:0/2298958430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 msgr2=0x55dfc6535e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:45.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.602+0000 7fe1c608b740 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 0x55dfc6535e70 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0b8a70 tx=0x7fe1ac03b790 comp rx=0 tx=0).stop 2024-09-16T00:37:45.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.602+0000 7fe1c608b740 1 -- 172.21.15.33:0/2298958430 shutdown_connections 2024-09-16T00:37:45.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.602+0000 7fe1c608b740 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dfc65409b0 0x55dfc65ae740 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.602+0000 7fe1c608b740 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dfc6530e00 0x55dfc65363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.602+0000 7fe1c608b740 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfc65253c0 0x55dfc6535e70 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.602+0000 7fe1a57fa640 0 client.19323 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:45.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.602+0000 7fe1c608b740 1 --2- 172.21.15.33:0/2298958430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dfc6524190 0x55dfc65393b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.602+0000 7fe1c608b740 1 -- 172.21.15.33:0/2298958430 >> 172.21.15.33:0/2298958430 conn(0x55dfc6520300 msgr2=0x55dfc65257a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:45.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c608b740 1 -- 172.21.15.33:0/2298958430 shutdown_connections 2024-09-16T00:37:45.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c608b740 1 -- 172.21.15.33:0/2298958430 wait complete. 2024-09-16T00:37:45.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c608b740 20 client.19323 trim_cache size 0 max 16384 2024-09-16T00:37:45.608 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:37:45.608 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:37:45.608 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "7af3c5c0-0f3e-4139-92aa-35ac00fbce59", 2024-09-16T00:37:45.608 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:37:45.608 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:37:45.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2429270742 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1a80a3240 msgr2=0x7fe1a80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:45.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c3cb2640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1a80a3240 0x7fe1a80a5700 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8005fd0 tx=0x7fe1b800b4a0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2429270742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1bc102570 msgr2=0x7fe1bc197910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:45.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c3cb2640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1bc102570 0x7fe1bc197910 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac005940 tx=0x7fe1ac003c10 comp rx=0 tx=0).stop 2024-09-16T00:37:45.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2429270742 shutdown_connections 2024-09-16T00:37:45.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c3cb2640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe1a80a3240 0x7fe1a80a5700 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c3cb2640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc102ef0 0x7fe1bc19bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c3cb2640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1bc102570 0x7fe1bc197910 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c3cb2640 1 --2- 172.21.15.33:0/2429270742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1bc100e20 0x7fe1bc1973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:45.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2429270742 >> 172.21.15.33:0/2429270742 conn(0x7fe1bc0fc7d0 msgr2=0x7fe1bc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:45.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2429270742 shutdown_connections 2024-09-16T00:37:45.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:45.603+0000 7fe1c3cb2640 1 -- 172.21.15.33:0/2429270742 wait complete. 2024-09-16T00:37:45.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:45 smithi033 ceph-mon[31243]: pgmap v968: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 12 op/s 2024-09-16T00:37:45.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:45 smithi083 ceph-mon[35242]: pgmap v968: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 12 op/s 2024-09-16T00:37:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:45 smithi060 ceph-mon[35185]: pgmap v968: 129 pgs: 129 active+clean; 611 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 12 op/s 2024-09-16T00:37:46.090 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 16.74 MB/sec execute 292 sec latency 22127.254 ms 2024-09-16T00:37:46.540 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:37:46.798 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:46.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:46 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:37:46.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:46.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:46.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:46.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:46.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:46.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:46.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:46.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:46 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:46.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:46 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:46.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:46 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:37:46.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:46.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:46 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:46 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:46.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:46 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:37:46.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:46.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:46.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:46 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:46.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:46 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.090 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 16.68 MB/sec execute 293 sec latency 23127.311 ms 2024-09-16T00:37:47.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.274+0000 7f2f5eca0640 1 -- 172.21.15.33:0/3960880546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f58101de0 msgr2=0x7f2f581021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:47.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.274+0000 7f2f5eca0640 1 --2- 172.21.15.33:0/3960880546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f58101de0 0x7f2f581021c0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c009900 tx=0x7f2f4c02a410 comp rx=0 tx=0).stop 2024-09-16T00:37:47.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.274+0000 7f2f5eca0640 1 -- 172.21.15.33:0/3960880546 shutdown_connections 2024-09-16T00:37:47.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.274+0000 7f2f5eca0640 1 --2- 172.21.15.33:0/3960880546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f58106f10 0x7f2f5810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.274+0000 7f2f5eca0640 1 --2- 172.21.15.33:0/3960880546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f58102790 0x7f2f581067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.274+0000 7f2f5eca0640 1 --2- 172.21.15.33:0/3960880546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f58101de0 0x7f2f581021c0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.274+0000 7f2f5eca0640 1 -- 172.21.15.33:0/3960880546 >> 172.21.15.33:0/3960880546 conn(0x7f2f580fd8c0 msgr2=0x7f2f580ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:47.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.274+0000 7f2f5eca0640 1 -- 172.21.15.33:0/3960880546 shutdown_connections 2024-09-16T00:37:47.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.274+0000 7f2f5eca0640 1 -- 172.21.15.33:0/3960880546 wait complete. 2024-09-16T00:37:47.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5eca0640 1 Processor -- start 2024-09-16T00:37:47.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5eca0640 1 -- start start 2024-09-16T00:37:47.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5eca0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f58101de0 0x7f2f58198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5eca0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f58102790 0x7f2f581989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5eca0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f58106f10 0x7f2f5819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5eca0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2f58111f90 con 0x7f2f58106f10 2024-09-16T00:37:47.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5eca0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2f58111e10 con 0x7f2f58101de0 2024-09-16T00:37:47.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5eca0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2f58112110 con 0x7f2f58102790 2024-09-16T00:37:47.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5d23d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f58106f10 0x7f2f5819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5d23d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f58106f10 0x7f2f5819cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34902/0 (socket says 172.21.15.33:34902) 2024-09-16T00:37:47.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5d23d640 1 -- 172.21.15.33:0/2621037926 learned_addr learned my addr 172.21.15.33:0/2621037926 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:47.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f58101de0 0x7f2f58198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f57fff640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f58102790 0x7f2f581989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5d23d640 1 -- 172.21.15.33:0/2621037926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f58102790 msgr2=0x7f2f581989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5d23d640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f58102790 0x7f2f581989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.275+0000 7f2f5d23d640 1 -- 172.21.15.33:0/2621037926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f58101de0 msgr2=0x7f2f58198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.276+0000 7f2f5d23d640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f58101de0 0x7f2f58198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.276+0000 7f2f5d23d640 1 -- 172.21.15.33:0/2621037926 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f5819d440 con 0x7f2f58106f10 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.276+0000 7f2f5d23d640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f58106f10 0x7f2f5819cd60 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000b970 tx=0x7f2f5000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.276+0000 7f2f55ffb640 1 -- 172.21.15.33:0/2621037926 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f50019070 con 0x7f2f58106f10 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.276+0000 7f2f55ffb640 1 -- 172.21.15.33:0/2621037926 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f5000cb40 con 0x7f2f58106f10 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.276+0000 7f2f5eca0640 1 -- 172.21.15.33:0/2621037926 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f5819d730 con 0x7f2f58106f10 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.276+0000 7f2f55ffb640 1 -- 172.21.15.33:0/2621037926 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f50007650 con 0x7f2f58106f10 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.276+0000 7f2f5eca0640 1 -- 172.21.15.33:0/2621037926 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f581a5010 con 0x7f2f58106f10 2024-09-16T00:37:47.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.278+0000 7f2f55ffb640 1 -- 172.21.15.33:0/2621037926 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2f5001c070 con 0x7f2f58106f10 2024-09-16T00:37:47.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f61079740 1 Processor -- start 2024-09-16T00:37:47.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f61079740 1 -- start start 2024-09-16T00:37:47.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f61079740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8d4a0 0x556334d8d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f61079740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 0x556334d98140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f61079740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d98680 0x556334d9aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f61079740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556334cf32e0 con 0x556334d8ddc0 2024-09-16T00:37:47.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f61079740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556334c9ace0 con 0x556334d8d4a0 2024-09-16T00:37:47.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f61079740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556334cef2a0 con 0x556334d98680 2024-09-16T00:37:47.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f55ffb640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2f480a3240 0x7f2f480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f5ca3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 0x556334d98140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f5ca3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 0x556334d98140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34918/0 (socket says 172.21.15.33:34918) 2024-09-16T00:37:47.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f5ca3c640 1 -- 172.21.15.33:0/1314536462 learned_addr learned my addr 172.21.15.33:0/1314536462 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:47.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f5d23d640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2f480a3240 0x7f2f480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f57fff640 1 --2- 172.21.15.33:0/1314536462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d98680 0x556334d9aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f55ffb640 1 -- 172.21.15.33:0/2621037926 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2f500cf2f0 con 0x7f2f58106f10 2024-09-16T00:37:47.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f57fff640 1 --2- 172.21.15.33:0/1314536462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8d4a0 0x556334d8d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f5ca3c640 1 -- 172.21.15.33:0/1314536462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d98680 msgr2=0x556334d9aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:47.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/1314536462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d98680 0x556334d9aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f5ca3c640 1 -- 172.21.15.33:0/1314536462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8d4a0 msgr2=0x556334d8d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:47.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/1314536462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8d4a0 0x556334d8d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f5ca3c640 1 -- 172.21.15.33:0/1314536462 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556334c7f440 con 0x556334d8ddc0 2024-09-16T00:37:47.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.279+0000 7f2f57fff640 1 --2- 172.21.15.33:0/1314536462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8d4a0 0x556334d8d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:37:47.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.280+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/1314536462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 0x556334d98140 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c009900 tx=0x7f2f4c031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d926e0c6aa463dc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:47.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.280+0000 7f2f5d23d640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2f480a3240 0x7f2f480a5700 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2f500979a0 tx=0x7f2f50065f80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.280+0000 7f2f42ffd640 1 -- 172.21.15.33:0/1314536462 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c040070 con 0x556334d8ddc0 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.280+0000 7f2f42ffd640 1 -- 172.21.15.33:0/1314536462 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f4c005020 con 0x556334d8ddc0 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.280+0000 7f2f42ffd640 1 -- 172.21.15.33:0/1314536462 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c035040 con 0x556334d8ddc0 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.280+0000 7f2f61079740 1 -- 172.21.15.33:0/1314536462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 msgr2=0x556334d98140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.280+0000 7f2f61079740 1 --2- 172.21.15.33:0/1314536462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 0x556334d98140 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c009900 tx=0x7f2f4c031630 comp rx=0 tx=0).stop 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.281+0000 7f2f61079740 1 -- 172.21.15.33:0/1314536462 shutdown_connections 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.281+0000 7f2f61079740 1 --2- 172.21.15.33:0/1314536462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d98680 0x556334d9aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.281+0000 7f2f61079740 1 --2- 172.21.15.33:0/1314536462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 0x556334d98140 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.281+0000 7f2f61079740 1 --2- 172.21.15.33:0/1314536462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8d4a0 0x556334d8d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.281+0000 7f2f61079740 1 -- 172.21.15.33:0/1314536462 >> 172.21.15.33:0/1314536462 conn(0x556334d89a30 msgr2=0x556334d89e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.281+0000 7f2f61079740 1 -- 172.21.15.33:0/1314536462 shutdown_connections 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.281+0000 7f2f61079740 1 -- 172.21.15.33:0/1314536462 wait complete. 2024-09-16T00:37:47.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.281+0000 7f2f61079740 1 -- start start 2024-09-16T00:37:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.281+0000 7f2f61079740 1 Processor -- start 2024-09-16T00:37:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f61079740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8d4a0 0x556334d97f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f61079740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8ddc0 0x556334d951d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f61079740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d98680 0x556334d95710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f61079740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556334d9b250 con 0x556334d8d4a0 2024-09-16T00:37:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f61079740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556334cae050 con 0x556334d8ddc0 2024-09-16T00:37:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f61079740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556334d9b3d0 con 0x556334d98680 2024-09-16T00:37:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f61079740 10 client.0 _ll_register_callbacks cb 0x556334cee160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f57fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8d4a0 0x556334d97f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f57fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8d4a0 0x556334d97f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34924/0 (socket says 172.21.15.33:34924) 2024-09-16T00:37:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f57fff640 1 -- 172.21.15.33:0/2890080067 learned_addr learned my addr 172.21.15.33:0/2890080067 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8ddc0 0x556334d951d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f57fff640 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d98680 0x556334d95710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f57fff640 1 -- 172.21.15.33:0/2890080067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d98680 msgr2=0x556334d95710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:47.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f57fff640 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d98680 0x556334d95710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f57fff640 1 -- 172.21.15.33:0/2890080067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8ddc0 msgr2=0x556334d951d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:47.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f57fff640 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8ddc0 0x556334d951d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f57fff640 1 -- 172.21.15.33:0/2890080067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556334a643c0 con 0x556334d8d4a0 2024-09-16T00:37:47.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8ddc0 0x556334d951d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:37:47.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f57fff640 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8d4a0 0x556334d97f20 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400cfd0 tx=0x7f2f4400cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=462ce3b3417fb952 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:47.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f37fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:47.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f37fff640 1 -- 172.21.15.33:0/2890080067 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44014280 con 0x556334d8d4a0 2024-09-16T00:37:47.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f37fff640 1 -- 172.21.15.33:0/2890080067 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2f440040f0 con 0x556334d8d4a0 2024-09-16T00:37:47.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.282+0000 7f2f61079740 1 -- 172.21.15.33:0/2890080067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f4c005020 con 0x556334d8d4a0 2024-09-16T00:37:47.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.283+0000 7f2f37fff640 1 -- 172.21.15.33:0/2890080067 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2f440042f0 con 0x556334d8d4a0 2024-09-16T00:37:47.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.283+0000 7f2f61079740 10 client.19335 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:47.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.283+0000 7f2f61079740 1 -- 172.21.15.33:0/2890080067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556334da6b70 con 0x556334d8d4a0 2024-09-16T00:37:47.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.283+0000 7f2f37fff640 1 -- 172.21.15.33:0/2890080067 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f4c005020 con 0x556334d8d4a0 2024-09-16T00:37:47.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.283+0000 7f2f37fff640 1 -- 172.21.15.33:0/2890080067 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2f44012550 con 0x556334d8d4a0 2024-09-16T00:37:47.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.283+0000 7f2f61079740 10 client.19335 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:47.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.283+0000 7f2f61079740 10 client.19335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:37:47.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.283+0000 7f2f61079740 20 client.19335 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:47.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.283+0000 7f2f61079740 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556334da8030 0x556334daa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.283+0000 7f2f61079740 4 client.19335 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.283+0000 7f2f61079740 1 -- 172.21.15.33:0/2890080067 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556334d00850 con 0x556334da8030 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.283+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556334da8030 0x556334daa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.284+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556334da8030 0x556334daa440 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e66057e1e4e885 server_cookie=75a9cb94e643d902 in_seq=0 out_seq=0 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.284+0000 7f2f37fff640 10 client.19335 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f37fff640 1 -- 172.21.15.33:0/2890080067 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556334d00850 con 0x556334da8030 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f37fff640 10 client.19335 handle_command_reply: tid=0 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f61079740 1 client.19335 shutdown 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f61079740 1 -- 172.21.15.33:0/2890080067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8d4a0 msgr2=0x556334d97f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f61079740 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8d4a0 0x556334d97f20 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400cfd0 tx=0x7f2f4400cb50 comp rx=0 tx=0).stop 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f61079740 1 -- 172.21.15.33:0/2890080067 shutdown_connections 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f61079740 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556334da8030 0x556334daa440 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f61079740 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d98680 0x556334d95710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f61079740 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8ddc0 0x556334d951d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f37fff640 0 client.19335 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:47.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f61079740 1 --2- 172.21.15.33:0/2890080067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8d4a0 0x556334d97f20 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f61079740 1 -- 172.21.15.33:0/2890080067 >> 172.21.15.33:0/2890080067 conn(0x556334d89a30 msgr2=0x556334d88b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:47.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f61079740 1 -- 172.21.15.33:0/2890080067 shutdown_connections 2024-09-16T00:37:47.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f61079740 1 -- 172.21.15.33:0/2890080067 wait complete. 2024-09-16T00:37:47.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.285+0000 7f2f61079740 20 client.19335 trim_cache size 0 max 16384 2024-09-16T00:37:47.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.302+0000 7f2f61079740 1 Processor -- start 2024-09-16T00:37:47.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f61079740 1 -- start start 2024-09-16T00:37:47.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f61079740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d8d4a0 0x556334da44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f61079740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 0x556334d93aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f61079740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d98680 0x556334dafd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f61079740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2f581b1a60 con 0x556334d8ddc0 2024-09-16T00:37:47.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f61079740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2f581af330 con 0x556334d98680 2024-09-16T00:37:47.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f61079740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2f581ad180 con 0x556334d8d4a0 2024-09-16T00:37:47.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f5ca3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 0x556334d93aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f5ca3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 0x556334d93aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34932/0 (socket says 172.21.15.33:34932) 2024-09-16T00:37:47.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f5ca3c640 1 -- 172.21.15.33:0/3148632518 learned_addr learned my addr 172.21.15.33:0/3148632518 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:47.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f57fff640 1 --2- 172.21.15.33:0/3148632518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d8d4a0 0x556334da44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f57fff640 1 --2- 172.21.15.33:0/3148632518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d98680 0x556334dafd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f5ca3c640 1 -- 172.21.15.33:0/3148632518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d8d4a0 msgr2=0x556334da44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/3148632518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d8d4a0 0x556334da44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f5ca3c640 1 -- 172.21.15.33:0/3148632518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d98680 msgr2=0x556334dafd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/3148632518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d98680 0x556334dafd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f5ca3c640 1 -- 172.21.15.33:0/3148632518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556334d94180 con 0x556334d8ddc0 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f57fff640 1 --2- 172.21.15.33:0/3148632518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d98680 0x556334dafd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.303+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/3148632518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 0x556334d93aa0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c031bf0 tx=0x7f2f4c0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12966e6e5a3bd48 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.304+0000 7f2f427fc640 1 -- 172.21.15.33:0/3148632518 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c054020 con 0x556334d8ddc0 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.304+0000 7f2f427fc640 1 -- 172.21.15.33:0/3148632518 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f4c004c30 con 0x556334d8ddc0 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.304+0000 7f2f427fc640 1 -- 172.21.15.33:0/3148632518 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c040040 con 0x556334d8ddc0 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.304+0000 7f2f61079740 1 -- 172.21.15.33:0/3148632518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 msgr2=0x556334d93aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.304+0000 7f2f61079740 1 --2- 172.21.15.33:0/3148632518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 0x556334d93aa0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c031bf0 tx=0x7f2f4c0058d0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.304+0000 7f2f61079740 1 -- 172.21.15.33:0/3148632518 shutdown_connections 2024-09-16T00:37:47.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.304+0000 7f2f61079740 1 --2- 172.21.15.33:0/3148632518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d98680 0x556334dafd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.304+0000 7f2f61079740 1 --2- 172.21.15.33:0/3148632518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d8ddc0 0x556334d93aa0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.304+0000 7f2f61079740 1 --2- 172.21.15.33:0/3148632518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d8d4a0 0x556334da44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.304+0000 7f2f61079740 1 -- 172.21.15.33:0/3148632518 >> 172.21.15.33:0/3148632518 conn(0x556334d89a30 msgr2=0x556334d907e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:47.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.304+0000 7f2f61079740 1 -- 172.21.15.33:0/3148632518 shutdown_connections 2024-09-16T00:37:47.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.304+0000 7f2f61079740 1 -- 172.21.15.33:0/3148632518 wait complete. 2024-09-16T00:37:47.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f61079740 1 -- start start 2024-09-16T00:37:47.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f61079740 1 Processor -- start 2024-09-16T00:37:47.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f61079740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d8d4a0 0x556334da0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f61079740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8ddc0 0x556334d9c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f61079740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d98680 0x556334d9c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f61079740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2f581ad7f0 con 0x556334d98680 2024-09-16T00:37:47.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f61079740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2f581aec60 con 0x556334d8ddc0 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f61079740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2f581adec0 con 0x556334d8d4a0 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f61079740 10 client.0 _ll_register_callbacks cb 0x7f2f4400cd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f57fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d98680 0x556334d9c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f57fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d98680 0x556334d9c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34940/0 (socket says 172.21.15.33:34940) 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f57fff640 1 -- 172.21.15.33:0/3561975823 learned_addr learned my addr 172.21.15.33:0/3561975823 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8ddc0 0x556334d9c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.305+0000 7f2f57fff640 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d8d4a0 0x556334da0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f57fff640 1 -- 172.21.15.33:0/3561975823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d8d4a0 msgr2=0x556334da0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f57fff640 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d8d4a0 0x556334da0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f57fff640 1 -- 172.21.15.33:0/3561975823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8ddc0 msgr2=0x556334d9c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f57fff640 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8ddc0 0x556334d9c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f57fff640 1 -- 172.21.15.33:0/3561975823 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556334d9cef0 con 0x556334d98680 2024-09-16T00:37:47.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8ddc0 0x556334d9c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f57fff640 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d98680 0x556334d9c6c0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400cfd0 tx=0x7f2f4400cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e959b42a5b2eddd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f37fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f37fff640 1 -- 172.21.15.33:0/3561975823 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44004490 con 0x556334d98680 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f37fff640 1 -- 172.21.15.33:0/3561975823 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2f4401d020 con 0x556334d98680 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f61079740 1 -- 172.21.15.33:0/3561975823 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f4c004c30 con 0x556334d98680 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f37fff640 1 -- 172.21.15.33:0/3561975823 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2f4403b050 con 0x556334d98680 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f61079740 10 client.19341 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f61079740 1 -- 172.21.15.33:0/3561975823 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556334e12d00 con 0x556334d98680 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.306+0000 7f2f37fff640 1 -- 172.21.15.33:0/3561975823 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f4c004c30 con 0x556334d98680 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f37fff640 1 -- 172.21.15.33:0/3561975823 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2f44003ad0 con 0x556334d98680 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f61079740 10 client.19341 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f61079740 10 client.19341 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:37:47.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f61079740 20 client.19341 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f61079740 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556334e133f0 0x556334e15810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f61079740 4 client.19341 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:37:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f61079740 1 -- 172.21.15.33:0/3561975823 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556334cfad60 con 0x556334e133f0 2024-09-16T00:37:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556334e133f0 0x556334e15810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f5ca3c640 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556334e133f0 0x556334e15810 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6656c668fc9ec692 server_cookie=2d8d39b9ef5d8fca in_seq=0 out_seq=0 2024-09-16T00:37:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f37fff640 10 client.19341 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f37fff640 1 -- 172.21.15.33:0/3561975823 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556334cfad60 con 0x556334e133f0 2024-09-16T00:37:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f37fff640 10 client.19341 handle_command_reply: tid=0 2024-09-16T00:37:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.307+0000 7f2f61079740 1 client.19341 shutdown 2024-09-16T00:37:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f61079740 1 -- 172.21.15.33:0/3561975823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d98680 msgr2=0x556334d9c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f61079740 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d98680 0x556334d9c6c0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400cfd0 tx=0x7f2f4400cb50 comp rx=0 tx=0).stop 2024-09-16T00:37:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f61079740 1 -- 172.21.15.33:0/3561975823 shutdown_connections 2024-09-16T00:37:47.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f61079740 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x556334e133f0 0x556334e15810 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f61079740 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556334d98680 0x556334d9c6c0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f61079740 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556334d8ddc0 0x556334d9c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f61079740 1 --2- 172.21.15.33:0/3561975823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556334d8d4a0 0x556334da0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f61079740 1 -- 172.21.15.33:0/3561975823 >> 172.21.15.33:0/3561975823 conn(0x556334d89a30 msgr2=0x556334d8eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:47.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f37fff640 0 client.19341 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:47.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f61079740 1 -- 172.21.15.33:0/3561975823 shutdown_connections 2024-09-16T00:37:47.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f61079740 1 -- 172.21.15.33:0/3561975823 wait complete. 2024-09-16T00:37:47.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f61079740 20 client.19341 trim_cache size 0 max 16384 2024-09-16T00:37:47.317 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:37:47.318 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:37:47.318 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:37:47.318 INFO:teuthology.orchestra.run.smithi033.stdout: "7af3c5c0-0f3e-4139-92aa-35ac00fbce59": { 2024-09-16T00:37:47.318 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:37:47.318 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "7af3c5c0-0f3e-4139-92aa-35ac00fbce59", 2024-09-16T00:37:47.318 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:37:47.318 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:47.318 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:47.318 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:37:47.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f5eca0640 1 -- 172.21.15.33:0/2621037926 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2f480a3240 msgr2=0x7f2f480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:47.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f5eca0640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2f480a3240 0x7f2f480a5700 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2f500979a0 tx=0x7f2f50065f80 comp rx=0 tx=0).stop 2024-09-16T00:37:47.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f5eca0640 1 -- 172.21.15.33:0/2621037926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f58106f10 msgr2=0x7f2f5819cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:47.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.308+0000 7f2f5eca0640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f58106f10 0x7f2f5819cd60 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000b970 tx=0x7f2f5000be30 comp rx=0 tx=0).stop 2024-09-16T00:37:47.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.309+0000 7f2f5eca0640 1 -- 172.21.15.33:0/2621037926 shutdown_connections 2024-09-16T00:37:47.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.309+0000 7f2f5eca0640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2f480a3240 0x7f2f480a5700 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.309+0000 7f2f5eca0640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f58106f10 0x7f2f5819cd60 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.309+0000 7f2f5eca0640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f58102790 0x7f2f581989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.309+0000 7f2f5eca0640 1 --2- 172.21.15.33:0/2621037926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f58101de0 0x7f2f58198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:47.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.309+0000 7f2f5eca0640 1 -- 172.21.15.33:0/2621037926 >> 172.21.15.33:0/2621037926 conn(0x7f2f580fd8c0 msgr2=0x7f2f580fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:47.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.309+0000 7f2f5eca0640 1 -- 172.21.15.33:0/2621037926 shutdown_connections 2024-09-16T00:37:47.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:47.309+0000 7f2f5eca0640 1 -- 172.21.15.33:0/2621037926 wait complete. 2024-09-16T00:37:47.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:37:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:47 smithi033 ceph-mon[31243]: pgmap v969: 129 pgs: 129 active+clean; 614 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 437 KiB/s wr, 18 op/s 2024-09-16T00:37:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:37:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:47 smithi083 ceph-mon[35242]: pgmap v969: 129 pgs: 129 active+clean; 614 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 437 KiB/s wr, 18 op/s 2024-09-16T00:37:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:47.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:47.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:47.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:47.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:37:47.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:37:47.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:47 smithi060 ceph-mon[35185]: pgmap v969: 129 pgs: 129 active+clean; 614 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 437 KiB/s wr, 18 op/s 2024-09-16T00:37:48.090 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 16.63 MB/sec execute 294 sec latency 24127.655 ms 2024-09-16T00:37:48.223 INFO:tasks.cephfs.filesystem:scrub status for tag:7af3c5c0-0f3e-4139-92aa-35ac00fbce59 - {'path': '/', 'tag': '7af3c5c0-0f3e-4139-92aa-35ac00fbce59', 'options': 'recursive,force'} 2024-09-16T00:37:49.076 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:37:49.090 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 16.57 MB/sec execute 295 sec latency 25127.726 ms 2024-09-16T00:37:49.346 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:49.750 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:49 smithi033 ceph-mon[31243]: pgmap v970: 129 pgs: 129 active+clean; 614 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 413 KiB/s wr, 15 op/s 2024-09-16T00:37:49.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.832+0000 7fc58cdcf640 1 -- 172.21.15.33:0/1298822351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc588101420 msgr2=0x7fc58810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:49.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.832+0000 7fc58cdcf640 1 --2- 172.21.15.33:0/1298822351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc588101420 0x7fc58810d950 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fc580009cd0 tx=0x7fc58002a8e0 comp rx=0 tx=0).stop 2024-09-16T00:37:49.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.833+0000 7fc58cdcf640 1 -- 172.21.15.33:0/1298822351 shutdown_connections 2024-09-16T00:37:49.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.833+0000 7fc58cdcf640 1 --2- 172.21.15.33:0/1298822351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc588101420 0x7fc58810d950 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:49.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.833+0000 7fc58cdcf640 1 --2- 172.21.15.33:0/1298822351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc588100a80 0x7fc588100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:49.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.833+0000 7fc58cdcf640 1 --2- 172.21.15.33:0/1298822351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc588106a80 0x7fc588106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:49.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.833+0000 7fc58cdcf640 1 -- 172.21.15.33:0/1298822351 >> 172.21.15.33:0/1298822351 conn(0x7fc5880fc7a0 msgr2=0x7fc5880febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:49.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.833+0000 7fc58cdcf640 1 -- 172.21.15.33:0/1298822351 shutdown_connections 2024-09-16T00:37:49.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.833+0000 7fc58cdcf640 1 -- 172.21.15.33:0/1298822351 wait complete. 2024-09-16T00:37:49.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.833+0000 7fc58cdcf640 1 Processor -- start 2024-09-16T00:37:49.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.833+0000 7fc58cdcf640 1 -- start start 2024-09-16T00:37:49.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc58cdcf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc588100a80 0x7fc58810e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:49.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc58cdcf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc588101420 0x7fc58810e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:49.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc58cdcf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc588106a80 0x7fc58810ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:49.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc58cdcf640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc588112020 con 0x7fc588106a80 2024-09-16T00:37:49.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc58cdcf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc588111ea0 con 0x7fc588100a80 2024-09-16T00:37:49.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc58cdcf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc5881121a0 con 0x7fc588101420 2024-09-16T00:37:49.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc586d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc588106a80 0x7fc58810ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc58659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc588100a80 0x7fc58810e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc586d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc588106a80 0x7fc58810ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34948/0 (socket says 172.21.15.33:34948) 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc586d9d640 1 -- 172.21.15.33:0/3676982249 learned_addr learned my addr 172.21.15.33:0/3676982249 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc585d9b640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc588101420 0x7fc58810e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc586d9d640 1 -- 172.21.15.33:0/3676982249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc588101420 msgr2=0x7fc58810e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc586d9d640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc588101420 0x7fc58810e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc586d9d640 1 -- 172.21.15.33:0/3676982249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc588100a80 msgr2=0x7fc58810e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc586d9d640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc588100a80 0x7fc58810e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc586d9d640 1 -- 172.21.15.33:0/3676982249 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5881a6a60 con 0x7fc588106a80 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.834+0000 7fc586d9d640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc588106a80 0x7fc58810ee60 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fc580006010 tx=0x7fc580003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.835+0000 7fc5777fe640 1 -- 172.21.15.33:0/3676982249 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580004450 con 0x7fc588106a80 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.835+0000 7fc5777fe640 1 -- 172.21.15.33:0/3676982249 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc580003e10 con 0x7fc588106a80 2024-09-16T00:37:49.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.835+0000 7fc58cdcf640 1 -- 172.21.15.33:0/3676982249 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5881a6cf0 con 0x7fc588106a80 2024-09-16T00:37:49.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.835+0000 7fc5777fe640 1 -- 172.21.15.33:0/3676982249 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580046480 con 0x7fc588106a80 2024-09-16T00:37:49.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.835+0000 7fc58cdcf640 1 -- 172.21.15.33:0/3676982249 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5881a7230 con 0x7fc588106a80 2024-09-16T00:37:49.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.836+0000 7fc58cdcf640 1 -- 172.21.15.33:0/3676982249 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc588100f60 con 0x7fc588106a80 2024-09-16T00:37:49.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.837+0000 7fc5777fe640 1 -- 172.21.15.33:0/3676982249 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc5800045f0 con 0x7fc588106a80 2024-09-16T00:37:49.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.838+0000 7fc5777fe640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc5700a3240 0x7fc5700a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:49.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.838+0000 7fc5777fe640 1 -- 172.21.15.33:0/3676982249 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc58002e030 con 0x7fc588106a80 2024-09-16T00:37:49.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.838+0000 7fc58659c640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc5700a3240 0x7fc5700a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:49.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.838+0000 7fc58659c640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc5700a3240 0x7fc5700a5700 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc578009990 tx=0x7fc578009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:49.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:49.841+0000 7fc5777fe640 1 -- 172.21.15.33:0/3676982249 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc58004b060 con 0x7fc588106a80 2024-09-16T00:37:49.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:49 smithi083 ceph-mon[35242]: pgmap v970: 129 pgs: 129 active+clean; 614 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 413 KiB/s wr, 15 op/s 2024-09-16T00:37:49.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:49 smithi060 ceph-mon[35185]: pgmap v970: 129 pgs: 129 active+clean; 614 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 413 KiB/s wr, 15 op/s 2024-09-16T00:37:50.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.053+0000 7fc58cdcf640 1 -- 172.21.15.33:0/3676982249 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc588109010 con 0x7fc588106a80 2024-09-16T00:37:50.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.054+0000 7fc5777fe640 1 -- 172.21.15.33:0/3676982249 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7fc5800bde20 con 0x7fc588106a80 2024-09-16T00:37:50.054 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:37:50.055 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:37:50.055 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:37:50.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.056+0000 7fc58cdcf640 1 -- 172.21.15.33:0/3676982249 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc5700a3240 msgr2=0x7fc5700a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:50.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.056+0000 7fc58cdcf640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc5700a3240 0x7fc5700a5700 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc578009990 tx=0x7fc578009110 comp rx=0 tx=0).stop 2024-09-16T00:37:50.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.056+0000 7fc58cdcf640 1 -- 172.21.15.33:0/3676982249 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc588106a80 msgr2=0x7fc58810ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:50.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.056+0000 7fc58cdcf640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc588106a80 0x7fc58810ee60 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fc580006010 tx=0x7fc580003c20 comp rx=0 tx=0).stop 2024-09-16T00:37:50.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.056+0000 7fc58cdcf640 1 -- 172.21.15.33:0/3676982249 shutdown_connections 2024-09-16T00:37:50.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.056+0000 7fc58cdcf640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc5700a3240 0x7fc5700a5700 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:50.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.056+0000 7fc58cdcf640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc588106a80 0x7fc58810ee60 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:50.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.056+0000 7fc58cdcf640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc588101420 0x7fc58810e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:50.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.056+0000 7fc58cdcf640 1 --2- 172.21.15.33:0/3676982249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc588100a80 0x7fc58810e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:50.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.056+0000 7fc58cdcf640 1 -- 172.21.15.33:0/3676982249 >> 172.21.15.33:0/3676982249 conn(0x7fc5880fc7a0 msgr2=0x7fc5880fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:50.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.056+0000 7fc58cdcf640 1 -- 172.21.15.33:0/3676982249 shutdown_connections 2024-09-16T00:37:50.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:50.056+0000 7fc58cdcf640 1 -- 172.21.15.33:0/3676982249 wait complete. 2024-09-16T00:37:50.090 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 16.52 MB/sec execute 296 sec latency 26127.794 ms 2024-09-16T00:37:50.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:50 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3676982249' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:37:50.745 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 50781c66 --release --await' 2024-09-16T00:37:50.745 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 50781c66 --release --await 2024-09-16T00:37:50.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:50 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3676982249' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:37:50.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:50 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3676982249' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:37:51.046 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:37:51.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134525 16.46 MB/sec execute 297 sec latency 27127.882 ms 2024-09-16T00:37:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.656+0000 7f007a175640 1 -- 172.21.15.33:0/3887819489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0074103660 msgr2=0x7f007410bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.656+0000 7f007a175640 1 --2- 172.21.15.33:0/3887819489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0074103660 0x7f007410bb50 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f0068009900 tx=0x7f006802a480 comp rx=0 tx=0).stop 2024-09-16T00:37:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.656+0000 7f007a175640 1 -- 172.21.15.33:0/3887819489 shutdown_connections 2024-09-16T00:37:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.656+0000 7f007a175640 1 --2- 172.21.15.33:0/3887819489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f007410c170 0x7f007410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.656+0000 7f007a175640 1 --2- 172.21.15.33:0/3887819489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0074103660 0x7f007410bb50 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.656+0000 7f007a175640 1 --2- 172.21.15.33:0/3887819489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074102d40 0x7f0074103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.656+0000 7f007a175640 1 -- 172.21.15.33:0/3887819489 >> 172.21.15.33:0/3887819489 conn(0x7f00740fc7a0 msgr2=0x7f00740febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.656+0000 7f007a175640 1 -- 172.21.15.33:0/3887819489 shutdown_connections 2024-09-16T00:37:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.656+0000 7f007a175640 1 -- 172.21.15.33:0/3887819489 wait complete. 2024-09-16T00:37:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.657+0000 7f007a175640 1 Processor -- start 2024-09-16T00:37:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.657+0000 7f007a175640 1 -- start start 2024-09-16T00:37:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.657+0000 7f007a175640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0074102d40 0x7f0074198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.657+0000 7f007a175640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074103660 0x7f00741989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.657+0000 7f007a175640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f007410c170 0x7f007419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.657+0000 7f007a175640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0074111fb0 con 0x7f007410c170 2024-09-16T00:37:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.657+0000 7f007a175640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0074111e30 con 0x7f0074102d40 2024-09-16T00:37:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.657+0000 7f007a175640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0074112130 con 0x7f0074103660 2024-09-16T00:37:51.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.657+0000 7f0073fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f007410c170 0x7f007419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0073fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f007410c170 0x7f007419cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53840/0 (socket says 172.21.15.33:53840) 2024-09-16T00:37:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f00737fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0074102d40 0x7f0074198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0073fff640 1 -- 172.21.15.33:0/2910180666 learned_addr learned my addr 172.21.15.33:0/2910180666 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0072ffd640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074103660 0x7f00741989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0073fff640 1 -- 172.21.15.33:0/2910180666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074103660 msgr2=0x7f00741989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0073fff640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074103660 0x7f00741989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0073fff640 1 -- 172.21.15.33:0/2910180666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0074102d40 msgr2=0x7f0074198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0073fff640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0074102d40 0x7f0074198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0073fff640 1 -- 172.21.15.33:0/2910180666 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007419d420 con 0x7f007410c170 2024-09-16T00:37:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0072ffd640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074103660 0x7f00741989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:37:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0073fff640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f007410c170 0x7f007419cd40 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f006c00b970 tx=0x7f006c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:51.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0070ff9640 1 -- 172.21.15.33:0/2910180666 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c019070 con 0x7f007410c170 2024-09-16T00:37:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f007a175640 1 -- 172.21.15.33:0/2910180666 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f007419d710 con 0x7f007410c170 2024-09-16T00:37:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0070ff9640 1 -- 172.21.15.33:0/2910180666 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f006c014440 con 0x7f007410c170 2024-09-16T00:37:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f0070ff9640 1 -- 172.21.15.33:0/2910180666 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c0075d0 con 0x7f007410c170 2024-09-16T00:37:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.658+0000 7f007a175640 1 -- 172.21.15.33:0/2910180666 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00741a4ff0 con 0x7f007410c170 2024-09-16T00:37:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f0070ff9640 1 -- 172.21.15.33:0/2910180666 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f006c01c070 con 0x7f007410c170 2024-09-16T00:37:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f007c54e740 1 Processor -- start 2024-09-16T00:37:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f007c54e740 1 -- start start 2024-09-16T00:37:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f007c54e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 0x561181d52300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f007c54e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 0x561181d61110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f007c54e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d61650 0x561181d63a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f007c54e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561181cb7620 con 0x561181d51f20 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f007c54e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561181c5ee50 con 0x561181d61650 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f007c54e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561181cb35e0 con 0x561181d52840 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f0072ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 0x561181d52300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f0070ff9640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f005c0a3240 0x7f005c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f0072ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 0x561181d52300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53844/0 (socket says 172.21.15.33:53844) 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.661+0000 7f0072ffd640 1 -- 172.21.15.33:0/3791807025 learned_addr learned my addr 172.21.15.33:0/3791807025 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0073fff640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f005c0a3240 0x7f005c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f00737fe640 1 --2- 172.21.15.33:0/3791807025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 0x561181d61110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0072ffd640 1 --2- 172.21.15.33:0/3791807025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d61650 0x561181d63a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0070ff9640 1 -- 172.21.15.33:0/2910180666 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f006c0cf3e0 con 0x7f007410c170 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0072ffd640 1 -- 172.21.15.33:0/3791807025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 msgr2=0x561181d61110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0072ffd640 1 --2- 172.21.15.33:0/3791807025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 0x561181d61110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0072ffd640 1 -- 172.21.15.33:0/3791807025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d61650 msgr2=0x561181d63a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0072ffd640 1 --2- 172.21.15.33:0/3791807025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d61650 0x561181d63a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0072ffd640 1 -- 172.21.15.33:0/3791807025 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561181c4fd40 con 0x561181d51f20 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0072ffd640 1 --2- 172.21.15.33:0/3791807025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 0x561181d52300 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f0068031550 tx=0x7f0068031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da79ce58fbd0a151 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0073fff640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f005c0a3240 0x7f005c0a5700 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f006c0ccdd0 tx=0x7f006c090a60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0059ffb640 1 -- 172.21.15.33:0/3791807025 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0068043030 con 0x561181d51f20 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0059ffb640 1 -- 172.21.15.33:0/3791807025 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0068005020 con 0x561181d51f20 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.662+0000 7f0059ffb640 1 -- 172.21.15.33:0/3791807025 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0068035050 con 0x561181d51f20 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.663+0000 7f007c54e740 1 -- 172.21.15.33:0/3791807025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 msgr2=0x561181d52300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.663+0000 7f007c54e740 1 --2- 172.21.15.33:0/3791807025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 0x561181d52300 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f0068031550 tx=0x7f0068031400 comp rx=0 tx=0).stop 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.663+0000 7f007c54e740 1 -- 172.21.15.33:0/3791807025 shutdown_connections 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.663+0000 7f007c54e740 1 --2- 172.21.15.33:0/3791807025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d61650 0x561181d63a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.663+0000 7f007c54e740 1 --2- 172.21.15.33:0/3791807025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 0x561181d61110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.663+0000 7f007c54e740 1 --2- 172.21.15.33:0/3791807025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 0x561181d52300 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.663+0000 7f007c54e740 1 -- 172.21.15.33:0/3791807025 >> 172.21.15.33:0/3791807025 conn(0x561181d4e5f0 msgr2=0x561181d4e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.663+0000 7f007c54e740 1 -- 172.21.15.33:0/3791807025 shutdown_connections 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.663+0000 7f007c54e740 1 -- 172.21.15.33:0/3791807025 wait complete. 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.663+0000 7f007c54e740 1 -- start start 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f007c54e740 1 Processor -- start 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f007c54e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d51f20 0x561181d5e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f007c54e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d52840 0x561181d5ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f007c54e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d61650 0x561181d5f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f007c54e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561181d641c0 con 0x561181d52840 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f007c54e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561181c3ff10 con 0x561181d51f20 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f007c54e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561181d64340 con 0x561181d61650 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f007c54e740 10 client.0 _ll_register_callbacks cb 0x561181c63db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f00737fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d52840 0x561181d5ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f00737fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d52840 0x561181d5ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53860/0 (socket says 172.21.15.33:53860) 2024-09-16T00:37:51.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f00737fe640 1 -- 172.21.15.33:0/2079300404 learned_addr learned my addr 172.21.15.33:0/2079300404 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:51.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f0072ffd640 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d51f20 0x561181d5e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f0072ffd640 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d61650 0x561181d5f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f00737fe640 1 -- 172.21.15.33:0/2079300404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d61650 msgr2=0x561181d5f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:51.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f00737fe640 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d61650 0x561181d5f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f00737fe640 1 -- 172.21.15.33:0/2079300404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d51f20 msgr2=0x561181d5e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:51.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f00737fe640 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d51f20 0x561181d5e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f00737fe640 1 -- 172.21.15.33:0/2079300404 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611819c6580 con 0x561181d52840 2024-09-16T00:37:51.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.664+0000 7f0072ffd640 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d51f20 0x561181d5e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:37:51.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f00737fe640 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d52840 0x561181d5ead0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f0064012870 tx=0x7f0064012d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24bfdeb14e6b9b28 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:51.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f0052ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f0052ffd640 1 -- 172.21.15.33:0/2079300404 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006401f060 con 0x561181d52840 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f007c54e740 1 -- 172.21.15.33:0/2079300404 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0068005020 con 0x561181d52840 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f0052ffd640 1 -- 172.21.15.33:0/2079300404 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0064010dc0 con 0x561181d52840 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f0052ffd640 1 -- 172.21.15.33:0/2079300404 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0064026050 con 0x561181d52840 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f007c54e740 10 client.19359 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f007c54e740 1 -- 172.21.15.33:0/2079300404 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561181a90960 con 0x561181d52840 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f0052ffd640 1 -- 172.21.15.33:0/2079300404 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0068005020 con 0x561181d52840 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f0052ffd640 1 -- 172.21.15.33:0/2079300404 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f006400d480 con 0x561181d52840 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f007c54e740 10 client.19359 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f007c54e740 10 client.19359 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f007c54e740 20 client.19359 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f007c54e740 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561181d711f0 0x561181d735e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f007c54e740 4 client.19359 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.665+0000 7f007c54e740 1 -- 172.21.15.33:0/2079300404 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561181cc4b90 con 0x561181d711f0 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.666+0000 7f0072ffd640 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561181d711f0 0x561181d735e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.666+0000 7f0052ffd640 10 client.19359 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.666+0000 7f0072ffd640 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561181d711f0 0x561181d735e0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56672f2846f6ba6f server_cookie=4697dc389d97e0d7 in_seq=0 out_seq=0 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f0052ffd640 1 -- 172.21.15.33:0/2079300404 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561181cc4b90 con 0x561181d711f0 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f0052ffd640 10 client.19359 handle_command_reply: tid=0 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f007c54e740 1 client.19359 shutdown 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f007c54e740 1 -- 172.21.15.33:0/2079300404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d52840 msgr2=0x561181d5ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f007c54e740 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d52840 0x561181d5ead0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f0064012870 tx=0x7f0064012d30 comp rx=0 tx=0).stop 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f007c54e740 1 -- 172.21.15.33:0/2079300404 shutdown_connections 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f007c54e740 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561181d711f0 0x561181d735e0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f007c54e740 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d61650 0x561181d5f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f007c54e740 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d52840 0x561181d5ead0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f0052ffd640 0 client.19359 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:51.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f007c54e740 1 --2- 172.21.15.33:0/2079300404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d51f20 0x561181d5e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f007c54e740 1 -- 172.21.15.33:0/2079300404 >> 172.21.15.33:0/2079300404 conn(0x561181d4e5f0 msgr2=0x561181d4d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:51.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f007c54e740 1 -- 172.21.15.33:0/2079300404 shutdown_connections 2024-09-16T00:37:51.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f007c54e740 1 -- 172.21.15.33:0/2079300404 wait complete. 2024-09-16T00:37:51.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.667+0000 7f007c54e740 20 client.19359 trim_cache size 0 max 16384 2024-09-16T00:37:51.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.684+0000 7f007c54e740 1 Processor -- start 2024-09-16T00:37:51.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.684+0000 7f007c54e740 1 -- start start 2024-09-16T00:37:51.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.684+0000 7f007c54e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 0x561181d492b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.684+0000 7f007c54e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 0x561181d4d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.684+0000 7f007c54e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d61650 0x561181d78ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f007c54e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00741af5e0 con 0x561181d51f20 2024-09-16T00:37:51.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f007c54e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f00741aceb0 con 0x561181d61650 2024-09-16T00:37:51.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f007c54e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00741aad00 con 0x561181d52840 2024-09-16T00:37:51.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f0072ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 0x561181d492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f0072ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 0x561181d492b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53864/0 (socket says 172.21.15.33:53864) 2024-09-16T00:37:51.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f0072ffd640 1 -- 172.21.15.33:0/839205834 learned_addr learned my addr 172.21.15.33:0/839205834 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:51.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f0072ffd640 1 -- 172.21.15.33:0/839205834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 msgr2=0x561181d4d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f0072ffd640 1 --2- 172.21.15.33:0/839205834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 0x561181d4d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f0072ffd640 1 -- 172.21.15.33:0/839205834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d61650 msgr2=0x561181d78ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f0072ffd640 1 --2- 172.21.15.33:0/839205834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d61650 0x561181d78ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f0072ffd640 1 -- 172.21.15.33:0/839205834 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561181d604d0 con 0x561181d51f20 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f0072ffd640 1 --2- 172.21.15.33:0/839205834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 0x561181d492b0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f0068031b20 tx=0x7f00680315e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b13d04d977f6ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f00597fa640 1 -- 172.21.15.33:0/839205834 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0068043040 con 0x561181d51f20 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f00597fa640 1 -- 172.21.15.33:0/839205834 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0068044040 con 0x561181d51f20 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.685+0000 7f00597fa640 1 -- 172.21.15.33:0/839205834 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0068035040 con 0x561181d51f20 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.686+0000 7f007c54e740 1 -- 172.21.15.33:0/839205834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 msgr2=0x561181d492b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.686+0000 7f007c54e740 1 --2- 172.21.15.33:0/839205834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 0x561181d492b0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f0068031b20 tx=0x7f00680315e0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.686+0000 7f007c54e740 1 -- 172.21.15.33:0/839205834 shutdown_connections 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.686+0000 7f007c54e740 1 --2- 172.21.15.33:0/839205834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d61650 0x561181d78ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.686+0000 7f007c54e740 1 --2- 172.21.15.33:0/839205834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 0x561181d4d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.686+0000 7f007c54e740 1 --2- 172.21.15.33:0/839205834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d51f20 0x561181d492b0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.686+0000 7f007c54e740 1 -- 172.21.15.33:0/839205834 >> 172.21.15.33:0/839205834 conn(0x561181d4e5f0 msgr2=0x561181d4cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.686+0000 7f007c54e740 1 -- 172.21.15.33:0/839205834 shutdown_connections 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.686+0000 7f007c54e740 1 -- 172.21.15.33:0/839205834 wait complete. 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.686+0000 7f007c54e740 1 -- start start 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f007c54e740 1 Processor -- start 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f007c54e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d51f20 0x561181d65a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f007c54e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 0x561181d65f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f007c54e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d61650 0x561181ddaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f007c54e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00741ab370 con 0x561181d61650 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f007c54e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f00741ac7e0 con 0x561181d51f20 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f007c54e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00741aba40 con 0x561181d52840 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f007c54e740 10 client.0 _ll_register_callbacks cb 0x7f0064012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:37:51.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f0072ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d61650 0x561181ddaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f0072ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d61650 0x561181ddaec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53878/0 (socket says 172.21.15.33:53878) 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f0072ffd640 1 -- 172.21.15.33:0/2166912062 learned_addr learned my addr 172.21.15.33:0/2166912062 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f00737fe640 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 0x561181d65f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f0072ffd640 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d51f20 0x561181d65a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f0072ffd640 1 -- 172.21.15.33:0/2166912062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 msgr2=0x561181d65f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f0072ffd640 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 0x561181d65f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f0072ffd640 1 -- 172.21.15.33:0/2166912062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d51f20 msgr2=0x561181d65a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f0072ffd640 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d51f20 0x561181d65a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.687+0000 7f0072ffd640 1 -- 172.21.15.33:0/2166912062 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561181d52670 con 0x561181d61650 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f0072ffd640 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d61650 0x561181ddaec0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f0068005420 tx=0x7f006804b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c7b01922d85d28 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f0052ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f0052ffd640 1 -- 172.21.15.33:0/2166912062 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0068043030 con 0x561181d61650 2024-09-16T00:37:51.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f0052ffd640 1 -- 172.21.15.33:0/2166912062 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f00680049c0 con 0x561181d61650 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f007c54e740 1 -- 172.21.15.33:0/2166912062 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0068044040 con 0x561181d61650 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f0052ffd640 1 -- 172.21.15.33:0/2166912062 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0068035070 con 0x561181d61650 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f007c54e740 10 client.19365 fetch_fsmap learned FSMap version 33 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f007c54e740 1 -- 172.21.15.33:0/2166912062 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561181ddc520 con 0x561181d61650 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f0052ffd640 1 -- 172.21.15.33:0/2166912062 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0068044040 con 0x561181d61650 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f0052ffd640 1 -- 172.21.15.33:0/2166912062 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0068056430 con 0x561181d61650 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f007c54e740 10 client.19365 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f007c54e740 10 client.19365 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f007c54e740 20 client.19365 populate_metadata read hostname 'smithi033' 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f007c54e740 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561181ddc7a0 0x561181ddeb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f007c54e740 4 client.19365 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "50781c66", "release": true, "await": true}] 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.688+0000 7f007c54e740 1 -- 172.21.15.33:0/2166912062 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "50781c66", "release": true, "await": true}) -- 0x561181cbf0a0 con 0x561181ddc7a0 2024-09-16T00:37:51.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.689+0000 7f00737fe640 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561181ddc7a0 0x561181ddeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:37:51.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.689+0000 7f00737fe640 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561181ddc7a0 0x561181ddeb90 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51c17f93263873f3 server_cookie=f1d9ad2bcc8d5cef in_seq=0 out_seq=0 2024-09-16T00:37:51.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.689+0000 7f0052ffd640 10 client.19365 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:51.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f0052ffd640 1 -- 172.21.15.33:0/2166912062 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x561181cbf0a0 con 0x561181ddc7a0 2024-09-16T00:37:51.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f0052ffd640 10 client.19365 handle_command_reply: tid=0 2024-09-16T00:37:51.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f007c54e740 1 client.19365 shutdown 2024-09-16T00:37:51.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f007c54e740 1 -- 172.21.15.33:0/2166912062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d61650 msgr2=0x561181ddaec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:51.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f007c54e740 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d61650 0x561181ddaec0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f0068005420 tx=0x7f006804b040 comp rx=0 tx=0).stop 2024-09-16T00:37:51.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f007c54e740 1 -- 172.21.15.33:0/2166912062 shutdown_connections 2024-09-16T00:37:51.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f007c54e740 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561181ddc7a0 0x561181ddeb90 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f007c54e740 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561181d61650 0x561181ddaec0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f007c54e740 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561181d52840 0x561181d65f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f007c54e740 1 --2- 172.21.15.33:0/2166912062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561181d51f20 0x561181d65a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f0052ffd640 0 client.19365 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:37:51.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f007c54e740 1 -- 172.21.15.33:0/2166912062 >> 172.21.15.33:0/2166912062 conn(0x561181d4e5f0 msgr2=0x561181d5ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:51.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.748+0000 7f007c54e740 1 -- 172.21.15.33:0/2166912062 shutdown_connections 2024-09-16T00:37:51.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.749+0000 7f007c54e740 1 -- 172.21.15.33:0/2166912062 wait complete. 2024-09-16T00:37:51.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.749+0000 7f007c54e740 20 client.19365 trim_cache size 0 max 16384 2024-09-16T00:37:51.750 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:37:51.750 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:37:51.750 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:37:51.750 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 76, 2024-09-16T00:37:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:37:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "50781c66": { 2024-09-16T00:37:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 76, 2024-09-16T00:37:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:37:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:37:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:37:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:37:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:37:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:37:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 148.2, 2024-09-16T00:37:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:37:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:37:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:37:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:37:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:37:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:37:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:37:51.752 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:37:51.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.749+0000 7f007a175640 1 -- 172.21.15.33:0/2910180666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f005c0a3240 msgr2=0x7f005c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:51.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.749+0000 7f007a175640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f005c0a3240 0x7f005c0a5700 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f006c0ccdd0 tx=0x7f006c090a60 comp rx=0 tx=0).stop 2024-09-16T00:37:51.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.749+0000 7f007a175640 1 -- 172.21.15.33:0/2910180666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f007410c170 msgr2=0x7f007419cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:37:51.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.749+0000 7f007a175640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f007410c170 0x7f007419cd40 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f006c00b970 tx=0x7f006c00bd70 comp rx=0 tx=0).stop 2024-09-16T00:37:51.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.749+0000 7f007a175640 1 -- 172.21.15.33:0/2910180666 shutdown_connections 2024-09-16T00:37:51.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.749+0000 7f007a175640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f005c0a3240 0x7f005c0a5700 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.749+0000 7f007a175640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f007410c170 0x7f007419cd40 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.749+0000 7f007a175640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074103660 0x7f00741989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.749+0000 7f007a175640 1 --2- 172.21.15.33:0/2910180666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0074102d40 0x7f0074198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:37:51.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.749+0000 7f007a175640 1 -- 172.21.15.33:0/2910180666 >> 172.21.15.33:0/2910180666 conn(0x7f00740fc7a0 msgr2=0x7f00740fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:37:51.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.750+0000 7f007a175640 1 -- 172.21.15.33:0/2910180666 shutdown_connections 2024-09-16T00:37:51.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:37:51.750+0000 7f007a175640 1 -- 172.21.15.33:0/2910180666 wait complete. 2024-09-16T00:37:51.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:51 smithi033 ceph-mon[31243]: pgmap v971: 129 pgs: 129 active+clean; 616 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-09-16T00:37:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:51 smithi083 ceph-mon[35242]: pgmap v971: 129 pgs: 129 active+clean; 616 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-09-16T00:37:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:51 smithi060 ceph-mon[35185]: pgmap v971: 129 pgs: 129 active+clean; 616 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-09-16T00:37:52.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 134689 16.43 MB/sec execute 298 sec latency 27813.546 ms 2024-09-16T00:37:52.574 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '50781c66', total seconds elapsed: 31.7 2024-09-16T00:37:53.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 135191 16.47 MB/sec execute 299 sec latency 47.615 ms 2024-09-16T00:37:53.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:53 smithi033 ceph-mon[31243]: pgmap v972: 129 pgs: 129 active+clean; 618 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 2.3 MiB/s wr, 107 op/s 2024-09-16T00:37:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:53 smithi083 ceph-mon[35242]: pgmap v972: 129 pgs: 129 active+clean; 618 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 2.3 MiB/s wr, 107 op/s 2024-09-16T00:37:53.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:53 smithi060 ceph-mon[35185]: pgmap v972: 129 pgs: 129 active+clean; 618 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 2.3 MiB/s wr, 107 op/s 2024-09-16T00:37:54.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 135719 16.49 MB/sec execute 300 sec latency 90.160 ms 2024-09-16T00:37:55.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 136225 16.50 MB/sec execute 301 sec latency 84.097 ms 2024-09-16T00:37:55.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:55 smithi033 ceph-mon[31243]: pgmap v973: 129 pgs: 129 active+clean; 619 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 122 op/s 2024-09-16T00:37:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:55 smithi083 ceph-mon[35242]: pgmap v973: 129 pgs: 129 active+clean; 619 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 122 op/s 2024-09-16T00:37:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:55 smithi060 ceph-mon[35185]: pgmap v973: 129 pgs: 129 active+clean; 619 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 122 op/s 2024-09-16T00:37:56.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 136849 16.60 MB/sec execute 302 sec latency 69.286 ms 2024-09-16T00:37:57.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 137448 16.64 MB/sec execute 303 sec latency 84.154 ms 2024-09-16T00:37:57.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:57 smithi033 ceph-mon[31243]: pgmap v974: 129 pgs: 129 active+clean; 623 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 3.0 MiB/s wr, 206 op/s 2024-09-16T00:37:57.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:57 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:37:57.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:57 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:37:57.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:57 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:37:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:57 smithi083 ceph-mon[35242]: pgmap v974: 129 pgs: 129 active+clean; 623 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 3.0 MiB/s wr, 206 op/s 2024-09-16T00:37:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:57 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:37:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:57 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:37:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:57 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:37:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:57 smithi060 ceph-mon[35185]: pgmap v974: 129 pgs: 129 active+clean; 623 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 3.0 MiB/s wr, 206 op/s 2024-09-16T00:37:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:57 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:37:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:57 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:37:57.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:57 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:37:58.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 138005 16.69 MB/sec execute 304 sec latency 52.761 ms 2024-09-16T00:37:59.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 138609 16.75 MB/sec execute 305 sec latency 69.530 ms 2024-09-16T00:37:59.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:37:59 smithi033 ceph-mon[31243]: pgmap v975: 129 pgs: 129 active+clean; 623 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.6 MiB/s wr, 196 op/s 2024-09-16T00:37:59.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:37:59 smithi083 ceph-mon[35242]: pgmap v975: 129 pgs: 129 active+clean; 623 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.6 MiB/s wr, 196 op/s 2024-09-16T00:37:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:37:59 smithi060 ceph-mon[35185]: pgmap v975: 129 pgs: 129 active+clean; 623 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.6 MiB/s wr, 196 op/s 2024-09-16T00:38:00.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 139009 16.77 MB/sec execute 306 sec latency 83.883 ms 2024-09-16T00:38:01.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 139518 16.78 MB/sec execute 307 sec latency 72.873 ms 2024-09-16T00:38:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:01 smithi060 ceph-mon[35185]: pgmap v976: 129 pgs: 129 active+clean; 629 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.5 MiB/s wr, 322 op/s 2024-09-16T00:38:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:01 smithi083 ceph-mon[35242]: pgmap v976: 129 pgs: 129 active+clean; 629 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.5 MiB/s wr, 322 op/s 2024-09-16T00:38:02.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:01 smithi033 ceph-mon[31243]: pgmap v976: 129 pgs: 129 active+clean; 629 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.5 MiB/s wr, 322 op/s 2024-09-16T00:38:02.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 140124 16.83 MB/sec execute 308 sec latency 109.530 ms 2024-09-16T00:38:03.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 140603 16.85 MB/sec execute 309 sec latency 82.399 ms 2024-09-16T00:38:03.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:03 smithi083 ceph-mon[35242]: pgmap v977: 129 pgs: 129 active+clean; 631 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 347 op/s 2024-09-16T00:38:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:03 smithi060 ceph-mon[35185]: pgmap v977: 129 pgs: 129 active+clean; 631 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 347 op/s 2024-09-16T00:38:04.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:03 smithi033 ceph-mon[31243]: pgmap v977: 129 pgs: 129 active+clean; 631 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 347 op/s 2024-09-16T00:38:04.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 141205 16.92 MB/sec execute 310 sec latency 63.530 ms 2024-09-16T00:38:05.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 141777 16.96 MB/sec execute 311 sec latency 68.068 ms 2024-09-16T00:38:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:05 smithi083 ceph-mon[35242]: pgmap v978: 129 pgs: 129 active+clean; 626 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.0 MiB/s wr, 408 op/s 2024-09-16T00:38:05.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:05 smithi060 ceph-mon[35185]: pgmap v978: 129 pgs: 129 active+clean; 626 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.0 MiB/s wr, 408 op/s 2024-09-16T00:38:06.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:05 smithi033 ceph-mon[31243]: pgmap v978: 129 pgs: 129 active+clean; 626 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.0 MiB/s wr, 408 op/s 2024-09-16T00:38:06.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 142439 17.03 MB/sec execute 312 sec latency 64.921 ms 2024-09-16T00:38:07.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 142987 17.08 MB/sec execute 313 sec latency 78.631 ms 2024-09-16T00:38:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:07 smithi083 ceph-mon[35242]: pgmap v979: 129 pgs: 129 active+clean; 618 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.3 MiB/s wr, 432 op/s 2024-09-16T00:38:07.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:07 smithi060 ceph-mon[35185]: pgmap v979: 129 pgs: 129 active+clean; 618 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.3 MiB/s wr, 432 op/s 2024-09-16T00:38:08.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:07 smithi033 ceph-mon[31243]: pgmap v979: 129 pgs: 129 active+clean; 618 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.3 MiB/s wr, 432 op/s 2024-09-16T00:38:08.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 143042 17.04 MB/sec execute 314 sec latency 903.862 ms 2024-09-16T00:38:09.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 143283 17.02 MB/sec execute 315 sec latency 1174.570 ms 2024-09-16T00:38:09.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:09 smithi083 ceph-mon[35242]: pgmap v980: 129 pgs: 129 active+clean; 618 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 346 op/s 2024-09-16T00:38:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:09 smithi060 ceph-mon[35185]: pgmap v980: 129 pgs: 129 active+clean; 618 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 346 op/s 2024-09-16T00:38:10.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:09 smithi033 ceph-mon[31243]: pgmap v980: 129 pgs: 129 active+clean; 618 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 346 op/s 2024-09-16T00:38:10.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 143530 17.00 MB/sec execute 316 sec latency 137.389 ms 2024-09-16T00:38:11.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 143839 17.00 MB/sec execute 317 sec latency 100.423 ms 2024-09-16T00:38:11.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:11 smithi083 ceph-mon[35242]: pgmap v981: 129 pgs: 129 active+clean; 657 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.6 MiB/s wr, 479 op/s 2024-09-16T00:38:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:11 smithi060 ceph-mon[35185]: pgmap v981: 129 pgs: 129 active+clean; 657 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.6 MiB/s wr, 479 op/s 2024-09-16T00:38:12.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:11 smithi033 ceph-mon[31243]: pgmap v981: 129 pgs: 129 active+clean; 657 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.6 MiB/s wr, 479 op/s 2024-09-16T00:38:12.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 144555 17.04 MB/sec execute 318 sec latency 83.013 ms 2024-09-16T00:38:13.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 145104 17.10 MB/sec execute 319 sec latency 78.899 ms 2024-09-16T00:38:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:13 smithi083 ceph-mon[35242]: pgmap v982: 129 pgs: 129 active+clean; 673 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 9.5 MiB/s wr, 494 op/s 2024-09-16T00:38:13.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:13 smithi060 ceph-mon[35185]: pgmap v982: 129 pgs: 129 active+clean; 673 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 9.5 MiB/s wr, 494 op/s 2024-09-16T00:38:14.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:13 smithi033 ceph-mon[31243]: pgmap v982: 129 pgs: 129 active+clean; 673 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 9.5 MiB/s wr, 494 op/s 2024-09-16T00:38:14.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 145755 17.15 MB/sec execute 320 sec latency 83.750 ms 2024-09-16T00:38:15.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 146354 17.20 MB/sec execute 321 sec latency 102.742 ms 2024-09-16T00:38:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:15 smithi083 ceph-mon[35242]: pgmap v983: 129 pgs: 129 active+clean; 646 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 9.2 MiB/s wr, 481 op/s 2024-09-16T00:38:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:15 smithi060 ceph-mon[35185]: pgmap v983: 129 pgs: 129 active+clean; 646 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 9.2 MiB/s wr, 481 op/s 2024-09-16T00:38:16.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:15 smithi033 ceph-mon[31243]: pgmap v983: 129 pgs: 129 active+clean; 646 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 9.2 MiB/s wr, 481 op/s 2024-09-16T00:38:16.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 147015 17.28 MB/sec execute 322 sec latency 74.291 ms 2024-09-16T00:38:17.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 147461 17.31 MB/sec execute 323 sec latency 119.375 ms 2024-09-16T00:38:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:17 smithi083 ceph-mon[35242]: pgmap v984: 129 pgs: 129 active+clean; 630 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 9.2 MiB/s wr, 496 op/s 2024-09-16T00:38:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:17 smithi060 ceph-mon[35185]: pgmap v984: 129 pgs: 129 active+clean; 630 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 9.2 MiB/s wr, 496 op/s 2024-09-16T00:38:18.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:17 smithi033 ceph-mon[31243]: pgmap v984: 129 pgs: 129 active+clean; 630 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 9.2 MiB/s wr, 496 op/s 2024-09-16T00:38:18.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 147819 17.30 MB/sec execute 324 sec latency 81.560 ms 2024-09-16T00:38:18.225 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:38:18.539 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:38:19.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 148389 17.33 MB/sec execute 325 sec latency 85.505 ms 2024-09-16T00:38:19.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.191+0000 7f5638b4e640 1 -- 172.21.15.33:0/2530299650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5634109450 msgr2=0x7f5634109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:19.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.191+0000 7f5638b4e640 1 --2- 172.21.15.33:0/2530299650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5634109450 0x7f5634109830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f562400af00 tx=0x7f5624032770 comp rx=0 tx=0).stop 2024-09-16T00:38:19.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.191+0000 7f5638b4e640 1 -- 172.21.15.33:0/2530299650 shutdown_connections 2024-09-16T00:38:19.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.191+0000 7f5638b4e640 1 --2- 172.21.15.33:0/2530299650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f563410ea80 0x7f56341154f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.191+0000 7f5638b4e640 1 --2- 172.21.15.33:0/2530299650 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5634109e00 0x7f563410e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.191+0000 7f5638b4e640 1 --2- 172.21.15.33:0/2530299650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5634109450 0x7f5634109830 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.191+0000 7f5638b4e640 1 -- 172.21.15.33:0/2530299650 >> 172.21.15.33:0/2530299650 conn(0x7f563406e2d0 msgr2=0x7f563406e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:19.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.191+0000 7f5638b4e640 1 -- 172.21.15.33:0/2530299650 shutdown_connections 2024-09-16T00:38:19.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.191+0000 7f5638b4e640 1 -- 172.21.15.33:0/2530299650 wait complete. 2024-09-16T00:38:19.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.192+0000 7f5638b4e640 1 Processor -- start 2024-09-16T00:38:19.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.192+0000 7f5638b4e640 1 -- start start 2024-09-16T00:38:19.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.193+0000 7f5638b4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5634109450 0x7f56341a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.193+0000 7f5638b4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5634109e00 0x7f56341a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.193+0000 7f5638b4e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f563410ea80 0x7f563419e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.193+0000 7f5638b4e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f56341185b0 con 0x7f5634109e00 2024-09-16T00:38:19.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.193+0000 7f5638b4e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5634118430 con 0x7f563410ea80 2024-09-16T00:38:19.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.193+0000 7f5638b4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5634118730 con 0x7f5634109450 2024-09-16T00:38:19.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.193+0000 7f5631d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5634109e00 0x7f56341a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.193+0000 7f5631d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5634109e00 0x7f56341a4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41110/0 (socket says 172.21.15.33:41110) 2024-09-16T00:38:19.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.193+0000 7f5632d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f563410ea80 0x7f563419e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.193+0000 7f5632d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f563410ea80 0x7f563419e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:53662/0 (socket says 172.21.15.33:53662) 2024-09-16T00:38:19.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.193+0000 7f5632d9d640 1 -- 172.21.15.33:0/3719820897 learned_addr learned my addr 172.21.15.33:0/3719820897 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:19.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.193+0000 7f563259c640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5634109450 0x7f56341a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.194+0000 7f5632d9d640 1 -- 172.21.15.33:0/3719820897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5634109450 msgr2=0x7f56341a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:19.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.194+0000 7f5632d9d640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5634109450 0x7f56341a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.194+0000 7f5632d9d640 1 -- 172.21.15.33:0/3719820897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5634109e00 msgr2=0x7f56341a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:19.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.194+0000 7f5632d9d640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5634109e00 0x7f56341a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.194+0000 7f5632d9d640 1 -- 172.21.15.33:0/3719820897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563419efc0 con 0x7f563410ea80 2024-09-16T00:38:19.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.195+0000 7f5632d9d640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f563410ea80 0x7f563419e890 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f562c00b970 tx=0x7f562c00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:19.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.195+0000 7f56237fe640 1 -- 172.21.15.33:0/3719820897 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562c019070 con 0x7f563410ea80 2024-09-16T00:38:19.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.195+0000 7f56237fe640 1 -- 172.21.15.33:0/3719820897 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f562c00cb40 con 0x7f563410ea80 2024-09-16T00:38:19.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.195+0000 7f56237fe640 1 -- 172.21.15.33:0/3719820897 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562c007650 con 0x7f563410ea80 2024-09-16T00:38:19.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.195+0000 7f5638b4e640 1 -- 172.21.15.33:0/3719820897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f563419f2b0 con 0x7f563410ea80 2024-09-16T00:38:19.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.196+0000 7f5638b4e640 1 -- 172.21.15.33:0/3719820897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56341ab5a0 con 0x7f563410ea80 2024-09-16T00:38:19.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.199+0000 7f563af27740 1 Processor -- start 2024-09-16T00:38:19.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.200+0000 7f563af27740 1 -- start start 2024-09-16T00:38:19.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.201+0000 7f563af27740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4aa4a0 0x56088f4aa880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.201+0000 7f563af27740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 0x56088f4b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.202+0000 7f563af27740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4b5680 0x56088f4b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.202+0000 7f563af27740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56088f4102e0 con 0x56088f4aa4a0 2024-09-16T00:38:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.202+0000 7f563af27740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56088f3b7ce0 con 0x56088f4b5680 2024-09-16T00:38:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.202+0000 7f563af27740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56088f40c2a0 con 0x56088f4aadc0 2024-09-16T00:38:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.203+0000 7f56237fe640 1 -- 172.21.15.33:0/3719820897 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f562c01c070 con 0x7f563410ea80 2024-09-16T00:38:19.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.205+0000 7f563259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 0x56088f4b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.205+0000 7f56237fe640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f561c0a3410 0x7f561c0a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.205+0000 7f563259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 0x56088f4b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:49224/0 (socket says 172.21.15.33:49224) 2024-09-16T00:38:19.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.205+0000 7f563259c640 1 -- 172.21.15.33:0/2147228169 learned_addr learned my addr 172.21.15.33:0/2147228169 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:19.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.205+0000 7f56237fe640 1 -- 172.21.15.33:0/3719820897 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f562c0cf2f0 con 0x7f563410ea80 2024-09-16T00:38:19.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.206+0000 7f563259c640 1 -- 172.21.15.33:0/2147228169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4b5680 msgr2=0x56088f4b7a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:38:19.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.206+0000 7f563259c640 1 --2- 172.21.15.33:0/2147228169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4b5680 0x56088f4b7a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.206+0000 7f563259c640 1 -- 172.21.15.33:0/2147228169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4aa4a0 msgr2=0x56088f4aa880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:19.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.206+0000 7f563259c640 1 --2- 172.21.15.33:0/2147228169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4aa4a0 0x56088f4aa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.206+0000 7f563259c640 1 -- 172.21.15.33:0/2147228169 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56088f39c440 con 0x56088f4aadc0 2024-09-16T00:38:19.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.206+0000 7f563259c640 1 --2- 172.21.15.33:0/2147228169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 0x56088f4b5140 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f562403b040 tx=0x7f5624009d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76b5439d9200e609 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:19.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.207+0000 7f5620ff9640 1 -- 172.21.15.33:0/2147228169 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624048070 con 0x56088f4aadc0 2024-09-16T00:38:19.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.207+0000 7f5620ff9640 1 -- 172.21.15.33:0/2147228169 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f562400a8f0 con 0x56088f4aadc0 2024-09-16T00:38:19.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.208+0000 7f563af27740 1 -- 172.21.15.33:0/2147228169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 msgr2=0x56088f4b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:19.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.208+0000 7f563af27740 1 --2- 172.21.15.33:0/2147228169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 0x56088f4b5140 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f562403b040 tx=0x7f5624009d80 comp rx=0 tx=0).stop 2024-09-16T00:38:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.209+0000 7f5632d9d640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f561c0a3410 0x7f561c0a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.210+0000 7f5632d9d640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f561c0a3410 0x7f561c0a58d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f562c0099b0 tx=0x7f562c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.210+0000 7f563af27740 1 -- 172.21.15.33:0/2147228169 shutdown_connections 2024-09-16T00:38:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.210+0000 7f563af27740 1 --2- 172.21.15.33:0/2147228169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4b5680 0x56088f4b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.210+0000 7f563af27740 1 --2- 172.21.15.33:0/2147228169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 0x56088f4b5140 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.210+0000 7f563af27740 1 --2- 172.21.15.33:0/2147228169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4aa4a0 0x56088f4aa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.210+0000 7f563af27740 1 -- 172.21.15.33:0/2147228169 >> 172.21.15.33:0/2147228169 conn(0x56088f4a6a30 msgr2=0x56088f4a6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:19.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.211+0000 7f563af27740 1 -- 172.21.15.33:0/2147228169 shutdown_connections 2024-09-16T00:38:19.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.211+0000 7f563af27740 1 -- 172.21.15.33:0/2147228169 wait complete. 2024-09-16T00:38:19.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.211+0000 7f563af27740 1 -- start start 2024-09-16T00:38:19.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.212+0000 7f563af27740 1 Processor -- start 2024-09-16T00:38:19.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.212+0000 7f563af27740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aa4a0 0x56088f4b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.212+0000 7f563af27740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 0x56088f4b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.212+0000 7f563af27740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4b5680 0x56088f4b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.212+0000 7f563af27740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56088f4b82f0 con 0x56088f4b5680 2024-09-16T00:38:19.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.212+0000 7f563af27740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56088f3cb050 con 0x56088f4aa4a0 2024-09-16T00:38:19.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.212+0000 7f563af27740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56088f4b8470 con 0x56088f4aadc0 2024-09-16T00:38:19.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.212+0000 7f563af27740 10 client.0 _ll_register_callbacks cb 0x56088f40b160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:38:19.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.213+0000 7f563259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 0x56088f4b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.213+0000 7f563259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 0x56088f4b2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:49236/0 (socket says 172.21.15.33:49236) 2024-09-16T00:38:19.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.213+0000 7f563259c640 1 -- 172.21.15.33:0/4048053391 learned_addr learned my addr 172.21.15.33:0/4048053391 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:19.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.214+0000 7f5631d9b640 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4b5680 0x56088f4b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.214+0000 7f5631d9b640 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aa4a0 0x56088f4b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.215+0000 7f5631d9b640 1 -- 172.21.15.33:0/4048053391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 msgr2=0x56088f4b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:19.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.215+0000 7f5631d9b640 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 0x56088f4b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.215+0000 7f5631d9b640 1 -- 172.21.15.33:0/4048053391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4b5680 msgr2=0x56088f4b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:19.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.215+0000 7f5631d9b640 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4b5680 0x56088f4b26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.215+0000 7f5631d9b640 1 -- 172.21.15.33:0/4048053391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56088f1813c0 con 0x56088f4aa4a0 2024-09-16T00:38:19.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.219+0000 7f5631d9b640 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aa4a0 0x56088f4b4e60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f562800b670 tx=0x7f5628005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d7ce83fd0f53895 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:19.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.219+0000 7f5619ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:38:19.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.220+0000 7f5619ffb640 1 -- 172.21.15.33:0/4048053391 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628004050 con 0x56088f4aa4a0 2024-09-16T00:38:19.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.220+0000 7f563af27740 1 -- 172.21.15.33:0/4048053391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f562400a8f0 con 0x56088f4aa4a0 2024-09-16T00:38:19.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.220+0000 7f5619ffb640 1 -- 172.21.15.33:0/4048053391 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f56280041f0 con 0x56088f4aa4a0 2024-09-16T00:38:19.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.220+0000 7f5619ffb640 1 -- 172.21.15.33:0/4048053391 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f56280043f0 con 0x56088f4aa4a0 2024-09-16T00:38:19.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.220+0000 7f563af27740 10 client.28574 fetch_fsmap learned FSMap version 33 2024-09-16T00:38:19.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.220+0000 7f563af27740 1 -- 172.21.15.33:0/4048053391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56088f4c3c70 con 0x56088f4aa4a0 2024-09-16T00:38:19.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.221+0000 7f5619ffb640 1 -- 172.21.15.33:0/4048053391 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f562400a8f0 con 0x56088f4aa4a0 2024-09-16T00:38:19.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.221+0000 7f5619ffb640 1 -- 172.21.15.33:0/4048053391 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f5628019500 con 0x56088f4aa4a0 2024-09-16T00:38:19.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.221+0000 7f563af27740 10 client.28574 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:38:19.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.221+0000 7f563af27740 10 client.28574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:38:19.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.221+0000 7f563af27740 20 client.28574 populate_metadata read hostname 'smithi033' 2024-09-16T00:38:19.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.221+0000 7f563af27740 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56088f4c5520 0x56088f4c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.221+0000 7f563af27740 4 client.28574 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:38:19.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.221+0000 7f563af27740 1 -- 172.21.15.33:0/4048053391 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56088f41d850 con 0x56088f4c5520 2024-09-16T00:38:19.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.222+0000 7f563259c640 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56088f4c5520 0x56088f4c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.222+0000 7f563259c640 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56088f4c5520 0x56088f4c79a0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f1172ff48e8eb3 server_cookie=6e0aa75b42b9aa6c in_seq=0 out_seq=0 2024-09-16T00:38:19.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.222+0000 7f5619ffb640 10 client.28574 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:19.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.224+0000 7f5619ffb640 1 -- 172.21.15.33:0/4048053391 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56088f41d850 con 0x56088f4c5520 2024-09-16T00:38:19.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.224+0000 7f5619ffb640 10 client.28574 handle_command_reply: tid=0 2024-09-16T00:38:19.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.224+0000 7f563af27740 1 client.28574 shutdown 2024-09-16T00:38:19.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.225+0000 7f563af27740 1 -- 172.21.15.33:0/4048053391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aa4a0 msgr2=0x56088f4b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:19.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.225+0000 7f563af27740 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aa4a0 0x56088f4b4e60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f562800b670 tx=0x7f5628005f90 comp rx=0 tx=0).stop 2024-09-16T00:38:19.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.225+0000 7f563af27740 1 -- 172.21.15.33:0/4048053391 shutdown_connections 2024-09-16T00:38:19.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.225+0000 7f563af27740 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56088f4c5520 0x56088f4c79a0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.225+0000 7f563af27740 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4b5680 0x56088f4b26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.225+0000 7f563af27740 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4aadc0 0x56088f4b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.225+0000 7f563af27740 1 --2- 172.21.15.33:0/4048053391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aa4a0 0x56088f4b4e60 secure :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f562800b670 tx=0x7f5628005f90 comp rx=0 tx=0).stop 2024-09-16T00:38:19.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.225+0000 7f563af27740 1 -- 172.21.15.33:0/4048053391 >> 172.21.15.33:0/4048053391 conn(0x56088f4a6a30 msgr2=0x56088f4ac760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:19.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.225+0000 7f5619ffb640 0 client.28574 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:19.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.226+0000 7f563af27740 1 -- 172.21.15.33:0/4048053391 shutdown_connections 2024-09-16T00:38:19.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.226+0000 7f563af27740 1 -- 172.21.15.33:0/4048053391 wait complete. 2024-09-16T00:38:19.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.227+0000 7f563af27740 20 client.28574 trim_cache size 0 max 16384 2024-09-16T00:38:19.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.265+0000 7f563af27740 1 Processor -- start 2024-09-16T00:38:19.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.265+0000 7f563af27740 1 -- start start 2024-09-16T00:38:19.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.265+0000 7f563af27740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4aadc0 0x56088f4c15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.265+0000 7f563af27740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4b5680 0x56088f4a6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.265+0000 7f563af27740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4ad7a0 0x56088f4cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.265+0000 7f563af27740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f56341b80e0 con 0x56088f4aadc0 2024-09-16T00:38:19.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.265+0000 7f563af27740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f56341b59b0 con 0x56088f4b5680 2024-09-16T00:38:19.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.265+0000 7f563af27740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f56341b3800 con 0x56088f4ad7a0 2024-09-16T00:38:19.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.266+0000 7f5631d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4aadc0 0x56088f4c15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.266+0000 7f5631d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4aadc0 0x56088f4c15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41138/0 (socket says 172.21.15.33:41138) 2024-09-16T00:38:19.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.266+0000 7f5631d9b640 1 -- 172.21.15.33:0/1064631954 learned_addr learned my addr 172.21.15.33:0/1064631954 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:19.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.266+0000 7f5631d9b640 1 --2- 172.21.15.33:0/1064631954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4ad7a0 0x56088f4cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.266+0000 7f5631d9b640 1 -- 172.21.15.33:0/1064631954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4ad7a0 msgr2=0x56088f4cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:19.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.266+0000 7f5631d9b640 1 --2- 172.21.15.33:0/1064631954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4ad7a0 0x56088f4cd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.266+0000 7f5631d9b640 1 -- 172.21.15.33:0/1064631954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4b5680 msgr2=0x56088f4a6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:19.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.266+0000 7f5631d9b640 1 --2- 172.21.15.33:0/1064631954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4b5680 0x56088f4a6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.266+0000 7f5631d9b640 1 -- 172.21.15.33:0/1064631954 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56088f4b1110 con 0x56088f4aadc0 2024-09-16T00:38:19.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.267+0000 7f5631d9b640 1 --2- 172.21.15.33:0/1064631954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4aadc0 0x56088f4c15e0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f56280058c0 tx=0x7f5628005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e120a9fecb1531 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:19.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.267+0000 7f561bfff640 1 -- 172.21.15.33:0/1064631954 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628019af0 con 0x56088f4aadc0 2024-09-16T00:38:19.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.267+0000 7f561bfff640 1 -- 172.21.15.33:0/1064631954 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f562802b070 con 0x56088f4aadc0 2024-09-16T00:38:19.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.267+0000 7f561bfff640 1 -- 172.21.15.33:0/1064631954 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56280196e0 con 0x56088f4aadc0 2024-09-16T00:38:19.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.268+0000 7f563af27740 1 -- 172.21.15.33:0/1064631954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4aadc0 msgr2=0x56088f4c15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:19.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.268+0000 7f563af27740 1 --2- 172.21.15.33:0/1064631954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4aadc0 0x56088f4c15e0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f56280058c0 tx=0x7f5628005850 comp rx=0 tx=0).stop 2024-09-16T00:38:19.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.269+0000 7f563af27740 1 -- 172.21.15.33:0/1064631954 shutdown_connections 2024-09-16T00:38:19.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.269+0000 7f563af27740 1 --2- 172.21.15.33:0/1064631954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4ad7a0 0x56088f4cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.269+0000 7f563af27740 1 --2- 172.21.15.33:0/1064631954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4b5680 0x56088f4a6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.269+0000 7f563af27740 1 --2- 172.21.15.33:0/1064631954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4aadc0 0x56088f4c15e0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.269+0000 7f563af27740 1 -- 172.21.15.33:0/1064631954 >> 172.21.15.33:0/1064631954 conn(0x56088f4a6a30 msgr2=0x56088f4ad090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:19.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.270+0000 7f563af27740 1 -- 172.21.15.33:0/1064631954 shutdown_connections 2024-09-16T00:38:19.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.271+0000 7f563af27740 1 -- 172.21.15.33:0/1064631954 wait complete. 2024-09-16T00:38:19.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.271+0000 7f563af27740 1 -- start start 2024-09-16T00:38:19.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.271+0000 7f563af27740 1 Processor -- start 2024-09-16T00:38:19.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.272+0000 7f563af27740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aadc0 0x56088f4bdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.272+0000 7f563af27740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4ad7a0 0x56088f4b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.272+0000 7f563af27740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4b5680 0x56088f4b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.272+0000 7f563af27740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f56341b3e70 con 0x56088f4ad7a0 2024-09-16T00:38:19.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.272+0000 7f563af27740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f56341b52e0 con 0x56088f4aadc0 2024-09-16T00:38:19.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.272+0000 7f563af27740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f56341b4540 con 0x56088f4b5680 2024-09-16T00:38:19.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.272+0000 7f563af27740 10 client.0 _ll_register_callbacks cb 0x7f562800bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:38:19.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.272+0000 7f5631d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aadc0 0x56088f4bdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.272+0000 7f5631d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aadc0 0x56088f4bdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:53686/0 (socket says 172.21.15.33:53686) 2024-09-16T00:38:19.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.272+0000 7f5631d9b640 1 -- 172.21.15.33:0/616795881 learned_addr learned my addr 172.21.15.33:0/616795881 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:19.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.273+0000 7f563259c640 1 --2- 172.21.15.33:0/616795881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4ad7a0 0x56088f4b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.273+0000 7f5631d9b640 1 -- 172.21.15.33:0/616795881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4b5680 msgr2=0x56088f4b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:19.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.273+0000 7f5631d9b640 1 --2- 172.21.15.33:0/616795881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4b5680 0x56088f4b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.273+0000 7f5631d9b640 1 -- 172.21.15.33:0/616795881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4ad7a0 msgr2=0x56088f4b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:19.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.273+0000 7f5631d9b640 1 --2- 172.21.15.33:0/616795881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4ad7a0 0x56088f4b6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.273+0000 7f5631d9b640 1 -- 172.21.15.33:0/616795881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56088f4b7390 con 0x56088f4aadc0 2024-09-16T00:38:19.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.273+0000 7f5631d9b640 1 --2- 172.21.15.33:0/616795881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aadc0 0x56088f4bdf20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5628019ec0 tx=0x7f56280193e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad20332e05821412 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.273+0000 7f5619ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:38:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.274+0000 7f5619ffb640 1 -- 172.21.15.33:0/616795881 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628005230 con 0x56088f4aadc0 2024-09-16T00:38:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.274+0000 7f563af27740 1 -- 172.21.15.33:0/616795881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f562802b070 con 0x56088f4aadc0 2024-09-16T00:38:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.274+0000 7f5619ffb640 1 -- 172.21.15.33:0/616795881 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f56280053d0 con 0x56088f4aadc0 2024-09-16T00:38:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.274+0000 7f5619ffb640 1 -- 172.21.15.33:0/616795881 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f56280196c0 con 0x56088f4aadc0 2024-09-16T00:38:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.274+0000 7f563af27740 10 client.28577 fetch_fsmap learned FSMap version 33 2024-09-16T00:38:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.274+0000 7f563af27740 1 -- 172.21.15.33:0/616795881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56088f5303a0 con 0x56088f4aadc0 2024-09-16T00:38:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.274+0000 7f5619ffb640 1 -- 172.21.15.33:0/616795881 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f562802b070 con 0x56088f4aadc0 2024-09-16T00:38:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.275+0000 7f5619ffb640 1 -- 172.21.15.33:0/616795881 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f56280404c0 con 0x56088f4aadc0 2024-09-16T00:38:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.275+0000 7f563af27740 10 client.28577 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:38:19.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.275+0000 7f563af27740 10 client.28577 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:38:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.275+0000 7f563af27740 20 client.28577 populate_metadata read hostname 'smithi033' 2024-09-16T00:38:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.275+0000 7f563af27740 1 --2- 172.21.15.33:0/616795881 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56088f530880 0x56088f532c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.275+0000 7f563af27740 4 client.28577 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:38:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.275+0000 7f563af27740 1 -- 172.21.15.33:0/616795881 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56088f417d60 con 0x56088f530880 2024-09-16T00:38:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.275+0000 7f563259c640 1 --2- 172.21.15.33:0/616795881 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56088f530880 0x56088f532c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:19.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.276+0000 7f563259c640 1 --2- 172.21.15.33:0/616795881 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56088f530880 0x56088f532c70 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a87a3234bb061aae server_cookie=76771598ae7b4afd in_seq=0 out_seq=0 2024-09-16T00:38:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.277+0000 7f5619ffb640 10 client.28577 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.278+0000 7f5619ffb640 1 -- 172.21.15.33:0/616795881 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56088f417d60 con 0x56088f530880 2024-09-16T00:38:19.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.278+0000 7f5619ffb640 10 client.28577 handle_command_reply: tid=0 2024-09-16T00:38:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.279+0000 7f563af27740 1 client.28577 shutdown 2024-09-16T00:38:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.280+0000 7f563af27740 1 -- 172.21.15.33:0/616795881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aadc0 msgr2=0x56088f4bdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:19.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.280+0000 7f563af27740 1 --2- 172.21.15.33:0/616795881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aadc0 0x56088f4bdf20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5628019ec0 tx=0x7f56280193e0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.280+0000 7f563af27740 1 -- 172.21.15.33:0/616795881 shutdown_connections 2024-09-16T00:38:19.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.280+0000 7f563af27740 1 --2- 172.21.15.33:0/616795881 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56088f530880 0x56088f532c70 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.280+0000 7f563af27740 1 --2- 172.21.15.33:0/616795881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56088f4b5680 0x56088f4b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.280+0000 7f563af27740 1 --2- 172.21.15.33:0/616795881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56088f4ad7a0 0x56088f4b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.280+0000 7f563af27740 1 --2- 172.21.15.33:0/616795881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56088f4aadc0 0x56088f4bdf20 secure :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5628019ec0 tx=0x7f56280193e0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.280+0000 7f563af27740 1 -- 172.21.15.33:0/616795881 >> 172.21.15.33:0/616795881 conn(0x56088f4a6a30 msgr2=0x56088f4a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.280+0000 7f5619ffb640 0 client.28577 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.281+0000 7f563af27740 1 -- 172.21.15.33:0/616795881 shutdown_connections 2024-09-16T00:38:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.281+0000 7f563af27740 1 -- 172.21.15.33:0/616795881 wait complete. 2024-09-16T00:38:19.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.281+0000 7f563af27740 20 client.28577 trim_cache size 0 max 16384 2024-09-16T00:38:19.284 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:38:19.284 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:38:19.284 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:38:19.284 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:38:19.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.282+0000 7f5638b4e640 1 -- 172.21.15.33:0/3719820897 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f561c0a3410 msgr2=0x7f561c0a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:19.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.282+0000 7f5638b4e640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f561c0a3410 0x7f561c0a58d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f562c0099b0 tx=0x7f562c004a40 comp rx=0 tx=0).stop 2024-09-16T00:38:19.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.282+0000 7f5638b4e640 1 -- 172.21.15.33:0/3719820897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f563410ea80 msgr2=0x7f563419e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:19.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.282+0000 7f5638b4e640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f563410ea80 0x7f563419e890 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f562c00b970 tx=0x7f562c00be30 comp rx=0 tx=0).stop 2024-09-16T00:38:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.283+0000 7f5638b4e640 1 -- 172.21.15.33:0/3719820897 shutdown_connections 2024-09-16T00:38:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.283+0000 7f5638b4e640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f561c0a3410 0x7f561c0a58d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.283+0000 7f5638b4e640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f563410ea80 0x7f563419e890 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.283+0000 7f5638b4e640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5634109e00 0x7f56341a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.283+0000 7f5638b4e640 1 --2- 172.21.15.33:0/3719820897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5634109450 0x7f56341a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.283+0000 7f5638b4e640 1 -- 172.21.15.33:0/3719820897 >> 172.21.15.33:0/3719820897 conn(0x7f563406e2d0 msgr2=0x7f563410c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.283+0000 7f5638b4e640 1 -- 172.21.15.33:0/3719820897 shutdown_connections 2024-09-16T00:38:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:19.283+0000 7f5638b4e640 1 -- 172.21.15.33:0/3719820897 wait complete. 2024-09-16T00:38:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:19 smithi083 ceph-mon[35242]: pgmap v985: 129 pgs: 129 active+clean; 630 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 8.8 MiB/s wr, 455 op/s 2024-09-16T00:38:19.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:19 smithi060 ceph-mon[35185]: pgmap v985: 129 pgs: 129 active+clean; 630 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 8.8 MiB/s wr, 455 op/s 2024-09-16T00:38:20.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:19 smithi033 ceph-mon[31243]: pgmap v985: 129 pgs: 129 active+clean; 630 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 8.8 MiB/s wr, 455 op/s 2024-09-16T00:38:20.091 INFO:tasks.workunit.client.0.smithi033.stdout: 10 149027 17.38 MB/sec execute 326 sec latency 77.211 ms 2024-09-16T00:38:20.262 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:38:20.263 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:38:20.579 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:38:21.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.074+0000 7fb9777ca640 1 -- 172.21.15.33:0/3871555623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb970102d30 msgr2=0x7fb970103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:21.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.074+0000 7fb9777ca640 1 --2- 172.21.15.33:0/3871555623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb970102d30 0x7fb970103110 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009900 tx=0x7fb96c02a480 comp rx=0 tx=0).stop 2024-09-16T00:38:21.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.074+0000 7fb9777ca640 1 -- 172.21.15.33:0/3871555623 shutdown_connections 2024-09-16T00:38:21.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.074+0000 7fb9777ca640 1 --2- 172.21.15.33:0/3871555623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb97010c160 0x7fb97010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:21.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.074+0000 7fb9777ca640 1 --2- 172.21.15.33:0/3871555623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb970103650 0x7fb97010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:21.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.074+0000 7fb9777ca640 1 --2- 172.21.15.33:0/3871555623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb970102d30 0x7fb970103110 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:21.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.074+0000 7fb9777ca640 1 -- 172.21.15.33:0/3871555623 >> 172.21.15.33:0/3871555623 conn(0x7fb9700fc7d0 msgr2=0x7fb9700febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:21.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.075+0000 7fb9777ca640 1 -- 172.21.15.33:0/3871555623 shutdown_connections 2024-09-16T00:38:21.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.075+0000 7fb9777ca640 1 -- 172.21.15.33:0/3871555623 wait complete. 2024-09-16T00:38:21.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.076+0000 7fb9777ca640 1 Processor -- start 2024-09-16T00:38:21.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.076+0000 7fb9777ca640 1 -- start start 2024-09-16T00:38:21.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.076+0000 7fb9777ca640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb970102d30 0x7fb970196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:21.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.077+0000 7fb9777ca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb970103650 0x7fb970196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:21.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.077+0000 7fb9777ca640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb97010c160 0x7fb97019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:21.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.077+0000 7fb9777ca640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb97010fea0 con 0x7fb97010c160 2024-09-16T00:38:21.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.077+0000 7fb9777ca640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb970102860 con 0x7fb970103650 2024-09-16T00:38:21.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.077+0000 7fb9777ca640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb970110020 con 0x7fb970102d30 2024-09-16T00:38:21.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.077+0000 7fb975566640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb970102d30 0x7fb970196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:21.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.077+0000 7fb975566640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb970102d30 0x7fb970196310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:43132/0 (socket says 172.21.15.33:43132) 2024-09-16T00:38:21.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.077+0000 7fb975d67640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb97010c160 0x7fb97019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:21.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.077+0000 7fb975566640 1 -- 172.21.15.33:0/3390506813 learned_addr learned my addr 172.21.15.33:0/3390506813 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:21.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.078+0000 7fb975566640 1 -- 172.21.15.33:0/3390506813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb970103650 msgr2=0x7fb970196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:21.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.078+0000 7fb975566640 1 --2- 172.21.15.33:0/3390506813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb970103650 0x7fb970196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:21.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.078+0000 7fb975566640 1 -- 172.21.15.33:0/3390506813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb97010c160 msgr2=0x7fb97019abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:21.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.078+0000 7fb975566640 1 --2- 172.21.15.33:0/3390506813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb97010c160 0x7fb97019abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:21.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.078+0000 7fb975566640 1 -- 172.21.15.33:0/3390506813 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97019b2c0 con 0x7fb970102d30 2024-09-16T00:38:21.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.078+0000 7fb975566640 1 --2- 172.21.15.33:0/3390506813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb970102d30 0x7fb970196310 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb96c006010 tx=0x7fb96c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:21.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.080+0000 7fb9667fc640 1 -- 172.21.15.33:0/3390506813 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c03f030 con 0x7fb970102d30 2024-09-16T00:38:21.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.080+0000 7fb9667fc640 1 -- 172.21.15.33:0/3390506813 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb96c003ce0 con 0x7fb970102d30 2024-09-16T00:38:21.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.081+0000 7fb9667fc640 1 -- 172.21.15.33:0/3390506813 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c035070 con 0x7fb970102d30 2024-09-16T00:38:21.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.081+0000 7fb9777ca640 1 -- 172.21.15.33:0/3390506813 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb97019b550 con 0x7fb970102d30 2024-09-16T00:38:21.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.081+0000 7fb9777ca640 1 -- 172.21.15.33:0/3390506813 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb97019b940 con 0x7fb970102d30 2024-09-16T00:38:21.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.083+0000 7fb9777ca640 1 -- 172.21.15.33:0/3390506813 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb97006b070 con 0x7fb970102d30 2024-09-16T00:38:21.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.084+0000 7fb9667fc640 1 -- 172.21.15.33:0/3390506813 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb96c042030 con 0x7fb970102d30 2024-09-16T00:38:21.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.085+0000 7fb9667fc640 1 --2- 172.21.15.33:0/3390506813 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb95c0a75f0 0x7fb95c0a9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:21.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.085+0000 7fb974d65640 1 --2- 172.21.15.33:0/3390506813 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb95c0a75f0 0x7fb95c0a9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:21.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.085+0000 7fb9667fc640 1 -- 172.21.15.33:0/3390506813 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb96c0f0490 con 0x7fb970102d30 2024-09-16T00:38:21.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.086+0000 7fb974d65640 1 --2- 172.21.15.33:0/3390506813 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb95c0a75f0 0x7fb95c0a9ab0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb960009460 tx=0x7fb9600093b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:21.096 INFO:tasks.workunit.client.0.smithi033.stdout: 10 149502 17.40 MB/sec execute 327 sec latency 248.721 ms 2024-09-16T00:38:21.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.093+0000 7fb9667fc640 1 -- 172.21.15.33:0/3390506813 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb96c0bcd30 con 0x7fb970102d30 2024-09-16T00:38:21.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.370+0000 7fb9777ca640 1 -- 172.21.15.33:0/3390506813 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb970197670 con 0x7fb970102d30 2024-09-16T00:38:21.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.371+0000 7fb9667fc640 1 -- 172.21.15.33:0/3390506813 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7fb96c0bc630 con 0x7fb970102d30 2024-09-16T00:38:21.372 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:38:21.372 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:38:21.373 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:38:21.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.373+0000 7fb9777ca640 1 -- 172.21.15.33:0/3390506813 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb95c0a75f0 msgr2=0x7fb95c0a9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:21.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.373+0000 7fb9777ca640 1 --2- 172.21.15.33:0/3390506813 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb95c0a75f0 0x7fb95c0a9ab0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb960009460 tx=0x7fb9600093b0 comp rx=0 tx=0).stop 2024-09-16T00:38:21.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.373+0000 7fb9777ca640 1 -- 172.21.15.33:0/3390506813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb970102d30 msgr2=0x7fb970196310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:21.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.373+0000 7fb9777ca640 1 --2- 172.21.15.33:0/3390506813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb970102d30 0x7fb970196310 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb96c006010 tx=0x7fb96c0042f0 comp rx=0 tx=0).stop 2024-09-16T00:38:21.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.373+0000 7fb9777ca640 1 -- 172.21.15.33:0/3390506813 shutdown_connections 2024-09-16T00:38:21.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.373+0000 7fb9777ca640 1 --2- 172.21.15.33:0/3390506813 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb95c0a75f0 0x7fb95c0a9ab0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:21.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.373+0000 7fb9777ca640 1 --2- 172.21.15.33:0/3390506813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb97010c160 0x7fb97019abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:21.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.373+0000 7fb9777ca640 1 --2- 172.21.15.33:0/3390506813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb970103650 0x7fb970196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:21.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.373+0000 7fb9777ca640 1 --2- 172.21.15.33:0/3390506813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb970102d30 0x7fb970196310 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:21.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.373+0000 7fb9777ca640 1 -- 172.21.15.33:0/3390506813 >> 172.21.15.33:0/3390506813 conn(0x7fb9700fc7d0 msgr2=0x7fb9700fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:21.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.373+0000 7fb9777ca640 1 -- 172.21.15.33:0/3390506813 shutdown_connections 2024-09-16T00:38:21.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:21.373+0000 7fb9777ca640 1 -- 172.21.15.33:0/3390506813 wait complete. 2024-09-16T00:38:21.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:21 smithi033 ceph-mon[31243]: pgmap v986: 129 pgs: 129 active+clean; 635 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 9.6 MiB/s wr, 562 op/s 2024-09-16T00:38:21.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:21 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3390506813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:38:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:21 smithi083 ceph-mon[35242]: pgmap v986: 129 pgs: 129 active+clean; 635 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 9.6 MiB/s wr, 562 op/s 2024-09-16T00:38:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:21 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3390506813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:38:21.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:21 smithi060 ceph-mon[35185]: pgmap v986: 129 pgs: 129 active+clean; 635 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 9.6 MiB/s wr, 562 op/s 2024-09-16T00:38:21.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:21 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3390506813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:38:22.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 149967 17.42 MB/sec execute 328 sec latency 226.494 ms 2024-09-16T00:38:22.376 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:38:22.651 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:38:23.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 150494 17.45 MB/sec execute 329 sec latency 285.525 ms 2024-09-16T00:38:23.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.181+0000 7ff78e8ff640 1 -- 172.21.15.33:0/1644608856 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff788102790 msgr2=0x7ff7881067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:23.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.181+0000 7ff78e8ff640 1 --2- 172.21.15.33:0/1644608856 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff788102790 0x7ff7881067e0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009900 tx=0x7ff77c02a440 comp rx=0 tx=0).stop 2024-09-16T00:38:23.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.181+0000 7ff78e8ff640 1 -- 172.21.15.33:0/1644608856 shutdown_connections 2024-09-16T00:38:23.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.181+0000 7ff78e8ff640 1 --2- 172.21.15.33:0/1644608856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff788106f10 0x7ff78810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.181+0000 7ff78e8ff640 1 --2- 172.21.15.33:0/1644608856 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff788102790 0x7ff7881067e0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.181+0000 7ff78e8ff640 1 --2- 172.21.15.33:0/1644608856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff788101de0 0x7ff7881021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.181+0000 7ff78e8ff640 1 -- 172.21.15.33:0/1644608856 >> 172.21.15.33:0/1644608856 conn(0x7ff7880fd8c0 msgr2=0x7ff7880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:23.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.182+0000 7ff78e8ff640 1 -- 172.21.15.33:0/1644608856 shutdown_connections 2024-09-16T00:38:23.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.182+0000 7ff78e8ff640 1 -- 172.21.15.33:0/1644608856 wait complete. 2024-09-16T00:38:23.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.182+0000 7ff78e8ff640 1 Processor -- start 2024-09-16T00:38:23.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.183+0000 7ff78e8ff640 1 -- start start 2024-09-16T00:38:23.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.183+0000 7ff78e8ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff788101de0 0x7ff78819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.183+0000 7ff78e8ff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff788102790 0x7ff78819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.183+0000 7ff78e8ff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff788106f10 0x7ff78819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.183+0000 7ff78e8ff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff788111fe0 con 0x7ff788102790 2024-09-16T00:38:23.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.183+0000 7ff78e8ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff788111e60 con 0x7ff788101de0 2024-09-16T00:38:23.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.183+0000 7ff78e8ff640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff788112160 con 0x7ff788106f10 2024-09-16T00:38:23.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.183+0000 7ff787fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff788101de0 0x7ff78819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.184+0000 7ff787fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff788101de0 0x7ff78819a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40222/0 (socket says 172.21.15.33:40222) 2024-09-16T00:38:23.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.184+0000 7ff787fff640 1 -- 172.21.15.33:0/405026531 learned_addr learned my addr 172.21.15.33:0/405026531 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:23.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.184+0000 7ff7877fe640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff788102790 0x7ff78819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.184+0000 7ff787fff640 1 -- 172.21.15.33:0/405026531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff788106f10 msgr2=0x7ff78819eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T00:38:23.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.184+0000 7ff787fff640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff788106f10 0x7ff78819eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.184+0000 7ff787fff640 1 -- 172.21.15.33:0/405026531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff788102790 msgr2=0x7ff78819ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:23.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.184+0000 7ff787fff640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff788102790 0x7ff78819ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.184+0000 7ff787fff640 1 -- 172.21.15.33:0/405026531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78819f6d0 con 0x7ff788101de0 2024-09-16T00:38:23.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.186+0000 7ff787fff640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff788101de0 0x7ff78819a720 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff77800c890 tx=0x7ff77800cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:23.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.187+0000 7ff7857fa640 1 -- 172.21.15.33:0/405026531 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff778004220 con 0x7ff788101de0 2024-09-16T00:38:23.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.187+0000 7ff78e8ff640 1 -- 172.21.15.33:0/405026531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78819f9c0 con 0x7ff788101de0 2024-09-16T00:38:23.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.187+0000 7ff78e8ff640 1 -- 172.21.15.33:0/405026531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7881a72a0 con 0x7ff788101de0 2024-09-16T00:38:23.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.187+0000 7ff7857fa640 1 -- 172.21.15.33:0/405026531 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7780043c0 con 0x7ff788101de0 2024-09-16T00:38:23.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.187+0000 7ff7857fa640 1 -- 172.21.15.33:0/405026531 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff778004a00 con 0x7ff788101de0 2024-09-16T00:38:23.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.191+0000 7ff7857fa640 1 -- 172.21.15.33:0/405026531 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff7780079b0 con 0x7ff788101de0 2024-09-16T00:38:23.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.191+0000 7ff790cd8740 1 Processor -- start 2024-09-16T00:38:23.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.191+0000 7ff790cd8740 1 -- start start 2024-09-16T00:38:23.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.192+0000 7ff790cd8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8c7b40 0x5647eb8c6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.192+0000 7ff790cd8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c65d0 0x5647eb8c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.192+0000 7ff790cd8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8d1370 0x5647eb8d3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.192+0000 7ff790cd8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5647eb82c2f0 con 0x5647eb8d1370 2024-09-16T00:38:23.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.192+0000 7ff790cd8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5647eb7d3cf0 con 0x5647eb8c65d0 2024-09-16T00:38:23.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.192+0000 7ff790cd8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5647eb8282b0 con 0x5647eb8c7b40 2024-09-16T00:38:23.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.192+0000 7ff7877fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8d1370 0x5647eb8d3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.192+0000 7ff7877fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8d1370 0x5647eb8d3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43988/0 (socket says 172.21.15.33:43988) 2024-09-16T00:38:23.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.192+0000 7ff7877fe640 1 -- 172.21.15.33:0/2579880125 learned_addr learned my addr 172.21.15.33:0/2579880125 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:23.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.192+0000 7ff787fff640 1 --2- 172.21.15.33:0/2579880125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c65d0 0x5647eb8c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.192+0000 7ff78ce9c640 1 --2- 172.21.15.33:0/2579880125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8c7b40 0x5647eb8c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.193+0000 7ff7877fe640 1 -- 172.21.15.33:0/2579880125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8c7b40 msgr2=0x5647eb8c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:23.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.193+0000 7ff7877fe640 1 --2- 172.21.15.33:0/2579880125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8c7b40 0x5647eb8c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.193+0000 7ff7877fe640 1 -- 172.21.15.33:0/2579880125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c65d0 msgr2=0x5647eb8c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:23.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.193+0000 7ff7877fe640 1 --2- 172.21.15.33:0/2579880125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c65d0 0x5647eb8c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.193+0000 7ff7877fe640 1 -- 172.21.15.33:0/2579880125 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647eb7b83a0 con 0x5647eb8d1370 2024-09-16T00:38:23.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.194+0000 7ff7857fa640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7700a3380 0x7ff7700a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.194+0000 7ff7857fa640 1 -- 172.21.15.33:0/405026531 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff7780cfa30 con 0x7ff788101de0 2024-09-16T00:38:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.194+0000 7ff78ce9c640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7700a3380 0x7ff7700a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.195+0000 7ff7877fe640 1 --2- 172.21.15.33:0/2579880125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8d1370 0x5647eb8d3760 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009900 tx=0x7ff77c009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76d98cd1ae2e231 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.196+0000 7ff76e7fc640 1 -- 172.21.15.33:0/2579880125 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c03f070 con 0x5647eb8d1370 2024-09-16T00:38:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.196+0000 7ff76e7fc640 1 -- 172.21.15.33:0/2579880125 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff77c003ad0 con 0x5647eb8d1370 2024-09-16T00:38:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.196+0000 7ff76e7fc640 1 -- 172.21.15.33:0/2579880125 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c035070 con 0x5647eb8d1370 2024-09-16T00:38:23.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.197+0000 7ff78ce9c640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7700a3380 0x7ff7700a5840 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff780009e20 tx=0x7ff780009410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:23.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.197+0000 7ff790cd8740 1 -- 172.21.15.33:0/2579880125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8d1370 msgr2=0x5647eb8d3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:23.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.197+0000 7ff790cd8740 1 --2- 172.21.15.33:0/2579880125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8d1370 0x5647eb8d3760 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009900 tx=0x7ff77c009680 comp rx=0 tx=0).stop 2024-09-16T00:38:23.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.197+0000 7ff790cd8740 1 -- 172.21.15.33:0/2579880125 shutdown_connections 2024-09-16T00:38:23.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.197+0000 7ff790cd8740 1 --2- 172.21.15.33:0/2579880125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8d1370 0x5647eb8d3760 secure :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009900 tx=0x7ff77c009680 comp rx=0 tx=0).stop 2024-09-16T00:38:23.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.197+0000 7ff790cd8740 1 --2- 172.21.15.33:0/2579880125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c65d0 0x5647eb8c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.197+0000 7ff790cd8740 1 --2- 172.21.15.33:0/2579880125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8c7b40 0x5647eb8c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.198+0000 7ff790cd8740 1 -- 172.21.15.33:0/2579880125 >> 172.21.15.33:0/2579880125 conn(0x5647eb88f310 msgr2=0x5647eb8c2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:23.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.200+0000 7ff790cd8740 1 -- 172.21.15.33:0/2579880125 shutdown_connections 2024-09-16T00:38:23.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.201+0000 7ff790cd8740 1 -- 172.21.15.33:0/2579880125 wait complete. 2024-09-16T00:38:23.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.202+0000 7ff790cd8740 1 -- start start 2024-09-16T00:38:23.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.202+0000 7ff790cd8740 1 Processor -- start 2024-09-16T00:38:23.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.203+0000 7ff790cd8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c65d0 0x5647eb8ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.203+0000 7ff790cd8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c7b40 0x5647eb8ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff790cd8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8cecc0 0x5647eb8dd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff790cd8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5647eb8d4010 con 0x5647eb8c65d0 2024-09-16T00:38:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff790cd8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5647eb7e7060 con 0x5647eb8c7b40 2024-09-16T00:38:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff790cd8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5647eb8d4190 con 0x5647eb8cecc0 2024-09-16T00:38:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff787fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c65d0 0x5647eb8ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff787fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c65d0 0x5647eb8ce240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43990/0 (socket says 172.21.15.33:43990) 2024-09-16T00:38:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff787fff640 1 -- 172.21.15.33:0/1211131291 learned_addr learned my addr 172.21.15.33:0/1211131291 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff790cd8740 10 client.0 _ll_register_callbacks cb 0x5647eb53a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:38:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff7877fe640 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c7b40 0x5647eb8ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff78ce9c640 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8cecc0 0x5647eb8dd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff787fff640 1 -- 172.21.15.33:0/1211131291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8cecc0 msgr2=0x5647eb8dd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff787fff640 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8cecc0 0x5647eb8dd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff787fff640 1 -- 172.21.15.33:0/1211131291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c7b40 msgr2=0x5647eb8ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff787fff640 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c7b40 0x5647eb8ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff787fff640 1 -- 172.21.15.33:0/1211131291 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647eb59d3c0 con 0x5647eb8c65d0 2024-09-16T00:38:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff7877fe640 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c7b40 0x5647eb8ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:38:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff787fff640 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c65d0 0x5647eb8ce240 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7ff77809bb00 tx=0x7ff77800a5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc5b75973fa8cfb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.204+0000 7ff7677fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:38:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.205+0000 7ff7677fe640 1 -- 172.21.15.33:0/1211131291 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77801d040 con 0x5647eb8c65d0 2024-09-16T00:38:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.205+0000 7ff7677fe640 1 -- 172.21.15.33:0/1211131291 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff778007ca0 con 0x5647eb8c65d0 2024-09-16T00:38:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.205+0000 7ff790cd8740 1 -- 172.21.15.33:0/1211131291 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff77c003ad0 con 0x5647eb8c65d0 2024-09-16T00:38:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.205+0000 7ff7677fe640 1 -- 172.21.15.33:0/1211131291 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff77809b4d0 con 0x5647eb8c65d0 2024-09-16T00:38:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.205+0000 7ff790cd8740 10 client.19398 fetch_fsmap learned FSMap version 33 2024-09-16T00:38:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.205+0000 7ff790cd8740 1 -- 172.21.15.33:0/1211131291 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5647eb8e01a0 con 0x5647eb8c65d0 2024-09-16T00:38:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.206+0000 7ff7677fe640 1 -- 172.21.15.33:0/1211131291 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff77c003ad0 con 0x5647eb8c65d0 2024-09-16T00:38:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.206+0000 7ff7677fe640 1 -- 172.21.15.33:0/1211131291 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ff7780860c0 con 0x5647eb8c65d0 2024-09-16T00:38:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.207+0000 7ff790cd8740 10 client.19398 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:38:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.207+0000 7ff790cd8740 10 client.19398 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:38:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.207+0000 7ff790cd8740 20 client.19398 populate_metadata read hostname 'smithi033' 2024-09-16T00:38:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.207+0000 7ff790cd8740 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5647eb8e1660 0x5647eb8e3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.207+0000 7ff790cd8740 4 client.19398 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:38:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.207+0000 7ff790cd8740 1 -- 172.21.15.33:0/1211131291 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647eb839860 con 0x5647eb8e1660 2024-09-16T00:38:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.208+0000 7ff7877fe640 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5647eb8e1660 0x5647eb8e3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.208+0000 7ff7877fe640 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5647eb8e1660 0x5647eb8e3ab0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=802b7dcb4df451b server_cookie=6595804b5ddf271f in_seq=0 out_seq=0 2024-09-16T00:38:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.208+0000 7ff7677fe640 10 client.19398 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:38:23.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.212+0000 7ff7677fe640 1 -- 172.21.15.33:0/1211131291 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5647eb839860 con 0x5647eb8e1660 2024-09-16T00:38:23.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.212+0000 7ff7677fe640 10 client.19398 handle_command_reply: tid=0 2024-09-16T00:38:23.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.212+0000 7ff790cd8740 1 client.19398 shutdown 2024-09-16T00:38:23.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.213+0000 7ff790cd8740 1 -- 172.21.15.33:0/1211131291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c65d0 msgr2=0x5647eb8ce240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:23.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.213+0000 7ff790cd8740 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c65d0 0x5647eb8ce240 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7ff77809bb00 tx=0x7ff77800a5a0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.213+0000 7ff790cd8740 1 -- 172.21.15.33:0/1211131291 shutdown_connections 2024-09-16T00:38:23.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.213+0000 7ff790cd8740 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5647eb8e1660 0x5647eb8e3ab0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.213+0000 7ff790cd8740 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8cecc0 0x5647eb8dd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.213+0000 7ff7677fe640 0 client.19398 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:38:23.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.213+0000 7ff790cd8740 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c7b40 0x5647eb8ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.213+0000 7ff790cd8740 1 --2- 172.21.15.33:0/1211131291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c65d0 0x5647eb8ce240 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.213+0000 7ff790cd8740 1 -- 172.21.15.33:0/1211131291 >> 172.21.15.33:0/1211131291 conn(0x5647eb88f310 msgr2=0x5647eb8c8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:23.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.213+0000 7ff790cd8740 1 -- 172.21.15.33:0/1211131291 shutdown_connections 2024-09-16T00:38:23.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.213+0000 7ff790cd8740 1 -- 172.21.15.33:0/1211131291 wait complete. 2024-09-16T00:38:23.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.213+0000 7ff790cd8740 20 client.19398 trim_cache size 0 max 16384 2024-09-16T00:38:23.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.239+0000 7ff790cd8740 1 Processor -- start 2024-09-16T00:38:23.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.239+0000 7ff790cd8740 1 -- start start 2024-09-16T00:38:23.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.240+0000 7ff790cd8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c65d0 0x5647eb8c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.240+0000 7ff790cd8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c7b40 0x5647eb8dd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.240+0000 7ff787fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c65d0 0x5647eb8c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.240+0000 7ff787fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c65d0 0x5647eb8c9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40250/0 (socket says 172.21.15.33:40250) 2024-09-16T00:38:23.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.240+0000 7ff787fff640 1 -- 172.21.15.33:0/3341685661 learned_addr learned my addr 172.21.15.33:0/3341685661 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:23.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.240+0000 7ff790cd8740 1 --2- 172.21.15.33:0/3341685661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8cecc0 0x5647eb8dd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.240+0000 7ff790cd8740 1 -- 172.21.15.33:0/3341685661 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff774012930 con 0x5647eb8c7b40 2024-09-16T00:38:23.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.240+0000 7ff790cd8740 1 -- 172.21.15.33:0/3341685661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff774010200 con 0x5647eb8c65d0 2024-09-16T00:38:23.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.240+0000 7ff790cd8740 1 -- 172.21.15.33:0/3341685661 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff77400e050 con 0x5647eb8cecc0 2024-09-16T00:38:23.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.240+0000 7ff7877fe640 1 --2- 172.21.15.33:0/3341685661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c7b40 0x5647eb8dd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.240+0000 7ff78ce9c640 1 --2- 172.21.15.33:0/3341685661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8cecc0 0x5647eb8dd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.241+0000 7ff78ce9c640 1 -- 172.21.15.33:0/3341685661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c65d0 msgr2=0x5647eb8c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:23.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.241+0000 7ff78ce9c640 1 --2- 172.21.15.33:0/3341685661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c65d0 0x5647eb8c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.241+0000 7ff78ce9c640 1 -- 172.21.15.33:0/3341685661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c7b40 msgr2=0x5647eb8dd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:23.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.241+0000 7ff78ce9c640 1 --2- 172.21.15.33:0/3341685661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c7b40 0x5647eb8dd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.241+0000 7ff78ce9c640 1 -- 172.21.15.33:0/3341685661 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647eb8c2e10 con 0x5647eb8cecc0 2024-09-16T00:38:23.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.241+0000 7ff7877fe640 1 --2- 172.21.15.33:0/3341685661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c7b40 0x5647eb8dd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:38:23.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.242+0000 7ff78ce9c640 1 --2- 172.21.15.33:0/3341685661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8cecc0 0x5647eb8dd880 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff78000cc80 tx=0x7ff780015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=612df22e1a373739 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:23.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.242+0000 7ff76dffb640 1 -- 172.21.15.33:0/3341685661 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff780022400 con 0x5647eb8cecc0 2024-09-16T00:38:23.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.242+0000 7ff76dffb640 1 -- 172.21.15.33:0/3341685661 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff780027070 con 0x5647eb8cecc0 2024-09-16T00:38:23.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.242+0000 7ff76dffb640 1 -- 172.21.15.33:0/3341685661 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff780020980 con 0x5647eb8cecc0 2024-09-16T00:38:23.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.243+0000 7ff790cd8740 1 -- 172.21.15.33:0/3341685661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8cecc0 msgr2=0x5647eb8dd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:23.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.243+0000 7ff790cd8740 1 --2- 172.21.15.33:0/3341685661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8cecc0 0x5647eb8dd880 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff78000cc80 tx=0x7ff780015000 comp rx=0 tx=0).stop 2024-09-16T00:38:23.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.243+0000 7ff790cd8740 1 -- 172.21.15.33:0/3341685661 shutdown_connections 2024-09-16T00:38:23.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.243+0000 7ff790cd8740 1 --2- 172.21.15.33:0/3341685661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8cecc0 0x5647eb8dd880 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.243+0000 7ff790cd8740 1 --2- 172.21.15.33:0/3341685661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c7b40 0x5647eb8dd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.243+0000 7ff790cd8740 1 --2- 172.21.15.33:0/3341685661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8c65d0 0x5647eb8c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.244+0000 7ff790cd8740 1 -- 172.21.15.33:0/3341685661 >> 172.21.15.33:0/3341685661 conn(0x5647eb88f310 msgr2=0x5647eb8c9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:23.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.244+0000 7ff790cd8740 1 -- 172.21.15.33:0/3341685661 shutdown_connections 2024-09-16T00:38:23.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.244+0000 7ff790cd8740 1 -- 172.21.15.33:0/3341685661 wait complete. 2024-09-16T00:38:23.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.244+0000 7ff790cd8740 1 -- start start 2024-09-16T00:38:23.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.245+0000 7ff790cd8740 1 Processor -- start 2024-09-16T00:38:23.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.245+0000 7ff790cd8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8c65d0 0x5647eb8d99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.246+0000 7ff790cd8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c7b40 0x5647eb8d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.246+0000 7ff790cd8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8cecc0 0x5647eb8d2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.246+0000 7ff790cd8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff77400e6c0 con 0x5647eb8c7b40 2024-09-16T00:38:23.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.246+0000 7ff790cd8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff77400fb30 con 0x5647eb8cecc0 2024-09-16T00:38:23.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.246+0000 7ff790cd8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff77400ed90 con 0x5647eb8c65d0 2024-09-16T00:38:23.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.246+0000 7ff790cd8740 10 client.0 _ll_register_callbacks cb 0x7ff7780058c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:38:23.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.247+0000 7ff78ce9c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8cecc0 0x5647eb8d2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.247+0000 7ff78ce9c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8cecc0 0x5647eb8d2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40254/0 (socket says 172.21.15.33:40254) 2024-09-16T00:38:23.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.247+0000 7ff78ce9c640 1 -- 172.21.15.33:0/2240291321 learned_addr learned my addr 172.21.15.33:0/2240291321 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:23.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.248+0000 7ff7877fe640 1 --2- 172.21.15.33:0/2240291321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c7b40 0x5647eb8d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.248+0000 7ff78ce9c640 1 -- 172.21.15.33:0/2240291321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8c65d0 msgr2=0x5647eb8d99d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:38:23.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.248+0000 7ff78ce9c640 1 --2- 172.21.15.33:0/2240291321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8c65d0 0x5647eb8d99d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.248+0000 7ff78ce9c640 1 -- 172.21.15.33:0/2240291321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c7b40 msgr2=0x5647eb8d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:23.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.248+0000 7ff78ce9c640 1 --2- 172.21.15.33:0/2240291321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c7b40 0x5647eb8d1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.248+0000 7ff78ce9c640 1 -- 172.21.15.33:0/2240291321 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647eb8d2900 con 0x5647eb8cecc0 2024-09-16T00:38:23.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.249+0000 7ff78ce9c640 1 --2- 172.21.15.33:0/2240291321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8cecc0 0x5647eb8d2390 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff78000cfd0 tx=0x7ff780004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c8cc41b490d2e93 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:23.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.249+0000 7ff7677fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:38:23.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.250+0000 7ff7677fe640 1 -- 172.21.15.33:0/2240291321 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff780031070 con 0x5647eb8cecc0 2024-09-16T00:38:23.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.250+0000 7ff7677fe640 1 -- 172.21.15.33:0/2240291321 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff780022810 con 0x5647eb8cecc0 2024-09-16T00:38:23.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.253+0000 7ff790cd8740 1 -- 172.21.15.33:0/2240291321 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff780027070 con 0x5647eb8cecc0 2024-09-16T00:38:23.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.253+0000 7ff7677fe640 1 -- 172.21.15.33:0/2240291321 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff78003b050 con 0x5647eb8cecc0 2024-09-16T00:38:23.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.253+0000 7ff790cd8740 10 client.28592 fetch_fsmap learned FSMap version 33 2024-09-16T00:38:23.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.253+0000 7ff790cd8740 1 -- 172.21.15.33:0/2240291321 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5647eb8d8250 con 0x5647eb8cecc0 2024-09-16T00:38:23.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.254+0000 7ff7677fe640 1 -- 172.21.15.33:0/2240291321 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff780027070 con 0x5647eb8cecc0 2024-09-16T00:38:23.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.254+0000 7ff7677fe640 1 -- 172.21.15.33:0/2240291321 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ff780036b70 con 0x5647eb8cecc0 2024-09-16T00:38:23.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.254+0000 7ff790cd8740 10 client.28592 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:38:23.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.254+0000 7ff790cd8740 10 client.28592 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:38:23.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.254+0000 7ff790cd8740 20 client.28592 populate_metadata read hostname 'smithi033' 2024-09-16T00:38:23.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.254+0000 7ff790cd8740 1 --2- 172.21.15.33:0/2240291321 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5647eb8e1660 0x5647eb94e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:23.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.254+0000 7ff790cd8740 4 client.28592 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:38:23.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.254+0000 7ff790cd8740 1 -- 172.21.15.33:0/2240291321 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5647eb833d70 con 0x5647eb8e1660 2024-09-16T00:38:23.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.255+0000 7ff787fff640 1 --2- 172.21.15.33:0/2240291321 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5647eb8e1660 0x5647eb94e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:23.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.256+0000 7ff787fff640 1 --2- 172.21.15.33:0/2240291321 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5647eb8e1660 0x5647eb94e800 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=30a4c89c624ac7ef server_cookie=e21638cfb9825486 in_seq=0 out_seq=0 2024-09-16T00:38:23.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.256+0000 7ff7677fe640 10 client.28592 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:38:23.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.258+0000 7ff7677fe640 1 -- 172.21.15.33:0/2240291321 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5647eb833d70 con 0x5647eb8e1660 2024-09-16T00:38:23.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.258+0000 7ff7677fe640 10 client.28592 handle_command_reply: tid=0 2024-09-16T00:38:23.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.258+0000 7ff790cd8740 1 client.28592 shutdown 2024-09-16T00:38:23.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.258+0000 7ff790cd8740 1 -- 172.21.15.33:0/2240291321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8cecc0 msgr2=0x5647eb8d2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:23.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.258+0000 7ff790cd8740 1 --2- 172.21.15.33:0/2240291321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8cecc0 0x5647eb8d2390 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff78000cfd0 tx=0x7ff780004a40 comp rx=0 tx=0).stop 2024-09-16T00:38:23.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.259+0000 7ff790cd8740 1 -- 172.21.15.33:0/2240291321 shutdown_connections 2024-09-16T00:38:23.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.259+0000 7ff790cd8740 1 --2- 172.21.15.33:0/2240291321 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5647eb8e1660 0x5647eb94e800 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.259+0000 7ff790cd8740 1 --2- 172.21.15.33:0/2240291321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5647eb8cecc0 0x5647eb8d2390 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.259+0000 7ff790cd8740 1 --2- 172.21.15.33:0/2240291321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647eb8c7b40 0x5647eb8d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.259+0000 7ff790cd8740 1 --2- 172.21.15.33:0/2240291321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647eb8c65d0 0x5647eb8d99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.259+0000 7ff7677fe640 0 client.28592 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:38:23.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.259+0000 7ff790cd8740 1 -- 172.21.15.33:0/2240291321 >> 172.21.15.33:0/2240291321 conn(0x5647eb88f310 msgr2=0x5647eb8c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:23.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.261+0000 7ff790cd8740 1 -- 172.21.15.33:0/2240291321 shutdown_connections 2024-09-16T00:38:23.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.261+0000 7ff790cd8740 1 -- 172.21.15.33:0/2240291321 wait complete. 2024-09-16T00:38:23.263 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:38:23.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.262+0000 7ff790cd8740 20 client.28592 trim_cache size 0 max 16384 2024-09-16T00:38:23.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.263+0000 7ff78e8ff640 1 -- 172.21.15.33:0/405026531 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7700a3380 msgr2=0x7ff7700a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:23.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.263+0000 7ff78e8ff640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7700a3380 0x7ff7700a5840 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff780009e20 tx=0x7ff780009410 comp rx=0 tx=0).stop 2024-09-16T00:38:23.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.263+0000 7ff78e8ff640 1 -- 172.21.15.33:0/405026531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff788101de0 msgr2=0x7ff78819a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:23.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.263+0000 7ff78e8ff640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff788101de0 0x7ff78819a720 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff77800c890 tx=0x7ff77800cd50 comp rx=0 tx=0).stop 2024-09-16T00:38:23.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.263+0000 7ff78e8ff640 1 -- 172.21.15.33:0/405026531 shutdown_connections 2024-09-16T00:38:23.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.263+0000 7ff78e8ff640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff7700a3380 0x7ff7700a5840 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.263+0000 7ff78e8ff640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff788106f10 0x7ff78819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.264+0000 7ff78e8ff640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff788102790 0x7ff78819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:23.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.264+0000 7ff78e8ff640 1 --2- 172.21.15.33:0/405026531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff788101de0 0x7ff78819a720 secure :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff77800c890 tx=0x7ff77800cd50 comp rx=0 tx=0).stop 2024-09-16T00:38:23.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.264+0000 7ff78e8ff640 1 -- 172.21.15.33:0/405026531 >> 172.21.15.33:0/405026531 conn(0x7ff7880fd8c0 msgr2=0x7ff7880feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:23.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.264+0000 7ff78e8ff640 1 -- 172.21.15.33:0/405026531 shutdown_connections 2024-09-16T00:38:23.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:23.264+0000 7ff78e8ff640 1 -- 172.21.15.33:0/405026531 wait complete. 2024-09-16T00:38:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:23 smithi083 ceph-mon[35242]: pgmap v987: 129 pgs: 129 active+clean; 641 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.4 MiB/s wr, 497 op/s 2024-09-16T00:38:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:23 smithi060 ceph-mon[35185]: pgmap v987: 129 pgs: 129 active+clean; 641 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.4 MiB/s wr, 497 op/s 2024-09-16T00:38:24.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:23 smithi033 ceph-mon[31243]: pgmap v987: 129 pgs: 129 active+clean; 641 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.4 MiB/s wr, 497 op/s 2024-09-16T00:38:24.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 151035 17.51 MB/sec execute 330 sec latency 91.845 ms 2024-09-16T00:38:24.265 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:38:24.599 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:38:25.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 151540 17.54 MB/sec execute 331 sec latency 88.856 ms 2024-09-16T00:38:25.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.113+0000 7fdb2923e640 1 -- 172.21.15.33:0/1441440954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb24100a90 msgr2=0x7fdb24100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:25.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.113+0000 7fdb2923e640 1 --2- 172.21.15.33:0/1441440954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb24100a90 0x7fdb24100ef0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009890 tx=0x7fdb1802a380 comp rx=0 tx=0).stop 2024-09-16T00:38:25.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.113+0000 7fdb2923e640 1 -- 172.21.15.33:0/1441440954 shutdown_connections 2024-09-16T00:38:25.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.113+0000 7fdb2923e640 1 --2- 172.21.15.33:0/1441440954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb24101430 0x7fdb2410d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.113+0000 7fdb2923e640 1 --2- 172.21.15.33:0/1441440954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb24100a90 0x7fdb24100ef0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.113+0000 7fdb2923e640 1 --2- 172.21.15.33:0/1441440954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb24106a90 0x7fdb24106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.113+0000 7fdb2923e640 1 -- 172.21.15.33:0/1441440954 >> 172.21.15.33:0/1441440954 conn(0x7fdb240fc7d0 msgr2=0x7fdb240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.114+0000 7fdb2923e640 1 -- 172.21.15.33:0/1441440954 shutdown_connections 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.114+0000 7fdb2923e640 1 -- 172.21.15.33:0/1441440954 wait complete. 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.115+0000 7fdb2923e640 1 Processor -- start 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.115+0000 7fdb2923e640 1 -- start start 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.116+0000 7fdb2923e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb24100a90 0x7fdb2410d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.116+0000 7fdb2923e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb24101430 0x7fdb2410a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.116+0000 7fdb2923e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb24106a90 0x7fdb2410ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.116+0000 7fdb2923e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2410fda0 con 0x7fdb24100a90 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.116+0000 7fdb2923e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2410fc20 con 0x7fdb24101430 2024-09-16T00:38:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.116+0000 7fdb2923e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2410ff20 con 0x7fdb24106a90 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.116+0000 7fdb22d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb24100a90 0x7fdb2410d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.117+0000 7fdb22d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb24100a90 0x7fdb2410d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44014/0 (socket says 172.21.15.33:44014) 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.117+0000 7fdb22d9d640 1 -- 172.21.15.33:0/4163228065 learned_addr learned my addr 172.21.15.33:0/4163228065 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.117+0000 7fdb2359e640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb24106a90 0x7fdb2410ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.117+0000 7fdb22d9d640 1 -- 172.21.15.33:0/4163228065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb24106a90 msgr2=0x7fdb2410ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.117+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb24106a90 0x7fdb2410ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.117+0000 7fdb22d9d640 1 -- 172.21.15.33:0/4163228065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb24101430 msgr2=0x7fdb2410a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.117+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb24101430 0x7fdb2410a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.117+0000 7fdb22d9d640 1 -- 172.21.15.33:0/4163228065 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2410b3f0 con 0x7fdb24100a90 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.117+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb24100a90 0x7fdb2410d680 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400e9c0 tx=0x7fdb1400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.119+0000 7fdb0bfff640 1 -- 172.21.15.33:0/4163228065 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1400ccc0 con 0x7fdb24100a90 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.119+0000 7fdb0bfff640 1 -- 172.21.15.33:0/4163228065 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb14005500 con 0x7fdb24100a90 2024-09-16T00:38:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.119+0000 7fdb0bfff640 1 -- 172.21.15.33:0/4163228065 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14010430 con 0x7fdb24100a90 2024-09-16T00:38:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.120+0000 7fdb2923e640 1 -- 172.21.15.33:0/4163228065 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb241a08d0 con 0x7fdb24100a90 2024-09-16T00:38:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.121+0000 7fdb2923e640 1 -- 172.21.15.33:0/4163228065 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb241a0d00 con 0x7fdb24100a90 2024-09-16T00:38:25.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.126+0000 7fdb2b617740 1 Processor -- start 2024-09-16T00:38:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.127+0000 7fdb2b617740 1 -- start start 2024-09-16T00:38:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.127+0000 7fdb2b617740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf98b40 0x564aabf97090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.127+0000 7fdb2b617740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf975d0 0x564aabf97a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.127+0000 7fdb2b617740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabfb3380 0x564aabfb5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.127+0000 7fdb2b617740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564aabefd2f0 con 0x564aabfb3380 2024-09-16T00:38:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.127+0000 7fdb2b617740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564aabea4cf0 con 0x564aabf975d0 2024-09-16T00:38:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.127+0000 7fdb2b617740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564aabef92b0 con 0x564aabf98b40 2024-09-16T00:38:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.127+0000 7fdb0bfff640 1 -- 172.21.15.33:0/4163228065 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fdb140046c0 con 0x7fdb24100a90 2024-09-16T00:38:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.128+0000 7fdb2359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf98b40 0x564aabf97090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.128+0000 7fdb2359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf98b40 0x564aabf97090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:43206/0 (socket says 172.21.15.33:43206) 2024-09-16T00:38:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.128+0000 7fdb2359e640 1 -- 172.21.15.33:0/3414041421 learned_addr learned my addr 172.21.15.33:0/3414041421 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.129+0000 7fdb0bfff640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdb0c0a3410 0x7fdb0c0a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.129+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/3414041421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf975d0 0x564aabf97a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.129+0000 7fdb0bfff640 1 -- 172.21.15.33:0/4163228065 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fdb140d0140 con 0x7fdb24100a90 2024-09-16T00:38:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.129+0000 7fdb22d9d640 1 -- 172.21.15.33:0/3414041421 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf98b40 msgr2=0x564aabf97090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.129+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/3414041421 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf98b40 0x564aabf97090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.129+0000 7fdb22d9d640 1 -- 172.21.15.33:0/3414041421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabfb3380 msgr2=0x564aabfb5770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:38:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.129+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/3414041421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabfb3380 0x564aabfb5770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.129+0000 7fdb22d9d640 1 -- 172.21.15.33:0/3414041421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564aabe893a0 con 0x564aabf975d0 2024-09-16T00:38:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.129+0000 7fdb2359e640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdb0c0a3410 0x7fdb0c0a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.130+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/3414041421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf975d0 0x564aabf97a10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdb1409bf80 tx=0x7fdb14004b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5be700f0c642fd88 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:25.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.131+0000 7fdb097fa640 1 -- 172.21.15.33:0/3414041421 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1409c280 con 0x564aabf975d0 2024-09-16T00:38:25.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.131+0000 7fdb097fa640 1 -- 172.21.15.33:0/3414041421 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb140a0630 con 0x564aabf975d0 2024-09-16T00:38:25.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.133+0000 7fdb097fa640 1 -- 172.21.15.33:0/3414041421 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1409c4e0 con 0x564aabf975d0 2024-09-16T00:38:25.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.133+0000 7fdb2b617740 1 -- 172.21.15.33:0/3414041421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf975d0 msgr2=0x564aabf97a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:25.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.133+0000 7fdb2359e640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdb0c0a3410 0x7fdb0c0a58d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009fd0 tx=0x7fdb1c02e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:25.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.133+0000 7fdb2b617740 1 --2- 172.21.15.33:0/3414041421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf975d0 0x564aabf97a10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdb1409bf80 tx=0x7fdb14004b00 comp rx=0 tx=0).stop 2024-09-16T00:38:25.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.133+0000 7fdb2b617740 1 -- 172.21.15.33:0/3414041421 shutdown_connections 2024-09-16T00:38:25.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.133+0000 7fdb2b617740 1 --2- 172.21.15.33:0/3414041421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabfb3380 0x564aabfb5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.133+0000 7fdb2b617740 1 --2- 172.21.15.33:0/3414041421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf975d0 0x564aabf97a10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.133+0000 7fdb2b617740 1 --2- 172.21.15.33:0/3414041421 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf98b40 0x564aabf97090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.133+0000 7fdb2b617740 1 -- 172.21.15.33:0/3414041421 >> 172.21.15.33:0/3414041421 conn(0x564aabf60310 msgr2=0x564aabf93c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:25.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.133+0000 7fdb2b617740 1 -- 172.21.15.33:0/3414041421 shutdown_connections 2024-09-16T00:38:25.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.133+0000 7fdb2b617740 1 -- 172.21.15.33:0/3414041421 wait complete. 2024-09-16T00:38:25.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2b617740 1 -- start start 2024-09-16T00:38:25.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2b617740 1 Processor -- start 2024-09-16T00:38:25.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2b617740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf975d0 0x564aabfa4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2b617740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf98b40 0x564aabfa4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2b617740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabfb3380 0x564aabf9f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2b617740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564aabf9a4e0 con 0x564aabf975d0 2024-09-16T00:38:25.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2b617740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564aabeb8060 con 0x564aabfb3380 2024-09-16T00:38:25.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2b617740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564aabf9a660 con 0x564aabf98b40 2024-09-16T00:38:25.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2b617740 10 client.0 _ll_register_callbacks cb 0x564aabc0b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:38:25.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabfb3380 0x564aabf9f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabfb3380 0x564aabf9f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40290/0 (socket says 172.21.15.33:40290) 2024-09-16T00:38:25.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2359e640 1 -- 172.21.15.33:0/3415004816 learned_addr learned my addr 172.21.15.33:0/3415004816 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:25.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf975d0 0x564aabfa4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.134+0000 7fdb2259c640 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf98b40 0x564aabfa4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.135+0000 7fdb22d9d640 1 -- 172.21.15.33:0/3415004816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf98b40 msgr2=0x564aabfa4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:25.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.135+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf98b40 0x564aabfa4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.135+0000 7fdb22d9d640 1 -- 172.21.15.33:0/3415004816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabfb3380 msgr2=0x564aabf9f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:25.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.135+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabfb3380 0x564aabf9f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.135+0000 7fdb22d9d640 1 -- 172.21.15.33:0/3415004816 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564aabc6e3c0 con 0x564aabf975d0 2024-09-16T00:38:25.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.135+0000 7fdb2359e640 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabfb3380 0x564aabf9f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:38:25.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.135+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf975d0 0x564aabfa4400 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fdb14098620 tx=0x7fdb1409bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7fe232bdb78cab4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:25.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.135+0000 7fdb06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:38:25.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.135+0000 7fdb06ff5640 1 -- 172.21.15.33:0/3415004816 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14016db0 con 0x564aabf975d0 2024-09-16T00:38:25.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.135+0000 7fdb06ff5640 1 -- 172.21.15.33:0/3415004816 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fdb1409c6f0 con 0x564aabf975d0 2024-09-16T00:38:25.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.135+0000 7fdb2b617740 1 -- 172.21.15.33:0/3415004816 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb140a0630 con 0x564aabf975d0 2024-09-16T00:38:25.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.136+0000 7fdb06ff5640 1 -- 172.21.15.33:0/3415004816 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdb1409c280 con 0x564aabf975d0 2024-09-16T00:38:25.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.136+0000 7fdb2b617740 10 client.19413 fetch_fsmap learned FSMap version 33 2024-09-16T00:38:25.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.136+0000 7fdb2b617740 1 -- 172.21.15.33:0/3415004816 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564aabfae0d0 con 0x564aabf975d0 2024-09-16T00:38:25.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.136+0000 7fdb06ff5640 1 -- 172.21.15.33:0/3415004816 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb140a0630 con 0x564aabf975d0 2024-09-16T00:38:25.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.136+0000 7fdb06ff5640 1 -- 172.21.15.33:0/3415004816 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fdb14097fd0 con 0x564aabf975d0 2024-09-16T00:38:25.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.136+0000 7fdb2b617740 10 client.19413 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:38:25.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.136+0000 7fdb2b617740 10 client.19413 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:38:25.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.136+0000 7fdb2b617740 20 client.19413 populate_metadata read hostname 'smithi033' 2024-09-16T00:38:25.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.136+0000 7fdb2b617740 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564aabfaf4e0 0x564aabfb1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.136+0000 7fdb2b617740 4 client.19413 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:38:25.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.136+0000 7fdb2b617740 1 -- 172.21.15.33:0/3415004816 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564aabf0a860 con 0x564aabfaf4e0 2024-09-16T00:38:25.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.136+0000 7fdb2259c640 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564aabfaf4e0 0x564aabfb1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.138+0000 7fdb2259c640 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564aabfaf4e0 0x564aabfb1910 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9f321efe2a5b9fe8 server_cookie=800168ed391bea5a in_seq=0 out_seq=0 2024-09-16T00:38:25.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.138+0000 7fdb06ff5640 10 client.19413 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:38:25.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.143+0000 7fdb06ff5640 1 -- 172.21.15.33:0/3415004816 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564aabf0a860 con 0x564aabfaf4e0 2024-09-16T00:38:25.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.143+0000 7fdb06ff5640 10 client.19413 handle_command_reply: tid=0 2024-09-16T00:38:25.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.143+0000 7fdb2b617740 1 client.19413 shutdown 2024-09-16T00:38:25.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.143+0000 7fdb2b617740 1 -- 172.21.15.33:0/3415004816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf975d0 msgr2=0x564aabfa4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:25.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.143+0000 7fdb2b617740 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf975d0 0x564aabfa4400 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fdb14098620 tx=0x7fdb1409bce0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.144+0000 7fdb2b617740 1 -- 172.21.15.33:0/3415004816 shutdown_connections 2024-09-16T00:38:25.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.144+0000 7fdb2b617740 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564aabfaf4e0 0x564aabfb1910 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.144+0000 7fdb2b617740 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabfb3380 0x564aabf9f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.144+0000 7fdb2b617740 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf98b40 0x564aabfa4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.144+0000 7fdb2b617740 1 --2- 172.21.15.33:0/3415004816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf975d0 0x564aabfa4400 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.144+0000 7fdb2b617740 1 -- 172.21.15.33:0/3415004816 >> 172.21.15.33:0/3415004816 conn(0x564aabf60310 msgr2=0x564aabf999c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:25.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.144+0000 7fdb06ff5640 0 client.19413 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:38:25.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.144+0000 7fdb2b617740 1 -- 172.21.15.33:0/3415004816 shutdown_connections 2024-09-16T00:38:25.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.144+0000 7fdb2b617740 1 -- 172.21.15.33:0/3415004816 wait complete. 2024-09-16T00:38:25.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.144+0000 7fdb2b617740 20 client.19413 trim_cache size 0 max 16384 2024-09-16T00:38:25.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2b617740 1 Processor -- start 2024-09-16T00:38:25.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2b617740 1 -- start start 2024-09-16T00:38:25.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2b617740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf975d0 0x564aabf9af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2b617740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf98b40 0x564aabf9e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2b617740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabfb3380 0x564aabfb28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2b617740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdb241b8050 con 0x564aabf98b40 2024-09-16T00:38:25.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2b617740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdb241b5920 con 0x564aabfb3380 2024-09-16T00:38:25.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2b617740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb24072ae0 con 0x564aabf975d0 2024-09-16T00:38:25.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf98b40 0x564aabf9e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf98b40 0x564aabf9e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44026/0 (socket says 172.21.15.33:44026) 2024-09-16T00:38:25.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2259c640 1 -- 172.21.15.33:0/2628533515 learned_addr learned my addr 172.21.15.33:0/2628533515 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:25.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/2628533515 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf975d0 0x564aabf9af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2359e640 1 --2- 172.21.15.33:0/2628533515 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabfb3380 0x564aabfb28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2259c640 1 -- 172.21.15.33:0/2628533515 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf975d0 msgr2=0x564aabf9af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:25.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2259c640 1 --2- 172.21.15.33:0/2628533515 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf975d0 0x564aabf9af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2259c640 1 -- 172.21.15.33:0/2628533515 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabfb3380 msgr2=0x564aabfb28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:25.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2259c640 1 --2- 172.21.15.33:0/2628533515 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabfb3380 0x564aabfb28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.169+0000 7fdb2259c640 1 -- 172.21.15.33:0/2628533515 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564aabf9dc70 con 0x564aabf98b40 2024-09-16T00:38:25.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.170+0000 7fdb2259c640 1 --2- 172.21.15.33:0/2628533515 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf98b40 0x564aabf9e5d0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fdb2410bf80 tx=0x7fdb18000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bae5ee89af9bcef server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:25.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.170+0000 7fdb08ff9640 1 -- 172.21.15.33:0/2628533515 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1804b070 con 0x564aabf98b40 2024-09-16T00:38:25.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.170+0000 7fdb08ff9640 1 -- 172.21.15.33:0/2628533515 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564aabf0a860 con 0x564aabf98b40 2024-09-16T00:38:25.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.170+0000 7fdb08ff9640 1 -- 172.21.15.33:0/2628533515 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18037040 con 0x564aabf98b40 2024-09-16T00:38:25.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.171+0000 7fdb2b617740 1 -- 172.21.15.33:0/2628533515 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf98b40 msgr2=0x564aabf9e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:25.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.171+0000 7fdb2b617740 1 --2- 172.21.15.33:0/2628533515 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf98b40 0x564aabf9e5d0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fdb2410bf80 tx=0x7fdb18000bc0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.171+0000 7fdb2b617740 1 -- 172.21.15.33:0/2628533515 shutdown_connections 2024-09-16T00:38:25.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.171+0000 7fdb2b617740 1 --2- 172.21.15.33:0/2628533515 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabfb3380 0x564aabfb28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.171+0000 7fdb2b617740 1 --2- 172.21.15.33:0/2628533515 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabf98b40 0x564aabf9e5d0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.171+0000 7fdb2b617740 1 --2- 172.21.15.33:0/2628533515 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf975d0 0x564aabf9af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.171+0000 7fdb2b617740 1 -- 172.21.15.33:0/2628533515 >> 172.21.15.33:0/2628533515 conn(0x564aabf60310 msgr2=0x564aabf9dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:25.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.171+0000 7fdb2b617740 1 -- 172.21.15.33:0/2628533515 shutdown_connections 2024-09-16T00:38:25.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.171+0000 7fdb2b617740 1 -- 172.21.15.33:0/2628533515 wait complete. 2024-09-16T00:38:25.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.172+0000 7fdb2b617740 1 -- start start 2024-09-16T00:38:25.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.172+0000 7fdb2b617740 1 Processor -- start 2024-09-16T00:38:25.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.172+0000 7fdb2b617740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf975d0 0x564aabfb7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.172+0000 7fdb2b617740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf98b40 0x564aabfb8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.172+0000 7fdb2b617740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabfb3380 0x564aac014e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.172+0000 7fdb2b617740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdb24073150 con 0x564aabfb3380 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.172+0000 7fdb2b617740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdb241b5250 con 0x564aabf98b40 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.172+0000 7fdb2b617740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb24073820 con 0x564aabf975d0 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb2b617740 10 client.0 _ll_register_callbacks cb 0x7fdb2410ddf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb22d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf975d0 0x564aabfb7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb2259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf98b40 0x564aabfb8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb22d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf975d0 0x564aabfb7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:43238/0 (socket says 172.21.15.33:43238) 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb2259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf98b40 0x564aabfb8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40316/0 (socket says 172.21.15.33:40316) 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb22d9d640 1 -- 172.21.15.33:0/797594874 learned_addr learned my addr 172.21.15.33:0/797594874 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb2359e640 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabfb3380 0x564aac014e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb2259c640 1 -- 172.21.15.33:0/797594874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf975d0 msgr2=0x564aabfb7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:25.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb2259c640 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf975d0 0x564aabfb7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb2259c640 1 -- 172.21.15.33:0/797594874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabfb3380 msgr2=0x564aac014e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:25.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb2259c640 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabfb3380 0x564aac014e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb2259c640 1 -- 172.21.15.33:0/797594874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564aac0154a0 con 0x564aabf98b40 2024-09-16T00:38:25.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf975d0 0x564aabfb7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:38:25.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb2259c640 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf98b40 0x564aabfb8300 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdb1802a8a0 tx=0x7fdb18005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd1f0a78ad581339 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:25.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:38:25.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb06ff5640 1 -- 172.21.15.33:0/797594874 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1804b030 con 0x564aabf98b40 2024-09-16T00:38:25.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb2b617740 1 -- 172.21.15.33:0/797594874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564aabf0a860 con 0x564aabf98b40 2024-09-16T00:38:25.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.173+0000 7fdb06ff5640 1 -- 172.21.15.33:0/797594874 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fdb18030550 con 0x564aabf98b40 2024-09-16T00:38:25.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.174+0000 7fdb06ff5640 1 -- 172.21.15.33:0/797594874 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdb18037040 con 0x564aabf98b40 2024-09-16T00:38:25.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.174+0000 7fdb2b617740 10 client.28607 fetch_fsmap learned FSMap version 33 2024-09-16T00:38:25.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.174+0000 7fdb2b617740 1 -- 172.21.15.33:0/797594874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564aac01c780 con 0x564aabf98b40 2024-09-16T00:38:25.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.174+0000 7fdb06ff5640 1 -- 172.21.15.33:0/797594874 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb1804d050 con 0x564aabf98b40 2024-09-16T00:38:25.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.174+0000 7fdb06ff5640 1 -- 172.21.15.33:0/797594874 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fdb18003b60 con 0x564aabf98b40 2024-09-16T00:38:25.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.174+0000 7fdb2b617740 10 client.28607 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:38:25.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.174+0000 7fdb2b617740 10 client.28607 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:38:25.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.174+0000 7fdb2b617740 20 client.28607 populate_metadata read hostname 'smithi033' 2024-09-16T00:38:25.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.174+0000 7fdb2b617740 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564aac01ce70 0x564aac01f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:25.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.174+0000 7fdb2b617740 4 client.28607 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:38:25.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.174+0000 7fdb2b617740 1 -- 172.21.15.33:0/797594874 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564aabf0a860 con 0x564aac01ce70 2024-09-16T00:38:25.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.174+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564aac01ce70 0x564aac01f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:25.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.175+0000 7fdb06ff5640 10 client.28607 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:38:25.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.175+0000 7fdb22d9d640 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564aac01ce70 0x564aac01f290 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f9dd531e7b8674c0 server_cookie=8857c76c546e409d in_seq=0 out_seq=0 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.175+0000 7fdb06ff5640 1 -- 172.21.15.33:0/797594874 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564aabf0a860 con 0x564aac01ce70 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.175+0000 7fdb06ff5640 10 client.28607 handle_command_reply: tid=0 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.175+0000 7fdb2b617740 1 client.28607 shutdown 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.176+0000 7fdb2b617740 1 -- 172.21.15.33:0/797594874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf98b40 msgr2=0x564aabfb8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.176+0000 7fdb2b617740 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf98b40 0x564aabfb8300 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdb1802a8a0 tx=0x7fdb18005f90 comp rx=0 tx=0).stop 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.176+0000 7fdb2b617740 1 -- 172.21.15.33:0/797594874 shutdown_connections 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.176+0000 7fdb2b617740 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564aabfb3380 0x564aac014e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.176+0000 7fdb2b617740 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x564aac01ce70 0x564aac01f290 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.176+0000 7fdb06ff5640 0 client.28607 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.176+0000 7fdb2b617740 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564aabf98b40 0x564aabfb8300 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.176+0000 7fdb2b617740 1 --2- 172.21.15.33:0/797594874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564aabf975d0 0x564aabfb7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.176+0000 7fdb2b617740 1 -- 172.21.15.33:0/797594874 >> 172.21.15.33:0/797594874 conn(0x564aabf60310 msgr2=0x564aabfab8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.176+0000 7fdb2b617740 1 -- 172.21.15.33:0/797594874 shutdown_connections 2024-09-16T00:38:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.176+0000 7fdb2b617740 1 -- 172.21.15.33:0/797594874 wait complete. 2024-09-16T00:38:25.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.176+0000 7fdb2b617740 20 client.28607 trim_cache size 0 max 16384 2024-09-16T00:38:25.182 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:38:25.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.177+0000 7fdb2923e640 1 -- 172.21.15.33:0/4163228065 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdb0c0a3410 msgr2=0x7fdb0c0a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:25.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.177+0000 7fdb2923e640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdb0c0a3410 0x7fdb0c0a58d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009fd0 tx=0x7fdb1c02e040 comp rx=0 tx=0).stop 2024-09-16T00:38:25.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.177+0000 7fdb2923e640 1 -- 172.21.15.33:0/4163228065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb24100a90 msgr2=0x7fdb2410d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:25.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.177+0000 7fdb2923e640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb24100a90 0x7fdb2410d680 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400e9c0 tx=0x7fdb1400ee80 comp rx=0 tx=0).stop 2024-09-16T00:38:25.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.177+0000 7fdb2923e640 1 -- 172.21.15.33:0/4163228065 shutdown_connections 2024-09-16T00:38:25.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.177+0000 7fdb2923e640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fdb0c0a3410 0x7fdb0c0a58d0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.177+0000 7fdb2923e640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb24106a90 0x7fdb2410ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.177+0000 7fdb2923e640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb24101430 0x7fdb2410a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.177+0000 7fdb2923e640 1 --2- 172.21.15.33:0/4163228065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb24100a90 0x7fdb2410d680 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:25.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.177+0000 7fdb2923e640 1 -- 172.21.15.33:0/4163228065 >> 172.21.15.33:0/4163228065 conn(0x7fdb240fc7d0 msgr2=0x7fdb240fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:25.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.177+0000 7fdb2923e640 1 -- 172.21.15.33:0/4163228065 shutdown_connections 2024-09-16T00:38:25.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:25.177+0000 7fdb2923e640 1 -- 172.21.15.33:0/4163228065 wait complete. 2024-09-16T00:38:25.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:25 smithi060 ceph-mon[35185]: pgmap v988: 129 pgs: 129 active+clean; 648 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.2 MiB/s wr, 441 op/s 2024-09-16T00:38:25.868 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:25 smithi033 ceph-mon[31243]: pgmap v988: 129 pgs: 129 active+clean; 648 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.2 MiB/s wr, 441 op/s 2024-09-16T00:38:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:25 smithi083 ceph-mon[35242]: pgmap v988: 129 pgs: 129 active+clean; 648 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.2 MiB/s wr, 441 op/s 2024-09-16T00:38:25.977 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:38:26.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 151981 17.56 MB/sec execute 332 sec latency 84.070 ms 2024-09-16T00:38:26.287 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:38:26.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.850+0000 7f2da1995640 1 -- 172.21.15.33:0/597707693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d9c102780 msgr2=0x7f2d9c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:26.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.850+0000 7f2da1995640 1 --2- 172.21.15.33:0/597707693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d9c102780 0x7f2d9c1067d0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009900 tx=0x7f2d9002a410 comp rx=0 tx=0).stop 2024-09-16T00:38:26.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.850+0000 7f2da1995640 1 -- 172.21.15.33:0/597707693 shutdown_connections 2024-09-16T00:38:26.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.850+0000 7f2da1995640 1 --2- 172.21.15.33:0/597707693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c106f00 0x7f2d9c10ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.850+0000 7f2da1995640 1 --2- 172.21.15.33:0/597707693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d9c102780 0x7f2d9c1067d0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.850+0000 7f2da1995640 1 --2- 172.21.15.33:0/597707693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d9c101dd0 0x7f2d9c1021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.850+0000 7f2da1995640 1 -- 172.21.15.33:0/597707693 >> 172.21.15.33:0/597707693 conn(0x7f2d9c0fd8b0 msgr2=0x7f2d9c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:26.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.850+0000 7f2da1995640 1 -- 172.21.15.33:0/597707693 shutdown_connections 2024-09-16T00:38:26.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.850+0000 7f2da1995640 1 -- 172.21.15.33:0/597707693 wait complete. 2024-09-16T00:38:26.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.851+0000 7f2da1995640 1 Processor -- start 2024-09-16T00:38:26.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.851+0000 7f2da1995640 1 -- start start 2024-09-16T00:38:26.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.851+0000 7f2da1995640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d9c101dd0 0x7f2d9c198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.851+0000 7f2da1995640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c102780 0x7f2d9c198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.851+0000 7f2da1995640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d9c106f00 0x7f2d9c19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2da1995640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c111f60 con 0x7f2d9c106f00 2024-09-16T00:38:26.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2da1995640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c111de0 con 0x7f2d9c102780 2024-09-16T00:38:26.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2da1995640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c1120e0 con 0x7f2d9c101dd0 2024-09-16T00:38:26.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d9c101dd0 0x7f2d9c198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c102780 0x7f2d9c198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d9c106f00 0x7f2d9c19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d9c101dd0 0x7f2d9c198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:43248/0 (socket says 172.21.15.33:43248) 2024-09-16T00:38:26.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c102780 0x7f2d9c198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40336/0 (socket says 172.21.15.33:40336) 2024-09-16T00:38:26.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9affd640 1 -- 172.21.15.33:0/937385984 learned_addr learned my addr 172.21.15.33:0/937385984 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:26.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9b7fe640 1 -- 172.21.15.33:0/937385984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d9c101dd0 msgr2=0x7f2d9c198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:26.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9b7fe640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d9c101dd0 0x7f2d9c198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9b7fe640 1 -- 172.21.15.33:0/937385984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c102780 msgr2=0x7f2d9c198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9b7fe640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c102780 0x7f2d9c198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9b7fe640 1 -- 172.21.15.33:0/937385984 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9c19d3f0 con 0x7f2d9c106f00 2024-09-16T00:38:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9a7fc640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c102780 0x7f2d9c198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:38:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2d9b7fe640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d9c106f00 0x7f2d9c19cd10 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f2d94009fd0 tx=0x7f2d9400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2da0993640 1 -- 172.21.15.33:0/937385984 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9400ce20 con 0x7f2d9c106f00 2024-09-16T00:38:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2da1995640 1 -- 172.21.15.33:0/937385984 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d9c19d6e0 con 0x7f2d9c106f00 2024-09-16T00:38:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2da0993640 1 -- 172.21.15.33:0/937385984 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d94005020 con 0x7f2d9c106f00 2024-09-16T00:38:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.852+0000 7f2da0993640 1 -- 172.21.15.33:0/937385984 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94010720 con 0x7f2d9c106f00 2024-09-16T00:38:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.853+0000 7f2da1995640 1 -- 172.21.15.33:0/937385984 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d9c1a4fc0 con 0x7f2d9c106f00 2024-09-16T00:38:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.855+0000 7f2da0993640 1 -- 172.21.15.33:0/937385984 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2d94005540 con 0x7f2d9c106f00 2024-09-16T00:38:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.855+0000 7f2da3d6e740 1 Processor -- start 2024-09-16T00:38:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.855+0000 7f2da3d6e740 1 -- start start 2024-09-16T00:38:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2da0993640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2d840a3240 0x7f2d840a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2da3d6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee80090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2da3d6e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee805d0 0x55595ee80a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2da3d6e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee8b370 0x55595ee8d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2da3d6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55595ede62f0 con 0x55595ee81b40 2024-09-16T00:38:26.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2da3d6e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55595ed8dcf0 con 0x55595ee8b370 2024-09-16T00:38:26.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2da3d6e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55595ede22b0 con 0x55595ee805d0 2024-09-16T00:38:26.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2da0993640 1 -- 172.21.15.33:0/937385984 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2d940d0410 con 0x7f2d9c106f00 2024-09-16T00:38:26.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2d9affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee80090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2d9affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee80090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44050/0 (socket says 172.21.15.33:44050) 2024-09-16T00:38:26.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2d9affd640 1 -- 172.21.15.33:0/2072556185 learned_addr learned my addr 172.21.15.33:0/2072556185 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:26.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2d9a7fc640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2d840a3240 0x7f2d840a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2d9b7fe640 1 --2- 172.21.15.33:0/2072556185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee8b370 0x55595ee8d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2d9a7fc640 1 --2- 172.21.15.33:0/2072556185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee805d0 0x55595ee80a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.856+0000 7f2d9affd640 1 -- 172.21.15.33:0/2072556185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee805d0 msgr2=0x55595ee80a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.857+0000 7f2d9affd640 1 --2- 172.21.15.33:0/2072556185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee805d0 0x55595ee80a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.857+0000 7f2d9affd640 1 -- 172.21.15.33:0/2072556185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee8b370 msgr2=0x55595ee8d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.857+0000 7f2d9affd640 1 --2- 172.21.15.33:0/2072556185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee8b370 0x55595ee8d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.857+0000 7f2d9affd640 1 -- 172.21.15.33:0/2072556185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55595ed723a0 con 0x55595ee81b40 2024-09-16T00:38:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.857+0000 7f2d9b7fe640 1 --2- 172.21.15.33:0/2072556185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee8b370 0x55595ee8d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:38:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.857+0000 7f2d9a7fc640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2d840a3240 0x7f2d840a5700 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009230 tx=0x7f2d90031820 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.857+0000 7f2d9affd640 1 --2- 172.21.15.33:0/2072556185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee80090 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c009e10 tx=0x7f2d8c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d5707da04686f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.857+0000 7f2d81ffb640 1 -- 172.21.15.33:0/2072556185 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c019620 con 0x55595ee81b40 2024-09-16T00:38:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.857+0000 7f2d81ffb640 1 -- 172.21.15.33:0/2072556185 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8c019ca0 con 0x55595ee81b40 2024-09-16T00:38:26.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.857+0000 7f2d81ffb640 1 -- 172.21.15.33:0/2072556185 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c018dc0 con 0x55595ee81b40 2024-09-16T00:38:26.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.858+0000 7f2da3d6e740 1 -- 172.21.15.33:0/2072556185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 msgr2=0x55595ee80090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:26.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.858+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/2072556185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee80090 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c009e10 tx=0x7f2d8c011810 comp rx=0 tx=0).stop 2024-09-16T00:38:26.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.858+0000 7f2da3d6e740 1 -- 172.21.15.33:0/2072556185 shutdown_connections 2024-09-16T00:38:26.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.858+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/2072556185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee8b370 0x55595ee8d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.858+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/2072556185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee805d0 0x55595ee80a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.858+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/2072556185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee80090 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.858+0000 7f2da3d6e740 1 -- 172.21.15.33:0/2072556185 >> 172.21.15.33:0/2072556185 conn(0x55595ee49310 msgr2=0x55595ee7cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.858+0000 7f2da3d6e740 1 -- 172.21.15.33:0/2072556185 shutdown_connections 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.858+0000 7f2da3d6e740 1 -- 172.21.15.33:0/2072556185 wait complete. 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.859+0000 7f2da3d6e740 1 -- start start 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.859+0000 7f2da3d6e740 1 Processor -- start 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.859+0000 7f2da3d6e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee805d0 0x55595ee8ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.859+0000 7f2da3d6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee87f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.859+0000 7f2da3d6e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595ee88480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.859+0000 7f2da3d6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55595ee8e070 con 0x55595ee81b40 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.859+0000 7f2da3d6e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55595eda1060 con 0x55595ee805d0 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.859+0000 7f2da3d6e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55595ee8e1f0 con 0x55595ee8b370 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.859+0000 7f2da3d6e740 10 client.0 _ll_register_callbacks cb 0x55595eaf48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:38:26.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.859+0000 7f2d9a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee87f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.859+0000 7f2d9a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee87f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44054/0 (socket says 172.21.15.33:44054) 2024-09-16T00:38:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.859+0000 7f2d9a7fc640 1 -- 172.21.15.33:0/3147465861 learned_addr learned my addr 172.21.15.33:0/3147465861 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.860+0000 7f2d9b7fe640 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595ee88480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.860+0000 7f2d9affd640 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee805d0 0x55595ee8ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.860+0000 7f2d9a7fc640 1 -- 172.21.15.33:0/3147465861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 msgr2=0x55595ee88480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.860+0000 7f2d9a7fc640 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595ee88480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.860+0000 7f2d9a7fc640 1 -- 172.21.15.33:0/3147465861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee805d0 msgr2=0x55595ee8ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.860+0000 7f2d9a7fc640 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee805d0 0x55595ee8ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.860+0000 7f2d9a7fc640 1 -- 172.21.15.33:0/3147465861 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55595eb573c0 con 0x55595ee81b40 2024-09-16T00:38:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.860+0000 7f2d9a7fc640 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee87f40 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009900 tx=0x7f2d9002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ebcddd2c856538 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.860+0000 7f2d7affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:38:26.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.860+0000 7f2d7affd640 1 -- 172.21.15.33:0/3147465861 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9004e070 con 0x55595ee81b40 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2d7affd640 1 -- 172.21.15.33:0/3147465861 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2d900037f0 con 0x55595ee81b40 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3147465861 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d8c019ca0 con 0x55595ee81b40 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2d7affd640 1 -- 172.21.15.33:0/3147465861 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2d9005a050 con 0x55595ee81b40 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2da3d6e740 10 client.19431 fetch_fsmap learned FSMap version 33 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3147465861 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55595ee99a30 con 0x55595ee81b40 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2d7affd640 1 -- 172.21.15.33:0/3147465861 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8c019ca0 con 0x55595ee81b40 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2d7affd640 1 -- 172.21.15.33:0/3147465861 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2d9003cae0 con 0x55595ee81b40 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2da3d6e740 10 client.19431 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2da3d6e740 10 client.19431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2da3d6e740 20 client.19431 populate_metadata read hostname 'smithi033' 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55595ee9ae90 0x55595ee9d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2da3d6e740 4 client.19431 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:38:26.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.861+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3147465861 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55595edf3860 con 0x55595ee9ae90 2024-09-16T00:38:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.862+0000 7f2d9affd640 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55595ee9ae90 0x55595ee9d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.862+0000 7f2d7affd640 10 client.19431 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.862+0000 7f2d9affd640 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55595ee9ae90 0x55595ee9d310 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e155f4de1f33987 server_cookie=4921993c552150f4 in_seq=0 out_seq=0 2024-09-16T00:38:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.863+0000 7f2d7affd640 1 -- 172.21.15.33:0/3147465861 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55595edf3860 con 0x55595ee9ae90 2024-09-16T00:38:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.863+0000 7f2d7affd640 10 client.19431 handle_command_reply: tid=0 2024-09-16T00:38:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.863+0000 7f2da3d6e740 1 client.19431 shutdown 2024-09-16T00:38:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.864+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3147465861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 msgr2=0x55595ee87f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.864+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee87f40 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009900 tx=0x7f2d9002a410 comp rx=0 tx=0).stop 2024-09-16T00:38:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.864+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3147465861 shutdown_connections 2024-09-16T00:38:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.864+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55595ee9ae90 0x55595ee9d310 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.864+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595ee88480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.864+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee87f40 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.864+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/3147465861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee805d0 0x55595ee8ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.864+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3147465861 >> 172.21.15.33:0/3147465861 conn(0x55595ee49310 msgr2=0x55595ee82960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.864+0000 7f2d7affd640 0 client.19431 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.864+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3147465861 shutdown_connections 2024-09-16T00:38:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.864+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3147465861 wait complete. 2024-09-16T00:38:26.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.864+0000 7f2da3d6e740 20 client.19431 trim_cache size 0 max 16384 2024-09-16T00:38:26.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2da3d6e740 1 Processor -- start 2024-09-16T00:38:26.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2da3d6e740 1 -- start start 2024-09-16T00:38:26.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2da3d6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee805d0 0x55595ee97310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2da3d6e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee81b40 0x55595eea2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2da3d6e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595eea3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2da3d6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c1b1930 con 0x55595ee805d0 2024-09-16T00:38:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2da3d6e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c1af200 con 0x55595ee81b40 2024-09-16T00:38:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2da3d6e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c1ad050 con 0x55595ee8b370 2024-09-16T00:38:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2d9affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee805d0 0x55595ee97310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2d9affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee805d0 0x55595ee97310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44060/0 (socket says 172.21.15.33:44060) 2024-09-16T00:38:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2d9affd640 1 -- 172.21.15.33:0/3623049999 learned_addr learned my addr 172.21.15.33:0/3623049999 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2d9a7fc640 1 --2- 172.21.15.33:0/3623049999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee81b40 0x55595eea2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2d9b7fe640 1 --2- 172.21.15.33:0/3623049999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595eea3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2d9affd640 1 -- 172.21.15.33:0/3623049999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 msgr2=0x55595eea3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2d9affd640 1 --2- 172.21.15.33:0/3623049999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595eea3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2d9affd640 1 -- 172.21.15.33:0/3623049999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee81b40 msgr2=0x55595eea2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:26.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.879+0000 7f2d9affd640 1 --2- 172.21.15.33:0/3623049999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee81b40 0x55595eea2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.880+0000 7f2d9affd640 1 -- 172.21.15.33:0/3623049999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55595ee86490 con 0x55595ee805d0 2024-09-16T00:38:26.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.880+0000 7f2d9b7fe640 1 --2- 172.21.15.33:0/3623049999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595eea3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:38:26.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.880+0000 7f2d9affd640 1 --2- 172.21.15.33:0/3623049999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee805d0 0x55595ee97310 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c018510 tx=0x7f2d8c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14dbba86782efcd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:26.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.880+0000 7f2d817fa640 1 -- 172.21.15.33:0/3623049999 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c0185c0 con 0x55595ee805d0 2024-09-16T00:38:26.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.880+0000 7f2d817fa640 1 -- 172.21.15.33:0/3623049999 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8c02f070 con 0x55595ee805d0 2024-09-16T00:38:26.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.880+0000 7f2d817fa640 1 -- 172.21.15.33:0/3623049999 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c004a00 con 0x55595ee805d0 2024-09-16T00:38:26.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.881+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3623049999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee805d0 msgr2=0x55595ee97310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:26.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.881+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/3623049999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee805d0 0x55595ee97310 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c018510 tx=0x7f2d8c009780 comp rx=0 tx=0).stop 2024-09-16T00:38:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.881+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3623049999 shutdown_connections 2024-09-16T00:38:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.881+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/3623049999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595eea3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.881+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/3623049999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee81b40 0x55595eea2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.881+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/3623049999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee805d0 0x55595ee97310 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.881+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3623049999 >> 172.21.15.33:0/3623049999 conn(0x55595ee49310 msgr2=0x55595ee87f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.881+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3623049999 shutdown_connections 2024-09-16T00:38:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.881+0000 7f2da3d6e740 1 -- 172.21.15.33:0/3623049999 wait complete. 2024-09-16T00:38:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.881+0000 7f2da3d6e740 1 -- start start 2024-09-16T00:38:26.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2da3d6e740 1 Processor -- start 2024-09-16T00:38:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2da3d6e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee805d0 0x55595ee93a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2da3d6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee8edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2da3d6e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595ee8f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2da3d6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c1ad6c0 con 0x55595ee81b40 2024-09-16T00:38:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2da3d6e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c1aeb30 con 0x55595ee805d0 2024-09-16T00:38:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2da3d6e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c1add90 con 0x55595ee8b370 2024-09-16T00:38:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2d9a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee8edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2da3d6e740 10 client.0 _ll_register_callbacks cb 0x7f2d8c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:38:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2d9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee805d0 0x55595ee93a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2d9b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595ee8f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2d9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee805d0 0x55595ee93a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40380/0 (socket says 172.21.15.33:40380) 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2d9affd640 1 -- 172.21.15.33:0/1098333239 learned_addr learned my addr 172.21.15.33:0/1098333239 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2d9a7fc640 1 -- 172.21.15.33:0/1098333239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 msgr2=0x55595ee8f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2d9a7fc640 1 --2- 172.21.15.33:0/1098333239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595ee8f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2d9a7fc640 1 -- 172.21.15.33:0/1098333239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee805d0 msgr2=0x55595ee93a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.882+0000 7f2d9a7fc640 1 --2- 172.21.15.33:0/1098333239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee805d0 0x55595ee93a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.883+0000 7f2d9a7fc640 1 -- 172.21.15.33:0/1098333239 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55595ee8f9e0 con 0x55595ee81b40 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.883+0000 7f2d9affd640 1 --2- 172.21.15.33:0/1098333239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee805d0 0x55595ee93a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.883+0000 7f2d9a7fc640 1 --2- 172.21.15.33:0/1098333239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee8edb0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f2d900318b0 tx=0x7f2d90009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc30e98bedd551e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.883+0000 7f2d7affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.883+0000 7f2d7affd640 1 -- 172.21.15.33:0/1098333239 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d90063060 con 0x55595ee81b40 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.883+0000 7f2d7affd640 1 -- 172.21.15.33:0/1098333239 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2d90005230 con 0x55595ee81b40 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.883+0000 7f2da3d6e740 1 -- 172.21.15.33:0/1098333239 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d8c02f070 con 0x55595ee81b40 2024-09-16T00:38:26.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.883+0000 7f2d7affd640 1 -- 172.21.15.33:0/1098333239 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2d9006a050 con 0x55595ee81b40 2024-09-16T00:38:26.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.883+0000 7f2da3d6e740 10 client.19437 fetch_fsmap learned FSMap version 33 2024-09-16T00:38:26.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.883+0000 7f2da3d6e740 1 -- 172.21.15.33:0/1098333239 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55595ef09e60 con 0x55595ee81b40 2024-09-16T00:38:26.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.884+0000 7f2d7affd640 1 -- 172.21.15.33:0/1098333239 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8c02f070 con 0x55595ee81b40 2024-09-16T00:38:26.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.884+0000 7f2d7affd640 1 -- 172.21.15.33:0/1098333239 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2d90004750 con 0x55595ee81b40 2024-09-16T00:38:26.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.884+0000 7f2da3d6e740 10 client.19437 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:38:26.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.884+0000 7f2da3d6e740 10 client.19437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:38:26.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.884+0000 7f2da3d6e740 20 client.19437 populate_metadata read hostname 'smithi033' 2024-09-16T00:38:26.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.884+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/1098333239 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55595ef0a580 0x55595ef0c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:26.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.884+0000 7f2da3d6e740 4 client.19437 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:38:26.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.884+0000 7f2da3d6e740 1 -- 172.21.15.33:0/1098333239 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55595ededd70 con 0x55595ef0a580 2024-09-16T00:38:26.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.884+0000 7f2d9affd640 1 --2- 172.21.15.33:0/1098333239 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55595ef0a580 0x55595ef0c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.884+0000 7f2d7affd640 10 client.19437 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.884+0000 7f2d9affd640 1 --2- 172.21.15.33:0/1098333239 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55595ef0a580 0x55595ef0c9a0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f8d20d57699724a server_cookie=5b6f64976eadfeae in_seq=0 out_seq=0 2024-09-16T00:38:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2d7affd640 1 -- 172.21.15.33:0/1098333239 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55595ededd70 con 0x55595ef0a580 2024-09-16T00:38:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2d7affd640 10 client.19437 handle_command_reply: tid=0 2024-09-16T00:38:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2da3d6e740 1 client.19437 shutdown 2024-09-16T00:38:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2da3d6e740 1 -- 172.21.15.33:0/1098333239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 msgr2=0x55595ee8edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/1098333239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee8edb0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f2d900318b0 tx=0x7f2d90009260 comp rx=0 tx=0).stop 2024-09-16T00:38:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2da3d6e740 1 -- 172.21.15.33:0/1098333239 shutdown_connections 2024-09-16T00:38:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/1098333239 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55595ef0a580 0x55595ef0c9a0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/1098333239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55595ee8b370 0x55595ee8f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/1098333239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55595ee81b40 0x55595ee8edb0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2d7affd640 0 client.19437 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2da3d6e740 1 --2- 172.21.15.33:0/1098333239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55595ee805d0 0x55595ee93a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2da3d6e740 1 -- 172.21.15.33:0/1098333239 >> 172.21.15.33:0/1098333239 conn(0x55595ee49310 msgr2=0x55595ee86810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2da3d6e740 1 -- 172.21.15.33:0/1098333239 shutdown_connections 2024-09-16T00:38:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2da3d6e740 1 -- 172.21.15.33:0/1098333239 wait complete. 2024-09-16T00:38:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.885+0000 7f2da3d6e740 20 client.19437 trim_cache size 0 max 16384 2024-09-16T00:38:26.893 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:38:26.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.886+0000 7f2da1995640 1 -- 172.21.15.33:0/937385984 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2d840a3240 msgr2=0x7f2d840a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:26.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.886+0000 7f2da1995640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2d840a3240 0x7f2d840a5700 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009230 tx=0x7f2d90031820 comp rx=0 tx=0).stop 2024-09-16T00:38:26.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.886+0000 7f2da1995640 1 -- 172.21.15.33:0/937385984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d9c106f00 msgr2=0x7f2d9c19cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:26.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.886+0000 7f2da1995640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d9c106f00 0x7f2d9c19cd10 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f2d94009fd0 tx=0x7f2d9400ef10 comp rx=0 tx=0).stop 2024-09-16T00:38:26.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.886+0000 7f2da1995640 1 -- 172.21.15.33:0/937385984 shutdown_connections 2024-09-16T00:38:26.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.886+0000 7f2da1995640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2d840a3240 0x7f2d840a5700 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.886+0000 7f2da1995640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d9c106f00 0x7f2d9c19cd10 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.886+0000 7f2da1995640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c102780 0x7f2d9c198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.886+0000 7f2da1995640 1 --2- 172.21.15.33:0/937385984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d9c101dd0 0x7f2d9c198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:26.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.886+0000 7f2da1995640 1 -- 172.21.15.33:0/937385984 >> 172.21.15.33:0/937385984 conn(0x7f2d9c0fd8b0 msgr2=0x7f2d9c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:26.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.886+0000 7f2da1995640 1 -- 172.21.15.33:0/937385984 shutdown_connections 2024-09-16T00:38:26.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:26.886+0000 7f2da1995640 1 -- 172.21.15.33:0/937385984 wait complete. 2024-09-16T00:38:27.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 152496 17.57 MB/sec execute 333 sec latency 108.137 ms 2024-09-16T00:38:27.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:27 smithi083 ceph-mon[35242]: pgmap v989: 129 pgs: 129 active+clean; 652 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.4 MiB/s wr, 417 op/s 2024-09-16T00:38:27.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:27 smithi060 ceph-mon[35185]: pgmap v989: 129 pgs: 129 active+clean; 652 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.4 MiB/s wr, 417 op/s 2024-09-16T00:38:28.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:27 smithi033 ceph-mon[31243]: pgmap v989: 129 pgs: 129 active+clean; 652 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.4 MiB/s wr, 417 op/s 2024-09-16T00:38:28.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 153036 17.60 MB/sec execute 334 sec latency 111.616 ms 2024-09-16T00:38:28.624 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:38:28.625 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:38:28.919 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:38:29.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 153717 17.66 MB/sec execute 335 sec latency 62.997 ms 2024-09-16T00:38:29.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.582+0000 7f273753d640 1 -- 172.21.15.33:0/2078921565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2730106f20 msgr2=0x7f273010eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:29.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.582+0000 7f273753d640 1 --2- 172.21.15.33:0/2078921565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2730106f20 0x7f273010eaa0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f2728009ca0 tx=0x7f272802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.582+0000 7f273753d640 1 -- 172.21.15.33:0/2078921565 shutdown_connections 2024-09-16T00:38:29.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.582+0000 7f273753d640 1 --2- 172.21.15.33:0/2078921565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2730106f20 0x7f273010eaa0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.582+0000 7f273753d640 1 --2- 172.21.15.33:0/2078921565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27301027a0 0x7f27301067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.582+0000 7f273753d640 1 --2- 172.21.15.33:0/2078921565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2730101df0 0x7f27301021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.582+0000 7f273753d640 1 -- 172.21.15.33:0/2078921565 >> 172.21.15.33:0/2078921565 conn(0x7f27300fd8f0 msgr2=0x7f27300ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:29.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.582+0000 7f273753d640 1 -- 172.21.15.33:0/2078921565 shutdown_connections 2024-09-16T00:38:29.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.582+0000 7f273753d640 1 -- 172.21.15.33:0/2078921565 wait complete. 2024-09-16T00:38:29.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f273753d640 1 Processor -- start 2024-09-16T00:38:29.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f273753d640 1 -- start start 2024-09-16T00:38:29.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f273753d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730101df0 0x7f27300744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f273753d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27301027a0 0x7f2730074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f273753d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2730106f20 0x7f2730074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f273753d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2730111f80 con 0x7f2730106f20 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f273753d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2730111e00 con 0x7f2730101df0 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f273753d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2730112100 con 0x7f27301027a0 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f2735ada640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2730106f20 0x7f2730074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f2735ada640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2730106f20 0x7f2730074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44084/0 (socket says 172.21.15.33:44084) 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f2734ad8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27301027a0 0x7f2730074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f2735ada640 1 -- 172.21.15.33:0/2126276877 learned_addr learned my addr 172.21.15.33:0/2126276877 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f27352d9640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730101df0 0x7f27300744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f27352d9640 1 -- 172.21.15.33:0/2126276877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27301027a0 msgr2=0x7f2730074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f27352d9640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27301027a0 0x7f2730074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f27352d9640 1 -- 172.21.15.33:0/2126276877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2730106f20 msgr2=0x7f2730074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:29.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.583+0000 7f27352d9640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2730106f20 0x7f2730074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.584+0000 7f27352d9640 1 -- 172.21.15.33:0/2126276877 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27301a68a0 con 0x7f2730101df0 2024-09-16T00:38:29.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.584+0000 7f2734ad8640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27301027a0 0x7f2730074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:38:29.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.584+0000 7f27352d9640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730101df0 0x7f27300744f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f272c00e9f0 tx=0x7f272c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:29.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.584+0000 7f27267fc640 1 -- 172.21.15.33:0/2126276877 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272c00ccc0 con 0x7f2730101df0 2024-09-16T00:38:29.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.584+0000 7f27267fc640 1 -- 172.21.15.33:0/2126276877 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f272c005020 con 0x7f2730101df0 2024-09-16T00:38:29.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.584+0000 7f273753d640 1 -- 172.21.15.33:0/2126276877 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27301a6b90 con 0x7f2730101df0 2024-09-16T00:38:29.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.584+0000 7f27267fc640 1 -- 172.21.15.33:0/2126276877 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272c0049e0 con 0x7f2730101df0 2024-09-16T00:38:29.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.584+0000 7f273753d640 1 -- 172.21.15.33:0/2126276877 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27301a7080 con 0x7f2730101df0 2024-09-16T00:38:29.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.587+0000 7f27267fc640 1 -- 172.21.15.33:0/2126276877 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f272c005560 con 0x7f2730101df0 2024-09-16T00:38:29.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.587+0000 7f2739916740 1 Processor -- start 2024-09-16T00:38:29.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.587+0000 7f2739916740 1 -- start start 2024-09-16T00:38:29.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2739916740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057eb60 0x55d3e057ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2739916740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057f480 0x55d3e0589800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2739916740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e0589d40 0x55d3e058c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2739916740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d3e04e4280 con 0x55d3e057f480 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2739916740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d3e048bb10 con 0x55d3e0589d40 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2739916740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d3e04e0240 con 0x55d3e057eb60 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2735ada640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057eb60 0x55d3e057ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f27352d9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057f480 0x55d3e0589800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2735ada640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057eb60 0x55d3e057ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:43304/0 (socket says 172.21.15.33:43304) 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2735ada640 1 -- 172.21.15.33:0/4155417194 learned_addr learned my addr 172.21.15.33:0/4155417194 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2734ad8640 1 --2- 172.21.15.33:0/4155417194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e0589d40 0x55d3e058c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2735ada640 1 -- 172.21.15.33:0/4155417194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e0589d40 msgr2=0x55d3e058c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2735ada640 1 --2- 172.21.15.33:0/4155417194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e0589d40 0x55d3e058c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2735ada640 1 -- 172.21.15.33:0/4155417194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057f480 msgr2=0x55d3e0589800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2735ada640 1 --2- 172.21.15.33:0/4155417194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057f480 0x55d3e0589800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2735ada640 1 -- 172.21.15.33:0/4155417194 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3e04981f0 con 0x55d3e057eb60 2024-09-16T00:38:29.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f27352d9640 1 --2- 172.21.15.33:0/4155417194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057f480 0x55d3e0589800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:38:29.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f27267fc640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f271c0a3240 0x7f271c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.588+0000 7f2735ada640 1 --2- 172.21.15.33:0/4155417194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057eb60 0x55d3e057ef40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f27280097b0 tx=0x7f2728005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bfd172d7a065032 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:29.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.589+0000 7f27352d9640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f271c0a3240 0x7f271c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.589+0000 7f271b7fe640 1 -- 172.21.15.33:0/4155417194 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2728031db0 con 0x55d3e057eb60 2024-09-16T00:38:29.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.589+0000 7f271b7fe640 1 -- 172.21.15.33:0/4155417194 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2728036040 con 0x55d3e057eb60 2024-09-16T00:38:29.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.589+0000 7f271b7fe640 1 -- 172.21.15.33:0/4155417194 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2728004360 con 0x55d3e057eb60 2024-09-16T00:38:29.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.589+0000 7f27267fc640 1 -- 172.21.15.33:0/2126276877 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f272c0cfcd0 con 0x7f2730101df0 2024-09-16T00:38:29.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.590+0000 7f2739916740 1 -- 172.21.15.33:0/4155417194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057eb60 msgr2=0x55d3e057ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:29.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.590+0000 7f27352d9640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f271c0a3240 0x7f271c0a5700 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f272c0cd6c0 tx=0x7f272c0096e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:29.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.590+0000 7f2739916740 1 --2- 172.21.15.33:0/4155417194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057eb60 0x55d3e057ef40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f27280097b0 tx=0x7f2728005f90 comp rx=0 tx=0).stop 2024-09-16T00:38:29.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.590+0000 7f2739916740 1 -- 172.21.15.33:0/4155417194 shutdown_connections 2024-09-16T00:38:29.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.590+0000 7f2739916740 1 --2- 172.21.15.33:0/4155417194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e0589d40 0x55d3e058c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.590+0000 7f2739916740 1 --2- 172.21.15.33:0/4155417194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057f480 0x55d3e0589800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.590+0000 7f2739916740 1 --2- 172.21.15.33:0/4155417194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057eb60 0x55d3e057ef40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.590+0000 7f2739916740 1 -- 172.21.15.33:0/4155417194 >> 172.21.15.33:0/4155417194 conn(0x55d3e057b0f0 msgr2=0x55d3e057b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:29.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.590+0000 7f2739916740 1 -- 172.21.15.33:0/4155417194 shutdown_connections 2024-09-16T00:38:29.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.590+0000 7f2739916740 1 -- 172.21.15.33:0/4155417194 wait complete. 2024-09-16T00:38:29.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.590+0000 7f2739916740 1 -- start start 2024-09-16T00:38:29.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.590+0000 7f2739916740 1 Processor -- start 2024-09-16T00:38:29.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2739916740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 0x55d3e0586c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2739916740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057f480 0x55d3e0587180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2739916740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e0589d40 0x55d3e05876c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2739916740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d3e058c790 con 0x55d3e057eb60 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2739916740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d3e049ef10 con 0x55d3e0589d40 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2739916740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d3e058c910 con 0x55d3e057f480 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2739916740 10 client.0 _ll_register_callbacks cb 0x55d3e0467a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2734ad8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057f480 0x55d3e0587180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2734ad8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057f480 0x55d3e0587180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:43316/0 (socket says 172.21.15.33:43316) 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2734ad8640 1 -- 172.21.15.33:0/3516430904 learned_addr learned my addr 172.21.15.33:0/3516430904 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2735ada640 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 0x55d3e0586c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2734ad8640 1 -- 172.21.15.33:0/3516430904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e0589d40 msgr2=0x55d3e05876c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2734ad8640 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e0589d40 0x55d3e05876c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.591+0000 7f2734ad8640 1 -- 172.21.15.33:0/3516430904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 msgr2=0x55d3e0586c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.592+0000 7f2734ad8640 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 0x55d3e0586c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.592+0000 7f2734ad8640 1 -- 172.21.15.33:0/3516430904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3e02556a0 con 0x55d3e057f480 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.592+0000 7f2735ada640 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 0x55d3e0586c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.592+0000 7f2734ad8640 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057f480 0x55d3e0587180 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2720012d00 tx=0x7f272000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17db983d4461d467 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.592+0000 7f2718ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.592+0000 7f2718ff9640 1 -- 172.21.15.33:0/3516430904 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2720012110 con 0x55d3e057f480 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.592+0000 7f2718ff9640 1 -- 172.21.15.33:0/3516430904 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2720004050 con 0x55d3e057f480 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.592+0000 7f2739916740 1 -- 172.21.15.33:0/3516430904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2728036040 con 0x55d3e057f480 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.593+0000 7f2718ff9640 1 -- 172.21.15.33:0/3516430904 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f27200122b0 con 0x55d3e057f480 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.593+0000 7f2739916740 10 client.28639 fetch_fsmap learned FSMap version 33 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.593+0000 7f2739916740 1 -- 172.21.15.33:0/3516430904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d3e0598470 con 0x55d3e057f480 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.593+0000 7f2718ff9640 1 -- 172.21.15.33:0/3516430904 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2728036040 con 0x55d3e057f480 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.593+0000 7f2718ff9640 1 -- 172.21.15.33:0/3516430904 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2720010430 con 0x55d3e057f480 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.593+0000 7f2739916740 10 client.28639 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.593+0000 7f2739916740 10 client.28639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:38:29.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.593+0000 7f2739916740 20 client.28639 populate_metadata read hostname 'smithi033' 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.593+0000 7f2739916740 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d3e05999d0 0x55d3e059be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.593+0000 7f2739916740 4 client.28639 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.593+0000 7f2739916740 1 -- 172.21.15.33:0/3516430904 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3e04f17f0 con 0x55d3e05999d0 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.593+0000 7f2735ada640 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d3e05999d0 0x55d3e059be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.594+0000 7f2718ff9640 10 client.28639 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.594+0000 7f2735ada640 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d3e05999d0 0x55d3e059be50 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=117bed1af827ae64 server_cookie=57d4a848b59760cb in_seq=0 out_seq=0 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2718ff9640 1 -- 172.21.15.33:0/3516430904 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d3e04f17f0 con 0x55d3e05999d0 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2718ff9640 10 client.28639 handle_command_reply: tid=0 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2739916740 1 client.28639 shutdown 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2739916740 1 -- 172.21.15.33:0/3516430904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057f480 msgr2=0x55d3e0587180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2739916740 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057f480 0x55d3e0587180 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2720012d00 tx=0x7f272000f790 comp rx=0 tx=0).stop 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2739916740 1 -- 172.21.15.33:0/3516430904 shutdown_connections 2024-09-16T00:38:29.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2739916740 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d3e05999d0 0x55d3e059be50 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2739916740 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e0589d40 0x55d3e05876c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2718ff9640 0 client.28639 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:29.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2739916740 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057f480 0x55d3e0587180 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2739916740 1 --2- 172.21.15.33:0/3516430904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 0x55d3e0586c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2739916740 1 -- 172.21.15.33:0/3516430904 >> 172.21.15.33:0/3516430904 conn(0x55d3e057b0f0 msgr2=0x55d3e0580e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:29.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2739916740 1 -- 172.21.15.33:0/3516430904 shutdown_connections 2024-09-16T00:38:29.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2739916740 1 -- 172.21.15.33:0/3516430904 wait complete. 2024-09-16T00:38:29.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.595+0000 7f2739916740 20 client.28639 trim_cache size 0 max 16384 2024-09-16T00:38:29.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.620+0000 7f2739916740 1 Processor -- start 2024-09-16T00:38:29.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.620+0000 7f2739916740 1 -- start start 2024-09-16T00:38:29.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.620+0000 7f2739916740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 0x55d3e0588aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.620+0000 7f2739916740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e057f480 0x55d3e05a1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.620+0000 7f2739916740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e0589d40 0x55d3e05a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.620+0000 7f2739916740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27301b3a60 con 0x55d3e057eb60 2024-09-16T00:38:29.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.620+0000 7f2739916740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f27301b1330 con 0x55d3e057f480 2024-09-16T00:38:29.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.620+0000 7f2739916740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f27301af180 con 0x55d3e0589d40 2024-09-16T00:38:29.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.621+0000 7f2735ada640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 0x55d3e0588aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.621+0000 7f2734ad8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e057f480 0x55d3e05a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.621+0000 7f2735ada640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 0x55d3e0588aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44112/0 (socket says 172.21.15.33:44112) 2024-09-16T00:38:29.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.621+0000 7f2735ada640 1 -- 172.21.15.33:0/2174797360 learned_addr learned my addr 172.21.15.33:0/2174797360 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:29.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.621+0000 7f2735ada640 1 --2- 172.21.15.33:0/2174797360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e0589d40 0x55d3e05a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.621+0000 7f2735ada640 1 -- 172.21.15.33:0/2174797360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e0589d40 msgr2=0x55d3e05a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:29.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.621+0000 7f2735ada640 1 --2- 172.21.15.33:0/2174797360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e0589d40 0x55d3e05a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.621+0000 7f2735ada640 1 -- 172.21.15.33:0/2174797360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e057f480 msgr2=0x55d3e05a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:29.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.621+0000 7f2735ada640 1 --2- 172.21.15.33:0/2174797360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e057f480 0x55d3e05a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.621+0000 7f2735ada640 1 -- 172.21.15.33:0/2174797360 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3e057f2b0 con 0x55d3e057eb60 2024-09-16T00:38:29.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.622+0000 7f2735ada640 1 --2- 172.21.15.33:0/2174797360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 0x55d3e0588aa0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f2728005f60 tx=0x7f27280319b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0eea33622838878 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:29.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.622+0000 7f271affd640 1 -- 172.21.15.33:0/2174797360 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272804a040 con 0x55d3e057eb60 2024-09-16T00:38:29.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.622+0000 7f271affd640 1 -- 172.21.15.33:0/2174797360 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2728036040 con 0x55d3e057eb60 2024-09-16T00:38:29.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.622+0000 7f271affd640 1 -- 172.21.15.33:0/2174797360 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2728004680 con 0x55d3e057eb60 2024-09-16T00:38:29.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.623+0000 7f2739916740 1 -- 172.21.15.33:0/2174797360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 msgr2=0x55d3e0588aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:29.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.623+0000 7f2739916740 1 --2- 172.21.15.33:0/2174797360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 0x55d3e0588aa0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f2728005f60 tx=0x7f27280319b0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.623+0000 7f2739916740 1 -- 172.21.15.33:0/2174797360 shutdown_connections 2024-09-16T00:38:29.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.623+0000 7f2739916740 1 --2- 172.21.15.33:0/2174797360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e0589d40 0x55d3e05a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.623+0000 7f2739916740 1 --2- 172.21.15.33:0/2174797360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e057f480 0x55d3e05a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.623+0000 7f2739916740 1 --2- 172.21.15.33:0/2174797360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e057eb60 0x55d3e0588aa0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.623+0000 7f2739916740 1 -- 172.21.15.33:0/2174797360 >> 172.21.15.33:0/2174797360 conn(0x55d3e057b0f0 msgr2=0x55d3e0588390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:29.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.624+0000 7f2739916740 1 -- 172.21.15.33:0/2174797360 shutdown_connections 2024-09-16T00:38:29.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.624+0000 7f2739916740 1 -- 172.21.15.33:0/2174797360 wait complete. 2024-09-16T00:38:29.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.624+0000 7f2739916740 1 -- start start 2024-09-16T00:38:29.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.624+0000 7f2739916740 1 Processor -- start 2024-09-16T00:38:29.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.625+0000 7f2739916740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057eb60 0x55d3e058da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.625+0000 7f2739916740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e057f480 0x55d3e058df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.625+0000 7f2739916740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e0589d40 0x55d3e058e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.625+0000 7f2739916740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27301af7f0 con 0x55d3e0589d40 2024-09-16T00:38:29.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.625+0000 7f2739916740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f27301b0c60 con 0x55d3e057f480 2024-09-16T00:38:29.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.625+0000 7f2739916740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f27301afec0 con 0x55d3e057eb60 2024-09-16T00:38:29.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.625+0000 7f2739916740 10 client.0 _ll_register_callbacks cb 0x7f2720009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:38:29.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.625+0000 7f2735ada640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057eb60 0x55d3e058da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.625+0000 7f2734ad8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e057f480 0x55d3e058df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.625+0000 7f2734ad8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e057f480 0x55d3e058df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:40424/0 (socket says 172.21.15.33:40424) 2024-09-16T00:38:29.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.625+0000 7f2734ad8640 1 -- 172.21.15.33:0/3436079342 learned_addr learned my addr 172.21.15.33:0/3436079342 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:29.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.626+0000 7f2735ada640 1 --2- 172.21.15.33:0/3436079342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e0589d40 0x55d3e058e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.626+0000 7f2734ad8640 1 -- 172.21.15.33:0/3436079342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057eb60 msgr2=0x55d3e058da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:29.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.626+0000 7f2734ad8640 1 --2- 172.21.15.33:0/3436079342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057eb60 0x55d3e058da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.626+0000 7f2734ad8640 1 -- 172.21.15.33:0/3436079342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e0589d40 msgr2=0x55d3e058e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:29.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.626+0000 7f2734ad8640 1 --2- 172.21.15.33:0/3436079342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e0589d40 0x55d3e058e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.626+0000 7f2734ad8640 1 -- 172.21.15.33:0/3436079342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3e060c060 con 0x55d3e057f480 2024-09-16T00:38:29.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.626+0000 7f2734ad8640 1 --2- 172.21.15.33:0/3436079342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e057f480 0x55d3e058df40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55d3e0588280 tx=0x7f2720010660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9417b46af030106 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:29.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.626+0000 7f2718ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:38:29.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.628+0000 7f2718ff9640 1 -- 172.21.15.33:0/3436079342 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2720012110 con 0x55d3e057f480 2024-09-16T00:38:29.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.628+0000 7f2718ff9640 1 -- 172.21.15.33:0/3436079342 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f27200052e0 con 0x55d3e057f480 2024-09-16T00:38:29.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.628+0000 7f2739916740 1 -- 172.21.15.33:0/3436079342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2728036040 con 0x55d3e057f480 2024-09-16T00:38:29.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.628+0000 7f2718ff9640 1 -- 172.21.15.33:0/3436079342 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f27200128a0 con 0x55d3e057f480 2024-09-16T00:38:29.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.628+0000 7f2739916740 10 client.28640 fetch_fsmap learned FSMap version 33 2024-09-16T00:38:29.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.628+0000 7f2739916740 1 -- 172.21.15.33:0/3436079342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d3e060d390 con 0x55d3e057f480 2024-09-16T00:38:29.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.629+0000 7f2718ff9640 1 -- 172.21.15.33:0/3436079342 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2728036040 con 0x55d3e057f480 2024-09-16T00:38:29.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.629+0000 7f2718ff9640 1 -- 172.21.15.33:0/3436079342 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2720029420 con 0x55d3e057f480 2024-09-16T00:38:29.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.629+0000 7f2739916740 10 client.28640 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:38:29.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.629+0000 7f2739916740 10 client.28640 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:38:29.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.629+0000 7f2739916740 20 client.28640 populate_metadata read hostname 'smithi033' 2024-09-16T00:38:29.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.629+0000 7f2739916740 1 --2- 172.21.15.33:0/3436079342 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d3e05999d0 0x55d3e060fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:29.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.629+0000 7f2739916740 4 client.28640 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:38:29.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.629+0000 7f2739916740 1 -- 172.21.15.33:0/3436079342 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d3e04ebd00 con 0x55d3e05999d0 2024-09-16T00:38:29.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.629+0000 7f2735ada640 1 --2- 172.21.15.33:0/3436079342 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d3e05999d0 0x55d3e060fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:29.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.630+0000 7f2718ff9640 10 client.28640 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:29.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.630+0000 7f2735ada640 1 --2- 172.21.15.33:0/3436079342 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d3e05999d0 0x55d3e060fb80 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=781570d32ef09b0f server_cookie=44f926e76ef2584 in_seq=0 out_seq=0 2024-09-16T00:38:29.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2718ff9640 1 -- 172.21.15.33:0/3436079342 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d3e04ebd00 con 0x55d3e05999d0 2024-09-16T00:38:29.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2718ff9640 10 client.28640 handle_command_reply: tid=0 2024-09-16T00:38:29.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2739916740 1 client.28640 shutdown 2024-09-16T00:38:29.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2739916740 1 -- 172.21.15.33:0/3436079342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e057f480 msgr2=0x55d3e058df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:29.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2739916740 1 --2- 172.21.15.33:0/3436079342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e057f480 0x55d3e058df40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55d3e0588280 tx=0x7f2720010660 comp rx=0 tx=0).stop 2024-09-16T00:38:29.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2739916740 1 -- 172.21.15.33:0/3436079342 shutdown_connections 2024-09-16T00:38:29.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2739916740 1 --2- 172.21.15.33:0/3436079342 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d3e05999d0 0x55d3e060fb80 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2739916740 1 --2- 172.21.15.33:0/3436079342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3e0589d40 0x55d3e058e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2739916740 1 --2- 172.21.15.33:0/3436079342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d3e057f480 0x55d3e058df40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2739916740 1 --2- 172.21.15.33:0/3436079342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d3e057eb60 0x55d3e058da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2718ff9640 0 client.28640 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:29.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2739916740 1 -- 172.21.15.33:0/3436079342 >> 172.21.15.33:0/3436079342 conn(0x55d3e057b0f0 msgr2=0x55d3e057a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:29.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2739916740 1 -- 172.21.15.33:0/3436079342 shutdown_connections 2024-09-16T00:38:29.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2739916740 1 -- 172.21.15.33:0/3436079342 wait complete. 2024-09-16T00:38:29.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.631+0000 7f2739916740 20 client.28640 trim_cache size 0 max 16384 2024-09-16T00:38:29.636 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:38:29.637 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:38:29.637 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "a75d63ee-dcad-4d93-9f78-69468a69648e", 2024-09-16T00:38:29.637 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:38:29.637 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:38:29.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.632+0000 7f273753d640 1 -- 172.21.15.33:0/2126276877 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f271c0a3240 msgr2=0x7f271c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:29.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.632+0000 7f273753d640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f271c0a3240 0x7f271c0a5700 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f272c0cd6c0 tx=0x7f272c0096e0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.632+0000 7f273753d640 1 -- 172.21.15.33:0/2126276877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730101df0 msgr2=0x7f27300744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:29.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.632+0000 7f273753d640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730101df0 0x7f27300744f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f272c00e9f0 tx=0x7f272c00eeb0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.632+0000 7f273753d640 1 -- 172.21.15.33:0/2126276877 shutdown_connections 2024-09-16T00:38:29.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.632+0000 7f273753d640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f271c0a3240 0x7f271c0a5700 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.632+0000 7f273753d640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2730106f20 0x7f2730074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.632+0000 7f273753d640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27301027a0 0x7f2730074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.632+0000 7f273753d640 1 --2- 172.21.15.33:0/2126276877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730101df0 0x7f27300744f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:29.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.632+0000 7f273753d640 1 -- 172.21.15.33:0/2126276877 >> 172.21.15.33:0/2126276877 conn(0x7f27300fd8f0 msgr2=0x7f27300fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:29.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.632+0000 7f273753d640 1 -- 172.21.15.33:0/2126276877 shutdown_connections 2024-09-16T00:38:29.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:29.632+0000 7f273753d640 1 -- 172.21.15.33:0/2126276877 wait complete. 2024-09-16T00:38:29.639 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:29 smithi033 ceph-mon[31243]: pgmap v990: 129 pgs: 129 active+clean; 652 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-09-16T00:38:29.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:29 smithi083 ceph-mon[35242]: pgmap v990: 129 pgs: 129 active+clean; 652 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-09-16T00:38:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:29 smithi060 ceph-mon[35185]: pgmap v990: 129 pgs: 129 active+clean; 652 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-09-16T00:38:30.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 154396 17.71 MB/sec execute 336 sec latency 45.229 ms 2024-09-16T00:38:30.671 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:38:30.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:38:30.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:38:30.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:38:30.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:38:30.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:38:30.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:38:30.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:38:30.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:38:30.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:38:30.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:30 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.974 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:38:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:38:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:38:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:38:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:38:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:38:30.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:38:30.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:38:30.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:38:30.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:30.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:38:30.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:30 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:38:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:38:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:38:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:38:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:38:31.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:38:31.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:38:31.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:38:31.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:38:31.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:30 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 155002 17.78 MB/sec execute 337 sec latency 50.995 ms 2024-09-16T00:38:31.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.546+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/429157081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d01082a0 msgr2=0x7fa4d010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:31.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.546+0000 7fa4d5cbd640 1 --2- 172.21.15.33:0/429157081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d01082a0 0x7fa4d010ec40 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009cd0 tx=0x7fa4c802a8e0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.546+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/429157081 shutdown_connections 2024-09-16T00:38:31.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.546+0000 7fa4d5cbd640 1 --2- 172.21.15.33:0/429157081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d01082a0 0x7fa4d010ec40 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.546+0000 7fa4d5cbd640 1 --2- 172.21.15.33:0/429157081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4d01036e0 0x7fa4d0107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.546+0000 7fa4d5cbd640 1 --2- 172.21.15.33:0/429157081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4d0102d30 0x7fa4d0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.546+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/429157081 >> 172.21.15.33:0/429157081 conn(0x7fa4d00fc7d0 msgr2=0x7fa4d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:31.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.547+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/429157081 shutdown_connections 2024-09-16T00:38:31.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.547+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/429157081 wait complete. 2024-09-16T00:38:31.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.548+0000 7fa4d5cbd640 1 Processor -- start 2024-09-16T00:38:31.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.548+0000 7fa4d5cbd640 1 -- start start 2024-09-16T00:38:31.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.548+0000 7fa4d5cbd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4d0102d30 0x7fa4d010e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.548+0000 7fa4d5cbd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d01036e0 0x7fa4d010e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.549+0000 7fa4cf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4d0102d30 0x7fa4d010e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.549+0000 7fa4cf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4d0102d30 0x7fa4d010e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:52018/0 (socket says 172.21.15.33:52018) 2024-09-16T00:38:31.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.549+0000 7fa4cf7fe640 1 -- 172.21.15.33:0/1101121194 learned_addr learned my addr 172.21.15.33:0/1101121194 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:31.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.549+0000 7fa4d5cbd640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4d01082a0 0x7fa4d01a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.549+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/1101121194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0112150 con 0x7fa4d01036e0 2024-09-16T00:38:31.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.549+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/1101121194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0111fd0 con 0x7fa4d0102d30 2024-09-16T00:38:31.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.549+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/1101121194 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01122d0 con 0x7fa4d01082a0 2024-09-16T00:38:31.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.549+0000 7fa4ceffd640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d01036e0 0x7fa4d010e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.550+0000 7fa4cf7fe640 1 -- 172.21.15.33:0/1101121194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4d01082a0 msgr2=0x7fa4d01a3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:31.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.550+0000 7fa4cf7fe640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4d01082a0 0x7fa4d01a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.550+0000 7fa4cf7fe640 1 -- 172.21.15.33:0/1101121194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d01036e0 msgr2=0x7fa4d010e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:31.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.550+0000 7fa4cf7fe640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d01036e0 0x7fa4d010e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.550+0000 7fa4cf7fe640 1 -- 172.21.15.33:0/1101121194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d01a3d60 con 0x7fa4d0102d30 2024-09-16T00:38:31.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.550+0000 7fa4cf7fe640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4d0102d30 0x7fa4d010e040 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000b4a0 tx=0x7fa4c000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:31.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.551+0000 7fa4ccff9640 1 -- 172.21.15.33:0/1101121194 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0014070 con 0x7fa4d0102d30 2024-09-16T00:38:31.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.551+0000 7fa4ccff9640 1 -- 172.21.15.33:0/1101121194 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c0010af0 con 0x7fa4d0102d30 2024-09-16T00:38:31.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.551+0000 7fa4ccff9640 1 -- 172.21.15.33:0/1101121194 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0017d10 con 0x7fa4d0102d30 2024-09-16T00:38:31.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.552+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/1101121194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d01a4050 con 0x7fa4d0102d30 2024-09-16T00:38:31.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.552+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/1101121194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d01ab8a0 con 0x7fa4d0102d30 2024-09-16T00:38:31.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.555+0000 7fa4d8096740 1 Processor -- start 2024-09-16T00:38:31.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.556+0000 7fa4d8096740 1 -- start start 2024-09-16T00:38:31.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.556+0000 7fa4d8096740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a64a0 0x55d0401a6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.556+0000 7fa4d8096740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d0401b1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.556+0000 7fa4d8096740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 0x55d0401b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.556+0000 7fa4d8096740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d04010c2e0 con 0x55d0401a6dc0 2024-09-16T00:38:31.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.556+0000 7fa4d8096740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d0400b3ce0 con 0x55d0401b1680 2024-09-16T00:38:31.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.556+0000 7fa4d8096740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d0401082a0 con 0x55d0401a64a0 2024-09-16T00:38:31.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.556+0000 7fa4cffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a64a0 0x55d0401a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.556+0000 7fa4cffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a64a0 0x55d0401a6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:39372/0 (socket says 172.21.15.33:39372) 2024-09-16T00:38:31.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.556+0000 7fa4cffff640 1 -- 172.21.15.33:0/1133283059 learned_addr learned my addr 172.21.15.33:0/1133283059 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:31.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.556+0000 7fa4ccff9640 1 -- 172.21.15.33:0/1101121194 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa4c0017070 con 0x7fa4d0102d30 2024-09-16T00:38:31.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.556+0000 7fa4ceffd640 1 --2- 172.21.15.33:0/1133283059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 0x55d0401b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.557+0000 7fa4cffff640 1 -- 172.21.15.33:0/1133283059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 msgr2=0x55d0401b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:31.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.557+0000 7fa4cffff640 1 --2- 172.21.15.33:0/1133283059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 0x55d0401b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.557+0000 7fa4cffff640 1 -- 172.21.15.33:0/1133283059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 msgr2=0x55d0401b1140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:38:31.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.557+0000 7fa4cffff640 1 --2- 172.21.15.33:0/1133283059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d0401b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.557+0000 7fa4cffff640 1 -- 172.21.15.33:0/1133283059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d040098440 con 0x55d0401a64a0 2024-09-16T00:38:31.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.557+0000 7fa4cffff640 1 --2- 172.21.15.33:0/1133283059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a64a0 0x55d0401a6880 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009ca0 tx=0x7fa4c8009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=761266af8bdf3e96 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:31.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.558+0000 7fa4ccff9640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa4b80a3390 0x7fa4b80a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.558+0000 7fa4b5ffb640 1 -- 172.21.15.33:0/1133283059 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c803f070 con 0x55d0401a64a0 2024-09-16T00:38:31.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.558+0000 7fa4cf7fe640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa4b80a3390 0x7fa4b80a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.558+0000 7fa4b5ffb640 1 -- 172.21.15.33:0/1133283059 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c8036040 con 0x55d0401a64a0 2024-09-16T00:38:31.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.558+0000 7fa4ccff9640 1 -- 172.21.15.33:0/1101121194 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa4c00cf000 con 0x7fa4d0102d30 2024-09-16T00:38:31.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.559+0000 7fa4b5ffb640 1 -- 172.21.15.33:0/1133283059 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8004280 con 0x55d0401a64a0 2024-09-16T00:38:31.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.559+0000 7fa4d8096740 1 -- 172.21.15.33:0/1133283059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a64a0 msgr2=0x55d0401a6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:31.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.559+0000 7fa4d8096740 1 --2- 172.21.15.33:0/1133283059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a64a0 0x55d0401a6880 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009ca0 tx=0x7fa4c8009a20 comp rx=0 tx=0).stop 2024-09-16T00:38:31.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.559+0000 7fa4cf7fe640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa4b80a3390 0x7fa4b80a5850 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00cc680 tx=0x7fa4c0009390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:31.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.561+0000 7fa4d8096740 1 -- 172.21.15.33:0/1133283059 shutdown_connections 2024-09-16T00:38:31.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.561+0000 7fa4d8096740 1 --2- 172.21.15.33:0/1133283059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 0x55d0401b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.561+0000 7fa4d8096740 1 --2- 172.21.15.33:0/1133283059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d0401b1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.561+0000 7fa4d8096740 1 --2- 172.21.15.33:0/1133283059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a64a0 0x55d0401a6880 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.561+0000 7fa4d8096740 1 -- 172.21.15.33:0/1133283059 >> 172.21.15.33:0/1133283059 conn(0x55d0401a2a30 msgr2=0x55d0401a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:31.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.562+0000 7fa4d8096740 1 -- 172.21.15.33:0/1133283059 shutdown_connections 2024-09-16T00:38:31.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.562+0000 7fa4d8096740 1 -- 172.21.15.33:0/1133283059 wait complete. 2024-09-16T00:38:31.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.563+0000 7fa4d8096740 1 -- start start 2024-09-16T00:38:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.563+0000 7fa4d8096740 1 Processor -- start 2024-09-16T00:38:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.563+0000 7fa4d8096740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401a64a0 0x55d0401b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.563+0000 7fa4d8096740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d0401ae220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.563+0000 7fa4d8096740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401b1680 0x55d0401ae760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.563+0000 7fa4d8096740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d0401b42a0 con 0x55d0401a6dc0 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.563+0000 7fa4d8096740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55d0400c7050 con 0x55d0401a64a0 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.563+0000 7fa4d8096740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d0401b4420 con 0x55d0401b1680 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.563+0000 7fa4d8096740 10 client.0 _ll_register_callbacks cb 0x55d040107160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.564+0000 7fa4cffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401a64a0 0x55d0401b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.564+0000 7fa4cffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401a64a0 0x55d0401b0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:52036/0 (socket says 172.21.15.33:52036) 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.564+0000 7fa4cffff640 1 -- 172.21.15.33:0/2910685040 learned_addr learned my addr 172.21.15.33:0/2910685040 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.564+0000 7fa4cffff640 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401b1680 0x55d0401ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.564+0000 7fa4ceffd640 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d0401ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.564+0000 7fa4cffff640 1 -- 172.21.15.33:0/2910685040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401b1680 msgr2=0x55d0401ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.564+0000 7fa4cffff640 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401b1680 0x55d0401ae760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.564+0000 7fa4cffff640 1 -- 172.21.15.33:0/2910685040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 msgr2=0x55d0401ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.564+0000 7fa4cffff640 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d0401ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.564+0000 7fa4cffff640 1 -- 172.21.15.33:0/2910685040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d03fe7d3c0 con 0x55d0401a64a0 2024-09-16T00:38:31.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.565+0000 7fa4cffff640 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401a64a0 0x55d0401b0f70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009620 tx=0x7fa4c8003e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ed63e1c8bb72cab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:31.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.565+0000 7fa4aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:38:31.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.565+0000 7fa4aeffd640 1 -- 172.21.15.33:0/2910685040 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c803f070 con 0x55d0401a64a0 2024-09-16T00:38:31.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.565+0000 7fa4d8096740 1 -- 172.21.15.33:0/2910685040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4c8036040 con 0x55d0401a64a0 2024-09-16T00:38:31.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.565+0000 7fa4aeffd640 1 -- 172.21.15.33:0/2910685040 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa4c804d040 con 0x55d0401a64a0 2024-09-16T00:38:31.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.565+0000 7fa4aeffd640 1 -- 172.21.15.33:0/2910685040 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa4c8004040 con 0x55d0401a64a0 2024-09-16T00:38:31.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.565+0000 7fa4d8096740 10 client.28649 fetch_fsmap learned FSMap version 33 2024-09-16T00:38:31.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.565+0000 7fa4d8096740 1 -- 172.21.15.33:0/2910685040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d0401bfc90 con 0x55d0401a64a0 2024-09-16T00:38:31.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.565+0000 7fa4aeffd640 1 -- 172.21.15.33:0/2910685040 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c8036040 con 0x55d0401a64a0 2024-09-16T00:38:31.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.566+0000 7fa4aeffd640 1 -- 172.21.15.33:0/2910685040 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fa4c8030070 con 0x55d0401a64a0 2024-09-16T00:38:31.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.566+0000 7fa4d8096740 10 client.28649 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:38:31.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.566+0000 7fa4d8096740 10 client.28649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:38:31.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.566+0000 7fa4d8096740 20 client.28649 populate_metadata read hostname 'smithi033' 2024-09-16T00:38:31.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.566+0000 7fa4d8096740 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d0401c0fe0 0x55d0401c33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.566+0000 7fa4d8096740 4 client.28649 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:38:31.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.566+0000 7fa4d8096740 1 -- 172.21.15.33:0/2910685040 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d040119850 con 0x55d0401c0fe0 2024-09-16T00:38:31.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.567+0000 7fa4ceffd640 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d0401c0fe0 0x55d0401c33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.567+0000 7fa4aeffd640 10 client.28649 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:31.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.567+0000 7fa4ceffd640 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d0401c0fe0 0x55d0401c33d0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd12070760d2b464 server_cookie=ee70917886fe7d0c in_seq=0 out_seq=0 2024-09-16T00:38:31.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.571+0000 7fa4aeffd640 1 -- 172.21.15.33:0/2910685040 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d040119850 con 0x55d0401c0fe0 2024-09-16T00:38:31.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.571+0000 7fa4aeffd640 10 client.28649 handle_command_reply: tid=0 2024-09-16T00:38:31.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.571+0000 7fa4d8096740 1 client.28649 shutdown 2024-09-16T00:38:31.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.572+0000 7fa4d8096740 1 -- 172.21.15.33:0/2910685040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401a64a0 msgr2=0x55d0401b0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:31.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.572+0000 7fa4d8096740 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401a64a0 0x55d0401b0f70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009620 tx=0x7fa4c8003e50 comp rx=0 tx=0).stop 2024-09-16T00:38:31.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.572+0000 7fa4d8096740 1 -- 172.21.15.33:0/2910685040 shutdown_connections 2024-09-16T00:38:31.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.572+0000 7fa4d8096740 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d0401c0fe0 0x55d0401c33d0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.572+0000 7fa4aeffd640 0 client.28649 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:31.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.572+0000 7fa4d8096740 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401b1680 0x55d0401ae760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.572+0000 7fa4d8096740 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d0401ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.572+0000 7fa4d8096740 1 --2- 172.21.15.33:0/2910685040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401a64a0 0x55d0401b0f70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.572+0000 7fa4d8096740 1 -- 172.21.15.33:0/2910685040 >> 172.21.15.33:0/2910685040 conn(0x55d0401a2a30 msgr2=0x55d0401a8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:31.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.572+0000 7fa4d8096740 1 -- 172.21.15.33:0/2910685040 shutdown_connections 2024-09-16T00:38:31.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.572+0000 7fa4d8096740 1 -- 172.21.15.33:0/2910685040 wait complete. 2024-09-16T00:38:31.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.572+0000 7fa4d8096740 20 client.28649 trim_cache size 0 max 16384 2024-09-16T00:38:31.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.602+0000 7fa4d8096740 1 Processor -- start 2024-09-16T00:38:31.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.602+0000 7fa4d8096740 1 -- start start 2024-09-16T00:38:31.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.603+0000 7fa4d8096740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a64a0 0x55d0401a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.603+0000 7fa4d8096740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a6dc0 0x55d0401bd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.603+0000 7fa4d8096740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 0x55d0401c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.603+0000 7fa4d8096740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b8150 con 0x55d0401a64a0 2024-09-16T00:38:31.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.603+0000 7fa4d8096740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b5a20 con 0x55d0401b1680 2024-09-16T00:38:31.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.603+0000 7fa4d8096740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b38c0 con 0x55d0401a6dc0 2024-09-16T00:38:31.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.603+0000 7fa4ceffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a6dc0 0x55d0401bd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.604+0000 7fa4ceffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a6dc0 0x55d0401bd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:39396/0 (socket says 172.21.15.33:39396) 2024-09-16T00:38:31.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.604+0000 7fa4ceffd640 1 -- 172.21.15.33:0/3033579639 learned_addr learned my addr 172.21.15.33:0/3033579639 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:31.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.604+0000 7fa4cffff640 1 --2- 172.21.15.33:0/3033579639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a64a0 0x55d0401a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.604+0000 7fa4cffff640 1 --2- 172.21.15.33:0/3033579639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 0x55d0401c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.604+0000 7fa4ceffd640 1 -- 172.21.15.33:0/3033579639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 msgr2=0x55d0401c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:31.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.604+0000 7fa4ceffd640 1 --2- 172.21.15.33:0/3033579639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 0x55d0401c89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.604+0000 7fa4ceffd640 1 -- 172.21.15.33:0/3033579639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a64a0 msgr2=0x55d0401a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:31.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.604+0000 7fa4ceffd640 1 --2- 172.21.15.33:0/3033579639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a64a0 0x55d0401a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.604+0000 7fa4ceffd640 1 -- 172.21.15.33:0/3033579639 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0401a6bf0 con 0x55d0401a6dc0 2024-09-16T00:38:31.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.606+0000 7fa4ceffd640 1 --2- 172.21.15.33:0/3033579639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a6dc0 0x55d0401bd4a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00697d0 tx=0x7fa4c4005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=129066a54244b109 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:31.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.607+0000 7fa4b57fa640 1 -- 172.21.15.33:0/3033579639 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4022070 con 0x55d0401a6dc0 2024-09-16T00:38:31.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.607+0000 7fa4b57fa640 1 -- 172.21.15.33:0/3033579639 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c40047a0 con 0x55d0401a6dc0 2024-09-16T00:38:31.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.607+0000 7fa4b57fa640 1 -- 172.21.15.33:0/3033579639 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c401f990 con 0x55d0401a6dc0 2024-09-16T00:38:31.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.608+0000 7fa4d8096740 1 -- 172.21.15.33:0/3033579639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a6dc0 msgr2=0x55d0401bd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:31.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.608+0000 7fa4d8096740 1 --2- 172.21.15.33:0/3033579639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a6dc0 0x55d0401bd4a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00697d0 tx=0x7fa4c4005f90 comp rx=0 tx=0).stop 2024-09-16T00:38:31.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.609+0000 7fa4d8096740 1 -- 172.21.15.33:0/3033579639 shutdown_connections 2024-09-16T00:38:31.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.609+0000 7fa4d8096740 1 --2- 172.21.15.33:0/3033579639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 0x55d0401c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.609+0000 7fa4d8096740 1 --2- 172.21.15.33:0/3033579639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a6dc0 0x55d0401bd4a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.609+0000 7fa4d8096740 1 --2- 172.21.15.33:0/3033579639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a64a0 0x55d0401a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.609+0000 7fa4d8096740 1 -- 172.21.15.33:0/3033579639 >> 172.21.15.33:0/3033579639 conn(0x55d0401a2a30 msgr2=0x55d0401bcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:31.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.610+0000 7fa4d8096740 1 -- 172.21.15.33:0/3033579639 shutdown_connections 2024-09-16T00:38:31.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.610+0000 7fa4d8096740 1 -- 172.21.15.33:0/3033579639 wait complete. 2024-09-16T00:38:31.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.610+0000 7fa4d8096740 1 -- start start 2024-09-16T00:38:31.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.611+0000 7fa4d8096740 1 Processor -- start 2024-09-16T00:38:31.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.612+0000 7fa4d8096740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a64a0 0x55d0402230d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.612+0000 7fa4d8096740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d040223610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.612+0000 7fa4d8096740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 0x55d0401b9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.612+0000 7fa4d8096740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b3f30 con 0x55d0401a6dc0 2024-09-16T00:38:31.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.612+0000 7fa4d8096740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b5350 con 0x55d0401b1680 2024-09-16T00:38:31.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.612+0000 7fa4d8096740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b45b0 con 0x55d0401a64a0 2024-09-16T00:38:31.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.612+0000 7fa4d8096740 10 client.0 _ll_register_callbacks cb 0x7fa4c8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:38:31.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.613+0000 7fa4ceffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d040223610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.613+0000 7fa4ceffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d040223610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58262/0 (socket says 172.21.15.33:58262) 2024-09-16T00:38:31.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.613+0000 7fa4ceffd640 1 -- 172.21.15.33:0/2714573132 learned_addr learned my addr 172.21.15.33:0/2714573132 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:38:31.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.613+0000 7fa4cffff640 1 --2- 172.21.15.33:0/2714573132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 0x55d0401b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.613+0000 7fa4ceffd640 1 -- 172.21.15.33:0/2714573132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a64a0 msgr2=0x55d0402230d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:31.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.613+0000 7fa4ceffd640 1 --2- 172.21.15.33:0/2714573132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a64a0 0x55d0402230d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.613+0000 7fa4ceffd640 1 -- 172.21.15.33:0/2714573132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 msgr2=0x55d0401b9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:38:31.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.613+0000 7fa4ceffd640 1 --2- 172.21.15.33:0/2714573132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 0x55d0401b9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.613+0000 7fa4ceffd640 1 -- 172.21.15.33:0/2714573132 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0401b9a50 con 0x55d0401a6dc0 2024-09-16T00:38:31.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.613+0000 7fa4ceffd640 1 --2- 172.21.15.33:0/2714573132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d040223610 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40057c0 tx=0x7fa4c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2e229da10ffbf4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:38:31.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.613+0000 7fa4aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:38:31.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.614+0000 7fa4aeffd640 1 -- 172.21.15.33:0/2714573132 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4022040 con 0x55d0401a6dc0 2024-09-16T00:38:31.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.614+0000 7fa4aeffd640 1 -- 172.21.15.33:0/2714573132 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa4c4005200 con 0x55d0401a6dc0 2024-09-16T00:38:31.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.614+0000 7fa4d8096740 1 -- 172.21.15.33:0/2714573132 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4c40047a0 con 0x55d0401a6dc0 2024-09-16T00:38:31.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.614+0000 7fa4aeffd640 1 -- 172.21.15.33:0/2714573132 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa4c403a050 con 0x55d0401a6dc0 2024-09-16T00:38:31.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.614+0000 7fa4d8096740 10 client.19470 fetch_fsmap learned FSMap version 33 2024-09-16T00:38:31.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.614+0000 7fa4d8096740 1 -- 172.21.15.33:0/2714573132 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d04022b9e0 con 0x55d0401a6dc0 2024-09-16T00:38:31.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.614+0000 7fa4aeffd640 1 -- 172.21.15.33:0/2714573132 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c4005440 con 0x55d0401a6dc0 2024-09-16T00:38:31.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.615+0000 7fa4aeffd640 1 -- 172.21.15.33:0/2714573132 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fa4c402f020 con 0x55d0401a6dc0 2024-09-16T00:38:31.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.615+0000 7fa4d8096740 10 client.19470 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:38:31.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.615+0000 7fa4d8096740 10 client.19470 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:38:31.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.615+0000 7fa4d8096740 20 client.19470 populate_metadata read hostname 'smithi033' 2024-09-16T00:38:31.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.615+0000 7fa4d8096740 1 --2- 172.21.15.33:0/2714573132 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d0401c0fe0 0x55d04022e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:38:31.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.615+0000 7fa4d8096740 4 client.19470 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:38:31.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.615+0000 7fa4d8096740 1 -- 172.21.15.33:0/2714573132 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fa4c40047a0 con 0x55d0401c0fe0 2024-09-16T00:38:31.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.615+0000 7fa4cffff640 1 --2- 172.21.15.33:0/2714573132 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d0401c0fe0 0x55d04022e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:38:31.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.619+0000 7fa4cffff640 1 --2- 172.21.15.33:0/2714573132 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d0401c0fe0 0x55d04022e0d0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db6cb40a8d90dc19 server_cookie=74688fd627d5d582 in_seq=0 out_seq=0 2024-09-16T00:38:31.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.619+0000 7fa4aeffd640 10 client.19470 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:31.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.619+0000 7fa4aeffd640 1 -- 172.21.15.33:0/2714573132 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fa4c40047a0 con 0x55d0401c0fe0 2024-09-16T00:38:31.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.620+0000 7fa4aeffd640 10 client.19470 handle_command_reply: tid=0 2024-09-16T00:38:31.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.620+0000 7fa4d8096740 1 client.19470 shutdown 2024-09-16T00:38:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.620+0000 7fa4d8096740 1 -- 172.21.15.33:0/2714573132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 msgr2=0x55d040223610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.620+0000 7fa4d8096740 1 --2- 172.21.15.33:0/2714573132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d040223610 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40057c0 tx=0x7fa4c4004730 comp rx=0 tx=0).stop 2024-09-16T00:38:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.621+0000 7fa4d8096740 1 -- 172.21.15.33:0/2714573132 shutdown_connections 2024-09-16T00:38:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.621+0000 7fa4d8096740 1 --2- 172.21.15.33:0/2714573132 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55d0401c0fe0 0x55d04022e0d0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.621+0000 7fa4d8096740 1 --2- 172.21.15.33:0/2714573132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d0401b1680 0x55d0401b9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.621+0000 7fa4d8096740 1 --2- 172.21.15.33:0/2714573132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d0401a6dc0 0x55d040223610 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.621+0000 7fa4d8096740 1 --2- 172.21.15.33:0/2714573132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d0401a64a0 0x55d0402230d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.621+0000 7fa4d8096740 1 -- 172.21.15.33:0/2714573132 >> 172.21.15.33:0/2714573132 conn(0x55d0401a2a30 msgr2=0x55d0401c7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.621+0000 7fa4aeffd640 0 client.19470 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:38:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.621+0000 7fa4d8096740 1 -- 172.21.15.33:0/2714573132 shutdown_connections 2024-09-16T00:38:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.621+0000 7fa4d8096740 1 -- 172.21.15.33:0/2714573132 wait complete. 2024-09-16T00:38:31.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.622+0000 7fa4d8096740 20 client.19470 trim_cache size 0 max 16384 2024-09-16T00:38:31.625 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:38:31.625 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:38:31.625 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:38:31.625 INFO:teuthology.orchestra.run.smithi033.stdout: "a75d63ee-dcad-4d93-9f78-69468a69648e": { 2024-09-16T00:38:31.625 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:38:31.625 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "a75d63ee-dcad-4d93-9f78-69468a69648e", 2024-09-16T00:38:31.625 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:38:31.625 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:38:31.625 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:38:31.626 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:38:31.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.623+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/1101121194 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa4b80a3390 msgr2=0x7fa4b80a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:31.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.623+0000 7fa4d5cbd640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa4b80a3390 0x7fa4b80a5850 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00cc680 tx=0x7fa4c0009390 comp rx=0 tx=0).stop 2024-09-16T00:38:31.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.623+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/1101121194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4d0102d30 msgr2=0x7fa4d010e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:38:31.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.623+0000 7fa4d5cbd640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4d0102d30 0x7fa4d010e040 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000b4a0 tx=0x7fa4c000b960 comp rx=0 tx=0).stop 2024-09-16T00:38:31.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.623+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/1101121194 shutdown_connections 2024-09-16T00:38:31.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.623+0000 7fa4d5cbd640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa4b80a3390 0x7fa4b80a5850 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.623+0000 7fa4d5cbd640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4d01082a0 0x7fa4d01a3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.623+0000 7fa4d5cbd640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d01036e0 0x7fa4d010e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.623+0000 7fa4d5cbd640 1 --2- 172.21.15.33:0/1101121194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4d0102d30 0x7fa4d010e040 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:38:31.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.623+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/1101121194 >> 172.21.15.33:0/1101121194 conn(0x7fa4d00fc7d0 msgr2=0x7fa4d00fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:38:31.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.623+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/1101121194 shutdown_connections 2024-09-16T00:38:31.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:38:31.623+0000 7fa4d5cbd640 1 -- 172.21.15.33:0/1101121194 wait complete. 2024-09-16T00:38:31.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:31 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:38:31.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:31 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:31 smithi033 ceph-mon[31243]: pgmap v991: 129 pgs: 129 active+clean; 662 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.7 MiB/s wr, 432 op/s 2024-09-16T00:38:31.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:31 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:31 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:38:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:31 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:31 smithi083 ceph-mon[35242]: pgmap v991: 129 pgs: 129 active+clean; 662 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.7 MiB/s wr, 432 op/s 2024-09-16T00:38:31.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:31 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:31 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:38:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:31 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:31 smithi060 ceph-mon[35185]: pgmap v991: 129 pgs: 129 active+clean; 662 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.7 MiB/s wr, 432 op/s 2024-09-16T00:38:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:31 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:38:32.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 155421 17.79 MB/sec execute 338 sec latency 116.874 ms 2024-09-16T00:38:32.462 INFO:tasks.cephfs.filesystem:scrub status for tag:a75d63ee-dcad-4d93-9f78-69468a69648e - {'path': '/', 'tag': 'a75d63ee-dcad-4d93-9f78-69468a69648e', 'options': 'recursive,force'} 2024-09-16T00:38:33.092 INFO:tasks.workunit.client.0.smithi033.stdout: 10 155829 17.81 MB/sec execute 339 sec latency 165.862 ms 2024-09-16T00:38:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:33 smithi083 ceph-mon[35242]: pgmap v992: 129 pgs: 129 active+clean; 664 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.1 MiB/s wr, 366 op/s 2024-09-16T00:38:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:38:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:33 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:33 smithi060 ceph-mon[35185]: pgmap v992: 129 pgs: 129 active+clean; 664 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.1 MiB/s wr, 366 op/s 2024-09-16T00:38:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:38:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:33 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:34.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:33 smithi033 ceph-mon[31243]: pgmap v992: 129 pgs: 129 active+clean; 664 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.1 MiB/s wr, 366 op/s 2024-09-16T00:38:34.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:38:34.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:34.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:33 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:34.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 156193 17.81 MB/sec execute 340 sec latency 284.858 ms 2024-09-16T00:38:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:34 smithi033 ceph-mon[31243]: pgmap v993: 129 pgs: 129 active+clean; 662 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.0 MiB/s wr, 396 op/s 2024-09-16T00:38:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:38:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:38:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:34 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.093 INFO:tasks.workunit.client.0.smithi033.stdout: 10 156224 17.76 MB/sec execute 341 sec latency 1284.897 ms 2024-09-16T00:38:35.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:34 smithi083 ceph-mon[35242]: pgmap v993: 129 pgs: 129 active+clean; 662 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.0 MiB/s wr, 396 op/s 2024-09-16T00:38:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:38:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:38:35.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:34 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:34 smithi060 ceph-mon[35185]: pgmap v993: 129 pgs: 129 active+clean; 662 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.0 MiB/s wr, 396 op/s 2024-09-16T00:38:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:38:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:38:35.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:34 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:38:36.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 156375 17.72 MB/sec execute 342 sec latency 1378.138 ms 2024-09-16T00:38:37.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 156655 17.71 MB/sec execute 343 sec latency 199.521 ms 2024-09-16T00:38:37.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:37 smithi083 ceph-mon[35242]: pgmap v994: 129 pgs: 129 active+clean; 688 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 6.8 MiB/s wr, 457 op/s 2024-09-16T00:38:37.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:37 smithi060 ceph-mon[35185]: pgmap v994: 129 pgs: 129 active+clean; 688 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 6.8 MiB/s wr, 457 op/s 2024-09-16T00:38:37.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:37 smithi033 ceph-mon[31243]: pgmap v994: 129 pgs: 129 active+clean; 688 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 6.8 MiB/s wr, 457 op/s 2024-09-16T00:38:38.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 157021 17.73 MB/sec execute 344 sec latency 98.438 ms 2024-09-16T00:38:39.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 157405 17.73 MB/sec execute 345 sec latency 172.566 ms 2024-09-16T00:38:39.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:39 smithi083 ceph-mon[35242]: pgmap v995: 129 pgs: 129 active+clean; 688 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.3 MiB/s wr, 415 op/s 2024-09-16T00:38:39.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:39 smithi060 ceph-mon[35185]: pgmap v995: 129 pgs: 129 active+clean; 688 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.3 MiB/s wr, 415 op/s 2024-09-16T00:38:39.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:39 smithi033 ceph-mon[31243]: pgmap v995: 129 pgs: 129 active+clean; 688 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.3 MiB/s wr, 415 op/s 2024-09-16T00:38:40.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 157858 17.75 MB/sec execute 346 sec latency 273.187 ms 2024-09-16T00:38:41.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 158460 17.78 MB/sec execute 347 sec latency 81.040 ms 2024-09-16T00:38:41.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:41 smithi083 ceph-mon[35242]: pgmap v996: 129 pgs: 129 active+clean; 684 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 11 MiB/s wr, 581 op/s 2024-09-16T00:38:41.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:41 smithi060 ceph-mon[35185]: pgmap v996: 129 pgs: 129 active+clean; 684 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 11 MiB/s wr, 581 op/s 2024-09-16T00:38:41.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:41 smithi033 ceph-mon[31243]: pgmap v996: 129 pgs: 129 active+clean; 684 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 11 MiB/s wr, 581 op/s 2024-09-16T00:38:42.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 159229 17.87 MB/sec execute 348 sec latency 67.391 ms 2024-09-16T00:38:42.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:42 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:38:42.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:42 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:38:42.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:42 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:38:42.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:42 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:38:42.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:42 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:38:42.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:42 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:38:42.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:42 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:38:42.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:42 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:38:42.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:42 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:38:43.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 159706 17.91 MB/sec execute 349 sec latency 69.929 ms 2024-09-16T00:38:43.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:43 smithi083 ceph-mon[35242]: pgmap v997: 129 pgs: 129 active+clean; 649 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 11 MiB/s wr, 614 op/s 2024-09-16T00:38:43.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:43 smithi060 ceph-mon[35185]: pgmap v997: 129 pgs: 129 active+clean; 649 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 11 MiB/s wr, 614 op/s 2024-09-16T00:38:43.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:43 smithi033 ceph-mon[31243]: pgmap v997: 129 pgs: 129 active+clean; 649 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 11 MiB/s wr, 614 op/s 2024-09-16T00:38:44.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 160178 17.92 MB/sec execute 350 sec latency 73.528 ms 2024-09-16T00:38:45.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 160670 17.92 MB/sec execute 351 sec latency 102.211 ms 2024-09-16T00:38:45.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:45 smithi083 ceph-mon[35242]: pgmap v998: 129 pgs: 129 active+clean; 654 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 12 MiB/s wr, 688 op/s 2024-09-16T00:38:45.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:45 smithi060 ceph-mon[35185]: pgmap v998: 129 pgs: 129 active+clean; 654 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 12 MiB/s wr, 688 op/s 2024-09-16T00:38:45.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:45 smithi033 ceph-mon[31243]: pgmap v998: 129 pgs: 129 active+clean; 654 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 12 MiB/s wr, 688 op/s 2024-09-16T00:38:46.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 161341 18.00 MB/sec execute 352 sec latency 52.658 ms 2024-09-16T00:38:47.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 161930 18.02 MB/sec execute 353 sec latency 71.165 ms 2024-09-16T00:38:47.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:47 smithi083 ceph-mon[35242]: pgmap v999: 129 pgs: 129 active+clean; 653 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 12 MiB/s wr, 634 op/s 2024-09-16T00:38:47.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:47 smithi060 ceph-mon[35185]: pgmap v999: 129 pgs: 129 active+clean; 653 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 12 MiB/s wr, 634 op/s 2024-09-16T00:38:47.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:47 smithi033 ceph-mon[31243]: pgmap v999: 129 pgs: 129 active+clean; 653 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 12 MiB/s wr, 634 op/s 2024-09-16T00:38:48.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 162437 18.05 MB/sec execute 354 sec latency 79.748 ms 2024-09-16T00:38:49.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 163165 18.09 MB/sec execute 355 sec latency 52.861 ms 2024-09-16T00:38:49.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:49 smithi083 ceph-mon[35242]: pgmap v1000: 129 pgs: 129 active+clean; 653 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.7 MiB/s wr, 488 op/s 2024-09-16T00:38:49.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:49 smithi060 ceph-mon[35185]: pgmap v1000: 129 pgs: 129 active+clean; 653 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.7 MiB/s wr, 488 op/s 2024-09-16T00:38:49.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:49 smithi033 ceph-mon[31243]: pgmap v1000: 129 pgs: 129 active+clean; 653 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.7 MiB/s wr, 488 op/s 2024-09-16T00:38:50.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 163759 18.17 MB/sec execute 356 sec latency 73.993 ms 2024-09-16T00:38:51.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 164248 18.18 MB/sec execute 357 sec latency 71.497 ms 2024-09-16T00:38:51.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:51 smithi083 ceph-mon[35242]: pgmap v1001: 129 pgs: 129 active+clean; 663 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.6 MiB/s wr, 594 op/s 2024-09-16T00:38:51.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:51 smithi060 ceph-mon[35185]: pgmap v1001: 129 pgs: 129 active+clean; 663 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.6 MiB/s wr, 594 op/s 2024-09-16T00:38:51.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:51 smithi033 ceph-mon[31243]: pgmap v1001: 129 pgs: 129 active+clean; 663 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.6 MiB/s wr, 594 op/s 2024-09-16T00:38:52.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 164672 18.19 MB/sec execute 358 sec latency 70.783 ms 2024-09-16T00:38:53.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 165356 18.25 MB/sec execute 359 sec latency 63.464 ms 2024-09-16T00:38:53.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:53 smithi083 ceph-mon[35242]: pgmap v1002: 129 pgs: 129 active+clean; 669 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.0 MiB/s wr, 511 op/s 2024-09-16T00:38:53.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:53 smithi060 ceph-mon[35185]: pgmap v1002: 129 pgs: 129 active+clean; 669 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.0 MiB/s wr, 511 op/s 2024-09-16T00:38:53.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:53 smithi033 ceph-mon[31243]: pgmap v1002: 129 pgs: 129 active+clean; 669 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.0 MiB/s wr, 511 op/s 2024-09-16T00:38:54.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 165959 18.27 MB/sec execute 360 sec latency 66.692 ms 2024-09-16T00:38:55.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 166533 18.30 MB/sec execute 361 sec latency 56.166 ms 2024-09-16T00:38:55.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:55 smithi083 ceph-mon[35242]: pgmap v1003: 129 pgs: 129 active+clean; 671 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.7 MiB/s wr, 378 op/s 2024-09-16T00:38:55.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:55 smithi060 ceph-mon[35185]: pgmap v1003: 129 pgs: 129 active+clean; 671 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.7 MiB/s wr, 378 op/s 2024-09-16T00:38:55.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:55 smithi033 ceph-mon[31243]: pgmap v1003: 129 pgs: 129 active+clean; 671 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.7 MiB/s wr, 378 op/s 2024-09-16T00:38:56.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 167095 18.34 MB/sec execute 362 sec latency 114.945 ms 2024-09-16T00:38:57.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 167798 18.41 MB/sec execute 363 sec latency 58.345 ms 2024-09-16T00:38:57.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:57 smithi083 ceph-mon[35242]: pgmap v1004: 129 pgs: 129 active+clean; 670 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.4 MiB/s wr, 389 op/s 2024-09-16T00:38:57.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:57 smithi060 ceph-mon[35185]: pgmap v1004: 129 pgs: 129 active+clean; 670 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.4 MiB/s wr, 389 op/s 2024-09-16T00:38:57.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:57 smithi033 ceph-mon[31243]: pgmap v1004: 129 pgs: 129 active+clean; 670 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.4 MiB/s wr, 389 op/s 2024-09-16T00:38:58.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 168354 18.42 MB/sec execute 364 sec latency 71.508 ms 2024-09-16T00:38:59.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 168721 18.44 MB/sec execute 365 sec latency 146.915 ms 2024-09-16T00:38:59.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:38:59 smithi083 ceph-mon[35242]: pgmap v1005: 129 pgs: 129 active+clean; 670 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-09-16T00:38:59.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:38:59 smithi060 ceph-mon[35185]: pgmap v1005: 129 pgs: 129 active+clean; 670 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-09-16T00:38:59.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:38:59 smithi033 ceph-mon[31243]: pgmap v1005: 129 pgs: 129 active+clean; 670 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-09-16T00:39:00.094 INFO:tasks.workunit.client.0.smithi033.stdout: 10 169265 18.46 MB/sec execute 366 sec latency 92.914 ms 2024-09-16T00:39:01.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 169927 18.52 MB/sec execute 367 sec latency 66.833 ms 2024-09-16T00:39:01.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:01 smithi083 ceph-mon[35242]: pgmap v1006: 129 pgs: 129 active+clean; 656 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.3 MiB/s wr, 376 op/s 2024-09-16T00:39:01.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:01 smithi060 ceph-mon[35185]: pgmap v1006: 129 pgs: 129 active+clean; 656 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.3 MiB/s wr, 376 op/s 2024-09-16T00:39:01.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:01 smithi033 ceph-mon[31243]: pgmap v1006: 129 pgs: 129 active+clean; 656 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.3 MiB/s wr, 376 op/s 2024-09-16T00:39:02.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 170458 18.52 MB/sec execute 368 sec latency 68.792 ms 2024-09-16T00:39:02.464 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:39:02.764 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:03.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 171092 18.58 MB/sec execute 369 sec latency 56.724 ms 2024-09-16T00:39:03.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.359+0000 7f431404a640 1 -- 172.21.15.33:0/2611626345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c103ff0 msgr2=0x7f430c1043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:03.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.359+0000 7f431404a640 1 --2- 172.21.15.33:0/2611626345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c103ff0 0x7f430c1043d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4308009900 tx=0x7f430802a3f0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.359+0000 7f431404a640 1 -- 172.21.15.33:0/2611626345 shutdown_connections 2024-09-16T00:39:03.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.359+0000 7f431404a640 1 --2- 172.21.15.33:0/2611626345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f430c109120 0x7f430c110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.359+0000 7f431404a640 1 --2- 172.21.15.33:0/2611626345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c1049a0 0x7f430c1089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.359+0000 7f431404a640 1 --2- 172.21.15.33:0/2611626345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c103ff0 0x7f430c1043d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.359+0000 7f431404a640 1 -- 172.21.15.33:0/2611626345 >> 172.21.15.33:0/2611626345 conn(0x7f430c0ffab0 msgr2=0x7f430c101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:03.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.359+0000 7f431404a640 1 -- 172.21.15.33:0/2611626345 shutdown_connections 2024-09-16T00:39:03.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.359+0000 7f431404a640 1 -- 172.21.15.33:0/2611626345 wait complete. 2024-09-16T00:39:03.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.360+0000 7f431404a640 1 Processor -- start 2024-09-16T00:39:03.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.360+0000 7f431404a640 1 -- start start 2024-09-16T00:39:03.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.360+0000 7f431404a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c103ff0 0x7f430c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f431404a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c1049a0 0x7f430c19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f431404a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f430c109120 0x7f430c19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f431404a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f430c114170 con 0x7f430c109120 2024-09-16T00:39:03.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f431404a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f430c113ff0 con 0x7f430c103ff0 2024-09-16T00:39:03.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f431404a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f430c1142f0 con 0x7f430c1049a0 2024-09-16T00:39:03.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f4311de6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c103ff0 0x7f430c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f4311de6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c103ff0 0x7f430c19a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:50504/0 (socket says 172.21.15.33:50504) 2024-09-16T00:39:03.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f4311de6640 1 -- 172.21.15.33:0/1668759839 learned_addr learned my addr 172.21.15.33:0/1668759839 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:03.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f43125e7640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f430c109120 0x7f430c19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f4311de6640 1 -- 172.21.15.33:0/1668759839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c1049a0 msgr2=0x7f430c19ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:03.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f4311de6640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c1049a0 0x7f430c19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f4311de6640 1 -- 172.21.15.33:0/1668759839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f430c109120 msgr2=0x7f430c19eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:03.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f4311de6640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f430c109120 0x7f430c19eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f4311de6640 1 -- 172.21.15.33:0/1668759839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f430c19f5c0 con 0x7f430c103ff0 2024-09-16T00:39:03.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f43125e7640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f430c109120 0x7f430c19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:39:03.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.361+0000 7f4311de6640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c103ff0 0x7f430c19a610 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4308004770 tx=0x7f4308003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:03.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.362+0000 7f4302ffd640 1 -- 172.21.15.33:0/1668759839 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430803f070 con 0x7f430c103ff0 2024-09-16T00:39:03.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.362+0000 7f431404a640 1 -- 172.21.15.33:0/1668759839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f430c19f850 con 0x7f430c103ff0 2024-09-16T00:39:03.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.362+0000 7f431404a640 1 -- 172.21.15.33:0/1668759839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f430c1a7190 con 0x7f430c103ff0 2024-09-16T00:39:03.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.362+0000 7f4302ffd640 1 -- 172.21.15.33:0/1668759839 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4308031ca0 con 0x7f430c103ff0 2024-09-16T00:39:03.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.362+0000 7f4302ffd640 1 -- 172.21.15.33:0/1668759839 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308035030 con 0x7f430c103ff0 2024-09-16T00:39:03.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.365+0000 7f4302ffd640 1 -- 172.21.15.33:0/1668759839 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4308042070 con 0x7f430c103ff0 2024-09-16T00:39:03.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.365+0000 7f4316423740 1 Processor -- start 2024-09-16T00:39:03.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.365+0000 7f4316423740 1 -- start start 2024-09-16T00:39:03.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.365+0000 7f4316423740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb651e3c0 0x55bfb651e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.365+0000 7f4316423740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651ece0 0x55bfb6531950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f4316423740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb6531e90 0x55bfb6534280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f4316423740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bfb64842e0 con 0x55bfb651ece0 2024-09-16T00:39:03.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f4316423740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55bfb642bce0 con 0x55bfb651e3c0 2024-09-16T00:39:03.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f4316423740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bfb64802a0 con 0x55bfb6531e90 2024-09-16T00:39:03.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f4311de6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651ece0 0x55bfb6531950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f43125e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb651e3c0 0x55bfb651e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f4311de6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651ece0 0x55bfb6531950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38114/0 (socket says 172.21.15.33:38114) 2024-09-16T00:39:03.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f4311de6640 1 -- 172.21.15.33:0/2295243618 learned_addr learned my addr 172.21.15.33:0/2295243618 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:03.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f4311de6640 1 -- 172.21.15.33:0/2295243618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb6531e90 msgr2=0x55bfb6534280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:03.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f4311de6640 1 --2- 172.21.15.33:0/2295243618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb6531e90 0x55bfb6534280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f4311de6640 1 -- 172.21.15.33:0/2295243618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb651e3c0 msgr2=0x55bfb651e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:03.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f4311de6640 1 --2- 172.21.15.33:0/2295243618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb651e3c0 0x55bfb651e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.366+0000 7f4311de6640 1 -- 172.21.15.33:0/2295243618 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfb6410440 con 0x55bfb651ece0 2024-09-16T00:39:03.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.367+0000 7f4302ffd640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42f80a32c0 0x7f42f80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.371+0000 7f43125e7640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42f80a32c0 0x7f42f80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.372+0000 7f43125e7640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42f80a32c0 0x7f42f80a5780 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f4304005fd0 tx=0x7f4304009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:03.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.372+0000 7f4302ffd640 1 -- 172.21.15.33:0/1668759839 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f43080f0490 con 0x7f430c103ff0 2024-09-16T00:39:03.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.373+0000 7f4311de6640 1 --2- 172.21.15.33:0/2295243618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651ece0 0x55bfb6531950 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f43080f85b0 tx=0x7f43080f86a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49d270a206b33f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:03.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.376+0000 7f42f7fff640 1 -- 172.21.15.33:0/2295243618 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308049020 con 0x55bfb651ece0 2024-09-16T00:39:03.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.376+0000 7f42f7fff640 1 -- 172.21.15.33:0/2295243618 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43080bccd0 con 0x55bfb651ece0 2024-09-16T00:39:03.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.376+0000 7f42f7fff640 1 -- 172.21.15.33:0/2295243618 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43080bc3a0 con 0x55bfb651ece0 2024-09-16T00:39:03.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.377+0000 7f4316423740 1 -- 172.21.15.33:0/2295243618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651ece0 msgr2=0x55bfb6531950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:03.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.377+0000 7f4316423740 1 --2- 172.21.15.33:0/2295243618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651ece0 0x55bfb6531950 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f43080f85b0 tx=0x7f43080f86a0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.379+0000 7f4316423740 1 -- 172.21.15.33:0/2295243618 shutdown_connections 2024-09-16T00:39:03.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.379+0000 7f4316423740 1 --2- 172.21.15.33:0/2295243618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb6531e90 0x55bfb6534280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.379+0000 7f4316423740 1 --2- 172.21.15.33:0/2295243618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651ece0 0x55bfb6531950 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.379+0000 7f4316423740 1 --2- 172.21.15.33:0/2295243618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb651e3c0 0x55bfb651e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.379+0000 7f4316423740 1 -- 172.21.15.33:0/2295243618 >> 172.21.15.33:0/2295243618 conn(0x55bfb651aa30 msgr2=0x55bfb651ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:03.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.380+0000 7f4316423740 1 -- 172.21.15.33:0/2295243618 shutdown_connections 2024-09-16T00:39:03.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.380+0000 7f4316423740 1 -- 172.21.15.33:0/2295243618 wait complete. 2024-09-16T00:39:03.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.381+0000 7f4316423740 1 -- start start 2024-09-16T00:39:03.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.381+0000 7f4316423740 1 Processor -- start 2024-09-16T00:39:03.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.382+0000 7f4316423740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb651e3c0 0x55bfb6531840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.382+0000 7f4316423740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651ece0 0x55bfb652eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.383+0000 7f4316423740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb6531e90 0x55bfb652f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.383+0000 7f43115e5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651ece0 0x55bfb652eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.383+0000 7f43115e5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651ece0 0x55bfb652eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34520/0 (socket says 172.21.15.33:34520) 2024-09-16T00:39:03.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.383+0000 7f43115e5640 1 -- 172.21.15.33:0/195487412 learned_addr learned my addr 172.21.15.33:0/195487412 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:03.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.383+0000 7f4311de6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb651e3c0 0x55bfb6531840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.383+0000 7f4316423740 1 -- 172.21.15.33:0/195487412 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bfb65347c0 con 0x55bfb6531e90 2024-09-16T00:39:03.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.383+0000 7f4316423740 1 -- 172.21.15.33:0/195487412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55bfb643f050 con 0x55bfb651e3c0 2024-09-16T00:39:03.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.383+0000 7f4316423740 1 -- 172.21.15.33:0/195487412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bfb6534940 con 0x55bfb651ece0 2024-09-16T00:39:03.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.384+0000 7f4316423740 10 client.0 _ll_register_callbacks cb 0x55bfb647f160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:03.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.384+0000 7f43125e7640 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb6531e90 0x55bfb652f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.384+0000 7f43115e5640 1 -- 172.21.15.33:0/195487412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb651e3c0 msgr2=0x55bfb6531840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:03.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.384+0000 7f43115e5640 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb651e3c0 0x55bfb6531840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.384+0000 7f43115e5640 1 -- 172.21.15.33:0/195487412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb6531e90 msgr2=0x55bfb652f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:03.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.384+0000 7f43115e5640 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb6531e90 0x55bfb652f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.384+0000 7f43115e5640 1 -- 172.21.15.33:0/195487412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfb61f53c0 con 0x55bfb651ece0 2024-09-16T00:39:03.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.384+0000 7f43125e7640 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb6531e90 0x55bfb652f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:39:03.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.384+0000 7f4316423740 1 -- 172.21.15.33:0/195487412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43080bccd0 con 0x55bfb651ece0 2024-09-16T00:39:03.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.385+0000 7f43115e5640 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651ece0 0x55bfb652eaf0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f42fc00bc50 tx=0x7f42fc00d6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e569be96ed1c1a1b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:03.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.386+0000 7f42f57fa640 10 client.28663 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:39:03.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.386+0000 7f42f57fa640 1 -- 172.21.15.33:0/195487412 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc016070 con 0x55bfb651ece0 2024-09-16T00:39:03.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.386+0000 7f42f57fa640 1 -- 172.21.15.33:0/195487412 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f42fc004220 con 0x55bfb651ece0 2024-09-16T00:39:03.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.387+0000 7f42f57fa640 1 -- 172.21.15.33:0/195487412 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f42fc004480 con 0x55bfb651ece0 2024-09-16T00:39:03.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.387+0000 7f4316423740 10 client.28663 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:03.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.387+0000 7f4316423740 1 -- 172.21.15.33:0/195487412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({fsmap=33}) -- 0x55bfb6540140 con 0x55bfb651ece0 2024-09-16T00:39:03.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.391+0000 7f42f57fa640 1 -- 172.21.15.33:0/195487412 <== mon.1 v2:172.21.15.83:3300/0 4 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f42fc022020 con 0x55bfb651ece0 2024-09-16T00:39:03.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.391+0000 7f4316423740 10 client.28663 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:03.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.391+0000 7f4316423740 10 client.28663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:39:03.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.391+0000 7f4316423740 20 client.28663 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:03.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.391+0000 7f4316423740 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bfb65414f0 0x55bfb6543950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.391+0000 7f4316423740 4 client.28663 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:39:03.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.392+0000 7f4316423740 1 -- 172.21.15.33:0/195487412 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfb6491850 con 0x55bfb65414f0 2024-09-16T00:39:03.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.392+0000 7f4311de6640 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bfb65414f0 0x55bfb6543950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.392+0000 7f4311de6640 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bfb65414f0 0x55bfb6543950 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fff63117f84d51a0 server_cookie=89005b12c8b722eb in_seq=0 out_seq=0 2024-09-16T00:39:03.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.392+0000 7f42f57fa640 10 client.28663 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:03.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.394+0000 7f42f57fa640 1 -- 172.21.15.33:0/195487412 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bfb6491850 con 0x55bfb65414f0 2024-09-16T00:39:03.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.394+0000 7f42f57fa640 10 client.28663 handle_command_reply: tid=0 2024-09-16T00:39:03.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.394+0000 7f4316423740 1 client.28663 shutdown 2024-09-16T00:39:03.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.395+0000 7f4316423740 1 -- 172.21.15.33:0/195487412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651ece0 msgr2=0x55bfb652eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:03.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.395+0000 7f4316423740 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651ece0 0x55bfb652eaf0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f42fc00bc50 tx=0x7f42fc00d6b0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.395+0000 7f4316423740 1 -- 172.21.15.33:0/195487412 shutdown_connections 2024-09-16T00:39:03.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.395+0000 7f4316423740 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bfb65414f0 0x55bfb6543950 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.395+0000 7f4316423740 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb6531e90 0x55bfb652f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.395+0000 7f4316423740 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651ece0 0x55bfb652eaf0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.395+0000 7f4316423740 1 --2- 172.21.15.33:0/195487412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb651e3c0 0x55bfb6531840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.395+0000 7f4316423740 1 -- 172.21.15.33:0/195487412 >> 172.21.15.33:0/195487412 conn(0x55bfb651aa30 msgr2=0x55bfb65206c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:03.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.395+0000 7f42f57fa640 0 client.28663 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:03.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.396+0000 7f4316423740 1 -- 172.21.15.33:0/195487412 shutdown_connections 2024-09-16T00:39:03.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.396+0000 7f4316423740 1 -- 172.21.15.33:0/195487412 wait complete. 2024-09-16T00:39:03.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.396+0000 7f4316423740 20 client.28663 trim_cache size 0 max 16384 2024-09-16T00:39:03.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:03 smithi083 ceph-mon[35242]: pgmap v1007: 129 pgs: 129 active+clean; 666 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 3.4 MiB/s wr, 394 op/s 2024-09-16T00:39:03.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.429+0000 7f4316423740 1 Processor -- start 2024-09-16T00:39:03.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.430+0000 7f4316423740 1 -- start start 2024-09-16T00:39:03.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.430+0000 7f4316423740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651e3c0 0x55bfb653da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.431+0000 7f4316423740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651ece0 0x55bfb652e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.431+0000 7f4316423740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb6531e90 0x55bfb6530a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.431+0000 7f4311de6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651e3c0 0x55bfb653da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.431+0000 7f4311de6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651e3c0 0x55bfb653da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34536/0 (socket says 172.21.15.33:34536) 2024-09-16T00:39:03.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.431+0000 7f4311de6640 1 -- 172.21.15.33:0/801612661 learned_addr learned my addr 172.21.15.33:0/801612661 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:03.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.431+0000 7f43125e7640 1 --2- 172.21.15.33:0/801612661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb6531e90 0x55bfb6530a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.431+0000 7f4316423740 1 -- 172.21.15.33:0/801612661 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f42f0012920 con 0x55bfb651ece0 2024-09-16T00:39:03.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.431+0000 7f4316423740 1 -- 172.21.15.33:0/801612661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f42f00101f0 con 0x55bfb6531e90 2024-09-16T00:39:03.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.432+0000 7f4316423740 1 -- 172.21.15.33:0/801612661 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f42f000e040 con 0x55bfb651e3c0 2024-09-16T00:39:03.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.432+0000 7f4311de6640 1 -- 172.21.15.33:0/801612661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb6531e90 msgr2=0x55bfb6530a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:03.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.432+0000 7f4311de6640 1 --2- 172.21.15.33:0/801612661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb6531e90 0x55bfb6530a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.432+0000 7f4311de6640 1 -- 172.21.15.33:0/801612661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651ece0 msgr2=0x55bfb652e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:03.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.432+0000 7f4311de6640 1 --2- 172.21.15.33:0/801612661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651ece0 0x55bfb652e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.432+0000 7f4311de6640 1 -- 172.21.15.33:0/801612661 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfb652daa0 con 0x55bfb651e3c0 2024-09-16T00:39:03.433 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:03 smithi033 ceph-mon[31243]: pgmap v1007: 129 pgs: 129 active+clean; 666 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 3.4 MiB/s wr, 394 op/s 2024-09-16T00:39:03.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.438+0000 7f4311de6640 1 --2- 172.21.15.33:0/801612661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651e3c0 0x55bfb653da60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f43080a80b0 tx=0x7f43080f3ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b276198e0539d14b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:03.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.439+0000 7f42f77fe640 1 -- 172.21.15.33:0/801612661 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308049040 con 0x55bfb651e3c0 2024-09-16T00:39:03.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.439+0000 7f42f77fe640 1 -- 172.21.15.33:0/801612661 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43080bc5d0 con 0x55bfb651e3c0 2024-09-16T00:39:03.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.439+0000 7f42f77fe640 1 -- 172.21.15.33:0/801612661 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308005d50 con 0x55bfb651e3c0 2024-09-16T00:39:03.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.441+0000 7f4316423740 1 -- 172.21.15.33:0/801612661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651e3c0 msgr2=0x55bfb653da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:03.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.441+0000 7f4316423740 1 --2- 172.21.15.33:0/801612661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651e3c0 0x55bfb653da60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f43080a80b0 tx=0x7f43080f3ed0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.442+0000 7f4316423740 1 -- 172.21.15.33:0/801612661 shutdown_connections 2024-09-16T00:39:03.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.442+0000 7f4316423740 1 --2- 172.21.15.33:0/801612661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb6531e90 0x55bfb6530a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.442+0000 7f4316423740 1 --2- 172.21.15.33:0/801612661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651ece0 0x55bfb652e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.442+0000 7f4316423740 1 --2- 172.21.15.33:0/801612661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651e3c0 0x55bfb653da60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.442+0000 7f4316423740 1 -- 172.21.15.33:0/801612661 >> 172.21.15.33:0/801612661 conn(0x55bfb651aa30 msgr2=0x55bfb6530350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:03.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.446+0000 7f4316423740 1 -- 172.21.15.33:0/801612661 shutdown_connections 2024-09-16T00:39:03.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.446+0000 7f4316423740 1 -- 172.21.15.33:0/801612661 wait complete. 2024-09-16T00:39:03.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.447+0000 7f4316423740 1 -- start start 2024-09-16T00:39:03.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.447+0000 7f4316423740 1 Processor -- start 2024-09-16T00:39:03.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.449+0000 7f4316423740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651e3c0 0x55bfb6539e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.450+0000 7f4316423740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651ece0 0x55bfb653a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.450+0000 7f4316423740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb6531e90 0x55bfb6536c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.450+0000 7f4316423740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f42f000e6b0 con 0x55bfb651e3c0 2024-09-16T00:39:03.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.450+0000 7f4316423740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f42f000fb20 con 0x55bfb6531e90 2024-09-16T00:39:03.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.450+0000 7f43125e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb6531e90 0x55bfb6536c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.450+0000 7f43125e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb6531e90 0x55bfb6536c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:50550/0 (socket says 172.21.15.33:50550) 2024-09-16T00:39:03.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.450+0000 7f43125e7640 1 -- 172.21.15.33:0/3176798984 learned_addr learned my addr 172.21.15.33:0/3176798984 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:03.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.450+0000 7f4316423740 1 -- 172.21.15.33:0/3176798984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f42f000ed80 con 0x55bfb651ece0 2024-09-16T00:39:03.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.451+0000 7f4316423740 10 client.0 _ll_register_callbacks cb 0x7f42fc00b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:03.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.451+0000 7f43125e7640 1 -- 172.21.15.33:0/3176798984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651ece0 msgr2=0x55bfb653a380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:39:03.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.451+0000 7f43125e7640 1 --2- 172.21.15.33:0/3176798984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651ece0 0x55bfb653a380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.451+0000 7f43125e7640 1 -- 172.21.15.33:0/3176798984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651e3c0 msgr2=0x55bfb6539e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:03.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.451+0000 7f43125e7640 1 --2- 172.21.15.33:0/3176798984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651e3c0 0x55bfb6539e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.451+0000 7f43125e7640 1 -- 172.21.15.33:0/3176798984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfb651eb10 con 0x55bfb6531e90 2024-09-16T00:39:03.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.452+0000 7f43125e7640 1 --2- 172.21.15.33:0/3176798984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb6531e90 0x55bfb6536c20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4304012010 tx=0x7f4304004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8177bf2cb37358b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:03.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.452+0000 7f42f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:39:03.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.452+0000 7f42f57fa640 1 -- 172.21.15.33:0/3176798984 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430402c060 con 0x55bfb6531e90 2024-09-16T00:39:03.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.452+0000 7f42f57fa640 1 -- 172.21.15.33:0/3176798984 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f430401cd10 con 0x55bfb6531e90 2024-09-16T00:39:03.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.452+0000 7f4316423740 1 -- 172.21.15.33:0/3176798984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43080bc5d0 con 0x55bfb6531e90 2024-09-16T00:39:03.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.453+0000 7f42f57fa640 1 -- 172.21.15.33:0/3176798984 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4304027400 con 0x55bfb6531e90 2024-09-16T00:39:03.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.454+0000 7f4316423740 10 client.28670 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:03.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.454+0000 7f4316423740 1 -- 172.21.15.33:0/3176798984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55bfb65acc50 con 0x55bfb6531e90 2024-09-16T00:39:03.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.456+0000 7f42f57fa640 1 -- 172.21.15.33:0/3176798984 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43080bc5d0 con 0x55bfb6531e90 2024-09-16T00:39:03.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.456+0000 7f42f57fa640 1 -- 172.21.15.33:0/3176798984 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f430400e710 con 0x55bfb6531e90 2024-09-16T00:39:03.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.456+0000 7f4316423740 10 client.28670 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:03.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.456+0000 7f4316423740 10 client.28670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:39:03.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.456+0000 7f4316423740 20 client.28670 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:03.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.456+0000 7f4316423740 1 --2- 172.21.15.33:0/3176798984 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bfb65ad360 0x55bfb65af7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:03.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.456+0000 7f4316423740 4 client.28670 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:39:03.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.456+0000 7f4316423740 1 -- 172.21.15.33:0/3176798984 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bfb648bd60 con 0x55bfb65ad360 2024-09-16T00:39:03.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.462+0000 7f4311de6640 1 --2- 172.21.15.33:0/3176798984 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bfb65ad360 0x55bfb65af7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:03.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.462+0000 7f4311de6640 1 --2- 172.21.15.33:0/3176798984 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bfb65ad360 0x55bfb65af7c0 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad806dc0df4c3bf3 server_cookie=a2bed97534746ef2 in_seq=0 out_seq=0 2024-09-16T00:39:03.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.462+0000 7f42f57fa640 10 client.28670 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:03.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.463+0000 7f42f57fa640 1 -- 172.21.15.33:0/3176798984 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bfb648bd60 con 0x55bfb65ad360 2024-09-16T00:39:03.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.463+0000 7f42f57fa640 10 client.28670 handle_command_reply: tid=0 2024-09-16T00:39:03.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.463+0000 7f4316423740 1 client.28670 shutdown 2024-09-16T00:39:03.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.466+0000 7f4316423740 1 -- 172.21.15.33:0/3176798984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb6531e90 msgr2=0x55bfb6536c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:03.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.466+0000 7f4316423740 1 --2- 172.21.15.33:0/3176798984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb6531e90 0x55bfb6536c20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4304012010 tx=0x7f4304004ab0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.466+0000 7f4316423740 1 -- 172.21.15.33:0/3176798984 shutdown_connections 2024-09-16T00:39:03.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.466+0000 7f4316423740 1 --2- 172.21.15.33:0/3176798984 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55bfb65ad360 0x55bfb65af7c0 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.466+0000 7f4316423740 1 --2- 172.21.15.33:0/3176798984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bfb6531e90 0x55bfb6536c20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.466+0000 7f4316423740 1 --2- 172.21.15.33:0/3176798984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfb651ece0 0x55bfb653a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.466+0000 7f4316423740 1 --2- 172.21.15.33:0/3176798984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bfb651e3c0 0x55bfb6539e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.466+0000 7f4316423740 1 -- 172.21.15.33:0/3176798984 >> 172.21.15.33:0/3176798984 conn(0x55bfb651aa30 msgr2=0x55bfb65206c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:03.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.468+0000 7f42f57fa640 0 client.28670 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:03.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.470+0000 7f4316423740 1 -- 172.21.15.33:0/3176798984 shutdown_connections 2024-09-16T00:39:03.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.470+0000 7f4316423740 1 -- 172.21.15.33:0/3176798984 wait complete. 2024-09-16T00:39:03.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.470+0000 7f4316423740 20 client.28670 trim_cache size 0 max 16384 2024-09-16T00:39:03.470 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:39:03.470 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:39:03.470 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:39:03.470 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:39:03.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.471+0000 7f431404a640 1 -- 172.21.15.33:0/1668759839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42f80a32c0 msgr2=0x7f42f80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:03.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.471+0000 7f431404a640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42f80a32c0 0x7f42f80a5780 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f4304005fd0 tx=0x7f4304009500 comp rx=0 tx=0).stop 2024-09-16T00:39:03.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.471+0000 7f431404a640 1 -- 172.21.15.33:0/1668759839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c103ff0 msgr2=0x7f430c19a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:03.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.471+0000 7f431404a640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c103ff0 0x7f430c19a610 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4308004770 tx=0x7f4308003c10 comp rx=0 tx=0).stop 2024-09-16T00:39:03.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.471+0000 7f431404a640 1 -- 172.21.15.33:0/1668759839 shutdown_connections 2024-09-16T00:39:03.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.471+0000 7f431404a640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f42f80a32c0 0x7f42f80a5780 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.471+0000 7f431404a640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f430c109120 0x7f430c19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.471+0000 7f431404a640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c1049a0 0x7f430c19ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:03.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.471+0000 7f431404a640 1 --2- 172.21.15.33:0/1668759839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c103ff0 0x7f430c19a610 secure :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4308004770 tx=0x7f4308003c10 comp rx=0 tx=0).stop 2024-09-16T00:39:03.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.471+0000 7f431404a640 1 -- 172.21.15.33:0/1668759839 >> 172.21.15.33:0/1668759839 conn(0x7f430c0ffab0 msgr2=0x7f430c100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:03.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.471+0000 7f431404a640 1 -- 172.21.15.33:0/1668759839 shutdown_connections 2024-09-16T00:39:03.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:03.471+0000 7f431404a640 1 -- 172.21.15.33:0/1668759839 wait complete. 2024-09-16T00:39:03.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:03 smithi060 ceph-mon[35185]: pgmap v1007: 129 pgs: 129 active+clean; 666 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 3.4 MiB/s wr, 394 op/s 2024-09-16T00:39:04.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 171610 18.62 MB/sec execute 370 sec latency 152.549 ms 2024-09-16T00:39:04.622 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:39:04.623 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:39:04.927 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:05.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 172138 18.64 MB/sec execute 371 sec latency 58.133 ms 2024-09-16T00:39:05.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:05 smithi033 ceph-mon[31243]: pgmap v1008: 129 pgs: 129 active+clean; 669 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 352 op/s 2024-09-16T00:39:05.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:05 smithi083 ceph-mon[35242]: pgmap v1008: 129 pgs: 129 active+clean; 669 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 352 op/s 2024-09-16T00:39:05.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.463+0000 7f74ec9c7640 1 -- 172.21.15.33:0/2636213601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74e00aab50 msgr2=0x7f74e00a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:05.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.463+0000 7f74ec9c7640 1 --2- 172.21.15.33:0/2636213601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74e00aab50 0x7f74e00a47f0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f74dc009860 tx=0x7f74dc02a340 comp rx=0 tx=0).stop 2024-09-16T00:39:05.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.464+0000 7f74ec9c7640 1 -- 172.21.15.33:0/2636213601 shutdown_connections 2024-09-16T00:39:05.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.464+0000 7f74ec9c7640 1 --2- 172.21.15.33:0/2636213601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74e00a4d30 0x7f74e00a51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:05.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.464+0000 7f74ec9c7640 1 --2- 172.21.15.33:0/2636213601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74e00aab50 0x7f74e00a47f0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:05.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.464+0000 7f74ec9c7640 1 --2- 172.21.15.33:0/2636213601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74e00aa230 0x7f74e00aa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:05.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.464+0000 7f74ec9c7640 1 -- 172.21.15.33:0/2636213601 >> 172.21.15.33:0/2636213601 conn(0x7f74e001a0a0 msgr2=0x7f74e001a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:05.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.464+0000 7f74ec9c7640 1 -- 172.21.15.33:0/2636213601 shutdown_connections 2024-09-16T00:39:05.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.464+0000 7f74ec9c7640 1 -- 172.21.15.33:0/2636213601 wait complete. 2024-09-16T00:39:05.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.465+0000 7f74ec9c7640 1 Processor -- start 2024-09-16T00:39:05.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.465+0000 7f74ec9c7640 1 -- start start 2024-09-16T00:39:05.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.465+0000 7f74ec9c7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74e00a4d30 0x7f74e01434d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:05.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.466+0000 7f74ec9c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74e00aa230 0x7f74e0143a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:05.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.466+0000 7f74ec9c7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74e00aab50 0x7f74e0147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:05.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.466+0000 7f74e659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74e00a4d30 0x7f74e01434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:05.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.466+0000 7f74e5d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74e00aa230 0x7f74e0143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:05.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.466+0000 7f74e5d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74e00aa230 0x7f74e0143a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34552/0 (socket says 172.21.15.33:34552) 2024-09-16T00:39:05.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.466+0000 7f74e5d9b640 1 -- 172.21.15.33:0/3967096206 learned_addr learned my addr 172.21.15.33:0/3967096206 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:05.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.466+0000 7f74e659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74e00a4d30 0x7f74e01434d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38156/0 (socket says 172.21.15.33:38156) 2024-09-16T00:39:05.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.466+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f74e00bcfb0 con 0x7f74e00a4d30 2024-09-16T00:39:05.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.466+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f74e00bce30 con 0x7f74e00aab50 2024-09-16T00:39:05.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.466+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f74e00bd130 con 0x7f74e00aa230 2024-09-16T00:39:05.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.466+0000 7f74e6d9d640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74e00aab50 0x7f74e0147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:05.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.467+0000 7f74e6d9d640 1 -- 172.21.15.33:0/3967096206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74e00aa230 msgr2=0x7f74e0143a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:05.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.467+0000 7f74e6d9d640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74e00aa230 0x7f74e0143a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:05.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.467+0000 7f74e6d9d640 1 -- 172.21.15.33:0/3967096206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74e00a4d30 msgr2=0x7f74e01434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:05.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.467+0000 7f74e6d9d640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74e00a4d30 0x7f74e01434d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:05.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.467+0000 7f74e6d9d640 1 -- 172.21.15.33:0/3967096206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e0148480 con 0x7f74e00aab50 2024-09-16T00:39:05.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.467+0000 7f74e5d9b640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74e00aa230 0x7f74e0143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:39:05.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.467+0000 7f74e6d9d640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74e00aab50 0x7f74e0147da0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f74e8068040 tx=0x7f74e806fe30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:05.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.467+0000 7f74d77fe640 1 -- 172.21.15.33:0/3967096206 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e807e070 con 0x7f74e00aab50 2024-09-16T00:39:05.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.467+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e0148770 con 0x7f74e00aab50 2024-09-16T00:39:05.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.467+0000 7f74d77fe640 1 -- 172.21.15.33:0/3967096206 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74e806f2e0 con 0x7f74e00aab50 2024-09-16T00:39:05.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.467+0000 7f74d77fe640 1 -- 172.21.15.33:0/3967096206 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e806de30 con 0x7f74e00aab50 2024-09-16T00:39:05.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.467+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e0150050 con 0x7f74e00aab50 2024-09-16T00:39:05.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.469+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e00aa610 con 0x7f74e00aab50 2024-09-16T00:39:05.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.471+0000 7f74d77fe640 1 -- 172.21.15.33:0/3967096206 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f74e8081070 con 0x7f74e00aab50 2024-09-16T00:39:05.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.472+0000 7f74d77fe640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74d00a3250 0x7f74d00a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:05.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.472+0000 7f74d77fe640 1 -- 172.21.15.33:0/3967096206 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f74e81333b0 con 0x7f74e00aab50 2024-09-16T00:39:05.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.472+0000 7f74e659c640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74d00a3250 0x7f74d00a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:05.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.473+0000 7f74e659c640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74d00a3250 0x7f74d00a5710 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f74d8005fd0 tx=0x7f74d8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:05.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.478+0000 7f74d77fe640 1 -- 172.21.15.33:0/3967096206 <== mon.2 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f74e8076070 con 0x7f74e00aab50 2024-09-16T00:39:05.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:05 smithi060 ceph-mon[35185]: pgmap v1008: 129 pgs: 129 active+clean; 669 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 352 op/s 2024-09-16T00:39:05.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.776+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74e0150280 con 0x7f74e00aab50 2024-09-16T00:39:05.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.777+0000 7f74d77fe640 1 -- 172.21.15.33:0/3967096206 <== mon.2 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f74e80ffc50 con 0x7f74e00aab50 2024-09-16T00:39:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:39:05.778 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:39:05.779 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:39:05.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.780+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74d00a3250 msgr2=0x7f74d00a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:05.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.780+0000 7f74ec9c7640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74d00a3250 0x7f74d00a5710 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f74d8005fd0 tx=0x7f74d8009260 comp rx=0 tx=0).stop 2024-09-16T00:39:05.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.780+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74e00aab50 msgr2=0x7f74e0147da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:05.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.780+0000 7f74ec9c7640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74e00aab50 0x7f74e0147da0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f74e8068040 tx=0x7f74e806fe30 comp rx=0 tx=0).stop 2024-09-16T00:39:05.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.780+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 shutdown_connections 2024-09-16T00:39:05.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.780+0000 7f74ec9c7640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74d00a3250 0x7f74d00a5710 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:05.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.780+0000 7f74ec9c7640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74e00aab50 0x7f74e0147da0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:05.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.780+0000 7f74ec9c7640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74e00aa230 0x7f74e0143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:05.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.780+0000 7f74ec9c7640 1 --2- 172.21.15.33:0/3967096206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74e00a4d30 0x7f74e01434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:05.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.780+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 >> 172.21.15.33:0/3967096206 conn(0x7f74e001a0a0 msgr2=0x7f74e00a6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:05.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.780+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 shutdown_connections 2024-09-16T00:39:05.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:05.780+0000 7f74ec9c7640 1 -- 172.21.15.33:0/3967096206 wait complete. 2024-09-16T00:39:06.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 172672 18.64 MB/sec execute 372 sec latency 59.736 ms 2024-09-16T00:39:06.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:06 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3967096206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:39:06.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:06 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3967096206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:39:06.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:06 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3967096206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:39:07.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 173179 18.69 MB/sec execute 373 sec latency 149.913 ms 2024-09-16T00:39:07.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:07 smithi033 ceph-mon[31243]: pgmap v1009: 129 pgs: 129 active+clean; 679 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.8 MiB/s wr, 431 op/s 2024-09-16T00:39:07.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:07 smithi083 ceph-mon[35242]: pgmap v1009: 129 pgs: 129 active+clean; 679 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.8 MiB/s wr, 431 op/s 2024-09-16T00:39:07.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:07 smithi060 ceph-mon[35185]: pgmap v1009: 129 pgs: 129 active+clean; 679 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.8 MiB/s wr, 431 op/s 2024-09-16T00:39:07.925 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:39:08.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 173706 18.73 MB/sec execute 374 sec latency 113.570 ms 2024-09-16T00:39:08.223 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:09.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 174160 18.73 MB/sec execute 375 sec latency 273.672 ms 2024-09-16T00:39:09.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:09 smithi083 ceph-mon[35242]: pgmap v1010: 129 pgs: 129 active+clean; 679 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 306 op/s 2024-09-16T00:39:09.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:09 smithi060 ceph-mon[35185]: pgmap v1010: 129 pgs: 129 active+clean; 679 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 306 op/s 2024-09-16T00:39:09.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:09 smithi033 ceph-mon[31243]: pgmap v1010: 129 pgs: 129 active+clean; 679 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 306 op/s 2024-09-16T00:39:09.824 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:39:09.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.182+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/4002275416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c88102d30 msgr2=0x7f1c88103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:09.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.182+0000 7f1c8d5ae640 1 --2- 172.21.15.33:0/4002275416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c88102d30 0x7f1c88103110 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009900 tx=0x7f1c7802a480 comp rx=0 tx=0).stop 2024-09-16T00:39:09.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.182+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/4002275416 shutdown_connections 2024-09-16T00:39:09.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.182+0000 7f1c8d5ae640 1 --2- 172.21.15.33:0/4002275416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c8810c160 0x7f1c8810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.182+0000 7f1c8d5ae640 1 --2- 172.21.15.33:0/4002275416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c88103650 0x7f1c8810bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.182+0000 7f1c8d5ae640 1 --2- 172.21.15.33:0/4002275416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c88102d30 0x7f1c88103110 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.182+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/4002275416 >> 172.21.15.33:0/4002275416 conn(0x7f1c880fc790 msgr2=0x7f1c880febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:09.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.182+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/4002275416 shutdown_connections 2024-09-16T00:39:09.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.182+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/4002275416 wait complete. 2024-09-16T00:39:09.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.182+0000 7f1c8d5ae640 1 Processor -- start 2024-09-16T00:39:09.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c8d5ae640 1 -- start start 2024-09-16T00:39:09.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c8d5ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c88102d30 0x7f1c88198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c8d5ae640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c88103650 0x7f1c88198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c8d5ae640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c8810c160 0x7f1c8819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c8d5ae640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1c88111f60 con 0x7f1c8810c160 2024-09-16T00:39:09.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c8d5ae640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1c88111de0 con 0x7f1c88103650 2024-09-16T00:39:09.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c8d5ae640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881120e0 con 0x7f1c88102d30 2024-09-16T00:39:09.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c7ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c88103650 0x7f1c88198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c86ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c88102d30 0x7f1c88198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c7ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c88103650 0x7f1c88198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:50592/0 (socket says 172.21.15.33:50592) 2024-09-16T00:39:09.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c86ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c88102d30 0x7f1c88198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34572/0 (socket says 172.21.15.33:34572) 2024-09-16T00:39:09.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c7ffff640 1 -- 172.21.15.33:0/2902868499 learned_addr learned my addr 172.21.15.33:0/2902868499 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:09.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c877fe640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c8810c160 0x7f1c8819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c7ffff640 1 -- 172.21.15.33:0/2902868499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c88102d30 msgr2=0x7f1c88198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c7ffff640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c88102d30 0x7f1c88198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c7ffff640 1 -- 172.21.15.33:0/2902868499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c8810c160 msgr2=0x7f1c8819cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c7ffff640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c8810c160 0x7f1c8819cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.183+0000 7f1c7ffff640 1 -- 172.21.15.33:0/2902868499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8819d4b0 con 0x7f1c88103650 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.184+0000 7f1c7ffff640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c88103650 0x7f1c88198a40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000e9f0 tx=0x7f1c7000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.184+0000 7f1c84ff9640 1 -- 172.21.15.33:0/2902868499 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7000ccc0 con 0x7f1c88103650 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.184+0000 7f1c84ff9640 1 -- 172.21.15.33:0/2902868499 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c7000ce60 con 0x7f1c88103650 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.184+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/2902868499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8819d7a0 con 0x7f1c88103650 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.184+0000 7f1c84ff9640 1 -- 172.21.15.33:0/2902868499 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70010690 con 0x7f1c88103650 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.184+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/2902868499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c881008c0 con 0x7f1c88103650 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.187+0000 7f1c84ff9640 1 -- 172.21.15.33:0/2902868499 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1c700046c0 con 0x7f1c88103650 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.187+0000 7f1c84ff9640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c740a32c0 0x7f1c740a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.187+0000 7f1c8f987740 1 Processor -- start 2024-09-16T00:39:09.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.188+0000 7f1c8f987740 1 -- start start 2024-09-16T00:39:09.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.188+0000 7f1c84ff9640 1 -- 172.21.15.33:0/2902868499 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1c700d0070 con 0x7f1c88103650 2024-09-16T00:39:09.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.188+0000 7f1c86ffd640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c740a32c0 0x7f1c740a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.188+0000 7f1c8f987740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528b5b40 0x5645528b4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.188+0000 7f1c8f987740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b45d0 0x5645528b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.188+0000 7f1c8f987740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528bf370 0x5645528c1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.188+0000 7f1c8f987740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56455281a2f0 con 0x5645528b5b40 2024-09-16T00:39:09.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.188+0000 7f1c8f987740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5645527c1cf0 con 0x5645528bf370 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.188+0000 7f1c8f987740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5645528162b0 con 0x5645528b45d0 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.189+0000 7f1c7ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b45d0 0x5645528b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.189+0000 7f1c7ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b45d0 0x5645528b4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34578/0 (socket says 172.21.15.33:34578) 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.189+0000 7f1c7ffff640 1 -- 172.21.15.33:0/1168971659 learned_addr learned my addr 172.21.15.33:0/1168971659 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.189+0000 7f1c86ffd640 1 --2- 172.21.15.33:0/1168971659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528b5b40 0x5645528b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.189+0000 7f1c86ffd640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c740a32c0 0x7f1c740a5780 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009230 tx=0x7f1c780058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.189+0000 7f1c7ffff640 1 -- 172.21.15.33:0/1168971659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528bf370 msgr2=0x5645528c1760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.189+0000 7f1c7ffff640 1 --2- 172.21.15.33:0/1168971659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528bf370 0x5645528c1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.189+0000 7f1c7ffff640 1 -- 172.21.15.33:0/1168971659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528b5b40 msgr2=0x5645528b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.189+0000 7f1c7ffff640 1 --2- 172.21.15.33:0/1168971659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528b5b40 0x5645528b4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.189+0000 7f1c7ffff640 1 -- 172.21.15.33:0/1168971659 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645527a63a0 con 0x5645528b45d0 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.190+0000 7f1c7ffff640 1 --2- 172.21.15.33:0/1168971659 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b45d0 0x5645528b4a10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000d010 tx=0x7f1c7009bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69a70f86873e04ff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.190+0000 7f1c7d7fa640 1 -- 172.21.15.33:0/1168971659 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c700a4e80 con 0x5645528b45d0 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.190+0000 7f1c7d7fa640 1 -- 172.21.15.33:0/1168971659 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c700bae50 con 0x5645528b45d0 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.190+0000 7f1c7d7fa640 1 -- 172.21.15.33:0/1168971659 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c700a49f0 con 0x5645528b45d0 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.191+0000 7f1c8f987740 1 -- 172.21.15.33:0/1168971659 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b45d0 msgr2=0x5645528b4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.191+0000 7f1c8f987740 1 --2- 172.21.15.33:0/1168971659 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b45d0 0x5645528b4a10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000d010 tx=0x7f1c7009bc30 comp rx=0 tx=0).stop 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.191+0000 7f1c8f987740 1 -- 172.21.15.33:0/1168971659 shutdown_connections 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.191+0000 7f1c8f987740 1 --2- 172.21.15.33:0/1168971659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528bf370 0x5645528c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.191+0000 7f1c8f987740 1 --2- 172.21.15.33:0/1168971659 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b45d0 0x5645528b4a10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.191+0000 7f1c8f987740 1 --2- 172.21.15.33:0/1168971659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528b5b40 0x5645528b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.191+0000 7f1c8f987740 1 -- 172.21.15.33:0/1168971659 >> 172.21.15.33:0/1168971659 conn(0x56455287d310 msgr2=0x5645528b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.191+0000 7f1c8f987740 1 -- 172.21.15.33:0/1168971659 shutdown_connections 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.191+0000 7f1c8f987740 1 -- 172.21.15.33:0/1168971659 wait complete. 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.191+0000 7f1c8f987740 1 -- start start 2024-09-16T00:39:09.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.191+0000 7f1c8f987740 1 Processor -- start 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c8f987740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528bed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c8f987740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b5b40 0x5645528bc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c8f987740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528bf370 0x5645528bc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c8f987740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5645528c2070 con 0x5645528bf370 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c8f987740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5645527d5060 con 0x5645528b45d0 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c8f987740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5645528c21f0 con 0x5645528b5b40 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c8f987740 10 client.0 _ll_register_callbacks cb 0x5645525288c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c86ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528bed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c7ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b5b40 0x5645528bc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c877fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528bf370 0x5645528bc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c86ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528bed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:50618/0 (socket says 172.21.15.33:50618) 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c86ffd640 1 -- 172.21.15.33:0/3653895505 learned_addr learned my addr 172.21.15.33:0/3653895505 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:09.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c877fe640 1 -- 172.21.15.33:0/3653895505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b5b40 msgr2=0x5645528bc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c877fe640 1 --2- 172.21.15.33:0/3653895505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b5b40 0x5645528bc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c877fe640 1 -- 172.21.15.33:0/3653895505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 msgr2=0x5645528bed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c877fe640 1 --2- 172.21.15.33:0/3653895505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528bed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c877fe640 1 -- 172.21.15.33:0/3653895505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56455258b3c0 con 0x5645528bf370 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.192+0000 7f1c877fe640 1 --2- 172.21.15.33:0/3653895505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528bf370 0x5645528bc650 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f1c80005fd0 tx=0x7f1c80011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4263e7965063fe0e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.193+0000 7f1c6e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.193+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3653895505 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8000fb70 con 0x5645528bf370 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.193+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3653895505 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1c8000fd10 con 0x5645528bf370 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.193+0000 7f1c8f987740 1 -- 172.21.15.33:0/3653895505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c700bae50 con 0x5645528bf370 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.193+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3653895505 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1c80004b20 con 0x5645528bf370 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.193+0000 7f1c8f987740 10 client.19497 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.193+0000 7f1c8f987740 1 -- 172.21.15.33:0/3653895505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5645528cdba0 con 0x5645528bf370 2024-09-16T00:39:09.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.193+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3653895505 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c700bae50 con 0x5645528bf370 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.193+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3653895505 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f1c80019410 con 0x5645528bf370 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.194+0000 7f1c8f987740 10 client.19497 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.194+0000 7f1c8f987740 10 client.19497 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.194+0000 7f1c8f987740 20 client.19497 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.194+0000 7f1c8f987740 1 --2- 172.21.15.33:0/3653895505 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5645528cf060 0x5645528d1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.194+0000 7f1c8f987740 4 client.19497 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.194+0000 7f1c8f987740 1 -- 172.21.15.33:0/3653895505 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564552827860 con 0x5645528cf060 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.194+0000 7f1c86ffd640 1 --2- 172.21.15.33:0/3653895505 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5645528cf060 0x5645528d1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.194+0000 7f1c86ffd640 1 --2- 172.21.15.33:0/3653895505 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5645528cf060 0x5645528d1450 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3f89ac3d50f006c0 server_cookie=7e1d603cfcbed23d in_seq=0 out_seq=0 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.194+0000 7f1c6e7fc640 10 client.19497 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.202+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/3653895505 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564552827860 con 0x5645528cf060 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c6e7fc640 10 client.19497 handle_command_reply: tid=0 2024-09-16T00:39:09.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c8f987740 1 client.19497 shutdown 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c8f987740 1 -- 172.21.15.33:0/3653895505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528bf370 msgr2=0x5645528bc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c8f987740 1 --2- 172.21.15.33:0/3653895505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528bf370 0x5645528bc650 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f1c80005fd0 tx=0x7f1c80011db0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c8f987740 1 -- 172.21.15.33:0/3653895505 shutdown_connections 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c8f987740 1 --2- 172.21.15.33:0/3653895505 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5645528cf060 0x5645528d1450 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c6e7fc640 0 client.19497 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c8f987740 1 --2- 172.21.15.33:0/3653895505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528bf370 0x5645528bc650 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c8f987740 1 --2- 172.21.15.33:0/3653895505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b5b40 0x5645528bc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c8f987740 1 --2- 172.21.15.33:0/3653895505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528bed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c8f987740 1 -- 172.21.15.33:0/3653895505 >> 172.21.15.33:0/3653895505 conn(0x56455287d310 msgr2=0x5645528b6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c8f987740 1 -- 172.21.15.33:0/3653895505 shutdown_connections 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.203+0000 7f1c8f987740 1 -- 172.21.15.33:0/3653895505 wait complete. 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.204+0000 7f1c8f987740 20 client.19497 trim_cache size 0 max 16384 2024-09-16T00:39:09.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.219+0000 7f1c8f987740 1 Processor -- start 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.219+0000 7f1c8f987740 1 -- start start 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c8f987740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56455287d310 0x5645528b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c8f987740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528b7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c8f987740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b5b40 0x5645528d6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c8f987740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881af7f0 con 0x56455287d310 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c8f987740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881ad0c0 con 0x5645528b45d0 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c8f987740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881aaf10 con 0x5645528b5b40 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c86ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56455287d310 0x5645528b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c86ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56455287d310 0x5645528b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38204/0 (socket says 172.21.15.33:38204) 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c86ffd640 1 -- 172.21.15.33:0/2170784061 learned_addr learned my addr 172.21.15.33:0/2170784061 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c7ffff640 1 --2- 172.21.15.33:0/2170784061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528b7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c877fe640 1 --2- 172.21.15.33:0/2170784061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b5b40 0x5645528d6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c86ffd640 1 -- 172.21.15.33:0/2170784061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b5b40 msgr2=0x5645528d6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:09.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c86ffd640 1 --2- 172.21.15.33:0/2170784061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b5b40 0x5645528d6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c86ffd640 1 -- 172.21.15.33:0/2170784061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 msgr2=0x5645528b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:09.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c86ffd640 1 --2- 172.21.15.33:0/2170784061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528b7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.220+0000 7f1c86ffd640 1 -- 172.21.15.33:0/2170784061 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645528bf100 con 0x56455287d310 2024-09-16T00:39:09.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.221+0000 7f1c86ffd640 1 --2- 172.21.15.33:0/2170784061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56455287d310 0x5645528b7120 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f1c78006010 tx=0x7f1c78046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9c4e8aeb6231fb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:09.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.221+0000 7f1c7cff9640 1 -- 172.21.15.33:0/2170784061 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c78035050 con 0x56455287d310 2024-09-16T00:39:09.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.221+0000 7f1c7cff9640 1 -- 172.21.15.33:0/2170784061 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c7803ab50 con 0x56455287d310 2024-09-16T00:39:09.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.221+0000 7f1c7cff9640 1 -- 172.21.15.33:0/2170784061 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c78003c40 con 0x56455287d310 2024-09-16T00:39:09.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.222+0000 7f1c8f987740 1 -- 172.21.15.33:0/2170784061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56455287d310 msgr2=0x5645528b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:09.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.222+0000 7f1c8f987740 1 --2- 172.21.15.33:0/2170784061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56455287d310 0x5645528b7120 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f1c78006010 tx=0x7f1c78046000 comp rx=0 tx=0).stop 2024-09-16T00:39:09.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.222+0000 7f1c8f987740 1 -- 172.21.15.33:0/2170784061 shutdown_connections 2024-09-16T00:39:09.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.222+0000 7f1c8f987740 1 --2- 172.21.15.33:0/2170784061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5645528b5b40 0x5645528d6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.222+0000 7f1c8f987740 1 --2- 172.21.15.33:0/2170784061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528b7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.222+0000 7f1c8f987740 1 --2- 172.21.15.33:0/2170784061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56455287d310 0x5645528b7120 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.222+0000 7f1c8f987740 1 -- 172.21.15.33:0/2170784061 >> 172.21.15.33:0/2170784061 conn(0x5645528d17c0 msgr2=0x5645528be0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.222+0000 7f1c8f987740 1 -- 172.21.15.33:0/2170784061 shutdown_connections 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.222+0000 7f1c8f987740 1 -- 172.21.15.33:0/2170784061 wait complete. 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c8f987740 1 -- start start 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c8f987740 1 Processor -- start 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c8f987740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56455287d310 0x5645528c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c8f987740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c8f987740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528b5b40 0x5645528c4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c8f987740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881ab580 con 0x5645528b5b40 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c8f987740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881ac9f0 con 0x5645528b45d0 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c8f987740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881abc50 con 0x56455287d310 2024-09-16T00:39:09.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c8f987740 10 client.0 _ll_register_callbacks cb 0x7f1c80009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:09.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c7ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c86ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56455287d310 0x5645528c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c7ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:50636/0 (socket says 172.21.15.33:50636) 2024-09-16T00:39:09.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c7ffff640 1 -- 172.21.15.33:0/631608729 learned_addr learned my addr 172.21.15.33:0/631608729 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:09.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.223+0000 7f1c877fe640 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528b5b40 0x5645528c4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c877fe640 1 -- 172.21.15.33:0/631608729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56455287d310 msgr2=0x5645528c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:09.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c877fe640 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56455287d310 0x5645528c7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c877fe640 1 -- 172.21.15.33:0/631608729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 msgr2=0x5645528c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:09.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c877fe640 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c877fe640 1 -- 172.21.15.33:0/631608729 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645528c4990 con 0x5645528b5b40 2024-09-16T00:39:09.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c7ffff640 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:39:09.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c877fe640 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528b5b40 0x5645528c4450 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f1c80019a40 tx=0x7f1c8000fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1559b93915b80d07 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c6e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/631608729 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8002d070 con 0x5645528b5b40 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/631608729 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1c80003b80 con 0x5645528b5b40 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c8f987740 1 -- 172.21.15.33:0/631608729 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c7803ab50 con 0x5645528b5b40 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/631608729 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1c800114f0 con 0x5645528b5b40 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c8f987740 10 client.19503 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.224+0000 7f1c8f987740 1 -- 172.21.15.33:0/631608729 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56455293a2d0 con 0x5645528b5b40 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.225+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/631608729 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c80005bd0 con 0x5645528b5b40 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.225+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/631608729 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f1c80027550 con 0x5645528b5b40 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.225+0000 7f1c8f987740 10 client.19503 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.225+0000 7f1c8f987740 10 client.19503 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.225+0000 7f1c8f987740 20 client.19503 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.225+0000 7f1c8f987740 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5645528bf370 0x56455293ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:09.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.225+0000 7f1c8f987740 4 client.19503 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.225+0000 7f1c8f987740 1 -- 172.21.15.33:0/631608729 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f1c7803ab50 con 0x5645528bf370 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.226+0000 7f1c86ffd640 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5645528bf370 0x56455293ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.226+0000 7f1c86ffd640 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5645528bf370 0x56455293ca30 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=495b7ee006609e98 server_cookie=a040c04bf520e9e4 in_seq=0 out_seq=0 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.226+0000 7f1c6e7fc640 10 client.19503 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c6e7fc640 1 -- 172.21.15.33:0/631608729 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f1c7803ab50 con 0x5645528bf370 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c6e7fc640 10 client.19503 handle_command_reply: tid=0 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c8f987740 1 client.19503 shutdown 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c8f987740 1 -- 172.21.15.33:0/631608729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528b5b40 msgr2=0x5645528c4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c8f987740 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528b5b40 0x5645528c4450 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f1c80019a40 tx=0x7f1c8000fe20 comp rx=0 tx=0).stop 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c8f987740 1 -- 172.21.15.33:0/631608729 shutdown_connections 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c8f987740 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5645528bf370 0x56455293ca30 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c8f987740 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645528b5b40 0x5645528c4450 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c8f987740 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5645528b45d0 0x5645528c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c6e7fc640 0 client.19503 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c8f987740 1 --2- 172.21.15.33:0/631608729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56455287d310 0x5645528c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c8f987740 1 -- 172.21.15.33:0/631608729 >> 172.21.15.33:0/631608729 conn(0x5645528d17c0 msgr2=0x5645528af4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c8f987740 1 -- 172.21.15.33:0/631608729 shutdown_connections 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c8f987740 1 -- 172.21.15.33:0/631608729 wait complete. 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.229+0000 7f1c8f987740 20 client.19503 trim_cache size 0 max 16384 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.230+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/2902868499 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c740a32c0 msgr2=0x7f1c740a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.230+0000 7f1c8d5ae640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c740a32c0 0x7f1c740a5780 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009230 tx=0x7f1c780058d0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.230+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/2902868499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c88103650 msgr2=0x7f1c88198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.230+0000 7f1c8d5ae640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c88103650 0x7f1c88198a40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000e9f0 tx=0x7f1c7000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.230+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/2902868499 shutdown_connections 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.230+0000 7f1c8d5ae640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c740a32c0 0x7f1c740a5780 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.230+0000 7f1c8d5ae640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c8810c160 0x7f1c8819cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.230+0000 7f1c8d5ae640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c88103650 0x7f1c88198a40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.230+0000 7f1c8d5ae640 1 --2- 172.21.15.33:0/2902868499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c88102d30 0x7f1c88198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:09.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.230+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/2902868499 >> 172.21.15.33:0/2902868499 conn(0x7f1c880fc790 msgr2=0x7f1c880fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:09.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.230+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/2902868499 shutdown_connections 2024-09-16T00:39:09.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:09.230+0000 7f1c8d5ae640 1 -- 172.21.15.33:0/2902868499 wait complete. 2024-09-16T00:39:10.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 174464 18.71 MB/sec execute 376 sec latency 427.575 ms 2024-09-16T00:39:10.802 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:39:11.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 174482 18.66 MB/sec execute 377 sec latency 1192.260 ms 2024-09-16T00:39:11.440 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:11.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:11 smithi060 ceph-mon[35185]: pgmap v1011: 129 pgs: 129 active+clean; 684 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 378 op/s 2024-09-16T00:39:11.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:11 smithi033 ceph-mon[31243]: pgmap v1011: 129 pgs: 129 active+clean; 684 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 378 op/s 2024-09-16T00:39:11.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:11 smithi083 ceph-mon[35242]: pgmap v1011: 129 pgs: 129 active+clean; 684 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 378 op/s 2024-09-16T00:39:12.095 INFO:tasks.workunit.client.0.smithi033.stdout: 10 174494 18.61 MB/sec execute 378 sec latency 2129.612 ms 2024-09-16T00:39:12.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.110+0000 7fb8d0841640 1 -- 172.21.15.33:0/2762904337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8cc11b140 msgr2=0x7fb8cc11d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:12.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.110+0000 7fb8d0841640 1 --2- 172.21.15.33:0/2762904337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8cc11b140 0x7fb8cc11d530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009ca0 tx=0x7fb8c402a8d0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.111+0000 7fb8d0841640 1 -- 172.21.15.33:0/2762904337 shutdown_connections 2024-09-16T00:39:12.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.111+0000 7fb8d0841640 1 --2- 172.21.15.33:0/2762904337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8cc11b140 0x7fb8cc11d530 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.111+0000 7fb8d0841640 1 --2- 172.21.15.33:0/2762904337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8cc06ea30 0x7fb8cc06ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.111+0000 7fb8d0841640 1 --2- 172.21.15.33:0/2762904337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8cc110190 0x7fb8cc06e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.111+0000 7fb8d0841640 1 -- 172.21.15.33:0/2762904337 >> 172.21.15.33:0/2762904337 conn(0x7fb8cc073180 msgr2=0x7fb8cc073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.114+0000 7fb8d0841640 1 -- 172.21.15.33:0/2762904337 shutdown_connections 2024-09-16T00:39:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.114+0000 7fb8d0841640 1 -- 172.21.15.33:0/2762904337 wait complete. 2024-09-16T00:39:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.115+0000 7fb8d0841640 1 Processor -- start 2024-09-16T00:39:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.116+0000 7fb8d0841640 1 -- start start 2024-09-16T00:39:12.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.116+0000 7fb8d0841640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8cc110190 0x7fb8cc1198f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.116+0000 7fb8d0841640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8cc11b140 0x7fb8cc112980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.116+0000 7fb8d0841640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8cc112ec0 0x7fb8cc113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.116+0000 7fb8d0841640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc120bf0 con 0x7fb8cc11b140 2024-09-16T00:39:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.116+0000 7fb8d0841640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc120a70 con 0x7fb8cc112ec0 2024-09-16T00:39:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.116+0000 7fb8d0841640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc120d70 con 0x7fb8cc110190 2024-09-16T00:39:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.117+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8cc11b140 0x7fb8cc112980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.117+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8cc11b140 0x7fb8cc112980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55326/0 (socket says 172.21.15.33:55326) 2024-09-16T00:39:12.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.117+0000 7fb8caffd640 1 -- 172.21.15.33:0/3707677864 learned_addr learned my addr 172.21.15.33:0/3707677864 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.117+0000 7fb8caffd640 1 -- 172.21.15.33:0/3707677864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8cc110190 msgr2=0x7fb8cc1198f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T00:39:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.117+0000 7fb8caffd640 1 --2- 172.21.15.33:0/3707677864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8cc110190 0x7fb8cc1198f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.117+0000 7fb8caffd640 1 -- 172.21.15.33:0/3707677864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8cc112ec0 msgr2=0x7fb8cc113370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T00:39:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.117+0000 7fb8caffd640 1 --2- 172.21.15.33:0/3707677864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8cc112ec0 0x7fb8cc113370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.117+0000 7fb8caffd640 1 -- 172.21.15.33:0/3707677864 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8cc117420 con 0x7fb8cc11b140 2024-09-16T00:39:12.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.118+0000 7fb8caffd640 1 --2- 172.21.15.33:0/3707677864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8cc11b140 0x7fb8cc112980 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc006fd0 tx=0x7fb8bc00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.121+0000 7fb8c8ff9640 1 -- 172.21.15.33:0/3707677864 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc015020 con 0x7fb8cc11b140 2024-09-16T00:39:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.121+0000 7fb8c8ff9640 1 -- 172.21.15.33:0/3707677864 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8bc011d10 con 0x7fb8cc11b140 2024-09-16T00:39:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.121+0000 7fb8c8ff9640 1 -- 172.21.15.33:0/3707677864 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc00f2e0 con 0x7fb8cc11b140 2024-09-16T00:39:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.122+0000 7fb8d0841640 1 -- 172.21.15.33:0/3707677864 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8cc117710 con 0x7fb8cc11b140 2024-09-16T00:39:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.122+0000 7fb8d0841640 1 -- 172.21.15.33:0/3707677864 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8cc117bd0 con 0x7fb8cc11b140 2024-09-16T00:39:12.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.129+0000 7fb8d3e7c740 1 Processor -- start 2024-09-16T00:39:12.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.129+0000 7fb8d3e7c740 1 -- start start 2024-09-16T00:39:12.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.129+0000 7fb8d3e7c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd4000cf0 0x55fbd3fff240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.129+0000 7fb8d3e7c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd3fff780 0x55fbd3fffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.129+0000 7fb8d3e7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 0x55fbd400c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.129+0000 7fb8d3e7c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fbd3f652f0 con 0x55fbd3fff780 2024-09-16T00:39:12.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.129+0000 7fb8d3e7c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55fbd3f0ccf0 con 0x55fbd4000cf0 2024-09-16T00:39:12.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.129+0000 7fb8d3e7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fbd3f612b0 con 0x55fbd400a520 2024-09-16T00:39:12.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.130+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 0x55fbd400c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:12.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.130+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 0x55fbd400c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:56558/0 (socket says 172.21.15.33:56558) 2024-09-16T00:39:12.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.130+0000 7fb8caffd640 1 -- 172.21.15.33:0/2503958851 learned_addr learned my addr 172.21.15.33:0/2503958851 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:12.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.131+0000 7fb8c8ff9640 1 -- 172.21.15.33:0/3707677864 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb8bc018020 con 0x7fb8cc11b140 2024-09-16T00:39:12.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.131+0000 7fb8caffd640 1 -- 172.21.15.33:0/2503958851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd4000cf0 msgr2=0x55fbd3fff240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:12.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.131+0000 7fb8caffd640 1 --2- 172.21.15.33:0/2503958851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd4000cf0 0x55fbd3fff240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.131+0000 7fb8caffd640 1 -- 172.21.15.33:0/2503958851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd3fff780 msgr2=0x55fbd3fffbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:12.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.131+0000 7fb8caffd640 1 --2- 172.21.15.33:0/2503958851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd3fff780 0x55fbd3fffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.131+0000 7fb8caffd640 1 -- 172.21.15.33:0/2503958851 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbd3ef13a0 con 0x55fbd400a520 2024-09-16T00:39:12.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.133+0000 7fb8c8ff9640 1 --2- 172.21.15.33:0/3707677864 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb8b80a3450 0x7fb8b80a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.133+0000 7fb8c8ff9640 1 -- 172.21.15.33:0/3707677864 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb8bc0cfba0 con 0x7fb8cc11b140 2024-09-16T00:39:12.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.134+0000 7fb8cbfff640 1 --2- 172.21.15.33:0/3707677864 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb8b80a3450 0x7fb8b80a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:12.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.134+0000 7fb8caffd640 1 --2- 172.21.15.33:0/2503958851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 0x55fbd400c910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc0d5700 tx=0x7fb8bc0d5bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf62c4de5d4d9552 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.135+0000 7fb8c1ffb640 1 -- 172.21.15.33:0/2503958851 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc098a60 con 0x55fbd400a520 2024-09-16T00:39:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.135+0000 7fb8c1ffb640 1 -- 172.21.15.33:0/2503958851 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8bc09cec0 con 0x55fbd400a520 2024-09-16T00:39:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.135+0000 7fb8c1ffb640 1 -- 172.21.15.33:0/2503958851 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc09c120 con 0x55fbd400a520 2024-09-16T00:39:12.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.136+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2503958851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 msgr2=0x55fbd400c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:12.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.136+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/2503958851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 0x55fbd400c910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc0d5700 tx=0x7fb8bc0d5bc0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.137+0000 7fb8cbfff640 1 --2- 172.21.15.33:0/3707677864 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb8b80a3450 0x7fb8b80a5910 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4006010 tx=0x7fb8c4005f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:12.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.137+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2503958851 shutdown_connections 2024-09-16T00:39:12.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.138+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/2503958851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 0x55fbd400c910 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.138+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/2503958851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd3fff780 0x55fbd3fffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.138+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/2503958851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd4000cf0 0x55fbd3fff240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.138+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2503958851 >> 172.21.15.33:0/2503958851 conn(0x55fbd3fc8310 msgr2=0x55fbd3ffbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:12.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.143+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2503958851 shutdown_connections 2024-09-16T00:39:12.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.146+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2503958851 wait complete. 2024-09-16T00:39:12.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.146+0000 7fb8d3e7c740 1 -- start start 2024-09-16T00:39:12.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.147+0000 7fb8d3e7c740 1 Processor -- start 2024-09-16T00:39:12.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.147+0000 7fb8d3e7c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd3fff780 0x55fbd4009f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.147+0000 7fb8d3e7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd4000cf0 0x55fbd4007200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.147+0000 7fb8d3e7c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd400a520 0x55fbd4007860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.147+0000 7fb8d3e7c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fbd400d0a0 con 0x55fbd400a520 2024-09-16T00:39:12.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.147+0000 7fb8d3e7c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55fbd3f20060 con 0x55fbd3fff780 2024-09-16T00:39:12.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.147+0000 7fb8d3e7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fbd400d220 con 0x55fbd4000cf0 2024-09-16T00:39:12.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.147+0000 7fb8d3e7c740 10 client.0 _ll_register_callbacks cb 0x55fbd3c738c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:12.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.148+0000 7fb8cbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd400a520 0x55fbd4007860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:12.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.148+0000 7fb8cbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd400a520 0x55fbd4007860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55344/0 (socket says 172.21.15.33:55344) 2024-09-16T00:39:12.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.148+0000 7fb8cbfff640 1 -- 172.21.15.33:0/2279127952 learned_addr learned my addr 172.21.15.33:0/2279127952 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:12.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.148+0000 7fb8caffd640 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd4000cf0 0x55fbd4007200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:12.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.150+0000 7fb8cb7fe640 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd3fff780 0x55fbd4009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:12.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.151+0000 7fb8caffd640 1 -- 172.21.15.33:0/2279127952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd3fff780 msgr2=0x55fbd4009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:12.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.151+0000 7fb8caffd640 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd3fff780 0x55fbd4009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.151+0000 7fb8caffd640 1 -- 172.21.15.33:0/2279127952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd400a520 msgr2=0x55fbd4007860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:12.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.151+0000 7fb8caffd640 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd400a520 0x55fbd4007860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.151+0000 7fb8caffd640 1 -- 172.21.15.33:0/2279127952 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbd3cd63c0 con 0x55fbd4000cf0 2024-09-16T00:39:12.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.151+0000 7fb8caffd640 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd4000cf0 0x55fbd4007200 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc0d51c0 tx=0x7fb8bc09c120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e098cc531cc3277 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:12.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.151+0000 7fb8cb7fe640 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd3fff780 0x55fbd4009f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:39:12.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.151+0000 7fb8cbfff640 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd400a520 0x55fbd4007860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:39:12.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.153+0000 7fb8aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:39:12.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.157+0000 7fb8aaffd640 1 -- 172.21.15.33:0/2279127952 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc0d4070 con 0x55fbd4000cf0 2024-09-16T00:39:12.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.157+0000 7fb8aaffd640 1 -- 172.21.15.33:0/2279127952 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb8bc0d5250 con 0x55fbd4000cf0 2024-09-16T00:39:12.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.157+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2279127952 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8bc09cec0 con 0x55fbd4000cf0 2024-09-16T00:39:12.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.165+0000 7fb8aaffd640 1 -- 172.21.15.33:0/2279127952 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb8bc09c300 con 0x55fbd4000cf0 2024-09-16T00:39:12.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.166+0000 7fb8d3e7c740 10 client.28699 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:12.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.166+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2279127952 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fbd400d0a0 con 0x55fbd4000cf0 2024-09-16T00:39:12.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.173+0000 7fb8aaffd640 1 -- 172.21.15.33:0/2279127952 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8bc09cec0 con 0x55fbd4000cf0 2024-09-16T00:39:12.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.173+0000 7fb8aaffd640 1 -- 172.21.15.33:0/2279127952 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb8bc0b0ce0 con 0x55fbd4000cf0 2024-09-16T00:39:12.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.174+0000 7fb8d3e7c740 10 client.28699 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:12.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.174+0000 7fb8d3e7c740 10 client.28699 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:39:12.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.174+0000 7fb8d3e7c740 20 client.28699 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:12.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.174+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55fbd4019d80 0x55fbd401c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.174+0000 7fb8d3e7c740 4 client.28699 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:39:12.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.174+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2279127952 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbd3f72860 con 0x55fbd4019d80 2024-09-16T00:39:12.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.176+0000 7fb8cb7fe640 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55fbd4019d80 0x55fbd401c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:12.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.178+0000 7fb8cb7fe640 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55fbd4019d80 0x55fbd401c170 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=62913094c9d6392 server_cookie=347a8021dafdb98 in_seq=0 out_seq=0 2024-09-16T00:39:12.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.179+0000 7fb8aaffd640 10 client.28699 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:39:12.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.179+0000 7fb8aaffd640 1 -- 172.21.15.33:0/2279127952 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fbd3f72860 con 0x55fbd4019d80 2024-09-16T00:39:12.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.180+0000 7fb8aaffd640 10 client.28699 handle_command_reply: tid=0 2024-09-16T00:39:12.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.180+0000 7fb8d3e7c740 1 client.28699 shutdown 2024-09-16T00:39:12.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.180+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2279127952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd4000cf0 msgr2=0x55fbd4007200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:12.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.180+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd4000cf0 0x55fbd4007200 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc0d51c0 tx=0x7fb8bc09c120 comp rx=0 tx=0).stop 2024-09-16T00:39:12.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.181+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2279127952 shutdown_connections 2024-09-16T00:39:12.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.181+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd400a520 0x55fbd4007860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.181+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd4000cf0 0x55fbd4007200 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.181+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55fbd4019d80 0x55fbd401c170 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.181+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/2279127952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd3fff780 0x55fbd4009f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.181+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2279127952 >> 172.21.15.33:0/2279127952 conn(0x55fbd3fc8310 msgr2=0x55fbd4001ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:12.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.181+0000 7fb8aaffd640 0 client.28699 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:39:12.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.186+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2279127952 shutdown_connections 2024-09-16T00:39:12.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.186+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/2279127952 wait complete. 2024-09-16T00:39:12.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.186+0000 7fb8d3e7c740 20 client.28699 trim_cache size 0 max 16384 2024-09-16T00:39:12.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.212+0000 7fb8d3e7c740 1 Processor -- start 2024-09-16T00:39:12.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.213+0000 7fb8d3e7c740 1 -- start start 2024-09-16T00:39:12.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.213+0000 7fb8d3e7c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd3fff780 0x55fbd4009170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.213+0000 7fb8d3e7c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd4000cf0 0x55fbd40022d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.213+0000 7fb8d3e7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 0x55fbd4002810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.213+0000 7fb8d3e7c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac012980 con 0x55fbd4000cf0 2024-09-16T00:39:12.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.214+0000 7fb8cb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd3fff780 0x55fbd4009170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:12.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.214+0000 7fb8cbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 0x55fbd4002810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:12.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.214+0000 7fb8cb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd3fff780 0x55fbd4009170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55362/0 (socket says 172.21.15.33:55362) 2024-09-16T00:39:12.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.214+0000 7fb8cb7fe640 1 -- 172.21.15.33:0/441155363 learned_addr learned my addr 172.21.15.33:0/441155363 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:12.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.214+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/441155363 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac010250 con 0x55fbd3fff780 2024-09-16T00:39:12.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.214+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/441155363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac00e0a0 con 0x55fbd400a520 2024-09-16T00:39:12.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.215+0000 7fb8cbfff640 1 -- 172.21.15.33:0/441155363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd3fff780 msgr2=0x55fbd4009170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:12.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.215+0000 7fb8cbfff640 1 --2- 172.21.15.33:0/441155363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd3fff780 0x55fbd4009170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.215+0000 7fb8cbfff640 1 -- 172.21.15.33:0/441155363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd4000cf0 msgr2=0x55fbd40022d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:12.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.215+0000 7fb8cbfff640 1 --2- 172.21.15.33:0/441155363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd4000cf0 0x55fbd40022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.215+0000 7fb8cbfff640 1 -- 172.21.15.33:0/441155363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbd3ffcf30 con 0x55fbd400a520 2024-09-16T00:39:12.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.215+0000 7fb8cbfff640 1 --2- 172.21.15.33:0/441155363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 0x55fbd4002810 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009ca0 tx=0x7fb8c403d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=968febda2b4d62a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:12.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.215+0000 7fb8c17fa640 1 -- 172.21.15.33:0/441155363 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40098a0 con 0x55fbd400a520 2024-09-16T00:39:12.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.216+0000 7fb8c17fa640 1 -- 172.21.15.33:0/441155363 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8c404d070 con 0x55fbd400a520 2024-09-16T00:39:12.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.217+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/441155363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 msgr2=0x55fbd4002810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:12.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.217+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/441155363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 0x55fbd4002810 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009ca0 tx=0x7fb8c403d000 comp rx=0 tx=0).stop 2024-09-16T00:39:12.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.219+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/441155363 shutdown_connections 2024-09-16T00:39:12.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.219+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/441155363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd400a520 0x55fbd4002810 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.219+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/441155363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd4000cf0 0x55fbd40022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.219+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/441155363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd3fff780 0x55fbd4009170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.219+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/441155363 >> 172.21.15.33:0/441155363 conn(0x55fbd3fc8310 msgr2=0x55fbd4008a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:12.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.220+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/441155363 shutdown_connections 2024-09-16T00:39:12.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.220+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/441155363 wait complete. 2024-09-16T00:39:12.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.221+0000 7fb8d3e7c740 1 -- start start 2024-09-16T00:39:12.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.221+0000 7fb8d3e7c740 1 Processor -- start 2024-09-16T00:39:12.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.221+0000 7fb8d3e7c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd3fff780 0x55fbd4012850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.221+0000 7fb8d3e7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd4000cf0 0x55fbd4012d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.221+0000 7fb8d3e7c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd400a520 0x55fbd400dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.221+0000 7fb8d3e7c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac00e710 con 0x55fbd3fff780 2024-09-16T00:39:12.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.221+0000 7fb8d3e7c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac00fb80 con 0x55fbd400a520 2024-09-16T00:39:12.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.221+0000 7fb8d3e7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac00ede0 con 0x55fbd4000cf0 2024-09-16T00:39:12.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.221+0000 7fb8d3e7c740 10 client.0 _ll_register_callbacks cb 0x7fb8bc0d5550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:12.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.222+0000 7fb8cb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd3fff780 0x55fbd4012850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:12.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.222+0000 7fb8cb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd3fff780 0x55fbd4012850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55368/0 (socket says 172.21.15.33:55368) 2024-09-16T00:39:12.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.222+0000 7fb8cb7fe640 1 -- 172.21.15.33:0/191337808 learned_addr learned my addr 172.21.15.33:0/191337808 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:12.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.222+0000 7fb8cb7fe640 1 -- 172.21.15.33:0/191337808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd4000cf0 msgr2=0x55fbd4012d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:39:12.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.222+0000 7fb8cb7fe640 1 --2- 172.21.15.33:0/191337808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd4000cf0 0x55fbd4012d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.222+0000 7fb8cb7fe640 1 -- 172.21.15.33:0/191337808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd400a520 msgr2=0x55fbd400dc10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:39:12.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.222+0000 7fb8cb7fe640 1 --2- 172.21.15.33:0/191337808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd400a520 0x55fbd400dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.222+0000 7fb8cb7fe640 1 -- 172.21.15.33:0/191337808 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbd400e440 con 0x55fbd3fff780 2024-09-16T00:39:12.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.222+0000 7fb8cb7fe640 1 --2- 172.21.15.33:0/191337808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd3fff780 0x55fbd4012850 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4005b90 tx=0x7fb8b4012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7287060833ccad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:12.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.223+0000 7fb8aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:12.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.223+0000 7fb8aaffd640 1 -- 172.21.15.33:0/191337808 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4023670 con 0x55fbd3fff780 2024-09-16T00:39:12.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.224+0000 7fb8aaffd640 1 -- 172.21.15.33:0/191337808 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb8b401a230 con 0x55fbd3fff780 2024-09-16T00:39:12.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.224+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/191337808 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8c404d070 con 0x55fbd3fff780 2024-09-16T00:39:12.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.224+0000 7fb8aaffd640 1 -- 172.21.15.33:0/191337808 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb8b4023810 con 0x55fbd3fff780 2024-09-16T00:39:12.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.225+0000 7fb8d3e7c740 10 client.19512 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:12.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.225+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/191337808 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fbd4084450 con 0x55fbd3fff780 2024-09-16T00:39:12.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.226+0000 7fb8aaffd640 1 -- 172.21.15.33:0/191337808 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8c404d070 con 0x55fbd3fff780 2024-09-16T00:39:12.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.227+0000 7fb8aaffd640 1 -- 172.21.15.33:0/191337808 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb8b400f040 con 0x55fbd3fff780 2024-09-16T00:39:12.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.227+0000 7fb8d3e7c740 10 client.19512 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:12.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.227+0000 7fb8d3e7c740 10 client.19512 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:39:12.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.227+0000 7fb8d3e7c740 20 client.19512 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:12.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.227+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/191337808 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55fbd4084b50 0x55fbd4086f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:12.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.227+0000 7fb8d3e7c740 4 client.19512 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:39:12.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.227+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/191337808 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fbd3f6cd70 con 0x55fbd4084b50 2024-09-16T00:39:12.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.230+0000 7fb8caffd640 1 --2- 172.21.15.33:0/191337808 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55fbd4084b50 0x55fbd4086f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.232+0000 7fb8caffd640 1 --2- 172.21.15.33:0/191337808 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55fbd4084b50 0x55fbd4086f90 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=750095399e894f61 server_cookie=f8e30af8a0426869 in_seq=0 out_seq=0 2024-09-16T00:39:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.232+0000 7fb8aaffd640 10 client.19512 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:39:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.233+0000 7fb8aaffd640 1 -- 172.21.15.33:0/191337808 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fbd3f6cd70 con 0x55fbd4084b50 2024-09-16T00:39:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.233+0000 7fb8aaffd640 10 client.19512 handle_command_reply: tid=0 2024-09-16T00:39:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.233+0000 7fb8d3e7c740 1 client.19512 shutdown 2024-09-16T00:39:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.234+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/191337808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd3fff780 msgr2=0x55fbd4012850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.234+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/191337808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd3fff780 0x55fbd4012850 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4005b90 tx=0x7fb8b4012820 comp rx=0 tx=0).stop 2024-09-16T00:39:12.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.234+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/191337808 shutdown_connections 2024-09-16T00:39:12.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.234+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/191337808 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55fbd4084b50 0x55fbd4086f90 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.234+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/191337808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fbd400a520 0x55fbd400dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.234+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/191337808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fbd4000cf0 0x55fbd4012d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.234+0000 7fb8d3e7c740 1 --2- 172.21.15.33:0/191337808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fbd3fff780 0x55fbd4012850 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.234+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/191337808 >> 172.21.15.33:0/191337808 conn(0x55fbd3fc8310 msgr2=0x55fbd4009280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:12.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.235+0000 7fb8aaffd640 0 client.19512 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:39:12.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.235+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/191337808 shutdown_connections 2024-09-16T00:39:12.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.235+0000 7fb8d3e7c740 1 -- 172.21.15.33:0/191337808 wait complete. 2024-09-16T00:39:12.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.236+0000 7fb8d3e7c740 20 client.19512 trim_cache size 0 max 16384 2024-09-16T00:39:12.238 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:39:12.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.237+0000 7fb8d0841640 1 -- 172.21.15.33:0/3707677864 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb8b80a3450 msgr2=0x7fb8b80a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:12.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.237+0000 7fb8d0841640 1 --2- 172.21.15.33:0/3707677864 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb8b80a3450 0x7fb8b80a5910 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4006010 tx=0x7fb8c4005f40 comp rx=0 tx=0).stop 2024-09-16T00:39:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.237+0000 7fb8d0841640 1 -- 172.21.15.33:0/3707677864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8cc11b140 msgr2=0x7fb8cc112980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.237+0000 7fb8d0841640 1 --2- 172.21.15.33:0/3707677864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8cc11b140 0x7fb8cc112980 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc006fd0 tx=0x7fb8bc00b890 comp rx=0 tx=0).stop 2024-09-16T00:39:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.237+0000 7fb8d0841640 1 -- 172.21.15.33:0/3707677864 shutdown_connections 2024-09-16T00:39:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.237+0000 7fb8d0841640 1 --2- 172.21.15.33:0/3707677864 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb8b80a3450 0x7fb8b80a5910 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.237+0000 7fb8d0841640 1 --2- 172.21.15.33:0/3707677864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8cc112ec0 0x7fb8cc113370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.237+0000 7fb8d0841640 1 --2- 172.21.15.33:0/3707677864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8cc11b140 0x7fb8cc112980 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.237+0000 7fb8d0841640 1 --2- 172.21.15.33:0/3707677864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8cc110190 0x7fb8cc1198f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.237+0000 7fb8d0841640 1 -- 172.21.15.33:0/3707677864 >> 172.21.15.33:0/3707677864 conn(0x7fb8cc073180 msgr2=0x7fb8cc06d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.237+0000 7fb8d0841640 1 -- 172.21.15.33:0/3707677864 shutdown_connections 2024-09-16T00:39:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:12.237+0000 7fb8d0841640 1 -- 172.21.15.33:0/3707677864 wait complete. 2024-09-16T00:39:13.036 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:39:13.096 INFO:tasks.workunit.client.0.smithi033.stdout: 10 174639 18.58 MB/sec execute 379 sec latency 461.994 ms 2024-09-16T00:39:13.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:13 smithi033 ceph-mon[31243]: pgmap v1012: 129 pgs: 129 active+clean; 715 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.5 MiB/s wr, 468 op/s 2024-09-16T00:39:13.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:13 smithi060 ceph-mon[35185]: pgmap v1012: 129 pgs: 129 active+clean; 715 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.5 MiB/s wr, 468 op/s 2024-09-16T00:39:13.514 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:13.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:13 smithi083 ceph-mon[35242]: pgmap v1012: 129 pgs: 129 active+clean; 715 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.5 MiB/s wr, 468 op/s 2024-09-16T00:39:14.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.088+0000 7f74db59e640 1 -- 172.21.15.33:0/724019586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74dc10bfc0 msgr2=0x7f74dc10e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:14.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.088+0000 7f74db59e640 1 --2- 172.21.15.33:0/724019586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74dc10bfc0 0x7f74dc10e460 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f74d4009ca0 tx=0x7f74d402a960 comp rx=0 tx=0).stop 2024-09-16T00:39:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.088+0000 7f74db59e640 1 -- 172.21.15.33:0/724019586 shutdown_connections 2024-09-16T00:39:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.088+0000 7f74db59e640 1 --2- 172.21.15.33:0/724019586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74dc10bfc0 0x7f74dc10e460 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.088+0000 7f74db59e640 1 --2- 172.21.15.33:0/724019586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74dc1034b0 0x7f74dc10b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.088+0000 7f74db59e640 1 --2- 172.21.15.33:0/724019586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74dc102b90 0x7f74dc102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.088+0000 7f74db59e640 1 -- 172.21.15.33:0/724019586 >> 172.21.15.33:0/724019586 conn(0x7f74dc0fc610 msgr2=0x7f74dc0fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.088+0000 7f74db59e640 1 -- 172.21.15.33:0/724019586 shutdown_connections 2024-09-16T00:39:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.088+0000 7f74db59e640 1 -- 172.21.15.33:0/724019586 wait complete. 2024-09-16T00:39:14.096 INFO:tasks.workunit.client.0.smithi033.stdout: 10 174973 18.58 MB/sec execute 380 sec latency 157.110 ms 2024-09-16T00:39:14.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.089+0000 7f74db59e640 1 Processor -- start 2024-09-16T00:39:14.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.089+0000 7f74db59e640 1 -- start start 2024-09-16T00:39:14.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.091+0000 7f74db59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74dc102b90 0x7f74dc19ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.095+0000 7f74db59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74dc1034b0 0x7f74dc19f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.096+0000 7f74d9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74dc1034b0 0x7f74dc19f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.096+0000 7f74da59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74dc102b90 0x7f74dc19ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.097+0000 7f74da59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74dc102b90 0x7f74dc19ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55384/0 (socket says 172.21.15.33:55384) 2024-09-16T00:39:14.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.097+0000 7f74db59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74dc10bfc0 0x7f74dc10ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.097+0000 7f74db59e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc111d90 con 0x7f74dc102b90 2024-09-16T00:39:14.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.097+0000 7f74db59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc111c10 con 0x7f74dc1034b0 2024-09-16T00:39:14.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.097+0000 7f74db59e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc111f10 con 0x7f74dc10bfc0 2024-09-16T00:39:14.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.097+0000 7f74da59c640 1 -- 172.21.15.33:0/3767967667 learned_addr learned my addr 172.21.15.33:0/3767967667 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:14.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.097+0000 7f74dad9d640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74dc10bfc0 0x7f74dc10ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.099+0000 7f74dad9d640 1 -- 172.21.15.33:0/3767967667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74dc1034b0 msgr2=0x7f74dc19f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:14.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.099+0000 7f74dad9d640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74dc1034b0 0x7f74dc19f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.099+0000 7f74dad9d640 1 -- 172.21.15.33:0/3767967667 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74dc102b90 msgr2=0x7f74dc19ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:14.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.099+0000 7f74dad9d640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74dc102b90 0x7f74dc19ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.099+0000 7f74dad9d640 1 -- 172.21.15.33:0/3767967667 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc10b430 con 0x7f74dc10bfc0 2024-09-16T00:39:14.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.099+0000 7f74dad9d640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74dc10bfc0 0x7f74dc10ab20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f74d402af50 tx=0x7f74d4003ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:14.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.102+0000 7f74cb7fe640 1 -- 172.21.15.33:0/3767967667 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4034030 con 0x7f74dc10bfc0 2024-09-16T00:39:14.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.102+0000 7f74cb7fe640 1 -- 172.21.15.33:0/3767967667 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d402fdc0 con 0x7f74dc10bfc0 2024-09-16T00:39:14.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.102+0000 7f74cb7fe640 1 -- 172.21.15.33:0/3767967667 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d40046a0 con 0x7f74dc10bfc0 2024-09-16T00:39:14.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.102+0000 7f74db59e640 1 -- 172.21.15.33:0/3767967667 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74dc10b660 con 0x7f74dc10bfc0 2024-09-16T00:39:14.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.103+0000 7f74db59e640 1 -- 172.21.15.33:0/3767967667 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74dc19aaa0 con 0x7f74dc10bfc0 2024-09-16T00:39:14.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.107+0000 7f74e304a740 1 Processor -- start 2024-09-16T00:39:14.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.109+0000 7f74cb7fe640 1 -- 172.21.15.33:0/3767967667 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f74d4037020 con 0x7f74dc10bfc0 2024-09-16T00:39:14.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.110+0000 7f74e304a740 1 -- start start 2024-09-16T00:39:14.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.114+0000 7f74e304a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b3b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.114+0000 7f74e304a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3b780 0x555dd4b3bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.114+0000 7f74e304a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b48910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.114+0000 7f74e304a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555dd4aa12f0 con 0x555dd4b3ccf0 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.114+0000 7f74e304a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555dd4a48cf0 con 0x555dd4b3b780 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.114+0000 7f74e304a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555dd4a9d2b0 con 0x555dd4b46520 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.115+0000 7f74da59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3b780 0x555dd4b3bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.115+0000 7f74d9d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b48910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.115+0000 7f74d9d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b48910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:56614/0 (socket says 172.21.15.33:56614) 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.115+0000 7f74da59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3b780 0x555dd4b3bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55398/0 (socket says 172.21.15.33:55398) 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.115+0000 7f74da59c640 1 -- 172.21.15.33:0/2879362470 learned_addr learned my addr 172.21.15.33:0/2879362470 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.116+0000 7f74d9d9b640 1 --2- 172.21.15.33:0/2879362470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b3b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.116+0000 7f74da59c640 1 -- 172.21.15.33:0/2879362470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 msgr2=0x555dd4b48910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.116+0000 7f74da59c640 1 --2- 172.21.15.33:0/2879362470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b48910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.116+0000 7f74da59c640 1 -- 172.21.15.33:0/2879362470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 msgr2=0x555dd4b3b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.116+0000 7f74da59c640 1 --2- 172.21.15.33:0/2879362470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b3b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.116+0000 7f74da59c640 1 -- 172.21.15.33:0/2879362470 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555dd4a2d3a0 con 0x555dd4b3b780 2024-09-16T00:39:14.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.116+0000 7f74da59c640 1 --2- 172.21.15.33:0/2879362470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3b780 0x555dd4b3bbc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009d30 tx=0x7f74d00117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1c90e46e66f8b24 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:14.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.118+0000 7f74cb7fe640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74c40a3450 0x7f74c40a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.118+0000 7f74d9d9b640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74c40a3450 0x7f74c40a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.118+0000 7f74d9d9b640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74c40a3450 0x7f74c40a5910 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x555dd4b3c220 tx=0x7f74cc004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:14.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.119+0000 7f74c8ff9640 1 -- 172.21.15.33:0/2879362470 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d0012650 con 0x555dd4b3b780 2024-09-16T00:39:14.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.119+0000 7f74cb7fe640 1 -- 172.21.15.33:0/3767967667 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f74d40f0870 con 0x7f74dc10bfc0 2024-09-16T00:39:14.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.119+0000 7f74c8ff9640 1 -- 172.21.15.33:0/2879362470 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d0009e20 con 0x555dd4b3b780 2024-09-16T00:39:14.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.120+0000 7f74e304a740 1 -- 172.21.15.33:0/2879362470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3b780 msgr2=0x555dd4b3bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:14.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.120+0000 7f74e304a740 1 --2- 172.21.15.33:0/2879362470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3b780 0x555dd4b3bbc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009d30 tx=0x7f74d00117b0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.121+0000 7f74e304a740 1 -- 172.21.15.33:0/2879362470 shutdown_connections 2024-09-16T00:39:14.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.121+0000 7f74e304a740 1 --2- 172.21.15.33:0/2879362470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b48910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.121+0000 7f74e304a740 1 --2- 172.21.15.33:0/2879362470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3b780 0x555dd4b3bbc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.121+0000 7f74e304a740 1 --2- 172.21.15.33:0/2879362470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b3b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.121+0000 7f74e304a740 1 -- 172.21.15.33:0/2879362470 >> 172.21.15.33:0/2879362470 conn(0x555dd4b04310 msgr2=0x555dd4b37e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:14.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.121+0000 7f74e304a740 1 -- 172.21.15.33:0/2879362470 shutdown_connections 2024-09-16T00:39:14.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.121+0000 7f74e304a740 1 -- 172.21.15.33:0/2879362470 wait complete. 2024-09-16T00:39:14.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.121+0000 7f74e304a740 1 -- start start 2024-09-16T00:39:14.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.122+0000 7f74e304a740 1 Processor -- start 2024-09-16T00:39:14.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.122+0000 7f74e304a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3b780 0x555dd4b45e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.122+0000 7f74e304a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b430b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.122+0000 7f74e304a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b435f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.122+0000 7f74e304a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555dd4b49270 con 0x555dd4b3ccf0 2024-09-16T00:39:14.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.122+0000 7f74e304a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555dd4a5c060 con 0x555dd4b3b780 2024-09-16T00:39:14.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.122+0000 7f74e304a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555dd4b493f0 con 0x555dd4b46520 2024-09-16T00:39:14.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.122+0000 7f74e304a740 10 client.0 _ll_register_callbacks cb 0x555dd47af8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:14.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.124+0000 7f74dad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.124+0000 7f74dad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b435f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:56626/0 (socket says 172.21.15.33:56626) 2024-09-16T00:39:14.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.124+0000 7f74dad9d640 1 -- 172.21.15.33:0/1255516664 learned_addr learned my addr 172.21.15.33:0/1255516664 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:14.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.124+0000 7f74da59c640 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3b780 0x555dd4b45e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.124+0000 7f74d9d9b640 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.125+0000 7f74d9d9b640 1 -- 172.21.15.33:0/1255516664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 msgr2=0x555dd4b435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.125+0000 7f74d9d9b640 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b435f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.125+0000 7f74d9d9b640 1 -- 172.21.15.33:0/1255516664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3b780 msgr2=0x555dd4b45e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.125+0000 7f74d9d9b640 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3b780 0x555dd4b45e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.125+0000 7f74d9d9b640 1 -- 172.21.15.33:0/1255516664 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555dd48123c0 con 0x555dd4b3ccf0 2024-09-16T00:39:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.125+0000 7f74d9d9b640 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b430b0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f74dc197600 tx=0x7f74cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b70cb8706c34499 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.126+0000 7f74c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.126+0000 7f74c1ffb640 1 -- 172.21.15.33:0/1255516664 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc00fac0 con 0x555dd4b3ccf0 2024-09-16T00:39:14.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.126+0000 7f74e304a740 1 -- 172.21.15.33:0/1255516664 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74d0009e20 con 0x555dd4b3ccf0 2024-09-16T00:39:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.126+0000 7f74c1ffb640 1 -- 172.21.15.33:0/1255516664 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f74cc00fc60 con 0x555dd4b3ccf0 2024-09-16T00:39:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.126+0000 7f74c1ffb640 1 -- 172.21.15.33:0/1255516664 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f74cc030050 con 0x555dd4b3ccf0 2024-09-16T00:39:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.127+0000 7f74e304a740 10 client.19524 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.127+0000 7f74e304a740 1 -- 172.21.15.33:0/1255516664 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555dd4b54d20 con 0x555dd4b3ccf0 2024-09-16T00:39:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.133+0000 7f74c1ffb640 1 -- 172.21.15.33:0/1255516664 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d0009e20 con 0x555dd4b3ccf0 2024-09-16T00:39:14.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.133+0000 7f74c1ffb640 1 -- 172.21.15.33:0/1255516664 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f74cc00f040 con 0x555dd4b3ccf0 2024-09-16T00:39:14.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.136+0000 7f74e304a740 10 client.19524 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:14.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.136+0000 7f74e304a740 10 client.19524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:39:14.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.136+0000 7f74e304a740 20 client.19524 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:14.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.136+0000 7f74e304a740 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555dd4b56200 0x555dd4b58610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.136+0000 7f74e304a740 4 client.19524 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:39:14.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.136+0000 7f74e304a740 1 -- 172.21.15.33:0/1255516664 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555dd4aae860 con 0x555dd4b56200 2024-09-16T00:39:14.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.138+0000 7f74da59c640 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555dd4b56200 0x555dd4b58610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.142+0000 7f74da59c640 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555dd4b56200 0x555dd4b58610 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d72d1bfc48747f4 server_cookie=be5b4f463ff8ce54 in_seq=0 out_seq=0 2024-09-16T00:39:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.142+0000 7f74c1ffb640 10 client.19524 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.143+0000 7f74c1ffb640 1 -- 172.21.15.33:0/1255516664 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555dd4aae860 con 0x555dd4b56200 2024-09-16T00:39:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.143+0000 7f74c1ffb640 10 client.19524 handle_command_reply: tid=0 2024-09-16T00:39:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.144+0000 7f74e304a740 1 client.19524 shutdown 2024-09-16T00:39:14.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.149+0000 7f74e304a740 1 -- 172.21.15.33:0/1255516664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 msgr2=0x555dd4b430b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:14.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.149+0000 7f74e304a740 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b430b0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f74dc197600 tx=0x7f74cc0046c0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.153+0000 7f74e304a740 1 -- 172.21.15.33:0/1255516664 shutdown_connections 2024-09-16T00:39:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.153+0000 7f74e304a740 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555dd4b56200 0x555dd4b58610 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.153+0000 7f74e304a740 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b435f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.153+0000 7f74e304a740 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b430b0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.153+0000 7f74e304a740 1 --2- 172.21.15.33:0/1255516664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3b780 0x555dd4b45e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.153+0000 7f74e304a740 1 -- 172.21.15.33:0/1255516664 >> 172.21.15.33:0/1255516664 conn(0x555dd4b04310 msgr2=0x555dd4b3dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:14.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.153+0000 7f74c1ffb640 0 client.19524 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:14.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.155+0000 7f74e304a740 1 -- 172.21.15.33:0/1255516664 shutdown_connections 2024-09-16T00:39:14.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.155+0000 7f74e304a740 1 -- 172.21.15.33:0/1255516664 wait complete. 2024-09-16T00:39:14.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.155+0000 7f74e304a740 20 client.19524 trim_cache size 0 max 16384 2024-09-16T00:39:14.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.185+0000 7f74e304a740 1 Processor -- start 2024-09-16T00:39:14.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.185+0000 7f74e304a740 1 -- start start 2024-09-16T00:39:14.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.186+0000 7f74e304a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3b780 0x555dd4b52560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.186+0000 7f74e304a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3ccf0 0x555dd4b47910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.186+0000 7f74e304a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b47e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.186+0000 7f74e304a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1c06d0 con 0x555dd4b3b780 2024-09-16T00:39:14.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.186+0000 7f74e304a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1bdfa0 con 0x555dd4b3ccf0 2024-09-16T00:39:14.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.186+0000 7f74e304a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1bbdf0 con 0x555dd4b46520 2024-09-16T00:39:14.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.187+0000 7f74da59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3b780 0x555dd4b52560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.187+0000 7f74da59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3b780 0x555dd4b52560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55406/0 (socket says 172.21.15.33:55406) 2024-09-16T00:39:14.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.187+0000 7f74da59c640 1 -- 172.21.15.33:0/1288670777 learned_addr learned my addr 172.21.15.33:0/1288670777 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.189+0000 7f74da59c640 1 -- 172.21.15.33:0/1288670777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 msgr2=0x555dd4b47e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:39:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.189+0000 7f74da59c640 1 --2- 172.21.15.33:0/1288670777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b47e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.189+0000 7f74da59c640 1 -- 172.21.15.33:0/1288670777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3ccf0 msgr2=0x555dd4b47910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:39:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.189+0000 7f74da59c640 1 --2- 172.21.15.33:0/1288670777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3ccf0 0x555dd4b47910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.189+0000 7f74da59c640 1 -- 172.21.15.33:0/1288670777 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555dd4b331b0 con 0x555dd4b3b780 2024-09-16T00:39:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.190+0000 7f74da59c640 1 --2- 172.21.15.33:0/1288670777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3b780 0x555dd4b52560 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009d30 tx=0x7f74d0014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff553286677f466c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.191+0000 7f74c3fff640 1 -- 172.21.15.33:0/1288670777 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d0009e20 con 0x555dd4b3b780 2024-09-16T00:39:14.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.191+0000 7f74c3fff640 1 -- 172.21.15.33:0/1288670777 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d0005590 con 0x555dd4b3b780 2024-09-16T00:39:14.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.191+0000 7f74c3fff640 1 -- 172.21.15.33:0/1288670777 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d0003a80 con 0x555dd4b3b780 2024-09-16T00:39:14.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.192+0000 7f74e304a740 1 -- 172.21.15.33:0/1288670777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3b780 msgr2=0x555dd4b52560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:14.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.192+0000 7f74e304a740 1 --2- 172.21.15.33:0/1288670777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3b780 0x555dd4b52560 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009d30 tx=0x7f74d0014040 comp rx=0 tx=0).stop 2024-09-16T00:39:14.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.197+0000 7f74e304a740 1 -- 172.21.15.33:0/1288670777 shutdown_connections 2024-09-16T00:39:14.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.197+0000 7f74e304a740 1 --2- 172.21.15.33:0/1288670777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b46520 0x555dd4b47e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.197+0000 7f74e304a740 1 --2- 172.21.15.33:0/1288670777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b3ccf0 0x555dd4b47910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.197+0000 7f74e304a740 1 --2- 172.21.15.33:0/1288670777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3b780 0x555dd4b52560 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.197+0000 7f74e304a740 1 -- 172.21.15.33:0/1288670777 >> 172.21.15.33:0/1288670777 conn(0x555dd4b04310 msgr2=0x555dd4b421c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:14.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.198+0000 7f74e304a740 1 -- 172.21.15.33:0/1288670777 shutdown_connections 2024-09-16T00:39:14.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.199+0000 7f74e304a740 1 -- 172.21.15.33:0/1288670777 wait complete. 2024-09-16T00:39:14.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.200+0000 7f74e304a740 1 -- start start 2024-09-16T00:39:14.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.200+0000 7f74e304a740 1 Processor -- start 2024-09-16T00:39:14.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.201+0000 7f74e304a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b3b780 0x555dd4b40c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.201+0000 7f74e304a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b4e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.201+0000 7f74e304a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b46520 0x555dd4b4ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.201+0000 7f74da59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b3b780 0x555dd4b40c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.201+0000 7f74e304a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1bc460 con 0x555dd4b3ccf0 2024-09-16T00:39:14.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.201+0000 7f74e304a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1bd8d0 con 0x555dd4b46520 2024-09-16T00:39:14.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.201+0000 7f74e304a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1bcb30 con 0x555dd4b3b780 2024-09-16T00:39:14.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.201+0000 7f74da59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b3b780 0x555dd4b40c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:56638/0 (socket says 172.21.15.33:56638) 2024-09-16T00:39:14.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.201+0000 7f74da59c640 1 -- 172.21.15.33:0/2843974834 learned_addr learned my addr 172.21.15.33:0/2843974834 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:14.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.201+0000 7f74e304a740 10 client.0 _ll_register_callbacks cb 0x7f74cc00e700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:14.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.201+0000 7f74dad9d640 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b46520 0x555dd4b4ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.202+0000 7f74d9d9b640 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b4e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.202+0000 7f74da59c640 1 -- 172.21.15.33:0/2843974834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b46520 msgr2=0x555dd4b4ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:14.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.202+0000 7f74da59c640 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b46520 0x555dd4b4ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.202+0000 7f74da59c640 1 -- 172.21.15.33:0/2843974834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 msgr2=0x555dd4b4e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:14.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.202+0000 7f74da59c640 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b4e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.202+0000 7f74da59c640 1 -- 172.21.15.33:0/2843974834 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555dd4b5d9b0 con 0x555dd4b3b780 2024-09-16T00:39:14.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.202+0000 7f74dad9d640 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b46520 0x555dd4b4ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:39:14.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.205+0000 7f74da59c640 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b3b780 0x555dd4b40c70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f74d000fa10 tx=0x7f74d00097c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e1333a484fc058 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:14.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.207+0000 7f74c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:39:14.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.207+0000 7f74c1ffb640 1 -- 172.21.15.33:0/2843974834 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d003c060 con 0x555dd4b3b780 2024-09-16T00:39:14.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.207+0000 7f74c1ffb640 1 -- 172.21.15.33:0/2843974834 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f74d0005250 con 0x555dd4b3b780 2024-09-16T00:39:14.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.207+0000 7f74e304a740 1 -- 172.21.15.33:0/2843974834 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74d0005590 con 0x555dd4b3b780 2024-09-16T00:39:14.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.211+0000 7f74c1ffb640 1 -- 172.21.15.33:0/2843974834 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f74d0003700 con 0x555dd4b3b780 2024-09-16T00:39:14.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.211+0000 7f74e304a740 10 client.28717 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:14.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.211+0000 7f74e304a740 1 -- 172.21.15.33:0/2843974834 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555dd4bc1090 con 0x555dd4b3b780 2024-09-16T00:39:14.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.212+0000 7f74c1ffb640 1 -- 172.21.15.33:0/2843974834 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d0005590 con 0x555dd4b3b780 2024-09-16T00:39:14.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.212+0000 7f74c1ffb640 1 -- 172.21.15.33:0/2843974834 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f74d002d030 con 0x555dd4b3b780 2024-09-16T00:39:14.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.213+0000 7f74e304a740 10 client.28717 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:14.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.213+0000 7f74e304a740 10 client.28717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:39:14.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.213+0000 7f74e304a740 20 client.28717 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:14.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.213+0000 7f74e304a740 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555dd4bc1470 0x555dd4bc3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:14.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.213+0000 7f74e304a740 4 client.28717 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:39:14.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.213+0000 7f74e304a740 1 -- 172.21.15.33:0/2843974834 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555dd4aa8d70 con 0x555dd4bc1470 2024-09-16T00:39:14.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.213+0000 7f74d9d9b640 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555dd4bc1470 0x555dd4bc3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:14.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.215+0000 7f74c1ffb640 10 client.28717 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:14.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.215+0000 7f74d9d9b640 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555dd4bc1470 0x555dd4bc3860 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3756676935e59d0f server_cookie=e2c367310a4da1e7 in_seq=0 out_seq=0 2024-09-16T00:39:14.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.216+0000 7f74c1ffb640 1 -- 172.21.15.33:0/2843974834 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555dd4aa8d70 con 0x555dd4bc1470 2024-09-16T00:39:14.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.216+0000 7f74c1ffb640 10 client.28717 handle_command_reply: tid=0 2024-09-16T00:39:14.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.216+0000 7f74e304a740 1 client.28717 shutdown 2024-09-16T00:39:14.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.217+0000 7f74e304a740 1 -- 172.21.15.33:0/2843974834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b3b780 msgr2=0x555dd4b40c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:14.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.217+0000 7f74e304a740 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b3b780 0x555dd4b40c70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f74d000fa10 tx=0x7f74d00097c0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.220+0000 7f74e304a740 1 -- 172.21.15.33:0/2843974834 shutdown_connections 2024-09-16T00:39:14.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.220+0000 7f74e304a740 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555dd4bc1470 0x555dd4bc3860 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.220+0000 7f74e304a740 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555dd4b46520 0x555dd4b4ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.220+0000 7f74e304a740 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555dd4b3ccf0 0x555dd4b4e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.220+0000 7f74e304a740 1 --2- 172.21.15.33:0/2843974834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555dd4b3b780 0x555dd4b40c70 secure :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f74d000fa10 tx=0x7f74d00097c0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.220+0000 7f74e304a740 1 -- 172.21.15.33:0/2843974834 >> 172.21.15.33:0/2843974834 conn(0x555dd4b04310 msgr2=0x555dd4b44910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:14.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.220+0000 7f74c1ffb640 0 client.28717 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:14.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.220+0000 7f74e304a740 1 -- 172.21.15.33:0/2843974834 shutdown_connections 2024-09-16T00:39:14.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.220+0000 7f74e304a740 1 -- 172.21.15.33:0/2843974834 wait complete. 2024-09-16T00:39:14.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.221+0000 7f74e304a740 20 client.28717 trim_cache size 0 max 16384 2024-09-16T00:39:14.223 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:39:14.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.225+0000 7f74db59e640 1 -- 172.21.15.33:0/3767967667 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74c40a3450 msgr2=0x7f74c40a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:14.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.225+0000 7f74db59e640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74c40a3450 0x7f74c40a5910 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x555dd4b3c220 tx=0x7f74cc004730 comp rx=0 tx=0).stop 2024-09-16T00:39:14.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.225+0000 7f74db59e640 1 -- 172.21.15.33:0/3767967667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74dc10bfc0 msgr2=0x7f74dc10ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:14.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.225+0000 7f74db59e640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74dc10bfc0 0x7f74dc10ab20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f74d402af50 tx=0x7f74d4003ef0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.225+0000 7f74db59e640 1 -- 172.21.15.33:0/3767967667 shutdown_connections 2024-09-16T00:39:14.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.225+0000 7f74db59e640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f74c40a3450 0x7f74c40a5910 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.225+0000 7f74db59e640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74dc10bfc0 0x7f74dc10ab20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.225+0000 7f74db59e640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74dc1034b0 0x7f74dc19f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.225+0000 7f74db59e640 1 --2- 172.21.15.33:0/3767967667 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74dc102b90 0x7f74dc19ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:14.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.225+0000 7f74db59e640 1 -- 172.21.15.33:0/3767967667 >> 172.21.15.33:0/3767967667 conn(0x7f74dc0fc610 msgr2=0x7f74dc10d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:14.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.228+0000 7f74db59e640 1 -- 172.21.15.33:0/3767967667 shutdown_connections 2024-09-16T00:39:14.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:14.228+0000 7f74db59e640 1 -- 172.21.15.33:0/3767967667 wait complete. 2024-09-16T00:39:15.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 175320 18.58 MB/sec execute 381 sec latency 120.547 ms 2024-09-16T00:39:15.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:15 smithi033 ceph-mon[31243]: pgmap v1013: 129 pgs: 129 active+clean; 754 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 11 MiB/s wr, 482 op/s 2024-09-16T00:39:15.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:15 smithi060 ceph-mon[35185]: pgmap v1013: 129 pgs: 129 active+clean; 754 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 11 MiB/s wr, 482 op/s 2024-09-16T00:39:15.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:15 smithi083 ceph-mon[35242]: pgmap v1013: 129 pgs: 129 active+clean; 754 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 11 MiB/s wr, 482 op/s 2024-09-16T00:39:15.978 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:39:15.978 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:39:16.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 175748 18.62 MB/sec execute 382 sec latency 108.372 ms 2024-09-16T00:39:16.369 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:16.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.943+0000 7f4370e9f640 1 -- 172.21.15.33:0/2097694720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f436c10efb0 msgr2=0x7f436c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:16.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.943+0000 7f4370e9f640 1 --2- 172.21.15.33:0/2097694720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f436c10efb0 0x7f436c109450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f435c009ca0 tx=0x7f435c02a960 comp rx=0 tx=0).stop 2024-09-16T00:39:16.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.945+0000 7f4370e9f640 1 -- 172.21.15.33:0/2097694720 shutdown_connections 2024-09-16T00:39:16.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.945+0000 7f4370e9f640 1 --2- 172.21.15.33:0/2097694720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f436c109990 0x7f436c109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.945+0000 7f4370e9f640 1 --2- 172.21.15.33:0/2097694720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f436c10efb0 0x7f436c109450 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.946+0000 7f4370e9f640 1 --2- 172.21.15.33:0/2097694720 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f436c10e690 0x7f436c10ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.946+0000 7f4370e9f640 1 -- 172.21.15.33:0/2097694720 >> 172.21.15.33:0/2097694720 conn(0x7f436c06e140 msgr2=0x7f436c06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:16.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.946+0000 7f4370e9f640 1 -- 172.21.15.33:0/2097694720 shutdown_connections 2024-09-16T00:39:16.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.946+0000 7f4370e9f640 1 -- 172.21.15.33:0/2097694720 wait complete. 2024-09-16T00:39:16.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.947+0000 7f4370e9f640 1 Processor -- start 2024-09-16T00:39:16.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.947+0000 7f4370e9f640 1 -- start start 2024-09-16T00:39:16.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4370e9f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f436c109990 0x7f436c117ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:16.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4370e9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f436c10e690 0x7f436c117ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:16.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4370e9f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f436c10efb0 0x7f436c118530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:16.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4370e9f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f436c11c9d0 con 0x7f436c10e690 2024-09-16T00:39:16.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4370e9f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f436c11c850 con 0x7f436c109990 2024-09-16T00:39:16.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4370e9f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f436c11cb50 con 0x7f436c10efb0 2024-09-16T00:39:16.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4369d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f436c10e690 0x7f436c117ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:16.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4369d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f436c10e690 0x7f436c117ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55422/0 (socket says 172.21.15.33:55422) 2024-09-16T00:39:16.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4369d9b640 1 -- 172.21.15.33:0/941328369 learned_addr learned my addr 172.21.15.33:0/941328369 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:16.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f436a59c640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f436c109990 0x7f436c117ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:16.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f436ad9d640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f436c10efb0 0x7f436c118530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:16.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4369d9b640 1 -- 172.21.15.33:0/941328369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f436c10efb0 msgr2=0x7f436c118530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:16.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4369d9b640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f436c10efb0 0x7f436c118530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4369d9b640 1 -- 172.21.15.33:0/941328369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f436c109990 msgr2=0x7f436c117ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:16.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.948+0000 7f4369d9b640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f436c109990 0x7f436c117ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.949+0000 7f4369d9b640 1 -- 172.21.15.33:0/941328369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436c1b33f0 con 0x7f436c10e690 2024-09-16T00:39:16.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.949+0000 7f4369d9b640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f436c10e690 0x7f436c117ff0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f435c006010 tx=0x7f435c02ff80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:16.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.950+0000 7f43637fe640 1 -- 172.21.15.33:0/941328369 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c034070 con 0x7f436c10e690 2024-09-16T00:39:16.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.950+0000 7f43637fe640 1 -- 172.21.15.33:0/941328369 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435c043440 con 0x7f436c10e690 2024-09-16T00:39:16.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.950+0000 7f43637fe640 1 -- 172.21.15.33:0/941328369 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c039680 con 0x7f436c10e690 2024-09-16T00:39:16.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.950+0000 7f4370e9f640 1 -- 172.21.15.33:0/941328369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436c1b3680 con 0x7f436c10e690 2024-09-16T00:39:16.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.950+0000 7f4370e9f640 1 -- 172.21.15.33:0/941328369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f436c1b3b10 con 0x7f436c10e690 2024-09-16T00:39:16.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.954+0000 7f43637fe640 1 -- 172.21.15.33:0/941328369 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f435c037020 con 0x7f436c10e690 2024-09-16T00:39:16.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.954+0000 7f4373278740 1 Processor -- start 2024-09-16T00:39:16.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.955+0000 7f4373278740 1 -- start start 2024-09-16T00:39:16.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.956+0000 7f4373278740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 0x558a106d5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.956+0000 7f4373278740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d6480 0x558a106e0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.956+0000 7f4373278740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106e3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.956+0000 7f4373278740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558a1063b280 con 0x558a106d6480 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.956+0000 7f43637fe640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f43580a3450 0x7f43580a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.956+0000 7f4373278740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558a105e2b10 con 0x558a106e0d40 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.956+0000 7f4373278740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558a10637240 con 0x558a106d5b60 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.956+0000 7f43637fe640 1 -- 172.21.15.33:0/941328369 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f435c0f0460 con 0x7f436c10e690 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.956+0000 7f436ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 0x558a106d5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.956+0000 7f436ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 0x558a106d5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:56664/0 (socket says 172.21.15.33:56664) 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.956+0000 7f436ad9d640 1 -- 172.21.15.33:0/2442084456 learned_addr learned my addr 172.21.15.33:0/2442084456 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.956+0000 7f436ad9d640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f43580a3450 0x7f43580a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.957+0000 7f436a59c640 1 --2- 172.21.15.33:0/2442084456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d6480 0x558a106e0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.957+0000 7f4369d9b640 1 --2- 172.21.15.33:0/2442084456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106e3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:16.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.957+0000 7f436ad9d640 1 -- 172.21.15.33:0/2442084456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 msgr2=0x558a106e3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:16.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.957+0000 7f436ad9d640 1 --2- 172.21.15.33:0/2442084456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106e3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.957+0000 7f436ad9d640 1 -- 172.21.15.33:0/2442084456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d6480 msgr2=0x558a106e0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:16.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.957+0000 7f436ad9d640 1 --2- 172.21.15.33:0/2442084456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d6480 0x558a106e0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.957+0000 7f436ad9d640 1 -- 172.21.15.33:0/2442084456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a105ef1f0 con 0x558a106d5b60 2024-09-16T00:39:16.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.957+0000 7f4369d9b640 1 --2- 172.21.15.33:0/2442084456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106e3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:39:16.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.957+0000 7f436ad9d640 1 --2- 172.21.15.33:0/2442084456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 0x558a106d5f40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f435400a5c0 tx=0x7f435401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=748f922d294f07d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:16.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.958+0000 7f436ad9d640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f43580a3450 0x7f43580a5910 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f4354011930 tx=0x7f435400a7f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:16.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.961+0000 7f4360ff9640 1 -- 172.21.15.33:0/2442084456 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354023070 con 0x558a106d5b60 2024-09-16T00:39:16.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.961+0000 7f4360ff9640 1 -- 172.21.15.33:0/2442084456 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435400fe50 con 0x558a106d5b60 2024-09-16T00:39:16.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.961+0000 7f4360ff9640 1 -- 172.21.15.33:0/2442084456 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354005280 con 0x558a106d5b60 2024-09-16T00:39:16.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.962+0000 7f4373278740 1 -- 172.21.15.33:0/2442084456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 msgr2=0x558a106d5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:16.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.962+0000 7f4373278740 1 --2- 172.21.15.33:0/2442084456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 0x558a106d5f40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f435400a5c0 tx=0x7f435401b000 comp rx=0 tx=0).stop 2024-09-16T00:39:16.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.963+0000 7f4373278740 1 -- 172.21.15.33:0/2442084456 shutdown_connections 2024-09-16T00:39:16.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.963+0000 7f4373278740 1 --2- 172.21.15.33:0/2442084456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106e3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.963+0000 7f4373278740 1 --2- 172.21.15.33:0/2442084456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d6480 0x558a106e0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.963+0000 7f4373278740 1 --2- 172.21.15.33:0/2442084456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 0x558a106d5f40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.963+0000 7f4373278740 1 -- 172.21.15.33:0/2442084456 >> 172.21.15.33:0/2442084456 conn(0x558a106d20f0 msgr2=0x558a106d24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:16.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.963+0000 7f4373278740 1 -- 172.21.15.33:0/2442084456 shutdown_connections 2024-09-16T00:39:16.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.963+0000 7f4373278740 1 -- 172.21.15.33:0/2442084456 wait complete. 2024-09-16T00:39:16.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.963+0000 7f4373278740 1 -- start start 2024-09-16T00:39:16.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.963+0000 7f4373278740 1 Processor -- start 2024-09-16T00:39:16.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.964+0000 7f4373278740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106d5b60 0x558a106ddc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:16.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.964+0000 7f4373278740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d6480 0x558a106de1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:16.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.964+0000 7f4373278740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106e0d40 0x558a106de700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:16.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.964+0000 7f436a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106d5b60 0x558a106ddc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:16.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.964+0000 7f436a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106d5b60 0x558a106ddc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55454/0 (socket says 172.21.15.33:55454) 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.964+0000 7f436a59c640 1 -- 172.21.15.33:0/4062403207 learned_addr learned my addr 172.21.15.33:0/4062403207 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.964+0000 7f4373278740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558a106e3910 con 0x558a106e0d40 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.964+0000 7f4373278740 1 -- 172.21.15.33:0/4062403207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558a105f5f10 con 0x558a106d5b60 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.964+0000 7f4373278740 1 -- 172.21.15.33:0/4062403207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558a106e3a90 con 0x558a106d6480 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.964+0000 7f4369d9b640 1 --2- 172.21.15.33:0/4062403207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d6480 0x558a106de1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.965+0000 7f4373278740 10 client.0 _ll_register_callbacks cb 0x558a105bea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.965+0000 7f436a59c640 1 -- 172.21.15.33:0/4062403207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d6480 msgr2=0x558a106de1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.965+0000 7f436a59c640 1 --2- 172.21.15.33:0/4062403207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d6480 0x558a106de1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.965+0000 7f436a59c640 1 -- 172.21.15.33:0/4062403207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106e0d40 msgr2=0x558a106de700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.965+0000 7f436a59c640 1 --2- 172.21.15.33:0/4062403207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106e0d40 0x558a106de700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.965+0000 7f436a59c640 1 -- 172.21.15.33:0/4062403207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a103ac6a0 con 0x558a106d5b60 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.965+0000 7f4351ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:39:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.965+0000 7f436a59c640 1 --2- 172.21.15.33:0/4062403207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106d5b60 0x558a106ddc80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4364012700 tx=0x7f4364012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd46cb5bf089abef server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:16.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.965+0000 7f4351ffb640 1 -- 172.21.15.33:0/4062403207 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364004050 con 0x558a106d5b60 2024-09-16T00:39:16.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.965+0000 7f4373278740 1 -- 172.21.15.33:0/4062403207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f435400fe50 con 0x558a106d5b60 2024-09-16T00:39:16.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.965+0000 7f4351ffb640 1 -- 172.21.15.33:0/4062403207 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4364018030 con 0x558a106d5b60 2024-09-16T00:39:16.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.966+0000 7f4351ffb640 1 -- 172.21.15.33:0/4062403207 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4364004050 con 0x558a106d5b60 2024-09-16T00:39:16.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.966+0000 7f4373278740 10 client.28718 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:16.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.966+0000 7f4373278740 1 -- 172.21.15.33:0/4062403207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558a106ef550 con 0x558a106d5b60 2024-09-16T00:39:16.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.967+0000 7f4351ffb640 1 -- 172.21.15.33:0/4062403207 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435400fe50 con 0x558a106d5b60 2024-09-16T00:39:16.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.967+0000 7f4351ffb640 1 -- 172.21.15.33:0/4062403207 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f43640038d0 con 0x558a106d5b60 2024-09-16T00:39:16.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.967+0000 7f4373278740 10 client.28718 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:16.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.967+0000 7f4373278740 10 client.28718 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:39:16.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.967+0000 7f4373278740 20 client.28718 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:16.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.968+0000 7f4373278740 1 --2- 172.21.15.33:0/4062403207 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a106f0960 0x558a106f2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:16.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.968+0000 7f4373278740 4 client.28718 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:39:16.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.968+0000 7f4373278740 1 -- 172.21.15.33:0/4062403207 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a106487f0 con 0x558a106f0960 2024-09-16T00:39:16.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.969+0000 7f4369d9b640 1 --2- 172.21.15.33:0/4062403207 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a106f0960 0x558a106f2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:16.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.969+0000 7f4369d9b640 1 --2- 172.21.15.33:0/4062403207 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a106f0960 0x558a106f2dd0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a410dd3e0e1d7dd server_cookie=24488e301dbc5264 in_seq=0 out_seq=0 2024-09-16T00:39:16.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.970+0000 7f4351ffb640 10 client.28718 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:16.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.970+0000 7f4351ffb640 1 -- 172.21.15.33:0/4062403207 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558a106487f0 con 0x558a106f0960 2024-09-16T00:39:16.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.970+0000 7f4351ffb640 10 client.28718 handle_command_reply: tid=0 2024-09-16T00:39:16.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.971+0000 7f4373278740 1 client.28718 shutdown 2024-09-16T00:39:16.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.972+0000 7f4373278740 1 -- 172.21.15.33:0/4062403207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106d5b60 msgr2=0x558a106ddc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:16.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.972+0000 7f4373278740 1 --2- 172.21.15.33:0/4062403207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106d5b60 0x558a106ddc80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4364012700 tx=0x7f4364012bc0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.972+0000 7f4373278740 1 -- 172.21.15.33:0/4062403207 shutdown_connections 2024-09-16T00:39:16.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.972+0000 7f4373278740 1 --2- 172.21.15.33:0/4062403207 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a106f0960 0x558a106f2dd0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.972+0000 7f4373278740 1 --2- 172.21.15.33:0/4062403207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106e0d40 0x558a106de700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.972+0000 7f4373278740 1 --2- 172.21.15.33:0/4062403207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d6480 0x558a106de1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.972+0000 7f4373278740 1 --2- 172.21.15.33:0/4062403207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106d5b60 0x558a106ddc80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:16.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.972+0000 7f4373278740 1 -- 172.21.15.33:0/4062403207 >> 172.21.15.33:0/4062403207 conn(0x558a106d20f0 msgr2=0x558a106d7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:16.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.972+0000 7f4351ffb640 0 client.28718 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:16.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.973+0000 7f4373278740 1 -- 172.21.15.33:0/4062403207 shutdown_connections 2024-09-16T00:39:16.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.973+0000 7f4373278740 1 -- 172.21.15.33:0/4062403207 wait complete. 2024-09-16T00:39:16.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:16.973+0000 7f4373278740 20 client.28718 trim_cache size 0 max 16384 2024-09-16T00:39:17.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.004+0000 7f4373278740 1 Processor -- start 2024-09-16T00:39:17.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.005+0000 7f4373278740 1 -- start start 2024-09-16T00:39:17.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.005+0000 7f4373278740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d5b60 0x558a106dfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:17.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.006+0000 7f4373278740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d6480 0x558a106f86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:17.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.006+0000 7f4373278740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106f8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:17.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.006+0000 7f4369d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d6480 0x558a106f86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:17.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.006+0000 7f4369d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d6480 0x558a106f86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:56686/0 (socket says 172.21.15.33:56686) 2024-09-16T00:39:17.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.006+0000 7f4369d9b640 1 -- 172.21.15.33:0/2842089005 learned_addr learned my addr 172.21.15.33:0/2842089005 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:17.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.007+0000 7f4373278740 1 -- 172.21.15.33:0/2842089005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f436c1c0690 con 0x558a106d5b60 2024-09-16T00:39:17.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.007+0000 7f436ad9d640 1 --2- 172.21.15.33:0/2842089005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106f8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:17.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.007+0000 7f4373278740 1 -- 172.21.15.33:0/2842089005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f436c1bdf60 con 0x558a106e0d40 2024-09-16T00:39:17.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.007+0000 7f4373278740 1 -- 172.21.15.33:0/2842089005 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f436c1bbdb0 con 0x558a106d6480 2024-09-16T00:39:17.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.007+0000 7f436a59c640 1 --2- 172.21.15.33:0/2842089005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d5b60 0x558a106dfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:17.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.008+0000 7f436a59c640 1 -- 172.21.15.33:0/2842089005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d6480 msgr2=0x558a106f86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:17.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.008+0000 7f436a59c640 1 --2- 172.21.15.33:0/2842089005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d6480 0x558a106f86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.008+0000 7f436a59c640 1 -- 172.21.15.33:0/2842089005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 msgr2=0x558a106f8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:17.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.008+0000 7f436a59c640 1 --2- 172.21.15.33:0/2842089005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106f8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.008+0000 7f436a59c640 1 -- 172.21.15.33:0/2842089005 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a106d3260 con 0x558a106d5b60 2024-09-16T00:39:17.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.008+0000 7f4369d9b640 1 --2- 172.21.15.33:0/2842089005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d6480 0x558a106f86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:39:17.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.008+0000 7f436ad9d640 1 --2- 172.21.15.33:0/2842089005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106f8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:39:17.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.008+0000 7f436a59c640 1 --2- 172.21.15.33:0/2842089005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d5b60 0x558a106dfae0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f4364003e60 tx=0x7f4364009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163042529be3bbc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:17.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.009+0000 7f4353fff640 1 -- 172.21.15.33:0/2842089005 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364010430 con 0x558a106d5b60 2024-09-16T00:39:17.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.009+0000 7f4353fff640 1 -- 172.21.15.33:0/2842089005 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43640035f0 con 0x558a106d5b60 2024-09-16T00:39:17.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.009+0000 7f4353fff640 1 -- 172.21.15.33:0/2842089005 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436400fc10 con 0x558a106d5b60 2024-09-16T00:39:17.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.010+0000 7f4373278740 1 -- 172.21.15.33:0/2842089005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d5b60 msgr2=0x558a106dfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:17.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.010+0000 7f4373278740 1 --2- 172.21.15.33:0/2842089005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d5b60 0x558a106dfae0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f4364003e60 tx=0x7f4364009640 comp rx=0 tx=0).stop 2024-09-16T00:39:17.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.014+0000 7f4373278740 1 -- 172.21.15.33:0/2842089005 shutdown_connections 2024-09-16T00:39:17.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.014+0000 7f4373278740 1 --2- 172.21.15.33:0/2842089005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106f8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.014+0000 7f4373278740 1 --2- 172.21.15.33:0/2842089005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d6480 0x558a106f86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.014+0000 7f4373278740 1 --2- 172.21.15.33:0/2842089005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d5b60 0x558a106dfae0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.014+0000 7f4373278740 1 -- 172.21.15.33:0/2842089005 >> 172.21.15.33:0/2842089005 conn(0x558a106d20f0 msgr2=0x558a106df3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:17.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.015+0000 7f4373278740 1 -- 172.21.15.33:0/2842089005 shutdown_connections 2024-09-16T00:39:17.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.015+0000 7f4373278740 1 -- 172.21.15.33:0/2842089005 wait complete. 2024-09-16T00:39:17.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.016+0000 7f4373278740 1 -- start start 2024-09-16T00:39:17.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.016+0000 7f4373278740 1 Processor -- start 2024-09-16T00:39:17.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.017+0000 7f4373278740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 0x558a106e8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:17.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.017+0000 7f4373278740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d6480 0x558a106e9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:17.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.017+0000 7f436a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 0x558a106e8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:17.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.017+0000 7f436a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 0x558a106e8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:56702/0 (socket says 172.21.15.33:56702) 2024-09-16T00:39:17.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.017+0000 7f436a59c640 1 -- 172.21.15.33:0/135657475 learned_addr learned my addr 172.21.15.33:0/135657475 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:17.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.018+0000 7f4373278740 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106e5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:17.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.018+0000 7f4373278740 1 -- 172.21.15.33:0/135657475 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f436c1bc420 con 0x558a106d6480 2024-09-16T00:39:17.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.018+0000 7f436ad9d640 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106e5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:17.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.018+0000 7f4373278740 1 -- 172.21.15.33:0/135657475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f436c1bd890 con 0x558a106e0d40 2024-09-16T00:39:17.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.018+0000 7f4373278740 1 -- 172.21.15.33:0/135657475 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f436c1bcaf0 con 0x558a106d5b60 2024-09-16T00:39:17.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.018+0000 7f4373278740 10 client.0 _ll_register_callbacks cb 0x7f43640097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:17.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.018+0000 7f436ad9d640 1 -- 172.21.15.33:0/135657475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 msgr2=0x558a106e8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:17.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.018+0000 7f436ad9d640 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 0x558a106e8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.018+0000 7f436ad9d640 1 -- 172.21.15.33:0/135657475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d6480 msgr2=0x558a106e9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:17.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.018+0000 7f4369d9b640 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d6480 0x558a106e9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:17.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.019+0000 7f436ad9d640 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d6480 0x558a106e9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.019+0000 7f436ad9d640 1 -- 172.21.15.33:0/135657475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a106e6220 con 0x558a106e0d40 2024-09-16T00:39:17.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.019+0000 7f4369d9b640 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d6480 0x558a106e9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:39:17.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.019+0000 7f436ad9d640 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106e5ce0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f435400d710 tx=0x7f435400def0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5588c0abb44e363 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:17.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.019+0000 7f4351ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:39:17.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.019+0000 7f4351ffb640 1 -- 172.21.15.33:0/135657475 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354023050 con 0x558a106e0d40 2024-09-16T00:39:17.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.020+0000 7f4351ffb640 1 -- 172.21.15.33:0/135657475 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4354005450 con 0x558a106e0d40 2024-09-16T00:39:17.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.020+0000 7f4373278740 1 -- 172.21.15.33:0/135657475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43640035f0 con 0x558a106e0d40 2024-09-16T00:39:17.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.022+0000 7f4351ffb640 1 -- 172.21.15.33:0/135657475 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4354005650 con 0x558a106e0d40 2024-09-16T00:39:17.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.025+0000 7f4373278740 10 client.28721 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:17.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.025+0000 7f4373278740 1 -- 172.21.15.33:0/135657475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558a1075bd90 con 0x558a106e0d40 2024-09-16T00:39:17.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.028+0000 7f4351ffb640 1 -- 172.21.15.33:0/135657475 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435400fe50 con 0x558a106e0d40 2024-09-16T00:39:17.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.028+0000 7f4351ffb640 1 -- 172.21.15.33:0/135657475 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f435401e070 con 0x558a106e0d40 2024-09-16T00:39:17.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.028+0000 7f4373278740 10 client.28721 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:17.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.028+0000 7f4373278740 10 client.28721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:39:17.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.028+0000 7f4373278740 20 client.28721 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:17.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.028+0000 7f4373278740 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a1075c320 0x558a1075e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:17.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.029+0000 7f4373278740 4 client.28721 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:39:17.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.029+0000 7f4373278740 1 -- 172.21.15.33:0/135657475 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f43640035f0 con 0x558a1075c320 2024-09-16T00:39:17.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.032+0000 7f436a59c640 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a1075c320 0x558a1075e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:17.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.038+0000 7f436a59c640 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a1075c320 0x558a1075e740 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54768b31937fee0b server_cookie=217aeee715c9e96a in_seq=0 out_seq=0 2024-09-16T00:39:17.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.038+0000 7f4351ffb640 10 client.28721 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:17.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.039+0000 7f4351ffb640 1 -- 172.21.15.33:0/135657475 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f43640035f0 con 0x558a1075c320 2024-09-16T00:39:17.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.039+0000 7f4351ffb640 10 client.28721 handle_command_reply: tid=0 2024-09-16T00:39:17.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.039+0000 7f4373278740 1 client.28721 shutdown 2024-09-16T00:39:17.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.039+0000 7f4373278740 1 -- 172.21.15.33:0/135657475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 msgr2=0x558a106e5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:17.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.039+0000 7f4373278740 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106e5ce0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f435400d710 tx=0x7f435400def0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.039+0000 7f4373278740 1 -- 172.21.15.33:0/135657475 shutdown_connections 2024-09-16T00:39:17.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.039+0000 7f4373278740 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558a1075c320 0x558a1075e740 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.039+0000 7f4351ffb640 0 client.28721 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:17.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.040+0000 7f4373278740 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a106e0d40 0x558a106e5ce0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.040+0000 7f4373278740 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558a106d6480 0x558a106e9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.040+0000 7f4373278740 1 --2- 172.21.15.33:0/135657475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558a106d5b60 0x558a106e8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.040+0000 7f4373278740 1 -- 172.21.15.33:0/135657475 >> 172.21.15.33:0/135657475 conn(0x558a106d20f0 msgr2=0x558a106de460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:17.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.040+0000 7f4373278740 1 -- 172.21.15.33:0/135657475 shutdown_connections 2024-09-16T00:39:17.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.040+0000 7f4373278740 1 -- 172.21.15.33:0/135657475 wait complete. 2024-09-16T00:39:17.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.041+0000 7f4373278740 20 client.28721 trim_cache size 0 max 16384 2024-09-16T00:39:17.043 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:39:17.043 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:39:17.043 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "ee9d3b85-8015-4981-a3bb-767aab4fa9c1", 2024-09-16T00:39:17.043 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:39:17.043 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:39:17.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.041+0000 7f4370e9f640 1 -- 172.21.15.33:0/941328369 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f43580a3450 msgr2=0x7f43580a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:17.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.042+0000 7f4370e9f640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f43580a3450 0x7f43580a5910 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f4354011930 tx=0x7f435400a7f0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.042+0000 7f4370e9f640 1 -- 172.21.15.33:0/941328369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f436c10e690 msgr2=0x7f436c117ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:17.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.042+0000 7f4370e9f640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f436c10e690 0x7f436c117ff0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f435c006010 tx=0x7f435c02ff80 comp rx=0 tx=0).stop 2024-09-16T00:39:17.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.042+0000 7f4370e9f640 1 -- 172.21.15.33:0/941328369 shutdown_connections 2024-09-16T00:39:17.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.042+0000 7f4370e9f640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f43580a3450 0x7f43580a5910 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.042+0000 7f4370e9f640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f436c10efb0 0x7f436c118530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.042+0000 7f4370e9f640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f436c10e690 0x7f436c117ff0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.042+0000 7f4370e9f640 1 --2- 172.21.15.33:0/941328369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f436c109990 0x7f436c117ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:17.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.042+0000 7f4370e9f640 1 -- 172.21.15.33:0/941328369 >> 172.21.15.33:0/941328369 conn(0x7f436c06e140 msgr2=0x7f436c10c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:17.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.042+0000 7f4370e9f640 1 -- 172.21.15.33:0/941328369 shutdown_connections 2024-09-16T00:39:17.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:17.042+0000 7f4370e9f640 1 -- 172.21.15.33:0/941328369 wait complete. 2024-09-16T00:39:17.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 176089 18.62 MB/sec execute 383 sec latency 109.184 ms 2024-09-16T00:39:17.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:17 smithi033 ceph-mon[31243]: pgmap v1014: 129 pgs: 129 active+clean; 743 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 16 MiB/s wr, 659 op/s 2024-09-16T00:39:17.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:17 smithi060 ceph-mon[35185]: pgmap v1014: 129 pgs: 129 active+clean; 743 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 16 MiB/s wr, 659 op/s 2024-09-16T00:39:17.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:17 smithi083 ceph-mon[35242]: pgmap v1014: 129 pgs: 129 active+clean; 743 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 16 MiB/s wr, 659 op/s 2024-09-16T00:39:17.831 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:39:18.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 176410 18.60 MB/sec execute 384 sec latency 189.960 ms 2024-09-16T00:39:18.214 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:18.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:39:18.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:39:18.481 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.482 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:39:18.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:39:18.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:39:18.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:39:18.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:39:18.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:39:18.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:39:18.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:39:18.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:39:18.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:39:18.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:39:18.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:39:18.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:39:18.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:39:18.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:39:18.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:39:18.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.669 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:39:18.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:39:18.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.670 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:18.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.780+0000 7fa2adde7640 1 -- 172.21.15.33:0/1892531257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2a8102d40 msgr2=0x7fa2a8103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:18.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.780+0000 7fa2adde7640 1 --2- 172.21.15.33:0/1892531257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2a8102d40 0x7fa2a8103120 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fa298009900 tx=0x7fa29802a480 comp rx=0 tx=0).stop 2024-09-16T00:39:18.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.780+0000 7fa2adde7640 1 -- 172.21.15.33:0/1892531257 shutdown_connections 2024-09-16T00:39:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.780+0000 7fa2adde7640 1 --2- 172.21.15.33:0/1892531257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2a810c170 0x7fa2a810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.780+0000 7fa2adde7640 1 --2- 172.21.15.33:0/1892531257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2a8103660 0x7fa2a810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.780+0000 7fa2adde7640 1 --2- 172.21.15.33:0/1892531257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2a8102d40 0x7fa2a8103120 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.780+0000 7fa2adde7640 1 -- 172.21.15.33:0/1892531257 >> 172.21.15.33:0/1892531257 conn(0x7fa2a80fc7a0 msgr2=0x7fa2a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.780+0000 7fa2adde7640 1 -- 172.21.15.33:0/1892531257 shutdown_connections 2024-09-16T00:39:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.780+0000 7fa2adde7640 1 -- 172.21.15.33:0/1892531257 wait complete. 2024-09-16T00:39:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.781+0000 7fa2adde7640 1 Processor -- start 2024-09-16T00:39:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.781+0000 7fa2adde7640 1 -- start start 2024-09-16T00:39:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.781+0000 7fa2adde7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2a8102d40 0x7fa2a8198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.781+0000 7fa2adde7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2a8103660 0x7fa2a8198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.781+0000 7fa2adde7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2a810c170 0x7fa2a819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2adde7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a8111f70 con 0x7fa2a8103660 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2adde7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a8111df0 con 0x7fa2a8102d40 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2adde7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a81120f0 con 0x7fa2a810c170 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2a8102d40 0x7fa2a8198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2a8103660 0x7fa2a8198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2a8102d40 0x7fa2a8198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55486/0 (socket says 172.21.15.33:55486) 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2a8103660 0x7fa2a8198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55484/0 (socket says 172.21.15.33:55484) 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a77fe640 1 -- 172.21.15.33:0/1672244198 learned_addr learned my addr 172.21.15.33:0/1672244198 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a7fff640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2a810c170 0x7fa2a819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a77fe640 1 -- 172.21.15.33:0/1672244198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2a810c170 msgr2=0x7fa2a819cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a77fe640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2a810c170 0x7fa2a819cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a77fe640 1 -- 172.21.15.33:0/1672244198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2a8103660 msgr2=0x7fa2a8198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a77fe640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2a8103660 0x7fa2a8198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a77fe640 1 -- 172.21.15.33:0/1672244198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2a819d3e0 con 0x7fa2a8102d40 2024-09-16T00:39:18.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a6ffd640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2a8103660 0x7fa2a8198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:39:18.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a7fff640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2a810c170 0x7fa2a819cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:39:18.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a77fe640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2a8102d40 0x7fa2a8198430 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa298006010 tx=0x7fa2980042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:18.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.782+0000 7fa2a4ff9640 1 -- 172.21.15.33:0/1672244198 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29803f030 con 0x7fa2a8102d40 2024-09-16T00:39:18.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.783+0000 7fa2adde7640 1 -- 172.21.15.33:0/1672244198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2a819d670 con 0x7fa2a8102d40 2024-09-16T00:39:18.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.783+0000 7fa2a4ff9640 1 -- 172.21.15.33:0/1672244198 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa298003ce0 con 0x7fa2a8102d40 2024-09-16T00:39:18.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.783+0000 7fa2a4ff9640 1 -- 172.21.15.33:0/1672244198 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa298035070 con 0x7fa2a8102d40 2024-09-16T00:39:18.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.783+0000 7fa2adde7640 1 -- 172.21.15.33:0/1672244198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2a81a4fb0 con 0x7fa2a8102d40 2024-09-16T00:39:18.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.785+0000 7fa2a4ff9640 1 -- 172.21.15.33:0/1672244198 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa298042030 con 0x7fa2a8102d40 2024-09-16T00:39:18.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.785+0000 7fa2b01c0740 1 Processor -- start 2024-09-16T00:39:18.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.785+0000 7fa2b01c0740 1 -- start start 2024-09-16T00:39:18.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2b01c0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75ac4a0 0x561de75ac880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2b01c0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75acdc0 0x561de75b7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2b01c0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75b7680 0x561de75b9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2b01c0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561de75122e0 con 0x561de75acdc0 2024-09-16T00:39:18.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2b01c0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561de74b9ce0 con 0x561de75b7680 2024-09-16T00:39:18.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2b01c0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561de750e2a0 con 0x561de75ac4a0 2024-09-16T00:39:18.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75acdc0 0x561de75b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a4ff9640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa2900a32c0 0x7fa2900a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75acdc0 0x561de75b7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55486/0 (socket says 172.21.15.33:55486) 2024-09-16T00:39:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a77fe640 1 -- 172.21.15.33:0/3724594832 learned_addr learned my addr 172.21.15.33:0/3724594832 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a6ffd640 1 --2- 172.21.15.33:0/3724594832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75b7680 0x561de75b9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a7fff640 1 --2- 172.21.15.33:0/3724594832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75ac4a0 0x561de75ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a7fff640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa2900a32c0 0x7fa2900a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a4ff9640 1 -- 172.21.15.33:0/1672244198 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa2980efee0 con 0x7fa2a8102d40 2024-09-16T00:39:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a6ffd640 1 -- 172.21.15.33:0/3724594832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75ac4a0 msgr2=0x561de75ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a6ffd640 1 --2- 172.21.15.33:0/3724594832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75ac4a0 0x561de75ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a6ffd640 1 -- 172.21.15.33:0/3724594832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75acdc0 msgr2=0x561de75b7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a6ffd640 1 --2- 172.21.15.33:0/3724594832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75acdc0 0x561de75b7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a6ffd640 1 -- 172.21.15.33:0/3724594832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561de749e440 con 0x561de75b7680 2024-09-16T00:39:18.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a7fff640 1 --2- 172.21.15.33:0/3724594832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75ac4a0 0x561de75ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:39:18.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.786+0000 7fa2a77fe640 1 --2- 172.21.15.33:0/3724594832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75acdc0 0x561de75b7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:39:18.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.787+0000 7fa2a6ffd640 1 --2- 172.21.15.33:0/3724594832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75b7680 0x561de75b9a70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00bfa0 tx=0x7fa29c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5c346a9979e84d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:18.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.787+0000 7fa28dffb640 1 -- 172.21.15.33:0/3724594832 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c011700 con 0x561de75b7680 2024-09-16T00:39:18.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.787+0000 7fa28dffb640 1 -- 172.21.15.33:0/3724594832 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa29c011d80 con 0x561de75b7680 2024-09-16T00:39:18.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.787+0000 7fa28dffb640 1 -- 172.21.15.33:0/3724594832 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c01ce60 con 0x561de75b7680 2024-09-16T00:39:18.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.787+0000 7fa2b01c0740 1 -- 172.21.15.33:0/3724594832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75b7680 msgr2=0x561de75b9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:18.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.787+0000 7fa2a7fff640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa2900a32c0 0x7fa2900a5780 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0005c40 tx=0x7fa2a0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:18.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.787+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/3724594832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75b7680 0x561de75b9a70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00bfa0 tx=0x7fa29c0107a0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.788+0000 7fa2b01c0740 1 -- 172.21.15.33:0/3724594832 shutdown_connections 2024-09-16T00:39:18.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.788+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/3724594832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75b7680 0x561de75b9a70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.788+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/3724594832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75acdc0 0x561de75b7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.788+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/3724594832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75ac4a0 0x561de75ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.788+0000 7fa2b01c0740 1 -- 172.21.15.33:0/3724594832 >> 172.21.15.33:0/3724594832 conn(0x561de75a8a30 msgr2=0x561de75a8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:18.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.788+0000 7fa2b01c0740 1 -- 172.21.15.33:0/3724594832 shutdown_connections 2024-09-16T00:39:18.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.788+0000 7fa2b01c0740 1 -- 172.21.15.33:0/3724594832 wait complete. 2024-09-16T00:39:18.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.788+0000 7fa2b01c0740 1 -- start start 2024-09-16T00:39:18.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.788+0000 7fa2b01c0740 1 Processor -- start 2024-09-16T00:39:18.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2b01c0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75b6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2b01c0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75acdc0 0x561de75b4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2b01c0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75b7680 0x561de75b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2b01c0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561de75ba250 con 0x561de75ac4a0 2024-09-16T00:39:18.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2b01c0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561de74cd050 con 0x561de75acdc0 2024-09-16T00:39:18.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2b01c0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561de75ba3d0 con 0x561de75b7680 2024-09-16T00:39:18.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75b6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2b01c0740 10 client.0 _ll_register_callbacks cb 0x561de750d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:18.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75b6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55496/0 (socket says 172.21.15.33:55496) 2024-09-16T00:39:18.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2a77fe640 1 -- 172.21.15.33:0/2269663369 learned_addr learned my addr 172.21.15.33:0/2269663369 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:18.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2a6ffd640 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75acdc0 0x561de75b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2a7fff640 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75b7680 0x561de75b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2a77fe640 1 -- 172.21.15.33:0/2269663369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75b7680 msgr2=0x561de75b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:18.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2a77fe640 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75b7680 0x561de75b47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2a77fe640 1 -- 172.21.15.33:0/2269663369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75acdc0 msgr2=0x561de75b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:18.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2a77fe640 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75acdc0 0x561de75b4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.789+0000 7fa2a77fe640 1 -- 172.21.15.33:0/2269663369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561de72833c0 con 0x561de75ac4a0 2024-09-16T00:39:18.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa2a77fe640 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75b6fd0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fa2980bc070 tx=0x7fa298003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718f6eeeb93bd92b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa286ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa286ffd640 1 -- 172.21.15.33:0/2269663369 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa298049030 con 0x561de75ac4a0 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa286ffd640 1 -- 172.21.15.33:0/2269663369 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa2980ed610 con 0x561de75ac4a0 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2269663369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa29c011d80 con 0x561de75ac4a0 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa286ffd640 1 -- 172.21.15.33:0/2269663369 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa2980bbc20 con 0x561de75ac4a0 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa2b01c0740 10 client.19554 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2269663369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561de75c5c20 con 0x561de75ac4a0 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa286ffd640 1 -- 172.21.15.33:0/2269663369 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa29c011d80 con 0x561de75ac4a0 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa286ffd640 1 -- 172.21.15.33:0/2269663369 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fa29804c060 con 0x561de75ac4a0 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa2b01c0740 10 client.19554 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa2b01c0740 10 client.19554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa2b01c0740 20 client.19554 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:18.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.790+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561de75c7030 0x561de75c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.791+0000 7fa2b01c0740 4 client.19554 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:39:18.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.791+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2269663369 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561de751f850 con 0x561de75c7030 2024-09-16T00:39:18.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.791+0000 7fa2a6ffd640 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561de75c7030 0x561de75c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.791+0000 7fa2a6ffd640 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561de75c7030 0x561de75c94a0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74241ff40eccd7e8 server_cookie=5adca0caaa0fbf7b in_seq=0 out_seq=0 2024-09-16T00:39:18.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.791+0000 7fa286ffd640 10 client.19554 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:18.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa286ffd640 1 -- 172.21.15.33:0/2269663369 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561de751f850 con 0x561de75c7030 2024-09-16T00:39:18.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa286ffd640 10 client.19554 handle_command_reply: tid=0 2024-09-16T00:39:18.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa2b01c0740 1 client.19554 shutdown 2024-09-16T00:39:18.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2269663369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 msgr2=0x561de75b6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:18.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75b6fd0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fa2980bc070 tx=0x7fa298003f50 comp rx=0 tx=0).stop 2024-09-16T00:39:18.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2269663369 shutdown_connections 2024-09-16T00:39:18.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561de75c7030 0x561de75c94a0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75b7680 0x561de75b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75acdc0 0x561de75b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/2269663369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75b6fd0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2269663369 >> 172.21.15.33:0/2269663369 conn(0x561de75a8a30 msgr2=0x561de75ae760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:18.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa286ffd640 0 client.19554 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:18.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2269663369 shutdown_connections 2024-09-16T00:39:18.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.792+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2269663369 wait complete. 2024-09-16T00:39:18.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.793+0000 7fa2b01c0740 20 client.19554 trim_cache size 0 max 16384 2024-09-16T00:39:18.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.816+0000 7fa2b01c0740 1 Processor -- start 2024-09-16T00:39:18.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.816+0000 7fa2b01c0740 1 -- start start 2024-09-16T00:39:18.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.816+0000 7fa2b01c0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.816+0000 7fa2b01c0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75acdc0 0x561de75af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.816+0000 7fa2b01c0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75b7680 0x561de75ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.816+0000 7fa2b01c0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a81af710 con 0x561de75ac4a0 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.816+0000 7fa2b01c0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a81acfe0 con 0x561de75acdc0 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.816+0000 7fa2b01c0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a81aae30 con 0x561de75b7680 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa2a6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75acdc0 0x561de75af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa2a7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75b7680 0x561de75ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa2a6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75acdc0 0x561de75af7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:55514/0 (socket says 172.21.15.33:55514) 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa2a6ffd640 1 -- 172.21.15.33:0/2020124179 learned_addr learned my addr 172.21.15.33:0/2020124179 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa2a77fe640 1 --2- 172.21.15.33:0/2020124179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa2a6ffd640 1 -- 172.21.15.33:0/2020124179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75b7680 msgr2=0x561de75ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa2a6ffd640 1 --2- 172.21.15.33:0/2020124179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75b7680 0x561de75ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa2a6ffd640 1 -- 172.21.15.33:0/2020124179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 msgr2=0x561de75c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa2a6ffd640 1 --2- 172.21.15.33:0/2020124179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75c34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa2a6ffd640 1 -- 172.21.15.33:0/2020124179 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561de75a8860 con 0x561de75acdc0 2024-09-16T00:39:18.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa2a6ffd640 1 --2- 172.21.15.33:0/2020124179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75acdc0 0x561de75af7f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa29c011010 tx=0x7fa29c01e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c35bd1087486672 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa28d7fa640 1 -- 172.21.15.33:0/2020124179 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c004020 con 0x561de75acdc0 2024-09-16T00:39:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa28d7fa640 1 -- 172.21.15.33:0/2020124179 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa29c01c960 con 0x561de75acdc0 2024-09-16T00:39:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.817+0000 7fa28d7fa640 1 -- 172.21.15.33:0/2020124179 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c0055a0 con 0x561de75acdc0 2024-09-16T00:39:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.818+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2020124179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75acdc0 msgr2=0x561de75af7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.818+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/2020124179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75acdc0 0x561de75af7f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa29c011010 tx=0x7fa29c01e040 comp rx=0 tx=0).stop 2024-09-16T00:39:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.818+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2020124179 shutdown_connections 2024-09-16T00:39:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.818+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/2020124179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75b7680 0x561de75ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.818+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/2020124179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75acdc0 0x561de75af7f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.818+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/2020124179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.818+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2020124179 >> 172.21.15.33:0/2020124179 conn(0x561de75a8a30 msgr2=0x561de75af0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.819+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2020124179 shutdown_connections 2024-09-16T00:39:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.819+0000 7fa2b01c0740 1 -- 172.21.15.33:0/2020124179 wait complete. 2024-09-16T00:39:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.819+0000 7fa2b01c0740 1 -- start start 2024-09-16T00:39:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.819+0000 7fa2b01c0740 1 Processor -- start 2024-09-16T00:39:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.820+0000 7fa2b01c0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75bf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.820+0000 7fa2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75bf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.820+0000 7fa2a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75bf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55506/0 (socket says 172.21.15.33:55506) 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.820+0000 7fa2a77fe640 1 -- 172.21.15.33:0/1842615541 learned_addr learned my addr 172.21.15.33:0/1842615541 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.820+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75acdc0 0x561de75bfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.820+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75b7680 0x561de75babc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.820+0000 7fa2b01c0740 1 -- 172.21.15.33:0/1842615541 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a81ab4a0 con 0x561de75ac4a0 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.820+0000 7fa2a7fff640 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75b7680 0x561de75babc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.820+0000 7fa2a6ffd640 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75acdc0 0x561de75bfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.820+0000 7fa2b01c0740 1 -- 172.21.15.33:0/1842615541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a81ac910 con 0x561de75b7680 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.821+0000 7fa2b01c0740 1 -- 172.21.15.33:0/1842615541 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a81abb70 con 0x561de75acdc0 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.821+0000 7fa2b01c0740 10 client.0 _ll_register_callbacks cb 0x7fa298005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.821+0000 7fa2a7fff640 1 -- 172.21.15.33:0/1842615541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75acdc0 msgr2=0x561de75bfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.821+0000 7fa2a7fff640 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75acdc0 0x561de75bfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.821+0000 7fa2a7fff640 1 -- 172.21.15.33:0/1842615541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 msgr2=0x561de75bf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.821+0000 7fa2a7fff640 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75bf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.821+0000 7fa2a7fff640 1 -- 172.21.15.33:0/1842615541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561de75bb3f0 con 0x561de75b7680 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.821+0000 7fa2a7fff640 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75b7680 0x561de75babc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x561de75b75f0 tx=0x7fa2a0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7725eedc5a33a3b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.821+0000 7fa286ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.822+0000 7fa286ffd640 1 -- 172.21.15.33:0/1842615541 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a002e060 con 0x561de75b7680 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.822+0000 7fa286ffd640 1 -- 172.21.15.33:0/1842615541 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa2a000f620 con 0x561de75b7680 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.822+0000 7fa2b01c0740 1 -- 172.21.15.33:0/1842615541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa29c01c960 con 0x561de75b7680 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.822+0000 7fa286ffd640 1 -- 172.21.15.33:0/1842615541 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa2a0018070 con 0x561de75b7680 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.822+0000 7fa2b01c0740 10 client.28739 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.822+0000 7fa2b01c0740 1 -- 172.21.15.33:0/1842615541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561de76317d0 con 0x561de75b7680 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.822+0000 7fa286ffd640 1 -- 172.21.15.33:0/1842615541 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa29c01c960 con 0x561de75b7680 2024-09-16T00:39:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.822+0000 7fa286ffd640 1 -- 172.21.15.33:0/1842615541 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fa2a0028070 con 0x561de75b7680 2024-09-16T00:39:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.823+0000 7fa2b01c0740 10 client.28739 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.823+0000 7fa2b01c0740 10 client.28739 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:39:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.823+0000 7fa2b01c0740 20 client.28739 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.823+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561de7631ef0 0x561de7634350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.823+0000 7fa2b01c0740 4 client.28739 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:39:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.823+0000 7fa2b01c0740 1 -- 172.21.15.33:0/1842615541 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561de7519d60 con 0x561de7631ef0 2024-09-16T00:39:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.823+0000 7fa2a77fe640 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561de7631ef0 0x561de7634350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.823+0000 7fa286ffd640 10 client.28739 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.823+0000 7fa2a77fe640 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561de7631ef0 0x561de7634350 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec4beb91a7815113 server_cookie=5840e47cb7af6442 in_seq=0 out_seq=0 2024-09-16T00:39:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa286ffd640 1 -- 172.21.15.33:0/1842615541 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561de7519d60 con 0x561de7631ef0 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa286ffd640 10 client.28739 handle_command_reply: tid=0 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa2b01c0740 1 client.28739 shutdown 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa2b01c0740 1 -- 172.21.15.33:0/1842615541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75b7680 msgr2=0x561de75babc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75b7680 0x561de75babc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x561de75b75f0 tx=0x7fa2a0004a40 comp rx=0 tx=0).stop 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa2b01c0740 1 -- 172.21.15.33:0/1842615541 shutdown_connections 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561de7631ef0 0x561de7634350 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561de75b7680 0x561de75babc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561de75acdc0 0x561de75bfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa2b01c0740 1 --2- 172.21.15.33:0/1842615541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561de75ac4a0 0x561de75bf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa2b01c0740 1 -- 172.21.15.33:0/1842615541 >> 172.21.15.33:0/1842615541 conn(0x561de75a8a30 msgr2=0x561de75ae760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa286ffd640 0 client.28739 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa2b01c0740 1 -- 172.21.15.33:0/1842615541 shutdown_connections 2024-09-16T00:39:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa2b01c0740 1 -- 172.21.15.33:0/1842615541 wait complete. 2024-09-16T00:39:18.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.824+0000 7fa2b01c0740 20 client.28739 trim_cache size 0 max 16384 2024-09-16T00:39:18.830 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:39:18.830 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:39:18.830 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:39:18.830 INFO:teuthology.orchestra.run.smithi033.stdout: "ee9d3b85-8015-4981-a3bb-767aab4fa9c1": { 2024-09-16T00:39:18.830 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:39:18.830 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "ee9d3b85-8015-4981-a3bb-767aab4fa9c1", 2024-09-16T00:39:18.830 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:39:18.830 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:39:18.830 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:39:18.831 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:39:18.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.825+0000 7fa2adde7640 1 -- 172.21.15.33:0/1672244198 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa2900a32c0 msgr2=0x7fa2900a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:18.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.825+0000 7fa2adde7640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa2900a32c0 0x7fa2900a5780 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0005c40 tx=0x7fa2a0009500 comp rx=0 tx=0).stop 2024-09-16T00:39:18.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.825+0000 7fa2adde7640 1 -- 172.21.15.33:0/1672244198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2a8102d40 msgr2=0x7fa2a8198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:18.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.825+0000 7fa2adde7640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2a8102d40 0x7fa2a8198430 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa298006010 tx=0x7fa2980042f0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.825+0000 7fa2adde7640 1 -- 172.21.15.33:0/1672244198 shutdown_connections 2024-09-16T00:39:18.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.825+0000 7fa2adde7640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa2900a32c0 0x7fa2900a5780 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.825+0000 7fa2adde7640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2a810c170 0x7fa2a819cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.825+0000 7fa2adde7640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2a8103660 0x7fa2a8198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.825+0000 7fa2adde7640 1 --2- 172.21.15.33:0/1672244198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2a8102d40 0x7fa2a8198430 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:18.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.825+0000 7fa2adde7640 1 -- 172.21.15.33:0/1672244198 >> 172.21.15.33:0/1672244198 conn(0x7fa2a80fc7a0 msgr2=0x7fa2a80fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:18.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.826+0000 7fa2adde7640 1 -- 172.21.15.33:0/1672244198 shutdown_connections 2024-09-16T00:39:18.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:18.826+0000 7fa2adde7640 1 -- 172.21.15.33:0/1672244198 wait complete. 2024-09-16T00:39:19.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 176864 18.60 MB/sec execute 385 sec latency 104.506 ms 2024-09-16T00:39:19.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:19 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:19.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:19 smithi060 ceph-mon[35185]: pgmap v1015: 129 pgs: 129 active+clean; 743 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 15 MiB/s wr, 548 op/s 2024-09-16T00:39:19.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:19 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:19.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:19 smithi033 ceph-mon[31243]: pgmap v1015: 129 pgs: 129 active+clean; 743 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 15 MiB/s wr, 548 op/s 2024-09-16T00:39:19.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:19 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:39:19.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:19 smithi083 ceph-mon[35242]: pgmap v1015: 129 pgs: 129 active+clean; 743 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 15 MiB/s wr, 548 op/s 2024-09-16T00:39:20.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 177478 18.65 MB/sec execute 386 sec latency 134.850 ms 2024-09-16T00:39:20.771 INFO:tasks.cephfs.filesystem:scrub status for tag:ee9d3b85-8015-4981-a3bb-767aab4fa9c1 - {'path': '/', 'tag': 'ee9d3b85-8015-4981-a3bb-767aab4fa9c1', 'options': 'recursive,force'} 2024-09-16T00:39:21.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 178017 18.71 MB/sec execute 387 sec latency 233.275 ms 2024-09-16T00:39:21.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:21 smithi060 ceph-mon[35185]: pgmap v1016: 129 pgs: 129 active+clean; 676 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 15 MiB/s wr, 673 op/s 2024-09-16T00:39:21.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:21 smithi033 ceph-mon[31243]: pgmap v1016: 129 pgs: 129 active+clean; 676 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 15 MiB/s wr, 673 op/s 2024-09-16T00:39:21.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:21 smithi083 ceph-mon[35242]: pgmap v1016: 129 pgs: 129 active+clean; 676 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 15 MiB/s wr, 673 op/s 2024-09-16T00:39:22.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 178543 18.71 MB/sec execute 388 sec latency 243.683 ms 2024-09-16T00:39:23.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 179046 18.72 MB/sec execute 389 sec latency 58.473 ms 2024-09-16T00:39:23.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:23 smithi060 ceph-mon[35185]: pgmap v1017: 129 pgs: 129 active+clean; 653 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 15 MiB/s wr, 748 op/s 2024-09-16T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:23 smithi033 ceph-mon[31243]: pgmap v1017: 129 pgs: 129 active+clean; 653 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 15 MiB/s wr, 748 op/s 2024-09-16T00:39:23.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:23 smithi083 ceph-mon[35242]: pgmap v1017: 129 pgs: 129 active+clean; 653 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 15 MiB/s wr, 748 op/s 2024-09-16T00:39:24.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 179361 18.71 MB/sec execute 390 sec latency 469.048 ms 2024-09-16T00:39:25.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 180003 18.78 MB/sec execute 391 sec latency 90.580 ms 2024-09-16T00:39:25.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:25 smithi060 ceph-mon[35185]: pgmap v1018: 129 pgs: 129 active+clean; 657 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 12 MiB/s wr, 680 op/s 2024-09-16T00:39:25.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:25 smithi033 ceph-mon[31243]: pgmap v1018: 129 pgs: 129 active+clean; 657 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 12 MiB/s wr, 680 op/s 2024-09-16T00:39:25.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:25 smithi083 ceph-mon[35242]: pgmap v1018: 129 pgs: 129 active+clean; 657 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 12 MiB/s wr, 680 op/s 2024-09-16T00:39:26.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 180553 18.80 MB/sec execute 392 sec latency 90.914 ms 2024-09-16T00:39:27.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 180988 18.79 MB/sec execute 393 sec latency 95.542 ms 2024-09-16T00:39:27.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:27 smithi083 ceph-mon[35242]: pgmap v1019: 129 pgs: 129 active+clean; 664 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.2 MiB/s wr, 618 op/s 2024-09-16T00:39:27.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:27 smithi060 ceph-mon[35185]: pgmap v1019: 129 pgs: 129 active+clean; 664 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.2 MiB/s wr, 618 op/s 2024-09-16T00:39:27.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:27 smithi033 ceph-mon[31243]: pgmap v1019: 129 pgs: 129 active+clean; 664 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.2 MiB/s wr, 618 op/s 2024-09-16T00:39:28.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 181540 18.83 MB/sec execute 394 sec latency 80.082 ms 2024-09-16T00:39:29.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 182154 18.89 MB/sec execute 395 sec latency 358.219 ms 2024-09-16T00:39:29.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:29 smithi060 ceph-mon[35185]: pgmap v1020: 129 pgs: 129 active+clean; 664 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.3 MiB/s wr, 399 op/s 2024-09-16T00:39:29.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:29 smithi033 ceph-mon[31243]: pgmap v1020: 129 pgs: 129 active+clean; 664 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.3 MiB/s wr, 399 op/s 2024-09-16T00:39:29.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:29 smithi083 ceph-mon[35242]: pgmap v1020: 129 pgs: 129 active+clean; 664 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.3 MiB/s wr, 399 op/s 2024-09-16T00:39:30.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 182704 18.90 MB/sec execute 396 sec latency 69.376 ms 2024-09-16T00:39:31.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 183178 18.91 MB/sec execute 397 sec latency 86.056 ms 2024-09-16T00:39:31.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:31 smithi060 ceph-mon[35185]: pgmap v1021: 129 pgs: 129 active+clean; 668 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.7 MiB/s wr, 467 op/s 2024-09-16T00:39:31.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:31 smithi033 ceph-mon[31243]: pgmap v1021: 129 pgs: 129 active+clean; 668 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.7 MiB/s wr, 467 op/s 2024-09-16T00:39:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:31 smithi083 ceph-mon[35242]: pgmap v1021: 129 pgs: 129 active+clean; 668 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.7 MiB/s wr, 467 op/s 2024-09-16T00:39:32.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 183794 18.95 MB/sec execute 398 sec latency 107.610 ms 2024-09-16T00:39:32.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:32 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:39:32.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:32 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:39:32.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:32 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:39:32.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:32 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:39:32.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:32 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:39:32.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:32 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:39:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:32 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:39:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:32 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:39:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:32 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:39:33.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 184367 18.99 MB/sec execute 399 sec latency 122.035 ms 2024-09-16T00:39:33.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:33 smithi060 ceph-mon[35185]: pgmap v1022: 129 pgs: 129 active+clean; 677 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.3 MiB/s wr, 424 op/s 2024-09-16T00:39:33.833 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:33 smithi033 ceph-mon[31243]: pgmap v1022: 129 pgs: 129 active+clean; 677 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.3 MiB/s wr, 424 op/s 2024-09-16T00:39:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:33 smithi083 ceph-mon[35242]: pgmap v1022: 129 pgs: 129 active+clean; 677 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.3 MiB/s wr, 424 op/s 2024-09-16T00:39:34.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 184855 18.99 MB/sec execute 400 sec latency 103.801 ms 2024-09-16T00:39:35.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 185307 19.00 MB/sec execute 401 sec latency 106.135 ms 2024-09-16T00:39:35.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:35 smithi060 ceph-mon[35185]: pgmap v1023: 129 pgs: 129 active+clean; 680 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.9 MiB/s wr, 315 op/s 2024-09-16T00:39:35.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:39:35.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:35.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:35 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:35.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:35 smithi033 ceph-mon[31243]: pgmap v1023: 129 pgs: 129 active+clean; 680 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.9 MiB/s wr, 315 op/s 2024-09-16T00:39:35.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:39:35.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:35.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:35 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:35 smithi083 ceph-mon[35242]: pgmap v1023: 129 pgs: 129 active+clean; 680 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.9 MiB/s wr, 315 op/s 2024-09-16T00:39:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:39:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:35 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:36.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 185962 19.05 MB/sec execute 402 sec latency 119.754 ms 2024-09-16T00:39:36.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:36 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:36.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:36 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:36.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:36 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:39:36.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:36 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:39:36.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:36 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:36.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:36.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:36.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:39:36.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:39:36.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:36.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:36 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:36.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:36 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:36.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:36 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:39:36.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:36 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:39:36.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:36 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:39:37.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 186555 19.09 MB/sec execute 403 sec latency 79.331 ms 2024-09-16T00:39:37.132 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 34.4 2024-09-16T00:39:37.133 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:39:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:37.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:37 smithi060 ceph-mon[35185]: pgmap v1024: 129 pgs: 129 active+clean; 690 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.5 MiB/s wr, 384 op/s 2024-09-16T00:39:37.732 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:37 smithi033 ceph-mon[31243]: pgmap v1024: 129 pgs: 129 active+clean; 690 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.5 MiB/s wr, 384 op/s 2024-09-16T00:39:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:37 smithi083 ceph-mon[35242]: pgmap v1024: 129 pgs: 129 active+clean; 690 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.5 MiB/s wr, 384 op/s 2024-09-16T00:39:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.971+0000 7fed3b3bb640 1 -- 172.21.15.33:0/4189289271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed34103660 msgr2=0x7fed3410bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:37.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.971+0000 7fed3b3bb640 1 --2- 172.21.15.33:0/4189289271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed34103660 0x7fed3410bb50 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fed28009900 tx=0x7fed2802a480 comp rx=0 tx=0).stop 2024-09-16T00:39:37.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.971+0000 7fed3b3bb640 1 -- 172.21.15.33:0/4189289271 shutdown_connections 2024-09-16T00:39:37.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.971+0000 7fed3b3bb640 1 --2- 172.21.15.33:0/4189289271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed3410c170 0x7fed3410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:37.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.971+0000 7fed3b3bb640 1 --2- 172.21.15.33:0/4189289271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed34103660 0x7fed3410bb50 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:37.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.971+0000 7fed3b3bb640 1 --2- 172.21.15.33:0/4189289271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed34102d40 0x7fed34103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:37.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.971+0000 7fed3b3bb640 1 -- 172.21.15.33:0/4189289271 >> 172.21.15.33:0/4189289271 conn(0x7fed340fc7a0 msgr2=0x7fed340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:37.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.971+0000 7fed3b3bb640 1 -- 172.21.15.33:0/4189289271 shutdown_connections 2024-09-16T00:39:37.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.971+0000 7fed3b3bb640 1 -- 172.21.15.33:0/4189289271 wait complete. 2024-09-16T00:39:37.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.972+0000 7fed3b3bb640 1 Processor -- start 2024-09-16T00:39:37.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.972+0000 7fed3b3bb640 1 -- start start 2024-09-16T00:39:37.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.972+0000 7fed3b3bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed34102d40 0x7fed34198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:37.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.972+0000 7fed3b3bb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed34103660 0x7fed34198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:37.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.972+0000 7fed3b3bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3410c170 0x7fed3419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:37.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.972+0000 7fed3b3bb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fed34111f70 con 0x7fed34103660 2024-09-16T00:39:37.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.972+0000 7fed3b3bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fed34111df0 con 0x7fed34102d40 2024-09-16T00:39:37.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.972+0000 7fed3b3bb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fed341120f0 con 0x7fed3410c170 2024-09-16T00:39:37.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed38956640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed34103660 0x7fed34198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:37.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed39157640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed34102d40 0x7fed34198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:37.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed39958640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3410c170 0x7fed3419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:37.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed38956640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed34103660 0x7fed34198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59066/0 (socket says 172.21.15.33:59066) 2024-09-16T00:39:37.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed38956640 1 -- 172.21.15.33:0/151381127 learned_addr learned my addr 172.21.15.33:0/151381127 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:37.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed39958640 1 -- 172.21.15.33:0/151381127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed34102d40 msgr2=0x7fed34198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:37.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed39958640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed34102d40 0x7fed34198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:37.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed39958640 1 -- 172.21.15.33:0/151381127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed34103660 msgr2=0x7fed34198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:37.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed39958640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed34103660 0x7fed34198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:37.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed39958640 1 -- 172.21.15.33:0/151381127 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3419d400 con 0x7fed3410c170 2024-09-16T00:39:37.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed39157640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed34102d40 0x7fed34198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:39:37.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed38956640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed34103660 0x7fed34198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:39:37.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed39958640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3410c170 0x7fed3419cd20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fed2c00e940 tx=0x7fed2c00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:37.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed327fc640 1 -- 172.21.15.33:0/151381127 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2c004d20 con 0x7fed3410c170 2024-09-16T00:39:37.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed327fc640 1 -- 172.21.15.33:0/151381127 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed2c005020 con 0x7fed3410c170 2024-09-16T00:39:37.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed3b3bb640 1 -- 172.21.15.33:0/151381127 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3419d6f0 con 0x7fed3410c170 2024-09-16T00:39:37.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed327fc640 1 -- 172.21.15.33:0/151381127 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2c0116b0 con 0x7fed3410c170 2024-09-16T00:39:37.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.973+0000 7fed3b3bb640 1 -- 172.21.15.33:0/151381127 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed341a4fd0 con 0x7fed3410c170 2024-09-16T00:39:37.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.974+0000 7fed3b3bb640 1 -- 172.21.15.33:0/151381127 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed34103120 con 0x7fed3410c170 2024-09-16T00:39:37.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.976+0000 7fed327fc640 1 -- 172.21.15.33:0/151381127 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fed2c0054d0 con 0x7fed3410c170 2024-09-16T00:39:37.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.976+0000 7fed327fc640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fed1c0a3240 0x7fed1c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:37.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.976+0000 7fed327fc640 1 -- 172.21.15.33:0/151381127 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fed2c0cff40 con 0x7fed3410c170 2024-09-16T00:39:37.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.976+0000 7fed39157640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fed1c0a3240 0x7fed1c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:37.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.977+0000 7fed39157640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fed1c0a3240 0x7fed1c0a5700 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fed2400bfd0 tx=0x7fed2400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:37.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:37.984+0000 7fed327fc640 1 -- 172.21.15.33:0/151381127 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fed2c016030 con 0x7fed3410c170 2024-09-16T00:39:38.097 INFO:tasks.workunit.client.0.smithi033.stdout: 10 186994 19.10 MB/sec execute 404 sec latency 97.872 ms 2024-09-16T00:39:38.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.211+0000 7fed3b3bb640 1 -- 172.21.15.33:0/151381127 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed341997b0 con 0x7fed3410c170 2024-09-16T00:39:38.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.212+0000 7fed327fc640 1 -- 172.21.15.33:0/151381127 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7fed2c09c190 con 0x7fed3410c170 2024-09-16T00:39:38.213 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:39:38.213 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:39:38.213 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:39:38.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.214+0000 7fed3b3bb640 1 -- 172.21.15.33:0/151381127 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fed1c0a3240 msgr2=0x7fed1c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:38.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.214+0000 7fed3b3bb640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fed1c0a3240 0x7fed1c0a5700 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fed2400bfd0 tx=0x7fed2400b390 comp rx=0 tx=0).stop 2024-09-16T00:39:38.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.214+0000 7fed3b3bb640 1 -- 172.21.15.33:0/151381127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3410c170 msgr2=0x7fed3419cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:38.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.214+0000 7fed3b3bb640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3410c170 0x7fed3419cd20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fed2c00e940 tx=0x7fed2c00ee00 comp rx=0 tx=0).stop 2024-09-16T00:39:38.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.214+0000 7fed3b3bb640 1 -- 172.21.15.33:0/151381127 shutdown_connections 2024-09-16T00:39:38.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.214+0000 7fed3b3bb640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fed1c0a3240 0x7fed1c0a5700 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:38.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.214+0000 7fed3b3bb640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3410c170 0x7fed3419cd20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:38.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.214+0000 7fed3b3bb640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed34103660 0x7fed34198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:38.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.214+0000 7fed3b3bb640 1 --2- 172.21.15.33:0/151381127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed34102d40 0x7fed34198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:38.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.215+0000 7fed3b3bb640 1 -- 172.21.15.33:0/151381127 >> 172.21.15.33:0/151381127 conn(0x7fed340fc7a0 msgr2=0x7fed340fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:38.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.215+0000 7fed3b3bb640 1 -- 172.21.15.33:0/151381127 shutdown_connections 2024-09-16T00:39:38.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:38.215+0000 7fed3b3bb640 1 -- 172.21.15.33:0/151381127 wait complete. 2024-09-16T00:39:38.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:38 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/151381127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:39:38.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:38 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/151381127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:39:38.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:38 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/151381127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:39:39.040 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 154.4 --await-for 45.0' 2024-09-16T00:39:39.041 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 154.4 --await-for 45.0 2024-09-16T00:39:39.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 187477 19.11 MB/sec execute 405 sec latency 85.818 ms 2024-09-16T00:39:39.348 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:39.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:39 smithi060 ceph-mon[35185]: pgmap v1025: 129 pgs: 129 active+clean; 690 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 309 op/s 2024-09-16T00:39:39.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:39 smithi033 ceph-mon[31243]: pgmap v1025: 129 pgs: 129 active+clean; 690 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 309 op/s 2024-09-16T00:39:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:39 smithi083 ceph-mon[35242]: pgmap v1025: 129 pgs: 129 active+clean; 690 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 309 op/s 2024-09-16T00:39:40.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188062 19.14 MB/sec execute 406 sec latency 56.528 ms 2024-09-16T00:39:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.371+0000 7f5aa3596640 1 -- 172.21.15.33:0/1852376708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a9c100b20 msgr2=0x7f5a9c100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.371+0000 7f5aa3596640 1 --2- 172.21.15.33:0/1852376708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a9c100b20 0x7f5a9c100f00 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009900 tx=0x7f5a9802a410 comp rx=0 tx=0).stop 2024-09-16T00:39:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.371+0000 7f5aa3596640 1 -- 172.21.15.33:0/1852376708 shutdown_connections 2024-09-16T00:39:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.371+0000 7f5aa3596640 1 --2- 172.21.15.33:0/1852376708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a9c10e500 0x7f5a9c110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.371+0000 7f5aa3596640 1 --2- 172.21.15.33:0/1852376708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a9c101440 0x7f5a9c10df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.371+0000 7f5aa3596640 1 --2- 172.21.15.33:0/1852376708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a9c100b20 0x7f5a9c100f00 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.371+0000 7f5aa3596640 1 -- 172.21.15.33:0/1852376708 >> 172.21.15.33:0/1852376708 conn(0x7f5a9c0fc7d0 msgr2=0x7f5a9c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.371+0000 7f5aa3596640 1 -- 172.21.15.33:0/1852376708 shutdown_connections 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.371+0000 7f5aa3596640 1 -- 172.21.15.33:0/1852376708 wait complete. 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.372+0000 7f5aa3596640 1 Processor -- start 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.372+0000 7f5aa3596640 1 -- start start 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.372+0000 7f5aa3596640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a9c100b20 0x7f5a9c1a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.372+0000 7f5aa3596640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a9c101440 0x7f5a9c1a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.372+0000 7f5aa3596640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a9c10e500 0x7f5a9c19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa3596640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c114280 con 0x7f5a9c10e500 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa3596640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c114100 con 0x7f5a9c101440 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa3596640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c114400 con 0x7f5a9c100b20 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa1332640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a9c100b20 0x7f5a9c1a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa1332640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a9c100b20 0x7f5a9c1a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34862/0 (socket says 172.21.15.33:34862) 2024-09-16T00:39:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa1332640 1 -- 172.21.15.33:0/2164876156 learned_addr learned my addr 172.21.15.33:0/2164876156 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa1b33640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a9c10e500 0x7f5a9c19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa0b31640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a9c101440 0x7f5a9c1a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa0b31640 1 -- 172.21.15.33:0/2164876156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a9c100b20 msgr2=0x7f5a9c1a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa0b31640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a9c100b20 0x7f5a9c1a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa0b31640 1 -- 172.21.15.33:0/2164876156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a9c10e500 msgr2=0x7f5a9c19a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa0b31640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a9c10e500 0x7f5a9c19a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa0b31640 1 -- 172.21.15.33:0/2164876156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a9c19ace0 con 0x7f5a9c101440 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa1332640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a9c100b20 0x7f5a9c1a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.373+0000 7f5aa0b31640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a9c101440 0x7f5a9c1a0930 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c00e9c0 tx=0x7f5a8c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.374+0000 7f5a927fc640 1 -- 172.21.15.33:0/2164876156 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c00ccc0 con 0x7f5a9c101440 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.374+0000 7f5a927fc640 1 -- 172.21.15.33:0/2164876156 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a8c0035f0 con 0x7f5a9c101440 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.374+0000 7f5a927fc640 1 -- 172.21.15.33:0/2164876156 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c00ce60 con 0x7f5a9c101440 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.374+0000 7f5aa3596640 1 -- 172.21.15.33:0/2164876156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a9c19afd0 con 0x7f5a9c101440 2024-09-16T00:39:40.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.374+0000 7f5aa3596640 1 -- 172.21.15.33:0/2164876156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a9c1a72c0 con 0x7f5a9c101440 2024-09-16T00:39:40.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.376+0000 7f5a927fc640 1 -- 172.21.15.33:0/2164876156 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5a8c0037a0 con 0x7f5a9c101440 2024-09-16T00:39:40.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.377+0000 7f5aa596f740 1 Processor -- start 2024-09-16T00:39:40.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.377+0000 7f5a927fc640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5a880a3240 0x7f5a880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.377+0000 7f5aa596f740 1 -- start start 2024-09-16T00:39:40.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa1b33640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5a880a3240 0x7f5a880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5a927fc640 1 -- 172.21.15.33:0/2164876156 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5a8c0d0120 con 0x7f5a9c101440 2024-09-16T00:39:40.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa596f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088c5760 0x5566088c5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa596f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c6080 0x5566088d0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa596f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088d0940 0x5566088d2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa596f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55660882b2a0 con 0x5566088c5760 2024-09-16T00:39:40.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa596f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5566087d2b10 con 0x5566088d0940 2024-09-16T00:39:40.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa596f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556608827260 con 0x5566088c6080 2024-09-16T00:39:40.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa0b31640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c6080 0x5566088d0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa0b31640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c6080 0x5566088d0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34864/0 (socket says 172.21.15.33:34864) 2024-09-16T00:39:40.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa1332640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088c5760 0x5566088c5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa0b31640 1 -- 172.21.15.33:0/1896252604 learned_addr learned my addr 172.21.15.33:0/1896252604 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:40.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa1b33640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5a880a3240 0x7f5a880a5700 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c069760 tx=0x7f5a94009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:40.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa1b33640 1 --2- 172.21.15.33:0/1896252604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088d0940 0x5566088d2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa1332640 1 -- 172.21.15.33:0/1896252604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c6080 msgr2=0x5566088d0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:40.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa1332640 1 --2- 172.21.15.33:0/1896252604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c6080 0x5566088d0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa1332640 1 -- 172.21.15.33:0/1896252604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088d0940 msgr2=0x5566088d2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:40.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa1332640 1 --2- 172.21.15.33:0/1896252604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088d0940 0x5566088d2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.378+0000 7f5aa1332640 1 -- 172.21.15.33:0/1896252604 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55660859cb90 con 0x5566088c5760 2024-09-16T00:39:40.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.379+0000 7f5aa1332640 1 --2- 172.21.15.33:0/1896252604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088c5760 0x5566088c5b40 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009230 tx=0x7f5a98005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd45e9817ce8181 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.379+0000 7f5a877fe640 1 -- 172.21.15.33:0/1896252604 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98040070 con 0x5566088c5760 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.379+0000 7f5a877fe640 1 -- 172.21.15.33:0/1896252604 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a98031970 con 0x5566088c5760 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.379+0000 7f5a877fe640 1 -- 172.21.15.33:0/1896252604 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98035040 con 0x5566088c5760 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.380+0000 7f5aa596f740 1 -- 172.21.15.33:0/1896252604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088c5760 msgr2=0x5566088c5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.380+0000 7f5aa596f740 1 --2- 172.21.15.33:0/1896252604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088c5760 0x5566088c5b40 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009230 tx=0x7f5a98005dd0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.380+0000 7f5aa596f740 1 -- 172.21.15.33:0/1896252604 shutdown_connections 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.380+0000 7f5aa596f740 1 --2- 172.21.15.33:0/1896252604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088d0940 0x5566088d2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.380+0000 7f5aa596f740 1 --2- 172.21.15.33:0/1896252604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c6080 0x5566088d0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.380+0000 7f5aa596f740 1 --2- 172.21.15.33:0/1896252604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088c5760 0x5566088c5b40 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.380+0000 7f5aa596f740 1 -- 172.21.15.33:0/1896252604 >> 172.21.15.33:0/1896252604 conn(0x5566088c1cf0 msgr2=0x5566088c20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.380+0000 7f5aa596f740 1 -- 172.21.15.33:0/1896252604 shutdown_connections 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.380+0000 7f5aa596f740 1 -- 172.21.15.33:0/1896252604 wait complete. 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.381+0000 7f5aa596f740 1 -- start start 2024-09-16T00:39:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.381+0000 7f5aa596f740 1 Processor -- start 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.381+0000 7f5aa596f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c5760 0x5566088cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.381+0000 7f5aa596f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088c6080 0x5566088cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.381+0000 7f5aa596f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088d0940 0x5566088ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.381+0000 7f5aa596f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5566088d3510 con 0x5566088c6080 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.381+0000 7f5aa596f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5566087b36e0 con 0x5566088c5760 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.381+0000 7f5aa596f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5566088d3690 con 0x5566088d0940 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.381+0000 7f5aa596f740 10 client.0 _ll_register_callbacks cb 0x5566087d7a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa1332640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c5760 0x5566088cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa1332640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c5760 0x5566088cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:37268/0 (socket says 172.21.15.33:37268) 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa1332640 1 -- 172.21.15.33:0/590469571 learned_addr learned my addr 172.21.15.33:0/590469571 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa0b31640 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088c6080 0x5566088cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa1b33640 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088d0940 0x5566088ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa1332640 1 -- 172.21.15.33:0/590469571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088d0940 msgr2=0x5566088ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa1332640 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088d0940 0x5566088ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa1332640 1 -- 172.21.15.33:0/590469571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088c6080 msgr2=0x5566088cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa1332640 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088c6080 0x5566088cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa1332640 1 -- 172.21.15.33:0/590469571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566088dca00 con 0x5566088c5760 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa0b31640 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088c6080 0x5566088cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa1b33640 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088d0940 0x5566088ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa1332640 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c5760 0x5566088cd810 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5a98005ab0 tx=0x7f5a98004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66cafbb01579828f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5a84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5a84ff9640 1 -- 172.21.15.33:0/590469571 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9804b070 con 0x5566088c5760 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5a84ff9640 1 -- 172.21.15.33:0/590469571 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5a980035f0 con 0x5566088c5760 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.382+0000 7f5aa596f740 1 -- 172.21.15.33:0/590469571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a98031970 con 0x5566088c5760 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.383+0000 7f5a84ff9640 1 -- 172.21.15.33:0/590469571 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5a98057050 con 0x5566088c5760 2024-09-16T00:39:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.383+0000 7f5aa596f740 10 client.28757 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.383+0000 7f5aa596f740 1 -- 172.21.15.33:0/590469571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5566088df370 con 0x5566088c5760 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.383+0000 7f5a84ff9640 1 -- 172.21.15.33:0/590469571 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a98031970 con 0x5566088c5760 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.383+0000 7f5a84ff9640 1 -- 172.21.15.33:0/590469571 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f5a9803c8b0 con 0x5566088c5760 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.383+0000 7f5aa596f740 10 client.28757 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.383+0000 7f5aa596f740 10 client.28757 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.383+0000 7f5aa596f740 20 client.28757 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.383+0000 7f5aa596f740 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566088e0910 0x5566088e2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.383+0000 7f5aa596f740 4 client.28757 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.383+0000 7f5aa596f740 1 -- 172.21.15.33:0/590469571 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566088389d0 con 0x5566088e0910 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.384+0000 7f5aa0b31640 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566088e0910 0x5566088e2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.384+0000 7f5a84ff9640 10 client.28757 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.384+0000 7f5aa0b31640 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566088e0910 0x5566088e2d80 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bff07c848b3edf85 server_cookie=21ff52f8fb440d26 in_seq=0 out_seq=0 2024-09-16T00:39:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.385+0000 7f5a84ff9640 1 -- 172.21.15.33:0/590469571 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5566088389d0 con 0x5566088e0910 2024-09-16T00:39:40.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.385+0000 7f5a84ff9640 10 client.28757 handle_command_reply: tid=0 2024-09-16T00:39:40.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.385+0000 7f5aa596f740 1 client.28757 shutdown 2024-09-16T00:39:40.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.386+0000 7f5aa596f740 1 -- 172.21.15.33:0/590469571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c5760 msgr2=0x5566088cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:40.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.386+0000 7f5aa596f740 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c5760 0x5566088cd810 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5a98005ab0 tx=0x7f5a98004270 comp rx=0 tx=0).stop 2024-09-16T00:39:40.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.386+0000 7f5aa596f740 1 -- 172.21.15.33:0/590469571 shutdown_connections 2024-09-16T00:39:40.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.386+0000 7f5aa596f740 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5566088e0910 0x5566088e2d80 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.386+0000 7f5aa596f740 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088d0940 0x5566088ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.386+0000 7f5aa596f740 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088c6080 0x5566088cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.386+0000 7f5aa596f740 1 --2- 172.21.15.33:0/590469571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c5760 0x5566088cd810 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.386+0000 7f5aa596f740 1 -- 172.21.15.33:0/590469571 >> 172.21.15.33:0/590469571 conn(0x5566088c1cf0 msgr2=0x5566088c0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:40.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.386+0000 7f5a84ff9640 0 client.28757 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:40.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.386+0000 7f5aa596f740 1 -- 172.21.15.33:0/590469571 shutdown_connections 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.386+0000 7f5aa596f740 1 -- 172.21.15.33:0/590469571 wait complete. 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.386+0000 7f5aa596f740 20 client.28757 trim_cache size 0 max 16384 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.404+0000 7f5aa596f740 1 Processor -- start 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.404+0000 7f5aa596f740 1 -- start start 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.404+0000 7f5aa596f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c5760 0x5566088e8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.404+0000 7f5aa596f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c6080 0x5566088e8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.404+0000 7f5aa596f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 0x5566088caa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.404+0000 7f5aa596f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c1b3be0 con 0x5566088d0940 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.404+0000 7f5aa596f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c1b14b0 con 0x5566088c6080 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.404+0000 7f5aa596f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c1af300 con 0x5566088c5760 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.404+0000 7f5aa1b33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 0x5566088caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.404+0000 7f5aa1b33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 0x5566088caa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42882/0 (socket says 172.21.15.33:42882) 2024-09-16T00:39:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.404+0000 7f5aa1b33640 1 -- 172.21.15.33:0/1998464183 learned_addr learned my addr 172.21.15.33:0/1998464183 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.405+0000 7f5aa1332640 1 --2- 172.21.15.33:0/1998464183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c5760 0x5566088e8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.405+0000 7f5aa0b31640 1 --2- 172.21.15.33:0/1998464183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c6080 0x5566088e8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.405+0000 7f5aa1b33640 1 -- 172.21.15.33:0/1998464183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c5760 msgr2=0x5566088e8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.405+0000 7f5aa1b33640 1 --2- 172.21.15.33:0/1998464183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c5760 0x5566088e8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.405+0000 7f5aa1b33640 1 -- 172.21.15.33:0/1998464183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c6080 msgr2=0x5566088e8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.405+0000 7f5aa1b33640 1 --2- 172.21.15.33:0/1998464183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c6080 0x5566088e8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.405+0000 7f5aa1b33640 1 -- 172.21.15.33:0/1998464183 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566088d0360 con 0x5566088d0940 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.405+0000 7f5aa1b33640 1 --2- 172.21.15.33:0/1998464183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 0x5566088caa70 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f5a94003cc0 tx=0x7f5a94014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2f6608a6eadaa3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.405+0000 7f5a86ffd640 1 -- 172.21.15.33:0/1998464183 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a94003db0 con 0x5566088d0940 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.405+0000 7f5a86ffd640 1 -- 172.21.15.33:0/1998464183 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a94005020 con 0x5566088d0940 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.405+0000 7f5a86ffd640 1 -- 172.21.15.33:0/1998464183 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9401a6b0 con 0x5566088d0940 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.406+0000 7f5aa596f740 1 -- 172.21.15.33:0/1998464183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 msgr2=0x5566088caa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:40.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.406+0000 7f5aa596f740 1 --2- 172.21.15.33:0/1998464183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 0x5566088caa70 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f5a94003cc0 tx=0x7f5a94014000 comp rx=0 tx=0).stop 2024-09-16T00:39:40.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.406+0000 7f5aa596f740 1 -- 172.21.15.33:0/1998464183 shutdown_connections 2024-09-16T00:39:40.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.406+0000 7f5aa596f740 1 --2- 172.21.15.33:0/1998464183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 0x5566088caa70 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.406+0000 7f5aa596f740 1 --2- 172.21.15.33:0/1998464183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c6080 0x5566088e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.406+0000 7f5aa596f740 1 --2- 172.21.15.33:0/1998464183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c5760 0x5566088e8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.406+0000 7f5aa596f740 1 -- 172.21.15.33:0/1998464183 >> 172.21.15.33:0/1998464183 conn(0x5566088c1cf0 msgr2=0x5566088cf890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:40.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.406+0000 7f5aa596f740 1 -- 172.21.15.33:0/1998464183 shutdown_connections 2024-09-16T00:39:40.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.406+0000 7f5aa596f740 1 -- 172.21.15.33:0/1998464183 wait complete. 2024-09-16T00:39:40.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.406+0000 7f5aa596f740 1 -- start start 2024-09-16T00:39:40.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa596f740 1 Processor -- start 2024-09-16T00:39:40.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa596f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c5760 0x5566088d8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa596f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c6080 0x5566088d41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa596f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 0x5566088d4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa596f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c1af970 con 0x5566088d0940 2024-09-16T00:39:40.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa596f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c1b0de0 con 0x5566088c6080 2024-09-16T00:39:40.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa596f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c1b0040 con 0x5566088c5760 2024-09-16T00:39:40.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa596f740 10 client.0 _ll_register_callbacks cb 0x7f5a98005650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:40.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa1332640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c5760 0x5566088d8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa1b33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 0x5566088d4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa0b31640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c6080 0x5566088d41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa1332640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c5760 0x5566088d8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34886/0 (socket says 172.21.15.33:34886) 2024-09-16T00:39:40.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.407+0000 7f5aa1332640 1 -- 172.21.15.33:0/815481583 learned_addr learned my addr 172.21.15.33:0/815481583 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:40.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5aa0b31640 1 -- 172.21.15.33:0/815481583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c5760 msgr2=0x5566088d8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:40.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5aa0b31640 1 --2- 172.21.15.33:0/815481583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c5760 0x5566088d8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5aa0b31640 1 -- 172.21.15.33:0/815481583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 msgr2=0x5566088d4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5aa0b31640 1 --2- 172.21.15.33:0/815481583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 0x5566088d4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5aa0b31640 1 -- 172.21.15.33:0/815481583 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566088d4d80 con 0x5566088c6080 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5aa1b33640 1 --2- 172.21.15.33:0/815481583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 0x5566088d4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5aa0b31640 1 --2- 172.21.15.33:0/815481583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c6080 0x5566088d41d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0a4a00 tx=0x7f5a8c09c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=613afbf049fa4f5a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5a84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5a84ff9640 1 -- 172.21.15.33:0/815481583 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c0b5170 con 0x5566088c6080 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5a84ff9640 1 -- 172.21.15.33:0/815481583 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f5a8c098930 con 0x5566088c6080 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5aa596f740 1 -- 172.21.15.33:0/815481583 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a94005020 con 0x5566088c6080 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5a84ff9640 1 -- 172.21.15.33:0/815481583 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5a8c0a4aa0 con 0x5566088c6080 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5aa596f740 10 client.28763 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.408+0000 7f5aa596f740 1 -- 172.21.15.33:0/815481583 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55660894b4d0 con 0x5566088c6080 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.409+0000 7f5a84ff9640 1 -- 172.21.15.33:0/815481583 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a94005020 con 0x5566088c6080 2024-09-16T00:39:40.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.409+0000 7f5a84ff9640 1 -- 172.21.15.33:0/815481583 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f5a8c0bc1d0 con 0x5566088c6080 2024-09-16T00:39:40.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.409+0000 7f5aa596f740 10 client.28763 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:40.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.409+0000 7f5aa596f740 10 client.28763 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:39:40.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.409+0000 7f5aa596f740 20 client.28763 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:40.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.409+0000 7f5aa596f740 1 --2- 172.21.15.33:0/815481583 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55660894bc70 0x55660894e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:40.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.409+0000 7f5aa596f740 4 client.28763 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 154.4, "await_for": 45.0}] 2024-09-16T00:39:40.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.409+0000 7f5aa596f740 1 -- 172.21.15.33:0/815481583 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 154.4, "await_for": 45.0}) -- 0x556608832d20 con 0x55660894bc70 2024-09-16T00:39:40.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.409+0000 7f5aa1332640 1 --2- 172.21.15.33:0/815481583 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55660894bc70 0x55660894e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:40.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.409+0000 7f5a84ff9640 10 client.28763 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:40.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:40.409+0000 7f5aa1332640 1 --2- 172.21.15.33:0/815481583 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55660894bc70 0x55660894e090 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1298d1a1937291e9 server_cookie=87939512b5f3e575 in_seq=0 out_seq=0 2024-09-16T00:39:41.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188250 19.12 MB/sec execute 407 sec latency 651.116 ms 2024-09-16T00:39:41.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:41 smithi060 ceph-mon[35185]: pgmap v1026: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.3 MiB/s wr, 377 op/s 2024-09-16T00:39:41.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:41 smithi033 ceph-mon[31243]: pgmap v1026: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.3 MiB/s wr, 377 op/s 2024-09-16T00:39:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:41 smithi083 ceph-mon[35242]: pgmap v1026: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.3 MiB/s wr, 377 op/s 2024-09-16T00:39:42.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188251 19.08 MB/sec execute 408 sec latency 1651.163 ms 2024-09-16T00:39:43.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188251 19.03 MB/sec execute 409 sec latency 2651.247 ms 2024-09-16T00:39:43.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.233+0000 7f5a84ff9640 1 -- 172.21.15.33:0/815481583 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x556608832d20 con 0x55660894bc70 2024-09-16T00:39:43.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.233+0000 7f5a84ff9640 10 client.28763 handle_command_reply: tid=0 2024-09-16T00:39:43.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa596f740 1 client.28763 shutdown 2024-09-16T00:39:43.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa596f740 1 -- 172.21.15.33:0/815481583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c6080 msgr2=0x5566088d41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:43.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa596f740 1 --2- 172.21.15.33:0/815481583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c6080 0x5566088d41d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0a4a00 tx=0x7f5a8c09c440 comp rx=0 tx=0).stop 2024-09-16T00:39:43.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa596f740 1 -- 172.21.15.33:0/815481583 shutdown_connections 2024-09-16T00:39:43.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa596f740 1 --2- 172.21.15.33:0/815481583 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55660894bc70 0x55660894e090 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:43.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa596f740 1 --2- 172.21.15.33:0/815481583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5566088d0940 0x5566088d4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:43.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa596f740 1 --2- 172.21.15.33:0/815481583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5566088c6080 0x5566088d41d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:43.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5a84ff9640 0 client.28763 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:43.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa596f740 1 --2- 172.21.15.33:0/815481583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5566088c5760 0x5566088d8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:43.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa596f740 1 -- 172.21.15.33:0/815481583 >> 172.21.15.33:0/815481583 conn(0x5566088c1cf0 msgr2=0x5566088cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:43.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa596f740 1 -- 172.21.15.33:0/815481583 shutdown_connections 2024-09-16T00:39:43.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa596f740 1 -- 172.21.15.33:0/815481583 wait complete. 2024-09-16T00:39:43.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa596f740 20 client.28763 trim_cache size 0 max 16384 2024-09-16T00:39:43.235 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:39:43.236 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:39:43.236 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:39:43.236 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 78, 2024-09-16T00:39:43.236 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:39:43.236 INFO:teuthology.orchestra.run.smithi033.stdout: "878947d6": { 2024-09-16T00:39:43.236 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 78, 2024-09-16T00:39:43.236 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:39:43.236 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:39:43.236 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:39:43.236 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:39:43.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 154.4, 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:39:43.237 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:39:43.238 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:39:43.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa3596640 1 -- 172.21.15.33:0/2164876156 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5a880a3240 msgr2=0x7f5a880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:43.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.234+0000 7f5aa3596640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5a880a3240 0x7f5a880a5700 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c069760 tx=0x7f5a94009500 comp rx=0 tx=0).stop 2024-09-16T00:39:43.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.235+0000 7f5aa3596640 1 -- 172.21.15.33:0/2164876156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a9c101440 msgr2=0x7f5a9c1a0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:43.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.235+0000 7f5aa3596640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a9c101440 0x7f5a9c1a0930 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c00e9c0 tx=0x7f5a8c00ee80 comp rx=0 tx=0).stop 2024-09-16T00:39:43.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.235+0000 7f5aa3596640 1 -- 172.21.15.33:0/2164876156 shutdown_connections 2024-09-16T00:39:43.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.235+0000 7f5aa3596640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f5a880a3240 0x7f5a880a5700 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:43.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.235+0000 7f5aa3596640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a9c10e500 0x7f5a9c19a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:43.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.235+0000 7f5aa3596640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a9c101440 0x7f5a9c1a0930 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:43.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.235+0000 7f5aa3596640 1 --2- 172.21.15.33:0/2164876156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a9c100b20 0x7f5a9c1a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:43.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.235+0000 7f5aa3596640 1 -- 172.21.15.33:0/2164876156 >> 172.21.15.33:0/2164876156 conn(0x7f5a9c0fc7d0 msgr2=0x7f5a9c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:43.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.235+0000 7f5aa3596640 1 -- 172.21.15.33:0/2164876156 shutdown_connections 2024-09-16T00:39:43.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:43.235+0000 7f5aa3596640 1 -- 172.21.15.33:0/2164876156 wait complete. 2024-09-16T00:39:43.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:43 smithi033 ceph-mon[31243]: pgmap v1027: 129 pgs: 129 active+clean; 700 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.5 MiB/s wr, 356 op/s 2024-09-16T00:39:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:43 smithi083 ceph-mon[35242]: pgmap v1027: 129 pgs: 129 active+clean; 700 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.5 MiB/s wr, 356 op/s 2024-09-16T00:39:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:43 smithi060 ceph-mon[35185]: pgmap v1027: 129 pgs: 129 active+clean; 700 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.5 MiB/s wr, 356 op/s 2024-09-16T00:39:44.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.98 MB/sec execute 410 sec latency 3651.298 ms 2024-09-16T00:39:44.124 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '878947d6', quiesce took 7.0 seconds. Will release after: 27.4 2024-09-16T00:39:45.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.94 MB/sec execute 411 sec latency 4651.338 ms 2024-09-16T00:39:45.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:45 smithi033 ceph-mon[31243]: pgmap v1028: 129 pgs: 129 active+clean; 741 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.3 MiB/s wr, 425 op/s 2024-09-16T00:39:45.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:45 smithi083 ceph-mon[35242]: pgmap v1028: 129 pgs: 129 active+clean; 741 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.3 MiB/s wr, 425 op/s 2024-09-16T00:39:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:45 smithi060 ceph-mon[35185]: pgmap v1028: 129 pgs: 129 active+clean; 741 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.3 MiB/s wr, 425 op/s 2024-09-16T00:39:46.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.89 MB/sec execute 412 sec latency 5651.379 ms 2024-09-16T00:39:47.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.85 MB/sec execute 413 sec latency 6651.490 ms 2024-09-16T00:39:47.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:47 smithi033 ceph-mon[31243]: pgmap v1029: 129 pgs: 129 active+clean; 750 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 10 MiB/s wr, 491 op/s 2024-09-16T00:39:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:47 smithi083 ceph-mon[35242]: pgmap v1029: 129 pgs: 129 active+clean; 750 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 10 MiB/s wr, 491 op/s 2024-09-16T00:39:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:47 smithi060 ceph-mon[35185]: pgmap v1029: 129 pgs: 129 active+clean; 750 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 10 MiB/s wr, 491 op/s 2024-09-16T00:39:48.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.80 MB/sec execute 414 sec latency 7651.598 ms 2024-09-16T00:39:49.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.76 MB/sec execute 415 sec latency 8651.734 ms 2024-09-16T00:39:49.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:49 smithi033 ceph-mon[31243]: pgmap v1030: 129 pgs: 129 active+clean; 752 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 11 MiB/s wr, 442 op/s 2024-09-16T00:39:49.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:49 smithi083 ceph-mon[35242]: pgmap v1030: 129 pgs: 129 active+clean; 752 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 11 MiB/s wr, 442 op/s 2024-09-16T00:39:49.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:49 smithi060 ceph-mon[35185]: pgmap v1030: 129 pgs: 129 active+clean; 752 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 11 MiB/s wr, 442 op/s 2024-09-16T00:39:50.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.71 MB/sec execute 416 sec latency 9651.811 ms 2024-09-16T00:39:50.772 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:39:51.043 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:51.099 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.67 MB/sec execute 417 sec latency 10651.899 ms 2024-09-16T00:39:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.577+0000 7f7decf8d640 1 -- 172.21.15.33:0/3505084280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7de8102d30 msgr2=0x7f7de8103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.577+0000 7f7decf8d640 1 --2- 172.21.15.33:0/3505084280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7de8102d30 0x7f7de8103110 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009cd0 tx=0x7f7dd802a8e0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.577+0000 7f7decf8d640 1 -- 172.21.15.33:0/3505084280 shutdown_connections 2024-09-16T00:39:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.577+0000 7f7decf8d640 1 --2- 172.21.15.33:0/3505084280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de81082a0 0x7f7de810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.577+0000 7f7decf8d640 1 --2- 172.21.15.33:0/3505084280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de81036e0 0x7f7de8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.577+0000 7f7decf8d640 1 --2- 172.21.15.33:0/3505084280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7de8102d30 0x7f7de8103110 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.577+0000 7f7decf8d640 1 -- 172.21.15.33:0/3505084280 >> 172.21.15.33:0/3505084280 conn(0x7f7de80fc7d0 msgr2=0x7f7de80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:51.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.577+0000 7f7decf8d640 1 -- 172.21.15.33:0/3505084280 shutdown_connections 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.577+0000 7f7decf8d640 1 -- 172.21.15.33:0/3505084280 wait complete. 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.578+0000 7f7decf8d640 1 Processor -- start 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.578+0000 7f7decf8d640 1 -- start start 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.578+0000 7f7decf8d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de8102d30 0x7f7de8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.578+0000 7f7decf8d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7de81036e0 0x7f7de8198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.578+0000 7f7decf8d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de81082a0 0x7f7de819ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.578+0000 7f7decf8d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7de81120a0 con 0x7f7de81036e0 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.578+0000 7f7decf8d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7de8111f20 con 0x7f7de81082a0 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.578+0000 7f7decf8d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7de8112220 con 0x7f7de8102d30 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.578+0000 7f7de5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7de81036e0 0x7f7de8198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7de5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7de81036e0 0x7f7de8198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50946/0 (socket says 172.21.15.33:50946) 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7de5d9b640 1 -- 172.21.15.33:0/282762506 learned_addr learned my addr 172.21.15.33:0/282762506 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:51.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7de659c640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de8102d30 0x7f7de8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7de6d9d640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de81082a0 0x7f7de819ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7de5d9b640 1 -- 172.21.15.33:0/282762506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de8102d30 msgr2=0x7f7de8198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:51.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de8102d30 0x7f7de8198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7de5d9b640 1 -- 172.21.15.33:0/282762506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de81082a0 msgr2=0x7f7de819ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:51.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de81082a0 0x7f7de819ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7de5d9b640 1 -- 172.21.15.33:0/282762506 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de819d4e0 con 0x7f7de81036e0 2024-09-16T00:39:51.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7de6d9d640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de81082a0 0x7f7de819ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:39:51.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7de81036e0 0x7f7de8198a70 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00d750 tx=0x7f7ddc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:51.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7dd77fe640 1 -- 172.21.15.33:0/282762506 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc017070 con 0x7f7de81036e0 2024-09-16T00:39:51.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7dd77fe640 1 -- 172.21.15.33:0/282762506 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ddc005500 con 0x7f7de81036e0 2024-09-16T00:39:51.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7decf8d640 1 -- 172.21.15.33:0/282762506 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de819d7d0 con 0x7f7de81036e0 2024-09-16T00:39:51.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7dd77fe640 1 -- 172.21.15.33:0/282762506 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc004e00 con 0x7f7de81036e0 2024-09-16T00:39:51.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.579+0000 7f7decf8d640 1 -- 172.21.15.33:0/282762506 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de81a50b0 con 0x7f7de81036e0 2024-09-16T00:39:51.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.582+0000 7f7dd77fe640 1 -- 172.21.15.33:0/282762506 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7ddc004fa0 con 0x7f7de81036e0 2024-09-16T00:39:51.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.582+0000 7f7def366740 1 Processor -- start 2024-09-16T00:39:51.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.582+0000 7f7def366740 1 -- start start 2024-09-16T00:39:51.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.582+0000 7f7def366740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b84a0 0x55dd209b8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.582+0000 7f7def366740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 0x55dd209c3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7def366740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209c3680 0x55dd209c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7def366740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dd2091e2e0 con 0x55dd209b8dc0 2024-09-16T00:39:51.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7def366740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55dd208c5ce0 con 0x55dd209b84a0 2024-09-16T00:39:51.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7def366740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dd2091a2a0 con 0x55dd209c3680 2024-09-16T00:39:51.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 0x55dd209c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 0x55dd209c3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50962/0 (socket says 172.21.15.33:50962) 2024-09-16T00:39:51.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de659c640 1 -- 172.21.15.33:0/2000994799 learned_addr learned my addr 172.21.15.33:0/2000994799 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:51.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7dd77fe640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7dd00a32c0 0x7f7dd00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de6d9d640 1 --2- 172.21.15.33:0/2000994799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b84a0 0x55dd209b8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/2000994799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209c3680 0x55dd209c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de6d9d640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7dd00a32c0 0x7f7dd00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de659c640 1 -- 172.21.15.33:0/2000994799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209c3680 msgr2=0x55dd209c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:51.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de659c640 1 --2- 172.21.15.33:0/2000994799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209c3680 0x55dd209c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de659c640 1 -- 172.21.15.33:0/2000994799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b84a0 msgr2=0x55dd209b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:51.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de659c640 1 --2- 172.21.15.33:0/2000994799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b84a0 0x55dd209b8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de659c640 1 -- 172.21.15.33:0/2000994799 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd208aa440 con 0x55dd209b8dc0 2024-09-16T00:39:51.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7dd77fe640 1 -- 172.21.15.33:0/282762506 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7ddc0cf6d0 con 0x7f7de81036e0 2024-09-16T00:39:51.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de6d9d640 1 --2- 172.21.15.33:0/2000994799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b84a0 0x55dd209b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:39:51.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de659c640 1 --2- 172.21.15.33:0/2000994799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 0x55dd209c3140 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009cd0 tx=0x7f7dd80314e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ada1e6bc0dd2fe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:51.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7de6d9d640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7dd00a32c0 0x7f7dd00a5780 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7de0005fd0 tx=0x7f7de0005d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:51.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.583+0000 7f7dd4ff9640 1 -- 172.21.15.33:0/2000994799 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8048070 con 0x55dd209b8dc0 2024-09-16T00:39:51.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.584+0000 7f7dd4ff9640 1 -- 172.21.15.33:0/2000994799 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dd8036070 con 0x55dd209b8dc0 2024-09-16T00:39:51.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.584+0000 7f7dd4ff9640 1 -- 172.21.15.33:0/2000994799 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8004630 con 0x55dd209b8dc0 2024-09-16T00:39:51.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.584+0000 7f7def366740 1 -- 172.21.15.33:0/2000994799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 msgr2=0x55dd209c3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:51.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.584+0000 7f7def366740 1 --2- 172.21.15.33:0/2000994799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 0x55dd209c3140 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009cd0 tx=0x7f7dd80314e0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.584+0000 7f7def366740 1 -- 172.21.15.33:0/2000994799 shutdown_connections 2024-09-16T00:39:51.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.584+0000 7f7def366740 1 --2- 172.21.15.33:0/2000994799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209c3680 0x55dd209c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.584+0000 7f7def366740 1 --2- 172.21.15.33:0/2000994799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 0x55dd209c3140 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.584+0000 7f7def366740 1 --2- 172.21.15.33:0/2000994799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b84a0 0x55dd209b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.584+0000 7f7def366740 1 -- 172.21.15.33:0/2000994799 >> 172.21.15.33:0/2000994799 conn(0x55dd209b4a30 msgr2=0x55dd209b4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:51.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7def366740 1 -- 172.21.15.33:0/2000994799 shutdown_connections 2024-09-16T00:39:51.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7def366740 1 -- 172.21.15.33:0/2000994799 wait complete. 2024-09-16T00:39:51.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7def366740 1 -- start start 2024-09-16T00:39:51.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7def366740 1 Processor -- start 2024-09-16T00:39:51.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7def366740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b84a0 0x55dd209c2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7def366740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 0x55dd209c0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7def366740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209c3680 0x55dd209c0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7def366740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dd209c6250 con 0x55dd209b8dc0 2024-09-16T00:39:51.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7def366740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55dd208d9050 con 0x55dd209c3680 2024-09-16T00:39:51.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7def366740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dd209c63d0 con 0x55dd209b84a0 2024-09-16T00:39:51.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7def366740 10 client.0 _ll_register_callbacks cb 0x55dd20919160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:51.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7de5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 0x55dd209c0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7de5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 0x55dd209c0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50966/0 (socket says 172.21.15.33:50966) 2024-09-16T00:39:51.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7de5d9b640 1 -- 172.21.15.33:0/2746436635 learned_addr learned my addr 172.21.15.33:0/2746436635 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:51.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7de659c640 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b84a0 0x55dd209c2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.585+0000 7f7de6d9d640 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209c3680 0x55dd209c0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7de5d9b640 1 -- 172.21.15.33:0/2746436635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b84a0 msgr2=0x55dd209c2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:51.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b84a0 0x55dd209c2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7de5d9b640 1 -- 172.21.15.33:0/2746436635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209c3680 msgr2=0x55dd209c0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:51.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209c3680 0x55dd209c0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7de5d9b640 1 -- 172.21.15.33:0/2746436635 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd2068f3c0 con 0x55dd209b8dc0 2024-09-16T00:39:51.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7de6d9d640 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209c3680 0x55dd209c0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:39:51.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 0x55dd209c0250 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc09b570 tx=0x7f7ddc066750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5ee341f7b9db66 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7dcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7dcdffb640 1 -- 172.21.15.33:0/2746436635 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc020070 con 0x55dd209b8dc0 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7dcdffb640 1 -- 172.21.15.33:0/2746436635 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7ddc099b70 con 0x55dd209b8dc0 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7def366740 1 -- 172.21.15.33:0/2746436635 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dd8036070 con 0x55dd209b8dc0 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7dcdffb640 1 -- 172.21.15.33:0/2746436635 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7ddc09bf70 con 0x55dd209b8dc0 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7def366740 10 client.19596 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7def366740 1 -- 172.21.15.33:0/2746436635 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dd209d1c20 con 0x55dd209b8dc0 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7dcdffb640 1 -- 172.21.15.33:0/2746436635 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dd8036070 con 0x55dd209b8dc0 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7dcdffb640 1 -- 172.21.15.33:0/2746436635 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7ddc097ee0 con 0x55dd209b8dc0 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7def366740 10 client.19596 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.586+0000 7f7def366740 10 client.19596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.587+0000 7f7def366740 20 client.19596 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:51.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.587+0000 7f7def366740 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dd209d3170 0x55dd209d5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.587+0000 7f7def366740 4 client.19596 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:39:51.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.587+0000 7f7def366740 1 -- 172.21.15.33:0/2746436635 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd2092b850 con 0x55dd209d3170 2024-09-16T00:39:51.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.587+0000 7f7de659c640 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dd209d3170 0x55dd209d5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.587+0000 7f7dcdffb640 10 client.19596 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:51.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.587+0000 7f7de659c640 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dd209d3170 0x55dd209d5560 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1633797fc20df76d server_cookie=8d6f6385e222b35b in_seq=0 out_seq=0 2024-09-16T00:39:51.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7dcdffb640 1 -- 172.21.15.33:0/2746436635 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dd2092b850 con 0x55dd209d3170 2024-09-16T00:39:51.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7dcdffb640 10 client.19596 handle_command_reply: tid=0 2024-09-16T00:39:51.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7def366740 1 client.19596 shutdown 2024-09-16T00:39:51.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7def366740 1 -- 172.21.15.33:0/2746436635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 msgr2=0x55dd209c0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:51.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7def366740 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 0x55dd209c0250 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc09b570 tx=0x7f7ddc066750 comp rx=0 tx=0).stop 2024-09-16T00:39:51.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7def366740 1 -- 172.21.15.33:0/2746436635 shutdown_connections 2024-09-16T00:39:51.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7def366740 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dd209d3170 0x55dd209d5560 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7def366740 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209c3680 0x55dd209c0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7def366740 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b8dc0 0x55dd209c0250 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7dcdffb640 0 client.19596 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:51.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7def366740 1 --2- 172.21.15.33:0/2746436635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b84a0 0x55dd209c2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7def366740 1 -- 172.21.15.33:0/2746436635 >> 172.21.15.33:0/2746436635 conn(0x55dd209b4a30 msgr2=0x55dd209b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:51.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7def366740 1 -- 172.21.15.33:0/2746436635 shutdown_connections 2024-09-16T00:39:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7def366740 1 -- 172.21.15.33:0/2746436635 wait complete. 2024-09-16T00:39:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.588+0000 7f7def366740 20 client.19596 trim_cache size 0 max 16384 2024-09-16T00:39:51.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.605+0000 7f7def366740 1 Processor -- start 2024-09-16T00:39:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.605+0000 7f7def366740 1 -- start start 2024-09-16T00:39:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7def366740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b4a30 0x55dd209cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7def366740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 0x55dd209bb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7def366740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b8dc0 0x55dd209cba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7def366740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7de81b1890 con 0x55dd209b84a0 2024-09-16T00:39:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7def366740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7de81af160 con 0x55dd209b8dc0 2024-09-16T00:39:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7def366740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7de81acfb0 con 0x55dd209b4a30 2024-09-16T00:39:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7de5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 0x55dd209bb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7de5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 0x55dd209bb7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50980/0 (socket says 172.21.15.33:50980) 2024-09-16T00:39:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7de5d9b640 1 -- 172.21.15.33:0/452264154 learned_addr learned my addr 172.21.15.33:0/452264154 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7de6d9d640 1 --2- 172.21.15.33:0/452264154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b8dc0 0x55dd209cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7de659c640 1 --2- 172.21.15.33:0/452264154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b4a30 0x55dd209cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7de5d9b640 1 -- 172.21.15.33:0/452264154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b4a30 msgr2=0x55dd209cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/452264154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b4a30 0x55dd209cf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7de5d9b640 1 -- 172.21.15.33:0/452264154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b8dc0 msgr2=0x55dd209cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/452264154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b8dc0 0x55dd209cba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7de5d9b640 1 -- 172.21.15.33:0/452264154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd209be7a0 con 0x55dd209b84a0 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/452264154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 0x55dd209bb7f0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc099b20 tx=0x7f7ddc09b640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ffe210dca601a84 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7dcffff640 1 -- 172.21.15.33:0/452264154 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc020070 con 0x55dd209b84a0 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7dcffff640 1 -- 172.21.15.33:0/452264154 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ddc09fc20 con 0x55dd209b84a0 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.606+0000 7f7dcffff640 1 -- 172.21.15.33:0/452264154 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc06b140 con 0x55dd209b84a0 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.607+0000 7f7def366740 1 -- 172.21.15.33:0/452264154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 msgr2=0x55dd209bb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:51.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.607+0000 7f7def366740 1 --2- 172.21.15.33:0/452264154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 0x55dd209bb7f0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc099b20 tx=0x7f7ddc09b640 comp rx=0 tx=0).stop 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.607+0000 7f7def366740 1 -- 172.21.15.33:0/452264154 shutdown_connections 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.607+0000 7f7def366740 1 --2- 172.21.15.33:0/452264154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b8dc0 0x55dd209cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.607+0000 7f7def366740 1 --2- 172.21.15.33:0/452264154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 0x55dd209bb7f0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.607+0000 7f7def366740 1 --2- 172.21.15.33:0/452264154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b4a30 0x55dd209cf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.607+0000 7f7def366740 1 -- 172.21.15.33:0/452264154 >> 172.21.15.33:0/452264154 conn(0x55dd209c08f0 msgr2=0x55dd209bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.607+0000 7f7def366740 1 -- 172.21.15.33:0/452264154 shutdown_connections 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.607+0000 7f7def366740 1 -- 172.21.15.33:0/452264154 wait complete. 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7def366740 1 -- start start 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7def366740 1 Processor -- start 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7def366740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b4a30 0x55dd20a35e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7def366740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 0x55dd20a363d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7def366740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b8dc0 0x55dd20a3ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7def366740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7de81ad620 con 0x55dd209b84a0 2024-09-16T00:39:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7def366740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7de81aea90 con 0x55dd209b4a30 2024-09-16T00:39:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7def366740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7de81adcf0 con 0x55dd209b8dc0 2024-09-16T00:39:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7def366740 10 client.0 _ll_register_callbacks cb 0x7f7ddc097450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7de5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 0x55dd20a363d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7de5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 0x55dd20a363d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50986/0 (socket says 172.21.15.33:50986) 2024-09-16T00:39:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7de5d9b640 1 -- 172.21.15.33:0/3391863427 learned_addr learned my addr 172.21.15.33:0/3391863427 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7de6d9d640 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b8dc0 0x55dd20a3ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7de659c640 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b4a30 0x55dd20a35e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7de5d9b640 1 -- 172.21.15.33:0/3391863427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b8dc0 msgr2=0x55dd20a3ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b8dc0 0x55dd20a3ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7de5d9b640 1 -- 172.21.15.33:0/3391863427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b4a30 msgr2=0x55dd20a35e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b4a30 0x55dd20a35e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.608+0000 7f7de5d9b640 1 -- 172.21.15.33:0/3391863427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd20a3b410 con 0x55dd209b84a0 2024-09-16T00:39:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7de5d9b640 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 0x55dd20a363d0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc098240 tx=0x7f7ddc099ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca087a213cf59043 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7dcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7dcdffb640 1 -- 172.21.15.33:0/3391863427 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc020070 con 0x55dd209b84a0 2024-09-16T00:39:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7dcdffb640 1 -- 172.21.15.33:0/3391863427 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7ddc0a4250 con 0x55dd209b84a0 2024-09-16T00:39:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7def366740 1 -- 172.21.15.33:0/3391863427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ddc09fc20 con 0x55dd209b84a0 2024-09-16T00:39:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7dcdffb640 1 -- 172.21.15.33:0/3391863427 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7ddc097b10 con 0x55dd209b84a0 2024-09-16T00:39:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7def366740 10 client.19602 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7def366740 1 -- 172.21.15.33:0/3391863427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dd20a427b0 con 0x55dd209b84a0 2024-09-16T00:39:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7dcdffb640 1 -- 172.21.15.33:0/3391863427 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ddc09fc20 con 0x55dd209b84a0 2024-09-16T00:39:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7dcdffb640 1 -- 172.21.15.33:0/3391863427 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7ddc09cba0 con 0x55dd209b84a0 2024-09-16T00:39:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7def366740 10 client.19602 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7def366740 10 client.19602 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:39:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7def366740 20 client.19602 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7def366740 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dd209c3680 0x55dd20a44c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7def366740 4 client.19602 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:39:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.609+0000 7f7def366740 1 -- 172.21.15.33:0/3391863427 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dd20925d60 con 0x55dd209c3680 2024-09-16T00:39:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7de659c640 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dd209c3680 0x55dd20a44c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7de659c640 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dd209c3680 0x55dd20a44c40 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=198dcc53728c2680 server_cookie=1c74c072783a9770 in_seq=0 out_seq=0 2024-09-16T00:39:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7dcdffb640 10 client.19602 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7dcdffb640 1 -- 172.21.15.33:0/3391863427 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55dd20925d60 con 0x55dd209c3680 2024-09-16T00:39:51.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7dcdffb640 10 client.19602 handle_command_reply: tid=0 2024-09-16T00:39:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7def366740 1 client.19602 shutdown 2024-09-16T00:39:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7def366740 1 -- 172.21.15.33:0/3391863427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 msgr2=0x55dd20a363d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7def366740 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 0x55dd20a363d0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc098240 tx=0x7f7ddc099ce0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7def366740 1 -- 172.21.15.33:0/3391863427 shutdown_connections 2024-09-16T00:39:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7def366740 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55dd209c3680 0x55dd20a44c40 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7def366740 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd209b8dc0 0x55dd20a3ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7def366740 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd209b84a0 0x55dd20a363d0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7def366740 1 --2- 172.21.15.33:0/3391863427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dd209b4a30 0x55dd20a35e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7dcdffb640 0 client.19602 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:51.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.610+0000 7f7def366740 1 -- 172.21.15.33:0/3391863427 >> 172.21.15.33:0/3391863427 conn(0x55dd209c08f0 msgr2=0x55dd209c22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:51.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7def366740 1 -- 172.21.15.33:0/3391863427 shutdown_connections 2024-09-16T00:39:51.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7def366740 1 -- 172.21.15.33:0/3391863427 wait complete. 2024-09-16T00:39:51.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7def366740 20 client.19602 trim_cache size 0 max 16384 2024-09-16T00:39:51.627 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:39:51.627 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:39:51.627 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:39:51.627 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:39:51.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7decf8d640 1 -- 172.21.15.33:0/282762506 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7dd00a32c0 msgr2=0x7f7dd00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:51.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7decf8d640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7dd00a32c0 0x7f7dd00a5780 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7de0005fd0 tx=0x7f7de0005d60 comp rx=0 tx=0).stop 2024-09-16T00:39:51.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7decf8d640 1 -- 172.21.15.33:0/282762506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7de81036e0 msgr2=0x7f7de8198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:51.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7decf8d640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7de81036e0 0x7f7de8198a70 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00d750 tx=0x7f7ddc00dc10 comp rx=0 tx=0).stop 2024-09-16T00:39:51.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7decf8d640 1 -- 172.21.15.33:0/282762506 shutdown_connections 2024-09-16T00:39:51.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7decf8d640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7dd00a32c0 0x7f7dd00a5780 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7decf8d640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de81082a0 0x7f7de819ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7decf8d640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7de81036e0 0x7f7de8198a70 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7decf8d640 1 --2- 172.21.15.33:0/282762506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de8102d30 0x7f7de8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:51.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7decf8d640 1 -- 172.21.15.33:0/282762506 >> 172.21.15.33:0/282762506 conn(0x7f7de80fc7d0 msgr2=0x7f7de8105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:51.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7decf8d640 1 -- 172.21.15.33:0/282762506 shutdown_connections 2024-09-16T00:39:51.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:51.611+0000 7f7decf8d640 1 -- 172.21.15.33:0/282762506 wait complete. 2024-09-16T00:39:51.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:51 smithi033 ceph-mon[31243]: pgmap v1031: 129 pgs: 129 active+clean; 707 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 11 MiB/s wr, 512 op/s 2024-09-16T00:39:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:51 smithi083 ceph-mon[35242]: pgmap v1031: 129 pgs: 129 active+clean; 707 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 11 MiB/s wr, 512 op/s 2024-09-16T00:39:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:51 smithi060 ceph-mon[35185]: pgmap v1031: 129 pgs: 129 active+clean; 707 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 11 MiB/s wr, 512 op/s 2024-09-16T00:39:52.098 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.62 MB/sec execute 418 sec latency 11651.963 ms 2024-09-16T00:39:52.466 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:39:52.466 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:39:52.727 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:53.099 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.58 MB/sec execute 419 sec latency 12652.064 ms 2024-09-16T00:39:53.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.226+0000 7fb6b2643640 1 -- 172.21.15.33:0/2893597641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6ac106f50 msgr2=0x7fb6ac10eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:53.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.226+0000 7fb6b2643640 1 --2- 172.21.15.33:0/2893597641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6ac106f50 0x7fb6ac10eaf0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4009cd0 tx=0x7fb6a402a8e0 comp rx=0 tx=0).stop 2024-09-16T00:39:53.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 -- 172.21.15.33:0/2893597641 shutdown_connections 2024-09-16T00:39:53.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 --2- 172.21.15.33:0/2893597641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6ac106f50 0x7fb6ac10eaf0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:53.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 --2- 172.21.15.33:0/2893597641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6ac100d00 0x7fb6ac106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:53.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 --2- 172.21.15.33:0/2893597641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6ac100350 0x7fb6ac100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:53.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 -- 172.21.15.33:0/2893597641 >> 172.21.15.33:0/2893597641 conn(0x7fb6ac077210 msgr2=0x7fb6ac0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:53.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 -- 172.21.15.33:0/2893597641 shutdown_connections 2024-09-16T00:39:53.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 -- 172.21.15.33:0/2893597641 wait complete. 2024-09-16T00:39:53.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 Processor -- start 2024-09-16T00:39:53.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 -- start start 2024-09-16T00:39:53.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6ac100350 0x7fb6ac198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:53.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6ac100d00 0x7fb6ac1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:53.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6ac106f50 0x7fb6ac19cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:53.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac111fa0 con 0x7fb6ac100d00 2024-09-16T00:39:53.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac111e20 con 0x7fb6ac106f50 2024-09-16T00:39:53.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.227+0000 7fb6b2643640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac112120 con 0x7fb6ac100350 2024-09-16T00:39:53.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6a3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6ac100d00 0x7fb6ac1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:53.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6a3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6ac100d00 0x7fb6ac1989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51008/0 (socket says 172.21.15.33:51008) 2024-09-16T00:39:53.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6a3fff640 1 -- 172.21.15.33:0/3540275275 learned_addr learned my addr 172.21.15.33:0/3540275275 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:53.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6b0be0640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6ac106f50 0x7fb6ac19cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:53.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6abfff640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6ac100350 0x7fb6ac198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:53.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6a3fff640 1 -- 172.21.15.33:0/3540275275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6ac100350 msgr2=0x7fb6ac198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:53.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6a3fff640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6ac100350 0x7fb6ac198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:53.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6a3fff640 1 -- 172.21.15.33:0/3540275275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6ac106f50 msgr2=0x7fb6ac19cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:53.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6a3fff640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6ac106f50 0x7fb6ac19cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:53.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6a3fff640 1 -- 172.21.15.33:0/3540275275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ac19d430 con 0x7fb6ac100d00 2024-09-16T00:39:53.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6b0be0640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6ac106f50 0x7fb6ac19cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:39:53.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6a3fff640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6ac100d00 0x7fb6ac1989c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fb694005fd0 tx=0x7fb69400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:53.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6a9ffb640 1 -- 172.21.15.33:0/3540275275 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb694017070 con 0x7fb6ac100d00 2024-09-16T00:39:53.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6a9ffb640 1 -- 172.21.15.33:0/3540275275 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb694009c30 con 0x7fb6ac100d00 2024-09-16T00:39:53.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.228+0000 7fb6b2643640 1 -- 172.21.15.33:0/3540275275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ac19d720 con 0x7fb6ac100d00 2024-09-16T00:39:53.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.229+0000 7fb6a9ffb640 1 -- 172.21.15.33:0/3540275275 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb694010460 con 0x7fb6ac100d00 2024-09-16T00:39:53.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.229+0000 7fb6b2643640 1 -- 172.21.15.33:0/3540275275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ac1a5000 con 0x7fb6ac100d00 2024-09-16T00:39:53.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.230+0000 7fb6b2643640 1 -- 172.21.15.33:0/3540275275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6ac100730 con 0x7fb6ac100d00 2024-09-16T00:39:53.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.231+0000 7fb6a9ffb640 1 -- 172.21.15.33:0/3540275275 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb694003e70 con 0x7fb6ac100d00 2024-09-16T00:39:53.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.232+0000 7fb6a9ffb640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb6980a32c0 0x7fb6980a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:53.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.232+0000 7fb6a9ffb640 1 -- 172.21.15.33:0/3540275275 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb6940d0120 con 0x7fb6ac100d00 2024-09-16T00:39:53.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.232+0000 7fb6abfff640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb6980a32c0 0x7fb6980a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:53.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.232+0000 7fb6abfff640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb6980a32c0 0x7fb6980a5780 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb69c005fd0 tx=0x7fb69c005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:53.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.235+0000 7fb6a9ffb640 1 -- 172.21.15.33:0/3540275275 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb694020030 con 0x7fb6ac100d00 2024-09-16T00:39:53.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.446+0000 7fb6b2643640 1 -- 172.21.15.33:0/3540275275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6ac1997e0 con 0x7fb6ac100d00 2024-09-16T00:39:53.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.447+0000 7fb6a9ffb640 1 -- 172.21.15.33:0/3540275275 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7fb6ac1997e0 con 0x7fb6ac100d00 2024-09-16T00:39:53.447 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:39:53.448 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:39:53.448 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:39:53.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.449+0000 7fb6b2643640 1 -- 172.21.15.33:0/3540275275 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb6980a32c0 msgr2=0x7fb6980a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:53.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.449+0000 7fb6b2643640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb6980a32c0 0x7fb6980a5780 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb69c005fd0 tx=0x7fb69c005e20 comp rx=0 tx=0).stop 2024-09-16T00:39:53.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.449+0000 7fb6b2643640 1 -- 172.21.15.33:0/3540275275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6ac100d00 msgr2=0x7fb6ac1989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:53.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.449+0000 7fb6b2643640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6ac100d00 0x7fb6ac1989c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fb694005fd0 tx=0x7fb69400dde0 comp rx=0 tx=0).stop 2024-09-16T00:39:53.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.449+0000 7fb6b2643640 1 -- 172.21.15.33:0/3540275275 shutdown_connections 2024-09-16T00:39:53.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.449+0000 7fb6b2643640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb6980a32c0 0x7fb6980a5780 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:53.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.449+0000 7fb6b2643640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6ac106f50 0x7fb6ac19cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:53.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.449+0000 7fb6b2643640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6ac100d00 0x7fb6ac1989c0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:53.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.449+0000 7fb6b2643640 1 --2- 172.21.15.33:0/3540275275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6ac100350 0x7fb6ac198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:53.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.449+0000 7fb6b2643640 1 -- 172.21.15.33:0/3540275275 >> 172.21.15.33:0/3540275275 conn(0x7fb6ac077210 msgr2=0x7fb6ac105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:53.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.449+0000 7fb6b2643640 1 -- 172.21.15.33:0/3540275275 shutdown_connections 2024-09-16T00:39:53.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:53.449+0000 7fb6b2643640 1 -- 172.21.15.33:0/3540275275 wait complete. 2024-09-16T00:39:53.782 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:53 smithi060 ceph-mon[35185]: pgmap v1032: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 11 MiB/s wr, 469 op/s 2024-09-16T00:39:53.783 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:53 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3540275275' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:39:53.807 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:53 smithi033 ceph-mon[31243]: pgmap v1032: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 11 MiB/s wr, 469 op/s 2024-09-16T00:39:53.807 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:53 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3540275275' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:39:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:53 smithi083 ceph-mon[35242]: pgmap v1032: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 11 MiB/s wr, 469 op/s 2024-09-16T00:39:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:53 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3540275275' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:39:54.099 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.53 MB/sec execute 420 sec latency 13652.184 ms 2024-09-16T00:39:54.155 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:39:54.415 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:54.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.912+0000 7fb6520e8640 1 -- 172.21.15.33:0/1727782345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb64c103660 msgr2=0x7fb64c10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:54.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.912+0000 7fb6520e8640 1 --2- 172.21.15.33:0/1727782345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb64c103660 0x7fb64c10bb50 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb640009900 tx=0x7fb64002a480 comp rx=0 tx=0).stop 2024-09-16T00:39:54.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.912+0000 7fb6520e8640 1 -- 172.21.15.33:0/1727782345 shutdown_connections 2024-09-16T00:39:54.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.912+0000 7fb6520e8640 1 --2- 172.21.15.33:0/1727782345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb64c10c170 0x7fb64c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.912+0000 7fb6520e8640 1 --2- 172.21.15.33:0/1727782345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb64c103660 0x7fb64c10bb50 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.912+0000 7fb6520e8640 1 --2- 172.21.15.33:0/1727782345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb64c102d40 0x7fb64c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.912+0000 7fb6520e8640 1 -- 172.21.15.33:0/1727782345 >> 172.21.15.33:0/1727782345 conn(0x7fb64c0fc7a0 msgr2=0x7fb64c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:54.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.912+0000 7fb6520e8640 1 -- 172.21.15.33:0/1727782345 shutdown_connections 2024-09-16T00:39:54.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.912+0000 7fb6520e8640 1 -- 172.21.15.33:0/1727782345 wait complete. 2024-09-16T00:39:54.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.912+0000 7fb6520e8640 1 Processor -- start 2024-09-16T00:39:54.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb6520e8640 1 -- start start 2024-09-16T00:39:54.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb6520e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb64c102d40 0x7fb64c1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb6520e8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb64c103660 0x7fb64c100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb6520e8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb64c10c170 0x7fb64c101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb6520e8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c111f70 con 0x7fb64c102d40 2024-09-16T00:39:54.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb6520e8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c111df0 con 0x7fb64c103660 2024-09-16T00:39:54.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb6520e8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c1120f0 con 0x7fb64c10c170 2024-09-16T00:39:54.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb64b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb64c102d40 0x7fb64c1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb64b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb64c102d40 0x7fb64c1009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51038/0 (socket says 172.21.15.33:51038) 2024-09-16T00:39:54.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb64b7fe640 1 -- 172.21.15.33:0/2073190202 learned_addr learned my addr 172.21.15.33:0/2073190202 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:54.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb64bfff640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb64c10c170 0x7fb64c101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb64affd640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb64c103660 0x7fb64c100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb64b7fe640 1 -- 172.21.15.33:0/2073190202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb64c10c170 msgr2=0x7fb64c101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:54.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb64b7fe640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb64c10c170 0x7fb64c101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb64b7fe640 1 -- 172.21.15.33:0/2073190202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb64c103660 msgr2=0x7fb64c100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:54.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb64b7fe640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb64c103660 0x7fb64c100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb64b7fe640 1 -- 172.21.15.33:0/2073190202 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64c1a6910 con 0x7fb64c102d40 2024-09-16T00:39:54.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb64bfff640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb64c10c170 0x7fb64c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:39:54.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.913+0000 7fb64affd640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb64c103660 0x7fb64c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:39:54.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.914+0000 7fb64b7fe640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb64c102d40 0x7fb64c1009a0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fb63c00d950 tx=0x7fb63c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:54.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.914+0000 7fb648ff9640 1 -- 172.21.15.33:0/2073190202 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c017030 con 0x7fb64c102d40 2024-09-16T00:39:54.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.914+0000 7fb6520e8640 1 -- 172.21.15.33:0/2073190202 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb64c1a6c00 con 0x7fb64c102d40 2024-09-16T00:39:54.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.914+0000 7fb648ff9640 1 -- 172.21.15.33:0/2073190202 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb63c004020 con 0x7fb64c102d40 2024-09-16T00:39:54.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.914+0000 7fb648ff9640 1 -- 172.21.15.33:0/2073190202 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c0047a0 con 0x7fb64c102d40 2024-09-16T00:39:54.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.914+0000 7fb6520e8640 1 -- 172.21.15.33:0/2073190202 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb64c1a7190 con 0x7fb64c102d40 2024-09-16T00:39:54.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.916+0000 7fb648ff9640 1 -- 172.21.15.33:0/2073190202 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb63c0041d0 con 0x7fb64c102d40 2024-09-16T00:39:54.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.916+0000 7fb6544c1740 1 Processor -- start 2024-09-16T00:39:54.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.916+0000 7fb6544c1740 1 -- start start 2024-09-16T00:39:54.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb6544c1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d9b40 0x55b3233d8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb6544c1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d85d0 0x55b3233d8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb6544c1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233e3370 0x55b3233e5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb6544c1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b32333e2f0 con 0x55b3233d9b40 2024-09-16T00:39:54.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb6544c1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55b3232e5cf0 con 0x55b3233d85d0 2024-09-16T00:39:54.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb6544c1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b32333a2b0 con 0x55b3233e3370 2024-09-16T00:39:54.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d9b40 0x55b3233d8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d9b40 0x55b3233d8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51048/0 (socket says 172.21.15.33:51048) 2024-09-16T00:39:54.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64bfff640 1 -- 172.21.15.33:0/2125305314 learned_addr learned my addr 172.21.15.33:0/2125305314 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:54.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb648ff9640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb6340a3240 0x7fb6340a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64affd640 1 --2- 172.21.15.33:0/2125305314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233e3370 0x55b3233e5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64b7fe640 1 --2- 172.21.15.33:0/2125305314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d85d0 0x55b3233d8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64bfff640 1 -- 172.21.15.33:0/2125305314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233e3370 msgr2=0x55b3233e5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:54.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64bfff640 1 --2- 172.21.15.33:0/2125305314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233e3370 0x55b3233e5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64bfff640 1 -- 172.21.15.33:0/2125305314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d85d0 msgr2=0x55b3233d8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:54.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64bfff640 1 --2- 172.21.15.33:0/2125305314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d85d0 0x55b3233d8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64bfff640 1 -- 172.21.15.33:0/2125305314 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3232ca3a0 con 0x55b3233d9b40 2024-09-16T00:39:54.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64b7fe640 1 --2- 172.21.15.33:0/2125305314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d85d0 0x55b3233d8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:39:54.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb648ff9640 1 -- 172.21.15.33:0/2073190202 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb63c0cf880 con 0x7fb64c102d40 2024-09-16T00:39:54.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64bfff640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb6340a3240 0x7fb6340a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.917+0000 7fb64bfff640 1 --2- 172.21.15.33:0/2125305314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d9b40 0x55b3233d8090 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fb644009fd0 tx=0x7fb644010a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345718a5b2cacaf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:54.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb631ffb640 1 -- 172.21.15.33:0/2125305314 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb644019070 con 0x55b3233d9b40 2024-09-16T00:39:54.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb631ffb640 1 -- 172.21.15.33:0/2125305314 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb644022070 con 0x55b3233d9b40 2024-09-16T00:39:54.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb631ffb640 1 -- 172.21.15.33:0/2125305314 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb644026c40 con 0x55b3233d9b40 2024-09-16T00:39:54.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb6544c1740 1 -- 172.21.15.33:0/2125305314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d9b40 msgr2=0x55b3233d8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:54.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb64bfff640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb6340a3240 0x7fb6340a5700 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb644021000 tx=0x7fb64401f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:54.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb6544c1740 1 --2- 172.21.15.33:0/2125305314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d9b40 0x55b3233d8090 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fb644009fd0 tx=0x7fb644010a00 comp rx=0 tx=0).stop 2024-09-16T00:39:54.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb6544c1740 1 -- 172.21.15.33:0/2125305314 shutdown_connections 2024-09-16T00:39:54.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb6544c1740 1 --2- 172.21.15.33:0/2125305314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233e3370 0x55b3233e5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb6544c1740 1 --2- 172.21.15.33:0/2125305314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d85d0 0x55b3233d8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb6544c1740 1 --2- 172.21.15.33:0/2125305314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d9b40 0x55b3233d8090 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb6544c1740 1 -- 172.21.15.33:0/2125305314 >> 172.21.15.33:0/2125305314 conn(0x55b3233a1310 msgr2=0x55b3233d4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:54.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb6544c1740 1 -- 172.21.15.33:0/2125305314 shutdown_connections 2024-09-16T00:39:54.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.918+0000 7fb6544c1740 1 -- 172.21.15.33:0/2125305314 wait complete. 2024-09-16T00:39:54.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.919+0000 7fb6544c1740 1 -- start start 2024-09-16T00:39:54.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.919+0000 7fb6544c1740 1 Processor -- start 2024-09-16T00:39:54.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.919+0000 7fb6544c1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d85d0 0x55b3233e2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.919+0000 7fb6544c1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d9b40 0x55b3233dffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.919+0000 7fb6544c1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233e3370 0x55b3233e05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.919+0000 7fb6544c1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b3233e6070 con 0x55b3233d85d0 2024-09-16T00:39:54.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.919+0000 7fb6544c1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55b3232f9060 con 0x55b3233e3370 2024-09-16T00:39:54.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.919+0000 7fb6544c1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b3233e61f0 con 0x55b3233d9b40 2024-09-16T00:39:54.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.919+0000 7fb6544c1740 10 client.0 _ll_register_callbacks cb 0x55b32304c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:54.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.919+0000 7fb64b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d85d0 0x55b3233e2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.919+0000 7fb64b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d85d0 0x55b3233e2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51064/0 (socket says 172.21.15.33:51064) 2024-09-16T00:39:54.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.919+0000 7fb64b7fe640 1 -- 172.21.15.33:0/555229770 learned_addr learned my addr 172.21.15.33:0/555229770 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:54.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb64bfff640 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233e3370 0x55b3233e05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb64affd640 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d9b40 0x55b3233dffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb64b7fe640 1 -- 172.21.15.33:0/555229770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d9b40 msgr2=0x55b3233dffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:54.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb64b7fe640 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d9b40 0x55b3233dffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb64b7fe640 1 -- 172.21.15.33:0/555229770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233e3370 msgr2=0x55b3233e05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:54.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb64b7fe640 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233e3370 0x55b3233e05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb64b7fe640 1 -- 172.21.15.33:0/555229770 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3230af3c0 con 0x55b3233d85d0 2024-09-16T00:39:54.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb64b7fe640 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d85d0 0x55b3233e2d30 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fb63c098390 tx=0x7fb63c09b500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d270ca898c425098 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:54.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb62affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:54.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb62affd640 1 -- 172.21.15.33:0/555229770 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c09b9c0 con 0x55b3233d85d0 2024-09-16T00:39:54.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb62affd640 1 -- 172.21.15.33:0/555229770 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb63c098030 con 0x55b3233d85d0 2024-09-16T00:39:54.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb6544c1740 1 -- 172.21.15.33:0/555229770 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb644022070 con 0x55b3233d85d0 2024-09-16T00:39:54.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb62affd640 1 -- 172.21.15.33:0/555229770 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb63c0b4050 con 0x55b3233d85d0 2024-09-16T00:39:54.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb6544c1740 10 client.19620 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:54.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb6544c1740 1 -- 172.21.15.33:0/555229770 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b3233f1a40 con 0x55b3233d85d0 2024-09-16T00:39:54.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.920+0000 7fb62affd640 1 -- 172.21.15.33:0/555229770 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb644022070 con 0x55b3233d85d0 2024-09-16T00:39:54.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.921+0000 7fb62affd640 1 -- 172.21.15.33:0/555229770 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb63c09ae10 con 0x55b3233d85d0 2024-09-16T00:39:54.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.921+0000 7fb6544c1740 10 client.19620 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:54.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.921+0000 7fb6544c1740 10 client.19620 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:39:54.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.921+0000 7fb6544c1740 20 client.19620 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:54.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.921+0000 7fb6544c1740 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55b3233f2e50 0x55b3233f5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.921+0000 7fb6544c1740 4 client.19620 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:39:54.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.921+0000 7fb6544c1740 1 -- 172.21.15.33:0/555229770 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b32334b860 con 0x55b3233f2e50 2024-09-16T00:39:54.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.921+0000 7fb64affd640 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55b3233f2e50 0x55b3233f5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.921+0000 7fb62affd640 10 client.19620 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:39:54.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.921+0000 7fb64affd640 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55b3233f2e50 0x55b3233f5240 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=baaaf8fcf2d83ccc server_cookie=5a037296c8a3c794 in_seq=0 out_seq=0 2024-09-16T00:39:54.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb62affd640 1 -- 172.21.15.33:0/555229770 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b32334b860 con 0x55b3233f2e50 2024-09-16T00:39:54.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb62affd640 10 client.19620 handle_command_reply: tid=0 2024-09-16T00:39:54.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb6544c1740 1 client.19620 shutdown 2024-09-16T00:39:54.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb6544c1740 1 -- 172.21.15.33:0/555229770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d85d0 msgr2=0x55b3233e2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:54.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb6544c1740 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d85d0 0x55b3233e2d30 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fb63c098390 tx=0x7fb63c09b500 comp rx=0 tx=0).stop 2024-09-16T00:39:54.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb6544c1740 1 -- 172.21.15.33:0/555229770 shutdown_connections 2024-09-16T00:39:54.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb6544c1740 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55b3233f2e50 0x55b3233f5240 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb6544c1740 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233e3370 0x55b3233e05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb6544c1740 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d9b40 0x55b3233dffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb6544c1740 1 --2- 172.21.15.33:0/555229770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233d85d0 0x55b3233e2d30 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb62affd640 0 client.19620 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:39:54.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb6544c1740 1 -- 172.21.15.33:0/555229770 >> 172.21.15.33:0/555229770 conn(0x55b3233a1310 msgr2=0x55b3233da930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:54.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb6544c1740 1 -- 172.21.15.33:0/555229770 shutdown_connections 2024-09-16T00:39:54.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.922+0000 7fb6544c1740 1 -- 172.21.15.33:0/555229770 wait complete. 2024-09-16T00:39:54.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.923+0000 7fb6544c1740 20 client.19620 trim_cache size 0 max 16384 2024-09-16T00:39:54.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb6544c1740 1 Processor -- start 2024-09-16T00:39:54.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb6544c1740 1 -- start start 2024-09-16T00:39:54.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb6544c1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d85d0 0x55b3233ef310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb6544c1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d9b40 0x55b3233d3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb6544c1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 0x55b3233faad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb6544c1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c1b3b70 con 0x55b3233e3370 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb6544c1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c1b1440 con 0x55b3233d9b40 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb6544c1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c1af290 con 0x55b3233d85d0 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb64bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 0x55b3233faad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb64bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 0x55b3233faad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51076/0 (socket says 172.21.15.33:51076) 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb64bfff640 1 -- 172.21.15.33:0/1545097878 learned_addr learned my addr 172.21.15.33:0/1545097878 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb64b7fe640 1 --2- 172.21.15.33:0/1545097878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d85d0 0x55b3233ef310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb64affd640 1 --2- 172.21.15.33:0/1545097878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d9b40 0x55b3233d3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb64bfff640 1 -- 172.21.15.33:0/1545097878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d85d0 msgr2=0x55b3233ef310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb64bfff640 1 --2- 172.21.15.33:0/1545097878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d85d0 0x55b3233ef310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb64bfff640 1 -- 172.21.15.33:0/1545097878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d9b40 msgr2=0x55b3233d3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:54.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb64bfff640 1 --2- 172.21.15.33:0/1545097878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d9b40 0x55b3233d3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.936+0000 7fb64bfff640 1 -- 172.21.15.33:0/1545097878 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3233f58a0 con 0x55b3233e3370 2024-09-16T00:39:54.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.937+0000 7fb64bfff640 1 --2- 172.21.15.33:0/1545097878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 0x55b3233faad0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fb644012000 tx=0x7fb644028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3269536678204623 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:54.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.937+0000 7fb6317fa640 1 -- 172.21.15.33:0/1545097878 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb644034070 con 0x55b3233e3370 2024-09-16T00:39:54.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.937+0000 7fb6317fa640 1 -- 172.21.15.33:0/1545097878 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb644022040 con 0x55b3233e3370 2024-09-16T00:39:54.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.937+0000 7fb6317fa640 1 -- 172.21.15.33:0/1545097878 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb644019050 con 0x55b3233e3370 2024-09-16T00:39:54.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.938+0000 7fb6544c1740 1 -- 172.21.15.33:0/1545097878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 msgr2=0x55b3233faad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:54.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.938+0000 7fb6544c1740 1 --2- 172.21.15.33:0/1545097878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 0x55b3233faad0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fb644012000 tx=0x7fb644028000 comp rx=0 tx=0).stop 2024-09-16T00:39:54.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.938+0000 7fb6544c1740 1 -- 172.21.15.33:0/1545097878 shutdown_connections 2024-09-16T00:39:54.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.938+0000 7fb6544c1740 1 --2- 172.21.15.33:0/1545097878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 0x55b3233faad0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.938+0000 7fb6544c1740 1 --2- 172.21.15.33:0/1545097878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d9b40 0x55b3233d3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.938+0000 7fb6544c1740 1 --2- 172.21.15.33:0/1545097878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d85d0 0x55b3233ef310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.938+0000 7fb6544c1740 1 -- 172.21.15.33:0/1545097878 >> 172.21.15.33:0/1545097878 conn(0x55b3233a1310 msgr2=0x55b3233d34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:54.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.938+0000 7fb6544c1740 1 -- 172.21.15.33:0/1545097878 shutdown_connections 2024-09-16T00:39:54.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.938+0000 7fb6544c1740 1 -- 172.21.15.33:0/1545097878 wait complete. 2024-09-16T00:39:54.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.938+0000 7fb6544c1740 1 -- start start 2024-09-16T00:39:54.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.938+0000 7fb6544c1740 1 Processor -- start 2024-09-16T00:39:54.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb6544c1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d85d0 0x55b3233e7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb6544c1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d9b40 0x55b3233e7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb6544c1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 0x55b323456a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb6544c1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c1af900 con 0x55b3233e3370 2024-09-16T00:39:54.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb6544c1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c1b0d70 con 0x55b3233d85d0 2024-09-16T00:39:54.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb6544c1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c1affd0 con 0x55b3233d9b40 2024-09-16T00:39:54.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb6544c1740 10 client.0 _ll_register_callbacks cb 0x7fb63c097c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:54.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb64bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 0x55b323456a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb64bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 0x55b323456a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51082/0 (socket says 172.21.15.33:51082) 2024-09-16T00:39:54.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb64bfff640 1 -- 172.21.15.33:0/2900343382 learned_addr learned my addr 172.21.15.33:0/2900343382 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:54.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb64affd640 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d9b40 0x55b3233e7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb64b7fe640 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d85d0 0x55b3233e7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb64bfff640 1 -- 172.21.15.33:0/2900343382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d9b40 msgr2=0x55b3233e7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:54.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb64bfff640 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d9b40 0x55b3233e7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb64bfff640 1 -- 172.21.15.33:0/2900343382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d85d0 msgr2=0x55b3233e7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb64bfff640 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d85d0 0x55b3233e7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb64bfff640 1 -- 172.21.15.33:0/2900343382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3234570b0 con 0x55b3233e3370 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb64bfff640 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 0x55b323456a70 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fb644032040 tx=0x7fb644013f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3ddc97c7c44c01 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb62affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb62affd640 1 -- 172.21.15.33:0/2900343382 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb644034040 con 0x55b3233e3370 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb62affd640 1 -- 172.21.15.33:0/2900343382 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fb6440138b0 con 0x55b3233e3370 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.939+0000 7fb6544c1740 1 -- 172.21.15.33:0/2900343382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb644022040 con 0x55b3233e3370 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.940+0000 7fb62affd640 1 -- 172.21.15.33:0/2900343382 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb644019020 con 0x55b3233e3370 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.940+0000 7fb6544c1740 10 client.19626 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.940+0000 7fb6544c1740 1 -- 172.21.15.33:0/2900343382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b32345e3e0 con 0x55b3233e3370 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.940+0000 7fb62affd640 1 -- 172.21.15.33:0/2900343382 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb644022040 con 0x55b3233e3370 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.940+0000 7fb62affd640 1 -- 172.21.15.33:0/2900343382 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb644042030 con 0x55b3233e3370 2024-09-16T00:39:54.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.940+0000 7fb6544c1740 10 client.19626 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:54.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.940+0000 7fb6544c1740 10 client.19626 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:39:54.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.940+0000 7fb6544c1740 20 client.19626 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:54.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.940+0000 7fb6544c1740 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55b32345e900 0x55b323460cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:54.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.940+0000 7fb6544c1740 4 client.19626 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:39:54.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.940+0000 7fb6544c1740 1 -- 172.21.15.33:0/2900343382 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b323345d70 con 0x55b32345e900 2024-09-16T00:39:54.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.940+0000 7fb64b7fe640 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55b32345e900 0x55b323460cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:54.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb64b7fe640 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55b32345e900 0x55b323460cf0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7806899a8a4f9985 server_cookie=ed4db017d994f47d in_seq=0 out_seq=0 2024-09-16T00:39:54.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb62affd640 10 client.19626 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:39:54.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb62affd640 1 -- 172.21.15.33:0/2900343382 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b323345d70 con 0x55b32345e900 2024-09-16T00:39:54.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb62affd640 10 client.19626 handle_command_reply: tid=0 2024-09-16T00:39:54.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb6544c1740 1 client.19626 shutdown 2024-09-16T00:39:54.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb6544c1740 1 -- 172.21.15.33:0/2900343382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 msgr2=0x55b323456a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:54.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb6544c1740 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 0x55b323456a70 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fb644032040 tx=0x7fb644013f10 comp rx=0 tx=0).stop 2024-09-16T00:39:54.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb6544c1740 1 -- 172.21.15.33:0/2900343382 shutdown_connections 2024-09-16T00:39:54.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb6544c1740 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55b32345e900 0x55b323460cf0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb6544c1740 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b3233e3370 0x55b323456a70 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb6544c1740 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b3233d9b40 0x55b3233e7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb6544c1740 1 --2- 172.21.15.33:0/2900343382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b3233d85d0 0x55b3233e7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb6544c1740 1 -- 172.21.15.33:0/2900343382 >> 172.21.15.33:0/2900343382 conn(0x55b3233a1310 msgr2=0x55b3233d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:54.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb62affd640 0 client.19626 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:39:54.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb6544c1740 1 -- 172.21.15.33:0/2900343382 shutdown_connections 2024-09-16T00:39:54.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb6544c1740 1 -- 172.21.15.33:0/2900343382 wait complete. 2024-09-16T00:39:54.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.941+0000 7fb6544c1740 20 client.19626 trim_cache size 0 max 16384 2024-09-16T00:39:54.962 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:39:54.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.942+0000 7fb6520e8640 1 -- 172.21.15.33:0/2073190202 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb6340a3240 msgr2=0x7fb6340a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:54.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.942+0000 7fb6520e8640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb6340a3240 0x7fb6340a5700 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb644021000 tx=0x7fb64401f000 comp rx=0 tx=0).stop 2024-09-16T00:39:54.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.942+0000 7fb6520e8640 1 -- 172.21.15.33:0/2073190202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb64c102d40 msgr2=0x7fb64c1009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:54.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.942+0000 7fb6520e8640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb64c102d40 0x7fb64c1009a0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fb63c00d950 tx=0x7fb63c00de10 comp rx=0 tx=0).stop 2024-09-16T00:39:54.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.942+0000 7fb6520e8640 1 -- 172.21.15.33:0/2073190202 shutdown_connections 2024-09-16T00:39:54.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.942+0000 7fb6520e8640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fb6340a3240 0x7fb6340a5700 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.942+0000 7fb6520e8640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb64c10c170 0x7fb64c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.942+0000 7fb6520e8640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb64c103660 0x7fb64c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.942+0000 7fb6520e8640 1 --2- 172.21.15.33:0/2073190202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb64c102d40 0x7fb64c1009a0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:54.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.942+0000 7fb6520e8640 1 -- 172.21.15.33:0/2073190202 >> 172.21.15.33:0/2073190202 conn(0x7fb64c0fc7a0 msgr2=0x7fb64c0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:54.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.942+0000 7fb6520e8640 1 -- 172.21.15.33:0/2073190202 shutdown_connections 2024-09-16T00:39:54.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:54.942+0000 7fb6520e8640 1 -- 172.21.15.33:0/2073190202 wait complete. 2024-09-16T00:39:55.099 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.49 MB/sec execute 421 sec latency 14652.224 ms 2024-09-16T00:39:55.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:55 smithi033 ceph-mon[31243]: pgmap v1033: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 10 MiB/s wr, 424 op/s 2024-09-16T00:39:55.779 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:39:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:55 smithi083 ceph-mon[35242]: pgmap v1033: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 10 MiB/s wr, 424 op/s 2024-09-16T00:39:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:55 smithi060 ceph-mon[35185]: pgmap v1033: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 10 MiB/s wr, 424 op/s 2024-09-16T00:39:56.044 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:56.099 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.44 MB/sec execute 422 sec latency 15652.263 ms 2024-09-16T00:39:56.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.531+0000 7f20a3cc7640 1 -- 172.21.15.33:0/1369325942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f209c102790 msgr2=0x7f209c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.531+0000 7f20a3cc7640 1 --2- 172.21.15.33:0/1369325942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f209c102790 0x7f209c1067e0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f2090009900 tx=0x7f209002a410 comp rx=0 tx=0).stop 2024-09-16T00:39:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.531+0000 7f20a3cc7640 1 -- 172.21.15.33:0/1369325942 shutdown_connections 2024-09-16T00:39:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.531+0000 7f20a3cc7640 1 --2- 172.21.15.33:0/1369325942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209c106f10 0x7f209c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.531+0000 7f20a3cc7640 1 --2- 172.21.15.33:0/1369325942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f209c102790 0x7f209c1067e0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.531+0000 7f20a3cc7640 1 --2- 172.21.15.33:0/1369325942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209c101de0 0x7f209c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.531+0000 7f20a3cc7640 1 -- 172.21.15.33:0/1369325942 >> 172.21.15.33:0/1369325942 conn(0x7f209c0fd8c0 msgr2=0x7f209c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.531+0000 7f20a3cc7640 1 -- 172.21.15.33:0/1369325942 shutdown_connections 2024-09-16T00:39:56.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.531+0000 7f20a3cc7640 1 -- 172.21.15.33:0/1369325942 wait complete. 2024-09-16T00:39:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a3cc7640 1 Processor -- start 2024-09-16T00:39:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a3cc7640 1 -- start start 2024-09-16T00:39:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a3cc7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209c101de0 0x7f209c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a3cc7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209c102790 0x7f209c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a3cc7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f209c106f10 0x7f209c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a3cc7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f209c111f70 con 0x7f209c106f10 2024-09-16T00:39:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a3cc7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f209c111df0 con 0x7f209c101de0 2024-09-16T00:39:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a3cc7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f209c1120f0 con 0x7f209c102790 2024-09-16T00:39:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a2264640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f209c106f10 0x7f209c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a2264640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f209c106f10 0x7f209c19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51110/0 (socket says 172.21.15.33:51110) 2024-09-16T00:39:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a2264640 1 -- 172.21.15.33:0/83481624 learned_addr learned my addr 172.21.15.33:0/83481624 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a1262640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209c102790 0x7f209c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.532+0000 7f20a1a63640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209c101de0 0x7f209c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.533+0000 7f20a2264640 1 -- 172.21.15.33:0/83481624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209c102790 msgr2=0x7f209c1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.533+0000 7f20a2264640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209c102790 0x7f209c1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.533+0000 7f20a2264640 1 -- 172.21.15.33:0/83481624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209c101de0 msgr2=0x7f209c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.533+0000 7f20a2264640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209c101de0 0x7f209c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.533+0000 7f20a2264640 1 -- 172.21.15.33:0/83481624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209c19d440 con 0x7f209c106f10 2024-09-16T00:39:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.533+0000 7f20a2264640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f209c106f10 0x7f209c19cd60 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f209400cad0 tx=0x7f209400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.533+0000 7f208effd640 1 -- 172.21.15.33:0/83481624 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209400ae20 con 0x7f209c106f10 2024-09-16T00:39:56.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.533+0000 7f20a3cc7640 1 -- 172.21.15.33:0/83481624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f209c19d730 con 0x7f209c106f10 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.533+0000 7f208effd640 1 -- 172.21.15.33:0/83481624 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2094005020 con 0x7f209c106f10 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.533+0000 7f208effd640 1 -- 172.21.15.33:0/83481624 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209400f6f0 con 0x7f209c106f10 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.533+0000 7f20a3cc7640 1 -- 172.21.15.33:0/83481624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f209c1a5010 con 0x7f209c106f10 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.535+0000 7f208effd640 1 -- 172.21.15.33:0/83481624 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f20940051d0 con 0x7f209c106f10 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a60a0740 1 Processor -- start 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a60a0740 1 -- start start 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a60a0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x56296329f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a60a0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 0x56296329fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a60a0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632aa370 0x5629632ac760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a60a0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5629632052f0 con 0x5629632a0b40 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a60a0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5629631accf0 con 0x56296329f5d0 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a60a0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5629632012b0 con 0x5629632aa370 2024-09-16T00:39:56.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a1262640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x56296329f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a1262640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x56296329f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51120/0 (socket says 172.21.15.33:51120) 2024-09-16T00:39:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a1262640 1 -- 172.21.15.33:0/1302592040 learned_addr learned my addr 172.21.15.33:0/1302592040 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a1a63640 1 --2- 172.21.15.33:0/1302592040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 0x56296329fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f208effd640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f20880a3240 0x7f20880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.536+0000 7f20a1262640 1 --2- 172.21.15.33:0/1302592040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632aa370 0x5629632ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.537+0000 7f20a2264640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f20880a3240 0x7f20880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.537+0000 7f20a1262640 1 -- 172.21.15.33:0/1302592040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632aa370 msgr2=0x5629632ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.537+0000 7f20a1262640 1 --2- 172.21.15.33:0/1302592040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632aa370 0x5629632ac760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.537+0000 7f20a1262640 1 -- 172.21.15.33:0/1302592040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 msgr2=0x56296329fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.537+0000 7f20a1262640 1 --2- 172.21.15.33:0/1302592040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 0x56296329fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.537+0000 7f208effd640 1 -- 172.21.15.33:0/83481624 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f20940cfd10 con 0x7f209c106f10 2024-09-16T00:39:56.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.537+0000 7f20a1262640 1 -- 172.21.15.33:0/1302592040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629631913a0 con 0x5629632a0b40 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.537+0000 7f20a1262640 1 --2- 172.21.15.33:0/1302592040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x56296329f090 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f20900096c0 tx=0x7f2090000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77a6cc88f11cb08 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.537+0000 7f20a2264640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f20880a3240 0x7f20880a5700 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f209409bef0 tx=0x7f2094005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.537+0000 7f2087fff640 1 -- 172.21.15.33:0/1302592040 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2090044030 con 0x5629632a0b40 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.537+0000 7f2087fff640 1 -- 172.21.15.33:0/1302592040 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2090005020 con 0x5629632a0b40 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.537+0000 7f2087fff640 1 -- 172.21.15.33:0/1302592040 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2090035040 con 0x5629632a0b40 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.538+0000 7f20a60a0740 1 -- 172.21.15.33:0/1302592040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 msgr2=0x56296329f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.538+0000 7f20a60a0740 1 --2- 172.21.15.33:0/1302592040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x56296329f090 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f20900096c0 tx=0x7f2090000bc0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.538+0000 7f20a60a0740 1 -- 172.21.15.33:0/1302592040 shutdown_connections 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.538+0000 7f20a60a0740 1 --2- 172.21.15.33:0/1302592040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632aa370 0x5629632ac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.538+0000 7f20a60a0740 1 --2- 172.21.15.33:0/1302592040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 0x56296329fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.538+0000 7f20a60a0740 1 --2- 172.21.15.33:0/1302592040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x56296329f090 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.538+0000 7f20a60a0740 1 -- 172.21.15.33:0/1302592040 >> 172.21.15.33:0/1302592040 conn(0x562963268310 msgr2=0x56296329bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.538+0000 7f20a60a0740 1 -- 172.21.15.33:0/1302592040 shutdown_connections 2024-09-16T00:39:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.538+0000 7f20a60a0740 1 -- 172.21.15.33:0/1302592040 wait complete. 2024-09-16T00:39:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.538+0000 7f20a60a0740 1 -- start start 2024-09-16T00:39:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.538+0000 7f20a60a0740 1 Processor -- start 2024-09-16T00:39:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a60a0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56296329f5d0 0x5629632a9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a60a0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x5629632a6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a60a0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629632aa370 0x5629632a7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a60a0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5629632ad010 con 0x5629632a0b40 2024-09-16T00:39:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a60a0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5629631c0060 con 0x5629632aa370 2024-09-16T00:39:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a60a0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5629632ad190 con 0x56296329f5d0 2024-09-16T00:39:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a60a0740 10 client.0 _ll_register_callbacks cb 0x562962f138c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:56.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a1a63640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x5629632a6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a1a63640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x5629632a6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51126/0 (socket says 172.21.15.33:51126) 2024-09-16T00:39:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a1a63640 1 -- 172.21.15.33:0/199962058 learned_addr learned my addr 172.21.15.33:0/199962058 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a1262640 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56296329f5d0 0x5629632a9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a1262640 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629632aa370 0x5629632a7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a1a63640 1 -- 172.21.15.33:0/199962058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56296329f5d0 msgr2=0x5629632a9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a1a63640 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56296329f5d0 0x5629632a9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a1a63640 1 -- 172.21.15.33:0/199962058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629632aa370 msgr2=0x5629632a7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a1a63640 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629632aa370 0x5629632a7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a1a63640 1 -- 172.21.15.33:0/199962058 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562962f763c0 con 0x5629632a0b40 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a1262640 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629632aa370 0x5629632a7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20a1a63640 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x5629632a6fb0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f209800bad0 tx=0x7f209800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4914e8c9d32bb84e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20857fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.539+0000 7f20857fa640 1 -- 172.21.15.33:0/199962058 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2098005020 con 0x5629632a0b40 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20857fa640 1 -- 172.21.15.33:0/199962058 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f20980051c0 con 0x5629632a0b40 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20a60a0740 1 -- 172.21.15.33:0/199962058 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2090005020 con 0x5629632a0b40 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20857fa640 1 -- 172.21.15.33:0/199962058 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2098005400 con 0x5629632a0b40 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20a60a0740 10 client.19638 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20a60a0740 1 -- 172.21.15.33:0/199962058 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5629632b8aa0 con 0x5629632a0b40 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20857fa640 1 -- 172.21.15.33:0/199962058 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2090005020 con 0x5629632a0b40 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20857fa640 1 -- 172.21.15.33:0/199962058 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f209801b740 con 0x5629632a0b40 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20a60a0740 10 client.19638 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:56.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20a60a0740 10 client.19638 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:39:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20a60a0740 20 client.19638 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20a60a0740 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5629632b9f60 0x5629632bc350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20a60a0740 4 client.19638 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:39:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.540+0000 7f20a60a0740 1 -- 172.21.15.33:0/199962058 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562963212860 con 0x5629632b9f60 2024-09-16T00:39:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.541+0000 7f20a1262640 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5629632b9f60 0x5629632bc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.541+0000 7f20a1262640 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5629632b9f60 0x5629632bc350 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6e0a9dd33dd6511c server_cookie=a9774c4b88896b58 in_seq=0 out_seq=0 2024-09-16T00:39:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.541+0000 7f20857fa640 10 client.19638 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:39:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20857fa640 1 -- 172.21.15.33:0/199962058 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562963212860 con 0x5629632b9f60 2024-09-16T00:39:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20857fa640 10 client.19638 handle_command_reply: tid=0 2024-09-16T00:39:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20a60a0740 1 client.19638 shutdown 2024-09-16T00:39:56.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20a60a0740 1 -- 172.21.15.33:0/199962058 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 msgr2=0x5629632a6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20a60a0740 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x5629632a6fb0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f209800bad0 tx=0x7f209800bed0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20a60a0740 1 -- 172.21.15.33:0/199962058 shutdown_connections 2024-09-16T00:39:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20a60a0740 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x5629632b9f60 0x5629632bc350 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20a60a0740 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5629632aa370 0x5629632a7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20a60a0740 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x5629632a6fb0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20a60a0740 1 --2- 172.21.15.33:0/199962058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56296329f5d0 0x5629632a9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20857fa640 0 client.19638 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:39:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20a60a0740 1 -- 172.21.15.33:0/199962058 >> 172.21.15.33:0/199962058 conn(0x562963268310 msgr2=0x5629632a1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20a60a0740 1 -- 172.21.15.33:0/199962058 shutdown_connections 2024-09-16T00:39:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20a60a0740 1 -- 172.21.15.33:0/199962058 wait complete. 2024-09-16T00:39:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.542+0000 7f20a60a0740 20 client.19638 trim_cache size 0 max 16384 2024-09-16T00:39:56.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a60a0740 1 Processor -- start 2024-09-16T00:39:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a60a0740 1 -- start start 2024-09-16T00:39:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a60a0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 0x5629632b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a60a0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x5629632a6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a60a0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632aa370 0x5629632c1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a60a0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f209c1b19a0 con 0x5629632a0b40 2024-09-16T00:39:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a60a0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f209c1af270 con 0x56296329f5d0 2024-09-16T00:39:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a60a0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f209c1ad0c0 con 0x5629632aa370 2024-09-16T00:39:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a1a63640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x5629632a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a1a63640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x5629632a6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51134/0 (socket says 172.21.15.33:51134) 2024-09-16T00:39:56.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a1a63640 1 -- 172.21.15.33:0/3759516471 learned_addr learned my addr 172.21.15.33:0/3759516471 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a1a63640 1 -- 172.21.15.33:0/3759516471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632aa370 msgr2=0x5629632c1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a1a63640 1 --2- 172.21.15.33:0/3759516471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632aa370 0x5629632c1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a1a63640 1 -- 172.21.15.33:0/3759516471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 msgr2=0x5629632b62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a1262640 1 --2- 172.21.15.33:0/3759516471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 0x5629632b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a1a63640 1 --2- 172.21.15.33:0/3759516471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 0x5629632b62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.555+0000 7f20a1a63640 1 -- 172.21.15.33:0/3759516471 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56296329c960 con 0x5629632a0b40 2024-09-16T00:39:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.556+0000 7f20a1a63640 1 --2- 172.21.15.33:0/3759516471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x5629632a6870 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f2098000c00 tx=0x7f2098004460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d58a6c9f8fb4ccc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.556+0000 7f20877fe640 1 -- 172.21.15.33:0/3759516471 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209802d020 con 0x5629632a0b40 2024-09-16T00:39:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.556+0000 7f20877fe640 1 -- 172.21.15.33:0/3759516471 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f209800b400 con 0x5629632a0b40 2024-09-16T00:39:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.556+0000 7f20877fe640 1 -- 172.21.15.33:0/3759516471 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2098011ae0 con 0x5629632a0b40 2024-09-16T00:39:56.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.556+0000 7f20a60a0740 1 -- 172.21.15.33:0/3759516471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 msgr2=0x5629632a6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.556+0000 7f20a60a0740 1 --2- 172.21.15.33:0/3759516471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x5629632a6870 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f2098000c00 tx=0x7f2098004460 comp rx=0 tx=0).stop 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 -- 172.21.15.33:0/3759516471 shutdown_connections 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 --2- 172.21.15.33:0/3759516471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632aa370 0x5629632c1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 --2- 172.21.15.33:0/3759516471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632a0b40 0x5629632a6870 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 --2- 172.21.15.33:0/3759516471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 0x5629632b62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 -- 172.21.15.33:0/3759516471 >> 172.21.15.33:0/3759516471 conn(0x562963268310 msgr2=0x5629632bbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 -- 172.21.15.33:0/3759516471 shutdown_connections 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 -- 172.21.15.33:0/3759516471 wait complete. 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 -- start start 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 Processor -- start 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 0x5629632ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632a0b40 0x5629632add00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632aa370 0x562963322b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f209c1ad730 con 0x5629632aa370 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f209c1aeba0 con 0x56296329f5d0 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.557+0000 7f20a60a0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f209c1ade00 con 0x5629632a0b40 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a60a0740 10 client.0 _ll_register_callbacks cb 0x7f20980040a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a1262640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632aa370 0x562963322b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a1262640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632aa370 0x562963322b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51150/0 (socket says 172.21.15.33:51150) 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a1262640 1 -- 172.21.15.33:0/3287468782 learned_addr learned my addr 172.21.15.33:0/3287468782 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a1262640 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 0x5629632ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a1a63640 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632a0b40 0x5629632add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a1262640 1 -- 172.21.15.33:0/3287468782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632a0b40 msgr2=0x5629632add00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a1262640 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632a0b40 0x5629632add00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a1262640 1 -- 172.21.15.33:0/3287468782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 msgr2=0x5629632ad7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:56.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a1262640 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 0x5629632ad7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a1262640 1 -- 172.21.15.33:0/3287468782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562963323080 con 0x5629632aa370 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a1262640 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632aa370 0x562963322b40 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f2090009340 tx=0x7f2090000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7cc54e310836bed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20857fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20857fa640 1 -- 172.21.15.33:0/3287468782 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2090035040 con 0x5629632aa370 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20857fa640 1 -- 172.21.15.33:0/3287468782 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2090004040 con 0x5629632aa370 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a60a0740 1 -- 172.21.15.33:0/3287468782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f209800b400 con 0x5629632aa370 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20857fa640 1 -- 172.21.15.33:0/3287468782 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2090004240 con 0x5629632aa370 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a60a0740 10 client.19644 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.558+0000 7f20a60a0740 1 -- 172.21.15.33:0/3287468782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562963324250 con 0x5629632aa370 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.559+0000 7f20857fa640 1 -- 172.21.15.33:0/3287468782 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f209800b400 con 0x5629632aa370 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.559+0000 7f20857fa640 1 -- 172.21.15.33:0/3287468782 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2090004750 con 0x5629632aa370 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.559+0000 7f20a60a0740 10 client.19644 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.559+0000 7f20a60a0740 10 client.19644 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.559+0000 7f20a60a0740 20 client.19644 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.559+0000 7f20a60a0740 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x562963324530 0x562963326920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.559+0000 7f20a60a0740 4 client.19644 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.559+0000 7f20a60a0740 1 -- 172.21.15.33:0/3287468782 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56296320cd70 con 0x562963324530 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.559+0000 7f20a1a63640 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x562963324530 0x562963326920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.559+0000 7f20a1a63640 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x562963324530 0x562963326920 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f05aed7a01dd662 server_cookie=ea463fffab1200cc in_seq=0 out_seq=0 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.559+0000 7f20857fa640 10 client.19644 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20857fa640 1 -- 172.21.15.33:0/3287468782 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56296320cd70 con 0x562963324530 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20857fa640 10 client.19644 handle_command_reply: tid=0 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20a60a0740 1 client.19644 shutdown 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20a60a0740 1 -- 172.21.15.33:0/3287468782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632aa370 msgr2=0x562963322b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20a60a0740 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632aa370 0x562963322b40 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f2090009340 tx=0x7f2090000bc0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20a60a0740 1 -- 172.21.15.33:0/3287468782 shutdown_connections 2024-09-16T00:39:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20a60a0740 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629632aa370 0x562963322b40 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20a60a0740 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629632a0b40 0x5629632add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20a60a0740 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x562963324530 0x562963326920 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20a60a0740 1 --2- 172.21.15.33:0/3287468782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56296329f5d0 0x5629632ad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20857fa640 0 client.19644 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:39:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20a60a0740 1 -- 172.21.15.33:0/3287468782 >> 172.21.15.33:0/3287468782 conn(0x562963268310 msgr2=0x5629632a9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20a60a0740 1 -- 172.21.15.33:0/3287468782 shutdown_connections 2024-09-16T00:39:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20a60a0740 1 -- 172.21.15.33:0/3287468782 wait complete. 2024-09-16T00:39:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.560+0000 7f20a60a0740 20 client.19644 trim_cache size 0 max 16384 2024-09-16T00:39:56.579 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:39:56.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.561+0000 7f20a3cc7640 1 -- 172.21.15.33:0/83481624 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f20880a3240 msgr2=0x7f20880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:56.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.561+0000 7f20a3cc7640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f20880a3240 0x7f20880a5700 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f209409bef0 tx=0x7f2094005960 comp rx=0 tx=0).stop 2024-09-16T00:39:56.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.561+0000 7f20a3cc7640 1 -- 172.21.15.33:0/83481624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f209c106f10 msgr2=0x7f209c19cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:56.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.561+0000 7f20a3cc7640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f209c106f10 0x7f209c19cd60 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f209400cad0 tx=0x7f209400cf90 comp rx=0 tx=0).stop 2024-09-16T00:39:56.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.561+0000 7f20a3cc7640 1 -- 172.21.15.33:0/83481624 shutdown_connections 2024-09-16T00:39:56.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.561+0000 7f20a3cc7640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f20880a3240 0x7f20880a5700 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.561+0000 7f20a3cc7640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f209c106f10 0x7f209c19cd60 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.561+0000 7f20a3cc7640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209c102790 0x7f209c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.561+0000 7f20a3cc7640 1 --2- 172.21.15.33:0/83481624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209c101de0 0x7f209c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:56.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.561+0000 7f20a3cc7640 1 -- 172.21.15.33:0/83481624 >> 172.21.15.33:0/83481624 conn(0x7f209c0fd8c0 msgr2=0x7f209c0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:56.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.561+0000 7f20a3cc7640 1 -- 172.21.15.33:0/83481624 shutdown_connections 2024-09-16T00:39:56.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:56.561+0000 7f20a3cc7640 1 -- 172.21.15.33:0/83481624 wait complete. 2024-09-16T00:39:57.099 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.40 MB/sec execute 423 sec latency 16652.393 ms 2024-09-16T00:39:57.464 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:39:57.730 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:39:57.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:57 smithi033 ceph-mon[31243]: pgmap v1034: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 5.8 MiB/s wr, 272 op/s 2024-09-16T00:39:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:57 smithi083 ceph-mon[35242]: pgmap v1034: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 5.8 MiB/s wr, 272 op/s 2024-09-16T00:39:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:57 smithi060 ceph-mon[35185]: pgmap v1034: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 5.8 MiB/s wr, 272 op/s 2024-09-16T00:39:58.099 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.36 MB/sec execute 424 sec latency 17652.473 ms 2024-09-16T00:39:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.256+0000 7f3a7d47e640 1 -- 172.21.15.33:0/4014856119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a78102790 msgr2=0x7f3a781067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.256+0000 7f3a7d47e640 1 --2- 172.21.15.33:0/4014856119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a78102790 0x7f3a781067e0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009900 tx=0x7f3a6002a410 comp rx=0 tx=0).stop 2024-09-16T00:39:58.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.256+0000 7f3a7d47e640 1 -- 172.21.15.33:0/4014856119 shutdown_connections 2024-09-16T00:39:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.256+0000 7f3a7d47e640 1 --2- 172.21.15.33:0/4014856119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a78106f10 0x7f3a7810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.256+0000 7f3a7d47e640 1 --2- 172.21.15.33:0/4014856119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a78102790 0x7f3a781067e0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.256+0000 7f3a7d47e640 1 --2- 172.21.15.33:0/4014856119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a78101de0 0x7f3a781021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.256+0000 7f3a7d47e640 1 -- 172.21.15.33:0/4014856119 >> 172.21.15.33:0/4014856119 conn(0x7f3a780fd8c0 msgr2=0x7f3a780ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a7d47e640 1 -- 172.21.15.33:0/4014856119 shutdown_connections 2024-09-16T00:39:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a7d47e640 1 -- 172.21.15.33:0/4014856119 wait complete. 2024-09-16T00:39:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a7d47e640 1 Processor -- start 2024-09-16T00:39:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a7d47e640 1 -- start start 2024-09-16T00:39:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a7d47e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a78101de0 0x7f3a78198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a7d47e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a78102790 0x7f3a78198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a7d47e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a78106f10 0x7f3a7819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a7d47e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a78111f30 con 0x7f3a78102790 2024-09-16T00:39:58.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a7d47e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3a78111db0 con 0x7f3a78101de0 2024-09-16T00:39:58.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a7d47e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3a781120b0 con 0x7f3a78106f10 2024-09-16T00:39:58.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a6ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a78102790 0x7f3a78198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a6ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a78102790 0x7f3a78198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51170/0 (socket says 172.21.15.33:51170) 2024-09-16T00:39:58.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.257+0000 7f3a6ffff640 1 -- 172.21.15.33:0/1830697653 learned_addr learned my addr 172.21.15.33:0/1830697653 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:58.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a78101de0 0x7f3a78198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a777fe640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a78106f10 0x7f3a7819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a6ffff640 1 -- 172.21.15.33:0/1830697653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a78106f10 msgr2=0x7f3a7819cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:58.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a6ffff640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a78106f10 0x7f3a7819cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a6ffff640 1 -- 172.21.15.33:0/1830697653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a78101de0 msgr2=0x7f3a78198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a6ffff640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a78101de0 0x7f3a78198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a6ffff640 1 -- 172.21.15.33:0/1830697653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a7819d3e0 con 0x7f3a78102790 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a6ffff640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a78102790 0x7f3a78198970 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f3a60006010 tx=0x7f3a600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a74ff9640 1 -- 172.21.15.33:0/1830697653 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6003f030 con 0x7f3a78102790 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a7d47e640 1 -- 172.21.15.33:0/1830697653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a7819d670 con 0x7f3a78102790 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a74ff9640 1 -- 172.21.15.33:0/1830697653 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a60003ce0 con 0x7f3a78102790 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a74ff9640 1 -- 172.21.15.33:0/1830697653 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a60035070 con 0x7f3a78102790 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.258+0000 7f3a7d47e640 1 -- 172.21.15.33:0/1830697653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a781a4fb0 con 0x7f3a78102790 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.260+0000 7f3a74ff9640 1 -- 172.21.15.33:0/1830697653 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3a60042030 con 0x7f3a78102790 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.261+0000 7f3a7f857740 1 Processor -- start 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.261+0000 7f3a7f857740 1 -- start start 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.261+0000 7f3a7f857740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216cf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.261+0000 7f3a7f857740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.261+0000 7f3a7f857740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216de780 0x5564216e0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.261+0000 7f3a7f857740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5564216352f0 con 0x5564216cf5d0 2024-09-16T00:39:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.261+0000 7f3a7f857740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5564215dccf0 con 0x5564216de780 2024-09-16T00:39:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.261+0000 7f3a7f857740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5564216312b0 con 0x5564216d0b40 2024-09-16T00:39:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.261+0000 7f3a76ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.261+0000 7f3a76ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216cfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51182/0 (socket says 172.21.15.33:51182) 2024-09-16T00:39:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.261+0000 7f3a76ffd640 1 -- 172.21.15.33:0/3647680267 learned_addr learned my addr 172.21.15.33:0/3647680267 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a777fe640 1 --2- 172.21.15.33:0/3647680267 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a6ffff640 1 --2- 172.21.15.33:0/3647680267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216de780 0x5564216e0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a74ff9640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3a640a32c0 0x7f3a640a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a76ffd640 1 -- 172.21.15.33:0/3647680267 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 msgr2=0x5564216cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:58.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/3647680267 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216cf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a76ffd640 1 -- 172.21.15.33:0/3647680267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216de780 msgr2=0x5564216e0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/3647680267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216de780 0x5564216e0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a76ffd640 1 -- 172.21.15.33:0/3647680267 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564215c13a0 con 0x5564216cf5d0 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a6ffff640 1 --2- 172.21.15.33:0/3647680267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216de780 0x5564216e0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a777fe640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3a640a32c0 0x7f3a640a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a74ff9640 1 -- 172.21.15.33:0/1830697653 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3a600f0270 con 0x7f3a78102790 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/3647680267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216cfa10 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800bd10 tx=0x7f3a680107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e461700de08ec77 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a6d7fa640 1 -- 172.21.15.33:0/3647680267 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a68004030 con 0x5564216cf5d0 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a6d7fa640 1 -- 172.21.15.33:0/3647680267 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a680041d0 con 0x5564216cf5d0 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.262+0000 7f3a6d7fa640 1 -- 172.21.15.33:0/3647680267 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a68011820 con 0x5564216cf5d0 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.263+0000 7f3a7f857740 1 -- 172.21.15.33:0/3647680267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 msgr2=0x5564216cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.263+0000 7f3a7f857740 1 --2- 172.21.15.33:0/3647680267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216cfa10 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800bd10 tx=0x7f3a680107a0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.263+0000 7f3a777fe640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3a640a32c0 0x7f3a640a5780 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3a70005fd0 tx=0x7f3a70004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:58.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.263+0000 7f3a7f857740 1 -- 172.21.15.33:0/3647680267 shutdown_connections 2024-09-16T00:39:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.263+0000 7f3a7f857740 1 --2- 172.21.15.33:0/3647680267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216de780 0x5564216e0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.263+0000 7f3a7f857740 1 --2- 172.21.15.33:0/3647680267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216cfa10 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.263+0000 7f3a7f857740 1 --2- 172.21.15.33:0/3647680267 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216cf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.263+0000 7f3a7f857740 1 -- 172.21.15.33:0/3647680267 >> 172.21.15.33:0/3647680267 conn(0x556421698310 msgr2=0x5564216cbc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.263+0000 7f3a7f857740 1 -- 172.21.15.33:0/3647680267 shutdown_connections 2024-09-16T00:39:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.263+0000 7f3a7f857740 1 -- 172.21.15.33:0/3647680267 wait complete. 2024-09-16T00:39:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.263+0000 7f3a7f857740 1 -- start start 2024-09-16T00:39:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.263+0000 7f3a7f857740 1 Processor -- start 2024-09-16T00:39:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a7f857740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216cf5d0 0x5564216de190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a7f857740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216db440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a7f857740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216de780 0x5564216dba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a7f857740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5564216e1480 con 0x5564216de780 2024-09-16T00:39:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a7f857740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5564215f0060 con 0x5564216cf5d0 2024-09-16T00:39:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a7f857740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5564216e1600 con 0x5564216d0b40 2024-09-16T00:39:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a7f857740 10 client.0 _ll_register_callbacks cb 0x5564213438c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216de780 0x5564216dba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216de780 0x5564216dba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51198/0 (socket says 172.21.15.33:51198) 2024-09-16T00:39:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a777fe640 1 -- 172.21.15.33:0/2457116144 learned_addr learned my addr 172.21.15.33:0/2457116144 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a6ffff640 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216db440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216cf5d0 0x5564216de190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a777fe640 1 -- 172.21.15.33:0/2457116144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 msgr2=0x5564216db440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a777fe640 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216db440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a777fe640 1 -- 172.21.15.33:0/2457116144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216cf5d0 msgr2=0x5564216de190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:58.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a777fe640 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216cf5d0 0x5564216de190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a777fe640 1 -- 172.21.15.33:0/2457116144 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564213a63c0 con 0x5564216de780 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a777fe640 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216de780 0x5564216dba50 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f3a70013720 tx=0x7f3a70004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ebbf0150e2fce7d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a5e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a5e7fc640 1 -- 172.21.15.33:0/2457116144 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7001c960 con 0x5564216de780 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a5e7fc640 1 -- 172.21.15.33:0/2457116144 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3a7001cb00 con 0x5564216de780 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.264+0000 7f3a7f857740 1 -- 172.21.15.33:0/2457116144 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a680041d0 con 0x5564216de780 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.265+0000 7f3a5e7fc640 1 -- 172.21.15.33:0/2457116144 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3a70030050 con 0x5564216de780 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.265+0000 7f3a7f857740 10 client.19656 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.265+0000 7f3a7f857740 1 -- 172.21.15.33:0/2457116144 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5564216ecf20 con 0x5564216de780 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.265+0000 7f3a5e7fc640 1 -- 172.21.15.33:0/2457116144 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a680041d0 con 0x5564216de780 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.265+0000 7f3a5e7fc640 1 -- 172.21.15.33:0/2457116144 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f3a70019030 con 0x5564216de780 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.265+0000 7f3a7f857740 10 client.19656 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:58.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.265+0000 7f3a7f857740 10 client.19656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:39:58.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.265+0000 7f3a7f857740 20 client.19656 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:58.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.265+0000 7f3a7f857740 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5564216ee3e0 0x5564216f0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.265+0000 7f3a7f857740 4 client.19656 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:39:58.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.265+0000 7f3a7f857740 1 -- 172.21.15.33:0/2457116144 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556421642860 con 0x5564216ee3e0 2024-09-16T00:39:58.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.265+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5564216ee3e0 0x5564216f0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.266+0000 7f3a5e7fc640 10 client.19656 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:58.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.266+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5564216ee3e0 0x5564216f0830 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=825ac63c21b05299 server_cookie=57f9cb26c12b0f1d in_seq=0 out_seq=0 2024-09-16T00:39:58.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.266+0000 7f3a5e7fc640 1 -- 172.21.15.33:0/2457116144 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556421642860 con 0x5564216ee3e0 2024-09-16T00:39:58.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.266+0000 7f3a5e7fc640 10 client.19656 handle_command_reply: tid=0 2024-09-16T00:39:58.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.266+0000 7f3a7f857740 1 client.19656 shutdown 2024-09-16T00:39:58.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.267+0000 7f3a7f857740 1 -- 172.21.15.33:0/2457116144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216de780 msgr2=0x5564216dba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:58.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.267+0000 7f3a7f857740 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216de780 0x5564216dba50 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f3a70013720 tx=0x7f3a70004ab0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.267+0000 7f3a7f857740 1 -- 172.21.15.33:0/2457116144 shutdown_connections 2024-09-16T00:39:58.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.267+0000 7f3a7f857740 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5564216ee3e0 0x5564216f0830 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.267+0000 7f3a5e7fc640 0 client.19656 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:58.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.267+0000 7f3a7f857740 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216de780 0x5564216dba50 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.267+0000 7f3a7f857740 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216db440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.267+0000 7f3a7f857740 1 --2- 172.21.15.33:0/2457116144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216cf5d0 0x5564216de190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.267+0000 7f3a7f857740 1 -- 172.21.15.33:0/2457116144 >> 172.21.15.33:0/2457116144 conn(0x556421698310 msgr2=0x5564216d1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:58.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.267+0000 7f3a7f857740 1 -- 172.21.15.33:0/2457116144 shutdown_connections 2024-09-16T00:39:58.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.267+0000 7f3a7f857740 1 -- 172.21.15.33:0/2457116144 wait complete. 2024-09-16T00:39:58.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.267+0000 7f3a7f857740 20 client.19656 trim_cache size 0 max 16384 2024-09-16T00:39:58.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a7f857740 1 Processor -- start 2024-09-16T00:39:58.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a7f857740 1 -- start start 2024-09-16T00:39:58.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a7f857740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216ea720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a7f857740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216cb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a7f857740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216de780 0x5564216cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a7f857740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a781af7e0 con 0x5564216cf5d0 2024-09-16T00:39:58.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a7f857740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3a781ad0b0 con 0x5564216de780 2024-09-16T00:39:58.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a7f857740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3a781aaf00 con 0x5564216d0b40 2024-09-16T00:39:58.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a76ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216ea720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a76ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216ea720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51214/0 (socket says 172.21.15.33:51214) 2024-09-16T00:39:58.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a76ffd640 1 -- 172.21.15.33:0/1268032821 learned_addr learned my addr 172.21.15.33:0/1268032821 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:58.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a6ffff640 1 --2- 172.21.15.33:0/1268032821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.280+0000 7f3a777fe640 1 --2- 172.21.15.33:0/1268032821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216de780 0x5564216cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.281+0000 7f3a76ffd640 1 -- 172.21.15.33:0/1268032821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 msgr2=0x5564216cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:58.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.281+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/1268032821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216cb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.281+0000 7f3a76ffd640 1 -- 172.21.15.33:0/1268032821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216de780 msgr2=0x5564216cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:58.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.281+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/1268032821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216de780 0x5564216cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.281+0000 7f3a76ffd640 1 -- 172.21.15.33:0/1268032821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564216cc180 con 0x5564216cf5d0 2024-09-16T00:39:58.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.281+0000 7f3a6ffff640 1 --2- 172.21.15.33:0/1268032821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216cb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:39:58.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.281+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/1268032821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216ea720 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800bd10 tx=0x7f3a68004000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c4cb0312c8a5ae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:58.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.281+0000 7f3a6cff9640 1 -- 172.21.15.33:0/1268032821 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a680035f0 con 0x5564216cf5d0 2024-09-16T00:39:58.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.281+0000 7f3a6cff9640 1 -- 172.21.15.33:0/1268032821 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a68005520 con 0x5564216cf5d0 2024-09-16T00:39:58.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.281+0000 7f3a6cff9640 1 -- 172.21.15.33:0/1268032821 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a68011b30 con 0x5564216cf5d0 2024-09-16T00:39:58.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.282+0000 7f3a7f857740 1 -- 172.21.15.33:0/1268032821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 msgr2=0x5564216ea720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:58.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.282+0000 7f3a7f857740 1 --2- 172.21.15.33:0/1268032821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216ea720 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800bd10 tx=0x7f3a68004000 comp rx=0 tx=0).stop 2024-09-16T00:39:58.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.282+0000 7f3a7f857740 1 -- 172.21.15.33:0/1268032821 shutdown_connections 2024-09-16T00:39:58.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.282+0000 7f3a7f857740 1 --2- 172.21.15.33:0/1268032821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216de780 0x5564216cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.282+0000 7f3a7f857740 1 --2- 172.21.15.33:0/1268032821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216d0b40 0x5564216cb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.282+0000 7f3a7f857740 1 --2- 172.21.15.33:0/1268032821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216ea720 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.282+0000 7f3a7f857740 1 -- 172.21.15.33:0/1268032821 >> 172.21.15.33:0/1268032821 conn(0x556421698310 msgr2=0x5564216db400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:58.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.282+0000 7f3a7f857740 1 -- 172.21.15.33:0/1268032821 shutdown_connections 2024-09-16T00:39:58.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.282+0000 7f3a7f857740 1 -- 172.21.15.33:0/1268032821 wait complete. 2024-09-16T00:39:58.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.282+0000 7f3a7f857740 1 -- start start 2024-09-16T00:39:58.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.282+0000 7f3a7f857740 1 Processor -- start 2024-09-16T00:39:58.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a7f857740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216e6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a7f857740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216d0b40 0x5564216d8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a7f857740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216de780 0x5564216d8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a7f857740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a781ab570 con 0x5564216cf5d0 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a7f857740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3a781ac9e0 con 0x5564216d0b40 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a7f857740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3a781abc40 con 0x5564216de780 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a7f857740 10 client.0 _ll_register_callbacks cb 0x7f3a70009980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a76ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216e6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a76ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216e6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51216/0 (socket says 172.21.15.33:51216) 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a76ffd640 1 -- 172.21.15.33:0/2096684151 learned_addr learned my addr 172.21.15.33:0/2096684151 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a6ffff640 1 --2- 172.21.15.33:0/2096684151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216d0b40 0x5564216d8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a76ffd640 1 -- 172.21.15.33:0/2096684151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216de780 msgr2=0x5564216d8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/2096684151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216de780 0x5564216d8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a76ffd640 1 -- 172.21.15.33:0/2096684151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216d0b40 msgr2=0x5564216d8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:39:58.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/2096684151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216d0b40 0x5564216d8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a76ffd640 1 -- 172.21.15.33:0/2096684151 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564216d8ec0 con 0x5564216cf5d0 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a76ffd640 1 --2- 172.21.15.33:0/2096684151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216e6fd0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800b4e0 tx=0x7f3a680107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f81208a86fa88acd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a5e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a5e7fc640 1 -- 172.21.15.33:0/2096684151 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a680050e0 con 0x5564216cf5d0 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a5e7fc640 1 -- 172.21.15.33:0/2096684151 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f3a68005280 con 0x5564216cf5d0 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.283+0000 7f3a7f857740 1 -- 172.21.15.33:0/2096684151 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a68005520 con 0x5564216cf5d0 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.284+0000 7f3a5e7fc640 1 -- 172.21.15.33:0/2096684151 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3a68042050 con 0x5564216cf5d0 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.284+0000 7f3a7f857740 10 client.19662 fetch_fsmap learned FSMap version 33 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.284+0000 7f3a7f857740 1 -- 172.21.15.33:0/2096684151 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556421759090 con 0x5564216cf5d0 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.284+0000 7f3a5e7fc640 1 -- 172.21.15.33:0/2096684151 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a68005520 con 0x5564216cf5d0 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.284+0000 7f3a5e7fc640 1 -- 172.21.15.33:0/2096684151 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f3a68003a30 con 0x5564216cf5d0 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.284+0000 7f3a7f857740 10 client.19662 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.284+0000 7f3a7f857740 10 client.19662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:39:58.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.284+0000 7f3a7f857740 20 client.19662 populate_metadata read hostname 'smithi033' 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.284+0000 7f3a7f857740 1 --2- 172.21.15.33:0/2096684151 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5564216ee3e0 0x55642175b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.284+0000 7f3a7f857740 4 client.19662 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.284+0000 7f3a7f857740 1 -- 172.21.15.33:0/2096684151 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55642163cd70 con 0x5564216ee3e0 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.284+0000 7f3a6ffff640 1 --2- 172.21.15.33:0/2096684151 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5564216ee3e0 0x55642175b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a6ffff640 1 --2- 172.21.15.33:0/2096684151 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5564216ee3e0 0x55642175b600 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=522fdcec3fe43cc4 server_cookie=84055809659a6090 in_seq=0 out_seq=0 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a5e7fc640 10 client.19662 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a5e7fc640 1 -- 172.21.15.33:0/2096684151 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55642163cd70 con 0x5564216ee3e0 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a5e7fc640 10 client.19662 handle_command_reply: tid=0 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a7f857740 1 client.19662 shutdown 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a7f857740 1 -- 172.21.15.33:0/2096684151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 msgr2=0x5564216e6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a7f857740 1 --2- 172.21.15.33:0/2096684151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216e6fd0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800b4e0 tx=0x7f3a680107a0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a7f857740 1 -- 172.21.15.33:0/2096684151 shutdown_connections 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a7f857740 1 --2- 172.21.15.33:0/2096684151 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5564216ee3e0 0x55642175b600 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a7f857740 1 --2- 172.21.15.33:0/2096684151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564216de780 0x5564216d8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a7f857740 1 --2- 172.21.15.33:0/2096684151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564216d0b40 0x5564216d8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a7f857740 1 --2- 172.21.15.33:0/2096684151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564216cf5d0 0x5564216e6fd0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a7f857740 1 -- 172.21.15.33:0/2096684151 >> 172.21.15.33:0/2096684151 conn(0x556421698310 msgr2=0x5564216dcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:58.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a5e7fc640 0 client.19662 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:39:58.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a7f857740 1 -- 172.21.15.33:0/2096684151 shutdown_connections 2024-09-16T00:39:58.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a7f857740 1 -- 172.21.15.33:0/2096684151 wait complete. 2024-09-16T00:39:58.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.285+0000 7f3a7f857740 20 client.19662 trim_cache size 0 max 16384 2024-09-16T00:39:58.304 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:39:58.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.286+0000 7f3a7d47e640 1 -- 172.21.15.33:0/1830697653 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3a640a32c0 msgr2=0x7f3a640a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:58.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.286+0000 7f3a7d47e640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3a640a32c0 0x7f3a640a5780 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3a70005fd0 tx=0x7f3a70004a40 comp rx=0 tx=0).stop 2024-09-16T00:39:58.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.286+0000 7f3a7d47e640 1 -- 172.21.15.33:0/1830697653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a78102790 msgr2=0x7f3a78198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:39:58.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.286+0000 7f3a7d47e640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a78102790 0x7f3a78198970 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f3a60006010 tx=0x7f3a600042f0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.286+0000 7f3a7d47e640 1 -- 172.21.15.33:0/1830697653 shutdown_connections 2024-09-16T00:39:58.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.286+0000 7f3a7d47e640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f3a640a32c0 0x7f3a640a5780 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.286+0000 7f3a7d47e640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a78106f10 0x7f3a7819cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.286+0000 7f3a7d47e640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a78102790 0x7f3a78198970 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.286+0000 7f3a7d47e640 1 --2- 172.21.15.33:0/1830697653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a78101de0 0x7f3a78198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:39:58.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.286+0000 7f3a7d47e640 1 -- 172.21.15.33:0/1830697653 >> 172.21.15.33:0/1830697653 conn(0x7f3a780fd8c0 msgr2=0x7f3a780fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:39:58.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.286+0000 7f3a7d47e640 1 -- 172.21.15.33:0/1830697653 shutdown_connections 2024-09-16T00:39:58.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:39:58.286+0000 7f3a7d47e640 1 -- 172.21.15.33:0/1830697653 wait complete. 2024-09-16T00:39:59.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.31 MB/sec execute 425 sec latency 18652.537 ms 2024-09-16T00:39:59.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:39:59 smithi033 ceph-mon[31243]: pgmap v1035: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 2.3 MiB/s wr, 173 op/s 2024-09-16T00:39:59.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:39:59 smithi083 ceph-mon[35242]: pgmap v1035: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 2.3 MiB/s wr, 173 op/s 2024-09-16T00:39:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:39:59 smithi060 ceph-mon[35185]: pgmap v1035: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 2.3 MiB/s wr, 173 op/s 2024-09-16T00:40:00.099 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.27 MB/sec execute 426 sec latency 19652.649 ms 2024-09-16T00:40:00.101 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:40:00.101 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:40:00.376 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:40:00.800 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:00 smithi033 ceph-mon[31243]: overall HEALTH_OK 2024-09-16T00:40:00.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.888+0000 7fc69ce50640 1 -- 172.21.15.33:0/1582138947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc698103780 msgr2=0x7fc698107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:00.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.888+0000 7fc69ce50640 1 --2- 172.21.15.33:0/1582138947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc698103780 0x7fc698107bc0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fc680009930 tx=0x7fc68002a420 comp rx=0 tx=0).stop 2024-09-16T00:40:00.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.888+0000 7fc69ce50640 1 -- 172.21.15.33:0/1582138947 shutdown_connections 2024-09-16T00:40:00.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.888+0000 7fc69ce50640 1 --2- 172.21.15.33:0/1582138947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc698108300 0x7fc69810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.888+0000 7fc69ce50640 1 --2- 172.21.15.33:0/1582138947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc698103780 0x7fc698107bc0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.888+0000 7fc69ce50640 1 --2- 172.21.15.33:0/1582138947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc698102dd0 0x7fc6981031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.888+0000 7fc69ce50640 1 -- 172.21.15.33:0/1582138947 >> 172.21.15.33:0/1582138947 conn(0x7fc6980fc7d0 msgr2=0x7fc6980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.888+0000 7fc69ce50640 1 -- 172.21.15.33:0/1582138947 shutdown_connections 2024-09-16T00:40:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.888+0000 7fc69ce50640 1 -- 172.21.15.33:0/1582138947 wait complete. 2024-09-16T00:40:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69ce50640 1 Processor -- start 2024-09-16T00:40:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69ce50640 1 -- start start 2024-09-16T00:40:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69ce50640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc698102dd0 0x7fc698100a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69ce50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc698103780 0x7fc698100f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69ce50640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc698108300 0x7fc6981014b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69ce50640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc6981120d0 con 0x7fc698102dd0 2024-09-16T00:40:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69ce50640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc698111f50 con 0x7fc698108300 2024-09-16T00:40:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69ce50640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc698112250 con 0x7fc698103780 2024-09-16T00:40:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc698102dd0 0x7fc698100a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc698102dd0 0x7fc698100a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52476/0 (socket says 172.21.15.33:52476) 2024-09-16T00:40:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69659c640 1 -- 172.21.15.33:0/2109637282 learned_addr learned my addr 172.21.15.33:0/2109637282 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc695d9b640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc698103780 0x7fc698100f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc696d9d640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc698108300 0x7fc6981014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69659c640 1 -- 172.21.15.33:0/2109637282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc698103780 msgr2=0x7fc698100f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69659c640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc698103780 0x7fc698100f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69659c640 1 -- 172.21.15.33:0/2109637282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc698108300 msgr2=0x7fc6981014b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69659c640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc698108300 0x7fc6981014b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69659c640 1 -- 172.21.15.33:0/2109637282 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6981a6af0 con 0x7fc698102dd0 2024-09-16T00:40:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc695d9b640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc698103780 0x7fc698100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:40:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.889+0000 7fc69659c640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc698102dd0 0x7fc698100a30 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fc68800d950 tx=0x7fc68800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.890+0000 7fc68f7fe640 1 -- 172.21.15.33:0/2109637282 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688017070 con 0x7fc698102dd0 2024-09-16T00:40:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.890+0000 7fc68f7fe640 1 -- 172.21.15.33:0/2109637282 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc688009c30 con 0x7fc698102dd0 2024-09-16T00:40:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.890+0000 7fc69ce50640 1 -- 172.21.15.33:0/2109637282 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6981a6de0 con 0x7fc698102dd0 2024-09-16T00:40:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.890+0000 7fc68f7fe640 1 -- 172.21.15.33:0/2109637282 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68800f6b0 con 0x7fc698102dd0 2024-09-16T00:40:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.890+0000 7fc69ce50640 1 -- 172.21.15.33:0/2109637282 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6981a7240 con 0x7fc698102dd0 2024-09-16T00:40:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.892+0000 7fc68f7fe640 1 -- 172.21.15.33:0/2109637282 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc6880109d0 con 0x7fc698102dd0 2024-09-16T00:40:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.892+0000 7fc69f229740 1 Processor -- start 2024-09-16T00:40:00.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.892+0000 7fc69f229740 1 -- start start 2024-09-16T00:40:00.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69f229740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f503b60 0x56457f503f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69f229740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f504480 0x56457f50e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69f229740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f50ed40 0x56457f511130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69f229740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56457f469280 con 0x56457f504480 2024-09-16T00:40:00.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69f229740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56457f410b10 con 0x56457f50ed40 2024-09-16T00:40:00.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69f229740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56457f465240 con 0x56457f503b60 2024-09-16T00:40:00.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f504480 0x56457f50e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f504480 0x56457f50e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52478/0 (socket says 172.21.15.33:52478) 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69659c640 1 -- 172.21.15.33:0/4180919000 learned_addr learned my addr 172.21.15.33:0/4180919000 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc68f7fe640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6840a3240 0x7fc6840a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc695d9b640 1 --2- 172.21.15.33:0/4180919000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f50ed40 0x56457f511130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc696d9d640 1 --2- 172.21.15.33:0/4180919000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f503b60 0x56457f503f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc696d9d640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6840a3240 0x7fc6840a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69659c640 1 -- 172.21.15.33:0/4180919000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f503b60 msgr2=0x56457f503f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69659c640 1 --2- 172.21.15.33:0/4180919000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f503b60 0x56457f503f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69659c640 1 -- 172.21.15.33:0/4180919000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f50ed40 msgr2=0x56457f511130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69659c640 1 --2- 172.21.15.33:0/4180919000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f50ed40 0x56457f511130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc69659c640 1 -- 172.21.15.33:0/4180919000 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56457f41d1f0 con 0x56457f504480 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.893+0000 7fc68f7fe640 1 -- 172.21.15.33:0/2109637282 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc6880cfd60 con 0x7fc698102dd0 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.894+0000 7fc69659c640 1 --2- 172.21.15.33:0/4180919000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f504480 0x56457f50e800 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fc688098300 tx=0x7fc68800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47e7d2f46a2f919 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:00.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.894+0000 7fc696d9d640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6840a3240 0x7fc6840a5700 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc690009900 tx=0x7fc690009650 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.894+0000 7fc68cff9640 1 -- 172.21.15.33:0/4180919000 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688020070 con 0x56457f504480 2024-09-16T00:40:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.894+0000 7fc68cff9640 1 -- 172.21.15.33:0/4180919000 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc688009de0 con 0x56457f504480 2024-09-16T00:40:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.894+0000 7fc68cff9640 1 -- 172.21.15.33:0/4180919000 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6880a4b10 con 0x56457f504480 2024-09-16T00:40:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.894+0000 7fc69f229740 1 -- 172.21.15.33:0/4180919000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f504480 msgr2=0x56457f50e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.894+0000 7fc69f229740 1 --2- 172.21.15.33:0/4180919000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f504480 0x56457f50e800 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fc688098300 tx=0x7fc68800bf60 comp rx=0 tx=0).stop 2024-09-16T00:40:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 -- 172.21.15.33:0/4180919000 shutdown_connections 2024-09-16T00:40:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 --2- 172.21.15.33:0/4180919000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f50ed40 0x56457f511130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 --2- 172.21.15.33:0/4180919000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f504480 0x56457f50e800 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 --2- 172.21.15.33:0/4180919000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f503b60 0x56457f503f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 -- 172.21.15.33:0/4180919000 >> 172.21.15.33:0/4180919000 conn(0x56457f5000f0 msgr2=0x56457f5004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 -- 172.21.15.33:0/4180919000 shutdown_connections 2024-09-16T00:40:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 -- 172.21.15.33:0/4180919000 wait complete. 2024-09-16T00:40:00.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 -- start start 2024-09-16T00:40:00.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 Processor -- start 2024-09-16T00:40:00.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f503b60 0x56457f50bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f504480 0x56457f50c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f50ed40 0x56457f50c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56457f5118b0 con 0x56457f503b60 2024-09-16T00:40:00.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56457f423f10 con 0x56457f504480 2024-09-16T00:40:00.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56457f511a30 con 0x56457f50ed40 2024-09-16T00:40:00.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69f229740 10 client.0 _ll_register_callbacks cb 0x56457f3eca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:00.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f503b60 0x56457f50bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f503b60 0x56457f50bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52486/0 (socket says 172.21.15.33:52486) 2024-09-16T00:40:00.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.895+0000 7fc69659c640 1 -- 172.21.15.33:0/94306394 learned_addr learned my addr 172.21.15.33:0/94306394 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:00.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc696d9d640 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f50ed40 0x56457f50c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc695d9b640 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f504480 0x56457f50c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc69659c640 1 -- 172.21.15.33:0/94306394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f50ed40 msgr2=0x56457f50c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:00.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc69659c640 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f50ed40 0x56457f50c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc69659c640 1 -- 172.21.15.33:0/94306394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f504480 msgr2=0x56457f50c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:00.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc69659c640 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f504480 0x56457f50c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc69659c640 1 -- 172.21.15.33:0/94306394 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56457f1da6a0 con 0x56457f503b60 2024-09-16T00:40:00.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc695d9b640 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f504480 0x56457f50c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:40:00.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc696d9d640 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f50ed40 0x56457f50c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:40:00.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc69659c640 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f503b60 0x56457f50bbe0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fc688009f90 tx=0x7fc688098210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b36ddf751882b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:00.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc67dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:40:00.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc67dffb640 1 -- 172.21.15.33:0/94306394 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688020040 con 0x56457f503b60 2024-09-16T00:40:00.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc67dffb640 1 -- 172.21.15.33:0/94306394 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc68809a1a0 con 0x56457f503b60 2024-09-16T00:40:00.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc69f229740 1 -- 172.21.15.33:0/94306394 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc688009de0 con 0x56457f503b60 2024-09-16T00:40:00.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc67dffb640 1 -- 172.21.15.33:0/94306394 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc6880a4b10 con 0x56457f503b60 2024-09-16T00:40:00.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc69f229740 10 client.19674 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:00.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc69f229740 1 -- 172.21.15.33:0/94306394 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56457f5118b0 con 0x56457f503b60 2024-09-16T00:40:00.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.896+0000 7fc67dffb640 1 -- 172.21.15.33:0/94306394 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc688009de0 con 0x56457f503b60 2024-09-16T00:40:00.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.897+0000 7fc67dffb640 1 -- 172.21.15.33:0/94306394 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fc6880a4710 con 0x56457f503b60 2024-09-16T00:40:00.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.897+0000 7fc69f229740 10 client.19674 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:00.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.897+0000 7fc69f229740 10 client.19674 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:40:00.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.897+0000 7fc69f229740 20 client.19674 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:00.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.897+0000 7fc69f229740 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56457f51e600 0x56457f5209f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.897+0000 7fc69f229740 4 client.19674 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:40:00.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.897+0000 7fc69f229740 1 -- 172.21.15.33:0/94306394 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56457f4767f0 con 0x56457f51e600 2024-09-16T00:40:00.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.897+0000 7fc695d9b640 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56457f51e600 0x56457f5209f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.897+0000 7fc695d9b640 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56457f51e600 0x56457f5209f0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4e9320a8007937a server_cookie=c460fc8765456df2 in_seq=0 out_seq=0 2024-09-16T00:40:00.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.897+0000 7fc67dffb640 10 client.19674 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:00.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc67dffb640 1 -- 172.21.15.33:0/94306394 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56457f4767f0 con 0x56457f51e600 2024-09-16T00:40:00.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc67dffb640 10 client.19674 handle_command_reply: tid=0 2024-09-16T00:40:00.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc69f229740 1 client.19674 shutdown 2024-09-16T00:40:00.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc69f229740 1 -- 172.21.15.33:0/94306394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f503b60 msgr2=0x56457f50bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:00.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc69f229740 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f503b60 0x56457f50bbe0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fc688009f90 tx=0x7fc688098210 comp rx=0 tx=0).stop 2024-09-16T00:40:00.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc69f229740 1 -- 172.21.15.33:0/94306394 shutdown_connections 2024-09-16T00:40:00.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc69f229740 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56457f51e600 0x56457f5209f0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc69f229740 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f50ed40 0x56457f50c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc69f229740 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f504480 0x56457f50c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc69f229740 1 --2- 172.21.15.33:0/94306394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f503b60 0x56457f50bbe0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc67dffb640 0 client.19674 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:00.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc69f229740 1 -- 172.21.15.33:0/94306394 >> 172.21.15.33:0/94306394 conn(0x56457f5000f0 msgr2=0x56457f505e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:00.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc69f229740 1 -- 172.21.15.33:0/94306394 shutdown_connections 2024-09-16T00:40:00.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc69f229740 1 -- 172.21.15.33:0/94306394 wait complete. 2024-09-16T00:40:00.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.898+0000 7fc69f229740 20 client.19674 trim_cache size 0 max 16384 2024-09-16T00:40:00.922 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:00 smithi083 ceph-mon[35242]: overall HEALTH_OK 2024-09-16T00:40:00.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.916+0000 7fc69f229740 1 Processor -- start 2024-09-16T00:40:00.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.916+0000 7fc69f229740 1 -- start start 2024-09-16T00:40:00.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.916+0000 7fc69f229740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f503b60 0x56457f50c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.916+0000 7fc69f229740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f504480 0x56457f526160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.916+0000 7fc69f229740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 0x56457f5266a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.916+0000 7fc69f229740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc6981b3c30 con 0x56457f50ed40 2024-09-16T00:40:00.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.916+0000 7fc69f229740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc6981b1500 con 0x56457f503b60 2024-09-16T00:40:00.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.916+0000 7fc69f229740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc6981af350 con 0x56457f504480 2024-09-16T00:40:00.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.916+0000 7fc696d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 0x56457f5266a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.916+0000 7fc696d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 0x56457f5266a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52502/0 (socket says 172.21.15.33:52502) 2024-09-16T00:40:00.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.916+0000 7fc696d9d640 1 -- 172.21.15.33:0/1004628135 learned_addr learned my addr 172.21.15.33:0/1004628135 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:00.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.916+0000 7fc69659c640 1 --2- 172.21.15.33:0/1004628135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f503b60 0x56457f50c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.917+0000 7fc695d9b640 1 --2- 172.21.15.33:0/1004628135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f504480 0x56457f526160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.917+0000 7fc696d9d640 1 -- 172.21.15.33:0/1004628135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f504480 msgr2=0x56457f526160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:00.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.917+0000 7fc696d9d640 1 --2- 172.21.15.33:0/1004628135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f504480 0x56457f526160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.917+0000 7fc696d9d640 1 -- 172.21.15.33:0/1004628135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f503b60 msgr2=0x56457f50c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:00.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.917+0000 7fc696d9d640 1 --2- 172.21.15.33:0/1004628135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f503b60 0x56457f50c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.917+0000 7fc696d9d640 1 -- 172.21.15.33:0/1004628135 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56457f4fff20 con 0x56457f50ed40 2024-09-16T00:40:00.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.917+0000 7fc696d9d640 1 --2- 172.21.15.33:0/1004628135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 0x56457f5266a0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fc6900086e0 tx=0x7fc690009ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d85a7e28d8d278f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:00.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.917+0000 7fc67ffff640 1 -- 172.21.15.33:0/1004628135 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690014040 con 0x56457f50ed40 2024-09-16T00:40:00.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.917+0000 7fc67ffff640 1 -- 172.21.15.33:0/1004628135 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc690018b10 con 0x56457f50ed40 2024-09-16T00:40:00.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.917+0000 7fc67ffff640 1 -- 172.21.15.33:0/1004628135 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690025ae0 con 0x56457f50ed40 2024-09-16T00:40:00.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 -- 172.21.15.33:0/1004628135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 msgr2=0x56457f5266a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 --2- 172.21.15.33:0/1004628135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 0x56457f5266a0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fc6900086e0 tx=0x7fc690009ab0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 -- 172.21.15.33:0/1004628135 shutdown_connections 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 --2- 172.21.15.33:0/1004628135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 0x56457f5266a0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 --2- 172.21.15.33:0/1004628135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f504480 0x56457f526160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 --2- 172.21.15.33:0/1004628135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f503b60 0x56457f50c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 -- 172.21.15.33:0/1004628135 >> 172.21.15.33:0/1004628135 conn(0x56457f5000f0 msgr2=0x56457f50aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 -- 172.21.15.33:0/1004628135 shutdown_connections 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 -- 172.21.15.33:0/1004628135 wait complete. 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 -- start start 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 Processor -- start 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f503b60 0x56457f513220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f504480 0x56457f513760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 0x56457f582100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc6981af9c0 con 0x56457f50ed40 2024-09-16T00:40:00.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc6981b0e30 con 0x56457f504480 2024-09-16T00:40:00.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.918+0000 7fc69f229740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc6981b0090 con 0x56457f503b60 2024-09-16T00:40:00.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc69f229740 10 client.0 _ll_register_callbacks cb 0x7fc680009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:00.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc69659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f503b60 0x56457f513220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc696d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 0x56457f582100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc695d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f504480 0x56457f513760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc69659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f503b60 0x56457f513220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40104/0 (socket says 172.21.15.33:40104) 2024-09-16T00:40:00.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc696d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 0x56457f582100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52504/0 (socket says 172.21.15.33:52504) 2024-09-16T00:40:00.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc69659c640 1 -- 172.21.15.33:0/2312447160 learned_addr learned my addr 172.21.15.33:0/2312447160 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:00.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc696d9d640 1 -- 172.21.15.33:0/2312447160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f503b60 msgr2=0x56457f513220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:00.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc696d9d640 1 --2- 172.21.15.33:0/2312447160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f503b60 0x56457f513220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc696d9d640 1 -- 172.21.15.33:0/2312447160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f504480 msgr2=0x56457f513760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc696d9d640 1 --2- 172.21.15.33:0/2312447160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f504480 0x56457f513760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc696d9d640 1 -- 172.21.15.33:0/2312447160 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56457f5826f0 con 0x56457f50ed40 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc695d9b640 1 --2- 172.21.15.33:0/2312447160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f504480 0x56457f513760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc696d9d640 1 --2- 172.21.15.33:0/2312447160 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 0x56457f582100 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fc690009da0 tx=0x7fc690004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4bd47139a184cb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc67dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc67dffb640 1 -- 172.21.15.33:0/2312447160 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690018830 con 0x56457f50ed40 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc69f229740 1 -- 172.21.15.33:0/2312447160 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc690018b10 con 0x56457f50ed40 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.919+0000 7fc67dffb640 1 -- 172.21.15.33:0/2312447160 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc690025760 con 0x56457f50ed40 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.920+0000 7fc67dffb640 1 -- 172.21.15.33:0/2312447160 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc690037050 con 0x56457f50ed40 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.920+0000 7fc69f229740 10 client.19680 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.920+0000 7fc69f229740 1 -- 172.21.15.33:0/2312447160 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56457f589ab0 con 0x56457f50ed40 2024-09-16T00:40:00.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.920+0000 7fc67dffb640 1 -- 172.21.15.33:0/2312447160 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc690018b10 con 0x56457f50ed40 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.920+0000 7fc67dffb640 1 -- 172.21.15.33:0/2312447160 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fc690023020 con 0x56457f50ed40 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.920+0000 7fc69f229740 10 client.19680 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.920+0000 7fc69f229740 10 client.19680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.920+0000 7fc69f229740 20 client.19680 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.920+0000 7fc69f229740 1 --2- 172.21.15.33:0/2312447160 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56457f58a100 0x56457f58c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.920+0000 7fc69f229740 4 client.19680 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.920+0000 7fc69f229740 1 -- 172.21.15.33:0/2312447160 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56457f470d00 con 0x56457f58a100 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.920+0000 7fc69659c640 1 --2- 172.21.15.33:0/2312447160 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56457f58a100 0x56457f58c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.921+0000 7fc69659c640 1 --2- 172.21.15.33:0/2312447160 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56457f58a100 0x56457f58c520 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=726fbbb7c15d28d1 server_cookie=7c5aabead830babb in_seq=0 out_seq=0 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.921+0000 7fc67dffb640 10 client.19680 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.921+0000 7fc67dffb640 1 -- 172.21.15.33:0/2312447160 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56457f470d00 con 0x56457f58a100 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc67dffb640 10 client.19680 handle_command_reply: tid=0 2024-09-16T00:40:00.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc69f229740 1 client.19680 shutdown 2024-09-16T00:40:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc69f229740 1 -- 172.21.15.33:0/2312447160 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 msgr2=0x56457f582100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc69f229740 1 --2- 172.21.15.33:0/2312447160 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 0x56457f582100 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fc690009da0 tx=0x7fc690004b50 comp rx=0 tx=0).stop 2024-09-16T00:40:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc69f229740 1 -- 172.21.15.33:0/2312447160 shutdown_connections 2024-09-16T00:40:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc69f229740 1 --2- 172.21.15.33:0/2312447160 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56457f58a100 0x56457f58c520 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc69f229740 1 --2- 172.21.15.33:0/2312447160 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56457f50ed40 0x56457f582100 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc69f229740 1 --2- 172.21.15.33:0/2312447160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56457f504480 0x56457f513760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc69f229740 1 --2- 172.21.15.33:0/2312447160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56457f503b60 0x56457f513220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc67dffb640 0 client.19680 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc69f229740 1 -- 172.21.15.33:0/2312447160 >> 172.21.15.33:0/2312447160 conn(0x56457f5000f0 msgr2=0x56457f5055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc69f229740 1 -- 172.21.15.33:0/2312447160 shutdown_connections 2024-09-16T00:40:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc69f229740 1 -- 172.21.15.33:0/2312447160 wait complete. 2024-09-16T00:40:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.922+0000 7fc69f229740 20 client.19680 trim_cache size 0 max 16384 2024-09-16T00:40:00.934 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:40:00.934 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:40:00.934 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "73c80774-a255-49e6-860f-ba6fe4154c29", 2024-09-16T00:40:00.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:40:00.934 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:40:00.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.923+0000 7fc69ce50640 1 -- 172.21.15.33:0/2109637282 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6840a3240 msgr2=0x7fc6840a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:00.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.923+0000 7fc69ce50640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6840a3240 0x7fc6840a5700 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc690009900 tx=0x7fc690009650 comp rx=0 tx=0).stop 2024-09-16T00:40:00.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.923+0000 7fc69ce50640 1 -- 172.21.15.33:0/2109637282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc698102dd0 msgr2=0x7fc698100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:00.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.923+0000 7fc69ce50640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc698102dd0 0x7fc698100a30 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fc68800d950 tx=0x7fc68800de10 comp rx=0 tx=0).stop 2024-09-16T00:40:00.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.923+0000 7fc69ce50640 1 -- 172.21.15.33:0/2109637282 shutdown_connections 2024-09-16T00:40:00.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.923+0000 7fc69ce50640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc6840a3240 0x7fc6840a5700 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.923+0000 7fc69ce50640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc698108300 0x7fc6981014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.923+0000 7fc69ce50640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc698103780 0x7fc698100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.923+0000 7fc69ce50640 1 --2- 172.21.15.33:0/2109637282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc698102dd0 0x7fc698100a30 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:00.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.923+0000 7fc69ce50640 1 -- 172.21.15.33:0/2109637282 >> 172.21.15.33:0/2109637282 conn(0x7fc6980fc7d0 msgr2=0x7fc698105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:00.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.923+0000 7fc69ce50640 1 -- 172.21.15.33:0/2109637282 shutdown_connections 2024-09-16T00:40:00.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:00.923+0000 7fc69ce50640 1 -- 172.21.15.33:0/2109637282 wait complete. 2024-09-16T00:40:00.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:00 smithi060 ceph-mon[35185]: overall HEALTH_OK 2024-09-16T00:40:01.099 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.23 MB/sec execute 427 sec latency 20652.740 ms 2024-09-16T00:40:01.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:01 smithi033 ceph-mon[31243]: pgmap v1036: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 596 KiB/s wr, 100 op/s 2024-09-16T00:40:01.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:01 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:40:01.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:01 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:01.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:01 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:01 smithi060 ceph-mon[35185]: pgmap v1036: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 596 KiB/s wr, 100 op/s 2024-09-16T00:40:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:01 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:40:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:01 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:01.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:01 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:01 smithi083 ceph-mon[35242]: pgmap v1036: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 596 KiB/s wr, 100 op/s 2024-09-16T00:40:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:01 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:40:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:01 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:01 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:01.927 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:40:02.099 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.19 MB/sec execute 428 sec latency 21652.804 ms 2024-09-16T00:40:02.195 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:40:02.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:02.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:02.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:02.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:02.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:02.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:02.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:02.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:02.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:02.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:02 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.695+0000 7faf11dbb640 1 -- 172.21.15.33:0/63596263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102d40 msgr2=0x7faf0c103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:02.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.695+0000 7faf11dbb640 1 --2- 172.21.15.33:0/63596263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102d40 0x7faf0c103120 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7faefc009ca0 tx=0x7faefc02a950 comp rx=0 tx=0).stop 2024-09-16T00:40:02.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.695+0000 7faf11dbb640 1 -- 172.21.15.33:0/63596263 shutdown_connections 2024-09-16T00:40:02.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.695+0000 7faf11dbb640 1 --2- 172.21.15.33:0/63596263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c10c170 0x7faf0c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.695+0000 7faf11dbb640 1 --2- 172.21.15.33:0/63596263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c103660 0x7faf0c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.695+0000 7faf11dbb640 1 --2- 172.21.15.33:0/63596263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102d40 0x7faf0c103120 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.695+0000 7faf11dbb640 1 -- 172.21.15.33:0/63596263 >> 172.21.15.33:0/63596263 conn(0x7faf0c0fc7a0 msgr2=0x7faf0c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:02.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.695+0000 7faf11dbb640 1 -- 172.21.15.33:0/63596263 shutdown_connections 2024-09-16T00:40:02.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.695+0000 7faf11dbb640 1 -- 172.21.15.33:0/63596263 wait complete. 2024-09-16T00:40:02.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.696+0000 7faf11dbb640 1 Processor -- start 2024-09-16T00:40:02.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.696+0000 7faf11dbb640 1 -- start start 2024-09-16T00:40:02.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.696+0000 7faf11dbb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102d40 0x7faf0c1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.696+0000 7faf11dbb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c103660 0x7faf0c100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.696+0000 7faf11dbb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c10c170 0x7faf0c101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.696+0000 7faf11dbb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c111f70 con 0x7faf0c102d40 2024-09-16T00:40:02.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.696+0000 7faf11dbb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c111df0 con 0x7faf0c10c170 2024-09-16T00:40:02.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.696+0000 7faf11dbb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1120f0 con 0x7faf0c103660 2024-09-16T00:40:02.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.696+0000 7faf0b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102d40 0x7faf0c1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.696+0000 7faf0b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102d40 0x7faf0c1009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52530/0 (socket says 172.21.15.33:52530) 2024-09-16T00:40:02.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.696+0000 7faf0b7fe640 1 -- 172.21.15.33:0/560242277 learned_addr learned my addr 172.21.15.33:0/560242277 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:02.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf0bfff640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c10c170 0x7faf0c101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf0affd640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c103660 0x7faf0c100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf0b7fe640 1 -- 172.21.15.33:0/560242277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c103660 msgr2=0x7faf0c100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:02.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf0b7fe640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c103660 0x7faf0c100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf0b7fe640 1 -- 172.21.15.33:0/560242277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c10c170 msgr2=0x7faf0c101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:02.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf0b7fe640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c10c170 0x7faf0c101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf0b7fe640 1 -- 172.21.15.33:0/560242277 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0c1a68e0 con 0x7faf0c102d40 2024-09-16T00:40:02.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf0affd640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c103660 0x7faf0c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:40:02.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf0b7fe640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102d40 0x7faf0c1009a0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7faefc005cc0 tx=0x7faefc02f630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:02.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf08ff9640 1 -- 172.21.15.33:0/560242277 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc034040 con 0x7faf0c102d40 2024-09-16T00:40:02.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf11dbb640 1 -- 172.21.15.33:0/560242277 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0c1a6b70 con 0x7faf0c102d40 2024-09-16T00:40:02.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf08ff9640 1 -- 172.21.15.33:0/560242277 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faefc030920 con 0x7faf0c102d40 2024-09-16T00:40:02.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.697+0000 7faf08ff9640 1 -- 172.21.15.33:0/560242277 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc004830 con 0x7faf0c102d40 2024-09-16T00:40:02.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.698+0000 7faf11dbb640 1 -- 172.21.15.33:0/560242277 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf0c1a70a0 con 0x7faf0c102d40 2024-09-16T00:40:02.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.700+0000 7faf08ff9640 1 -- 172.21.15.33:0/560242277 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7faefc037020 con 0x7faf0c102d40 2024-09-16T00:40:02.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.700+0000 7faf14194740 1 Processor -- start 2024-09-16T00:40:02.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.700+0000 7faf14194740 1 -- start start 2024-09-16T00:40:02.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.700+0000 7faf14194740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a604a0 0x5597c1a60880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.700+0000 7faf14194740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a60dc0 0x5597c1a6b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.700+0000 7faf14194740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a6da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf14194740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5597c19c62e0 con 0x5597c1a6b680 2024-09-16T00:40:02.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf14194740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5597c196dce0 con 0x5597c1a604a0 2024-09-16T00:40:02.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf14194740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5597c19c22a0 con 0x5597c1a60dc0 2024-09-16T00:40:02.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a6da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a6da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52532/0 (socket says 172.21.15.33:52532) 2024-09-16T00:40:02.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0affd640 1 -- 172.21.15.33:0/2556471628 learned_addr learned my addr 172.21.15.33:0/2556471628 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:02.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf08ff9640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faef40a3240 0x7faef40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0bfff640 1 --2- 172.21.15.33:0/2556471628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a604a0 0x5597c1a60880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0b7fe640 1 --2- 172.21.15.33:0/2556471628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a60dc0 0x5597c1a6b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0affd640 1 -- 172.21.15.33:0/2556471628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a60dc0 msgr2=0x5597c1a6b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:02.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0affd640 1 --2- 172.21.15.33:0/2556471628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a60dc0 0x5597c1a6b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0bfff640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faef40a3240 0x7faef40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0affd640 1 -- 172.21.15.33:0/2556471628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a604a0 msgr2=0x5597c1a60880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:02.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0affd640 1 --2- 172.21.15.33:0/2556471628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a604a0 0x5597c1a60880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0affd640 1 -- 172.21.15.33:0/2556471628 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597c1952440 con 0x5597c1a6b680 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf08ff9640 1 -- 172.21.15.33:0/560242277 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7faefc0f0d50 con 0x7faf0c102d40 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0affd640 1 --2- 172.21.15.33:0/2556471628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a6da70 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7faf0000bb30 tx=0x7faf0000d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4de9e71739e2ca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faf0bfff640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faef40a3240 0x7faef40a5700 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7faf0400bba0 tx=0x7faf04004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.701+0000 7faef1ffb640 1 -- 172.21.15.33:0/2556471628 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00017420 con 0x5597c1a6b680 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.702+0000 7faef1ffb640 1 -- 172.21.15.33:0/2556471628 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf00017aa0 con 0x5597c1a6b680 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.702+0000 7faef1ffb640 1 -- 172.21.15.33:0/2556471628 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0000fbd0 con 0x5597c1a6b680 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.702+0000 7faf14194740 1 -- 172.21.15.33:0/2556471628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 msgr2=0x5597c1a6da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.702+0000 7faf14194740 1 --2- 172.21.15.33:0/2556471628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a6da70 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7faf0000bb30 tx=0x7faf0000d340 comp rx=0 tx=0).stop 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.702+0000 7faf14194740 1 -- 172.21.15.33:0/2556471628 shutdown_connections 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.702+0000 7faf14194740 1 --2- 172.21.15.33:0/2556471628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a6da70 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.702+0000 7faf14194740 1 --2- 172.21.15.33:0/2556471628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a60dc0 0x5597c1a6b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.702+0000 7faf14194740 1 --2- 172.21.15.33:0/2556471628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a604a0 0x5597c1a60880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.702+0000 7faf14194740 1 -- 172.21.15.33:0/2556471628 >> 172.21.15.33:0/2556471628 conn(0x5597c1a5ca30 msgr2=0x5597c1a5ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.702+0000 7faf14194740 1 -- 172.21.15.33:0/2556471628 shutdown_connections 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.702+0000 7faf14194740 1 -- 172.21.15.33:0/2556471628 wait complete. 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf14194740 1 -- start start 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf14194740 1 Processor -- start 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf14194740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a604a0 0x5597c1a6aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf14194740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a681c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf14194740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a68700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf14194740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5597c1a6e250 con 0x5597c1a6b680 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf14194740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5597c1981050 con 0x5597c1a60dc0 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf14194740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5597c1a6e3d0 con 0x5597c1a604a0 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf14194740 10 client.0 _ll_register_callbacks cb 0x5597c19c1160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf0bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a68700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf0bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a68700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52544/0 (socket says 172.21.15.33:52544) 2024-09-16T00:40:02.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf0bfff640 1 -- 172.21.15.33:0/4060000732 learned_addr learned my addr 172.21.15.33:0/4060000732 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:02.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf0affd640 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a681c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.703+0000 7faf0b7fe640 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a604a0 0x5597c1a6aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faf0bfff640 1 -- 172.21.15.33:0/4060000732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a604a0 msgr2=0x5597c1a6aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:02.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faf0bfff640 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a604a0 0x5597c1a6aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faf0bfff640 1 -- 172.21.15.33:0/4060000732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 msgr2=0x5597c1a681c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:02.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faf0bfff640 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a681c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faf0bfff640 1 -- 172.21.15.33:0/4060000732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597c17373c0 con 0x5597c1a6b680 2024-09-16T00:40:02.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faf0b7fe640 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a604a0 0x5597c1a6aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:40:02.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faf0affd640 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a681c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:40:02.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faf0bfff640 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a68700 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7faf04003fc0 tx=0x7faf040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ae434178cb479c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:02.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faeeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:40:02.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faeeaffd640 1 -- 172.21.15.33:0/4060000732 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf04025020 con 0x5597c1a6b680 2024-09-16T00:40:02.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faeeaffd640 1 -- 172.21.15.33:0/4060000732 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7faf04021890 con 0x5597c1a6b680 2024-09-16T00:40:02.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faf14194740 1 -- 172.21.15.33:0/4060000732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf00017aa0 con 0x5597c1a6b680 2024-09-16T00:40:02.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faeeaffd640 1 -- 172.21.15.33:0/4060000732 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7faf04030050 con 0x5597c1a6b680 2024-09-16T00:40:02.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faf14194740 10 client.19692 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:02.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faf14194740 1 -- 172.21.15.33:0/4060000732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5597c1a79c00 con 0x5597c1a6b680 2024-09-16T00:40:02.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.704+0000 7faeeaffd640 1 -- 172.21.15.33:0/4060000732 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf00017aa0 con 0x5597c1a6b680 2024-09-16T00:40:02.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.705+0000 7faeeaffd640 1 -- 172.21.15.33:0/4060000732 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7faf0401e070 con 0x5597c1a6b680 2024-09-16T00:40:02.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.705+0000 7faf14194740 10 client.19692 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:02.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.705+0000 7faf14194740 10 client.19692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:40:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.705+0000 7faf14194740 20 client.19692 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.705+0000 7faf14194740 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5597c1a7b110 0x5597c1a7d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.705+0000 7faf14194740 4 client.19692 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:40:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.705+0000 7faf14194740 1 -- 172.21.15.33:0/4060000732 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597c19d3850 con 0x5597c1a7b110 2024-09-16T00:40:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.705+0000 7faf0b7fe640 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5597c1a7b110 0x5597c1a7d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.705+0000 7faf0b7fe640 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5597c1a7b110 0x5597c1a7d590 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a700c3522a2e2432 server_cookie=5784c2a73bfd5b4a in_seq=0 out_seq=0 2024-09-16T00:40:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.705+0000 7faeeaffd640 10 client.19692 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.706+0000 7faeeaffd640 1 -- 172.21.15.33:0/4060000732 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5597c19d3850 con 0x5597c1a7b110 2024-09-16T00:40:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.706+0000 7faeeaffd640 10 client.19692 handle_command_reply: tid=0 2024-09-16T00:40:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.706+0000 7faf14194740 1 client.19692 shutdown 2024-09-16T00:40:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.706+0000 7faf14194740 1 -- 172.21.15.33:0/4060000732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 msgr2=0x5597c1a68700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.706+0000 7faf14194740 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a68700 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7faf04003fc0 tx=0x7faf040046c0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.706+0000 7faf14194740 1 -- 172.21.15.33:0/4060000732 shutdown_connections 2024-09-16T00:40:02.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.706+0000 7faf14194740 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5597c1a7b110 0x5597c1a7d590 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.706+0000 7faf14194740 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a68700 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.706+0000 7faf14194740 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a681c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.706+0000 7faf14194740 1 --2- 172.21.15.33:0/4060000732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a604a0 0x5597c1a6aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.706+0000 7faeeaffd640 0 client.19692 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:02.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.706+0000 7faf14194740 1 -- 172.21.15.33:0/4060000732 >> 172.21.15.33:0/4060000732 conn(0x5597c1a5ca30 msgr2=0x5597c1a62710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:02.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.707+0000 7faf14194740 1 -- 172.21.15.33:0/4060000732 shutdown_connections 2024-09-16T00:40:02.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.707+0000 7faf14194740 1 -- 172.21.15.33:0/4060000732 wait complete. 2024-09-16T00:40:02.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.707+0000 7faf14194740 20 client.19692 trim_cache size 0 max 16384 2024-09-16T00:40:02.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.724+0000 7faf14194740 1 Processor -- start 2024-09-16T00:40:02.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.724+0000 7faf14194740 1 -- start start 2024-09-16T00:40:02.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf14194740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a604a0 0x5597c1a82e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf14194740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a833a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf14194740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a65420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf14194740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1b3b20 con 0x5597c1a6b680 2024-09-16T00:40:02.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf14194740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1b13f0 con 0x5597c1a60dc0 2024-09-16T00:40:02.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf14194740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1af240 con 0x5597c1a604a0 2024-09-16T00:40:02.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf0bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a65420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf0bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a65420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52554/0 (socket says 172.21.15.33:52554) 2024-09-16T00:40:02.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf0bfff640 1 -- 172.21.15.33:0/1891612945 learned_addr learned my addr 172.21.15.33:0/1891612945 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:02.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf0b7fe640 1 --2- 172.21.15.33:0/1891612945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a604a0 0x5597c1a82e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf0bfff640 1 -- 172.21.15.33:0/1891612945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a604a0 msgr2=0x5597c1a82e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf0bfff640 1 --2- 172.21.15.33:0/1891612945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a604a0 0x5597c1a82e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf0bfff640 1 -- 172.21.15.33:0/1891612945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 msgr2=0x5597c1a833a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf0affd640 1 --2- 172.21.15.33:0/1891612945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a833a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf0bfff640 1 --2- 172.21.15.33:0/1891612945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a833a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf0bfff640 1 -- 172.21.15.33:0/1891612945 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597c1a68d00 con 0x5597c1a6b680 2024-09-16T00:40:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf0affd640 1 --2- 172.21.15.33:0/1891612945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a833a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:40:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.725+0000 7faf0bfff640 1 --2- 172.21.15.33:0/1891612945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a65420 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7faf04003ba0 tx=0x7faf04004620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb2318868db5be1e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.726+0000 7faef17fa640 1 -- 172.21.15.33:0/1891612945 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf04025070 con 0x5597c1a6b680 2024-09-16T00:40:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.726+0000 7faef17fa640 1 -- 172.21.15.33:0/1891612945 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf04020560 con 0x5597c1a6b680 2024-09-16T00:40:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.726+0000 7faef17fa640 1 -- 172.21.15.33:0/1891612945 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf04020e60 con 0x5597c1a6b680 2024-09-16T00:40:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.726+0000 7faf14194740 1 -- 172.21.15.33:0/1891612945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 msgr2=0x5597c1a65420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.726+0000 7faf14194740 1 --2- 172.21.15.33:0/1891612945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a65420 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7faf04003ba0 tx=0x7faf04004620 comp rx=0 tx=0).stop 2024-09-16T00:40:02.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.726+0000 7faf14194740 1 -- 172.21.15.33:0/1891612945 shutdown_connections 2024-09-16T00:40:02.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.726+0000 7faf14194740 1 --2- 172.21.15.33:0/1891612945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a6b680 0x5597c1a65420 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.726+0000 7faf14194740 1 --2- 172.21.15.33:0/1891612945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a833a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.726+0000 7faf14194740 1 --2- 172.21.15.33:0/1891612945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a604a0 0x5597c1a82e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.726+0000 7faf14194740 1 -- 172.21.15.33:0/1891612945 >> 172.21.15.33:0/1891612945 conn(0x5597c1a5ca30 msgr2=0x5597c1a63870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:02.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.726+0000 7faf14194740 1 -- 172.21.15.33:0/1891612945 shutdown_connections 2024-09-16T00:40:02.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.726+0000 7faf14194740 1 -- 172.21.15.33:0/1891612945 wait complete. 2024-09-16T00:40:02.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.727+0000 7faf14194740 1 -- start start 2024-09-16T00:40:02.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.727+0000 7faf14194740 1 Processor -- start 2024-09-16T00:40:02.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.727+0000 7faf14194740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a604a0 0x5597c1a6c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.727+0000 7faf14194740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a6cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.727+0000 7faf14194740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a6b680 0x5597c1a6d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.727+0000 7faf14194740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1af8b0 con 0x5597c1a604a0 2024-09-16T00:40:02.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.727+0000 7faf14194740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1b0d20 con 0x5597c1a60dc0 2024-09-16T00:40:02.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.727+0000 7faf14194740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1aff80 con 0x5597c1a6b680 2024-09-16T00:40:02.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.727+0000 7faf14194740 10 client.0 _ll_register_callbacks cb 0x7faf04003c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:02.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf0b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a604a0 0x5597c1a6c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf0b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a604a0 0x5597c1a6c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52568/0 (socket says 172.21.15.33:52568) 2024-09-16T00:40:02.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf0b7fe640 1 -- 172.21.15.33:0/496878197 learned_addr learned my addr 172.21.15.33:0/496878197 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:02.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf0bfff640 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a6b680 0x5597c1a6d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf0affd640 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a6cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf0b7fe640 1 -- 172.21.15.33:0/496878197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a6b680 msgr2=0x5597c1a6d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:02.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf0b7fe640 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a6b680 0x5597c1a6d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf0b7fe640 1 -- 172.21.15.33:0/496878197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 msgr2=0x5597c1a6cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:02.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf0b7fe640 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a6cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf0b7fe640 1 -- 172.21.15.33:0/496878197 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597c1a5c860 con 0x5597c1a604a0 2024-09-16T00:40:02.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf0affd640 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a6cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:40:02.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf0b7fe640 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a604a0 0x5597c1a6c7c0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7faefc0bcbf0 tx=0x7faefc08cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff84f35ee4fbceb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:02.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faeeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:40:02.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faeeaffd640 1 -- 172.21.15.33:0/496878197 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc0c12a0 con 0x5597c1a604a0 2024-09-16T00:40:02.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faeeaffd640 1 -- 172.21.15.33:0/496878197 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7faefc009800 con 0x5597c1a604a0 2024-09-16T00:40:02.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf14194740 1 -- 172.21.15.33:0/496878197 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf04020560 con 0x5597c1a604a0 2024-09-16T00:40:02.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faeeaffd640 1 -- 172.21.15.33:0/496878197 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7faefc0c5b60 con 0x5597c1a604a0 2024-09-16T00:40:02.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf14194740 10 client.19698 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:02.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.728+0000 7faf14194740 1 -- 172.21.15.33:0/496878197 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5597c1ae5e30 con 0x5597c1a604a0 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.729+0000 7faeeaffd640 1 -- 172.21.15.33:0/496878197 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf04020560 con 0x5597c1a604a0 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.729+0000 7faeeaffd640 1 -- 172.21.15.33:0/496878197 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7faefc04d030 con 0x5597c1a604a0 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.729+0000 7faf14194740 10 client.19698 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.729+0000 7faf14194740 10 client.19698 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.729+0000 7faf14194740 20 client.19698 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.729+0000 7faf14194740 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5597c1ae64e0 0x5597c1ae8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.729+0000 7faf14194740 4 client.19698 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.729+0000 7faf14194740 1 -- 172.21.15.33:0/496878197 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5597c19cdd60 con 0x5597c1ae64e0 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.729+0000 7faf0affd640 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5597c1ae64e0 0x5597c1ae8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf0affd640 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5597c1ae64e0 0x5597c1ae8900 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6db3f47c4c221a43 server_cookie=5b9e8ba71d4c797f in_seq=0 out_seq=0 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faeeaffd640 10 client.19698 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faeeaffd640 1 -- 172.21.15.33:0/496878197 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5597c19cdd60 con 0x5597c1ae64e0 2024-09-16T00:40:02.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faeeaffd640 10 client.19698 handle_command_reply: tid=0 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf14194740 1 client.19698 shutdown 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf14194740 1 -- 172.21.15.33:0/496878197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a604a0 msgr2=0x5597c1a6c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf14194740 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a604a0 0x5597c1a6c7c0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7faefc0bcbf0 tx=0x7faefc08cb60 comp rx=0 tx=0).stop 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf14194740 1 -- 172.21.15.33:0/496878197 shutdown_connections 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf14194740 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5597c1ae64e0 0x5597c1ae8900 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf14194740 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597c1a6b680 0x5597c1a6d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faeeaffd640 0 client.19698 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf14194740 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597c1a60dc0 0x5597c1a6cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf14194740 1 --2- 172.21.15.33:0/496878197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5597c1a604a0 0x5597c1a6c7c0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf14194740 1 -- 172.21.15.33:0/496878197 >> 172.21.15.33:0/496878197 conn(0x5597c1a5ca30 msgr2=0x5597c1a69a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf14194740 1 -- 172.21.15.33:0/496878197 shutdown_connections 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf14194740 1 -- 172.21.15.33:0/496878197 wait complete. 2024-09-16T00:40:02.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.730+0000 7faf14194740 20 client.19698 trim_cache size 0 max 16384 2024-09-16T00:40:02.747 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:40:02.747 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:40:02.747 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:40:02.747 INFO:teuthology.orchestra.run.smithi033.stdout: "73c80774-a255-49e6-860f-ba6fe4154c29": { 2024-09-16T00:40:02.747 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:40:02.747 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "73c80774-a255-49e6-860f-ba6fe4154c29", 2024-09-16T00:40:02.747 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:40:02.747 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:40:02.747 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:40:02.747 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:40:02.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.731+0000 7faf11dbb640 1 -- 172.21.15.33:0/560242277 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faef40a3240 msgr2=0x7faef40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:02.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.731+0000 7faf11dbb640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faef40a3240 0x7faef40a5700 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7faf0400bba0 tx=0x7faf04004730 comp rx=0 tx=0).stop 2024-09-16T00:40:02.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.731+0000 7faf11dbb640 1 -- 172.21.15.33:0/560242277 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102d40 msgr2=0x7faf0c1009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:02.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.731+0000 7faf11dbb640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102d40 0x7faf0c1009a0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7faefc005cc0 tx=0x7faefc02f630 comp rx=0 tx=0).stop 2024-09-16T00:40:02.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.731+0000 7faf11dbb640 1 -- 172.21.15.33:0/560242277 shutdown_connections 2024-09-16T00:40:02.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.731+0000 7faf11dbb640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7faef40a3240 0x7faef40a5700 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.731+0000 7faf11dbb640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0c10c170 0x7faf0c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.731+0000 7faf11dbb640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf0c103660 0x7faf0c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.731+0000 7faf11dbb640 1 --2- 172.21.15.33:0/560242277 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf0c102d40 0x7faf0c1009a0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:02.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.731+0000 7faf11dbb640 1 -- 172.21.15.33:0/560242277 >> 172.21.15.33:0/560242277 conn(0x7faf0c0fc7a0 msgr2=0x7faf0c0fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:02.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.731+0000 7faf11dbb640 1 -- 172.21.15.33:0/560242277 shutdown_connections 2024-09-16T00:40:02.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:02.731+0000 7faf11dbb640 1 -- 172.21.15.33:0/560242277 wait complete. 2024-09-16T00:40:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:02 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:02.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:02 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:03.099 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.14 MB/sec execute 429 sec latency 22652.899 ms 2024-09-16T00:40:03.568 INFO:tasks.cephfs.filesystem:scrub status for tag:73c80774-a255-49e6-860f-ba6fe4154c29 - {'path': '/', 'tag': '73c80774-a255-49e6-860f-ba6fe4154c29', 'options': 'recursive,force'} 2024-09-16T00:40:03.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:03 smithi033 ceph-mon[31243]: pgmap v1037: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 268 KiB/s wr, 41 op/s 2024-09-16T00:40:03.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:03 smithi083 ceph-mon[35242]: pgmap v1037: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 268 KiB/s wr, 41 op/s 2024-09-16T00:40:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:03 smithi060 ceph-mon[35185]: pgmap v1037: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 268 KiB/s wr, 41 op/s 2024-09-16T00:40:04.099 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.10 MB/sec execute 430 sec latency 23653.021 ms 2024-09-16T00:40:05.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.06 MB/sec execute 431 sec latency 24653.163 ms 2024-09-16T00:40:05.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:05 smithi033 ceph-mon[31243]: pgmap v1038: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 23 KiB/s wr, 39 op/s 2024-09-16T00:40:05.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:05 smithi083 ceph-mon[35242]: pgmap v1038: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 23 KiB/s wr, 39 op/s 2024-09-16T00:40:05.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:05 smithi060 ceph-mon[35185]: pgmap v1038: 129 pgs: 129 active+clean; 696 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 23 KiB/s wr, 39 op/s 2024-09-16T00:40:06.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 18.02 MB/sec execute 432 sec latency 25653.265 ms 2024-09-16T00:40:07.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 17.98 MB/sec execute 433 sec latency 26653.360 ms 2024-09-16T00:40:07.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:07 smithi033 ceph-mon[31243]: pgmap v1039: 129 pgs: 129 active+clean; 688 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 22 KiB/s wr, 53 op/s 2024-09-16T00:40:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:07 smithi083 ceph-mon[35242]: pgmap v1039: 129 pgs: 129 active+clean; 688 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 22 KiB/s wr, 53 op/s 2024-09-16T00:40:07.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:07 smithi060 ceph-mon[35185]: pgmap v1039: 129 pgs: 129 active+clean; 688 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 22 KiB/s wr, 53 op/s 2024-09-16T00:40:08.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 17.93 MB/sec execute 434 sec latency 27653.423 ms 2024-09-16T00:40:09.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 17.89 MB/sec execute 435 sec latency 28653.530 ms 2024-09-16T00:40:09.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:09 smithi033 ceph-mon[31243]: pgmap v1040: 129 pgs: 129 active+clean; 685 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 28 KiB/s wr, 54 op/s 2024-09-16T00:40:09.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:09 smithi083 ceph-mon[35242]: pgmap v1040: 129 pgs: 129 active+clean; 685 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 28 KiB/s wr, 54 op/s 2024-09-16T00:40:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:09 smithi060 ceph-mon[35185]: pgmap v1040: 129 pgs: 129 active+clean; 685 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 28 KiB/s wr, 54 op/s 2024-09-16T00:40:10.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 17.85 MB/sec execute 436 sec latency 29653.641 ms 2024-09-16T00:40:11.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 17.81 MB/sec execute 437 sec latency 30653.758 ms 2024-09-16T00:40:11.525 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:40:11.791 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:40:11.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:11 smithi033 ceph-mon[31243]: pgmap v1041: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 34 KiB/s wr, 55 op/s 2024-09-16T00:40:11.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:11 smithi083 ceph-mon[35242]: pgmap v1041: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 34 KiB/s wr, 55 op/s 2024-09-16T00:40:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:11 smithi060 ceph-mon[35185]: pgmap v1041: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 34 KiB/s wr, 55 op/s 2024-09-16T00:40:12.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 17.77 MB/sec execute 438 sec latency 31653.789 ms 2024-09-16T00:40:12.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.312+0000 7f189f345640 1 -- 172.21.15.33:0/1968329219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1898102570 msgr2=0x7f18981029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:12.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.312+0000 7f189f345640 1 --2- 172.21.15.33:0/1968329219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1898102570 0x7f18981029b0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f1888009900 tx=0x7f188802a480 comp rx=0 tx=0).stop 2024-09-16T00:40:12.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.312+0000 7f189f345640 1 -- 172.21.15.33:0/1968329219 shutdown_connections 2024-09-16T00:40:12.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.312+0000 7f189f345640 1 --2- 172.21.15.33:0/1968329219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1898102ef0 0x7f189810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:12.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.312+0000 7f189f345640 1 --2- 172.21.15.33:0/1968329219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1898102570 0x7f18981029b0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.312+0000 7f189f345640 1 --2- 172.21.15.33:0/1968329219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898100e20 0x7f1898101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.312+0000 7f189f345640 1 -- 172.21.15.33:0/1968329219 >> 172.21.15.33:0/1968329219 conn(0x7f18980fc7d0 msgr2=0x7f18980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.312+0000 7f189f345640 1 -- 172.21.15.33:0/1968329219 shutdown_connections 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.312+0000 7f189f345640 1 -- 172.21.15.33:0/1968329219 wait complete. 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189f345640 1 Processor -- start 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189f345640 1 -- start start 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189f345640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898100e20 0x7f18980fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189f345640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1898102570 0x7f18980fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189f345640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1898102ef0 0x7f18980fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189f345640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1898110f80 con 0x7f1898102ef0 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189f345640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1898110e00 con 0x7f1898100e20 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189f345640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1898111100 con 0x7f1898102570 2024-09-16T00:40:12.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189d8e2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1898102ef0 0x7f18980fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189d8e2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1898102ef0 0x7f18980fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56950/0 (socket says 172.21.15.33:56950) 2024-09-16T00:40:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189d8e2640 1 -- 172.21.15.33:0/3047614021 learned_addr learned my addr 172.21.15.33:0/3047614021 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189c8e0640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1898102570 0x7f18980fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189d0e1640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898100e20 0x7f18980fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189d8e2640 1 -- 172.21.15.33:0/3047614021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1898102570 msgr2=0x7f18980fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189d8e2640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1898102570 0x7f18980fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189d8e2640 1 -- 172.21.15.33:0/3047614021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898100e20 msgr2=0x7f18980fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189d8e2640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898100e20 0x7f18980fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.313+0000 7f189d8e2640 1 -- 172.21.15.33:0/3047614021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18981a6970 con 0x7f1898102ef0 2024-09-16T00:40:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.314+0000 7f189d8e2640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1898102ef0 0x7f18980fd700 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f189000eaa0 tx=0x7f189000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.314+0000 7f188e7fc640 1 -- 172.21.15.33:0/3047614021 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189000cdb0 con 0x7f1898102ef0 2024-09-16T00:40:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.314+0000 7f188e7fc640 1 -- 172.21.15.33:0/3047614021 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1890005020 con 0x7f1898102ef0 2024-09-16T00:40:12.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.314+0000 7f189f345640 1 -- 172.21.15.33:0/3047614021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18981a6c60 con 0x7f1898102ef0 2024-09-16T00:40:12.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.314+0000 7f188e7fc640 1 -- 172.21.15.33:0/3047614021 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18900116b0 con 0x7f1898102ef0 2024-09-16T00:40:12.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.314+0000 7f189f345640 1 -- 172.21.15.33:0/3047614021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18981a7170 con 0x7f1898102ef0 2024-09-16T00:40:12.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.315+0000 7f189f345640 1 -- 172.21.15.33:0/3047614021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1898101200 con 0x7f1898102ef0 2024-09-16T00:40:12.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.316+0000 7f188e7fc640 1 -- 172.21.15.33:0/3047614021 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1890005560 con 0x7f1898102ef0 2024-09-16T00:40:12.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.317+0000 7f188e7fc640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18840a3240 0x7f18840a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:12.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.317+0000 7f189d0e1640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18840a3240 0x7f18840a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:12.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.317+0000 7f188e7fc640 1 -- 172.21.15.33:0/3047614021 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f18900cfec0 con 0x7f1898102ef0 2024-09-16T00:40:12.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.317+0000 7f189d0e1640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18840a3240 0x7f18840a5700 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1894009bd0 tx=0x7f1894005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:12.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.321+0000 7f188e7fc640 1 -- 172.21.15.33:0/3047614021 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1890016030 con 0x7f1898102ef0 2024-09-16T00:40:12.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.533+0000 7f189f345640 1 -- 172.21.15.33:0/3047614021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18980fe1b0 con 0x7f1898102ef0 2024-09-16T00:40:12.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.535+0000 7f188e7fc640 1 -- 172.21.15.33:0/3047614021 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f18980fe1b0 con 0x7f1898102ef0 2024-09-16T00:40:12.536 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:40:12.536 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:40:12.537 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:40:12.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.537+0000 7f189f345640 1 -- 172.21.15.33:0/3047614021 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18840a3240 msgr2=0x7f18840a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:12.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.537+0000 7f189f345640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18840a3240 0x7f18840a5700 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1894009bd0 tx=0x7f1894005f90 comp rx=0 tx=0).stop 2024-09-16T00:40:12.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.537+0000 7f189f345640 1 -- 172.21.15.33:0/3047614021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1898102ef0 msgr2=0x7f18980fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:12.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.537+0000 7f189f345640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1898102ef0 0x7f18980fd700 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f189000eaa0 tx=0x7f189000ef60 comp rx=0 tx=0).stop 2024-09-16T00:40:12.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.537+0000 7f189f345640 1 -- 172.21.15.33:0/3047614021 shutdown_connections 2024-09-16T00:40:12.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.537+0000 7f189f345640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f18840a3240 0x7f18840a5700 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:12.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.537+0000 7f189f345640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1898102ef0 0x7f18980fd700 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:12.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.537+0000 7f189f345640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1898102570 0x7f18980fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:12.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.537+0000 7f189f345640 1 --2- 172.21.15.33:0/3047614021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1898100e20 0x7f18980fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:12.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.537+0000 7f189f345640 1 -- 172.21.15.33:0/3047614021 >> 172.21.15.33:0/3047614021 conn(0x7f18980fc7d0 msgr2=0x7f1898069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:12.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.537+0000 7f189f345640 1 -- 172.21.15.33:0/3047614021 shutdown_connections 2024-09-16T00:40:12.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:12.537+0000 7f189f345640 1 -- 172.21.15.33:0/3047614021 wait complete. 2024-09-16T00:40:12.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:12 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:40:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:12 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:40:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:12 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:40:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:12 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3047614021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:40:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:12 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:40:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:12 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:40:12.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:12 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:40:12.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:12 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3047614021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:40:12.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:12 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:40:12.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:12 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:40:12.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:12 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:40:12.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:12 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3047614021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:40:13.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188253 17.73 MB/sec execute 439 sec latency 32653.875 ms 2024-09-16T00:40:13.274 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 878947d6 --release --await' 2024-09-16T00:40:13.274 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 878947d6 --release --await 2024-09-16T00:40:13.537 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:40:13.802 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:13 smithi033 ceph-mon[31243]: pgmap v1042: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 35 KiB/s wr, 55 op/s 2024-09-16T00:40:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:13 smithi083 ceph-mon[35242]: pgmap v1042: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 35 KiB/s wr, 55 op/s 2024-09-16T00:40:13.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:13 smithi060 ceph-mon[35185]: pgmap v1042: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 35 KiB/s wr, 55 op/s 2024-09-16T00:40:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.054+0000 7fd97034a640 1 -- 172.21.15.33:0/223524063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd968101de0 msgr2=0x7fd9681021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.054+0000 7fd97034a640 1 --2- 172.21.15.33:0/223524063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd968101de0 0x7fd9681021c0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fd964009900 tx=0x7fd96402a410 comp rx=0 tx=0).stop 2024-09-16T00:40:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.054+0000 7fd97034a640 1 -- 172.21.15.33:0/223524063 shutdown_connections 2024-09-16T00:40:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.054+0000 7fd97034a640 1 --2- 172.21.15.33:0/223524063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd968106f10 0x7fd96810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.054+0000 7fd97034a640 1 --2- 172.21.15.33:0/223524063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd968102790 0x7fd9681067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.054+0000 7fd97034a640 1 --2- 172.21.15.33:0/223524063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd968101de0 0x7fd9681021c0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.054+0000 7fd97034a640 1 -- 172.21.15.33:0/223524063 >> 172.21.15.33:0/223524063 conn(0x7fd9680fd8c0 msgr2=0x7fd9680ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.054+0000 7fd97034a640 1 -- 172.21.15.33:0/223524063 shutdown_connections 2024-09-16T00:40:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.054+0000 7fd97034a640 1 -- 172.21.15.33:0/223524063 wait complete. 2024-09-16T00:40:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.055+0000 7fd97034a640 1 Processor -- start 2024-09-16T00:40:14.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.055+0000 7fd97034a640 1 -- start start 2024-09-16T00:40:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.055+0000 7fd97034a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd968101de0 0x7fd9680744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.055+0000 7fd97034a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd968102790 0x7fd968074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.055+0000 7fd97034a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd968106f10 0x7fd968074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.055+0000 7fd97034a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd968111f70 con 0x7fd968101de0 2024-09-16T00:40:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.055+0000 7fd97034a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd968111df0 con 0x7fd968102790 2024-09-16T00:40:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.055+0000 7fd97034a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681120f0 con 0x7fd968106f10 2024-09-16T00:40:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.055+0000 7fd96e0e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd968101de0 0x7fd9680744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd96e0e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd968101de0 0x7fd9680744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56974/0 (socket says 172.21.15.33:56974) 2024-09-16T00:40:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd96e0e6640 1 -- 172.21.15.33:0/1806055240 learned_addr learned my addr 172.21.15.33:0/1806055240 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd96d8e5640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd968102790 0x7fd968074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd96e8e7640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd968106f10 0x7fd968074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd96e0e6640 1 -- 172.21.15.33:0/1806055240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd968106f10 msgr2=0x7fd968074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd968106f10 0x7fd968074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd96e0e6640 1 -- 172.21.15.33:0/1806055240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd968102790 msgr2=0x7fd968074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd968102790 0x7fd968074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd96e0e6640 1 -- 172.21.15.33:0/1806055240 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9681a67e0 con 0x7fd968101de0 2024-09-16T00:40:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd96d8e5640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd968102790 0x7fd968074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:40:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd96e8e7640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd968106f10 0x7fd968074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:40:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd968101de0 0x7fd9680744f0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fd964005ab0 tx=0x7fd9640042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd95f7fe640 1 -- 172.21.15.33:0/1806055240 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96403f030 con 0x7fd968101de0 2024-09-16T00:40:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd95f7fe640 1 -- 172.21.15.33:0/1806055240 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd964003ce0 con 0x7fd968101de0 2024-09-16T00:40:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd97034a640 1 -- 172.21.15.33:0/1806055240 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9681a6a70 con 0x7fd968101de0 2024-09-16T00:40:14.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd95f7fe640 1 -- 172.21.15.33:0/1806055240 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964035070 con 0x7fd968101de0 2024-09-16T00:40:14.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.056+0000 7fd97034a640 1 -- 172.21.15.33:0/1806055240 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9681a6fa0 con 0x7fd968101de0 2024-09-16T00:40:14.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.059+0000 7fd95f7fe640 1 -- 172.21.15.33:0/1806055240 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd964042030 con 0x7fd968101de0 2024-09-16T00:40:14.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.059+0000 7fd972723740 1 Processor -- start 2024-09-16T00:40:14.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd972723740 1 -- start start 2024-09-16T00:40:14.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd972723740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c362f1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd972723740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362f1980 0x555c362fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd972723740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362fc240 0x555c362fe630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd972723740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555c36256620 con 0x555c362f1060 2024-09-16T00:40:14.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd972723740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555c361fde50 con 0x555c362fc240 2024-09-16T00:40:14.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd972723740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555c362525e0 con 0x555c362f1980 2024-09-16T00:40:14.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd96e8e7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c362f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd96e8e7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c362f1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56984/0 (socket says 172.21.15.33:56984) 2024-09-16T00:40:14.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd96e8e7640 1 -- 172.21.15.33:0/4214818599 learned_addr learned my addr 172.21.15.33:0/4214818599 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:14.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/4214818599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362f1980 0x555c362fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd96d8e5640 1 --2- 172.21.15.33:0/4214818599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362fc240 0x555c362fe630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd95f7fe640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd9540a32c0 0x7fd9540a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd96e8e7640 1 -- 172.21.15.33:0/4214818599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362f1980 msgr2=0x555c362fbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:14.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd96e8e7640 1 --2- 172.21.15.33:0/4214818599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362f1980 0x555c362fbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd96e8e7640 1 -- 172.21.15.33:0/4214818599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362fc240 msgr2=0x555c362fe630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:14.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd96e8e7640 1 --2- 172.21.15.33:0/4214818599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362fc240 0x555c362fe630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd96e8e7640 1 -- 172.21.15.33:0/4214818599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c361eed40 con 0x555c362f1060 2024-09-16T00:40:14.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.060+0000 7fd96d8e5640 1 --2- 172.21.15.33:0/4214818599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362fc240 0x555c362fe630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:40:14.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.061+0000 7fd95f7fe640 1 -- 172.21.15.33:0/1806055240 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd9640f0730 con 0x7fd968101de0 2024-09-16T00:40:14.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.061+0000 7fd96e8e7640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd9540a32c0 0x7fd9540a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.061+0000 7fd96e8e7640 1 --2- 172.21.15.33:0/4214818599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c362f1440 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fd960008de0 tx=0x7fd96000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d1380f962d7a47 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:14.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.061+0000 7fd95cdea640 1 -- 172.21.15.33:0/4214818599 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96001f070 con 0x555c362f1060 2024-09-16T00:40:14.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.061+0000 7fd95cdea640 1 -- 172.21.15.33:0/4214818599 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd96000e3b0 con 0x555c362f1060 2024-09-16T00:40:14.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.061+0000 7fd95cdea640 1 -- 172.21.15.33:0/4214818599 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96000b5a0 con 0x555c362f1060 2024-09-16T00:40:14.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.062+0000 7fd972723740 1 -- 172.21.15.33:0/4214818599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 msgr2=0x555c362f1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:14.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.062+0000 7fd96e8e7640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd9540a32c0 0x7fd9540a5780 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd96001c000 tx=0x7fd96001f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:14.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.062+0000 7fd972723740 1 --2- 172.21.15.33:0/4214818599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c362f1440 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fd960008de0 tx=0x7fd96000d710 comp rx=0 tx=0).stop 2024-09-16T00:40:14.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.062+0000 7fd972723740 1 -- 172.21.15.33:0/4214818599 shutdown_connections 2024-09-16T00:40:14.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.062+0000 7fd972723740 1 --2- 172.21.15.33:0/4214818599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362fc240 0x555c362fe630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.062+0000 7fd972723740 1 --2- 172.21.15.33:0/4214818599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362f1980 0x555c362fbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.062+0000 7fd972723740 1 --2- 172.21.15.33:0/4214818599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c362f1440 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.062+0000 7fd972723740 1 -- 172.21.15.33:0/4214818599 >> 172.21.15.33:0/4214818599 conn(0x555c362ed5f0 msgr2=0x555c362ed9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:14.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.062+0000 7fd972723740 1 -- 172.21.15.33:0/4214818599 shutdown_connections 2024-09-16T00:40:14.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.062+0000 7fd972723740 1 -- 172.21.15.33:0/4214818599 wait complete. 2024-09-16T00:40:14.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.062+0000 7fd972723740 1 -- start start 2024-09-16T00:40:14.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.062+0000 7fd972723740 1 Processor -- start 2024-09-16T00:40:14.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd972723740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1060 0x555c362f9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd972723740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1980 0x555c362f9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd972723740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 0x555c362f9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd972723740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555c362fee10 con 0x555c362f1980 2024-09-16T00:40:14.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd972723740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555c361def10 con 0x555c362f1060 2024-09-16T00:40:14.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd972723740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555c362fef90 con 0x555c362fc240 2024-09-16T00:40:14.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd972723740 10 client.0 _ll_register_callbacks cb 0x555c36202db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:14.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd96d8e5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1980 0x555c362f9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd96d8e5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1980 0x555c362f9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56998/0 (socket says 172.21.15.33:56998) 2024-09-16T00:40:14.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd96d8e5640 1 -- 172.21.15.33:0/2484496247 learned_addr learned my addr 172.21.15.33:0/2484496247 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:14.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd96e8e7640 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 0x555c362f9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1060 0x555c362f9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd96d8e5640 1 -- 172.21.15.33:0/2484496247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 msgr2=0x555c362f9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:14.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd96d8e5640 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 0x555c362f9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd96d8e5640 1 -- 172.21.15.33:0/2484496247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1060 msgr2=0x555c362f9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:14.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd96d8e5640 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1060 0x555c362f9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd96d8e5640 1 -- 172.21.15.33:0/2484496247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c35f65580 con 0x555c362f1980 2024-09-16T00:40:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd96d8e5640 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1980 0x555c362f9690 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fd95800f660 tx=0x7fd95800fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb49896a2496ba1f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd951ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:40:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.063+0000 7fd951ffb640 1 -- 172.21.15.33:0/2484496247 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958004220 con 0x555c362f1980 2024-09-16T00:40:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd951ffb640 1 -- 172.21.15.33:0/2484496247 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd9580043c0 con 0x555c362f1980 2024-09-16T00:40:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd972723740 1 -- 172.21.15.33:0/2484496247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd96000e3b0 con 0x555c362f1980 2024-09-16T00:40:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd951ffb640 1 -- 172.21.15.33:0/2484496247 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd95800fe60 con 0x555c362f1980 2024-09-16T00:40:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd972723740 10 client.19716 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd972723740 1 -- 172.21.15.33:0/2484496247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555c3602f960 con 0x555c362f1980 2024-09-16T00:40:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd951ffb640 1 -- 172.21.15.33:0/2484496247 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd96000e3b0 con 0x555c362f1980 2024-09-16T00:40:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd951ffb640 1 -- 172.21.15.33:0/2484496247 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fd9580047a0 con 0x555c362f1980 2024-09-16T00:40:14.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd972723740 10 client.19716 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:14.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd972723740 10 client.19716 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:40:14.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd972723740 20 client.19716 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:14.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd972723740 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c3630bde0 0x555c3630e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd972723740 4 client.19716 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:40:14.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.064+0000 7fd972723740 1 -- 172.21.15.33:0/2484496247 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c36263b90 con 0x555c3630bde0 2024-09-16T00:40:14.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.065+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c3630bde0 0x555c3630e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.065+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c3630bde0 0x555c3630e2a0 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d9c95caefeaf02d server_cookie=5a9f25ce225afcbf in_seq=0 out_seq=0 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.065+0000 7fd951ffb640 10 client.19716 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd951ffb640 1 -- 172.21.15.33:0/2484496247 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555c36263b90 con 0x555c3630bde0 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd951ffb640 10 client.19716 handle_command_reply: tid=0 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd972723740 1 client.19716 shutdown 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd972723740 1 -- 172.21.15.33:0/2484496247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1980 msgr2=0x555c362f9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd972723740 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1980 0x555c362f9690 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fd95800f660 tx=0x7fd95800fb20 comp rx=0 tx=0).stop 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd972723740 1 -- 172.21.15.33:0/2484496247 shutdown_connections 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd972723740 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c3630bde0 0x555c3630e2a0 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd972723740 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 0x555c362f9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd972723740 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1980 0x555c362f9690 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd972723740 1 --2- 172.21.15.33:0/2484496247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1060 0x555c362f9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd972723740 1 -- 172.21.15.33:0/2484496247 >> 172.21.15.33:0/2484496247 conn(0x555c362ed5f0 msgr2=0x555c362f32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:14.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd951ffb640 0 client.19716 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:14.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd972723740 1 -- 172.21.15.33:0/2484496247 shutdown_connections 2024-09-16T00:40:14.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd972723740 1 -- 172.21.15.33:0/2484496247 wait complete. 2024-09-16T00:40:14.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.066+0000 7fd972723740 20 client.19716 trim_cache size 0 max 16384 2024-09-16T00:40:14.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.083+0000 7fd972723740 1 Processor -- start 2024-09-16T00:40:14.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.083+0000 7fd972723740 1 -- start start 2024-09-16T00:40:14.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.083+0000 7fd972723740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c362fb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.083+0000 7fd972723740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1980 0x555c36313bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.083+0000 7fd972723740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 0x555c363140f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.083+0000 7fd972723740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681b3aa0 con 0x555c362f1060 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.083+0000 7fd972723740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681b1370 con 0x555c362f1980 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.083+0000 7fd972723740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681af1c0 con 0x555c362fc240 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.084+0000 7fd96e0e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c362fb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.084+0000 7fd96e0e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c362fb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57000/0 (socket says 172.21.15.33:57000) 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.084+0000 7fd96e0e6640 1 -- 172.21.15.33:0/3593070696 learned_addr learned my addr 172.21.15.33:0/3593070696 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.084+0000 7fd96e0e6640 1 -- 172.21.15.33:0/3593070696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 msgr2=0x555c363140f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.084+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/3593070696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 0x555c363140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.084+0000 7fd96e0e6640 1 -- 172.21.15.33:0/3593070696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1980 msgr2=0x555c36313bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.084+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/3593070696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1980 0x555c36313bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.084+0000 7fd96e0e6640 1 -- 172.21.15.33:0/3593070696 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c362f8100 con 0x555c362f1060 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.084+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/3593070696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c362fb0c0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fd9640b8cd0 tx=0x7fd9640f2b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f03ffdb432d61f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.084+0000 7fd953fff640 1 -- 172.21.15.33:0/3593070696 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964049040 con 0x555c362f1060 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.084+0000 7fd953fff640 1 -- 172.21.15.33:0/3593070696 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9640bcf70 con 0x555c362f1060 2024-09-16T00:40:14.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.084+0000 7fd953fff640 1 -- 172.21.15.33:0/3593070696 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96403bd70 con 0x555c362f1060 2024-09-16T00:40:14.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.085+0000 7fd972723740 1 -- 172.21.15.33:0/3593070696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 msgr2=0x555c362fb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:14.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.085+0000 7fd972723740 1 --2- 172.21.15.33:0/3593070696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c362fb0c0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fd9640b8cd0 tx=0x7fd9640f2b80 comp rx=0 tx=0).stop 2024-09-16T00:40:14.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.085+0000 7fd972723740 1 -- 172.21.15.33:0/3593070696 shutdown_connections 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.085+0000 7fd972723740 1 --2- 172.21.15.33:0/3593070696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 0x555c363140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.085+0000 7fd972723740 1 --2- 172.21.15.33:0/3593070696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1980 0x555c36313bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.085+0000 7fd972723740 1 --2- 172.21.15.33:0/3593070696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c362fb0c0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.085+0000 7fd972723740 1 -- 172.21.15.33:0/3593070696 >> 172.21.15.33:0/3593070696 conn(0x555c362ed5f0 msgr2=0x555c362fa9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.085+0000 7fd972723740 1 -- 172.21.15.33:0/3593070696 shutdown_connections 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.085+0000 7fd972723740 1 -- 172.21.15.33:0/3593070696 wait complete. 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.085+0000 7fd972723740 1 -- start start 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.085+0000 7fd972723740 1 Processor -- start 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd972723740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c36300130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd972723740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1980 0x555c36300670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd972723740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 0x555c36300bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd972723740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681af830 con 0x555c362f1060 2024-09-16T00:40:14.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd972723740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681b0ca0 con 0x555c362f1980 2024-09-16T00:40:14.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd972723740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681aff00 con 0x555c362fc240 2024-09-16T00:40:14.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd972723740 10 client.0 _ll_register_callbacks cb 0x7fd95800ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:14.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd96e0e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c36300130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd96e0e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c36300130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57006/0 (socket says 172.21.15.33:57006) 2024-09-16T00:40:14.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd96e0e6640 1 -- 172.21.15.33:0/2930368278 learned_addr learned my addr 172.21.15.33:0/2930368278 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:14.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd96d8e5640 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1980 0x555c36300670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd96e8e7640 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 0x555c36300bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd96e0e6640 1 -- 172.21.15.33:0/2930368278 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 msgr2=0x555c36300bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:14.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 0x555c36300bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd96e0e6640 1 -- 172.21.15.33:0/2930368278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1980 msgr2=0x555c36300670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:14.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1980 0x555c36300670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd96e0e6640 1 -- 172.21.15.33:0/2930368278 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c362f17b0 con 0x555c362f1060 2024-09-16T00:40:14.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd96e0e6640 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c36300130 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fd9640bc800 tx=0x7fd9640f6150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39b1d5acaded2ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:14.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd951ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:40:14.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd951ffb640 1 -- 172.21.15.33:0/2930368278 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964049040 con 0x555c362f1060 2024-09-16T00:40:14.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd951ffb640 1 -- 172.21.15.33:0/2930368278 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fd96403bd70 con 0x555c362f1060 2024-09-16T00:40:14.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.086+0000 7fd972723740 1 -- 172.21.15.33:0/2930368278 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9640bcf70 con 0x555c362f1060 2024-09-16T00:40:14.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.087+0000 7fd951ffb640 1 -- 172.21.15.33:0/2930368278 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd9640bc870 con 0x555c362f1060 2024-09-16T00:40:14.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.087+0000 7fd972723740 10 client.19722 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:14.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.087+0000 7fd972723740 1 -- 172.21.15.33:0/2930368278 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555c36376e90 con 0x555c362f1060 2024-09-16T00:40:14.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.087+0000 7fd951ffb640 1 -- 172.21.15.33:0/2930368278 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9640bcf70 con 0x555c362f1060 2024-09-16T00:40:14.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.087+0000 7fd951ffb640 1 -- 172.21.15.33:0/2930368278 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fd9640f0310 con 0x555c362f1060 2024-09-16T00:40:14.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.087+0000 7fd972723740 10 client.19722 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:14.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.087+0000 7fd972723740 10 client.19722 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:40:14.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.087+0000 7fd972723740 20 client.19722 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:14.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.087+0000 7fd972723740 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c36377550 0x555c36379940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:14.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.087+0000 7fd972723740 4 client.19722 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "878947d6", "release": true, "await": true}] 2024-09-16T00:40:14.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.087+0000 7fd972723740 1 -- 172.21.15.33:0/2930368278 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "878947d6", "release": true, "await": true}) -- 0x555c3625e0a0 con 0x555c36377550 2024-09-16T00:40:14.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.087+0000 7fd96d8e5640 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c36377550 0x555c36379940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:14.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.088+0000 7fd951ffb640 10 client.19722 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:14.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.088+0000 7fd96d8e5640 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c36377550 0x555c36379940 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce433460833de975 server_cookie=7ad409fd82696203 in_seq=0 out_seq=0 2024-09-16T00:40:14.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188254 17.69 MB/sec execute 440 sec latency 33653.933 ms 2024-09-16T00:40:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.140+0000 7fd951ffb640 1 -- 172.21.15.33:0/2930368278 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555c3625e0a0 con 0x555c36377550 2024-09-16T00:40:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.140+0000 7fd951ffb640 10 client.19722 handle_command_reply: tid=0 2024-09-16T00:40:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.140+0000 7fd972723740 1 client.19722 shutdown 2024-09-16T00:40:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.141+0000 7fd972723740 1 -- 172.21.15.33:0/2930368278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 msgr2=0x555c36300130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.141+0000 7fd972723740 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c36300130 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fd9640bc800 tx=0x7fd9640f6150 comp rx=0 tx=0).stop 2024-09-16T00:40:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.141+0000 7fd972723740 1 -- 172.21.15.33:0/2930368278 shutdown_connections 2024-09-16T00:40:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.141+0000 7fd972723740 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x555c36377550 0x555c36379940 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.141+0000 7fd972723740 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c362fc240 0x555c36300bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.141+0000 7fd972723740 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555c362f1980 0x555c36300670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.141+0000 7fd972723740 1 --2- 172.21.15.33:0/2930368278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555c362f1060 0x555c36300130 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.141+0000 7fd951ffb640 0 client.19722 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.141+0000 7fd972723740 1 -- 172.21.15.33:0/2930368278 >> 172.21.15.33:0/2930368278 conn(0x555c362ed5f0 msgr2=0x555c362ec750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.141+0000 7fd972723740 1 -- 172.21.15.33:0/2930368278 shutdown_connections 2024-09-16T00:40:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.141+0000 7fd972723740 1 -- 172.21.15.33:0/2930368278 wait complete. 2024-09-16T00:40:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.141+0000 7fd972723740 20 client.19722 trim_cache size 0 max 16384 2024-09-16T00:40:14.144 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:40:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:40:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:40:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 80, 2024-09-16T00:40:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:40:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: "878947d6": { 2024-09-16T00:40:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 80, 2024-09-16T00:40:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:40:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:40:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:40:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:40:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:40:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:40:14.146 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 154.4, 2024-09-16T00:40:14.146 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:40:14.146 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:40:14.146 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:40:14.146 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:40:14.146 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:40:14.146 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:40:14.146 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:40:14.146 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:40:14.146 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:40:14.146 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:40:14.147 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:40:14.147 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:40:14.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.142+0000 7fd97034a640 1 -- 172.21.15.33:0/1806055240 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd9540a32c0 msgr2=0x7fd9540a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:14.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.142+0000 7fd97034a640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd9540a32c0 0x7fd9540a5780 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd96001c000 tx=0x7fd96001f000 comp rx=0 tx=0).stop 2024-09-16T00:40:14.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.142+0000 7fd97034a640 1 -- 172.21.15.33:0/1806055240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd968101de0 msgr2=0x7fd9680744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:14.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.142+0000 7fd97034a640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd968101de0 0x7fd9680744f0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fd964005ab0 tx=0x7fd9640042f0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.142+0000 7fd97034a640 1 -- 172.21.15.33:0/1806055240 shutdown_connections 2024-09-16T00:40:14.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.142+0000 7fd97034a640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fd9540a32c0 0x7fd9540a5780 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.142+0000 7fd97034a640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd968106f10 0x7fd968074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.142+0000 7fd97034a640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd968102790 0x7fd968074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.142+0000 7fd97034a640 1 --2- 172.21.15.33:0/1806055240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd968101de0 0x7fd9680744f0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:14.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.142+0000 7fd97034a640 1 -- 172.21.15.33:0/1806055240 >> 172.21.15.33:0/1806055240 conn(0x7fd9680fd8c0 msgr2=0x7fd9680fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:14.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.142+0000 7fd97034a640 1 -- 172.21.15.33:0/1806055240 shutdown_connections 2024-09-16T00:40:14.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:14.142+0000 7fd97034a640 1 -- 172.21.15.33:0/1806055240 wait complete. 2024-09-16T00:40:14.996 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '878947d6', total seconds elapsed: 37.9 2024-09-16T00:40:15.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 188811 17.72 MB/sec execute 441 sec latency 33717.972 ms 2024-09-16T00:40:15.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:15 smithi033 ceph-mon[31243]: pgmap v1043: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 23 KiB/s wr, 47 op/s 2024-09-16T00:40:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:15 smithi083 ceph-mon[35242]: pgmap v1043: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 23 KiB/s wr, 47 op/s 2024-09-16T00:40:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:15 smithi060 ceph-mon[35185]: pgmap v1043: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 23 KiB/s wr, 47 op/s 2024-09-16T00:40:16.100 INFO:tasks.workunit.client.0.smithi033.stdout: 10 189355 17.73 MB/sec execute 442 sec latency 90.683 ms 2024-09-16T00:40:17.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 189894 17.77 MB/sec execute 443 sec latency 66.814 ms 2024-09-16T00:40:17.837 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:17 smithi033 ceph-mon[31243]: pgmap v1044: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 75 KiB/s wr, 33 op/s 2024-09-16T00:40:19.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:17 smithi083 ceph-mon[35242]: pgmap v1044: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 75 KiB/s wr, 33 op/s 2024-09-16T00:40:19.068 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:17 smithi060 ceph-mon[35185]: pgmap v1044: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 75 KiB/s wr, 33 op/s 2024-09-16T00:40:19.069 INFO:tasks.workunit.client.0.smithi033.stdout: 10 190412 17.78 MB/sec execute 444 sec latency 86.644 ms 2024-09-16T00:40:19.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 190930 17.82 MB/sec execute 445 sec latency 96.514 ms 2024-09-16T00:40:19.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:19 smithi033 ceph-mon[31243]: pgmap v1045: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 240 KiB/s wr, 43 op/s 2024-09-16T00:40:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:19 smithi083 ceph-mon[35242]: pgmap v1045: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 240 KiB/s wr, 43 op/s 2024-09-16T00:40:19.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:19 smithi060 ceph-mon[35185]: pgmap v1045: 129 pgs: 129 active+clean; 677 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 240 KiB/s wr, 43 op/s 2024-09-16T00:40:20.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 191593 17.85 MB/sec execute 446 sec latency 54.120 ms 2024-09-16T00:40:21.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 192086 17.87 MB/sec execute 447 sec latency 82.634 ms 2024-09-16T00:40:21.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:21 smithi033 ceph-mon[31243]: pgmap v1046: 129 pgs: 129 active+clean; 679 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 896 KiB/s wr, 129 op/s 2024-09-16T00:40:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:21 smithi083 ceph-mon[35242]: pgmap v1046: 129 pgs: 129 active+clean; 679 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 896 KiB/s wr, 129 op/s 2024-09-16T00:40:21.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:21 smithi060 ceph-mon[35185]: pgmap v1046: 129 pgs: 129 active+clean; 679 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 896 KiB/s wr, 129 op/s 2024-09-16T00:40:22.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 192563 17.89 MB/sec execute 448 sec latency 100.628 ms 2024-09-16T00:40:23.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 193194 17.93 MB/sec execute 449 sec latency 60.793 ms 2024-09-16T00:40:23.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:23 smithi033 ceph-mon[31243]: pgmap v1047: 129 pgs: 129 active+clean; 684 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 1.8 MiB/s wr, 245 op/s 2024-09-16T00:40:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:23 smithi083 ceph-mon[35242]: pgmap v1047: 129 pgs: 129 active+clean; 684 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 1.8 MiB/s wr, 245 op/s 2024-09-16T00:40:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:23 smithi060 ceph-mon[35185]: pgmap v1047: 129 pgs: 129 active+clean; 684 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 1.8 MiB/s wr, 245 op/s 2024-09-16T00:40:24.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 193757 17.96 MB/sec execute 450 sec latency 68.766 ms 2024-09-16T00:40:25.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 194124 17.96 MB/sec execute 451 sec latency 94.927 ms 2024-09-16T00:40:25.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:25 smithi033 ceph-mon[31243]: pgmap v1048: 129 pgs: 129 active+clean; 687 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 300 op/s 2024-09-16T00:40:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:25 smithi083 ceph-mon[35242]: pgmap v1048: 129 pgs: 129 active+clean; 687 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 300 op/s 2024-09-16T00:40:25.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:25 smithi060 ceph-mon[35185]: pgmap v1048: 129 pgs: 129 active+clean; 687 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 300 op/s 2024-09-16T00:40:26.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 194688 17.98 MB/sec execute 452 sec latency 108.139 ms 2024-09-16T00:40:27.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 195221 18.00 MB/sec execute 453 sec latency 140.558 ms 2024-09-16T00:40:27.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:27 smithi033 ceph-mon[31243]: pgmap v1049: 129 pgs: 129 active+clean; 691 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-09-16T00:40:27.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:27 smithi083 ceph-mon[35242]: pgmap v1049: 129 pgs: 129 active+clean; 691 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-09-16T00:40:27.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:27 smithi060 ceph-mon[35185]: pgmap v1049: 129 pgs: 129 active+clean; 691 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-09-16T00:40:28.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 195875 18.05 MB/sec execute 454 sec latency 89.476 ms 2024-09-16T00:40:29.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 196375 18.07 MB/sec execute 455 sec latency 62.687 ms 2024-09-16T00:40:29.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:29 smithi083 ceph-mon[35242]: pgmap v1050: 129 pgs: 129 active+clean; 692 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 362 op/s 2024-09-16T00:40:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:29 smithi060 ceph-mon[35185]: pgmap v1050: 129 pgs: 129 active+clean; 692 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 362 op/s 2024-09-16T00:40:30.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:29 smithi033 ceph-mon[31243]: pgmap v1050: 129 pgs: 129 active+clean; 692 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 362 op/s 2024-09-16T00:40:30.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 196893 18.09 MB/sec execute 456 sec latency 98.529 ms 2024-09-16T00:40:31.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 197422 18.12 MB/sec execute 457 sec latency 171.909 ms 2024-09-16T00:40:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:31 smithi083 ceph-mon[35242]: pgmap v1051: 129 pgs: 129 active+clean; 702 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 449 op/s 2024-09-16T00:40:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:31 smithi060 ceph-mon[35185]: pgmap v1051: 129 pgs: 129 active+clean; 702 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 449 op/s 2024-09-16T00:40:32.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:31 smithi033 ceph-mon[31243]: pgmap v1051: 129 pgs: 129 active+clean; 702 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 449 op/s 2024-09-16T00:40:32.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 197944 18.14 MB/sec execute 458 sec latency 195.963 ms 2024-09-16T00:40:33.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 198304 18.14 MB/sec execute 459 sec latency 92.842 ms 2024-09-16T00:40:33.569 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:40:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:33 smithi083 ceph-mon[35242]: pgmap v1052: 129 pgs: 129 active+clean; 705 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.2 MiB/s wr, 399 op/s 2024-09-16T00:40:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:33 smithi060 ceph-mon[35185]: pgmap v1052: 129 pgs: 129 active+clean; 705 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.2 MiB/s wr, 399 op/s 2024-09-16T00:40:33.986 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:40:34.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:33 smithi033 ceph-mon[31243]: pgmap v1052: 129 pgs: 129 active+clean; 705 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.2 MiB/s wr, 399 op/s 2024-09-16T00:40:34.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 198770 18.14 MB/sec execute 460 sec latency 86.720 ms 2024-09-16T00:40:34.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.537+0000 7fe52340a640 1 -- 172.21.15.33:0/2012565270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe51c102570 msgr2=0x7fe51c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:34.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.537+0000 7fe52340a640 1 --2- 172.21.15.33:0/2012565270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe51c102570 0x7fe51c1029b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe50c009890 tx=0x7fe50c02a410 comp rx=0 tx=0).stop 2024-09-16T00:40:34.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.537+0000 7fe52340a640 1 -- 172.21.15.33:0/2012565270 shutdown_connections 2024-09-16T00:40:34.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.537+0000 7fe52340a640 1 --2- 172.21.15.33:0/2012565270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe51c102ef0 0x7fe51c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.537+0000 7fe52340a640 1 --2- 172.21.15.33:0/2012565270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe51c102570 0x7fe51c1029b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.537+0000 7fe52340a640 1 --2- 172.21.15.33:0/2012565270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe51c100e20 0x7fe51c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.537+0000 7fe52340a640 1 -- 172.21.15.33:0/2012565270 >> 172.21.15.33:0/2012565270 conn(0x7fe51c0fc7d0 msgr2=0x7fe51c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.537+0000 7fe52340a640 1 -- 172.21.15.33:0/2012565270 shutdown_connections 2024-09-16T00:40:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.537+0000 7fe52340a640 1 -- 172.21.15.33:0/2012565270 wait complete. 2024-09-16T00:40:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe52340a640 1 Processor -- start 2024-09-16T00:40:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe52340a640 1 -- start start 2024-09-16T00:40:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe52340a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe51c100e20 0x7fe51c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe52340a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe51c102570 0x7fe51c198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe52340a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe51c102ef0 0x7fe51c19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe52340a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c1120d0 con 0x7fe51c100e20 2024-09-16T00:40:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe52340a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c111f50 con 0x7fe51c102ef0 2024-09-16T00:40:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe52340a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c112250 con 0x7fe51c102570 2024-09-16T00:40:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe5209a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe51c102570 0x7fe51c198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe5211a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe51c100e20 0x7fe51c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe5209a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe51c102570 0x7fe51c198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42856/0 (socket says 172.21.15.33:42856) 2024-09-16T00:40:34.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe5211a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe51c100e20 0x7fe51c198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45252/0 (socket says 172.21.15.33:45252) 2024-09-16T00:40:34.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe5209a5640 1 -- 172.21.15.33:0/3743605397 learned_addr learned my addr 172.21.15.33:0/3743605397 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:34.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.538+0000 7fe5219a7640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe51c102ef0 0x7fe51c19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.539+0000 7fe5211a6640 1 -- 172.21.15.33:0/3743605397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe51c102570 msgr2=0x7fe51c198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:34.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.539+0000 7fe5211a6640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe51c102570 0x7fe51c198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.539+0000 7fe5211a6640 1 -- 172.21.15.33:0/3743605397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe51c102ef0 msgr2=0x7fe51c19ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:34.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.539+0000 7fe5211a6640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe51c102ef0 0x7fe51c19ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.539+0000 7fe5211a6640 1 -- 172.21.15.33:0/3743605397 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c19d4e0 con 0x7fe51c100e20 2024-09-16T00:40:34.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.539+0000 7fe5209a5640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe51c102570 0x7fe51c198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:40:34.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.539+0000 7fe5211a6640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe51c100e20 0x7fe51c198530 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fe51800e790 tx=0x7fe51800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:34.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.539+0000 7fe5127fc640 1 -- 172.21.15.33:0/3743605397 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51800eea0 con 0x7fe51c100e20 2024-09-16T00:40:34.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.539+0000 7fe52340a640 1 -- 172.21.15.33:0/3743605397 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51c19d7d0 con 0x7fe51c100e20 2024-09-16T00:40:34.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.539+0000 7fe52340a640 1 -- 172.21.15.33:0/3743605397 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe51c1a50b0 con 0x7fe51c100e20 2024-09-16T00:40:34.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.539+0000 7fe5127fc640 1 -- 172.21.15.33:0/3743605397 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe51800ccc0 con 0x7fe51c100e20 2024-09-16T00:40:34.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.539+0000 7fe5127fc640 1 -- 172.21.15.33:0/3743605397 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518018660 con 0x7fe51c100e20 2024-09-16T00:40:34.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.543+0000 7fe5127fc640 1 -- 172.21.15.33:0/3743605397 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe518010430 con 0x7fe51c100e20 2024-09-16T00:40:34.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.543+0000 7fe5257e3740 1 Processor -- start 2024-09-16T00:40:34.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.543+0000 7fe5127fc640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe5080a3240 0x7fe5080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5257e3740 1 -- start start 2024-09-16T00:40:34.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5219a7640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe5080a3240 0x7fe5080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5127fc640 1 -- 172.21.15.33:0/3743605397 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe5180cffb0 con 0x7fe51c100e20 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5257e3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e41780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5257e3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e50550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5257e3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e52e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5257e3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559ea0da72e0 con 0x559ea0e41cc0 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5257e3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559ea0d4ece0 con 0x559ea0e413a0 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5257e3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559ea0da32a0 con 0x559ea0e50a90 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5211a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e41780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5211a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e41780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:44740/0 (socket says 172.21.15.33:44740) 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5211a6640 1 -- 172.21.15.33:0/3317744963 learned_addr learned my addr 172.21.15.33:0/3317744963 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5209a5640 1 --2- 172.21.15.33:0/3317744963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e50550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5219a7640 1 --2- 172.21.15.33:0/3317744963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e52e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5219a7640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe5080a3240 0x7fe5080a5700 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe514005fd0 tx=0x7fe514011810 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5211a6640 1 -- 172.21.15.33:0/3317744963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 msgr2=0x559ea0e52e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5211a6640 1 --2- 172.21.15.33:0/3317744963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e52e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5211a6640 1 -- 172.21.15.33:0/3317744963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 msgr2=0x559ea0e50550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.544+0000 7fe5211a6640 1 --2- 172.21.15.33:0/3317744963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e50550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.545+0000 7fe5211a6640 1 -- 172.21.15.33:0/3317744963 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ea0d33440 con 0x559ea0e413a0 2024-09-16T00:40:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.545+0000 7fe5209a5640 1 --2- 172.21.15.33:0/3317744963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e50550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:40:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.545+0000 7fe5211a6640 1 --2- 172.21.15.33:0/3317744963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e41780 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe5180a4890 tx=0x7fe51809c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b9ec8b73c9e56b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.545+0000 7fe5077fe640 1 -- 172.21.15.33:0/3317744963 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5180a4dc0 con 0x559ea0e413a0 2024-09-16T00:40:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.545+0000 7fe5077fe640 1 -- 172.21.15.33:0/3317744963 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe51809a500 con 0x559ea0e413a0 2024-09-16T00:40:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.545+0000 7fe5077fe640 1 -- 172.21.15.33:0/3317744963 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5180a0300 con 0x559ea0e413a0 2024-09-16T00:40:34.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.546+0000 7fe5257e3740 1 -- 172.21.15.33:0/3317744963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 msgr2=0x559ea0e41780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:34.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.546+0000 7fe5257e3740 1 --2- 172.21.15.33:0/3317744963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e41780 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe5180a4890 tx=0x7fe51809c200 comp rx=0 tx=0).stop 2024-09-16T00:40:34.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.546+0000 7fe5257e3740 1 -- 172.21.15.33:0/3317744963 shutdown_connections 2024-09-16T00:40:34.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.546+0000 7fe5257e3740 1 --2- 172.21.15.33:0/3317744963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e52e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.546+0000 7fe5257e3740 1 --2- 172.21.15.33:0/3317744963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e50550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.546+0000 7fe5257e3740 1 --2- 172.21.15.33:0/3317744963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e41780 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.546+0000 7fe5257e3740 1 -- 172.21.15.33:0/3317744963 >> 172.21.15.33:0/3317744963 conn(0x559ea0e3da30 msgr2=0x559ea0e3de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:34.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.546+0000 7fe5257e3740 1 -- 172.21.15.33:0/3317744963 shutdown_connections 2024-09-16T00:40:34.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.546+0000 7fe5257e3740 1 -- 172.21.15.33:0/3317744963 wait complete. 2024-09-16T00:40:34.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.546+0000 7fe5257e3740 1 -- start start 2024-09-16T00:40:34.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5257e3740 1 Processor -- start 2024-09-16T00:40:34.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5257e3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e413a0 0x559ea0e504d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5257e3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e41cc0 0x559ea0e4d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5257e3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e50a90 0x559ea0e4df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5257e3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559ea0e536c0 con 0x559ea0e413a0 2024-09-16T00:40:34.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5257e3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x559ea0d62050 con 0x559ea0e50a90 2024-09-16T00:40:34.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5257e3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559ea0e53840 con 0x559ea0e41cc0 2024-09-16T00:40:34.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5257e3740 10 client.0 _ll_register_callbacks cb 0x559ea0da2160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:34.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5209a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e41cc0 0x559ea0e4d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5219a7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e50a90 0x559ea0e4df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5209a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e41cc0 0x559ea0e4d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42882/0 (socket says 172.21.15.33:42882) 2024-09-16T00:40:34.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5209a5640 1 -- 172.21.15.33:0/418612561 learned_addr learned my addr 172.21.15.33:0/418612561 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:34.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.547+0000 7fe5211a6640 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e413a0 0x559ea0e504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe5219a7640 1 -- 172.21.15.33:0/418612561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e41cc0 msgr2=0x559ea0e4d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe5219a7640 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e41cc0 0x559ea0e4d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe5219a7640 1 -- 172.21.15.33:0/418612561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e413a0 msgr2=0x559ea0e504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe5219a7640 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e413a0 0x559ea0e504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe5219a7640 1 -- 172.21.15.33:0/418612561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ea0b183c0 con 0x559ea0e50a90 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe5211a6640 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e413a0 0x559ea0e504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe5219a7640 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e50a90 0x559ea0e4df10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe514009d60 tx=0x7fe514004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e09454aed24cfbc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe504ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe504ff9640 1 -- 172.21.15.33:0/418612561 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe514017040 con 0x559ea0e50a90 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe5257e3740 1 -- 172.21.15.33:0/418612561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe51809a500 con 0x559ea0e50a90 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe504ff9640 1 -- 172.21.15.33:0/418612561 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe514009040 con 0x559ea0e50a90 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe504ff9640 1 -- 172.21.15.33:0/418612561 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe51402f050 con 0x559ea0e50a90 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe5257e3740 10 client.28877 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.548+0000 7fe5257e3740 1 -- 172.21.15.33:0/418612561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559ea0e442f0 con 0x559ea0e50a90 2024-09-16T00:40:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.549+0000 7fe504ff9640 1 -- 172.21.15.33:0/418612561 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe51809a500 con 0x559ea0e50a90 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.549+0000 7fe504ff9640 1 -- 172.21.15.33:0/418612561 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fe5140052f0 con 0x559ea0e50a90 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.549+0000 7fe5257e3740 10 client.28877 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.549+0000 7fe5257e3740 10 client.28877 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.549+0000 7fe5257e3740 20 client.28877 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.549+0000 7fe5257e3740 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559ea0e457b0 0x559ea0e47c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.549+0000 7fe5257e3740 4 client.28877 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.549+0000 7fe5257e3740 1 -- 172.21.15.33:0/418612561 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ea0db4850 con 0x559ea0e457b0 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.549+0000 7fe5211a6640 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559ea0e457b0 0x559ea0e47c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.550+0000 7fe504ff9640 10 client.28877 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.550+0000 7fe5211a6640 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559ea0e457b0 0x559ea0e47c00 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bb93036e734c50c server_cookie=5743c281661fde0b in_seq=0 out_seq=0 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe504ff9640 1 -- 172.21.15.33:0/418612561 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559ea0db4850 con 0x559ea0e457b0 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe504ff9640 10 client.28877 handle_command_reply: tid=0 2024-09-16T00:40:34.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe5257e3740 1 client.28877 shutdown 2024-09-16T00:40:34.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe5257e3740 1 -- 172.21.15.33:0/418612561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e50a90 msgr2=0x559ea0e4df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:34.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe5257e3740 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e50a90 0x559ea0e4df10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe514009d60 tx=0x7fe514004ab0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe5257e3740 1 -- 172.21.15.33:0/418612561 shutdown_connections 2024-09-16T00:40:34.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe5257e3740 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559ea0e457b0 0x559ea0e47c00 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe5257e3740 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e50a90 0x559ea0e4df10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe5257e3740 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e41cc0 0x559ea0e4d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe5257e3740 1 --2- 172.21.15.33:0/418612561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e413a0 0x559ea0e504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe504ff9640 0 client.28877 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:34.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe5257e3740 1 -- 172.21.15.33:0/418612561 >> 172.21.15.33:0/418612561 conn(0x559ea0e3da30 msgr2=0x559ea0e43790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:34.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.551+0000 7fe5257e3740 1 -- 172.21.15.33:0/418612561 shutdown_connections 2024-09-16T00:40:34.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.552+0000 7fe5257e3740 1 -- 172.21.15.33:0/418612561 wait complete. 2024-09-16T00:40:34.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.552+0000 7fe5257e3740 20 client.28877 trim_cache size 0 max 16384 2024-09-16T00:40:34.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5257e3740 1 Processor -- start 2024-09-16T00:40:34.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5257e3740 1 -- start start 2024-09-16T00:40:34.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5257e3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e4d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5257e3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e4f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5257e3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e6c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5257e3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c1af830 con 0x559ea0e41cc0 2024-09-16T00:40:34.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5257e3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c1ad100 con 0x559ea0e413a0 2024-09-16T00:40:34.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5257e3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c1aaf50 con 0x559ea0e50a90 2024-09-16T00:40:34.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5209a5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e4f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5211a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e4d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5219a7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e6c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5209a5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e4f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45278/0 (socket says 172.21.15.33:45278) 2024-09-16T00:40:34.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5219a7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e6c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42894/0 (socket says 172.21.15.33:42894) 2024-09-16T00:40:34.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5209a5640 1 -- 172.21.15.33:0/491441113 learned_addr learned my addr 172.21.15.33:0/491441113 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:34.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.571+0000 7fe5211a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e4d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:44756/0 (socket says 172.21.15.33:44756) 2024-09-16T00:40:34.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.572+0000 7fe5211a6640 1 -- 172.21.15.33:0/491441113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 msgr2=0x559ea0e6c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:34.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.572+0000 7fe5211a6640 1 --2- 172.21.15.33:0/491441113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e6c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.572+0000 7fe5211a6640 1 -- 172.21.15.33:0/491441113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 msgr2=0x559ea0e4f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.572+0000 7fe5211a6640 1 --2- 172.21.15.33:0/491441113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e4f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.572+0000 7fe5211a6640 1 -- 172.21.15.33:0/491441113 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ea0e3e990 con 0x559ea0e413a0 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.572+0000 7fe5219a7640 1 --2- 172.21.15.33:0/491441113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e6c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.572+0000 7fe5209a5640 1 --2- 172.21.15.33:0/491441113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e4f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.572+0000 7fe5211a6640 1 --2- 172.21.15.33:0/491441113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e4d040 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe51809bf00 tx=0x7fe51809a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26a035b525cbd03d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.572+0000 7fe506ffd640 1 -- 172.21.15.33:0/491441113 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5180bc070 con 0x559ea0e413a0 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.572+0000 7fe506ffd640 1 -- 172.21.15.33:0/491441113 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5180bc210 con 0x559ea0e413a0 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.572+0000 7fe506ffd640 1 -- 172.21.15.33:0/491441113 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5180981b0 con 0x559ea0e413a0 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.573+0000 7fe5257e3740 1 -- 172.21.15.33:0/491441113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 msgr2=0x559ea0e4d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.573+0000 7fe5257e3740 1 --2- 172.21.15.33:0/491441113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e4d040 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe51809bf00 tx=0x7fe51809a6d0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.573+0000 7fe5257e3740 1 -- 172.21.15.33:0/491441113 shutdown_connections 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.573+0000 7fe5257e3740 1 --2- 172.21.15.33:0/491441113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e6c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.573+0000 7fe5257e3740 1 --2- 172.21.15.33:0/491441113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e4f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.573+0000 7fe5257e3740 1 --2- 172.21.15.33:0/491441113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e4d040 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.573+0000 7fe5257e3740 1 -- 172.21.15.33:0/491441113 >> 172.21.15.33:0/491441113 conn(0x559ea0e3da30 msgr2=0x559ea0e3de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.573+0000 7fe5257e3740 1 -- 172.21.15.33:0/491441113 shutdown_connections 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.573+0000 7fe5257e3740 1 -- 172.21.15.33:0/491441113 wait complete. 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.573+0000 7fe5257e3740 1 -- start start 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.573+0000 7fe5257e3740 1 Processor -- start 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5257e3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e548f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5257e3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e54e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5257e3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e55370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5257e3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c1ab5c0 con 0x559ea0e41cc0 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5257e3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c1aca30 con 0x559ea0e413a0 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5257e3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c1abc90 con 0x559ea0e50a90 2024-09-16T00:40:34.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5257e3740 10 client.0 _ll_register_callbacks cb 0x7fe51401d5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:34.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5219a7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e55370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5209a5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e54e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5219a7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e55370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42898/0 (socket says 172.21.15.33:42898) 2024-09-16T00:40:34.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5209a5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e54e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45290/0 (socket says 172.21.15.33:45290) 2024-09-16T00:40:34.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5219a7640 1 -- 172.21.15.33:0/3806727033 learned_addr learned my addr 172.21.15.33:0/3806727033 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:34.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5211a6640 1 --2- 172.21.15.33:0/3806727033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5209a5640 1 -- 172.21.15.33:0/3806727033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 msgr2=0x559ea0e55370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:34.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5209a5640 1 --2- 172.21.15.33:0/3806727033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e55370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5209a5640 1 -- 172.21.15.33:0/3806727033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 msgr2=0x559ea0e548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:34.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5209a5640 1 --2- 172.21.15.33:0/3806727033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.574+0000 7fe5209a5640 1 -- 172.21.15.33:0/3806727033 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ea0ece8e0 con 0x559ea0e41cc0 2024-09-16T00:40:34.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.575+0000 7fe5209a5640 1 --2- 172.21.15.33:0/3806727033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e54e30 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fe50c009250 tx=0x7fe50c0305b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b823a4c379288e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:34.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.575+0000 7fe504ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:40:34.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.575+0000 7fe504ff9640 1 -- 172.21.15.33:0/3806727033 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c0308b0 con 0x559ea0e41cc0 2024-09-16T00:40:34.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.575+0000 7fe504ff9640 1 -- 172.21.15.33:0/3806727033 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe50c005050 con 0x559ea0e41cc0 2024-09-16T00:40:34.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.575+0000 7fe5257e3740 1 -- 172.21.15.33:0/3806727033 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5180bc210 con 0x559ea0e41cc0 2024-09-16T00:40:34.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.575+0000 7fe504ff9640 1 -- 172.21.15.33:0/3806727033 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe50c050050 con 0x559ea0e41cc0 2024-09-16T00:40:34.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.575+0000 7fe5257e3740 10 client.19740 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:34.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.575+0000 7fe5257e3740 1 -- 172.21.15.33:0/3806727033 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559ea0ecfc00 con 0x559ea0e41cc0 2024-09-16T00:40:34.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.575+0000 7fe504ff9640 1 -- 172.21.15.33:0/3806727033 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe50c0046c0 con 0x559ea0e41cc0 2024-09-16T00:40:34.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.576+0000 7fe504ff9640 1 -- 172.21.15.33:0/3806727033 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fe50c0035f0 con 0x559ea0e41cc0 2024-09-16T00:40:34.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.576+0000 7fe5257e3740 10 client.19740 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:34.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.576+0000 7fe5257e3740 10 client.19740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:40:34.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.576+0000 7fe5257e3740 20 client.19740 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:34.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.576+0000 7fe5257e3740 1 --2- 172.21.15.33:0/3806727033 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559ea0ed03b0 0x559ea0ed27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:34.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.576+0000 7fe5257e3740 4 client.19740 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:40:34.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.576+0000 7fe5257e3740 1 -- 172.21.15.33:0/3806727033 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fe5180bc210 con 0x559ea0ed03b0 2024-09-16T00:40:34.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.576+0000 7fe5211a6640 1 --2- 172.21.15.33:0/3806727033 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559ea0ed03b0 0x559ea0ed27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:34.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.576+0000 7fe5211a6640 1 --2- 172.21.15.33:0/3806727033 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559ea0ed03b0 0x559ea0ed27d0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3edf724d0415350c server_cookie=9d10c60cfc345369 in_seq=0 out_seq=0 2024-09-16T00:40:34.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.576+0000 7fe504ff9640 10 client.19740 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:34.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe504ff9640 1 -- 172.21.15.33:0/3806727033 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fe5180bc210 con 0x559ea0ed03b0 2024-09-16T00:40:34.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe504ff9640 10 client.19740 handle_command_reply: tid=0 2024-09-16T00:40:34.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe5257e3740 1 client.19740 shutdown 2024-09-16T00:40:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe5257e3740 1 -- 172.21.15.33:0/3806727033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 msgr2=0x559ea0e54e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe5257e3740 1 --2- 172.21.15.33:0/3806727033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e54e30 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fe50c009250 tx=0x7fe50c0305b0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe5257e3740 1 -- 172.21.15.33:0/3806727033 shutdown_connections 2024-09-16T00:40:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe5257e3740 1 --2- 172.21.15.33:0/3806727033 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x559ea0ed03b0 0x559ea0ed27d0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe504ff9640 0 client.19740 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe5257e3740 1 --2- 172.21.15.33:0/3806727033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559ea0e50a90 0x559ea0e55370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe5257e3740 1 --2- 172.21.15.33:0/3806727033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ea0e41cc0 0x559ea0e54e30 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe5257e3740 1 --2- 172.21.15.33:0/3806727033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559ea0e413a0 0x559ea0e548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe5257e3740 1 -- 172.21.15.33:0/3806727033 >> 172.21.15.33:0/3806727033 conn(0x559ea0e3da30 msgr2=0x559ea0e4efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe5257e3740 1 -- 172.21.15.33:0/3806727033 shutdown_connections 2024-09-16T00:40:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe5257e3740 1 -- 172.21.15.33:0/3806727033 wait complete. 2024-09-16T00:40:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.577+0000 7fe5257e3740 20 client.19740 trim_cache size 0 max 16384 2024-09-16T00:40:34.584 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:40:34.584 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:40:34.584 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:40:34.584 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:40:34.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.578+0000 7fe52340a640 1 -- 172.21.15.33:0/3743605397 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe5080a3240 msgr2=0x7fe5080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:34.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.578+0000 7fe52340a640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe5080a3240 0x7fe5080a5700 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe514005fd0 tx=0x7fe514011810 comp rx=0 tx=0).stop 2024-09-16T00:40:34.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.579+0000 7fe52340a640 1 -- 172.21.15.33:0/3743605397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe51c100e20 msgr2=0x7fe51c198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:34.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.579+0000 7fe52340a640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe51c100e20 0x7fe51c198530 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fe51800e790 tx=0x7fe51800ec50 comp rx=0 tx=0).stop 2024-09-16T00:40:34.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.579+0000 7fe52340a640 1 -- 172.21.15.33:0/3743605397 shutdown_connections 2024-09-16T00:40:34.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.579+0000 7fe52340a640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe5080a3240 0x7fe5080a5700 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.579+0000 7fe52340a640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe51c102ef0 0x7fe51c19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.579+0000 7fe52340a640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe51c102570 0x7fe51c198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.579+0000 7fe52340a640 1 --2- 172.21.15.33:0/3743605397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe51c100e20 0x7fe51c198530 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:34.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.579+0000 7fe52340a640 1 -- 172.21.15.33:0/3743605397 >> 172.21.15.33:0/3743605397 conn(0x7fe51c0fc7d0 msgr2=0x7fe51c1048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:34.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.579+0000 7fe52340a640 1 -- 172.21.15.33:0/3743605397 shutdown_connections 2024-09-16T00:40:34.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:34.579+0000 7fe52340a640 1 -- 172.21.15.33:0/3743605397 wait complete. 2024-09-16T00:40:35.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 199345 18.18 MB/sec execute 461 sec latency 84.175 ms 2024-09-16T00:40:35.481 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:40:35.482 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:40:35.825 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:40:35.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:35 smithi033 ceph-mon[31243]: pgmap v1053: 129 pgs: 129 active+clean; 713 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 393 op/s 2024-09-16T00:40:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:35 smithi083 ceph-mon[35242]: pgmap v1053: 129 pgs: 129 active+clean; 713 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 393 op/s 2024-09-16T00:40:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:35 smithi060 ceph-mon[35185]: pgmap v1053: 129 pgs: 129 active+clean; 713 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 393 op/s 2024-09-16T00:40:36.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 200039 18.21 MB/sec execute 462 sec latency 171.169 ms 2024-09-16T00:40:36.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.377+0000 7f61344bb640 1 -- 172.21.15.33:0/3247747462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c102d30 msgr2=0x7f612c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:36.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.378+0000 7f61344bb640 1 --2- 172.21.15.33:0/3247747462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c102d30 0x7f612c103110 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f6128009930 tx=0x7f612802a420 comp rx=0 tx=0).stop 2024-09-16T00:40:36.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.378+0000 7f61344bb640 1 -- 172.21.15.33:0/3247747462 shutdown_connections 2024-09-16T00:40:36.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.378+0000 7f61344bb640 1 --2- 172.21.15.33:0/3247747462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612c1082a0 0x7f612c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:36.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.378+0000 7f61344bb640 1 --2- 172.21.15.33:0/3247747462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f612c1036e0 0x7f612c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:36.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.378+0000 7f61344bb640 1 --2- 172.21.15.33:0/3247747462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c102d30 0x7f612c103110 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:36.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.378+0000 7f61344bb640 1 -- 172.21.15.33:0/3247747462 >> 172.21.15.33:0/3247747462 conn(0x7f612c0fc7d0 msgr2=0x7f612c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:36.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.378+0000 7f61344bb640 1 -- 172.21.15.33:0/3247747462 shutdown_connections 2024-09-16T00:40:36.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.378+0000 7f61344bb640 1 -- 172.21.15.33:0/3247747462 wait complete. 2024-09-16T00:40:36.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f61344bb640 1 Processor -- start 2024-09-16T00:40:36.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f61344bb640 1 -- start start 2024-09-16T00:40:36.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f61344bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f612c102d30 0x7f612c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:36.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f61344bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612c1036e0 0x7f612c198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:36.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f61344bb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1082a0 0x7f612c19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:36.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f61344bb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1120a0 con 0x7f612c1082a0 2024-09-16T00:40:36.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f61344bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f612c111f20 con 0x7f612c1036e0 2024-09-16T00:40:36.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f61344bb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f612c112220 con 0x7f612c102d30 2024-09-16T00:40:36.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f6132257640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f612c102d30 0x7f612c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:36.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f6132257640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f612c102d30 0x7f612c1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:42926/0 (socket says 172.21.15.33:42926) 2024-09-16T00:40:36.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f6132257640 1 -- 172.21.15.33:0/2455377351 learned_addr learned my addr 172.21.15.33:0/2455377351 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:36.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f6132a58640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1082a0 0x7f612c19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:36.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f6131a56640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612c1036e0 0x7f612c198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:36.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f6132257640 1 -- 172.21.15.33:0/2455377351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612c1036e0 msgr2=0x7f612c198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:36.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f6132257640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612c1036e0 0x7f612c198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:36.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f6132257640 1 -- 172.21.15.33:0/2455377351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1082a0 msgr2=0x7f612c19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:36.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.379+0000 7f6132257640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1082a0 0x7f612c19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:36.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.380+0000 7f6132257640 1 -- 172.21.15.33:0/2455377351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612c19d590 con 0x7f612c102d30 2024-09-16T00:40:36.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.380+0000 7f6132257640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f612c102d30 0x7f612c1985e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6128009230 tx=0x7f6128004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:36.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.380+0000 7f61237fe640 1 -- 172.21.15.33:0/2455377351 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612803f030 con 0x7f612c102d30 2024-09-16T00:40:36.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.380+0000 7f61237fe640 1 -- 172.21.15.33:0/2455377351 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6128004440 con 0x7f612c102d30 2024-09-16T00:40:36.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.380+0000 7f61344bb640 1 -- 172.21.15.33:0/2455377351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612c19d820 con 0x7f612c102d30 2024-09-16T00:40:36.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.380+0000 7f61237fe640 1 -- 172.21.15.33:0/2455377351 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128035070 con 0x7f612c102d30 2024-09-16T00:40:36.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.380+0000 7f61344bb640 1 -- 172.21.15.33:0/2455377351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f612c1a5160 con 0x7f612c102d30 2024-09-16T00:40:36.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.381+0000 7f61344bb640 1 -- 172.21.15.33:0/2455377351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f612c103110 con 0x7f612c102d30 2024-09-16T00:40:36.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.383+0000 7f61237fe640 1 -- 172.21.15.33:0/2455377351 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6128042030 con 0x7f612c102d30 2024-09-16T00:40:36.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.383+0000 7f61237fe640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f61180a32c0 0x7f61180a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:36.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.384+0000 7f61237fe640 1 -- 172.21.15.33:0/2455377351 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f61280f00c0 con 0x7f612c102d30 2024-09-16T00:40:36.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.384+0000 7f6131a56640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f61180a32c0 0x7f61180a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:36.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.384+0000 7f6131a56640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f61180a32c0 0x7f61180a5780 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f612c0fee70 tx=0x7f611c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:36.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.386+0000 7f61237fe640 1 -- 172.21.15.33:0/2455377351 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f61280bc960 con 0x7f612c102d30 2024-09-16T00:40:36.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.701+0000 7f61344bb640 1 -- 172.21.15.33:0/2455377351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f612c10a2b0 con 0x7f612c102d30 2024-09-16T00:40:36.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.702+0000 7f61237fe640 1 -- 172.21.15.33:0/2455377351 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f61280bc260 con 0x7f612c102d30 2024-09-16T00:40:36.703 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:40:36.704 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:40:36.704 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:40:36.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.705+0000 7f61344bb640 1 -- 172.21.15.33:0/2455377351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f61180a32c0 msgr2=0x7f61180a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:36.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.705+0000 7f61344bb640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f61180a32c0 0x7f61180a5780 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f612c0fee70 tx=0x7f611c005f90 comp rx=0 tx=0).stop 2024-09-16T00:40:36.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.705+0000 7f61344bb640 1 -- 172.21.15.33:0/2455377351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f612c102d30 msgr2=0x7f612c1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:36.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.705+0000 7f61344bb640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f612c102d30 0x7f612c1985e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6128009230 tx=0x7f6128004040 comp rx=0 tx=0).stop 2024-09-16T00:40:36.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.705+0000 7f61344bb640 1 -- 172.21.15.33:0/2455377351 shutdown_connections 2024-09-16T00:40:36.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.705+0000 7f61344bb640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f61180a32c0 0x7f61180a5780 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:36.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.705+0000 7f61344bb640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1082a0 0x7f612c19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:36.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.705+0000 7f61344bb640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612c1036e0 0x7f612c198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:36.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.705+0000 7f61344bb640 1 --2- 172.21.15.33:0/2455377351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f612c102d30 0x7f612c1985e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:36.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.705+0000 7f61344bb640 1 -- 172.21.15.33:0/2455377351 >> 172.21.15.33:0/2455377351 conn(0x7f612c0fc7d0 msgr2=0x7f612c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:36.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.705+0000 7f61344bb640 1 -- 172.21.15.33:0/2455377351 shutdown_connections 2024-09-16T00:40:36.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:36.705+0000 7f61344bb640 1 -- 172.21.15.33:0/2455377351 wait complete. 2024-09-16T00:40:36.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:36 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:40:36.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:36 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:40:37.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:36 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:40:37.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 200510 18.24 MB/sec execute 463 sec latency 119.343 ms 2024-09-16T00:40:37.825 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:37 smithi033 ceph-mon[31243]: pgmap v1054: 129 pgs: 129 active+clean; 716 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.1 MiB/s wr, 391 op/s 2024-09-16T00:40:37.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:37 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2455377351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:40:37.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:40:37.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:40:37.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:40:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:37 smithi083 ceph-mon[35242]: pgmap v1054: 129 pgs: 129 active+clean; 716 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.1 MiB/s wr, 391 op/s 2024-09-16T00:40:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:37 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2455377351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:40:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:40:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:40:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:40:37.946 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:40:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:37 smithi060 ceph-mon[35185]: pgmap v1054: 129 pgs: 129 active+clean; 716 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.1 MiB/s wr, 391 op/s 2024-09-16T00:40:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:37 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2455377351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:40:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:40:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:40:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:40:38.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 201016 18.26 MB/sec execute 464 sec latency 70.809 ms 2024-09-16T00:40:38.254 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:40:39.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.012+0000 7f7c53f9b640 1 -- 172.21.15.33:0/297167071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4c1082a0 msgr2=0x7f7c4c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:39.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.012+0000 7f7c53f9b640 1 --2- 172.21.15.33:0/297167071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4c1082a0 0x7f7c4c10ec40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009c10 tx=0x7f7c4402a6e0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.012+0000 7f7c53f9b640 1 -- 172.21.15.33:0/297167071 shutdown_connections 2024-09-16T00:40:39.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.012+0000 7f7c53f9b640 1 --2- 172.21.15.33:0/297167071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4c1082a0 0x7f7c4c10ec40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.012+0000 7f7c53f9b640 1 --2- 172.21.15.33:0/297167071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c4c1036e0 0x7f7c4c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.012+0000 7f7c53f9b640 1 --2- 172.21.15.33:0/297167071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c102d30 0x7f7c4c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.012+0000 7f7c53f9b640 1 -- 172.21.15.33:0/297167071 >> 172.21.15.33:0/297167071 conn(0x7f7c4c0fc7d0 msgr2=0x7f7c4c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:39.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.012+0000 7f7c53f9b640 1 -- 172.21.15.33:0/297167071 shutdown_connections 2024-09-16T00:40:39.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.012+0000 7f7c53f9b640 1 -- 172.21.15.33:0/297167071 wait complete. 2024-09-16T00:40:39.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.013+0000 7f7c53f9b640 1 Processor -- start 2024-09-16T00:40:39.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.013+0000 7f7c53f9b640 1 -- start start 2024-09-16T00:40:39.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.013+0000 7f7c53f9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c4c102d30 0x7f7c4c073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.013+0000 7f7c53f9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1036e0 0x7f7c4c06dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.013+0000 7f7c53f9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4c1082a0 0x7f7c4c06e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c53f9b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c06c620 con 0x7f7c4c1036e0 2024-09-16T00:40:39.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c53f9b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c06c4a0 con 0x7f7c4c1082a0 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c53f9b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c06c7a0 con 0x7f7c4c102d30 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c51536640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1036e0 0x7f7c4c06dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c51536640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1036e0 0x7f7c4c06dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45320/0 (socket says 172.21.15.33:45320) 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c51536640 1 -- 172.21.15.33:0/3448515427 learned_addr learned my addr 172.21.15.33:0/3448515427 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c51d37640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c4c102d30 0x7f7c4c073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c52538640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4c1082a0 0x7f7c4c06e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c51d37640 1 -- 172.21.15.33:0/3448515427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4c1082a0 msgr2=0x7f7c4c06e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c51d37640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4c1082a0 0x7f7c4c06e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c51d37640 1 -- 172.21.15.33:0/3448515427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1036e0 msgr2=0x7f7c4c06dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c51d37640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1036e0 0x7f7c4c06dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c51d37640 1 -- 172.21.15.33:0/3448515427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c06e960 con 0x7f7c4c102d30 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.014+0000 7f7c51536640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1036e0 0x7f7c4c06dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:40:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.015+0000 7f7c51d37640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c4c102d30 0x7f7c4c073bd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7c48005fd0 tx=0x7f7c4800edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:39.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.015+0000 7f7c42ffd640 1 -- 172.21.15.33:0/3448515427 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4800ccc0 con 0x7f7c4c102d30 2024-09-16T00:40:39.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.015+0000 7f7c42ffd640 1 -- 172.21.15.33:0/3448515427 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c480035f0 con 0x7f7c4c102d30 2024-09-16T00:40:39.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.015+0000 7f7c53f9b640 1 -- 172.21.15.33:0/3448515427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4c1ad3e0 con 0x7f7c4c102d30 2024-09-16T00:40:39.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.015+0000 7f7c42ffd640 1 -- 172.21.15.33:0/3448515427 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4800ce60 con 0x7f7c4c102d30 2024-09-16T00:40:39.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.015+0000 7f7c53f9b640 1 -- 172.21.15.33:0/3448515427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4c1ad8f0 con 0x7f7c4c102d30 2024-09-16T00:40:39.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.019+0000 7f7c42ffd640 1 -- 172.21.15.33:0/3448515427 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7c480037a0 con 0x7f7c4c102d30 2024-09-16T00:40:39.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.019+0000 7f7c42ffd640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7c380a3300 0x7f7c380a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.020+0000 7f7c42ffd640 1 -- 172.21.15.33:0/3448515427 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7c480cff10 con 0x7f7c4c102d30 2024-09-16T00:40:39.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.020+0000 7f7c51536640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7c380a3300 0x7f7c380a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.020+0000 7f7c56374740 1 Processor -- start 2024-09-16T00:40:39.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.020+0000 7f7c56374740 1 -- start start 2024-09-16T00:40:39.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.020+0000 7f7c56374740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b31b40 0x5644e4b30090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.020+0000 7f7c56374740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b305d0 0x5644e4b30a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.020+0000 7f7c56374740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b41b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c56374740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5644e4a962f0 con 0x5644e4b3f780 2024-09-16T00:40:39.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c56374740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5644e4a3dcf0 con 0x5644e4b31b40 2024-09-16T00:40:39.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c56374740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5644e4a922b0 con 0x5644e4b305d0 2024-09-16T00:40:39.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c51d37640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b31b40 0x5644e4b30090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c52538640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b41b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c52538640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b41b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45332/0 (socket says 172.21.15.33:45332) 2024-09-16T00:40:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c52538640 1 -- 172.21.15.33:0/3057746745 learned_addr learned my addr 172.21.15.33:0/3057746745 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c51536640 1 --2- 172.21.15.33:0/3057746745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b305d0 0x5644e4b30a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c52538640 1 -- 172.21.15.33:0/3057746745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b305d0 msgr2=0x5644e4b30a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c51536640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7c380a3300 0x7f7c380a57c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c0095f0 tx=0x7f7c3c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c52538640 1 --2- 172.21.15.33:0/3057746745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b305d0 0x5644e4b30a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c52538640 1 -- 172.21.15.33:0/3057746745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b31b40 msgr2=0x5644e4b30090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:39.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c52538640 1 --2- 172.21.15.33:0/3057746745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b31b40 0x5644e4b30090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c52538640 1 -- 172.21.15.33:0/3057746745 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644e4a223a0 con 0x5644e4b3f780 2024-09-16T00:40:39.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.021+0000 7f7c51d37640 1 --2- 172.21.15.33:0/3057746745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b31b40 0x5644e4b30090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:40:39.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.022+0000 7f7c52538640 1 --2- 172.21.15.33:0/3057746745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b41b70 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009be0 tx=0x7f7c44032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6831465b557946c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:39.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.022+0000 7f7c37fff640 1 -- 172.21.15.33:0/3057746745 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44041030 con 0x5644e4b3f780 2024-09-16T00:40:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.022+0000 7f7c37fff640 1 -- 172.21.15.33:0/3057746745 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c440303a0 con 0x5644e4b3f780 2024-09-16T00:40:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.022+0000 7f7c37fff640 1 -- 172.21.15.33:0/3057746745 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44036040 con 0x5644e4b3f780 2024-09-16T00:40:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.023+0000 7f7c56374740 1 -- 172.21.15.33:0/3057746745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 msgr2=0x5644e4b41b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.023+0000 7f7c56374740 1 --2- 172.21.15.33:0/3057746745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b41b70 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009be0 tx=0x7f7c44032e20 comp rx=0 tx=0).stop 2024-09-16T00:40:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.023+0000 7f7c56374740 1 -- 172.21.15.33:0/3057746745 shutdown_connections 2024-09-16T00:40:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.023+0000 7f7c56374740 1 --2- 172.21.15.33:0/3057746745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b41b70 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.023+0000 7f7c56374740 1 --2- 172.21.15.33:0/3057746745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b305d0 0x5644e4b30a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.023+0000 7f7c56374740 1 --2- 172.21.15.33:0/3057746745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b31b40 0x5644e4b30090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.023+0000 7f7c56374740 1 -- 172.21.15.33:0/3057746745 >> 172.21.15.33:0/3057746745 conn(0x5644e4af9310 msgr2=0x5644e4b2cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.023+0000 7f7c56374740 1 -- 172.21.15.33:0/3057746745 shutdown_connections 2024-09-16T00:40:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.023+0000 7f7c56374740 1 -- 172.21.15.33:0/3057746745 wait complete. 2024-09-16T00:40:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.024+0000 7f7c56374740 1 -- start start 2024-09-16T00:40:39.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.024+0000 7f7c56374740 1 Processor -- start 2024-09-16T00:40:39.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c56374740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b305d0 0x5644e4b385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c56374740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b38b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c56374740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b39070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c51d37640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b305d0 0x5644e4b385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c51d37640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b305d0 0x5644e4b385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:44808/0 (socket says 172.21.15.33:44808) 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c51d37640 1 -- 172.21.15.33:0/4205815215 learned_addr learned my addr 172.21.15.33:0/4205815215 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c51536640 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b38b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c56374740 1 -- 172.21.15.33:0/4205815215 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5644e4b334f0 con 0x5644e4b3f780 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c56374740 1 -- 172.21.15.33:0/4205815215 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5644e4a51060 con 0x5644e4b305d0 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c56374740 1 -- 172.21.15.33:0/4205815215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5644e4b33670 con 0x5644e4b31b40 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c52538640 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b39070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c56374740 10 client.0 _ll_register_callbacks cb 0x5644e47a48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.025+0000 7f7c51536640 1 -- 172.21.15.33:0/4205815215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b305d0 msgr2=0x5644e4b385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.026+0000 7f7c51536640 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b305d0 0x5644e4b385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.026+0000 7f7c51536640 1 -- 172.21.15.33:0/4205815215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 msgr2=0x5644e4b39070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.026+0000 7f7c51536640 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b39070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.026+0000 7f7c51536640 1 -- 172.21.15.33:0/4205815215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644e48073c0 con 0x5644e4b31b40 2024-09-16T00:40:39.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.026+0000 7f7c51536640 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b38b30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c01b000 tx=0x7f7c3c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb6e2531eae31a15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.026+0000 7f7c357fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.026+0000 7f7c357fa640 1 -- 172.21.15.33:0/4205815215 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c01cda0 con 0x5644e4b31b40 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.026+0000 7f7c56374740 1 -- 172.21.15.33:0/4205815215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c440303a0 con 0x5644e4b31b40 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.026+0000 7f7c357fa640 1 -- 172.21.15.33:0/4205815215 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7c3c022040 con 0x5644e4b31b40 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.026+0000 7f7c357fa640 1 -- 172.21.15.33:0/4205815215 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7c3c032050 con 0x5644e4b31b40 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.027+0000 7f7c56374740 10 client.28915 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.027+0000 7f7c56374740 1 -- 172.21.15.33:0/4205815215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5644e4b49b00 con 0x5644e4b31b40 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.027+0000 7f7c357fa640 1 -- 172.21.15.33:0/4205815215 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c440303a0 con 0x5644e4b31b40 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.027+0000 7f7c357fa640 1 -- 172.21.15.33:0/4205815215 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7c3c01a050 con 0x5644e4b31b40 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.027+0000 7f7c56374740 10 client.28915 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.027+0000 7f7c56374740 10 client.28915 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.027+0000 7f7c56374740 20 client.28915 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:39.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.027+0000 7f7c56374740 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5644e4b4b300 0x5644e4b4d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.027+0000 7f7c56374740 4 client.28915 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:40:39.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.027+0000 7f7c56374740 1 -- 172.21.15.33:0/4205815215 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644e4aa3860 con 0x5644e4b4b300 2024-09-16T00:40:39.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.027+0000 7f7c51d37640 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5644e4b4b300 0x5644e4b4d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.028+0000 7f7c357fa640 10 client.28915 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:40:39.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.028+0000 7f7c51d37640 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5644e4b4b300 0x5644e4b4d7a0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=48a1403e827ddeaa server_cookie=c19364976117afb2 in_seq=0 out_seq=0 2024-09-16T00:40:39.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c357fa640 1 -- 172.21.15.33:0/4205815215 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5644e4aa3860 con 0x5644e4b4b300 2024-09-16T00:40:39.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c357fa640 10 client.28915 handle_command_reply: tid=0 2024-09-16T00:40:39.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c56374740 1 client.28915 shutdown 2024-09-16T00:40:39.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c56374740 1 -- 172.21.15.33:0/4205815215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 msgr2=0x5644e4b38b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:39.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c56374740 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b38b30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c01b000 tx=0x7f7c3c004730 comp rx=0 tx=0).stop 2024-09-16T00:40:39.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c56374740 1 -- 172.21.15.33:0/4205815215 shutdown_connections 2024-09-16T00:40:39.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c56374740 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5644e4b4b300 0x5644e4b4d7a0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c56374740 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b39070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c56374740 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b38b30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c56374740 1 --2- 172.21.15.33:0/4205815215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b305d0 0x5644e4b385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c56374740 1 -- 172.21.15.33:0/4205815215 >> 172.21.15.33:0/4205815215 conn(0x5644e4af9310 msgr2=0x5644e4b32960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:39.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c357fa640 0 client.28915 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:40:39.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c56374740 1 -- 172.21.15.33:0/4205815215 shutdown_connections 2024-09-16T00:40:39.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c56374740 1 -- 172.21.15.33:0/4205815215 wait complete. 2024-09-16T00:40:39.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.030+0000 7f7c56374740 20 client.28915 trim_cache size 0 max 16384 2024-09-16T00:40:39.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c56374740 1 Processor -- start 2024-09-16T00:40:39.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c56374740 1 -- start start 2024-09-16T00:40:39.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c56374740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b305d0 0x5644e4b38ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c56374740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b3c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c56374740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b530b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c56374740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1b8050 con 0x5644e4b3f780 2024-09-16T00:40:39.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c56374740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1b5920 con 0x5644e4b305d0 2024-09-16T00:40:39.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c56374740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1b3770 con 0x5644e4b31b40 2024-09-16T00:40:39.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c51d37640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b305d0 0x5644e4b38ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c52538640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c52538640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b530b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45350/0 (socket says 172.21.15.33:45350) 2024-09-16T00:40:39.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c52538640 1 -- 172.21.15.33:0/2863907158 learned_addr learned my addr 172.21.15.33:0/2863907158 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:39.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.046+0000 7f7c51536640 1 --2- 172.21.15.33:0/2863907158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b3c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.047+0000 7f7c52538640 1 -- 172.21.15.33:0/2863907158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 msgr2=0x5644e4b3c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:39.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.047+0000 7f7c52538640 1 --2- 172.21.15.33:0/2863907158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b3c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.047+0000 7f7c52538640 1 -- 172.21.15.33:0/2863907158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b305d0 msgr2=0x5644e4b38ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:39.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.047+0000 7f7c52538640 1 --2- 172.21.15.33:0/2863907158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b305d0 0x5644e4b38ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.047+0000 7f7c52538640 1 -- 172.21.15.33:0/2863907158 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644e4b2d960 con 0x5644e4b3f780 2024-09-16T00:40:39.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.047+0000 7f7c52538640 1 --2- 172.21.15.33:0/2863907158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b530b0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f7c440039e0 tx=0x7f7c44003ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953500aac0ac1706 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:39.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.047+0000 7f7c377fe640 1 -- 172.21.15.33:0/2863907158 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4404d070 con 0x5644e4b3f780 2024-09-16T00:40:39.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.047+0000 7f7c377fe640 1 -- 172.21.15.33:0/2863907158 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c44030c30 con 0x5644e4b3f780 2024-09-16T00:40:39.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.047+0000 7f7c377fe640 1 -- 172.21.15.33:0/2863907158 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44036040 con 0x5644e4b3f780 2024-09-16T00:40:39.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.048+0000 7f7c56374740 1 -- 172.21.15.33:0/2863907158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 msgr2=0x5644e4b530b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:39.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.048+0000 7f7c56374740 1 --2- 172.21.15.33:0/2863907158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b530b0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f7c440039e0 tx=0x7f7c44003ab0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.048+0000 7f7c56374740 1 -- 172.21.15.33:0/2863907158 shutdown_connections 2024-09-16T00:40:39.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.048+0000 7f7c56374740 1 --2- 172.21.15.33:0/2863907158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b3f780 0x5644e4b530b0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.048+0000 7f7c56374740 1 --2- 172.21.15.33:0/2863907158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b3c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.048+0000 7f7c56374740 1 --2- 172.21.15.33:0/2863907158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b305d0 0x5644e4b38ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.048+0000 7f7c56374740 1 -- 172.21.15.33:0/2863907158 >> 172.21.15.33:0/2863907158 conn(0x5644e4af9310 msgr2=0x5644e4b3be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:39.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.048+0000 7f7c56374740 1 -- 172.21.15.33:0/2863907158 shutdown_connections 2024-09-16T00:40:39.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.048+0000 7f7c56374740 1 -- 172.21.15.33:0/2863907158 wait complete. 2024-09-16T00:40:39.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.048+0000 7f7c56374740 1 -- start start 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.048+0000 7f7c56374740 1 Processor -- start 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c56374740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b305d0 0x5644e4b35690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c56374740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b35bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c56374740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b3f780 0x5644e4b36110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c56374740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1b3de0 con 0x5644e4b305d0 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c56374740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1b5250 con 0x5644e4b3f780 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c56374740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1b44b0 con 0x5644e4b31b40 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c56374740 10 client.0 _ll_register_callbacks cb 0x7f7c3c00fa90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c51d37640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b305d0 0x5644e4b35690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c51536640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b35bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c51d37640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b305d0 0x5644e4b35690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45358/0 (socket says 172.21.15.33:45358) 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c51d37640 1 -- 172.21.15.33:0/3843970101 learned_addr learned my addr 172.21.15.33:0/3843970101 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c52538640 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b3f780 0x5644e4b36110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c51d37640 1 -- 172.21.15.33:0/3843970101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 msgr2=0x5644e4b35bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c51d37640 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b35bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c51d37640 1 -- 172.21.15.33:0/3843970101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b3f780 msgr2=0x5644e4b36110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c51d37640 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b3f780 0x5644e4b36110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c51d37640 1 -- 172.21.15.33:0/3843970101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644e4b33660 con 0x5644e4b305d0 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c52538640 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b3f780 0x5644e4b36110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.049+0000 7f7c51536640 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b35bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c357fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c51d37640 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b305d0 0x5644e4b35690 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800c6a0 tx=0x7f7c4809bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c9ff30bdfa3138 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c357fa640 1 -- 172.21.15.33:0/3843970101 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c480bc0b0 con 0x5644e4b305d0 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c357fa640 1 -- 172.21.15.33:0/3843970101 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7c48098720 con 0x5644e4b305d0 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c56374740 1 -- 172.21.15.33:0/3843970101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c44030c30 con 0x5644e4b305d0 2024-09-16T00:40:39.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c357fa640 1 -- 172.21.15.33:0/3843970101 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7c480a00b0 con 0x5644e4b305d0 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c56374740 10 client.19764 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c56374740 1 -- 172.21.15.33:0/3843970101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5644e4bb60b0 con 0x5644e4b305d0 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c357fa640 1 -- 172.21.15.33:0/3843970101 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c44030c30 con 0x5644e4b305d0 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c357fa640 1 -- 172.21.15.33:0/3843970101 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7c4807ace0 con 0x5644e4b305d0 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c56374740 10 client.19764 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c56374740 10 client.19764 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c56374740 20 client.19764 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.050+0000 7f7c56374740 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5644e4bb6730 0x5644e4bb8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.051+0000 7f7c56374740 4 client.19764 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.051+0000 7f7c56374740 1 -- 172.21.15.33:0/3843970101 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5644e4a9dd70 con 0x5644e4bb6730 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.051+0000 7f7c51536640 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5644e4bb6730 0x5644e4bb8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.051+0000 7f7c357fa640 10 client.19764 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:40:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.051+0000 7f7c51536640 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5644e4bb6730 0x5644e4bb8b50 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2f44b90fd4916269 server_cookie=5b5169cf8eaf28bf in_seq=0 out_seq=0 2024-09-16T00:40:39.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c357fa640 1 -- 172.21.15.33:0/3843970101 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5644e4a9dd70 con 0x5644e4bb6730 2024-09-16T00:40:39.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c357fa640 10 client.19764 handle_command_reply: tid=0 2024-09-16T00:40:39.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c56374740 1 client.19764 shutdown 2024-09-16T00:40:39.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c56374740 1 -- 172.21.15.33:0/3843970101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b305d0 msgr2=0x5644e4b35690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:39.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c56374740 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b305d0 0x5644e4b35690 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800c6a0 tx=0x7f7c4809bbf0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c56374740 1 -- 172.21.15.33:0/3843970101 shutdown_connections 2024-09-16T00:40:39.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c56374740 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x5644e4bb6730 0x5644e4bb8b50 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c56374740 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5644e4b3f780 0x5644e4b36110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c56374740 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5644e4b31b40 0x5644e4b35bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c357fa640 0 client.19764 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:40:39.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c56374740 1 --2- 172.21.15.33:0/3843970101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5644e4b305d0 0x5644e4b35690 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c56374740 1 -- 172.21.15.33:0/3843970101 >> 172.21.15.33:0/3843970101 conn(0x5644e4af9310 msgr2=0x5644e4b2bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:39.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.052+0000 7f7c56374740 1 -- 172.21.15.33:0/3843970101 shutdown_connections 2024-09-16T00:40:39.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.053+0000 7f7c56374740 1 -- 172.21.15.33:0/3843970101 wait complete. 2024-09-16T00:40:39.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.053+0000 7f7c56374740 20 client.19764 trim_cache size 0 max 16384 2024-09-16T00:40:39.058 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:40:39.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.053+0000 7f7c53f9b640 1 -- 172.21.15.33:0/3448515427 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7c380a3300 msgr2=0x7f7c380a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:39.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.053+0000 7f7c53f9b640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7c380a3300 0x7f7c380a57c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c0095f0 tx=0x7f7c3c005f90 comp rx=0 tx=0).stop 2024-09-16T00:40:39.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.053+0000 7f7c53f9b640 1 -- 172.21.15.33:0/3448515427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c4c102d30 msgr2=0x7f7c4c073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:39.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.053+0000 7f7c53f9b640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c4c102d30 0x7f7c4c073bd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7c48005fd0 tx=0x7f7c4800edd0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.053+0000 7f7c53f9b640 1 -- 172.21.15.33:0/3448515427 shutdown_connections 2024-09-16T00:40:39.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.053+0000 7f7c53f9b640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7c380a3300 0x7f7c380a57c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.053+0000 7f7c53f9b640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4c1082a0 0x7f7c4c06e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.053+0000 7f7c53f9b640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1036e0 0x7f7c4c06dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.053+0000 7f7c53f9b640 1 --2- 172.21.15.33:0/3448515427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c4c102d30 0x7f7c4c073bd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:39.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.053+0000 7f7c53f9b640 1 -- 172.21.15.33:0/3448515427 >> 172.21.15.33:0/3448515427 conn(0x7f7c4c0fc7d0 msgr2=0x7f7c4c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:39.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.054+0000 7f7c53f9b640 1 -- 172.21.15.33:0/3448515427 shutdown_connections 2024-09-16T00:40:39.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:39.054+0000 7f7c53f9b640 1 -- 172.21.15.33:0/3448515427 wait complete. 2024-09-16T00:40:39.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 201518 18.28 MB/sec execute 465 sec latency 88.460 ms 2024-09-16T00:40:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:39 smithi083 ceph-mon[35242]: pgmap v1055: 129 pgs: 129 active+clean; 717 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 2.8 MiB/s wr, 359 op/s 2024-09-16T00:40:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:39 smithi060 ceph-mon[35185]: pgmap v1055: 129 pgs: 129 active+clean; 717 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 2.8 MiB/s wr, 359 op/s 2024-09-16T00:40:40.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:39 smithi033 ceph-mon[31243]: pgmap v1055: 129 pgs: 129 active+clean; 717 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 2.8 MiB/s wr, 359 op/s 2024-09-16T00:40:40.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 202060 18.30 MB/sec execute 466 sec latency 180.019 ms 2024-09-16T00:40:40.163 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:40:40.473 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:40:41.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.069+0000 7f3397e91640 1 -- 172.21.15.33:0/3297182197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f339011b140 msgr2=0x7f339011d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:41.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.070+0000 7f3397e91640 1 --2- 172.21.15.33:0/3297182197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f339011b140 0x7f339011d530 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f3388009ca0 tx=0x7f338802a8f0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.071+0000 7f3397e91640 1 -- 172.21.15.33:0/3297182197 shutdown_connections 2024-09-16T00:40:41.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.071+0000 7f3397e91640 1 --2- 172.21.15.33:0/3297182197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f339011b140 0x7f339011d530 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.071+0000 7f3397e91640 1 --2- 172.21.15.33:0/3297182197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3390109830 0x7f3390109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.071+0000 7f3397e91640 1 --2- 172.21.15.33:0/3297182197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3390109450 0x7f3390110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.071+0000 7f3397e91640 1 -- 172.21.15.33:0/3297182197 >> 172.21.15.33:0/3297182197 conn(0x7f339006e140 msgr2=0x7f339006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:41.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.071+0000 7f3397e91640 1 -- 172.21.15.33:0/3297182197 shutdown_connections 2024-09-16T00:40:41.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.071+0000 7f3397e91640 1 -- 172.21.15.33:0/3297182197 wait complete. 2024-09-16T00:40:41.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.072+0000 7f3397e91640 1 Processor -- start 2024-09-16T00:40:41.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.072+0000 7f3397e91640 1 -- start start 2024-09-16T00:40:41.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.073+0000 7f3397e91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3390109450 0x7f33901a7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.073+0000 7f3397e91640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3390109830 0x7f33901a76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.074+0000 7f3395c2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3390109450 0x7f33901a7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.074+0000 7f3395c2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3390109450 0x7f33901a7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47400/0 (socket says 172.21.15.33:47400) 2024-09-16T00:40:41.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.074+0000 7f3395c2d640 1 -- 172.21.15.33:0/2074109908 learned_addr learned my addr 172.21.15.33:0/2074109908 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:41.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.074+0000 7f3397e91640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f339011b140 0x7f33901aba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.075+0000 7f3397e91640 1 -- 172.21.15.33:0/2074109908 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3390120eb0 con 0x7f3390109450 2024-09-16T00:40:41.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.075+0000 7f3397e91640 1 -- 172.21.15.33:0/2074109908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3390120d30 con 0x7f3390109830 2024-09-16T00:40:41.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.075+0000 7f3397e91640 1 -- 172.21.15.33:0/2074109908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3390121030 con 0x7f339011b140 2024-09-16T00:40:41.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.075+0000 7f3395c2d640 1 -- 172.21.15.33:0/2074109908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f339011b140 msgr2=0x7f33901aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:41.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.075+0000 7f3395c2d640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f339011b140 0x7f33901aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.075+0000 7f3395c2d640 1 -- 172.21.15.33:0/2074109908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3390109830 msgr2=0x7f33901a76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:41.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.075+0000 7f3395c2d640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3390109830 0x7f33901a76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.075+0000 7f3395c2d640 1 -- 172.21.15.33:0/2074109908 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33901ac140 con 0x7f3390109450 2024-09-16T00:40:41.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.076+0000 7f3395c2d640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3390109450 0x7f33901a7190 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f338c00b4a0 tx=0x7f338c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:41.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.077+0000 7f3386ffd640 1 -- 172.21.15.33:0/2074109908 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338c014070 con 0x7f3390109450 2024-09-16T00:40:41.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.077+0000 7f3386ffd640 1 -- 172.21.15.33:0/2074109908 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f338c010af0 con 0x7f3390109450 2024-09-16T00:40:41.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.077+0000 7f3397e91640 1 -- 172.21.15.33:0/2074109908 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33901ac430 con 0x7f3390109450 2024-09-16T00:40:41.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.077+0000 7f3386ffd640 1 -- 172.21.15.33:0/2074109908 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338c017d10 con 0x7f3390109450 2024-09-16T00:40:41.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.077+0000 7f3397e91640 1 -- 172.21.15.33:0/2074109908 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33901b3d10 con 0x7f3390109450 2024-09-16T00:40:41.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.080+0000 7f339a26a740 1 Processor -- start 2024-09-16T00:40:41.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.083+0000 7f3386ffd640 1 -- 172.21.15.33:0/2074109908 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f338c017070 con 0x7f3390109450 2024-09-16T00:40:41.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.083+0000 7f339a26a740 1 -- start start 2024-09-16T00:40:41.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.083+0000 7f339a26a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 0x55cd8399f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.083+0000 7f339a26a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd8399fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.083+0000 7f339a26a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839aa370 0x55cd839ac760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.083+0000 7f339a26a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cd839052f0 con 0x55cd839a0b40 2024-09-16T00:40:41.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.083+0000 7f339a26a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55cd838accf0 con 0x55cd839aa370 2024-09-16T00:40:41.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.083+0000 7f339a26a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cd839012b0 con 0x55cd8399f5d0 2024-09-16T00:40:41.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.084+0000 7f339642e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 0x55cd8399f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.085+0000 7f3386ffd640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f337c0a3390 0x7f337c0a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.085+0000 7f339542c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839aa370 0x55cd839ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.085+0000 7f3386ffd640 1 -- 172.21.15.33:0/2074109908 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f338c0cf0f0 con 0x7f3390109450 2024-09-16T00:40:41.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.085+0000 7f339642e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 0x55cd8399f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47404/0 (socket says 172.21.15.33:47404) 2024-09-16T00:40:41.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.085+0000 7f339642e640 1 -- 172.21.15.33:0/4065250081 learned_addr learned my addr 172.21.15.33:0/4065250081 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:41.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.086+0000 7f339642e640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f337c0a3390 0x7f337c0a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.086+0000 7f339642e640 1 -- 172.21.15.33:0/4065250081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 msgr2=0x55cd8399fa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:40:41.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.086+0000 7f339642e640 1 --2- 172.21.15.33:0/4065250081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd8399fa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.086+0000 7f339642e640 1 -- 172.21.15.33:0/4065250081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839aa370 msgr2=0x55cd839ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:41.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.086+0000 7f339642e640 1 --2- 172.21.15.33:0/4065250081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839aa370 0x55cd839ac760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.086+0000 7f339642e640 1 -- 172.21.15.33:0/4065250081 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd838913a0 con 0x55cd839a0b40 2024-09-16T00:40:41.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.087+0000 7f339642e640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f337c0a3390 0x7f337c0a5850 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3388031570 tx=0x7f3388048000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:41.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.087+0000 7f339642e640 1 --2- 172.21.15.33:0/4065250081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 0x55cd8399f090 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f3388009bf0 tx=0x7f3388031500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7fda9c3dd35dd1c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:41.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.087+0000 7f337bfff640 1 -- 172.21.15.33:0/4065250081 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338802fed0 con 0x55cd839a0b40 2024-09-16T00:40:41.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.087+0000 7f337bfff640 1 -- 172.21.15.33:0/4065250081 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3388036040 con 0x55cd839a0b40 2024-09-16T00:40:41.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.088+0000 7f339a26a740 1 -- 172.21.15.33:0/4065250081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 msgr2=0x55cd8399f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:41.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.088+0000 7f339a26a740 1 --2- 172.21.15.33:0/4065250081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 0x55cd8399f090 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f3388009bf0 tx=0x7f3388031500 comp rx=0 tx=0).stop 2024-09-16T00:40:41.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.090+0000 7f339a26a740 1 -- 172.21.15.33:0/4065250081 shutdown_connections 2024-09-16T00:40:41.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.090+0000 7f339a26a740 1 --2- 172.21.15.33:0/4065250081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839aa370 0x55cd839ac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.090+0000 7f339a26a740 1 --2- 172.21.15.33:0/4065250081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd8399fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.090+0000 7f339a26a740 1 --2- 172.21.15.33:0/4065250081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 0x55cd8399f090 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.090+0000 7f339a26a740 1 -- 172.21.15.33:0/4065250081 >> 172.21.15.33:0/4065250081 conn(0x55cd83968310 msgr2=0x55cd8399bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:41.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.091+0000 7f339a26a740 1 -- 172.21.15.33:0/4065250081 shutdown_connections 2024-09-16T00:40:41.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.091+0000 7f339a26a740 1 -- 172.21.15.33:0/4065250081 wait complete. 2024-09-16T00:40:41.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.091+0000 7f339a26a740 1 -- start start 2024-09-16T00:40:41.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.091+0000 7f339a26a740 1 Processor -- start 2024-09-16T00:40:41.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.092+0000 7f339a26a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd839a9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.092+0000 7f339a26a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 0x55cd839a6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.092+0000 7f339a26a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839aa370 0x55cd839a7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.092+0000 7f339a26a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cd839ad060 con 0x55cd839a0b40 2024-09-16T00:40:41.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.092+0000 7f339a26a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55cd838c0060 con 0x55cd839aa370 2024-09-16T00:40:41.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.092+0000 7f339a26a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cd839ad1e0 con 0x55cd8399f5d0 2024-09-16T00:40:41.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.092+0000 7f339a26a740 10 client.0 _ll_register_callbacks cb 0x55cd836138c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:41.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.092+0000 7f339542c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 0x55cd839a6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.092+0000 7f339542c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 0x55cd839a6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47408/0 (socket says 172.21.15.33:47408) 2024-09-16T00:40:41.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.092+0000 7f339542c640 1 -- 172.21.15.33:0/2030942689 learned_addr learned my addr 172.21.15.33:0/2030942689 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:41.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.093+0000 7f339642e640 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839aa370 0x55cd839a7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.093+0000 7f3395c2d640 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd839a9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.094+0000 7f339542c640 1 -- 172.21.15.33:0/2030942689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 msgr2=0x55cd839a9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:41.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.094+0000 7f339542c640 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd839a9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.094+0000 7f339542c640 1 -- 172.21.15.33:0/2030942689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839aa370 msgr2=0x55cd839a7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:41.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.094+0000 7f339542c640 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839aa370 0x55cd839a7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.094+0000 7f339542c640 1 -- 172.21.15.33:0/2030942689 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd836763c0 con 0x55cd839a0b40 2024-09-16T00:40:41.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.094+0000 7f339542c640 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 0x55cd839a6fc0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f338000d160 tx=0x7f338000d620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2167b8743b5f5318 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:41.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.095+0000 7f33797fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:40:41.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.095+0000 7f33797fa640 1 -- 172.21.15.33:0/2030942689 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338000de50 con 0x55cd839a0b40 2024-09-16T00:40:41.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.095+0000 7f33797fa640 1 -- 172.21.15.33:0/2030942689 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f33800125f0 con 0x55cd839a0b40 2024-09-16T00:40:41.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.095+0000 7f339a26a740 1 -- 172.21.15.33:0/2030942689 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3388036040 con 0x55cd839a0b40 2024-09-16T00:40:41.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.095+0000 7f33797fa640 1 -- 172.21.15.33:0/2030942689 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3380022050 con 0x55cd839a0b40 2024-09-16T00:40:41.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.095+0000 7f339a26a740 10 client.19776 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:41.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.095+0000 7f339a26a740 1 -- 172.21.15.33:0/2030942689 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55cd839c9b70 con 0x55cd839a0b40 2024-09-16T00:40:41.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.096+0000 7f33797fa640 1 -- 172.21.15.33:0/2030942689 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3388036040 con 0x55cd839a0b40 2024-09-16T00:40:41.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.096+0000 7f33797fa640 1 -- 172.21.15.33:0/2030942689 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f3380013070 con 0x55cd839a0b40 2024-09-16T00:40:41.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.096+0000 7f339a26a740 10 client.19776 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:41.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.096+0000 7f339a26a740 10 client.19776 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:40:41.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.097+0000 7f339a26a740 20 client.19776 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:41.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.097+0000 7f339a26a740 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55cd839cb080 0x55cd839cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.097+0000 7f339a26a740 4 client.19776 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:40:41.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.097+0000 7f339a26a740 1 -- 172.21.15.33:0/2030942689 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd83912860 con 0x55cd839cb080 2024-09-16T00:40:41.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.097+0000 7f3395c2d640 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55cd839cb080 0x55cd839cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.098+0000 7f3395c2d640 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55cd839cb080 0x55cd839cd4a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f7eb04e0ed725c7a server_cookie=dd277d062267eab8 in_seq=0 out_seq=0 2024-09-16T00:40:41.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.098+0000 7f33797fa640 10 client.19776 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:40:41.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.101+0000 7f33797fa640 1 -- 172.21.15.33:0/2030942689 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cd83912860 con 0x55cd839cb080 2024-09-16T00:40:41.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.101+0000 7f33797fa640 10 client.19776 handle_command_reply: tid=0 2024-09-16T00:40:41.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.101+0000 7f339a26a740 1 client.19776 shutdown 2024-09-16T00:40:41.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 202466 18.30 MB/sec execute 467 sec latency 209.581 ms 2024-09-16T00:40:41.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.101+0000 7f339a26a740 1 -- 172.21.15.33:0/2030942689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 msgr2=0x55cd839a6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:41.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.101+0000 7f339a26a740 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 0x55cd839a6fc0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f338000d160 tx=0x7f338000d620 comp rx=0 tx=0).stop 2024-09-16T00:40:41.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.101+0000 7f339a26a740 1 -- 172.21.15.33:0/2030942689 shutdown_connections 2024-09-16T00:40:41.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.101+0000 7f339a26a740 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55cd839cb080 0x55cd839cd4a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.101+0000 7f339a26a740 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839aa370 0x55cd839a7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.101+0000 7f339a26a740 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839a0b40 0x55cd839a6fc0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.102+0000 7f339a26a740 1 --2- 172.21.15.33:0/2030942689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd839a9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.102+0000 7f339a26a740 1 -- 172.21.15.33:0/2030942689 >> 172.21.15.33:0/2030942689 conn(0x55cd83968310 msgr2=0x55cd839a1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:41.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.102+0000 7f33797fa640 0 client.19776 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:40:41.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.102+0000 7f339a26a740 1 -- 172.21.15.33:0/2030942689 shutdown_connections 2024-09-16T00:40:41.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.102+0000 7f339a26a740 1 -- 172.21.15.33:0/2030942689 wait complete. 2024-09-16T00:40:41.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.103+0000 7f339a26a740 20 client.19776 trim_cache size 0 max 16384 2024-09-16T00:40:41.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.127+0000 7f339a26a740 1 Processor -- start 2024-09-16T00:40:41.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.127+0000 7f339a26a740 1 -- start start 2024-09-16T00:40:41.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.128+0000 7f339a26a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd8399f5d0 0x55cd839d2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.128+0000 7f339a26a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd839a0b40 0x55cd839d3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.128+0000 7f339a26a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839aa370 0x55cd8399b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.128+0000 7f339a26a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f33901b82e0 con 0x55cd839aa370 2024-09-16T00:40:41.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.128+0000 7f339a26a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f33901b5bb0 con 0x55cd8399f5d0 2024-09-16T00:40:41.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.128+0000 7f339a26a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3390118c40 con 0x55cd839a0b40 2024-09-16T00:40:41.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.128+0000 7f339642e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839aa370 0x55cd8399b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.128+0000 7f339642e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839aa370 0x55cd8399b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47416/0 (socket says 172.21.15.33:47416) 2024-09-16T00:40:41.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.128+0000 7f339642e640 1 -- 172.21.15.33:0/1652410928 learned_addr learned my addr 172.21.15.33:0/1652410928 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:41.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.128+0000 7f339542c640 1 --2- 172.21.15.33:0/1652410928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd839a0b40 0x55cd839d3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.128+0000 7f3395c2d640 1 --2- 172.21.15.33:0/1652410928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd8399f5d0 0x55cd839d2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.129+0000 7f339542c640 1 -- 172.21.15.33:0/1652410928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd8399f5d0 msgr2=0x55cd839d2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:41.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.129+0000 7f339542c640 1 --2- 172.21.15.33:0/1652410928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd8399f5d0 0x55cd839d2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.129+0000 7f339542c640 1 -- 172.21.15.33:0/1652410928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839aa370 msgr2=0x55cd8399b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:41.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.129+0000 7f339542c640 1 --2- 172.21.15.33:0/1652410928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839aa370 0x55cd8399b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.129+0000 7f339542c640 1 -- 172.21.15.33:0/1652410928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd839aa080 con 0x55cd839a0b40 2024-09-16T00:40:41.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.129+0000 7f339542c640 1 --2- 172.21.15.33:0/1652410928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd839a0b40 0x55cd839d3290 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3380014b10 tx=0x7f338000bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fbb972cca9774dd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:41.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.130+0000 7f337b7fe640 1 -- 172.21.15.33:0/1652410928 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338002a020 con 0x55cd839a0b40 2024-09-16T00:40:41.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.130+0000 7f337b7fe640 1 -- 172.21.15.33:0/1652410928 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3380019070 con 0x55cd839a0b40 2024-09-16T00:40:41.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.130+0000 7f337b7fe640 1 -- 172.21.15.33:0/1652410928 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380012ba0 con 0x55cd839a0b40 2024-09-16T00:40:41.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.131+0000 7f339a26a740 1 -- 172.21.15.33:0/1652410928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd839a0b40 msgr2=0x55cd839d3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:41.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.131+0000 7f339a26a740 1 --2- 172.21.15.33:0/1652410928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd839a0b40 0x55cd839d3290 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3380014b10 tx=0x7f338000bc40 comp rx=0 tx=0).stop 2024-09-16T00:40:41.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.132+0000 7f339a26a740 1 -- 172.21.15.33:0/1652410928 shutdown_connections 2024-09-16T00:40:41.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.132+0000 7f339a26a740 1 --2- 172.21.15.33:0/1652410928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839aa370 0x55cd8399b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.132+0000 7f339a26a740 1 --2- 172.21.15.33:0/1652410928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd839a0b40 0x55cd839d3290 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.132+0000 7f339a26a740 1 --2- 172.21.15.33:0/1652410928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd8399f5d0 0x55cd839d2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.132+0000 7f339a26a740 1 -- 172.21.15.33:0/1652410928 >> 172.21.15.33:0/1652410928 conn(0x55cd83968310 msgr2=0x55cd839a60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:41.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.133+0000 7f339a26a740 1 -- 172.21.15.33:0/1652410928 shutdown_connections 2024-09-16T00:40:41.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.133+0000 7f339a26a740 1 -- 172.21.15.33:0/1652410928 wait complete. 2024-09-16T00:40:41.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.134+0000 7f339a26a740 1 -- start start 2024-09-16T00:40:41.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.134+0000 7f339a26a740 1 Processor -- start 2024-09-16T00:40:41.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.135+0000 7f339a26a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd839b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.135+0000 7f339a26a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839a0b40 0x55cd839b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.136+0000 7f339a26a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839aa370 0x55cd839a3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.136+0000 7f339a26a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f33901b4070 con 0x55cd839aa370 2024-09-16T00:40:41.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.136+0000 7f339a26a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f33901b54e0 con 0x55cd839a0b40 2024-09-16T00:40:41.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.136+0000 7f339a26a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f33901b4740 con 0x55cd8399f5d0 2024-09-16T00:40:41.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.136+0000 7f339a26a740 10 client.0 _ll_register_callbacks cb 0x7f338000b100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:41.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.136+0000 7f339542c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839a0b40 0x55cd839b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.136+0000 7f3395c2d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd839b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.136+0000 7f3395c2d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd839b27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34848/0 (socket says 172.21.15.33:34848) 2024-09-16T00:40:41.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.136+0000 7f3395c2d640 1 -- 172.21.15.33:0/774804806 learned_addr learned my addr 172.21.15.33:0/774804806 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:41.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.136+0000 7f339642e640 1 --2- 172.21.15.33:0/774804806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839aa370 0x55cd839a3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.137+0000 7f3395c2d640 1 -- 172.21.15.33:0/774804806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839a0b40 msgr2=0x55cd839b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:41.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.137+0000 7f3395c2d640 1 --2- 172.21.15.33:0/774804806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839a0b40 0x55cd839b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.137+0000 7f3395c2d640 1 -- 172.21.15.33:0/774804806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839aa370 msgr2=0x55cd839a3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:41.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.137+0000 7f3395c2d640 1 --2- 172.21.15.33:0/774804806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839aa370 0x55cd839a3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.137+0000 7f3395c2d640 1 -- 172.21.15.33:0/774804806 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd839a4230 con 0x55cd8399f5d0 2024-09-16T00:40:41.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.137+0000 7f3395c2d640 1 --2- 172.21.15.33:0/774804806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd839b27a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f338c0cc710 tx=0x7f338c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7d680a847dae881 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:41.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.137+0000 7f33797fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:40:41.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.137+0000 7f33797fa640 1 -- 172.21.15.33:0/774804806 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338c01e040 con 0x55cd8399f5d0 2024-09-16T00:40:41.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.137+0000 7f33797fa640 1 -- 172.21.15.33:0/774804806 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f338c0978a0 con 0x55cd8399f5d0 2024-09-16T00:40:41.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.138+0000 7f339a26a740 1 -- 172.21.15.33:0/774804806 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3380019070 con 0x55cd8399f5d0 2024-09-16T00:40:41.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.138+0000 7f33797fa640 1 -- 172.21.15.33:0/774804806 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f338c010ca0 con 0x55cd8399f5d0 2024-09-16T00:40:41.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.138+0000 7f339a26a740 10 client.28927 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:41.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.138+0000 7f339a26a740 1 -- 172.21.15.33:0/774804806 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55cd83a35940 con 0x55cd8399f5d0 2024-09-16T00:40:41.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.139+0000 7f33797fa640 1 -- 172.21.15.33:0/774804806 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f338c0af290 con 0x55cd8399f5d0 2024-09-16T00:40:41.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.140+0000 7f33797fa640 1 -- 172.21.15.33:0/774804806 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f338c0bb040 con 0x55cd8399f5d0 2024-09-16T00:40:41.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.140+0000 7f339a26a740 10 client.28927 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:41.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.140+0000 7f339a26a740 10 client.28927 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:40:41.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.140+0000 7f339a26a740 20 client.28927 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:41.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.140+0000 7f339a26a740 1 --2- 172.21.15.33:0/774804806 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55cd83a36060 0x55cd83a38480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:41.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.140+0000 7f339a26a740 4 client.28927 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:40:41.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.140+0000 7f339a26a740 1 -- 172.21.15.33:0/774804806 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f3380019070 con 0x55cd83a36060 2024-09-16T00:40:41.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.141+0000 7f339542c640 1 --2- 172.21.15.33:0/774804806 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55cd83a36060 0x55cd83a38480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:41.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.143+0000 7f339542c640 1 --2- 172.21.15.33:0/774804806 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55cd83a36060 0x55cd83a38480 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9ee094eed923c899 server_cookie=18d02fa1348dc44a in_seq=0 out_seq=0 2024-09-16T00:40:41.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.143+0000 7f33797fa640 10 client.28927 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:40:41.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.145+0000 7f33797fa640 1 -- 172.21.15.33:0/774804806 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f3380019070 con 0x55cd83a36060 2024-09-16T00:40:41.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.145+0000 7f33797fa640 10 client.28927 handle_command_reply: tid=0 2024-09-16T00:40:41.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.145+0000 7f339a26a740 1 client.28927 shutdown 2024-09-16T00:40:41.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.145+0000 7f339a26a740 1 -- 172.21.15.33:0/774804806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 msgr2=0x55cd839b27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:41.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.145+0000 7f339a26a740 1 --2- 172.21.15.33:0/774804806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd839b27a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f338c0cc710 tx=0x7f338c0046c0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.146+0000 7f339a26a740 1 -- 172.21.15.33:0/774804806 shutdown_connections 2024-09-16T00:40:41.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.146+0000 7f339a26a740 1 --2- 172.21.15.33:0/774804806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd839aa370 0x55cd839a3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.146+0000 7f339a26a740 1 --2- 172.21.15.33:0/774804806 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55cd83a36060 0x55cd83a38480 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.146+0000 7f339a26a740 1 --2- 172.21.15.33:0/774804806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd839a0b40 0x55cd839b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.146+0000 7f33797fa640 0 client.28927 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:40:41.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.146+0000 7f339a26a740 1 --2- 172.21.15.33:0/774804806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd8399f5d0 0x55cd839b27a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.146+0000 7f339a26a740 1 -- 172.21.15.33:0/774804806 >> 172.21.15.33:0/774804806 conn(0x55cd83968310 msgr2=0x55cd839a1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:41.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.146+0000 7f339a26a740 1 -- 172.21.15.33:0/774804806 shutdown_connections 2024-09-16T00:40:41.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.146+0000 7f339a26a740 1 -- 172.21.15.33:0/774804806 wait complete. 2024-09-16T00:40:41.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.147+0000 7f339a26a740 20 client.28927 trim_cache size 0 max 16384 2024-09-16T00:40:41.147 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:40:41.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.148+0000 7f3397e91640 1 -- 172.21.15.33:0/2074109908 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f337c0a3390 msgr2=0x7f337c0a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:41.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.148+0000 7f3397e91640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f337c0a3390 0x7f337c0a5850 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3388031570 tx=0x7f3388048000 comp rx=0 tx=0).stop 2024-09-16T00:40:41.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.148+0000 7f3397e91640 1 -- 172.21.15.33:0/2074109908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3390109450 msgr2=0x7f33901a7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:41.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.148+0000 7f3397e91640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3390109450 0x7f33901a7190 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f338c00b4a0 tx=0x7f338c00b960 comp rx=0 tx=0).stop 2024-09-16T00:40:41.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.148+0000 7f3397e91640 1 -- 172.21.15.33:0/2074109908 shutdown_connections 2024-09-16T00:40:41.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.148+0000 7f3397e91640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f337c0a3390 0x7f337c0a5850 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.148+0000 7f3397e91640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f339011b140 0x7f33901aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.148+0000 7f3397e91640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3390109830 0x7f33901a76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.148+0000 7f3397e91640 1 --2- 172.21.15.33:0/2074109908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3390109450 0x7f33901a7190 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:41.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.148+0000 7f3397e91640 1 -- 172.21.15.33:0/2074109908 >> 172.21.15.33:0/2074109908 conn(0x7f339006e140 msgr2=0x7f339011be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:41.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.148+0000 7f3397e91640 1 -- 172.21.15.33:0/2074109908 shutdown_connections 2024-09-16T00:40:41.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:41.149+0000 7f3397e91640 1 -- 172.21.15.33:0/2074109908 wait complete. 2024-09-16T00:40:41.884 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:41 smithi033 ceph-mon[31243]: pgmap v1056: 129 pgs: 129 active+clean; 722 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.1 MiB/s wr, 393 op/s 2024-09-16T00:40:41.885 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:40:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:41 smithi083 ceph-mon[35242]: pgmap v1056: 129 pgs: 129 active+clean; 722 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.1 MiB/s wr, 393 op/s 2024-09-16T00:40:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:41 smithi060 ceph-mon[35185]: pgmap v1056: 129 pgs: 129 active+clean; 722 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.1 MiB/s wr, 393 op/s 2024-09-16T00:40:42.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 203024 18.33 MB/sec execute 468 sec latency 96.782 ms 2024-09-16T00:40:42.179 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:40:43.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 203580 18.35 MB/sec execute 469 sec latency 59.911 ms 2024-09-16T00:40:43.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.485+0000 7f50fe7ec640 1 -- 172.21.15.33:0/530369935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f806ea30 msgr2=0x7f50f806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:43.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.485+0000 7f50fe7ec640 1 --2- 172.21.15.33:0/530369935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f806ea30 0x7f50f806ee70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009ca0 tx=0x7f50f002a8c0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.486+0000 7f50fe7ec640 1 -- 172.21.15.33:0/530369935 shutdown_connections 2024-09-16T00:40:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.486+0000 7f50fe7ec640 1 --2- 172.21.15.33:0/530369935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50f811b140 0x7f50f811d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.486+0000 7f50fe7ec640 1 --2- 172.21.15.33:0/530369935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f806ea30 0x7f50f806ee70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.486+0000 7f50fe7ec640 1 --2- 172.21.15.33:0/530369935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50f8110190 0x7f50f806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.486+0000 7f50fe7ec640 1 -- 172.21.15.33:0/530369935 >> 172.21.15.33:0/530369935 conn(0x7f50f8073180 msgr2=0x7f50f8073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.486+0000 7f50fe7ec640 1 -- 172.21.15.33:0/530369935 shutdown_connections 2024-09-16T00:40:43.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.486+0000 7f50fe7ec640 1 -- 172.21.15.33:0/530369935 wait complete. 2024-09-16T00:40:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.487+0000 7f50fe7ec640 1 Processor -- start 2024-09-16T00:40:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.487+0000 7f50fe7ec640 1 -- start start 2024-09-16T00:40:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fe7ec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50f806ea30 0x7f50f81a7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fe7ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f8110190 0x7f50f81a7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fe7ec640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50f811b140 0x7f50f81ab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fe7ec640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50f8120be0 con 0x7f50f811b140 2024-09-16T00:40:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fe7ec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f50f8120a60 con 0x7f50f8110190 2024-09-16T00:40:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fe7ec640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f50f8120d60 con 0x7f50f806ea30 2024-09-16T00:40:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fd7ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50f806ea30 0x7f50f81a7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fdfeb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50f811b140 0x7f50f81ab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fd7ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50f806ea30 0x7f50f81a7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34880/0 (socket says 172.21.15.33:34880) 2024-09-16T00:40:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fd7ea640 1 -- 172.21.15.33:0/3999964273 learned_addr learned my addr 172.21.15.33:0/3999964273 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fd7ea640 1 -- 172.21.15.33:0/3999964273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f8110190 msgr2=0x7f50f81a7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/3999964273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f8110190 0x7f50f81a7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fd7ea640 1 -- 172.21.15.33:0/3999964273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50f811b140 msgr2=0x7f50f81ab9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.488+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/3999964273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50f811b140 0x7f50f81ab9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.489+0000 7f50fd7ea640 1 -- 172.21.15.33:0/3999964273 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f81ac0d0 con 0x7f50f806ea30 2024-09-16T00:40:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.489+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/3999964273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50f806ea30 0x7f50f81a7120 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f50e800d950 tx=0x7f50e800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.489+0000 7f50ee7fc640 1 -- 172.21.15.33:0/3999964273 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e8017070 con 0x7f50f806ea30 2024-09-16T00:40:43.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.489+0000 7f50fe7ec640 1 -- 172.21.15.33:0/3999964273 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50f81ac3c0 con 0x7f50f806ea30 2024-09-16T00:40:43.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.489+0000 7f50fe7ec640 1 -- 172.21.15.33:0/3999964273 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50f81b3ca0 con 0x7f50f806ea30 2024-09-16T00:40:43.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.490+0000 7f50ee7fc640 1 -- 172.21.15.33:0/3999964273 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50e8009c30 con 0x7f50f806ea30 2024-09-16T00:40:43.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.490+0000 7f50ee7fc640 1 -- 172.21.15.33:0/3999964273 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e800f6b0 con 0x7f50f806ea30 2024-09-16T00:40:43.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.493+0000 7f50ee7fc640 1 -- 172.21.15.33:0/3999964273 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f50e800f850 con 0x7f50f806ea30 2024-09-16T00:40:43.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.494+0000 7f5101e27740 1 Processor -- start 2024-09-16T00:40:43.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.495+0000 7f5101e27740 1 -- start start 2024-09-16T00:40:43.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.496+0000 7f5101e27740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90dd20 0x55e12d90c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.496+0000 7f5101e27740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 0x55e12d90cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.496+0000 7f5101e27740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d919940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.496+0000 7f50fdfeb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90dd20 0x55e12d90c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.496+0000 7f5101e27740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e12d8722f0 con 0x55e12d90c7b0 2024-09-16T00:40:43.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.496+0000 7f5101e27740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e12d819cf0 con 0x55e12d90dd20 2024-09-16T00:40:43.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.496+0000 7f50fdfeb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90dd20 0x55e12d90c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42334/0 (socket says 172.21.15.33:42334) 2024-09-16T00:40:43.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.496+0000 7f50fdfeb640 1 -- 172.21.15.33:0/2492140623 learned_addr learned my addr 172.21.15.33:0/2492140623 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:43.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.496+0000 7f5101e27740 1 -- 172.21.15.33:0/2492140623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e12d86e2b0 con 0x55e12d917550 2024-09-16T00:40:43.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.497+0000 7f50fcfe9640 1 --2- 172.21.15.33:0/2492140623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d919940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.497+0000 7f50ee7fc640 1 --2- 172.21.15.33:0/3999964273 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f50e40a3340 0x7f50e40a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.497+0000 7f50fdfeb640 1 --2- 172.21.15.33:0/3999964273 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f50e40a3340 0x7f50e40a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.497+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/2492140623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 0x55e12d90cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.498+0000 7f50ee7fc640 1 -- 172.21.15.33:0/3999964273 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f50e80cfc50 con 0x7f50f806ea30 2024-09-16T00:40:43.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.499+0000 7f50fcfe9640 1 -- 172.21.15.33:0/2492140623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90dd20 msgr2=0x55e12d90c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:43.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.499+0000 7f50fcfe9640 1 --2- 172.21.15.33:0/2492140623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90dd20 0x55e12d90c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.499+0000 7f50fcfe9640 1 -- 172.21.15.33:0/2492140623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 msgr2=0x55e12d90cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:43.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.499+0000 7f50fcfe9640 1 --2- 172.21.15.33:0/2492140623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 0x55e12d90cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.499+0000 7f50fcfe9640 1 -- 172.21.15.33:0/2492140623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e12d7fe3a0 con 0x55e12d917550 2024-09-16T00:40:43.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.499+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/2492140623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 0x55e12d90cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:40:43.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.499+0000 7f50fdfeb640 1 --2- 172.21.15.33:0/2492140623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90dd20 0x55e12d90c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:40:43.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.499+0000 7f50fcfe9640 1 --2- 172.21.15.33:0/2492140623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d919940 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f50f002aeb0 tx=0x7f50f0005fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ef6aaa04c9e275e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:43.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.499+0000 7f50fdfeb640 1 --2- 172.21.15.33:0/3999964273 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f50e40a3340 0x7f50e40a5800 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f50f400ba20 tx=0x7f50f4004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.500+0000 7f50e37fe640 1 -- 172.21.15.33:0/2492140623 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f0031db0 con 0x55e12d917550 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.500+0000 7f50e37fe640 1 -- 172.21.15.33:0/2492140623 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f003f070 con 0x55e12d917550 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.500+0000 7f50e37fe640 1 -- 172.21.15.33:0/2492140623 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f00042a0 con 0x55e12d917550 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.501+0000 7f5101e27740 1 -- 172.21.15.33:0/2492140623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 msgr2=0x55e12d919940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.501+0000 7f5101e27740 1 --2- 172.21.15.33:0/2492140623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d919940 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f50f002aeb0 tx=0x7f50f0005fc0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.501+0000 7f5101e27740 1 -- 172.21.15.33:0/2492140623 shutdown_connections 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.501+0000 7f5101e27740 1 --2- 172.21.15.33:0/2492140623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d919940 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.501+0000 7f5101e27740 1 --2- 172.21.15.33:0/2492140623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 0x55e12d90cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.501+0000 7f5101e27740 1 --2- 172.21.15.33:0/2492140623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90dd20 0x55e12d90c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.501+0000 7f5101e27740 1 -- 172.21.15.33:0/2492140623 >> 172.21.15.33:0/2492140623 conn(0x55e12d8d5310 msgr2=0x55e12d908e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.501+0000 7f5101e27740 1 -- 172.21.15.33:0/2492140623 shutdown_connections 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.501+0000 7f5101e27740 1 -- 172.21.15.33:0/2492140623 wait complete. 2024-09-16T00:40:43.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.501+0000 7f5101e27740 1 -- start start 2024-09-16T00:40:43.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.502+0000 7f5101e27740 1 Processor -- start 2024-09-16T00:40:43.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.502+0000 7f5101e27740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 0x55e12d916f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f5101e27740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d90dd20 0x55e12d9141c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f5101e27740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d917550 0x55e12d9147b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f50fd7ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 0x55e12d916f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f5101e27740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e12d91a1f0 con 0x55e12d90c7b0 2024-09-16T00:40:43.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f5101e27740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e12d82d060 con 0x55e12d917550 2024-09-16T00:40:43.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f5101e27740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e12d91a370 con 0x55e12d90dd20 2024-09-16T00:40:43.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f50fd7ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 0x55e12d916f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47450/0 (socket says 172.21.15.33:47450) 2024-09-16T00:40:43.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f50fd7ea640 1 -- 172.21.15.33:0/1854596733 learned_addr learned my addr 172.21.15.33:0/1854596733 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:43.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f50fcfe9640 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d90dd20 0x55e12d9141c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f5101e27740 10 client.0 _ll_register_callbacks cb 0x55e12d5808c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:43.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f50fdfeb640 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d917550 0x55e12d9147b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f50fdfeb640 1 -- 172.21.15.33:0/1854596733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d90dd20 msgr2=0x55e12d9141c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:43.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f50fdfeb640 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d90dd20 0x55e12d9141c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f50fdfeb640 1 -- 172.21.15.33:0/1854596733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 msgr2=0x55e12d916f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:43.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f50fdfeb640 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 0x55e12d916f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f50fdfeb640 1 -- 172.21.15.33:0/1854596733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e12d5e33c0 con 0x55e12d917550 2024-09-16T00:40:43.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 0x55e12d916f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:40:43.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.503+0000 7f50fcfe9640 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d90dd20 0x55e12d9141c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:40:43.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.504+0000 7f50fdfeb640 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d917550 0x55e12d9147b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f50f4004530 tx=0x7f50f400b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=269dd941efce3dfd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:43.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.504+0000 7f50e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:40:43.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.504+0000 7f50e0ff9640 1 -- 172.21.15.33:0/1854596733 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f4010840 con 0x55e12d917550 2024-09-16T00:40:43.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.504+0000 7f50e0ff9640 1 -- 172.21.15.33:0/1854596733 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f50f40109e0 con 0x55e12d917550 2024-09-16T00:40:43.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.504+0000 7f5101e27740 1 -- 172.21.15.33:0/1854596733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50f003f070 con 0x55e12d917550 2024-09-16T00:40:43.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.505+0000 7f50e0ff9640 1 -- 172.21.15.33:0/1854596733 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f50f4011e50 con 0x55e12d917550 2024-09-16T00:40:43.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.505+0000 7f5101e27740 10 client.28925 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:43.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.505+0000 7f5101e27740 1 -- 172.21.15.33:0/1854596733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e12d925c00 con 0x55e12d917550 2024-09-16T00:40:43.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.505+0000 7f50e0ff9640 1 -- 172.21.15.33:0/1854596733 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f003f070 con 0x55e12d917550 2024-09-16T00:40:43.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.505+0000 7f50e0ff9640 1 -- 172.21.15.33:0/1854596733 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f50f402b450 con 0x55e12d917550 2024-09-16T00:40:43.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.505+0000 7f5101e27740 10 client.28925 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:43.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.505+0000 7f5101e27740 10 client.28925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:40:43.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.505+0000 7f5101e27740 20 client.28925 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:43.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.505+0000 7f5101e27740 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e12d9270c0 0x55e12d929560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.505+0000 7f5101e27740 4 client.28925 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:40:43.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.505+0000 7f5101e27740 1 -- 172.21.15.33:0/1854596733 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e12d87f860 con 0x55e12d9270c0 2024-09-16T00:40:43.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.506+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e12d9270c0 0x55e12d929560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.506+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e12d9270c0 0x55e12d929560 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3794cab734bf3bdb server_cookie=64c4709aa6694bc4 in_seq=0 out_seq=0 2024-09-16T00:40:43.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.506+0000 7f50e0ff9640 10 client.28925 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.507+0000 7f50e0ff9640 1 -- 172.21.15.33:0/1854596733 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e12d87f860 con 0x55e12d9270c0 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.507+0000 7f50e0ff9640 10 client.28925 handle_command_reply: tid=0 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.507+0000 7f5101e27740 1 client.28925 shutdown 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.507+0000 7f5101e27740 1 -- 172.21.15.33:0/1854596733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d917550 msgr2=0x55e12d9147b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.507+0000 7f5101e27740 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d917550 0x55e12d9147b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f50f4004530 tx=0x7f50f400b260 comp rx=0 tx=0).stop 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.508+0000 7f5101e27740 1 -- 172.21.15.33:0/1854596733 shutdown_connections 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.508+0000 7f5101e27740 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e12d9270c0 0x55e12d929560 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.508+0000 7f50e0ff9640 0 client.28925 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.508+0000 7f5101e27740 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d917550 0x55e12d9147b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.508+0000 7f5101e27740 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d90dd20 0x55e12d9141c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.508+0000 7f5101e27740 1 --2- 172.21.15.33:0/1854596733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90c7b0 0x55e12d916f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.508+0000 7f5101e27740 1 -- 172.21.15.33:0/1854596733 >> 172.21.15.33:0/1854596733 conn(0x55e12d8d5310 msgr2=0x55e12d90eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:43.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.508+0000 7f5101e27740 1 -- 172.21.15.33:0/1854596733 shutdown_connections 2024-09-16T00:40:43.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.508+0000 7f5101e27740 1 -- 172.21.15.33:0/1854596733 wait complete. 2024-09-16T00:40:43.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.508+0000 7f5101e27740 20 client.28925 trim_cache size 0 max 16384 2024-09-16T00:40:43.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f5101e27740 1 Processor -- start 2024-09-16T00:40:43.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f5101e27740 1 -- start start 2024-09-16T00:40:43.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f5101e27740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 0x55e12d9234f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f5101e27740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90dd20 0x55e12d913a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f5101e27740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d912ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f5101e27740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc012930 con 0x55e12d90dd20 2024-09-16T00:40:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f5101e27740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc010200 con 0x55e12d90c7b0 2024-09-16T00:40:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f5101e27740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc00e050 con 0x55e12d917550 2024-09-16T00:40:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f50fd7ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 0x55e12d9234f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f50fdfeb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d912ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f50fd7ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 0x55e12d9234f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42350/0 (socket says 172.21.15.33:42350) 2024-09-16T00:40:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f50fdfeb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d912ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34898/0 (socket says 172.21.15.33:34898) 2024-09-16T00:40:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.531+0000 7f50fd7ea640 1 -- 172.21.15.33:0/2911435289 learned_addr learned my addr 172.21.15.33:0/2911435289 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.532+0000 7f50fcfe9640 1 --2- 172.21.15.33:0/2911435289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90dd20 0x55e12d913a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.532+0000 7f50fd7ea640 1 -- 172.21.15.33:0/2911435289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 msgr2=0x55e12d912ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.532+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/2911435289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d912ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.532+0000 7f50fd7ea640 1 -- 172.21.15.33:0/2911435289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90dd20 msgr2=0x55e12d913a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.532+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/2911435289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90dd20 0x55e12d913a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.532+0000 7f50fd7ea640 1 -- 172.21.15.33:0/2911435289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e12d917240 con 0x55e12d90c7b0 2024-09-16T00:40:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.532+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/2911435289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 0x55e12d9234f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f50e809ba90 tx=0x7f50e80a6040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b5e037584f27e30 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.532+0000 7f50e2ffd640 1 -- 172.21.15.33:0/2911435289 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e8020040 con 0x55e12d90c7b0 2024-09-16T00:40:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.532+0000 7f50e2ffd640 1 -- 172.21.15.33:0/2911435289 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50e80a0140 con 0x55e12d90c7b0 2024-09-16T00:40:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.532+0000 7f50e2ffd640 1 -- 172.21.15.33:0/2911435289 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e80a4570 con 0x55e12d90c7b0 2024-09-16T00:40:43.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.533+0000 7f5101e27740 1 -- 172.21.15.33:0/2911435289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 msgr2=0x55e12d9234f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:43.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.533+0000 7f5101e27740 1 --2- 172.21.15.33:0/2911435289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 0x55e12d9234f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f50e809ba90 tx=0x7f50e80a6040 comp rx=0 tx=0).stop 2024-09-16T00:40:43.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.533+0000 7f5101e27740 1 -- 172.21.15.33:0/2911435289 shutdown_connections 2024-09-16T00:40:43.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.533+0000 7f5101e27740 1 --2- 172.21.15.33:0/2911435289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d912ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.533+0000 7f5101e27740 1 --2- 172.21.15.33:0/2911435289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90dd20 0x55e12d913a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.533+0000 7f5101e27740 1 --2- 172.21.15.33:0/2911435289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 0x55e12d9234f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.533+0000 7f5101e27740 1 -- 172.21.15.33:0/2911435289 >> 172.21.15.33:0/2911435289 conn(0x55e12d8d5310 msgr2=0x55e12d914180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:43.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.534+0000 7f5101e27740 1 -- 172.21.15.33:0/2911435289 shutdown_connections 2024-09-16T00:40:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.534+0000 7f5101e27740 1 -- 172.21.15.33:0/2911435289 wait complete. 2024-09-16T00:40:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.534+0000 7f5101e27740 1 -- start start 2024-09-16T00:40:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.534+0000 7f5101e27740 1 Processor -- start 2024-09-16T00:40:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.535+0000 7f5101e27740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 0x55e12d91fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.535+0000 7f5101e27740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90dd20 0x55e12d91ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.535+0000 7f5101e27740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d910de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.535+0000 7f50fd7ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 0x55e12d91fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.535+0000 7f5101e27740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc00e6c0 con 0x55e12d90dd20 2024-09-16T00:40:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.535+0000 7f5101e27740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc00fb30 con 0x55e12d90c7b0 2024-09-16T00:40:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.535+0000 7f5101e27740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc00ed90 con 0x55e12d917550 2024-09-16T00:40:43.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.535+0000 7f50fd7ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 0x55e12d91fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42366/0 (socket says 172.21.15.33:42366) 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.535+0000 7f50fd7ea640 1 -- 172.21.15.33:0/2182616846 learned_addr learned my addr 172.21.15.33:0/2182616846 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.535+0000 7f50fcfe9640 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90dd20 0x55e12d91ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.535+0000 7f5101e27740 10 client.0 _ll_register_callbacks cb 0x7f50f400b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.535+0000 7f50fdfeb640 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d910de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.536+0000 7f50fd7ea640 1 -- 172.21.15.33:0/2182616846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 msgr2=0x55e12d910de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.536+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d910de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.536+0000 7f50fd7ea640 1 -- 172.21.15.33:0/2182616846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90dd20 msgr2=0x55e12d91ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.536+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90dd20 0x55e12d91ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.536+0000 7f50fd7ea640 1 -- 172.21.15.33:0/2182616846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e12d9115e0 con 0x55e12d90c7b0 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.536+0000 7f50fd7ea640 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 0x55e12d91fa20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f50e809b7c0 tx=0x7f50e809bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f32738195b0f0f6c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.536+0000 7f50e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.536+0000 7f50e0ff9640 1 -- 172.21.15.33:0/2182616846 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e8020070 con 0x55e12d90c7b0 2024-09-16T00:40:43.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.536+0000 7f50e0ff9640 1 -- 172.21.15.33:0/2182616846 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f50e809a090 con 0x55e12d90c7b0 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.536+0000 7f5101e27740 1 -- 172.21.15.33:0/2182616846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50e80a0140 con 0x55e12d90c7b0 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.536+0000 7f50e0ff9640 1 -- 172.21.15.33:0/2182616846 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f50e80a4a00 con 0x55e12d90c7b0 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.537+0000 7f5101e27740 10 client.28931 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.537+0000 7f5101e27740 1 -- 172.21.15.33:0/2182616846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e12d9919e0 con 0x55e12d90c7b0 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.537+0000 7f50e0ff9640 1 -- 172.21.15.33:0/2182616846 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50e8015080 con 0x55e12d90c7b0 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.537+0000 7f50e0ff9640 1 -- 172.21.15.33:0/2182616846 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f50e800fbd0 con 0x55e12d90c7b0 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.537+0000 7f5101e27740 10 client.28931 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.537+0000 7f5101e27740 10 client.28931 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.537+0000 7f5101e27740 20 client.28931 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.537+0000 7f5101e27740 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e12d9270c0 0x55e12d994090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.537+0000 7f5101e27740 4 client.28931 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.537+0000 7f5101e27740 1 -- 172.21.15.33:0/2182616846 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e12d879d70 con 0x55e12d9270c0 2024-09-16T00:40:43.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.537+0000 7f50fcfe9640 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e12d9270c0 0x55e12d994090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f50fcfe9640 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e12d9270c0 0x55e12d994090 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c15c306e117a58b server_cookie=928bde726f454676 in_seq=0 out_seq=0 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f50e0ff9640 10 client.28931 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f50e0ff9640 1 -- 172.21.15.33:0/2182616846 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e12d879d70 con 0x55e12d9270c0 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f50e0ff9640 10 client.28931 handle_command_reply: tid=0 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f5101e27740 1 client.28931 shutdown 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f5101e27740 1 -- 172.21.15.33:0/2182616846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 msgr2=0x55e12d91fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f5101e27740 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 0x55e12d91fa20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f50e809b7c0 tx=0x7f50e809bc20 comp rx=0 tx=0).stop 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f5101e27740 1 -- 172.21.15.33:0/2182616846 shutdown_connections 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f5101e27740 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e12d9270c0 0x55e12d994090 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f5101e27740 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e12d917550 0x55e12d910de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f50e0ff9640 0 client.28931 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f5101e27740 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12d90dd20 0x55e12d91ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f5101e27740 1 --2- 172.21.15.33:0/2182616846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e12d90c7b0 0x55e12d91fa20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f5101e27740 1 -- 172.21.15.33:0/2182616846 >> 172.21.15.33:0/2182616846 conn(0x55e12d8d5310 msgr2=0x55e12d914180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:43.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f5101e27740 1 -- 172.21.15.33:0/2182616846 shutdown_connections 2024-09-16T00:40:43.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f5101e27740 1 -- 172.21.15.33:0/2182616846 wait complete. 2024-09-16T00:40:43.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.538+0000 7f5101e27740 20 client.28931 trim_cache size 0 max 16384 2024-09-16T00:40:43.544 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:40:43.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.539+0000 7f50fe7ec640 1 -- 172.21.15.33:0/3999964273 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f50e40a3340 msgr2=0x7f50e40a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:43.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.539+0000 7f50fe7ec640 1 --2- 172.21.15.33:0/3999964273 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f50e40a3340 0x7f50e40a5800 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f50f400ba20 tx=0x7f50f4004730 comp rx=0 tx=0).stop 2024-09-16T00:40:43.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.539+0000 7f50fe7ec640 1 -- 172.21.15.33:0/3999964273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50f806ea30 msgr2=0x7f50f81a7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:43.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.539+0000 7f50fe7ec640 1 --2- 172.21.15.33:0/3999964273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50f806ea30 0x7f50f81a7120 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f50e800d950 tx=0x7f50e800de10 comp rx=0 tx=0).stop 2024-09-16T00:40:43.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.539+0000 7f50fe7ec640 1 -- 172.21.15.33:0/3999964273 shutdown_connections 2024-09-16T00:40:43.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.539+0000 7f50fe7ec640 1 --2- 172.21.15.33:0/3999964273 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f50e40a3340 0x7f50e40a5800 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.539+0000 7f50fe7ec640 1 --2- 172.21.15.33:0/3999964273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50f811b140 0x7f50f81ab9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.539+0000 7f50fe7ec640 1 --2- 172.21.15.33:0/3999964273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f8110190 0x7f50f81a7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.539+0000 7f50fe7ec640 1 --2- 172.21.15.33:0/3999964273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50f806ea30 0x7f50f81a7120 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:43.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.539+0000 7f50fe7ec640 1 -- 172.21.15.33:0/3999964273 >> 172.21.15.33:0/3999964273 conn(0x7f50f8073180 msgr2=0x7f50f8119370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:43.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.539+0000 7f50fe7ec640 1 -- 172.21.15.33:0/3999964273 shutdown_connections 2024-09-16T00:40:43.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:43.539+0000 7f50fe7ec640 1 -- 172.21.15.33:0/3999964273 wait complete. 2024-09-16T00:40:43.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:43 smithi033 ceph-mon[31243]: pgmap v1057: 129 pgs: 129 active+clean; 730 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 2.9 MiB/s wr, 377 op/s 2024-09-16T00:40:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:43 smithi083 ceph-mon[35242]: pgmap v1057: 129 pgs: 129 active+clean; 730 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 2.9 MiB/s wr, 377 op/s 2024-09-16T00:40:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:43 smithi060 ceph-mon[35185]: pgmap v1057: 129 pgs: 129 active+clean; 730 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 2.9 MiB/s wr, 377 op/s 2024-09-16T00:40:44.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 204290 18.39 MB/sec execute 470 sec latency 63.231 ms 2024-09-16T00:40:45.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 204785 18.40 MB/sec execute 471 sec latency 229.654 ms 2024-09-16T00:40:45.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:45 smithi083 ceph-mon[35242]: pgmap v1058: 129 pgs: 129 active+clean; 716 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.3 MiB/s wr, 424 op/s 2024-09-16T00:40:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:45 smithi060 ceph-mon[35185]: pgmap v1058: 129 pgs: 129 active+clean; 716 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.3 MiB/s wr, 424 op/s 2024-09-16T00:40:46.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:45 smithi033 ceph-mon[31243]: pgmap v1058: 129 pgs: 129 active+clean; 716 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.3 MiB/s wr, 424 op/s 2024-09-16T00:40:46.101 INFO:tasks.workunit.client.0.smithi033.stdout: 10 204804 18.36 MB/sec execute 472 sec latency 1169.861 ms 2024-09-16T00:40:46.166 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:40:46.166 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:40:46.710 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:40:47.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 204846 18.33 MB/sec execute 473 sec latency 1425.084 ms 2024-09-16T00:40:47.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.321+0000 7efd4e351640 1 -- 172.21.15.33:0/881098714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd48102790 msgr2=0x7efd481067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:47.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.321+0000 7efd4e351640 1 --2- 172.21.15.33:0/881098714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd48102790 0x7efd481067e0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7efd38009900 tx=0x7efd3802a400 comp rx=0 tx=0).stop 2024-09-16T00:40:47.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.321+0000 7efd4e351640 1 -- 172.21.15.33:0/881098714 shutdown_connections 2024-09-16T00:40:47.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.321+0000 7efd4e351640 1 --2- 172.21.15.33:0/881098714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd48106f10 0x7efd4810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.321+0000 7efd4e351640 1 --2- 172.21.15.33:0/881098714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd48102790 0x7efd481067e0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.321+0000 7efd4e351640 1 --2- 172.21.15.33:0/881098714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd48101de0 0x7efd481021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.321+0000 7efd4e351640 1 -- 172.21.15.33:0/881098714 >> 172.21.15.33:0/881098714 conn(0x7efd480fd880 msgr2=0x7efd480ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:47.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.321+0000 7efd4e351640 1 -- 172.21.15.33:0/881098714 shutdown_connections 2024-09-16T00:40:47.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.325+0000 7efd4e351640 1 -- 172.21.15.33:0/881098714 wait complete. 2024-09-16T00:40:47.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.326+0000 7efd4e351640 1 Processor -- start 2024-09-16T00:40:47.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.326+0000 7efd4e351640 1 -- start start 2024-09-16T00:40:47.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.326+0000 7efd4e351640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd48101de0 0x7efd48117350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.326+0000 7efd4e351640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd48102790 0x7efd48117890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.326+0000 7efd4e351640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd48106f10 0x7efd481115e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.326+0000 7efd4e351640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd4810fdd0 con 0x7efd48106f10 2024-09-16T00:40:47.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.326+0000 7efd4e351640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efd4810fc50 con 0x7efd48102790 2024-09-16T00:40:47.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.326+0000 7efd4e351640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efd4810ff50 con 0x7efd48101de0 2024-09-16T00:40:47.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.327+0000 7efd4c8ee640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd48106f10 0x7efd481115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.327+0000 7efd477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd48102790 0x7efd48117890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.327+0000 7efd4c8ee640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd48106f10 0x7efd481115e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47474/0 (socket says 172.21.15.33:47474) 2024-09-16T00:40:47.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.327+0000 7efd477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd48102790 0x7efd48117890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42386/0 (socket says 172.21.15.33:42386) 2024-09-16T00:40:47.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.327+0000 7efd4c8ee640 1 -- 172.21.15.33:0/3325575066 learned_addr learned my addr 172.21.15.33:0/3325575066 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:47.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.327+0000 7efd4c8ee640 1 -- 172.21.15.33:0/3325575066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd48101de0 msgr2=0x7efd48117350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:47.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.327+0000 7efd4c8ee640 1 --2- 172.21.15.33:0/3325575066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd48101de0 0x7efd48117350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.327+0000 7efd4c8ee640 1 -- 172.21.15.33:0/3325575066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd48102790 msgr2=0x7efd48117890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:47.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.327+0000 7efd4c8ee640 1 --2- 172.21.15.33:0/3325575066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd48102790 0x7efd48117890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.327+0000 7efd4c8ee640 1 -- 172.21.15.33:0/3325575066 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd48111ea0 con 0x7efd48106f10 2024-09-16T00:40:47.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.328+0000 7efd4c8ee640 1 --2- 172.21.15.33:0/3325575066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd48106f10 0x7efd481115e0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00d750 tx=0x7efd3c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:47.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.329+0000 7efd457fa640 1 -- 172.21.15.33:0/3325575066 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c017070 con 0x7efd48106f10 2024-09-16T00:40:47.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.329+0000 7efd4e351640 1 -- 172.21.15.33:0/3325575066 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd48112190 con 0x7efd48106f10 2024-09-16T00:40:47.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.329+0000 7efd4e351640 1 -- 172.21.15.33:0/3325575066 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd48071ec0 con 0x7efd48106f10 2024-09-16T00:40:47.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.330+0000 7efd457fa640 1 -- 172.21.15.33:0/3325575066 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd3c005500 con 0x7efd48106f10 2024-09-16T00:40:47.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.330+0000 7efd457fa640 1 -- 172.21.15.33:0/3325575066 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c004cd0 con 0x7efd48106f10 2024-09-16T00:40:47.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.333+0000 7efd5072a740 1 Processor -- start 2024-09-16T00:40:47.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.333+0000 7efd457fa640 1 -- 172.21.15.33:0/3325575066 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7efd3c010460 con 0x7efd48106f10 2024-09-16T00:40:47.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.334+0000 7efd5072a740 1 -- start start 2024-09-16T00:40:47.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.334+0000 7efd5072a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50506fb60 0x55c50506ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.334+0000 7efd5072a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c505070480 0x55c50507a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.334+0000 7efd5072a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50507ad40 0x55c50507d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.334+0000 7efd5072a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c504fd5280 con 0x55c50507ad40 2024-09-16T00:40:47.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.334+0000 7efd5072a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c504f7cb10 con 0x55c505070480 2024-09-16T00:40:47.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.334+0000 7efd5072a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c504fd1240 con 0x55c50506fb60 2024-09-16T00:40:47.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.334+0000 7efd477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50507ad40 0x55c50507d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.334+0000 7efd477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50507ad40 0x55c50507d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47486/0 (socket says 172.21.15.33:47486) 2024-09-16T00:40:47.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.334+0000 7efd477fe640 1 -- 172.21.15.33:0/524567914 learned_addr learned my addr 172.21.15.33:0/524567914 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:47.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.335+0000 7efd477fe640 1 --2- 172.21.15.33:0/524567914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50506fb60 0x55c50506ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.335+0000 7efd47fff640 1 --2- 172.21.15.33:0/524567914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c505070480 0x55c50507a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.335+0000 7efd477fe640 1 -- 172.21.15.33:0/524567914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c505070480 msgr2=0x55c50507a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.335+0000 7efd477fe640 1 --2- 172.21.15.33:0/524567914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c505070480 0x55c50507a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.335+0000 7efd477fe640 1 -- 172.21.15.33:0/524567914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50507ad40 msgr2=0x55c50507d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.335+0000 7efd477fe640 1 --2- 172.21.15.33:0/524567914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50507ad40 0x55c50507d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.335+0000 7efd477fe640 1 -- 172.21.15.33:0/524567914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c504f891f0 con 0x55c50506fb60 2024-09-16T00:40:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.336+0000 7efd457fa640 1 --2- 172.21.15.33:0/3325575066 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efd300a3300 0x7efd300a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.336+0000 7efd477fe640 1 --2- 172.21.15.33:0/524567914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50506fb60 0x55c50506ff40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55c50506d470 tx=0x7efd38005a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d03f50c518a58e0b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.336+0000 7efd47fff640 1 --2- 172.21.15.33:0/3325575066 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efd300a3300 0x7efd300a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.336+0000 7efd457fa640 1 -- 172.21.15.33:0/3325575066 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7efd3c0cf5d0 con 0x7efd48106f10 2024-09-16T00:40:47.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.336+0000 7efd2e7fc640 1 -- 172.21.15.33:0/524567914 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38035070 con 0x55c50506fb60 2024-09-16T00:40:47.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.336+0000 7efd2e7fc640 1 -- 172.21.15.33:0/524567914 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd3803f070 con 0x55c50506fb60 2024-09-16T00:40:47.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.336+0000 7efd2e7fc640 1 -- 172.21.15.33:0/524567914 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd380035f0 con 0x55c50506fb60 2024-09-16T00:40:47.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.337+0000 7efd5072a740 1 -- 172.21.15.33:0/524567914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50506fb60 msgr2=0x55c50506ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:47.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.337+0000 7efd5072a740 1 --2- 172.21.15.33:0/524567914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50506fb60 0x55c50506ff40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55c50506d470 tx=0x7efd38005a40 comp rx=0 tx=0).stop 2024-09-16T00:40:47.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.340+0000 7efd5072a740 1 -- 172.21.15.33:0/524567914 shutdown_connections 2024-09-16T00:40:47.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.340+0000 7efd5072a740 1 --2- 172.21.15.33:0/524567914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50507ad40 0x55c50507d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.340+0000 7efd5072a740 1 --2- 172.21.15.33:0/524567914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c505070480 0x55c50507a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.340+0000 7efd5072a740 1 --2- 172.21.15.33:0/524567914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50506fb60 0x55c50506ff40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.340+0000 7efd5072a740 1 -- 172.21.15.33:0/524567914 >> 172.21.15.33:0/524567914 conn(0x55c50506c0f0 msgr2=0x55c50506c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:47.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.341+0000 7efd47fff640 1 --2- 172.21.15.33:0/3325575066 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efd300a3300 0x7efd300a57c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x55c50506cf50 tx=0x7efd4000d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:47.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.344+0000 7efd5072a740 1 -- 172.21.15.33:0/524567914 shutdown_connections 2024-09-16T00:40:47.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.344+0000 7efd5072a740 1 -- 172.21.15.33:0/524567914 wait complete. 2024-09-16T00:40:47.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.344+0000 7efd5072a740 1 -- start start 2024-09-16T00:40:47.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.344+0000 7efd5072a740 1 Processor -- start 2024-09-16T00:40:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.347+0000 7efd5072a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50506fb60 0x55c505077c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.347+0000 7efd5072a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c505070480 0x55c505078160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.347+0000 7efd5072a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50507ad40 0x55c5050786a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.347+0000 7efd5072a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c50507d910 con 0x55c50507ad40 2024-09-16T00:40:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.347+0000 7efd5072a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c504f8ff10 con 0x55c505070480 2024-09-16T00:40:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.347+0000 7efd5072a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c50507da90 con 0x55c50506fb60 2024-09-16T00:40:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.347+0000 7efd5072a740 10 client.0 _ll_register_callbacks cb 0x55c504f58a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.347+0000 7efd477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c505070480 0x55c505078160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.347+0000 7efd477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c505070480 0x55c505078160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42408/0 (socket says 172.21.15.33:42408) 2024-09-16T00:40:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.347+0000 7efd477fe640 1 -- 172.21.15.33:0/2346495195 learned_addr learned my addr 172.21.15.33:0/2346495195 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.348+0000 7efd477fe640 1 -- 172.21.15.33:0/2346495195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50506fb60 msgr2=0x55c505077c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:40:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.348+0000 7efd477fe640 1 --2- 172.21.15.33:0/2346495195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50506fb60 0x55c505077c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.348+0000 7efd477fe640 1 -- 172.21.15.33:0/2346495195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50507ad40 msgr2=0x55c5050786a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:47.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.348+0000 7efd477fe640 1 --2- 172.21.15.33:0/2346495195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50507ad40 0x55c5050786a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.348+0000 7efd477fe640 1 -- 172.21.15.33:0/2346495195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c504d466a0 con 0x55c505070480 2024-09-16T00:40:47.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.349+0000 7efd477fe640 1 --2- 172.21.15.33:0/2346495195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c505070480 0x55c505078160 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efd3803fbb0 tx=0x7efd3802a400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=469822634193e8a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:47.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.349+0000 7efd277fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:40:47.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.349+0000 7efd277fe640 1 -- 172.21.15.33:0/2346495195 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38035040 con 0x55c505070480 2024-09-16T00:40:47.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.349+0000 7efd5072a740 1 -- 172.21.15.33:0/2346495195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd3803f070 con 0x55c505070480 2024-09-16T00:40:47.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.349+0000 7efd277fe640 1 -- 172.21.15.33:0/2346495195 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7efd38004240 con 0x55c505070480 2024-09-16T00:40:47.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.350+0000 7efd277fe640 1 -- 172.21.15.33:0/2346495195 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7efd3803f430 con 0x55c505070480 2024-09-16T00:40:47.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.351+0000 7efd5072a740 10 client.28943 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:47.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.351+0000 7efd5072a740 1 -- 172.21.15.33:0/2346495195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c505089580 con 0x55c505070480 2024-09-16T00:40:47.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.353+0000 7efd277fe640 1 -- 172.21.15.33:0/2346495195 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd38004440 con 0x55c505070480 2024-09-16T00:40:47.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.353+0000 7efd277fe640 1 -- 172.21.15.33:0/2346495195 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7efd3803a040 con 0x55c505070480 2024-09-16T00:40:47.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.354+0000 7efd5072a740 10 client.28943 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:47.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.354+0000 7efd5072a740 10 client.28943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:40:47.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.354+0000 7efd5072a740 20 client.28943 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:47.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.354+0000 7efd5072a740 1 --2- 172.21.15.33:0/2346495195 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c50508a8d0 0x55c50508ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.354+0000 7efd5072a740 4 client.28943 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:40:47.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.354+0000 7efd5072a740 1 -- 172.21.15.33:0/2346495195 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7efd3803f070 con 0x55c50508a8d0 2024-09-16T00:40:47.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.357+0000 7efd47fff640 1 --2- 172.21.15.33:0/2346495195 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c50508a8d0 0x55c50508ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.364+0000 7efd47fff640 1 --2- 172.21.15.33:0/2346495195 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c50508a8d0 0x55c50508ccc0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fb0c8a11fc50558 server_cookie=671c0ab8a79934d7 in_seq=0 out_seq=0 2024-09-16T00:40:47.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.364+0000 7efd277fe640 10 client.28943 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:47.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.367+0000 7efd277fe640 1 -- 172.21.15.33:0/2346495195 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7efd3803f070 con 0x55c50508a8d0 2024-09-16T00:40:47.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.367+0000 7efd277fe640 10 client.28943 handle_command_reply: tid=0 2024-09-16T00:40:47.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.368+0000 7efd5072a740 1 client.28943 shutdown 2024-09-16T00:40:47.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.368+0000 7efd5072a740 1 -- 172.21.15.33:0/2346495195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c505070480 msgr2=0x55c505078160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:47.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.368+0000 7efd5072a740 1 --2- 172.21.15.33:0/2346495195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c505070480 0x55c505078160 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efd3803fbb0 tx=0x7efd3802a400 comp rx=0 tx=0).stop 2024-09-16T00:40:47.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.370+0000 7efd5072a740 1 -- 172.21.15.33:0/2346495195 shutdown_connections 2024-09-16T00:40:47.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.370+0000 7efd5072a740 1 --2- 172.21.15.33:0/2346495195 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c50508a8d0 0x55c50508ccc0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.370+0000 7efd5072a740 1 --2- 172.21.15.33:0/2346495195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50507ad40 0x55c5050786a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.370+0000 7efd5072a740 1 --2- 172.21.15.33:0/2346495195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c505070480 0x55c505078160 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.370+0000 7efd5072a740 1 --2- 172.21.15.33:0/2346495195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50506fb60 0x55c505077c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.370+0000 7efd5072a740 1 -- 172.21.15.33:0/2346495195 >> 172.21.15.33:0/2346495195 conn(0x55c50506c0f0 msgr2=0x55c50506b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:47.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.371+0000 7efd277fe640 0 client.28943 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:47.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.377+0000 7efd5072a740 1 -- 172.21.15.33:0/2346495195 shutdown_connections 2024-09-16T00:40:47.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.380+0000 7efd5072a740 1 -- 172.21.15.33:0/2346495195 wait complete. 2024-09-16T00:40:47.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.381+0000 7efd5072a740 20 client.28943 trim_cache size 0 max 16384 2024-09-16T00:40:47.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.414+0000 7efd5072a740 1 Processor -- start 2024-09-16T00:40:47.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.416+0000 7efd5072a740 1 -- start start 2024-09-16T00:40:47.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.417+0000 7efd5072a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c50506fb60 0x55c505076c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.417+0000 7efd5072a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c505070480 0x55c5050922f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.417+0000 7efd5072a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50507ad40 0x55c505092830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.417+0000 7efd5072a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd34012970 con 0x55c505070480 2024-09-16T00:40:47.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.417+0000 7efd5072a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efd34010240 con 0x55c50506fb60 2024-09-16T00:40:47.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.417+0000 7efd5072a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efd3400e090 con 0x55c50507ad40 2024-09-16T00:40:47.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.418+0000 7efd477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c505070480 0x55c5050922f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.418+0000 7efd477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c505070480 0x55c5050922f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47510/0 (socket says 172.21.15.33:47510) 2024-09-16T00:40:47.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.418+0000 7efd477fe640 1 -- 172.21.15.33:0/1197941681 learned_addr learned my addr 172.21.15.33:0/1197941681 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:47.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.418+0000 7efd47fff640 1 --2- 172.21.15.33:0/1197941681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c50506fb60 0x55c505076c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.418+0000 7efd477fe640 1 -- 172.21.15.33:0/1197941681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50507ad40 msgr2=0x55c505092830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:40:47.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.418+0000 7efd477fe640 1 --2- 172.21.15.33:0/1197941681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50507ad40 0x55c505092830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.418+0000 7efd477fe640 1 -- 172.21.15.33:0/1197941681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c50506fb60 msgr2=0x55c505076c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:47.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.418+0000 7efd477fe640 1 --2- 172.21.15.33:0/1197941681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c50506fb60 0x55c505076c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.418+0000 7efd477fe640 1 -- 172.21.15.33:0/1197941681 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c505076170 con 0x55c505070480 2024-09-16T00:40:47.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.420+0000 7efd477fe640 1 --2- 172.21.15.33:0/1197941681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c505070480 0x55c5050922f0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7efd3803d900 tx=0x7efd38009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf7f0320998ba46 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:47.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.420+0000 7efd2dffb640 1 -- 172.21.15.33:0/1197941681 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38035070 con 0x55c505070480 2024-09-16T00:40:47.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.420+0000 7efd2dffb640 1 -- 172.21.15.33:0/1197941681 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd3803d980 con 0x55c505070480 2024-09-16T00:40:47.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.421+0000 7efd2dffb640 1 -- 172.21.15.33:0/1197941681 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd380052d0 con 0x55c505070480 2024-09-16T00:40:47.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.422+0000 7efd5072a740 1 -- 172.21.15.33:0/1197941681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c505070480 msgr2=0x55c5050922f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:47.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.422+0000 7efd5072a740 1 --2- 172.21.15.33:0/1197941681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c505070480 0x55c5050922f0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7efd3803d900 tx=0x7efd38009660 comp rx=0 tx=0).stop 2024-09-16T00:40:47.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.422+0000 7efd5072a740 1 -- 172.21.15.33:0/1197941681 shutdown_connections 2024-09-16T00:40:47.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.422+0000 7efd5072a740 1 --2- 172.21.15.33:0/1197941681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50507ad40 0x55c505092830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.422+0000 7efd5072a740 1 --2- 172.21.15.33:0/1197941681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c505070480 0x55c5050922f0 secure :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7efd3803d900 tx=0x7efd38009660 comp rx=0 tx=0).stop 2024-09-16T00:40:47.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.422+0000 7efd5072a740 1 --2- 172.21.15.33:0/1197941681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c50506fb60 0x55c505076c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.422+0000 7efd5072a740 1 -- 172.21.15.33:0/1197941681 >> 172.21.15.33:0/1197941681 conn(0x55c50506c0f0 msgr2=0x55c5050764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:47.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.422+0000 7efd5072a740 1 -- 172.21.15.33:0/1197941681 shutdown_connections 2024-09-16T00:40:47.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.422+0000 7efd5072a740 1 -- 172.21.15.33:0/1197941681 wait complete. 2024-09-16T00:40:47.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.422+0000 7efd5072a740 1 -- start start 2024-09-16T00:40:47.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.423+0000 7efd5072a740 1 Processor -- start 2024-09-16T00:40:47.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.425+0000 7efd5072a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50506fb60 0x55c505083610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.425+0000 7efd5072a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50507ad40 0x55c50507f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.425+0000 7efd5072a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c50507ffa0 0x55c5050f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.425+0000 7efd5072a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd3400e700 con 0x55c50506fb60 2024-09-16T00:40:47.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.425+0000 7efd5072a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efd3400fb70 con 0x55c50507ffa0 2024-09-16T00:40:47.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.425+0000 7efd5072a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efd3400edd0 con 0x55c50507ad40 2024-09-16T00:40:47.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.425+0000 7efd5072a740 10 client.0 _ll_register_callbacks cb 0x7efd380315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:47.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.426+0000 7efd4c8ee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c50507ffa0 0x55c5050f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.426+0000 7efd4c8ee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c50507ffa0 0x55c5050f52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:42432/0 (socket says 172.21.15.33:42432) 2024-09-16T00:40:47.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.426+0000 7efd4c8ee640 1 -- 172.21.15.33:0/187388122 learned_addr learned my addr 172.21.15.33:0/187388122 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:47.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.426+0000 7efd477fe640 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50507ad40 0x55c50507f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.426+0000 7efd47fff640 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50506fb60 0x55c505083610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.426+0000 7efd477fe640 1 -- 172.21.15.33:0/187388122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c50507ffa0 msgr2=0x55c5050f52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:47.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.426+0000 7efd477fe640 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c50507ffa0 0x55c5050f52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.426+0000 7efd477fe640 1 -- 172.21.15.33:0/187388122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50506fb60 msgr2=0x55c505083610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:47.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.426+0000 7efd477fe640 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50506fb60 0x55c505083610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.426+0000 7efd477fe640 1 -- 172.21.15.33:0/187388122 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5050f5810 con 0x55c50507ad40 2024-09-16T00:40:47.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.427+0000 7efd477fe640 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50507ad40 0x55c50507f8b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55c505078d10 tx=0x7efd38005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b02a0de771483d4f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:47.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.427+0000 7efd277fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:40:47.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.427+0000 7efd277fe640 1 -- 172.21.15.33:0/187388122 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38035070 con 0x55c50507ad40 2024-09-16T00:40:47.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.427+0000 7efd5072a740 1 -- 172.21.15.33:0/187388122 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd3803d980 con 0x55c50507ad40 2024-09-16T00:40:47.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.427+0000 7efd277fe640 1 -- 172.21.15.33:0/187388122 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7efd38005220 con 0x55c50507ad40 2024-09-16T00:40:47.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.428+0000 7efd277fe640 1 -- 172.21.15.33:0/187388122 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7efd3802aa80 con 0x55c50507ad40 2024-09-16T00:40:47.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.428+0000 7efd5072a740 10 client.28951 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:47.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.428+0000 7efd5072a740 1 -- 172.21.15.33:0/187388122 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c5050f6860 con 0x55c50507ad40 2024-09-16T00:40:47.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.429+0000 7efd277fe640 1 -- 172.21.15.33:0/187388122 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd3803d980 con 0x55c50507ad40 2024-09-16T00:40:47.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.429+0000 7efd277fe640 1 -- 172.21.15.33:0/187388122 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7efd3805f630 con 0x55c50507ad40 2024-09-16T00:40:47.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.432+0000 7efd5072a740 10 client.28951 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:47.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.432+0000 7efd5072a740 10 client.28951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:40:47.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.432+0000 7efd5072a740 20 client.28951 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:47.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.432+0000 7efd5072a740 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c5050f6c40 0x55c5050f9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:47.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.432+0000 7efd5072a740 4 client.28951 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:40:47.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.432+0000 7efd5072a740 1 -- 172.21.15.33:0/187388122 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c504fe27f0 con 0x55c5050f6c40 2024-09-16T00:40:47.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.432+0000 7efd47fff640 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c5050f6c40 0x55c5050f9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:47.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.436+0000 7efd47fff640 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c5050f6c40 0x55c5050f9030 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7715e9f2a30e79fe server_cookie=9790b9dedef53e79 in_seq=0 out_seq=0 2024-09-16T00:40:47.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.436+0000 7efd277fe640 10 client.28951 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:47.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.436+0000 7efd277fe640 1 -- 172.21.15.33:0/187388122 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c504fe27f0 con 0x55c5050f6c40 2024-09-16T00:40:47.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.436+0000 7efd277fe640 10 client.28951 handle_command_reply: tid=0 2024-09-16T00:40:47.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.437+0000 7efd5072a740 1 client.28951 shutdown 2024-09-16T00:40:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.442+0000 7efd5072a740 1 -- 172.21.15.33:0/187388122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50507ad40 msgr2=0x55c50507f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.442+0000 7efd5072a740 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50507ad40 0x55c50507f8b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55c505078d10 tx=0x7efd38005b50 comp rx=0 tx=0).stop 2024-09-16T00:40:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.443+0000 7efd5072a740 1 -- 172.21.15.33:0/187388122 shutdown_connections 2024-09-16T00:40:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.443+0000 7efd5072a740 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55c5050f6c40 0x55c5050f9030 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.443+0000 7efd5072a740 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c50507ffa0 0x55c5050f52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.443+0000 7efd5072a740 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c50507ad40 0x55c50507f8b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.443+0000 7efd5072a740 1 --2- 172.21.15.33:0/187388122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c50506fb60 0x55c505083610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.443+0000 7efd5072a740 1 -- 172.21.15.33:0/187388122 >> 172.21.15.33:0/187388122 conn(0x55c50506c0f0 msgr2=0x55c505076d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.443+0000 7efd277fe640 0 client.28951 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:47.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.444+0000 7efd5072a740 1 -- 172.21.15.33:0/187388122 shutdown_connections 2024-09-16T00:40:47.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.445+0000 7efd5072a740 1 -- 172.21.15.33:0/187388122 wait complete. 2024-09-16T00:40:47.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.445+0000 7efd5072a740 20 client.28951 trim_cache size 0 max 16384 2024-09-16T00:40:47.446 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:40:47.446 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:40:47.446 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "91abd9e8-d263-4a11-ae27-19e597a813cd", 2024-09-16T00:40:47.446 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:40:47.447 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:40:47.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.446+0000 7efd4e351640 1 -- 172.21.15.33:0/3325575066 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efd300a3300 msgr2=0x7efd300a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:47.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.446+0000 7efd4e351640 1 --2- 172.21.15.33:0/3325575066 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efd300a3300 0x7efd300a57c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x55c50506cf50 tx=0x7efd4000d040 comp rx=0 tx=0).stop 2024-09-16T00:40:47.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.446+0000 7efd4e351640 1 -- 172.21.15.33:0/3325575066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd48106f10 msgr2=0x7efd481115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:47.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.446+0000 7efd4e351640 1 --2- 172.21.15.33:0/3325575066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd48106f10 0x7efd481115e0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00d750 tx=0x7efd3c00dc10 comp rx=0 tx=0).stop 2024-09-16T00:40:47.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.447+0000 7efd4e351640 1 -- 172.21.15.33:0/3325575066 shutdown_connections 2024-09-16T00:40:47.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.447+0000 7efd4e351640 1 --2- 172.21.15.33:0/3325575066 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efd300a3300 0x7efd300a57c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.447+0000 7efd4e351640 1 --2- 172.21.15.33:0/3325575066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd48106f10 0x7efd481115e0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.447+0000 7efd4e351640 1 --2- 172.21.15.33:0/3325575066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd48102790 0x7efd48117890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.447+0000 7efd4e351640 1 --2- 172.21.15.33:0/3325575066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd48101de0 0x7efd48117350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:47.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.447+0000 7efd4e351640 1 -- 172.21.15.33:0/3325575066 >> 172.21.15.33:0/3325575066 conn(0x7efd480fd880 msgr2=0x7efd480febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:47.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.447+0000 7efd4e351640 1 -- 172.21.15.33:0/3325575066 shutdown_connections 2024-09-16T00:40:47.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:47.447+0000 7efd4e351640 1 -- 172.21.15.33:0/3325575066 wait complete. 2024-09-16T00:40:47.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:47 smithi033 ceph-mon[31243]: pgmap v1059: 129 pgs: 129 active+clean; 722 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 4.0 MiB/s wr, 354 op/s 2024-09-16T00:40:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:47 smithi083 ceph-mon[35242]: pgmap v1059: 129 pgs: 129 active+clean; 722 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 4.0 MiB/s wr, 354 op/s 2024-09-16T00:40:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:47 smithi060 ceph-mon[35185]: pgmap v1059: 129 pgs: 129 active+clean; 722 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 4.0 MiB/s wr, 354 op/s 2024-09-16T00:40:48.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 205085 18.33 MB/sec execute 474 sec latency 272.027 ms 2024-09-16T00:40:48.219 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:40:48.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:40:48.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:48.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:48.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.653 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:48.654 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:40:48.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:40:48.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:48 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.975 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:40:48.976 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.976 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:48.976 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:48.976 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:48.976 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:48.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:48.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:48.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:48.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:48.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:48.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:40:48.979 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:48 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:40:48.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:48.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:48.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:48.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:48.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:48.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:48.982 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.983 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.983 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:48.983 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.983 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:48.983 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.983 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:48.983 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:48.984 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:40:48.984 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:48 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:49.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 205462 18.35 MB/sec execute 475 sec latency 149.233 ms 2024-09-16T00:40:49.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.273+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2591403062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa2811b140 msgr2=0x7ffa2811d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:49.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.273+0000 7ffa2f24d640 1 --2- 172.21.15.33:0/2591403062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa2811b140 0x7ffa2811d530 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c009900 tx=0x7ffa1c02a3f0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.274+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2591403062 shutdown_connections 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.274+0000 7ffa2f24d640 1 --2- 172.21.15.33:0/2591403062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa2811b140 0x7ffa2811d530 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.274+0000 7ffa2f24d640 1 --2- 172.21.15.33:0/2591403062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa28109830 0x7ffa28109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.274+0000 7ffa2f24d640 1 --2- 172.21.15.33:0/2591403062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa28109450 0x7ffa28110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.274+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2591403062 >> 172.21.15.33:0/2591403062 conn(0x7ffa2806e140 msgr2=0x7ffa2806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.274+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2591403062 shutdown_connections 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.274+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2591403062 wait complete. 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.275+0000 7ffa2f24d640 1 Processor -- start 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.275+0000 7ffa2f24d640 1 -- start start 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.275+0000 7ffa2f24d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa28109450 0x7ffa281a7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.275+0000 7ffa2f24d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa28109830 0x7ffa281a7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.275+0000 7ffa2f24d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa2811b140 0x7ffa281abb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.275+0000 7ffa2f24d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa28120dc0 con 0x7ffa2811b140 2024-09-16T00:40:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.276+0000 7ffa2f24d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffa28120c40 con 0x7ffa28109830 2024-09-16T00:40:49.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.276+0000 7ffa2f24d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffa28120f40 con 0x7ffa28109450 2024-09-16T00:40:49.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.276+0000 7ffa2d7ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa2811b140 0x7ffa281abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.276+0000 7ffa2d7ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa2811b140 0x7ffa281abb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47536/0 (socket says 172.21.15.33:47536) 2024-09-16T00:40:49.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.276+0000 7ffa2d7ea640 1 -- 172.21.15.33:0/2717910795 learned_addr learned my addr 172.21.15.33:0/2717910795 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:49.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.276+0000 7ffa2d7ea640 1 -- 172.21.15.33:0/2717910795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa28109450 msgr2=0x7ffa281a7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:49.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.276+0000 7ffa2d7ea640 1 --2- 172.21.15.33:0/2717910795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa28109450 0x7ffa281a7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.276+0000 7ffa2d7ea640 1 -- 172.21.15.33:0/2717910795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa28109830 msgr2=0x7ffa281a7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:49.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.276+0000 7ffa2d7ea640 1 --2- 172.21.15.33:0/2717910795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa28109830 0x7ffa281a7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.276+0000 7ffa2d7ea640 1 -- 172.21.15.33:0/2717910795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa281ac1f0 con 0x7ffa2811b140 2024-09-16T00:40:49.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.276+0000 7ffa2d7ea640 1 --2- 172.21.15.33:0/2717910795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa2811b140 0x7ffa281abb10 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c02a960 tx=0x7ffa1c0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:49.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.277+0000 7ffa25ffb640 1 -- 172.21.15.33:0/2717910795 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c03f030 con 0x7ffa2811b140 2024-09-16T00:40:49.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.277+0000 7ffa25ffb640 1 -- 172.21.15.33:0/2717910795 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa1c003ad0 con 0x7ffa2811b140 2024-09-16T00:40:49.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.277+0000 7ffa25ffb640 1 -- 172.21.15.33:0/2717910795 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c035070 con 0x7ffa2811b140 2024-09-16T00:40:49.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.277+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2717910795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa281ac480 con 0x7ffa2811b140 2024-09-16T00:40:49.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.278+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2717910795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa281ac910 con 0x7ffa2811b140 2024-09-16T00:40:49.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.281+0000 7ffa25ffb640 1 -- 172.21.15.33:0/2717910795 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ffa1c042030 con 0x7ffa2811b140 2024-09-16T00:40:49.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.282+0000 7ffa25ffb640 1 --2- 172.21.15.33:0/2717910795 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffa100a3300 0x7ffa100a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.282+0000 7ffa31626740 1 Processor -- start 2024-09-16T00:40:49.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.282+0000 7ffa25ffb640 1 -- 172.21.15.33:0/2717910795 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ffa1c0f0310 con 0x7ffa2811b140 2024-09-16T00:40:49.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.282+0000 7ffa2cfe9640 1 --2- 172.21.15.33:0/2717910795 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffa100a3300 0x7ffa100a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.283+0000 7ffa31626740 1 -- start start 2024-09-16T00:40:49.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.283+0000 7ffa31626740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea94a0 0x558164ea9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.283+0000 7ffa31626740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea9dc0 0x558164eb4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.283+0000 7ffa31626740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164eb4680 0x558164eb6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.283+0000 7ffa31626740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558164e0f2e0 con 0x558164ea9dc0 2024-09-16T00:40:49.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.283+0000 7ffa31626740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558164db6ce0 con 0x558164eb4680 2024-09-16T00:40:49.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.283+0000 7ffa31626740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558164e0b2a0 con 0x558164ea94a0 2024-09-16T00:40:49.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.283+0000 7ffa2cfe9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea94a0 0x558164ea9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.283+0000 7ffa2d7ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164eb4680 0x558164eb6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.283+0000 7ffa27fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea9dc0 0x558164eb4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.283+0000 7ffa27fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea9dc0 0x558164eb4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47546/0 (socket says 172.21.15.33:47546) 2024-09-16T00:40:49.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.283+0000 7ffa27fff640 1 -- 172.21.15.33:0/110791360 learned_addr learned my addr 172.21.15.33:0/110791360 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:49.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.284+0000 7ffa27fff640 1 -- 172.21.15.33:0/110791360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea94a0 msgr2=0x558164ea9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:49.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.284+0000 7ffa27fff640 1 --2- 172.21.15.33:0/110791360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea94a0 0x558164ea9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.284+0000 7ffa27fff640 1 -- 172.21.15.33:0/110791360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164eb4680 msgr2=0x558164eb6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:49.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.284+0000 7ffa27fff640 1 --2- 172.21.15.33:0/110791360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164eb4680 0x558164eb6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.284+0000 7ffa27fff640 1 -- 172.21.15.33:0/110791360 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558164d9b440 con 0x558164ea9dc0 2024-09-16T00:40:49.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.285+0000 7ffa27fff640 1 --2- 172.21.15.33:0/110791360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea9dc0 0x558164eb4140 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009d50 tx=0x7ffa2000b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e255cbbe5412d938 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:49.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.285+0000 7ffa2cfe9640 1 --2- 172.21.15.33:0/2717910795 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffa100a3300 0x7ffa100a57c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800a8f0 tx=0x7ffa18005bf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:49.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.285+0000 7ffa0effd640 1 -- 172.21.15.33:0/110791360 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2000bd40 con 0x558164ea9dc0 2024-09-16T00:40:49.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.285+0000 7ffa0effd640 1 -- 172.21.15.33:0/110791360 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa20017070 con 0x558164ea9dc0 2024-09-16T00:40:49.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.285+0000 7ffa0effd640 1 -- 172.21.15.33:0/110791360 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20011750 con 0x558164ea9dc0 2024-09-16T00:40:49.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.286+0000 7ffa31626740 1 -- 172.21.15.33:0/110791360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea9dc0 msgr2=0x558164eb4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:49.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.286+0000 7ffa31626740 1 --2- 172.21.15.33:0/110791360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea9dc0 0x558164eb4140 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009d50 tx=0x7ffa2000b420 comp rx=0 tx=0).stop 2024-09-16T00:40:49.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.286+0000 7ffa31626740 1 -- 172.21.15.33:0/110791360 shutdown_connections 2024-09-16T00:40:49.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.287+0000 7ffa31626740 1 --2- 172.21.15.33:0/110791360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164eb4680 0x558164eb6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.287+0000 7ffa31626740 1 --2- 172.21.15.33:0/110791360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea9dc0 0x558164eb4140 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.287+0000 7ffa31626740 1 --2- 172.21.15.33:0/110791360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea94a0 0x558164ea9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.287+0000 7ffa31626740 1 -- 172.21.15.33:0/110791360 >> 172.21.15.33:0/110791360 conn(0x558164ea5a30 msgr2=0x558164ea5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:49.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.287+0000 7ffa31626740 1 -- 172.21.15.33:0/110791360 shutdown_connections 2024-09-16T00:40:49.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.287+0000 7ffa31626740 1 -- 172.21.15.33:0/110791360 wait complete. 2024-09-16T00:40:49.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.288+0000 7ffa31626740 1 -- start start 2024-09-16T00:40:49.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.288+0000 7ffa31626740 1 Processor -- start 2024-09-16T00:40:49.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.289+0000 7ffa31626740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea94a0 0x558164eb40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.289+0000 7ffa31626740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 0x558164eb1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.289+0000 7ffa2cfe9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea94a0 0x558164eb40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.289+0000 7ffa31626740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164eb4680 0x558164eb1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.289+0000 7ffa2cfe9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea94a0 0x558164eb40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47562/0 (socket says 172.21.15.33:47562) 2024-09-16T00:40:49.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.290+0000 7ffa27fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 0x558164eb1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.290+0000 7ffa27fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 0x558164eb1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:34998/0 (socket says 172.21.15.33:34998) 2024-09-16T00:40:49.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.290+0000 7ffa2cfe9640 1 -- 172.21.15.33:0/3630860999 learned_addr learned my addr 172.21.15.33:0/3630860999 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:49.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.290+0000 7ffa31626740 1 -- 172.21.15.33:0/3630860999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558164eb7250 con 0x558164ea94a0 2024-09-16T00:40:49.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.290+0000 7ffa2d7ea640 1 --2- 172.21.15.33:0/3630860999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164eb4680 0x558164eb1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.290+0000 7ffa31626740 1 -- 172.21.15.33:0/3630860999 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558164dca050 con 0x558164eb4680 2024-09-16T00:40:49.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.290+0000 7ffa31626740 1 -- 172.21.15.33:0/3630860999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558164eb73d0 con 0x558164ea9dc0 2024-09-16T00:40:49.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.290+0000 7ffa31626740 10 client.0 _ll_register_callbacks cb 0x558164e0a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:49.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.291+0000 7ffa2d7ea640 1 -- 172.21.15.33:0/3630860999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 msgr2=0x558164eb1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:49.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.291+0000 7ffa2d7ea640 1 --2- 172.21.15.33:0/3630860999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 0x558164eb1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.291+0000 7ffa2d7ea640 1 -- 172.21.15.33:0/3630860999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea94a0 msgr2=0x558164eb40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:49.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.291+0000 7ffa2d7ea640 1 --2- 172.21.15.33:0/3630860999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea94a0 0x558164eb40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.291+0000 7ffa2d7ea640 1 -- 172.21.15.33:0/3630860999 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558164b803c0 con 0x558164eb4680 2024-09-16T00:40:49.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.291+0000 7ffa2d7ea640 1 --2- 172.21.15.33:0/3630860999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164eb4680 0x558164eb1960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0ed990 tx=0x7ffa1c031b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a0825e731c1ce5e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:49.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.291+0000 7ffa07fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:40:49.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.291+0000 7ffa07fff640 1 -- 172.21.15.33:0/3630860999 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c0d5060 con 0x558164eb4680 2024-09-16T00:40:49.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.291+0000 7ffa07fff640 1 -- 172.21.15.33:0/3630860999 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ffa1c003c80 con 0x558164eb4680 2024-09-16T00:40:49.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.292+0000 7ffa31626740 1 -- 172.21.15.33:0/3630860999 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa20017070 con 0x558164eb4680 2024-09-16T00:40:49.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.292+0000 7ffa07fff640 1 -- 172.21.15.33:0/3630860999 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ffa1c049040 con 0x558164eb4680 2024-09-16T00:40:49.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.292+0000 7ffa31626740 10 client.28955 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:49.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.292+0000 7ffa31626740 1 -- 172.21.15.33:0/3630860999 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558164ec2ce0 con 0x558164eb4680 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.292+0000 7ffa07fff640 1 -- 172.21.15.33:0/3630860999 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa1c0ede50 con 0x558164eb4680 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.293+0000 7ffa07fff640 1 -- 172.21.15.33:0/3630860999 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ffa1c0c4c90 con 0x558164eb4680 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.293+0000 7ffa31626740 10 client.28955 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.293+0000 7ffa31626740 10 client.28955 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.293+0000 7ffa31626740 20 client.28955 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.293+0000 7ffa31626740 1 --2- 172.21.15.33:0/3630860999 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558164ec40d0 0x558164ec6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.293+0000 7ffa31626740 4 client.28955 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.293+0000 7ffa31626740 1 -- 172.21.15.33:0/3630860999 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffa20017070 con 0x558164ec40d0 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.293+0000 7ffa2cfe9640 1 --2- 172.21.15.33:0/3630860999 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558164ec40d0 0x558164ec6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.294+0000 7ffa2cfe9640 1 --2- 172.21.15.33:0/3630860999 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558164ec40d0 0x558164ec6500 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d31e8625a57e329 server_cookie=7a0147faec3604f2 in_seq=0 out_seq=0 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.294+0000 7ffa07fff640 10 client.28955 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.295+0000 7ffa07fff640 1 -- 172.21.15.33:0/3630860999 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7ffa20017070 con 0x558164ec40d0 2024-09-16T00:40:49.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.295+0000 7ffa07fff640 10 client.28955 handle_command_reply: tid=0 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.295+0000 7ffa31626740 1 client.28955 shutdown 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.295+0000 7ffa31626740 1 -- 172.21.15.33:0/3630860999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164eb4680 msgr2=0x558164eb1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.295+0000 7ffa31626740 1 --2- 172.21.15.33:0/3630860999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164eb4680 0x558164eb1960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0ed990 tx=0x7ffa1c031b60 comp rx=0 tx=0).stop 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.295+0000 7ffa31626740 1 -- 172.21.15.33:0/3630860999 shutdown_connections 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.295+0000 7ffa31626740 1 --2- 172.21.15.33:0/3630860999 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558164ec40d0 0x558164ec6500 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.295+0000 7ffa07fff640 0 client.28955 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.295+0000 7ffa31626740 1 --2- 172.21.15.33:0/3630860999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164eb4680 0x558164eb1960 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.295+0000 7ffa31626740 1 --2- 172.21.15.33:0/3630860999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 0x558164eb1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.296+0000 7ffa31626740 1 --2- 172.21.15.33:0/3630860999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164ea94a0 0x558164eb40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.296+0000 7ffa31626740 1 -- 172.21.15.33:0/3630860999 >> 172.21.15.33:0/3630860999 conn(0x558164ea5a30 msgr2=0x558164ea4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.296+0000 7ffa31626740 1 -- 172.21.15.33:0/3630860999 shutdown_connections 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.296+0000 7ffa31626740 1 -- 172.21.15.33:0/3630860999 wait complete. 2024-09-16T00:40:49.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.296+0000 7ffa31626740 20 client.28955 trim_cache size 0 max 16384 2024-09-16T00:40:49.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.328+0000 7ffa31626740 1 Processor -- start 2024-09-16T00:40:49.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.330+0000 7ffa31626740 1 -- start start 2024-09-16T00:40:49.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.330+0000 7ffa31626740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea94a0 0x558164ec04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.330+0000 7ffa31626740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164ea9dc0 0x558164eac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.330+0000 7ffa31626740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 0x558164ecbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.330+0000 7ffa31626740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa281c08a0 con 0x558164eb4680 2024-09-16T00:40:49.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.330+0000 7ffa31626740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffa281be170 con 0x558164ea9dc0 2024-09-16T00:40:49.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.330+0000 7ffa31626740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffa281bbfc0 con 0x558164ea94a0 2024-09-16T00:40:49.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.331+0000 7ffa2d7ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 0x558164ecbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.331+0000 7ffa2d7ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 0x558164ecbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47568/0 (socket says 172.21.15.33:47568) 2024-09-16T00:40:49.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.331+0000 7ffa2d7ea640 1 -- 172.21.15.33:0/2088114164 learned_addr learned my addr 172.21.15.33:0/2088114164 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:49.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.332+0000 7ffa2d7ea640 1 -- 172.21.15.33:0/2088114164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea94a0 msgr2=0x558164ec04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:49.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.332+0000 7ffa2d7ea640 1 --2- 172.21.15.33:0/2088114164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea94a0 0x558164ec04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.332+0000 7ffa2d7ea640 1 -- 172.21.15.33:0/2088114164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164ea9dc0 msgr2=0x558164eac8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:40:49.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.332+0000 7ffa2d7ea640 1 --2- 172.21.15.33:0/2088114164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164ea9dc0 0x558164eac8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.332+0000 7ffa2d7ea640 1 -- 172.21.15.33:0/2088114164 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558164ea66c0 con 0x558164eb4680 2024-09-16T00:40:49.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.335+0000 7ffa2d7ea640 1 --2- 172.21.15.33:0/2088114164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 0x558164ecbdd0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0bc1b0 tx=0x7ffa1c009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d60ffb4e0e593e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:49.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.335+0000 7ffa0e7fc640 1 -- 172.21.15.33:0/2088114164 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c0bbff0 con 0x558164eb4680 2024-09-16T00:40:49.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.336+0000 7ffa0e7fc640 1 -- 172.21.15.33:0/2088114164 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa1c0ede50 con 0x558164eb4680 2024-09-16T00:40:49.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.336+0000 7ffa0e7fc640 1 -- 172.21.15.33:0/2088114164 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c0d5350 con 0x558164eb4680 2024-09-16T00:40:49.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.337+0000 7ffa31626740 1 -- 172.21.15.33:0/2088114164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 msgr2=0x558164ecbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:49.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.337+0000 7ffa31626740 1 --2- 172.21.15.33:0/2088114164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 0x558164ecbdd0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0bc1b0 tx=0x7ffa1c009830 comp rx=0 tx=0).stop 2024-09-16T00:40:49.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.337+0000 7ffa31626740 1 -- 172.21.15.33:0/2088114164 shutdown_connections 2024-09-16T00:40:49.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.337+0000 7ffa31626740 1 --2- 172.21.15.33:0/2088114164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 0x558164ecbdd0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.337+0000 7ffa31626740 1 --2- 172.21.15.33:0/2088114164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164ea9dc0 0x558164eac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.337+0000 7ffa31626740 1 --2- 172.21.15.33:0/2088114164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea94a0 0x558164ec04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.337+0000 7ffa31626740 1 -- 172.21.15.33:0/2088114164 >> 172.21.15.33:0/2088114164 conn(0x558164ea5a30 msgr2=0x558164eb3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:49.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.342+0000 7ffa31626740 1 -- 172.21.15.33:0/2088114164 shutdown_connections 2024-09-16T00:40:49.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.343+0000 7ffa31626740 1 -- 172.21.15.33:0/2088114164 wait complete. 2024-09-16T00:40:49.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.343+0000 7ffa31626740 1 -- start start 2024-09-16T00:40:49.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.343+0000 7ffa31626740 1 Processor -- start 2024-09-16T00:40:49.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.344+0000 7ffa31626740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164ea94a0 0x558164eb8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.344+0000 7ffa31626740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 0x558164eb9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.344+0000 7ffa31626740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 0x558164f2dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.344+0000 7ffa31626740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa281bc630 con 0x558164eb4680 2024-09-16T00:40:49.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.344+0000 7ffa31626740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffa281bdaa0 con 0x558164ea94a0 2024-09-16T00:40:49.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.344+0000 7ffa31626740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffa281bcd00 con 0x558164ea9dc0 2024-09-16T00:40:49.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.344+0000 7ffa31626740 10 client.0 _ll_register_callbacks cb 0x7ffa1c0bbe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:40:49.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.344+0000 7ffa27fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 0x558164eb9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.344+0000 7ffa27fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 0x558164eb9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:35012/0 (socket says 172.21.15.33:35012) 2024-09-16T00:40:49.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.344+0000 7ffa27fff640 1 -- 172.21.15.33:0/291199907 learned_addr learned my addr 172.21.15.33:0/291199907 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:40:49.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.345+0000 7ffa2d7ea640 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 0x558164f2dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.346+0000 7ffa2cfe9640 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164ea94a0 0x558164eb8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.347+0000 7ffa27fff640 1 -- 172.21.15.33:0/291199907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164ea94a0 msgr2=0x558164eb8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:49.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.347+0000 7ffa27fff640 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164ea94a0 0x558164eb8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.347+0000 7ffa27fff640 1 -- 172.21.15.33:0/291199907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 msgr2=0x558164f2dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:40:49.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.347+0000 7ffa27fff640 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 0x558164f2dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.347+0000 7ffa27fff640 1 -- 172.21.15.33:0/291199907 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558164f2e390 con 0x558164ea9dc0 2024-09-16T00:40:49.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.347+0000 7ffa2d7ea640 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 0x558164f2dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:40:49.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.348+0000 7ffa27fff640 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 0x558164eb9070 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009d50 tx=0x7ffa20012e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae2f600097d4c181 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:40:49.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.348+0000 7ffa07fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:40:49.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.348+0000 7ffa07fff640 1 -- 172.21.15.33:0/291199907 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20021070 con 0x558164ea9dc0 2024-09-16T00:40:49.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.348+0000 7ffa07fff640 1 -- 172.21.15.33:0/291199907 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ffa20025600 con 0x558164ea9dc0 2024-09-16T00:40:49.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.348+0000 7ffa31626740 1 -- 172.21.15.33:0/291199907 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa1c0ede50 con 0x558164ea9dc0 2024-09-16T00:40:49.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.350+0000 7ffa07fff640 1 -- 172.21.15.33:0/291199907 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ffa20025800 con 0x558164ea9dc0 2024-09-16T00:40:49.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.350+0000 7ffa31626740 10 client.28957 fetch_fsmap learned FSMap version 33 2024-09-16T00:40:49.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.350+0000 7ffa31626740 1 -- 172.21.15.33:0/291199907 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558164f2f690 con 0x558164ea9dc0 2024-09-16T00:40:49.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.351+0000 7ffa07fff640 1 -- 172.21.15.33:0/291199907 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa1c0ede50 con 0x558164ea9dc0 2024-09-16T00:40:49.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.351+0000 7ffa07fff640 1 -- 172.21.15.33:0/291199907 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ffa20011070 con 0x558164ea9dc0 2024-09-16T00:40:49.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.351+0000 7ffa31626740 10 client.28957 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:40:49.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.351+0000 7ffa31626740 10 client.28957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:40:49.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.351+0000 7ffa31626740 20 client.28957 populate_metadata read hostname 'smithi033' 2024-09-16T00:40:49.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.351+0000 7ffa31626740 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558164ec40d0 0x558164f31e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:40:49.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.351+0000 7ffa31626740 4 client.28957 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:40:49.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.351+0000 7ffa31626740 1 -- 172.21.15.33:0/291199907 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558164e1c850 con 0x558164ec40d0 2024-09-16T00:40:49.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.351+0000 7ffa2cfe9640 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558164ec40d0 0x558164f31e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:40:49.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.352+0000 7ffa07fff640 10 client.28957 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:49.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.352+0000 7ffa2cfe9640 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558164ec40d0 0x558164f31e10 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8177b0b7ac3e1f8 server_cookie=bda256dbe3627f4d in_seq=0 out_seq=0 2024-09-16T00:40:49.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.352+0000 7ffa07fff640 1 -- 172.21.15.33:0/291199907 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558164e1c850 con 0x558164ec40d0 2024-09-16T00:40:49.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.352+0000 7ffa07fff640 10 client.28957 handle_command_reply: tid=0 2024-09-16T00:40:49.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.352+0000 7ffa31626740 1 client.28957 shutdown 2024-09-16T00:40:49.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.353+0000 7ffa31626740 1 -- 172.21.15.33:0/291199907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 msgr2=0x558164eb9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:49.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.353+0000 7ffa31626740 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 0x558164eb9070 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009d50 tx=0x7ffa20012e00 comp rx=0 tx=0).stop 2024-09-16T00:40:49.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.353+0000 7ffa31626740 1 -- 172.21.15.33:0/291199907 shutdown_connections 2024-09-16T00:40:49.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.353+0000 7ffa31626740 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558164ec40d0 0x558164f31e10 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.353+0000 7ffa31626740 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558164eb4680 0x558164f2dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.353+0000 7ffa07fff640 0 client.28957 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:40:49.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.353+0000 7ffa31626740 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558164ea9dc0 0x558164eb9070 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.353+0000 7ffa31626740 1 --2- 172.21.15.33:0/291199907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558164ea94a0 0x558164eb8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.353+0000 7ffa31626740 1 -- 172.21.15.33:0/291199907 >> 172.21.15.33:0/291199907 conn(0x558164ea5a30 msgr2=0x558164ea4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:49.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.353+0000 7ffa31626740 1 -- 172.21.15.33:0/291199907 shutdown_connections 2024-09-16T00:40:49.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.353+0000 7ffa31626740 1 -- 172.21.15.33:0/291199907 wait complete. 2024-09-16T00:40:49.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.354+0000 7ffa31626740 20 client.28957 trim_cache size 0 max 16384 2024-09-16T00:40:49.355 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:40:49.355 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:40:49.355 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:40:49.355 INFO:teuthology.orchestra.run.smithi033.stdout: "91abd9e8-d263-4a11-ae27-19e597a813cd": { 2024-09-16T00:40:49.355 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:40:49.356 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "91abd9e8-d263-4a11-ae27-19e597a813cd", 2024-09-16T00:40:49.356 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:40:49.356 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:40:49.356 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:40:49.356 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:40:49.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.354+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2717910795 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffa100a3300 msgr2=0x7ffa100a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:49.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.354+0000 7ffa2f24d640 1 --2- 172.21.15.33:0/2717910795 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffa100a3300 0x7ffa100a57c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800a8f0 tx=0x7ffa18005bf0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.354+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2717910795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa2811b140 msgr2=0x7ffa281abb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:40:49.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.354+0000 7ffa2f24d640 1 --2- 172.21.15.33:0/2717910795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa2811b140 0x7ffa281abb10 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c02a960 tx=0x7ffa1c0317e0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.354+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2717910795 shutdown_connections 2024-09-16T00:40:49.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.354+0000 7ffa2f24d640 1 --2- 172.21.15.33:0/2717910795 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffa100a3300 0x7ffa100a57c0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.354+0000 7ffa2f24d640 1 --2- 172.21.15.33:0/2717910795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa2811b140 0x7ffa281abb10 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.355+0000 7ffa2f24d640 1 --2- 172.21.15.33:0/2717910795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa28109830 0x7ffa281a7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.355+0000 7ffa2f24d640 1 --2- 172.21.15.33:0/2717910795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa28109450 0x7ffa281a7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:40:49.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.355+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2717910795 >> 172.21.15.33:0/2717910795 conn(0x7ffa2806e140 msgr2=0x7ffa2811be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:40:49.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.355+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2717910795 shutdown_connections 2024-09-16T00:40:49.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:40:49.355+0000 7ffa2f24d640 1 -- 172.21.15.33:0/2717910795 wait complete. 2024-09-16T00:40:49.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:49 smithi083 ceph-mon[35242]: pgmap v1060: 129 pgs: 129 active+clean; 743 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.6 MiB/s wr, 344 op/s 2024-09-16T00:40:49.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:49 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:49.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:49 smithi060 ceph-mon[35185]: pgmap v1060: 129 pgs: 129 active+clean; 743 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.6 MiB/s wr, 344 op/s 2024-09-16T00:40:49.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:49 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:49.981 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:49 smithi033 ceph-mon[31243]: pgmap v1060: 129 pgs: 129 active+clean; 743 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.6 MiB/s wr, 344 op/s 2024-09-16T00:40:49.981 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:49 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:40:50.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 205815 18.35 MB/sec execute 476 sec latency 112.444 ms 2024-09-16T00:40:50.181 INFO:tasks.cephfs.filesystem:scrub status for tag:91abd9e8-d263-4a11-ae27-19e597a813cd - {'path': '/', 'tag': '91abd9e8-d263-4a11-ae27-19e597a813cd', 'options': 'recursive,force'} 2024-09-16T00:40:51.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 205908 18.32 MB/sec execute 477 sec latency 465.678 ms 2024-09-16T00:40:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:51 smithi083 ceph-mon[35242]: pgmap v1061: 129 pgs: 129 active+clean; 763 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 10 MiB/s wr, 524 op/s 2024-09-16T00:40:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:51 smithi060 ceph-mon[35185]: pgmap v1061: 129 pgs: 129 active+clean; 763 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 10 MiB/s wr, 524 op/s 2024-09-16T00:40:52.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:51 smithi033 ceph-mon[31243]: pgmap v1061: 129 pgs: 129 active+clean; 763 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 10 MiB/s wr, 524 op/s 2024-09-16T00:40:52.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 206206 18.32 MB/sec execute 478 sec latency 102.412 ms 2024-09-16T00:40:53.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 206544 18.31 MB/sec execute 479 sec latency 86.800 ms 2024-09-16T00:40:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:53 smithi083 ceph-mon[35242]: pgmap v1062: 129 pgs: 129 active+clean; 745 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 14 MiB/s wr, 685 op/s 2024-09-16T00:40:53.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:53 smithi060 ceph-mon[35185]: pgmap v1062: 129 pgs: 129 active+clean; 745 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 14 MiB/s wr, 685 op/s 2024-09-16T00:40:54.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:53 smithi033 ceph-mon[31243]: pgmap v1062: 129 pgs: 129 active+clean; 745 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 14 MiB/s wr, 685 op/s 2024-09-16T00:40:54.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 206970 18.31 MB/sec execute 480 sec latency 80.850 ms 2024-09-16T00:40:55.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 207580 18.35 MB/sec execute 481 sec latency 61.351 ms 2024-09-16T00:40:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:55 smithi083 ceph-mon[35242]: pgmap v1063: 129 pgs: 129 active+clean; 720 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 14 MiB/s wr, 674 op/s 2024-09-16T00:40:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:55 smithi060 ceph-mon[35185]: pgmap v1063: 129 pgs: 129 active+clean; 720 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 14 MiB/s wr, 674 op/s 2024-09-16T00:40:56.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:55 smithi033 ceph-mon[31243]: pgmap v1063: 129 pgs: 129 active+clean; 720 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 14 MiB/s wr, 674 op/s 2024-09-16T00:40:56.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 208273 18.39 MB/sec execute 482 sec latency 47.605 ms 2024-09-16T00:40:57.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 208835 18.40 MB/sec execute 483 sec latency 53.476 ms 2024-09-16T00:40:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:57 smithi060 ceph-mon[35185]: pgmap v1064: 129 pgs: 129 active+clean; 705 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 13 MiB/s wr, 635 op/s 2024-09-16T00:40:58.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:57 smithi033 ceph-mon[31243]: pgmap v1064: 129 pgs: 129 active+clean; 705 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 13 MiB/s wr, 635 op/s 2024-09-16T00:40:58.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 209397 18.44 MB/sec execute 484 sec latency 126.496 ms 2024-09-16T00:40:58.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:57 smithi083 ceph-mon[35242]: pgmap v1064: 129 pgs: 129 active+clean; 705 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 13 MiB/s wr, 635 op/s 2024-09-16T00:40:59.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 209959 18.47 MB/sec execute 485 sec latency 68.424 ms 2024-09-16T00:40:59.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:40:59 smithi083 ceph-mon[35242]: pgmap v1065: 129 pgs: 129 active+clean; 710 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 13 MiB/s wr, 697 op/s 2024-09-16T00:40:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:40:59 smithi060 ceph-mon[35185]: pgmap v1065: 129 pgs: 129 active+clean; 710 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 13 MiB/s wr, 697 op/s 2024-09-16T00:41:00.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:40:59 smithi033 ceph-mon[31243]: pgmap v1065: 129 pgs: 129 active+clean; 710 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 13 MiB/s wr, 697 op/s 2024-09-16T00:41:00.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 210469 18.49 MB/sec execute 486 sec latency 137.147 ms 2024-09-16T00:41:01.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 210837 18.48 MB/sec execute 487 sec latency 452.810 ms 2024-09-16T00:41:01.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:01 smithi060 ceph-mon[35185]: pgmap v1066: 129 pgs: 129 active+clean; 712 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 11 MiB/s wr, 692 op/s 2024-09-16T00:41:02.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:01 smithi033 ceph-mon[31243]: pgmap v1066: 129 pgs: 129 active+clean; 712 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 11 MiB/s wr, 692 op/s 2024-09-16T00:41:02.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 211299 18.49 MB/sec execute 488 sec latency 70.720 ms 2024-09-16T00:41:02.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:01 smithi083 ceph-mon[35242]: pgmap v1066: 129 pgs: 129 active+clean; 712 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 11 MiB/s wr, 692 op/s 2024-09-16T00:41:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:02 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:41:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:02 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:41:02.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:02 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:41:03.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:02 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:41:03.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:02 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:41:03.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:02 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:41:03.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 211871 18.51 MB/sec execute 489 sec latency 87.319 ms 2024-09-16T00:41:03.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:02 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:41:03.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:02 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:41:03.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:02 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:41:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:03 smithi060 ceph-mon[35185]: pgmap v1067: 129 pgs: 129 active+clean; 717 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.7 MiB/s wr, 570 op/s 2024-09-16T00:41:04.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:03 smithi033 ceph-mon[31243]: pgmap v1067: 129 pgs: 129 active+clean; 717 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.7 MiB/s wr, 570 op/s 2024-09-16T00:41:04.102 INFO:tasks.workunit.client.0.smithi033.stdout: 10 212667 18.57 MB/sec execute 490 sec latency 41.374 ms 2024-09-16T00:41:04.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:03 smithi083 ceph-mon[35242]: pgmap v1067: 129 pgs: 129 active+clean; 717 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.7 MiB/s wr, 570 op/s 2024-09-16T00:41:05.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 213259 18.59 MB/sec execute 491 sec latency 60.774 ms 2024-09-16T00:41:05.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:05 smithi060 ceph-mon[35185]: pgmap v1068: 129 pgs: 129 active+clean; 723 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.5 MiB/s wr, 419 op/s 2024-09-16T00:41:06.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:05 smithi033 ceph-mon[31243]: pgmap v1068: 129 pgs: 129 active+clean; 723 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.5 MiB/s wr, 419 op/s 2024-09-16T00:41:06.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 213734 18.61 MB/sec execute 492 sec latency 78.789 ms 2024-09-16T00:41:06.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:05 smithi083 ceph-mon[35242]: pgmap v1068: 129 pgs: 129 active+clean; 723 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.5 MiB/s wr, 419 op/s 2024-09-16T00:41:07.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 214234 18.63 MB/sec execute 493 sec latency 95.001 ms 2024-09-16T00:41:07.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:07 smithi033 ceph-mon[31243]: pgmap v1069: 129 pgs: 129 active+clean; 715 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.1 MiB/s wr, 385 op/s 2024-09-16T00:41:07.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:07 smithi060 ceph-mon[35185]: pgmap v1069: 129 pgs: 129 active+clean; 715 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.1 MiB/s wr, 385 op/s 2024-09-16T00:41:08.104 INFO:tasks.workunit.client.0.smithi033.stdout: 10 214789 18.66 MB/sec execute 494 sec latency 68.154 ms 2024-09-16T00:41:08.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:07 smithi083 ceph-mon[35242]: pgmap v1069: 129 pgs: 129 active+clean; 715 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.1 MiB/s wr, 385 op/s 2024-09-16T00:41:09.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 215151 18.66 MB/sec execute 495 sec latency 86.330 ms 2024-09-16T00:41:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:09 smithi060 ceph-mon[35185]: pgmap v1070: 129 pgs: 129 active+clean; 713 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.3 MiB/s wr, 398 op/s 2024-09-16T00:41:10.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:09 smithi033 ceph-mon[31243]: pgmap v1070: 129 pgs: 129 active+clean; 713 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.3 MiB/s wr, 398 op/s 2024-09-16T00:41:10.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 215592 18.66 MB/sec execute 496 sec latency 68.549 ms 2024-09-16T00:41:10.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:09 smithi083 ceph-mon[35242]: pgmap v1070: 129 pgs: 129 active+clean; 713 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.3 MiB/s wr, 398 op/s 2024-09-16T00:41:11.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 216172 18.67 MB/sec execute 497 sec latency 99.003 ms 2024-09-16T00:41:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:11 smithi060 ceph-mon[35185]: pgmap v1071: 129 pgs: 129 active+clean; 713 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.0 MiB/s wr, 360 op/s 2024-09-16T00:41:12.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:11 smithi033 ceph-mon[31243]: pgmap v1071: 129 pgs: 129 active+clean; 713 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.0 MiB/s wr, 360 op/s 2024-09-16T00:41:12.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 216819 18.72 MB/sec execute 498 sec latency 67.967 ms 2024-09-16T00:41:12.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:11 smithi083 ceph-mon[35242]: pgmap v1071: 129 pgs: 129 active+clean; 713 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.0 MiB/s wr, 360 op/s 2024-09-16T00:41:13.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 217438 18.75 MB/sec execute 499 sec latency 68.207 ms 2024-09-16T00:41:13.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:13 smithi060 ceph-mon[35185]: pgmap v1072: 129 pgs: 129 active+clean; 719 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.3 MiB/s wr, 384 op/s 2024-09-16T00:41:14.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:13 smithi033 ceph-mon[31243]: pgmap v1072: 129 pgs: 129 active+clean; 719 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.3 MiB/s wr, 384 op/s 2024-09-16T00:41:14.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 217915 18.77 MB/sec execute 500 sec latency 51.831 ms 2024-09-16T00:41:14.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:13 smithi083 ceph-mon[35242]: pgmap v1072: 129 pgs: 129 active+clean; 719 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.3 MiB/s wr, 384 op/s 2024-09-16T00:41:15.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 218416 18.78 MB/sec execute 501 sec latency 80.578 ms 2024-09-16T00:41:15.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:15 smithi060 ceph-mon[35185]: pgmap v1073: 129 pgs: 129 active+clean; 725 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.3 MiB/s wr, 380 op/s 2024-09-16T00:41:16.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:15 smithi033 ceph-mon[31243]: pgmap v1073: 129 pgs: 129 active+clean; 725 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.3 MiB/s wr, 380 op/s 2024-09-16T00:41:16.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 218764 18.79 MB/sec execute 502 sec latency 499.798 ms 2024-09-16T00:41:16.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:15 smithi083 ceph-mon[35242]: pgmap v1073: 129 pgs: 129 active+clean; 725 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.3 MiB/s wr, 380 op/s 2024-09-16T00:41:17.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 218807 18.76 MB/sec execute 503 sec latency 1470.101 ms 2024-09-16T00:41:17.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:17 smithi060 ceph-mon[35185]: pgmap v1074: 129 pgs: 129 active+clean; 729 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.1 MiB/s wr, 356 op/s 2024-09-16T00:41:18.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:17 smithi033 ceph-mon[31243]: pgmap v1074: 129 pgs: 129 active+clean; 729 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.1 MiB/s wr, 356 op/s 2024-09-16T00:41:18.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 218863 18.73 MB/sec execute 504 sec latency 1825.323 ms 2024-09-16T00:41:18.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:17 smithi083 ceph-mon[35242]: pgmap v1074: 129 pgs: 129 active+clean; 729 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.1 MiB/s wr, 356 op/s 2024-09-16T00:41:18.883 INFO:tasks.quiescer.fs.[cephfs]:Total duration (54.4) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T00:41:18.883 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.2 2024-09-16T00:41:18.884 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:41:19.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 219108 18.71 MB/sec execute 505 sec latency 154.040 ms 2024-09-16T00:41:19.330 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:19.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.945+0000 7f1c510b2640 1 -- 172.21.15.33:0/4163947302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c4c106a90 msgr2=0x7f1c4c106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:19.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.945+0000 7f1c510b2640 1 --2- 172.21.15.33:0/4163947302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c4c106a90 0x7f1c4c106e70 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c009930 tx=0x7f1c3c02a480 comp rx=0 tx=0).stop 2024-09-16T00:41:19.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.948+0000 7f1c510b2640 1 -- 172.21.15.33:0/4163947302 shutdown_connections 2024-09-16T00:41:19.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.948+0000 7f1c510b2640 1 --2- 172.21.15.33:0/4163947302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c4c101430 0x7f1c4c10d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:19.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.948+0000 7f1c510b2640 1 --2- 172.21.15.33:0/4163947302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c4c100a90 0x7f1c4c100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:19.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.948+0000 7f1c510b2640 1 --2- 172.21.15.33:0/4163947302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c4c106a90 0x7f1c4c106e70 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:19.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.948+0000 7f1c510b2640 1 -- 172.21.15.33:0/4163947302 >> 172.21.15.33:0/4163947302 conn(0x7f1c4c0fc7d0 msgr2=0x7f1c4c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:19.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.950+0000 7f1c510b2640 1 -- 172.21.15.33:0/4163947302 shutdown_connections 2024-09-16T00:41:19.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.950+0000 7f1c510b2640 1 -- 172.21.15.33:0/4163947302 wait complete. 2024-09-16T00:41:19.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.951+0000 7f1c510b2640 1 Processor -- start 2024-09-16T00:41:19.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.951+0000 7f1c510b2640 1 -- start start 2024-09-16T00:41:19.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.951+0000 7f1c510b2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c4c100a90 0x7f1c4c109c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:19.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.951+0000 7f1c510b2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c4c101430 0x7f1c4c10a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:19.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.951+0000 7f1c510b2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c4c106a90 0x7f1c4c10a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:19.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.951+0000 7f1c510b2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c10fe90 con 0x7f1c4c101430 2024-09-16T00:41:19.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.952+0000 7f1c510b2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c10fd10 con 0x7f1c4c106a90 2024-09-16T00:41:19.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.952+0000 7f1c510b2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c110010 con 0x7f1c4c100a90 2024-09-16T00:41:19.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.952+0000 7f1c4a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c4c101430 0x7f1c4c10a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:19.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.952+0000 7f1c4a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c4c101430 0x7f1c4c10a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35532/0 (socket says 172.21.15.33:35532) 2024-09-16T00:41:19.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.952+0000 7f1c4a59c640 1 -- 172.21.15.33:0/908790691 learned_addr learned my addr 172.21.15.33:0/908790691 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:19.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.953+0000 7f1c4a59c640 1 -- 172.21.15.33:0/908790691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c4c100a90 msgr2=0x7f1c4c109c10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T00:41:19.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.953+0000 7f1c4a59c640 1 --2- 172.21.15.33:0/908790691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c4c100a90 0x7f1c4c109c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:19.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.953+0000 7f1c4a59c640 1 -- 172.21.15.33:0/908790691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c4c106a90 msgr2=0x7f1c4c10a690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T00:41:19.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.953+0000 7f1c4a59c640 1 --2- 172.21.15.33:0/908790691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c4c106a90 0x7f1c4c10a690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:19.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.953+0000 7f1c4a59c640 1 -- 172.21.15.33:0/908790691 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4c115770 con 0x7f1c4c101430 2024-09-16T00:41:19.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.954+0000 7f1c4a59c640 1 --2- 172.21.15.33:0/908790691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c4c101430 0x7f1c4c10a150 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000b4a0 tx=0x7f1c4000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:19.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.954+0000 7f1c33fff640 1 -- 172.21.15.33:0/908790691 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c40014070 con 0x7f1c4c101430 2024-09-16T00:41:19.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.954+0000 7f1c510b2640 1 -- 172.21.15.33:0/908790691 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c4c115a60 con 0x7f1c4c101430 2024-09-16T00:41:19.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.954+0000 7f1c510b2640 1 -- 172.21.15.33:0/908790691 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c4c115ff0 con 0x7f1c4c101430 2024-09-16T00:41:19.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.956+0000 7f1c33fff640 1 -- 172.21.15.33:0/908790691 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c40010af0 con 0x7f1c4c101430 2024-09-16T00:41:19.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.956+0000 7f1c33fff640 1 -- 172.21.15.33:0/908790691 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4001e060 con 0x7f1c4c101430 2024-09-16T00:41:19.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.958+0000 7f1c510b2640 1 -- 172.21.15.33:0/908790691 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c4c1021d0 con 0x7f1c4c101430 2024-09-16T00:41:19.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.962+0000 7f1c33fff640 1 -- 172.21.15.33:0/908790691 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1c40017070 con 0x7f1c4c101430 2024-09-16T00:41:19.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.963+0000 7f1c33fff640 1 --2- 172.21.15.33:0/908790691 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c340a3380 0x7f1c340a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:19.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.963+0000 7f1c33fff640 1 -- 172.21.15.33:0/908790691 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1c400cf190 con 0x7f1c4c101430 2024-09-16T00:41:19.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.965+0000 7f1c4ad9d640 1 --2- 172.21.15.33:0/908790691 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c340a3380 0x7f1c340a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:19.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.967+0000 7f1c4ad9d640 1 --2- 172.21.15.33:0/908790691 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c340a3380 0x7f1c340a5840 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c0096c0 tx=0x7f1c3c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:19.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:19.972+0000 7f1c33fff640 1 -- 172.21.15.33:0/908790691 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1c400a3f40 con 0x7f1c4c101430 2024-09-16T00:41:19.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:19 smithi060 ceph-mon[35185]: pgmap v1075: 129 pgs: 129 active+clean; 749 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 349 op/s 2024-09-16T00:41:20.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:19 smithi033 ceph-mon[31243]: pgmap v1075: 129 pgs: 129 active+clean; 749 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 349 op/s 2024-09-16T00:41:20.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 219405 18.69 MB/sec execute 506 sec latency 97.682 ms 2024-09-16T00:41:20.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:19 smithi083 ceph-mon[35242]: pgmap v1075: 129 pgs: 129 active+clean; 749 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 349 op/s 2024-09-16T00:41:20.183 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:41:20.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.380+0000 7f1c510b2640 1 -- 172.21.15.33:0/908790691 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c4c116300 con 0x7f1c4c101430 2024-09-16T00:41:20.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.383+0000 7f1c33fff640 1 -- 172.21.15.33:0/908790691 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f1c40017360 con 0x7f1c4c101430 2024-09-16T00:41:20.384 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:41:20.385 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:41:20.385 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:41:20.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.387+0000 7f1c510b2640 1 -- 172.21.15.33:0/908790691 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c340a3380 msgr2=0x7f1c340a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:20.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.387+0000 7f1c510b2640 1 --2- 172.21.15.33:0/908790691 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c340a3380 0x7f1c340a5840 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c0096c0 tx=0x7f1c3c005fd0 comp rx=0 tx=0).stop 2024-09-16T00:41:20.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.387+0000 7f1c510b2640 1 -- 172.21.15.33:0/908790691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c4c101430 msgr2=0x7f1c4c10a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:20.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.387+0000 7f1c510b2640 1 --2- 172.21.15.33:0/908790691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c4c101430 0x7f1c4c10a150 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000b4a0 tx=0x7f1c4000b960 comp rx=0 tx=0).stop 2024-09-16T00:41:20.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.388+0000 7f1c510b2640 1 -- 172.21.15.33:0/908790691 shutdown_connections 2024-09-16T00:41:20.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.388+0000 7f1c510b2640 1 --2- 172.21.15.33:0/908790691 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1c340a3380 0x7f1c340a5840 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:20.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.388+0000 7f1c510b2640 1 --2- 172.21.15.33:0/908790691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c4c106a90 0x7f1c4c10a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:20.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.388+0000 7f1c510b2640 1 --2- 172.21.15.33:0/908790691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c4c101430 0x7f1c4c10a150 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:20.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.388+0000 7f1c510b2640 1 --2- 172.21.15.33:0/908790691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c4c100a90 0x7f1c4c109c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:20.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.388+0000 7f1c510b2640 1 -- 172.21.15.33:0/908790691 >> 172.21.15.33:0/908790691 conn(0x7f1c4c0fc7d0 msgr2=0x7f1c4c0fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:20.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.390+0000 7f1c510b2640 1 -- 172.21.15.33:0/908790691 shutdown_connections 2024-09-16T00:41:20.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:20.391+0000 7f1c510b2640 1 -- 172.21.15.33:0/908790691 wait complete. 2024-09-16T00:41:21.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:20 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/908790691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:41:21.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 219753 18.70 MB/sec execute 507 sec latency 146.846 ms 2024-09-16T00:41:21.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:20 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/908790691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:41:21.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:20 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/908790691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:41:21.767 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 147.2 --await-for 45.0' 2024-09-16T00:41:21.768 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 147.2 --await-for 45.0 2024-09-16T00:41:21.931 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:22.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:21 smithi033 ceph-mon[31243]: pgmap v1076: 129 pgs: 129 active+clean; 789 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 9.9 MiB/s wr, 461 op/s 2024-09-16T00:41:22.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 219990 18.68 MB/sec execute 508 sec latency 248.612 ms 2024-09-16T00:41:22.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:21 smithi083 ceph-mon[35242]: pgmap v1076: 129 pgs: 129 active+clean; 789 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 9.9 MiB/s wr, 461 op/s 2024-09-16T00:41:22.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:21 smithi060 ceph-mon[35185]: pgmap v1076: 129 pgs: 129 active+clean; 789 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 9.9 MiB/s wr, 461 op/s 2024-09-16T00:41:22.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.572+0000 7f1bfd47d640 1 -- 172.21.15.33:0/3475236671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bf8100e20 msgr2=0x7f1bf8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:22.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.572+0000 7f1bfd47d640 1 --2- 172.21.15.33:0/3475236671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bf8100e20 0x7f1bf8101200 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009930 tx=0x7f1be802a4d0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.572+0000 7f1bfd47d640 1 -- 172.21.15.33:0/3475236671 shutdown_connections 2024-09-16T00:41:22.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.572+0000 7f1bfd47d640 1 --2- 172.21.15.33:0/3475236671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1bf8102ef0 0x7f1bf810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.572+0000 7f1bfd47d640 1 --2- 172.21.15.33:0/3475236671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bf8102570 0x7f1bf81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.572+0000 7f1bfd47d640 1 --2- 172.21.15.33:0/3475236671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bf8100e20 0x7f1bf8101200 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.572+0000 7f1bfd47d640 1 -- 172.21.15.33:0/3475236671 >> 172.21.15.33:0/3475236671 conn(0x7f1bf80fc7d0 msgr2=0x7f1bf80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:22.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.572+0000 7f1bfd47d640 1 -- 172.21.15.33:0/3475236671 shutdown_connections 2024-09-16T00:41:22.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.572+0000 7f1bfd47d640 1 -- 172.21.15.33:0/3475236671 wait complete. 2024-09-16T00:41:22.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.573+0000 7f1bfd47d640 1 Processor -- start 2024-09-16T00:41:22.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.573+0000 7f1bfd47d640 1 -- start start 2024-09-16T00:41:22.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.574+0000 7f1bfd47d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1bf8100e20 0x7f1bf819d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.574+0000 7f1bfd47d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bf8102570 0x7f1bf8196180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.574+0000 7f1bfd47d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bf8102ef0 0x7f1bf81966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.574+0000 7f1bfd47d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf810fe40 con 0x7f1bf8100e20 2024-09-16T00:41:22.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.574+0000 7f1bf77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bf8102ef0 0x7f1bf81966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bf67fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bf8102570 0x7f1bf8196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bf67fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bf8102570 0x7f1bf8196180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:33102/0 (socket says 172.21.15.33:33102) 2024-09-16T00:41:22.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bf77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bf8102ef0 0x7f1bf81966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40332/0 (socket says 172.21.15.33:40332) 2024-09-16T00:41:22.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bf67fc640 1 -- 172.21.15.33:0/4212095890 learned_addr learned my addr 172.21.15.33:0/4212095890 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:22.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.574+0000 7f1bf6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1bf8100e20 0x7f1bf819d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bfd47d640 1 -- 172.21.15.33:0/4212095890 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf810fcc0 con 0x7f1bf8102570 2024-09-16T00:41:22.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bfd47d640 1 -- 172.21.15.33:0/4212095890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf810ffc0 con 0x7f1bf8102ef0 2024-09-16T00:41:22.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/4212095890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bf8102ef0 msgr2=0x7f1bf81966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:22.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bf8102ef0 0x7f1bf81966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/4212095890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bf8102570 msgr2=0x7f1bf8196180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:22.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bf8102570 0x7f1bf8196180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/4212095890 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf8196cb0 con 0x7f1bf8100e20 2024-09-16T00:41:22.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bf67fc640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bf8102570 0x7f1bf8196180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:41:22.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.575+0000 7f1bf77fe640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bf8102ef0 0x7f1bf81966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:41:22.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.576+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1bf8100e20 0x7f1bf819d0f0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f1be8005940 tx=0x7f1be80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:22.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.576+0000 7f1bdffff640 1 -- 172.21.15.33:0/4212095890 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be803f070 con 0x7f1bf8100e20 2024-09-16T00:41:22.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.576+0000 7f1bdffff640 1 -- 172.21.15.33:0/4212095890 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1be80044c0 con 0x7f1bf8100e20 2024-09-16T00:41:22.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.576+0000 7f1bdffff640 1 -- 172.21.15.33:0/4212095890 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8035070 con 0x7f1bf8100e20 2024-09-16T00:41:22.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.576+0000 7f1bfd47d640 1 -- 172.21.15.33:0/4212095890 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf8196f40 con 0x7f1bf8100e20 2024-09-16T00:41:22.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.577+0000 7f1bfd47d640 1 -- 172.21.15.33:0/4212095890 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf819adf0 con 0x7f1bf8100e20 2024-09-16T00:41:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.580+0000 7f1bdffff640 1 -- 172.21.15.33:0/4212095890 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1be803b460 con 0x7f1bf8100e20 2024-09-16T00:41:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.580+0000 7f1bff856740 1 Processor -- start 2024-09-16T00:41:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.581+0000 7f1bff856740 1 -- start start 2024-09-16T00:41:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.581+0000 7f1bff856740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcf4a0 0x558815bcf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.581+0000 7f1bdffff640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1be00a3390 0x7f1be00a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.582+0000 7f1bff856740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcfdc0 0x558815bda140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.582+0000 7f1bff856740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bda680 0x558815bdca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.582+0000 7f1bdffff640 1 -- 172.21.15.33:0/4212095890 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1be80f00f0 con 0x7f1bf8100e20 2024-09-16T00:41:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.582+0000 7f1bf67fc640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1be00a3390 0x7f1be00a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.582+0000 7f1bf77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bda680 0x558815bdca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.582+0000 7f1bf6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcfdc0 0x558815bda140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.582+0000 7f1bff856740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558815b352e0 con 0x558815bcf4a0 2024-09-16T00:41:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.582+0000 7f1bf6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcfdc0 0x558815bda140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:33118/0 (socket says 172.21.15.33:33118) 2024-09-16T00:41:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.582+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/401917478 learned_addr learned my addr 172.21.15.33:0/401917478 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.582+0000 7f1bff856740 1 -- 172.21.15.33:0/401917478 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558815adcce0 con 0x558815bcfdc0 2024-09-16T00:41:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.582+0000 7f1bff856740 1 -- 172.21.15.33:0/401917478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558815b312a0 con 0x558815bda680 2024-09-16T00:41:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.583+0000 7f1bf67fc640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1be00a3390 0x7f1be00a5850 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8197940 tx=0x7f1bec00d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.583+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/401917478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bda680 msgr2=0x558815bdca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.583+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/401917478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bda680 0x558815bdca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.583+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/401917478 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcf4a0 msgr2=0x558815bcf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.583+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/401917478 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcf4a0 0x558815bcf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.583+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/401917478 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558815ac1440 con 0x558815bcfdc0 2024-09-16T00:41:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.584+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/401917478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcfdc0 0x558815bda140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1be80b87a0 tx=0x7f1be80bbe80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf05b37f2282f27d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.584+0000 7f1bdd7fa640 1 -- 172.21.15.33:0/401917478 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8048040 con 0x558815bcfdc0 2024-09-16T00:41:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.585+0000 7f1bdd7fa640 1 -- 172.21.15.33:0/401917478 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1be80bc340 con 0x558815bcfdc0 2024-09-16T00:41:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.585+0000 7f1bdd7fa640 1 -- 172.21.15.33:0/401917478 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be80ed960 con 0x558815bcfdc0 2024-09-16T00:41:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.586+0000 7f1bff856740 1 -- 172.21.15.33:0/401917478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcfdc0 msgr2=0x558815bda140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.586+0000 7f1bff856740 1 --2- 172.21.15.33:0/401917478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcfdc0 0x558815bda140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1be80b87a0 tx=0x7f1be80bbe80 comp rx=0 tx=0).stop 2024-09-16T00:41:22.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.589+0000 7f1bff856740 1 -- 172.21.15.33:0/401917478 shutdown_connections 2024-09-16T00:41:22.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.590+0000 7f1bff856740 1 --2- 172.21.15.33:0/401917478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bda680 0x558815bdca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.590+0000 7f1bff856740 1 --2- 172.21.15.33:0/401917478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcfdc0 0x558815bda140 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.590+0000 7f1bff856740 1 --2- 172.21.15.33:0/401917478 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcf4a0 0x558815bcf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.590+0000 7f1bff856740 1 -- 172.21.15.33:0/401917478 >> 172.21.15.33:0/401917478 conn(0x558815bcba30 msgr2=0x558815bcbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:22.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.591+0000 7f1bff856740 1 -- 172.21.15.33:0/401917478 shutdown_connections 2024-09-16T00:41:22.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.591+0000 7f1bff856740 1 -- 172.21.15.33:0/401917478 wait complete. 2024-09-16T00:41:22.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.592+0000 7f1bff856740 1 -- start start 2024-09-16T00:41:22.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.592+0000 7f1bff856740 1 Processor -- start 2024-09-16T00:41:22.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.592+0000 7f1bff856740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bcf4a0 0x558815bd9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.592+0000 7f1bff856740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcfdc0 0x558815bd7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.592+0000 7f1bff856740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bda680 0x558815bd7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bff856740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558815bdd010 con 0x558815bcfdc0 2024-09-16T00:41:22.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bff856740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x558815af0050 con 0x558815bda680 2024-09-16T00:41:22.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bff856740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558815bdd190 con 0x558815bcf4a0 2024-09-16T00:41:22.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bff856740 10 client.0 _ll_register_callbacks cb 0x558815b30160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:22.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bf6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bcf4a0 0x558815bd9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bf6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bcf4a0 0x558815bd9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40346/0 (socket says 172.21.15.33:40346) 2024-09-16T00:41:22.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/2562354335 learned_addr learned my addr 172.21.15.33:0/2562354335 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:22.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bf67fc640 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcfdc0 0x558815bd7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bf77fe640 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bda680 0x558815bd7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/2562354335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bda680 msgr2=0x558815bd7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:22.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bda680 0x558815bd7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/2562354335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcfdc0 msgr2=0x558815bd7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:22.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcfdc0 0x558815bd7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.593+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/2562354335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588158a63c0 con 0x558815bcf4a0 2024-09-16T00:41:22.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.594+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bcf4a0 0x558815bd9f90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1be8006010 tx=0x7f1be80eda30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=183ab3c50af62073 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:22.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.594+0000 7f1bdaff5640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:41:22.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.594+0000 7f1bdaff5640 1 -- 172.21.15.33:0/2562354335 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be80d4020 con 0x558815bcf4a0 2024-09-16T00:41:22.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.594+0000 7f1bdaff5640 1 -- 172.21.15.33:0/2562354335 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1be80dc110 con 0x558815bcf4a0 2024-09-16T00:41:22.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.595+0000 7f1bff856740 1 -- 172.21.15.33:0/2562354335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1be80bc340 con 0x558815bcf4a0 2024-09-16T00:41:22.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.598+0000 7f1bdaff5640 1 -- 172.21.15.33:0/2562354335 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1be8048070 con 0x558815bcf4a0 2024-09-16T00:41:22.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.599+0000 7f1bff856740 10 client.28969 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:22.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.599+0000 7f1bff856740 1 -- 172.21.15.33:0/2562354335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558815bdd010 con 0x558815bcf4a0 2024-09-16T00:41:22.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.602+0000 7f1bdaff5640 1 -- 172.21.15.33:0/2562354335 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1be80bc340 con 0x558815bcf4a0 2024-09-16T00:41:22.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.602+0000 7f1bdaff5640 1 -- 172.21.15.33:0/2562354335 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f1be804b060 con 0x558815bcf4a0 2024-09-16T00:41:22.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.602+0000 7f1bff856740 10 client.28969 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:22.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.602+0000 7f1bff856740 10 client.28969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:41:22.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.602+0000 7f1bff856740 20 client.28969 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:22.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.602+0000 7f1bff856740 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558815be9a80 0x558815bebe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.602+0000 7f1bff856740 4 client.28969 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:41:22.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.602+0000 7f1bff856740 1 -- 172.21.15.33:0/2562354335 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558815b42850 con 0x558815be9a80 2024-09-16T00:41:22.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.603+0000 7f1bf67fc640 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558815be9a80 0x558815bebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.608+0000 7f1bf67fc640 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558815be9a80 0x558815bebe70 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b439d59e310aa160 server_cookie=3a6403338e8d0694 in_seq=0 out_seq=0 2024-09-16T00:41:22.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.608+0000 7f1bdaff5640 10 client.28969 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:22.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.609+0000 7f1bdaff5640 1 -- 172.21.15.33:0/2562354335 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558815b42850 con 0x558815be9a80 2024-09-16T00:41:22.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.609+0000 7f1bdaff5640 10 client.28969 handle_command_reply: tid=0 2024-09-16T00:41:22.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.609+0000 7f1bff856740 1 client.28969 shutdown 2024-09-16T00:41:22.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.609+0000 7f1bff856740 1 -- 172.21.15.33:0/2562354335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bcf4a0 msgr2=0x558815bd9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:22.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.609+0000 7f1bff856740 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bcf4a0 0x558815bd9f90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1be8006010 tx=0x7f1be80eda30 comp rx=0 tx=0).stop 2024-09-16T00:41:22.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.609+0000 7f1bff856740 1 -- 172.21.15.33:0/2562354335 shutdown_connections 2024-09-16T00:41:22.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.610+0000 7f1bff856740 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558815be9a80 0x558815bebe70 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.610+0000 7f1bff856740 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bda680 0x558815bd7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.610+0000 7f1bff856740 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcfdc0 0x558815bd7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.610+0000 7f1bff856740 1 --2- 172.21.15.33:0/2562354335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bcf4a0 0x558815bd9f90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.610+0000 7f1bdaff5640 0 client.28969 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:22.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.610+0000 7f1bff856740 1 -- 172.21.15.33:0/2562354335 >> 172.21.15.33:0/2562354335 conn(0x558815bcba30 msgr2=0x558815bd1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:22.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.610+0000 7f1bff856740 1 -- 172.21.15.33:0/2562354335 shutdown_connections 2024-09-16T00:41:22.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.610+0000 7f1bff856740 1 -- 172.21.15.33:0/2562354335 wait complete. 2024-09-16T00:41:22.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.610+0000 7f1bff856740 20 client.28969 trim_cache size 0 max 16384 2024-09-16T00:41:22.646 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:22.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.645+0000 7f1bff856740 1 Processor -- start 2024-09-16T00:41:22.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.645+0000 7f1bff856740 1 -- start start 2024-09-16T00:41:22.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.645+0000 7f1bff856740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 0x558815bcae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.645+0000 7f1bff856740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcfdc0 0x558815bf16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.645+0000 7f1bff856740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bda680 0x558815bf1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.645+0000 7f1bf67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcfdc0 0x558815bf16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.645+0000 7f1bf67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcfdc0 0x558815bf16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38222/0 (socket says 172.21.15.33:38222) 2024-09-16T00:41:22.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.645+0000 7f1bf67fc640 1 -- 172.21.15.33:0/2785373477 learned_addr learned my addr 172.21.15.33:0/2785373477 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:22.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.645+0000 7f1bff856740 1 -- 172.21.15.33:0/2785373477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81c08f0 con 0x558815bcfdc0 2024-09-16T00:41:22.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.646+0000 7f1bff856740 1 -- 172.21.15.33:0/2785373477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81be1c0 con 0x558815bcf4a0 2024-09-16T00:41:22.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.646+0000 7f1bff856740 1 -- 172.21.15.33:0/2785373477 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81bc010 con 0x558815bda680 2024-09-16T00:41:22.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.646+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/2785373477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 0x558815bcae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.646+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/2785373477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bda680 msgr2=0x558815bf1c30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:41:22.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.646+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/2785373477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bda680 0x558815bf1c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.646+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/2785373477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcfdc0 msgr2=0x558815bf16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:22.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.646+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/2785373477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcfdc0 0x558815bf16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.646+0000 7f1bf6ffd640 1 -- 172.21.15.33:0/2785373477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558815bcb860 con 0x558815bcf4a0 2024-09-16T00:41:22.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.648+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/2785373477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 0x558815bcae40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1be80bc2d0 tx=0x7f1be803b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93ae9b2309de2425 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:22.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.649+0000 7f1bf67fc640 1 --2- 172.21.15.33:0/2785373477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcfdc0 0x558815bf16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:41:22.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.650+0000 7f1bdcff9640 1 -- 172.21.15.33:0/2785373477 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be80d4070 con 0x558815bcf4a0 2024-09-16T00:41:22.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.650+0000 7f1bdcff9640 1 -- 172.21.15.33:0/2785373477 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1be80d5220 con 0x558815bcf4a0 2024-09-16T00:41:22.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.650+0000 7f1bdcff9640 1 -- 172.21.15.33:0/2785373477 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8031e30 con 0x558815bcf4a0 2024-09-16T00:41:22.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.651+0000 7f1bff856740 1 -- 172.21.15.33:0/2785373477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 msgr2=0x558815bcae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:22.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.652+0000 7f1bff856740 1 --2- 172.21.15.33:0/2785373477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 0x558815bcae40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1be80bc2d0 tx=0x7f1be803b800 comp rx=0 tx=0).stop 2024-09-16T00:41:22.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.652+0000 7f1bff856740 1 -- 172.21.15.33:0/2785373477 shutdown_connections 2024-09-16T00:41:22.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.652+0000 7f1bff856740 1 --2- 172.21.15.33:0/2785373477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bda680 0x558815bf1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.652+0000 7f1bff856740 1 --2- 172.21.15.33:0/2785373477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bcfdc0 0x558815bf16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.652+0000 7f1bff856740 1 --2- 172.21.15.33:0/2785373477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 0x558815bcae40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.652+0000 7f1bff856740 1 -- 172.21.15.33:0/2785373477 >> 172.21.15.33:0/2785373477 conn(0x558815bcba30 msgr2=0x558815bd7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:22.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.653+0000 7f1bff856740 1 -- 172.21.15.33:0/2785373477 shutdown_connections 2024-09-16T00:41:22.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.653+0000 7f1bff856740 1 -- 172.21.15.33:0/2785373477 wait complete. 2024-09-16T00:41:22.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.653+0000 7f1bff856740 1 -- start start 2024-09-16T00:41:22.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.654+0000 7f1bff856740 1 Processor -- start 2024-09-16T00:41:22.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.654+0000 7f1bff856740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 0x558815be2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.654+0000 7f1bff856740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bcfdc0 0x558815bdfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.655+0000 7f1bff856740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bda680 0x558815be0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.655+0000 7f1bf6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 0x558815be2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.655+0000 7f1bf6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 0x558815be2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:33136/0 (socket says 172.21.15.33:33136) 2024-09-16T00:41:22.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.655+0000 7f1bff856740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81bc680 con 0x558815bda680 2024-09-16T00:41:22.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.655+0000 7f1bff856740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81bdaf0 con 0x558815bcf4a0 2024-09-16T00:41:22.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.655+0000 7f1bff856740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81bcd50 con 0x558815bcfdc0 2024-09-16T00:41:22.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.655+0000 7f1bff856740 10 client.0 _ll_register_callbacks cb 0x7f1bf0009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:22.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.655+0000 7f1bf77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bda680 0x558815be0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.655+0000 7f1bf77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bda680 0x558815be0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38224/0 (socket says 172.21.15.33:38224) 2024-09-16T00:41:22.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.655+0000 7f1bf77fe640 1 -- 172.21.15.33:0/2599370974 learned_addr learned my addr 172.21.15.33:0/2599370974 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:22.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.655+0000 7f1bf67fc640 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bcfdc0 0x558815bdfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.656+0000 7f1bf77fe640 1 -- 172.21.15.33:0/2599370974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bcfdc0 msgr2=0x558815bdfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:22.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.656+0000 7f1bf77fe640 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bcfdc0 0x558815bdfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.656+0000 7f1bf77fe640 1 -- 172.21.15.33:0/2599370974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 msgr2=0x558815be2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:22.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.656+0000 7f1bf77fe640 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 0x558815be2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.656+0000 7f1bf77fe640 1 -- 172.21.15.33:0/2599370974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558815be0800 con 0x558815bda680 2024-09-16T00:41:22.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.656+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 0x558815be2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-09-16T00:41:22.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.656+0000 7f1bf77fe640 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bda680 0x558815be0190 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0010e10 tx=0x7f1bf000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b861c4ad54783d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:22.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.656+0000 7f1bdaff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:22.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.657+0000 7f1bdaff5640 1 -- 172.21.15.33:0/2599370974 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0022060 con 0x558815bda680 2024-09-16T00:41:22.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.657+0000 7f1bff856740 1 -- 172.21.15.33:0/2599370974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1be80d5220 con 0x558815bda680 2024-09-16T00:41:22.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.657+0000 7f1bdaff5640 1 -- 172.21.15.33:0/2599370974 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1bf0003d80 con 0x558815bda680 2024-09-16T00:41:22.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.657+0000 7f1bdaff5640 1 -- 172.21.15.33:0/2599370974 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1bf0004050 con 0x558815bda680 2024-09-16T00:41:22.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.660+0000 7f1bff856740 10 client.19848 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:22.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.660+0000 7f1bff856740 1 -- 172.21.15.33:0/2599370974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558815c54570 con 0x558815bda680 2024-09-16T00:41:22.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.661+0000 7f1bdaff5640 1 -- 172.21.15.33:0/2599370974 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bf00046d0 con 0x558815bda680 2024-09-16T00:41:22.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.661+0000 7f1bdaff5640 1 -- 172.21.15.33:0/2599370974 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f1bf001db20 con 0x558815bda680 2024-09-16T00:41:22.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.661+0000 7f1bff856740 10 client.19848 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:22.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.661+0000 7f1bff856740 10 client.19848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:41:22.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.661+0000 7f1bff856740 20 client.19848 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:22.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.661+0000 7f1bff856740 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558815c54c70 0x558815c57090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:22.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.661+0000 7f1bff856740 4 client.19848 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:41:22.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.661+0000 7f1bff856740 1 -- 172.21.15.33:0/2599370974 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f1be80d5220 con 0x558815c54c70 2024-09-16T00:41:22.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.662+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558815c54c70 0x558815c57090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:22.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.664+0000 7f1bf6ffd640 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558815c54c70 0x558815c57090 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75bc10e945082e4 server_cookie=979449b93299a326 in_seq=0 out_seq=0 2024-09-16T00:41:22.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.665+0000 7f1bdaff5640 10 client.19848 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.665+0000 7f1bdaff5640 1 -- 172.21.15.33:0/2599370974 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f1be80d5220 con 0x558815c54c70 2024-09-16T00:41:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.665+0000 7f1bdaff5640 10 client.19848 handle_command_reply: tid=0 2024-09-16T00:41:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.665+0000 7f1bff856740 1 client.19848 shutdown 2024-09-16T00:41:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.665+0000 7f1bff856740 1 -- 172.21.15.33:0/2599370974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bda680 msgr2=0x558815be0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.665+0000 7f1bff856740 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bda680 0x558815be0190 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0010e10 tx=0x7f1bf000f790 comp rx=0 tx=0).stop 2024-09-16T00:41:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.666+0000 7f1bff856740 1 -- 172.21.15.33:0/2599370974 shutdown_connections 2024-09-16T00:41:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.666+0000 7f1bff856740 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x558815c54c70 0x558815c57090 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.667+0000 7f1bff856740 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558815bda680 0x558815be0190 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.667+0000 7f1bff856740 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558815bcfdc0 0x558815bdfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.667+0000 7f1bff856740 1 --2- 172.21.15.33:0/2599370974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558815bcf4a0 0x558815be2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.667+0000 7f1bff856740 1 -- 172.21.15.33:0/2599370974 >> 172.21.15.33:0/2599370974 conn(0x558815bcba30 msgr2=0x558815bd8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.667+0000 7f1bdaff5640 0 client.19848 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:22.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.670+0000 7f1bff856740 1 -- 172.21.15.33:0/2599370974 shutdown_connections 2024-09-16T00:41:22.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.670+0000 7f1bff856740 1 -- 172.21.15.33:0/2599370974 wait complete. 2024-09-16T00:41:22.672 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:41:22.672 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:41:22.672 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:41:22.672 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:41:22.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.670+0000 7f1bff856740 20 client.19848 trim_cache size 0 max 16384 2024-09-16T00:41:22.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.672+0000 7f1bfd47d640 1 -- 172.21.15.33:0/4212095890 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1be00a3390 msgr2=0x7f1be00a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:22.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.672+0000 7f1bfd47d640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1be00a3390 0x7f1be00a5850 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8197940 tx=0x7f1bec00d040 comp rx=0 tx=0).stop 2024-09-16T00:41:22.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.672+0000 7f1bfd47d640 1 -- 172.21.15.33:0/4212095890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1bf8100e20 msgr2=0x7f1bf819d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:22.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.672+0000 7f1bfd47d640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1bf8100e20 0x7f1bf819d0f0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f1be8005940 tx=0x7f1be80040c0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.673+0000 7f1bfd47d640 1 -- 172.21.15.33:0/4212095890 shutdown_connections 2024-09-16T00:41:22.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.673+0000 7f1bfd47d640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1be00a3390 0x7f1be00a5850 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.673+0000 7f1bfd47d640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bf8102ef0 0x7f1bf81966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.673+0000 7f1bfd47d640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bf8102570 0x7f1bf8196180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.673+0000 7f1bfd47d640 1 --2- 172.21.15.33:0/4212095890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1bf8100e20 0x7f1bf819d0f0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:22.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.673+0000 7f1bfd47d640 1 -- 172.21.15.33:0/4212095890 >> 172.21.15.33:0/4212095890 conn(0x7f1bf80fc7d0 msgr2=0x7f1bf8103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:22.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.673+0000 7f1bfd47d640 1 -- 172.21.15.33:0/4212095890 shutdown_connections 2024-09-16T00:41:22.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:22.673+0000 7f1bfd47d640 1 -- 172.21.15.33:0/4212095890 wait complete. 2024-09-16T00:41:23.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 220379 18.69 MB/sec execute 509 sec latency 221.156 ms 2024-09-16T00:41:24.018 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:23 smithi033 ceph-mon[31243]: pgmap v1077: 129 pgs: 129 active+clean; 777 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 12 MiB/s wr, 516 op/s 2024-09-16T00:41:24.019 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:41:24.019 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:41:24.032 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:23 smithi060 ceph-mon[35185]: pgmap v1077: 129 pgs: 129 active+clean; 777 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 12 MiB/s wr, 516 op/s 2024-09-16T00:41:24.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221012 18.72 MB/sec execute 510 sec latency 72.154 ms 2024-09-16T00:41:24.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:23 smithi083 ceph-mon[35242]: pgmap v1077: 129 pgs: 129 active+clean; 777 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 12 MiB/s wr, 516 op/s 2024-09-16T00:41:24.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.270+0000 7f9738f40640 1 -- 172.21.15.33:0/84970358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97341027a0 msgr2=0x7f97341067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:24.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.270+0000 7f9738f40640 1 --2- 172.21.15.33:0/84970358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97341027a0 0x7f97341067f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f971c009900 tx=0x7f971c02a3f0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.270+0000 7f9738f40640 1 -- 172.21.15.33:0/84970358 shutdown_connections 2024-09-16T00:41:24.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.270+0000 7f9738f40640 1 --2- 172.21.15.33:0/84970358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9734106f20 0x7f973410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.270+0000 7f9738f40640 1 --2- 172.21.15.33:0/84970358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97341027a0 0x7f97341067f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.270+0000 7f9738f40640 1 --2- 172.21.15.33:0/84970358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9734101df0 0x7f97341021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.270+0000 7f9738f40640 1 -- 172.21.15.33:0/84970358 >> 172.21.15.33:0/84970358 conn(0x7f97340fd8f0 msgr2=0x7f97340ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:24.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.270+0000 7f9738f40640 1 -- 172.21.15.33:0/84970358 shutdown_connections 2024-09-16T00:41:24.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.270+0000 7f9738f40640 1 -- 172.21.15.33:0/84970358 wait complete. 2024-09-16T00:41:24.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.272+0000 7f9738f40640 1 Processor -- start 2024-09-16T00:41:24.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.272+0000 7f9738f40640 1 -- start start 2024-09-16T00:41:24.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.272+0000 7f9738f40640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9734101df0 0x7f9734198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.272+0000 7f9738f40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97341027a0 0x7f9734198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.272+0000 7f9738f40640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9734106f20 0x7f973419cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.272+0000 7f9738f40640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9734112010 con 0x7f9734106f20 2024-09-16T00:41:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.272+0000 7f9738f40640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9734111e90 con 0x7f97341027a0 2024-09-16T00:41:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.272+0000 7f9738f40640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9734112190 con 0x7f9734101df0 2024-09-16T00:41:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.273+0000 7f9732d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9734106f20 0x7f973419cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.273+0000 7f973259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9734101df0 0x7f9734198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.273+0000 7f9732d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9734106f20 0x7f973419cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38238/0 (socket says 172.21.15.33:38238) 2024-09-16T00:41:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.273+0000 7f9732d9d640 1 -- 172.21.15.33:0/1691418811 learned_addr learned my addr 172.21.15.33:0/1691418811 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.273+0000 7f972bfff640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97341027a0 0x7f9734198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.273+0000 7f9732d9d640 1 -- 172.21.15.33:0/1691418811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9734101df0 msgr2=0x7f9734198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.273+0000 7f9732d9d640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9734101df0 0x7f9734198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.273+0000 7f9732d9d640 1 -- 172.21.15.33:0/1691418811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97341027a0 msgr2=0x7f9734198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.273+0000 7f9732d9d640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97341027a0 0x7f9734198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.273+0000 7f9732d9d640 1 -- 172.21.15.33:0/1691418811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973419d4c0 con 0x7f9734106f20 2024-09-16T00:41:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.274+0000 7f9732d9d640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9734106f20 0x7f973419cde0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f972c00eb10 tx=0x7f972c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.274+0000 7f972b7fe640 1 -- 172.21.15.33:0/1691418811 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972c00ce20 con 0x7f9734106f20 2024-09-16T00:41:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.274+0000 7f9738f40640 1 -- 172.21.15.33:0/1691418811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f973419d7b0 con 0x7f9734106f20 2024-09-16T00:41:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.274+0000 7f9738f40640 1 -- 172.21.15.33:0/1691418811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9734074420 con 0x7f9734106f20 2024-09-16T00:41:24.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.274+0000 7f972b7fe640 1 -- 172.21.15.33:0/1691418811 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f972c005020 con 0x7f9734106f20 2024-09-16T00:41:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.274+0000 7f972b7fe640 1 -- 172.21.15.33:0/1691418811 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972c010690 con 0x7f9734106f20 2024-09-16T00:41:24.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.277+0000 7f972b7fe640 1 -- 172.21.15.33:0/1691418811 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f972c004a40 con 0x7f9734106f20 2024-09-16T00:41:24.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.278+0000 7f972b7fe640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f97200a32c0 0x7f97200a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.278+0000 7f973259c640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f97200a32c0 0x7f97200a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.278+0000 7f972b7fe640 1 -- 172.21.15.33:0/1691418811 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f972c0d01a0 con 0x7f9734106f20 2024-09-16T00:41:24.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.279+0000 7f973259c640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f97200a32c0 0x7f97200a5780 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f972400de30 tx=0x7f972400f040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:24.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.279+0000 7f973b319740 1 Processor -- start 2024-09-16T00:41:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.279+0000 7f973b319740 1 -- start start 2024-09-16T00:41:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973b319740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 0x564856abcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973b319740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856abd080 0x564856ac7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973b319740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856ac7940 0x564856ac9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973b319740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564856a222a0 con 0x564856ac7940 2024-09-16T00:41:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973b319740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5648569c9b10 con 0x564856abc760 2024-09-16T00:41:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973b319740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564856a1e260 con 0x564856abd080 2024-09-16T00:41:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 0x564856abcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f9732d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856ac7940 0x564856ac9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 0x564856abcb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:33158/0 (socket says 172.21.15.33:33158) 2024-09-16T00:41:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973259c640 1 -- 172.21.15.33:0/4121318855 learned_addr learned my addr 172.21.15.33:0/4121318855 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:24.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f972bfff640 1 --2- 172.21.15.33:0/4121318855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856abd080 0x564856ac7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973259c640 1 -- 172.21.15.33:0/4121318855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856abd080 msgr2=0x564856ac7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:24.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973259c640 1 --2- 172.21.15.33:0/4121318855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856abd080 0x564856ac7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973259c640 1 -- 172.21.15.33:0/4121318855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856ac7940 msgr2=0x564856ac9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:24.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973259c640 1 --2- 172.21.15.33:0/4121318855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856ac7940 0x564856ac9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.280+0000 7f973259c640 1 -- 172.21.15.33:0/4121318855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564856793b90 con 0x564856abc760 2024-09-16T00:41:24.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.281+0000 7f973259c640 1 --2- 172.21.15.33:0/4121318855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 0x564856abcb40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9724012ef0 tx=0x7f9724015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50002216317c7504 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:24.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.281+0000 7f9728ff9640 1 -- 172.21.15.33:0/4121318855 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972401f040 con 0x564856abc760 2024-09-16T00:41:24.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.281+0000 7f9728ff9640 1 -- 172.21.15.33:0/4121318855 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9724028070 con 0x564856abc760 2024-09-16T00:41:24.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.281+0000 7f9728ff9640 1 -- 172.21.15.33:0/4121318855 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9724017810 con 0x564856abc760 2024-09-16T00:41:24.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.282+0000 7f973b319740 1 -- 172.21.15.33:0/4121318855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 msgr2=0x564856abcb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:24.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.282+0000 7f973b319740 1 --2- 172.21.15.33:0/4121318855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 0x564856abcb40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9724012ef0 tx=0x7f9724015000 comp rx=0 tx=0).stop 2024-09-16T00:41:24.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.282+0000 7f973b319740 1 -- 172.21.15.33:0/4121318855 shutdown_connections 2024-09-16T00:41:24.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.282+0000 7f973b319740 1 --2- 172.21.15.33:0/4121318855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856ac7940 0x564856ac9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.282+0000 7f973b319740 1 --2- 172.21.15.33:0/4121318855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856abd080 0x564856ac7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.282+0000 7f973b319740 1 --2- 172.21.15.33:0/4121318855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 0x564856abcb40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.282+0000 7f973b319740 1 -- 172.21.15.33:0/4121318855 >> 172.21.15.33:0/4121318855 conn(0x564856ab8cf0 msgr2=0x564856ab90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:24.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.282+0000 7f973b319740 1 -- 172.21.15.33:0/4121318855 shutdown_connections 2024-09-16T00:41:24.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.282+0000 7f973b319740 1 -- 172.21.15.33:0/4121318855 wait complete. 2024-09-16T00:41:24.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f973b319740 1 -- start start 2024-09-16T00:41:24.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f973b319740 1 Processor -- start 2024-09-16T00:41:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f973b319740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abc760 0x564856ac4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f973b319740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abd080 0x564856ac4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f973b319740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856ac7940 0x564856ac51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f973b319740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564856aca510 con 0x564856abc760 2024-09-16T00:41:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f973b319740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5648569aa6e0 con 0x564856abd080 2024-09-16T00:41:24.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f973b319740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564856aca690 con 0x564856ac7940 2024-09-16T00:41:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f973b319740 10 client.0 _ll_register_callbacks cb 0x5648569cea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f973259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abc760 0x564856ac4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f9732d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856ac7940 0x564856ac51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f972bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abd080 0x564856ac4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f973259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abc760 0x564856ac4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38254/0 (socket says 172.21.15.33:38254) 2024-09-16T00:41:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.283+0000 7f973259c640 1 -- 172.21.15.33:0/1311890292 learned_addr learned my addr 172.21.15.33:0/1311890292 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f9732d9d640 1 -- 172.21.15.33:0/1311890292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abd080 msgr2=0x564856ac4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f9732d9d640 1 --2- 172.21.15.33:0/1311890292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abd080 0x564856ac4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f9732d9d640 1 -- 172.21.15.33:0/1311890292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abc760 msgr2=0x564856ac4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f9732d9d640 1 --2- 172.21.15.33:0/1311890292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abc760 0x564856ac4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f9732d9d640 1 -- 172.21.15.33:0/1311890292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564856ad39b0 con 0x564856ac7940 2024-09-16T00:41:24.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f973259c640 1 --2- 172.21.15.33:0/1311890292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abc760 0x564856ac4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f9732d9d640 1 --2- 172.21.15.33:0/1311890292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856ac7940 0x564856ac51e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f972c005a50 tx=0x7f972c09c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=842540e642fcc793 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f9719ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f9719ffb640 1 -- 172.21.15.33:0/1311890292 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972c0a4b20 con 0x564856ac7940 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f973b319740 1 -- 172.21.15.33:0/1311890292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9724028070 con 0x564856ac7940 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f9719ffb640 1 -- 172.21.15.33:0/1311890292 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f972c0a4cc0 con 0x564856ac7940 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f9719ffb640 1 -- 172.21.15.33:0/1311890292 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f972c0b5200 con 0x564856ac7940 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f973b319740 10 client.28984 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.284+0000 7f973b319740 1 -- 172.21.15.33:0/1311890292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564856ad6340 con 0x564856ac7940 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.285+0000 7f9719ffb640 1 -- 172.21.15.33:0/1311890292 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9724028070 con 0x564856ac7940 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.285+0000 7f9719ffb640 1 -- 172.21.15.33:0/1311890292 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f972c0cdce0 con 0x564856ac7940 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.285+0000 7f973b319740 10 client.28984 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.285+0000 7f973b319740 10 client.28984 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:41:24.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.285+0000 7f973b319740 20 client.28984 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.285+0000 7f973b319740 1 --2- 172.21.15.33:0/1311890292 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564856ad7750 0x564856ad9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.285+0000 7f973b319740 4 client.28984 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:41:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.285+0000 7f973b319740 1 -- 172.21.15.33:0/1311890292 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564856a2f9d0 con 0x564856ad7750 2024-09-16T00:41:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.285+0000 7f973259c640 1 --2- 172.21.15.33:0/1311890292 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564856ad7750 0x564856ad9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.286+0000 7f9719ffb640 10 client.28984 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.286+0000 7f973259c640 1 --2- 172.21.15.33:0/1311890292 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564856ad7750 0x564856ad9b60 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b3762f335a59275 server_cookie=66e13731fc5a3c3a in_seq=0 out_seq=0 2024-09-16T00:41:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.286+0000 7f9719ffb640 1 -- 172.21.15.33:0/1311890292 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564856a2f9d0 con 0x564856ad7750 2024-09-16T00:41:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.286+0000 7f9719ffb640 10 client.28984 handle_command_reply: tid=0 2024-09-16T00:41:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f973b319740 1 client.28984 shutdown 2024-09-16T00:41:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f973b319740 1 -- 172.21.15.33:0/1311890292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856ac7940 msgr2=0x564856ac51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f973b319740 1 --2- 172.21.15.33:0/1311890292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856ac7940 0x564856ac51e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f972c005a50 tx=0x7f972c09c400 comp rx=0 tx=0).stop 2024-09-16T00:41:24.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f973b319740 1 -- 172.21.15.33:0/1311890292 shutdown_connections 2024-09-16T00:41:24.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f973b319740 1 --2- 172.21.15.33:0/1311890292 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564856ad7750 0x564856ad9b60 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f973b319740 1 --2- 172.21.15.33:0/1311890292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856ac7940 0x564856ac51e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f973b319740 1 --2- 172.21.15.33:0/1311890292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abd080 0x564856ac4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f973b319740 1 --2- 172.21.15.33:0/1311890292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abc760 0x564856ac4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f973b319740 1 -- 172.21.15.33:0/1311890292 >> 172.21.15.33:0/1311890292 conn(0x564856ab8cf0 msgr2=0x564856ab7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:24.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f9719ffb640 0 client.28984 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:24.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f973b319740 1 -- 172.21.15.33:0/1311890292 shutdown_connections 2024-09-16T00:41:24.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f973b319740 1 -- 172.21.15.33:0/1311890292 wait complete. 2024-09-16T00:41:24.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.287+0000 7f973b319740 20 client.28984 trim_cache size 0 max 16384 2024-09-16T00:41:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.309+0000 7f973b319740 1 Processor -- start 2024-09-16T00:41:24.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.309+0000 7f973b319740 1 -- start start 2024-09-16T00:41:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.309+0000 7f973b319740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856abc760 0x564856abfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.309+0000 7f973b319740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 0x564856ac36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.309+0000 7f973b319740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856ac7940 0x564856adf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.309+0000 7f973b319740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9714012860 con 0x564856abd080 2024-09-16T00:41:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.309+0000 7f973b319740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9714010130 con 0x564856ac7940 2024-09-16T00:41:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.309+0000 7f973b319740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f971400df80 con 0x564856abc760 2024-09-16T00:41:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f972bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 0x564856ac36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f972bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 0x564856ac36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38256/0 (socket says 172.21.15.33:38256) 2024-09-16T00:41:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f972bfff640 1 -- 172.21.15.33:0/2258614368 learned_addr learned my addr 172.21.15.33:0/2258614368 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:24.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f9732d9d640 1 --2- 172.21.15.33:0/2258614368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856ac7940 0x564856adf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f972bfff640 1 -- 172.21.15.33:0/2258614368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856abc760 msgr2=0x564856abfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f972bfff640 1 --2- 172.21.15.33:0/2258614368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856abc760 0x564856abfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f972bfff640 1 -- 172.21.15.33:0/2258614368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856ac7940 msgr2=0x564856adf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f972bfff640 1 --2- 172.21.15.33:0/2258614368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856ac7940 0x564856adf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f972bfff640 1 -- 172.21.15.33:0/2258614368 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564856ab8b20 con 0x564856abd080 2024-09-16T00:41:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f9732d9d640 1 --2- 172.21.15.33:0/2258614368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856ac7940 0x564856adf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:41:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f972bfff640 1 --2- 172.21.15.33:0/2258614368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 0x564856ac36a0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f971c009230 tx=0x7f971c0044a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b85c39957c22f003 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f971bfff640 1 -- 172.21.15.33:0/2258614368 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c035040 con 0x564856abd080 2024-09-16T00:41:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f971bfff640 1 -- 172.21.15.33:0/2258614368 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f971c0046c0 con 0x564856abd080 2024-09-16T00:41:24.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.310+0000 7f971bfff640 1 -- 172.21.15.33:0/2258614368 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c0039e0 con 0x564856abd080 2024-09-16T00:41:24.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.311+0000 7f973b319740 1 -- 172.21.15.33:0/2258614368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 msgr2=0x564856ac36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:24.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.311+0000 7f973b319740 1 --2- 172.21.15.33:0/2258614368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 0x564856ac36a0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f971c009230 tx=0x7f971c0044a0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.312+0000 7f973b319740 1 -- 172.21.15.33:0/2258614368 shutdown_connections 2024-09-16T00:41:24.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.312+0000 7f973b319740 1 --2- 172.21.15.33:0/2258614368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856ac7940 0x564856adf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.312+0000 7f973b319740 1 --2- 172.21.15.33:0/2258614368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 0x564856ac36a0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.312+0000 7f973b319740 1 --2- 172.21.15.33:0/2258614368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856abc760 0x564856abfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.312+0000 7f973b319740 1 -- 172.21.15.33:0/2258614368 >> 172.21.15.33:0/2258614368 conn(0x564856ab8cf0 msgr2=0x564856ac67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:24.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.312+0000 7f973b319740 1 -- 172.21.15.33:0/2258614368 shutdown_connections 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.312+0000 7f973b319740 1 -- 172.21.15.33:0/2258614368 wait complete. 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.312+0000 7f973b319740 1 -- start start 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f973b319740 1 Processor -- start 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f973b319740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 0x564856acf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f973b319740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 0x564856acfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f973b319740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856ac7940 0x564856acc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f973b319740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f971400e5f0 con 0x564856abd080 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f973b319740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f971400fa60 con 0x564856abc760 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f973b319740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f971400ecc0 con 0x564856ac7940 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f973b319740 10 client.0 _ll_register_callbacks cb 0x7f972c09ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f972bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 0x564856acfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f9732d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856ac7940 0x564856acc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f972bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 0x564856acfe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38268/0 (socket says 172.21.15.33:38268) 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f973259c640 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 0x564856acf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f972bfff640 1 -- 172.21.15.33:0/1913665993 learned_addr learned my addr 172.21.15.33:0/1913665993 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f972bfff640 1 -- 172.21.15.33:0/1913665993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856ac7940 msgr2=0x564856acc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f972bfff640 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856ac7940 0x564856acc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f972bfff640 1 -- 172.21.15.33:0/1913665993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 msgr2=0x564856acf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f972bfff640 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 0x564856acf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f972bfff640 1 -- 172.21.15.33:0/1913665993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564856ab4260 con 0x564856abd080 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.313+0000 7f973259c640 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 0x564856acf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.314+0000 7f972bfff640 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 0x564856acfe50 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f971c031550 tx=0x7f971c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1cbb30e61859561 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.314+0000 7f9719ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.314+0000 7f9719ffb640 1 -- 172.21.15.33:0/1913665993 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c035040 con 0x564856abd080 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.314+0000 7f9719ffb640 1 -- 172.21.15.33:0/1913665993 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f971c005510 con 0x564856abd080 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.314+0000 7f973b319740 1 -- 172.21.15.33:0/1913665993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f971c0046c0 con 0x564856abd080 2024-09-16T00:41:24.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.314+0000 7f9719ffb640 1 -- 172.21.15.33:0/1913665993 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f971c059050 con 0x564856abd080 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.314+0000 7f973b319740 10 client.19863 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.314+0000 7f973b319740 1 -- 172.21.15.33:0/1913665993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564856ab9da0 con 0x564856abd080 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.315+0000 7f9719ffb640 1 -- 172.21.15.33:0/1913665993 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f971c050070 con 0x564856abd080 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.315+0000 7f9719ffb640 1 -- 172.21.15.33:0/1913665993 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f971c045020 con 0x564856abd080 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.315+0000 7f973b319740 10 client.19863 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.315+0000 7f973b319740 10 client.19863 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.315+0000 7f973b319740 20 client.19863 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.315+0000 7f973b319740 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564856b4b810 0x564856b4dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.315+0000 7f973b319740 4 client.19863 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.2, "await_for": 45.0}] 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.315+0000 7f973b319740 1 -- 172.21.15.33:0/1913665993 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.2, "await_for": 45.0}) -- 0x7f971c0046c0 con 0x564856b4b810 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.316+0000 7f973259c640 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564856b4b810 0x564856b4dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.316+0000 7f973259c640 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564856b4b810 0x564856b4dc30 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=422cabc87cd28bef server_cookie=ba431bccaea2bc82 in_seq=0 out_seq=0 2024-09-16T00:41:24.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.316+0000 7f9719ffb640 10 client.19863 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:24.367 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.896+0000 7f899be2b640 1 -- 172.21.15.33:0/618172255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994101430 msgr2=0x7f8994105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.896+0000 7f899be2b640 1 --2- 172.21.15.33:0/618172255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994101430 0x7f8994105900 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f8990009900 tx=0x7f899002a470 comp rx=0 tx=0).stop 2024-09-16T00:41:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.896+0000 7f899be2b640 1 -- 172.21.15.33:0/618172255 shutdown_connections 2024-09-16T00:41:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.896+0000 7f899be2b640 1 --2- 172.21.15.33:0/618172255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8994105f20 0x7f89941063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.896+0000 7f899be2b640 1 --2- 172.21.15.33:0/618172255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994101430 0x7f8994105900 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.896+0000 7f899be2b640 1 --2- 172.21.15.33:0/618172255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8994100b10 0x7f8994100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.896+0000 7f899be2b640 1 -- 172.21.15.33:0/618172255 >> 172.21.15.33:0/618172255 conn(0x7f89940fc780 msgr2=0x7f89940feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.896+0000 7f899be2b640 1 -- 172.21.15.33:0/618172255 shutdown_connections 2024-09-16T00:41:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.896+0000 7f899be2b640 1 -- 172.21.15.33:0/618172255 wait complete. 2024-09-16T00:41:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.896+0000 7f899be2b640 1 Processor -- start 2024-09-16T00:41:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f899be2b640 1 -- start start 2024-09-16T00:41:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f899be2b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994100b10 0x7f89941a4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f899be2b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8994101430 0x7f89941a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f899be2b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8994105f20 0x7f899419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f899be2b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8994118520 con 0x7f8994100b10 2024-09-16T00:41:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f899be2b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f89941183a0 con 0x7f8994101430 2024-09-16T00:41:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f899be2b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89941186a0 con 0x7f8994105f20 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f8999bc7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994100b10 0x7f89941a4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f8999bc7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994100b10 0x7f89941a4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38280/0 (socket says 172.21.15.33:38280) 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f8999bc7640 1 -- 172.21.15.33:0/2910381560 learned_addr learned my addr 172.21.15.33:0/2910381560 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f89993c6640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8994101430 0x7f89941a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f899a3c8640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8994105f20 0x7f899419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f8999bc7640 1 -- 172.21.15.33:0/2910381560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8994105f20 msgr2=0x7f899419e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f8999bc7640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8994105f20 0x7f899419e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f8999bc7640 1 -- 172.21.15.33:0/2910381560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8994101430 msgr2=0x7f89941a4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f8999bc7640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8994101430 0x7f89941a4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f8999bc7640 1 -- 172.21.15.33:0/2910381560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f899419f020 con 0x7f8994100b10 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.897+0000 7f8999bc7640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994100b10 0x7f89941a4730 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f898400e9f0 tx=0x7f898400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.898+0000 7f898affd640 1 -- 172.21.15.33:0/2910381560 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898400cd80 con 0x7f8994100b10 2024-09-16T00:41:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.898+0000 7f899be2b640 1 -- 172.21.15.33:0/2910381560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f899419f310 con 0x7f8994100b10 2024-09-16T00:41:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.898+0000 7f898affd640 1 -- 172.21.15.33:0/2910381560 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8984010910 con 0x7f8994100b10 2024-09-16T00:41:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.898+0000 7f898affd640 1 -- 172.21.15.33:0/2910381560 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89840188f0 con 0x7f8994100b10 2024-09-16T00:41:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.898+0000 7f899be2b640 1 -- 172.21.15.33:0/2910381560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89941ab600 con 0x7f8994100b10 2024-09-16T00:41:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.899+0000 7f899be2b640 1 -- 172.21.15.33:0/2910381560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8994105900 con 0x7f8994100b10 2024-09-16T00:41:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.900+0000 7f898affd640 1 -- 172.21.15.33:0/2910381560 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8984010ac0 con 0x7f8994100b10 2024-09-16T00:41:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.900+0000 7f898affd640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89800a3240 0x7f89800a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.901+0000 7f89993c6640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89800a3240 0x7f89800a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.901+0000 7f898affd640 1 -- 172.21.15.33:0/2910381560 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f89840d0520 con 0x7f8994100b10 2024-09-16T00:41:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.901+0000 7f89993c6640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89800a3240 0x7f89800a5700 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f8990009230 tx=0x7f89900058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:24.904+0000 7f898affd640 1 -- 172.21.15.33:0/2910381560 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f898409c770 con 0x7f8994100b10 2024-09-16T00:41:25.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.69 MB/sec execute 511 sec latency 770.789 ms 2024-09-16T00:41:25.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.157+0000 7f899be2b640 1 -- 172.21.15.33:0/2910381560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89941ab7d0 con 0x7f8994100b10 2024-09-16T00:41:25.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.158+0000 7f898affd640 1 -- 172.21.15.33:0/2910381560 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f89840a0a70 con 0x7f8994100b10 2024-09-16T00:41:25.160 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:41:25.160 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:41:25.160 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:41:25.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.160+0000 7f899be2b640 1 -- 172.21.15.33:0/2910381560 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89800a3240 msgr2=0x7f89800a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:25.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.160+0000 7f899be2b640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89800a3240 0x7f89800a5700 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f8990009230 tx=0x7f89900058d0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.160+0000 7f899be2b640 1 -- 172.21.15.33:0/2910381560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994100b10 msgr2=0x7f89941a4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:25.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.160+0000 7f899be2b640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994100b10 0x7f89941a4730 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f898400e9f0 tx=0x7f898400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.160+0000 7f899be2b640 1 -- 172.21.15.33:0/2910381560 shutdown_connections 2024-09-16T00:41:25.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.160+0000 7f899be2b640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89800a3240 0x7f89800a5700 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.160+0000 7f899be2b640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8994105f20 0x7f899419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.160+0000 7f899be2b640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8994101430 0x7f89941a4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.160+0000 7f899be2b640 1 --2- 172.21.15.33:0/2910381560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8994100b10 0x7f89941a4730 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.160+0000 7f899be2b640 1 -- 172.21.15.33:0/2910381560 >> 172.21.15.33:0/2910381560 conn(0x7f89940fc780 msgr2=0x7f899410f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:25.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.160+0000 7f899be2b640 1 -- 172.21.15.33:0/2910381560 shutdown_connections 2024-09-16T00:41:25.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.160+0000 7f899be2b640 1 -- 172.21.15.33:0/2910381560 wait complete. 2024-09-16T00:41:25.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f9719ffb640 1 -- 172.21.15.33:0/1913665993 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f971c0046c0 con 0x564856b4b810 2024-09-16T00:41:25.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f9719ffb640 10 client.19863 handle_command_reply: tid=0 2024-09-16T00:41:25.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f973b319740 1 client.19863 shutdown 2024-09-16T00:41:25.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f973b319740 1 -- 172.21.15.33:0/1913665993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 msgr2=0x564856acfe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:25.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f973b319740 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 0x564856acfe50 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f971c031550 tx=0x7f971c009260 comp rx=0 tx=0).stop 2024-09-16T00:41:25.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f973b319740 1 -- 172.21.15.33:0/1913665993 shutdown_connections 2024-09-16T00:41:25.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f973b319740 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x564856b4b810 0x564856b4dc30 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f973b319740 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564856ac7940 0x564856acc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f973b319740 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564856abd080 0x564856acfe50 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f9719ffb640 0 client.19863 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f973b319740 1 --2- 172.21.15.33:0/1913665993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564856abc760 0x564856acf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f973b319740 1 -- 172.21.15.33:0/1913665993 >> 172.21.15.33:0/1913665993 conn(0x564856ab8cf0 msgr2=0x564856abe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f973b319740 1 -- 172.21.15.33:0/1913665993 shutdown_connections 2024-09-16T00:41:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f973b319740 1 -- 172.21.15.33:0/1913665993 wait complete. 2024-09-16T00:41:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.300+0000 7f973b319740 20 client.19863 trim_cache size 0 max 16384 2024-09-16T00:41:25.303 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:41:25.303 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:41:25.303 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:41:25.303 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 82, 2024-09-16T00:41:25.303 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: "987c2a04": { 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 82, 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 147.2, 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:41:25.305 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:41:25.305 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:41:25.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:25.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:25.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:25.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:25.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:25.305 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:41:25.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.301+0000 7f9738f40640 1 -- 172.21.15.33:0/1691418811 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f97200a32c0 msgr2=0x7f97200a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:25.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.301+0000 7f9738f40640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f97200a32c0 0x7f97200a5780 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f972400de30 tx=0x7f972400f040 comp rx=0 tx=0).stop 2024-09-16T00:41:25.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.301+0000 7f9738f40640 1 -- 172.21.15.33:0/1691418811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9734106f20 msgr2=0x7f973419cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:25.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.301+0000 7f9738f40640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9734106f20 0x7f973419cde0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f972c00eb10 tx=0x7f972c00ef10 comp rx=0 tx=0).stop 2024-09-16T00:41:25.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.301+0000 7f9738f40640 1 -- 172.21.15.33:0/1691418811 shutdown_connections 2024-09-16T00:41:25.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.301+0000 7f9738f40640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f97200a32c0 0x7f97200a5780 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.301+0000 7f9738f40640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9734106f20 0x7f973419cde0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.301+0000 7f9738f40640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97341027a0 0x7f9734198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.301+0000 7f9738f40640 1 --2- 172.21.15.33:0/1691418811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9734101df0 0x7f9734198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:25.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.301+0000 7f9738f40640 1 -- 172.21.15.33:0/1691418811 >> 172.21.15.33:0/1691418811 conn(0x7f97340fd8f0 msgr2=0x7f97340fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:25.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.301+0000 7f9738f40640 1 -- 172.21.15.33:0/1691418811 shutdown_connections 2024-09-16T00:41:25.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:25.301+0000 7f9738f40640 1 -- 172.21.15.33:0/1691418811 wait complete. 2024-09-16T00:41:26.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:25 smithi033 ceph-mon[31243]: pgmap v1078: 129 pgs: 129 active+clean; 745 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 12 MiB/s wr, 577 op/s 2024-09-16T00:41:26.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:25 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2910381560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:41:26.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.66 MB/sec execute 512 sec latency 1770.864 ms 2024-09-16T00:41:26.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:25 smithi083 ceph-mon[35242]: pgmap v1078: 129 pgs: 129 active+clean; 745 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 12 MiB/s wr, 577 op/s 2024-09-16T00:41:26.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:25 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2910381560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:41:26.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:25 smithi060 ceph-mon[35185]: pgmap v1078: 129 pgs: 129 active+clean; 745 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 12 MiB/s wr, 577 op/s 2024-09-16T00:41:26.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:25 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2910381560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:41:26.565 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:41:27.103 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.62 MB/sec execute 513 sec latency 2770.899 ms 2024-09-16T00:41:27.230 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '987c2a04', quiesce took 8.3 seconds. Will release after: 18.9 2024-09-16T00:41:27.267 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:27.832 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:27 smithi033 ceph-mon[31243]: pgmap v1079: 129 pgs: 129 active+clean; 729 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 559 op/s 2024-09-16T00:41:28.104 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.59 MB/sec execute 514 sec latency 3770.950 ms 2024-09-16T00:41:28.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:27 smithi083 ceph-mon[35242]: pgmap v1079: 129 pgs: 129 active+clean; 729 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 559 op/s 2024-09-16T00:41:28.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:27 smithi060 ceph-mon[35185]: pgmap v1079: 129 pgs: 129 active+clean; 729 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 559 op/s 2024-09-16T00:41:28.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.239+0000 7f0d7dfba640 1 -- 172.21.15.33:0/3579664726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d78101de0 msgr2=0x7f0d781021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:28.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.239+0000 7f0d7dfba640 1 --2- 172.21.15.33:0/3579664726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d78101de0 0x7f0d781021c0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009900 tx=0x7f0d6802a410 comp rx=0 tx=0).stop 2024-09-16T00:41:28.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.239+0000 7f0d7dfba640 1 -- 172.21.15.33:0/3579664726 shutdown_connections 2024-09-16T00:41:28.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.239+0000 7f0d7dfba640 1 --2- 172.21.15.33:0/3579664726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d78106f10 0x7f0d7810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.239+0000 7f0d7dfba640 1 --2- 172.21.15.33:0/3579664726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d78102790 0x7f0d781067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.239+0000 7f0d7dfba640 1 --2- 172.21.15.33:0/3579664726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d78101de0 0x7f0d781021c0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.239+0000 7f0d7dfba640 1 -- 172.21.15.33:0/3579664726 >> 172.21.15.33:0/3579664726 conn(0x7f0d780fd8c0 msgr2=0x7f0d780ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:28.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.239+0000 7f0d7dfba640 1 -- 172.21.15.33:0/3579664726 shutdown_connections 2024-09-16T00:41:28.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.239+0000 7f0d7dfba640 1 -- 172.21.15.33:0/3579664726 wait complete. 2024-09-16T00:41:28.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.240+0000 7f0d7dfba640 1 Processor -- start 2024-09-16T00:41:28.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.240+0000 7f0d7dfba640 1 -- start start 2024-09-16T00:41:28.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.240+0000 7f0d7dfba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d78101de0 0x7f0d781984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.240+0000 7f0d7dfba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d78102790 0x7f0d781989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.240+0000 7f0d7dfba640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d78106f10 0x7f0d7819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.240+0000 7f0d7dfba640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d78111f90 con 0x7f0d78106f10 2024-09-16T00:41:28.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.240+0000 7f0d7dfba640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0d78111e10 con 0x7f0d78101de0 2024-09-16T00:41:28.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.240+0000 7f0d7dfba640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0d78112110 con 0x7f0d78102790 2024-09-16T00:41:28.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.240+0000 7f0d77fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d78106f10 0x7f0d7819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.240+0000 7f0d77fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d78106f10 0x7f0d7819cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38298/0 (socket says 172.21.15.33:38298) 2024-09-16T00:41:28.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.240+0000 7f0d77fff640 1 -- 172.21.15.33:0/2899506837 learned_addr learned my addr 172.21.15.33:0/2899506837 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:28.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d78102790 0x7f0d781989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d777fe640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d78101de0 0x7f0d781984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d77fff640 1 -- 172.21.15.33:0/2899506837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d78102790 msgr2=0x7f0d781989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:28.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d77fff640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d78102790 0x7f0d781989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d77fff640 1 -- 172.21.15.33:0/2899506837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d78101de0 msgr2=0x7f0d781984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:28.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d77fff640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d78101de0 0x7f0d781984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d77fff640 1 -- 172.21.15.33:0/2899506837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7819d460 con 0x7f0d78106f10 2024-09-16T00:41:28.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d777fe640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d78101de0 0x7f0d781984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:41:28.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d77fff640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d78106f10 0x7f0d7819cd80 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000cce0 tx=0x7f0d7000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:28.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d74ff9640 1 -- 172.21.15.33:0/2899506837 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7000f450 con 0x7f0d78106f10 2024-09-16T00:41:28.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d74ff9640 1 -- 172.21.15.33:0/2899506837 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d7000fad0 con 0x7f0d78106f10 2024-09-16T00:41:28.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d7dfba640 1 -- 172.21.15.33:0/2899506837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d7819d750 con 0x7f0d78106f10 2024-09-16T00:41:28.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d74ff9640 1 -- 172.21.15.33:0/2899506837 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d70018b60 con 0x7f0d78106f10 2024-09-16T00:41:28.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.241+0000 7f0d7dfba640 1 -- 172.21.15.33:0/2899506837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d781a5030 con 0x7f0d78106f10 2024-09-16T00:41:28.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.244+0000 7f0d74ff9640 1 -- 172.21.15.33:0/2899506837 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0d7000fc80 con 0x7f0d78106f10 2024-09-16T00:41:28.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.244+0000 7f0d80393740 1 Processor -- start 2024-09-16T00:41:28.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.244+0000 7f0d80393740 1 -- start start 2024-09-16T00:41:28.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.244+0000 7f0d80393740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefeb40 0x55a4bfefd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.244+0000 7f0d80393740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefd5d0 0x55a4bfefda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.244+0000 7f0d80393740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bff08370 0x55a4bff0a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.244+0000 7f0d80393740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a4bfe632f0 con 0x55a4bfefd5d0 2024-09-16T00:41:28.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.244+0000 7f0d80393740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a4bfe0acf0 con 0x55a4bff08370 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.244+0000 7f0d80393740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a4bfe5f2b0 con 0x55a4bfefeb40 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.244+0000 7f0d777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefd5d0 0x55a4bfefda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.244+0000 7f0d74ff9640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d600a3240 0x7f0d600a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefd5d0 0x55a4bfefda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38304/0 (socket says 172.21.15.33:38304) 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d777fe640 1 -- 172.21.15.33:0/2419137554 learned_addr learned my addr 172.21.15.33:0/2419137554 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/2419137554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bff08370 0x55a4bff0a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/2419137554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefeb40 0x55a4bfefd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d77fff640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d600a3240 0x7f0d600a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d777fe640 1 -- 172.21.15.33:0/2419137554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefeb40 msgr2=0x55a4bfefd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d777fe640 1 --2- 172.21.15.33:0/2419137554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefeb40 0x55a4bfefd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d777fe640 1 -- 172.21.15.33:0/2419137554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bff08370 msgr2=0x55a4bff0a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d777fe640 1 --2- 172.21.15.33:0/2419137554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bff08370 0x55a4bff0a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d777fe640 1 -- 172.21.15.33:0/2419137554 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4bfdef3a0 con 0x55a4bfefd5d0 2024-09-16T00:41:28.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d74ff9640 1 -- 172.21.15.33:0/2899506837 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0d700d02c0 con 0x7f0d78106f10 2024-09-16T00:41:28.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d77fff640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d600a3240 0x7f0d600a5700 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0d700899a0 tx=0x7f0d700980e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:28.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.245+0000 7f0d777fe640 1 --2- 172.21.15.33:0/2419137554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefd5d0 0x55a4bfefda10 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009900 tx=0x7f0d680316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de46d44e7f44209 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:28.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.246+0000 7f0d5dffb640 1 -- 172.21.15.33:0/2419137554 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6803f070 con 0x55a4bfefd5d0 2024-09-16T00:41:28.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.246+0000 7f0d5dffb640 1 -- 172.21.15.33:0/2419137554 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d680035f0 con 0x55a4bfefd5d0 2024-09-16T00:41:28.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.246+0000 7f0d5dffb640 1 -- 172.21.15.33:0/2419137554 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68035070 con 0x55a4bfefd5d0 2024-09-16T00:41:28.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.246+0000 7f0d80393740 1 -- 172.21.15.33:0/2419137554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefd5d0 msgr2=0x55a4bfefda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:28.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.246+0000 7f0d80393740 1 --2- 172.21.15.33:0/2419137554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefd5d0 0x55a4bfefda10 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009900 tx=0x7f0d680316e0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.246+0000 7f0d80393740 1 -- 172.21.15.33:0/2419137554 shutdown_connections 2024-09-16T00:41:28.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.246+0000 7f0d80393740 1 --2- 172.21.15.33:0/2419137554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bff08370 0x55a4bff0a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.246+0000 7f0d80393740 1 --2- 172.21.15.33:0/2419137554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefd5d0 0x55a4bfefda10 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.246+0000 7f0d80393740 1 --2- 172.21.15.33:0/2419137554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefeb40 0x55a4bfefd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.246+0000 7f0d80393740 1 -- 172.21.15.33:0/2419137554 >> 172.21.15.33:0/2419137554 conn(0x55a4bfec6310 msgr2=0x55a4bfef9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.247+0000 7f0d80393740 1 -- 172.21.15.33:0/2419137554 shutdown_connections 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.247+0000 7f0d80393740 1 -- 172.21.15.33:0/2419137554 wait complete. 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.247+0000 7f0d80393740 1 -- start start 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.247+0000 7f0d80393740 1 Processor -- start 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.247+0000 7f0d80393740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefd5d0 0x55a4bff07d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.247+0000 7f0d80393740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefeb40 0x55a4bff05030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.247+0000 7f0d80393740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 0x55a4bff05640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.247+0000 7f0d80393740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a4bff0b010 con 0x55a4bff08370 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.247+0000 7f0d80393740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55a4bfe1e060 con 0x55a4bfefeb40 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.247+0000 7f0d80393740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a4bff0b190 con 0x55a4bfefd5d0 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d80393740 10 client.0 _ll_register_callbacks cb 0x55a4bfb718c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d76ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 0x55a4bff05640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d76ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 0x55a4bff05640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38310/0 (socket says 172.21.15.33:38310) 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d76ffd640 1 -- 172.21.15.33:0/3939750822 learned_addr learned my addr 172.21.15.33:0/3939750822 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d777fe640 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefeb40 0x55a4bff05030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefd5d0 0x55a4bff07d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d76ffd640 1 -- 172.21.15.33:0/3939750822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefd5d0 msgr2=0x55a4bff07d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefd5d0 0x55a4bff07d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d76ffd640 1 -- 172.21.15.33:0/3939750822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefeb40 msgr2=0x55a4bff05030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefeb40 0x55a4bff05030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d76ffd640 1 -- 172.21.15.33:0/3939750822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4bfbd43c0 con 0x55a4bff08370 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 0x55a4bff05640 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c019b20 tx=0x7f0d6c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d15bf398924129b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d56ffd640 1 -- 172.21.15.33:0/3939750822 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c013c90 con 0x55a4bff08370 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d56ffd640 1 -- 172.21.15.33:0/3939750822 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0d6c00f040 con 0x55a4bff08370 2024-09-16T00:41:28.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.248+0000 7f0d80393740 1 -- 172.21.15.33:0/3939750822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d680035f0 con 0x55a4bff08370 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.249+0000 7f0d56ffd640 1 -- 172.21.15.33:0/3939750822 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0d6c02b050 con 0x55a4bff08370 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.249+0000 7f0d80393740 10 client.19881 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.249+0000 7f0d80393740 1 -- 172.21.15.33:0/3939750822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a4bff16b40 con 0x55a4bff08370 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.249+0000 7f0d56ffd640 1 -- 172.21.15.33:0/3939750822 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d680035f0 con 0x55a4bff08370 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.249+0000 7f0d56ffd640 1 -- 172.21.15.33:0/3939750822 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0d6c0163d0 con 0x55a4bff08370 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.249+0000 7f0d80393740 10 client.19881 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.249+0000 7f0d80393740 10 client.19881 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.249+0000 7f0d80393740 20 client.19881 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.249+0000 7f0d80393740 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a4bff17f50 0x55a4bff1a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.249+0000 7f0d80393740 4 client.19881 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.249+0000 7f0d80393740 1 -- 172.21.15.33:0/3939750822 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4bfe70860 con 0x55a4bff17f50 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.249+0000 7f0d777fe640 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a4bff17f50 0x55a4bff1a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.250+0000 7f0d777fe640 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a4bff17f50 0x55a4bff1a340 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7c49aec421bcda09 server_cookie=14de0da8292d419e in_seq=0 out_seq=0 2024-09-16T00:41:28.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.250+0000 7f0d56ffd640 10 client.19881 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:41:28.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d56ffd640 1 -- 172.21.15.33:0/3939750822 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a4bfe70860 con 0x55a4bff17f50 2024-09-16T00:41:28.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d56ffd640 10 client.19881 handle_command_reply: tid=0 2024-09-16T00:41:28.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d80393740 1 client.19881 shutdown 2024-09-16T00:41:28.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d80393740 1 -- 172.21.15.33:0/3939750822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 msgr2=0x55a4bff05640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:28.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d80393740 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 0x55a4bff05640 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c019b20 tx=0x7f0d6c008040 comp rx=0 tx=0).stop 2024-09-16T00:41:28.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d80393740 1 -- 172.21.15.33:0/3939750822 shutdown_connections 2024-09-16T00:41:28.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d80393740 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a4bff17f50 0x55a4bff1a340 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d80393740 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 0x55a4bff05640 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d80393740 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefeb40 0x55a4bff05030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d80393740 1 --2- 172.21.15.33:0/3939750822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefd5d0 0x55a4bff07d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d80393740 1 -- 172.21.15.33:0/3939750822 >> 172.21.15.33:0/3939750822 conn(0x55a4bfec6310 msgr2=0x55a4bfeff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:28.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d56ffd640 0 client.19881 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:41:28.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d80393740 1 -- 172.21.15.33:0/3939750822 shutdown_connections 2024-09-16T00:41:28.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d80393740 1 -- 172.21.15.33:0/3939750822 wait complete. 2024-09-16T00:41:28.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.251+0000 7f0d80393740 20 client.19881 trim_cache size 0 max 16384 2024-09-16T00:41:28.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.264+0000 7f0d80393740 1 Processor -- start 2024-09-16T00:41:28.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.264+0000 7f0d80393740 1 -- start start 2024-09-16T00:41:28.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.264+0000 7f0d80393740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefd5d0 0x55a4bfef9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.264+0000 7f0d80393740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefeb40 0x55a4bff142b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.264+0000 7f0d80393740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bff08370 0x55a4bff00120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.264+0000 7f0d80393740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781b1890 con 0x55a4bfefeb40 2024-09-16T00:41:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.264+0000 7f0d80393740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781af160 con 0x55a4bfefd5d0 2024-09-16T00:41:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.264+0000 7f0d80393740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781acfb0 con 0x55a4bff08370 2024-09-16T00:41:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefeb40 0x55a4bff142b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefeb40 0x55a4bff142b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38316/0 (socket says 172.21.15.33:38316) 2024-09-16T00:41:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d777fe640 1 -- 172.21.15.33:0/2010516608 learned_addr learned my addr 172.21.15.33:0/2010516608 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/2010516608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefd5d0 0x55a4bfef9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/2010516608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bff08370 0x55a4bff00120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d777fe640 1 -- 172.21.15.33:0/2010516608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bff08370 msgr2=0x55a4bff00120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d777fe640 1 --2- 172.21.15.33:0/2010516608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bff08370 0x55a4bff00120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d777fe640 1 -- 172.21.15.33:0/2010516608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefd5d0 msgr2=0x55a4bfef9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:28.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d777fe640 1 --2- 172.21.15.33:0/2010516608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefd5d0 0x55a4bfef9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d777fe640 1 -- 172.21.15.33:0/2010516608 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4bff035a0 con 0x55a4bfefeb40 2024-09-16T00:41:28.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d777fe640 1 --2- 172.21.15.33:0/2010516608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefeb40 0x55a4bff142b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009900 tx=0x7f0d6802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9046c792ec984907 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:28.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d5d7fa640 1 -- 172.21.15.33:0/2010516608 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68035040 con 0x55a4bfefeb40 2024-09-16T00:41:28.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d5d7fa640 1 -- 172.21.15.33:0/2010516608 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d68003f70 con 0x55a4bfefeb40 2024-09-16T00:41:28.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.265+0000 7f0d5d7fa640 1 -- 172.21.15.33:0/2010516608 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6803f040 con 0x55a4bfefeb40 2024-09-16T00:41:28.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.266+0000 7f0d80393740 1 -- 172.21.15.33:0/2010516608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefeb40 msgr2=0x55a4bff142b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.266+0000 7f0d80393740 1 --2- 172.21.15.33:0/2010516608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefeb40 0x55a4bff142b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009900 tx=0x7f0d6802a410 comp rx=0 tx=0).stop 2024-09-16T00:41:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.266+0000 7f0d80393740 1 -- 172.21.15.33:0/2010516608 shutdown_connections 2024-09-16T00:41:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.266+0000 7f0d80393740 1 --2- 172.21.15.33:0/2010516608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bff08370 0x55a4bff00120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.266+0000 7f0d80393740 1 --2- 172.21.15.33:0/2010516608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bfefeb40 0x55a4bff142b0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.266+0000 7f0d80393740 1 --2- 172.21.15.33:0/2010516608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefd5d0 0x55a4bfef9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.266+0000 7f0d80393740 1 -- 172.21.15.33:0/2010516608 >> 172.21.15.33:0/2010516608 conn(0x55a4bfec6310 msgr2=0x55a4bfef9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.266+0000 7f0d80393740 1 -- 172.21.15.33:0/2010516608 shutdown_connections 2024-09-16T00:41:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.266+0000 7f0d80393740 1 -- 172.21.15.33:0/2010516608 wait complete. 2024-09-16T00:41:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d80393740 1 -- start start 2024-09-16T00:41:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d80393740 1 Processor -- start 2024-09-16T00:41:28.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d80393740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefd5d0 0x55a4bff01880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d80393740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefeb40 0x55a4bff01dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d80393740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 0x55a4bff02300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d80393740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781ad620 con 0x55a4bff08370 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d80393740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781aea90 con 0x55a4bfefeb40 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d80393740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781adcf0 con 0x55a4bfefd5d0 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d80393740 10 client.0 _ll_register_callbacks cb 0x7f0d6c009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d76ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 0x55a4bff02300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d76ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 0x55a4bff02300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38332/0 (socket says 172.21.15.33:38332) 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d76ffd640 1 -- 172.21.15.33:0/1197406439 learned_addr learned my addr 172.21.15.33:0/1197406439 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.267+0000 7f0d777fe640 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefeb40 0x55a4bff01dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefd5d0 0x55a4bff01880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d76ffd640 1 -- 172.21.15.33:0/1197406439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefd5d0 msgr2=0x55a4bff01880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefd5d0 0x55a4bff01880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d76ffd640 1 -- 172.21.15.33:0/1197406439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefeb40 msgr2=0x55a4bff01dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefeb40 0x55a4bff01dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d76ffd640 1 -- 172.21.15.33:0/1197406439 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4bff0f4c0 con 0x55a4bff08370 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d777fe640 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefeb40 0x55a4bff01dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d76ffd640 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 0x55a4bff02300 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c01c040 tx=0x7f0d6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29770073e1d4f1d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d56ffd640 1 -- 172.21.15.33:0/1197406439 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c0165e0 con 0x55a4bff08370 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d56ffd640 1 -- 172.21.15.33:0/1197406439 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0d6c016e40 con 0x55a4bff08370 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d80393740 1 -- 172.21.15.33:0/1197406439 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d68003f70 con 0x55a4bff08370 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d56ffd640 1 -- 172.21.15.33:0/1197406439 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0d6c03f050 con 0x55a4bff08370 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d80393740 10 client.19887 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.268+0000 7f0d80393740 1 -- 172.21.15.33:0/1197406439 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a4bff107c0 con 0x55a4bff08370 2024-09-16T00:41:28.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.269+0000 7f0d56ffd640 1 -- 172.21.15.33:0/1197406439 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d68003f70 con 0x55a4bff08370 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.269+0000 7f0d56ffd640 1 -- 172.21.15.33:0/1197406439 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0d6c0190c0 con 0x55a4bff08370 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.269+0000 7f0d80393740 10 client.19887 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.269+0000 7f0d80393740 10 client.19887 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.269+0000 7f0d80393740 20 client.19887 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.269+0000 7f0d80393740 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a4bff8b770 0x55a4bff8db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.269+0000 7f0d80393740 4 client.19887 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.269+0000 7f0d80393740 1 -- 172.21.15.33:0/1197406439 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a4bfe6ad70 con 0x55a4bff8b770 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.269+0000 7f0d777fe640 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a4bff8b770 0x55a4bff8db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d777fe640 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a4bff8b770 0x55a4bff8db60 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e3b1ea425c5138c5 server_cookie=e49ec5b1551274ce in_seq=0 out_seq=0 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d56ffd640 10 client.19887 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d56ffd640 1 -- 172.21.15.33:0/1197406439 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a4bfe6ad70 con 0x55a4bff8b770 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d56ffd640 10 client.19887 handle_command_reply: tid=0 2024-09-16T00:41:28.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d80393740 1 client.19887 shutdown 2024-09-16T00:41:28.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d80393740 1 -- 172.21.15.33:0/1197406439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 msgr2=0x55a4bff02300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:28.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d80393740 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 0x55a4bff02300 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c01c040 tx=0x7f0d6c0046c0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d80393740 1 -- 172.21.15.33:0/1197406439 shutdown_connections 2024-09-16T00:41:28.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d80393740 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55a4bff8b770 0x55a4bff8db60 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d56ffd640 0 client.19887 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:41:28.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d80393740 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4bff08370 0x55a4bff02300 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d80393740 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a4bfefeb40 0x55a4bff01dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d80393740 1 --2- 172.21.15.33:0/1197406439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4bfefd5d0 0x55a4bff01880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d80393740 1 -- 172.21.15.33:0/1197406439 >> 172.21.15.33:0/1197406439 conn(0x55a4bfec6310 msgr2=0x55a4bff06890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:28.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d80393740 1 -- 172.21.15.33:0/1197406439 shutdown_connections 2024-09-16T00:41:28.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d80393740 1 -- 172.21.15.33:0/1197406439 wait complete. 2024-09-16T00:41:28.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.270+0000 7f0d80393740 20 client.19887 trim_cache size 0 max 16384 2024-09-16T00:41:28.277 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:41:28.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.271+0000 7f0d7dfba640 1 -- 172.21.15.33:0/2899506837 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d600a3240 msgr2=0x7f0d600a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:28.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.271+0000 7f0d7dfba640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d600a3240 0x7f0d600a5700 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0d700899a0 tx=0x7f0d700980e0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.271+0000 7f0d7dfba640 1 -- 172.21.15.33:0/2899506837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d78106f10 msgr2=0x7f0d7819cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:28.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.271+0000 7f0d7dfba640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d78106f10 0x7f0d7819cd80 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000cce0 tx=0x7f0d7000a620 comp rx=0 tx=0).stop 2024-09-16T00:41:28.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.271+0000 7f0d7dfba640 1 -- 172.21.15.33:0/2899506837 shutdown_connections 2024-09-16T00:41:28.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.271+0000 7f0d7dfba640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d600a3240 0x7f0d600a5700 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.271+0000 7f0d7dfba640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d78106f10 0x7f0d7819cd80 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.271+0000 7f0d7dfba640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d78102790 0x7f0d781989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.271+0000 7f0d7dfba640 1 --2- 172.21.15.33:0/2899506837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d78101de0 0x7f0d781984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:28.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.271+0000 7f0d7dfba640 1 -- 172.21.15.33:0/2899506837 >> 172.21.15.33:0/2899506837 conn(0x7f0d780fd8c0 msgr2=0x7f0d780feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:28.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.271+0000 7f0d7dfba640 1 -- 172.21.15.33:0/2899506837 shutdown_connections 2024-09-16T00:41:28.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:28.271+0000 7f0d7dfba640 1 -- 172.21.15.33:0/2899506837 wait complete. 2024-09-16T00:41:29.104 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.55 MB/sec execute 515 sec latency 4771.029 ms 2024-09-16T00:41:29.506 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:41:29.819 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:30.072 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:29 smithi033 ceph-mon[31243]: pgmap v1080: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 585 op/s 2024-09-16T00:41:30.104 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.51 MB/sec execute 516 sec latency 5771.074 ms 2024-09-16T00:41:30.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:29 smithi083 ceph-mon[35242]: pgmap v1080: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 585 op/s 2024-09-16T00:41:30.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:29 smithi060 ceph-mon[35185]: pgmap v1080: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 585 op/s 2024-09-16T00:41:30.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.314+0000 7fc189ef7640 1 -- 172.21.15.33:0/2606389031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184101de0 msgr2=0x7fc1841021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:30.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.314+0000 7fc189ef7640 1 --2- 172.21.15.33:0/2606389031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184101de0 0x7fc1841021c0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc174009900 tx=0x7fc17402a410 comp rx=0 tx=0).stop 2024-09-16T00:41:30.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.314+0000 7fc189ef7640 1 -- 172.21.15.33:0/2606389031 shutdown_connections 2024-09-16T00:41:30.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.314+0000 7fc189ef7640 1 --2- 172.21.15.33:0/2606389031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc184106f10 0x7fc18410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.314+0000 7fc189ef7640 1 --2- 172.21.15.33:0/2606389031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc184102790 0x7fc1841067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.314+0000 7fc189ef7640 1 --2- 172.21.15.33:0/2606389031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184101de0 0x7fc1841021c0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.314+0000 7fc189ef7640 1 -- 172.21.15.33:0/2606389031 >> 172.21.15.33:0/2606389031 conn(0x7fc1840fd8c0 msgr2=0x7fc1840ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:30.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.314+0000 7fc189ef7640 1 -- 172.21.15.33:0/2606389031 shutdown_connections 2024-09-16T00:41:30.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.314+0000 7fc189ef7640 1 -- 172.21.15.33:0/2606389031 wait complete. 2024-09-16T00:41:30.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.315+0000 7fc189ef7640 1 Processor -- start 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.315+0000 7fc189ef7640 1 -- start start 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.315+0000 7fc189ef7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc184101de0 0x7fc184198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.315+0000 7fc189ef7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc184102790 0x7fc184198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.315+0000 7fc189ef7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184106f10 0x7fc18419cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.315+0000 7fc189ef7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc184111f30 con 0x7fc184106f10 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.315+0000 7fc189ef7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc184111db0 con 0x7fc184101de0 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.315+0000 7fc189ef7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc1841120b0 con 0x7fc184102790 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.315+0000 7fc183fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184106f10 0x7fc18419cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc183fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184106f10 0x7fc18419cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35748/0 (socket says 172.21.15.33:35748) 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc183fff640 1 -- 172.21.15.33:0/2197186744 learned_addr learned my addr 172.21.15.33:0/2197186744 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc182ffd640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc184102790 0x7fc184198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc1837fe640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc184101de0 0x7fc184198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc183fff640 1 -- 172.21.15.33:0/2197186744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc184102790 msgr2=0x7fc184198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc183fff640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc184102790 0x7fc184198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc183fff640 1 -- 172.21.15.33:0/2197186744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc184101de0 msgr2=0x7fc184198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc183fff640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc184101de0 0x7fc184198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc183fff640 1 -- 172.21.15.33:0/2197186744 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18419d4c0 con 0x7fc184106f10 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc1837fe640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc184101de0 0x7fc184198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc183fff640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184106f10 0x7fc18419cde0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fc17c00eb10 tx=0x7fc17c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc180ff9640 1 -- 172.21.15.33:0/2197186744 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c004b20 con 0x7fc184106f10 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc180ff9640 1 -- 172.21.15.33:0/2197186744 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc17c004cc0 con 0x7fc184106f10 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc180ff9640 1 -- 172.21.15.33:0/2197186744 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c010760 con 0x7fc184106f10 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc189ef7640 1 -- 172.21.15.33:0/2197186744 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc18419d7b0 con 0x7fc184106f10 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.316+0000 7fc189ef7640 1 -- 172.21.15.33:0/2197186744 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1841a5090 con 0x7fc184106f10 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc180ff9640 1 -- 172.21.15.33:0/2197186744 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc17c005020 con 0x7fc184106f10 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc18c2d0740 1 Processor -- start 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc18c2d0740 1 -- start start 2024-09-16T00:41:30.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc18c2d0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c72489ab40 0x55c724899090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc18c2d0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 0x55c724899a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc18c2d0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7248a4370 0x55c7248a6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc18c2d0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c7247ff2f0 con 0x55c72489ab40 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc18c2d0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c7247a6cf0 con 0x55c7248995d0 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc18c2d0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c7247fb2b0 con 0x55c7248a4370 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc180ff9640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc16c0a32c0 0x7fc16c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc182ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c72489ab40 0x55c724899090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc182ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c72489ab40 0x55c724899090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35764/0 (socket says 172.21.15.33:35764) 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.319+0000 7fc182ffd640 1 -- 172.21.15.33:0/3093902613 learned_addr learned my addr 172.21.15.33:0/3093902613 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc183fff640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc16c0a32c0 0x7fc16c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc180ff9640 1 -- 172.21.15.33:0/2197186744 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc17c0d02d0 con 0x7fc184106f10 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc1837fe640 1 --2- 172.21.15.33:0/3093902613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 0x55c724899a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc182ffd640 1 --2- 172.21.15.33:0/3093902613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7248a4370 0x55c7248a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc182ffd640 1 -- 172.21.15.33:0/3093902613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7248a4370 msgr2=0x55c7248a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:30.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc182ffd640 1 --2- 172.21.15.33:0/3093902613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7248a4370 0x55c7248a6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc182ffd640 1 -- 172.21.15.33:0/3093902613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 msgr2=0x55c724899a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:30.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc182ffd640 1 --2- 172.21.15.33:0/3093902613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 0x55c724899a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc182ffd640 1 -- 172.21.15.33:0/3093902613 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c72478b3a0 con 0x55c72489ab40 2024-09-16T00:41:30.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc182ffd640 1 --2- 172.21.15.33:0/3093902613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c72489ab40 0x55c724899090 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc178005bc0 tx=0x7fc178009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7209cea7fe30e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:30.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc183fff640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc16c0a32c0 0x7fc16c0a5780 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0988d0 tx=0x7fc17c091950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:30.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc169ffb640 1 -- 172.21.15.33:0/3093902613 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178004030 con 0x55c72489ab40 2024-09-16T00:41:30.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc169ffb640 1 -- 172.21.15.33:0/3093902613 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1780041d0 con 0x55c72489ab40 2024-09-16T00:41:30.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.320+0000 7fc169ffb640 1 -- 172.21.15.33:0/3093902613 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178010770 con 0x55c72489ab40 2024-09-16T00:41:30.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.321+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3093902613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c72489ab40 msgr2=0x55c724899090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:30.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.321+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3093902613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c72489ab40 0x55c724899090 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc178005bc0 tx=0x7fc178009850 comp rx=0 tx=0).stop 2024-09-16T00:41:30.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.321+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3093902613 shutdown_connections 2024-09-16T00:41:30.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.321+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3093902613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7248a4370 0x55c7248a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.321+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3093902613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 0x55c724899a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.321+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3093902613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c72489ab40 0x55c724899090 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.321+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3093902613 >> 172.21.15.33:0/3093902613 conn(0x55c724862310 msgr2=0x55c724895c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:30.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.321+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3093902613 shutdown_connections 2024-09-16T00:41:30.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.321+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3093902613 wait complete. 2024-09-16T00:41:30.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.321+0000 7fc18c2d0740 1 -- start start 2024-09-16T00:41:30.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc18c2d0740 1 Processor -- start 2024-09-16T00:41:30.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc18c2d0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248995d0 0x55c7248a3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc18c2d0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c72489ab40 0x55c7248a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc18c2d0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7248a4370 0x55c7248a14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc18c2d0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c7248a7070 con 0x55c7248995d0 2024-09-16T00:41:30.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc18c2d0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c7247ba060 con 0x55c72489ab40 2024-09-16T00:41:30.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc18c2d0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c7248a71f0 con 0x55c7248a4370 2024-09-16T00:41:30.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc18c2d0740 10 client.0 _ll_register_callbacks cb 0x55c72450d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:30.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc182ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248995d0 0x55c7248a3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc182ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248995d0 0x55c7248a3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35778/0 (socket says 172.21.15.33:35778) 2024-09-16T00:41:30.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc182ffd640 1 -- 172.21.15.33:0/3343835765 learned_addr learned my addr 172.21.15.33:0/3343835765 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:30.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc1837fe640 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c72489ab40 0x55c7248a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.322+0000 7fc182ffd640 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7248a4370 0x55c7248a14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc182ffd640 1 -- 172.21.15.33:0/3343835765 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7248a4370 msgr2=0x55c7248a14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:30.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc182ffd640 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7248a4370 0x55c7248a14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc182ffd640 1 -- 172.21.15.33:0/3343835765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c72489ab40 msgr2=0x55c7248a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:30.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc182ffd640 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c72489ab40 0x55c7248a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc182ffd640 1 -- 172.21.15.33:0/3343835765 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7245703c0 con 0x55c7248995d0 2024-09-16T00:41:30.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc182ffd640 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248995d0 0x55c7248a3ce0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fc178003e50 tx=0x7fc178005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89155bc00e221da8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:30.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc162ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:30.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc162ffd640 1 -- 172.21.15.33:0/3343835765 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178010510 con 0x55c7248995d0 2024-09-16T00:41:30.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc162ffd640 1 -- 172.21.15.33:0/3343835765 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc1780106b0 con 0x55c7248995d0 2024-09-16T00:41:30.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3343835765 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1780041d0 con 0x55c7248995d0 2024-09-16T00:41:30.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc162ffd640 1 -- 172.21.15.33:0/3343835765 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc1780108b0 con 0x55c7248995d0 2024-09-16T00:41:30.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc18c2d0740 10 client.19899 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:30.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3343835765 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c7248b2a80 con 0x55c7248995d0 2024-09-16T00:41:30.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.323+0000 7fc162ffd640 1 -- 172.21.15.33:0/3343835765 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1780041d0 con 0x55c7248995d0 2024-09-16T00:41:30.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.324+0000 7fc162ffd640 1 -- 172.21.15.33:0/3343835765 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fc17801dba0 con 0x55c7248995d0 2024-09-16T00:41:30.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.324+0000 7fc18c2d0740 10 client.19899 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:30.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.324+0000 7fc18c2d0740 10 client.19899 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:41:30.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.324+0000 7fc18c2d0740 20 client.19899 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:30.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.324+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c7248b3ee0 0x55c7248b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.324+0000 7fc18c2d0740 4 client.19899 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.324+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3343835765 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c72480c860 con 0x55c7248b3ee0 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.324+0000 7fc1837fe640 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c7248b3ee0 0x55c7248b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.324+0000 7fc1837fe640 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c7248b3ee0 0x55c7248b62d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6dbe792b3111c03e server_cookie=1eff3ad610e326f4 in_seq=0 out_seq=0 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.324+0000 7fc162ffd640 10 client.19899 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc162ffd640 1 -- 172.21.15.33:0/3343835765 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c72480c860 con 0x55c7248b3ee0 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc162ffd640 10 client.19899 handle_command_reply: tid=0 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc18c2d0740 1 client.19899 shutdown 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3343835765 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248995d0 msgr2=0x55c7248a3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248995d0 0x55c7248a3ce0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fc178003e50 tx=0x7fc178005fb0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3343835765 shutdown_connections 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7248a4370 0x55c7248a14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c7248b3ee0 0x55c7248b62d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc162ffd640 0 client.19899 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:41:30.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c72489ab40 0x55c7248a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3343835765 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248995d0 0x55c7248a3ce0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3343835765 >> 172.21.15.33:0/3343835765 conn(0x55c724862310 msgr2=0x55c72489b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:30.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.325+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3343835765 shutdown_connections 2024-09-16T00:41:30.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.326+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3343835765 wait complete. 2024-09-16T00:41:30.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.326+0000 7fc18c2d0740 20 client.19899 trim_cache size 0 max 16384 2024-09-16T00:41:30.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.339+0000 7fc18c2d0740 1 Processor -- start 2024-09-16T00:41:30.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.339+0000 7fc18c2d0740 1 -- start start 2024-09-16T00:41:30.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.339+0000 7fc18c2d0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 0x55c7248b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.339+0000 7fc18c2d0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c72489ab40 0x55c724894c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.339+0000 7fc18c2d0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 0x55c7248bb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.339+0000 7fc18c2d0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc1841b19d0 con 0x55c7248a4370 2024-09-16T00:41:30.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.339+0000 7fc18c2d0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc1841af2a0 con 0x55c7248995d0 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.339+0000 7fc18c2d0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc1841ad0f0 con 0x55c72489ab40 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc182ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 0x55c7248bb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc182ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 0x55c7248bb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35786/0 (socket says 172.21.15.33:35786) 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc182ffd640 1 -- 172.21.15.33:0/1166144568 learned_addr learned my addr 172.21.15.33:0/1166144568 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc1837fe640 1 --2- 172.21.15.33:0/1166144568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c72489ab40 0x55c724894c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc182ffd640 1 --2- 172.21.15.33:0/1166144568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 0x55c7248b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc182ffd640 1 -- 172.21.15.33:0/1166144568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c72489ab40 msgr2=0x55c724894c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc182ffd640 1 --2- 172.21.15.33:0/1166144568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c72489ab40 0x55c724894c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc182ffd640 1 -- 172.21.15.33:0/1166144568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 msgr2=0x55c7248b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc182ffd640 1 --2- 172.21.15.33:0/1166144568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 0x55c7248b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc182ffd640 1 -- 172.21.15.33:0/1166144568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c724896960 con 0x55c7248a4370 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc182ffd640 1 --2- 172.21.15.33:0/1166144568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 0x55c7248bb9c0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fc17802c7f0 tx=0x7fc178000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6d2f1694abe7bb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc1697fa640 1 -- 172.21.15.33:0/1166144568 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178003e80 con 0x55c7248a4370 2024-09-16T00:41:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc1697fa640 1 -- 172.21.15.33:0/1166144568 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc178028040 con 0x55c7248a4370 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.340+0000 7fc1697fa640 1 -- 172.21.15.33:0/1166144568 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178010690 con 0x55c7248a4370 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.341+0000 7fc18c2d0740 1 -- 172.21.15.33:0/1166144568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 msgr2=0x55c7248bb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.341+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/1166144568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 0x55c7248bb9c0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fc17802c7f0 tx=0x7fc178000bc0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.341+0000 7fc18c2d0740 1 -- 172.21.15.33:0/1166144568 shutdown_connections 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.341+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/1166144568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 0x55c7248bb9c0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.341+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/1166144568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c72489ab40 0x55c724894c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.341+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/1166144568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 0x55c7248b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.341+0000 7fc18c2d0740 1 -- 172.21.15.33:0/1166144568 >> 172.21.15.33:0/1166144568 conn(0x55c724862310 msgr2=0x55c724894500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.341+0000 7fc18c2d0740 1 -- 172.21.15.33:0/1166144568 shutdown_connections 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.341+0000 7fc18c2d0740 1 -- 172.21.15.33:0/1166144568 wait complete. 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.342+0000 7fc18c2d0740 1 -- start start 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.342+0000 7fc18c2d0740 1 Processor -- start 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.342+0000 7fc18c2d0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 0x55c7248ac480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.342+0000 7fc18c2d0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c72489ab40 0x55c7248ac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.342+0000 7fc18c2d0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 0x55c72489d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.342+0000 7fc18c2d0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc1841ad760 con 0x55c7248a4370 2024-09-16T00:41:30.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.342+0000 7fc18c2d0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc1841aebd0 con 0x55c7248995d0 2024-09-16T00:41:30.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.342+0000 7fc18c2d0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc1841ade30 con 0x55c72489ab40 2024-09-16T00:41:30.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.342+0000 7fc18c2d0740 10 client.0 _ll_register_callbacks cb 0x7fc178009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:30.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.342+0000 7fc182ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 0x55c72489d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.342+0000 7fc182ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 0x55c72489d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35788/0 (socket says 172.21.15.33:35788) 2024-09-16T00:41:30.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.342+0000 7fc182ffd640 1 -- 172.21.15.33:0/3689310964 learned_addr learned my addr 172.21.15.33:0/3689310964 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:30.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc1837fe640 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c72489ab40 0x55c7248ac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc182ffd640 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 0x55c7248ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc182ffd640 1 -- 172.21.15.33:0/3689310964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c72489ab40 msgr2=0x55c7248ac9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:30.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc182ffd640 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c72489ab40 0x55c7248ac9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc182ffd640 1 -- 172.21.15.33:0/3689310964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 msgr2=0x55c7248ac480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:30.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc182ffd640 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 0x55c7248ac480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc182ffd640 1 -- 172.21.15.33:0/3689310964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c72489dbb0 con 0x55c7248a4370 2024-09-16T00:41:30.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc1837fe640 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c72489ab40 0x55c7248ac9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:41:30.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc182ffd640 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 0x55c72489d5c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fc1780038c0 tx=0x7fc178003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667f5a85cc242c78 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:30.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc162ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:30.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc162ffd640 1 -- 172.21.15.33:0/3689310964 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17804b060 con 0x55c7248a4370 2024-09-16T00:41:30.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc162ffd640 1 -- 172.21.15.33:0/3689310964 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fc178010640 con 0x55c7248a4370 2024-09-16T00:41:30.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3689310964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc178028040 con 0x55c7248a4370 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc162ffd640 1 -- 172.21.15.33:0/3689310964 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc178003ba0 con 0x55c7248a4370 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc18c2d0740 10 client.19905 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.343+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3689310964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c72491e250 con 0x55c7248a4370 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.344+0000 7fc162ffd640 1 -- 172.21.15.33:0/3689310964 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc178028040 con 0x55c7248a4370 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.344+0000 7fc162ffd640 1 -- 172.21.15.33:0/3689310964 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fc17804e080 con 0x55c7248a4370 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.344+0000 7fc18c2d0740 10 client.19905 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.344+0000 7fc18c2d0740 10 client.19905 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.344+0000 7fc18c2d0740 20 client.19905 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.344+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c7248b3ee0 0x55c72491e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.344+0000 7fc18c2d0740 4 client.19905 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.344+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3689310964 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c724806d70 con 0x55c7248b3ee0 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.344+0000 7fc1837fe640 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c7248b3ee0 0x55c72491e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc1837fe640 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c7248b3ee0 0x55c72491e980 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9917f58b039583c2 server_cookie=88fe3b1db1bf8717 in_seq=0 out_seq=0 2024-09-16T00:41:30.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc162ffd640 10 client.19905 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc162ffd640 1 -- 172.21.15.33:0/3689310964 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c724806d70 con 0x55c7248b3ee0 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc162ffd640 10 client.19905 handle_command_reply: tid=0 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc18c2d0740 1 client.19905 shutdown 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3689310964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 msgr2=0x55c72489d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 0x55c72489d5c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fc1780038c0 tx=0x7fc178003b00 comp rx=0 tx=0).stop 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3689310964 shutdown_connections 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7248a4370 0x55c72489d5c0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c72489ab40 0x55c7248ac9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55c7248b3ee0 0x55c72491e980 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc18c2d0740 1 --2- 172.21.15.33:0/3689310964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c7248995d0 0x55c7248ac480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3689310964 >> 172.21.15.33:0/3689310964 conn(0x55c724862310 msgr2=0x55c7248944e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc162ffd640 0 client.19905 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:41:30.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3689310964 shutdown_connections 2024-09-16T00:41:30.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc18c2d0740 1 -- 172.21.15.33:0/3689310964 wait complete. 2024-09-16T00:41:30.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.345+0000 7fc18c2d0740 20 client.19905 trim_cache size 0 max 16384 2024-09-16T00:41:30.365 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:41:30.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.346+0000 7fc189ef7640 1 -- 172.21.15.33:0/2197186744 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc16c0a32c0 msgr2=0x7fc16c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:30.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.346+0000 7fc189ef7640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc16c0a32c0 0x7fc16c0a5780 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0988d0 tx=0x7fc17c091950 comp rx=0 tx=0).stop 2024-09-16T00:41:30.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.346+0000 7fc189ef7640 1 -- 172.21.15.33:0/2197186744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184106f10 msgr2=0x7fc18419cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:30.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.346+0000 7fc189ef7640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184106f10 0x7fc18419cde0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fc17c00eb10 tx=0x7fc17c00c2d0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.346+0000 7fc189ef7640 1 -- 172.21.15.33:0/2197186744 shutdown_connections 2024-09-16T00:41:30.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.346+0000 7fc189ef7640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fc16c0a32c0 0x7fc16c0a5780 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.346+0000 7fc189ef7640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184106f10 0x7fc18419cde0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.346+0000 7fc189ef7640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc184102790 0x7fc184198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.346+0000 7fc189ef7640 1 --2- 172.21.15.33:0/2197186744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc184101de0 0x7fc184198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:30.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.346+0000 7fc189ef7640 1 -- 172.21.15.33:0/2197186744 >> 172.21.15.33:0/2197186744 conn(0x7fc1840fd8c0 msgr2=0x7fc1840fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:30.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.346+0000 7fc189ef7640 1 -- 172.21.15.33:0/2197186744 shutdown_connections 2024-09-16T00:41:30.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:30.346+0000 7fc189ef7640 1 -- 172.21.15.33:0/2197186744 wait complete. 2024-09-16T00:41:31.088 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:41:31.104 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.48 MB/sec execute 517 sec latency 6771.110 ms 2024-09-16T00:41:31.352 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:31.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.841+0000 7f1528b14640 1 -- 172.21.15.33:0/1989917449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1524101de0 msgr2=0x7f15241021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.841+0000 7f1528b14640 1 --2- 172.21.15.33:0/1989917449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1524101de0 0x7f15241021c0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f1514009ca0 tx=0x7f151402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.841+0000 7f1528b14640 1 -- 172.21.15.33:0/1989917449 shutdown_connections 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.841+0000 7f1528b14640 1 --2- 172.21.15.33:0/1989917449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1524106f10 0x7f152410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.841+0000 7f1528b14640 1 --2- 172.21.15.33:0/1989917449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1524102790 0x7f15241067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.841+0000 7f1528b14640 1 --2- 172.21.15.33:0/1989917449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1524101de0 0x7f15241021c0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.841+0000 7f1528b14640 1 -- 172.21.15.33:0/1989917449 >> 172.21.15.33:0/1989917449 conn(0x7f15240fd8c0 msgr2=0x7f15240ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.842+0000 7f1528b14640 1 -- 172.21.15.33:0/1989917449 shutdown_connections 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.842+0000 7f1528b14640 1 -- 172.21.15.33:0/1989917449 wait complete. 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.842+0000 7f1528b14640 1 Processor -- start 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.842+0000 7f1528b14640 1 -- start start 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.842+0000 7f1528b14640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1524101de0 0x7f1524198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.842+0000 7f1528b14640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1524102790 0x7f1524198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.842+0000 7f1528b14640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1524106f10 0x7f152419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1528b14640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1524111f70 con 0x7f1524102790 2024-09-16T00:41:31.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1528b14640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1524111df0 con 0x7f1524106f10 2024-09-16T00:41:31.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1528b14640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f15241120f0 con 0x7f1524101de0 2024-09-16T00:41:31.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1521d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1524102790 0x7f1524198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1521d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1524102790 0x7f1524198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35808/0 (socket says 172.21.15.33:35808) 2024-09-16T00:41:31.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1521d9b640 1 -- 172.21.15.33:0/1416333471 learned_addr learned my addr 172.21.15.33:0/1416333471 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:31.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f152259c640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1524101de0 0x7f1524198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1522d9d640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1524106f10 0x7f152419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1521d9b640 1 -- 172.21.15.33:0/1416333471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1524101de0 msgr2=0x7f1524198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:31.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1521d9b640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1524101de0 0x7f1524198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1521d9b640 1 -- 172.21.15.33:0/1416333471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1524106f10 msgr2=0x7f152419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1521d9b640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1524106f10 0x7f152419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1521d9b640 1 -- 172.21.15.33:0/1416333471 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152419d3e0 con 0x7f1524102790 2024-09-16T00:41:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1521d9b640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1524102790 0x7f1524198970 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f151800d950 tx=0x7f151800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f15137fe640 1 -- 172.21.15.33:0/1416333471 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1518017070 con 0x7f1524102790 2024-09-16T00:41:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f15137fe640 1 -- 172.21.15.33:0/1416333471 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1518009c30 con 0x7f1524102790 2024-09-16T00:41:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1528b14640 1 -- 172.21.15.33:0/1416333471 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f152419d6d0 con 0x7f1524102790 2024-09-16T00:41:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f15137fe640 1 -- 172.21.15.33:0/1416333471 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151800f6b0 con 0x7f1524102790 2024-09-16T00:41:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.843+0000 7f1528b14640 1 -- 172.21.15.33:0/1416333471 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15241a4fb0 con 0x7f1524102790 2024-09-16T00:41:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.846+0000 7f15137fe640 1 -- 172.21.15.33:0/1416333471 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f15180109d0 con 0x7f1524102790 2024-09-16T00:41:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.846+0000 7f152aeed740 1 Processor -- start 2024-09-16T00:41:31.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.846+0000 7f152aeed740 1 -- start start 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.846+0000 7f152aeed740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3f8d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.846+0000 7f152aeed740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3f8da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.846+0000 7f152aeed740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f98370 0x557bb3f9a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f152aeed740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557bb3ef32f0 con 0x557bb3f8eb40 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f152aeed740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x557bb3e9acf0 con 0x557bb3f8d5d0 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f152aeed740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557bb3eef2b0 con 0x557bb3f98370 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f15137fe640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f150c0a32c0 0x7f150c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f1522d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3f8d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f1522d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3f8d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35820/0 (socket says 172.21.15.33:35820) 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f1522d9d640 1 -- 172.21.15.33:0/3741028601 learned_addr learned my addr 172.21.15.33:0/3741028601 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f152259c640 1 --2- 172.21.15.33:0/3741028601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3f8da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f1522d9d640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f150c0a32c0 0x7f150c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f1521d9b640 1 --2- 172.21.15.33:0/3741028601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f98370 0x557bb3f9a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f15137fe640 1 -- 172.21.15.33:0/1416333471 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f15180cf8d0 con 0x7f1524102790 2024-09-16T00:41:31.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f1522d9d640 1 -- 172.21.15.33:0/3741028601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f98370 msgr2=0x557bb3f9a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:31.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f1522d9d640 1 --2- 172.21.15.33:0/3741028601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f98370 0x557bb3f9a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f1522d9d640 1 -- 172.21.15.33:0/3741028601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 msgr2=0x557bb3f8da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:31.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f1522d9d640 1 --2- 172.21.15.33:0/3741028601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3f8da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f1522d9d640 1 -- 172.21.15.33:0/3741028601 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bb3e7f3a0 con 0x557bb3f8eb40 2024-09-16T00:41:31.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f152259c640 1 --2- 172.21.15.33:0/3741028601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3f8da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:41:31.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.847+0000 7f1522d9d640 1 --2- 172.21.15.33:0/3741028601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3f8d090 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f151c013690 tx=0x7f151c013b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcaa5e49b782f4c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:31.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.848+0000 7f1510ff9640 1 -- 172.21.15.33:0/3741028601 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151c021020 con 0x557bb3f8eb40 2024-09-16T00:41:31.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.848+0000 7f1510ff9640 1 -- 172.21.15.33:0/3741028601 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f151c00ab50 con 0x557bb3f8eb40 2024-09-16T00:41:31.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.848+0000 7f1510ff9640 1 -- 172.21.15.33:0/3741028601 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151c00ea10 con 0x557bb3f8eb40 2024-09-16T00:41:31.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.848+0000 7f1522d9d640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f150c0a32c0 0x7f150c0a5780 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f151c022000 tx=0x7f151c020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:31.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.848+0000 7f152aeed740 1 -- 172.21.15.33:0/3741028601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 msgr2=0x557bb3f8d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:31.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.848+0000 7f152aeed740 1 --2- 172.21.15.33:0/3741028601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3f8d090 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f151c013690 tx=0x7f151c013b50 comp rx=0 tx=0).stop 2024-09-16T00:41:31.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.848+0000 7f152aeed740 1 -- 172.21.15.33:0/3741028601 shutdown_connections 2024-09-16T00:41:31.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.848+0000 7f152aeed740 1 --2- 172.21.15.33:0/3741028601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f98370 0x557bb3f9a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.848+0000 7f152aeed740 1 --2- 172.21.15.33:0/3741028601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3f8da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.848+0000 7f152aeed740 1 --2- 172.21.15.33:0/3741028601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3f8d090 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.848+0000 7f152aeed740 1 -- 172.21.15.33:0/3741028601 >> 172.21.15.33:0/3741028601 conn(0x557bb3f56310 msgr2=0x557bb3f89c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:31.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.848+0000 7f152aeed740 1 -- 172.21.15.33:0/3741028601 shutdown_connections 2024-09-16T00:41:31.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f152aeed740 1 -- 172.21.15.33:0/3741028601 wait complete. 2024-09-16T00:41:31.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f152aeed740 1 -- start start 2024-09-16T00:41:31.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f152aeed740 1 Processor -- start 2024-09-16T00:41:31.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f152aeed740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f8d5d0 0x557bb3f97d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f152aeed740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3f94ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f152aeed740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f98370 0x557bb3f955e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f152aeed740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557bb3f9b070 con 0x557bb3f8eb40 2024-09-16T00:41:31.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f152aeed740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x557bb3eae060 con 0x557bb3f98370 2024-09-16T00:41:31.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f152aeed740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557bb3f9b1f0 con 0x557bb3f8d5d0 2024-09-16T00:41:31.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f152aeed740 10 client.0 _ll_register_callbacks cb 0x557bb3c018c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:31.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f1521d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3f94ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f1521d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3f94ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35834/0 (socket says 172.21.15.33:35834) 2024-09-16T00:41:31.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.849+0000 7f1521d9b640 1 -- 172.21.15.33:0/2814076491 learned_addr learned my addr 172.21.15.33:0/2814076491 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:31.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f152259c640 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f8d5d0 0x557bb3f97d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1521d9b640 1 -- 172.21.15.33:0/2814076491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f8d5d0 msgr2=0x557bb3f97d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:31.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1521d9b640 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f8d5d0 0x557bb3f97d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1521d9b640 1 -- 172.21.15.33:0/2814076491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f98370 msgr2=0x557bb3f955e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:31.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1522d9d640 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f98370 0x557bb3f955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1521d9b640 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f98370 0x557bb3f955e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1521d9b640 1 -- 172.21.15.33:0/2814076491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bb3c643c0 con 0x557bb3f8eb40 2024-09-16T00:41:31.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1521d9b640 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3f94ff0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f151809ba50 tx=0x7f1518000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be76fdbb723f1b42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:31.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1509ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:31.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1509ffb640 1 -- 172.21.15.33:0/2814076491 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1518020040 con 0x557bb3f8eb40 2024-09-16T00:41:31.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1509ffb640 1 -- 172.21.15.33:0/2814076491 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f15180860b0 con 0x557bb3f8eb40 2024-09-16T00:41:31.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f152aeed740 1 -- 172.21.15.33:0/2814076491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f151c00ab50 con 0x557bb3f8eb40 2024-09-16T00:41:31.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1509ffb640 1 -- 172.21.15.33:0/2814076491 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f15180862b0 con 0x557bb3f8eb40 2024-09-16T00:41:31.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f152aeed740 10 client.19917 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:31.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f152aeed740 1 -- 172.21.15.33:0/2814076491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x557bb3fa6aa0 con 0x557bb3f8eb40 2024-09-16T00:41:31.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1509ffb640 1 -- 172.21.15.33:0/2814076491 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f151c00ab50 con 0x557bb3f8eb40 2024-09-16T00:41:31.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.850+0000 7f1509ffb640 1 -- 172.21.15.33:0/2814076491 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f1518010460 con 0x557bb3f8eb40 2024-09-16T00:41:31.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.851+0000 7f152aeed740 10 client.19917 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:31.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.851+0000 7f152aeed740 10 client.19917 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:41:31.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.851+0000 7f152aeed740 20 client.19917 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:31.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.851+0000 7f152aeed740 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557bb3fa7fb0 0x557bb3faa400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.851+0000 7f152aeed740 4 client.19917 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:41:31.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.851+0000 7f152aeed740 1 -- 172.21.15.33:0/2814076491 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bb3f00860 con 0x557bb3fa7fb0 2024-09-16T00:41:31.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.851+0000 7f152259c640 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557bb3fa7fb0 0x557bb3faa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.851+0000 7f152259c640 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557bb3fa7fb0 0x557bb3faa400 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=252f945798aa513c server_cookie=493386a106ca9f39 in_seq=0 out_seq=0 2024-09-16T00:41:31.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.851+0000 7f1509ffb640 10 client.19917 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:31.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f1509ffb640 1 -- 172.21.15.33:0/2814076491 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557bb3f00860 con 0x557bb3fa7fb0 2024-09-16T00:41:31.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f1509ffb640 10 client.19917 handle_command_reply: tid=0 2024-09-16T00:41:31.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f152aeed740 1 client.19917 shutdown 2024-09-16T00:41:31.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f152aeed740 1 -- 172.21.15.33:0/2814076491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 msgr2=0x557bb3f94ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:31.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f152aeed740 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3f94ff0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f151809ba50 tx=0x7f1518000bc0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f152aeed740 1 -- 172.21.15.33:0/2814076491 shutdown_connections 2024-09-16T00:41:31.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f152aeed740 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557bb3fa7fb0 0x557bb3faa400 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f152aeed740 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f98370 0x557bb3f955e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f152aeed740 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3f94ff0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f152aeed740 1 --2- 172.21.15.33:0/2814076491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f8d5d0 0x557bb3f97d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f152aeed740 1 -- 172.21.15.33:0/2814076491 >> 172.21.15.33:0/2814076491 conn(0x557bb3f56310 msgr2=0x557bb3f8f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:31.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f1509ffb640 0 client.19917 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:31.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f152aeed740 1 -- 172.21.15.33:0/2814076491 shutdown_connections 2024-09-16T00:41:31.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f152aeed740 1 -- 172.21.15.33:0/2814076491 wait complete. 2024-09-16T00:41:31.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.852+0000 7f152aeed740 20 client.19917 trim_cache size 0 max 16384 2024-09-16T00:41:31.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.865+0000 7f152aeed740 1 Processor -- start 2024-09-16T00:41:31.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.865+0000 7f152aeed740 1 -- start start 2024-09-16T00:41:31.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f152aeed740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3fa4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f152aeed740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3fa9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f152aeed740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f98370 0x557bb3f90120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f152aeed740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f15241b1920 con 0x557bb3f8eb40 2024-09-16T00:41:31.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f152aeed740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f15241af1f0 con 0x557bb3f8d5d0 2024-09-16T00:41:31.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f152aeed740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f15241ad040 con 0x557bb3f98370 2024-09-16T00:41:31.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f1521d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3fa9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f1521d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3fa9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35842/0 (socket says 172.21.15.33:35842) 2024-09-16T00:41:31.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f1521d9b640 1 -- 172.21.15.33:0/327089422 learned_addr learned my addr 172.21.15.33:0/327089422 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:31.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f152259c640 1 --2- 172.21.15.33:0/327089422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3fa4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f1522d9d640 1 --2- 172.21.15.33:0/327089422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f98370 0x557bb3f90120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f1521d9b640 1 -- 172.21.15.33:0/327089422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f98370 msgr2=0x557bb3f90120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f1521d9b640 1 --2- 172.21.15.33:0/327089422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f98370 0x557bb3f90120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f1521d9b640 1 -- 172.21.15.33:0/327089422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 msgr2=0x557bb3fa4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f1521d9b640 1 --2- 172.21.15.33:0/327089422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3fa4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f1521d9b640 1 -- 172.21.15.33:0/327089422 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bb3f97960 con 0x557bb3f8eb40 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f1522d9d640 1 --2- 172.21.15.33:0/327089422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f98370 0x557bb3f90120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f1521d9b640 1 --2- 172.21.15.33:0/327089422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3fa9fc0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f151809b7c0 tx=0x7f151800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3c8b3922c12a6a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f150bfff640 1 -- 172.21.15.33:0/327089422 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15180b9070 con 0x557bb3f8eb40 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f150bfff640 1 -- 172.21.15.33:0/327089422 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15180a4680 con 0x557bb3f8eb40 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.866+0000 7f150bfff640 1 -- 172.21.15.33:0/327089422 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1518020070 con 0x557bb3f8eb40 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.867+0000 7f152aeed740 1 -- 172.21.15.33:0/327089422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 msgr2=0x557bb3fa9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.867+0000 7f152aeed740 1 --2- 172.21.15.33:0/327089422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3fa9fc0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f151809b7c0 tx=0x7f151800bf60 comp rx=0 tx=0).stop 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.867+0000 7f152aeed740 1 -- 172.21.15.33:0/327089422 shutdown_connections 2024-09-16T00:41:31.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.867+0000 7f152aeed740 1 --2- 172.21.15.33:0/327089422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f98370 0x557bb3f90120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.867+0000 7f152aeed740 1 --2- 172.21.15.33:0/327089422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f8eb40 0x557bb3fa9fc0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.867+0000 7f152aeed740 1 --2- 172.21.15.33:0/327089422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3fa4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.867+0000 7f152aeed740 1 -- 172.21.15.33:0/327089422 >> 172.21.15.33:0/327089422 conn(0x557bb3f56310 msgr2=0x557bb3f940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.867+0000 7f152aeed740 1 -- 172.21.15.33:0/327089422 shutdown_connections 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.867+0000 7f152aeed740 1 -- 172.21.15.33:0/327089422 wait complete. 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.867+0000 7f152aeed740 1 -- start start 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f152aeed740 1 Processor -- start 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f152aeed740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3fa0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f152aeed740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f8eb40 0x557bb3f9bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f152aeed740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f98370 0x557bb3f9c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f152aeed740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f15241ad6b0 con 0x557bb3f98370 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f152aeed740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f15241aeb20 con 0x557bb3f8d5d0 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f152aeed740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f15241add80 con 0x557bb3f8eb40 2024-09-16T00:41:31.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f152aeed740 10 client.0 _ll_register_callbacks cb 0x7f1518097d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f1522d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f98370 0x557bb3f9c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f1522d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f98370 0x557bb3f9c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35856/0 (socket says 172.21.15.33:35856) 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f1522d9d640 1 -- 172.21.15.33:0/2784103659 learned_addr learned my addr 172.21.15.33:0/2784103659 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f152259c640 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3fa0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f1521d9b640 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f8eb40 0x557bb3f9bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f1522d9d640 1 -- 172.21.15.33:0/2784103659 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f8eb40 msgr2=0x557bb3f9bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f1522d9d640 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f8eb40 0x557bb3f9bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f1522d9d640 1 -- 172.21.15.33:0/2784103659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 msgr2=0x557bb3fa0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f1522d9d640 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3fa0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f1522d9d640 1 -- 172.21.15.33:0/2784103659 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bb3f9cd10 con 0x557bb3f98370 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f1522d9d640 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f98370 0x557bb3f9c4e0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f151c013690 tx=0x7f151c005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d436b06bd419ad63 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.868+0000 7f1509ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:31.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f1509ffb640 1 -- 172.21.15.33:0/2784103659 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151c036060 con 0x557bb3f98370 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f1509ffb640 1 -- 172.21.15.33:0/2784103659 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f151c00ed30 con 0x557bb3f98370 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f152aeed740 1 -- 172.21.15.33:0/2784103659 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15180a4680 con 0x557bb3f98370 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f1509ffb640 1 -- 172.21.15.33:0/2784103659 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f151c03d050 con 0x557bb3f98370 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f152aeed740 10 client.19923 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f152aeed740 1 -- 172.21.15.33:0/2784103659 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x557bb4012ca0 con 0x557bb3f98370 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f1509ffb640 1 -- 172.21.15.33:0/2784103659 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15180a4680 con 0x557bb3f98370 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f1509ffb640 1 -- 172.21.15.33:0/2784103659 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f151c01db30 con 0x557bb3f98370 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f152aeed740 10 client.19923 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f152aeed740 10 client.19923 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f152aeed740 20 client.19923 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f152aeed740 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557bb40130e0 0x557bb4015500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f152aeed740 4 client.19923 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:41:31.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.869+0000 7f152aeed740 1 -- 172.21.15.33:0/2784103659 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557bb3faa790 con 0x557bb40130e0 2024-09-16T00:41:31.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f152259c640 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557bb40130e0 0x557bb4015500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:31.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f1509ffb640 10 client.19923 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:31.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f152259c640 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557bb40130e0 0x557bb4015500 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81b6a85d76f6927a server_cookie=80f753a5574dec86 in_seq=0 out_seq=0 2024-09-16T00:41:31.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f1509ffb640 1 -- 172.21.15.33:0/2784103659 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557bb3faa790 con 0x557bb40130e0 2024-09-16T00:41:31.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f1509ffb640 10 client.19923 handle_command_reply: tid=0 2024-09-16T00:41:31.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f152aeed740 1 client.19923 shutdown 2024-09-16T00:41:31.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f152aeed740 1 -- 172.21.15.33:0/2784103659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f98370 msgr2=0x557bb3f9c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:31.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f152aeed740 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f98370 0x557bb3f9c4e0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f151c013690 tx=0x7f151c005cb0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f152aeed740 1 -- 172.21.15.33:0/2784103659 shutdown_connections 2024-09-16T00:41:31.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f152aeed740 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x557bb40130e0 0x557bb4015500 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f152aeed740 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bb3f98370 0x557bb3f9c4e0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f152aeed740 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557bb3f8eb40 0x557bb3f9bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f152aeed740 1 --2- 172.21.15.33:0/2784103659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557bb3f8d5d0 0x557bb3fa0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f152aeed740 1 -- 172.21.15.33:0/2784103659 >> 172.21.15.33:0/2784103659 conn(0x557bb3f56310 msgr2=0x557bb3f884e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:31.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.870+0000 7f1509ffb640 0 client.19923 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:31.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f152aeed740 1 -- 172.21.15.33:0/2784103659 shutdown_connections 2024-09-16T00:41:31.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f152aeed740 1 -- 172.21.15.33:0/2784103659 wait complete. 2024-09-16T00:41:31.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f152aeed740 20 client.19923 trim_cache size 0 max 16384 2024-09-16T00:41:31.882 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:41:31.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:31 smithi033 ceph-mon[31243]: pgmap v1081: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 10 MiB/s wr, 560 op/s 2024-09-16T00:41:31.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f1528b14640 1 -- 172.21.15.33:0/1416333471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f150c0a32c0 msgr2=0x7f150c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:31.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f1528b14640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f150c0a32c0 0x7f150c0a5780 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f151c022000 tx=0x7f151c020000 comp rx=0 tx=0).stop 2024-09-16T00:41:31.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f1528b14640 1 -- 172.21.15.33:0/1416333471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1524102790 msgr2=0x7f1524198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:31.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f1528b14640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1524102790 0x7f1524198970 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f151800d950 tx=0x7f151800de10 comp rx=0 tx=0).stop 2024-09-16T00:41:31.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f1528b14640 1 -- 172.21.15.33:0/1416333471 shutdown_connections 2024-09-16T00:41:31.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f1528b14640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f150c0a32c0 0x7f150c0a5780 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f1528b14640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1524106f10 0x7f152419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f1528b14640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1524102790 0x7f1524198970 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f1528b14640 1 --2- 172.21.15.33:0/1416333471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1524101de0 0x7f1524198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:31.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f1528b14640 1 -- 172.21.15.33:0/1416333471 >> 172.21.15.33:0/1416333471 conn(0x7f15240fd8c0 msgr2=0x7f15240fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:31.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f1528b14640 1 -- 172.21.15.33:0/1416333471 shutdown_connections 2024-09-16T00:41:31.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:31.871+0000 7f1528b14640 1 -- 172.21.15.33:0/1416333471 wait complete. 2024-09-16T00:41:32.104 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.44 MB/sec execute 518 sec latency 7771.168 ms 2024-09-16T00:41:32.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:31 smithi083 ceph-mon[35242]: pgmap v1081: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 10 MiB/s wr, 560 op/s 2024-09-16T00:41:32.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:31 smithi060 ceph-mon[35185]: pgmap v1081: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 10 MiB/s wr, 560 op/s 2024-09-16T00:41:33.104 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.41 MB/sec execute 519 sec latency 8771.273 ms 2024-09-16T00:41:33.619 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:41:33.619 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:41:33.891 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:34.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:33 smithi033 ceph-mon[31243]: pgmap v1082: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.6 MiB/s wr, 391 op/s 2024-09-16T00:41:34.104 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.37 MB/sec execute 520 sec latency 9771.376 ms 2024-09-16T00:41:34.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:33 smithi083 ceph-mon[35242]: pgmap v1082: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.6 MiB/s wr, 391 op/s 2024-09-16T00:41:34.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:33 smithi060 ceph-mon[35185]: pgmap v1082: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.6 MiB/s wr, 391 op/s 2024-09-16T00:41:34.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.397+0000 7f818b996640 1 -- 172.21.15.33:0/407167990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8184101380 msgr2=0x7f818410dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:34.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.397+0000 7f818b996640 1 --2- 172.21.15.33:0/407167990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8184101380 0x7f818410dfb0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f8174009900 tx=0x7f817402a410 comp rx=0 tx=0).stop 2024-09-16T00:41:34.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.397+0000 7f818b996640 1 -- 172.21.15.33:0/407167990 shutdown_connections 2024-09-16T00:41:34.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.397+0000 7f818b996640 1 --2- 172.21.15.33:0/407167990 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f818410e4f0 0x7f8184110900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.397+0000 7f818b996640 1 --2- 172.21.15.33:0/407167990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8184101380 0x7f818410dfb0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.397+0000 7f818b996640 1 --2- 172.21.15.33:0/407167990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81841009d0 0x7f8184100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.397+0000 7f818b996640 1 -- 172.21.15.33:0/407167990 >> 172.21.15.33:0/407167990 conn(0x7f81840fc7d0 msgr2=0x7f81840febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:34.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.397+0000 7f818b996640 1 -- 172.21.15.33:0/407167990 shutdown_connections 2024-09-16T00:41:34.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.397+0000 7f818b996640 1 -- 172.21.15.33:0/407167990 wait complete. 2024-09-16T00:41:34.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.397+0000 7f818b996640 1 Processor -- start 2024-09-16T00:41:34.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.397+0000 7f818b996640 1 -- start start 2024-09-16T00:41:34.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f818b996640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81841009d0 0x7f81841a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f818b996640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8184101380 0x7f81841a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f818b996640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f818410e4f0 0x7f818419a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f818b996640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8184114260 con 0x7f81841009d0 2024-09-16T00:41:34.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f818b996640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f81841140e0 con 0x7f818410e4f0 2024-09-16T00:41:34.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f818b996640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f81841143e0 con 0x7f8184101380 2024-09-16T00:41:34.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f8189732640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81841009d0 0x7f81841a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f8189732640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81841009d0 0x7f81841a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35872/0 (socket says 172.21.15.33:35872) 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f8189732640 1 -- 172.21.15.33:0/277951546 learned_addr learned my addr 172.21.15.33:0/277951546 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f8189f33640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f818410e4f0 0x7f818419a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f8188f31640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8184101380 0x7f81841a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f8189732640 1 -- 172.21.15.33:0/277951546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8184101380 msgr2=0x7f81841a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f8189732640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8184101380 0x7f81841a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f8189732640 1 -- 172.21.15.33:0/277951546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f818410e4f0 msgr2=0x7f818419a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f8189732640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f818410e4f0 0x7f818419a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f8189732640 1 -- 172.21.15.33:0/277951546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818419ace0 con 0x7f81841009d0 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f8189732640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81841009d0 0x7f81841a03f0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f818000d770 tx=0x7f818000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f817a7fc640 1 -- 172.21.15.33:0/277951546 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8180017070 con 0x7f81841009d0 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.398+0000 7f817a7fc640 1 -- 172.21.15.33:0/277951546 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8180005500 con 0x7f81841009d0 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.399+0000 7f818b996640 1 -- 172.21.15.33:0/277951546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f818419afd0 con 0x7f81841009d0 2024-09-16T00:41:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.399+0000 7f817a7fc640 1 -- 172.21.15.33:0/277951546 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8180004e00 con 0x7f81841009d0 2024-09-16T00:41:34.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.399+0000 7f818b996640 1 -- 172.21.15.33:0/277951546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81841a72c0 con 0x7f81841009d0 2024-09-16T00:41:34.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.401+0000 7f817a7fc640 1 -- 172.21.15.33:0/277951546 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8180004fa0 con 0x7f81841009d0 2024-09-16T00:41:34.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.401+0000 7f818dd6f740 1 Processor -- start 2024-09-16T00:41:34.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f818dd6f740 1 -- start start 2024-09-16T00:41:34.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f818dd6f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec24f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f818dd6f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec25480 0x55ccaec2f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f818dd6f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec2fd40 0x55ccaec32130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f818dd6f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ccaeb8a280 con 0x55ccaec24b60 2024-09-16T00:41:34.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f818dd6f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ccaeb31b10 con 0x55ccaec2fd40 2024-09-16T00:41:34.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f818dd6f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ccaeb86240 con 0x55ccaec25480 2024-09-16T00:41:34.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f8189f33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec24f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f8189f33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec24f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35888/0 (socket says 172.21.15.33:35888) 2024-09-16T00:41:34.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f8189f33640 1 -- 172.21.15.33:0/1877316212 learned_addr learned my addr 172.21.15.33:0/1877316212 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:34.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f817a7fc640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81700a32c0 0x7f81700a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f8189732640 1 --2- 172.21.15.33:0/1877316212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec25480 0x55ccaec2f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f8188f31640 1 --2- 172.21.15.33:0/1877316212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec2fd40 0x55ccaec32130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f8189f33640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81700a32c0 0x7f81700a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f817a7fc640 1 -- 172.21.15.33:0/277951546 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f81800cf870 con 0x7f81841009d0 2024-09-16T00:41:34.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f8189f33640 1 -- 172.21.15.33:0/1877316212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec25480 msgr2=0x55ccaec2f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:34.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f8189f33640 1 --2- 172.21.15.33:0/1877316212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec25480 0x55ccaec2f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f8189f33640 1 -- 172.21.15.33:0/1877316212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec2fd40 msgr2=0x55ccaec32130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:34.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f8189f33640 1 --2- 172.21.15.33:0/1877316212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec2fd40 0x55ccaec32130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f8189f33640 1 -- 172.21.15.33:0/1877316212 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccaeb3e1f0 con 0x55ccaec24b60 2024-09-16T00:41:34.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.402+0000 7f8188f31640 1 --2- 172.21.15.33:0/1877316212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec2fd40 0x55ccaec32130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:41:34.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.403+0000 7f8189f33640 1 --2- 172.21.15.33:0/1877316212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec24f40 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f817c0139c0 tx=0x7f817c013e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c23754f6748480e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:34.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.403+0000 7f816f7fe640 1 -- 172.21.15.33:0/1877316212 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c013460 con 0x55ccaec24b60 2024-09-16T00:41:34.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.403+0000 7f8189f33640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81700a32c0 0x7f81700a5780 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f817c021000 tx=0x7f817c01f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:34.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.403+0000 7f816f7fe640 1 -- 172.21.15.33:0/1877316212 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f817c022070 con 0x55ccaec24b60 2024-09-16T00:41:34.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.403+0000 7f816f7fe640 1 -- 172.21.15.33:0/1877316212 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c01cc50 con 0x55ccaec24b60 2024-09-16T00:41:34.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.404+0000 7f818dd6f740 1 -- 172.21.15.33:0/1877316212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 msgr2=0x55ccaec24f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:34.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.404+0000 7f818dd6f740 1 --2- 172.21.15.33:0/1877316212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec24f40 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f817c0139c0 tx=0x7f817c013e80 comp rx=0 tx=0).stop 2024-09-16T00:41:34.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.404+0000 7f818dd6f740 1 -- 172.21.15.33:0/1877316212 shutdown_connections 2024-09-16T00:41:34.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.404+0000 7f818dd6f740 1 --2- 172.21.15.33:0/1877316212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec2fd40 0x55ccaec32130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.404+0000 7f818dd6f740 1 --2- 172.21.15.33:0/1877316212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec25480 0x55ccaec2f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.404+0000 7f818dd6f740 1 --2- 172.21.15.33:0/1877316212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec24f40 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.404+0000 7f818dd6f740 1 -- 172.21.15.33:0/1877316212 >> 172.21.15.33:0/1877316212 conn(0x55ccaec210f0 msgr2=0x55ccaec214d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:34.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.404+0000 7f818dd6f740 1 -- 172.21.15.33:0/1877316212 shutdown_connections 2024-09-16T00:41:34.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.404+0000 7f818dd6f740 1 -- 172.21.15.33:0/1877316212 wait complete. 2024-09-16T00:41:34.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.404+0000 7f818dd6f740 1 -- start start 2024-09-16T00:41:34.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.404+0000 7f818dd6f740 1 Processor -- start 2024-09-16T00:41:34.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f818dd6f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec2cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f818dd6f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec25480 0x55ccaec2d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f818dd6f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec2fd40 0x55ccaec2d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f818dd6f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ccaec32910 con 0x55ccaec24b60 2024-09-16T00:41:34.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f818dd6f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ccaeb44f10 con 0x55ccaec25480 2024-09-16T00:41:34.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f818dd6f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ccaec32a90 con 0x55ccaec2fd40 2024-09-16T00:41:34.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f818dd6f740 10 client.0 _ll_register_callbacks cb 0x55ccaeb0da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:34.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f8189732640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec2cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f8189732640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec2cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35892/0 (socket says 172.21.15.33:35892) 2024-09-16T00:41:34.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f8189732640 1 -- 172.21.15.33:0/2990401567 learned_addr learned my addr 172.21.15.33:0/2990401567 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:34.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f8188f31640 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec25480 0x55ccaec2d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f8189f33640 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec2fd40 0x55ccaec2d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f8189732640 1 -- 172.21.15.33:0/2990401567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec2fd40 msgr2=0x55ccaec2d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:34.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f8189732640 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec2fd40 0x55ccaec2d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f8189732640 1 -- 172.21.15.33:0/2990401567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec25480 msgr2=0x55ccaec2d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:34.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f8189732640 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec25480 0x55ccaec2d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f8189732640 1 -- 172.21.15.33:0/2990401567 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccae8fb6a0 con 0x55ccaec24b60 2024-09-16T00:41:34.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f8188f31640 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec25480 0x55ccaec2d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:41:34.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f8189732640 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec2cc50 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f8180098380 tx=0x7f8180009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff51d8d2afb5c6e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:34.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f816cff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:34.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f816cff9640 1 -- 172.21.15.33:0/2990401567 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8180020040 con 0x55ccaec24b60 2024-09-16T00:41:34.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f816cff9640 1 -- 172.21.15.33:0/2990401567 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f81800860b0 con 0x55ccaec24b60 2024-09-16T00:41:34.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.405+0000 7f818dd6f740 1 -- 172.21.15.33:0/2990401567 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f817c022070 con 0x55ccaec24b60 2024-09-16T00:41:34.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.406+0000 7f816cff9640 1 -- 172.21.15.33:0/2990401567 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f818009fd60 con 0x55ccaec24b60 2024-09-16T00:41:34.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.406+0000 7f818dd6f740 10 client.19935 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:34.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.406+0000 7f818dd6f740 1 -- 172.21.15.33:0/2990401567 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ccaec3e590 con 0x55ccaec24b60 2024-09-16T00:41:34.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.406+0000 7f816cff9640 1 -- 172.21.15.33:0/2990401567 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f817c022070 con 0x55ccaec24b60 2024-09-16T00:41:34.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.406+0000 7f816cff9640 1 -- 172.21.15.33:0/2990401567 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f818009cce0 con 0x55ccaec24b60 2024-09-16T00:41:34.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.406+0000 7f818dd6f740 10 client.19935 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:34.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.406+0000 7f818dd6f740 10 client.19935 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:41:34.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.406+0000 7f818dd6f740 20 client.19935 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.406+0000 7f818dd6f740 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccaec3fa50 0x55ccaec41e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.406+0000 7f818dd6f740 4 client.19935 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.406+0000 7f818dd6f740 1 -- 172.21.15.33:0/2990401567 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccaeb977f0 con 0x55ccaec3fa50 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.406+0000 7f8188f31640 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccaec3fa50 0x55ccaec41e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.407+0000 7f816cff9640 10 client.19935 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.407+0000 7f8188f31640 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccaec3fa50 0x55ccaec41e40 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9ef3731486766c9 server_cookie=545beb12ebde099f in_seq=0 out_seq=0 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.407+0000 7f816cff9640 1 -- 172.21.15.33:0/2990401567 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ccaeb977f0 con 0x55ccaec3fa50 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.407+0000 7f816cff9640 10 client.19935 handle_command_reply: tid=0 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.407+0000 7f818dd6f740 1 client.19935 shutdown 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.407+0000 7f818dd6f740 1 -- 172.21.15.33:0/2990401567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 msgr2=0x55ccaec2cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.407+0000 7f818dd6f740 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec2cc50 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f8180098380 tx=0x7f8180009c30 comp rx=0 tx=0).stop 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.408+0000 7f818dd6f740 1 -- 172.21.15.33:0/2990401567 shutdown_connections 2024-09-16T00:41:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.408+0000 7f818dd6f740 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccaec3fa50 0x55ccaec41e40 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.408+0000 7f818dd6f740 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec2fd40 0x55ccaec2d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.408+0000 7f818dd6f740 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec25480 0x55ccaec2d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.408+0000 7f818dd6f740 1 --2- 172.21.15.33:0/2990401567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec2cc50 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.408+0000 7f816cff9640 0 client.19935 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:34.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.408+0000 7f818dd6f740 1 -- 172.21.15.33:0/2990401567 >> 172.21.15.33:0/2990401567 conn(0x55ccaec210f0 msgr2=0x55ccaec20270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:34.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.408+0000 7f818dd6f740 1 -- 172.21.15.33:0/2990401567 shutdown_connections 2024-09-16T00:41:34.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.408+0000 7f818dd6f740 1 -- 172.21.15.33:0/2990401567 wait complete. 2024-09-16T00:41:34.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.408+0000 7f818dd6f740 20 client.19935 trim_cache size 0 max 16384 2024-09-16T00:41:34.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f818dd6f740 1 Processor -- start 2024-09-16T00:41:34.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f818dd6f740 1 -- start start 2024-09-16T00:41:34.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f818dd6f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec275c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f818dd6f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec25480 0x55ccaec47670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f818dd6f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec2fd40 0x55ccaec47bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f818dd6f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f81841b3a90 con 0x55ccaec24b60 2024-09-16T00:41:34.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f818dd6f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f81841b1360 con 0x55ccaec2fd40 2024-09-16T00:41:34.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f818dd6f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f81841af1b0 con 0x55ccaec25480 2024-09-16T00:41:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f8189732640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec275c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f8189732640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec275c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35894/0 (socket says 172.21.15.33:35894) 2024-09-16T00:41:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f8189732640 1 -- 172.21.15.33:0/1080497709 learned_addr learned my addr 172.21.15.33:0/1080497709 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f8188f31640 1 --2- 172.21.15.33:0/1080497709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec25480 0x55ccaec47670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f8189f33640 1 --2- 172.21.15.33:0/1080497709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec2fd40 0x55ccaec47bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f8189732640 1 -- 172.21.15.33:0/1080497709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec25480 msgr2=0x55ccaec47670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f8189732640 1 --2- 172.21.15.33:0/1080497709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec25480 0x55ccaec47670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f8189732640 1 -- 172.21.15.33:0/1080497709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec2fd40 msgr2=0x55ccaec47bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f8189732640 1 --2- 172.21.15.33:0/1080497709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec2fd40 0x55ccaec47bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f8189732640 1 -- 172.21.15.33:0/1080497709 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccaec421b0 con 0x55ccaec24b60 2024-09-16T00:41:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.426+0000 7f8189732640 1 --2- 172.21.15.33:0/1080497709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec275c0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f8180021000 tx=0x7f8180099ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0b0d0d18e32ffd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.427+0000 7f816effd640 1 -- 172.21.15.33:0/1080497709 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8180014020 con 0x55ccaec24b60 2024-09-16T00:41:34.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.427+0000 7f816effd640 1 -- 172.21.15.33:0/1080497709 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81800b4bb0 con 0x55ccaec24b60 2024-09-16T00:41:34.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.427+0000 7f816effd640 1 -- 172.21.15.33:0/1080497709 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81800b4e60 con 0x55ccaec24b60 2024-09-16T00:41:34.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.427+0000 7f818dd6f740 1 -- 172.21.15.33:0/1080497709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 msgr2=0x55ccaec275c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:34.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.427+0000 7f818dd6f740 1 --2- 172.21.15.33:0/1080497709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec275c0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f8180021000 tx=0x7f8180099ec0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.427+0000 7f818dd6f740 1 -- 172.21.15.33:0/1080497709 shutdown_connections 2024-09-16T00:41:34.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.427+0000 7f818dd6f740 1 --2- 172.21.15.33:0/1080497709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec2fd40 0x55ccaec47bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.427+0000 7f818dd6f740 1 --2- 172.21.15.33:0/1080497709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec25480 0x55ccaec47670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.427+0000 7f818dd6f740 1 --2- 172.21.15.33:0/1080497709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec24b60 0x55ccaec275c0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.427+0000 7f818dd6f740 1 -- 172.21.15.33:0/1080497709 >> 172.21.15.33:0/1080497709 conn(0x55ccaec210f0 msgr2=0x55ccaec2ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:34.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.428+0000 7f818dd6f740 1 -- 172.21.15.33:0/1080497709 shutdown_connections 2024-09-16T00:41:34.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.428+0000 7f818dd6f740 1 -- 172.21.15.33:0/1080497709 wait complete. 2024-09-16T00:41:34.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.428+0000 7f818dd6f740 1 -- start start 2024-09-16T00:41:34.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.428+0000 7f818dd6f740 1 Processor -- start 2024-09-16T00:41:34.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.428+0000 7f818dd6f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec24b60 0x55ccaec34210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.428+0000 7f818dd6f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec25480 0x55ccaec34750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.428+0000 7f818dd6f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec2fd40 0x55ccaeca9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.428+0000 7f818dd6f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f81841af820 con 0x55ccaec25480 2024-09-16T00:41:34.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.428+0000 7f818dd6f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f81841b0c90 con 0x55ccaec24b60 2024-09-16T00:41:34.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.428+0000 7f818dd6f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f81841afef0 con 0x55ccaec2fd40 2024-09-16T00:41:34.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.428+0000 7f818dd6f740 10 client.0 _ll_register_callbacks cb 0x7f818000dea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:34.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f8188f31640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec25480 0x55ccaec34750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f8188f31640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec25480 0x55ccaec34750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35896/0 (socket says 172.21.15.33:35896) 2024-09-16T00:41:34.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f8188f31640 1 -- 172.21.15.33:0/2209180015 learned_addr learned my addr 172.21.15.33:0/2209180015 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:34.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f8189f33640 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec2fd40 0x55ccaeca9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f8189732640 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec24b60 0x55ccaec34210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f8188f31640 1 -- 172.21.15.33:0/2209180015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec2fd40 msgr2=0x55ccaeca9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:34.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f8188f31640 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec2fd40 0x55ccaeca9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f8188f31640 1 -- 172.21.15.33:0/2209180015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec24b60 msgr2=0x55ccaec34210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:34.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f8188f31640 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec24b60 0x55ccaec34210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f8188f31640 1 -- 172.21.15.33:0/2209180015 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccaeca9c30 con 0x55ccaec25480 2024-09-16T00:41:34.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f8189732640 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec24b60 0x55ccaec34210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:41:34.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f8188f31640 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec25480 0x55ccaec34750 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f81740092a0 tx=0x7f81740040d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9762429d0a7fff35 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:34.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f816cff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:34.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f816cff9640 1 -- 172.21.15.33:0/2209180015 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8174035040 con 0x55ccaec25480 2024-09-16T00:41:34.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f816cff9640 1 -- 172.21.15.33:0/2209180015 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8174005040 con 0x55ccaec25480 2024-09-16T00:41:34.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f818dd6f740 1 -- 172.21.15.33:0/2209180015 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81800b4bb0 con 0x55ccaec25480 2024-09-16T00:41:34.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f816cff9640 1 -- 172.21.15.33:0/2209180015 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8174051050 con 0x55ccaec25480 2024-09-16T00:41:34.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f818dd6f740 10 client.19941 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:34.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.429+0000 7f818dd6f740 1 -- 172.21.15.33:0/2209180015 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ccaecaafe0 con 0x55ccaec25480 2024-09-16T00:41:34.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.430+0000 7f816cff9640 1 -- 172.21.15.33:0/2209180015 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81800b4bb0 con 0x55ccaec25480 2024-09-16T00:41:34.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.430+0000 7f816cff9640 1 -- 172.21.15.33:0/2209180015 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f81740035f0 con 0x55ccaec25480 2024-09-16T00:41:34.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.430+0000 7f818dd6f740 10 client.19941 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:34.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.430+0000 7f818dd6f740 10 client.19941 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:41:34.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.430+0000 7f818dd6f740 20 client.19941 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:34.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.430+0000 7f818dd6f740 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccaecab7e0 0x55ccaecadc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:34.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.430+0000 7f818dd6f740 4 client.19941 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:41:34.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.430+0000 7f818dd6f740 1 -- 172.21.15.33:0/2209180015 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ccaeb91d00 con 0x55ccaecab7e0 2024-09-16T00:41:34.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.430+0000 7f8189732640 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccaecab7e0 0x55ccaecadc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:34.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.431+0000 7f8189732640 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccaecab7e0 0x55ccaecadc00 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9b88703fc71d508 server_cookie=af9d5e6c4ad19231 in_seq=0 out_seq=0 2024-09-16T00:41:34.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.431+0000 7f816cff9640 10 client.19941 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:34.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.432+0000 7f816cff9640 1 -- 172.21.15.33:0/2209180015 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ccaeb91d00 con 0x55ccaecab7e0 2024-09-16T00:41:34.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.432+0000 7f816cff9640 10 client.19941 handle_command_reply: tid=0 2024-09-16T00:41:34.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.432+0000 7f818dd6f740 1 client.19941 shutdown 2024-09-16T00:41:34.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.432+0000 7f818dd6f740 1 -- 172.21.15.33:0/2209180015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec25480 msgr2=0x55ccaec34750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:34.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.432+0000 7f818dd6f740 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec25480 0x55ccaec34750 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f81740092a0 tx=0x7f81740040d0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.432+0000 7f818dd6f740 1 -- 172.21.15.33:0/2209180015 shutdown_connections 2024-09-16T00:41:34.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.432+0000 7f818dd6f740 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ccaecab7e0 0x55ccaecadc00 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818dd6f740 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ccaec2fd40 0x55ccaeca9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818dd6f740 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccaec25480 0x55ccaec34750 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f816cff9640 0 client.19941 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:34.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818dd6f740 1 --2- 172.21.15.33:0/2209180015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ccaec24b60 0x55ccaec34210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818dd6f740 1 -- 172.21.15.33:0/2209180015 >> 172.21.15.33:0/2209180015 conn(0x55ccaec210f0 msgr2=0x55ccaec3ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:34.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818dd6f740 1 -- 172.21.15.33:0/2209180015 shutdown_connections 2024-09-16T00:41:34.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818dd6f740 1 -- 172.21.15.33:0/2209180015 wait complete. 2024-09-16T00:41:34.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818dd6f740 20 client.19941 trim_cache size 0 max 16384 2024-09-16T00:41:34.439 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:41:34.439 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:41:34.440 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "4d0baafc-c6a6-439d-bf0a-2a033c35b22a", 2024-09-16T00:41:34.440 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:41:34.440 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:41:34.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818b996640 1 -- 172.21.15.33:0/277951546 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81700a32c0 msgr2=0x7f81700a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:34.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818b996640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81700a32c0 0x7f81700a5780 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f817c021000 tx=0x7f817c01f000 comp rx=0 tx=0).stop 2024-09-16T00:41:34.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818b996640 1 -- 172.21.15.33:0/277951546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81841009d0 msgr2=0x7f81841a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:34.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818b996640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81841009d0 0x7f81841a03f0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f818000d770 tx=0x7f818000dc30 comp rx=0 tx=0).stop 2024-09-16T00:41:34.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818b996640 1 -- 172.21.15.33:0/277951546 shutdown_connections 2024-09-16T00:41:34.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818b996640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f81700a32c0 0x7f81700a5780 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818b996640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f818410e4f0 0x7f818419a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.433+0000 7f818b996640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8184101380 0x7f81841a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.434+0000 7f818b996640 1 --2- 172.21.15.33:0/277951546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81841009d0 0x7f81841a03f0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:34.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.434+0000 7f818b996640 1 -- 172.21.15.33:0/277951546 >> 172.21.15.33:0/277951546 conn(0x7f81840fc7d0 msgr2=0x7f81840febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:34.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.434+0000 7f818b996640 1 -- 172.21.15.33:0/277951546 shutdown_connections 2024-09-16T00:41:34.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:34.434+0000 7f818b996640 1 -- 172.21.15.33:0/277951546 wait complete. 2024-09-16T00:41:35.104 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.34 MB/sec execute 521 sec latency 10771.460 ms 2024-09-16T00:41:35.161 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:41:35.419 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:35.846 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: pgmap v1083: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.7 MiB/s wr, 271 op/s 2024-09-16T00:41:35.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:41:35.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:35.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:41:35.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:41:35.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:41:35.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:35.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:35.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:41:35.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:41:35.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:35.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:41:35.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:35.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:41:35.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:35.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:35.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:41:35.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:35.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:41:35.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:35.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:41:35.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:35.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:35 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:35.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.929+0000 7fe9e68a3640 1 -- 172.21.15.33:0/20354850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9e0102ef0 msgr2=0x7fe9e010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:35.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.929+0000 7fe9e68a3640 1 --2- 172.21.15.33:0/20354850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9e0102ef0 0x7fe9e010da40 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009ca0 tx=0x7fe9d802a950 comp rx=0 tx=0).stop 2024-09-16T00:41:35.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.929+0000 7fe9e68a3640 1 -- 172.21.15.33:0/20354850 shutdown_connections 2024-09-16T00:41:35.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.929+0000 7fe9e68a3640 1 --2- 172.21.15.33:0/20354850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9e0102ef0 0x7fe9e010da40 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.929+0000 7fe9e68a3640 1 --2- 172.21.15.33:0/20354850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9e0102570 0x7fe9e01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.929+0000 7fe9e68a3640 1 --2- 172.21.15.33:0/20354850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9e0100e20 0x7fe9e0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.929+0000 7fe9e68a3640 1 -- 172.21.15.33:0/20354850 >> 172.21.15.33:0/20354850 conn(0x7fe9e00fc7d0 msgr2=0x7fe9e00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:35.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.929+0000 7fe9e68a3640 1 -- 172.21.15.33:0/20354850 shutdown_connections 2024-09-16T00:41:35.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.929+0000 7fe9e68a3640 1 -- 172.21.15.33:0/20354850 wait complete. 2024-09-16T00:41:35.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.929+0000 7fe9e68a3640 1 Processor -- start 2024-09-16T00:41:35.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9e68a3640 1 -- start start 2024-09-16T00:41:35.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9e68a3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9e0100e20 0x7fe9e01984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9e68a3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9e0102570 0x7fe9e01989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9e68a3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9e0102ef0 0x7fe9e019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9e68a3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01120a0 con 0x7fe9e0102570 2024-09-16T00:41:35.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9e68a3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e0111f20 con 0x7fe9e0102ef0 2024-09-16T00:41:35.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9e68a3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e0112220 con 0x7fe9e0100e20 2024-09-16T00:41:35.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9df7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9e0102570 0x7fe9e01989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9df7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9e0102570 0x7fe9e01989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35916/0 (socket says 172.21.15.33:35916) 2024-09-16T00:41:35.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9df7fe640 1 -- 172.21.15.33:0/2608022177 learned_addr learned my addr 172.21.15.33:0/2608022177 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:35.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9dffff640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9e0100e20 0x7fe9e01984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9e4e40640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9e0102ef0 0x7fe9e019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9df7fe640 1 -- 172.21.15.33:0/2608022177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9e0100e20 msgr2=0x7fe9e01984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:35.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9df7fe640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9e0100e20 0x7fe9e01984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9df7fe640 1 -- 172.21.15.33:0/2608022177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9e0102ef0 msgr2=0x7fe9e019cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:35.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9df7fe640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9e0102ef0 0x7fe9e019cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9df7fe640 1 -- 172.21.15.33:0/2608022177 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e019d450 con 0x7fe9e0102570 2024-09-16T00:41:35.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9e4e40640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9e0102ef0 0x7fe9e019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:41:35.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.930+0000 7fe9dffff640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9e0100e20 0x7fe9e01984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:41:35.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.931+0000 7fe9df7fe640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9e0102570 0x7fe9e01989e0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400e9f0 tx=0x7fe9d400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:35.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.931+0000 7fe9dd7fa640 1 -- 172.21.15.33:0/2608022177 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d400cd80 con 0x7fe9e0102570 2024-09-16T00:41:35.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.931+0000 7fe9dd7fa640 1 -- 172.21.15.33:0/2608022177 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d4005500 con 0x7fe9e0102570 2024-09-16T00:41:35.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.931+0000 7fe9e68a3640 1 -- 172.21.15.33:0/2608022177 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e019d740 con 0x7fe9e0102570 2024-09-16T00:41:35.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.931+0000 7fe9dd7fa640 1 -- 172.21.15.33:0/2608022177 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4010690 con 0x7fe9e0102570 2024-09-16T00:41:35.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.931+0000 7fe9e68a3640 1 -- 172.21.15.33:0/2608022177 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e01a5020 con 0x7fe9e0102570 2024-09-16T00:41:35.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.933+0000 7fe9dd7fa640 1 -- 172.21.15.33:0/2608022177 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe9d40046c0 con 0x7fe9e0102570 2024-09-16T00:41:35.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.933+0000 7fe9e8c7c740 1 Processor -- start 2024-09-16T00:41:35.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.933+0000 7fe9e8c7c740 1 -- start start 2024-09-16T00:41:35.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9e8c7c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bba380 0x561626bba760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9e8c7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bbaca0 0x561626bc9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9e8c7c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 0x561626bcbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9e8c7c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561626b202e0 con 0x561626bc9a90 2024-09-16T00:41:35.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9e8c7c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561626ac7ce0 con 0x561626bba380 2024-09-16T00:41:35.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9e8c7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561626b1c2a0 con 0x561626bbaca0 2024-09-16T00:41:35.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9df7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 0x561626bcbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9e4e40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bba380 0x561626bba760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9df7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 0x561626bcbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35930/0 (socket says 172.21.15.33:35930) 2024-09-16T00:41:35.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9df7fe640 1 -- 172.21.15.33:0/2095384443 learned_addr learned my addr 172.21.15.33:0/2095384443 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:35.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9dd7fa640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9c80a3240 0x7fe9c80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9dffff640 1 --2- 172.21.15.33:0/2095384443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bbaca0 0x561626bc9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9e4e40640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9c80a3240 0x7fe9c80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9dd7fa640 1 -- 172.21.15.33:0/2608022177 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe9d40d0350 con 0x7fe9e0102570 2024-09-16T00:41:35.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9dffff640 1 -- 172.21.15.33:0/2095384443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bba380 msgr2=0x561626bba760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:35.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9dffff640 1 --2- 172.21.15.33:0/2095384443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bba380 0x561626bba760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9dffff640 1 -- 172.21.15.33:0/2095384443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 msgr2=0x561626bcbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:35.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9dffff640 1 --2- 172.21.15.33:0/2095384443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 0x561626bcbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9dffff640 1 -- 172.21.15.33:0/2095384443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561626aac440 con 0x561626bbaca0 2024-09-16T00:41:35.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.934+0000 7fe9df7fe640 1 --2- 172.21.15.33:0/2095384443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 0x561626bcbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:41:35.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.935+0000 7fe9dffff640 1 --2- 172.21.15.33:0/2095384443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bbaca0 0x561626bc9550 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009c30 tx=0x7fe9d00117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a064270f9d0cd13 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:35.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.935+0000 7fe9e4e40640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9c80a3240 0x7fe9c80a5700 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8030040 tx=0x7fe9d802a950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:35.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.935+0000 7fe9c67fc640 1 -- 172.21.15.33:0/2095384443 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0012450 con 0x561626bbaca0 2024-09-16T00:41:35.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.935+0000 7fe9c67fc640 1 -- 172.21.15.33:0/2095384443 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d0012ad0 con 0x561626bbaca0 2024-09-16T00:41:35.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.935+0000 7fe9c67fc640 1 -- 172.21.15.33:0/2095384443 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000fab0 con 0x561626bbaca0 2024-09-16T00:41:35.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/2095384443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bbaca0 msgr2=0x561626bc9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:35.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/2095384443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bbaca0 0x561626bc9550 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009c30 tx=0x7fe9d00117b0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/2095384443 shutdown_connections 2024-09-16T00:41:35.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/2095384443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 0x561626bcbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/2095384443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bbaca0 0x561626bc9550 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/2095384443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bba380 0x561626bba760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/2095384443 >> 172.21.15.33:0/2095384443 conn(0x561626bb6a30 msgr2=0x561626bb6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:35.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/2095384443 shutdown_connections 2024-09-16T00:41:35.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/2095384443 wait complete. 2024-09-16T00:41:35.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 -- start start 2024-09-16T00:41:35.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 Processor -- start 2024-09-16T00:41:35.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 0x561626bc9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 0x561626bc6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.936+0000 7fe9e8c7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bc9a90 0x561626bc6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9e8c7c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561626bcc660 con 0x561626bba380 2024-09-16T00:41:35.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9e8c7c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561626adb050 con 0x561626bbaca0 2024-09-16T00:41:35.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9e8c7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561626bcc7e0 con 0x561626bc9a90 2024-09-16T00:41:35.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9e8c7c740 10 client.0 _ll_register_callbacks cb 0x561626b1b160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:35.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 0x561626bc9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 0x561626bc9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35944/0 (socket says 172.21.15.33:35944) 2024-09-16T00:41:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9dffff640 1 -- 172.21.15.33:0/3954366523 learned_addr learned my addr 172.21.15.33:0/3954366523 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9df7fe640 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 0x561626bc6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9e4e40640 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bc9a90 0x561626bc6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9dffff640 1 -- 172.21.15.33:0/3954366523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bc9a90 msgr2=0x561626bc6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9dffff640 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bc9a90 0x561626bc6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9dffff640 1 -- 172.21.15.33:0/3954366523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 msgr2=0x561626bc6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9dffff640 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 0x561626bc6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9dffff640 1 -- 172.21.15.33:0/3954366523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616268913c0 con 0x561626bba380 2024-09-16T00:41:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9dffff640 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 0x561626bc9360 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0000c00 tx=0x7fe9d0012f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e72c9e96a9c179 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9bf7fe640 1 -- 172.21.15.33:0/3954366523 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0029020 con 0x561626bba380 2024-09-16T00:41:35.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9bf7fe640 1 -- 172.21.15.33:0/3954366523 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe9d0012660 con 0x561626bba380 2024-09-16T00:41:35.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3954366523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9d0012ad0 con 0x561626bba380 2024-09-16T00:41:35.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9bf7fe640 1 -- 172.21.15.33:0/3954366523 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe9d0034050 con 0x561626bba380 2024-09-16T00:41:35.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9e8c7c740 10 client.19953 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:35.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.937+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3954366523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561626bd8020 con 0x561626bba380 2024-09-16T00:41:35.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.938+0000 7fe9bf7fe640 1 -- 172.21.15.33:0/3954366523 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d0012ad0 con 0x561626bba380 2024-09-16T00:41:35.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.938+0000 7fe9bf7fe640 1 -- 172.21.15.33:0/3954366523 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fe9d000fce0 con 0x561626bba380 2024-09-16T00:41:35.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.938+0000 7fe9e8c7c740 10 client.19953 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:35.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.938+0000 7fe9e8c7c740 10 client.19953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:41:35.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.938+0000 7fe9e8c7c740 20 client.19953 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:35.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.938+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561626bd9480 0x561626bdb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.938+0000 7fe9e8c7c740 4 client.19953 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:41:35.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.938+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3954366523 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561626b2d850 con 0x561626bd9480 2024-09-16T00:41:35.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.938+0000 7fe9df7fe640 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561626bd9480 0x561626bdb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9df7fe640 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561626bd9480 0x561626bdb870 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae409b1fee10f72e server_cookie=ef62a534ad147a3 in_seq=0 out_seq=0 2024-09-16T00:41:35.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9bf7fe640 10 client.19953 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:35.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9bf7fe640 1 -- 172.21.15.33:0/3954366523 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561626b2d850 con 0x561626bd9480 2024-09-16T00:41:35.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9bf7fe640 10 client.19953 handle_command_reply: tid=0 2024-09-16T00:41:35.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9e8c7c740 1 client.19953 shutdown 2024-09-16T00:41:35.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3954366523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 msgr2=0x561626bc9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:35.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 0x561626bc9360 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0000c00 tx=0x7fe9d0012f30 comp rx=0 tx=0).stop 2024-09-16T00:41:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3954366523 shutdown_connections 2024-09-16T00:41:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561626bd9480 0x561626bdb870 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bc9a90 0x561626bc6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 0x561626bc6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9bf7fe640 0 client.19953 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.939+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/3954366523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 0x561626bc9360 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.940+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3954366523 >> 172.21.15.33:0/3954366523 conn(0x561626bb6a30 msgr2=0x561626bbc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.940+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3954366523 shutdown_connections 2024-09-16T00:41:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.940+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3954366523 wait complete. 2024-09-16T00:41:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.940+0000 7fe9e8c7c740 20 client.19953 trim_cache size 0 max 16384 2024-09-16T00:41:35.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.957+0000 7fe9e8c7c740 1 Processor -- start 2024-09-16T00:41:35.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.957+0000 7fe9e8c7c740 1 -- start start 2024-09-16T00:41:35.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9e8c7c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 0x561626bb5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9e8c7c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 0x561626bc5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9e8c7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bc9a90 0x561626be0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9e8c7c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01af740 con 0x561626bba380 2024-09-16T00:41:35.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9e8c7c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01ad010 con 0x561626bbaca0 2024-09-16T00:41:35.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9e8c7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01aae60 con 0x561626bc9a90 2024-09-16T00:41:35.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 0x561626bb5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 0x561626bb5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35958/0 (socket says 172.21.15.33:35958) 2024-09-16T00:41:35.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9dffff640 1 -- 172.21.15.33:0/3005255227 learned_addr learned my addr 172.21.15.33:0/3005255227 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:35.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9df7fe640 1 --2- 172.21.15.33:0/3005255227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 0x561626bc5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9e4e40640 1 --2- 172.21.15.33:0/3005255227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bc9a90 0x561626be0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9dffff640 1 -- 172.21.15.33:0/3005255227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bc9a90 msgr2=0x561626be0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:35.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9dffff640 1 --2- 172.21.15.33:0/3005255227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bc9a90 0x561626be0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9dffff640 1 -- 172.21.15.33:0/3005255227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 msgr2=0x561626bc5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:35.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9dffff640 1 --2- 172.21.15.33:0/3005255227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 0x561626bc5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9dffff640 1 -- 172.21.15.33:0/3005255227 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561626bc2260 con 0x561626bba380 2024-09-16T00:41:35.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9e4e40640 1 --2- 172.21.15.33:0/3005255227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bc9a90 0x561626be0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:41:35.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9dffff640 1 --2- 172.21.15.33:0/3005255227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 0x561626bb5e20 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00117f0 tx=0x7fe9d0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1facc2a48fc8e22 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:35.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.958+0000 7fe9c5ffb640 1 -- 172.21.15.33:0/3005255227 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d003d070 con 0x561626bba380 2024-09-16T00:41:35.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.959+0000 7fe9c5ffb640 1 -- 172.21.15.33:0/3005255227 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d00092a0 con 0x561626bba380 2024-09-16T00:41:35.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.959+0000 7fe9c5ffb640 1 -- 172.21.15.33:0/3005255227 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0029030 con 0x561626bba380 2024-09-16T00:41:35.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.959+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3005255227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 msgr2=0x561626bb5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:35.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.959+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/3005255227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 0x561626bb5e20 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00117f0 tx=0x7fe9d0004600 comp rx=0 tx=0).stop 2024-09-16T00:41:35.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.959+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3005255227 shutdown_connections 2024-09-16T00:41:35.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.959+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/3005255227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bc9a90 0x561626be0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.959+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/3005255227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 0x561626bc5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.959+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/3005255227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bba380 0x561626bb5e20 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.959+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3005255227 >> 172.21.15.33:0/3005255227 conn(0x561626bb6a30 msgr2=0x561626bbc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:35.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.959+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3005255227 shutdown_connections 2024-09-16T00:41:35.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.959+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/3005255227 wait complete. 2024-09-16T00:41:35.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9e8c7c740 1 -- start start 2024-09-16T00:41:35.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9e8c7c740 1 Processor -- start 2024-09-16T00:41:35.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9e8c7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bba380 0x561626bd22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9e8c7c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 0x561626bcd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9e8c7c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 0x561626bcdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9e8c7c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01ab4d0 con 0x561626bc9a90 2024-09-16T00:41:35.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9e8c7c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01ac940 con 0x561626bbaca0 2024-09-16T00:41:35.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9e8c7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01abba0 con 0x561626bba380 2024-09-16T00:41:35.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9e8c7c740 10 client.0 _ll_register_callbacks cb 0x7fe9d4098560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:35.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9e4e40640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 0x561626bcdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9e4e40640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 0x561626bcdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35968/0 (socket says 172.21.15.33:35968) 2024-09-16T00:41:35.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9e4e40640 1 -- 172.21.15.33:0/1688792889 learned_addr learned my addr 172.21.15.33:0/1688792889 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:35.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.960+0000 7fe9df7fe640 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 0x561626bcd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9dffff640 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bba380 0x561626bd22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9e4e40640 1 -- 172.21.15.33:0/1688792889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bba380 msgr2=0x561626bd22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:35.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9e4e40640 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bba380 0x561626bd22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9e4e40640 1 -- 172.21.15.33:0/1688792889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 msgr2=0x561626bcd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:35.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9e4e40640 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 0x561626bcd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9e4e40640 1 -- 172.21.15.33:0/1688792889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561626bce460 con 0x561626bc9a90 2024-09-16T00:41:35.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9dffff640 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bba380 0x561626bd22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:41:35.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9e4e40640 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 0x561626bcdc30 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80099a0 tx=0x7fe9d80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6247e990b5502253 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:35.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:35.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9bf7fe640 1 -- 172.21.15.33:0/1688792889 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d804f070 con 0x561626bc9a90 2024-09-16T00:41:35.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9bf7fe640 1 -- 172.21.15.33:0/1688792889 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe9d8004450 con 0x561626bc9a90 2024-09-16T00:41:35.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/1688792889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9d00092a0 con 0x561626bc9a90 2024-09-16T00:41:35.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9bf7fe640 1 -- 172.21.15.33:0/1688792889 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe9d8034040 con 0x561626bc9a90 2024-09-16T00:41:35.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9e8c7c740 10 client.19959 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:35.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.961+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/1688792889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561626c43fe0 con 0x561626bc9a90 2024-09-16T00:41:35.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.962+0000 7fe9bf7fe640 1 -- 172.21.15.33:0/1688792889 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d00092a0 con 0x561626bc9a90 2024-09-16T00:41:35.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.962+0000 7fe9bf7fe640 1 -- 172.21.15.33:0/1688792889 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fe9d8048040 con 0x561626bc9a90 2024-09-16T00:41:35.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.962+0000 7fe9e8c7c740 10 client.19959 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:35.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.962+0000 7fe9e8c7c740 10 client.19959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:41:35.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.962+0000 7fe9e8c7c740 20 client.19959 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:35.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.962+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561626c44700 0x561626c46af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:35.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.962+0000 7fe9e8c7c740 4 client.19959 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:41:35.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.962+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/1688792889 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561626b27d60 con 0x561626c44700 2024-09-16T00:41:35.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.962+0000 7fe9dffff640 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561626c44700 0x561626c46af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:35.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.962+0000 7fe9dffff640 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561626c44700 0x561626c46af0 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbeea922138cab80 server_cookie=37867f2a5ff43e0a in_seq=0 out_seq=0 2024-09-16T00:41:35.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.962+0000 7fe9bf7fe640 10 client.19959 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:35.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9bf7fe640 1 -- 172.21.15.33:0/1688792889 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561626b27d60 con 0x561626c44700 2024-09-16T00:41:35.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9bf7fe640 10 client.19959 handle_command_reply: tid=0 2024-09-16T00:41:35.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9e8c7c740 1 client.19959 shutdown 2024-09-16T00:41:35.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/1688792889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 msgr2=0x561626bcdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:35.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 0x561626bcdc30 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80099a0 tx=0x7fe9d80095c0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/1688792889 shutdown_connections 2024-09-16T00:41:35.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561626c44700 0x561626c46af0 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9bf7fe640 0 client.19959 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:35.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561626bc9a90 0x561626bcdc30 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561626bbaca0 0x561626bcd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9e8c7c740 1 --2- 172.21.15.33:0/1688792889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561626bba380 0x561626bd22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/1688792889 >> 172.21.15.33:0/1688792889 conn(0x561626bb6a30 msgr2=0x561626bc7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:35.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/1688792889 shutdown_connections 2024-09-16T00:41:35.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9e8c7c740 1 -- 172.21.15.33:0/1688792889 wait complete. 2024-09-16T00:41:35.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.963+0000 7fe9e8c7c740 20 client.19959 trim_cache size 0 max 16384 2024-09-16T00:41:35.972 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:41:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:41:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:41:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "4d0baafc-c6a6-439d-bf0a-2a033c35b22a": { 2024-09-16T00:41:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:41:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "4d0baafc-c6a6-439d-bf0a-2a033c35b22a", 2024-09-16T00:41:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:41:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:35.973 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:41:35.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.964+0000 7fe9e68a3640 1 -- 172.21.15.33:0/2608022177 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9c80a3240 msgr2=0x7fe9c80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:35.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.964+0000 7fe9e68a3640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9c80a3240 0x7fe9c80a5700 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8030040 tx=0x7fe9d802a950 comp rx=0 tx=0).stop 2024-09-16T00:41:35.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.964+0000 7fe9e68a3640 1 -- 172.21.15.33:0/2608022177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9e0102570 msgr2=0x7fe9e01989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:35.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.964+0000 7fe9e68a3640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9e0102570 0x7fe9e01989e0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400e9f0 tx=0x7fe9d400edf0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.964+0000 7fe9e68a3640 1 -- 172.21.15.33:0/2608022177 shutdown_connections 2024-09-16T00:41:35.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.964+0000 7fe9e68a3640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe9c80a3240 0x7fe9c80a5700 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.964+0000 7fe9e68a3640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9e0102ef0 0x7fe9e019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.964+0000 7fe9e68a3640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9e0102570 0x7fe9e01989e0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.964+0000 7fe9e68a3640 1 --2- 172.21.15.33:0/2608022177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9e0100e20 0x7fe9e01984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:35.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.964+0000 7fe9e68a3640 1 -- 172.21.15.33:0/2608022177 >> 172.21.15.33:0/2608022177 conn(0x7fe9e00fc7d0 msgr2=0x7fe9e0106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:35.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.964+0000 7fe9e68a3640 1 -- 172.21.15.33:0/2608022177 shutdown_connections 2024-09-16T00:41:35.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:35.964+0000 7fe9e68a3640 1 -- 172.21.15.33:0/2608022177 wait complete. 2024-09-16T00:41:36.104 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.30 MB/sec execute 522 sec latency 11771.497 ms 2024-09-16T00:41:36.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: pgmap v1083: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.7 MiB/s wr, 271 op/s 2024-09-16T00:41:36.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:41:36.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:41:36.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:41:36.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:41:36.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:41:36.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:41:36.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:41:36.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:41:36.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:41:36.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:41:36.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:41:36.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:35 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: pgmap v1083: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.7 MiB/s wr, 271 op/s 2024-09-16T00:41:36.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:41:36.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:41:36.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:41:36.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:41:36.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:41:36.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:41:36.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:41:36.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:41:36.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:41:36.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:41:36.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:41:36.232 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.233 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:35 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:41:36.722 INFO:tasks.cephfs.filesystem:scrub status for tag:4d0baafc-c6a6-439d-bf0a-2a033c35b22a - {'path': '/', 'tag': '4d0baafc-c6a6-439d-bf0a-2a033c35b22a', 'options': 'recursive,force'} 2024-09-16T00:41:37.104 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.27 MB/sec execute 523 sec latency 12771.544 ms 2024-09-16T00:41:38.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:37 smithi033 ceph-mon[31243]: pgmap v1084: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 898 KiB/s wr, 162 op/s 2024-09-16T00:41:38.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:41:38.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:41:38.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:41:38.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:41:38.104 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.23 MB/sec execute 524 sec latency 13771.679 ms 2024-09-16T00:41:38.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:37 smithi083 ceph-mon[35242]: pgmap v1084: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 898 KiB/s wr, 162 op/s 2024-09-16T00:41:38.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:41:38.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:41:38.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:41:38.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:41:38.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:37 smithi060 ceph-mon[35185]: pgmap v1084: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 898 KiB/s wr, 162 op/s 2024-09-16T00:41:38.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:41:38.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:41:38.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:41:38.231 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:41:39.105 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.20 MB/sec execute 525 sec latency 14771.781 ms 2024-09-16T00:41:40.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:39 smithi033 ceph-mon[31243]: pgmap v1085: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 468 KiB/s wr, 132 op/s 2024-09-16T00:41:40.105 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.16 MB/sec execute 526 sec latency 15771.921 ms 2024-09-16T00:41:40.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:39 smithi083 ceph-mon[35242]: pgmap v1085: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 468 KiB/s wr, 132 op/s 2024-09-16T00:41:40.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:39 smithi060 ceph-mon[35185]: pgmap v1085: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 468 KiB/s wr, 132 op/s 2024-09-16T00:41:41.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.13 MB/sec execute 527 sec latency 16771.977 ms 2024-09-16T00:41:42.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:41 smithi033 ceph-mon[31243]: pgmap v1086: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 114 KiB/s wr, 84 op/s 2024-09-16T00:41:42.105 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.09 MB/sec execute 528 sec latency 17772.066 ms 2024-09-16T00:41:42.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:41 smithi083 ceph-mon[35242]: pgmap v1086: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 114 KiB/s wr, 84 op/s 2024-09-16T00:41:42.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:41 smithi060 ceph-mon[35185]: pgmap v1086: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 114 KiB/s wr, 84 op/s 2024-09-16T00:41:43.105 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.06 MB/sec execute 529 sec latency 18772.175 ms 2024-09-16T00:41:44.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:43 smithi033 ceph-mon[31243]: pgmap v1087: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 12 KiB/s wr, 65 op/s 2024-09-16T00:41:44.105 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 18.02 MB/sec execute 530 sec latency 19772.281 ms 2024-09-16T00:41:44.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:43 smithi083 ceph-mon[35242]: pgmap v1087: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 12 KiB/s wr, 65 op/s 2024-09-16T00:41:44.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:43 smithi060 ceph-mon[35185]: pgmap v1087: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 12 KiB/s wr, 65 op/s 2024-09-16T00:41:45.105 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 17.99 MB/sec execute 531 sec latency 20772.404 ms 2024-09-16T00:41:46.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:45 smithi033 ceph-mon[31243]: pgmap v1088: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 KiB/s wr, 65 op/s 2024-09-16T00:41:46.105 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 17.96 MB/sec execute 532 sec latency 21772.530 ms 2024-09-16T00:41:46.131 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:41:46.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:45 smithi083 ceph-mon[35242]: pgmap v1088: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 KiB/s wr, 65 op/s 2024-09-16T00:41:46.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:45 smithi060 ceph-mon[35185]: pgmap v1088: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 KiB/s wr, 65 op/s 2024-09-16T00:41:46.637 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:47.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:46 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:41:47.105 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 17.92 MB/sec execute 533 sec latency 22772.629 ms 2024-09-16T00:41:47.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:46 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:41:47.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:46 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:41:47.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.417+0000 7f65ff67c640 1 -- 172.21.15.33:0/718899796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81082a0 msgr2=0x7f65f810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:47.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.417+0000 7f65ff67c640 1 --2- 172.21.15.33:0/718899796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81082a0 0x7f65f810ec40 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009cd0 tx=0x7f65f002a8e0 comp rx=0 tx=0).stop 2024-09-16T00:41:47.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.417+0000 7f65ff67c640 1 -- 172.21.15.33:0/718899796 shutdown_connections 2024-09-16T00:41:47.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.417+0000 7f65ff67c640 1 --2- 172.21.15.33:0/718899796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81082a0 0x7f65f810ec40 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:47.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.417+0000 7f65ff67c640 1 --2- 172.21.15.33:0/718899796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65f81036e0 0x7f65f8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:47.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.417+0000 7f65ff67c640 1 --2- 172.21.15.33:0/718899796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102d30 0x7f65f8103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:47.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.417+0000 7f65ff67c640 1 -- 172.21.15.33:0/718899796 >> 172.21.15.33:0/718899796 conn(0x7f65f80fc7d0 msgr2=0x7f65f80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:47.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.417+0000 7f65ff67c640 1 -- 172.21.15.33:0/718899796 shutdown_connections 2024-09-16T00:41:47.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.417+0000 7f65ff67c640 1 -- 172.21.15.33:0/718899796 wait complete. 2024-09-16T00:41:47.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65ff67c640 1 Processor -- start 2024-09-16T00:41:47.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65ff67c640 1 -- start start 2024-09-16T00:41:47.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65ff67c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102d30 0x7f65f8198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:47.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65ff67c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65f81036e0 0x7f65f8198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:47.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65ff67c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81082a0 0x7f65f819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:47.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65ff67c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f65f81120a0 con 0x7f65f81082a0 2024-09-16T00:41:47.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65ff67c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8111f20 con 0x7f65f81036e0 2024-09-16T00:41:47.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65ff67c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8112220 con 0x7f65f8102d30 2024-09-16T00:41:47.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65fdc19640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81082a0 0x7f65f819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:47.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65fdc19640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81082a0 0x7f65f819ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45108/0 (socket says 172.21.15.33:45108) 2024-09-16T00:41:47.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65fdc19640 1 -- 172.21.15.33:0/2068927942 learned_addr learned my addr 172.21.15.33:0/2068927942 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:47.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65fcc17640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65f81036e0 0x7f65f8198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:47.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.418+0000 7f65fd418640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102d30 0x7f65f8198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.419+0000 7f65fdc19640 1 -- 172.21.15.33:0/2068927942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102d30 msgr2=0x7f65f8198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.419+0000 7f65fdc19640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102d30 0x7f65f8198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.419+0000 7f65fdc19640 1 -- 172.21.15.33:0/2068927942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65f81036e0 msgr2=0x7f65f8198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.419+0000 7f65fdc19640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65f81036e0 0x7f65f8198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.419+0000 7f65fdc19640 1 -- 172.21.15.33:0/2068927942 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f819d500 con 0x7f65f81082a0 2024-09-16T00:41:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.419+0000 7f65fcc17640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65f81036e0 0x7f65f8198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:41:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.419+0000 7f65fdc19640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81082a0 0x7f65f819ce20 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f65f0005cc0 tx=0x7f65f0003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.419+0000 7f65ee7fc640 1 -- 172.21.15.33:0/2068927942 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0004040 con 0x7f65f81082a0 2024-09-16T00:41:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.419+0000 7f65ee7fc640 1 -- 172.21.15.33:0/2068927942 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f00046c0 con 0x7f65f81082a0 2024-09-16T00:41:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.419+0000 7f65ff67c640 1 -- 172.21.15.33:0/2068927942 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f819d790 con 0x7f65f81082a0 2024-09-16T00:41:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.419+0000 7f65ee7fc640 1 -- 172.21.15.33:0/2068927942 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0046480 con 0x7f65f81082a0 2024-09-16T00:41:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.419+0000 7f65ff67c640 1 -- 172.21.15.33:0/2068927942 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f81a50d0 con 0x7f65f81082a0 2024-09-16T00:41:47.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.420+0000 7f65ff67c640 1 -- 172.21.15.33:0/2068927942 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65f8103110 con 0x7f65f81082a0 2024-09-16T00:41:47.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.421+0000 7f65ee7fc640 1 -- 172.21.15.33:0/2068927942 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f65f00305a0 con 0x7f65f81082a0 2024-09-16T00:41:47.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.422+0000 7f65ee7fc640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f65e40a32c0 0x7f65e40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:47.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.422+0000 7f65ee7fc640 1 -- 172.21.15.33:0/2068927942 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f65f002e030 con 0x7f65f81082a0 2024-09-16T00:41:47.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.422+0000 7f65fd418640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f65e40a32c0 0x7f65e40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:47.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.423+0000 7f65fd418640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f65e40a32c0 0x7f65e40a5780 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f65f4005fd0 tx=0x7f65f400b590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:47.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.425+0000 7f65ee7fc640 1 -- 172.21.15.33:0/2068927942 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f65f004b060 con 0x7f65f81082a0 2024-09-16T00:41:47.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.669+0000 7f65ff67c640 1 -- 172.21.15.33:0/2068927942 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65f81998b0 con 0x7f65f81082a0 2024-09-16T00:41:47.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.670+0000 7f65ee7fc640 1 -- 172.21.15.33:0/2068927942 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f65f81998b0 con 0x7f65f81082a0 2024-09-16T00:41:47.671 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:41:47.671 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:41:47.671 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:41:47.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.672+0000 7f65ff67c640 1 -- 172.21.15.33:0/2068927942 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f65e40a32c0 msgr2=0x7f65e40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:47.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.672+0000 7f65ff67c640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f65e40a32c0 0x7f65e40a5780 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f65f4005fd0 tx=0x7f65f400b590 comp rx=0 tx=0).stop 2024-09-16T00:41:47.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.672+0000 7f65ff67c640 1 -- 172.21.15.33:0/2068927942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81082a0 msgr2=0x7f65f819ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:47.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.672+0000 7f65ff67c640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81082a0 0x7f65f819ce20 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f65f0005cc0 tx=0x7f65f0003970 comp rx=0 tx=0).stop 2024-09-16T00:41:47.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.672+0000 7f65ff67c640 1 -- 172.21.15.33:0/2068927942 shutdown_connections 2024-09-16T00:41:47.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.672+0000 7f65ff67c640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f65e40a32c0 0x7f65e40a5780 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:47.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.672+0000 7f65ff67c640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81082a0 0x7f65f819ce20 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:47.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.672+0000 7f65ff67c640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65f81036e0 0x7f65f8198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:47.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.672+0000 7f65ff67c640 1 --2- 172.21.15.33:0/2068927942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102d30 0x7f65f8198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:47.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.672+0000 7f65ff67c640 1 -- 172.21.15.33:0/2068927942 >> 172.21.15.33:0/2068927942 conn(0x7f65f80fc7d0 msgr2=0x7f65f8105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:47.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.672+0000 7f65ff67c640 1 -- 172.21.15.33:0/2068927942 shutdown_connections 2024-09-16T00:41:47.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:47.672+0000 7f65ff67c640 1 -- 172.21.15.33:0/2068927942 wait complete. 2024-09-16T00:41:48.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:47 smithi033 ceph-mon[31243]: pgmap v1089: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 10 KiB/s wr, 62 op/s 2024-09-16T00:41:48.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:47 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:41:48.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:47 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:41:48.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:47 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2068927942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:41:48.105 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 17.89 MB/sec execute 534 sec latency 23772.679 ms 2024-09-16T00:41:48.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:47 smithi083 ceph-mon[35242]: pgmap v1089: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 10 KiB/s wr, 62 op/s 2024-09-16T00:41:48.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:47 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:41:48.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:47 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:41:48.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:47 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2068927942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:41:48.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:47 smithi060 ceph-mon[35185]: pgmap v1089: 129 pgs: 129 active+clean; 705 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 10 KiB/s wr, 62 op/s 2024-09-16T00:41:48.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:47 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:41:48.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:47 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:41:48.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:47 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2068927942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:41:48.383 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 987c2a04 --release --await' 2024-09-16T00:41:48.383 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 987c2a04 --release --await 2024-09-16T00:41:48.639 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:49.105 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221143 17.86 MB/sec execute 535 sec latency 24772.726 ms 2024-09-16T00:41:49.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.143+0000 7f873c6a8640 1 -- 172.21.15.33:0/359475411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8734101de0 msgr2=0x7f87341021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:49.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.143+0000 7f873c6a8640 1 --2- 172.21.15.33:0/359475411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8734101de0 0x7f87341021c0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f8730009900 tx=0x7f873002a410 comp rx=0 tx=0).stop 2024-09-16T00:41:49.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.143+0000 7f873c6a8640 1 -- 172.21.15.33:0/359475411 shutdown_connections 2024-09-16T00:41:49.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.143+0000 7f873c6a8640 1 --2- 172.21.15.33:0/359475411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8734106f10 0x7f873410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.143+0000 7f873c6a8640 1 --2- 172.21.15.33:0/359475411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8734102790 0x7f87341067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.143+0000 7f873c6a8640 1 --2- 172.21.15.33:0/359475411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8734101de0 0x7f87341021c0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.143+0000 7f873c6a8640 1 -- 172.21.15.33:0/359475411 >> 172.21.15.33:0/359475411 conn(0x7f87340fd8c0 msgr2=0x7f87340ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:49.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.144+0000 7f873c6a8640 1 -- 172.21.15.33:0/359475411 shutdown_connections 2024-09-16T00:41:49.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.144+0000 7f873c6a8640 1 -- 172.21.15.33:0/359475411 wait complete. 2024-09-16T00:41:49.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.144+0000 7f873c6a8640 1 Processor -- start 2024-09-16T00:41:49.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.144+0000 7f873c6a8640 1 -- start start 2024-09-16T00:41:49.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f873c6a8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8734101de0 0x7f8734198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f873c6a8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8734102790 0x7f87341989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f873c6a8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8734106f10 0x7f873419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f873c6a8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8734111f70 con 0x7f8734102790 2024-09-16T00:41:49.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f873c6a8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8734111df0 con 0x7f8734101de0 2024-09-16T00:41:49.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f873c6a8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f87341120f0 con 0x7f8734106f10 2024-09-16T00:41:49.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f8739c43640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8734102790 0x7f87341989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f8739c43640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8734102790 0x7f87341989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45126/0 (socket says 172.21.15.33:45126) 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f8739c43640 1 -- 172.21.15.33:0/4234452037 learned_addr learned my addr 172.21.15.33:0/4234452037 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f873a444640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8734101de0 0x7f8734198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f873ac45640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8734106f10 0x7f873419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f8739c43640 1 -- 172.21.15.33:0/4234452037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8734106f10 msgr2=0x7f873419cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f8739c43640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8734106f10 0x7f873419cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f8739c43640 1 -- 172.21.15.33:0/4234452037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8734101de0 msgr2=0x7f8734198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f8739c43640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8734101de0 0x7f8734198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f8739c43640 1 -- 172.21.15.33:0/4234452037 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f873419d430 con 0x7f8734102790 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.145+0000 7f8739c43640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8734102790 0x7f87341989c0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f872400c890 tx=0x7f872400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.146+0000 7f872b7fe640 1 -- 172.21.15.33:0/4234452037 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8724004220 con 0x7f8734102790 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.146+0000 7f872b7fe640 1 -- 172.21.15.33:0/4234452037 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87240043c0 con 0x7f8734102790 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.146+0000 7f873c6a8640 1 -- 172.21.15.33:0/4234452037 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f873419d720 con 0x7f8734102790 2024-09-16T00:41:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.146+0000 7f872b7fe640 1 -- 172.21.15.33:0/4234452037 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87240049e0 con 0x7f8734102790 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.146+0000 7f873c6a8640 1 -- 172.21.15.33:0/4234452037 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87341a5000 con 0x7f8734102790 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.148+0000 7f872b7fe640 1 -- 172.21.15.33:0/4234452037 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f87240079b0 con 0x7f8734102790 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.148+0000 7f873ea81740 1 Processor -- start 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f873ea81740 1 -- start start 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f873ea81740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7060 0x55ad9dfa7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f873ea81740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7980 0x55ad9dfb1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f873ea81740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfb4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f873ea81740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ad9df0c620 con 0x55ad9dfb2240 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f873ea81740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ad9deb3e50 con 0x55ad9dfa7980 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f873ea81740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ad9df085e0 con 0x55ad9dfa7060 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f8739c43640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfb4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f8739c43640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfb4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45132/0 (socket says 172.21.15.33:45132) 2024-09-16T00:41:49.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f8739c43640 1 -- 172.21.15.33:0/398359575 learned_addr learned my addr 172.21.15.33:0/398359575 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:49.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f873a444640 1 --2- 172.21.15.33:0/398359575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7980 0x55ad9dfb1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f873ac45640 1 --2- 172.21.15.33:0/398359575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7060 0x55ad9dfa7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f872b7fe640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f87200a3240 0x7f87200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.149+0000 7f873ac45640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f87200a3240 0x7f87200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.150+0000 7f8739c43640 1 -- 172.21.15.33:0/398359575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7060 msgr2=0x55ad9dfa7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:49.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.150+0000 7f8739c43640 1 --2- 172.21.15.33:0/398359575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7060 0x55ad9dfa7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.150+0000 7f8739c43640 1 -- 172.21.15.33:0/398359575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7980 msgr2=0x55ad9dfb1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:49.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.150+0000 7f8739c43640 1 --2- 172.21.15.33:0/398359575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7980 0x55ad9dfb1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.150+0000 7f8739c43640 1 -- 172.21.15.33:0/398359575 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad9dea4d40 con 0x55ad9dfb2240 2024-09-16T00:41:49.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.150+0000 7f872b7fe640 1 -- 172.21.15.33:0/4234452037 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f87240cfe50 con 0x7f8734102790 2024-09-16T00:41:49.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.150+0000 7f873a444640 1 --2- 172.21.15.33:0/398359575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7980 0x55ad9dfb1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:41:49.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.150+0000 7f8739c43640 1 --2- 172.21.15.33:0/398359575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfb4630 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f872409bc90 tx=0x7f8724098360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255263bd72db76dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.150+0000 7f873ac45640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f87200a3240 0x7f87200a5700 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f872c005fd0 tx=0x7f872c004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.150+0000 7f8728ff9640 1 -- 172.21.15.33:0/398359575 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872401d070 con 0x55ad9dfb2240 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.150+0000 7f8728ff9640 1 -- 172.21.15.33:0/398359575 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f872409c690 con 0x55ad9dfb2240 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.150+0000 7f8728ff9640 1 -- 172.21.15.33:0/398359575 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87240a0340 con 0x55ad9dfb2240 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.151+0000 7f873ea81740 1 -- 172.21.15.33:0/398359575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 msgr2=0x55ad9dfb4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.151+0000 7f873ea81740 1 --2- 172.21.15.33:0/398359575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfb4630 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f872409bc90 tx=0x7f8724098360 comp rx=0 tx=0).stop 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.151+0000 7f873ea81740 1 -- 172.21.15.33:0/398359575 shutdown_connections 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.151+0000 7f873ea81740 1 --2- 172.21.15.33:0/398359575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfb4630 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.151+0000 7f873ea81740 1 --2- 172.21.15.33:0/398359575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7980 0x55ad9dfb1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.151+0000 7f873ea81740 1 --2- 172.21.15.33:0/398359575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7060 0x55ad9dfa7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.151+0000 7f873ea81740 1 -- 172.21.15.33:0/398359575 >> 172.21.15.33:0/398359575 conn(0x55ad9dfa35f0 msgr2=0x55ad9dfa39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.151+0000 7f873ea81740 1 -- 172.21.15.33:0/398359575 shutdown_connections 2024-09-16T00:41:49.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.151+0000 7f873ea81740 1 -- 172.21.15.33:0/398359575 wait complete. 2024-09-16T00:41:49.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.151+0000 7f873ea81740 1 -- start start 2024-09-16T00:41:49.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.151+0000 7f873ea81740 1 Processor -- start 2024-09-16T00:41:49.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ea81740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7060 0x55ad9dfaf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ea81740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7980 0x55ad9dfaf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ea81740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfafb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ea81740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ad9dfb4e10 con 0x55ad9dfb2240 2024-09-16T00:41:49.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ea81740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ad9de94f10 con 0x55ad9dfa7060 2024-09-16T00:41:49.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ea81740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ad9dfb4f90 con 0x55ad9dfa7980 2024-09-16T00:41:49.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ea81740 10 client.0 _ll_register_callbacks cb 0x55ad9deb8db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:49.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ac45640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfafb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ac45640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfafb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45144/0 (socket says 172.21.15.33:45144) 2024-09-16T00:41:49.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ac45640 1 -- 172.21.15.33:0/465796951 learned_addr learned my addr 172.21.15.33:0/465796951 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:49.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f8739c43640 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7980 0x55ad9dfaf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873a444640 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7060 0x55ad9dfaf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ac45640 1 -- 172.21.15.33:0/465796951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7980 msgr2=0x55ad9dfaf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:49.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ac45640 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7980 0x55ad9dfaf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ac45640 1 -- 172.21.15.33:0/465796951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7060 msgr2=0x55ad9dfaf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:49.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ac45640 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7060 0x55ad9dfaf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ac45640 1 -- 172.21.15.33:0/465796951 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad9dc1b580 con 0x55ad9dfb2240 2024-09-16T00:41:49.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f873ac45640 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfafb90 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f872c014330 tx=0x7f872c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3823d563908e5625 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:49.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.152+0000 7f871dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:49.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f871dffb640 1 -- 172.21.15.33:0/465796951 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c025070 con 0x55ad9dfb2240 2024-09-16T00:41:49.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f871dffb640 1 -- 172.21.15.33:0/465796951 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f872c004b20 con 0x55ad9dfb2240 2024-09-16T00:41:49.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f873ea81740 1 -- 172.21.15.33:0/465796951 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f872409c690 con 0x55ad9dfb2240 2024-09-16T00:41:49.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f871dffb640 1 -- 172.21.15.33:0/465796951 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f872c030050 con 0x55ad9dfb2240 2024-09-16T00:41:49.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f873ea81740 10 client.19977 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:49.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f873ea81740 1 -- 172.21.15.33:0/465796951 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ad9dce5960 con 0x55ad9dfb2240 2024-09-16T00:41:49.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f871dffb640 1 -- 172.21.15.33:0/465796951 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f872409c690 con 0x55ad9dfb2240 2024-09-16T00:41:49.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f871dffb640 1 -- 172.21.15.33:0/465796951 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f872c020040 con 0x55ad9dfb2240 2024-09-16T00:41:49.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f873ea81740 10 client.19977 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:49.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f873ea81740 10 client.19977 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:41:49.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f873ea81740 20 client.19977 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:49.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f873ea81740 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad9dfc1d40 0x55ad9dfc4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f873ea81740 4 client.19977 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:41:49.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.153+0000 7f873ea81740 1 -- 172.21.15.33:0/465796951 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad9df19b90 con 0x55ad9dfc1d40 2024-09-16T00:41:49.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.154+0000 7f873a444640 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad9dfc1d40 0x55ad9dfc4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.154+0000 7f871dffb640 10 client.19977 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:49.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.154+0000 7f873a444640 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad9dfc1d40 0x55ad9dfc4150 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0af7432fb30c43b server_cookie=108012844e7f7a61 in_seq=0 out_seq=0 2024-09-16T00:41:49.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f871dffb640 1 -- 172.21.15.33:0/465796951 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ad9df19b90 con 0x55ad9dfc1d40 2024-09-16T00:41:49.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f871dffb640 10 client.19977 handle_command_reply: tid=0 2024-09-16T00:41:49.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f873ea81740 1 client.19977 shutdown 2024-09-16T00:41:49.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f873ea81740 1 -- 172.21.15.33:0/465796951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 msgr2=0x55ad9dfafb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:49.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f873ea81740 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfafb90 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f872c014330 tx=0x7f872c004a40 comp rx=0 tx=0).stop 2024-09-16T00:41:49.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f873ea81740 1 -- 172.21.15.33:0/465796951 shutdown_connections 2024-09-16T00:41:49.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f873ea81740 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad9dfc1d40 0x55ad9dfc4150 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f873ea81740 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfafb90 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f873ea81740 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7980 0x55ad9dfaf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f873ea81740 1 --2- 172.21.15.33:0/465796951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7060 0x55ad9dfaf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f873ea81740 1 -- 172.21.15.33:0/465796951 >> 172.21.15.33:0/465796951 conn(0x55ad9dfa35f0 msgr2=0x55ad9dfa9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:49.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f871dffb640 0 client.19977 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:49.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f873ea81740 1 -- 172.21.15.33:0/465796951 shutdown_connections 2024-09-16T00:41:49.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f873ea81740 1 -- 172.21.15.33:0/465796951 wait complete. 2024-09-16T00:41:49.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.155+0000 7f873ea81740 20 client.19977 trim_cache size 0 max 16384 2024-09-16T00:41:49.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.172+0000 7f873ea81740 1 Processor -- start 2024-09-16T00:41:49.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.172+0000 7f873ea81740 1 -- start start 2024-09-16T00:41:49.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ea81740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7060 0x55ad9dfa2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ea81740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7980 0x55ad9dfc9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ea81740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfc9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ea81740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f87341b3bf0 con 0x55ad9dfb2240 2024-09-16T00:41:49.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ea81740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f87341b14c0 con 0x55ad9dfa7980 2024-09-16T00:41:49.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ea81740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f87341af310 con 0x55ad9dfa7060 2024-09-16T00:41:49.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ac45640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfc9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f8739c43640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7980 0x55ad9dfc9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ac45640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfc9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45160/0 (socket says 172.21.15.33:45160) 2024-09-16T00:41:49.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ac45640 1 -- 172.21.15.33:0/2113932271 learned_addr learned my addr 172.21.15.33:0/2113932271 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:49.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873a444640 1 --2- 172.21.15.33:0/2113932271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7060 0x55ad9dfa2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ac45640 1 -- 172.21.15.33:0/2113932271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7060 msgr2=0x55ad9dfa2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:49.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ac45640 1 --2- 172.21.15.33:0/2113932271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7060 0x55ad9dfa2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ac45640 1 -- 172.21.15.33:0/2113932271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7980 msgr2=0x55ad9dfc9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:49.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ac45640 1 --2- 172.21.15.33:0/2113932271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7980 0x55ad9dfc9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ac45640 1 -- 172.21.15.33:0/2113932271 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad9dfae0c0 con 0x55ad9dfb2240 2024-09-16T00:41:49.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f8739c43640 1 --2- 172.21.15.33:0/2113932271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7980 0x55ad9dfc9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:41:49.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.173+0000 7f873ac45640 1 --2- 172.21.15.33:0/2113932271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfc9f60 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f872c009790 tx=0x7f872c012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5c23d364f84463 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:49.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.174+0000 7f871ffff640 1 -- 172.21.15.33:0/2113932271 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c025030 con 0x55ad9dfb2240 2024-09-16T00:41:49.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.174+0000 7f871ffff640 1 -- 172.21.15.33:0/2113932271 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f872c011d10 con 0x55ad9dfb2240 2024-09-16T00:41:49.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.174+0000 7f871ffff640 1 -- 172.21.15.33:0/2113932271 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c01a040 con 0x55ad9dfb2240 2024-09-16T00:41:49.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.174+0000 7f873ea81740 1 -- 172.21.15.33:0/2113932271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 msgr2=0x55ad9dfc9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:49.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.174+0000 7f873ea81740 1 --2- 172.21.15.33:0/2113932271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfc9f60 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f872c009790 tx=0x7f872c012eb0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.174+0000 7f873ea81740 1 -- 172.21.15.33:0/2113932271 shutdown_connections 2024-09-16T00:41:49.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.174+0000 7f873ea81740 1 --2- 172.21.15.33:0/2113932271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfc9f60 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.174+0000 7f873ea81740 1 --2- 172.21.15.33:0/2113932271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7980 0x55ad9dfc9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.174+0000 7f873ea81740 1 --2- 172.21.15.33:0/2113932271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7060 0x55ad9dfa2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.174+0000 7f873ea81740 1 -- 172.21.15.33:0/2113932271 >> 172.21.15.33:0/2113932271 conn(0x55ad9dfa35f0 msgr2=0x55ad9dfa1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:49.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ea81740 1 -- 172.21.15.33:0/2113932271 shutdown_connections 2024-09-16T00:41:49.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ea81740 1 -- 172.21.15.33:0/2113932271 wait complete. 2024-09-16T00:41:49.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ea81740 1 -- start start 2024-09-16T00:41:49.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ea81740 1 Processor -- start 2024-09-16T00:41:49.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ea81740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7060 0x55ad9dfb6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ea81740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7980 0x55ad9dfb66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ea81740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfb6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ea81740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f87341af980 con 0x55ad9dfb2240 2024-09-16T00:41:49.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ea81740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f87341b0df0 con 0x55ad9dfa7060 2024-09-16T00:41:49.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ea81740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f87341b0050 con 0x55ad9dfa7980 2024-09-16T00:41:49.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ac45640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfb6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ea81740 10 client.0 _ll_register_callbacks cb 0x7f872c009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:49.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ac45640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfb6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45176/0 (socket says 172.21.15.33:45176) 2024-09-16T00:41:49.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.175+0000 7f873ac45640 1 -- 172.21.15.33:0/319886568 learned_addr learned my addr 172.21.15.33:0/319886568 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:49.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f8739c43640 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7980 0x55ad9dfb66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f873a444640 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7060 0x55ad9dfb6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f873ac45640 1 -- 172.21.15.33:0/319886568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7980 msgr2=0x55ad9dfb66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f873ac45640 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7980 0x55ad9dfb66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f873ac45640 1 -- 172.21.15.33:0/319886568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7060 msgr2=0x55ad9dfb6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f873ac45640 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7060 0x55ad9dfb6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f873ac45640 1 -- 172.21.15.33:0/319886568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad9dfa77b0 con 0x55ad9dfb2240 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f873a444640 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7060 0x55ad9dfb6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f873ac45640 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfb6be0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f872c0092d0 tx=0x7f872c0058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc05f91e0840768 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f871dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f871dffb640 1 -- 172.21.15.33:0/319886568 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c01a030 con 0x55ad9dfb2240 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f873ea81740 1 -- 172.21.15.33:0/319886568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f872c011d10 con 0x55ad9dfb2240 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f871dffb640 1 -- 172.21.15.33:0/319886568 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f872c011a40 con 0x55ad9dfb2240 2024-09-16T00:41:49.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f871dffb640 1 -- 172.21.15.33:0/319886568 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f872c025030 con 0x55ad9dfb2240 2024-09-16T00:41:49.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f873ea81740 10 client.19983 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:49.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.176+0000 7f873ea81740 1 -- 172.21.15.33:0/319886568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ad9e02cd70 con 0x55ad9dfb2240 2024-09-16T00:41:49.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.177+0000 7f871dffb640 1 -- 172.21.15.33:0/319886568 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f872c011d10 con 0x55ad9dfb2240 2024-09-16T00:41:49.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.177+0000 7f871dffb640 1 -- 172.21.15.33:0/319886568 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f872c020040 con 0x55ad9dfb2240 2024-09-16T00:41:49.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.177+0000 7f873ea81740 10 client.19983 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:49.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.177+0000 7f873ea81740 10 client.19983 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:41:49.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.177+0000 7f873ea81740 20 client.19983 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:49.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.177+0000 7f873ea81740 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad9e02d370 0x55ad9e02f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:49.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.177+0000 7f873ea81740 4 client.19983 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "987c2a04", "release": true, "await": true}] 2024-09-16T00:41:49.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.177+0000 7f873ea81740 1 -- 172.21.15.33:0/319886568 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "987c2a04", "release": true, "await": true}) -- 0x55ad9df140a0 con 0x55ad9e02d370 2024-09-16T00:41:49.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.177+0000 7f873a444640 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad9e02d370 0x55ad9e02f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:49.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.177+0000 7f873a444640 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad9e02d370 0x55ad9e02f760 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e787c152652ed8fa server_cookie=b28bc9fe24c821c1 in_seq=0 out_seq=0 2024-09-16T00:41:49.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.177+0000 7f871dffb640 10 client.19983 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:49.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.236+0000 7f871dffb640 1 -- 172.21.15.33:0/319886568 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ad9df140a0 con 0x55ad9e02d370 2024-09-16T00:41:49.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.236+0000 7f871dffb640 10 client.19983 handle_command_reply: tid=0 2024-09-16T00:41:49.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.236+0000 7f873ea81740 1 client.19983 shutdown 2024-09-16T00:41:49.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.236+0000 7f873ea81740 1 -- 172.21.15.33:0/319886568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 msgr2=0x55ad9dfb6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:49.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.236+0000 7f873ea81740 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfb6be0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f872c0092d0 tx=0x7f872c0058c0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.236+0000 7f873ea81740 1 -- 172.21.15.33:0/319886568 shutdown_connections 2024-09-16T00:41:49.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.236+0000 7f873ea81740 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55ad9e02d370 0x55ad9e02f760 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.236+0000 7f873ea81740 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad9dfb2240 0x55ad9dfb6be0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.236+0000 7f873ea81740 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ad9dfa7980 0x55ad9dfb66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.236+0000 7f873ea81740 1 --2- 172.21.15.33:0/319886568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad9dfa7060 0x55ad9dfb6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.236+0000 7f871dffb640 0 client.19983 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:49.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.236+0000 7f873ea81740 1 -- 172.21.15.33:0/319886568 >> 172.21.15.33:0/319886568 conn(0x55ad9dfa35f0 msgr2=0x55ad9dfa92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:49.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.237+0000 7f873ea81740 1 -- 172.21.15.33:0/319886568 shutdown_connections 2024-09-16T00:41:49.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.237+0000 7f873ea81740 1 -- 172.21.15.33:0/319886568 wait complete. 2024-09-16T00:41:49.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.237+0000 7f873ea81740 20 client.19983 trim_cache size 0 max 16384 2024-09-16T00:41:49.239 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:41:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:41:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:41:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 84, 2024-09-16T00:41:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:41:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "987c2a04": { 2024-09-16T00:41:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 84, 2024-09-16T00:41:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:41:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 147.2, 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:49.241 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:49.241 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:49.241 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:49.241 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:41:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.237+0000 7f873c6a8640 1 -- 172.21.15.33:0/4234452037 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f87200a3240 msgr2=0x7f87200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.237+0000 7f873c6a8640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f87200a3240 0x7f87200a5700 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f872c005fd0 tx=0x7f872c004ab0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.237+0000 7f873c6a8640 1 -- 172.21.15.33:0/4234452037 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8734102790 msgr2=0x7f87341989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.237+0000 7f873c6a8640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8734102790 0x7f87341989c0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f872400c890 tx=0x7f872400cd50 comp rx=0 tx=0).stop 2024-09-16T00:41:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.237+0000 7f873c6a8640 1 -- 172.21.15.33:0/4234452037 shutdown_connections 2024-09-16T00:41:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.237+0000 7f873c6a8640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f87200a3240 0x7f87200a5700 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.237+0000 7f873c6a8640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8734106f10 0x7f873419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.237+0000 7f873c6a8640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8734102790 0x7f87341989c0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.237+0000 7f873c6a8640 1 --2- 172.21.15.33:0/4234452037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8734101de0 0x7f8734198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.238+0000 7f873c6a8640 1 -- 172.21.15.33:0/4234452037 >> 172.21.15.33:0/4234452037 conn(0x7f87340fd8c0 msgr2=0x7f87340fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:49.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.238+0000 7f873c6a8640 1 -- 172.21.15.33:0/4234452037 shutdown_connections 2024-09-16T00:41:49.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:49.238+0000 7f873c6a8640 1 -- 172.21.15.33:0/4234452037 wait complete. 2024-09-16T00:41:50.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:49 smithi033 ceph-mon[31243]: pgmap v1090: 129 pgs: 129 active+clean; 707 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 509 KiB/s wr, 61 op/s 2024-09-16T00:41:50.105 INFO:tasks.workunit.client.0.smithi033.stdout: 10 221619 17.88 MB/sec execute 536 sec latency 24905.423 ms 2024-09-16T00:41:50.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:49 smithi083 ceph-mon[35242]: pgmap v1090: 129 pgs: 129 active+clean; 707 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 509 KiB/s wr, 61 op/s 2024-09-16T00:41:50.193 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '987c2a04', total seconds elapsed: 31.3 2024-09-16T00:41:50.194 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.2 2024-09-16T00:41:50.194 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:41:50.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:49 smithi060 ceph-mon[35185]: pgmap v1090: 129 pgs: 129 active+clean; 707 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 509 KiB/s wr, 61 op/s 2024-09-16T00:41:50.554 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:51.106 INFO:tasks.workunit.client.0.smithi033.stdout: 10 222089 17.89 MB/sec execute 537 sec latency 87.516 ms 2024-09-16T00:41:51.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.118+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3572967812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3f4106f10 msgr2=0x7fa3f410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:51.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.118+0000 7fa3fb25f640 1 --2- 172.21.15.33:0/3572967812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3f4106f10 0x7fa3f410ea90 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec009be0 tx=0x7fa3ec02a6c0 comp rx=0 tx=0).stop 2024-09-16T00:41:51.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.118+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3572967812 shutdown_connections 2024-09-16T00:41:51.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.118+0000 7fa3fb25f640 1 --2- 172.21.15.33:0/3572967812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3f4106f10 0x7fa3f410ea90 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:51.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.118+0000 7fa3fb25f640 1 --2- 172.21.15.33:0/3572967812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3f4102790 0x7fa3f41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:51.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.118+0000 7fa3fb25f640 1 --2- 172.21.15.33:0/3572967812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f4101de0 0x7fa3f41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.118+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3572967812 >> 172.21.15.33:0/3572967812 conn(0x7fa3f40fd880 msgr2=0x7fa3f40ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.118+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3572967812 shutdown_connections 2024-09-16T00:41:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.118+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3572967812 wait complete. 2024-09-16T00:41:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.119+0000 7fa3fb25f640 1 Processor -- start 2024-09-16T00:41:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.119+0000 7fa3fb25f640 1 -- start start 2024-09-16T00:41:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3fb25f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3f4101de0 0x7fa3f41983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3fb25f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f4102790 0x7fa3f41988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3fb25f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3f4106f10 0x7fa3f419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3fb25f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f4111f20 con 0x7fa3f4106f10 2024-09-16T00:41:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3fb25f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f4111da0 con 0x7fa3f4102790 2024-09-16T00:41:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3fb25f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f41120a0 con 0x7fa3f4101de0 2024-09-16T00:41:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f8ffb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3f4101de0 0x7fa3f41983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f97fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3f4106f10 0x7fa3f419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f8ffb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3f4101de0 0x7fa3f41983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:54838/0 (socket says 172.21.15.33:54838) 2024-09-16T00:41:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f97fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3f4106f10 0x7fa3f419cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47562/0 (socket says 172.21.15.33:47562) 2024-09-16T00:41:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f8ffb640 1 -- 172.21.15.33:0/3150539692 learned_addr learned my addr 172.21.15.33:0/3150539692 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f3fff640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f4102790 0x7fa3f41988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f97fc640 1 -- 172.21.15.33:0/3150539692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3f4101de0 msgr2=0x7fa3f41983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f97fc640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3f4101de0 0x7fa3f41983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f97fc640 1 -- 172.21.15.33:0/3150539692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f4102790 msgr2=0x7fa3f41988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f97fc640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f4102790 0x7fa3f41988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f97fc640 1 -- 172.21.15.33:0/3150539692 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f419d350 con 0x7fa3f4106f10 2024-09-16T00:41:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f8ffb640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3f4101de0 0x7fa3f41983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:41:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.120+0000 7fa3f97fc640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3f4106f10 0x7fa3f419cc70 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec034040 tx=0x7fa3ec0305d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.121+0000 7fa3f1ffb640 1 -- 172.21.15.33:0/3150539692 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec041020 con 0x7fa3f4106f10 2024-09-16T00:41:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.121+0000 7fa3f1ffb640 1 -- 172.21.15.33:0/3150539692 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3ec037040 con 0x7fa3f4106f10 2024-09-16T00:41:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.122+0000 7fa3f1ffb640 1 -- 172.21.15.33:0/3150539692 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec036070 con 0x7fa3f4106f10 2024-09-16T00:41:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.122+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3150539692 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3f419d5e0 con 0x7fa3f4106f10 2024-09-16T00:41:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.122+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3150539692 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3f41a4f20 con 0x7fa3f4106f10 2024-09-16T00:41:51.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.123+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3150539692 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3f41032c0 con 0x7fa3f4106f10 2024-09-16T00:41:51.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.125+0000 7fa3f1ffb640 1 -- 172.21.15.33:0/3150539692 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa3ec044020 con 0x7fa3f4106f10 2024-09-16T00:41:51.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.126+0000 7fa3f1ffb640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3e40a3390 0x7fa3e40a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:51.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.126+0000 7fa3f1ffb640 1 -- 172.21.15.33:0/3150539692 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa3ec0f09c0 con 0x7fa3f4106f10 2024-09-16T00:41:51.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.126+0000 7fa3f8ffb640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3e40a3390 0x7fa3e40a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:51.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.127+0000 7fa3f8ffb640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3e40a3390 0x7fa3e40a5850 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80095f0 tx=0x7fa3e8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:51.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.130+0000 7fa3f1ffb640 1 -- 172.21.15.33:0/3150539692 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa3ec04b020 con 0x7fa3f4106f10 2024-09-16T00:41:51.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.469+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3150539692 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3f41021c0 con 0x7fa3f4106f10 2024-09-16T00:41:51.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.470+0000 7fa3f1ffb640 1 -- 172.21.15.33:0/3150539692 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7fa3f41021c0 con 0x7fa3f4106f10 2024-09-16T00:41:51.471 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:41:51.472 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:41:51.472 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:41:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.474+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3150539692 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3e40a3390 msgr2=0x7fa3e40a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.474+0000 7fa3fb25f640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3e40a3390 0x7fa3e40a5850 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80095f0 tx=0x7fa3e8005f90 comp rx=0 tx=0).stop 2024-09-16T00:41:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.474+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3150539692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3f4106f10 msgr2=0x7fa3f419cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.474+0000 7fa3fb25f640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3f4106f10 0x7fa3f419cc70 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec034040 tx=0x7fa3ec0305d0 comp rx=0 tx=0).stop 2024-09-16T00:41:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.474+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3150539692 shutdown_connections 2024-09-16T00:41:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.474+0000 7fa3fb25f640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa3e40a3390 0x7fa3e40a5850 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.474+0000 7fa3fb25f640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3f4106f10 0x7fa3f419cc70 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.474+0000 7fa3fb25f640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f4102790 0x7fa3f41988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.474+0000 7fa3fb25f640 1 --2- 172.21.15.33:0/3150539692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3f4101de0 0x7fa3f41983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.474+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3150539692 >> 172.21.15.33:0/3150539692 conn(0x7fa3f40fd880 msgr2=0x7fa3f40fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.475+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3150539692 shutdown_connections 2024-09-16T00:41:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:51.475+0000 7fa3fb25f640 1 -- 172.21.15.33:0/3150539692 wait complete. 2024-09-16T00:41:52.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:51 smithi033 ceph-mon[31243]: pgmap v1091: 129 pgs: 129 active+clean; 713 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 1.6 MiB/s wr, 72 op/s 2024-09-16T00:41:52.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:51 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3150539692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:41:52.106 INFO:tasks.workunit.client.0.smithi033.stdout: 10 222481 17.90 MB/sec execute 538 sec latency 64.834 ms 2024-09-16T00:41:52.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:51 smithi083 ceph-mon[35242]: pgmap v1091: 129 pgs: 129 active+clean; 713 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 1.6 MiB/s wr, 72 op/s 2024-09-16T00:41:52.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:51 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3150539692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:41:52.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:51 smithi060 ceph-mon[35185]: pgmap v1091: 129 pgs: 129 active+clean; 713 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 1.6 MiB/s wr, 72 op/s 2024-09-16T00:41:52.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:51 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3150539692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:41:52.710 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db / --timeout 90 --expiration 147.2 --await-for 45.0' 2024-09-16T00:41:52.710 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db / --timeout 90 --expiration 147.2 --await-for 45.0 2024-09-16T00:41:53.035 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:41:53.106 INFO:tasks.workunit.client.0.smithi033.stdout: 10 222978 17.91 MB/sec execute 539 sec latency 58.425 ms 2024-09-16T00:41:53.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.584+0000 7ff21649e640 1 -- 172.21.15.33:0/1345089299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff210108e00 msgr2=0x7ff21010ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:53.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.584+0000 7ff21649e640 1 --2- 172.21.15.33:0/1345089299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff210108e00 0x7ff21010ce00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff204009900 tx=0x7ff20402a420 comp rx=0 tx=0).stop 2024-09-16T00:41:53.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.584+0000 7ff21649e640 1 -- 172.21.15.33:0/1345089299 shutdown_connections 2024-09-16T00:41:53.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.584+0000 7ff21649e640 1 --2- 172.21.15.33:0/1345089299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff21010d530 0x7ff2101150b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.584+0000 7ff21649e640 1 --2- 172.21.15.33:0/1345089299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff210108e00 0x7ff21010ce00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.584+0000 7ff21649e640 1 --2- 172.21.15.33:0/1345089299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff210108450 0x7ff210108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.584+0000 7ff21649e640 1 -- 172.21.15.33:0/1345089299 >> 172.21.15.33:0/1345089299 conn(0x7ff210077ba0 msgr2=0x7ff210079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:53.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.584+0000 7ff21649e640 1 -- 172.21.15.33:0/1345089299 shutdown_connections 2024-09-16T00:41:53.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.584+0000 7ff21649e640 1 -- 172.21.15.33:0/1345089299 wait complete. 2024-09-16T00:41:53.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.585+0000 7ff21649e640 1 Processor -- start 2024-09-16T00:41:53.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.585+0000 7ff21649e640 1 -- start start 2024-09-16T00:41:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.585+0000 7ff21649e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff210108450 0x7ff2101a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff21649e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff210108e00 0x7ff2101a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff21649e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff21010d530 0x7ff21019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff21649e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff210118530 con 0x7ff210108450 2024-09-16T00:41:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff21649e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff2101183b0 con 0x7ff210108e00 2024-09-16T00:41:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff21649e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2101186b0 con 0x7ff21010d530 2024-09-16T00:41:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff20f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff210108e00 0x7ff2101a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff20f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff210108e00 0x7ff2101a4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:60064/0 (socket says 172.21.15.33:60064) 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff20f7fe640 1 -- 172.21.15.33:0/3952073955 learned_addr learned my addr 172.21.15.33:0/3952073955 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff20ffff640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff210108450 0x7ff2101a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff214a3b640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff21010d530 0x7ff21019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff20f7fe640 1 -- 172.21.15.33:0/3952073955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff21010d530 msgr2=0x7ff21019e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff20f7fe640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff21010d530 0x7ff21019e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff20f7fe640 1 -- 172.21.15.33:0/3952073955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff210108450 msgr2=0x7ff2101a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff20f7fe640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff210108450 0x7ff2101a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff20f7fe640 1 -- 172.21.15.33:0/3952073955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21019efc0 con 0x7ff210108e00 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.586+0000 7ff20f7fe640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff210108e00 0x7ff2101a4be0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff204006010 tx=0x7ff2040042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.587+0000 7ff20d7fa640 1 -- 172.21.15.33:0/3952073955 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20403f030 con 0x7ff210108e00 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.587+0000 7ff21649e640 1 -- 172.21.15.33:0/3952073955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff21019f250 con 0x7ff210108e00 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.587+0000 7ff20d7fa640 1 -- 172.21.15.33:0/3952073955 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff204003ce0 con 0x7ff210108e00 2024-09-16T00:41:53.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.587+0000 7ff20d7fa640 1 -- 172.21.15.33:0/3952073955 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff204035070 con 0x7ff210108e00 2024-09-16T00:41:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.587+0000 7ff21649e640 1 -- 172.21.15.33:0/3952073955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2101ab500 con 0x7ff210108e00 2024-09-16T00:41:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.589+0000 7ff218877740 1 Processor -- start 2024-09-16T00:41:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff218877740 1 -- start start 2024-09-16T00:41:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff20d7fa640 1 -- 172.21.15.33:0/3952073955 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff204042030 con 0x7ff210108e00 2024-09-16T00:41:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff218877740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2ba760 0x55e94a2bab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff218877740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff218877740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff218877740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e94a2202a0 con 0x55e94a2c5940 2024-09-16T00:41:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff218877740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e94a1c7b10 con 0x55e94a2bb080 2024-09-16T00:41:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff218877740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e94a21c260 con 0x55e94a2ba760 2024-09-16T00:41:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff20ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff20f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff20ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:60080/0 (socket says 172.21.15.33:60080) 2024-09-16T00:41:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff20ffff640 1 -- 172.21.15.33:0/1562993871 learned_addr learned my addr 172.21.15.33:0/1562993871 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff214a3b640 1 --2- 172.21.15.33:0/1562993871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2ba760 0x55e94a2bab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff20ffff640 1 -- 172.21.15.33:0/1562993871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2ba760 msgr2=0x55e94a2bab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff20ffff640 1 --2- 172.21.15.33:0/1562993871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2ba760 0x55e94a2bab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff20ffff640 1 -- 172.21.15.33:0/1562993871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 msgr2=0x55e94a2c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff20ffff640 1 --2- 172.21.15.33:0/1562993871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.590+0000 7ff20ffff640 1 -- 172.21.15.33:0/1562993871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e949f91b90 con 0x55e94a2bb080 2024-09-16T00:41:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.591+0000 7ff20f7fe640 1 --2- 172.21.15.33:0/1562993871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:41:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.591+0000 7ff20d7fa640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff1f80a3240 0x7ff1f80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.591+0000 7ff214a3b640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff1f80a3240 0x7ff1f80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.591+0000 7ff20d7fa640 1 -- 172.21.15.33:0/3952073955 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff2040efed0 con 0x7ff210108e00 2024-09-16T00:41:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.591+0000 7ff20ffff640 1 --2- 172.21.15.33:0/1562993871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c5400 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff20000bd10 tx=0x7ff2000107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68fb206a5a2f5091 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.591+0000 7ff214a3b640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff1f80a3240 0x7ff1f80a5700 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff208005fd0 tx=0x7ff208009550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.591+0000 7ff1f67fc640 1 -- 172.21.15.33:0/1562993871 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff200004030 con 0x55e94a2bb080 2024-09-16T00:41:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.591+0000 7ff1f67fc640 1 -- 172.21.15.33:0/1562993871 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2000041d0 con 0x55e94a2bb080 2024-09-16T00:41:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.591+0000 7ff1f67fc640 1 -- 172.21.15.33:0/1562993871 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff200011840 con 0x55e94a2bb080 2024-09-16T00:41:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.592+0000 7ff218877740 1 -- 172.21.15.33:0/1562993871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 msgr2=0x55e94a2c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.592+0000 7ff218877740 1 --2- 172.21.15.33:0/1562993871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c5400 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff20000bd10 tx=0x7ff2000107a0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.592+0000 7ff218877740 1 -- 172.21.15.33:0/1562993871 shutdown_connections 2024-09-16T00:41:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.592+0000 7ff218877740 1 --2- 172.21.15.33:0/1562993871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.592+0000 7ff218877740 1 --2- 172.21.15.33:0/1562993871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c5400 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.592+0000 7ff218877740 1 --2- 172.21.15.33:0/1562993871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2ba760 0x55e94a2bab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.592+0000 7ff218877740 1 -- 172.21.15.33:0/1562993871 >> 172.21.15.33:0/1562993871 conn(0x55e94a2b6cf0 msgr2=0x55e94a2b70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.592+0000 7ff218877740 1 -- 172.21.15.33:0/1562993871 shutdown_connections 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.592+0000 7ff218877740 1 -- 172.21.15.33:0/1562993871 wait complete. 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.592+0000 7ff218877740 1 -- start start 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff218877740 1 Processor -- start 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff218877740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2ba760 0x55e94a2c27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff218877740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff218877740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2c5940 0x55e94a2c3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff218877740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e94a2c8510 con 0x55e94a2ba760 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff218877740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e94a1a86e0 con 0x55e94a2bb080 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff218877740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e94a2c8690 con 0x55e94a2c5940 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff218877740 10 client.0 _ll_register_callbacks cb 0x55e94a1cca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff20f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff20ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2ba760 0x55e94a2c27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff20f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:60084/0 (socket says 172.21.15.33:60084) 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff20f7fe640 1 -- 172.21.15.33:0/2044802302 learned_addr learned my addr 172.21.15.33:0/2044802302 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.593+0000 7ff214a3b640 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2c5940 0x55e94a2c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff20f7fe640 1 -- 172.21.15.33:0/2044802302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2c5940 msgr2=0x55e94a2c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff20f7fe640 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2c5940 0x55e94a2c3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff20f7fe640 1 -- 172.21.15.33:0/2044802302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2ba760 msgr2=0x55e94a2c27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff20f7fe640 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2ba760 0x55e94a2c27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff20f7fe640 1 -- 172.21.15.33:0/2044802302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e94a2d19d0 con 0x55e94a2bb080 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff20ffff640 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2ba760 0x55e94a2c27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff20f7fe640 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c2d20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff2040b8580 tx=0x7ff20403b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5551348037584cc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff1ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff1ef7fe640 1 -- 172.21.15.33:0/2044802302 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff204049070 con 0x55e94a2bb080 2024-09-16T00:41:53.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff1ef7fe640 1 -- 172.21.15.33:0/2044802302 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff20403b460 con 0x55e94a2bb080 2024-09-16T00:41:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff218877740 1 -- 172.21.15.33:0/2044802302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2000041d0 con 0x55e94a2bb080 2024-09-16T00:41:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff1ef7fe640 1 -- 172.21.15.33:0/2044802302 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff2040b83c0 con 0x55e94a2bb080 2024-09-16T00:41:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff218877740 10 client.29108 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff218877740 1 -- 172.21.15.33:0/2044802302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e94a2d4340 con 0x55e94a2bb080 2024-09-16T00:41:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.594+0000 7ff1ef7fe640 1 -- 172.21.15.33:0/2044802302 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2000041d0 con 0x55e94a2bb080 2024-09-16T00:41:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.595+0000 7ff1ef7fe640 1 -- 172.21.15.33:0/2044802302 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ff20404c060 con 0x55e94a2bb080 2024-09-16T00:41:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.595+0000 7ff218877740 10 client.29108 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.595+0000 7ff218877740 10 client.29108 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:41:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.595+0000 7ff218877740 20 client.29108 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.595+0000 7ff218877740 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e94a2d5690 0x55e94a2d7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.595+0000 7ff218877740 4 client.29108 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:41:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.595+0000 7ff218877740 1 -- 172.21.15.33:0/2044802302 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e94a22d9d0 con 0x55e94a2d5690 2024-09-16T00:41:53.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.595+0000 7ff20ffff640 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e94a2d5690 0x55e94a2d7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.595+0000 7ff1ef7fe640 10 client.29108 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:53.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.595+0000 7ff20ffff640 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e94a2d5690 0x55e94a2d7aa0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5555480e4e879dd2 server_cookie=728ea026992fbc61 in_seq=0 out_seq=0 2024-09-16T00:41:53.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff1ef7fe640 1 -- 172.21.15.33:0/2044802302 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e94a22d9d0 con 0x55e94a2d5690 2024-09-16T00:41:53.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff1ef7fe640 10 client.29108 handle_command_reply: tid=0 2024-09-16T00:41:53.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff218877740 1 client.29108 shutdown 2024-09-16T00:41:53.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff218877740 1 -- 172.21.15.33:0/2044802302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 msgr2=0x55e94a2c2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:53.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff218877740 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c2d20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff2040b8580 tx=0x7ff20403b720 comp rx=0 tx=0).stop 2024-09-16T00:41:53.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff218877740 1 -- 172.21.15.33:0/2044802302 shutdown_connections 2024-09-16T00:41:53.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff218877740 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e94a2d5690 0x55e94a2d7aa0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff218877740 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2c5940 0x55e94a2c3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff1ef7fe640 0 client.29108 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff218877740 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c2d20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff218877740 1 --2- 172.21.15.33:0/2044802302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2ba760 0x55e94a2c27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff218877740 1 -- 172.21.15.33:0/2044802302 >> 172.21.15.33:0/2044802302 conn(0x55e94a2b6cf0 msgr2=0x55e94a2b5e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff218877740 1 -- 172.21.15.33:0/2044802302 shutdown_connections 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff218877740 1 -- 172.21.15.33:0/2044802302 wait complete. 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.596+0000 7ff218877740 20 client.29108 trim_cache size 0 max 16384 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.614+0000 7ff218877740 1 Processor -- start 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.614+0000 7ff218877740 1 -- start start 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff218877740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2ba760 0x55e94a2c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff218877740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2bb080 0x55e94a2c10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff218877740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2dd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff218877740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff2101b7f90 con 0x55e94a2c5940 2024-09-16T00:41:53.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff218877740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff2101b5860 con 0x55e94a2ba760 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff218877740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2101b36b0 con 0x55e94a2bb080 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff214a3b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2dd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff20ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2ba760 0x55e94a2c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff214a3b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2dd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47604/0 (socket says 172.21.15.33:47604) 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff214a3b640 1 -- 172.21.15.33:0/1967518275 learned_addr learned my addr 172.21.15.33:0/1967518275 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff20ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2ba760 0x55e94a2c46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:60088/0 (socket says 172.21.15.33:60088) 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff20f7fe640 1 --2- 172.21.15.33:0/1967518275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2bb080 0x55e94a2c10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff214a3b640 1 -- 172.21.15.33:0/1967518275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2bb080 msgr2=0x55e94a2c10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff214a3b640 1 --2- 172.21.15.33:0/1967518275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2bb080 0x55e94a2c10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff214a3b640 1 -- 172.21.15.33:0/1967518275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2ba760 msgr2=0x55e94a2c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff214a3b640 1 --2- 172.21.15.33:0/1967518275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2ba760 0x55e94a2c46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff214a3b640 1 -- 172.21.15.33:0/1967518275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e94a2b2260 con 0x55e94a2c5940 2024-09-16T00:41:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff20f7fe640 1 --2- 172.21.15.33:0/1967518275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2bb080 0x55e94a2c10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.615+0000 7ff214a3b640 1 --2- 172.21.15.33:0/1967518275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2dd370 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7ff2080092f0 tx=0x7ff208016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8236fe29c82ad2a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.616+0000 7ff1f5ffb640 1 -- 172.21.15.33:0/1967518275 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff208011500 con 0x55e94a2c5940 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.616+0000 7ff1f5ffb640 1 -- 172.21.15.33:0/1967518275 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff208025070 con 0x55e94a2c5940 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.616+0000 7ff1f5ffb640 1 -- 172.21.15.33:0/1967518275 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20802abe0 con 0x55e94a2c5940 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.616+0000 7ff218877740 1 -- 172.21.15.33:0/1967518275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 msgr2=0x55e94a2dd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.616+0000 7ff218877740 1 --2- 172.21.15.33:0/1967518275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2dd370 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7ff2080092f0 tx=0x7ff208016000 comp rx=0 tx=0).stop 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.617+0000 7ff218877740 1 -- 172.21.15.33:0/1967518275 shutdown_connections 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.617+0000 7ff218877740 1 --2- 172.21.15.33:0/1967518275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2dd370 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.617+0000 7ff218877740 1 --2- 172.21.15.33:0/1967518275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2bb080 0x55e94a2c10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.617+0000 7ff218877740 1 --2- 172.21.15.33:0/1967518275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2ba760 0x55e94a2c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.617+0000 7ff218877740 1 -- 172.21.15.33:0/1967518275 >> 172.21.15.33:0/1967518275 conn(0x55e94a2b6cf0 msgr2=0x55e94a2b5e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.617+0000 7ff218877740 1 -- 172.21.15.33:0/1967518275 shutdown_connections 2024-09-16T00:41:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.617+0000 7ff218877740 1 -- 172.21.15.33:0/1967518275 wait complete. 2024-09-16T00:41:53.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.617+0000 7ff218877740 1 -- start start 2024-09-16T00:41:53.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.617+0000 7ff218877740 1 Processor -- start 2024-09-16T00:41:53.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff218877740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2ba760 0x55e94a2c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff218877740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff218877740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff218877740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff2101b3d20 con 0x55e94a2c5940 2024-09-16T00:41:53.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff218877740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff2101b5190 con 0x55e94a2bb080 2024-09-16T00:41:53.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff218877740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2101b43f0 con 0x55e94a2ba760 2024-09-16T00:41:53.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff218877740 10 client.0 _ll_register_callbacks cb 0x7ff2040f36a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:41:53.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff214a3b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff20f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff214a3b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47606/0 (socket says 172.21.15.33:47606) 2024-09-16T00:41:53.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff20f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:60100/0 (socket says 172.21.15.33:60100) 2024-09-16T00:41:53.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff214a3b640 1 -- 172.21.15.33:0/3949598299 learned_addr learned my addr 172.21.15.33:0/3949598299 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:41:53.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff20ffff640 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2ba760 0x55e94a2c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff214a3b640 1 -- 172.21.15.33:0/3949598299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2ba760 msgr2=0x55e94a2c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:53.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff214a3b640 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2ba760 0x55e94a2c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff214a3b640 1 -- 172.21.15.33:0/3949598299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 msgr2=0x55e94a2c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:41:53.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff214a3b640 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:53.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff214a3b640 1 -- 172.21.15.33:0/3949598299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e94a2bd5d0 con 0x55e94a2c5940 2024-09-16T00:41:53.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.618+0000 7ff20f7fe640 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:41:53.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.619+0000 7ff214a3b640 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2ca360 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7ff208003bb0 tx=0x7ff208004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e83aa037248e29 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:41:53.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.619+0000 7ff1ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:41:53.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.619+0000 7ff1ef7fe640 1 -- 172.21.15.33:0/3949598299 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff208004c50 con 0x55e94a2c5940 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.619+0000 7ff1ef7fe640 1 -- 172.21.15.33:0/3949598299 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7ff208004df0 con 0x55e94a2c5940 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.619+0000 7ff218877740 1 -- 172.21.15.33:0/3949598299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff208025070 con 0x55e94a2c5940 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.619+0000 7ff1ef7fe640 1 -- 172.21.15.33:0/3949598299 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff208011500 con 0x55e94a2c5940 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.619+0000 7ff218877740 10 client.20007 fetch_fsmap learned FSMap version 33 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.619+0000 7ff218877740 1 -- 172.21.15.33:0/3949598299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e94a3406d0 con 0x55e94a2c5940 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.619+0000 7ff1ef7fe640 1 -- 172.21.15.33:0/3949598299 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff208025070 con 0x55e94a2c5940 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.620+0000 7ff1ef7fe640 1 -- 172.21.15.33:0/3949598299 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ff20800f040 con 0x55e94a2c5940 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.620+0000 7ff218877740 10 client.20007 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.620+0000 7ff218877740 10 client.20007 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.620+0000 7ff218877740 20 client.20007 populate_metadata read hostname 'smithi033' 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.620+0000 7ff218877740 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e94a340dc0 0x55e94a3431e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.620+0000 7ff218877740 4 client.20007 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.2, "await_for": 45.0}] 2024-09-16T00:41:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.620+0000 7ff218877740 1 -- 172.21.15.33:0/3949598299 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.2, "await_for": 45.0}) -- 0x55e94a227d20 con 0x55e94a340dc0 2024-09-16T00:41:53.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.620+0000 7ff20ffff640 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e94a340dc0 0x55e94a3431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:41:53.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.620+0000 7ff1ef7fe640 10 client.20007 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:53.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:53.620+0000 7ff20ffff640 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e94a340dc0 0x55e94a3431e0 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88114bc12ec5ffe7 server_cookie=e25fa974636111de in_seq=0 out_seq=0 2024-09-16T00:41:54.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:53 smithi033 ceph-mon[31243]: pgmap v1092: 129 pgs: 129 active+clean; 707 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 83 op/s 2024-09-16T00:41:54.106 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223241 17.91 MB/sec execute 540 sec latency 484.765 ms 2024-09-16T00:41:54.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:53 smithi083 ceph-mon[35242]: pgmap v1092: 129 pgs: 129 active+clean; 707 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 83 op/s 2024-09-16T00:41:54.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:53 smithi060 ceph-mon[35185]: pgmap v1092: 129 pgs: 129 active+clean; 707 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 83 op/s 2024-09-16T00:41:54.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff1ef7fe640 1 -- 172.21.15.33:0/3949598299 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e94a227d20 con 0x55e94a340dc0 2024-09-16T00:41:54.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff1ef7fe640 10 client.20007 handle_command_reply: tid=0 2024-09-16T00:41:54.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff218877740 1 client.20007 shutdown 2024-09-16T00:41:54.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff218877740 1 -- 172.21.15.33:0/3949598299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 msgr2=0x55e94a2ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:54.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff218877740 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2ca360 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7ff208003bb0 tx=0x7ff208004970 comp rx=0 tx=0).stop 2024-09-16T00:41:54.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff218877740 1 -- 172.21.15.33:0/3949598299 shutdown_connections 2024-09-16T00:41:54.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff218877740 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e94a340dc0 0x55e94a3431e0 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:54.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff218877740 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e94a2c5940 0x55e94a2ca360 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:54.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff218877740 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e94a2bb080 0x55e94a2c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:54.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff218877740 1 --2- 172.21.15.33:0/3949598299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e94a2ba760 0x55e94a2c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:54.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff1ef7fe640 0 client.20007 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:41:54.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff218877740 1 -- 172.21.15.33:0/3949598299 >> 172.21.15.33:0/3949598299 conn(0x55e94a2b6cf0 msgr2=0x55e94a2bc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:54.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff218877740 1 -- 172.21.15.33:0/3949598299 shutdown_connections 2024-09-16T00:41:54.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.408+0000 7ff218877740 1 -- 172.21.15.33:0/3949598299 wait complete. 2024-09-16T00:41:54.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff218877740 20 client.20007 trim_cache size 0 max 16384 2024-09-16T00:41:54.410 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:41:54.410 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:41:54.411 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:41:54.411 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 86, 2024-09-16T00:41:54.411 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:41:54.411 INFO:teuthology.orchestra.run.smithi033.stdout: "78738c5c": { 2024-09-16T00:41:54.411 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 86, 2024-09-16T00:41:54.411 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:41:54.411 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:41:54.411 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:41:54.411 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:41:54.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:41:54.411 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:41:54.412 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 147.2, 2024-09-16T00:41:54.412 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:41:54.412 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:41:54.412 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:41:54.412 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:41:54.412 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-09-16T00:41:54.412 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:41:54.412 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:54.412 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:54.412 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:54.412 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:54.412 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:41:54.413 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:41:54.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff21649e640 1 -- 172.21.15.33:0/3952073955 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff1f80a3240 msgr2=0x7ff1f80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:54.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff21649e640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff1f80a3240 0x7ff1f80a5700 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff208005fd0 tx=0x7ff208009550 comp rx=0 tx=0).stop 2024-09-16T00:41:54.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff21649e640 1 -- 172.21.15.33:0/3952073955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff210108e00 msgr2=0x7ff2101a4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:41:54.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff21649e640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff210108e00 0x7ff2101a4be0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff204006010 tx=0x7ff2040042f0 comp rx=0 tx=0).stop 2024-09-16T00:41:54.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff21649e640 1 -- 172.21.15.33:0/3952073955 shutdown_connections 2024-09-16T00:41:54.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff21649e640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ff1f80a3240 0x7ff1f80a5700 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:54.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff21649e640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff21010d530 0x7ff21019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:54.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff21649e640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff210108e00 0x7ff2101a4be0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:54.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff21649e640 1 --2- 172.21.15.33:0/3952073955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff210108450 0x7ff2101a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:41:54.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff21649e640 1 -- 172.21.15.33:0/3952073955 >> 172.21.15.33:0/3952073955 conn(0x7ff210077ba0 msgr2=0x7ff210078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:41:54.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff21649e640 1 -- 172.21.15.33:0/3952073955 shutdown_connections 2024-09-16T00:41:54.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:41:54.409+0000 7ff21649e640 1 -- 172.21.15.33:0/3952073955 wait complete. 2024-09-16T00:41:55.106 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.88 MB/sec execute 541 sec latency 1484.863 ms 2024-09-16T00:41:55.537 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '78738c5c', quiesce took 5.3 seconds. Will release after: 21.9 2024-09-16T00:41:56.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:55 smithi033 ceph-mon[31243]: pgmap v1093: 129 pgs: 129 active+clean; 697 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-09-16T00:41:56.106 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.85 MB/sec execute 542 sec latency 2484.960 ms 2024-09-16T00:41:56.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:55 smithi083 ceph-mon[35242]: pgmap v1093: 129 pgs: 129 active+clean; 697 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-09-16T00:41:56.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:55 smithi060 ceph-mon[35185]: pgmap v1093: 129 pgs: 129 active+clean; 697 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-09-16T00:41:57.106 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.81 MB/sec execute 543 sec latency 3485.066 ms 2024-09-16T00:41:58.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:57 smithi033 ceph-mon[31243]: pgmap v1094: 129 pgs: 129 active+clean; 692 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 218 op/s 2024-09-16T00:41:58.106 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.78 MB/sec execute 544 sec latency 4485.173 ms 2024-09-16T00:41:58.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:57 smithi083 ceph-mon[35242]: pgmap v1094: 129 pgs: 129 active+clean; 692 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 218 op/s 2024-09-16T00:41:58.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:57 smithi060 ceph-mon[35185]: pgmap v1094: 129 pgs: 129 active+clean; 692 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 218 op/s 2024-09-16T00:41:59.106 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.75 MB/sec execute 545 sec latency 5485.226 ms 2024-09-16T00:42:00.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:41:59 smithi033 ceph-mon[31243]: pgmap v1095: 129 pgs: 129 active+clean; 690 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.4 MiB/s wr, 228 op/s 2024-09-16T00:42:00.107 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.72 MB/sec execute 546 sec latency 6485.342 ms 2024-09-16T00:42:00.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:41:59 smithi083 ceph-mon[35242]: pgmap v1095: 129 pgs: 129 active+clean; 690 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.4 MiB/s wr, 228 op/s 2024-09-16T00:42:00.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:41:59 smithi060 ceph-mon[35185]: pgmap v1095: 129 pgs: 129 active+clean; 690 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.4 MiB/s wr, 228 op/s 2024-09-16T00:42:01.107 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.68 MB/sec execute 547 sec latency 7485.392 ms 2024-09-16T00:42:02.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:01 smithi033 ceph-mon[31243]: pgmap v1096: 129 pgs: 129 active+clean; 690 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.9 MiB/s wr, 238 op/s 2024-09-16T00:42:02.107 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.65 MB/sec execute 548 sec latency 8485.492 ms 2024-09-16T00:42:02.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:01 smithi083 ceph-mon[35242]: pgmap v1096: 129 pgs: 129 active+clean; 690 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.9 MiB/s wr, 238 op/s 2024-09-16T00:42:02.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:01 smithi060 ceph-mon[35185]: pgmap v1096: 129 pgs: 129 active+clean; 690 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.9 MiB/s wr, 238 op/s 2024-09-16T00:42:03.107 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.62 MB/sec execute 549 sec latency 9485.603 ms 2024-09-16T00:42:03.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:02 smithi083 ceph-mon[35242]: pgmap v1097: 129 pgs: 129 active+clean; 688 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.8 MiB/s wr, 208 op/s 2024-09-16T00:42:03.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:02 smithi060 ceph-mon[35185]: pgmap v1097: 129 pgs: 129 active+clean; 688 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.8 MiB/s wr, 208 op/s 2024-09-16T00:42:03.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:02 smithi033 ceph-mon[31243]: pgmap v1097: 129 pgs: 129 active+clean; 688 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.8 MiB/s wr, 208 op/s 2024-09-16T00:42:04.107 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.59 MB/sec execute 550 sec latency 10485.706 ms 2024-09-16T00:42:05.107 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.55 MB/sec execute 551 sec latency 11485.826 ms 2024-09-16T00:42:05.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:05 smithi060 ceph-mon[35185]: pgmap v1098: 129 pgs: 129 active+clean; 688 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.1 MiB/s wr, 177 op/s 2024-09-16T00:42:05.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:05 smithi033 ceph-mon[31243]: pgmap v1098: 129 pgs: 129 active+clean; 688 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.1 MiB/s wr, 177 op/s 2024-09-16T00:42:05.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:05 smithi083 ceph-mon[35242]: pgmap v1098: 129 pgs: 129 active+clean; 688 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.1 MiB/s wr, 177 op/s 2024-09-16T00:42:06.107 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.52 MB/sec execute 552 sec latency 12485.884 ms 2024-09-16T00:42:06.724 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:42:06.985 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:07.107 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.49 MB/sec execute 553 sec latency 13485.924 ms 2024-09-16T00:42:07.448 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:07 smithi033 ceph-mon[31243]: pgmap v1099: 129 pgs: 129 active+clean; 688 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 622 KiB/s wr, 99 op/s 2024-09-16T00:42:07.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:07 smithi060 ceph-mon[35185]: pgmap v1099: 129 pgs: 129 active+clean; 688 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 622 KiB/s wr, 99 op/s 2024-09-16T00:42:07.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.533+0000 7f25a0dc9640 1 -- 172.21.15.33:0/2318536627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f259c1036c0 msgr2=0x7f259c10bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:07.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.533+0000 7f25a0dc9640 1 --2- 172.21.15.33:0/2318536627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f259c1036c0 0x7f259c10bbb0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f2584009900 tx=0x7f258402a480 comp rx=0 tx=0).stop 2024-09-16T00:42:07.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.533+0000 7f25a0dc9640 1 -- 172.21.15.33:0/2318536627 shutdown_connections 2024-09-16T00:42:07.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.533+0000 7f25a0dc9640 1 --2- 172.21.15.33:0/2318536627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f259c10c180 0x7f259c10e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.533+0000 7f25a0dc9640 1 --2- 172.21.15.33:0/2318536627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f259c1036c0 0x7f259c10bbb0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.533+0000 7f25a0dc9640 1 --2- 172.21.15.33:0/2318536627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c102da0 0x7f259c103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.533+0000 7f25a0dc9640 1 -- 172.21.15.33:0/2318536627 >> 172.21.15.33:0/2318536627 conn(0x7f259c0fc7a0 msgr2=0x7f259c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:07.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.533+0000 7f25a0dc9640 1 -- 172.21.15.33:0/2318536627 shutdown_connections 2024-09-16T00:42:07.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.533+0000 7f25a0dc9640 1 -- 172.21.15.33:0/2318536627 wait complete. 2024-09-16T00:42:07.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.534+0000 7f25a0dc9640 1 Processor -- start 2024-09-16T00:42:07.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.534+0000 7f25a0dc9640 1 -- start start 2024-09-16T00:42:07.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.534+0000 7f25a0dc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c102da0 0x7f259c19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.534+0000 7f25a0dc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f259c1036c0 0x7f259c19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f25a0dc9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f259c10c180 0x7f259c19ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f25a0dc9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f259c111f60 con 0x7f259c10c180 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f25a0dc9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f259c111de0 con 0x7f259c102da0 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f25a0dc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f259c1120e0 con 0x7f259c1036c0 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f259ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f259c10c180 0x7f259c19ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f259ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f259c10c180 0x7f259c19ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34358/0 (socket says 172.21.15.33:34358) 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f259ad9d640 1 -- 172.21.15.33:0/3241126962 learned_addr learned my addr 172.21.15.33:0/3241126962 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f259a59c640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c102da0 0x7f259c19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f2591d9b640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f259c1036c0 0x7f259c19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f259ad9d640 1 -- 172.21.15.33:0/3241126962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f259c1036c0 msgr2=0x7f259c19aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f259ad9d640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f259c1036c0 0x7f259c19aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f259ad9d640 1 -- 172.21.15.33:0/3241126962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c102da0 msgr2=0x7f259c19a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f259ad9d640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c102da0 0x7f259c19a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f259ad9d640 1 -- 172.21.15.33:0/3241126962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f259c19f560 con 0x7f259c10c180 2024-09-16T00:42:07.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f259ad9d640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f259c10c180 0x7f259c19ee80 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f259400cad0 tx=0x7f259400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f2593fff640 1 -- 172.21.15.33:0/3241126962 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259400ad60 con 0x7f259c10c180 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f2593fff640 1 -- 172.21.15.33:0/3241126962 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2594005020 con 0x7f259c10c180 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.535+0000 7f25a0dc9640 1 -- 172.21.15.33:0/3241126962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f259c19f850 con 0x7f259c10c180 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.536+0000 7f2593fff640 1 -- 172.21.15.33:0/3241126962 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259400f6b0 con 0x7f259c10c180 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.536+0000 7f25a0dc9640 1 -- 172.21.15.33:0/3241126962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f259c1a7130 con 0x7f259c10c180 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f2593fff640 1 -- 172.21.15.33:0/3241126962 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2594005560 con 0x7f259c10c180 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f25a31a2740 1 Processor -- start 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f25a31a2740 1 -- start start 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f25a31a2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 0x55aa7525d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f25a31a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525ddc0 0x55aa75268140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f25a31a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 0x55aa7526aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f25a31a2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55aa751c32e0 con 0x55aa7525d4a0 2024-09-16T00:42:07.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f25a31a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55aa7516ace0 con 0x55aa75268680 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f25a31a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55aa751bf2a0 con 0x55aa7525ddc0 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f2591d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 0x55aa7525d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f2591d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 0x55aa7525d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34364/0 (socket says 172.21.15.33:34364) 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f2591d9b640 1 -- 172.21.15.33:0/3006021701 learned_addr learned my addr 172.21.15.33:0/3006021701 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.539+0000 7f2593fff640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f25880a3240 0x7f25880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f259a59c640 1 --2- 172.21.15.33:0/3006021701 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525ddc0 0x55aa75268140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f259ad9d640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f25880a3240 0x7f25880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f2591d9b640 1 --2- 172.21.15.33:0/3006021701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 0x55aa7526aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f2593fff640 1 -- 172.21.15.33:0/3241126962 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f25940cfc80 con 0x7f259c10c180 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f2591d9b640 1 -- 172.21.15.33:0/3006021701 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525ddc0 msgr2=0x55aa75268140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f2591d9b640 1 --2- 172.21.15.33:0/3006021701 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525ddc0 0x55aa75268140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f2591d9b640 1 -- 172.21.15.33:0/3006021701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 msgr2=0x55aa7526aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f2591d9b640 1 --2- 172.21.15.33:0/3006021701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 0x55aa7526aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f2591d9b640 1 -- 172.21.15.33:0/3006021701 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa7514f440 con 0x55aa7525d4a0 2024-09-16T00:42:07.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f2591d9b640 1 --2- 172.21.15.33:0/3006021701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 0x55aa7525d880 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f2584005ec0 tx=0x7f2584005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf6e5f4010dc8b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:07.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f259ad9d640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f25880a3240 0x7f25880a5700 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f259409bb20 tx=0x7f2594066860 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:07.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f2590d99640 1 -- 172.21.15.33:0/3006021701 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584035040 con 0x55aa7525d4a0 2024-09-16T00:42:07.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f2590d99640 1 -- 172.21.15.33:0/3006021701 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2584042070 con 0x55aa7525d4a0 2024-09-16T00:42:07.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.540+0000 7f2590d99640 1 -- 172.21.15.33:0/3006021701 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584004b20 con 0x55aa7525d4a0 2024-09-16T00:42:07.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.541+0000 7f25a31a2740 1 -- 172.21.15.33:0/3006021701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 msgr2=0x55aa7525d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:07.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.541+0000 7f25a31a2740 1 --2- 172.21.15.33:0/3006021701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 0x55aa7525d880 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f2584005ec0 tx=0x7f2584005d90 comp rx=0 tx=0).stop 2024-09-16T00:42:07.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.541+0000 7f25a31a2740 1 -- 172.21.15.33:0/3006021701 shutdown_connections 2024-09-16T00:42:07.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.541+0000 7f25a31a2740 1 --2- 172.21.15.33:0/3006021701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 0x55aa7526aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.541+0000 7f25a31a2740 1 --2- 172.21.15.33:0/3006021701 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525ddc0 0x55aa75268140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.541+0000 7f25a31a2740 1 --2- 172.21.15.33:0/3006021701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 0x55aa7525d880 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.541+0000 7f25a31a2740 1 -- 172.21.15.33:0/3006021701 >> 172.21.15.33:0/3006021701 conn(0x55aa75259a30 msgr2=0x55aa75259e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.541+0000 7f25a31a2740 1 -- 172.21.15.33:0/3006021701 shutdown_connections 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.541+0000 7f25a31a2740 1 -- 172.21.15.33:0/3006021701 wait complete. 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.541+0000 7f25a31a2740 1 -- start start 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.541+0000 7f25a31a2740 1 Processor -- start 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f25a31a2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 0x55aa75267f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f25a31a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525ddc0 0x55aa75265220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f25a31a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 0x55aa75265760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f25a31a2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55aa7526b250 con 0x55aa7525d4a0 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f25a31a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55aa7517e050 con 0x55aa75268680 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f25a31a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55aa7526b3d0 con 0x55aa7525ddc0 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f25a31a2740 10 client.0 _ll_register_callbacks cb 0x55aa751be160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f2591d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 0x55aa75267f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f2591d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 0x55aa75267f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34380/0 (socket says 172.21.15.33:34380) 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f2591d9b640 1 -- 172.21.15.33:0/1585854375 learned_addr learned my addr 172.21.15.33:0/1585854375 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f259a59c640 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525ddc0 0x55aa75265220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f2591d9b640 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 0x55aa75265760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f259a59c640 1 -- 172.21.15.33:0/1585854375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 msgr2=0x55aa75265760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f259a59c640 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 0x55aa75265760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f259a59c640 1 -- 172.21.15.33:0/1585854375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 msgr2=0x55aa75267f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f259a59c640 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 0x55aa75267f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f259a59c640 1 -- 172.21.15.33:0/1585854375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa74f343c0 con 0x55aa7525ddc0 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f2591d9b640 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 0x55aa75267f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f259a59c640 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525ddc0 0x55aa75265220 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f258c010810 tx=0x7f258c010cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5ee28dcb5c452d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.542+0000 7f2581ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f2581ffb640 1 -- 172.21.15.33:0/1585854375 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c01e060 con 0x55aa7525ddc0 2024-09-16T00:42:07.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f25a31a2740 1 -- 172.21.15.33:0/1585854375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2584042070 con 0x55aa7525ddc0 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f2581ffb640 1 -- 172.21.15.33:0/1585854375 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f258c00fb90 con 0x55aa7525ddc0 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f2581ffb640 1 -- 172.21.15.33:0/1585854375 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f258c025050 con 0x55aa7525ddc0 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f25a31a2740 10 client.29122 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f25a31a2740 1 -- 172.21.15.33:0/1585854375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55aa75276c70 con 0x55aa7525ddc0 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f2581ffb640 1 -- 172.21.15.33:0/1585854375 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2584042070 con 0x55aa7525ddc0 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f2581ffb640 1 -- 172.21.15.33:0/1585854375 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f258c00e420 con 0x55aa7525ddc0 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f25a31a2740 10 client.29122 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f25a31a2740 10 client.29122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f25a31a2740 20 client.29122 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f25a31a2740 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55aa752780d0 0x55aa7527a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f25a31a2740 4 client.29122 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.543+0000 7f25a31a2740 1 -- 172.21.15.33:0/1585854375 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa751d0850 con 0x55aa752780d0 2024-09-16T00:42:07.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.544+0000 7f2591d9b640 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55aa752780d0 0x55aa7527a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.544+0000 7f2581ffb640 10 client.29122 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.544+0000 7f2591d9b640 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55aa752780d0 0x55aa7527a4f0 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=792999c94fba5546 server_cookie=c9db5f881b4f94db in_seq=0 out_seq=0 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f2581ffb640 1 -- 172.21.15.33:0/1585854375 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55aa751d0850 con 0x55aa752780d0 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f2581ffb640 10 client.29122 handle_command_reply: tid=0 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f25a31a2740 1 client.29122 shutdown 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f25a31a2740 1 -- 172.21.15.33:0/1585854375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525ddc0 msgr2=0x55aa75265220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f25a31a2740 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525ddc0 0x55aa75265220 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f258c010810 tx=0x7f258c010cd0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f25a31a2740 1 -- 172.21.15.33:0/1585854375 shutdown_connections 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f25a31a2740 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55aa752780d0 0x55aa7527a4f0 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f25a31a2740 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 0x55aa75265760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f25a31a2740 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525ddc0 0x55aa75265220 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f25a31a2740 1 --2- 172.21.15.33:0/1585854375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525d4a0 0x55aa75267f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f25a31a2740 1 -- 172.21.15.33:0/1585854375 >> 172.21.15.33:0/1585854375 conn(0x55aa75259a30 msgr2=0x55aa7525f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:07.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f2581ffb640 0 client.29122 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:07.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f25a31a2740 1 -- 172.21.15.33:0/1585854375 shutdown_connections 2024-09-16T00:42:07.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f25a31a2740 1 -- 172.21.15.33:0/1585854375 wait complete. 2024-09-16T00:42:07.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.545+0000 7f25a31a2740 20 client.29122 trim_cache size 0 max 16384 2024-09-16T00:42:07.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.562+0000 7f25a31a2740 1 Processor -- start 2024-09-16T00:42:07.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.562+0000 7f25a31a2740 1 -- start start 2024-09-16T00:42:07.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f25a31a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525d4a0 0x55aa752744b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f25a31a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa7525ddc0 0x55aa7527fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f25a31a2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa75268680 0x55aa75280300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f25a31a2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f259c1b3af0 con 0x55aa75268680 2024-09-16T00:42:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f25a31a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f259c1b13c0 con 0x55aa7525ddc0 2024-09-16T00:42:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f25a31a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f259c1af210 con 0x55aa7525d4a0 2024-09-16T00:42:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2591d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa75268680 0x55aa75280300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2591d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa75268680 0x55aa75280300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34384/0 (socket says 172.21.15.33:34384) 2024-09-16T00:42:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2591d9b640 1 -- 172.21.15.33:0/1475408866 learned_addr learned my addr 172.21.15.33:0/1475408866 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2591d9b640 1 --2- 172.21.15.33:0/1475408866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525d4a0 0x55aa752744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2591d9b640 1 -- 172.21.15.33:0/1475408866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525d4a0 msgr2=0x55aa752744b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f259a59c640 1 --2- 172.21.15.33:0/1475408866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa7525ddc0 0x55aa7527fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2591d9b640 1 --2- 172.21.15.33:0/1475408866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525d4a0 0x55aa752744b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2591d9b640 1 -- 172.21.15.33:0/1475408866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa7525ddc0 msgr2=0x55aa7527fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:07.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2591d9b640 1 --2- 172.21.15.33:0/1475408866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa7525ddc0 0x55aa7527fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2591d9b640 1 -- 172.21.15.33:0/1475408866 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa7525e330 con 0x55aa75268680 2024-09-16T00:42:07.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f259a59c640 1 --2- 172.21.15.33:0/1475408866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa7525ddc0 0x55aa7527fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:42:07.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2591d9b640 1 --2- 172.21.15.33:0/1475408866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa75268680 0x55aa75280300 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f2584005ec0 tx=0x7f258403c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac7760b4f126c22 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:07.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2583fff640 1 -- 172.21.15.33:0/1475408866 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584035040 con 0x55aa75268680 2024-09-16T00:42:07.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2583fff640 1 -- 172.21.15.33:0/1475408866 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2584042040 con 0x55aa75268680 2024-09-16T00:42:07.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.563+0000 7f2583fff640 1 -- 172.21.15.33:0/1475408866 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584004bc0 con 0x55aa75268680 2024-09-16T00:42:07.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.564+0000 7f25a31a2740 1 -- 172.21.15.33:0/1475408866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa75268680 msgr2=0x55aa75280300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:07.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.564+0000 7f25a31a2740 1 --2- 172.21.15.33:0/1475408866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa75268680 0x55aa75280300 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f2584005ec0 tx=0x7f258403c040 comp rx=0 tx=0).stop 2024-09-16T00:42:07.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.564+0000 7f25a31a2740 1 -- 172.21.15.33:0/1475408866 shutdown_connections 2024-09-16T00:42:07.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.564+0000 7f25a31a2740 1 --2- 172.21.15.33:0/1475408866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa75268680 0x55aa75280300 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.564+0000 7f25a31a2740 1 --2- 172.21.15.33:0/1475408866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa7525ddc0 0x55aa7527fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.564+0000 7f25a31a2740 1 --2- 172.21.15.33:0/1475408866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525d4a0 0x55aa752744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.564+0000 7f25a31a2740 1 -- 172.21.15.33:0/1475408866 >> 172.21.15.33:0/1475408866 conn(0x55aa75259a30 msgr2=0x55aa75264310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:07.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.564+0000 7f25a31a2740 1 -- 172.21.15.33:0/1475408866 shutdown_connections 2024-09-16T00:42:07.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.564+0000 7f25a31a2740 1 -- 172.21.15.33:0/1475408866 wait complete. 2024-09-16T00:42:07.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f25a31a2740 1 -- start start 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f25a31a2740 1 Processor -- start 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f25a31a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525d4a0 0x55aa7526d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f25a31a2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525ddc0 0x55aa7526dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f25a31a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 0x55aa752e0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f25a31a2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f259c1af880 con 0x55aa7525ddc0 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f25a31a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f259c1b0cf0 con 0x55aa75268680 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f25a31a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f259c1aff50 con 0x55aa7525d4a0 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f25a31a2740 10 client.0 _ll_register_callbacks cb 0x7f258c010450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f259a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525ddc0 0x55aa7526dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f259a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525ddc0 0x55aa7526dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34398/0 (socket says 172.21.15.33:34398) 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f259a59c640 1 -- 172.21.15.33:0/111943947 learned_addr learned my addr 172.21.15.33:0/111943947 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f2591d9b640 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525d4a0 0x55aa7526d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f2591d9b640 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 0x55aa752e0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f259a59c640 1 -- 172.21.15.33:0/111943947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525d4a0 msgr2=0x55aa7526d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:07.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f259a59c640 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525d4a0 0x55aa7526d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f259a59c640 1 -- 172.21.15.33:0/111943947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 msgr2=0x55aa752e0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:07.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f259a59c640 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 0x55aa752e0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.565+0000 7f259a59c640 1 -- 172.21.15.33:0/111943947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa7526e560 con 0x55aa7525ddc0 2024-09-16T00:42:07.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f259a59c640 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525ddc0 0x55aa7526dc60 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f258c00e7f0 tx=0x7f258c00a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd03f24d4824d40a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:07.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f2581ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:07.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f2581ffb640 1 -- 172.21.15.33:0/111943947 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c00ec70 con 0x55aa7525ddc0 2024-09-16T00:42:07.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f2581ffb640 1 -- 172.21.15.33:0/111943947 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f258c00e3b0 con 0x55aa7525ddc0 2024-09-16T00:42:07.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f25a31a2740 1 -- 172.21.15.33:0/111943947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2584042040 con 0x55aa7525ddc0 2024-09-16T00:42:07.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f2581ffb640 1 -- 172.21.15.33:0/111943947 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f258c01e020 con 0x55aa7525ddc0 2024-09-16T00:42:07.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f25a31a2740 10 client.20025 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f25a31a2740 1 -- 172.21.15.33:0/111943947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55aa752e2380 con 0x55aa7525ddc0 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f2581ffb640 1 -- 172.21.15.33:0/111943947 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2584042040 con 0x55aa7525ddc0 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f2581ffb640 1 -- 172.21.15.33:0/111943947 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f258c029be0 con 0x55aa7525ddc0 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f25a31a2740 10 client.20025 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f25a31a2740 10 client.20025 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f25a31a2740 20 client.20025 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f25a31a2740 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55aa752e2b80 0x55aa752e4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f25a31a2740 4 client.20025 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.566+0000 7f25a31a2740 1 -- 172.21.15.33:0/111943947 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55aa751cad60 con 0x55aa752e2b80 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f2591d9b640 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55aa752e2b80 0x55aa752e4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f2591d9b640 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55aa752e2b80 0x55aa752e4fa0 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a441ba0b1d361ddb server_cookie=577ea66f19cbe09 in_seq=0 out_seq=0 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f2581ffb640 10 client.20025 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:07.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f2581ffb640 1 -- 172.21.15.33:0/111943947 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55aa751cad60 con 0x55aa752e2b80 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f2581ffb640 10 client.20025 handle_command_reply: tid=0 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f25a31a2740 1 client.20025 shutdown 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f25a31a2740 1 -- 172.21.15.33:0/111943947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525ddc0 msgr2=0x55aa7526dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f25a31a2740 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525ddc0 0x55aa7526dc60 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f258c00e7f0 tx=0x7f258c00a780 comp rx=0 tx=0).stop 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f25a31a2740 1 -- 172.21.15.33:0/111943947 shutdown_connections 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f25a31a2740 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55aa752e2b80 0x55aa752e4fa0 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f25a31a2740 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aa75268680 0x55aa752e0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f25a31a2740 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa7525ddc0 0x55aa7526dc60 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f2581ffb640 0 client.20025 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.567+0000 7f25a31a2740 1 --2- 172.21.15.33:0/111943947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aa7525d4a0 0x55aa7526d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a31a2740 1 -- 172.21.15.33:0/111943947 >> 172.21.15.33:0/111943947 conn(0x55aa75259a30 msgr2=0x55aa75266a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a31a2740 1 -- 172.21.15.33:0/111943947 shutdown_connections 2024-09-16T00:42:07.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a31a2740 1 -- 172.21.15.33:0/111943947 wait complete. 2024-09-16T00:42:07.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a31a2740 20 client.20025 trim_cache size 0 max 16384 2024-09-16T00:42:07.574 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:42:07.574 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:42:07.574 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:42:07.574 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:42:07.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a0dc9640 1 -- 172.21.15.33:0/3241126962 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f25880a3240 msgr2=0x7f25880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:07.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a0dc9640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f25880a3240 0x7f25880a5700 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f259409bb20 tx=0x7f2594066860 comp rx=0 tx=0).stop 2024-09-16T00:42:07.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a0dc9640 1 -- 172.21.15.33:0/3241126962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f259c10c180 msgr2=0x7f259c19ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:07.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a0dc9640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f259c10c180 0x7f259c19ee80 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f259400cad0 tx=0x7f259400ced0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a0dc9640 1 -- 172.21.15.33:0/3241126962 shutdown_connections 2024-09-16T00:42:07.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a0dc9640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f25880a3240 0x7f25880a5700 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a0dc9640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f259c10c180 0x7f259c19ee80 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a0dc9640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f259c1036c0 0x7f259c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a0dc9640 1 --2- 172.21.15.33:0/3241126962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c102da0 0x7f259c19a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:07.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a0dc9640 1 -- 172.21.15.33:0/3241126962 >> 172.21.15.33:0/3241126962 conn(0x7f259c0fc7a0 msgr2=0x7f259c100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:07.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a0dc9640 1 -- 172.21.15.33:0/3241126962 shutdown_connections 2024-09-16T00:42:07.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:07.568+0000 7f25a0dc9640 1 -- 172.21.15.33:0/3241126962 wait complete. 2024-09-16T00:42:07.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:07 smithi083 ceph-mon[35242]: pgmap v1099: 129 pgs: 129 active+clean; 688 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 622 KiB/s wr, 99 op/s 2024-09-16T00:42:08.107 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.46 MB/sec execute 554 sec latency 14486.042 ms 2024-09-16T00:42:08.754 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:42:08.754 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:42:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:09.107 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.43 MB/sec execute 555 sec latency 15486.145 ms 2024-09-16T00:42:09.451 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:09 smithi033 ceph-mon[31243]: pgmap v1100: 129 pgs: 129 active+clean; 688 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 124 KiB/s wr, 41 op/s 2024-09-16T00:42:09.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:09 smithi060 ceph-mon[35185]: pgmap v1100: 129 pgs: 129 active+clean; 688 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 124 KiB/s wr, 41 op/s 2024-09-16T00:42:09.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.532+0000 7f89c83eb640 1 -- 172.21.15.33:0/1204486380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89c0101420 msgr2=0x7f89c010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:09.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.532+0000 7f89c83eb640 1 --2- 172.21.15.33:0/1204486380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89c0101420 0x7f89c010d950 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f89b8009cd0 tx=0x7f89b802a8e0 comp rx=0 tx=0).stop 2024-09-16T00:42:09.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.532+0000 7f89c83eb640 1 -- 172.21.15.33:0/1204486380 shutdown_connections 2024-09-16T00:42:09.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.532+0000 7f89c83eb640 1 --2- 172.21.15.33:0/1204486380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89c0101420 0x7f89c010d950 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:09.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.532+0000 7f89c83eb640 1 --2- 172.21.15.33:0/1204486380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89c0100a80 0x7f89c0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:09.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.532+0000 7f89c83eb640 1 --2- 172.21.15.33:0/1204486380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a80 0x7f89c0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:09.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.532+0000 7f89c83eb640 1 -- 172.21.15.33:0/1204486380 >> 172.21.15.33:0/1204486380 conn(0x7f89c00fc7a0 msgr2=0x7f89c00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:09.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.532+0000 7f89c83eb640 1 -- 172.21.15.33:0/1204486380 shutdown_connections 2024-09-16T00:42:09.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.532+0000 7f89c83eb640 1 -- 172.21.15.33:0/1204486380 wait complete. 2024-09-16T00:42:09.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c83eb640 1 Processor -- start 2024-09-16T00:42:09.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c83eb640 1 -- start start 2024-09-16T00:42:09.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c83eb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89c0100a80 0x7f89c019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:09.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c83eb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0101420 0x7f89c019ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:09.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c83eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89c0106a80 0x7f89c019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:09.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c83eb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f89c0110e80 con 0x7f89c0100a80 2024-09-16T00:42:09.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c83eb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f89c0110d00 con 0x7f89c0101420 2024-09-16T00:42:09.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c83eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89c0111000 con 0x7f89c0106a80 2024-09-16T00:42:09.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c6187640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89c0100a80 0x7f89c019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:09.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c6187640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89c0100a80 0x7f89c019a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34406/0 (socket says 172.21.15.33:34406) 2024-09-16T00:42:09.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c6187640 1 -- 172.21.15.33:0/3566873100 learned_addr learned my addr 172.21.15.33:0/3566873100 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:09.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c5986640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0101420 0x7f89c019ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:09.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c6988640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89c0106a80 0x7f89c019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:09.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c6187640 1 -- 172.21.15.33:0/3566873100 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89c0106a80 msgr2=0x7f89c019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:09.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c6187640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89c0106a80 0x7f89c019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:09.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c6187640 1 -- 172.21.15.33:0/3566873100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0101420 msgr2=0x7f89c019ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:09.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.533+0000 7f89c6187640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0101420 0x7f89c019ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.534+0000 7f89c6187640 1 -- 172.21.15.33:0/3566873100 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c019f5e0 con 0x7f89c0100a80 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.534+0000 7f89c6187640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89c0100a80 0x7f89c019a630 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00e9f0 tx=0x7f89bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.534+0000 7f89b77fe640 1 -- 172.21.15.33:0/3566873100 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc00cd80 con 0x7f89c0100a80 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.534+0000 7f89b77fe640 1 -- 172.21.15.33:0/3566873100 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89bc010910 con 0x7f89c0100a80 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.534+0000 7f89c83eb640 1 -- 172.21.15.33:0/3566873100 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c019f8d0 con 0x7f89c0100a80 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.534+0000 7f89b77fe640 1 -- 172.21.15.33:0/3566873100 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc018990 con 0x7f89c0100a80 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.534+0000 7f89c83eb640 1 -- 172.21.15.33:0/3566873100 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c01a71b0 con 0x7f89c0100a80 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.535+0000 7f89c83eb640 1 -- 172.21.15.33:0/3566873100 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c0100ee0 con 0x7f89c0100a80 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.536+0000 7f89b77fe640 1 -- 172.21.15.33:0/3566873100 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f89bc010ac0 con 0x7f89c0100a80 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.537+0000 7f89b77fe640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89ac0a32c0 0x7f89ac0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.537+0000 7f89c5986640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89ac0a32c0 0x7f89ac0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.537+0000 7f89b77fe640 1 -- 172.21.15.33:0/3566873100 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f89bc0d0010 con 0x7f89c0100a80 2024-09-16T00:42:09.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.537+0000 7f89c5986640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89ac0a32c0 0x7f89ac0a5780 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f89c00febe0 tx=0x7f89b000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:09.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.541+0000 7f89b77fe640 1 -- 172.21.15.33:0/3566873100 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f89bc09c8b0 con 0x7f89c0100a80 2024-09-16T00:42:09.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:09 smithi083 ceph-mon[35242]: pgmap v1100: 129 pgs: 129 active+clean; 688 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 124 KiB/s wr, 41 op/s 2024-09-16T00:42:09.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.759+0000 7f89c83eb640 1 -- 172.21.15.33:0/3566873100 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89c019b990 con 0x7f89c0100a80 2024-09-16T00:42:09.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.759+0000 7f89b77fe640 1 -- 172.21.15.33:0/3566873100 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f89bc09c1b0 con 0x7f89c0100a80 2024-09-16T00:42:09.760 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:42:09.760 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:42:09.760 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:42:09.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.762+0000 7f89c83eb640 1 -- 172.21.15.33:0/3566873100 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89ac0a32c0 msgr2=0x7f89ac0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:09.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.762+0000 7f89c83eb640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89ac0a32c0 0x7f89ac0a5780 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f89c00febe0 tx=0x7f89b000b390 comp rx=0 tx=0).stop 2024-09-16T00:42:09.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.762+0000 7f89c83eb640 1 -- 172.21.15.33:0/3566873100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89c0100a80 msgr2=0x7f89c019a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:09.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.762+0000 7f89c83eb640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89c0100a80 0x7f89c019a630 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00e9f0 tx=0x7f89bc00eeb0 comp rx=0 tx=0).stop 2024-09-16T00:42:09.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.762+0000 7f89c83eb640 1 -- 172.21.15.33:0/3566873100 shutdown_connections 2024-09-16T00:42:09.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.762+0000 7f89c83eb640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f89ac0a32c0 0x7f89ac0a5780 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:09.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.762+0000 7f89c83eb640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89c0106a80 0x7f89c019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:09.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.762+0000 7f89c83eb640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0101420 0x7f89c019ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:09.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.762+0000 7f89c83eb640 1 --2- 172.21.15.33:0/3566873100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f89c0100a80 0x7f89c019a630 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:09.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.762+0000 7f89c83eb640 1 -- 172.21.15.33:0/3566873100 >> 172.21.15.33:0/3566873100 conn(0x7f89c00fc7a0 msgr2=0x7f89c006ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:09.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.762+0000 7f89c83eb640 1 -- 172.21.15.33:0/3566873100 shutdown_connections 2024-09-16T00:42:09.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:09.762+0000 7f89c83eb640 1 -- 172.21.15.33:0/3566873100 wait complete. 2024-09-16T00:42:10.107 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.40 MB/sec execute 556 sec latency 16486.183 ms 2024-09-16T00:42:10.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:10 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3566873100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:42:10.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:10 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3566873100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:42:10.484 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:42:10.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:10 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3566873100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:42:10.742 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:11.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.37 MB/sec execute 557 sec latency 17486.285 ms 2024-09-16T00:42:11.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.247+0000 7f21bd38c640 1 -- 172.21.15.33:0/2104140199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21b8102d40 msgr2=0x7f21b8103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:11.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.247+0000 7f21bd38c640 1 --2- 172.21.15.33:0/2104140199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21b8102d40 0x7f21b8103120 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f21a8009900 tx=0x7f21a802a480 comp rx=0 tx=0).stop 2024-09-16T00:42:11.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.247+0000 7f21bd38c640 1 -- 172.21.15.33:0/2104140199 shutdown_connections 2024-09-16T00:42:11.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.247+0000 7f21bd38c640 1 --2- 172.21.15.33:0/2104140199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21b810c170 0x7f21b810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.247+0000 7f21bd38c640 1 --2- 172.21.15.33:0/2104140199 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b8103660 0x7f21b810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.247+0000 7f21bd38c640 1 --2- 172.21.15.33:0/2104140199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21b8102d40 0x7f21b8103120 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.247+0000 7f21bd38c640 1 -- 172.21.15.33:0/2104140199 >> 172.21.15.33:0/2104140199 conn(0x7f21b80fc7a0 msgr2=0x7f21b80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:11.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.247+0000 7f21bd38c640 1 -- 172.21.15.33:0/2104140199 shutdown_connections 2024-09-16T00:42:11.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.247+0000 7f21bd38c640 1 -- 172.21.15.33:0/2104140199 wait complete. 2024-09-16T00:42:11.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.248+0000 7f21bd38c640 1 Processor -- start 2024-09-16T00:42:11.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.248+0000 7f21bd38c640 1 -- start start 2024-09-16T00:42:11.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.248+0000 7f21bd38c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21b8102d40 0x7f21b81984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.248+0000 7f21bd38c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b8103660 0x7f21b8198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.248+0000 7f21bd38c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21b810c170 0x7f21b819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.248+0000 7f21bd38c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21b8111fd0 con 0x7f21b810c170 2024-09-16T00:42:11.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.248+0000 7f21bd38c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f21b8111e50 con 0x7f21b8103660 2024-09-16T00:42:11.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.248+0000 7f21bd38c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21b8112150 con 0x7f21b8102d40 2024-09-16T00:42:11.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21b810c170 0x7f21b819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21b810c170 0x7f21b819cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42442/0 (socket says 172.21.15.33:42442) 2024-09-16T00:42:11.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21b77fe640 1 -- 172.21.15.33:0/1497349705 learned_addr learned my addr 172.21.15.33:0/1497349705 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:11.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21b8102d40 0x7f21b81984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21b67fc640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b8103660 0x7f21b8198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21b77fe640 1 -- 172.21.15.33:0/1497349705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21b8102d40 msgr2=0x7f21b81984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:11.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21b77fe640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21b8102d40 0x7f21b81984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21b77fe640 1 -- 172.21.15.33:0/1497349705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b8103660 msgr2=0x7f21b8198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21b77fe640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b8103660 0x7f21b8198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21b77fe640 1 -- 172.21.15.33:0/1497349705 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b819d480 con 0x7f21b810c170 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21b8102d40 0x7f21b81984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21b77fe640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21b810c170 0x7f21b819cda0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f21b000d750 tx=0x7f21b000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f219ffff640 1 -- 172.21.15.33:0/1497349705 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b0017070 con 0x7f21b810c170 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f219ffff640 1 -- 172.21.15.33:0/1497349705 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21b0005500 con 0x7f21b810c170 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21bd38c640 1 -- 172.21.15.33:0/1497349705 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b819d770 con 0x7f21b810c170 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f219ffff640 1 -- 172.21.15.33:0/1497349705 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b0004d90 con 0x7f21b810c170 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.249+0000 7f21bd38c640 1 -- 172.21.15.33:0/1497349705 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b81a5050 con 0x7f21b810c170 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f219ffff640 1 -- 172.21.15.33:0/1497349705 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f21b0004f30 con 0x7f21b810c170 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f21bf765740 1 Processor -- start 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f21bf765740 1 -- start start 2024-09-16T00:42:11.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f21bf765740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd98455090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f21bf765740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 0x55cd98455a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f21bf765740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98464780 0x55cd98466b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f21bf765740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cd983bb2f0 con 0x55cd984555d0 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f21bf765740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55cd98362cf0 con 0x55cd98456b40 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f21bf765740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cd983b72b0 con 0x55cd98464780 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f21b6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 0x55cd98455a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f21b6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 0x55cd98455a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42450/0 (socket says 172.21.15.33:42450) 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f21b6ffd640 1 -- 172.21.15.33:0/3167341340 learned_addr learned my addr 172.21.15.33:0/3167341340 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.252+0000 7f219ffff640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21a00a3240 0x7f21a00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f21b67fc640 1 --2- 172.21.15.33:0/3167341340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98464780 0x55cd98466b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f21b67fc640 1 --2- 172.21.15.33:0/3167341340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd98455090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f21b6ffd640 1 -- 172.21.15.33:0/3167341340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98464780 msgr2=0x55cd98466b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:11.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/3167341340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98464780 0x55cd98466b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f21b6ffd640 1 -- 172.21.15.33:0/3167341340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 msgr2=0x55cd98455090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f21b77fe640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21a00a3240 0x7f21a00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/3167341340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd98455090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f21b6ffd640 1 -- 172.21.15.33:0/3167341340 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd983473a0 con 0x55cd984555d0 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f21b67fc640 1 --2- 172.21.15.33:0/3167341340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd98455090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f219ffff640 1 -- 172.21.15.33:0/1497349705 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f21b00cf810 con 0x7f21b810c170 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/3167341340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 0x55cd98455a10 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f21a8033000 tx=0x7f21a8032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d124e060537da46a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f219d7fa640 1 -- 172.21.15.33:0/3167341340 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a803f070 con 0x55cd984555d0 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f21b77fe640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21a00a3240 0x7f21a00a5700 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f21b00ccee0 tx=0x7f21b00c6780 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f219d7fa640 1 -- 172.21.15.33:0/3167341340 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21a80317b0 con 0x55cd984555d0 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.253+0000 7f219d7fa640 1 -- 172.21.15.33:0/3167341340 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a8035070 con 0x55cd984555d0 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.254+0000 7f21bf765740 1 -- 172.21.15.33:0/3167341340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 msgr2=0x55cd98455a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:11.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.254+0000 7f21bf765740 1 --2- 172.21.15.33:0/3167341340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 0x55cd98455a10 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f21a8033000 tx=0x7f21a8032d60 comp rx=0 tx=0).stop 2024-09-16T00:42:11.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.254+0000 7f21bf765740 1 -- 172.21.15.33:0/3167341340 shutdown_connections 2024-09-16T00:42:11.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.254+0000 7f21bf765740 1 --2- 172.21.15.33:0/3167341340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98464780 0x55cd98466b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.254+0000 7f21bf765740 1 --2- 172.21.15.33:0/3167341340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 0x55cd98455a10 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.254+0000 7f21bf765740 1 --2- 172.21.15.33:0/3167341340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd98455090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.254+0000 7f21bf765740 1 -- 172.21.15.33:0/3167341340 >> 172.21.15.33:0/3167341340 conn(0x55cd9841e310 msgr2=0x55cd98451c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:11.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.254+0000 7f21bf765740 1 -- 172.21.15.33:0/3167341340 shutdown_connections 2024-09-16T00:42:11.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.254+0000 7f21bf765740 1 -- 172.21.15.33:0/3167341340 wait complete. 2024-09-16T00:42:11.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.254+0000 7f21bf765740 1 -- start start 2024-09-16T00:42:11.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21bf765740 1 Processor -- start 2024-09-16T00:42:11.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21bf765740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd984555d0 0x55cd98464190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21bf765740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98456b40 0x55cd98461440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21bf765740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 0x55cd98461a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21bf765740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cd98467420 con 0x55cd98464780 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21bf765740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55cd98376060 con 0x55cd984555d0 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21bf765740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cd984675a0 con 0x55cd98456b40 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21bf765740 10 client.0 _ll_register_callbacks cb 0x55cd980c98c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21b67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 0x55cd98461a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21b67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 0x55cd98461a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42460/0 (socket says 172.21.15.33:42460) 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21b6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98456b40 0x55cd98461440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21b67fc640 1 -- 172.21.15.33:0/3056107675 learned_addr learned my addr 172.21.15.33:0/3056107675 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21b67fc640 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd984555d0 0x55cd98464190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21b67fc640 1 -- 172.21.15.33:0/3056107675 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98456b40 msgr2=0x55cd98461440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21b67fc640 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98456b40 0x55cd98461440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21b67fc640 1 -- 172.21.15.33:0/3056107675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd984555d0 msgr2=0x55cd98464190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:11.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21b67fc640 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd984555d0 0x55cd98464190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21b67fc640 1 -- 172.21.15.33:0/3056107675 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd9812c3c0 con 0x55cd98464780 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.255+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98456b40 0x55cd98461440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f21b67fc640 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 0x55cd98461a50 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00ae90 tx=0x7f21ac00c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65264d99d6fdfb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f219aff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f219aff5640 1 -- 172.21.15.33:0/3056107675 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac020060 con 0x55cd98464780 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f219aff5640 1 -- 172.21.15.33:0/3056107675 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f21ac017560 con 0x55cd98464780 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f21bf765740 1 -- 172.21.15.33:0/3056107675 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21a80317b0 con 0x55cd98464780 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f219aff5640 1 -- 172.21.15.33:0/3056107675 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f21ac027050 con 0x55cd98464780 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f21bf765740 10 client.20043 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f21bf765740 1 -- 172.21.15.33:0/3056107675 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55cd98472f10 con 0x55cd98464780 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f219aff5640 1 -- 172.21.15.33:0/3056107675 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21a80317b0 con 0x55cd98464780 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f219aff5640 1 -- 172.21.15.33:0/3056107675 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f21ac01a600 con 0x55cd98464780 2024-09-16T00:42:11.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f21bf765740 10 client.20043 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f21bf765740 10 client.20043 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f21bf765740 20 client.20043 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.256+0000 7f21bf765740 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cd984743d0 0x55cd984767c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.257+0000 7f21bf765740 4 client.20043 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.257+0000 7f21bf765740 1 -- 172.21.15.33:0/3056107675 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd983c8860 con 0x55cd984743d0 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.257+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cd984743d0 0x55cd984767c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.257+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cd984743d0 0x55cd984767c0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d61202901887d9b2 server_cookie=a5f3eb914a100254 in_seq=0 out_seq=0 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.257+0000 7f219aff5640 10 client.20043 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f219aff5640 1 -- 172.21.15.33:0/3056107675 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cd983c8860 con 0x55cd984743d0 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f219aff5640 10 client.20043 handle_command_reply: tid=0 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f21bf765740 1 client.20043 shutdown 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f21bf765740 1 -- 172.21.15.33:0/3056107675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 msgr2=0x55cd98461a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:11.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f21bf765740 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 0x55cd98461a50 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00ae90 tx=0x7f21ac00c040 comp rx=0 tx=0).stop 2024-09-16T00:42:11.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f21bf765740 1 -- 172.21.15.33:0/3056107675 shutdown_connections 2024-09-16T00:42:11.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f21bf765740 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cd984743d0 0x55cd984767c0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f21bf765740 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 0x55cd98461a50 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f21bf765740 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98456b40 0x55cd98461440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f21bf765740 1 --2- 172.21.15.33:0/3056107675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd984555d0 0x55cd98464190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f21bf765740 1 -- 172.21.15.33:0/3056107675 >> 172.21.15.33:0/3056107675 conn(0x55cd9841e310 msgr2=0x55cd98457930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:11.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f219aff5640 0 client.20043 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:42:11.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f21bf765740 1 -- 172.21.15.33:0/3056107675 shutdown_connections 2024-09-16T00:42:11.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f21bf765740 1 -- 172.21.15.33:0/3056107675 wait complete. 2024-09-16T00:42:11.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.258+0000 7f21bf765740 20 client.20043 trim_cache size 0 max 16384 2024-09-16T00:42:11.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.271+0000 7f21bf765740 1 Processor -- start 2024-09-16T00:42:11.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.271+0000 7f21bf765740 1 -- start start 2024-09-16T00:42:11.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21bf765740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 0x55cd984706c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21bf765740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd98451700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21bf765740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98464780 0x55cd98451c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21bf765740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81af730 con 0x55cd984555d0 2024-09-16T00:42:11.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21bf765740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81ad000 con 0x55cd98456b40 2024-09-16T00:42:11.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21bf765740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81aae50 con 0x55cd98464780 2024-09-16T00:42:11.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21b67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 0x55cd984706c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21b67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 0x55cd984706c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42464/0 (socket says 172.21.15.33:42464) 2024-09-16T00:42:11.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21b67fc640 1 -- 172.21.15.33:0/2355809018 learned_addr learned my addr 172.21.15.33:0/2355809018 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:11.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/2355809018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd98451700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21b67fc640 1 -- 172.21.15.33:0/2355809018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98464780 msgr2=0x55cd98451c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:11.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21b67fc640 1 --2- 172.21.15.33:0/2355809018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98464780 0x55cd98451c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21b67fc640 1 -- 172.21.15.33:0/2355809018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 msgr2=0x55cd98451700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:11.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21b67fc640 1 --2- 172.21.15.33:0/2355809018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd98451700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21b67fc640 1 -- 172.21.15.33:0/2355809018 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd98452180 con 0x55cd984555d0 2024-09-16T00:42:11.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/2355809018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd98451700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:42:11.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.272+0000 7f21b67fc640 1 --2- 172.21.15.33:0/2355809018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 0x55cd984706c0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00aa80 tx=0x7f21ac00a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c68c7111429476b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:11.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.273+0000 7f219cff9640 1 -- 172.21.15.33:0/2355809018 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac030070 con 0x55cd984555d0 2024-09-16T00:42:11.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.273+0000 7f219cff9640 1 -- 172.21.15.33:0/2355809018 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21ac019e40 con 0x55cd984555d0 2024-09-16T00:42:11.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.273+0000 7f219cff9640 1 -- 172.21.15.33:0/2355809018 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac017ca0 con 0x55cd984555d0 2024-09-16T00:42:11.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.273+0000 7f21bf765740 1 -- 172.21.15.33:0/2355809018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 msgr2=0x55cd984706c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:11.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.273+0000 7f21bf765740 1 --2- 172.21.15.33:0/2355809018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 0x55cd984706c0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00aa80 tx=0x7f21ac00a480 comp rx=0 tx=0).stop 2024-09-16T00:42:11.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.273+0000 7f21bf765740 1 -- 172.21.15.33:0/2355809018 shutdown_connections 2024-09-16T00:42:11.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.273+0000 7f21bf765740 1 --2- 172.21.15.33:0/2355809018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd98464780 0x55cd98451c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.273+0000 7f21bf765740 1 --2- 172.21.15.33:0/2355809018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd98451700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.273+0000 7f21bf765740 1 --2- 172.21.15.33:0/2355809018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd984555d0 0x55cd984706c0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.273+0000 7f21bf765740 1 -- 172.21.15.33:0/2355809018 >> 172.21.15.33:0/2355809018 conn(0x55cd9841e310 msgr2=0x55cd98461400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:11.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.273+0000 7f21bf765740 1 -- 172.21.15.33:0/2355809018 shutdown_connections 2024-09-16T00:42:11.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.273+0000 7f21bf765740 1 -- 172.21.15.33:0/2355809018 wait complete. 2024-09-16T00:42:11.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21bf765740 1 -- start start 2024-09-16T00:42:11.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21bf765740 1 Processor -- start 2024-09-16T00:42:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21bf765740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd984555d0 0x55cd9846ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21bf765740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd9845dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21bf765740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 0x55cd9845e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21bf765740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81ab4c0 con 0x55cd98464780 2024-09-16T00:42:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21bf765740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81ac930 con 0x55cd98456b40 2024-09-16T00:42:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21bf765740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81abb90 con 0x55cd984555d0 2024-09-16T00:42:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21bf765740 10 client.0 _ll_register_callbacks cb 0x7f21b806b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21b67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 0x55cd9845e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21b67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 0x55cd9845e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42474/0 (socket says 172.21.15.33:42474) 2024-09-16T00:42:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21b67fc640 1 -- 172.21.15.33:0/697933819 learned_addr learned my addr 172.21.15.33:0/697933819 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21b67fc640 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd984555d0 0x55cd9846ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd9845dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21b67fc640 1 -- 172.21.15.33:0/697933819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd984555d0 msgr2=0x55cd9846ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21b67fc640 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd984555d0 0x55cd9846ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21b67fc640 1 -- 172.21.15.33:0/697933819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 msgr2=0x55cd9845dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21b67fc640 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd9845dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21b67fc640 1 -- 172.21.15.33:0/697933819 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd9845ebf0 con 0x55cd98464780 2024-09-16T00:42:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.274+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd9845dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.275+0000 7f21b67fc640 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 0x55cd9845e500 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f21ac01ab70 tx=0x7f21ac017880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107788d6ed93e5b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.275+0000 7f219aff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.275+0000 7f219aff5640 1 -- 172.21.15.33:0/697933819 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac030040 con 0x55cd98464780 2024-09-16T00:42:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.275+0000 7f219aff5640 1 -- 172.21.15.33:0/697933819 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f21ac01abf0 con 0x55cd98464780 2024-09-16T00:42:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.275+0000 7f21bf765740 1 -- 172.21.15.33:0/697933819 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cd983c2d70 con 0x55cd98464780 2024-09-16T00:42:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.275+0000 7f219aff5640 1 -- 172.21.15.33:0/697933819 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f21ac01a3d0 con 0x55cd98464780 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.275+0000 7f21bf765740 10 client.20049 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.275+0000 7f21bf765740 1 -- 172.21.15.33:0/697933819 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55cd984dee50 con 0x55cd98464780 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.275+0000 7f219aff5640 1 -- 172.21.15.33:0/697933819 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55cd983c2d70 con 0x55cd98464780 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.275+0000 7f219aff5640 1 -- 172.21.15.33:0/697933819 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f21ac033070 con 0x55cd98464780 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.276+0000 7f21bf765740 10 client.20049 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.276+0000 7f21bf765740 10 client.20049 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.276+0000 7f21bf765740 20 client.20049 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.276+0000 7f21bf765740 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cd984df4e0 0x55cd984e1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.276+0000 7f21bf765740 4 client.20049 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.276+0000 7f21bf765740 1 -- 172.21.15.33:0/697933819 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cd9833f170 con 0x55cd984df4e0 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.276+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cd984df4e0 0x55cd984e1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.276+0000 7f21b6ffd640 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cd984df4e0 0x55cd984e1900 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5ea24db368aeb80e server_cookie=52f2e0cec3404485 in_seq=0 out_seq=0 2024-09-16T00:42:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.276+0000 7f219aff5640 10 client.20049 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f219aff5640 1 -- 172.21.15.33:0/697933819 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cd9833f170 con 0x55cd984df4e0 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f219aff5640 10 client.20049 handle_command_reply: tid=0 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bf765740 1 client.20049 shutdown 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bf765740 1 -- 172.21.15.33:0/697933819 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 msgr2=0x55cd9845e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bf765740 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 0x55cd9845e500 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f21ac01ab70 tx=0x7f21ac017880 comp rx=0 tx=0).stop 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bf765740 1 -- 172.21.15.33:0/697933819 shutdown_connections 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bf765740 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x55cd984df4e0 0x55cd984e1900 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f219aff5640 0 client.20049 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bf765740 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd98464780 0x55cd9845e500 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bf765740 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cd98456b40 0x55cd9845dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bf765740 1 --2- 172.21.15.33:0/697933819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd984555d0 0x55cd9846ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bf765740 1 -- 172.21.15.33:0/697933819 >> 172.21.15.33:0/697933819 conn(0x55cd9841e310 msgr2=0x55cd9845f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bf765740 1 -- 172.21.15.33:0/697933819 shutdown_connections 2024-09-16T00:42:11.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bf765740 1 -- 172.21.15.33:0/697933819 wait complete. 2024-09-16T00:42:11.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bf765740 20 client.20049 trim_cache size 0 max 16384 2024-09-16T00:42:11.283 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:42:11.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.277+0000 7f21bd38c640 1 -- 172.21.15.33:0/1497349705 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21a00a3240 msgr2=0x7f21a00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:11.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.278+0000 7f21bd38c640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21a00a3240 0x7f21a00a5700 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f21b00ccee0 tx=0x7f21b00c6780 comp rx=0 tx=0).stop 2024-09-16T00:42:11.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.278+0000 7f21bd38c640 1 -- 172.21.15.33:0/1497349705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21b810c170 msgr2=0x7f21b819cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:11.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.278+0000 7f21bd38c640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21b810c170 0x7f21b819cda0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f21b000d750 tx=0x7f21b000dc10 comp rx=0 tx=0).stop 2024-09-16T00:42:11.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.278+0000 7f21bd38c640 1 -- 172.21.15.33:0/1497349705 shutdown_connections 2024-09-16T00:42:11.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.278+0000 7f21bd38c640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f21a00a3240 0x7f21a00a5700 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.278+0000 7f21bd38c640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21b810c170 0x7f21b819cda0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.278+0000 7f21bd38c640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b8103660 0x7f21b8198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.278+0000 7f21bd38c640 1 --2- 172.21.15.33:0/1497349705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21b8102d40 0x7f21b81984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:11.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.278+0000 7f21bd38c640 1 -- 172.21.15.33:0/1497349705 >> 172.21.15.33:0/1497349705 conn(0x7f21b80fc7a0 msgr2=0x7f21b80fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:11.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.278+0000 7f21bd38c640 1 -- 172.21.15.33:0/1497349705 shutdown_connections 2024-09-16T00:42:11.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:11.278+0000 7f21bd38c640 1 -- 172.21.15.33:0/1497349705 wait complete. 2024-09-16T00:42:11.480 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:11 smithi060 ceph-mon[35185]: pgmap v1101: 129 pgs: 129 active+clean; 688 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 64 KiB/s wr, 33 op/s 2024-09-16T00:42:11.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:11 smithi033 ceph-mon[31243]: pgmap v1101: 129 pgs: 129 active+clean; 688 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 64 KiB/s wr, 33 op/s 2024-09-16T00:42:11.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:11 smithi083 ceph-mon[35242]: pgmap v1101: 129 pgs: 129 active+clean; 688 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 64 KiB/s wr, 33 op/s 2024-09-16T00:42:12.012 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:42:12.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.33 MB/sec execute 558 sec latency 18486.348 ms 2024-09-16T00:42:12.270 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:12.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.862+0000 7f55a76fc640 1 -- 172.21.15.33:0/993927060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55a0110220 msgr2=0x7f55a006e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:12.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.862+0000 7f55a76fc640 1 --2- 172.21.15.33:0/993927060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55a0110220 0x7f55a006e4f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f559c009900 tx=0x7f559c02a440 comp rx=0 tx=0).stop 2024-09-16T00:42:12.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.867+0000 7f55a76fc640 1 -- 172.21.15.33:0/993927060 shutdown_connections 2024-09-16T00:42:12.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.867+0000 7f55a76fc640 1 --2- 172.21.15.33:0/993927060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55a011b140 0x7f55a011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.867+0000 7f55a76fc640 1 --2- 172.21.15.33:0/993927060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a006ea30 0x7f55a006ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.867+0000 7f55a76fc640 1 --2- 172.21.15.33:0/993927060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55a0110220 0x7f55a006e4f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.867+0000 7f55a76fc640 1 -- 172.21.15.33:0/993927060 >> 172.21.15.33:0/993927060 conn(0x7f55a0073210 msgr2=0x7f55a0073620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:12.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.867+0000 7f55a76fc640 1 -- 172.21.15.33:0/993927060 shutdown_connections 2024-09-16T00:42:12.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.868+0000 7f55a76fc640 1 -- 172.21.15.33:0/993927060 wait complete. 2024-09-16T00:42:12.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.868+0000 7f55a76fc640 1 Processor -- start 2024-09-16T00:42:12.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.869+0000 7f55a76fc640 1 -- start start 2024-09-16T00:42:12.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.869+0000 7f55a76fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a006ea30 0x7f55a01a71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.869+0000 7f55a76fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55a0110220 0x7f55a01a7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.869+0000 7f55a76fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55a011b140 0x7f55a01abaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.869+0000 7f55a76fc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f55a0120e00 con 0x7f55a006ea30 2024-09-16T00:42:12.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.869+0000 7f55a76fc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f55a0120c80 con 0x7f55a0110220 2024-09-16T00:42:12.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.869+0000 7f55a76fc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f55a0120f80 con 0x7f55a011b140 2024-09-16T00:42:12.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.870+0000 7f55a5498640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a006ea30 0x7f55a01a71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.870+0000 7f55a5498640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a006ea30 0x7f55a01a71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42490/0 (socket says 172.21.15.33:42490) 2024-09-16T00:42:12.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.870+0000 7f55a5498640 1 -- 172.21.15.33:0/2470667541 learned_addr learned my addr 172.21.15.33:0/2470667541 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:12.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.870+0000 7f55a5c99640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55a011b140 0x7f55a01abaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.871+0000 7f55a5c99640 1 -- 172.21.15.33:0/2470667541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55a0110220 msgr2=0x7f55a01a7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:12.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.871+0000 7f55a5c99640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55a0110220 0x7f55a01a7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.871+0000 7f55a5c99640 1 -- 172.21.15.33:0/2470667541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a006ea30 msgr2=0x7f55a01a71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:12.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.871+0000 7f55a5c99640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a006ea30 0x7f55a01a71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.871+0000 7f55a5c99640 1 -- 172.21.15.33:0/2470667541 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a01ac180 con 0x7f55a011b140 2024-09-16T00:42:12.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.871+0000 7f55a5c99640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55a011b140 0x7f55a01abaa0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f559800be20 tx=0x7f559800bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:12.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.871+0000 7f55967fc640 1 -- 172.21.15.33:0/2470667541 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5598019070 con 0x7f55a011b140 2024-09-16T00:42:12.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.871+0000 7f55a76fc640 1 -- 172.21.15.33:0/2470667541 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a01ac470 con 0x7f55a011b140 2024-09-16T00:42:12.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.871+0000 7f55967fc640 1 -- 172.21.15.33:0/2470667541 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559800cdf0 con 0x7f55a011b140 2024-09-16T00:42:12.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.871+0000 7f55967fc640 1 -- 172.21.15.33:0/2470667541 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55980128b0 con 0x7f55a011b140 2024-09-16T00:42:12.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.872+0000 7f55a76fc640 1 -- 172.21.15.33:0/2470667541 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a01b3d50 con 0x7f55a011b140 2024-09-16T00:42:12.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.876+0000 7f55a9ad5740 1 Processor -- start 2024-09-16T00:42:12.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.877+0000 7f55967fc640 1 -- 172.21.15.33:0/2470667541 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f559801c070 con 0x7f55a011b140 2024-09-16T00:42:12.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.878+0000 7f55967fc640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f558c0a3380 0x7f558c0a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.879+0000 7f55967fc640 1 -- 172.21.15.33:0/2470667541 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f55980cf570 con 0x7f55a011b140 2024-09-16T00:42:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.879+0000 7f55a4c97640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f558c0a3380 0x7f558c0a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.879+0000 7f55a9ad5740 1 -- start start 2024-09-16T00:42:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.880+0000 7f55a9ad5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad9cf0 0x556a95ad8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.880+0000 7f55a9ad5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ad8780 0x556a95ad8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.880+0000 7f55a9ad5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ae3520 0x556a95ae5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.880+0000 7f55a9ad5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556a95a3e2f0 con 0x556a95ad9cf0 2024-09-16T00:42:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.880+0000 7f55a9ad5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556a959e5cf0 con 0x556a95ae3520 2024-09-16T00:42:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.880+0000 7f55a9ad5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556a95a3a2b0 con 0x556a95ad8780 2024-09-16T00:42:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.880+0000 7f55a4c97640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f558c0a3380 0x7f558c0a5840 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f55a010e5e0 tx=0x7f5590006c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.880+0000 7f55a4c97640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ad8780 0x556a95ad8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.880+0000 7f55a4c97640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ad8780 0x556a95ad8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40502/0 (socket says 172.21.15.33:40502) 2024-09-16T00:42:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.880+0000 7f55a4c97640 1 -- 172.21.15.33:0/3941504193 learned_addr learned my addr 172.21.15.33:0/3941504193 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.880+0000 7f55a5498640 1 --2- 172.21.15.33:0/3941504193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad9cf0 0x556a95ad8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.880+0000 7f55a5c99640 1 --2- 172.21.15.33:0/3941504193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ae3520 0x556a95ae5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.881+0000 7f55a5498640 1 -- 172.21.15.33:0/3941504193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ad8780 msgr2=0x556a95ad8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.881+0000 7f55a5498640 1 --2- 172.21.15.33:0/3941504193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ad8780 0x556a95ad8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.881+0000 7f55a5498640 1 -- 172.21.15.33:0/3941504193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ae3520 msgr2=0x556a95ae5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.881+0000 7f55a5498640 1 --2- 172.21.15.33:0/3941504193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ae3520 0x556a95ae5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.881+0000 7f55a5498640 1 -- 172.21.15.33:0/3941504193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a959ca3a0 con 0x556a95ad9cf0 2024-09-16T00:42:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.881+0000 7f55a5498640 1 --2- 172.21.15.33:0/3941504193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad9cf0 0x556a95ad8240 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f559c000c00 tx=0x7f559c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a016f5a2067971e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.881+0000 7f558b7fe640 1 -- 172.21.15.33:0/3941504193 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c040070 con 0x556a95ad9cf0 2024-09-16T00:42:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.881+0000 7f558b7fe640 1 -- 172.21.15.33:0/3941504193 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559c0047a0 con 0x556a95ad9cf0 2024-09-16T00:42:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.882+0000 7f558b7fe640 1 -- 172.21.15.33:0/3941504193 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c035040 con 0x556a95ad9cf0 2024-09-16T00:42:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.882+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3941504193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad9cf0 msgr2=0x556a95ad8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.882+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3941504193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad9cf0 0x556a95ad8240 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f559c000c00 tx=0x7f559c0043b0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.884+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3941504193 shutdown_connections 2024-09-16T00:42:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.884+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3941504193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ae3520 0x556a95ae5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.884+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3941504193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ad8780 0x556a95ad8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.884+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3941504193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad9cf0 0x556a95ad8240 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.884+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3941504193 >> 172.21.15.33:0/3941504193 conn(0x556a95aa1310 msgr2=0x556a95ad4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.884+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3941504193 shutdown_connections 2024-09-16T00:42:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.884+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3941504193 wait complete. 2024-09-16T00:42:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.884+0000 7f55a9ad5740 1 -- start start 2024-09-16T00:42:12.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.884+0000 7f55a9ad5740 1 Processor -- start 2024-09-16T00:42:12.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.885+0000 7f55a9ad5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad8780 0x556a95ae2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.885+0000 7f55a9ad5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad9cf0 0x556a95ae0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.885+0000 7f55a9ad5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ae3520 0x556a95ae0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.885+0000 7f55a9ad5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556a95ae6210 con 0x556a95ad9cf0 2024-09-16T00:42:12.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.885+0000 7f55a9ad5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556a959f9060 con 0x556a95ad8780 2024-09-16T00:42:12.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.885+0000 7f55a9ad5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556a95ae6390 con 0x556a95ae3520 2024-09-16T00:42:12.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.885+0000 7f55a9ad5740 10 client.0 _ll_register_callbacks cb 0x556a9574c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:12.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.886+0000 7f55a5c99640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ae3520 0x556a95ae0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.886+0000 7f55a5c99640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ae3520 0x556a95ae0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:40516/0 (socket says 172.21.15.33:40516) 2024-09-16T00:42:12.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.886+0000 7f55a5c99640 1 -- 172.21.15.33:0/324715638 learned_addr learned my addr 172.21.15.33:0/324715638 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:12.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.886+0000 7f55a5c99640 1 -- 172.21.15.33:0/324715638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad8780 msgr2=0x556a95ae2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:12.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.886+0000 7f55a5c99640 1 --2- 172.21.15.33:0/324715638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad8780 0x556a95ae2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.886+0000 7f55a5c99640 1 -- 172.21.15.33:0/324715638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad9cf0 msgr2=0x556a95ae0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:12.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.886+0000 7f55a5c99640 1 --2- 172.21.15.33:0/324715638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad9cf0 0x556a95ae0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.886+0000 7f55a5c99640 1 -- 172.21.15.33:0/324715638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a957af3c0 con 0x556a95ae3520 2024-09-16T00:42:12.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.886+0000 7f55a5c99640 1 --2- 172.21.15.33:0/324715638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ae3520 0x556a95ae0720 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f559800bf80 tx=0x7f5598004b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca7299b2a9ca253f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.889+0000 7f5588ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:42:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.889+0000 7f5588ff9640 1 -- 172.21.15.33:0/324715638 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55980cf360 con 0x556a95ae3520 2024-09-16T00:42:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.889+0000 7f5588ff9640 1 -- 172.21.15.33:0/324715638 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f55980afbd0 con 0x556a95ae3520 2024-09-16T00:42:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.889+0000 7f55a9ad5740 1 -- 172.21.15.33:0/324715638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f559c0047a0 con 0x556a95ae3520 2024-09-16T00:42:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.889+0000 7f5588ff9640 1 -- 172.21.15.33:0/324715638 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f55980ba0d0 con 0x556a95ae3520 2024-09-16T00:42:12.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.891+0000 7f55a9ad5740 10 client.29158 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:12.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.891+0000 7f55a9ad5740 1 -- 172.21.15.33:0/324715638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556a95af1cc0 con 0x556a95ae3520 2024-09-16T00:42:12.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.893+0000 7f5588ff9640 1 -- 172.21.15.33:0/324715638 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559c0047a0 con 0x556a95ae3520 2024-09-16T00:42:12.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.893+0000 7f5588ff9640 1 -- 172.21.15.33:0/324715638 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f55980bb040 con 0x556a95ae3520 2024-09-16T00:42:12.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.894+0000 7f55a9ad5740 10 client.29158 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:12.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.894+0000 7f55a9ad5740 10 client.29158 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:42:12.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.894+0000 7f55a9ad5740 20 client.29158 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:12.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.894+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/324715638 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x556a95af30d0 0x556a95af5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.894+0000 7f55a9ad5740 4 client.29158 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:42:12.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.894+0000 7f55a9ad5740 1 -- 172.21.15.33:0/324715638 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a95a4b860 con 0x556a95af30d0 2024-09-16T00:42:12.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.895+0000 7f55a5498640 1 --2- 172.21.15.33:0/324715638 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x556a95af30d0 0x556a95af5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.897+0000 7f55a5498640 1 --2- 172.21.15.33:0/324715638 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x556a95af30d0 0x556a95af5560 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=627d4312918b121 server_cookie=bb548bc369f83e5b in_seq=0 out_seq=0 2024-09-16T00:42:12.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.897+0000 7f5588ff9640 10 client.29158 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:42:12.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.897+0000 7f5588ff9640 1 -- 172.21.15.33:0/324715638 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556a95a4b860 con 0x556a95af30d0 2024-09-16T00:42:12.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.898+0000 7f5588ff9640 10 client.29158 handle_command_reply: tid=0 2024-09-16T00:42:12.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.898+0000 7f55a9ad5740 1 client.29158 shutdown 2024-09-16T00:42:12.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.898+0000 7f55a9ad5740 1 -- 172.21.15.33:0/324715638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ae3520 msgr2=0x556a95ae0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:12.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.898+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/324715638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ae3520 0x556a95ae0720 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f559800bf80 tx=0x7f5598004b50 comp rx=0 tx=0).stop 2024-09-16T00:42:12.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.898+0000 7f55a9ad5740 1 -- 172.21.15.33:0/324715638 shutdown_connections 2024-09-16T00:42:12.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.898+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/324715638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ae3520 0x556a95ae0720 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.898+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/324715638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad9cf0 0x556a95ae0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.898+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/324715638 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x556a95af30d0 0x556a95af5560 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.898+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/324715638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad8780 0x556a95ae2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.898+0000 7f55a9ad5740 1 -- 172.21.15.33:0/324715638 >> 172.21.15.33:0/324715638 conn(0x556a95aa1310 msgr2=0x556a95adab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:12.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.898+0000 7f5588ff9640 0 client.29158 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:42:12.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.904+0000 7f55a9ad5740 1 -- 172.21.15.33:0/324715638 shutdown_connections 2024-09-16T00:42:12.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.904+0000 7f55a9ad5740 1 -- 172.21.15.33:0/324715638 wait complete. 2024-09-16T00:42:12.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.906+0000 7f55a9ad5740 20 client.29158 trim_cache size 0 max 16384 2024-09-16T00:42:12.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.930+0000 7f55a9ad5740 1 Processor -- start 2024-09-16T00:42:12.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.930+0000 7f55a9ad5740 1 -- start start 2024-09-16T00:42:12.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a9ad5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad8780 0x556a95aef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a9ad5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad9cf0 0x556a95adb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a9ad5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ae3520 0x556a95adb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a9ad5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f55a01c0740 con 0x556a95ad8780 2024-09-16T00:42:12.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a9ad5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f55a01be010 con 0x556a95ad9cf0 2024-09-16T00:42:12.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a5498640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad8780 0x556a95aef4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a9ad5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f55a01bbe60 con 0x556a95ae3520 2024-09-16T00:42:12.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a4c97640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad9cf0 0x556a95adb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a5498640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad8780 0x556a95aef4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42522/0 (socket says 172.21.15.33:42522) 2024-09-16T00:42:12.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a5498640 1 -- 172.21.15.33:0/3638441315 learned_addr learned my addr 172.21.15.33:0/3638441315 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:12.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a5c99640 1 --2- 172.21.15.33:0/3638441315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ae3520 0x556a95adb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a5498640 1 -- 172.21.15.33:0/3638441315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ae3520 msgr2=0x556a95adb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:12.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a5498640 1 --2- 172.21.15.33:0/3638441315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ae3520 0x556a95adb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a5498640 1 -- 172.21.15.33:0/3638441315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad9cf0 msgr2=0x556a95adb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:12.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a5498640 1 --2- 172.21.15.33:0/3638441315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad9cf0 0x556a95adb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.931+0000 7f55a5498640 1 -- 172.21.15.33:0/3638441315 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a95adbd50 con 0x556a95ad8780 2024-09-16T00:42:12.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.932+0000 7f55a5498640 1 --2- 172.21.15.33:0/3638441315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad8780 0x556a95aef4b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f559c004650 tx=0x7f559c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e51f87535d86068 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:12.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.932+0000 7f558affd640 1 -- 172.21.15.33:0/3638441315 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c052070 con 0x556a95ad8780 2024-09-16T00:42:12.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.932+0000 7f558affd640 1 -- 172.21.15.33:0/3638441315 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559c0047a0 con 0x556a95ad8780 2024-09-16T00:42:12.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.932+0000 7f558affd640 1 -- 172.21.15.33:0/3638441315 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c040040 con 0x556a95ad8780 2024-09-16T00:42:12.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.933+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3638441315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad8780 msgr2=0x556a95aef4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:12.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.933+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3638441315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad8780 0x556a95aef4b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f559c004650 tx=0x7f559c0045e0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.933+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3638441315 shutdown_connections 2024-09-16T00:42:12.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.933+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3638441315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ae3520 0x556a95adb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.933+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3638441315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad9cf0 0x556a95adb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.933+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3638441315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ad8780 0x556a95aef4b0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.933+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3638441315 >> 172.21.15.33:0/3638441315 conn(0x556a95aa1310 msgr2=0x556a95ae0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:12.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.933+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3638441315 shutdown_connections 2024-09-16T00:42:12.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.933+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3638441315 wait complete. 2024-09-16T00:42:12.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.933+0000 7f55a9ad5740 1 -- start start 2024-09-16T00:42:12.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.933+0000 7f55a9ad5740 1 Processor -- start 2024-09-16T00:42:12.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a9ad5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad8780 0x556a95ae74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a9ad5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ad9cf0 0x556a95ae7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a9ad5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ae3520 0x556a95ae7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a9ad5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f55a01bc4d0 con 0x556a95ae3520 2024-09-16T00:42:12.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a9ad5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f55a01bd940 con 0x556a95ad8780 2024-09-16T00:42:12.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a9ad5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f55a01bcba0 con 0x556a95ad9cf0 2024-09-16T00:42:12.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a5c99640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ae3520 0x556a95ae7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a5498640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad8780 0x556a95ae74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a5498640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad8780 0x556a95ae74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:35994/0 (socket says 172.21.15.33:35994) 2024-09-16T00:42:12.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a5c99640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ae3520 0x556a95ae7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42534/0 (socket says 172.21.15.33:42534) 2024-09-16T00:42:12.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a9ad5740 10 client.0 _ll_register_callbacks cb 0x7f559809b0c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:12.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a5498640 1 -- 172.21.15.33:0/3906397591 learned_addr learned my addr 172.21.15.33:0/3906397591 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:12.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a4c97640 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ad9cf0 0x556a95ae7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a5498640 1 -- 172.21.15.33:0/3906397591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ad9cf0 msgr2=0x556a95ae7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:12.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a5498640 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ad9cf0 0x556a95ae7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a5498640 1 -- 172.21.15.33:0/3906397591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ae3520 msgr2=0x556a95ae7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:12.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a5498640 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ae3520 0x556a95ae7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a5498640 1 -- 172.21.15.33:0/3906397591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a95b5cfa0 con 0x556a95ad8780 2024-09-16T00:42:12.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a5c99640 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ae3520 0x556a95ae7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:12.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f55a5498640 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad8780 0x556a95ae74c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f559c000c00 tx=0x7f559c02a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc827021dd19b57 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:12.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.934+0000 7f5588ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-09-16T00:42:12.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f5588ff9640 1 -- 172.21.15.33:0/3906397591 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c052030 con 0x556a95ad8780 2024-09-16T00:42:12.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f5588ff9640 1 -- 172.21.15.33:0/3906397591 <== mon.2 v2:172.21.15.60:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f559c003ce0 con 0x556a95ad8780 2024-09-16T00:42:12.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3906397591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f559c0047a0 con 0x556a95ad8780 2024-09-16T00:42:12.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f5588ff9640 1 -- 172.21.15.33:0/3906397591 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f559c05f050 con 0x556a95ad8780 2024-09-16T00:42:12.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f55a9ad5740 10 client.29150 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:12.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3906397591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556a95b5e260 con 0x556a95ad8780 2024-09-16T00:42:12.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f5588ff9640 1 -- 172.21.15.33:0/3906397591 <== mon.2 v2:172.21.15.60:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559c0047a0 con 0x556a95ad8780 2024-09-16T00:42:12.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f5588ff9640 1 -- 172.21.15.33:0/3906397591 <== mon.2 v2:172.21.15.60:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f559c03c540 con 0x556a95ad8780 2024-09-16T00:42:12.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f55a9ad5740 10 client.29150 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:12.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f55a9ad5740 10 client.29150 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:42:12.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f55a9ad5740 20 client.29150 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:12.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x556a95b5e490 0x556a95b60880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:12.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f55a9ad5740 4 client.29150 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:42:12.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.935+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3906397591 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556a95a45d70 con 0x556a95b5e490 2024-09-16T00:42:12.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f55a4c97640 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x556a95b5e490 0x556a95b60880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:12.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f5588ff9640 10 client.29150 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:42:12.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f55a4c97640 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x556a95b5e490 0x556a95b60880 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2a0118cab401786b server_cookie=e1bd3ec217d523c4 in_seq=0 out_seq=0 2024-09-16T00:42:12.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f5588ff9640 1 -- 172.21.15.33:0/3906397591 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556a95a45d70 con 0x556a95b5e490 2024-09-16T00:42:12.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f5588ff9640 10 client.29150 handle_command_reply: tid=0 2024-09-16T00:42:12.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f55a9ad5740 1 client.29150 shutdown 2024-09-16T00:42:12.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3906397591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad8780 msgr2=0x556a95ae74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:12.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad8780 0x556a95ae74c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f559c000c00 tx=0x7f559c02a440 comp rx=0 tx=0).stop 2024-09-16T00:42:12.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3906397591 shutdown_connections 2024-09-16T00:42:12.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a95ae3520 0x556a95ae7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556a95ad9cf0 0x556a95ae7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x556a95b5e490 0x556a95b60880 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f5588ff9640 0 client.29150 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:42:12.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f55a9ad5740 1 --2- 172.21.15.33:0/3906397591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a95ad8780 0x556a95ae74c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.936+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3906397591 >> 172.21.15.33:0/3906397591 conn(0x556a95aa1310 msgr2=0x556a95ae19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:12.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3906397591 shutdown_connections 2024-09-16T00:42:12.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a9ad5740 1 -- 172.21.15.33:0/3906397591 wait complete. 2024-09-16T00:42:12.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a9ad5740 20 client.29150 trim_cache size 0 max 16384 2024-09-16T00:42:12.947 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:42:12.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a76fc640 1 -- 172.21.15.33:0/2470667541 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f558c0a3380 msgr2=0x7f558c0a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:12.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a76fc640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f558c0a3380 0x7f558c0a5840 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f55a010e5e0 tx=0x7f5590006c90 comp rx=0 tx=0).stop 2024-09-16T00:42:12.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a76fc640 1 -- 172.21.15.33:0/2470667541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55a011b140 msgr2=0x7f55a01abaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:12.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a76fc640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55a011b140 0x7f55a01abaa0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f559800be20 tx=0x7f559800bf10 comp rx=0 tx=0).stop 2024-09-16T00:42:12.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a76fc640 1 -- 172.21.15.33:0/2470667541 shutdown_connections 2024-09-16T00:42:12.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a76fc640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f558c0a3380 0x7f558c0a5840 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a76fc640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55a011b140 0x7f55a01abaa0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a76fc640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55a0110220 0x7f55a01a7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a76fc640 1 --2- 172.21.15.33:0/2470667541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a006ea30 0x7f55a01a71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:12.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.937+0000 7f55a76fc640 1 -- 172.21.15.33:0/2470667541 >> 172.21.15.33:0/2470667541 conn(0x7f55a0073210 msgr2=0x7f55a01191d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:12.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.938+0000 7f55a76fc640 1 -- 172.21.15.33:0/2470667541 shutdown_connections 2024-09-16T00:42:12.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:12.938+0000 7f55a76fc640 1 -- 172.21.15.33:0/2470667541 wait complete. 2024-09-16T00:42:13.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.30 MB/sec execute 559 sec latency 19486.384 ms 2024-09-16T00:42:13.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:13 smithi060 ceph-mon[35185]: pgmap v1102: 129 pgs: 129 active+clean; 696 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.1 MiB/s wr, 29 op/s 2024-09-16T00:42:13.743 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:13 smithi033 ceph-mon[31243]: pgmap v1102: 129 pgs: 129 active+clean; 696 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.1 MiB/s wr, 29 op/s 2024-09-16T00:42:13.772 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:42:13.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:13 smithi083 ceph-mon[35242]: pgmap v1102: 129 pgs: 129 active+clean; 696 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.1 MiB/s wr, 29 op/s 2024-09-16T00:42:14.028 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:14.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.27 MB/sec execute 560 sec latency 20486.497 ms 2024-09-16T00:42:14.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.513+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/2916030923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ba0106f20 msgr2=0x7f0ba010eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:14.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.513+0000 7f0ba4ec9640 1 --2- 172.21.15.33:0/2916030923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ba0106f20 0x7f0ba010eaa0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f0b98009ca0 tx=0x7f0b9802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.513+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/2916030923 shutdown_connections 2024-09-16T00:42:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.513+0000 7f0ba4ec9640 1 --2- 172.21.15.33:0/2916030923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ba0106f20 0x7f0ba010eaa0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.513+0000 7f0ba4ec9640 1 --2- 172.21.15.33:0/2916030923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba01027a0 0x7f0ba01067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.513+0000 7f0ba4ec9640 1 --2- 172.21.15.33:0/2916030923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ba0101df0 0x7f0ba01021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.513+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/2916030923 >> 172.21.15.33:0/2916030923 conn(0x7f0ba00fd8f0 msgr2=0x7f0ba00ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.513+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/2916030923 shutdown_connections 2024-09-16T00:42:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.513+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/2916030923 wait complete. 2024-09-16T00:42:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.514+0000 7f0ba4ec9640 1 Processor -- start 2024-09-16T00:42:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.514+0000 7f0ba4ec9640 1 -- start start 2024-09-16T00:42:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.514+0000 7f0ba4ec9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba0101df0 0x7f0ba00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.514+0000 7f0ba4ec9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ba01027a0 0x7f0ba0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.514+0000 7f0ba4ec9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ba0106f20 0x7f0ba0074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.514+0000 7f0ba4ec9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba0111f80 con 0x7f0ba01027a0 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.514+0000 7f0ba4ec9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba0111e00 con 0x7f0ba0106f20 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.514+0000 7f0ba4ec9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba0112100 con 0x7f0ba0101df0 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.514+0000 7f0b9dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ba01027a0 0x7f0ba0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b9dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ba01027a0 0x7f0ba0074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42554/0 (socket says 172.21.15.33:42554) 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b9dd9b640 1 -- 172.21.15.33:0/1273209734 learned_addr learned my addr 172.21.15.33:0/1273209734 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b9e59c640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba0101df0 0x7f0ba00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ba0106f20 0x7f0ba0074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b9dd9b640 1 -- 172.21.15.33:0/1273209734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba0101df0 msgr2=0x7f0ba00744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba0101df0 0x7f0ba00744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b9dd9b640 1 -- 172.21.15.33:0/1273209734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ba0106f20 msgr2=0x7f0ba0074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ba0106f20 0x7f0ba0074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b9dd9b640 1 -- 172.21.15.33:0/1273209734 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba01a6880 con 0x7f0ba01027a0 2024-09-16T00:42:14.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ba0106f20 0x7f0ba0074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:42:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ba01027a0 0x7f0ba0074a30 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400d950 tx=0x7f0b9400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b8f7fe640 1 -- 172.21.15.33:0/1273209734 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b94017030 con 0x7f0ba01027a0 2024-09-16T00:42:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/1273209734 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba01a6b70 con 0x7f0ba01027a0 2024-09-16T00:42:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b8f7fe640 1 -- 172.21.15.33:0/1273209734 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b94004020 con 0x7f0ba01027a0 2024-09-16T00:42:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0b8f7fe640 1 -- 172.21.15.33:0/1273209734 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b940047a0 con 0x7f0ba01027a0 2024-09-16T00:42:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.515+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/1273209734 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba01a70b0 con 0x7f0ba01027a0 2024-09-16T00:42:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.517+0000 7f0b8f7fe640 1 -- 172.21.15.33:0/1273209734 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0b940041d0 con 0x7f0ba01027a0 2024-09-16T00:42:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.518+0000 7f0ba72a2740 1 Processor -- start 2024-09-16T00:42:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.518+0000 7f0ba72a2740 1 -- start start 2024-09-16T00:42:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.518+0000 7f0ba72a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.518+0000 7f0ba72a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5aba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.518+0000 7f0ba72a2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5b8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.518+0000 7f0ba72a2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f9ea5112f0 con 0x55f9ea5b6370 2024-09-16T00:42:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.518+0000 7f0ba72a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f9ea4b8cf0 con 0x55f9ea5acb40 2024-09-16T00:42:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.518+0000 7f0ba72a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f9ea50d2b0 con 0x55f9ea5ab5d0 2024-09-16T00:42:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.518+0000 7f0b8f7fe640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b880a3240 0x7f0b880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5b8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42562/0 (socket says 172.21.15.33:42562) 2024-09-16T00:42:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9dd9b640 1 -- 172.21.15.33:0/2834405065 learned_addr learned my addr 172.21.15.33:0/2834405065 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b880a3240 0x7f0b880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9e59c640 1 --2- 172.21.15.33:0/2834405065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b8f7fe640 1 -- 172.21.15.33:0/1273209734 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0b940cf780 con 0x7f0ba01027a0 2024-09-16T00:42:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/2834405065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9dd9b640 1 -- 172.21.15.33:0/2834405065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 msgr2=0x55f9ea5aba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/2834405065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5aba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9dd9b640 1 -- 172.21.15.33:0/2834405065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 msgr2=0x55f9ea5ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/2834405065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9dd9b640 1 -- 172.21.15.33:0/2834405065 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9ea49d3a0 con 0x55f9ea5b6370 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/2834405065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/2834405065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5b8760 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f0b94005cd0 tx=0x7f0b94005d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b317b9cceac7aa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b880a3240 0x7f0b880a5700 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0b98006010 tx=0x7f0b98005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b8cff9640 1 -- 172.21.15.33:0/2834405065 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9409b220 con 0x55f9ea5b6370 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b8cff9640 1 -- 172.21.15.33:0/2834405065 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b9409bfc0 con 0x55f9ea5b6370 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.519+0000 7f0b8cff9640 1 -- 172.21.15.33:0/2834405065 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b940ccf80 con 0x55f9ea5b6370 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.520+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2834405065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 msgr2=0x55f9ea5b8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.520+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2834405065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5b8760 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f0b94005cd0 tx=0x7f0b94005d00 comp rx=0 tx=0).stop 2024-09-16T00:42:14.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.520+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2834405065 shutdown_connections 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.520+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2834405065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5b8760 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.520+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2834405065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5aba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.520+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2834405065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.520+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2834405065 >> 172.21.15.33:0/2834405065 conn(0x55f9ea574310 msgr2=0x55f9ea5a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.520+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2834405065 shutdown_connections 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.520+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2834405065 wait complete. 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0ba72a2740 1 -- start start 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0ba72a2740 1 Processor -- start 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0ba72a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0ba72a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0ba72a2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0ba72a2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f9ea5b9010 con 0x55f9ea5b6370 2024-09-16T00:42:14.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0ba72a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f9ea4cc060 con 0x55f9ea5acb40 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0ba72a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f9ea5b9190 con 0x55f9ea5ab5d0 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0ba72a2740 10 client.0 _ll_register_callbacks cb 0x55f9ea21f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0b9ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0b9ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42576/0 (socket says 172.21.15.33:42576) 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0b9ed9d640 1 -- 172.21.15.33:0/2322120974 learned_addr learned my addr 172.21.15.33:0/2322120974 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.521+0000 7f0b9e59c640 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b9ed9d640 1 -- 172.21.15.33:0/2322120974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 msgr2=0x55f9ea5b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b9ed9d640 1 -- 172.21.15.33:0/2322120974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 msgr2=0x55f9ea5b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b9ed9d640 1 -- 172.21.15.33:0/2322120974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9ea2823c0 con 0x55f9ea5b6370 2024-09-16T00:42:14.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b9e59c640 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5b34d0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f0b980099f0 tx=0x7f0b980095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7e9b52c6183420 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b85ffb640 1 -- 172.21.15.33:0/2322120974 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98031df0 con 0x55f9ea5b6370 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b85ffb640 1 -- 172.21.15.33:0/2322120974 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0b980305c0 con 0x55f9ea5b6370 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2322120974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b9409bfc0 con 0x55f9ea5b6370 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b85ffb640 1 -- 172.21.15.33:0/2322120974 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0b980307c0 con 0x55f9ea5b6370 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0ba72a2740 10 client.20073 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2322120974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f9ea5c4a50 con 0x55f9ea5b6370 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b85ffb640 1 -- 172.21.15.33:0/2322120974 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b9409bfc0 con 0x55f9ea5b6370 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.522+0000 7f0b85ffb640 1 -- 172.21.15.33:0/2322120974 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0b9804aad0 con 0x55f9ea5b6370 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.523+0000 7f0ba72a2740 10 client.20073 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:14.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.523+0000 7f0ba72a2740 10 client.20073 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:42:14.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.523+0000 7f0ba72a2740 20 client.20073 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:14.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.523+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f9ea5c5f10 0x55f9ea5c8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.523+0000 7f0ba72a2740 4 client.20073 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:42:14.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.523+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2322120974 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9ea51e860 con 0x55f9ea5c5f10 2024-09-16T00:42:14.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.523+0000 7f0b9e59c640 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f9ea5c5f10 0x55f9ea5c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.523+0000 7f0b9e59c640 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f9ea5c5f10 0x55f9ea5c8390 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2943965403771de8 server_cookie=bb60ceba33c7fece in_seq=0 out_seq=0 2024-09-16T00:42:14.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.523+0000 7f0b85ffb640 10 client.20073 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:14.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0b85ffb640 1 -- 172.21.15.33:0/2322120974 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f9ea51e860 con 0x55f9ea5c5f10 2024-09-16T00:42:14.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0b85ffb640 10 client.20073 handle_command_reply: tid=0 2024-09-16T00:42:14.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0ba72a2740 1 client.20073 shutdown 2024-09-16T00:42:14.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2322120974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 msgr2=0x55f9ea5b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:14.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5b34d0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f0b980099f0 tx=0x7f0b980095c0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2322120974 shutdown_connections 2024-09-16T00:42:14.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f9ea5c5f10 0x55f9ea5c8390 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5b34d0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2322120974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2322120974 >> 172.21.15.33:0/2322120974 conn(0x55f9ea574310 msgr2=0x55f9ea5ad990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:14.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0b85ffb640 0 client.20073 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:14.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.524+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2322120974 shutdown_connections 2024-09-16T00:42:14.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.525+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2322120974 wait complete. 2024-09-16T00:42:14.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.525+0000 7f0ba72a2740 20 client.20073 trim_cache size 0 max 16384 2024-09-16T00:42:14.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.537+0000 7f0ba72a2740 1 Processor -- start 2024-09-16T00:42:14.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0ba72a2740 1 -- start start 2024-09-16T00:42:14.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0ba72a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5c22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0ba72a2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0ba72a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5b6370 0x55f9ea5cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0ba72a2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01b3ae0 con 0x55f9ea5acb40 2024-09-16T00:42:14.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0ba72a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01b13b0 con 0x55f9ea5b6370 2024-09-16T00:42:14.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0ba72a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01af200 con 0x55f9ea5ab5d0 2024-09-16T00:42:14.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b9dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b9dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42580/0 (socket says 172.21.15.33:42580) 2024-09-16T00:42:14.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b9dd9b640 1 -- 172.21.15.33:0/2628313263 learned_addr learned my addr 172.21.15.33:0/2628313263 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:14.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/2628313263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5b6370 0x55f9ea5cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b9e59c640 1 --2- 172.21.15.33:0/2628313263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5c22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b9dd9b640 1 -- 172.21.15.33:0/2628313263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 msgr2=0x55f9ea5c22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/2628313263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5c22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b9dd9b640 1 -- 172.21.15.33:0/2628313263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5b6370 msgr2=0x55f9ea5cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/2628313263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5b6370 0x55f9ea5cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b9dd9b640 1 -- 172.21.15.33:0/2628313263 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9ea5b8f50 con 0x55f9ea5acb40 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/2628313263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5b6370 0x55f9ea5cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/2628313263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b1f50 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f0b940a4000 tx=0x7f0b940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36033ed5461824eb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b87fff640 1 -- 172.21.15.33:0/2628313263 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b940860b0 con 0x55f9ea5acb40 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.538+0000 7f0b87fff640 1 -- 172.21.15.33:0/2628313263 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b9409bfc0 con 0x55f9ea5acb40 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.539+0000 7f0b87fff640 1 -- 172.21.15.33:0/2628313263 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b940cf350 con 0x55f9ea5acb40 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.539+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2628313263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5acb40 msgr2=0x55f9ea5b1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:14.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.539+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2628313263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b1f50 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f0b940a4000 tx=0x7f0b940046c0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.539+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2628313263 shutdown_connections 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.539+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2628313263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5b6370 0x55f9ea5cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.539+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2628313263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b1f50 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.539+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/2628313263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5c22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.539+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2628313263 >> 172.21.15.33:0/2628313263 conn(0x55f9ea574310 msgr2=0x55f9ea5b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.539+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2628313263 shutdown_connections 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.539+0000 7f0ba72a2740 1 -- 172.21.15.33:0/2628313263 wait complete. 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.539+0000 7f0ba72a2740 1 -- start start 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0ba72a2740 1 Processor -- start 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0ba72a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5be890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0ba72a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0ba72a2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5ba100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0ba72a2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01af870 con 0x55f9ea5b6370 2024-09-16T00:42:14.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0ba72a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01b0ce0 con 0x55f9ea5acb40 2024-09-16T00:42:14.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0ba72a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01aff40 con 0x55f9ea5ab5d0 2024-09-16T00:42:14.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0ba72a2740 10 client.0 _ll_register_callbacks cb 0x7f0b98031810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:14.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0b9ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5ba100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0b9e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5be890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0b9ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5ba100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42596/0 (socket says 172.21.15.33:42596) 2024-09-16T00:42:14.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0b9ed9d640 1 -- 172.21.15.33:0/3402718495 learned_addr learned my addr 172.21.15.33:0/3402718495 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:14.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0b9ed9d640 1 -- 172.21.15.33:0/3402718495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 msgr2=0x55f9ea5be890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:14.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5be890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0b9ed9d640 1 -- 172.21.15.33:0/3402718495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 msgr2=0x55f9ea5b9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:14.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0b9ed9d640 1 -- 172.21.15.33:0/3402718495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9ea5ba640 con 0x55f9ea5b6370 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0b9e59c640 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5be890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.540+0000 7f0b9dd9b640 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0b9ed9d640 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5ba100 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x55f9ea5a3690 tx=0x7f0b9803df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a79ce107bde7af server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0b85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0b85ffb640 1 -- 172.21.15.33:0/3402718495 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98066070 con 0x55f9ea5b6370 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0b85ffb640 1 -- 172.21.15.33:0/3402718495 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0b9802fda0 con 0x55f9ea5b6370 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0ba72a2740 1 -- 172.21.15.33:0/3402718495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b9409bfc0 con 0x55f9ea5b6370 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0b85ffb640 1 -- 172.21.15.33:0/3402718495 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0b98072050 con 0x55f9ea5b6370 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0ba72a2740 10 client.20079 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0ba72a2740 1 -- 172.21.15.33:0/3402718495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f9ea630920 con 0x55f9ea5b6370 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0b85ffb640 1 -- 172.21.15.33:0/3402718495 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b9409bfc0 con 0x55f9ea5b6370 2024-09-16T00:42:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0b85ffb640 1 -- 172.21.15.33:0/3402718495 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0b9804a550 con 0x55f9ea5b6370 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0ba72a2740 10 client.20079 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0ba72a2740 10 client.20079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0ba72a2740 20 client.20079 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f9ea631010 0x55f9ea633430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0ba72a2740 4 client.20079 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.541+0000 7f0ba72a2740 1 -- 172.21.15.33:0/3402718495 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f9ea518d70 con 0x55f9ea631010 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.542+0000 7f0b9e59c640 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f9ea631010 0x55f9ea633430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.542+0000 7f0b9e59c640 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f9ea631010 0x55f9ea633430 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fb3414c843cc06f server_cookie=e7ba39f2c77f976d in_seq=0 out_seq=0 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.542+0000 7f0b85ffb640 10 client.20079 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.542+0000 7f0b85ffb640 1 -- 172.21.15.33:0/3402718495 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f9ea518d70 con 0x55f9ea631010 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.542+0000 7f0b85ffb640 10 client.20079 handle_command_reply: tid=0 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.542+0000 7f0ba72a2740 1 client.20079 shutdown 2024-09-16T00:42:14.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba72a2740 1 -- 172.21.15.33:0/3402718495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 msgr2=0x55f9ea5ba100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:14.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5ba100 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x55f9ea5a3690 tx=0x7f0b9803df60 comp rx=0 tx=0).stop 2024-09-16T00:42:14.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba72a2740 1 -- 172.21.15.33:0/3402718495 shutdown_connections 2024-09-16T00:42:14.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55f9ea631010 0x55f9ea633430 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9ea5b6370 0x55f9ea5ba100 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f9ea5acb40 0x55f9ea5b9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba72a2740 1 --2- 172.21.15.33:0/3402718495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f9ea5ab5d0 0x55f9ea5be890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0b85ffb640 0 client.20079 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:14.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba72a2740 1 -- 172.21.15.33:0/3402718495 >> 172.21.15.33:0/3402718495 conn(0x55f9ea574310 msgr2=0x55f9ea5b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:14.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba72a2740 1 -- 172.21.15.33:0/3402718495 shutdown_connections 2024-09-16T00:42:14.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba72a2740 1 -- 172.21.15.33:0/3402718495 wait complete. 2024-09-16T00:42:14.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba72a2740 20 client.20079 trim_cache size 0 max 16384 2024-09-16T00:42:14.554 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:42:14.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/1273209734 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b880a3240 msgr2=0x7f0b880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:14.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba4ec9640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b880a3240 0x7f0b880a5700 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0b98006010 tx=0x7f0b98005ef0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/1273209734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ba01027a0 msgr2=0x7f0ba0074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:14.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba4ec9640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ba01027a0 0x7f0ba0074a30 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400d950 tx=0x7f0b9400de10 comp rx=0 tx=0).stop 2024-09-16T00:42:14.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/1273209734 shutdown_connections 2024-09-16T00:42:14.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba4ec9640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0b880a3240 0x7f0b880a5700 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba4ec9640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ba0106f20 0x7f0ba0074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba4ec9640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ba01027a0 0x7f0ba0074a30 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba4ec9640 1 --2- 172.21.15.33:0/1273209734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba0101df0 0x7f0ba00744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:14.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/1273209734 >> 172.21.15.33:0/1273209734 conn(0x7f0ba00fd8f0 msgr2=0x7f0ba00febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:14.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/1273209734 shutdown_connections 2024-09-16T00:42:14.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:14.543+0000 7f0ba4ec9640 1 -- 172.21.15.33:0/1273209734 wait complete. 2024-09-16T00:42:15.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.24 MB/sec execute 561 sec latency 21486.571 ms 2024-09-16T00:42:15.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:15 smithi033 ceph-mon[31243]: pgmap v1103: 129 pgs: 129 active+clean; 701 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.1 MiB/s wr, 52 op/s 2024-09-16T00:42:15.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:15 smithi060 ceph-mon[35185]: pgmap v1103: 129 pgs: 129 active+clean; 701 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.1 MiB/s wr, 52 op/s 2024-09-16T00:42:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:15 smithi083 ceph-mon[35242]: pgmap v1103: 129 pgs: 129 active+clean; 701 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.1 MiB/s wr, 52 op/s 2024-09-16T00:42:16.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.21 MB/sec execute 562 sec latency 22486.696 ms 2024-09-16T00:42:16.342 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-16T00:42:16.343 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-16T00:42:16.612 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:17.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.18 MB/sec execute 563 sec latency 23486.758 ms 2024-09-16T00:42:17.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.092+0000 7f2fb4e70640 1 -- 172.21.15.33:0/3146897108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fb0102d40 msgr2=0x7f2fb0103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:17.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.092+0000 7f2fb4e70640 1 --2- 172.21.15.33:0/3146897108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fb0102d40 0x7f2fb0103120 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0009900 tx=0x7f2fa002a480 comp rx=0 tx=0).stop 2024-09-16T00:42:17.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.092+0000 7f2fb4e70640 1 -- 172.21.15.33:0/3146897108 shutdown_connections 2024-09-16T00:42:17.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.092+0000 7f2fb4e70640 1 --2- 172.21.15.33:0/3146897108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb010c170 0x7f2fb010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.092+0000 7f2fb4e70640 1 --2- 172.21.15.33:0/3146897108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fb0103660 0x7f2fb010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.092+0000 7f2fb4e70640 1 --2- 172.21.15.33:0/3146897108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fb0102d40 0x7f2fb0103120 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.092+0000 7f2fb4e70640 1 -- 172.21.15.33:0/3146897108 >> 172.21.15.33:0/3146897108 conn(0x7f2fb00fc7a0 msgr2=0x7f2fb00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.092+0000 7f2fb4e70640 1 -- 172.21.15.33:0/3146897108 shutdown_connections 2024-09-16T00:42:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.092+0000 7f2fb4e70640 1 -- 172.21.15.33:0/3146897108 wait complete. 2024-09-16T00:42:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.093+0000 7f2fb4e70640 1 Processor -- start 2024-09-16T00:42:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.093+0000 7f2fb4e70640 1 -- start start 2024-09-16T00:42:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fb4e70640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fb0102d40 0x7f2fb01984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fb4e70640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb0103660 0x7f2fb0198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fb4e70640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fb010c170 0x7f2fb019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fb4e70640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb0111f70 con 0x7f2fb0102d40 2024-09-16T00:42:17.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fb4e70640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb0111df0 con 0x7f2fb010c170 2024-09-16T00:42:17.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fb4e70640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01120f0 con 0x7f2fb0103660 2024-09-16T00:42:17.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fae59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fb0102d40 0x7f2fb01984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fae59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fb0102d40 0x7f2fb01984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42610/0 (socket says 172.21.15.33:42610) 2024-09-16T00:42:17.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fae59c640 1 -- 172.21.15.33:0/1966495917 learned_addr learned my addr 172.21.15.33:0/1966495917 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:17.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2faed9d640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fb010c170 0x7f2fb019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fadd9b640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb0103660 0x7f2fb0198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fae59c640 1 -- 172.21.15.33:0/1966495917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb0103660 msgr2=0x7f2fb0198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:17.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fae59c640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb0103660 0x7f2fb0198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fae59c640 1 -- 172.21.15.33:0/1966495917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fb010c170 msgr2=0x7f2fb019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:17.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fae59c640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fb010c170 0x7f2fb019cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fae59c640 1 -- 172.21.15.33:0/1966495917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb019d480 con 0x7f2fb0102d40 2024-09-16T00:42:17.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fadd9b640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb0103660 0x7f2fb0198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:17.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.094+0000 7f2fae59c640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fb0102d40 0x7f2fb01984d0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0006010 tx=0x7f2fa00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:17.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.095+0000 7f2f9f7fe640 1 -- 172.21.15.33:0/1966495917 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa003f030 con 0x7f2fb0102d40 2024-09-16T00:42:17.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.095+0000 7f2f9f7fe640 1 -- 172.21.15.33:0/1966495917 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fa0003ce0 con 0x7f2fb0102d40 2024-09-16T00:42:17.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.095+0000 7f2f9f7fe640 1 -- 172.21.15.33:0/1966495917 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0035070 con 0x7f2fb0102d40 2024-09-16T00:42:17.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.095+0000 7f2fb4e70640 1 -- 172.21.15.33:0/1966495917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb019d710 con 0x7f2fb0102d40 2024-09-16T00:42:17.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.095+0000 7f2fb4e70640 1 -- 172.21.15.33:0/1966495917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb019dba0 con 0x7f2fb0102d40 2024-09-16T00:42:17.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.097+0000 7f2f9f7fe640 1 -- 172.21.15.33:0/1966495917 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2fa0042030 con 0x7f2fb0102d40 2024-09-16T00:42:17.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.097+0000 7f2fb7249740 1 Processor -- start 2024-09-16T00:42:17.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fb7249740 1 -- start start 2024-09-16T00:42:17.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fb7249740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac4a40 0x5638c4ac4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2f9f7fe640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2f980a32c0 0x7f2f980a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fb7249740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 0x5638c4ad3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fb7249740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ad4150 0x5638c4ad6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fb7249740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5638c4a2a280 con 0x5638c4ac5360 2024-09-16T00:42:17.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fb7249740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5638c49d1b10 con 0x5638c4ad4150 2024-09-16T00:42:17.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fb7249740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5638c4a26240 con 0x5638c4ac4a40 2024-09-16T00:42:17.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fadd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 0x5638c4ad3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fae59c640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2f980a32c0 0x7f2f980a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fadd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 0x5638c4ad3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42616/0 (socket says 172.21.15.33:42616) 2024-09-16T00:42:17.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fadd9b640 1 -- 172.21.15.33:0/2488916560 learned_addr learned my addr 172.21.15.33:0/2488916560 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:17.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2f9f7fe640 1 -- 172.21.15.33:0/1966495917 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2fa00f03a0 con 0x7f2fb0102d40 2024-09-16T00:42:17.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2faed9d640 1 --2- 172.21.15.33:0/2488916560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ad4150 0x5638c4ad6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2faed9d640 1 --2- 172.21.15.33:0/2488916560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac4a40 0x5638c4ac4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fadd9b640 1 -- 172.21.15.33:0/2488916560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac4a40 msgr2=0x5638c4ac4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:17.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fadd9b640 1 --2- 172.21.15.33:0/2488916560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac4a40 0x5638c4ac4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fadd9b640 1 -- 172.21.15.33:0/2488916560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ad4150 msgr2=0x5638c4ad6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:17.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fadd9b640 1 --2- 172.21.15.33:0/2488916560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ad4150 0x5638c4ad6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2fadd9b640 1 -- 172.21.15.33:0/2488916560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638c49de1f0 con 0x5638c4ac5360 2024-09-16T00:42:17.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.098+0000 7f2faed9d640 1 --2- 172.21.15.33:0/2488916560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac4a40 0x5638c4ac4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:42:17.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.099+0000 7f2fae59c640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2f980a32c0 0x7f2f980a5780 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0000c00 tx=0x7f2fa0049bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:17.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.099+0000 7f2fadd9b640 1 --2- 172.21.15.33:0/2488916560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 0x5638c4ad3c10 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400bcc0 tx=0x7f2fa400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fab29869dbb2cb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:17.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.099+0000 7f2f9cff9640 1 -- 172.21.15.33:0/2488916560 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4011440 con 0x5638c4ac5360 2024-09-16T00:42:17.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.099+0000 7f2f9cff9640 1 -- 172.21.15.33:0/2488916560 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fa4011ac0 con 0x5638c4ac5360 2024-09-16T00:42:17.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.099+0000 7f2f9cff9640 1 -- 172.21.15.33:0/2488916560 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa401bb80 con 0x5638c4ac5360 2024-09-16T00:42:17.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.100+0000 7f2fb7249740 1 -- 172.21.15.33:0/2488916560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 msgr2=0x5638c4ad3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:17.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.100+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2488916560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 0x5638c4ad3c10 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400bcc0 tx=0x7f2fa400f730 comp rx=0 tx=0).stop 2024-09-16T00:42:17.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.100+0000 7f2fb7249740 1 -- 172.21.15.33:0/2488916560 shutdown_connections 2024-09-16T00:42:17.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.100+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2488916560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ad4150 0x5638c4ad6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.100+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2488916560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 0x5638c4ad3c10 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.100+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2488916560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac4a40 0x5638c4ac4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.100+0000 7f2fb7249740 1 -- 172.21.15.33:0/2488916560 >> 172.21.15.33:0/2488916560 conn(0x5638c4ac10f0 msgr2=0x5638c4ac14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:17.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.100+0000 7f2fb7249740 1 -- 172.21.15.33:0/2488916560 shutdown_connections 2024-09-16T00:42:17.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.100+0000 7f2fb7249740 1 -- 172.21.15.33:0/2488916560 wait complete. 2024-09-16T00:42:17.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.100+0000 7f2fb7249740 1 -- start start 2024-09-16T00:42:17.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.100+0000 7f2fb7249740 1 Processor -- start 2024-09-16T00:42:17.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2fb7249740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac4a40 0x5638c4ad10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2fb7249740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac5360 0x5638c4ad15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2fb7249740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 0x5638c4ad1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2fb7249740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5638c4ad6d20 con 0x5638c4ad4150 2024-09-16T00:42:17.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2fb7249740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5638c49e4f10 con 0x5638c4ac5360 2024-09-16T00:42:17.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2fb7249740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5638c4ad6ea0 con 0x5638c4ac4a40 2024-09-16T00:42:17.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2fb7249740 10 client.0 _ll_register_callbacks cb 0x5638c49ada40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:17.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2faed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 0x5638c4ad1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2faed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 0x5638c4ad1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42628/0 (socket says 172.21.15.33:42628) 2024-09-16T00:42:17.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2faed9d640 1 -- 172.21.15.33:0/1495624449 learned_addr learned my addr 172.21.15.33:0/1495624449 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:17.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2fadd9b640 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac5360 0x5638c4ad15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2faed9d640 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac4a40 0x5638c4ad10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2faed9d640 1 -- 172.21.15.33:0/1495624449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac4a40 msgr2=0x5638c4ad10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:17.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2faed9d640 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac4a40 0x5638c4ad10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2faed9d640 1 -- 172.21.15.33:0/1495624449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac5360 msgr2=0x5638c4ad15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:17.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2faed9d640 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac5360 0x5638c4ad15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2faed9d640 1 -- 172.21.15.33:0/1495624449 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638c479b6a0 con 0x5638c4ad4150 2024-09-16T00:42:17.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2faed9d640 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 0x5638c4ad1b30 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8011440 tx=0x7f2fa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a3b7ff8a7e467f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:17.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2f95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:17.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2f95ffb640 1 -- 172.21.15.33:0/1495624449 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa80142d0 con 0x5638c4ad4150 2024-09-16T00:42:17.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2f95ffb640 1 -- 172.21.15.33:0/1495624449 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2fa800f630 con 0x5638c4ad4150 2024-09-16T00:42:17.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.101+0000 7f2fb7249740 1 -- 172.21.15.33:0/1495624449 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fa4011ac0 con 0x5638c4ad4150 2024-09-16T00:42:17.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.102+0000 7f2f95ffb640 1 -- 172.21.15.33:0/1495624449 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2fa8004b40 con 0x5638c4ad4150 2024-09-16T00:42:17.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.102+0000 7f2fb7249740 10 client.20091 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:17.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.102+0000 7f2fb7249740 1 -- 172.21.15.33:0/1495624449 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5638c4ae2990 con 0x5638c4ad4150 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.102+0000 7f2f95ffb640 1 -- 172.21.15.33:0/1495624449 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fa4011ac0 con 0x5638c4ad4150 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.102+0000 7f2f95ffb640 1 -- 172.21.15.33:0/1495624449 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2fa80051e0 con 0x5638c4ad4150 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.102+0000 7f2fb7249740 10 client.20091 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.102+0000 7f2fb7249740 10 client.20091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.102+0000 7f2fb7249740 20 client.20091 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.102+0000 7f2fb7249740 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638c4ae3da0 0x5638c4ae61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.102+0000 7f2fb7249740 4 client.20091 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.102+0000 7f2fb7249740 1 -- 172.21.15.33:0/1495624449 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638c4a377f0 con 0x5638c4ae3da0 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.102+0000 7f2fadd9b640 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638c4ae3da0 0x5638c4ae61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.103+0000 7f2fadd9b640 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638c4ae3da0 0x5638c4ae61f0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7bb64e480cfe7cc server_cookie=eb2df3d5ff12c13b in_seq=0 out_seq=0 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.103+0000 7f2f95ffb640 10 client.20091 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2f95ffb640 1 -- 172.21.15.33:0/1495624449 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5638c4a377f0 con 0x5638c4ae3da0 2024-09-16T00:42:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2f95ffb640 10 client.20091 handle_command_reply: tid=0 2024-09-16T00:42:17.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2fb7249740 1 client.20091 shutdown 2024-09-16T00:42:17.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2fb7249740 1 -- 172.21.15.33:0/1495624449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 msgr2=0x5638c4ad1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:17.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2fb7249740 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 0x5638c4ad1b30 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8011440 tx=0x7f2fa8004a40 comp rx=0 tx=0).stop 2024-09-16T00:42:17.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2fb7249740 1 -- 172.21.15.33:0/1495624449 shutdown_connections 2024-09-16T00:42:17.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2fb7249740 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638c4ae3da0 0x5638c4ae61f0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2fb7249740 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 0x5638c4ad1b30 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2fb7249740 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac5360 0x5638c4ad15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2fb7249740 1 --2- 172.21.15.33:0/1495624449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac4a40 0x5638c4ad10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2f95ffb640 0 client.20091 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:17.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2fb7249740 1 -- 172.21.15.33:0/1495624449 >> 172.21.15.33:0/1495624449 conn(0x5638c4ac10f0 msgr2=0x5638c4ac6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:17.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2fb7249740 1 -- 172.21.15.33:0/1495624449 shutdown_connections 2024-09-16T00:42:17.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2fb7249740 1 -- 172.21.15.33:0/1495624449 wait complete. 2024-09-16T00:42:17.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.104+0000 7f2fb7249740 20 client.20091 trim_cache size 0 max 16384 2024-09-16T00:42:17.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.121+0000 7f2fb7249740 1 Processor -- start 2024-09-16T00:42:17.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.121+0000 7f2fb7249740 1 -- start start 2024-09-16T00:42:17.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2fb7249740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac4a40 0x5638c4aebac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2fb7249740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac5360 0x5638c4aec000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2fb7249740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 0x5638c4abfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2fb7249740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01af810 con 0x5638c4ad4150 2024-09-16T00:42:17.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2fb7249740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01ad0e0 con 0x5638c4ac4a40 2024-09-16T00:42:17.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2fb7249740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01aaf30 con 0x5638c4ac5360 2024-09-16T00:42:17.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2faed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 0x5638c4abfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2faed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 0x5638c4abfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42636/0 (socket says 172.21.15.33:42636) 2024-09-16T00:42:17.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2faed9d640 1 -- 172.21.15.33:0/2439073518 learned_addr learned my addr 172.21.15.33:0/2439073518 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:17.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2faed9d640 1 -- 172.21.15.33:0/2439073518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac5360 msgr2=0x5638c4aec000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:17.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2faed9d640 1 --2- 172.21.15.33:0/2439073518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac5360 0x5638c4aec000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2faed9d640 1 -- 172.21.15.33:0/2439073518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac4a40 msgr2=0x5638c4aebac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:17.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2faed9d640 1 --2- 172.21.15.33:0/2439073518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac4a40 0x5638c4aebac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2faed9d640 1 -- 172.21.15.33:0/2439073518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638c4ac7560 con 0x5638c4ad4150 2024-09-16T00:42:17.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2faed9d640 1 --2- 172.21.15.33:0/2439073518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 0x5638c4abfa50 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8014fd0 tx=0x7f2fa800a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f354d5193f95719b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:17.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2f97fff640 1 -- 172.21.15.33:0/2439073518 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8011d90 con 0x5638c4ad4150 2024-09-16T00:42:17.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2f97fff640 1 -- 172.21.15.33:0/2439073518 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fa8004ce0 con 0x5638c4ad4150 2024-09-16T00:42:17.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.122+0000 7f2f97fff640 1 -- 172.21.15.33:0/2439073518 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa80039b0 con 0x5638c4ad4150 2024-09-16T00:42:17.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.123+0000 7f2fb7249740 1 -- 172.21.15.33:0/2439073518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 msgr2=0x5638c4abfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:17.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.123+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2439073518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 0x5638c4abfa50 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8014fd0 tx=0x7f2fa800a550 comp rx=0 tx=0).stop 2024-09-16T00:42:17.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.123+0000 7f2fb7249740 1 -- 172.21.15.33:0/2439073518 shutdown_connections 2024-09-16T00:42:17.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.123+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2439073518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ad4150 0x5638c4abfa50 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.123+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2439073518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ac5360 0x5638c4aec000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.123+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2439073518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac4a40 0x5638c4aebac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.123+0000 7f2fb7249740 1 -- 172.21.15.33:0/2439073518 >> 172.21.15.33:0/2439073518 conn(0x5638c4ac10f0 msgr2=0x5638c4ad01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:17.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.123+0000 7f2fb7249740 1 -- 172.21.15.33:0/2439073518 shutdown_connections 2024-09-16T00:42:17.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.123+0000 7f2fb7249740 1 -- 172.21.15.33:0/2439073518 wait complete. 2024-09-16T00:42:17.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.123+0000 7f2fb7249740 1 -- start start 2024-09-16T00:42:17.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.123+0000 7f2fb7249740 1 Processor -- start 2024-09-16T00:42:17.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fb7249740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac4a40 0x5638c4ad8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fb7249740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 0x5638c4ad8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fb7249740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ad4150 0x5638c4ad8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fb7249740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01ab5a0 con 0x5638c4ac5360 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fb7249740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01aca10 con 0x5638c4ac4a40 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fb7249740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01abc70 con 0x5638c4ad4150 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fb7249740 10 client.0 _ll_register_callbacks cb 0x7f2fa800a9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fadd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 0x5638c4ad8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fadd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 0x5638c4ad8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42644/0 (socket says 172.21.15.33:42644) 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fadd9b640 1 -- 172.21.15.33:0/2230230995 learned_addr learned my addr 172.21.15.33:0/2230230995 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2faed9d640 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac4a40 0x5638c4ad8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2faed9d640 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ad4150 0x5638c4ad8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fadd9b640 1 -- 172.21.15.33:0/2230230995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ad4150 msgr2=0x5638c4ad8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fadd9b640 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ad4150 0x5638c4ad8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fadd9b640 1 -- 172.21.15.33:0/2230230995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac4a40 msgr2=0x5638c4ad8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fadd9b640 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac4a40 0x5638c4ad8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fadd9b640 1 -- 172.21.15.33:0/2230230995 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638c4b4dbf0 con 0x5638c4ac5360 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2faed9d640 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ad4150 0x5638c4ad8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2fadd9b640 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 0x5638c4ad8560 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x5638c4ac7750 tx=0x7f2fa4005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466aaa14efdfbb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.124+0000 7f2f95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2f95ffb640 1 -- 172.21.15.33:0/2230230995 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa402b020 con 0x5638c4ac5360 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2f95ffb640 1 -- 172.21.15.33:0/2230230995 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f2fa4004240 con 0x5638c4ac5360 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2fb7249740 1 -- 172.21.15.33:0/2230230995 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fa8004ce0 con 0x5638c4ac5360 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2f95ffb640 1 -- 172.21.15.33:0/2230230995 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2fa4033050 con 0x5638c4ac5360 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2fb7249740 10 client.20097 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2fb7249740 1 -- 172.21.15.33:0/2230230995 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5638c4b4eef0 con 0x5638c4ac5360 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2f95ffb640 1 -- 172.21.15.33:0/2230230995 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fa8004ce0 con 0x5638c4ac5360 2024-09-16T00:42:17.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2f95ffb640 1 -- 172.21.15.33:0/2230230995 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2fa401bb50 con 0x5638c4ac5360 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2fb7249740 10 client.20097 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2fb7249740 10 client.20097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2fb7249740 20 client.20097 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638c4b4f390 0x5638c4b51780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2fb7249740 4 client.20097 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.125+0000 7f2fb7249740 1 -- 172.21.15.33:0/2230230995 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5638c4a31d00 con 0x5638c4b4f390 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.126+0000 7f2faed9d640 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638c4b4f390 0x5638c4b51780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.126+0000 7f2f95ffb640 10 client.20097 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.126+0000 7f2faed9d640 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638c4b4f390 0x5638c4b51780 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d94f3b545c5b98e server_cookie=7ef70042988a26d9 in_seq=0 out_seq=0 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2f95ffb640 1 -- 172.21.15.33:0/2230230995 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5638c4a31d00 con 0x5638c4b4f390 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2f95ffb640 10 client.20097 handle_command_reply: tid=0 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2fb7249740 1 client.20097 shutdown 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2fb7249740 1 -- 172.21.15.33:0/2230230995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 msgr2=0x5638c4ad8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:17.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 0x5638c4ad8560 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x5638c4ac7750 tx=0x7f2fa4005fb0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2fb7249740 1 -- 172.21.15.33:0/2230230995 shutdown_connections 2024-09-16T00:42:17.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5638c4b4f390 0x5638c4b51780 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5638c4ad4150 0x5638c4ad8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638c4ac5360 0x5638c4ad8560 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2fb7249740 1 --2- 172.21.15.33:0/2230230995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5638c4ac4a40 0x5638c4ad8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2f95ffb640 0 client.20097 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:17.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2fb7249740 1 -- 172.21.15.33:0/2230230995 >> 172.21.15.33:0/2230230995 conn(0x5638c4ac10f0 msgr2=0x5638c4ad1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:17.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2fb7249740 1 -- 172.21.15.33:0/2230230995 shutdown_connections 2024-09-16T00:42:17.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2fb7249740 1 -- 172.21.15.33:0/2230230995 wait complete. 2024-09-16T00:42:17.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.127+0000 7f2fb7249740 20 client.20097 trim_cache size 0 max 16384 2024-09-16T00:42:17.148 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:42:17.148 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-09-16T00:42:17.148 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "5e969d48-5420-4759-8c78-d6b4221fdc7c", 2024-09-16T00:42:17.148 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-09-16T00:42:17.148 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:42:17.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.128+0000 7f2fb4e70640 1 -- 172.21.15.33:0/1966495917 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2f980a32c0 msgr2=0x7f2f980a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:17.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.128+0000 7f2fb4e70640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2f980a32c0 0x7f2f980a5780 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0000c00 tx=0x7f2fa0049bd0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.128+0000 7f2fb4e70640 1 -- 172.21.15.33:0/1966495917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fb0102d40 msgr2=0x7f2fb01984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:17.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.128+0000 7f2fb4e70640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fb0102d40 0x7f2fb01984d0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0006010 tx=0x7f2fa00042f0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.128+0000 7f2fb4e70640 1 -- 172.21.15.33:0/1966495917 shutdown_connections 2024-09-16T00:42:17.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.128+0000 7f2fb4e70640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f2f980a32c0 0x7f2f980a5780 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.128+0000 7f2fb4e70640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fb010c170 0x7f2fb019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.128+0000 7f2fb4e70640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb0103660 0x7f2fb0198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.128+0000 7f2fb4e70640 1 --2- 172.21.15.33:0/1966495917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fb0102d40 0x7f2fb01984d0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:17.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.128+0000 7f2fb4e70640 1 -- 172.21.15.33:0/1966495917 >> 172.21.15.33:0/1966495917 conn(0x7f2fb00fc7a0 msgr2=0x7f2fb00fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:17.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.128+0000 7f2fb4e70640 1 -- 172.21.15.33:0/1966495917 shutdown_connections 2024-09-16T00:42:17.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:17.128+0000 7f2fb4e70640 1 -- 172.21.15.33:0/1966495917 wait complete. 2024-09-16T00:42:17.438 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:42:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:17 smithi060 ceph-mon[35185]: pgmap v1104: 129 pgs: 129 active+clean; 698 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 2.1 MiB/s wr, 51 op/s 2024-09-16T00:42:17.748 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:17 smithi033 ceph-mon[31243]: pgmap v1104: 129 pgs: 129 active+clean; 698 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 2.1 MiB/s wr, 51 op/s 2024-09-16T00:42:17.839 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:42:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:17 smithi083 ceph-mon[35242]: pgmap v1104: 129 pgs: 129 active+clean; 698 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 2.1 MiB/s wr, 51 op/s 2024-09-16T00:42:17.931 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:18.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.15 MB/sec execute 564 sec latency 24486.791 ms 2024-09-16T00:42:18.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.483+0000 7f104f066640 1 -- 172.21.15.33:0/1868333815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1048100a90 msgr2=0x7f1048100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:18.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.483+0000 7f104f066640 1 --2- 172.21.15.33:0/1868333815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1048100a90 0x7f1048100ef0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f1034009930 tx=0x7f103402a420 comp rx=0 tx=0).stop 2024-09-16T00:42:18.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.483+0000 7f104f066640 1 -- 172.21.15.33:0/1868333815 shutdown_connections 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.483+0000 7f104f066640 1 --2- 172.21.15.33:0/1868333815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1048101430 0x7f104810d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.483+0000 7f104f066640 1 --2- 172.21.15.33:0/1868333815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1048100a90 0x7f1048100ef0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.483+0000 7f104f066640 1 --2- 172.21.15.33:0/1868333815 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1048106a90 0x7f1048106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.483+0000 7f104f066640 1 -- 172.21.15.33:0/1868333815 >> 172.21.15.33:0/1868333815 conn(0x7f10480fc7d0 msgr2=0x7f10480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.483+0000 7f104f066640 1 -- 172.21.15.33:0/1868333815 shutdown_connections 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.483+0000 7f104f066640 1 -- 172.21.15.33:0/1868333815 wait complete. 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104f066640 1 Processor -- start 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104f066640 1 -- start start 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104f066640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1048100a90 0x7f10481992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104f066640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1048101430 0x7f1048199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104f066640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1048106a90 0x7f10481a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104f066640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1048110e90 con 0x7f1048100a90 2024-09-16T00:42:18.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104f066640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1048110d10 con 0x7f1048101430 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104f066640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1048111010 con 0x7f1048106a90 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104ce02640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1048100a90 0x7f10481992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104ce02640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1048100a90 0x7f10481992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42660/0 (socket says 172.21.15.33:42660) 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104ce02640 1 -- 172.21.15.33:0/4164597992 learned_addr learned my addr 172.21.15.33:0/4164597992 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f1047fff640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1048101430 0x7f1048199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104d603640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1048106a90 0x7f10481a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104ce02640 1 -- 172.21.15.33:0/4164597992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1048106a90 msgr2=0x7f10481a4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104ce02640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1048106a90 0x7f10481a4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104ce02640 1 -- 172.21.15.33:0/4164597992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1048101430 msgr2=0x7f1048199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104ce02640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1048101430 0x7f1048199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104ce02640 1 -- 172.21.15.33:0/4164597992 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10481a48a0 con 0x7f1048100a90 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f104d603640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1048106a90 0x7f10481a4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.484+0000 7f1047fff640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1048101430 0x7f1048199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:18.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.485+0000 7f104ce02640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1048100a90 0x7f10481992e0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f103c00c970 tx=0x7f103c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.485+0000 7f1045ffb640 1 -- 172.21.15.33:0/4164597992 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c007bb0 con 0x7f1048100a90 2024-09-16T00:42:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.485+0000 7f1045ffb640 1 -- 172.21.15.33:0/4164597992 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f103c007d50 con 0x7f1048100a90 2024-09-16T00:42:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.485+0000 7f104f066640 1 -- 172.21.15.33:0/4164597992 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10481a4a70 con 0x7f1048100a90 2024-09-16T00:42:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.485+0000 7f1045ffb640 1 -- 172.21.15.33:0/4164597992 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c00f450 con 0x7f1048100a90 2024-09-16T00:42:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.485+0000 7f104f066640 1 -- 172.21.15.33:0/4164597992 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10481a4f30 con 0x7f1048100a90 2024-09-16T00:42:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.486+0000 7f104f066640 1 -- 172.21.15.33:0/4164597992 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1048100ef0 con 0x7f1048100a90 2024-09-16T00:42:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.487+0000 7f1045ffb640 1 -- 172.21.15.33:0/4164597992 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f103c005020 con 0x7f1048100a90 2024-09-16T00:42:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.488+0000 7f1045ffb640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10380a32c0 0x7f10380a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.488+0000 7f1045ffb640 1 -- 172.21.15.33:0/4164597992 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f103c0cfb90 con 0x7f1048100a90 2024-09-16T00:42:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.488+0000 7f1047fff640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10380a32c0 0x7f10380a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.488+0000 7f1047fff640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10380a32c0 0x7f10380a5780 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1034000c00 tx=0x7f1034005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.492+0000 7f1045ffb640 1 -- 172.21.15.33:0/4164597992 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f103c014030 con 0x7f1048100a90 2024-09-16T00:42:18.536 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:18.662 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub queued for path: / 2024-09-16T00:42:18.662 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.662 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:42:18.662 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:42:18.662 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:42:18.662 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.662 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:42:18.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:42:18.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:42:18.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:42:18.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: active paths [/] 2024-09-16T00:42:18.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:18 smithi033 ceph-mon[31243]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.713+0000 7f104f066640 1 -- 172.21.15.33:0/4164597992 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1048109020 con 0x7f1048100a90 2024-09-16T00:42:18.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.714+0000 7f1045ffb640 1 -- 172.21.15.33:0/4164597992 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f103c09c3d0 con 0x7f1048100a90 2024-09-16T00:42:18.715 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:42:18.715 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:42:18.716 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:42:18.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.716+0000 7f104f066640 1 -- 172.21.15.33:0/4164597992 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10380a32c0 msgr2=0x7f10380a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:18.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.716+0000 7f104f066640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10380a32c0 0x7f10380a5780 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1034000c00 tx=0x7f1034005a20 comp rx=0 tx=0).stop 2024-09-16T00:42:18.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.716+0000 7f104f066640 1 -- 172.21.15.33:0/4164597992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1048100a90 msgr2=0x7f10481992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:18.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.716+0000 7f104f066640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1048100a90 0x7f10481992e0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f103c00c970 tx=0x7f103c00ce30 comp rx=0 tx=0).stop 2024-09-16T00:42:18.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.716+0000 7f104f066640 1 -- 172.21.15.33:0/4164597992 shutdown_connections 2024-09-16T00:42:18.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.716+0000 7f104f066640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f10380a32c0 0x7f10380a5780 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:18.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.716+0000 7f104f066640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1048106a90 0x7f10481a4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:18.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.716+0000 7f104f066640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1048101430 0x7f1048199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:18.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.716+0000 7f104f066640 1 --2- 172.21.15.33:0/4164597992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1048100a90 0x7f10481992e0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:18.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.716+0000 7f104f066640 1 -- 172.21.15.33:0/4164597992 >> 172.21.15.33:0/4164597992 conn(0x7f10480fc7d0 msgr2=0x7f10480fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:18.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.716+0000 7f104f066640 1 -- 172.21.15.33:0/4164597992 shutdown_connections 2024-09-16T00:42:18.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:18.716+0000 7f104f066640 1 -- 172.21.15.33:0/4164597992 wait complete. 2024-09-16T00:42:18.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub queued for path: / 2024-09-16T00:42:18.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:42:18.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:42:18.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:42:18.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:42:18.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:42:18.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:42:18.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:42:18.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.731 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: active paths [/] 2024-09-16T00:42:18.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.732 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:18 smithi060 ceph-mon[35185]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub queued for path: / 2024-09-16T00:42:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:42:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:42:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:42:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:42:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:42:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:42:18.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:42:18.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: active paths [/] 2024-09-16T00:42:18.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:18.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:18 smithi083 ceph-mon[35242]: scrub summary: idle+waiting paths [/] 2024-09-16T00:42:19.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.12 MB/sec execute 565 sec latency 25486.880 ms 2024-09-16T00:42:19.576 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --set-id 78738c5c --release --await' 2024-09-16T00:42:19.576 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --set-id 78738c5c --release --await 2024-09-16T00:42:19.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:19 smithi060 ceph-mon[35185]: pgmap v1105: 129 pgs: 129 active+clean; 696 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 2.1 MiB/s wr, 58 op/s 2024-09-16T00:42:19.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:19 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4164597992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:42:19.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:19 smithi033 ceph-mon[31243]: pgmap v1105: 129 pgs: 129 active+clean; 696 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 2.1 MiB/s wr, 58 op/s 2024-09-16T00:42:19.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:19 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4164597992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:42:19.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.771+0000 7f1d34d4e640 1 -- 172.21.15.33:0/2242071902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d30102d90 msgr2=0x7f1d30103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:19.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.771+0000 7f1d34d4e640 1 --2- 172.21.15.33:0/2242071902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d30102d90 0x7f1d30103170 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009cd0 tx=0x7f1d2002a8e0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.771+0000 7f1d34d4e640 1 -- 172.21.15.33:0/2242071902 shutdown_connections 2024-09-16T00:42:19.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.771+0000 7f1d34d4e640 1 --2- 172.21.15.33:0/2242071902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d30108290 0x7f1d3010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.771+0000 7f1d34d4e640 1 --2- 172.21.15.33:0/2242071902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d30103740 0x7f1d30107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.771+0000 7f1d34d4e640 1 --2- 172.21.15.33:0/2242071902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d30102d90 0x7f1d30103170 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.771+0000 7f1d34d4e640 1 -- 172.21.15.33:0/2242071902 >> 172.21.15.33:0/2242071902 conn(0x7f1d300fc7d0 msgr2=0x7f1d300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:19.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.772+0000 7f1d34d4e640 1 -- 172.21.15.33:0/2242071902 shutdown_connections 2024-09-16T00:42:19.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.772+0000 7f1d34d4e640 1 -- 172.21.15.33:0/2242071902 wait complete. 2024-09-16T00:42:19.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.772+0000 7f1d34d4e640 1 Processor -- start 2024-09-16T00:42:19.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.772+0000 7f1d34d4e640 1 -- start start 2024-09-16T00:42:19.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d34d4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d30102d90 0x7f1d301985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d34d4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d30103740 0x7f1d30198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d34d4e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d30108290 0x7f1d3019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d34d4e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30112090 con 0x7f1d30102d90 2024-09-16T00:42:19.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d34d4e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30111f10 con 0x7f1d30108290 2024-09-16T00:42:19.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d34d4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30112210 con 0x7f1d30103740 2024-09-16T00:42:19.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d2e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d30102d90 0x7f1d301985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d2e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d30102d90 0x7f1d301985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42678/0 (socket says 172.21.15.33:42678) 2024-09-16T00:42:19.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d2e59c640 1 -- 172.21.15.33:0/1301168317 learned_addr learned my addr 172.21.15.33:0/1301168317 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:19.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d30108290 0x7f1d3019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d2dd9b640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d30103740 0x7f1d30198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d2e59c640 1 -- 172.21.15.33:0/1301168317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d30103740 msgr2=0x7f1d30198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:19.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d2e59c640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d30103740 0x7f1d30198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d2e59c640 1 -- 172.21.15.33:0/1301168317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d30108290 msgr2=0x7f1d3019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:19.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d2e59c640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d30108290 0x7f1d3019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.773+0000 7f1d2e59c640 1 -- 172.21.15.33:0/1301168317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3019d580 con 0x7f1d30102d90 2024-09-16T00:42:19.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.774+0000 7f1d2e59c640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d30102d90 0x7f1d301985d0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f1d2002ae00 tx=0x7f1d20003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:19.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.774+0000 7f1d277fe640 1 -- 172.21.15.33:0/1301168317 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20004040 con 0x7f1d30102d90 2024-09-16T00:42:19.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.774+0000 7f1d34d4e640 1 -- 172.21.15.33:0/1301168317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d3019d810 con 0x7f1d30102d90 2024-09-16T00:42:19.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.774+0000 7f1d277fe640 1 -- 172.21.15.33:0/1301168317 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d200046c0 con 0x7f1d30102d90 2024-09-16T00:42:19.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.774+0000 7f1d277fe640 1 -- 172.21.15.33:0/1301168317 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20046480 con 0x7f1d30102d90 2024-09-16T00:42:19.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.774+0000 7f1d34d4e640 1 -- 172.21.15.33:0/1301168317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d301a5150 con 0x7f1d30102d90 2024-09-16T00:42:19.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.776+0000 7f1d277fe640 1 -- 172.21.15.33:0/1301168317 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1d200305a0 con 0x7f1d30102d90 2024-09-16T00:42:19.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d37127740 1 Processor -- start 2024-09-16T00:42:19.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d37127740 1 -- start start 2024-09-16T00:42:19.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d37127740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857e360 0x56259857e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d277fe640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1d1c0a32c0 0x7f1d1c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d37127740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857ec80 0x56259858d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d37127740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x56259858fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d37127740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5625984e42e0 con 0x56259858da90 2024-09-16T00:42:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d37127740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56259848bce0 con 0x56259857e360 2024-09-16T00:42:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d37127740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5625984e02a0 con 0x56259857ec80 2024-09-16T00:42:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d277fe640 1 -- 172.21.15.33:0/1301168317 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1d2002e030 con 0x7f1d30102d90 2024-09-16T00:42:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x56259858fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d2e59c640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1d1c0a32c0 0x7f1d1c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x56259858fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42688/0 (socket says 172.21.15.33:42688) 2024-09-16T00:42:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.777+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/2663767053 learned_addr learned my addr 172.21.15.33:0/2663767053 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:19.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/2663767053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857e360 0x56259857e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d2dd9b640 1 --2- 172.21.15.33:0/2663767053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857ec80 0x56259858d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/2663767053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857ec80 msgr2=0x56259858d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:19.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/2663767053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857ec80 0x56259858d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/2663767053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857e360 msgr2=0x56259857e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:19.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/2663767053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857e360 0x56259857e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/2663767053 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562598470440 con 0x56259858da90 2024-09-16T00:42:19.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d2e59c640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1d1c0a32c0 0x7f1d1c0a5780 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1d200c5e70 tx=0x7f1d200a8040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:19.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d2dd9b640 1 --2- 172.21.15.33:0/2663767053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857ec80 0x56259858d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:19.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/2663767053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x56259858fe80 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f1d28014900 tx=0x7f1d2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f899f4baa6f42a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:19.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d24ff9640 1 -- 172.21.15.33:0/2663767053 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d280142d0 con 0x56259858da90 2024-09-16T00:42:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d24ff9640 1 -- 172.21.15.33:0/2663767053 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d2800fc40 con 0x56259858da90 2024-09-16T00:42:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.778+0000 7f1d24ff9640 1 -- 172.21.15.33:0/2663767053 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d28012920 con 0x56259858da90 2024-09-16T00:42:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.779+0000 7f1d37127740 1 -- 172.21.15.33:0/2663767053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 msgr2=0x56259858fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.779+0000 7f1d37127740 1 --2- 172.21.15.33:0/2663767053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x56259858fe80 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f1d28014900 tx=0x7f1d2800c490 comp rx=0 tx=0).stop 2024-09-16T00:42:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.779+0000 7f1d37127740 1 -- 172.21.15.33:0/2663767053 shutdown_connections 2024-09-16T00:42:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.779+0000 7f1d37127740 1 --2- 172.21.15.33:0/2663767053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x56259858fe80 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.779+0000 7f1d37127740 1 --2- 172.21.15.33:0/2663767053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857ec80 0x56259858d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.779+0000 7f1d37127740 1 --2- 172.21.15.33:0/2663767053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857e360 0x56259857e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.779+0000 7f1d37127740 1 -- 172.21.15.33:0/2663767053 >> 172.21.15.33:0/2663767053 conn(0x56259857aa30 msgr2=0x56259857ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.779+0000 7f1d37127740 1 -- 172.21.15.33:0/2663767053 shutdown_connections 2024-09-16T00:42:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.779+0000 7f1d37127740 1 -- 172.21.15.33:0/2663767053 wait complete. 2024-09-16T00:42:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.779+0000 7f1d37127740 1 -- start start 2024-09-16T00:42:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d37127740 1 Processor -- start 2024-09-16T00:42:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d37127740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259857e360 0x56259858d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d37127740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857ec80 0x56259858a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d37127740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259858da90 0x56259858ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d37127740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562598590660 con 0x56259857e360 2024-09-16T00:42:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d37127740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56259849f050 con 0x56259857ec80 2024-09-16T00:42:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d37127740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5625985907e0 con 0x56259858da90 2024-09-16T00:42:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d37127740 10 client.0 _ll_register_callbacks cb 0x5625984df160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259857e360 0x56259858d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259857e360 0x56259858d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42704/0 (socket says 172.21.15.33:42704) 2024-09-16T00:42:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/49221633 learned_addr learned my addr 172.21.15.33:0/49221633 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:19.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259858da90 0x56259858ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d2dd9b640 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857ec80 0x56259858a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/49221633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259858da90 msgr2=0x56259858ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:19.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259858da90 0x56259858ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/49221633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857ec80 msgr2=0x56259858a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:19.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857ec80 0x56259858a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.780+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/49221633 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625982553c0 con 0x56259857e360 2024-09-16T00:42:19.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259857e360 0x56259858d460 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800c8c0 tx=0x7f1d2800c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb315c243e9a286 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:19.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:19.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d15ffb640 1 -- 172.21.15.33:0/49221633 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d280142d0 con 0x56259857e360 2024-09-16T00:42:19.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d15ffb640 1 -- 172.21.15.33:0/49221633 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1d2800f730 con 0x56259857e360 2024-09-16T00:42:19.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d37127740 1 -- 172.21.15.33:0/49221633 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d2800fc40 con 0x56259857e360 2024-09-16T00:42:19.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d15ffb640 1 -- 172.21.15.33:0/49221633 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1d28035050 con 0x56259857e360 2024-09-16T00:42:19.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d37127740 10 client.20115 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:19.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d37127740 1 -- 172.21.15.33:0/49221633 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56259859c0d0 con 0x56259857e360 2024-09-16T00:42:19.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d15ffb640 1 -- 172.21.15.33:0/49221633 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d2800fc40 con 0x56259857e360 2024-09-16T00:42:19.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d15ffb640 1 -- 172.21.15.33:0/49221633 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f1d28011600 con 0x56259857e360 2024-09-16T00:42:19.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d37127740 10 client.20115 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:19.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d37127740 10 client.20115 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:42:19.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d37127740 20 client.20115 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:19.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d37127740 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56259859d420 0x56259859f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.781+0000 7f1d37127740 4 client.20115 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:42:19.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.782+0000 7f1d37127740 1 -- 172.21.15.33:0/49221633 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625984f1850 con 0x56259859d420 2024-09-16T00:42:19.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.782+0000 7f1d2dd9b640 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56259859d420 0x56259859f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.782+0000 7f1d2dd9b640 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56259859d420 0x56259859f810 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8414763aa6a9fb1 server_cookie=b52bddae512126f8 in_seq=0 out_seq=0 2024-09-16T00:42:19.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.782+0000 7f1d15ffb640 10 client.20115 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:19.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d15ffb640 1 -- 172.21.15.33:0/49221633 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5625984f1850 con 0x56259859d420 2024-09-16T00:42:19.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d15ffb640 10 client.20115 handle_command_reply: tid=0 2024-09-16T00:42:19.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d37127740 1 client.20115 shutdown 2024-09-16T00:42:19.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d37127740 1 -- 172.21.15.33:0/49221633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259857e360 msgr2=0x56259858d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:19.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d37127740 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259857e360 0x56259858d460 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800c8c0 tx=0x7f1d2800c3f0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d37127740 1 -- 172.21.15.33:0/49221633 shutdown_connections 2024-09-16T00:42:19.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d37127740 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x56259859d420 0x56259859f810 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d37127740 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259858da90 0x56259858ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d15ffb640 0 client.20115 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:19.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d37127740 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857ec80 0x56259858a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d37127740 1 --2- 172.21.15.33:0/49221633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259857e360 0x56259858d460 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d37127740 1 -- 172.21.15.33:0/49221633 >> 172.21.15.33:0/49221633 conn(0x56259857aa30 msgr2=0x562598580790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:19.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d37127740 1 -- 172.21.15.33:0/49221633 shutdown_connections 2024-09-16T00:42:19.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d37127740 1 -- 172.21.15.33:0/49221633 wait complete. 2024-09-16T00:42:19.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.783+0000 7f1d37127740 20 client.20115 trim_cache size 0 max 16384 2024-09-16T00:42:19.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.801+0000 7f1d37127740 1 Processor -- start 2024-09-16T00:42:19.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.801+0000 7f1d37127740 1 -- start start 2024-09-16T00:42:19.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d37127740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857e360 0x562598599900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d37127740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857ec80 0x5625985a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d37127740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x5625985a5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d37127740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301af880 con 0x56259858da90 2024-09-16T00:42:19.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d37127740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301ad150 con 0x56259857ec80 2024-09-16T00:42:19.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d37127740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301aafa0 con 0x56259857e360 2024-09-16T00:42:19.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x5625985a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x5625985a5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42720/0 (socket says 172.21.15.33:42720) 2024-09-16T00:42:19.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/180182622 learned_addr learned my addr 172.21.15.33:0/180182622 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:19.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/180182622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857e360 0x562598599900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d2dd9b640 1 --2- 172.21.15.33:0/180182622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857ec80 0x5625985a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/180182622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857e360 msgr2=0x562598599900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:19.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/180182622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857e360 0x562598599900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/180182622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857ec80 msgr2=0x5625985a4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:19.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/180182622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857ec80 0x5625985a4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/180182622 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56259858d7d0 con 0x56259858da90 2024-09-16T00:42:19.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.802+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/180182622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x5625985a5470 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800c780 tx=0x7f1d28024040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d445738ee9ebc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:19.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.803+0000 7f1d17fff640 1 -- 172.21.15.33:0/180182622 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2800ccd0 con 0x56259858da90 2024-09-16T00:42:19.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.803+0000 7f1d17fff640 1 -- 172.21.15.33:0/180182622 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d28014110 con 0x56259858da90 2024-09-16T00:42:19.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.803+0000 7f1d17fff640 1 -- 172.21.15.33:0/180182622 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d28005030 con 0x56259858da90 2024-09-16T00:42:19.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.803+0000 7f1d37127740 1 -- 172.21.15.33:0/180182622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 msgr2=0x5625985a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:19.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.803+0000 7f1d37127740 1 --2- 172.21.15.33:0/180182622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x5625985a5470 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800c780 tx=0x7f1d28024040 comp rx=0 tx=0).stop 2024-09-16T00:42:19.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.803+0000 7f1d37127740 1 -- 172.21.15.33:0/180182622 shutdown_connections 2024-09-16T00:42:19.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.803+0000 7f1d37127740 1 --2- 172.21.15.33:0/180182622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x5625985a5470 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.803+0000 7f1d37127740 1 --2- 172.21.15.33:0/180182622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857ec80 0x5625985a4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.803+0000 7f1d37127740 1 --2- 172.21.15.33:0/180182622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857e360 0x562598599900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.803+0000 7f1d37127740 1 -- 172.21.15.33:0/180182622 >> 172.21.15.33:0/180182622 conn(0x56259857aa30 msgr2=0x5625985896d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.803+0000 7f1d37127740 1 -- 172.21.15.33:0/180182622 shutdown_connections 2024-09-16T00:42:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.804+0000 7f1d37127740 1 -- 172.21.15.33:0/180182622 wait complete. 2024-09-16T00:42:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.804+0000 7f1d37127740 1 -- start start 2024-09-16T00:42:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.804+0000 7f1d37127740 1 Processor -- start 2024-09-16T00:42:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.804+0000 7f1d37127740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857e360 0x562598596300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.804+0000 7f1d37127740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857ec80 0x5625985875c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.804+0000 7f1d37127740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x562598587bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.804+0000 7f1d37127740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301ab610 con 0x56259858da90 2024-09-16T00:42:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.804+0000 7f1d37127740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301aca80 con 0x56259857e360 2024-09-16T00:42:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.804+0000 7f1d37127740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301abce0 con 0x56259857ec80 2024-09-16T00:42:19.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.804+0000 7f1d37127740 10 client.0 _ll_register_callbacks cb 0x7f1d2800ce70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:19.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.804+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x562598587bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.804+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x562598587bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42732/0 (socket says 172.21.15.33:42732) 2024-09-16T00:42:19.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/1324189858 learned_addr learned my addr 172.21.15.33:0/1324189858 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:19.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857e360 0x562598596300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d2dd9b640 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857ec80 0x5625985875c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/1324189858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857ec80 msgr2=0x5625985875c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:19.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857ec80 0x5625985875c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/1324189858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857e360 msgr2=0x562598596300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:19.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857e360 0x562598596300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d2ed9d640 1 -- 172.21.15.33:0/1324189858 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562598588400 con 0x56259858da90 2024-09-16T00:42:19.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d2ed9d640 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x562598587bd0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1d28005f50 tx=0x7f1d280124f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d2ece85c34324b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:19.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:19.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d15ffb640 1 -- 172.21.15.33:0/1324189858 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2800f730 con 0x56259858da90 2024-09-16T00:42:19.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d15ffb640 1 -- 172.21.15.33:0/1324189858 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1d2800cad0 con 0x56259858da90 2024-09-16T00:42:19.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.805+0000 7f1d37127740 1 -- 172.21.15.33:0/1324189858 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d28014110 con 0x56259858da90 2024-09-16T00:42:19.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.806+0000 7f1d15ffb640 1 -- 172.21.15.33:0/1324189858 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1d2804c050 con 0x56259858da90 2024-09-16T00:42:19.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.806+0000 7f1d37127740 10 client.20121 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:19.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.806+0000 7f1d37127740 1 -- 172.21.15.33:0/1324189858 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562598608040 con 0x56259858da90 2024-09-16T00:42:19.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.806+0000 7f1d15ffb640 1 -- 172.21.15.33:0/1324189858 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d28014110 con 0x56259858da90 2024-09-16T00:42:19.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.806+0000 7f1d15ffb640 1 -- 172.21.15.33:0/1324189858 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f1d28012910 con 0x56259858da90 2024-09-16T00:42:19.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.806+0000 7f1d37127740 10 client.20121 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:19.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.806+0000 7f1d37127740 10 client.20121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:42:19.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.806+0000 7f1d37127740 20 client.20121 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:19.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.806+0000 7f1d37127740 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562598608650 0x56259860aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:19.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.806+0000 7f1d37127740 4 client.20121 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:42:19.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.806+0000 7f1d37127740 1 -- 172.21.15.33:0/1324189858 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5625984ebd60 con 0x562598608650 2024-09-16T00:42:19.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.806+0000 7f1d2dd9b640 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562598608650 0x56259860aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:19.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d2dd9b640 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562598608650 0x56259860aa40 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57fa12ad65d98f7e server_cookie=7ea139ee9b41d639 in_seq=0 out_seq=0 2024-09-16T00:42:19.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d15ffb640 10 client.20121 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:19.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d15ffb640 1 -- 172.21.15.33:0/1324189858 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5625984ebd60 con 0x562598608650 2024-09-16T00:42:19.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d15ffb640 10 client.20121 handle_command_reply: tid=0 2024-09-16T00:42:19.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d37127740 1 client.20121 shutdown 2024-09-16T00:42:19.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d37127740 1 -- 172.21.15.33:0/1324189858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 msgr2=0x562598587bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:19.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d37127740 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x562598587bd0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1d28005f50 tx=0x7f1d280124f0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d37127740 1 -- 172.21.15.33:0/1324189858 shutdown_connections 2024-09-16T00:42:19.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d37127740 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x562598608650 0x56259860aa40 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d37127740 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259858da90 0x562598587bd0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d37127740 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56259857ec80 0x5625985875c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d37127740 1 --2- 172.21.15.33:0/1324189858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56259857e360 0x562598596300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d37127740 1 -- 172.21.15.33:0/1324189858 >> 172.21.15.33:0/1324189858 conn(0x56259857aa30 msgr2=0x56259857fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:19.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.807+0000 7f1d15ffb640 0 client.20121 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:19.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.808+0000 7f1d37127740 1 -- 172.21.15.33:0/1324189858 shutdown_connections 2024-09-16T00:42:19.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.808+0000 7f1d37127740 1 -- 172.21.15.33:0/1324189858 wait complete. 2024-09-16T00:42:19.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.808+0000 7f1d37127740 20 client.20121 trim_cache size 0 max 16384 2024-09-16T00:42:19.819 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:42:19.819 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T00:42:19.819 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-09-16T00:42:19.819 INFO:teuthology.orchestra.run.smithi033.stdout: "5e969d48-5420-4759-8c78-d6b4221fdc7c": { 2024-09-16T00:42:19.819 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-09-16T00:42:19.819 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "5e969d48-5420-4759-8c78-d6b4221fdc7c", 2024-09-16T00:42:19.819 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-09-16T00:42:19.820 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:42:19.820 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:42:19.820 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:42:19.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.808+0000 7f1d34d4e640 1 -- 172.21.15.33:0/1301168317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1d1c0a32c0 msgr2=0x7f1d1c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:19.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.808+0000 7f1d34d4e640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1d1c0a32c0 0x7f1d1c0a5780 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1d200c5e70 tx=0x7f1d200a8040 comp rx=0 tx=0).stop 2024-09-16T00:42:19.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.808+0000 7f1d34d4e640 1 -- 172.21.15.33:0/1301168317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d30102d90 msgr2=0x7f1d301985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:19.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.808+0000 7f1d34d4e640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d30102d90 0x7f1d301985d0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f1d2002ae00 tx=0x7f1d20003970 comp rx=0 tx=0).stop 2024-09-16T00:42:19.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.809+0000 7f1d34d4e640 1 -- 172.21.15.33:0/1301168317 shutdown_connections 2024-09-16T00:42:19.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.809+0000 7f1d34d4e640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1d1c0a32c0 0x7f1d1c0a5780 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.809+0000 7f1d34d4e640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d30108290 0x7f1d3019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.809+0000 7f1d34d4e640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d30103740 0x7f1d30198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.809+0000 7f1d34d4e640 1 --2- 172.21.15.33:0/1301168317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d30102d90 0x7f1d301985d0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:19.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.809+0000 7f1d34d4e640 1 -- 172.21.15.33:0/1301168317 >> 172.21.15.33:0/1301168317 conn(0x7f1d300fc7d0 msgr2=0x7f1d30106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:19.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.809+0000 7f1d34d4e640 1 -- 172.21.15.33:0/1301168317 shutdown_connections 2024-09-16T00:42:19.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:19.809+0000 7f1d34d4e640 1 -- 172.21.15.33:0/1301168317 wait complete. 2024-09-16T00:42:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:19 smithi083 ceph-mon[35242]: pgmap v1105: 129 pgs: 129 active+clean; 696 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 2.1 MiB/s wr, 58 op/s 2024-09-16T00:42:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:19 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4164597992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:42:20.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.09 MB/sec execute 566 sec latency 26486.978 ms 2024-09-16T00:42:20.603 INFO:tasks.cephfs.filesystem:scrub status for tag:5e969d48-5420-4759-8c78-d6b4221fdc7c - {'path': '/', 'tag': '5e969d48-5420-4759-8c78-d6b4221fdc7c', 'options': 'recursive,force'} 2024-09-16T00:42:20.631 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:21.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223247 17.06 MB/sec execute 567 sec latency 27487.043 ms 2024-09-16T00:42:21.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.135+0000 7fcd311a5640 1 -- 172.21.15.33:0/2719487815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd2c100e20 msgr2=0x7fcd2c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:21.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.135+0000 7fcd311a5640 1 --2- 172.21.15.33:0/2719487815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd2c100e20 0x7fcd2c101200 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009900 tx=0x7fcd1c02a480 comp rx=0 tx=0).stop 2024-09-16T00:42:21.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.135+0000 7fcd311a5640 1 -- 172.21.15.33:0/2719487815 shutdown_connections 2024-09-16T00:42:21.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.135+0000 7fcd311a5640 1 --2- 172.21.15.33:0/2719487815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd2c102ef0 0x7fcd2c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.135+0000 7fcd311a5640 1 --2- 172.21.15.33:0/2719487815 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd2c102570 0x7fcd2c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.135+0000 7fcd311a5640 1 --2- 172.21.15.33:0/2719487815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd2c100e20 0x7fcd2c101200 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.135+0000 7fcd311a5640 1 -- 172.21.15.33:0/2719487815 >> 172.21.15.33:0/2719487815 conn(0x7fcd2c0fc7d0 msgr2=0x7fcd2c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:21.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.135+0000 7fcd311a5640 1 -- 172.21.15.33:0/2719487815 shutdown_connections 2024-09-16T00:42:21.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.135+0000 7fcd311a5640 1 -- 172.21.15.33:0/2719487815 wait complete. 2024-09-16T00:42:21.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.136+0000 7fcd311a5640 1 Processor -- start 2024-09-16T00:42:21.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.136+0000 7fcd311a5640 1 -- start start 2024-09-16T00:42:21.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd311a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd2c100e20 0x7fcd2c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd311a5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd2c102570 0x7fcd2c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd311a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd2c102ef0 0x7fcd2c19cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd311a5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2c06a980 con 0x7fcd2c102570 2024-09-16T00:42:21.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd311a5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2c06a800 con 0x7fcd2c100e20 2024-09-16T00:42:21.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd311a5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2c06ab00 con 0x7fcd2c102ef0 2024-09-16T00:42:21.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd2a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd2c102570 0x7fcd2c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd2a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd2c102570 0x7fcd2c198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35676/0 (socket says 172.21.15.33:35676) 2024-09-16T00:42:21.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd2a59c640 1 -- 172.21.15.33:0/268242152 learned_addr learned my addr 172.21.15.33:0/268242152 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:21.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd2b59e640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd2c102ef0 0x7fcd2c19cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd2ad9d640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd2c100e20 0x7fcd2c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd2a59c640 1 -- 172.21.15.33:0/268242152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd2c102ef0 msgr2=0x7fcd2c19cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:21.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd2c102ef0 0x7fcd2c19cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd2a59c640 1 -- 172.21.15.33:0/268242152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd2c100e20 msgr2=0x7fcd2c198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:21.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd2c100e20 0x7fcd2c198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd2a59c640 1 -- 172.21.15.33:0/268242152 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd2c19d4c0 con 0x7fcd2c102570 2024-09-16T00:42:21.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.137+0000 7fcd2ad9d640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd2c100e20 0x7fcd2c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:21.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.138+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd2c102570 0x7fcd2c198a50 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000e9f0 tx=0x7fcd2000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:21.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.138+0000 7fcd13fff640 1 -- 172.21.15.33:0/268242152 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2000cd80 con 0x7fcd2c102570 2024-09-16T00:42:21.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.138+0000 7fcd311a5640 1 -- 172.21.15.33:0/268242152 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd2c19d7b0 con 0x7fcd2c102570 2024-09-16T00:42:21.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.138+0000 7fcd13fff640 1 -- 172.21.15.33:0/268242152 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd20005500 con 0x7fcd2c102570 2024-09-16T00:42:21.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.138+0000 7fcd13fff640 1 -- 172.21.15.33:0/268242152 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd20010690 con 0x7fcd2c102570 2024-09-16T00:42:21.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.138+0000 7fcd311a5640 1 -- 172.21.15.33:0/268242152 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd2c1a5090 con 0x7fcd2c102570 2024-09-16T00:42:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd3357e740 1 Processor -- start 2024-09-16T00:42:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd3357e740 1 -- start start 2024-09-16T00:42:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd13fff640 1 -- 172.21.15.33:0/268242152 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fcd20010830 con 0x7fcd2c102570 2024-09-16T00:42:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd3357e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd3357e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd3357e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2facd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd3357e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e2fa24a620 con 0x55e2fa2f88e0 2024-09-16T00:42:21.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd3357e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e2fa1f1e50 con 0x55e2fa2e65e0 2024-09-16T00:42:21.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd3357e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e2fa2465e0 con 0x55e2fa2e5440 2024-09-16T00:42:21.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd2a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2facd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd2a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2facd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35686/0 (socket says 172.21.15.33:35686) 2024-09-16T00:42:21.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd2a59c640 1 -- 172.21.15.33:0/2755401346 learned_addr learned my addr 172.21.15.33:0/2755401346 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:21.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd2ad9d640 1 --2- 172.21.15.33:0/2755401346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.141+0000 7fcd2b59e640 1 --2- 172.21.15.33:0/2755401346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.142+0000 7fcd2a59c640 1 -- 172.21.15.33:0/2755401346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e5440 msgr2=0x55e2fa2e5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:21.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.142+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/2755401346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.142+0000 7fcd2a59c640 1 -- 172.21.15.33:0/2755401346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2e65e0 msgr2=0x55e2fa2e4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:21.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.142+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/2755401346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2e4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.142+0000 7fcd2a59c640 1 -- 172.21.15.33:0/2755401346 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2fa1e2d40 con 0x55e2fa2f88e0 2024-09-16T00:42:21.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.142+0000 7fcd2b59e640 1 --2- 172.21.15.33:0/2755401346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2e4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:21.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.142+0000 7fcd13fff640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcd140a3240 0x7fcd140a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.142+0000 7fcd2b59e640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcd140a3240 0x7fcd140a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.142+0000 7fcd13fff640 1 -- 172.21.15.33:0/268242152 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fcd200d0260 con 0x7fcd2c102570 2024-09-16T00:42:21.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.142+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/2755401346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2facd0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fcd200d3fd0 tx=0x7fcd200d83b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3e37774a61a279 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:21.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.142+0000 7fcd117fa640 1 -- 172.21.15.33:0/2755401346 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd200986a0 con 0x55e2fa2f88e0 2024-09-16T00:42:21.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.143+0000 7fcd2b59e640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcd140a3240 0x7fcd140a5700 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009fd0 tx=0x7fcd24009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:21.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.143+0000 7fcd117fa640 1 -- 172.21.15.33:0/2755401346 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd2009cb00 con 0x55e2fa2f88e0 2024-09-16T00:42:21.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.143+0000 7fcd117fa640 1 -- 172.21.15.33:0/2755401346 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2009bd60 con 0x55e2fa2f88e0 2024-09-16T00:42:21.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.143+0000 7fcd3357e740 1 -- 172.21.15.33:0/2755401346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 msgr2=0x55e2fa2facd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:21.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.143+0000 7fcd3357e740 1 --2- 172.21.15.33:0/2755401346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2facd0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fcd200d3fd0 tx=0x7fcd200d83b0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.143+0000 7fcd3357e740 1 -- 172.21.15.33:0/2755401346 shutdown_connections 2024-09-16T00:42:21.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.143+0000 7fcd3357e740 1 --2- 172.21.15.33:0/2755401346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2facd0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.143+0000 7fcd3357e740 1 --2- 172.21.15.33:0/2755401346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.143+0000 7fcd3357e740 1 --2- 172.21.15.33:0/2755401346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2e4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.143+0000 7fcd3357e740 1 -- 172.21.15.33:0/2755401346 >> 172.21.15.33:0/2755401346 conn(0x55e2fa2e15f0 msgr2=0x55e2fa2e19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:21.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.143+0000 7fcd3357e740 1 -- 172.21.15.33:0/2755401346 shutdown_connections 2024-09-16T00:42:21.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.144+0000 7fcd3357e740 1 -- 172.21.15.33:0/2755401346 wait complete. 2024-09-16T00:42:21.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.144+0000 7fcd3357e740 1 -- start start 2024-09-16T00:42:21.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.144+0000 7fcd3357e740 1 Processor -- start 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd3357e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd3357e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2e9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd3357e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2e97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd3357e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e2fa2fb610 con 0x55e2fa2f88e0 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd3357e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55e2fa1d2f10 con 0x55e2fa2e5440 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd3357e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e2fa2fb790 con 0x55e2fa2e65e0 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd3357e740 10 client.0 _ll_register_callbacks cb 0x55e2fa1f6db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd2ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd2b59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2e97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd2ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:49290/0 (socket says 172.21.15.33:49290) 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd2ad9d640 1 -- 172.21.15.33:0/3337495096 learned_addr learned my addr 172.21.15.33:0/3337495096 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2e9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd2a59c640 1 -- 172.21.15.33:0/3337495096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2e5440 msgr2=0x55e2fa2e8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:21.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.145+0000 7fcd2a59c640 1 -- 172.21.15.33:0/3337495096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 msgr2=0x55e2fa2e97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:21.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.146+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2e97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.146+0000 7fcd2a59c640 1 -- 172.21.15.33:0/3337495096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2f9f59580 con 0x55e2fa2e65e0 2024-09-16T00:42:21.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.146+0000 7fcd2b59e640 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2e97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:21.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.146+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2e9260 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55e2fa2e26e0 tx=0x7fcd200d05d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fea1f9287d559688 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:21.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.146+0000 7fcd0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:42:21.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.146+0000 7fcd0eff5640 1 -- 172.21.15.33:0/3337495096 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2009bf40 con 0x55e2fa2e65e0 2024-09-16T00:42:21.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.146+0000 7fcd3357e740 1 -- 172.21.15.33:0/3337495096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd2009cb00 con 0x55e2fa2e65e0 2024-09-16T00:42:21.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.146+0000 7fcd0eff5640 1 -- 172.21.15.33:0/3337495096 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fcd2009c660 con 0x55e2fa2e65e0 2024-09-16T00:42:21.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.146+0000 7fcd0eff5640 1 -- 172.21.15.33:0/3337495096 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fcd2009c1d0 con 0x55e2fa2e65e0 2024-09-16T00:42:21.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.146+0000 7fcd3357e740 10 client.29221 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:21.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.146+0000 7fcd3357e740 1 -- 172.21.15.33:0/3337495096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e2fa023960 con 0x55e2fa2e65e0 2024-09-16T00:42:21.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.147+0000 7fcd0eff5640 1 -- 172.21.15.33:0/3337495096 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd2009cb00 con 0x55e2fa2e65e0 2024-09-16T00:42:21.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.147+0000 7fcd0eff5640 1 -- 172.21.15.33:0/3337495096 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fcd200a4be0 con 0x55e2fa2e65e0 2024-09-16T00:42:21.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.147+0000 7fcd3357e740 10 client.29221 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:21.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.147+0000 7fcd3357e740 10 client.29221 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:42:21.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.147+0000 7fcd3357e740 20 client.29221 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:21.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.147+0000 7fcd3357e740 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e2fa308570 0x55e2fa30a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.147+0000 7fcd3357e740 4 client.29221 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:42:21.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.147+0000 7fcd3357e740 1 -- 172.21.15.33:0/3337495096 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2fa257b90 con 0x55e2fa308570 2024-09-16T00:42:21.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.147+0000 7fcd2ad9d640 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e2fa308570 0x55e2fa30a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.148+0000 7fcd0eff5640 10 client.29221 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:21.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.148+0000 7fcd2ad9d640 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e2fa308570 0x55e2fa30a960 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76c9db7e21acae83 server_cookie=44e980a96856f922 in_seq=0 out_seq=0 2024-09-16T00:42:21.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.148+0000 7fcd0eff5640 1 -- 172.21.15.33:0/3337495096 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e2fa257b90 con 0x55e2fa308570 2024-09-16T00:42:21.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.148+0000 7fcd0eff5640 10 client.29221 handle_command_reply: tid=0 2024-09-16T00:42:21.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.148+0000 7fcd3357e740 1 client.29221 shutdown 2024-09-16T00:42:21.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.149+0000 7fcd3357e740 1 -- 172.21.15.33:0/3337495096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e65e0 msgr2=0x55e2fa2e9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:21.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.149+0000 7fcd3357e740 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2e9260 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55e2fa2e26e0 tx=0x7fcd200d05d0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.149+0000 7fcd3357e740 1 -- 172.21.15.33:0/3337495096 shutdown_connections 2024-09-16T00:42:21.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.149+0000 7fcd3357e740 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e2fa308570 0x55e2fa30a960 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.149+0000 7fcd3357e740 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2e97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.149+0000 7fcd0eff5640 0 client.29221 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:21.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.149+0000 7fcd3357e740 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2e9260 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.149+0000 7fcd3357e740 1 --2- 172.21.15.33:0/3337495096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.149+0000 7fcd3357e740 1 -- 172.21.15.33:0/3337495096 >> 172.21.15.33:0/3337495096 conn(0x55e2fa2e15f0 msgr2=0x55e2fa2e7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:21.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.149+0000 7fcd3357e740 1 -- 172.21.15.33:0/3337495096 shutdown_connections 2024-09-16T00:42:21.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.149+0000 7fcd3357e740 1 -- 172.21.15.33:0/3337495096 wait complete. 2024-09-16T00:42:21.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.149+0000 7fcd3357e740 20 client.29221 trim_cache size 0 max 16384 2024-09-16T00:42:21.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.181+0000 7fcd3357e740 1 Processor -- start 2024-09-16T00:42:21.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.181+0000 7fcd3357e740 1 -- start start 2024-09-16T00:42:21.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.181+0000 7fcd3357e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.181+0000 7fcd3357e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.181+0000 7fcd3357e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2f7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.181+0000 7fcd3357e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2c1af830 con 0x55e2fa2e65e0 2024-09-16T00:42:21.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.181+0000 7fcd3357e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2c1ad100 con 0x55e2fa2f88e0 2024-09-16T00:42:21.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.181+0000 7fcd3357e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2c1aaf50 con 0x55e2fa2e5440 2024-09-16T00:42:21.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.182+0000 7fcd2a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.182+0000 7fcd2a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2f6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35696/0 (socket says 172.21.15.33:35696) 2024-09-16T00:42:21.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.182+0000 7fcd2a59c640 1 -- 172.21.15.33:0/2299111854 learned_addr learned my addr 172.21.15.33:0/2299111854 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:21.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.182+0000 7fcd2b59e640 1 --2- 172.21.15.33:0/2299111854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.182+0000 7fcd2ad9d640 1 --2- 172.21.15.33:0/2299111854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.182+0000 7fcd2b59e640 1 -- 172.21.15.33:0/2299111854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e5440 msgr2=0x55e2fa2e0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:21.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.182+0000 7fcd2b59e640 1 --2- 172.21.15.33:0/2299111854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.182+0000 7fcd2b59e640 1 -- 172.21.15.33:0/2299111854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e65e0 msgr2=0x55e2fa2f6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:21.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.182+0000 7fcd2b59e640 1 --2- 172.21.15.33:0/2299111854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2f6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.182+0000 7fcd2b59e640 1 -- 172.21.15.33:0/2299111854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2fa2f7eb0 con 0x55e2fa2f88e0 2024-09-16T00:42:21.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.182+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/2299111854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:21.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.182+0000 7fcd2b59e640 1 --2- 172.21.15.33:0/2299111854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2f7100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcd24005fd0 tx=0x7fcd24019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=753e367db9dffaee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:21.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.183+0000 7fcd10ff9640 1 -- 172.21.15.33:0/2299111854 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2400fda0 con 0x55e2fa2f88e0 2024-09-16T00:42:21.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.183+0000 7fcd10ff9640 1 -- 172.21.15.33:0/2299111854 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd24029480 con 0x55e2fa2f88e0 2024-09-16T00:42:21.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.183+0000 7fcd10ff9640 1 -- 172.21.15.33:0/2299111854 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2401e660 con 0x55e2fa2f88e0 2024-09-16T00:42:21.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.184+0000 7fcd3357e740 1 -- 172.21.15.33:0/2299111854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2f88e0 msgr2=0x55e2fa2f7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:21.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.184+0000 7fcd3357e740 1 --2- 172.21.15.33:0/2299111854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2f7100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcd24005fd0 tx=0x7fcd24019000 comp rx=0 tx=0).stop 2024-09-16T00:42:21.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.184+0000 7fcd3357e740 1 -- 172.21.15.33:0/2299111854 shutdown_connections 2024-09-16T00:42:21.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.184+0000 7fcd3357e740 1 --2- 172.21.15.33:0/2299111854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2f88e0 0x55e2fa2f7100 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.184+0000 7fcd3357e740 1 --2- 172.21.15.33:0/2299111854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e65e0 0x55e2fa2f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.184+0000 7fcd3357e740 1 --2- 172.21.15.33:0/2299111854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e5440 0x55e2fa2e0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.184+0000 7fcd3357e740 1 -- 172.21.15.33:0/2299111854 >> 172.21.15.33:0/2299111854 conn(0x55e2fa2e15f0 msgr2=0x55e2fa2fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:21.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.184+0000 7fcd3357e740 1 -- 172.21.15.33:0/2299111854 shutdown_connections 2024-09-16T00:42:21.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.184+0000 7fcd3357e740 1 -- 172.21.15.33:0/2299111854 wait complete. 2024-09-16T00:42:21.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.184+0000 7fcd3357e740 1 -- start start 2024-09-16T00:42:21.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.184+0000 7fcd3357e740 1 Processor -- start 2024-09-16T00:42:21.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd3357e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e5440 0x55e2fa309dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd3357e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e65e0 0x55e2fa30a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd3357e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2f88e0 0x55e2fa371ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd2ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e5440 0x55e2fa309dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd2ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e5440 0x55e2fa309dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35706/0 (socket says 172.21.15.33:35706) 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd2ad9d640 1 -- 172.21.15.33:0/4160264731 learned_addr learned my addr 172.21.15.33:0/4160264731 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd3357e740 1 -- 172.21.15.33:0/4160264731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2c1ab5c0 con 0x55e2fa2e5440 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd3357e740 1 -- 172.21.15.33:0/4160264731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2c1aca30 con 0x55e2fa2f88e0 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd3357e740 1 -- 172.21.15.33:0/4160264731 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2c1abc90 con 0x55e2fa2e65e0 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd3357e740 10 client.0 _ll_register_callbacks cb 0x7fcd200d8170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e65e0 0x55e2fa30a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd2b59e640 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2f88e0 0x55e2fa371ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd2ad9d640 1 -- 172.21.15.33:0/4160264731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e65e0 msgr2=0x55e2fa30a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd2ad9d640 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e65e0 0x55e2fa30a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd2ad9d640 1 -- 172.21.15.33:0/4160264731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2f88e0 msgr2=0x55e2fa371ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd2ad9d640 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2f88e0 0x55e2fa371ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd2ad9d640 1 -- 172.21.15.33:0/4160264731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2fa3723e0 con 0x55e2fa2e5440 2024-09-16T00:42:21.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.185+0000 7fcd2b59e640 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2f88e0 0x55e2fa371ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:21.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.186+0000 7fcd2ad9d640 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e5440 0x55e2fa309dc0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c006010 tx=0x7fcd1c0044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ccbd8ade2ce6df5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:21.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.186+0000 7fcd0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:21.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.186+0000 7fcd0eff5640 1 -- 172.21.15.33:0/4160264731 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c035040 con 0x55e2fa2e5440 2024-09-16T00:42:21.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.186+0000 7fcd0eff5640 1 -- 172.21.15.33:0/4160264731 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fcd1c005450 con 0x55e2fa2e5440 2024-09-16T00:42:21.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.186+0000 7fcd3357e740 1 -- 172.21.15.33:0/4160264731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd24029480 con 0x55e2fa2e5440 2024-09-16T00:42:21.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.186+0000 7fcd0eff5640 1 -- 172.21.15.33:0/4160264731 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fcd1c055050 con 0x55e2fa2e5440 2024-09-16T00:42:21.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.186+0000 7fcd3357e740 10 client.20139 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:21.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.186+0000 7fcd3357e740 1 -- 172.21.15.33:0/4160264731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e2fa3736e0 con 0x55e2fa2e5440 2024-09-16T00:42:21.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.186+0000 7fcd0eff5640 1 -- 172.21.15.33:0/4160264731 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd24029480 con 0x55e2fa2e5440 2024-09-16T00:42:21.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.186+0000 7fcd0eff5640 1 -- 172.21.15.33:0/4160264731 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fcd1c003890 con 0x55e2fa2e5440 2024-09-16T00:42:21.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.187+0000 7fcd3357e740 10 client.20139 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:21.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.187+0000 7fcd3357e740 10 client.20139 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:42:21.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.187+0000 7fcd3357e740 20 client.20139 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:21.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.187+0000 7fcd3357e740 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e2fa308570 0x55e2fa375ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:21.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.187+0000 7fcd3357e740 4 client.20139 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "78738c5c", "release": true, "await": true}] 2024-09-16T00:42:21.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.187+0000 7fcd3357e740 1 -- 172.21.15.33:0/4160264731 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "set_id": "78738c5c", "release": true, "await": true}) -- 0x55e2fa2520a0 con 0x55e2fa308570 2024-09-16T00:42:21.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.187+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e2fa308570 0x55e2fa375ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:21.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.187+0000 7fcd0eff5640 10 client.20139 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:21.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.187+0000 7fcd2a59c640 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e2fa308570 0x55e2fa375ec0 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75ba381652897b02 server_cookie=5aaff85542f2ecd6 in_seq=0 out_seq=0 2024-09-16T00:42:21.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.247+0000 7fcd0eff5640 1 -- 172.21.15.33:0/4160264731 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e2fa2520a0 con 0x55e2fa308570 2024-09-16T00:42:21.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.247+0000 7fcd0eff5640 10 client.20139 handle_command_reply: tid=0 2024-09-16T00:42:21.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.247+0000 7fcd3357e740 1 client.20139 shutdown 2024-09-16T00:42:21.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.247+0000 7fcd3357e740 1 -- 172.21.15.33:0/4160264731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e5440 msgr2=0x55e2fa309dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:21.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.247+0000 7fcd3357e740 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e5440 0x55e2fa309dc0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c006010 tx=0x7fcd1c0044b0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.247+0000 7fcd3357e740 1 -- 172.21.15.33:0/4160264731 shutdown_connections 2024-09-16T00:42:21.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.247+0000 7fcd3357e740 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x55e2fa308570 0x55e2fa375ec0 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.247+0000 7fcd3357e740 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e2fa2f88e0 0x55e2fa371ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.247+0000 7fcd3357e740 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2fa2e65e0 0x55e2fa30a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.247+0000 7fcd3357e740 1 --2- 172.21.15.33:0/4160264731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e2fa2e5440 0x55e2fa309dc0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.247+0000 7fcd3357e740 1 -- 172.21.15.33:0/4160264731 >> 172.21.15.33:0/4160264731 conn(0x55e2fa2e15f0 msgr2=0x55e2fa2fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:21.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.247+0000 7fcd0eff5640 0 client.20139 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:21.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.248+0000 7fcd3357e740 1 -- 172.21.15.33:0/4160264731 shutdown_connections 2024-09-16T00:42:21.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.248+0000 7fcd3357e740 1 -- 172.21.15.33:0/4160264731 wait complete. 2024-09-16T00:42:21.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.248+0000 7fcd3357e740 20 client.20139 trim_cache size 0 max 16384 2024-09-16T00:42:21.250 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:42:21.250 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 33, 2024-09-16T00:42:21.250 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 88, 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: "78738c5c": { 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 88, 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 90.0, 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 147.2, 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-09-16T00:42:21.251 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-09-16T00:42:21.252 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-09-16T00:42:21.252 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-09-16T00:42:21.252 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-09-16T00:42:21.252 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-09-16T00:42:21.252 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:42:21.252 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:42:21.252 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:42:21.252 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:42:21.252 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:42:21.252 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:42:21.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.248+0000 7fcd311a5640 1 -- 172.21.15.33:0/268242152 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcd140a3240 msgr2=0x7fcd140a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:21.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.248+0000 7fcd311a5640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcd140a3240 0x7fcd140a5700 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009fd0 tx=0x7fcd24009500 comp rx=0 tx=0).stop 2024-09-16T00:42:21.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.248+0000 7fcd311a5640 1 -- 172.21.15.33:0/268242152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd2c102570 msgr2=0x7fcd2c198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:21.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.248+0000 7fcd311a5640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd2c102570 0x7fcd2c198a50 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000e9f0 tx=0x7fcd2000edf0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.249+0000 7fcd311a5640 1 -- 172.21.15.33:0/268242152 shutdown_connections 2024-09-16T00:42:21.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.249+0000 7fcd311a5640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fcd140a3240 0x7fcd140a5700 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.249+0000 7fcd311a5640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd2c102ef0 0x7fcd2c19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.249+0000 7fcd311a5640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd2c102570 0x7fcd2c198a50 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.249+0000 7fcd311a5640 1 --2- 172.21.15.33:0/268242152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd2c100e20 0x7fcd2c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:21.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.249+0000 7fcd311a5640 1 -- 172.21.15.33:0/268242152 >> 172.21.15.33:0/268242152 conn(0x7fcd2c0fc7d0 msgr2=0x7fcd2c0fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:21.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.249+0000 7fcd311a5640 1 -- 172.21.15.33:0/268242152 shutdown_connections 2024-09-16T00:42:21.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:21.249+0000 7fcd311a5640 1 -- 172.21.15.33:0/268242152 wait complete. 2024-09-16T00:42:21.788 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:21 smithi033 ceph-mon[31243]: pgmap v1106: 129 pgs: 129 active+clean; 696 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 2.2 MiB/s wr, 89 op/s 2024-09-16T00:42:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:21 smithi083 ceph-mon[35242]: pgmap v1106: 129 pgs: 129 active+clean; 696 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 2.2 MiB/s wr, 89 op/s 2024-09-16T00:42:21.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:21 smithi060 ceph-mon[35185]: pgmap v1106: 129 pgs: 129 active+clean; 696 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 2.2 MiB/s wr, 89 op/s 2024-09-16T00:42:22.036 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '78738c5c', total seconds elapsed: 31.8 2024-09-16T00:42:22.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 223623 17.05 MB/sec execute 568 sec latency 27630.970 ms 2024-09-16T00:42:23.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 224141 17.07 MB/sec execute 569 sec latency 93.651 ms 2024-09-16T00:42:23.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:23 smithi033 ceph-mon[31243]: pgmap v1107: 129 pgs: 129 active+clean; 696 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 2.2 MiB/s wr, 114 op/s 2024-09-16T00:42:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:23 smithi083 ceph-mon[35242]: pgmap v1107: 129 pgs: 129 active+clean; 696 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 2.2 MiB/s wr, 114 op/s 2024-09-16T00:42:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:23 smithi060 ceph-mon[35185]: pgmap v1107: 129 pgs: 129 active+clean; 696 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 2.2 MiB/s wr, 114 op/s 2024-09-16T00:42:24.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 224854 17.11 MB/sec execute 570 sec latency 83.092 ms 2024-09-16T00:42:25.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 225440 17.15 MB/sec execute 571 sec latency 85.851 ms 2024-09-16T00:42:25.782 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:25 smithi060 ceph-mon[35185]: pgmap v1108: 129 pgs: 129 active+clean; 697 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 162 op/s 2024-09-16T00:42:25.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:25 smithi033 ceph-mon[31243]: pgmap v1108: 129 pgs: 129 active+clean; 697 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 162 op/s 2024-09-16T00:42:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:25 smithi083 ceph-mon[35242]: pgmap v1108: 129 pgs: 129 active+clean; 697 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 162 op/s 2024-09-16T00:42:26.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 225938 17.16 MB/sec execute 572 sec latency 67.099 ms 2024-09-16T00:42:27.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 226359 17.17 MB/sec execute 573 sec latency 55.697 ms 2024-09-16T00:42:27.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:27 smithi033 ceph-mon[31243]: pgmap v1109: 129 pgs: 129 active+clean; 701 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.1 MiB/s wr, 214 op/s 2024-09-16T00:42:27.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:27 smithi083 ceph-mon[35242]: pgmap v1109: 129 pgs: 129 active+clean; 701 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.1 MiB/s wr, 214 op/s 2024-09-16T00:42:27.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:27 smithi060 ceph-mon[35185]: pgmap v1109: 129 pgs: 129 active+clean; 701 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.1 MiB/s wr, 214 op/s 2024-09-16T00:42:28.108 INFO:tasks.workunit.client.0.smithi033.stdout: 10 226887 17.19 MB/sec execute 574 sec latency 104.162 ms 2024-09-16T00:42:29.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 227397 17.21 MB/sec execute 575 sec latency 75.210 ms 2024-09-16T00:42:29.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:29 smithi033 ceph-mon[31243]: pgmap v1110: 129 pgs: 129 active+clean; 700 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 1.2 MiB/s wr, 241 op/s 2024-09-16T00:42:29.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:29 smithi083 ceph-mon[35242]: pgmap v1110: 129 pgs: 129 active+clean; 700 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 1.2 MiB/s wr, 241 op/s 2024-09-16T00:42:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:29 smithi060 ceph-mon[35185]: pgmap v1110: 129 pgs: 129 active+clean; 700 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 1.2 MiB/s wr, 241 op/s 2024-09-16T00:42:30.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 227803 17.20 MB/sec execute 576 sec latency 405.677 ms 2024-09-16T00:42:31.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 227972 17.19 MB/sec execute 577 sec latency 696.298 ms 2024-09-16T00:42:31.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:31 smithi033 ceph-mon[31243]: pgmap v1111: 129 pgs: 129 active+clean; 701 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-09-16T00:42:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:31 smithi083 ceph-mon[35242]: pgmap v1111: 129 pgs: 129 active+clean; 701 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-09-16T00:42:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:31 smithi060 ceph-mon[35185]: pgmap v1111: 129 pgs: 129 active+clean; 701 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-09-16T00:42:32.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 228291 17.19 MB/sec execute 578 sec latency 158.367 ms 2024-09-16T00:42:32.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:32 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:42:32.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:32 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:42:32.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:32 smithi033 ceph-mon[31243]: scrub summary: idle 2024-09-16T00:42:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:32 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:42:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:32 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:42:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:32 smithi083 ceph-mon[35242]: scrub summary: idle 2024-09-16T00:42:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:32 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:42:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:32 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:42:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:32 smithi060 ceph-mon[35185]: scrub summary: idle 2024-09-16T00:42:33.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 228976 17.22 MB/sec execute 579 sec latency 87.784 ms 2024-09-16T00:42:33.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:33 smithi033 ceph-mon[31243]: pgmap v1112: 129 pgs: 129 active+clean; 717 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.4 MiB/s wr, 380 op/s 2024-09-16T00:42:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:33 smithi083 ceph-mon[35242]: pgmap v1112: 129 pgs: 129 active+clean; 717 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.4 MiB/s wr, 380 op/s 2024-09-16T00:42:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:33 smithi060 ceph-mon[35185]: pgmap v1112: 129 pgs: 129 active+clean; 717 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.4 MiB/s wr, 380 op/s 2024-09-16T00:42:34.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 229718 17.28 MB/sec execute 580 sec latency 46.527 ms 2024-09-16T00:42:35.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 230123 17.29 MB/sec execute 581 sec latency 73.392 ms 2024-09-16T00:42:35.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:35 smithi033 ceph-mon[31243]: pgmap v1113: 129 pgs: 129 active+clean; 721 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.3 MiB/s wr, 460 op/s 2024-09-16T00:42:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:35 smithi083 ceph-mon[35242]: pgmap v1113: 129 pgs: 129 active+clean; 721 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.3 MiB/s wr, 460 op/s 2024-09-16T00:42:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:35 smithi060 ceph-mon[35185]: pgmap v1113: 129 pgs: 129 active+clean; 721 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.3 MiB/s wr, 460 op/s 2024-09-16T00:42:36.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 230476 17.29 MB/sec execute 582 sec latency 81.188 ms 2024-09-16T00:42:37.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 230647 17.28 MB/sec execute 583 sec latency 514.191 ms 2024-09-16T00:42:37.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:37 smithi033 ceph-mon[31243]: pgmap v1114: 129 pgs: 129 active+clean; 714 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.2 MiB/s wr, 425 op/s 2024-09-16T00:42:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:37 smithi083 ceph-mon[35242]: pgmap v1114: 129 pgs: 129 active+clean; 714 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.2 MiB/s wr, 425 op/s 2024-09-16T00:42:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:37 smithi060 ceph-mon[35185]: pgmap v1114: 129 pgs: 129 active+clean; 714 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.2 MiB/s wr, 425 op/s 2024-09-16T00:42:38.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 230813 17.26 MB/sec execute 584 sec latency 699.205 ms 2024-09-16T00:42:38.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:38 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:42:38.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:38 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:42:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:38 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:42:39.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 231377 17.27 MB/sec execute 585 sec latency 147.995 ms 2024-09-16T00:42:39.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:39 smithi033 ceph-mon[31243]: pgmap v1115: 129 pgs: 129 active+clean; 721 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 6.9 MiB/s wr, 430 op/s 2024-09-16T00:42:39.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:39 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:42:39.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:39 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:42:39.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:39 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:42:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:39 smithi083 ceph-mon[35242]: pgmap v1115: 129 pgs: 129 active+clean; 721 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 6.9 MiB/s wr, 430 op/s 2024-09-16T00:42:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:39 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:42:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:39 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:42:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:39 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:42:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:39 smithi060 ceph-mon[35185]: pgmap v1115: 129 pgs: 129 active+clean; 721 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 6.9 MiB/s wr, 430 op/s 2024-09-16T00:42:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:39 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:42:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:39 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:42:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:39 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:42:40.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 231904 17.30 MB/sec execute 586 sec latency 61.487 ms 2024-09-16T00:42:41.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 232408 17.30 MB/sec execute 587 sec latency 75.631 ms 2024-09-16T00:42:41.833 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:41 smithi033 ceph-mon[31243]: pgmap v1116: 129 pgs: 129 active+clean; 724 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 8.2 MiB/s wr, 477 op/s 2024-09-16T00:42:41.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:41 smithi083 ceph-mon[35242]: pgmap v1116: 129 pgs: 129 active+clean; 724 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 8.2 MiB/s wr, 477 op/s 2024-09-16T00:42:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:41 smithi060 ceph-mon[35185]: pgmap v1116: 129 pgs: 129 active+clean; 724 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 8.2 MiB/s wr, 477 op/s 2024-09-16T00:42:42.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 233004 17.34 MB/sec execute 588 sec latency 90.155 ms 2024-09-16T00:42:43.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 233394 17.34 MB/sec execute 589 sec latency 451.810 ms 2024-09-16T00:42:43.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:43 smithi033 ceph-mon[31243]: pgmap v1117: 129 pgs: 129 active+clean; 723 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 9.2 MiB/s wr, 521 op/s 2024-09-16T00:42:43.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:43 smithi083 ceph-mon[35242]: pgmap v1117: 129 pgs: 129 active+clean; 723 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 9.2 MiB/s wr, 521 op/s 2024-09-16T00:42:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:43 smithi060 ceph-mon[35185]: pgmap v1117: 129 pgs: 129 active+clean; 723 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 9.2 MiB/s wr, 521 op/s 2024-09-16T00:42:44.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 233929 17.37 MB/sec execute 590 sec latency 82.493 ms 2024-09-16T00:42:45.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 234298 17.38 MB/sec execute 591 sec latency 138.921 ms 2024-09-16T00:42:45.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:45 smithi033 ceph-mon[31243]: pgmap v1118: 129 pgs: 129 active+clean; 715 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.9 MiB/s wr, 484 op/s 2024-09-16T00:42:45.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:45 smithi083 ceph-mon[35242]: pgmap v1118: 129 pgs: 129 active+clean; 715 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.9 MiB/s wr, 484 op/s 2024-09-16T00:42:45.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:45 smithi060 ceph-mon[35185]: pgmap v1118: 129 pgs: 129 active+clean; 715 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.9 MiB/s wr, 484 op/s 2024-09-16T00:42:46.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 234644 17.37 MB/sec execute 592 sec latency 80.100 ms 2024-09-16T00:42:47.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 234936 17.37 MB/sec execute 593 sec latency 184.266 ms 2024-09-16T00:42:47.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:47 smithi033 ceph-mon[31243]: pgmap v1119: 129 pgs: 129 active+clean; 718 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.0 MiB/s wr, 439 op/s 2024-09-16T00:42:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:47 smithi083 ceph-mon[35242]: pgmap v1119: 129 pgs: 129 active+clean; 718 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.0 MiB/s wr, 439 op/s 2024-09-16T00:42:47.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:47 smithi060 ceph-mon[35185]: pgmap v1119: 129 pgs: 129 active+clean; 718 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.0 MiB/s wr, 439 op/s 2024-09-16T00:42:48.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 235445 17.38 MB/sec execute 594 sec latency 292.161 ms 2024-09-16T00:42:49.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 235984 17.39 MB/sec execute 595 sec latency 63.733 ms 2024-09-16T00:42:49.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:49 smithi033 ceph-mon[31243]: pgmap v1120: 129 pgs: 129 active+clean; 712 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.9 MiB/s wr, 447 op/s 2024-09-16T00:42:49.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:49 smithi083 ceph-mon[35242]: pgmap v1120: 129 pgs: 129 active+clean; 712 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.9 MiB/s wr, 447 op/s 2024-09-16T00:42:49.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:49 smithi060 ceph-mon[35185]: pgmap v1120: 129 pgs: 129 active+clean; 712 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.9 MiB/s wr, 447 op/s 2024-09-16T00:42:50.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 236598 17.43 MB/sec execute 596 sec latency 90.717 ms 2024-09-16T00:42:50.605 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-16T00:42:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:51.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 237077 17.44 MB/sec execute 597 sec latency 78.048 ms 2024-09-16T00:42:51.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.469+0000 7f4ef3b91640 1 -- 172.21.15.33:0/578559445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4eec102790 msgr2=0x7f4eec1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:51.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.469+0000 7f4ef3b91640 1 --2- 172.21.15.33:0/578559445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4eec102790 0x7f4eec1067e0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009900 tx=0x7f4edc02a410 comp rx=0 tx=0).stop 2024-09-16T00:42:51.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.470+0000 7f4ef3b91640 1 -- 172.21.15.33:0/578559445 shutdown_connections 2024-09-16T00:42:51.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.470+0000 7f4ef3b91640 1 --2- 172.21.15.33:0/578559445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eec106f10 0x7f4eec10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.470+0000 7f4ef3b91640 1 --2- 172.21.15.33:0/578559445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4eec102790 0x7f4eec1067e0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.470+0000 7f4ef3b91640 1 --2- 172.21.15.33:0/578559445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eec101de0 0x7f4eec1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.470+0000 7f4ef3b91640 1 -- 172.21.15.33:0/578559445 >> 172.21.15.33:0/578559445 conn(0x7f4eec0fd8e0 msgr2=0x7f4eec0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:51.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.470+0000 7f4ef3b91640 1 -- 172.21.15.33:0/578559445 shutdown_connections 2024-09-16T00:42:51.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.470+0000 7f4ef3b91640 1 -- 172.21.15.33:0/578559445 wait complete. 2024-09-16T00:42:51.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.471+0000 7f4ef3b91640 1 Processor -- start 2024-09-16T00:42:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.471+0000 7f4ef3b91640 1 -- start start 2024-09-16T00:42:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.471+0000 7f4ef3b91640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eec101de0 0x7f4eec198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.471+0000 7f4ef3b91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4eec102790 0x7f4eec1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.471+0000 7f4ef3b91640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eec106f10 0x7f4eec19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.471+0000 7f4ef3b91640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec111fc0 con 0x7f4eec102790 2024-09-16T00:42:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.471+0000 7f4ef3b91640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec111e40 con 0x7f4eec106f10 2024-09-16T00:42:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.471+0000 7f4ef3b91640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec112140 con 0x7f4eec101de0 2024-09-16T00:42:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.472+0000 7f4ef112c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4eec102790 0x7f4eec1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.472+0000 7f4ef192d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eec101de0 0x7f4eec198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.472+0000 7f4ef192d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eec101de0 0x7f4eec198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:35650/0 (socket says 172.21.15.33:35650) 2024-09-16T00:42:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.472+0000 7f4ef192d640 1 -- 172.21.15.33:0/1200373245 learned_addr learned my addr 172.21.15.33:0/1200373245 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:51.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.472+0000 7f4ef112c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4eec102790 0x7f4eec1989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56478/0 (socket says 172.21.15.33:56478) 2024-09-16T00:42:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.472+0000 7f4ef212e640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eec106f10 0x7f4eec19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.472+0000 7f4ef112c640 1 -- 172.21.15.33:0/1200373245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eec101de0 msgr2=0x7f4eec198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.472+0000 7f4ef112c640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eec101de0 0x7f4eec198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.473+0000 7f4ef112c640 1 -- 172.21.15.33:0/1200373245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eec106f10 msgr2=0x7f4eec19cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.473+0000 7f4ef112c640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eec106f10 0x7f4eec19cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.473+0000 7f4ef112c640 1 -- 172.21.15.33:0/1200373245 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eec19d410 con 0x7f4eec102790 2024-09-16T00:42:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.473+0000 7f4ef192d640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eec101de0 0x7f4eec198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.473+0000 7f4ef112c640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4eec102790 0x7f4eec1989a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f4edc02a930 tx=0x7f4edc003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.473+0000 7f4ee2ffd640 1 -- 172.21.15.33:0/1200373245 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4edc03f070 con 0x7f4eec102790 2024-09-16T00:42:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.473+0000 7f4ef3b91640 1 -- 172.21.15.33:0/1200373245 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eec19d6a0 con 0x7f4eec102790 2024-09-16T00:42:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.473+0000 7f4ee2ffd640 1 -- 172.21.15.33:0/1200373245 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4edc031c90 con 0x7f4eec102790 2024-09-16T00:42:51.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.474+0000 7f4ee2ffd640 1 -- 172.21.15.33:0/1200373245 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4edc035030 con 0x7f4eec102790 2024-09-16T00:42:51.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.474+0000 7f4ef3b91640 1 -- 172.21.15.33:0/1200373245 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eec1a4fe0 con 0x7f4eec102790 2024-09-16T00:42:51.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.476+0000 7f4ee2ffd640 1 -- 172.21.15.33:0/1200373245 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4edc042070 con 0x7f4eec102790 2024-09-16T00:42:51.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.477+0000 7f4ef5f6a740 1 Processor -- start 2024-09-16T00:42:51.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.477+0000 7f4ef5f6a740 1 -- start start 2024-09-16T00:42:51.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.477+0000 7f4ef5f6a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd124a0 0x561b6cd12880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.477+0000 7f4ef5f6a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd12dc0 0x561b6cd1d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.477+0000 7f4ef5f6a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd1d680 0x561b6cd1fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.477+0000 7f4ef5f6a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561b6cc782e0 con 0x561b6cd1d680 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.477+0000 7f4ef5f6a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561b6cc1fce0 con 0x561b6cd124a0 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.477+0000 7f4ef5f6a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561b6cc742a0 con 0x561b6cd12dc0 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.478+0000 7f4ef212e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd124a0 0x561b6cd12880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.478+0000 7f4ef212e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd124a0 0x561b6cd12880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:47732/0 (socket says 172.21.15.33:47732) 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.478+0000 7f4ef212e640 1 -- 172.21.15.33:0/2020948112 learned_addr learned my addr 172.21.15.33:0/2020948112 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.478+0000 7f4ef192d640 1 --2- 172.21.15.33:0/2020948112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd12dc0 0x561b6cd1d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.478+0000 7f4ef112c640 1 --2- 172.21.15.33:0/2020948112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd1d680 0x561b6cd1fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.478+0000 7f4ee2ffd640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4ed80a32c0 0x7f4ed80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.478+0000 7f4ef192d640 1 -- 172.21.15.33:0/2020948112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd124a0 msgr2=0x561b6cd12880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.478+0000 7f4ef192d640 1 --2- 172.21.15.33:0/2020948112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd124a0 0x561b6cd12880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.478+0000 7f4ef192d640 1 -- 172.21.15.33:0/2020948112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd1d680 msgr2=0x561b6cd1fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.478+0000 7f4ef192d640 1 --2- 172.21.15.33:0/2020948112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd1d680 0x561b6cd1fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.478+0000 7f4ef192d640 1 -- 172.21.15.33:0/2020948112 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b6cc04440 con 0x561b6cd12dc0 2024-09-16T00:42:51.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.478+0000 7f4ee2ffd640 1 -- 172.21.15.33:0/1200373245 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4edc0f0340 con 0x7f4eec102790 2024-09-16T00:42:51.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.479+0000 7f4ef192d640 1 --2- 172.21.15.33:0/2020948112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd12dc0 0x561b6cd1d140 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8011510 tx=0x7f4ee80119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=545596d82cef7f8a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:51.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.479+0000 7f4ef212e640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4ed80a32c0 0x7f4ed80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.479+0000 7f4ed7fff640 1 -- 172.21.15.33:0/2020948112 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee801e070 con 0x561b6cd12dc0 2024-09-16T00:42:51.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.479+0000 7f4ed7fff640 1 -- 172.21.15.33:0/2020948112 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee800faa0 con 0x561b6cd12dc0 2024-09-16T00:42:51.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.479+0000 7f4ed7fff640 1 -- 172.21.15.33:0/2020948112 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8019bd0 con 0x561b6cd12dc0 2024-09-16T00:42:51.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.480+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2020948112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd12dc0 msgr2=0x561b6cd1d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:51.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.480+0000 7f4ef212e640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4ed80a32c0 0x7f4ed80a5780 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4005f40 tx=0x7f4ee4004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:51.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.480+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/2020948112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd12dc0 0x561b6cd1d140 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8011510 tx=0x7f4ee80119d0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.480+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2020948112 shutdown_connections 2024-09-16T00:42:51.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.480+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/2020948112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd1d680 0x561b6cd1fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.480+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/2020948112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd12dc0 0x561b6cd1d140 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.480+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/2020948112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd124a0 0x561b6cd12880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.480+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2020948112 >> 172.21.15.33:0/2020948112 conn(0x561b6cd0ea30 msgr2=0x561b6cd0ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:51.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.480+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2020948112 shutdown_connections 2024-09-16T00:42:51.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.480+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2020948112 wait complete. 2024-09-16T00:42:51.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef5f6a740 1 -- start start 2024-09-16T00:42:51.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef5f6a740 1 Processor -- start 2024-09-16T00:42:51.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef5f6a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd124a0 0x561b6cd1cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef5f6a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd12dc0 0x561b6cd1a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef5f6a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd1d680 0x561b6cd1a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef5f6a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561b6cd24660 con 0x561b6cd12dc0 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef5f6a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x561b6cc33050 con 0x561b6cd1d680 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef5f6a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561b6cd247e0 con 0x561b6cd124a0 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef5f6a740 10 client.0 _ll_register_callbacks cb 0x561b6cc73160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef212e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd1d680 0x561b6cd1a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef192d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd124a0 0x561b6cd1cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef112c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd12dc0 0x561b6cd1a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef192d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd124a0 0x561b6cd1cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:35670/0 (socket says 172.21.15.33:35670) 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef192d640 1 -- 172.21.15.33:0/2089768445 learned_addr learned my addr 172.21.15.33:0/2089768445 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.481+0000 7f4ef212e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd1d680 0x561b6cd1a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:47738/0 (socket says 172.21.15.33:47738) 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.482+0000 7f4ef192d640 1 -- 172.21.15.33:0/2089768445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd1d680 msgr2=0x561b6cd1a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.482+0000 7f4ef192d640 1 --2- 172.21.15.33:0/2089768445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd1d680 0x561b6cd1a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.482+0000 7f4ef192d640 1 -- 172.21.15.33:0/2089768445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd12dc0 msgr2=0x561b6cd1a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.482+0000 7f4ef192d640 1 --2- 172.21.15.33:0/2089768445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd12dc0 0x561b6cd1a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.482+0000 7f4ef192d640 1 -- 172.21.15.33:0/2089768445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b6c9e93c0 con 0x561b6cd124a0 2024-09-16T00:42:51.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.482+0000 7f4ef112c640 1 --2- 172.21.15.33:0/2089768445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd12dc0 0x561b6cd1a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:51.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.482+0000 7f4ef192d640 1 --2- 172.21.15.33:0/2089768445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd124a0 0x561b6cd1cf90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80097e0 tx=0x7f4ee800f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc956a487897a7ae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:51.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.482+0000 7f4ed57fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:42:51.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.482+0000 7f4ed57fa640 1 -- 172.21.15.33:0/2089768445 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee801e040 con 0x561b6cd124a0 2024-09-16T00:42:51.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.482+0000 7f4ed57fa640 1 -- 172.21.15.33:0/2089768445 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4ee8019950 con 0x561b6cd124a0 2024-09-16T00:42:51.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.483+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2089768445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ee800faa0 con 0x561b6cd124a0 2024-09-16T00:42:51.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.483+0000 7f4ed57fa640 1 -- 172.21.15.33:0/2089768445 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4ee8032050 con 0x561b6cd124a0 2024-09-16T00:42:51.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.483+0000 7f4ef5f6a740 10 client.29239 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:51.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.483+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2089768445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561b6cd2ffe0 con 0x561b6cd124a0 2024-09-16T00:42:51.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.483+0000 7f4ed57fa640 1 -- 172.21.15.33:0/2089768445 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee800faa0 con 0x561b6cd124a0 2024-09-16T00:42:51.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.483+0000 7f4ed57fa640 1 -- 172.21.15.33:0/2089768445 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f4ee8021030 con 0x561b6cd124a0 2024-09-16T00:42:51.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.483+0000 7f4ef5f6a740 10 client.29239 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:51.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.483+0000 7f4ef5f6a740 10 client.29239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:42:51.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.483+0000 7f4ef5f6a740 20 client.29239 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:51.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.483+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/2089768445 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561b6cd31440 0x561b6cd33830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.483+0000 7f4ef5f6a740 4 client.29239 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:42:51.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.483+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2089768445 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b6cc85850 con 0x561b6cd31440 2024-09-16T00:42:51.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.484+0000 7f4ef112c640 1 --2- 172.21.15.33:0/2089768445 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561b6cd31440 0x561b6cd33830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.484+0000 7f4ed57fa640 10 client.29239 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:51.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.484+0000 7f4ef112c640 1 --2- 172.21.15.33:0/2089768445 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561b6cd31440 0x561b6cd33830 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=145aee0c6a198ba5 server_cookie=b516e873919512d4 in_seq=0 out_seq=0 2024-09-16T00:42:51.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.487+0000 7f4ed57fa640 1 -- 172.21.15.33:0/2089768445 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561b6cc85850 con 0x561b6cd31440 2024-09-16T00:42:51.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.487+0000 7f4ed57fa640 10 client.29239 handle_command_reply: tid=0 2024-09-16T00:42:51.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.487+0000 7f4ef5f6a740 1 client.29239 shutdown 2024-09-16T00:42:51.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.487+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2089768445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd124a0 msgr2=0x561b6cd1cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:51.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.487+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/2089768445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd124a0 0x561b6cd1cf90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80097e0 tx=0x7f4ee800f7f0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.487+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2089768445 shutdown_connections 2024-09-16T00:42:51.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.487+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/2089768445 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561b6cd31440 0x561b6cd33830 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.487+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/2089768445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd1d680 0x561b6cd1a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.487+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/2089768445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd12dc0 0x561b6cd1a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.487+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/2089768445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd124a0 0x561b6cd1cf90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.487+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2089768445 >> 172.21.15.33:0/2089768445 conn(0x561b6cd0ea30 msgr2=0x561b6cd14710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:51.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.488+0000 7f4ed57fa640 0 client.29239 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:51.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.488+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2089768445 shutdown_connections 2024-09-16T00:42:51.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.488+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/2089768445 wait complete. 2024-09-16T00:42:51.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.488+0000 7f4ef5f6a740 20 client.29239 trim_cache size 0 max 16384 2024-09-16T00:42:51.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.508+0000 7f4ef5f6a740 1 Processor -- start 2024-09-16T00:42:51.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.508+0000 7f4ef5f6a740 1 -- start start 2024-09-16T00:42:51.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef5f6a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd124a0 0x561b6cd2d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef5f6a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd12dc0 0x561b6cd38fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef5f6a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd1d680 0x561b6cd39500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef5f6a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec1af7d0 con 0x561b6cd124a0 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef5f6a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec1ad0a0 con 0x561b6cd12dc0 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef5f6a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec1aaef0 con 0x561b6cd1d680 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef192d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd124a0 0x561b6cd2d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef192d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd124a0 0x561b6cd2d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56506/0 (socket says 172.21.15.33:56506) 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef192d640 1 -- 172.21.15.33:0/1998067345 learned_addr learned my addr 172.21.15.33:0/1998067345 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef112c640 1 --2- 172.21.15.33:0/1998067345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd12dc0 0x561b6cd38fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef212e640 1 --2- 172.21.15.33:0/1998067345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd1d680 0x561b6cd39500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef192d640 1 -- 172.21.15.33:0/1998067345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd1d680 msgr2=0x561b6cd39500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef192d640 1 --2- 172.21.15.33:0/1998067345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd1d680 0x561b6cd39500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef192d640 1 -- 172.21.15.33:0/1998067345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd12dc0 msgr2=0x561b6cd38fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:51.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef192d640 1 --2- 172.21.15.33:0/1998067345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd12dc0 0x561b6cd38fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef192d640 1 -- 172.21.15.33:0/1998067345 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b6cd0e4d0 con 0x561b6cd124a0 2024-09-16T00:42:51.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ef192d640 1 --2- 172.21.15.33:0/1998067345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd124a0 0x561b6cd2d8c0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8000c00 tx=0x7f4ee8004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce63aee52b532163 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:51.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.509+0000 7f4ed77fe640 1 -- 172.21.15.33:0/1998067345 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee801e040 con 0x561b6cd124a0 2024-09-16T00:42:51.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.510+0000 7f4ed77fe640 1 -- 172.21.15.33:0/1998067345 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee8016020 con 0x561b6cd124a0 2024-09-16T00:42:51.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.510+0000 7f4ed77fe640 1 -- 172.21.15.33:0/1998067345 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8005040 con 0x561b6cd124a0 2024-09-16T00:42:51.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.510+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/1998067345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd124a0 msgr2=0x561b6cd2d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:51.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.510+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/1998067345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd124a0 0x561b6cd2d8c0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8000c00 tx=0x7f4ee8004a80 comp rx=0 tx=0).stop 2024-09-16T00:42:51.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.510+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/1998067345 shutdown_connections 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.510+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/1998067345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd1d680 0x561b6cd39500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.511+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/1998067345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd12dc0 0x561b6cd38fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.511+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/1998067345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd124a0 0x561b6cd2d8c0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.511+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/1998067345 >> 172.21.15.33:0/1998067345 conn(0x561b6cd0ea30 msgr2=0x561b6cd18b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.511+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/1998067345 shutdown_connections 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.511+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/1998067345 wait complete. 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.511+0000 7f4ef5f6a740 1 -- start start 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.511+0000 7f4ef5f6a740 1 Processor -- start 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.512+0000 7f4ef5f6a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd124a0 0x561b6cd26a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.512+0000 7f4ef5f6a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd12dc0 0x561b6cd26fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.512+0000 7f4ef5f6a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd1d680 0x561b6cd27510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.512+0000 7f4ef5f6a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec1ab560 con 0x561b6cd1d680 2024-09-16T00:42:51.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.512+0000 7f4ef5f6a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec1ac9d0 con 0x561b6cd124a0 2024-09-16T00:42:51.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.512+0000 7f4ef5f6a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec1abc30 con 0x561b6cd12dc0 2024-09-16T00:42:51.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.512+0000 7f4ef212e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd1d680 0x561b6cd27510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.512+0000 7f4ef212e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd1d680 0x561b6cd27510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56522/0 (socket says 172.21.15.33:56522) 2024-09-16T00:42:51.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.512+0000 7f4ef212e640 1 -- 172.21.15.33:0/3595834827 learned_addr learned my addr 172.21.15.33:0/3595834827 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:51.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.512+0000 7f4ef5f6a740 10 client.0 _ll_register_callbacks cb 0x7f4ee801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:51.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.512+0000 7f4ef192d640 1 --2- 172.21.15.33:0/3595834827 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd124a0 0x561b6cd26a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.512+0000 7f4ef212e640 1 -- 172.21.15.33:0/3595834827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd12dc0 msgr2=0x561b6cd26fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:51.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.513+0000 7f4ef212e640 1 --2- 172.21.15.33:0/3595834827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd12dc0 0x561b6cd26fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.513+0000 7f4ef212e640 1 -- 172.21.15.33:0/3595834827 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd124a0 msgr2=0x561b6cd26a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:51.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.513+0000 7f4ef212e640 1 --2- 172.21.15.33:0/3595834827 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd124a0 0x561b6cd26a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.513+0000 7f4ef212e640 1 -- 172.21.15.33:0/3595834827 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b6cd9a230 con 0x561b6cd1d680 2024-09-16T00:42:51.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.513+0000 7f4ef212e640 1 --2- 172.21.15.33:0/3595834827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd1d680 0x561b6cd27510 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4009fd0 tx=0x7f4ee4009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f479aaa16d7fc417 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:51.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.513+0000 7f4ed57fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:51.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.513+0000 7f4ed57fa640 1 -- 172.21.15.33:0/3595834827 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4022b20 con 0x561b6cd1d680 2024-09-16T00:42:51.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.513+0000 7f4ed57fa640 1 -- 172.21.15.33:0/3595834827 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f4ee4022cc0 con 0x561b6cd1d680 2024-09-16T00:42:51.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.513+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/3595834827 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ee8016020 con 0x561b6cd1d680 2024-09-16T00:42:51.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.514+0000 7f4ed57fa640 1 -- 172.21.15.33:0/3595834827 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4ee4012440 con 0x561b6cd1d680 2024-09-16T00:42:51.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.514+0000 7f4ef5f6a740 10 client.20157 fetch_fsmap learned FSMap version 33 2024-09-16T00:42:51.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.514+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/3595834827 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561b6cd9b560 con 0x561b6cd1d680 2024-09-16T00:42:51.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.514+0000 7f4ed57fa640 1 -- 172.21.15.33:0/3595834827 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee8016020 con 0x561b6cd1d680 2024-09-16T00:42:51.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.514+0000 7f4ed57fa640 1 -- 172.21.15.33:0/3595834827 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f4ee400f040 con 0x561b6cd1d680 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.514+0000 7f4ef5f6a740 10 client.20157 fetch_fsmap finished waiting for FSMap version 33 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.515+0000 7f4ef5f6a740 10 client.20157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.515+0000 7f4ef5f6a740 20 client.20157 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.515+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/3595834827 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561b6cd9bc40 0x561b6cd9e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.515+0000 7f4ef5f6a740 4 client.20157 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.515+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/3595834827 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561b6cd1ab10 con 0x561b6cd9bc40 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.515+0000 7f4ef192d640 1 --2- 172.21.15.33:0/3595834827 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561b6cd9bc40 0x561b6cd9e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ed57fa640 10 client.20157 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ef192d640 1 --2- 172.21.15.33:0/3595834827 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561b6cd9bc40 0x561b6cd9e0c0 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf80c67e40ceedb1 server_cookie=6adce0eceaf4015e in_seq=0 out_seq=0 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ed57fa640 1 -- 172.21.15.33:0/3595834827 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561b6cd1ab10 con 0x561b6cd9bc40 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ed57fa640 10 client.20157 handle_command_reply: tid=0 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ef5f6a740 1 client.20157 shutdown 2024-09-16T00:42:51.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/3595834827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd1d680 msgr2=0x561b6cd27510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:51.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/3595834827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd1d680 0x561b6cd27510 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4009fd0 tx=0x7f4ee4009600 comp rx=0 tx=0).stop 2024-09-16T00:42:51.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/3595834827 shutdown_connections 2024-09-16T00:42:51.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/3595834827 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x561b6cd9bc40 0x561b6cd9e0c0 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/3595834827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b6cd1d680 0x561b6cd27510 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/3595834827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561b6cd12dc0 0x561b6cd26fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ef5f6a740 1 --2- 172.21.15.33:0/3595834827 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561b6cd124a0 0x561b6cd26a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.516+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/3595834827 >> 172.21.15.33:0/3595834827 conn(0x561b6cd0ea30 msgr2=0x561b6cd0d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:51.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.517+0000 7f4ed57fa640 0 client.20157 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:42:51.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.517+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/3595834827 shutdown_connections 2024-09-16T00:42:51.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.517+0000 7f4ef5f6a740 1 -- 172.21.15.33:0/3595834827 wait complete. 2024-09-16T00:42:51.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.517+0000 7f4ef5f6a740 20 client.20157 trim_cache size 0 max 16384 2024-09-16T00:42:51.521 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:42:51.521 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-09-16T00:42:51.521 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-09-16T00:42:51.521 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:42:51.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.518+0000 7f4ef3b91640 1 -- 172.21.15.33:0/1200373245 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4ed80a32c0 msgr2=0x7f4ed80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:51.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.518+0000 7f4ef3b91640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4ed80a32c0 0x7f4ed80a5780 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4005f40 tx=0x7f4ee4004ab0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.518+0000 7f4ef3b91640 1 -- 172.21.15.33:0/1200373245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4eec102790 msgr2=0x7f4eec1989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:51.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.518+0000 7f4ef3b91640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4eec102790 0x7f4eec1989a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f4edc02a930 tx=0x7f4edc003b90 comp rx=0 tx=0).stop 2024-09-16T00:42:51.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.518+0000 7f4ef3b91640 1 -- 172.21.15.33:0/1200373245 shutdown_connections 2024-09-16T00:42:51.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.518+0000 7f4ef3b91640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f4ed80a32c0 0x7f4ed80a5780 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.518+0000 7f4ef3b91640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eec106f10 0x7f4eec19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.518+0000 7f4ef3b91640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4eec102790 0x7f4eec1989a0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.518+0000 7f4ef3b91640 1 --2- 172.21.15.33:0/1200373245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eec101de0 0x7f4eec198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:51.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.518+0000 7f4ef3b91640 1 -- 172.21.15.33:0/1200373245 >> 172.21.15.33:0/1200373245 conn(0x7f4eec0fd8e0 msgr2=0x7f4eec0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:51.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.519+0000 7f4ef3b91640 1 -- 172.21.15.33:0/1200373245 shutdown_connections 2024-09-16T00:42:51.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:51.519+0000 7f4ef3b91640 1 -- 172.21.15.33:0/1200373245 wait complete. 2024-09-16T00:42:51.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:51 smithi033 ceph-mon[31243]: pgmap v1121: 129 pgs: 129 active+clean; 717 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 8.3 MiB/s wr, 487 op/s 2024-09-16T00:42:51.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:51 smithi083 ceph-mon[35242]: pgmap v1121: 129 pgs: 129 active+clean; 717 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 8.3 MiB/s wr, 487 op/s 2024-09-16T00:42:51.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:51 smithi060 ceph-mon[35185]: pgmap v1121: 129 pgs: 129 active+clean; 717 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 8.3 MiB/s wr, 487 op/s 2024-09-16T00:42:52.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 237560 17.46 MB/sec execute 598 sec latency 270.924 ms 2024-09-16T00:42:53.109 INFO:tasks.workunit.client.0.smithi033.stdout: 10 237758 17.44 MB/sec execute 599 sec latency 910.876 ms 2024-09-16T00:42:53.536 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T00:42:53.699 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:42:53.701 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:53 smithi033 ceph-mon[31243]: pgmap v1122: 129 pgs: 129 active+clean; 716 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.7 MiB/s wr, 438 op/s 2024-09-16T00:42:53.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:53 smithi083 ceph-mon[35242]: pgmap v1122: 129 pgs: 129 active+clean; 716 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.7 MiB/s wr, 438 op/s 2024-09-16T00:42:53.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:53 smithi060 ceph-mon[35185]: pgmap v1122: 129 pgs: 129 active+clean; 716 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.7 MiB/s wr, 438 op/s 2024-09-16T00:42:54.113 INFO:tasks.workunit.client.0.smithi033.stdout: 10 cleanup 600 sec 2024-09-16T00:42:54.191 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:54.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.705+0000 7f58aa03e640 1 -- 172.21.15.33:0/3979779623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58a4102570 msgr2=0x7f58a41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:54.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.705+0000 7f58aa03e640 1 --2- 172.21.15.33:0/3979779623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58a4102570 0x7f58a41029b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5898009890 tx=0x7f589802a410 comp rx=0 tx=0).stop 2024-09-16T00:42:54.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.705+0000 7f58aa03e640 1 -- 172.21.15.33:0/3979779623 shutdown_connections 2024-09-16T00:42:54.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.705+0000 7f58aa03e640 1 --2- 172.21.15.33:0/3979779623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58a4102ef0 0x7f58a410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:54.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.705+0000 7f58aa03e640 1 --2- 172.21.15.33:0/3979779623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58a4102570 0x7f58a41029b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:54.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.705+0000 7f58aa03e640 1 --2- 172.21.15.33:0/3979779623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58a4100e20 0x7f58a4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:54.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.705+0000 7f58aa03e640 1 -- 172.21.15.33:0/3979779623 >> 172.21.15.33:0/3979779623 conn(0x7f58a40fc7d0 msgr2=0x7f58a40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.705+0000 7f58aa03e640 1 -- 172.21.15.33:0/3979779623 shutdown_connections 2024-09-16T00:42:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.705+0000 7f58aa03e640 1 -- 172.21.15.33:0/3979779623 wait complete. 2024-09-16T00:42:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.706+0000 7f58aa03e640 1 Processor -- start 2024-09-16T00:42:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.706+0000 7f58aa03e640 1 -- start start 2024-09-16T00:42:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58aa03e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58a4100e20 0x7f58a4198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58aa03e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58a4102570 0x7f58a4198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58aa03e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58a4102ef0 0x7f58a419ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58aa03e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58a406a540 con 0x7f58a4100e20 2024-09-16T00:42:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58aa03e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f58a406a3c0 con 0x7f58a4102ef0 2024-09-16T00:42:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58aa03e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f58a406a6c0 con 0x7f58a4102570 2024-09-16T00:42:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58a37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58a4100e20 0x7f58a4198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58a37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58a4100e20 0x7f58a4198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56534/0 (socket says 172.21.15.33:56534) 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58a2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58a4102570 0x7f58a4198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58a37fe640 1 -- 172.21.15.33:0/1850588597 learned_addr learned my addr 172.21.15.33:0/1850588597 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58a3fff640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58a4102ef0 0x7f58a419ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58a37fe640 1 -- 172.21.15.33:0/1850588597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58a4102570 msgr2=0x7f58a4198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58a37fe640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58a4102570 0x7f58a4198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58a37fe640 1 -- 172.21.15.33:0/1850588597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58a4102ef0 msgr2=0x7f58a419ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58a37fe640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58a4102ef0 0x7f58a419ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58a37fe640 1 -- 172.21.15.33:0/1850588597 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a419d520 con 0x7f58a4100e20 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58a2ffd640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58a4102570 0x7f58a4198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.707+0000 7f58a37fe640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58a4100e20 0x7f58a4198570 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f589400e9f0 tx=0x7f589400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.708+0000 7f58a0ff9640 1 -- 172.21.15.33:0/1850588597 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589400ccc0 con 0x7f58a4100e20 2024-09-16T00:42:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.708+0000 7f58aa03e640 1 -- 172.21.15.33:0/1850588597 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a419d810 con 0x7f58a4100e20 2024-09-16T00:42:54.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.708+0000 7f58a0ff9640 1 -- 172.21.15.33:0/1850588597 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5894005500 con 0x7f58a4100e20 2024-09-16T00:42:54.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.708+0000 7f58a0ff9640 1 -- 172.21.15.33:0/1850588597 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5894010690 con 0x7f58a4100e20 2024-09-16T00:42:54.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.708+0000 7f58aa03e640 1 -- 172.21.15.33:0/1850588597 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a41a50f0 con 0x7f58a4100e20 2024-09-16T00:42:54.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.709+0000 7f58aa03e640 1 -- 172.21.15.33:0/1850588597 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a4101200 con 0x7f58a4100e20 2024-09-16T00:42:54.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.710+0000 7f58a0ff9640 1 -- 172.21.15.33:0/1850588597 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f58940046c0 con 0x7f58a4100e20 2024-09-16T00:42:54.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.711+0000 7f58a0ff9640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f588c0a3240 0x7f588c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:54.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.711+0000 7f58a2ffd640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f588c0a3240 0x7f588c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:54.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.711+0000 7f58a0ff9640 1 -- 172.21.15.33:0/1850588597 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f58940d0220 con 0x7f58a4100e20 2024-09-16T00:42:54.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.711+0000 7f58a2ffd640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f588c0a3240 0x7f588c0a5700 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5898031000 tx=0x7f58980094b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:54.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.714+0000 7f58a0ff9640 1 -- 172.21.15.33:0/1850588597 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f58940541b0 con 0x7f58a4100e20 2024-09-16T00:42:54.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.976+0000 7f58aa03e640 1 -- 172.21.15.33:0/1850588597 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58a41998d0 con 0x7f58a4100e20 2024-09-16T00:42:54.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.983+0000 7f58a0ff9640 1 -- 172.21.15.33:0/1850588597 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11903 (secure 0 0 0) 0x7f58a41998d0 con 0x7f58a4100e20 2024-09-16T00:42:54.984 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:42:54.984 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":33,"btime":"2024-09-16T00:33:38:068030+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:33:37.073163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:42:54.985 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 33 2024-09-16T00:42:54.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.989+0000 7f588a7fc640 1 -- 172.21.15.33:0/1850588597 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f588c0a3240 msgr2=0x7f588c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:54.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.989+0000 7f588a7fc640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f588c0a3240 0x7f588c0a5700 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5898031000 tx=0x7f58980094b0 comp rx=0 tx=0).stop 2024-09-16T00:42:54.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.989+0000 7f588a7fc640 1 -- 172.21.15.33:0/1850588597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58a4100e20 msgr2=0x7f58a4198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:54.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.989+0000 7f588a7fc640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58a4100e20 0x7f58a4198570 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f589400e9f0 tx=0x7f589400eeb0 comp rx=0 tx=0).stop 2024-09-16T00:42:54.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.989+0000 7f588a7fc640 1 -- 172.21.15.33:0/1850588597 shutdown_connections 2024-09-16T00:42:54.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.989+0000 7f588a7fc640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f588c0a3240 0x7f588c0a5700 secure :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5898031000 tx=0x7f58980094b0 comp rx=0 tx=0).stop 2024-09-16T00:42:54.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.989+0000 7f588a7fc640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58a4102ef0 0x7f58a419ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:54.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.989+0000 7f588a7fc640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58a4102570 0x7f58a4198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:54.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.989+0000 7f588a7fc640 1 --2- 172.21.15.33:0/1850588597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58a4100e20 0x7f58a4198570 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:54.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.989+0000 7f588a7fc640 1 -- 172.21.15.33:0/1850588597 >> 172.21.15.33:0/1850588597 conn(0x7f58a40fc7d0 msgr2=0x7f58a4106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:54.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.991+0000 7f588a7fc640 1 -- 172.21.15.33:0/1850588597 shutdown_connections 2024-09-16T00:42:54.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:54.992+0000 7f588a7fc640 1 -- 172.21.15.33:0/1850588597 wait complete. 2024-09-16T00:42:55.113 INFO:tasks.workunit.client.0.smithi033.stdout: 8 cleanup 601 sec 2024-09-16T00:42:55.442 INFO:tasks.workunit.client.0.smithi033.stdout: 0 cleanup 601 sec 2024-09-16T00:42:55.443 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-09-16T00:42:55.443 INFO:tasks.workunit.client.0.smithi033.stdout: Operation Count AvgLat MaxLat 2024-09-16T00:42:55.443 INFO:tasks.workunit.client.0.smithi033.stdout: ---------------------------------------- 2024-09-16T00:42:55.443 INFO:tasks.workunit.client.0.smithi033.stdout: NTCreateX 335711 7.429 33717.969 2024-09-16T00:42:55.443 INFO:tasks.workunit.client.0.smithi033.stdout: Close 246493 0.003 0.289 2024-09-16T00:42:55.443 INFO:tasks.workunit.client.0.smithi033.stdout: Rename 14232 41.411 32672.728 2024-09-16T00:42:55.443 INFO:tasks.workunit.client.0.smithi033.stdout: Unlink 67884 15.334 33453.958 2024-09-16T00:42:55.443 INFO:tasks.workunit.client.0.smithi033.stdout: Qpathinfo 304624 3.627 2484.410 2024-09-16T00:42:55.443 INFO:tasks.workunit.client.0.smithi033.stdout: Qfileinfo 53129 0.001 0.079 2024-09-16T00:42:55.444 INFO:tasks.workunit.client.0.smithi033.stdout: Qfsinfo 55855 0.220 47.547 2024-09-16T00:42:55.444 INFO:tasks.workunit.client.0.smithi033.stdout: Sfileinfo 27328 7.846 33623.680 2024-09-16T00:42:55.444 INFO:tasks.workunit.client.0.smithi033.stdout: Find 117725 0.465 679.303 2024-09-16T00:42:55.444 INFO:tasks.workunit.client.0.smithi033.stdout: WriteX 165922 1.431 33526.940 2024-09-16T00:42:55.444 INFO:tasks.workunit.client.0.smithi033.stdout: ReadX 527320 0.072 1281.005 2024-09-16T00:42:55.444 INFO:tasks.workunit.client.0.smithi033.stdout: LockX 1098 5.271 84.736 2024-09-16T00:42:55.444 INFO:tasks.workunit.client.0.smithi033.stdout: UnlockX 1098 4.752 210.833 2024-09-16T00:42:55.444 INFO:tasks.workunit.client.0.smithi033.stdout: Flush 23531 14.112 2129.602 2024-09-16T00:42:55.444 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-09-16T00:42:55.444 INFO:tasks.workunit.client.0.smithi033.stdout:Throughput 17.4386 MB/sec 10 clients 10 procs max_latency=33717.972 ms 2024-09-16T00:42:55.445 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T00:42:55.445 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T00:42:55.692 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:55 smithi033 ceph-mon[31243]: pgmap v1123: 129 pgs: 129 active+clean; 723 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 8.8 MiB/s wr, 512 op/s 2024-09-16T00:42:55.692 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:55 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1850588597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:42:55.746 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-16T00:42:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:55 smithi083 ceph-mon[35242]: pgmap v1123: 129 pgs: 129 active+clean; 723 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 8.8 MiB/s wr, 512 op/s 2024-09-16T00:42:55.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:55 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1850588597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:42:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:55 smithi060 ceph-mon[35185]: pgmap v1123: 129 pgs: 129 active+clean; 723 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 8.8 MiB/s wr, 512 op/s 2024-09-16T00:42:55.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:55 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1850588597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:42:56.186 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.773+0000 7f37c12b9640 1 -- 172.21.15.33:0/2633295622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37bc110190 msgr2=0x7f37bc06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.773+0000 7f37c12b9640 1 --2- 172.21.15.33:0/2633295622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37bc110190 0x7f37bc06e4f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f37ac01a910 tx=0x7f37ac03b470 comp rx=0 tx=0).stop 2024-09-16T00:42:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.774+0000 7f37c12b9640 1 -- 172.21.15.33:0/2633295622 shutdown_connections 2024-09-16T00:42:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.774+0000 7f37c12b9640 1 --2- 172.21.15.33:0/2633295622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37bc11b140 0x7f37bc11d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.774+0000 7f37c12b9640 1 --2- 172.21.15.33:0/2633295622 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37bc06ea30 0x7f37bc06ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.774+0000 7f37c12b9640 1 --2- 172.21.15.33:0/2633295622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37bc110190 0x7f37bc06e4f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.774+0000 7f37c12b9640 1 -- 172.21.15.33:0/2633295622 >> 172.21.15.33:0/2633295622 conn(0x7f37bc073180 msgr2=0x7f37bc073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.774+0000 7f37c12b9640 1 -- 172.21.15.33:0/2633295622 shutdown_connections 2024-09-16T00:42:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.774+0000 7f37c12b9640 1 -- 172.21.15.33:0/2633295622 wait complete. 2024-09-16T00:42:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.775+0000 7f37c12b9640 1 Processor -- start 2024-09-16T00:42:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.775+0000 7f37c12b9640 1 -- start start 2024-09-16T00:42:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.776+0000 7f37c12b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37bc06ea30 0x7f37bc11aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.776+0000 7f37c12b9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37bc110190 0x7f37bc116ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.776+0000 7f37c12b9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37bc11b140 0x7f37bc116ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.776+0000 7f37c12b9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc113400 con 0x7f37bc11b140 2024-09-16T00:42:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.776+0000 7f37c12b9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc113280 con 0x7f37bc110190 2024-09-16T00:42:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.776+0000 7f37c12b9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc113580 con 0x7f37bc06ea30 2024-09-16T00:42:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.776+0000 7f37bb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37bc110190 0x7f37bc116ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.776+0000 7f37bb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37bc110190 0x7f37bc116ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.33:47794/0 (socket says 172.21.15.33:47794) 2024-09-16T00:42:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.776+0000 7f37bb7fe640 1 -- 172.21.15.33:0/1206887101 learned_addr learned my addr 172.21.15.33:0/1206887101 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.778+0000 7f37bb7fe640 1 -- 172.21.15.33:0/1206887101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37bc06ea30 msgr2=0x7f37bc11aa10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T00:42:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.778+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/1206887101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37bc06ea30 0x7f37bc11aa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.778+0000 7f37bb7fe640 1 -- 172.21.15.33:0/1206887101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37bc11b140 msgr2=0x7f37bc116ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.778+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/1206887101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37bc11b140 0x7f37bc116ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.778+0000 7f37bb7fe640 1 -- 172.21.15.33:0/1206887101 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37bc117880 con 0x7f37bc110190 2024-09-16T00:42:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.779+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/1206887101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37bc110190 0x7f37bc116ab0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f37a400ba50 tx=0x7f37a400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.779+0000 7f37b97fa640 1 -- 172.21.15.33:0/1206887101 <== mon.2 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a4019070 con 0x7f37bc110190 2024-09-16T00:42:56.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.779+0000 7f37b97fa640 1 -- 172.21.15.33:0/1206887101 <== mon.2 v2:172.21.15.60:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37a400cbe0 con 0x7f37bc110190 2024-09-16T00:42:56.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.779+0000 7f37c12b9640 1 -- 172.21.15.33:0/1206887101 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37bc1aafc0 con 0x7f37bc110190 2024-09-16T00:42:56.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.779+0000 7f37b97fa640 1 -- 172.21.15.33:0/1206887101 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a4007680 con 0x7f37bc110190 2024-09-16T00:42:56.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.779+0000 7f37c12b9640 1 -- 172.21.15.33:0/1206887101 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37bc1ab410 con 0x7f37bc110190 2024-09-16T00:42:56.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.786+0000 7f37b97fa640 1 -- 172.21.15.33:0/1206887101 <== mon.2 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f37a401c020 con 0x7f37bc110190 2024-09-16T00:42:56.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.786+0000 7f37c48f4740 1 Processor -- start 2024-09-16T00:42:56.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.786+0000 7f37c48f4740 1 -- start start 2024-09-16T00:42:56.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c48f4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ebf9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c48f4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebf9780 0x563c7ebf9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c48f4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec06910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c48f4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563c7eb5f2f0 con 0x563c7ebf9780 2024-09-16T00:42:56.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c48f4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x563c7eb06cf0 con 0x563c7ec04520 2024-09-16T00:42:56.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c48f4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563c7eb5b2b0 con 0x563c7ebfacf0 2024-09-16T00:42:56.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c0ab8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ebf9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c0ab8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ebf9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:35716/0 (socket says 172.21.15.33:35716) 2024-09-16T00:42:56.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c0ab8640 1 -- 172.21.15.33:0/2405592485 learned_addr learned my addr 172.21.15.33:0/2405592485 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:56.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/2405592485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec06910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37bbfff640 1 --2- 172.21.15.33:0/2405592485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebf9780 0x563c7ebf9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c0ab8640 1 -- 172.21.15.33:0/2405592485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 msgr2=0x563c7ec06910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:56.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c0ab8640 1 --2- 172.21.15.33:0/2405592485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec06910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c0ab8640 1 -- 172.21.15.33:0/2405592485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebf9780 msgr2=0x563c7ebf9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:56.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c0ab8640 1 --2- 172.21.15.33:0/2405592485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebf9780 0x563c7ebf9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37c0ab8640 1 -- 172.21.15.33:0/2405592485 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c7eaeb3a0 con 0x563c7ebfacf0 2024-09-16T00:42:56.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.787+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/2405592485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec06910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:56.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.788+0000 7f37c0ab8640 1 --2- 172.21.15.33:0/2405592485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ebf9240 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f37b4009750 tx=0x7f37b4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad98476a9a92e38a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:56.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.788+0000 7f37b27fc640 1 -- 172.21.15.33:0/2405592485 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4015020 con 0x563c7ebfacf0 2024-09-16T00:42:56.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.788+0000 7f37b27fc640 1 -- 172.21.15.33:0/2405592485 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37b400bda0 con 0x563c7ebfacf0 2024-09-16T00:42:56.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.788+0000 7f37b27fc640 1 -- 172.21.15.33:0/2405592485 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4010380 con 0x563c7ebfacf0 2024-09-16T00:42:56.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.788+0000 7f37b97fa640 1 --2- 172.21.15.33:0/1206887101 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f379c0a3300 0x7f379c0a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.789+0000 7f37b97fa640 1 -- 172.21.15.33:0/1206887101 <== mon.2 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f37a40cf550 con 0x7f37bc110190 2024-09-16T00:42:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.789+0000 7f37bbfff640 1 --2- 172.21.15.33:0/1206887101 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f379c0a3300 0x7f379c0a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.789+0000 7f37c48f4740 1 -- 172.21.15.33:0/2405592485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 msgr2=0x563c7ebf9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.789+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2405592485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ebf9240 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f37b4009750 tx=0x7f37b4009b50 comp rx=0 tx=0).stop 2024-09-16T00:42:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.789+0000 7f37c48f4740 1 -- 172.21.15.33:0/2405592485 shutdown_connections 2024-09-16T00:42:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.789+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2405592485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec06910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.789+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2405592485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebf9780 0x563c7ebf9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.789+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2405592485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ebf9240 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.789+0000 7f37c48f4740 1 -- 172.21.15.33:0/2405592485 >> 172.21.15.33:0/2405592485 conn(0x563c7ebc2310 msgr2=0x563c7ebf5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.789+0000 7f37c48f4740 1 -- 172.21.15.33:0/2405592485 shutdown_connections 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.789+0000 7f37c48f4740 1 -- 172.21.15.33:0/2405592485 wait complete. 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.789+0000 7f37c48f4740 1 -- start start 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37bbfff640 1 --2- 172.21.15.33:0/1206887101 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f379c0a3300 0x7f379c0a57c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f37ac03ba60 tx=0x7f37ac000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37c48f4740 1 Processor -- start 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37c48f4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebf9780 0x563c7ec03dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37c48f4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebfacf0 0x563c7ec01070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37c48f4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec015b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37c48f4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563c7ec07270 con 0x563c7ebfacf0 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37c48f4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x563c7eb1a060 con 0x563c7ec04520 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37c48f4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563c7ec073f0 con 0x563c7ebf9780 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37c48f4740 10 client.0 _ll_register_callbacks cb 0x563c7e86d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebf9780 0x563c7ec03dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37bb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebfacf0 0x563c7ec01070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebf9780 0x563c7ec03dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:35730/0 (socket says 172.21.15.33:35730) 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37bbfff640 1 -- 172.21.15.33:0/2977952280 learned_addr learned my addr 172.21.15.33:0/2977952280 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.790+0000 7f37c0ab8640 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37bb7fe640 1 -- 172.21.15.33:0/2977952280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebf9780 msgr2=0x563c7ec03dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebf9780 0x563c7ec03dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37bb7fe640 1 -- 172.21.15.33:0/2977952280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 msgr2=0x563c7ec015b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec015b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37bb7fe640 1 -- 172.21.15.33:0/2977952280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c7e8d03c0 con 0x563c7ebfacf0 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37c0ab8640 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebfacf0 0x563c7ec01070 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x563c7ebfa220 tx=0x7f37a409b520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef761752a64d213 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37ab7fe640 1 -- 172.21.15.33:0/2977952280 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a409bd90 con 0x563c7ebfacf0 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37ab7fe640 1 -- 172.21.15.33:0/2977952280 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f37a40cf130 con 0x563c7ebfacf0 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37c48f4740 1 -- 172.21.15.33:0/2977952280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37b400bda0 con 0x563c7ebfacf0 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37ab7fe640 1 -- 172.21.15.33:0/2977952280 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f37a40cf330 con 0x563c7ebfacf0 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37c48f4740 10 client.20169 fetch_fsmap learned FSMap version 34 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.791+0000 7f37c48f4740 1 -- 172.21.15.33:0/2977952280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563c7ec12c50 con 0x563c7ebfacf0 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.792+0000 7f37ab7fe640 1 -- 172.21.15.33:0/2977952280 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37b400bda0 con 0x563c7ebfacf0 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.792+0000 7f37ab7fe640 1 -- 172.21.15.33:0/2977952280 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f37a409b980 con 0x563c7ebfacf0 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.792+0000 7f37c48f4740 10 client.20169 fetch_fsmap finished waiting for FSMap version 34 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.792+0000 7f37c48f4740 10 client.20169 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.792+0000 7f37c48f4740 20 client.20169 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.792+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x563c7ec140b0 0x563c7ec164a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.792+0000 7f37c48f4740 4 client.20169 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:42:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.792+0000 7f37c48f4740 1 -- 172.21.15.33:0/2977952280 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c7eb6c860 con 0x563c7ec140b0 2024-09-16T00:42:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.792+0000 7f37bbfff640 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x563c7ec140b0 0x563c7ec164a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.793+0000 7f37ab7fe640 10 client.20169 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:42:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.793+0000 7f37bbfff640 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x563c7ec140b0 0x563c7ec164a0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e644be7d3d5cc577 server_cookie=d3012d0018616aed in_seq=0 out_seq=0 2024-09-16T00:42:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.793+0000 7f37ab7fe640 1 -- 172.21.15.33:0/2977952280 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563c7eb6c860 con 0x563c7ec140b0 2024-09-16T00:42:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.793+0000 7f37ab7fe640 10 client.20169 handle_command_reply: tid=0 2024-09-16T00:42:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.793+0000 7f37c48f4740 1 client.20169 shutdown 2024-09-16T00:42:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.793+0000 7f37c48f4740 1 -- 172.21.15.33:0/2977952280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebfacf0 msgr2=0x563c7ec01070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.793+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebfacf0 0x563c7ec01070 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x563c7ebfa220 tx=0x7f37a409b520 comp rx=0 tx=0).stop 2024-09-16T00:42:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.794+0000 7f37c48f4740 1 -- 172.21.15.33:0/2977952280 shutdown_connections 2024-09-16T00:42:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.794+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x563c7ec140b0 0x563c7ec164a0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.794+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.794+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebfacf0 0x563c7ec01070 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.794+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2977952280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebf9780 0x563c7ec03dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.794+0000 7f37ab7fe640 0 client.20169 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:42:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.794+0000 7f37c48f4740 1 -- 172.21.15.33:0/2977952280 >> 172.21.15.33:0/2977952280 conn(0x563c7ebc2310 msgr2=0x563c7ebfbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.794+0000 7f37c48f4740 1 -- 172.21.15.33:0/2977952280 shutdown_connections 2024-09-16T00:42:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.794+0000 7f37c48f4740 1 -- 172.21.15.33:0/2977952280 wait complete. 2024-09-16T00:42:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.794+0000 7f37c48f4740 20 client.20169 trim_cache size 0 max 16384 2024-09-16T00:42:56.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37c48f4740 1 Processor -- start 2024-09-16T00:42:56.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37c48f4740 1 -- start start 2024-09-16T00:42:56.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37c48f4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ebf9780 0x563c7ec105a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37c48f4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ec00050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37c48f4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ec04520 0x563c7ebfc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37c48f4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1b7ee0 con 0x563c7ec04520 2024-09-16T00:42:56.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37c48f4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1b57b0 con 0x563c7ebf9780 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37c48f4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1b3600 con 0x563c7ebfacf0 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37c0ab8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ec04520 0x563c7ebfc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37bb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ec00050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37bb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ec00050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:35742/0 (socket says 172.21.15.33:35742) 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37bb7fe640 1 -- 172.21.15.33:0/1597766153 learned_addr learned my addr 172.21.15.33:0/1597766153 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.808+0000 7f37bbfff640 1 --2- 172.21.15.33:0/1597766153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ebf9780 0x563c7ec105a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.809+0000 7f37bb7fe640 1 -- 172.21.15.33:0/1597766153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ebf9780 msgr2=0x563c7ec105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.809+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/1597766153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ebf9780 0x563c7ec105a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.809+0000 7f37bb7fe640 1 -- 172.21.15.33:0/1597766153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ec04520 msgr2=0x563c7ebfc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.809+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/1597766153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ec04520 0x563c7ebfc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.809+0000 7f37bb7fe640 1 -- 172.21.15.33:0/1597766153 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c7ebf6350 con 0x563c7ebfacf0 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.809+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/1597766153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ec00050 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f37a409b390 tx=0x7f37a409b590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd78620cdefbce4c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:56.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.809+0000 7f37b1ffb640 1 -- 172.21.15.33:0/1597766153 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a40b8030 con 0x563c7ebfacf0 2024-09-16T00:42:56.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.809+0000 7f37b1ffb640 1 -- 172.21.15.33:0/1597766153 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37a409bd90 con 0x563c7ebfacf0 2024-09-16T00:42:56.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.809+0000 7f37b1ffb640 1 -- 172.21.15.33:0/1597766153 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a40a4300 con 0x563c7ebfacf0 2024-09-16T00:42:56.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.810+0000 7f37c48f4740 1 -- 172.21.15.33:0/1597766153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 msgr2=0x563c7ec00050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:56.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.810+0000 7f37c48f4740 1 --2- 172.21.15.33:0/1597766153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ec00050 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f37a409b390 tx=0x7f37a409b590 comp rx=0 tx=0).stop 2024-09-16T00:42:56.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.810+0000 7f37c48f4740 1 -- 172.21.15.33:0/1597766153 shutdown_connections 2024-09-16T00:42:56.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.810+0000 7f37c48f4740 1 --2- 172.21.15.33:0/1597766153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ec04520 0x563c7ebfc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.810+0000 7f37c48f4740 1 --2- 172.21.15.33:0/1597766153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ec00050 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.810+0000 7f37c48f4740 1 --2- 172.21.15.33:0/1597766153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ebf9780 0x563c7ec105a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.810+0000 7f37c48f4740 1 -- 172.21.15.33:0/1597766153 >> 172.21.15.33:0/1597766153 conn(0x563c7ebc2310 msgr2=0x563c7ebff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:56.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.810+0000 7f37c48f4740 1 -- 172.21.15.33:0/1597766153 shutdown_connections 2024-09-16T00:42:56.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.810+0000 7f37c48f4740 1 -- 172.21.15.33:0/1597766153 wait complete. 2024-09-16T00:42:56.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.810+0000 7f37c48f4740 1 -- start start 2024-09-16T00:42:56.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.810+0000 7f37c48f4740 1 Processor -- start 2024-09-16T00:42:56.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37c48f4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebf9780 0x563c7ec0cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37c48f4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ec08240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37c48f4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec087c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37c48f4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1b3c70 con 0x563c7ebf9780 2024-09-16T00:42:56.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37c48f4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1b50e0 con 0x563c7ec04520 2024-09-16T00:42:56.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37c48f4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1b4340 con 0x563c7ebfacf0 2024-09-16T00:42:56.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37c48f4740 10 client.0 _ll_register_callbacks cb 0x7f37a4097930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:56.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37bbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebf9780 0x563c7ec0cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37bb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ec08240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37bbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebf9780 0x563c7ec0cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56562/0 (socket says 172.21.15.33:56562) 2024-09-16T00:42:56.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37bbfff640 1 -- 172.21.15.33:0/2948432205 learned_addr learned my addr 172.21.15.33:0/2948432205 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37c0ab8640 1 --2- 172.21.15.33:0/2948432205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37bb7fe640 1 -- 172.21.15.33:0/2948432205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 msgr2=0x563c7ec087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/2948432205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec087c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37bb7fe640 1 -- 172.21.15.33:0/2948432205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebf9780 msgr2=0x563c7ec0cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/2948432205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebf9780 0x563c7ec0cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37bb7fe640 1 -- 172.21.15.33:0/2948432205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c7ebf6b10 con 0x563c7ebfacf0 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37bb7fe640 1 --2- 172.21.15.33:0/2948432205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ec08240 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x563c7ebfa1f0 tx=0x7f37a409bb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e49a841ca648825d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.811+0000 7f37ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37ab7fe640 1 -- 172.21.15.33:0/2948432205 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a4082060 con 0x563c7ebfacf0 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37ab7fe640 1 -- 172.21.15.33:0/2948432205 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f37a40afdc0 con 0x563c7ebfacf0 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37c48f4740 1 -- 172.21.15.33:0/2948432205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37a409bd90 con 0x563c7ebfacf0 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37ab7fe640 1 -- 172.21.15.33:0/2948432205 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f37a40b8070 con 0x563c7ebfacf0 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37c48f4740 10 client.29260 fetch_fsmap learned FSMap version 34 2024-09-16T00:42:56.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37c48f4740 1 -- 172.21.15.33:0/2948432205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563c7ec7eb70 con 0x563c7ebfacf0 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37ab7fe640 1 -- 172.21.15.33:0/2948432205 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37a409bd90 con 0x563c7ebfacf0 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37ab7fe640 1 -- 172.21.15.33:0/2948432205 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 34) ==== 8362+0+0 (secure 0 0 0) 0x7f37a409aae0 con 0x563c7ebfacf0 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37c48f4740 10 client.29260 fetch_fsmap finished waiting for FSMap version 34 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37c48f4740 10 client.29260 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37c48f4740 20 client.29260 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2948432205 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x563c7ec7eff0 0x563c7ec813e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37c48f4740 4 client.29260 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37c48f4740 1 -- 172.21.15.33:0/2948432205 --> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563c7eb66d70 con 0x563c7ec7eff0 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.812+0000 7f37bbfff640 1 --2- 172.21.15.33:0/2948432205 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x563c7ec7eff0 0x563c7ec813e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.813+0000 7f37bbfff640 1 --2- 172.21.15.33:0/2948432205 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x563c7ec7eff0 0x563c7ec813e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c42fd2e92ecaba4 server_cookie=20270469c289e746 in_seq=0 out_seq=0 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.813+0000 7f37ab7fe640 10 client.29260 ms_handle_connect on v2:172.21.15.83:6836/4244918448 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37ab7fe640 1 -- 172.21.15.33:0/2948432205 <== mds.2 v2:172.21.15.83:6836/4244918448 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563c7eb66d70 con 0x563c7ec7eff0 2024-09-16T00:42:56.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37ab7fe640 10 client.29260 handle_command_reply: tid=0 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37c48f4740 1 client.29260 shutdown 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37c48f4740 1 -- 172.21.15.33:0/2948432205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 msgr2=0x563c7ec08240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2948432205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ec08240 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x563c7ebfa1f0 tx=0x7f37a409bb80 comp rx=0 tx=0).stop 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37c48f4740 1 -- 172.21.15.33:0/2948432205 shutdown_connections 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2948432205 >> [v2:172.21.15.83:6836/4244918448,v1:172.21.15.83:6837/4244918448] conn(0x563c7ec7eff0 0x563c7ec813e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2948432205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563c7ec04520 0x563c7ec087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2948432205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c7ebfacf0 0x563c7ec08240 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37ab7fe640 0 client.29260 ms_handle_reset on v2:172.21.15.83:6836/4244918448 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37c48f4740 1 --2- 172.21.15.33:0/2948432205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c7ebf9780 0x563c7ec0cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37c48f4740 1 -- 172.21.15.33:0/2948432205 >> 172.21.15.33:0/2948432205 conn(0x563c7ebc2310 msgr2=0x563c7ec00160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37c48f4740 1 -- 172.21.15.33:0/2948432205 shutdown_connections 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37c48f4740 1 -- 172.21.15.33:0/2948432205 wait complete. 2024-09-16T00:42:56.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.814+0000 7f37c48f4740 20 client.29260 trim_cache size 0 max 16384 2024-09-16T00:42:56.820 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:42:56.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.815+0000 7f37c12b9640 1 -- 172.21.15.33:0/1206887101 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f379c0a3300 msgr2=0x7f379c0a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:56.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.815+0000 7f37c12b9640 1 --2- 172.21.15.33:0/1206887101 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f379c0a3300 0x7f379c0a57c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f37ac03ba60 tx=0x7f37ac000bc0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.815+0000 7f37c12b9640 1 -- 172.21.15.33:0/1206887101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37bc110190 msgr2=0x7f37bc116ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:56.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.815+0000 7f37c12b9640 1 --2- 172.21.15.33:0/1206887101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37bc110190 0x7f37bc116ab0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f37a400ba50 tx=0x7f37a400bf10 comp rx=0 tx=0).stop 2024-09-16T00:42:56.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.815+0000 7f37c12b9640 1 -- 172.21.15.33:0/1206887101 shutdown_connections 2024-09-16T00:42:56.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.815+0000 7f37c12b9640 1 --2- 172.21.15.33:0/1206887101 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f379c0a3300 0x7f379c0a57c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.815+0000 7f37c12b9640 1 --2- 172.21.15.33:0/1206887101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37bc11b140 0x7f37bc116ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.815+0000 7f37c12b9640 1 --2- 172.21.15.33:0/1206887101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37bc110190 0x7f37bc116ab0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.815+0000 7f37c12b9640 1 --2- 172.21.15.33:0/1206887101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37bc06ea30 0x7f37bc11aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:56.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.815+0000 7f37c12b9640 1 -- 172.21.15.33:0/1206887101 >> 172.21.15.33:0/1206887101 conn(0x7f37bc073180 msgr2=0x7f37bc11b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:56.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.815+0000 7f37c12b9640 1 -- 172.21.15.33:0/1206887101 shutdown_connections 2024-09-16T00:42:56.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:56.816+0000 7f37c12b9640 1 -- 172.21.15.33:0/1206887101 wait complete. 2024-09-16T00:42:57.076 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-09-16T00:42:57.077 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T00:42:57.416 DEBUG:teuthology.parallel:result is None 2024-09-16T00:42:57.417 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T00:42:57.467 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T00:42:57.467 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-09-16T00:42:57.478 INFO:tasks.quiescer:joining Quiescers 2024-09-16T00:42:57.478 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-09-16T00:42:57.478 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:42:57.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:57 smithi033 ceph-mon[31243]: pgmap v1124: 129 pgs: 129 active+clean; 709 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 7.7 MiB/s wr, 431 op/s 2024-09-16T00:42:57.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:57 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:42:57.810 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-16T00:42:57.876 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:57 smithi083 ceph-mon[35242]: pgmap v1124: 129 pgs: 129 active+clean; 709 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 7.7 MiB/s wr, 431 op/s 2024-09-16T00:42:57.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:57 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:42:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:57 smithi060 ceph-mon[35185]: pgmap v1124: 129 pgs: 129 active+clean; 709 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 7.7 MiB/s wr, 431 op/s 2024-09-16T00:42:57.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:57 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:42:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.378+0000 7f9f27337640 1 -- 172.21.15.33:0/2272833230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f2010e480 msgr2=0x7f9f20110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.378+0000 7f9f27337640 1 --2- 172.21.15.33:0/2272833230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f2010e480 0x7f9f20110870 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009ca0 tx=0x7f9f1802a8c0 comp rx=0 tx=0).stop 2024-09-16T00:42:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.379+0000 7f9f27337640 1 -- 172.21.15.33:0/2272833230 shutdown_connections 2024-09-16T00:42:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.379+0000 7f9f27337640 1 --2- 172.21.15.33:0/2272833230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f2010e480 0x7f9f20110870 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.379+0000 7f9f27337640 1 --2- 172.21.15.33:0/2272833230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f20101280 0x7f9f2010df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.379+0000 7f9f27337640 1 --2- 172.21.15.33:0/2272833230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f201008d0 0x7f9f20100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.379+0000 7f9f27337640 1 -- 172.21.15.33:0/2272833230 >> 172.21.15.33:0/2272833230 conn(0x7f9f200fc7a0 msgr2=0x7f9f200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:58.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.379+0000 7f9f27337640 1 -- 172.21.15.33:0/2272833230 shutdown_connections 2024-09-16T00:42:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.379+0000 7f9f27337640 1 -- 172.21.15.33:0/2272833230 wait complete. 2024-09-16T00:42:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.379+0000 7f9f27337640 1 Processor -- start 2024-09-16T00:42:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.379+0000 7f9f27337640 1 -- start start 2024-09-16T00:42:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.379+0000 7f9f27337640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f201008d0 0x7f9f2019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.379+0000 7f9f27337640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f20101280 0x7f9f2019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f27337640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f2010e480 0x7f9f2019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f27337640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9f20114220 con 0x7f9f20101280 2024-09-16T00:42:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f27337640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9f201140a0 con 0x7f9f201008d0 2024-09-16T00:42:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f27337640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9f201143a0 con 0x7f9f2010e480 2024-09-16T00:42:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f248d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f20101280 0x7f9f2019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f248d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f20101280 0x7f9f2019abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56590/0 (socket says 172.21.15.33:56590) 2024-09-16T00:42:58.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f248d2640 1 -- 172.21.15.33:0/1681290932 learned_addr learned my addr 172.21.15.33:0/1681290932 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f250d3640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f201008d0 0x7f9f2019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f258d4640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f2010e480 0x7f9f2019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f248d2640 1 -- 172.21.15.33:0/1681290932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f2010e480 msgr2=0x7f9f2019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f248d2640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f2010e480 0x7f9f2019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f248d2640 1 -- 172.21.15.33:0/1681290932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f201008d0 msgr2=0x7f9f2019a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f248d2640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f201008d0 0x7f9f2019a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f248d2640 1 -- 172.21.15.33:0/1681290932 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2019f650 con 0x7f9f20101280 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f250d3640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f201008d0 0x7f9f2019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f248d2640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f20101280 0x7f9f2019abe0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000c920 tx=0x7f9f1000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f167fc640 1 -- 172.21.15.33:0/1681290932 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f10007c70 con 0x7f9f20101280 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f27337640 1 -- 172.21.15.33:0/1681290932 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f2019f940 con 0x7f9f20101280 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f167fc640 1 -- 172.21.15.33:0/1681290932 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f10007e10 con 0x7f9f20101280 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f167fc640 1 -- 172.21.15.33:0/1681290932 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f10018890 con 0x7f9f20101280 2024-09-16T00:42:58.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.380+0000 7f9f27337640 1 -- 172.21.15.33:0/1681290932 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f201a7220 con 0x7f9f20101280 2024-09-16T00:42:58.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.381+0000 7f9f27337640 1 -- 172.21.15.33:0/1681290932 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f20100cb0 con 0x7f9f20101280 2024-09-16T00:42:58.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.383+0000 7f9f167fc640 1 -- 172.21.15.33:0/1681290932 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9f10018a70 con 0x7f9f20101280 2024-09-16T00:42:58.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.383+0000 7f9f167fc640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9f0c0a3240 0x7f9f0c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:58.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.383+0000 7f9f250d3640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9f0c0a3240 0x7f9f0c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:58.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.383+0000 7f9f167fc640 1 -- 172.21.15.33:0/1681290932 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9f100cfff0 con 0x7f9f20101280 2024-09-16T00:42:58.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.384+0000 7f9f250d3640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9f0c0a3240 0x7f9f0c0a5700 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c009930 tx=0x7f9f1c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:58.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.388+0000 7f9f167fc640 1 -- 172.21.15.33:0/1681290932 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9f10014030 con 0x7f9f20101280 2024-09-16T00:42:58.444 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:42:58.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.609+0000 7f9f27337640 1 -- 172.21.15.33:0/1681290932 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f201a7480 con 0x7f9f20101280 2024-09-16T00:42:58.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.610+0000 7f9f167fc640 1 -- 172.21.15.33:0/1681290932 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11908 (secure 0 0 0) 0x7f9f1000de20 con 0x7f9f20101280 2024-09-16T00:42:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:42:58.610 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":35,"btime":"2024-09-16T00:42:57:562466+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:42:57.479147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24472,24493,24494]},"id":1}]} 2024-09-16T00:42:58.611 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 35 2024-09-16T00:42:58.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.612+0000 7f9f27337640 1 -- 172.21.15.33:0/1681290932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9f0c0a3240 msgr2=0x7f9f0c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:58.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.612+0000 7f9f27337640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9f0c0a3240 0x7f9f0c0a5700 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c009930 tx=0x7f9f1c008040 comp rx=0 tx=0).stop 2024-09-16T00:42:58.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.612+0000 7f9f27337640 1 -- 172.21.15.33:0/1681290932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f20101280 msgr2=0x7f9f2019abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:58.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.612+0000 7f9f27337640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f20101280 0x7f9f2019abe0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000c920 tx=0x7f9f1000cde0 comp rx=0 tx=0).stop 2024-09-16T00:42:58.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.612+0000 7f9f27337640 1 -- 172.21.15.33:0/1681290932 shutdown_connections 2024-09-16T00:42:58.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.612+0000 7f9f27337640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9f0c0a3240 0x7f9f0c0a5700 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:58.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.612+0000 7f9f27337640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f2010e480 0x7f9f2019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:58.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.612+0000 7f9f27337640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f20101280 0x7f9f2019abe0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:58.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.612+0000 7f9f27337640 1 --2- 172.21.15.33:0/1681290932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f201008d0 0x7f9f2019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:58.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.612+0000 7f9f27337640 1 -- 172.21.15.33:0/1681290932 >> 172.21.15.33:0/1681290932 conn(0x7f9f200fc7a0 msgr2=0x7f9f200feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:58.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.613+0000 7f9f27337640 1 -- 172.21.15.33:0/1681290932 shutdown_connections 2024-09-16T00:42:58.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:58.613+0000 7f9f27337640 1 -- 172.21.15.33:0/1681290932 wait complete. 2024-09-16T00:42:58.726 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:58 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:42:58.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:58 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:42:58.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:58 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:42:59.409 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24494 quiesce db --cancel --all' 2024-09-16T00:42:59.409 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24494 quiesce db --cancel --all 2024-09-16T00:42:59.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.584+0000 7f0af75b3640 1 -- 172.21.15.33:0/2224753250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0af0074620 msgr2=0x7f0af0074a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:59.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.584+0000 7f0af75b3640 1 --2- 172.21.15.33:0/2224753250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0af0074620 0x7f0af0074a00 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009900 tx=0x7f0aec02a480 comp rx=0 tx=0).stop 2024-09-16T00:42:59.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.584+0000 7f0af75b3640 1 -- 172.21.15.33:0/2224753250 shutdown_connections 2024-09-16T00:42:59.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.584+0000 7f0af75b3640 1 --2- 172.21.15.33:0/2224753250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0af010e3d0 0x7f0af0110820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.584+0000 7f0af75b3640 1 --2- 172.21.15.33:0/2224753250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0af0074f40 0x7f0af010de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.584+0000 7f0af75b3640 1 --2- 172.21.15.33:0/2224753250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0af0074620 0x7f0af0074a00 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.584+0000 7f0af75b3640 1 -- 172.21.15.33:0/2224753250 >> 172.21.15.33:0/2224753250 conn(0x7f0af00fd8f0 msgr2=0x7f0af00ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:59.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.584+0000 7f0af75b3640 1 -- 172.21.15.33:0/2224753250 shutdown_connections 2024-09-16T00:42:59.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.584+0000 7f0af75b3640 1 -- 172.21.15.33:0/2224753250 wait complete. 2024-09-16T00:42:59.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af75b3640 1 Processor -- start 2024-09-16T00:42:59.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af75b3640 1 -- start start 2024-09-16T00:42:59.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af75b3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0af0074620 0x7f0af019a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af75b3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0af0074f40 0x7f0af019abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af75b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0af010e3d0 0x7f0af019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af75b3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01141d0 con 0x7f0af0074f40 2024-09-16T00:42:59.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af75b3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0af0114050 con 0x7f0af0074620 2024-09-16T00:42:59.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af75b3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0af0114350 con 0x7f0af010e3d0 2024-09-16T00:42:59.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af4b4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0af0074f40 0x7f0af019abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af4b4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0af0074f40 0x7f0af019abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56606/0 (socket says 172.21.15.33:56606) 2024-09-16T00:42:59.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af4b4e640 1 -- 172.21.15.33:0/2107669412 learned_addr learned my addr 172.21.15.33:0/2107669412 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:59.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af534f640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0af0074620 0x7f0af019a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af5b50640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0af010e3d0 0x7f0af019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af4b4e640 1 -- 172.21.15.33:0/2107669412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0af010e3d0 msgr2=0x7f0af019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:59.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0af010e3d0 0x7f0af019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af4b4e640 1 -- 172.21.15.33:0/2107669412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0af0074620 msgr2=0x7f0af019a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:59.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0af0074620 0x7f0af019a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.585+0000 7f0af4b4e640 1 -- 172.21.15.33:0/2107669412 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af019f620 con 0x7f0af0074f40 2024-09-16T00:42:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.586+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0af0074f40 0x7f0af019abb0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f0ae000e9f0 tx=0x7f0ae000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.586+0000 7f0ae67fc640 1 -- 172.21.15.33:0/2107669412 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae000cd80 con 0x7f0af0074f40 2024-09-16T00:42:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.586+0000 7f0af75b3640 1 -- 172.21.15.33:0/2107669412 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0af019f910 con 0x7f0af0074f40 2024-09-16T00:42:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.586+0000 7f0ae67fc640 1 -- 172.21.15.33:0/2107669412 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ae0010910 con 0x7f0af0074f40 2024-09-16T00:42:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.586+0000 7f0ae67fc640 1 -- 172.21.15.33:0/2107669412 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae00188f0 con 0x7f0af0074f40 2024-09-16T00:42:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.586+0000 7f0af75b3640 1 -- 172.21.15.33:0/2107669412 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0af01a71f0 con 0x7f0af0074f40 2024-09-16T00:42:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.589+0000 7f0af998c740 1 Processor -- start 2024-09-16T00:42:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.589+0000 7f0af998c740 1 -- start start 2024-09-16T00:42:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.589+0000 7f0ae67fc640 1 -- 172.21.15.33:0/2107669412 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0ae0018b10 con 0x7f0af0074f40 2024-09-16T00:42:59.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.589+0000 7f0af998c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba50b40 0x55bcbba4f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.589+0000 7f0af998c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba4f5d0 0x55bcbba4fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.589+0000 7f0af998c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba5a370 0x55bcbba5c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.589+0000 7f0af998c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bcbb9b52f0 con 0x55bcbba5a370 2024-09-16T00:42:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.589+0000 7f0af998c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55bcbb95ccf0 con 0x55bcbba4f5d0 2024-09-16T00:42:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.589+0000 7f0af998c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bcbb9b12b0 con 0x55bcbba50b40 2024-09-16T00:42:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0af4b4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba5a370 0x55bcbba5c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0af4b4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba5a370 0x55bcbba5c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56610/0 (socket says 172.21.15.33:56610) 2024-09-16T00:42:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0af4b4e640 1 -- 172.21.15.33:0/1888216064 learned_addr learned my addr 172.21.15.33:0/1888216064 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0af5b50640 1 --2- 172.21.15.33:0/1888216064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba50b40 0x55bcbba4f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0af534f640 1 --2- 172.21.15.33:0/1888216064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba4f5d0 0x55bcbba4fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0af4b4e640 1 -- 172.21.15.33:0/1888216064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba50b40 msgr2=0x55bcbba4f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/1888216064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba50b40 0x55bcbba4f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0af4b4e640 1 -- 172.21.15.33:0/1888216064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba4f5d0 msgr2=0x55bcbba4fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/1888216064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba4f5d0 0x55bcbba4fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0af4b4e640 1 -- 172.21.15.33:0/1888216064 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcbb9413a0 con 0x55bcbba5a370 2024-09-16T00:42:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0af534f640 1 --2- 172.21.15.33:0/1888216064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba4f5d0 0x55bcbba4fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0ae67fc640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0adc0a3240 0x7f0adc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.590+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/1888216064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba5a370 0x55bcbba5c760 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00d8480 tx=0x7f0ae00d8940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a47f64aebe3c85 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.591+0000 7f0af5b50640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0adc0a3240 0x7f0adc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.591+0000 7f0adb7fe640 1 -- 172.21.15.33:0/1888216064 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0098600 con 0x55bcbba5a370 2024-09-16T00:42:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.591+0000 7f0ae67fc640 1 -- 172.21.15.33:0/2107669412 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0ae00d01c0 con 0x7f0af0074f40 2024-09-16T00:42:59.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.591+0000 7f0adb7fe640 1 -- 172.21.15.33:0/1888216064 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ae009ca60 con 0x55bcbba5a370 2024-09-16T00:42:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.592+0000 7f0adb7fe640 1 -- 172.21.15.33:0/1888216064 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae00b6240 con 0x55bcbba5a370 2024-09-16T00:42:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.592+0000 7f0af998c740 1 -- 172.21.15.33:0/1888216064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba5a370 msgr2=0x55bcbba5c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.592+0000 7f0af5b50640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0adc0a3240 0x7f0adc0a5700 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8005fd0 tx=0x7f0ae8009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.592+0000 7f0af998c740 1 --2- 172.21.15.33:0/1888216064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba5a370 0x55bcbba5c760 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00d8480 tx=0x7f0ae00d8940 comp rx=0 tx=0).stop 2024-09-16T00:42:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.592+0000 7f0af998c740 1 -- 172.21.15.33:0/1888216064 shutdown_connections 2024-09-16T00:42:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.592+0000 7f0af998c740 1 --2- 172.21.15.33:0/1888216064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba5a370 0x55bcbba5c760 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.592+0000 7f0af998c740 1 --2- 172.21.15.33:0/1888216064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba4f5d0 0x55bcbba4fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.592+0000 7f0af998c740 1 --2- 172.21.15.33:0/1888216064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba50b40 0x55bcbba4f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.592+0000 7f0af998c740 1 -- 172.21.15.33:0/1888216064 >> 172.21.15.33:0/1888216064 conn(0x55bcbba18310 msgr2=0x55bcbba4bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.592+0000 7f0af998c740 1 -- 172.21.15.33:0/1888216064 shutdown_connections 2024-09-16T00:42:59.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.592+0000 7f0af998c740 1 -- 172.21.15.33:0/1888216064 wait complete. 2024-09-16T00:42:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.592+0000 7f0af998c740 1 -- start start 2024-09-16T00:42:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af998c740 1 Processor -- start 2024-09-16T00:42:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af998c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba4f5d0 0x55bcbba59cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af998c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbba56f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af998c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba5a370 0x55bcbba574a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af998c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bcbba5d060 con 0x55bcbba50b40 2024-09-16T00:42:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af998c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55bcbb970060 con 0x55bcbba5a370 2024-09-16T00:42:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af998c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bcbba5d1e0 con 0x55bcbba4f5d0 2024-09-16T00:42:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af998c740 10 client.0 _ll_register_callbacks cb 0x55bcbb6c38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af4b4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbba56f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af534f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba4f5d0 0x55bcbba59cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af4b4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbba56f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56618/0 (socket says 172.21.15.33:56618) 2024-09-16T00:42:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af4b4e640 1 -- 172.21.15.33:0/4034961902 learned_addr learned my addr 172.21.15.33:0/4034961902 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.593+0000 7f0af5b50640 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba5a370 0x55bcbba574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0af534f640 1 -- 172.21.15.33:0/4034961902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba5a370 msgr2=0x55bcbba574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0af534f640 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba5a370 0x55bcbba574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0af534f640 1 -- 172.21.15.33:0/4034961902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 msgr2=0x55bcbba56f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0af534f640 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbba56f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0af534f640 1 -- 172.21.15.33:0/4034961902 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcbb7263c0 con 0x55bcbba4f5d0 2024-09-16T00:42:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbba56f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:42:59.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0af534f640 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba4f5d0 0x55bcbba59cb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0aec02aa70 tx=0x7f0aec0043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bda3fee93cdbfbb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0ad8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-16T00:42:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0ad8ff9640 1 -- 172.21.15.33:0/4034961902 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec035040 con 0x55bcbba4f5d0 2024-09-16T00:42:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0ad8ff9640 1 -- 172.21.15.33:0/4034961902 <== mon.1 v2:172.21.15.83:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0aec005450 con 0x55bcbba4f5d0 2024-09-16T00:42:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0af998c740 1 -- 172.21.15.33:0/4034961902 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ae009ca60 con 0x55bcbba4f5d0 2024-09-16T00:42:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0ad8ff9640 1 -- 172.21.15.33:0/4034961902 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0aec005650 con 0x55bcbba4f5d0 2024-09-16T00:42:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0af998c740 10 client.29275 fetch_fsmap learned FSMap version 35 2024-09-16T00:42:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0af998c740 1 -- 172.21.15.33:0/4034961902 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bcbba68aa0 con 0x55bcbba4f5d0 2024-09-16T00:42:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.594+0000 7f0ad8ff9640 1 -- 172.21.15.33:0/4034961902 <== mon.1 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ae009ca60 con 0x55bcbba4f5d0 2024-09-16T00:42:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.595+0000 7f0ad8ff9640 1 -- 172.21.15.33:0/4034961902 <== mon.1 v2:172.21.15.83:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f0aec003a80 con 0x55bcbba4f5d0 2024-09-16T00:42:59.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.595+0000 7f0af998c740 10 client.29275 fetch_fsmap finished waiting for FSMap version 35 2024-09-16T00:42:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.595+0000 7f0af998c740 10 client.29275 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:42:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.595+0000 7f0af998c740 20 client.29275 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.595+0000 7f0af998c740 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55bcbba69f60 0x55bcbba6c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.595+0000 7f0af998c740 4 client.29275 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:42:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.595+0000 7f0af998c740 1 -- 172.21.15.33:0/4034961902 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcbb9c2860 con 0x55bcbba69f60 2024-09-16T00:42:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.595+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55bcbba69f60 0x55bcbba6c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.595+0000 7f0ad8ff9640 10 client.29275 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:42:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.595+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55bcbba69f60 0x55bcbba6c390 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ed233eed15673ce3 server_cookie=a46e3538ed1a01af in_seq=0 out_seq=0 2024-09-16T00:42:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0ad8ff9640 1 -- 172.21.15.33:0/4034961902 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bcbb9c2860 con 0x55bcbba69f60 2024-09-16T00:42:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0ad8ff9640 10 client.29275 handle_command_reply: tid=0 2024-09-16T00:42:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0af998c740 1 client.29275 shutdown 2024-09-16T00:42:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0af998c740 1 -- 172.21.15.33:0/4034961902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba4f5d0 msgr2=0x55bcbba59cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0af998c740 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba4f5d0 0x55bcbba59cb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0aec02aa70 tx=0x7f0aec0043b0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0af998c740 1 -- 172.21.15.33:0/4034961902 shutdown_connections 2024-09-16T00:42:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0af998c740 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55bcbba69f60 0x55bcbba6c390 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0af998c740 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba5a370 0x55bcbba574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0ad8ff9640 0 client.29275 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:42:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0af998c740 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbba56f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0af998c740 1 --2- 172.21.15.33:0/4034961902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba4f5d0 0x55bcbba59cb0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0af998c740 1 -- 172.21.15.33:0/4034961902 >> 172.21.15.33:0/4034961902 conn(0x55bcbba18310 msgr2=0x55bcbba51960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:59.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.596+0000 7f0af998c740 1 -- 172.21.15.33:0/4034961902 shutdown_connections 2024-09-16T00:42:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.597+0000 7f0af998c740 1 -- 172.21.15.33:0/4034961902 wait complete. 2024-09-16T00:42:59.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.597+0000 7f0af998c740 20 client.29275 trim_cache size 0 max 16384 2024-09-16T00:42:59.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.611+0000 7f0af998c740 1 Processor -- start 2024-09-16T00:42:59.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.611+0000 7f0af998c740 1 -- start start 2024-09-16T00:42:59.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af998c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba4f5d0 0x55bcbba66350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af998c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbba52120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af998c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba5a370 0x55bcbba52660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af998c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01b3cc0 con 0x55bcbba50b40 2024-09-16T00:42:59.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af998c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01b1530 con 0x55bcbba5a370 2024-09-16T00:42:59.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af998c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01af380 con 0x55bcbba4f5d0 2024-09-16T00:42:59.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af4b4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbba52120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af4b4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbba52120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56634/0 (socket says 172.21.15.33:56634) 2024-09-16T00:42:59.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af4b4e640 1 -- 172.21.15.33:0/1441636390 learned_addr learned my addr 172.21.15.33:0/1441636390 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:59.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af534f640 1 --2- 172.21.15.33:0/1441636390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba4f5d0 0x55bcbba66350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af5b50640 1 --2- 172.21.15.33:0/1441636390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba5a370 0x55bcbba52660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af4b4e640 1 -- 172.21.15.33:0/1441636390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba4f5d0 msgr2=0x55bcbba66350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:59.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/1441636390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba4f5d0 0x55bcbba66350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af4b4e640 1 -- 172.21.15.33:0/1441636390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba5a370 msgr2=0x55bcbba52660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:59.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/1441636390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba5a370 0x55bcbba52660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af4b4e640 1 -- 172.21.15.33:0/1441636390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcbba52ba0 con 0x55bcbba50b40 2024-09-16T00:42:59.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af5b50640 1 --2- 172.21.15.33:0/1441636390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba5a370 0x55bcbba52660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:42:59.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.612+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/1441636390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbba52120 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f0ae000cfd0 tx=0x7f0ae000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7abf172f441bc862 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:59.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.613+0000 7f0adaffd640 1 -- 172.21.15.33:0/1441636390 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae00b4020 con 0x55bcbba50b40 2024-09-16T00:42:59.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.613+0000 7f0adaffd640 1 -- 172.21.15.33:0/1441636390 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ae00d3a70 con 0x55bcbba50b40 2024-09-16T00:42:59.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.613+0000 7f0adaffd640 1 -- 172.21.15.33:0/1441636390 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae00b4890 con 0x55bcbba50b40 2024-09-16T00:42:59.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.614+0000 7f0af998c740 1 -- 172.21.15.33:0/1441636390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 msgr2=0x55bcbba52120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:59.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.614+0000 7f0af998c740 1 --2- 172.21.15.33:0/1441636390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbba52120 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f0ae000cfd0 tx=0x7f0ae000cf60 comp rx=0 tx=0).stop 2024-09-16T00:42:59.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.614+0000 7f0af998c740 1 -- 172.21.15.33:0/1441636390 shutdown_connections 2024-09-16T00:42:59.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.614+0000 7f0af998c740 1 --2- 172.21.15.33:0/1441636390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba5a370 0x55bcbba52660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.614+0000 7f0af998c740 1 --2- 172.21.15.33:0/1441636390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbba52120 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.614+0000 7f0af998c740 1 --2- 172.21.15.33:0/1441636390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba4f5d0 0x55bcbba66350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.614+0000 7f0af998c740 1 -- 172.21.15.33:0/1441636390 >> 172.21.15.33:0/1441636390 conn(0x55bcbba18310 msgr2=0x55bcbba55830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:59.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.614+0000 7f0af998c740 1 -- 172.21.15.33:0/1441636390 shutdown_connections 2024-09-16T00:42:59.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.614+0000 7f0af998c740 1 -- 172.21.15.33:0/1441636390 wait complete. 2024-09-16T00:42:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.614+0000 7f0af998c740 1 -- start start 2024-09-16T00:42:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.614+0000 7f0af998c740 1 Processor -- start 2024-09-16T00:42:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af998c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba4f5d0 0x55bcbbaccf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af998c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbbacd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af998c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba5a370 0x55bcbba5daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af998c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01af9f0 con 0x55bcbba50b40 2024-09-16T00:42:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af998c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01b0e60 con 0x55bcbba4f5d0 2024-09-16T00:42:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af998c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01b00c0 con 0x55bcbba5a370 2024-09-16T00:42:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af998c740 10 client.0 _ll_register_callbacks cb 0x7f0aec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:42:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af4b4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbbacd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af4b4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbbacd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56640/0 (socket says 172.21.15.33:56640) 2024-09-16T00:42:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af4b4e640 1 -- 172.21.15.33:0/1379492030 learned_addr learned my addr 172.21.15.33:0/1379492030 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:42:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af5b50640 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba5a370 0x55bcbba5daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af534f640 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba4f5d0 0x55bcbbaccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af4b4e640 1 -- 172.21.15.33:0/1379492030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba5a370 msgr2=0x55bcbba5daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba5a370 0x55bcbba5daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af4b4e640 1 -- 172.21.15.33:0/1379492030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba4f5d0 msgr2=0x55bcbbaccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:42:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba4f5d0 0x55bcbbaccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af4b4e640 1 -- 172.21.15.33:0/1379492030 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcbba5e190 con 0x55bcbba50b40 2024-09-16T00:42:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0af4b4e640 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbbacd460 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00d8e40 tx=0x7f0ae00d8430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1712ddf756e3ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:42:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.615+0000 7f0ad8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:42:59.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0ad8ff9640 1 -- 172.21.15.33:0/1379492030 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae00b4070 con 0x55bcbba50b40 2024-09-16T00:42:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0ad8ff9640 1 -- 172.21.15.33:0/1379492030 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0ae00b52b0 con 0x55bcbba50b40 2024-09-16T00:42:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0af998c740 1 -- 172.21.15.33:0/1379492030 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ae00d3a70 con 0x55bcbba50b40 2024-09-16T00:42:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0ad8ff9640 1 -- 172.21.15.33:0/1379492030 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0ae00b5560 con 0x55bcbba50b40 2024-09-16T00:42:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0af998c740 10 client.20199 fetch_fsmap learned FSMap version 35 2024-09-16T00:42:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0af998c740 1 -- 172.21.15.33:0/1379492030 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bcbbad46f0 con 0x55bcbba50b40 2024-09-16T00:42:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0ad8ff9640 1 -- 172.21.15.33:0/1379492030 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ae00d3a70 con 0x55bcbba50b40 2024-09-16T00:42:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0ad8ff9640 1 -- 172.21.15.33:0/1379492030 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f0ae009c5c0 con 0x55bcbba50b40 2024-09-16T00:42:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0af998c740 10 client.20199 fetch_fsmap finished waiting for FSMap version 35 2024-09-16T00:42:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0af998c740 10 client.20199 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-16T00:42:59.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0af998c740 20 client.20199 populate_metadata read hostname 'smithi033' 2024-09-16T00:42:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0af998c740 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55bcbba69f60 0x55bcbbad69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:42:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0af998c740 4 client.20199 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:42:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.616+0000 7f0af998c740 1 -- 172.21.15.33:0/1379492030 --> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bcbb9bcd70 con 0x55bcbba69f60 2024-09-16T00:42:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0af534f640 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55bcbba69f60 0x55bcbbad69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:42:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0af534f640 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55bcbba69f60 0x55bcbbad69d0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf01832c1c92d38b server_cookie=a7ade1cc72d1a3cd in_seq=0 out_seq=0 2024-09-16T00:42:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0ad8ff9640 10 client.20199 ms_handle_connect on v2:172.21.15.83:6838/3202868586 2024-09-16T00:42:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0ad8ff9640 1 -- 172.21.15.33:0/1379492030 <== mds.1 v2:172.21.15.83:6838/3202868586 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bcbb9bcd70 con 0x55bcbba69f60 2024-09-16T00:42:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0ad8ff9640 10 client.20199 handle_command_reply: tid=0 2024-09-16T00:42:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0af998c740 1 client.20199 shutdown 2024-09-16T00:42:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0af998c740 1 -- 172.21.15.33:0/1379492030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 msgr2=0x55bcbbacd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:59.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0af998c740 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbbacd460 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00d8e40 tx=0x7f0ae00d8430 comp rx=0 tx=0).stop 2024-09-16T00:42:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0af998c740 1 -- 172.21.15.33:0/1379492030 shutdown_connections 2024-09-16T00:42:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0af998c740 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bcbba5a370 0x55bcbba5daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0af998c740 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcbba50b40 0x55bcbbacd460 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0af998c740 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.83:6838/3202868586,v1:172.21.15.83:6839/3202868586] conn(0x55bcbba69f60 0x55bcbbad69d0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0af998c740 1 --2- 172.21.15.33:0/1379492030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bcbba4f5d0 0x55bcbbaccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0af998c740 1 -- 172.21.15.33:0/1379492030 >> 172.21.15.33:0/1379492030 conn(0x55bcbba18310 msgr2=0x55bcbba587c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.617+0000 7f0ad8ff9640 0 client.20199 ms_handle_reset on v2:172.21.15.83:6838/3202868586 2024-09-16T00:42:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af998c740 1 -- 172.21.15.33:0/1379492030 shutdown_connections 2024-09-16T00:42:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af998c740 1 -- 172.21.15.33:0/1379492030 wait complete. 2024-09-16T00:42:59.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af998c740 20 client.20199 trim_cache size 0 max 16384 2024-09-16T00:42:59.625 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:42:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af75b3640 1 -- 172.21.15.33:0/2107669412 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0adc0a3240 msgr2=0x7f0adc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:59.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af75b3640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0adc0a3240 0x7f0adc0a5700 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8005fd0 tx=0x7f0ae8009500 comp rx=0 tx=0).stop 2024-09-16T00:42:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af75b3640 1 -- 172.21.15.33:0/2107669412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0af0074f40 msgr2=0x7f0af019abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:42:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af75b3640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0af0074f40 0x7f0af019abb0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f0ae000e9f0 tx=0x7f0ae000eeb0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af75b3640 1 -- 172.21.15.33:0/2107669412 shutdown_connections 2024-09-16T00:42:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af75b3640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0adc0a3240 0x7f0adc0a5700 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af75b3640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0af010e3d0 0x7f0af019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af75b3640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0af0074f40 0x7f0af019abb0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af75b3640 1 --2- 172.21.15.33:0/2107669412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0af0074620 0x7f0af019a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:42:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.618+0000 7f0af75b3640 1 -- 172.21.15.33:0/2107669412 >> 172.21.15.33:0/2107669412 conn(0x7f0af00fd8f0 msgr2=0x7f0af00fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:42:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.619+0000 7f0af75b3640 1 -- 172.21.15.33:0/2107669412 shutdown_connections 2024-09-16T00:42:59.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:42:59.619+0000 7f0af75b3640 1 -- 172.21.15.33:0/2107669412 wait complete. 2024-09-16T00:42:59.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:59 smithi033 ceph-mon[31243]: pgmap v1125: 129 pgs: 129 active+clean; 621 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 7.2 MiB/s wr, 445 op/s 2024-09-16T00:42:59.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:42:59 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1681290932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:42:59.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:59 smithi083 ceph-mon[35242]: pgmap v1125: 129 pgs: 129 active+clean; 621 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 7.2 MiB/s wr, 445 op/s 2024-09-16T00:42:59.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:42:59 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1681290932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:42:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:59 smithi060 ceph-mon[35185]: pgmap v1125: 129 pgs: 129 active+clean; 621 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 7.2 MiB/s wr, 445 op/s 2024-09-16T00:42:59.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:42:59 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1681290932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:43:00.336 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-16T00:43:00.342 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:00.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.852+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1592718883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d48102570 msgr2=0x7f0d481029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:00.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.852+0000 7f0d4f94d640 1 --2- 172.21.15.33:0/1592718883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d48102570 0x7f0d481029b0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009ca0 tx=0x7f0d4002a950 comp rx=0 tx=0).stop 2024-09-16T00:43:00.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.852+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1592718883 shutdown_connections 2024-09-16T00:43:00.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.852+0000 7f0d4f94d640 1 --2- 172.21.15.33:0/1592718883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d48102ef0 0x7f0d4810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.852+0000 7f0d4f94d640 1 --2- 172.21.15.33:0/1592718883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d48102570 0x7f0d481029b0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.852+0000 7f0d4f94d640 1 --2- 172.21.15.33:0/1592718883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d48100e20 0x7f0d48101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.852+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1592718883 >> 172.21.15.33:0/1592718883 conn(0x7f0d480fc7d0 msgr2=0x7f0d480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:00.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.852+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1592718883 shutdown_connections 2024-09-16T00:43:00.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.852+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1592718883 wait complete. 2024-09-16T00:43:00.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.853+0000 7f0d4f94d640 1 Processor -- start 2024-09-16T00:43:00.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.853+0000 7f0d4f94d640 1 -- start start 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4f94d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d48100e20 0x7f0d481970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4f94d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d48102570 0x7f0d481975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4f94d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d48102ef0 0x7f0d48197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4f94d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d48110f50 con 0x7f0d48100e20 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4f94d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0d48110dd0 con 0x7f0d48102570 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4f94d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0d481110d0 con 0x7f0d48102ef0 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4d6e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d48100e20 0x7f0d481970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4d6e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d48100e20 0x7f0d481970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39000/0 (socket says 172.21.15.33:39000) 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/1000923181 learned_addr learned my addr 172.21.15.33:0/1000923181 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4deea640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d48102ef0 0x7f0d48197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4cee8640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d48102570 0x7f0d481975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/1000923181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d48102ef0 msgr2=0x7f0d48197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:00.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4d6e9640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d48102ef0 0x7f0d48197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/1000923181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d48102570 msgr2=0x7f0d481975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:00.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4d6e9640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d48102570 0x7f0d481975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/1000923181 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d481a6920 con 0x7f0d48100e20 2024-09-16T00:43:00.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4deea640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d48102ef0 0x7f0d48197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:43:00.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.854+0000 7f0d4cee8640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d48102570 0x7f0d481975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:43:00.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.855+0000 7f0d4d6e9640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d48100e20 0x7f0d481970b0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400c920 tx=0x7f0d4400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:00.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.855+0000 7f0d3e7fc640 1 -- 172.21.15.33:0/1000923181 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d44007c70 con 0x7f0d48100e20 2024-09-16T00:43:00.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.855+0000 7f0d3e7fc640 1 -- 172.21.15.33:0/1000923181 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d44007e10 con 0x7f0d48100e20 2024-09-16T00:43:00.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.855+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1000923181 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d481a6c10 con 0x7f0d48100e20 2024-09-16T00:43:00.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.855+0000 7f0d3e7fc640 1 -- 172.21.15.33:0/1000923181 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d44018890 con 0x7f0d48100e20 2024-09-16T00:43:00.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.855+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1000923181 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d481a71a0 con 0x7f0d48100e20 2024-09-16T00:43:00.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.857+0000 7f0d3e7fc640 1 -- 172.21.15.33:0/1000923181 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0d44018a70 con 0x7f0d48100e20 2024-09-16T00:43:00.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.858+0000 7f0d51d26740 1 Processor -- start 2024-09-16T00:43:00.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.858+0000 7f0d51d26740 1 -- start start 2024-09-16T00:43:00.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.858+0000 7f0d3e7fc640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d340a32c0 0x7f0d340a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.858+0000 7f0d51d26740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb3ba00 0x5634cbb3bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.858+0000 7f0d51d26740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3c320 0x5634cbb466a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.858+0000 7f0d51d26740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb46be0 0x5634cbb48fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d51d26740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5634cbaa0200 con 0x5634cbb46be0 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d51d26740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5634cba46dc0 con 0x5634cbb3c320 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d51d26740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5634cba9c1c0 con 0x5634cbb3ba00 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d3e7fc640 1 -- 172.21.15.33:0/1000923181 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0d440cfef0 con 0x7f0d48100e20 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d4deea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb46be0 0x5634cbb48fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d4deea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb46be0 0x5634cbb48fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39010/0 (socket says 172.21.15.33:39010) 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d4deea640 1 -- 172.21.15.33:0/3884781219 learned_addr learned my addr 172.21.15.33:0/3884781219 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d4deea640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d340a32c0 0x7f0d340a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d4deea640 1 -- 172.21.15.33:0/3884781219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb3ba00 msgr2=0x5634cbb3bde0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d4deea640 1 --2- 172.21.15.33:0/3884781219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb3ba00 0x5634cbb3bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d4deea640 1 -- 172.21.15.33:0/3884781219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3c320 msgr2=0x5634cbb466a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T00:43:00.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d4deea640 1 --2- 172.21.15.33:0/3884781219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3c320 0x5634cbb466a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d4deea640 1 -- 172.21.15.33:0/3884781219 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634cb811620 con 0x5634cbb46be0 2024-09-16T00:43:00.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d4deea640 1 --2- 172.21.15.33:0/3884781219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb46be0 0x5634cbb48fd0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f0d38012ab0 tx=0x7f0d38012f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff76c03b9ce6952 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:00.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.859+0000 7f0d4deea640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d340a32c0 0x7f0d340a5780 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0d38010860 tx=0x7f0d38008000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:00.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.860+0000 7f0d337fe640 1 -- 172.21.15.33:0/3884781219 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d38023070 con 0x5634cbb46be0 2024-09-16T00:43:00.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.860+0000 7f0d337fe640 1 -- 172.21.15.33:0/3884781219 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d38009a10 con 0x5634cbb46be0 2024-09-16T00:43:00.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.860+0000 7f0d337fe640 1 -- 172.21.15.33:0/3884781219 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d38012050 con 0x5634cbb46be0 2024-09-16T00:43:00.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.860+0000 7f0d51d26740 1 -- 172.21.15.33:0/3884781219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb46be0 msgr2=0x5634cbb48fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:00.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.860+0000 7f0d51d26740 1 --2- 172.21.15.33:0/3884781219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb46be0 0x5634cbb48fd0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f0d38012ab0 tx=0x7f0d38012f70 comp rx=0 tx=0).stop 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.860+0000 7f0d51d26740 1 -- 172.21.15.33:0/3884781219 shutdown_connections 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.860+0000 7f0d51d26740 1 --2- 172.21.15.33:0/3884781219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb46be0 0x5634cbb48fd0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.860+0000 7f0d51d26740 1 --2- 172.21.15.33:0/3884781219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3c320 0x5634cbb466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.860+0000 7f0d51d26740 1 --2- 172.21.15.33:0/3884781219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb3ba00 0x5634cbb3bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.860+0000 7f0d51d26740 1 -- 172.21.15.33:0/3884781219 >> 172.21.15.33:0/3884781219 conn(0x5634cbb37f90 msgr2=0x5634cbb38370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.861+0000 7f0d51d26740 1 -- 172.21.15.33:0/3884781219 shutdown_connections 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.861+0000 7f0d51d26740 1 -- 172.21.15.33:0/3884781219 wait complete. 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.861+0000 7f0d51d26740 1 -- start start 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.861+0000 7f0d51d26740 1 Processor -- start 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.861+0000 7f0d51d26740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3ba00 0x5634cbb46600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.861+0000 7f0d51d26740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3c320 0x5634cbb438b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.861+0000 7f0d51d26740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb46be0 0x5634cbb43ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.861+0000 7f0d51d26740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5634cbb497b0 con 0x5634cbb3ba00 2024-09-16T00:43:00.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.861+0000 7f0d51d26740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5634cba3ac50 con 0x5634cbb3c320 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.861+0000 7f0d51d26740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5634cbb49930 con 0x5634cbb46be0 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d51d26740 10 client.0 _ll_register_callbacks cb 0x5634cba4cba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d4d6e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3ba00 0x5634cbb46600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d4d6e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3ba00 0x5634cbb46600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39024/0 (socket says 172.21.15.33:39024) 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/2349136053 learned_addr learned my addr 172.21.15.33:0/2349136053 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d4deea640 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb46be0 0x5634cbb43ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d4cee8640 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3c320 0x5634cbb438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/2349136053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb46be0 msgr2=0x5634cbb43ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d4d6e9640 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb46be0 0x5634cbb43ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/2349136053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3c320 msgr2=0x5634cbb438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d4d6e9640 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3c320 0x5634cbb438b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/2349136053 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634cbb446f0 con 0x5634cbb3ba00 2024-09-16T00:43:00.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d4cee8640 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3c320 0x5634cbb438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d4d6e9640 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3ba00 0x5634cbb46600 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f0d440984f0 tx=0x7f0d440056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f36f80326d6954 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.862+0000 7f0d30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d30ff9640 1 -- 172.21.15.33:0/2349136053 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4409c090 con 0x5634cbb3ba00 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d30ff9640 1 -- 172.21.15.33:0/2349136053 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0d44018d60 con 0x5634cbb3ba00 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d51d26740 1 -- 172.21.15.33:0/2349136053 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d38009a10 con 0x5634cbb3ba00 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d30ff9640 1 -- 172.21.15.33:0/2349136053 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0d440a0440 con 0x5634cbb3ba00 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d51d26740 10 client.20211 fetch_fsmap learned FSMap version 35 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d51d26740 1 -- 172.21.15.33:0/2349136053 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5634cbb55490 con 0x5634cbb3ba00 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d30ff9640 1 -- 172.21.15.33:0/2349136053 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d38009a10 con 0x5634cbb3ba00 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d30ff9640 1 -- 172.21.15.33:0/2349136053 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f0d440a4870 con 0x5634cbb3ba00 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d51d26740 10 client.20211 fetch_fsmap finished waiting for FSMap version 35 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d51d26740 10 client.20211 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d51d26740 20 client.20211 populate_metadata read hostname 'smithi033' 2024-09-16T00:43:00.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d51d26740 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634cbb568a0 0x5634cbb58cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d51d26740 4 client.20211 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:43:00.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.863+0000 7f0d51d26740 1 -- 172.21.15.33:0/2349136053 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634cbaad770 con 0x5634cbb568a0 2024-09-16T00:43:00.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.864+0000 7f0d4cee8640 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634cbb568a0 0x5634cbb58cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.864+0000 7f0d4cee8640 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634cbb568a0 0x5634cbb58cd0 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37188f0dfe4e3fdc server_cookie=e84f15c81f05643d in_seq=0 out_seq=0 2024-09-16T00:43:00.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.864+0000 7f0d30ff9640 10 client.20211 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:43:00.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d30ff9640 1 -- 172.21.15.33:0/2349136053 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5634cbaad770 con 0x5634cbb568a0 2024-09-16T00:43:00.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d30ff9640 10 client.20211 handle_command_reply: tid=0 2024-09-16T00:43:00.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d51d26740 1 client.20211 shutdown 2024-09-16T00:43:00.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d51d26740 1 -- 172.21.15.33:0/2349136053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3ba00 msgr2=0x5634cbb46600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:00.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d51d26740 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3ba00 0x5634cbb46600 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f0d440984f0 tx=0x7f0d440056d0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d51d26740 1 -- 172.21.15.33:0/2349136053 shutdown_connections 2024-09-16T00:43:00.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d51d26740 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634cbb568a0 0x5634cbb58cd0 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d51d26740 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb46be0 0x5634cbb43ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d51d26740 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3c320 0x5634cbb438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d30ff9640 0 client.20211 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:43:00.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d51d26740 1 --2- 172.21.15.33:0/2349136053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3ba00 0x5634cbb46600 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d51d26740 1 -- 172.21.15.33:0/2349136053 >> 172.21.15.33:0/2349136053 conn(0x5634cbb37f90 msgr2=0x5634cbb371d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:00.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d51d26740 1 -- 172.21.15.33:0/2349136053 shutdown_connections 2024-09-16T00:43:00.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.865+0000 7f0d51d26740 1 -- 172.21.15.33:0/2349136053 wait complete. 2024-09-16T00:43:00.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.866+0000 7f0d51d26740 20 client.20211 trim_cache size 0 max 16384 2024-09-16T00:43:00.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.893+0000 7f0d51d26740 1 Processor -- start 2024-09-16T00:43:00.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.893+0000 7f0d51d26740 1 -- start start 2024-09-16T00:43:00.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.893+0000 7f0d51d26740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb37f90 0x5634cbb5e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.893+0000 7f0d51d26740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3ba00 0x5634cbb5eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.893+0000 7f0d51d26740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3c320 0x5634cbb4eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.893+0000 7f0d51d26740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d481b3b00 con 0x5634cbb3c320 2024-09-16T00:43:00.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.893+0000 7f0d51d26740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0d481b1370 con 0x5634cbb3ba00 2024-09-16T00:43:00.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.893+0000 7f0d51d26740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0d481af1c0 con 0x5634cbb37f90 2024-09-16T00:43:00.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.893+0000 7f0d4d6e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb37f90 0x5634cbb5e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.893+0000 7f0d4deea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3c320 0x5634cbb4eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.893+0000 7f0d4d6e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb37f90 0x5634cbb5e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.33:59150/0 (socket says 172.21.15.33:59150) 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.893+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/862638873 learned_addr learned my addr 172.21.15.33:0/862638873 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.894+0000 7f0d4cee8640 1 --2- 172.21.15.33:0/862638873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3ba00 0x5634cbb5eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.894+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/862638873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3ba00 msgr2=0x5634cbb5eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.894+0000 7f0d4d6e9640 1 --2- 172.21.15.33:0/862638873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3ba00 0x5634cbb5eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.894+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/862638873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3c320 msgr2=0x5634cbb4eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.894+0000 7f0d4d6e9640 1 --2- 172.21.15.33:0/862638873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3c320 0x5634cbb4eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.894+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/862638873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634cbb44230 con 0x5634cbb37f90 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.894+0000 7f0d4deea640 1 --2- 172.21.15.33:0/862638873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3c320 0x5634cbb4eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.894+0000 7f0d4d6e9640 1 --2- 172.21.15.33:0/862638873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb37f90 0x5634cbb5e5d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0d4409c020 tx=0x7f0d4400cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15587648f019e3a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.894+0000 7f0d32ffd640 1 -- 172.21.15.33:0/862638873 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4409c090 con 0x5634cbb37f90 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.894+0000 7f0d32ffd640 1 -- 172.21.15.33:0/862638873 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d440bc0b0 con 0x5634cbb37f90 2024-09-16T00:43:00.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.894+0000 7f0d32ffd640 1 -- 172.21.15.33:0/862638873 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4409b4e0 con 0x5634cbb37f90 2024-09-16T00:43:00.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.895+0000 7f0d51d26740 1 -- 172.21.15.33:0/862638873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb37f90 msgr2=0x5634cbb5e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:00.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.895+0000 7f0d51d26740 1 --2- 172.21.15.33:0/862638873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb37f90 0x5634cbb5e5d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0d4409c020 tx=0x7f0d4400cf10 comp rx=0 tx=0).stop 2024-09-16T00:43:00.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.895+0000 7f0d51d26740 1 -- 172.21.15.33:0/862638873 shutdown_connections 2024-09-16T00:43:00.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.895+0000 7f0d51d26740 1 --2- 172.21.15.33:0/862638873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb3c320 0x5634cbb4eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.895+0000 7f0d51d26740 1 --2- 172.21.15.33:0/862638873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3ba00 0x5634cbb5eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.895+0000 7f0d51d26740 1 --2- 172.21.15.33:0/862638873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb37f90 0x5634cbb5e5d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.895+0000 7f0d51d26740 1 -- 172.21.15.33:0/862638873 >> 172.21.15.33:0/862638873 conn(0x5634cbb33520 msgr2=0x5634cbb371d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:00.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.896+0000 7f0d51d26740 1 -- 172.21.15.33:0/862638873 shutdown_connections 2024-09-16T00:43:00.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.896+0000 7f0d51d26740 1 -- 172.21.15.33:0/862638873 wait complete. 2024-09-16T00:43:00.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.896+0000 7f0d51d26740 1 -- start start 2024-09-16T00:43:00.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.896+0000 7f0d51d26740 1 Processor -- start 2024-09-16T00:43:00.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d51d26740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb37f90 0x5634cbb4f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d51d26740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3ba00 0x5634cbb4b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d51d26740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb3c320 0x5634cbb4b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d51d26740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d481af830 con 0x5634cbb37f90 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d51d26740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0d481b0ca0 con 0x5634cbb3ba00 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d51d26740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0d481aff00 con 0x5634cbb3c320 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d51d26740 10 client.0 _ll_register_callbacks cb 0x7f0d4409bbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d4d6e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb37f90 0x5634cbb4f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d4d6e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb37f90 0x5634cbb4f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39056/0 (socket says 172.21.15.33:39056) 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/80516508 learned_addr learned my addr 172.21.15.33:0/80516508 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d4cee8640 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3ba00 0x5634cbb4b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d4deea640 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb3c320 0x5634cbb4b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/80516508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb3c320 msgr2=0x5634cbb4b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d4d6e9640 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb3c320 0x5634cbb4b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/80516508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3ba00 msgr2=0x5634cbb4b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:00.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d4d6e9640 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3ba00 0x5634cbb4b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d4d6e9640 1 -- 172.21.15.33:0/80516508 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634cbb3c150 con 0x5634cbb37f90 2024-09-16T00:43:00.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d4cee8640 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3ba00 0x5634cbb4b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:43:00.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d4d6e9640 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb37f90 0x5634cbb4f130 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f0d440984f0 tx=0x7f0d4409bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163c7eece2704ff8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:00.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.897+0000 7f0d30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:43:00.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d30ff9640 1 -- 172.21.15.33:0/80516508 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4409c090 con 0x5634cbb37f90 2024-09-16T00:43:00.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d30ff9640 1 -- 172.21.15.33:0/80516508 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0d4400de00 con 0x5634cbb37f90 2024-09-16T00:43:00.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d51d26740 1 -- 172.21.15.33:0/80516508 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d440bc0b0 con 0x5634cbb37f90 2024-09-16T00:43:00.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d30ff9640 1 -- 172.21.15.33:0/80516508 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0d440a9050 con 0x5634cbb37f90 2024-09-16T00:43:00.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d51d26740 10 client.20217 fetch_fsmap learned FSMap version 35 2024-09-16T00:43:00.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d51d26740 1 -- 172.21.15.33:0/80516508 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5634cbbc21f0 con 0x5634cbb37f90 2024-09-16T00:43:00.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d30ff9640 1 -- 172.21.15.33:0/80516508 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d4409b4e0 con 0x5634cbb37f90 2024-09-16T00:43:00.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d30ff9640 1 -- 172.21.15.33:0/80516508 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f0d440a15d0 con 0x5634cbb37f90 2024-09-16T00:43:00.901 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:00.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d51d26740 10 client.20217 fetch_fsmap finished waiting for FSMap version 35 2024-09-16T00:43:00.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d51d26740 10 client.20217 resolve_mds: validated gid 24494 aka daemon mds.b 2024-09-16T00:43:00.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d51d26740 20 client.20217 populate_metadata read hostname 'smithi033' 2024-09-16T00:43:00.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d51d26740 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634cbb46be0 0x5634cbbc4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d51d26740 4 client.20217 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.898+0000 7f0d51d26740 1 -- 172.21.15.33:0/80516508 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x7f0d440bc0b0 con 0x5634cbb46be0 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.899+0000 7f0d4cee8640 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634cbb46be0 0x5634cbbc4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.899+0000 7f0d4cee8640 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634cbb46be0 0x5634cbbc4690 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f378e3c13d88513 server_cookie=631eb84f5f3361c0 in_seq=0 out_seq=0 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.899+0000 7f0d30ff9640 10 client.20217 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.899+0000 7f0d30ff9640 1 -- 172.21.15.33:0/80516508 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x7f0d440bc0b0 con 0x5634cbb46be0 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.899+0000 7f0d30ff9640 10 client.20217 handle_command_reply: tid=0 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.899+0000 7f0d51d26740 1 client.20217 shutdown 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.899+0000 7f0d51d26740 1 -- 172.21.15.33:0/80516508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb37f90 msgr2=0x5634cbb4f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.899+0000 7f0d51d26740 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb37f90 0x5634cbb4f130 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f0d440984f0 tx=0x7f0d4409bd00 comp rx=0 tx=0).stop 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.900+0000 7f0d51d26740 1 -- 172.21.15.33:0/80516508 shutdown_connections 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.900+0000 7f0d51d26740 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5634cbb46be0 0x5634cbbc4690 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.900+0000 7f0d51d26740 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5634cbb3c320 0x5634cbb4b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.900+0000 7f0d51d26740 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5634cbb3ba00 0x5634cbb4b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.900+0000 7f0d51d26740 1 --2- 172.21.15.33:0/80516508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634cbb37f90 0x5634cbb4f130 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.900+0000 7f0d51d26740 1 -- 172.21.15.33:0/80516508 >> 172.21.15.33:0/80516508 conn(0x5634cbb33520 msgr2=0x5634cbb371d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:00.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.900+0000 7f0d30ff9640 0 client.20217 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:43:00.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.900+0000 7f0d51d26740 1 -- 172.21.15.33:0/80516508 shutdown_connections 2024-09-16T00:43:00.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.900+0000 7f0d51d26740 1 -- 172.21.15.33:0/80516508 wait complete. 2024-09-16T00:43:00.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.900+0000 7f0d51d26740 20 client.20217 trim_cache size 0 max 16384 2024-09-16T00:43:00.904 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:43:00.904 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 35, 2024-09-16T00:43:00.904 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24494, 2024-09-16T00:43:00.904 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 88, 2024-09-16T00:43:00.904 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": {} 2024-09-16T00:43:00.905 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:43:00.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.901+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1000923181 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d340a32c0 msgr2=0x7f0d340a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:00.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.901+0000 7f0d4f94d640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d340a32c0 0x7f0d340a5780 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0d38010860 tx=0x7f0d38008000 comp rx=0 tx=0).stop 2024-09-16T00:43:00.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.901+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1000923181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d48100e20 msgr2=0x7f0d481970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:00.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.901+0000 7f0d4f94d640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d48100e20 0x7f0d481970b0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400c920 tx=0x7f0d4400cde0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.901+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1000923181 shutdown_connections 2024-09-16T00:43:00.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.901+0000 7f0d4f94d640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0d340a32c0 0x7f0d340a5780 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.901+0000 7f0d4f94d640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d48102ef0 0x7f0d48197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.901+0000 7f0d4f94d640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d48102570 0x7f0d481975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.901+0000 7f0d4f94d640 1 --2- 172.21.15.33:0/1000923181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d48100e20 0x7f0d481970b0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:00.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.901+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1000923181 >> 172.21.15.33:0/1000923181 conn(0x7f0d480fc7d0 msgr2=0x7f0d48105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:00.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.901+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1000923181 shutdown_connections 2024-09-16T00:43:00.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:00.901+0000 7f0d4f94d640 1 -- 172.21.15.33:0/1000923181 wait complete. 2024-09-16T00:43:01.827 INFO:tasks.quiescer:done joining Quiescers 2024-09-16T00:43:01.827 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-09-16T00:43:01.838 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-09-16T00:43:01.838 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:01 smithi033 ceph-mon[31243]: pgmap v1126: 129 pgs: 129 active+clean; 560 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.4 MiB/s wr, 480 op/s 2024-09-16T00:43:01.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:01 smithi083 ceph-mon[35242]: pgmap v1126: 129 pgs: 129 active+clean; 560 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.4 MiB/s wr, 480 op/s 2024-09-16T00:43:01.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:01 smithi060 ceph-mon[35185]: pgmap v1126: 129 pgs: 129 active+clean; 560 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.4 MiB/s wr, 480 op/s 2024-09-16T00:43:02.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.035+0000 7f1f02dfd640 1 -- 172.21.15.33:0/727554890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1efc100a80 msgr2=0x7f1efc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:02.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.035+0000 7f1f02dfd640 1 --2- 172.21.15.33:0/727554890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1efc100a80 0x7f1efc100ee0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8009930 tx=0x7f1ee802a420 comp rx=0 tx=0).stop 2024-09-16T00:43:02.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.035+0000 7f1f02dfd640 1 -- 172.21.15.33:0/727554890 shutdown_connections 2024-09-16T00:43:02.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.035+0000 7f1f02dfd640 1 --2- 172.21.15.33:0/727554890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1efc101420 0x7f1efc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.035+0000 7f1f02dfd640 1 --2- 172.21.15.33:0/727554890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1efc100a80 0x7f1efc100ee0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.035+0000 7f1f02dfd640 1 --2- 172.21.15.33:0/727554890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1efc106a80 0x7f1efc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.035+0000 7f1f02dfd640 1 -- 172.21.15.33:0/727554890 >> 172.21.15.33:0/727554890 conn(0x7f1efc0fc7a0 msgr2=0x7f1efc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:02.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.035+0000 7f1f02dfd640 1 -- 172.21.15.33:0/727554890 shutdown_connections 2024-09-16T00:43:02.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.035+0000 7f1f02dfd640 1 -- 172.21.15.33:0/727554890 wait complete. 2024-09-16T00:43:02.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f02dfd640 1 Processor -- start 2024-09-16T00:43:02.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f02dfd640 1 -- start start 2024-09-16T00:43:02.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f02dfd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1efc100a80 0x7f1efc0fce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f02dfd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1efc101420 0x7f1efc0fd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f02dfd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1efc106a80 0x7f1efc0fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f02dfd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc110e80 con 0x7f1efc106a80 2024-09-16T00:43:02.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f02dfd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc110d00 con 0x7f1efc100a80 2024-09-16T00:43:02.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f02dfd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc111000 con 0x7f1efc101420 2024-09-16T00:43:02.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f0139a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1efc106a80 0x7f1efc0fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f0139a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1efc106a80 0x7f1efc0fd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39080/0 (socket says 172.21.15.33:39080) 2024-09-16T00:43:02.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f0139a640 1 -- 172.21.15.33:0/537198991 learned_addr learned my addr 172.21.15.33:0/537198991 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:02.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f00b99640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1efc100a80 0x7f1efc0fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1efbfff640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1efc101420 0x7f1efc0fd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f0139a640 1 -- 172.21.15.33:0/537198991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1efc101420 msgr2=0x7f1efc0fd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:02.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f0139a640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1efc101420 0x7f1efc0fd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f0139a640 1 -- 172.21.15.33:0/537198991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1efc100a80 msgr2=0x7f1efc0fce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:02.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f0139a640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1efc100a80 0x7f1efc0fce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.036+0000 7f1f0139a640 1 -- 172.21.15.33:0/537198991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1efc1a6960 con 0x7f1efc106a80 2024-09-16T00:43:02.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.037+0000 7f1f00b99640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1efc100a80 0x7f1efc0fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:43:02.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.037+0000 7f1f0139a640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1efc106a80 0x7f1efc0fd8f0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009fd0 tx=0x7f1ef400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:02.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.037+0000 7f1ef9ffb640 1 -- 172.21.15.33:0/537198991 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4017070 con 0x7f1efc106a80 2024-09-16T00:43:02.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.037+0000 7f1ef9ffb640 1 -- 172.21.15.33:0/537198991 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef4003e50 con 0x7f1efc106a80 2024-09-16T00:43:02.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.037+0000 7f1ef9ffb640 1 -- 172.21.15.33:0/537198991 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef40106c0 con 0x7f1efc106a80 2024-09-16T00:43:02.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.037+0000 7f1f02dfd640 1 -- 172.21.15.33:0/537198991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1efc1a6c50 con 0x7f1efc106a80 2024-09-16T00:43:02.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.037+0000 7f1f02dfd640 1 -- 172.21.15.33:0/537198991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1efc1a7110 con 0x7f1efc106a80 2024-09-16T00:43:02.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.039+0000 7f1ef9ffb640 1 -- 172.21.15.33:0/537198991 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1ef4003970 con 0x7f1efc106a80 2024-09-16T00:43:02.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f051d6740 1 Processor -- start 2024-09-16T00:43:02.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f051d6740 1 -- start start 2024-09-16T00:43:02.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f051d6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61e6b40 0x5621f61e5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f051d6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61e5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f051d6740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61f0370 0x5621f61f2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f051d6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5621f614b2f0 con 0x5621f61e55d0 2024-09-16T00:43:02.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f051d6740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5621f60f2cf0 con 0x5621f61f0370 2024-09-16T00:43:02.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f051d6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5621f61472b0 con 0x5621f61e6b40 2024-09-16T00:43:02.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f00b99640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61e5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f00b99640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61e5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39096/0 (socket says 172.21.15.33:39096) 2024-09-16T00:43:02.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f00b99640 1 -- 172.21.15.33:0/4017753133 learned_addr learned my addr 172.21.15.33:0/4017753133 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:02.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1efbfff640 1 --2- 172.21.15.33:0/4017753133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61e6b40 0x5621f61e5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1ef9ffb640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1eec0a3240 0x7f1eec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1efbfff640 1 --2- 172.21.15.33:0/4017753133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61f0370 0x5621f61f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f00b99640 1 -- 172.21.15.33:0/4017753133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61e6b40 msgr2=0x5621f61e5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:02.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f00b99640 1 --2- 172.21.15.33:0/4017753133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61e6b40 0x5621f61e5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f00b99640 1 -- 172.21.15.33:0/4017753133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61f0370 msgr2=0x5621f61f2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:02.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f00b99640 1 --2- 172.21.15.33:0/4017753133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61f0370 0x5621f61f2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f00b99640 1 -- 172.21.15.33:0/4017753133 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621f60d73a0 con 0x5621f61e55d0 2024-09-16T00:43:02.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1efbfff640 1 --2- 172.21.15.33:0/4017753133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61e6b40 0x5621f61e5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:43:02.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1ef9ffb640 1 -- 172.21.15.33:0/537198991 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1ef40cfc40 con 0x7f1efc106a80 2024-09-16T00:43:02.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.040+0000 7f1f0139a640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1eec0a3240 0x7f1eec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.041+0000 7f1f00b99640 1 --2- 172.21.15.33:0/4017753133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61e5a10 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000bcc0 tx=0x7f1ef000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f20714fb95e29b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:02.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.041+0000 7f1ee6ffd640 1 -- 172.21.15.33:0/4017753133 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0015070 con 0x5621f61e55d0 2024-09-16T00:43:02.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.041+0000 7f1ee6ffd640 1 -- 172.21.15.33:0/4017753133 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef001a940 con 0x5621f61e55d0 2024-09-16T00:43:02.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.041+0000 7f1ee6ffd640 1 -- 172.21.15.33:0/4017753133 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0011a30 con 0x5621f61e55d0 2024-09-16T00:43:02.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.042+0000 7f1f051d6740 1 -- 172.21.15.33:0/4017753133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 msgr2=0x5621f61e5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:02.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.042+0000 7f1f0139a640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1eec0a3240 0x7f1eec0a5700 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40981e0 tx=0x7f1ef4004020 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:02.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.042+0000 7f1f051d6740 1 --2- 172.21.15.33:0/4017753133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61e5a10 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000bcc0 tx=0x7f1ef000f730 comp rx=0 tx=0).stop 2024-09-16T00:43:02.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.042+0000 7f1f051d6740 1 -- 172.21.15.33:0/4017753133 shutdown_connections 2024-09-16T00:43:02.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.042+0000 7f1f051d6740 1 --2- 172.21.15.33:0/4017753133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61f0370 0x5621f61f2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.042+0000 7f1f051d6740 1 --2- 172.21.15.33:0/4017753133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61e5a10 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.042+0000 7f1f051d6740 1 --2- 172.21.15.33:0/4017753133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61e6b40 0x5621f61e5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.042+0000 7f1f051d6740 1 -- 172.21.15.33:0/4017753133 >> 172.21.15.33:0/4017753133 conn(0x5621f61ae310 msgr2=0x5621f61e1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:02.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.042+0000 7f1f051d6740 1 -- 172.21.15.33:0/4017753133 shutdown_connections 2024-09-16T00:43:02.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.042+0000 7f1f051d6740 1 -- 172.21.15.33:0/4017753133 wait complete. 2024-09-16T00:43:02.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.042+0000 7f1f051d6740 1 -- start start 2024-09-16T00:43:02.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.042+0000 7f1f051d6740 1 Processor -- start 2024-09-16T00:43:02.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f051d6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61e55d0 0x5621f61efd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f051d6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e6b40 0x5621f61ecfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f051d6740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61f0370 0x5621f61ed560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f051d6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5621f61f3070 con 0x5621f61e6b40 2024-09-16T00:43:02.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f051d6740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5621f6106060 con 0x5621f61f0370 2024-09-16T00:43:02.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f051d6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5621f61f31f0 con 0x5621f61e55d0 2024-09-16T00:43:02.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f051d6740 10 client.0 _ll_register_callbacks cb 0x5621f5e598c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:43:02.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f00b99640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e6b40 0x5621f61ecfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f00b99640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e6b40 0x5621f61ecfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39104/0 (socket says 172.21.15.33:39104) 2024-09-16T00:43:02.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f00b99640 1 -- 172.21.15.33:0/3158699405 learned_addr learned my addr 172.21.15.33:0/3158699405 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:02.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1efbfff640 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61f0370 0x5621f61ed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f00b99640 1 -- 172.21.15.33:0/3158699405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61e55d0 msgr2=0x5621f61efd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:02.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1efbfff640 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61e55d0 0x5621f61efd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f00b99640 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61e55d0 0x5621f61efd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f00b99640 1 -- 172.21.15.33:0/3158699405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61f0370 msgr2=0x5621f61ed560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:02.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f00b99640 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61f0370 0x5621f61ed560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f00b99640 1 -- 172.21.15.33:0/3158699405 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621f5ebc3c0 con 0x5621f61e6b40 2024-09-16T00:43:02.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1efbfff640 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61e55d0 0x5621f61efd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:43:02.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1f00b99640 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e6b40 0x5621f61ecfc0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000fc30 tx=0x7f1ef000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a19684d2bb94255 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:02.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1edbfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:43:02.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.043+0000 7f1edbfff640 1 -- 172.21.15.33:0/3158699405 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0015070 con 0x5621f61e6b40 2024-09-16T00:43:02.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1edbfff640 1 -- 172.21.15.33:0/3158699405 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1ef001a460 con 0x5621f61e6b40 2024-09-16T00:43:02.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1f051d6740 1 -- 172.21.15.33:0/3158699405 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ef001a940 con 0x5621f61e6b40 2024-09-16T00:43:02.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1edbfff640 1 -- 172.21.15.33:0/3158699405 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1ef002e050 con 0x5621f61e6b40 2024-09-16T00:43:02.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1f051d6740 10 client.20229 fetch_fsmap learned FSMap version 35 2024-09-16T00:43:02.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1f051d6740 1 -- 172.21.15.33:0/3158699405 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5621f61feaf0 con 0x5621f61e6b40 2024-09-16T00:43:02.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1edbfff640 1 -- 172.21.15.33:0/3158699405 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef001a940 con 0x5621f61e6b40 2024-09-16T00:43:02.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1edbfff640 1 -- 172.21.15.33:0/3158699405 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f1ef00117b0 con 0x5621f61e6b40 2024-09-16T00:43:02.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1f051d6740 10 client.20229 fetch_fsmap finished waiting for FSMap version 35 2024-09-16T00:43:02.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1f051d6740 10 client.20229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:43:02.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1f051d6740 20 client.20229 populate_metadata read hostname 'smithi033' 2024-09-16T00:43:02.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1f051d6740 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5621f61fff50 0x5621f6202340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1f051d6740 4 client.20229 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T00:43:02.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.044+0000 7f1f051d6740 1 -- 172.21.15.33:0/3158699405 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621f6158860 con 0x5621f61fff50 2024-09-16T00:43:02.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.045+0000 7f1efbfff640 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5621f61fff50 0x5621f6202340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.045+0000 7f1efbfff640 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5621f61fff50 0x5621f6202340 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5c3ca1d9b0bd36 server_cookie=edc42da49e142513 in_seq=0 out_seq=0 2024-09-16T00:43:02.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.045+0000 7f1edbfff640 10 client.20229 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:43:02.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1edbfff640 1 -- 172.21.15.33:0/3158699405 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5621f6158860 con 0x5621f61fff50 2024-09-16T00:43:02.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1edbfff640 10 client.20229 handle_command_reply: tid=0 2024-09-16T00:43:02.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1f051d6740 1 client.20229 shutdown 2024-09-16T00:43:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1f051d6740 1 -- 172.21.15.33:0/3158699405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e6b40 msgr2=0x5621f61ecfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1f051d6740 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e6b40 0x5621f61ecfc0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000fc30 tx=0x7f1ef000b990 comp rx=0 tx=0).stop 2024-09-16T00:43:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1f051d6740 1 -- 172.21.15.33:0/3158699405 shutdown_connections 2024-09-16T00:43:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1f051d6740 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5621f61fff50 0x5621f6202340 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1f051d6740 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61f0370 0x5621f61ed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1f051d6740 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e6b40 0x5621f61ecfc0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1f051d6740 1 --2- 172.21.15.33:0/3158699405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61e55d0 0x5621f61efd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1edbfff640 0 client.20229 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:43:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1f051d6740 1 -- 172.21.15.33:0/3158699405 >> 172.21.15.33:0/3158699405 conn(0x5621f61ae310 msgr2=0x5621f61e7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1f051d6740 1 -- 172.21.15.33:0/3158699405 shutdown_connections 2024-09-16T00:43:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1f051d6740 1 -- 172.21.15.33:0/3158699405 wait complete. 2024-09-16T00:43:02.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.046+0000 7f1f051d6740 20 client.20229 trim_cache size 0 max 16384 2024-09-16T00:43:02.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1f051d6740 1 Processor -- start 2024-09-16T00:43:02.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1f051d6740 1 -- start start 2024-09-16T00:43:02.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1f051d6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61fc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1f051d6740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61e6b40 0x5621f61ec880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1f051d6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61f0370 0x5621f62079f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1f051d6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1b3bd0 con 0x5621f61e55d0 2024-09-16T00:43:02.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1f051d6740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1b1440 con 0x5621f61e6b40 2024-09-16T00:43:02.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1f051d6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1af290 con 0x5621f61f0370 2024-09-16T00:43:02.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1efbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61fc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1efbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61fc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39114/0 (socket says 172.21.15.33:39114) 2024-09-16T00:43:02.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1efbfff640 1 -- 172.21.15.33:0/2932321891 learned_addr learned my addr 172.21.15.33:0/2932321891 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:02.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1f00b99640 1 --2- 172.21.15.33:0/2932321891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61e6b40 0x5621f61ec880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1efbfff640 1 -- 172.21.15.33:0/2932321891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61f0370 msgr2=0x5621f62079f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:02.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1efbfff640 1 --2- 172.21.15.33:0/2932321891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61f0370 0x5621f62079f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1efbfff640 1 -- 172.21.15.33:0/2932321891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61e6b40 msgr2=0x5621f61ec880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:02.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1efbfff640 1 --2- 172.21.15.33:0/2932321891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61e6b40 0x5621f61ec880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.060+0000 7f1efbfff640 1 -- 172.21.15.33:0/2932321891 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621f61e1de0 con 0x5621f61e55d0 2024-09-16T00:43:02.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.061+0000 7f1efbfff640 1 --2- 172.21.15.33:0/2932321891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61fc340 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8000c00 tx=0x7f1ee8005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a5b2142a06b4d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:02.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.061+0000 7f1ee67fc640 1 -- 172.21.15.33:0/2932321891 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8035040 con 0x5621f61e55d0 2024-09-16T00:43:02.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.061+0000 7f1ee67fc640 1 -- 172.21.15.33:0/2932321891 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ee804c070 con 0x5621f61e55d0 2024-09-16T00:43:02.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.061+0000 7f1ee67fc640 1 -- 172.21.15.33:0/2932321891 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8004bf0 con 0x5621f61e55d0 2024-09-16T00:43:02.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.062+0000 7f1f051d6740 1 -- 172.21.15.33:0/2932321891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 msgr2=0x5621f61fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:02.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.062+0000 7f1f051d6740 1 --2- 172.21.15.33:0/2932321891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61fc340 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8000c00 tx=0x7f1ee8005b50 comp rx=0 tx=0).stop 2024-09-16T00:43:02.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.062+0000 7f1f051d6740 1 -- 172.21.15.33:0/2932321891 shutdown_connections 2024-09-16T00:43:02.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.062+0000 7f1f051d6740 1 --2- 172.21.15.33:0/2932321891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61f0370 0x5621f62079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.062+0000 7f1f051d6740 1 --2- 172.21.15.33:0/2932321891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61e6b40 0x5621f61ec880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.062+0000 7f1f051d6740 1 --2- 172.21.15.33:0/2932321891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61fc340 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.062+0000 7f1f051d6740 1 -- 172.21.15.33:0/2932321891 >> 172.21.15.33:0/2932321891 conn(0x5621f61ae310 msgr2=0x5621f61ec0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:02.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.062+0000 7f1f051d6740 1 -- 172.21.15.33:0/2932321891 shutdown_connections 2024-09-16T00:43:02.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.062+0000 7f1f051d6740 1 -- 172.21.15.33:0/2932321891 wait complete. 2024-09-16T00:43:02.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.062+0000 7f1f051d6740 1 -- start start 2024-09-16T00:43:02.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.062+0000 7f1f051d6740 1 Processor -- start 2024-09-16T00:43:02.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1f051d6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61f88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1f051d6740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61e6b40 0x5621f61e9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1f051d6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61f0370 0x5621f61e9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1f051d6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1af900 con 0x5621f61e55d0 2024-09-16T00:43:02.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1f051d6740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1b0d70 con 0x5621f61e6b40 2024-09-16T00:43:02.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1f051d6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1affd0 con 0x5621f61f0370 2024-09-16T00:43:02.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1f051d6740 10 client.0 _ll_register_callbacks cb 0x7f1ef000bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T00:43:02.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1efbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61f88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1efbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61f88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39120/0 (socket says 172.21.15.33:39120) 2024-09-16T00:43:02.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1efbfff640 1 -- 172.21.15.33:0/4116609252 learned_addr learned my addr 172.21.15.33:0/4116609252 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:02.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1f00b99640 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61e6b40 0x5621f61e9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1efbfff640 1 -- 172.21.15.33:0/4116609252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61f0370 msgr2=0x5621f61e9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:02.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1efbfff640 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61f0370 0x5621f61e9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1efbfff640 1 -- 172.21.15.33:0/4116609252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61e6b40 msgr2=0x5621f61e9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T00:43:02.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1efbfff640 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61e6b40 0x5621f61e9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1efbfff640 1 -- 172.21.15.33:0/4116609252 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621f61ea3e0 con 0x5621f61e55d0 2024-09-16T00:43:02.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1f00b99640 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61e6b40 0x5621f61e9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:43:02.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1efbfff640 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61f88d0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8004810 tx=0x7f1ee80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae01b752ad50f506 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:02.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1edbfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-16T00:43:02.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1edbfff640 1 -- 172.21.15.33:0/4116609252 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8035040 con 0x5621f61e55d0 2024-09-16T00:43:02.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1f051d6740 1 -- 172.21.15.33:0/4116609252 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ee804c070 con 0x5621f61e55d0 2024-09-16T00:43:02.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.063+0000 7f1edbfff640 1 -- 172.21.15.33:0/4116609252 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f1ee8003800 con 0x5621f61e55d0 2024-09-16T00:43:02.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.064+0000 7f1edbfff640 1 -- 172.21.15.33:0/4116609252 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1ee8003a00 con 0x5621f61e55d0 2024-09-16T00:43:02.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.064+0000 7f1f051d6740 10 client.20235 fetch_fsmap learned FSMap version 35 2024-09-16T00:43:02.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.064+0000 7f1f051d6740 1 -- 172.21.15.33:0/4116609252 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5621f626a6b0 con 0x5621f61e55d0 2024-09-16T00:43:02.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.064+0000 7f1edbfff640 1 -- 172.21.15.33:0/4116609252 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ee804c070 con 0x5621f61e55d0 2024-09-16T00:43:02.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.064+0000 7f1edbfff640 1 -- 172.21.15.33:0/4116609252 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f1ee803a040 con 0x5621f61e55d0 2024-09-16T00:43:02.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.064+0000 7f1f051d6740 10 client.20235 fetch_fsmap finished waiting for FSMap version 35 2024-09-16T00:43:02.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.064+0000 7f1f051d6740 10 client.20235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-16T00:43:02.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.064+0000 7f1f051d6740 20 client.20235 populate_metadata read hostname 'smithi033' 2024-09-16T00:43:02.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.064+0000 7f1f051d6740 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5621f626ad80 0x5621f626d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:02.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.064+0000 7f1f051d6740 4 client.20235 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-16T00:43:02.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.064+0000 7f1f051d6740 1 -- 172.21.15.33:0/4116609252 --> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5621f6152d70 con 0x5621f626ad80 2024-09-16T00:43:02.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.064+0000 7f1f00b99640 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5621f626ad80 0x5621f626d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:02.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1f00b99640 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5621f626ad80 0x5621f626d1a0 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b29ea3334f614bcc server_cookie=de7a4889e27b0a96 in_seq=0 out_seq=0 2024-09-16T00:43:02.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1edbfff640 10 client.20235 ms_handle_connect on v2:172.21.15.60:6838/1222932045 2024-09-16T00:43:02.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1edbfff640 1 -- 172.21.15.33:0/4116609252 <== mds.0 v2:172.21.15.60:6838/1222932045 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5621f6152d70 con 0x5621f626ad80 2024-09-16T00:43:02.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1edbfff640 10 client.20235 handle_command_reply: tid=0 2024-09-16T00:43:02.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1f051d6740 1 client.20235 shutdown 2024-09-16T00:43:02.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1f051d6740 1 -- 172.21.15.33:0/4116609252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 msgr2=0x5621f61f88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:02.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1f051d6740 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61f88d0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8004810 tx=0x7f1ee80042f0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1f051d6740 1 -- 172.21.15.33:0/4116609252 shutdown_connections 2024-09-16T00:43:02.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1f051d6740 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.60:6838/1222932045,v1:172.21.15.60:6839/1222932045] conn(0x5621f626ad80 0x5621f626d1a0 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1f051d6740 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621f61f0370 0x5621f61e9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1f051d6740 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5621f61e6b40 0x5621f61e9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1edbfff640 0 client.20235 ms_handle_reset on v2:172.21.15.60:6838/1222932045 2024-09-16T00:43:02.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1f051d6740 1 --2- 172.21.15.33:0/4116609252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621f61e55d0 0x5621f61f88d0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1f051d6740 1 -- 172.21.15.33:0/4116609252 >> 172.21.15.33:0/4116609252 conn(0x5621f61ae310 msgr2=0x5621f61e7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:02.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1f051d6740 1 -- 172.21.15.33:0/4116609252 shutdown_connections 2024-09-16T00:43:02.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.065+0000 7f1f051d6740 1 -- 172.21.15.33:0/4116609252 wait complete. 2024-09-16T00:43:02.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f051d6740 20 client.20235 trim_cache size 0 max 16384 2024-09-16T00:43:02.075 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-09-16T00:43:02.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f02dfd640 1 -- 172.21.15.33:0/537198991 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1eec0a3240 msgr2=0x7f1eec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:02.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f02dfd640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1eec0a3240 0x7f1eec0a5700 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40981e0 tx=0x7f1ef4004020 comp rx=0 tx=0).stop 2024-09-16T00:43:02.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f02dfd640 1 -- 172.21.15.33:0/537198991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1efc106a80 msgr2=0x7f1efc0fd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:02.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f02dfd640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1efc106a80 0x7f1efc0fd8f0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009fd0 tx=0x7f1ef400b260 comp rx=0 tx=0).stop 2024-09-16T00:43:02.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f02dfd640 1 -- 172.21.15.33:0/537198991 shutdown_connections 2024-09-16T00:43:02.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f02dfd640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f1eec0a3240 0x7f1eec0a5700 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f02dfd640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1efc106a80 0x7f1efc0fd8f0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f02dfd640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1efc101420 0x7f1efc0fd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f02dfd640 1 --2- 172.21.15.33:0/537198991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1efc100a80 0x7f1efc0fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:02.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f02dfd640 1 -- 172.21.15.33:0/537198991 >> 172.21.15.33:0/537198991 conn(0x7f1efc0fc7a0 msgr2=0x7f1efc06aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:02.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f02dfd640 1 -- 172.21.15.33:0/537198991 shutdown_connections 2024-09-16T00:43:02.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:02.066+0000 7f1f02dfd640 1 -- 172.21.15.33:0/537198991 wait complete. 2024-09-16T00:43:02.866 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-09-16T00:43:02.867 INFO:tasks.fwd_scrub:done joining 2024-09-16T00:43:02.867 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-09-16T00:43:02.877 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T00:43:03.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.011+0000 7f9bf112a640 1 -- 172.21.15.33:0/2593436475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bec104f00 msgr2=0x7f9bec1072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.011+0000 7f9bf112a640 1 --2- 172.21.15.33:0/2593436475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bec104f00 0x7f9bec1072f0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009900 tx=0x7f9be002a410 comp rx=0 tx=0).stop 2024-09-16T00:43:03.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.011+0000 7f9bf112a640 1 -- 172.21.15.33:0/2593436475 shutdown_connections 2024-09-16T00:43:03.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.011+0000 7f9bf112a640 1 --2- 172.21.15.33:0/2593436475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bec107830 0x7f9bec109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.011+0000 7f9bf112a640 1 --2- 172.21.15.33:0/2593436475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bec104f00 0x7f9bec1072f0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.011+0000 7f9bf112a640 1 --2- 172.21.15.33:0/2593436475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bec1025d0 0x7f9bec1049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.011+0000 7f9bf112a640 1 -- 172.21.15.33:0/2593436475 >> 172.21.15.33:0/2593436475 conn(0x7f9bec0fc190 msgr2=0x7f9bec0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:03.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.011+0000 7f9bf112a640 1 -- 172.21.15.33:0/2593436475 shutdown_connections 2024-09-16T00:43:03.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.011+0000 7f9bf112a640 1 -- 172.21.15.33:0/2593436475 wait complete. 2024-09-16T00:43:03.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.012+0000 7f9bf112a640 1 Processor -- start 2024-09-16T00:43:03.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.012+0000 7f9bf112a640 1 -- start start 2024-09-16T00:43:03.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.012+0000 7f9bf112a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bec1025d0 0x7f9bec197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.012+0000 7f9bf112a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bec104f00 0x7f9bec198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.012+0000 7f9bf112a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bec107830 0x7f9bec19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.012+0000 7f9bf112a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec10d810 con 0x7f9bec104f00 2024-09-16T00:43:03.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.012+0000 7f9bf112a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec10d690 con 0x7f9bec1025d0 2024-09-16T00:43:03.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.012+0000 7f9bf112a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec10d990 con 0x7f9bec107830 2024-09-16T00:43:03.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.012+0000 7f9bea59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bec104f00 0x7f9bec198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.012+0000 7f9bea59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bec104f00 0x7f9bec198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39140/0 (socket says 172.21.15.33:39140) 2024-09-16T00:43:03.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.012+0000 7f9bea59c640 1 -- 172.21.15.33:0/4237130616 learned_addr learned my addr 172.21.15.33:0/4237130616 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:03.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.012+0000 7f9beb59e640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bec107830 0x7f9bec19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bead9d640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bec1025d0 0x7f9bec197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bea59c640 1 -- 172.21.15.33:0/4237130616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bec107830 msgr2=0x7f9bec19f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bea59c640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bec107830 0x7f9bec19f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bea59c640 1 -- 172.21.15.33:0/4237130616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bec1025d0 msgr2=0x7f9bec197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bea59c640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bec1025d0 0x7f9bec197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bea59c640 1 -- 172.21.15.33:0/4237130616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bec19fae0 con 0x7f9bec104f00 2024-09-16T00:43:03.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9beb59e640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bec107830 0x7f9bec19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:43:03.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bead9d640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bec1025d0 0x7f9bec197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:43:03.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bea59c640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bec104f00 0x7f9bec198400 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f9be0031610 tx=0x7f9be00043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:03.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bd3fff640 1 -- 172.21.15.33:0/4237130616 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be003f030 con 0x7f9bec104f00 2024-09-16T00:43:03.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bd3fff640 1 -- 172.21.15.33:0/4237130616 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be0003ce0 con 0x7f9bec104f00 2024-09-16T00:43:03.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bf112a640 1 -- 172.21.15.33:0/4237130616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bec19fd70 con 0x7f9bec104f00 2024-09-16T00:43:03.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bd3fff640 1 -- 172.21.15.33:0/4237130616 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0035070 con 0x7f9bec104f00 2024-09-16T00:43:03.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.013+0000 7f9bf112a640 1 -- 172.21.15.33:0/4237130616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bec1a0200 con 0x7f9bec104f00 2024-09-16T00:43:03.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.014+0000 7f9bf112a640 1 -- 172.21.15.33:0/4237130616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bec0fdc80 con 0x7f9bec104f00 2024-09-16T00:43:03.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.015+0000 7f9bd3fff640 1 -- 172.21.15.33:0/4237130616 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9be0042030 con 0x7f9bec104f00 2024-09-16T00:43:03.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.016+0000 7f9bd3fff640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9bd40a3250 0x7f9bd40a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.016+0000 7f9bead9d640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9bd40a3250 0x7f9bd40a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.016+0000 7f9bd3fff640 1 -- 172.21.15.33:0/4237130616 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9be00f0440 con 0x7f9bec104f00 2024-09-16T00:43:03.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.017+0000 7f9bead9d640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9bd40a3250 0x7f9bd40a5710 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc005d00 tx=0x7f9bdc005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:03.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.021+0000 7f9bd3fff640 1 -- 172.21.15.33:0/4237130616 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9be00bcc80 con 0x7f9bec104f00 2024-09-16T00:43:03.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.197+0000 7f9bf112a640 1 -- 172.21.15.33:0/4237130616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9bec062e90 con 0x7f9bec104f00 2024-09-16T00:43:03.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.198+0000 7f9bd3fff640 1 -- 172.21.15.33:0/4237130616 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25076 (secure 0 0 0) 0x7f9be00a2210 con 0x7f9bec104f00 2024-09-16T00:43:03.199 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:43:03.199 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":90,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:17:09.569270+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2e","mappings":[{"from":1,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:43:03.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.200+0000 7f9bf112a640 1 -- 172.21.15.33:0/4237130616 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9bd40a3250 msgr2=0x7f9bd40a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.200+0000 7f9bf112a640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9bd40a3250 0x7f9bd40a5710 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc005d00 tx=0x7f9bdc005ed0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.200+0000 7f9bf112a640 1 -- 172.21.15.33:0/4237130616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bec104f00 msgr2=0x7f9bec198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.200+0000 7f9bf112a640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bec104f00 0x7f9bec198400 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f9be0031610 tx=0x7f9be00043b0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.201+0000 7f9bf112a640 1 -- 172.21.15.33:0/4237130616 shutdown_connections 2024-09-16T00:43:03.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.201+0000 7f9bf112a640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9bd40a3250 0x7f9bd40a5710 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.201+0000 7f9bf112a640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bec107830 0x7f9bec19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.201+0000 7f9bf112a640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bec104f00 0x7f9bec198400 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.201+0000 7f9bf112a640 1 --2- 172.21.15.33:0/4237130616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bec1025d0 0x7f9bec197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.201+0000 7f9bf112a640 1 -- 172.21.15.33:0/4237130616 >> 172.21.15.33:0/4237130616 conn(0x7f9bec0fc190 msgr2=0x7f9bec102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:03.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.201+0000 7f9bf112a640 1 -- 172.21.15.33:0/4237130616 shutdown_connections 2024-09-16T00:43:03.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.201+0000 7f9bf112a640 1 -- 172.21.15.33:0/4237130616 wait complete. 2024-09-16T00:43:03.213 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T00:10:24.062476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-16T00:15:02.871178+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-16T00:15:05.511764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-16T00:15:11.993036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-16T00:43:03.213 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T00:43:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.323+0000 7efd01726640 1 -- 172.21.15.33:0/1098655085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcfc0ffe10 msgr2=0x7efcfc100210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.323+0000 7efd01726640 1 --2- 172.21.15.33:0/1098655085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcfc0ffe10 0x7efcfc100210 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7efcec009900 tx=0x7efcec02a410 comp rx=0 tx=0).stop 2024-09-16T00:43:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.323+0000 7efd01726640 1 -- 172.21.15.33:0/1098655085 shutdown_connections 2024-09-16T00:43:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.323+0000 7efd01726640 1 --2- 172.21.15.33:0/1098655085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcfc103840 0x7efcfc10a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.323+0000 7efd01726640 1 --2- 172.21.15.33:0/1098655085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcfc102e80 0x7efcfc103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.323+0000 7efd01726640 1 --2- 172.21.15.33:0/1098655085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcfc0ffe10 0x7efcfc100210 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.323+0000 7efd01726640 1 -- 172.21.15.33:0/1098655085 >> 172.21.15.33:0/1098655085 conn(0x7efcfc0fb8f0 msgr2=0x7efcfc0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.323+0000 7efd01726640 1 -- 172.21.15.33:0/1098655085 shutdown_connections 2024-09-16T00:43:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.323+0000 7efd01726640 1 -- 172.21.15.33:0/1098655085 wait complete. 2024-09-16T00:43:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.324+0000 7efd01726640 1 Processor -- start 2024-09-16T00:43:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.324+0000 7efd01726640 1 -- start start 2024-09-16T00:43:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.324+0000 7efd01726640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcfc0ffe10 0x7efcfc197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.324+0000 7efd01726640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcfc102e80 0x7efcfc1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.324+0000 7efd01726640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcfc103840 0x7efcfc19f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.324+0000 7efd01726640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efcfc10d7c0 con 0x7efcfc103840 2024-09-16T00:43:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.324+0000 7efd01726640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efcfc10d640 con 0x7efcfc0ffe10 2024-09-16T00:43:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.324+0000 7efd01726640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efcfc10d940 con 0x7efcfc102e80 2024-09-16T00:43:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efcfb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcfc103840 0x7efcfc19f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efcfb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcfc103840 0x7efcfc19f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39166/0 (socket says 172.21.15.33:39166) 2024-09-16T00:43:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efcfb7fe640 1 -- 172.21.15.33:0/2037972045 learned_addr learned my addr 172.21.15.33:0/2037972045 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efcfa7fc640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcfc102e80 0x7efcfc1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efcfaffd640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcfc0ffe10 0x7efcfc197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efcfb7fe640 1 -- 172.21.15.33:0/2037972045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcfc102e80 msgr2=0x7efcfc1983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efcfb7fe640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcfc102e80 0x7efcfc1983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efcfb7fe640 1 -- 172.21.15.33:0/2037972045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcfc0ffe10 msgr2=0x7efcfc197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efcfb7fe640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcfc0ffe10 0x7efcfc197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efcfb7fe640 1 -- 172.21.15.33:0/2037972045 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcfc19fac0 con 0x7efcfc103840 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efcfb7fe640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcfc103840 0x7efcfc19f450 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7efcf400cad0 tx=0x7efcf400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efce3fff640 1 -- 172.21.15.33:0/2037972045 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf400ae20 con 0x7efcfc103840 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efd01726640 1 -- 172.21.15.33:0/2037972045 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcfc19fdb0 con 0x7efcfc103840 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efce3fff640 1 -- 172.21.15.33:0/2037972045 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcf4005020 con 0x7efcfc103840 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efce3fff640 1 -- 172.21.15.33:0/2037972045 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf400f6f0 con 0x7efcfc103840 2024-09-16T00:43:03.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.325+0000 7efd01726640 1 -- 172.21.15.33:0/2037972045 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcfc1a0270 con 0x7efcfc103840 2024-09-16T00:43:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.326+0000 7efd01726640 1 -- 172.21.15.33:0/2037972045 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcfc109070 con 0x7efcfc103840 2024-09-16T00:43:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.328+0000 7efce3fff640 1 -- 172.21.15.33:0/2037972045 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7efcf40051d0 con 0x7efcfc103840 2024-09-16T00:43:03.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.329+0000 7efce3fff640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efce40a3250 0x7efce40a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.329+0000 7efce3fff640 1 -- 172.21.15.33:0/2037972045 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7efcf40d00e0 con 0x7efcfc103840 2024-09-16T00:43:03.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.329+0000 7efcfaffd640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efce40a3250 0x7efce40a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.330+0000 7efcfaffd640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efce40a3250 0x7efce40a5710 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7efcec005ab0 tx=0x7efcec000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:03.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.332+0000 7efce3fff640 1 -- 172.21.15.33:0/2037972045 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7efcf4014030 con 0x7efcfc103840 2024-09-16T00:43:03.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.510+0000 7efd01726640 1 -- 172.21.15.33:0/2037972045 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efcfc02d950 con 0x7efcfc103840 2024-09-16T00:43:03.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.511+0000 7efce3fff640 1 -- 172.21.15.33:0/2037972045 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7efcf409c330 con 0x7efcfc103840 2024-09-16T00:43:03.511 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-09-16T00:43:03.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.513+0000 7efd01726640 1 -- 172.21.15.33:0/2037972045 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efce40a3250 msgr2=0x7efce40a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.513+0000 7efd01726640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efce40a3250 0x7efce40a5710 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7efcec005ab0 tx=0x7efcec000bc0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.513+0000 7efd01726640 1 -- 172.21.15.33:0/2037972045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcfc103840 msgr2=0x7efcfc19f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.513+0000 7efd01726640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcfc103840 0x7efcfc19f450 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7efcf400cad0 tx=0x7efcf400cf90 comp rx=0 tx=0).stop 2024-09-16T00:43:03.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.513+0000 7efd01726640 1 -- 172.21.15.33:0/2037972045 shutdown_connections 2024-09-16T00:43:03.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.513+0000 7efd01726640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7efce40a3250 0x7efce40a5710 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.513+0000 7efd01726640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcfc103840 0x7efcfc19f450 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.513+0000 7efd01726640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcfc102e80 0x7efcfc1983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.513+0000 7efd01726640 1 --2- 172.21.15.33:0/2037972045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcfc0ffe10 0x7efcfc197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.513+0000 7efd01726640 1 -- 172.21.15.33:0/2037972045 >> 172.21.15.33:0/2037972045 conn(0x7efcfc0fb8f0 msgr2=0x7efcfc106240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:03.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.513+0000 7efd01726640 1 -- 172.21.15.33:0/2037972045 shutdown_connections 2024-09-16T00:43:03.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.513+0000 7efd01726640 1 -- 172.21.15.33:0/2037972045 wait complete. 2024-09-16T00:43:03.525 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-16T00:43:03.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.632+0000 7f0c93ea6640 1 -- 172.21.15.33:0/2852625461 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c107830 msgr2=0x7f0c8c109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.632+0000 7f0c93ea6640 1 --2- 172.21.15.33:0/2852625461 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c107830 0x7f0c8c109ca0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009ca0 tx=0x7f0c8402a8c0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.632+0000 7f0c93ea6640 1 -- 172.21.15.33:0/2852625461 shutdown_connections 2024-09-16T00:43:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.632+0000 7f0c93ea6640 1 --2- 172.21.15.33:0/2852625461 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c107830 0x7f0c8c109ca0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.632+0000 7f0c93ea6640 1 --2- 172.21.15.33:0/2852625461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c8c104f00 0x7f0c8c1072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.632+0000 7f0c93ea6640 1 --2- 172.21.15.33:0/2852625461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c8c1025d0 0x7f0c8c1049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.632+0000 7f0c93ea6640 1 -- 172.21.15.33:0/2852625461 >> 172.21.15.33:0/2852625461 conn(0x7f0c8c0fc190 msgr2=0x7f0c8c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.632+0000 7f0c93ea6640 1 -- 172.21.15.33:0/2852625461 shutdown_connections 2024-09-16T00:43:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.632+0000 7f0c93ea6640 1 -- 172.21.15.33:0/2852625461 wait complete. 2024-09-16T00:43:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c93ea6640 1 Processor -- start 2024-09-16T00:43:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c93ea6640 1 -- start start 2024-09-16T00:43:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c93ea6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c1025d0 0x7f0c8c101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c93ea6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c8c104f00 0x7f0c8c100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c93ea6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c8c107830 0x7f0c8c100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c93ea6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c10d860 con 0x7f0c8c1025d0 2024-09-16T00:43:03.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c93ea6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c10d6e0 con 0x7f0c8c107830 2024-09-16T00:43:03.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c93ea6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c10d9e0 con 0x7f0c8c104f00 2024-09-16T00:43:03.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c91c42640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c1025d0 0x7f0c8c101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c91c42640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c1025d0 0x7f0c8c101f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39172/0 (socket says 172.21.15.33:39172) 2024-09-16T00:43:03.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c91c42640 1 -- 172.21.15.33:0/9893406 learned_addr learned my addr 172.21.15.33:0/9893406 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:03.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c91441640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c8c104f00 0x7f0c8c100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.633+0000 7f0c92443640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c8c107830 0x7f0c8c100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.634+0000 7f0c91c42640 1 -- 172.21.15.33:0/9893406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c8c104f00 msgr2=0x7f0c8c100580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.634+0000 7f0c91c42640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c8c104f00 0x7f0c8c100580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.634+0000 7f0c91c42640 1 -- 172.21.15.33:0/9893406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c8c107830 msgr2=0x7f0c8c100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.634+0000 7f0c91c42640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c8c107830 0x7f0c8c100ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.634+0000 7f0c91c42640 1 -- 172.21.15.33:0/9893406 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c8c101350 con 0x7f0c8c1025d0 2024-09-16T00:43:03.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.634+0000 7f0c91c42640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c1025d0 0x7f0c8c101f40 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800d950 tx=0x7f0c8800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:03.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.634+0000 7f0c82ffd640 1 -- 172.21.15.33:0/9893406 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c88017030 con 0x7f0c8c1025d0 2024-09-16T00:43:03.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.634+0000 7f0c82ffd640 1 -- 172.21.15.33:0/9893406 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c88004020 con 0x7f0c8c1025d0 2024-09-16T00:43:03.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.634+0000 7f0c93ea6640 1 -- 172.21.15.33:0/9893406 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c8c1a1fb0 con 0x7f0c8c1025d0 2024-09-16T00:43:03.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.634+0000 7f0c82ffd640 1 -- 172.21.15.33:0/9893406 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c880047a0 con 0x7f0c8c1025d0 2024-09-16T00:43:03.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.634+0000 7f0c93ea6640 1 -- 172.21.15.33:0/9893406 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c8c1a2410 con 0x7f0c8c1025d0 2024-09-16T00:43:03.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.635+0000 7f0c93ea6640 1 -- 172.21.15.33:0/9893406 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c8c1052e0 con 0x7f0c8c1025d0 2024-09-16T00:43:03.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.636+0000 7f0c82ffd640 1 -- 172.21.15.33:0/9893406 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0c880041d0 con 0x7f0c8c1025d0 2024-09-16T00:43:03.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.637+0000 7f0c82ffd640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0c780a3250 0x7f0c780a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.637+0000 7f0c91441640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0c780a3250 0x7f0c780a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.637+0000 7f0c82ffd640 1 -- 172.21.15.33:0/9893406 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0c880cf8c0 con 0x7f0c8c1025d0 2024-09-16T00:43:03.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.638+0000 7f0c91441640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0c780a3250 0x7f0c780a5710 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c101cb0 tx=0x7f0c7c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:03.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.642+0000 7f0c82ffd640 1 -- 172.21.15.33:0/9893406 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0c8809c100 con 0x7f0c8c1025d0 2024-09-16T00:43:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.819+0000 7f0c93ea6640 1 -- 172.21.15.33:0/9893406 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0c8c062df0 con 0x7f0c8c1025d0 2024-09-16T00:43:03.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.819+0000 7f0c82ffd640 1 -- 172.21.15.33:0/9893406 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f0c8809ba00 con 0x7f0c8c1025d0 2024-09-16T00:43:03.824 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-09-16T00:43:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.821+0000 7f0c93ea6640 1 -- 172.21.15.33:0/9893406 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0c780a3250 msgr2=0x7f0c780a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.821+0000 7f0c93ea6640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0c780a3250 0x7f0c780a5710 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c101cb0 tx=0x7f0c7c005f90 comp rx=0 tx=0).stop 2024-09-16T00:43:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.821+0000 7f0c93ea6640 1 -- 172.21.15.33:0/9893406 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c1025d0 msgr2=0x7f0c8c101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.821+0000 7f0c93ea6640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c1025d0 0x7f0c8c101f40 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800d950 tx=0x7f0c8800de10 comp rx=0 tx=0).stop 2024-09-16T00:43:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.821+0000 7f0c93ea6640 1 -- 172.21.15.33:0/9893406 shutdown_connections 2024-09-16T00:43:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.821+0000 7f0c93ea6640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0c780a3250 0x7f0c780a5710 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.821+0000 7f0c93ea6640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c8c107830 0x7f0c8c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.821+0000 7f0c93ea6640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c8c104f00 0x7f0c8c100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.821+0000 7f0c93ea6640 1 --2- 172.21.15.33:0/9893406 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c8c1025d0 0x7f0c8c101f40 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.821+0000 7f0c93ea6640 1 -- 172.21.15.33:0/9893406 >> 172.21.15.33:0/9893406 conn(0x7f0c8c0fc190 msgr2=0x7f0c8c102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.822+0000 7f0c93ea6640 1 -- 172.21.15.33:0/9893406 shutdown_connections 2024-09-16T00:43:03.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.822+0000 7f0c93ea6640 1 -- 172.21.15.33:0/9893406 wait complete. 2024-09-16T00:43:03.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:03 smithi033 ceph-mon[31243]: pgmap v1127: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 370 op/s 2024-09-16T00:43:03.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:03 smithi033 ceph-mon[31243]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:43:03.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:03 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/4237130616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:43:03.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:03 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2037972045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T00:43:03.833 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-16T00:43:03.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:03 smithi083 ceph-mon[35242]: pgmap v1127: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 370 op/s 2024-09-16T00:43:03.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:03 smithi083 ceph-mon[35242]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:43:03.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:03 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/4237130616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:43:03.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:03 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2037972045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T00:43:03.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.942+0000 7f72d27f6640 1 -- 172.21.15.33:0/2012641791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72cc100a20 msgr2=0x7f72cc100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.942+0000 7f72d27f6640 1 --2- 172.21.15.33:0/2012641791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72cc100a20 0x7f72cc100e20 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009980 tx=0x7f72bc02c620 comp rx=0 tx=0).stop 2024-09-16T00:43:03.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.942+0000 7f72d27f6640 1 -- 172.21.15.33:0/2012641791 shutdown_connections 2024-09-16T00:43:03.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.942+0000 7f72d27f6640 1 --2- 172.21.15.33:0/2012641791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72cc1025d0 0x7f72cc108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.942+0000 7f72d27f6640 1 --2- 172.21.15.33:0/2012641791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72cc101c10 0x7f72cc102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.942+0000 7f72d27f6640 1 --2- 172.21.15.33:0/2012641791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72cc100a20 0x7f72cc100e20 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.942+0000 7f72d27f6640 1 -- 172.21.15.33:0/2012641791 >> 172.21.15.33:0/2012641791 conn(0x7f72cc0fc1a0 msgr2=0x7f72cc0fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:03.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.942+0000 7f72d27f6640 1 -- 172.21.15.33:0/2012641791 shutdown_connections 2024-09-16T00:43:03.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.942+0000 7f72d27f6640 1 -- 172.21.15.33:0/2012641791 wait complete. 2024-09-16T00:43:03.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d27f6640 1 Processor -- start 2024-09-16T00:43:03.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d27f6640 1 -- start start 2024-09-16T00:43:03.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d27f6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72cc100a20 0x7f72cc195c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d27f6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72cc101c10 0x7f72cc1961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d27f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72cc1025d0 0x7f72cc19d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d27f6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc10b5e0 con 0x7f72cc1025d0 2024-09-16T00:43:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d27f6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc10b460 con 0x7f72cc101c10 2024-09-16T00:43:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d27f6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc10b760 con 0x7f72cc100a20 2024-09-16T00:43:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d0d93640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72cc1025d0 0x7f72cc19d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d0d93640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72cc1025d0 0x7f72cc19d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39182/0 (socket says 172.21.15.33:39182) 2024-09-16T00:43:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d0d93640 1 -- 172.21.15.33:0/946037590 learned_addr learned my addr 172.21.15.33:0/946037590 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72cbfff640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72cc100a20 0x7f72cc195c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72cb7fe640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72cc101c10 0x7f72cc1961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d0d93640 1 -- 172.21.15.33:0/946037590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72cc100a20 msgr2=0x7f72cc195c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d0d93640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72cc100a20 0x7f72cc195c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d0d93640 1 -- 172.21.15.33:0/946037590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72cc101c10 msgr2=0x7f72cc1961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d0d93640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72cc101c10 0x7f72cc1961c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72d0d93640 1 -- 172.21.15.33:0/946037590 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72cc19d890 con 0x7f72cc1025d0 2024-09-16T00:43:03.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.943+0000 7f72cb7fe640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72cc101c10 0x7f72cc1961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:43:03.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.944+0000 7f72d0d93640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72cc1025d0 0x7f72cc19d1f0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f72c400cce0 tx=0x7f72c400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:03.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.944+0000 7f72c97fa640 1 -- 172.21.15.33:0/946037590 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c400f450 con 0x7f72cc1025d0 2024-09-16T00:43:03.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.944+0000 7f72c97fa640 1 -- 172.21.15.33:0/946037590 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72c400fad0 con 0x7f72cc1025d0 2024-09-16T00:43:03.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.944+0000 7f72d27f6640 1 -- 172.21.15.33:0/946037590 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72cc19db80 con 0x7f72cc1025d0 2024-09-16T00:43:03.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.944+0000 7f72c97fa640 1 -- 172.21.15.33:0/946037590 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4018c10 con 0x7f72cc1025d0 2024-09-16T00:43:03.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.944+0000 7f72d27f6640 1 -- 172.21.15.33:0/946037590 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72cc19e0a0 con 0x7f72cc1025d0 2024-09-16T00:43:03.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.945+0000 7f72d27f6640 1 -- 172.21.15.33:0/946037590 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72cc107de0 con 0x7f72cc1025d0 2024-09-16T00:43:03.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.946+0000 7f72c97fa640 1 -- 172.21.15.33:0/946037590 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f72c400f5f0 con 0x7f72cc1025d0 2024-09-16T00:43:03.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.947+0000 7f72c97fa640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f72b40a32d0 0x7f72b40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:03.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.947+0000 7f72cbfff640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f72b40a32d0 0x7f72b40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:03.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.947+0000 7f72c97fa640 1 -- 172.21.15.33:0/946037590 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f72c40d0470 con 0x7f72cc1025d0 2024-09-16T00:43:03.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.948+0000 7f72cbfff640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f72b40a32d0 0x7f72b40a5790 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f72bc005ab0 tx=0x7f72bc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:03.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:03.951+0000 7f72c97fa640 1 -- 172.21.15.33:0/946037590 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f72c4014030 con 0x7f72cc1025d0 2024-09-16T00:43:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:03 smithi060 ceph-mon[35185]: pgmap v1127: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 370 op/s 2024-09-16T00:43:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:03 smithi060 ceph-mon[35185]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T00:43:03.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:03 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/4237130616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:43:03.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:03 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2037972045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T00:43:04.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.126+0000 7f72d27f6640 1 -- 172.21.15.33:0/946037590 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f72cc062df0 con 0x7f72cc1025d0 2024-09-16T00:43:04.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.126+0000 7f72c97fa640 1 -- 172.21.15.33:0/946037590 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f72c409cd10 con 0x7f72cc1025d0 2024-09-16T00:43:04.127 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-09-16T00:43:04.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.128+0000 7f72d27f6640 1 -- 172.21.15.33:0/946037590 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f72b40a32d0 msgr2=0x7f72b40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:04.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.128+0000 7f72d27f6640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f72b40a32d0 0x7f72b40a5790 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f72bc005ab0 tx=0x7f72bc000bc0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.128+0000 7f72d27f6640 1 -- 172.21.15.33:0/946037590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72cc1025d0 msgr2=0x7f72cc19d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:04.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.128+0000 7f72d27f6640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72cc1025d0 0x7f72cc19d1f0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f72c400cce0 tx=0x7f72c400a620 comp rx=0 tx=0).stop 2024-09-16T00:43:04.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.128+0000 7f72d27f6640 1 -- 172.21.15.33:0/946037590 shutdown_connections 2024-09-16T00:43:04.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.128+0000 7f72d27f6640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f72b40a32d0 0x7f72b40a5790 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.128+0000 7f72d27f6640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72cc1025d0 0x7f72cc19d1f0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.128+0000 7f72d27f6640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72cc101c10 0x7f72cc1961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.128+0000 7f72d27f6640 1 --2- 172.21.15.33:0/946037590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72cc100a20 0x7f72cc195c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.128+0000 7f72d27f6640 1 -- 172.21.15.33:0/946037590 >> 172.21.15.33:0/946037590 conn(0x7f72cc0fc1a0 msgr2=0x7f72cc104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:04.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.128+0000 7f72d27f6640 1 -- 172.21.15.33:0/946037590 shutdown_connections 2024-09-16T00:43:04.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.128+0000 7f72d27f6640 1 -- 172.21.15.33:0/946037590 wait complete. 2024-09-16T00:43:04.140 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-16T00:43:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.246+0000 7f8ea1731640 1 -- 172.21.15.33:0/658537643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e9c102d10 msgr2=0x7f8e9c103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.246+0000 7f8ea1731640 1 --2- 172.21.15.33:0/658537643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e9c102d10 0x7f8e9c103190 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009900 tx=0x7f8e9002a410 comp rx=0 tx=0).stop 2024-09-16T00:43:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.246+0000 7f8ea1731640 1 -- 172.21.15.33:0/658537643 shutdown_connections 2024-09-16T00:43:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.246+0000 7f8ea1731640 1 --2- 172.21.15.33:0/658537643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e9c1036d0 0x7f8e9c109f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.246+0000 7f8ea1731640 1 --2- 172.21.15.33:0/658537643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e9c102d10 0x7f8e9c103190 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.246+0000 7f8ea1731640 1 --2- 172.21.15.33:0/658537643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c101b20 0x7f8e9c101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.246+0000 7f8ea1731640 1 -- 172.21.15.33:0/658537643 >> 172.21.15.33:0/658537643 conn(0x7f8e9c0fd2a0 msgr2=0x7f8e9c0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.246+0000 7f8ea1731640 1 -- 172.21.15.33:0/658537643 shutdown_connections 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.246+0000 7f8ea1731640 1 -- 172.21.15.33:0/658537643 wait complete. 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.246+0000 7f8ea1731640 1 Processor -- start 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8ea1731640 1 -- start start 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8ea1731640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e9c101b20 0x7f8e9c075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8ea1731640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c102d10 0x7f8e9c0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8ea1731640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e9c1036d0 0x7f8e9c074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8ea1731640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9c10d7a0 con 0x7f8e9c101b20 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8ea1731640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9c10d620 con 0x7f8e9c1036d0 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8ea1731640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9c10d920 con 0x7f8e9c102d10 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8e9affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e9c101b20 0x7f8e9c075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8e9affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e9c101b20 0x7f8e9c075fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39192/0 (socket says 172.21.15.33:39192) 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8e9affd640 1 -- 172.21.15.33:0/3542825756 learned_addr learned my addr 172.21.15.33:0/3542825756 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:04.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8e9a7fc640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c102d10 0x7f8e9c0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8e9b7fe640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e9c1036d0 0x7f8e9c074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8e9affd640 1 -- 172.21.15.33:0/3542825756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c102d10 msgr2=0x7f8e9c0745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8e9affd640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c102d10 0x7f8e9c0745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8e9affd640 1 -- 172.21.15.33:0/3542825756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e9c1036d0 msgr2=0x7f8e9c074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8e9affd640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e9c1036d0 0x7f8e9c074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8e9affd640 1 -- 172.21.15.33:0/3542825756 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9c0753b0 con 0x7f8e9c101b20 2024-09-16T00:43:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.247+0000 7f8e9affd640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e9c101b20 0x7f8e9c075fa0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00e9c0 tx=0x7f8e8c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.248+0000 7f8e83fff640 1 -- 172.21.15.33:0/3542825756 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c00cd80 con 0x7f8e9c101b20 2024-09-16T00:43:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.248+0000 7f8ea1731640 1 -- 172.21.15.33:0/3542825756 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e9c1a1fa0 con 0x7f8e9c101b20 2024-09-16T00:43:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.248+0000 7f8e83fff640 1 -- 172.21.15.33:0/3542825756 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e8c0035f0 con 0x7f8e9c101b20 2024-09-16T00:43:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.248+0000 7f8e83fff640 1 -- 172.21.15.33:0/3542825756 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c010690 con 0x7f8e9c101b20 2024-09-16T00:43:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.248+0000 7f8ea1731640 1 -- 172.21.15.33:0/3542825756 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e9c1a23a0 con 0x7f8e9c101b20 2024-09-16T00:43:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.249+0000 7f8ea1731640 1 -- 172.21.15.33:0/3542825756 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e9c108f50 con 0x7f8e9c101b20 2024-09-16T00:43:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.250+0000 7f8e83fff640 1 -- 172.21.15.33:0/3542825756 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8e8c0037a0 con 0x7f8e9c101b20 2024-09-16T00:43:04.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.251+0000 7f8e83fff640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e840a3250 0x7f8e840a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.251+0000 7f8e9a7fc640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e840a3250 0x7f8e840a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.251+0000 7f8e83fff640 1 -- 172.21.15.33:0/3542825756 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f8e8c0d0100 con 0x7f8e9c101b20 2024-09-16T00:43:04.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.251+0000 7f8e9a7fc640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e840a3250 0x7f8e840a5710 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8e900314e0 tx=0x7f8e90000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:04.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.255+0000 7f8e83fff640 1 -- 172.21.15.33:0/3542825756 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8e8c09c9a0 con 0x7f8e9c101b20 2024-09-16T00:43:04.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.431+0000 7f8ea1731640 1 -- 172.21.15.33:0/3542825756 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8e9c062e20 con 0x7f8e9c101b20 2024-09-16T00:43:04.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.432+0000 7f8e83fff640 1 -- 172.21.15.33:0/3542825756 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f8e8c09c2a0 con 0x7f8e9c101b20 2024-09-16T00:43:04.432 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-09-16T00:43:04.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.434+0000 7f8ea1731640 1 -- 172.21.15.33:0/3542825756 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e840a3250 msgr2=0x7f8e840a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:04.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.434+0000 7f8ea1731640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e840a3250 0x7f8e840a5710 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8e900314e0 tx=0x7f8e90000bc0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.434+0000 7f8ea1731640 1 -- 172.21.15.33:0/3542825756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e9c101b20 msgr2=0x7f8e9c075fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:04.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.434+0000 7f8ea1731640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e9c101b20 0x7f8e9c075fa0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00e9c0 tx=0x7f8e8c00ee80 comp rx=0 tx=0).stop 2024-09-16T00:43:04.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.434+0000 7f8ea1731640 1 -- 172.21.15.33:0/3542825756 shutdown_connections 2024-09-16T00:43:04.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.434+0000 7f8ea1731640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8e840a3250 0x7f8e840a5710 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.434+0000 7f8ea1731640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e9c1036d0 0x7f8e9c074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.434+0000 7f8ea1731640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c102d10 0x7f8e9c0745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.434+0000 7f8ea1731640 1 --2- 172.21.15.33:0/3542825756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e9c101b20 0x7f8e9c075fa0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.434+0000 7f8ea1731640 1 -- 172.21.15.33:0/3542825756 >> 172.21.15.33:0/3542825756 conn(0x7f8e9c0fd2a0 msgr2=0x7f8e9c106120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:04.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.434+0000 7f8ea1731640 1 -- 172.21.15.33:0/3542825756 shutdown_connections 2024-09-16T00:43:04.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.434+0000 7f8ea1731640 1 -- 172.21.15.33:0/3542825756 wait complete. 2024-09-16T00:43:04.445 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-16T00:43:04.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.549+0000 7f0a36c0e640 1 -- 172.21.15.33:0/2690000405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a301025d0 msgr2=0x7f0a301049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:04.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.549+0000 7f0a36c0e640 1 --2- 172.21.15.33:0/2690000405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a301025d0 0x7f0a301049c0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f0a20009900 tx=0x7f0a2002a410 comp rx=0 tx=0).stop 2024-09-16T00:43:04.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.549+0000 7f0a36c0e640 1 -- 172.21.15.33:0/2690000405 shutdown_connections 2024-09-16T00:43:04.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.549+0000 7f0a36c0e640 1 --2- 172.21.15.33:0/2690000405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a30107830 0x7f0a30109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.549+0000 7f0a36c0e640 1 --2- 172.21.15.33:0/2690000405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a30104f00 0x7f0a301072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.549+0000 7f0a36c0e640 1 --2- 172.21.15.33:0/2690000405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a301025d0 0x7f0a301049c0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.549+0000 7f0a36c0e640 1 -- 172.21.15.33:0/2690000405 >> 172.21.15.33:0/2690000405 conn(0x7f0a300fc190 msgr2=0x7f0a300fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:04.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.549+0000 7f0a36c0e640 1 -- 172.21.15.33:0/2690000405 shutdown_connections 2024-09-16T00:43:04.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.550+0000 7f0a36c0e640 1 -- 172.21.15.33:0/2690000405 wait complete. 2024-09-16T00:43:04.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.550+0000 7f0a36c0e640 1 Processor -- start 2024-09-16T00:43:04.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.550+0000 7f0a36c0e640 1 -- start start 2024-09-16T00:43:04.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.550+0000 7f0a36c0e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a301025d0 0x7f0a30101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:04.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.550+0000 7f0a36c0e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a30104f00 0x7f0a30100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:04.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.550+0000 7f0a36c0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a30107830 0x7f0a30100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:04.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.550+0000 7f0a36c0e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3010d810 con 0x7f0a301025d0 2024-09-16T00:43:04.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.550+0000 7f0a36c0e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3010d690 con 0x7f0a30104f00 2024-09-16T00:43:04.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.550+0000 7f0a36c0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3010d990 con 0x7f0a30107830 2024-09-16T00:43:04.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.550+0000 7f0a349aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a301025d0 0x7f0a30101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:04.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.550+0000 7f0a349aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a301025d0 0x7f0a30101f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39222/0 (socket says 172.21.15.33:39222) 2024-09-16T00:43:04.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.550+0000 7f0a349aa640 1 -- 172.21.15.33:0/3034687166 learned_addr learned my addr 172.21.15.33:0/3034687166 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:04.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a351ab640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a30107830 0x7f0a30100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:04.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a2ffff640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a30104f00 0x7f0a30100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a349aa640 1 -- 172.21.15.33:0/3034687166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a30107830 msgr2=0x7f0a30100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a349aa640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a30107830 0x7f0a30100ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a349aa640 1 -- 172.21.15.33:0/3034687166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a30104f00 msgr2=0x7f0a30100580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a349aa640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a30104f00 0x7f0a30100580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a349aa640 1 -- 172.21.15.33:0/3034687166 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a30101350 con 0x7f0a301025d0 2024-09-16T00:43:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a2ffff640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a30104f00 0x7f0a30100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:43:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a349aa640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a301025d0 0x7f0a30101f40 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f0a20009230 tx=0x7f0a20004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a2dffb640 1 -- 172.21.15.33:0/3034687166 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2003f030 con 0x7f0a301025d0 2024-09-16T00:43:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a2dffb640 1 -- 172.21.15.33:0/3034687166 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a20003ce0 con 0x7f0a301025d0 2024-09-16T00:43:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a36c0e640 1 -- 172.21.15.33:0/3034687166 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a301a1f60 con 0x7f0a301025d0 2024-09-16T00:43:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a2dffb640 1 -- 172.21.15.33:0/3034687166 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20035070 con 0x7f0a301025d0 2024-09-16T00:43:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.551+0000 7f0a36c0e640 1 -- 172.21.15.33:0/3034687166 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a301a2370 con 0x7f0a301025d0 2024-09-16T00:43:04.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.552+0000 7f0a36c0e640 1 -- 172.21.15.33:0/3034687166 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a300fdc80 con 0x7f0a301025d0 2024-09-16T00:43:04.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.554+0000 7f0a2dffb640 1 -- 172.21.15.33:0/3034687166 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0a20042030 con 0x7f0a301025d0 2024-09-16T00:43:04.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.554+0000 7f0a2dffb640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0a180a3250 0x7f0a180a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:04.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.554+0000 7f0a2dffb640 1 -- 172.21.15.33:0/3034687166 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f0a200f06f0 con 0x7f0a301025d0 2024-09-16T00:43:04.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.554+0000 7f0a2ffff640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0a180a3250 0x7f0a180a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:04.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.555+0000 7f0a2ffff640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0a180a3250 0x7f0a180a5710 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0a30101cb0 tx=0x7f0a24005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:04.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.557+0000 7f0a2dffb640 1 -- 172.21.15.33:0/3034687166 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0a200bcf90 con 0x7f0a301025d0 2024-09-16T00:43:04.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.783+0000 7f0a36c0e640 1 -- 172.21.15.33:0/3034687166 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f0a3002d940 con 0x7f0a301025d0 2024-09-16T00:43:04.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.791+0000 7f0a2dffb640 1 -- 172.21.15.33:0/3034687166 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+742341 (secure 0 0 0) 0x7f0a200bc890 con 0x7f0a301025d0 2024-09-16T00:43:04.801 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:43:04.801 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:43:04.802 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 20, 2024-09-16T00:43:04.802 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.802 INFO:teuthology.orchestra.run.smithi033.stdout: "active_gid": 14150, 2024-09-16T00:43:04.802 INFO:teuthology.orchestra.run.smithi033.stdout: "active_name": "x", 2024-09-16T00:43:04.802 INFO:teuthology.orchestra.run.smithi033.stdout: "active_addrs": { 2024-09-16T00:43:04.802 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-09-16T00:43:04.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:04.802 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-09-16T00:43:04.802 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6800", 2024-09-16T00:43:04.802 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3427860355 2024-09-16T00:43:04.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6801", 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3427860355 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: "active_addr": "172.21.15.33:6801/3427860355", 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: "active_change": "2024-09-16T00:08:07.548656+0000", 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: "active_mgr_features": 4540701547738038271, 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: "available": true, 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: "standbys": [ 2024-09-16T00:43:04.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:04.804 INFO:teuthology.orchestra.run.smithi033.stdout: "gid": 24109, 2024-09-16T00:43:04.804 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "z", 2024-09-16T00:43:04.804 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_features": 4540701547738038271, 2024-09-16T00:43:04.804 INFO:teuthology.orchestra.run.smithi033.stdout: "available_modules": [ 2024-09-16T00:43:04.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:04.804 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "alerts", 2024-09-16T00:43:04.804 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:04.804 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:04.804 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:04.804 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:04.804 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:04.805 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:04.805 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.805 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.805 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:04.805 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.805 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.805 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.805 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How frequently to reexamine health status", 2024-09-16T00:43:04.805 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.805 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.805 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:04.806 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.807 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.808 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.808 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.808 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.808 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.808 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.808 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.808 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.808 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.808 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:04.808 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:04.808 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.809 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.809 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.809 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:04.809 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.809 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.809 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.809 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:04.809 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:04.809 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:04.809 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.809 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.810 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.811 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.811 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.811 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.811 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.811 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.811 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.811 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.811 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_destination": { 2024-09-16T00:43:04.811 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_destination", 2024-09-16T00:43:04.811 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.811 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.812 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.812 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.812 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.812 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.812 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.812 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Email address to send alerts to", 2024-09-16T00:43:04.812 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.812 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.812 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.812 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_from_name": { 2024-09-16T00:43:04.812 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_from_name", 2024-09-16T00:43:04.813 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.813 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.813 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.813 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "Ceph", 2024-09-16T00:43:04.813 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.813 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.813 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.813 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Email From: name", 2024-09-16T00:43:04.813 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.813 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.813 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_host": { 2024-09-16T00:43:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_host", 2024-09-16T00:43:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP server", 2024-09-16T00:43:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_password": { 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_password", 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Password to authenticate with", 2024-09-16T00:43:04.815 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_port": { 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_port", 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "465", 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.816 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP port", 2024-09-16T00:43:04.817 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.817 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.817 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.817 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_sender": { 2024-09-16T00:43:04.817 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_sender", 2024-09-16T00:43:04.817 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.817 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.817 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.817 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.817 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.817 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.818 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.818 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP envelope sender", 2024-09-16T00:43:04.818 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.818 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.818 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.818 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_ssl": { 2024-09-16T00:43:04.818 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_ssl", 2024-09-16T00:43:04.818 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.818 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.818 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.818 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.819 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.819 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.819 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.819 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-16T00:43:04.819 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.819 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.819 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.819 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_user": { 2024-09-16T00:43:04.819 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_user", 2024-09-16T00:43:04.819 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.819 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.820 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.820 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.820 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.820 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.820 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.820 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "User to authenticate as", 2024-09-16T00:43:04.820 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.820 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.820 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.820 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:04.820 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:04.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:04.821 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "balancer", 2024-09-16T00:43:04.821 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:04.821 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:04.821 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:04.821 INFO:teuthology.orchestra.run.smithi033.stdout: "active": { 2024-09-16T00:43:04.821 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "active", 2024-09-16T00:43:04.821 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.821 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.821 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.821 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "automatically balance PGs across cluster", 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: "begin_time": { 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "begin_time", 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.822 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.823 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0000", 2024-09-16T00:43:04.823 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.823 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.823 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.823 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "beginning time of day to automatically balance", 2024-09-16T00:43:04.823 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T00:43:04.823 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.823 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.823 INFO:teuthology.orchestra.run.smithi033.stdout: "begin_weekday": { 2024-09-16T00:43:04.823 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "begin_weekday", 2024-09-16T00:43:04.823 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0", 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "6", 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_max_iterations": { 2024-09-16T00:43:04.824 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_max_iterations", 2024-09-16T00:43:04.825 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:04.825 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.825 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.825 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "25", 2024-09-16T00:43:04.825 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-09-16T00:43:04.825 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "250", 2024-09-16T00:43:04.825 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.825 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-16T00:43:04.825 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.825 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.825 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_metrics": { 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_metrics", 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "pgs,objects,bytes", 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.826 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_step": { 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_step", 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0.5", 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0.001", 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "0.999", 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "aggressiveness of optimization", 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-16T00:43:04.827 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: "end_time": { 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "end_time", 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "2359", 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "ending time of day to automatically balance", 2024-09-16T00:43:04.828 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T00:43:04.829 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.829 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.829 INFO:teuthology.orchestra.run.smithi033.stdout: "end_weekday": { 2024-09-16T00:43:04.829 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "end_weekday", 2024-09-16T00:43:04.829 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:04.829 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.829 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.829 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:04.829 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0", 2024-09-16T00:43:04.829 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "6", 2024-09-16T00:43:04.829 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.830 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:04.831 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:04.832 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.832 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.832 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.832 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.832 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.832 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.832 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.832 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.832 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.832 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.832 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.834 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: "min_score": { 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "min_score", 2024-09-16T00:43:04.835 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:04.836 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.836 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.836 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:04.836 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.836 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.836 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.836 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-16T00:43:04.836 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.836 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.836 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.836 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": { 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mode", 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "upmap", 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-compat", 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "none", 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "read", 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap", 2024-09-16T00:43:04.837 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap-read" 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Balancer mode", 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_ids": { 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pool_ids", 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.838 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.839 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap_max_deviation": { 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "upmap_max_deviation", 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.840 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.841 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:04.841 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-09-16T00:43:04.841 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.841 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.841 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-16T00:43:04.841 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-16T00:43:04.841 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.841 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.841 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap_max_optimizations": { 2024-09-16T00:43:04.841 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "upmap_max_optimizations", 2024-09-16T00:43:04.841 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:04.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cephadm", 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_down_multiplier": { 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_down_multiplier", 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3.0", 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.843 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_refresh_rate": { 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_refresh_rate", 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "20", 2024-09-16T00:43:04.844 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_starting_port": { 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_starting_port", 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.845 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "4721", 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_ptrace": { 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_ptrace", 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.846 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.847 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.847 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.847 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.847 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.847 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-16T00:43:04.847 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-16T00:43:04.847 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.847 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.847 INFO:teuthology.orchestra.run.smithi033.stdout: "autotune_interval": { 2024-09-16T00:43:04.847 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "autotune_interval", 2024-09-16T00:43:04.847 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:04.848 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.848 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.848 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:04.848 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.848 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.848 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.848 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-16T00:43:04.848 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.848 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.848 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.848 INFO:teuthology.orchestra.run.smithi033.stdout: "autotune_memory_target_ratio": { 2024-09-16T00:43:04.849 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "autotune_memory_target_ratio", 2024-09-16T00:43:04.849 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:04.849 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.849 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.849 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0.7", 2024-09-16T00:43:04.849 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.849 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.849 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.849 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-16T00:43:04.849 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.849 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.849 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm_log_destination": { 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cephadm_log_destination", 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: "file", 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: "file,syslog", 2024-09-16T00:43:04.850 INFO:teuthology.orchestra.run.smithi033.stdout: "syslog" 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: "cgroups_split": { 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cgroups_split", 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.851 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.852 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.852 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.852 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-16T00:43:04.852 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.852 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.852 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.852 INFO:teuthology.orchestra.run.smithi033.stdout: "config_checks_enabled": { 2024-09-16T00:43:04.852 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_checks_enabled", 2024-09-16T00:43:04.852 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.852 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.852 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: "config_dashboard": { 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_dashboard", 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.853 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_alertmanager": { 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_alertmanager", 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.854 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_base": { 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_base", 2024-09-16T00:43:04.855 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Container image name, without the tag", 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_elasticsearch": { 2024-09-16T00:43:04.856 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_elasticsearch", 2024-09-16T00:43:04.857 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.857 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.857 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.857 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-16T00:43:04.857 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.857 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.857 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.857 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "elasticsearch container image", 2024-09-16T00:43:04.857 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.857 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.857 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_grafana": { 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_grafana", 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.859 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_haproxy": { 2024-09-16T00:43:04.859 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_haproxy", 2024-09-16T00:43:04.859 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.859 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.859 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.859 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-16T00:43:04.859 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.859 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.859 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.859 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "HAproxy container image", 2024-09-16T00:43:04.859 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.859 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_agent": { 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_agent", 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger agent container image", 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.860 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.861 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.861 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_collector": { 2024-09-16T00:43:04.861 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_collector", 2024-09-16T00:43:04.861 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.861 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.861 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.861 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-16T00:43:04.861 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.861 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.861 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.861 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger collector container image", 2024-09-16T00:43:04.862 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.862 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.862 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.862 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_query": { 2024-09-16T00:43:04.862 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_query", 2024-09-16T00:43:04.862 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.862 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.862 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.862 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-16T00:43:04.862 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.862 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger query container image", 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_keepalived": { 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_keepalived", 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-16T00:43:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Keepalived container image", 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_loki": { 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_loki", 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.864 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Loki container image", 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_node_exporter": { 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_node_exporter", 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.865 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_nvmeof": { 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_nvmeof", 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.866 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Nvme-of container image", 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_prometheus": { 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_prometheus", 2024-09-16T00:43:04.867 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_promtail": { 2024-09-16T00:43:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_promtail", 2024-09-16T00:43:04.869 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.869 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.869 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.869 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-16T00:43:04.869 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.869 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.869 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.869 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Promtail container image", 2024-09-16T00:43:04.869 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.869 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.869 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.870 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_samba": { 2024-09-16T00:43:04.870 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_samba", 2024-09-16T00:43:04.870 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.870 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.870 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.870 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-16T00:43:04.870 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.870 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.870 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.870 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Samba/SMB container image", 2024-09-16T00:43:04.870 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.870 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_snmp_gateway": { 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_snmp_gateway", 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SNMP Gateway container image", 2024-09-16T00:43:04.871 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.872 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.872 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.872 INFO:teuthology.orchestra.run.smithi033.stdout: "container_init": { 2024-09-16T00:43:04.872 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_init", 2024-09-16T00:43:04.872 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.872 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.872 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.872 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.872 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.872 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.872 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.873 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Run podman/docker with `--init`", 2024-09-16T00:43:04.873 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.873 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.873 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.873 INFO:teuthology.orchestra.run.smithi033.stdout: "daemon_cache_timeout": { 2024-09-16T00:43:04.873 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "daemon_cache_timeout", 2024-09-16T00:43:04.873 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:04.873 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.873 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.873 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:04.873 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: "default_cephadm_command_timeout": { 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "default_cephadm_command_timeout", 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.874 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "900", 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: "default_registry": { 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "default_registry", 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.875 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.876 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io", 2024-09-16T00:43:04.876 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.876 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.876 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.876 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-16T00:43:04.876 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.876 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.876 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.876 INFO:teuthology.orchestra.run.smithi033.stdout: "device_cache_timeout": { 2024-09-16T00:43:04.876 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_cache_timeout", 2024-09-16T00:43:04.876 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:04.877 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.877 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.877 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1800", 2024-09-16T00:43:04.877 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.877 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.877 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.877 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache device inventory", 2024-09-16T00:43:04.877 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.877 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.877 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.877 INFO:teuthology.orchestra.run.smithi033.stdout: "device_enhanced_scan": { 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_enhanced_scan", 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "facts_cache_timeout": { 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "facts_cache_timeout", 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache host facts data", 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.879 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.880 INFO:teuthology.orchestra.run.smithi033.stdout: "grafana_dashboards_path": { 2024-09-16T00:43:04.880 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "grafana_dashboards_path", 2024-09-16T00:43:04.880 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.880 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.880 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.880 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-16T00:43:04.880 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.880 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.880 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.880 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-16T00:43:04.880 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: "host_check_interval": { 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "host_check_interval", 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.881 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to perform a host check", 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: "hw_monitoring": { 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "hw_monitoring", 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.882 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: "inventory_list_all": { 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "inventory_list_all", 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.883 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.884 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.884 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-16T00:43:04.884 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.884 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.884 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.884 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:04.884 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:04.884 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.884 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.884 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.884 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.885 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.886 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.886 INFO:teuthology.orchestra.run.smithi033.stdout: "log_refresh_metadata": { 2024-09-16T00:43:04.886 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_refresh_metadata", 2024-09-16T00:43:04.886 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.886 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.886 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.886 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.886 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.886 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.886 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.887 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.888 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.888 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-16T00:43:04.888 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.888 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.888 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.888 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:04.888 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:04.888 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.888 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.888 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.888 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.889 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.890 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.891 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.891 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.891 INFO:teuthology.orchestra.run.smithi033.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-16T00:43:04.891 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-16T00:43:04.891 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.891 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.891 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.891 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.891 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.891 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.891 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "*", 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.892 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.893 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.893 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-16T00:43:04.893 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.893 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.893 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.893 INFO:teuthology.orchestra.run.smithi033.stdout: "max_count_per_host": { 2024-09-16T00:43:04.893 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_count_per_host", 2024-09-16T00:43:04.893 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.893 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.893 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "max number of daemons per service per host", 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd_draining_count": { 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_osd_draining_count", 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.894 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: "migration_current": { 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "migration_current", 2024-09-16T00:43:04.895 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "internal - do not modify", 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": { 2024-09-16T00:43:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mode", 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "root", 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm-package", 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: "root" 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "mode for remote execution of cephadm", 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.897 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.898 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.898 INFO:teuthology.orchestra.run.smithi033.stdout: "oob_default_addr": { 2024-09-16T00:43:04.898 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "oob_default_addr", 2024-09-16T00:43:04.898 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.898 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.898 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.898 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "169.254.1.1", 2024-09-16T00:43:04.898 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.898 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.898 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.898 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "prometheus_alerts_path": { 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prometheus_alerts_path", 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.899 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-16T00:43:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_insecure": { 2024-09-16T00:43:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_insecure", 2024-09-16T00:43:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_password": { 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_password", 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.901 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_url": { 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_url", 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.902 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.903 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.903 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.903 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.903 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.903 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-16T00:43:04.903 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.903 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.903 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.903 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_username": { 2024-09-16T00:43:04.903 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_username", 2024-09-16T00:43:04.903 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.904 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.904 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.904 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.904 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.904 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.904 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.904 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-16T00:43:04.904 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.904 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.904 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.904 INFO:teuthology.orchestra.run.smithi033.stdout: "secure_monitoring_stack": { 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "secure_monitoring_stack", 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.905 INFO:teuthology.orchestra.run.smithi033.stdout: "service_discovery_port": { 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "service_discovery_port", 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8765", 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "cephadm service discovery port", 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "ssh_config_file": { 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssh_config_file", 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: "use_agent": { 2024-09-16T00:43:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "use_agent", 2024-09-16T00:43:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-16T00:43:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "use_repo_digest": { 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "use_repo_digest", 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-16T00:43:04.909 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.910 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.910 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.910 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.910 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_failed_host_check": { 2024-09-16T00:43:04.910 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_failed_host_check", 2024-09-16T00:43:04.910 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.910 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.910 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.910 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.910 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.910 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.910 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if the host check fails", 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_stray_daemons": { 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_stray_daemons", 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.911 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.912 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-16T00:43:04.912 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.912 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.912 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.912 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_stray_hosts": { 2024-09-16T00:43:04.912 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_stray_hosts", 2024-09-16T00:43:04.912 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.912 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.912 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.912 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.912 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.913 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.913 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.913 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-16T00:43:04.913 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.913 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.913 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.913 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:04.913 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:04.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.913 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:04.913 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crash", 2024-09-16T00:43:04.913 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:04.914 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:04.914 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:04.914 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:04.914 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:04.914 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.914 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.914 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.914 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.914 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.914 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.914 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.914 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:04.915 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:04.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:04.915 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.915 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:04.915 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:04.915 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.915 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.915 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.915 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.915 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.915 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.917 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:04.917 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:04.917 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.917 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.917 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.917 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:04.917 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.917 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.917 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.917 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:04.917 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:04.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.918 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: "retain_interval": { 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "retain_interval", 2024-09-16T00:43:04.919 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:04.920 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.920 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.920 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "31536000", 2024-09-16T00:43:04.920 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.920 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.920 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.920 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-16T00:43:04.920 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.920 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.920 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.920 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_recent_interval": { 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_recent_interval", 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1209600", 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.921 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "dashboard", 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.922 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: "ALERTMANAGER_API_HOST": { 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.923 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.924 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: "AUDIT_API_ENABLED": { 2024-09-16T00:43:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "AUDIT_API_ENABLED", 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-16T00:43:04.926 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.927 INFO:teuthology.orchestra.run.smithi033.stdout: "ENABLE_BROWSABLE_API": { 2024-09-16T00:43:04.928 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-16T00:43:04.928 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.928 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.928 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.928 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.928 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.928 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.928 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.928 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.928 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.928 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.928 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.929 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.929 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-16T00:43:04.929 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-16T00:43:04.929 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.929 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.929 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.929 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.929 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.929 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.929 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.929 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.929 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.930 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.931 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.932 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.932 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.932 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.932 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-16T00:43:04.932 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-16T00:43:04.932 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.932 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.932 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.932 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.932 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.932 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.933 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.934 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.935 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.936 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_PASSWORD": { 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-16T00:43:04.937 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.938 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.938 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.938 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-09-16T00:43:04.938 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.938 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.938 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.938 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.938 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.938 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.938 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.938 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.938 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-16T00:43:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-16T00:43:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_URL": { 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_URL", 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.940 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_USERNAME": { 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.941 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.942 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.942 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.942 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.942 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-16T00:43:04.942 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-16T00:43:04.942 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.942 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.942 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.942 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.942 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.942 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.942 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.943 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.943 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.943 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.943 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.943 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-16T00:43:04.943 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-16T00:43:04.943 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.943 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.943 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.943 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.943 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.944 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.944 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.944 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.944 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.944 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.944 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.944 INFO:teuthology.orchestra.run.smithi033.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-16T00:43:04.944 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-16T00:43:04.944 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.944 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.944 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.945 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: "PROMETHEUS_API_HOST": { 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-16T00:43:04.946 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-16T00:43:04.947 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.948 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-16T00:43:04.949 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-16T00:43:04.949 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.949 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.949 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.949 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.949 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.949 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.949 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.949 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.949 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.949 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.949 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.950 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.952 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.952 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.952 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.952 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-16T00:43:04.952 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-16T00:43:04.952 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.952 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.952 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.952 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.952 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.952 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.953 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.954 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.955 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.955 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.955 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.955 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.955 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.955 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.955 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.955 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-16T00:43:04.955 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-16T00:43:04.955 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.955 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_ENABLED": { 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-16T00:43:04.956 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.957 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.959 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-16T00:43:04.959 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-16T00:43:04.959 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.959 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.959 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.959 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:04.959 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.959 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.959 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.959 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.959 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8", 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.960 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "45", 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.961 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_ACCESS_KEY": { 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-09-16T00:43:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_SECRET_KEY": { 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_SSL_VERIFY": { 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.965 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.966 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.967 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.968 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: "cross_origin_url": { 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cross_origin_url", 2024-09-16T00:43:04.969 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: "crt_file": { 2024-09-16T00:43:04.970 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crt_file", 2024-09-16T00:43:04.971 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.971 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.971 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.971 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.971 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.971 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.971 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.971 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.971 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.971 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.971 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.971 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.972 INFO:teuthology.orchestra.run.smithi033.stdout: "debug": { 2024-09-16T00:43:04.972 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "debug", 2024-09-16T00:43:04.972 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.972 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.972 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.972 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.972 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.972 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.972 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.972 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable/disable debug options", 2024-09-16T00:43:04.972 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.972 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.973 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.973 INFO:teuthology.orchestra.run.smithi033.stdout: "jwt_token_ttl": { 2024-09-16T00:43:04.973 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "jwt_token_ttl", 2024-09-16T00:43:04.973 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.973 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.973 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.973 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "28800", 2024-09-16T00:43:04.973 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.973 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.973 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.973 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: "key_file": { 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "key_file", 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.974 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.975 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.976 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.976 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:04.976 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:04.976 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:04.976 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.976 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:04.976 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:04.976 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.976 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.976 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.976 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.976 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:04.978 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.979 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.980 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.980 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.980 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.980 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.980 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.980 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.980 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.980 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.980 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.980 INFO:teuthology.orchestra.run.smithi033.stdout: "motd": { 2024-09-16T00:43:04.980 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "motd", 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "The message of the day", 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.981 INFO:teuthology.orchestra.run.smithi033.stdout: "redirect_resolve_ip_addr": { 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "redirect_resolve_ip_addr", 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.982 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "::", 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8080", 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.984 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl": { 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl", 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.985 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl_server_port": { 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl_server_port", 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8443", 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.986 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.987 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.987 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.987 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_behaviour": { 2024-09-16T00:43:04.987 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_behaviour", 2024-09-16T00:43:04.987 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.987 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.987 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.987 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "redirect", 2024-09-16T00:43:04.987 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.987 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: "redirect" 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_error_status_code": { 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_error_status_code", 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:04.988 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.989 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.989 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "500", 2024-09-16T00:43:04.989 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "400", 2024-09-16T00:43:04.989 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "599", 2024-09-16T00:43:04.989 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.989 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.989 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.989 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.989 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.989 INFO:teuthology.orchestra.run.smithi033.stdout: "url_prefix": { 2024-09-16T00:43:04.989 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "url_prefix", 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:04.990 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "devicehealth", 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_monitoring": { 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enable_monitoring", 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:04.991 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "monitor device health metrics", 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.992 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.994 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:04.995 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:04.996 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.997 INFO:teuthology.orchestra.run.smithi033.stdout: "mark_out_threshold": { 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mark_out_threshold", 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "2419200", 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:04.998 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": { 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pool_name", 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "device_health_metrics", 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:04.999 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "retention_period": { 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "retention_period", 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15552000", 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long to retain device health metrics", 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.000 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.001 INFO:teuthology.orchestra.run.smithi033.stdout: "scrape_frequency": { 2024-09-16T00:43:05.001 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "scrape_frequency", 2024-09-16T00:43:05.001 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.001 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.001 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.001 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "86400", 2024-09-16T00:43:05.001 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.001 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.001 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.001 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-16T00:43:05.001 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: "self_heal": { 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "self_heal", 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.002 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.003 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to wake up and check device health", 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_threshold": { 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_threshold", 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "7257600", 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.004 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "diskprediction_local", 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.005 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.006 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.006 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.006 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.006 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.006 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.006 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.006 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.006 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.006 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.006 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.006 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.006 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.007 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.008 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.008 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.008 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.008 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.008 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.008 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.008 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.008 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.008 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.008 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.008 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.009 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.010 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: "predict_interval": { 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "predict_interval", 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.011 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "86400", 2024-09-16T00:43:05.012 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.012 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.012 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.012 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.012 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.012 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.012 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.012 INFO:teuthology.orchestra.run.smithi033.stdout: "predictor_model": { 2024-09-16T00:43:05.012 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "predictor_model", 2024-09-16T00:43:05.012 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.012 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "prophetstor", 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:05.013 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.015 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.015 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "influx", 2024-09-16T00:43:05.015 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": false, 2024-09-16T00:43:05.015 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "influxdb python module not found", 2024-09-16T00:43:05.015 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.015 INFO:teuthology.orchestra.run.smithi033.stdout: "batch_size": { 2024-09-16T00:43:05.015 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "batch_size", 2024-09-16T00:43:05.015 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.015 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.015 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.015 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5000", 2024-09-16T00:43:05.015 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: "database": { 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "database", 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.016 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "ceph", 2024-09-16T00:43:05.017 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.017 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.017 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.017 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-16T00:43:05.017 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.017 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.017 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.017 INFO:teuthology.orchestra.run.smithi033.stdout: "hostname": { 2024-09-16T00:43:05.017 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "hostname", 2024-09-16T00:43:05.017 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.017 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.018 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.018 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.018 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.018 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.018 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.018 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB server hostname", 2024-09-16T00:43:05.018 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.018 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.018 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.018 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:05.018 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "30", 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "5", 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.019 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.020 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.020 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.020 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.020 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.020 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.020 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.020 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.020 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.020 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.020 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.020 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.021 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.022 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.022 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.022 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.022 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.022 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.022 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.022 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.022 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.022 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.022 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.022 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.023 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.024 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.024 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.024 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.024 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.024 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.024 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.024 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.024 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.024 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.024 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.024 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: "password": { 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "password", 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.025 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "password of InfluxDB server user", 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: "port": { 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "port", 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.026 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.027 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.027 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8086", 2024-09-16T00:43:05.027 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.027 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.027 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.027 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB server port", 2024-09-16T00:43:05.027 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.027 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.027 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.027 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.027 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl": { 2024-09-16T00:43:05.027 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl", 2024-09-16T00:43:05.028 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.028 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.028 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.028 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "false", 2024-09-16T00:43:05.028 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.028 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.028 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.028 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T00:43:05.028 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.028 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.028 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "threads": { 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "threads", 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "32", 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.029 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: "username": { 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "username", 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "username of InfluxDB server user", 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.030 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: "verify_ssl": { 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "verify_ssl", 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "true", 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T00:43:05.031 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.032 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.032 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.032 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.032 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.032 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.032 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.032 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "insights", 2024-09-16T00:43:05.032 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.032 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.032 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.032 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.032 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.033 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.034 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.035 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.036 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.037 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "iostat", 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.038 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.039 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.040 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.041 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.042 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.042 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.042 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.042 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.042 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.042 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.042 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.042 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.042 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.042 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.042 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.042 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.043 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "k8sevents", 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_event_retention_days": { 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ceph_event_retention_days", 2024-09-16T00:43:05.044 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "7", 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: "config_check_secs": { 2024-09-16T00:43:05.045 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_check_secs", 2024-09-16T00:43:05.046 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.046 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.046 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.046 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.046 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "10", 2024-09-16T00:43:05.046 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.046 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.046 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-16T00:43:05.046 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.046 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.046 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.047 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.048 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.049 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.050 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.051 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.051 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.051 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.051 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.051 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.051 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.051 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.051 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.051 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.051 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.051 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "localpool", 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.052 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "failure_domain": { 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "failure_domain", 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "host", 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "failure domain for any created local pool", 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.053 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.054 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.055 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.057 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.058 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.059 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.059 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.059 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.059 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.059 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.059 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": { 2024-09-16T00:43:05.059 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "min_size", 2024-09-16T00:43:05.059 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.059 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.059 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.059 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default min_size for any created local pool", 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: "num_rep": { 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "num_rep", 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.060 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.061 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3", 2024-09-16T00:43:05.061 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.061 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.061 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.061 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default replica count for any created local pool", 2024-09-16T00:43:05.061 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.061 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.061 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.061 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": { 2024-09-16T00:43:05.061 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pg_num", 2024-09-16T00:43:05.061 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "128", 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default pg_num for any created local pool", 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: "prefix": { 2024-09-16T00:43:05.062 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prefix", 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "name prefix for any created local pool", 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "subtree": { 2024-09-16T00:43:05.063 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "subtree", 2024-09-16T00:43:05.064 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.064 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.064 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.064 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "rack", 2024-09-16T00:43:05.064 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.064 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.064 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.064 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-16T00:43:05.064 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-16T00:43:05.064 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.064 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.064 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.065 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.065 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mds_autoscaler", 2024-09-16T00:43:05.065 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.065 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.065 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.065 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.065 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.065 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.065 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.065 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.066 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.067 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.068 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.068 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.068 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.068 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.068 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.068 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.068 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.068 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.068 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.068 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.068 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.069 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.070 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mirroring", 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.071 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.072 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.073 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.074 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.075 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.076 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.076 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.076 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.076 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.076 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.076 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.076 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.076 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.076 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.076 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.076 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.076 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "nfs", 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.077 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.078 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.079 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.080 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.081 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "orchestrator", 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "fail_fs": { 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "fail_fs", 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.083 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.084 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.086 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.088 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator": { 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "orchestrator", 2024-09-16T00:43:05.089 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm", 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: "rook", 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: "test_orchestrator" 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Orchestrator backend", 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.090 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.091 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.091 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.091 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.091 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.091 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "osd_perf_query", 2024-09-16T00:43:05.091 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.091 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.091 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.091 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.091 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.091 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.092 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.093 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.093 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.093 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.093 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.093 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.093 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.093 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.093 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.093 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.093 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.093 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.097 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.097 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.097 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.097 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.097 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.097 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "osd_support", 2024-09-16T00:43:05.097 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.097 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.097 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.097 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.097 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.098 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.099 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.100 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.100 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.100 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.100 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.100 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.100 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.100 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.100 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.100 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.100 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.100 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.101 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.102 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pg_autoscaler", 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.103 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.105 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.106 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.107 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.108 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.109 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: "threshold": { 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "threshold", 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3.0", 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1.0", 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "scaling threshold", 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "progress", 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_pg_recovery_event": { 2024-09-16T00:43:05.111 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_pg_recovery_event", 2024-09-16T00:43:05.112 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.112 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.112 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.112 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.112 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.112 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.112 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.112 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-16T00:43:05.112 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.112 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.112 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": { 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enabled", 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.113 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.114 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.115 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.116 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.117 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "max_completed_events": { 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_completed_events", 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "50", 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.120 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "number of past completed events to remember", 2024-09-16T00:43:05.120 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.120 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.120 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.120 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.120 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:05.120 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:05.120 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.120 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.120 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.120 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:05.120 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long the module is going to sleep", 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prometheus", 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "cache": { 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cache", 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "exclude_perf_counters": { 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "exclude_perf_counters", 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-16T00:43:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-16T00:43:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_stats_pools": { 2024-09-16T00:43:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_stats_pools", 2024-09-16T00:43:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-16T00:43:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "300", 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "scrape_interval": { 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "scrape_interval", 2024-09-16T00:43:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15.0", 2024-09-16T00:43:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "::", 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "9283", 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "stale_cache_strategy": { 2024-09-16T00:43:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "stale_cache_strategy", 2024-09-16T00:43:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "log", 2024-09-16T00:43:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_behaviour": { 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_behaviour", 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "default", 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "default", 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "error" 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_error_status_code": { 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_error_status_code", 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "500", 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "400", 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "599", 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_support", 2024-09-16T00:43:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "max_concurrent_snap_create": { 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_concurrent_snap_create", 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "mirror_snapshot_schedule": { 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mirror_snapshot_schedule", 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "trash_purge_schedule": { 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "trash_purge_schedule", 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "restful", 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_auth": { 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enable_auth", 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "key_file": { 2024-09-16T00:43:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "key_file", 2024-09-16T00:43:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rgw", 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "secondary_zone_period_retry_limit": { 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rook", 2024-09-16T00:43:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.170 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.171 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.171 INFO:teuthology.orchestra.run.smithi033.stdout: "storage_class": { 2024-09-16T00:43:05.171 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "storage_class", 2024-09-16T00:43:05.171 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.171 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.171 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.171 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "local", 2024-09-16T00:43:05.171 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.171 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.171 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.171 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "selftest", 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.172 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.173 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.174 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.175 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.176 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.177 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: "roption1": { 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "roption1", 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.178 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: "roption2": { 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "roption2", 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "xyz", 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption1": { 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption1", 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.180 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption2": { 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption2", 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.181 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption3": { 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption3", 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.182 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption4": { 2024-09-16T00:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption4", 2024-09-16T00:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption5": { 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption5", 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.184 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption6": { 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption6", 2024-09-16T00:43:05.185 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption7": { 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption7", 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "42", 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.187 INFO:teuthology.orchestra.run.smithi033.stdout: "testkey": { 2024-09-16T00:43:05.188 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testkey", 2024-09-16T00:43:05.188 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.188 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.188 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.188 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.188 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.188 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.188 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.188 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.188 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.188 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.188 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "testlkey": { 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testlkey", 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.189 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: "testnewline": { 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testnewline", 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.190 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snap_schedule", 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_m_granularity": { 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_m_granularity", 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.191 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow minute scheduled snapshots", 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: "dump_on_update": { 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "dump_on_update", 2024-09-16T00:43:05.192 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "dump database to debug log on update", 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.193 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.194 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.195 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.195 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.195 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.195 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.195 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.195 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.195 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.195 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.195 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.195 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.195 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.195 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.196 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.197 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.198 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "stats", 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.199 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.200 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.200 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.200 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.200 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.200 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.200 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.200 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.200 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.200 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.200 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.200 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.200 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.201 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.202 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.202 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.202 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.202 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.202 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.202 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.202 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.202 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.202 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.202 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.202 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.202 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.203 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.204 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.204 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.204 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.204 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.204 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.204 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.204 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.204 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.204 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.204 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.204 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.204 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "status", 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.206 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.207 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.208 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.209 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.210 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.211 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.211 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.211 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.211 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "telegraf", 2024-09-16T00:43:05.211 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.211 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.211 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.211 INFO:teuthology.orchestra.run.smithi033.stdout: "address": { 2024-09-16T00:43:05.211 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "address", 2024-09-16T00:43:05.211 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:05.212 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15", 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.213 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.215 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.216 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.216 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.216 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.216 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.216 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.216 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.216 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.216 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.216 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.216 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.216 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.217 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.218 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "telemetry", 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_basic": { 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_basic", 2024-09-16T00:43:05.219 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_crash": { 2024-09-16T00:43:05.220 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_crash", 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.221 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_device": { 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_device", 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_ident": { 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_ident", 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_perf": { 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_perf", 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.224 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.225 INFO:teuthology.orchestra.run.smithi033.stdout: "contact": { 2024-09-16T00:43:05.225 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "contact", 2024-09-16T00:43:05.225 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.225 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.225 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.225 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.225 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.225 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.225 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.225 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.225 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "description": { 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "description", 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.226 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: "device_url": { 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_url", 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.227 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.228 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.228 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.228 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.228 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": { 2024-09-16T00:43:05.228 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enabled", 2024-09-16T00:43:05.228 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.228 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.228 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.228 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.228 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.228 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.229 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.229 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.229 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.229 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.229 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.229 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:05.229 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:05.229 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.229 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.229 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.229 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "24", 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "8", 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: "last_opt_revision": { 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "last_opt_revision", 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.230 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1", 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: "leaderboard": { 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "leaderboard", 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.231 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: "leaderboard_description": { 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "leaderboard_description", 2024-09-16T00:43:05.232 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.233 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.234 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "organization": { 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "organization", 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "proxy": { 2024-09-16T00:43:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "proxy", 2024-09-16T00:43:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "url": { 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "url", 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "test_orchestrator", 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.247 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "volumes", 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.248 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.249 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.250 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.251 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.252 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.252 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.252 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.252 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.252 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.252 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.252 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.252 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.252 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.252 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.252 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.252 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.253 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: "max_concurrent_clones": { 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_concurrent_clones", 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "4", 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.254 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.255 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.255 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-16T00:43:05.255 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.255 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.255 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.255 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.255 INFO:teuthology.orchestra.run.smithi033.stdout: "periodic_async_work": { 2024-09-16T00:43:05.255 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "periodic_async_work", 2024-09-16T00:43:05.255 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.255 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.255 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.255 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Periodically check for async work", 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: "snapshot_clone_delay": { 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snapshot_clone_delay", 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.256 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: "snapshot_clone_no_wait": { 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snapshot_clone_no_wait", 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.257 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.258 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.258 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.258 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.258 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.258 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.258 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-16T00:43:05.258 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.258 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.258 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.258 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.258 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix", 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: "discovery_interval": { 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "discovery_interval", 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "100", 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.259 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: "identifier": { 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "identifier", 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.260 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.261 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.262 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.264 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.265 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.265 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.265 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.265 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.265 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.265 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.265 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.265 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.265 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.265 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.265 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.265 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.267 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_host": { 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_host", 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.268 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_port": { 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_port", 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10051", 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.269 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.270 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.270 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.270 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_sender": { 2024-09-16T00:43:05.270 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_sender", 2024-09-16T00:43:05.270 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.270 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.270 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.270 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-16T00:43:05.270 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.270 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.270 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: "gid": 24113, 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "y", 2024-09-16T00:43:05.271 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_features": 4540701547738038271, 2024-09-16T00:43:05.272 INFO:teuthology.orchestra.run.smithi033.stdout: "available_modules": [ 2024-09-16T00:43:05.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.272 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "alerts", 2024-09-16T00:43:05.272 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.272 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.272 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.272 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:05.272 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:05.272 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.272 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.272 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.272 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How frequently to reexamine health status", 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.273 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_destination": { 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_destination", 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Email address to send alerts to", 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_from_name": { 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_from_name", 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "Ceph", 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Email From: name", 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_host": { 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_host", 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP server", 2024-09-16T00:43:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_password": { 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_password", 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Password to authenticate with", 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_port": { 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_port", 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "465", 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP port", 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_sender": { 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_sender", 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP envelope sender", 2024-09-16T00:43:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_ssl": { 2024-09-16T00:43:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_ssl", 2024-09-16T00:43:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-16T00:43:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_user": { 2024-09-16T00:43:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_user", 2024-09-16T00:43:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "User to authenticate as", 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "balancer", 2024-09-16T00:43:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "active": { 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "active", 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "automatically balance PGs across cluster", 2024-09-16T00:43:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "begin_time": { 2024-09-16T00:43:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "begin_time", 2024-09-16T00:43:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0000", 2024-09-16T00:43:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "beginning time of day to automatically balance", 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "begin_weekday": { 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "begin_weekday", 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0", 2024-09-16T00:43:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "6", 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_max_iterations": { 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_max_iterations", 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "25", 2024-09-16T00:43:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-09-16T00:43:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "250", 2024-09-16T00:43:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-16T00:43:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_metrics": { 2024-09-16T00:43:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_metrics", 2024-09-16T00:43:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "pgs,objects,bytes", 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_step": { 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_step", 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0.5", 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0.001", 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "0.999", 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "aggressiveness of optimization", 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "end_time": { 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "end_time", 2024-09-16T00:43:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "2359", 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "ending time of day to automatically balance", 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "end_weekday": { 2024-09-16T00:43:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "end_weekday", 2024-09-16T00:43:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0", 2024-09-16T00:43:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "6", 2024-09-16T00:43:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-16T00:43:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T00:43:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "min_score": { 2024-09-16T00:43:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "min_score", 2024-09-16T00:43:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": { 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mode", 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "upmap", 2024-09-16T00:43:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-compat", 2024-09-16T00:43:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "none", 2024-09-16T00:43:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "read", 2024-09-16T00:43:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap", 2024-09-16T00:43:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap-read" 2024-09-16T00:43:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Balancer mode", 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_ids": { 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pool_ids", 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap_max_deviation": { 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "upmap_max_deviation", 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-09-16T00:43:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-16T00:43:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-16T00:43:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap_max_optimizations": { 2024-09-16T00:43:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "upmap_max_optimizations", 2024-09-16T00:43:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cephadm", 2024-09-16T00:43:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_down_multiplier": { 2024-09-16T00:43:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_down_multiplier", 2024-09-16T00:43:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3.0", 2024-09-16T00:43:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_refresh_rate": { 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_refresh_rate", 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "20", 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-16T00:43:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_starting_port": { 2024-09-16T00:43:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_starting_port", 2024-09-16T00:43:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "4721", 2024-09-16T00:43:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-16T00:43:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_ptrace": { 2024-09-16T00:43:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_ptrace", 2024-09-16T00:43:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "autotune_interval": { 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "autotune_interval", 2024-09-16T00:43:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "autotune_memory_target_ratio": { 2024-09-16T00:43:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "autotune_memory_target_ratio", 2024-09-16T00:43:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0.7", 2024-09-16T00:43:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-16T00:43:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm_log_destination": { 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cephadm_log_destination", 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "file", 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "file,syslog", 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "syslog" 2024-09-16T00:43:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-16T00:43:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "cgroups_split": { 2024-09-16T00:43:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cgroups_split", 2024-09-16T00:43:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "config_checks_enabled": { 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_checks_enabled", 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-16T00:43:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "config_dashboard": { 2024-09-16T00:43:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_dashboard", 2024-09-16T00:43:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_alertmanager": { 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_alertmanager", 2024-09-16T00:43:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-16T00:43:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-09-16T00:43:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_base": { 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_base", 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Container image name, without the tag", 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_elasticsearch": { 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_elasticsearch", 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "elasticsearch container image", 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_grafana": { 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_grafana", 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_haproxy": { 2024-09-16T00:43:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_haproxy", 2024-09-16T00:43:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-16T00:43:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "HAproxy container image", 2024-09-16T00:43:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_agent": { 2024-09-16T00:43:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_agent", 2024-09-16T00:43:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-16T00:43:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger agent container image", 2024-09-16T00:43:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_collector": { 2024-09-16T00:43:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_collector", 2024-09-16T00:43:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger collector container image", 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_query": { 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_query", 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-16T00:43:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger query container image", 2024-09-16T00:43:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_keepalived": { 2024-09-16T00:43:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_keepalived", 2024-09-16T00:43:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Keepalived container image", 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_loki": { 2024-09-16T00:43:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_loki", 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Loki container image", 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_node_exporter": { 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_node_exporter", 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_nvmeof": { 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_nvmeof", 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Nvme-of container image", 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_prometheus": { 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_prometheus", 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_promtail": { 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_promtail", 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Promtail container image", 2024-09-16T00:43:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_samba": { 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_samba", 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Samba/SMB container image", 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_snmp_gateway": { 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_snmp_gateway", 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SNMP Gateway container image", 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "container_init": { 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_init", 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Run podman/docker with `--init`", 2024-09-16T00:43:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "daemon_cache_timeout": { 2024-09-16T00:43:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "daemon_cache_timeout", 2024-09-16T00:43:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-16T00:43:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "default_cephadm_command_timeout": { 2024-09-16T00:43:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "default_cephadm_command_timeout", 2024-09-16T00:43:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "900", 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "default_registry": { 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "default_registry", 2024-09-16T00:43:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io", 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "device_cache_timeout": { 2024-09-16T00:43:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_cache_timeout", 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1800", 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache device inventory", 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "device_enhanced_scan": { 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_enhanced_scan", 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "facts_cache_timeout": { 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "facts_cache_timeout", 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache host facts data", 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "grafana_dashboards_path": { 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "grafana_dashboards_path", 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "host_check_interval": { 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "host_check_interval", 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to perform a host check", 2024-09-16T00:43:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "hw_monitoring": { 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "hw_monitoring", 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "inventory_list_all": { 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "inventory_list_all", 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-16T00:43:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "log_refresh_metadata": { 2024-09-16T00:43:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_refresh_metadata", 2024-09-16T00:43:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-16T00:43:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-16T00:43:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "*", 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-16T00:43:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "max_count_per_host": { 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_count_per_host", 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "max number of daemons per service per host", 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd_draining_count": { 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_osd_draining_count", 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "migration_current": { 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "migration_current", 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "internal - do not modify", 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": { 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mode", 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "root", 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm-package", 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "root" 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "mode for remote execution of cephadm", 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "oob_default_addr": { 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "oob_default_addr", 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "169.254.1.1", 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "prometheus_alerts_path": { 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prometheus_alerts_path", 2024-09-16T00:43:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-16T00:43:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-16T00:43:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_insecure": { 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_insecure", 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_password": { 2024-09-16T00:43:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_password", 2024-09-16T00:43:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-16T00:43:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_url": { 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_url", 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-16T00:43:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_username": { 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_username", 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "secure_monitoring_stack": { 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "secure_monitoring_stack", 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "service_discovery_port": { 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "service_discovery_port", 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8765", 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "cephadm service discovery port", 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "ssh_config_file": { 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssh_config_file", 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "use_agent": { 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "use_agent", 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "use_repo_digest": { 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "use_repo_digest", 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-16T00:43:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_failed_host_check": { 2024-09-16T00:43:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_failed_host_check", 2024-09-16T00:43:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if the host check fails", 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_stray_daemons": { 2024-09-16T00:43:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_stray_daemons", 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_stray_hosts": { 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_stray_hosts", 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crash", 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.382 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.383 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.384 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.385 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "retain_interval": { 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "retain_interval", 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "31536000", 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.386 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_recent_interval": { 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_recent_interval", 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1209600", 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.387 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "dashboard", 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: "ALERTMANAGER_API_HOST": { 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-16T00:43:05.389 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-16T00:43:05.390 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "AUDIT_API_ENABLED": { 2024-09-16T00:43:05.391 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "AUDIT_API_ENABLED", 2024-09-16T00:43:05.392 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.392 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.392 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.392 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.392 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.392 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.392 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.392 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.392 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.392 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.392 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.393 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: "ENABLE_BROWSABLE_API": { 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.394 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.395 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.396 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.397 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.398 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.398 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.398 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.398 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.398 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.398 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-16T00:43:05.398 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-16T00:43:05.398 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.398 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.398 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.398 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.398 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.399 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.400 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.400 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.400 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.400 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.400 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.400 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.400 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.400 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.400 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-16T00:43:05.400 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-16T00:43:05.400 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.400 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-16T00:43:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-16T00:43:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_PASSWORD": { 2024-09-16T00:43:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_URL": { 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_URL", 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_USERNAME": { 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-16T00:43:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-16T00:43:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-16T00:43:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-16T00:43:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "PROMETHEUS_API_HOST": { 2024-09-16T00:43:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-16T00:43:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-16T00:43:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-16T00:43:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-16T00:43:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-16T00:43:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-16T00:43:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-16T00:43:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_ENABLED": { 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8", 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "45", 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_ACCESS_KEY": { 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-16T00:43:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-16T00:43:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_SECRET_KEY": { 2024-09-16T00:43:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-16T00:43:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_SSL_VERIFY": { 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "cross_origin_url": { 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cross_origin_url", 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "crt_file": { 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crt_file", 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "debug": { 2024-09-16T00:43:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "debug", 2024-09-16T00:43:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable/disable debug options", 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "jwt_token_ttl": { 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "jwt_token_ttl", 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "28800", 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "key_file": { 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "key_file", 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "motd": { 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "motd", 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "The message of the day", 2024-09-16T00:43:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "redirect_resolve_ip_addr": { 2024-09-16T00:43:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "redirect_resolve_ip_addr", 2024-09-16T00:43:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "::", 2024-09-16T00:43:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8080", 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl": { 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl", 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl_server_port": { 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl_server_port", 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8443", 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_behaviour": { 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_behaviour", 2024-09-16T00:43:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "redirect", 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "redirect" 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_error_status_code": { 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_error_status_code", 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "500", 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "400", 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "599", 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "url_prefix": { 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "url_prefix", 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "devicehealth", 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_monitoring": { 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enable_monitoring", 2024-09-16T00:43:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "monitor device health metrics", 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "mark_out_threshold": { 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mark_out_threshold", 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "2419200", 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": { 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pool_name", 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "device_health_metrics", 2024-09-16T00:43:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-16T00:43:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "retention_period": { 2024-09-16T00:43:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "retention_period", 2024-09-16T00:43:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15552000", 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long to retain device health metrics", 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "scrape_frequency": { 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "scrape_frequency", 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "86400", 2024-09-16T00:43:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-16T00:43:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "self_heal": { 2024-09-16T00:43:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "self_heal", 2024-09-16T00:43:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-16T00:43:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to wake up and check device health", 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_threshold": { 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_threshold", 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "7257600", 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "diskprediction_local", 2024-09-16T00:43:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "predict_interval": { 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "predict_interval", 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "86400", 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "predictor_model": { 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "predictor_model", 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "prophetstor", 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "influx", 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": false, 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "influxdb python module not found", 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "batch_size": { 2024-09-16T00:43:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "batch_size", 2024-09-16T00:43:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5000", 2024-09-16T00:43:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-16T00:43:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "database": { 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "database", 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "ceph", 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "hostname": { 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "hostname", 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB server hostname", 2024-09-16T00:43:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "30", 2024-09-16T00:43:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "5", 2024-09-16T00:43:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "password": { 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "password", 2024-09-16T00:43:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "password of InfluxDB server user", 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "port": { 2024-09-16T00:43:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "port", 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8086", 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB server port", 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl": { 2024-09-16T00:43:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl", 2024-09-16T00:43:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "false", 2024-09-16T00:43:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T00:43:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "threads": { 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "threads", 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "32", 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "username": { 2024-09-16T00:43:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "username", 2024-09-16T00:43:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "username of InfluxDB server user", 2024-09-16T00:43:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "verify_ssl": { 2024-09-16T00:43:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "verify_ssl", 2024-09-16T00:43:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "true", 2024-09-16T00:43:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "insights", 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.502 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "iostat", 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.503 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.505 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.506 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.508 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.508 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.508 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.508 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.508 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.508 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.508 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.508 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.508 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.508 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.508 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "k8sevents", 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_event_retention_days": { 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ceph_event_retention_days", 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "7", 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: "config_check_secs": { 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_check_secs", 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "10", 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-16T00:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.513 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.514 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.515 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.515 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.515 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.515 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.515 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.515 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.515 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.515 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.515 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.515 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.515 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.515 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "localpool", 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "failure_domain": { 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "failure_domain", 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "host", 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "failure domain for any created local pool", 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": { 2024-09-16T00:43:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "min_size", 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default min_size for any created local pool", 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "num_rep": { 2024-09-16T00:43:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "num_rep", 2024-09-16T00:43:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3", 2024-09-16T00:43:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default replica count for any created local pool", 2024-09-16T00:43:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": { 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pg_num", 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "128", 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default pg_num for any created local pool", 2024-09-16T00:43:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "prefix": { 2024-09-16T00:43:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prefix", 2024-09-16T00:43:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "name prefix for any created local pool", 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "subtree": { 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "subtree", 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "rack", 2024-09-16T00:43:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-16T00:43:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-16T00:43:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mds_autoscaler", 2024-09-16T00:43:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mirroring", 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "nfs", 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "orchestrator", 2024-09-16T00:43:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "fail_fs": { 2024-09-16T00:43:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "fail_fs", 2024-09-16T00:43:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-16T00:43:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator": { 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "orchestrator", 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm", 2024-09-16T00:43:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "rook", 2024-09-16T00:43:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "test_orchestrator" 2024-09-16T00:43:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Orchestrator backend", 2024-09-16T00:43:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "osd_perf_query", 2024-09-16T00:43:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "osd_support", 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pg_autoscaler", 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.572 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.572 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.572 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.572 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.572 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.572 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.572 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.572 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.572 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.572 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.572 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.572 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.573 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.574 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.574 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.574 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.574 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.574 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.574 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.574 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.574 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.574 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.574 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.574 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.575 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.575 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.575 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.575 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.575 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.575 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.575 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.575 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.575 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.575 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.575 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.576 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:05.576 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:05.576 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.576 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.576 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.576 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:05.576 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.576 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.576 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.576 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.576 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.576 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: "threshold": { 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "threshold", 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3.0", 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1.0", 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "scaling threshold", 2024-09-16T00:43:05.577 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "progress", 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_pg_recovery_event": { 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_pg_recovery_event", 2024-09-16T00:43:05.578 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": { 2024-09-16T00:43:05.579 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enabled", 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.580 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.581 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.581 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.581 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.581 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.581 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.581 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.581 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.581 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.581 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.581 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.581 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.581 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.582 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.582 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.582 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.582 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.582 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.582 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.582 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.582 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.582 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.582 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.582 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.583 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.584 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.584 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.584 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.584 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.584 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.584 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.584 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.584 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.584 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.584 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.584 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.584 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.585 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "max_completed_events": { 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_completed_events", 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "50", 2024-09-16T00:43:05.586 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.587 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.587 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.587 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "number of past completed events to remember", 2024-09-16T00:43:05.587 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.587 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.587 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.587 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:05.587 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:05.587 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.587 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.588 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.588 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:05.588 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.588 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.588 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.588 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long the module is going to sleep", 2024-09-16T00:43:05.588 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.588 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.588 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.588 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.588 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.589 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prometheus", 2024-09-16T00:43:05.589 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.589 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.589 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.589 INFO:teuthology.orchestra.run.smithi033.stdout: "cache": { 2024-09-16T00:43:05.589 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cache", 2024-09-16T00:43:05.589 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.589 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.589 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.589 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.589 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: "exclude_perf_counters": { 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "exclude_perf_counters", 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.590 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.591 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.591 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.591 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.591 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-16T00:43:05.591 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-16T00:43:05.591 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.591 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.591 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.591 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.591 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.591 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.592 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.592 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.592 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.592 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.592 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.592 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.592 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.592 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.592 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.592 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.592 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.592 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.593 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.593 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.593 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.593 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.593 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.593 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.593 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.593 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.593 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.593 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.593 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.594 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.594 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.594 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.594 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.594 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.594 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.594 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.594 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.594 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.594 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.594 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.595 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.596 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.596 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.596 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.596 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.596 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.596 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.596 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.596 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.596 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.596 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.596 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_stats_pools": { 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_stats_pools", 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.597 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "300", 2024-09-16T00:43:05.598 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "scrape_interval": { 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "scrape_interval", 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15.0", 2024-09-16T00:43:05.599 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.600 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.600 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.600 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.600 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.600 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.600 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.600 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-09-16T00:43:05.600 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-09-16T00:43:05.600 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.600 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.600 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "::", 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.601 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.602 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.602 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "9283", 2024-09-16T00:43:05.602 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.602 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.602 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.602 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-16T00:43:05.602 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.602 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.602 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.602 INFO:teuthology.orchestra.run.smithi033.stdout: "stale_cache_strategy": { 2024-09-16T00:43:05.602 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "stale_cache_strategy", 2024-09-16T00:43:05.603 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.603 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.603 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.603 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "log", 2024-09-16T00:43:05.603 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.603 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.603 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.603 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.603 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.603 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.603 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.604 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_behaviour": { 2024-09-16T00:43:05.604 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_behaviour", 2024-09-16T00:43:05.604 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.604 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.604 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.604 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "default", 2024-09-16T00:43:05.604 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.604 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.604 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.604 INFO:teuthology.orchestra.run.smithi033.stdout: "default", 2024-09-16T00:43:05.604 INFO:teuthology.orchestra.run.smithi033.stdout: "error" 2024-09-16T00:43:05.604 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_error_status_code": { 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_error_status_code", 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "500", 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "400", 2024-09-16T00:43:05.605 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "599", 2024-09-16T00:43:05.606 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.606 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.606 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.606 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.606 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.606 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.606 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.606 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_support", 2024-09-16T00:43:05.606 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.606 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.607 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.608 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.608 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.608 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.608 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.608 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.608 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.608 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.608 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.608 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.608 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.608 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.609 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.610 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.611 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: "max_concurrent_snap_create": { 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_concurrent_snap_create", 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.612 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.613 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.613 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.613 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.613 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.613 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.613 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.613 INFO:teuthology.orchestra.run.smithi033.stdout: "mirror_snapshot_schedule": { 2024-09-16T00:43:05.613 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mirror_snapshot_schedule", 2024-09-16T00:43:05.613 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.613 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.613 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: "trash_purge_schedule": { 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "trash_purge_schedule", 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.614 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "restful", 2024-09-16T00:43:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_auth": { 2024-09-16T00:43:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enable_auth", 2024-09-16T00:43:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: "key_file": { 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "key_file", 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.617 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.618 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.618 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.618 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.618 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.618 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.618 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.618 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.618 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.618 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.618 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.619 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.619 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.619 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.619 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.619 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.619 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.619 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.619 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.619 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.619 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.619 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.619 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.620 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.621 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.622 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.623 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.624 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.624 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-09-16T00:43:05.624 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-09-16T00:43:05.624 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.624 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.624 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.624 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.624 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.624 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.624 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.624 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.625 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rgw", 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.626 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.627 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.628 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.628 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.628 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.628 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.628 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.628 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.628 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.628 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.628 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.628 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.628 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.629 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.630 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.630 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.630 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.630 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.630 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.630 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.630 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.630 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.630 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.630 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.630 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.630 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.631 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.632 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.632 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.632 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.632 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.632 INFO:teuthology.orchestra.run.smithi033.stdout: "secondary_zone_period_retry_limit": { 2024-09-16T00:43:05.632 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-16T00:43:05.632 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.632 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.632 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.632 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:05.632 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.633 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.633 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.633 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-16T00:43:05.633 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.633 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.633 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.633 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.633 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.633 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rook", 2024-09-16T00:43:05.633 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.634 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.634 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.634 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.634 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.634 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.634 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.634 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.634 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.634 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.634 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.634 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.634 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.636 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.636 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.636 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.636 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.636 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.636 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.636 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.636 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.636 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.636 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.636 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.637 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.638 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.638 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.638 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.638 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.638 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.638 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.638 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.638 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.638 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.638 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.638 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.639 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.639 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.639 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.639 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.639 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.639 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.639 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.639 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.639 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.639 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.639 INFO:teuthology.orchestra.run.smithi033.stdout: "storage_class": { 2024-09-16T00:43:05.640 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "storage_class", 2024-09-16T00:43:05.640 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.640 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.640 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.640 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "local", 2024-09-16T00:43:05.640 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.640 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.640 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.640 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-16T00:43:05.640 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.640 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.641 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.641 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.641 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.641 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "selftest", 2024-09-16T00:43:05.641 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.641 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.641 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.641 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.641 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.641 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.642 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.643 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.643 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.643 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.643 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.643 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.643 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.643 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.643 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.643 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.643 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.643 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.644 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.644 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.644 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.644 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.644 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.644 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.644 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.644 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.644 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.644 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.644 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.645 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.645 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.645 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.645 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.645 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.645 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.645 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.645 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.645 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.645 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.645 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.645 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.646 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.646 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.646 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.646 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.646 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.646 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.646 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.646 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.646 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.646 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.646 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.647 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.647 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.647 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.647 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.647 INFO:teuthology.orchestra.run.smithi033.stdout: "roption1": { 2024-09-16T00:43:05.647 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "roption1", 2024-09-16T00:43:05.647 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.647 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.647 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.647 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.647 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: "roption2": { 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "roption2", 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.648 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "xyz", 2024-09-16T00:43:05.649 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.649 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.649 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.649 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.649 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.649 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.649 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.649 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption1": { 2024-09-16T00:43:05.649 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption1", 2024-09-16T00:43:05.649 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.650 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.650 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.650 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.650 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.650 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.650 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.650 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.650 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.650 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.650 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.650 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption2": { 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption2", 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.652 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption3": { 2024-09-16T00:43:05.652 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption3", 2024-09-16T00:43:05.652 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.652 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.652 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.652 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.652 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.652 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.652 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.652 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.652 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.652 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.653 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.653 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption4": { 2024-09-16T00:43:05.653 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption4", 2024-09-16T00:43:05.653 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.653 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.653 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.653 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.653 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.653 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.653 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.653 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption5": { 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption5", 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.654 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.655 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.655 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.655 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.655 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.655 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption6": { 2024-09-16T00:43:05.655 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption6", 2024-09-16T00:43:05.655 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.655 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.655 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.655 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.655 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.656 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.656 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.656 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.656 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.656 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.656 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.656 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption7": { 2024-09-16T00:43:05.656 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption7", 2024-09-16T00:43:05.656 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.656 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.656 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "42", 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: "testkey": { 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testkey", 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.657 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: "testlkey": { 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testlkey", 2024-09-16T00:43:05.658 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: "testnewline": { 2024-09-16T00:43:05.659 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testnewline", 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.660 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.661 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snap_schedule", 2024-09-16T00:43:05.661 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.661 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.661 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.661 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_m_granularity": { 2024-09-16T00:43:05.661 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_m_granularity", 2024-09-16T00:43:05.661 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.661 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.661 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.661 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.662 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.662 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.662 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.662 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow minute scheduled snapshots", 2024-09-16T00:43:05.662 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.662 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.662 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.662 INFO:teuthology.orchestra.run.smithi033.stdout: "dump_on_update": { 2024-09-16T00:43:05.662 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "dump_on_update", 2024-09-16T00:43:05.662 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.662 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "dump database to debug log on update", 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.663 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.664 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.664 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.664 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.664 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.664 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.664 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.664 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.664 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.664 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.664 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.664 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.664 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.665 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.665 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.665 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.665 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.665 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.665 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.665 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.665 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.665 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.665 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.665 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.666 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.667 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.668 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.668 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.668 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.668 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.668 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.668 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.668 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.668 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.668 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.668 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.668 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "stats", 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.669 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.671 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.671 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.671 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.671 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.671 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.671 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.671 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.671 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.671 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.671 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.671 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.672 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.673 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.674 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "status", 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.675 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.676 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.676 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.676 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.676 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.676 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.676 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.676 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.676 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.676 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.676 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.676 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.676 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.677 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.678 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.679 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.679 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.679 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.679 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.679 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.679 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.679 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.679 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.679 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.679 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.679 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.679 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.680 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.680 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.680 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.680 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.680 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.680 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.680 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.680 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.680 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.680 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.680 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "telegraf", 2024-09-16T00:43:05.681 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "address": { 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "address", 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.682 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15", 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.683 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.684 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.686 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.687 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.688 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.689 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "telemetry", 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_basic": { 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_basic", 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.690 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_crash": { 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_crash", 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.691 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_device": { 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_device", 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.692 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_ident": { 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_ident", 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.693 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-16T00:43:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_perf": { 2024-09-16T00:43:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_perf", 2024-09-16T00:43:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.695 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.695 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.695 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.695 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.695 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-16T00:43:05.695 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.695 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.695 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.695 INFO:teuthology.orchestra.run.smithi033.stdout: "contact": { 2024-09-16T00:43:05.695 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "contact", 2024-09-16T00:43:05.695 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.696 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.696 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.696 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.696 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.696 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.696 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.696 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.696 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.696 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.696 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.696 INFO:teuthology.orchestra.run.smithi033.stdout: "description": { 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "description", 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.698 INFO:teuthology.orchestra.run.smithi033.stdout: "device_url": { 2024-09-16T00:43:05.698 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_url", 2024-09-16T00:43:05.698 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.698 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.698 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.698 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-16T00:43:05.698 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.698 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.698 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.698 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.698 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.698 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.699 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.699 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.699 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": { 2024-09-16T00:43:05.699 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enabled", 2024-09-16T00:43:05.699 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.699 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.699 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.699 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.699 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.699 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.699 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.699 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.700 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.700 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.700 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.700 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:05.700 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:05.700 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.700 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.700 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.700 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "24", 2024-09-16T00:43:05.700 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "8", 2024-09-16T00:43:05.700 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: "last_opt_revision": { 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "last_opt_revision", 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1", 2024-09-16T00:43:05.701 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.702 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.702 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.702 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.702 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.702 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.702 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.702 INFO:teuthology.orchestra.run.smithi033.stdout: "leaderboard": { 2024-09-16T00:43:05.702 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "leaderboard", 2024-09-16T00:43:05.702 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.702 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.702 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.703 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.703 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.703 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.703 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.703 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.703 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.703 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.703 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.703 INFO:teuthology.orchestra.run.smithi033.stdout: "leaderboard_description": { 2024-09-16T00:43:05.703 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "leaderboard_description", 2024-09-16T00:43:05.703 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.704 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.705 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.705 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.705 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.705 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.705 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.705 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.705 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.705 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.705 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.705 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.705 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.705 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.706 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.707 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.707 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.707 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.707 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.707 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.707 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.707 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.707 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.707 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.707 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.707 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.708 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.709 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: "organization": { 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "organization", 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.710 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: "proxy": { 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "proxy", 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.711 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: "url": { 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "url", 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-16T00:43:05.712 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.713 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.713 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.713 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.713 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.713 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.713 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.713 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.713 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.713 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "test_orchestrator", 2024-09-16T00:43:05.713 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.714 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.715 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.716 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.717 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.718 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.719 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.719 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.719 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.719 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.719 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.719 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.719 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.719 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.719 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "volumes", 2024-09-16T00:43:05.719 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.719 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.720 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.720 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.720 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.720 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.720 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.720 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.720 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.720 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.720 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.720 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.720 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.720 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.721 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.722 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.722 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.722 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.722 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.722 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.722 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.722 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.722 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.722 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.722 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.722 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.723 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.724 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.725 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.725 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.725 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.725 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.725 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.725 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.725 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.725 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.725 INFO:teuthology.orchestra.run.smithi033.stdout: "max_concurrent_clones": { 2024-09-16T00:43:05.725 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_concurrent_clones", 2024-09-16T00:43:05.725 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.725 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.726 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.726 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "4", 2024-09-16T00:43:05.726 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.726 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.726 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.726 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-16T00:43:05.726 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.726 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.726 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.726 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.726 INFO:teuthology.orchestra.run.smithi033.stdout: "periodic_async_work": { 2024-09-16T00:43:05.726 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "periodic_async_work", 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Periodically check for async work", 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.727 INFO:teuthology.orchestra.run.smithi033.stdout: "snapshot_clone_delay": { 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snapshot_clone_delay", 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.728 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.729 INFO:teuthology.orchestra.run.smithi033.stdout: "snapshot_clone_no_wait": { 2024-09-16T00:43:05.729 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snapshot_clone_no_wait", 2024-09-16T00:43:05.729 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.729 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.729 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.729 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.729 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.729 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.729 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.729 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-16T00:43:05.729 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.729 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix", 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: "discovery_interval": { 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "discovery_interval", 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.730 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "100", 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: "identifier": { 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "identifier", 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.731 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.734 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.735 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.736 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.736 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.736 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.736 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.736 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.736 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.736 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.736 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.736 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.736 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.736 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.736 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.737 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.738 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_host": { 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_host", 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.739 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.740 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.740 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.740 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.740 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.740 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_port": { 2024-09-16T00:43:05.740 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_port", 2024-09-16T00:43:05.740 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.740 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.740 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.740 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10051", 2024-09-16T00:43:05.740 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_sender": { 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_sender", 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.741 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-16T00:43:05.742 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.742 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.742 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.742 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.742 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.742 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.742 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.742 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.742 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.742 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.742 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-09-16T00:43:05.742 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: "modules": [ 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm", 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: "dashboard", 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: "iostat", 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: "nfs", 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: "restful" 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: "available_modules": [ 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "alerts", 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.743 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How frequently to reexamine health status", 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.744 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.745 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.746 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.747 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.747 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.747 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.747 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.747 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.747 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.747 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.747 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.747 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.747 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.747 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.747 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.748 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.749 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_destination": { 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_destination", 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.750 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Email address to send alerts to", 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_from_name": { 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_from_name", 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "Ceph", 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.751 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.752 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Email From: name", 2024-09-16T00:43:05.752 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.752 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.752 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.752 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.752 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_host": { 2024-09-16T00:43:05.752 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_host", 2024-09-16T00:43:05.752 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.752 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.752 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.752 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.752 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.753 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.753 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.753 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP server", 2024-09-16T00:43:05.753 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.753 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.753 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.753 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.753 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_password": { 2024-09-16T00:43:05.753 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_password", 2024-09-16T00:43:05.753 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.753 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.753 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Password to authenticate with", 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_port": { 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_port", 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.754 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "465", 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP port", 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_sender": { 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_sender", 2024-09-16T00:43:05.755 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP envelope sender", 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_ssl": { 2024-09-16T00:43:05.756 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_ssl", 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.757 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_user": { 2024-09-16T00:43:05.758 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_user", 2024-09-16T00:43:05.758 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.758 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.758 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.758 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.758 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.758 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.758 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.758 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "User to authenticate as", 2024-09-16T00:43:05.758 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.758 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.758 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.759 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.759 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.759 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.759 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.759 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "balancer", 2024-09-16T00:43:05.759 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.759 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.759 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.759 INFO:teuthology.orchestra.run.smithi033.stdout: "active": { 2024-09-16T00:43:05.759 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "active", 2024-09-16T00:43:05.759 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.759 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "automatically balance PGs across cluster", 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: "begin_time": { 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "begin_time", 2024-09-16T00:43:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.761 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.761 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.761 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0000", 2024-09-16T00:43:05.761 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.761 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.761 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.761 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "beginning time of day to automatically balance", 2024-09-16T00:43:05.761 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T00:43:05.761 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.761 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.761 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.761 INFO:teuthology.orchestra.run.smithi033.stdout: "begin_weekday": { 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "begin_weekday", 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0", 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "6", 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.762 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_max_iterations": { 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_max_iterations", 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "25", 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "250", 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.763 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_metrics": { 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_metrics", 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "pgs,objects,bytes", 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.764 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_step": { 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_step", 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0.5", 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0.001", 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "0.999", 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "aggressiveness of optimization", 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-16T00:43:05.765 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.766 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.766 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.766 INFO:teuthology.orchestra.run.smithi033.stdout: "end_time": { 2024-09-16T00:43:05.766 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "end_time", 2024-09-16T00:43:05.766 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.766 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.766 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.766 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "2359", 2024-09-16T00:43:05.766 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.766 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.766 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.766 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "ending time of day to automatically balance", 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: "end_weekday": { 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "end_weekday", 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0", 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "6", 2024-09-16T00:43:05.767 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.768 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.769 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.770 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.770 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.770 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.770 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.770 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.770 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.770 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.770 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.770 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.770 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.770 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.770 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.771 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.771 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.771 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.771 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.771 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.771 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.771 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.771 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.771 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.771 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.771 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.771 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.772 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.773 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.773 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.773 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.773 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.773 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.773 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.773 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.773 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.773 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.773 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.773 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.773 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "min_score": { 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "min_score", 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.774 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": { 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mode", 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "upmap", 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-compat", 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "none", 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "read", 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap", 2024-09-16T00:43:05.775 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap-read" 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Balancer mode", 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_ids": { 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pool_ids", 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.776 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.777 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.777 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.777 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-16T00:43:05.777 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.777 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.777 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.777 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.777 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:05.777 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:05.777 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.777 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.777 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap_max_deviation": { 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "upmap_max_deviation", 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.778 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.779 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.779 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:05.779 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-09-16T00:43:05.779 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.779 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.779 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-16T00:43:05.779 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-16T00:43:05.779 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.779 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.779 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.779 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap_max_optimizations": { 2024-09-16T00:43:05.779 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "upmap_max_optimizations", 2024-09-16T00:43:05.780 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:05.780 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.780 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.780 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.780 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.780 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.780 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.780 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-16T00:43:05.780 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.780 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.780 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.780 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cephadm", 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_down_multiplier": { 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_down_multiplier", 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.781 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3.0", 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_refresh_rate": { 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_refresh_rate", 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.782 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "20", 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_starting_port": { 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_starting_port", 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.783 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.784 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "4721", 2024-09-16T00:43:05.784 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.784 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.784 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.784 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-16T00:43:05.784 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.784 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.784 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.784 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_ptrace": { 2024-09-16T00:43:05.784 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_ptrace", 2024-09-16T00:43:05.784 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: "autotune_interval": { 2024-09-16T00:43:05.785 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "autotune_interval", 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "autotune_memory_target_ratio": { 2024-09-16T00:43:05.786 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "autotune_memory_target_ratio", 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0.7", 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.787 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm_log_destination": { 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cephadm_log_destination", 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: "file", 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: "file,syslog", 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: "syslog" 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.788 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-16T00:43:05.789 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.789 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.789 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.789 INFO:teuthology.orchestra.run.smithi033.stdout: "cgroups_split": { 2024-09-16T00:43:05.789 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cgroups_split", 2024-09-16T00:43:05.789 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.789 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.789 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.789 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.789 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.789 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: "config_checks_enabled": { 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_checks_enabled", 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.790 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: "config_dashboard": { 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_dashboard", 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_alertmanager": { 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_alertmanager", 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.792 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_base": { 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_base", 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.793 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.794 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.794 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-16T00:43:05.794 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.794 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.794 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.794 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Container image name, without the tag", 2024-09-16T00:43:05.794 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.794 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.794 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.794 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_elasticsearch": { 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_elasticsearch", 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "elasticsearch container image", 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_grafana": { 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_grafana", 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.796 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_haproxy": { 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_haproxy", 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "HAproxy container image", 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.797 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_agent": { 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_agent", 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger agent container image", 2024-09-16T00:43:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.799 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.799 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.799 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_collector": { 2024-09-16T00:43:05.799 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_collector", 2024-09-16T00:43:05.799 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.799 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.799 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.799 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-16T00:43:05.799 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.799 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.799 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger collector container image", 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_query": { 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_query", 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger query container image", 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_keepalived": { 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_keepalived", 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-16T00:43:05.801 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Keepalived container image", 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_loki": { 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_loki", 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.802 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Loki container image", 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_node_exporter": { 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_node_exporter", 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.803 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.804 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-16T00:43:05.804 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.804 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.804 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.804 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-09-16T00:43:05.804 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.804 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.804 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.804 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_nvmeof": { 2024-09-16T00:43:05.804 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_nvmeof", 2024-09-16T00:43:05.804 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Nvme-of container image", 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_prometheus": { 2024-09-16T00:43:05.805 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_prometheus", 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_promtail": { 2024-09-16T00:43:05.806 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_promtail", 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Promtail container image", 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.807 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_samba": { 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_samba", 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Samba/SMB container image", 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.809 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_snmp_gateway": { 2024-09-16T00:43:05.809 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_snmp_gateway", 2024-09-16T00:43:05.809 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.809 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.809 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.809 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-16T00:43:05.809 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.809 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.809 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.809 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SNMP Gateway container image", 2024-09-16T00:43:05.809 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "container_init": { 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_init", 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Run podman/docker with `--init`", 2024-09-16T00:43:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.811 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.811 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.811 INFO:teuthology.orchestra.run.smithi033.stdout: "daemon_cache_timeout": { 2024-09-16T00:43:05.811 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "daemon_cache_timeout", 2024-09-16T00:43:05.811 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.811 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.811 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.811 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:05.811 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.811 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.811 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.812 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-16T00:43:05.812 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.812 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.812 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.812 INFO:teuthology.orchestra.run.smithi033.stdout: "default_cephadm_command_timeout": { 2024-09-16T00:43:05.812 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "default_cephadm_command_timeout", 2024-09-16T00:43:05.812 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.812 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.812 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.812 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "900", 2024-09-16T00:43:05.812 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: "default_registry": { 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "default_registry", 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.813 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io", 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: "device_cache_timeout": { 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_cache_timeout", 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.814 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1800", 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache device inventory", 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: "device_enhanced_scan": { 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_enhanced_scan", 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.815 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: "facts_cache_timeout": { 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "facts_cache_timeout", 2024-09-16T00:43:05.816 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.817 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.817 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.817 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:05.817 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.817 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.817 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.817 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache host facts data", 2024-09-16T00:43:05.817 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.817 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.817 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.817 INFO:teuthology.orchestra.run.smithi033.stdout: "grafana_dashboards_path": { 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "grafana_dashboards_path", 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "host_check_interval": { 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "host_check_interval", 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to perform a host check", 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.819 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.820 INFO:teuthology.orchestra.run.smithi033.stdout: "hw_monitoring": { 2024-09-16T00:43:05.820 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "hw_monitoring", 2024-09-16T00:43:05.820 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.820 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.820 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.820 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.820 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.820 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.820 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.820 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-16T00:43:05.820 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.821 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.821 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.821 INFO:teuthology.orchestra.run.smithi033.stdout: "inventory_list_all": { 2024-09-16T00:43:05.821 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "inventory_list_all", 2024-09-16T00:43:05.821 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.821 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.821 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.821 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.821 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.821 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.821 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.822 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "log_refresh_metadata": { 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_refresh_metadata", 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.824 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.825 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.826 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.826 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.826 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.826 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.826 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.826 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.826 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.826 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.826 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.826 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.826 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.827 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.827 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.827 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.827 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.827 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.827 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.827 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.827 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.827 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.827 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.827 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.828 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.828 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.828 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.828 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.828 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.828 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.828 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.828 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.828 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.828 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.828 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.828 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.829 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "*", 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-16T00:43:05.830 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.831 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.831 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.831 INFO:teuthology.orchestra.run.smithi033.stdout: "max_count_per_host": { 2024-09-16T00:43:05.831 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_count_per_host", 2024-09-16T00:43:05.831 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.831 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.831 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.831 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.831 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.831 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.831 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "max number of daemons per service per host", 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd_draining_count": { 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_osd_draining_count", 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.832 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: "migration_current": { 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "migration_current", 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.833 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "internal - do not modify", 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": { 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mode", 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "root", 2024-09-16T00:43:05.834 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm-package", 2024-09-16T00:43:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: "root" 2024-09-16T00:43:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "mode for remote execution of cephadm", 2024-09-16T00:43:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: "oob_default_addr": { 2024-09-16T00:43:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "oob_default_addr", 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "169.254.1.1", 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.836 INFO:teuthology.orchestra.run.smithi033.stdout: "prometheus_alerts_path": { 2024-09-16T00:43:05.837 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prometheus_alerts_path", 2024-09-16T00:43:05.837 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.837 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.837 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.837 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-16T00:43:05.837 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.837 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.837 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.837 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-16T00:43:05.837 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.837 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.837 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_insecure": { 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_insecure", 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.838 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_password": { 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_password", 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.839 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.840 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.840 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_url": { 2024-09-16T00:43:05.840 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_url", 2024-09-16T00:43:05.840 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.840 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.840 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.840 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.840 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.840 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.840 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.840 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_username": { 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_username", 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.841 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.842 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-16T00:43:05.842 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.842 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.842 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.842 INFO:teuthology.orchestra.run.smithi033.stdout: "secure_monitoring_stack": { 2024-09-16T00:43:05.842 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "secure_monitoring_stack", 2024-09-16T00:43:05.842 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.842 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.842 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.842 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.842 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.843 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.843 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.843 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-16T00:43:05.843 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.843 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.843 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.843 INFO:teuthology.orchestra.run.smithi033.stdout: "service_discovery_port": { 2024-09-16T00:43:05.843 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "service_discovery_port", 2024-09-16T00:43:05.843 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.843 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.843 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.844 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8765", 2024-09-16T00:43:05.844 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.844 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.844 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.844 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "cephadm service discovery port", 2024-09-16T00:43:05.844 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.844 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.844 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.844 INFO:teuthology.orchestra.run.smithi033.stdout: "ssh_config_file": { 2024-09-16T00:43:05.844 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssh_config_file", 2024-09-16T00:43:05.844 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: "use_agent": { 2024-09-16T00:43:05.845 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "use_agent", 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.846 INFO:teuthology.orchestra.run.smithi033.stdout: "use_repo_digest": { 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "use_repo_digest", 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_failed_host_check": { 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_failed_host_check", 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if the host check fails", 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.848 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_stray_daemons": { 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_stray_daemons", 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.849 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.850 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.850 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.850 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_stray_hosts": { 2024-09-16T00:43:05.850 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_stray_hosts", 2024-09-16T00:43:05.850 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.850 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.850 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.850 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.850 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.850 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.850 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.850 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crash", 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.851 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.852 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.852 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.852 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.852 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.852 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.852 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.852 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.852 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.852 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.852 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.852 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.852 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.853 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.853 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.853 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.853 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.853 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.853 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.853 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.853 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.853 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.853 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.853 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.854 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.855 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.856 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.857 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.857 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.857 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.857 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.857 INFO:teuthology.orchestra.run.smithi033.stdout: "retain_interval": { 2024-09-16T00:43:05.857 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "retain_interval", 2024-09-16T00:43:05.857 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.857 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.857 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.857 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "31536000", 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_recent_interval": { 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_recent_interval", 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.858 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1209600", 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.859 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "dashboard", 2024-09-16T00:43:05.860 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.860 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.860 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.860 INFO:teuthology.orchestra.run.smithi033.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-16T00:43:05.860 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-16T00:43:05.860 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.860 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.860 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.860 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.860 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.860 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: "ALERTMANAGER_API_HOST": { 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.861 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.862 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.862 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.862 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.862 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.862 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.862 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.862 INFO:teuthology.orchestra.run.smithi033.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-16T00:43:05.862 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-16T00:43:05.862 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.862 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.862 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: "AUDIT_API_ENABLED": { 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "AUDIT_API_ENABLED", 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.863 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.864 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.864 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.864 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.864 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.864 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.864 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.864 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.864 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.864 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.864 INFO:teuthology.orchestra.run.smithi033.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-16T00:43:05.864 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.865 INFO:teuthology.orchestra.run.smithi033.stdout: "ENABLE_BROWSABLE_API": { 2024-09-16T00:43:05.866 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-16T00:43:05.866 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.866 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.866 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.866 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.866 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.866 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.866 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.866 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.866 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.866 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.866 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.867 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.868 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.868 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-16T00:43:05.868 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-16T00:43:05.868 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.868 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.868 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.868 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.868 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.868 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.868 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.868 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.869 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.869 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.869 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.869 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-16T00:43:05.869 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-16T00:43:05.869 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.869 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.869 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.869 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.869 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.869 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.870 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.871 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.871 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.871 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.871 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.871 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.871 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.871 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-16T00:43:05.871 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-16T00:43:05.871 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.871 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.871 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.872 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.872 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.872 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.872 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.872 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.872 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.872 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.872 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.872 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-16T00:43:05.872 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-16T00:43:05.872 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-16T00:43:05.873 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-16T00:43:05.874 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-16T00:43:05.875 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.875 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.875 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.875 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.875 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.875 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.875 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.875 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.875 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.875 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.875 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_PASSWORD": { 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.876 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.877 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-16T00:43:05.877 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-16T00:43:05.877 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.877 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.877 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.877 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.877 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.877 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.877 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.877 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.877 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_URL": { 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_URL", 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.878 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_USERNAME": { 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.879 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.880 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.880 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.880 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.880 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.880 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-16T00:43:05.880 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-16T00:43:05.880 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.880 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.880 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.880 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.881 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.882 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.882 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.882 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.882 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.882 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.882 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.882 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.882 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.882 INFO:teuthology.orchestra.run.smithi033.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-16T00:43:05.882 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-16T00:43:05.882 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-16T00:43:05.883 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.884 INFO:teuthology.orchestra.run.smithi033.stdout: "PROMETHEUS_API_HOST": { 2024-09-16T00:43:05.885 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-16T00:43:05.885 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.885 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.885 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.885 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.885 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.885 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.885 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.885 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.885 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.885 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.885 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.886 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.887 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.888 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.888 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.888 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-16T00:43:05.888 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-16T00:43:05.888 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.888 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.888 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.888 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.888 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.888 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.888 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.889 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.889 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.889 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.889 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.889 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-16T00:43:05.889 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-16T00:43:05.889 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.889 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.889 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.889 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.889 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.890 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.891 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.892 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.892 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.892 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.892 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.892 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.892 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.892 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.892 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.892 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-16T00:43:05.892 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-16T00:43:05.892 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-16T00:43:05.893 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.894 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_ENABLED": { 2024-09-16T00:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-16T00:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.896 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.897 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.897 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.897 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-16T00:43:05.897 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-16T00:43:05.897 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.897 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.897 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.897 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.897 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.897 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.897 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.897 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8", 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.898 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "45", 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.899 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_ACCESS_KEY": { 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.900 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-09-16T00:43:05.901 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_SECRET_KEY": { 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.902 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_SSL_VERIFY": { 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.903 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.904 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.904 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.904 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.904 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.904 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.904 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.904 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.904 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-16T00:43:05.904 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-16T00:43:05.904 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.904 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.905 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.905 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:05.905 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.905 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.905 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.905 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.905 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.905 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.905 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.905 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-16T00:43:05.905 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.906 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "cross_origin_url": { 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cross_origin_url", 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "crt_file": { 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crt_file", 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.909 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.910 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.910 INFO:teuthology.orchestra.run.smithi033.stdout: "debug": { 2024-09-16T00:43:05.910 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "debug", 2024-09-16T00:43:05.910 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.910 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.910 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.910 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.910 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.910 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.910 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.910 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable/disable debug options", 2024-09-16T00:43:05.910 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: "jwt_token_ttl": { 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "jwt_token_ttl", 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "28800", 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.911 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: "key_file": { 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "key_file", 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.912 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.913 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.913 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.913 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.913 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.913 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.913 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.913 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.913 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.913 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.913 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.913 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.914 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.915 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.917 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "motd": { 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "motd", 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.918 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "The message of the day", 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "redirect_resolve_ip_addr": { 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "redirect_resolve_ip_addr", 2024-09-16T00:43:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-09-16T00:43:05.920 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.921 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.921 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.921 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "::", 2024-09-16T00:43:05.921 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.921 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.921 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.921 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.921 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.921 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.921 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.921 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8080", 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl": { 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl", 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.923 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.924 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl_server_port": { 2024-09-16T00:43:05.924 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl_server_port", 2024-09-16T00:43:05.924 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.924 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.924 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.924 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8443", 2024-09-16T00:43:05.924 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.924 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.924 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.924 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.924 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.925 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.925 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.925 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_behaviour": { 2024-09-16T00:43:05.925 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_behaviour", 2024-09-16T00:43:05.925 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.925 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.925 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.925 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "redirect", 2024-09-16T00:43:05.925 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.925 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.925 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: "redirect" 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_error_status_code": { 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_error_status_code", 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.926 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "500", 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "400", 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "599", 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "url_prefix": { 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "url_prefix", 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.927 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.928 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.928 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.928 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.928 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.928 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.928 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.928 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.928 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.928 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.928 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "devicehealth", 2024-09-16T00:43:05.929 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.929 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.929 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.929 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_monitoring": { 2024-09-16T00:43:05.929 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enable_monitoring", 2024-09-16T00:43:05.929 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.929 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.929 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.929 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.929 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.929 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.929 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "monitor device health metrics", 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.930 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.933 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.933 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.933 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.933 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.933 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.933 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.933 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.933 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.933 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.933 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.933 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.933 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.934 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mark_out_threshold": { 2024-09-16T00:43:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mark_out_threshold", 2024-09-16T00:43:05.935 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "2419200", 2024-09-16T00:43:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-16T00:43:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.936 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": { 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pool_name", 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "device_health_metrics", 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.937 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "retention_period": { 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "retention_period", 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15552000", 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long to retain device health metrics", 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.938 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "scrape_frequency": { 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "scrape_frequency", 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "86400", 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.939 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: "self_heal": { 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "self_heal", 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.940 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to wake up and check device health", 2024-09-16T00:43:05.941 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_threshold": { 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_threshold", 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "7257600", 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-16T00:43:05.942 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "diskprediction_local", 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.943 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.944 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.945 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.946 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.946 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.946 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.946 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.946 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.946 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.946 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.946 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.946 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.947 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.948 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.948 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.948 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.948 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.948 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.948 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.948 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.948 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.948 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.948 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.948 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.948 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: "predict_interval": { 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "predict_interval", 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.949 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "86400", 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "predictor_model": { 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "predictor_model", 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.950 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "prophetstor", 2024-09-16T00:43:05.951 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.951 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.951 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.951 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.951 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.951 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.951 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.951 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.951 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:05.951 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:05.951 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.951 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.952 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.952 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-09-16T00:43:05.952 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.952 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.952 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.952 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.952 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.952 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.952 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.952 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.952 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.952 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.953 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.953 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "influx", 2024-09-16T00:43:05.953 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": false, 2024-09-16T00:43:05.953 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "influxdb python module not found", 2024-09-16T00:43:05.953 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.953 INFO:teuthology.orchestra.run.smithi033.stdout: "batch_size": { 2024-09-16T00:43:05.953 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "batch_size", 2024-09-16T00:43:05.953 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.953 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.953 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.953 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5000", 2024-09-16T00:43:05.953 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: "database": { 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "database", 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.954 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "ceph", 2024-09-16T00:43:05.955 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.955 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.955 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.955 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-16T00:43:05.955 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.955 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.955 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.955 INFO:teuthology.orchestra.run.smithi033.stdout: "hostname": { 2024-09-16T00:43:05.955 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "hostname", 2024-09-16T00:43:05.955 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.955 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB server hostname", 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:05.956 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "30", 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "5", 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.957 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.958 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.959 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.960 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.960 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.960 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.960 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.960 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.960 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.960 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.960 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.960 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.960 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.960 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.961 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.961 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.961 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.961 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.961 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.961 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.961 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.961 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.961 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.961 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.961 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.961 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.962 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: "password": { 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "password", 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.963 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.964 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.964 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "password of InfluxDB server user", 2024-09-16T00:43:05.964 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.964 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.964 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.964 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.964 INFO:teuthology.orchestra.run.smithi033.stdout: "port": { 2024-09-16T00:43:05.964 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "port", 2024-09-16T00:43:05.964 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.964 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.964 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.964 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8086", 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB server port", 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl": { 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl", 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.965 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "false", 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: "threads": { 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "threads", 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.966 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "32", 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "username": { 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "username", 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.967 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.968 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.968 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.968 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.968 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.968 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "username of InfluxDB server user", 2024-09-16T00:43:05.968 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.968 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.968 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.968 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.968 INFO:teuthology.orchestra.run.smithi033.stdout: "verify_ssl": { 2024-09-16T00:43:05.968 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "verify_ssl", 2024-09-16T00:43:05.968 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "true", 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.969 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "insights", 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.970 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.971 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.972 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.973 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.974 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "iostat", 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.975 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.976 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.976 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.976 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.976 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.976 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.976 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.976 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.976 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.976 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.976 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.976 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.976 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.977 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.978 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.979 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.980 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "k8sevents", 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.981 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_event_retention_days": { 2024-09-16T00:43:05.982 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ceph_event_retention_days", 2024-09-16T00:43:05.982 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.982 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.982 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.982 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "7", 2024-09-16T00:43:05.982 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.982 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.982 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.982 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-16T00:43:05.982 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.982 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.982 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: "config_check_secs": { 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_check_secs", 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "10", 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.983 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.984 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.985 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.986 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "localpool", 2024-09-16T00:43:05.989 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:05.990 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:05.990 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:05.990 INFO:teuthology.orchestra.run.smithi033.stdout: "failure_domain": { 2024-09-16T00:43:05.990 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "failure_domain", 2024-09-16T00:43:05.990 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.990 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.990 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.990 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "host", 2024-09-16T00:43:05.990 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.990 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.990 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.990 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "failure domain for any created local pool", 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.991 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.992 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.992 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.992 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.992 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.992 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.992 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.992 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.992 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.992 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.992 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.992 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.993 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:05.994 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:05.995 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:05.995 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:05.995 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:05.995 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:05.995 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:05.995 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.995 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.995 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.995 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.995 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:05.995 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": { 2024-09-16T00:43:05.996 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "min_size", 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default min_size for any created local pool", 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.997 INFO:teuthology.orchestra.run.smithi033.stdout: "num_rep": { 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "num_rep", 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3", 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default replica count for any created local pool", 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:05.998 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": { 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pg_num", 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "128", 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default pg_num for any created local pool", 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:05.999 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.000 INFO:teuthology.orchestra.run.smithi033.stdout: "prefix": { 2024-09-16T00:43:06.000 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prefix", 2024-09-16T00:43:06.000 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.000 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.000 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.000 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.000 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.000 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.000 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.000 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "name prefix for any created local pool", 2024-09-16T00:43:06.000 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.000 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: "subtree": { 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "subtree", 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "rack", 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-16T00:43:06.001 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-16T00:43:06.002 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.002 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.002 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.002 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.002 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mds_autoscaler", 2024-09-16T00:43:06.002 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.002 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.002 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.002 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.002 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.004 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.005 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.005 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.005 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.005 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.005 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.005 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.005 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.005 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.005 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.005 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.006 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.007 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mirroring", 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.008 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.009 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.010 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.011 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.012 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.013 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.013 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.013 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.013 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.013 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.013 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.013 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.013 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.013 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.013 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.013 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.013 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "nfs", 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.014 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.015 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.016 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.017 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.017 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.017 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.017 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.017 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.017 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.017 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.017 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.017 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.017 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.017 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.019 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "orchestrator", 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: "fail_fs": { 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "fail_fs", 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.020 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.021 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.021 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.021 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.021 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.021 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-16T00:43:06.021 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.021 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.021 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.021 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.021 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.021 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.022 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.023 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.024 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.025 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.026 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.026 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.026 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.026 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.026 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.026 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.026 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.026 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.026 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.026 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.026 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator": { 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "orchestrator", 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.027 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm", 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: "rook", 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: "test_orchestrator" 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Orchestrator backend", 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "osd_perf_query", 2024-09-16T00:43:06.028 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.029 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.029 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.029 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.029 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.029 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.029 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.029 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.029 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.029 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.029 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.029 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.029 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.030 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.031 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.031 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.031 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.031 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.031 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.031 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.031 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.031 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.031 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.031 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.031 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.031 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "osd_support", 2024-09-16T00:43:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pg_autoscaler", 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "threshold": { 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "threshold", 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3.0", 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1.0", 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "scaling threshold", 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-16T00:43:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "progress", 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_pg_recovery_event": { 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_pg_recovery_event", 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": { 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enabled", 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "max_completed_events": { 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_completed_events", 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "50", 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "number of past completed events to remember", 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long the module is going to sleep", 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prometheus", 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "cache": { 2024-09-16T00:43:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cache", 2024-09-16T00:43:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "exclude_perf_counters": { 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "exclude_perf_counters", 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_stats_pools": { 2024-09-16T00:43:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_stats_pools", 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-16T00:43:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-16T00:43:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "300", 2024-09-16T00:43:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "scrape_interval": { 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "scrape_interval", 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15.0", 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "::", 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-16T00:43:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-09-16T00:43:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-09-16T00:43:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "9283", 2024-09-16T00:43:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "stale_cache_strategy": { 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "stale_cache_strategy", 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "log", 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_behaviour": { 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_behaviour", 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "default", 2024-09-16T00:43:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "default", 2024-09-16T00:43:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "error" 2024-09-16T00:43:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_error_status_code": { 2024-09-16T00:43:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_error_status_code", 2024-09-16T00:43:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "500", 2024-09-16T00:43:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "400", 2024-09-16T00:43:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "599", 2024-09-16T00:43:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_support", 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "max_concurrent_snap_create": { 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_concurrent_snap_create", 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "mirror_snapshot_schedule": { 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mirror_snapshot_schedule", 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "trash_purge_schedule": { 2024-09-16T00:43:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "trash_purge_schedule", 2024-09-16T00:43:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "restful", 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_auth": { 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enable_auth", 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "key_file": { 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "key_file", 2024-09-16T00:43:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-09-16T00:43:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-09-16T00:43:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rgw", 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "secondary_zone_period_retry_limit": { 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-16T00:43:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rook", 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "storage_class": { 2024-09-16T00:43:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "storage_class", 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "local", 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "selftest", 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "roption1": { 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "roption1", 2024-09-16T00:43:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "roption2": { 2024-09-16T00:43:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "roption2", 2024-09-16T00:43:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "xyz", 2024-09-16T00:43:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption1": { 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption1", 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption2": { 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption2", 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption3": { 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption3", 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption4": { 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption4", 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption5": { 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption5", 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption6": { 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption6", 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption7": { 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption7", 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "42", 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "testkey": { 2024-09-16T00:43:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testkey", 2024-09-16T00:43:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "testlkey": { 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testlkey", 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "testnewline": { 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testnewline", 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snap_schedule", 2024-09-16T00:43:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_m_granularity": { 2024-09-16T00:43:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_m_granularity", 2024-09-16T00:43:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow minute scheduled snapshots", 2024-09-16T00:43:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "dump_on_update": { 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "dump_on_update", 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "dump database to debug log on update", 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "stats", 2024-09-16T00:43:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "status", 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "telegraf", 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "address": { 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "address", 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-16T00:43:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15", 2024-09-16T00:43:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "telemetry", 2024-09-16T00:43:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_basic": { 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_basic", 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_crash": { 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_crash", 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_device": { 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_device", 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_ident": { 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_ident", 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_perf": { 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_perf", 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "contact": { 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "contact", 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "description": { 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "description", 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "device_url": { 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_url", 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": { 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enabled", 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "24", 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "8", 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "last_opt_revision": { 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "last_opt_revision", 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1", 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "leaderboard": { 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "leaderboard", 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "leaderboard_description": { 2024-09-16T00:43:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "leaderboard_description", 2024-09-16T00:43:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "organization": { 2024-09-16T00:43:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "organization", 2024-09-16T00:43:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "proxy": { 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "proxy", 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "url": { 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "url", 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "test_orchestrator", 2024-09-16T00:43:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "volumes", 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "max_concurrent_clones": { 2024-09-16T00:43:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_concurrent_clones", 2024-09-16T00:43:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "4", 2024-09-16T00:43:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-16T00:43:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "periodic_async_work": { 2024-09-16T00:43:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "periodic_async_work", 2024-09-16T00:43:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Periodically check for async work", 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "snapshot_clone_delay": { 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snapshot_clone_delay", 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "snapshot_clone_no_wait": { 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snapshot_clone_no_wait", 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix", 2024-09-16T00:43:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "discovery_interval": { 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "discovery_interval", 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "100", 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "identifier": { 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "identifier", 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.203 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.204 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.205 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-09-16T00:43:06.206 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-09-16T00:43:06.207 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-09-16T00:43:06.207 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-09-16T00:43:06.207 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-09-16T00:43:06.207 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.207 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.207 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.207 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.207 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.207 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-09-16T00:43:06.207 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-09-16T00:43:06.207 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_host": { 2024-09-16T00:43:06.208 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_host", 2024-09-16T00:43:06.209 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.209 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.209 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.209 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-09-16T00:43:06.209 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.209 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.209 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.209 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.209 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.209 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.209 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.209 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_port": { 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_port", 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10051", 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_sender": { 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_sender", 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-09-16T00:43:06.211 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.212 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.212 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.212 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.212 INFO:teuthology.orchestra.run.smithi033.stdout: "services": { 2024-09-16T00:43:06.212 INFO:teuthology.orchestra.run.smithi033.stdout: "dashboard": "https://172.21.15.33:8443/" 2024-09-16T00:43:06.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.212 INFO:teuthology.orchestra.run.smithi033.stdout: "always_on_modules": { 2024-09-16T00:43:06.212 INFO:teuthology.orchestra.run.smithi033.stdout: "octopus": [ 2024-09-16T00:43:06.212 INFO:teuthology.orchestra.run.smithi033.stdout: "balancer", 2024-09-16T00:43:06.212 INFO:teuthology.orchestra.run.smithi033.stdout: "crash", 2024-09-16T00:43:06.212 INFO:teuthology.orchestra.run.smithi033.stdout: "devicehealth", 2024-09-16T00:43:06.212 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator", 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscaler", 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "progress", 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_support", 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "status", 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "telemetry", 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "volumes" 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "pacific": [ 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "balancer", 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "crash", 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "devicehealth", 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator", 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscaler", 2024-09-16T00:43:06.213 INFO:teuthology.orchestra.run.smithi033.stdout: "progress", 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_support", 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: "status", 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: "telemetry", 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: "volumes" 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: "quincy": [ 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: "balancer", 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: "crash", 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: "devicehealth", 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator", 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscaler", 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: "progress", 2024-09-16T00:43:06.214 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_support", 2024-09-16T00:43:06.215 INFO:teuthology.orchestra.run.smithi033.stdout: "status", 2024-09-16T00:43:06.215 INFO:teuthology.orchestra.run.smithi033.stdout: "telemetry", 2024-09-16T00:43:06.215 INFO:teuthology.orchestra.run.smithi033.stdout: "volumes" 2024-09-16T00:43:06.215 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.215 INFO:teuthology.orchestra.run.smithi033.stdout: "reef": [ 2024-09-16T00:43:06.215 INFO:teuthology.orchestra.run.smithi033.stdout: "balancer", 2024-09-16T00:43:06.215 INFO:teuthology.orchestra.run.smithi033.stdout: "crash", 2024-09-16T00:43:06.215 INFO:teuthology.orchestra.run.smithi033.stdout: "devicehealth", 2024-09-16T00:43:06.215 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator", 2024-09-16T00:43:06.215 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscaler", 2024-09-16T00:43:06.215 INFO:teuthology.orchestra.run.smithi033.stdout: "progress", 2024-09-16T00:43:06.215 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_support", 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: "status", 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: "telemetry", 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: "volumes" 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: "squid": [ 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: "balancer", 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: "crash", 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: "devicehealth", 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator", 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscaler", 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: "progress", 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_support", 2024-09-16T00:43:06.216 INFO:teuthology.orchestra.run.smithi033.stdout: "status", 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: "telemetry", 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: "volumes" 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: "last_failure_osd_epoch": 3, 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: "active_clients": [ 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "devicehealth", 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:0", 2024-09-16T00:43:06.217 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3761745238 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "libcephsqlite", 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:0", 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1917125516 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-09-16T00:43:06.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_support", 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:0", 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1628705799 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "volumes", 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-09-16T00:43:06.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-09-16T00:43:06.220 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-09-16T00:43:06.220 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:0", 2024-09-16T00:43:06.220 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2583695893 2024-09-16T00:43:06.220 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.220 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-09-16T00:43:06.220 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:06.220 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-09-16T00:43:06.220 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:43:06.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.796+0000 7f0a36c0e640 1 -- 172.21.15.33:0/3034687166 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0a180a3250 msgr2=0x7f0a180a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:06.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.796+0000 7f0a36c0e640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0a180a3250 0x7f0a180a5710 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0a30101cb0 tx=0x7f0a24005e50 comp rx=0 tx=0).stop 2024-09-16T00:43:06.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.796+0000 7f0a36c0e640 1 -- 172.21.15.33:0/3034687166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a301025d0 msgr2=0x7f0a30101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:06.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.796+0000 7f0a36c0e640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a301025d0 0x7f0a30101f40 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f0a20009230 tx=0x7f0a20004360 comp rx=0 tx=0).stop 2024-09-16T00:43:06.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.796+0000 7f0a36c0e640 1 -- 172.21.15.33:0/3034687166 shutdown_connections 2024-09-16T00:43:06.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.796+0000 7f0a36c0e640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f0a180a3250 0x7f0a180a5710 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:06.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.796+0000 7f0a36c0e640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a30107830 0x7f0a30100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:06.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.796+0000 7f0a36c0e640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a30104f00 0x7f0a30100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:06.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.796+0000 7f0a36c0e640 1 --2- 172.21.15.33:0/3034687166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a301025d0 0x7f0a30101f40 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:06.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.796+0000 7f0a36c0e640 1 -- 172.21.15.33:0/3034687166 >> 172.21.15.33:0/3034687166 conn(0x7f0a300fc190 msgr2=0x7f0a30102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:06.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.797+0000 7f0a36c0e640 1 -- 172.21.15.33:0/3034687166 shutdown_connections 2024-09-16T00:43:06.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:04.797+0000 7f0a36c0e640 1 -- 172.21.15.33:0/3034687166 wait complete. 2024-09-16T00:43:06.223 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:04 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/9893406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-16T00:43:06.223 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:04 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/946037590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-16T00:43:06.223 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:04 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3542825756' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-16T00:43:06.223 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:04 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/9893406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-16T00:43:06.223 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:04 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/946037590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-16T00:43:06.223 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:04 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3542825756' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-16T00:43:06.223 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:05 smithi060 ceph-mon[35185]: pgmap v1128: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 345 op/s 2024-09-16T00:43:06.223 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:05 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3034687166' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-16T00:43:06.223 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:04 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/9893406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-16T00:43:06.224 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:04 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/946037590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-16T00:43:06.224 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:04 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3542825756' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-16T00:43:06.224 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:05 smithi083 ceph-mon[35242]: pgmap v1128: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 345 op/s 2024-09-16T00:43:06.224 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:05 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3034687166' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-16T00:43:06.281 DEBUG:tasks.check_counter:Getting stats from a 2024-09-16T00:43:06.281 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mds.a perf dump 2024-09-16T00:43:06.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:05 smithi033 ceph-mon[31243]: pgmap v1128: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 345 op/s 2024-09-16T00:43:06.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:05 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3034687166' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-16T00:43:06.556 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:07.085 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:43:07.086 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T00:43:07.086 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 454, 2024-09-16T00:43:07.086 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 449, 2024-09-16T00:43:07.086 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 823776, 2024-09-16T00:43:07.086 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 278032, 2024-09-16T00:43:07.086 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-09-16T00:43:07.086 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 0, 2024-09-16T00:43:07.086 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.056636089, 2024-09-16T00:43:07.086 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.026803969, 2024-09-16T00:43:07.086 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.033850990, 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 448, 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.021476002, 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000047937 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 823776, 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 278032 2024-09-16T00:43:07.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:07.088 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T00:43:07.088 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 6, 2024-09-16T00:43:07.088 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 345, 2024-09-16T00:43:07.088 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 3344, 2024-09-16T00:43:07.088 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 218224, 2024-09-16T00:43:07.088 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-09-16T00:43:07.088 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 2, 2024-09-16T00:43:07.088 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.025152712, 2024-09-16T00:43:07.088 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.018451281, 2024-09-16T00:43:07.088 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.003426525, 2024-09-16T00:43:07.088 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:07.088 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 344, 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.018164800, 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000052804 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 218224 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T00:43:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 0, 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 2, 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 0, 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 0, 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 2, 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 1, 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.000327789, 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.000000000, 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.000225579, 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:07.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 0, 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:07.091 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-09-16T00:43:07.092 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-09-16T00:43:07.092 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-09-16T00:43:07.092 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T00:43:07.092 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-09-16T00:43:07.092 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T00:43:07.092 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-09-16T00:43:07.092 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T00:43:07.092 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T00:43:07.092 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T00:43:07.092 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T00:43:07.092 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T00:43:07.092 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-09-16T00:43:07.093 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 148744, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 30, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-09-16T00:43:07.094 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 0, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 0, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 928, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 2, 2024-09-16T00:43:07.095 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 456, 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 759562, 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 456, 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-16T00:43:07.096 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-16T00:43:07.097 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 456, 2024-09-16T00:43:07.097 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 759562, 2024-09-16T00:43:07.097 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-16T00:43:07.097 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:07.097 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:07.097 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:07.097 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:07.097 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:07.097 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:43:07.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:07 smithi033 ceph-mon[31243]: pgmap v1129: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 1.5 MiB/s wr, 198 op/s 2024-09-16T00:43:07.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:07 smithi083 ceph-mon[35242]: pgmap v1129: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 1.5 MiB/s wr, 198 op/s 2024-09-16T00:43:07.961 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-09-16T00:43:07.961 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-09-16T00:43:07.961 DEBUG:tasks.check_counter:Getting stats from d 2024-09-16T00:43:07.962 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mds.d perf dump 2024-09-16T00:43:07.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:07 smithi060 ceph-mon[35185]: pgmap v1129: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 1.5 MiB/s wr, 198 op/s 2024-09-16T00:43:08.220 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:08.979 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:43:08.980 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T00:43:08.980 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 0, 2024-09-16T00:43:08.980 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 2, 2024-09-16T00:43:08.980 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 0, 2024-09-16T00:43:08.980 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 0, 2024-09-16T00:43:08.980 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-09-16T00:43:08.980 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 0, 2024-09-16T00:43:08.980 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.001557900, 2024-09-16T00:43:08.980 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.000000000, 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.000150026, 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:08.981 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 463, 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 456, 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 828160, 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 281680, 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 2, 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 1, 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.058701618, 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.028500971, 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.036014967, 2024-09-16T00:43:08.982 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:08.983 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:08.983 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 456, 2024-09-16T00:43:08.983 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.027234089, 2024-09-16T00:43:08.983 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000059723 2024-09-16T00:43:08.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:08.983 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:08.983 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:08.983 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:08.983 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:08.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:08.984 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 828160, 2024-09-16T00:43:08.984 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 281680 2024-09-16T00:43:08.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:08.984 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T00:43:08.984 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 2, 2024-09-16T00:43:08.984 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 347, 2024-09-16T00:43:08.984 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 320, 2024-09-16T00:43:08.985 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 218480, 2024-09-16T00:43:08.985 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-09-16T00:43:08.985 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 2, 2024-09-16T00:43:08.985 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.023942442, 2024-09-16T00:43:08.985 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.019296909, 2024-09-16T00:43:08.986 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.001438879, 2024-09-16T00:43:08.986 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:08.986 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:08.986 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 345, 2024-09-16T00:43:08.986 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.017057242, 2024-09-16T00:43:08.986 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000049441 2024-09-16T00:43:08.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:08.986 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:08.986 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:08.986 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 218480 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 0, 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-09-16T00:43:08.987 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-09-16T00:43:08.988 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T00:43:08.989 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-09-16T00:43:08.990 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-09-16T00:43:08.990 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-09-16T00:43:08.990 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-09-16T00:43:08.990 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T00:43:08.990 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-09-16T00:43:08.990 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T00:43:08.990 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-09-16T00:43:08.990 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 148704, 2024-09-16T00:43:08.990 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 29, 2024-09-16T00:43:08.990 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-09-16T00:43:08.990 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-09-16T00:43:08.990 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 0, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 0, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 928, 2024-09-16T00:43:08.991 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 2, 2024-09-16T00:43:08.992 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-09-16T00:43:08.992 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-09-16T00:43:08.992 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-09-16T00:43:08.992 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-09-16T00:43:08.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:08.992 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T00:43:08.992 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-16T00:43:08.992 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-09-16T00:43:08.992 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-16T00:43:08.992 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 461, 2024-09-16T00:43:08.992 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 760212, 2024-09-16T00:43:08.993 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:08.993 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 461, 2024-09-16T00:43:08.993 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-16T00:43:08.993 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-16T00:43:08.993 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 461, 2024-09-16T00:43:08.993 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 760212, 2024-09-16T00:43:08.993 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-16T00:43:08.993 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:08.993 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:08.993 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:08.994 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:08.994 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:08.994 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:43:09.737 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-09-16T00:43:09.737 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-09-16T00:43:09.738 DEBUG:tasks.check_counter:Getting stats from g 2024-09-16T00:43:09.738 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mds.g perf dump 2024-09-16T00:43:09.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:09 smithi083 ceph-mon[35242]: pgmap v1130: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 879 KiB/s wr, 167 op/s 2024-09-16T00:43:09.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:09 smithi060 ceph-mon[35185]: pgmap v1130: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 879 KiB/s wr, 167 op/s 2024-09-16T00:43:09.994 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:10.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:09 smithi033 ceph-mon[31243]: pgmap v1130: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 879 KiB/s wr, 167 op/s 2024-09-16T00:43:10.484 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:43:10.484 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T00:43:10.484 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 456, 2024-09-16T00:43:10.484 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 451, 2024-09-16T00:43:10.485 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 824320, 2024-09-16T00:43:10.485 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 279312, 2024-09-16T00:43:10.485 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-09-16T00:43:10.485 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 0, 2024-09-16T00:43:10.485 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.058621548, 2024-09-16T00:43:10.485 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.027747000, 2024-09-16T00:43:10.485 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.035770932, 2024-09-16T00:43:10.485 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:10.485 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:10.485 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 450, 2024-09-16T00:43:10.485 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.021580493, 2024-09-16T00:43:10.486 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000047956 2024-09-16T00:43:10.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:10.486 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:10.486 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:10.486 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:10.486 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:10.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:10.486 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 824320, 2024-09-16T00:43:10.486 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 279312 2024-09-16T00:43:10.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:10.487 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T00:43:10.487 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 2, 2024-09-16T00:43:10.487 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 344, 2024-09-16T00:43:10.487 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 320, 2024-09-16T00:43:10.487 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 217952, 2024-09-16T00:43:10.487 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-09-16T00:43:10.487 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 2, 2024-09-16T00:43:10.487 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.024576029, 2024-09-16T00:43:10.487 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.019704751, 2024-09-16T00:43:10.487 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.001444486, 2024-09-16T00:43:10.487 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:10.487 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:10.488 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 342, 2024-09-16T00:43:10.488 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.023404403, 2024-09-16T00:43:10.488 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000068433 2024-09-16T00:43:10.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:10.488 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:10.488 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:10.488 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:10.488 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:10.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:10.488 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-16T00:43:10.488 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 217952 2024-09-16T00:43:10.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 4, 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 4, 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 3024, 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 448, 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 2, 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 1, 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.002362499, 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.000039548, 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.002297583, 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:10.489 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 3, 2024-09-16T00:43:10.490 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.004058081, 2024-09-16T00:43:10.490 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.001352693 2024-09-16T00:43:10.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:10.490 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:10.490 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:10.490 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:10.490 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:10.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:10.490 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-09-16T00:43:10.490 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 448 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 0, 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-09-16T00:43:10.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T00:43:10.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T00:43:10.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T00:43:10.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T00:43:10.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T00:43:10.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-09-16T00:43:10.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T00:43:10.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T00:43:10.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-09-16T00:43:10.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-09-16T00:43:10.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-09-16T00:43:10.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-09-16T00:43:10.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T00:43:10.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T00:43:10.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T00:43:10.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T00:43:10.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T00:43:10.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-09-16T00:43:10.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-09-16T00:43:10.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-09-16T00:43:10.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T00:43:10.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T00:43:10.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-09-16T00:43:10.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-09-16T00:43:10.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 148744, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 30, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-09-16T00:43:10.494 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 0, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 0, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 928, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 2, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-09-16T00:43:10.495 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 458, 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 759822, 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 458, 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 458, 2024-09-16T00:43:10.496 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 759822, 2024-09-16T00:43:10.497 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-16T00:43:10.497 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:10.497 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:10.497 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:10.497 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:10.497 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:10.497 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:43:11.231 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-09-16T00:43:11.232 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-09-16T00:43:11.232 DEBUG:tasks.check_counter:Getting stats from j 2024-09-16T00:43:11.232 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mds.j perf dump 2024-09-16T00:43:11.490 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:11.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:11 smithi033 ceph-mon[31243]: pgmap v1131: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 380 KiB/s wr, 94 op/s 2024-09-16T00:43:11.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:11 smithi083 ceph-mon[35242]: pgmap v1131: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 380 KiB/s wr, 94 op/s 2024-09-16T00:43:11.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:11 smithi060 ceph-mon[35185]: pgmap v1131: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 380 KiB/s wr, 94 op/s 2024-09-16T00:43:12.019 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-16T00:43:12.019 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T00:43:12.019 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 4, 2024-09-16T00:43:12.019 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 4, 2024-09-16T00:43:12.019 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 3024, 2024-09-16T00:43:12.019 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 448, 2024-09-16T00:43:12.019 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-09-16T00:43:12.019 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 0, 2024-09-16T00:43:12.019 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.004537836, 2024-09-16T00:43:12.020 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.000058495, 2024-09-16T00:43:12.020 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.003572123, 2024-09-16T00:43:12.020 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:12.020 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:12.020 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 3, 2024-09-16T00:43:12.020 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.006507545, 2024-09-16T00:43:12.020 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.002169181 2024-09-16T00:43:12.020 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:12.020 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:12.020 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:12.020 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:12.020 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-09-16T00:43:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 448 2024-09-16T00:43:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T00:43:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 2, 2024-09-16T00:43:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 349, 2024-09-16T00:43:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 320, 2024-09-16T00:43:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 218832, 2024-09-16T00:43:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-09-16T00:43:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 2, 2024-09-16T00:43:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.022912517, 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.018406800, 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.001385447, 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 347, 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.016968130, 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000048899 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 218832 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 461, 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 456, 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 825680, 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 282512, 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 2, 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 1, 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.057798605, 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.027872504, 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.035154655, 2024-09-16T00:43:12.023 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:12.024 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:12.024 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 455, 2024-09-16T00:43:12.024 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.023364421, 2024-09-16T00:43:12.024 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000051350 2024-09-16T00:43:12.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:12.024 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:12.024 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:12.024 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:12.024 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:12.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:12.024 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 825680, 2024-09-16T00:43:12.024 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 282512 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 0, 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-09-16T00:43:12.025 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T00:43:12.026 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T00:43:12.027 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 148722, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 30, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-09-16T00:43:12.028 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 0, 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 0, 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 928, 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 2, 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-16T00:43:12.029 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 463, 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 760472, 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 463, 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 463, 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 760472, 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-16T00:43:12.030 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-16T00:43:12.031 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-16T00:43:12.031 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:12.031 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-16T00:43:12.031 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-16T00:43:12.802 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-09-16T00:43:12.802 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-09-16T00:43:12.802 DEBUG:tasks.check_counter:Getting stats from b 2024-09-16T00:43:12.802 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mds.b perf dump 2024-09-16T00:43:13.136 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T00:43:13.892 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:13 smithi060 ceph-mon[35185]: pgmap v1132: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.5 KiB/s wr, 19 op/s 2024-09-16T00:43:13.904 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-16T00:43:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T00:43:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 227457, 2024-09-16T00:43:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 138720, 2024-09-16T00:43:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 67422152, 2024-09-16T00:43:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 58055904, 2024-09-16T00:43:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 357, 2024-09-16T00:43:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 5, 2024-09-16T00:43:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 10.474540195, 2024-09-16T00:43:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 3.653958979, 2024-09-16T00:43:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 8.858146371, 2024-09-16T00:43:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.414581861, 2024-09-16T00:43:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 138719, 2024-09-16T00:43:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 3.137775695, 2024-09-16T00:43:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000022619 2024-09-16T00:43:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 342321, 2024-09-16T00:43:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.047991749, 2024-09-16T00:43:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000140 2024-09-16T00:43:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 37952, 2024-09-16T00:43:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 5568 2024-09-16T00:43:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T00:43:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 109034, 2024-09-16T00:43:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 144372, 2024-09-16T00:43:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 39430139, 2024-09-16T00:43:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 118837295, 2024-09-16T00:43:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 29, 2024-09-16T00:43:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 5, 2024-09-16T00:43:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 8.119065741, 2024-09-16T00:43:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 3.475159342, 2024-09-16T00:43:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 5.439515093, 2024-09-16T00:43:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.395888665, 2024-09-16T00:43:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 144371, 2024-09-16T00:43:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 4.985564469, 2024-09-16T00:43:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000034533 2024-09-16T00:43:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 211971, 2024-09-16T00:43:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.031509510, 2024-09-16T00:43:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000148 2024-09-16T00:43:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 36425712, 2024-09-16T00:43:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 98341184 2024-09-16T00:43:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 46213, 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 36140, 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 12460316, 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 17905747, 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 14, 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 6, 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 2.550474764, 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.886153759, 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 1.698625869, 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.307857871, 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 36138, 2024-09-16T00:43:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 1.119889334, 2024-09-16T00:43:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000030989 2024-09-16T00:43:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 67148, 2024-09-16T00:43:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.010224223, 2024-09-16T00:43:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000152 2024-09-16T00:43:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-16T00:43:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 789792 2024-09-16T00:43:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: "cct": { 2024-09-16T00:43:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "total_workers": 1, 2024-09-16T00:43:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "unhealthy_workers": 0 2024-09-16T00:43:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "finisher-MDSRank": { 2024-09-16T00:43:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "queue_len": 0, 2024-09-16T00:43:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "complete_latency": { 2024-09-16T00:43:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 22717, 2024-09-16T00:43:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 13.086714727, 2024-09-16T00:43:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000576075 2024-09-16T00:43:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "finisher-PurgeQueue": { 2024-09-16T00:43:13.912 INFO:teuthology.orchestra.run.smithi060.stdout: "queue_len": 0, 2024-09-16T00:43:13.912 INFO:teuthology.orchestra.run.smithi060.stdout: "complete_latency": { 2024-09-16T00:43:13.912 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 11861, 2024-09-16T00:43:13.912 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 1.221508799, 2024-09-16T00:43:13.912 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000102985 2024-09-16T00:43:13.912 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:13.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-09-16T00:43:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "request": 64336, 2024-09-16T00:43:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "reply": 35204, 2024-09-16T00:43:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "reply_latency": { 2024-09-16T00:43:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 35204, 2024-09-16T00:43:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 411.568147680, 2024-09-16T00:43:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.011690948 2024-09-16T00:43:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "slow_reply": 0, 2024-09-16T00:43:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "forward": 29132, 2024-09-16T00:43:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_fetch_complete": 0, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_fetch_keys": 0, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_commit": 12, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_split": 0, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_merge": 0, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes": 44, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_top": 0, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_bottom": 2, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_pin_tail": 42, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_pinned": 44, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_expired": 302, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_with_caps": 21, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": 21, 2024-09-16T00:43:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "subtrees": 4, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse": 110206, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_hit": 35256, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_forward": 29131, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_discover": 38, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_dir_fetch": 0, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_remote_ino": 0, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_lock": 999, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "load_cent": 0, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "q": 0, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "exported": 6, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "exported_inodes": 233, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "imported": 6, 2024-09-16T00:43:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "imported_inodes": 74, 2024-09-16T00:43:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_dir_fetch": 0, 2024-09-16T00:43:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_backtrace_fetch": 1108, 2024-09-16T00:43:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_peer_discover": 0, 2024-09-16T00:43:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rfiles": 4, 2024-09-16T00:43:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rbytes": 234, 2024-09-16T00:43:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rsnaps": 0, 2024-09-16T00:43:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_backtrace_fetch": 1108, 2024-09-16T00:43:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_set_tag": 0, 2024-09-16T00:43:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_backtrace_repaired": 0, 2024-09-16T00:43:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_inotable_repaired": 0, 2024-09-16T00:43:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dir_inodes": 271, 2024-09-16T00:43:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dir_base_inodes": 34, 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dirfrag_rstats": 237, 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_file_inodes": 837, 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_inode_file_caps": 2, 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_revoke": 10458, 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_grant": 35982, 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_trunc": 0, 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_caps": 25299, 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_caps_dirty": 5602, 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_cap_release": 2615, 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "process_request_cap_release": 32323 2024-09-16T00:43:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cache": { 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_update": 2, 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_update_receipt": 43, 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_try_discover": 29, 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_send_discover": 53, 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_handle_discover": 8, 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays": 19, 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays_delayed": 0, 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays_enqueuing": 0, 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_created": 6241, 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_enqueued": 6222, 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_reintegrated": 0, 2024-09-16T00:43:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_migrated": 0, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_processing": 0, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_enqueued": 0, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_prioritized": 0, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_started": 0, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_completed": 0, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_quiesce_path": 22, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_quiesce_inode": 1185, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_enqueue_scrub": 34, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_exportdir": 6, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_flush": 0, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_fragmentdir": 0, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_fragstats": 0, 2024-09-16T00:43:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_inodestats": 0 2024-09-16T00:43:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_log": { 2024-09-16T00:43:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "evlrg": 0, 2024-09-16T00:43:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "evadd": 39115, 2024-09-16T00:43:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "evex": 0, 2024-09-16T00:43:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "evtrm": 0, 2024-09-16T00:43:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "ev": 39115, 2024-09-16T00:43:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "evexg": 0, 2024-09-16T00:43:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "evexd": 0, 2024-09-16T00:43:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "segadd": 48, 2024-09-16T00:43:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "segex": 0, 2024-09-16T00:43:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "segtrm": 0, 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "seg": 48, 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "segmjr": 5, 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "segexg": 0, 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "segexd": 0, 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "expos": 4194304, 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "wrpos": 56537443, 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "rdpos": 0, 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "jlat": { 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 11038, 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 70.541748739, 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.006390808 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "replayed": 0 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_mem": { 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 48, 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "ino+": 6450, 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "ino-": 6402, 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "dir": 24, 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "dir+": 356, 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "dir-": 332, 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "dn": 105, 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "dn+": 11060, 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "dn-": 10955, 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "cap": 21, 2024-09-16T00:43:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "cap+": 7538, 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "cap-": 7517, 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "rss": 70944, 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "heap": 231712 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_server": { 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "dispatch_client_request": 97530, 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "dispatch_server_request": 42, 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_request": 64336, 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_session": 895, 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_peer_request": 12732, 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "req_create_latency": { 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 6259, 2024-09-16T00:43:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 183.897046771, 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.029381218 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getattr_latency": { 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 165, 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.110174035, 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000667721 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getfilelock_latency": { 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "req_link_latency": { 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookup_latency": { 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 9829, 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 52.853245238, 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.005377275 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookuphash_latency": { 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupino_latency": { 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 1, 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000199059, 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000199059 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupname_latency": { 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupparent_latency": { 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.927 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.927 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupsnap_latency": { 2024-09-16T00:43:13.927 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.927 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:13 smithi083 ceph-mon[35242]: pgmap v1132: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.5 KiB/s wr, 19 op/s 2024-09-16T00:43:13.928 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.928 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lssnap_latency": { 2024-09-16T00:43:13.928 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mkdir_latency": { 2024-09-16T00:43:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 20, 2024-09-16T00:43:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.020022625, 2024-09-16T00:43:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.001001131 2024-09-16T00:43:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mknod_latency": { 2024-09-16T00:43:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mksnap_latency": { 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "req_open_latency": { 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 1, 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000339645, 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000339645 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "req_readdir_latency": { 2024-09-16T00:43:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 11358, 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 8.861424330, 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000780192 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rename_latency": { 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 1057, 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 57.741994623, 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.054628187 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: "req_renamesnap_latency": { 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: "req_snapdiff_latency": { 2024-09-16T00:43:13.932 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.932 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.932 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.932 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmdir_latency": { 2024-09-16T00:43:13.932 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 1, 2024-09-16T00:43:13.932 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.015643955, 2024-09-16T00:43:13.932 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.015643955 2024-09-16T00:43:13.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.932 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmsnap_latency": { 2024-09-16T00:43:13.932 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.932 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmxattr_latency": { 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setattr_latency": { 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 20, 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.120951121, 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.006047556 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.933 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setdirlayout_latency": { 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setfilelock_latency": { 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 244, 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.128147636, 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000525195 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setlayout_latency": { 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.934 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.935 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setxattr_latency": { 2024-09-16T00:43:13.935 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 9, 2024-09-16T00:43:13.935 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.020865220, 2024-09-16T00:43:13.935 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.002318357 2024-09-16T00:43:13.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.935 INFO:teuthology.orchestra.run.smithi060.stdout: "req_symlink_latency": { 2024-09-16T00:43:13.935 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.935 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.936 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.936 INFO:teuthology.orchestra.run.smithi060.stdout: "req_unlink_latency": { 2024-09-16T00:43:13.936 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 6240, 2024-09-16T00:43:13.936 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 107.798093422, 2024-09-16T00:43:13.936 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.017275335 2024-09-16T00:43:13.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.936 INFO:teuthology.orchestra.run.smithi060.stdout: "cap_revoke_eviction": 0, 2024-09-16T00:43:13.936 INFO:teuthology.orchestra.run.smithi060.stdout: "cap_acquisition_throttle": 0, 2024-09-16T00:43:13.936 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getvxattr_latency": { 2024-09-16T00:43:13.936 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.936 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_sessions": { 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: "session_count": 1, 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: "session_add": 4, 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: "session_remove": 3, 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: "sessions_open": 1, 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: "sessions_stale": 0, 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: "total_load": 3438, 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: "average_load": 3438, 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: "avg_session_uptime": 1543, 2024-09-16T00:43:13.937 INFO:teuthology.orchestra.run.smithi060.stdout: "mdthresh_evicted": 0 2024-09-16T00:43:13.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.938 INFO:teuthology.orchestra.run.smithi060.stdout: "mempool": { 2024-09-16T00:43:13.938 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_bytes": 0, 2024-09-16T00:43:13.938 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_items": 0, 2024-09-16T00:43:13.938 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T00:43:13.938 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_items": 0, 2024-09-16T00:43:13.938 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T00:43:13.938 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_items": 0, 2024-09-16T00:43:13.938 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T00:43:13.938 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T00:43:13.938 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T00:43:13.938 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T00:43:13.939 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T00:43:13.939 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_items": 0, 2024-09-16T00:43:13.939 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T00:43:13.939 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T00:43:13.939 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_bytes": 0, 2024-09-16T00:43:13.939 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_items": 0, 2024-09-16T00:43:13.939 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_bytes": 0, 2024-09-16T00:43:13.939 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_items": 0, 2024-09-16T00:43:13.939 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T00:43:13.939 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_items": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_bytes": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_items": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_bytes": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_items": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_bytes": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_items": 0, 2024-09-16T00:43:13.940 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_items": 0, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_items": 0, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_bytes": 304528, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_items": 126, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_bytes": 88, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_items": 1, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_bytes": 0, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_items": 0, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_bytes": 0, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_items": 0, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_bytes": 0, 2024-09-16T00:43:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_items": 0, 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_bytes": 9624, 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_items": 182, 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_items": 0, 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_bytes": 0, 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_items": 0, 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_bytes": 239652, 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_items": 4572, 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_bytes": 0, 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_items": 0, 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_bytes": 0, 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_items": 0 2024-09-16T00:43:13.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter": { 2024-09-16T00:43:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: "op_active": 0, 2024-09-16T00:43:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: "op_laggy": 0, 2024-09-16T00:43:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: "op_send": 26760, 2024-09-16T00:43:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: "op_send_bytes": 53783833, 2024-09-16T00:43:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: "op_resend": 37, 2024-09-16T00:43:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: "op_reply": 26760, 2024-09-16T00:43:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: "op_latency": { 2024-09-16T00:43:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 26760, 2024-09-16T00:43:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 343.408767588, 2024-09-16T00:43:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.012832913 2024-09-16T00:43:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: "op_inflight": 0, 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: "oplen_avg": { 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 26760, 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 27533 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: "op": 26760, 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: "op_r": 1550, 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: "op_w": 25210, 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: "op_rmw": 0, 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: "op_pg": 0, 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_stat": 0, 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_create": 11, 2024-09-16T00:43:13.944 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_read": 408, 2024-09-16T00:43:13.945 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_write": 11412, 2024-09-16T00:43:13.945 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_writefull": 291, 2024-09-16T00:43:13.945 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_writesame": 0, 2024-09-16T00:43:13.945 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_append": 0, 2024-09-16T00:43:13.945 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_zero": 0, 2024-09-16T00:43:13.945 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_truncate": 0, 2024-09-16T00:43:13.945 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_delete": 12967, 2024-09-16T00:43:13.945 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_mapext": 0, 2024-09-16T00:43:13.945 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_sparse_read": 0, 2024-09-16T00:43:13.945 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_clonerange": 0, 2024-09-16T00:43:13.945 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_getxattr": 1142, 2024-09-16T00:43:13.945 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_setxattr": 22, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_cmpxattr": 0, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_rmxattr": 0, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_resetxattrs": 0, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_call": 0, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_watch": 0, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_notify": 0, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_src_cmpxattr": 0, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_pgls": 0, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_pgls_filter": 0, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_other": 0, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_active": 0, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_send": 0, 2024-09-16T00:43:13.946 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_resend": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_ping": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_active": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_send": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_resend": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_active": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_send": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_resend": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_active": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_send": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_resend": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "command_active": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "command_send": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "command_resend": 0, 2024-09-16T00:43:13.947 INFO:teuthology.orchestra.run.smithi060.stdout: "map_epoch": 90, 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: "map_full": 0, 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: "map_inc": 10, 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_sessions": 12, 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_session_open": 12, 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_session_close": 0, 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_laggy": 0, 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_wr": 979, 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_rd": 0, 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_del": 301 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: "oft": { 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_objs": 1, 2024-09-16T00:43:13.948 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_kv_pairs": 0, 2024-09-16T00:43:13.949 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_updates": 0, 2024-09-16T00:43:13.949 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_removes": 0 2024-09-16T00:43:13.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.949 INFO:teuthology.orchestra.run.smithi060.stdout: "osd-slow-ops": { 2024-09-16T00:43:13.949 INFO:teuthology.orchestra.run.smithi060.stdout: "slow_ops_count": 1216 2024-09-16T00:43:13.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.949 INFO:teuthology.orchestra.run.smithi060.stdout: "purge_queue": { 2024-09-16T00:43:13.949 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_ops": 0, 2024-09-16T00:43:13.949 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_ops_high_water": 16, 2024-09-16T00:43:13.949 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing": 0, 2024-09-16T00:43:13.949 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_high_water": 8, 2024-09-16T00:43:13.949 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executed_ops": 12443, 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executed": 6222, 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_item_in_journal": 0 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 382700, 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 80359822, 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 382700, 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-09-16T00:43:13.950 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 382700, 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 80359822, 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-objecter_bytes": { 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-09-16T00:43:13.951 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 26760, 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 54462023, 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 26760, 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 54462023, 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.952 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-objecter_ops": { 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 1024, 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 26760, 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 26760, 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 26760, 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 26760, 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.953 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-write_buf_throttle": { 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 3758096384, 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 6222, 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 678190, 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 6222, 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-09-16T00:43:13.954 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 374, 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 678190, 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-write_buf_throttle-0x559f9ebaa500": { 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 3758096384, 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:13.955 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 39115, 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 52343139, 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 39115, 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 11038, 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 52343139, 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:13.956 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:13.957 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-16T00:43:14.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:13 smithi033 ceph-mon[31243]: pgmap v1132: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.5 KiB/s wr, 19 op/s 2024-09-16T00:43:15.034 INFO:tasks.check_counter:Daemon mds.b mds.exported=6 2024-09-16T00:43:15.034 INFO:tasks.check_counter:Daemon mds.b mds.imported=6 2024-09-16T00:43:15.035 DEBUG:tasks.check_counter:Getting stats from e 2024-09-16T00:43:15.035 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mds.e perf dump 2024-09-16T00:43:15.289 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T00:43:15.695 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:15 smithi060 ceph-mon[35185]: pgmap v1133: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-09-16T00:43:15.803 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-16T00:43:15.804 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T00:43:15.804 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 1939, 2024-09-16T00:43:15.804 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 1941, 2024-09-16T00:43:15.804 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 339569300, 2024-09-16T00:43:15.804 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 616666, 2024-09-16T00:43:15.804 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 8, 2024-09-16T00:43:15.804 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 4, 2024-09-16T00:43:15.804 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.294941117, 2024-09-16T00:43:15.804 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.045481874, 2024-09-16T00:43:15.804 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.193337048, 2024-09-16T00:43:15.804 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.042877952, 2024-09-16T00:43:15.804 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:15.805 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 1939, 2024-09-16T00:43:15.805 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.060458439, 2024-09-16T00:43:15.805 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000031180 2024-09-16T00:43:15.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.805 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:15.805 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.805 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.805 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.805 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T00:43:15.805 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T00:43:15.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.806 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T00:43:15.806 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 2098, 2024-09-16T00:43:15.806 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 2444, 2024-09-16T00:43:15.806 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 386418620, 2024-09-16T00:43:15.806 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 1347395, 2024-09-16T00:43:15.806 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 7, 2024-09-16T00:43:15.806 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 6, 2024-09-16T00:43:15.806 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.379632172, 2024-09-16T00:43:15.806 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.054487944, 2024-09-16T00:43:15.806 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.257285586, 2024-09-16T00:43:15.806 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.050218176, 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 2442, 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.080312131, 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000032887 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 680544 2024-09-16T00:43:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.808 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T00:43:15.808 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 3055, 2024-09-16T00:43:15.808 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 3036, 2024-09-16T00:43:15.808 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 215591234, 2024-09-16T00:43:15.808 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 1101116, 2024-09-16T00:43:15.808 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 5, 2024-09-16T00:43:15.808 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 4, 2024-09-16T00:43:15.808 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.394997422, 2024-09-16T00:43:15.808 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.092774523, 2024-09-16T00:43:15.808 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.219720662, 2024-09-16T00:43:15.808 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.071629179, 2024-09-16T00:43:15.809 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:15.809 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 3036, 2024-09-16T00:43:15.809 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.132442226, 2024-09-16T00:43:15.809 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000043623 2024-09-16T00:43:15.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.809 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:15.809 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.809 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.809 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.809 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 896912, 2024-09-16T00:43:15.809 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 283856 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: "cct": { 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: "total_workers": 1, 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: "unhealthy_workers": 0 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: "finisher-MDSRank": { 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: "queue_len": 0, 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: "complete_latency": { 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 10702, 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.393255340, 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000036745 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:15.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: "finisher-PurgeQueue": { 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: "queue_len": 0, 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: "complete_latency": { 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: "request": 0, 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: "reply": 0, 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: "reply_latency": { 2024-09-16T00:43:15.811 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: "slow_reply": 0, 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: "forward": 0, 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_fetch_complete": 0, 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_fetch_keys": 0, 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_commit": 0, 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_split": 0, 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_merge": 0, 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes": 93378, 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_top": 48691, 2024-09-16T00:43:15.812 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_bottom": 44687, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_pin_tail": 0, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_pinned": 23819, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_expired": 0, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_with_caps": 0, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": 0, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "subtrees": 1, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse": 0, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_hit": 0, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_forward": 0, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_discover": 0, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_dir_fetch": 0, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_remote_ino": 0, 2024-09-16T00:43:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_lock": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "load_cent": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "q": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "exported": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "exported_inodes": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "imported": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "imported_inodes": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_dir_fetch": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_backtrace_fetch": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_peer_discover": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rfiles": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rbytes": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rsnaps": 0, 2024-09-16T00:43:15.814 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_backtrace_fetch": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_set_tag": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_backtrace_repaired": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_inotable_repaired": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dir_inodes": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dir_base_inodes": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dirfrag_rstats": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_file_inodes": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_inode_file_caps": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_revoke": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_grant": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_trunc": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-16T00:43:15.815 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_caps": 0, 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_caps_dirty": 0, 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_cap_release": 0, 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: "process_request_cap_release": 0 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cache": { 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_update": 0, 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_update_receipt": 0, 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_try_discover": 0, 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_send_discover": 0, 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_handle_discover": 0, 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays": 0, 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays_delayed": 0, 2024-09-16T00:43:15.816 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays_enqueuing": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_created": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_enqueued": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_reintegrated": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_migrated": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_processing": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_enqueued": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_prioritized": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_started": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_completed": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_quiesce_path": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_quiesce_inode": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_enqueue_scrub": 0, 2024-09-16T00:43:15.817 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_exportdir": 0, 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_flush": 0, 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_fragmentdir": 0, 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_fragstats": 0, 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_inodestats": 0 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_log": { 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "evlrg": 0, 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "evadd": 0, 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "evex": 0, 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "evtrm": 0, 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "ev": 446958, 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "evexg": 0, 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "evexd": 0, 2024-09-16T00:43:15.818 INFO:teuthology.orchestra.run.smithi060.stdout: "segadd": 0, 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "segex": 0, 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "segtrm": 0, 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "seg": 135, 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "segmjr": 38, 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "segexg": 0, 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "segexd": 0, 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "expos": 702333088, 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "wrpos": 943050630, 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "rdpos": 943048346, 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "jlat": { 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.819 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: "replayed": 446958 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_mem": { 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 115, 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: "ino+": 93413, 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: "ino-": 93298, 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: "dir": 30, 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: "dir+": 629, 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: "dir-": 599, 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: "dn": 93378, 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: "dn+": 94625, 2024-09-16T00:43:15.820 INFO:teuthology.orchestra.run.smithi060.stdout: "dn-": 1247, 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: "cap": 0, 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: "cap+": 0, 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: "cap-": 0, 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: "rss": 133072, 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: "heap": 215328 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_server": { 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: "dispatch_client_request": 0, 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: "dispatch_server_request": 0, 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_request": 0, 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_session": 0, 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_peer_request": 0, 2024-09-16T00:43:15.821 INFO:teuthology.orchestra.run.smithi060.stdout: "req_create_latency": { 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getattr_latency": { 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getfilelock_latency": { 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.822 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: "req_link_latency": { 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookup_latency": { 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookuphash_latency": { 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.823 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupino_latency": { 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupname_latency": { 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.824 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupparent_latency": { 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupsnap_latency": { 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lssnap_latency": { 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.825 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.826 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mkdir_latency": { 2024-09-16T00:43:15.826 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.826 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.826 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.826 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mknod_latency": { 2024-09-16T00:43:15.826 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.826 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.826 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.826 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mksnap_latency": { 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: "req_open_latency": { 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: "req_readdir_latency": { 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.827 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rename_latency": { 2024-09-16T00:43:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: "req_renamesnap_latency": { 2024-09-16T00:43:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: "req_snapdiff_latency": { 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmdir_latency": { 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmsnap_latency": { 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmxattr_latency": { 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setattr_latency": { 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setdirlayout_latency": { 2024-09-16T00:43:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setfilelock_latency": { 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setlayout_latency": { 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setxattr_latency": { 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: "req_symlink_latency": { 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: "req_unlink_latency": { 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.833 INFO:teuthology.orchestra.run.smithi060.stdout: "cap_revoke_eviction": 0, 2024-09-16T00:43:15.833 INFO:teuthology.orchestra.run.smithi060.stdout: "cap_acquisition_throttle": 0, 2024-09-16T00:43:15.833 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getvxattr_latency": { 2024-09-16T00:43:15.833 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.833 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.833 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.833 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:15.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.834 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_sessions": { 2024-09-16T00:43:15.834 INFO:teuthology.orchestra.run.smithi060.stdout: "session_count": 1, 2024-09-16T00:43:15.834 INFO:teuthology.orchestra.run.smithi060.stdout: "session_add": 3, 2024-09-16T00:43:15.834 INFO:teuthology.orchestra.run.smithi060.stdout: "session_remove": 2, 2024-09-16T00:43:15.834 INFO:teuthology.orchestra.run.smithi060.stdout: "sessions_open": 2, 2024-09-16T00:43:15.834 INFO:teuthology.orchestra.run.smithi060.stdout: "sessions_stale": 0, 2024-09-16T00:43:15.834 INFO:teuthology.orchestra.run.smithi060.stdout: "total_load": 0, 2024-09-16T00:43:15.834 INFO:teuthology.orchestra.run.smithi060.stdout: "average_load": 0, 2024-09-16T00:43:15.834 INFO:teuthology.orchestra.run.smithi060.stdout: "avg_session_uptime": 4094, 2024-09-16T00:43:15.834 INFO:teuthology.orchestra.run.smithi060.stdout: "mdthresh_evicted": 0 2024-09-16T00:43:15.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.834 INFO:teuthology.orchestra.run.smithi060.stdout: "mempool": { 2024-09-16T00:43:15.835 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_bytes": 0, 2024-09-16T00:43:15.835 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_items": 0, 2024-09-16T00:43:15.835 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T00:43:15.835 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_items": 0, 2024-09-16T00:43:15.835 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T00:43:15.835 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_items": 0, 2024-09-16T00:43:15.835 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T00:43:15.835 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T00:43:15.835 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T00:43:15.835 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T00:43:15.835 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T00:43:15.835 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_items": 0, 2024-09-16T00:43:15.836 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T00:43:15.836 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T00:43:15.836 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_bytes": 0, 2024-09-16T00:43:15.836 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_items": 0, 2024-09-16T00:43:15.836 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_bytes": 0, 2024-09-16T00:43:15.836 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_items": 0, 2024-09-16T00:43:15.836 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T00:43:15.836 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T00:43:15.836 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T00:43:15.836 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T00:43:15.836 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T00:43:15.836 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_items": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_bytes": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_items": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_bytes": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_items": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_bytes": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_items": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_items": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_items": 0, 2024-09-16T00:43:15.837 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_bytes": 244773, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_items": 184, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_bytes": 0, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_items": 0, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_bytes": 0, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_items": 0, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_bytes": 0, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_items": 0, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_bytes": 0, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_items": 0, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_bytes": 9624, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_items": 182, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T00:43:15.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_items": 0, 2024-09-16T00:43:15.839 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_bytes": 0, 2024-09-16T00:43:15.839 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_items": 0, 2024-09-16T00:43:15.839 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_bytes": 66060637, 2024-09-16T00:43:15.839 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_items": 386842, 2024-09-16T00:43:15.839 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_bytes": 0, 2024-09-16T00:43:15.839 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_items": 0, 2024-09-16T00:43:15.839 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_bytes": 0, 2024-09-16T00:43:15.839 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_items": 0 2024-09-16T00:43:15.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.839 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter": { 2024-09-16T00:43:15.839 INFO:teuthology.orchestra.run.smithi060.stdout: "op_active": 0, 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: "op_laggy": 0, 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: "op_send": 6606, 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: "op_send_bytes": 21, 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: "op_resend": 3, 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: "op_reply": 6606, 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: "op_latency": { 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 6606, 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 21.495797098, 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.003253980 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: "op_inflight": 0, 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: "oplen_avg": { 2024-09-16T00:43:15.840 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 6606, 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 6607 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: "op": 6606, 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: "op_r": 6606, 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: "op_w": 0, 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: "op_rmw": 0, 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: "op_pg": 0, 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_stat": 3756, 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_create": 0, 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_read": 2848, 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_write": 0, 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_writefull": 0, 2024-09-16T00:43:15.841 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_writesame": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_append": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_zero": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_truncate": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_delete": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_mapext": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_sparse_read": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_clonerange": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_getxattr": 1, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_setxattr": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_cmpxattr": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_rmxattr": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_resetxattrs": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_call": 0, 2024-09-16T00:43:15.842 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_watch": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_notify": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_src_cmpxattr": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_pgls": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_pgls_filter": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_other": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_active": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_send": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_resend": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_ping": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_active": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_send": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_resend": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_active": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_send": 0, 2024-09-16T00:43:15.843 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_resend": 0, 2024-09-16T00:43:15.844 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_active": 0, 2024-09-16T00:43:15.844 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_send": 0, 2024-09-16T00:43:15.844 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_resend": 0, 2024-09-16T00:43:15.844 INFO:teuthology.orchestra.run.smithi060.stdout: "command_active": 0, 2024-09-16T00:43:15.844 INFO:teuthology.orchestra.run.smithi060.stdout: "command_send": 0, 2024-09-16T00:43:15.844 INFO:teuthology.orchestra.run.smithi060.stdout: "command_resend": 0, 2024-09-16T00:43:15.844 INFO:teuthology.orchestra.run.smithi060.stdout: "map_epoch": 90, 2024-09-16T00:43:15.844 INFO:teuthology.orchestra.run.smithi060.stdout: "map_full": 0, 2024-09-16T00:43:15.844 INFO:teuthology.orchestra.run.smithi060.stdout: "map_inc": 4, 2024-09-16T00:43:15.844 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_sessions": 11, 2024-09-16T00:43:15.844 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_session_open": 11, 2024-09-16T00:43:15.844 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_session_close": 0, 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_laggy": 0, 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_wr": 0, 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_rd": 2, 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_del": 0 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: "oft": { 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_objs": 0, 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_kv_pairs": 0, 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_updates": 0, 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_removes": 0 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: "osd-slow-ops": { 2024-09-16T00:43:15.845 INFO:teuthology.orchestra.run.smithi060.stdout: "slow_ops_count": 0 2024-09-16T00:43:15.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.846 INFO:teuthology.orchestra.run.smithi060.stdout: "purge_queue": { 2024-09-16T00:43:15.846 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_ops": 0, 2024-09-16T00:43:15.846 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_ops_high_water": 0, 2024-09-16T00:43:15.846 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing": 0, 2024-09-16T00:43:15.846 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_high_water": 0, 2024-09-16T00:43:15.846 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executed_ops": 0, 2024-09-16T00:43:15.846 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executed": 0, 2024-09-16T00:43:15.846 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_item_in_journal": 0 2024-09-16T00:43:15.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.846 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T00:43:15.846 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 7088, 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 940913456, 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 7088, 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 7088, 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 940913456, 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.847 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-objecter_bytes": { 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 6606, 2024-09-16T00:43:15.848 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 938863384, 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 6606, 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 938863384, 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-objecter_ops": { 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 1024, 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:15.849 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 6606, 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 6606, 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 6606, 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 6606, 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:15.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-write_buf_throttle": { 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 3758096384, 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 0, 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 0, 2024-09-16T00:43:15.851 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-write_buf_throttle-0x5595541ea6e0": { 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 3758096384, 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:15.852 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:15.853 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-09-16T00:43:15.853 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-09-16T00:43:15.853 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 0, 2024-09-16T00:43:15.853 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 0, 2024-09-16T00:43:15.853 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:15.853 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:15.853 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:15.853 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:15.853 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:15.853 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:15.853 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-16T00:43:15.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:15 smithi083 ceph-mon[35242]: pgmap v1133: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-09-16T00:43:16.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:15 smithi033 ceph-mon[31243]: pgmap v1133: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-09-16T00:43:16.654 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-09-16T00:43:16.654 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-09-16T00:43:16.655 DEBUG:tasks.check_counter:Getting stats from h 2024-09-16T00:43:16.655 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mds.h perf dump 2024-09-16T00:43:16.907 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T00:43:17.501 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-16T00:43:17.502 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T00:43:17.502 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 3853, 2024-09-16T00:43:17.502 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 3853, 2024-09-16T00:43:17.502 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 91678149, 2024-09-16T00:43:17.502 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 1224437, 2024-09-16T00:43:17.502 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 8, 2024-09-16T00:43:17.502 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 4, 2024-09-16T00:43:17.502 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.320014631, 2024-09-16T00:43:17.502 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.087827399, 2024-09-16T00:43:17.502 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.119678227, 2024-09-16T00:43:17.502 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.090316488, 2024-09-16T00:43:17.502 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:17.503 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 3852, 2024-09-16T00:43:17.503 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.126473207, 2024-09-16T00:43:17.503 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000032833 2024-09-16T00:43:17.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.503 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:17.503 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.503 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.503 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.503 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-09-16T00:43:17.503 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 448 2024-09-16T00:43:17.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.504 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T00:43:17.504 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 1153, 2024-09-16T00:43:17.504 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 1136, 2024-09-16T00:43:17.504 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 80760165, 2024-09-16T00:43:17.504 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 496844, 2024-09-16T00:43:17.504 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 6, 2024-09-16T00:43:17.504 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 5, 2024-09-16T00:43:17.504 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.146998698, 2024-09-16T00:43:17.504 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.041328347, 2024-09-16T00:43:17.504 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.094910118, 2024-09-16T00:43:17.504 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.015818172, 2024-09-16T00:43:17.504 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:17.505 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 1135, 2024-09-16T00:43:17.505 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.043569182, 2024-09-16T00:43:17.505 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000038386 2024-09-16T00:43:17.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.505 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:17.505 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.505 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.505 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.505 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 893616, 2024-09-16T00:43:17.505 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 282768 2024-09-16T00:43:17.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 864, 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 1209, 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 92495716, 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 1042820, 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 8, 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 5, 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.129072196, 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.029219051, 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.071445050, 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.020448026, 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:17.506 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 1207, 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.044648770, 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000036991 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 768704 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: "cct": { 2024-09-16T00:43:17.507 INFO:teuthology.orchestra.run.smithi060.stdout: "total_workers": 1, 2024-09-16T00:43:17.508 INFO:teuthology.orchestra.run.smithi060.stdout: "unhealthy_workers": 0 2024-09-16T00:43:17.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.508 INFO:teuthology.orchestra.run.smithi060.stdout: "finisher-MDSRank": { 2024-09-16T00:43:17.508 INFO:teuthology.orchestra.run.smithi060.stdout: "queue_len": 0, 2024-09-16T00:43:17.508 INFO:teuthology.orchestra.run.smithi060.stdout: "complete_latency": { 2024-09-16T00:43:17.508 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 9108, 2024-09-16T00:43:17.508 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.191347083, 2024-09-16T00:43:17.508 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000021008 2024-09-16T00:43:17.508 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:17.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.508 INFO:teuthology.orchestra.run.smithi060.stdout: "finisher-PurgeQueue": { 2024-09-16T00:43:17.508 INFO:teuthology.orchestra.run.smithi060.stdout: "queue_len": 0, 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: "complete_latency": { 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: "request": 0, 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: "reply": 0, 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: "reply_latency": { 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.509 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: "slow_reply": 0, 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: "forward": 0, 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_fetch_complete": 0, 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_fetch_keys": 0, 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_commit": 0, 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_split": 0, 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_merge": 0, 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes": 26449, 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_top": 1542, 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_bottom": 24907, 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_pin_tail": 0, 2024-09-16T00:43:17.510 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_pinned": 24245, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_expired": 0, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_with_caps": 0, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": 0, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "subtrees": 1, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse": 0, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_hit": 0, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_forward": 0, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_discover": 0, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_dir_fetch": 0, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_remote_ino": 0, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_lock": 0, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "load_cent": 0, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "q": 0, 2024-09-16T00:43:17.511 INFO:teuthology.orchestra.run.smithi060.stdout: "exported": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "exported_inodes": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "imported": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "imported_inodes": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_dir_fetch": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_backtrace_fetch": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_peer_discover": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rfiles": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rbytes": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rsnaps": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_backtrace_fetch": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_set_tag": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_backtrace_repaired": 0, 2024-09-16T00:43:17.512 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_inotable_repaired": 0, 2024-09-16T00:43:17.513 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dir_inodes": 0, 2024-09-16T00:43:17.513 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dir_base_inodes": 0, 2024-09-16T00:43:17.513 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dirfrag_rstats": 0, 2024-09-16T00:43:17.513 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_file_inodes": 0, 2024-09-16T00:43:17.513 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_inode_file_caps": 0, 2024-09-16T00:43:17.513 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_revoke": 0, 2024-09-16T00:43:17.513 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_grant": 0, 2024-09-16T00:43:17.513 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_trunc": 0, 2024-09-16T00:43:17.513 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-16T00:43:17.513 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-16T00:43:17.513 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_caps": 0, 2024-09-16T00:43:17.513 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_caps_dirty": 0, 2024-09-16T00:43:17.514 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_cap_release": 0, 2024-09-16T00:43:17.514 INFO:teuthology.orchestra.run.smithi060.stdout: "process_request_cap_release": 0 2024-09-16T00:43:17.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.514 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cache": { 2024-09-16T00:43:17.514 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_update": 0, 2024-09-16T00:43:17.514 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_update_receipt": 0, 2024-09-16T00:43:17.514 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_try_discover": 0, 2024-09-16T00:43:17.514 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_send_discover": 0, 2024-09-16T00:43:17.514 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_handle_discover": 0, 2024-09-16T00:43:17.514 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays": 0, 2024-09-16T00:43:17.515 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays_delayed": 0, 2024-09-16T00:43:17.515 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays_enqueuing": 0, 2024-09-16T00:43:17.515 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_created": 0, 2024-09-16T00:43:17.515 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_enqueued": 0, 2024-09-16T00:43:17.515 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_reintegrated": 0, 2024-09-16T00:43:17.515 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_migrated": 0, 2024-09-16T00:43:17.515 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_processing": 0, 2024-09-16T00:43:17.515 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_enqueued": 0, 2024-09-16T00:43:17.515 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_prioritized": 0, 2024-09-16T00:43:17.515 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_started": 0, 2024-09-16T00:43:17.515 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_completed": 0, 2024-09-16T00:43:17.515 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_quiesce_path": 0, 2024-09-16T00:43:17.516 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_quiesce_inode": 0, 2024-09-16T00:43:17.516 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_enqueue_scrub": 0, 2024-09-16T00:43:17.516 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_exportdir": 0, 2024-09-16T00:43:17.516 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_flush": 0, 2024-09-16T00:43:17.516 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_fragmentdir": 0, 2024-09-16T00:43:17.516 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_fragstats": 0, 2024-09-16T00:43:17.516 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_inodestats": 0 2024-09-16T00:43:17.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_log": { 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "evlrg": 0, 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "evadd": 0, 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "evex": 0, 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "evtrm": 0, 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "ev": 130678, 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "evexg": 0, 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "evexd": 0, 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "segadd": 0, 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "segex": 0, 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "segtrm": 0, 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "seg": 141, 2024-09-16T00:43:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: "segmjr": 12, 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: "segexg": 0, 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: "segexd": 0, 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: "expos": 29679705, 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: "wrpos": 266722233, 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: "rdpos": 266721634, 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: "jlat": { 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: "replayed": 130678 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.518 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_mem": { 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 42, 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "ino+": 26410, 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "ino-": 26368, 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "dir": 13, 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "dir+": 58, 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "dir-": 45, 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "dn": 26449, 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "dn+": 26723, 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "dn-": 274, 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "cap": 0, 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "cap+": 0, 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "cap-": 0, 2024-09-16T00:43:17.519 INFO:teuthology.orchestra.run.smithi060.stdout: "rss": 74668, 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: "heap": 215328 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_server": { 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: "dispatch_client_request": 0, 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: "dispatch_server_request": 0, 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_request": 0, 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_session": 0, 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_peer_request": 0, 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: "req_create_latency": { 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getattr_latency": { 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getfilelock_latency": { 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: "req_link_latency": { 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.521 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookup_latency": { 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookuphash_latency": { 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.522 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupino_latency": { 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupname_latency": { 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupparent_latency": { 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.524 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupsnap_latency": { 2024-09-16T00:43:17.524 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.524 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.524 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.524 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lssnap_latency": { 2024-09-16T00:43:17.524 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.524 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.524 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.525 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mkdir_latency": { 2024-09-16T00:43:17.525 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.525 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.525 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.525 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mknod_latency": { 2024-09-16T00:43:17.525 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.525 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.526 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.526 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mksnap_latency": { 2024-09-16T00:43:17.526 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.526 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.526 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.526 INFO:teuthology.orchestra.run.smithi060.stdout: "req_open_latency": { 2024-09-16T00:43:17.526 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.526 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.526 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.527 INFO:teuthology.orchestra.run.smithi060.stdout: "req_readdir_latency": { 2024-09-16T00:43:17.527 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.527 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.527 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.527 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rename_latency": { 2024-09-16T00:43:17.527 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.527 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.527 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.527 INFO:teuthology.orchestra.run.smithi060.stdout: "req_renamesnap_latency": { 2024-09-16T00:43:17.527 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: "req_snapdiff_latency": { 2024-09-16T00:43:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmdir_latency": { 2024-09-16T00:43:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmsnap_latency": { 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmxattr_latency": { 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setattr_latency": { 2024-09-16T00:43:17.529 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setdirlayout_latency": { 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setfilelock_latency": { 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.531 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setlayout_latency": { 2024-09-16T00:43:17.531 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.531 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.531 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.531 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setxattr_latency": { 2024-09-16T00:43:17.531 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.531 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.531 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.531 INFO:teuthology.orchestra.run.smithi060.stdout: "req_symlink_latency": { 2024-09-16T00:43:17.531 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: "req_unlink_latency": { 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: "cap_revoke_eviction": 0, 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: "cap_acquisition_throttle": 0, 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getvxattr_latency": { 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.532 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_sessions": { 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: "session_count": 1, 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: "session_add": 1, 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: "session_remove": 0, 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: "sessions_open": 1, 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: "sessions_stale": 0, 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: "total_load": 0, 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: "average_load": 0, 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: "avg_session_uptime": 2821, 2024-09-16T00:43:17.533 INFO:teuthology.orchestra.run.smithi060.stdout: "mdthresh_evicted": 0 2024-09-16T00:43:17.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.534 INFO:teuthology.orchestra.run.smithi060.stdout: "mempool": { 2024-09-16T00:43:17.534 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_bytes": 0, 2024-09-16T00:43:17.534 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_items": 0, 2024-09-16T00:43:17.534 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T00:43:17.534 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_items": 0, 2024-09-16T00:43:17.534 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T00:43:17.534 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_items": 0, 2024-09-16T00:43:17.534 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T00:43:17.534 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T00:43:17.534 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T00:43:17.534 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T00:43:17.535 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T00:43:17.535 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_items": 0, 2024-09-16T00:43:17.535 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T00:43:17.535 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T00:43:17.535 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_bytes": 0, 2024-09-16T00:43:17.535 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_items": 0, 2024-09-16T00:43:17.535 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_bytes": 0, 2024-09-16T00:43:17.535 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_items": 0, 2024-09-16T00:43:17.535 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T00:43:17.535 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T00:43:17.535 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T00:43:17.535 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T00:43:17.536 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T00:43:17.536 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_items": 0, 2024-09-16T00:43:17.536 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_bytes": 0, 2024-09-16T00:43:17.536 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_items": 0, 2024-09-16T00:43:17.536 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T00:43:17.536 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T00:43:17.536 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_bytes": 0, 2024-09-16T00:43:17.536 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_items": 0, 2024-09-16T00:43:17.536 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_bytes": 0, 2024-09-16T00:43:17.536 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_items": 0, 2024-09-16T00:43:17.536 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T00:43:17.536 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_items": 0, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_items": 0, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_bytes": 241328, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_items": 110, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_bytes": 0, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_items": 0, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_bytes": 0, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_items": 0, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_bytes": 0, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_items": 0, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_bytes": 0, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_items": 0, 2024-09-16T00:43:17.537 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_bytes": 9624, 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_items": 182, 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_items": 0, 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_bytes": 0, 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_items": 0, 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_bytes": 18738103, 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_items": 109694, 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_bytes": 0, 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_items": 0, 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_bytes": 0, 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_items": 0 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.538 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter": { 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: "op_active": 0, 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: "op_laggy": 0, 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: "op_send": 5384, 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: "op_send_bytes": 21, 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: "op_resend": 3, 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: "op_reply": 5384, 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: "op_latency": { 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 5384, 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 11.541891232, 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.002143739 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: "op_inflight": 0, 2024-09-16T00:43:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: "oplen_avg": { 2024-09-16T00:43:17.540 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 5384, 2024-09-16T00:43:17.540 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 5385 2024-09-16T00:43:17.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.540 INFO:teuthology.orchestra.run.smithi060.stdout: "op": 5384, 2024-09-16T00:43:17.540 INFO:teuthology.orchestra.run.smithi060.stdout: "op_r": 5384, 2024-09-16T00:43:17.540 INFO:teuthology.orchestra.run.smithi060.stdout: "op_w": 0, 2024-09-16T00:43:17.540 INFO:teuthology.orchestra.run.smithi060.stdout: "op_rmw": 0, 2024-09-16T00:43:17.540 INFO:teuthology.orchestra.run.smithi060.stdout: "op_pg": 0, 2024-09-16T00:43:17.540 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_stat": 3166, 2024-09-16T00:43:17.540 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_create": 0, 2024-09-16T00:43:17.540 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_read": 2216, 2024-09-16T00:43:17.541 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_write": 0, 2024-09-16T00:43:17.541 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_writefull": 0, 2024-09-16T00:43:17.541 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_writesame": 0, 2024-09-16T00:43:17.541 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_append": 0, 2024-09-16T00:43:17.541 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_zero": 0, 2024-09-16T00:43:17.541 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_truncate": 0, 2024-09-16T00:43:17.541 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_delete": 0, 2024-09-16T00:43:17.541 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_mapext": 0, 2024-09-16T00:43:17.541 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_sparse_read": 0, 2024-09-16T00:43:17.541 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_clonerange": 0, 2024-09-16T00:43:17.541 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_getxattr": 1, 2024-09-16T00:43:17.541 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_setxattr": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_cmpxattr": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_rmxattr": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_resetxattrs": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_call": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_watch": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_notify": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_src_cmpxattr": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_pgls": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_pgls_filter": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_other": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_active": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_send": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_resend": 0, 2024-09-16T00:43:17.542 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_ping": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_active": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_send": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_resend": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_active": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_send": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_resend": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_active": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_send": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_resend": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "command_active": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "command_send": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "command_resend": 0, 2024-09-16T00:43:17.543 INFO:teuthology.orchestra.run.smithi060.stdout: "map_epoch": 90, 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: "map_full": 0, 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: "map_inc": 4, 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_sessions": 11, 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_session_open": 12, 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_session_close": 1, 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_laggy": 0, 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_wr": 0, 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_rd": 2, 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_del": 0 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: "oft": { 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_objs": 0, 2024-09-16T00:43:17.544 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_kv_pairs": 0, 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_updates": 0, 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_removes": 0 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: "osd-slow-ops": { 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: "slow_ops_count": 0 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: "purge_queue": { 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_ops": 0, 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_ops_high_water": 0, 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing": 0, 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_high_water": 0, 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executed_ops": 0, 2024-09-16T00:43:17.545 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executed": 0, 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_item_in_journal": 0 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 5866, 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 264378321, 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 5866, 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-09-16T00:43:17.546 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 5866, 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 264378321, 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-objecter_bytes": { 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-09-16T00:43:17.547 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:17.548 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:17.548 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 5384, 2024-09-16T00:43:17.548 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 262531588, 2024-09-16T00:43:17.548 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 5384, 2024-09-16T00:43:17.548 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 262531588, 2024-09-16T00:43:17.548 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:17.548 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.548 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.548 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.548 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:17.548 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.548 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-objecter_ops": { 2024-09-16T00:43:17.549 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:17.549 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 1024, 2024-09-16T00:43:17.549 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:17.549 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-09-16T00:43:17.549 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-09-16T00:43:17.549 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:17.549 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:17.549 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 5384, 2024-09-16T00:43:17.549 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 5384, 2024-09-16T00:43:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 5384, 2024-09-16T00:43:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 5384, 2024-09-16T00:43:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-write_buf_throttle": { 2024-09-16T00:43:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 3758096384, 2024-09-16T00:43:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 0, 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 0, 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.553 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-write_buf_throttle-0x55b01a71c6e0": { 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 3758096384, 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 0, 2024-09-16T00:43:17.554 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 0, 2024-09-16T00:43:17.555 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:17.555 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:17.555 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:17.555 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:17.555 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:17.555 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:17.555 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-16T00:43:17.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:17 smithi060 ceph-mon[35185]: pgmap v1134: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 5 op/s 2024-09-16T00:43:17.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:17 smithi083 ceph-mon[35242]: pgmap v1134: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 5 op/s 2024-09-16T00:43:18.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:17 smithi033 ceph-mon[31243]: pgmap v1134: 129 pgs: 129 active+clean; 536 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 5 op/s 2024-09-16T00:43:18.565 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-09-16T00:43:18.566 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-09-16T00:43:18.566 DEBUG:tasks.check_counter:Getting stats from k 2024-09-16T00:43:18.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mds.k perf dump 2024-09-16T00:43:18.820 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T00:43:19.312 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-16T00:43:19.312 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T00:43:19.312 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 0, 2024-09-16T00:43:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 2, 2024-09-16T00:43:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 0, 2024-09-16T00:43:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 0, 2024-09-16T00:43:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 4, 2024-09-16T00:43:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 0, 2024-09-16T00:43:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.001032636, 2024-09-16T00:43:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.000000000, 2024-09-16T00:43:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.000151921, 2024-09-16T00:43:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T00:43:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 2, 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 351, 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 320, 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 219184, 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 4, 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 2, 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.018327409, 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.013697859, 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.001366071, 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 349, 2024-09-16T00:43:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.016463374, 2024-09-16T00:43:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000047172 2024-09-16T00:43:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-16T00:43:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 219184 2024-09-16T00:43:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T00:43:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 467, 2024-09-16T00:43:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 460, 2024-09-16T00:43:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 829248, 2024-09-16T00:43:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 284240, 2024-09-16T00:43:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 2, 2024-09-16T00:43:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 1, 2024-09-16T00:43:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.060732465, 2024-09-16T00:43:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.026994802, 2024-09-16T00:43:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.041821778, 2024-09-16T00:43:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 460, 2024-09-16T00:43:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.025280230, 2024-09-16T00:43:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000054957 2024-09-16T00:43:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 829248, 2024-09-16T00:43:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 284240 2024-09-16T00:43:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: "cct": { 2024-09-16T00:43:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: "total_workers": 0, 2024-09-16T00:43:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: "unhealthy_workers": 0 2024-09-16T00:43:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: "mempool": { 2024-09-16T00:43:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_bytes": 0, 2024-09-16T00:43:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_items": 0, 2024-09-16T00:43:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T00:43:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_items": 0, 2024-09-16T00:43:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T00:43:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_items": 0, 2024-09-16T00:43:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T00:43:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T00:43:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T00:43:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T00:43:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T00:43:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_items": 0, 2024-09-16T00:43:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T00:43:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T00:43:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_bytes": 0, 2024-09-16T00:43:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_items": 0, 2024-09-16T00:43:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_bytes": 0, 2024-09-16T00:43:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_items": 0, 2024-09-16T00:43:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T00:43:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T00:43:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T00:43:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T00:43:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T00:43:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_items": 0, 2024-09-16T00:43:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_bytes": 0, 2024-09-16T00:43:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_items": 0, 2024-09-16T00:43:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T00:43:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T00:43:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_bytes": 0, 2024-09-16T00:43:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_items": 0, 2024-09-16T00:43:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_bytes": 0, 2024-09-16T00:43:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_items": 0, 2024-09-16T00:43:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T00:43:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_items": 0, 2024-09-16T00:43:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T00:43:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_items": 0, 2024-09-16T00:43:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_bytes": 148682, 2024-09-16T00:43:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_items": 29, 2024-09-16T00:43:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_bytes": 0, 2024-09-16T00:43:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_items": 0, 2024-09-16T00:43:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_bytes": 0, 2024-09-16T00:43:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_items": 0, 2024-09-16T00:43:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_bytes": 0, 2024-09-16T00:43:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_items": 0, 2024-09-16T00:43:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_bytes": 0, 2024-09-16T00:43:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_items": 0, 2024-09-16T00:43:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_bytes": 0, 2024-09-16T00:43:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_items": 0, 2024-09-16T00:43:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T00:43:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_items": 0, 2024-09-16T00:43:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_bytes": 0, 2024-09-16T00:43:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_items": 0, 2024-09-16T00:43:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_bytes": 928, 2024-09-16T00:43:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_items": 2, 2024-09-16T00:43:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_bytes": 0, 2024-09-16T00:43:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_items": 0, 2024-09-16T00:43:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_bytes": 0, 2024-09-16T00:43:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_items": 0 2024-09-16T00:43:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-16T00:43:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T00:43:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-09-16T00:43:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-09-16T00:43:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-09-16T00:43:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 465, 2024-09-16T00:43:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 760732, 2024-09-16T00:43:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 465, 2024-09-16T00:43:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-09-16T00:43:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-09-16T00:43:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 465, 2024-09-16T00:43:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 760732, 2024-09-16T00:43:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-09-16T00:43:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-09-16T00:43:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-09-16T00:43:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-09-16T00:43:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-16T00:43:19.326 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-16T00:43:19.916 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:19 smithi060 ceph-mon[35185]: pgmap v1135: 129 pgs: 129 active+clean; 537 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 121 KiB/s wr, 6 op/s 2024-09-16T00:43:19.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:19 smithi083 ceph-mon[35242]: pgmap v1135: 129 pgs: 129 active+clean; 537 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 121 KiB/s wr, 6 op/s 2024-09-16T00:43:19.947 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-09-16T00:43:19.947 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-09-16T00:43:19.947 DEBUG:tasks.check_counter:Getting stats from c 2024-09-16T00:43:19.947 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mds.c perf dump 2024-09-16T00:43:20.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:19 smithi033 ceph-mon[31243]: pgmap v1135: 129 pgs: 129 active+clean; 537 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 121 KiB/s wr, 6 op/s 2024-09-16T00:43:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:43:20.848 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-16T00:43:20.848 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T00:43:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 253981, 2024-09-16T00:43:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 291988, 2024-09-16T00:43:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 60182428, 2024-09-16T00:43:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 368012755, 2024-09-16T00:43:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 77, 2024-09-16T00:43:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 9, 2024-09-16T00:43:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 21.724165201, 2024-09-16T00:43:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 10.360138553, 2024-09-16T00:43:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 8.660007929, 2024-09-16T00:43:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 3.503745869, 2024-09-16T00:43:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 291986, 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 26.194043133, 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000089709 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 347229, 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.062640482, 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000180 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 920800, 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 319440 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T00:43:20.850 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 912793, 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 1380020, 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 318940980, 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 1431295183, 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 10, 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 6, 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 66.208007712, 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 30.116142348, 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 40.405521882, 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 5.186373881, 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 1380018, 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 136.330968348, 2024-09-16T00:43:20.851 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000098789 2024-09-16T00:43:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 1879723, 2024-09-16T00:43:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.333915368, 2024-09-16T00:43:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000177 2024-09-16T00:43:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 291287600, 2024-09-16T00:43:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 1130481792 2024-09-16T00:43:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T00:43:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 256905, 2024-09-16T00:43:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 345575, 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 66547149, 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 514743410, 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 14, 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 5, 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 20.381879044, 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 7.620129845, 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 7.549460576, 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 5.078923324, 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 345575, 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 31.054891809, 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000089864 2024-09-16T00:43:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 336911, 2024-09-16T00:43:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.060039470, 2024-09-16T00:43:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000178 2024-09-16T00:43:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T00:43:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T00:43:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: "cct": { 2024-09-16T00:43:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: "total_workers": 1, 2024-09-16T00:43:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: "unhealthy_workers": 0 2024-09-16T00:43:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: "finisher-MDSRank": { 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: "queue_len": 0, 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: "complete_latency": { 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 220927, 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 341.069576444, 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.001543811 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: "finisher-PurgeQueue": { 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: "queue_len": 0, 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: "complete_latency": { 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 156870, 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 38.858524518, 2024-09-16T00:43:20.855 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000247711 2024-09-16T00:43:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": { 2024-09-16T00:43:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: "request": 581654, 2024-09-16T00:43:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: "reply": 563974, 2024-09-16T00:43:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: "reply_latency": { 2024-09-16T00:43:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 563974, 2024-09-16T00:43:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 5343.109184352, 2024-09-16T00:43:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.009474034 2024-09-16T00:43:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: "slow_reply": 0, 2024-09-16T00:43:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: "forward": 17680, 2024-09-16T00:43:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_fetch_complete": 0, 2024-09-16T00:43:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_fetch_keys": 0, 2024-09-16T00:43:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_commit": 603, 2024-09-16T00:43:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_split": 54, 2024-09-16T00:43:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_merge": 64, 2024-09-16T00:43:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes": 6869, 2024-09-16T00:43:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_top": 4716, 2024-09-16T00:43:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_bottom": 2051, 2024-09-16T00:43:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_pin_tail": 102, 2024-09-16T00:43:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_pinned": 126, 2024-09-16T00:43:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_expired": 342, 2024-09-16T00:43:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_with_caps": 84, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": 84, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "subtrees": 5, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse": 863023, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_hit": 498591, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_forward": 17678, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_discover": 4, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_dir_fetch": 0, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_remote_ino": 0, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_lock": 7235, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "load_cent": 0, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "q": 0, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "exported": 19, 2024-09-16T00:43:20.858 INFO:teuthology.orchestra.run.smithi083.stdout: "exported_inodes": 232, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "imported": 9, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "imported_inodes": 192, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_dir_fetch": 0, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_backtrace_fetch": 11491, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_peer_discover": 0, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rfiles": 702, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rbytes": 214210665, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rsnaps": 0, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_backtrace_fetch": 11491, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_set_tag": 0, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_backtrace_repaired": 0, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_inotable_repaired": 0, 2024-09-16T00:43:20.859 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dir_inodes": 1696, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dir_base_inodes": 0, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dirfrag_rstats": 1696, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_file_inodes": 9795, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_inode_file_caps": 111, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_revoke": 35681, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_grant": 370116, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_trunc": 0, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_caps": 176852, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_caps_dirty": 74085, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_cap_release": 31168, 2024-09-16T00:43:20.860 INFO:teuthology.orchestra.run.smithi083.stdout: "process_request_cap_release": 356253 2024-09-16T00:43:20.861 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.861 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache": { 2024-09-16T00:43:20.861 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_update": 22, 2024-09-16T00:43:20.861 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_update_receipt": 9, 2024-09-16T00:43:20.861 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_try_discover": 6, 2024-09-16T00:43:20.861 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_send_discover": 8, 2024-09-16T00:43:20.861 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_handle_discover": 81, 2024-09-16T00:43:20.861 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays": 102, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays_delayed": 0, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays_enqueuing": 0, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_created": 93224, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_enqueued": 93122, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_reintegrated": 0, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_migrated": 0, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_processing": 0, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_enqueued": 0, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_prioritized": 0, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_started": 0, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_completed": 0, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_quiesce_path": 22, 2024-09-16T00:43:20.862 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_quiesce_inode": 7533, 2024-09-16T00:43:20.863 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_enqueue_scrub": 0, 2024-09-16T00:43:20.863 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_exportdir": 19, 2024-09-16T00:43:20.863 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_flush": 0, 2024-09-16T00:43:20.863 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_fragmentdir": 118, 2024-09-16T00:43:20.863 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_fragstats": 0, 2024-09-16T00:43:20.863 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_inodestats": 0 2024-09-16T00:43:20.863 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.863 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log": { 2024-09-16T00:43:20.863 INFO:teuthology.orchestra.run.smithi083.stdout: "evlrg": 0, 2024-09-16T00:43:20.864 INFO:teuthology.orchestra.run.smithi083.stdout: "evadd": 447041, 2024-09-16T00:43:20.864 INFO:teuthology.orchestra.run.smithi083.stdout: "evex": 337923, 2024-09-16T00:43:20.864 INFO:teuthology.orchestra.run.smithi083.stdout: "evtrm": 331831, 2024-09-16T00:43:20.864 INFO:teuthology.orchestra.run.smithi083.stdout: "ev": 115210, 2024-09-16T00:43:20.864 INFO:teuthology.orchestra.run.smithi083.stdout: "evexg": 0, 2024-09-16T00:43:20.864 INFO:teuthology.orchestra.run.smithi083.stdout: "evexd": 6092, 2024-09-16T00:43:20.864 INFO:teuthology.orchestra.run.smithi083.stdout: "segadd": 523, 2024-09-16T00:43:20.864 INFO:teuthology.orchestra.run.smithi083.stdout: "segex": 395, 2024-09-16T00:43:20.864 INFO:teuthology.orchestra.run.smithi083.stdout: "segtrm": 388, 2024-09-16T00:43:20.864 INFO:teuthology.orchestra.run.smithi083.stdout: "seg": 135, 2024-09-16T00:43:20.864 INFO:teuthology.orchestra.run.smithi083.stdout: "segmjr": 10, 2024-09-16T00:43:20.864 INFO:teuthology.orchestra.run.smithi083.stdout: "segexg": 0, 2024-09-16T00:43:20.865 INFO:teuthology.orchestra.run.smithi083.stdout: "segexd": 7, 2024-09-16T00:43:20.865 INFO:teuthology.orchestra.run.smithi083.stdout: "expos": 702333088, 2024-09-16T00:43:20.865 INFO:teuthology.orchestra.run.smithi083.stdout: "wrpos": 943224099, 2024-09-16T00:43:20.865 INFO:teuthology.orchestra.run.smithi083.stdout: "rdpos": 0, 2024-09-16T00:43:20.865 INFO:teuthology.orchestra.run.smithi083.stdout: "jlat": { 2024-09-16T00:43:20.865 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 125708, 2024-09-16T00:43:20.865 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 1477.353408233, 2024-09-16T00:43:20.865 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.011752262 2024-09-16T00:43:20.865 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.866 INFO:teuthology.orchestra.run.smithi083.stdout: "replayed": 0 2024-09-16T00:43:20.866 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_mem": { 2024-09-16T00:43:20.866 INFO:teuthology.orchestra.run.smithi083.stdout: "ino": 124, 2024-09-16T00:43:20.866 INFO:teuthology.orchestra.run.smithi083.stdout: "ino+": 93421, 2024-09-16T00:43:20.866 INFO:teuthology.orchestra.run.smithi083.stdout: "ino-": 93297, 2024-09-16T00:43:20.866 INFO:teuthology.orchestra.run.smithi083.stdout: "dir": 40, 2024-09-16T00:43:20.866 INFO:teuthology.orchestra.run.smithi083.stdout: "dir+": 633, 2024-09-16T00:43:20.866 INFO:teuthology.orchestra.run.smithi083.stdout: "dir-": 593, 2024-09-16T00:43:20.866 INFO:teuthology.orchestra.run.smithi083.stdout: "dn": 6869, 2024-09-16T00:43:20.866 INFO:teuthology.orchestra.run.smithi083.stdout: "dn+": 192508, 2024-09-16T00:43:20.867 INFO:teuthology.orchestra.run.smithi083.stdout: "dn-": 185639, 2024-09-16T00:43:20.867 INFO:teuthology.orchestra.run.smithi083.stdout: "cap": 1, 2024-09-16T00:43:20.867 INFO:teuthology.orchestra.run.smithi083.stdout: "cap+": 101298, 2024-09-16T00:43:20.867 INFO:teuthology.orchestra.run.smithi083.stdout: "cap-": 101297, 2024-09-16T00:43:20.867 INFO:teuthology.orchestra.run.smithi083.stdout: "rss": 106504, 2024-09-16T00:43:20.867 INFO:teuthology.orchestra.run.smithi083.stdout: "heap": 223520 2024-09-16T00:43:20.867 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_server": { 2024-09-16T00:43:20.867 INFO:teuthology.orchestra.run.smithi083.stdout: "dispatch_client_request": 754633, 2024-09-16T00:43:20.867 INFO:teuthology.orchestra.run.smithi083.stdout: "dispatch_server_request": 12686, 2024-09-16T00:43:20.867 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_request": 581654, 2024-09-16T00:43:20.868 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_session": 12394, 2024-09-16T00:43:20.868 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_peer_request": 28369, 2024-09-16T00:43:20.868 INFO:teuthology.orchestra.run.smithi083.stdout: "req_create_latency": { 2024-09-16T00:43:20.868 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 93220, 2024-09-16T00:43:20.868 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 1373.739575038, 2024-09-16T00:43:20.868 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.014736532 2024-09-16T00:43:20.868 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.868 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getattr_latency": { 2024-09-16T00:43:20.868 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 16188, 2024-09-16T00:43:20.868 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 348.295651974, 2024-09-16T00:43:20.868 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.021515669 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getfilelock_latency": { 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: "req_link_latency": { 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookup_latency": { 2024-09-16T00:43:20.869 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 194605, 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 805.716175107, 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.004140264 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookuphash_latency": { 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupino_latency": { 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 1, 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000147391, 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000147391 2024-09-16T00:43:20.870 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.871 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupname_latency": { 2024-09-16T00:43:20.871 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.871 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.871 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.871 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.871 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupparent_latency": { 2024-09-16T00:43:20.871 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.871 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.871 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.871 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.872 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupsnap_latency": { 2024-09-16T00:43:20.872 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.872 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.872 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.873 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.873 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lssnap_latency": { 2024-09-16T00:43:20.873 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.873 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.873 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.873 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.873 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mkdir_latency": { 2024-09-16T00:43:20.874 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 99, 2024-09-16T00:43:20.874 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.539989493, 2024-09-16T00:43:20.874 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.005454439 2024-09-16T00:43:20.874 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.874 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mknod_latency": { 2024-09-16T00:43:20.874 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.874 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.874 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.874 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.874 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mksnap_latency": { 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: "req_open_latency": { 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 9, 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.014442239, 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.001604693 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: "req_readdir_latency": { 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 143706, 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 1276.738046486, 2024-09-16T00:43:20.875 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.008884375 2024-09-16T00:43:20.876 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.876 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rename_latency": { 2024-09-16T00:43:20.876 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 19702, 2024-09-16T00:43:20.876 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 333.068579436, 2024-09-16T00:43:20.876 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.016905318 2024-09-16T00:43:20.876 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.876 INFO:teuthology.orchestra.run.smithi083.stdout: "req_renamesnap_latency": { 2024-09-16T00:43:20.876 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.876 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.876 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.877 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.877 INFO:teuthology.orchestra.run.smithi083.stdout: "req_snapdiff_latency": { 2024-09-16T00:43:20.877 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.877 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.877 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.877 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.877 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmdir_latency": { 2024-09-16T00:43:20.877 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 104, 2024-09-16T00:43:20.877 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 1.078364045, 2024-09-16T00:43:20.877 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.010368885 2024-09-16T00:43:20.877 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmsnap_latency": { 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmxattr_latency": { 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setattr_latency": { 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 313, 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 182.552528145, 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.583234914 2024-09-16T00:43:20.878 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setdirlayout_latency": { 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setfilelock_latency": { 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 2896, 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 11.687028400, 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.004035576 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setlayout_latency": { 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.879 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setxattr_latency": { 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: "req_symlink_latency": { 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.880 INFO:teuthology.orchestra.run.smithi083.stdout: "req_unlink_latency": { 2024-09-16T00:43:20.881 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 93131, 2024-09-16T00:43:20.881 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 1009.678656598, 2024-09-16T00:43:20.881 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.010841488 2024-09-16T00:43:20.881 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.881 INFO:teuthology.orchestra.run.smithi083.stdout: "cap_revoke_eviction": 0, 2024-09-16T00:43:20.881 INFO:teuthology.orchestra.run.smithi083.stdout: "cap_acquisition_throttle": 0, 2024-09-16T00:43:20.881 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getvxattr_latency": { 2024-09-16T00:43:20.881 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.881 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.881 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.881 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:20.881 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_sessions": { 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "session_count": 1, 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "session_add": 3, 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "session_remove": 2, 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "sessions_open": 1, 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "sessions_stale": 0, 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "total_load": 33152, 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "average_load": 33152, 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "avg_session_uptime": 1526, 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mdthresh_evicted": 0 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mempool": { 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_bytes": 0, 2024-09-16T00:43:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_items": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_items": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_items": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_items": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_bytes": 0, 2024-09-16T00:43:20.883 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_items": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_bytes": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_items": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_items": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_bytes": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_items": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_bytes": 0, 2024-09-16T00:43:20.884 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_items": 0, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_bytes": 0, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_items": 0, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_items": 0, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_items": 0, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_bytes": 293573, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_items": 216, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_bytes": 352, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_items": 4, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_bytes": 0, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_items": 0, 2024-09-16T00:43:20.885 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_bytes": 0, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_items": 0, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_bytes": 0, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_items": 0, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_bytes": 9624, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_items": 182, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_items": 0, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_bytes": 0, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_items": 0, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_bytes": 5610746, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_items": 89587, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_bytes": 0, 2024-09-16T00:43:20.886 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_items": 0, 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_bytes": 0, 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_items": 0 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter": { 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: "op_active": 0, 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: "op_laggy": 0, 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: "op_send": 332181, 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: "op_send_bytes": 964082311, 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: "op_resend": 33, 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: "op_reply": 332181, 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: "op_latency": { 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 332181, 2024-09-16T00:43:20.887 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 3155.465386409, 2024-09-16T00:43:20.888 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.009499235 2024-09-16T00:43:20.888 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.888 INFO:teuthology.orchestra.run.smithi083.stdout: "op_inflight": 0, 2024-09-16T00:43:20.888 INFO:teuthology.orchestra.run.smithi083.stdout: "oplen_avg": { 2024-09-16T00:43:20.888 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 332181, 2024-09-16T00:43:20.888 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 336369 2024-09-16T00:43:20.888 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.888 INFO:teuthology.orchestra.run.smithi083.stdout: "op": 332181, 2024-09-16T00:43:20.888 INFO:teuthology.orchestra.run.smithi083.stdout: "op_r": 13629, 2024-09-16T00:43:20.888 INFO:teuthology.orchestra.run.smithi083.stdout: "op_w": 318552, 2024-09-16T00:43:20.888 INFO:teuthology.orchestra.run.smithi083.stdout: "op_rmw": 0, 2024-09-16T00:43:20.888 INFO:teuthology.orchestra.run.smithi083.stdout: "op_pg": 0, 2024-09-16T00:43:20.889 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_stat": 9, 2024-09-16T00:43:20.889 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_create": 984, 2024-09-16T00:43:20.889 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_read": 2138, 2024-09-16T00:43:20.889 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_write": 127847, 2024-09-16T00:43:20.889 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_writefull": 983, 2024-09-16T00:43:20.889 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_writesame": 0, 2024-09-16T00:43:20.889 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_append": 0, 2024-09-16T00:43:20.889 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_zero": 0, 2024-09-16T00:43:20.889 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_truncate": 54, 2024-09-16T00:43:20.889 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_delete": 187017, 2024-09-16T00:43:20.889 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_mapext": 0, 2024-09-16T00:43:20.889 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_sparse_read": 0, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_clonerange": 0, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_getxattr": 11491, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_setxattr": 1533, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_cmpxattr": 0, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_rmxattr": 0, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_resetxattrs": 0, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_call": 0, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_watch": 0, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_notify": 0, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_src_cmpxattr": 0, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_pgls": 0, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_pgls_filter": 0, 2024-09-16T00:43:20.890 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_other": 0, 2024-09-16T00:43:20.891 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_active": 0, 2024-09-16T00:43:20.891 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_send": 0, 2024-09-16T00:43:20.891 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_resend": 0, 2024-09-16T00:43:20.891 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_ping": 0, 2024-09-16T00:43:20.891 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_active": 0, 2024-09-16T00:43:20.891 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_send": 0, 2024-09-16T00:43:20.891 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_resend": 0, 2024-09-16T00:43:20.891 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_active": 0, 2024-09-16T00:43:20.891 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_send": 0, 2024-09-16T00:43:20.891 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_resend": 0, 2024-09-16T00:43:20.891 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_active": 0, 2024-09-16T00:43:20.892 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_send": 0, 2024-09-16T00:43:20.892 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_resend": 0, 2024-09-16T00:43:20.892 INFO:teuthology.orchestra.run.smithi083.stdout: "command_active": 0, 2024-09-16T00:43:20.892 INFO:teuthology.orchestra.run.smithi083.stdout: "command_send": 0, 2024-09-16T00:43:20.892 INFO:teuthology.orchestra.run.smithi083.stdout: "command_resend": 0, 2024-09-16T00:43:20.892 INFO:teuthology.orchestra.run.smithi083.stdout: "map_epoch": 90, 2024-09-16T00:43:20.892 INFO:teuthology.orchestra.run.smithi083.stdout: "map_full": 0, 2024-09-16T00:43:20.892 INFO:teuthology.orchestra.run.smithi083.stdout: "map_inc": 4, 2024-09-16T00:43:20.892 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_sessions": 12, 2024-09-16T00:43:20.892 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_session_open": 12, 2024-09-16T00:43:20.892 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_session_close": 0, 2024-09-16T00:43:20.892 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_laggy": 0, 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_wr": 3206, 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_rd": 0, 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_del": 1107 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: "oft": { 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_objs": 1, 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_kv_pairs": 0, 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_updates": 0, 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_removes": 0 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: "osd-slow-ops": { 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: "slow_ops_count": 7631 2024-09-16T00:43:20.893 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: "purge_queue": { 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_ops": 0, 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_ops_high_water": 16, 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing": 0, 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_high_water": 16, 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executed_ops": 186170, 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executed": 93122, 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_item_in_journal": 0 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-09-16T00:43:20.894 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:20.895 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 1423675, 2024-09-16T00:43:20.895 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 281043171, 2024-09-16T00:43:20.895 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:20.895 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 1423675, 2024-09-16T00:43:20.895 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-09-16T00:43:20.895 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-09-16T00:43:20.895 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 1423675, 2024-09-16T00:43:20.895 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 281043171, 2024-09-16T00:43:20.895 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:20.896 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.896 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.896 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.896 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:20.896 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.896 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-objecter_bytes": { 2024-09-16T00:43:20.896 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:20.896 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-09-16T00:43:20.896 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:20.896 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-09-16T00:43:20.896 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-09-16T00:43:20.897 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:20.897 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:20.897 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 332181, 2024-09-16T00:43:20.897 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 974232017, 2024-09-16T00:43:20.897 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 332181, 2024-09-16T00:43:20.897 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 974232017, 2024-09-16T00:43:20.897 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:20.897 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.897 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.897 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.897 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:20.897 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.898 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-objecter_ops": { 2024-09-16T00:43:20.898 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:20.898 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 1024, 2024-09-16T00:43:20.898 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:20.898 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-09-16T00:43:20.898 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-09-16T00:43:20.898 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:20.898 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:20.898 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 332181, 2024-09-16T00:43:20.898 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 332181, 2024-09-16T00:43:20.898 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 332181, 2024-09-16T00:43:20.899 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 332181, 2024-09-16T00:43:20.899 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:20.899 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.899 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.899 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.899 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:20.899 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.899 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-write_buf_throttle": { 2024-09-16T00:43:20.899 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:20.899 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 3758096384, 2024-09-16T00:43:20.900 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:20.900 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 93122, 2024-09-16T00:43:20.900 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 10149706, 2024-09-16T00:43:20.900 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:20.900 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 93122, 2024-09-16T00:43:20.900 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-09-16T00:43:20.900 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-09-16T00:43:20.900 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 2139, 2024-09-16T00:43:20.900 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 10149706, 2024-09-16T00:43:20.900 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:20.900 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.900 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.901 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.901 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:20.901 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:20.901 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-write_buf_throttle-0x556fa0e3e500": { 2024-09-16T00:43:20.901 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 173469, 2024-09-16T00:43:20.901 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 3758096384, 2024-09-16T00:43:20.901 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:20.901 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 447041, 2024-09-16T00:43:20.901 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 939029795, 2024-09-16T00:43:20.901 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:20.901 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 447041, 2024-09-16T00:43:20.902 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-09-16T00:43:20.902 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-09-16T00:43:20.902 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 125708, 2024-09-16T00:43:20.902 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 938856326, 2024-09-16T00:43:20.902 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:20.902 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:20.902 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:20.902 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:20.902 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:20.902 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:20.902 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-16T00:43:21.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:21 smithi083 ceph-mon[35242]: pgmap v1136: 129 pgs: 129 active+clean; 539 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 386 KiB/s wr, 12 op/s 2024-09-16T00:43:21.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:21 smithi060 ceph-mon[35185]: pgmap v1136: 129 pgs: 129 active+clean; 539 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 386 KiB/s wr, 12 op/s 2024-09-16T00:43:22.080 INFO:tasks.check_counter:Daemon mds.c mds.exported=19 2024-09-16T00:43:22.080 INFO:tasks.check_counter:Daemon mds.c mds.imported=9 2024-09-16T00:43:22.081 DEBUG:tasks.check_counter:Getting stats from f 2024-09-16T00:43:22.081 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mds.f perf dump 2024-09-16T00:43:22.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:21 smithi033 ceph-mon[31243]: pgmap v1136: 129 pgs: 129 active+clean; 539 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 386 KiB/s wr, 12 op/s 2024-09-16T00:43:22.380 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:43:22.940 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-16T00:43:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T00:43:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 2906, 2024-09-16T00:43:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 2886, 2024-09-16T00:43:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 18428068, 2024-09-16T00:43:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 1053584, 2024-09-16T00:43:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 6, 2024-09-16T00:43:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 2, 2024-09-16T00:43:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.279298997, 2024-09-16T00:43:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.090439444, 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.107867200, 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.070470747, 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 2886, 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.135494020, 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000046948 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 897344, 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 284656 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 1526, 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 1873, 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 21353230, 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 1167352, 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 7, 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 5, 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.145884412, 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.050567698, 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.041768887, 2024-09-16T00:43:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.041566622, 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 1871, 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.080053497, 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000042786 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 682336 2024-09-16T00:43:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T00:43:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 1593, 2024-09-16T00:43:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 1595, 2024-09-16T00:43:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 15042471, 2024-09-16T00:43:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 506581, 2024-09-16T00:43:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 5, 2024-09-16T00:43:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 4, 2024-09-16T00:43:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.132366577, 2024-09-16T00:43:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.044296061, 2024-09-16T00:43:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.038846018, 2024-09-16T00:43:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.038471845, 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 1593, 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.058595826, 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000036783 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T00:43:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "cct": { 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "total_workers": 1, 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "unhealthy_workers": 0 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "finisher-MDSRank": { 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "queue_len": 0, 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "complete_latency": { 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 9218, 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.217483324, 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000023593 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "finisher-PurgeQueue": { 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "queue_len": 0, 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "complete_latency": { 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": { 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "request": 0, 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "reply": 0, 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "reply_latency": { 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "slow_reply": 0, 2024-09-16T00:43:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "forward": 0, 2024-09-16T00:43:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_fetch_complete": 0, 2024-09-16T00:43:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_fetch_keys": 0, 2024-09-16T00:43:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_commit": 0, 2024-09-16T00:43:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_split": 0, 2024-09-16T00:43:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_merge": 0, 2024-09-16T00:43:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes": 6264, 2024-09-16T00:43:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_top": 0, 2024-09-16T00:43:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_bottom": 6264, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_pin_tail": 0, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_pinned": 6264, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_expired": 0, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_with_caps": 0, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": 0, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "subtrees": 2, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse": 0, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_hit": 0, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_forward": 0, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_discover": 0, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_dir_fetch": 0, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_remote_ino": 0, 2024-09-16T00:43:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_lock": 0, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "load_cent": 0, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "q": 0, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "exported": 0, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "exported_inodes": 0, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "imported": 0, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "imported_inodes": 0, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_dir_fetch": 0, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_backtrace_fetch": 0, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_peer_discover": 0, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rfiles": 4, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rbytes": 234, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rsnaps": 0, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_backtrace_fetch": 0, 2024-09-16T00:43:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_set_tag": 0, 2024-09-16T00:43:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_backtrace_repaired": 0, 2024-09-16T00:43:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_inotable_repaired": 0, 2024-09-16T00:43:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dir_inodes": 0, 2024-09-16T00:43:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dir_base_inodes": 0, 2024-09-16T00:43:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dirfrag_rstats": 0, 2024-09-16T00:43:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_file_inodes": 0, 2024-09-16T00:43:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_inode_file_caps": 0, 2024-09-16T00:43:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_revoke": 0, 2024-09-16T00:43:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_grant": 0, 2024-09-16T00:43:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_trunc": 0, 2024-09-16T00:43:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-16T00:43:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_caps": 0, 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_caps_dirty": 0, 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_cap_release": 0, 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "process_request_cap_release": 0 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache": { 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_update": 0, 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_update_receipt": 0, 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_try_discover": 0, 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_send_discover": 0, 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_handle_discover": 0, 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays": 0, 2024-09-16T00:43:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays_delayed": 0, 2024-09-16T00:43:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays_enqueuing": 0, 2024-09-16T00:43:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_created": 0, 2024-09-16T00:43:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_enqueued": 0, 2024-09-16T00:43:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_reintegrated": 0, 2024-09-16T00:43:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_migrated": 0, 2024-09-16T00:43:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_processing": 0, 2024-09-16T00:43:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_enqueued": 0, 2024-09-16T00:43:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_prioritized": 0, 2024-09-16T00:43:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_started": 0, 2024-09-16T00:43:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_completed": 0, 2024-09-16T00:43:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_quiesce_path": 0, 2024-09-16T00:43:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_quiesce_inode": 0, 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_enqueue_scrub": 0, 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_exportdir": 0, 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_flush": 0, 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_fragmentdir": 0, 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_fragstats": 0, 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_inodestats": 0 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log": { 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "evlrg": 0, 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "evadd": 0, 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "evex": 0, 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "evtrm": 0, 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "ev": 39134, 2024-09-16T00:43:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "evexg": 0, 2024-09-16T00:43:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "evexd": 0, 2024-09-16T00:43:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "segadd": 0, 2024-09-16T00:43:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "segex": 0, 2024-09-16T00:43:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "segtrm": 0, 2024-09-16T00:43:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "seg": 48, 2024-09-16T00:43:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "segmjr": 5, 2024-09-16T00:43:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "segexg": 0, 2024-09-16T00:43:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "segexd": 0, 2024-09-16T00:43:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "expos": 4194304, 2024-09-16T00:43:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "wrpos": 56577619, 2024-09-16T00:43:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rdpos": 56575537, 2024-09-16T00:43:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "jlat": { 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "replayed": 39134 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_mem": { 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "ino": 42, 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "ino+": 6355, 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "ino-": 6313, 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "dir": 18, 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "dir+": 51, 2024-09-16T00:43:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "dir-": 33, 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "dn": 6264, 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "dn+": 6538, 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "dn-": 274, 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "cap": 0, 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "cap+": 0, 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "cap-": 0, 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rss": 60428, 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "heap": 215328 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_server": { 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "dispatch_client_request": 0, 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "dispatch_server_request": 0, 2024-09-16T00:43:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_request": 0, 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_session": 0, 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_peer_request": 0, 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "req_create_latency": { 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getattr_latency": { 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getfilelock_latency": { 2024-09-16T00:43:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "req_link_latency": { 2024-09-16T00:43:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookup_latency": { 2024-09-16T00:43:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookuphash_latency": { 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupino_latency": { 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupname_latency": { 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupparent_latency": { 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupsnap_latency": { 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lssnap_latency": { 2024-09-16T00:43:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mkdir_latency": { 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mknod_latency": { 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mksnap_latency": { 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "req_open_latency": { 2024-09-16T00:43:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "req_readdir_latency": { 2024-09-16T00:43:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rename_latency": { 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "req_renamesnap_latency": { 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "req_snapdiff_latency": { 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmdir_latency": { 2024-09-16T00:43:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmsnap_latency": { 2024-09-16T00:43:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmxattr_latency": { 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setattr_latency": { 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setdirlayout_latency": { 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.967 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.968 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.968 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setfilelock_latency": { 2024-09-16T00:43:22.968 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.968 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.968 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.968 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.968 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setlayout_latency": { 2024-09-16T00:43:22.968 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.968 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.968 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.968 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.968 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setxattr_latency": { 2024-09-16T00:43:22.969 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.969 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.969 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.969 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.969 INFO:teuthology.orchestra.run.smithi083.stdout: "req_symlink_latency": { 2024-09-16T00:43:22.969 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.969 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.969 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.969 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.969 INFO:teuthology.orchestra.run.smithi083.stdout: "req_unlink_latency": { 2024-09-16T00:43:22.969 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.969 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: "cap_revoke_eviction": 0, 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: "cap_acquisition_throttle": 0, 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getvxattr_latency": { 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_sessions": { 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: "session_count": 1, 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: "session_add": 4, 2024-09-16T00:43:22.970 INFO:teuthology.orchestra.run.smithi083.stdout: "session_remove": 3, 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "sessions_open": 2, 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "sessions_stale": 0, 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "total_load": 0, 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "average_load": 0, 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "avg_session_uptime": 4065, 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "mdthresh_evicted": 0 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "mempool": { 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_bytes": 0, 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_items": 0, 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_items": 0, 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T00:43:22.971 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_items": 0, 2024-09-16T00:43:22.972 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T00:43:22.972 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T00:43:22.972 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T00:43:22.972 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T00:43:22.972 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T00:43:22.972 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_items": 0, 2024-09-16T00:43:22.972 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T00:43:22.972 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T00:43:22.972 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_bytes": 0, 2024-09-16T00:43:22.972 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_items": 0, 2024-09-16T00:43:22.972 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_bytes": 0, 2024-09-16T00:43:22.972 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_items": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_items": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_bytes": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_items": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_bytes": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_items": 0, 2024-09-16T00:43:22.973 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_bytes": 0, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_items": 0, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_items": 0, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_items": 0, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_bytes": 217462, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_items": 89, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_bytes": 0, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_items": 0, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_bytes": 0, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_items": 0, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_bytes": 0, 2024-09-16T00:43:22.974 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_items": 0, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_bytes": 0, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_items": 0, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_bytes": 9624, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_items": 182, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_items": 0, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_bytes": 0, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_items": 0, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_bytes": 4531783, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_items": 27889, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_bytes": 0, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_items": 0, 2024-09-16T00:43:22.975 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_bytes": 0, 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_items": 0 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter": { 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: "op_active": 0, 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: "op_laggy": 0, 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: "op_send": 5536, 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: "op_send_bytes": 26, 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: "op_resend": 4, 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: "op_reply": 5536, 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: "op_latency": { 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 5536, 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 7.834809343, 2024-09-16T00:43:22.976 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.001415247 2024-09-16T00:43:22.977 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.977 INFO:teuthology.orchestra.run.smithi083.stdout: "op_inflight": 0, 2024-09-16T00:43:22.977 INFO:teuthology.orchestra.run.smithi083.stdout: "oplen_avg": { 2024-09-16T00:43:22.977 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 5536, 2024-09-16T00:43:22.977 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 5537 2024-09-16T00:43:22.977 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.977 INFO:teuthology.orchestra.run.smithi083.stdout: "op": 5536, 2024-09-16T00:43:22.977 INFO:teuthology.orchestra.run.smithi083.stdout: "op_r": 5536, 2024-09-16T00:43:22.977 INFO:teuthology.orchestra.run.smithi083.stdout: "op_w": 0, 2024-09-16T00:43:22.977 INFO:teuthology.orchestra.run.smithi083.stdout: "op_rmw": 0, 2024-09-16T00:43:22.978 INFO:teuthology.orchestra.run.smithi083.stdout: "op_pg": 0, 2024-09-16T00:43:22.978 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_stat": 3284, 2024-09-16T00:43:22.978 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_create": 0, 2024-09-16T00:43:22.978 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_read": 2249, 2024-09-16T00:43:22.978 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_write": 0, 2024-09-16T00:43:22.978 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_writefull": 0, 2024-09-16T00:43:22.978 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_writesame": 0, 2024-09-16T00:43:22.978 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_append": 0, 2024-09-16T00:43:22.978 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_zero": 0, 2024-09-16T00:43:22.978 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_truncate": 0, 2024-09-16T00:43:22.978 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_delete": 0, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_mapext": 0, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_sparse_read": 0, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_clonerange": 0, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_getxattr": 2, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_setxattr": 0, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_cmpxattr": 0, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_rmxattr": 0, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_resetxattrs": 0, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_call": 0, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_watch": 0, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_notify": 0, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_src_cmpxattr": 0, 2024-09-16T00:43:22.979 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_pgls": 0, 2024-09-16T00:43:22.980 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_pgls_filter": 0, 2024-09-16T00:43:22.980 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_other": 0, 2024-09-16T00:43:22.980 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_active": 0, 2024-09-16T00:43:22.980 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_send": 0, 2024-09-16T00:43:22.980 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_resend": 0, 2024-09-16T00:43:22.980 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_ping": 0, 2024-09-16T00:43:22.980 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_active": 0, 2024-09-16T00:43:22.980 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_send": 0, 2024-09-16T00:43:22.980 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_resend": 0, 2024-09-16T00:43:22.980 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_active": 0, 2024-09-16T00:43:22.980 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_send": 0, 2024-09-16T00:43:22.980 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_resend": 0, 2024-09-16T00:43:22.981 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_active": 0, 2024-09-16T00:43:22.981 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_send": 0, 2024-09-16T00:43:22.981 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_resend": 0, 2024-09-16T00:43:22.981 INFO:teuthology.orchestra.run.smithi083.stdout: "command_active": 0, 2024-09-16T00:43:22.981 INFO:teuthology.orchestra.run.smithi083.stdout: "command_send": 0, 2024-09-16T00:43:22.981 INFO:teuthology.orchestra.run.smithi083.stdout: "command_resend": 0, 2024-09-16T00:43:22.981 INFO:teuthology.orchestra.run.smithi083.stdout: "map_epoch": 90, 2024-09-16T00:43:22.981 INFO:teuthology.orchestra.run.smithi083.stdout: "map_full": 0, 2024-09-16T00:43:22.981 INFO:teuthology.orchestra.run.smithi083.stdout: "map_inc": 4, 2024-09-16T00:43:22.981 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_sessions": 8, 2024-09-16T00:43:22.981 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_session_open": 9, 2024-09-16T00:43:22.981 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_session_close": 1, 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_laggy": 0, 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_wr": 0, 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_rd": 2, 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_del": 0 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: "oft": { 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_objs": 0, 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_kv_pairs": 0, 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_updates": 0, 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_removes": 0 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: "osd-slow-ops": { 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: "slow_ops_count": 0 2024-09-16T00:43:22.982 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "purge_queue": { 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_ops": 0, 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_ops_high_water": 0, 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing": 0, 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_high_water": 0, 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executed_ops": 0, 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executed": 0, 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_item_in_journal": 0 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:22.983 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 6021, 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 54254003, 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 6021, 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 6021, 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 54254003, 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.984 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-objecter_bytes": { 2024-09-16T00:43:22.985 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:22.985 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-09-16T00:43:22.985 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:22.985 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-09-16T00:43:22.985 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-09-16T00:43:22.985 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:22.985 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:22.985 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 5536, 2024-09-16T00:43:22.985 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 52386857, 2024-09-16T00:43:22.985 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 5536, 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 52386857, 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-objecter_ops": { 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 1024, 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-09-16T00:43:22.986 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 5536, 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 5536, 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 5536, 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 5536, 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-write_buf_throttle": { 2024-09-16T00:43:22.987 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 3758096384, 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 0, 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 0, 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.988 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-write_buf_throttle-0x562fe159e6e0": { 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 3758096384, 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-09-16T00:43:22.989 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-09-16T00:43:22.990 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 0, 2024-09-16T00:43:22.990 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 0, 2024-09-16T00:43:22.990 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:22.990 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:22.990 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:22.990 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:22.990 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:22.990 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:22.990 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-16T00:43:23.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:23 smithi083 ceph-mon[35242]: pgmap v1137: 129 pgs: 129 active+clean; 542 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 817 KiB/s wr, 15 op/s 2024-09-16T00:43:23.884 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-09-16T00:43:23.885 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-09-16T00:43:23.885 DEBUG:tasks.check_counter:Getting stats from i 2024-09-16T00:43:23.885 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mds.i perf dump 2024-09-16T00:43:23.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:23 smithi060 ceph-mon[35185]: pgmap v1137: 129 pgs: 129 active+clean; 542 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 817 KiB/s wr, 15 op/s 2024-09-16T00:43:24.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:23 smithi033 ceph-mon[31243]: pgmap v1137: 129 pgs: 129 active+clean; 542 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 817 KiB/s wr, 15 op/s 2024-09-16T00:43:24.146 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:43:24.653 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-16T00:43:24.655 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T00:43:24.655 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 227147, 2024-09-16T00:43:24.655 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 189130, 2024-09-16T00:43:24.655 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 57243079, 2024-09-16T00:43:24.655 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 138815046, 2024-09-16T00:43:24.655 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 84, 2024-09-16T00:43:24.655 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 7, 2024-09-16T00:43:24.655 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 14.078113961, 2024-09-16T00:43:24.655 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 6.157160876, 2024-09-16T00:43:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 7.965422169, 2024-09-16T00:43:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 1.287017385, 2024-09-16T00:43:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 189126, 2024-09-16T00:43:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 10.190766916, 2024-09-16T00:43:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000053883 2024-09-16T00:43:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 346052, 2024-09-16T00:43:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.053001859, 2024-09-16T00:43:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000153 2024-09-16T00:43:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.657 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T00:43:24.657 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T00:43:24.657 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.657 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T00:43:24.657 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 60171, 2024-09-16T00:43:24.657 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 70558, 2024-09-16T00:43:24.657 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 14594621, 2024-09-16T00:43:24.657 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 121025150, 2024-09-16T00:43:24.657 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 9, 2024-09-16T00:43:24.657 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 5, 2024-09-16T00:43:24.657 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 4.914072672, 2024-09-16T00:43:24.657 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 1.782441596, 2024-09-16T00:43:24.658 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 1.811345271, 2024-09-16T00:43:24.658 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 1.278646751, 2024-09-16T00:43:24.658 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:24.658 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 70560, 2024-09-16T00:43:24.658 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 6.696051496, 2024-09-16T00:43:24.658 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000094898 2024-09-16T00:43:24.658 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.658 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:24.658 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 66941, 2024-09-16T00:43:24.658 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.012191595, 2024-09-16T00:43:24.658 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000182 2024-09-16T00:43:24.658 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.659 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 915664, 2024-09-16T00:43:24.659 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 311536 2024-09-16T00:43:24.659 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.659 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T00:43:24.659 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 292936, 2024-09-16T00:43:24.659 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 409194, 2024-09-16T00:43:24.659 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 104422384, 2024-09-16T00:43:24.659 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 393089240, 2024-09-16T00:43:24.659 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 10, 2024-09-16T00:43:24.659 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 6, 2024-09-16T00:43:24.659 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 22.466261143, 2024-09-16T00:43:24.660 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 9.577596439, 2024-09-16T00:43:24.660 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 13.794038264, 2024-09-16T00:43:24.660 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 1.560390775, 2024-09-16T00:43:24.660 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:24.660 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 409192, 2024-09-16T00:43:24.660 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 36.558993679, 2024-09-16T00:43:24.660 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000089344 2024-09-16T00:43:24.660 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.661 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:24.661 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 583452, 2024-09-16T00:43:24.661 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.104281890, 2024-09-16T00:43:24.661 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000178 2024-09-16T00:43:24.661 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.661 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 92655664, 2024-09-16T00:43:24.661 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 308167600 2024-09-16T00:43:24.661 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.661 INFO:teuthology.orchestra.run.smithi083.stdout: "cct": { 2024-09-16T00:43:24.661 INFO:teuthology.orchestra.run.smithi083.stdout: "total_workers": 1, 2024-09-16T00:43:24.661 INFO:teuthology.orchestra.run.smithi083.stdout: "unhealthy_workers": 0 2024-09-16T00:43:24.661 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.662 INFO:teuthology.orchestra.run.smithi083.stdout: "finisher-MDSRank": { 2024-09-16T00:43:24.662 INFO:teuthology.orchestra.run.smithi083.stdout: "queue_len": 0, 2024-09-16T00:43:24.662 INFO:teuthology.orchestra.run.smithi083.stdout: "complete_latency": { 2024-09-16T00:43:24.662 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 76763, 2024-09-16T00:43:24.662 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 97.531344593, 2024-09-16T00:43:24.662 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.001270551 2024-09-16T00:43:24.662 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:24.662 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.662 INFO:teuthology.orchestra.run.smithi083.stdout: "finisher-PurgeQueue": { 2024-09-16T00:43:24.662 INFO:teuthology.orchestra.run.smithi083.stdout: "queue_len": 0, 2024-09-16T00:43:24.662 INFO:teuthology.orchestra.run.smithi083.stdout: "complete_latency": { 2024-09-16T00:43:24.662 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 46439, 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 5.807400899, 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000125054 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": { 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: "request": 190753, 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: "reply": 163873, 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: "reply_latency": { 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 163873, 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 1000.272001534, 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.006103946 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.663 INFO:teuthology.orchestra.run.smithi083.stdout: "slow_reply": 0, 2024-09-16T00:43:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: "forward": 26880, 2024-09-16T00:43:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_fetch_complete": 0, 2024-09-16T00:43:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_fetch_keys": 0, 2024-09-16T00:43:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_commit": 41, 2024-09-16T00:43:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_split": 1, 2024-09-16T00:43:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_merge": 1, 2024-09-16T00:43:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes": 190, 2024-09-16T00:43:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_top": 70, 2024-09-16T00:43:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_bottom": 91, 2024-09-16T00:43:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_pin_tail": 29, 2024-09-16T00:43:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_pinned": 79, 2024-09-16T00:43:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_expired": 92, 2024-09-16T00:43:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_with_caps": 0, 2024-09-16T00:43:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": 0, 2024-09-16T00:43:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: "subtrees": 4, 2024-09-16T00:43:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse": 290603, 2024-09-16T00:43:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_hit": 152825, 2024-09-16T00:43:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_forward": 26880, 2024-09-16T00:43:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_discover": 25, 2024-09-16T00:43:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_dir_fetch": 0, 2024-09-16T00:43:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_remote_ino": 0, 2024-09-16T00:43:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_lock": 2199, 2024-09-16T00:43:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: "load_cent": 0, 2024-09-16T00:43:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: "q": 0, 2024-09-16T00:43:24.666 INFO:teuthology.orchestra.run.smithi083.stdout: "exported": 4, 2024-09-16T00:43:24.666 INFO:teuthology.orchestra.run.smithi083.stdout: "exported_inodes": 10, 2024-09-16T00:43:24.666 INFO:teuthology.orchestra.run.smithi083.stdout: "imported": 14, 2024-09-16T00:43:24.666 INFO:teuthology.orchestra.run.smithi083.stdout: "imported_inodes": 209, 2024-09-16T00:43:24.666 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_dir_fetch": 0, 2024-09-16T00:43:24.666 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_backtrace_fetch": 2077, 2024-09-16T00:43:24.666 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_peer_discover": 0, 2024-09-16T00:43:24.666 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rfiles": 702, 2024-09-16T00:43:24.666 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rbytes": 214210665, 2024-09-16T00:43:24.666 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rsnaps": 0, 2024-09-16T00:43:24.666 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_backtrace_fetch": 2077, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_set_tag": 0, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_backtrace_repaired": 0, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_inotable_repaired": 0, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dir_inodes": 304, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dir_base_inodes": 0, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dirfrag_rstats": 304, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_file_inodes": 1773, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_inode_file_caps": 7, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_revoke": 10240, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_grant": 92488, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_trunc": 0, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-16T00:43:24.667 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_caps": 54112, 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_caps_dirty": 21245, 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_cap_release": 8401, 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: "process_request_cap_release": 102880 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache": { 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_update": 5, 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_update_receipt": 36, 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_try_discover": 24, 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_send_discover": 39, 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_handle_discover": 11, 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays": 2, 2024-09-16T00:43:24.668 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays_delayed": 0, 2024-09-16T00:43:24.669 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays_enqueuing": 0, 2024-09-16T00:43:24.669 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_created": 26379, 2024-09-16T00:43:24.669 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_enqueued": 26377, 2024-09-16T00:43:24.669 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_reintegrated": 0, 2024-09-16T00:43:24.669 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_migrated": 0, 2024-09-16T00:43:24.669 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_processing": 0, 2024-09-16T00:43:24.669 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_enqueued": 0, 2024-09-16T00:43:24.669 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_prioritized": 0, 2024-09-16T00:43:24.669 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_started": 0, 2024-09-16T00:43:24.669 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_completed": 0, 2024-09-16T00:43:24.670 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_quiesce_path": 22, 2024-09-16T00:43:24.670 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_quiesce_inode": 1865, 2024-09-16T00:43:24.670 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_enqueue_scrub": 0, 2024-09-16T00:43:24.670 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_exportdir": 4, 2024-09-16T00:43:24.670 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_flush": 0, 2024-09-16T00:43:24.670 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_fragmentdir": 2, 2024-09-16T00:43:24.670 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_fragstats": 0, 2024-09-16T00:43:24.670 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_inodestats": 0 2024-09-16T00:43:24.670 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.670 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log": { 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "evlrg": 0, 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "evadd": 130732, 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "evex": 23552, 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "evtrm": 12291, 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "ev": 118441, 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "evexg": 0, 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "evexd": 11261, 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "segadd": 157, 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "segex": 29, 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "segtrm": 16, 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "seg": 141, 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "segmjr": 10, 2024-09-16T00:43:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "segexg": 0, 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "segexd": 13, 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "expos": 29679705, 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "wrpos": 266795226, 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "rdpos": 0, 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "jlat": { 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 40917, 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 249.879617245, 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.006106987 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "replayed": 0 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_mem": { 2024-09-16T00:43:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "ino": 50, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "ino+": 26436, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "ino-": 26386, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "dir": 22, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "dir+": 85, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "dir-": 63, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "dn": 171, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "dn+": 52218, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "dn-": 52047, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "cap": 0, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "cap+": 29360, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "cap-": 29360, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "rss": 77976, 2024-09-16T00:43:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "heap": 223520 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_server": { 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: "dispatch_client_request": 251805, 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: "dispatch_server_request": 6623, 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_request": 190753, 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_session": 3942, 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_peer_request": 16932, 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: "req_create_latency": { 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 26256, 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 205.626824378, 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.007831612 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getattr_latency": { 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 4757, 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 79.508553228, 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.016714011 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getfilelock_latency": { 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "req_link_latency": { 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookup_latency": { 2024-09-16T00:43:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 58584, 2024-09-16T00:43:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 60.941450456, 2024-09-16T00:43:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.001040240 2024-09-16T00:43:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookuphash_latency": { 2024-09-16T00:43:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupino_latency": { 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupname_latency": { 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupparent_latency": { 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.678 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.678 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.678 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupsnap_latency": { 2024-09-16T00:43:24.678 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.678 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.678 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lssnap_latency": { 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mkdir_latency": { 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mknod_latency": { 2024-09-16T00:43:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mksnap_latency": { 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "req_open_latency": { 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "req_readdir_latency": { 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 41390, 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 232.697294401, 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.005622065 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rename_latency": { 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 5497, 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 152.268562491, 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.027700302 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "req_renamesnap_latency": { 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: "req_snapdiff_latency": { 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmdir_latency": { 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 19, 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.191617776, 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.010085146 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmsnap_latency": { 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmxattr_latency": { 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setattr_latency": { 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 118, 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 56.378403899, 2024-09-16T00:43:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.477783083 2024-09-16T00:43:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setdirlayout_latency": { 2024-09-16T00:43:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setfilelock_latency": { 2024-09-16T00:43:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 892, 2024-09-16T00:43:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 1.021044156, 2024-09-16T00:43:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.001144668 2024-09-16T00:43:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setlayout_latency": { 2024-09-16T00:43:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setxattr_latency": { 2024-09-16T00:43:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: "req_symlink_latency": { 2024-09-16T00:43:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: "req_unlink_latency": { 2024-09-16T00:43:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 26360, 2024-09-16T00:43:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 211.638250749, 2024-09-16T00:43:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.008028765 2024-09-16T00:43:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: "cap_revoke_eviction": 0, 2024-09-16T00:43:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: "cap_acquisition_throttle": 0, 2024-09-16T00:43:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getvxattr_latency": { 2024-09-16T00:43:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_sessions": { 2024-09-16T00:43:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: "session_count": 1, 2024-09-16T00:43:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: "session_add": 1, 2024-09-16T00:43:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: "session_remove": 0, 2024-09-16T00:43:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: "sessions_open": 1, 2024-09-16T00:43:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: "sessions_stale": 0, 2024-09-16T00:43:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: "total_load": 16309, 2024-09-16T00:43:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: "average_load": 16309, 2024-09-16T00:43:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "avg_session_uptime": 1460, 2024-09-16T00:43:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "mdthresh_evicted": 0 2024-09-16T00:43:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "mempool": { 2024-09-16T00:43:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_bytes": 0, 2024-09-16T00:43:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_items": 0, 2024-09-16T00:43:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T00:43:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_items": 0, 2024-09-16T00:43:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T00:43:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_items": 0, 2024-09-16T00:43:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T00:43:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T00:43:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T00:43:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T00:43:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T00:43:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_items": 0, 2024-09-16T00:43:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T00:43:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T00:43:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_bytes": 0, 2024-09-16T00:43:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_items": 0, 2024-09-16T00:43:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_bytes": 0, 2024-09-16T00:43:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_items": 0, 2024-09-16T00:43:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T00:43:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T00:43:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T00:43:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T00:43:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T00:43:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_items": 0, 2024-09-16T00:43:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_bytes": 0, 2024-09-16T00:43:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_items": 0, 2024-09-16T00:43:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T00:43:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T00:43:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_bytes": 0, 2024-09-16T00:43:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_items": 0, 2024-09-16T00:43:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_bytes": 0, 2024-09-16T00:43:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_items": 0, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_items": 0, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_items": 0, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_bytes": 613456, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_items": 210, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_bytes": 0, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_items": 0, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_bytes": 0, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_items": 0, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_bytes": 0, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_items": 0, 2024-09-16T00:43:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_bytes": 0, 2024-09-16T00:43:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_items": 0, 2024-09-16T00:43:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_bytes": 9624, 2024-09-16T00:43:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_items": 182, 2024-09-16T00:43:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T00:43:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_items": 0, 2024-09-16T00:43:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_bytes": 0, 2024-09-16T00:43:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_items": 0, 2024-09-16T00:43:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_bytes": 352210, 2024-09-16T00:43:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_items": 9166, 2024-09-16T00:43:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_bytes": 0, 2024-09-16T00:43:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_items": 0, 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_bytes": 0, 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_items": 0 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter": { 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: "op_active": 0, 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: "op_laggy": 0, 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: "op_send": 98653, 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: "op_send_bytes": 268294417, 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: "op_resend": 28, 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: "op_reply": 98653, 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: "op_latency": { 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 98653, 2024-09-16T00:43:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 1073.316976681, 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.010879719 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: "op_inflight": 0, 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: "oplen_avg": { 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 98653, 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 99995 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: "op": 98653, 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: "op_r": 2885, 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: "op_w": 95768, 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: "op_rmw": 0, 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: "op_pg": 0, 2024-09-16T00:43:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_stat": 0, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_create": 222, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_read": 808, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_write": 41725, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_writefull": 433, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_writesame": 0, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_append": 0, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_zero": 0, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_truncate": 1, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_delete": 52825, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_mapext": 0, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_sparse_read": 0, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_clonerange": 0, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_getxattr": 2077, 2024-09-16T00:43:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_setxattr": 353, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_cmpxattr": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_rmxattr": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_resetxattrs": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_call": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_watch": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_notify": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_src_cmpxattr": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_pgls": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_pgls_filter": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_other": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_active": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_send": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_resend": 0, 2024-09-16T00:43:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_ping": 0, 2024-09-16T00:43:24.697 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_active": 0, 2024-09-16T00:43:24.697 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_send": 0, 2024-09-16T00:43:24.697 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_resend": 0, 2024-09-16T00:43:24.697 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_active": 0, 2024-09-16T00:43:24.697 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_send": 0, 2024-09-16T00:43:24.697 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_resend": 0, 2024-09-16T00:43:24.697 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_active": 0, 2024-09-16T00:43:24.697 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_send": 0, 2024-09-16T00:43:24.697 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_resend": 0, 2024-09-16T00:43:24.697 INFO:teuthology.orchestra.run.smithi083.stdout: "command_active": 0, 2024-09-16T00:43:24.697 INFO:teuthology.orchestra.run.smithi083.stdout: "command_send": 0, 2024-09-16T00:43:24.697 INFO:teuthology.orchestra.run.smithi083.stdout: "command_resend": 0, 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: "map_epoch": 90, 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: "map_full": 0, 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: "map_inc": 4, 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_sessions": 12, 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_session_open": 16, 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_session_close": 4, 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_laggy": 0, 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_wr": 1078, 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_rd": 0, 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_del": 473 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: "oft": { 2024-09-16T00:43:24.698 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_objs": 1, 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_kv_pairs": 0, 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_updates": 0, 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_removes": 0 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: "osd-slow-ops": { 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: "slow_ops_count": 1905 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: "purge_queue": { 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_ops": 0, 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_ops_high_water": 16, 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing": 0, 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_high_water": 10, 2024-09-16T00:43:24.699 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executed_ops": 52737, 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executed": 26377, 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_item_in_journal": 0 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 580250, 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 112419783, 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 580250, 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-09-16T00:43:24.700 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-09-16T00:43:24.701 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 580250, 2024-09-16T00:43:24.701 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 112419783, 2024-09-16T00:43:24.701 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:24.701 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.701 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.701 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.701 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:24.701 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.701 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-objecter_bytes": { 2024-09-16T00:43:24.701 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:24.701 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-09-16T00:43:24.701 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:24.702 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-09-16T00:43:24.702 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-09-16T00:43:24.702 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:24.702 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:24.702 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 98653, 2024-09-16T00:43:24.702 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 271169374, 2024-09-16T00:43:24.702 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 98653, 2024-09-16T00:43:24.702 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 271169374, 2024-09-16T00:43:24.702 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:24.702 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.702 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.702 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.703 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:24.703 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.703 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-objecter_ops": { 2024-09-16T00:43:24.703 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:24.703 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 1024, 2024-09-16T00:43:24.703 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:24.703 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-09-16T00:43:24.703 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-09-16T00:43:24.703 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:24.703 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-09-16T00:43:24.703 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 98653, 2024-09-16T00:43:24.703 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 98653, 2024-09-16T00:43:24.704 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 98653, 2024-09-16T00:43:24.704 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 98653, 2024-09-16T00:43:24.704 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:24.704 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.704 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.704 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.704 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:24.704 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.704 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-write_buf_throttle": { 2024-09-16T00:43:24.704 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:24.704 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 3758096384, 2024-09-16T00:43:24.704 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 26377, 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 2874957, 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 26377, 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 808, 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 2874957, 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.705 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-write_buf_throttle-0x5559b56dc500": { 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 16173, 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 3758096384, 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 130732, 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 262600922, 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 130732, 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 40917, 2024-09-16T00:43:24.706 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 262584749, 2024-09-16T00:43:24.707 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:24.707 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:24.707 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:24.707 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:24.707 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:24.707 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:24.707 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-16T00:43:25.590 INFO:tasks.check_counter:Daemon mds.i mds.exported=4 2024-09-16T00:43:25.590 INFO:tasks.check_counter:Daemon mds.i mds.imported=14 2024-09-16T00:43:25.590 DEBUG:tasks.check_counter:Getting stats from l 2024-09-16T00:43:25.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph daemon mds.l perf dump 2024-09-16T00:43:25.837 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T00:43:25.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:25 smithi083 ceph-mon[35242]: pgmap v1138: 129 pgs: 129 active+clean; 525 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 909 KiB/s wr, 29 op/s 2024-09-16T00:43:25.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:25 smithi060 ceph-mon[35185]: pgmap v1138: 129 pgs: 129 active+clean; 525 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 909 KiB/s wr, 29 op/s 2024-09-16T00:43:26.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:25 smithi033 ceph-mon[31243]: pgmap v1138: 129 pgs: 129 active+clean; 525 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 909 KiB/s wr, 29 op/s 2024-09-16T00:43:26.337 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-16T00:43:26.337 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T00:43:26.337 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 470, 2024-09-16T00:43:26.337 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 463, 2024-09-16T00:43:26.337 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 830064, 2024-09-16T00:43:26.337 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 286160, 2024-09-16T00:43:26.337 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 4, 2024-09-16T00:43:26.337 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 0, 2024-09-16T00:43:26.337 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.057663397, 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.024898749, 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.041114186, 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 463, 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.032772663, 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000070783 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:26.338 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 830064, 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 286160 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 2, 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 353, 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 320, 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 219536, 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 4, 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 2, 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.017312242, 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.012965306, 2024-09-16T00:43:26.339 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.001043136, 2024-09-16T00:43:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 351, 2024-09-16T00:43:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.019869367, 2024-09-16T00:43:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000056607 2024-09-16T00:43:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 219536 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 0, 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 2, 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 0, 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 0, 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 2, 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 1, 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.000334128, 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.000000000, 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.000256548, 2024-09-16T00:43:26.341 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T00:43:26.342 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:26.343 INFO:teuthology.orchestra.run.smithi083.stdout: "cct": { 2024-09-16T00:43:26.343 INFO:teuthology.orchestra.run.smithi083.stdout: "total_workers": 0, 2024-09-16T00:43:26.343 INFO:teuthology.orchestra.run.smithi083.stdout: "unhealthy_workers": 0 2024-09-16T00:43:26.343 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:26.343 INFO:teuthology.orchestra.run.smithi083.stdout: "mempool": { 2024-09-16T00:43:26.343 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_bytes": 0, 2024-09-16T00:43:26.343 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_items": 0, 2024-09-16T00:43:26.343 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T00:43:26.343 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_items": 0, 2024-09-16T00:43:26.343 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T00:43:26.343 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_items": 0, 2024-09-16T00:43:26.343 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T00:43:26.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T00:43:26.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T00:43:26.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T00:43:26.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T00:43:26.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_items": 0, 2024-09-16T00:43:26.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T00:43:26.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T00:43:26.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_bytes": 0, 2024-09-16T00:43:26.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_items": 0, 2024-09-16T00:43:26.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_bytes": 0, 2024-09-16T00:43:26.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_items": 0, 2024-09-16T00:43:26.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_items": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_bytes": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_items": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_bytes": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_items": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_bytes": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_items": 0, 2024-09-16T00:43:26.345 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T00:43:26.346 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_items": 0, 2024-09-16T00:43:26.346 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T00:43:26.346 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_items": 0, 2024-09-16T00:43:26.346 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_bytes": 148721, 2024-09-16T00:43:26.346 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_items": 30, 2024-09-16T00:43:26.346 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_bytes": 0, 2024-09-16T00:43:26.346 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_items": 0, 2024-09-16T00:43:26.346 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_bytes": 0, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_items": 0, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_bytes": 0, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_items": 0, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_bytes": 0, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_items": 0, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_bytes": 0, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_items": 0, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_items": 0, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_bytes": 0, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_items": 0, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_bytes": 928, 2024-09-16T00:43:26.347 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_items": 2, 2024-09-16T00:43:26.348 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_bytes": 0, 2024-09-16T00:43:26.348 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_items": 0, 2024-09-16T00:43:26.348 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_bytes": 0, 2024-09-16T00:43:26.348 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_items": 0 2024-09-16T00:43:26.348 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-16T00:43:26.348 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T00:43:26.348 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-09-16T00:43:26.348 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-09-16T00:43:26.348 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-09-16T00:43:26.348 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 468, 2024-09-16T00:43:26.348 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 761122, 2024-09-16T00:43:26.348 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-09-16T00:43:26.349 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 468, 2024-09-16T00:43:26.349 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-09-16T00:43:26.349 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-09-16T00:43:26.349 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 468, 2024-09-16T00:43:26.349 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 761122, 2024-09-16T00:43:26.349 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-09-16T00:43:26.349 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-09-16T00:43:26.349 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-09-16T00:43:26.349 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-09-16T00:43:26.349 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:26.349 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-16T00:43:26.349 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-16T00:43:27.058 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-09-16T00:43:27.058 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-09-16T00:43:27.059 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-16T00:43:27.069 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-16T00:43:27.069 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:43:27.069 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T00:43:27.101 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-16T00:43:27.101 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T00:43:27.161 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-16T00:43:27.469 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:27.890 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:27 smithi033 ceph-mon[31243]: pgmap v1139: 129 pgs: 129 active+clean; 519 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 908 KiB/s wr, 29 op/s 2024-09-16T00:43:27.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:27 smithi083 ceph-mon[35242]: pgmap v1139: 129 pgs: 129 active+clean; 519 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 908 KiB/s wr, 29 op/s 2024-09-16T00:43:27.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:27 smithi060 ceph-mon[35185]: pgmap v1139: 129 pgs: 129 active+clean; 519 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 908 KiB/s wr, 29 op/s 2024-09-16T00:43:27.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.976+0000 7f9af29cc640 1 -- 172.21.15.33:0/3664892908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aec102570 msgr2=0x7f9aec1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:27.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.976+0000 7f9af29cc640 1 --2- 172.21.15.33:0/3664892908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aec102570 0x7f9aec1029b0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009900 tx=0x7f9ad402a480 comp rx=0 tx=0).stop 2024-09-16T00:43:27.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.976+0000 7f9af29cc640 1 -- 172.21.15.33:0/3664892908 shutdown_connections 2024-09-16T00:43:27.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.976+0000 7f9af29cc640 1 --2- 172.21.15.33:0/3664892908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aec102ef0 0x7f9aec10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:27.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.976+0000 7f9af29cc640 1 --2- 172.21.15.33:0/3664892908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aec102570 0x7f9aec1029b0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:27.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.976+0000 7f9af29cc640 1 --2- 172.21.15.33:0/3664892908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec100e20 0x7f9aec101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:27.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.976+0000 7f9af29cc640 1 -- 172.21.15.33:0/3664892908 >> 172.21.15.33:0/3664892908 conn(0x7f9aec0fc7d0 msgr2=0x7f9aec0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:27.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.976+0000 7f9af29cc640 1 -- 172.21.15.33:0/3664892908 shutdown_connections 2024-09-16T00:43:27.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.976+0000 7f9af29cc640 1 -- 172.21.15.33:0/3664892908 wait complete. 2024-09-16T00:43:27.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.976+0000 7f9af29cc640 1 Processor -- start 2024-09-16T00:43:27.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.976+0000 7f9af29cc640 1 -- start start 2024-09-16T00:43:27.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9af29cc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aec100e20 0x7f9aec0fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:27.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9af29cc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec102570 0x7f9aec0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:27.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9af29cc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aec102ef0 0x7f9aec0fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:27.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9af29cc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9aec110f80 con 0x7f9aec100e20 2024-09-16T00:43:27.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9af29cc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9aec110e00 con 0x7f9aec102ef0 2024-09-16T00:43:27.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9af29cc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9aec111100 con 0x7f9aec102570 2024-09-16T00:43:27.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9aebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aec100e20 0x7f9aec0fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:27.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9aebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aec100e20 0x7f9aec0fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59438/0 (socket says 172.21.15.33:59438) 2024-09-16T00:43:27.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9aebfff640 1 -- 172.21.15.33:0/2776301867 learned_addr learned my addr 172.21.15.33:0/2776301867 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:27.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9af0f69640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aec102ef0 0x7f9aec0fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:27.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9ae3fff640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec102570 0x7f9aec0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:27.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9aebfff640 1 -- 172.21.15.33:0/2776301867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec102570 msgr2=0x7f9aec0fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:27.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9aebfff640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec102570 0x7f9aec0fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:27.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9aebfff640 1 -- 172.21.15.33:0/2776301867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aec102ef0 msgr2=0x7f9aec0fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:27.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9aebfff640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aec102ef0 0x7f9aec0fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:27.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9aebfff640 1 -- 172.21.15.33:0/2776301867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aec1a6a70 con 0x7f9aec100e20 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9af0f69640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aec102ef0 0x7f9aec0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9aebfff640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aec100e20 0x7f9aec0fcc80 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f9adc00d770 tx=0x7f9adc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.977+0000 7f9ae9ffb640 1 -- 172.21.15.33:0/2776301867 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc017070 con 0x7f9aec100e20 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.978+0000 7f9af29cc640 1 -- 172.21.15.33:0/2776301867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aec1a6d60 con 0x7f9aec100e20 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.978+0000 7f9ae9ffb640 1 -- 172.21.15.33:0/2776301867 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9adc005500 con 0x7f9aec100e20 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.978+0000 7f9ae9ffb640 1 -- 172.21.15.33:0/2776301867 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc004e00 con 0x7f9aec100e20 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.978+0000 7f9af29cc640 1 -- 172.21.15.33:0/2776301867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aec1a71f0 con 0x7f9aec100e20 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.979+0000 7f9af29cc640 1 -- 172.21.15.33:0/2776301867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aec101200 con 0x7f9aec100e20 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.980+0000 7f9ae9ffb640 1 -- 172.21.15.33:0/2776301867 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9adc004fa0 con 0x7f9aec100e20 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.981+0000 7f9ae9ffb640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9ad80a3240 0x7f9ad80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.981+0000 7f9ae3fff640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9ad80a3240 0x7f9ad80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.981+0000 7f9ae9ffb640 1 -- 172.21.15.33:0/2776301867 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f9adc0cfcf0 con 0x7f9aec100e20 2024-09-16T00:43:27.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.982+0000 7f9ae3fff640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9ad80a3240 0x7f9ad80a5700 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0fe710 tx=0x7f9ad4005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:27.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:27.985+0000 7f9ae9ffb640 1 -- 172.21.15.33:0/2776301867 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9adc09c530 con 0x7f9aec100e20 2024-09-16T00:43:28.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.174+0000 7f9af29cc640 1 -- 172.21.15.33:0/2776301867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9aec10a0a0 con 0x7f9aec100e20 2024-09-16T00:43:28.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.174+0000 7f9ae9ffb640 1 -- 172.21.15.33:0/2776301867 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+588 (secure 0 0 0) 0x7f9adc09be30 con 0x7f9aec100e20 2024-09-16T00:43:28.175 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/3553033370 2024-09-17T00:08:07.548547+0000 2024-09-16T00:43:28.175 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/122116489 2024-09-17T00:08:07.548547+0000 2024-09-16T00:43:28.176 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/1410359644 2024-09-17T00:08:07.548547+0000 2024-09-16T00:43:28.176 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2692588880 2024-09-17T00:08:07.548547+0000 2024-09-16T00:43:28.176 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/122116489 2024-09-17T00:08:07.548547+0000 2024-09-16T00:43:28.176 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/961233150 2024-09-17T00:07:43.600446+0000 2024-09-16T00:43:28.176 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/1160344784 2024-09-17T00:07:43.600446+0000 2024-09-16T00:43:28.176 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/961233150 2024-09-17T00:07:43.600446+0000 2024-09-16T00:43:28.176 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2039144880 2024-09-17T00:07:43.600446+0000 2024-09-16T00:43:28.176 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/3586056783 2024-09-17T00:07:43.600446+0000 2024-09-16T00:43:28.176 INFO:teuthology.orchestra.run.smithi033.stderr:listed 10 entries 2024-09-16T00:43:28.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.176+0000 7f9af29cc640 1 -- 172.21.15.33:0/2776301867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9ad80a3240 msgr2=0x7f9ad80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:28.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.176+0000 7f9af29cc640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9ad80a3240 0x7f9ad80a5700 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0fe710 tx=0x7f9ad4005a20 comp rx=0 tx=0).stop 2024-09-16T00:43:28.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.176+0000 7f9af29cc640 1 -- 172.21.15.33:0/2776301867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aec100e20 msgr2=0x7f9aec0fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:28.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.176+0000 7f9af29cc640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aec100e20 0x7f9aec0fcc80 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f9adc00d770 tx=0x7f9adc00dc30 comp rx=0 tx=0).stop 2024-09-16T00:43:28.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.176+0000 7f9af29cc640 1 -- 172.21.15.33:0/2776301867 shutdown_connections 2024-09-16T00:43:28.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.176+0000 7f9af29cc640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f9ad80a3240 0x7f9ad80a5700 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:28.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.176+0000 7f9af29cc640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aec102ef0 0x7f9aec0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:28.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.176+0000 7f9af29cc640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec102570 0x7f9aec0fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:28.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.176+0000 7f9af29cc640 1 --2- 172.21.15.33:0/2776301867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aec100e20 0x7f9aec0fcc80 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:28.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.176+0000 7f9af29cc640 1 -- 172.21.15.33:0/2776301867 >> 172.21.15.33:0/2776301867 conn(0x7f9aec0fc7d0 msgr2=0x7f9aec06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:28.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.176+0000 7f9af29cc640 1 -- 172.21.15.33:0/2776301867 shutdown_connections 2024-09-16T00:43:28.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:28.176+0000 7f9af29cc640 1 -- 172.21.15.33:0/2776301867 wait complete. 2024-09-16T00:43:28.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:28 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2776301867' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T00:43:28.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:28 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2776301867' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T00:43:29.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:28 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2776301867' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T00:43:29.108 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T00:43:29.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:43:29.108 DEBUG:teuthology.orchestra.run.smithi033:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T00:43:29.192 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi033.front.sepia.ceph.com 2024-09-16T00:43:29.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:43:29.192 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T00:43:29.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:43:29.242 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list 2024-09-16T00:43:29.286 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T00:43:29.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:43:29.286 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T00:43:29.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T00:43:29.391 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link delete ceph-brx 2024-09-16T00:43:29.466 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-16T00:43:29.477 INFO:tasks.cephadm:Teardown begin 2024-09-16T00:43:29.477 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T00:43:29.477 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:43:29.740 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:29.890 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:29 smithi033 ceph-mon[31243]: pgmap v1140: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 909 KiB/s wr, 32 op/s 2024-09-16T00:43:29.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:29 smithi083 ceph-mon[35242]: pgmap v1140: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 909 KiB/s wr, 32 op/s 2024-09-16T00:43:29.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:29 smithi060 ceph-mon[35185]: pgmap v1140: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 909 KiB/s wr, 32 op/s 2024-09-16T00:43:30.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.233+0000 7f7ba6a14640 1 -- 172.21.15.33:0/2146155593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ba0102da0 msgr2=0x7f7ba0103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:30.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.233+0000 7f7ba6a14640 1 --2- 172.21.15.33:0/2146155593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ba0102da0 0x7f7ba0103180 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f7b90009900 tx=0x7f7b9002a480 comp rx=0 tx=0).stop 2024-09-16T00:43:30.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.233+0000 7f7ba6a14640 1 -- 172.21.15.33:0/2146155593 shutdown_connections 2024-09-16T00:43:30.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.233+0000 7f7ba6a14640 1 --2- 172.21.15.33:0/2146155593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ba010c180 0x7f7ba010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:30.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.233+0000 7f7ba6a14640 1 --2- 172.21.15.33:0/2146155593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ba01036c0 0x7f7ba010bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:30.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.233+0000 7f7ba6a14640 1 --2- 172.21.15.33:0/2146155593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ba0102da0 0x7f7ba0103180 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:30.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.233+0000 7f7ba6a14640 1 -- 172.21.15.33:0/2146155593 >> 172.21.15.33:0/2146155593 conn(0x7f7ba00fc7a0 msgr2=0x7f7ba00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:30.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.234+0000 7f7ba6a14640 1 -- 172.21.15.33:0/2146155593 shutdown_connections 2024-09-16T00:43:30.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.234+0000 7f7ba6a14640 1 -- 172.21.15.33:0/2146155593 wait complete. 2024-09-16T00:43:30.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.234+0000 7f7ba6a14640 1 Processor -- start 2024-09-16T00:43:30.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.234+0000 7f7ba6a14640 1 -- start start 2024-09-16T00:43:30.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba6a14640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ba0102da0 0x7f7ba0100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:30.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba6a14640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ba01036c0 0x7f7ba0100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:30.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba6a14640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ba010c180 0x7f7ba0101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:30.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba6a14640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba0111f60 con 0x7f7ba010c180 2024-09-16T00:43:30.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba6a14640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba0111de0 con 0x7f7ba0102da0 2024-09-16T00:43:30.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba6a14640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba01120e0 con 0x7f7ba01036c0 2024-09-16T00:43:30.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba4fb1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ba010c180 0x7f7ba0101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:30.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba4fb1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ba010c180 0x7f7ba0101480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59444/0 (socket says 172.21.15.33:59444) 2024-09-16T00:43:30.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba4fb1640 1 -- 172.21.15.33:0/3738734910 learned_addr learned my addr 172.21.15.33:0/3738734910 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:30.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7b9ffff640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ba0102da0 0x7f7ba0100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7b9f7fe640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ba01036c0 0x7f7ba0100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba4fb1640 1 -- 172.21.15.33:0/3738734910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ba01036c0 msgr2=0x7f7ba0100f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba4fb1640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ba01036c0 0x7f7ba0100f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba4fb1640 1 -- 172.21.15.33:0/3738734910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ba0102da0 msgr2=0x7f7ba0100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba4fb1640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ba0102da0 0x7f7ba0100a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba4fb1640 1 -- 172.21.15.33:0/3738734910 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba01a6920 con 0x7f7ba010c180 2024-09-16T00:43:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba4fb1640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ba010c180 0x7f7ba0101480 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009fd0 tx=0x7f7b9800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7b9d7fa640 1 -- 172.21.15.33:0/3738734910 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98017070 con 0x7f7ba010c180 2024-09-16T00:43:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7b9d7fa640 1 -- 172.21.15.33:0/3738734910 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b98003e50 con 0x7f7ba010c180 2024-09-16T00:43:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7ba6a14640 1 -- 172.21.15.33:0/3738734910 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba01a6c10 con 0x7f7ba010c180 2024-09-16T00:43:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.235+0000 7f7b9d7fa640 1 -- 172.21.15.33:0/3738734910 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b980106c0 con 0x7f7ba010c180 2024-09-16T00:43:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.236+0000 7f7ba6a14640 1 -- 172.21.15.33:0/3738734910 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba01a7100 con 0x7f7ba010c180 2024-09-16T00:43:30.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.236+0000 7f7ba6a14640 1 -- 172.21.15.33:0/3738734910 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ba0103180 con 0x7f7ba010c180 2024-09-16T00:43:30.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.238+0000 7f7b9d7fa640 1 -- 172.21.15.33:0/3738734910 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7b98003970 con 0x7f7ba010c180 2024-09-16T00:43:30.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.238+0000 7f7b9d7fa640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b880a3240 0x7f7b880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:30.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.239+0000 7f7b9ffff640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b880a3240 0x7f7b880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:30.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.239+0000 7f7b9d7fa640 1 -- 172.21.15.33:0/3738734910 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7f7b980cfab0 con 0x7f7ba010c180 2024-09-16T00:43:30.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.239+0000 7f7b9ffff640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b880a3240 0x7f7b880a5700 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7b90009230 tx=0x7f7b900058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:30.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.242+0000 7f7b9d7fa640 1 -- 172.21.15.33:0/3738734910 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7b9809c2f0 con 0x7f7ba010c180 2024-09-16T00:43:30.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.456+0000 7f7ba6a14640 1 -- 172.21.15.33:0/3738734910 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ba0101f30 con 0x7f7ba010c180 2024-09-16T00:43:30.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.457+0000 7f7b9d7fa640 1 -- 172.21.15.33:0/3738734910 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11912 (secure 0 0 0) 0x7f7b9809bbf0 con 0x7f7ba010c180 2024-09-16T00:43:30.459 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:43:30.459 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":36,"btime":"2024-09-16T00:43:02:576958+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:43:02.479297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:43:30.459 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 36 2024-09-16T00:43:30.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.459+0000 7f7ba6a14640 1 -- 172.21.15.33:0/3738734910 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b880a3240 msgr2=0x7f7b880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:30.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.459+0000 7f7ba6a14640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b880a3240 0x7f7b880a5700 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7b90009230 tx=0x7f7b900058d0 comp rx=0 tx=0).stop 2024-09-16T00:43:30.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.459+0000 7f7ba6a14640 1 -- 172.21.15.33:0/3738734910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ba010c180 msgr2=0x7f7ba0101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:30.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.459+0000 7f7ba6a14640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ba010c180 0x7f7ba0101480 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009fd0 tx=0x7f7b9800b260 comp rx=0 tx=0).stop 2024-09-16T00:43:30.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.460+0000 7f7ba6a14640 1 -- 172.21.15.33:0/3738734910 shutdown_connections 2024-09-16T00:43:30.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.460+0000 7f7ba6a14640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f7b880a3240 0x7f7b880a5700 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:30.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.460+0000 7f7ba6a14640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ba010c180 0x7f7ba0101480 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:30.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.460+0000 7f7ba6a14640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ba01036c0 0x7f7ba0100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:30.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.460+0000 7f7ba6a14640 1 --2- 172.21.15.33:0/3738734910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ba0102da0 0x7f7ba0100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:30.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.460+0000 7f7ba6a14640 1 -- 172.21.15.33:0/3738734910 >> 172.21.15.33:0/3738734910 conn(0x7f7ba00fc7a0 msgr2=0x7f7ba00fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:30.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.460+0000 7f7ba6a14640 1 -- 172.21.15.33:0/3738734910 shutdown_connections 2024-09-16T00:43:30.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:30.460+0000 7f7ba6a14640 1 -- 172.21.15.33:0/3738734910 wait complete. 2024-09-16T00:43:30.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:30 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3738734910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:43:30.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:30 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3738734910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:43:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:30 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3738734910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:43:31.307 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T00:43:31.568 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:31.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:31 smithi083 ceph-mon[35242]: pgmap v1141: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 795 KiB/s wr, 34 op/s 2024-09-16T00:43:31.972 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:31 smithi033 ceph-mon[31243]: pgmap v1141: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 795 KiB/s wr, 34 op/s 2024-09-16T00:43:31.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:31 smithi060 ceph-mon[35185]: pgmap v1141: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 795 KiB/s wr, 34 op/s 2024-09-16T00:43:32.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.053+0000 7fe402874640 1 -- 172.21.15.33:0/824532249 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3fc10c170 msgr2=0x7fe3fc10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:32.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.053+0000 7fe402874640 1 --2- 172.21.15.33:0/824532249 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3fc10c170 0x7fe3fc10e610 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009ca0 tx=0x7fe3f402a950 comp rx=0 tx=0).stop 2024-09-16T00:43:32.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.053+0000 7fe402874640 1 -- 172.21.15.33:0/824532249 shutdown_connections 2024-09-16T00:43:32.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.053+0000 7fe402874640 1 --2- 172.21.15.33:0/824532249 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3fc10c170 0x7fe3fc10e610 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:32.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.053+0000 7fe402874640 1 --2- 172.21.15.33:0/824532249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3fc103660 0x7fe3fc10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:32.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.053+0000 7fe402874640 1 --2- 172.21.15.33:0/824532249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3fc102d40 0x7fe3fc103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:32.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.053+0000 7fe402874640 1 -- 172.21.15.33:0/824532249 >> 172.21.15.33:0/824532249 conn(0x7fe3fc0fc7a0 msgr2=0x7fe3fc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:32.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.053+0000 7fe402874640 1 -- 172.21.15.33:0/824532249 shutdown_connections 2024-09-16T00:43:32.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.053+0000 7fe402874640 1 -- 172.21.15.33:0/824532249 wait complete. 2024-09-16T00:43:32.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.054+0000 7fe402874640 1 Processor -- start 2024-09-16T00:43:32.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.054+0000 7fe402874640 1 -- start start 2024-09-16T00:43:32.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.054+0000 7fe402874640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3fc102d40 0x7fe3fc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:32.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.054+0000 7fe402874640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3fc103660 0x7fe3fc198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:32.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.054+0000 7fe402874640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3fc10c170 0x7fe3fc19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:32.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.054+0000 7fe402874640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc111f70 con 0x7fe3fc103660 2024-09-16T00:43:32.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.054+0000 7fe402874640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc111df0 con 0x7fe3fc10c170 2024-09-16T00:43:32.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.054+0000 7fe402874640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc1120f0 con 0x7fe3fc102d40 2024-09-16T00:43:32.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.054+0000 7fe3fb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3fc103660 0x7fe3fc198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:32.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.054+0000 7fe3fb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3fc103660 0x7fe3fc198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38560/0 (socket says 172.21.15.33:38560) 2024-09-16T00:43:32.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.054+0000 7fe3fb7fe640 1 -- 172.21.15.33:0/2479548450 learned_addr learned my addr 172.21.15.33:0/2479548450 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:32.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe400e11640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3fc10c170 0x7fe3fc19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:32.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe3fbfff640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3fc102d40 0x7fe3fc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:32.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe3fb7fe640 1 -- 172.21.15.33:0/2479548450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3fc102d40 msgr2=0x7fe3fc198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:32.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe3fb7fe640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3fc102d40 0x7fe3fc198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:32.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe3fb7fe640 1 -- 172.21.15.33:0/2479548450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3fc10c170 msgr2=0x7fe3fc19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:32.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe3fb7fe640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3fc10c170 0x7fe3fc19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:32.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe3fb7fe640 1 -- 172.21.15.33:0/2479548450 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3fc19d3c0 con 0x7fe3fc103660 2024-09-16T00:43:32.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe3fbfff640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3fc102d40 0x7fe3fc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:43:32.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe3fb7fe640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3fc103660 0x7fe3fc198950 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c940 tx=0x7fe3f000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:32.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe3f97fa640 1 -- 172.21.15.33:0/2479548450 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0004220 con 0x7fe3fc103660 2024-09-16T00:43:32.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe402874640 1 -- 172.21.15.33:0/2479548450 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3fc19d6b0 con 0x7fe3fc103660 2024-09-16T00:43:32.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe3f97fa640 1 -- 172.21.15.33:0/2479548450 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3f00043c0 con 0x7fe3fc103660 2024-09-16T00:43:32.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe3f97fa640 1 -- 172.21.15.33:0/2479548450 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0004a00 con 0x7fe3fc103660 2024-09-16T00:43:32.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.055+0000 7fe402874640 1 -- 172.21.15.33:0/2479548450 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3fc1a4f90 con 0x7fe3fc103660 2024-09-16T00:43:32.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.056+0000 7fe402874640 1 -- 172.21.15.33:0/2479548450 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3fc103120 con 0x7fe3fc103660 2024-09-16T00:43:32.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.057+0000 7fe3f97fa640 1 -- 172.21.15.33:0/2479548450 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe3f00079b0 con 0x7fe3fc103660 2024-09-16T00:43:32.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.058+0000 7fe3f97fa640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3e40a3240 0x7fe3e40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:32.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.058+0000 7fe3fbfff640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3e40a3240 0x7fe3e40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:32.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.058+0000 7fe3f97fa640 1 -- 172.21.15.33:0/2479548450 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fe3f00cfce0 con 0x7fe3fc103660 2024-09-16T00:43:32.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.059+0000 7fe3fbfff640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3e40a3240 0x7fe3e40a5700 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0058c0 tx=0x7fe3ec005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:32.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.062+0000 7fe3f97fa640 1 -- 172.21.15.33:0/2479548450 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe3f0014030 con 0x7fe3fc103660 2024-09-16T00:43:32.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.277+0000 7fe402874640 1 -- 172.21.15.33:0/2479548450 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3fc1a5280 con 0x7fe3fc103660 2024-09-16T00:43:32.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.278+0000 7fe3f97fa640 1 -- 172.21.15.33:0/2479548450 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11912 (secure 0 0 0) 0x7fe3f009c580 con 0x7fe3fc103660 2024-09-16T00:43:32.280 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:43:32.280 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":36,"btime":"2024-09-16T00:43:02:576958+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2093179667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2093179667},{"type":"v1","addr":"172.21.15.33:6835","nonce":2093179667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3526002638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3526002638},{"type":"v1","addr":"172.21.15.33:6837","nonce":3526002638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/4092505131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":4092505131},{"type":"v1","addr":"172.21.15.33:6839","nonce":4092505131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1588024134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1588024134},{"type":"v1","addr":"172.21.15.33:6841","nonce":1588024134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2130595683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2130595683},{"type":"v1","addr":"172.21.15.60:6833","nonce":2130595683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/1083835397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1083835397},{"type":"v1","addr":"172.21.15.83:6833","nonce":1083835397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T00:15:07.246355+0000","modified":"2024-09-16T00:43:02.479297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24493,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6835/1002409397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1002409397},{"type":"v1","addr":"172.21.15.60:6835","nonce":1002409397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6837/452012251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":452012251},{"type":"v1","addr":"172.21.15.60:6837","nonce":452012251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/2667354509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2667354509},{"type":"v1","addr":"172.21.15.83:6835","nonce":2667354509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.83:6837/4244918448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4244918448},{"type":"v1","addr":"172.21.15.83:6837","nonce":4244918448}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.83:6839/3202868586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":3202868586},{"type":"v1","addr":"172.21.15.83:6839","nonce":3202868586}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.60:6839/1222932045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":1222932045},{"type":"v1","addr":"172.21.15.60:6839","nonce":1222932045}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24493,24472]},"id":1}]} 2024-09-16T00:43:32.280 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 36 2024-09-16T00:43:32.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.281+0000 7fe402874640 1 -- 172.21.15.33:0/2479548450 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3e40a3240 msgr2=0x7fe3e40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:32.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.281+0000 7fe402874640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3e40a3240 0x7fe3e40a5700 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0058c0 tx=0x7fe3ec005850 comp rx=0 tx=0).stop 2024-09-16T00:43:32.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.281+0000 7fe402874640 1 -- 172.21.15.33:0/2479548450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3fc103660 msgr2=0x7fe3fc198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:32.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.281+0000 7fe402874640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3fc103660 0x7fe3fc198950 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c940 tx=0x7fe3f000ce00 comp rx=0 tx=0).stop 2024-09-16T00:43:32.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.281+0000 7fe402874640 1 -- 172.21.15.33:0/2479548450 shutdown_connections 2024-09-16T00:43:32.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.281+0000 7fe402874640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fe3e40a3240 0x7fe3e40a5700 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:32.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.281+0000 7fe402874640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3fc10c170 0x7fe3fc19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:32.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.281+0000 7fe402874640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3fc103660 0x7fe3fc198950 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:32.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.281+0000 7fe402874640 1 --2- 172.21.15.33:0/2479548450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3fc102d40 0x7fe3fc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:32.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.281+0000 7fe402874640 1 -- 172.21.15.33:0/2479548450 >> 172.21.15.33:0/2479548450 conn(0x7fe3fc0fc7a0 msgr2=0x7fe3fc0fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:32.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.281+0000 7fe402874640 1 -- 172.21.15.33:0/2479548450 shutdown_connections 2024-09-16T00:43:32.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:32.281+0000 7fe402874640 1 -- 172.21.15.33:0/2479548450 wait complete. 2024-09-16T00:43:32.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:32 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2479548450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:43:32.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:32 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2479548450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:43:33.069 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:32 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2479548450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T00:43:33.103 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T00:43:33.365 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:33.806 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:33 smithi033 ceph-mon[31243]: pgmap v1142: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 530 KiB/s wr, 27 op/s 2024-09-16T00:43:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.888+0000 7fa7e9f81640 1 -- 172.21.15.33:0/2926690382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e4102d40 msgr2=0x7fa7e4103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.888+0000 7fa7e9f81640 1 --2- 172.21.15.33:0/2926690382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e4102d40 0x7fa7e4103120 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009900 tx=0x7fa7d402a480 comp rx=0 tx=0).stop 2024-09-16T00:43:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.888+0000 7fa7e9f81640 1 -- 172.21.15.33:0/2926690382 shutdown_connections 2024-09-16T00:43:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.888+0000 7fa7e9f81640 1 --2- 172.21.15.33:0/2926690382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7e410c170 0x7fa7e410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.889+0000 7fa7e9f81640 1 --2- 172.21.15.33:0/2926690382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e4103660 0x7fa7e410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.889+0000 7fa7e9f81640 1 --2- 172.21.15.33:0/2926690382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e4102d40 0x7fa7e4103120 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.889+0000 7fa7e9f81640 1 -- 172.21.15.33:0/2926690382 >> 172.21.15.33:0/2926690382 conn(0x7fa7e40fc7a0 msgr2=0x7fa7e40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.889+0000 7fa7e9f81640 1 -- 172.21.15.33:0/2926690382 shutdown_connections 2024-09-16T00:43:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.889+0000 7fa7e9f81640 1 -- 172.21.15.33:0/2926690382 wait complete. 2024-09-16T00:43:33.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.889+0000 7fa7e9f81640 1 Processor -- start 2024-09-16T00:43:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.889+0000 7fa7e9f81640 1 -- start start 2024-09-16T00:43:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.889+0000 7fa7e9f81640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e4102d40 0x7fa7e41009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e9f81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e4103660 0x7fa7e4100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e9f81640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7e410c170 0x7fa7e4101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e9f81640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4111fb0 con 0x7fa7e4102d40 2024-09-16T00:43:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e9f81640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4111e30 con 0x7fa7e410c170 2024-09-16T00:43:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e9f81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4112130 con 0x7fa7e4103660 2024-09-16T00:43:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e4102d40 0x7fa7e41009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e4102d40 0x7fa7e41009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38574/0 (socket says 172.21.15.33:38574) 2024-09-16T00:43:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e37fe640 1 -- 172.21.15.33:0/1550773197 learned_addr learned my addr 172.21.15.33:0/1550773197 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e3fff640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7e410c170 0x7fa7e4101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:33.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e2ffd640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e4103660 0x7fa7e4100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e37fe640 1 -- 172.21.15.33:0/1550773197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e4103660 msgr2=0x7fa7e4100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e37fe640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e4103660 0x7fa7e4100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e37fe640 1 -- 172.21.15.33:0/1550773197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7e410c170 msgr2=0x7fa7e4101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e37fe640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7e410c170 0x7fa7e4101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e37fe640 1 -- 172.21.15.33:0/1550773197 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e41a6970 con 0x7fa7e4102d40 2024-09-16T00:43:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e2ffd640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e4103660 0x7fa7e4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:43:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e37fe640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e4102d40 0x7fa7e41009a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4006010 tx=0x7fa7d40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e0ff9640 1 -- 172.21.15.33:0/1550773197 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d403f030 con 0x7fa7e4102d40 2024-09-16T00:43:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e9f81640 1 -- 172.21.15.33:0/1550773197 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e41a6c00 con 0x7fa7e4102d40 2024-09-16T00:43:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e0ff9640 1 -- 172.21.15.33:0/1550773197 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7d4003ce0 con 0x7fa7e4102d40 2024-09-16T00:43:33.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.890+0000 7fa7e0ff9640 1 -- 172.21.15.33:0/1550773197 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4035070 con 0x7fa7e4102d40 2024-09-16T00:43:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.891+0000 7fa7e9f81640 1 -- 172.21.15.33:0/1550773197 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e41a7130 con 0x7fa7e4102d40 2024-09-16T00:43:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.891+0000 7fa7e9f81640 1 -- 172.21.15.33:0/1550773197 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e4103120 con 0x7fa7e4102d40 2024-09-16T00:43:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.893+0000 7fa7e0ff9640 1 -- 172.21.15.33:0/1550773197 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa7d4042030 con 0x7fa7e4102d40 2024-09-16T00:43:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.894+0000 7fa7e0ff9640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa7cc0a3240 0x7fa7cc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.894+0000 7fa7e2ffd640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa7cc0a3240 0x7fa7cc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.894+0000 7fa7e0ff9640 1 -- 172.21.15.33:0/1550773197 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7fa7d40f0530 con 0x7fa7e4102d40 2024-09-16T00:43:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.894+0000 7fa7e2ffd640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa7cc0a3240 0x7fa7cc0a5700 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8006fd0 tx=0x7fa7d8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:33.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:33.897+0000 7fa7e0ff9640 1 -- 172.21.15.33:0/1550773197 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa7d40bce90 con 0x7fa7e4102d40 2024-09-16T00:43:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:33 smithi083 ceph-mon[35242]: pgmap v1142: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 530 KiB/s wr, 27 op/s 2024-09-16T00:43:33.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:33 smithi060 ceph-mon[35185]: pgmap v1142: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 530 KiB/s wr, 27 op/s 2024-09-16T00:43:34.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.076+0000 7fa7e9f81640 1 -- 172.21.15.33:0/1550773197 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa7e41071e0 con 0x7fa7e4102d40 2024-09-16T00:43:34.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.077+0000 7fa7e0ff9640 1 -- 172.21.15.33:0/1550773197 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25076 (secure 0 0 0) 0x7fa7d40bc790 con 0x7fa7e4102d40 2024-09-16T00:43:34.079 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-16T00:43:34.079 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":90,"fsid":"7dde3d1a-73bf-11ef-bceb-c7b262605968","created":"2024-09-16T00:07:21.167733+0000","modified":"2024-09-16T00:17:09.569270+0000","last_up_change":"2024-09-16T00:14:14.482899+0000","last_in_change":"2024-09-16T00:13:51.100252+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T00:10:24.062476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T00:15:02.871178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T00:15:05.511764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T00:15:11.993036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"12847d0f-d226-40db-bd81-100ef8fddf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6803","nonce":1463114656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6805","nonce":1463114656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6809","nonce":1463114656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1463114656},{"type":"v1","addr":"172.21.15.33:6807","nonce":1463114656}]},"public_addr":"172.21.15.33:6803/1463114656","cluster_addr":"172.21.15.33:6805/1463114656","heartbeat_back_addr":"172.21.15.33:6809/1463114656","heartbeat_front_addr":"172.21.15.33:6807/1463114656","state":["exists","up"]},{"osd":1,"uuid":"a01a2f0e-e6f6-4ed2-9b6e-53bb0702a64d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6801","nonce":1932822239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6803","nonce":1932822239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6807","nonce":1932822239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1932822239},{"type":"v1","addr":"172.21.15.60:6805","nonce":1932822239}]},"public_addr":"172.21.15.60:6801/1932822239","cluster_addr":"172.21.15.60:6803/1932822239","heartbeat_back_addr":"172.21.15.60:6807/1932822239","heartbeat_front_addr":"172.21.15.60:6805/1932822239","state":["exists","up"]},{"osd":2,"uuid":"b978c9fd-d8e4-4b4a-8bc2-1671f0232a49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6801","nonce":409428043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6803","nonce":409428043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6807","nonce":409428043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":409428043},{"type":"v1","addr":"172.21.15.83:6805","nonce":409428043}]},"public_addr":"172.21.15.83:6801/409428043","cluster_addr":"172.21.15.83:6803/409428043","heartbeat_back_addr":"172.21.15.83:6807/409428043","heartbeat_front_addr":"172.21.15.83:6805/409428043","state":["exists","up"]},{"osd":3,"uuid":"94dae86a-b8b7-4563-84f4-c14c9a07bda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6811","nonce":712525362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6813","nonce":712525362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6817","nonce":712525362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":712525362},{"type":"v1","addr":"172.21.15.33:6815","nonce":712525362}]},"public_addr":"172.21.15.33:6811/712525362","cluster_addr":"172.21.15.33:6813/712525362","heartbeat_back_addr":"172.21.15.33:6817/712525362","heartbeat_front_addr":"172.21.15.33:6815/712525362","state":["exists","up"]},{"osd":4,"uuid":"c3fa6bf5-b880-4cd0-9912-c64f33b78b34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6809","nonce":3488110488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6811","nonce":3488110488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6815","nonce":3488110488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3488110488},{"type":"v1","addr":"172.21.15.60:6813","nonce":3488110488}]},"public_addr":"172.21.15.60:6809/3488110488","cluster_addr":"172.21.15.60:6811/3488110488","heartbeat_back_addr":"172.21.15.60:6815/3488110488","heartbeat_front_addr":"172.21.15.60:6813/3488110488","state":["exists","up"]},{"osd":5,"uuid":"cf0905ff-87f4-4fa5-a61f-c9f49e6fd3ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6809","nonce":2896581095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6811","nonce":2896581095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6815","nonce":2896581095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2896581095},{"type":"v1","addr":"172.21.15.83:6813","nonce":2896581095}]},"public_addr":"172.21.15.83:6809/2896581095","cluster_addr":"172.21.15.83:6811/2896581095","heartbeat_back_addr":"172.21.15.83:6815/2896581095","heartbeat_front_addr":"172.21.15.83:6813/2896581095","state":["exists","up"]},{"osd":6,"uuid":"48de98ea-b98b-4f4f-b771-38a7d816e784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6819","nonce":2633676155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6821","nonce":2633676155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6825","nonce":2633676155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2633676155},{"type":"v1","addr":"172.21.15.33:6823","nonce":2633676155}]},"public_addr":"172.21.15.33:6819/2633676155","cluster_addr":"172.21.15.33:6821/2633676155","heartbeat_back_addr":"172.21.15.33:6825/2633676155","heartbeat_front_addr":"172.21.15.33:6823/2633676155","state":["exists","up"]},{"osd":7,"uuid":"44266591-c35b-48fe-9a90-e1597d7e311a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6817","nonce":1473789559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6819","nonce":1473789559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6823","nonce":1473789559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1473789559},{"type":"v1","addr":"172.21.15.60:6821","nonce":1473789559}]},"public_addr":"172.21.15.60:6817/1473789559","cluster_addr":"172.21.15.60:6819/1473789559","heartbeat_back_addr":"172.21.15.60:6823/1473789559","heartbeat_front_addr":"172.21.15.60:6821/1473789559","state":["exists","up"]},{"osd":8,"uuid":"4fadf88b-ff77-46ed-8c31-34b974af9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6817","nonce":2115210527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6819","nonce":2115210527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6823","nonce":2115210527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2115210527},{"type":"v1","addr":"172.21.15.83:6821","nonce":2115210527}]},"public_addr":"172.21.15.83:6817/2115210527","cluster_addr":"172.21.15.83:6819/2115210527","heartbeat_back_addr":"172.21.15.83:6823/2115210527","heartbeat_front_addr":"172.21.15.83:6821/2115210527","state":["exists","up"]},{"osd":9,"uuid":"03160394-ba77-4a71-80fe-6c20837c3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6827","nonce":2906942241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6829","nonce":2906942241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6833","nonce":2906942241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":2906942241},{"type":"v1","addr":"172.21.15.33:6831","nonce":2906942241}]},"public_addr":"172.21.15.33:6827/2906942241","cluster_addr":"172.21.15.33:6829/2906942241","heartbeat_back_addr":"172.21.15.33:6833/2906942241","heartbeat_front_addr":"172.21.15.33:6831/2906942241","state":["exists","up"]},{"osd":10,"uuid":"ae2336fa-0b07-42a2-b67f-c77b0f7db0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6825","nonce":3903707079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6827","nonce":3903707079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6831","nonce":3903707079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":3903707079},{"type":"v1","addr":"172.21.15.60:6829","nonce":3903707079}]},"public_addr":"172.21.15.60:6825/3903707079","cluster_addr":"172.21.15.60:6827/3903707079","heartbeat_back_addr":"172.21.15.60:6831/3903707079","heartbeat_front_addr":"172.21.15.60:6829/3903707079","state":["exists","up"]},{"osd":11,"uuid":"02536f55-f270-4950-8fb3-0b528f79dac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6825","nonce":2178604075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6827","nonce":2178604075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6831","nonce":2178604075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2178604075},{"type":"v1","addr":"172.21.15.83:6829","nonce":2178604075}]},"public_addr":"172.21.15.83:6825/2178604075","cluster_addr":"172.21.15.83:6827/2178604075","heartbeat_back_addr":"172.21.15.83:6831/2178604075","heartbeat_front_addr":"172.21.15.83:6829/2178604075","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:31.120666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:09:56.213944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:20.772660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:10:45.504328+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:11.966466+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:11:37.054936+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:03.040547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:30.083116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:12:55.278261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:21.441756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:13:46.535922+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T00:14:12.726429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2e","mappings":[{"from":1,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/3553033370":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6801/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/1410359644":"2024-09-17T00:08:07.548547+0000","172.21.15.33:0/2692588880":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/122116489":"2024-09-17T00:08:07.548547+0000","172.21.15.33:6800/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/1160344784":"2024-09-17T00:07:43.600446+0000","172.21.15.33:6801/961233150":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/2039144880":"2024-09-17T00:07:43.600446+0000","172.21.15.33:0/3586056783":"2024-09-17T00:07:43.600446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T00:43:34.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.080+0000 7fa7e9f81640 1 -- 172.21.15.33:0/1550773197 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa7cc0a3240 msgr2=0x7fa7cc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:34.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.080+0000 7fa7e9f81640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa7cc0a3240 0x7fa7cc0a5700 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8006fd0 tx=0x7fa7d8008040 comp rx=0 tx=0).stop 2024-09-16T00:43:34.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.080+0000 7fa7e9f81640 1 -- 172.21.15.33:0/1550773197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e4102d40 msgr2=0x7fa7e41009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:34.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.080+0000 7fa7e9f81640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e4102d40 0x7fa7e41009a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4006010 tx=0x7fa7d40042f0 comp rx=0 tx=0).stop 2024-09-16T00:43:34.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.080+0000 7fa7e9f81640 1 -- 172.21.15.33:0/1550773197 shutdown_connections 2024-09-16T00:43:34.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.080+0000 7fa7e9f81640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fa7cc0a3240 0x7fa7cc0a5700 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:34.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.080+0000 7fa7e9f81640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7e410c170 0x7fa7e4101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.080+0000 7fa7e9f81640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e4103660 0x7fa7e4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.080+0000 7fa7e9f81640 1 --2- 172.21.15.33:0/1550773197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e4102d40 0x7fa7e41009a0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.080+0000 7fa7e9f81640 1 -- 172.21.15.33:0/1550773197 >> 172.21.15.33:0/1550773197 conn(0x7fa7e40fc7a0 msgr2=0x7fa7e40fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.080+0000 7fa7e9f81640 1 -- 172.21.15.33:0/1550773197 shutdown_connections 2024-09-16T00:43:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:34.080+0000 7fa7e9f81640 1 -- 172.21.15.33:0/1550773197 wait complete. 2024-09-16T00:43:34.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:34 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1550773197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:43:34.905 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs fail cephfs 2024-09-16T00:43:34.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:34 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1550773197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:43:34.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:34 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1550773197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T00:43:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:35.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.731+0000 7fecba216640 1 -- 172.21.15.33:0/2954047488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecb4102790 msgr2=0x7fecb41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:35.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.731+0000 7fecba216640 1 --2- 172.21.15.33:0/2954047488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecb4102790 0x7fecb41067e0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7feca8009900 tx=0x7feca802a410 comp rx=0 tx=0).stop 2024-09-16T00:43:35.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.731+0000 7fecba216640 1 -- 172.21.15.33:0/2954047488 shutdown_connections 2024-09-16T00:43:35.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.731+0000 7fecba216640 1 --2- 172.21.15.33:0/2954047488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecb4106f10 0x7fecb410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:35.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.731+0000 7fecba216640 1 --2- 172.21.15.33:0/2954047488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecb4102790 0x7fecb41067e0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:35.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.731+0000 7fecba216640 1 --2- 172.21.15.33:0/2954047488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecb4101de0 0x7fecb41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:35.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.731+0000 7fecba216640 1 -- 172.21.15.33:0/2954047488 >> 172.21.15.33:0/2954047488 conn(0x7fecb40fd8c0 msgr2=0x7fecb40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:35.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.731+0000 7fecba216640 1 -- 172.21.15.33:0/2954047488 shutdown_connections 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.731+0000 7fecba216640 1 -- 172.21.15.33:0/2954047488 wait complete. 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecba216640 1 Processor -- start 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecba216640 1 -- start start 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecba216640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecb4101de0 0x7fecb4198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecba216640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecb4102790 0x7fecb41989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecba216640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecb4106f10 0x7fecb419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecba216640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fecb4111f90 con 0x7fecb4102790 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecba216640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fecb4111e10 con 0x7fecb4101de0 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecba216640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fecb4112110 con 0x7fecb4106f10 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecb2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecb4102790 0x7fecb41989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecb2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecb4102790 0x7fecb41989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38584/0 (socket says 172.21.15.33:38584) 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecb2ffd640 1 -- 172.21.15.33:0/3517599718 learned_addr learned my addr 172.21.15.33:0/3517599718 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:35.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecb37fe640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecb4101de0 0x7fecb4198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:35.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecb3fff640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecb4106f10 0x7fecb419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:35.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecb2ffd640 1 -- 172.21.15.33:0/3517599718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecb4106f10 msgr2=0x7fecb419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:35.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecb2ffd640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecb4106f10 0x7fecb419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:35.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecb2ffd640 1 -- 172.21.15.33:0/3517599718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecb4101de0 msgr2=0x7fecb4198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:35.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.732+0000 7fecb2ffd640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecb4101de0 0x7fecb4198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:35.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.733+0000 7fecb2ffd640 1 -- 172.21.15.33:0/3517599718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb419d420 con 0x7fecb4102790 2024-09-16T00:43:35.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.733+0000 7fecb2ffd640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecb4102790 0x7fecb41989b0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7feca8006010 tx=0x7feca80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:35.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.733+0000 7fecb0ff9640 1 -- 172.21.15.33:0/3517599718 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca803f030 con 0x7fecb4102790 2024-09-16T00:43:35.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.733+0000 7fecb0ff9640 1 -- 172.21.15.33:0/3517599718 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feca8003ce0 con 0x7fecb4102790 2024-09-16T00:43:35.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.733+0000 7fecba216640 1 -- 172.21.15.33:0/3517599718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecb419d6b0 con 0x7fecb4102790 2024-09-16T00:43:35.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.733+0000 7fecb0ff9640 1 -- 172.21.15.33:0/3517599718 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca8035070 con 0x7fecb4102790 2024-09-16T00:43:35.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.733+0000 7fecba216640 1 -- 172.21.15.33:0/3517599718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecb41a4ff0 con 0x7fecb4102790 2024-09-16T00:43:35.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.734+0000 7fecba216640 1 -- 172.21.15.33:0/3517599718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecb41032c0 con 0x7fecb4102790 2024-09-16T00:43:35.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.735+0000 7fecb0ff9640 1 -- 172.21.15.33:0/3517599718 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7feca8042030 con 0x7fecb4102790 2024-09-16T00:43:35.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.736+0000 7fecb0ff9640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec9c0a3240 0x7fec9c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:35.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.736+0000 7fecb37fe640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec9c0a3240 0x7fec9c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:35.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.736+0000 7fecb0ff9640 1 -- 172.21.15.33:0/3517599718 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9382+0+0 (secure 0 0 0) 0x7feca80f0400 con 0x7fecb4102790 2024-09-16T00:43:35.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.736+0000 7fecb37fe640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec9c0a3240 0x7fec9c0a5700 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7feca4005d00 tx=0x7feca400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:35.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.740+0000 7fecb0ff9640 1 -- 172.21.15.33:0/3517599718 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7feca80bc6b0 con 0x7fecb4102790 2024-09-16T00:43:35.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:35 smithi083 ceph-mon[35242]: pgmap v1143: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 99 KiB/s wr, 26 op/s 2024-09-16T00:43:35.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.950+0000 7fecba216640 1 -- 172.21.15.33:0/3517599718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fecb410a0f0 con 0x7fecb4102790 2024-09-16T00:43:35.952 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:35 smithi033 ceph-mon[31243]: pgmap v1143: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 99 KiB/s wr, 26 op/s 2024-09-16T00:43:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.954+0000 7fecb0ff9640 1 -- 172.21.15.33:0/3517599718 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) ==== 161+0+0 (secure 0 0 0) 0x7feca80c09b0 con 0x7fecb4102790 2024-09-16T00:43:35.955 INFO:teuthology.orchestra.run.smithi033.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T00:43:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.957+0000 7fecba216640 1 -- 172.21.15.33:0/3517599718 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec9c0a3240 msgr2=0x7fec9c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.957+0000 7fecba216640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec9c0a3240 0x7fec9c0a5700 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7feca4005d00 tx=0x7feca400b410 comp rx=0 tx=0).stop 2024-09-16T00:43:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.957+0000 7fecba216640 1 -- 172.21.15.33:0/3517599718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecb4102790 msgr2=0x7fecb41989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:35.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.957+0000 7fecba216640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecb4102790 0x7fecb41989b0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7feca8006010 tx=0x7feca80042f0 comp rx=0 tx=0).stop 2024-09-16T00:43:35.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.957+0000 7fecba216640 1 -- 172.21.15.33:0/3517599718 shutdown_connections 2024-09-16T00:43:35.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.957+0000 7fecba216640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fec9c0a3240 0x7fec9c0a5700 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:35.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.957+0000 7fecba216640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecb4106f10 0x7fecb419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:35.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.957+0000 7fecba216640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecb4102790 0x7fecb41989b0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:35.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.957+0000 7fecba216640 1 --2- 172.21.15.33:0/3517599718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecb4101de0 0x7fecb4198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:35.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.957+0000 7fecba216640 1 -- 172.21.15.33:0/3517599718 >> 172.21.15.33:0/3517599718 conn(0x7fecb40fd8c0 msgr2=0x7fecb40fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:35.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.957+0000 7fecba216640 1 -- 172.21.15.33:0/3517599718 shutdown_connections 2024-09-16T00:43:35.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:35.957+0000 7fecba216640 1 -- 172.21.15.33:0/3517599718 wait complete. 2024-09-16T00:43:35.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:35 smithi060 ceph-mon[35185]: pgmap v1143: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 99 KiB/s wr, 26 op/s 2024-09-16T00:43:36.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:35 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a[31239]: 2024-09-16T00:43:35.951+0000 7fa1d7d80640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T00:43:36.418 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-f[64761]: ignoring --setuser ceph since I am not root 2024-09-16T00:43:36.418 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-f[64761]: ignoring --setgroup ceph since I am not root 2024-09-16T00:43:36.418 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-mds[64765]: -- 172.21.15.83:0/4220093507 <== mon.2 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f970083860 con 0x55f970061000 2024-09-16T00:43:36.419 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-f[64761]: starting mds.f at 2024-09-16T00:43:36.419 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i[65364]: ignoring --setuser ceph since I am not root 2024-09-16T00:43:36.419 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i[65364]: ignoring --setgroup ceph since I am not root 2024-09-16T00:43:36.419 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-mds[65368]: -- 172.21.15.83:0/204793534 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a2acb22680 con 0x55a2acaf7000 2024-09-16T00:43:36.419 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-mds[65368]: -- 172.21.15.83:0/204793534 <== mon.2 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a2acb19860 con 0x55a2acaf7000 2024-09-16T00:43:36.419 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i[65364]: starting mds.i at 2024-09-16T00:43:36.419 INFO:journalctl@ceph.mds.c.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-c[65064]: ignoring --setuser ceph since I am not root 2024-09-16T00:43:36.420 INFO:journalctl@ceph.mds.c.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-c[65064]: ignoring --setgroup ceph since I am not root 2024-09-16T00:43:36.420 INFO:journalctl@ceph.mds.c.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-mds[65068]: -- 172.21.15.83:0/3791553835 <== mon.2 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56258a8b5860 con 0x56258a893400 2024-09-16T00:43:36.420 INFO:journalctl@ceph.mds.c.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-c[65064]: starting mds.c at 2024-09-16T00:43:36.480 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b[65594]: ignoring --setuser ceph since I am not root 2024-09-16T00:43:36.480 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b[65594]: ignoring --setgroup ceph since I am not root 2024-09-16T00:43:36.481 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-mds[65598]: -- 172.21.15.60:0/2875758853 <== mon.2 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d6306f5860 con 0x55d6306d3000 2024-09-16T00:43:36.481 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b[65594]: starting mds.b at 2024-09-16T00:43:36.481 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:43:35 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-h[65278]: ignoring --setuser ceph since I am not root 2024-09-16T00:43:36.481 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:43:35 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-h[65278]: ignoring --setgroup ceph since I am not root 2024-09-16T00:43:36.481 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:43:35 smithi060 ceph-mds[65282]: -- 172.21.15.60:0/541464256 <== mon.1 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561e507c1860 con 0x561e5079e800 2024-09-16T00:43:36.481 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-h[65278]: starting mds.h at 2024-09-16T00:43:36.481 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-e[64974]: ignoring --setuser ceph since I am not root 2024-09-16T00:43:36.481 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-e[64974]: ignoring --setgroup ceph since I am not root 2024-09-16T00:43:36.482 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-mds[64978]: -- 172.21.15.60:0/3769904946 <== mon.2 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558811ad0680 con 0x558811aa5400 2024-09-16T00:43:36.482 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-mds[64978]: -- 172.21.15.60:0/3769904946 <== mon.2 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558811ac7860 con 0x558811aa5400 2024-09-16T00:43:36.482 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-e[64974]: starting mds.e at 2024-09-16T00:43:36.684 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T00:43:36.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3517599718' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-16T00:43:36.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-mon[35242]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-16T00:43:36.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-mon[35242]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T00:43:36.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3517599718' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-16T00:43:36.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-mon[35242]: osdmap e91: 12 total, 12 up, 12 in 2024-09-16T00:43:36.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:36 smithi083 ceph-mon[35242]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-09-16T00:43:36.941 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:36.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3517599718' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-16T00:43:36.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-mon[35185]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-16T00:43:36.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-mon[35185]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T00:43:36.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3517599718' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-16T00:43:36.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-mon[35185]: osdmap e91: 12 total, 12 up, 12 in 2024-09-16T00:43:36.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:36 smithi060 ceph-mon[35185]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-09-16T00:43:37.064 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:36 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3517599718' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-16T00:43:37.064 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:36 smithi033 ceph-mon[31243]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-16T00:43:37.064 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:36 smithi033 ceph-mon[31243]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T00:43:37.064 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:36 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3517599718' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-16T00:43:37.064 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:36 smithi033 ceph-mon[31243]: osdmap e91: 12 total, 12 up, 12 in 2024-09-16T00:43:37.064 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:36 smithi033 ceph-mon[31243]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-09-16T00:43:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.418+0000 7fac547e0640 1 -- 172.21.15.33:0/590268693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac4c101420 msgr2=0x7fac4c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.418+0000 7fac547e0640 1 --2- 172.21.15.33:0/590268693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac4c101420 0x7fac4c10d950 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009cd0 tx=0x7fac3c02a8e0 comp rx=0 tx=0).stop 2024-09-16T00:43:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.418+0000 7fac547e0640 1 -- 172.21.15.33:0/590268693 shutdown_connections 2024-09-16T00:43:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.418+0000 7fac547e0640 1 --2- 172.21.15.33:0/590268693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac4c101420 0x7fac4c10d950 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.418+0000 7fac547e0640 1 --2- 172.21.15.33:0/590268693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac4c100a80 0x7fac4c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.418+0000 7fac547e0640 1 --2- 172.21.15.33:0/590268693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac4c106a80 0x7fac4c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.418+0000 7fac547e0640 1 -- 172.21.15.33:0/590268693 >> 172.21.15.33:0/590268693 conn(0x7fac4c0fc7a0 msgr2=0x7fac4c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.418+0000 7fac547e0640 1 -- 172.21.15.33:0/590268693 shutdown_connections 2024-09-16T00:43:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.418+0000 7fac547e0640 1 -- 172.21.15.33:0/590268693 wait complete. 2024-09-16T00:43:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.418+0000 7fac547e0640 1 Processor -- start 2024-09-16T00:43:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.418+0000 7fac547e0640 1 -- start start 2024-09-16T00:43:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac547e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac4c100a80 0x7fac4c196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac547e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac4c101420 0x7fac4c1974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac547e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac4c106a80 0x7fac4c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac547e0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c110e80 con 0x7fac4c106a80 2024-09-16T00:43:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac547e0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c110d00 con 0x7fac4c101420 2024-09-16T00:43:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac547e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c111000 con 0x7fac4c100a80 2024-09-16T00:43:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac52d7d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac4c106a80 0x7fac4c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac52d7d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac4c106a80 0x7fac4c197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38596/0 (socket says 172.21.15.33:38596) 2024-09-16T00:43:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac52d7d640 1 -- 172.21.15.33:0/2958011935 learned_addr learned my addr 172.21.15.33:0/2958011935 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac5257c640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac4c100a80 0x7fac4c196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac51d7b640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac4c101420 0x7fac4c1974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac52d7d640 1 -- 172.21.15.33:0/2958011935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac4c100a80 msgr2=0x7fac4c196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac52d7d640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac4c100a80 0x7fac4c196fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac52d7d640 1 -- 172.21.15.33:0/2958011935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac4c101420 msgr2=0x7fac4c1974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac52d7d640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac4c101420 0x7fac4c1974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac52d7d640 1 -- 172.21.15.33:0/2958011935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c1a6820 con 0x7fac4c106a80 2024-09-16T00:43:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac52d7d640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac4c106a80 0x7fac4c197a30 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fac3c005cc0 tx=0x7fac3c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac4b7fe640 1 -- 172.21.15.33:0/2958011935 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c004040 con 0x7fac4c106a80 2024-09-16T00:43:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac4b7fe640 1 -- 172.21.15.33:0/2958011935 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac3c0046c0 con 0x7fac4c106a80 2024-09-16T00:43:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.419+0000 7fac547e0640 1 -- 172.21.15.33:0/2958011935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac4c1a6ab0 con 0x7fac4c106a80 2024-09-16T00:43:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.420+0000 7fac4b7fe640 1 -- 172.21.15.33:0/2958011935 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c030e70 con 0x7fac4c106a80 2024-09-16T00:43:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.420+0000 7fac547e0640 1 -- 172.21.15.33:0/2958011935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac4c1a6ff0 con 0x7fac4c106a80 2024-09-16T00:43:37.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.420+0000 7fac547e0640 1 -- 172.21.15.33:0/2958011935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac4c100ee0 con 0x7fac4c106a80 2024-09-16T00:43:37.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.422+0000 7fac4b7fe640 1 -- 172.21.15.33:0/2958011935 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fac3c0305a0 con 0x7fac4c106a80 2024-09-16T00:43:37.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.422+0000 7fac4b7fe640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac340a3240 0x7fac340a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:37.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.423+0000 7fac5257c640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac340a3240 0x7fac340a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:37.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.423+0000 7fac4b7fe640 1 -- 172.21.15.33:0/2958011935 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9640+0+0 (secure 0 0 0) 0x7fac3c02e030 con 0x7fac4c106a80 2024-09-16T00:43:37.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.423+0000 7fac5257c640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac340a3240 0x7fac340a5700 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fac4400ab60 tx=0x7fac4400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:37.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.426+0000 7fac4b7fe640 1 -- 172.21.15.33:0/2958011935 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fac3c0bdc90 con 0x7fac4c106a80 2024-09-16T00:43:37.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.638+0000 7fac547e0640 1 -- 172.21.15.33:0/2958011935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fac4c1a7350 con 0x7fac4c106a80 2024-09-16T00:43:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: pgmap v1145: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.5 KiB/s wr, 11 op/s 2024-09-16T00:43:37.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.60:6834/3132658191,v1:172.21.15.60:6835/3132658191] up:boot 2024-09-16T00:43:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.60:6836/1256041478,v1:172.21.15.60:6837/1256041478] up:boot 2024-09-16T00:43:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.60:6838/3888934231,v1:172.21.15.60:6839/3888934231] up:boot 2024-09-16T00:43:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.83:6834/2206499551,v1:172.21.15.83:6837/2206499551] up:boot 2024-09-16T00:43:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.83:6835/3190997871,v1:172.21.15.83:6838/3190997871] up:boot 2024-09-16T00:43:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: mds.? [v2:172.21.15.83:6836/1233007750,v1:172.21.15.83:6839/1233007750] up:boot 2024-09-16T00:43:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-09-16T00:43:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T00:43:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T00:43:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T00:43:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T00:43:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T00:43:37.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T00:43:37.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:37 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2958011935' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T00:43:37.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.960+0000 7fac4b7fe640 1 -- 172.21.15.33:0/2958011935 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) ==== 104+0+0 (secure 0 0 0) 0x7fac3c0098b0 con 0x7fac4c106a80 2024-09-16T00:43:37.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: pgmap v1145: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.5 KiB/s wr, 11 op/s 2024-09-16T00:43:37.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.60:6834/3132658191,v1:172.21.15.60:6835/3132658191] up:boot 2024-09-16T00:43:37.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.60:6836/1256041478,v1:172.21.15.60:6837/1256041478] up:boot 2024-09-16T00:43:37.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.60:6838/3888934231,v1:172.21.15.60:6839/3888934231] up:boot 2024-09-16T00:43:37.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.83:6834/2206499551,v1:172.21.15.83:6837/2206499551] up:boot 2024-09-16T00:43:37.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.83:6835/3190997871,v1:172.21.15.83:6838/3190997871] up:boot 2024-09-16T00:43:37.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: mds.? [v2:172.21.15.83:6836/1233007750,v1:172.21.15.83:6839/1233007750] up:boot 2024-09-16T00:43:37.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-09-16T00:43:37.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T00:43:37.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T00:43:37.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T00:43:37.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T00:43:37.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T00:43:37.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T00:43:37.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:37 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2958011935' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T00:43:37.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.963+0000 7fac547e0640 1 -- 172.21.15.33:0/2958011935 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac340a3240 msgr2=0x7fac340a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:37.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.963+0000 7fac547e0640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac340a3240 0x7fac340a5700 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fac4400ab60 tx=0x7fac4400a2d0 comp rx=0 tx=0).stop 2024-09-16T00:43:37.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.963+0000 7fac547e0640 1 -- 172.21.15.33:0/2958011935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac4c106a80 msgr2=0x7fac4c197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:37.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.963+0000 7fac547e0640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac4c106a80 0x7fac4c197a30 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fac3c005cc0 tx=0x7fac3c003970 comp rx=0 tx=0).stop 2024-09-16T00:43:37.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.963+0000 7fac547e0640 1 -- 172.21.15.33:0/2958011935 shutdown_connections 2024-09-16T00:43:37.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.963+0000 7fac547e0640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7fac340a3240 0x7fac340a5700 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:37.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.963+0000 7fac547e0640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac4c106a80 0x7fac4c197a30 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:37.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.963+0000 7fac547e0640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac4c101420 0x7fac4c1974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:37.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.963+0000 7fac547e0640 1 --2- 172.21.15.33:0/2958011935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac4c100a80 0x7fac4c196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:37.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.963+0000 7fac547e0640 1 -- 172.21.15.33:0/2958011935 >> 172.21.15.33:0/2958011935 conn(0x7fac4c0fc7a0 msgr2=0x7fac4c0fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:37.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.963+0000 7fac547e0640 1 -- 172.21.15.33:0/2958011935 shutdown_connections 2024-09-16T00:43:37.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:37.963+0000 7fac547e0640 1 -- 172.21.15.33:0/2958011935 wait complete. 2024-09-16T00:43:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: pgmap v1145: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.5 KiB/s wr, 11 op/s 2024-09-16T00:43:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.60:6834/3132658191,v1:172.21.15.60:6835/3132658191] up:boot 2024-09-16T00:43:37.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.60:6836/1256041478,v1:172.21.15.60:6837/1256041478] up:boot 2024-09-16T00:43:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.60:6838/3888934231,v1:172.21.15.60:6839/3888934231] up:boot 2024-09-16T00:43:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.83:6834/2206499551,v1:172.21.15.83:6837/2206499551] up:boot 2024-09-16T00:43:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.83:6835/3190997871,v1:172.21.15.83:6838/3190997871] up:boot 2024-09-16T00:43:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: mds.? [v2:172.21.15.83:6836/1233007750,v1:172.21.15.83:6839/1233007750] up:boot 2024-09-16T00:43:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-09-16T00:43:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T00:43:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T00:43:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T00:43:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T00:43:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T00:43:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T00:43:37.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:37 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2958011935' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T00:43:38.715 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T00:43:38.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:38 smithi083 ceph-mon[35242]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-16T00:43:38.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:38 smithi083 ceph-mon[35242]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-16T00:43:38.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:38 smithi083 ceph-mon[35242]: Cluster is now healthy 2024-09-16T00:43:38.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:38 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/2958011935' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-16T00:43:38.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:38 smithi083 ceph-mon[35242]: fsmap 12 up:standby 2024-09-16T00:43:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:38 smithi060 ceph-mon[35185]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-16T00:43:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:38 smithi060 ceph-mon[35185]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-16T00:43:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:38 smithi060 ceph-mon[35185]: Cluster is now healthy 2024-09-16T00:43:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:38 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/2958011935' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-16T00:43:38.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:38 smithi060 ceph-mon[35185]: fsmap 12 up:standby 2024-09-16T00:43:38.990 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:39.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:38 smithi033 ceph-mon[31243]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-16T00:43:39.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:38 smithi033 ceph-mon[31243]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-16T00:43:39.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:38 smithi033 ceph-mon[31243]: Cluster is now healthy 2024-09-16T00:43:39.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:38 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/2958011935' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-16T00:43:39.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:38 smithi033 ceph-mon[31243]: fsmap 12 up:standby 2024-09-16T00:43:39.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.554+0000 7f1320e66640 1 -- 172.21.15.33:0/865278847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f131c101dd0 msgr2=0x7f131c1021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:39.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.554+0000 7f1320e66640 1 --2- 172.21.15.33:0/865278847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f131c101dd0 0x7f131c1021b0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f130c009900 tx=0x7f130c02a410 comp rx=0 tx=0).stop 2024-09-16T00:43:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.554+0000 7f1320e66640 1 -- 172.21.15.33:0/865278847 shutdown_connections 2024-09-16T00:43:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.554+0000 7f1320e66640 1 --2- 172.21.15.33:0/865278847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f131c106f00 0x7f131c10ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.554+0000 7f1320e66640 1 --2- 172.21.15.33:0/865278847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f131c102780 0x7f131c1067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.554+0000 7f1320e66640 1 --2- 172.21.15.33:0/865278847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f131c101dd0 0x7f131c1021b0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.554+0000 7f1320e66640 1 -- 172.21.15.33:0/865278847 >> 172.21.15.33:0/865278847 conn(0x7f131c0fd8b0 msgr2=0x7f131c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.554+0000 7f1320e66640 1 -- 172.21.15.33:0/865278847 shutdown_connections 2024-09-16T00:43:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.554+0000 7f1320e66640 1 -- 172.21.15.33:0/865278847 wait complete. 2024-09-16T00:43:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.555+0000 7f1320e66640 1 Processor -- start 2024-09-16T00:43:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.555+0000 7f1320e66640 1 -- start start 2024-09-16T00:43:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.555+0000 7f1320e66640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f131c101dd0 0x7f131c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.555+0000 7f1320e66640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f131c102780 0x7f131c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:39.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.555+0000 7f1320e66640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f131c106f00 0x7f131c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.555+0000 7f1320e66640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f131c111f60 con 0x7f131c101dd0 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.555+0000 7f1320e66640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f131c111de0 con 0x7f131c106f00 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.555+0000 7f1320e66640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f131c1120e0 con 0x7f131c102780 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.555+0000 7f131a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f131c101dd0 0x7f131c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.555+0000 7f131a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f131c101dd0 0x7f131c198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38604/0 (socket says 172.21.15.33:38604) 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f131a59c640 1 -- 172.21.15.33:0/3097422896 learned_addr learned my addr 172.21.15.33:0/3097422896 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f1319d9b640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f131c102780 0x7f131c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f131ad9d640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f131c106f00 0x7f131c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f131a59c640 1 -- 172.21.15.33:0/3097422896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f131c102780 msgr2=0x7f131c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f131a59c640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f131c102780 0x7f131c198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f131a59c640 1 -- 172.21.15.33:0/3097422896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f131c106f00 msgr2=0x7f131c19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f131a59c640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f131c106f00 0x7f131c19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:39.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f131a59c640 1 -- 172.21.15.33:0/3097422896 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131c19d3c0 con 0x7f131c101dd0 2024-09-16T00:43:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f131a59c640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f131c101dd0 0x7f131c198410 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f130c006010 tx=0x7f130c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f130b7fe640 1 -- 172.21.15.33:0/3097422896 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c03f030 con 0x7f131c101dd0 2024-09-16T00:43:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f1320e66640 1 -- 172.21.15.33:0/3097422896 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f131c19d650 con 0x7f131c101dd0 2024-09-16T00:43:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f130b7fe640 1 -- 172.21.15.33:0/3097422896 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f130c003ce0 con 0x7f131c101dd0 2024-09-16T00:43:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f130b7fe640 1 -- 172.21.15.33:0/3097422896 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c035070 con 0x7f131c101dd0 2024-09-16T00:43:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.556+0000 7f1320e66640 1 -- 172.21.15.33:0/3097422896 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f131c1a4f90 con 0x7f131c101dd0 2024-09-16T00:43:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.557+0000 7f1320e66640 1 -- 172.21.15.33:0/3097422896 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f131c1032b0 con 0x7f131c101dd0 2024-09-16T00:43:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.558+0000 7f130b7fe640 1 -- 172.21.15.33:0/3097422896 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f130c042030 con 0x7f131c101dd0 2024-09-16T00:43:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.559+0000 7f130b7fe640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f13040a3240 0x7f13040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.559+0000 7f130b7fe640 1 -- 172.21.15.33:0/3097422896 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9640+0+0 (secure 0 0 0) 0x7f130c0f03c0 con 0x7f131c101dd0 2024-09-16T00:43:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.559+0000 7f1319d9b640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f13040a3240 0x7f13040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:39.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.560+0000 7f1319d9b640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f13040a3240 0x7f13040a5700 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f131c0fdcd0 tx=0x7f131000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:39.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.563+0000 7f130b7fe640 1 -- 172.21.15.33:0/3097422896 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f130c0bc5d0 con 0x7f131c101dd0 2024-09-16T00:43:39.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:39.750+0000 7f1320e66640 1 -- 172.21.15.33:0/3097422896 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f131c10a0e0 con 0x7f131c101dd0 2024-09-16T00:43:39.751 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:39 smithi033 ceph-mon[31243]: pgmap v1146: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.9 KiB/s wr, 8 op/s 2024-09-16T00:43:39.751 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:39 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:43:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:39 smithi083 ceph-mon[35242]: pgmap v1146: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.9 KiB/s wr, 8 op/s 2024-09-16T00:43:39.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:39 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:43:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:39 smithi060 ceph-mon[35185]: pgmap v1146: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.9 KiB/s wr, 8 op/s 2024-09-16T00:43:39.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:39 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T00:43:40.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.668+0000 7f130b7fe640 1 -- 172.21.15.33:0/3097422896 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v92) ==== 181+0+0 (secure 0 0 0) 0x7f130c0c08d0 con 0x7f131c101dd0 2024-09-16T00:43:40.669 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_metadata' removed 2024-09-16T00:43:40.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.671+0000 7f1320e66640 1 -- 172.21.15.33:0/3097422896 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f13040a3240 msgr2=0x7f13040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:40.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.671+0000 7f1320e66640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f13040a3240 0x7f13040a5700 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f131c0fdcd0 tx=0x7f131000b390 comp rx=0 tx=0).stop 2024-09-16T00:43:40.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.671+0000 7f1320e66640 1 -- 172.21.15.33:0/3097422896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f131c101dd0 msgr2=0x7f131c198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:40.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.671+0000 7f1320e66640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f131c101dd0 0x7f131c198410 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f130c006010 tx=0x7f130c0042f0 comp rx=0 tx=0).stop 2024-09-16T00:43:40.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.673+0000 7f1320e66640 1 -- 172.21.15.33:0/3097422896 shutdown_connections 2024-09-16T00:43:40.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.673+0000 7f1320e66640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f13040a3240 0x7f13040a5700 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:40.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.673+0000 7f1320e66640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f131c106f00 0x7f131c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:40.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.673+0000 7f1320e66640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f131c102780 0x7f131c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:40.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.673+0000 7f1320e66640 1 --2- 172.21.15.33:0/3097422896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f131c101dd0 0x7f131c198410 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:40.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.673+0000 7f1320e66640 1 -- 172.21.15.33:0/3097422896 >> 172.21.15.33:0/3097422896 conn(0x7f131c0fd8b0 msgr2=0x7f131c0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.674+0000 7f1320e66640 1 -- 172.21.15.33:0/3097422896 shutdown_connections 2024-09-16T00:43:40.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:40.675+0000 7f1320e66640 1 -- 172.21.15.33:0/3097422896 wait complete. 2024-09-16T00:43:40.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:40 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3097422896' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T00:43:40.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:40 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3097422896' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T00:43:41.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:40 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3097422896' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T00:43:41.553 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T00:43:41.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:41 smithi033 ceph-mon[31243]: pgmap v1147: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 614 B/s wr, 6 op/s 2024-09-16T00:43:41.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:41 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3097422896' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T00:43:41.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:41 smithi033 ceph-mon[31243]: osdmap e92: 12 total, 12 up, 12 in 2024-09-16T00:43:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:41 smithi060 ceph-mon[35185]: pgmap v1147: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 614 B/s wr, 6 op/s 2024-09-16T00:43:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:41 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3097422896' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T00:43:41.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:41 smithi060 ceph-mon[35185]: osdmap e92: 12 total, 12 up, 12 in 2024-09-16T00:43:42.122 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:42.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:41 smithi083 ceph-mon[35242]: pgmap v1147: 129 pgs: 129 active+clean; 514 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 614 B/s wr, 6 op/s 2024-09-16T00:43:42.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:41 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3097422896' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T00:43:42.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:41 smithi083 ceph-mon[35242]: osdmap e92: 12 total, 12 up, 12 in 2024-09-16T00:43:42.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.664+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/2485694809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bc8100e20 msgr2=0x7f8bc8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:42.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.664+0000 7f8bcfb3e640 1 --2- 172.21.15.33:0/2485694809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bc8100e20 0x7f8bc8101200 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4009ca0 tx=0x7f8bc402a950 comp rx=0 tx=0).stop 2024-09-16T00:43:42.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.664+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/2485694809 shutdown_connections 2024-09-16T00:43:42.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.664+0000 7f8bcfb3e640 1 --2- 172.21.15.33:0/2485694809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc8102ef0 0x7f8bc810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:42.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.664+0000 7f8bcfb3e640 1 --2- 172.21.15.33:0/2485694809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc8102570 0x7f8bc81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:42.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.664+0000 7f8bcfb3e640 1 --2- 172.21.15.33:0/2485694809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bc8100e20 0x7f8bc8101200 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:42.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.664+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/2485694809 >> 172.21.15.33:0/2485694809 conn(0x7f8bc80fc7d0 msgr2=0x7f8bc80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:42.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.664+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/2485694809 shutdown_connections 2024-09-16T00:43:42.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.664+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/2485694809 wait complete. 2024-09-16T00:43:42.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.665+0000 7f8bcfb3e640 1 Processor -- start 2024-09-16T00:43:42.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.665+0000 7f8bcfb3e640 1 -- start start 2024-09-16T00:43:42.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.665+0000 7f8bcfb3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bc8100e20 0x7f8bc8198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:42.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.665+0000 7f8bcfb3e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc8102570 0x7f8bc8198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:42.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.665+0000 7f8bcfb3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc8102ef0 0x7f8bc819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:42.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcfb3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc8110f80 con 0x7f8bc8100e20 2024-09-16T00:43:42.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcfb3e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc8110e00 con 0x7f8bc8102570 2024-09-16T00:43:42.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcfb3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc8111100 con 0x7f8bc8102ef0 2024-09-16T00:43:42.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcd8da640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bc8100e20 0x7f8bc8198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:42.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcd8da640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bc8100e20 0x7f8bc8198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56226/0 (socket says 172.21.15.33:56226) 2024-09-16T00:43:42.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcd8da640 1 -- 172.21.15.33:0/1548997211 learned_addr learned my addr 172.21.15.33:0/1548997211 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:42.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcd0d9640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc8102570 0x7f8bc8198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:42.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bce0db640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc8102ef0 0x7f8bc819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:42.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcd8da640 1 -- 172.21.15.33:0/1548997211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc8102ef0 msgr2=0x7f8bc819ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:42.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcd8da640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc8102ef0 0x7f8bc819ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:42.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcd8da640 1 -- 172.21.15.33:0/1548997211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc8102570 msgr2=0x7f8bc8198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:42.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcd8da640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc8102570 0x7f8bc8198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:42.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcd8da640 1 -- 172.21.15.33:0/1548997211 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc819d530 con 0x7f8bc8100e20 2024-09-16T00:43:42.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bce0db640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc8102ef0 0x7f8bc819ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:43:42.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcd0d9640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc8102570 0x7f8bc8198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T00:43:42.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bcd8da640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bc8100e20 0x7f8bc8198580 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4005cc0 tx=0x7f8bc402f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:42.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.666+0000 7f8bbeffd640 1 -- 172.21.15.33:0/1548997211 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4034040 con 0x7f8bc8100e20 2024-09-16T00:43:42.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.667+0000 7f8bbeffd640 1 -- 172.21.15.33:0/1548997211 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bc4030920 con 0x7f8bc8100e20 2024-09-16T00:43:42.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.667+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/1548997211 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bc819d7c0 con 0x7f8bc8100e20 2024-09-16T00:43:42.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.667+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/1548997211 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bc81a5100 con 0x7f8bc8100e20 2024-09-16T00:43:42.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.667+0000 7f8bbeffd640 1 -- 172.21.15.33:0/1548997211 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4030d20 con 0x7f8bc8100e20 2024-09-16T00:43:42.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.668+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/1548997211 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bc8101200 con 0x7f8bc8100e20 2024-09-16T00:43:42.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.669+0000 7f8bbeffd640 1 -- 172.21.15.33:0/1548997211 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8bc4037020 con 0x7f8bc8100e20 2024-09-16T00:43:42.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.670+0000 7f8bbeffd640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8bb40a3240 0x7f8bb40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:42.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.670+0000 7f8bcd0d9640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8bb40a3240 0x7f8bb40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:42.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.670+0000 7f8bbeffd640 1 -- 172.21.15.33:0/1548997211 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9028+0+0 (secure 0 0 0) 0x7f8bc40f0d40 con 0x7f8bc8100e20 2024-09-16T00:43:42.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.671+0000 7f8bcd0d9640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8bb40a3240 0x7f8bb40a5700 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80ffd80 tx=0x7f8bb8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:42.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.673+0000 7f8bbeffd640 1 -- 172.21.15.33:0/1548997211 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8bc40bd740 con 0x7f8bc8100e20 2024-09-16T00:43:42.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:42.856+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/1548997211 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8bc81a52d0 con 0x7f8bc8100e20 2024-09-16T00:43:43.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.678+0000 7f8bbeffd640 1 -- 172.21.15.33:0/1548997211 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v93) ==== 169+0+0 (secure 0 0 0) 0x7f8bc4043e40 con 0x7f8bc8100e20 2024-09-16T00:43:43.679 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_data' removed 2024-09-16T00:43:43.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.680+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/1548997211 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8bb40a3240 msgr2=0x7f8bb40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:43.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.680+0000 7f8bcfb3e640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8bb40a3240 0x7f8bb40a5700 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80ffd80 tx=0x7f8bb8005f90 comp rx=0 tx=0).stop 2024-09-16T00:43:43.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.680+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/1548997211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bc8100e20 msgr2=0x7f8bc8198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:43.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.680+0000 7f8bcfb3e640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bc8100e20 0x7f8bc8198580 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4005cc0 tx=0x7f8bc402f420 comp rx=0 tx=0).stop 2024-09-16T00:43:43.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.681+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/1548997211 shutdown_connections 2024-09-16T00:43:43.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.681+0000 7f8bcfb3e640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7f8bb40a3240 0x7f8bb40a5700 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:43.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.681+0000 7f8bcfb3e640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc8102ef0 0x7f8bc819ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:43.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.681+0000 7f8bcfb3e640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc8102570 0x7f8bc8198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:43.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.681+0000 7f8bcfb3e640 1 --2- 172.21.15.33:0/1548997211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bc8100e20 0x7f8bc8198580 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:43.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.681+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/1548997211 >> 172.21.15.33:0/1548997211 conn(0x7f8bc80fc7d0 msgr2=0x7f8bc806a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:43.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.681+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/1548997211 shutdown_connections 2024-09-16T00:43:43.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:43.681+0000 7f8bcfb3e640 1 -- 172.21.15.33:0/1548997211 wait complete. 2024-09-16T00:43:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:43 smithi060 ceph-mon[35185]: pgmap v1149: 97 pgs: 97 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:43:43.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:43 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1548997211' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T00:43:43.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:43:43.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:43:43.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:43:43.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:43:43.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:43 smithi060 ceph-mon[35185]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:43:44.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:43 smithi033 ceph-mon[31243]: pgmap v1149: 97 pgs: 97 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:43:44.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:43 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1548997211' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T00:43:44.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:43:44.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:43:44.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:43:44.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:43:44.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:43 smithi033 ceph-mon[31243]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:43:44.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:43 smithi083 ceph-mon[35242]: pgmap v1149: 97 pgs: 97 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:43:44.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:43 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1548997211' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T00:43:44.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:43:44.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:43:44.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T00:43:44.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T00:43:44.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:43 smithi083 ceph-mon[35242]: from='mgr.14150 172.21.15.33:0/781755715' entity='mgr.x' 2024-09-16T00:43:44.731 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-09-16T00:43:44.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:44 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/1548997211' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T00:43:44.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:44 smithi060 ceph-mon[35185]: osdmap e93: 12 total, 12 up, 12 in 2024-09-16T00:43:44.994 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T00:43:45.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:44 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/1548997211' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T00:43:45.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:44 smithi033 ceph-mon[31243]: osdmap e93: 12 total, 12 up, 12 in 2024-09-16T00:43:45.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:44 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/1548997211' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T00:43:45.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:44 smithi083 ceph-mon[35242]: osdmap e93: 12 total, 12 up, 12 in 2024-09-16T00:43:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.744+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/1012786341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbc0102ef0 msgr2=0x7ffbc010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.744+0000 7ffbc5a0e640 1 --2- 172.21.15.33:0/1012786341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbc0102ef0 0x7ffbc010da40 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009ca0 tx=0x7ffbb802a950 comp rx=0 tx=0).stop 2024-09-16T00:43:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.744+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/1012786341 shutdown_connections 2024-09-16T00:43:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.744+0000 7ffbc5a0e640 1 --2- 172.21.15.33:0/1012786341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbc0102ef0 0x7ffbc010da40 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.744+0000 7ffbc5a0e640 1 --2- 172.21.15.33:0/1012786341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc0102570 0x7ffbc01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.744+0000 7ffbc5a0e640 1 --2- 172.21.15.33:0/1012786341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbc0100e20 0x7ffbc0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.744+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/1012786341 >> 172.21.15.33:0/1012786341 conn(0x7ffbc00fc7d0 msgr2=0x7ffbc00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.744+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/1012786341 shutdown_connections 2024-09-16T00:43:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.744+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/1012786341 wait complete. 2024-09-16T00:43:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.744+0000 7ffbc5a0e640 1 Processor -- start 2024-09-16T00:43:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbc5a0e640 1 -- start start 2024-09-16T00:43:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbc5a0e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc0100e20 0x7ffbc0198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbc5a0e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbc0102570 0x7ffbc0198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbc5a0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbc0102ef0 0x7ffbc019ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbc5a0e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01120a0 con 0x7ffbc0102570 2024-09-16T00:43:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbc5a0e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc0111f20 con 0x7ffbc0100e20 2024-09-16T00:43:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbc5a0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc0112220 con 0x7ffbc0102ef0 2024-09-16T00:43:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbbe7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbc0102570 0x7ffbc0198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbbe7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbc0102570 0x7ffbc0198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56240/0 (socket says 172.21.15.33:56240) 2024-09-16T00:43:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbbe7fc640 1 -- 172.21.15.33:0/3759016518 learned_addr learned my addr 172.21.15.33:0/3759016518 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-16T00:43:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbbf7fe640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbc0102ef0 0x7ffbc019ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbbeffd640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc0100e20 0x7ffbc0198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbbe7fc640 1 -- 172.21.15.33:0/3759016518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbc0102ef0 msgr2=0x7ffbc019ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbbe7fc640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbc0102ef0 0x7ffbc019ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbbe7fc640 1 -- 172.21.15.33:0/3759016518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc0100e20 msgr2=0x7ffbc0198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbbe7fc640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc0100e20 0x7ffbc0198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbbe7fc640 1 -- 172.21.15.33:0/3759016518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc019d3d0 con 0x7ffbc0102570 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.745+0000 7ffbbeffd640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc0100e20 0x7ffbc0198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.746+0000 7ffbbe7fc640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbc0102570 0x7ffbc0198960 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400c920 tx=0x7ffbb400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.746+0000 7ffbc4a0c640 1 -- 172.21.15.33:0/3759016518 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4007c70 con 0x7ffbc0102570 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.746+0000 7ffbc4a0c640 1 -- 172.21.15.33:0/3759016518 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbb4007e10 con 0x7ffbc0102570 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.746+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/3759016518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc019d6c0 con 0x7ffbc0102570 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.746+0000 7ffbc4a0c640 1 -- 172.21.15.33:0/3759016518 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4018890 con 0x7ffbc0102570 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.746+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/3759016518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc01a4fa0 con 0x7ffbc0102570 2024-09-16T00:43:45.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.747+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/3759016518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbc0101200 con 0x7ffbc0102570 2024-09-16T00:43:45.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.748+0000 7ffbc4a0c640 1 -- 172.21.15.33:0/3759016518 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ffbb4018a50 con 0x7ffbc0102570 2024-09-16T00:43:45.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.749+0000 7ffbc4a0c640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffba80a3240 0x7ffba80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T00:43:45.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.749+0000 7ffbbeffd640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffba80a3240 0x7ffba80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T00:43:45.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.749+0000 7ffbc4a0c640 1 -- 172.21.15.33:0/3759016518 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8324+0+0 (secure 0 0 0) 0x7ffbb40cfe10 con 0x7ffbc0102570 2024-09-16T00:43:45.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.749+0000 7ffbbeffd640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffba80a3240 0x7ffba80a5700 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009b40 tx=0x7ffbb0005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T00:43:45.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.752+0000 7ffbc4a0c640 1 -- 172.21.15.33:0/3759016518 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffbb4014030 con 0x7ffbc0102570 2024-09-16T00:43:45.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:45.929+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/3759016518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7ffbc01a5170 con 0x7ffbc0102570 2024-09-16T00:43:45.930 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:45 smithi033 ceph-mon[31243]: pgmap v1151: 33 pgs: 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:43:45.981 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:45 smithi060 ceph-mon[35185]: pgmap v1151: 33 pgs: 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:43:46.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:45 smithi083 ceph-mon[35242]: pgmap v1151: 33 pgs: 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:43:46.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.702+0000 7ffbc4a0c640 1 -- 172.21.15.33:0/3759016518 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v94) ==== 178+0+0 (secure 0 0 0) 0x7ffbb409cb60 con 0x7ffbc0102570 2024-09-16T00:43:46.703 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_data_ec' removed 2024-09-16T00:43:46.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.705+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/3759016518 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffba80a3240 msgr2=0x7ffba80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:46.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.705+0000 7ffbc5a0e640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffba80a3240 0x7ffba80a5700 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009b40 tx=0x7ffbb0005e50 comp rx=0 tx=0).stop 2024-09-16T00:43:46.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.705+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/3759016518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbc0102570 msgr2=0x7ffbc0198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T00:43:46.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.705+0000 7ffbc5a0e640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbc0102570 0x7ffbc0198960 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400c920 tx=0x7ffbb400cde0 comp rx=0 tx=0).stop 2024-09-16T00:43:46.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.705+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/3759016518 shutdown_connections 2024-09-16T00:43:46.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.705+0000 7ffbc5a0e640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.33:6800/3427860355,v1:172.21.15.33:6801/3427860355] conn(0x7ffba80a3240 0x7ffba80a5700 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:46.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.705+0000 7ffbc5a0e640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbc0102ef0 0x7ffbc019ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:46.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.705+0000 7ffbc5a0e640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbc0102570 0x7ffbc0198960 secure :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400c920 tx=0x7ffbb400cde0 comp rx=0 tx=0).stop 2024-09-16T00:43:46.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.705+0000 7ffbc5a0e640 1 --2- 172.21.15.33:0/3759016518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc0100e20 0x7ffbc0198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T00:43:46.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.705+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/3759016518 >> 172.21.15.33:0/3759016518 conn(0x7ffbc00fc7d0 msgr2=0x7ffbc01067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T00:43:46.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.707+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/3759016518 shutdown_connections 2024-09-16T00:43:46.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-16T00:43:46.707+0000 7ffbc5a0e640 1 -- 172.21.15.33:0/3759016518 wait complete. 2024-09-16T00:43:46.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:46 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3759016518' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T00:43:47.054 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:46 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3759016518' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T00:43:47.168 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:46 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3759016518' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T00:43:47.427 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T00:43:47.454 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T00:43:47.487 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T00:43:47.529 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-16T00:43:47.529 DEBUG:teuthology.orchestra.run.smithi033:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T00:43:47.545 DEBUG:teuthology.orchestra.run.smithi060:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T00:43:47.560 DEBUG:teuthology.orchestra.run.smithi083:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T00:43:47.584 INFO:tasks.cephadm:Stopping all daemons... 2024-09-16T00:43:47.585 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-16T00:43:47.585 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.a 2024-09-16T00:43:47.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:47 smithi033 systemd[1]: Stopping Ceph mon.a for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:43:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:47 smithi060 ceph-mon[35185]: pgmap v1152: 33 pgs: 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:43:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:47 smithi060 ceph-mon[35185]: from='client.? 172.21.15.33:0/3759016518' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T00:43:47.730 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:47 smithi060 ceph-mon[35185]: osdmap e94: 12 total, 12 up, 12 in 2024-09-16T00:43:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:47 smithi083 ceph-mon[35242]: pgmap v1152: 33 pgs: 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:43:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:47 smithi083 ceph-mon[35242]: from='client.? 172.21.15.33:0/3759016518' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T00:43:47.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:47 smithi083 ceph-mon[35242]: osdmap e94: 12 total, 12 up, 12 in 2024-09-16T00:43:48.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:47 smithi033 ceph-mon[31243]: pgmap v1152: 33 pgs: 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T00:43:48.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:47 smithi033 ceph-mon[31243]: from='client.? 172.21.15.33:0/3759016518' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T00:43:48.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:47 smithi033 ceph-mon[31243]: osdmap e94: 12 total, 12 up, 12 in 2024-09-16T00:43:48.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:47 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a[31239]: 2024-09-16T00:43:47.900+0000 7fa1dd58b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:43:48.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:47 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a[31239]: 2024-09-16T00:43:47.900+0000 7fa1dd58b640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-16T00:43:48.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:47 smithi033 podman[137002]: 2024-09-16 00:43:47.932055216 +0000 UTC m=+0.151625634 container died 831049806ec5419bdf268e2d4f962b16da7021d08b247c2c780919d4774ddace (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-16T00:43:48.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:48 smithi033 podman[137002]: 2024-09-16 00:43:48.342811705 +0000 UTC m=+0.562382121 container cleanup 831049806ec5419bdf268e2d4f962b16da7021d08b247c2c780919d4774ddace (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True) 2024-09-16T00:43:48.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:48 smithi033 bash[137002]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a 2024-09-16T00:43:48.829 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.a.service' 2024-09-16T00:43:48.853 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:48 smithi033 podman[137014]: 2024-09-16 00:43:48.586042321 +0000 UTC m=+0.652918801 container remove 831049806ec5419bdf268e2d4f962b16da7021d08b247c2c780919d4774ddace (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-16T00:43:48.853 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:48 smithi033 systemd[1]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-16T00:43:48.853 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:48 smithi033 systemd[1]: Stopped Ceph mon.a for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:43:48.853 INFO:journalctl@ceph.mon.a.smithi033.stdout:Sep 16 00:43:48 smithi033 systemd[1]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.a.service: Consumed 48.542s CPU time. 2024-09-16T00:43:48.885 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:43:48.885 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-16T00:43:48.885 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-16T00:43:48.885 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.b 2024-09-16T00:43:49.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:48 smithi060 systemd[1]: Stopping Ceph mon.b for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:43:49.973 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:49 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-b[35181]: 2024-09-16T00:43:49.870+0000 7f90624dd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:43:49.973 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:49 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-b[35181]: 2024-09-16T00:43:49.870+0000 7f90624dd640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-16T00:43:50.230 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:50 smithi060 podman[74064]: 2024-09-16 00:43:50.102981207 +0000 UTC m=+0.379076616 container died d741ec7069ef54a9f396be3774bfec01933851e43141ebaf6e336041d8dd3123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T00:43:50.766 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:50 smithi060 podman[74064]: 2024-09-16 00:43:50.506180075 +0000 UTC m=+0.782275483 container cleanup d741ec7069ef54a9f396be3774bfec01933851e43141ebaf6e336041d8dd3123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-b, GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-16T00:43:50.767 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:50 smithi060 bash[74064]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-b 2024-09-16T00:43:51.229 INFO:journalctl@ceph.mon.b.smithi060.stdout:Sep 16 00:43:50 smithi060 podman[74091]: 2024-09-16 00:43:50.766639658 +0000 UTC m=+0.680186912 container remove d741ec7069ef54a9f396be3774bfec01933851e43141ebaf6e336041d8dd3123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-b, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True) 2024-09-16T00:43:51.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.b.service' 2024-09-16T00:43:51.298 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:43:51.299 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-16T00:43:51.299 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-16T00:43:51.299 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.c 2024-09-16T00:43:51.668 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:51 smithi083 systemd[1]: Stopping Ceph mon.c for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:43:51.939 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:51 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-c[35238]: 2024-09-16T00:43:51.683+0000 7fc07dfaf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:43:51.939 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:51 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-c[35238]: 2024-09-16T00:43:51.683+0000 7fc07dfaf640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-16T00:43:52.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:51 smithi083 podman[72780]: 2024-09-16 00:43:51.939708574 +0000 UTC m=+0.368977003 container died f13645e976358456c99d33ac2ad78687cfe9b443550acdbe2952c717e37f616f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-c, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True) 2024-09-16T00:43:52.619 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:52 smithi083 podman[72780]: 2024-09-16 00:43:52.335085941 +0000 UTC m=+0.764354370 container cleanup f13645e976358456c99d33ac2ad78687cfe9b443550acdbe2952c717e37f616f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-16T00:43:52.620 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 16 00:43:52 smithi083 bash[72780]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mon-c 2024-09-16T00:43:52.749 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mon.c.service' 2024-09-16T00:43:52.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:43:52.799 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-16T00:43:52.799 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-16T00:43:52.799 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.x 2024-09-16T00:43:53.080 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:43:52 smithi033 systemd[1]: Stopping Ceph mgr.x for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:43:53.490 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:43:53 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x[31469]: 2024-09-16T00:43:53.122+0000 7f75b7c05640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:43:53.491 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:43:53 smithi033 podman[137121]: 2024-09-16 00:43:53.153823191 +0000 UTC m=+0.153516368 container died e964b53ad2d0c02ad6a4c1de177522c175e7178997af6feefa553e0241333d31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-16T00:43:53.829 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:43:53 smithi033 podman[137121]: 2024-09-16 00:43:53.606773018 +0000 UTC m=+0.606466189 container cleanup e964b53ad2d0c02ad6a4c1de177522c175e7178997af6feefa553e0241333d31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-16T00:43:53.830 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Sep 16 00:43:53 smithi033 bash[137121]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-x 2024-09-16T00:43:54.082 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-16T00:43:54.119 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:43:54.119 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-16T00:43:54.119 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-16T00:43:54.119 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.y 2024-09-16T00:43:54.480 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:43:54 smithi060 systemd[1]: Stopping Ceph mgr.y for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:43:54.779 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:43:54 smithi060 podman[76464]: 2024-09-16 00:43:54.481104029 +0000 UTC m=+0.160694106 container died 9a7609fbd20b31a3041ee24a5e1821ea2cf6b7c7752a683139cc1ba25f7f9b99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, ceph=True, org.label-schema.license=GPLv2) 2024-09-16T00:43:55.154 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:43:54 smithi060 podman[76464]: 2024-09-16 00:43:54.869770476 +0000 UTC m=+0.549360599 container cleanup 9a7609fbd20b31a3041ee24a5e1821ea2cf6b7c7752a683139cc1ba25f7f9b99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T00:43:55.154 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Sep 16 00:43:54 smithi060 bash[76464]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-y 2024-09-16T00:43:55.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-16T00:43:55.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:43:55.350 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-16T00:43:55.350 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-09-16T00:43:55.350 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.z 2024-09-16T00:43:55.668 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:43:55 smithi083 systemd[1]: Stopping Ceph mgr.z for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:43:56.107 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:43:55 smithi083 podman[72904]: 2024-09-16 00:43:55.80302485 +0000 UTC m=+0.259498917 container died 44d46e2f0d9dce3aecf338096adacf718e006a9a806c4f48bae046a12d9e1e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-16T00:43:56.418 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:43:56 smithi083 podman[72904]: 2024-09-16 00:43:56.199211448 +0000 UTC m=+0.655685515 container cleanup 44d46e2f0d9dce3aecf338096adacf718e006a9a806c4f48bae046a12d9e1e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-16T00:43:56.418 INFO:journalctl@ceph.mgr.z.smithi083.stdout:Sep 16 00:43:56 smithi083 bash[72904]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mgr-z 2024-09-16T00:43:56.633 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mgr.z.service' 2024-09-16T00:43:56.668 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:43:56.668 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-09-16T00:43:56.668 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-16T00:43:56.669 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.0 2024-09-16T00:43:56.986 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:43:56 smithi033 systemd[1]: Stopping Ceph osd.0 for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:43:57.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:43:56 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0[42929]: 2024-09-16T00:43:56.986+0000 7faceca3d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:43:57.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:43:56 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0[42929]: 2024-09-16T00:43:56.986+0000 7faceca3d640 -1 osd.0 94 *** Got signal Terminated *** 2024-09-16T00:43:57.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:43:56 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0[42929]: 2024-09-16T00:43:56.986+0000 7faceca3d640 -1 osd.0 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T00:44:02.830 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:44:02 smithi033 podman[137244]: 2024-09-16 00:44:02.563826227 +0000 UTC m=+5.695250649 container died 662c4a8efa4c3c0ea2c668df7cb8cd0fa881ec65ce3a19277d774a68e3c9f03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-16T00:44:03.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:44:02 smithi033 podman[137244]: 2024-09-16 00:44:02.978743143 +0000 UTC m=+6.110167566 container cleanup 662c4a8efa4c3c0ea2c668df7cb8cd0fa881ec65ce3a19277d774a68e3c9f03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-16T00:44:03.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:44:02 smithi033 bash[137244]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0 2024-09-16T00:44:03.755 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:44:03 smithi033 podman[137259]: 2024-09-16 00:44:03.355699789 +0000 UTC m=+0.784081711 container remove 662c4a8efa4c3c0ea2c668df7cb8cd0fa881ec65ce3a19277d774a68e3c9f03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-16T00:44:04.042 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:44:03 smithi033 podman[137351]: 2024-09-16 00:44:03.658202919 +0000 UTC m=+0.016391161 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:44:04.042 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:44:03 smithi033 podman[137351]: 2024-09-16 00:44:03.798321613 +0000 UTC m=+0.156509884 container create 1d514e95f17a32ae9d83bfd0d417af935ffbbe21714ec77f57382e9e64d17d96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e) 2024-09-16T00:44:04.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:44:04 smithi033 podman[137351]: 2024-09-16 00:44:04.042693815 +0000 UTC m=+0.400882049 container init 1d514e95f17a32ae9d83bfd0d417af935ffbbe21714ec77f57382e9e64d17d96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:44:04.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:44:04 smithi033 podman[137351]: 2024-09-16 00:44:04.046344389 +0000 UTC m=+0.404532617 container start 1d514e95f17a32ae9d83bfd0d417af935ffbbe21714ec77f57382e9e64d17d96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-16T00:44:04.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:44:04 smithi033 podman[137351]: 2024-09-16 00:44:04.142735327 +0000 UTC m=+0.500923563 container attach 1d514e95f17a32ae9d83bfd0d417af935ffbbe21714ec77f57382e9e64d17d96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-16T00:44:04.330 INFO:journalctl@ceph.osd.0.smithi033.stdout:Sep 16 00:44:04 smithi033 podman[137351]: 2024-09-16 00:44:04.242901815 +0000 UTC m=+0.601090043 container died 1d514e95f17a32ae9d83bfd0d417af935ffbbe21714ec77f57382e9e64d17d96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:44:04.922 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.0.service' 2024-09-16T00:44:04.963 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:44:04.963 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-16T00:44:04.963 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-16T00:44:04.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.1 2024-09-16T00:44:05.291 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:05 smithi060 systemd[1]: Stopping Ceph osd.1 for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:44:05.730 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:05 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1[39634]: 2024-09-16T00:44:05.291+0000 7ff6a3d8b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:44:05.730 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:05 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1[39634]: 2024-09-16T00:44:05.291+0000 7ff6a3d8b640 -1 osd.1 94 *** Got signal Terminated *** 2024-09-16T00:44:05.730 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:05 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1[39634]: 2024-09-16T00:44:05.291+0000 7ff6a3d8b640 -1 osd.1 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T00:44:11.174 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:10 smithi060 podman[76588]: 2024-09-16 00:44:10.857266711 +0000 UTC m=+5.691939894 container died e8867ba4f9d049bff120a8fa7f7689f58428586b4055adda424c80aa3401996b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-16T00:44:11.480 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:11 smithi060 podman[76588]: 2024-09-16 00:44:11.275018331 +0000 UTC m=+6.109691501 container cleanup e8867ba4f9d049bff120a8fa7f7689f58428586b4055adda424c80aa3401996b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-16T00:44:11.480 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:11 smithi060 bash[76588]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1 2024-09-16T00:44:11.934 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:11 smithi060 podman[76605]: 2024-09-16 00:44:11.526871444 +0000 UTC m=+0.668746400 container remove e8867ba4f9d049bff120a8fa7f7689f58428586b4055adda424c80aa3401996b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-16T00:44:12.222 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:11 smithi060 podman[76695]: 2024-09-16 00:44:11.836915014 +0000 UTC m=+0.016195217 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:44:12.222 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:11 smithi060 podman[76695]: 2024-09-16 00:44:11.952785058 +0000 UTC m=+0.132065294 container create e1c57a9f9627707d98997b08c32333093458fd30c3bcd0536bfc3cdf6a98bf7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-16T00:44:12.480 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:12 smithi060 podman[76695]: 2024-09-16 00:44:12.222267465 +0000 UTC m=+0.401547659 container init e1c57a9f9627707d98997b08c32333093458fd30c3bcd0536bfc3cdf6a98bf7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:44:12.480 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:12 smithi060 podman[76695]: 2024-09-16 00:44:12.225875735 +0000 UTC m=+0.405155928 container start e1c57a9f9627707d98997b08c32333093458fd30c3bcd0536bfc3cdf6a98bf7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-16T00:44:12.480 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:12 smithi060 podman[76695]: 2024-09-16 00:44:12.322228765 +0000 UTC m=+0.501508970 container attach e1c57a9f9627707d98997b08c32333093458fd30c3bcd0536bfc3cdf6a98bf7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-09-16T00:44:12.481 INFO:journalctl@ceph.osd.1.smithi060.stdout:Sep 16 00:44:12 smithi060 podman[76695]: 2024-09-16 00:44:12.416947911 +0000 UTC m=+0.596228110 container died e1c57a9f9627707d98997b08c32333093458fd30c3bcd0536bfc3cdf6a98bf7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-16T00:44:13.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.1.service' 2024-09-16T00:44:13.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:44:13.140 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-16T00:44:13.140 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-16T00:44:13.140 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.2 2024-09-16T00:44:13.454 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:13 smithi083 systemd[1]: Stopping Ceph osd.2 for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:44:13.918 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2[40190]: 2024-09-16T00:44:13.452+0000 7fa24d762640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:44:13.918 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2[40190]: 2024-09-16T00:44:13.452+0000 7fa24d762640 -1 osd.2 94 *** Got signal Terminated *** 2024-09-16T00:44:13.918 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2[40190]: 2024-09-16T00:44:13.452+0000 7fa24d762640 -1 osd.2 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T00:44:19.156 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:18 smithi083 podman[73044]: 2024-09-16 00:44:18.891877435 +0000 UTC m=+5.557286167 container died 4eebd2012a21eeaad0ac202db5913d59db44c84772a3a0dad40ec7a05b8eb8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e) 2024-09-16T00:44:19.418 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:19 smithi083 podman[73044]: 2024-09-16 00:44:19.271107749 +0000 UTC m=+5.936516473 container cleanup 4eebd2012a21eeaad0ac202db5913d59db44c84772a3a0dad40ec7a05b8eb8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:44:19.418 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:19 smithi083 bash[73044]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2 2024-09-16T00:44:19.877 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:19 smithi083 podman[73061]: 2024-09-16 00:44:19.481461321 +0000 UTC m=+0.591340159 container remove 4eebd2012a21eeaad0ac202db5913d59db44c84772a3a0dad40ec7a05b8eb8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-16T00:44:20.168 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:19 smithi083 podman[73151]: 2024-09-16 00:44:19.780628828 +0000 UTC m=+0.015950688 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:44:20.168 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:19 smithi083 podman[73151]: 2024-09-16 00:44:19.907318285 +0000 UTC m=+0.142640130 container create 2f2c89648d28cd4d40e1d11abbc8a223ee033f6ddb336d814403e4c48bc736ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:44:20.418 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:20 smithi083 podman[73151]: 2024-09-16 00:44:20.168408795 +0000 UTC m=+0.403730641 container init 2f2c89648d28cd4d40e1d11abbc8a223ee033f6ddb336d814403e4c48bc736ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:44:20.418 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:20 smithi083 podman[73151]: 2024-09-16 00:44:20.171750076 +0000 UTC m=+0.407071923 container start 2f2c89648d28cd4d40e1d11abbc8a223ee033f6ddb336d814403e4c48bc736ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-16T00:44:20.418 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:20 smithi083 podman[73151]: 2024-09-16 00:44:20.268530491 +0000 UTC m=+0.503852346 container attach 2f2c89648d28cd4d40e1d11abbc8a223ee033f6ddb336d814403e4c48bc736ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-16T00:44:20.847 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 16 00:44:20 smithi083 podman[73151]: 2024-09-16 00:44:20.525669263 +0000 UTC m=+0.760991112 container died 2f2c89648d28cd4d40e1d11abbc8a223ee033f6ddb336d814403e4c48bc736ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-16T00:44:21.152 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.2.service' 2024-09-16T00:44:21.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:44:21.193 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-16T00:44:21.193 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-16T00:44:21.193 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.3 2024-09-16T00:44:21.541 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:21 smithi033 systemd[1]: Stopping Ceph osd.3 for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:44:21.829 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:21 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3[49398]: 2024-09-16T00:44:21.540+0000 7fac07936640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:44:21.830 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:21 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3[49398]: 2024-09-16T00:44:21.540+0000 7fac07936640 -1 osd.3 94 *** Got signal Terminated *** 2024-09-16T00:44:21.830 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:21 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3[49398]: 2024-09-16T00:44:21.540+0000 7fac07936640 -1 osd.3 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T00:44:23.840 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:23 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:23.495+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:24.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:23 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:23.839+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:24.829 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:24 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3[49398]: 2024-09-16T00:44:24.544+0000 7fac03f4c640 -1 osd.3 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:58.219738+0000 front 2024-09-16T00:43:58.219768+0000 (oldest deadline 2024-09-16T00:44:24.119876+0000) 2024-09-16T00:44:24.830 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:24 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:24.461+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:25.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:24 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:24.830+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:25.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:25 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:25.500+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:25.669 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:25 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:25.359+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:25.829 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:25 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3[49398]: 2024-09-16T00:44:25.544+0000 7fac03f4c640 -1 osd.3 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:58.219738+0000 front 2024-09-16T00:43:58.219768+0000 (oldest deadline 2024-09-16T00:44:24.119876+0000) 2024-09-16T00:44:26.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:25 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:25.827+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:26.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:26.511+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:26.668 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:26.404+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:26.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:26 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:26.390+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:27.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:26.781+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:27.525 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:27 smithi033 podman[137468]: 2024-09-16 00:44:27.204114343 +0000 UTC m=+5.809402443 container died d57387010c76e624e62dd9c7160a9e91b762c3b05f8f1a767cd6589bdc6e5354 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-16T00:44:27.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:27 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:27.395+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:27.782 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:27 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:27.436+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:27.782 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:27 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:27.515+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:27.829 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:27 smithi033 podman[137468]: 2024-09-16 00:44:27.73313107 +0000 UTC m=+6.338419167 container cleanup d57387010c76e624e62dd9c7160a9e91b762c3b05f8f1a767cd6589bdc6e5354 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-16T00:44:27.829 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:27 smithi033 bash[137468]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3 2024-09-16T00:44:28.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:27 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:27.781+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:28.329 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:27 smithi033 podman[137483]: 2024-09-16 00:44:27.959716049 +0000 UTC m=+0.752394548 container remove d57387010c76e624e62dd9c7160a9e91b762c3b05f8f1a767cd6589bdc6e5354 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, ceph=True) 2024-09-16T00:44:28.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:28 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:28.362+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:28.738 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:28 smithi033 podman[137575]: 2024-09-16 00:44:28.319907782 +0000 UTC m=+0.016315545 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:44:28.738 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:28 smithi033 podman[137575]: 2024-09-16 00:44:28.444046811 +0000 UTC m=+0.140454616 container create 07720324bbabea1699b040a700da8c895d5f20b7a281d6ddc682a408b847f0aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:44:28.799 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:28 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:28.430+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:28.799 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:28 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:28.498+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:29.080 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:28 smithi033 podman[137575]: 2024-09-16 00:44:28.738324302 +0000 UTC m=+0.434732066 container init 07720324bbabea1699b040a700da8c895d5f20b7a281d6ddc682a408b847f0aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e) 2024-09-16T00:44:29.080 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:28 smithi033 podman[137575]: 2024-09-16 00:44:28.742066526 +0000 UTC m=+0.438474298 container start 07720324bbabea1699b040a700da8c895d5f20b7a281d6ddc682a408b847f0aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3-deactivate, RELEASE=squid-8921c8e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T00:44:29.080 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:28 smithi033 podman[137575]: 2024-09-16 00:44:28.854947078 +0000 UTC m=+0.551354860 container attach 07720324bbabea1699b040a700da8c895d5f20b7a281d6ddc682a408b847f0aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e) 2024-09-16T00:44:29.080 INFO:journalctl@ceph.osd.3.smithi033.stdout:Sep 16 00:44:28 smithi033 podman[137575]: 2024-09-16 00:44:28.935301777 +0000 UTC m=+0.631709541 container died 07720324bbabea1699b040a700da8c895d5f20b7a281d6ddc682a408b847f0aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=) 2024-09-16T00:44:29.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:28 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:28.799+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:29.598 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.3.service' 2024-09-16T00:44:29.644 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:44:29.644 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-16T00:44:29.644 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-16T00:44:29.644 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.4 2024-09-16T00:44:29.672 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:29 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:29.362+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:29.796 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:29 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:29.454+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:29.796 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:29 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:29.494+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:29.971 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:29 smithi060 systemd[1]: Stopping Ceph osd.4 for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:44:30.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:29 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:29.795+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:30.230 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:29 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4[46117]: 2024-09-16T00:44:29.970+0000 7f8240629640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:44:30.230 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:29 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4[46117]: 2024-09-16T00:44:29.970+0000 7f8240629640 -1 osd.4 94 *** Got signal Terminated *** 2024-09-16T00:44:30.230 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:29 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4[46117]: 2024-09-16T00:44:29.970+0000 7f8240629640 -1 osd.4 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T00:44:30.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:30 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:30.363+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:30.801 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:30 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:30.469+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:30.802 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:30 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:30.455+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:31.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:30 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:30.801+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:31.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:31 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:31.391+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:31.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:31 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:31.286+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:31.814 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:31 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:31.434+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:31.814 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:31 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:31.491+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:32.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:31 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:31.813+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:32.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:32 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:32.279+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:32.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:32 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:32.389+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:32.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:32 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:32.389+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:32.824 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:32 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:32.515+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:32.825 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:32 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:32.468+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:32.825 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:32 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:32.468+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:05.988978+0000 front 2024-09-16T00:44:05.989117+0000 (oldest deadline 2024-09-16T00:44:31.888936+0000) 2024-09-16T00:44:33.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:32 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:32.824+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:33.329 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:32 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:32.854+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:33.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:33 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:33.282+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:33.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:33 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:33.358+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:33.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:33 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:33.358+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:33.815 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:33 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:33.469+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:33.815 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:33 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:33.508+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:33.815 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:33 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:33.508+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:05.988978+0000 front 2024-09-16T00:44:05.989117+0000 (oldest deadline 2024-09-16T00:44:31.888936+0000) 2024-09-16T00:44:34.079 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:33 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:33.810+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:34.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:33 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:33.582+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:34.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:33 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:33.814+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:34.269 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:33 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4[46117]: 2024-09-16T00:44:33.981+0000 7f8238c37640 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:07.970519+0000 front 2024-09-16T00:44:07.970474+0000 (oldest deadline 2024-09-16T00:44:33.870687+0000) 2024-09-16T00:44:34.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:34 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:34.269+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:34.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:34 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:34.329+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:34.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:34 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:34.329+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:34.778 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:34 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:34.447+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:34.779 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:34 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:34.506+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:34.779 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:34 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:34.506+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:05.988978+0000 front 2024-09-16T00:44:05.989117+0000 (oldest deadline 2024-09-16T00:44:31.888936+0000) 2024-09-16T00:44:34.829 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:34 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:34.770+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:34.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:34 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:34.534+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:35.082 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:34 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:34.778+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:35.230 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:34 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4[46117]: 2024-09-16T00:44:34.962+0000 7f8238c37640 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:07.970519+0000 front 2024-09-16T00:44:07.970474+0000 (oldest deadline 2024-09-16T00:44:33.870687+0000) 2024-09-16T00:44:35.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:35 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:35.286+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:35.730 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:35 smithi060 podman[76810]: 2024-09-16 00:44:35.505111542 +0000 UTC m=+5.657512083 container died 9706d3b46d6546492260a2345aef28cffc90761f7ed15f03f42fa2a38eff1fc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:44:35.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:35 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:35.301+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:35.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:35 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:35.301+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:35.820 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:35 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:35.443+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:35.821 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:35 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:35.467+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:35.821 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:35 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:35.467+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:05.988978+0000 front 2024-09-16T00:44:05.989117+0000 (oldest deadline 2024-09-16T00:44:31.888936+0000) 2024-09-16T00:44:35.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:35 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:35.561+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:35.829 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:35 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:35.759+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:36.156 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:35 smithi060 podman[76810]: 2024-09-16 00:44:35.904392211 +0000 UTC m=+6.056792739 container cleanup 9706d3b46d6546492260a2345aef28cffc90761f7ed15f03f42fa2a38eff1fc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-09-16T00:44:36.156 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:35 smithi060 bash[76810]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4 2024-09-16T00:44:36.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:35 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:35.820+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:36.480 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:36 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:36.324+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:36.480 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:36 smithi060 podman[76824]: 2024-09-16 00:44:36.15616701 +0000 UTC m=+0.650879536 container remove 9706d3b46d6546492260a2345aef28cffc90761f7ed15f03f42fa2a38eff1fc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:44:36.480 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:36 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:36.351+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:36.481 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:36 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:36.351+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:36.776 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:36 smithi060 podman[76914]: 2024-09-16 00:44:36.422653207 +0000 UTC m=+0.015771562 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:44:36.776 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:36 smithi060 podman[76914]: 2024-09-16 00:44:36.523643153 +0000 UTC m=+0.116761506 container create 5aa3c43cc3eac30b5f7371ce168b675da436ddb492ba8edd93495da7a5fb4b85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:44:36.821 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:36.423+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:36.822 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:36.494+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:36.822 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:36.494+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:05.988978+0000 front 2024-09-16T00:44:05.989117+0000 (oldest deadline 2024-09-16T00:44:31.888936+0000) 2024-09-16T00:44:36.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:36 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:36.544+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:36.829 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:36 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:36.778+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:37.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:36 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:36.821+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:37.230 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:36 smithi060 podman[76914]: 2024-09-16 00:44:36.776294667 +0000 UTC m=+0.369413034 container init 5aa3c43cc3eac30b5f7371ce168b675da436ddb492ba8edd93495da7a5fb4b85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e) 2024-09-16T00:44:37.230 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:36 smithi060 podman[76914]: 2024-09-16 00:44:36.779524447 +0000 UTC m=+0.372642800 container start 5aa3c43cc3eac30b5f7371ce168b675da436ddb492ba8edd93495da7a5fb4b85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e) 2024-09-16T00:44:37.230 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:36 smithi060 podman[76914]: 2024-09-16 00:44:36.859643301 +0000 UTC m=+0.452761669 container attach 5aa3c43cc3eac30b5f7371ce168b675da436ddb492ba8edd93495da7a5fb4b85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e) 2024-09-16T00:44:37.230 INFO:journalctl@ceph.osd.4.smithi060.stdout:Sep 16 00:44:36 smithi060 podman[76914]: 2024-09-16 00:44:36.971732114 +0000 UTC m=+0.564850467 container died 5aa3c43cc3eac30b5f7371ce168b675da436ddb492ba8edd93495da7a5fb4b85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True) 2024-09-16T00:44:37.546 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:37 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:37.366+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:37.547 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:37 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:37.384+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:37.547 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:37 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:37.384+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:37.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.4.service' 2024-09-16T00:44:37.646 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:44:37.646 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-16T00:44:37.646 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-16T00:44:37.646 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.5 2024-09-16T00:44:37.696 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:37 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:37.482+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:37.697 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:37 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:37.482+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:05.988978+0000 front 2024-09-16T00:44:05.989117+0000 (oldest deadline 2024-09-16T00:44:31.888936+0000) 2024-09-16T00:44:37.697 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:37 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:37.433+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:37.697 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:37 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:37.433+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:37.829 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:37 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:37.751+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:37.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:37 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:37.528+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:37.976 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:37 smithi083 systemd[1]: Stopping Ceph osd.5 for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:44:37.977 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:37 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:37.838+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:38.418 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:37 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:37.976+0000 7f62a5ba4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:44:38.419 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:37 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:37.976+0000 7f62a5ba4640 -1 osd.5 94 *** Got signal Terminated *** 2024-09-16T00:44:38.419 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:37 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:37.976+0000 7f62a5ba4640 -1 osd.5 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T00:44:38.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:38 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:38.342+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:38.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:38 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:38.376+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:38.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:38 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:38.376+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:38.766 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:38 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:38.500+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:38.880 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:38 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:38.522+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:38.881 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:38 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:38.522+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:05.988978+0000 front 2024-09-16T00:44:05.989117+0000 (oldest deadline 2024-09-16T00:44:31.888936+0000) 2024-09-16T00:44:38.881 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:38 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:38.477+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:38.881 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:38 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:38.477+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:39.079 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:38 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:38.766+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:39.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:38 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:38.880+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:39.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:39 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:39.295+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:39.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:39 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:39.380+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:39.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:39 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:39.380+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:39.813 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:39 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:39.534+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:39.813 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:39 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:39.534+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:39.874 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:39 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:39.487+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:39.874 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:39 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:39.487+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:05.988978+0000 front 2024-09-16T00:44:05.989117+0000 (oldest deadline 2024-09-16T00:44:31.888936+0000) 2024-09-16T00:44:39.874 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:39 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:39.497+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:39.874 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:39 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:39.497+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:40.079 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:39 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:39.812+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:40.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:39 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:39.873+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:40.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:40 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:40.341+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:40.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:40 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:40.401+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:40.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:40 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:40.401+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:40.829 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:40 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:40.763+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:40.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:40 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:40.526+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:40.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:40 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:40.526+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:40.896 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:40 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:40.520+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:40.897 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:40 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:40.520+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:05.988978+0000 front 2024-09-16T00:44:05.989117+0000 (oldest deadline 2024-09-16T00:44:31.888936+0000) 2024-09-16T00:44:40.897 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:40 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:40.520+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:14.189635+0000 front 2024-09-16T00:44:14.189661+0000 (oldest deadline 2024-09-16T00:44:40.089735+0000) 2024-09-16T00:44:40.897 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:40 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:40.475+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:40.897 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:40 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:40.475+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:41.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:40 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:40.896+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:41.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:41 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:41.336+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:41.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:41 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:41.364+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:41.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:41 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:41.364+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:41.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:41 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:41.562+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:41.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:41 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:41.562+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:41.830 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:41 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:41.775+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:41.830 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:41 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:41.775+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.666752+0000 front 2024-09-16T00:44:15.666813+0000 (oldest deadline 2024-09-16T00:44:40.966878+0000) 2024-09-16T00:44:41.852 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:41 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:41.559+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:41.852 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:41 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:41.559+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:05.988978+0000 front 2024-09-16T00:44:05.989117+0000 (oldest deadline 2024-09-16T00:44:31.888936+0000) 2024-09-16T00:44:41.852 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:41 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:41.559+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:14.189635+0000 front 2024-09-16T00:44:14.189661+0000 (oldest deadline 2024-09-16T00:44:40.089735+0000) 2024-09-16T00:44:41.852 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:41 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:41.498+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:41.852 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:41 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:41.498+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:42.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:41 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:41.851+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:42.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:41 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:41.851+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:42.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:42 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:42.330+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:42.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:42 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:42.408+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:42.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:42 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:42.408+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:42.811 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:42.538+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:42.811 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:42.538+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:42.851 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:42 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:42.539+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.487303+0000 front 2024-09-16T00:44:01.487284+0000 (oldest deadline 2024-09-16T00:44:24.387318+0000) 2024-09-16T00:44:42.851 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:42 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:42.539+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:05.988978+0000 front 2024-09-16T00:44:05.989117+0000 (oldest deadline 2024-09-16T00:44:31.888936+0000) 2024-09-16T00:44:42.851 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:42 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5[46694]: 2024-09-16T00:44:42.539+0000 7f62a21ba640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:14.189635+0000 front 2024-09-16T00:44:14.189661+0000 (oldest deadline 2024-09-16T00:44:40.089735+0000) 2024-09-16T00:44:42.851 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:42 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:42.476+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:42.851 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:42 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:42.476+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:43.079 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:42.811+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:43.080 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:42 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:42.811+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.666752+0000 front 2024-09-16T00:44:15.666813+0000 (oldest deadline 2024-09-16T00:44:40.966878+0000) 2024-09-16T00:44:43.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:42 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:42.850+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:43.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:42 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:42.850+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:43.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:43 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:43.369+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:43.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:43 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:43.369+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:43.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:43 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:43.419+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:43.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:43 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:43.419+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:43.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:43 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:43.419+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:43.767 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:43 smithi083 podman[73276]: 2024-09-16 00:44:43.502881468 +0000 UTC m=+5.657226288 container died f8ef70f296b8238927fb816fa1064d7a7b80a5cd42264874ba13f5f0c34e19e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=squid-8921c8e) 2024-09-16T00:44:43.767 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:43 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:43.491+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:43.767 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:43 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:43.491+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:43.829 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:43 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:43.764+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:43.830 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:43 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:43.764+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.666752+0000 front 2024-09-16T00:44:15.666813+0000 (oldest deadline 2024-09-16T00:44:40.966878+0000) 2024-09-16T00:44:43.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:43 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:43.561+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:43.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:43 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:43.561+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:44.144 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:43 smithi083 podman[73276]: 2024-09-16 00:44:43.851536426 +0000 UTC m=+6.005881254 container cleanup f8ef70f296b8238927fb816fa1064d7a7b80a5cd42264874ba13f5f0c34e19e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-16T00:44:44.145 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:43 smithi083 bash[73276]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5 2024-09-16T00:44:44.145 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:43 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:43.891+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:44.145 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:43 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:43.891+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:44.418 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:44 smithi083 podman[73291]: 2024-09-16 00:44:44.144533804 +0000 UTC m=+0.644223609 container remove f8ef70f296b8238927fb816fa1064d7a7b80a5cd42264874ba13f5f0c34e19e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-16T00:44:44.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:44 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:44.419+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:44.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:44 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:44.419+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:44.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:44 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:44.434+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:44.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:44 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:44.434+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:44.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:44 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:44.434+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:44.747 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:44 smithi083 podman[73382]: 2024-09-16 00:44:44.376310484 +0000 UTC m=+0.015538100 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:44:44.748 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:44 smithi083 podman[73382]: 2024-09-16 00:44:44.487250132 +0000 UTC m=+0.126477749 container create 269523eed3c2cad7f340982c886a368dda4b565c87daf08490e3c5de1bdc8229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, io.buildah.version=1.37.2) 2024-09-16T00:44:44.748 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:44 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:44.463+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:44.748 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:44 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:44.463+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:45.079 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:44 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:44.782+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:45.079 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:44 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:44.782+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.666752+0000 front 2024-09-16T00:44:15.666813+0000 (oldest deadline 2024-09-16T00:44:40.966878+0000) 2024-09-16T00:44:45.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:44 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:44.588+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:45.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:44 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:44.588+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:45.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:44 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:44.929+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:45.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:44 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:44.929+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:45.169 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:44 smithi083 podman[73382]: 2024-09-16 00:44:44.74826954 +0000 UTC m=+0.387497169 container init 269523eed3c2cad7f340982c886a368dda4b565c87daf08490e3c5de1bdc8229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-16T00:44:45.169 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:44 smithi083 podman[73382]: 2024-09-16 00:44:44.751954118 +0000 UTC m=+0.391181735 container start 269523eed3c2cad7f340982c886a368dda4b565c87daf08490e3c5de1bdc8229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-16T00:44:45.169 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:44 smithi083 podman[73382]: 2024-09-16 00:44:44.848302903 +0000 UTC m=+0.487530519 container attach 269523eed3c2cad7f340982c886a368dda4b565c87daf08490e3c5de1bdc8229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-16T00:44:45.169 INFO:journalctl@ceph.osd.5.smithi083.stdout:Sep 16 00:44:44 smithi083 podman[73382]: 2024-09-16 00:44:44.941894087 +0000 UTC m=+0.581121704 container died 269523eed3c2cad7f340982c886a368dda4b565c87daf08490e3c5de1bdc8229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-16T00:44:45.518 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:45 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:45.445+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:45.519 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:45 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:45.445+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:45.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.5.service' 2024-09-16T00:44:45.582 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:44:45.582 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-16T00:44:45.582 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-16T00:44:45.582 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.6 2024-09-16T00:44:45.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:45 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:45.388+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:45.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:45 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:45.388+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:45.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:45 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:45.402+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:45.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:45 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:45.402+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:45.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:45 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:45.402+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:45.920 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:45 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:45.618+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:45.920 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:45 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:45.618+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:45.920 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:45 smithi033 systemd[1]: Stopping Ceph osd.6 for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:44:45.921 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:45 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:45.746+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:45.921 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:45 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:45.746+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.666752+0000 front 2024-09-16T00:44:15.666813+0000 (oldest deadline 2024-09-16T00:44:40.966878+0000) 2024-09-16T00:44:46.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:45 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:45.897+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:46.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:45 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:45.897+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:46.329 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:45 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:45.918+0000 7ffb31177640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:44:46.329 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:45 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:45.918+0000 7ffb31177640 -1 osd.6 94 *** Got signal Terminated *** 2024-09-16T00:44:46.330 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:45 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:45.918+0000 7ffb31177640 -1 osd.6 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T00:44:46.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:46 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:46.361+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:46.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:46 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:46.361+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:46.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:46 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:46.396+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:46.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:46 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:46.396+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:46.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:46 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:46.396+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:46.829 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:46.697+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:46.830 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:46.697+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.666752+0000 front 2024-09-16T00:44:15.666813+0000 (oldest deadline 2024-09-16T00:44:40.966878+0000) 2024-09-16T00:44:46.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:46.637+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:46.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:46 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:46.637+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:46.918 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:46 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:46.439+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:46.918 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:46 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:46.439+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:47.391 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:46 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:46.921+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:47.392 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:46 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:46.921+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:47.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:47 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:47.390+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:47.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:47 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:47.390+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:47.733 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:47 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:47.324+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:47.733 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:47 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:47.324+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:47.734 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:47 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:47.357+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:47.734 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:47 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:47.357+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:47.734 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:47 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:47.357+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:48.079 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:47 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:47.675+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:48.079 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:47 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:47.675+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.666752+0000 front 2024-09-16T00:44:15.666813+0000 (oldest deadline 2024-09-16T00:44:40.966878+0000) 2024-09-16T00:44:48.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:47 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:47.631+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:48.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:47 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:47.631+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:48.418 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:47 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:47.934+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:48.418 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:47 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:47.934+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:48.418 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:47 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:47.934+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:44:48.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:48 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:48.367+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:48.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:48 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:48.367+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:48.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:48 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:48.367+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:48.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:48 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:48.367+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:44:48.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:48 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:48.285+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:48.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:48 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:48.285+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:48.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:48 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:48.285+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:22.576001+0000 front 2024-09-16T00:44:22.576014+0000 (oldest deadline 2024-09-16T00:44:47.875902+0000) 2024-09-16T00:44:48.918 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:48 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:48.427+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:48.918 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:48 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:48.427+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:48.919 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:48 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:48.427+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:44:49.079 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:48.632+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:49.080 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:48.632+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.666752+0000 front 2024-09-16T00:44:15.666813+0000 (oldest deadline 2024-09-16T00:44:40.966878+0000) 2024-09-16T00:44:49.080 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:48.632+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:25.067959+0000 front 2024-09-16T00:44:25.067981+0000 (oldest deadline 2024-09-16T00:44:48.568087+0000) 2024-09-16T00:44:49.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:48.677+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:49.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:48 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:48.677+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:49.386 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:48 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:48.932+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:49.386 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:48 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:48.932+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:49.386 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:48 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:48.932+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:44:49.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:49 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:49.386+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:49.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:49 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:49.386+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:49.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:49 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:49.386+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:44:49.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:49 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:49.319+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:49.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:49 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:49.319+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:49.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:49 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:49.319+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:22.576001+0000 front 2024-09-16T00:44:22.576014+0000 (oldest deadline 2024-09-16T00:44:47.875902+0000) 2024-09-16T00:44:49.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:49 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:49.399+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:49.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:49 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:49.399+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:49.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:49 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:49.399+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:49.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:49 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:49.399+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:44:50.079 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:49 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:49.610+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:50.079 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:49 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:49.610+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.666752+0000 front 2024-09-16T00:44:15.666813+0000 (oldest deadline 2024-09-16T00:44:40.966878+0000) 2024-09-16T00:44:50.080 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:49 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:49.610+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:25.067959+0000 front 2024-09-16T00:44:25.067981+0000 (oldest deadline 2024-09-16T00:44:48.568087+0000) 2024-09-16T00:44:50.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:49 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:49.664+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:50.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:49 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:49.664+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:50.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:49 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:49.904+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:50.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:49 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:49.904+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:50.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:49 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:49.904+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:44:50.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:50 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:50.381+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:50.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:50 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:50.381+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:50.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:50 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:50.381+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:44:50.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:50 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:50.331+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:50.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:50 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:50.331+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:50.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:50 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:50.331+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:22.576001+0000 front 2024-09-16T00:44:22.576014+0000 (oldest deadline 2024-09-16T00:44:47.875902+0000) 2024-09-16T00:44:50.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:50 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:50.436+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:50.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:50 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:50.436+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:50.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:50 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:50.436+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:50.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:50 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:50.436+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:44:51.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:50 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:50.627+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:51.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:50 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:50.627+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:51.080 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:50 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:50.606+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:09.364699+0000 front 2024-09-16T00:44:09.364910+0000 (oldest deadline 2024-09-16T00:44:32.264943+0000) 2024-09-16T00:44:51.080 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:50 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:50.606+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.666752+0000 front 2024-09-16T00:44:15.666813+0000 (oldest deadline 2024-09-16T00:44:40.966878+0000) 2024-09-16T00:44:51.080 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:50 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6[56312]: 2024-09-16T00:44:50.606+0000 7ffb2d78d640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:25.067959+0000 front 2024-09-16T00:44:25.067981+0000 (oldest deadline 2024-09-16T00:44:48.568087+0000) 2024-09-16T00:44:51.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:50 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:50.898+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:51.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:50 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:50.898+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:51.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:50 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:50.898+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:44:51.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:51 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:51.410+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:51.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:51 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:51.410+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:51.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:51 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:51.410+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:44:51.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:51 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:51.331+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:51.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:51 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:51.331+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:51.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:51 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:51.331+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:22.576001+0000 front 2024-09-16T00:44:22.576014+0000 (oldest deadline 2024-09-16T00:44:47.875902+0000) 2024-09-16T00:44:51.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:51 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:51.462+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:51.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:51 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:51.462+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:51.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:51 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:51.462+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:51.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:51 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:51.462+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:44:51.829 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:51 smithi033 podman[137690]: 2024-09-16 00:44:51.522726647 +0000 UTC m=+5.724418662 container died 19ddb956e0c70dc2034d3712182f881e652ea6fb428175f60c248652328b7663 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, io.buildah.version=1.37.2) 2024-09-16T00:44:51.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:51 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:51.589+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:51.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:51 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:51.589+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:51.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:51 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:51.589+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:44:52.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:51 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:51.915+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:52.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:51 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:51.915+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:52.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:51 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:51.915+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:44:52.287 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:52 smithi033 podman[137690]: 2024-09-16 00:44:52.044301899 +0000 UTC m=+6.245993913 container cleanup 19ddb956e0c70dc2034d3712182f881e652ea6fb428175f60c248652328b7663 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-16T00:44:52.287 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:52 smithi033 bash[137690]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6 2024-09-16T00:44:52.579 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:52 smithi033 podman[137704]: 2024-09-16 00:44:52.287683303 +0000 UTC m=+0.762821238 container remove 19ddb956e0c70dc2034d3712182f881e652ea6fb428175f60c248652328b7663 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-09-16T00:44:52.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:52 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:52.374+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:52.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:52 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:52.374+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:52.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:52 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:52.374+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:44:52.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:52 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:52.309+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:52.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:52 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:52.309+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:52.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:52 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:52.309+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:22.576001+0000 front 2024-09-16T00:44:22.576014+0000 (oldest deadline 2024-09-16T00:44:47.875902+0000) 2024-09-16T00:44:52.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:52 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:52.416+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:52.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:52 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:52.416+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:52.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:52 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:52.416+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:52.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:52 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:52.416+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:44:52.869 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:52 smithi033 podman[137793]: 2024-09-16 00:44:52.619973754 +0000 UTC m=+0.016147507 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:44:52.869 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:52 smithi033 podman[137793]: 2024-09-16 00:44:52.738720907 +0000 UTC m=+0.134894661 container create 7cb7effab4edcba2be1a8a6e174d0db630f89787135d2a619f61192e033db513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e) 2024-09-16T00:44:52.869 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:52 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:52.598+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:52.869 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:52 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:52.598+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:52.869 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:52 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:52.598+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:44:53.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:52 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:52.888+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:53.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:52 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:52.888+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:53.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:52 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:52.888+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:44:53.220 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:53 smithi033 podman[137793]: 2024-09-16 00:44:53.016089013 +0000 UTC m=+0.412262768 container init 7cb7effab4edcba2be1a8a6e174d0db630f89787135d2a619f61192e033db513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-16T00:44:53.220 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:53 smithi033 podman[137793]: 2024-09-16 00:44:53.019765533 +0000 UTC m=+0.415939286 container start 7cb7effab4edcba2be1a8a6e174d0db630f89787135d2a619f61192e033db513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e) 2024-09-16T00:44:53.220 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:53 smithi033 podman[137793]: 2024-09-16 00:44:53.082666616 +0000 UTC m=+0.478840368 container attach 7cb7effab4edcba2be1a8a6e174d0db630f89787135d2a619f61192e033db513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-16T00:44:53.505 INFO:journalctl@ceph.osd.6.smithi033.stdout:Sep 16 00:44:53 smithi033 podman[137793]: 2024-09-16 00:44:53.220153614 +0000 UTC m=+0.616327372 container died 7cb7effab4edcba2be1a8a6e174d0db630f89787135d2a619f61192e033db513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2) 2024-09-16T00:44:53.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:53 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:53.361+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:53.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:53 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:53.361+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:53.669 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:53 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:53.361+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:44:53.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:53 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:53.420+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:53.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:53 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:53.420+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:53.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:53 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:53.420+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:53.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:53 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:53.420+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:44:53.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:53 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:53.294+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:53.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:53 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:53.294+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:53.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:53 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:53.294+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:22.576001+0000 front 2024-09-16T00:44:22.576014+0000 (oldest deadline 2024-09-16T00:44:47.875902+0000) 2024-09-16T00:44:53.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:53 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:53.550+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:53.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:53 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:53.550+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:53.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:53 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:53.550+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:44:54.087 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.6.service' 2024-09-16T00:44:54.128 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:44:54.128 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-16T00:44:54.128 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-16T00:44:54.128 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.7 2024-09-16T00:44:54.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:53 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:53.876+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:54.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:53 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:53.876+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:54.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:53 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:53.876+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:44:54.480 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:54 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:54.393+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:54.480 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:54 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:54.393+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:54.480 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:54 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:54.393+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:54.481 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:54 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:54.393+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:44:54.481 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:54 smithi060 systemd[1]: Stopping Ceph osd.7 for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:44:54.481 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:54 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:54.320+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:54.481 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:54 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:54.320+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:54.481 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:54 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:54.320+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:22.576001+0000 front 2024-09-16T00:44:22.576014+0000 (oldest deadline 2024-09-16T00:44:47.875902+0000) 2024-09-16T00:44:54.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:54 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:54.369+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:54.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:54 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:54.369+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:54.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:54 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:54.369+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:44:54.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:54 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:54.579+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:54.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:54 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:54.579+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:54.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:54 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:54.579+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:44:54.980 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:54 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:54.574+0000 7f3b34065640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:44:54.980 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:54 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:54.574+0000 7f3b34065640 -1 osd.7 94 *** Got signal Terminated *** 2024-09-16T00:44:54.981 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:54 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:54.574+0000 7f3b34065640 -1 osd.7 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T00:44:55.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:54 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:54.876+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:55.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:54 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:54.876+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:55.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:54 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:54.876+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:44:55.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:55 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:55.411+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:55.669 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:55 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:55.411+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:55.669 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:55 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:55.411+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:44:55.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:55 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:55.356+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:55.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:55 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:55.356+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:55.730 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:55 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:55.356+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:22.576001+0000 front 2024-09-16T00:44:22.576014+0000 (oldest deadline 2024-09-16T00:44:47.875902+0000) 2024-09-16T00:44:55.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:55 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:55.381+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:55.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:55 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:55.381+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:55.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:55 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:55.381+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:55.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:55 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:55.381+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:44:56.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:55 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:55.622+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:56.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:55 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:55.622+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:56.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:55 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:55.623+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:44:56.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:55 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:55.907+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:56.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:55 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:55.907+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:56.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:55 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:55.907+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:44:56.632 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:56 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:56.385+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:56.632 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:56 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:56.385+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:56.632 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:56 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:56.385+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:56.632 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:56 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:56.385+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:44:56.633 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:56 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:56.344+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:56.633 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:56 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:56.344+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:56.633 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:56 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:56.344+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:22.576001+0000 front 2024-09-16T00:44:22.576014+0000 (oldest deadline 2024-09-16T00:44:47.875902+0000) 2024-09-16T00:44:56.633 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:56 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:56.344+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.277448+0000 front 2024-09-16T00:44:34.277491+0000 (oldest deadline 2024-09-16T00:44:55.977603+0000) 2024-09-16T00:44:56.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:56 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:56.396+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:56.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:56 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:56.396+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:56.669 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:56 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:56.396+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:44:57.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:56 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:56.634+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:57.162 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:56 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:56.634+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:57.162 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:56 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:56.634+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:44:57.162 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:56 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:56.634+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:44:57.408 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:56 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:56.922+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:57.409 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:56 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:56.922+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:57.409 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:56 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:56.922+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:44:57.648 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:57 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:57.351+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:57.648 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:57 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:57.351+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:57.648 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:57 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:57.351+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:57.649 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:57 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:57.351+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:44:57.649 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:57 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:57.319+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:57.649 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:57 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:57.319+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:57.649 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:57 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:57.319+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:22.576001+0000 front 2024-09-16T00:44:22.576014+0000 (oldest deadline 2024-09-16T00:44:47.875902+0000) 2024-09-16T00:44:57.649 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:57 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:57.319+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.277448+0000 front 2024-09-16T00:44:34.277491+0000 (oldest deadline 2024-09-16T00:44:55.977603+0000) 2024-09-16T00:44:57.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:57 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:57.407+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:57.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:57 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:57.407+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:57.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:57 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:57.407+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:44:57.669 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:57 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:57.407+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.005609+0000 front 2024-09-16T00:44:33.005718+0000 (oldest deadline 2024-09-16T00:44:56.505414+0000) 2024-09-16T00:44:58.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:57 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:57.645+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:58.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:57 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:57.645+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:58.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:57 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:57.645+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:44:58.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:57 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:57.645+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:44:58.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:57 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:57.916+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:58.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:57 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:57.916+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:58.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:57 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:57.916+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:44:58.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:58 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:58.385+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:58.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:58 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:58.385+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:58.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:58 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:58.385+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:58.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:58 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:58.385+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:44:58.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:58 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:58.385+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:44:58.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:58 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:58.331+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:58.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:58 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:58.331+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:58.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:58 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:58.331+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:22.576001+0000 front 2024-09-16T00:44:22.576014+0000 (oldest deadline 2024-09-16T00:44:47.875902+0000) 2024-09-16T00:44:58.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:58 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:58.331+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.277448+0000 front 2024-09-16T00:44:34.277491+0000 (oldest deadline 2024-09-16T00:44:55.977603+0000) 2024-09-16T00:44:58.918 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:58 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:58.426+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:58.918 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:58 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:58.426+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:58.918 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:58 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:58.426+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:44:58.919 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:58 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:58.426+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.005609+0000 front 2024-09-16T00:44:33.005718+0000 (oldest deadline 2024-09-16T00:44:56.505414+0000) 2024-09-16T00:44:59.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:58 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:58.681+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:44:59.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:58 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:58.681+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:44:59.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:58 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:58.681+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:44:59.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:58 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:58.681+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:44:59.416 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:58 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:58.936+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:44:59.416 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:58 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:58.936+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:44:59.416 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:58 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:58.936+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:44:59.416 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:58 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:58.936+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:44:59.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:59 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:59.415+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:44:59.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:59 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:59.415+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:44:59.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:59 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:59.415+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:44:59.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:44:59 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:44:59.415+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.005609+0000 front 2024-09-16T00:44:33.005718+0000 (oldest deadline 2024-09-16T00:44:56.505414+0000) 2024-09-16T00:44:59.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:59 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:59.338+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:44:59.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:59 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:59.338+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:44:59.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:59 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:59.338+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:44:59.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:59 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:59.338+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:44:59.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:44:59 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:44:59.338+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:44:59.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:59 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:59.340+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.272821+0000 front 2024-09-16T00:44:06.272847+0000 (oldest deadline 2024-09-16T00:44:30.972984+0000) 2024-09-16T00:44:59.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:59 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:59.340+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:17.374607+0000 front 2024-09-16T00:44:17.374586+0000 (oldest deadline 2024-09-16T00:44:42.575260+0000) 2024-09-16T00:44:59.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:59 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:59.340+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:22.576001+0000 front 2024-09-16T00:44:22.576014+0000 (oldest deadline 2024-09-16T00:44:47.875902+0000) 2024-09-16T00:44:59.731 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:44:59 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7[53080]: 2024-09-16T00:44:59.340+0000 7f3b2c673640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.277448+0000 front 2024-09-16T00:44:34.277491+0000 (oldest deadline 2024-09-16T00:44:55.977603+0000) 2024-09-16T00:45:00.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:59 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:59.728+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:00.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:59 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:59.728+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:00.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:59 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:59.728+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:00.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:44:59 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:44:59.728+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:00.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:59 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:59.888+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:00.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:59 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:59.888+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:00.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:59 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:59.888+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:00.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:44:59 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:44:59.888+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:00.480 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:45:00 smithi060 podman[77016]: 2024-09-16 00:45:00.193247201 +0000 UTC m=+5.868212601 container died 6ab9d8a0208526301a885c515c8b56f3cd708bbdb27b0fc162b9e572703ea792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-16T00:45:00.481 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:00 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:00.357+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:00.481 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:00 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:00.357+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:00.481 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:00 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:00.357+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:00.481 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:00 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:00.357+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:00.481 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:00 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:00.357+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:00.826 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:45:00 smithi060 podman[77016]: 2024-09-16 00:45:00.608592531 +0000 UTC m=+6.283557915 container cleanup 6ab9d8a0208526301a885c515c8b56f3cd708bbdb27b0fc162b9e572703ea792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:45:00.826 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:45:00 smithi060 bash[77016]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7 2024-09-16T00:45:00.865 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:00 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:00.426+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:45:00.865 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:00 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:00.426+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:45:00.865 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:00 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:00.426+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:45:00.865 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:00 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:00.426+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.005609+0000 front 2024-09-16T00:44:33.005718+0000 (oldest deadline 2024-09-16T00:44:56.505414+0000) 2024-09-16T00:45:01.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:00 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:00.748+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:01.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:00 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:00.748+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:01.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:00 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:00.748+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:01.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:00 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:00.748+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:01.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:00 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:00.864+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:01.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:00 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:00.864+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:01.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:00 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:00.864+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:01.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:00 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:00.864+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:01.230 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:45:00 smithi060 podman[77045]: 2024-09-16 00:45:00.826686676 +0000 UTC m=+0.631329224 container remove 6ab9d8a0208526301a885c515c8b56f3cd708bbdb27b0fc162b9e572703ea792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e) 2024-09-16T00:45:01.588 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:45:01 smithi060 podman[77138]: 2024-09-16 00:45:01.134974801 +0000 UTC m=+0.015742500 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:45:01.588 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:45:01 smithi060 podman[77138]: 2024-09-16 00:45:01.344050426 +0000 UTC m=+0.224818157 container create 6f8ea732298e2d402257ef3733c370f64cb9b4b5b00c104c9ad0e646aac07973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-09-16T00:45:01.588 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:01 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:01.332+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:01.588 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:01 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:01.332+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:01.588 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:01 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:01.332+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:01.589 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:01 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:01.332+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:01.589 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:01 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:01.332+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:01.841 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:01 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:01.474+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:45:01.841 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:01 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:01.474+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:45:01.841 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:01 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:01.474+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:45:01.841 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:01 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:01.474+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.005609+0000 front 2024-09-16T00:44:33.005718+0000 (oldest deadline 2024-09-16T00:44:56.505414+0000) 2024-09-16T00:45:01.981 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:45:01 smithi060 podman[77138]: 2024-09-16 00:45:01.588458917 +0000 UTC m=+0.469226609 container init 6f8ea732298e2d402257ef3733c370f64cb9b4b5b00c104c9ad0e646aac07973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-16T00:45:01.981 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:45:01 smithi060 podman[77138]: 2024-09-16 00:45:01.591873737 +0000 UTC m=+0.472641426 container start 6f8ea732298e2d402257ef3733c370f64cb9b4b5b00c104c9ad0e646aac07973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-16T00:45:01.981 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:45:01 smithi060 podman[77138]: 2024-09-16 00:45:01.68000464 +0000 UTC m=+0.560772339 container attach 6f8ea732298e2d402257ef3733c370f64cb9b4b5b00c104c9ad0e646aac07973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-09-16T00:45:01.981 INFO:journalctl@ceph.osd.7.smithi060.stdout:Sep 16 00:45:01 smithi060 podman[77138]: 2024-09-16 00:45:01.786134146 +0000 UTC m=+0.666901845 container died 6f8ea732298e2d402257ef3733c370f64cb9b4b5b00c104c9ad0e646aac07973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-16T00:45:02.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:01 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:01.761+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:02.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:01 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:01.761+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:02.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:01 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:01.761+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:02.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:01 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:01.761+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:02.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:01 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:01.840+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:02.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:01 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:01.840+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:02.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:01 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:01.840+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:02.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:01 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:01.840+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:02.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.7.service' 2024-09-16T00:45:02.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:45:02.427 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-16T00:45:02.427 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-16T00:45:02.428 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.8 2024-09-16T00:45:02.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:02 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:02.339+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:02.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:02 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:02.339+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:02.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:02 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:02.339+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:02.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:02 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:02.339+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:02.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:02 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:02.339+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:02.746 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:02 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:02.445+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:45:02.746 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:02 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:02.445+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:45:02.746 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:02 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:02.445+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:45:02.747 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:02 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:02.445+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.005609+0000 front 2024-09-16T00:44:33.005718+0000 (oldest deadline 2024-09-16T00:44:56.505414+0000) 2024-09-16T00:45:02.747 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:02 smithi083 systemd[1]: Stopping Ceph osd.8 for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:45:03.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:02 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:02.752+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:03.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:02 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:02.752+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:03.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:02 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:02.752+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:03.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:02 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:02.752+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:03.168 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:02 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:02.746+0000 7fb4ac4c4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:45:03.168 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:02 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:02.746+0000 7fb4ac4c4640 -1 osd.8 94 *** Got signal Terminated *** 2024-09-16T00:45:03.168 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:02 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:02.746+0000 7fb4ac4c4640 -1 osd.8 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T00:45:03.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:02 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:02.824+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:03.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:02 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:02.824+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:03.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:02 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:02.824+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:03.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:02 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:02.824+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:03.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:03 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:03.373+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:03.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:03 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:03.373+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:03.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:03 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:03.373+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:03.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:03 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:03.373+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:03.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:03 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:03.373+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:03.790 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:03 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:03.434+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:45:03.790 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:03 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:03.434+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:45:03.790 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:03 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:03.434+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:45:03.790 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:03 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:03.434+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.005609+0000 front 2024-09-16T00:44:33.005718+0000 (oldest deadline 2024-09-16T00:44:56.505414+0000) 2024-09-16T00:45:04.003 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:03 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:03.703+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:04.004 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:03 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:03.703+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:04.004 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:03 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:03.703+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:04.004 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:03 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:03.703+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:04.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:03 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:03.790+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:04.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:03 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:03.790+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:04.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:03 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:03.790+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:04.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:03 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:03.790+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:04.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:03 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:03.790+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:04.668 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:04 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:04.454+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:45:04.669 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:04 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:04.454+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:45:04.669 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:04 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:04.454+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:45:04.669 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:04 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:04.454+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.005609+0000 front 2024-09-16T00:44:33.005718+0000 (oldest deadline 2024-09-16T00:44:56.505414+0000) 2024-09-16T00:45:04.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:04 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:04.408+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:04.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:04 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:04.408+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:04.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:04 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:04.408+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:04.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:04 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:04.408+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:04.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:04 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:04.408+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:05.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:04 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:04.679+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:05.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:04 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:04.680+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:05.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:04 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:04.680+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:05.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:04 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:04.680+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:05.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:04 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:04.764+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:05.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:04 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:04.764+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:05.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:04 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:04.764+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:05.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:04 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:04.764+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:05.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:04 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:04.765+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:05.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:05 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:05.395+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:05.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:05 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:05.395+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:05.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:05 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:05.395+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:05.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:05 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:05.395+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:05.732 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:05 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:05.395+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:05.747 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:05.468+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:45:05.748 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:05.468+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:45:05.748 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:05.468+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:45:05.748 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:05.468+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.005609+0000 front 2024-09-16T00:44:33.005718+0000 (oldest deadline 2024-09-16T00:44:56.505414+0000) 2024-09-16T00:45:06.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:05 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:05.633+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:06.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:05 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:05.633+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:06.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:05 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:05.633+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:06.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:05 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:05.633+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:06.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:05 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:05.633+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:40.322323+0000 front 2024-09-16T00:44:40.322115+0000 (oldest deadline 2024-09-16T00:45:05.022073+0000) 2024-09-16T00:45:06.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:05.747+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:06.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:05.747+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:06.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:05.747+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:06.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:05.747+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:06.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:05 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:05.747+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:06.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:06 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:06.433+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:06.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:06 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:06.433+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:06.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:06 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:06.433+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:06.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:06 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:06.433+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:06.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:06 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:06.433+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:06.736 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:06.465+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:45:06.736 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:06.465+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:45:06.736 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:06.465+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:45:06.737 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:06.465+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.005609+0000 front 2024-09-16T00:44:33.005718+0000 (oldest deadline 2024-09-16T00:44:56.505414+0000) 2024-09-16T00:45:06.737 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:06.465+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.806088+0000 front 2024-09-16T00:44:41.806109+0000 (oldest deadline 2024-09-16T00:45:05.906184+0000) 2024-09-16T00:45:06.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:06.594+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:06.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:06.594+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:06.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:06.594+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:06.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:06.594+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:06.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:06 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:06.594+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:40.322323+0000 front 2024-09-16T00:44:40.322115+0000 (oldest deadline 2024-09-16T00:45:05.022073+0000) 2024-09-16T00:45:07.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:06.735+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:07.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:06.735+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:07.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:06.735+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:07.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:06.735+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:07.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:06 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:06.735+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:07.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:07 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:07.430+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:07.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:07 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:07.430+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:07.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:07 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:07.430+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:07.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:07 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:07.430+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:07.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:07 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:07.430+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:07.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:07 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:07.545+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:07.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:07 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:07.545+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:07.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:07 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:07.545+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:07.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:07 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:07.545+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:07.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:07 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:07.545+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:40.322323+0000 front 2024-09-16T00:44:40.322115+0000 (oldest deadline 2024-09-16T00:45:05.022073+0000) 2024-09-16T00:45:07.918 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:07.702+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:07.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:07.702+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:07.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:07.702+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:07.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:07.702+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:07.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:07.702+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:07.919 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:07.509+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.600066+0000 front 2024-09-16T00:44:01.600094+0000 (oldest deadline 2024-09-16T00:44:22.700198+0000) 2024-09-16T00:45:07.919 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:07.509+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.101674+0000 front 2024-09-16T00:44:06.101820+0000 (oldest deadline 2024-09-16T00:44:36.702407+0000) 2024-09-16T00:45:07.919 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:07.509+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:24.904110+0000 front 2024-09-16T00:44:24.903918+0000 (oldest deadline 2024-09-16T00:44:47.803965+0000) 2024-09-16T00:45:07.919 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:07.509+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.005609+0000 front 2024-09-16T00:44:33.005718+0000 (oldest deadline 2024-09-16T00:44:56.505414+0000) 2024-09-16T00:45:07.919 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:07 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8[53219]: 2024-09-16T00:45:07.509+0000 7fb4a8ada640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.806088+0000 front 2024-09-16T00:44:41.806109+0000 (oldest deadline 2024-09-16T00:45:05.906184+0000) 2024-09-16T00:45:08.530 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:08 smithi083 podman[73486]: 2024-09-16 00:45:08.21628515 +0000 UTC m=+5.595306585 container died c6b122500d37b4ac0da47068a933ccbed1273a89dd166df24563c92825ac6ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-16T00:45:08.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:08 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:08.441+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:08.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:08 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:08.441+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:08.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:08 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:08.441+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:08.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:08 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:08.441+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:08.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:08 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:08.441+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:08.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:08 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:08.441+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:08.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:08 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:08.517+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:08.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:08 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:08.517+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:08.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:08 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:08.517+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:08.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:08 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:08.517+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:08.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:08 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:08.517+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:40.322323+0000 front 2024-09-16T00:44:40.322115+0000 (oldest deadline 2024-09-16T00:45:05.022073+0000) 2024-09-16T00:45:08.872 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:08 smithi083 podman[73486]: 2024-09-16 00:45:08.633499486 +0000 UTC m=+6.012520921 container cleanup c6b122500d37b4ac0da47068a933ccbed1273a89dd166df24563c92825ac6ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-16T00:45:08.873 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:08 smithi083 bash[73486]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8 2024-09-16T00:45:08.873 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:08.751+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:08.873 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:08.751+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:08.873 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:08.751+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:08.873 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:08.751+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:08.873 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:08.751+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:09.168 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:08 smithi083 podman[73513]: 2024-09-16 00:45:08.873005766 +0000 UTC m=+0.649219090 container remove c6b122500d37b4ac0da47068a933ccbed1273a89dd166df24563c92825ac6ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-16T00:45:09.576 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:09 smithi083 podman[73601]: 2024-09-16 00:45:09.186475223 +0000 UTC m=+0.015976365 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:45:09.576 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:09 smithi083 podman[73601]: 2024-09-16 00:45:09.290526676 +0000 UTC m=+0.120027860 container create c75c6d26d27d07092229d437788a9ae14c0d8987311d7b72ab0011d9d96cb57d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-16T00:45:09.576 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:09 smithi083 podman[73601]: 2024-09-16 00:45:09.49288672 +0000 UTC m=+0.322387862 container init c75c6d26d27d07092229d437788a9ae14c0d8987311d7b72ab0011d9d96cb57d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8-deactivate, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:45:09.576 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:09 smithi083 podman[73601]: 2024-09-16 00:45:09.496451901 +0000 UTC m=+0.325953044 container start c75c6d26d27d07092229d437788a9ae14c0d8987311d7b72ab0011d9d96cb57d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_CLEAN=True) 2024-09-16T00:45:09.576 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:09 smithi083 podman[73601]: 2024-09-16 00:45:09.576147267 +0000 UTC m=+0.405648441 container attach c75c6d26d27d07092229d437788a9ae14c0d8987311d7b72ab0011d9d96cb57d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:45:09.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:09 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:09.418+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:09.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:09 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:09.418+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:09.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:09 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:09.418+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:09.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:09 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:09.418+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:09.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:09 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:09.418+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:09.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:09 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:09.418+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:09.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:09.551+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:09.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:09.551+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:09.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:09.551+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:09.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:09.551+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:09.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:09 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:09.551+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:40.322323+0000 front 2024-09-16T00:44:40.322115+0000 (oldest deadline 2024-09-16T00:45:05.022073+0000) 2024-09-16T00:45:09.918 INFO:journalctl@ceph.osd.8.smithi083.stdout:Sep 16 00:45:09 smithi083 podman[73601]: 2024-09-16 00:45:09.683194741 +0000 UTC m=+0.512695883 container died c75c6d26d27d07092229d437788a9ae14c0d8987311d7b72ab0011d9d96cb57d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T00:45:09.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:09 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:09.779+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:09.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:09 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:09.779+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:09.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:09 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:09.779+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:09.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:09 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:09.779+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:09.920 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:09 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:09.779+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:10.298 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.8.service' 2024-09-16T00:45:10.337 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:45:10.337 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-16T00:45:10.337 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-16T00:45:10.337 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.9 2024-09-16T00:45:10.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:10.381+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:10.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:10.381+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:10.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:10.381+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:10.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:10.381+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:10.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:10.381+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:10.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:10 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:10.381+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:10.732 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:10 smithi033 systemd[1]: Stopping Ceph osd.9 for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:45:10.732 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:10 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:10.506+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:10.732 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:10 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:10.506+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:10.732 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:10 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:10.506+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:10.732 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:10 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:10.506+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:10.732 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:10 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:10.506+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:40.322323+0000 front 2024-09-16T00:44:40.322115+0000 (oldest deadline 2024-09-16T00:45:05.022073+0000) 2024-09-16T00:45:11.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:10 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:10.731+0000 7f2ce230e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:45:11.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:10 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:10.731+0000 7f2ce230e640 -1 osd.9 94 *** Got signal Terminated *** 2024-09-16T00:45:11.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:10 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:10.731+0000 7f2ce230e640 -1 osd.9 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T00:45:11.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:10 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:10.754+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:11.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:10 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:10.754+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:11.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:10 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:10.754+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:11.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:10 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:10.754+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:11.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:10 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:10.754+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:11.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:11 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:11.392+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:11.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:11 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:11.392+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:11.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:11 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:11.392+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:11.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:11 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:11.392+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:11.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:11 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:11.392+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:11.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:11 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:11.392+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:11.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:11 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:11.458+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:11.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:11 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:11.458+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:11.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:11 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:11.458+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:11.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:11 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:11.458+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:11.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:11 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:11.458+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:40.322323+0000 front 2024-09-16T00:44:40.322115+0000 (oldest deadline 2024-09-16T00:45:05.022073+0000) 2024-09-16T00:45:12.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:11 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:11.740+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:12.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:11 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:11.740+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:12.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:11 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:11.740+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:12.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:11 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:11.740+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:12.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:11 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:11.740+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:12.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:11 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:11.740+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:12.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:12 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:12.412+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:12.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:12 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:12.412+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:12.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:12 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:12.412+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:12.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:12 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:12.412+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:12.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:12 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:12.412+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:12.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:12 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:12.412+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:12.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:12 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:12.456+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:12.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:12 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:12.456+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:12.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:12 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:12.456+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:12.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:12 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:12.456+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:12.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:12 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:12.456+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:40.322323+0000 front 2024-09-16T00:44:40.322115+0000 (oldest deadline 2024-09-16T00:45:05.022073+0000) 2024-09-16T00:45:13.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:12 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:12.787+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:13.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:12 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:12.787+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:13.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:12 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:12.787+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:13.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:12 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:12.787+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:13.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:12 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:12.787+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:13.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:12 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:12.787+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:13.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:13 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:13.371+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:13.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:13 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:13.371+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:13.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:13 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:13.371+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:13.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:13 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:13.371+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:13.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:13 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:13.371+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:13.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:13 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:13.371+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:13.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:13 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:13.371+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:50.565640+0000 front 2024-09-16T00:44:50.565608+0000 (oldest deadline 2024-09-16T00:45:12.865461+0000) 2024-09-16T00:45:13.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:13 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:13.490+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:13.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:13 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:13.490+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:13.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:13 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:13.490+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:13.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:13 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:13.490+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:13.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:13 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:13.490+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:40.322323+0000 front 2024-09-16T00:44:40.322115+0000 (oldest deadline 2024-09-16T00:45:05.022073+0000) 2024-09-16T00:45:14.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:13.754+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:14.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:13.754+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:14.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:13.754+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:14.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:13.754+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:14.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:13.754+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:14.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:13.754+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:14.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:14 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:14.374+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:14.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:14 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:14.374+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:14.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:14 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:14.374+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:14.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:14 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:14.374+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:14.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:14 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:14.374+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:14.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:14 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:14.374+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:14.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:14 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:14.374+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:50.565640+0000 front 2024-09-16T00:44:50.565608+0000 (oldest deadline 2024-09-16T00:45:12.865461+0000) 2024-09-16T00:45:14.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:14 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:14.477+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:14.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:14 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:14.477+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:14.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:14 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:14.477+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:14.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:14 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:14.477+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:14.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:14 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:14.477+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:40.322323+0000 front 2024-09-16T00:44:40.322115+0000 (oldest deadline 2024-09-16T00:45:05.022073+0000) 2024-09-16T00:45:14.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:14 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:14.477+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:49.722922+0000 front 2024-09-16T00:44:49.722944+0000 (oldest deadline 2024-09-16T00:45:14.423039+0000) 2024-09-16T00:45:15.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:14 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:14.710+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:15.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:14 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:14.710+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:15.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:14 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:14.710+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:15.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:14 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:14.710+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:15.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:14 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:14.710+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:15.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:14 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:14.710+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:15.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:15 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:15.359+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:15.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:15 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:15.359+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:15.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:15 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:15.359+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:15.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:15 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:15.359+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:15.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:15 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:15.359+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:15.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:15 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:15.359+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:15.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:15 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:15.359+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:50.565640+0000 front 2024-09-16T00:44:50.565608+0000 (oldest deadline 2024-09-16T00:45:12.865461+0000) 2024-09-16T00:45:15.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:15 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:15.517+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.115324+0000 front 2024-09-16T00:44:06.115351+0000 (oldest deadline 2024-09-16T00:44:33.115955+0000) 2024-09-16T00:45:15.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:15 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:15.517+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:13.116574+0000 front 2024-09-16T00:44:13.116715+0000 (oldest deadline 2024-09-16T00:44:39.016546+0000) 2024-09-16T00:45:15.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:15 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:15.517+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.419189+0000 front 2024-09-16T00:44:26.419233+0000 (oldest deadline 2024-09-16T00:44:51.119343+0000) 2024-09-16T00:45:15.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:15 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:15.517+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.521118+0000 front 2024-09-16T00:44:34.521193+0000 (oldest deadline 2024-09-16T00:44:56.220952+0000) 2024-09-16T00:45:15.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:15 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:15.517+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:40.322323+0000 front 2024-09-16T00:44:40.322115+0000 (oldest deadline 2024-09-16T00:45:05.022073+0000) 2024-09-16T00:45:15.830 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:15 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9[62836]: 2024-09-16T00:45:15.517+0000 7f2cda91c640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:49.722922+0000 front 2024-09-16T00:44:49.722944+0000 (oldest deadline 2024-09-16T00:45:14.423039+0000) 2024-09-16T00:45:15.918 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:15 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:15.666+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:15.918 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:15 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:15.666+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:15.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:15 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:15.666+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:15.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:15 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:15.666+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:15.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:15 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:15.666+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:15.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:15 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:15.666+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:16.523 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:16 smithi033 podman[137920]: 2024-09-16 00:45:16.217781712 +0000 UTC m=+5.674017072 container died 181e2246b1c04fac6ae8645d7e4640217a212aed8779dfb27b0412f924031b4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-16T00:45:16.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:16 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:16.388+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:16.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:16 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:16.388+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:16.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:16 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:16.388+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:16.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:16 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:16.388+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:16.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:16 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:16.388+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:16.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:16 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:16.388+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:16.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:16 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:16.388+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:50.565640+0000 front 2024-09-16T00:44:50.565608+0000 (oldest deadline 2024-09-16T00:45:12.865461+0000) 2024-09-16T00:45:16.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:16 smithi033 podman[137920]: 2024-09-16 00:45:16.714344571 +0000 UTC m=+6.170579931 container cleanup 181e2246b1c04fac6ae8645d7e4640217a212aed8779dfb27b0412f924031b4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-16T00:45:16.829 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:16 smithi033 bash[137920]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9 2024-09-16T00:45:17.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:16 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:16.683+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:17.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:16 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:16.683+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:17.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:16 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:16.683+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:17.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:16 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:16.683+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:17.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:16 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:16.683+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:17.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:16 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:16.683+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:17.329 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:16 smithi033 podman[137935]: 2024-09-16 00:45:16.999293199 +0000 UTC m=+0.770664600 container remove 181e2246b1c04fac6ae8645d7e4640217a212aed8779dfb27b0412f924031b4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-16T00:45:17.594 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:17 smithi033 podman[138031]: 2024-09-16 00:45:17.240292489 +0000 UTC m=+0.016269695 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:45:17.594 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:17 smithi033 podman[138031]: 2024-09-16 00:45:17.375402835 +0000 UTC m=+0.151380044 container create b716bc1a0b5ed4b2ea5dada08af318a01e05dc9e69596c024e5ee2952276f269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:45:17.594 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:17 smithi033 podman[138031]: 2024-09-16 00:45:17.594546228 +0000 UTC m=+0.370523434 container init b716bc1a0b5ed4b2ea5dada08af318a01e05dc9e69596c024e5ee2952276f269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:45:17.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:17 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:17.397+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:17.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:17 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:17.397+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:17.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:17 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:17.397+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:17.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:17 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:17.397+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:17.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:17 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:17.397+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:17.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:17 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:17.397+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:17.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:17 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:17.397+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:50.565640+0000 front 2024-09-16T00:44:50.565608+0000 (oldest deadline 2024-09-16T00:45:12.865461+0000) 2024-09-16T00:45:17.918 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:17 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:17.651+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:17.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:17 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:17.651+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:17.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:17 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:17.651+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:17.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:17 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:17.651+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:17.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:17 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:17.651+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:17.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:17 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:17.651+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:18.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:17 smithi033 podman[138031]: 2024-09-16 00:45:17.597986762 +0000 UTC m=+0.373963973 container start b716bc1a0b5ed4b2ea5dada08af318a01e05dc9e69596c024e5ee2952276f269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:45:18.079 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:17 smithi033 podman[138031]: 2024-09-16 00:45:17.677660548 +0000 UTC m=+0.453637770 container attach b716bc1a0b5ed4b2ea5dada08af318a01e05dc9e69596c024e5ee2952276f269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:45:18.080 INFO:journalctl@ceph.osd.9.smithi033.stdout:Sep 16 00:45:17 smithi033 podman[138031]: 2024-09-16 00:45:17.790731459 +0000 UTC m=+0.566708670 container died b716bc1a0b5ed4b2ea5dada08af318a01e05dc9e69596c024e5ee2952276f269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-16T00:45:18.378 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.9.service' 2024-09-16T00:45:18.421 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:45:18.421 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-16T00:45:18.421 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-16T00:45:18.421 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.10 2024-09-16T00:45:18.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:18 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:18.413+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:18.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:18 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:18.413+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:18.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:18 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:18.413+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:18.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:18 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:18.413+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:18.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:18 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:18.413+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:18.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:18 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:18.413+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:18.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:18 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:18.413+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:50.565640+0000 front 2024-09-16T00:44:50.565608+0000 (oldest deadline 2024-09-16T00:45:12.865461+0000) 2024-09-16T00:45:18.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:18 smithi060 systemd[1]: Stopping Ceph osd.10 for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:45:19.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:18 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:18.670+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:19.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:18 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:18.670+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:19.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:18 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:18.670+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:19.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:18 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:18.670+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:19.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:18 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:18.670+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:19.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:18 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:18.670+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:19.230 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:18 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:18.877+0000 7fafca0ca640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:45:19.230 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:18 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:18.877+0000 7fafca0ca640 -1 osd.10 94 *** Got signal Terminated *** 2024-09-16T00:45:19.230 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:18 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:18.877+0000 7fafca0ca640 -1 osd.10 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T00:45:19.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:19 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:19.445+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:19.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:19 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:19.445+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:19.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:19 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:19.445+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:19.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:19 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:19.445+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:19.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:19 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:19.445+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:19.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:19 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:19.445+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:19.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:19 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:19.445+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:50.565640+0000 front 2024-09-16T00:44:50.565608+0000 (oldest deadline 2024-09-16T00:45:12.865461+0000) 2024-09-16T00:45:19.918 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:19 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:19.646+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:19.918 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:19 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:19.646+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:19.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:19 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:19.646+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:19.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:19 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:19.646+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:19.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:19 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:19.646+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:19.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:19 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:19.646+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:20.980 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:20 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:20.483+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:20.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:20 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:20.483+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:20.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:20 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:20.483+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:20.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:20 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:20.483+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:20.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:20 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:20.483+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:20.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:20 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:20.483+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:20.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:20 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:20.483+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:50.565640+0000 front 2024-09-16T00:44:50.565608+0000 (oldest deadline 2024-09-16T00:45:12.865461+0000) 2024-09-16T00:45:21.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:20 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:20.678+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:21.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:20 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:20.678+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:21.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:20 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:20.678+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:21.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:20 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:20.678+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:21.170 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:20 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:20.678+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:21.170 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:20 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:20.678+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:21.170 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:20 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:20.678+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6820 osd.7 since back 2024-09-16T00:44:55.645353+0000 front 2024-09-16T00:44:55.645305+0000 (oldest deadline 2024-09-16T00:45:20.345226+0000) 2024-09-16T00:45:21.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:21 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:21.667+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:21.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:21 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:21.667+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:21.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:21 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:21.667+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:21.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:21 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:21.667+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:21.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:21 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:21.667+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:21.920 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:21 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:21.667+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:21.920 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:21 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:21.667+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6820 osd.7 since back 2024-09-16T00:44:55.645353+0000 front 2024-09-16T00:44:55.645305+0000 (oldest deadline 2024-09-16T00:45:20.345226+0000) 2024-09-16T00:45:21.980 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:21 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:21.484+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:21.980 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:21 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:21.484+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:21.980 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:21 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:21.484+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:21.980 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:21 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:21.484+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:21.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:21 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:21.484+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:21.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:21 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:21.484+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:21.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:21 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:21.484+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:50.565640+0000 front 2024-09-16T00:44:50.565608+0000 (oldest deadline 2024-09-16T00:45:12.865461+0000) 2024-09-16T00:45:22.980 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:22 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:22.496+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:22.980 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:22 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:22.496+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:22.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:22 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:22.496+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:22.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:22 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:22.496+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:22.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:22 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:22.496+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:22.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:22 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:22.496+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:22.981 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:22 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:22.496+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:50.565640+0000 front 2024-09-16T00:44:50.565608+0000 (oldest deadline 2024-09-16T00:45:12.865461+0000) 2024-09-16T00:45:23.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:22 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:22.669+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:23.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:22 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:22.669+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:23.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:22 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:22.669+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:23.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:22 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:22.669+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:23.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:22 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:22.669+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:23.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:22 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:22.669+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:23.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:22 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:22.669+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6820 osd.7 since back 2024-09-16T00:44:55.645353+0000 front 2024-09-16T00:44:55.645305+0000 (oldest deadline 2024-09-16T00:45:20.345226+0000) 2024-09-16T00:45:23.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:23 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:23.467+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:44:01.358981+0000 front 2024-09-16T00:44:01.359135+0000 (oldest deadline 2024-09-16T00:44:26.058991+0000) 2024-09-16T00:45:23.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:23 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:23.467+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6804 osd.1 since back 2024-09-16T00:44:06.059428+0000 front 2024-09-16T00:44:06.059377+0000 (oldest deadline 2024-09-16T00:44:31.959478+0000) 2024-09-16T00:45:23.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:23 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:23.467+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:18.361225+0000 front 2024-09-16T00:44:18.361256+0000 (oldest deadline 2024-09-16T00:44:43.061411+0000) 2024-09-16T00:45:23.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:23 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:23.467+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:23.062026+0000 front 2024-09-16T00:44:23.061978+0000 (oldest deadline 2024-09-16T00:44:47.762041+0000) 2024-09-16T00:45:23.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:23 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:23.467+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:33.563231+0000 front 2024-09-16T00:44:33.563208+0000 (oldest deadline 2024-09-16T00:44:58.263400+0000) 2024-09-16T00:45:23.731 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:23 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:23.467+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:41.764300+0000 front 2024-09-16T00:44:41.764367+0000 (oldest deadline 2024-09-16T00:45:07.664276+0000) 2024-09-16T00:45:23.732 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:23 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10[60079]: 2024-09-16T00:45:23.467+0000 7fafc26d8640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:50.565640+0000 front 2024-09-16T00:44:50.565608+0000 (oldest deadline 2024-09-16T00:45:12.865461+0000) 2024-09-16T00:45:24.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:23 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:23.689+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:24.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:23 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:23.689+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:24.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:23 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:23.689+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:24.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:23 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:23.689+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:24.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:23 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:23.689+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:24.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:23 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:23.689+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:24.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:23 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:23.689+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6820 osd.7 since back 2024-09-16T00:44:55.645353+0000 front 2024-09-16T00:44:55.645305+0000 (oldest deadline 2024-09-16T00:45:20.345226+0000) 2024-09-16T00:45:24.595 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:24 smithi060 podman[77241]: 2024-09-16 00:45:24.282767538 +0000 UTC m=+5.658912013 container died c6601bbb2bab60feffae536d16362a8b4373d90eb045ffd91636897c051ac12c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:45:24.980 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:24 smithi060 podman[77241]: 2024-09-16 00:45:24.67775688 +0000 UTC m=+6.053901344 container cleanup c6601bbb2bab60feffae536d16362a8b4373d90eb045ffd91636897c051ac12c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, ceph=True, io.buildah.version=1.37.2) 2024-09-16T00:45:24.980 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:24 smithi060 bash[77241]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10 2024-09-16T00:45:25.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:24 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:24.721+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:25.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:24 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:24.721+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:25.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:24 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:24.721+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:25.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:24 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:24.721+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:25.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:24 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:24.721+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:25.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:24 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:24.721+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:25.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:24 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:24.721+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6820 osd.7 since back 2024-09-16T00:44:55.645353+0000 front 2024-09-16T00:44:55.645305+0000 (oldest deadline 2024-09-16T00:45:20.345226+0000) 2024-09-16T00:45:25.316 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:25 smithi060 podman[77256]: 2024-09-16 00:45:25.003938017 +0000 UTC m=+0.676188961 container remove c6601bbb2bab60feffae536d16362a8b4373d90eb045ffd91636897c051ac12c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-16T00:45:25.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:25 smithi060 podman[77342]: 2024-09-16 00:45:25.218951586 +0000 UTC m=+0.016112397 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:45:25.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:25 smithi060 podman[77342]: 2024-09-16 00:45:25.321454968 +0000 UTC m=+0.118615814 container create 838204bf4860cc0be5789a298fc0180fc8a0421ab15a94344754c91df5d854df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e) 2024-09-16T00:45:25.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:25 smithi060 podman[77342]: 2024-09-16 00:45:25.54051926 +0000 UTC m=+0.337680061 container init 838204bf4860cc0be5789a298fc0180fc8a0421ab15a94344754c91df5d854df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10-deactivate, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-16T00:45:25.730 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:25 smithi060 podman[77342]: 2024-09-16 00:45:25.543863188 +0000 UTC m=+0.341023995 container start 838204bf4860cc0be5789a298fc0180fc8a0421ab15a94344754c91df5d854df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:45:26.026 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:25 smithi060 podman[77342]: 2024-09-16 00:45:25.759952285 +0000 UTC m=+0.557113098 container attach 838204bf4860cc0be5789a298fc0180fc8a0421ab15a94344754c91df5d854df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-16T00:45:26.026 INFO:journalctl@ceph.osd.10.smithi060.stdout:Sep 16 00:45:25 smithi060 podman[77342]: 2024-09-16 00:45:25.76015307 +0000 UTC m=+0.557313873 container died 838204bf4860cc0be5789a298fc0180fc8a0421ab15a94344754c91df5d854df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-8921c8e) 2024-09-16T00:45:26.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:25 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:25.720+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:26.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:25 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:25.720+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:26.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:25 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:25.720+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:26.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:25 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:25.720+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:26.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:25 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:25.720+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:26.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:25 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:25.720+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:26.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:25 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:25.720+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6820 osd.7 since back 2024-09-16T00:44:55.645353+0000 front 2024-09-16T00:44:55.645305+0000 (oldest deadline 2024-09-16T00:45:20.345226+0000) 2024-09-16T00:45:26.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.10.service' 2024-09-16T00:45:26.436 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:45:26.436 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-16T00:45:26.436 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-16T00:45:26.437 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.11 2024-09-16T00:45:26.728 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:26 smithi083 systemd[1]: Stopping Ceph osd.11 for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:45:27.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:26.728+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:27.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:26.728+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:27.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:26.728+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:27.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:26.728+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:27.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:26.728+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:27.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:26.728+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:27.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:26.728+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6820 osd.7 since back 2024-09-16T00:44:55.645353+0000 front 2024-09-16T00:44:55.645305+0000 (oldest deadline 2024-09-16T00:45:20.345226+0000) 2024-09-16T00:45:27.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:26.757+0000 7f86224b7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:45:27.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:26.757+0000 7f86224b7640 -1 osd.11 94 *** Got signal Terminated *** 2024-09-16T00:45:27.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:26 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:26.757+0000 7f86224b7640 -1 osd.11 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T00:45:28.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:27 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:27.704+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:28.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:27 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:27.704+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:28.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:27 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:27.704+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:28.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:27 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:27.704+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:28.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:27 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:27.704+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:28.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:27 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:27.704+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:28.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:27 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:27.704+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6820 osd.7 since back 2024-09-16T00:44:55.645353+0000 front 2024-09-16T00:44:55.645305+0000 (oldest deadline 2024-09-16T00:45:20.345226+0000) 2024-09-16T00:45:28.918 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:28 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:28.661+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:28.918 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:28 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:28.661+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:28.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:28 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:28.661+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:28.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:28 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:28.661+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:28.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:28 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:28.661+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:28.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:28 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:28.661+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:28.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:28 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:28.661+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6820 osd.7 since back 2024-09-16T00:44:55.645353+0000 front 2024-09-16T00:44:55.645305+0000 (oldest deadline 2024-09-16T00:45:20.345226+0000) 2024-09-16T00:45:28.919 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:28 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:28.661+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6820 osd.8 since back 2024-09-16T00:45:04.346561+0000 front 2024-09-16T00:45:04.346575+0000 (oldest deadline 2024-09-16T00:45:28.446574+0000) 2024-09-16T00:45:30.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:29 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:29.701+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:30.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:29 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:29.701+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:30.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:29 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:29.701+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:30.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:29 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:29.701+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:30.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:29 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:29.701+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:30.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:29 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:29.701+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:30.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:29 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:29.701+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6820 osd.7 since back 2024-09-16T00:44:55.645353+0000 front 2024-09-16T00:44:55.645305+0000 (oldest deadline 2024-09-16T00:45:20.345226+0000) 2024-09-16T00:45:30.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:29 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:29.701+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6820 osd.8 since back 2024-09-16T00:45:04.346561+0000 front 2024-09-16T00:45:04.346575+0000 (oldest deadline 2024-09-16T00:45:28.446574+0000) 2024-09-16T00:45:31.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:30 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:30.727+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-09-16T00:43:57.634539+0000 front 2024-09-16T00:43:57.634513+0000 (oldest deadline 2024-09-16T00:44:22.835398+0000) 2024-09-16T00:45:31.168 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:30 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:30.727+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6804 osd.2 since back 2024-09-16T00:44:15.637962+0000 front 2024-09-16T00:44:15.638033+0000 (oldest deadline 2024-09-16T00:44:40.938147+0000) 2024-09-16T00:45:31.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:30 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:30.727+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-09-16T00:44:26.139675+0000 front 2024-09-16T00:44:26.139613+0000 (oldest deadline 2024-09-16T00:44:47.239793+0000) 2024-09-16T00:45:31.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:30 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:30.727+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6812 osd.4 since back 2024-09-16T00:44:34.241645+0000 front 2024-09-16T00:44:34.241543+0000 (oldest deadline 2024-09-16T00:44:58.842038+0000) 2024-09-16T00:45:31.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:30 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:30.727+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6812 osd.5 since back 2024-09-16T00:44:38.842642+0000 front 2024-09-16T00:44:38.842522+0000 (oldest deadline 2024-09-16T00:45:03.542663+0000) 2024-09-16T00:45:31.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:30 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:30.727+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-09-16T00:44:46.844506+0000 front 2024-09-16T00:44:46.844555+0000 (oldest deadline 2024-09-16T00:45:11.544436+0000) 2024-09-16T00:45:31.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:30 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:30.727+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.60:6820 osd.7 since back 2024-09-16T00:44:55.645353+0000 front 2024-09-16T00:44:55.645305+0000 (oldest deadline 2024-09-16T00:45:20.345226+0000) 2024-09-16T00:45:31.169 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:30 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11[60241]: 2024-09-16T00:45:30.727+0000 7f861eacd640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.83:6820 osd.8 since back 2024-09-16T00:45:04.346561+0000 front 2024-09-16T00:45:04.346575+0000 (oldest deadline 2024-09-16T00:45:28.446574+0000) 2024-09-16T00:45:32.609 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:32 smithi083 podman[73705]: 2024-09-16 00:45:32.338993031 +0000 UTC m=+5.705075594 container died 49e47c89fb97e171792b32797b0b87aa41494c81626c19855677667c3ceed706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-16T00:45:32.918 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:32 smithi083 podman[73705]: 2024-09-16 00:45:32.724298736 +0000 UTC m=+6.090381298 container cleanup 49e47c89fb97e171792b32797b0b87aa41494c81626c19855677667c3ceed706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, io.buildah.version=1.37.2) 2024-09-16T00:45:32.918 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:32 smithi083 bash[73705]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11 2024-09-16T00:45:33.326 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:32 smithi083 podman[73720]: 2024-09-16 00:45:32.975599208 +0000 UTC m=+0.630284065 container remove 49e47c89fb97e171792b32797b0b87aa41494c81626c19855677667c3ceed706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-16T00:45:33.637 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:33 smithi083 podman[73806]: 2024-09-16 00:45:33.326364241 +0000 UTC m=+0.109211210 container create 51a05f11fdb75760216c00022894825001c1af85c3eba68bbbccbfd815a0f1e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2) 2024-09-16T00:45:33.637 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:33 smithi083 podman[73806]: 2024-09-16 00:45:33.233198005 +0000 UTC m=+0.016044938 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T00:45:33.637 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:33 smithi083 podman[73806]: 2024-09-16 00:45:33.545584119 +0000 UTC m=+0.328431042 container init 51a05f11fdb75760216c00022894825001c1af85c3eba68bbbccbfd815a0f1e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-16T00:45:33.637 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:33 smithi083 podman[73806]: 2024-09-16 00:45:33.549647083 +0000 UTC m=+0.332494008 container start 51a05f11fdb75760216c00022894825001c1af85c3eba68bbbccbfd815a0f1e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-16T00:45:33.637 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:33 smithi083 podman[73806]: 2024-09-16 00:45:33.637203482 +0000 UTC m=+0.420050420 container attach 51a05f11fdb75760216c00022894825001c1af85c3eba68bbbccbfd815a0f1e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e) 2024-09-16T00:45:33.918 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:33 smithi083 podman[73806]: 2024-09-16 00:45:33.734727903 +0000 UTC m=+0.517574841 container died 51a05f11fdb75760216c00022894825001c1af85c3eba68bbbccbfd815a0f1e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T00:45:34.276 INFO:journalctl@ceph.osd.11.smithi083.stdout:Sep 16 00:45:34 smithi083 podman[73829]: 2024-09-16 00:45:34.25723703 +0000 UTC m=+0.516340729 container remove 51a05f11fdb75760216c00022894825001c1af85c3eba68bbbccbfd815a0f1e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T00:45:34.281 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@osd.11.service' 2024-09-16T00:45:34.320 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:45:34.321 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-16T00:45:34.321 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-09-16T00:45:34.321 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.a 2024-09-16T00:45:34.792 INFO:journalctl@ceph.mds.a.smithi033.stdout:Sep 16 00:45:34 smithi033 systemd[1]: Stopping Ceph mds.a for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:45:35.079 INFO:journalctl@ceph.mds.a.smithi033.stdout:Sep 16 00:45:34 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-a[69575]: 2024-09-16T00:45:34.789+0000 7f9498114640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:45:35.080 INFO:journalctl@ceph.mds.a.smithi033.stdout:Sep 16 00:45:34 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-a[69575]: 2024-09-16T00:45:34.789+0000 7f9498114640 -1 mds.a *** got signal Terminated *** 2024-09-16T00:45:37.983 INFO:journalctl@ceph.mds.a.smithi033.stdout:Sep 16 00:45:37 smithi033 podman[138134]: 2024-09-16 00:45:37.691631103 +0000 UTC m=+3.168571094 container died 03ad602848ca3eed95b8345d96ed1c9a03f54f86ca7c7b5985b37b580c26ae32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-16T00:45:38.329 INFO:journalctl@ceph.mds.a.smithi033.stdout:Sep 16 00:45:38 smithi033 podman[138134]: 2024-09-16 00:45:38.157127607 +0000 UTC m=+3.634067597 container cleanup 03ad602848ca3eed95b8345d96ed1c9a03f54f86ca7c7b5985b37b580c26ae32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-16T00:45:38.330 INFO:journalctl@ceph.mds.a.smithi033.stdout:Sep 16 00:45:38 smithi033 bash[138134]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-a 2024-09-16T00:45:38.562 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.a.service' 2024-09-16T00:45:38.600 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:45:38.600 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-09-16T00:45:38.601 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-09-16T00:45:38.601 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.d 2024-09-16T00:45:39.079 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:45:38 smithi033 systemd[1]: Stopping Ceph mds.d for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:45:39.080 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:45:38 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-d[68391]: 2024-09-16T00:45:38.926+0000 7f119d6f6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:45:39.080 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:45:38 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-d[68391]: 2024-09-16T00:45:38.926+0000 7f119d6f6640 -1 mds.d *** got signal Terminated *** 2024-09-16T00:45:42.803 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:45:42 smithi033 podman[138268]: 2024-09-16 00:45:42.507772518 +0000 UTC m=+3.675699178 container died 7ed5f1632dbd749930b874544805379e3d78845343eaa0a5bc91886f6016484d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-d, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:45:43.079 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:45:42 smithi033 podman[138268]: 2024-09-16 00:45:42.977423912 +0000 UTC m=+4.145350628 container cleanup 7ed5f1632dbd749930b874544805379e3d78845343eaa0a5bc91886f6016484d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-d, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-8921c8e) 2024-09-16T00:45:43.079 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:45:42 smithi033 bash[138268]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-d 2024-09-16T00:45:43.417 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.d.service' 2024-09-16T00:45:43.439 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:45:43 smithi033 podman[138281]: 2024-09-16 00:45:43.18715612 +0000 UTC m=+0.680198129 container remove 7ed5f1632dbd749930b874544805379e3d78845343eaa0a5bc91886f6016484d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T00:45:43.439 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:45:43 smithi033 systemd[1]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.d.service: Deactivated successfully. 2024-09-16T00:45:43.439 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:45:43 smithi033 systemd[1]: Stopped Ceph mds.d for 7dde3d1a-73bf-11ef-bceb-c7b262605968. 2024-09-16T00:45:43.440 INFO:journalctl@ceph.mds.d.smithi033.stdout:Sep 16 00:45:43 smithi033 systemd[1]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.d.service: Consumed 1.234s CPU time. 2024-09-16T00:45:43.452 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:45:43.452 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-09-16T00:45:43.452 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-09-16T00:45:43.452 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.g 2024-09-16T00:45:43.790 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:45:43 smithi033 systemd[1]: Stopping Ceph mds.g for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:45:44.079 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:45:43 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-g[69179]: 2024-09-16T00:45:43.789+0000 7eff821ab640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:45:44.079 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:45:43 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-g[69179]: 2024-09-16T00:45:43.789+0000 7eff821ab640 -1 mds.g *** got signal Terminated *** 2024-09-16T00:45:46.329 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:45:46 smithi033 podman[138388]: 2024-09-16 00:45:46.11261472 +0000 UTC m=+2.436506376 container died 093a62828f9a2921f7182c120d9c7bc7104dd1faa113c82ab998ae14c86f13ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:45:46.824 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:45:46 smithi033 podman[138388]: 2024-09-16 00:45:46.573253954 +0000 UTC m=+2.897145643 container cleanup 093a62828f9a2921f7182c120d9c7bc7104dd1faa113c82ab998ae14c86f13ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-g, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T00:45:46.824 INFO:journalctl@ceph.mds.g.smithi033.stdout:Sep 16 00:45:46 smithi033 bash[138388]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-g 2024-09-16T00:45:46.999 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.g.service' 2024-09-16T00:45:47.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:45:47.033 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-09-16T00:45:47.033 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-09-16T00:45:47.034 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.j 2024-09-16T00:45:47.360 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:45:47 smithi033 systemd[1]: Stopping Ceph mds.j for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:45:47.829 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:45:47 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-j[68003]: 2024-09-16T00:45:47.359+0000 7f0dcb964640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:45:47.829 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:45:47 smithi033 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-j[68003]: 2024-09-16T00:45:47.359+0000 7f0dcb964640 -1 mds.j *** got signal Terminated *** 2024-09-16T00:45:51.068 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:45:50 smithi033 podman[138509]: 2024-09-16 00:45:50.787795938 +0000 UTC m=+3.520304924 container died fdf1e0d0550706c463a952074c0f3530768eb13a3d22b28e312276303f00b51f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-j, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e) 2024-09-16T00:45:51.329 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:45:51 smithi033 podman[138509]: 2024-09-16 00:45:51.260080868 +0000 UTC m=+3.992589855 container cleanup fdf1e0d0550706c463a952074c0f3530768eb13a3d22b28e312276303f00b51f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-j, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-16T00:45:51.329 INFO:journalctl@ceph.mds.j.smithi033.stdout:Sep 16 00:45:51 smithi033 bash[138509]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-j 2024-09-16T00:45:51.625 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.j.service' 2024-09-16T00:45:51.659 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:45:51.659 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-09-16T00:45:51.659 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-09-16T00:45:51.659 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.b 2024-09-16T00:45:51.979 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:45:51 smithi060 systemd[1]: Stopping Ceph mds.b for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:45:52.230 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:45:51 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b[65594]: 2024-09-16T00:45:51.978+0000 7fa55d650640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:45:52.230 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:45:51 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b[65594]: 2024-09-16T00:45:51.978+0000 7fa55d650640 -1 mds.b *** got signal Terminated *** 2024-09-16T00:45:56.395 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:45:55 smithi060 podman[77456]: 2024-09-16 00:45:55.993017318 +0000 UTC m=+4.124348929 container died 42c946fdaf398492d9f2b4e5ac2611c5613d7efe141d775602e43e5a4d164084 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-16T00:45:56.720 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:45:56 smithi060 podman[77456]: 2024-09-16 00:45:56.494411417 +0000 UTC m=+4.625743033 container cleanup 42c946fdaf398492d9f2b4e5ac2611c5613d7efe141d775602e43e5a4d164084 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:45:56.720 INFO:journalctl@ceph.mds.b.smithi060.stdout:Sep 16 00:45:56 smithi060 bash[77456]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-b 2024-09-16T00:45:56.872 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.b.service' 2024-09-16T00:45:56.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:45:56.907 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-09-16T00:45:56.907 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-09-16T00:45:56.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.e 2024-09-16T00:45:57.293 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:45:56 smithi060 systemd[1]: Stopping Ceph mds.e for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:45:57.730 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:45:57 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-e[64974]: 2024-09-16T00:45:57.290+0000 7f52ec898640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:45:57.730 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:45:57 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-e[64974]: 2024-09-16T00:45:57.290+0000 7f52ec898640 -1 mds.e *** got signal Terminated *** 2024-09-16T00:46:00.300 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:46:00 smithi060 podman[77578]: 2024-09-16 00:46:00.024197398 +0000 UTC m=+2.898902557 container died 34c56ec6ab56af859dbe470626758f4c8464f31dde68344abad222222c464491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-16T00:46:00.642 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:46:00 smithi060 podman[77578]: 2024-09-16 00:46:00.407179797 +0000 UTC m=+3.281884954 container cleanup 34c56ec6ab56af859dbe470626758f4c8464f31dde68344abad222222c464491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-16T00:46:00.642 INFO:journalctl@ceph.mds.e.smithi060.stdout:Sep 16 00:46:00 smithi060 bash[77578]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-e 2024-09-16T00:46:00.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.e.service' 2024-09-16T00:46:00.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:46:00.817 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-09-16T00:46:00.818 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-09-16T00:46:00.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.h 2024-09-16T00:46:01.161 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:46:00 smithi060 systemd[1]: Stopping Ceph mds.h for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:46:01.480 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:46:01 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-h[65278]: 2024-09-16T00:46:01.161+0000 7fe22570c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:46:01.480 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:46:01 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-h[65278]: 2024-09-16T00:46:01.161+0000 7fe22570c640 -1 mds.h *** got signal Terminated *** 2024-09-16T00:46:04.270 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:46:04 smithi060 podman[77711]: 2024-09-16 00:46:04.020470762 +0000 UTC m=+2.981392282 container died 4b6511ad6bcd02c165f491c012c6cb2a5a4c1df50a018a13881c0c2e27a3470a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-h, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T00:46:04.554 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:46:04 smithi060 podman[77711]: 2024-09-16 00:46:04.352740626 +0000 UTC m=+3.313662148 container cleanup 4b6511ad6bcd02c165f491c012c6cb2a5a4c1df50a018a13881c0c2e27a3470a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0) 2024-09-16T00:46:04.554 INFO:journalctl@ceph.mds.h.smithi060.stdout:Sep 16 00:46:04 smithi060 bash[77711]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-h 2024-09-16T00:46:04.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.h.service' 2024-09-16T00:46:04.814 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:46:04.814 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-09-16T00:46:04.814 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-09-16T00:46:04.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.k 2024-09-16T00:46:05.149 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:46:04 smithi060 systemd[1]: Stopping Ceph mds.k for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:46:05.480 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:46:05 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-k[64662]: 2024-09-16T00:46:05.148+0000 7f345d81d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:46:05.480 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:46:05 smithi060 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-k[64662]: 2024-09-16T00:46:05.148+0000 7f345d81d640 -1 mds.k *** got signal Terminated *** 2024-09-16T00:46:07.624 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:46:07 smithi060 podman[77830]: 2024-09-16 00:46:07.36079308 +0000 UTC m=+2.321320687 container died 30c350dd320bd9093927f2f0e4c1623d5a4cde0abf4a41287d59f1f18437ace3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-k, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True) 2024-09-16T00:46:07.980 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:46:07 smithi060 podman[77830]: 2024-09-16 00:46:07.723527926 +0000 UTC m=+2.684055524 container cleanup 30c350dd320bd9093927f2f0e4c1623d5a4cde0abf4a41287d59f1f18437ace3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-k, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-16T00:46:07.980 INFO:journalctl@ceph.mds.k.smithi060.stdout:Sep 16 00:46:07 smithi060 bash[77830]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-k 2024-09-16T00:46:08.161 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.k.service' 2024-09-16T00:46:08.193 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:46:08.194 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-09-16T00:46:08.194 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-09-16T00:46:08.194 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.c 2024-09-16T00:46:08.491 INFO:journalctl@ceph.mds.c.smithi083.stdout:Sep 16 00:46:08 smithi083 systemd[1]: Stopping Ceph mds.c for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:46:08.902 INFO:journalctl@ceph.mds.c.smithi083.stdout:Sep 16 00:46:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-c[65064]: 2024-09-16T00:46:08.490+0000 7facc477b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:46:08.902 INFO:journalctl@ceph.mds.c.smithi083.stdout:Sep 16 00:46:08 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-c[65064]: 2024-09-16T00:46:08.490+0000 7facc477b640 -1 mds.c *** got signal Terminated *** 2024-09-16T00:46:12.349 INFO:journalctl@ceph.mds.c.smithi083.stdout:Sep 16 00:46:12 smithi083 podman[73920]: 2024-09-16 00:46:12.066231294 +0000 UTC m=+3.676182944 container died 56133162cff8c987aee3ba87cf0aaa77d17d316a460f63b9e94e24e97a9012a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-16T00:46:12.668 INFO:journalctl@ceph.mds.c.smithi083.stdout:Sep 16 00:46:12 smithi083 podman[73920]: 2024-09-16 00:46:12.449183575 +0000 UTC m=+4.059135223 container cleanup 56133162cff8c987aee3ba87cf0aaa77d17d316a460f63b9e94e24e97a9012a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-c, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-16T00:46:12.668 INFO:journalctl@ceph.mds.c.smithi083.stdout:Sep 16 00:46:12 smithi083 bash[73920]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-c 2024-09-16T00:46:12.849 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.c.service' 2024-09-16T00:46:12.885 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:46:12.885 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-09-16T00:46:12.885 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-09-16T00:46:12.885 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.f 2024-09-16T00:46:13.418 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:46:12 smithi083 systemd[1]: Stopping Ceph mds.f for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:46:13.418 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:46:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-f[64761]: 2024-09-16T00:46:13.202+0000 7f4fa1a3f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:46:13.419 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:46:13 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-f[64761]: 2024-09-16T00:46:13.202+0000 7f4fa1a3f640 -1 mds.f *** got signal Terminated *** 2024-09-16T00:46:16.369 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:46:16 smithi083 podman[74054]: 2024-09-16 00:46:16.062527069 +0000 UTC m=+2.951901647 container died 2bb32a434d0ad4ca3d847bf74ab6604e0df2b8d28e4ad7f2f972ee5fee065f68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=squid-8921c8e) 2024-09-16T00:46:16.668 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:46:16 smithi083 podman[74054]: 2024-09-16 00:46:16.452553125 +0000 UTC m=+3.341927704 container cleanup 2bb32a434d0ad4ca3d847bf74ab6604e0df2b8d28e4ad7f2f972ee5fee065f68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-f, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e) 2024-09-16T00:46:16.668 INFO:journalctl@ceph.mds.f.smithi083.stdout:Sep 16 00:46:16 smithi083 bash[74054]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-f 2024-09-16T00:46:16.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.f.service' 2024-09-16T00:46:16.921 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:46:16.922 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-09-16T00:46:16.922 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-09-16T00:46:16.922 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.i 2024-09-16T00:46:17.287 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:46:17 smithi083 systemd[1]: Stopping Ceph mds.i for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:46:17.668 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:46:17 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i[65364]: 2024-09-16T00:46:17.286+0000 7f17b2b98640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:46:17.668 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:46:17 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i[65364]: 2024-09-16T00:46:17.286+0000 7f17b2b98640 -1 mds.i *** got signal Terminated *** 2024-09-16T00:46:20.364 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:46:20 smithi083 podman[74172]: 2024-09-16 00:46:20.061088548 +0000 UTC m=+2.915545281 container died 8b79c590acfae9b681cb1ee3798345307be38699bb9525336b1ca5d0508d6b84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-16T00:46:20.364 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:46:20 smithi083 podman[74172]: 2024-09-16 00:46:20.364898357 +0000 UTC m=+3.219355102 container cleanup 8b79c590acfae9b681cb1ee3798345307be38699bb9525336b1ca5d0508d6b84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-16T00:46:20.668 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:46:20 smithi083 bash[74172]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i 2024-09-16T00:46:20.668 INFO:journalctl@ceph.mds.i.smithi083.stdout:Sep 16 00:46:20 smithi083 podman[74186]: 2024-09-16 00:46:20.591221767 +0000 UTC m=+0.526731230 container remove 8b79c590acfae9b681cb1ee3798345307be38699bb9525336b1ca5d0508d6b84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-i, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-09-16T00:46:20.728 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.i.service' 2024-09-16T00:46:20.760 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:46:20.760 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-09-16T00:46:20.760 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-09-16T00:46:20.761 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.l 2024-09-16T00:46:21.068 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:46:20 smithi083 systemd[1]: Stopping Ceph mds.l for 7dde3d1a-73bf-11ef-bceb-c7b262605968... 2024-09-16T00:46:21.418 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:46:21 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-l[64460]: 2024-09-16T00:46:21.067+0000 7f0d87332640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T00:46:21.418 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:46:21 smithi083 ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-l[64460]: 2024-09-16T00:46:21.067+0000 7f0d87332640 -1 mds.l *** got signal Terminated *** 2024-09-16T00:46:24.918 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:46:24 smithi083 podman[74294]: 2024-09-16 00:46:24.622470567 +0000 UTC m=+3.648308766 container died cc64a2a63a5d6cb29cad91338e0ec552f7e3e99821a75d47dbb0973b63c1f977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-16T00:46:25.294 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:46:25 smithi083 podman[74294]: 2024-09-16 00:46:25.07705843 +0000 UTC m=+4.102896629 container cleanup cc64a2a63a5d6cb29cad91338e0ec552f7e3e99821a75d47dbb0973b63c1f977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-l, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-16T00:46:25.294 INFO:journalctl@ceph.mds.l.smithi083.stdout:Sep 16 00:46:25 smithi083 bash[74294]: ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968-mds-l 2024-09-16T00:46:25.506 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7dde3d1a-73bf-11ef-bceb-c7b262605968@mds.l.service' 2024-09-16T00:46:25.539 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T00:46:25.540 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-09-16T00:46:25.540 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T00:46:25.751 INFO:teuthology.orchestra.run.smithi033.stdout:Deleting cluster with fsid: 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:46:28.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T00:46:28.575 INFO:teuthology.orchestra.run.smithi060.stdout:Deleting cluster with fsid: 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:46:31.107 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T00:46:31.307 INFO:teuthology.orchestra.run.smithi083.stdout:Deleting cluster with fsid: 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T00:46:33.754 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T00:46:33.789 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T00:46:33.822 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T00:46:33.848 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-16T00:46:33.851 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375/remote/smithi033/crash 2024-09-16T00:46:33.852 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/crash -- . 2024-09-16T00:46:33.890 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T00:46:33.890 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2024-09-16T00:46:33.892 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375/remote/smithi060/crash 2024-09-16T00:46:33.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/crash -- . 2024-09-16T00:46:33.969 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T00:46:33.969 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2024-09-16T00:46:33.971 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375/remote/smithi083/crash 2024-09-16T00:46:33.972 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/crash -- . 2024-09-16T00:46:34.006 INFO:teuthology.orchestra.run.smithi083.stderr:tar: /var/lib/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T00:46:34.006 INFO:teuthology.orchestra.run.smithi083.stderr:tar: Error is not recoverable: exiting now 2024-09-16T00:46:34.007 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-16T00:46:34.008 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-16T00:46:34.069 INFO:tasks.cephadm:Compressing logs... 2024-09-16T00:46:34.069 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T00:46:34.072 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T00:46:34.075 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T00:46:34.098 INFO:teuthology.orchestra.run.smithi033.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-16T00:46:34.099 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-09-16T00:46:34.099 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-16T00:46:34.099 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.log: -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.9.log 2024-09-16T00:46:34.099 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T00:46:34.100 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.0.log: /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T00:46:34.101 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.a.log 2024-09-16T00:46:34.101 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T00:46:34.101 INFO:teuthology.orchestra.run.smithi033.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-09-16T00:46:34.103 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-16T00:46:34.103 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.g.log 2024-09-16T00:46:34.103 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-16T00:46:34.103 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.log 2024-09-16T00:46:34.103 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.d.log 2024-09-16T00:46:34.104 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-16T00:46:34.104 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.j.log 2024-09-16T00:46:34.104 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-16T00:46:34.104 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-16T00:46:34.105 INFO:teuthology.orchestra.run.smithi083.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-16T00:46:34.106 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T00:46:34.106 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T00:46:34.106 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-16T00:46:34.107 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.i.log 2024-09-16T00:46:34.107 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-volume.log: /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-16T00:46:34.107 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T00:46:34.109 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.j.log: /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 87.2% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T00:46:34.109 INFO:teuthology.orchestra.run.smithi033.stderr: 94.7% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.a.log.gz 2024-09-16T00:46:34.109 INFO:teuthology.orchestra.run.smithi033.stderr: 94.7% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.d.log.gz 2024-09-16T00:46:34.113 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.11.log 2024-09-16T00:46:34.113 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-16T00:46:34.113 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-16T00:46:34.113 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose --/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.2.log: /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.log 2024-09-16T00:46:34.113 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.l.log 2024-09-16T00:46:34.113 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.8.log 2024-09-16T00:46:34.114 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mgr.z.log 2024-09-16T00:46:34.114 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.l.log: /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.c.log 2024-09-16T00:46:34.114 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.f.log 2024-09-16T00:46:34.114 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.c.log: --verbose -- /var/log/ceph/cephadm.log 2024-09-16T00:46:34.114 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.10.log 2024-09-16T00:46:34.114 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-16T00:46:34.114 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-16T00:46:34.114 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.1.log: -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T00:46:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.b.log 2024-09-16T00:46:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.k.log 2024-09-16T00:46:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-16T00:46:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.b.log: /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T00:46:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T00:46:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-16T00:46:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.cephadm.log: -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.log 2024-09-16T00:46:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.h.log 2024-09-16T00:46:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-16T00:46:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.e.log 2024-09-16T00:46:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-16T00:46:34.173 INFO:teuthology.orchestra.run.smithi033.stderr: 90.3% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T00:46:34.183 INFO:teuthology.orchestra.run.smithi033.stderr: 83.0% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T00:46:34.186 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.e.log: /var/log/ceph/cephadm.log: 83.4% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T00:46:34.254 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.f.log: /var/log/ceph/cephadm.log: 94.7% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.l.log.gz 2024-09-16T00:46:34.256 INFO:teuthology.orchestra.run.smithi033.stderr: 94.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T00:46:34.268 INFO:teuthology.orchestra.run.smithi083.stderr: 93.5% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mgr.z.log.gz 2024-09-16T00:46:34.273 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-16T00:46:34.278 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T00:46:34.309 INFO:teuthology.orchestra.run.smithi083.stderr: 83.4% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T00:46:34.330 INFO:teuthology.orchestra.run.smithi033.stderr: 91.1% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T00:46:34.331 INFO:teuthology.orchestra.run.smithi060.stderr: 94.6% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.k.log.gz 2024-09-16T00:46:34.353 INFO:teuthology.orchestra.run.smithi033.stderr: 94.8% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.j.log.gz 2024-09-16T00:46:34.366 INFO:teuthology.orchestra.run.smithi083.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T00:46:34.387 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T00:46:34.389 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T00:46:34.415 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T00:46:34.418 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T00:46:34.574 INFO:teuthology.orchestra.run.smithi033.stderr: 94.7% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.g.log.gz 2024-09-16T00:46:34.584 INFO:teuthology.orchestra.run.smithi033.stderr: 90.6% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-16T00:46:34.991 INFO:teuthology.orchestra.run.smithi083.stderr: 90.9% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T00:46:34.992 INFO:teuthology.orchestra.run.smithi083.stderr: 90.5% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T00:46:35.828 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-16T00:46:38.445 INFO:teuthology.orchestra.run.smithi033.stderr: 92.2% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-16T00:46:48.088 INFO:teuthology.orchestra.run.smithi083.stderr: 91.7% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-16T00:47:21.169 INFO:teuthology.orchestra.run.smithi060.stderr: 95.4% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.h.log.gz 2024-09-16T00:47:27.051 INFO:teuthology.orchestra.run.smithi033.stderr: 91.6% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-16T00:47:35.374 INFO:teuthology.orchestra.run.smithi033.stderr: 91.1% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-16T00:47:38.069 INFO:teuthology.orchestra.run.smithi083.stderr: 95.2% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.f.log.gz 2024-09-16T00:47:38.243 INFO:teuthology.orchestra.run.smithi033.stderr: 91.2% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.9.log.gz 2024-09-16T00:47:38.418 INFO:teuthology.orchestra.run.smithi033.stderr: 91.2% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-16T00:47:38.420 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-09-16T00:47:38.420 INFO:teuthology.orchestra.run.smithi033.stderr:real 1m4.333s 2024-09-16T00:47:38.420 INFO:teuthology.orchestra.run.smithi033.stderr:user 3m29.191s 2024-09-16T00:47:38.420 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m6.410s 2024-09-16T00:48:00.401 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.10.log.gz 2024-09-16T00:48:05.757 INFO:teuthology.orchestra.run.smithi060.stderr: 94.8% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.e.log.gz 2024-09-16T00:48:23.536 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-16T00:48:25.466 INFO:teuthology.orchestra.run.smithi060.stderr: 93.8% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.b.log.gz 2024-09-16T00:49:02.605 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-16T00:49:12.610 INFO:teuthology.orchestra.run.smithi060.stderr: 91.2% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-16T00:49:12.612 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-16T00:49:12.612 INFO:teuthology.orchestra.run.smithi060.stderr:real 2m38.522s 2024-09-16T00:49:12.612 INFO:teuthology.orchestra.run.smithi060.stderr:user 4m24.315s 2024-09-16T00:49:12.612 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m8.562s 2024-09-16T00:57:41.476 INFO:teuthology.orchestra.run.smithi083.stderr: 91.1% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.8.log.gz 2024-09-16T00:57:45.830 INFO:teuthology.orchestra.run.smithi083.stderr: 91.3% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.11.log.gz 2024-09-16T00:59:30.153 INFO:teuthology.orchestra.run.smithi083.stderr: 91.3% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-16T01:01:16.600 INFO:teuthology.orchestra.run.smithi083.stderr: 91.0% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-16T01:11:36.595 INFO:teuthology.orchestra.run.smithi083.stderr: 93.3% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.i.log.gz 2024-09-16T01:17:28.601 INFO:teuthology.orchestra.run.smithi083.stderr: 93.2% -- replaced with /var/log/ceph/7dde3d1a-73bf-11ef-bceb-c7b262605968/ceph-mds.c.log.gz 2024-09-16T01:17:28.653 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-16T01:17:28.653 INFO:teuthology.orchestra.run.smithi083.stderr:real 30m54.561s 2024-09-16T01:17:28.653 INFO:teuthology.orchestra.run.smithi083.stderr:user 12m43.445s 2024-09-16T01:17:28.653 INFO:teuthology.orchestra.run.smithi083.stderr:sys 0m36.182s 2024-09-16T01:17:28.653 INFO:tasks.cephadm:Archiving logs... 2024-09-16T01:17:28.659 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375/remote/smithi033/log 2024-09-16T01:17:28.661 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T01:18:04.981 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375/remote/smithi060/log 2024-09-16T01:18:04.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T01:18:50.758 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375/remote/smithi083/log 2024-09-16T01:18:50.759 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T01:20:53.487 INFO:tasks.cephadm:Removing cluster... 2024-09-16T01:20:53.488 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 --force 2024-09-16T01:20:53.702 INFO:teuthology.orchestra.run.smithi033.stdout:Deleting cluster with fsid: 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T01:20:54.285 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 --force 2024-09-16T01:20:54.496 INFO:teuthology.orchestra.run.smithi060.stdout:Deleting cluster with fsid: 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T01:20:55.088 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7dde3d1a-73bf-11ef-bceb-c7b262605968 --force 2024-09-16T01:20:56.195 INFO:teuthology.orchestra.run.smithi083.stdout:Deleting cluster with fsid: 7dde3d1a-73bf-11ef-bceb-c7b262605968 2024-09-16T01:20:57.136 INFO:tasks.cephadm:Removing cephadm ... 2024-09-16T01:20:57.136 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T01:20:57.157 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T01:20:57.177 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T01:20:57.198 INFO:tasks.cephadm:Teardown complete 2024-09-16T01:20:57.199 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T01:20:57.211 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T01:20:57.211 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T01:20:57.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T01:20:57.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T01:20:57.349 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T01:20:57.411 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T01:20:57.412 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:20:57.415 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:20:57.417 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:20:57.430 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:20:57.432 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:20:57.432 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:20:57.435 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:20:57.435 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-09-16T01:20:57.435 INFO:teuthology.orchestra.run.smithi033.stdout:^* hv01.front.sepia.ceph.com 3 6 377 47 -357us[ -356us] +/- 47ms 2024-09-16T01:20:57.435 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 44 -170us[ -170us] +/- 54ms 2024-09-16T01:20:57.436 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +948us[ +948us] +/- 76ms 2024-09-16T01:20:57.436 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:20:57.462 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:20:57.462 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-09-16T01:20:57.462 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv01.front.sepia.ceph.com 3 6 377 46 -378us[ -373us] +/- 47ms 2024-09-16T01:20:57.462 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 47 -178us[ -173us] +/- 54ms 2024-09-16T01:20:57.462 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +962us[ +962us] +/- 76ms 2024-09-16T01:20:57.462 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:20:57.515 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:20:57.515 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-09-16T01:20:57.515 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 -391us[ -438us] +/- 47ms 2024-09-16T01:20:57.515 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 49 -200us[ -244us] +/- 53ms 2024-09-16T01:20:57.515 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +924us[ +924us] +/- 76ms 2024-09-16T01:20:57.515 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:20:57.531 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T01:20:57.542 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T01:20:57.542 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T01:20:57.551 DEBUG:teuthology.orchestra.run.smithi033:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T01:20:57.721 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T01:20:57.812 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T01:20:57.956 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:20:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:20:57.992 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:20:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:20:58.024 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:20:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:20:58.052 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T01:20:58.063 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T01:20:58.073 INFO:teuthology.task.internal:Duration was 4827.533390 seconds 2024-09-16T01:20:58.073 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T01:20:58.082 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T01:20:58.082 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T01:20:58.085 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T01:20:58.088 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T01:20:58.137 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:20:58.162 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:20:58.167 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:20:58.549 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T01:20:58.549 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2024-09-16T01:20:58.549 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T01:20:58.573 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-09-16T01:20:58.573 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T01:20:58.605 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-09-16T01:20:58.606 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T01:20:58.633 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T01:20:58.634 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T01:20:58.636 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T01:20:58.648 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T01:20:58.712 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T01:20:58.712 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T01:20:58.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T01:20:58.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T01:21:00.924 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T01:21:01.000 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T01:21:01.000 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T01:21:01.026 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T01:21:01.074 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T01:21:01.119 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T01:21:01.133 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:21:01.138 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:21:01.142 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:21:01.160 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2024-09-16T01:21:01.171 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-09-16T01:21:01.192 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-09-16T01:21:01.227 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:21:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:21:01.242 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:21:01.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:21:01.258 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:21:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:21:01.281 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T01:21:01.293 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T01:21:01.295 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375/remote/smithi033 2024-09-16T01:21:01.295 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T01:21:01.382 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375/remote/smithi060 2024-09-16T01:21:01.382 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T01:21:01.424 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906375/remote/smithi083 2024-09-16T01:21:01.424 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T01:21:01.462 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T01:21:01.462 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T01:21:01.465 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T01:21:01.467 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T01:21:01.510 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T01:21:01.524 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T01:21:01.524 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T01:21:01.535 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T01:21:01.535 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T01:21:01.538 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T01:21:01.540 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T01:21:01.551 INFO:teuthology.orchestra.run.smithi033.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 01:21 /home/ubuntu/cephtest 2024-09-16T01:21:01.554 INFO:teuthology.orchestra.run.smithi060.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 01:21 /home/ubuntu/cephtest 2024-09-16T01:21:01.565 INFO:teuthology.orchestra.run.smithi083.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 01:21 /home/ubuntu/cephtest 2024-09-16T01:21:01.566 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T01:21:01.577 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T01:21:01.611 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 4827.533389568329 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T01:21:01.612 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T01:21:01.712 INFO:teuthology.run:pass